2024-06-03T08:17:56.730 INFO:root:teuthology version: 0.0.1.dev303+g861a8dc 2024-06-03T08:17:56.730 DEBUG:teuthology.run:Teuthology command: teuthology --archive /home/teuthworker/archive/teuthology-2024-06-02_22:00:03-rados-reef-distro-default-smithi/7739049 --description rados/verify/{centos_latest ceph clusters/{fixed-2 openstack} d-thrash/none mon_election/connectivity msgr-failures/few msgr/async-v1only objectstore/bluestore-stupid rados tasks/rados_api_tests validater/valgrind} --name teuthology-2024-06-02_22:00:03-rados-reef-distro-default-smithi --verbose --owner scheduled_teuthology@teuthology -- /home/teuthworker/archive/teuthology-2024-06-02_22:00:03-rados-reef-distro-default-smithi/7739049/orig.config.yaml 2024-06-03T08:17:56.770 DEBUG:teuthology.report:Pushing job info to https://paddles.front.sepia.ceph.com/ 2024-06-03T08:17:56.855 INFO:teuthology.run:Config: archive_path: /home/teuthworker/archive/teuthology-2024-06-02_22:00:03-rados-reef-distro-default-smithi/7739049 branch: reef description: rados/verify/{centos_latest ceph clusters/{fixed-2 openstack} d-thrash/none mon_election/connectivity msgr-failures/few msgr/async-v1only objectstore/bluestore-stupid rados tasks/rados_api_tests validater/valgrind} email: ceph-qa@ceph.com first_in_suite: false job_id: '7739049' kernel: kdb: 1 sha1: distro last_in_suite: false machine_type: smithi name: teuthology-2024-06-02_22:00:03-rados-reef-distro-default-smithi no_nested_subset: false openstack: - volumes: count: 4 size: 10 - volumes: count: 4 size: 10 os_type: centos os_version: 9.stream overrides: admin_socket: branch: reef ceph: conf: client: debug monc: 20 debug ms: 1 debug objecter: 20 debug rados: 20 global: mon client directed command retry: 5 mon election default strategy: 3 ms bind msgr2: false ms inject socket failures: 5000 ms type: async osd heartbeat grace: 80 osd op thread timeout: 150 mgr: debug mgr: 20 debug ms: 1 mon: debug mon: 20 debug ms: 1 debug paxos: 20 mon min osdmap epochs: 50 mon osd crush smoke test: false mon osdmap full prune interval: 2 mon osdmap full prune min: 15 mon osdmap full prune txsize: 2 mon scrub interval: 300 mon warn on pool no app: false paxos service trim min: 10 osd: bdev async discard: true bdev enable discard: true bluestore allocator: stupid bluestore block size: 96636764160 bluestore fsck on mount: true bluestore zero block detection: true debug bluefs: 1 debug bluestore: 1 debug monc: 20 debug ms: 1 debug osd: 20 debug rocksdb: 10 mon osd backfillfull_ratio: 0.85 mon osd full ratio: 0.9 mon osd nearfull ratio: 0.8 osd class default list: '*' osd class load list: '*' osd client watch timeout: 120 osd debug verify cached snaps: true osd debug verify missing on start: true osd failsafe full ratio: 0.95 osd fast shutdown: false osd mclock override recovery settings: true osd mclock profile: high_recovery_ops osd objectstore: bluestore osd op queue: debug_random osd op queue cut off: debug_random osd shutdown pgref assert: true flavor: default fs: xfs log-ignorelist: - \(MDS_ALL_DOWN\) - \(MDS_UP_LESS_THAN_MAX\) - \(OSD_SLOW_PING_TIME - reached quota - overall HEALTH_ - \(CACHE_POOL_NO_HIT_SET\) - \(POOL_FULL\) - \(SMALLER_PGP_NUM\) - \(SLOW_OPS\) - \(CACHE_POOL_NEAR_FULL\) - \(POOL_APP_NOT_ENABLED\) - \(PG_AVAILABILITY\) - \(OBJECT_MISPLACED\) - slow request - overall HEALTH_ - \(PG_ - \(MON_DOWN\) - \(SLOW_OPS\) - slow request mon_bind_msgr2: false sha1: f1abf795f4fc15bf2f30ac90c811a53ab21d5d65 valgrind: mds: - --tool=memcheck mon: - --tool=memcheck - --leak-check=full - --show-reachable=yes osd: - --tool=memcheck ceph-deploy: bluestore: true conf: client: log file: /var/log/ceph/ceph-$name.$pid.log mon: {} osd: bdev async discard: true bdev enable discard: true bluestore block size: 96636764160 bluestore fsck on mount: true debug bluefs: 20 debug bluestore: 20 debug rocksdb: 10 mon osd backfillfull_ratio: 0.85 mon osd full ratio: 0.9 mon osd nearfull ratio: 0.8 osd failsafe full ratio: 0.95 osd objectstore: bluestore fs: xfs install: ceph: debuginfo: true flavor: default sha1: f1abf795f4fc15bf2f30ac90c811a53ab21d5d65 thrashosds: bdev_inject_crash: 2 bdev_inject_crash_probability: 0.5 workunit: branch: reef sha1: f1abf795f4fc15bf2f30ac90c811a53ab21d5d65 owner: scheduled_teuthology@teuthology priority: 931 repo: https://git.ceph.com/ceph.git roles: - - mon.a - mon.c - mgr.y - osd.0 - osd.1 - osd.2 - osd.3 - client.0 - node-exporter.a - - mon.b - mgr.x - osd.4 - osd.5 - osd.6 - osd.7 - client.1 - prometheus.a - node-exporter.b seed: 6006 sha1: f1abf795f4fc15bf2f30ac90c811a53ab21d5d65 sleep_before_teardown: 0 subset: 1861/100000 suite: rados suite_branch: reef suite_path: /home/teuthworker/src/git.ceph.com_ceph_f1abf795f4fc15bf2f30ac90c811a53ab21d5d65/qa suite_relpath: qa suite_repo: https://git.ceph.com/ceph.git suite_sha1: f1abf795f4fc15bf2f30ac90c811a53ab21d5d65 targets: smithi088.front.sepia.ceph.com: ecdsa-sha2-nistp256 AAAAE2VjZHNhLXNoYTItbmlzdHAyNTYAAAAIbmlzdHAyNTYAAABBBN5r+pnqR9ZU5/EEmFtgISdvjW+oxZ59NEzqi3i5Mif/oZX9TtsPQCe4xqEQYbrq3l+HbFDgc7ykrEhar97ndF0= smithi192.front.sepia.ceph.com: ecdsa-sha2-nistp256 AAAAE2VjZHNhLXNoYTItbmlzdHAyNTYAAAAIbmlzdHAyNTYAAABBBOonkgHrAzRMX4IChrWrragyGABZjlHr7O8NAeLBaTiHcz1SyKGXxzg2Mlqd38WgrTD784FoVD4YaVw9nmM2wRU= tasks: - install: null - ceph: null - workunit: clients: client.0: - rados/test.sh env: ALLOW_TIMEOUTS: '1' timeout: 6h teuthology: fragments_dropped: [] meta: {} postmerge: [] teuthology_branch: main teuthology_sha1: 861a8dcf7aa816a26e13f039336f7ed0a9aec0fa timestamp: 2024-06-02_22:00:03 tube: smithi user: teuthology verbose: false worker_log: /home/teuthworker/archive/worker_logs/dispatcher.smithi.2698841 2024-06-03T08:17:56.855 INFO:teuthology.run:suite_path is set to /home/teuthworker/src/git.ceph.com_ceph_f1abf795f4fc15bf2f30ac90c811a53ab21d5d65/qa; will attempt to use it 2024-06-03T08:17:56.856 INFO:teuthology.run:Found tasks at /home/teuthworker/src/git.ceph.com_ceph_f1abf795f4fc15bf2f30ac90c811a53ab21d5d65/qa/tasks 2024-06-03T08:17:56.856 INFO:teuthology.run_tasks:Running task internal.check_packages... 2024-06-03T08:17:56.857 INFO:teuthology.task.internal:Checking packages... 2024-06-03T08:17:56.877 INFO:teuthology.task.internal:Checking packages for os_type 'centos', flavor 'default' and ceph hash 'f1abf795f4fc15bf2f30ac90c811a53ab21d5d65' 2024-06-03T08:17:56.877 WARNING:teuthology.packaging:More than one of ref, tag, branch, or sha1 supplied; using branch 2024-06-03T08:17:56.877 INFO:teuthology.packaging:ref: None 2024-06-03T08:17:56.878 INFO:teuthology.packaging:tag: None 2024-06-03T08:17:56.878 INFO:teuthology.packaging:branch: reef 2024-06-03T08:17:56.878 INFO:teuthology.packaging:sha1: f1abf795f4fc15bf2f30ac90c811a53ab21d5d65 2024-06-03T08:17:56.878 DEBUG:teuthology.packaging:Querying https://shaman.ceph.com/api/search?status=ready&project=ceph&flavor=default&distros=centos%2F9%2Fx86_64&ref=reef 2024-06-03T08:17:57.137 INFO:teuthology.task.internal:Found packages for ceph version 18.2.2-1389.gf1abf795 2024-06-03T08:17:57.138 INFO:teuthology.run_tasks:Running task internal.buildpackages_prep... 2024-06-03T08:17:57.145 INFO:teuthology.task.internal:no buildpackages task found 2024-06-03T08:17:57.145 INFO:teuthology.run_tasks:Running task internal.save_config... 2024-06-03T08:17:57.151 INFO:teuthology.task.internal:Saving configuration 2024-06-03T08:17:57.165 INFO:teuthology.run_tasks:Running task internal.check_lock... 2024-06-03T08:17:57.171 INFO:teuthology.task.internal.check_lock:Checking locks... 2024-06-03T08:17:57.196 DEBUG:teuthology.task.internal.check_lock:machine status is {'name': 'smithi088.front.sepia.ceph.com', 'description': '/home/teuthworker/archive/teuthology-2024-06-02_22:00:03-rados-reef-distro-default-smithi/7739049', 'up': True, 'machine_type': 'smithi', 'is_vm': False, 'vm_host': None, 'os_type': 'centos', 'os_version': '9', 'arch': 'x86_64', 'locked': True, 'locked_since': '2024-06-03 08:10:48.730102', 'locked_by': 'scheduled_teuthology@teuthology', 'mac_address': None, 'ssh_pub_key': 'ecdsa-sha2-nistp256 AAAAE2VjZHNhLXNoYTItbmlzdHAyNTYAAAAIbmlzdHAyNTYAAABBBN5r+pnqR9ZU5/EEmFtgISdvjW+oxZ59NEzqi3i5Mif/oZX9TtsPQCe4xqEQYbrq3l+HbFDgc7ykrEhar97ndF0='} 2024-06-03T08:17:57.218 DEBUG:teuthology.task.internal.check_lock:machine status is {'name': 'smithi192.front.sepia.ceph.com', 'description': '/home/teuthworker/archive/teuthology-2024-06-02_22:00:03-rados-reef-distro-default-smithi/7739049', 'up': True, 'machine_type': 'smithi', 'is_vm': False, 'vm_host': None, 'os_type': 'centos', 'os_version': '9', 'arch': 'x86_64', 'locked': True, 'locked_since': '2024-06-03 08:10:48.728780', 'locked_by': 'scheduled_teuthology@teuthology', 'mac_address': None, 'ssh_pub_key': 'ecdsa-sha2-nistp256 AAAAE2VjZHNhLXNoYTItbmlzdHAyNTYAAAAIbmlzdHAyNTYAAABBBOonkgHrAzRMX4IChrWrragyGABZjlHr7O8NAeLBaTiHcz1SyKGXxzg2Mlqd38WgrTD784FoVD4YaVw9nmM2wRU='} 2024-06-03T08:17:57.218 INFO:teuthology.run_tasks:Running task internal.add_remotes... 2024-06-03T08:17:57.223 INFO:teuthology.task.internal:roles: ubuntu@smithi088.front.sepia.ceph.com - ['mon.a', 'mon.c', 'mgr.y', 'osd.0', 'osd.1', 'osd.2', 'osd.3', 'client.0', 'node-exporter.a'] 2024-06-03T08:17:57.223 INFO:teuthology.task.internal:roles: ubuntu@smithi192.front.sepia.ceph.com - ['mon.b', 'mgr.x', 'osd.4', 'osd.5', 'osd.6', 'osd.7', 'client.1', 'prometheus.a', 'node-exporter.b'] 2024-06-03T08:17:57.223 INFO:teuthology.run_tasks:Running task console_log... 2024-06-03T08:17:57.281 DEBUG:teuthology.exit:Installing handler: Handler(exiter=, func=.kill_console_loggers at 0x7fd397c93ac0>, signals=[15]) 2024-06-03T08:17:57.281 INFO:teuthology.run_tasks:Running task internal.connect... 2024-06-03T08:17:57.287 INFO:teuthology.task.internal:Opening connections... 2024-06-03T08:17:57.287 DEBUG:teuthology.task.internal:connecting to ubuntu@smithi088.front.sepia.ceph.com 2024-06-03T08:17:57.288 DEBUG:teuthology.orchestra.connection:{'hostname': 'smithi088.front.sepia.ceph.com', 'username': 'ubuntu', 'timeout': 60} 2024-06-03T08:17:57.370 DEBUG:teuthology.task.internal:connecting to ubuntu@smithi192.front.sepia.ceph.com 2024-06-03T08:17:57.371 DEBUG:teuthology.orchestra.connection:{'hostname': 'smithi192.front.sepia.ceph.com', 'username': 'ubuntu', 'timeout': 60} 2024-06-03T08:17:57.454 INFO:teuthology.run_tasks:Running task internal.push_inventory... 2024-06-03T08:17:57.461 DEBUG:teuthology.orchestra.run.smithi088:> uname -m 2024-06-03T08:17:57.479 INFO:teuthology.orchestra.run.smithi088.stdout:x86_64 2024-06-03T08:17:57.480 DEBUG:teuthology.orchestra.run.smithi088:> cat /etc/os-release 2024-06-03T08:17:57.536 INFO:teuthology.orchestra.run.smithi088.stdout:NAME="CentOS Stream" 2024-06-03T08:17:57.537 INFO:teuthology.orchestra.run.smithi088.stdout:VERSION="9" 2024-06-03T08:17:57.537 INFO:teuthology.orchestra.run.smithi088.stdout:ID="centos" 2024-06-03T08:17:57.537 INFO:teuthology.orchestra.run.smithi088.stdout:ID_LIKE="rhel fedora" 2024-06-03T08:17:57.537 INFO:teuthology.orchestra.run.smithi088.stdout:VERSION_ID="9" 2024-06-03T08:17:57.537 INFO:teuthology.orchestra.run.smithi088.stdout:PLATFORM_ID="platform:el9" 2024-06-03T08:17:57.537 INFO:teuthology.orchestra.run.smithi088.stdout:PRETTY_NAME="CentOS Stream 9" 2024-06-03T08:17:57.537 INFO:teuthology.orchestra.run.smithi088.stdout:ANSI_COLOR="0;31" 2024-06-03T08:17:57.537 INFO:teuthology.orchestra.run.smithi088.stdout:LOGO="fedora-logo-icon" 2024-06-03T08:17:57.537 INFO:teuthology.orchestra.run.smithi088.stdout:CPE_NAME="cpe:/o:centos:centos:9" 2024-06-03T08:17:57.537 INFO:teuthology.orchestra.run.smithi088.stdout:HOME_URL="https://centos.org/" 2024-06-03T08:17:57.537 INFO:teuthology.orchestra.run.smithi088.stdout:BUG_REPORT_URL="https://bugzilla.redhat.com/" 2024-06-03T08:17:57.537 INFO:teuthology.orchestra.run.smithi088.stdout:REDHAT_SUPPORT_PRODUCT="Red Hat Enterprise Linux 9" 2024-06-03T08:17:57.538 INFO:teuthology.orchestra.run.smithi088.stdout:REDHAT_SUPPORT_PRODUCT_VERSION="CentOS Stream" 2024-06-03T08:17:57.538 INFO:teuthology.lock.ops:Updating smithi088.front.sepia.ceph.com on lock server 2024-06-03T08:17:57.561 DEBUG:teuthology.orchestra.run.smithi192:> uname -m 2024-06-03T08:17:57.579 INFO:teuthology.orchestra.run.smithi192.stdout:x86_64 2024-06-03T08:17:57.579 DEBUG:teuthology.orchestra.run.smithi192:> cat /etc/os-release 2024-06-03T08:17:57.635 INFO:teuthology.orchestra.run.smithi192.stdout:NAME="CentOS Stream" 2024-06-03T08:17:57.636 INFO:teuthology.orchestra.run.smithi192.stdout:VERSION="9" 2024-06-03T08:17:57.636 INFO:teuthology.orchestra.run.smithi192.stdout:ID="centos" 2024-06-03T08:17:57.636 INFO:teuthology.orchestra.run.smithi192.stdout:ID_LIKE="rhel fedora" 2024-06-03T08:17:57.636 INFO:teuthology.orchestra.run.smithi192.stdout:VERSION_ID="9" 2024-06-03T08:17:57.636 INFO:teuthology.orchestra.run.smithi192.stdout:PLATFORM_ID="platform:el9" 2024-06-03T08:17:57.636 INFO:teuthology.orchestra.run.smithi192.stdout:PRETTY_NAME="CentOS Stream 9" 2024-06-03T08:17:57.636 INFO:teuthology.orchestra.run.smithi192.stdout:ANSI_COLOR="0;31" 2024-06-03T08:17:57.636 INFO:teuthology.orchestra.run.smithi192.stdout:LOGO="fedora-logo-icon" 2024-06-03T08:17:57.636 INFO:teuthology.orchestra.run.smithi192.stdout:CPE_NAME="cpe:/o:centos:centos:9" 2024-06-03T08:17:57.637 INFO:teuthology.orchestra.run.smithi192.stdout:HOME_URL="https://centos.org/" 2024-06-03T08:17:57.637 INFO:teuthology.orchestra.run.smithi192.stdout:BUG_REPORT_URL="https://bugzilla.redhat.com/" 2024-06-03T08:17:57.637 INFO:teuthology.orchestra.run.smithi192.stdout:REDHAT_SUPPORT_PRODUCT="Red Hat Enterprise Linux 9" 2024-06-03T08:17:57.637 INFO:teuthology.orchestra.run.smithi192.stdout:REDHAT_SUPPORT_PRODUCT_VERSION="CentOS Stream" 2024-06-03T08:17:57.637 INFO:teuthology.lock.ops:Updating smithi192.front.sepia.ceph.com on lock server 2024-06-03T08:17:57.661 INFO:teuthology.run_tasks:Running task internal.serialize_remote_roles... 2024-06-03T08:17:57.670 INFO:teuthology.run_tasks:Running task internal.check_conflict... 2024-06-03T08:17:57.675 INFO:teuthology.task.internal:Checking for old test directory... 2024-06-03T08:17:57.675 DEBUG:teuthology.orchestra.run.smithi088:> test '!' -e /home/ubuntu/cephtest 2024-06-03T08:17:57.678 DEBUG:teuthology.orchestra.run.smithi192:> test '!' -e /home/ubuntu/cephtest 2024-06-03T08:17:57.693 INFO:teuthology.run_tasks:Running task internal.check_ceph_data... 2024-06-03T08:17:57.702 INFO:teuthology.task.internal:Checking for non-empty /var/lib/ceph... 2024-06-03T08:17:57.703 DEBUG:teuthology.orchestra.run.smithi088:> test -z $(ls -A /var/lib/ceph) 2024-06-03T08:17:57.733 DEBUG:teuthology.orchestra.run.smithi192:> test -z $(ls -A /var/lib/ceph) 2024-06-03T08:17:57.851 INFO:teuthology.run_tasks:Running task internal.vm_setup... 2024-06-03T08:17:57.905 INFO:teuthology.run_tasks:Running task kernel... 2024-06-03T08:17:57.919 INFO:teuthology.task.kernel:normalize config orig: {'kdb': 1, 'sha1': 'distro'} 2024-06-03T08:17:57.919 INFO:teuthology.task.kernel:config {'mon.a': {'kdb': 1, 'sha1': 'distro'}, 'mon.b': {'kdb': 1, 'sha1': 'distro'}}, timeout 300 2024-06-03T08:17:57.919 DEBUG:teuthology.orchestra.run.smithi088:> test -f /run/.containerenv -o -f /.dockerenv 2024-06-03T08:17:57.920 DEBUG:teuthology.orchestra.run.smithi192:> test -f /run/.containerenv -o -f /.dockerenv 2024-06-03T08:17:57.936 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-03T08:17:57.936 DEBUG:teuthology.orchestra.run.smithi088:> uname -r 2024-06-03T08:17:57.937 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-03T08:17:57.937 DEBUG:teuthology.orchestra.run.smithi192:> uname -r 2024-06-03T08:17:57.992 INFO:teuthology.orchestra.run.smithi088.stdout:5.14.0-437.el9.x86_64 2024-06-03T08:17:57.992 INFO:teuthology.task.kernel:Running kernel on smithi088: 5.14.0-437.el9.x86_64 2024-06-03T08:17:57.992 DEBUG:teuthology.orchestra.run.smithi088:> sudo yum install -y kernel 2024-06-03T08:17:57.994 INFO:teuthology.orchestra.run.smithi192.stdout:5.14.0-437.el9.x86_64 2024-06-03T08:17:57.995 INFO:teuthology.task.kernel:Running kernel on smithi192: 5.14.0-437.el9.x86_64 2024-06-03T08:17:57.995 DEBUG:teuthology.orchestra.run.smithi192:> sudo yum install -y kernel 2024-06-03T08:18:00.273 INFO:teuthology.orchestra.run.smithi088.stdout:CentOS Stream 9 - BaseOS 13 MB/s | 8.1 MB 00:00 2024-06-03T08:18:02.157 INFO:teuthology.orchestra.run.smithi088.stdout:CentOS Stream 9 - AppStream 25 MB/s | 19 MB 00:00 2024-06-03T08:18:04.312 INFO:teuthology.orchestra.run.smithi192.stdout:CentOS Stream 9 - BaseOS 1.7 MB/s | 8.1 MB 00:04 2024-06-03T08:18:06.021 INFO:teuthology.orchestra.run.smithi192.stdout:CentOS Stream 9 - AppStream 31 MB/s | 19 MB 00:00 2024-06-03T08:18:06.865 INFO:teuthology.orchestra.run.smithi088.stdout:CentOS Stream 9 - CRB 5.5 MB/s | 6.2 MB 00:01 2024-06-03T08:18:08.449 INFO:teuthology.orchestra.run.smithi088.stdout:CentOS Stream 9 - Extras packages 66 kB/s | 17 kB 00:00 2024-06-03T08:18:10.067 INFO:teuthology.orchestra.run.smithi088.stdout:Extra Packages for Enterprise Linux 14 MB/s | 21 MB 00:01 2024-06-03T08:18:10.130 INFO:teuthology.orchestra.run.smithi192.stdout:CentOS Stream 9 - CRB 13 MB/s | 6.2 MB 00:00 2024-06-03T08:18:12.339 INFO:teuthology.orchestra.run.smithi192.stdout:CentOS Stream 9 - Extras packages 16 kB/s | 17 kB 00:01 2024-06-03T08:18:13.760 INFO:teuthology.orchestra.run.smithi192.stdout:Extra Packages for Enterprise Linux 17 MB/s | 21 MB 00:01 2024-06-03T08:18:15.300 INFO:teuthology.orchestra.run.smithi088.stdout:lab-extras 26 kB/s | 1.7 kB 00:00 2024-06-03T08:18:17.314 INFO:teuthology.orchestra.run.smithi088.stdout:Package kernel-5.14.0-239.el9.x86_64 is already installed. 2024-06-03T08:18:17.315 INFO:teuthology.orchestra.run.smithi088.stdout:Package kernel-5.14.0-437.el9.x86_64 is already installed. 2024-06-03T08:18:17.411 INFO:teuthology.orchestra.run.smithi088.stdout:Dependencies resolved. 2024-06-03T08:18:17.423 INFO:teuthology.orchestra.run.smithi088.stdout:================================================================================ 2024-06-03T08:18:17.423 INFO:teuthology.orchestra.run.smithi088.stdout: Package Architecture Version Repository Size 2024-06-03T08:18:17.423 INFO:teuthology.orchestra.run.smithi088.stdout:================================================================================ 2024-06-03T08:18:17.423 INFO:teuthology.orchestra.run.smithi088.stdout:Installing: 2024-06-03T08:18:17.423 INFO:teuthology.orchestra.run.smithi088.stdout: kernel x86_64 5.14.0-452.el9 baseos 893 k 2024-06-03T08:18:17.423 INFO:teuthology.orchestra.run.smithi088.stdout:Installing dependencies: 2024-06-03T08:18:17.424 INFO:teuthology.orchestra.run.smithi088.stdout: kernel-core x86_64 5.14.0-452.el9 baseos 16 M 2024-06-03T08:18:17.424 INFO:teuthology.orchestra.run.smithi088.stdout: kernel-modules x86_64 5.14.0-452.el9 baseos 35 M 2024-06-03T08:18:17.424 INFO:teuthology.orchestra.run.smithi088.stdout: kernel-modules-core x86_64 5.14.0-452.el9 baseos 29 M 2024-06-03T08:18:17.424 INFO:teuthology.orchestra.run.smithi088.stdout: 2024-06-03T08:18:17.424 INFO:teuthology.orchestra.run.smithi088.stdout:Transaction Summary 2024-06-03T08:18:17.424 INFO:teuthology.orchestra.run.smithi088.stdout:================================================================================ 2024-06-03T08:18:17.424 INFO:teuthology.orchestra.run.smithi088.stdout:Install 4 Packages 2024-06-03T08:18:17.424 INFO:teuthology.orchestra.run.smithi088.stdout: 2024-06-03T08:18:17.424 INFO:teuthology.orchestra.run.smithi088.stdout:Total download size: 80 M 2024-06-03T08:18:17.425 INFO:teuthology.orchestra.run.smithi088.stdout:Installed size: 124 M 2024-06-03T08:18:17.425 INFO:teuthology.orchestra.run.smithi088.stdout:Downloading Packages: 2024-06-03T08:18:17.868 INFO:teuthology.orchestra.run.smithi088.stdout:(1/4): kernel-5.14.0-452.el9.x86_64.rpm 2.9 MB/s | 893 kB 00:00 2024-06-03T08:18:18.849 INFO:teuthology.orchestra.run.smithi192.stdout:lab-extras 40 kB/s | 1.7 kB 00:00 2024-06-03T08:18:18.985 INFO:teuthology.orchestra.run.smithi088.stdout:(2/4): kernel-core-5.14.0-452.el9.x86_64.rpm 11 MB/s | 16 MB 00:01 2024-06-03T08:18:19.568 INFO:teuthology.orchestra.run.smithi088.stdout:(3/4): kernel-modules-core-5.14.0-452.el9.x86_6 17 MB/s | 29 MB 00:01 2024-06-03T08:18:19.885 INFO:teuthology.orchestra.run.smithi088.stdout:(4/4): kernel-modules-5.14.0-452.el9.x86_64.rpm 15 MB/s | 35 MB 00:02 2024-06-03T08:18:19.885 INFO:teuthology.orchestra.run.smithi088.stdout:-------------------------------------------------------------------------------- 2024-06-03T08:18:19.885 INFO:teuthology.orchestra.run.smithi088.stdout:Total 33 MB/s | 80 MB 00:02 2024-06-03T08:18:20.583 INFO:teuthology.orchestra.run.smithi088.stdout:Running transaction check 2024-06-03T08:18:20.831 INFO:teuthology.orchestra.run.smithi192.stdout:Package kernel-5.14.0-239.el9.x86_64 is already installed. 2024-06-03T08:18:20.831 INFO:teuthology.orchestra.run.smithi192.stdout:Package kernel-5.14.0-437.el9.x86_64 is already installed. 2024-06-03T08:18:20.924 INFO:teuthology.orchestra.run.smithi192.stdout:Dependencies resolved. 2024-06-03T08:18:20.929 INFO:teuthology.orchestra.run.smithi088.stdout:Transaction check succeeded. 2024-06-03T08:18:20.929 INFO:teuthology.orchestra.run.smithi088.stdout:Running transaction test 2024-06-03T08:18:20.930 INFO:teuthology.orchestra.run.smithi192.stdout:================================================================================ 2024-06-03T08:18:20.930 INFO:teuthology.orchestra.run.smithi192.stdout: Package Architecture Version Repository Size 2024-06-03T08:18:20.930 INFO:teuthology.orchestra.run.smithi192.stdout:================================================================================ 2024-06-03T08:18:20.930 INFO:teuthology.orchestra.run.smithi192.stdout:Installing: 2024-06-03T08:18:20.930 INFO:teuthology.orchestra.run.smithi192.stdout: kernel x86_64 5.14.0-452.el9 baseos 893 k 2024-06-03T08:18:20.931 INFO:teuthology.orchestra.run.smithi192.stdout:Installing dependencies: 2024-06-03T08:18:20.931 INFO:teuthology.orchestra.run.smithi192.stdout: kernel-core x86_64 5.14.0-452.el9 baseos 16 M 2024-06-03T08:18:20.931 INFO:teuthology.orchestra.run.smithi192.stdout: kernel-modules x86_64 5.14.0-452.el9 baseos 35 M 2024-06-03T08:18:20.931 INFO:teuthology.orchestra.run.smithi192.stdout: kernel-modules-core x86_64 5.14.0-452.el9 baseos 29 M 2024-06-03T08:18:20.931 INFO:teuthology.orchestra.run.smithi192.stdout: 2024-06-03T08:18:20.931 INFO:teuthology.orchestra.run.smithi192.stdout:Transaction Summary 2024-06-03T08:18:20.931 INFO:teuthology.orchestra.run.smithi192.stdout:================================================================================ 2024-06-03T08:18:20.931 INFO:teuthology.orchestra.run.smithi192.stdout:Install 4 Packages 2024-06-03T08:18:20.931 INFO:teuthology.orchestra.run.smithi192.stdout: 2024-06-03T08:18:20.932 INFO:teuthology.orchestra.run.smithi192.stdout:Total download size: 80 M 2024-06-03T08:18:20.932 INFO:teuthology.orchestra.run.smithi192.stdout:Installed size: 124 M 2024-06-03T08:18:20.932 INFO:teuthology.orchestra.run.smithi192.stdout:Downloading Packages: 2024-06-03T08:18:21.134 INFO:teuthology.orchestra.run.smithi192.stdout:(1/4): kernel-5.14.0-452.el9.x86_64.rpm 6.3 MB/s | 893 kB 00:00 2024-06-03T08:18:21.425 INFO:teuthology.orchestra.run.smithi192.stdout:(2/4): kernel-core-5.14.0-452.el9.x86_64.rpm 37 MB/s | 16 MB 00:00 2024-06-03T08:18:21.776 INFO:teuthology.orchestra.run.smithi192.stdout:(3/4): kernel-modules-core-5.14.0-452.el9.x86_6 45 MB/s | 29 MB 00:00 2024-06-03T08:18:21.866 INFO:teuthology.orchestra.run.smithi088.stdout:Transaction test succeeded. 2024-06-03T08:18:21.867 INFO:teuthology.orchestra.run.smithi088.stdout:Running transaction 2024-06-03T08:18:22.126 INFO:teuthology.orchestra.run.smithi192.stdout:(4/4): kernel-modules-5.14.0-452.el9.x86_64.rpm 31 MB/s | 35 MB 00:01 2024-06-03T08:18:22.126 INFO:teuthology.orchestra.run.smithi192.stdout:-------------------------------------------------------------------------------- 2024-06-03T08:18:22.126 INFO:teuthology.orchestra.run.smithi192.stdout:Total 67 MB/s | 80 MB 00:01 2024-06-03T08:18:22.824 INFO:teuthology.orchestra.run.smithi192.stdout:Running transaction check 2024-06-03T08:18:23.164 INFO:teuthology.orchestra.run.smithi192.stdout:Transaction check succeeded. 2024-06-03T08:18:23.164 INFO:teuthology.orchestra.run.smithi192.stdout:Running transaction test 2024-06-03T08:18:24.113 INFO:teuthology.orchestra.run.smithi192.stdout:Transaction test succeeded. 2024-06-03T08:18:24.114 INFO:teuthology.orchestra.run.smithi192.stdout:Running transaction 2024-06-03T08:18:24.261 INFO:teuthology.orchestra.run.smithi088.stdout: Preparing : 1/1 2024-06-03T08:18:24.880 INFO:teuthology.orchestra.run.smithi088.stdout: Installing : kernel-modules-core-5.14.0-452.el9.x86_64 1/4 2024-06-03T08:18:24.984 INFO:teuthology.orchestra.run.smithi088.stdout: Installing : kernel-core-5.14.0-452.el9.x86_64 2/4 2024-06-03T08:18:26.532 INFO:teuthology.orchestra.run.smithi192.stdout: Preparing : 1/1 2024-06-03T08:18:27.171 INFO:teuthology.orchestra.run.smithi192.stdout: Installing : kernel-modules-core-5.14.0-452.el9.x86_64 1/4 2024-06-03T08:18:27.251 INFO:teuthology.orchestra.run.smithi192.stdout: Installing : kernel-core-5.14.0-452.el9.x86_64 2/4 2024-06-03T08:18:27.635 INFO:teuthology.orchestra.run.smithi088.stdout: Running scriptlet: kernel-core-5.14.0-452.el9.x86_64 2/4 2024-06-03T08:18:27.786 INFO:teuthology.orchestra.run.smithi088.stdout: Installing : kernel-modules-5.14.0-452.el9.x86_64 3/4 2024-06-03T08:18:29.882 INFO:teuthology.orchestra.run.smithi192.stdout: Running scriptlet: kernel-core-5.14.0-452.el9.x86_64 2/4 2024-06-03T08:18:30.022 INFO:teuthology.orchestra.run.smithi192.stdout: Installing : kernel-modules-5.14.0-452.el9.x86_64 3/4 2024-06-03T08:18:32.544 INFO:teuthology.orchestra.run.smithi088.stdout: Running scriptlet: kernel-modules-5.14.0-452.el9.x86_64 3/4 2024-06-03T08:18:32.649 INFO:teuthology.orchestra.run.smithi088.stdout: Installing : kernel-5.14.0-452.el9.x86_64 4/4 2024-06-03T08:18:34.763 INFO:teuthology.orchestra.run.smithi192.stdout: Running scriptlet: kernel-modules-5.14.0-452.el9.x86_64 3/4 2024-06-03T08:18:34.818 INFO:teuthology.orchestra.run.smithi192.stdout: Installing : kernel-5.14.0-452.el9.x86_64 4/4 2024-06-03T08:18:37.381 INFO:teuthology.orchestra.run.smithi088.stdout: Running scriptlet: kernel-modules-core-5.14.0-452.el9.x86_64 4/4 2024-06-03T08:18:39.580 INFO:teuthology.orchestra.run.smithi192.stdout: Running scriptlet: kernel-modules-core-5.14.0-452.el9.x86_64 4/4 2024-06-03T08:19:06.998 INFO:teuthology.orchestra.run.smithi088.stdout: Running scriptlet: kernel-core-5.14.0-452.el9.x86_64 4/4 2024-06-03T08:19:07.039 INFO:teuthology.orchestra.run.smithi088.stdout: Running scriptlet: kernel-modules-5.14.0-452.el9.x86_64 4/4 2024-06-03T08:19:07.380 INFO:teuthology.orchestra.run.smithi088.stdout: Running scriptlet: kernel-5.14.0-452.el9.x86_64 4/4 2024-06-03T08:19:07.380 INFO:teuthology.orchestra.run.smithi088.stdout: Verifying : kernel-5.14.0-452.el9.x86_64 1/4 2024-06-03T08:19:07.380 INFO:teuthology.orchestra.run.smithi088.stdout: Verifying : kernel-core-5.14.0-452.el9.x86_64 2/4 2024-06-03T08:19:07.380 INFO:teuthology.orchestra.run.smithi088.stdout: Verifying : kernel-modules-5.14.0-452.el9.x86_64 3/4 2024-06-03T08:19:08.160 INFO:teuthology.orchestra.run.smithi088.stdout: Verifying : kernel-modules-core-5.14.0-452.el9.x86_64 4/4 2024-06-03T08:19:08.161 INFO:teuthology.orchestra.run.smithi088.stdout: 2024-06-03T08:19:08.161 INFO:teuthology.orchestra.run.smithi088.stdout:Installed: 2024-06-03T08:19:08.161 INFO:teuthology.orchestra.run.smithi088.stdout: kernel-5.14.0-452.el9.x86_64 2024-06-03T08:19:08.161 INFO:teuthology.orchestra.run.smithi088.stdout: kernel-core-5.14.0-452.el9.x86_64 2024-06-03T08:19:08.161 INFO:teuthology.orchestra.run.smithi088.stdout: kernel-modules-5.14.0-452.el9.x86_64 2024-06-03T08:19:08.161 INFO:teuthology.orchestra.run.smithi088.stdout: kernel-modules-core-5.14.0-452.el9.x86_64 2024-06-03T08:19:08.161 INFO:teuthology.orchestra.run.smithi088.stdout: 2024-06-03T08:19:08.161 INFO:teuthology.orchestra.run.smithi088.stdout:Complete! 2024-06-03T08:19:08.684 DEBUG:teuthology.orchestra.run.smithi088:> rpm -q kernel | sort -rV | head -n 1 2024-06-03T08:19:08.769 INFO:teuthology.orchestra.run.smithi088.stdout:kernel-5.14.0-452.el9.x86_64 2024-06-03T08:19:08.770 DEBUG:teuthology.task.kernel:get_latest_image_version_rpm: 5.14.0-452.el9.x86_64 2024-06-03T08:19:08.770 INFO:teuthology.task.kernel:Not newest distro kernel. Current: 5.14.0-437.el9.x86_64 Expected: 5.14.0-452.el9.x86_64 2024-06-03T08:19:08.770 INFO:teuthology.task.kernel:Skipping firmware on distro kernel 2024-06-03T08:19:08.770 DEBUG:teuthology.task.kernel:src is distro, skipping download 2024-06-03T08:19:08.770 INFO:teuthology.task.kernel:Installing distro kernel on mon.a... 2024-06-03T08:19:08.770 DEBUG:teuthology.task.kernel:install_kernel(remote=ubuntu@smithi088.front.sepia.ceph.com, path=None, version=distro) 2024-06-03T08:19:08.770 DEBUG:teuthology.orchestra.run.smithi088:> rpm -q kernel | sort -rV | head -n 1 2024-06-03T08:19:08.823 INFO:teuthology.orchestra.run.smithi088.stdout:kernel-5.14.0-452.el9.x86_64 2024-06-03T08:19:08.824 DEBUG:teuthology.task.kernel:get_latest_image_version_rpm: 5.14.0-452.el9.x86_64 2024-06-03T08:19:08.824 DEBUG:teuthology.orchestra.run.smithi088:> sudo rpm -qi grub2-tools 2024-06-03T08:19:08.854 INFO:teuthology.orchestra.run.smithi088.stdout:Name : grub2-tools 2024-06-03T08:19:08.855 INFO:teuthology.orchestra.run.smithi088.stdout:Epoch : 1 2024-06-03T08:19:08.855 INFO:teuthology.orchestra.run.smithi088.stdout:Version : 2.06 2024-06-03T08:19:08.855 INFO:teuthology.orchestra.run.smithi088.stdout:Release : 68.el9 2024-06-03T08:19:08.855 INFO:teuthology.orchestra.run.smithi088.stdout:Architecture: x86_64 2024-06-03T08:19:08.855 INFO:teuthology.orchestra.run.smithi088.stdout:Install Date: Fri 19 Apr 2024 09:32:39 PM UTC 2024-06-03T08:19:08.855 INFO:teuthology.orchestra.run.smithi088.stdout:Group : Unspecified 2024-06-03T08:19:08.855 INFO:teuthology.orchestra.run.smithi088.stdout:Size : 8264254 2024-06-03T08:19:08.855 INFO:teuthology.orchestra.run.smithi088.stdout:License : GPLv3+ 2024-06-03T08:19:08.855 INFO:teuthology.orchestra.run.smithi088.stdout:Signature : RSA/SHA256, Mon 07 Aug 2023 01:57:16 PM UTC, Key ID 05b555b38483c65d 2024-06-03T08:19:08.855 INFO:teuthology.orchestra.run.smithi088.stdout:Source RPM : grub2-2.06-68.el9.src.rpm 2024-06-03T08:19:08.855 INFO:teuthology.orchestra.run.smithi088.stdout:Build Date : Wed 02 Aug 2023 03:15:48 PM UTC 2024-06-03T08:19:08.855 INFO:teuthology.orchestra.run.smithi088.stdout:Build Host : x86-05.stream.rdu2.redhat.com 2024-06-03T08:19:08.856 INFO:teuthology.orchestra.run.smithi088.stdout:Packager : builder@centos.org 2024-06-03T08:19:08.856 INFO:teuthology.orchestra.run.smithi088.stdout:Vendor : CentOS 2024-06-03T08:19:08.856 INFO:teuthology.orchestra.run.smithi088.stdout:URL : http://www.gnu.org/software/grub/ 2024-06-03T08:19:08.856 INFO:teuthology.orchestra.run.smithi088.stdout:Summary : Support tools for GRUB. 2024-06-03T08:19:08.856 INFO:teuthology.orchestra.run.smithi088.stdout:Description : 2024-06-03T08:19:08.856 INFO:teuthology.orchestra.run.smithi088.stdout: 2024-06-03T08:19:08.856 INFO:teuthology.orchestra.run.smithi088.stdout:The GRand Unified Bootloader (GRUB) is a highly configurable and 2024-06-03T08:19:08.856 INFO:teuthology.orchestra.run.smithi088.stdout:customizable bootloader with modular architecture. It supports a rich 2024-06-03T08:19:08.856 INFO:teuthology.orchestra.run.smithi088.stdout:variety of kernel formats, file systems, computer architectures and 2024-06-03T08:19:08.856 INFO:teuthology.orchestra.run.smithi088.stdout:hardware devices. 2024-06-03T08:19:08.856 INFO:teuthology.orchestra.run.smithi088.stdout: 2024-06-03T08:19:08.856 INFO:teuthology.orchestra.run.smithi088.stdout:This subpackage provides tools for support of all platforms. 2024-06-03T08:19:08.857 INFO:teuthology.task.kernel:Updating Grub Version: grub2 2024-06-03T08:19:08.857 INFO:teuthology.task.kernel:Updating grub on smithi088 to boot 5.14.0-452.el9.x86_64 2024-06-03T08:19:08.857 DEBUG:teuthology.orchestra.run.smithi088:> sudo grub2-mkconfig -o /boot/grub2/grub.cfg 2024-06-03T08:19:09.423 INFO:teuthology.orchestra.run.smithi192.stdout: Running scriptlet: kernel-core-5.14.0-452.el9.x86_64 4/4 2024-06-03T08:19:09.463 INFO:teuthology.orchestra.run.smithi192.stdout: Running scriptlet: kernel-modules-5.14.0-452.el9.x86_64 4/4 2024-06-03T08:19:09.485 INFO:teuthology.orchestra.run.smithi088.stderr:Generating grub configuration file ... 2024-06-03T08:19:09.777 INFO:teuthology.orchestra.run.smithi192.stdout: Running scriptlet: kernel-5.14.0-452.el9.x86_64 4/4 2024-06-03T08:19:09.777 INFO:teuthology.orchestra.run.smithi192.stdout: Verifying : kernel-5.14.0-452.el9.x86_64 1/4 2024-06-03T08:19:09.777 INFO:teuthology.orchestra.run.smithi192.stdout: Verifying : kernel-core-5.14.0-452.el9.x86_64 2/4 2024-06-03T08:19:09.777 INFO:teuthology.orchestra.run.smithi192.stdout: Verifying : kernel-modules-5.14.0-452.el9.x86_64 3/4 2024-06-03T08:19:10.578 INFO:teuthology.orchestra.run.smithi192.stdout: Verifying : kernel-modules-core-5.14.0-452.el9.x86_64 4/4 2024-06-03T08:19:10.579 INFO:teuthology.orchestra.run.smithi192.stdout: 2024-06-03T08:19:10.579 INFO:teuthology.orchestra.run.smithi192.stdout:Installed: 2024-06-03T08:19:10.579 INFO:teuthology.orchestra.run.smithi192.stdout: kernel-5.14.0-452.el9.x86_64 2024-06-03T08:19:10.579 INFO:teuthology.orchestra.run.smithi192.stdout: kernel-core-5.14.0-452.el9.x86_64 2024-06-03T08:19:10.579 INFO:teuthology.orchestra.run.smithi192.stdout: kernel-modules-5.14.0-452.el9.x86_64 2024-06-03T08:19:10.579 INFO:teuthology.orchestra.run.smithi192.stdout: kernel-modules-core-5.14.0-452.el9.x86_64 2024-06-03T08:19:10.579 INFO:teuthology.orchestra.run.smithi192.stdout: 2024-06-03T08:19:10.579 INFO:teuthology.orchestra.run.smithi192.stdout:Complete! 2024-06-03T08:19:10.908 INFO:teuthology.orchestra.run.smithi088.stderr:Adding boot menu entry for UEFI Firmware Settings ... 2024-06-03T08:19:10.938 INFO:teuthology.orchestra.run.smithi088.stderr:done 2024-06-03T08:19:10.940 DEBUG:teuthology.orchestra.run.smithi088:> mktemp 2024-06-03T08:19:10.951 DEBUG:teuthology.orchestra.run.smithi192:> rpm -q kernel | sort -rV | head -n 1 2024-06-03T08:19:10.955 INFO:teuthology.orchestra.run.smithi088.stdout:/tmp/tmp.73ZmRq0724 2024-06-03T08:19:10.956 DEBUG:teuthology.orchestra.run.smithi088:> sudo cp /boot/grub2/grub.cfg /tmp/tmp.73ZmRq0724 2024-06-03T08:19:11.021 DEBUG:teuthology.orchestra.run.smithi088:> sudo chmod 0666 /tmp/tmp.73ZmRq0724 2024-06-03T08:19:11.028 INFO:teuthology.orchestra.run.smithi192.stdout:kernel-5.14.0-452.el9.x86_64 2024-06-03T08:19:11.028 DEBUG:teuthology.task.kernel:get_latest_image_version_rpm: 5.14.0-452.el9.x86_64 2024-06-03T08:19:11.028 INFO:teuthology.task.kernel:Not newest distro kernel. Current: 5.14.0-437.el9.x86_64 Expected: 5.14.0-452.el9.x86_64 2024-06-03T08:19:11.029 INFO:teuthology.task.kernel:Skipping firmware on distro kernel 2024-06-03T08:19:11.029 DEBUG:teuthology.task.kernel:src is distro, skipping download 2024-06-03T08:19:11.029 INFO:teuthology.task.kernel:Installing distro kernel on mon.b... 2024-06-03T08:19:11.029 DEBUG:teuthology.task.kernel:install_kernel(remote=ubuntu@smithi192.front.sepia.ceph.com, path=None, version=distro) 2024-06-03T08:19:11.029 DEBUG:teuthology.orchestra.run.smithi192:> rpm -q kernel | sort -rV | head -n 1 2024-06-03T08:19:11.081 INFO:teuthology.orchestra.run.smithi192.stdout:kernel-5.14.0-452.el9.x86_64 2024-06-03T08:19:11.081 DEBUG:teuthology.task.kernel:get_latest_image_version_rpm: 5.14.0-452.el9.x86_64 2024-06-03T08:19:11.082 DEBUG:teuthology.orchestra.run.smithi192:> sudo rpm -qi grub2-tools 2024-06-03T08:19:11.112 INFO:teuthology.orchestra.run.smithi192.stdout:Name : grub2-tools 2024-06-03T08:19:11.112 INFO:teuthology.orchestra.run.smithi192.stdout:Epoch : 1 2024-06-03T08:19:11.112 INFO:teuthology.orchestra.run.smithi192.stdout:Version : 2.06 2024-06-03T08:19:11.112 INFO:teuthology.orchestra.run.smithi192.stdout:Release : 68.el9 2024-06-03T08:19:11.112 INFO:teuthology.orchestra.run.smithi192.stdout:Architecture: x86_64 2024-06-03T08:19:11.112 INFO:teuthology.orchestra.run.smithi192.stdout:Install Date: Fri 19 Apr 2024 09:32:39 PM UTC 2024-06-03T08:19:11.113 INFO:teuthology.orchestra.run.smithi192.stdout:Group : Unspecified 2024-06-03T08:19:11.113 INFO:teuthology.orchestra.run.smithi192.stdout:Size : 8264254 2024-06-03T08:19:11.113 INFO:teuthology.orchestra.run.smithi192.stdout:License : GPLv3+ 2024-06-03T08:19:11.113 INFO:teuthology.orchestra.run.smithi192.stdout:Signature : RSA/SHA256, Mon 07 Aug 2023 01:57:16 PM UTC, Key ID 05b555b38483c65d 2024-06-03T08:19:11.113 INFO:teuthology.orchestra.run.smithi192.stdout:Source RPM : grub2-2.06-68.el9.src.rpm 2024-06-03T08:19:11.113 INFO:teuthology.orchestra.run.smithi192.stdout:Build Date : Wed 02 Aug 2023 03:15:48 PM UTC 2024-06-03T08:19:11.113 INFO:teuthology.orchestra.run.smithi192.stdout:Build Host : x86-05.stream.rdu2.redhat.com 2024-06-03T08:19:11.113 INFO:teuthology.orchestra.run.smithi192.stdout:Packager : builder@centos.org 2024-06-03T08:19:11.113 INFO:teuthology.orchestra.run.smithi192.stdout:Vendor : CentOS 2024-06-03T08:19:11.113 INFO:teuthology.orchestra.run.smithi192.stdout:URL : http://www.gnu.org/software/grub/ 2024-06-03T08:19:11.113 INFO:teuthology.orchestra.run.smithi192.stdout:Summary : Support tools for GRUB. 2024-06-03T08:19:11.113 INFO:teuthology.orchestra.run.smithi192.stdout:Description : 2024-06-03T08:19:11.114 INFO:teuthology.orchestra.run.smithi192.stdout: 2024-06-03T08:19:11.114 INFO:teuthology.orchestra.run.smithi192.stdout:The GRand Unified Bootloader (GRUB) is a highly configurable and 2024-06-03T08:19:11.114 INFO:teuthology.orchestra.run.smithi192.stdout:customizable bootloader with modular architecture. It supports a rich 2024-06-03T08:19:11.114 INFO:teuthology.orchestra.run.smithi192.stdout:variety of kernel formats, file systems, computer architectures and 2024-06-03T08:19:11.114 INFO:teuthology.orchestra.run.smithi192.stdout:hardware devices. 2024-06-03T08:19:11.114 INFO:teuthology.orchestra.run.smithi192.stdout: 2024-06-03T08:19:11.114 INFO:teuthology.orchestra.run.smithi192.stdout:This subpackage provides tools for support of all platforms. 2024-06-03T08:19:11.115 INFO:teuthology.task.kernel:Updating Grub Version: grub2 2024-06-03T08:19:11.115 INFO:teuthology.task.kernel:Updating grub on smithi192 to boot 5.14.0-452.el9.x86_64 2024-06-03T08:19:11.115 DEBUG:teuthology.orchestra.run.smithi192:> sudo grub2-mkconfig -o /boot/grub2/grub.cfg 2024-06-03T08:19:11.152 DEBUG:teuthology.orchestra.remote:smithi088:/tmp/tmp.73ZmRq0724 is 6KB 2024-06-03T08:19:11.202 DEBUG:teuthology.orchestra.run.smithi088:> rm -fr /tmp/tmp.73ZmRq0724 2024-06-03T08:19:11.216 DEBUG:teuthology.orchestra.run.smithi088:> sudo /bin/ls /boot/loader/entries || true 2024-06-03T08:19:11.281 INFO:teuthology.orchestra.run.smithi088.stdout:7fc8828e06d0482694bcf4a447d13cb6-0-rescue.conf 2024-06-03T08:19:11.281 INFO:teuthology.orchestra.run.smithi088.stdout:7fc8828e06d0482694bcf4a447d13cb6-5.14.0-239.el9.x86_64.conf 2024-06-03T08:19:11.281 INFO:teuthology.orchestra.run.smithi088.stdout:7fc8828e06d0482694bcf4a447d13cb6-5.14.0-437.el9.x86_64.conf 2024-06-03T08:19:11.281 INFO:teuthology.orchestra.run.smithi088.stdout:7fc8828e06d0482694bcf4a447d13cb6-5.14.0-452.el9.x86_64.conf 2024-06-03T08:19:11.282 DEBUG:teuthology.orchestra.run.smithi088:> sudo grub2-set-default 7fc8828e06d0482694bcf4a447d13cb6-5.14.0-452.el9.x86_64 2024-06-03T08:19:11.430 DEBUG:teuthology.orchestra.run.smithi088:> sudo shutdown -r now 2024-06-03T08:19:11.747 INFO:teuthology.orchestra.run.smithi192.stderr:Generating grub configuration file ... 2024-06-03T08:19:13.154 INFO:teuthology.orchestra.run.smithi192.stderr:Adding boot menu entry for UEFI Firmware Settings ... 2024-06-03T08:19:13.190 INFO:teuthology.orchestra.run.smithi192.stderr:done 2024-06-03T08:19:13.193 DEBUG:teuthology.orchestra.run.smithi192:> mktemp 2024-06-03T08:19:13.224 INFO:teuthology.orchestra.run.smithi192.stdout:/tmp/tmp.2r0RduNLMk 2024-06-03T08:19:13.224 DEBUG:teuthology.orchestra.run.smithi192:> sudo cp /boot/grub2/grub.cfg /tmp/tmp.2r0RduNLMk 2024-06-03T08:19:13.290 DEBUG:teuthology.orchestra.run.smithi192:> sudo chmod 0666 /tmp/tmp.2r0RduNLMk 2024-06-03T08:19:13.424 DEBUG:teuthology.orchestra.remote:smithi192:/tmp/tmp.2r0RduNLMk is 6KB 2024-06-03T08:19:13.473 DEBUG:teuthology.orchestra.run.smithi192:> rm -fr /tmp/tmp.2r0RduNLMk 2024-06-03T08:19:13.487 DEBUG:teuthology.orchestra.run.smithi192:> sudo /bin/ls /boot/loader/entries || true 2024-06-03T08:19:13.549 INFO:teuthology.orchestra.run.smithi192.stdout:7fc8828e06d0482694bcf4a447d13cb6-0-rescue.conf 2024-06-03T08:19:13.549 INFO:teuthology.orchestra.run.smithi192.stdout:7fc8828e06d0482694bcf4a447d13cb6-5.14.0-239.el9.x86_64.conf 2024-06-03T08:19:13.549 INFO:teuthology.orchestra.run.smithi192.stdout:7fc8828e06d0482694bcf4a447d13cb6-5.14.0-437.el9.x86_64.conf 2024-06-03T08:19:13.549 INFO:teuthology.orchestra.run.smithi192.stdout:7fc8828e06d0482694bcf4a447d13cb6-5.14.0-452.el9.x86_64.conf 2024-06-03T08:19:13.550 DEBUG:teuthology.orchestra.run.smithi192:> sudo grub2-set-default 7fc8828e06d0482694bcf4a447d13cb6-5.14.0-452.el9.x86_64 2024-06-03T08:19:13.703 DEBUG:teuthology.orchestra.run.smithi192:> sudo shutdown -r now 2024-06-03T08:19:41.459 INFO:teuthology.task.kernel:Checking client mon.a for new kernel version... 2024-06-03T08:19:41.460 INFO:teuthology.orchestra.remote:Trying to reconnect to host 'ubuntu@smithi088.front.sepia.ceph.com' 2024-06-03T08:19:41.461 DEBUG:teuthology.orchestra.connection:{'hostname': 'smithi088.front.sepia.ceph.com', 'username': 'ubuntu', 'timeout': 60} 2024-06-03T08:19:43.706 INFO:teuthology.task.kernel:Checking client mon.b for new kernel version... 2024-06-03T08:19:43.706 INFO:teuthology.orchestra.remote:Trying to reconnect to host 'ubuntu@smithi192.front.sepia.ceph.com' 2024-06-03T08:19:43.707 DEBUG:teuthology.orchestra.connection:{'hostname': 'smithi192.front.sepia.ceph.com', 'username': 'ubuntu', 'timeout': 60} 2024-06-03T08:19:59.902 DEBUG:teuthology.orchestra.remote:[Errno None] Unable to connect to port 22 on 172.21.15.88 2024-06-03T08:20:02.206 DEBUG:teuthology.orchestra.remote:[Errno None] Unable to connect to port 22 on 172.21.15.192 2024-06-03T08:20:08.910 INFO:teuthology.orchestra.remote:Trying to reconnect to host 'ubuntu@smithi088.front.sepia.ceph.com' 2024-06-03T08:20:08.911 DEBUG:teuthology.orchestra.connection:{'hostname': 'smithi088.front.sepia.ceph.com', 'username': 'ubuntu', 'timeout': 60} 2024-06-03T08:20:11.209 INFO:teuthology.orchestra.remote:Trying to reconnect to host 'ubuntu@smithi192.front.sepia.ceph.com' 2024-06-03T08:20:11.210 DEBUG:teuthology.orchestra.connection:{'hostname': 'smithi192.front.sepia.ceph.com', 'username': 'ubuntu', 'timeout': 60} 2024-06-03T08:20:11.966 DEBUG:teuthology.orchestra.remote:[Errno None] Unable to connect to port 22 on 172.21.15.88 2024-06-03T08:20:14.270 DEBUG:teuthology.orchestra.remote:[Errno None] Unable to connect to port 22 on 172.21.15.192 2024-06-03T08:20:23.975 INFO:teuthology.orchestra.remote:Trying to reconnect to host 'ubuntu@smithi088.front.sepia.ceph.com' 2024-06-03T08:20:23.975 DEBUG:teuthology.orchestra.connection:{'hostname': 'smithi088.front.sepia.ceph.com', 'username': 'ubuntu', 'timeout': 60} 2024-06-03T08:20:26.274 INFO:teuthology.orchestra.remote:Trying to reconnect to host 'ubuntu@smithi192.front.sepia.ceph.com' 2024-06-03T08:20:26.275 DEBUG:teuthology.orchestra.connection:{'hostname': 'smithi192.front.sepia.ceph.com', 'username': 'ubuntu', 'timeout': 60} 2024-06-03T08:20:27.038 DEBUG:teuthology.orchestra.remote:[Errno None] Unable to connect to port 22 on 172.21.15.88 2024-06-03T08:20:42.050 INFO:teuthology.orchestra.remote:Trying to reconnect to host 'ubuntu@smithi088.front.sepia.ceph.com' 2024-06-03T08:20:42.051 DEBUG:teuthology.orchestra.connection:{'hostname': 'smithi088.front.sepia.ceph.com', 'username': 'ubuntu', 'timeout': 60} 2024-06-03T08:20:44.702 DEBUG:teuthology.orchestra.remote:[Errno None] Unable to connect to port 22 on 172.21.15.192 2024-06-03T08:20:59.706 INFO:teuthology.orchestra.remote:Trying to reconnect to host 'ubuntu@smithi192.front.sepia.ceph.com' 2024-06-03T08:20:59.707 DEBUG:teuthology.orchestra.connection:{'hostname': 'smithi192.front.sepia.ceph.com', 'username': 'ubuntu', 'timeout': 60} 2024-06-03T08:21:00.177 DEBUG:teuthology.orchestra.run.smithi192:> true 2024-06-03T08:21:00.579 INFO:teuthology.orchestra.remote:Successfully reconnected to host 'ubuntu@smithi192.front.sepia.ceph.com' 2024-06-03T08:21:00.579 INFO:teuthology.task.kernel:Checking kernel version of mon.b, want "5.14.0-452.el9.x86_64"... 2024-06-03T08:21:00.579 DEBUG:teuthology.orchestra.run.smithi192:> uname -r 2024-06-03T08:21:00.594 INFO:teuthology.orchestra.run.smithi192.stdout:5.14.0-452.el9.x86_64 2024-06-03T08:21:00.594 DEBUG:teuthology.task.kernel:current kernel version is 5.14.0-452.el9.x86_64 vs 5.14.0-452.el9.x86_64 2024-06-03T08:21:00.594 DEBUG:teuthology.task.kernel:utsrelease strings match, do not need to install 2024-06-03T08:21:00.594 DEBUG:teuthology.task.kernel:Distro of this test job: centos 2024-06-03T08:21:01.595 INFO:teuthology.task.kernel:Enabling kdb on mon.b... 2024-06-03T08:21:01.596 DEBUG:teuthology.orchestra.run.smithi192:> echo ttyS1 | sudo tee /sys/module/kgdboc/parameters/kgdboc 2024-06-03T08:21:01.687 INFO:teuthology.orchestra.run.smithi192.stdout:ttyS1 2024-06-03T08:21:01.726 DEBUG:teuthology.parallel:result is None 2024-06-03T08:21:14.035 DEBUG:teuthology.orchestra.run.smithi088:> true 2024-06-03T08:21:14.409 INFO:teuthology.orchestra.remote:Successfully reconnected to host 'ubuntu@smithi088.front.sepia.ceph.com' 2024-06-03T08:21:14.409 INFO:teuthology.task.kernel:Checking kernel version of mon.a, want "5.14.0-452.el9.x86_64"... 2024-06-03T08:21:14.409 DEBUG:teuthology.orchestra.run.smithi088:> uname -r 2024-06-03T08:21:14.427 INFO:teuthology.orchestra.run.smithi088.stdout:5.14.0-452.el9.x86_64 2024-06-03T08:21:14.427 DEBUG:teuthology.task.kernel:current kernel version is 5.14.0-452.el9.x86_64 vs 5.14.0-452.el9.x86_64 2024-06-03T08:21:14.427 DEBUG:teuthology.task.kernel:utsrelease strings match, do not need to install 2024-06-03T08:21:14.427 DEBUG:teuthology.task.kernel:Distro of this test job: centos 2024-06-03T08:21:15.428 INFO:teuthology.task.kernel:Enabling kdb on mon.a... 2024-06-03T08:21:15.428 DEBUG:teuthology.orchestra.run.smithi088:> echo ttyS1 | sudo tee /sys/module/kgdboc/parameters/kgdboc 2024-06-03T08:21:15.499 INFO:teuthology.orchestra.run.smithi088.stdout:ttyS1 2024-06-03T08:21:15.528 DEBUG:teuthology.parallel:result is None 2024-06-03T08:21:15.529 INFO:teuthology.run_tasks:Running task internal.base... 2024-06-03T08:21:15.537 INFO:teuthology.task.internal:Creating test directory... 2024-06-03T08:21:15.537 DEBUG:teuthology.orchestra.run.smithi088:> mkdir -p -m0755 -- /home/ubuntu/cephtest 2024-06-03T08:21:15.540 DEBUG:teuthology.orchestra.run.smithi192:> mkdir -p -m0755 -- /home/ubuntu/cephtest 2024-06-03T08:21:15.559 INFO:teuthology.run_tasks:Running task internal.archive_upload... 2024-06-03T08:21:15.566 INFO:teuthology.run_tasks:Running task internal.archive... 2024-06-03T08:21:15.623 INFO:teuthology.task.internal:Creating archive directory... 2024-06-03T08:21:15.624 DEBUG:teuthology.orchestra.run.smithi088:> install -d -m0755 -- /home/ubuntu/cephtest/archive 2024-06-03T08:21:15.626 DEBUG:teuthology.orchestra.run.smithi192:> install -d -m0755 -- /home/ubuntu/cephtest/archive 2024-06-03T08:21:15.658 INFO:teuthology.run_tasks:Running task internal.coredump... 2024-06-03T08:21:15.665 INFO:teuthology.task.internal:Enabling coredump saving... 2024-06-03T08:21:15.665 DEBUG:teuthology.orchestra.run.smithi088:> install -d -m0755 -- /home/ubuntu/cephtest/archive/coredump && sudo sysctl -w kernel.core_pattern=/home/ubuntu/cephtest/archive/coredump/%t.%p.core && echo kernel.core_pattern=/home/ubuntu/cephtest/archive/coredump/%t.%p.core | sudo tee -a /etc/sysctl.conf 2024-06-03T08:21:15.694 DEBUG:teuthology.orchestra.run.smithi192:> install -d -m0755 -- /home/ubuntu/cephtest/archive/coredump && sudo sysctl -w kernel.core_pattern=/home/ubuntu/cephtest/archive/coredump/%t.%p.core && echo kernel.core_pattern=/home/ubuntu/cephtest/archive/coredump/%t.%p.core | sudo tee -a /etc/sysctl.conf 2024-06-03T08:21:15.724 INFO:teuthology.orchestra.run.smithi192.stdout:kernel.core_pattern = /home/ubuntu/cephtest/archive/coredump/%t.%p.core 2024-06-03T08:21:15.729 INFO:teuthology.orchestra.run.smithi088.stdout:kernel.core_pattern = /home/ubuntu/cephtest/archive/coredump/%t.%p.core 2024-06-03T08:21:15.736 INFO:teuthology.orchestra.run.smithi192.stdout:kernel.core_pattern=/home/ubuntu/cephtest/archive/coredump/%t.%p.core 2024-06-03T08:21:15.741 INFO:teuthology.orchestra.run.smithi088.stdout:kernel.core_pattern=/home/ubuntu/cephtest/archive/coredump/%t.%p.core 2024-06-03T08:21:15.743 INFO:teuthology.run_tasks:Running task internal.sudo... 2024-06-03T08:21:15.753 INFO:teuthology.task.internal:Configuring sudo... 2024-06-03T08:21:15.753 DEBUG:teuthology.orchestra.run.smithi088:> sudo sed -i.orig.teuthology -e 's/^\([^#]*\) \(requiretty\)/\1 !\2/g' -e 's/^\([^#]*\) !\(visiblepw\)/\1 \2/g' /etc/sudoers 2024-06-03T08:21:15.786 DEBUG:teuthology.orchestra.run.smithi192:> sudo sed -i.orig.teuthology -e 's/^\([^#]*\) \(requiretty\)/\1 !\2/g' -e 's/^\([^#]*\) !\(visiblepw\)/\1 \2/g' /etc/sudoers 2024-06-03T08:21:15.814 INFO:teuthology.run_tasks:Running task internal.syslog... 2024-06-03T08:21:15.823 INFO:teuthology.task.internal.syslog:Starting syslog monitoring... 2024-06-03T08:21:15.823 DEBUG:teuthology.orchestra.run.smithi088:> mkdir -p -m0755 -- /home/ubuntu/cephtest/archive/syslog 2024-06-03T08:21:15.856 DEBUG:teuthology.orchestra.run.smithi192:> mkdir -p -m0755 -- /home/ubuntu/cephtest/archive/syslog 2024-06-03T08:21:15.872 DEBUG:teuthology.orchestra.run.smithi088:> install -m 666 /dev/null /home/ubuntu/cephtest/archive/syslog/kern.log 2024-06-03T08:21:15.958 DEBUG:teuthology.orchestra.run.smithi088:> sudo chcon system_u:object_r:var_log_t:s0 /home/ubuntu/cephtest/archive/syslog/kern.log 2024-06-03T08:21:16.007 DEBUG:teuthology.orchestra.run.smithi088:> install -m 666 /dev/null /home/ubuntu/cephtest/archive/syslog/misc.log 2024-06-03T08:21:16.093 DEBUG:teuthology.orchestra.run.smithi088:> sudo chcon system_u:object_r:var_log_t:s0 /home/ubuntu/cephtest/archive/syslog/misc.log 2024-06-03T08:21:16.138 DEBUG:teuthology.orchestra.run.smithi088:> set -ex 2024-06-03T08:21:16.138 DEBUG:teuthology.orchestra.run.smithi088:> sudo dd of=/etc/rsyslog.d/80-cephtest.conf 2024-06-03T08:21:16.207 DEBUG:teuthology.orchestra.run.smithi192:> install -m 666 /dev/null /home/ubuntu/cephtest/archive/syslog/kern.log 2024-06-03T08:21:16.252 DEBUG:teuthology.orchestra.run.smithi192:> sudo chcon system_u:object_r:var_log_t:s0 /home/ubuntu/cephtest/archive/syslog/kern.log 2024-06-03T08:21:16.303 DEBUG:teuthology.orchestra.run.smithi192:> install -m 666 /dev/null /home/ubuntu/cephtest/archive/syslog/misc.log 2024-06-03T08:21:16.385 DEBUG:teuthology.orchestra.run.smithi192:> sudo chcon system_u:object_r:var_log_t:s0 /home/ubuntu/cephtest/archive/syslog/misc.log 2024-06-03T08:21:16.430 DEBUG:teuthology.orchestra.run.smithi192:> set -ex 2024-06-03T08:21:16.430 DEBUG:teuthology.orchestra.run.smithi192:> sudo dd of=/etc/rsyslog.d/80-cephtest.conf 2024-06-03T08:21:16.497 DEBUG:teuthology.orchestra.run.smithi088:> sudo service rsyslog restart 2024-06-03T08:21:16.500 DEBUG:teuthology.orchestra.run.smithi192:> sudo service rsyslog restart 2024-06-03T08:21:16.544 INFO:teuthology.orchestra.run.smithi088.stderr:Redirecting to /bin/systemctl restart rsyslog.service 2024-06-03T08:21:16.593 INFO:teuthology.orchestra.run.smithi192.stderr:Redirecting to /bin/systemctl restart rsyslog.service 2024-06-03T08:21:17.097 INFO:teuthology.run_tasks:Running task internal.timer... 2024-06-03T08:21:17.106 INFO:teuthology.task.internal:Starting timer... 2024-06-03T08:21:17.106 INFO:teuthology.run_tasks:Running task pcp... 2024-06-03T08:21:17.116 INFO:teuthology.run_tasks:Running task selinux... 2024-06-03T08:21:17.124 DEBUG:teuthology.orchestra.run.smithi088:> sudo service auditd rotate 2024-06-03T08:21:17.171 INFO:teuthology.orchestra.run.smithi088.stdout:Rotating logs: 2024-06-03T08:21:17.172 DEBUG:teuthology.orchestra.run.smithi192:> sudo service auditd rotate 2024-06-03T08:21:17.237 INFO:teuthology.orchestra.run.smithi192.stdout:Rotating logs: 2024-06-03T08:21:17.239 DEBUG:teuthology.task.selinux:Getting current SELinux state 2024-06-03T08:21:17.239 DEBUG:teuthology.orchestra.run.smithi088:> /usr/sbin/getenforce 2024-06-03T08:21:17.263 INFO:teuthology.orchestra.run.smithi088.stdout:Permissive 2024-06-03T08:21:17.263 DEBUG:teuthology.orchestra.run.smithi192:> /usr/sbin/getenforce 2024-06-03T08:21:17.307 INFO:teuthology.orchestra.run.smithi192.stdout:Permissive 2024-06-03T08:21:17.308 DEBUG:teuthology.task.selinux:Existing SELinux modes: {'ubuntu@smithi088.front.sepia.ceph.com': 'permissive', 'ubuntu@smithi192.front.sepia.ceph.com': 'permissive'} 2024-06-03T08:21:17.308 DEBUG:teuthology.orchestra.run.smithi088:> sudo grep -a 'avc: .*denied' /var/log/audit/audit.log | grep -av -e 'comm="dmidecode"' -e chronyd.service -e 'name="cephtest"' -e scontext=system_u:system_r:nrpe_t:s0 -e scontext=system_u:system_r:pcp_pmlogger_t -e scontext=system_u:system_r:pcp_pmcd_t:s0 -e 'comm="rhsmd"' -e scontext=system_u:system_r:syslogd_t:s0 -e tcontext=system_u:system_r:nrpe_t:s0 -e 'comm="updatedb"' -e 'comm="smartd"' -e 'comm="rhsmcertd-worke"' -e 'comm="setroubleshootd"' -e 'comm="rpm"' -e tcontext=system_u:object_r:container_runtime_exec_t:s0 -e 'comm="ksmtuned"' -e 'comm="sssd"' -e 'comm="sss_cache"' -e context=system_u:system_r:NetworkManager_dispatcher_t:s0 2024-06-03T08:21:17.336 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-03T08:21:17.337 DEBUG:teuthology.orchestra.run.smithi192:> sudo grep -a 'avc: .*denied' /var/log/audit/audit.log | grep -av -e 'comm="dmidecode"' -e chronyd.service -e 'name="cephtest"' -e scontext=system_u:system_r:nrpe_t:s0 -e scontext=system_u:system_r:pcp_pmlogger_t -e scontext=system_u:system_r:pcp_pmcd_t:s0 -e 'comm="rhsmd"' -e scontext=system_u:system_r:syslogd_t:s0 -e tcontext=system_u:system_r:nrpe_t:s0 -e 'comm="updatedb"' -e 'comm="smartd"' -e 'comm="rhsmcertd-worke"' -e 'comm="setroubleshootd"' -e 'comm="rpm"' -e tcontext=system_u:object_r:container_runtime_exec_t:s0 -e 'comm="ksmtuned"' -e 'comm="sssd"' -e 'comm="sss_cache"' -e context=system_u:system_r:NetworkManager_dispatcher_t:s0 2024-06-03T08:21:17.376 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-03T08:21:17.376 INFO:teuthology.task.selinux:Putting SELinux into permissive mode 2024-06-03T08:21:17.376 DEBUG:teuthology.orchestra.run.smithi088:> sudo /usr/sbin/setenforce permissive 2024-06-03T08:21:17.407 DEBUG:teuthology.orchestra.run.smithi192:> sudo /usr/sbin/setenforce permissive 2024-06-03T08:21:17.443 INFO:teuthology.run_tasks:Running task ansible.cephlab... 2024-06-03T08:21:17.452 DEBUG:teuthology.repo_utils:Setting repo remote to https://git.ceph.com/ceph-cm-ansible.git 2024-06-03T08:21:17.455 INFO:teuthology.repo_utils:Fetching git.ceph.com_ceph-cm-ansible_main from origin 2024-06-03T08:21:17.585 DEBUG:teuthology.repo_utils:Resetting repo at /home/teuthworker/src/git.ceph.com_ceph-cm-ansible_main to origin/main 2024-06-03T08:21:17.597 INFO:teuthology.task.ansible:Playbook: [{'import_playbook': 'ansible_managed.yml'}, {'import_playbook': 'teuthology.yml'}, {'hosts': 'testnodes', 'tasks': [{'set_fact': {'ran_from_cephlab_playbook': True}}]}, {'import_playbook': 'testnodes.yml'}, {'import_playbook': 'container-host.yml'}, {'import_playbook': 'cobbler.yml'}, {'import_playbook': 'paddles.yml'}, {'import_playbook': 'pulpito.yml'}, {'hosts': 'testnodes', 'become': True, 'tasks': [{'name': 'Touch /ceph-qa-ready', 'file': {'path': '/ceph-qa-ready', 'state': 'touch'}, 'when': 'ran_from_cephlab_playbook|bool'}]}] 2024-06-03T08:21:17.597 DEBUG:teuthology.task.ansible:Running ansible-playbook -v --extra-vars '{"ansible_ssh_user": "ubuntu"}' -i /etc/ansible/hosts --limit smithi088.front.sepia.ceph.com,smithi192.front.sepia.ceph.com /home/teuthworker/src/git.ceph.com_ceph-cm-ansible_main/cephlab.yml 2024-06-03T08:25:38.158 DEBUG:teuthology.task.ansible:Reconnecting to [Remote(name='ubuntu@smithi088.front.sepia.ceph.com'), Remote(name='ubuntu@smithi192.front.sepia.ceph.com')] 2024-06-03T08:25:38.160 INFO:teuthology.orchestra.remote:Trying to reconnect to host 'ubuntu@smithi088.front.sepia.ceph.com' 2024-06-03T08:25:38.161 DEBUG:teuthology.orchestra.connection:{'hostname': 'smithi088.front.sepia.ceph.com', 'username': 'ubuntu', 'timeout': 60} 2024-06-03T08:25:38.235 DEBUG:teuthology.orchestra.run.smithi088:> true 2024-06-03T08:25:38.317 INFO:teuthology.orchestra.remote:Successfully reconnected to host 'ubuntu@smithi088.front.sepia.ceph.com' 2024-06-03T08:25:38.318 INFO:teuthology.orchestra.remote:Trying to reconnect to host 'ubuntu@smithi192.front.sepia.ceph.com' 2024-06-03T08:25:38.318 DEBUG:teuthology.orchestra.connection:{'hostname': 'smithi192.front.sepia.ceph.com', 'username': 'ubuntu', 'timeout': 60} 2024-06-03T08:25:38.394 DEBUG:teuthology.orchestra.run.smithi192:> true 2024-06-03T08:25:38.482 INFO:teuthology.orchestra.remote:Successfully reconnected to host 'ubuntu@smithi192.front.sepia.ceph.com' 2024-06-03T08:25:38.483 INFO:teuthology.run_tasks:Running task clock... 2024-06-03T08:25:38.494 INFO:teuthology.task.clock:Syncing clocks and checking initial clock skew... 2024-06-03T08:25:38.494 INFO:teuthology.orchestra.run:Running command with timeout 360 2024-06-03T08:25:38.495 DEBUG:teuthology.orchestra.run.smithi088:> sudo systemctl stop ntp.service || sudo systemctl stop ntpd.service || sudo systemctl stop chronyd.service ; sudo ntpd -gq || sudo chronyc makestep ; sudo systemctl start ntp.service || sudo systemctl start ntpd.service || sudo systemctl start chronyd.service ; PATH=/usr/bin:/usr/sbin ntpq -p || PATH=/usr/bin:/usr/sbin chronyc sources || true 2024-06-03T08:25:38.497 INFO:teuthology.orchestra.run:Running command with timeout 360 2024-06-03T08:25:38.497 DEBUG:teuthology.orchestra.run.smithi192:> sudo systemctl stop ntp.service || sudo systemctl stop ntpd.service || sudo systemctl stop chronyd.service ; sudo ntpd -gq || sudo chronyc makestep ; sudo systemctl start ntp.service || sudo systemctl start ntpd.service || sudo systemctl start chronyd.service ; PATH=/usr/bin:/usr/sbin ntpq -p || PATH=/usr/bin:/usr/sbin chronyc sources || true 2024-06-03T08:25:38.524 INFO:teuthology.orchestra.run.smithi088.stderr:Failed to stop ntp.service: Unit ntp.service not loaded. 2024-06-03T08:25:38.538 INFO:teuthology.orchestra.run.smithi088.stderr:Failed to stop ntpd.service: Unit ntpd.service not loaded. 2024-06-03T08:25:38.553 INFO:teuthology.orchestra.run.smithi192.stderr:Failed to stop ntp.service: Unit ntp.service not loaded. 2024-06-03T08:25:38.562 INFO:teuthology.orchestra.run.smithi088.stderr:sudo: ntpd: command not found 2024-06-03T08:25:38.569 INFO:teuthology.orchestra.run.smithi192.stderr:Failed to stop ntpd.service: Unit ntpd.service not loaded. 2024-06-03T08:25:38.573 INFO:teuthology.orchestra.run.smithi088.stdout:506 Cannot talk to daemon 2024-06-03T08:25:38.587 INFO:teuthology.orchestra.run.smithi088.stderr:Failed to start ntp.service: Unit ntp.service not found. 2024-06-03T08:25:38.594 INFO:teuthology.orchestra.run.smithi192.stderr:sudo: ntpd: command not found 2024-06-03T08:25:38.600 INFO:teuthology.orchestra.run.smithi088.stderr:Failed to start ntpd.service: Unit ntpd.service not found. 2024-06-03T08:25:38.605 INFO:teuthology.orchestra.run.smithi192.stdout:506 Cannot talk to daemon 2024-06-03T08:25:38.619 INFO:teuthology.orchestra.run.smithi192.stderr:Failed to start ntp.service: Unit ntp.service not found. 2024-06-03T08:25:38.633 INFO:teuthology.orchestra.run.smithi192.stderr:Failed to start ntpd.service: Unit ntpd.service not found. 2024-06-03T08:25:38.656 INFO:teuthology.orchestra.run.smithi088.stderr:bash: line 1: ntpq: command not found 2024-06-03T08:25:38.660 INFO:teuthology.orchestra.run.smithi088.stdout:MS Name/IP address Stratum Poll Reach LastRx Last sample 2024-06-03T08:25:38.660 INFO:teuthology.orchestra.run.smithi088.stdout:=============================================================================== 2024-06-03T08:25:38.660 INFO:teuthology.orchestra.run.smithi088.stdout:^? hv01.front.sepia.ceph.com 0 6 0 - +0ns[ +0ns] +/- 0ns 2024-06-03T08:25:38.661 INFO:teuthology.orchestra.run.smithi088.stdout:^? hv02.front.sepia.ceph.com 0 6 0 - +0ns[ +0ns] +/- 0ns 2024-06-03T08:25:38.661 INFO:teuthology.orchestra.run.smithi088.stdout:^? hv03.front.sepia.ceph.com 0 6 0 - +0ns[ +0ns] +/- 0ns 2024-06-03T08:25:38.661 INFO:teuthology.orchestra.run.smithi088.stdout:^? hv04.front.sepia.ceph.com 0 6 0 - +0ns[ +0ns] +/- 0ns 2024-06-03T08:25:38.688 INFO:teuthology.orchestra.run.smithi192.stderr:bash: line 1: ntpq: command not found 2024-06-03T08:25:38.691 INFO:teuthology.orchestra.run.smithi192.stdout:MS Name/IP address Stratum Poll Reach LastRx Last sample 2024-06-03T08:25:38.691 INFO:teuthology.orchestra.run.smithi192.stdout:=============================================================================== 2024-06-03T08:25:38.691 INFO:teuthology.orchestra.run.smithi192.stdout:^? hv01.front.sepia.ceph.com 0 6 0 - +0ns[ +0ns] +/- 0ns 2024-06-03T08:25:38.692 INFO:teuthology.orchestra.run.smithi192.stdout:^? hv02.front.sepia.ceph.com 0 6 0 - +0ns[ +0ns] +/- 0ns 2024-06-03T08:25:38.692 INFO:teuthology.orchestra.run.smithi192.stdout:^? hv03.front.sepia.ceph.com 0 6 0 - +0ns[ +0ns] +/- 0ns 2024-06-03T08:25:38.692 INFO:teuthology.orchestra.run.smithi192.stdout:^? hv04.front.sepia.ceph.com 0 6 0 - +0ns[ +0ns] +/- 0ns 2024-06-03T08:25:38.693 INFO:teuthology.run_tasks:Running task install... 2024-06-03T08:25:38.701 DEBUG:teuthology.task.install:project ceph 2024-06-03T08:25:38.701 DEBUG:teuthology.task.install:INSTALL overrides: {'ceph': {'debuginfo': True, 'flavor': 'default', 'sha1': 'f1abf795f4fc15bf2f30ac90c811a53ab21d5d65'}} 2024-06-03T08:25:38.701 DEBUG:teuthology.task.install:config {'debuginfo': True, 'flavor': 'default', 'sha1': 'f1abf795f4fc15bf2f30ac90c811a53ab21d5d65'} 2024-06-03T08:25:38.701 INFO:teuthology.task.install:Using flavor: default 2024-06-03T08:25:38.707 DEBUG:teuthology.task.install:Package list is: {'deb': ['ceph', 'cephadm', 'ceph-mds', 'ceph-mgr', 'ceph-common', 'ceph-fuse', 'ceph-test', 'ceph-volume', 'radosgw', 'python3-rados', 'python3-rgw', 'python3-cephfs', 'python3-rbd', 'libcephfs2', 'libcephfs-dev', 'librados2', 'librbd1', 'rbd-fuse', 'ceph-common-dbg', 'ceph-fuse-dbg', 'ceph-mds-dbg', 'ceph-mgr-dbg', 'ceph-mon-dbg', 'ceph-osd-dbg', 'ceph-test-dbg', 'libcephfs2-dbg', 'librados2-dbg', 'libradosstriper1-dbg', 'librbd1-dbg', 'librgw2-dbg', 'radosgw-dbg', 'rbd-fuse-dbg', 'rbd-mirror-dbg', 'rbd-nbd-dbg'], 'rpm': ['ceph-radosgw', 'ceph-test', 'ceph', 'ceph-base', 'cephadm', 'ceph-immutable-object-cache', 'ceph-mgr', 'ceph-mgr-dashboard', 'ceph-mgr-diskprediction-local', 'ceph-mgr-rook', 'ceph-mgr-cephadm', 'ceph-fuse', 'ceph-volume', 'librados-devel', 'libcephfs2', 'libcephfs-devel', 'librados2', 'librbd1', 'python3-rados', 'python3-rgw', 'python3-cephfs', 'python3-rbd', 'rbd-fuse', 'rbd-mirror', 'rbd-nbd', 'ceph-base-debuginfo', 'ceph-common-debuginfo', 'ceph-immutable-object-cache-debuginfo', 'ceph-radosgw-debuginfo', 'ceph-test-debuginfo', 'ceph-base-debuginfo', 'ceph-mgr-debuginfo', 'ceph-mds-debuginfo', 'ceph-mon-debuginfo', 'ceph-osd-debuginfo', 'ceph-fuse-debuginfo', 'librados-devel-debuginfo', 'libcephfs2-debuginfo', 'librados2-debuginfo', 'librbd1-debuginfo', 'python3-cephfs-debuginfo', 'python3-rados-debuginfo', 'python3-rbd-debuginfo', 'python3-rgw-debuginfo', 'rbd-fuse-debuginfo', 'rbd-mirror-debuginfo', 'rbd-nbd-debuginfo']} 2024-06-03T08:25:38.707 INFO:teuthology.task.install:extra packages: [] 2024-06-03T08:25:38.707 DEBUG:teuthology.task.install.rpm:_update_package_list_and_install: config is {'branch': None, 'cleanup': None, 'debuginfo': True, 'downgrade_packages': [], 'exclude_packages': [], 'extra_packages': [], 'extra_system_packages': [], 'extras': None, 'enable_coprs': [], 'flavor': 'default', 'install_ceph_packages': True, 'packages': {}, 'project': 'ceph', 'repos_only': False, 'sha1': 'f1abf795f4fc15bf2f30ac90c811a53ab21d5d65', 'tag': None, 'wait_for_package': False} 2024-06-03T08:25:38.708 DEBUG:teuthology.packaging:Querying https://shaman.ceph.com/api/search?status=ready&project=ceph&flavor=default&distros=centos%2F9%2Fx86_64&sha1=f1abf795f4fc15bf2f30ac90c811a53ab21d5d65 2024-06-03T08:25:38.710 DEBUG:teuthology.task.install.rpm:_update_package_list_and_install: config is {'branch': None, 'cleanup': None, 'debuginfo': True, 'downgrade_packages': [], 'exclude_packages': [], 'extra_packages': [], 'extra_system_packages': [], 'extras': None, 'enable_coprs': [], 'flavor': 'default', 'install_ceph_packages': True, 'packages': {}, 'project': 'ceph', 'repos_only': False, 'sha1': 'f1abf795f4fc15bf2f30ac90c811a53ab21d5d65', 'tag': None, 'wait_for_package': False} 2024-06-03T08:25:38.710 DEBUG:teuthology.packaging:Querying https://shaman.ceph.com/api/search?status=ready&project=ceph&flavor=default&distros=centos%2F9%2Fx86_64&sha1=f1abf795f4fc15bf2f30ac90c811a53ab21d5d65 2024-06-03T08:25:38.848 INFO:teuthology.task.install.rpm:Pulling from https://1.chacra.ceph.com/r/ceph/reef/f1abf795f4fc15bf2f30ac90c811a53ab21d5d65/centos/9/flavors/default/ 2024-06-03T08:25:38.849 INFO:teuthology.task.install.rpm:Package version is 18.2.2-1389.gf1abf795 2024-06-03T08:25:38.922 INFO:teuthology.task.install.rpm:Pulling from https://1.chacra.ceph.com/r/ceph/reef/f1abf795f4fc15bf2f30ac90c811a53ab21d5d65/centos/9/flavors/default/ 2024-06-03T08:25:38.922 INFO:teuthology.task.install.rpm:Package version is 18.2.2-1389.gf1abf795 2024-06-03T08:25:39.026 INFO:teuthology.packaging:Writing yum repo: [ceph] name=ceph packages for $basearch baseurl=https://1.chacra.ceph.com/r/ceph/reef/f1abf795f4fc15bf2f30ac90c811a53ab21d5d65/centos/9/flavors/default/$basearch enabled=1 gpgcheck=0 type=rpm-md [ceph-noarch] name=ceph noarch packages baseurl=https://1.chacra.ceph.com/r/ceph/reef/f1abf795f4fc15bf2f30ac90c811a53ab21d5d65/centos/9/flavors/default/noarch enabled=1 gpgcheck=0 type=rpm-md [ceph-source] name=ceph source packages baseurl=https://1.chacra.ceph.com/r/ceph/reef/f1abf795f4fc15bf2f30ac90c811a53ab21d5d65/centos/9/flavors/default/SRPMS enabled=1 gpgcheck=0 type=rpm-md 2024-06-03T08:25:39.026 DEBUG:teuthology.orchestra.run.smithi192:> set -ex 2024-06-03T08:25:39.026 DEBUG:teuthology.orchestra.run.smithi192:> sudo dd of=/etc/yum.repos.d/ceph.repo 2024-06-03T08:25:39.052 INFO:teuthology.task.install.rpm:Installing packages: ceph-radosgw, ceph-test, ceph, ceph-base, cephadm, ceph-immutable-object-cache, ceph-mgr, ceph-mgr-dashboard, ceph-mgr-diskprediction-local, ceph-mgr-rook, ceph-mgr-cephadm, ceph-fuse, ceph-volume, librados-devel, libcephfs2, libcephfs-devel, librados2, librbd1, python3-rados, python3-rgw, python3-cephfs, python3-rbd, rbd-fuse, rbd-mirror, rbd-nbd, ceph-base-debuginfo, ceph-common-debuginfo, ceph-immutable-object-cache-debuginfo, ceph-radosgw-debuginfo, ceph-test-debuginfo, ceph-base-debuginfo, ceph-mgr-debuginfo, ceph-mds-debuginfo, ceph-mon-debuginfo, ceph-osd-debuginfo, ceph-fuse-debuginfo, librados-devel-debuginfo, libcephfs2-debuginfo, librados2-debuginfo, librbd1-debuginfo, python3-cephfs-debuginfo, python3-rados-debuginfo, python3-rbd-debuginfo, python3-rgw-debuginfo, rbd-fuse-debuginfo, rbd-mirror-debuginfo, rbd-nbd-debuginfo on remote rpm x86_64 2024-06-03T08:25:39.052 DEBUG:teuthology.orchestra.run.smithi192:> if test -f /etc/yum.repos.d/ceph.repo ; then sudo sed -i -e ':a;N;$!ba;s/enabled=1\ngpg/enabled=1\npriority=1\ngpg/g' -e 's;ref/[a-zA-Z0-9_-]*/;sha1/f1abf795f4fc15bf2f30ac90c811a53ab21d5d65/;g' /etc/yum.repos.d/ceph.repo ; fi 2024-06-03T08:25:39.094 INFO:teuthology.packaging:Writing yum repo: [ceph] name=ceph packages for $basearch baseurl=https://1.chacra.ceph.com/r/ceph/reef/f1abf795f4fc15bf2f30ac90c811a53ab21d5d65/centos/9/flavors/default/$basearch enabled=1 gpgcheck=0 type=rpm-md [ceph-noarch] name=ceph noarch packages baseurl=https://1.chacra.ceph.com/r/ceph/reef/f1abf795f4fc15bf2f30ac90c811a53ab21d5d65/centos/9/flavors/default/noarch enabled=1 gpgcheck=0 type=rpm-md [ceph-source] name=ceph source packages baseurl=https://1.chacra.ceph.com/r/ceph/reef/f1abf795f4fc15bf2f30ac90c811a53ab21d5d65/centos/9/flavors/default/SRPMS enabled=1 gpgcheck=0 type=rpm-md 2024-06-03T08:25:39.094 DEBUG:teuthology.orchestra.run.smithi088:> set -ex 2024-06-03T08:25:39.094 DEBUG:teuthology.orchestra.run.smithi088:> sudo dd of=/etc/yum.repos.d/ceph.repo 2024-06-03T08:25:39.120 DEBUG:teuthology.orchestra.run.smithi192:> sudo touch -a /etc/yum/pluginconf.d/priorities.conf ; test -e /etc/yum/pluginconf.d/priorities.conf.orig || sudo cp -af /etc/yum/pluginconf.d/priorities.conf /etc/yum/pluginconf.d/priorities.conf.orig 2024-06-03T08:25:39.121 INFO:teuthology.task.install.rpm:Installing packages: ceph-radosgw, ceph-test, ceph, ceph-base, cephadm, ceph-immutable-object-cache, ceph-mgr, ceph-mgr-dashboard, ceph-mgr-diskprediction-local, ceph-mgr-rook, ceph-mgr-cephadm, ceph-fuse, ceph-volume, librados-devel, libcephfs2, libcephfs-devel, librados2, librbd1, python3-rados, python3-rgw, python3-cephfs, python3-rbd, rbd-fuse, rbd-mirror, rbd-nbd, ceph-base-debuginfo, ceph-common-debuginfo, ceph-immutable-object-cache-debuginfo, ceph-radosgw-debuginfo, ceph-test-debuginfo, ceph-base-debuginfo, ceph-mgr-debuginfo, ceph-mds-debuginfo, ceph-mon-debuginfo, ceph-osd-debuginfo, ceph-fuse-debuginfo, librados-devel-debuginfo, libcephfs2-debuginfo, librados2-debuginfo, librbd1-debuginfo, python3-cephfs-debuginfo, python3-rados-debuginfo, python3-rbd-debuginfo, python3-rgw-debuginfo, rbd-fuse-debuginfo, rbd-mirror-debuginfo, rbd-nbd-debuginfo on remote rpm x86_64 2024-06-03T08:25:39.121 DEBUG:teuthology.orchestra.run.smithi088:> if test -f /etc/yum.repos.d/ceph.repo ; then sudo sed -i -e ':a;N;$!ba;s/enabled=1\ngpg/enabled=1\npriority=1\ngpg/g' -e 's;ref/[a-zA-Z0-9_-]*/;sha1/f1abf795f4fc15bf2f30ac90c811a53ab21d5d65/;g' /etc/yum.repos.d/ceph.repo ; fi 2024-06-03T08:25:39.187 DEBUG:teuthology.orchestra.run.smithi088:> sudo touch -a /etc/yum/pluginconf.d/priorities.conf ; test -e /etc/yum/pluginconf.d/priorities.conf.orig || sudo cp -af /etc/yum/pluginconf.d/priorities.conf /etc/yum/pluginconf.d/priorities.conf.orig 2024-06-03T08:25:39.195 DEBUG:teuthology.orchestra.run.smithi192:> grep check_obsoletes /etc/yum/pluginconf.d/priorities.conf && sudo sed -i 's/check_obsoletes.*0/check_obsoletes = 1/g' /etc/yum/pluginconf.d/priorities.conf || echo 'check_obsoletes = 1' | sudo tee -a /etc/yum/pluginconf.d/priorities.conf 2024-06-03T08:25:39.259 INFO:teuthology.orchestra.run.smithi192.stdout:check_obsoletes = 1 2024-06-03T08:25:39.261 DEBUG:teuthology.orchestra.run.smithi192:> sudo yum clean all 2024-06-03T08:25:39.263 DEBUG:teuthology.orchestra.run.smithi088:> grep check_obsoletes /etc/yum/pluginconf.d/priorities.conf && sudo sed -i 's/check_obsoletes.*0/check_obsoletes = 1/g' /etc/yum/pluginconf.d/priorities.conf || echo 'check_obsoletes = 1' | sudo tee -a /etc/yum/pluginconf.d/priorities.conf 2024-06-03T08:25:39.327 INFO:teuthology.orchestra.run.smithi088.stdout:check_obsoletes = 1 2024-06-03T08:25:39.329 DEBUG:teuthology.orchestra.run.smithi088:> sudo yum clean all 2024-06-03T08:25:39.490 INFO:teuthology.orchestra.run.smithi192.stdout:45 files removed 2024-06-03T08:25:39.517 DEBUG:teuthology.orchestra.run.smithi192:> sudo yum -y install ceph-radosgw 2024-06-03T08:25:39.568 INFO:teuthology.orchestra.run.smithi088.stdout:45 files removed 2024-06-03T08:25:39.602 DEBUG:teuthology.orchestra.run.smithi088:> sudo yum -y install ceph-radosgw 2024-06-03T08:25:40.172 INFO:teuthology.orchestra.run.smithi192.stdout:ceph packages for x86_64 204 kB/s | 77 kB 00:00 2024-06-03T08:25:40.220 INFO:teuthology.orchestra.run.smithi088.stdout:ceph packages for x86_64 225 kB/s | 77 kB 00:00 2024-06-03T08:25:40.595 INFO:teuthology.orchestra.run.smithi088.stdout:ceph noarch packages 35 kB/s | 11 kB 00:00 2024-06-03T08:25:40.605 INFO:teuthology.orchestra.run.smithi192.stdout:ceph noarch packages 30 kB/s | 11 kB 00:00 2024-06-03T08:25:40.956 INFO:teuthology.orchestra.run.smithi192.stdout:ceph source packages 6.1 kB/s | 1.9 kB 00:00 2024-06-03T08:25:41.046 INFO:teuthology.orchestra.run.smithi088.stdout:ceph source packages 4.6 kB/s | 1.9 kB 00:00 2024-06-03T08:25:41.439 INFO:teuthology.orchestra.run.smithi192.stdout:CentOS Stream 9 - BaseOS 18 MB/s | 8.1 MB 00:00 2024-06-03T08:25:41.471 INFO:teuthology.orchestra.run.smithi088.stdout:CentOS Stream 9 - BaseOS 21 MB/s | 8.1 MB 00:00 2024-06-03T08:25:43.405 INFO:teuthology.orchestra.run.smithi088.stdout:CentOS Stream 9 - AppStream 23 MB/s | 19 MB 00:00 2024-06-03T08:25:48.339 INFO:teuthology.orchestra.run.smithi088.stdout:CentOS Stream 9 - CRB 4.5 MB/s | 6.2 MB 00:01 2024-06-03T08:25:48.448 INFO:teuthology.orchestra.run.smithi192.stdout:CentOS Stream 9 - AppStream 3.2 MB/s | 19 MB 00:05 2024-06-03T08:25:49.773 INFO:teuthology.orchestra.run.smithi088.stdout:CentOS Stream 9 - Extras packages 67 kB/s | 17 kB 00:00 2024-06-03T08:25:51.252 INFO:teuthology.orchestra.run.smithi088.stdout:Extra Packages for Enterprise Linux 16 MB/s | 21 MB 00:01 2024-06-03T08:25:52.682 INFO:teuthology.orchestra.run.smithi192.stdout:CentOS Stream 9 - CRB 9.4 MB/s | 6.2 MB 00:00 2024-06-03T08:25:54.350 INFO:teuthology.orchestra.run.smithi192.stdout:CentOS Stream 9 - Extras packages 51 kB/s | 17 kB 00:00 2024-06-03T08:25:55.787 INFO:teuthology.orchestra.run.smithi192.stdout:Extra Packages for Enterprise Linux 16 MB/s | 21 MB 00:01 2024-06-03T08:25:56.441 INFO:teuthology.orchestra.run.smithi088.stdout:lab-extras 40 kB/s | 1.7 kB 00:00 2024-06-03T08:25:58.807 INFO:teuthology.orchestra.run.smithi088.stdout:Dependencies resolved. 2024-06-03T08:25:58.821 INFO:teuthology.orchestra.run.smithi088.stdout:================================================================================ 2024-06-03T08:25:58.821 INFO:teuthology.orchestra.run.smithi088.stdout: Package Arch Version Repository Size 2024-06-03T08:25:58.821 INFO:teuthology.orchestra.run.smithi088.stdout:================================================================================ 2024-06-03T08:25:58.821 INFO:teuthology.orchestra.run.smithi088.stdout:Installing: 2024-06-03T08:25:58.821 INFO:teuthology.orchestra.run.smithi088.stdout: ceph-radosgw x86_64 2:18.2.2-1389.gf1abf795.el9 ceph 7.7 M 2024-06-03T08:25:58.821 INFO:teuthology.orchestra.run.smithi088.stdout:Upgrading: 2024-06-03T08:25:58.821 INFO:teuthology.orchestra.run.smithi088.stdout: librados2 x86_64 2:18.2.2-1389.gf1abf795.el9 ceph 3.3 M 2024-06-03T08:25:58.821 INFO:teuthology.orchestra.run.smithi088.stdout: librbd1 x86_64 2:18.2.2-1389.gf1abf795.el9 ceph 3.0 M 2024-06-03T08:25:58.821 INFO:teuthology.orchestra.run.smithi088.stdout:Installing dependencies: 2024-06-03T08:25:58.821 INFO:teuthology.orchestra.run.smithi088.stdout: boost-program-options x86_64 1.75.0-8.el9 appstream 107 k 2024-06-03T08:25:58.821 INFO:teuthology.orchestra.run.smithi088.stdout: ceph-base x86_64 2:18.2.2-1389.gf1abf795.el9 ceph 5.1 M 2024-06-03T08:25:58.821 INFO:teuthology.orchestra.run.smithi088.stdout: ceph-common x86_64 2:18.2.2-1389.gf1abf795.el9 ceph 18 M 2024-06-03T08:25:58.822 INFO:teuthology.orchestra.run.smithi088.stdout: ceph-selinux x86_64 2:18.2.2-1389.gf1abf795.el9 ceph 25 k 2024-06-03T08:25:58.822 INFO:teuthology.orchestra.run.smithi088.stdout: fmt x86_64 8.1.1-5.el9 epel 111 k 2024-06-03T08:25:58.822 INFO:teuthology.orchestra.run.smithi088.stdout: gperftools-libs x86_64 2.9.1-2.el9 epel 309 k 2024-06-03T08:25:58.822 INFO:teuthology.orchestra.run.smithi088.stdout: libarrow x86_64 9.0.0-11.el9 epel 4.4 M 2024-06-03T08:25:58.822 INFO:teuthology.orchestra.run.smithi088.stdout: libarrow-doc noarch 9.0.0-11.el9 epel 26 k 2024-06-03T08:25:58.822 INFO:teuthology.orchestra.run.smithi088.stdout: libcephfs2 x86_64 2:18.2.2-1389.gf1abf795.el9 ceph 710 k 2024-06-03T08:25:58.822 INFO:teuthology.orchestra.run.smithi088.stdout: liboath x86_64 2.6.7-2.el9 epel 49 k 2024-06-03T08:25:58.822 INFO:teuthology.orchestra.run.smithi088.stdout: libpmemobj x86_64 1.12.1-1.el9 appstream 160 k 2024-06-03T08:25:58.822 INFO:teuthology.orchestra.run.smithi088.stdout: librabbitmq x86_64 0.11.0-7.el9 appstream 45 k 2024-06-03T08:25:58.822 INFO:teuthology.orchestra.run.smithi088.stdout: libradosstriper1 x86_64 2:18.2.2-1389.gf1abf795.el9 ceph 474 k 2024-06-03T08:25:58.822 INFO:teuthology.orchestra.run.smithi088.stdout: librdkafka x86_64 1.6.1-102.el9 appstream 662 k 2024-06-03T08:25:58.823 INFO:teuthology.orchestra.run.smithi088.stdout: librgw2 x86_64 2:18.2.2-1389.gf1abf795.el9 ceph 4.5 M 2024-06-03T08:25:58.823 INFO:teuthology.orchestra.run.smithi088.stdout: libunwind x86_64 1.6.2-1.el9 epel 67 k 2024-06-03T08:25:58.823 INFO:teuthology.orchestra.run.smithi088.stdout: lttng-ust x86_64 2.12.0-6.el9 appstream 292 k 2024-06-03T08:25:58.823 INFO:teuthology.orchestra.run.smithi088.stdout: parquet-libs x86_64 9.0.0-11.el9 epel 839 k 2024-06-03T08:25:58.823 INFO:teuthology.orchestra.run.smithi088.stdout: python3-ceph-argparse x86_64 2:18.2.2-1389.gf1abf795.el9 ceph 45 k 2024-06-03T08:25:58.823 INFO:teuthology.orchestra.run.smithi088.stdout: python3-ceph-common x86_64 2:18.2.2-1389.gf1abf795.el9 ceph 129 k 2024-06-03T08:25:58.823 INFO:teuthology.orchestra.run.smithi088.stdout: python3-cephfs x86_64 2:18.2.2-1389.gf1abf795.el9 ceph 162 k 2024-06-03T08:25:58.823 INFO:teuthology.orchestra.run.smithi088.stdout: python3-prettytable noarch 0.7.2-27.el9 appstream 42 k 2024-06-03T08:25:58.823 INFO:teuthology.orchestra.run.smithi088.stdout: python3-rados x86_64 2:18.2.2-1389.gf1abf795.el9 ceph 319 k 2024-06-03T08:25:58.823 INFO:teuthology.orchestra.run.smithi088.stdout: python3-rbd x86_64 2:18.2.2-1389.gf1abf795.el9 ceph 298 k 2024-06-03T08:25:58.823 INFO:teuthology.orchestra.run.smithi088.stdout: python3-rgw x86_64 2:18.2.2-1389.gf1abf795.el9 ceph 99 k 2024-06-03T08:25:58.823 INFO:teuthology.orchestra.run.smithi088.stdout: re2 x86_64 1:20211101-3.el9 epel 192 k 2024-06-03T08:25:58.823 INFO:teuthology.orchestra.run.smithi088.stdout: thrift x86_64 0.15.0-2.el9 epel 1.6 M 2024-06-03T08:25:58.823 INFO:teuthology.orchestra.run.smithi088.stdout: 2024-06-03T08:25:58.824 INFO:teuthology.orchestra.run.smithi088.stdout:Transaction Summary 2024-06-03T08:25:58.824 INFO:teuthology.orchestra.run.smithi088.stdout:================================================================================ 2024-06-03T08:25:58.824 INFO:teuthology.orchestra.run.smithi088.stdout:Install 28 Packages 2024-06-03T08:25:58.824 INFO:teuthology.orchestra.run.smithi088.stdout:Upgrade 2 Packages 2024-06-03T08:25:58.824 INFO:teuthology.orchestra.run.smithi088.stdout: 2024-06-03T08:25:58.824 INFO:teuthology.orchestra.run.smithi088.stdout:Total download size: 53 M 2024-06-03T08:25:58.824 INFO:teuthology.orchestra.run.smithi088.stdout:Downloading Packages: 2024-06-03T08:26:00.258 INFO:teuthology.orchestra.run.smithi088.stdout:(1/30): ceph-base-18.2.2-1389.gf1abf795.el9.x86 4.1 MB/s | 5.1 MB 00:01 2024-06-03T08:26:00.317 INFO:teuthology.orchestra.run.smithi088.stdout:(2/30): ceph-selinux-18.2.2-1389.gf1abf795.el9. 432 kB/s | 25 kB 00:00 2024-06-03T08:26:00.442 INFO:teuthology.orchestra.run.smithi088.stdout:(3/30): libcephfs2-18.2.2-1389.gf1abf795.el9.x8 5.6 MB/s | 710 kB 00:00 2024-06-03T08:26:00.617 INFO:teuthology.orchestra.run.smithi088.stdout:(4/30): ceph-radosgw-18.2.2-1389.gf1abf795.el9. 4.9 MB/s | 7.7 MB 00:01 2024-06-03T08:26:00.642 INFO:teuthology.orchestra.run.smithi088.stdout:(5/30): libradosstriper1-18.2.2-1389.gf1abf795. 2.3 MB/s | 474 kB 00:00 2024-06-03T08:26:00.710 INFO:teuthology.orchestra.run.smithi088.stdout:(6/30): python3-ceph-argparse-18.2.2-1389.gf1ab 674 kB/s | 45 kB 00:00 2024-06-03T08:26:00.793 INFO:teuthology.orchestra.run.smithi088.stdout:(7/30): python3-ceph-common-18.2.2-1389.gf1abf7 1.5 MB/s | 129 kB 00:00 2024-06-03T08:26:00.877 INFO:teuthology.orchestra.run.smithi088.stdout:(8/30): python3-cephfs-18.2.2-1389.gf1abf795.el 1.9 MB/s | 162 kB 00:00 2024-06-03T08:26:00.968 INFO:teuthology.orchestra.run.smithi192.stdout:lab-extras 34 kB/s | 1.7 kB 00:00 2024-06-03T08:26:00.968 INFO:teuthology.orchestra.run.smithi088.stdout:(9/30): python3-rados-18.2.2-1389.gf1abf795.el9 3.4 MB/s | 319 kB 00:00 2024-06-03T08:26:01.061 INFO:teuthology.orchestra.run.smithi088.stdout:(10/30): python3-rbd-18.2.2-1389.gf1abf795.el9. 3.2 MB/s | 298 kB 00:00 2024-06-03T08:26:01.144 INFO:teuthology.orchestra.run.smithi088.stdout:(11/30): librgw2-18.2.2-1389.gf1abf795.el9.x86_ 8.5 MB/s | 4.5 MB 00:00 2024-06-03T08:26:01.190 INFO:teuthology.orchestra.run.smithi088.stdout:(12/30): python3-rgw-18.2.2-1389.gf1abf795.el9. 771 kB/s | 99 kB 00:00 2024-06-03T08:26:01.387 INFO:teuthology.orchestra.run.smithi088.stdout:(13/30): ceph-common-18.2.2-1389.gf1abf795.el9. 7.8 MB/s | 18 MB 00:02 2024-06-03T08:26:01.454 INFO:teuthology.orchestra.run.smithi088.stdout:(14/30): boost-program-options-1.75.0-8.el9.x86 346 kB/s | 107 kB 00:00 2024-06-03T08:26:01.478 INFO:teuthology.orchestra.run.smithi088.stdout:(15/30): libpmemobj-1.12.1-1.el9.x86_64.rpm 555 kB/s | 160 kB 00:00 2024-06-03T08:26:01.504 INFO:teuthology.orchestra.run.smithi088.stdout:(16/30): librabbitmq-0.11.0-7.el9.x86_64.rpm 387 kB/s | 45 kB 00:00 2024-06-03T08:26:01.563 INFO:teuthology.orchestra.run.smithi088.stdout:(17/30): librdkafka-1.6.1-102.el9.x86_64.rpm 5.9 MB/s | 662 kB 00:00 2024-06-03T08:26:01.588 INFO:teuthology.orchestra.run.smithi088.stdout:(18/30): python3-prettytable-0.7.2-27.el9.noarc 505 kB/s | 42 kB 00:00 2024-06-03T08:26:01.613 INFO:teuthology.orchestra.run.smithi088.stdout:(19/30): lttng-ust-2.12.0-6.el9.x86_64.rpm 2.1 MB/s | 292 kB 00:00 2024-06-03T08:26:01.780 INFO:teuthology.orchestra.run.smithi088.stdout:(20/30): fmt-8.1.1-5.el9.x86_64.rpm 510 kB/s | 111 kB 00:00 2024-06-03T08:26:01.838 INFO:teuthology.orchestra.run.smithi088.stdout:(21/30): gperftools-libs-2.9.1-2.el9.x86_64.rpm 1.2 MB/s | 309 kB 00:00 2024-06-03T08:26:01.863 INFO:teuthology.orchestra.run.smithi088.stdout:(22/30): libarrow-doc-9.0.0-11.el9.noarch.rpm 311 kB/s | 26 kB 00:00 2024-06-03T08:26:01.914 INFO:teuthology.orchestra.run.smithi088.stdout:(23/30): libunwind-1.6.2-1.el9.x86_64.rpm 1.3 MB/s | 67 kB 00:00 2024-06-03T08:26:01.939 INFO:teuthology.orchestra.run.smithi088.stdout:(24/30): liboath-2.6.7-2.el9.x86_64.rpm 645 kB/s | 49 kB 00:00 2024-06-03T08:26:02.023 INFO:teuthology.orchestra.run.smithi088.stdout:(25/30): libarrow-9.0.0-11.el9.x86_64.rpm 11 MB/s | 4.4 MB 00:00 2024-06-03T08:26:02.057 INFO:teuthology.orchestra.run.smithi088.stdout:(26/30): parquet-libs-9.0.0-11.el9.x86_64.rpm 5.8 MB/s | 839 kB 00:00 2024-06-03T08:26:02.082 INFO:teuthology.orchestra.run.smithi088.stdout:(27/30): re2-20211101-3.el9.x86_64.rpm 1.3 MB/s | 192 kB 00:00 2024-06-03T08:26:02.124 INFO:teuthology.orchestra.run.smithi088.stdout:(28/30): thrift-0.15.0-2.el9.x86_64.rpm 16 MB/s | 1.6 MB 00:00 2024-06-03T08:26:02.374 INFO:teuthology.orchestra.run.smithi088.stdout:(29/30): librbd1-18.2.2-1389.gf1abf795.el9.x86_ 10 MB/s | 3.0 MB 00:00 2024-06-03T08:26:02.591 INFO:teuthology.orchestra.run.smithi088.stdout:(30/30): librados2-18.2.2-1389.gf1abf795.el9.x8 6.2 MB/s | 3.3 MB 00:00 2024-06-03T08:26:02.593 INFO:teuthology.orchestra.run.smithi088.stdout:-------------------------------------------------------------------------------- 2024-06-03T08:26:02.593 INFO:teuthology.orchestra.run.smithi088.stdout:Total 14 MB/s | 53 MB 00:03 2024-06-03T08:26:02.663 INFO:teuthology.orchestra.run.smithi088.stdout:Running transaction check 2024-06-03T08:26:02.696 INFO:teuthology.orchestra.run.smithi088.stdout:Transaction check succeeded. 2024-06-03T08:26:02.697 INFO:teuthology.orchestra.run.smithi088.stdout:Running transaction test 2024-06-03T08:26:03.200 INFO:teuthology.orchestra.run.smithi088.stdout:Transaction test succeeded. 2024-06-03T08:26:03.200 INFO:teuthology.orchestra.run.smithi088.stdout:Running transaction 2024-06-03T08:26:03.351 INFO:teuthology.orchestra.run.smithi192.stdout:Dependencies resolved. 2024-06-03T08:26:03.361 INFO:teuthology.orchestra.run.smithi192.stdout:================================================================================ 2024-06-03T08:26:03.361 INFO:teuthology.orchestra.run.smithi192.stdout: Package Arch Version Repository Size 2024-06-03T08:26:03.362 INFO:teuthology.orchestra.run.smithi192.stdout:================================================================================ 2024-06-03T08:26:03.362 INFO:teuthology.orchestra.run.smithi192.stdout:Installing: 2024-06-03T08:26:03.362 INFO:teuthology.orchestra.run.smithi192.stdout: ceph-radosgw x86_64 2:18.2.2-1389.gf1abf795.el9 ceph 7.7 M 2024-06-03T08:26:03.362 INFO:teuthology.orchestra.run.smithi192.stdout:Upgrading: 2024-06-03T08:26:03.362 INFO:teuthology.orchestra.run.smithi192.stdout: librados2 x86_64 2:18.2.2-1389.gf1abf795.el9 ceph 3.3 M 2024-06-03T08:26:03.362 INFO:teuthology.orchestra.run.smithi192.stdout: librbd1 x86_64 2:18.2.2-1389.gf1abf795.el9 ceph 3.0 M 2024-06-03T08:26:03.362 INFO:teuthology.orchestra.run.smithi192.stdout:Installing dependencies: 2024-06-03T08:26:03.362 INFO:teuthology.orchestra.run.smithi192.stdout: boost-program-options x86_64 1.75.0-8.el9 appstream 107 k 2024-06-03T08:26:03.362 INFO:teuthology.orchestra.run.smithi192.stdout: ceph-base x86_64 2:18.2.2-1389.gf1abf795.el9 ceph 5.1 M 2024-06-03T08:26:03.362 INFO:teuthology.orchestra.run.smithi192.stdout: ceph-common x86_64 2:18.2.2-1389.gf1abf795.el9 ceph 18 M 2024-06-03T08:26:03.362 INFO:teuthology.orchestra.run.smithi192.stdout: ceph-selinux x86_64 2:18.2.2-1389.gf1abf795.el9 ceph 25 k 2024-06-03T08:26:03.362 INFO:teuthology.orchestra.run.smithi192.stdout: fmt x86_64 8.1.1-5.el9 epel 111 k 2024-06-03T08:26:03.362 INFO:teuthology.orchestra.run.smithi192.stdout: gperftools-libs x86_64 2.9.1-2.el9 epel 309 k 2024-06-03T08:26:03.363 INFO:teuthology.orchestra.run.smithi192.stdout: libarrow x86_64 9.0.0-11.el9 epel 4.4 M 2024-06-03T08:26:03.363 INFO:teuthology.orchestra.run.smithi192.stdout: libarrow-doc noarch 9.0.0-11.el9 epel 26 k 2024-06-03T08:26:03.363 INFO:teuthology.orchestra.run.smithi192.stdout: libcephfs2 x86_64 2:18.2.2-1389.gf1abf795.el9 ceph 710 k 2024-06-03T08:26:03.363 INFO:teuthology.orchestra.run.smithi192.stdout: liboath x86_64 2.6.7-2.el9 epel 49 k 2024-06-03T08:26:03.363 INFO:teuthology.orchestra.run.smithi192.stdout: libpmemobj x86_64 1.12.1-1.el9 appstream 160 k 2024-06-03T08:26:03.363 INFO:teuthology.orchestra.run.smithi192.stdout: librabbitmq x86_64 0.11.0-7.el9 appstream 45 k 2024-06-03T08:26:03.363 INFO:teuthology.orchestra.run.smithi192.stdout: libradosstriper1 x86_64 2:18.2.2-1389.gf1abf795.el9 ceph 474 k 2024-06-03T08:26:03.363 INFO:teuthology.orchestra.run.smithi192.stdout: librdkafka x86_64 1.6.1-102.el9 appstream 662 k 2024-06-03T08:26:03.363 INFO:teuthology.orchestra.run.smithi192.stdout: librgw2 x86_64 2:18.2.2-1389.gf1abf795.el9 ceph 4.5 M 2024-06-03T08:26:03.363 INFO:teuthology.orchestra.run.smithi192.stdout: libunwind x86_64 1.6.2-1.el9 epel 67 k 2024-06-03T08:26:03.363 INFO:teuthology.orchestra.run.smithi192.stdout: lttng-ust x86_64 2.12.0-6.el9 appstream 292 k 2024-06-03T08:26:03.363 INFO:teuthology.orchestra.run.smithi192.stdout: parquet-libs x86_64 9.0.0-11.el9 epel 839 k 2024-06-03T08:26:03.363 INFO:teuthology.orchestra.run.smithi192.stdout: python3-ceph-argparse x86_64 2:18.2.2-1389.gf1abf795.el9 ceph 45 k 2024-06-03T08:26:03.364 INFO:teuthology.orchestra.run.smithi192.stdout: python3-ceph-common x86_64 2:18.2.2-1389.gf1abf795.el9 ceph 129 k 2024-06-03T08:26:03.364 INFO:teuthology.orchestra.run.smithi192.stdout: python3-cephfs x86_64 2:18.2.2-1389.gf1abf795.el9 ceph 162 k 2024-06-03T08:26:03.364 INFO:teuthology.orchestra.run.smithi192.stdout: python3-prettytable noarch 0.7.2-27.el9 appstream 42 k 2024-06-03T08:26:03.364 INFO:teuthology.orchestra.run.smithi192.stdout: python3-rados x86_64 2:18.2.2-1389.gf1abf795.el9 ceph 319 k 2024-06-03T08:26:03.364 INFO:teuthology.orchestra.run.smithi192.stdout: python3-rbd x86_64 2:18.2.2-1389.gf1abf795.el9 ceph 298 k 2024-06-03T08:26:03.364 INFO:teuthology.orchestra.run.smithi192.stdout: python3-rgw x86_64 2:18.2.2-1389.gf1abf795.el9 ceph 99 k 2024-06-03T08:26:03.364 INFO:teuthology.orchestra.run.smithi192.stdout: re2 x86_64 1:20211101-3.el9 epel 192 k 2024-06-03T08:26:03.364 INFO:teuthology.orchestra.run.smithi192.stdout: thrift x86_64 0.15.0-2.el9 epel 1.6 M 2024-06-03T08:26:03.364 INFO:teuthology.orchestra.run.smithi192.stdout: 2024-06-03T08:26:03.364 INFO:teuthology.orchestra.run.smithi192.stdout:Transaction Summary 2024-06-03T08:26:03.364 INFO:teuthology.orchestra.run.smithi192.stdout:================================================================================ 2024-06-03T08:26:03.364 INFO:teuthology.orchestra.run.smithi192.stdout:Install 28 Packages 2024-06-03T08:26:03.364 INFO:teuthology.orchestra.run.smithi192.stdout:Upgrade 2 Packages 2024-06-03T08:26:03.364 INFO:teuthology.orchestra.run.smithi192.stdout: 2024-06-03T08:26:03.365 INFO:teuthology.orchestra.run.smithi192.stdout:Total download size: 53 M 2024-06-03T08:26:03.365 INFO:teuthology.orchestra.run.smithi192.stdout:Downloading Packages: 2024-06-03T08:26:04.361 INFO:teuthology.orchestra.run.smithi192.stdout:(1/30): ceph-base-18.2.2-1389.gf1abf795.el9.x86 6.3 MB/s | 5.1 MB 00:00 2024-06-03T08:26:04.419 INFO:teuthology.orchestra.run.smithi192.stdout:(2/30): ceph-selinux-18.2.2-1389.gf1abf795.el9. 432 kB/s | 25 kB 00:00 2024-06-03T08:26:04.523 INFO:teuthology.orchestra.run.smithi088.stdout: Preparing : 1/1 2024-06-03T08:26:04.529 INFO:teuthology.orchestra.run.smithi192.stdout:(3/30): libcephfs2-18.2.2-1389.gf1abf795.el9.x8 6.4 MB/s | 710 kB 00:00 2024-06-03T08:26:04.604 INFO:teuthology.orchestra.run.smithi088.stdout: Installing : lttng-ust-2.12.0-6.el9.x86_64 1/32 2024-06-03T08:26:04.628 INFO:teuthology.orchestra.run.smithi192.stdout:(4/30): libradosstriper1-18.2.2-1389.gf1abf795. 4.6 MB/s | 474 kB 00:00 2024-06-03T08:26:04.637 INFO:teuthology.orchestra.run.smithi088.stdout: Installing : librdkafka-1.6.1-102.el9.x86_64 2/32 2024-06-03T08:26:04.688 INFO:teuthology.orchestra.run.smithi088.stdout: Installing : librabbitmq-0.11.0-7.el9.x86_64 3/32 2024-06-03T08:26:05.061 INFO:teuthology.orchestra.run.smithi088.stdout: Installing : thrift-0.15.0-2.el9.x86_64 4/32 2024-06-03T08:26:05.120 INFO:teuthology.orchestra.run.smithi088.stdout: Upgrading : librados2-2:18.2.2-1389.gf1abf795.el9.x86_64 5/32 2024-06-03T08:26:05.137 INFO:teuthology.orchestra.run.smithi192.stdout:(5/30): librgw2-18.2.2-1389.gf1abf795.el9.x86_6 8.8 MB/s | 4.5 MB 00:00 2024-06-03T08:26:05.195 INFO:teuthology.orchestra.run.smithi088.stdout: Running scriptlet: librados2-2:18.2.2-1389.gf1abf795.el9.x86_64 5/32 2024-06-03T08:26:05.280 INFO:teuthology.orchestra.run.smithi088.stdout: Installing : python3-rados-2:18.2.2-1389.gf1abf795.el9.x86_64 6/32 2024-06-03T08:26:05.304 INFO:teuthology.orchestra.run.smithi088.stdout: Installing : libcephfs2-2:18.2.2-1389.gf1abf795.el9.x86_64 7/32 2024-06-03T08:26:05.320 INFO:teuthology.orchestra.run.smithi192.stdout:(6/30): ceph-common-18.2.2-1389.gf1abf795.el9.x 10 MB/s | 18 MB 00:01 2024-06-03T08:26:05.346 INFO:teuthology.orchestra.run.smithi192.stdout:(7/30): python3-ceph-argparse-18.2.2-1389.gf1ab 215 kB/s | 45 kB 00:00 2024-06-03T08:26:05.367 INFO:teuthology.orchestra.run.smithi088.stdout: Running scriptlet: libcephfs2-2:18.2.2-1389.gf1abf795.el9.x86_64 7/32 2024-06-03T08:26:05.404 INFO:teuthology.orchestra.run.smithi088.stdout: Installing : liboath-2.6.7-2.el9.x86_64 8/32 2024-06-03T08:26:05.438 INFO:teuthology.orchestra.run.smithi192.stdout:(8/30): ceph-radosgw-18.2.2-1389.gf1abf795.el9. 4.1 MB/s | 7.7 MB 00:01 2024-06-03T08:26:05.448 INFO:teuthology.orchestra.run.smithi088.stdout: Installing : python3-ceph-argparse-2:18.2.2-1389.gf1abf795.el9. 9/32 2024-06-03T08:26:05.463 INFO:teuthology.orchestra.run.smithi192.stdout:(9/30): python3-ceph-common-18.2.2-1389.gf1abf7 908 kB/s | 129 kB 00:00 2024-06-03T08:26:05.480 INFO:teuthology.orchestra.run.smithi192.stdout:(10/30): python3-cephfs-18.2.2-1389.gf1abf795.e 1.2 MB/s | 162 kB 00:00 2024-06-03T08:26:05.505 INFO:teuthology.orchestra.run.smithi192.stdout:(11/30): python3-rados-18.2.2-1389.gf1abf795.el 4.6 MB/s | 319 kB 00:00 2024-06-03T08:26:05.514 INFO:teuthology.orchestra.run.smithi088.stdout: Installing : python3-cephfs-2:18.2.2-1389.gf1abf795.el9.x86_64 10/32 2024-06-03T08:26:05.530 INFO:teuthology.orchestra.run.smithi192.stdout:(12/30): python3-rbd-18.2.2-1389.gf1abf795.el9. 4.3 MB/s | 298 kB 00:00 2024-06-03T08:26:05.540 INFO:teuthology.orchestra.run.smithi088.stdout: Installing : libradosstriper1-2:18.2.2-1389.gf1abf795.el9.x86_6 11/32 2024-06-03T08:26:05.556 INFO:teuthology.orchestra.run.smithi192.stdout:(13/30): python3-rgw-18.2.2-1389.gf1abf795.el9. 1.3 MB/s | 99 kB 00:00 2024-06-03T08:26:05.606 INFO:teuthology.orchestra.run.smithi088.stdout: Running scriptlet: libradosstriper1-2:18.2.2-1389.gf1abf795.el9.x86_6 11/32 2024-06-03T08:26:05.711 INFO:teuthology.orchestra.run.smithi088.stdout: Installing : re2-1:20211101-3.el9.x86_64 12/32 2024-06-03T08:26:05.740 INFO:teuthology.orchestra.run.smithi088.stdout: Installing : libarrow-9.0.0-11.el9.x86_64 13/32 2024-06-03T08:26:05.780 INFO:teuthology.orchestra.run.smithi088.stdout: Installing : libunwind-1.6.2-1.el9.x86_64 14/32 2024-06-03T08:26:05.812 INFO:teuthology.orchestra.run.smithi088.stdout: Installing : gperftools-libs-2.9.1-2.el9.x86_64 15/32 2024-06-03T08:26:05.839 INFO:teuthology.orchestra.run.smithi088.stdout: Installing : libarrow-doc-9.0.0-11.el9.noarch 16/32 2024-06-03T08:26:05.875 INFO:teuthology.orchestra.run.smithi088.stdout: Installing : fmt-8.1.1-5.el9.x86_64 17/32 2024-06-03T08:26:05.908 INFO:teuthology.orchestra.run.smithi088.stdout: Installing : python3-prettytable-0.7.2-27.el9.noarch 18/32 2024-06-03T08:26:05.914 INFO:teuthology.orchestra.run.smithi192.stdout:(14/30): librabbitmq-0.11.0-7.el9.x86_64.rpm 126 kB/s | 45 kB 00:00 2024-06-03T08:26:06.015 INFO:teuthology.orchestra.run.smithi192.stdout:(15/30): boost-program-options-1.75.0-8.el9.x86 210 kB/s | 107 kB 00:00 2024-06-03T08:26:06.115 INFO:teuthology.orchestra.run.smithi192.stdout:(16/30): libpmemobj-1.12.1-1.el9.x86_64.rpm 274 kB/s | 160 kB 00:00 2024-06-03T08:26:06.207 INFO:teuthology.orchestra.run.smithi088.stdout: Installing : libpmemobj-1.12.1-1.el9.x86_64 19/32 2024-06-03T08:26:06.246 INFO:teuthology.orchestra.run.smithi088.stdout: Upgrading : librbd1-2:18.2.2-1389.gf1abf795.el9.x86_64 20/32 2024-06-03T08:26:06.337 INFO:teuthology.orchestra.run.smithi088.stdout: Running scriptlet: librbd1-2:18.2.2-1389.gf1abf795.el9.x86_64 20/32 2024-06-03T08:26:06.367 INFO:teuthology.orchestra.run.smithi088.stdout: Installing : python3-rbd-2:18.2.2-1389.gf1abf795.el9.x86_64 21/32 2024-06-03T08:26:06.403 INFO:teuthology.orchestra.run.smithi088.stdout: Installing : boost-program-options-1.75.0-8.el9.x86_64 22/32 2024-06-03T08:26:06.465 INFO:teuthology.orchestra.run.smithi192.stdout:(17/30): python3-prettytable-0.7.2-27.el9.noarc 121 kB/s | 42 kB 00:00 2024-06-03T08:26:06.666 INFO:teuthology.orchestra.run.smithi192.stdout:(18/30): librdkafka-1.6.1-102.el9.x86_64.rpm 882 kB/s | 662 kB 00:00 2024-06-03T08:26:06.691 INFO:teuthology.orchestra.run.smithi192.stdout:(19/30): fmt-8.1.1-5.el9.x86_64.rpm 491 kB/s | 111 kB 00:00 2024-06-03T08:26:06.716 INFO:teuthology.orchestra.run.smithi192.stdout:(20/30): lttng-ust-2.12.0-6.el9.x86_64.rpm 417 kB/s | 292 kB 00:00 2024-06-03T08:26:06.805 INFO:teuthology.orchestra.run.smithi088.stdout: Installing : parquet-libs-9.0.0-11.el9.x86_64 23/32 2024-06-03T08:26:06.841 INFO:teuthology.orchestra.run.smithi192.stdout:(21/30): libarrow-doc-9.0.0-11.el9.noarch.rpm 208 kB/s | 26 kB 00:00 2024-06-03T08:26:06.856 INFO:teuthology.orchestra.run.smithi088.stdout: Installing : librgw2-2:18.2.2-1389.gf1abf795.el9.x86_64 24/32 2024-06-03T08:26:06.917 INFO:teuthology.orchestra.run.smithi192.stdout:(22/30): gperftools-libs-2.9.1-2.el9.x86_64.rpm 1.2 MB/s | 309 kB 00:00 2024-06-03T08:26:06.927 INFO:teuthology.orchestra.run.smithi088.stdout: Running scriptlet: librgw2-2:18.2.2-1389.gf1abf795.el9.x86_64 24/32 2024-06-03T08:26:06.942 INFO:teuthology.orchestra.run.smithi192.stdout:(23/30): liboath-2.6.7-2.el9.x86_64.rpm 484 kB/s | 49 kB 00:00 2024-06-03T08:26:06.997 INFO:teuthology.orchestra.run.smithi088.stdout: Installing : python3-rgw-2:18.2.2-1389.gf1abf795.el9.x86_64 25/32 2024-06-03T08:26:07.009 INFO:teuthology.orchestra.run.smithi192.stdout:(24/30): libarrow-9.0.0-11.el9.x86_64.rpm 14 MB/s | 4.4 MB 00:00 2024-06-03T08:26:07.026 INFO:teuthology.orchestra.run.smithi088.stdout: Installing : python3-ceph-common-2:18.2.2-1389.gf1abf795.el9.x8 26/32 2024-06-03T08:26:07.034 INFO:teuthology.orchestra.run.smithi192.stdout:(25/30): libunwind-1.6.2-1.el9.x86_64.rpm 575 kB/s | 67 kB 00:00 2024-06-03T08:26:07.068 INFO:teuthology.orchestra.run.smithi192.stdout:(26/30): re2-20211101-3.el9.x86_64.rpm 3.2 MB/s | 192 kB 00:00 2024-06-03T08:26:07.152 INFO:teuthology.orchestra.run.smithi192.stdout:(27/30): parquet-libs-9.0.0-11.el9.x86_64.rpm 3.9 MB/s | 839 kB 00:00 2024-06-03T08:26:07.202 INFO:teuthology.orchestra.run.smithi192.stdout:(28/30): thrift-0.15.0-2.el9.x86_64.rpm 9.5 MB/s | 1.6 MB 00:00 2024-06-03T08:26:07.502 INFO:teuthology.orchestra.run.smithi192.stdout:(29/30): librados2-18.2.2-1389.gf1abf795.el9.x8 7.6 MB/s | 3.3 MB 00:00 2024-06-03T08:26:07.569 INFO:teuthology.orchestra.run.smithi192.stdout:(30/30): librbd1-18.2.2-1389.gf1abf795.el9.x86_ 7.2 MB/s | 3.0 MB 00:00 2024-06-03T08:26:07.571 INFO:teuthology.orchestra.run.smithi192.stdout:-------------------------------------------------------------------------------- 2024-06-03T08:26:07.572 INFO:teuthology.orchestra.run.smithi192.stdout:Total 13 MB/s | 53 MB 00:04 2024-06-03T08:26:07.644 INFO:teuthology.orchestra.run.smithi192.stdout:Running transaction check 2024-06-03T08:26:07.680 INFO:teuthology.orchestra.run.smithi192.stdout:Transaction check succeeded. 2024-06-03T08:26:07.680 INFO:teuthology.orchestra.run.smithi192.stdout:Running transaction test 2024-06-03T08:26:08.201 INFO:teuthology.orchestra.run.smithi192.stdout:Transaction test succeeded. 2024-06-03T08:26:08.202 INFO:teuthology.orchestra.run.smithi192.stdout:Running transaction 2024-06-03T08:26:09.076 INFO:teuthology.orchestra.run.smithi088.stdout: Running scriptlet: ceph-common-2:18.2.2-1389.gf1abf795.el9.x86_64 27/32 2024-06-03T08:26:09.106 INFO:teuthology.orchestra.run.smithi088.stdout: Installing : ceph-common-2:18.2.2-1389.gf1abf795.el9.x86_64 27/32 2024-06-03T08:26:09.517 INFO:teuthology.orchestra.run.smithi192.stdout: Preparing : 1/1 2024-06-03T08:26:09.603 INFO:teuthology.orchestra.run.smithi192.stdout: Installing : lttng-ust-2.12.0-6.el9.x86_64 1/32 2024-06-03T08:26:09.632 INFO:teuthology.orchestra.run.smithi192.stdout: Installing : librdkafka-1.6.1-102.el9.x86_64 2/32 2024-06-03T08:26:09.640 INFO:teuthology.orchestra.run.smithi088.stdout: Running scriptlet: ceph-common-2:18.2.2-1389.gf1abf795.el9.x86_64 27/32 2024-06-03T08:26:09.685 INFO:teuthology.orchestra.run.smithi088.stdout: Installing : ceph-base-2:18.2.2-1389.gf1abf795.el9.x86_64 28/32 2024-06-03T08:26:09.699 INFO:teuthology.orchestra.run.smithi192.stdout: Installing : librabbitmq-0.11.0-7.el9.x86_64 3/32 2024-06-03T08:26:09.784 INFO:teuthology.orchestra.run.smithi088.stdout: Running scriptlet: ceph-base-2:18.2.2-1389.gf1abf795.el9.x86_64 28/32 2024-06-03T08:26:09.784 INFO:teuthology.orchestra.run.smithi088.stdout:Created symlink /etc/systemd/system/multi-user.target.wants/ceph.target → /usr/lib/systemd/system/ceph.target. 2024-06-03T08:26:09.784 INFO:teuthology.orchestra.run.smithi088.stdout:Created symlink /etc/systemd/system/ceph.target.wants/ceph-crash.service → /usr/lib/systemd/system/ceph-crash.service. 2024-06-03T08:26:09.784 INFO:teuthology.orchestra.run.smithi088.stdout: 2024-06-03T08:26:09.810 INFO:teuthology.orchestra.run.smithi088.stdout: Installing : ceph-selinux-2:18.2.2-1389.gf1abf795.el9.x86_64 29/32 2024-06-03T08:26:10.077 INFO:teuthology.orchestra.run.smithi192.stdout: Installing : thrift-0.15.0-2.el9.x86_64 4/32 2024-06-03T08:26:10.131 INFO:teuthology.orchestra.run.smithi192.stdout: Upgrading : librados2-2:18.2.2-1389.gf1abf795.el9.x86_64 5/32 2024-06-03T08:26:10.214 INFO:teuthology.orchestra.run.smithi192.stdout: Running scriptlet: librados2-2:18.2.2-1389.gf1abf795.el9.x86_64 5/32 2024-06-03T08:26:10.299 INFO:teuthology.orchestra.run.smithi192.stdout: Installing : python3-rados-2:18.2.2-1389.gf1abf795.el9.x86_64 6/32 2024-06-03T08:26:10.325 INFO:teuthology.orchestra.run.smithi192.stdout: Installing : libcephfs2-2:18.2.2-1389.gf1abf795.el9.x86_64 7/32 2024-06-03T08:26:10.387 INFO:teuthology.orchestra.run.smithi192.stdout: Running scriptlet: libcephfs2-2:18.2.2-1389.gf1abf795.el9.x86_64 7/32 2024-06-03T08:26:10.416 INFO:teuthology.orchestra.run.smithi192.stdout: Installing : liboath-2.6.7-2.el9.x86_64 8/32 2024-06-03T08:26:10.460 INFO:teuthology.orchestra.run.smithi192.stdout: Installing : python3-ceph-argparse-2:18.2.2-1389.gf1abf795.el9. 9/32 2024-06-03T08:26:10.525 INFO:teuthology.orchestra.run.smithi192.stdout: Installing : python3-cephfs-2:18.2.2-1389.gf1abf795.el9.x86_64 10/32 2024-06-03T08:26:10.552 INFO:teuthology.orchestra.run.smithi192.stdout: Installing : libradosstriper1-2:18.2.2-1389.gf1abf795.el9.x86_6 11/32 2024-06-03T08:26:10.617 INFO:teuthology.orchestra.run.smithi192.stdout: Running scriptlet: libradosstriper1-2:18.2.2-1389.gf1abf795.el9.x86_6 11/32 2024-06-03T08:26:10.724 INFO:teuthology.orchestra.run.smithi192.stdout: Installing : re2-1:20211101-3.el9.x86_64 12/32 2024-06-03T08:26:10.760 INFO:teuthology.orchestra.run.smithi192.stdout: Installing : libarrow-9.0.0-11.el9.x86_64 13/32 2024-06-03T08:26:10.802 INFO:teuthology.orchestra.run.smithi192.stdout: Installing : libunwind-1.6.2-1.el9.x86_64 14/32 2024-06-03T08:26:10.841 INFO:teuthology.orchestra.run.smithi192.stdout: Installing : gperftools-libs-2.9.1-2.el9.x86_64 15/32 2024-06-03T08:26:10.868 INFO:teuthology.orchestra.run.smithi192.stdout: Installing : libarrow-doc-9.0.0-11.el9.noarch 16/32 2024-06-03T08:26:10.904 INFO:teuthology.orchestra.run.smithi192.stdout: Installing : fmt-8.1.1-5.el9.x86_64 17/32 2024-06-03T08:26:10.937 INFO:teuthology.orchestra.run.smithi192.stdout: Installing : python3-prettytable-0.7.2-27.el9.noarch 18/32 2024-06-03T08:26:11.237 INFO:teuthology.orchestra.run.smithi192.stdout: Installing : libpmemobj-1.12.1-1.el9.x86_64 19/32 2024-06-03T08:26:11.283 INFO:teuthology.orchestra.run.smithi192.stdout: Upgrading : librbd1-2:18.2.2-1389.gf1abf795.el9.x86_64 20/32 2024-06-03T08:26:11.357 INFO:teuthology.orchestra.run.smithi192.stdout: Running scriptlet: librbd1-2:18.2.2-1389.gf1abf795.el9.x86_64 20/32 2024-06-03T08:26:11.388 INFO:teuthology.orchestra.run.smithi192.stdout: Installing : python3-rbd-2:18.2.2-1389.gf1abf795.el9.x86_64 21/32 2024-06-03T08:26:11.424 INFO:teuthology.orchestra.run.smithi192.stdout: Installing : boost-program-options-1.75.0-8.el9.x86_64 22/32 2024-06-03T08:26:11.824 INFO:teuthology.orchestra.run.smithi192.stdout: Installing : parquet-libs-9.0.0-11.el9.x86_64 23/32 2024-06-03T08:26:11.852 INFO:teuthology.orchestra.run.smithi192.stdout: Installing : librgw2-2:18.2.2-1389.gf1abf795.el9.x86_64 24/32 2024-06-03T08:26:11.923 INFO:teuthology.orchestra.run.smithi192.stdout: Running scriptlet: librgw2-2:18.2.2-1389.gf1abf795.el9.x86_64 24/32 2024-06-03T08:26:11.991 INFO:teuthology.orchestra.run.smithi192.stdout: Installing : python3-rgw-2:18.2.2-1389.gf1abf795.el9.x86_64 25/32 2024-06-03T08:26:12.022 INFO:teuthology.orchestra.run.smithi192.stdout: Installing : python3-ceph-common-2:18.2.2-1389.gf1abf795.el9.x8 26/32 2024-06-03T08:26:14.140 INFO:teuthology.orchestra.run.smithi192.stdout: Running scriptlet: ceph-common-2:18.2.2-1389.gf1abf795.el9.x86_64 27/32 2024-06-03T08:26:14.169 INFO:teuthology.orchestra.run.smithi192.stdout: Installing : ceph-common-2:18.2.2-1389.gf1abf795.el9.x86_64 27/32 2024-06-03T08:26:14.728 INFO:teuthology.orchestra.run.smithi192.stdout: Running scriptlet: ceph-common-2:18.2.2-1389.gf1abf795.el9.x86_64 27/32 2024-06-03T08:26:14.798 INFO:teuthology.orchestra.run.smithi192.stdout: Installing : ceph-base-2:18.2.2-1389.gf1abf795.el9.x86_64 28/32 2024-06-03T08:26:14.892 INFO:teuthology.orchestra.run.smithi192.stdout: Running scriptlet: ceph-base-2:18.2.2-1389.gf1abf795.el9.x86_64 28/32 2024-06-03T08:26:14.892 INFO:teuthology.orchestra.run.smithi192.stdout:Created symlink /etc/systemd/system/multi-user.target.wants/ceph.target → /usr/lib/systemd/system/ceph.target. 2024-06-03T08:26:14.893 INFO:teuthology.orchestra.run.smithi192.stdout:Created symlink /etc/systemd/system/ceph.target.wants/ceph-crash.service → /usr/lib/systemd/system/ceph-crash.service. 2024-06-03T08:26:14.893 INFO:teuthology.orchestra.run.smithi192.stdout: 2024-06-03T08:26:14.924 INFO:teuthology.orchestra.run.smithi192.stdout: Installing : ceph-selinux-2:18.2.2-1389.gf1abf795.el9.x86_64 29/32 2024-06-03T08:26:19.506 INFO:teuthology.orchestra.run.smithi088.stdout: Running scriptlet: ceph-selinux-2:18.2.2-1389.gf1abf795.el9.x86_64 29/32 2024-06-03T08:26:19.506 INFO:teuthology.orchestra.run.smithi088.stdout:skipping the directory /sys 2024-06-03T08:26:19.506 INFO:teuthology.orchestra.run.smithi088.stdout:skipping the directory /proc 2024-06-03T08:26:19.506 INFO:teuthology.orchestra.run.smithi088.stdout:skipping the directory /mnt 2024-06-03T08:26:19.506 INFO:teuthology.orchestra.run.smithi088.stdout:skipping the directory /var/tmp 2024-06-03T08:26:19.506 INFO:teuthology.orchestra.run.smithi088.stdout:skipping the directory /home 2024-06-03T08:26:19.506 INFO:teuthology.orchestra.run.smithi088.stdout:skipping the directory /root 2024-06-03T08:26:19.506 INFO:teuthology.orchestra.run.smithi088.stdout:skipping the directory /tmp 2024-06-03T08:26:19.506 INFO:teuthology.orchestra.run.smithi088.stdout: 2024-06-03T08:26:20.260 INFO:teuthology.orchestra.run.smithi088.stdout: Installing : ceph-radosgw-2:18.2.2-1389.gf1abf795.el9.x86_64 30/32 2024-06-03T08:26:20.285 INFO:teuthology.orchestra.run.smithi088.stdout: Running scriptlet: ceph-radosgw-2:18.2.2-1389.gf1abf795.el9.x86_64 30/32 2024-06-03T08:26:20.286 INFO:teuthology.orchestra.run.smithi088.stdout:Glob pattern passed to enable, but globs are not supported for this. 2024-06-03T08:26:20.286 INFO:teuthology.orchestra.run.smithi088.stdout:Invalid unit name "ceph-radosgw@*.service" escaped as "ceph-radosgw@\x2a.service". 2024-06-03T08:26:20.286 INFO:teuthology.orchestra.run.smithi088.stdout:Created symlink /etc/systemd/system/multi-user.target.wants/ceph-radosgw.target → /usr/lib/systemd/system/ceph-radosgw.target. 2024-06-03T08:26:20.286 INFO:teuthology.orchestra.run.smithi088.stdout:Created symlink /etc/systemd/system/ceph.target.wants/ceph-radosgw.target → /usr/lib/systemd/system/ceph-radosgw.target. 2024-06-03T08:26:20.286 INFO:teuthology.orchestra.run.smithi088.stdout: 2024-06-03T08:26:20.288 INFO:teuthology.orchestra.run.smithi088.stdout: Cleanup : librbd1-2:16.2.4-5.el9.x86_64 31/32 2024-06-03T08:26:20.372 INFO:teuthology.orchestra.run.smithi088.stdout: Running scriptlet: librbd1-2:16.2.4-5.el9.x86_64 31/32 2024-06-03T08:26:20.372 INFO:teuthology.orchestra.run.smithi088.stdout: Cleanup : librados2-2:16.2.4-5.el9.x86_64 32/32 2024-06-03T08:26:21.432 INFO:teuthology.orchestra.run.smithi088.stdout: Running scriptlet: librados2-2:16.2.4-5.el9.x86_64 32/32 2024-06-03T08:26:21.433 INFO:teuthology.orchestra.run.smithi088.stdout: Verifying : ceph-base-2:18.2.2-1389.gf1abf795.el9.x86_64 1/32 2024-06-03T08:26:21.433 INFO:teuthology.orchestra.run.smithi088.stdout: Verifying : ceph-common-2:18.2.2-1389.gf1abf795.el9.x86_64 2/32 2024-06-03T08:26:21.433 INFO:teuthology.orchestra.run.smithi088.stdout: Verifying : ceph-radosgw-2:18.2.2-1389.gf1abf795.el9.x86_64 3/32 2024-06-03T08:26:21.433 INFO:teuthology.orchestra.run.smithi088.stdout: Verifying : ceph-selinux-2:18.2.2-1389.gf1abf795.el9.x86_64 4/32 2024-06-03T08:26:21.433 INFO:teuthology.orchestra.run.smithi088.stdout: Verifying : libcephfs2-2:18.2.2-1389.gf1abf795.el9.x86_64 5/32 2024-06-03T08:26:21.433 INFO:teuthology.orchestra.run.smithi088.stdout: Verifying : libradosstriper1-2:18.2.2-1389.gf1abf795.el9.x86_6 6/32 2024-06-03T08:26:21.434 INFO:teuthology.orchestra.run.smithi088.stdout: Verifying : librgw2-2:18.2.2-1389.gf1abf795.el9.x86_64 7/32 2024-06-03T08:26:21.434 INFO:teuthology.orchestra.run.smithi088.stdout: Verifying : python3-ceph-argparse-2:18.2.2-1389.gf1abf795.el9. 8/32 2024-06-03T08:26:21.434 INFO:teuthology.orchestra.run.smithi088.stdout: Verifying : python3-ceph-common-2:18.2.2-1389.gf1abf795.el9.x8 9/32 2024-06-03T08:26:21.434 INFO:teuthology.orchestra.run.smithi088.stdout: Verifying : python3-cephfs-2:18.2.2-1389.gf1abf795.el9.x86_64 10/32 2024-06-03T08:26:21.434 INFO:teuthology.orchestra.run.smithi088.stdout: Verifying : python3-rados-2:18.2.2-1389.gf1abf795.el9.x86_64 11/32 2024-06-03T08:26:21.434 INFO:teuthology.orchestra.run.smithi088.stdout: Verifying : python3-rbd-2:18.2.2-1389.gf1abf795.el9.x86_64 12/32 2024-06-03T08:26:21.434 INFO:teuthology.orchestra.run.smithi088.stdout: Verifying : python3-rgw-2:18.2.2-1389.gf1abf795.el9.x86_64 13/32 2024-06-03T08:26:21.435 INFO:teuthology.orchestra.run.smithi088.stdout: Verifying : boost-program-options-1.75.0-8.el9.x86_64 14/32 2024-06-03T08:26:21.435 INFO:teuthology.orchestra.run.smithi088.stdout: Verifying : libpmemobj-1.12.1-1.el9.x86_64 15/32 2024-06-03T08:26:21.435 INFO:teuthology.orchestra.run.smithi088.stdout: Verifying : librabbitmq-0.11.0-7.el9.x86_64 16/32 2024-06-03T08:26:21.435 INFO:teuthology.orchestra.run.smithi088.stdout: Verifying : librdkafka-1.6.1-102.el9.x86_64 17/32 2024-06-03T08:26:21.435 INFO:teuthology.orchestra.run.smithi088.stdout: Verifying : lttng-ust-2.12.0-6.el9.x86_64 18/32 2024-06-03T08:26:21.435 INFO:teuthology.orchestra.run.smithi088.stdout: Verifying : python3-prettytable-0.7.2-27.el9.noarch 19/32 2024-06-03T08:26:21.436 INFO:teuthology.orchestra.run.smithi088.stdout: Verifying : fmt-8.1.1-5.el9.x86_64 20/32 2024-06-03T08:26:21.436 INFO:teuthology.orchestra.run.smithi088.stdout: Verifying : gperftools-libs-2.9.1-2.el9.x86_64 21/32 2024-06-03T08:26:21.436 INFO:teuthology.orchestra.run.smithi088.stdout: Verifying : libarrow-9.0.0-11.el9.x86_64 22/32 2024-06-03T08:26:21.436 INFO:teuthology.orchestra.run.smithi088.stdout: Verifying : libarrow-doc-9.0.0-11.el9.noarch 23/32 2024-06-03T08:26:21.436 INFO:teuthology.orchestra.run.smithi088.stdout: Verifying : liboath-2.6.7-2.el9.x86_64 24/32 2024-06-03T08:26:21.436 INFO:teuthology.orchestra.run.smithi088.stdout: Verifying : libunwind-1.6.2-1.el9.x86_64 25/32 2024-06-03T08:26:21.437 INFO:teuthology.orchestra.run.smithi088.stdout: Verifying : parquet-libs-9.0.0-11.el9.x86_64 26/32 2024-06-03T08:26:21.437 INFO:teuthology.orchestra.run.smithi088.stdout: Verifying : re2-1:20211101-3.el9.x86_64 27/32 2024-06-03T08:26:21.437 INFO:teuthology.orchestra.run.smithi088.stdout: Verifying : thrift-0.15.0-2.el9.x86_64 28/32 2024-06-03T08:26:21.437 INFO:teuthology.orchestra.run.smithi088.stdout: Verifying : librados2-2:18.2.2-1389.gf1abf795.el9.x86_64 29/32 2024-06-03T08:26:21.437 INFO:teuthology.orchestra.run.smithi088.stdout: Verifying : librados2-2:16.2.4-5.el9.x86_64 30/32 2024-06-03T08:26:21.437 INFO:teuthology.orchestra.run.smithi088.stdout: Verifying : librbd1-2:18.2.2-1389.gf1abf795.el9.x86_64 31/32 2024-06-03T08:26:22.198 INFO:teuthology.orchestra.run.smithi088.stdout: Verifying : librbd1-2:16.2.4-5.el9.x86_64 32/32 2024-06-03T08:26:22.198 INFO:teuthology.orchestra.run.smithi088.stdout: 2024-06-03T08:26:22.198 INFO:teuthology.orchestra.run.smithi088.stdout:Upgraded: 2024-06-03T08:26:22.198 INFO:teuthology.orchestra.run.smithi088.stdout: librados2-2:18.2.2-1389.gf1abf795.el9.x86_64 2024-06-03T08:26:22.198 INFO:teuthology.orchestra.run.smithi088.stdout: librbd1-2:18.2.2-1389.gf1abf795.el9.x86_64 2024-06-03T08:26:22.199 INFO:teuthology.orchestra.run.smithi088.stdout:Installed: 2024-06-03T08:26:22.199 INFO:teuthology.orchestra.run.smithi088.stdout: boost-program-options-1.75.0-8.el9.x86_64 2024-06-03T08:26:22.199 INFO:teuthology.orchestra.run.smithi088.stdout: ceph-base-2:18.2.2-1389.gf1abf795.el9.x86_64 2024-06-03T08:26:22.199 INFO:teuthology.orchestra.run.smithi088.stdout: ceph-common-2:18.2.2-1389.gf1abf795.el9.x86_64 2024-06-03T08:26:22.199 INFO:teuthology.orchestra.run.smithi088.stdout: ceph-radosgw-2:18.2.2-1389.gf1abf795.el9.x86_64 2024-06-03T08:26:22.199 INFO:teuthology.orchestra.run.smithi088.stdout: ceph-selinux-2:18.2.2-1389.gf1abf795.el9.x86_64 2024-06-03T08:26:22.199 INFO:teuthology.orchestra.run.smithi088.stdout: fmt-8.1.1-5.el9.x86_64 2024-06-03T08:26:22.199 INFO:teuthology.orchestra.run.smithi088.stdout: gperftools-libs-2.9.1-2.el9.x86_64 2024-06-03T08:26:22.199 INFO:teuthology.orchestra.run.smithi088.stdout: libarrow-9.0.0-11.el9.x86_64 2024-06-03T08:26:22.199 INFO:teuthology.orchestra.run.smithi088.stdout: libarrow-doc-9.0.0-11.el9.noarch 2024-06-03T08:26:22.199 INFO:teuthology.orchestra.run.smithi088.stdout: libcephfs2-2:18.2.2-1389.gf1abf795.el9.x86_64 2024-06-03T08:26:22.199 INFO:teuthology.orchestra.run.smithi088.stdout: liboath-2.6.7-2.el9.x86_64 2024-06-03T08:26:22.199 INFO:teuthology.orchestra.run.smithi088.stdout: libpmemobj-1.12.1-1.el9.x86_64 2024-06-03T08:26:22.200 INFO:teuthology.orchestra.run.smithi088.stdout: librabbitmq-0.11.0-7.el9.x86_64 2024-06-03T08:26:22.200 INFO:teuthology.orchestra.run.smithi088.stdout: libradosstriper1-2:18.2.2-1389.gf1abf795.el9.x86_64 2024-06-03T08:26:22.200 INFO:teuthology.orchestra.run.smithi088.stdout: librdkafka-1.6.1-102.el9.x86_64 2024-06-03T08:26:22.200 INFO:teuthology.orchestra.run.smithi088.stdout: librgw2-2:18.2.2-1389.gf1abf795.el9.x86_64 2024-06-03T08:26:22.200 INFO:teuthology.orchestra.run.smithi088.stdout: libunwind-1.6.2-1.el9.x86_64 2024-06-03T08:26:22.200 INFO:teuthology.orchestra.run.smithi088.stdout: lttng-ust-2.12.0-6.el9.x86_64 2024-06-03T08:26:22.200 INFO:teuthology.orchestra.run.smithi088.stdout: parquet-libs-9.0.0-11.el9.x86_64 2024-06-03T08:26:22.200 INFO:teuthology.orchestra.run.smithi088.stdout: python3-ceph-argparse-2:18.2.2-1389.gf1abf795.el9.x86_64 2024-06-03T08:26:22.200 INFO:teuthology.orchestra.run.smithi088.stdout: python3-ceph-common-2:18.2.2-1389.gf1abf795.el9.x86_64 2024-06-03T08:26:22.200 INFO:teuthology.orchestra.run.smithi088.stdout: python3-cephfs-2:18.2.2-1389.gf1abf795.el9.x86_64 2024-06-03T08:26:22.200 INFO:teuthology.orchestra.run.smithi088.stdout: python3-prettytable-0.7.2-27.el9.noarch 2024-06-03T08:26:22.200 INFO:teuthology.orchestra.run.smithi088.stdout: python3-rados-2:18.2.2-1389.gf1abf795.el9.x86_64 2024-06-03T08:26:22.200 INFO:teuthology.orchestra.run.smithi088.stdout: python3-rbd-2:18.2.2-1389.gf1abf795.el9.x86_64 2024-06-03T08:26:22.201 INFO:teuthology.orchestra.run.smithi088.stdout: python3-rgw-2:18.2.2-1389.gf1abf795.el9.x86_64 2024-06-03T08:26:22.201 INFO:teuthology.orchestra.run.smithi088.stdout: re2-1:20211101-3.el9.x86_64 2024-06-03T08:26:22.201 INFO:teuthology.orchestra.run.smithi088.stdout: thrift-0.15.0-2.el9.x86_64 2024-06-03T08:26:22.201 INFO:teuthology.orchestra.run.smithi088.stdout: 2024-06-03T08:26:22.201 INFO:teuthology.orchestra.run.smithi088.stdout:Complete! 2024-06-03T08:26:22.574 DEBUG:teuthology.orchestra.run.smithi088:> sudo yum -y install ceph-test 2024-06-03T08:26:23.148 INFO:teuthology.orchestra.run.smithi088.stdout:Last metadata expiration check: 0:00:27 ago on Mon 03 Jun 2024 08:25:56 AM UTC. 2024-06-03T08:26:23.268 INFO:teuthology.orchestra.run.smithi088.stdout:Dependencies resolved. 2024-06-03T08:26:23.268 INFO:teuthology.orchestra.run.smithi088.stdout:================================================================================ 2024-06-03T08:26:23.269 INFO:teuthology.orchestra.run.smithi088.stdout: Package Arch Version Repository Size 2024-06-03T08:26:23.269 INFO:teuthology.orchestra.run.smithi088.stdout:================================================================================ 2024-06-03T08:26:23.269 INFO:teuthology.orchestra.run.smithi088.stdout:Installing: 2024-06-03T08:26:23.269 INFO:teuthology.orchestra.run.smithi088.stdout: ceph-test x86_64 2:18.2.2-1389.gf1abf795.el9 ceph 40 M 2024-06-03T08:26:23.269 INFO:teuthology.orchestra.run.smithi088.stdout:Installing dependencies: 2024-06-03T08:26:23.269 INFO:teuthology.orchestra.run.smithi088.stdout: libcephsqlite x86_64 2:18.2.2-1389.gf1abf795.el9 ceph 166 k 2024-06-03T08:26:23.269 INFO:teuthology.orchestra.run.smithi088.stdout: socat x86_64 1.7.4.1-5.el9 appstream 305 k 2024-06-03T08:26:23.269 INFO:teuthology.orchestra.run.smithi088.stdout: xmlstarlet x86_64 1.6.1-20.el9 appstream 64 k 2024-06-03T08:26:23.269 INFO:teuthology.orchestra.run.smithi088.stdout: 2024-06-03T08:26:23.269 INFO:teuthology.orchestra.run.smithi088.stdout:Transaction Summary 2024-06-03T08:26:23.269 INFO:teuthology.orchestra.run.smithi088.stdout:================================================================================ 2024-06-03T08:26:23.269 INFO:teuthology.orchestra.run.smithi088.stdout:Install 4 Packages 2024-06-03T08:26:23.270 INFO:teuthology.orchestra.run.smithi088.stdout: 2024-06-03T08:26:23.270 INFO:teuthology.orchestra.run.smithi088.stdout:Total download size: 40 M 2024-06-03T08:26:23.270 INFO:teuthology.orchestra.run.smithi088.stdout:Installed size: 163 M 2024-06-03T08:26:23.270 INFO:teuthology.orchestra.run.smithi088.stdout:Downloading Packages: 2024-06-03T08:26:23.494 INFO:teuthology.orchestra.run.smithi088.stdout:(1/4): socat-1.7.4.1-5.el9.x86_64.rpm 2.0 MB/s | 305 kB 00:00 2024-06-03T08:26:23.527 INFO:teuthology.orchestra.run.smithi088.stdout:(2/4): xmlstarlet-1.6.1-20.el9.x86_64.rpm 1.9 MB/s | 64 kB 00:00 2024-06-03T08:26:23.561 INFO:teuthology.orchestra.run.smithi088.stdout:(3/4): libcephsqlite-18.2.2-1389.gf1abf795.el9. 758 kB/s | 166 kB 00:00 2024-06-03T08:26:24.482 INFO:teuthology.orchestra.run.smithi192.stdout: Running scriptlet: ceph-selinux-2:18.2.2-1389.gf1abf795.el9.x86_64 29/32 2024-06-03T08:26:24.483 INFO:teuthology.orchestra.run.smithi192.stdout:skipping the directory /sys 2024-06-03T08:26:24.483 INFO:teuthology.orchestra.run.smithi192.stdout:skipping the directory /proc 2024-06-03T08:26:24.483 INFO:teuthology.orchestra.run.smithi192.stdout:skipping the directory /mnt 2024-06-03T08:26:24.483 INFO:teuthology.orchestra.run.smithi192.stdout:skipping the directory /var/tmp 2024-06-03T08:26:24.483 INFO:teuthology.orchestra.run.smithi192.stdout:skipping the directory /home 2024-06-03T08:26:24.483 INFO:teuthology.orchestra.run.smithi192.stdout:skipping the directory /root 2024-06-03T08:26:24.483 INFO:teuthology.orchestra.run.smithi192.stdout:skipping the directory /tmp 2024-06-03T08:26:24.483 INFO:teuthology.orchestra.run.smithi192.stdout: 2024-06-03T08:26:25.252 INFO:teuthology.orchestra.run.smithi192.stdout: Installing : ceph-radosgw-2:18.2.2-1389.gf1abf795.el9.x86_64 30/32 2024-06-03T08:26:25.277 INFO:teuthology.orchestra.run.smithi192.stdout: Running scriptlet: ceph-radosgw-2:18.2.2-1389.gf1abf795.el9.x86_64 30/32 2024-06-03T08:26:25.277 INFO:teuthology.orchestra.run.smithi192.stdout:Glob pattern passed to enable, but globs are not supported for this. 2024-06-03T08:26:25.277 INFO:teuthology.orchestra.run.smithi192.stdout:Invalid unit name "ceph-radosgw@*.service" escaped as "ceph-radosgw@\x2a.service". 2024-06-03T08:26:25.277 INFO:teuthology.orchestra.run.smithi192.stdout:Created symlink /etc/systemd/system/multi-user.target.wants/ceph-radosgw.target → /usr/lib/systemd/system/ceph-radosgw.target. 2024-06-03T08:26:25.277 INFO:teuthology.orchestra.run.smithi192.stdout:Created symlink /etc/systemd/system/ceph.target.wants/ceph-radosgw.target → /usr/lib/systemd/system/ceph-radosgw.target. 2024-06-03T08:26:25.277 INFO:teuthology.orchestra.run.smithi192.stdout: 2024-06-03T08:26:25.279 INFO:teuthology.orchestra.run.smithi192.stdout: Cleanup : librbd1-2:16.2.4-5.el9.x86_64 31/32 2024-06-03T08:26:25.372 INFO:teuthology.orchestra.run.smithi192.stdout: Running scriptlet: librbd1-2:16.2.4-5.el9.x86_64 31/32 2024-06-03T08:26:25.372 INFO:teuthology.orchestra.run.smithi192.stdout: Cleanup : librados2-2:16.2.4-5.el9.x86_64 32/32 2024-06-03T08:26:25.494 INFO:teuthology.orchestra.run.smithi088.stdout:(4/4): ceph-test-18.2.2-1389.gf1abf795.el9.x86_ 18 MB/s | 40 MB 00:02 2024-06-03T08:26:25.496 INFO:teuthology.orchestra.run.smithi088.stdout:-------------------------------------------------------------------------------- 2024-06-03T08:26:25.497 INFO:teuthology.orchestra.run.smithi088.stdout:Total 18 MB/s | 40 MB 00:02 2024-06-03T08:26:25.523 INFO:teuthology.orchestra.run.smithi088.stdout:Running transaction check 2024-06-03T08:26:25.543 INFO:teuthology.orchestra.run.smithi088.stdout:Transaction check succeeded. 2024-06-03T08:26:25.543 INFO:teuthology.orchestra.run.smithi088.stdout:Running transaction test 2024-06-03T08:26:25.861 INFO:teuthology.orchestra.run.smithi088.stdout:Transaction test succeeded. 2024-06-03T08:26:25.861 INFO:teuthology.orchestra.run.smithi088.stdout:Running transaction 2024-06-03T08:26:26.343 INFO:teuthology.orchestra.run.smithi088.stdout: Preparing : 1/1 2024-06-03T08:26:26.440 INFO:teuthology.orchestra.run.smithi088.stdout: Installing : xmlstarlet-1.6.1-20.el9.x86_64 1/4 2024-06-03T08:26:26.470 INFO:teuthology.orchestra.run.smithi192.stdout: Running scriptlet: librados2-2:16.2.4-5.el9.x86_64 32/32 2024-06-03T08:26:26.470 INFO:teuthology.orchestra.run.smithi192.stdout: Verifying : ceph-base-2:18.2.2-1389.gf1abf795.el9.x86_64 1/32 2024-06-03T08:26:26.470 INFO:teuthology.orchestra.run.smithi192.stdout: Verifying : ceph-common-2:18.2.2-1389.gf1abf795.el9.x86_64 2/32 2024-06-03T08:26:26.470 INFO:teuthology.orchestra.run.smithi192.stdout: Verifying : ceph-radosgw-2:18.2.2-1389.gf1abf795.el9.x86_64 3/32 2024-06-03T08:26:26.471 INFO:teuthology.orchestra.run.smithi192.stdout: Verifying : ceph-selinux-2:18.2.2-1389.gf1abf795.el9.x86_64 4/32 2024-06-03T08:26:26.471 INFO:teuthology.orchestra.run.smithi192.stdout: Verifying : libcephfs2-2:18.2.2-1389.gf1abf795.el9.x86_64 5/32 2024-06-03T08:26:26.471 INFO:teuthology.orchestra.run.smithi192.stdout: Verifying : libradosstriper1-2:18.2.2-1389.gf1abf795.el9.x86_6 6/32 2024-06-03T08:26:26.471 INFO:teuthology.orchestra.run.smithi192.stdout: Verifying : librgw2-2:18.2.2-1389.gf1abf795.el9.x86_64 7/32 2024-06-03T08:26:26.471 INFO:teuthology.orchestra.run.smithi192.stdout: Verifying : python3-ceph-argparse-2:18.2.2-1389.gf1abf795.el9. 8/32 2024-06-03T08:26:26.471 INFO:teuthology.orchestra.run.smithi192.stdout: Verifying : python3-ceph-common-2:18.2.2-1389.gf1abf795.el9.x8 9/32 2024-06-03T08:26:26.471 INFO:teuthology.orchestra.run.smithi192.stdout: Verifying : python3-cephfs-2:18.2.2-1389.gf1abf795.el9.x86_64 10/32 2024-06-03T08:26:26.471 INFO:teuthology.orchestra.run.smithi192.stdout: Verifying : python3-rados-2:18.2.2-1389.gf1abf795.el9.x86_64 11/32 2024-06-03T08:26:26.471 INFO:teuthology.orchestra.run.smithi192.stdout: Verifying : python3-rbd-2:18.2.2-1389.gf1abf795.el9.x86_64 12/32 2024-06-03T08:26:26.471 INFO:teuthology.orchestra.run.smithi192.stdout: Verifying : python3-rgw-2:18.2.2-1389.gf1abf795.el9.x86_64 13/32 2024-06-03T08:26:26.471 INFO:teuthology.orchestra.run.smithi192.stdout: Verifying : boost-program-options-1.75.0-8.el9.x86_64 14/32 2024-06-03T08:26:26.471 INFO:teuthology.orchestra.run.smithi192.stdout: Verifying : libpmemobj-1.12.1-1.el9.x86_64 15/32 2024-06-03T08:26:26.472 INFO:teuthology.orchestra.run.smithi192.stdout: Verifying : librabbitmq-0.11.0-7.el9.x86_64 16/32 2024-06-03T08:26:26.472 INFO:teuthology.orchestra.run.smithi192.stdout: Verifying : librdkafka-1.6.1-102.el9.x86_64 17/32 2024-06-03T08:26:26.472 INFO:teuthology.orchestra.run.smithi192.stdout: Verifying : lttng-ust-2.12.0-6.el9.x86_64 18/32 2024-06-03T08:26:26.472 INFO:teuthology.orchestra.run.smithi192.stdout: Verifying : python3-prettytable-0.7.2-27.el9.noarch 19/32 2024-06-03T08:26:26.472 INFO:teuthology.orchestra.run.smithi192.stdout: Verifying : fmt-8.1.1-5.el9.x86_64 20/32 2024-06-03T08:26:26.472 INFO:teuthology.orchestra.run.smithi192.stdout: Verifying : gperftools-libs-2.9.1-2.el9.x86_64 21/32 2024-06-03T08:26:26.472 INFO:teuthology.orchestra.run.smithi192.stdout: Verifying : libarrow-9.0.0-11.el9.x86_64 22/32 2024-06-03T08:26:26.472 INFO:teuthology.orchestra.run.smithi192.stdout: Verifying : libarrow-doc-9.0.0-11.el9.noarch 23/32 2024-06-03T08:26:26.472 INFO:teuthology.orchestra.run.smithi192.stdout: Verifying : liboath-2.6.7-2.el9.x86_64 24/32 2024-06-03T08:26:26.472 INFO:teuthology.orchestra.run.smithi192.stdout: Verifying : libunwind-1.6.2-1.el9.x86_64 25/32 2024-06-03T08:26:26.473 INFO:teuthology.orchestra.run.smithi192.stdout: Verifying : parquet-libs-9.0.0-11.el9.x86_64 26/32 2024-06-03T08:26:26.473 INFO:teuthology.orchestra.run.smithi192.stdout: Verifying : re2-1:20211101-3.el9.x86_64 27/32 2024-06-03T08:26:26.473 INFO:teuthology.orchestra.run.smithi192.stdout: Verifying : thrift-0.15.0-2.el9.x86_64 28/32 2024-06-03T08:26:26.473 INFO:teuthology.orchestra.run.smithi192.stdout: Verifying : librados2-2:18.2.2-1389.gf1abf795.el9.x86_64 29/32 2024-06-03T08:26:26.473 INFO:teuthology.orchestra.run.smithi192.stdout: Verifying : librados2-2:16.2.4-5.el9.x86_64 30/32 2024-06-03T08:26:26.473 INFO:teuthology.orchestra.run.smithi192.stdout: Verifying : librbd1-2:18.2.2-1389.gf1abf795.el9.x86_64 31/32 2024-06-03T08:26:26.486 INFO:teuthology.orchestra.run.smithi088.stdout: Installing : socat-1.7.4.1-5.el9.x86_64 2/4 2024-06-03T08:26:26.513 INFO:teuthology.orchestra.run.smithi088.stdout: Installing : libcephsqlite-2:18.2.2-1389.gf1abf795.el9.x86_64 3/4 2024-06-03T08:26:27.222 INFO:teuthology.orchestra.run.smithi192.stdout: Verifying : librbd1-2:16.2.4-5.el9.x86_64 32/32 2024-06-03T08:26:27.222 INFO:teuthology.orchestra.run.smithi192.stdout: 2024-06-03T08:26:27.222 INFO:teuthology.orchestra.run.smithi192.stdout:Upgraded: 2024-06-03T08:26:27.222 INFO:teuthology.orchestra.run.smithi192.stdout: librados2-2:18.2.2-1389.gf1abf795.el9.x86_64 2024-06-03T08:26:27.222 INFO:teuthology.orchestra.run.smithi192.stdout: librbd1-2:18.2.2-1389.gf1abf795.el9.x86_64 2024-06-03T08:26:27.223 INFO:teuthology.orchestra.run.smithi192.stdout:Installed: 2024-06-03T08:26:27.223 INFO:teuthology.orchestra.run.smithi192.stdout: boost-program-options-1.75.0-8.el9.x86_64 2024-06-03T08:26:27.223 INFO:teuthology.orchestra.run.smithi192.stdout: ceph-base-2:18.2.2-1389.gf1abf795.el9.x86_64 2024-06-03T08:26:27.223 INFO:teuthology.orchestra.run.smithi192.stdout: ceph-common-2:18.2.2-1389.gf1abf795.el9.x86_64 2024-06-03T08:26:27.223 INFO:teuthology.orchestra.run.smithi192.stdout: ceph-radosgw-2:18.2.2-1389.gf1abf795.el9.x86_64 2024-06-03T08:26:27.223 INFO:teuthology.orchestra.run.smithi192.stdout: ceph-selinux-2:18.2.2-1389.gf1abf795.el9.x86_64 2024-06-03T08:26:27.223 INFO:teuthology.orchestra.run.smithi192.stdout: fmt-8.1.1-5.el9.x86_64 2024-06-03T08:26:27.223 INFO:teuthology.orchestra.run.smithi192.stdout: gperftools-libs-2.9.1-2.el9.x86_64 2024-06-03T08:26:27.223 INFO:teuthology.orchestra.run.smithi192.stdout: libarrow-9.0.0-11.el9.x86_64 2024-06-03T08:26:27.223 INFO:teuthology.orchestra.run.smithi192.stdout: libarrow-doc-9.0.0-11.el9.noarch 2024-06-03T08:26:27.223 INFO:teuthology.orchestra.run.smithi192.stdout: libcephfs2-2:18.2.2-1389.gf1abf795.el9.x86_64 2024-06-03T08:26:27.224 INFO:teuthology.orchestra.run.smithi192.stdout: liboath-2.6.7-2.el9.x86_64 2024-06-03T08:26:27.224 INFO:teuthology.orchestra.run.smithi192.stdout: libpmemobj-1.12.1-1.el9.x86_64 2024-06-03T08:26:27.224 INFO:teuthology.orchestra.run.smithi192.stdout: librabbitmq-0.11.0-7.el9.x86_64 2024-06-03T08:26:27.224 INFO:teuthology.orchestra.run.smithi192.stdout: libradosstriper1-2:18.2.2-1389.gf1abf795.el9.x86_64 2024-06-03T08:26:27.224 INFO:teuthology.orchestra.run.smithi192.stdout: librdkafka-1.6.1-102.el9.x86_64 2024-06-03T08:26:27.224 INFO:teuthology.orchestra.run.smithi192.stdout: librgw2-2:18.2.2-1389.gf1abf795.el9.x86_64 2024-06-03T08:26:27.224 INFO:teuthology.orchestra.run.smithi192.stdout: libunwind-1.6.2-1.el9.x86_64 2024-06-03T08:26:27.224 INFO:teuthology.orchestra.run.smithi192.stdout: lttng-ust-2.12.0-6.el9.x86_64 2024-06-03T08:26:27.224 INFO:teuthology.orchestra.run.smithi192.stdout: parquet-libs-9.0.0-11.el9.x86_64 2024-06-03T08:26:27.224 INFO:teuthology.orchestra.run.smithi192.stdout: python3-ceph-argparse-2:18.2.2-1389.gf1abf795.el9.x86_64 2024-06-03T08:26:27.224 INFO:teuthology.orchestra.run.smithi192.stdout: python3-ceph-common-2:18.2.2-1389.gf1abf795.el9.x86_64 2024-06-03T08:26:27.224 INFO:teuthology.orchestra.run.smithi192.stdout: python3-cephfs-2:18.2.2-1389.gf1abf795.el9.x86_64 2024-06-03T08:26:27.225 INFO:teuthology.orchestra.run.smithi192.stdout: python3-prettytable-0.7.2-27.el9.noarch 2024-06-03T08:26:27.225 INFO:teuthology.orchestra.run.smithi192.stdout: python3-rados-2:18.2.2-1389.gf1abf795.el9.x86_64 2024-06-03T08:26:27.225 INFO:teuthology.orchestra.run.smithi192.stdout: python3-rbd-2:18.2.2-1389.gf1abf795.el9.x86_64 2024-06-03T08:26:27.225 INFO:teuthology.orchestra.run.smithi192.stdout: python3-rgw-2:18.2.2-1389.gf1abf795.el9.x86_64 2024-06-03T08:26:27.225 INFO:teuthology.orchestra.run.smithi192.stdout: re2-1:20211101-3.el9.x86_64 2024-06-03T08:26:27.225 INFO:teuthology.orchestra.run.smithi192.stdout: thrift-0.15.0-2.el9.x86_64 2024-06-03T08:26:27.225 INFO:teuthology.orchestra.run.smithi192.stdout: 2024-06-03T08:26:27.225 INFO:teuthology.orchestra.run.smithi192.stdout:Complete! 2024-06-03T08:26:27.551 DEBUG:teuthology.orchestra.run.smithi192:> sudo yum -y install ceph-test 2024-06-03T08:26:28.133 INFO:teuthology.orchestra.run.smithi192.stdout:Last metadata expiration check: 0:00:28 ago on Mon 03 Jun 2024 08:26:00 AM UTC. 2024-06-03T08:26:28.255 INFO:teuthology.orchestra.run.smithi192.stdout:Dependencies resolved. 2024-06-03T08:26:28.256 INFO:teuthology.orchestra.run.smithi192.stdout:================================================================================ 2024-06-03T08:26:28.256 INFO:teuthology.orchestra.run.smithi192.stdout: Package Arch Version Repository Size 2024-06-03T08:26:28.256 INFO:teuthology.orchestra.run.smithi192.stdout:================================================================================ 2024-06-03T08:26:28.256 INFO:teuthology.orchestra.run.smithi192.stdout:Installing: 2024-06-03T08:26:28.256 INFO:teuthology.orchestra.run.smithi192.stdout: ceph-test x86_64 2:18.2.2-1389.gf1abf795.el9 ceph 40 M 2024-06-03T08:26:28.256 INFO:teuthology.orchestra.run.smithi192.stdout:Installing dependencies: 2024-06-03T08:26:28.256 INFO:teuthology.orchestra.run.smithi192.stdout: libcephsqlite x86_64 2:18.2.2-1389.gf1abf795.el9 ceph 166 k 2024-06-03T08:26:28.256 INFO:teuthology.orchestra.run.smithi192.stdout: socat x86_64 1.7.4.1-5.el9 appstream 305 k 2024-06-03T08:26:28.257 INFO:teuthology.orchestra.run.smithi192.stdout: xmlstarlet x86_64 1.6.1-20.el9 appstream 64 k 2024-06-03T08:26:28.257 INFO:teuthology.orchestra.run.smithi192.stdout: 2024-06-03T08:26:28.257 INFO:teuthology.orchestra.run.smithi192.stdout:Transaction Summary 2024-06-03T08:26:28.257 INFO:teuthology.orchestra.run.smithi192.stdout:================================================================================ 2024-06-03T08:26:28.257 INFO:teuthology.orchestra.run.smithi192.stdout:Install 4 Packages 2024-06-03T08:26:28.257 INFO:teuthology.orchestra.run.smithi192.stdout: 2024-06-03T08:26:28.257 INFO:teuthology.orchestra.run.smithi192.stdout:Total download size: 40 M 2024-06-03T08:26:28.257 INFO:teuthology.orchestra.run.smithi192.stdout:Installed size: 163 M 2024-06-03T08:26:28.258 INFO:teuthology.orchestra.run.smithi192.stdout:Downloading Packages: 2024-06-03T08:26:28.769 INFO:teuthology.orchestra.run.smithi192.stdout:(1/4): socat-1.7.4.1-5.el9.x86_64.rpm 1.6 MB/s | 305 kB 00:00 2024-06-03T08:26:28.813 INFO:teuthology.orchestra.run.smithi192.stdout:(2/4): xmlstarlet-1.6.1-20.el9.x86_64.rpm 1.5 MB/s | 64 kB 00:00 2024-06-03T08:26:29.078 INFO:teuthology.orchestra.run.smithi192.stdout:(3/4): libcephsqlite-18.2.2-1389.gf1abf795.el9. 335 kB/s | 166 kB 00:00 2024-06-03T08:26:30.203 INFO:teuthology.orchestra.run.smithi088.stdout: Running scriptlet: libcephsqlite-2:18.2.2-1389.gf1abf795.el9.x86_64 3/4 2024-06-03T08:26:30.286 INFO:teuthology.orchestra.run.smithi088.stdout: Installing : ceph-test-2:18.2.2-1389.gf1abf795.el9.x86_64 4/4 2024-06-03T08:26:30.553 INFO:teuthology.orchestra.run.smithi192.stdout:(4/4): ceph-test-18.2.2-1389.gf1abf795.el9.x86_ 20 MB/s | 40 MB 00:01 2024-06-03T08:26:30.555 INFO:teuthology.orchestra.run.smithi192.stdout:-------------------------------------------------------------------------------- 2024-06-03T08:26:30.556 INFO:teuthology.orchestra.run.smithi192.stdout:Total 17 MB/s | 40 MB 00:02 2024-06-03T08:26:30.583 INFO:teuthology.orchestra.run.smithi192.stdout:Running transaction check 2024-06-03T08:26:30.602 INFO:teuthology.orchestra.run.smithi192.stdout:Transaction check succeeded. 2024-06-03T08:26:30.602 INFO:teuthology.orchestra.run.smithi192.stdout:Running transaction test 2024-06-03T08:26:30.918 INFO:teuthology.orchestra.run.smithi192.stdout:Transaction test succeeded. 2024-06-03T08:26:30.918 INFO:teuthology.orchestra.run.smithi192.stdout:Running transaction 2024-06-03T08:26:31.400 INFO:teuthology.orchestra.run.smithi192.stdout: Preparing : 1/1 2024-06-03T08:26:31.490 INFO:teuthology.orchestra.run.smithi192.stdout: Installing : xmlstarlet-1.6.1-20.el9.x86_64 1/4 2024-06-03T08:26:31.536 INFO:teuthology.orchestra.run.smithi192.stdout: Installing : socat-1.7.4.1-5.el9.x86_64 2/4 2024-06-03T08:26:31.563 INFO:teuthology.orchestra.run.smithi192.stdout: Installing : libcephsqlite-2:18.2.2-1389.gf1abf795.el9.x86_64 3/4 2024-06-03T08:26:32.335 INFO:teuthology.orchestra.run.smithi088.stdout: Running scriptlet: ceph-test-2:18.2.2-1389.gf1abf795.el9.x86_64 4/4 2024-06-03T08:26:32.335 INFO:teuthology.orchestra.run.smithi088.stdout: Verifying : ceph-test-2:18.2.2-1389.gf1abf795.el9.x86_64 1/4 2024-06-03T08:26:32.335 INFO:teuthology.orchestra.run.smithi088.stdout: Verifying : libcephsqlite-2:18.2.2-1389.gf1abf795.el9.x86_64 2/4 2024-06-03T08:26:32.335 INFO:teuthology.orchestra.run.smithi088.stdout: Verifying : socat-1.7.4.1-5.el9.x86_64 3/4 2024-06-03T08:26:32.658 INFO:teuthology.orchestra.run.smithi088.stdout: Verifying : xmlstarlet-1.6.1-20.el9.x86_64 4/4 2024-06-03T08:26:32.659 INFO:teuthology.orchestra.run.smithi088.stdout: 2024-06-03T08:26:32.659 INFO:teuthology.orchestra.run.smithi088.stdout:Installed: 2024-06-03T08:26:32.659 INFO:teuthology.orchestra.run.smithi088.stdout: ceph-test-2:18.2.2-1389.gf1abf795.el9.x86_64 2024-06-03T08:26:32.659 INFO:teuthology.orchestra.run.smithi088.stdout: libcephsqlite-2:18.2.2-1389.gf1abf795.el9.x86_64 2024-06-03T08:26:32.659 INFO:teuthology.orchestra.run.smithi088.stdout: socat-1.7.4.1-5.el9.x86_64 2024-06-03T08:26:32.659 INFO:teuthology.orchestra.run.smithi088.stdout: xmlstarlet-1.6.1-20.el9.x86_64 2024-06-03T08:26:32.659 INFO:teuthology.orchestra.run.smithi088.stdout: 2024-06-03T08:26:32.659 INFO:teuthology.orchestra.run.smithi088.stdout:Complete! 2024-06-03T08:26:32.919 DEBUG:teuthology.orchestra.run.smithi088:> sudo yum -y install ceph 2024-06-03T08:26:33.498 INFO:teuthology.orchestra.run.smithi088.stdout:Last metadata expiration check: 0:00:37 ago on Mon 03 Jun 2024 08:25:56 AM UTC. 2024-06-03T08:26:33.622 INFO:teuthology.orchestra.run.smithi088.stdout:Dependencies resolved. 2024-06-03T08:26:33.624 INFO:teuthology.orchestra.run.smithi088.stdout:================================================================================ 2024-06-03T08:26:33.625 INFO:teuthology.orchestra.run.smithi088.stdout: Package Arch Version Repository Size 2024-06-03T08:26:33.625 INFO:teuthology.orchestra.run.smithi088.stdout:================================================================================ 2024-06-03T08:26:33.625 INFO:teuthology.orchestra.run.smithi088.stdout:Installing: 2024-06-03T08:26:33.625 INFO:teuthology.orchestra.run.smithi088.stdout: ceph x86_64 2:18.2.2-1389.gf1abf795.el9 ceph 6.5 k 2024-06-03T08:26:33.625 INFO:teuthology.orchestra.run.smithi088.stdout:Installing dependencies: 2024-06-03T08:26:33.625 INFO:teuthology.orchestra.run.smithi088.stdout: ceph-mds x86_64 2:18.2.2-1389.gf1abf795.el9 ceph 2.1 M 2024-06-03T08:26:33.625 INFO:teuthology.orchestra.run.smithi088.stdout: ceph-mgr x86_64 2:18.2.2-1389.gf1abf795.el9 ceph 1.5 M 2024-06-03T08:26:33.625 INFO:teuthology.orchestra.run.smithi088.stdout: ceph-mgr-modules-core noarch 2:18.2.2-1389.gf1abf795.el9 ceph-noarch 246 k 2024-06-03T08:26:33.625 INFO:teuthology.orchestra.run.smithi088.stdout: ceph-mon x86_64 2:18.2.2-1389.gf1abf795.el9 ceph 4.7 M 2024-06-03T08:26:33.625 INFO:teuthology.orchestra.run.smithi088.stdout: ceph-osd x86_64 2:18.2.2-1389.gf1abf795.el9 ceph 17 M 2024-06-03T08:26:33.626 INFO:teuthology.orchestra.run.smithi088.stdout: python3-bcrypt x86_64 3.2.2-1.el9 epel 43 k 2024-06-03T08:26:33.626 INFO:teuthology.orchestra.run.smithi088.stdout: python3-cffi x86_64 1.14.5-5.el9 baseos 253 k 2024-06-03T08:26:33.626 INFO:teuthology.orchestra.run.smithi088.stdout: python3-cheroot noarch 8.6.0-4.el9 epel 172 k 2024-06-03T08:26:33.626 INFO:teuthology.orchestra.run.smithi088.stdout: python3-cherrypy noarch 18.6.1-2.el9 epel 358 k 2024-06-03T08:26:33.626 INFO:teuthology.orchestra.run.smithi088.stdout: python3-cryptography x86_64 36.0.1-4.el9 baseos 1.2 M 2024-06-03T08:26:33.626 INFO:teuthology.orchestra.run.smithi088.stdout: python3-jaraco noarch 8.2.1-3.el9 epel 11 k 2024-06-03T08:26:33.626 INFO:teuthology.orchestra.run.smithi088.stdout: python3-jaraco-classes noarch 3.2.1-5.el9 epel 18 k 2024-06-03T08:26:33.626 INFO:teuthology.orchestra.run.smithi088.stdout: python3-jaraco-collections 2024-06-03T08:26:33.626 INFO:teuthology.orchestra.run.smithi088.stdout: noarch 3.0.0-8.el9 epel 23 k 2024-06-03T08:26:33.626 INFO:teuthology.orchestra.run.smithi088.stdout: python3-jaraco-functools noarch 3.5.0-2.el9 epel 19 k 2024-06-03T08:26:33.626 INFO:teuthology.orchestra.run.smithi088.stdout: python3-jaraco-text noarch 3.2.0-6.el9 epel 20 k 2024-06-03T08:26:33.626 INFO:teuthology.orchestra.run.smithi088.stdout: python3-logutils noarch 0.3.5-21.el9 epel 46 k 2024-06-03T08:26:33.627 INFO:teuthology.orchestra.run.smithi088.stdout: python3-mako noarch 1.1.4-6.el9 appstream 172 k 2024-06-03T08:26:33.627 INFO:teuthology.orchestra.run.smithi088.stdout: python3-markupsafe x86_64 1.1.1-12.el9 appstream 35 k 2024-06-03T08:26:33.627 INFO:teuthology.orchestra.run.smithi088.stdout: python3-more-itertools noarch 8.12.0-2.el9 epel 79 k 2024-06-03T08:26:33.627 INFO:teuthology.orchestra.run.smithi088.stdout: python3-pecan noarch 1.4.2-3.el9 epel 272 k 2024-06-03T08:26:33.627 INFO:teuthology.orchestra.run.smithi088.stdout: python3-ply noarch 3.11-14.el9 baseos 106 k 2024-06-03T08:26:33.627 INFO:teuthology.orchestra.run.smithi088.stdout: python3-portend noarch 3.1.0-2.el9 epel 16 k 2024-06-03T08:26:33.627 INFO:teuthology.orchestra.run.smithi088.stdout: python3-pyOpenSSL noarch 21.0.0-1.el9 epel 90 k 2024-06-03T08:26:33.627 INFO:teuthology.orchestra.run.smithi088.stdout: python3-pycparser noarch 2.20-6.el9 baseos 135 k 2024-06-03T08:26:33.627 INFO:teuthology.orchestra.run.smithi088.stdout: python3-pytz noarch 2021.1-5.el9 appstream 51 k 2024-06-03T08:26:33.627 INFO:teuthology.orchestra.run.smithi088.stdout: python3-requests noarch 2.25.1-8.el9 baseos 125 k 2024-06-03T08:26:33.627 INFO:teuthology.orchestra.run.smithi088.stdout: python3-tempora noarch 5.0.0-2.el9 epel 36 k 2024-06-03T08:26:33.627 INFO:teuthology.orchestra.run.smithi088.stdout: python3-toml noarch 0.10.2-6.el9 appstream 42 k 2024-06-03T08:26:33.627 INFO:teuthology.orchestra.run.smithi088.stdout: python3-urllib3 noarch 1.26.5-5.el9 baseos 215 k 2024-06-03T08:26:33.628 INFO:teuthology.orchestra.run.smithi088.stdout: python3-webob noarch 1.8.7-6.el9 epel 230 k 2024-06-03T08:26:33.628 INFO:teuthology.orchestra.run.smithi088.stdout: python3-werkzeug noarch 2.0.3-3.el9.1 epel 427 k 2024-06-03T08:26:33.628 INFO:teuthology.orchestra.run.smithi088.stdout: python3-zc-lockfile noarch 2.0-10.el9 epel 20 k 2024-06-03T08:26:33.628 INFO:teuthology.orchestra.run.smithi088.stdout: 2024-06-03T08:26:33.628 INFO:teuthology.orchestra.run.smithi088.stdout:Transaction Summary 2024-06-03T08:26:33.628 INFO:teuthology.orchestra.run.smithi088.stdout:================================================================================ 2024-06-03T08:26:33.628 INFO:teuthology.orchestra.run.smithi088.stdout:Install 33 Packages 2024-06-03T08:26:33.628 INFO:teuthology.orchestra.run.smithi088.stdout: 2024-06-03T08:26:33.629 INFO:teuthology.orchestra.run.smithi088.stdout:Total download size: 29 M 2024-06-03T08:26:33.629 INFO:teuthology.orchestra.run.smithi088.stdout:Installed size: 104 M 2024-06-03T08:26:33.629 INFO:teuthology.orchestra.run.smithi088.stdout:Downloading Packages: 2024-06-03T08:26:34.139 INFO:teuthology.orchestra.run.smithi088.stdout:(1/33): ceph-18.2.2-1389.gf1abf795.el9.x86_64.r 38 kB/s | 6.5 kB 00:00 2024-06-03T08:26:34.531 INFO:teuthology.orchestra.run.smithi088.stdout:(2/33): ceph-mon-18.2.2-1389.gf1abf795.el9.x86_ 12 MB/s | 4.7 MB 00:00 2024-06-03T08:26:34.689 INFO:teuthology.orchestra.run.smithi088.stdout:(3/33): ceph-mgr-18.2.2-1389.gf1abf795.el9.x86_ 2.0 MB/s | 1.5 MB 00:00 2024-06-03T08:26:34.798 INFO:teuthology.orchestra.run.smithi088.stdout:(4/33): ceph-mgr-modules-core-18.2.2-1389.gf1ab 2.2 MB/s | 246 kB 00:00 2024-06-03T08:26:34.848 INFO:teuthology.orchestra.run.smithi088.stdout:(5/33): ceph-mds-18.2.2-1389.gf1abf795.el9.x86_ 2.4 MB/s | 2.1 MB 00:00 2024-06-03T08:26:35.006 INFO:teuthology.orchestra.run.smithi088.stdout:(6/33): python3-cffi-1.14.5-5.el9.x86_64.rpm 1.2 MB/s | 253 kB 00:00 2024-06-03T08:26:35.057 INFO:teuthology.orchestra.run.smithi088.stdout:(7/33): python3-ply-3.11-14.el9.noarch.rpm 2.1 MB/s | 106 kB 00:00 2024-06-03T08:26:35.107 INFO:teuthology.orchestra.run.smithi088.stdout:(8/33): python3-cryptography-36.0.1-4.el9.x86_6 4.7 MB/s | 1.2 MB 00:00 2024-06-03T08:26:35.284 INFO:teuthology.orchestra.run.smithi192.stdout: Running scriptlet: libcephsqlite-2:18.2.2-1389.gf1abf795.el9.x86_64 3/4 2024-06-03T08:26:35.316 INFO:teuthology.orchestra.run.smithi088.stdout:(9/33): ceph-osd-18.2.2-1389.gf1abf795.el9.x86_ 21 MB/s | 17 MB 00:00 2024-06-03T08:26:35.341 INFO:teuthology.orchestra.run.smithi088.stdout:(10/33): python3-requests-2.25.1-8.el9.noarch.r 535 kB/s | 125 kB 00:00 2024-06-03T08:26:35.366 INFO:teuthology.orchestra.run.smithi088.stdout:(11/33): python3-pycparser-2.20-6.el9.noarch.rp 437 kB/s | 135 kB 00:00 2024-06-03T08:26:35.371 INFO:teuthology.orchestra.run.smithi192.stdout: Installing : ceph-test-2:18.2.2-1389.gf1abf795.el9.x86_64 4/4 2024-06-03T08:26:35.425 INFO:teuthology.orchestra.run.smithi088.stdout:(12/33): python3-markupsafe-1.1.1-12.el9.x86_64 595 kB/s | 35 kB 00:00 2024-06-03T08:26:35.458 INFO:teuthology.orchestra.run.smithi088.stdout:(13/33): python3-mako-1.1.4-6.el9.noarch.rpm 1.4 MB/s | 172 kB 00:00 2024-06-03T08:26:35.475 INFO:teuthology.orchestra.run.smithi088.stdout:(14/33): python3-pytz-2021.1-5.el9.noarch.rpm 1.0 MB/s | 51 kB 00:00 2024-06-03T08:26:35.500 INFO:teuthology.orchestra.run.smithi088.stdout:(15/33): python3-urllib3-1.26.5-5.el9.noarch.rp 1.1 MB/s | 215 kB 00:00 2024-06-03T08:26:35.517 INFO:teuthology.orchestra.run.smithi088.stdout:(16/33): python3-toml-0.10.2-6.el9.noarch.rpm 707 kB/s | 42 kB 00:00 2024-06-03T08:26:35.826 INFO:teuthology.orchestra.run.smithi088.stdout:(17/33): python3-bcrypt-3.2.2-1.el9.x86_64.rpm 124 kB/s | 43 kB 00:00 2024-06-03T08:26:35.935 INFO:teuthology.orchestra.run.smithi088.stdout:(18/33): python3-jaraco-8.2.1-3.el9.noarch.rpm 99 kB/s | 11 kB 00:00 2024-06-03T08:26:35.977 INFO:teuthology.orchestra.run.smithi088.stdout:(19/33): python3-cheroot-8.6.0-4.el9.noarch.rpm 361 kB/s | 172 kB 00:00 2024-06-03T08:26:36.044 INFO:teuthology.orchestra.run.smithi088.stdout:(20/33): python3-jaraco-classes-3.2.1-5.el9.noa 163 kB/s | 18 kB 00:00 2024-06-03T08:26:36.069 INFO:teuthology.orchestra.run.smithi088.stdout:(21/33): python3-jaraco-collections-3.0.0-8.el9 252 kB/s | 23 kB 00:00 2024-06-03T08:26:36.094 INFO:teuthology.orchestra.run.smithi088.stdout:(22/33): python3-cherrypy-18.6.1-2.el9.noarch.r 622 kB/s | 358 kB 00:00 2024-06-03T08:26:36.153 INFO:teuthology.orchestra.run.smithi088.stdout:(23/33): python3-jaraco-functools-3.5.0-2.el9.n 178 kB/s | 19 kB 00:00 2024-06-03T08:26:36.170 INFO:teuthology.orchestra.run.smithi088.stdout:(24/33): python3-jaraco-text-3.2.0-6.el9.noarch 195 kB/s | 20 kB 00:00 2024-06-03T08:26:36.187 INFO:teuthology.orchestra.run.smithi088.stdout:(25/33): python3-logutils-0.3.5-21.el9.noarch.r 502 kB/s | 46 kB 00:00 2024-06-03T08:26:36.278 INFO:teuthology.orchestra.run.smithi088.stdout:(26/33): python3-portend-3.1.0-2.el9.noarch.rpm 179 kB/s | 16 kB 00:00 2024-06-03T08:26:36.321 INFO:teuthology.orchestra.run.smithi088.stdout:(27/33): python3-more-itertools-8.12.0-2.el9.no 470 kB/s | 79 kB 00:00 2024-06-03T08:26:36.363 INFO:teuthology.orchestra.run.smithi088.stdout:(28/33): python3-pecan-1.4.2-3.el9.noarch.rpm 1.4 MB/s | 272 kB 00:00 2024-06-03T08:26:36.388 INFO:teuthology.orchestra.run.smithi088.stdout:(29/33): python3-pyOpenSSL-21.0.0-1.el9.noarch. 824 kB/s | 90 kB 00:00 2024-06-03T08:26:36.430 INFO:teuthology.orchestra.run.smithi088.stdout:(30/33): python3-tempora-5.0.0-2.el9.noarch.rpm 328 kB/s | 36 kB 00:00 2024-06-03T08:26:36.480 INFO:teuthology.orchestra.run.smithi088.stdout:(31/33): python3-webob-1.8.7-6.el9.noarch.rpm 1.9 MB/s | 230 kB 00:00 2024-06-03T08:26:36.555 INFO:teuthology.orchestra.run.smithi088.stdout:(32/33): python3-zc-lockfile-2.0-10.el9.noarch. 159 kB/s | 20 kB 00:00 2024-06-03T08:26:36.580 INFO:teuthology.orchestra.run.smithi088.stdout:(33/33): python3-werkzeug-2.0.3-3.el9.1.noarch. 2.2 MB/s | 427 kB 00:00 2024-06-03T08:26:36.582 INFO:teuthology.orchestra.run.smithi088.stdout:-------------------------------------------------------------------------------- 2024-06-03T08:26:36.582 INFO:teuthology.orchestra.run.smithi088.stdout:Total 9.9 MB/s | 29 MB 00:02 2024-06-03T08:26:36.703 INFO:teuthology.orchestra.run.smithi088.stdout:Running transaction check 2024-06-03T08:26:36.739 INFO:teuthology.orchestra.run.smithi088.stdout:Transaction check succeeded. 2024-06-03T08:26:36.739 INFO:teuthology.orchestra.run.smithi088.stdout:Running transaction test 2024-06-03T08:26:37.119 INFO:teuthology.orchestra.run.smithi088.stdout:Transaction test succeeded. 2024-06-03T08:26:37.119 INFO:teuthology.orchestra.run.smithi088.stdout:Running transaction 2024-06-03T08:26:37.425 INFO:teuthology.orchestra.run.smithi192.stdout: Running scriptlet: ceph-test-2:18.2.2-1389.gf1abf795.el9.x86_64 4/4 2024-06-03T08:26:37.425 INFO:teuthology.orchestra.run.smithi192.stdout: Verifying : ceph-test-2:18.2.2-1389.gf1abf795.el9.x86_64 1/4 2024-06-03T08:26:37.425 INFO:teuthology.orchestra.run.smithi192.stdout: Verifying : libcephsqlite-2:18.2.2-1389.gf1abf795.el9.x86_64 2/4 2024-06-03T08:26:37.425 INFO:teuthology.orchestra.run.smithi192.stdout: Verifying : socat-1.7.4.1-5.el9.x86_64 3/4 2024-06-03T08:26:37.695 INFO:teuthology.orchestra.run.smithi192.stdout: Verifying : xmlstarlet-1.6.1-20.el9.x86_64 4/4 2024-06-03T08:26:37.695 INFO:teuthology.orchestra.run.smithi192.stdout: 2024-06-03T08:26:37.695 INFO:teuthology.orchestra.run.smithi192.stdout:Installed: 2024-06-03T08:26:37.696 INFO:teuthology.orchestra.run.smithi192.stdout: ceph-test-2:18.2.2-1389.gf1abf795.el9.x86_64 2024-06-03T08:26:37.696 INFO:teuthology.orchestra.run.smithi192.stdout: libcephsqlite-2:18.2.2-1389.gf1abf795.el9.x86_64 2024-06-03T08:26:37.696 INFO:teuthology.orchestra.run.smithi192.stdout: socat-1.7.4.1-5.el9.x86_64 2024-06-03T08:26:37.696 INFO:teuthology.orchestra.run.smithi192.stdout: xmlstarlet-1.6.1-20.el9.x86_64 2024-06-03T08:26:37.696 INFO:teuthology.orchestra.run.smithi192.stdout: 2024-06-03T08:26:37.696 INFO:teuthology.orchestra.run.smithi192.stdout:Complete! 2024-06-03T08:26:37.979 DEBUG:teuthology.orchestra.run.smithi192:> sudo yum -y install ceph 2024-06-03T08:26:38.264 INFO:teuthology.orchestra.run.smithi088.stdout: Preparing : 1/1 2024-06-03T08:26:38.334 INFO:teuthology.orchestra.run.smithi088.stdout: Installing : python3-more-itertools-8.12.0-2.el9.noarch 1/33 2024-06-03T08:26:38.366 INFO:teuthology.orchestra.run.smithi088.stdout: Installing : python3-jaraco-8.2.1-3.el9.noarch 2/33 2024-06-03T08:26:38.398 INFO:teuthology.orchestra.run.smithi088.stdout: Installing : python3-jaraco-classes-3.2.1-5.el9.noarch 3/33 2024-06-03T08:26:38.493 INFO:teuthology.orchestra.run.smithi088.stdout: Installing : python3-zc-lockfile-2.0-10.el9.noarch 4/33 2024-06-03T08:26:38.563 INFO:teuthology.orchestra.run.smithi192.stdout:Last metadata expiration check: 0:00:38 ago on Mon 03 Jun 2024 08:26:00 AM UTC. 2024-06-03T08:26:38.578 INFO:teuthology.orchestra.run.smithi088.stdout: Installing : python3-werkzeug-2.0.3-3.el9.1.noarch 5/33 2024-06-03T08:26:38.627 INFO:teuthology.orchestra.run.smithi088.stdout: Installing : python3-webob-1.8.7-6.el9.noarch 6/33 2024-06-03T08:26:38.662 INFO:teuthology.orchestra.run.smithi088.stdout: Installing : python3-logutils-0.3.5-21.el9.noarch 7/33 2024-06-03T08:26:38.688 INFO:teuthology.orchestra.run.smithi192.stdout:Dependencies resolved. 2024-06-03T08:26:38.690 INFO:teuthology.orchestra.run.smithi192.stdout:================================================================================ 2024-06-03T08:26:38.690 INFO:teuthology.orchestra.run.smithi192.stdout: Package Arch Version Repository Size 2024-06-03T08:26:38.690 INFO:teuthology.orchestra.run.smithi192.stdout:================================================================================ 2024-06-03T08:26:38.691 INFO:teuthology.orchestra.run.smithi192.stdout:Installing: 2024-06-03T08:26:38.691 INFO:teuthology.orchestra.run.smithi192.stdout: ceph x86_64 2:18.2.2-1389.gf1abf795.el9 ceph 6.5 k 2024-06-03T08:26:38.691 INFO:teuthology.orchestra.run.smithi192.stdout:Installing dependencies: 2024-06-03T08:26:38.691 INFO:teuthology.orchestra.run.smithi192.stdout: ceph-mds x86_64 2:18.2.2-1389.gf1abf795.el9 ceph 2.1 M 2024-06-03T08:26:38.691 INFO:teuthology.orchestra.run.smithi192.stdout: ceph-mgr x86_64 2:18.2.2-1389.gf1abf795.el9 ceph 1.5 M 2024-06-03T08:26:38.691 INFO:teuthology.orchestra.run.smithi192.stdout: ceph-mgr-modules-core noarch 2:18.2.2-1389.gf1abf795.el9 ceph-noarch 246 k 2024-06-03T08:26:38.691 INFO:teuthology.orchestra.run.smithi192.stdout: ceph-mon x86_64 2:18.2.2-1389.gf1abf795.el9 ceph 4.7 M 2024-06-03T08:26:38.691 INFO:teuthology.orchestra.run.smithi192.stdout: ceph-osd x86_64 2:18.2.2-1389.gf1abf795.el9 ceph 17 M 2024-06-03T08:26:38.691 INFO:teuthology.orchestra.run.smithi192.stdout: python3-bcrypt x86_64 3.2.2-1.el9 epel 43 k 2024-06-03T08:26:38.691 INFO:teuthology.orchestra.run.smithi192.stdout: python3-cffi x86_64 1.14.5-5.el9 baseos 253 k 2024-06-03T08:26:38.691 INFO:teuthology.orchestra.run.smithi192.stdout: python3-cheroot noarch 8.6.0-4.el9 epel 172 k 2024-06-03T08:26:38.691 INFO:teuthology.orchestra.run.smithi192.stdout: python3-cherrypy noarch 18.6.1-2.el9 epel 358 k 2024-06-03T08:26:38.691 INFO:teuthology.orchestra.run.smithi192.stdout: python3-cryptography x86_64 36.0.1-4.el9 baseos 1.2 M 2024-06-03T08:26:38.692 INFO:teuthology.orchestra.run.smithi192.stdout: python3-jaraco noarch 8.2.1-3.el9 epel 11 k 2024-06-03T08:26:38.692 INFO:teuthology.orchestra.run.smithi192.stdout: python3-jaraco-classes noarch 3.2.1-5.el9 epel 18 k 2024-06-03T08:26:38.692 INFO:teuthology.orchestra.run.smithi192.stdout: python3-jaraco-collections 2024-06-03T08:26:38.692 INFO:teuthology.orchestra.run.smithi192.stdout: noarch 3.0.0-8.el9 epel 23 k 2024-06-03T08:26:38.692 INFO:teuthology.orchestra.run.smithi192.stdout: python3-jaraco-functools noarch 3.5.0-2.el9 epel 19 k 2024-06-03T08:26:38.692 INFO:teuthology.orchestra.run.smithi192.stdout: python3-jaraco-text noarch 3.2.0-6.el9 epel 20 k 2024-06-03T08:26:38.692 INFO:teuthology.orchestra.run.smithi192.stdout: python3-logutils noarch 0.3.5-21.el9 epel 46 k 2024-06-03T08:26:38.692 INFO:teuthology.orchestra.run.smithi192.stdout: python3-mako noarch 1.1.4-6.el9 appstream 172 k 2024-06-03T08:26:38.692 INFO:teuthology.orchestra.run.smithi192.stdout: python3-markupsafe x86_64 1.1.1-12.el9 appstream 35 k 2024-06-03T08:26:38.692 INFO:teuthology.orchestra.run.smithi192.stdout: python3-more-itertools noarch 8.12.0-2.el9 epel 79 k 2024-06-03T08:26:38.692 INFO:teuthology.orchestra.run.smithi192.stdout: python3-pecan noarch 1.4.2-3.el9 epel 272 k 2024-06-03T08:26:38.692 INFO:teuthology.orchestra.run.smithi192.stdout: python3-ply noarch 3.11-14.el9 baseos 106 k 2024-06-03T08:26:38.692 INFO:teuthology.orchestra.run.smithi192.stdout: python3-portend noarch 3.1.0-2.el9 epel 16 k 2024-06-03T08:26:38.692 INFO:teuthology.orchestra.run.smithi192.stdout: python3-pyOpenSSL noarch 21.0.0-1.el9 epel 90 k 2024-06-03T08:26:38.693 INFO:teuthology.orchestra.run.smithi192.stdout: python3-pycparser noarch 2.20-6.el9 baseos 135 k 2024-06-03T08:26:38.693 INFO:teuthology.orchestra.run.smithi192.stdout: python3-pytz noarch 2021.1-5.el9 appstream 51 k 2024-06-03T08:26:38.693 INFO:teuthology.orchestra.run.smithi192.stdout: python3-requests noarch 2.25.1-8.el9 baseos 125 k 2024-06-03T08:26:38.693 INFO:teuthology.orchestra.run.smithi192.stdout: python3-tempora noarch 5.0.0-2.el9 epel 36 k 2024-06-03T08:26:38.693 INFO:teuthology.orchestra.run.smithi192.stdout: python3-toml noarch 0.10.2-6.el9 appstream 42 k 2024-06-03T08:26:38.693 INFO:teuthology.orchestra.run.smithi192.stdout: python3-urllib3 noarch 1.26.5-5.el9 baseos 215 k 2024-06-03T08:26:38.693 INFO:teuthology.orchestra.run.smithi192.stdout: python3-webob noarch 1.8.7-6.el9 epel 230 k 2024-06-03T08:26:38.693 INFO:teuthology.orchestra.run.smithi192.stdout: python3-werkzeug noarch 2.0.3-3.el9.1 epel 427 k 2024-06-03T08:26:38.693 INFO:teuthology.orchestra.run.smithi192.stdout: python3-zc-lockfile noarch 2.0-10.el9 epel 20 k 2024-06-03T08:26:38.693 INFO:teuthology.orchestra.run.smithi192.stdout: 2024-06-03T08:26:38.693 INFO:teuthology.orchestra.run.smithi192.stdout:Transaction Summary 2024-06-03T08:26:38.693 INFO:teuthology.orchestra.run.smithi192.stdout:================================================================================ 2024-06-03T08:26:38.693 INFO:teuthology.orchestra.run.smithi192.stdout:Install 33 Packages 2024-06-03T08:26:38.693 INFO:teuthology.orchestra.run.smithi192.stdout: 2024-06-03T08:26:38.694 INFO:teuthology.orchestra.run.smithi088.stdout: Installing : python3-toml-0.10.2-6.el9.noarch 8/33 2024-06-03T08:26:38.694 INFO:teuthology.orchestra.run.smithi192.stdout:Total download size: 29 M 2024-06-03T08:26:38.694 INFO:teuthology.orchestra.run.smithi192.stdout:Installed size: 104 M 2024-06-03T08:26:38.695 INFO:teuthology.orchestra.run.smithi192.stdout:Downloading Packages: 2024-06-03T08:26:38.726 INFO:teuthology.orchestra.run.smithi088.stdout: Installing : python3-jaraco-functools-3.5.0-2.el9.noarch 9/33 2024-06-03T08:26:38.760 INFO:teuthology.orchestra.run.smithi088.stdout: Installing : python3-jaraco-text-3.2.0-6.el9.noarch 10/33 2024-06-03T08:26:38.800 INFO:teuthology.orchestra.run.smithi088.stdout: Installing : python3-jaraco-collections-3.0.0-8.el9.noarch 11/33 2024-06-03T08:26:38.854 INFO:teuthology.orchestra.run.smithi088.stdout: Installing : python3-pytz-2021.1-5.el9.noarch 12/33 2024-06-03T08:26:38.887 INFO:teuthology.orchestra.run.smithi088.stdout: Installing : python3-tempora-5.0.0-2.el9.noarch 13/33 2024-06-03T08:26:38.928 INFO:teuthology.orchestra.run.smithi088.stdout: Installing : python3-portend-3.1.0-2.el9.noarch 14/33 2024-06-03T08:26:39.007 INFO:teuthology.orchestra.run.smithi088.stdout: Installing : python3-markupsafe-1.1.1-12.el9.x86_64 15/33 2024-06-03T08:26:39.146 INFO:teuthology.orchestra.run.smithi088.stdout: Installing : python3-mako-1.1.4-6.el9.noarch 16/33 2024-06-03T08:26:39.225 INFO:teuthology.orchestra.run.smithi088.stdout: Installing : python3-pecan-1.4.2-3.el9.noarch 17/33 2024-06-03T08:26:39.278 INFO:teuthology.orchestra.run.smithi088.stdout: Installing : python3-urllib3-1.26.5-5.el9.noarch 18/33 2024-06-03T08:26:39.316 INFO:teuthology.orchestra.run.smithi088.stdout: Installing : python3-requests-2.25.1-8.el9.noarch 19/33 2024-06-03T08:26:39.323 INFO:teuthology.orchestra.run.smithi192.stdout:(1/33): ceph-18.2.2-1389.gf1abf795.el9.x86_64.r 33 kB/s | 6.5 kB 00:00 2024-06-03T08:26:39.383 INFO:teuthology.orchestra.run.smithi088.stdout: Installing : python3-ply-3.11-14.el9.noarch 20/33 2024-06-03T08:26:39.438 INFO:teuthology.orchestra.run.smithi088.stdout: Installing : python3-pycparser-2.20-6.el9.noarch 21/33 2024-06-03T08:26:39.506 INFO:teuthology.orchestra.run.smithi192.stdout:(2/33): ceph-mds-18.2.2-1389.gf1abf795.el9.x86_ 5.5 MB/s | 2.1 MB 00:00 2024-06-03T08:26:39.548 INFO:teuthology.orchestra.run.smithi192.stdout:(3/33): ceph-mgr-18.2.2-1389.gf1abf795.el9.x86_ 3.5 MB/s | 1.5 MB 00:00 2024-06-03T08:26:39.623 INFO:teuthology.orchestra.run.smithi088.stdout: Installing : python3-cffi-1.14.5-5.el9.x86_64 22/33 2024-06-03T08:26:39.656 INFO:teuthology.orchestra.run.smithi192.stdout:(4/33): ceph-mgr-modules-core-18.2.2-1389.gf1ab 2.2 MB/s | 246 kB 00:00 2024-06-03T08:26:39.704 INFO:teuthology.orchestra.run.smithi088.stdout: Installing : python3-cryptography-36.0.1-4.el9.x86_64 23/33 2024-06-03T08:26:39.774 INFO:teuthology.orchestra.run.smithi088.stdout: Installing : python3-pyOpenSSL-21.0.0-1.el9.noarch 24/33 2024-06-03T08:26:39.860 INFO:teuthology.orchestra.run.smithi088.stdout: Installing : python3-cheroot-8.6.0-4.el9.noarch 25/33 2024-06-03T08:26:39.894 INFO:teuthology.orchestra.run.smithi088.stdout: Installing : python3-cherrypy-18.6.1-2.el9.noarch 26/33 2024-06-03T08:26:40.010 INFO:teuthology.orchestra.run.smithi088.stdout: Installing : python3-bcrypt-3.2.2-1.el9.x86_64 27/33 2024-06-03T08:26:40.161 INFO:teuthology.orchestra.run.smithi088.stdout: Installing : ceph-mgr-modules-core-2:18.2.2-1389.gf1abf795.el9. 28/33 2024-06-03T08:26:40.189 INFO:teuthology.orchestra.run.smithi088.stdout: Installing : ceph-mgr-2:18.2.2-1389.gf1abf795.el9.x86_64 29/33 2024-06-03T08:26:40.216 INFO:teuthology.orchestra.run.smithi088.stdout: Running scriptlet: ceph-mgr-2:18.2.2-1389.gf1abf795.el9.x86_64 29/33 2024-06-03T08:26:40.216 INFO:teuthology.orchestra.run.smithi088.stdout:Glob pattern passed to enable, but globs are not supported for this. 2024-06-03T08:26:40.216 INFO:teuthology.orchestra.run.smithi088.stdout:Invalid unit name "ceph-mgr@*.service" escaped as "ceph-mgr@\x2a.service". 2024-06-03T08:26:40.216 INFO:teuthology.orchestra.run.smithi088.stdout:Created symlink /etc/systemd/system/multi-user.target.wants/ceph-mgr.target → /usr/lib/systemd/system/ceph-mgr.target. 2024-06-03T08:26:40.216 INFO:teuthology.orchestra.run.smithi088.stdout:Created symlink /etc/systemd/system/ceph.target.wants/ceph-mgr.target → /usr/lib/systemd/system/ceph-mgr.target. 2024-06-03T08:26:40.216 INFO:teuthology.orchestra.run.smithi088.stdout: 2024-06-03T08:26:40.307 INFO:teuthology.orchestra.run.smithi192.stdout:(5/33): ceph-osd-18.2.2-1389.gf1abf795.el9.x86_ 21 MB/s | 17 MB 00:00 2024-06-03T08:26:40.332 INFO:teuthology.orchestra.run.smithi192.stdout:(6/33): python3-cffi-1.14.5-5.el9.x86_64.rpm 375 kB/s | 253 kB 00:00 2024-06-03T08:26:40.607 INFO:teuthology.orchestra.run.smithi192.stdout:(7/33): ceph-mon-18.2.2-1389.gf1abf795.el9.x86_ 3.6 MB/s | 4.7 MB 00:01 2024-06-03T08:26:40.941 INFO:teuthology.orchestra.run.smithi192.stdout:(8/33): python3-ply-3.11-14.el9.noarch.rpm 175 kB/s | 106 kB 00:00 2024-06-03T08:26:41.116 INFO:teuthology.orchestra.run.smithi192.stdout:(9/33): python3-pycparser-2.20-6.el9.noarch.rpm 266 kB/s | 135 kB 00:00 2024-06-03T08:26:41.367 INFO:teuthology.orchestra.run.smithi192.stdout:(10/33): python3-requests-2.25.1-8.el9.noarch.r 294 kB/s | 125 kB 00:00 2024-06-03T08:26:41.492 INFO:teuthology.orchestra.run.smithi192.stdout:(11/33): python3-mako-1.1.4-6.el9.noarch.rpm 1.3 MB/s | 172 kB 00:00 2024-06-03T08:26:41.534 INFO:teuthology.orchestra.run.smithi192.stdout:(12/33): python3-markupsafe-1.1.1-12.el9.x86_64 832 kB/s | 35 kB 00:00 2024-06-03T08:26:41.617 INFO:teuthology.orchestra.run.smithi192.stdout:(13/33): python3-urllib3-1.26.5-5.el9.noarch.rp 428 kB/s | 215 kB 00:00 2024-06-03T08:26:41.623 INFO:teuthology.orchestra.run.smithi088.stdout: Installing : ceph-osd-2:18.2.2-1389.gf1abf795.el9.x86_64 30/33 2024-06-03T08:26:41.650 INFO:teuthology.orchestra.run.smithi088.stdout: Running scriptlet: ceph-osd-2:18.2.2-1389.gf1abf795.el9.x86_64 30/33 2024-06-03T08:26:41.650 INFO:teuthology.orchestra.run.smithi088.stdout:Glob pattern passed to enable, but globs are not supported for this. 2024-06-03T08:26:41.650 INFO:teuthology.orchestra.run.smithi088.stdout:Invalid unit name "ceph-osd@*.service" escaped as "ceph-osd@\x2a.service". 2024-06-03T08:26:41.651 INFO:teuthology.orchestra.run.smithi088.stdout:Created symlink /etc/systemd/system/multi-user.target.wants/ceph-osd.target → /usr/lib/systemd/system/ceph-osd.target. 2024-06-03T08:26:41.651 INFO:teuthology.orchestra.run.smithi088.stdout:Created symlink /etc/systemd/system/ceph.target.wants/ceph-osd.target → /usr/lib/systemd/system/ceph-osd.target. 2024-06-03T08:26:41.651 INFO:teuthology.orchestra.run.smithi088.stdout: 2024-06-03T08:26:41.684 INFO:teuthology.orchestra.run.smithi192.stdout:(14/33): python3-pytz-2021.1-5.el9.noarch.rpm 339 kB/s | 51 kB 00:00 2024-06-03T08:26:41.726 INFO:teuthology.orchestra.run.smithi192.stdout:(15/33): python3-toml-0.10.2-6.el9.noarch.rpm 383 kB/s | 42 kB 00:00 2024-06-03T08:26:41.860 INFO:teuthology.orchestra.run.smithi192.stdout:(16/33): python3-bcrypt-3.2.2-1.el9.x86_64.rpm 248 kB/s | 43 kB 00:00 2024-06-03T08:26:41.952 INFO:teuthology.orchestra.run.smithi192.stdout:(17/33): python3-cheroot-8.6.0-4.el9.noarch.rpm 763 kB/s | 172 kB 00:00 2024-06-03T08:26:42.002 INFO:teuthology.orchestra.run.smithi192.stdout:(18/33): python3-jaraco-8.2.1-3.el9.noarch.rpm 213 kB/s | 11 kB 00:00 2024-06-03T08:26:42.061 INFO:teuthology.orchestra.run.smithi192.stdout:(19/33): python3-cherrypy-18.6.1-2.el9.noarch.r 1.7 MB/s | 358 kB 00:00 2024-06-03T08:26:42.086 INFO:teuthology.orchestra.run.smithi192.stdout:(20/33): python3-jaraco-classes-3.2.1-5.el9.noa 211 kB/s | 18 kB 00:00 2024-06-03T08:26:42.090 INFO:teuthology.orchestra.run.smithi088.stdout: Installing : ceph-mon-2:18.2.2-1389.gf1abf795.el9.x86_64 31/33 2024-06-03T08:26:42.114 INFO:teuthology.orchestra.run.smithi088.stdout: Running scriptlet: ceph-mon-2:18.2.2-1389.gf1abf795.el9.x86_64 31/33 2024-06-03T08:26:42.114 INFO:teuthology.orchestra.run.smithi088.stdout:Glob pattern passed to enable, but globs are not supported for this. 2024-06-03T08:26:42.114 INFO:teuthology.orchestra.run.smithi088.stdout:Invalid unit name "ceph-mon@*.service" escaped as "ceph-mon@\x2a.service". 2024-06-03T08:26:42.115 INFO:teuthology.orchestra.run.smithi088.stdout:Created symlink /etc/systemd/system/multi-user.target.wants/ceph-mon.target → /usr/lib/systemd/system/ceph-mon.target. 2024-06-03T08:26:42.115 INFO:teuthology.orchestra.run.smithi088.stdout:Created symlink /etc/systemd/system/ceph.target.wants/ceph-mon.target → /usr/lib/systemd/system/ceph-mon.target. 2024-06-03T08:26:42.115 INFO:teuthology.orchestra.run.smithi088.stdout: 2024-06-03T08:26:42.120 INFO:teuthology.orchestra.run.smithi192.stdout:(21/33): python3-jaraco-collections-3.0.0-8.el9 396 kB/s | 23 kB 00:00 2024-06-03T08:26:42.145 INFO:teuthology.orchestra.run.smithi192.stdout:(22/33): python3-jaraco-functools-3.5.0-2.el9.n 332 kB/s | 19 kB 00:00 2024-06-03T08:26:42.195 INFO:teuthology.orchestra.run.smithi192.stdout:(23/33): python3-jaraco-text-3.2.0-6.el9.noarch 262 kB/s | 20 kB 00:00 2024-06-03T08:26:42.221 INFO:teuthology.orchestra.run.smithi192.stdout:(24/33): python3-logutils-0.3.5-21.el9.noarch.r 615 kB/s | 46 kB 00:00 2024-06-03T08:26:42.246 INFO:teuthology.orchestra.run.smithi192.stdout:(25/33): python3-more-itertools-8.12.0-2.el9.no 1.5 MB/s | 79 kB 00:00 2024-06-03T08:26:42.313 INFO:teuthology.orchestra.run.smithi192.stdout:(26/33): python3-portend-3.1.0-2.el9.noarch.rpm 247 kB/s | 16 kB 00:00 2024-06-03T08:26:42.316 INFO:teuthology.orchestra.run.smithi088.stdout: Installing : ceph-mds-2:18.2.2-1389.gf1abf795.el9.x86_64 32/33 2024-06-03T08:26:42.338 INFO:teuthology.orchestra.run.smithi192.stdout:(27/33): python3-pecan-1.4.2-3.el9.noarch.rpm 2.3 MB/s | 272 kB 00:00 2024-06-03T08:26:42.339 INFO:teuthology.orchestra.run.smithi088.stdout: Running scriptlet: ceph-mds-2:18.2.2-1389.gf1abf795.el9.x86_64 32/33 2024-06-03T08:26:42.339 INFO:teuthology.orchestra.run.smithi088.stdout:Glob pattern passed to enable, but globs are not supported for this. 2024-06-03T08:26:42.340 INFO:teuthology.orchestra.run.smithi088.stdout:Invalid unit name "ceph-mds@*.service" escaped as "ceph-mds@\x2a.service". 2024-06-03T08:26:42.340 INFO:teuthology.orchestra.run.smithi088.stdout:Created symlink /etc/systemd/system/multi-user.target.wants/ceph-mds.target → /usr/lib/systemd/system/ceph-mds.target. 2024-06-03T08:26:42.340 INFO:teuthology.orchestra.run.smithi088.stdout:Created symlink /etc/systemd/system/ceph.target.wants/ceph-mds.target → /usr/lib/systemd/system/ceph-mds.target. 2024-06-03T08:26:42.340 INFO:teuthology.orchestra.run.smithi088.stdout: 2024-06-03T08:26:42.378 INFO:teuthology.orchestra.run.smithi192.stdout:(28/33): python3-pyOpenSSL-21.0.0-1.el9.noarch. 1.4 MB/s | 90 kB 00:00 2024-06-03T08:26:42.404 INFO:teuthology.orchestra.run.smithi088.stdout: Installing : ceph-2:18.2.2-1389.gf1abf795.el9.x86_64 33/33 2024-06-03T08:26:42.406 INFO:teuthology.orchestra.run.smithi192.stdout:(29/33): python3-tempora-5.0.0-2.el9.noarch.rpm 535 kB/s | 36 kB 00:00 2024-06-03T08:26:42.456 INFO:teuthology.orchestra.run.smithi192.stdout:(30/33): python3-webob-1.8.7-6.el9.noarch.rpm 2.9 MB/s | 230 kB 00:00 2024-06-03T08:26:42.490 INFO:teuthology.orchestra.run.smithi192.stdout:(31/33): python3-werkzeug-2.0.3-3.el9.1.noarch. 5.0 MB/s | 427 kB 00:00 2024-06-03T08:26:42.515 INFO:teuthology.orchestra.run.smithi192.stdout:(32/33): python3-zc-lockfile-2.0-10.el9.noarch. 342 kB/s | 20 kB 00:00 2024-06-03T08:26:42.682 INFO:teuthology.orchestra.run.smithi192.stdout:(33/33): python3-cryptography-36.0.1-4.el9.x86_ 529 kB/s | 1.2 MB 00:02 2024-06-03T08:26:42.685 INFO:teuthology.orchestra.run.smithi192.stdout:-------------------------------------------------------------------------------- 2024-06-03T08:26:42.685 INFO:teuthology.orchestra.run.smithi192.stdout:Total 7.3 MB/s | 29 MB 00:03 2024-06-03T08:26:42.808 INFO:teuthology.orchestra.run.smithi192.stdout:Running transaction check 2024-06-03T08:26:42.844 INFO:teuthology.orchestra.run.smithi192.stdout:Transaction check succeeded. 2024-06-03T08:26:42.844 INFO:teuthology.orchestra.run.smithi192.stdout:Running transaction test 2024-06-03T08:26:43.232 INFO:teuthology.orchestra.run.smithi192.stdout:Transaction test succeeded. 2024-06-03T08:26:43.233 INFO:teuthology.orchestra.run.smithi192.stdout:Running transaction 2024-06-03T08:26:44.016 INFO:teuthology.orchestra.run.smithi088.stdout: Running scriptlet: ceph-2:18.2.2-1389.gf1abf795.el9.x86_64 33/33 2024-06-03T08:26:44.016 INFO:teuthology.orchestra.run.smithi088.stdout: Verifying : ceph-2:18.2.2-1389.gf1abf795.el9.x86_64 1/33 2024-06-03T08:26:44.016 INFO:teuthology.orchestra.run.smithi088.stdout: Verifying : ceph-mds-2:18.2.2-1389.gf1abf795.el9.x86_64 2/33 2024-06-03T08:26:44.016 INFO:teuthology.orchestra.run.smithi088.stdout: Verifying : ceph-mgr-2:18.2.2-1389.gf1abf795.el9.x86_64 3/33 2024-06-03T08:26:44.017 INFO:teuthology.orchestra.run.smithi088.stdout: Verifying : ceph-mon-2:18.2.2-1389.gf1abf795.el9.x86_64 4/33 2024-06-03T08:26:44.017 INFO:teuthology.orchestra.run.smithi088.stdout: Verifying : ceph-osd-2:18.2.2-1389.gf1abf795.el9.x86_64 5/33 2024-06-03T08:26:44.017 INFO:teuthology.orchestra.run.smithi088.stdout: Verifying : ceph-mgr-modules-core-2:18.2.2-1389.gf1abf795.el9. 6/33 2024-06-03T08:26:44.017 INFO:teuthology.orchestra.run.smithi088.stdout: Verifying : python3-cffi-1.14.5-5.el9.x86_64 7/33 2024-06-03T08:26:44.017 INFO:teuthology.orchestra.run.smithi088.stdout: Verifying : python3-cryptography-36.0.1-4.el9.x86_64 8/33 2024-06-03T08:26:44.017 INFO:teuthology.orchestra.run.smithi088.stdout: Verifying : python3-ply-3.11-14.el9.noarch 9/33 2024-06-03T08:26:44.017 INFO:teuthology.orchestra.run.smithi088.stdout: Verifying : python3-pycparser-2.20-6.el9.noarch 10/33 2024-06-03T08:26:44.017 INFO:teuthology.orchestra.run.smithi088.stdout: Verifying : python3-requests-2.25.1-8.el9.noarch 11/33 2024-06-03T08:26:44.017 INFO:teuthology.orchestra.run.smithi088.stdout: Verifying : python3-urllib3-1.26.5-5.el9.noarch 12/33 2024-06-03T08:26:44.017 INFO:teuthology.orchestra.run.smithi088.stdout: Verifying : python3-mako-1.1.4-6.el9.noarch 13/33 2024-06-03T08:26:44.017 INFO:teuthology.orchestra.run.smithi088.stdout: Verifying : python3-markupsafe-1.1.1-12.el9.x86_64 14/33 2024-06-03T08:26:44.017 INFO:teuthology.orchestra.run.smithi088.stdout: Verifying : python3-pytz-2021.1-5.el9.noarch 15/33 2024-06-03T08:26:44.017 INFO:teuthology.orchestra.run.smithi088.stdout: Verifying : python3-toml-0.10.2-6.el9.noarch 16/33 2024-06-03T08:26:44.018 INFO:teuthology.orchestra.run.smithi088.stdout: Verifying : python3-bcrypt-3.2.2-1.el9.x86_64 17/33 2024-06-03T08:26:44.018 INFO:teuthology.orchestra.run.smithi088.stdout: Verifying : python3-cheroot-8.6.0-4.el9.noarch 18/33 2024-06-03T08:26:44.018 INFO:teuthology.orchestra.run.smithi088.stdout: Verifying : python3-cherrypy-18.6.1-2.el9.noarch 19/33 2024-06-03T08:26:44.018 INFO:teuthology.orchestra.run.smithi088.stdout: Verifying : python3-jaraco-8.2.1-3.el9.noarch 20/33 2024-06-03T08:26:44.018 INFO:teuthology.orchestra.run.smithi088.stdout: Verifying : python3-jaraco-classes-3.2.1-5.el9.noarch 21/33 2024-06-03T08:26:44.018 INFO:teuthology.orchestra.run.smithi088.stdout: Verifying : python3-jaraco-collections-3.0.0-8.el9.noarch 22/33 2024-06-03T08:26:44.018 INFO:teuthology.orchestra.run.smithi088.stdout: Verifying : python3-jaraco-functools-3.5.0-2.el9.noarch 23/33 2024-06-03T08:26:44.018 INFO:teuthology.orchestra.run.smithi088.stdout: Verifying : python3-jaraco-text-3.2.0-6.el9.noarch 24/33 2024-06-03T08:26:44.018 INFO:teuthology.orchestra.run.smithi088.stdout: Verifying : python3-logutils-0.3.5-21.el9.noarch 25/33 2024-06-03T08:26:44.018 INFO:teuthology.orchestra.run.smithi088.stdout: Verifying : python3-more-itertools-8.12.0-2.el9.noarch 26/33 2024-06-03T08:26:44.018 INFO:teuthology.orchestra.run.smithi088.stdout: Verifying : python3-pecan-1.4.2-3.el9.noarch 27/33 2024-06-03T08:26:44.018 INFO:teuthology.orchestra.run.smithi088.stdout: Verifying : python3-portend-3.1.0-2.el9.noarch 28/33 2024-06-03T08:26:44.019 INFO:teuthology.orchestra.run.smithi088.stdout: Verifying : python3-pyOpenSSL-21.0.0-1.el9.noarch 29/33 2024-06-03T08:26:44.019 INFO:teuthology.orchestra.run.smithi088.stdout: Verifying : python3-tempora-5.0.0-2.el9.noarch 30/33 2024-06-03T08:26:44.019 INFO:teuthology.orchestra.run.smithi088.stdout: Verifying : python3-webob-1.8.7-6.el9.noarch 31/33 2024-06-03T08:26:44.019 INFO:teuthology.orchestra.run.smithi088.stdout: Verifying : python3-werkzeug-2.0.3-3.el9.1.noarch 32/33 2024-06-03T08:26:44.386 INFO:teuthology.orchestra.run.smithi192.stdout: Preparing : 1/1 2024-06-03T08:26:44.460 INFO:teuthology.orchestra.run.smithi192.stdout: Installing : python3-more-itertools-8.12.0-2.el9.noarch 1/33 2024-06-03T08:26:44.491 INFO:teuthology.orchestra.run.smithi192.stdout: Installing : python3-jaraco-8.2.1-3.el9.noarch 2/33 2024-06-03T08:26:44.525 INFO:teuthology.orchestra.run.smithi192.stdout: Installing : python3-jaraco-classes-3.2.1-5.el9.noarch 3/33 2024-06-03T08:26:44.616 INFO:teuthology.orchestra.run.smithi192.stdout: Installing : python3-zc-lockfile-2.0-10.el9.noarch 4/33 2024-06-03T08:26:44.703 INFO:teuthology.orchestra.run.smithi192.stdout: Installing : python3-werkzeug-2.0.3-3.el9.1.noarch 5/33 2024-06-03T08:26:44.744 INFO:teuthology.orchestra.run.smithi192.stdout: Installing : python3-webob-1.8.7-6.el9.noarch 6/33 2024-06-03T08:26:44.779 INFO:teuthology.orchestra.run.smithi192.stdout: Installing : python3-logutils-0.3.5-21.el9.noarch 7/33 2024-06-03T08:26:44.809 INFO:teuthology.orchestra.run.smithi192.stdout: Installing : python3-toml-0.10.2-6.el9.noarch 8/33 2024-06-03T08:26:44.844 INFO:teuthology.orchestra.run.smithi192.stdout: Installing : python3-jaraco-functools-3.5.0-2.el9.noarch 9/33 2024-06-03T08:26:44.877 INFO:teuthology.orchestra.run.smithi192.stdout: Installing : python3-jaraco-text-3.2.0-6.el9.noarch 10/33 2024-06-03T08:26:44.918 INFO:teuthology.orchestra.run.smithi192.stdout: Installing : python3-jaraco-collections-3.0.0-8.el9.noarch 11/33 2024-06-03T08:26:44.952 INFO:teuthology.orchestra.run.smithi192.stdout: Installing : python3-pytz-2021.1-5.el9.noarch 12/33 2024-06-03T08:26:44.987 INFO:teuthology.orchestra.run.smithi192.stdout: Installing : python3-tempora-5.0.0-2.el9.noarch 13/33 2024-06-03T08:26:45.029 INFO:teuthology.orchestra.run.smithi192.stdout: Installing : python3-portend-3.1.0-2.el9.noarch 14/33 2024-06-03T08:26:45.044 INFO:teuthology.orchestra.run.smithi088.stdout: Verifying : python3-zc-lockfile-2.0-10.el9.noarch 33/33 2024-06-03T08:26:45.044 INFO:teuthology.orchestra.run.smithi088.stdout: 2024-06-03T08:26:45.044 INFO:teuthology.orchestra.run.smithi088.stdout:Installed: 2024-06-03T08:26:45.044 INFO:teuthology.orchestra.run.smithi088.stdout: ceph-2:18.2.2-1389.gf1abf795.el9.x86_64 2024-06-03T08:26:45.045 INFO:teuthology.orchestra.run.smithi088.stdout: ceph-mds-2:18.2.2-1389.gf1abf795.el9.x86_64 2024-06-03T08:26:45.045 INFO:teuthology.orchestra.run.smithi088.stdout: ceph-mgr-2:18.2.2-1389.gf1abf795.el9.x86_64 2024-06-03T08:26:45.045 INFO:teuthology.orchestra.run.smithi088.stdout: ceph-mgr-modules-core-2:18.2.2-1389.gf1abf795.el9.noarch 2024-06-03T08:26:45.045 INFO:teuthology.orchestra.run.smithi088.stdout: ceph-mon-2:18.2.2-1389.gf1abf795.el9.x86_64 2024-06-03T08:26:45.045 INFO:teuthology.orchestra.run.smithi088.stdout: ceph-osd-2:18.2.2-1389.gf1abf795.el9.x86_64 2024-06-03T08:26:45.045 INFO:teuthology.orchestra.run.smithi088.stdout: python3-bcrypt-3.2.2-1.el9.x86_64 2024-06-03T08:26:45.045 INFO:teuthology.orchestra.run.smithi088.stdout: python3-cffi-1.14.5-5.el9.x86_64 2024-06-03T08:26:45.045 INFO:teuthology.orchestra.run.smithi088.stdout: python3-cheroot-8.6.0-4.el9.noarch 2024-06-03T08:26:45.045 INFO:teuthology.orchestra.run.smithi088.stdout: python3-cherrypy-18.6.1-2.el9.noarch 2024-06-03T08:26:45.045 INFO:teuthology.orchestra.run.smithi088.stdout: python3-cryptography-36.0.1-4.el9.x86_64 2024-06-03T08:26:45.045 INFO:teuthology.orchestra.run.smithi088.stdout: python3-jaraco-8.2.1-3.el9.noarch 2024-06-03T08:26:45.045 INFO:teuthology.orchestra.run.smithi088.stdout: python3-jaraco-classes-3.2.1-5.el9.noarch 2024-06-03T08:26:45.046 INFO:teuthology.orchestra.run.smithi088.stdout: python3-jaraco-collections-3.0.0-8.el9.noarch 2024-06-03T08:26:45.046 INFO:teuthology.orchestra.run.smithi088.stdout: python3-jaraco-functools-3.5.0-2.el9.noarch 2024-06-03T08:26:45.046 INFO:teuthology.orchestra.run.smithi088.stdout: python3-jaraco-text-3.2.0-6.el9.noarch 2024-06-03T08:26:45.046 INFO:teuthology.orchestra.run.smithi088.stdout: python3-logutils-0.3.5-21.el9.noarch 2024-06-03T08:26:45.046 INFO:teuthology.orchestra.run.smithi088.stdout: python3-mako-1.1.4-6.el9.noarch 2024-06-03T08:26:45.046 INFO:teuthology.orchestra.run.smithi088.stdout: python3-markupsafe-1.1.1-12.el9.x86_64 2024-06-03T08:26:45.046 INFO:teuthology.orchestra.run.smithi088.stdout: python3-more-itertools-8.12.0-2.el9.noarch 2024-06-03T08:26:45.046 INFO:teuthology.orchestra.run.smithi088.stdout: python3-pecan-1.4.2-3.el9.noarch 2024-06-03T08:26:45.046 INFO:teuthology.orchestra.run.smithi088.stdout: python3-ply-3.11-14.el9.noarch 2024-06-03T08:26:45.046 INFO:teuthology.orchestra.run.smithi088.stdout: python3-portend-3.1.0-2.el9.noarch 2024-06-03T08:26:45.046 INFO:teuthology.orchestra.run.smithi088.stdout: python3-pyOpenSSL-21.0.0-1.el9.noarch 2024-06-03T08:26:45.047 INFO:teuthology.orchestra.run.smithi088.stdout: python3-pycparser-2.20-6.el9.noarch 2024-06-03T08:26:45.047 INFO:teuthology.orchestra.run.smithi088.stdout: python3-pytz-2021.1-5.el9.noarch 2024-06-03T08:26:45.047 INFO:teuthology.orchestra.run.smithi088.stdout: python3-requests-2.25.1-8.el9.noarch 2024-06-03T08:26:45.047 INFO:teuthology.orchestra.run.smithi088.stdout: python3-tempora-5.0.0-2.el9.noarch 2024-06-03T08:26:45.047 INFO:teuthology.orchestra.run.smithi088.stdout: python3-toml-0.10.2-6.el9.noarch 2024-06-03T08:26:45.047 INFO:teuthology.orchestra.run.smithi088.stdout: python3-urllib3-1.26.5-5.el9.noarch 2024-06-03T08:26:45.047 INFO:teuthology.orchestra.run.smithi088.stdout: python3-webob-1.8.7-6.el9.noarch 2024-06-03T08:26:45.047 INFO:teuthology.orchestra.run.smithi088.stdout: python3-werkzeug-2.0.3-3.el9.1.noarch 2024-06-03T08:26:45.047 INFO:teuthology.orchestra.run.smithi088.stdout: python3-zc-lockfile-2.0-10.el9.noarch 2024-06-03T08:26:45.047 INFO:teuthology.orchestra.run.smithi088.stdout: 2024-06-03T08:26:45.047 INFO:teuthology.orchestra.run.smithi088.stdout:Complete! 2024-06-03T08:26:45.107 INFO:teuthology.orchestra.run.smithi192.stdout: Installing : python3-markupsafe-1.1.1-12.el9.x86_64 15/33 2024-06-03T08:26:45.254 INFO:teuthology.orchestra.run.smithi192.stdout: Installing : python3-mako-1.1.4-6.el9.noarch 16/33 2024-06-03T08:26:45.333 INFO:teuthology.orchestra.run.smithi192.stdout: Installing : python3-pecan-1.4.2-3.el9.noarch 17/33 2024-06-03T08:26:45.378 DEBUG:teuthology.orchestra.run.smithi088:> sudo yum -y install ceph-base 2024-06-03T08:26:45.387 INFO:teuthology.orchestra.run.smithi192.stdout: Installing : python3-urllib3-1.26.5-5.el9.noarch 18/33 2024-06-03T08:26:45.426 INFO:teuthology.orchestra.run.smithi192.stdout: Installing : python3-requests-2.25.1-8.el9.noarch 19/33 2024-06-03T08:26:45.483 INFO:teuthology.orchestra.run.smithi192.stdout: Installing : python3-ply-3.11-14.el9.noarch 20/33 2024-06-03T08:26:45.546 INFO:teuthology.orchestra.run.smithi192.stdout: Installing : python3-pycparser-2.20-6.el9.noarch 21/33 2024-06-03T08:26:45.734 INFO:teuthology.orchestra.run.smithi192.stdout: Installing : python3-cffi-1.14.5-5.el9.x86_64 22/33 2024-06-03T08:26:45.780 INFO:teuthology.orchestra.run.smithi192.stdout: Installing : python3-cryptography-36.0.1-4.el9.x86_64 23/33 2024-06-03T08:26:45.858 INFO:teuthology.orchestra.run.smithi192.stdout: Installing : python3-pyOpenSSL-21.0.0-1.el9.noarch 24/33 2024-06-03T08:26:45.946 INFO:teuthology.orchestra.run.smithi192.stdout: Installing : python3-cheroot-8.6.0-4.el9.noarch 25/33 2024-06-03T08:26:45.964 INFO:teuthology.orchestra.run.smithi088.stdout:Last metadata expiration check: 0:00:49 ago on Mon 03 Jun 2024 08:25:56 AM UTC. 2024-06-03T08:26:45.987 INFO:teuthology.orchestra.run.smithi192.stdout: Installing : python3-cherrypy-18.6.1-2.el9.noarch 26/33 2024-06-03T08:26:46.045 INFO:teuthology.orchestra.run.smithi088.stdout:Package ceph-base-2:18.2.2-1389.gf1abf795.el9.x86_64 is already installed. 2024-06-03T08:26:46.088 INFO:teuthology.orchestra.run.smithi088.stdout:Dependencies resolved. 2024-06-03T08:26:46.089 INFO:teuthology.orchestra.run.smithi088.stdout:Nothing to do. 2024-06-03T08:26:46.089 INFO:teuthology.orchestra.run.smithi088.stdout:Complete! 2024-06-03T08:26:46.103 INFO:teuthology.orchestra.run.smithi192.stdout: Installing : python3-bcrypt-3.2.2-1.el9.x86_64 27/33 2024-06-03T08:26:46.152 DEBUG:teuthology.orchestra.run.smithi088:> sudo yum -y install cephadm 2024-06-03T08:26:46.255 INFO:teuthology.orchestra.run.smithi192.stdout: Installing : ceph-mgr-modules-core-2:18.2.2-1389.gf1abf795.el9. 28/33 2024-06-03T08:26:46.282 INFO:teuthology.orchestra.run.smithi192.stdout: Installing : ceph-mgr-2:18.2.2-1389.gf1abf795.el9.x86_64 29/33 2024-06-03T08:26:46.307 INFO:teuthology.orchestra.run.smithi192.stdout: Running scriptlet: ceph-mgr-2:18.2.2-1389.gf1abf795.el9.x86_64 29/33 2024-06-03T08:26:46.307 INFO:teuthology.orchestra.run.smithi192.stdout:Glob pattern passed to enable, but globs are not supported for this. 2024-06-03T08:26:46.307 INFO:teuthology.orchestra.run.smithi192.stdout:Invalid unit name "ceph-mgr@*.service" escaped as "ceph-mgr@\x2a.service". 2024-06-03T08:26:46.308 INFO:teuthology.orchestra.run.smithi192.stdout:Created symlink /etc/systemd/system/multi-user.target.wants/ceph-mgr.target → /usr/lib/systemd/system/ceph-mgr.target. 2024-06-03T08:26:46.308 INFO:teuthology.orchestra.run.smithi192.stdout:Created symlink /etc/systemd/system/ceph.target.wants/ceph-mgr.target → /usr/lib/systemd/system/ceph-mgr.target. 2024-06-03T08:26:46.308 INFO:teuthology.orchestra.run.smithi192.stdout: 2024-06-03T08:26:46.728 INFO:teuthology.orchestra.run.smithi088.stdout:Last metadata expiration check: 0:00:50 ago on Mon 03 Jun 2024 08:25:56 AM UTC. 2024-06-03T08:26:46.851 INFO:teuthology.orchestra.run.smithi088.stdout:Dependencies resolved. 2024-06-03T08:26:46.852 INFO:teuthology.orchestra.run.smithi088.stdout:================================================================================ 2024-06-03T08:26:46.852 INFO:teuthology.orchestra.run.smithi088.stdout: Package Arch Version Repository Size 2024-06-03T08:26:46.852 INFO:teuthology.orchestra.run.smithi088.stdout:================================================================================ 2024-06-03T08:26:46.852 INFO:teuthology.orchestra.run.smithi088.stdout:Installing: 2024-06-03T08:26:46.852 INFO:teuthology.orchestra.run.smithi088.stdout: cephadm noarch 2:18.2.2-1389.gf1abf795.el9 ceph-noarch 224 k 2024-06-03T08:26:46.852 INFO:teuthology.orchestra.run.smithi088.stdout: 2024-06-03T08:26:46.852 INFO:teuthology.orchestra.run.smithi088.stdout:Transaction Summary 2024-06-03T08:26:46.852 INFO:teuthology.orchestra.run.smithi088.stdout:================================================================================ 2024-06-03T08:26:46.852 INFO:teuthology.orchestra.run.smithi088.stdout:Install 1 Package 2024-06-03T08:26:46.852 INFO:teuthology.orchestra.run.smithi088.stdout: 2024-06-03T08:26:46.853 INFO:teuthology.orchestra.run.smithi088.stdout:Total download size: 224 k 2024-06-03T08:26:46.853 INFO:teuthology.orchestra.run.smithi088.stdout:Installed size: 216 k 2024-06-03T08:26:46.853 INFO:teuthology.orchestra.run.smithi088.stdout:Downloading Packages: 2024-06-03T08:26:47.233 INFO:teuthology.orchestra.run.smithi088.stdout:cephadm-18.2.2-1389.gf1abf795.el9.noarch.rpm 590 kB/s | 224 kB 00:00 2024-06-03T08:26:47.234 INFO:teuthology.orchestra.run.smithi088.stdout:-------------------------------------------------------------------------------- 2024-06-03T08:26:47.234 INFO:teuthology.orchestra.run.smithi088.stdout:Total 587 kB/s | 224 kB 00:00 2024-06-03T08:26:47.234 INFO:teuthology.orchestra.run.smithi088.stdout:Running transaction check 2024-06-03T08:26:47.242 INFO:teuthology.orchestra.run.smithi088.stdout:Transaction check succeeded. 2024-06-03T08:26:47.242 INFO:teuthology.orchestra.run.smithi088.stdout:Running transaction test 2024-06-03T08:26:47.243 INFO:teuthology.orchestra.run.smithi088.stdout:Transaction test succeeded. 2024-06-03T08:26:47.243 INFO:teuthology.orchestra.run.smithi088.stdout:Running transaction 2024-06-03T08:26:47.310 INFO:teuthology.orchestra.run.smithi088.stdout: Preparing : 1/1 2024-06-03T08:26:47.754 INFO:teuthology.orchestra.run.smithi088.stdout: Running scriptlet: cephadm-2:18.2.2-1389.gf1abf795.el9.noarch 1/1 2024-06-03T08:26:47.767 INFO:teuthology.orchestra.run.smithi192.stdout: Installing : ceph-osd-2:18.2.2-1389.gf1abf795.el9.x86_64 30/33 2024-06-03T08:26:47.796 INFO:teuthology.orchestra.run.smithi192.stdout: Running scriptlet: ceph-osd-2:18.2.2-1389.gf1abf795.el9.x86_64 30/33 2024-06-03T08:26:47.796 INFO:teuthology.orchestra.run.smithi192.stdout:Glob pattern passed to enable, but globs are not supported for this. 2024-06-03T08:26:47.796 INFO:teuthology.orchestra.run.smithi192.stdout:Invalid unit name "ceph-osd@*.service" escaped as "ceph-osd@\x2a.service". 2024-06-03T08:26:47.796 INFO:teuthology.orchestra.run.smithi192.stdout:Created symlink /etc/systemd/system/multi-user.target.wants/ceph-osd.target → /usr/lib/systemd/system/ceph-osd.target. 2024-06-03T08:26:47.796 INFO:teuthology.orchestra.run.smithi192.stdout:Created symlink /etc/systemd/system/ceph.target.wants/ceph-osd.target → /usr/lib/systemd/system/ceph-osd.target. 2024-06-03T08:26:47.796 INFO:teuthology.orchestra.run.smithi192.stdout: 2024-06-03T08:26:47.846 INFO:teuthology.orchestra.run.smithi088.stdout: Installing : cephadm-2:18.2.2-1389.gf1abf795.el9.noarch 1/1 2024-06-03T08:26:48.202 INFO:teuthology.orchestra.run.smithi088.stdout: Running scriptlet: cephadm-2:18.2.2-1389.gf1abf795.el9.noarch 1/1 2024-06-03T08:26:48.251 INFO:teuthology.orchestra.run.smithi192.stdout: Installing : ceph-mon-2:18.2.2-1389.gf1abf795.el9.x86_64 31/33 2024-06-03T08:26:48.276 INFO:teuthology.orchestra.run.smithi192.stdout: Running scriptlet: ceph-mon-2:18.2.2-1389.gf1abf795.el9.x86_64 31/33 2024-06-03T08:26:48.276 INFO:teuthology.orchestra.run.smithi192.stdout:Glob pattern passed to enable, but globs are not supported for this. 2024-06-03T08:26:48.276 INFO:teuthology.orchestra.run.smithi192.stdout:Invalid unit name "ceph-mon@*.service" escaped as "ceph-mon@\x2a.service". 2024-06-03T08:26:48.276 INFO:teuthology.orchestra.run.smithi192.stdout:Created symlink /etc/systemd/system/multi-user.target.wants/ceph-mon.target → /usr/lib/systemd/system/ceph-mon.target. 2024-06-03T08:26:48.276 INFO:teuthology.orchestra.run.smithi192.stdout:Created symlink /etc/systemd/system/ceph.target.wants/ceph-mon.target → /usr/lib/systemd/system/ceph-mon.target. 2024-06-03T08:26:48.276 INFO:teuthology.orchestra.run.smithi192.stdout: 2024-06-03T08:26:48.432 INFO:teuthology.orchestra.run.smithi088.stdout: Verifying : cephadm-2:18.2.2-1389.gf1abf795.el9.noarch 1/1 2024-06-03T08:26:48.432 INFO:teuthology.orchestra.run.smithi088.stdout: 2024-06-03T08:26:48.432 INFO:teuthology.orchestra.run.smithi088.stdout:Installed: 2024-06-03T08:26:48.432 INFO:teuthology.orchestra.run.smithi088.stdout: cephadm-2:18.2.2-1389.gf1abf795.el9.noarch 2024-06-03T08:26:48.433 INFO:teuthology.orchestra.run.smithi088.stdout: 2024-06-03T08:26:48.433 INFO:teuthology.orchestra.run.smithi088.stdout:Complete! 2024-06-03T08:26:48.485 INFO:teuthology.orchestra.run.smithi192.stdout: Installing : ceph-mds-2:18.2.2-1389.gf1abf795.el9.x86_64 32/33 2024-06-03T08:26:48.508 INFO:teuthology.orchestra.run.smithi192.stdout: Running scriptlet: ceph-mds-2:18.2.2-1389.gf1abf795.el9.x86_64 32/33 2024-06-03T08:26:48.508 INFO:teuthology.orchestra.run.smithi192.stdout:Glob pattern passed to enable, but globs are not supported for this. 2024-06-03T08:26:48.509 INFO:teuthology.orchestra.run.smithi192.stdout:Invalid unit name "ceph-mds@*.service" escaped as "ceph-mds@\x2a.service". 2024-06-03T08:26:48.509 INFO:teuthology.orchestra.run.smithi192.stdout:Created symlink /etc/systemd/system/multi-user.target.wants/ceph-mds.target → /usr/lib/systemd/system/ceph-mds.target. 2024-06-03T08:26:48.509 INFO:teuthology.orchestra.run.smithi192.stdout:Created symlink /etc/systemd/system/ceph.target.wants/ceph-mds.target → /usr/lib/systemd/system/ceph-mds.target. 2024-06-03T08:26:48.509 INFO:teuthology.orchestra.run.smithi192.stdout: 2024-06-03T08:26:48.574 INFO:teuthology.orchestra.run.smithi192.stdout: Installing : ceph-2:18.2.2-1389.gf1abf795.el9.x86_64 33/33 2024-06-03T08:26:48.629 DEBUG:teuthology.orchestra.run.smithi088:> sudo yum -y install ceph-immutable-object-cache 2024-06-03T08:26:49.221 INFO:teuthology.orchestra.run.smithi088.stdout:Last metadata expiration check: 0:00:53 ago on Mon 03 Jun 2024 08:25:56 AM UTC. 2024-06-03T08:26:49.345 INFO:teuthology.orchestra.run.smithi088.stdout:Dependencies resolved. 2024-06-03T08:26:49.346 INFO:teuthology.orchestra.run.smithi088.stdout:================================================================================ 2024-06-03T08:26:49.346 INFO:teuthology.orchestra.run.smithi088.stdout: Package Arch Version Repo Size 2024-06-03T08:26:49.346 INFO:teuthology.orchestra.run.smithi088.stdout:================================================================================ 2024-06-03T08:26:49.346 INFO:teuthology.orchestra.run.smithi088.stdout:Installing: 2024-06-03T08:26:49.346 INFO:teuthology.orchestra.run.smithi088.stdout: ceph-immutable-object-cache x86_64 2:18.2.2-1389.gf1abf795.el9 ceph 143 k 2024-06-03T08:26:49.346 INFO:teuthology.orchestra.run.smithi088.stdout: 2024-06-03T08:26:49.346 INFO:teuthology.orchestra.run.smithi088.stdout:Transaction Summary 2024-06-03T08:26:49.347 INFO:teuthology.orchestra.run.smithi088.stdout:================================================================================ 2024-06-03T08:26:49.347 INFO:teuthology.orchestra.run.smithi088.stdout:Install 1 Package 2024-06-03T08:26:49.347 INFO:teuthology.orchestra.run.smithi088.stdout: 2024-06-03T08:26:49.347 INFO:teuthology.orchestra.run.smithi088.stdout:Total download size: 143 k 2024-06-03T08:26:49.347 INFO:teuthology.orchestra.run.smithi088.stdout:Installed size: 395 k 2024-06-03T08:26:49.347 INFO:teuthology.orchestra.run.smithi088.stdout:Downloading Packages: 2024-06-03T08:26:49.565 INFO:teuthology.orchestra.run.smithi088.stdout:ceph-immutable-object-cache-18.2.2-1389.gf1abf7 656 kB/s | 143 kB 00:00 2024-06-03T08:26:49.565 INFO:teuthology.orchestra.run.smithi088.stdout:-------------------------------------------------------------------------------- 2024-06-03T08:26:49.566 INFO:teuthology.orchestra.run.smithi088.stdout:Total 651 kB/s | 143 kB 00:00 2024-06-03T08:26:49.566 INFO:teuthology.orchestra.run.smithi088.stdout:Running transaction check 2024-06-03T08:26:49.573 INFO:teuthology.orchestra.run.smithi088.stdout:Transaction check succeeded. 2024-06-03T08:26:49.574 INFO:teuthology.orchestra.run.smithi088.stdout:Running transaction test 2024-06-03T08:26:49.631 INFO:teuthology.orchestra.run.smithi088.stdout:Transaction test succeeded. 2024-06-03T08:26:49.631 INFO:teuthology.orchestra.run.smithi088.stdout:Running transaction 2024-06-03T08:26:49.772 INFO:teuthology.orchestra.run.smithi088.stdout: Preparing : 1/1 2024-06-03T08:26:49.832 INFO:teuthology.orchestra.run.smithi088.stdout: Installing : ceph-immutable-object-cache-2:18.2.2-1389.gf1abf795. 1/1 2024-06-03T08:26:49.857 INFO:teuthology.orchestra.run.smithi088.stdout: Running scriptlet: ceph-immutable-object-cache-2:18.2.2-1389.gf1abf795. 1/1 2024-06-03T08:26:49.858 INFO:teuthology.orchestra.run.smithi088.stdout:Glob pattern passed to enable, but globs are not supported for this. 2024-06-03T08:26:49.858 INFO:teuthology.orchestra.run.smithi088.stdout:Invalid unit name "ceph-immutable-object-cache@*.service" escaped as "ceph-immutable-object-cache@\x2a.service". 2024-06-03T08:26:49.858 INFO:teuthology.orchestra.run.smithi088.stdout: 2024-06-03T08:26:50.247 INFO:teuthology.orchestra.run.smithi192.stdout: Running scriptlet: ceph-2:18.2.2-1389.gf1abf795.el9.x86_64 33/33 2024-06-03T08:26:50.248 INFO:teuthology.orchestra.run.smithi192.stdout: Verifying : ceph-2:18.2.2-1389.gf1abf795.el9.x86_64 1/33 2024-06-03T08:26:50.248 INFO:teuthology.orchestra.run.smithi192.stdout: Verifying : ceph-mds-2:18.2.2-1389.gf1abf795.el9.x86_64 2/33 2024-06-03T08:26:50.248 INFO:teuthology.orchestra.run.smithi192.stdout: Verifying : ceph-mgr-2:18.2.2-1389.gf1abf795.el9.x86_64 3/33 2024-06-03T08:26:50.248 INFO:teuthology.orchestra.run.smithi192.stdout: Verifying : ceph-mon-2:18.2.2-1389.gf1abf795.el9.x86_64 4/33 2024-06-03T08:26:50.248 INFO:teuthology.orchestra.run.smithi192.stdout: Verifying : ceph-osd-2:18.2.2-1389.gf1abf795.el9.x86_64 5/33 2024-06-03T08:26:50.248 INFO:teuthology.orchestra.run.smithi192.stdout: Verifying : ceph-mgr-modules-core-2:18.2.2-1389.gf1abf795.el9. 6/33 2024-06-03T08:26:50.248 INFO:teuthology.orchestra.run.smithi192.stdout: Verifying : python3-cffi-1.14.5-5.el9.x86_64 7/33 2024-06-03T08:26:50.248 INFO:teuthology.orchestra.run.smithi192.stdout: Verifying : python3-cryptography-36.0.1-4.el9.x86_64 8/33 2024-06-03T08:26:50.248 INFO:teuthology.orchestra.run.smithi192.stdout: Verifying : python3-ply-3.11-14.el9.noarch 9/33 2024-06-03T08:26:50.248 INFO:teuthology.orchestra.run.smithi192.stdout: Verifying : python3-pycparser-2.20-6.el9.noarch 10/33 2024-06-03T08:26:50.249 INFO:teuthology.orchestra.run.smithi192.stdout: Verifying : python3-requests-2.25.1-8.el9.noarch 11/33 2024-06-03T08:26:50.249 INFO:teuthology.orchestra.run.smithi192.stdout: Verifying : python3-urllib3-1.26.5-5.el9.noarch 12/33 2024-06-03T08:26:50.249 INFO:teuthology.orchestra.run.smithi192.stdout: Verifying : python3-mako-1.1.4-6.el9.noarch 13/33 2024-06-03T08:26:50.249 INFO:teuthology.orchestra.run.smithi192.stdout: Verifying : python3-markupsafe-1.1.1-12.el9.x86_64 14/33 2024-06-03T08:26:50.249 INFO:teuthology.orchestra.run.smithi192.stdout: Verifying : python3-pytz-2021.1-5.el9.noarch 15/33 2024-06-03T08:26:50.249 INFO:teuthology.orchestra.run.smithi192.stdout: Verifying : python3-toml-0.10.2-6.el9.noarch 16/33 2024-06-03T08:26:50.249 INFO:teuthology.orchestra.run.smithi192.stdout: Verifying : python3-bcrypt-3.2.2-1.el9.x86_64 17/33 2024-06-03T08:26:50.249 INFO:teuthology.orchestra.run.smithi192.stdout: Verifying : python3-cheroot-8.6.0-4.el9.noarch 18/33 2024-06-03T08:26:50.249 INFO:teuthology.orchestra.run.smithi192.stdout: Verifying : python3-cherrypy-18.6.1-2.el9.noarch 19/33 2024-06-03T08:26:50.249 INFO:teuthology.orchestra.run.smithi192.stdout: Verifying : python3-jaraco-8.2.1-3.el9.noarch 20/33 2024-06-03T08:26:50.249 INFO:teuthology.orchestra.run.smithi192.stdout: Verifying : python3-jaraco-classes-3.2.1-5.el9.noarch 21/33 2024-06-03T08:26:50.249 INFO:teuthology.orchestra.run.smithi192.stdout: Verifying : python3-jaraco-collections-3.0.0-8.el9.noarch 22/33 2024-06-03T08:26:50.249 INFO:teuthology.orchestra.run.smithi192.stdout: Verifying : python3-jaraco-functools-3.5.0-2.el9.noarch 23/33 2024-06-03T08:26:50.249 INFO:teuthology.orchestra.run.smithi192.stdout: Verifying : python3-jaraco-text-3.2.0-6.el9.noarch 24/33 2024-06-03T08:26:50.250 INFO:teuthology.orchestra.run.smithi192.stdout: Verifying : python3-logutils-0.3.5-21.el9.noarch 25/33 2024-06-03T08:26:50.250 INFO:teuthology.orchestra.run.smithi192.stdout: Verifying : python3-more-itertools-8.12.0-2.el9.noarch 26/33 2024-06-03T08:26:50.250 INFO:teuthology.orchestra.run.smithi192.stdout: Verifying : python3-pecan-1.4.2-3.el9.noarch 27/33 2024-06-03T08:26:50.250 INFO:teuthology.orchestra.run.smithi192.stdout: Verifying : python3-portend-3.1.0-2.el9.noarch 28/33 2024-06-03T08:26:50.250 INFO:teuthology.orchestra.run.smithi192.stdout: Verifying : python3-pyOpenSSL-21.0.0-1.el9.noarch 29/33 2024-06-03T08:26:50.250 INFO:teuthology.orchestra.run.smithi192.stdout: Verifying : python3-tempora-5.0.0-2.el9.noarch 30/33 2024-06-03T08:26:50.250 INFO:teuthology.orchestra.run.smithi192.stdout: Verifying : python3-webob-1.8.7-6.el9.noarch 31/33 2024-06-03T08:26:50.250 INFO:teuthology.orchestra.run.smithi192.stdout: Verifying : python3-werkzeug-2.0.3-3.el9.1.noarch 32/33 2024-06-03T08:26:50.707 INFO:teuthology.orchestra.run.smithi088.stdout: Verifying : ceph-immutable-object-cache-2:18.2.2-1389.gf1abf795. 1/1 2024-06-03T08:26:50.708 INFO:teuthology.orchestra.run.smithi088.stdout: 2024-06-03T08:26:50.708 INFO:teuthology.orchestra.run.smithi088.stdout:Installed: 2024-06-03T08:26:50.708 INFO:teuthology.orchestra.run.smithi088.stdout: ceph-immutable-object-cache-2:18.2.2-1389.gf1abf795.el9.x86_64 2024-06-03T08:26:50.708 INFO:teuthology.orchestra.run.smithi088.stdout: 2024-06-03T08:26:50.708 INFO:teuthology.orchestra.run.smithi088.stdout:Complete! 2024-06-03T08:26:50.971 DEBUG:teuthology.orchestra.run.smithi088:> sudo yum -y install ceph-mgr 2024-06-03T08:26:51.095 INFO:teuthology.orchestra.run.smithi192.stdout: Verifying : python3-zc-lockfile-2.0-10.el9.noarch 33/33 2024-06-03T08:26:51.095 INFO:teuthology.orchestra.run.smithi192.stdout: 2024-06-03T08:26:51.095 INFO:teuthology.orchestra.run.smithi192.stdout:Installed: 2024-06-03T08:26:51.095 INFO:teuthology.orchestra.run.smithi192.stdout: ceph-2:18.2.2-1389.gf1abf795.el9.x86_64 2024-06-03T08:26:51.095 INFO:teuthology.orchestra.run.smithi192.stdout: ceph-mds-2:18.2.2-1389.gf1abf795.el9.x86_64 2024-06-03T08:26:51.095 INFO:teuthology.orchestra.run.smithi192.stdout: ceph-mgr-2:18.2.2-1389.gf1abf795.el9.x86_64 2024-06-03T08:26:51.095 INFO:teuthology.orchestra.run.smithi192.stdout: ceph-mgr-modules-core-2:18.2.2-1389.gf1abf795.el9.noarch 2024-06-03T08:26:51.095 INFO:teuthology.orchestra.run.smithi192.stdout: ceph-mon-2:18.2.2-1389.gf1abf795.el9.x86_64 2024-06-03T08:26:51.096 INFO:teuthology.orchestra.run.smithi192.stdout: ceph-osd-2:18.2.2-1389.gf1abf795.el9.x86_64 2024-06-03T08:26:51.096 INFO:teuthology.orchestra.run.smithi192.stdout: python3-bcrypt-3.2.2-1.el9.x86_64 2024-06-03T08:26:51.096 INFO:teuthology.orchestra.run.smithi192.stdout: python3-cffi-1.14.5-5.el9.x86_64 2024-06-03T08:26:51.096 INFO:teuthology.orchestra.run.smithi192.stdout: python3-cheroot-8.6.0-4.el9.noarch 2024-06-03T08:26:51.096 INFO:teuthology.orchestra.run.smithi192.stdout: python3-cherrypy-18.6.1-2.el9.noarch 2024-06-03T08:26:51.096 INFO:teuthology.orchestra.run.smithi192.stdout: python3-cryptography-36.0.1-4.el9.x86_64 2024-06-03T08:26:51.096 INFO:teuthology.orchestra.run.smithi192.stdout: python3-jaraco-8.2.1-3.el9.noarch 2024-06-03T08:26:51.096 INFO:teuthology.orchestra.run.smithi192.stdout: python3-jaraco-classes-3.2.1-5.el9.noarch 2024-06-03T08:26:51.096 INFO:teuthology.orchestra.run.smithi192.stdout: python3-jaraco-collections-3.0.0-8.el9.noarch 2024-06-03T08:26:51.096 INFO:teuthology.orchestra.run.smithi192.stdout: python3-jaraco-functools-3.5.0-2.el9.noarch 2024-06-03T08:26:51.096 INFO:teuthology.orchestra.run.smithi192.stdout: python3-jaraco-text-3.2.0-6.el9.noarch 2024-06-03T08:26:51.096 INFO:teuthology.orchestra.run.smithi192.stdout: python3-logutils-0.3.5-21.el9.noarch 2024-06-03T08:26:51.096 INFO:teuthology.orchestra.run.smithi192.stdout: python3-mako-1.1.4-6.el9.noarch 2024-06-03T08:26:51.097 INFO:teuthology.orchestra.run.smithi192.stdout: python3-markupsafe-1.1.1-12.el9.x86_64 2024-06-03T08:26:51.097 INFO:teuthology.orchestra.run.smithi192.stdout: python3-more-itertools-8.12.0-2.el9.noarch 2024-06-03T08:26:51.097 INFO:teuthology.orchestra.run.smithi192.stdout: python3-pecan-1.4.2-3.el9.noarch 2024-06-03T08:26:51.097 INFO:teuthology.orchestra.run.smithi192.stdout: python3-ply-3.11-14.el9.noarch 2024-06-03T08:26:51.097 INFO:teuthology.orchestra.run.smithi192.stdout: python3-portend-3.1.0-2.el9.noarch 2024-06-03T08:26:51.097 INFO:teuthology.orchestra.run.smithi192.stdout: python3-pyOpenSSL-21.0.0-1.el9.noarch 2024-06-03T08:26:51.097 INFO:teuthology.orchestra.run.smithi192.stdout: python3-pycparser-2.20-6.el9.noarch 2024-06-03T08:26:51.097 INFO:teuthology.orchestra.run.smithi192.stdout: python3-pytz-2021.1-5.el9.noarch 2024-06-03T08:26:51.097 INFO:teuthology.orchestra.run.smithi192.stdout: python3-requests-2.25.1-8.el9.noarch 2024-06-03T08:26:51.097 INFO:teuthology.orchestra.run.smithi192.stdout: python3-tempora-5.0.0-2.el9.noarch 2024-06-03T08:26:51.097 INFO:teuthology.orchestra.run.smithi192.stdout: python3-toml-0.10.2-6.el9.noarch 2024-06-03T08:26:51.097 INFO:teuthology.orchestra.run.smithi192.stdout: python3-urllib3-1.26.5-5.el9.noarch 2024-06-03T08:26:51.098 INFO:teuthology.orchestra.run.smithi192.stdout: python3-webob-1.8.7-6.el9.noarch 2024-06-03T08:26:51.098 INFO:teuthology.orchestra.run.smithi192.stdout: python3-werkzeug-2.0.3-3.el9.1.noarch 2024-06-03T08:26:51.098 INFO:teuthology.orchestra.run.smithi192.stdout: python3-zc-lockfile-2.0-10.el9.noarch 2024-06-03T08:26:51.098 INFO:teuthology.orchestra.run.smithi192.stdout: 2024-06-03T08:26:51.098 INFO:teuthology.orchestra.run.smithi192.stdout:Complete! 2024-06-03T08:26:51.448 DEBUG:teuthology.orchestra.run.smithi192:> sudo yum -y install ceph-base 2024-06-03T08:26:51.567 INFO:teuthology.orchestra.run.smithi088.stdout:Last metadata expiration check: 0:00:55 ago on Mon 03 Jun 2024 08:25:56 AM UTC. 2024-06-03T08:26:51.648 INFO:teuthology.orchestra.run.smithi088.stdout:Package ceph-mgr-2:18.2.2-1389.gf1abf795.el9.x86_64 is already installed. 2024-06-03T08:26:51.690 INFO:teuthology.orchestra.run.smithi088.stdout:Dependencies resolved. 2024-06-03T08:26:51.691 INFO:teuthology.orchestra.run.smithi088.stdout:Nothing to do. 2024-06-03T08:26:51.692 INFO:teuthology.orchestra.run.smithi088.stdout:Complete! 2024-06-03T08:26:51.754 DEBUG:teuthology.orchestra.run.smithi088:> sudo yum -y install ceph-mgr-dashboard 2024-06-03T08:26:52.029 INFO:teuthology.orchestra.run.smithi192.stdout:Last metadata expiration check: 0:00:52 ago on Mon 03 Jun 2024 08:26:00 AM UTC. 2024-06-03T08:26:52.110 INFO:teuthology.orchestra.run.smithi192.stdout:Package ceph-base-2:18.2.2-1389.gf1abf795.el9.x86_64 is already installed. 2024-06-03T08:26:52.152 INFO:teuthology.orchestra.run.smithi192.stdout:Dependencies resolved. 2024-06-03T08:26:52.153 INFO:teuthology.orchestra.run.smithi192.stdout:Nothing to do. 2024-06-03T08:26:52.154 INFO:teuthology.orchestra.run.smithi192.stdout:Complete! 2024-06-03T08:26:52.216 DEBUG:teuthology.orchestra.run.smithi192:> sudo yum -y install cephadm 2024-06-03T08:26:52.329 INFO:teuthology.orchestra.run.smithi088.stdout:Last metadata expiration check: 0:00:56 ago on Mon 03 Jun 2024 08:25:56 AM UTC. 2024-06-03T08:26:52.452 INFO:teuthology.orchestra.run.smithi088.stdout:Dependencies resolved. 2024-06-03T08:26:52.452 INFO:teuthology.orchestra.run.smithi088.stdout:================================================================================ 2024-06-03T08:26:52.453 INFO:teuthology.orchestra.run.smithi088.stdout: Package Arch Version Repository Size 2024-06-03T08:26:52.453 INFO:teuthology.orchestra.run.smithi088.stdout:================================================================================ 2024-06-03T08:26:52.453 INFO:teuthology.orchestra.run.smithi088.stdout:Installing: 2024-06-03T08:26:52.453 INFO:teuthology.orchestra.run.smithi088.stdout: ceph-mgr-dashboard noarch 2:18.2.2-1389.gf1abf795.el9 ceph-noarch 3.5 M 2024-06-03T08:26:52.453 INFO:teuthology.orchestra.run.smithi088.stdout:Installing dependencies: 2024-06-03T08:26:52.453 INFO:teuthology.orchestra.run.smithi088.stdout: ceph-grafana-dashboards noarch 2:18.2.2-1389.gf1abf795.el9 ceph-noarch 24 k 2024-06-03T08:26:52.453 INFO:teuthology.orchestra.run.smithi088.stdout: ceph-prometheus-alerts noarch 2:18.2.2-1389.gf1abf795.el9 ceph-noarch 16 k 2024-06-03T08:26:52.453 INFO:teuthology.orchestra.run.smithi088.stdout: python3-repoze-lru noarch 0.7-16.el9 epel 31 k 2024-06-03T08:26:52.453 INFO:teuthology.orchestra.run.smithi088.stdout: python3-routes noarch 2.5.1-5.el9 epel 188 k 2024-06-03T08:26:52.453 INFO:teuthology.orchestra.run.smithi088.stdout: 2024-06-03T08:26:52.453 INFO:teuthology.orchestra.run.smithi088.stdout:Transaction Summary 2024-06-03T08:26:52.453 INFO:teuthology.orchestra.run.smithi088.stdout:================================================================================ 2024-06-03T08:26:52.454 INFO:teuthology.orchestra.run.smithi088.stdout:Install 5 Packages 2024-06-03T08:26:52.454 INFO:teuthology.orchestra.run.smithi088.stdout: 2024-06-03T08:26:52.454 INFO:teuthology.orchestra.run.smithi088.stdout:Total download size: 3.8 M 2024-06-03T08:26:52.454 INFO:teuthology.orchestra.run.smithi088.stdout:Installed size: 87 M 2024-06-03T08:26:52.454 INFO:teuthology.orchestra.run.smithi088.stdout:Downloading Packages: 2024-06-03T08:26:52.792 INFO:teuthology.orchestra.run.smithi192.stdout:Last metadata expiration check: 0:00:52 ago on Mon 03 Jun 2024 08:26:00 AM UTC. 2024-06-03T08:26:52.853 INFO:teuthology.orchestra.run.smithi088.stdout:(1/5): ceph-grafana-dashboards-18.2.2-1389.gf1a 93 kB/s | 24 kB 00:00 2024-06-03T08:26:52.878 INFO:teuthology.orchestra.run.smithi088.stdout:(2/5): ceph-prometheus-alerts-18.2.2-1389.gf1ab 57 kB/s | 16 kB 00:00 2024-06-03T08:26:52.916 INFO:teuthology.orchestra.run.smithi192.stdout:Dependencies resolved. 2024-06-03T08:26:52.916 INFO:teuthology.orchestra.run.smithi192.stdout:================================================================================ 2024-06-03T08:26:52.916 INFO:teuthology.orchestra.run.smithi192.stdout: Package Arch Version Repository Size 2024-06-03T08:26:52.916 INFO:teuthology.orchestra.run.smithi192.stdout:================================================================================ 2024-06-03T08:26:52.916 INFO:teuthology.orchestra.run.smithi192.stdout:Installing: 2024-06-03T08:26:52.916 INFO:teuthology.orchestra.run.smithi192.stdout: cephadm noarch 2:18.2.2-1389.gf1abf795.el9 ceph-noarch 224 k 2024-06-03T08:26:52.916 INFO:teuthology.orchestra.run.smithi192.stdout: 2024-06-03T08:26:52.917 INFO:teuthology.orchestra.run.smithi192.stdout:Transaction Summary 2024-06-03T08:26:52.917 INFO:teuthology.orchestra.run.smithi192.stdout:================================================================================ 2024-06-03T08:26:52.917 INFO:teuthology.orchestra.run.smithi192.stdout:Install 1 Package 2024-06-03T08:26:52.917 INFO:teuthology.orchestra.run.smithi192.stdout: 2024-06-03T08:26:52.917 INFO:teuthology.orchestra.run.smithi192.stdout:Total download size: 224 k 2024-06-03T08:26:52.917 INFO:teuthology.orchestra.run.smithi192.stdout:Installed size: 216 k 2024-06-03T08:26:52.917 INFO:teuthology.orchestra.run.smithi192.stdout:Downloading Packages: 2024-06-03T08:26:53.145 INFO:teuthology.orchestra.run.smithi192.stdout:cephadm-18.2.2-1389.gf1abf795.el9.noarch.rpm 988 kB/s | 224 kB 00:00 2024-06-03T08:26:53.145 INFO:teuthology.orchestra.run.smithi192.stdout:-------------------------------------------------------------------------------- 2024-06-03T08:26:53.145 INFO:teuthology.orchestra.run.smithi192.stdout:Total 981 kB/s | 224 kB 00:00 2024-06-03T08:26:53.146 INFO:teuthology.orchestra.run.smithi192.stdout:Running transaction check 2024-06-03T08:26:53.150 INFO:teuthology.orchestra.run.smithi192.stdout:Transaction check succeeded. 2024-06-03T08:26:53.151 INFO:teuthology.orchestra.run.smithi192.stdout:Running transaction test 2024-06-03T08:26:53.154 INFO:teuthology.orchestra.run.smithi088.stdout:(3/5): python3-repoze-lru-0.7-16.el9.noarch.rpm 102 kB/s | 31 kB 00:00 2024-06-03T08:26:53.154 INFO:teuthology.orchestra.run.smithi192.stdout:Transaction test succeeded. 2024-06-03T08:26:53.154 INFO:teuthology.orchestra.run.smithi192.stdout:Running transaction 2024-06-03T08:26:53.229 INFO:teuthology.orchestra.run.smithi088.stdout:(4/5): python3-routes-2.5.1-5.el9.noarch.rpm 538 kB/s | 188 kB 00:00 2024-06-03T08:26:53.235 INFO:teuthology.orchestra.run.smithi192.stdout: Preparing : 1/1 2024-06-03T08:26:53.554 INFO:teuthology.orchestra.run.smithi088.stdout:(5/5): ceph-mgr-dashboard-18.2.2-1389.gf1abf795 3.7 MB/s | 3.5 MB 00:00 2024-06-03T08:26:53.557 INFO:teuthology.orchestra.run.smithi088.stdout:-------------------------------------------------------------------------------- 2024-06-03T08:26:53.558 INFO:teuthology.orchestra.run.smithi088.stdout:Total 3.4 MB/s | 3.8 MB 00:01 2024-06-03T08:26:53.558 INFO:teuthology.orchestra.run.smithi088.stdout:Running transaction check 2024-06-03T08:26:53.565 INFO:teuthology.orchestra.run.smithi088.stdout:Transaction check succeeded. 2024-06-03T08:26:53.565 INFO:teuthology.orchestra.run.smithi088.stdout:Running transaction test 2024-06-03T08:26:53.624 INFO:teuthology.orchestra.run.smithi088.stdout:Transaction test succeeded. 2024-06-03T08:26:53.625 INFO:teuthology.orchestra.run.smithi088.stdout:Running transaction 2024-06-03T08:26:53.641 INFO:teuthology.orchestra.run.smithi192.stdout: Running scriptlet: cephadm-2:18.2.2-1389.gf1abf795.el9.noarch 1/1 2024-06-03T08:26:53.716 INFO:teuthology.orchestra.run.smithi192.stdout: Installing : cephadm-2:18.2.2-1389.gf1abf795.el9.noarch 1/1 2024-06-03T08:26:53.948 INFO:teuthology.orchestra.run.smithi088.stdout: Preparing : 1/1 2024-06-03T08:26:54.035 INFO:teuthology.orchestra.run.smithi088.stdout: Installing : python3-repoze-lru-0.7-16.el9.noarch 1/5 2024-06-03T08:26:54.067 INFO:teuthology.orchestra.run.smithi088.stdout: Installing : python3-routes-2.5.1-5.el9.noarch 2/5 2024-06-03T08:26:54.096 INFO:teuthology.orchestra.run.smithi088.stdout: Installing : ceph-prometheus-alerts-2:18.2.2-1389.gf1abf795.el9.n 3/5 2024-06-03T08:26:54.112 INFO:teuthology.orchestra.run.smithi192.stdout: Running scriptlet: cephadm-2:18.2.2-1389.gf1abf795.el9.noarch 1/1 2024-06-03T08:26:54.398 INFO:teuthology.orchestra.run.smithi192.stdout: Verifying : cephadm-2:18.2.2-1389.gf1abf795.el9.noarch 1/1 2024-06-03T08:26:54.399 INFO:teuthology.orchestra.run.smithi192.stdout: 2024-06-03T08:26:54.399 INFO:teuthology.orchestra.run.smithi192.stdout:Installed: 2024-06-03T08:26:54.399 INFO:teuthology.orchestra.run.smithi192.stdout: cephadm-2:18.2.2-1389.gf1abf795.el9.noarch 2024-06-03T08:26:54.399 INFO:teuthology.orchestra.run.smithi192.stdout: 2024-06-03T08:26:54.399 INFO:teuthology.orchestra.run.smithi192.stdout:Complete! 2024-06-03T08:26:54.585 DEBUG:teuthology.orchestra.run.smithi192:> sudo yum -y install ceph-immutable-object-cache 2024-06-03T08:26:55.138 INFO:teuthology.orchestra.run.smithi088.stdout: Installing : ceph-grafana-dashboards-2:18.2.2-1389.gf1abf795.el9. 4/5 2024-06-03T08:26:55.164 INFO:teuthology.orchestra.run.smithi192.stdout:Last metadata expiration check: 0:00:55 ago on Mon 03 Jun 2024 08:26:00 AM UTC. 2024-06-03T08:26:55.174 INFO:teuthology.orchestra.run.smithi088.stdout: Installing : ceph-mgr-dashboard-2:18.2.2-1389.gf1abf795.el9.noarc 5/5 2024-06-03T08:26:55.288 INFO:teuthology.orchestra.run.smithi192.stdout:Dependencies resolved. 2024-06-03T08:26:55.289 INFO:teuthology.orchestra.run.smithi192.stdout:================================================================================ 2024-06-03T08:26:55.289 INFO:teuthology.orchestra.run.smithi192.stdout: Package Arch Version Repo Size 2024-06-03T08:26:55.289 INFO:teuthology.orchestra.run.smithi192.stdout:================================================================================ 2024-06-03T08:26:55.289 INFO:teuthology.orchestra.run.smithi192.stdout:Installing: 2024-06-03T08:26:55.289 INFO:teuthology.orchestra.run.smithi192.stdout: ceph-immutable-object-cache x86_64 2:18.2.2-1389.gf1abf795.el9 ceph 143 k 2024-06-03T08:26:55.289 INFO:teuthology.orchestra.run.smithi192.stdout: 2024-06-03T08:26:55.289 INFO:teuthology.orchestra.run.smithi192.stdout:Transaction Summary 2024-06-03T08:26:55.289 INFO:teuthology.orchestra.run.smithi192.stdout:================================================================================ 2024-06-03T08:26:55.290 INFO:teuthology.orchestra.run.smithi192.stdout:Install 1 Package 2024-06-03T08:26:55.290 INFO:teuthology.orchestra.run.smithi192.stdout: 2024-06-03T08:26:55.290 INFO:teuthology.orchestra.run.smithi192.stdout:Total download size: 143 k 2024-06-03T08:26:55.290 INFO:teuthology.orchestra.run.smithi192.stdout:Installed size: 395 k 2024-06-03T08:26:55.290 INFO:teuthology.orchestra.run.smithi192.stdout:Downloading Packages: 2024-06-03T08:26:55.510 INFO:teuthology.orchestra.run.smithi192.stdout:ceph-immutable-object-cache-18.2.2-1389.gf1abf7 650 kB/s | 143 kB 00:00 2024-06-03T08:26:55.511 INFO:teuthology.orchestra.run.smithi192.stdout:-------------------------------------------------------------------------------- 2024-06-03T08:26:55.511 INFO:teuthology.orchestra.run.smithi192.stdout:Total 645 kB/s | 143 kB 00:00 2024-06-03T08:26:55.511 INFO:teuthology.orchestra.run.smithi192.stdout:Running transaction check 2024-06-03T08:26:55.518 INFO:teuthology.orchestra.run.smithi192.stdout:Transaction check succeeded. 2024-06-03T08:26:55.519 INFO:teuthology.orchestra.run.smithi192.stdout:Running transaction test 2024-06-03T08:26:55.575 INFO:teuthology.orchestra.run.smithi192.stdout:Transaction test succeeded. 2024-06-03T08:26:55.575 INFO:teuthology.orchestra.run.smithi192.stdout:Running transaction 2024-06-03T08:26:55.713 INFO:teuthology.orchestra.run.smithi192.stdout: Preparing : 1/1 2024-06-03T08:26:55.769 INFO:teuthology.orchestra.run.smithi192.stdout: Installing : ceph-immutable-object-cache-2:18.2.2-1389.gf1abf795. 1/1 2024-06-03T08:26:55.794 INFO:teuthology.orchestra.run.smithi192.stdout: Running scriptlet: ceph-immutable-object-cache-2:18.2.2-1389.gf1abf795. 1/1 2024-06-03T08:26:55.794 INFO:teuthology.orchestra.run.smithi192.stdout:Glob pattern passed to enable, but globs are not supported for this. 2024-06-03T08:26:55.794 INFO:teuthology.orchestra.run.smithi192.stdout:Invalid unit name "ceph-immutable-object-cache@*.service" escaped as "ceph-immutable-object-cache@\x2a.service". 2024-06-03T08:26:55.795 INFO:teuthology.orchestra.run.smithi192.stdout: 2024-06-03T08:26:56.210 INFO:teuthology.orchestra.run.smithi088.stdout: Running scriptlet: ceph-mgr-dashboard-2:18.2.2-1389.gf1abf795.el9.noarc 5/5 2024-06-03T08:26:56.210 INFO:teuthology.orchestra.run.smithi088.stdout: Verifying : ceph-grafana-dashboards-2:18.2.2-1389.gf1abf795.el9. 1/5 2024-06-03T08:26:56.210 INFO:teuthology.orchestra.run.smithi088.stdout: Verifying : ceph-mgr-dashboard-2:18.2.2-1389.gf1abf795.el9.noarc 2/5 2024-06-03T08:26:56.210 INFO:teuthology.orchestra.run.smithi088.stdout: Verifying : ceph-prometheus-alerts-2:18.2.2-1389.gf1abf795.el9.n 3/5 2024-06-03T08:26:56.210 INFO:teuthology.orchestra.run.smithi088.stdout: Verifying : python3-repoze-lru-0.7-16.el9.noarch 4/5 2024-06-03T08:26:56.461 INFO:teuthology.orchestra.run.smithi088.stdout: Verifying : python3-routes-2.5.1-5.el9.noarch 5/5 2024-06-03T08:26:56.462 INFO:teuthology.orchestra.run.smithi088.stdout: 2024-06-03T08:26:56.462 INFO:teuthology.orchestra.run.smithi088.stdout:Installed: 2024-06-03T08:26:56.462 INFO:teuthology.orchestra.run.smithi088.stdout: ceph-grafana-dashboards-2:18.2.2-1389.gf1abf795.el9.noarch 2024-06-03T08:26:56.462 INFO:teuthology.orchestra.run.smithi088.stdout: ceph-mgr-dashboard-2:18.2.2-1389.gf1abf795.el9.noarch 2024-06-03T08:26:56.462 INFO:teuthology.orchestra.run.smithi088.stdout: ceph-prometheus-alerts-2:18.2.2-1389.gf1abf795.el9.noarch 2024-06-03T08:26:56.462 INFO:teuthology.orchestra.run.smithi088.stdout: python3-repoze-lru-0.7-16.el9.noarch 2024-06-03T08:26:56.462 INFO:teuthology.orchestra.run.smithi088.stdout: python3-routes-2.5.1-5.el9.noarch 2024-06-03T08:26:56.462 INFO:teuthology.orchestra.run.smithi088.stdout: 2024-06-03T08:26:56.462 INFO:teuthology.orchestra.run.smithi088.stdout:Complete! 2024-06-03T08:26:56.656 DEBUG:teuthology.orchestra.run.smithi088:> sudo yum -y install ceph-mgr-diskprediction-local 2024-06-03T08:26:56.748 INFO:teuthology.orchestra.run.smithi192.stdout: Verifying : ceph-immutable-object-cache-2:18.2.2-1389.gf1abf795. 1/1 2024-06-03T08:26:56.748 INFO:teuthology.orchestra.run.smithi192.stdout: 2024-06-03T08:26:56.748 INFO:teuthology.orchestra.run.smithi192.stdout:Installed: 2024-06-03T08:26:56.748 INFO:teuthology.orchestra.run.smithi192.stdout: ceph-immutable-object-cache-2:18.2.2-1389.gf1abf795.el9.x86_64 2024-06-03T08:26:56.748 INFO:teuthology.orchestra.run.smithi192.stdout: 2024-06-03T08:26:56.748 INFO:teuthology.orchestra.run.smithi192.stdout:Complete! 2024-06-03T08:26:56.989 DEBUG:teuthology.orchestra.run.smithi192:> sudo yum -y install ceph-mgr 2024-06-03T08:26:57.223 INFO:teuthology.orchestra.run.smithi088.stdout:Last metadata expiration check: 0:01:01 ago on Mon 03 Jun 2024 08:25:56 AM UTC. 2024-06-03T08:26:57.348 INFO:teuthology.orchestra.run.smithi088.stdout:Dependencies resolved. 2024-06-03T08:26:57.349 INFO:teuthology.orchestra.run.smithi088.stdout:======================================================================================= 2024-06-03T08:26:57.349 INFO:teuthology.orchestra.run.smithi088.stdout: Package Arch Version Repository Size 2024-06-03T08:26:57.349 INFO:teuthology.orchestra.run.smithi088.stdout:======================================================================================= 2024-06-03T08:26:57.349 INFO:teuthology.orchestra.run.smithi088.stdout:Installing: 2024-06-03T08:26:57.349 INFO:teuthology.orchestra.run.smithi088.stdout: ceph-mgr-diskprediction-local noarch 2:18.2.2-1389.gf1abf795.el9 ceph-noarch 7.4 M 2024-06-03T08:26:57.349 INFO:teuthology.orchestra.run.smithi088.stdout:Installing dependencies: 2024-06-03T08:26:57.349 INFO:teuthology.orchestra.run.smithi088.stdout: flexiblas x86_64 3.0.4-8.el9 appstream 33 k 2024-06-03T08:26:57.349 INFO:teuthology.orchestra.run.smithi088.stdout: flexiblas-netlib x86_64 3.0.4-8.el9 appstream 3.0 M 2024-06-03T08:26:57.349 INFO:teuthology.orchestra.run.smithi088.stdout: flexiblas-openblas-openmp x86_64 3.0.4-8.el9 appstream 18 k 2024-06-03T08:26:57.349 INFO:teuthology.orchestra.run.smithi088.stdout: libgfortran x86_64 11.4.1-3.el9 baseos 803 k 2024-06-03T08:26:57.349 INFO:teuthology.orchestra.run.smithi088.stdout: libquadmath x86_64 11.4.1-3.el9 baseos 195 k 2024-06-03T08:26:57.350 INFO:teuthology.orchestra.run.smithi088.stdout: openblas x86_64 0.3.21-2.el9 appstream 35 k 2024-06-03T08:26:57.350 INFO:teuthology.orchestra.run.smithi088.stdout: openblas-openmp x86_64 0.3.21-2.el9 appstream 4.7 M 2024-06-03T08:26:57.350 INFO:teuthology.orchestra.run.smithi088.stdout: python3-devel x86_64 3.9.18-3.el9 appstream 244 k 2024-06-03T08:26:57.350 INFO:teuthology.orchestra.run.smithi088.stdout: python3-numpy x86_64 1:1.20.1-5.el9 appstream 5.1 M 2024-06-03T08:26:57.350 INFO:teuthology.orchestra.run.smithi088.stdout: python3-numpy-f2py x86_64 1:1.20.1-5.el9 appstream 250 k 2024-06-03T08:26:57.350 INFO:teuthology.orchestra.run.smithi088.stdout: python3-scipy x86_64 1.6.2-8.el9 appstream 16 M 2024-06-03T08:26:57.350 INFO:teuthology.orchestra.run.smithi088.stdout: 2024-06-03T08:26:57.350 INFO:teuthology.orchestra.run.smithi088.stdout:Transaction Summary 2024-06-03T08:26:57.350 INFO:teuthology.orchestra.run.smithi088.stdout:======================================================================================= 2024-06-03T08:26:57.350 INFO:teuthology.orchestra.run.smithi088.stdout:Install 12 Packages 2024-06-03T08:26:57.350 INFO:teuthology.orchestra.run.smithi088.stdout: 2024-06-03T08:26:57.351 INFO:teuthology.orchestra.run.smithi088.stdout:Total download size: 38 M 2024-06-03T08:26:57.351 INFO:teuthology.orchestra.run.smithi088.stdout:Installed size: 209 M 2024-06-03T08:26:57.351 INFO:teuthology.orchestra.run.smithi088.stdout:Downloading Packages: 2024-06-03T08:26:57.594 INFO:teuthology.orchestra.run.smithi192.stdout:Last metadata expiration check: 0:00:57 ago on Mon 03 Jun 2024 08:26:00 AM UTC. 2024-06-03T08:26:57.682 INFO:teuthology.orchestra.run.smithi192.stdout:Package ceph-mgr-2:18.2.2-1389.gf1abf795.el9.x86_64 is already installed. 2024-06-03T08:26:57.722 INFO:teuthology.orchestra.run.smithi192.stdout:Dependencies resolved. 2024-06-03T08:26:57.723 INFO:teuthology.orchestra.run.smithi192.stdout:Nothing to do. 2024-06-03T08:26:57.724 INFO:teuthology.orchestra.run.smithi192.stdout:Complete! 2024-06-03T08:26:57.789 DEBUG:teuthology.orchestra.run.smithi192:> sudo yum -y install ceph-mgr-dashboard 2024-06-03T08:26:58.370 INFO:teuthology.orchestra.run.smithi192.stdout:Last metadata expiration check: 0:00:58 ago on Mon 03 Jun 2024 08:26:00 AM UTC. 2024-06-03T08:26:58.453 INFO:teuthology.orchestra.run.smithi088.stdout:(1/12): libquadmath-11.4.1-3.el9.x86_64.rpm 284 kB/s | 195 kB 00:00 2024-06-03T08:26:58.493 INFO:teuthology.orchestra.run.smithi192.stdout:Dependencies resolved. 2024-06-03T08:26:58.493 INFO:teuthology.orchestra.run.smithi192.stdout:================================================================================ 2024-06-03T08:26:58.494 INFO:teuthology.orchestra.run.smithi192.stdout: Package Arch Version Repository Size 2024-06-03T08:26:58.494 INFO:teuthology.orchestra.run.smithi192.stdout:================================================================================ 2024-06-03T08:26:58.494 INFO:teuthology.orchestra.run.smithi192.stdout:Installing: 2024-06-03T08:26:58.494 INFO:teuthology.orchestra.run.smithi192.stdout: ceph-mgr-dashboard noarch 2:18.2.2-1389.gf1abf795.el9 ceph-noarch 3.5 M 2024-06-03T08:26:58.494 INFO:teuthology.orchestra.run.smithi192.stdout:Installing dependencies: 2024-06-03T08:26:58.494 INFO:teuthology.orchestra.run.smithi192.stdout: ceph-grafana-dashboards noarch 2:18.2.2-1389.gf1abf795.el9 ceph-noarch 24 k 2024-06-03T08:26:58.494 INFO:teuthology.orchestra.run.smithi192.stdout: ceph-prometheus-alerts noarch 2:18.2.2-1389.gf1abf795.el9 ceph-noarch 16 k 2024-06-03T08:26:58.494 INFO:teuthology.orchestra.run.smithi192.stdout: python3-repoze-lru noarch 0.7-16.el9 epel 31 k 2024-06-03T08:26:58.494 INFO:teuthology.orchestra.run.smithi192.stdout: python3-routes noarch 2.5.1-5.el9 epel 188 k 2024-06-03T08:26:58.494 INFO:teuthology.orchestra.run.smithi192.stdout: 2024-06-03T08:26:58.494 INFO:teuthology.orchestra.run.smithi192.stdout:Transaction Summary 2024-06-03T08:26:58.495 INFO:teuthology.orchestra.run.smithi192.stdout:================================================================================ 2024-06-03T08:26:58.495 INFO:teuthology.orchestra.run.smithi192.stdout:Install 5 Packages 2024-06-03T08:26:58.495 INFO:teuthology.orchestra.run.smithi192.stdout: 2024-06-03T08:26:58.495 INFO:teuthology.orchestra.run.smithi192.stdout:Total download size: 3.8 M 2024-06-03T08:26:58.495 INFO:teuthology.orchestra.run.smithi192.stdout:Installed size: 87 M 2024-06-03T08:26:58.496 INFO:teuthology.orchestra.run.smithi192.stdout:Downloading Packages: 2024-06-03T08:26:58.545 INFO:teuthology.orchestra.run.smithi088.stdout:(2/12): flexiblas-3.0.4-8.el9.x86_64.rpm 357 kB/s | 33 kB 00:00 2024-06-03T08:26:58.786 INFO:teuthology.orchestra.run.smithi088.stdout:(3/12): flexiblas-netlib-3.0.4-8.el9.x86_64.rpm 12 MB/s | 3.0 MB 00:00 2024-06-03T08:26:58.790 INFO:teuthology.orchestra.run.smithi192.stdout:(1/5): ceph-prometheus-alerts-18.2.2-1389.gf1ab 65 kB/s | 16 kB 00:00 2024-06-03T08:26:58.815 INFO:teuthology.orchestra.run.smithi192.stdout:(2/5): ceph-grafana-dashboards-18.2.2-1389.gf1a 87 kB/s | 24 kB 00:00 2024-06-03T08:26:58.820 INFO:teuthology.orchestra.run.smithi088.stdout:(4/12): flexiblas-openblas-openmp-3.0.4-8.el9.x 548 kB/s | 18 kB 00:00 2024-06-03T08:26:58.854 INFO:teuthology.orchestra.run.smithi088.stdout:(5/12): openblas-0.3.21-2.el9.x86_64.rpm 1.0 MB/s | 35 kB 00:00 2024-06-03T08:26:59.074 INFO:teuthology.orchestra.run.smithi192.stdout:(3/5): python3-repoze-lru-0.7-16.el9.noarch.rpm 108 kB/s | 31 kB 00:00 2024-06-03T08:26:59.129 INFO:teuthology.orchestra.run.smithi088.stdout:(6/12): ceph-mgr-diskprediction-local-18.2.2-13 5.4 MB/s | 7.4 MB 00:01 2024-06-03T08:26:59.183 INFO:teuthology.orchestra.run.smithi192.stdout:(4/5): ceph-mgr-dashboard-18.2.2-1389.gf1abf795 5.5 MB/s | 3.5 MB 00:00 2024-06-03T08:26:59.196 INFO:teuthology.orchestra.run.smithi088.stdout:(7/12): openblas-openmp-0.3.21-2.el9.x86_64.rpm 14 MB/s | 4.7 MB 00:00 2024-06-03T08:26:59.267 INFO:teuthology.orchestra.run.smithi192.stdout:(5/5): python3-routes-2.5.1-5.el9.noarch.rpm 418 kB/s | 188 kB 00:00 2024-06-03T08:26:59.270 INFO:teuthology.orchestra.run.smithi192.stdout:-------------------------------------------------------------------------------- 2024-06-03T08:26:59.270 INFO:teuthology.orchestra.run.smithi192.stdout:Total 4.9 MB/s | 3.8 MB 00:00 2024-06-03T08:26:59.270 INFO:teuthology.orchestra.run.smithi192.stdout:Running transaction check 2024-06-03T08:26:59.277 INFO:teuthology.orchestra.run.smithi192.stdout:Transaction check succeeded. 2024-06-03T08:26:59.278 INFO:teuthology.orchestra.run.smithi192.stdout:Running transaction test 2024-06-03T08:26:59.321 INFO:teuthology.orchestra.run.smithi088.stdout:(8/12): python3-devel-3.9.18-3.el9.x86_64.rpm 1.2 MB/s | 244 kB 00:00 2024-06-03T08:26:59.340 INFO:teuthology.orchestra.run.smithi192.stdout:Transaction test succeeded. 2024-06-03T08:26:59.340 INFO:teuthology.orchestra.run.smithi192.stdout:Running transaction 2024-06-03T08:26:59.396 INFO:teuthology.orchestra.run.smithi088.stdout:(9/12): python3-numpy-1.20.1-5.el9.x86_64.rpm 25 MB/s | 5.1 MB 00:00 2024-06-03T08:26:59.422 INFO:teuthology.orchestra.run.smithi088.stdout:(10/12): libgfortran-11.4.1-3.el9.x86_64.rpm 485 kB/s | 803 kB 00:01 2024-06-03T08:26:59.447 INFO:teuthology.orchestra.run.smithi088.stdout:(11/12): python3-numpy-f2py-1.20.1-5.el9.x86_64 2.0 MB/s | 250 kB 00:00 2024-06-03T08:26:59.657 INFO:teuthology.orchestra.run.smithi192.stdout: Preparing : 1/1 2024-06-03T08:26:59.739 INFO:teuthology.orchestra.run.smithi192.stdout: Installing : python3-repoze-lru-0.7-16.el9.noarch 1/5 2024-06-03T08:26:59.772 INFO:teuthology.orchestra.run.smithi192.stdout: Installing : python3-routes-2.5.1-5.el9.noarch 2/5 2024-06-03T08:26:59.800 INFO:teuthology.orchestra.run.smithi192.stdout: Installing : ceph-prometheus-alerts-2:18.2.2-1389.gf1abf795.el9.n 3/5 2024-06-03T08:26:59.847 INFO:teuthology.orchestra.run.smithi088.stdout:(12/12): python3-scipy-1.6.2-8.el9.x86_64.rpm 36 MB/s | 16 MB 00:00 2024-06-03T08:26:59.848 INFO:teuthology.orchestra.run.smithi088.stdout:-------------------------------------------------------------------------------- 2024-06-03T08:26:59.849 INFO:teuthology.orchestra.run.smithi088.stdout:Total 15 MB/s | 38 MB 00:02 2024-06-03T08:27:00.215 INFO:teuthology.orchestra.run.smithi088.stdout:Running transaction check 2024-06-03T08:27:00.242 INFO:teuthology.orchestra.run.smithi088.stdout:Transaction check succeeded. 2024-06-03T08:27:00.242 INFO:teuthology.orchestra.run.smithi088.stdout:Running transaction test 2024-06-03T08:27:00.743 INFO:teuthology.orchestra.run.smithi088.stdout:Transaction test succeeded. 2024-06-03T08:27:00.744 INFO:teuthology.orchestra.run.smithi088.stdout:Running transaction 2024-06-03T08:27:00.840 INFO:teuthology.orchestra.run.smithi192.stdout: Installing : ceph-grafana-dashboards-2:18.2.2-1389.gf1abf795.el9. 4/5 2024-06-03T08:27:00.880 INFO:teuthology.orchestra.run.smithi192.stdout: Installing : ceph-mgr-dashboard-2:18.2.2-1389.gf1abf795.el9.noarc 5/5 2024-06-03T08:27:01.503 INFO:teuthology.orchestra.run.smithi088.stdout: Preparing : 1/1 2024-06-03T08:27:01.577 INFO:teuthology.orchestra.run.smithi088.stdout: Installing : flexiblas-3.0.4-8.el9.x86_64 1/12 2024-06-03T08:27:01.620 INFO:teuthology.orchestra.run.smithi088.stdout: Installing : libquadmath-11.4.1-3.el9.x86_64 2/12 2024-06-03T08:27:01.720 INFO:teuthology.orchestra.run.smithi088.stdout: Installing : libgfortran-11.4.1-3.el9.x86_64 3/12 2024-06-03T08:27:01.763 INFO:teuthology.orchestra.run.smithi088.stdout: Installing : python3-devel-3.9.18-3.el9.x86_64 4/12 2024-06-03T08:27:01.952 INFO:teuthology.orchestra.run.smithi088.stdout: Installing : openblas-0.3.21-2.el9.x86_64 5/12 2024-06-03T08:27:01.958 INFO:teuthology.orchestra.run.smithi192.stdout: Running scriptlet: ceph-mgr-dashboard-2:18.2.2-1389.gf1abf795.el9.noarc 5/5 2024-06-03T08:27:01.958 INFO:teuthology.orchestra.run.smithi192.stdout: Verifying : ceph-grafana-dashboards-2:18.2.2-1389.gf1abf795.el9. 1/5 2024-06-03T08:27:01.958 INFO:teuthology.orchestra.run.smithi192.stdout: Verifying : ceph-mgr-dashboard-2:18.2.2-1389.gf1abf795.el9.noarc 2/5 2024-06-03T08:27:01.958 INFO:teuthology.orchestra.run.smithi192.stdout: Verifying : ceph-prometheus-alerts-2:18.2.2-1389.gf1abf795.el9.n 3/5 2024-06-03T08:27:01.958 INFO:teuthology.orchestra.run.smithi192.stdout: Verifying : python3-repoze-lru-0.7-16.el9.noarch 4/5 2024-06-03T08:27:01.995 INFO:teuthology.orchestra.run.smithi088.stdout: Installing : openblas-openmp-0.3.21-2.el9.x86_64 6/12 2024-06-03T08:27:02.079 INFO:teuthology.orchestra.run.smithi088.stdout: Installing : flexiblas-openblas-openmp-3.0.4-8.el9.x86_64 7/12 2024-06-03T08:27:02.213 INFO:teuthology.orchestra.run.smithi192.stdout: Verifying : python3-routes-2.5.1-5.el9.noarch 5/5 2024-06-03T08:27:02.213 INFO:teuthology.orchestra.run.smithi192.stdout: 2024-06-03T08:27:02.213 INFO:teuthology.orchestra.run.smithi192.stdout:Installed: 2024-06-03T08:27:02.213 INFO:teuthology.orchestra.run.smithi192.stdout: ceph-grafana-dashboards-2:18.2.2-1389.gf1abf795.el9.noarch 2024-06-03T08:27:02.213 INFO:teuthology.orchestra.run.smithi192.stdout: ceph-mgr-dashboard-2:18.2.2-1389.gf1abf795.el9.noarch 2024-06-03T08:27:02.213 INFO:teuthology.orchestra.run.smithi192.stdout: ceph-prometheus-alerts-2:18.2.2-1389.gf1abf795.el9.noarch 2024-06-03T08:27:02.213 INFO:teuthology.orchestra.run.smithi192.stdout: python3-repoze-lru-0.7-16.el9.noarch 2024-06-03T08:27:02.214 INFO:teuthology.orchestra.run.smithi192.stdout: python3-routes-2.5.1-5.el9.noarch 2024-06-03T08:27:02.214 INFO:teuthology.orchestra.run.smithi192.stdout: 2024-06-03T08:27:02.214 INFO:teuthology.orchestra.run.smithi192.stdout:Complete! 2024-06-03T08:27:02.429 DEBUG:teuthology.orchestra.run.smithi192:> sudo yum -y install ceph-mgr-diskprediction-local 2024-06-03T08:27:02.747 INFO:teuthology.orchestra.run.smithi088.stdout: Installing : flexiblas-netlib-3.0.4-8.el9.x86_64 8/12 2024-06-03T08:27:02.867 INFO:teuthology.orchestra.run.smithi088.stdout: Installing : python3-numpy-1:1.20.1-5.el9.x86_64 9/12 2024-06-03T08:27:02.999 INFO:teuthology.orchestra.run.smithi192.stdout:Last metadata expiration check: 0:01:02 ago on Mon 03 Jun 2024 08:26:00 AM UTC. 2024-06-03T08:27:03.121 INFO:teuthology.orchestra.run.smithi192.stdout:Dependencies resolved. 2024-06-03T08:27:03.122 INFO:teuthology.orchestra.run.smithi192.stdout:======================================================================================= 2024-06-03T08:27:03.122 INFO:teuthology.orchestra.run.smithi192.stdout: Package Arch Version Repository Size 2024-06-03T08:27:03.122 INFO:teuthology.orchestra.run.smithi192.stdout:======================================================================================= 2024-06-03T08:27:03.123 INFO:teuthology.orchestra.run.smithi192.stdout:Installing: 2024-06-03T08:27:03.123 INFO:teuthology.orchestra.run.smithi192.stdout: ceph-mgr-diskprediction-local noarch 2:18.2.2-1389.gf1abf795.el9 ceph-noarch 7.4 M 2024-06-03T08:27:03.123 INFO:teuthology.orchestra.run.smithi192.stdout:Installing dependencies: 2024-06-03T08:27:03.123 INFO:teuthology.orchestra.run.smithi192.stdout: flexiblas x86_64 3.0.4-8.el9 appstream 33 k 2024-06-03T08:27:03.123 INFO:teuthology.orchestra.run.smithi192.stdout: flexiblas-netlib x86_64 3.0.4-8.el9 appstream 3.0 M 2024-06-03T08:27:03.123 INFO:teuthology.orchestra.run.smithi192.stdout: flexiblas-openblas-openmp x86_64 3.0.4-8.el9 appstream 18 k 2024-06-03T08:27:03.123 INFO:teuthology.orchestra.run.smithi192.stdout: libgfortran x86_64 11.4.1-3.el9 baseos 803 k 2024-06-03T08:27:03.123 INFO:teuthology.orchestra.run.smithi192.stdout: libquadmath x86_64 11.4.1-3.el9 baseos 195 k 2024-06-03T08:27:03.123 INFO:teuthology.orchestra.run.smithi192.stdout: openblas x86_64 0.3.21-2.el9 appstream 35 k 2024-06-03T08:27:03.123 INFO:teuthology.orchestra.run.smithi192.stdout: openblas-openmp x86_64 0.3.21-2.el9 appstream 4.7 M 2024-06-03T08:27:03.123 INFO:teuthology.orchestra.run.smithi192.stdout: python3-devel x86_64 3.9.18-3.el9 appstream 244 k 2024-06-03T08:27:03.123 INFO:teuthology.orchestra.run.smithi192.stdout: python3-numpy x86_64 1:1.20.1-5.el9 appstream 5.1 M 2024-06-03T08:27:03.123 INFO:teuthology.orchestra.run.smithi192.stdout: python3-numpy-f2py x86_64 1:1.20.1-5.el9 appstream 250 k 2024-06-03T08:27:03.123 INFO:teuthology.orchestra.run.smithi192.stdout: python3-scipy x86_64 1.6.2-8.el9 appstream 16 M 2024-06-03T08:27:03.124 INFO:teuthology.orchestra.run.smithi192.stdout: 2024-06-03T08:27:03.124 INFO:teuthology.orchestra.run.smithi192.stdout:Transaction Summary 2024-06-03T08:27:03.124 INFO:teuthology.orchestra.run.smithi192.stdout:======================================================================================= 2024-06-03T08:27:03.124 INFO:teuthology.orchestra.run.smithi192.stdout:Install 12 Packages 2024-06-03T08:27:03.124 INFO:teuthology.orchestra.run.smithi192.stdout: 2024-06-03T08:27:03.124 INFO:teuthology.orchestra.run.smithi192.stdout:Total download size: 38 M 2024-06-03T08:27:03.124 INFO:teuthology.orchestra.run.smithi192.stdout:Installed size: 209 M 2024-06-03T08:27:03.125 INFO:teuthology.orchestra.run.smithi192.stdout:Downloading Packages: 2024-06-03T08:27:03.383 INFO:teuthology.orchestra.run.smithi192.stdout:(1/12): libquadmath-11.4.1-3.el9.x86_64.rpm 1.3 MB/s | 195 kB 00:00 2024-06-03T08:27:03.416 INFO:teuthology.orchestra.run.smithi192.stdout:(2/12): libgfortran-11.4.1-3.el9.x86_64.rpm 4.3 MB/s | 803 kB 00:00 2024-06-03T08:27:03.442 INFO:teuthology.orchestra.run.smithi192.stdout:(3/12): flexiblas-3.0.4-8.el9.x86_64.rpm 568 kB/s | 33 kB 00:00 2024-06-03T08:27:03.475 INFO:teuthology.orchestra.run.smithi192.stdout:(4/12): flexiblas-openblas-openmp-3.0.4-8.el9.x 547 kB/s | 18 kB 00:00 2024-06-03T08:27:03.509 INFO:teuthology.orchestra.run.smithi192.stdout:(5/12): openblas-0.3.21-2.el9.x86_64.rpm 1.0 MB/s | 35 kB 00:00 2024-06-03T08:27:03.568 INFO:teuthology.orchestra.run.smithi192.stdout:(6/12): flexiblas-netlib-3.0.4-8.el9.x86_64.rpm 20 MB/s | 3.0 MB 00:00 2024-06-03T08:27:03.618 INFO:teuthology.orchestra.run.smithi192.stdout:(7/12): python3-devel-3.9.18-3.el9.x86_64.rpm 4.8 MB/s | 244 kB 00:00 2024-06-03T08:27:03.685 INFO:teuthology.orchestra.run.smithi192.stdout:(8/12): openblas-openmp-0.3.21-2.el9.x86_64.rpm 27 MB/s | 4.7 MB 00:00 2024-06-03T08:27:03.727 INFO:teuthology.orchestra.run.smithi192.stdout:(9/12): python3-numpy-f2py-1.20.1-5.el9.x86_64. 5.9 MB/s | 250 kB 00:00 2024-06-03T08:27:03.794 INFO:teuthology.orchestra.run.smithi192.stdout:(10/12): python3-numpy-1.20.1-5.el9.x86_64.rpm 29 MB/s | 5.1 MB 00:00 2024-06-03T08:27:04.077 INFO:teuthology.orchestra.run.smithi192.stdout:(11/12): python3-scipy-1.6.2-8.el9.x86_64.rpm 46 MB/s | 16 MB 00:00 2024-06-03T08:27:04.112 INFO:teuthology.orchestra.run.smithi088.stdout: Installing : python3-numpy-f2py-1:1.20.1-5.el9.x86_64 10/12 2024-06-03T08:27:04.486 INFO:teuthology.orchestra.run.smithi192.stdout:(12/12): ceph-mgr-diskprediction-local-18.2.2-1 5.9 MB/s | 7.4 MB 00:01 2024-06-03T08:27:04.487 INFO:teuthology.orchestra.run.smithi192.stdout:-------------------------------------------------------------------------------- 2024-06-03T08:27:04.487 INFO:teuthology.orchestra.run.smithi192.stdout:Total 28 MB/s | 38 MB 00:01 2024-06-03T08:27:04.838 INFO:teuthology.orchestra.run.smithi192.stdout:Running transaction check 2024-06-03T08:27:04.865 INFO:teuthology.orchestra.run.smithi192.stdout:Transaction check succeeded. 2024-06-03T08:27:04.865 INFO:teuthology.orchestra.run.smithi192.stdout:Running transaction test 2024-06-03T08:27:05.160 INFO:teuthology.orchestra.run.smithi088.stdout: Installing : python3-scipy-1.6.2-8.el9.x86_64 11/12 2024-06-03T08:27:05.196 INFO:teuthology.orchestra.run.smithi088.stdout: Installing : ceph-mgr-diskprediction-local-2:18.2.2-1389.gf1abf 12/12 2024-06-03T08:27:05.379 INFO:teuthology.orchestra.run.smithi192.stdout:Transaction test succeeded. 2024-06-03T08:27:05.379 INFO:teuthology.orchestra.run.smithi192.stdout:Running transaction 2024-06-03T08:27:06.140 INFO:teuthology.orchestra.run.smithi192.stdout: Preparing : 1/1 2024-06-03T08:27:06.216 INFO:teuthology.orchestra.run.smithi192.stdout: Installing : flexiblas-3.0.4-8.el9.x86_64 1/12 2024-06-03T08:27:06.251 INFO:teuthology.orchestra.run.smithi192.stdout: Installing : libquadmath-11.4.1-3.el9.x86_64 2/12 2024-06-03T08:27:06.350 INFO:teuthology.orchestra.run.smithi192.stdout: Installing : libgfortran-11.4.1-3.el9.x86_64 3/12 2024-06-03T08:27:06.394 INFO:teuthology.orchestra.run.smithi192.stdout: Installing : python3-devel-3.9.18-3.el9.x86_64 4/12 2024-06-03T08:27:06.579 INFO:teuthology.orchestra.run.smithi192.stdout: Installing : openblas-0.3.21-2.el9.x86_64 5/12 2024-06-03T08:27:06.610 INFO:teuthology.orchestra.run.smithi192.stdout: Installing : openblas-openmp-0.3.21-2.el9.x86_64 6/12 2024-06-03T08:27:06.687 INFO:teuthology.orchestra.run.smithi192.stdout: Installing : flexiblas-openblas-openmp-3.0.4-8.el9.x86_64 7/12 2024-06-03T08:27:07.336 INFO:teuthology.orchestra.run.smithi192.stdout: Installing : flexiblas-netlib-3.0.4-8.el9.x86_64 8/12 2024-06-03T08:27:07.439 INFO:teuthology.orchestra.run.smithi192.stdout: Installing : python3-numpy-1:1.20.1-5.el9.x86_64 9/12 2024-06-03T08:27:07.773 INFO:teuthology.orchestra.run.smithi088.stdout: Running scriptlet: ceph-mgr-diskprediction-local-2:18.2.2-1389.gf1abf 12/12 2024-06-03T08:27:07.773 INFO:teuthology.orchestra.run.smithi088.stdout: Verifying : ceph-mgr-diskprediction-local-2:18.2.2-1389.gf1abf 1/12 2024-06-03T08:27:07.773 INFO:teuthology.orchestra.run.smithi088.stdout: Verifying : libgfortran-11.4.1-3.el9.x86_64 2/12 2024-06-03T08:27:07.773 INFO:teuthology.orchestra.run.smithi088.stdout: Verifying : libquadmath-11.4.1-3.el9.x86_64 3/12 2024-06-03T08:27:07.774 INFO:teuthology.orchestra.run.smithi088.stdout: Verifying : flexiblas-3.0.4-8.el9.x86_64 4/12 2024-06-03T08:27:07.774 INFO:teuthology.orchestra.run.smithi088.stdout: Verifying : flexiblas-netlib-3.0.4-8.el9.x86_64 5/12 2024-06-03T08:27:07.774 INFO:teuthology.orchestra.run.smithi088.stdout: Verifying : flexiblas-openblas-openmp-3.0.4-8.el9.x86_64 6/12 2024-06-03T08:27:07.774 INFO:teuthology.orchestra.run.smithi088.stdout: Verifying : openblas-0.3.21-2.el9.x86_64 7/12 2024-06-03T08:27:07.774 INFO:teuthology.orchestra.run.smithi088.stdout: Verifying : openblas-openmp-0.3.21-2.el9.x86_64 8/12 2024-06-03T08:27:07.774 INFO:teuthology.orchestra.run.smithi088.stdout: Verifying : python3-devel-3.9.18-3.el9.x86_64 9/12 2024-06-03T08:27:07.774 INFO:teuthology.orchestra.run.smithi088.stdout: Verifying : python3-numpy-1:1.20.1-5.el9.x86_64 10/12 2024-06-03T08:27:07.774 INFO:teuthology.orchestra.run.smithi088.stdout: Verifying : python3-numpy-f2py-1:1.20.1-5.el9.x86_64 11/12 2024-06-03T08:27:08.197 INFO:teuthology.orchestra.run.smithi088.stdout: Verifying : python3-scipy-1.6.2-8.el9.x86_64 12/12 2024-06-03T08:27:08.197 INFO:teuthology.orchestra.run.smithi088.stdout: 2024-06-03T08:27:08.197 INFO:teuthology.orchestra.run.smithi088.stdout:Installed: 2024-06-03T08:27:08.197 INFO:teuthology.orchestra.run.smithi088.stdout: ceph-mgr-diskprediction-local-2:18.2.2-1389.gf1abf795.el9.noarch 2024-06-03T08:27:08.198 INFO:teuthology.orchestra.run.smithi088.stdout: flexiblas-3.0.4-8.el9.x86_64 2024-06-03T08:27:08.198 INFO:teuthology.orchestra.run.smithi088.stdout: flexiblas-netlib-3.0.4-8.el9.x86_64 2024-06-03T08:27:08.198 INFO:teuthology.orchestra.run.smithi088.stdout: flexiblas-openblas-openmp-3.0.4-8.el9.x86_64 2024-06-03T08:27:08.198 INFO:teuthology.orchestra.run.smithi088.stdout: libgfortran-11.4.1-3.el9.x86_64 2024-06-03T08:27:08.198 INFO:teuthology.orchestra.run.smithi088.stdout: libquadmath-11.4.1-3.el9.x86_64 2024-06-03T08:27:08.198 INFO:teuthology.orchestra.run.smithi088.stdout: openblas-0.3.21-2.el9.x86_64 2024-06-03T08:27:08.198 INFO:teuthology.orchestra.run.smithi088.stdout: openblas-openmp-0.3.21-2.el9.x86_64 2024-06-03T08:27:08.198 INFO:teuthology.orchestra.run.smithi088.stdout: python3-devel-3.9.18-3.el9.x86_64 2024-06-03T08:27:08.198 INFO:teuthology.orchestra.run.smithi088.stdout: python3-numpy-1:1.20.1-5.el9.x86_64 2024-06-03T08:27:08.198 INFO:teuthology.orchestra.run.smithi088.stdout: python3-numpy-f2py-1:1.20.1-5.el9.x86_64 2024-06-03T08:27:08.198 INFO:teuthology.orchestra.run.smithi088.stdout: python3-scipy-1.6.2-8.el9.x86_64 2024-06-03T08:27:08.198 INFO:teuthology.orchestra.run.smithi088.stdout: 2024-06-03T08:27:08.199 INFO:teuthology.orchestra.run.smithi088.stdout:Complete! 2024-06-03T08:27:08.545 DEBUG:teuthology.orchestra.run.smithi088:> sudo yum -y install ceph-mgr-rook 2024-06-03T08:27:08.703 INFO:teuthology.orchestra.run.smithi192.stdout: Installing : python3-numpy-f2py-1:1.20.1-5.el9.x86_64 10/12 2024-06-03T08:27:09.134 INFO:teuthology.orchestra.run.smithi088.stdout:Last metadata expiration check: 0:01:13 ago on Mon 03 Jun 2024 08:25:56 AM UTC. 2024-06-03T08:27:09.264 INFO:teuthology.orchestra.run.smithi088.stdout:Dependencies resolved. 2024-06-03T08:27:09.265 INFO:teuthology.orchestra.run.smithi088.stdout:================================================================================ 2024-06-03T08:27:09.265 INFO:teuthology.orchestra.run.smithi088.stdout: Package Arch Version Repository Size 2024-06-03T08:27:09.265 INFO:teuthology.orchestra.run.smithi088.stdout:================================================================================ 2024-06-03T08:27:09.265 INFO:teuthology.orchestra.run.smithi088.stdout:Installing: 2024-06-03T08:27:09.266 INFO:teuthology.orchestra.run.smithi088.stdout: ceph-mgr-rook noarch 2:18.2.2-1389.gf1abf795.el9 ceph-noarch 49 k 2024-06-03T08:27:09.266 INFO:teuthology.orchestra.run.smithi088.stdout:Installing dependencies: 2024-06-03T08:27:09.266 INFO:teuthology.orchestra.run.smithi088.stdout: python3-cachetools noarch 4.2.4-1.el9 epel 32 k 2024-06-03T08:27:09.266 INFO:teuthology.orchestra.run.smithi088.stdout: python3-certifi noarch 2021.10.8-2.el9 epel 15 k 2024-06-03T08:27:09.266 INFO:teuthology.orchestra.run.smithi088.stdout: python3-google-auth noarch 1:2.29.0-1.el9 epel 206 k 2024-06-03T08:27:09.266 INFO:teuthology.orchestra.run.smithi088.stdout: python3-jsonpatch noarch 1.21-16.el9 appstream 26 k 2024-06-03T08:27:09.266 INFO:teuthology.orchestra.run.smithi088.stdout: python3-jsonpointer noarch 2.0-4.el9 appstream 19 k 2024-06-03T08:27:09.266 INFO:teuthology.orchestra.run.smithi088.stdout: python3-kubernetes noarch 1:26.1.0-2.el9 epel 1.0 M 2024-06-03T08:27:09.266 INFO:teuthology.orchestra.run.smithi088.stdout: python3-oauthlib noarch 3.1.1-5.el9 appstream 222 k 2024-06-03T08:27:09.266 INFO:teuthology.orchestra.run.smithi088.stdout: python3-pyasn1 noarch 0.4.8-6.el9 appstream 159 k 2024-06-03T08:27:09.266 INFO:teuthology.orchestra.run.smithi088.stdout: python3-pyasn1-modules noarch 0.4.8-6.el9 appstream 279 k 2024-06-03T08:27:09.267 INFO:teuthology.orchestra.run.smithi088.stdout: python3-requests-oauthlib noarch 1.3.0-12.el9 appstream 54 k 2024-06-03T08:27:09.267 INFO:teuthology.orchestra.run.smithi088.stdout: python3-rsa noarch 4.9-2.el9 epel 59 k 2024-06-03T08:27:09.267 INFO:teuthology.orchestra.run.smithi088.stdout: python3-websocket-client noarch 1.2.3-2.el9 epel 90 k 2024-06-03T08:27:09.267 INFO:teuthology.orchestra.run.smithi088.stdout: 2024-06-03T08:27:09.267 INFO:teuthology.orchestra.run.smithi088.stdout:Transaction Summary 2024-06-03T08:27:09.267 INFO:teuthology.orchestra.run.smithi088.stdout:================================================================================ 2024-06-03T08:27:09.267 INFO:teuthology.orchestra.run.smithi088.stdout:Install 13 Packages 2024-06-03T08:27:09.267 INFO:teuthology.orchestra.run.smithi088.stdout: 2024-06-03T08:27:09.268 INFO:teuthology.orchestra.run.smithi088.stdout:Total download size: 2.2 M 2024-06-03T08:27:09.268 INFO:teuthology.orchestra.run.smithi088.stdout:Installed size: 26 M 2024-06-03T08:27:09.268 INFO:teuthology.orchestra.run.smithi088.stdout:Downloading Packages: 2024-06-03T08:27:09.538 INFO:teuthology.orchestra.run.smithi088.stdout:(1/13): python3-jsonpointer-2.0-4.el9.noarch.rp 198 kB/s | 19 kB 00:00 2024-06-03T08:27:09.563 INFO:teuthology.orchestra.run.smithi088.stdout:(2/13): python3-jsonpatch-1.21-16.el9.noarch.rp 210 kB/s | 26 kB 00:00 2024-06-03T08:27:09.663 INFO:teuthology.orchestra.run.smithi088.stdout:(3/13): python3-oauthlib-3.1.1-5.el9.noarch.rpm 1.7 MB/s | 222 kB 00:00 2024-06-03T08:27:09.688 INFO:teuthology.orchestra.run.smithi088.stdout:(4/13): ceph-mgr-rook-18.2.2-1389.gf1abf795.el9 199 kB/s | 49 kB 00:00 2024-06-03T08:27:09.714 INFO:teuthology.orchestra.run.smithi088.stdout:(5/13): python3-pyasn1-0.4.8-6.el9.noarch.rpm 1.0 MB/s | 159 kB 00:00 2024-06-03T08:27:09.733 INFO:teuthology.orchestra.run.smithi192.stdout: Installing : python3-scipy-1.6.2-8.el9.x86_64 11/12 2024-06-03T08:27:09.739 INFO:teuthology.orchestra.run.smithi088.stdout:(6/13): python3-pyasn1-modules-0.4.8-6.el9.noar 3.6 MB/s | 279 kB 00:00 2024-06-03T08:27:09.769 INFO:teuthology.orchestra.run.smithi192.stdout: Installing : ceph-mgr-diskprediction-local-2:18.2.2-1389.gf1abf 12/12 2024-06-03T08:27:09.789 INFO:teuthology.orchestra.run.smithi088.stdout:(7/13): python3-cachetools-4.2.4-1.el9.noarch.r 428 kB/s | 32 kB 00:00 2024-06-03T08:27:09.814 INFO:teuthology.orchestra.run.smithi088.stdout:(8/13): python3-requests-oauthlib-1.3.0-12.el9. 426 kB/s | 54 kB 00:00 2024-06-03T08:27:09.840 INFO:teuthology.orchestra.run.smithi088.stdout:(9/13): python3-certifi-2021.10.8-2.el9.noarch. 149 kB/s | 15 kB 00:00 2024-06-03T08:27:09.890 INFO:teuthology.orchestra.run.smithi088.stdout:(10/13): python3-google-auth-2.29.0-1.el9.noarc 2.0 MB/s | 206 kB 00:00 2024-06-03T08:27:09.912 INFO:teuthology.orchestra.run.smithi088.stdout:(11/13): python3-rsa-4.9-2.el9.noarch.rpm 827 kB/s | 59 kB 00:00 2024-06-03T08:27:09.937 INFO:teuthology.orchestra.run.smithi088.stdout:(12/13): python3-websocket-client-1.2.3-2.el9.n 1.9 MB/s | 90 kB 00:00 2024-06-03T08:27:09.987 INFO:teuthology.orchestra.run.smithi088.stdout:(13/13): python3-kubernetes-26.1.0-2.el9.noarch 6.9 MB/s | 1.0 MB 00:00 2024-06-03T08:27:09.987 INFO:teuthology.orchestra.run.smithi088.stdout:-------------------------------------------------------------------------------- 2024-06-03T08:27:09.988 INFO:teuthology.orchestra.run.smithi088.stdout:Total 3.1 MB/s | 2.2 MB 00:00 2024-06-03T08:27:10.053 INFO:teuthology.orchestra.run.smithi088.stdout:Running transaction check 2024-06-03T08:27:10.067 INFO:teuthology.orchestra.run.smithi088.stdout:Transaction check succeeded. 2024-06-03T08:27:10.067 INFO:teuthology.orchestra.run.smithi088.stdout:Running transaction test 2024-06-03T08:27:10.151 INFO:teuthology.orchestra.run.smithi088.stdout:Transaction test succeeded. 2024-06-03T08:27:10.151 INFO:teuthology.orchestra.run.smithi088.stdout:Running transaction 2024-06-03T08:27:10.652 INFO:teuthology.orchestra.run.smithi088.stdout: Preparing : 1/1 2024-06-03T08:27:10.825 INFO:teuthology.orchestra.run.smithi088.stdout: Installing : python3-pyasn1-0.4.8-6.el9.noarch 1/13 2024-06-03T08:27:10.880 INFO:teuthology.orchestra.run.smithi088.stdout: Installing : python3-pyasn1-modules-0.4.8-6.el9.noarch 2/13 2024-06-03T08:27:10.944 INFO:teuthology.orchestra.run.smithi088.stdout: Installing : python3-rsa-4.9-2.el9.noarch 3/13 2024-06-03T08:27:10.982 INFO:teuthology.orchestra.run.smithi088.stdout: Installing : python3-websocket-client-1.2.3-2.el9.noarch 4/13 2024-06-03T08:27:11.023 INFO:teuthology.orchestra.run.smithi088.stdout: Installing : python3-certifi-2021.10.8-2.el9.noarch 5/13 2024-06-03T08:27:11.124 INFO:teuthology.orchestra.run.smithi088.stdout: Installing : python3-cachetools-4.2.4-1.el9.noarch 6/13 2024-06-03T08:27:11.226 INFO:teuthology.orchestra.run.smithi088.stdout: Installing : python3-google-auth-1:2.29.0-1.el9.noarch 7/13 2024-06-03T08:27:11.274 INFO:teuthology.orchestra.run.smithi088.stdout: Installing : python3-oauthlib-3.1.1-5.el9.noarch 8/13 2024-06-03T08:27:11.988 INFO:teuthology.orchestra.run.smithi088.stdout: Installing : python3-requests-oauthlib-1.3.0-12.el9.noarch 9/13 2024-06-03T08:27:12.038 INFO:teuthology.orchestra.run.smithi088.stdout: Installing : python3-kubernetes-1:26.1.0-2.el9.noarch 10/13 2024-06-03T08:27:12.072 INFO:teuthology.orchestra.run.smithi088.stdout: Installing : python3-jsonpointer-2.0-4.el9.noarch 11/13 2024-06-03T08:27:12.120 INFO:teuthology.orchestra.run.smithi088.stdout: Installing : python3-jsonpatch-1.21-16.el9.noarch 12/13 2024-06-03T08:27:12.150 INFO:teuthology.orchestra.run.smithi088.stdout: Installing : ceph-mgr-rook-2:18.2.2-1389.gf1abf795.el9.noarch 13/13 2024-06-03T08:27:12.307 INFO:teuthology.orchestra.run.smithi192.stdout: Running scriptlet: ceph-mgr-diskprediction-local-2:18.2.2-1389.gf1abf 12/12 2024-06-03T08:27:12.307 INFO:teuthology.orchestra.run.smithi192.stdout: Verifying : ceph-mgr-diskprediction-local-2:18.2.2-1389.gf1abf 1/12 2024-06-03T08:27:12.307 INFO:teuthology.orchestra.run.smithi192.stdout: Verifying : libgfortran-11.4.1-3.el9.x86_64 2/12 2024-06-03T08:27:12.307 INFO:teuthology.orchestra.run.smithi192.stdout: Verifying : libquadmath-11.4.1-3.el9.x86_64 3/12 2024-06-03T08:27:12.307 INFO:teuthology.orchestra.run.smithi192.stdout: Verifying : flexiblas-3.0.4-8.el9.x86_64 4/12 2024-06-03T08:27:12.307 INFO:teuthology.orchestra.run.smithi192.stdout: Verifying : flexiblas-netlib-3.0.4-8.el9.x86_64 5/12 2024-06-03T08:27:12.307 INFO:teuthology.orchestra.run.smithi192.stdout: Verifying : flexiblas-openblas-openmp-3.0.4-8.el9.x86_64 6/12 2024-06-03T08:27:12.307 INFO:teuthology.orchestra.run.smithi192.stdout: Verifying : openblas-0.3.21-2.el9.x86_64 7/12 2024-06-03T08:27:12.307 INFO:teuthology.orchestra.run.smithi192.stdout: Verifying : openblas-openmp-0.3.21-2.el9.x86_64 8/12 2024-06-03T08:27:12.308 INFO:teuthology.orchestra.run.smithi192.stdout: Verifying : python3-devel-3.9.18-3.el9.x86_64 9/12 2024-06-03T08:27:12.308 INFO:teuthology.orchestra.run.smithi192.stdout: Verifying : python3-numpy-1:1.20.1-5.el9.x86_64 10/12 2024-06-03T08:27:12.308 INFO:teuthology.orchestra.run.smithi192.stdout: Verifying : python3-numpy-f2py-1:1.20.1-5.el9.x86_64 11/12 2024-06-03T08:27:12.749 INFO:teuthology.orchestra.run.smithi192.stdout: Verifying : python3-scipy-1.6.2-8.el9.x86_64 12/12 2024-06-03T08:27:12.749 INFO:teuthology.orchestra.run.smithi192.stdout: 2024-06-03T08:27:12.749 INFO:teuthology.orchestra.run.smithi192.stdout:Installed: 2024-06-03T08:27:12.750 INFO:teuthology.orchestra.run.smithi192.stdout: ceph-mgr-diskprediction-local-2:18.2.2-1389.gf1abf795.el9.noarch 2024-06-03T08:27:12.750 INFO:teuthology.orchestra.run.smithi192.stdout: flexiblas-3.0.4-8.el9.x86_64 2024-06-03T08:27:12.750 INFO:teuthology.orchestra.run.smithi192.stdout: flexiblas-netlib-3.0.4-8.el9.x86_64 2024-06-03T08:27:12.750 INFO:teuthology.orchestra.run.smithi192.stdout: flexiblas-openblas-openmp-3.0.4-8.el9.x86_64 2024-06-03T08:27:12.750 INFO:teuthology.orchestra.run.smithi192.stdout: libgfortran-11.4.1-3.el9.x86_64 2024-06-03T08:27:12.750 INFO:teuthology.orchestra.run.smithi192.stdout: libquadmath-11.4.1-3.el9.x86_64 2024-06-03T08:27:12.750 INFO:teuthology.orchestra.run.smithi192.stdout: openblas-0.3.21-2.el9.x86_64 2024-06-03T08:27:12.750 INFO:teuthology.orchestra.run.smithi192.stdout: openblas-openmp-0.3.21-2.el9.x86_64 2024-06-03T08:27:12.750 INFO:teuthology.orchestra.run.smithi192.stdout: python3-devel-3.9.18-3.el9.x86_64 2024-06-03T08:27:12.750 INFO:teuthology.orchestra.run.smithi192.stdout: python3-numpy-1:1.20.1-5.el9.x86_64 2024-06-03T08:27:12.750 INFO:teuthology.orchestra.run.smithi192.stdout: python3-numpy-f2py-1:1.20.1-5.el9.x86_64 2024-06-03T08:27:12.750 INFO:teuthology.orchestra.run.smithi192.stdout: python3-scipy-1.6.2-8.el9.x86_64 2024-06-03T08:27:12.751 INFO:teuthology.orchestra.run.smithi192.stdout: 2024-06-03T08:27:12.751 INFO:teuthology.orchestra.run.smithi192.stdout:Complete! 2024-06-03T08:27:12.910 INFO:teuthology.orchestra.run.smithi088.stdout: Running scriptlet: ceph-mgr-rook-2:18.2.2-1389.gf1abf795.el9.noarch 13/13 2024-06-03T08:27:12.910 INFO:teuthology.orchestra.run.smithi088.stdout: Verifying : ceph-mgr-rook-2:18.2.2-1389.gf1abf795.el9.noarch 1/13 2024-06-03T08:27:12.910 INFO:teuthology.orchestra.run.smithi088.stdout: Verifying : python3-jsonpatch-1.21-16.el9.noarch 2/13 2024-06-03T08:27:12.910 INFO:teuthology.orchestra.run.smithi088.stdout: Verifying : python3-jsonpointer-2.0-4.el9.noarch 3/13 2024-06-03T08:27:12.910 INFO:teuthology.orchestra.run.smithi088.stdout: Verifying : python3-oauthlib-3.1.1-5.el9.noarch 4/13 2024-06-03T08:27:12.910 INFO:teuthology.orchestra.run.smithi088.stdout: Verifying : python3-pyasn1-0.4.8-6.el9.noarch 5/13 2024-06-03T08:27:12.911 INFO:teuthology.orchestra.run.smithi088.stdout: Verifying : python3-pyasn1-modules-0.4.8-6.el9.noarch 6/13 2024-06-03T08:27:12.911 INFO:teuthology.orchestra.run.smithi088.stdout: Verifying : python3-requests-oauthlib-1.3.0-12.el9.noarch 7/13 2024-06-03T08:27:12.911 INFO:teuthology.orchestra.run.smithi088.stdout: Verifying : python3-cachetools-4.2.4-1.el9.noarch 8/13 2024-06-03T08:27:12.911 INFO:teuthology.orchestra.run.smithi088.stdout: Verifying : python3-certifi-2021.10.8-2.el9.noarch 9/13 2024-06-03T08:27:12.911 INFO:teuthology.orchestra.run.smithi088.stdout: Verifying : python3-google-auth-1:2.29.0-1.el9.noarch 10/13 2024-06-03T08:27:12.911 INFO:teuthology.orchestra.run.smithi088.stdout: Verifying : python3-kubernetes-1:26.1.0-2.el9.noarch 11/13 2024-06-03T08:27:12.911 INFO:teuthology.orchestra.run.smithi088.stdout: Verifying : python3-rsa-4.9-2.el9.noarch 12/13 2024-06-03T08:27:13.130 DEBUG:teuthology.orchestra.run.smithi192:> sudo yum -y install ceph-mgr-rook 2024-06-03T08:27:13.259 INFO:teuthology.orchestra.run.smithi088.stdout: Verifying : python3-websocket-client-1.2.3-2.el9.noarch 13/13 2024-06-03T08:27:13.259 INFO:teuthology.orchestra.run.smithi088.stdout: 2024-06-03T08:27:13.259 INFO:teuthology.orchestra.run.smithi088.stdout:Installed: 2024-06-03T08:27:13.259 INFO:teuthology.orchestra.run.smithi088.stdout: ceph-mgr-rook-2:18.2.2-1389.gf1abf795.el9.noarch 2024-06-03T08:27:13.260 INFO:teuthology.orchestra.run.smithi088.stdout: python3-cachetools-4.2.4-1.el9.noarch 2024-06-03T08:27:13.260 INFO:teuthology.orchestra.run.smithi088.stdout: python3-certifi-2021.10.8-2.el9.noarch 2024-06-03T08:27:13.260 INFO:teuthology.orchestra.run.smithi088.stdout: python3-google-auth-1:2.29.0-1.el9.noarch 2024-06-03T08:27:13.260 INFO:teuthology.orchestra.run.smithi088.stdout: python3-jsonpatch-1.21-16.el9.noarch 2024-06-03T08:27:13.260 INFO:teuthology.orchestra.run.smithi088.stdout: python3-jsonpointer-2.0-4.el9.noarch 2024-06-03T08:27:13.260 INFO:teuthology.orchestra.run.smithi088.stdout: python3-kubernetes-1:26.1.0-2.el9.noarch 2024-06-03T08:27:13.260 INFO:teuthology.orchestra.run.smithi088.stdout: python3-oauthlib-3.1.1-5.el9.noarch 2024-06-03T08:27:13.260 INFO:teuthology.orchestra.run.smithi088.stdout: python3-pyasn1-0.4.8-6.el9.noarch 2024-06-03T08:27:13.260 INFO:teuthology.orchestra.run.smithi088.stdout: python3-pyasn1-modules-0.4.8-6.el9.noarch 2024-06-03T08:27:13.260 INFO:teuthology.orchestra.run.smithi088.stdout: python3-requests-oauthlib-1.3.0-12.el9.noarch 2024-06-03T08:27:13.260 INFO:teuthology.orchestra.run.smithi088.stdout: python3-rsa-4.9-2.el9.noarch 2024-06-03T08:27:13.260 INFO:teuthology.orchestra.run.smithi088.stdout: python3-websocket-client-1.2.3-2.el9.noarch 2024-06-03T08:27:13.260 INFO:teuthology.orchestra.run.smithi088.stdout: 2024-06-03T08:27:13.261 INFO:teuthology.orchestra.run.smithi088.stdout:Complete! 2024-06-03T08:27:13.518 DEBUG:teuthology.orchestra.run.smithi088:> sudo yum -y install ceph-mgr-cephadm 2024-06-03T08:27:13.717 INFO:teuthology.orchestra.run.smithi192.stdout:Last metadata expiration check: 0:01:13 ago on Mon 03 Jun 2024 08:26:00 AM UTC. 2024-06-03T08:27:13.840 INFO:teuthology.orchestra.run.smithi192.stdout:Dependencies resolved. 2024-06-03T08:27:13.842 INFO:teuthology.orchestra.run.smithi192.stdout:================================================================================ 2024-06-03T08:27:13.842 INFO:teuthology.orchestra.run.smithi192.stdout: Package Arch Version Repository Size 2024-06-03T08:27:13.842 INFO:teuthology.orchestra.run.smithi192.stdout:================================================================================ 2024-06-03T08:27:13.842 INFO:teuthology.orchestra.run.smithi192.stdout:Installing: 2024-06-03T08:27:13.842 INFO:teuthology.orchestra.run.smithi192.stdout: ceph-mgr-rook noarch 2:18.2.2-1389.gf1abf795.el9 ceph-noarch 49 k 2024-06-03T08:27:13.842 INFO:teuthology.orchestra.run.smithi192.stdout:Installing dependencies: 2024-06-03T08:27:13.842 INFO:teuthology.orchestra.run.smithi192.stdout: python3-cachetools noarch 4.2.4-1.el9 epel 32 k 2024-06-03T08:27:13.842 INFO:teuthology.orchestra.run.smithi192.stdout: python3-certifi noarch 2021.10.8-2.el9 epel 15 k 2024-06-03T08:27:13.842 INFO:teuthology.orchestra.run.smithi192.stdout: python3-google-auth noarch 1:2.29.0-1.el9 epel 206 k 2024-06-03T08:27:13.842 INFO:teuthology.orchestra.run.smithi192.stdout: python3-jsonpatch noarch 1.21-16.el9 appstream 26 k 2024-06-03T08:27:13.842 INFO:teuthology.orchestra.run.smithi192.stdout: python3-jsonpointer noarch 2.0-4.el9 appstream 19 k 2024-06-03T08:27:13.843 INFO:teuthology.orchestra.run.smithi192.stdout: python3-kubernetes noarch 1:26.1.0-2.el9 epel 1.0 M 2024-06-03T08:27:13.843 INFO:teuthology.orchestra.run.smithi192.stdout: python3-oauthlib noarch 3.1.1-5.el9 appstream 222 k 2024-06-03T08:27:13.843 INFO:teuthology.orchestra.run.smithi192.stdout: python3-pyasn1 noarch 0.4.8-6.el9 appstream 159 k 2024-06-03T08:27:13.843 INFO:teuthology.orchestra.run.smithi192.stdout: python3-pyasn1-modules noarch 0.4.8-6.el9 appstream 279 k 2024-06-03T08:27:13.843 INFO:teuthology.orchestra.run.smithi192.stdout: python3-requests-oauthlib noarch 1.3.0-12.el9 appstream 54 k 2024-06-03T08:27:13.843 INFO:teuthology.orchestra.run.smithi192.stdout: python3-rsa noarch 4.9-2.el9 epel 59 k 2024-06-03T08:27:13.843 INFO:teuthology.orchestra.run.smithi192.stdout: python3-websocket-client noarch 1.2.3-2.el9 epel 90 k 2024-06-03T08:27:13.843 INFO:teuthology.orchestra.run.smithi192.stdout: 2024-06-03T08:27:13.843 INFO:teuthology.orchestra.run.smithi192.stdout:Transaction Summary 2024-06-03T08:27:13.843 INFO:teuthology.orchestra.run.smithi192.stdout:================================================================================ 2024-06-03T08:27:13.843 INFO:teuthology.orchestra.run.smithi192.stdout:Install 13 Packages 2024-06-03T08:27:13.843 INFO:teuthology.orchestra.run.smithi192.stdout: 2024-06-03T08:27:13.844 INFO:teuthology.orchestra.run.smithi192.stdout:Total download size: 2.2 M 2024-06-03T08:27:13.844 INFO:teuthology.orchestra.run.smithi192.stdout:Installed size: 26 M 2024-06-03T08:27:13.844 INFO:teuthology.orchestra.run.smithi192.stdout:Downloading Packages: 2024-06-03T08:27:14.098 INFO:teuthology.orchestra.run.smithi088.stdout:Last metadata expiration check: 0:01:18 ago on Mon 03 Jun 2024 08:25:56 AM UTC. 2024-06-03T08:27:14.113 INFO:teuthology.orchestra.run.smithi192.stdout:(1/13): python3-jsonpatch-1.21-16.el9.noarch.rp 272 kB/s | 26 kB 00:00 2024-06-03T08:27:14.138 INFO:teuthology.orchestra.run.smithi192.stdout:(2/13): python3-jsonpointer-2.0-4.el9.noarch.rp 161 kB/s | 19 kB 00:00 2024-06-03T08:27:14.221 INFO:teuthology.orchestra.run.smithi088.stdout:Dependencies resolved. 2024-06-03T08:27:14.222 INFO:teuthology.orchestra.run.smithi088.stdout:================================================================================ 2024-06-03T08:27:14.222 INFO:teuthology.orchestra.run.smithi088.stdout: Package Arch Version Repository Size 2024-06-03T08:27:14.222 INFO:teuthology.orchestra.run.smithi088.stdout:================================================================================ 2024-06-03T08:27:14.223 INFO:teuthology.orchestra.run.smithi088.stdout:Installing: 2024-06-03T08:27:14.223 INFO:teuthology.orchestra.run.smithi088.stdout: ceph-mgr-cephadm noarch 2:18.2.2-1389.gf1abf795.el9 ceph-noarch 139 k 2024-06-03T08:27:14.223 INFO:teuthology.orchestra.run.smithi088.stdout:Installing dependencies: 2024-06-03T08:27:14.223 INFO:teuthology.orchestra.run.smithi088.stdout: python3-asyncssh noarch 2.13.2-5.el9 epel 548 k 2024-06-03T08:27:14.223 INFO:teuthology.orchestra.run.smithi088.stdout: python3-babel noarch 2.9.1-2.el9 appstream 6.0 M 2024-06-03T08:27:14.223 INFO:teuthology.orchestra.run.smithi088.stdout: python3-jinja2 noarch 2.11.3-6.el9 appstream 248 k 2024-06-03T08:27:14.223 INFO:teuthology.orchestra.run.smithi088.stdout: python3-natsort noarch 7.1.1-5.el9 epel 58 k 2024-06-03T08:27:14.223 INFO:teuthology.orchestra.run.smithi088.stdout: python3-typing-extensions noarch 4.4.0-2.el9 epel 51 k 2024-06-03T08:27:14.223 INFO:teuthology.orchestra.run.smithi088.stdout: 2024-06-03T08:27:14.223 INFO:teuthology.orchestra.run.smithi088.stdout:Transaction Summary 2024-06-03T08:27:14.223 INFO:teuthology.orchestra.run.smithi088.stdout:================================================================================ 2024-06-03T08:27:14.223 INFO:teuthology.orchestra.run.smithi088.stdout:Install 6 Packages 2024-06-03T08:27:14.223 INFO:teuthology.orchestra.run.smithi088.stdout: 2024-06-03T08:27:14.224 INFO:teuthology.orchestra.run.smithi088.stdout:Total download size: 7.0 M 2024-06-03T08:27:14.224 INFO:teuthology.orchestra.run.smithi088.stdout:Installed size: 33 M 2024-06-03T08:27:14.224 INFO:teuthology.orchestra.run.smithi088.stdout:Downloading Packages: 2024-06-03T08:27:14.271 INFO:teuthology.orchestra.run.smithi192.stdout:(3/13): python3-oauthlib-3.1.1-5.el9.noarch.rpm 1.6 MB/s | 222 kB 00:00 2024-06-03T08:27:14.288 INFO:teuthology.orchestra.run.smithi192.stdout:(4/13): python3-pyasn1-0.4.8-6.el9.noarch.rpm 1.0 MB/s | 159 kB 00:00 2024-06-03T08:27:14.314 INFO:teuthology.orchestra.run.smithi192.stdout:(5/13): ceph-mgr-rook-18.2.2-1389.gf1abf795.el9 167 kB/s | 49 kB 00:00 2024-06-03T08:27:14.339 INFO:teuthology.orchestra.run.smithi192.stdout:(6/13): python3-requests-oauthlib-1.3.0-12.el9. 1.0 MB/s | 54 kB 00:00 2024-06-03T08:27:14.364 INFO:teuthology.orchestra.run.smithi192.stdout:(7/13): python3-pyasn1-modules-0.4.8-6.el9.noar 3.6 MB/s | 279 kB 00:00 2024-06-03T08:27:14.414 INFO:teuthology.orchestra.run.smithi192.stdout:(8/13): python3-certifi-2021.10.8-2.el9.noarch. 199 kB/s | 15 kB 00:00 2024-06-03T08:27:14.440 INFO:teuthology.orchestra.run.smithi192.stdout:(9/13): python3-cachetools-4.2.4-1.el9.noarch.r 256 kB/s | 32 kB 00:00 2024-06-03T08:27:14.482 INFO:teuthology.orchestra.run.smithi192.stdout:(10/13): python3-google-auth-2.29.0-1.el9.noarc 1.7 MB/s | 206 kB 00:00 2024-06-03T08:27:14.514 INFO:teuthology.orchestra.run.smithi192.stdout:(11/13): python3-rsa-4.9-2.el9.noarch.rpm 799 kB/s | 59 kB 00:00 2024-06-03T08:27:14.539 INFO:teuthology.orchestra.run.smithi192.stdout:(12/13): python3-websocket-client-1.2.3-2.el9.n 1.5 MB/s | 90 kB 00:00 2024-06-03T08:27:14.573 INFO:teuthology.orchestra.run.smithi192.stdout:(13/13): python3-kubernetes-26.1.0-2.el9.noarch 6.4 MB/s | 1.0 MB 00:00 2024-06-03T08:27:14.573 INFO:teuthology.orchestra.run.smithi192.stdout:-------------------------------------------------------------------------------- 2024-06-03T08:27:14.574 INFO:teuthology.orchestra.run.smithi192.stdout:Total 3.0 MB/s | 2.2 MB 00:00 2024-06-03T08:27:14.637 INFO:teuthology.orchestra.run.smithi192.stdout:Running transaction check 2024-06-03T08:27:14.651 INFO:teuthology.orchestra.run.smithi192.stdout:Transaction check succeeded. 2024-06-03T08:27:14.651 INFO:teuthology.orchestra.run.smithi192.stdout:Running transaction test 2024-06-03T08:27:14.735 INFO:teuthology.orchestra.run.smithi192.stdout:Transaction test succeeded. 2024-06-03T08:27:14.736 INFO:teuthology.orchestra.run.smithi192.stdout:Running transaction 2024-06-03T08:27:14.848 INFO:teuthology.orchestra.run.smithi088.stdout:(1/6): python3-jinja2-2.11.3-6.el9.noarch.rpm 1.4 MB/s | 248 kB 00:00 2024-06-03T08:27:14.998 INFO:teuthology.orchestra.run.smithi088.stdout:(2/6): python3-babel-2.9.1-2.el9.noarch.rpm 18 MB/s | 6.0 MB 00:00 2024-06-03T08:27:15.023 INFO:teuthology.orchestra.run.smithi088.stdout:(3/6): ceph-mgr-cephadm-18.2.2-1389.gf1abf795.e 396 kB/s | 139 kB 00:00 2024-06-03T08:27:15.157 INFO:teuthology.orchestra.run.smithi088.stdout:(4/6): python3-natsort-7.1.1-5.el9.noarch.rpm 364 kB/s | 58 kB 00:00 2024-06-03T08:27:15.182 INFO:teuthology.orchestra.run.smithi088.stdout:(5/6): python3-typing-extensions-4.4.0-2.el9.no 322 kB/s | 51 kB 00:00 2024-06-03T08:27:15.207 INFO:teuthology.orchestra.run.smithi088.stdout:(6/6): python3-asyncssh-2.13.2-5.el9.noarch.rpm 1.5 MB/s | 548 kB 00:00 2024-06-03T08:27:15.208 INFO:teuthology.orchestra.run.smithi088.stdout:-------------------------------------------------------------------------------- 2024-06-03T08:27:15.208 INFO:teuthology.orchestra.run.smithi088.stdout:Total 7.1 MB/s | 7.0 MB 00:00 2024-06-03T08:27:15.239 INFO:teuthology.orchestra.run.smithi192.stdout: Preparing : 1/1 2024-06-03T08:27:15.279 INFO:teuthology.orchestra.run.smithi088.stdout:Running transaction check 2024-06-03T08:27:15.291 INFO:teuthology.orchestra.run.smithi088.stdout:Transaction check succeeded. 2024-06-03T08:27:15.291 INFO:teuthology.orchestra.run.smithi088.stdout:Running transaction test 2024-06-03T08:27:15.388 INFO:teuthology.orchestra.run.smithi088.stdout:Transaction test succeeded. 2024-06-03T08:27:15.388 INFO:teuthology.orchestra.run.smithi088.stdout:Running transaction 2024-06-03T08:27:15.412 INFO:teuthology.orchestra.run.smithi192.stdout: Installing : python3-pyasn1-0.4.8-6.el9.noarch 1/13 2024-06-03T08:27:15.474 INFO:teuthology.orchestra.run.smithi192.stdout: Installing : python3-pyasn1-modules-0.4.8-6.el9.noarch 2/13 2024-06-03T08:27:15.530 INFO:teuthology.orchestra.run.smithi192.stdout: Installing : python3-rsa-4.9-2.el9.noarch 3/13 2024-06-03T08:27:15.567 INFO:teuthology.orchestra.run.smithi192.stdout: Installing : python3-websocket-client-1.2.3-2.el9.noarch 4/13 2024-06-03T08:27:15.608 INFO:teuthology.orchestra.run.smithi192.stdout: Installing : python3-certifi-2021.10.8-2.el9.noarch 5/13 2024-06-03T08:27:15.684 INFO:teuthology.orchestra.run.smithi088.stdout: Preparing : 1/1 2024-06-03T08:27:15.702 INFO:teuthology.orchestra.run.smithi192.stdout: Installing : python3-cachetools-4.2.4-1.el9.noarch 6/13 2024-06-03T08:27:15.820 INFO:teuthology.orchestra.run.smithi192.stdout: Installing : python3-google-auth-1:2.29.0-1.el9.noarch 7/13 2024-06-03T08:27:15.862 INFO:teuthology.orchestra.run.smithi088.stdout: Installing : python3-typing-extensions-4.4.0-2.el9.noarch 1/6 2024-06-03T08:27:15.876 INFO:teuthology.orchestra.run.smithi192.stdout: Installing : python3-oauthlib-3.1.1-5.el9.noarch 8/13 2024-06-03T08:27:15.917 INFO:teuthology.orchestra.run.smithi088.stdout: Installing : python3-asyncssh-2.13.2-5.el9.noarch 2/6 2024-06-03T08:27:16.427 INFO:teuthology.orchestra.run.smithi088.stdout: Installing : python3-natsort-7.1.1-5.el9.noarch 3/6 2024-06-03T08:27:16.511 INFO:teuthology.orchestra.run.smithi088.stdout: Installing : python3-babel-2.9.1-2.el9.noarch 4/6 2024-06-03T08:27:16.574 INFO:teuthology.orchestra.run.smithi088.stdout: Installing : python3-jinja2-2.11.3-6.el9.noarch 5/6 2024-06-03T08:27:16.587 INFO:teuthology.orchestra.run.smithi192.stdout: Installing : python3-requests-oauthlib-1.3.0-12.el9.noarch 9/13 2024-06-03T08:27:16.604 INFO:teuthology.orchestra.run.smithi088.stdout: Installing : ceph-mgr-cephadm-2:18.2.2-1389.gf1abf795.el9.noarch 6/6 2024-06-03T08:27:16.632 INFO:teuthology.orchestra.run.smithi192.stdout: Installing : python3-kubernetes-1:26.1.0-2.el9.noarch 10/13 2024-06-03T08:27:16.666 INFO:teuthology.orchestra.run.smithi192.stdout: Installing : python3-jsonpointer-2.0-4.el9.noarch 11/13 2024-06-03T08:27:16.713 INFO:teuthology.orchestra.run.smithi192.stdout: Installing : python3-jsonpatch-1.21-16.el9.noarch 12/13 2024-06-03T08:27:16.735 INFO:teuthology.orchestra.run.smithi192.stdout: Installing : ceph-mgr-rook-2:18.2.2-1389.gf1abf795.el9.noarch 13/13 2024-06-03T08:27:17.361 INFO:teuthology.orchestra.run.smithi088.stdout: Running scriptlet: ceph-mgr-cephadm-2:18.2.2-1389.gf1abf795.el9.noarch 6/6 2024-06-03T08:27:17.361 INFO:teuthology.orchestra.run.smithi088.stdout: Verifying : ceph-mgr-cephadm-2:18.2.2-1389.gf1abf795.el9.noarch 1/6 2024-06-03T08:27:17.361 INFO:teuthology.orchestra.run.smithi088.stdout: Verifying : python3-babel-2.9.1-2.el9.noarch 2/6 2024-06-03T08:27:17.362 INFO:teuthology.orchestra.run.smithi088.stdout: Verifying : python3-jinja2-2.11.3-6.el9.noarch 3/6 2024-06-03T08:27:17.362 INFO:teuthology.orchestra.run.smithi088.stdout: Verifying : python3-asyncssh-2.13.2-5.el9.noarch 4/6 2024-06-03T08:27:17.362 INFO:teuthology.orchestra.run.smithi088.stdout: Verifying : python3-natsort-7.1.1-5.el9.noarch 5/6 2024-06-03T08:27:17.452 INFO:teuthology.orchestra.run.smithi192.stdout: Running scriptlet: ceph-mgr-rook-2:18.2.2-1389.gf1abf795.el9.noarch 13/13 2024-06-03T08:27:17.453 INFO:teuthology.orchestra.run.smithi192.stdout: Verifying : ceph-mgr-rook-2:18.2.2-1389.gf1abf795.el9.noarch 1/13 2024-06-03T08:27:17.453 INFO:teuthology.orchestra.run.smithi192.stdout: Verifying : python3-jsonpatch-1.21-16.el9.noarch 2/13 2024-06-03T08:27:17.453 INFO:teuthology.orchestra.run.smithi192.stdout: Verifying : python3-jsonpointer-2.0-4.el9.noarch 3/13 2024-06-03T08:27:17.453 INFO:teuthology.orchestra.run.smithi192.stdout: Verifying : python3-oauthlib-3.1.1-5.el9.noarch 4/13 2024-06-03T08:27:17.453 INFO:teuthology.orchestra.run.smithi192.stdout: Verifying : python3-pyasn1-0.4.8-6.el9.noarch 5/13 2024-06-03T08:27:17.453 INFO:teuthology.orchestra.run.smithi192.stdout: Verifying : python3-pyasn1-modules-0.4.8-6.el9.noarch 6/13 2024-06-03T08:27:17.453 INFO:teuthology.orchestra.run.smithi192.stdout: Verifying : python3-requests-oauthlib-1.3.0-12.el9.noarch 7/13 2024-06-03T08:27:17.453 INFO:teuthology.orchestra.run.smithi192.stdout: Verifying : python3-cachetools-4.2.4-1.el9.noarch 8/13 2024-06-03T08:27:17.453 INFO:teuthology.orchestra.run.smithi192.stdout: Verifying : python3-certifi-2021.10.8-2.el9.noarch 9/13 2024-06-03T08:27:17.453 INFO:teuthology.orchestra.run.smithi192.stdout: Verifying : python3-google-auth-1:2.29.0-1.el9.noarch 10/13 2024-06-03T08:27:17.453 INFO:teuthology.orchestra.run.smithi192.stdout: Verifying : python3-kubernetes-1:26.1.0-2.el9.noarch 11/13 2024-06-03T08:27:17.453 INFO:teuthology.orchestra.run.smithi192.stdout: Verifying : python3-rsa-4.9-2.el9.noarch 12/13 2024-06-03T08:27:17.624 INFO:teuthology.orchestra.run.smithi088.stdout: Verifying : python3-typing-extensions-4.4.0-2.el9.noarch 6/6 2024-06-03T08:27:17.624 INFO:teuthology.orchestra.run.smithi088.stdout: 2024-06-03T08:27:17.624 INFO:teuthology.orchestra.run.smithi088.stdout:Installed: 2024-06-03T08:27:17.624 INFO:teuthology.orchestra.run.smithi088.stdout: ceph-mgr-cephadm-2:18.2.2-1389.gf1abf795.el9.noarch 2024-06-03T08:27:17.624 INFO:teuthology.orchestra.run.smithi088.stdout: python3-asyncssh-2.13.2-5.el9.noarch 2024-06-03T08:27:17.624 INFO:teuthology.orchestra.run.smithi088.stdout: python3-babel-2.9.1-2.el9.noarch 2024-06-03T08:27:17.624 INFO:teuthology.orchestra.run.smithi088.stdout: python3-jinja2-2.11.3-6.el9.noarch 2024-06-03T08:27:17.624 INFO:teuthology.orchestra.run.smithi088.stdout: python3-natsort-7.1.1-5.el9.noarch 2024-06-03T08:27:17.624 INFO:teuthology.orchestra.run.smithi088.stdout: python3-typing-extensions-4.4.0-2.el9.noarch 2024-06-03T08:27:17.625 INFO:teuthology.orchestra.run.smithi088.stdout: 2024-06-03T08:27:17.625 INFO:teuthology.orchestra.run.smithi088.stdout:Complete! 2024-06-03T08:27:17.796 INFO:teuthology.orchestra.run.smithi192.stdout: Verifying : python3-websocket-client-1.2.3-2.el9.noarch 13/13 2024-06-03T08:27:17.796 INFO:teuthology.orchestra.run.smithi192.stdout: 2024-06-03T08:27:17.796 INFO:teuthology.orchestra.run.smithi192.stdout:Installed: 2024-06-03T08:27:17.796 INFO:teuthology.orchestra.run.smithi192.stdout: ceph-mgr-rook-2:18.2.2-1389.gf1abf795.el9.noarch 2024-06-03T08:27:17.797 INFO:teuthology.orchestra.run.smithi192.stdout: python3-cachetools-4.2.4-1.el9.noarch 2024-06-03T08:27:17.797 INFO:teuthology.orchestra.run.smithi192.stdout: python3-certifi-2021.10.8-2.el9.noarch 2024-06-03T08:27:17.797 INFO:teuthology.orchestra.run.smithi192.stdout: python3-google-auth-1:2.29.0-1.el9.noarch 2024-06-03T08:27:17.797 INFO:teuthology.orchestra.run.smithi192.stdout: python3-jsonpatch-1.21-16.el9.noarch 2024-06-03T08:27:17.797 INFO:teuthology.orchestra.run.smithi192.stdout: python3-jsonpointer-2.0-4.el9.noarch 2024-06-03T08:27:17.797 INFO:teuthology.orchestra.run.smithi192.stdout: python3-kubernetes-1:26.1.0-2.el9.noarch 2024-06-03T08:27:17.797 INFO:teuthology.orchestra.run.smithi192.stdout: python3-oauthlib-3.1.1-5.el9.noarch 2024-06-03T08:27:17.797 INFO:teuthology.orchestra.run.smithi192.stdout: python3-pyasn1-0.4.8-6.el9.noarch 2024-06-03T08:27:17.797 INFO:teuthology.orchestra.run.smithi192.stdout: python3-pyasn1-modules-0.4.8-6.el9.noarch 2024-06-03T08:27:17.797 INFO:teuthology.orchestra.run.smithi192.stdout: python3-requests-oauthlib-1.3.0-12.el9.noarch 2024-06-03T08:27:17.797 INFO:teuthology.orchestra.run.smithi192.stdout: python3-rsa-4.9-2.el9.noarch 2024-06-03T08:27:17.797 INFO:teuthology.orchestra.run.smithi192.stdout: python3-websocket-client-1.2.3-2.el9.noarch 2024-06-03T08:27:17.797 INFO:teuthology.orchestra.run.smithi192.stdout: 2024-06-03T08:27:17.797 INFO:teuthology.orchestra.run.smithi192.stdout:Complete! 2024-06-03T08:27:17.879 DEBUG:teuthology.orchestra.run.smithi088:> sudo yum -y install ceph-fuse 2024-06-03T08:27:18.061 DEBUG:teuthology.orchestra.run.smithi192:> sudo yum -y install ceph-mgr-cephadm 2024-06-03T08:27:18.453 INFO:teuthology.orchestra.run.smithi088.stdout:Last metadata expiration check: 0:01:22 ago on Mon 03 Jun 2024 08:25:56 AM UTC. 2024-06-03T08:27:18.574 INFO:teuthology.orchestra.run.smithi088.stdout:Dependencies resolved. 2024-06-03T08:27:18.574 INFO:teuthology.orchestra.run.smithi088.stdout:================================================================================ 2024-06-03T08:27:18.574 INFO:teuthology.orchestra.run.smithi088.stdout: Package Arch Version Repository Size 2024-06-03T08:27:18.574 INFO:teuthology.orchestra.run.smithi088.stdout:================================================================================ 2024-06-03T08:27:18.574 INFO:teuthology.orchestra.run.smithi088.stdout:Installing: 2024-06-03T08:27:18.575 INFO:teuthology.orchestra.run.smithi088.stdout: ceph-fuse x86_64 2:18.2.2-1389.gf1abf795.el9 ceph 849 k 2024-06-03T08:27:18.575 INFO:teuthology.orchestra.run.smithi088.stdout:Installing dependencies: 2024-06-03T08:27:18.575 INFO:teuthology.orchestra.run.smithi088.stdout: fuse x86_64 2.9.9-15.el9 baseos 80 k 2024-06-03T08:27:18.575 INFO:teuthology.orchestra.run.smithi088.stdout: 2024-06-03T08:27:18.575 INFO:teuthology.orchestra.run.smithi088.stdout:Transaction Summary 2024-06-03T08:27:18.575 INFO:teuthology.orchestra.run.smithi088.stdout:================================================================================ 2024-06-03T08:27:18.575 INFO:teuthology.orchestra.run.smithi088.stdout:Install 2 Packages 2024-06-03T08:27:18.575 INFO:teuthology.orchestra.run.smithi088.stdout: 2024-06-03T08:27:18.576 INFO:teuthology.orchestra.run.smithi088.stdout:Total download size: 929 k 2024-06-03T08:27:18.576 INFO:teuthology.orchestra.run.smithi088.stdout:Installed size: 2.7 M 2024-06-03T08:27:18.576 INFO:teuthology.orchestra.run.smithi088.stdout:Downloading Packages: 2024-06-03T08:27:18.647 INFO:teuthology.orchestra.run.smithi192.stdout:Last metadata expiration check: 0:01:18 ago on Mon 03 Jun 2024 08:26:00 AM UTC. 2024-06-03T08:27:18.770 INFO:teuthology.orchestra.run.smithi192.stdout:Dependencies resolved. 2024-06-03T08:27:18.771 INFO:teuthology.orchestra.run.smithi192.stdout:================================================================================ 2024-06-03T08:27:18.771 INFO:teuthology.orchestra.run.smithi192.stdout: Package Arch Version Repository Size 2024-06-03T08:27:18.771 INFO:teuthology.orchestra.run.smithi192.stdout:================================================================================ 2024-06-03T08:27:18.771 INFO:teuthology.orchestra.run.smithi192.stdout:Installing: 2024-06-03T08:27:18.771 INFO:teuthology.orchestra.run.smithi192.stdout: ceph-mgr-cephadm noarch 2:18.2.2-1389.gf1abf795.el9 ceph-noarch 139 k 2024-06-03T08:27:18.771 INFO:teuthology.orchestra.run.smithi192.stdout:Installing dependencies: 2024-06-03T08:27:18.771 INFO:teuthology.orchestra.run.smithi192.stdout: python3-asyncssh noarch 2.13.2-5.el9 epel 548 k 2024-06-03T08:27:18.771 INFO:teuthology.orchestra.run.smithi192.stdout: python3-babel noarch 2.9.1-2.el9 appstream 6.0 M 2024-06-03T08:27:18.771 INFO:teuthology.orchestra.run.smithi192.stdout: python3-jinja2 noarch 2.11.3-6.el9 appstream 248 k 2024-06-03T08:27:18.771 INFO:teuthology.orchestra.run.smithi192.stdout: python3-natsort noarch 7.1.1-5.el9 epel 58 k 2024-06-03T08:27:18.771 INFO:teuthology.orchestra.run.smithi192.stdout: python3-typing-extensions noarch 4.4.0-2.el9 epel 51 k 2024-06-03T08:27:18.772 INFO:teuthology.orchestra.run.smithi192.stdout: 2024-06-03T08:27:18.772 INFO:teuthology.orchestra.run.smithi192.stdout:Transaction Summary 2024-06-03T08:27:18.772 INFO:teuthology.orchestra.run.smithi192.stdout:================================================================================ 2024-06-03T08:27:18.772 INFO:teuthology.orchestra.run.smithi192.stdout:Install 6 Packages 2024-06-03T08:27:18.772 INFO:teuthology.orchestra.run.smithi192.stdout: 2024-06-03T08:27:18.772 INFO:teuthology.orchestra.run.smithi192.stdout:Total download size: 7.0 M 2024-06-03T08:27:18.773 INFO:teuthology.orchestra.run.smithi192.stdout:Installed size: 33 M 2024-06-03T08:27:18.773 INFO:teuthology.orchestra.run.smithi192.stdout:Downloading Packages: 2024-06-03T08:27:19.042 INFO:teuthology.orchestra.run.smithi088.stdout:(1/2): fuse-2.9.9-15.el9.x86_64.rpm 425 kB/s | 80 kB 00:00 2024-06-03T08:27:19.192 INFO:teuthology.orchestra.run.smithi088.stdout:(2/2): ceph-fuse-18.2.2-1389.gf1abf795.el9.x86_ 2.4 MB/s | 849 kB 00:00 2024-06-03T08:27:19.193 INFO:teuthology.orchestra.run.smithi088.stdout:-------------------------------------------------------------------------------- 2024-06-03T08:27:19.193 INFO:teuthology.orchestra.run.smithi088.stdout:Total 1.5 MB/s | 929 kB 00:00 2024-06-03T08:27:19.205 INFO:teuthology.orchestra.run.smithi088.stdout:Running transaction check 2024-06-03T08:27:19.227 INFO:teuthology.orchestra.run.smithi088.stdout:Transaction check succeeded. 2024-06-03T08:27:19.227 INFO:teuthology.orchestra.run.smithi088.stdout:Running transaction test 2024-06-03T08:27:19.305 INFO:teuthology.orchestra.run.smithi088.stdout:Transaction test succeeded. 2024-06-03T08:27:19.306 INFO:teuthology.orchestra.run.smithi088.stdout:Running transaction 2024-06-03T08:27:19.494 INFO:teuthology.orchestra.run.smithi088.stdout: Preparing : 1/1 2024-06-03T08:27:19.593 INFO:teuthology.orchestra.run.smithi192.stdout:(1/6): ceph-mgr-cephadm-18.2.2-1389.gf1abf795.e 655 kB/s | 139 kB 00:00 2024-06-03T08:27:19.631 INFO:teuthology.orchestra.run.smithi088.stdout: Installing : fuse-2.9.9-15.el9.x86_64 1/2 2024-06-03T08:27:19.700 INFO:teuthology.orchestra.run.smithi088.stdout: Installing : ceph-fuse-2:18.2.2-1389.gf1abf795.el9.x86_64 2/2 2024-06-03T08:27:19.868 INFO:teuthology.orchestra.run.smithi192.stdout:(2/6): python3-asyncssh-2.13.2-5.el9.noarch.rpm 1.9 MB/s | 548 kB 00:00 2024-06-03T08:27:19.926 INFO:teuthology.orchestra.run.smithi192.stdout:(3/6): python3-natsort-7.1.1-5.el9.noarch.rpm 992 kB/s | 58 kB 00:00 2024-06-03T08:27:19.976 INFO:teuthology.orchestra.run.smithi192.stdout:(4/6): python3-typing-extensions-4.4.0-2.el9.no 1.0 MB/s | 51 kB 00:00 2024-06-03T08:27:20.210 INFO:teuthology.orchestra.run.smithi192.stdout:(5/6): python3-jinja2-2.11.3-6.el9.noarch.rpm 299 kB/s | 248 kB 00:00 2024-06-03T08:27:20.352 INFO:teuthology.orchestra.run.smithi088.stdout: Running scriptlet: ceph-fuse-2:18.2.2-1389.gf1abf795.el9.x86_64 2/2 2024-06-03T08:27:20.353 INFO:teuthology.orchestra.run.smithi088.stdout: Verifying : ceph-fuse-2:18.2.2-1389.gf1abf795.el9.x86_64 1/2 2024-06-03T08:27:20.625 INFO:teuthology.orchestra.run.smithi088.stdout: Verifying : fuse-2.9.9-15.el9.x86_64 2/2 2024-06-03T08:27:20.626 INFO:teuthology.orchestra.run.smithi088.stdout: 2024-06-03T08:27:20.626 INFO:teuthology.orchestra.run.smithi088.stdout:Installed: 2024-06-03T08:27:20.626 INFO:teuthology.orchestra.run.smithi088.stdout: ceph-fuse-2:18.2.2-1389.gf1abf795.el9.x86_64 fuse-2.9.9-15.el9.x86_64 2024-06-03T08:27:20.626 INFO:teuthology.orchestra.run.smithi088.stdout: 2024-06-03T08:27:20.626 INFO:teuthology.orchestra.run.smithi088.stdout:Complete! 2024-06-03T08:27:20.890 DEBUG:teuthology.orchestra.run.smithi088:> sudo yum -y install ceph-volume 2024-06-03T08:27:21.475 INFO:teuthology.orchestra.run.smithi088.stdout:Last metadata expiration check: 0:01:25 ago on Mon 03 Jun 2024 08:25:56 AM UTC. 2024-06-03T08:27:21.602 INFO:teuthology.orchestra.run.smithi088.stdout:Dependencies resolved. 2024-06-03T08:27:21.602 INFO:teuthology.orchestra.run.smithi088.stdout:================================================================================ 2024-06-03T08:27:21.603 INFO:teuthology.orchestra.run.smithi088.stdout: Package Arch Version Repository Size 2024-06-03T08:27:21.603 INFO:teuthology.orchestra.run.smithi088.stdout:================================================================================ 2024-06-03T08:27:21.603 INFO:teuthology.orchestra.run.smithi088.stdout:Installing: 2024-06-03T08:27:21.603 INFO:teuthology.orchestra.run.smithi088.stdout: ceph-volume noarch 2:18.2.2-1389.gf1abf795.el9 ceph-noarch 264 k 2024-06-03T08:27:21.603 INFO:teuthology.orchestra.run.smithi088.stdout:Installing dependencies: 2024-06-03T08:27:21.603 INFO:teuthology.orchestra.run.smithi088.stdout: python3-packaging noarch 20.9-5.el9 appstream 77 k 2024-06-03T08:27:21.603 INFO:teuthology.orchestra.run.smithi088.stdout: 2024-06-03T08:27:21.603 INFO:teuthology.orchestra.run.smithi088.stdout:Transaction Summary 2024-06-03T08:27:21.603 INFO:teuthology.orchestra.run.smithi088.stdout:================================================================================ 2024-06-03T08:27:21.603 INFO:teuthology.orchestra.run.smithi088.stdout:Install 2 Packages 2024-06-03T08:27:21.604 INFO:teuthology.orchestra.run.smithi088.stdout: 2024-06-03T08:27:21.604 INFO:teuthology.orchestra.run.smithi088.stdout:Total download size: 341 k 2024-06-03T08:27:21.604 INFO:teuthology.orchestra.run.smithi088.stdout:Installed size: 1.5 M 2024-06-03T08:27:21.604 INFO:teuthology.orchestra.run.smithi088.stdout:Downloading Packages: 2024-06-03T08:27:21.996 INFO:teuthology.orchestra.run.smithi088.stdout:(1/2): python3-packaging-20.9-5.el9.noarch.rpm 602 kB/s | 77 kB 00:00 2024-06-03T08:27:22.297 INFO:teuthology.orchestra.run.smithi088.stdout:(2/2): ceph-volume-18.2.2-1389.gf1abf795.el9.no 615 kB/s | 264 kB 00:00 2024-06-03T08:27:22.297 INFO:teuthology.orchestra.run.smithi088.stdout:-------------------------------------------------------------------------------- 2024-06-03T08:27:22.298 INFO:teuthology.orchestra.run.smithi088.stdout:Total 491 kB/s | 341 kB 00:00 2024-06-03T08:27:22.309 INFO:teuthology.orchestra.run.smithi088.stdout:Running transaction check 2024-06-03T08:27:22.319 INFO:teuthology.orchestra.run.smithi088.stdout:Transaction check succeeded. 2024-06-03T08:27:22.319 INFO:teuthology.orchestra.run.smithi088.stdout:Running transaction test 2024-06-03T08:27:22.349 INFO:teuthology.orchestra.run.smithi088.stdout:Transaction test succeeded. 2024-06-03T08:27:22.350 INFO:teuthology.orchestra.run.smithi088.stdout:Running transaction 2024-06-03T08:27:22.495 INFO:teuthology.orchestra.run.smithi088.stdout: Preparing : 1/1 2024-06-03T08:27:22.670 INFO:teuthology.orchestra.run.smithi088.stdout: Installing : python3-packaging-20.9-5.el9.noarch 1/2 2024-06-03T08:27:22.716 INFO:teuthology.orchestra.run.smithi088.stdout: Installing : ceph-volume-2:18.2.2-1389.gf1abf795.el9.noarch 2/2 2024-06-03T08:27:22.732 INFO:teuthology.orchestra.run.smithi088.stdout: Running scriptlet: ceph-volume-2:18.2.2-1389.gf1abf795.el9.noarch 2/2 2024-06-03T08:27:22.732 INFO:teuthology.orchestra.run.smithi088.stdout:Glob pattern passed to enable, but globs are not supported for this. 2024-06-03T08:27:22.732 INFO:teuthology.orchestra.run.smithi088.stdout:Invalid unit name "ceph-volume@*.service" escaped as "ceph-volume@\x2a.service". 2024-06-03T08:27:22.732 INFO:teuthology.orchestra.run.smithi088.stdout: 2024-06-03T08:27:23.498 INFO:teuthology.orchestra.run.smithi088.stdout: Verifying : ceph-volume-2:18.2.2-1389.gf1abf795.el9.noarch 1/2 2024-06-03T08:27:23.710 INFO:teuthology.orchestra.run.smithi192.stdout:(6/6): python3-babel-2.9.1-2.el9.noarch.rpm 1.4 MB/s | 6.0 MB 00:04 2024-06-03T08:27:23.711 INFO:teuthology.orchestra.run.smithi192.stdout:-------------------------------------------------------------------------------- 2024-06-03T08:27:23.712 INFO:teuthology.orchestra.run.smithi192.stdout:Total 1.4 MB/s | 7.0 MB 00:04 2024-06-03T08:27:23.784 INFO:teuthology.orchestra.run.smithi192.stdout:Running transaction check 2024-06-03T08:27:23.795 INFO:teuthology.orchestra.run.smithi088.stdout: Verifying : python3-packaging-20.9-5.el9.noarch 2/2 2024-06-03T08:27:23.795 INFO:teuthology.orchestra.run.smithi088.stdout: 2024-06-03T08:27:23.795 INFO:teuthology.orchestra.run.smithi088.stdout:Installed: 2024-06-03T08:27:23.795 INFO:teuthology.orchestra.run.smithi088.stdout: ceph-volume-2:18.2.2-1389.gf1abf795.el9.noarch 2024-06-03T08:27:23.795 INFO:teuthology.orchestra.run.smithi088.stdout: python3-packaging-20.9-5.el9.noarch 2024-06-03T08:27:23.795 INFO:teuthology.orchestra.run.smithi088.stdout: 2024-06-03T08:27:23.795 INFO:teuthology.orchestra.run.smithi088.stdout:Complete! 2024-06-03T08:27:23.796 INFO:teuthology.orchestra.run.smithi192.stdout:Transaction check succeeded. 2024-06-03T08:27:23.796 INFO:teuthology.orchestra.run.smithi192.stdout:Running transaction test 2024-06-03T08:27:23.891 INFO:teuthology.orchestra.run.smithi192.stdout:Transaction test succeeded. 2024-06-03T08:27:23.891 INFO:teuthology.orchestra.run.smithi192.stdout:Running transaction 2024-06-03T08:27:24.121 DEBUG:teuthology.orchestra.run.smithi088:> sudo yum -y install librados-devel 2024-06-03T08:27:24.192 INFO:teuthology.orchestra.run.smithi192.stdout: Preparing : 1/1 2024-06-03T08:27:24.365 INFO:teuthology.orchestra.run.smithi192.stdout: Installing : python3-typing-extensions-4.4.0-2.el9.noarch 1/6 2024-06-03T08:27:24.411 INFO:teuthology.orchestra.run.smithi192.stdout: Installing : python3-asyncssh-2.13.2-5.el9.noarch 2/6 2024-06-03T08:27:24.714 INFO:teuthology.orchestra.run.smithi088.stdout:Last metadata expiration check: 0:01:28 ago on Mon 03 Jun 2024 08:25:56 AM UTC. 2024-06-03T08:27:24.841 INFO:teuthology.orchestra.run.smithi088.stdout:Dependencies resolved. 2024-06-03T08:27:24.841 INFO:teuthology.orchestra.run.smithi088.stdout:================================================================================ 2024-06-03T08:27:24.842 INFO:teuthology.orchestra.run.smithi088.stdout: Package Arch Version Repo Size 2024-06-03T08:27:24.842 INFO:teuthology.orchestra.run.smithi088.stdout:================================================================================ 2024-06-03T08:27:24.842 INFO:teuthology.orchestra.run.smithi088.stdout:Installing: 2024-06-03T08:27:24.842 INFO:teuthology.orchestra.run.smithi088.stdout: librados-devel x86_64 2:18.2.2-1389.gf1abf795.el9 ceph 127 k 2024-06-03T08:27:24.842 INFO:teuthology.orchestra.run.smithi088.stdout: 2024-06-03T08:27:24.842 INFO:teuthology.orchestra.run.smithi088.stdout:Transaction Summary 2024-06-03T08:27:24.842 INFO:teuthology.orchestra.run.smithi088.stdout:================================================================================ 2024-06-03T08:27:24.842 INFO:teuthology.orchestra.run.smithi088.stdout:Install 1 Package 2024-06-03T08:27:24.842 INFO:teuthology.orchestra.run.smithi088.stdout: 2024-06-03T08:27:24.842 INFO:teuthology.orchestra.run.smithi088.stdout:Total download size: 127 k 2024-06-03T08:27:24.842 INFO:teuthology.orchestra.run.smithi088.stdout:Installed size: 456 k 2024-06-03T08:27:24.842 INFO:teuthology.orchestra.run.smithi088.stdout:Downloading Packages: 2024-06-03T08:27:24.919 INFO:teuthology.orchestra.run.smithi192.stdout: Installing : python3-natsort-7.1.1-5.el9.noarch 3/6 2024-06-03T08:27:25.005 INFO:teuthology.orchestra.run.smithi192.stdout: Installing : python3-babel-2.9.1-2.el9.noarch 4/6 2024-06-03T08:27:25.067 INFO:teuthology.orchestra.run.smithi192.stdout: Installing : python3-jinja2-2.11.3-6.el9.noarch 5/6 2024-06-03T08:27:25.091 INFO:teuthology.orchestra.run.smithi192.stdout: Installing : ceph-mgr-cephadm-2:18.2.2-1389.gf1abf795.el9.noarch 6/6 2024-06-03T08:27:25.152 INFO:teuthology.orchestra.run.smithi088.stdout:librados-devel-18.2.2-1389.gf1abf795.el9.x86_64 409 kB/s | 127 kB 00:00 2024-06-03T08:27:25.153 INFO:teuthology.orchestra.run.smithi088.stdout:-------------------------------------------------------------------------------- 2024-06-03T08:27:25.153 INFO:teuthology.orchestra.run.smithi088.stdout:Total 407 kB/s | 127 kB 00:00 2024-06-03T08:27:25.153 INFO:teuthology.orchestra.run.smithi088.stdout:Running transaction check 2024-06-03T08:27:25.159 INFO:teuthology.orchestra.run.smithi088.stdout:Transaction check succeeded. 2024-06-03T08:27:25.159 INFO:teuthology.orchestra.run.smithi088.stdout:Running transaction test 2024-06-03T08:27:25.221 INFO:teuthology.orchestra.run.smithi088.stdout:Transaction test succeeded. 2024-06-03T08:27:25.221 INFO:teuthology.orchestra.run.smithi088.stdout:Running transaction 2024-06-03T08:27:25.369 INFO:teuthology.orchestra.run.smithi088.stdout: Preparing : 1/1 2024-06-03T08:27:25.482 INFO:teuthology.orchestra.run.smithi088.stdout: Installing : librados-devel-2:18.2.2-1389.gf1abf795.el9.x86_64 1/1 2024-06-03T08:27:25.825 INFO:teuthology.orchestra.run.smithi192.stdout: Running scriptlet: ceph-mgr-cephadm-2:18.2.2-1389.gf1abf795.el9.noarch 6/6 2024-06-03T08:27:25.825 INFO:teuthology.orchestra.run.smithi192.stdout: Verifying : ceph-mgr-cephadm-2:18.2.2-1389.gf1abf795.el9.noarch 1/6 2024-06-03T08:27:25.825 INFO:teuthology.orchestra.run.smithi192.stdout: Verifying : python3-babel-2.9.1-2.el9.noarch 2/6 2024-06-03T08:27:25.825 INFO:teuthology.orchestra.run.smithi192.stdout: Verifying : python3-jinja2-2.11.3-6.el9.noarch 3/6 2024-06-03T08:27:25.825 INFO:teuthology.orchestra.run.smithi192.stdout: Verifying : python3-asyncssh-2.13.2-5.el9.noarch 4/6 2024-06-03T08:27:25.825 INFO:teuthology.orchestra.run.smithi192.stdout: Verifying : python3-natsort-7.1.1-5.el9.noarch 5/6 2024-06-03T08:27:25.870 INFO:teuthology.orchestra.run.smithi088.stdout: Running scriptlet: librados-devel-2:18.2.2-1389.gf1abf795.el9.x86_64 1/1 2024-06-03T08:27:26.093 INFO:teuthology.orchestra.run.smithi192.stdout: Verifying : python3-typing-extensions-4.4.0-2.el9.noarch 6/6 2024-06-03T08:27:26.093 INFO:teuthology.orchestra.run.smithi192.stdout: 2024-06-03T08:27:26.093 INFO:teuthology.orchestra.run.smithi192.stdout:Installed: 2024-06-03T08:27:26.093 INFO:teuthology.orchestra.run.smithi192.stdout: ceph-mgr-cephadm-2:18.2.2-1389.gf1abf795.el9.noarch 2024-06-03T08:27:26.093 INFO:teuthology.orchestra.run.smithi192.stdout: python3-asyncssh-2.13.2-5.el9.noarch 2024-06-03T08:27:26.093 INFO:teuthology.orchestra.run.smithi192.stdout: python3-babel-2.9.1-2.el9.noarch 2024-06-03T08:27:26.093 INFO:teuthology.orchestra.run.smithi192.stdout: python3-jinja2-2.11.3-6.el9.noarch 2024-06-03T08:27:26.093 INFO:teuthology.orchestra.run.smithi192.stdout: python3-natsort-7.1.1-5.el9.noarch 2024-06-03T08:27:26.093 INFO:teuthology.orchestra.run.smithi192.stdout: python3-typing-extensions-4.4.0-2.el9.noarch 2024-06-03T08:27:26.093 INFO:teuthology.orchestra.run.smithi192.stdout: 2024-06-03T08:27:26.093 INFO:teuthology.orchestra.run.smithi192.stdout:Complete! 2024-06-03T08:27:26.097 INFO:teuthology.orchestra.run.smithi088.stdout: Verifying : librados-devel-2:18.2.2-1389.gf1abf795.el9.x86_64 1/1 2024-06-03T08:27:26.097 INFO:teuthology.orchestra.run.smithi088.stdout: 2024-06-03T08:27:26.097 INFO:teuthology.orchestra.run.smithi088.stdout:Installed: 2024-06-03T08:27:26.098 INFO:teuthology.orchestra.run.smithi088.stdout: librados-devel-2:18.2.2-1389.gf1abf795.el9.x86_64 2024-06-03T08:27:26.098 INFO:teuthology.orchestra.run.smithi088.stdout: 2024-06-03T08:27:26.098 INFO:teuthology.orchestra.run.smithi088.stdout:Complete! 2024-06-03T08:27:26.332 DEBUG:teuthology.orchestra.run.smithi192:> sudo yum -y install ceph-fuse 2024-06-03T08:27:26.354 DEBUG:teuthology.orchestra.run.smithi088:> sudo yum -y install libcephfs2 2024-06-03T08:27:26.911 INFO:teuthology.orchestra.run.smithi192.stdout:Last metadata expiration check: 0:01:26 ago on Mon 03 Jun 2024 08:26:00 AM UTC. 2024-06-03T08:27:26.951 INFO:teuthology.orchestra.run.smithi088.stdout:Last metadata expiration check: 0:01:30 ago on Mon 03 Jun 2024 08:25:56 AM UTC. 2024-06-03T08:27:27.035 INFO:teuthology.orchestra.run.smithi088.stdout:Package libcephfs2-2:18.2.2-1389.gf1abf795.el9.x86_64 is already installed. 2024-06-03T08:27:27.035 INFO:teuthology.orchestra.run.smithi192.stdout:Dependencies resolved. 2024-06-03T08:27:27.036 INFO:teuthology.orchestra.run.smithi192.stdout:================================================================================ 2024-06-03T08:27:27.036 INFO:teuthology.orchestra.run.smithi192.stdout: Package Arch Version Repository Size 2024-06-03T08:27:27.036 INFO:teuthology.orchestra.run.smithi192.stdout:================================================================================ 2024-06-03T08:27:27.036 INFO:teuthology.orchestra.run.smithi192.stdout:Installing: 2024-06-03T08:27:27.036 INFO:teuthology.orchestra.run.smithi192.stdout: ceph-fuse x86_64 2:18.2.2-1389.gf1abf795.el9 ceph 849 k 2024-06-03T08:27:27.036 INFO:teuthology.orchestra.run.smithi192.stdout:Installing dependencies: 2024-06-03T08:27:27.036 INFO:teuthology.orchestra.run.smithi192.stdout: fuse x86_64 2.9.9-15.el9 baseos 80 k 2024-06-03T08:27:27.036 INFO:teuthology.orchestra.run.smithi192.stdout: 2024-06-03T08:27:27.037 INFO:teuthology.orchestra.run.smithi192.stdout:Transaction Summary 2024-06-03T08:27:27.037 INFO:teuthology.orchestra.run.smithi192.stdout:================================================================================ 2024-06-03T08:27:27.037 INFO:teuthology.orchestra.run.smithi192.stdout:Install 2 Packages 2024-06-03T08:27:27.037 INFO:teuthology.orchestra.run.smithi192.stdout: 2024-06-03T08:27:27.037 INFO:teuthology.orchestra.run.smithi192.stdout:Total download size: 929 k 2024-06-03T08:27:27.037 INFO:teuthology.orchestra.run.smithi192.stdout:Installed size: 2.7 M 2024-06-03T08:27:27.037 INFO:teuthology.orchestra.run.smithi192.stdout:Downloading Packages: 2024-06-03T08:27:27.079 INFO:teuthology.orchestra.run.smithi088.stdout:Dependencies resolved. 2024-06-03T08:27:27.080 INFO:teuthology.orchestra.run.smithi088.stdout:Nothing to do. 2024-06-03T08:27:27.080 INFO:teuthology.orchestra.run.smithi088.stdout:Complete! 2024-06-03T08:27:27.147 DEBUG:teuthology.orchestra.run.smithi088:> sudo yum -y install libcephfs-devel 2024-06-03T08:27:27.212 INFO:teuthology.orchestra.run.smithi192.stdout:(1/2): fuse-2.9.9-15.el9.x86_64.rpm 703 kB/s | 80 kB 00:00 2024-06-03T08:27:27.529 INFO:teuthology.orchestra.run.smithi192.stdout:(2/2): ceph-fuse-18.2.2-1389.gf1abf795.el9.x86_ 1.9 MB/s | 849 kB 00:00 2024-06-03T08:27:27.530 INFO:teuthology.orchestra.run.smithi192.stdout:-------------------------------------------------------------------------------- 2024-06-03T08:27:27.530 INFO:teuthology.orchestra.run.smithi192.stdout:Total 1.8 MB/s | 929 kB 00:00 2024-06-03T08:27:27.541 INFO:teuthology.orchestra.run.smithi192.stdout:Running transaction check 2024-06-03T08:27:27.563 INFO:teuthology.orchestra.run.smithi192.stdout:Transaction check succeeded. 2024-06-03T08:27:27.563 INFO:teuthology.orchestra.run.smithi192.stdout:Running transaction test 2024-06-03T08:27:27.642 INFO:teuthology.orchestra.run.smithi192.stdout:Transaction test succeeded. 2024-06-03T08:27:27.642 INFO:teuthology.orchestra.run.smithi192.stdout:Running transaction 2024-06-03T08:27:27.720 INFO:teuthology.orchestra.run.smithi088.stdout:Last metadata expiration check: 0:01:31 ago on Mon 03 Jun 2024 08:25:56 AM UTC. 2024-06-03T08:27:27.824 INFO:teuthology.orchestra.run.smithi192.stdout: Preparing : 1/1 2024-06-03T08:27:27.844 INFO:teuthology.orchestra.run.smithi088.stdout:Dependencies resolved. 2024-06-03T08:27:27.844 INFO:teuthology.orchestra.run.smithi088.stdout:================================================================================ 2024-06-03T08:27:27.844 INFO:teuthology.orchestra.run.smithi088.stdout: Package Arch Version Repo Size 2024-06-03T08:27:27.844 INFO:teuthology.orchestra.run.smithi088.stdout:================================================================================ 2024-06-03T08:27:27.844 INFO:teuthology.orchestra.run.smithi088.stdout:Installing: 2024-06-03T08:27:27.845 INFO:teuthology.orchestra.run.smithi088.stdout: libcephfs-devel x86_64 2:18.2.2-1389.gf1abf795.el9 ceph 31 k 2024-06-03T08:27:27.845 INFO:teuthology.orchestra.run.smithi088.stdout: 2024-06-03T08:27:27.845 INFO:teuthology.orchestra.run.smithi088.stdout:Transaction Summary 2024-06-03T08:27:27.845 INFO:teuthology.orchestra.run.smithi088.stdout:================================================================================ 2024-06-03T08:27:27.845 INFO:teuthology.orchestra.run.smithi088.stdout:Install 1 Package 2024-06-03T08:27:27.845 INFO:teuthology.orchestra.run.smithi088.stdout: 2024-06-03T08:27:27.845 INFO:teuthology.orchestra.run.smithi088.stdout:Total download size: 31 k 2024-06-03T08:27:27.846 INFO:teuthology.orchestra.run.smithi088.stdout:Installed size: 139 k 2024-06-03T08:27:27.846 INFO:teuthology.orchestra.run.smithi088.stdout:Downloading Packages: 2024-06-03T08:27:27.961 INFO:teuthology.orchestra.run.smithi192.stdout: Installing : fuse-2.9.9-15.el9.x86_64 1/2 2024-06-03T08:27:28.014 INFO:teuthology.orchestra.run.smithi088.stdout:libcephfs-devel-18.2.2-1389.gf1abf795.el9.x86_6 184 kB/s | 31 kB 00:00 2024-06-03T08:27:28.015 INFO:teuthology.orchestra.run.smithi088.stdout:-------------------------------------------------------------------------------- 2024-06-03T08:27:28.015 INFO:teuthology.orchestra.run.smithi088.stdout:Total 183 kB/s | 31 kB 00:00 2024-06-03T08:27:28.015 INFO:teuthology.orchestra.run.smithi088.stdout:Running transaction check 2024-06-03T08:27:28.018 INFO:teuthology.orchestra.run.smithi088.stdout:Transaction check succeeded. 2024-06-03T08:27:28.019 INFO:teuthology.orchestra.run.smithi088.stdout:Running transaction test 2024-06-03T08:27:28.028 INFO:teuthology.orchestra.run.smithi192.stdout: Installing : ceph-fuse-2:18.2.2-1389.gf1abf795.el9.x86_64 2/2 2024-06-03T08:27:28.040 INFO:teuthology.orchestra.run.smithi088.stdout:Transaction test succeeded. 2024-06-03T08:27:28.040 INFO:teuthology.orchestra.run.smithi088.stdout:Running transaction 2024-06-03T08:27:28.116 INFO:teuthology.orchestra.run.smithi088.stdout: Preparing : 1/1 2024-06-03T08:27:28.213 INFO:teuthology.orchestra.run.smithi088.stdout: Installing : libcephfs-devel-2:18.2.2-1389.gf1abf795.el9.x86_64 1/1 2024-06-03T08:27:28.592 INFO:teuthology.orchestra.run.smithi088.stdout: Running scriptlet: libcephfs-devel-2:18.2.2-1389.gf1abf795.el9.x86_64 1/1 2024-06-03T08:27:28.642 INFO:teuthology.orchestra.run.smithi192.stdout: Running scriptlet: ceph-fuse-2:18.2.2-1389.gf1abf795.el9.x86_64 2/2 2024-06-03T08:27:28.643 INFO:teuthology.orchestra.run.smithi192.stdout: Verifying : ceph-fuse-2:18.2.2-1389.gf1abf795.el9.x86_64 1/2 2024-06-03T08:27:28.804 INFO:teuthology.orchestra.run.smithi088.stdout: Verifying : libcephfs-devel-2:18.2.2-1389.gf1abf795.el9.x86_64 1/1 2024-06-03T08:27:28.805 INFO:teuthology.orchestra.run.smithi088.stdout: 2024-06-03T08:27:28.805 INFO:teuthology.orchestra.run.smithi088.stdout:Installed: 2024-06-03T08:27:28.805 INFO:teuthology.orchestra.run.smithi088.stdout: libcephfs-devel-2:18.2.2-1389.gf1abf795.el9.x86_64 2024-06-03T08:27:28.805 INFO:teuthology.orchestra.run.smithi088.stdout: 2024-06-03T08:27:28.805 INFO:teuthology.orchestra.run.smithi088.stdout:Complete! 2024-06-03T08:27:28.892 INFO:teuthology.orchestra.run.smithi192.stdout: Verifying : fuse-2.9.9-15.el9.x86_64 2/2 2024-06-03T08:27:28.892 INFO:teuthology.orchestra.run.smithi192.stdout: 2024-06-03T08:27:28.892 INFO:teuthology.orchestra.run.smithi192.stdout:Installed: 2024-06-03T08:27:28.892 INFO:teuthology.orchestra.run.smithi192.stdout: ceph-fuse-2:18.2.2-1389.gf1abf795.el9.x86_64 fuse-2.9.9-15.el9.x86_64 2024-06-03T08:27:28.892 INFO:teuthology.orchestra.run.smithi192.stdout: 2024-06-03T08:27:28.892 INFO:teuthology.orchestra.run.smithi192.stdout:Complete! 2024-06-03T08:27:29.005 DEBUG:teuthology.orchestra.run.smithi088:> sudo yum -y install librados2 2024-06-03T08:27:29.138 DEBUG:teuthology.orchestra.run.smithi192:> sudo yum -y install ceph-volume 2024-06-03T08:27:29.580 INFO:teuthology.orchestra.run.smithi088.stdout:Last metadata expiration check: 0:01:33 ago on Mon 03 Jun 2024 08:25:56 AM UTC. 2024-06-03T08:27:29.661 INFO:teuthology.orchestra.run.smithi088.stdout:Package librados2-2:18.2.2-1389.gf1abf795.el9.x86_64 is already installed. 2024-06-03T08:27:29.703 INFO:teuthology.orchestra.run.smithi088.stdout:Dependencies resolved. 2024-06-03T08:27:29.704 INFO:teuthology.orchestra.run.smithi088.stdout:Nothing to do. 2024-06-03T08:27:29.704 INFO:teuthology.orchestra.run.smithi088.stdout:Complete! 2024-06-03T08:27:29.734 INFO:teuthology.orchestra.run.smithi192.stdout:Last metadata expiration check: 0:01:29 ago on Mon 03 Jun 2024 08:26:00 AM UTC. 2024-06-03T08:27:29.771 DEBUG:teuthology.orchestra.run.smithi088:> sudo yum -y install librbd1 2024-06-03T08:27:29.862 INFO:teuthology.orchestra.run.smithi192.stdout:Dependencies resolved. 2024-06-03T08:27:29.862 INFO:teuthology.orchestra.run.smithi192.stdout:================================================================================ 2024-06-03T08:27:29.862 INFO:teuthology.orchestra.run.smithi192.stdout: Package Arch Version Repository Size 2024-06-03T08:27:29.862 INFO:teuthology.orchestra.run.smithi192.stdout:================================================================================ 2024-06-03T08:27:29.863 INFO:teuthology.orchestra.run.smithi192.stdout:Installing: 2024-06-03T08:27:29.863 INFO:teuthology.orchestra.run.smithi192.stdout: ceph-volume noarch 2:18.2.2-1389.gf1abf795.el9 ceph-noarch 264 k 2024-06-03T08:27:29.863 INFO:teuthology.orchestra.run.smithi192.stdout:Installing dependencies: 2024-06-03T08:27:29.863 INFO:teuthology.orchestra.run.smithi192.stdout: python3-packaging noarch 20.9-5.el9 appstream 77 k 2024-06-03T08:27:29.863 INFO:teuthology.orchestra.run.smithi192.stdout: 2024-06-03T08:27:29.863 INFO:teuthology.orchestra.run.smithi192.stdout:Transaction Summary 2024-06-03T08:27:29.863 INFO:teuthology.orchestra.run.smithi192.stdout:================================================================================ 2024-06-03T08:27:29.863 INFO:teuthology.orchestra.run.smithi192.stdout:Install 2 Packages 2024-06-03T08:27:29.863 INFO:teuthology.orchestra.run.smithi192.stdout: 2024-06-03T08:27:29.864 INFO:teuthology.orchestra.run.smithi192.stdout:Total download size: 341 k 2024-06-03T08:27:29.864 INFO:teuthology.orchestra.run.smithi192.stdout:Installed size: 1.5 M 2024-06-03T08:27:29.864 INFO:teuthology.orchestra.run.smithi192.stdout:Downloading Packages: 2024-06-03T08:27:30.283 INFO:teuthology.orchestra.run.smithi192.stdout:(1/2): python3-packaging-20.9-5.el9.noarch.rpm 526 kB/s | 77 kB 00:00 2024-06-03T08:27:30.348 INFO:teuthology.orchestra.run.smithi088.stdout:Last metadata expiration check: 0:01:34 ago on Mon 03 Jun 2024 08:25:56 AM UTC. 2024-06-03T08:27:30.430 INFO:teuthology.orchestra.run.smithi088.stdout:Package librbd1-2:18.2.2-1389.gf1abf795.el9.x86_64 is already installed. 2024-06-03T08:27:30.470 INFO:teuthology.orchestra.run.smithi088.stdout:Dependencies resolved. 2024-06-03T08:27:30.471 INFO:teuthology.orchestra.run.smithi088.stdout:Nothing to do. 2024-06-03T08:27:30.471 INFO:teuthology.orchestra.run.smithi088.stdout:Complete! 2024-06-03T08:27:30.525 INFO:teuthology.orchestra.run.smithi192.stdout:(2/2): ceph-volume-18.2.2-1389.gf1abf795.el9.no 678 kB/s | 264 kB 00:00 2024-06-03T08:27:30.526 INFO:teuthology.orchestra.run.smithi192.stdout:-------------------------------------------------------------------------------- 2024-06-03T08:27:30.526 INFO:teuthology.orchestra.run.smithi192.stdout:Total 514 kB/s | 341 kB 00:00 2024-06-03T08:27:30.538 DEBUG:teuthology.orchestra.run.smithi088:> sudo yum -y install python3-rados 2024-06-03T08:27:30.538 INFO:teuthology.orchestra.run.smithi192.stdout:Running transaction check 2024-06-03T08:27:30.547 INFO:teuthology.orchestra.run.smithi192.stdout:Transaction check succeeded. 2024-06-03T08:27:30.548 INFO:teuthology.orchestra.run.smithi192.stdout:Running transaction test 2024-06-03T08:27:30.584 INFO:teuthology.orchestra.run.smithi192.stdout:Transaction test succeeded. 2024-06-03T08:27:30.584 INFO:teuthology.orchestra.run.smithi192.stdout:Running transaction 2024-06-03T08:27:30.723 INFO:teuthology.orchestra.run.smithi192.stdout: Preparing : 1/1 2024-06-03T08:27:30.900 INFO:teuthology.orchestra.run.smithi192.stdout: Installing : python3-packaging-20.9-5.el9.noarch 1/2 2024-06-03T08:27:30.936 INFO:teuthology.orchestra.run.smithi192.stdout: Installing : ceph-volume-2:18.2.2-1389.gf1abf795.el9.noarch 2/2 2024-06-03T08:27:30.952 INFO:teuthology.orchestra.run.smithi192.stdout: Running scriptlet: ceph-volume-2:18.2.2-1389.gf1abf795.el9.noarch 2/2 2024-06-03T08:27:30.952 INFO:teuthology.orchestra.run.smithi192.stdout:Glob pattern passed to enable, but globs are not supported for this. 2024-06-03T08:27:30.952 INFO:teuthology.orchestra.run.smithi192.stdout:Invalid unit name "ceph-volume@*.service" escaped as "ceph-volume@\x2a.service". 2024-06-03T08:27:30.953 INFO:teuthology.orchestra.run.smithi192.stdout: 2024-06-03T08:27:31.120 INFO:teuthology.orchestra.run.smithi088.stdout:Last metadata expiration check: 0:01:35 ago on Mon 03 Jun 2024 08:25:56 AM UTC. 2024-06-03T08:27:31.201 INFO:teuthology.orchestra.run.smithi088.stdout:Package python3-rados-2:18.2.2-1389.gf1abf795.el9.x86_64 is already installed. 2024-06-03T08:27:31.242 INFO:teuthology.orchestra.run.smithi088.stdout:Dependencies resolved. 2024-06-03T08:27:31.243 INFO:teuthology.orchestra.run.smithi088.stdout:Nothing to do. 2024-06-03T08:27:31.243 INFO:teuthology.orchestra.run.smithi088.stdout:Complete! 2024-06-03T08:27:31.312 DEBUG:teuthology.orchestra.run.smithi088:> sudo yum -y install python3-rgw 2024-06-03T08:27:31.663 INFO:teuthology.orchestra.run.smithi192.stdout: Verifying : ceph-volume-2:18.2.2-1389.gf1abf795.el9.noarch 1/2 2024-06-03T08:27:31.892 INFO:teuthology.orchestra.run.smithi088.stdout:Last metadata expiration check: 0:01:35 ago on Mon 03 Jun 2024 08:25:56 AM UTC. 2024-06-03T08:27:31.969 INFO:teuthology.orchestra.run.smithi192.stdout: Verifying : python3-packaging-20.9-5.el9.noarch 2/2 2024-06-03T08:27:31.969 INFO:teuthology.orchestra.run.smithi192.stdout: 2024-06-03T08:27:31.969 INFO:teuthology.orchestra.run.smithi192.stdout:Installed: 2024-06-03T08:27:31.970 INFO:teuthology.orchestra.run.smithi192.stdout: ceph-volume-2:18.2.2-1389.gf1abf795.el9.noarch 2024-06-03T08:27:31.970 INFO:teuthology.orchestra.run.smithi192.stdout: python3-packaging-20.9-5.el9.noarch 2024-06-03T08:27:31.970 INFO:teuthology.orchestra.run.smithi192.stdout: 2024-06-03T08:27:31.970 INFO:teuthology.orchestra.run.smithi192.stdout:Complete! 2024-06-03T08:27:31.972 INFO:teuthology.orchestra.run.smithi088.stdout:Package python3-rgw-2:18.2.2-1389.gf1abf795.el9.x86_64 is already installed. 2024-06-03T08:27:32.014 INFO:teuthology.orchestra.run.smithi088.stdout:Dependencies resolved. 2024-06-03T08:27:32.015 INFO:teuthology.orchestra.run.smithi088.stdout:Nothing to do. 2024-06-03T08:27:32.015 INFO:teuthology.orchestra.run.smithi088.stdout:Complete! 2024-06-03T08:27:32.087 DEBUG:teuthology.orchestra.run.smithi088:> sudo yum -y install python3-cephfs 2024-06-03T08:27:32.282 DEBUG:teuthology.orchestra.run.smithi192:> sudo yum -y install librados-devel 2024-06-03T08:27:32.661 INFO:teuthology.orchestra.run.smithi088.stdout:Last metadata expiration check: 0:01:36 ago on Mon 03 Jun 2024 08:25:56 AM UTC. 2024-06-03T08:27:32.742 INFO:teuthology.orchestra.run.smithi088.stdout:Package python3-cephfs-2:18.2.2-1389.gf1abf795.el9.x86_64 is already installed. 2024-06-03T08:27:32.784 INFO:teuthology.orchestra.run.smithi088.stdout:Dependencies resolved. 2024-06-03T08:27:32.785 INFO:teuthology.orchestra.run.smithi088.stdout:Nothing to do. 2024-06-03T08:27:32.785 INFO:teuthology.orchestra.run.smithi088.stdout:Complete! 2024-06-03T08:27:32.853 DEBUG:teuthology.orchestra.run.smithi088:> sudo yum -y install python3-rbd 2024-06-03T08:27:32.872 INFO:teuthology.orchestra.run.smithi192.stdout:Last metadata expiration check: 0:01:32 ago on Mon 03 Jun 2024 08:26:00 AM UTC. 2024-06-03T08:27:32.999 INFO:teuthology.orchestra.run.smithi192.stdout:Dependencies resolved. 2024-06-03T08:27:33.000 INFO:teuthology.orchestra.run.smithi192.stdout:================================================================================ 2024-06-03T08:27:33.000 INFO:teuthology.orchestra.run.smithi192.stdout: Package Arch Version Repo Size 2024-06-03T08:27:33.000 INFO:teuthology.orchestra.run.smithi192.stdout:================================================================================ 2024-06-03T08:27:33.000 INFO:teuthology.orchestra.run.smithi192.stdout:Installing: 2024-06-03T08:27:33.000 INFO:teuthology.orchestra.run.smithi192.stdout: librados-devel x86_64 2:18.2.2-1389.gf1abf795.el9 ceph 127 k 2024-06-03T08:27:33.001 INFO:teuthology.orchestra.run.smithi192.stdout: 2024-06-03T08:27:33.001 INFO:teuthology.orchestra.run.smithi192.stdout:Transaction Summary 2024-06-03T08:27:33.001 INFO:teuthology.orchestra.run.smithi192.stdout:================================================================================ 2024-06-03T08:27:33.001 INFO:teuthology.orchestra.run.smithi192.stdout:Install 1 Package 2024-06-03T08:27:33.001 INFO:teuthology.orchestra.run.smithi192.stdout: 2024-06-03T08:27:33.001 INFO:teuthology.orchestra.run.smithi192.stdout:Total download size: 127 k 2024-06-03T08:27:33.001 INFO:teuthology.orchestra.run.smithi192.stdout:Installed size: 456 k 2024-06-03T08:27:33.001 INFO:teuthology.orchestra.run.smithi192.stdout:Downloading Packages: 2024-06-03T08:27:33.223 INFO:teuthology.orchestra.run.smithi192.stdout:librados-devel-18.2.2-1389.gf1abf795.el9.x86_64 570 kB/s | 127 kB 00:00 2024-06-03T08:27:33.224 INFO:teuthology.orchestra.run.smithi192.stdout:-------------------------------------------------------------------------------- 2024-06-03T08:27:33.224 INFO:teuthology.orchestra.run.smithi192.stdout:Total 566 kB/s | 127 kB 00:00 2024-06-03T08:27:33.224 INFO:teuthology.orchestra.run.smithi192.stdout:Running transaction check 2024-06-03T08:27:33.230 INFO:teuthology.orchestra.run.smithi192.stdout:Transaction check succeeded. 2024-06-03T08:27:33.230 INFO:teuthology.orchestra.run.smithi192.stdout:Running transaction test 2024-06-03T08:27:33.293 INFO:teuthology.orchestra.run.smithi192.stdout:Transaction test succeeded. 2024-06-03T08:27:33.293 INFO:teuthology.orchestra.run.smithi192.stdout:Running transaction 2024-06-03T08:27:33.428 INFO:teuthology.orchestra.run.smithi088.stdout:Last metadata expiration check: 0:01:37 ago on Mon 03 Jun 2024 08:25:56 AM UTC. 2024-06-03T08:27:33.471 INFO:teuthology.orchestra.run.smithi192.stdout: Preparing : 1/1 2024-06-03T08:27:33.510 INFO:teuthology.orchestra.run.smithi088.stdout:Package python3-rbd-2:18.2.2-1389.gf1abf795.el9.x86_64 is already installed. 2024-06-03T08:27:33.552 INFO:teuthology.orchestra.run.smithi088.stdout:Dependencies resolved. 2024-06-03T08:27:33.553 INFO:teuthology.orchestra.run.smithi088.stdout:Nothing to do. 2024-06-03T08:27:33.553 INFO:teuthology.orchestra.run.smithi088.stdout:Complete! 2024-06-03T08:27:33.563 INFO:teuthology.orchestra.run.smithi192.stdout: Installing : librados-devel-2:18.2.2-1389.gf1abf795.el9.x86_64 1/1 2024-06-03T08:27:33.628 DEBUG:teuthology.orchestra.run.smithi088:> sudo yum -y install rbd-fuse 2024-06-03T08:27:33.977 INFO:teuthology.orchestra.run.smithi192.stdout: Running scriptlet: librados-devel-2:18.2.2-1389.gf1abf795.el9.x86_64 1/1 2024-06-03T08:27:34.205 INFO:teuthology.orchestra.run.smithi088.stdout:Last metadata expiration check: 0:01:38 ago on Mon 03 Jun 2024 08:25:56 AM UTC. 2024-06-03T08:27:34.232 INFO:teuthology.orchestra.run.smithi192.stdout: Verifying : librados-devel-2:18.2.2-1389.gf1abf795.el9.x86_64 1/1 2024-06-03T08:27:34.233 INFO:teuthology.orchestra.run.smithi192.stdout: 2024-06-03T08:27:34.233 INFO:teuthology.orchestra.run.smithi192.stdout:Installed: 2024-06-03T08:27:34.233 INFO:teuthology.orchestra.run.smithi192.stdout: librados-devel-2:18.2.2-1389.gf1abf795.el9.x86_64 2024-06-03T08:27:34.233 INFO:teuthology.orchestra.run.smithi192.stdout: 2024-06-03T08:27:34.233 INFO:teuthology.orchestra.run.smithi192.stdout:Complete! 2024-06-03T08:27:34.327 INFO:teuthology.orchestra.run.smithi088.stdout:Dependencies resolved. 2024-06-03T08:27:34.328 INFO:teuthology.orchestra.run.smithi088.stdout:================================================================================ 2024-06-03T08:27:34.328 INFO:teuthology.orchestra.run.smithi088.stdout: Package Architecture Version Repository Size 2024-06-03T08:27:34.328 INFO:teuthology.orchestra.run.smithi088.stdout:================================================================================ 2024-06-03T08:27:34.328 INFO:teuthology.orchestra.run.smithi088.stdout:Installing: 2024-06-03T08:27:34.328 INFO:teuthology.orchestra.run.smithi088.stdout: rbd-fuse x86_64 2:18.2.2-1389.gf1abf795.el9 ceph 87 k 2024-06-03T08:27:34.328 INFO:teuthology.orchestra.run.smithi088.stdout: 2024-06-03T08:27:34.328 INFO:teuthology.orchestra.run.smithi088.stdout:Transaction Summary 2024-06-03T08:27:34.328 INFO:teuthology.orchestra.run.smithi088.stdout:================================================================================ 2024-06-03T08:27:34.328 INFO:teuthology.orchestra.run.smithi088.stdout:Install 1 Package 2024-06-03T08:27:34.328 INFO:teuthology.orchestra.run.smithi088.stdout: 2024-06-03T08:27:34.328 INFO:teuthology.orchestra.run.smithi088.stdout:Total download size: 87 k 2024-06-03T08:27:34.328 INFO:teuthology.orchestra.run.smithi088.stdout:Installed size: 227 k 2024-06-03T08:27:34.328 INFO:teuthology.orchestra.run.smithi088.stdout:Downloading Packages: 2024-06-03T08:27:34.503 DEBUG:teuthology.orchestra.run.smithi192:> sudo yum -y install libcephfs2 2024-06-03T08:27:34.658 INFO:teuthology.orchestra.run.smithi088.stdout:rbd-fuse-18.2.2-1389.gf1abf795.el9.x86_64.rpm 263 kB/s | 87 kB 00:00 2024-06-03T08:27:34.659 INFO:teuthology.orchestra.run.smithi088.stdout:-------------------------------------------------------------------------------- 2024-06-03T08:27:34.659 INFO:teuthology.orchestra.run.smithi088.stdout:Total 262 kB/s | 87 kB 00:00 2024-06-03T08:27:34.659 INFO:teuthology.orchestra.run.smithi088.stdout:Running transaction check 2024-06-03T08:27:34.666 INFO:teuthology.orchestra.run.smithi088.stdout:Transaction check succeeded. 2024-06-03T08:27:34.666 INFO:teuthology.orchestra.run.smithi088.stdout:Running transaction test 2024-06-03T08:27:34.729 INFO:teuthology.orchestra.run.smithi088.stdout:Transaction test succeeded. 2024-06-03T08:27:34.729 INFO:teuthology.orchestra.run.smithi088.stdout:Running transaction 2024-06-03T08:27:34.870 INFO:teuthology.orchestra.run.smithi088.stdout: Preparing : 1/1 2024-06-03T08:27:34.973 INFO:teuthology.orchestra.run.smithi088.stdout: Installing : rbd-fuse-2:18.2.2-1389.gf1abf795.el9.x86_64 1/1 2024-06-03T08:27:35.101 INFO:teuthology.orchestra.run.smithi192.stdout:Last metadata expiration check: 0:01:35 ago on Mon 03 Jun 2024 08:26:00 AM UTC. 2024-06-03T08:27:35.185 INFO:teuthology.orchestra.run.smithi192.stdout:Package libcephfs2-2:18.2.2-1389.gf1abf795.el9.x86_64 is already installed. 2024-06-03T08:27:35.229 INFO:teuthology.orchestra.run.smithi192.stdout:Dependencies resolved. 2024-06-03T08:27:35.230 INFO:teuthology.orchestra.run.smithi192.stdout:Nothing to do. 2024-06-03T08:27:35.230 INFO:teuthology.orchestra.run.smithi192.stdout:Complete! 2024-06-03T08:27:35.294 DEBUG:teuthology.orchestra.run.smithi192:> sudo yum -y install libcephfs-devel 2024-06-03T08:27:35.355 INFO:teuthology.orchestra.run.smithi088.stdout: Running scriptlet: rbd-fuse-2:18.2.2-1389.gf1abf795.el9.x86_64 1/1 2024-06-03T08:27:35.605 INFO:teuthology.orchestra.run.smithi088.stdout: Verifying : rbd-fuse-2:18.2.2-1389.gf1abf795.el9.x86_64 1/1 2024-06-03T08:27:35.605 INFO:teuthology.orchestra.run.smithi088.stdout: 2024-06-03T08:27:35.605 INFO:teuthology.orchestra.run.smithi088.stdout:Installed: 2024-06-03T08:27:35.606 INFO:teuthology.orchestra.run.smithi088.stdout: rbd-fuse-2:18.2.2-1389.gf1abf795.el9.x86_64 2024-06-03T08:27:35.606 INFO:teuthology.orchestra.run.smithi088.stdout: 2024-06-03T08:27:35.606 INFO:teuthology.orchestra.run.smithi088.stdout:Complete! 2024-06-03T08:27:35.869 INFO:teuthology.orchestra.run.smithi192.stdout:Last metadata expiration check: 0:01:35 ago on Mon 03 Jun 2024 08:26:00 AM UTC. 2024-06-03T08:27:35.876 DEBUG:teuthology.orchestra.run.smithi088:> sudo yum -y install rbd-mirror 2024-06-03T08:27:35.995 INFO:teuthology.orchestra.run.smithi192.stdout:Dependencies resolved. 2024-06-03T08:27:35.996 INFO:teuthology.orchestra.run.smithi192.stdout:================================================================================ 2024-06-03T08:27:35.996 INFO:teuthology.orchestra.run.smithi192.stdout: Package Arch Version Repo Size 2024-06-03T08:27:35.996 INFO:teuthology.orchestra.run.smithi192.stdout:================================================================================ 2024-06-03T08:27:35.996 INFO:teuthology.orchestra.run.smithi192.stdout:Installing: 2024-06-03T08:27:35.996 INFO:teuthology.orchestra.run.smithi192.stdout: libcephfs-devel x86_64 2:18.2.2-1389.gf1abf795.el9 ceph 31 k 2024-06-03T08:27:35.996 INFO:teuthology.orchestra.run.smithi192.stdout: 2024-06-03T08:27:35.996 INFO:teuthology.orchestra.run.smithi192.stdout:Transaction Summary 2024-06-03T08:27:35.997 INFO:teuthology.orchestra.run.smithi192.stdout:================================================================================ 2024-06-03T08:27:35.997 INFO:teuthology.orchestra.run.smithi192.stdout:Install 1 Package 2024-06-03T08:27:35.997 INFO:teuthology.orchestra.run.smithi192.stdout: 2024-06-03T08:27:35.997 INFO:teuthology.orchestra.run.smithi192.stdout:Total download size: 31 k 2024-06-03T08:27:35.997 INFO:teuthology.orchestra.run.smithi192.stdout:Installed size: 139 k 2024-06-03T08:27:35.997 INFO:teuthology.orchestra.run.smithi192.stdout:Downloading Packages: 2024-06-03T08:27:36.169 INFO:teuthology.orchestra.run.smithi192.stdout:libcephfs-devel-18.2.2-1389.gf1abf795.el9.x86_6 180 kB/s | 31 kB 00:00 2024-06-03T08:27:36.169 INFO:teuthology.orchestra.run.smithi192.stdout:-------------------------------------------------------------------------------- 2024-06-03T08:27:36.170 INFO:teuthology.orchestra.run.smithi192.stdout:Total 179 kB/s | 31 kB 00:00 2024-06-03T08:27:36.170 INFO:teuthology.orchestra.run.smithi192.stdout:Running transaction check 2024-06-03T08:27:36.173 INFO:teuthology.orchestra.run.smithi192.stdout:Transaction check succeeded. 2024-06-03T08:27:36.173 INFO:teuthology.orchestra.run.smithi192.stdout:Running transaction test 2024-06-03T08:27:36.188 INFO:teuthology.orchestra.run.smithi192.stdout:Transaction test succeeded. 2024-06-03T08:27:36.189 INFO:teuthology.orchestra.run.smithi192.stdout:Running transaction 2024-06-03T08:27:36.268 INFO:teuthology.orchestra.run.smithi192.stdout: Preparing : 1/1 2024-06-03T08:27:36.391 INFO:teuthology.orchestra.run.smithi192.stdout: Installing : libcephfs-devel-2:18.2.2-1389.gf1abf795.el9.x86_64 1/1 2024-06-03T08:27:36.472 INFO:teuthology.orchestra.run.smithi088.stdout:Last metadata expiration check: 0:01:40 ago on Mon 03 Jun 2024 08:25:56 AM UTC. 2024-06-03T08:27:36.600 INFO:teuthology.orchestra.run.smithi088.stdout:Dependencies resolved. 2024-06-03T08:27:36.601 INFO:teuthology.orchestra.run.smithi088.stdout:================================================================================ 2024-06-03T08:27:36.601 INFO:teuthology.orchestra.run.smithi088.stdout: Package Arch Version Repo Size 2024-06-03T08:27:36.601 INFO:teuthology.orchestra.run.smithi088.stdout:================================================================================ 2024-06-03T08:27:36.601 INFO:teuthology.orchestra.run.smithi088.stdout:Installing: 2024-06-03T08:27:36.601 INFO:teuthology.orchestra.run.smithi088.stdout: rbd-mirror x86_64 2:18.2.2-1389.gf1abf795.el9 ceph 3.0 M 2024-06-03T08:27:36.601 INFO:teuthology.orchestra.run.smithi088.stdout: 2024-06-03T08:27:36.601 INFO:teuthology.orchestra.run.smithi088.stdout:Transaction Summary 2024-06-03T08:27:36.602 INFO:teuthology.orchestra.run.smithi088.stdout:================================================================================ 2024-06-03T08:27:36.602 INFO:teuthology.orchestra.run.smithi088.stdout:Install 1 Package 2024-06-03T08:27:36.602 INFO:teuthology.orchestra.run.smithi088.stdout: 2024-06-03T08:27:36.602 INFO:teuthology.orchestra.run.smithi088.stdout:Total download size: 3.0 M 2024-06-03T08:27:36.602 INFO:teuthology.orchestra.run.smithi088.stdout:Installed size: 12 M 2024-06-03T08:27:36.602 INFO:teuthology.orchestra.run.smithi088.stdout:Downloading Packages: 2024-06-03T08:27:36.782 INFO:teuthology.orchestra.run.smithi192.stdout: Running scriptlet: libcephfs-devel-2:18.2.2-1389.gf1abf795.el9.x86_64 1/1 2024-06-03T08:27:37.003 INFO:teuthology.orchestra.run.smithi088.stdout:rbd-mirror-18.2.2-1389.gf1abf795.el9.x86_64.rpm 7.4 MB/s | 3.0 MB 00:00 2024-06-03T08:27:37.004 INFO:teuthology.orchestra.run.smithi088.stdout:-------------------------------------------------------------------------------- 2024-06-03T08:27:37.004 INFO:teuthology.orchestra.run.smithi088.stdout:Total 7.4 MB/s | 3.0 MB 00:00 2024-06-03T08:27:37.004 INFO:teuthology.orchestra.run.smithi088.stdout:Running transaction check 2024-06-03T08:27:37.008 INFO:teuthology.orchestra.run.smithi192.stdout: Verifying : libcephfs-devel-2:18.2.2-1389.gf1abf795.el9.x86_64 1/1 2024-06-03T08:27:37.008 INFO:teuthology.orchestra.run.smithi192.stdout: 2024-06-03T08:27:37.008 INFO:teuthology.orchestra.run.smithi192.stdout:Installed: 2024-06-03T08:27:37.008 INFO:teuthology.orchestra.run.smithi192.stdout: libcephfs-devel-2:18.2.2-1389.gf1abf795.el9.x86_64 2024-06-03T08:27:37.008 INFO:teuthology.orchestra.run.smithi192.stdout: 2024-06-03T08:27:37.008 INFO:teuthology.orchestra.run.smithi192.stdout:Complete! 2024-06-03T08:27:37.013 INFO:teuthology.orchestra.run.smithi088.stdout:Transaction check succeeded. 2024-06-03T08:27:37.013 INFO:teuthology.orchestra.run.smithi088.stdout:Running transaction test 2024-06-03T08:27:37.093 INFO:teuthology.orchestra.run.smithi088.stdout:Transaction test succeeded. 2024-06-03T08:27:37.093 INFO:teuthology.orchestra.run.smithi088.stdout:Running transaction 2024-06-03T08:27:37.210 DEBUG:teuthology.orchestra.run.smithi192:> sudo yum -y install librados2 2024-06-03T08:27:37.508 INFO:teuthology.orchestra.run.smithi088.stdout: Preparing : 1/1 2024-06-03T08:27:37.570 INFO:teuthology.orchestra.run.smithi088.stdout: Installing : rbd-mirror-2:18.2.2-1389.gf1abf795.el9.x86_64 1/1 2024-06-03T08:27:37.595 INFO:teuthology.orchestra.run.smithi088.stdout: Running scriptlet: rbd-mirror-2:18.2.2-1389.gf1abf795.el9.x86_64 1/1 2024-06-03T08:27:37.595 INFO:teuthology.orchestra.run.smithi088.stdout:Glob pattern passed to enable, but globs are not supported for this. 2024-06-03T08:27:37.595 INFO:teuthology.orchestra.run.smithi088.stdout:Invalid unit name "ceph-rbd-mirror@*.service" escaped as "ceph-rbd-mirror@\x2a.service". 2024-06-03T08:27:37.595 INFO:teuthology.orchestra.run.smithi088.stdout:Created symlink /etc/systemd/system/multi-user.target.wants/ceph-rbd-mirror.target → /usr/lib/systemd/system/ceph-rbd-mirror.target. 2024-06-03T08:27:37.595 INFO:teuthology.orchestra.run.smithi088.stdout:Created symlink /etc/systemd/system/ceph.target.wants/ceph-rbd-mirror.target → /usr/lib/systemd/system/ceph-rbd-mirror.target. 2024-06-03T08:27:37.595 INFO:teuthology.orchestra.run.smithi088.stdout: 2024-06-03T08:27:37.784 INFO:teuthology.orchestra.run.smithi192.stdout:Last metadata expiration check: 0:01:37 ago on Mon 03 Jun 2024 08:26:00 AM UTC. 2024-06-03T08:27:37.865 INFO:teuthology.orchestra.run.smithi192.stdout:Package librados2-2:18.2.2-1389.gf1abf795.el9.x86_64 is already installed. 2024-06-03T08:27:37.906 INFO:teuthology.orchestra.run.smithi192.stdout:Dependencies resolved. 2024-06-03T08:27:37.907 INFO:teuthology.orchestra.run.smithi192.stdout:Nothing to do. 2024-06-03T08:27:37.907 INFO:teuthology.orchestra.run.smithi192.stdout:Complete! 2024-06-03T08:27:37.970 DEBUG:teuthology.orchestra.run.smithi192:> sudo yum -y install librbd1 2024-06-03T08:27:38.545 INFO:teuthology.orchestra.run.smithi192.stdout:Last metadata expiration check: 0:01:38 ago on Mon 03 Jun 2024 08:26:00 AM UTC. 2024-06-03T08:27:38.627 INFO:teuthology.orchestra.run.smithi192.stdout:Package librbd1-2:18.2.2-1389.gf1abf795.el9.x86_64 is already installed. 2024-06-03T08:27:38.637 INFO:teuthology.orchestra.run.smithi088.stdout: Verifying : rbd-mirror-2:18.2.2-1389.gf1abf795.el9.x86_64 1/1 2024-06-03T08:27:38.637 INFO:teuthology.orchestra.run.smithi088.stdout: 2024-06-03T08:27:38.637 INFO:teuthology.orchestra.run.smithi088.stdout:Installed: 2024-06-03T08:27:38.637 INFO:teuthology.orchestra.run.smithi088.stdout: rbd-mirror-2:18.2.2-1389.gf1abf795.el9.x86_64 2024-06-03T08:27:38.637 INFO:teuthology.orchestra.run.smithi088.stdout: 2024-06-03T08:27:38.638 INFO:teuthology.orchestra.run.smithi088.stdout:Complete! 2024-06-03T08:27:38.669 INFO:teuthology.orchestra.run.smithi192.stdout:Dependencies resolved. 2024-06-03T08:27:38.670 INFO:teuthology.orchestra.run.smithi192.stdout:Nothing to do. 2024-06-03T08:27:38.671 INFO:teuthology.orchestra.run.smithi192.stdout:Complete! 2024-06-03T08:27:38.737 DEBUG:teuthology.orchestra.run.smithi192:> sudo yum -y install python3-rados 2024-06-03T08:27:38.884 DEBUG:teuthology.orchestra.run.smithi088:> sudo yum -y install rbd-nbd 2024-06-03T08:27:39.317 INFO:teuthology.orchestra.run.smithi192.stdout:Last metadata expiration check: 0:01:39 ago on Mon 03 Jun 2024 08:26:00 AM UTC. 2024-06-03T08:27:39.399 INFO:teuthology.orchestra.run.smithi192.stdout:Package python3-rados-2:18.2.2-1389.gf1abf795.el9.x86_64 is already installed. 2024-06-03T08:27:39.441 INFO:teuthology.orchestra.run.smithi192.stdout:Dependencies resolved. 2024-06-03T08:27:39.442 INFO:teuthology.orchestra.run.smithi192.stdout:Nothing to do. 2024-06-03T08:27:39.442 INFO:teuthology.orchestra.run.smithi192.stdout:Complete! 2024-06-03T08:27:39.477 INFO:teuthology.orchestra.run.smithi088.stdout:Last metadata expiration check: 0:01:43 ago on Mon 03 Jun 2024 08:25:56 AM UTC. 2024-06-03T08:27:39.507 DEBUG:teuthology.orchestra.run.smithi192:> sudo yum -y install python3-rgw 2024-06-03T08:27:39.607 INFO:teuthology.orchestra.run.smithi088.stdout:Dependencies resolved. 2024-06-03T08:27:39.607 INFO:teuthology.orchestra.run.smithi088.stdout:================================================================================ 2024-06-03T08:27:39.607 INFO:teuthology.orchestra.run.smithi088.stdout: Package Architecture Version Repository Size 2024-06-03T08:27:39.607 INFO:teuthology.orchestra.run.smithi088.stdout:================================================================================ 2024-06-03T08:27:39.608 INFO:teuthology.orchestra.run.smithi088.stdout:Installing: 2024-06-03T08:27:39.608 INFO:teuthology.orchestra.run.smithi088.stdout: rbd-nbd x86_64 2:18.2.2-1389.gf1abf795.el9 ceph 172 k 2024-06-03T08:27:39.608 INFO:teuthology.orchestra.run.smithi088.stdout: 2024-06-03T08:27:39.608 INFO:teuthology.orchestra.run.smithi088.stdout:Transaction Summary 2024-06-03T08:27:39.608 INFO:teuthology.orchestra.run.smithi088.stdout:================================================================================ 2024-06-03T08:27:39.608 INFO:teuthology.orchestra.run.smithi088.stdout:Install 1 Package 2024-06-03T08:27:39.608 INFO:teuthology.orchestra.run.smithi088.stdout: 2024-06-03T08:27:39.609 INFO:teuthology.orchestra.run.smithi088.stdout:Total download size: 172 k 2024-06-03T08:27:39.609 INFO:teuthology.orchestra.run.smithi088.stdout:Installed size: 494 k 2024-06-03T08:27:39.609 INFO:teuthology.orchestra.run.smithi088.stdout:Downloading Packages: 2024-06-03T08:27:39.952 INFO:teuthology.orchestra.run.smithi088.stdout:rbd-nbd-18.2.2-1389.gf1abf795.el9.x86_64.rpm 501 kB/s | 172 kB 00:00 2024-06-03T08:27:39.953 INFO:teuthology.orchestra.run.smithi088.stdout:-------------------------------------------------------------------------------- 2024-06-03T08:27:39.953 INFO:teuthology.orchestra.run.smithi088.stdout:Total 499 kB/s | 172 kB 00:00 2024-06-03T08:27:39.953 INFO:teuthology.orchestra.run.smithi088.stdout:Running transaction check 2024-06-03T08:27:39.961 INFO:teuthology.orchestra.run.smithi088.stdout:Transaction check succeeded. 2024-06-03T08:27:39.961 INFO:teuthology.orchestra.run.smithi088.stdout:Running transaction test 2024-06-03T08:27:40.027 INFO:teuthology.orchestra.run.smithi088.stdout:Transaction test succeeded. 2024-06-03T08:27:40.027 INFO:teuthology.orchestra.run.smithi088.stdout:Running transaction 2024-06-03T08:27:40.079 INFO:teuthology.orchestra.run.smithi192.stdout:Last metadata expiration check: 0:01:40 ago on Mon 03 Jun 2024 08:26:00 AM UTC. 2024-06-03T08:27:40.162 INFO:teuthology.orchestra.run.smithi192.stdout:Package python3-rgw-2:18.2.2-1389.gf1abf795.el9.x86_64 is already installed. 2024-06-03T08:27:40.183 INFO:teuthology.orchestra.run.smithi088.stdout: Preparing : 1/1 2024-06-03T08:27:40.203 INFO:teuthology.orchestra.run.smithi192.stdout:Dependencies resolved. 2024-06-03T08:27:40.205 INFO:teuthology.orchestra.run.smithi192.stdout:Nothing to do. 2024-06-03T08:27:40.205 INFO:teuthology.orchestra.run.smithi192.stdout:Complete! 2024-06-03T08:27:40.270 DEBUG:teuthology.orchestra.run.smithi192:> sudo yum -y install python3-cephfs 2024-06-03T08:27:40.292 INFO:teuthology.orchestra.run.smithi088.stdout: Installing : rbd-nbd-2:18.2.2-1389.gf1abf795.el9.x86_64 1/1 2024-06-03T08:27:40.731 INFO:teuthology.orchestra.run.smithi088.stdout: Running scriptlet: rbd-nbd-2:18.2.2-1389.gf1abf795.el9.x86_64 1/1 2024-06-03T08:27:40.845 INFO:teuthology.orchestra.run.smithi192.stdout:Last metadata expiration check: 0:01:40 ago on Mon 03 Jun 2024 08:26:00 AM UTC. 2024-06-03T08:27:40.926 INFO:teuthology.orchestra.run.smithi192.stdout:Package python3-cephfs-2:18.2.2-1389.gf1abf795.el9.x86_64 is already installed. 2024-06-03T08:27:40.969 INFO:teuthology.orchestra.run.smithi192.stdout:Dependencies resolved. 2024-06-03T08:27:40.970 INFO:teuthology.orchestra.run.smithi192.stdout:Nothing to do. 2024-06-03T08:27:40.970 INFO:teuthology.orchestra.run.smithi192.stdout:Complete! 2024-06-03T08:27:40.987 INFO:teuthology.orchestra.run.smithi088.stdout: Verifying : rbd-nbd-2:18.2.2-1389.gf1abf795.el9.x86_64 1/1 2024-06-03T08:27:40.987 INFO:teuthology.orchestra.run.smithi088.stdout: 2024-06-03T08:27:40.987 INFO:teuthology.orchestra.run.smithi088.stdout:Installed: 2024-06-03T08:27:40.987 INFO:teuthology.orchestra.run.smithi088.stdout: rbd-nbd-2:18.2.2-1389.gf1abf795.el9.x86_64 2024-06-03T08:27:40.987 INFO:teuthology.orchestra.run.smithi088.stdout: 2024-06-03T08:27:40.987 INFO:teuthology.orchestra.run.smithi088.stdout:Complete! 2024-06-03T08:27:41.037 DEBUG:teuthology.orchestra.run.smithi192:> sudo yum -y install python3-rbd 2024-06-03T08:27:41.249 DEBUG:teuthology.orchestra.run.smithi088:> sudo yum -y install ceph-base-debuginfo 2024-06-03T08:27:41.613 INFO:teuthology.orchestra.run.smithi192.stdout:Last metadata expiration check: 0:01:41 ago on Mon 03 Jun 2024 08:26:00 AM UTC. 2024-06-03T08:27:41.694 INFO:teuthology.orchestra.run.smithi192.stdout:Package python3-rbd-2:18.2.2-1389.gf1abf795.el9.x86_64 is already installed. 2024-06-03T08:27:41.736 INFO:teuthology.orchestra.run.smithi192.stdout:Dependencies resolved. 2024-06-03T08:27:41.737 INFO:teuthology.orchestra.run.smithi192.stdout:Nothing to do. 2024-06-03T08:27:41.737 INFO:teuthology.orchestra.run.smithi192.stdout:Complete! 2024-06-03T08:27:41.806 DEBUG:teuthology.orchestra.run.smithi192:> sudo yum -y install rbd-fuse 2024-06-03T08:27:41.838 INFO:teuthology.orchestra.run.smithi088.stdout:Last metadata expiration check: 0:01:45 ago on Mon 03 Jun 2024 08:25:56 AM UTC. 2024-06-03T08:27:41.963 INFO:teuthology.orchestra.run.smithi088.stdout:Dependencies resolved. 2024-06-03T08:27:41.964 INFO:teuthology.orchestra.run.smithi088.stdout:================================================================================ 2024-06-03T08:27:41.964 INFO:teuthology.orchestra.run.smithi088.stdout: Package Arch Version Repo Size 2024-06-03T08:27:41.964 INFO:teuthology.orchestra.run.smithi088.stdout:================================================================================ 2024-06-03T08:27:41.964 INFO:teuthology.orchestra.run.smithi088.stdout:Installing: 2024-06-03T08:27:41.964 INFO:teuthology.orchestra.run.smithi088.stdout: ceph-base-debuginfo x86_64 2:18.2.2-1389.gf1abf795.el9 ceph 67 M 2024-06-03T08:27:41.964 INFO:teuthology.orchestra.run.smithi088.stdout:Installing dependencies: 2024-06-03T08:27:41.964 INFO:teuthology.orchestra.run.smithi088.stdout: ceph-debuginfo x86_64 2:18.2.2-1389.gf1abf795.el9 ceph 20 M 2024-06-03T08:27:41.964 INFO:teuthology.orchestra.run.smithi088.stdout: 2024-06-03T08:27:41.964 INFO:teuthology.orchestra.run.smithi088.stdout:Transaction Summary 2024-06-03T08:27:41.964 INFO:teuthology.orchestra.run.smithi088.stdout:================================================================================ 2024-06-03T08:27:41.965 INFO:teuthology.orchestra.run.smithi088.stdout:Install 2 Packages 2024-06-03T08:27:41.965 INFO:teuthology.orchestra.run.smithi088.stdout: 2024-06-03T08:27:41.965 INFO:teuthology.orchestra.run.smithi088.stdout:Total download size: 87 M 2024-06-03T08:27:41.965 INFO:teuthology.orchestra.run.smithi088.stdout:Installed size: 482 M 2024-06-03T08:27:41.965 INFO:teuthology.orchestra.run.smithi088.stdout:Downloading Packages: 2024-06-03T08:27:42.383 INFO:teuthology.orchestra.run.smithi192.stdout:Last metadata expiration check: 0:01:42 ago on Mon 03 Jun 2024 08:26:00 AM UTC. 2024-06-03T08:27:42.507 INFO:teuthology.orchestra.run.smithi192.stdout:Dependencies resolved. 2024-06-03T08:27:42.507 INFO:teuthology.orchestra.run.smithi192.stdout:================================================================================ 2024-06-03T08:27:42.507 INFO:teuthology.orchestra.run.smithi192.stdout: Package Architecture Version Repository Size 2024-06-03T08:27:42.507 INFO:teuthology.orchestra.run.smithi192.stdout:================================================================================ 2024-06-03T08:27:42.507 INFO:teuthology.orchestra.run.smithi192.stdout:Installing: 2024-06-03T08:27:42.508 INFO:teuthology.orchestra.run.smithi192.stdout: rbd-fuse x86_64 2:18.2.2-1389.gf1abf795.el9 ceph 87 k 2024-06-03T08:27:42.508 INFO:teuthology.orchestra.run.smithi192.stdout: 2024-06-03T08:27:42.508 INFO:teuthology.orchestra.run.smithi192.stdout:Transaction Summary 2024-06-03T08:27:42.508 INFO:teuthology.orchestra.run.smithi192.stdout:================================================================================ 2024-06-03T08:27:42.508 INFO:teuthology.orchestra.run.smithi192.stdout:Install 1 Package 2024-06-03T08:27:42.508 INFO:teuthology.orchestra.run.smithi192.stdout: 2024-06-03T08:27:42.508 INFO:teuthology.orchestra.run.smithi192.stdout:Total download size: 87 k 2024-06-03T08:27:42.508 INFO:teuthology.orchestra.run.smithi192.stdout:Installed size: 227 k 2024-06-03T08:27:42.509 INFO:teuthology.orchestra.run.smithi192.stdout:Downloading Packages: 2024-06-03T08:27:42.847 INFO:teuthology.orchestra.run.smithi192.stdout:rbd-fuse-18.2.2-1389.gf1abf795.el9.x86_64.rpm 256 kB/s | 87 kB 00:00 2024-06-03T08:27:42.847 INFO:teuthology.orchestra.run.smithi192.stdout:-------------------------------------------------------------------------------- 2024-06-03T08:27:42.848 INFO:teuthology.orchestra.run.smithi192.stdout:Total 255 kB/s | 87 kB 00:00 2024-06-03T08:27:42.848 INFO:teuthology.orchestra.run.smithi192.stdout:Running transaction check 2024-06-03T08:27:42.855 INFO:teuthology.orchestra.run.smithi192.stdout:Transaction check succeeded. 2024-06-03T08:27:42.855 INFO:teuthology.orchestra.run.smithi192.stdout:Running transaction test 2024-06-03T08:27:42.918 INFO:teuthology.orchestra.run.smithi192.stdout:Transaction test succeeded. 2024-06-03T08:27:42.919 INFO:teuthology.orchestra.run.smithi192.stdout:Running transaction 2024-06-03T08:27:43.065 INFO:teuthology.orchestra.run.smithi192.stdout: Preparing : 1/1 2024-06-03T08:27:43.169 INFO:teuthology.orchestra.run.smithi192.stdout: Installing : rbd-fuse-2:18.2.2-1389.gf1abf795.el9.x86_64 1/1 2024-06-03T08:27:43.527 INFO:teuthology.orchestra.run.smithi192.stdout: Running scriptlet: rbd-fuse-2:18.2.2-1389.gf1abf795.el9.x86_64 1/1 2024-06-03T08:27:43.778 INFO:teuthology.orchestra.run.smithi192.stdout: Verifying : rbd-fuse-2:18.2.2-1389.gf1abf795.el9.x86_64 1/1 2024-06-03T08:27:43.778 INFO:teuthology.orchestra.run.smithi192.stdout: 2024-06-03T08:27:43.778 INFO:teuthology.orchestra.run.smithi192.stdout:Installed: 2024-06-03T08:27:43.779 INFO:teuthology.orchestra.run.smithi192.stdout: rbd-fuse-2:18.2.2-1389.gf1abf795.el9.x86_64 2024-06-03T08:27:43.779 INFO:teuthology.orchestra.run.smithi192.stdout: 2024-06-03T08:27:43.779 INFO:teuthology.orchestra.run.smithi192.stdout:Complete! 2024-06-03T08:27:44.015 DEBUG:teuthology.orchestra.run.smithi192:> sudo yum -y install rbd-mirror 2024-06-03T08:27:44.612 INFO:teuthology.orchestra.run.smithi192.stdout:Last metadata expiration check: 0:01:44 ago on Mon 03 Jun 2024 08:26:00 AM UTC. 2024-06-03T08:27:44.708 INFO:teuthology.orchestra.run.smithi088.stdout:(1/2): ceph-debuginfo-18.2.2-1389.gf1abf795.el9 7.4 MB/s | 20 MB 00:02 2024-06-03T08:27:44.739 INFO:teuthology.orchestra.run.smithi192.stdout:Dependencies resolved. 2024-06-03T08:27:44.740 INFO:teuthology.orchestra.run.smithi192.stdout:================================================================================ 2024-06-03T08:27:44.740 INFO:teuthology.orchestra.run.smithi192.stdout: Package Arch Version Repo Size 2024-06-03T08:27:44.740 INFO:teuthology.orchestra.run.smithi192.stdout:================================================================================ 2024-06-03T08:27:44.740 INFO:teuthology.orchestra.run.smithi192.stdout:Installing: 2024-06-03T08:27:44.740 INFO:teuthology.orchestra.run.smithi192.stdout: rbd-mirror x86_64 2:18.2.2-1389.gf1abf795.el9 ceph 3.0 M 2024-06-03T08:27:44.740 INFO:teuthology.orchestra.run.smithi192.stdout: 2024-06-03T08:27:44.740 INFO:teuthology.orchestra.run.smithi192.stdout:Transaction Summary 2024-06-03T08:27:44.740 INFO:teuthology.orchestra.run.smithi192.stdout:================================================================================ 2024-06-03T08:27:44.740 INFO:teuthology.orchestra.run.smithi192.stdout:Install 1 Package 2024-06-03T08:27:44.740 INFO:teuthology.orchestra.run.smithi192.stdout: 2024-06-03T08:27:44.741 INFO:teuthology.orchestra.run.smithi192.stdout:Total download size: 3.0 M 2024-06-03T08:27:44.741 INFO:teuthology.orchestra.run.smithi192.stdout:Installed size: 12 M 2024-06-03T08:27:44.741 INFO:teuthology.orchestra.run.smithi192.stdout:Downloading Packages: 2024-06-03T08:27:45.983 INFO:teuthology.orchestra.run.smithi088.stdout:(2/2): ceph-base-debuginfo-18.2.2-1389.gf1abf79 17 MB/s | 67 MB 00:04 2024-06-03T08:27:45.985 INFO:teuthology.orchestra.run.smithi088.stdout:-------------------------------------------------------------------------------- 2024-06-03T08:27:45.986 INFO:teuthology.orchestra.run.smithi088.stdout:Total 22 MB/s | 87 MB 00:04 2024-06-03T08:27:45.986 INFO:teuthology.orchestra.run.smithi088.stdout:Running transaction check 2024-06-03T08:27:45.991 INFO:teuthology.orchestra.run.smithi088.stdout:Transaction check succeeded. 2024-06-03T08:27:45.991 INFO:teuthology.orchestra.run.smithi088.stdout:Running transaction test 2024-06-03T08:27:46.200 INFO:teuthology.orchestra.run.smithi192.stdout:rbd-mirror-18.2.2-1389.gf1abf795.el9.x86_64.rpm 2.0 MB/s | 3.0 MB 00:01 2024-06-03T08:27:46.201 INFO:teuthology.orchestra.run.smithi192.stdout:-------------------------------------------------------------------------------- 2024-06-03T08:27:46.201 INFO:teuthology.orchestra.run.smithi192.stdout:Total 2.0 MB/s | 3.0 MB 00:01 2024-06-03T08:27:46.201 INFO:teuthology.orchestra.run.smithi192.stdout:Running transaction check 2024-06-03T08:27:46.210 INFO:teuthology.orchestra.run.smithi192.stdout:Transaction check succeeded. 2024-06-03T08:27:46.210 INFO:teuthology.orchestra.run.smithi192.stdout:Running transaction test 2024-06-03T08:27:46.290 INFO:teuthology.orchestra.run.smithi192.stdout:Transaction test succeeded. 2024-06-03T08:27:46.290 INFO:teuthology.orchestra.run.smithi192.stdout:Running transaction 2024-06-03T08:27:46.547 INFO:teuthology.orchestra.run.smithi088.stdout:Transaction test succeeded. 2024-06-03T08:27:46.548 INFO:teuthology.orchestra.run.smithi088.stdout:Running transaction 2024-06-03T08:27:46.705 INFO:teuthology.orchestra.run.smithi192.stdout: Preparing : 1/1 2024-06-03T08:27:46.777 INFO:teuthology.orchestra.run.smithi192.stdout: Installing : rbd-mirror-2:18.2.2-1389.gf1abf795.el9.x86_64 1/1 2024-06-03T08:27:46.799 INFO:teuthology.orchestra.run.smithi192.stdout: Running scriptlet: rbd-mirror-2:18.2.2-1389.gf1abf795.el9.x86_64 1/1 2024-06-03T08:27:46.800 INFO:teuthology.orchestra.run.smithi192.stdout:Glob pattern passed to enable, but globs are not supported for this. 2024-06-03T08:27:46.800 INFO:teuthology.orchestra.run.smithi192.stdout:Invalid unit name "ceph-rbd-mirror@*.service" escaped as "ceph-rbd-mirror@\x2a.service". 2024-06-03T08:27:46.800 INFO:teuthology.orchestra.run.smithi192.stdout:Created symlink /etc/systemd/system/multi-user.target.wants/ceph-rbd-mirror.target → /usr/lib/systemd/system/ceph-rbd-mirror.target. 2024-06-03T08:27:46.800 INFO:teuthology.orchestra.run.smithi192.stdout:Created symlink /etc/systemd/system/ceph.target.wants/ceph-rbd-mirror.target → /usr/lib/systemd/system/ceph-rbd-mirror.target. 2024-06-03T08:27:46.800 INFO:teuthology.orchestra.run.smithi192.stdout: 2024-06-03T08:27:47.872 INFO:teuthology.orchestra.run.smithi192.stdout: Verifying : rbd-mirror-2:18.2.2-1389.gf1abf795.el9.x86_64 1/1 2024-06-03T08:27:47.873 INFO:teuthology.orchestra.run.smithi192.stdout: 2024-06-03T08:27:47.873 INFO:teuthology.orchestra.run.smithi192.stdout:Installed: 2024-06-03T08:27:47.873 INFO:teuthology.orchestra.run.smithi192.stdout: rbd-mirror-2:18.2.2-1389.gf1abf795.el9.x86_64 2024-06-03T08:27:47.873 INFO:teuthology.orchestra.run.smithi192.stdout: 2024-06-03T08:27:47.873 INFO:teuthology.orchestra.run.smithi192.stdout:Complete! 2024-06-03T08:27:48.134 DEBUG:teuthology.orchestra.run.smithi192:> sudo yum -y install rbd-nbd 2024-06-03T08:27:48.727 INFO:teuthology.orchestra.run.smithi192.stdout:Last metadata expiration check: 0:01:48 ago on Mon 03 Jun 2024 08:26:00 AM UTC. 2024-06-03T08:27:48.852 INFO:teuthology.orchestra.run.smithi192.stdout:Dependencies resolved. 2024-06-03T08:27:48.852 INFO:teuthology.orchestra.run.smithi192.stdout:================================================================================ 2024-06-03T08:27:48.852 INFO:teuthology.orchestra.run.smithi192.stdout: Package Architecture Version Repository Size 2024-06-03T08:27:48.852 INFO:teuthology.orchestra.run.smithi192.stdout:================================================================================ 2024-06-03T08:27:48.852 INFO:teuthology.orchestra.run.smithi192.stdout:Installing: 2024-06-03T08:27:48.852 INFO:teuthology.orchestra.run.smithi192.stdout: rbd-nbd x86_64 2:18.2.2-1389.gf1abf795.el9 ceph 172 k 2024-06-03T08:27:48.853 INFO:teuthology.orchestra.run.smithi192.stdout: 2024-06-03T08:27:48.853 INFO:teuthology.orchestra.run.smithi192.stdout:Transaction Summary 2024-06-03T08:27:48.853 INFO:teuthology.orchestra.run.smithi192.stdout:================================================================================ 2024-06-03T08:27:48.853 INFO:teuthology.orchestra.run.smithi192.stdout:Install 1 Package 2024-06-03T08:27:48.853 INFO:teuthology.orchestra.run.smithi192.stdout: 2024-06-03T08:27:48.853 INFO:teuthology.orchestra.run.smithi192.stdout:Total download size: 172 k 2024-06-03T08:27:48.853 INFO:teuthology.orchestra.run.smithi192.stdout:Installed size: 494 k 2024-06-03T08:27:48.854 INFO:teuthology.orchestra.run.smithi192.stdout:Downloading Packages: 2024-06-03T08:27:49.201 INFO:teuthology.orchestra.run.smithi192.stdout:rbd-nbd-18.2.2-1389.gf1abf795.el9.x86_64.rpm 495 kB/s | 172 kB 00:00 2024-06-03T08:27:49.202 INFO:teuthology.orchestra.run.smithi192.stdout:-------------------------------------------------------------------------------- 2024-06-03T08:27:49.202 INFO:teuthology.orchestra.run.smithi192.stdout:Total 492 kB/s | 172 kB 00:00 2024-06-03T08:27:49.202 INFO:teuthology.orchestra.run.smithi192.stdout:Running transaction check 2024-06-03T08:27:49.210 INFO:teuthology.orchestra.run.smithi192.stdout:Transaction check succeeded. 2024-06-03T08:27:49.210 INFO:teuthology.orchestra.run.smithi192.stdout:Running transaction test 2024-06-03T08:27:49.273 INFO:teuthology.orchestra.run.smithi192.stdout:Transaction test succeeded. 2024-06-03T08:27:49.274 INFO:teuthology.orchestra.run.smithi192.stdout:Running transaction 2024-06-03T08:27:49.407 INFO:teuthology.orchestra.run.smithi088.stdout: Preparing : 1/1 2024-06-03T08:27:49.423 INFO:teuthology.orchestra.run.smithi192.stdout: Preparing : 1/1 2024-06-03T08:27:49.541 INFO:teuthology.orchestra.run.smithi192.stdout: Installing : rbd-nbd-2:18.2.2-1389.gf1abf795.el9.x86_64 1/1 2024-06-03T08:27:49.974 INFO:teuthology.orchestra.run.smithi192.stdout: Running scriptlet: rbd-nbd-2:18.2.2-1389.gf1abf795.el9.x86_64 1/1 2024-06-03T08:27:50.241 INFO:teuthology.orchestra.run.smithi192.stdout: Verifying : rbd-nbd-2:18.2.2-1389.gf1abf795.el9.x86_64 1/1 2024-06-03T08:27:50.241 INFO:teuthology.orchestra.run.smithi192.stdout: 2024-06-03T08:27:50.241 INFO:teuthology.orchestra.run.smithi192.stdout:Installed: 2024-06-03T08:27:50.241 INFO:teuthology.orchestra.run.smithi192.stdout: rbd-nbd-2:18.2.2-1389.gf1abf795.el9.x86_64 2024-06-03T08:27:50.241 INFO:teuthology.orchestra.run.smithi192.stdout: 2024-06-03T08:27:50.241 INFO:teuthology.orchestra.run.smithi192.stdout:Complete! 2024-06-03T08:27:50.513 DEBUG:teuthology.orchestra.run.smithi192:> sudo yum -y install ceph-base-debuginfo 2024-06-03T08:27:51.118 INFO:teuthology.orchestra.run.smithi192.stdout:Last metadata expiration check: 0:01:51 ago on Mon 03 Jun 2024 08:26:00 AM UTC. 2024-06-03T08:27:51.246 INFO:teuthology.orchestra.run.smithi192.stdout:Dependencies resolved. 2024-06-03T08:27:51.247 INFO:teuthology.orchestra.run.smithi192.stdout:================================================================================ 2024-06-03T08:27:51.247 INFO:teuthology.orchestra.run.smithi192.stdout: Package Arch Version Repo Size 2024-06-03T08:27:51.247 INFO:teuthology.orchestra.run.smithi192.stdout:================================================================================ 2024-06-03T08:27:51.247 INFO:teuthology.orchestra.run.smithi192.stdout:Installing: 2024-06-03T08:27:51.247 INFO:teuthology.orchestra.run.smithi192.stdout: ceph-base-debuginfo x86_64 2:18.2.2-1389.gf1abf795.el9 ceph 67 M 2024-06-03T08:27:51.247 INFO:teuthology.orchestra.run.smithi192.stdout:Installing dependencies: 2024-06-03T08:27:51.247 INFO:teuthology.orchestra.run.smithi192.stdout: ceph-debuginfo x86_64 2:18.2.2-1389.gf1abf795.el9 ceph 20 M 2024-06-03T08:27:51.247 INFO:teuthology.orchestra.run.smithi192.stdout: 2024-06-03T08:27:51.247 INFO:teuthology.orchestra.run.smithi192.stdout:Transaction Summary 2024-06-03T08:27:51.247 INFO:teuthology.orchestra.run.smithi192.stdout:================================================================================ 2024-06-03T08:27:51.247 INFO:teuthology.orchestra.run.smithi192.stdout:Install 2 Packages 2024-06-03T08:27:51.248 INFO:teuthology.orchestra.run.smithi192.stdout: 2024-06-03T08:27:51.248 INFO:teuthology.orchestra.run.smithi192.stdout:Total download size: 87 M 2024-06-03T08:27:51.248 INFO:teuthology.orchestra.run.smithi192.stdout:Installed size: 482 M 2024-06-03T08:27:51.248 INFO:teuthology.orchestra.run.smithi192.stdout:Downloading Packages: 2024-06-03T08:27:54.800 INFO:teuthology.orchestra.run.smithi192.stdout:(1/2): ceph-debuginfo-18.2.2-1389.gf1abf795.el9 5.7 MB/s | 20 MB 00:03 2024-06-03T08:27:55.475 INFO:teuthology.orchestra.run.smithi192.stdout:(2/2): ceph-base-debuginfo-18.2.2-1389.gf1abf79 16 MB/s | 67 MB 00:04 2024-06-03T08:27:55.477 INFO:teuthology.orchestra.run.smithi192.stdout:-------------------------------------------------------------------------------- 2024-06-03T08:27:55.478 INFO:teuthology.orchestra.run.smithi192.stdout:Total 21 MB/s | 87 MB 00:04 2024-06-03T08:27:55.478 INFO:teuthology.orchestra.run.smithi192.stdout:Running transaction check 2024-06-03T08:27:55.483 INFO:teuthology.orchestra.run.smithi192.stdout:Transaction check succeeded. 2024-06-03T08:27:55.483 INFO:teuthology.orchestra.run.smithi192.stdout:Running transaction test 2024-06-03T08:27:56.049 INFO:teuthology.orchestra.run.smithi192.stdout:Transaction test succeeded. 2024-06-03T08:27:56.049 INFO:teuthology.orchestra.run.smithi192.stdout:Running transaction 2024-06-03T08:27:56.413 INFO:teuthology.orchestra.run.smithi088.stdout: Installing : ceph-debuginfo-2:18.2.2-1389.gf1abf795.el9.x86_64 1/2 2024-06-03T08:27:56.493 INFO:teuthology.orchestra.run.smithi088.stdout: Installing : ceph-base-debuginfo-2:18.2.2-1389.gf1abf795.el9.x86_ 2/2 2024-06-03T08:27:58.883 INFO:teuthology.orchestra.run.smithi192.stdout: Preparing : 1/1 2024-06-03T08:27:59.745 INFO:teuthology.orchestra.run.smithi088.stdout: Running scriptlet: ceph-base-debuginfo-2:18.2.2-1389.gf1abf795.el9.x86_ 2/2 2024-06-03T08:27:59.745 INFO:teuthology.orchestra.run.smithi088.stdout: Verifying : ceph-base-debuginfo-2:18.2.2-1389.gf1abf795.el9.x86_ 1/2 2024-06-03T08:27:59.964 INFO:teuthology.orchestra.run.smithi088.stdout: Verifying : ceph-debuginfo-2:18.2.2-1389.gf1abf795.el9.x86_64 2/2 2024-06-03T08:27:59.964 INFO:teuthology.orchestra.run.smithi088.stdout: 2024-06-03T08:27:59.965 INFO:teuthology.orchestra.run.smithi088.stdout:Installed: 2024-06-03T08:27:59.965 INFO:teuthology.orchestra.run.smithi088.stdout: ceph-base-debuginfo-2:18.2.2-1389.gf1abf795.el9.x86_64 2024-06-03T08:27:59.965 INFO:teuthology.orchestra.run.smithi088.stdout: ceph-debuginfo-2:18.2.2-1389.gf1abf795.el9.x86_64 2024-06-03T08:27:59.965 INFO:teuthology.orchestra.run.smithi088.stdout: 2024-06-03T08:27:59.965 INFO:teuthology.orchestra.run.smithi088.stdout:Complete! 2024-06-03T08:28:00.177 DEBUG:teuthology.orchestra.run.smithi088:> sudo yum -y install ceph-common-debuginfo 2024-06-03T08:28:00.763 INFO:teuthology.orchestra.run.smithi088.stdout:Last metadata expiration check: 0:02:04 ago on Mon 03 Jun 2024 08:25:56 AM UTC. 2024-06-03T08:28:00.887 INFO:teuthology.orchestra.run.smithi088.stdout:Dependencies resolved. 2024-06-03T08:28:00.887 INFO:teuthology.orchestra.run.smithi088.stdout:================================================================================ 2024-06-03T08:28:00.888 INFO:teuthology.orchestra.run.smithi088.stdout: Package Arch Version Repo Size 2024-06-03T08:28:00.888 INFO:teuthology.orchestra.run.smithi088.stdout:================================================================================ 2024-06-03T08:28:00.888 INFO:teuthology.orchestra.run.smithi088.stdout:Installing: 2024-06-03T08:28:00.888 INFO:teuthology.orchestra.run.smithi088.stdout: ceph-common-debuginfo x86_64 2:18.2.2-1389.gf1abf795.el9 ceph 400 M 2024-06-03T08:28:00.888 INFO:teuthology.orchestra.run.smithi088.stdout: 2024-06-03T08:28:00.888 INFO:teuthology.orchestra.run.smithi088.stdout:Transaction Summary 2024-06-03T08:28:00.888 INFO:teuthology.orchestra.run.smithi088.stdout:================================================================================ 2024-06-03T08:28:00.888 INFO:teuthology.orchestra.run.smithi088.stdout:Install 1 Package 2024-06-03T08:28:00.888 INFO:teuthology.orchestra.run.smithi088.stdout: 2024-06-03T08:28:00.888 INFO:teuthology.orchestra.run.smithi088.stdout:Total download size: 400 M 2024-06-03T08:28:00.888 INFO:teuthology.orchestra.run.smithi088.stdout:Installed size: 1.6 G 2024-06-03T08:28:00.888 INFO:teuthology.orchestra.run.smithi088.stdout:Downloading Packages: 2024-06-03T08:28:05.916 INFO:teuthology.orchestra.run.smithi192.stdout: Installing : ceph-debuginfo-2:18.2.2-1389.gf1abf795.el9.x86_64 1/2 2024-06-03T08:28:05.994 INFO:teuthology.orchestra.run.smithi192.stdout: Installing : ceph-base-debuginfo-2:18.2.2-1389.gf1abf795.el9.x86_ 2/2 2024-06-03T08:28:09.234 INFO:teuthology.orchestra.run.smithi192.stdout: Running scriptlet: ceph-base-debuginfo-2:18.2.2-1389.gf1abf795.el9.x86_ 2/2 2024-06-03T08:28:09.235 INFO:teuthology.orchestra.run.smithi192.stdout: Verifying : ceph-base-debuginfo-2:18.2.2-1389.gf1abf795.el9.x86_ 1/2 2024-06-03T08:28:09.469 INFO:teuthology.orchestra.run.smithi192.stdout: Verifying : ceph-debuginfo-2:18.2.2-1389.gf1abf795.el9.x86_64 2/2 2024-06-03T08:28:09.469 INFO:teuthology.orchestra.run.smithi192.stdout: 2024-06-03T08:28:09.469 INFO:teuthology.orchestra.run.smithi192.stdout:Installed: 2024-06-03T08:28:09.469 INFO:teuthology.orchestra.run.smithi192.stdout: ceph-base-debuginfo-2:18.2.2-1389.gf1abf795.el9.x86_64 2024-06-03T08:28:09.469 INFO:teuthology.orchestra.run.smithi192.stdout: ceph-debuginfo-2:18.2.2-1389.gf1abf795.el9.x86_64 2024-06-03T08:28:09.470 INFO:teuthology.orchestra.run.smithi192.stdout: 2024-06-03T08:28:09.470 INFO:teuthology.orchestra.run.smithi192.stdout:Complete! 2024-06-03T08:28:09.667 DEBUG:teuthology.orchestra.run.smithi192:> sudo yum -y install ceph-common-debuginfo 2024-06-03T08:28:10.250 INFO:teuthology.orchestra.run.smithi192.stdout:Last metadata expiration check: 0:02:10 ago on Mon 03 Jun 2024 08:26:00 AM UTC. 2024-06-03T08:28:10.374 INFO:teuthology.orchestra.run.smithi192.stdout:Dependencies resolved. 2024-06-03T08:28:10.374 INFO:teuthology.orchestra.run.smithi192.stdout:================================================================================ 2024-06-03T08:28:10.374 INFO:teuthology.orchestra.run.smithi192.stdout: Package Arch Version Repo Size 2024-06-03T08:28:10.374 INFO:teuthology.orchestra.run.smithi192.stdout:================================================================================ 2024-06-03T08:28:10.374 INFO:teuthology.orchestra.run.smithi192.stdout:Installing: 2024-06-03T08:28:10.374 INFO:teuthology.orchestra.run.smithi192.stdout: ceph-common-debuginfo x86_64 2:18.2.2-1389.gf1abf795.el9 ceph 400 M 2024-06-03T08:28:10.375 INFO:teuthology.orchestra.run.smithi192.stdout: 2024-06-03T08:28:10.375 INFO:teuthology.orchestra.run.smithi192.stdout:Transaction Summary 2024-06-03T08:28:10.375 INFO:teuthology.orchestra.run.smithi192.stdout:================================================================================ 2024-06-03T08:28:10.375 INFO:teuthology.orchestra.run.smithi192.stdout:Install 1 Package 2024-06-03T08:28:10.375 INFO:teuthology.orchestra.run.smithi192.stdout: 2024-06-03T08:28:10.375 INFO:teuthology.orchestra.run.smithi192.stdout:Total download size: 400 M 2024-06-03T08:28:10.375 INFO:teuthology.orchestra.run.smithi192.stdout:Installed size: 1.6 G 2024-06-03T08:28:10.375 INFO:teuthology.orchestra.run.smithi192.stdout:Downloading Packages: 2024-06-03T08:28:25.079 INFO:teuthology.orchestra.run.smithi088.stdout:ceph-common-debuginfo-18.2.2-1389.gf1abf795.el9 17 MB/s | 400 MB 00:24 2024-06-03T08:28:25.079 INFO:teuthology.orchestra.run.smithi088.stdout:-------------------------------------------------------------------------------- 2024-06-03T08:28:25.080 INFO:teuthology.orchestra.run.smithi088.stdout:Total 17 MB/s | 400 MB 00:24 2024-06-03T08:28:25.080 INFO:teuthology.orchestra.run.smithi088.stdout:Running transaction check 2024-06-03T08:28:25.084 INFO:teuthology.orchestra.run.smithi088.stdout:Transaction check succeeded. 2024-06-03T08:28:25.084 INFO:teuthology.orchestra.run.smithi088.stdout:Running transaction test 2024-06-03T08:28:27.452 INFO:teuthology.orchestra.run.smithi088.stdout:Transaction test succeeded. 2024-06-03T08:28:27.453 INFO:teuthology.orchestra.run.smithi088.stdout:Running transaction 2024-06-03T08:28:34.339 INFO:teuthology.orchestra.run.smithi192.stdout:ceph-common-debuginfo-18.2.2-1389.gf1abf795.el9 17 MB/s | 400 MB 00:23 2024-06-03T08:28:34.339 INFO:teuthology.orchestra.run.smithi192.stdout:-------------------------------------------------------------------------------- 2024-06-03T08:28:34.340 INFO:teuthology.orchestra.run.smithi192.stdout:Total 17 MB/s | 400 MB 00:23 2024-06-03T08:28:34.340 INFO:teuthology.orchestra.run.smithi192.stdout:Running transaction check 2024-06-03T08:28:34.344 INFO:teuthology.orchestra.run.smithi192.stdout:Transaction check succeeded. 2024-06-03T08:28:34.344 INFO:teuthology.orchestra.run.smithi192.stdout:Running transaction test 2024-06-03T08:28:36.714 INFO:teuthology.orchestra.run.smithi192.stdout:Transaction test succeeded. 2024-06-03T08:28:36.714 INFO:teuthology.orchestra.run.smithi192.stdout:Running transaction 2024-06-03T08:29:08.012 INFO:teuthology.orchestra.run.smithi088.stdout: Preparing : 1/1 2024-06-03T08:29:08.321 INFO:teuthology.orchestra.run.smithi088.stdout: Installing : ceph-common-debuginfo-2:18.2.2-1389.gf1abf795.el9.x8 1/1 2024-06-03T08:29:16.464 INFO:teuthology.orchestra.run.smithi088.stdout: Running scriptlet: ceph-common-debuginfo-2:18.2.2-1389.gf1abf795.el9.x8 1/1 2024-06-03T08:29:16.678 INFO:teuthology.orchestra.run.smithi088.stdout: Verifying : ceph-common-debuginfo-2:18.2.2-1389.gf1abf795.el9.x8 1/1 2024-06-03T08:29:16.678 INFO:teuthology.orchestra.run.smithi088.stdout: 2024-06-03T08:29:16.678 INFO:teuthology.orchestra.run.smithi088.stdout:Installed: 2024-06-03T08:29:16.678 INFO:teuthology.orchestra.run.smithi088.stdout: ceph-common-debuginfo-2:18.2.2-1389.gf1abf795.el9.x86_64 2024-06-03T08:29:16.678 INFO:teuthology.orchestra.run.smithi088.stdout: 2024-06-03T08:29:16.678 INFO:teuthology.orchestra.run.smithi088.stdout:Complete! 2024-06-03T08:29:16.887 DEBUG:teuthology.orchestra.run.smithi088:> sudo yum -y install ceph-immutable-object-cache-debuginfo 2024-06-03T08:29:17.297 INFO:teuthology.orchestra.run.smithi192.stdout: Preparing : 1/1 2024-06-03T08:29:17.464 INFO:teuthology.orchestra.run.smithi088.stdout:Last metadata expiration check: 0:03:21 ago on Mon 03 Jun 2024 08:25:56 AM UTC. 2024-06-03T08:29:17.589 INFO:teuthology.orchestra.run.smithi088.stdout:Dependencies resolved. 2024-06-03T08:29:17.590 INFO:teuthology.orchestra.run.smithi088.stdout:======================================================================================== 2024-06-03T08:29:17.590 INFO:teuthology.orchestra.run.smithi088.stdout: Package Arch Version Repo Size 2024-06-03T08:29:17.590 INFO:teuthology.orchestra.run.smithi088.stdout:======================================================================================== 2024-06-03T08:29:17.590 INFO:teuthology.orchestra.run.smithi088.stdout:Installing: 2024-06-03T08:29:17.590 INFO:teuthology.orchestra.run.smithi088.stdout: ceph-immutable-object-cache-debuginfo x86_64 2:18.2.2-1389.gf1abf795.el9 ceph 1.4 M 2024-06-03T08:29:17.590 INFO:teuthology.orchestra.run.smithi088.stdout: 2024-06-03T08:29:17.590 INFO:teuthology.orchestra.run.smithi088.stdout:Transaction Summary 2024-06-03T08:29:17.590 INFO:teuthology.orchestra.run.smithi088.stdout:======================================================================================== 2024-06-03T08:29:17.590 INFO:teuthology.orchestra.run.smithi088.stdout:Install 1 Package 2024-06-03T08:29:17.590 INFO:teuthology.orchestra.run.smithi088.stdout: 2024-06-03T08:29:17.591 INFO:teuthology.orchestra.run.smithi088.stdout:Total download size: 1.4 M 2024-06-03T08:29:17.591 INFO:teuthology.orchestra.run.smithi088.stdout:Installed size: 6.2 M 2024-06-03T08:29:17.591 INFO:teuthology.orchestra.run.smithi088.stdout:Downloading Packages: 2024-06-03T08:29:17.804 INFO:teuthology.orchestra.run.smithi192.stdout: Installing : ceph-common-debuginfo-2:18.2.2-1389.gf1abf795.el9.x8 1/1 2024-06-03T08:29:18.001 INFO:teuthology.orchestra.run.smithi088.stdout:ceph-immutable-object-cache-debuginfo-18.2.2-13 3.4 MB/s | 1.4 MB 00:00 2024-06-03T08:29:18.001 INFO:teuthology.orchestra.run.smithi088.stdout:-------------------------------------------------------------------------------- 2024-06-03T08:29:18.001 INFO:teuthology.orchestra.run.smithi088.stdout:Total 3.4 MB/s | 1.4 MB 00:00 2024-06-03T08:29:18.002 INFO:teuthology.orchestra.run.smithi088.stdout:Running transaction check 2024-06-03T08:29:18.005 INFO:teuthology.orchestra.run.smithi088.stdout:Transaction check succeeded. 2024-06-03T08:29:18.005 INFO:teuthology.orchestra.run.smithi088.stdout:Running transaction test 2024-06-03T08:29:18.048 INFO:teuthology.orchestra.run.smithi088.stdout:Transaction test succeeded. 2024-06-03T08:29:18.048 INFO:teuthology.orchestra.run.smithi088.stdout:Running transaction 2024-06-03T08:29:18.302 INFO:teuthology.orchestra.run.smithi088.stdout: Preparing : 1/1 2024-06-03T08:29:18.407 INFO:teuthology.orchestra.run.smithi088.stdout: Installing : ceph-immutable-object-cache-debuginfo-2:18.2.2-1389. 1/1 2024-06-03T08:29:18.824 INFO:teuthology.orchestra.run.smithi088.stdout: Running scriptlet: ceph-immutable-object-cache-debuginfo-2:18.2.2-1389. 1/1 2024-06-03T08:29:19.048 INFO:teuthology.orchestra.run.smithi088.stdout: Verifying : ceph-immutable-object-cache-debuginfo-2:18.2.2-1389. 1/1 2024-06-03T08:29:19.048 INFO:teuthology.orchestra.run.smithi088.stdout: 2024-06-03T08:29:19.048 INFO:teuthology.orchestra.run.smithi088.stdout:Installed: 2024-06-03T08:29:19.048 INFO:teuthology.orchestra.run.smithi088.stdout: ceph-immutable-object-cache-debuginfo-2:18.2.2-1389.gf1abf795.el9.x86_64 2024-06-03T08:29:19.048 INFO:teuthology.orchestra.run.smithi088.stdout: 2024-06-03T08:29:19.048 INFO:teuthology.orchestra.run.smithi088.stdout:Complete! 2024-06-03T08:29:19.209 DEBUG:teuthology.orchestra.run.smithi088:> sudo yum -y install ceph-radosgw-debuginfo 2024-06-03T08:29:19.785 INFO:teuthology.orchestra.run.smithi088.stdout:Last metadata expiration check: 0:03:23 ago on Mon 03 Jun 2024 08:25:56 AM UTC. 2024-06-03T08:29:19.906 INFO:teuthology.orchestra.run.smithi088.stdout:Dependencies resolved. 2024-06-03T08:29:19.908 INFO:teuthology.orchestra.run.smithi088.stdout:================================================================================ 2024-06-03T08:29:19.908 INFO:teuthology.orchestra.run.smithi088.stdout: Package Arch Version Repo Size 2024-06-03T08:29:19.908 INFO:teuthology.orchestra.run.smithi088.stdout:================================================================================ 2024-06-03T08:29:19.908 INFO:teuthology.orchestra.run.smithi088.stdout:Installing: 2024-06-03T08:29:19.908 INFO:teuthology.orchestra.run.smithi088.stdout: ceph-radosgw-debuginfo x86_64 2:18.2.2-1389.gf1abf795.el9 ceph 162 M 2024-06-03T08:29:19.908 INFO:teuthology.orchestra.run.smithi088.stdout: 2024-06-03T08:29:19.909 INFO:teuthology.orchestra.run.smithi088.stdout:Transaction Summary 2024-06-03T08:29:19.909 INFO:teuthology.orchestra.run.smithi088.stdout:================================================================================ 2024-06-03T08:29:19.909 INFO:teuthology.orchestra.run.smithi088.stdout:Install 1 Package 2024-06-03T08:29:19.909 INFO:teuthology.orchestra.run.smithi088.stdout: 2024-06-03T08:29:19.909 INFO:teuthology.orchestra.run.smithi088.stdout:Total download size: 162 M 2024-06-03T08:29:19.909 INFO:teuthology.orchestra.run.smithi088.stdout:Installed size: 679 M 2024-06-03T08:29:19.909 INFO:teuthology.orchestra.run.smithi088.stdout:Downloading Packages: 2024-06-03T08:29:25.543 INFO:teuthology.orchestra.run.smithi192.stdout: Running scriptlet: ceph-common-debuginfo-2:18.2.2-1389.gf1abf795.el9.x8 1/1 2024-06-03T08:29:25.766 INFO:teuthology.orchestra.run.smithi192.stdout: Verifying : ceph-common-debuginfo-2:18.2.2-1389.gf1abf795.el9.x8 1/1 2024-06-03T08:29:25.766 INFO:teuthology.orchestra.run.smithi192.stdout: 2024-06-03T08:29:25.766 INFO:teuthology.orchestra.run.smithi192.stdout:Installed: 2024-06-03T08:29:25.766 INFO:teuthology.orchestra.run.smithi192.stdout: ceph-common-debuginfo-2:18.2.2-1389.gf1abf795.el9.x86_64 2024-06-03T08:29:25.767 INFO:teuthology.orchestra.run.smithi192.stdout: 2024-06-03T08:29:25.767 INFO:teuthology.orchestra.run.smithi192.stdout:Complete! 2024-06-03T08:29:25.998 DEBUG:teuthology.orchestra.run.smithi192:> sudo yum -y install ceph-immutable-object-cache-debuginfo 2024-06-03T08:29:26.586 INFO:teuthology.orchestra.run.smithi192.stdout:Last metadata expiration check: 0:03:26 ago on Mon 03 Jun 2024 08:26:00 AM UTC. 2024-06-03T08:29:26.718 INFO:teuthology.orchestra.run.smithi192.stdout:Dependencies resolved. 2024-06-03T08:29:26.719 INFO:teuthology.orchestra.run.smithi192.stdout:======================================================================================== 2024-06-03T08:29:26.719 INFO:teuthology.orchestra.run.smithi192.stdout: Package Arch Version Repo Size 2024-06-03T08:29:26.719 INFO:teuthology.orchestra.run.smithi192.stdout:======================================================================================== 2024-06-03T08:29:26.719 INFO:teuthology.orchestra.run.smithi192.stdout:Installing: 2024-06-03T08:29:26.719 INFO:teuthology.orchestra.run.smithi192.stdout: ceph-immutable-object-cache-debuginfo x86_64 2:18.2.2-1389.gf1abf795.el9 ceph 1.4 M 2024-06-03T08:29:26.719 INFO:teuthology.orchestra.run.smithi192.stdout: 2024-06-03T08:29:26.719 INFO:teuthology.orchestra.run.smithi192.stdout:Transaction Summary 2024-06-03T08:29:26.719 INFO:teuthology.orchestra.run.smithi192.stdout:======================================================================================== 2024-06-03T08:29:26.719 INFO:teuthology.orchestra.run.smithi192.stdout:Install 1 Package 2024-06-03T08:29:26.720 INFO:teuthology.orchestra.run.smithi192.stdout: 2024-06-03T08:29:26.720 INFO:teuthology.orchestra.run.smithi192.stdout:Total download size: 1.4 M 2024-06-03T08:29:26.720 INFO:teuthology.orchestra.run.smithi192.stdout:Installed size: 6.2 M 2024-06-03T08:29:26.720 INFO:teuthology.orchestra.run.smithi192.stdout:Downloading Packages: 2024-06-03T08:29:27.363 INFO:teuthology.orchestra.run.smithi088.stdout:ceph-radosgw-debuginfo-18.2.2-1389.gf1abf795.el 22 MB/s | 162 MB 00:07 2024-06-03T08:29:27.364 INFO:teuthology.orchestra.run.smithi088.stdout:-------------------------------------------------------------------------------- 2024-06-03T08:29:27.364 INFO:teuthology.orchestra.run.smithi088.stdout:Total 22 MB/s | 162 MB 00:07 2024-06-03T08:29:27.365 INFO:teuthology.orchestra.run.smithi088.stdout:Running transaction check 2024-06-03T08:29:27.368 INFO:teuthology.orchestra.run.smithi088.stdout:Transaction check succeeded. 2024-06-03T08:29:27.368 INFO:teuthology.orchestra.run.smithi088.stdout:Running transaction test 2024-06-03T08:29:27.383 INFO:teuthology.orchestra.run.smithi192.stdout:ceph-immutable-object-cache-debuginfo-18.2.2-13 2.1 MB/s | 1.4 MB 00:00 2024-06-03T08:29:27.384 INFO:teuthology.orchestra.run.smithi192.stdout:-------------------------------------------------------------------------------- 2024-06-03T08:29:27.384 INFO:teuthology.orchestra.run.smithi192.stdout:Total 2.1 MB/s | 1.4 MB 00:00 2024-06-03T08:29:27.384 INFO:teuthology.orchestra.run.smithi192.stdout:Running transaction check 2024-06-03T08:29:27.388 INFO:teuthology.orchestra.run.smithi192.stdout:Transaction check succeeded. 2024-06-03T08:29:27.388 INFO:teuthology.orchestra.run.smithi192.stdout:Running transaction test 2024-06-03T08:29:27.430 INFO:teuthology.orchestra.run.smithi192.stdout:Transaction test succeeded. 2024-06-03T08:29:27.431 INFO:teuthology.orchestra.run.smithi192.stdout:Running transaction 2024-06-03T08:29:27.683 INFO:teuthology.orchestra.run.smithi192.stdout: Preparing : 1/1 2024-06-03T08:29:27.797 INFO:teuthology.orchestra.run.smithi192.stdout: Installing : ceph-immutable-object-cache-debuginfo-2:18.2.2-1389. 1/1 2024-06-03T08:29:28.160 INFO:teuthology.orchestra.run.smithi192.stdout: Running scriptlet: ceph-immutable-object-cache-debuginfo-2:18.2.2-1389. 1/1 2024-06-03T08:29:28.348 INFO:teuthology.orchestra.run.smithi088.stdout:Transaction test succeeded. 2024-06-03T08:29:28.348 INFO:teuthology.orchestra.run.smithi088.stdout:Running transaction 2024-06-03T08:29:28.382 INFO:teuthology.orchestra.run.smithi192.stdout: Verifying : ceph-immutable-object-cache-debuginfo-2:18.2.2-1389. 1/1 2024-06-03T08:29:28.382 INFO:teuthology.orchestra.run.smithi192.stdout: 2024-06-03T08:29:28.382 INFO:teuthology.orchestra.run.smithi192.stdout:Installed: 2024-06-03T08:29:28.382 INFO:teuthology.orchestra.run.smithi192.stdout: ceph-immutable-object-cache-debuginfo-2:18.2.2-1389.gf1abf795.el9.x86_64 2024-06-03T08:29:28.382 INFO:teuthology.orchestra.run.smithi192.stdout: 2024-06-03T08:29:28.382 INFO:teuthology.orchestra.run.smithi192.stdout:Complete! 2024-06-03T08:29:28.531 DEBUG:teuthology.orchestra.run.smithi192:> sudo yum -y install ceph-radosgw-debuginfo 2024-06-03T08:29:29.107 INFO:teuthology.orchestra.run.smithi192.stdout:Last metadata expiration check: 0:03:29 ago on Mon 03 Jun 2024 08:26:00 AM UTC. 2024-06-03T08:29:29.230 INFO:teuthology.orchestra.run.smithi192.stdout:Dependencies resolved. 2024-06-03T08:29:29.230 INFO:teuthology.orchestra.run.smithi192.stdout:================================================================================ 2024-06-03T08:29:29.230 INFO:teuthology.orchestra.run.smithi192.stdout: Package Arch Version Repo Size 2024-06-03T08:29:29.230 INFO:teuthology.orchestra.run.smithi192.stdout:================================================================================ 2024-06-03T08:29:29.230 INFO:teuthology.orchestra.run.smithi192.stdout:Installing: 2024-06-03T08:29:29.230 INFO:teuthology.orchestra.run.smithi192.stdout: ceph-radosgw-debuginfo x86_64 2:18.2.2-1389.gf1abf795.el9 ceph 162 M 2024-06-03T08:29:29.230 INFO:teuthology.orchestra.run.smithi192.stdout: 2024-06-03T08:29:29.231 INFO:teuthology.orchestra.run.smithi192.stdout:Transaction Summary 2024-06-03T08:29:29.231 INFO:teuthology.orchestra.run.smithi192.stdout:================================================================================ 2024-06-03T08:29:29.231 INFO:teuthology.orchestra.run.smithi192.stdout:Install 1 Package 2024-06-03T08:29:29.231 INFO:teuthology.orchestra.run.smithi192.stdout: 2024-06-03T08:29:29.231 INFO:teuthology.orchestra.run.smithi192.stdout:Total download size: 162 M 2024-06-03T08:29:29.231 INFO:teuthology.orchestra.run.smithi192.stdout:Installed size: 679 M 2024-06-03T08:29:29.231 INFO:teuthology.orchestra.run.smithi192.stdout:Downloading Packages: 2024-06-03T08:29:37.587 INFO:teuthology.orchestra.run.smithi192.stdout:ceph-radosgw-debuginfo-18.2.2-1389.gf1abf795.el 19 MB/s | 162 MB 00:08 2024-06-03T08:29:37.587 INFO:teuthology.orchestra.run.smithi192.stdout:-------------------------------------------------------------------------------- 2024-06-03T08:29:37.588 INFO:teuthology.orchestra.run.smithi192.stdout:Total 19 MB/s | 162 MB 00:08 2024-06-03T08:29:37.588 INFO:teuthology.orchestra.run.smithi192.stdout:Running transaction check 2024-06-03T08:29:37.592 INFO:teuthology.orchestra.run.smithi192.stdout:Transaction check succeeded. 2024-06-03T08:29:37.592 INFO:teuthology.orchestra.run.smithi192.stdout:Running transaction test 2024-06-03T08:29:38.575 INFO:teuthology.orchestra.run.smithi192.stdout:Transaction test succeeded. 2024-06-03T08:29:38.575 INFO:teuthology.orchestra.run.smithi192.stdout:Running transaction 2024-06-03T08:29:44.932 INFO:teuthology.orchestra.run.smithi088.stdout: Preparing : 1/1 2024-06-03T08:29:45.041 INFO:teuthology.orchestra.run.smithi088.stdout: Installing : ceph-radosgw-debuginfo-2:18.2.2-1389.gf1abf795.el9.x 1/1 2024-06-03T08:29:49.207 INFO:teuthology.orchestra.run.smithi088.stdout: Running scriptlet: ceph-radosgw-debuginfo-2:18.2.2-1389.gf1abf795.el9.x 1/1 2024-06-03T08:29:49.423 INFO:teuthology.orchestra.run.smithi088.stdout: Verifying : ceph-radosgw-debuginfo-2:18.2.2-1389.gf1abf795.el9.x 1/1 2024-06-03T08:29:49.424 INFO:teuthology.orchestra.run.smithi088.stdout: 2024-06-03T08:29:49.424 INFO:teuthology.orchestra.run.smithi088.stdout:Installed: 2024-06-03T08:29:49.424 INFO:teuthology.orchestra.run.smithi088.stdout: ceph-radosgw-debuginfo-2:18.2.2-1389.gf1abf795.el9.x86_64 2024-06-03T08:29:49.424 INFO:teuthology.orchestra.run.smithi088.stdout: 2024-06-03T08:29:49.424 INFO:teuthology.orchestra.run.smithi088.stdout:Complete! 2024-06-03T08:29:49.603 DEBUG:teuthology.orchestra.run.smithi088:> sudo yum -y install ceph-test-debuginfo 2024-06-03T08:29:50.181 INFO:teuthology.orchestra.run.smithi088.stdout:Last metadata expiration check: 0:03:54 ago on Mon 03 Jun 2024 08:25:56 AM UTC. 2024-06-03T08:29:50.303 INFO:teuthology.orchestra.run.smithi088.stdout:Dependencies resolved. 2024-06-03T08:29:50.304 INFO:teuthology.orchestra.run.smithi088.stdout:================================================================================ 2024-06-03T08:29:50.304 INFO:teuthology.orchestra.run.smithi088.stdout: Package Arch Version Repo Size 2024-06-03T08:29:50.304 INFO:teuthology.orchestra.run.smithi088.stdout:================================================================================ 2024-06-03T08:29:50.305 INFO:teuthology.orchestra.run.smithi088.stdout:Installing: 2024-06-03T08:29:50.305 INFO:teuthology.orchestra.run.smithi088.stdout: ceph-test-debuginfo x86_64 2:18.2.2-1389.gf1abf795.el9 ceph 582 M 2024-06-03T08:29:50.305 INFO:teuthology.orchestra.run.smithi088.stdout: 2024-06-03T08:29:50.305 INFO:teuthology.orchestra.run.smithi088.stdout:Transaction Summary 2024-06-03T08:29:50.305 INFO:teuthology.orchestra.run.smithi088.stdout:================================================================================ 2024-06-03T08:29:50.305 INFO:teuthology.orchestra.run.smithi088.stdout:Install 1 Package 2024-06-03T08:29:50.305 INFO:teuthology.orchestra.run.smithi088.stdout: 2024-06-03T08:29:50.305 INFO:teuthology.orchestra.run.smithi088.stdout:Total download size: 582 M 2024-06-03T08:29:50.305 INFO:teuthology.orchestra.run.smithi088.stdout:Installed size: 2.6 G 2024-06-03T08:29:50.305 INFO:teuthology.orchestra.run.smithi088.stdout:Downloading Packages: 2024-06-03T08:29:55.187 INFO:teuthology.orchestra.run.smithi192.stdout: Preparing : 1/1 2024-06-03T08:29:55.276 INFO:teuthology.orchestra.run.smithi192.stdout: Installing : ceph-radosgw-debuginfo-2:18.2.2-1389.gf1abf795.el9.x 1/1 2024-06-03T08:29:59.517 INFO:teuthology.orchestra.run.smithi192.stdout: Running scriptlet: ceph-radosgw-debuginfo-2:18.2.2-1389.gf1abf795.el9.x 1/1 2024-06-03T08:29:59.753 INFO:teuthology.orchestra.run.smithi192.stdout: Verifying : ceph-radosgw-debuginfo-2:18.2.2-1389.gf1abf795.el9.x 1/1 2024-06-03T08:29:59.753 INFO:teuthology.orchestra.run.smithi192.stdout: 2024-06-03T08:29:59.753 INFO:teuthology.orchestra.run.smithi192.stdout:Installed: 2024-06-03T08:29:59.753 INFO:teuthology.orchestra.run.smithi192.stdout: ceph-radosgw-debuginfo-2:18.2.2-1389.gf1abf795.el9.x86_64 2024-06-03T08:29:59.754 INFO:teuthology.orchestra.run.smithi192.stdout: 2024-06-03T08:29:59.754 INFO:teuthology.orchestra.run.smithi192.stdout:Complete! 2024-06-03T08:29:59.929 DEBUG:teuthology.orchestra.run.smithi192:> sudo yum -y install ceph-test-debuginfo 2024-06-03T08:30:00.511 INFO:teuthology.orchestra.run.smithi192.stdout:Last metadata expiration check: 0:04:00 ago on Mon 03 Jun 2024 08:26:00 AM UTC. 2024-06-03T08:30:00.634 INFO:teuthology.orchestra.run.smithi192.stdout:Dependencies resolved. 2024-06-03T08:30:00.635 INFO:teuthology.orchestra.run.smithi192.stdout:================================================================================ 2024-06-03T08:30:00.635 INFO:teuthology.orchestra.run.smithi192.stdout: Package Arch Version Repo Size 2024-06-03T08:30:00.635 INFO:teuthology.orchestra.run.smithi192.stdout:================================================================================ 2024-06-03T08:30:00.635 INFO:teuthology.orchestra.run.smithi192.stdout:Installing: 2024-06-03T08:30:00.635 INFO:teuthology.orchestra.run.smithi192.stdout: ceph-test-debuginfo x86_64 2:18.2.2-1389.gf1abf795.el9 ceph 582 M 2024-06-03T08:30:00.635 INFO:teuthology.orchestra.run.smithi192.stdout: 2024-06-03T08:30:00.636 INFO:teuthology.orchestra.run.smithi192.stdout:Transaction Summary 2024-06-03T08:30:00.636 INFO:teuthology.orchestra.run.smithi192.stdout:================================================================================ 2024-06-03T08:30:00.636 INFO:teuthology.orchestra.run.smithi192.stdout:Install 1 Package 2024-06-03T08:30:00.636 INFO:teuthology.orchestra.run.smithi192.stdout: 2024-06-03T08:30:00.636 INFO:teuthology.orchestra.run.smithi192.stdout:Total download size: 582 M 2024-06-03T08:30:00.636 INFO:teuthology.orchestra.run.smithi192.stdout:Installed size: 2.6 G 2024-06-03T08:30:00.636 INFO:teuthology.orchestra.run.smithi192.stdout:Downloading Packages: 2024-06-03T08:30:30.338 INFO:teuthology.orchestra.run.smithi192.stdout:ceph-test-debuginfo-18.2.2-1389.gf1abf795.el9.x 20 MB/s | 582 MB 00:29 2024-06-03T08:30:30.339 INFO:teuthology.orchestra.run.smithi192.stdout:-------------------------------------------------------------------------------- 2024-06-03T08:30:30.339 INFO:teuthology.orchestra.run.smithi192.stdout:Total 20 MB/s | 582 MB 00:29 2024-06-03T08:30:30.339 INFO:teuthology.orchestra.run.smithi192.stdout:Running transaction check 2024-06-03T08:30:30.345 INFO:teuthology.orchestra.run.smithi192.stdout:Transaction check succeeded. 2024-06-03T08:30:30.345 INFO:teuthology.orchestra.run.smithi192.stdout:Running transaction test 2024-06-03T08:30:33.768 INFO:teuthology.orchestra.run.smithi192.stdout:Transaction test succeeded. 2024-06-03T08:30:33.768 INFO:teuthology.orchestra.run.smithi192.stdout:Running transaction 2024-06-03T08:30:36.033 INFO:teuthology.orchestra.run.smithi088.stdout:ceph-test-debuginfo-18.2.2-1389.gf1abf795.el9.x 13 MB/s | 582 MB 00:45 2024-06-03T08:30:36.033 INFO:teuthology.orchestra.run.smithi088.stdout:-------------------------------------------------------------------------------- 2024-06-03T08:30:36.034 INFO:teuthology.orchestra.run.smithi088.stdout:Total 13 MB/s | 582 MB 00:45 2024-06-03T08:30:36.034 INFO:teuthology.orchestra.run.smithi088.stdout:Running transaction check 2024-06-03T08:30:36.039 INFO:teuthology.orchestra.run.smithi088.stdout:Transaction check succeeded. 2024-06-03T08:30:36.040 INFO:teuthology.orchestra.run.smithi088.stdout:Running transaction test 2024-06-03T08:30:39.463 INFO:teuthology.orchestra.run.smithi088.stdout:Transaction test succeeded. 2024-06-03T08:30:39.463 INFO:teuthology.orchestra.run.smithi088.stdout:Running transaction 2024-06-03T08:31:34.449 INFO:teuthology.orchestra.run.smithi192.stdout: Preparing : 1/1 2024-06-03T08:31:34.547 INFO:teuthology.orchestra.run.smithi192.stdout: Installing : ceph-test-debuginfo-2:18.2.2-1389.gf1abf795.el9.x86_ 1/1 2024-06-03T08:31:40.026 INFO:teuthology.orchestra.run.smithi088.stdout: Preparing : 1/1 2024-06-03T08:31:40.196 INFO:teuthology.orchestra.run.smithi088.stdout: Installing : ceph-test-debuginfo-2:18.2.2-1389.gf1abf795.el9.x86_ 1/1 2024-06-03T08:31:43.154 INFO:teuthology.orchestra.run.smithi192.stdout: Running scriptlet: ceph-test-debuginfo-2:18.2.2-1389.gf1abf795.el9.x86_ 1/1 2024-06-03T08:31:43.379 INFO:teuthology.orchestra.run.smithi192.stdout: Verifying : ceph-test-debuginfo-2:18.2.2-1389.gf1abf795.el9.x86_ 1/1 2024-06-03T08:31:43.379 INFO:teuthology.orchestra.run.smithi192.stdout: 2024-06-03T08:31:43.379 INFO:teuthology.orchestra.run.smithi192.stdout:Installed: 2024-06-03T08:31:43.379 INFO:teuthology.orchestra.run.smithi192.stdout: ceph-test-debuginfo-2:18.2.2-1389.gf1abf795.el9.x86_64 2024-06-03T08:31:43.379 INFO:teuthology.orchestra.run.smithi192.stdout: 2024-06-03T08:31:43.379 INFO:teuthology.orchestra.run.smithi192.stdout:Complete! 2024-06-03T08:31:43.632 DEBUG:teuthology.orchestra.run.smithi192:> sudo yum -y install ceph-base-debuginfo 2024-06-03T08:31:44.212 INFO:teuthology.orchestra.run.smithi192.stdout:Last metadata expiration check: 0:05:44 ago on Mon 03 Jun 2024 08:26:00 AM UTC. 2024-06-03T08:31:44.294 INFO:teuthology.orchestra.run.smithi192.stdout:Package ceph-base-debuginfo-2:18.2.2-1389.gf1abf795.el9.x86_64 is already installed. 2024-06-03T08:31:44.337 INFO:teuthology.orchestra.run.smithi192.stdout:Dependencies resolved. 2024-06-03T08:31:44.338 INFO:teuthology.orchestra.run.smithi192.stdout:Nothing to do. 2024-06-03T08:31:44.338 INFO:teuthology.orchestra.run.smithi192.stdout:Complete! 2024-06-03T08:31:44.396 DEBUG:teuthology.orchestra.run.smithi192:> sudo yum -y install ceph-mgr-debuginfo 2024-06-03T08:31:44.968 INFO:teuthology.orchestra.run.smithi192.stdout:Last metadata expiration check: 0:05:44 ago on Mon 03 Jun 2024 08:26:00 AM UTC. 2024-06-03T08:31:45.091 INFO:teuthology.orchestra.run.smithi192.stdout:Dependencies resolved. 2024-06-03T08:31:45.092 INFO:teuthology.orchestra.run.smithi192.stdout:================================================================================ 2024-06-03T08:31:45.092 INFO:teuthology.orchestra.run.smithi192.stdout: Package Arch Version Repo Size 2024-06-03T08:31:45.092 INFO:teuthology.orchestra.run.smithi192.stdout:================================================================================ 2024-06-03T08:31:45.092 INFO:teuthology.orchestra.run.smithi192.stdout:Installing: 2024-06-03T08:31:45.092 INFO:teuthology.orchestra.run.smithi192.stdout: ceph-mgr-debuginfo x86_64 2:18.2.2-1389.gf1abf795.el9 ceph 19 M 2024-06-03T08:31:45.092 INFO:teuthology.orchestra.run.smithi192.stdout: 2024-06-03T08:31:45.092 INFO:teuthology.orchestra.run.smithi192.stdout:Transaction Summary 2024-06-03T08:31:45.092 INFO:teuthology.orchestra.run.smithi192.stdout:================================================================================ 2024-06-03T08:31:45.092 INFO:teuthology.orchestra.run.smithi192.stdout:Install 1 Package 2024-06-03T08:31:45.092 INFO:teuthology.orchestra.run.smithi192.stdout: 2024-06-03T08:31:45.092 INFO:teuthology.orchestra.run.smithi192.stdout:Total download size: 19 M 2024-06-03T08:31:45.093 INFO:teuthology.orchestra.run.smithi192.stdout:Installed size: 101 M 2024-06-03T08:31:45.093 INFO:teuthology.orchestra.run.smithi192.stdout:Downloading Packages: 2024-06-03T08:31:46.793 INFO:teuthology.orchestra.run.smithi192.stdout:ceph-mgr-debuginfo-18.2.2-1389.gf1abf795.el9.x8 11 MB/s | 19 MB 00:01 2024-06-03T08:31:46.794 INFO:teuthology.orchestra.run.smithi192.stdout:-------------------------------------------------------------------------------- 2024-06-03T08:31:46.794 INFO:teuthology.orchestra.run.smithi192.stdout:Total 11 MB/s | 19 MB 00:01 2024-06-03T08:31:46.795 INFO:teuthology.orchestra.run.smithi192.stdout:Running transaction check 2024-06-03T08:31:46.798 INFO:teuthology.orchestra.run.smithi192.stdout:Transaction check succeeded. 2024-06-03T08:31:46.798 INFO:teuthology.orchestra.run.smithi192.stdout:Running transaction test 2024-06-03T08:31:46.946 INFO:teuthology.orchestra.run.smithi192.stdout:Transaction test succeeded. 2024-06-03T08:31:46.947 INFO:teuthology.orchestra.run.smithi192.stdout:Running transaction 2024-06-03T08:31:48.710 INFO:teuthology.orchestra.run.smithi088.stdout: Running scriptlet: ceph-test-debuginfo-2:18.2.2-1389.gf1abf795.el9.x86_ 1/1 2024-06-03T08:31:48.931 INFO:teuthology.orchestra.run.smithi088.stdout: Verifying : ceph-test-debuginfo-2:18.2.2-1389.gf1abf795.el9.x86_ 1/1 2024-06-03T08:31:48.931 INFO:teuthology.orchestra.run.smithi088.stdout: 2024-06-03T08:31:48.931 INFO:teuthology.orchestra.run.smithi088.stdout:Installed: 2024-06-03T08:31:48.932 INFO:teuthology.orchestra.run.smithi088.stdout: ceph-test-debuginfo-2:18.2.2-1389.gf1abf795.el9.x86_64 2024-06-03T08:31:48.932 INFO:teuthology.orchestra.run.smithi088.stdout: 2024-06-03T08:31:48.932 INFO:teuthology.orchestra.run.smithi088.stdout:Complete! 2024-06-03T08:31:49.137 INFO:teuthology.orchestra.run.smithi192.stdout: Preparing : 1/1 2024-06-03T08:31:49.193 DEBUG:teuthology.orchestra.run.smithi088:> sudo yum -y install ceph-base-debuginfo 2024-06-03T08:31:49.241 INFO:teuthology.orchestra.run.smithi192.stdout: Installing : ceph-mgr-debuginfo-2:18.2.2-1389.gf1abf795.el9.x86_6 1/1 2024-06-03T08:31:49.763 INFO:teuthology.orchestra.run.smithi088.stdout:Last metadata expiration check: 0:05:53 ago on Mon 03 Jun 2024 08:25:56 AM UTC. 2024-06-03T08:31:49.844 INFO:teuthology.orchestra.run.smithi088.stdout:Package ceph-base-debuginfo-2:18.2.2-1389.gf1abf795.el9.x86_64 is already installed. 2024-06-03T08:31:49.885 INFO:teuthology.orchestra.run.smithi088.stdout:Dependencies resolved. 2024-06-03T08:31:49.886 INFO:teuthology.orchestra.run.smithi088.stdout:Nothing to do. 2024-06-03T08:31:49.887 INFO:teuthology.orchestra.run.smithi088.stdout:Complete! 2024-06-03T08:31:49.949 DEBUG:teuthology.orchestra.run.smithi088:> sudo yum -y install ceph-mgr-debuginfo 2024-06-03T08:31:50.218 INFO:teuthology.orchestra.run.smithi192.stdout: Running scriptlet: ceph-mgr-debuginfo-2:18.2.2-1389.gf1abf795.el9.x86_6 1/1 2024-06-03T08:31:50.435 INFO:teuthology.orchestra.run.smithi192.stdout: Verifying : ceph-mgr-debuginfo-2:18.2.2-1389.gf1abf795.el9.x86_6 1/1 2024-06-03T08:31:50.435 INFO:teuthology.orchestra.run.smithi192.stdout: 2024-06-03T08:31:50.436 INFO:teuthology.orchestra.run.smithi192.stdout:Installed: 2024-06-03T08:31:50.436 INFO:teuthology.orchestra.run.smithi192.stdout: ceph-mgr-debuginfo-2:18.2.2-1389.gf1abf795.el9.x86_64 2024-06-03T08:31:50.436 INFO:teuthology.orchestra.run.smithi192.stdout: 2024-06-03T08:31:50.436 INFO:teuthology.orchestra.run.smithi192.stdout:Complete! 2024-06-03T08:31:50.529 INFO:teuthology.orchestra.run.smithi088.stdout:Last metadata expiration check: 0:05:54 ago on Mon 03 Jun 2024 08:25:56 AM UTC. 2024-06-03T08:31:50.600 DEBUG:teuthology.orchestra.run.smithi192:> sudo yum -y install ceph-mds-debuginfo 2024-06-03T08:31:50.653 INFO:teuthology.orchestra.run.smithi088.stdout:Dependencies resolved. 2024-06-03T08:31:50.654 INFO:teuthology.orchestra.run.smithi088.stdout:================================================================================ 2024-06-03T08:31:50.654 INFO:teuthology.orchestra.run.smithi088.stdout: Package Arch Version Repo Size 2024-06-03T08:31:50.654 INFO:teuthology.orchestra.run.smithi088.stdout:================================================================================ 2024-06-03T08:31:50.654 INFO:teuthology.orchestra.run.smithi088.stdout:Installing: 2024-06-03T08:31:50.654 INFO:teuthology.orchestra.run.smithi088.stdout: ceph-mgr-debuginfo x86_64 2:18.2.2-1389.gf1abf795.el9 ceph 19 M 2024-06-03T08:31:50.654 INFO:teuthology.orchestra.run.smithi088.stdout: 2024-06-03T08:31:50.654 INFO:teuthology.orchestra.run.smithi088.stdout:Transaction Summary 2024-06-03T08:31:50.654 INFO:teuthology.orchestra.run.smithi088.stdout:================================================================================ 2024-06-03T08:31:50.655 INFO:teuthology.orchestra.run.smithi088.stdout:Install 1 Package 2024-06-03T08:31:50.655 INFO:teuthology.orchestra.run.smithi088.stdout: 2024-06-03T08:31:50.655 INFO:teuthology.orchestra.run.smithi088.stdout:Total download size: 19 M 2024-06-03T08:31:50.655 INFO:teuthology.orchestra.run.smithi088.stdout:Installed size: 101 M 2024-06-03T08:31:50.655 INFO:teuthology.orchestra.run.smithi088.stdout:Downloading Packages: 2024-06-03T08:31:51.179 INFO:teuthology.orchestra.run.smithi192.stdout:Last metadata expiration check: 0:05:51 ago on Mon 03 Jun 2024 08:26:00 AM UTC. 2024-06-03T08:31:51.303 INFO:teuthology.orchestra.run.smithi192.stdout:Dependencies resolved. 2024-06-03T08:31:51.304 INFO:teuthology.orchestra.run.smithi192.stdout:================================================================================ 2024-06-03T08:31:51.304 INFO:teuthology.orchestra.run.smithi192.stdout: Package Arch Version Repo Size 2024-06-03T08:31:51.304 INFO:teuthology.orchestra.run.smithi192.stdout:================================================================================ 2024-06-03T08:31:51.304 INFO:teuthology.orchestra.run.smithi192.stdout:Installing: 2024-06-03T08:31:51.305 INFO:teuthology.orchestra.run.smithi192.stdout: ceph-mds-debuginfo x86_64 2:18.2.2-1389.gf1abf795.el9 ceph 47 M 2024-06-03T08:31:51.305 INFO:teuthology.orchestra.run.smithi192.stdout: 2024-06-03T08:31:51.305 INFO:teuthology.orchestra.run.smithi192.stdout:Transaction Summary 2024-06-03T08:31:51.305 INFO:teuthology.orchestra.run.smithi192.stdout:================================================================================ 2024-06-03T08:31:51.305 INFO:teuthology.orchestra.run.smithi192.stdout:Install 1 Package 2024-06-03T08:31:51.305 INFO:teuthology.orchestra.run.smithi192.stdout: 2024-06-03T08:31:51.305 INFO:teuthology.orchestra.run.smithi192.stdout:Total download size: 47 M 2024-06-03T08:31:51.305 INFO:teuthology.orchestra.run.smithi192.stdout:Installed size: 171 M 2024-06-03T08:31:51.305 INFO:teuthology.orchestra.run.smithi192.stdout:Downloading Packages: 2024-06-03T08:31:52.324 INFO:teuthology.orchestra.run.smithi088.stdout:ceph-mgr-debuginfo-18.2.2-1389.gf1abf795.el9.x8 12 MB/s | 19 MB 00:01 2024-06-03T08:31:52.325 INFO:teuthology.orchestra.run.smithi088.stdout:-------------------------------------------------------------------------------- 2024-06-03T08:31:52.326 INFO:teuthology.orchestra.run.smithi088.stdout:Total 12 MB/s | 19 MB 00:01 2024-06-03T08:31:52.326 INFO:teuthology.orchestra.run.smithi088.stdout:Running transaction check 2024-06-03T08:31:52.330 INFO:teuthology.orchestra.run.smithi088.stdout:Transaction check succeeded. 2024-06-03T08:31:52.330 INFO:teuthology.orchestra.run.smithi088.stdout:Running transaction test 2024-06-03T08:31:52.481 INFO:teuthology.orchestra.run.smithi088.stdout:Transaction test succeeded. 2024-06-03T08:31:52.481 INFO:teuthology.orchestra.run.smithi088.stdout:Running transaction 2024-06-03T08:31:54.683 INFO:teuthology.orchestra.run.smithi088.stdout: Preparing : 1/1 2024-06-03T08:31:54.779 INFO:teuthology.orchestra.run.smithi088.stdout: Installing : ceph-mgr-debuginfo-2:18.2.2-1389.gf1abf795.el9.x86_6 1/1 2024-06-03T08:31:55.719 INFO:teuthology.orchestra.run.smithi088.stdout: Running scriptlet: ceph-mgr-debuginfo-2:18.2.2-1389.gf1abf795.el9.x86_6 1/1 2024-06-03T08:31:55.936 INFO:teuthology.orchestra.run.smithi088.stdout: Verifying : ceph-mgr-debuginfo-2:18.2.2-1389.gf1abf795.el9.x86_6 1/1 2024-06-03T08:31:55.936 INFO:teuthology.orchestra.run.smithi088.stdout: 2024-06-03T08:31:55.936 INFO:teuthology.orchestra.run.smithi088.stdout:Installed: 2024-06-03T08:31:55.936 INFO:teuthology.orchestra.run.smithi088.stdout: ceph-mgr-debuginfo-2:18.2.2-1389.gf1abf795.el9.x86_64 2024-06-03T08:31:55.936 INFO:teuthology.orchestra.run.smithi088.stdout: 2024-06-03T08:31:55.936 INFO:teuthology.orchestra.run.smithi088.stdout:Complete! 2024-06-03T08:31:56.080 DEBUG:teuthology.orchestra.run.smithi088:> sudo yum -y install ceph-mds-debuginfo 2024-06-03T08:31:56.659 INFO:teuthology.orchestra.run.smithi088.stdout:Last metadata expiration check: 0:06:00 ago on Mon 03 Jun 2024 08:25:56 AM UTC. 2024-06-03T08:31:56.781 INFO:teuthology.orchestra.run.smithi088.stdout:Dependencies resolved. 2024-06-03T08:31:56.782 INFO:teuthology.orchestra.run.smithi088.stdout:================================================================================ 2024-06-03T08:31:56.782 INFO:teuthology.orchestra.run.smithi088.stdout: Package Arch Version Repo Size 2024-06-03T08:31:56.782 INFO:teuthology.orchestra.run.smithi088.stdout:================================================================================ 2024-06-03T08:31:56.782 INFO:teuthology.orchestra.run.smithi088.stdout:Installing: 2024-06-03T08:31:56.782 INFO:teuthology.orchestra.run.smithi088.stdout: ceph-mds-debuginfo x86_64 2:18.2.2-1389.gf1abf795.el9 ceph 47 M 2024-06-03T08:31:56.782 INFO:teuthology.orchestra.run.smithi088.stdout: 2024-06-03T08:31:56.782 INFO:teuthology.orchestra.run.smithi088.stdout:Transaction Summary 2024-06-03T08:31:56.782 INFO:teuthology.orchestra.run.smithi088.stdout:================================================================================ 2024-06-03T08:31:56.782 INFO:teuthology.orchestra.run.smithi088.stdout:Install 1 Package 2024-06-03T08:31:56.783 INFO:teuthology.orchestra.run.smithi088.stdout: 2024-06-03T08:31:56.783 INFO:teuthology.orchestra.run.smithi088.stdout:Total download size: 47 M 2024-06-03T08:31:56.783 INFO:teuthology.orchestra.run.smithi088.stdout:Installed size: 171 M 2024-06-03T08:31:56.783 INFO:teuthology.orchestra.run.smithi088.stdout:Downloading Packages: 2024-06-03T08:31:58.428 INFO:teuthology.orchestra.run.smithi192.stdout:ceph-mds-debuginfo-18.2.2-1389.gf1abf795.el9.x8 6.7 MB/s | 47 MB 00:07 2024-06-03T08:31:58.429 INFO:teuthology.orchestra.run.smithi192.stdout:-------------------------------------------------------------------------------- 2024-06-03T08:31:58.430 INFO:teuthology.orchestra.run.smithi192.stdout:Total 6.7 MB/s | 47 MB 00:07 2024-06-03T08:31:58.430 INFO:teuthology.orchestra.run.smithi192.stdout:Running transaction check 2024-06-03T08:31:58.434 INFO:teuthology.orchestra.run.smithi192.stdout:Transaction check succeeded. 2024-06-03T08:31:58.434 INFO:teuthology.orchestra.run.smithi192.stdout:Running transaction test 2024-06-03T08:31:58.746 INFO:teuthology.orchestra.run.smithi192.stdout:Transaction test succeeded. 2024-06-03T08:31:58.746 INFO:teuthology.orchestra.run.smithi192.stdout:Running transaction 2024-06-03T08:32:01.293 INFO:teuthology.orchestra.run.smithi088.stdout:ceph-mds-debuginfo-18.2.2-1389.gf1abf795.el9.x8 11 MB/s | 47 MB 00:04 2024-06-03T08:32:01.293 INFO:teuthology.orchestra.run.smithi088.stdout:-------------------------------------------------------------------------------- 2024-06-03T08:32:01.294 INFO:teuthology.orchestra.run.smithi088.stdout:Total 11 MB/s | 47 MB 00:04 2024-06-03T08:32:01.294 INFO:teuthology.orchestra.run.smithi088.stdout:Running transaction check 2024-06-03T08:32:01.298 INFO:teuthology.orchestra.run.smithi088.stdout:Transaction check succeeded. 2024-06-03T08:32:01.298 INFO:teuthology.orchestra.run.smithi088.stdout:Running transaction test 2024-06-03T08:32:01.612 INFO:teuthology.orchestra.run.smithi088.stdout:Transaction test succeeded. 2024-06-03T08:32:01.612 INFO:teuthology.orchestra.run.smithi088.stdout:Running transaction 2024-06-03T08:32:03.437 INFO:teuthology.orchestra.run.smithi192.stdout: Preparing : 1/1 2024-06-03T08:32:03.542 INFO:teuthology.orchestra.run.smithi192.stdout: Installing : ceph-mds-debuginfo-2:18.2.2-1389.gf1abf795.el9.x86_6 1/1 2024-06-03T08:32:04.858 INFO:teuthology.orchestra.run.smithi192.stdout: Running scriptlet: ceph-mds-debuginfo-2:18.2.2-1389.gf1abf795.el9.x86_6 1/1 2024-06-03T08:32:05.076 INFO:teuthology.orchestra.run.smithi192.stdout: Verifying : ceph-mds-debuginfo-2:18.2.2-1389.gf1abf795.el9.x86_6 1/1 2024-06-03T08:32:05.076 INFO:teuthology.orchestra.run.smithi192.stdout: 2024-06-03T08:32:05.077 INFO:teuthology.orchestra.run.smithi192.stdout:Installed: 2024-06-03T08:32:05.077 INFO:teuthology.orchestra.run.smithi192.stdout: ceph-mds-debuginfo-2:18.2.2-1389.gf1abf795.el9.x86_64 2024-06-03T08:32:05.077 INFO:teuthology.orchestra.run.smithi192.stdout: 2024-06-03T08:32:05.077 INFO:teuthology.orchestra.run.smithi192.stdout:Complete! 2024-06-03T08:32:05.231 DEBUG:teuthology.orchestra.run.smithi192:> sudo yum -y install ceph-mon-debuginfo 2024-06-03T08:32:05.814 INFO:teuthology.orchestra.run.smithi192.stdout:Last metadata expiration check: 0:06:05 ago on Mon 03 Jun 2024 08:26:00 AM UTC. 2024-06-03T08:32:05.938 INFO:teuthology.orchestra.run.smithi192.stdout:Dependencies resolved. 2024-06-03T08:32:05.938 INFO:teuthology.orchestra.run.smithi192.stdout:================================================================================ 2024-06-03T08:32:05.938 INFO:teuthology.orchestra.run.smithi192.stdout: Package Arch Version Repo Size 2024-06-03T08:32:05.939 INFO:teuthology.orchestra.run.smithi192.stdout:================================================================================ 2024-06-03T08:32:05.939 INFO:teuthology.orchestra.run.smithi192.stdout:Installing: 2024-06-03T08:32:05.939 INFO:teuthology.orchestra.run.smithi192.stdout: ceph-mon-debuginfo x86_64 2:18.2.2-1389.gf1abf795.el9 ceph 98 M 2024-06-03T08:32:05.939 INFO:teuthology.orchestra.run.smithi192.stdout: 2024-06-03T08:32:05.939 INFO:teuthology.orchestra.run.smithi192.stdout:Transaction Summary 2024-06-03T08:32:05.939 INFO:teuthology.orchestra.run.smithi192.stdout:================================================================================ 2024-06-03T08:32:05.939 INFO:teuthology.orchestra.run.smithi192.stdout:Install 1 Package 2024-06-03T08:32:05.939 INFO:teuthology.orchestra.run.smithi192.stdout: 2024-06-03T08:32:05.940 INFO:teuthology.orchestra.run.smithi192.stdout:Total download size: 98 M 2024-06-03T08:32:05.940 INFO:teuthology.orchestra.run.smithi192.stdout:Installed size: 450 M 2024-06-03T08:32:05.940 INFO:teuthology.orchestra.run.smithi192.stdout:Downloading Packages: 2024-06-03T08:32:06.291 INFO:teuthology.orchestra.run.smithi088.stdout: Preparing : 1/1 2024-06-03T08:32:06.439 INFO:teuthology.orchestra.run.smithi088.stdout: Installing : ceph-mds-debuginfo-2:18.2.2-1389.gf1abf795.el9.x86_6 1/1 2024-06-03T08:32:07.774 INFO:teuthology.orchestra.run.smithi088.stdout: Running scriptlet: ceph-mds-debuginfo-2:18.2.2-1389.gf1abf795.el9.x86_6 1/1 2024-06-03T08:32:08.009 INFO:teuthology.orchestra.run.smithi088.stdout: Verifying : ceph-mds-debuginfo-2:18.2.2-1389.gf1abf795.el9.x86_6 1/1 2024-06-03T08:32:08.009 INFO:teuthology.orchestra.run.smithi088.stdout: 2024-06-03T08:32:08.010 INFO:teuthology.orchestra.run.smithi088.stdout:Installed: 2024-06-03T08:32:08.010 INFO:teuthology.orchestra.run.smithi088.stdout: ceph-mds-debuginfo-2:18.2.2-1389.gf1abf795.el9.x86_64 2024-06-03T08:32:08.010 INFO:teuthology.orchestra.run.smithi088.stdout: 2024-06-03T08:32:08.010 INFO:teuthology.orchestra.run.smithi088.stdout:Complete! 2024-06-03T08:32:08.155 DEBUG:teuthology.orchestra.run.smithi088:> sudo yum -y install ceph-mon-debuginfo 2024-06-03T08:32:08.733 INFO:teuthology.orchestra.run.smithi088.stdout:Last metadata expiration check: 0:06:12 ago on Mon 03 Jun 2024 08:25:56 AM UTC. 2024-06-03T08:32:08.855 INFO:teuthology.orchestra.run.smithi088.stdout:Dependencies resolved. 2024-06-03T08:32:08.856 INFO:teuthology.orchestra.run.smithi088.stdout:================================================================================ 2024-06-03T08:32:08.856 INFO:teuthology.orchestra.run.smithi088.stdout: Package Arch Version Repo Size 2024-06-03T08:32:08.856 INFO:teuthology.orchestra.run.smithi088.stdout:================================================================================ 2024-06-03T08:32:08.857 INFO:teuthology.orchestra.run.smithi088.stdout:Installing: 2024-06-03T08:32:08.857 INFO:teuthology.orchestra.run.smithi088.stdout: ceph-mon-debuginfo x86_64 2:18.2.2-1389.gf1abf795.el9 ceph 98 M 2024-06-03T08:32:08.857 INFO:teuthology.orchestra.run.smithi088.stdout: 2024-06-03T08:32:08.857 INFO:teuthology.orchestra.run.smithi088.stdout:Transaction Summary 2024-06-03T08:32:08.857 INFO:teuthology.orchestra.run.smithi088.stdout:================================================================================ 2024-06-03T08:32:08.857 INFO:teuthology.orchestra.run.smithi088.stdout:Install 1 Package 2024-06-03T08:32:08.857 INFO:teuthology.orchestra.run.smithi088.stdout: 2024-06-03T08:32:08.857 INFO:teuthology.orchestra.run.smithi088.stdout:Total download size: 98 M 2024-06-03T08:32:08.857 INFO:teuthology.orchestra.run.smithi088.stdout:Installed size: 450 M 2024-06-03T08:32:08.858 INFO:teuthology.orchestra.run.smithi088.stdout:Downloading Packages: 2024-06-03T08:32:11.324 INFO:teuthology.orchestra.run.smithi192.stdout:ceph-mon-debuginfo-18.2.2-1389.gf1abf795.el9.x8 18 MB/s | 98 MB 00:05 2024-06-03T08:32:11.325 INFO:teuthology.orchestra.run.smithi192.stdout:-------------------------------------------------------------------------------- 2024-06-03T08:32:11.326 INFO:teuthology.orchestra.run.smithi192.stdout:Total 18 MB/s | 98 MB 00:05 2024-06-03T08:32:11.326 INFO:teuthology.orchestra.run.smithi192.stdout:Running transaction check 2024-06-03T08:32:11.330 INFO:teuthology.orchestra.run.smithi192.stdout:Transaction check succeeded. 2024-06-03T08:32:11.330 INFO:teuthology.orchestra.run.smithi192.stdout:Running transaction test 2024-06-03T08:32:11.938 INFO:teuthology.orchestra.run.smithi192.stdout:Transaction test succeeded. 2024-06-03T08:32:11.939 INFO:teuthology.orchestra.run.smithi192.stdout:Running transaction 2024-06-03T08:32:15.835 INFO:teuthology.orchestra.run.smithi088.stdout:ceph-mon-debuginfo-18.2.2-1389.gf1abf795.el9.x8 14 MB/s | 98 MB 00:06 2024-06-03T08:32:15.835 INFO:teuthology.orchestra.run.smithi088.stdout:-------------------------------------------------------------------------------- 2024-06-03T08:32:15.836 INFO:teuthology.orchestra.run.smithi088.stdout:Total 14 MB/s | 98 MB 00:06 2024-06-03T08:32:15.836 INFO:teuthology.orchestra.run.smithi088.stdout:Running transaction check 2024-06-03T08:32:15.840 INFO:teuthology.orchestra.run.smithi088.stdout:Transaction check succeeded. 2024-06-03T08:32:15.840 INFO:teuthology.orchestra.run.smithi088.stdout:Running transaction test 2024-06-03T08:32:16.451 INFO:teuthology.orchestra.run.smithi088.stdout:Transaction test succeeded. 2024-06-03T08:32:16.451 INFO:teuthology.orchestra.run.smithi088.stdout:Running transaction 2024-06-03T08:32:22.272 INFO:teuthology.orchestra.run.smithi192.stdout: Preparing : 1/1 2024-06-03T08:32:22.362 INFO:teuthology.orchestra.run.smithi192.stdout: Installing : ceph-mon-debuginfo-2:18.2.2-1389.gf1abf795.el9.x86_6 1/1 2024-06-03T08:32:25.314 INFO:teuthology.orchestra.run.smithi192.stdout: Running scriptlet: ceph-mon-debuginfo-2:18.2.2-1389.gf1abf795.el9.x86_6 1/1 2024-06-03T08:32:25.549 INFO:teuthology.orchestra.run.smithi192.stdout: Verifying : ceph-mon-debuginfo-2:18.2.2-1389.gf1abf795.el9.x86_6 1/1 2024-06-03T08:32:25.549 INFO:teuthology.orchestra.run.smithi192.stdout: 2024-06-03T08:32:25.549 INFO:teuthology.orchestra.run.smithi192.stdout:Installed: 2024-06-03T08:32:25.549 INFO:teuthology.orchestra.run.smithi192.stdout: ceph-mon-debuginfo-2:18.2.2-1389.gf1abf795.el9.x86_64 2024-06-03T08:32:25.550 INFO:teuthology.orchestra.run.smithi192.stdout: 2024-06-03T08:32:25.550 INFO:teuthology.orchestra.run.smithi192.stdout:Complete! 2024-06-03T08:32:25.749 DEBUG:teuthology.orchestra.run.smithi192:> sudo yum -y install ceph-osd-debuginfo 2024-06-03T08:32:26.324 INFO:teuthology.orchestra.run.smithi192.stdout:Last metadata expiration check: 0:06:26 ago on Mon 03 Jun 2024 08:26:00 AM UTC. 2024-06-03T08:32:26.449 INFO:teuthology.orchestra.run.smithi192.stdout:Dependencies resolved. 2024-06-03T08:32:26.450 INFO:teuthology.orchestra.run.smithi192.stdout:================================================================================ 2024-06-03T08:32:26.450 INFO:teuthology.orchestra.run.smithi192.stdout: Package Arch Version Repo Size 2024-06-03T08:32:26.450 INFO:teuthology.orchestra.run.smithi192.stdout:================================================================================ 2024-06-03T08:32:26.450 INFO:teuthology.orchestra.run.smithi192.stdout:Installing: 2024-06-03T08:32:26.450 INFO:teuthology.orchestra.run.smithi192.stdout: ceph-osd-debuginfo x86_64 2:18.2.2-1389.gf1abf795.el9 ceph 286 M 2024-06-03T08:32:26.450 INFO:teuthology.orchestra.run.smithi192.stdout: 2024-06-03T08:32:26.450 INFO:teuthology.orchestra.run.smithi192.stdout:Transaction Summary 2024-06-03T08:32:26.451 INFO:teuthology.orchestra.run.smithi192.stdout:================================================================================ 2024-06-03T08:32:26.451 INFO:teuthology.orchestra.run.smithi192.stdout:Install 1 Package 2024-06-03T08:32:26.451 INFO:teuthology.orchestra.run.smithi192.stdout: 2024-06-03T08:32:26.451 INFO:teuthology.orchestra.run.smithi192.stdout:Total download size: 286 M 2024-06-03T08:32:26.451 INFO:teuthology.orchestra.run.smithi192.stdout:Installed size: 1.2 G 2024-06-03T08:32:26.451 INFO:teuthology.orchestra.run.smithi192.stdout:Downloading Packages: 2024-06-03T08:32:26.703 INFO:teuthology.orchestra.run.smithi088.stdout: Preparing : 1/1 2024-06-03T08:32:26.822 INFO:teuthology.orchestra.run.smithi088.stdout: Installing : ceph-mon-debuginfo-2:18.2.2-1389.gf1abf795.el9.x86_6 1/1 2024-06-03T08:32:29.674 INFO:teuthology.orchestra.run.smithi088.stdout: Running scriptlet: ceph-mon-debuginfo-2:18.2.2-1389.gf1abf795.el9.x86_6 1/1 2024-06-03T08:32:29.904 INFO:teuthology.orchestra.run.smithi088.stdout: Verifying : ceph-mon-debuginfo-2:18.2.2-1389.gf1abf795.el9.x86_6 1/1 2024-06-03T08:32:29.904 INFO:teuthology.orchestra.run.smithi088.stdout: 2024-06-03T08:32:29.905 INFO:teuthology.orchestra.run.smithi088.stdout:Installed: 2024-06-03T08:32:29.905 INFO:teuthology.orchestra.run.smithi088.stdout: ceph-mon-debuginfo-2:18.2.2-1389.gf1abf795.el9.x86_64 2024-06-03T08:32:29.905 INFO:teuthology.orchestra.run.smithi088.stdout: 2024-06-03T08:32:29.905 INFO:teuthology.orchestra.run.smithi088.stdout:Complete! 2024-06-03T08:32:30.076 DEBUG:teuthology.orchestra.run.smithi088:> sudo yum -y install ceph-osd-debuginfo 2024-06-03T08:32:30.649 INFO:teuthology.orchestra.run.smithi088.stdout:Last metadata expiration check: 0:06:34 ago on Mon 03 Jun 2024 08:25:56 AM UTC. 2024-06-03T08:32:30.772 INFO:teuthology.orchestra.run.smithi088.stdout:Dependencies resolved. 2024-06-03T08:32:30.773 INFO:teuthology.orchestra.run.smithi088.stdout:================================================================================ 2024-06-03T08:32:30.773 INFO:teuthology.orchestra.run.smithi088.stdout: Package Arch Version Repo Size 2024-06-03T08:32:30.773 INFO:teuthology.orchestra.run.smithi088.stdout:================================================================================ 2024-06-03T08:32:30.773 INFO:teuthology.orchestra.run.smithi088.stdout:Installing: 2024-06-03T08:32:30.773 INFO:teuthology.orchestra.run.smithi088.stdout: ceph-osd-debuginfo x86_64 2:18.2.2-1389.gf1abf795.el9 ceph 286 M 2024-06-03T08:32:30.773 INFO:teuthology.orchestra.run.smithi088.stdout: 2024-06-03T08:32:30.773 INFO:teuthology.orchestra.run.smithi088.stdout:Transaction Summary 2024-06-03T08:32:30.773 INFO:teuthology.orchestra.run.smithi088.stdout:================================================================================ 2024-06-03T08:32:30.773 INFO:teuthology.orchestra.run.smithi088.stdout:Install 1 Package 2024-06-03T08:32:30.773 INFO:teuthology.orchestra.run.smithi088.stdout: 2024-06-03T08:32:30.774 INFO:teuthology.orchestra.run.smithi088.stdout:Total download size: 286 M 2024-06-03T08:32:30.774 INFO:teuthology.orchestra.run.smithi088.stdout:Installed size: 1.2 G 2024-06-03T08:32:30.774 INFO:teuthology.orchestra.run.smithi088.stdout:Downloading Packages: 2024-06-03T08:32:48.618 INFO:teuthology.orchestra.run.smithi088.stdout:ceph-osd-debuginfo-18.2.2-1389.gf1abf795.el9.x8 16 MB/s | 286 MB 00:17 2024-06-03T08:32:48.619 INFO:teuthology.orchestra.run.smithi088.stdout:-------------------------------------------------------------------------------- 2024-06-03T08:32:48.619 INFO:teuthology.orchestra.run.smithi088.stdout:Total 16 MB/s | 286 MB 00:17 2024-06-03T08:32:48.619 INFO:teuthology.orchestra.run.smithi088.stdout:Running transaction check 2024-06-03T08:32:48.623 INFO:teuthology.orchestra.run.smithi088.stdout:Transaction check succeeded. 2024-06-03T08:32:48.623 INFO:teuthology.orchestra.run.smithi088.stdout:Running transaction test 2024-06-03T08:32:50.316 INFO:teuthology.orchestra.run.smithi192.stdout:ceph-osd-debuginfo-18.2.2-1389.gf1abf795.el9.x8 12 MB/s | 286 MB 00:23 2024-06-03T08:32:50.317 INFO:teuthology.orchestra.run.smithi192.stdout:-------------------------------------------------------------------------------- 2024-06-03T08:32:50.317 INFO:teuthology.orchestra.run.smithi192.stdout:Total 12 MB/s | 286 MB 00:23 2024-06-03T08:32:50.318 INFO:teuthology.orchestra.run.smithi192.stdout:Running transaction check 2024-06-03T08:32:50.322 INFO:teuthology.orchestra.run.smithi192.stdout:Transaction check succeeded. 2024-06-03T08:32:50.322 INFO:teuthology.orchestra.run.smithi192.stdout:Running transaction test 2024-06-03T08:32:50.326 INFO:teuthology.orchestra.run.smithi088.stdout:Transaction test succeeded. 2024-06-03T08:32:50.326 INFO:teuthology.orchestra.run.smithi088.stdout:Running transaction 2024-06-03T08:32:52.031 INFO:teuthology.orchestra.run.smithi192.stdout:Transaction test succeeded. 2024-06-03T08:32:52.031 INFO:teuthology.orchestra.run.smithi192.stdout:Running transaction 2024-06-03T08:33:19.629 INFO:teuthology.orchestra.run.smithi088.stdout: Preparing : 1/1 2024-06-03T08:33:19.736 INFO:teuthology.orchestra.run.smithi088.stdout: Installing : ceph-osd-debuginfo-2:18.2.2-1389.gf1abf795.el9.x86_6 1/1 2024-06-03T08:33:21.347 INFO:teuthology.orchestra.run.smithi192.stdout: Preparing : 1/1 2024-06-03T08:33:21.444 INFO:teuthology.orchestra.run.smithi192.stdout: Installing : ceph-osd-debuginfo-2:18.2.2-1389.gf1abf795.el9.x86_6 1/1 2024-06-03T08:33:27.156 INFO:teuthology.orchestra.run.smithi088.stdout: Running scriptlet: ceph-osd-debuginfo-2:18.2.2-1389.gf1abf795.el9.x86_6 1/1 2024-06-03T08:33:27.382 INFO:teuthology.orchestra.run.smithi088.stdout: Verifying : ceph-osd-debuginfo-2:18.2.2-1389.gf1abf795.el9.x86_6 1/1 2024-06-03T08:33:27.382 INFO:teuthology.orchestra.run.smithi088.stdout: 2024-06-03T08:33:27.382 INFO:teuthology.orchestra.run.smithi088.stdout:Installed: 2024-06-03T08:33:27.382 INFO:teuthology.orchestra.run.smithi088.stdout: ceph-osd-debuginfo-2:18.2.2-1389.gf1abf795.el9.x86_64 2024-06-03T08:33:27.382 INFO:teuthology.orchestra.run.smithi088.stdout: 2024-06-03T08:33:27.382 INFO:teuthology.orchestra.run.smithi088.stdout:Complete! 2024-06-03T08:33:27.568 DEBUG:teuthology.orchestra.run.smithi088:> sudo yum -y install ceph-fuse-debuginfo 2024-06-03T08:33:28.149 INFO:teuthology.orchestra.run.smithi088.stdout:Last metadata expiration check: 0:07:32 ago on Mon 03 Jun 2024 08:25:56 AM UTC. 2024-06-03T08:33:28.273 INFO:teuthology.orchestra.run.smithi088.stdout:Dependencies resolved. 2024-06-03T08:33:28.273 INFO:teuthology.orchestra.run.smithi088.stdout:================================================================================ 2024-06-03T08:33:28.273 INFO:teuthology.orchestra.run.smithi088.stdout: Package Arch Version Repo Size 2024-06-03T08:33:28.273 INFO:teuthology.orchestra.run.smithi088.stdout:================================================================================ 2024-06-03T08:33:28.274 INFO:teuthology.orchestra.run.smithi088.stdout:Installing: 2024-06-03T08:33:28.274 INFO:teuthology.orchestra.run.smithi088.stdout: ceph-fuse-debuginfo x86_64 2:18.2.2-1389.gf1abf795.el9 ceph 5.8 M 2024-06-03T08:33:28.274 INFO:teuthology.orchestra.run.smithi088.stdout: 2024-06-03T08:33:28.274 INFO:teuthology.orchestra.run.smithi088.stdout:Transaction Summary 2024-06-03T08:33:28.274 INFO:teuthology.orchestra.run.smithi088.stdout:================================================================================ 2024-06-03T08:33:28.274 INFO:teuthology.orchestra.run.smithi088.stdout:Install 1 Package 2024-06-03T08:33:28.274 INFO:teuthology.orchestra.run.smithi088.stdout: 2024-06-03T08:33:28.274 INFO:teuthology.orchestra.run.smithi088.stdout:Total download size: 5.8 M 2024-06-03T08:33:28.274 INFO:teuthology.orchestra.run.smithi088.stdout:Installed size: 28 M 2024-06-03T08:33:28.274 INFO:teuthology.orchestra.run.smithi088.stdout:Downloading Packages: 2024-06-03T08:33:28.790 INFO:teuthology.orchestra.run.smithi088.stdout:ceph-fuse-debuginfo-18.2.2-1389.gf1abf795.el9.x 11 MB/s | 5.8 MB 00:00 2024-06-03T08:33:28.791 INFO:teuthology.orchestra.run.smithi088.stdout:-------------------------------------------------------------------------------- 2024-06-03T08:33:28.791 INFO:teuthology.orchestra.run.smithi088.stdout:Total 11 MB/s | 5.8 MB 00:00 2024-06-03T08:33:28.791 INFO:teuthology.orchestra.run.smithi088.stdout:Running transaction check 2024-06-03T08:33:28.794 INFO:teuthology.orchestra.run.smithi088.stdout:Transaction check succeeded. 2024-06-03T08:33:28.795 INFO:teuthology.orchestra.run.smithi088.stdout:Running transaction test 2024-06-03T08:33:28.862 INFO:teuthology.orchestra.run.smithi088.stdout:Transaction test succeeded. 2024-06-03T08:33:28.862 INFO:teuthology.orchestra.run.smithi088.stdout:Running transaction 2024-06-03T08:33:28.931 INFO:teuthology.orchestra.run.smithi192.stdout: Running scriptlet: ceph-osd-debuginfo-2:18.2.2-1389.gf1abf795.el9.x86_6 1/1 2024-06-03T08:33:29.155 INFO:teuthology.orchestra.run.smithi192.stdout: Verifying : ceph-osd-debuginfo-2:18.2.2-1389.gf1abf795.el9.x86_6 1/1 2024-06-03T08:33:29.155 INFO:teuthology.orchestra.run.smithi192.stdout: 2024-06-03T08:33:29.155 INFO:teuthology.orchestra.run.smithi192.stdout:Installed: 2024-06-03T08:33:29.155 INFO:teuthology.orchestra.run.smithi192.stdout: ceph-osd-debuginfo-2:18.2.2-1389.gf1abf795.el9.x86_64 2024-06-03T08:33:29.155 INFO:teuthology.orchestra.run.smithi192.stdout: 2024-06-03T08:33:29.155 INFO:teuthology.orchestra.run.smithi192.stdout:Complete! 2024-06-03T08:33:29.345 DEBUG:teuthology.orchestra.run.smithi192:> sudo yum -y install ceph-fuse-debuginfo 2024-06-03T08:33:29.575 INFO:teuthology.orchestra.run.smithi088.stdout: Preparing : 1/1 2024-06-03T08:33:29.669 INFO:teuthology.orchestra.run.smithi088.stdout: Installing : ceph-fuse-debuginfo-2:18.2.2-1389.gf1abf795.el9.x86_ 1/1 2024-06-03T08:33:29.928 INFO:teuthology.orchestra.run.smithi192.stdout:Last metadata expiration check: 0:07:29 ago on Mon 03 Jun 2024 08:26:00 AM UTC. 2024-06-03T08:33:30.050 INFO:teuthology.orchestra.run.smithi192.stdout:Dependencies resolved. 2024-06-03T08:33:30.051 INFO:teuthology.orchestra.run.smithi192.stdout:================================================================================ 2024-06-03T08:33:30.051 INFO:teuthology.orchestra.run.smithi192.stdout: Package Arch Version Repo Size 2024-06-03T08:33:30.051 INFO:teuthology.orchestra.run.smithi192.stdout:================================================================================ 2024-06-03T08:33:30.051 INFO:teuthology.orchestra.run.smithi192.stdout:Installing: 2024-06-03T08:33:30.051 INFO:teuthology.orchestra.run.smithi192.stdout: ceph-fuse-debuginfo x86_64 2:18.2.2-1389.gf1abf795.el9 ceph 5.8 M 2024-06-03T08:33:30.051 INFO:teuthology.orchestra.run.smithi192.stdout: 2024-06-03T08:33:30.051 INFO:teuthology.orchestra.run.smithi192.stdout:Transaction Summary 2024-06-03T08:33:30.052 INFO:teuthology.orchestra.run.smithi192.stdout:================================================================================ 2024-06-03T08:33:30.052 INFO:teuthology.orchestra.run.smithi192.stdout:Install 1 Package 2024-06-03T08:33:30.052 INFO:teuthology.orchestra.run.smithi192.stdout: 2024-06-03T08:33:30.052 INFO:teuthology.orchestra.run.smithi192.stdout:Total download size: 5.8 M 2024-06-03T08:33:30.052 INFO:teuthology.orchestra.run.smithi192.stdout:Installed size: 28 M 2024-06-03T08:33:30.052 INFO:teuthology.orchestra.run.smithi192.stdout:Downloading Packages: 2024-06-03T08:33:30.164 INFO:teuthology.orchestra.run.smithi088.stdout: Running scriptlet: ceph-fuse-debuginfo-2:18.2.2-1389.gf1abf795.el9.x86_ 1/1 2024-06-03T08:33:30.390 INFO:teuthology.orchestra.run.smithi088.stdout: Verifying : ceph-fuse-debuginfo-2:18.2.2-1389.gf1abf795.el9.x86_ 1/1 2024-06-03T08:33:30.390 INFO:teuthology.orchestra.run.smithi088.stdout: 2024-06-03T08:33:30.390 INFO:teuthology.orchestra.run.smithi088.stdout:Installed: 2024-06-03T08:33:30.390 INFO:teuthology.orchestra.run.smithi088.stdout: ceph-fuse-debuginfo-2:18.2.2-1389.gf1abf795.el9.x86_64 2024-06-03T08:33:30.390 INFO:teuthology.orchestra.run.smithi088.stdout: 2024-06-03T08:33:30.390 INFO:teuthology.orchestra.run.smithi088.stdout:Complete! 2024-06-03T08:33:30.535 DEBUG:teuthology.orchestra.run.smithi088:> sudo yum -y install librados-devel-debuginfo 2024-06-03T08:33:30.702 INFO:teuthology.orchestra.run.smithi192.stdout:ceph-fuse-debuginfo-18.2.2-1389.gf1abf795.el9.x 8.9 MB/s | 5.8 MB 00:00 2024-06-03T08:33:30.702 INFO:teuthology.orchestra.run.smithi192.stdout:-------------------------------------------------------------------------------- 2024-06-03T08:33:30.702 INFO:teuthology.orchestra.run.smithi192.stdout:Total 8.9 MB/s | 5.8 MB 00:00 2024-06-03T08:33:30.702 INFO:teuthology.orchestra.run.smithi192.stdout:Running transaction check 2024-06-03T08:33:30.706 INFO:teuthology.orchestra.run.smithi192.stdout:Transaction check succeeded. 2024-06-03T08:33:30.706 INFO:teuthology.orchestra.run.smithi192.stdout:Running transaction test 2024-06-03T08:33:30.773 INFO:teuthology.orchestra.run.smithi192.stdout:Transaction test succeeded. 2024-06-03T08:33:30.774 INFO:teuthology.orchestra.run.smithi192.stdout:Running transaction 2024-06-03T08:33:31.102 INFO:teuthology.orchestra.run.smithi088.stdout:Last metadata expiration check: 0:07:35 ago on Mon 03 Jun 2024 08:25:56 AM UTC. 2024-06-03T08:33:31.224 INFO:teuthology.orchestra.run.smithi088.stdout:Dependencies resolved. 2024-06-03T08:33:31.225 INFO:teuthology.orchestra.run.smithi088.stdout:================================================================================ 2024-06-03T08:33:31.225 INFO:teuthology.orchestra.run.smithi088.stdout: Package Arch Version Repo Size 2024-06-03T08:33:31.225 INFO:teuthology.orchestra.run.smithi088.stdout:================================================================================ 2024-06-03T08:33:31.225 INFO:teuthology.orchestra.run.smithi088.stdout:Installing: 2024-06-03T08:33:31.225 INFO:teuthology.orchestra.run.smithi088.stdout: librados-devel-debuginfo x86_64 2:18.2.2-1389.gf1abf795.el9 ceph 108 k 2024-06-03T08:33:31.225 INFO:teuthology.orchestra.run.smithi088.stdout: 2024-06-03T08:33:31.225 INFO:teuthology.orchestra.run.smithi088.stdout:Transaction Summary 2024-06-03T08:33:31.225 INFO:teuthology.orchestra.run.smithi088.stdout:================================================================================ 2024-06-03T08:33:31.226 INFO:teuthology.orchestra.run.smithi088.stdout:Install 1 Package 2024-06-03T08:33:31.226 INFO:teuthology.orchestra.run.smithi088.stdout: 2024-06-03T08:33:31.226 INFO:teuthology.orchestra.run.smithi088.stdout:Total download size: 108 k 2024-06-03T08:33:31.226 INFO:teuthology.orchestra.run.smithi088.stdout:Installed size: 440 k 2024-06-03T08:33:31.226 INFO:teuthology.orchestra.run.smithi088.stdout:Downloading Packages: 2024-06-03T08:33:31.462 INFO:teuthology.orchestra.run.smithi088.stdout:librados-devel-debuginfo-18.2.2-1389.gf1abf795. 460 kB/s | 108 kB 00:00 2024-06-03T08:33:31.462 INFO:teuthology.orchestra.run.smithi088.stdout:-------------------------------------------------------------------------------- 2024-06-03T08:33:31.462 INFO:teuthology.orchestra.run.smithi088.stdout:Total 457 kB/s | 108 kB 00:00 2024-06-03T08:33:31.463 INFO:teuthology.orchestra.run.smithi088.stdout:Running transaction check 2024-06-03T08:33:31.466 INFO:teuthology.orchestra.run.smithi088.stdout:Transaction check succeeded. 2024-06-03T08:33:31.466 INFO:teuthology.orchestra.run.smithi088.stdout:Running transaction test 2024-06-03T08:33:31.494 INFO:teuthology.orchestra.run.smithi192.stdout: Preparing : 1/1 2024-06-03T08:33:31.502 INFO:teuthology.orchestra.run.smithi088.stdout:Transaction test succeeded. 2024-06-03T08:33:31.502 INFO:teuthology.orchestra.run.smithi088.stdout:Running transaction 2024-06-03T08:33:31.589 INFO:teuthology.orchestra.run.smithi192.stdout: Installing : ceph-fuse-debuginfo-2:18.2.2-1389.gf1abf795.el9.x86_ 1/1 2024-06-03T08:33:31.620 INFO:teuthology.orchestra.run.smithi088.stdout: Preparing : 1/1 2024-06-03T08:33:31.735 INFO:teuthology.orchestra.run.smithi088.stdout: Installing : librados-devel-debuginfo-2:18.2.2-1389.gf1abf795.el9 1/1 2024-06-03T08:33:32.066 INFO:teuthology.orchestra.run.smithi088.stdout: Running scriptlet: librados-devel-debuginfo-2:18.2.2-1389.gf1abf795.el9 1/1 2024-06-03T08:33:32.096 INFO:teuthology.orchestra.run.smithi192.stdout: Running scriptlet: ceph-fuse-debuginfo-2:18.2.2-1389.gf1abf795.el9.x86_ 1/1 2024-06-03T08:33:32.289 INFO:teuthology.orchestra.run.smithi088.stdout: Verifying : librados-devel-debuginfo-2:18.2.2-1389.gf1abf795.el9 1/1 2024-06-03T08:33:32.290 INFO:teuthology.orchestra.run.smithi088.stdout: 2024-06-03T08:33:32.290 INFO:teuthology.orchestra.run.smithi088.stdout:Installed: 2024-06-03T08:33:32.290 INFO:teuthology.orchestra.run.smithi088.stdout: librados-devel-debuginfo-2:18.2.2-1389.gf1abf795.el9.x86_64 2024-06-03T08:33:32.290 INFO:teuthology.orchestra.run.smithi088.stdout: 2024-06-03T08:33:32.290 INFO:teuthology.orchestra.run.smithi088.stdout:Complete! 2024-06-03T08:33:32.329 INFO:teuthology.orchestra.run.smithi192.stdout: Verifying : ceph-fuse-debuginfo-2:18.2.2-1389.gf1abf795.el9.x86_ 1/1 2024-06-03T08:33:32.329 INFO:teuthology.orchestra.run.smithi192.stdout: 2024-06-03T08:33:32.329 INFO:teuthology.orchestra.run.smithi192.stdout:Installed: 2024-06-03T08:33:32.329 INFO:teuthology.orchestra.run.smithi192.stdout: ceph-fuse-debuginfo-2:18.2.2-1389.gf1abf795.el9.x86_64 2024-06-03T08:33:32.329 INFO:teuthology.orchestra.run.smithi192.stdout: 2024-06-03T08:33:32.329 INFO:teuthology.orchestra.run.smithi192.stdout:Complete! 2024-06-03T08:33:32.469 DEBUG:teuthology.orchestra.run.smithi088:> sudo yum -y install libcephfs2-debuginfo 2024-06-03T08:33:32.506 DEBUG:teuthology.orchestra.run.smithi192:> sudo yum -y install librados-devel-debuginfo 2024-06-03T08:33:33.043 INFO:teuthology.orchestra.run.smithi088.stdout:Last metadata expiration check: 0:07:37 ago on Mon 03 Jun 2024 08:25:56 AM UTC. 2024-06-03T08:33:33.078 INFO:teuthology.orchestra.run.smithi192.stdout:Last metadata expiration check: 0:07:33 ago on Mon 03 Jun 2024 08:26:00 AM UTC. 2024-06-03T08:33:33.166 INFO:teuthology.orchestra.run.smithi088.stdout:Dependencies resolved. 2024-06-03T08:33:33.167 INFO:teuthology.orchestra.run.smithi088.stdout:================================================================================ 2024-06-03T08:33:33.167 INFO:teuthology.orchestra.run.smithi088.stdout: Package Arch Version Repo Size 2024-06-03T08:33:33.167 INFO:teuthology.orchestra.run.smithi088.stdout:================================================================================ 2024-06-03T08:33:33.167 INFO:teuthology.orchestra.run.smithi088.stdout:Installing: 2024-06-03T08:33:33.167 INFO:teuthology.orchestra.run.smithi088.stdout: libcephfs2-debuginfo x86_64 2:18.2.2-1389.gf1abf795.el9 ceph 5.1 M 2024-06-03T08:33:33.167 INFO:teuthology.orchestra.run.smithi088.stdout: 2024-06-03T08:33:33.167 INFO:teuthology.orchestra.run.smithi088.stdout:Transaction Summary 2024-06-03T08:33:33.167 INFO:teuthology.orchestra.run.smithi088.stdout:================================================================================ 2024-06-03T08:33:33.167 INFO:teuthology.orchestra.run.smithi088.stdout:Install 1 Package 2024-06-03T08:33:33.167 INFO:teuthology.orchestra.run.smithi088.stdout: 2024-06-03T08:33:33.168 INFO:teuthology.orchestra.run.smithi088.stdout:Total download size: 5.1 M 2024-06-03T08:33:33.168 INFO:teuthology.orchestra.run.smithi088.stdout:Installed size: 24 M 2024-06-03T08:33:33.168 INFO:teuthology.orchestra.run.smithi088.stdout:Downloading Packages: 2024-06-03T08:33:33.200 INFO:teuthology.orchestra.run.smithi192.stdout:Dependencies resolved. 2024-06-03T08:33:33.201 INFO:teuthology.orchestra.run.smithi192.stdout:================================================================================ 2024-06-03T08:33:33.201 INFO:teuthology.orchestra.run.smithi192.stdout: Package Arch Version Repo Size 2024-06-03T08:33:33.201 INFO:teuthology.orchestra.run.smithi192.stdout:================================================================================ 2024-06-03T08:33:33.201 INFO:teuthology.orchestra.run.smithi192.stdout:Installing: 2024-06-03T08:33:33.201 INFO:teuthology.orchestra.run.smithi192.stdout: librados-devel-debuginfo x86_64 2:18.2.2-1389.gf1abf795.el9 ceph 108 k 2024-06-03T08:33:33.201 INFO:teuthology.orchestra.run.smithi192.stdout: 2024-06-03T08:33:33.201 INFO:teuthology.orchestra.run.smithi192.stdout:Transaction Summary 2024-06-03T08:33:33.201 INFO:teuthology.orchestra.run.smithi192.stdout:================================================================================ 2024-06-03T08:33:33.201 INFO:teuthology.orchestra.run.smithi192.stdout:Install 1 Package 2024-06-03T08:33:33.201 INFO:teuthology.orchestra.run.smithi192.stdout: 2024-06-03T08:33:33.201 INFO:teuthology.orchestra.run.smithi192.stdout:Total download size: 108 k 2024-06-03T08:33:33.201 INFO:teuthology.orchestra.run.smithi192.stdout:Installed size: 440 k 2024-06-03T08:33:33.202 INFO:teuthology.orchestra.run.smithi192.stdout:Downloading Packages: 2024-06-03T08:33:33.437 INFO:teuthology.orchestra.run.smithi192.stdout:librados-devel-debuginfo-18.2.2-1389.gf1abf795. 460 kB/s | 108 kB 00:00 2024-06-03T08:33:33.438 INFO:teuthology.orchestra.run.smithi192.stdout:-------------------------------------------------------------------------------- 2024-06-03T08:33:33.438 INFO:teuthology.orchestra.run.smithi192.stdout:Total 458 kB/s | 108 kB 00:00 2024-06-03T08:33:33.438 INFO:teuthology.orchestra.run.smithi192.stdout:Running transaction check 2024-06-03T08:33:33.441 INFO:teuthology.orchestra.run.smithi192.stdout:Transaction check succeeded. 2024-06-03T08:33:33.441 INFO:teuthology.orchestra.run.smithi192.stdout:Running transaction test 2024-06-03T08:33:33.477 INFO:teuthology.orchestra.run.smithi192.stdout:Transaction test succeeded. 2024-06-03T08:33:33.477 INFO:teuthology.orchestra.run.smithi192.stdout:Running transaction 2024-06-03T08:33:33.590 INFO:teuthology.orchestra.run.smithi192.stdout: Preparing : 1/1 2024-06-03T08:33:33.664 INFO:teuthology.orchestra.run.smithi088.stdout:libcephfs2-debuginfo-18.2.2-1389.gf1abf795.el9. 10 MB/s | 5.1 MB 00:00 2024-06-03T08:33:33.665 INFO:teuthology.orchestra.run.smithi088.stdout:-------------------------------------------------------------------------------- 2024-06-03T08:33:33.665 INFO:teuthology.orchestra.run.smithi088.stdout:Total 10 MB/s | 5.1 MB 00:00 2024-06-03T08:33:33.666 INFO:teuthology.orchestra.run.smithi088.stdout:Running transaction check 2024-06-03T08:33:33.669 INFO:teuthology.orchestra.run.smithi088.stdout:Transaction check succeeded. 2024-06-03T08:33:33.669 INFO:teuthology.orchestra.run.smithi088.stdout:Running transaction test 2024-06-03T08:33:33.693 INFO:teuthology.orchestra.run.smithi192.stdout: Installing : librados-devel-debuginfo-2:18.2.2-1389.gf1abf795.el9 1/1 2024-06-03T08:33:33.733 INFO:teuthology.orchestra.run.smithi088.stdout:Transaction test succeeded. 2024-06-03T08:33:33.734 INFO:teuthology.orchestra.run.smithi088.stdout:Running transaction 2024-06-03T08:33:34.007 INFO:teuthology.orchestra.run.smithi192.stdout: Running scriptlet: librados-devel-debuginfo-2:18.2.2-1389.gf1abf795.el9 1/1 2024-06-03T08:33:34.237 INFO:teuthology.orchestra.run.smithi192.stdout: Verifying : librados-devel-debuginfo-2:18.2.2-1389.gf1abf795.el9 1/1 2024-06-03T08:33:34.238 INFO:teuthology.orchestra.run.smithi192.stdout: 2024-06-03T08:33:34.238 INFO:teuthology.orchestra.run.smithi192.stdout:Installed: 2024-06-03T08:33:34.238 INFO:teuthology.orchestra.run.smithi192.stdout: librados-devel-debuginfo-2:18.2.2-1389.gf1abf795.el9.x86_64 2024-06-03T08:33:34.238 INFO:teuthology.orchestra.run.smithi192.stdout: 2024-06-03T08:33:34.238 INFO:teuthology.orchestra.run.smithi192.stdout:Complete! 2024-06-03T08:33:34.373 INFO:teuthology.orchestra.run.smithi088.stdout: Preparing : 1/1 2024-06-03T08:33:34.420 DEBUG:teuthology.orchestra.run.smithi192:> sudo yum -y install libcephfs2-debuginfo 2024-06-03T08:33:34.469 INFO:teuthology.orchestra.run.smithi088.stdout: Installing : libcephfs2-debuginfo-2:18.2.2-1389.gf1abf795.el9.x86 1/1 2024-06-03T08:33:34.948 INFO:teuthology.orchestra.run.smithi088.stdout: Running scriptlet: libcephfs2-debuginfo-2:18.2.2-1389.gf1abf795.el9.x86 1/1 2024-06-03T08:33:34.997 INFO:teuthology.orchestra.run.smithi192.stdout:Last metadata expiration check: 0:07:34 ago on Mon 03 Jun 2024 08:26:00 AM UTC. 2024-06-03T08:33:35.120 INFO:teuthology.orchestra.run.smithi192.stdout:Dependencies resolved. 2024-06-03T08:33:35.121 INFO:teuthology.orchestra.run.smithi192.stdout:================================================================================ 2024-06-03T08:33:35.121 INFO:teuthology.orchestra.run.smithi192.stdout: Package Arch Version Repo Size 2024-06-03T08:33:35.121 INFO:teuthology.orchestra.run.smithi192.stdout:================================================================================ 2024-06-03T08:33:35.121 INFO:teuthology.orchestra.run.smithi192.stdout:Installing: 2024-06-03T08:33:35.121 INFO:teuthology.orchestra.run.smithi192.stdout: libcephfs2-debuginfo x86_64 2:18.2.2-1389.gf1abf795.el9 ceph 5.1 M 2024-06-03T08:33:35.121 INFO:teuthology.orchestra.run.smithi192.stdout: 2024-06-03T08:33:35.121 INFO:teuthology.orchestra.run.smithi192.stdout:Transaction Summary 2024-06-03T08:33:35.121 INFO:teuthology.orchestra.run.smithi192.stdout:================================================================================ 2024-06-03T08:33:35.121 INFO:teuthology.orchestra.run.smithi192.stdout:Install 1 Package 2024-06-03T08:33:35.121 INFO:teuthology.orchestra.run.smithi192.stdout: 2024-06-03T08:33:35.121 INFO:teuthology.orchestra.run.smithi192.stdout:Total download size: 5.1 M 2024-06-03T08:33:35.121 INFO:teuthology.orchestra.run.smithi192.stdout:Installed size: 24 M 2024-06-03T08:33:35.121 INFO:teuthology.orchestra.run.smithi192.stdout:Downloading Packages: 2024-06-03T08:33:35.161 INFO:teuthology.orchestra.run.smithi088.stdout: Verifying : libcephfs2-debuginfo-2:18.2.2-1389.gf1abf795.el9.x86 1/1 2024-06-03T08:33:35.161 INFO:teuthology.orchestra.run.smithi088.stdout: 2024-06-03T08:33:35.161 INFO:teuthology.orchestra.run.smithi088.stdout:Installed: 2024-06-03T08:33:35.161 INFO:teuthology.orchestra.run.smithi088.stdout: libcephfs2-debuginfo-2:18.2.2-1389.gf1abf795.el9.x86_64 2024-06-03T08:33:35.161 INFO:teuthology.orchestra.run.smithi088.stdout: 2024-06-03T08:33:35.162 INFO:teuthology.orchestra.run.smithi088.stdout:Complete! 2024-06-03T08:33:35.336 DEBUG:teuthology.orchestra.run.smithi088:> sudo yum -y install librados2-debuginfo 2024-06-03T08:33:35.757 INFO:teuthology.orchestra.run.smithi192.stdout:libcephfs2-debuginfo-18.2.2-1389.gf1abf795.el9. 8.0 MB/s | 5.1 MB 00:00 2024-06-03T08:33:35.758 INFO:teuthology.orchestra.run.smithi192.stdout:-------------------------------------------------------------------------------- 2024-06-03T08:33:35.758 INFO:teuthology.orchestra.run.smithi192.stdout:Total 8.0 MB/s | 5.1 MB 00:00 2024-06-03T08:33:35.758 INFO:teuthology.orchestra.run.smithi192.stdout:Running transaction check 2024-06-03T08:33:35.762 INFO:teuthology.orchestra.run.smithi192.stdout:Transaction check succeeded. 2024-06-03T08:33:35.762 INFO:teuthology.orchestra.run.smithi192.stdout:Running transaction test 2024-06-03T08:33:35.826 INFO:teuthology.orchestra.run.smithi192.stdout:Transaction test succeeded. 2024-06-03T08:33:35.826 INFO:teuthology.orchestra.run.smithi192.stdout:Running transaction 2024-06-03T08:33:35.915 INFO:teuthology.orchestra.run.smithi088.stdout:Last metadata expiration check: 0:07:39 ago on Mon 03 Jun 2024 08:25:56 AM UTC. 2024-06-03T08:33:36.038 INFO:teuthology.orchestra.run.smithi088.stdout:Dependencies resolved. 2024-06-03T08:33:36.039 INFO:teuthology.orchestra.run.smithi088.stdout:================================================================================ 2024-06-03T08:33:36.039 INFO:teuthology.orchestra.run.smithi088.stdout: Package Arch Version Repo Size 2024-06-03T08:33:36.039 INFO:teuthology.orchestra.run.smithi088.stdout:================================================================================ 2024-06-03T08:33:36.039 INFO:teuthology.orchestra.run.smithi088.stdout:Installing: 2024-06-03T08:33:36.039 INFO:teuthology.orchestra.run.smithi088.stdout: librados2-debuginfo x86_64 2:18.2.2-1389.gf1abf795.el9 ceph 45 M 2024-06-03T08:33:36.039 INFO:teuthology.orchestra.run.smithi088.stdout: 2024-06-03T08:33:36.039 INFO:teuthology.orchestra.run.smithi088.stdout:Transaction Summary 2024-06-03T08:33:36.039 INFO:teuthology.orchestra.run.smithi088.stdout:================================================================================ 2024-06-03T08:33:36.039 INFO:teuthology.orchestra.run.smithi088.stdout:Install 1 Package 2024-06-03T08:33:36.039 INFO:teuthology.orchestra.run.smithi088.stdout: 2024-06-03T08:33:36.039 INFO:teuthology.orchestra.run.smithi088.stdout:Total download size: 45 M 2024-06-03T08:33:36.040 INFO:teuthology.orchestra.run.smithi088.stdout:Installed size: 256 M 2024-06-03T08:33:36.040 INFO:teuthology.orchestra.run.smithi088.stdout:Downloading Packages: 2024-06-03T08:33:36.471 INFO:teuthology.orchestra.run.smithi192.stdout: Preparing : 1/1 2024-06-03T08:33:36.570 INFO:teuthology.orchestra.run.smithi192.stdout: Installing : libcephfs2-debuginfo-2:18.2.2-1389.gf1abf795.el9.x86 1/1 2024-06-03T08:33:37.068 INFO:teuthology.orchestra.run.smithi192.stdout: Running scriptlet: libcephfs2-debuginfo-2:18.2.2-1389.gf1abf795.el9.x86 1/1 2024-06-03T08:33:37.294 INFO:teuthology.orchestra.run.smithi192.stdout: Verifying : libcephfs2-debuginfo-2:18.2.2-1389.gf1abf795.el9.x86 1/1 2024-06-03T08:33:37.294 INFO:teuthology.orchestra.run.smithi192.stdout: 2024-06-03T08:33:37.295 INFO:teuthology.orchestra.run.smithi192.stdout:Installed: 2024-06-03T08:33:37.295 INFO:teuthology.orchestra.run.smithi192.stdout: libcephfs2-debuginfo-2:18.2.2-1389.gf1abf795.el9.x86_64 2024-06-03T08:33:37.295 INFO:teuthology.orchestra.run.smithi192.stdout: 2024-06-03T08:33:37.295 INFO:teuthology.orchestra.run.smithi192.stdout:Complete! 2024-06-03T08:33:37.469 DEBUG:teuthology.orchestra.run.smithi192:> sudo yum -y install librados2-debuginfo 2024-06-03T08:33:38.050 INFO:teuthology.orchestra.run.smithi192.stdout:Last metadata expiration check: 0:07:38 ago on Mon 03 Jun 2024 08:26:00 AM UTC. 2024-06-03T08:33:38.171 INFO:teuthology.orchestra.run.smithi192.stdout:Dependencies resolved. 2024-06-03T08:33:38.172 INFO:teuthology.orchestra.run.smithi192.stdout:================================================================================ 2024-06-03T08:33:38.172 INFO:teuthology.orchestra.run.smithi192.stdout: Package Arch Version Repo Size 2024-06-03T08:33:38.172 INFO:teuthology.orchestra.run.smithi192.stdout:================================================================================ 2024-06-03T08:33:38.172 INFO:teuthology.orchestra.run.smithi192.stdout:Installing: 2024-06-03T08:33:38.173 INFO:teuthology.orchestra.run.smithi192.stdout: librados2-debuginfo x86_64 2:18.2.2-1389.gf1abf795.el9 ceph 45 M 2024-06-03T08:33:38.173 INFO:teuthology.orchestra.run.smithi192.stdout: 2024-06-03T08:33:38.173 INFO:teuthology.orchestra.run.smithi192.stdout:Transaction Summary 2024-06-03T08:33:38.173 INFO:teuthology.orchestra.run.smithi192.stdout:================================================================================ 2024-06-03T08:33:38.173 INFO:teuthology.orchestra.run.smithi192.stdout:Install 1 Package 2024-06-03T08:33:38.173 INFO:teuthology.orchestra.run.smithi192.stdout: 2024-06-03T08:33:38.173 INFO:teuthology.orchestra.run.smithi192.stdout:Total download size: 45 M 2024-06-03T08:33:38.173 INFO:teuthology.orchestra.run.smithi192.stdout:Installed size: 256 M 2024-06-03T08:33:38.173 INFO:teuthology.orchestra.run.smithi192.stdout:Downloading Packages: 2024-06-03T08:33:38.484 INFO:teuthology.orchestra.run.smithi088.stdout:librados2-debuginfo-18.2.2-1389.gf1abf795.el9.x 19 MB/s | 45 MB 00:02 2024-06-03T08:33:38.486 INFO:teuthology.orchestra.run.smithi088.stdout:-------------------------------------------------------------------------------- 2024-06-03T08:33:38.486 INFO:teuthology.orchestra.run.smithi088.stdout:Total 19 MB/s | 45 MB 00:02 2024-06-03T08:33:38.486 INFO:teuthology.orchestra.run.smithi088.stdout:Running transaction check 2024-06-03T08:33:38.489 INFO:teuthology.orchestra.run.smithi088.stdout:Transaction check succeeded. 2024-06-03T08:33:38.489 INFO:teuthology.orchestra.run.smithi088.stdout:Running transaction test 2024-06-03T08:33:38.797 INFO:teuthology.orchestra.run.smithi088.stdout:Transaction test succeeded. 2024-06-03T08:33:38.797 INFO:teuthology.orchestra.run.smithi088.stdout:Running transaction 2024-06-03T08:33:42.368 INFO:teuthology.orchestra.run.smithi192.stdout:librados2-debuginfo-18.2.2-1389.gf1abf795.el9.x 11 MB/s | 45 MB 00:04 2024-06-03T08:33:42.369 INFO:teuthology.orchestra.run.smithi192.stdout:-------------------------------------------------------------------------------- 2024-06-03T08:33:42.369 INFO:teuthology.orchestra.run.smithi192.stdout:Total 11 MB/s | 45 MB 00:04 2024-06-03T08:33:42.370 INFO:teuthology.orchestra.run.smithi192.stdout:Running transaction check 2024-06-03T08:33:42.373 INFO:teuthology.orchestra.run.smithi192.stdout:Transaction check succeeded. 2024-06-03T08:33:42.374 INFO:teuthology.orchestra.run.smithi192.stdout:Running transaction test 2024-06-03T08:33:42.680 INFO:teuthology.orchestra.run.smithi192.stdout:Transaction test succeeded. 2024-06-03T08:33:42.680 INFO:teuthology.orchestra.run.smithi192.stdout:Running transaction 2024-06-03T08:33:43.934 INFO:teuthology.orchestra.run.smithi088.stdout: Preparing : 1/1 2024-06-03T08:33:44.030 INFO:teuthology.orchestra.run.smithi088.stdout: Installing : librados2-debuginfo-2:18.2.2-1389.gf1abf795.el9.x86_ 1/1 2024-06-03T08:33:45.851 INFO:teuthology.orchestra.run.smithi088.stdout: Running scriptlet: librados2-debuginfo-2:18.2.2-1389.gf1abf795.el9.x86_ 1/1 2024-06-03T08:33:46.087 INFO:teuthology.orchestra.run.smithi088.stdout: Verifying : librados2-debuginfo-2:18.2.2-1389.gf1abf795.el9.x86_ 1/1 2024-06-03T08:33:46.087 INFO:teuthology.orchestra.run.smithi088.stdout: 2024-06-03T08:33:46.087 INFO:teuthology.orchestra.run.smithi088.stdout:Installed: 2024-06-03T08:33:46.087 INFO:teuthology.orchestra.run.smithi088.stdout: librados2-debuginfo-2:18.2.2-1389.gf1abf795.el9.x86_64 2024-06-03T08:33:46.087 INFO:teuthology.orchestra.run.smithi088.stdout: 2024-06-03T08:33:46.087 INFO:teuthology.orchestra.run.smithi088.stdout:Complete! 2024-06-03T08:33:46.238 DEBUG:teuthology.orchestra.run.smithi088:> sudo yum -y install librbd1-debuginfo 2024-06-03T08:33:46.815 INFO:teuthology.orchestra.run.smithi088.stdout:Last metadata expiration check: 0:07:50 ago on Mon 03 Jun 2024 08:25:56 AM UTC. 2024-06-03T08:33:46.939 INFO:teuthology.orchestra.run.smithi088.stdout:Dependencies resolved. 2024-06-03T08:33:46.940 INFO:teuthology.orchestra.run.smithi088.stdout:================================================================================ 2024-06-03T08:33:46.940 INFO:teuthology.orchestra.run.smithi088.stdout: Package Arch Version Repo Size 2024-06-03T08:33:46.940 INFO:teuthology.orchestra.run.smithi088.stdout:================================================================================ 2024-06-03T08:33:46.940 INFO:teuthology.orchestra.run.smithi088.stdout:Installing: 2024-06-03T08:33:46.940 INFO:teuthology.orchestra.run.smithi088.stdout: librbd1-debuginfo x86_64 2:18.2.2-1389.gf1abf795.el9 ceph 50 M 2024-06-03T08:33:46.940 INFO:teuthology.orchestra.run.smithi088.stdout: 2024-06-03T08:33:46.940 INFO:teuthology.orchestra.run.smithi088.stdout:Transaction Summary 2024-06-03T08:33:46.940 INFO:teuthology.orchestra.run.smithi088.stdout:================================================================================ 2024-06-03T08:33:46.940 INFO:teuthology.orchestra.run.smithi088.stdout:Install 1 Package 2024-06-03T08:33:46.940 INFO:teuthology.orchestra.run.smithi088.stdout: 2024-06-03T08:33:46.941 INFO:teuthology.orchestra.run.smithi088.stdout:Total download size: 50 M 2024-06-03T08:33:46.941 INFO:teuthology.orchestra.run.smithi088.stdout:Installed size: 291 M 2024-06-03T08:33:46.941 INFO:teuthology.orchestra.run.smithi088.stdout:Downloading Packages: 2024-06-03T08:33:47.847 INFO:teuthology.orchestra.run.smithi192.stdout: Preparing : 1/1 2024-06-03T08:33:47.948 INFO:teuthology.orchestra.run.smithi192.stdout: Installing : librados2-debuginfo-2:18.2.2-1389.gf1abf795.el9.x86_ 1/1 2024-06-03T08:33:49.619 INFO:teuthology.orchestra.run.smithi088.stdout:librbd1-debuginfo-18.2.2-1389.gf1abf795.el9.x86 19 MB/s | 50 MB 00:02 2024-06-03T08:33:49.619 INFO:teuthology.orchestra.run.smithi088.stdout:-------------------------------------------------------------------------------- 2024-06-03T08:33:49.620 INFO:teuthology.orchestra.run.smithi088.stdout:Total 19 MB/s | 50 MB 00:02 2024-06-03T08:33:49.620 INFO:teuthology.orchestra.run.smithi088.stdout:Running transaction check 2024-06-03T08:33:49.624 INFO:teuthology.orchestra.run.smithi088.stdout:Transaction check succeeded. 2024-06-03T08:33:49.624 INFO:teuthology.orchestra.run.smithi088.stdout:Running transaction test 2024-06-03T08:33:49.741 INFO:teuthology.orchestra.run.smithi192.stdout: Running scriptlet: librados2-debuginfo-2:18.2.2-1389.gf1abf795.el9.x86_ 1/1 2024-06-03T08:33:49.960 INFO:teuthology.orchestra.run.smithi088.stdout:Transaction test succeeded. 2024-06-03T08:33:49.960 INFO:teuthology.orchestra.run.smithi088.stdout:Running transaction 2024-06-03T08:33:49.989 INFO:teuthology.orchestra.run.smithi192.stdout: Verifying : librados2-debuginfo-2:18.2.2-1389.gf1abf795.el9.x86_ 1/1 2024-06-03T08:33:49.989 INFO:teuthology.orchestra.run.smithi192.stdout: 2024-06-03T08:33:49.989 INFO:teuthology.orchestra.run.smithi192.stdout:Installed: 2024-06-03T08:33:49.990 INFO:teuthology.orchestra.run.smithi192.stdout: librados2-debuginfo-2:18.2.2-1389.gf1abf795.el9.x86_64 2024-06-03T08:33:49.990 INFO:teuthology.orchestra.run.smithi192.stdout: 2024-06-03T08:33:49.990 INFO:teuthology.orchestra.run.smithi192.stdout:Complete! 2024-06-03T08:33:50.176 DEBUG:teuthology.orchestra.run.smithi192:> sudo yum -y install librbd1-debuginfo 2024-06-03T08:33:50.753 INFO:teuthology.orchestra.run.smithi192.stdout:Last metadata expiration check: 0:07:50 ago on Mon 03 Jun 2024 08:26:00 AM UTC. 2024-06-03T08:33:50.877 INFO:teuthology.orchestra.run.smithi192.stdout:Dependencies resolved. 2024-06-03T08:33:50.878 INFO:teuthology.orchestra.run.smithi192.stdout:================================================================================ 2024-06-03T08:33:50.878 INFO:teuthology.orchestra.run.smithi192.stdout: Package Arch Version Repo Size 2024-06-03T08:33:50.878 INFO:teuthology.orchestra.run.smithi192.stdout:================================================================================ 2024-06-03T08:33:50.878 INFO:teuthology.orchestra.run.smithi192.stdout:Installing: 2024-06-03T08:33:50.878 INFO:teuthology.orchestra.run.smithi192.stdout: librbd1-debuginfo x86_64 2:18.2.2-1389.gf1abf795.el9 ceph 50 M 2024-06-03T08:33:50.878 INFO:teuthology.orchestra.run.smithi192.stdout: 2024-06-03T08:33:50.878 INFO:teuthology.orchestra.run.smithi192.stdout:Transaction Summary 2024-06-03T08:33:50.879 INFO:teuthology.orchestra.run.smithi192.stdout:================================================================================ 2024-06-03T08:33:50.879 INFO:teuthology.orchestra.run.smithi192.stdout:Install 1 Package 2024-06-03T08:33:50.879 INFO:teuthology.orchestra.run.smithi192.stdout: 2024-06-03T08:33:50.879 INFO:teuthology.orchestra.run.smithi192.stdout:Total download size: 50 M 2024-06-03T08:33:50.879 INFO:teuthology.orchestra.run.smithi192.stdout:Installed size: 291 M 2024-06-03T08:33:50.879 INFO:teuthology.orchestra.run.smithi192.stdout:Downloading Packages: 2024-06-03T08:33:54.754 INFO:teuthology.orchestra.run.smithi192.stdout:librbd1-debuginfo-18.2.2-1389.gf1abf795.el9.x86 13 MB/s | 50 MB 00:03 2024-06-03T08:33:54.754 INFO:teuthology.orchestra.run.smithi192.stdout:-------------------------------------------------------------------------------- 2024-06-03T08:33:54.755 INFO:teuthology.orchestra.run.smithi192.stdout:Total 13 MB/s | 50 MB 00:03 2024-06-03T08:33:54.756 INFO:teuthology.orchestra.run.smithi192.stdout:Running transaction check 2024-06-03T08:33:54.759 INFO:teuthology.orchestra.run.smithi192.stdout:Transaction check succeeded. 2024-06-03T08:33:54.759 INFO:teuthology.orchestra.run.smithi192.stdout:Running transaction test 2024-06-03T08:33:55.093 INFO:teuthology.orchestra.run.smithi192.stdout:Transaction test succeeded. 2024-06-03T08:33:55.093 INFO:teuthology.orchestra.run.smithi192.stdout:Running transaction 2024-06-03T08:33:55.718 INFO:teuthology.orchestra.run.smithi088.stdout: Preparing : 1/1 2024-06-03T08:33:55.815 INFO:teuthology.orchestra.run.smithi088.stdout: Installing : librbd1-debuginfo-2:18.2.2-1389.gf1abf795.el9.x86_64 1/1 2024-06-03T08:33:57.863 INFO:teuthology.orchestra.run.smithi088.stdout: Running scriptlet: librbd1-debuginfo-2:18.2.2-1389.gf1abf795.el9.x86_64 1/1 2024-06-03T08:33:58.084 INFO:teuthology.orchestra.run.smithi088.stdout: Verifying : librbd1-debuginfo-2:18.2.2-1389.gf1abf795.el9.x86_64 1/1 2024-06-03T08:33:58.084 INFO:teuthology.orchestra.run.smithi088.stdout: 2024-06-03T08:33:58.084 INFO:teuthology.orchestra.run.smithi088.stdout:Installed: 2024-06-03T08:33:58.084 INFO:teuthology.orchestra.run.smithi088.stdout: librbd1-debuginfo-2:18.2.2-1389.gf1abf795.el9.x86_64 2024-06-03T08:33:58.084 INFO:teuthology.orchestra.run.smithi088.stdout: 2024-06-03T08:33:58.084 INFO:teuthology.orchestra.run.smithi088.stdout:Complete! 2024-06-03T08:33:58.239 DEBUG:teuthology.orchestra.run.smithi088:> sudo yum -y install python3-cephfs-debuginfo 2024-06-03T08:33:58.813 INFO:teuthology.orchestra.run.smithi088.stdout:Last metadata expiration check: 0:08:02 ago on Mon 03 Jun 2024 08:25:56 AM UTC. 2024-06-03T08:33:58.936 INFO:teuthology.orchestra.run.smithi088.stdout:Dependencies resolved. 2024-06-03T08:33:58.936 INFO:teuthology.orchestra.run.smithi088.stdout:================================================================================ 2024-06-03T08:33:58.936 INFO:teuthology.orchestra.run.smithi088.stdout: Package Arch Version Repo Size 2024-06-03T08:33:58.936 INFO:teuthology.orchestra.run.smithi088.stdout:================================================================================ 2024-06-03T08:33:58.936 INFO:teuthology.orchestra.run.smithi088.stdout:Installing: 2024-06-03T08:33:58.936 INFO:teuthology.orchestra.run.smithi088.stdout: python3-cephfs-debuginfo x86_64 2:18.2.2-1389.gf1abf795.el9 ceph 615 k 2024-06-03T08:33:58.936 INFO:teuthology.orchestra.run.smithi088.stdout: 2024-06-03T08:33:58.936 INFO:teuthology.orchestra.run.smithi088.stdout:Transaction Summary 2024-06-03T08:33:58.937 INFO:teuthology.orchestra.run.smithi088.stdout:================================================================================ 2024-06-03T08:33:58.937 INFO:teuthology.orchestra.run.smithi088.stdout:Install 1 Package 2024-06-03T08:33:58.937 INFO:teuthology.orchestra.run.smithi088.stdout: 2024-06-03T08:33:58.937 INFO:teuthology.orchestra.run.smithi088.stdout:Total download size: 615 k 2024-06-03T08:33:58.937 INFO:teuthology.orchestra.run.smithi088.stdout:Installed size: 2.1 M 2024-06-03T08:33:58.937 INFO:teuthology.orchestra.run.smithi088.stdout:Downloading Packages: 2024-06-03T08:33:59.704 INFO:teuthology.orchestra.run.smithi088.stdout:python3-cephfs-debuginfo-18.2.2-1389.gf1abf795. 802 kB/s | 615 kB 00:00 2024-06-03T08:33:59.704 INFO:teuthology.orchestra.run.smithi088.stdout:-------------------------------------------------------------------------------- 2024-06-03T08:33:59.705 INFO:teuthology.orchestra.run.smithi088.stdout:Total 800 kB/s | 615 kB 00:00 2024-06-03T08:33:59.705 INFO:teuthology.orchestra.run.smithi088.stdout:Running transaction check 2024-06-03T08:33:59.709 INFO:teuthology.orchestra.run.smithi088.stdout:Transaction check succeeded. 2024-06-03T08:33:59.709 INFO:teuthology.orchestra.run.smithi088.stdout:Running transaction test 2024-06-03T08:33:59.748 INFO:teuthology.orchestra.run.smithi088.stdout:Transaction test succeeded. 2024-06-03T08:33:59.748 INFO:teuthology.orchestra.run.smithi088.stdout:Running transaction 2024-06-03T08:33:59.916 INFO:teuthology.orchestra.run.smithi088.stdout: Preparing : 1/1 2024-06-03T08:34:00.019 INFO:teuthology.orchestra.run.smithi088.stdout: Installing : python3-cephfs-debuginfo-2:18.2.2-1389.gf1abf795.el9 1/1 2024-06-03T08:34:00.431 INFO:teuthology.orchestra.run.smithi088.stdout: Running scriptlet: python3-cephfs-debuginfo-2:18.2.2-1389.gf1abf795.el9 1/1 2024-06-03T08:34:00.645 INFO:teuthology.orchestra.run.smithi088.stdout: Verifying : python3-cephfs-debuginfo-2:18.2.2-1389.gf1abf795.el9 1/1 2024-06-03T08:34:00.646 INFO:teuthology.orchestra.run.smithi088.stdout: 2024-06-03T08:34:00.646 INFO:teuthology.orchestra.run.smithi088.stdout:Installed: 2024-06-03T08:34:00.646 INFO:teuthology.orchestra.run.smithi088.stdout: python3-cephfs-debuginfo-2:18.2.2-1389.gf1abf795.el9.x86_64 2024-06-03T08:34:00.646 INFO:teuthology.orchestra.run.smithi088.stdout: 2024-06-03T08:34:00.646 INFO:teuthology.orchestra.run.smithi088.stdout:Complete! 2024-06-03T08:34:00.800 DEBUG:teuthology.orchestra.run.smithi088:> sudo yum -y install python3-rados-debuginfo 2024-06-03T08:34:00.844 INFO:teuthology.orchestra.run.smithi192.stdout: Preparing : 1/1 2024-06-03T08:34:01.209 INFO:teuthology.orchestra.run.smithi192.stdout: Installing : librbd1-debuginfo-2:18.2.2-1389.gf1abf795.el9.x86_64 1/1 2024-06-03T08:34:01.378 INFO:teuthology.orchestra.run.smithi088.stdout:Last metadata expiration check: 0:08:05 ago on Mon 03 Jun 2024 08:25:56 AM UTC. 2024-06-03T08:34:01.501 INFO:teuthology.orchestra.run.smithi088.stdout:Dependencies resolved. 2024-06-03T08:34:01.502 INFO:teuthology.orchestra.run.smithi088.stdout:================================================================================ 2024-06-03T08:34:01.502 INFO:teuthology.orchestra.run.smithi088.stdout: Package Arch Version Repo Size 2024-06-03T08:34:01.502 INFO:teuthology.orchestra.run.smithi088.stdout:================================================================================ 2024-06-03T08:34:01.502 INFO:teuthology.orchestra.run.smithi088.stdout:Installing: 2024-06-03T08:34:01.503 INFO:teuthology.orchestra.run.smithi088.stdout: python3-rados-debuginfo x86_64 2:18.2.2-1389.gf1abf795.el9 ceph 770 k 2024-06-03T08:34:01.503 INFO:teuthology.orchestra.run.smithi088.stdout: 2024-06-03T08:34:01.503 INFO:teuthology.orchestra.run.smithi088.stdout:Transaction Summary 2024-06-03T08:34:01.503 INFO:teuthology.orchestra.run.smithi088.stdout:================================================================================ 2024-06-03T08:34:01.503 INFO:teuthology.orchestra.run.smithi088.stdout:Install 1 Package 2024-06-03T08:34:01.503 INFO:teuthology.orchestra.run.smithi088.stdout: 2024-06-03T08:34:01.503 INFO:teuthology.orchestra.run.smithi088.stdout:Total download size: 770 k 2024-06-03T08:34:01.503 INFO:teuthology.orchestra.run.smithi088.stdout:Installed size: 2.7 M 2024-06-03T08:34:01.503 INFO:teuthology.orchestra.run.smithi088.stdout:Downloading Packages: 2024-06-03T08:34:01.807 INFO:teuthology.orchestra.run.smithi088.stdout:python3-rados-debuginfo-18.2.2-1389.gf1abf795.e 2.5 MB/s | 770 kB 00:00 2024-06-03T08:34:01.807 INFO:teuthology.orchestra.run.smithi088.stdout:-------------------------------------------------------------------------------- 2024-06-03T08:34:01.808 INFO:teuthology.orchestra.run.smithi088.stdout:Total 2.5 MB/s | 770 kB 00:00 2024-06-03T08:34:01.808 INFO:teuthology.orchestra.run.smithi088.stdout:Running transaction check 2024-06-03T08:34:01.811 INFO:teuthology.orchestra.run.smithi088.stdout:Transaction check succeeded. 2024-06-03T08:34:01.811 INFO:teuthology.orchestra.run.smithi088.stdout:Running transaction test 2024-06-03T08:34:01.849 INFO:teuthology.orchestra.run.smithi088.stdout:Transaction test succeeded. 2024-06-03T08:34:01.849 INFO:teuthology.orchestra.run.smithi088.stdout:Running transaction 2024-06-03T08:34:02.027 INFO:teuthology.orchestra.run.smithi088.stdout: Preparing : 1/1 2024-06-03T08:34:02.129 INFO:teuthology.orchestra.run.smithi088.stdout: Installing : python3-rados-debuginfo-2:18.2.2-1389.gf1abf795.el9. 1/1 2024-06-03T08:34:02.492 INFO:teuthology.orchestra.run.smithi088.stdout: Running scriptlet: python3-rados-debuginfo-2:18.2.2-1389.gf1abf795.el9. 1/1 2024-06-03T08:34:02.728 INFO:teuthology.orchestra.run.smithi088.stdout: Verifying : python3-rados-debuginfo-2:18.2.2-1389.gf1abf795.el9. 1/1 2024-06-03T08:34:02.728 INFO:teuthology.orchestra.run.smithi088.stdout: 2024-06-03T08:34:02.728 INFO:teuthology.orchestra.run.smithi088.stdout:Installed: 2024-06-03T08:34:02.728 INFO:teuthology.orchestra.run.smithi088.stdout: python3-rados-debuginfo-2:18.2.2-1389.gf1abf795.el9.x86_64 2024-06-03T08:34:02.729 INFO:teuthology.orchestra.run.smithi088.stdout: 2024-06-03T08:34:02.729 INFO:teuthology.orchestra.run.smithi088.stdout:Complete! 2024-06-03T08:34:02.887 DEBUG:teuthology.orchestra.run.smithi088:> sudo yum -y install python3-rbd-debuginfo 2024-06-03T08:34:03.265 INFO:teuthology.orchestra.run.smithi192.stdout: Running scriptlet: librbd1-debuginfo-2:18.2.2-1389.gf1abf795.el9.x86_64 1/1 2024-06-03T08:34:03.463 INFO:teuthology.orchestra.run.smithi088.stdout:Last metadata expiration check: 0:08:07 ago on Mon 03 Jun 2024 08:25:56 AM UTC. 2024-06-03T08:34:03.497 INFO:teuthology.orchestra.run.smithi192.stdout: Verifying : librbd1-debuginfo-2:18.2.2-1389.gf1abf795.el9.x86_64 1/1 2024-06-03T08:34:03.498 INFO:teuthology.orchestra.run.smithi192.stdout: 2024-06-03T08:34:03.498 INFO:teuthology.orchestra.run.smithi192.stdout:Installed: 2024-06-03T08:34:03.498 INFO:teuthology.orchestra.run.smithi192.stdout: librbd1-debuginfo-2:18.2.2-1389.gf1abf795.el9.x86_64 2024-06-03T08:34:03.498 INFO:teuthology.orchestra.run.smithi192.stdout: 2024-06-03T08:34:03.498 INFO:teuthology.orchestra.run.smithi192.stdout:Complete! 2024-06-03T08:34:03.586 INFO:teuthology.orchestra.run.smithi088.stdout:Dependencies resolved. 2024-06-03T08:34:03.587 INFO:teuthology.orchestra.run.smithi088.stdout:================================================================================ 2024-06-03T08:34:03.587 INFO:teuthology.orchestra.run.smithi088.stdout: Package Arch Version Repo Size 2024-06-03T08:34:03.587 INFO:teuthology.orchestra.run.smithi088.stdout:================================================================================ 2024-06-03T08:34:03.587 INFO:teuthology.orchestra.run.smithi088.stdout:Installing: 2024-06-03T08:34:03.587 INFO:teuthology.orchestra.run.smithi088.stdout: python3-rbd-debuginfo x86_64 2:18.2.2-1389.gf1abf795.el9 ceph 1.1 M 2024-06-03T08:34:03.587 INFO:teuthology.orchestra.run.smithi088.stdout: 2024-06-03T08:34:03.587 INFO:teuthology.orchestra.run.smithi088.stdout:Transaction Summary 2024-06-03T08:34:03.587 INFO:teuthology.orchestra.run.smithi088.stdout:================================================================================ 2024-06-03T08:34:03.587 INFO:teuthology.orchestra.run.smithi088.stdout:Install 1 Package 2024-06-03T08:34:03.587 INFO:teuthology.orchestra.run.smithi088.stdout: 2024-06-03T08:34:03.587 INFO:teuthology.orchestra.run.smithi088.stdout:Total download size: 1.1 M 2024-06-03T08:34:03.588 INFO:teuthology.orchestra.run.smithi088.stdout:Installed size: 4.3 M 2024-06-03T08:34:03.588 INFO:teuthology.orchestra.run.smithi088.stdout:Downloading Packages: 2024-06-03T08:34:03.650 DEBUG:teuthology.orchestra.run.smithi192:> sudo yum -y install python3-cephfs-debuginfo 2024-06-03T08:34:04.227 INFO:teuthology.orchestra.run.smithi192.stdout:Last metadata expiration check: 0:08:04 ago on Mon 03 Jun 2024 08:26:00 AM UTC. 2024-06-03T08:34:04.268 INFO:teuthology.orchestra.run.smithi088.stdout:python3-rbd-debuginfo-18.2.2-1389.gf1abf795.el9 1.7 MB/s | 1.1 MB 00:00 2024-06-03T08:34:04.268 INFO:teuthology.orchestra.run.smithi088.stdout:-------------------------------------------------------------------------------- 2024-06-03T08:34:04.269 INFO:teuthology.orchestra.run.smithi088.stdout:Total 1.7 MB/s | 1.1 MB 00:00 2024-06-03T08:34:04.269 INFO:teuthology.orchestra.run.smithi088.stdout:Running transaction check 2024-06-03T08:34:04.272 INFO:teuthology.orchestra.run.smithi088.stdout:Transaction check succeeded. 2024-06-03T08:34:04.272 INFO:teuthology.orchestra.run.smithi088.stdout:Running transaction test 2024-06-03T08:34:04.315 INFO:teuthology.orchestra.run.smithi088.stdout:Transaction test succeeded. 2024-06-03T08:34:04.315 INFO:teuthology.orchestra.run.smithi088.stdout:Running transaction 2024-06-03T08:34:04.350 INFO:teuthology.orchestra.run.smithi192.stdout:Dependencies resolved. 2024-06-03T08:34:04.350 INFO:teuthology.orchestra.run.smithi192.stdout:================================================================================ 2024-06-03T08:34:04.350 INFO:teuthology.orchestra.run.smithi192.stdout: Package Arch Version Repo Size 2024-06-03T08:34:04.351 INFO:teuthology.orchestra.run.smithi192.stdout:================================================================================ 2024-06-03T08:34:04.351 INFO:teuthology.orchestra.run.smithi192.stdout:Installing: 2024-06-03T08:34:04.351 INFO:teuthology.orchestra.run.smithi192.stdout: python3-cephfs-debuginfo x86_64 2:18.2.2-1389.gf1abf795.el9 ceph 615 k 2024-06-03T08:34:04.351 INFO:teuthology.orchestra.run.smithi192.stdout: 2024-06-03T08:34:04.351 INFO:teuthology.orchestra.run.smithi192.stdout:Transaction Summary 2024-06-03T08:34:04.351 INFO:teuthology.orchestra.run.smithi192.stdout:================================================================================ 2024-06-03T08:34:04.351 INFO:teuthology.orchestra.run.smithi192.stdout:Install 1 Package 2024-06-03T08:34:04.351 INFO:teuthology.orchestra.run.smithi192.stdout: 2024-06-03T08:34:04.351 INFO:teuthology.orchestra.run.smithi192.stdout:Total download size: 615 k 2024-06-03T08:34:04.351 INFO:teuthology.orchestra.run.smithi192.stdout:Installed size: 2.1 M 2024-06-03T08:34:04.351 INFO:teuthology.orchestra.run.smithi192.stdout:Downloading Packages: 2024-06-03T08:34:04.530 INFO:teuthology.orchestra.run.smithi088.stdout: Preparing : 1/1 2024-06-03T08:34:04.632 INFO:teuthology.orchestra.run.smithi088.stdout: Installing : python3-rbd-debuginfo-2:18.2.2-1389.gf1abf795.el9.x8 1/1 2024-06-03T08:34:04.948 INFO:teuthology.orchestra.run.smithi192.stdout:python3-cephfs-debuginfo-18.2.2-1389.gf1abf795. 1.0 MB/s | 615 kB 00:00 2024-06-03T08:34:04.949 INFO:teuthology.orchestra.run.smithi192.stdout:-------------------------------------------------------------------------------- 2024-06-03T08:34:04.949 INFO:teuthology.orchestra.run.smithi192.stdout:Total 1.0 MB/s | 615 kB 00:00 2024-06-03T08:34:04.949 INFO:teuthology.orchestra.run.smithi192.stdout:Running transaction check 2024-06-03T08:34:04.952 INFO:teuthology.orchestra.run.smithi192.stdout:Transaction check succeeded. 2024-06-03T08:34:04.952 INFO:teuthology.orchestra.run.smithi192.stdout:Running transaction test 2024-06-03T08:34:04.981 INFO:teuthology.orchestra.run.smithi088.stdout: Running scriptlet: python3-rbd-debuginfo-2:18.2.2-1389.gf1abf795.el9.x8 1/1 2024-06-03T08:34:04.989 INFO:teuthology.orchestra.run.smithi192.stdout:Transaction test succeeded. 2024-06-03T08:34:04.989 INFO:teuthology.orchestra.run.smithi192.stdout:Running transaction 2024-06-03T08:34:05.153 INFO:teuthology.orchestra.run.smithi192.stdout: Preparing : 1/1 2024-06-03T08:34:05.199 INFO:teuthology.orchestra.run.smithi088.stdout: Verifying : python3-rbd-debuginfo-2:18.2.2-1389.gf1abf795.el9.x8 1/1 2024-06-03T08:34:05.199 INFO:teuthology.orchestra.run.smithi088.stdout: 2024-06-03T08:34:05.199 INFO:teuthology.orchestra.run.smithi088.stdout:Installed: 2024-06-03T08:34:05.199 INFO:teuthology.orchestra.run.smithi088.stdout: python3-rbd-debuginfo-2:18.2.2-1389.gf1abf795.el9.x86_64 2024-06-03T08:34:05.199 INFO:teuthology.orchestra.run.smithi088.stdout: 2024-06-03T08:34:05.199 INFO:teuthology.orchestra.run.smithi088.stdout:Complete! 2024-06-03T08:34:05.247 INFO:teuthology.orchestra.run.smithi192.stdout: Installing : python3-cephfs-debuginfo-2:18.2.2-1389.gf1abf795.el9 1/1 2024-06-03T08:34:05.384 DEBUG:teuthology.orchestra.run.smithi088:> sudo yum -y install python3-rgw-debuginfo 2024-06-03T08:34:05.633 INFO:teuthology.orchestra.run.smithi192.stdout: Running scriptlet: python3-cephfs-debuginfo-2:18.2.2-1389.gf1abf795.el9 1/1 2024-06-03T08:34:05.857 INFO:teuthology.orchestra.run.smithi192.stdout: Verifying : python3-cephfs-debuginfo-2:18.2.2-1389.gf1abf795.el9 1/1 2024-06-03T08:34:05.857 INFO:teuthology.orchestra.run.smithi192.stdout: 2024-06-03T08:34:05.857 INFO:teuthology.orchestra.run.smithi192.stdout:Installed: 2024-06-03T08:34:05.858 INFO:teuthology.orchestra.run.smithi192.stdout: python3-cephfs-debuginfo-2:18.2.2-1389.gf1abf795.el9.x86_64 2024-06-03T08:34:05.858 INFO:teuthology.orchestra.run.smithi192.stdout: 2024-06-03T08:34:05.858 INFO:teuthology.orchestra.run.smithi192.stdout:Complete! 2024-06-03T08:34:05.959 INFO:teuthology.orchestra.run.smithi088.stdout:Last metadata expiration check: 0:08:09 ago on Mon 03 Jun 2024 08:25:56 AM UTC. 2024-06-03T08:34:06.036 DEBUG:teuthology.orchestra.run.smithi192:> sudo yum -y install python3-rados-debuginfo 2024-06-03T08:34:06.082 INFO:teuthology.orchestra.run.smithi088.stdout:Dependencies resolved. 2024-06-03T08:34:06.083 INFO:teuthology.orchestra.run.smithi088.stdout:================================================================================ 2024-06-03T08:34:06.083 INFO:teuthology.orchestra.run.smithi088.stdout: Package Arch Version Repo Size 2024-06-03T08:34:06.083 INFO:teuthology.orchestra.run.smithi088.stdout:================================================================================ 2024-06-03T08:34:06.083 INFO:teuthology.orchestra.run.smithi088.stdout:Installing: 2024-06-03T08:34:06.083 INFO:teuthology.orchestra.run.smithi088.stdout: python3-rgw-debuginfo x86_64 2:18.2.2-1389.gf1abf795.el9 ceph 337 k 2024-06-03T08:34:06.083 INFO:teuthology.orchestra.run.smithi088.stdout: 2024-06-03T08:34:06.083 INFO:teuthology.orchestra.run.smithi088.stdout:Transaction Summary 2024-06-03T08:34:06.083 INFO:teuthology.orchestra.run.smithi088.stdout:================================================================================ 2024-06-03T08:34:06.083 INFO:teuthology.orchestra.run.smithi088.stdout:Install 1 Package 2024-06-03T08:34:06.084 INFO:teuthology.orchestra.run.smithi088.stdout: 2024-06-03T08:34:06.084 INFO:teuthology.orchestra.run.smithi088.stdout:Total download size: 337 k 2024-06-03T08:34:06.084 INFO:teuthology.orchestra.run.smithi088.stdout:Installed size: 1.0 M 2024-06-03T08:34:06.084 INFO:teuthology.orchestra.run.smithi088.stdout:Downloading Packages: 2024-06-03T08:34:06.363 INFO:teuthology.orchestra.run.smithi088.stdout:python3-rgw-debuginfo-18.2.2-1389.gf1abf795.el9 1.2 MB/s | 337 kB 00:00 2024-06-03T08:34:06.363 INFO:teuthology.orchestra.run.smithi088.stdout:-------------------------------------------------------------------------------- 2024-06-03T08:34:06.363 INFO:teuthology.orchestra.run.smithi088.stdout:Total 1.2 MB/s | 337 kB 00:00 2024-06-03T08:34:06.363 INFO:teuthology.orchestra.run.smithi088.stdout:Running transaction check 2024-06-03T08:34:06.367 INFO:teuthology.orchestra.run.smithi088.stdout:Transaction check succeeded. 2024-06-03T08:34:06.367 INFO:teuthology.orchestra.run.smithi088.stdout:Running transaction test 2024-06-03T08:34:06.403 INFO:teuthology.orchestra.run.smithi088.stdout:Transaction test succeeded. 2024-06-03T08:34:06.404 INFO:teuthology.orchestra.run.smithi088.stdout:Running transaction 2024-06-03T08:34:06.542 INFO:teuthology.orchestra.run.smithi088.stdout: Preparing : 1/1 2024-06-03T08:34:06.612 INFO:teuthology.orchestra.run.smithi192.stdout:Last metadata expiration check: 0:08:06 ago on Mon 03 Jun 2024 08:26:00 AM UTC. 2024-06-03T08:34:06.652 INFO:teuthology.orchestra.run.smithi088.stdout: Installing : python3-rgw-debuginfo-2:18.2.2-1389.gf1abf795.el9.x8 1/1 2024-06-03T08:34:06.737 INFO:teuthology.orchestra.run.smithi192.stdout:Dependencies resolved. 2024-06-03T08:34:06.738 INFO:teuthology.orchestra.run.smithi192.stdout:================================================================================ 2024-06-03T08:34:06.738 INFO:teuthology.orchestra.run.smithi192.stdout: Package Arch Version Repo Size 2024-06-03T08:34:06.738 INFO:teuthology.orchestra.run.smithi192.stdout:================================================================================ 2024-06-03T08:34:06.738 INFO:teuthology.orchestra.run.smithi192.stdout:Installing: 2024-06-03T08:34:06.738 INFO:teuthology.orchestra.run.smithi192.stdout: python3-rados-debuginfo x86_64 2:18.2.2-1389.gf1abf795.el9 ceph 770 k 2024-06-03T08:34:06.739 INFO:teuthology.orchestra.run.smithi192.stdout: 2024-06-03T08:34:06.739 INFO:teuthology.orchestra.run.smithi192.stdout:Transaction Summary 2024-06-03T08:34:06.739 INFO:teuthology.orchestra.run.smithi192.stdout:================================================================================ 2024-06-03T08:34:06.739 INFO:teuthology.orchestra.run.smithi192.stdout:Install 1 Package 2024-06-03T08:34:06.739 INFO:teuthology.orchestra.run.smithi192.stdout: 2024-06-03T08:34:06.739 INFO:teuthology.orchestra.run.smithi192.stdout:Total download size: 770 k 2024-06-03T08:34:06.739 INFO:teuthology.orchestra.run.smithi192.stdout:Installed size: 2.7 M 2024-06-03T08:34:06.739 INFO:teuthology.orchestra.run.smithi192.stdout:Downloading Packages: 2024-06-03T08:34:06.971 INFO:teuthology.orchestra.run.smithi088.stdout: Running scriptlet: python3-rgw-debuginfo-2:18.2.2-1389.gf1abf795.el9.x8 1/1 2024-06-03T08:34:07.043 INFO:teuthology.orchestra.run.smithi192.stdout:python3-rados-debuginfo-18.2.2-1389.gf1abf795.e 2.5 MB/s | 770 kB 00:00 2024-06-03T08:34:07.043 INFO:teuthology.orchestra.run.smithi192.stdout:-------------------------------------------------------------------------------- 2024-06-03T08:34:07.044 INFO:teuthology.orchestra.run.smithi192.stdout:Total 2.5 MB/s | 770 kB 00:00 2024-06-03T08:34:07.044 INFO:teuthology.orchestra.run.smithi192.stdout:Running transaction check 2024-06-03T08:34:07.047 INFO:teuthology.orchestra.run.smithi192.stdout:Transaction check succeeded. 2024-06-03T08:34:07.047 INFO:teuthology.orchestra.run.smithi192.stdout:Running transaction test 2024-06-03T08:34:07.085 INFO:teuthology.orchestra.run.smithi192.stdout:Transaction test succeeded. 2024-06-03T08:34:07.086 INFO:teuthology.orchestra.run.smithi192.stdout:Running transaction 2024-06-03T08:34:07.186 INFO:teuthology.orchestra.run.smithi088.stdout: Verifying : python3-rgw-debuginfo-2:18.2.2-1389.gf1abf795.el9.x8 1/1 2024-06-03T08:34:07.186 INFO:teuthology.orchestra.run.smithi088.stdout: 2024-06-03T08:34:07.186 INFO:teuthology.orchestra.run.smithi088.stdout:Installed: 2024-06-03T08:34:07.186 INFO:teuthology.orchestra.run.smithi088.stdout: python3-rgw-debuginfo-2:18.2.2-1389.gf1abf795.el9.x86_64 2024-06-03T08:34:07.186 INFO:teuthology.orchestra.run.smithi088.stdout: 2024-06-03T08:34:07.187 INFO:teuthology.orchestra.run.smithi088.stdout:Complete! 2024-06-03T08:34:07.267 INFO:teuthology.orchestra.run.smithi192.stdout: Preparing : 1/1 2024-06-03T08:34:07.354 DEBUG:teuthology.orchestra.run.smithi088:> sudo yum -y install rbd-fuse-debuginfo 2024-06-03T08:34:07.357 INFO:teuthology.orchestra.run.smithi192.stdout: Installing : python3-rados-debuginfo-2:18.2.2-1389.gf1abf795.el9. 1/1 2024-06-03T08:34:07.708 INFO:teuthology.orchestra.run.smithi192.stdout: Running scriptlet: python3-rados-debuginfo-2:18.2.2-1389.gf1abf795.el9. 1/1 2024-06-03T08:34:07.941 INFO:teuthology.orchestra.run.smithi088.stdout:Last metadata expiration check: 0:08:11 ago on Mon 03 Jun 2024 08:25:56 AM UTC. 2024-06-03T08:34:07.955 INFO:teuthology.orchestra.run.smithi192.stdout: Verifying : python3-rados-debuginfo-2:18.2.2-1389.gf1abf795.el9. 1/1 2024-06-03T08:34:07.955 INFO:teuthology.orchestra.run.smithi192.stdout: 2024-06-03T08:34:07.955 INFO:teuthology.orchestra.run.smithi192.stdout:Installed: 2024-06-03T08:34:07.956 INFO:teuthology.orchestra.run.smithi192.stdout: python3-rados-debuginfo-2:18.2.2-1389.gf1abf795.el9.x86_64 2024-06-03T08:34:07.956 INFO:teuthology.orchestra.run.smithi192.stdout: 2024-06-03T08:34:07.956 INFO:teuthology.orchestra.run.smithi192.stdout:Complete! 2024-06-03T08:34:08.063 INFO:teuthology.orchestra.run.smithi088.stdout:Dependencies resolved. 2024-06-03T08:34:08.064 INFO:teuthology.orchestra.run.smithi088.stdout:================================================================================ 2024-06-03T08:34:08.064 INFO:teuthology.orchestra.run.smithi088.stdout: Package Arch Version Repo Size 2024-06-03T08:34:08.064 INFO:teuthology.orchestra.run.smithi088.stdout:================================================================================ 2024-06-03T08:34:08.064 INFO:teuthology.orchestra.run.smithi088.stdout:Installing: 2024-06-03T08:34:08.064 INFO:teuthology.orchestra.run.smithi088.stdout: rbd-fuse-debuginfo x86_64 2:18.2.2-1389.gf1abf795.el9 ceph 511 k 2024-06-03T08:34:08.064 INFO:teuthology.orchestra.run.smithi088.stdout: 2024-06-03T08:34:08.064 INFO:teuthology.orchestra.run.smithi088.stdout:Transaction Summary 2024-06-03T08:34:08.064 INFO:teuthology.orchestra.run.smithi088.stdout:================================================================================ 2024-06-03T08:34:08.065 INFO:teuthology.orchestra.run.smithi088.stdout:Install 1 Package 2024-06-03T08:34:08.065 INFO:teuthology.orchestra.run.smithi088.stdout: 2024-06-03T08:34:08.065 INFO:teuthology.orchestra.run.smithi088.stdout:Total download size: 511 k 2024-06-03T08:34:08.065 INFO:teuthology.orchestra.run.smithi088.stdout:Installed size: 2.5 M 2024-06-03T08:34:08.065 INFO:teuthology.orchestra.run.smithi088.stdout:Downloading Packages: 2024-06-03T08:34:08.114 DEBUG:teuthology.orchestra.run.smithi192:> sudo yum -y install python3-rbd-debuginfo 2024-06-03T08:34:08.324 INFO:teuthology.orchestra.run.smithi088.stdout:rbd-fuse-debuginfo-18.2.2-1389.gf1abf795.el9.x8 1.9 MB/s | 511 kB 00:00 2024-06-03T08:34:08.324 INFO:teuthology.orchestra.run.smithi088.stdout:-------------------------------------------------------------------------------- 2024-06-03T08:34:08.325 INFO:teuthology.orchestra.run.smithi088.stdout:Total 1.9 MB/s | 511 kB 00:00 2024-06-03T08:34:08.325 INFO:teuthology.orchestra.run.smithi088.stdout:Running transaction check 2024-06-03T08:34:08.328 INFO:teuthology.orchestra.run.smithi088.stdout:Transaction check succeeded. 2024-06-03T08:34:08.328 INFO:teuthology.orchestra.run.smithi088.stdout:Running transaction test 2024-06-03T08:34:08.365 INFO:teuthology.orchestra.run.smithi088.stdout:Transaction test succeeded. 2024-06-03T08:34:08.366 INFO:teuthology.orchestra.run.smithi088.stdout:Running transaction 2024-06-03T08:34:08.523 INFO:teuthology.orchestra.run.smithi088.stdout: Preparing : 1/1 2024-06-03T08:34:08.621 INFO:teuthology.orchestra.run.smithi088.stdout: Installing : rbd-fuse-debuginfo-2:18.2.2-1389.gf1abf795.el9.x86_6 1/1 2024-06-03T08:34:08.688 INFO:teuthology.orchestra.run.smithi192.stdout:Last metadata expiration check: 0:08:08 ago on Mon 03 Jun 2024 08:26:00 AM UTC. 2024-06-03T08:34:08.812 INFO:teuthology.orchestra.run.smithi192.stdout:Dependencies resolved. 2024-06-03T08:34:08.812 INFO:teuthology.orchestra.run.smithi192.stdout:================================================================================ 2024-06-03T08:34:08.812 INFO:teuthology.orchestra.run.smithi192.stdout: Package Arch Version Repo Size 2024-06-03T08:34:08.812 INFO:teuthology.orchestra.run.smithi192.stdout:================================================================================ 2024-06-03T08:34:08.812 INFO:teuthology.orchestra.run.smithi192.stdout:Installing: 2024-06-03T08:34:08.812 INFO:teuthology.orchestra.run.smithi192.stdout: python3-rbd-debuginfo x86_64 2:18.2.2-1389.gf1abf795.el9 ceph 1.1 M 2024-06-03T08:34:08.813 INFO:teuthology.orchestra.run.smithi192.stdout: 2024-06-03T08:34:08.813 INFO:teuthology.orchestra.run.smithi192.stdout:Transaction Summary 2024-06-03T08:34:08.813 INFO:teuthology.orchestra.run.smithi192.stdout:================================================================================ 2024-06-03T08:34:08.813 INFO:teuthology.orchestra.run.smithi192.stdout:Install 1 Package 2024-06-03T08:34:08.813 INFO:teuthology.orchestra.run.smithi192.stdout: 2024-06-03T08:34:08.813 INFO:teuthology.orchestra.run.smithi192.stdout:Total download size: 1.1 M 2024-06-03T08:34:08.813 INFO:teuthology.orchestra.run.smithi192.stdout:Installed size: 4.3 M 2024-06-03T08:34:08.813 INFO:teuthology.orchestra.run.smithi192.stdout:Downloading Packages: 2024-06-03T08:34:08.940 INFO:teuthology.orchestra.run.smithi088.stdout: Running scriptlet: rbd-fuse-debuginfo-2:18.2.2-1389.gf1abf795.el9.x86_6 1/1 2024-06-03T08:34:09.121 INFO:teuthology.orchestra.run.smithi192.stdout:python3-rbd-debuginfo-18.2.2-1389.gf1abf795.el9 3.6 MB/s | 1.1 MB 00:00 2024-06-03T08:34:09.122 INFO:teuthology.orchestra.run.smithi192.stdout:-------------------------------------------------------------------------------- 2024-06-03T08:34:09.122 INFO:teuthology.orchestra.run.smithi192.stdout:Total 3.6 MB/s | 1.1 MB 00:00 2024-06-03T08:34:09.122 INFO:teuthology.orchestra.run.smithi192.stdout:Running transaction check 2024-06-03T08:34:09.125 INFO:teuthology.orchestra.run.smithi192.stdout:Transaction check succeeded. 2024-06-03T08:34:09.125 INFO:teuthology.orchestra.run.smithi192.stdout:Running transaction test 2024-06-03T08:34:09.152 INFO:teuthology.orchestra.run.smithi088.stdout: Verifying : rbd-fuse-debuginfo-2:18.2.2-1389.gf1abf795.el9.x86_6 1/1 2024-06-03T08:34:09.152 INFO:teuthology.orchestra.run.smithi088.stdout: 2024-06-03T08:34:09.152 INFO:teuthology.orchestra.run.smithi088.stdout:Installed: 2024-06-03T08:34:09.152 INFO:teuthology.orchestra.run.smithi088.stdout: rbd-fuse-debuginfo-2:18.2.2-1389.gf1abf795.el9.x86_64 2024-06-03T08:34:09.152 INFO:teuthology.orchestra.run.smithi088.stdout: 2024-06-03T08:34:09.152 INFO:teuthology.orchestra.run.smithi088.stdout:Complete! 2024-06-03T08:34:09.165 INFO:teuthology.orchestra.run.smithi192.stdout:Transaction test succeeded. 2024-06-03T08:34:09.166 INFO:teuthology.orchestra.run.smithi192.stdout:Running transaction 2024-06-03T08:34:09.296 DEBUG:teuthology.orchestra.run.smithi088:> sudo yum -y install rbd-mirror-debuginfo 2024-06-03T08:34:09.387 INFO:teuthology.orchestra.run.smithi192.stdout: Preparing : 1/1 2024-06-03T08:34:09.477 INFO:teuthology.orchestra.run.smithi192.stdout: Installing : python3-rbd-debuginfo-2:18.2.2-1389.gf1abf795.el9.x8 1/1 2024-06-03T08:34:09.840 INFO:teuthology.orchestra.run.smithi192.stdout: Running scriptlet: python3-rbd-debuginfo-2:18.2.2-1389.gf1abf795.el9.x8 1/1 2024-06-03T08:34:09.872 INFO:teuthology.orchestra.run.smithi088.stdout:Last metadata expiration check: 0:08:13 ago on Mon 03 Jun 2024 08:25:56 AM UTC. 2024-06-03T08:34:09.995 INFO:teuthology.orchestra.run.smithi088.stdout:Dependencies resolved. 2024-06-03T08:34:09.996 INFO:teuthology.orchestra.run.smithi088.stdout:================================================================================ 2024-06-03T08:34:09.996 INFO:teuthology.orchestra.run.smithi088.stdout: Package Arch Version Repo Size 2024-06-03T08:34:09.996 INFO:teuthology.orchestra.run.smithi088.stdout:================================================================================ 2024-06-03T08:34:09.996 INFO:teuthology.orchestra.run.smithi088.stdout:Installing: 2024-06-03T08:34:09.996 INFO:teuthology.orchestra.run.smithi088.stdout: rbd-mirror-debuginfo x86_64 2:18.2.2-1389.gf1abf795.el9 ceph 50 M 2024-06-03T08:34:09.996 INFO:teuthology.orchestra.run.smithi088.stdout: 2024-06-03T08:34:09.996 INFO:teuthology.orchestra.run.smithi088.stdout:Transaction Summary 2024-06-03T08:34:09.996 INFO:teuthology.orchestra.run.smithi088.stdout:================================================================================ 2024-06-03T08:34:09.996 INFO:teuthology.orchestra.run.smithi088.stdout:Install 1 Package 2024-06-03T08:34:09.996 INFO:teuthology.orchestra.run.smithi088.stdout: 2024-06-03T08:34:09.996 INFO:teuthology.orchestra.run.smithi088.stdout:Total download size: 50 M 2024-06-03T08:34:09.997 INFO:teuthology.orchestra.run.smithi088.stdout:Installed size: 281 M 2024-06-03T08:34:09.997 INFO:teuthology.orchestra.run.smithi088.stdout:Downloading Packages: 2024-06-03T08:34:10.063 INFO:teuthology.orchestra.run.smithi192.stdout: Verifying : python3-rbd-debuginfo-2:18.2.2-1389.gf1abf795.el9.x8 1/1 2024-06-03T08:34:10.063 INFO:teuthology.orchestra.run.smithi192.stdout: 2024-06-03T08:34:10.064 INFO:teuthology.orchestra.run.smithi192.stdout:Installed: 2024-06-03T08:34:10.064 INFO:teuthology.orchestra.run.smithi192.stdout: python3-rbd-debuginfo-2:18.2.2-1389.gf1abf795.el9.x86_64 2024-06-03T08:34:10.064 INFO:teuthology.orchestra.run.smithi192.stdout: 2024-06-03T08:34:10.064 INFO:teuthology.orchestra.run.smithi192.stdout:Complete! 2024-06-03T08:34:10.235 DEBUG:teuthology.orchestra.run.smithi192:> sudo yum -y install python3-rgw-debuginfo 2024-06-03T08:34:10.814 INFO:teuthology.orchestra.run.smithi192.stdout:Last metadata expiration check: 0:08:10 ago on Mon 03 Jun 2024 08:26:00 AM UTC. 2024-06-03T08:34:10.938 INFO:teuthology.orchestra.run.smithi192.stdout:Dependencies resolved. 2024-06-03T08:34:10.939 INFO:teuthology.orchestra.run.smithi192.stdout:================================================================================ 2024-06-03T08:34:10.939 INFO:teuthology.orchestra.run.smithi192.stdout: Package Arch Version Repo Size 2024-06-03T08:34:10.939 INFO:teuthology.orchestra.run.smithi192.stdout:================================================================================ 2024-06-03T08:34:10.939 INFO:teuthology.orchestra.run.smithi192.stdout:Installing: 2024-06-03T08:34:10.939 INFO:teuthology.orchestra.run.smithi192.stdout: python3-rgw-debuginfo x86_64 2:18.2.2-1389.gf1abf795.el9 ceph 337 k 2024-06-03T08:34:10.940 INFO:teuthology.orchestra.run.smithi192.stdout: 2024-06-03T08:34:10.940 INFO:teuthology.orchestra.run.smithi192.stdout:Transaction Summary 2024-06-03T08:34:10.940 INFO:teuthology.orchestra.run.smithi192.stdout:================================================================================ 2024-06-03T08:34:10.940 INFO:teuthology.orchestra.run.smithi192.stdout:Install 1 Package 2024-06-03T08:34:10.940 INFO:teuthology.orchestra.run.smithi192.stdout: 2024-06-03T08:34:10.940 INFO:teuthology.orchestra.run.smithi192.stdout:Total download size: 337 k 2024-06-03T08:34:10.940 INFO:teuthology.orchestra.run.smithi192.stdout:Installed size: 1.0 M 2024-06-03T08:34:10.940 INFO:teuthology.orchestra.run.smithi192.stdout:Downloading Packages: 2024-06-03T08:34:11.608 INFO:teuthology.orchestra.run.smithi192.stdout:python3-rgw-debuginfo-18.2.2-1389.gf1abf795.el9 505 kB/s | 337 kB 00:00 2024-06-03T08:34:11.608 INFO:teuthology.orchestra.run.smithi192.stdout:-------------------------------------------------------------------------------- 2024-06-03T08:34:11.608 INFO:teuthology.orchestra.run.smithi192.stdout:Total 504 kB/s | 337 kB 00:00 2024-06-03T08:34:11.609 INFO:teuthology.orchestra.run.smithi192.stdout:Running transaction check 2024-06-03T08:34:11.612 INFO:teuthology.orchestra.run.smithi192.stdout:Transaction check succeeded. 2024-06-03T08:34:11.612 INFO:teuthology.orchestra.run.smithi192.stdout:Running transaction test 2024-06-03T08:34:11.650 INFO:teuthology.orchestra.run.smithi192.stdout:Transaction test succeeded. 2024-06-03T08:34:11.650 INFO:teuthology.orchestra.run.smithi192.stdout:Running transaction 2024-06-03T08:34:11.791 INFO:teuthology.orchestra.run.smithi192.stdout: Preparing : 1/1 2024-06-03T08:34:11.889 INFO:teuthology.orchestra.run.smithi192.stdout: Installing : python3-rgw-debuginfo-2:18.2.2-1389.gf1abf795.el9.x8 1/1 2024-06-03T08:34:12.241 INFO:teuthology.orchestra.run.smithi192.stdout: Running scriptlet: python3-rgw-debuginfo-2:18.2.2-1389.gf1abf795.el9.x8 1/1 2024-06-03T08:34:12.471 INFO:teuthology.orchestra.run.smithi192.stdout: Verifying : python3-rgw-debuginfo-2:18.2.2-1389.gf1abf795.el9.x8 1/1 2024-06-03T08:34:12.472 INFO:teuthology.orchestra.run.smithi192.stdout: 2024-06-03T08:34:12.472 INFO:teuthology.orchestra.run.smithi192.stdout:Installed: 2024-06-03T08:34:12.472 INFO:teuthology.orchestra.run.smithi192.stdout: python3-rgw-debuginfo-2:18.2.2-1389.gf1abf795.el9.x86_64 2024-06-03T08:34:12.472 INFO:teuthology.orchestra.run.smithi192.stdout: 2024-06-03T08:34:12.472 INFO:teuthology.orchestra.run.smithi192.stdout:Complete! 2024-06-03T08:34:12.610 INFO:teuthology.orchestra.run.smithi088.stdout:rbd-mirror-debuginfo-18.2.2-1389.gf1abf795.el9. 19 MB/s | 50 MB 00:02 2024-06-03T08:34:12.610 INFO:teuthology.orchestra.run.smithi088.stdout:-------------------------------------------------------------------------------- 2024-06-03T08:34:12.611 INFO:teuthology.orchestra.run.smithi088.stdout:Total 19 MB/s | 50 MB 00:02 2024-06-03T08:34:12.611 INFO:teuthology.orchestra.run.smithi088.stdout:Running transaction check 2024-06-03T08:34:12.615 INFO:teuthology.orchestra.run.smithi088.stdout:Transaction check succeeded. 2024-06-03T08:34:12.615 INFO:teuthology.orchestra.run.smithi088.stdout:Running transaction test 2024-06-03T08:34:12.671 DEBUG:teuthology.orchestra.run.smithi192:> sudo yum -y install rbd-fuse-debuginfo 2024-06-03T08:34:12.946 INFO:teuthology.orchestra.run.smithi088.stdout:Transaction test succeeded. 2024-06-03T08:34:12.947 INFO:teuthology.orchestra.run.smithi088.stdout:Running transaction 2024-06-03T08:34:13.246 INFO:teuthology.orchestra.run.smithi192.stdout:Last metadata expiration check: 0:08:13 ago on Mon 03 Jun 2024 08:26:00 AM UTC. 2024-06-03T08:34:13.370 INFO:teuthology.orchestra.run.smithi192.stdout:Dependencies resolved. 2024-06-03T08:34:13.371 INFO:teuthology.orchestra.run.smithi192.stdout:================================================================================ 2024-06-03T08:34:13.371 INFO:teuthology.orchestra.run.smithi192.stdout: Package Arch Version Repo Size 2024-06-03T08:34:13.371 INFO:teuthology.orchestra.run.smithi192.stdout:================================================================================ 2024-06-03T08:34:13.371 INFO:teuthology.orchestra.run.smithi192.stdout:Installing: 2024-06-03T08:34:13.371 INFO:teuthology.orchestra.run.smithi192.stdout: rbd-fuse-debuginfo x86_64 2:18.2.2-1389.gf1abf795.el9 ceph 511 k 2024-06-03T08:34:13.371 INFO:teuthology.orchestra.run.smithi192.stdout: 2024-06-03T08:34:13.371 INFO:teuthology.orchestra.run.smithi192.stdout:Transaction Summary 2024-06-03T08:34:13.371 INFO:teuthology.orchestra.run.smithi192.stdout:================================================================================ 2024-06-03T08:34:13.372 INFO:teuthology.orchestra.run.smithi192.stdout:Install 1 Package 2024-06-03T08:34:13.372 INFO:teuthology.orchestra.run.smithi192.stdout: 2024-06-03T08:34:13.372 INFO:teuthology.orchestra.run.smithi192.stdout:Total download size: 511 k 2024-06-03T08:34:13.372 INFO:teuthology.orchestra.run.smithi192.stdout:Installed size: 2.5 M 2024-06-03T08:34:13.372 INFO:teuthology.orchestra.run.smithi192.stdout:Downloading Packages: 2024-06-03T08:34:13.661 INFO:teuthology.orchestra.run.smithi192.stdout:rbd-fuse-debuginfo-18.2.2-1389.gf1abf795.el9.x8 1.7 MB/s | 511 kB 00:00 2024-06-03T08:34:13.662 INFO:teuthology.orchestra.run.smithi192.stdout:-------------------------------------------------------------------------------- 2024-06-03T08:34:13.662 INFO:teuthology.orchestra.run.smithi192.stdout:Total 1.7 MB/s | 511 kB 00:00 2024-06-03T08:34:13.662 INFO:teuthology.orchestra.run.smithi192.stdout:Running transaction check 2024-06-03T08:34:13.665 INFO:teuthology.orchestra.run.smithi192.stdout:Transaction check succeeded. 2024-06-03T08:34:13.665 INFO:teuthology.orchestra.run.smithi192.stdout:Running transaction test 2024-06-03T08:34:13.703 INFO:teuthology.orchestra.run.smithi192.stdout:Transaction test succeeded. 2024-06-03T08:34:13.704 INFO:teuthology.orchestra.run.smithi192.stdout:Running transaction 2024-06-03T08:34:13.862 INFO:teuthology.orchestra.run.smithi192.stdout: Preparing : 1/1 2024-06-03T08:34:13.951 INFO:teuthology.orchestra.run.smithi192.stdout: Installing : rbd-fuse-debuginfo-2:18.2.2-1389.gf1abf795.el9.x86_6 1/1 2024-06-03T08:34:14.313 INFO:teuthology.orchestra.run.smithi192.stdout: Running scriptlet: rbd-fuse-debuginfo-2:18.2.2-1389.gf1abf795.el9.x86_6 1/1 2024-06-03T08:34:14.530 INFO:teuthology.orchestra.run.smithi192.stdout: Verifying : rbd-fuse-debuginfo-2:18.2.2-1389.gf1abf795.el9.x86_6 1/1 2024-06-03T08:34:14.533 INFO:teuthology.orchestra.run.smithi192.stdout: 2024-06-03T08:34:14.533 INFO:teuthology.orchestra.run.smithi192.stdout:Installed: 2024-06-03T08:34:14.533 INFO:teuthology.orchestra.run.smithi192.stdout: rbd-fuse-debuginfo-2:18.2.2-1389.gf1abf795.el9.x86_64 2024-06-03T08:34:14.533 INFO:teuthology.orchestra.run.smithi192.stdout: 2024-06-03T08:34:14.533 INFO:teuthology.orchestra.run.smithi192.stdout:Complete! 2024-06-03T08:34:14.708 DEBUG:teuthology.orchestra.run.smithi192:> sudo yum -y install rbd-mirror-debuginfo 2024-06-03T08:34:15.283 INFO:teuthology.orchestra.run.smithi192.stdout:Last metadata expiration check: 0:08:15 ago on Mon 03 Jun 2024 08:26:00 AM UTC. 2024-06-03T08:34:15.408 INFO:teuthology.orchestra.run.smithi192.stdout:Dependencies resolved. 2024-06-03T08:34:15.408 INFO:teuthology.orchestra.run.smithi192.stdout:================================================================================ 2024-06-03T08:34:15.408 INFO:teuthology.orchestra.run.smithi192.stdout: Package Arch Version Repo Size 2024-06-03T08:34:15.409 INFO:teuthology.orchestra.run.smithi192.stdout:================================================================================ 2024-06-03T08:34:15.409 INFO:teuthology.orchestra.run.smithi192.stdout:Installing: 2024-06-03T08:34:15.409 INFO:teuthology.orchestra.run.smithi192.stdout: rbd-mirror-debuginfo x86_64 2:18.2.2-1389.gf1abf795.el9 ceph 50 M 2024-06-03T08:34:15.409 INFO:teuthology.orchestra.run.smithi192.stdout: 2024-06-03T08:34:15.409 INFO:teuthology.orchestra.run.smithi192.stdout:Transaction Summary 2024-06-03T08:34:15.409 INFO:teuthology.orchestra.run.smithi192.stdout:================================================================================ 2024-06-03T08:34:15.409 INFO:teuthology.orchestra.run.smithi192.stdout:Install 1 Package 2024-06-03T08:34:15.409 INFO:teuthology.orchestra.run.smithi192.stdout: 2024-06-03T08:34:15.409 INFO:teuthology.orchestra.run.smithi192.stdout:Total download size: 50 M 2024-06-03T08:34:15.409 INFO:teuthology.orchestra.run.smithi192.stdout:Installed size: 281 M 2024-06-03T08:34:15.409 INFO:teuthology.orchestra.run.smithi192.stdout:Downloading Packages: 2024-06-03T08:34:17.914 INFO:teuthology.orchestra.run.smithi192.stdout:rbd-mirror-debuginfo-18.2.2-1389.gf1abf795.el9. 20 MB/s | 50 MB 00:02 2024-06-03T08:34:17.914 INFO:teuthology.orchestra.run.smithi192.stdout:-------------------------------------------------------------------------------- 2024-06-03T08:34:17.915 INFO:teuthology.orchestra.run.smithi192.stdout:Total 20 MB/s | 50 MB 00:02 2024-06-03T08:34:17.915 INFO:teuthology.orchestra.run.smithi192.stdout:Running transaction check 2024-06-03T08:34:17.919 INFO:teuthology.orchestra.run.smithi192.stdout:Transaction check succeeded. 2024-06-03T08:34:17.919 INFO:teuthology.orchestra.run.smithi192.stdout:Running transaction test 2024-06-03T08:34:18.249 INFO:teuthology.orchestra.run.smithi192.stdout:Transaction test succeeded. 2024-06-03T08:34:18.249 INFO:teuthology.orchestra.run.smithi192.stdout:Running transaction 2024-06-03T08:34:18.663 INFO:teuthology.orchestra.run.smithi088.stdout: Preparing : 1/1 2024-06-03T08:34:18.773 INFO:teuthology.orchestra.run.smithi088.stdout: Installing : rbd-mirror-debuginfo-2:18.2.2-1389.gf1abf795.el9.x86 1/1 2024-06-03T08:34:20.692 INFO:teuthology.orchestra.run.smithi088.stdout: Running scriptlet: rbd-mirror-debuginfo-2:18.2.2-1389.gf1abf795.el9.x86 1/1 2024-06-03T08:34:20.929 INFO:teuthology.orchestra.run.smithi088.stdout: Verifying : rbd-mirror-debuginfo-2:18.2.2-1389.gf1abf795.el9.x86 1/1 2024-06-03T08:34:20.929 INFO:teuthology.orchestra.run.smithi088.stdout: 2024-06-03T08:34:20.929 INFO:teuthology.orchestra.run.smithi088.stdout:Installed: 2024-06-03T08:34:20.929 INFO:teuthology.orchestra.run.smithi088.stdout: rbd-mirror-debuginfo-2:18.2.2-1389.gf1abf795.el9.x86_64 2024-06-03T08:34:20.929 INFO:teuthology.orchestra.run.smithi088.stdout: 2024-06-03T08:34:20.929 INFO:teuthology.orchestra.run.smithi088.stdout:Complete! 2024-06-03T08:34:21.070 DEBUG:teuthology.orchestra.run.smithi088:> sudo yum -y install rbd-nbd-debuginfo 2024-06-03T08:34:21.643 INFO:teuthology.orchestra.run.smithi088.stdout:Last metadata expiration check: 0:08:25 ago on Mon 03 Jun 2024 08:25:56 AM UTC. 2024-06-03T08:34:21.766 INFO:teuthology.orchestra.run.smithi088.stdout:Dependencies resolved. 2024-06-03T08:34:21.767 INFO:teuthology.orchestra.run.smithi088.stdout:================================================================================ 2024-06-03T08:34:21.767 INFO:teuthology.orchestra.run.smithi088.stdout: Package Arch Version Repo Size 2024-06-03T08:34:21.767 INFO:teuthology.orchestra.run.smithi088.stdout:================================================================================ 2024-06-03T08:34:21.767 INFO:teuthology.orchestra.run.smithi088.stdout:Installing: 2024-06-03T08:34:21.767 INFO:teuthology.orchestra.run.smithi088.stdout: rbd-nbd-debuginfo x86_64 2:18.2.2-1389.gf1abf795.el9 ceph 1.3 M 2024-06-03T08:34:21.767 INFO:teuthology.orchestra.run.smithi088.stdout: 2024-06-03T08:34:21.767 INFO:teuthology.orchestra.run.smithi088.stdout:Transaction Summary 2024-06-03T08:34:21.767 INFO:teuthology.orchestra.run.smithi088.stdout:================================================================================ 2024-06-03T08:34:21.767 INFO:teuthology.orchestra.run.smithi088.stdout:Install 1 Package 2024-06-03T08:34:21.768 INFO:teuthology.orchestra.run.smithi088.stdout: 2024-06-03T08:34:21.768 INFO:teuthology.orchestra.run.smithi088.stdout:Total download size: 1.3 M 2024-06-03T08:34:21.768 INFO:teuthology.orchestra.run.smithi088.stdout:Installed size: 5.5 M 2024-06-03T08:34:21.768 INFO:teuthology.orchestra.run.smithi088.stdout:Downloading Packages: 2024-06-03T08:34:22.661 INFO:teuthology.orchestra.run.smithi088.stdout:rbd-nbd-debuginfo-18.2.2-1389.gf1abf795.el9.x86 1.4 MB/s | 1.3 MB 00:00 2024-06-03T08:34:22.661 INFO:teuthology.orchestra.run.smithi088.stdout:-------------------------------------------------------------------------------- 2024-06-03T08:34:22.662 INFO:teuthology.orchestra.run.smithi088.stdout:Total 1.4 MB/s | 1.3 MB 00:00 2024-06-03T08:34:22.662 INFO:teuthology.orchestra.run.smithi088.stdout:Running transaction check 2024-06-03T08:34:22.666 INFO:teuthology.orchestra.run.smithi088.stdout:Transaction check succeeded. 2024-06-03T08:34:22.666 INFO:teuthology.orchestra.run.smithi088.stdout:Running transaction test 2024-06-03T08:34:22.709 INFO:teuthology.orchestra.run.smithi088.stdout:Transaction test succeeded. 2024-06-03T08:34:22.710 INFO:teuthology.orchestra.run.smithi088.stdout:Running transaction 2024-06-03T08:34:22.963 INFO:teuthology.orchestra.run.smithi088.stdout: Preparing : 1/1 2024-06-03T08:34:23.058 INFO:teuthology.orchestra.run.smithi088.stdout: Installing : rbd-nbd-debuginfo-2:18.2.2-1389.gf1abf795.el9.x86_64 1/1 2024-06-03T08:34:23.403 INFO:teuthology.orchestra.run.smithi088.stdout: Running scriptlet: rbd-nbd-debuginfo-2:18.2.2-1389.gf1abf795.el9.x86_64 1/1 2024-06-03T08:34:23.615 INFO:teuthology.orchestra.run.smithi088.stdout: Verifying : rbd-nbd-debuginfo-2:18.2.2-1389.gf1abf795.el9.x86_64 1/1 2024-06-03T08:34:23.615 INFO:teuthology.orchestra.run.smithi088.stdout: 2024-06-03T08:34:23.615 INFO:teuthology.orchestra.run.smithi088.stdout:Installed: 2024-06-03T08:34:23.616 INFO:teuthology.orchestra.run.smithi088.stdout: rbd-nbd-debuginfo-2:18.2.2-1389.gf1abf795.el9.x86_64 2024-06-03T08:34:23.616 INFO:teuthology.orchestra.run.smithi088.stdout: 2024-06-03T08:34:23.616 INFO:teuthology.orchestra.run.smithi088.stdout:Complete! 2024-06-03T08:34:23.768 DEBUG:teuthology.parallel:result is None 2024-06-03T08:34:23.944 INFO:teuthology.orchestra.run.smithi192.stdout: Preparing : 1/1 2024-06-03T08:34:24.044 INFO:teuthology.orchestra.run.smithi192.stdout: Installing : rbd-mirror-debuginfo-2:18.2.2-1389.gf1abf795.el9.x86 1/1 2024-06-03T08:34:25.994 INFO:teuthology.orchestra.run.smithi192.stdout: Running scriptlet: rbd-mirror-debuginfo-2:18.2.2-1389.gf1abf795.el9.x86 1/1 2024-06-03T08:34:26.218 INFO:teuthology.orchestra.run.smithi192.stdout: Verifying : rbd-mirror-debuginfo-2:18.2.2-1389.gf1abf795.el9.x86 1/1 2024-06-03T08:34:26.219 INFO:teuthology.orchestra.run.smithi192.stdout: 2024-06-03T08:34:26.219 INFO:teuthology.orchestra.run.smithi192.stdout:Installed: 2024-06-03T08:34:26.219 INFO:teuthology.orchestra.run.smithi192.stdout: rbd-mirror-debuginfo-2:18.2.2-1389.gf1abf795.el9.x86_64 2024-06-03T08:34:26.219 INFO:teuthology.orchestra.run.smithi192.stdout: 2024-06-03T08:34:26.219 INFO:teuthology.orchestra.run.smithi192.stdout:Complete! 2024-06-03T08:34:26.396 DEBUG:teuthology.orchestra.run.smithi192:> sudo yum -y install rbd-nbd-debuginfo 2024-06-03T08:34:26.972 INFO:teuthology.orchestra.run.smithi192.stdout:Last metadata expiration check: 0:08:26 ago on Mon 03 Jun 2024 08:26:00 AM UTC. 2024-06-03T08:34:27.096 INFO:teuthology.orchestra.run.smithi192.stdout:Dependencies resolved. 2024-06-03T08:34:27.097 INFO:teuthology.orchestra.run.smithi192.stdout:================================================================================ 2024-06-03T08:34:27.098 INFO:teuthology.orchestra.run.smithi192.stdout: Package Arch Version Repo Size 2024-06-03T08:34:27.098 INFO:teuthology.orchestra.run.smithi192.stdout:================================================================================ 2024-06-03T08:34:27.098 INFO:teuthology.orchestra.run.smithi192.stdout:Installing: 2024-06-03T08:34:27.098 INFO:teuthology.orchestra.run.smithi192.stdout: rbd-nbd-debuginfo x86_64 2:18.2.2-1389.gf1abf795.el9 ceph 1.3 M 2024-06-03T08:34:27.098 INFO:teuthology.orchestra.run.smithi192.stdout: 2024-06-03T08:34:27.098 INFO:teuthology.orchestra.run.smithi192.stdout:Transaction Summary 2024-06-03T08:34:27.098 INFO:teuthology.orchestra.run.smithi192.stdout:================================================================================ 2024-06-03T08:34:27.098 INFO:teuthology.orchestra.run.smithi192.stdout:Install 1 Package 2024-06-03T08:34:27.098 INFO:teuthology.orchestra.run.smithi192.stdout: 2024-06-03T08:34:27.098 INFO:teuthology.orchestra.run.smithi192.stdout:Total download size: 1.3 M 2024-06-03T08:34:27.098 INFO:teuthology.orchestra.run.smithi192.stdout:Installed size: 5.5 M 2024-06-03T08:34:27.098 INFO:teuthology.orchestra.run.smithi192.stdout:Downloading Packages: 2024-06-03T08:34:27.574 INFO:teuthology.orchestra.run.smithi192.stdout:rbd-nbd-debuginfo-18.2.2-1389.gf1abf795.el9.x86 2.6 MB/s | 1.3 MB 00:00 2024-06-03T08:34:27.575 INFO:teuthology.orchestra.run.smithi192.stdout:-------------------------------------------------------------------------------- 2024-06-03T08:34:27.575 INFO:teuthology.orchestra.run.smithi192.stdout:Total 2.6 MB/s | 1.3 MB 00:00 2024-06-03T08:34:27.575 INFO:teuthology.orchestra.run.smithi192.stdout:Running transaction check 2024-06-03T08:34:27.578 INFO:teuthology.orchestra.run.smithi192.stdout:Transaction check succeeded. 2024-06-03T08:34:27.578 INFO:teuthology.orchestra.run.smithi192.stdout:Running transaction test 2024-06-03T08:34:27.620 INFO:teuthology.orchestra.run.smithi192.stdout:Transaction test succeeded. 2024-06-03T08:34:27.620 INFO:teuthology.orchestra.run.smithi192.stdout:Running transaction 2024-06-03T08:34:27.854 INFO:teuthology.orchestra.run.smithi192.stdout: Preparing : 1/1 2024-06-03T08:34:27.980 INFO:teuthology.orchestra.run.smithi192.stdout: Installing : rbd-nbd-debuginfo-2:18.2.2-1389.gf1abf795.el9.x86_64 1/1 2024-06-03T08:34:28.376 INFO:teuthology.orchestra.run.smithi192.stdout: Running scriptlet: rbd-nbd-debuginfo-2:18.2.2-1389.gf1abf795.el9.x86_64 1/1 2024-06-03T08:34:28.603 INFO:teuthology.orchestra.run.smithi192.stdout: Verifying : rbd-nbd-debuginfo-2:18.2.2-1389.gf1abf795.el9.x86_64 1/1 2024-06-03T08:34:28.603 INFO:teuthology.orchestra.run.smithi192.stdout: 2024-06-03T08:34:28.603 INFO:teuthology.orchestra.run.smithi192.stdout:Installed: 2024-06-03T08:34:28.603 INFO:teuthology.orchestra.run.smithi192.stdout: rbd-nbd-debuginfo-2:18.2.2-1389.gf1abf795.el9.x86_64 2024-06-03T08:34:28.603 INFO:teuthology.orchestra.run.smithi192.stdout: 2024-06-03T08:34:28.603 INFO:teuthology.orchestra.run.smithi192.stdout:Complete! 2024-06-03T08:34:28.784 DEBUG:teuthology.parallel:result is None 2024-06-03T08:34:28.785 DEBUG:teuthology.packaging:Querying https://shaman.ceph.com/api/search?status=ready&project=ceph&flavor=default&distros=centos%2F9%2Fx86_64&sha1=f1abf795f4fc15bf2f30ac90c811a53ab21d5d65 2024-06-03T08:34:28.927 DEBUG:teuthology.orchestra.run.smithi088:> rpm -q ceph --qf '%{VERSION}-%{RELEASE}' 2024-06-03T08:34:28.952 INFO:teuthology.orchestra.run.smithi088.stdout:18.2.2-1389.gf1abf795.el9 2024-06-03T08:34:28.953 INFO:teuthology.packaging:The installed version of ceph is 18.2.2-1389.gf1abf795.el9 2024-06-03T08:34:28.953 INFO:teuthology.task.install:The correct ceph version 18.2.2-1389.gf1abf795 is installed. 2024-06-03T08:34:28.955 DEBUG:teuthology.packaging:Querying https://shaman.ceph.com/api/search?status=ready&project=ceph&flavor=default&distros=centos%2F9%2Fx86_64&sha1=f1abf795f4fc15bf2f30ac90c811a53ab21d5d65 2024-06-03T08:34:29.102 DEBUG:teuthology.orchestra.run.smithi192:> rpm -q ceph --qf '%{VERSION}-%{RELEASE}' 2024-06-03T08:34:29.123 INFO:teuthology.orchestra.run.smithi192.stdout:18.2.2-1389.gf1abf795.el9 2024-06-03T08:34:29.123 INFO:teuthology.packaging:The installed version of ceph is 18.2.2-1389.gf1abf795.el9 2024-06-03T08:34:29.124 INFO:teuthology.task.install:The correct ceph version 18.2.2-1389.gf1abf795 is installed. 2024-06-03T08:34:29.126 INFO:teuthology.task.install.util:Shipping valgrind.supp... 2024-06-03T08:34:29.126 DEBUG:teuthology.orchestra.run.smithi088:> set -ex 2024-06-03T08:34:29.126 DEBUG:teuthology.orchestra.run.smithi088:> sudo dd of=/home/ubuntu/cephtest/valgrind.supp 2024-06-03T08:34:29.156 DEBUG:teuthology.orchestra.run.smithi192:> set -ex 2024-06-03T08:34:29.156 DEBUG:teuthology.orchestra.run.smithi192:> sudo dd of=/home/ubuntu/cephtest/valgrind.supp 2024-06-03T08:34:29.189 INFO:teuthology.task.install.util:Shipping 'daemon-helper'... 2024-06-03T08:34:29.189 DEBUG:teuthology.orchestra.run.smithi088:> set -ex 2024-06-03T08:34:29.189 DEBUG:teuthology.orchestra.run.smithi088:> sudo dd of=/usr/bin/daemon-helper 2024-06-03T08:34:29.224 DEBUG:teuthology.orchestra.run.smithi088:> sudo chmod a=rx -- /usr/bin/daemon-helper 2024-06-03T08:34:29.292 DEBUG:teuthology.orchestra.run.smithi192:> set -ex 2024-06-03T08:34:29.292 DEBUG:teuthology.orchestra.run.smithi192:> sudo dd of=/usr/bin/daemon-helper 2024-06-03T08:34:29.316 DEBUG:teuthology.orchestra.run.smithi192:> sudo chmod a=rx -- /usr/bin/daemon-helper 2024-06-03T08:34:29.380 INFO:teuthology.task.install.util:Shipping 'adjust-ulimits'... 2024-06-03T08:34:29.380 DEBUG:teuthology.orchestra.run.smithi088:> set -ex 2024-06-03T08:34:29.380 DEBUG:teuthology.orchestra.run.smithi088:> sudo dd of=/usr/bin/adjust-ulimits 2024-06-03T08:34:29.408 DEBUG:teuthology.orchestra.run.smithi088:> sudo chmod a=rx -- /usr/bin/adjust-ulimits 2024-06-03T08:34:29.475 DEBUG:teuthology.orchestra.run.smithi192:> set -ex 2024-06-03T08:34:29.475 DEBUG:teuthology.orchestra.run.smithi192:> sudo dd of=/usr/bin/adjust-ulimits 2024-06-03T08:34:29.500 DEBUG:teuthology.orchestra.run.smithi192:> sudo chmod a=rx -- /usr/bin/adjust-ulimits 2024-06-03T08:34:29.565 INFO:teuthology.task.install.util:Shipping 'stdin-killer'... 2024-06-03T08:34:29.565 DEBUG:teuthology.orchestra.run.smithi088:> set -ex 2024-06-03T08:34:29.565 DEBUG:teuthology.orchestra.run.smithi088:> sudo dd of=/usr/bin/stdin-killer 2024-06-03T08:34:29.592 DEBUG:teuthology.orchestra.run.smithi088:> sudo chmod a=rx -- /usr/bin/stdin-killer 2024-06-03T08:34:29.658 DEBUG:teuthology.orchestra.run.smithi192:> set -ex 2024-06-03T08:34:29.658 DEBUG:teuthology.orchestra.run.smithi192:> sudo dd of=/usr/bin/stdin-killer 2024-06-03T08:34:29.682 DEBUG:teuthology.orchestra.run.smithi192:> sudo chmod a=rx -- /usr/bin/stdin-killer 2024-06-03T08:34:29.748 INFO:teuthology.run_tasks:Running task ceph... 2024-06-03T08:34:29.829 INFO:tasks.ceph:Making ceph log dir writeable by non-root... 2024-06-03T08:34:29.829 DEBUG:teuthology.orchestra.run.smithi088:> sudo chmod 777 /var/log/ceph 2024-06-03T08:34:29.832 DEBUG:teuthology.orchestra.run.smithi192:> sudo chmod 777 /var/log/ceph 2024-06-03T08:34:29.857 INFO:tasks.ceph:Disabling ceph logrotate... 2024-06-03T08:34:29.857 DEBUG:teuthology.orchestra.run.smithi088:> sudo rm -f -- /etc/logrotate.d/ceph 2024-06-03T08:34:29.900 DEBUG:teuthology.orchestra.run.smithi192:> sudo rm -f -- /etc/logrotate.d/ceph 2024-06-03T08:34:29.924 INFO:tasks.ceph:Creating extra log directories... 2024-06-03T08:34:29.925 DEBUG:teuthology.orchestra.run.smithi088:> sudo install -d -m0777 -- /var/log/ceph/valgrind /var/log/ceph/profiling-logger 2024-06-03T08:34:29.967 DEBUG:teuthology.orchestra.run.smithi192:> sudo install -d -m0777 -- /var/log/ceph/valgrind /var/log/ceph/profiling-logger 2024-06-03T08:34:30.001 INFO:tasks.ceph:Creating ceph cluster ceph... 2024-06-03T08:34:30.002 INFO:tasks.ceph:config {'conf': {'client': {'debug monc': 20, 'debug ms': 1, 'debug objecter': 20, 'debug rados': 20}, 'global': {'mon client directed command retry': 5, 'mon election default strategy': 3, 'ms bind msgr2': False, 'ms inject socket failures': 5000, 'ms type': 'async', 'osd heartbeat grace': 80, 'osd op thread timeout': 150}, 'mgr': {'debug mgr': 20, 'debug ms': 1}, 'mon': {'debug mon': 20, 'debug ms': 1, 'debug paxos': 20, 'mon min osdmap epochs': 50, 'mon osd crush smoke test': False, 'mon osdmap full prune interval': 2, 'mon osdmap full prune min': 15, 'mon osdmap full prune txsize': 2, 'mon scrub interval': 300, 'mon warn on pool no app': False, 'paxos service trim min': 10}, 'osd': {'bdev async discard': True, 'bdev enable discard': True, 'bluestore allocator': 'stupid', 'bluestore block size': 96636764160, 'bluestore fsck on mount': True, 'bluestore zero block detection': True, 'debug bluefs': 1, 'debug bluestore': 1, 'debug monc': 20, 'debug ms': 1, 'debug osd': 20, 'debug rocksdb': 10, 'mon osd backfillfull_ratio': 0.85, 'mon osd full ratio': 0.9, 'mon osd nearfull ratio': 0.8, 'osd class default list': '*', 'osd class load list': '*', 'osd client watch timeout': 120, 'osd debug verify cached snaps': True, 'osd debug verify missing on start': True, 'osd failsafe full ratio': 0.95, 'osd fast shutdown': False, 'osd mclock override recovery settings': True, 'osd mclock profile': 'high_recovery_ops', 'osd objectstore': 'bluestore', 'osd op queue': 'debug_random', 'osd op queue cut off': 'debug_random', 'osd shutdown pgref assert': True}}, 'fs': 'xfs', 'mkfs_options': None, 'mount_options': None, 'skip_mgr_daemons': False, 'log_ignorelist': ['\\(MDS_ALL_DOWN\\)', '\\(MDS_UP_LESS_THAN_MAX\\)', '\\(OSD_SLOW_PING_TIME', 'reached quota', 'overall HEALTH_', '\\(CACHE_POOL_NO_HIT_SET\\)', '\\(POOL_FULL\\)', '\\(SMALLER_PGP_NUM\\)', '\\(SLOW_OPS\\)', '\\(CACHE_POOL_NEAR_FULL\\)', '\\(POOL_APP_NOT_ENABLED\\)', '\\(PG_AVAILABILITY\\)', '\\(OBJECT_MISPLACED\\)', 'slow request', 'overall HEALTH_', '\\(PG_', '\\(MON_DOWN\\)', '\\(SLOW_OPS\\)', 'slow request'], 'cpu_profile': set(), 'cluster': 'ceph', 'mon_bind_msgr2': False, 'mon_bind_addrvec': True} 2024-06-03T08:34:30.002 INFO:tasks.ceph:ctx.config {'archive_path': '/home/teuthworker/archive/teuthology-2024-06-02_22:00:03-rados-reef-distro-default-smithi/7739049', 'branch': 'reef', 'description': 'rados/verify/{centos_latest ceph clusters/{fixed-2 openstack} d-thrash/none mon_election/connectivity msgr-failures/few msgr/async-v1only objectstore/bluestore-stupid rados tasks/rados_api_tests validater/valgrind}', 'email': 'ceph-qa@ceph.com', 'first_in_suite': False, 'job_id': '7739049', 'kernel': {'kdb': 1, 'sha1': 'distro'}, 'last_in_suite': False, 'machine_type': 'smithi', 'name': 'teuthology-2024-06-02_22:00:03-rados-reef-distro-default-smithi', 'no_nested_subset': False, 'openstack': [{'volumes': {'count': 4, 'size': 10}}, {'volumes': {'count': 4, 'size': 10}}], 'os_type': 'centos', 'os_version': '9.stream', 'overrides': {'admin_socket': {'branch': 'reef'}, 'ceph': {'conf': {'client': {'debug monc': 20, 'debug ms': 1, 'debug objecter': 20, 'debug rados': 20}, 'global': {'mon client directed command retry': 5, 'mon election default strategy': 3, 'ms bind msgr2': False, 'ms inject socket failures': 5000, 'ms type': 'async', 'osd heartbeat grace': 80, 'osd op thread timeout': 150}, 'mgr': {'debug mgr': 20, 'debug ms': 1}, 'mon': {'debug mon': 20, 'debug ms': 1, 'debug paxos': 20, 'mon min osdmap epochs': 50, 'mon osd crush smoke test': False, 'mon osdmap full prune interval': 2, 'mon osdmap full prune min': 15, 'mon osdmap full prune txsize': 2, 'mon scrub interval': 300, 'mon warn on pool no app': False, 'paxos service trim min': 10}, 'osd': {'bdev async discard': True, 'bdev enable discard': True, 'bluestore allocator': 'stupid', 'bluestore block size': 96636764160, 'bluestore fsck on mount': True, 'bluestore zero block detection': True, 'debug bluefs': 1, 'debug bluestore': 1, 'debug monc': 20, 'debug ms': 1, 'debug osd': 20, 'debug rocksdb': 10, 'mon osd backfillfull_ratio': 0.85, 'mon osd full ratio': 0.9, 'mon osd nearfull ratio': 0.8, 'osd class default list': '*', 'osd class load list': '*', 'osd client watch timeout': 120, 'osd debug verify cached snaps': True, 'osd debug verify missing on start': True, 'osd failsafe full ratio': 0.95, 'osd fast shutdown': False, 'osd mclock override recovery settings': True, 'osd mclock profile': 'high_recovery_ops', 'osd objectstore': 'bluestore', 'osd op queue': 'debug_random', 'osd op queue cut off': 'debug_random', 'osd shutdown pgref assert': True}}, 'flavor': 'default', 'fs': 'xfs', 'log-ignorelist': ['\\(MDS_ALL_DOWN\\)', '\\(MDS_UP_LESS_THAN_MAX\\)', '\\(OSD_SLOW_PING_TIME', 'reached quota', 'overall HEALTH_', '\\(CACHE_POOL_NO_HIT_SET\\)', '\\(POOL_FULL\\)', '\\(SMALLER_PGP_NUM\\)', '\\(SLOW_OPS\\)', '\\(CACHE_POOL_NEAR_FULL\\)', '\\(POOL_APP_NOT_ENABLED\\)', '\\(PG_AVAILABILITY\\)', '\\(OBJECT_MISPLACED\\)', 'slow request', 'overall HEALTH_', '\\(PG_', '\\(MON_DOWN\\)', '\\(SLOW_OPS\\)', 'slow request'], 'mon_bind_msgr2': False, 'sha1': 'f1abf795f4fc15bf2f30ac90c811a53ab21d5d65', 'valgrind': {'mds': ['--tool=memcheck'], 'mon': ['--tool=memcheck', '--leak-check=full', '--show-reachable=yes'], 'osd': ['--tool=memcheck']}}, 'ceph-deploy': {'bluestore': True, 'conf': {'client': {'log file': '/var/log/ceph/ceph-$name.$pid.log'}, 'mon': {}, 'osd': {'bdev async discard': True, 'bdev enable discard': True, 'bluestore block size': 96636764160, 'bluestore fsck on mount': True, 'debug bluefs': 20, 'debug bluestore': 20, 'debug rocksdb': 10, 'mon osd backfillfull_ratio': 0.85, 'mon osd full ratio': 0.9, 'mon osd nearfull ratio': 0.8, 'osd failsafe full ratio': 0.95, 'osd objectstore': 'bluestore'}}, 'fs': 'xfs'}, 'install': {'ceph': {'debuginfo': True, 'flavor': 'default', 'sha1': 'f1abf795f4fc15bf2f30ac90c811a53ab21d5d65'}}, 'thrashosds': {'bdev_inject_crash': 2, 'bdev_inject_crash_probability': 0.5}, 'workunit': {'branch': 'reef', 'sha1': 'f1abf795f4fc15bf2f30ac90c811a53ab21d5d65'}}, 'owner': 'scheduled_teuthology@teuthology', 'priority': 931, 'repo': 'https://git.ceph.com/ceph.git', 'roles': [['mon.a', 'mon.c', 'mgr.y', 'osd.0', 'osd.1', 'osd.2', 'osd.3', 'client.0', 'node-exporter.a'], ['mon.b', 'mgr.x', 'osd.4', 'osd.5', 'osd.6', 'osd.7', 'client.1', 'prometheus.a', 'node-exporter.b']], 'seed': 6006, 'sha1': 'f1abf795f4fc15bf2f30ac90c811a53ab21d5d65', 'sleep_before_teardown': 0, 'subset': '1861/100000', 'suite': 'rados', 'suite_branch': 'reef', 'suite_path': '/home/teuthworker/src/git.ceph.com_ceph_f1abf795f4fc15bf2f30ac90c811a53ab21d5d65/qa', 'suite_relpath': 'qa', 'suite_repo': 'https://git.ceph.com/ceph.git', 'suite_sha1': 'f1abf795f4fc15bf2f30ac90c811a53ab21d5d65', 'targets': {'smithi088.front.sepia.ceph.com': 'ecdsa-sha2-nistp256 AAAAE2VjZHNhLXNoYTItbmlzdHAyNTYAAAAIbmlzdHAyNTYAAABBBN5r+pnqR9ZU5/EEmFtgISdvjW+oxZ59NEzqi3i5Mif/oZX9TtsPQCe4xqEQYbrq3l+HbFDgc7ykrEhar97ndF0=', 'smithi192.front.sepia.ceph.com': 'ecdsa-sha2-nistp256 AAAAE2VjZHNhLXNoYTItbmlzdHAyNTYAAAAIbmlzdHAyNTYAAABBBOonkgHrAzRMX4IChrWrragyGABZjlHr7O8NAeLBaTiHcz1SyKGXxzg2Mlqd38WgrTD784FoVD4YaVw9nmM2wRU='}, 'tasks': [{'internal.check_packages': None}, {'internal.buildpackages_prep': None}, {'internal.save_config': None}, {'internal.check_lock': None}, {'internal.add_remotes': None}, {'console_log': None}, {'internal.connect': None}, {'internal.push_inventory': None}, {'internal.serialize_remote_roles': None}, {'internal.check_conflict': None}, {'internal.check_ceph_data': None}, {'internal.vm_setup': None}, {'kernel': {'kdb': 1, 'sha1': 'distro'}}, {'internal.base': None}, {'internal.archive_upload': None}, {'internal.archive': None}, {'internal.coredump': None}, {'internal.sudo': None}, {'internal.syslog': None}, {'internal.timer': None}, {'pcp': None}, {'selinux': None}, {'ansible.cephlab': None}, {'clock': None}, {'install': None}, {'ceph': None}, {'workunit': {'clients': {'client.0': ['rados/test.sh']}, 'env': {'ALLOW_TIMEOUTS': '1'}, 'timeout': '6h'}}], 'teuthology': {'fragments_dropped': [], 'meta': {}, 'postmerge': []}, 'teuthology_branch': 'main', 'teuthology_sha1': '861a8dcf7aa816a26e13f039336f7ed0a9aec0fa', 'timestamp': '2024-06-02_22:00:03', 'tube': 'smithi', 'user': 'teuthology', 'verbose': False, 'worker_log': '/home/teuthworker/archive/worker_logs/dispatcher.smithi.2698841'} 2024-06-03T08:34:30.002 DEBUG:teuthology.orchestra.run.smithi088:> install -d -m0755 -- /home/ubuntu/cephtest/ceph.data 2024-06-03T08:34:30.033 DEBUG:teuthology.orchestra.run.smithi192:> install -d -m0755 -- /home/ubuntu/cephtest/ceph.data 2024-06-03T08:34:30.050 DEBUG:teuthology.orchestra.run.smithi088:> sudo install -d -m0777 -- /var/run/ceph 2024-06-03T08:34:30.089 DEBUG:teuthology.orchestra.run.smithi192:> sudo install -d -m0777 -- /var/run/ceph 2024-06-03T08:34:30.116 DEBUG:teuthology.orchestra.run.smithi088:> set -ex 2024-06-03T08:34:30.116 DEBUG:teuthology.orchestra.run.smithi088:> dd if=/scratch_devs of=/dev/stdout 2024-06-03T08:34:30.167 DEBUG:teuthology.misc:devs=['/dev/vg_nvme/lv_1', '/dev/vg_nvme/lv_2', '/dev/vg_nvme/lv_3', '/dev/vg_nvme/lv_4'] 2024-06-03T08:34:30.168 DEBUG:teuthology.orchestra.run.smithi088:> stat /dev/vg_nvme/lv_1 2024-06-03T08:34:30.223 INFO:teuthology.orchestra.run.smithi088.stdout: File: /dev/vg_nvme/lv_1 -> ../dm-0 2024-06-03T08:34:30.223 INFO:teuthology.orchestra.run.smithi088.stdout: Size: 7 Blocks: 0 IO Block: 4096 symbolic link 2024-06-03T08:34:30.223 INFO:teuthology.orchestra.run.smithi088.stdout:Device: 5h/5d Inode: 668 Links: 1 2024-06-03T08:34:30.223 INFO:teuthology.orchestra.run.smithi088.stdout:Access: (0777/lrwxrwxrwx) Uid: ( 0/ root) Gid: ( 0/ root) 2024-06-03T08:34:30.223 INFO:teuthology.orchestra.run.smithi088.stdout:Context: system_u:object_r:device_t:s0 2024-06-03T08:34:30.223 INFO:teuthology.orchestra.run.smithi088.stdout:Access: 2024-06-03 08:27:37.830569195 +0000 2024-06-03T08:34:30.223 INFO:teuthology.orchestra.run.smithi088.stdout:Modify: 2024-06-03 08:24:59.940037360 +0000 2024-06-03T08:34:30.223 INFO:teuthology.orchestra.run.smithi088.stdout:Change: 2024-06-03 08:24:59.940037360 +0000 2024-06-03T08:34:30.223 INFO:teuthology.orchestra.run.smithi088.stdout: Birth: 2024-06-03 08:24:59.940037360 +0000 2024-06-03T08:34:30.224 DEBUG:teuthology.orchestra.run.smithi088:> sudo dd if=/dev/vg_nvme/lv_1 of=/dev/null count=1 2024-06-03T08:34:30.286 INFO:teuthology.orchestra.run.smithi088.stderr:1+0 records in 2024-06-03T08:34:30.286 INFO:teuthology.orchestra.run.smithi088.stderr:1+0 records out 2024-06-03T08:34:30.286 INFO:teuthology.orchestra.run.smithi088.stderr:512 bytes copied, 0.000189715 s, 2.7 MB/s 2024-06-03T08:34:30.288 DEBUG:teuthology.orchestra.run.smithi088:> ! mount | grep -v devtmpfs | grep -q /dev/vg_nvme/lv_1 2024-06-03T08:34:30.345 DEBUG:teuthology.orchestra.run.smithi088:> stat /dev/vg_nvme/lv_2 2024-06-03T08:34:30.400 INFO:teuthology.orchestra.run.smithi088.stdout: File: /dev/vg_nvme/lv_2 -> ../dm-1 2024-06-03T08:34:30.401 INFO:teuthology.orchestra.run.smithi088.stdout: Size: 7 Blocks: 0 IO Block: 4096 symbolic link 2024-06-03T08:34:30.401 INFO:teuthology.orchestra.run.smithi088.stdout:Device: 5h/5d Inode: 687 Links: 1 2024-06-03T08:34:30.401 INFO:teuthology.orchestra.run.smithi088.stdout:Access: (0777/lrwxrwxrwx) Uid: ( 0/ root) Gid: ( 0/ root) 2024-06-03T08:34:30.401 INFO:teuthology.orchestra.run.smithi088.stdout:Context: system_u:object_r:device_t:s0 2024-06-03T08:34:30.401 INFO:teuthology.orchestra.run.smithi088.stdout:Access: 2024-06-03 08:27:37.830569195 +0000 2024-06-03T08:34:30.401 INFO:teuthology.orchestra.run.smithi088.stdout:Modify: 2024-06-03 08:25:00.274030731 +0000 2024-06-03T08:34:30.401 INFO:teuthology.orchestra.run.smithi088.stdout:Change: 2024-06-03 08:25:00.274030731 +0000 2024-06-03T08:34:30.401 INFO:teuthology.orchestra.run.smithi088.stdout: Birth: 2024-06-03 08:25:00.274030731 +0000 2024-06-03T08:34:30.401 DEBUG:teuthology.orchestra.run.smithi088:> sudo dd if=/dev/vg_nvme/lv_2 of=/dev/null count=1 2024-06-03T08:34:30.464 INFO:teuthology.orchestra.run.smithi088.stderr:1+0 records in 2024-06-03T08:34:30.465 INFO:teuthology.orchestra.run.smithi088.stderr:1+0 records out 2024-06-03T08:34:30.465 INFO:teuthology.orchestra.run.smithi088.stderr:512 bytes copied, 0.000208614 s, 2.5 MB/s 2024-06-03T08:34:30.466 DEBUG:teuthology.orchestra.run.smithi088:> ! mount | grep -v devtmpfs | grep -q /dev/vg_nvme/lv_2 2024-06-03T08:34:30.522 DEBUG:teuthology.orchestra.run.smithi088:> stat /dev/vg_nvme/lv_3 2024-06-03T08:34:30.577 INFO:teuthology.orchestra.run.smithi088.stdout: File: /dev/vg_nvme/lv_3 -> ../dm-2 2024-06-03T08:34:30.578 INFO:teuthology.orchestra.run.smithi088.stdout: Size: 7 Blocks: 0 IO Block: 4096 symbolic link 2024-06-03T08:34:30.578 INFO:teuthology.orchestra.run.smithi088.stdout:Device: 5h/5d Inode: 705 Links: 1 2024-06-03T08:34:30.578 INFO:teuthology.orchestra.run.smithi088.stdout:Access: (0777/lrwxrwxrwx) Uid: ( 0/ root) Gid: ( 0/ root) 2024-06-03T08:34:30.578 INFO:teuthology.orchestra.run.smithi088.stdout:Context: system_u:object_r:device_t:s0 2024-06-03T08:34:30.578 INFO:teuthology.orchestra.run.smithi088.stdout:Access: 2024-06-03 08:27:37.830569195 +0000 2024-06-03T08:34:30.578 INFO:teuthology.orchestra.run.smithi088.stdout:Modify: 2024-06-03 08:25:00.615023964 +0000 2024-06-03T08:34:30.578 INFO:teuthology.orchestra.run.smithi088.stdout:Change: 2024-06-03 08:25:00.615023964 +0000 2024-06-03T08:34:30.578 INFO:teuthology.orchestra.run.smithi088.stdout: Birth: 2024-06-03 08:25:00.615023964 +0000 2024-06-03T08:34:30.579 DEBUG:teuthology.orchestra.run.smithi088:> sudo dd if=/dev/vg_nvme/lv_3 of=/dev/null count=1 2024-06-03T08:34:30.640 INFO:teuthology.orchestra.run.smithi088.stderr:1+0 records in 2024-06-03T08:34:30.640 INFO:teuthology.orchestra.run.smithi088.stderr:1+0 records out 2024-06-03T08:34:30.640 INFO:teuthology.orchestra.run.smithi088.stderr:512 bytes copied, 0.000184448 s, 2.8 MB/s 2024-06-03T08:34:30.641 DEBUG:teuthology.orchestra.run.smithi088:> ! mount | grep -v devtmpfs | grep -q /dev/vg_nvme/lv_3 2024-06-03T08:34:30.697 DEBUG:teuthology.orchestra.run.smithi088:> stat /dev/vg_nvme/lv_4 2024-06-03T08:34:30.752 INFO:teuthology.orchestra.run.smithi088.stdout: File: /dev/vg_nvme/lv_4 -> ../dm-3 2024-06-03T08:34:30.752 INFO:teuthology.orchestra.run.smithi088.stdout: Size: 7 Blocks: 0 IO Block: 4096 symbolic link 2024-06-03T08:34:30.752 INFO:teuthology.orchestra.run.smithi088.stdout:Device: 5h/5d Inode: 724 Links: 1 2024-06-03T08:34:30.752 INFO:teuthology.orchestra.run.smithi088.stdout:Access: (0777/lrwxrwxrwx) Uid: ( 0/ root) Gid: ( 0/ root) 2024-06-03T08:34:30.753 INFO:teuthology.orchestra.run.smithi088.stdout:Context: system_u:object_r:device_t:s0 2024-06-03T08:34:30.753 INFO:teuthology.orchestra.run.smithi088.stdout:Access: 2024-06-03 08:27:37.831569175 +0000 2024-06-03T08:34:30.753 INFO:teuthology.orchestra.run.smithi088.stdout:Modify: 2024-06-03 08:25:00.950017314 +0000 2024-06-03T08:34:30.753 INFO:teuthology.orchestra.run.smithi088.stdout:Change: 2024-06-03 08:25:00.950017314 +0000 2024-06-03T08:34:30.753 INFO:teuthology.orchestra.run.smithi088.stdout: Birth: 2024-06-03 08:25:00.950017314 +0000 2024-06-03T08:34:30.753 DEBUG:teuthology.orchestra.run.smithi088:> sudo dd if=/dev/vg_nvme/lv_4 of=/dev/null count=1 2024-06-03T08:34:30.815 INFO:teuthology.orchestra.run.smithi088.stderr:1+0 records in 2024-06-03T08:34:30.815 INFO:teuthology.orchestra.run.smithi088.stderr:1+0 records out 2024-06-03T08:34:30.815 INFO:teuthology.orchestra.run.smithi088.stderr:512 bytes copied, 0.000230244 s, 2.2 MB/s 2024-06-03T08:34:30.817 DEBUG:teuthology.orchestra.run.smithi088:> ! mount | grep -v devtmpfs | grep -q /dev/vg_nvme/lv_4 2024-06-03T08:34:30.873 INFO:tasks.ceph:osd dev map: {'osd.0': '/dev/vg_nvme/lv_1', 'osd.1': '/dev/vg_nvme/lv_2', 'osd.2': '/dev/vg_nvme/lv_3', 'osd.3': '/dev/vg_nvme/lv_4'} 2024-06-03T08:34:30.873 DEBUG:teuthology.orchestra.run.smithi192:> set -ex 2024-06-03T08:34:30.873 DEBUG:teuthology.orchestra.run.smithi192:> dd if=/scratch_devs of=/dev/stdout 2024-06-03T08:34:30.891 DEBUG:teuthology.misc:devs=['/dev/vg_nvme/lv_1', '/dev/vg_nvme/lv_2', '/dev/vg_nvme/lv_3', '/dev/vg_nvme/lv_4'] 2024-06-03T08:34:30.891 DEBUG:teuthology.orchestra.run.smithi192:> stat /dev/vg_nvme/lv_1 2024-06-03T08:34:30.948 INFO:teuthology.orchestra.run.smithi192.stdout: File: /dev/vg_nvme/lv_1 -> ../dm-0 2024-06-03T08:34:30.948 INFO:teuthology.orchestra.run.smithi192.stdout: Size: 7 Blocks: 0 IO Block: 4096 symbolic link 2024-06-03T08:34:30.948 INFO:teuthology.orchestra.run.smithi192.stdout:Device: 5h/5d Inode: 668 Links: 1 2024-06-03T08:34:30.948 INFO:teuthology.orchestra.run.smithi192.stdout:Access: (0777/lrwxrwxrwx) Uid: ( 0/ root) Gid: ( 0/ root) 2024-06-03T08:34:30.948 INFO:teuthology.orchestra.run.smithi192.stdout:Context: system_u:object_r:device_t:s0 2024-06-03T08:34:30.948 INFO:teuthology.orchestra.run.smithi192.stdout:Access: 2024-06-03 08:27:47.034259000 +0000 2024-06-03T08:34:30.949 INFO:teuthology.orchestra.run.smithi192.stdout:Modify: 2024-06-03 08:24:59.958512917 +0000 2024-06-03T08:34:30.949 INFO:teuthology.orchestra.run.smithi192.stdout:Change: 2024-06-03 08:24:59.958512917 +0000 2024-06-03T08:34:30.949 INFO:teuthology.orchestra.run.smithi192.stdout: Birth: 2024-06-03 08:24:59.958512917 +0000 2024-06-03T08:34:30.949 DEBUG:teuthology.orchestra.run.smithi192:> sudo dd if=/dev/vg_nvme/lv_1 of=/dev/null count=1 2024-06-03T08:34:31.013 INFO:teuthology.orchestra.run.smithi192.stderr:1+0 records in 2024-06-03T08:34:31.013 INFO:teuthology.orchestra.run.smithi192.stderr:1+0 records out 2024-06-03T08:34:31.013 INFO:teuthology.orchestra.run.smithi192.stderr:512 bytes copied, 0.000286654 s, 1.8 MB/s 2024-06-03T08:34:31.015 DEBUG:teuthology.orchestra.run.smithi192:> ! mount | grep -v devtmpfs | grep -q /dev/vg_nvme/lv_1 2024-06-03T08:34:31.072 DEBUG:teuthology.orchestra.run.smithi192:> stat /dev/vg_nvme/lv_2 2024-06-03T08:34:31.128 INFO:teuthology.orchestra.run.smithi192.stdout: File: /dev/vg_nvme/lv_2 -> ../dm-1 2024-06-03T08:34:31.128 INFO:teuthology.orchestra.run.smithi192.stdout: Size: 7 Blocks: 0 IO Block: 4096 symbolic link 2024-06-03T08:34:31.128 INFO:teuthology.orchestra.run.smithi192.stdout:Device: 5h/5d Inode: 687 Links: 1 2024-06-03T08:34:31.128 INFO:teuthology.orchestra.run.smithi192.stdout:Access: (0777/lrwxrwxrwx) Uid: ( 0/ root) Gid: ( 0/ root) 2024-06-03T08:34:31.129 INFO:teuthology.orchestra.run.smithi192.stdout:Context: system_u:object_r:device_t:s0 2024-06-03T08:34:31.129 INFO:teuthology.orchestra.run.smithi192.stdout:Access: 2024-06-03 08:27:47.035258981 +0000 2024-06-03T08:34:31.129 INFO:teuthology.orchestra.run.smithi192.stdout:Modify: 2024-06-03 08:25:00.292506527 +0000 2024-06-03T08:34:31.129 INFO:teuthology.orchestra.run.smithi192.stdout:Change: 2024-06-03 08:25:00.292506527 +0000 2024-06-03T08:34:31.129 INFO:teuthology.orchestra.run.smithi192.stdout: Birth: 2024-06-03 08:25:00.292506527 +0000 2024-06-03T08:34:31.129 DEBUG:teuthology.orchestra.run.smithi192:> sudo dd if=/dev/vg_nvme/lv_2 of=/dev/null count=1 2024-06-03T08:34:31.192 INFO:teuthology.orchestra.run.smithi192.stderr:1+0 records in 2024-06-03T08:34:31.192 INFO:teuthology.orchestra.run.smithi192.stderr:1+0 records out 2024-06-03T08:34:31.192 INFO:teuthology.orchestra.run.smithi192.stderr:512 bytes copied, 0.000256292 s, 2.0 MB/s 2024-06-03T08:34:31.194 DEBUG:teuthology.orchestra.run.smithi192:> ! mount | grep -v devtmpfs | grep -q /dev/vg_nvme/lv_2 2024-06-03T08:34:31.250 DEBUG:teuthology.orchestra.run.smithi192:> stat /dev/vg_nvme/lv_3 2024-06-03T08:34:31.307 INFO:teuthology.orchestra.run.smithi192.stdout: File: /dev/vg_nvme/lv_3 -> ../dm-2 2024-06-03T08:34:31.307 INFO:teuthology.orchestra.run.smithi192.stdout: Size: 7 Blocks: 0 IO Block: 4096 symbolic link 2024-06-03T08:34:31.307 INFO:teuthology.orchestra.run.smithi192.stdout:Device: 5h/5d Inode: 705 Links: 1 2024-06-03T08:34:31.307 INFO:teuthology.orchestra.run.smithi192.stdout:Access: (0777/lrwxrwxrwx) Uid: ( 0/ root) Gid: ( 0/ root) 2024-06-03T08:34:31.307 INFO:teuthology.orchestra.run.smithi192.stdout:Context: system_u:object_r:device_t:s0 2024-06-03T08:34:31.307 INFO:teuthology.orchestra.run.smithi192.stdout:Access: 2024-06-03 08:27:47.035258981 +0000 2024-06-03T08:34:31.307 INFO:teuthology.orchestra.run.smithi192.stdout:Modify: 2024-06-03 08:25:00.614500367 +0000 2024-06-03T08:34:31.307 INFO:teuthology.orchestra.run.smithi192.stdout:Change: 2024-06-03 08:25:00.614500367 +0000 2024-06-03T08:34:31.307 INFO:teuthology.orchestra.run.smithi192.stdout: Birth: 2024-06-03 08:25:00.614500367 +0000 2024-06-03T08:34:31.308 DEBUG:teuthology.orchestra.run.smithi192:> sudo dd if=/dev/vg_nvme/lv_3 of=/dev/null count=1 2024-06-03T08:34:31.370 INFO:teuthology.orchestra.run.smithi192.stderr:1+0 records in 2024-06-03T08:34:31.370 INFO:teuthology.orchestra.run.smithi192.stderr:1+0 records out 2024-06-03T08:34:31.370 INFO:teuthology.orchestra.run.smithi192.stderr:512 bytes copied, 0.000228722 s, 2.2 MB/s 2024-06-03T08:34:31.372 DEBUG:teuthology.orchestra.run.smithi192:> ! mount | grep -v devtmpfs | grep -q /dev/vg_nvme/lv_3 2024-06-03T08:34:31.428 DEBUG:teuthology.orchestra.run.smithi192:> stat /dev/vg_nvme/lv_4 2024-06-03T08:34:31.484 INFO:teuthology.orchestra.run.smithi192.stdout: File: /dev/vg_nvme/lv_4 -> ../dm-3 2024-06-03T08:34:31.484 INFO:teuthology.orchestra.run.smithi192.stdout: Size: 7 Blocks: 0 IO Block: 4096 symbolic link 2024-06-03T08:34:31.484 INFO:teuthology.orchestra.run.smithi192.stdout:Device: 5h/5d Inode: 724 Links: 1 2024-06-03T08:34:31.484 INFO:teuthology.orchestra.run.smithi192.stdout:Access: (0777/lrwxrwxrwx) Uid: ( 0/ root) Gid: ( 0/ root) 2024-06-03T08:34:31.484 INFO:teuthology.orchestra.run.smithi192.stdout:Context: system_u:object_r:device_t:s0 2024-06-03T08:34:31.485 INFO:teuthology.orchestra.run.smithi192.stdout:Access: 2024-06-03 08:27:47.036258962 +0000 2024-06-03T08:34:31.485 INFO:teuthology.orchestra.run.smithi192.stdout:Modify: 2024-06-03 08:25:00.945494034 +0000 2024-06-03T08:34:31.485 INFO:teuthology.orchestra.run.smithi192.stdout:Change: 2024-06-03 08:25:00.945494034 +0000 2024-06-03T08:34:31.485 INFO:teuthology.orchestra.run.smithi192.stdout: Birth: 2024-06-03 08:25:00.945494034 +0000 2024-06-03T08:34:31.485 DEBUG:teuthology.orchestra.run.smithi192:> sudo dd if=/dev/vg_nvme/lv_4 of=/dev/null count=1 2024-06-03T08:34:31.548 INFO:teuthology.orchestra.run.smithi192.stderr:1+0 records in 2024-06-03T08:34:31.548 INFO:teuthology.orchestra.run.smithi192.stderr:1+0 records out 2024-06-03T08:34:31.548 INFO:teuthology.orchestra.run.smithi192.stderr:512 bytes copied, 0.000194819 s, 2.6 MB/s 2024-06-03T08:34:31.549 DEBUG:teuthology.orchestra.run.smithi192:> ! mount | grep -v devtmpfs | grep -q /dev/vg_nvme/lv_4 2024-06-03T08:34:31.605 INFO:tasks.ceph:osd dev map: {'osd.4': '/dev/vg_nvme/lv_1', 'osd.5': '/dev/vg_nvme/lv_2', 'osd.6': '/dev/vg_nvme/lv_3', 'osd.7': '/dev/vg_nvme/lv_4'} 2024-06-03T08:34:31.605 INFO:tasks.ceph:remote_to_roles_to_devs: {Remote(name='ubuntu@smithi088.front.sepia.ceph.com'): {'osd.0': '/dev/vg_nvme/lv_1', 'osd.1': '/dev/vg_nvme/lv_2', 'osd.2': '/dev/vg_nvme/lv_3', 'osd.3': '/dev/vg_nvme/lv_4'}, Remote(name='ubuntu@smithi192.front.sepia.ceph.com'): {'osd.4': '/dev/vg_nvme/lv_1', 'osd.5': '/dev/vg_nvme/lv_2', 'osd.6': '/dev/vg_nvme/lv_3', 'osd.7': '/dev/vg_nvme/lv_4'}} 2024-06-03T08:34:31.605 INFO:tasks.ceph:Generating config... 2024-06-03T08:34:31.607 INFO:tasks.ceph:[client] debug monc = 20 2024-06-03T08:34:31.607 INFO:tasks.ceph:[client] debug ms = 1 2024-06-03T08:34:31.607 INFO:tasks.ceph:[client] debug objecter = 20 2024-06-03T08:34:31.607 INFO:tasks.ceph:[client] debug rados = 20 2024-06-03T08:34:31.607 INFO:tasks.ceph:[global] mon client directed command retry = 5 2024-06-03T08:34:31.607 INFO:tasks.ceph:[global] mon election default strategy = 3 2024-06-03T08:34:31.607 INFO:tasks.ceph:[global] ms bind msgr2 = False 2024-06-03T08:34:31.607 INFO:tasks.ceph:[global] ms inject socket failures = 5000 2024-06-03T08:34:31.607 INFO:tasks.ceph:[global] ms type = async 2024-06-03T08:34:31.607 INFO:tasks.ceph:[global] osd heartbeat grace = 80 2024-06-03T08:34:31.607 INFO:tasks.ceph:[global] osd op thread timeout = 150 2024-06-03T08:34:31.607 INFO:tasks.ceph:[mgr] debug mgr = 20 2024-06-03T08:34:31.608 INFO:tasks.ceph:[mgr] debug ms = 1 2024-06-03T08:34:31.608 INFO:tasks.ceph:[mon] debug mon = 20 2024-06-03T08:34:31.608 INFO:tasks.ceph:[mon] debug ms = 1 2024-06-03T08:34:31.608 INFO:tasks.ceph:[mon] debug paxos = 20 2024-06-03T08:34:31.608 INFO:tasks.ceph:[mon] mon min osdmap epochs = 50 2024-06-03T08:34:31.608 INFO:tasks.ceph:[mon] mon osd crush smoke test = False 2024-06-03T08:34:31.608 INFO:tasks.ceph:[mon] mon osdmap full prune interval = 2 2024-06-03T08:34:31.608 INFO:tasks.ceph:[mon] mon osdmap full prune min = 15 2024-06-03T08:34:31.608 INFO:tasks.ceph:[mon] mon osdmap full prune txsize = 2 2024-06-03T08:34:31.608 INFO:tasks.ceph:[mon] mon scrub interval = 300 2024-06-03T08:34:31.608 INFO:tasks.ceph:[mon] mon warn on pool no app = False 2024-06-03T08:34:31.608 INFO:tasks.ceph:[mon] paxos service trim min = 10 2024-06-03T08:34:31.608 INFO:tasks.ceph:[osd] bdev async discard = True 2024-06-03T08:34:31.609 INFO:tasks.ceph:[osd] bdev enable discard = True 2024-06-03T08:34:31.609 INFO:tasks.ceph:[osd] bluestore allocator = stupid 2024-06-03T08:34:31.609 INFO:tasks.ceph:[osd] bluestore block size = 96636764160 2024-06-03T08:34:31.609 INFO:tasks.ceph:[osd] bluestore fsck on mount = True 2024-06-03T08:34:31.609 INFO:tasks.ceph:[osd] bluestore zero block detection = True 2024-06-03T08:34:31.609 INFO:tasks.ceph:[osd] debug bluefs = 1 2024-06-03T08:34:31.609 INFO:tasks.ceph:[osd] debug bluestore = 1 2024-06-03T08:34:31.609 INFO:tasks.ceph:[osd] debug monc = 20 2024-06-03T08:34:31.609 INFO:tasks.ceph:[osd] debug ms = 1 2024-06-03T08:34:31.609 INFO:tasks.ceph:[osd] debug osd = 20 2024-06-03T08:34:31.609 INFO:tasks.ceph:[osd] debug rocksdb = 10 2024-06-03T08:34:31.609 INFO:tasks.ceph:[osd] mon osd backfillfull_ratio = 0.85 2024-06-03T08:34:31.609 INFO:tasks.ceph:[osd] mon osd full ratio = 0.9 2024-06-03T08:34:31.609 INFO:tasks.ceph:[osd] mon osd nearfull ratio = 0.8 2024-06-03T08:34:31.610 INFO:tasks.ceph:[osd] osd class default list = * 2024-06-03T08:34:31.610 INFO:tasks.ceph:[osd] osd class load list = * 2024-06-03T08:34:31.610 INFO:tasks.ceph:[osd] osd client watch timeout = 120 2024-06-03T08:34:31.610 INFO:tasks.ceph:[osd] osd debug verify cached snaps = True 2024-06-03T08:34:31.610 INFO:tasks.ceph:[osd] osd debug verify missing on start = True 2024-06-03T08:34:31.610 INFO:tasks.ceph:[osd] osd failsafe full ratio = 0.95 2024-06-03T08:34:31.610 INFO:tasks.ceph:[osd] osd fast shutdown = False 2024-06-03T08:34:31.610 INFO:tasks.ceph:[osd] osd mclock override recovery settings = True 2024-06-03T08:34:31.610 INFO:tasks.ceph:[osd] osd mclock profile = high_recovery_ops 2024-06-03T08:34:31.610 INFO:tasks.ceph:[osd] osd objectstore = bluestore 2024-06-03T08:34:31.610 INFO:tasks.ceph:[osd] osd op queue = debug_random 2024-06-03T08:34:31.610 INFO:tasks.ceph:[osd] osd op queue cut off = debug_random 2024-06-03T08:34:31.610 INFO:tasks.ceph:[osd] osd shutdown pgref assert = True 2024-06-03T08:34:31.611 INFO:tasks.ceph:Setting up mon.a... 2024-06-03T08:34:31.611 DEBUG:teuthology.orchestra.run.smithi088:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-authtool --create-keyring /etc/ceph/ceph.keyring 2024-06-03T08:34:31.668 INFO:teuthology.orchestra.run.smithi088.stdout:creating /etc/ceph/ceph.keyring 2024-06-03T08:34:31.671 DEBUG:teuthology.orchestra.run.smithi088:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-authtool --gen-key --name=mon. /etc/ceph/ceph.keyring 2024-06-03T08:34:31.715 DEBUG:teuthology.orchestra.run.smithi088:> sudo chmod 0644 /etc/ceph/ceph.keyring 2024-06-03T08:34:31.740 DEBUG:tasks.ceph:Ceph mon addresses: [('mon.a', '[v1:172.21.15.88:6789]'), ('mon.c', '[v1:172.21.15.88:6790]'), ('mon.b', '[v1:172.21.15.192:6789]')] 2024-06-03T08:34:31.740 DEBUG:tasks.ceph:writing out conf {'global': {'chdir': '', 'pid file': '/var/run/ceph/$cluster-$name.pid', 'auth supported': 'cephx', 'filestore xattr use omap': 'true', 'mon clock drift allowed': '1.000', 'osd crush chooseleaf type': '0', 'auth debug': 'true', 'ms die on old message': 'true', 'ms die on bug': 'true', 'mon max pg per osd': '10000', 'mon pg warn max object skew': '0', 'osd_pool_default_pg_autoscale_mode': 'off', 'osd pool default size': '2', 'mon osd allow primary affinity': 'true', 'mon osd allow pg remap': 'true', 'mon warn on legacy crush tunables': 'false', 'mon warn on crush straw calc version zero': 'false', 'mon warn on no sortbitwise': 'false', 'mon warn on osd down out interval zero': 'false', 'mon warn on too few osds': 'false', 'mon_warn_on_pool_pg_num_not_power_of_two': 'false', 'mon_warn_on_pool_no_redundancy': 'false', 'mon_allow_pool_size_one': 'true', 'osd pool default erasure code profile': 'plugin=jerasure technique=reed_sol_van k=2 m=1 crush-failure-domain=osd', 'osd default data pool replay window': '5', 'mon allow pool delete': 'true', 'mon cluster log file level': 'debug', 'debug asserts on shutdown': 'true', 'mon health detail to clog': 'false', 'mon host': '[v1:172.21.15.88:6789],[v1:172.21.15.88:6790],[v1:172.21.15.192:6789]', 'mon client directed command retry': 5, 'mon election default strategy': 3, 'ms bind msgr2': False, 'ms inject socket failures': 5000, 'ms type': 'async', 'osd heartbeat grace': 80, 'osd op thread timeout': 150}, 'osd': {'osd journal size': '100', 'osd scrub load threshold': '5.0', 'osd scrub max interval': '600', 'osd mclock profile': 'high_recovery_ops', 'osd recover clone overlap': 'true', 'osd recovery max chunk': '1048576', 'osd debug shutdown': 'true', 'osd debug op order': 'true', 'osd debug verify stray on activate': 'true', 'osd open classes on start': 'true', 'osd debug pg log writeout': 'true', 'osd deep scrub update digest min age': '30', 'osd map max advance': '10', 'journal zero on create': 'true', 'filestore ondisk finisher threads': '3', 'filestore apply finisher threads': '3', 'bdev debug aio': 'true', 'osd debug misdirected ops': 'true', 'bdev async discard': True, 'bdev enable discard': True, 'bluestore allocator': 'stupid', 'bluestore block size': 96636764160, 'bluestore fsck on mount': True, 'bluestore zero block detection': True, 'debug bluefs': 1, 'debug bluestore': 1, 'debug monc': 20, 'debug ms': 1, 'debug osd': 20, 'debug rocksdb': 10, 'mon osd backfillfull_ratio': 0.85, 'mon osd full ratio': 0.9, 'mon osd nearfull ratio': 0.8, 'osd class default list': '*', 'osd class load list': '*', 'osd client watch timeout': 120, 'osd debug verify cached snaps': True, 'osd debug verify missing on start': True, 'osd failsafe full ratio': 0.95, 'osd fast shutdown': False, 'osd mclock override recovery settings': True, 'osd objectstore': 'bluestore', 'osd op queue': 'debug_random', 'osd op queue cut off': 'debug_random', 'osd shutdown pgref assert': True}, 'mgr': {'debug ms': 1, 'debug mgr': 20, 'debug mon': '20', 'debug auth': '20', 'mon reweight min pgs per osd': '4', 'mon reweight min bytes per osd': '10', 'mgr/telemetry/nag': 'false'}, 'mon': {'debug ms': 1, 'debug mon': 20, 'debug paxos': 20, 'debug auth': '20', 'mon data avail warn': '5', 'mon mgr mkfs grace': '240', 'mon reweight min pgs per osd': '4', 'mon osd reporter subtree level': 'osd', 'mon osd prime pg temp': 'true', 'mon reweight min bytes per osd': '10', 'auth mon ticket ttl': '660', 'auth service ticket ttl': '240', 'mon_warn_on_insecure_global_id_reclaim': 'false', 'mon_warn_on_insecure_global_id_reclaim_allowed': 'false', 'mon_down_mkfs_grace': '2m', 'mon_warn_on_filestore_osds': 'false', 'mon min osdmap epochs': 50, 'mon osd crush smoke test': False, 'mon osdmap full prune interval': 2, 'mon osdmap full prune min': 15, 'mon osdmap full prune txsize': 2, 'mon scrub interval': 300, 'mon warn on pool no app': False, 'paxos service trim min': 10}, 'client': {'rgw cache enabled': 'true', 'rgw enable ops log': 'true', 'rgw enable usage log': 'true', 'log file': '/var/log/ceph/$cluster-$name.$pid.log', 'admin socket': '/var/run/ceph/$cluster-$name.$pid.asok', 'debug monc': 20, 'debug ms': 1, 'debug objecter': 20, 'debug rados': 20}, 'mon.a': {}, 'mon.c': {}, 'mon.b': {}} 2024-06-03T08:34:31.741 DEBUG:teuthology.orchestra.run.smithi088:> set -ex 2024-06-03T08:34:31.741 DEBUG:teuthology.orchestra.run.smithi088:> dd of=/home/ubuntu/cephtest/ceph.tmp.conf 2024-06-03T08:34:31.797 DEBUG:teuthology.orchestra.run.smithi088:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage monmaptool -c /home/ubuntu/cephtest/ceph.tmp.conf --create --clobber --enable-all-features --addv a '[v1:172.21.15.88:6789]' --addv c '[v1:172.21.15.88:6790]' --addv b '[v1:172.21.15.192:6789]' --print /home/ubuntu/cephtest/ceph.monmap 2024-06-03T08:34:31.884 INFO:teuthology.orchestra.run.smithi088.stdout:monmaptool: monmap file /home/ubuntu/cephtest/ceph.monmap 2024-06-03T08:34:31.884 INFO:teuthology.orchestra.run.smithi088.stdout:monmaptool: generated fsid ec3c3fdd-3ac6-4d7d-9412-683130ed8b04 2024-06-03T08:34:31.884 INFO:teuthology.orchestra.run.smithi088.stdout:setting min_mon_release = pacific 2024-06-03T08:34:31.885 INFO:teuthology.orchestra.run.smithi088.stdout:epoch 0 2024-06-03T08:34:31.885 INFO:teuthology.orchestra.run.smithi088.stdout:fsid ec3c3fdd-3ac6-4d7d-9412-683130ed8b04 2024-06-03T08:34:31.885 INFO:teuthology.orchestra.run.smithi088.stdout:last_changed 2024-06-03T08:34:31.885125+0000 2024-06-03T08:34:31.885 INFO:teuthology.orchestra.run.smithi088.stdout:created 2024-06-03T08:34:31.885125+0000 2024-06-03T08:34:31.885 INFO:teuthology.orchestra.run.smithi088.stdout:min_mon_release 16 (pacific) 2024-06-03T08:34:31.885 INFO:teuthology.orchestra.run.smithi088.stdout:election_strategy: 3 2024-06-03T08:34:31.885 INFO:teuthology.orchestra.run.smithi088.stdout:0: v1:172.21.15.88:6789/0 mon.a 2024-06-03T08:34:31.885 INFO:teuthology.orchestra.run.smithi088.stdout:1: v1:172.21.15.192:6789/0 mon.b 2024-06-03T08:34:31.885 INFO:teuthology.orchestra.run.smithi088.stdout:2: v1:172.21.15.88:6790/0 mon.c 2024-06-03T08:34:31.885 INFO:teuthology.orchestra.run.smithi088.stdout:monmaptool: writing epoch 0 to /home/ubuntu/cephtest/ceph.monmap (3 monitors) 2024-06-03T08:34:31.886 DEBUG:teuthology.orchestra.run.smithi088:> rm -- /home/ubuntu/cephtest/ceph.tmp.conf 2024-06-03T08:34:31.901 INFO:tasks.ceph:Writing /etc/ceph/ceph.conf for FSID ec3c3fdd-3ac6-4d7d-9412-683130ed8b04... 2024-06-03T08:34:31.902 DEBUG:teuthology.orchestra.run.smithi088:> sudo mkdir -p /etc/ceph && sudo chmod 0755 /etc/ceph && sudo tee /etc/ceph/ceph.conf && sudo chmod 0644 /etc/ceph/ceph.conf > /dev/null 2024-06-03T08:34:31.944 DEBUG:teuthology.orchestra.run.smithi192:> sudo mkdir -p /etc/ceph && sudo chmod 0755 /etc/ceph && sudo tee /etc/ceph/ceph.conf && sudo chmod 0644 /etc/ceph/ceph.conf > /dev/null 2024-06-03T08:34:31.984 INFO:teuthology.orchestra.run.smithi088.stdout:[global] 2024-06-03T08:34:31.984 INFO:teuthology.orchestra.run.smithi088.stdout: chdir = "" 2024-06-03T08:34:31.984 INFO:teuthology.orchestra.run.smithi088.stdout: pid file = /var/run/ceph/$cluster-$name.pid 2024-06-03T08:34:31.984 INFO:teuthology.orchestra.run.smithi088.stdout: auth supported = cephx 2024-06-03T08:34:31.984 INFO:teuthology.orchestra.run.smithi088.stdout: 2024-06-03T08:34:31.984 INFO:teuthology.orchestra.run.smithi088.stdout: filestore xattr use omap = true 2024-06-03T08:34:31.984 INFO:teuthology.orchestra.run.smithi088.stdout: 2024-06-03T08:34:31.984 INFO:teuthology.orchestra.run.smithi088.stdout: mon clock drift allowed = 1.000 2024-06-03T08:34:31.984 INFO:teuthology.orchestra.run.smithi088.stdout: 2024-06-03T08:34:31.985 INFO:teuthology.orchestra.run.smithi088.stdout: osd crush chooseleaf type = 0 2024-06-03T08:34:31.985 INFO:teuthology.orchestra.run.smithi088.stdout: auth debug = true 2024-06-03T08:34:31.985 INFO:teuthology.orchestra.run.smithi088.stdout: 2024-06-03T08:34:31.985 INFO:teuthology.orchestra.run.smithi088.stdout: ms die on old message = true 2024-06-03T08:34:31.985 INFO:teuthology.orchestra.run.smithi088.stdout: ms die on bug = true 2024-06-03T08:34:31.985 INFO:teuthology.orchestra.run.smithi088.stdout: 2024-06-03T08:34:31.985 INFO:teuthology.orchestra.run.smithi088.stdout: mon max pg per osd = 10000 # >= luminous 2024-06-03T08:34:31.985 INFO:teuthology.orchestra.run.smithi088.stdout: mon pg warn max object skew = 0 2024-06-03T08:34:31.985 INFO:teuthology.orchestra.run.smithi088.stdout: 2024-06-03T08:34:31.985 INFO:teuthology.orchestra.run.smithi088.stdout: # disable pg_autoscaler by default for new pools 2024-06-03T08:34:31.985 INFO:teuthology.orchestra.run.smithi088.stdout: osd_pool_default_pg_autoscale_mode = off 2024-06-03T08:34:31.985 INFO:teuthology.orchestra.run.smithi088.stdout: 2024-06-03T08:34:31.985 INFO:teuthology.orchestra.run.smithi088.stdout: osd pool default size = 2 2024-06-03T08:34:31.986 INFO:teuthology.orchestra.run.smithi088.stdout: 2024-06-03T08:34:31.986 INFO:teuthology.orchestra.run.smithi088.stdout: mon osd allow primary affinity = true 2024-06-03T08:34:31.986 INFO:teuthology.orchestra.run.smithi088.stdout: mon osd allow pg remap = true 2024-06-03T08:34:31.986 INFO:teuthology.orchestra.run.smithi088.stdout: mon warn on legacy crush tunables = false 2024-06-03T08:34:31.986 INFO:teuthology.orchestra.run.smithi088.stdout: mon warn on crush straw calc version zero = false 2024-06-03T08:34:31.986 INFO:teuthology.orchestra.run.smithi088.stdout: mon warn on no sortbitwise = false 2024-06-03T08:34:31.986 INFO:teuthology.orchestra.run.smithi088.stdout: mon warn on osd down out interval zero = false 2024-06-03T08:34:31.986 INFO:teuthology.orchestra.run.smithi088.stdout: mon warn on too few osds = false 2024-06-03T08:34:31.986 INFO:teuthology.orchestra.run.smithi088.stdout: mon_warn_on_pool_pg_num_not_power_of_two = false 2024-06-03T08:34:31.987 INFO:teuthology.orchestra.run.smithi088.stdout: mon_warn_on_pool_no_redundancy = false 2024-06-03T08:34:31.987 INFO:teuthology.orchestra.run.smithi088.stdout: mon_allow_pool_size_one = true 2024-06-03T08:34:31.987 INFO:teuthology.orchestra.run.smithi088.stdout: 2024-06-03T08:34:31.987 INFO:teuthology.orchestra.run.smithi088.stdout: osd pool default erasure code profile = plugin=jerasure technique=reed_sol_van k=2 m=1 crush-failure-domain=osd 2024-06-03T08:34:31.987 INFO:teuthology.orchestra.run.smithi088.stdout: 2024-06-03T08:34:31.987 INFO:teuthology.orchestra.run.smithi088.stdout: osd default data pool replay window = 5 2024-06-03T08:34:31.987 INFO:teuthology.orchestra.run.smithi088.stdout: 2024-06-03T08:34:31.987 INFO:teuthology.orchestra.run.smithi088.stdout: mon allow pool delete = true 2024-06-03T08:34:31.987 INFO:teuthology.orchestra.run.smithi088.stdout: 2024-06-03T08:34:31.988 INFO:teuthology.orchestra.run.smithi088.stdout: mon cluster log file level = debug 2024-06-03T08:34:31.988 INFO:teuthology.orchestra.run.smithi088.stdout: debug asserts on shutdown = true 2024-06-03T08:34:31.988 INFO:teuthology.orchestra.run.smithi088.stdout: mon health detail to clog = false 2024-06-03T08:34:31.988 INFO:teuthology.orchestra.run.smithi088.stdout: mon host = "[v1:172.21.15.88:6789],[v1:172.21.15.88:6790],[v1:172.21.15.192:6789]" 2024-06-03T08:34:31.988 INFO:teuthology.orchestra.run.smithi088.stdout: mon client directed command retry = 5 2024-06-03T08:34:31.988 INFO:teuthology.orchestra.run.smithi088.stdout: mon election default strategy = 3 2024-06-03T08:34:31.988 INFO:teuthology.orchestra.run.smithi088.stdout: ms bind msgr2 = False 2024-06-03T08:34:31.988 INFO:teuthology.orchestra.run.smithi088.stdout: ms inject socket failures = 5000 2024-06-03T08:34:31.988 INFO:teuthology.orchestra.run.smithi088.stdout: ms type = async 2024-06-03T08:34:31.988 INFO:teuthology.orchestra.run.smithi088.stdout: osd heartbeat grace = 80 2024-06-03T08:34:31.988 INFO:teuthology.orchestra.run.smithi088.stdout: osd op thread timeout = 150 2024-06-03T08:34:31.988 INFO:teuthology.orchestra.run.smithi088.stdout: fsid = ec3c3fdd-3ac6-4d7d-9412-683130ed8b04 2024-06-03T08:34:31.988 INFO:teuthology.orchestra.run.smithi088.stdout: 2024-06-03T08:34:31.989 INFO:teuthology.orchestra.run.smithi088.stdout:[osd] 2024-06-03T08:34:31.989 INFO:teuthology.orchestra.run.smithi088.stdout: osd journal size = 100 2024-06-03T08:34:31.989 INFO:teuthology.orchestra.run.smithi088.stdout: 2024-06-03T08:34:31.989 INFO:teuthology.orchestra.run.smithi088.stdout: osd scrub load threshold = 5.0 2024-06-03T08:34:31.989 INFO:teuthology.orchestra.run.smithi088.stdout: osd scrub max interval = 600 2024-06-03T08:34:31.989 INFO:teuthology.orchestra.run.smithi088.stdout: osd mclock profile = high_recovery_ops 2024-06-03T08:34:31.989 INFO:teuthology.orchestra.run.smithi088.stdout: 2024-06-03T08:34:31.989 INFO:teuthology.orchestra.run.smithi088.stdout: osd recover clone overlap = true 2024-06-03T08:34:31.989 INFO:teuthology.orchestra.run.smithi088.stdout: osd recovery max chunk = 1048576 2024-06-03T08:34:31.989 INFO:teuthology.orchestra.run.smithi088.stdout: 2024-06-03T08:34:31.989 INFO:teuthology.orchestra.run.smithi088.stdout: osd debug shutdown = true 2024-06-03T08:34:31.990 INFO:teuthology.orchestra.run.smithi088.stdout: osd debug op order = true 2024-06-03T08:34:31.990 INFO:teuthology.orchestra.run.smithi088.stdout: osd debug verify stray on activate = true 2024-06-03T08:34:31.990 INFO:teuthology.orchestra.run.smithi088.stdout: 2024-06-03T08:34:31.990 INFO:teuthology.orchestra.run.smithi088.stdout: osd open classes on start = true 2024-06-03T08:34:31.990 INFO:teuthology.orchestra.run.smithi088.stdout: osd debug pg log writeout = true 2024-06-03T08:34:31.990 INFO:teuthology.orchestra.run.smithi088.stdout: 2024-06-03T08:34:31.990 INFO:teuthology.orchestra.run.smithi088.stdout: osd deep scrub update digest min age = 30 2024-06-03T08:34:31.990 INFO:teuthology.orchestra.run.smithi088.stdout: 2024-06-03T08:34:31.990 INFO:teuthology.orchestra.run.smithi088.stdout: osd map max advance = 10 2024-06-03T08:34:31.990 INFO:teuthology.orchestra.run.smithi088.stdout: 2024-06-03T08:34:31.991 INFO:teuthology.orchestra.run.smithi088.stdout: journal zero on create = true 2024-06-03T08:34:31.991 INFO:teuthology.orchestra.run.smithi088.stdout: 2024-06-03T08:34:31.991 INFO:teuthology.orchestra.run.smithi088.stdout: filestore ondisk finisher threads = 3 2024-06-03T08:34:31.991 INFO:teuthology.orchestra.run.smithi088.stdout: filestore apply finisher threads = 3 2024-06-03T08:34:31.991 INFO:teuthology.orchestra.run.smithi088.stdout: 2024-06-03T08:34:31.991 INFO:teuthology.orchestra.run.smithi088.stdout: bdev debug aio = true 2024-06-03T08:34:31.991 INFO:teuthology.orchestra.run.smithi088.stdout: osd debug misdirected ops = true 2024-06-03T08:34:31.991 INFO:teuthology.orchestra.run.smithi088.stdout: bdev async discard = True 2024-06-03T08:34:31.991 INFO:teuthology.orchestra.run.smithi088.stdout: bdev enable discard = True 2024-06-03T08:34:31.991 INFO:teuthology.orchestra.run.smithi088.stdout: bluestore allocator = stupid 2024-06-03T08:34:31.991 INFO:teuthology.orchestra.run.smithi088.stdout: bluestore block size = 96636764160 2024-06-03T08:34:31.991 INFO:teuthology.orchestra.run.smithi088.stdout: bluestore fsck on mount = True 2024-06-03T08:34:31.991 INFO:teuthology.orchestra.run.smithi088.stdout: bluestore zero block detection = True 2024-06-03T08:34:31.991 INFO:teuthology.orchestra.run.smithi088.stdout: debug bluefs = 1 2024-06-03T08:34:31.992 INFO:teuthology.orchestra.run.smithi088.stdout: debug bluestore = 1 2024-06-03T08:34:31.992 INFO:teuthology.orchestra.run.smithi088.stdout: debug monc = 20 2024-06-03T08:34:31.992 INFO:teuthology.orchestra.run.smithi088.stdout: debug ms = 1 2024-06-03T08:34:31.992 INFO:teuthology.orchestra.run.smithi088.stdout: debug osd = 20 2024-06-03T08:34:31.992 INFO:teuthology.orchestra.run.smithi088.stdout: debug rocksdb = 10 2024-06-03T08:34:31.992 INFO:teuthology.orchestra.run.smithi088.stdout: mon osd backfillfull_ratio = 0.85 2024-06-03T08:34:31.992 INFO:teuthology.orchestra.run.smithi088.stdout: mon osd full ratio = 0.9 2024-06-03T08:34:31.992 INFO:teuthology.orchestra.run.smithi088.stdout: mon osd nearfull ratio = 0.8 2024-06-03T08:34:31.992 INFO:teuthology.orchestra.run.smithi088.stdout: osd class default list = * 2024-06-03T08:34:31.992 INFO:teuthology.orchestra.run.smithi088.stdout: osd class load list = * 2024-06-03T08:34:31.992 INFO:teuthology.orchestra.run.smithi088.stdout: osd client watch timeout = 120 2024-06-03T08:34:31.992 INFO:teuthology.orchestra.run.smithi088.stdout: osd debug verify cached snaps = True 2024-06-03T08:34:31.992 INFO:teuthology.orchestra.run.smithi088.stdout: osd debug verify missing on start = True 2024-06-03T08:34:31.993 INFO:teuthology.orchestra.run.smithi088.stdout: osd failsafe full ratio = 0.95 2024-06-03T08:34:31.993 INFO:teuthology.orchestra.run.smithi088.stdout: osd fast shutdown = False 2024-06-03T08:34:31.993 INFO:teuthology.orchestra.run.smithi088.stdout: osd mclock override recovery settings = True 2024-06-03T08:34:31.993 INFO:teuthology.orchestra.run.smithi088.stdout: osd objectstore = bluestore 2024-06-03T08:34:31.993 INFO:teuthology.orchestra.run.smithi088.stdout: osd op queue = debug_random 2024-06-03T08:34:31.993 INFO:teuthology.orchestra.run.smithi088.stdout: osd op queue cut off = debug_random 2024-06-03T08:34:31.993 INFO:teuthology.orchestra.run.smithi088.stdout: osd shutdown pgref assert = True 2024-06-03T08:34:31.993 INFO:teuthology.orchestra.run.smithi088.stdout: 2024-06-03T08:34:31.993 INFO:teuthology.orchestra.run.smithi088.stdout:[mgr] 2024-06-03T08:34:31.993 INFO:teuthology.orchestra.run.smithi088.stdout: debug ms = 1 2024-06-03T08:34:31.993 INFO:teuthology.orchestra.run.smithi088.stdout: debug mgr = 20 2024-06-03T08:34:31.994 INFO:teuthology.orchestra.run.smithi088.stdout: debug mon = 20 2024-06-03T08:34:31.994 INFO:teuthology.orchestra.run.smithi088.stdout: debug auth = 20 2024-06-03T08:34:31.994 INFO:teuthology.orchestra.run.smithi088.stdout: mon reweight min pgs per osd = 4 2024-06-03T08:34:31.994 INFO:teuthology.orchestra.run.smithi088.stdout: mon reweight min bytes per osd = 10 2024-06-03T08:34:31.994 INFO:teuthology.orchestra.run.smithi088.stdout: mgr/telemetry/nag = false 2024-06-03T08:34:31.994 INFO:teuthology.orchestra.run.smithi088.stdout: 2024-06-03T08:34:31.994 INFO:teuthology.orchestra.run.smithi088.stdout:[mon] 2024-06-03T08:34:31.994 INFO:teuthology.orchestra.run.smithi088.stdout: debug ms = 1 2024-06-03T08:34:31.994 INFO:teuthology.orchestra.run.smithi088.stdout: debug mon = 20 2024-06-03T08:34:31.994 INFO:teuthology.orchestra.run.smithi088.stdout: debug paxos = 20 2024-06-03T08:34:31.994 INFO:teuthology.orchestra.run.smithi088.stdout: debug auth = 20 2024-06-03T08:34:31.994 INFO:teuthology.orchestra.run.smithi088.stdout: mon data avail warn = 5 2024-06-03T08:34:31.994 INFO:teuthology.orchestra.run.smithi088.stdout: mon mgr mkfs grace = 240 2024-06-03T08:34:31.995 INFO:teuthology.orchestra.run.smithi088.stdout: mon reweight min pgs per osd = 4 2024-06-03T08:34:31.995 INFO:teuthology.orchestra.run.smithi088.stdout: mon osd reporter subtree level = osd 2024-06-03T08:34:31.995 INFO:teuthology.orchestra.run.smithi088.stdout: mon osd prime pg temp = true 2024-06-03T08:34:31.995 INFO:teuthology.orchestra.run.smithi088.stdout: mon reweight min bytes per osd = 10 2024-06-03T08:34:31.995 INFO:teuthology.orchestra.run.smithi088.stdout: 2024-06-03T08:34:31.995 INFO:teuthology.orchestra.run.smithi088.stdout: # rotate auth tickets quickly to exercise renewal paths 2024-06-03T08:34:31.995 INFO:teuthology.orchestra.run.smithi088.stdout: auth mon ticket ttl = 660 # 11m 2024-06-03T08:34:31.995 INFO:teuthology.orchestra.run.smithi088.stdout: auth service ticket ttl = 240 # 4m 2024-06-03T08:34:31.995 INFO:teuthology.orchestra.run.smithi088.stdout: 2024-06-03T08:34:31.995 INFO:teuthology.orchestra.run.smithi088.stdout: # don't complain about insecure global_id in the test suite 2024-06-03T08:34:31.995 INFO:teuthology.orchestra.run.smithi088.stdout: mon_warn_on_insecure_global_id_reclaim = false 2024-06-03T08:34:31.995 INFO:teuthology.orchestra.run.smithi088.stdout: mon_warn_on_insecure_global_id_reclaim_allowed = false 2024-06-03T08:34:31.995 INFO:teuthology.orchestra.run.smithi088.stdout: 2024-06-03T08:34:31.996 INFO:teuthology.orchestra.run.smithi088.stdout: # 1m isn't quite enough 2024-06-03T08:34:31.996 INFO:teuthology.orchestra.run.smithi088.stdout: mon_down_mkfs_grace = 2m 2024-06-03T08:34:31.996 INFO:teuthology.orchestra.run.smithi088.stdout: 2024-06-03T08:34:31.996 INFO:teuthology.orchestra.run.smithi088.stdout: mon_warn_on_filestore_osds = false 2024-06-03T08:34:31.996 INFO:teuthology.orchestra.run.smithi088.stdout: mon min osdmap epochs = 50 2024-06-03T08:34:31.996 INFO:teuthology.orchestra.run.smithi088.stdout: mon osd crush smoke test = False 2024-06-03T08:34:31.996 INFO:teuthology.orchestra.run.smithi088.stdout: mon osdmap full prune interval = 2 2024-06-03T08:34:31.996 INFO:teuthology.orchestra.run.smithi088.stdout: mon osdmap full prune min = 15 2024-06-03T08:34:31.996 INFO:teuthology.orchestra.run.smithi088.stdout: mon osdmap full prune txsize = 2 2024-06-03T08:34:31.996 INFO:teuthology.orchestra.run.smithi088.stdout: mon scrub interval = 300 2024-06-03T08:34:31.996 INFO:teuthology.orchestra.run.smithi088.stdout: mon warn on pool no app = False 2024-06-03T08:34:31.996 INFO:teuthology.orchestra.run.smithi088.stdout: paxos service trim min = 10 2024-06-03T08:34:31.996 INFO:teuthology.orchestra.run.smithi088.stdout: 2024-06-03T08:34:31.996 INFO:teuthology.orchestra.run.smithi088.stdout:[client] 2024-06-03T08:34:31.997 INFO:teuthology.orchestra.run.smithi088.stdout: rgw cache enabled = true 2024-06-03T08:34:31.997 INFO:teuthology.orchestra.run.smithi088.stdout: rgw enable ops log = true 2024-06-03T08:34:31.997 INFO:teuthology.orchestra.run.smithi088.stdout: rgw enable usage log = true 2024-06-03T08:34:31.997 INFO:teuthology.orchestra.run.smithi088.stdout: log file = /var/log/ceph/$cluster-$name.$pid.log 2024-06-03T08:34:31.997 INFO:teuthology.orchestra.run.smithi088.stdout: admin socket = /var/run/ceph/$cluster-$name.$pid.asok 2024-06-03T08:34:31.997 INFO:teuthology.orchestra.run.smithi088.stdout: debug monc = 20 2024-06-03T08:34:31.997 INFO:teuthology.orchestra.run.smithi088.stdout: debug ms = 1 2024-06-03T08:34:31.997 INFO:teuthology.orchestra.run.smithi088.stdout: debug objecter = 20 2024-06-03T08:34:31.997 INFO:teuthology.orchestra.run.smithi088.stdout: debug rados = 20 2024-06-03T08:34:31.998 INFO:teuthology.orchestra.run.smithi088.stdout:[mon.a] 2024-06-03T08:34:31.998 INFO:teuthology.orchestra.run.smithi088.stdout:[mon.c] 2024-06-03T08:34:31.998 INFO:teuthology.orchestra.run.smithi088.stdout:[mon.b] 2024-06-03T08:34:31.999 INFO:teuthology.orchestra.run.smithi192.stdout:[global] 2024-06-03T08:34:32.000 INFO:teuthology.orchestra.run.smithi192.stdout: chdir = "" 2024-06-03T08:34:32.000 INFO:teuthology.orchestra.run.smithi192.stdout: pid file = /var/run/ceph/$cluster-$name.pid 2024-06-03T08:34:32.000 INFO:teuthology.orchestra.run.smithi192.stdout: auth supported = cephx 2024-06-03T08:34:32.000 INFO:teuthology.orchestra.run.smithi192.stdout: 2024-06-03T08:34:32.000 INFO:teuthology.orchestra.run.smithi192.stdout: filestore xattr use omap = true 2024-06-03T08:34:32.000 INFO:teuthology.orchestra.run.smithi192.stdout: 2024-06-03T08:34:32.000 INFO:teuthology.orchestra.run.smithi192.stdout: mon clock drift allowed = 1.000 2024-06-03T08:34:32.000 INFO:teuthology.orchestra.run.smithi192.stdout: 2024-06-03T08:34:32.000 INFO:teuthology.orchestra.run.smithi192.stdout: osd crush chooseleaf type = 0 2024-06-03T08:34:32.000 INFO:teuthology.orchestra.run.smithi192.stdout: auth debug = true 2024-06-03T08:34:32.000 INFO:teuthology.orchestra.run.smithi192.stdout: 2024-06-03T08:34:32.000 INFO:teuthology.orchestra.run.smithi192.stdout: ms die on old message = true 2024-06-03T08:34:32.000 INFO:teuthology.orchestra.run.smithi192.stdout: ms die on bug = true 2024-06-03T08:34:32.001 INFO:teuthology.orchestra.run.smithi192.stdout: 2024-06-03T08:34:32.001 INFO:teuthology.orchestra.run.smithi192.stdout: mon max pg per osd = 10000 # >= luminous 2024-06-03T08:34:32.001 INFO:teuthology.orchestra.run.smithi192.stdout: mon pg warn max object skew = 0 2024-06-03T08:34:32.001 INFO:teuthology.orchestra.run.smithi192.stdout: 2024-06-03T08:34:32.001 INFO:teuthology.orchestra.run.smithi192.stdout: # disable pg_autoscaler by default for new pools 2024-06-03T08:34:32.001 INFO:teuthology.orchestra.run.smithi192.stdout: osd_pool_default_pg_autoscale_mode = off 2024-06-03T08:34:32.001 INFO:teuthology.orchestra.run.smithi192.stdout: 2024-06-03T08:34:32.001 INFO:teuthology.orchestra.run.smithi192.stdout: osd pool default size = 2 2024-06-03T08:34:32.001 INFO:teuthology.orchestra.run.smithi192.stdout: 2024-06-03T08:34:32.001 INFO:teuthology.orchestra.run.smithi192.stdout: mon osd allow primary affinity = true 2024-06-03T08:34:32.001 INFO:teuthology.orchestra.run.smithi192.stdout: mon osd allow pg remap = true 2024-06-03T08:34:32.001 INFO:teuthology.orchestra.run.smithi192.stdout: mon warn on legacy crush tunables = false 2024-06-03T08:34:32.001 INFO:teuthology.orchestra.run.smithi192.stdout: mon warn on crush straw calc version zero = false 2024-06-03T08:34:32.001 INFO:teuthology.orchestra.run.smithi192.stdout: mon warn on no sortbitwise = false 2024-06-03T08:34:32.002 INFO:teuthology.orchestra.run.smithi192.stdout: mon warn on osd down out interval zero = false 2024-06-03T08:34:32.002 INFO:teuthology.orchestra.run.smithi192.stdout: mon warn on too few osds = false 2024-06-03T08:34:32.002 INFO:teuthology.orchestra.run.smithi192.stdout: mon_warn_on_pool_pg_num_not_power_of_two = false 2024-06-03T08:34:32.002 INFO:teuthology.orchestra.run.smithi192.stdout: mon_warn_on_pool_no_redundancy = false 2024-06-03T08:34:32.002 INFO:teuthology.orchestra.run.smithi192.stdout: mon_allow_pool_size_one = true 2024-06-03T08:34:32.002 INFO:teuthology.orchestra.run.smithi192.stdout: 2024-06-03T08:34:32.002 INFO:teuthology.orchestra.run.smithi192.stdout: osd pool default erasure code profile = plugin=jerasure technique=reed_sol_van k=2 m=1 crush-failure-domain=osd 2024-06-03T08:34:32.002 INFO:teuthology.orchestra.run.smithi192.stdout: 2024-06-03T08:34:32.002 INFO:teuthology.orchestra.run.smithi192.stdout: osd default data pool replay window = 5 2024-06-03T08:34:32.002 INFO:teuthology.orchestra.run.smithi192.stdout: 2024-06-03T08:34:32.002 INFO:teuthology.orchestra.run.smithi192.stdout: mon allow pool delete = true 2024-06-03T08:34:32.002 INFO:teuthology.orchestra.run.smithi192.stdout: 2024-06-03T08:34:32.002 INFO:teuthology.orchestra.run.smithi192.stdout: mon cluster log file level = debug 2024-06-03T08:34:32.003 INFO:teuthology.orchestra.run.smithi192.stdout: debug asserts on shutdown = true 2024-06-03T08:34:32.003 INFO:teuthology.orchestra.run.smithi192.stdout: mon health detail to clog = false 2024-06-03T08:34:32.003 INFO:teuthology.orchestra.run.smithi192.stdout: mon host = "[v1:172.21.15.88:6789],[v1:172.21.15.88:6790],[v1:172.21.15.192:6789]" 2024-06-03T08:34:32.003 INFO:teuthology.orchestra.run.smithi192.stdout: mon client directed command retry = 5 2024-06-03T08:34:32.003 INFO:teuthology.orchestra.run.smithi192.stdout: mon election default strategy = 3 2024-06-03T08:34:32.003 INFO:teuthology.orchestra.run.smithi192.stdout: ms bind msgr2 = False 2024-06-03T08:34:32.003 INFO:teuthology.orchestra.run.smithi192.stdout: ms inject socket failures = 5000 2024-06-03T08:34:32.003 INFO:teuthology.orchestra.run.smithi192.stdout: ms type = async 2024-06-03T08:34:32.003 INFO:teuthology.orchestra.run.smithi192.stdout: osd heartbeat grace = 80 2024-06-03T08:34:32.003 INFO:teuthology.orchestra.run.smithi192.stdout: osd op thread timeout = 150 2024-06-03T08:34:32.003 INFO:teuthology.orchestra.run.smithi192.stdout: fsid = ec3c3fdd-3ac6-4d7d-9412-683130ed8b04 2024-06-03T08:34:32.003 INFO:teuthology.orchestra.run.smithi192.stdout: 2024-06-03T08:34:32.003 INFO:teuthology.orchestra.run.smithi192.stdout:[osd] 2024-06-03T08:34:32.003 INFO:teuthology.orchestra.run.smithi192.stdout: osd journal size = 100 2024-06-03T08:34:32.004 INFO:teuthology.orchestra.run.smithi192.stdout: 2024-06-03T08:34:32.004 INFO:teuthology.orchestra.run.smithi192.stdout: osd scrub load threshold = 5.0 2024-06-03T08:34:32.004 INFO:teuthology.orchestra.run.smithi192.stdout: osd scrub max interval = 600 2024-06-03T08:34:32.004 INFO:teuthology.orchestra.run.smithi192.stdout: osd mclock profile = high_recovery_ops 2024-06-03T08:34:32.004 INFO:teuthology.orchestra.run.smithi192.stdout: 2024-06-03T08:34:32.004 INFO:teuthology.orchestra.run.smithi192.stdout: osd recover clone overlap = true 2024-06-03T08:34:32.004 INFO:teuthology.orchestra.run.smithi192.stdout: osd recovery max chunk = 1048576 2024-06-03T08:34:32.004 INFO:teuthology.orchestra.run.smithi192.stdout: 2024-06-03T08:34:32.004 INFO:teuthology.orchestra.run.smithi192.stdout: osd debug shutdown = true 2024-06-03T08:34:32.004 INFO:teuthology.orchestra.run.smithi192.stdout: osd debug op order = true 2024-06-03T08:34:32.004 INFO:teuthology.orchestra.run.smithi192.stdout: osd debug verify stray on activate = true 2024-06-03T08:34:32.004 INFO:teuthology.orchestra.run.smithi192.stdout: 2024-06-03T08:34:32.004 INFO:teuthology.orchestra.run.smithi192.stdout: osd open classes on start = true 2024-06-03T08:34:32.005 INFO:teuthology.orchestra.run.smithi192.stdout: osd debug pg log writeout = true 2024-06-03T08:34:32.005 INFO:teuthology.orchestra.run.smithi192.stdout: 2024-06-03T08:34:32.005 INFO:teuthology.orchestra.run.smithi192.stdout: osd deep scrub update digest min age = 30 2024-06-03T08:34:32.005 INFO:teuthology.orchestra.run.smithi192.stdout: 2024-06-03T08:34:32.005 INFO:teuthology.orchestra.run.smithi192.stdout: osd map max advance = 10 2024-06-03T08:34:32.005 INFO:teuthology.orchestra.run.smithi192.stdout: 2024-06-03T08:34:32.005 INFO:teuthology.orchestra.run.smithi192.stdout: journal zero on create = true 2024-06-03T08:34:32.005 INFO:teuthology.orchestra.run.smithi192.stdout: 2024-06-03T08:34:32.005 INFO:teuthology.orchestra.run.smithi192.stdout: filestore ondisk finisher threads = 3 2024-06-03T08:34:32.005 INFO:teuthology.orchestra.run.smithi192.stdout: filestore apply finisher threads = 3 2024-06-03T08:34:32.005 INFO:teuthology.orchestra.run.smithi192.stdout: 2024-06-03T08:34:32.005 INFO:teuthology.orchestra.run.smithi192.stdout: bdev debug aio = true 2024-06-03T08:34:32.005 INFO:teuthology.orchestra.run.smithi192.stdout: osd debug misdirected ops = true 2024-06-03T08:34:32.005 INFO:teuthology.orchestra.run.smithi192.stdout: bdev async discard = True 2024-06-03T08:34:32.006 INFO:teuthology.orchestra.run.smithi192.stdout: bdev enable discard = True 2024-06-03T08:34:32.006 INFO:teuthology.orchestra.run.smithi192.stdout: bluestore allocator = stupid 2024-06-03T08:34:32.006 INFO:teuthology.orchestra.run.smithi192.stdout: bluestore block size = 96636764160 2024-06-03T08:34:32.006 INFO:teuthology.orchestra.run.smithi192.stdout: bluestore fsck on mount = True 2024-06-03T08:34:32.006 INFO:teuthology.orchestra.run.smithi192.stdout: bluestore zero block detection = True 2024-06-03T08:34:32.006 INFO:teuthology.orchestra.run.smithi192.stdout: debug bluefs = 1 2024-06-03T08:34:32.006 INFO:teuthology.orchestra.run.smithi192.stdout: debug bluestore = 1 2024-06-03T08:34:32.006 INFO:teuthology.orchestra.run.smithi192.stdout: debug monc = 20 2024-06-03T08:34:32.006 INFO:teuthology.orchestra.run.smithi192.stdout: debug ms = 1 2024-06-03T08:34:32.006 INFO:teuthology.orchestra.run.smithi192.stdout: debug osd = 20 2024-06-03T08:34:32.006 INFO:teuthology.orchestra.run.smithi192.stdout: debug rocksdb = 10 2024-06-03T08:34:32.006 INFO:teuthology.orchestra.run.smithi192.stdout: mon osd backfillfull_ratio = 0.85 2024-06-03T08:34:32.006 INFO:teuthology.orchestra.run.smithi192.stdout: mon osd full ratio = 0.9 2024-06-03T08:34:32.007 INFO:teuthology.orchestra.run.smithi192.stdout: mon osd nearfull ratio = 0.8 2024-06-03T08:34:32.007 INFO:teuthology.orchestra.run.smithi192.stdout: osd class default list = * 2024-06-03T08:34:32.007 INFO:teuthology.orchestra.run.smithi192.stdout: osd class load list = * 2024-06-03T08:34:32.007 INFO:teuthology.orchestra.run.smithi192.stdout: osd client watch timeout = 120 2024-06-03T08:34:32.007 INFO:teuthology.orchestra.run.smithi192.stdout: osd debug verify cached snaps = True 2024-06-03T08:34:32.007 INFO:teuthology.orchestra.run.smithi192.stdout: osd debug verify missing on start = True 2024-06-03T08:34:32.007 INFO:teuthology.orchestra.run.smithi192.stdout: osd failsafe full ratio = 0.95 2024-06-03T08:34:32.007 INFO:teuthology.orchestra.run.smithi192.stdout: osd fast shutdown = False 2024-06-03T08:34:32.007 INFO:teuthology.orchestra.run.smithi192.stdout: osd mclock override recovery settings = True 2024-06-03T08:34:32.007 INFO:teuthology.orchestra.run.smithi192.stdout: osd objectstore = bluestore 2024-06-03T08:34:32.007 INFO:teuthology.orchestra.run.smithi192.stdout: osd op queue = debug_random 2024-06-03T08:34:32.007 INFO:teuthology.orchestra.run.smithi192.stdout: osd op queue cut off = debug_random 2024-06-03T08:34:32.007 INFO:teuthology.orchestra.run.smithi192.stdout: osd shutdown pgref assert = True 2024-06-03T08:34:32.007 INFO:teuthology.orchestra.run.smithi192.stdout: 2024-06-03T08:34:32.008 INFO:teuthology.orchestra.run.smithi192.stdout:[mgr] 2024-06-03T08:34:32.008 INFO:teuthology.orchestra.run.smithi192.stdout: debug ms = 1 2024-06-03T08:34:32.008 INFO:teuthology.orchestra.run.smithi192.stdout: debug mgr = 20 2024-06-03T08:34:32.008 INFO:teuthology.orchestra.run.smithi192.stdout: debug mon = 20 2024-06-03T08:34:32.008 INFO:teuthology.orchestra.run.smithi192.stdout: debug auth = 20 2024-06-03T08:34:32.008 INFO:teuthology.orchestra.run.smithi192.stdout: mon reweight min pgs per osd = 4 2024-06-03T08:34:32.008 INFO:teuthology.orchestra.run.smithi192.stdout: mon reweight min bytes per osd = 10 2024-06-03T08:34:32.008 INFO:teuthology.orchestra.run.smithi192.stdout: mgr/telemetry/nag = false 2024-06-03T08:34:32.009 INFO:teuthology.orchestra.run.smithi192.stdout: 2024-06-03T08:34:32.009 INFO:teuthology.orchestra.run.smithi192.stdout:[mon] 2024-06-03T08:34:32.009 INFO:teuthology.orchestra.run.smithi192.stdout: debug ms = 1 2024-06-03T08:34:32.009 INFO:teuthology.orchestra.run.smithi192.stdout: debug mon = 20 2024-06-03T08:34:32.009 INFO:teuthology.orchestra.run.smithi192.stdout: debug paxos = 20 2024-06-03T08:34:32.009 INFO:teuthology.orchestra.run.smithi192.stdout: debug auth = 20 2024-06-03T08:34:32.009 INFO:teuthology.orchestra.run.smithi192.stdout: mon data avail warn = 5 2024-06-03T08:34:32.009 INFO:teuthology.orchestra.run.smithi192.stdout: mon mgr mkfs grace = 240 2024-06-03T08:34:32.009 INFO:teuthology.orchestra.run.smithi192.stdout: mon reweight min pgs per osd = 4 2024-06-03T08:34:32.009 INFO:teuthology.orchestra.run.smithi192.stdout: mon osd reporter subtree level = osd 2024-06-03T08:34:32.009 INFO:teuthology.orchestra.run.smithi192.stdout: mon osd prime pg temp = true 2024-06-03T08:34:32.009 INFO:teuthology.orchestra.run.smithi192.stdout: mon reweight min bytes per osd = 10 2024-06-03T08:34:32.009 INFO:teuthology.orchestra.run.smithi192.stdout: 2024-06-03T08:34:32.010 INFO:teuthology.orchestra.run.smithi192.stdout: # rotate auth tickets quickly to exercise renewal paths 2024-06-03T08:34:32.010 INFO:teuthology.orchestra.run.smithi192.stdout: auth mon ticket ttl = 660 # 11m 2024-06-03T08:34:32.010 INFO:teuthology.orchestra.run.smithi192.stdout: auth service ticket ttl = 240 # 4m 2024-06-03T08:34:32.010 INFO:teuthology.orchestra.run.smithi192.stdout: 2024-06-03T08:34:32.010 INFO:teuthology.orchestra.run.smithi192.stdout: # don't complain about insecure global_id in the test suite 2024-06-03T08:34:32.010 INFO:teuthology.orchestra.run.smithi192.stdout: mon_warn_on_insecure_global_id_reclaim = false 2024-06-03T08:34:32.010 INFO:teuthology.orchestra.run.smithi192.stdout: mon_warn_on_insecure_global_id_reclaim_allowed = false 2024-06-03T08:34:32.010 INFO:teuthology.orchestra.run.smithi192.stdout: 2024-06-03T08:34:32.010 INFO:teuthology.orchestra.run.smithi192.stdout: # 1m isn't quite enough 2024-06-03T08:34:32.010 INFO:teuthology.orchestra.run.smithi192.stdout: mon_down_mkfs_grace = 2m 2024-06-03T08:34:32.010 INFO:teuthology.orchestra.run.smithi192.stdout: 2024-06-03T08:34:32.010 INFO:teuthology.orchestra.run.smithi192.stdout: mon_warn_on_filestore_osds = false 2024-06-03T08:34:32.010 INFO:teuthology.orchestra.run.smithi192.stdout: mon min osdmap epochs = 50 2024-06-03T08:34:32.011 INFO:teuthology.orchestra.run.smithi192.stdout: mon osd crush smoke test = False 2024-06-03T08:34:32.011 INFO:teuthology.orchestra.run.smithi192.stdout: mon osdmap full prune interval = 2 2024-06-03T08:34:32.011 INFO:teuthology.orchestra.run.smithi192.stdout: mon osdmap full prune min = 15 2024-06-03T08:34:32.011 INFO:teuthology.orchestra.run.smithi192.stdout: mon osdmap full prune txsize = 2 2024-06-03T08:34:32.011 INFO:teuthology.orchestra.run.smithi192.stdout: mon scrub interval = 300 2024-06-03T08:34:32.011 INFO:teuthology.orchestra.run.smithi192.stdout: mon warn on pool no app = False 2024-06-03T08:34:32.011 INFO:teuthology.orchestra.run.smithi192.stdout: paxos service trim min = 10 2024-06-03T08:34:32.011 INFO:teuthology.orchestra.run.smithi192.stdout: 2024-06-03T08:34:32.011 INFO:teuthology.orchestra.run.smithi192.stdout:[client] 2024-06-03T08:34:32.011 INFO:teuthology.orchestra.run.smithi192.stdout: rgw cache enabled = true 2024-06-03T08:34:32.011 INFO:teuthology.orchestra.run.smithi192.stdout: rgw enable ops log = true 2024-06-03T08:34:32.011 INFO:teuthology.orchestra.run.smithi192.stdout: rgw enable usage log = true 2024-06-03T08:34:32.011 INFO:teuthology.orchestra.run.smithi192.stdout: log file = /var/log/ceph/$cluster-$name.$pid.log 2024-06-03T08:34:32.012 INFO:teuthology.orchestra.run.smithi192.stdout: admin socket = /var/run/ceph/$cluster-$name.$pid.asok 2024-06-03T08:34:32.012 INFO:teuthology.orchestra.run.smithi192.stdout: debug monc = 20 2024-06-03T08:34:32.012 INFO:teuthology.orchestra.run.smithi192.stdout: debug ms = 1 2024-06-03T08:34:32.012 INFO:teuthology.orchestra.run.smithi192.stdout: debug objecter = 20 2024-06-03T08:34:32.012 INFO:teuthology.orchestra.run.smithi192.stdout: debug rados = 20 2024-06-03T08:34:32.012 INFO:teuthology.orchestra.run.smithi192.stdout:[mon.a] 2024-06-03T08:34:32.012 INFO:teuthology.orchestra.run.smithi192.stdout:[mon.c] 2024-06-03T08:34:32.012 INFO:teuthology.orchestra.run.smithi192.stdout:[mon.b] 2024-06-03T08:34:32.012 INFO:tasks.ceph:Creating admin key on mon.a... 2024-06-03T08:34:32.012 DEBUG:teuthology.orchestra.run.smithi088:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-authtool --gen-key --name=client.admin --cap mon 'allow *' --cap osd 'allow *' --cap mds 'allow *' --cap mgr 'allow *' /etc/ceph/ceph.keyring 2024-06-03T08:34:32.095 INFO:tasks.ceph:Copying monmap to all nodes... 2024-06-03T08:34:32.095 DEBUG:teuthology.orchestra.run.smithi088:> set -ex 2024-06-03T08:34:32.095 DEBUG:teuthology.orchestra.run.smithi088:> dd if=/etc/ceph/ceph.keyring of=/dev/stdout 2024-06-03T08:34:32.111 DEBUG:teuthology.orchestra.run.smithi088:> set -ex 2024-06-03T08:34:32.111 DEBUG:teuthology.orchestra.run.smithi088:> dd if=/home/ubuntu/cephtest/ceph.monmap of=/dev/stdout 2024-06-03T08:34:32.166 INFO:tasks.ceph:Sending monmap to node ubuntu@smithi088.front.sepia.ceph.com 2024-06-03T08:34:32.166 DEBUG:teuthology.orchestra.run.smithi088:> set -ex 2024-06-03T08:34:32.166 DEBUG:teuthology.orchestra.run.smithi088:> sudo dd of=/etc/ceph/ceph.keyring 2024-06-03T08:34:32.166 DEBUG:teuthology.orchestra.run.smithi088:> sudo chmod 0644 /etc/ceph/ceph.keyring 2024-06-03T08:34:32.240 DEBUG:teuthology.orchestra.run.smithi088:> set -ex 2024-06-03T08:34:32.240 DEBUG:teuthology.orchestra.run.smithi088:> dd of=/home/ubuntu/cephtest/ceph.monmap 2024-06-03T08:34:32.295 INFO:tasks.ceph:Sending monmap to node ubuntu@smithi192.front.sepia.ceph.com 2024-06-03T08:34:32.296 DEBUG:teuthology.orchestra.run.smithi192:> set -ex 2024-06-03T08:34:32.296 DEBUG:teuthology.orchestra.run.smithi192:> sudo dd of=/etc/ceph/ceph.keyring 2024-06-03T08:34:32.296 DEBUG:teuthology.orchestra.run.smithi192:> sudo chmod 0644 /etc/ceph/ceph.keyring 2024-06-03T08:34:32.331 DEBUG:teuthology.orchestra.run.smithi192:> set -ex 2024-06-03T08:34:32.331 DEBUG:teuthology.orchestra.run.smithi192:> dd of=/home/ubuntu/cephtest/ceph.monmap 2024-06-03T08:34:32.387 INFO:tasks.ceph:Setting up mon nodes... 2024-06-03T08:34:32.387 INFO:tasks.ceph:Setting up mgr nodes... 2024-06-03T08:34:32.387 DEBUG:teuthology.orchestra.run.smithi088:> sudo mkdir -p /var/lib/ceph/mgr/ceph-y && sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-authtool --create-keyring --gen-key --name=mgr.y /var/lib/ceph/mgr/ceph-y/keyring 2024-06-03T08:34:32.438 INFO:teuthology.orchestra.run.smithi088.stdout:creating /var/lib/ceph/mgr/ceph-y/keyring 2024-06-03T08:34:32.441 DEBUG:teuthology.orchestra.run.smithi192:> sudo mkdir -p /var/lib/ceph/mgr/ceph-x && sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-authtool --create-keyring --gen-key --name=mgr.x /var/lib/ceph/mgr/ceph-x/keyring 2024-06-03T08:34:32.521 INFO:teuthology.orchestra.run.smithi192.stdout:creating /var/lib/ceph/mgr/ceph-x/keyring 2024-06-03T08:34:32.524 INFO:tasks.ceph:Setting up mds nodes... 2024-06-03T08:34:32.524 INFO:tasks.ceph_client:Setting up client nodes... 2024-06-03T08:34:32.524 DEBUG:teuthology.orchestra.run.smithi088:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-authtool --create-keyring --gen-key --name=client.0 /etc/ceph/ceph.client.0.keyring && sudo chmod 0644 /etc/ceph/ceph.client.0.keyring 2024-06-03T08:34:32.565 INFO:teuthology.orchestra.run.smithi088.stdout:creating /etc/ceph/ceph.client.0.keyring 2024-06-03T08:34:32.577 DEBUG:teuthology.orchestra.run.smithi192:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-authtool --create-keyring --gen-key --name=client.1 /etc/ceph/ceph.client.1.keyring && sudo chmod 0644 /etc/ceph/ceph.client.1.keyring 2024-06-03T08:34:32.617 INFO:teuthology.orchestra.run.smithi192.stdout:creating /etc/ceph/ceph.client.1.keyring 2024-06-03T08:34:32.630 INFO:tasks.ceph:Running mkfs on osd nodes... 2024-06-03T08:34:32.630 INFO:tasks.ceph:ctx.disk_config.remote_to_roles_to_dev: {Remote(name='ubuntu@smithi088.front.sepia.ceph.com'): {'osd.0': '/dev/vg_nvme/lv_1', 'osd.1': '/dev/vg_nvme/lv_2', 'osd.2': '/dev/vg_nvme/lv_3', 'osd.3': '/dev/vg_nvme/lv_4'}, Remote(name='ubuntu@smithi192.front.sepia.ceph.com'): {'osd.4': '/dev/vg_nvme/lv_1', 'osd.5': '/dev/vg_nvme/lv_2', 'osd.6': '/dev/vg_nvme/lv_3', 'osd.7': '/dev/vg_nvme/lv_4'}} 2024-06-03T08:34:32.630 DEBUG:teuthology.orchestra.run.smithi088:> sudo mkdir -p /var/lib/ceph/osd/ceph-0 2024-06-03T08:34:32.654 INFO:tasks.ceph:roles_to_devs: {'osd.0': '/dev/vg_nvme/lv_1', 'osd.1': '/dev/vg_nvme/lv_2', 'osd.2': '/dev/vg_nvme/lv_3', 'osd.3': '/dev/vg_nvme/lv_4'} 2024-06-03T08:34:32.654 INFO:tasks.ceph:role: osd.0 2024-06-03T08:34:32.655 INFO:tasks.ceph:['mkfs.xfs', '-f', '-i', 'size=2048'] on /dev/vg_nvme/lv_1 on ubuntu@smithi088.front.sepia.ceph.com 2024-06-03T08:34:32.655 DEBUG:teuthology.orchestra.run.smithi088:> yes | sudo mkfs.xfs -f -i size=2048 /dev/vg_nvme/lv_1 2024-06-03T08:34:32.726 INFO:teuthology.orchestra.run.smithi088.stdout:meta-data=/dev/vg_nvme/lv_1 isize=2048 agcount=4, agsize=5859072 blks 2024-06-03T08:34:32.726 INFO:teuthology.orchestra.run.smithi088.stdout: = sectsz=512 attr=2, projid32bit=1 2024-06-03T08:34:32.726 INFO:teuthology.orchestra.run.smithi088.stdout: = crc=1 finobt=1, sparse=1, rmapbt=0 2024-06-03T08:34:32.726 INFO:teuthology.orchestra.run.smithi088.stdout: = reflink=1 bigtime=1 inobtcount=1 nrext64=0 2024-06-03T08:34:32.726 INFO:teuthology.orchestra.run.smithi088.stdout:data = bsize=4096 blocks=23436288, imaxpct=25 2024-06-03T08:34:32.726 INFO:teuthology.orchestra.run.smithi088.stdout: = sunit=0 swidth=0 blks 2024-06-03T08:34:32.726 INFO:teuthology.orchestra.run.smithi088.stdout:naming =version 2 bsize=4096 ascii-ci=0, ftype=1 2024-06-03T08:34:32.726 INFO:teuthology.orchestra.run.smithi088.stdout:log =internal log bsize=4096 blocks=16384, version=2 2024-06-03T08:34:32.726 INFO:teuthology.orchestra.run.smithi088.stdout: = sectsz=512 sunit=0 blks, lazy-count=1 2024-06-03T08:34:32.726 INFO:teuthology.orchestra.run.smithi088.stdout:realtime =none extsz=4096 blocks=0, rtextents=0 2024-06-03T08:34:33.224 INFO:teuthology.orchestra.run.smithi088.stdout:Discarding blocks...Done. 2024-06-03T08:34:33.226 INFO:tasks.ceph:mount /dev/vg_nvme/lv_1 on ubuntu@smithi088.front.sepia.ceph.com -o noatime 2024-06-03T08:34:33.226 DEBUG:teuthology.orchestra.run.smithi088:> sudo mount -t xfs -o noatime /dev/vg_nvme/lv_1 /var/lib/ceph/osd/ceph-0 2024-06-03T08:34:33.345 DEBUG:teuthology.orchestra.run.smithi088:> sudo /sbin/restorecon /var/lib/ceph/osd/ceph-0 2024-06-03T08:34:33.377 DEBUG:teuthology.orchestra.run.smithi088:> sudo mkdir -p /var/lib/ceph/osd/ceph-1 2024-06-03T08:34:33.444 INFO:tasks.ceph:roles_to_devs: {'osd.0': '/dev/vg_nvme/lv_1', 'osd.1': '/dev/vg_nvme/lv_2', 'osd.2': '/dev/vg_nvme/lv_3', 'osd.3': '/dev/vg_nvme/lv_4'} 2024-06-03T08:34:33.444 INFO:tasks.ceph:role: osd.1 2024-06-03T08:34:33.444 INFO:tasks.ceph:['mkfs.xfs', '-f', '-i', 'size=2048'] on /dev/vg_nvme/lv_2 on ubuntu@smithi088.front.sepia.ceph.com 2024-06-03T08:34:33.444 DEBUG:teuthology.orchestra.run.smithi088:> yes | sudo mkfs.xfs -f -i size=2048 /dev/vg_nvme/lv_2 2024-06-03T08:34:33.517 INFO:teuthology.orchestra.run.smithi088.stdout:meta-data=/dev/vg_nvme/lv_2 isize=2048 agcount=4, agsize=5859072 blks 2024-06-03T08:34:33.518 INFO:teuthology.orchestra.run.smithi088.stdout: = sectsz=512 attr=2, projid32bit=1 2024-06-03T08:34:33.518 INFO:teuthology.orchestra.run.smithi088.stdout: = crc=1 finobt=1, sparse=1, rmapbt=0 2024-06-03T08:34:33.518 INFO:teuthology.orchestra.run.smithi088.stdout: = reflink=1 bigtime=1 inobtcount=1 nrext64=0 2024-06-03T08:34:33.518 INFO:teuthology.orchestra.run.smithi088.stdout:data = bsize=4096 blocks=23436288, imaxpct=25 2024-06-03T08:34:33.518 INFO:teuthology.orchestra.run.smithi088.stdout: = sunit=0 swidth=0 blks 2024-06-03T08:34:33.518 INFO:teuthology.orchestra.run.smithi088.stdout:naming =version 2 bsize=4096 ascii-ci=0, ftype=1 2024-06-03T08:34:33.518 INFO:teuthology.orchestra.run.smithi088.stdout:log =internal log bsize=4096 blocks=16384, version=2 2024-06-03T08:34:33.518 INFO:teuthology.orchestra.run.smithi088.stdout: = sectsz=512 sunit=0 blks, lazy-count=1 2024-06-03T08:34:33.518 INFO:teuthology.orchestra.run.smithi088.stdout:realtime =none extsz=4096 blocks=0, rtextents=0 2024-06-03T08:34:33.977 INFO:teuthology.orchestra.run.smithi088.stdout:Discarding blocks...Done. 2024-06-03T08:34:33.979 INFO:tasks.ceph:mount /dev/vg_nvme/lv_2 on ubuntu@smithi088.front.sepia.ceph.com -o noatime 2024-06-03T08:34:33.979 DEBUG:teuthology.orchestra.run.smithi088:> sudo mount -t xfs -o noatime /dev/vg_nvme/lv_2 /var/lib/ceph/osd/ceph-1 2024-06-03T08:34:34.059 DEBUG:teuthology.orchestra.run.smithi088:> sudo /sbin/restorecon /var/lib/ceph/osd/ceph-1 2024-06-03T08:34:34.130 DEBUG:teuthology.orchestra.run.smithi088:> sudo mkdir -p /var/lib/ceph/osd/ceph-2 2024-06-03T08:34:34.195 INFO:tasks.ceph:roles_to_devs: {'osd.0': '/dev/vg_nvme/lv_1', 'osd.1': '/dev/vg_nvme/lv_2', 'osd.2': '/dev/vg_nvme/lv_3', 'osd.3': '/dev/vg_nvme/lv_4'} 2024-06-03T08:34:34.195 INFO:tasks.ceph:role: osd.2 2024-06-03T08:34:34.195 INFO:tasks.ceph:['mkfs.xfs', '-f', '-i', 'size=2048'] on /dev/vg_nvme/lv_3 on ubuntu@smithi088.front.sepia.ceph.com 2024-06-03T08:34:34.196 DEBUG:teuthology.orchestra.run.smithi088:> yes | sudo mkfs.xfs -f -i size=2048 /dev/vg_nvme/lv_3 2024-06-03T08:34:34.268 INFO:teuthology.orchestra.run.smithi088.stdout:meta-data=/dev/vg_nvme/lv_3 isize=2048 agcount=4, agsize=5859072 blks 2024-06-03T08:34:34.268 INFO:teuthology.orchestra.run.smithi088.stdout: = sectsz=512 attr=2, projid32bit=1 2024-06-03T08:34:34.268 INFO:teuthology.orchestra.run.smithi088.stdout: = crc=1 finobt=1, sparse=1, rmapbt=0 2024-06-03T08:34:34.268 INFO:teuthology.orchestra.run.smithi088.stdout: = reflink=1 bigtime=1 inobtcount=1 nrext64=0 2024-06-03T08:34:34.268 INFO:teuthology.orchestra.run.smithi088.stdout:data = bsize=4096 blocks=23436288, imaxpct=25 2024-06-03T08:34:34.268 INFO:teuthology.orchestra.run.smithi088.stdout: = sunit=0 swidth=0 blks 2024-06-03T08:34:34.268 INFO:teuthology.orchestra.run.smithi088.stdout:naming =version 2 bsize=4096 ascii-ci=0, ftype=1 2024-06-03T08:34:34.268 INFO:teuthology.orchestra.run.smithi088.stdout:log =internal log bsize=4096 blocks=16384, version=2 2024-06-03T08:34:34.268 INFO:teuthology.orchestra.run.smithi088.stdout: = sectsz=512 sunit=0 blks, lazy-count=1 2024-06-03T08:34:34.269 INFO:teuthology.orchestra.run.smithi088.stdout:realtime =none extsz=4096 blocks=0, rtextents=0 2024-06-03T08:34:34.769 INFO:teuthology.orchestra.run.smithi088.stdout:Discarding blocks...Done. 2024-06-03T08:34:34.771 INFO:tasks.ceph:mount /dev/vg_nvme/lv_3 on ubuntu@smithi088.front.sepia.ceph.com -o noatime 2024-06-03T08:34:34.771 DEBUG:teuthology.orchestra.run.smithi088:> sudo mount -t xfs -o noatime /dev/vg_nvme/lv_3 /var/lib/ceph/osd/ceph-2 2024-06-03T08:34:34.904 DEBUG:teuthology.orchestra.run.smithi088:> sudo /sbin/restorecon /var/lib/ceph/osd/ceph-2 2024-06-03T08:34:34.934 DEBUG:teuthology.orchestra.run.smithi088:> sudo mkdir -p /var/lib/ceph/osd/ceph-3 2024-06-03T08:34:34.999 INFO:tasks.ceph:roles_to_devs: {'osd.0': '/dev/vg_nvme/lv_1', 'osd.1': '/dev/vg_nvme/lv_2', 'osd.2': '/dev/vg_nvme/lv_3', 'osd.3': '/dev/vg_nvme/lv_4'} 2024-06-03T08:34:34.999 INFO:tasks.ceph:role: osd.3 2024-06-03T08:34:34.999 INFO:tasks.ceph:['mkfs.xfs', '-f', '-i', 'size=2048'] on /dev/vg_nvme/lv_4 on ubuntu@smithi088.front.sepia.ceph.com 2024-06-03T08:34:34.999 DEBUG:teuthology.orchestra.run.smithi088:> yes | sudo mkfs.xfs -f -i size=2048 /dev/vg_nvme/lv_4 2024-06-03T08:34:35.071 INFO:teuthology.orchestra.run.smithi088.stdout:meta-data=/dev/vg_nvme/lv_4 isize=2048 agcount=4, agsize=5859072 blks 2024-06-03T08:34:35.072 INFO:teuthology.orchestra.run.smithi088.stdout: = sectsz=512 attr=2, projid32bit=1 2024-06-03T08:34:35.072 INFO:teuthology.orchestra.run.smithi088.stdout: = crc=1 finobt=1, sparse=1, rmapbt=0 2024-06-03T08:34:35.072 INFO:teuthology.orchestra.run.smithi088.stdout: = reflink=1 bigtime=1 inobtcount=1 nrext64=0 2024-06-03T08:34:35.072 INFO:teuthology.orchestra.run.smithi088.stdout:data = bsize=4096 blocks=23436288, imaxpct=25 2024-06-03T08:34:35.072 INFO:teuthology.orchestra.run.smithi088.stdout: = sunit=0 swidth=0 blks 2024-06-03T08:34:35.072 INFO:teuthology.orchestra.run.smithi088.stdout:naming =version 2 bsize=4096 ascii-ci=0, ftype=1 2024-06-03T08:34:35.072 INFO:teuthology.orchestra.run.smithi088.stdout:log =internal log bsize=4096 blocks=16384, version=2 2024-06-03T08:34:35.072 INFO:teuthology.orchestra.run.smithi088.stdout: = sectsz=512 sunit=0 blks, lazy-count=1 2024-06-03T08:34:35.072 INFO:teuthology.orchestra.run.smithi088.stdout:realtime =none extsz=4096 blocks=0, rtextents=0 2024-06-03T08:34:35.569 INFO:teuthology.orchestra.run.smithi088.stdout:Discarding blocks...Done. 2024-06-03T08:34:35.571 INFO:tasks.ceph:mount /dev/vg_nvme/lv_4 on ubuntu@smithi088.front.sepia.ceph.com -o noatime 2024-06-03T08:34:35.571 DEBUG:teuthology.orchestra.run.smithi088:> sudo mount -t xfs -o noatime /dev/vg_nvme/lv_4 /var/lib/ceph/osd/ceph-3 2024-06-03T08:34:35.679 DEBUG:teuthology.orchestra.run.smithi088:> sudo /sbin/restorecon /var/lib/ceph/osd/ceph-3 2024-06-03T08:34:35.710 DEBUG:teuthology.orchestra.run.smithi088:> sudo MALLOC_CHECK_=3 adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-osd --no-mon-config --cluster ceph --mkfs --mkkey -i 0 --monmap /home/ubuntu/cephtest/ceph.monmap 2024-06-03T08:34:35.841 INFO:teuthology.orchestra.run.smithi088.stderr:2024-06-03T08:34:35.840+0000 7fbb30815640 -1 auth: error reading file: /var/lib/ceph/osd/ceph-0/keyring: can't open /var/lib/ceph/osd/ceph-0/keyring: (2) No such file or directory 2024-06-03T08:34:35.841 INFO:teuthology.orchestra.run.smithi088.stderr:2024-06-03T08:34:35.840+0000 7fbb30815640 -1 created new key in keyring /var/lib/ceph/osd/ceph-0/keyring 2024-06-03T08:34:35.841 INFO:teuthology.orchestra.run.smithi088.stderr:2024-06-03T08:34:35.840+0000 7fbb30815640 -1 bluestore(/var/lib/ceph/osd/ceph-0/block) _read_bdev_label failed to open /var/lib/ceph/osd/ceph-0/block: (2) No such file or directory 2024-06-03T08:34:35.841 INFO:teuthology.orchestra.run.smithi088.stderr:2024-06-03T08:34:35.840+0000 7fbb30815640 -1 bluestore(/var/lib/ceph/osd/ceph-0/block) _read_bdev_label failed to open /var/lib/ceph/osd/ceph-0/block: (2) No such file or directory 2024-06-03T08:34:35.841 INFO:teuthology.orchestra.run.smithi088.stderr:2024-06-03T08:34:35.840+0000 7fbb30815640 -1 bluestore(/var/lib/ceph/osd/ceph-0/block) _read_bdev_label failed to open /var/lib/ceph/osd/ceph-0/block: (2) No such file or directory 2024-06-03T08:34:35.841 INFO:teuthology.orchestra.run.smithi088.stderr:2024-06-03T08:34:35.841+0000 7fbb30815640 -1 bluestore(/var/lib/ceph/osd/ceph-0) _read_fsid unparsable uuid 2024-06-03T08:34:38.696 DEBUG:teuthology.orchestra.run.smithi088:> sudo chown -R ceph:ceph /var/lib/ceph/osd/ceph-0 2024-06-03T08:34:38.726 DEBUG:teuthology.orchestra.run.smithi088:> sudo MALLOC_CHECK_=3 adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-osd --no-mon-config --cluster ceph --mkfs --mkkey -i 1 --monmap /home/ubuntu/cephtest/ceph.monmap 2024-06-03T08:34:38.816 INFO:teuthology.orchestra.run.smithi088.stderr:2024-06-03T08:34:38.816+0000 7f9b93412640 -1 auth: error reading file: /var/lib/ceph/osd/ceph-1/keyring: can't open /var/lib/ceph/osd/ceph-1/keyring: (2) No such file or directory 2024-06-03T08:34:38.817 INFO:teuthology.orchestra.run.smithi088.stderr:2024-06-03T08:34:38.816+0000 7f9b93412640 -1 created new key in keyring /var/lib/ceph/osd/ceph-1/keyring 2024-06-03T08:34:38.817 INFO:teuthology.orchestra.run.smithi088.stderr:2024-06-03T08:34:38.816+0000 7f9b93412640 -1 bluestore(/var/lib/ceph/osd/ceph-1/block) _read_bdev_label failed to open /var/lib/ceph/osd/ceph-1/block: (2) No such file or directory 2024-06-03T08:34:38.817 INFO:teuthology.orchestra.run.smithi088.stderr:2024-06-03T08:34:38.816+0000 7f9b93412640 -1 bluestore(/var/lib/ceph/osd/ceph-1/block) _read_bdev_label failed to open /var/lib/ceph/osd/ceph-1/block: (2) No such file or directory 2024-06-03T08:34:38.817 INFO:teuthology.orchestra.run.smithi088.stderr:2024-06-03T08:34:38.816+0000 7f9b93412640 -1 bluestore(/var/lib/ceph/osd/ceph-1/block) _read_bdev_label failed to open /var/lib/ceph/osd/ceph-1/block: (2) No such file or directory 2024-06-03T08:34:38.817 INFO:teuthology.orchestra.run.smithi088.stderr:2024-06-03T08:34:38.817+0000 7f9b93412640 -1 bluestore(/var/lib/ceph/osd/ceph-1) _read_fsid unparsable uuid 2024-06-03T08:34:41.660 DEBUG:teuthology.orchestra.run.smithi088:> sudo chown -R ceph:ceph /var/lib/ceph/osd/ceph-1 2024-06-03T08:34:41.690 DEBUG:teuthology.orchestra.run.smithi088:> sudo MALLOC_CHECK_=3 adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-osd --no-mon-config --cluster ceph --mkfs --mkkey -i 2 --monmap /home/ubuntu/cephtest/ceph.monmap 2024-06-03T08:34:41.780 INFO:teuthology.orchestra.run.smithi088.stderr:2024-06-03T08:34:41.779+0000 7fabae508640 -1 auth: error reading file: /var/lib/ceph/osd/ceph-2/keyring: can't open /var/lib/ceph/osd/ceph-2/keyring: (2) No such file or directory 2024-06-03T08:34:41.780 INFO:teuthology.orchestra.run.smithi088.stderr:2024-06-03T08:34:41.779+0000 7fabae508640 -1 created new key in keyring /var/lib/ceph/osd/ceph-2/keyring 2024-06-03T08:34:41.780 INFO:teuthology.orchestra.run.smithi088.stderr:2024-06-03T08:34:41.779+0000 7fabae508640 -1 bluestore(/var/lib/ceph/osd/ceph-2/block) _read_bdev_label failed to open /var/lib/ceph/osd/ceph-2/block: (2) No such file or directory 2024-06-03T08:34:41.780 INFO:teuthology.orchestra.run.smithi088.stderr:2024-06-03T08:34:41.779+0000 7fabae508640 -1 bluestore(/var/lib/ceph/osd/ceph-2/block) _read_bdev_label failed to open /var/lib/ceph/osd/ceph-2/block: (2) No such file or directory 2024-06-03T08:34:41.780 INFO:teuthology.orchestra.run.smithi088.stderr:2024-06-03T08:34:41.779+0000 7fabae508640 -1 bluestore(/var/lib/ceph/osd/ceph-2/block) _read_bdev_label failed to open /var/lib/ceph/osd/ceph-2/block: (2) No such file or directory 2024-06-03T08:34:41.781 INFO:teuthology.orchestra.run.smithi088.stderr:2024-06-03T08:34:41.780+0000 7fabae508640 -1 bluestore(/var/lib/ceph/osd/ceph-2) _read_fsid unparsable uuid 2024-06-03T08:34:44.622 DEBUG:teuthology.orchestra.run.smithi088:> sudo chown -R ceph:ceph /var/lib/ceph/osd/ceph-2 2024-06-03T08:34:44.650 DEBUG:teuthology.orchestra.run.smithi088:> sudo MALLOC_CHECK_=3 adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-osd --no-mon-config --cluster ceph --mkfs --mkkey -i 3 --monmap /home/ubuntu/cephtest/ceph.monmap 2024-06-03T08:34:44.739 INFO:teuthology.orchestra.run.smithi088.stderr:2024-06-03T08:34:44.738+0000 7fc096c0c640 -1 auth: error reading file: /var/lib/ceph/osd/ceph-3/keyring: can't open /var/lib/ceph/osd/ceph-3/keyring: (2) No such file or directory 2024-06-03T08:34:44.740 INFO:teuthology.orchestra.run.smithi088.stderr:2024-06-03T08:34:44.738+0000 7fc096c0c640 -1 created new key in keyring /var/lib/ceph/osd/ceph-3/keyring 2024-06-03T08:34:44.740 INFO:teuthology.orchestra.run.smithi088.stderr:2024-06-03T08:34:44.738+0000 7fc096c0c640 -1 bluestore(/var/lib/ceph/osd/ceph-3/block) _read_bdev_label failed to open /var/lib/ceph/osd/ceph-3/block: (2) No such file or directory 2024-06-03T08:34:44.740 INFO:teuthology.orchestra.run.smithi088.stderr:2024-06-03T08:34:44.738+0000 7fc096c0c640 -1 bluestore(/var/lib/ceph/osd/ceph-3/block) _read_bdev_label failed to open /var/lib/ceph/osd/ceph-3/block: (2) No such file or directory 2024-06-03T08:34:44.740 INFO:teuthology.orchestra.run.smithi088.stderr:2024-06-03T08:34:44.738+0000 7fc096c0c640 -1 bluestore(/var/lib/ceph/osd/ceph-3/block) _read_bdev_label failed to open /var/lib/ceph/osd/ceph-3/block: (2) No such file or directory 2024-06-03T08:34:44.740 INFO:teuthology.orchestra.run.smithi088.stderr:2024-06-03T08:34:44.739+0000 7fc096c0c640 -1 bluestore(/var/lib/ceph/osd/ceph-3) _read_fsid unparsable uuid 2024-06-03T08:34:47.582 DEBUG:teuthology.orchestra.run.smithi088:> sudo chown -R ceph:ceph /var/lib/ceph/osd/ceph-3 2024-06-03T08:34:47.611 DEBUG:teuthology.orchestra.run.smithi192:> sudo mkdir -p /var/lib/ceph/osd/ceph-4 2024-06-03T08:34:47.641 INFO:tasks.ceph:roles_to_devs: {'osd.4': '/dev/vg_nvme/lv_1', 'osd.5': '/dev/vg_nvme/lv_2', 'osd.6': '/dev/vg_nvme/lv_3', 'osd.7': '/dev/vg_nvme/lv_4'} 2024-06-03T08:34:47.641 INFO:tasks.ceph:role: osd.4 2024-06-03T08:34:47.642 INFO:tasks.ceph:['mkfs.xfs', '-f', '-i', 'size=2048'] on /dev/vg_nvme/lv_1 on ubuntu@smithi192.front.sepia.ceph.com 2024-06-03T08:34:47.642 DEBUG:teuthology.orchestra.run.smithi192:> yes | sudo mkfs.xfs -f -i size=2048 /dev/vg_nvme/lv_1 2024-06-03T08:34:47.717 INFO:teuthology.orchestra.run.smithi192.stdout:meta-data=/dev/vg_nvme/lv_1 isize=2048 agcount=4, agsize=5859072 blks 2024-06-03T08:34:47.718 INFO:teuthology.orchestra.run.smithi192.stdout: = sectsz=512 attr=2, projid32bit=1 2024-06-03T08:34:47.718 INFO:teuthology.orchestra.run.smithi192.stdout: = crc=1 finobt=1, sparse=1, rmapbt=0 2024-06-03T08:34:47.718 INFO:teuthology.orchestra.run.smithi192.stdout: = reflink=1 bigtime=1 inobtcount=1 nrext64=0 2024-06-03T08:34:47.718 INFO:teuthology.orchestra.run.smithi192.stdout:data = bsize=4096 blocks=23436288, imaxpct=25 2024-06-03T08:34:47.718 INFO:teuthology.orchestra.run.smithi192.stdout: = sunit=0 swidth=0 blks 2024-06-03T08:34:47.718 INFO:teuthology.orchestra.run.smithi192.stdout:naming =version 2 bsize=4096 ascii-ci=0, ftype=1 2024-06-03T08:34:47.719 INFO:teuthology.orchestra.run.smithi192.stdout:log =internal log bsize=4096 blocks=16384, version=2 2024-06-03T08:34:47.719 INFO:teuthology.orchestra.run.smithi192.stdout: = sectsz=512 sunit=0 blks, lazy-count=1 2024-06-03T08:34:47.719 INFO:teuthology.orchestra.run.smithi192.stdout:realtime =none extsz=4096 blocks=0, rtextents=0 2024-06-03T08:34:48.222 INFO:teuthology.orchestra.run.smithi192.stdout:Discarding blocks...Done. 2024-06-03T08:34:48.225 INFO:tasks.ceph:mount /dev/vg_nvme/lv_1 on ubuntu@smithi192.front.sepia.ceph.com -o noatime 2024-06-03T08:34:48.225 DEBUG:teuthology.orchestra.run.smithi192:> sudo mount -t xfs -o noatime /dev/vg_nvme/lv_1 /var/lib/ceph/osd/ceph-4 2024-06-03T08:34:48.333 DEBUG:teuthology.orchestra.run.smithi192:> sudo /sbin/restorecon /var/lib/ceph/osd/ceph-4 2024-06-03T08:34:48.367 DEBUG:teuthology.orchestra.run.smithi192:> sudo mkdir -p /var/lib/ceph/osd/ceph-5 2024-06-03T08:34:48.434 INFO:tasks.ceph:roles_to_devs: {'osd.4': '/dev/vg_nvme/lv_1', 'osd.5': '/dev/vg_nvme/lv_2', 'osd.6': '/dev/vg_nvme/lv_3', 'osd.7': '/dev/vg_nvme/lv_4'} 2024-06-03T08:34:48.434 INFO:tasks.ceph:role: osd.5 2024-06-03T08:34:48.434 INFO:tasks.ceph:['mkfs.xfs', '-f', '-i', 'size=2048'] on /dev/vg_nvme/lv_2 on ubuntu@smithi192.front.sepia.ceph.com 2024-06-03T08:34:48.434 DEBUG:teuthology.orchestra.run.smithi192:> yes | sudo mkfs.xfs -f -i size=2048 /dev/vg_nvme/lv_2 2024-06-03T08:34:48.507 INFO:teuthology.orchestra.run.smithi192.stdout:meta-data=/dev/vg_nvme/lv_2 isize=2048 agcount=4, agsize=5859072 blks 2024-06-03T08:34:48.508 INFO:teuthology.orchestra.run.smithi192.stdout: = sectsz=512 attr=2, projid32bit=1 2024-06-03T08:34:48.508 INFO:teuthology.orchestra.run.smithi192.stdout: = crc=1 finobt=1, sparse=1, rmapbt=0 2024-06-03T08:34:48.508 INFO:teuthology.orchestra.run.smithi192.stdout: = reflink=1 bigtime=1 inobtcount=1 nrext64=0 2024-06-03T08:34:48.508 INFO:teuthology.orchestra.run.smithi192.stdout:data = bsize=4096 blocks=23436288, imaxpct=25 2024-06-03T08:34:48.508 INFO:teuthology.orchestra.run.smithi192.stdout: = sunit=0 swidth=0 blks 2024-06-03T08:34:48.508 INFO:teuthology.orchestra.run.smithi192.stdout:naming =version 2 bsize=4096 ascii-ci=0, ftype=1 2024-06-03T08:34:48.508 INFO:teuthology.orchestra.run.smithi192.stdout:log =internal log bsize=4096 blocks=16384, version=2 2024-06-03T08:34:48.508 INFO:teuthology.orchestra.run.smithi192.stdout: = sectsz=512 sunit=0 blks, lazy-count=1 2024-06-03T08:34:48.508 INFO:teuthology.orchestra.run.smithi192.stdout:realtime =none extsz=4096 blocks=0, rtextents=0 2024-06-03T08:34:48.970 INFO:teuthology.orchestra.run.smithi192.stdout:Discarding blocks...Done. 2024-06-03T08:34:48.972 INFO:tasks.ceph:mount /dev/vg_nvme/lv_2 on ubuntu@smithi192.front.sepia.ceph.com -o noatime 2024-06-03T08:34:48.973 DEBUG:teuthology.orchestra.run.smithi192:> sudo mount -t xfs -o noatime /dev/vg_nvme/lv_2 /var/lib/ceph/osd/ceph-5 2024-06-03T08:34:49.075 DEBUG:teuthology.orchestra.run.smithi192:> sudo /sbin/restorecon /var/lib/ceph/osd/ceph-5 2024-06-03T08:34:49.106 DEBUG:teuthology.orchestra.run.smithi192:> sudo mkdir -p /var/lib/ceph/osd/ceph-6 2024-06-03T08:34:49.172 INFO:tasks.ceph:roles_to_devs: {'osd.4': '/dev/vg_nvme/lv_1', 'osd.5': '/dev/vg_nvme/lv_2', 'osd.6': '/dev/vg_nvme/lv_3', 'osd.7': '/dev/vg_nvme/lv_4'} 2024-06-03T08:34:49.172 INFO:tasks.ceph:role: osd.6 2024-06-03T08:34:49.172 INFO:tasks.ceph:['mkfs.xfs', '-f', '-i', 'size=2048'] on /dev/vg_nvme/lv_3 on ubuntu@smithi192.front.sepia.ceph.com 2024-06-03T08:34:49.172 DEBUG:teuthology.orchestra.run.smithi192:> yes | sudo mkfs.xfs -f -i size=2048 /dev/vg_nvme/lv_3 2024-06-03T08:34:49.245 INFO:teuthology.orchestra.run.smithi192.stdout:meta-data=/dev/vg_nvme/lv_3 isize=2048 agcount=4, agsize=5859072 blks 2024-06-03T08:34:49.245 INFO:teuthology.orchestra.run.smithi192.stdout: = sectsz=512 attr=2, projid32bit=1 2024-06-03T08:34:49.245 INFO:teuthology.orchestra.run.smithi192.stdout: = crc=1 finobt=1, sparse=1, rmapbt=0 2024-06-03T08:34:49.245 INFO:teuthology.orchestra.run.smithi192.stdout: = reflink=1 bigtime=1 inobtcount=1 nrext64=0 2024-06-03T08:34:49.245 INFO:teuthology.orchestra.run.smithi192.stdout:data = bsize=4096 blocks=23436288, imaxpct=25 2024-06-03T08:34:49.245 INFO:teuthology.orchestra.run.smithi192.stdout: = sunit=0 swidth=0 blks 2024-06-03T08:34:49.246 INFO:teuthology.orchestra.run.smithi192.stdout:naming =version 2 bsize=4096 ascii-ci=0, ftype=1 2024-06-03T08:34:49.246 INFO:teuthology.orchestra.run.smithi192.stdout:log =internal log bsize=4096 blocks=16384, version=2 2024-06-03T08:34:49.246 INFO:teuthology.orchestra.run.smithi192.stdout: = sectsz=512 sunit=0 blks, lazy-count=1 2024-06-03T08:34:49.246 INFO:teuthology.orchestra.run.smithi192.stdout:realtime =none extsz=4096 blocks=0, rtextents=0 2024-06-03T08:34:49.740 INFO:teuthology.orchestra.run.smithi192.stdout:Discarding blocks...Done. 2024-06-03T08:34:49.742 INFO:tasks.ceph:mount /dev/vg_nvme/lv_3 on ubuntu@smithi192.front.sepia.ceph.com -o noatime 2024-06-03T08:34:49.743 DEBUG:teuthology.orchestra.run.smithi192:> sudo mount -t xfs -o noatime /dev/vg_nvme/lv_3 /var/lib/ceph/osd/ceph-6 2024-06-03T08:34:49.893 DEBUG:teuthology.orchestra.run.smithi192:> sudo /sbin/restorecon /var/lib/ceph/osd/ceph-6 2024-06-03T08:34:49.926 DEBUG:teuthology.orchestra.run.smithi192:> sudo mkdir -p /var/lib/ceph/osd/ceph-7 2024-06-03T08:34:49.993 INFO:tasks.ceph:roles_to_devs: {'osd.4': '/dev/vg_nvme/lv_1', 'osd.5': '/dev/vg_nvme/lv_2', 'osd.6': '/dev/vg_nvme/lv_3', 'osd.7': '/dev/vg_nvme/lv_4'} 2024-06-03T08:34:49.993 INFO:tasks.ceph:role: osd.7 2024-06-03T08:34:49.993 INFO:tasks.ceph:['mkfs.xfs', '-f', '-i', 'size=2048'] on /dev/vg_nvme/lv_4 on ubuntu@smithi192.front.sepia.ceph.com 2024-06-03T08:34:49.993 DEBUG:teuthology.orchestra.run.smithi192:> yes | sudo mkfs.xfs -f -i size=2048 /dev/vg_nvme/lv_4 2024-06-03T08:34:50.067 INFO:teuthology.orchestra.run.smithi192.stdout:meta-data=/dev/vg_nvme/lv_4 isize=2048 agcount=4, agsize=5859072 blks 2024-06-03T08:34:50.067 INFO:teuthology.orchestra.run.smithi192.stdout: = sectsz=512 attr=2, projid32bit=1 2024-06-03T08:34:50.067 INFO:teuthology.orchestra.run.smithi192.stdout: = crc=1 finobt=1, sparse=1, rmapbt=0 2024-06-03T08:34:50.067 INFO:teuthology.orchestra.run.smithi192.stdout: = reflink=1 bigtime=1 inobtcount=1 nrext64=0 2024-06-03T08:34:50.067 INFO:teuthology.orchestra.run.smithi192.stdout:data = bsize=4096 blocks=23436288, imaxpct=25 2024-06-03T08:34:50.067 INFO:teuthology.orchestra.run.smithi192.stdout: = sunit=0 swidth=0 blks 2024-06-03T08:34:50.067 INFO:teuthology.orchestra.run.smithi192.stdout:naming =version 2 bsize=4096 ascii-ci=0, ftype=1 2024-06-03T08:34:50.068 INFO:teuthology.orchestra.run.smithi192.stdout:log =internal log bsize=4096 blocks=16384, version=2 2024-06-03T08:34:50.068 INFO:teuthology.orchestra.run.smithi192.stdout: = sectsz=512 sunit=0 blks, lazy-count=1 2024-06-03T08:34:50.068 INFO:teuthology.orchestra.run.smithi192.stdout:realtime =none extsz=4096 blocks=0, rtextents=0 2024-06-03T08:34:50.582 INFO:teuthology.orchestra.run.smithi192.stdout:Discarding blocks...Done. 2024-06-03T08:34:50.584 INFO:tasks.ceph:mount /dev/vg_nvme/lv_4 on ubuntu@smithi192.front.sepia.ceph.com -o noatime 2024-06-03T08:34:50.584 DEBUG:teuthology.orchestra.run.smithi192:> sudo mount -t xfs -o noatime /dev/vg_nvme/lv_4 /var/lib/ceph/osd/ceph-7 2024-06-03T08:34:50.705 DEBUG:teuthology.orchestra.run.smithi192:> sudo /sbin/restorecon /var/lib/ceph/osd/ceph-7 2024-06-03T08:34:50.741 DEBUG:teuthology.orchestra.run.smithi192:> sudo MALLOC_CHECK_=3 adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-osd --no-mon-config --cluster ceph --mkfs --mkkey -i 4 --monmap /home/ubuntu/cephtest/ceph.monmap 2024-06-03T08:34:50.841 INFO:teuthology.orchestra.run.smithi192.stderr:2024-06-03T08:34:50.840+0000 7fa9cc611640 -1 auth: error reading file: /var/lib/ceph/osd/ceph-4/keyring: can't open /var/lib/ceph/osd/ceph-4/keyring: (2) No such file or directory 2024-06-03T08:34:50.841 INFO:teuthology.orchestra.run.smithi192.stderr:2024-06-03T08:34:50.840+0000 7fa9cc611640 -1 created new key in keyring /var/lib/ceph/osd/ceph-4/keyring 2024-06-03T08:34:50.841 INFO:teuthology.orchestra.run.smithi192.stderr:2024-06-03T08:34:50.840+0000 7fa9cc611640 -1 bluestore(/var/lib/ceph/osd/ceph-4/block) _read_bdev_label failed to open /var/lib/ceph/osd/ceph-4/block: (2) No such file or directory 2024-06-03T08:34:50.841 INFO:teuthology.orchestra.run.smithi192.stderr:2024-06-03T08:34:50.840+0000 7fa9cc611640 -1 bluestore(/var/lib/ceph/osd/ceph-4/block) _read_bdev_label failed to open /var/lib/ceph/osd/ceph-4/block: (2) No such file or directory 2024-06-03T08:34:50.841 INFO:teuthology.orchestra.run.smithi192.stderr:2024-06-03T08:34:50.840+0000 7fa9cc611640 -1 bluestore(/var/lib/ceph/osd/ceph-4/block) _read_bdev_label failed to open /var/lib/ceph/osd/ceph-4/block: (2) No such file or directory 2024-06-03T08:34:50.842 INFO:teuthology.orchestra.run.smithi192.stderr:2024-06-03T08:34:50.842+0000 7fa9cc611640 -1 bluestore(/var/lib/ceph/osd/ceph-4) _read_fsid unparsable uuid 2024-06-03T08:34:53.692 DEBUG:teuthology.orchestra.run.smithi192:> sudo chown -R ceph:ceph /var/lib/ceph/osd/ceph-4 2024-06-03T08:34:53.722 DEBUG:teuthology.orchestra.run.smithi192:> sudo MALLOC_CHECK_=3 adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-osd --no-mon-config --cluster ceph --mkfs --mkkey -i 5 --monmap /home/ubuntu/cephtest/ceph.monmap 2024-06-03T08:34:53.813 INFO:teuthology.orchestra.run.smithi192.stderr:2024-06-03T08:34:53.812+0000 7ffa06f21640 -1 auth: error reading file: /var/lib/ceph/osd/ceph-5/keyring: can't open /var/lib/ceph/osd/ceph-5/keyring: (2) No such file or directory 2024-06-03T08:34:53.813 INFO:teuthology.orchestra.run.smithi192.stderr:2024-06-03T08:34:53.812+0000 7ffa06f21640 -1 created new key in keyring /var/lib/ceph/osd/ceph-5/keyring 2024-06-03T08:34:53.813 INFO:teuthology.orchestra.run.smithi192.stderr:2024-06-03T08:34:53.812+0000 7ffa06f21640 -1 bluestore(/var/lib/ceph/osd/ceph-5/block) _read_bdev_label failed to open /var/lib/ceph/osd/ceph-5/block: (2) No such file or directory 2024-06-03T08:34:53.813 INFO:teuthology.orchestra.run.smithi192.stderr:2024-06-03T08:34:53.812+0000 7ffa06f21640 -1 bluestore(/var/lib/ceph/osd/ceph-5/block) _read_bdev_label failed to open /var/lib/ceph/osd/ceph-5/block: (2) No such file or directory 2024-06-03T08:34:53.813 INFO:teuthology.orchestra.run.smithi192.stderr:2024-06-03T08:34:53.812+0000 7ffa06f21640 -1 bluestore(/var/lib/ceph/osd/ceph-5/block) _read_bdev_label failed to open /var/lib/ceph/osd/ceph-5/block: (2) No such file or directory 2024-06-03T08:34:53.813 INFO:teuthology.orchestra.run.smithi192.stderr:2024-06-03T08:34:53.813+0000 7ffa06f21640 -1 bluestore(/var/lib/ceph/osd/ceph-5) _read_fsid unparsable uuid 2024-06-03T08:34:56.661 DEBUG:teuthology.orchestra.run.smithi192:> sudo chown -R ceph:ceph /var/lib/ceph/osd/ceph-5 2024-06-03T08:34:56.690 DEBUG:teuthology.orchestra.run.smithi192:> sudo MALLOC_CHECK_=3 adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-osd --no-mon-config --cluster ceph --mkfs --mkkey -i 6 --monmap /home/ubuntu/cephtest/ceph.monmap 2024-06-03T08:34:56.780 INFO:teuthology.orchestra.run.smithi192.stderr:2024-06-03T08:34:56.780+0000 7f246baae640 -1 auth: error reading file: /var/lib/ceph/osd/ceph-6/keyring: can't open /var/lib/ceph/osd/ceph-6/keyring: (2) No such file or directory 2024-06-03T08:34:56.781 INFO:teuthology.orchestra.run.smithi192.stderr:2024-06-03T08:34:56.780+0000 7f246baae640 -1 created new key in keyring /var/lib/ceph/osd/ceph-6/keyring 2024-06-03T08:34:56.781 INFO:teuthology.orchestra.run.smithi192.stderr:2024-06-03T08:34:56.780+0000 7f246baae640 -1 bluestore(/var/lib/ceph/osd/ceph-6/block) _read_bdev_label failed to open /var/lib/ceph/osd/ceph-6/block: (2) No such file or directory 2024-06-03T08:34:56.781 INFO:teuthology.orchestra.run.smithi192.stderr:2024-06-03T08:34:56.780+0000 7f246baae640 -1 bluestore(/var/lib/ceph/osd/ceph-6/block) _read_bdev_label failed to open /var/lib/ceph/osd/ceph-6/block: (2) No such file or directory 2024-06-03T08:34:56.781 INFO:teuthology.orchestra.run.smithi192.stderr:2024-06-03T08:34:56.780+0000 7f246baae640 -1 bluestore(/var/lib/ceph/osd/ceph-6/block) _read_bdev_label failed to open /var/lib/ceph/osd/ceph-6/block: (2) No such file or directory 2024-06-03T08:34:56.781 INFO:teuthology.orchestra.run.smithi192.stderr:2024-06-03T08:34:56.781+0000 7f246baae640 -1 bluestore(/var/lib/ceph/osd/ceph-6) _read_fsid unparsable uuid 2024-06-03T08:34:59.631 DEBUG:teuthology.orchestra.run.smithi192:> sudo chown -R ceph:ceph /var/lib/ceph/osd/ceph-6 2024-06-03T08:34:59.659 DEBUG:teuthology.orchestra.run.smithi192:> sudo MALLOC_CHECK_=3 adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-osd --no-mon-config --cluster ceph --mkfs --mkkey -i 7 --monmap /home/ubuntu/cephtest/ceph.monmap 2024-06-03T08:34:59.748 INFO:teuthology.orchestra.run.smithi192.stderr:2024-06-03T08:34:59.747+0000 7f8104c0e640 -1 auth: error reading file: /var/lib/ceph/osd/ceph-7/keyring: can't open /var/lib/ceph/osd/ceph-7/keyring: (2) No such file or directory 2024-06-03T08:34:59.748 INFO:teuthology.orchestra.run.smithi192.stderr:2024-06-03T08:34:59.747+0000 7f8104c0e640 -1 created new key in keyring /var/lib/ceph/osd/ceph-7/keyring 2024-06-03T08:34:59.748 INFO:teuthology.orchestra.run.smithi192.stderr:2024-06-03T08:34:59.747+0000 7f8104c0e640 -1 bluestore(/var/lib/ceph/osd/ceph-7/block) _read_bdev_label failed to open /var/lib/ceph/osd/ceph-7/block: (2) No such file or directory 2024-06-03T08:34:59.748 INFO:teuthology.orchestra.run.smithi192.stderr:2024-06-03T08:34:59.747+0000 7f8104c0e640 -1 bluestore(/var/lib/ceph/osd/ceph-7/block) _read_bdev_label failed to open /var/lib/ceph/osd/ceph-7/block: (2) No such file or directory 2024-06-03T08:34:59.748 INFO:teuthology.orchestra.run.smithi192.stderr:2024-06-03T08:34:59.747+0000 7f8104c0e640 -1 bluestore(/var/lib/ceph/osd/ceph-7/block) _read_bdev_label failed to open /var/lib/ceph/osd/ceph-7/block: (2) No such file or directory 2024-06-03T08:34:59.748 INFO:teuthology.orchestra.run.smithi192.stderr:2024-06-03T08:34:59.748+0000 7f8104c0e640 -1 bluestore(/var/lib/ceph/osd/ceph-7) _read_fsid unparsable uuid 2024-06-03T08:35:02.592 DEBUG:teuthology.orchestra.run.smithi192:> sudo chown -R ceph:ceph /var/lib/ceph/osd/ceph-7 2024-06-03T08:35:02.620 INFO:tasks.ceph:Reading keys from all nodes... 2024-06-03T08:35:02.620 DEBUG:teuthology.orchestra.run.smithi088:> set -ex 2024-06-03T08:35:02.620 DEBUG:teuthology.orchestra.run.smithi088:> sudo dd if=/var/lib/ceph/mgr/ceph-y/keyring of=/dev/stdout 2024-06-03T08:35:02.650 DEBUG:teuthology.orchestra.run.smithi088:> set -ex 2024-06-03T08:35:02.650 DEBUG:teuthology.orchestra.run.smithi088:> sudo dd if=/var/lib/ceph/osd/ceph-0/keyring of=/dev/stdout 2024-06-03T08:35:02.718 DEBUG:teuthology.orchestra.run.smithi088:> set -ex 2024-06-03T08:35:02.718 DEBUG:teuthology.orchestra.run.smithi088:> sudo dd if=/var/lib/ceph/osd/ceph-1/keyring of=/dev/stdout 2024-06-03T08:35:02.786 DEBUG:teuthology.orchestra.run.smithi088:> set -ex 2024-06-03T08:35:02.786 DEBUG:teuthology.orchestra.run.smithi088:> sudo dd if=/var/lib/ceph/osd/ceph-2/keyring of=/dev/stdout 2024-06-03T08:35:02.852 DEBUG:teuthology.orchestra.run.smithi088:> set -ex 2024-06-03T08:35:02.852 DEBUG:teuthology.orchestra.run.smithi088:> sudo dd if=/var/lib/ceph/osd/ceph-3/keyring of=/dev/stdout 2024-06-03T08:35:02.918 DEBUG:teuthology.orchestra.run.smithi192:> set -ex 2024-06-03T08:35:02.918 DEBUG:teuthology.orchestra.run.smithi192:> sudo dd if=/var/lib/ceph/mgr/ceph-x/keyring of=/dev/stdout 2024-06-03T08:35:02.948 DEBUG:teuthology.orchestra.run.smithi192:> set -ex 2024-06-03T08:35:02.948 DEBUG:teuthology.orchestra.run.smithi192:> sudo dd if=/var/lib/ceph/osd/ceph-4/keyring of=/dev/stdout 2024-06-03T08:35:03.016 DEBUG:teuthology.orchestra.run.smithi192:> set -ex 2024-06-03T08:35:03.016 DEBUG:teuthology.orchestra.run.smithi192:> sudo dd if=/var/lib/ceph/osd/ceph-5/keyring of=/dev/stdout 2024-06-03T08:35:03.083 DEBUG:teuthology.orchestra.run.smithi192:> set -ex 2024-06-03T08:35:03.083 DEBUG:teuthology.orchestra.run.smithi192:> sudo dd if=/var/lib/ceph/osd/ceph-6/keyring of=/dev/stdout 2024-06-03T08:35:03.150 DEBUG:teuthology.orchestra.run.smithi192:> set -ex 2024-06-03T08:35:03.150 DEBUG:teuthology.orchestra.run.smithi192:> sudo dd if=/var/lib/ceph/osd/ceph-7/keyring of=/dev/stdout 2024-06-03T08:35:03.216 DEBUG:teuthology.orchestra.run.smithi088:> set -ex 2024-06-03T08:35:03.216 DEBUG:teuthology.orchestra.run.smithi088:> dd if=/etc/ceph/ceph.client.0.keyring of=/dev/stdout 2024-06-03T08:35:03.233 DEBUG:teuthology.orchestra.run.smithi192:> set -ex 2024-06-03T08:35:03.233 DEBUG:teuthology.orchestra.run.smithi192:> dd if=/etc/ceph/ceph.client.1.keyring of=/dev/stdout 2024-06-03T08:35:03.272 INFO:tasks.ceph:Adding keys to all mons... 2024-06-03T08:35:03.272 DEBUG:teuthology.orchestra.run.smithi088:> sudo tee -a /etc/ceph/ceph.keyring 2024-06-03T08:35:03.276 DEBUG:teuthology.orchestra.run.smithi192:> sudo tee -a /etc/ceph/ceph.keyring 2024-06-03T08:35:03.315 INFO:teuthology.orchestra.run.smithi088.stdout:[mgr.y] 2024-06-03T08:35:03.315 INFO:teuthology.orchestra.run.smithi088.stdout: key = AQAYgF1mjHswGhAAiJAQ4n0JyK267H+oWrwlgw== 2024-06-03T08:35:03.316 INFO:teuthology.orchestra.run.smithi088.stdout:[osd.0] 2024-06-03T08:35:03.316 INFO:teuthology.orchestra.run.smithi088.stdout: key = AQAbgF1mZEkoMhAA/xeuwW1MZ+T0I0UMewWfIQ== 2024-06-03T08:35:03.316 INFO:teuthology.orchestra.run.smithi088.stdout:[osd.1] 2024-06-03T08:35:03.316 INFO:teuthology.orchestra.run.smithi088.stdout: key = AQAegF1m3SO4MBAALn5/yVuW0oit+6YV7Wdk3Q== 2024-06-03T08:35:03.316 INFO:teuthology.orchestra.run.smithi088.stdout:[osd.2] 2024-06-03T08:35:03.316 INFO:teuthology.orchestra.run.smithi088.stdout: key = AQAhgF1mqSyFLhAAlX7F2oUtEXjfS2a/Zfj+AA== 2024-06-03T08:35:03.316 INFO:teuthology.orchestra.run.smithi088.stdout:[osd.3] 2024-06-03T08:35:03.316 INFO:teuthology.orchestra.run.smithi088.stdout: key = AQAkgF1m/kIQLBAACim8FL31JRVlHCpclv/7lw== 2024-06-03T08:35:03.316 INFO:teuthology.orchestra.run.smithi088.stdout:[mgr.x] 2024-06-03T08:35:03.316 INFO:teuthology.orchestra.run.smithi088.stdout: key = AQAYgF1m60EjHxAA8yQrduFQTK5m02tS6nFZOA== 2024-06-03T08:35:03.316 INFO:teuthology.orchestra.run.smithi088.stdout:[osd.4] 2024-06-03T08:35:03.317 INFO:teuthology.orchestra.run.smithi088.stdout: key = AQAqgF1mUB4mMhAAngxITJXbPUohGTPNqv8szA== 2024-06-03T08:35:03.317 INFO:teuthology.orchestra.run.smithi088.stdout:[osd.5] 2024-06-03T08:35:03.317 INFO:teuthology.orchestra.run.smithi088.stdout: key = AQAtgF1miMB6MBAA9ukDxXgyOFMzyhi7JxcSCw== 2024-06-03T08:35:03.317 INFO:teuthology.orchestra.run.smithi088.stdout:[osd.6] 2024-06-03T08:35:03.317 INFO:teuthology.orchestra.run.smithi088.stdout: key = AQAwgF1mVYePLhAAByE18vTjAv5XrQpACrlMxQ== 2024-06-03T08:35:03.317 INFO:teuthology.orchestra.run.smithi088.stdout:[osd.7] 2024-06-03T08:35:03.317 INFO:teuthology.orchestra.run.smithi088.stdout: key = AQAzgF1mUiKcLBAAg0Iv0XTT4zh7PAUZ9lsdYA== 2024-06-03T08:35:03.317 INFO:teuthology.orchestra.run.smithi088.stdout:[client.0] 2024-06-03T08:35:03.317 INFO:teuthology.orchestra.run.smithi088.stdout: key = AQAYgF1m6sq3IRAA7IQuEjvi93PwJRHUjclViQ== 2024-06-03T08:35:03.317 INFO:teuthology.orchestra.run.smithi088.stdout:[client.1] 2024-06-03T08:35:03.317 INFO:teuthology.orchestra.run.smithi088.stdout: key = AQAYgF1mdmXdJBAA/+IIsJ4fl/2evLjPvit3Bw== 2024-06-03T08:35:03.336 INFO:teuthology.orchestra.run.smithi192.stdout:[mgr.y] 2024-06-03T08:35:03.336 INFO:teuthology.orchestra.run.smithi192.stdout: key = AQAYgF1mjHswGhAAiJAQ4n0JyK267H+oWrwlgw== 2024-06-03T08:35:03.336 INFO:teuthology.orchestra.run.smithi192.stdout:[osd.0] 2024-06-03T08:35:03.336 INFO:teuthology.orchestra.run.smithi192.stdout: key = AQAbgF1mZEkoMhAA/xeuwW1MZ+T0I0UMewWfIQ== 2024-06-03T08:35:03.336 INFO:teuthology.orchestra.run.smithi192.stdout:[osd.1] 2024-06-03T08:35:03.336 INFO:teuthology.orchestra.run.smithi192.stdout: key = AQAegF1m3SO4MBAALn5/yVuW0oit+6YV7Wdk3Q== 2024-06-03T08:35:03.336 INFO:teuthology.orchestra.run.smithi192.stdout:[osd.2] 2024-06-03T08:35:03.336 INFO:teuthology.orchestra.run.smithi192.stdout: key = AQAhgF1mqSyFLhAAlX7F2oUtEXjfS2a/Zfj+AA== 2024-06-03T08:35:03.337 INFO:teuthology.orchestra.run.smithi192.stdout:[osd.3] 2024-06-03T08:35:03.337 INFO:teuthology.orchestra.run.smithi192.stdout: key = AQAkgF1m/kIQLBAACim8FL31JRVlHCpclv/7lw== 2024-06-03T08:35:03.337 INFO:teuthology.orchestra.run.smithi192.stdout:[mgr.x] 2024-06-03T08:35:03.337 INFO:teuthology.orchestra.run.smithi192.stdout: key = AQAYgF1m60EjHxAA8yQrduFQTK5m02tS6nFZOA== 2024-06-03T08:35:03.337 INFO:teuthology.orchestra.run.smithi192.stdout:[osd.4] 2024-06-03T08:35:03.337 INFO:teuthology.orchestra.run.smithi192.stdout: key = AQAqgF1mUB4mMhAAngxITJXbPUohGTPNqv8szA== 2024-06-03T08:35:03.337 INFO:teuthology.orchestra.run.smithi192.stdout:[osd.5] 2024-06-03T08:35:03.337 INFO:teuthology.orchestra.run.smithi192.stdout: key = AQAtgF1miMB6MBAA9ukDxXgyOFMzyhi7JxcSCw== 2024-06-03T08:35:03.337 INFO:teuthology.orchestra.run.smithi192.stdout:[osd.6] 2024-06-03T08:35:03.337 INFO:teuthology.orchestra.run.smithi192.stdout: key = AQAwgF1mVYePLhAAByE18vTjAv5XrQpACrlMxQ== 2024-06-03T08:35:03.337 INFO:teuthology.orchestra.run.smithi192.stdout:[osd.7] 2024-06-03T08:35:03.337 INFO:teuthology.orchestra.run.smithi192.stdout: key = AQAzgF1mUiKcLBAAg0Iv0XTT4zh7PAUZ9lsdYA== 2024-06-03T08:35:03.337 INFO:teuthology.orchestra.run.smithi192.stdout:[client.0] 2024-06-03T08:35:03.338 INFO:teuthology.orchestra.run.smithi192.stdout: key = AQAYgF1m6sq3IRAA7IQuEjvi93PwJRHUjclViQ== 2024-06-03T08:35:03.338 INFO:teuthology.orchestra.run.smithi192.stdout:[client.1] 2024-06-03T08:35:03.338 INFO:teuthology.orchestra.run.smithi192.stdout: key = AQAYgF1mdmXdJBAA/+IIsJ4fl/2evLjPvit3Bw== 2024-06-03T08:35:03.339 DEBUG:teuthology.orchestra.run.smithi088:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-authtool /etc/ceph/ceph.keyring --name=mgr.y --cap mon 'allow profile mgr' --cap osd 'allow *' --cap mds 'allow *' 2024-06-03T08:35:03.360 DEBUG:teuthology.orchestra.run.smithi192:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-authtool /etc/ceph/ceph.keyring --name=mgr.y --cap mon 'allow profile mgr' --cap osd 'allow *' --cap mds 'allow *' 2024-06-03T08:35:03.430 DEBUG:teuthology.orchestra.run.smithi088:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-authtool /etc/ceph/ceph.keyring --name=osd.0 --cap mon 'allow profile osd' --cap mgr 'allow profile osd' --cap osd 'allow *' 2024-06-03T08:35:03.432 DEBUG:teuthology.orchestra.run.smithi192:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-authtool /etc/ceph/ceph.keyring --name=osd.0 --cap mon 'allow profile osd' --cap mgr 'allow profile osd' --cap osd 'allow *' 2024-06-03T08:35:03.482 DEBUG:teuthology.orchestra.run.smithi088:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-authtool /etc/ceph/ceph.keyring --name=osd.1 --cap mon 'allow profile osd' --cap mgr 'allow profile osd' --cap osd 'allow *' 2024-06-03T08:35:03.485 DEBUG:teuthology.orchestra.run.smithi192:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-authtool /etc/ceph/ceph.keyring --name=osd.1 --cap mon 'allow profile osd' --cap mgr 'allow profile osd' --cap osd 'allow *' 2024-06-03T08:35:03.534 DEBUG:teuthology.orchestra.run.smithi088:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-authtool /etc/ceph/ceph.keyring --name=osd.2 --cap mon 'allow profile osd' --cap mgr 'allow profile osd' --cap osd 'allow *' 2024-06-03T08:35:03.537 DEBUG:teuthology.orchestra.run.smithi192:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-authtool /etc/ceph/ceph.keyring --name=osd.2 --cap mon 'allow profile osd' --cap mgr 'allow profile osd' --cap osd 'allow *' 2024-06-03T08:35:03.586 DEBUG:teuthology.orchestra.run.smithi088:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-authtool /etc/ceph/ceph.keyring --name=osd.3 --cap mon 'allow profile osd' --cap mgr 'allow profile osd' --cap osd 'allow *' 2024-06-03T08:35:03.589 DEBUG:teuthology.orchestra.run.smithi192:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-authtool /etc/ceph/ceph.keyring --name=osd.3 --cap mon 'allow profile osd' --cap mgr 'allow profile osd' --cap osd 'allow *' 2024-06-03T08:35:03.639 DEBUG:teuthology.orchestra.run.smithi088:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-authtool /etc/ceph/ceph.keyring --name=mgr.x --cap mon 'allow profile mgr' --cap osd 'allow *' --cap mds 'allow *' 2024-06-03T08:35:03.641 DEBUG:teuthology.orchestra.run.smithi192:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-authtool /etc/ceph/ceph.keyring --name=mgr.x --cap mon 'allow profile mgr' --cap osd 'allow *' --cap mds 'allow *' 2024-06-03T08:35:03.691 DEBUG:teuthology.orchestra.run.smithi088:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-authtool /etc/ceph/ceph.keyring --name=osd.4 --cap mon 'allow profile osd' --cap mgr 'allow profile osd' --cap osd 'allow *' 2024-06-03T08:35:03.693 DEBUG:teuthology.orchestra.run.smithi192:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-authtool /etc/ceph/ceph.keyring --name=osd.4 --cap mon 'allow profile osd' --cap mgr 'allow profile osd' --cap osd 'allow *' 2024-06-03T08:35:03.743 DEBUG:teuthology.orchestra.run.smithi088:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-authtool /etc/ceph/ceph.keyring --name=osd.5 --cap mon 'allow profile osd' --cap mgr 'allow profile osd' --cap osd 'allow *' 2024-06-03T08:35:03.745 DEBUG:teuthology.orchestra.run.smithi192:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-authtool /etc/ceph/ceph.keyring --name=osd.5 --cap mon 'allow profile osd' --cap mgr 'allow profile osd' --cap osd 'allow *' 2024-06-03T08:35:03.795 DEBUG:teuthology.orchestra.run.smithi088:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-authtool /etc/ceph/ceph.keyring --name=osd.6 --cap mon 'allow profile osd' --cap mgr 'allow profile osd' --cap osd 'allow *' 2024-06-03T08:35:03.797 DEBUG:teuthology.orchestra.run.smithi192:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-authtool /etc/ceph/ceph.keyring --name=osd.6 --cap mon 'allow profile osd' --cap mgr 'allow profile osd' --cap osd 'allow *' 2024-06-03T08:35:03.847 DEBUG:teuthology.orchestra.run.smithi088:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-authtool /etc/ceph/ceph.keyring --name=osd.7 --cap mon 'allow profile osd' --cap mgr 'allow profile osd' --cap osd 'allow *' 2024-06-03T08:35:03.849 DEBUG:teuthology.orchestra.run.smithi192:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-authtool /etc/ceph/ceph.keyring --name=osd.7 --cap mon 'allow profile osd' --cap mgr 'allow profile osd' --cap osd 'allow *' 2024-06-03T08:35:03.899 DEBUG:teuthology.orchestra.run.smithi088:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-authtool /etc/ceph/ceph.keyring --name=client.0 --cap mon 'allow rw' --cap mgr 'allow r' --cap osd 'allow rwx' --cap mds allow 2024-06-03T08:35:03.901 DEBUG:teuthology.orchestra.run.smithi192:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-authtool /etc/ceph/ceph.keyring --name=client.0 --cap mon 'allow rw' --cap mgr 'allow r' --cap osd 'allow rwx' --cap mds allow 2024-06-03T08:35:03.954 DEBUG:teuthology.orchestra.run.smithi088:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-authtool /etc/ceph/ceph.keyring --name=client.1 --cap mon 'allow rw' --cap mgr 'allow r' --cap osd 'allow rwx' --cap mds allow 2024-06-03T08:35:03.956 DEBUG:teuthology.orchestra.run.smithi192:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-authtool /etc/ceph/ceph.keyring --name=client.1 --cap mon 'allow rw' --cap mgr 'allow r' --cap osd 'allow rwx' --cap mds allow 2024-06-03T08:35:04.009 INFO:tasks.ceph:Running mkfs on mon nodes... 2024-06-03T08:35:04.009 DEBUG:teuthology.orchestra.run.smithi088:> sudo mkdir -p /var/lib/ceph/mon/ceph-a 2024-06-03T08:35:04.034 DEBUG:teuthology.orchestra.run.smithi088:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-mon --cluster ceph --mkfs -i a --monmap /home/ubuntu/cephtest/ceph.monmap --keyring /etc/ceph/ceph.keyring 2024-06-03T08:35:04.130 DEBUG:teuthology.orchestra.run.smithi088:> sudo chown -R ceph:ceph /var/lib/ceph/mon/ceph-a 2024-06-03T08:35:04.155 DEBUG:teuthology.orchestra.run.smithi088:> sudo mkdir -p /var/lib/ceph/mon/ceph-c 2024-06-03T08:35:04.219 DEBUG:teuthology.orchestra.run.smithi088:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-mon --cluster ceph --mkfs -i c --monmap /home/ubuntu/cephtest/ceph.monmap --keyring /etc/ceph/ceph.keyring 2024-06-03T08:35:04.314 DEBUG:teuthology.orchestra.run.smithi088:> sudo chown -R ceph:ceph /var/lib/ceph/mon/ceph-c 2024-06-03T08:35:04.339 DEBUG:teuthology.orchestra.run.smithi192:> sudo mkdir -p /var/lib/ceph/mon/ceph-b 2024-06-03T08:35:04.364 DEBUG:teuthology.orchestra.run.smithi192:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-mon --cluster ceph --mkfs -i b --monmap /home/ubuntu/cephtest/ceph.monmap --keyring /etc/ceph/ceph.keyring 2024-06-03T08:35:04.460 DEBUG:teuthology.orchestra.run.smithi192:> sudo chown -R ceph:ceph /var/lib/ceph/mon/ceph-b 2024-06-03T08:35:04.484 DEBUG:teuthology.orchestra.run.smithi088:> rm -- /home/ubuntu/cephtest/ceph.monmap 2024-06-03T08:35:04.487 DEBUG:teuthology.orchestra.run.smithi192:> rm -- /home/ubuntu/cephtest/ceph.monmap 2024-06-03T08:35:04.540 INFO:tasks.ceph:Starting mon daemons in cluster ceph... 2024-06-03T08:35:04.541 DEBUG:tasks.ceph_manager:running mon.a under valgrind with args ['cd', '/home/ubuntu/cephtest', Raw('&&'), 'sudo', 'adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'daemon-helper', 'term', 'env', 'OPENSSL_ia32cap=~0x1000000000000000', 'valgrind', '--trace-children=no', '--child-silent-after-fork=yes', '--soname-synonyms=somalloc=*tcmalloc*', '--num-callers=50', '--suppressions=/home/ubuntu/cephtest/valgrind.supp', '--xml=yes', '--xml-file=/var/log/ceph/valgrind/mon.a.log', '--time-stamp=yes', '--vgdb=yes', '--exit-on-first-error=yes', '--error-exitcode=42', '--tool=memcheck', '--leak-check=full', '--show-reachable=yes'] 2024-06-03T08:35:04.541 INFO:tasks.ceph.mon.a:Restarting daemon 2024-06-03T08:35:04.541 DEBUG:teuthology.orchestra.run.smithi088:> cd /home/ubuntu/cephtest && sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage daemon-helper term env 'OPENSSL_ia32cap=~0x1000000000000000' valgrind --trace-children=no --child-silent-after-fork=yes '--soname-synonyms=somalloc=*tcmalloc*' --num-callers=50 --suppressions=/home/ubuntu/cephtest/valgrind.supp --xml=yes --xml-file=/var/log/ceph/valgrind/mon.a.log --time-stamp=yes --vgdb=yes --exit-on-first-error=yes --error-exitcode=42 --tool=memcheck --leak-check=full --show-reachable=yes ceph-mon -f --cluster ceph -i a 2024-06-03T08:35:04.543 INFO:tasks.ceph.mon.a:Started 2024-06-03T08:35:04.543 DEBUG:tasks.ceph_manager:running mon.c under valgrind with args ['cd', '/home/ubuntu/cephtest', Raw('&&'), 'sudo', 'adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'daemon-helper', 'term', 'env', 'OPENSSL_ia32cap=~0x1000000000000000', 'valgrind', '--trace-children=no', '--child-silent-after-fork=yes', '--soname-synonyms=somalloc=*tcmalloc*', '--num-callers=50', '--suppressions=/home/ubuntu/cephtest/valgrind.supp', '--xml=yes', '--xml-file=/var/log/ceph/valgrind/mon.c.log', '--time-stamp=yes', '--vgdb=yes', '--exit-on-first-error=yes', '--error-exitcode=42', '--tool=memcheck', '--leak-check=full', '--show-reachable=yes'] 2024-06-03T08:35:04.543 INFO:tasks.ceph.mon.c:Restarting daemon 2024-06-03T08:35:04.544 DEBUG:teuthology.orchestra.run.smithi088:> cd /home/ubuntu/cephtest && sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage daemon-helper term env 'OPENSSL_ia32cap=~0x1000000000000000' valgrind --trace-children=no --child-silent-after-fork=yes '--soname-synonyms=somalloc=*tcmalloc*' --num-callers=50 --suppressions=/home/ubuntu/cephtest/valgrind.supp --xml=yes --xml-file=/var/log/ceph/valgrind/mon.c.log --time-stamp=yes --vgdb=yes --exit-on-first-error=yes --error-exitcode=42 --tool=memcheck --leak-check=full --show-reachable=yes ceph-mon -f --cluster ceph -i c 2024-06-03T08:35:04.545 INFO:tasks.ceph.mon.c:Started 2024-06-03T08:35:04.546 DEBUG:tasks.ceph_manager:running mon.b under valgrind with args ['cd', '/home/ubuntu/cephtest', Raw('&&'), 'sudo', 'adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'daemon-helper', 'term', 'env', 'OPENSSL_ia32cap=~0x1000000000000000', 'valgrind', '--trace-children=no', '--child-silent-after-fork=yes', '--soname-synonyms=somalloc=*tcmalloc*', '--num-callers=50', '--suppressions=/home/ubuntu/cephtest/valgrind.supp', '--xml=yes', '--xml-file=/var/log/ceph/valgrind/mon.b.log', '--time-stamp=yes', '--vgdb=yes', '--exit-on-first-error=yes', '--error-exitcode=42', '--tool=memcheck', '--leak-check=full', '--show-reachable=yes'] 2024-06-03T08:35:04.546 INFO:tasks.ceph.mon.b:Restarting daemon 2024-06-03T08:35:04.546 DEBUG:teuthology.orchestra.run.smithi192:> cd /home/ubuntu/cephtest && sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage daemon-helper term env 'OPENSSL_ia32cap=~0x1000000000000000' valgrind --trace-children=no --child-silent-after-fork=yes '--soname-synonyms=somalloc=*tcmalloc*' --num-callers=50 --suppressions=/home/ubuntu/cephtest/valgrind.supp --xml=yes --xml-file=/var/log/ceph/valgrind/mon.b.log --time-stamp=yes --vgdb=yes --exit-on-first-error=yes --error-exitcode=42 --tool=memcheck --leak-check=full --show-reachable=yes ceph-mon -f --cluster ceph -i b 2024-06-03T08:35:04.582 INFO:tasks.ceph.mon.b:Started 2024-06-03T08:35:04.583 INFO:tasks.ceph:Starting mgr daemons in cluster ceph... 2024-06-03T08:35:04.583 INFO:tasks.ceph.mgr.y:Restarting daemon 2024-06-03T08:35:04.583 DEBUG:teuthology.orchestra.run.smithi088:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage daemon-helper term ceph-mgr -f --cluster ceph -i y 2024-06-03T08:35:04.585 INFO:tasks.ceph.mgr.y:Started 2024-06-03T08:35:04.585 INFO:tasks.ceph.mgr.x:Restarting daemon 2024-06-03T08:35:04.586 DEBUG:teuthology.orchestra.run.smithi192:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage daemon-helper term ceph-mgr -f --cluster ceph -i x 2024-06-03T08:35:04.587 INFO:tasks.ceph.mgr.x:Started 2024-06-03T08:35:04.587 INFO:tasks.ceph:Setting crush tunables to default 2024-06-03T08:35:04.588 DEBUG:teuthology.orchestra.run.smithi088:> sudo ceph --cluster ceph osd crush tunables default 2024-06-03T08:35:04.685 INFO:teuthology.orchestra.run.smithi088.stderr:2024-06-03T08:35:04.685+0000 7ff30f8d5640 10 monclient: get_monmap_and_config 2024-06-03T08:35:04.686 INFO:teuthology.orchestra.run.smithi088.stderr:2024-06-03T08:35:04.685+0000 7ff30f8d5640 10 monclient: build_initial_monmap 2024-06-03T08:35:04.686 INFO:teuthology.orchestra.run.smithi088.stderr:2024-06-03T08:35:04.685+0000 7ff30f8d5640 10 monclient: monmap: 2024-06-03T08:35:04.686 INFO:teuthology.orchestra.run.smithi088.stderr:epoch 0 2024-06-03T08:35:04.686 INFO:teuthology.orchestra.run.smithi088.stderr:fsid ec3c3fdd-3ac6-4d7d-9412-683130ed8b04 2024-06-03T08:35:04.686 INFO:teuthology.orchestra.run.smithi088.stderr:last_changed 2024-06-03T08:35:04.686556+0000 2024-06-03T08:35:04.686 INFO:teuthology.orchestra.run.smithi088.stderr:created 2024-06-03T08:35:04.686556+0000 2024-06-03T08:35:04.686 INFO:teuthology.orchestra.run.smithi088.stderr:min_mon_release 0 (unknown) 2024-06-03T08:35:04.686 INFO:teuthology.orchestra.run.smithi088.stderr:election_strategy: 3 2024-06-03T08:35:04.686 INFO:teuthology.orchestra.run.smithi088.stderr:0: v1:172.21.15.88:6789/0 mon.noname-a 2024-06-03T08:35:04.686 INFO:teuthology.orchestra.run.smithi088.stderr:1: v1:172.21.15.192:6789/0 mon.noname-c 2024-06-03T08:35:04.687 INFO:teuthology.orchestra.run.smithi088.stderr:2: v1:172.21.15.88:6790/0 mon.noname-b 2024-06-03T08:35:04.687 INFO:teuthology.orchestra.run.smithi088.stderr: 2024-06-03T08:35:04.688 INFO:teuthology.orchestra.run.smithi088.stderr:2024-06-03T08:35:04.686+0000 7ff30f8d5640 1 Processor -- start 2024-06-03T08:35:04.688 INFO:teuthology.orchestra.run.smithi088.stderr:2024-06-03T08:35:04.686+0000 7ff30f8d5640 1 -- start start 2024-06-03T08:35:04.688 INFO:teuthology.orchestra.run.smithi088.stderr:2024-06-03T08:35:04.686+0000 7ff30f8d5640 10 monclient: init 2024-06-03T08:35:04.688 INFO:teuthology.orchestra.run.smithi088.stderr:2024-06-03T08:35:04.686+0000 7ff30f8d5640 10 monclient: _reopen_session rank -1 2024-06-03T08:35:04.688 INFO:teuthology.orchestra.run.smithi088.stderr:2024-06-03T08:35:04.686+0000 7ff30f8d5640 10 monclient: _add_conns ranks=[1,0,2] 2024-06-03T08:35:04.688 INFO:teuthology.orchestra.run.smithi088.stderr:2024-06-03T08:35:04.686+0000 7ff30f8d5640 10 monclient(hunting): picked mon.noname-c con 0x7ff3080a8960 addr v1:172.21.15.192:6789/0 2024-06-03T08:35:04.688 INFO:teuthology.orchestra.run.smithi088.stderr:2024-06-03T08:35:04.686+0000 7ff30f8d5640 10 monclient(hunting): picked mon.noname-a con 0x7ff3081450f0 addr v1:172.21.15.88:6789/0 2024-06-03T08:35:04.689 INFO:teuthology.orchestra.run.smithi088.stderr:2024-06-03T08:35:04.686+0000 7ff30f8d5640 10 monclient(hunting): picked mon.noname-b con 0x7ff30813eec0 addr v1:172.21.15.88:6790/0 2024-06-03T08:35:04.689 INFO:teuthology.orchestra.run.smithi088.stderr:2024-06-03T08:35:04.686+0000 7ff30f8d5640 1 -- --> v1:172.21.15.88:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7ff308144c60 con 0x7ff3081450f0 2024-06-03T08:35:04.689 INFO:teuthology.orchestra.run.smithi088.stderr:2024-06-03T08:35:04.686+0000 7ff30f8d5640 1 -- --> v1:172.21.15.192:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7ff3080b3930 con 0x7ff3080a8960 2024-06-03T08:35:04.689 INFO:teuthology.orchestra.run.smithi088.stderr:2024-06-03T08:35:04.686+0000 7ff30f8d5640 1 -- --> v1:172.21.15.88:6790/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7ff3080b1ad0 con 0x7ff30813eec0 2024-06-03T08:35:04.689 INFO:teuthology.orchestra.run.smithi088.stderr:2024-06-03T08:35:04.686+0000 7ff30f8d5640 10 monclient(hunting): _renew_subs 2024-06-03T08:35:04.689 INFO:teuthology.orchestra.run.smithi088.stderr:2024-06-03T08:35:04.686+0000 7ff30f8d5640 10 monclient(hunting): authenticate will time out at 1160.496917s 2024-06-03T08:35:04.689 INFO:teuthology.orchestra.run.smithi088.stderr:2024-06-03T08:35:04.687+0000 7ff30de72640 1 -- >> v1:172.21.15.88:6790/0 conn(0x7ff30813eec0 legacy=0x7ff30814ce60 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v1:172.21.15.88:6790/0 2024-06-03T08:35:04.689 INFO:teuthology.orchestra.run.smithi088.stderr:2024-06-03T08:35:04.686+0000 7ff30ce70640 1 -- >> v1:172.21.15.88:6789/0 conn(0x7ff3081450f0 legacy=0x7ff30813e7b0 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v1:172.21.15.88:6789/0 2024-06-03T08:35:04.689 INFO:teuthology.orchestra.run.smithi088.stderr:2024-06-03T08:35:04.687+0000 7ff30d671640 1 -- >> v1:172.21.15.192:6789/0 conn(0x7ff3080a8960 legacy=0x7ff3080a8d40 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v1:172.21.15.192:6789/0 2024-06-03T08:35:04.888 INFO:teuthology.orchestra.run.smithi088.stderr:2024-06-03T08:35:04.888+0000 7ff30de72640 1 -- >> v1:172.21.15.88:6790/0 conn(0x7ff30813eec0 legacy=0x7ff30814ce60 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v1:172.21.15.88:6790/0 2024-06-03T08:35:04.888 INFO:teuthology.orchestra.run.smithi088.stderr:2024-06-03T08:35:04.888+0000 7ff30ce70640 1 -- >> v1:172.21.15.88:6789/0 conn(0x7ff3081450f0 legacy=0x7ff30813e7b0 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v1:172.21.15.88:6789/0 2024-06-03T08:35:04.888 INFO:teuthology.orchestra.run.smithi088.stderr:2024-06-03T08:35:04.888+0000 7ff30d671640 1 -- >> v1:172.21.15.192:6789/0 conn(0x7ff3080a8960 legacy=0x7ff3080a8d40 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v1:172.21.15.192:6789/0 2024-06-03T08:35:05.289 INFO:teuthology.orchestra.run.smithi088.stderr:2024-06-03T08:35:05.289+0000 7ff30de72640 1 -- >> v1:172.21.15.88:6790/0 conn(0x7ff30813eec0 legacy=0x7ff30814ce60 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v1:172.21.15.88:6790/0 2024-06-03T08:35:05.289 INFO:teuthology.orchestra.run.smithi088.stderr:2024-06-03T08:35:05.289+0000 7ff30ce70640 1 -- >> v1:172.21.15.88:6789/0 conn(0x7ff3081450f0 legacy=0x7ff30813e7b0 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v1:172.21.15.88:6789/0 2024-06-03T08:35:05.289 INFO:teuthology.orchestra.run.smithi088.stderr:2024-06-03T08:35:05.289+0000 7ff30d671640 1 -- >> v1:172.21.15.192:6789/0 conn(0x7ff3080a8960 legacy=0x7ff3080a8d40 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v1:172.21.15.192:6789/0 2024-06-03T08:35:06.090 INFO:teuthology.orchestra.run.smithi088.stderr:2024-06-03T08:35:06.090+0000 7ff30de72640 1 -- >> v1:172.21.15.88:6790/0 conn(0x7ff30813eec0 legacy=0x7ff30814ce60 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v1:172.21.15.88:6790/0 2024-06-03T08:35:06.090 INFO:teuthology.orchestra.run.smithi088.stderr:2024-06-03T08:35:06.090+0000 7ff30ce70640 1 -- >> v1:172.21.15.88:6789/0 conn(0x7ff3081450f0 legacy=0x7ff30813e7b0 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v1:172.21.15.88:6789/0 2024-06-03T08:35:06.091 INFO:teuthology.orchestra.run.smithi088.stderr:2024-06-03T08:35:06.090+0000 7ff30d671640 1 -- >> v1:172.21.15.192:6789/0 conn(0x7ff3080a8960 legacy=0x7ff3080a8d40 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v1:172.21.15.192:6789/0 2024-06-03T08:35:07.688 INFO:teuthology.orchestra.run.smithi088.stderr:2024-06-03T08:35:07.686+0000 7ff2feffd640 10 monclient(hunting): tick 2024-06-03T08:35:07.688 INFO:teuthology.orchestra.run.smithi088.stderr:2024-06-03T08:35:07.687+0000 7ff2feffd640 10 monclient(hunting): _check_auth_tickets 2024-06-03T08:35:07.688 INFO:teuthology.orchestra.run.smithi088.stderr:2024-06-03T08:35:07.687+0000 7ff2feffd640 1 monclient(hunting): continuing hunt 2024-06-03T08:35:07.688 INFO:teuthology.orchestra.run.smithi088.stderr:2024-06-03T08:35:07.687+0000 7ff2feffd640 10 monclient(hunting): _reopen_session rank -1 2024-06-03T08:35:07.688 INFO:teuthology.orchestra.run.smithi088.stderr:2024-06-03T08:35:07.687+0000 7ff2feffd640 1 -- >> v1:172.21.15.88:6790/0 conn(0x7ff30813eec0 legacy=0x7ff30814ce60 unknown :-1 s=STATE_CONNECTING l=0).mark_down 2024-06-03T08:35:07.688 INFO:teuthology.orchestra.run.smithi088.stderr:2024-06-03T08:35:07.687+0000 7ff2feffd640 1 -- >> v1:172.21.15.192:6789/0 conn(0x7ff3080a8960 legacy=0x7ff3080a8d40 unknown :-1 s=STATE_CONNECTING l=0).mark_down 2024-06-03T08:35:07.688 INFO:teuthology.orchestra.run.smithi088.stderr:2024-06-03T08:35:07.687+0000 7ff2feffd640 1 -- >> v1:172.21.15.88:6789/0 conn(0x7ff3081450f0 legacy=0x7ff30813e7b0 unknown :-1 s=STATE_CONNECTING l=0).mark_down 2024-06-03T08:35:07.688 INFO:teuthology.orchestra.run.smithi088.stderr:2024-06-03T08:35:07.687+0000 7ff2feffd640 10 monclient: _add_conns ranks=[0,2,1] 2024-06-03T08:35:07.688 INFO:teuthology.orchestra.run.smithi088.stderr:2024-06-03T08:35:07.687+0000 7ff2feffd640 10 monclient(hunting): picked mon.noname-a con 0x7ff2ec003680 addr v1:172.21.15.88:6789/0 2024-06-03T08:35:07.689 INFO:teuthology.orchestra.run.smithi088.stderr:2024-06-03T08:35:07.687+0000 7ff2feffd640 10 monclient(hunting): picked mon.noname-b con 0x7ff30813eec0 addr v1:172.21.15.88:6790/0 2024-06-03T08:35:07.689 INFO:teuthology.orchestra.run.smithi088.stderr:2024-06-03T08:35:07.687+0000 7ff2feffd640 10 monclient(hunting): picked mon.noname-c con 0x7ff3080a8960 addr v1:172.21.15.192:6789/0 2024-06-03T08:35:07.689 INFO:teuthology.orchestra.run.smithi088.stderr:2024-06-03T08:35:07.687+0000 7ff2feffd640 1 -- --> v1:172.21.15.88:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7ff3080b1c60 con 0x7ff2ec003680 2024-06-03T08:35:07.689 INFO:teuthology.orchestra.run.smithi088.stderr:2024-06-03T08:35:07.687+0000 7ff2feffd640 1 -- --> v1:172.21.15.192:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7ff308144c60 con 0x7ff3080a8960 2024-06-03T08:35:07.689 INFO:teuthology.orchestra.run.smithi088.stderr:2024-06-03T08:35:07.687+0000 7ff2feffd640 1 -- --> v1:172.21.15.88:6790/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7ff3080b3930 con 0x7ff30813eec0 2024-06-03T08:35:07.689 INFO:teuthology.orchestra.run.smithi088.stderr:2024-06-03T08:35:07.687+0000 7ff30d671640 1 -- >> v1:172.21.15.88:6789/0 conn(0x7ff2ec003680 legacy=0x7ff2ec005ad0 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v1:172.21.15.88:6789/0 2024-06-03T08:35:07.689 INFO:teuthology.orchestra.run.smithi088.stderr:2024-06-03T08:35:07.687+0000 7ff2feffd640 10 monclient(hunting): _renew_subs 2024-06-03T08:35:07.689 INFO:teuthology.orchestra.run.smithi088.stderr:2024-06-03T08:35:07.687+0000 7ff30ce70640 1 -- >> v1:172.21.15.88:6790/0 conn(0x7ff30813eec0 legacy=0x7ff2ec009330 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v1:172.21.15.88:6790/0 2024-06-03T08:35:07.689 INFO:teuthology.orchestra.run.smithi088.stderr:2024-06-03T08:35:07.687+0000 7ff30de72640 1 -- >> v1:172.21.15.192:6789/0 conn(0x7ff3080a8960 legacy=0x7ff2ec00cb30 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v1:172.21.15.192:6789/0 2024-06-03T08:35:07.888 INFO:teuthology.orchestra.run.smithi088.stderr:2024-06-03T08:35:07.888+0000 7ff30d671640 1 -- >> v1:172.21.15.88:6789/0 conn(0x7ff2ec003680 legacy=0x7ff2ec005ad0 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v1:172.21.15.88:6789/0 2024-06-03T08:35:07.888 INFO:teuthology.orchestra.run.smithi088.stderr:2024-06-03T08:35:07.888+0000 7ff30ce70640 1 -- >> v1:172.21.15.88:6790/0 conn(0x7ff30813eec0 legacy=0x7ff2ec009330 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v1:172.21.15.88:6790/0 2024-06-03T08:35:07.888 INFO:teuthology.orchestra.run.smithi088.stderr:2024-06-03T08:35:07.888+0000 7ff30de72640 1 -- >> v1:172.21.15.192:6789/0 conn(0x7ff3080a8960 legacy=0x7ff2ec00cb30 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v1:172.21.15.192:6789/0 2024-06-03T08:35:08.289 INFO:teuthology.orchestra.run.smithi088.stderr:2024-06-03T08:35:08.289+0000 7ff30d671640 1 -- >> v1:172.21.15.88:6789/0 conn(0x7ff2ec003680 legacy=0x7ff2ec005ad0 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v1:172.21.15.88:6789/0 2024-06-03T08:35:08.289 INFO:teuthology.orchestra.run.smithi088.stderr:2024-06-03T08:35:08.289+0000 7ff30ce70640 1 -- >> v1:172.21.15.88:6790/0 conn(0x7ff30813eec0 legacy=0x7ff2ec009330 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v1:172.21.15.88:6790/0 2024-06-03T08:35:08.289 INFO:teuthology.orchestra.run.smithi088.stderr:2024-06-03T08:35:08.289+0000 7ff30de72640 1 -- >> v1:172.21.15.192:6789/0 conn(0x7ff3080a8960 legacy=0x7ff2ec00cb30 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v1:172.21.15.192:6789/0 2024-06-03T08:35:09.090 INFO:teuthology.orchestra.run.smithi088.stderr:2024-06-03T08:35:09.090+0000 7ff30d671640 1 -- >> v1:172.21.15.88:6789/0 conn(0x7ff2ec003680 legacy=0x7ff2ec005ad0 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v1:172.21.15.88:6789/0 2024-06-03T08:35:09.090 INFO:teuthology.orchestra.run.smithi088.stderr:2024-06-03T08:35:09.090+0000 7ff30ce70640 1 -- >> v1:172.21.15.88:6790/0 conn(0x7ff30813eec0 legacy=0x7ff2ec009330 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v1:172.21.15.88:6790/0 2024-06-03T08:35:09.090 INFO:teuthology.orchestra.run.smithi088.stderr:2024-06-03T08:35:09.090+0000 7ff30de72640 1 -- >> v1:172.21.15.192:6789/0 conn(0x7ff3080a8960 legacy=0x7ff2ec00cb30 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v1:172.21.15.192:6789/0 2024-06-03T08:35:10.688 INFO:teuthology.orchestra.run.smithi088.stderr:2024-06-03T08:35:10.687+0000 7ff2feffd640 10 monclient(hunting): tick 2024-06-03T08:35:10.688 INFO:teuthology.orchestra.run.smithi088.stderr:2024-06-03T08:35:10.687+0000 7ff2feffd640 10 monclient(hunting): _check_auth_tickets 2024-06-03T08:35:10.688 INFO:teuthology.orchestra.run.smithi088.stderr:2024-06-03T08:35:10.687+0000 7ff2feffd640 1 monclient(hunting): continuing hunt 2024-06-03T08:35:10.688 INFO:teuthology.orchestra.run.smithi088.stderr:2024-06-03T08:35:10.687+0000 7ff2feffd640 10 monclient(hunting): _reopen_session rank -1 2024-06-03T08:35:10.688 INFO:teuthology.orchestra.run.smithi088.stderr:2024-06-03T08:35:10.687+0000 7ff2feffd640 1 -- >> v1:172.21.15.88:6790/0 conn(0x7ff30813eec0 legacy=0x7ff2ec009330 unknown :-1 s=STATE_CONNECTING l=0).mark_down 2024-06-03T08:35:10.688 INFO:teuthology.orchestra.run.smithi088.stderr:2024-06-03T08:35:10.687+0000 7ff2feffd640 1 -- >> v1:172.21.15.192:6789/0 conn(0x7ff3080a8960 legacy=0x7ff2ec00cb30 unknown :-1 s=STATE_CONNECTING l=0).mark_down 2024-06-03T08:35:10.688 INFO:teuthology.orchestra.run.smithi088.stderr:2024-06-03T08:35:10.687+0000 7ff2feffd640 1 -- >> v1:172.21.15.88:6789/0 conn(0x7ff2ec003680 legacy=0x7ff2ec005ad0 unknown :-1 s=STATE_CONNECTING l=0).mark_down 2024-06-03T08:35:10.688 INFO:teuthology.orchestra.run.smithi088.stderr:2024-06-03T08:35:10.687+0000 7ff2feffd640 10 monclient: _add_conns ranks=[1,0,2] 2024-06-03T08:35:10.688 INFO:teuthology.orchestra.run.smithi088.stderr:2024-06-03T08:35:10.687+0000 7ff2feffd640 10 monclient(hunting): picked mon.noname-c con 0x7ff3080a8960 addr v1:172.21.15.192:6789/0 2024-06-03T08:35:10.689 INFO:teuthology.orchestra.run.smithi088.stderr:2024-06-03T08:35:10.687+0000 7ff2feffd640 10 monclient(hunting): picked mon.noname-a con 0x7ff2ec003680 addr v1:172.21.15.88:6789/0 2024-06-03T08:35:10.689 INFO:teuthology.orchestra.run.smithi088.stderr:2024-06-03T08:35:10.687+0000 7ff2feffd640 10 monclient(hunting): picked mon.noname-b con 0x7ff30813eec0 addr v1:172.21.15.88:6790/0 2024-06-03T08:35:10.689 INFO:teuthology.orchestra.run.smithi088.stderr:2024-06-03T08:35:10.687+0000 7ff2feffd640 1 -- --> v1:172.21.15.88:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7ff3080b1ad0 con 0x7ff2ec003680 2024-06-03T08:35:10.689 INFO:teuthology.orchestra.run.smithi088.stderr:2024-06-03T08:35:10.687+0000 7ff2feffd640 1 -- --> v1:172.21.15.192:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7ff3080b1c60 con 0x7ff3080a8960 2024-06-03T08:35:10.689 INFO:teuthology.orchestra.run.smithi088.stderr:2024-06-03T08:35:10.687+0000 7ff2feffd640 1 -- --> v1:172.21.15.88:6790/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7ff308144c60 con 0x7ff30813eec0 2024-06-03T08:35:10.689 INFO:teuthology.orchestra.run.smithi088.stderr:2024-06-03T08:35:10.687+0000 7ff2feffd640 10 monclient(hunting): _renew_subs 2024-06-03T08:35:10.689 INFO:teuthology.orchestra.run.smithi088.stderr:2024-06-03T08:35:10.687+0000 7ff30ce70640 1 -- >> v1:172.21.15.88:6789/0 conn(0x7ff2ec003680 legacy=0x7ff2ec005a70 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v1:172.21.15.88:6789/0 2024-06-03T08:35:10.689 INFO:teuthology.orchestra.run.smithi088.stderr:2024-06-03T08:35:10.687+0000 7ff30de72640 1 -- >> v1:172.21.15.88:6790/0 conn(0x7ff30813eec0 legacy=0x7ff2ec009330 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v1:172.21.15.88:6790/0 2024-06-03T08:35:10.689 INFO:teuthology.orchestra.run.smithi088.stderr:2024-06-03T08:35:10.687+0000 7ff30d671640 1 -- >> v1:172.21.15.192:6789/0 conn(0x7ff3080a8960 legacy=0x7ff2ec00cab0 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v1:172.21.15.192:6789/0 2024-06-03T08:35:10.888 INFO:teuthology.orchestra.run.smithi088.stderr:2024-06-03T08:35:10.888+0000 7ff30de72640 1 -- >> v1:172.21.15.88:6790/0 conn(0x7ff30813eec0 legacy=0x7ff2ec009330 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v1:172.21.15.88:6790/0 2024-06-03T08:35:10.888 INFO:teuthology.orchestra.run.smithi088.stderr:2024-06-03T08:35:10.888+0000 7ff30ce70640 1 -- >> v1:172.21.15.88:6789/0 conn(0x7ff2ec003680 legacy=0x7ff2ec005a70 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v1:172.21.15.88:6789/0 2024-06-03T08:35:10.888 INFO:teuthology.orchestra.run.smithi088.stderr:2024-06-03T08:35:10.888+0000 7ff30d671640 1 -- >> v1:172.21.15.192:6789/0 conn(0x7ff3080a8960 legacy=0x7ff2ec00cab0 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v1:172.21.15.192:6789/0 2024-06-03T08:35:11.289 INFO:teuthology.orchestra.run.smithi088.stderr:2024-06-03T08:35:11.289+0000 7ff30de72640 1 -- >> v1:172.21.15.88:6790/0 conn(0x7ff30813eec0 legacy=0x7ff2ec009330 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v1:172.21.15.88:6790/0 2024-06-03T08:35:11.289 INFO:teuthology.orchestra.run.smithi088.stderr:2024-06-03T08:35:11.289+0000 7ff30ce70640 1 -- >> v1:172.21.15.88:6789/0 conn(0x7ff2ec003680 legacy=0x7ff2ec005a70 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v1:172.21.15.88:6789/0 2024-06-03T08:35:11.289 INFO:teuthology.orchestra.run.smithi088.stderr:2024-06-03T08:35:11.289+0000 7ff30d671640 1 -- >> v1:172.21.15.192:6789/0 conn(0x7ff3080a8960 legacy=0x7ff2ec00cab0 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v1:172.21.15.192:6789/0 2024-06-03T08:35:11.549 INFO:tasks.ceph.mon.b.smithi192.stderr:==00:00:00:06.851 34228== Warning: unimplemented fcntl command: 1036 2024-06-03T08:35:11.678 INFO:tasks.ceph.mon.a.smithi088.stderr:==00:00:00:07.003 34743== Warning: unimplemented fcntl command: 1036 2024-06-03T08:35:11.678 INFO:tasks.ceph.mon.c.smithi088.stderr:==00:00:00:07.003 34749== Warning: unimplemented fcntl command: 1036 2024-06-03T08:35:11.853 INFO:tasks.ceph.mon.b.smithi192.stderr:==00:00:00:07.155 34228== Warning: unimplemented fcntl command: 1036 2024-06-03T08:35:11.997 INFO:tasks.ceph.mon.a.smithi088.stderr:==00:00:00:07.322 34743== Warning: unimplemented fcntl command: 1036 2024-06-03T08:35:11.997 INFO:tasks.ceph.mon.c.smithi088.stderr:==00:00:00:07.322 34749== Warning: unimplemented fcntl command: 1036 2024-06-03T08:35:12.090 INFO:teuthology.orchestra.run.smithi088.stderr:2024-06-03T08:35:12.090+0000 7ff30de72640 1 -- >> v1:172.21.15.88:6790/0 conn(0x7ff30813eec0 legacy=0x7ff2ec009330 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v1:172.21.15.88:6790/0 2024-06-03T08:35:12.090 INFO:teuthology.orchestra.run.smithi088.stderr:2024-06-03T08:35:12.090+0000 7ff30ce70640 1 -- >> v1:172.21.15.88:6789/0 conn(0x7ff2ec003680 legacy=0x7ff2ec005a70 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v1:172.21.15.88:6789/0 2024-06-03T08:35:12.090 INFO:teuthology.orchestra.run.smithi088.stderr:2024-06-03T08:35:12.090+0000 7ff30d671640 1 -- >> v1:172.21.15.192:6789/0 conn(0x7ff3080a8960 legacy=0x7ff2ec00cab0 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v1:172.21.15.192:6789/0 2024-06-03T08:35:13.688 INFO:teuthology.orchestra.run.smithi088.stderr:2024-06-03T08:35:13.687+0000 7ff2feffd640 10 monclient(hunting): tick 2024-06-03T08:35:13.688 INFO:teuthology.orchestra.run.smithi088.stderr:2024-06-03T08:35:13.687+0000 7ff2feffd640 10 monclient(hunting): _check_auth_tickets 2024-06-03T08:35:13.688 INFO:teuthology.orchestra.run.smithi088.stderr:2024-06-03T08:35:13.687+0000 7ff2feffd640 1 monclient(hunting): continuing hunt 2024-06-03T08:35:13.688 INFO:teuthology.orchestra.run.smithi088.stderr:2024-06-03T08:35:13.687+0000 7ff2feffd640 10 monclient(hunting): _reopen_session rank -1 2024-06-03T08:35:13.688 INFO:teuthology.orchestra.run.smithi088.stderr:2024-06-03T08:35:13.687+0000 7ff2feffd640 1 -- >> v1:172.21.15.88:6790/0 conn(0x7ff30813eec0 legacy=0x7ff2ec009330 unknown :-1 s=STATE_CONNECTING l=0).mark_down 2024-06-03T08:35:13.688 INFO:teuthology.orchestra.run.smithi088.stderr:2024-06-03T08:35:13.687+0000 7ff2feffd640 1 -- >> v1:172.21.15.192:6789/0 conn(0x7ff3080a8960 legacy=0x7ff2ec00cab0 unknown :-1 s=STATE_CONNECTING l=0).mark_down 2024-06-03T08:35:13.688 INFO:teuthology.orchestra.run.smithi088.stderr:2024-06-03T08:35:13.687+0000 7ff2feffd640 1 -- >> v1:172.21.15.88:6789/0 conn(0x7ff2ec003680 legacy=0x7ff2ec005a70 unknown :-1 s=STATE_CONNECTING l=0).mark_down 2024-06-03T08:35:13.688 INFO:teuthology.orchestra.run.smithi088.stderr:2024-06-03T08:35:13.687+0000 7ff2feffd640 10 monclient: _add_conns ranks=[0,1,2] 2024-06-03T08:35:13.688 INFO:teuthology.orchestra.run.smithi088.stderr:2024-06-03T08:35:13.687+0000 7ff2feffd640 10 monclient(hunting): picked mon.noname-a con 0x7ff2ec00f410 addr v1:172.21.15.88:6789/0 2024-06-03T08:35:13.688 INFO:teuthology.orchestra.run.smithi088.stderr:2024-06-03T08:35:13.687+0000 7ff2feffd640 10 monclient(hunting): picked mon.noname-c con 0x7ff3080a8960 addr v1:172.21.15.192:6789/0 2024-06-03T08:35:13.688 INFO:teuthology.orchestra.run.smithi088.stderr:2024-06-03T08:35:13.687+0000 7ff2feffd640 10 monclient(hunting): picked mon.noname-b con 0x7ff30813eec0 addr v1:172.21.15.88:6790/0 2024-06-03T08:35:13.689 INFO:teuthology.orchestra.run.smithi088.stderr:2024-06-03T08:35:13.687+0000 7ff2feffd640 1 -- --> v1:172.21.15.88:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7ff3080b3930 con 0x7ff2ec00f410 2024-06-03T08:35:13.689 INFO:teuthology.orchestra.run.smithi088.stderr:2024-06-03T08:35:13.687+0000 7ff2feffd640 1 -- --> v1:172.21.15.192:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7ff3080b1ad0 con 0x7ff3080a8960 2024-06-03T08:35:13.689 INFO:teuthology.orchestra.run.smithi088.stderr:2024-06-03T08:35:13.687+0000 7ff2feffd640 1 -- --> v1:172.21.15.88:6790/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7ff3080b1c60 con 0x7ff30813eec0 2024-06-03T08:35:13.689 INFO:teuthology.orchestra.run.smithi088.stderr:2024-06-03T08:35:13.687+0000 7ff2feffd640 10 monclient(hunting): _renew_subs 2024-06-03T08:35:13.754 INFO:teuthology.orchestra.run.smithi088.stderr:2024-06-03T08:35:13.753+0000 7ff30ce70640 1 --1- >> v1:172.21.15.192:6789/0 conn(0x7ff3080a8960 0x7ff2ec00cab0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.192:6789/0 says I am v1:172.21.15.88:53438/0 (socket says 172.21.15.88:53438) 2024-06-03T08:35:13.754 INFO:teuthology.orchestra.run.smithi088.stderr:2024-06-03T08:35:13.753+0000 7ff30ce70640 1 -- 172.21.15.88:0/2715150535 learned_addr learned my addr 172.21.15.88:0/2715150535 (peer_addr_for_me v1:172.21.15.88:0/0) 2024-06-03T08:35:16.688 INFO:teuthology.orchestra.run.smithi088.stderr:2024-06-03T08:35:16.687+0000 7ff2feffd640 10 monclient(hunting): tick 2024-06-03T08:35:16.688 INFO:teuthology.orchestra.run.smithi088.stderr:2024-06-03T08:35:16.687+0000 7ff2feffd640 10 monclient(hunting): _check_auth_tickets 2024-06-03T08:35:16.688 INFO:teuthology.orchestra.run.smithi088.stderr:2024-06-03T08:35:16.687+0000 7ff2feffd640 1 monclient(hunting): continuing hunt 2024-06-03T08:35:16.688 INFO:teuthology.orchestra.run.smithi088.stderr:2024-06-03T08:35:16.687+0000 7ff2feffd640 10 monclient(hunting): _reopen_session rank -1 2024-06-03T08:35:16.688 INFO:teuthology.orchestra.run.smithi088.stderr:2024-06-03T08:35:16.687+0000 7ff2feffd640 1 -- 172.21.15.88:0/2715150535 >> v1:172.21.15.88:6790/0 conn(0x7ff30813eec0 legacy=0x7ff2ec009330 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-06-03T08:35:16.689 INFO:teuthology.orchestra.run.smithi088.stderr:2024-06-03T08:35:16.687+0000 7ff2feffd640 1 -- 172.21.15.88:0/2715150535 >> v1:172.21.15.192:6789/0 conn(0x7ff3080a8960 legacy=0x7ff2ec00cab0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-06-03T08:35:16.689 INFO:teuthology.orchestra.run.smithi088.stderr:2024-06-03T08:35:16.687+0000 7ff2feffd640 1 -- 172.21.15.88:0/2715150535 >> v1:172.21.15.88:6789/0 conn(0x7ff2ec00f410 legacy=0x7ff2ec00e300 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-06-03T08:35:16.689 INFO:teuthology.orchestra.run.smithi088.stderr:2024-06-03T08:35:16.687+0000 7ff2feffd640 10 monclient: _add_conns ranks=[2,0,1] 2024-06-03T08:35:16.689 INFO:teuthology.orchestra.run.smithi088.stderr:2024-06-03T08:35:16.687+0000 7ff2feffd640 10 monclient(hunting): picked mon.noname-b con 0x7ff2ec00ec20 addr v1:172.21.15.88:6790/0 2024-06-03T08:35:16.689 INFO:teuthology.orchestra.run.smithi088.stderr:2024-06-03T08:35:16.687+0000 7ff2feffd640 10 monclient(hunting): picked mon.noname-a con 0x7ff2ec010800 addr v1:172.21.15.88:6789/0 2024-06-03T08:35:16.689 INFO:teuthology.orchestra.run.smithi088.stderr:2024-06-03T08:35:16.687+0000 7ff2feffd640 10 monclient(hunting): picked mon.noname-c con 0x7ff3080a8960 addr v1:172.21.15.192:6789/0 2024-06-03T08:35:16.689 INFO:teuthology.orchestra.run.smithi088.stderr:2024-06-03T08:35:16.687+0000 7ff2feffd640 1 -- 172.21.15.88:0/2715150535 --> v1:172.21.15.88:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7ff308144c60 con 0x7ff2ec010800 2024-06-03T08:35:16.689 INFO:teuthology.orchestra.run.smithi088.stderr:2024-06-03T08:35:16.687+0000 7ff2feffd640 1 -- 172.21.15.88:0/2715150535 --> v1:172.21.15.192:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7ff2ec010c80 con 0x7ff3080a8960 2024-06-03T08:35:16.689 INFO:teuthology.orchestra.run.smithi088.stderr:2024-06-03T08:35:16.687+0000 7ff2feffd640 1 -- 172.21.15.88:0/2715150535 --> v1:172.21.15.88:6790/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7ff2ec010e60 con 0x7ff2ec00ec20 2024-06-03T08:35:16.689 INFO:teuthology.orchestra.run.smithi088.stderr:2024-06-03T08:35:16.687+0000 7ff2feffd640 10 monclient(hunting): _renew_subs 2024-06-03T08:35:19.689 INFO:teuthology.orchestra.run.smithi088.stderr:2024-06-03T08:35:19.687+0000 7ff2feffd640 10 monclient(hunting): tick 2024-06-03T08:35:19.689 INFO:teuthology.orchestra.run.smithi088.stderr:2024-06-03T08:35:19.687+0000 7ff2feffd640 10 monclient(hunting): _check_auth_tickets 2024-06-03T08:35:19.689 INFO:teuthology.orchestra.run.smithi088.stderr:2024-06-03T08:35:19.687+0000 7ff2feffd640 1 monclient(hunting): continuing hunt 2024-06-03T08:35:19.689 INFO:teuthology.orchestra.run.smithi088.stderr:2024-06-03T08:35:19.687+0000 7ff2feffd640 10 monclient(hunting): _reopen_session rank -1 2024-06-03T08:35:19.689 INFO:teuthology.orchestra.run.smithi088.stderr:2024-06-03T08:35:19.687+0000 7ff2feffd640 1 -- 172.21.15.88:0/2715150535 >> v1:172.21.15.88:6790/0 conn(0x7ff2ec00ec20 legacy=0x7ff2ec005a70 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-06-03T08:35:19.689 INFO:teuthology.orchestra.run.smithi088.stderr:2024-06-03T08:35:19.687+0000 7ff2feffd640 1 -- 172.21.15.88:0/2715150535 >> v1:172.21.15.192:6789/0 conn(0x7ff3080a8960 legacy=0x7ff2ec00e300 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-06-03T08:35:19.689 INFO:teuthology.orchestra.run.smithi088.stderr:2024-06-03T08:35:19.688+0000 7ff2feffd640 1 -- 172.21.15.88:0/2715150535 >> v1:172.21.15.88:6789/0 conn(0x7ff2ec010800 legacy=0x7ff2ec0154d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-06-03T08:35:19.690 INFO:teuthology.orchestra.run.smithi088.stderr:2024-06-03T08:35:19.688+0000 7ff2feffd640 10 monclient: _add_conns ranks=[1,0,2] 2024-06-03T08:35:19.690 INFO:teuthology.orchestra.run.smithi088.stderr:2024-06-03T08:35:19.688+0000 7ff2feffd640 10 monclient(hunting): picked mon.noname-c con 0x7ff2ec009a60 addr v1:172.21.15.192:6789/0 2024-06-03T08:35:19.690 INFO:teuthology.orchestra.run.smithi088.stderr:2024-06-03T08:35:19.688+0000 7ff2feffd640 10 monclient(hunting): picked mon.noname-a con 0x7ff2ec009ea0 addr v1:172.21.15.88:6789/0 2024-06-03T08:35:19.690 INFO:teuthology.orchestra.run.smithi088.stderr:2024-06-03T08:35:19.688+0000 7ff2feffd640 10 monclient(hunting): picked mon.noname-b con 0x7ff2ec00ec20 addr v1:172.21.15.88:6790/0 2024-06-03T08:35:19.690 INFO:teuthology.orchestra.run.smithi088.stderr:2024-06-03T08:35:19.688+0000 7ff2feffd640 1 -- 172.21.15.88:0/2715150535 --> v1:172.21.15.88:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7ff2ec011040 con 0x7ff2ec009ea0 2024-06-03T08:35:19.690 INFO:teuthology.orchestra.run.smithi088.stderr:2024-06-03T08:35:19.688+0000 7ff2feffd640 1 -- 172.21.15.88:0/2715150535 --> v1:172.21.15.192:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7ff2ec00a3b0 con 0x7ff2ec009a60 2024-06-03T08:35:19.690 INFO:teuthology.orchestra.run.smithi088.stderr:2024-06-03T08:35:19.688+0000 7ff2feffd640 1 -- 172.21.15.88:0/2715150535 --> v1:172.21.15.88:6790/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7ff2ec00a590 con 0x7ff2ec00ec20 2024-06-03T08:35:19.690 INFO:teuthology.orchestra.run.smithi088.stderr:2024-06-03T08:35:19.688+0000 7ff2feffd640 10 monclient(hunting): _renew_subs 2024-06-03T08:35:22.689 INFO:teuthology.orchestra.run.smithi088.stderr:2024-06-03T08:35:22.688+0000 7ff2feffd640 10 monclient(hunting): tick 2024-06-03T08:35:22.689 INFO:teuthology.orchestra.run.smithi088.stderr:2024-06-03T08:35:22.688+0000 7ff2feffd640 10 monclient(hunting): _check_auth_tickets 2024-06-03T08:35:22.689 INFO:teuthology.orchestra.run.smithi088.stderr:2024-06-03T08:35:22.688+0000 7ff2feffd640 1 monclient(hunting): continuing hunt 2024-06-03T08:35:22.689 INFO:teuthology.orchestra.run.smithi088.stderr:2024-06-03T08:35:22.688+0000 7ff2feffd640 10 monclient(hunting): _reopen_session rank -1 2024-06-03T08:35:22.689 INFO:teuthology.orchestra.run.smithi088.stderr:2024-06-03T08:35:22.688+0000 7ff2feffd640 1 -- 172.21.15.88:0/2715150535 >> v1:172.21.15.88:6790/0 conn(0x7ff2ec00ec20 legacy=0x7ff2ec005a70 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-06-03T08:35:22.689 INFO:teuthology.orchestra.run.smithi088.stderr:2024-06-03T08:35:22.688+0000 7ff2feffd640 1 -- 172.21.15.88:0/2715150535 >> v1:172.21.15.192:6789/0 conn(0x7ff2ec009a60 legacy=0x7ff2ec00cab0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-06-03T08:35:22.689 INFO:teuthology.orchestra.run.smithi088.stderr:2024-06-03T08:35:22.688+0000 7ff2feffd640 1 -- 172.21.15.88:0/2715150535 >> v1:172.21.15.88:6789/0 conn(0x7ff2ec009ea0 legacy=0x7ff2ec009330 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-06-03T08:35:22.689 INFO:teuthology.orchestra.run.smithi088.stderr:2024-06-03T08:35:22.688+0000 7ff2feffd640 10 monclient: _add_conns ranks=[0,1,2] 2024-06-03T08:35:22.690 INFO:teuthology.orchestra.run.smithi088.stderr:2024-06-03T08:35:22.688+0000 7ff2feffd640 10 monclient(hunting): picked mon.noname-a con 0x7ff2ec00e300 addr v1:172.21.15.88:6789/0 2024-06-03T08:35:22.690 INFO:teuthology.orchestra.run.smithi088.stderr:2024-06-03T08:35:22.688+0000 7ff2feffd640 10 monclient(hunting): picked mon.noname-c con 0x7ff2ec00d970 addr v1:172.21.15.192:6789/0 2024-06-03T08:35:22.690 INFO:teuthology.orchestra.run.smithi088.stderr:2024-06-03T08:35:22.688+0000 7ff2feffd640 10 monclient(hunting): picked mon.noname-b con 0x7ff2ec00ec20 addr v1:172.21.15.88:6790/0 2024-06-03T08:35:22.690 INFO:teuthology.orchestra.run.smithi088.stderr:2024-06-03T08:35:22.688+0000 7ff2feffd640 1 -- 172.21.15.88:0/2715150535 --> v1:172.21.15.88:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7ff2ec00a7b0 con 0x7ff2ec00e300 2024-06-03T08:35:22.690 INFO:teuthology.orchestra.run.smithi088.stderr:2024-06-03T08:35:22.688+0000 7ff2feffd640 1 -- 172.21.15.88:0/2715150535 --> v1:172.21.15.192:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7ff2ec00ddf0 con 0x7ff2ec00d970 2024-06-03T08:35:22.690 INFO:teuthology.orchestra.run.smithi088.stderr:2024-06-03T08:35:22.688+0000 7ff2feffd640 1 -- 172.21.15.88:0/2715150535 --> v1:172.21.15.88:6790/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7ff2ec00df80 con 0x7ff2ec00ec20 2024-06-03T08:35:22.690 INFO:teuthology.orchestra.run.smithi088.stderr:2024-06-03T08:35:22.688+0000 7ff2feffd640 10 monclient(hunting): _renew_subs 2024-06-03T08:35:25.691 INFO:teuthology.orchestra.run.smithi088.stderr:2024-06-03T08:35:25.688+0000 7ff2feffd640 10 monclient(hunting): tick 2024-06-03T08:35:25.691 INFO:teuthology.orchestra.run.smithi088.stderr:2024-06-03T08:35:25.688+0000 7ff2feffd640 10 monclient(hunting): _check_auth_tickets 2024-06-03T08:35:25.691 INFO:teuthology.orchestra.run.smithi088.stderr:2024-06-03T08:35:25.688+0000 7ff2feffd640 1 monclient(hunting): continuing hunt 2024-06-03T08:35:25.691 INFO:teuthology.orchestra.run.smithi088.stderr:2024-06-03T08:35:25.688+0000 7ff2feffd640 10 monclient(hunting): _reopen_session rank -1 2024-06-03T08:35:25.691 INFO:teuthology.orchestra.run.smithi088.stderr:2024-06-03T08:35:25.688+0000 7ff2feffd640 1 -- 172.21.15.88:0/2715150535 >> v1:172.21.15.88:6790/0 conn(0x7ff2ec00ec20 legacy=0x7ff2ec005a70 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-06-03T08:35:25.691 INFO:teuthology.orchestra.run.smithi088.stderr:2024-06-03T08:35:25.688+0000 7ff2feffd640 1 -- 172.21.15.88:0/2715150535 >> v1:172.21.15.192:6789/0 conn(0x7ff2ec00d970 legacy=0x7ff2ec0164e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-06-03T08:35:25.691 INFO:teuthology.orchestra.run.smithi088.stderr:2024-06-03T08:35:25.688+0000 7ff2feffd640 1 -- 172.21.15.88:0/2715150535 >> v1:172.21.15.88:6789/0 conn(0x7ff2ec00e300 legacy=0x7ff2ec00d260 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-06-03T08:35:25.691 INFO:teuthology.orchestra.run.smithi088.stderr:2024-06-03T08:35:25.688+0000 7ff2feffd640 10 monclient: _add_conns ranks=[2,0,1] 2024-06-03T08:35:25.691 INFO:teuthology.orchestra.run.smithi088.stderr:2024-06-03T08:35:25.688+0000 7ff2feffd640 10 monclient(hunting): picked mon.noname-b con 0x7ff2ec00f7f0 addr v1:172.21.15.88:6790/0 2024-06-03T08:35:25.691 INFO:teuthology.orchestra.run.smithi088.stderr:2024-06-03T08:35:25.688+0000 7ff2feffd640 10 monclient(hunting): picked mon.noname-a con 0x7ff2ec00fbd0 addr v1:172.21.15.88:6789/0 2024-06-03T08:35:25.691 INFO:teuthology.orchestra.run.smithi088.stderr:2024-06-03T08:35:25.688+0000 7ff2feffd640 10 monclient(hunting): picked mon.noname-c con 0x7ff2ec00d970 addr v1:172.21.15.192:6789/0 2024-06-03T08:35:25.692 INFO:teuthology.orchestra.run.smithi088.stderr:2024-06-03T08:35:25.688+0000 7ff2feffd640 1 -- 172.21.15.88:0/2715150535 --> v1:172.21.15.88:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7ff2ec008330 con 0x7ff2ec00fbd0 2024-06-03T08:35:25.692 INFO:teuthology.orchestra.run.smithi088.stderr:2024-06-03T08:35:25.688+0000 7ff2feffd640 1 -- 172.21.15.88:0/2715150535 --> v1:172.21.15.192:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7ff2ec00d260 con 0x7ff2ec00d970 2024-06-03T08:35:25.692 INFO:teuthology.orchestra.run.smithi088.stderr:2024-06-03T08:35:25.688+0000 7ff2feffd640 1 -- 172.21.15.88:0/2715150535 --> v1:172.21.15.88:6790/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7ff2ec009550 con 0x7ff2ec00f7f0 2024-06-03T08:35:25.692 INFO:teuthology.orchestra.run.smithi088.stderr:2024-06-03T08:35:25.688+0000 7ff2feffd640 10 monclient(hunting): _renew_subs 2024-06-03T08:35:27.676 INFO:teuthology.orchestra.run.smithi088.stderr:2024-06-03T08:35:27.675+0000 7ff2fffff640 1 -- 172.21.15.88:0/2715150535 <== mon.0 v1:172.21.15.88:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 1672027186 0 0) 0x7ff2ec008330 con 0x7ff2ec00fbd0 2024-06-03T08:35:27.676 INFO:teuthology.orchestra.run.smithi088.stderr:2024-06-03T08:35:27.675+0000 7ff2fffff640 10 monclient(hunting): _negotiate 2024-06-03T08:35:27.676 INFO:teuthology.orchestra.run.smithi088.stderr:2024-06-03T08:35:27.675+0000 7ff2fffff640 10 monclient(hunting): _init_auth method 2 2024-06-03T08:35:27.676 INFO:teuthology.orchestra.run.smithi088.stderr:2024-06-03T08:35:27.675+0000 7ff2fffff640 10 monclient(hunting): _init_auth creating new auth 2024-06-03T08:35:27.676 INFO:teuthology.orchestra.run.smithi088.stderr:2024-06-03T08:35:27.675+0000 7ff2fffff640 10 monclient(hunting): my global_id is 4101 2024-06-03T08:35:27.676 INFO:teuthology.orchestra.run.smithi088.stderr:2024-06-03T08:35:27.675+0000 7ff2fffff640 1 -- 172.21.15.88:0/2715150535 --> v1:172.21.15.88:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7ff2f4003520 con 0x7ff2ec00fbd0 2024-06-03T08:35:27.702 INFO:teuthology.orchestra.run.smithi088.stderr:2024-06-03T08:35:27.702+0000 7ff2fffff640 1 -- 172.21.15.88:0/2715150535 <== mon.1 v1:172.21.15.192:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 243761106 0 0) 0x7ff2ec00d260 con 0x7ff2ec00d970 2024-06-03T08:35:27.702 INFO:teuthology.orchestra.run.smithi088.stderr:2024-06-03T08:35:27.702+0000 7ff2fffff640 10 monclient(hunting): _negotiate 2024-06-03T08:35:27.703 INFO:teuthology.orchestra.run.smithi088.stderr:2024-06-03T08:35:27.702+0000 7ff2fffff640 10 monclient(hunting): _init_auth method 2 2024-06-03T08:35:27.703 INFO:teuthology.orchestra.run.smithi088.stderr:2024-06-03T08:35:27.702+0000 7ff2fffff640 10 monclient(hunting): _init_auth creating new auth 2024-06-03T08:35:27.703 INFO:teuthology.orchestra.run.smithi088.stderr:2024-06-03T08:35:27.702+0000 7ff2fffff640 10 monclient(hunting): my global_id is 4105 2024-06-03T08:35:27.703 INFO:teuthology.orchestra.run.smithi088.stderr:2024-06-03T08:35:27.702+0000 7ff2fffff640 1 -- 172.21.15.88:0/2715150535 --> v1:172.21.15.192:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7ff2ec008330 con 0x7ff2ec00d970 2024-06-03T08:35:27.714 INFO:teuthology.orchestra.run.smithi088.stderr:2024-06-03T08:35:27.714+0000 7ff2fffff640 1 -- 172.21.15.88:0/2715150535 <== mon.2 v1:172.21.15.88:6790/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 1056078341 0 0) 0x7ff2ec009550 con 0x7ff2ec00f7f0 2024-06-03T08:35:27.714 INFO:teuthology.orchestra.run.smithi088.stderr:2024-06-03T08:35:27.714+0000 7ff2fffff640 10 monclient(hunting): _negotiate 2024-06-03T08:35:27.715 INFO:teuthology.orchestra.run.smithi088.stderr:2024-06-03T08:35:27.714+0000 7ff2fffff640 10 monclient(hunting): _init_auth method 2 2024-06-03T08:35:27.715 INFO:teuthology.orchestra.run.smithi088.stderr:2024-06-03T08:35:27.714+0000 7ff2fffff640 10 monclient(hunting): _init_auth creating new auth 2024-06-03T08:35:27.715 INFO:teuthology.orchestra.run.smithi088.stderr:2024-06-03T08:35:27.714+0000 7ff2fffff640 10 monclient(hunting): my global_id is 4106 2024-06-03T08:35:27.715 INFO:teuthology.orchestra.run.smithi088.stderr:2024-06-03T08:35:27.714+0000 7ff2fffff640 1 -- 172.21.15.88:0/2715150535 --> v1:172.21.15.88:6790/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7ff2ec00d260 con 0x7ff2ec00f7f0 2024-06-03T08:35:27.739 INFO:teuthology.orchestra.run.smithi088.stderr:2024-06-03T08:35:27.738+0000 7ff2fffff640 1 -- 172.21.15.88:0/2715150535 <== mon.0 v1:172.21.15.88:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 214+0+0 (unknown 3566469989 0 0) 0x7ff2f4003520 con 0x7ff2ec00fbd0 2024-06-03T08:35:27.739 INFO:teuthology.orchestra.run.smithi088.stderr:2024-06-03T08:35:27.738+0000 7ff2fffff640 1 -- 172.21.15.88:0/2715150535 >> v1:172.21.15.88:6790/0 conn(0x7ff2ec00f7f0 legacy=0x7ff2ec00cab0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-06-03T08:35:27.739 INFO:teuthology.orchestra.run.smithi088.stderr:2024-06-03T08:35:27.738+0000 7ff2fffff640 1 -- 172.21.15.88:0/2715150535 >> v1:172.21.15.192:6789/0 conn(0x7ff2ec00d970 legacy=0x7ff2ec005a70 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-06-03T08:35:27.739 INFO:teuthology.orchestra.run.smithi088.stderr:2024-06-03T08:35:27.738+0000 7ff2fffff640 10 monclient: _finish_hunting 0 2024-06-03T08:35:27.740 INFO:teuthology.orchestra.run.smithi088.stderr:2024-06-03T08:35:27.738+0000 7ff2fffff640 1 monclient: found mon.noname-a 2024-06-03T08:35:27.740 INFO:teuthology.orchestra.run.smithi088.stderr:2024-06-03T08:35:27.738+0000 7ff2fffff640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-06-03T08:35:27.740 INFO:teuthology.orchestra.run.smithi088.stderr:2024-06-03T08:35:27.738+0000 7ff2fffff640 10 monclient: _send_mon_message to mon.noname-a at v1:172.21.15.88:6789/0 2024-06-03T08:35:27.740 INFO:teuthology.orchestra.run.smithi088.stderr:2024-06-03T08:35:27.738+0000 7ff2fffff640 1 -- 172.21.15.88:0/2715150535 --> v1:172.21.15.88:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7ff2ec0096e0 con 0x7ff2ec00fbd0 2024-06-03T08:35:27.740 INFO:teuthology.orchestra.run.smithi088.stderr:2024-06-03T08:35:27.738+0000 7ff2fffff640 10 monclient: _finish_auth 0 2024-06-03T08:35:27.740 INFO:teuthology.orchestra.run.smithi088.stderr:2024-06-03T08:35:27.738+0000 7ff2fffff640 10 monclient: _check_auth_tickets 2024-06-03T08:35:27.740 INFO:teuthology.orchestra.run.smithi088.stderr:2024-06-03T08:35:27.738+0000 7ff2fffff640 20 monclient: _check_auth_rotating not needed by client.admin 2024-06-03T08:35:27.740 INFO:teuthology.orchestra.run.smithi088.stderr:2024-06-03T08:35:27.738+0000 7ff2fffff640 1 -- 172.21.15.88:0/2715150535 <== mon.0 v1:172.21.15.88:6789/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 2234374065 0 0) 0x7ff2f8003e20 con 0x7ff2ec00fbd0 2024-06-03T08:35:27.740 INFO:teuthology.orchestra.run.smithi088.stderr:2024-06-03T08:35:27.738+0000 7ff30f8d5640 5 monclient: authenticate success, global_id 4101 2024-06-03T08:35:27.740 INFO:teuthology.orchestra.run.smithi088.stderr:2024-06-03T08:35:27.738+0000 7ff30f8d5640 20 monclient: get_monmap_and_config waiting for monmap|config 2024-06-03T08:35:27.740 INFO:teuthology.orchestra.run.smithi088.stderr:2024-06-03T08:35:27.738+0000 7ff2fffff640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-06-03T08:35:27.740 INFO:teuthology.orchestra.run.smithi088.stderr:2024-06-03T08:35:27.738+0000 7ff2fffff640 10 monclient: got monmap 1 from mon.noname-a (according to old e1) 2024-06-03T08:35:27.740 INFO:teuthology.orchestra.run.smithi088.stderr:2024-06-03T08:35:27.738+0000 7ff2fffff640 10 monclient: dump: 2024-06-03T08:35:27.741 INFO:teuthology.orchestra.run.smithi088.stderr:epoch 1 2024-06-03T08:35:27.741 INFO:teuthology.orchestra.run.smithi088.stderr:fsid ec3c3fdd-3ac6-4d7d-9412-683130ed8b04 2024-06-03T08:35:27.741 INFO:teuthology.orchestra.run.smithi088.stderr:last_changed 2024-06-03T08:34:31.885125+0000 2024-06-03T08:35:27.741 INFO:teuthology.orchestra.run.smithi088.stderr:created 2024-06-03T08:34:31.885125+0000 2024-06-03T08:35:27.741 INFO:teuthology.orchestra.run.smithi088.stderr:min_mon_release 18 (reef) 2024-06-03T08:35:27.741 INFO:teuthology.orchestra.run.smithi088.stderr:election_strategy: 3 2024-06-03T08:35:27.741 INFO:teuthology.orchestra.run.smithi088.stderr:0: v1:172.21.15.88:6789/0 mon.a 2024-06-03T08:35:27.741 INFO:teuthology.orchestra.run.smithi088.stderr:1: v1:172.21.15.192:6789/0 mon.b 2024-06-03T08:35:27.741 INFO:teuthology.orchestra.run.smithi088.stderr:2: v1:172.21.15.88:6790/0 mon.c 2024-06-03T08:35:27.741 INFO:teuthology.orchestra.run.smithi088.stderr: 2024-06-03T08:35:27.741 INFO:teuthology.orchestra.run.smithi088.stderr:2024-06-03T08:35:27.738+0000 7ff30f8d5640 20 monclient: get_monmap_and_config waiting for monmap|config 2024-06-03T08:35:27.772 INFO:teuthology.orchestra.run.smithi088.stderr:2024-06-03T08:35:27.771+0000 7ff2fffff640 1 -- 172.21.15.88:0/2715150535 <== mon.0 v1:172.21.15.88:6789/0 4 ==== config(0 keys) v1 ==== 4+0+0 (unknown 0 0 0) 0x7ff2f8004040 con 0x7ff2ec00fbd0 2024-06-03T08:35:27.772 INFO:teuthology.orchestra.run.smithi088.stderr:2024-06-03T08:35:27.771+0000 7ff2fffff640 10 monclient: handle_config config(0 keys) v1 2024-06-03T08:35:27.772 INFO:teuthology.orchestra.run.smithi088.stderr:2024-06-03T08:35:27.771+0000 7ff2fffff640 1 -- 172.21.15.88:0/2715150535 <== mon.0 v1:172.21.15.88:6789/0 5 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 2234374065 0 0) 0x7ff2f8004450 con 0x7ff2ec00fbd0 2024-06-03T08:35:27.772 INFO:teuthology.orchestra.run.smithi088.stderr:2024-06-03T08:35:27.771+0000 7ff30f8d5640 10 monclient: get_monmap_and_config success 2024-06-03T08:35:27.772 INFO:teuthology.orchestra.run.smithi088.stderr:2024-06-03T08:35:27.771+0000 7ff30f8d5640 4 set_mon_vals no callback set 2024-06-03T08:35:27.772 INFO:teuthology.orchestra.run.smithi088.stderr:2024-06-03T08:35:27.771+0000 7ff2fffff640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-06-03T08:35:27.773 INFO:teuthology.orchestra.run.smithi088.stderr:2024-06-03T08:35:27.771+0000 7ff2fffff640 10 monclient: got monmap 1 from mon.a (according to old e1) 2024-06-03T08:35:27.773 INFO:teuthology.orchestra.run.smithi088.stderr:2024-06-03T08:35:27.771+0000 7ff2fffff640 10 monclient: dump: 2024-06-03T08:35:27.773 INFO:teuthology.orchestra.run.smithi088.stderr:epoch 1 2024-06-03T08:35:27.773 INFO:teuthology.orchestra.run.smithi088.stderr:fsid ec3c3fdd-3ac6-4d7d-9412-683130ed8b04 2024-06-03T08:35:27.773 INFO:teuthology.orchestra.run.smithi088.stderr:last_changed 2024-06-03T08:34:31.885125+0000 2024-06-03T08:35:27.773 INFO:teuthology.orchestra.run.smithi088.stderr:created 2024-06-03T08:34:31.885125+0000 2024-06-03T08:35:27.773 INFO:teuthology.orchestra.run.smithi088.stderr:min_mon_release 18 (reef) 2024-06-03T08:35:27.773 INFO:teuthology.orchestra.run.smithi088.stderr:election_strategy: 3 2024-06-03T08:35:27.773 INFO:teuthology.orchestra.run.smithi088.stderr:0: v1:172.21.15.88:6789/0 mon.a 2024-06-03T08:35:27.773 INFO:teuthology.orchestra.run.smithi088.stderr:1: v1:172.21.15.192:6789/0 mon.b 2024-06-03T08:35:27.773 INFO:teuthology.orchestra.run.smithi088.stderr:2: v1:172.21.15.88:6790/0 mon.c 2024-06-03T08:35:27.773 INFO:teuthology.orchestra.run.smithi088.stderr: 2024-06-03T08:35:27.775 INFO:teuthology.orchestra.run.smithi088.stderr:2024-06-03T08:35:27.772+0000 7ff30f8d5640 10 monclient: shutdown 2024-06-03T08:35:27.776 INFO:teuthology.orchestra.run.smithi088.stderr:2024-06-03T08:35:27.772+0000 7ff30f8d5640 20 monclient: shutdown discarding 0 pending message(s) 2024-06-03T08:35:27.776 INFO:teuthology.orchestra.run.smithi088.stderr:2024-06-03T08:35:27.772+0000 7ff30f8d5640 1 -- 172.21.15.88:0/2715150535 >> v1:172.21.15.88:6789/0 conn(0x7ff2ec00fbd0 legacy=0x7ff2ec010050 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-06-03T08:35:27.776 INFO:teuthology.orchestra.run.smithi088.stderr:2024-06-03T08:35:27.772+0000 7ff30f8d5640 1 -- 172.21.15.88:0/2715150535 shutdown_connections 2024-06-03T08:35:27.776 INFO:teuthology.orchestra.run.smithi088.stderr:2024-06-03T08:35:27.772+0000 7ff30f8d5640 1 -- 172.21.15.88:0/2715150535 >> 172.21.15.88:0/2715150535 conn(0x7ff3080b70d0 msgr2=0x7ff3080b74e0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-06-03T08:35:27.776 INFO:teuthology.orchestra.run.smithi088.stderr:2024-06-03T08:35:27.772+0000 7ff30f8d5640 1 -- 172.21.15.88:0/2715150535 shutdown_connections 2024-06-03T08:35:27.776 INFO:teuthology.orchestra.run.smithi088.stderr:2024-06-03T08:35:27.772+0000 7ff30f8d5640 1 -- 172.21.15.88:0/2715150535 wait complete. 2024-06-03T08:35:27.776 INFO:teuthology.orchestra.run.smithi088.stderr:2024-06-03T08:35:27.773+0000 7ff30f8d5640 10 monclient: build_initial_monmap 2024-06-03T08:35:27.776 INFO:teuthology.orchestra.run.smithi088.stderr:2024-06-03T08:35:27.773+0000 7ff30f8d5640 10 monclient: monmap: 2024-06-03T08:35:27.776 INFO:teuthology.orchestra.run.smithi088.stderr:epoch 0 2024-06-03T08:35:27.776 INFO:teuthology.orchestra.run.smithi088.stderr:fsid 00000000-0000-0000-0000-000000000000 2024-06-03T08:35:27.776 INFO:teuthology.orchestra.run.smithi088.stderr:last_changed 0.000000 2024-06-03T08:35:27.776 INFO:teuthology.orchestra.run.smithi088.stderr:created 0.000000 2024-06-03T08:35:27.777 INFO:teuthology.orchestra.run.smithi088.stderr:min_mon_release 0 (unknown) 2024-06-03T08:35:27.777 INFO:teuthology.orchestra.run.smithi088.stderr:election_strategy: 1 2024-06-03T08:35:27.777 INFO:teuthology.orchestra.run.smithi088.stderr:0: v1:172.21.15.88:6789/0 mon.noname-a 2024-06-03T08:35:27.777 INFO:teuthology.orchestra.run.smithi088.stderr:1: v1:172.21.15.192:6789/0 mon.noname-b 2024-06-03T08:35:27.777 INFO:teuthology.orchestra.run.smithi088.stderr:2: v1:172.21.15.88:6790/0 mon.noname-c 2024-06-03T08:35:27.777 INFO:teuthology.orchestra.run.smithi088.stderr: 2024-06-03T08:35:27.777 INFO:teuthology.orchestra.run.smithi088.stderr:2024-06-03T08:35:27.773+0000 7ff30f8d5640 1 librados: starting msgr at 2024-06-03T08:35:27.777 INFO:teuthology.orchestra.run.smithi088.stderr:2024-06-03T08:35:27.773+0000 7ff30f8d5640 1 librados: starting objecter 2024-06-03T08:35:27.777 INFO:teuthology.orchestra.run.smithi088.stderr:2024-06-03T08:35:27.773+0000 7ff30f8d5640 1 Processor -- start 2024-06-03T08:35:27.777 INFO:teuthology.orchestra.run.smithi088.stderr:2024-06-03T08:35:27.773+0000 7ff30f8d5640 1 -- start start 2024-06-03T08:35:27.777 INFO:teuthology.orchestra.run.smithi088.stderr:2024-06-03T08:35:27.773+0000 7ff30f8d5640 1 librados: setting wanted keys 2024-06-03T08:35:27.777 INFO:teuthology.orchestra.run.smithi088.stderr:2024-06-03T08:35:27.773+0000 7ff30f8d5640 1 librados: calling monclient init 2024-06-03T08:35:27.777 INFO:teuthology.orchestra.run.smithi088.stderr:2024-06-03T08:35:27.773+0000 7ff30f8d5640 10 monclient: init 2024-06-03T08:35:27.777 INFO:teuthology.orchestra.run.smithi088.stderr:2024-06-03T08:35:27.774+0000 7ff30f8d5640 10 monclient: _reopen_session rank -1 2024-06-03T08:35:27.778 INFO:teuthology.orchestra.run.smithi088.stderr:2024-06-03T08:35:27.774+0000 7ff30f8d5640 10 monclient: _add_conns ranks=[1,0,2] 2024-06-03T08:35:27.778 INFO:teuthology.orchestra.run.smithi088.stderr:2024-06-03T08:35:27.774+0000 7ff30f8d5640 10 monclient(hunting): picked mon.noname-b con 0x7ff2ec00d970 addr v1:172.21.15.192:6789/0 2024-06-03T08:35:27.778 INFO:teuthology.orchestra.run.smithi088.stderr:2024-06-03T08:35:27.774+0000 7ff30f8d5640 10 monclient(hunting): picked mon.noname-a con 0x7ff2ec00f7f0 addr v1:172.21.15.88:6789/0 2024-06-03T08:35:27.778 INFO:teuthology.orchestra.run.smithi088.stderr:2024-06-03T08:35:27.774+0000 7ff30f8d5640 10 monclient(hunting): picked mon.noname-c con 0x7ff2ec00fbd0 addr v1:172.21.15.88:6790/0 2024-06-03T08:35:27.778 INFO:teuthology.orchestra.run.smithi088.stderr:2024-06-03T08:35:27.774+0000 7ff30f8d5640 1 -- --> v1:172.21.15.88:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7ff3081420e0 con 0x7ff2ec00f7f0 2024-06-03T08:35:27.778 INFO:teuthology.orchestra.run.smithi088.stderr:2024-06-03T08:35:27.774+0000 7ff30f8d5640 1 -- --> v1:172.21.15.192:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7ff3080b6530 con 0x7ff2ec00d970 2024-06-03T08:35:27.778 INFO:teuthology.orchestra.run.smithi088.stderr:2024-06-03T08:35:27.774+0000 7ff30f8d5640 1 -- --> v1:172.21.15.88:6790/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7ff308141cc0 con 0x7ff2ec00fbd0 2024-06-03T08:35:27.778 INFO:teuthology.orchestra.run.smithi088.stderr:2024-06-03T08:35:27.774+0000 7ff30f8d5640 10 monclient(hunting): _renew_subs 2024-06-03T08:35:27.778 INFO:teuthology.orchestra.run.smithi088.stderr:2024-06-03T08:35:27.774+0000 7ff30f8d5640 10 monclient(hunting): authenticate will time out at 1183.584495s 2024-06-03T08:35:27.778 INFO:teuthology.orchestra.run.smithi088.stderr:2024-06-03T08:35:27.775+0000 7ff30de72640 1 --1- >> v1:172.21.15.88:6790/0 conn(0x7ff2ec00fbd0 0x7ff3081d6e30 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=1).handle_server_banner_and_identify peer v1:172.21.15.88:6790/0 says I am v1:172.21.15.88:42600/0 (socket says 172.21.15.88:42600) 2024-06-03T08:35:27.778 INFO:teuthology.orchestra.run.smithi088.stderr:2024-06-03T08:35:27.775+0000 7ff30de72640 1 -- 172.21.15.88:0/1111225272 learned_addr learned my addr 172.21.15.88:0/1111225272 (peer_addr_for_me v1:172.21.15.88:0/0) 2024-06-03T08:35:27.778 INFO:teuthology.orchestra.run.smithi088.stderr:2024-06-03T08:35:27.775+0000 7ff2fdffb640 10 client.?.objecter ms_handle_connect 0x7ff2ec00fbd0 2024-06-03T08:35:27.778 INFO:teuthology.orchestra.run.smithi088.stderr:2024-06-03T08:35:27.775+0000 7ff2fdffb640 10 client.?.objecter resend_mon_ops 2024-06-03T08:35:27.779 INFO:teuthology.orchestra.run.smithi088.stderr:2024-06-03T08:35:27.775+0000 7ff2fdffb640 10 client.?.objecter ms_handle_connect 0x7ff2ec00d970 2024-06-03T08:35:27.779 INFO:teuthology.orchestra.run.smithi088.stderr:2024-06-03T08:35:27.775+0000 7ff2fdffb640 10 client.?.objecter resend_mon_ops 2024-06-03T08:35:27.806 INFO:teuthology.orchestra.run.smithi088.stderr:2024-06-03T08:35:27.806+0000 7ff2fdffb640 10 client.?.objecter ms_handle_connect 0x7ff2ec00f7f0 2024-06-03T08:35:27.807 INFO:teuthology.orchestra.run.smithi088.stderr:2024-06-03T08:35:27.806+0000 7ff2fdffb640 10 client.?.objecter resend_mon_ops 2024-06-03T08:35:27.826 INFO:teuthology.orchestra.run.smithi088.stderr:2024-06-03T08:35:27.826+0000 7ff2fdffb640 1 -- 172.21.15.88:0/1111225272 <== mon.0 v1:172.21.15.88:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 712411950 0 0) 0x7ff3081420e0 con 0x7ff2ec00f7f0 2024-06-03T08:35:27.826 INFO:teuthology.orchestra.run.smithi088.stderr:2024-06-03T08:35:27.826+0000 7ff2fdffb640 10 monclient(hunting): _negotiate 2024-06-03T08:35:27.827 INFO:teuthology.orchestra.run.smithi088.stderr:2024-06-03T08:35:27.826+0000 7ff2fdffb640 10 monclient(hunting): _init_auth method 2 2024-06-03T08:35:27.827 INFO:teuthology.orchestra.run.smithi088.stderr:2024-06-03T08:35:27.826+0000 7ff2fdffb640 10 monclient(hunting): _init_auth creating new auth 2024-06-03T08:35:27.827 INFO:teuthology.orchestra.run.smithi088.stderr:2024-06-03T08:35:27.826+0000 7ff2fdffb640 10 monclient(hunting): my global_id is 4107 2024-06-03T08:35:27.827 INFO:teuthology.orchestra.run.smithi088.stderr:2024-06-03T08:35:27.826+0000 7ff2fdffb640 1 -- 172.21.15.88:0/1111225272 --> v1:172.21.15.88:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7ff2ec009870 con 0x7ff2ec00f7f0 2024-06-03T08:35:27.830 INFO:teuthology.orchestra.run.smithi088.stderr:2024-06-03T08:35:27.830+0000 7ff2fdffb640 1 -- 172.21.15.88:0/1111225272 <== mon.1 v1:172.21.15.192:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 943593366 0 0) 0x7ff3080b6530 con 0x7ff2ec00d970 2024-06-03T08:35:27.831 INFO:teuthology.orchestra.run.smithi088.stderr:2024-06-03T08:35:27.830+0000 7ff2fdffb640 10 monclient(hunting): _negotiate 2024-06-03T08:35:27.831 INFO:teuthology.orchestra.run.smithi088.stderr:2024-06-03T08:35:27.830+0000 7ff2fdffb640 10 monclient(hunting): _init_auth method 2 2024-06-03T08:35:27.831 INFO:teuthology.orchestra.run.smithi088.stderr:2024-06-03T08:35:27.830+0000 7ff2fdffb640 10 monclient(hunting): _init_auth creating new auth 2024-06-03T08:35:27.831 INFO:teuthology.orchestra.run.smithi088.stderr:2024-06-03T08:35:27.830+0000 7ff2fdffb640 10 monclient(hunting): my global_id is 4108 2024-06-03T08:35:27.831 INFO:teuthology.orchestra.run.smithi088.stderr:2024-06-03T08:35:27.830+0000 7ff2fdffb640 1 -- 172.21.15.88:0/1111225272 --> v1:172.21.15.192:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7ff3081420e0 con 0x7ff2ec00d970 2024-06-03T08:35:27.831 INFO:teuthology.orchestra.run.smithi088.stderr:2024-06-03T08:35:27.831+0000 7ff2fdffb640 1 -- 172.21.15.88:0/1111225272 <== mon.2 v1:172.21.15.88:6790/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 1096444311 0 0) 0x7ff308141cc0 con 0x7ff2ec00fbd0 2024-06-03T08:35:27.831 INFO:teuthology.orchestra.run.smithi088.stderr:2024-06-03T08:35:27.831+0000 7ff2fdffb640 10 monclient(hunting): _negotiate 2024-06-03T08:35:27.831 INFO:teuthology.orchestra.run.smithi088.stderr:2024-06-03T08:35:27.831+0000 7ff2fdffb640 10 monclient(hunting): _init_auth method 2 2024-06-03T08:35:27.832 INFO:teuthology.orchestra.run.smithi088.stderr:2024-06-03T08:35:27.831+0000 7ff2fdffb640 10 monclient(hunting): _init_auth creating new auth 2024-06-03T08:35:27.832 INFO:teuthology.orchestra.run.smithi088.stderr:2024-06-03T08:35:27.831+0000 7ff2fdffb640 10 monclient(hunting): my global_id is 4109 2024-06-03T08:35:27.832 INFO:teuthology.orchestra.run.smithi088.stderr:2024-06-03T08:35:27.831+0000 7ff2fdffb640 1 -- 172.21.15.88:0/1111225272 --> v1:172.21.15.88:6790/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7ff3080b6530 con 0x7ff2ec00fbd0 2024-06-03T08:35:27.845 INFO:teuthology.orchestra.run.smithi088.stderr:2024-06-03T08:35:27.844+0000 7ff2fdffb640 1 -- 172.21.15.88:0/1111225272 <== mon.0 v1:172.21.15.88:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 764+0+0 (unknown 3884306067 0 0) 0x7ff2ec009870 con 0x7ff2ec00f7f0 2024-06-03T08:35:27.845 INFO:teuthology.orchestra.run.smithi088.stderr:2024-06-03T08:35:27.844+0000 7ff2fdffb640 1 -- 172.21.15.88:0/1111225272 --> v1:172.21.15.88:6789/0 -- auth(proto 2 165 bytes epoch 0) v1 -- 0x7ff308141cc0 con 0x7ff2ec00f7f0 2024-06-03T08:35:27.845 INFO:teuthology.orchestra.run.smithi088.stderr:2024-06-03T08:35:27.844+0000 7ff2fdffb640 1 -- 172.21.15.88:0/1111225272 <== mon.0 v1:172.21.15.88:6789/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 2234374065 0 0) 0x7ff2f8004410 con 0x7ff2ec00f7f0 2024-06-03T08:35:27.845 INFO:teuthology.orchestra.run.smithi088.stderr:2024-06-03T08:35:27.844+0000 7ff2fdffb640 10 monclient(hunting): handle_monmap mon_map magic: 0 v1 2024-06-03T08:35:27.845 INFO:teuthology.orchestra.run.smithi088.stderr:2024-06-03T08:35:27.844+0000 7ff2fdffb640 10 monclient(hunting): got monmap 1 from mon.noname-a (according to old e1) 2024-06-03T08:35:27.845 INFO:teuthology.orchestra.run.smithi088.stderr:2024-06-03T08:35:27.844+0000 7ff2fdffb640 10 monclient(hunting): dump: 2024-06-03T08:35:27.845 INFO:teuthology.orchestra.run.smithi088.stderr:epoch 1 2024-06-03T08:35:27.845 INFO:teuthology.orchestra.run.smithi088.stderr:fsid ec3c3fdd-3ac6-4d7d-9412-683130ed8b04 2024-06-03T08:35:27.845 INFO:teuthology.orchestra.run.smithi088.stderr:last_changed 2024-06-03T08:34:31.885125+0000 2024-06-03T08:35:27.845 INFO:teuthology.orchestra.run.smithi088.stderr:created 2024-06-03T08:34:31.885125+0000 2024-06-03T08:35:27.846 INFO:teuthology.orchestra.run.smithi088.stderr:min_mon_release 18 (reef) 2024-06-03T08:35:27.846 INFO:teuthology.orchestra.run.smithi088.stderr:election_strategy: 3 2024-06-03T08:35:27.846 INFO:teuthology.orchestra.run.smithi088.stderr:0: v1:172.21.15.88:6789/0 mon.a 2024-06-03T08:35:27.846 INFO:teuthology.orchestra.run.smithi088.stderr:1: v1:172.21.15.192:6789/0 mon.b 2024-06-03T08:35:27.846 INFO:teuthology.orchestra.run.smithi088.stderr:2: v1:172.21.15.88:6790/0 mon.c 2024-06-03T08:35:27.846 INFO:teuthology.orchestra.run.smithi088.stderr: 2024-06-03T08:35:27.846 INFO:teuthology.orchestra.run.smithi088.stderr:2024-06-03T08:35:27.844+0000 7ff2fdffb640 10 monclient(hunting): _finish_auth 0 2024-06-03T08:35:27.859 INFO:teuthology.orchestra.run.smithi088.stderr:2024-06-03T08:35:27.858+0000 7ff2fdffb640 1 -- 172.21.15.88:0/1111225272 <== mon.0 v1:172.21.15.88:6789/0 4 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 580+0+0 (unknown 993326948 0 0) 0x7ff308141cc0 con 0x7ff2ec00f7f0 2024-06-03T08:35:27.860 INFO:teuthology.orchestra.run.smithi088.stderr:2024-06-03T08:35:27.859+0000 7ff2fdffb640 1 -- 172.21.15.88:0/1111225272 >> v1:172.21.15.88:6790/0 conn(0x7ff2ec00fbd0 legacy=0x7ff3081d6e30 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-06-03T08:35:27.860 INFO:teuthology.orchestra.run.smithi088.stderr:2024-06-03T08:35:27.859+0000 7ff2fdffb640 1 -- 172.21.15.88:0/1111225272 >> v1:172.21.15.192:6789/0 conn(0x7ff2ec00d970 legacy=0x7ff3081415b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-06-03T08:35:27.860 INFO:teuthology.orchestra.run.smithi088.stderr:2024-06-03T08:35:27.859+0000 7ff2fdffb640 10 monclient: _finish_hunting 0 2024-06-03T08:35:27.860 INFO:teuthology.orchestra.run.smithi088.stderr:2024-06-03T08:35:27.859+0000 7ff2fdffb640 1 monclient: found mon.a 2024-06-03T08:35:27.860 INFO:teuthology.orchestra.run.smithi088.stderr:2024-06-03T08:35:27.859+0000 7ff2fdffb640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-06-03T08:35:27.860 INFO:teuthology.orchestra.run.smithi088.stderr:2024-06-03T08:35:27.859+0000 7ff2fdffb640 10 monclient: _send_mon_message to mon.a at v1:172.21.15.88:6789/0 2024-06-03T08:35:27.860 INFO:teuthology.orchestra.run.smithi088.stderr:2024-06-03T08:35:27.859+0000 7ff2fdffb640 1 -- 172.21.15.88:0/1111225272 --> v1:172.21.15.88:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7ff308140f80 con 0x7ff2ec00f7f0 2024-06-03T08:35:27.861 INFO:teuthology.orchestra.run.smithi088.stderr:2024-06-03T08:35:27.859+0000 7ff2fdffb640 10 monclient: _finish_auth 0 2024-06-03T08:35:27.861 INFO:teuthology.orchestra.run.smithi088.stderr:2024-06-03T08:35:27.859+0000 7ff2fdffb640 10 monclient: _check_auth_tickets 2024-06-03T08:35:27.861 INFO:teuthology.orchestra.run.smithi088.stderr:2024-06-03T08:35:27.859+0000 7ff2fdffb640 20 monclient: _check_auth_rotating not needed by client.admin 2024-06-03T08:35:27.861 INFO:teuthology.orchestra.run.smithi088.stderr:2024-06-03T08:35:27.859+0000 7ff30f8d5640 5 monclient: authenticate success, global_id 4107 2024-06-03T08:35:27.861 INFO:teuthology.orchestra.run.smithi088.stderr:2024-06-03T08:35:27.859+0000 7ff30f8d5640 10 monclient: _renew_subs 2024-06-03T08:35:27.861 INFO:teuthology.orchestra.run.smithi088.stderr:2024-06-03T08:35:27.859+0000 7ff30f8d5640 10 monclient: _send_mon_message to mon.a at v1:172.21.15.88:6789/0 2024-06-03T08:35:27.861 INFO:teuthology.orchestra.run.smithi088.stderr:2024-06-03T08:35:27.859+0000 7ff30f8d5640 1 -- 172.21.15.88:0/1111225272 --> v1:172.21.15.88:6789/0 -- mon_subscribe({mgrmap=0+}) v3 -- 0x7ff3081411f0 con 0x7ff2ec00f7f0 2024-06-03T08:35:27.861 INFO:teuthology.orchestra.run.smithi088.stderr:2024-06-03T08:35:27.859+0000 7ff30f8d5640 10 client.4107.objecter _maybe_request_map subscribing (onetime) to next osd map 2024-06-03T08:35:27.861 INFO:teuthology.orchestra.run.smithi088.stderr:2024-06-03T08:35:27.859+0000 7ff30f8d5640 10 monclient: _renew_subs 2024-06-03T08:35:27.861 INFO:teuthology.orchestra.run.smithi088.stderr:2024-06-03T08:35:27.859+0000 7ff30f8d5640 10 monclient: _send_mon_message to mon.a at v1:172.21.15.88:6789/0 2024-06-03T08:35:27.861 INFO:teuthology.orchestra.run.smithi088.stderr:2024-06-03T08:35:27.859+0000 7ff30f8d5640 1 -- 172.21.15.88:0/1111225272 --> v1:172.21.15.88:6789/0 -- mon_subscribe({osdmap=0}) v3 -- 0x7ff3081e21e0 con 0x7ff2ec00f7f0 2024-06-03T08:35:27.861 INFO:teuthology.orchestra.run.smithi088.stderr:2024-06-03T08:35:27.859+0000 7ff30f8d5640 1 librados: init done 2024-06-03T08:35:27.862 INFO:teuthology.orchestra.run.smithi088.stderr:2024-06-03T08:35:27.860+0000 7ff30f8d5640 10 start_mon_command cmd=[{"prefix": "get_command_descriptions"}] 2024-06-03T08:35:27.862 INFO:teuthology.orchestra.run.smithi088.stderr:2024-06-03T08:35:27.860+0000 7ff30f8d5640 10 monclient: _send_command 1 [{"prefix": "get_command_descriptions"}] 2024-06-03T08:35:27.862 INFO:teuthology.orchestra.run.smithi088.stderr:2024-06-03T08:35:27.860+0000 7ff30f8d5640 10 monclient: _send_mon_message to mon.a at v1:172.21.15.88:6789/0 2024-06-03T08:35:27.863 INFO:teuthology.orchestra.run.smithi088.stderr:2024-06-03T08:35:27.860+0000 7ff30f8d5640 1 -- 172.21.15.88:0/1111225272 --> v1:172.21.15.88:6789/0 -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7ff3080a9e60 con 0x7ff2ec00f7f0 2024-06-03T08:35:28.077 INFO:teuthology.orchestra.run.smithi088.stderr:2024-06-03T08:35:28.077+0000 7ff2fdffb640 1 -- 172.21.15.88:0/1111225272 <== mon.0 v1:172.21.15.88:6789/0 5 ==== config(0 keys) v1 ==== 4+0+0 (unknown 0 0 0) 0x7ff2f8003a40 con 0x7ff2ec00f7f0 2024-06-03T08:35:28.078 INFO:teuthology.orchestra.run.smithi088.stderr:2024-06-03T08:35:28.077+0000 7ff2fdffb640 10 monclient: handle_config config(0 keys) v1 2024-06-03T08:35:28.078 INFO:teuthology.orchestra.run.smithi088.stderr:2024-06-03T08:35:28.077+0000 7ff2fdffb640 1 -- 172.21.15.88:0/1111225272 <== mon.0 v1:172.21.15.88:6789/0 6 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 2234374065 0 0) 0x7ff2f8004b00 con 0x7ff2ec00f7f0 2024-06-03T08:35:28.078 INFO:teuthology.orchestra.run.smithi088.stderr:2024-06-03T08:35:28.077+0000 7ff2fdffb640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-06-03T08:35:28.078 INFO:teuthology.orchestra.run.smithi088.stderr:2024-06-03T08:35:28.077+0000 7ff2fdffb640 10 monclient: got monmap 1 from mon.a (according to old e1) 2024-06-03T08:35:28.078 INFO:teuthology.orchestra.run.smithi088.stderr:2024-06-03T08:35:28.077+0000 7ff2fdffb640 10 monclient: dump: 2024-06-03T08:35:28.078 INFO:teuthology.orchestra.run.smithi088.stderr:epoch 1 2024-06-03T08:35:28.078 INFO:teuthology.orchestra.run.smithi088.stderr:fsid ec3c3fdd-3ac6-4d7d-9412-683130ed8b04 2024-06-03T08:35:28.078 INFO:teuthology.orchestra.run.smithi088.stderr:last_changed 2024-06-03T08:34:31.885125+0000 2024-06-03T08:35:28.078 INFO:teuthology.orchestra.run.smithi088.stderr:created 2024-06-03T08:34:31.885125+0000 2024-06-03T08:35:28.078 INFO:teuthology.orchestra.run.smithi088.stderr:min_mon_release 18 (reef) 2024-06-03T08:35:28.078 INFO:teuthology.orchestra.run.smithi088.stderr:election_strategy: 3 2024-06-03T08:35:28.078 INFO:teuthology.orchestra.run.smithi088.stderr:0: v1:172.21.15.88:6789/0 mon.a 2024-06-03T08:35:28.079 INFO:teuthology.orchestra.run.smithi088.stderr:1: v1:172.21.15.192:6789/0 mon.b 2024-06-03T08:35:28.079 INFO:teuthology.orchestra.run.smithi088.stderr:2: v1:172.21.15.88:6790/0 mon.c 2024-06-03T08:35:28.079 INFO:teuthology.orchestra.run.smithi088.stderr: 2024-06-03T08:35:28.079 INFO:teuthology.orchestra.run.smithi088.stderr:2024-06-03T08:35:28.077+0000 7ff2fdffb640 1 -- 172.21.15.88:0/1111225272 <== mon.0 v1:172.21.15.88:6789/0 7 ==== mgrmap(e 1) v1 ==== 668+0+0 (unknown 3836259515 0 0) 0x7ff2f8004f60 con 0x7ff2ec00f7f0 2024-06-03T08:35:28.079 INFO:teuthology.orchestra.run.smithi088.stderr:2024-06-03T08:35:28.077+0000 7ff2ff7fe640 4 set_mon_vals no callback set 2024-06-03T08:35:28.079 INFO:teuthology.orchestra.run.smithi088.stderr:2024-06-03T08:35:28.077+0000 7ff2fdffb640 1 -- 172.21.15.88:0/1111225272 <== mon.0 v1:172.21.15.88:6789/0 8 ==== osd_map(1..1 src has 1..1) v4 ==== 725+0+0 (unknown 3900859628 0 0) 0x7ff2f8006220 con 0x7ff2ec00f7f0 2024-06-03T08:35:28.079 INFO:teuthology.orchestra.run.smithi088.stderr:2024-06-03T08:35:28.077+0000 7ff2fdffb640 10 client.4107.objecter ms_dispatch 0x7ff308000b90 osd_map(1..1 src has 1..1) v4 2024-06-03T08:35:28.079 INFO:teuthology.orchestra.run.smithi088.stderr:2024-06-03T08:35:28.077+0000 7ff2fdffb640 3 client.4107.objecter handle_osd_map got epochs [1,1] > 0 2024-06-03T08:35:28.079 INFO:teuthology.orchestra.run.smithi088.stderr:2024-06-03T08:35:28.077+0000 7ff2fdffb640 3 client.4107.objecter handle_osd_map decoding full epoch 1 2024-06-03T08:35:28.079 INFO:teuthology.orchestra.run.smithi088.stderr:2024-06-03T08:35:28.077+0000 7ff2fdffb640 20 client.4107.objecter dump_active .. 0 homeless 2024-06-03T08:35:28.079 INFO:teuthology.orchestra.run.smithi088.stderr:2024-06-03T08:35:28.077+0000 7ff2fdffb640 1 -- 172.21.15.88:0/1111225272 <== mon.0 v1:172.21.15.88:6789/0 9 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+73121 (unknown 1092875540 0 1417653156) 0x7ff2f8006710 con 0x7ff2ec00f7f0 2024-06-03T08:35:28.079 INFO:teuthology.orchestra.run.smithi088.stderr:2024-06-03T08:35:28.077+0000 7ff2fdffb640 10 monclient: handle_mon_command_ack 1 [{"prefix": "get_command_descriptions"}] 2024-06-03T08:35:28.079 INFO:teuthology.orchestra.run.smithi088.stderr:2024-06-03T08:35:28.077+0000 7ff2fdffb640 10 monclient: _finish_command 1 = system:0 2024-06-03T08:35:28.155 INFO:teuthology.orchestra.run.smithi088.stderr:2024-06-03T08:35:28.155+0000 7ff30f8d5640 10 start_mon_command cmd=[{"prefix": "osd crush tunables", "profile": "default"}] 2024-06-03T08:35:28.156 INFO:teuthology.orchestra.run.smithi088.stderr:2024-06-03T08:35:28.155+0000 7ff30f8d5640 10 monclient: _send_command 2 [{"prefix": "osd crush tunables", "profile": "default"}] 2024-06-03T08:35:28.156 INFO:teuthology.orchestra.run.smithi088.stderr:2024-06-03T08:35:28.155+0000 7ff30f8d5640 10 monclient: _send_mon_message to mon.a at v1:172.21.15.88:6789/0 2024-06-03T08:35:28.156 INFO:teuthology.orchestra.run.smithi088.stderr:2024-06-03T08:35:28.155+0000 7ff30f8d5640 1 -- 172.21.15.88:0/1111225272 --> v1:172.21.15.88:6789/0 -- mon_command({"prefix": "osd crush tunables", "profile": "default"} v 0) v1 -- 0x7ff3080a9e60 con 0x7ff2ec00f7f0 2024-06-03T08:35:28.207 INFO:tasks.ceph.mgr.x.smithi192.stderr:2024-06-03T08:35:28.207+0000 7f0e6be43040 -1 mgr[py] Module balancer has missing NOTIFY_TYPES member 2024-06-03T08:35:28.322 INFO:tasks.ceph.mgr.y.smithi088.stderr:2024-06-03T08:35:28.322+0000 7ff0aea1d040 -1 mgr[py] Module balancer has missing NOTIFY_TYPES member 2024-06-03T08:35:28.364 INFO:tasks.ceph.mgr.x.smithi192.stderr:2024-06-03T08:35:28.364+0000 7f0e6be43040 -1 mgr[py] Module orchestrator has missing NOTIFY_TYPES member 2024-06-03T08:35:28.418 INFO:tasks.ceph.mgr.x.smithi192.stderr:2024-06-03T08:35:28.418+0000 7f0e6be43040 -1 mgr[py] Module snap_schedule has missing NOTIFY_TYPES member 2024-06-03T08:35:28.429 INFO:teuthology.orchestra.run.smithi088.stderr:2024-06-03T08:35:28.429+0000 7ff2fdffb640 1 -- 172.21.15.88:0/1111225272 <== mon.0 v1:172.21.15.88:6789/0 10 ==== mon_command_ack([{"prefix": "osd crush tunables", "profile": "default"}]=0 adjusted tunables profile to default v2) v1 ==== 124+0+0 (unknown 2277712955 0 0) 0x7ff2f8005440 con 0x7ff2ec00f7f0 2024-06-03T08:35:28.429 INFO:teuthology.orchestra.run.smithi088.stderr:2024-06-03T08:35:28.429+0000 7ff2fdffb640 10 monclient: handle_mon_command_ack 2 [{"prefix": "osd crush tunables", "profile": "default"}] 2024-06-03T08:35:28.429 INFO:teuthology.orchestra.run.smithi088.stderr:2024-06-03T08:35:28.429+0000 7ff2fdffb640 10 monclient: _finish_command 2 = system:0 adjusted tunables profile to default 2024-06-03T08:35:28.430 INFO:teuthology.orchestra.run.smithi088.stderr:adjusted tunables profile to default 2024-06-03T08:35:28.431 INFO:teuthology.orchestra.run.smithi088.stderr:2024-06-03T08:35:28.430+0000 7ff30f8d5640 10 librados: watch_flush enter 2024-06-03T08:35:28.431 INFO:teuthology.orchestra.run.smithi088.stderr:2024-06-03T08:35:28.430+0000 7ff30f8d5640 10 librados: watch_flush exit 2024-06-03T08:35:28.432 INFO:teuthology.orchestra.run.smithi088.stderr:2024-06-03T08:35:28.430+0000 7ff30f8d5640 20 client.4107.objecter shutdown clearing up homeless session... 2024-06-03T08:35:28.432 INFO:teuthology.orchestra.run.smithi088.stderr:2024-06-03T08:35:28.430+0000 7ff30f8d5640 10 client.4107.objecter successfully canceled tick 2024-06-03T08:35:28.432 INFO:teuthology.orchestra.run.smithi088.stderr:2024-06-03T08:35:28.430+0000 7ff30f8d5640 10 monclient: shutdown 2024-06-03T08:35:28.432 INFO:teuthology.orchestra.run.smithi088.stderr:2024-06-03T08:35:28.430+0000 7ff30f8d5640 20 monclient: shutdown discarding 0 pending message(s) 2024-06-03T08:35:28.432 INFO:teuthology.orchestra.run.smithi088.stderr:2024-06-03T08:35:28.430+0000 7ff30f8d5640 1 -- 172.21.15.88:0/1111225272 >> v1:172.21.15.88:6789/0 conn(0x7ff2ec00f7f0 legacy=0x7ff3081d6720 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-06-03T08:35:28.432 INFO:teuthology.orchestra.run.smithi088.stderr:2024-06-03T08:35:28.430+0000 7ff30f8d5640 1 -- 172.21.15.88:0/1111225272 shutdown_connections 2024-06-03T08:35:28.432 INFO:teuthology.orchestra.run.smithi088.stderr:2024-06-03T08:35:28.430+0000 7ff30f8d5640 1 -- 172.21.15.88:0/1111225272 >> 172.21.15.88:0/1111225272 conn(0x7ff3080b70d0 msgr2=0x7ff3080b74b0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-06-03T08:35:28.432 INFO:teuthology.orchestra.run.smithi088.stderr:2024-06-03T08:35:28.430+0000 7ff30f8d5640 1 -- 172.21.15.88:0/1111225272 shutdown_connections 2024-06-03T08:35:28.432 INFO:teuthology.orchestra.run.smithi088.stderr:2024-06-03T08:35:28.430+0000 7ff30f8d5640 1 -- 172.21.15.88:0/1111225272 wait complete. 2024-06-03T08:35:28.432 INFO:teuthology.orchestra.run.smithi088.stderr:2024-06-03T08:35:28.430+0000 7ff30f8d5640 1 librados: shutdown 2024-06-03T08:35:28.442 INFO:tasks.ceph:check_enable_crimson: False 2024-06-03T08:35:28.442 INFO:tasks.ceph:Starting osd daemons in cluster ceph... 2024-06-03T08:35:28.443 DEBUG:teuthology.orchestra.run.smithi088:> set -ex 2024-06-03T08:35:28.443 DEBUG:teuthology.orchestra.run.smithi088:> sudo dd if=/var/lib/ceph/osd/ceph-0/fsid of=/dev/stdout 2024-06-03T08:35:28.470 DEBUG:teuthology.orchestra.run.smithi088:> set -ex 2024-06-03T08:35:28.470 DEBUG:teuthology.orchestra.run.smithi088:> sudo dd if=/var/lib/ceph/osd/ceph-1/fsid of=/dev/stdout 2024-06-03T08:35:28.507 INFO:tasks.ceph.mgr.y.smithi088.stderr:2024-06-03T08:35:28.506+0000 7ff0aea1d040 -1 mgr[py] Module orchestrator has missing NOTIFY_TYPES member 2024-06-03T08:35:28.532 DEBUG:teuthology.orchestra.run.smithi088:> set -ex 2024-06-03T08:35:28.532 DEBUG:teuthology.orchestra.run.smithi088:> sudo dd if=/var/lib/ceph/osd/ceph-2/fsid of=/dev/stdout 2024-06-03T08:35:28.569 INFO:tasks.ceph.mgr.y.smithi088.stderr:2024-06-03T08:35:28.569+0000 7ff0aea1d040 -1 mgr[py] Module snap_schedule has missing NOTIFY_TYPES member 2024-06-03T08:35:28.596 DEBUG:teuthology.orchestra.run.smithi088:> set -ex 2024-06-03T08:35:28.596 DEBUG:teuthology.orchestra.run.smithi088:> sudo dd if=/var/lib/ceph/osd/ceph-3/fsid of=/dev/stdout 2024-06-03T08:35:28.662 DEBUG:teuthology.orchestra.run.smithi192:> set -ex 2024-06-03T08:35:28.662 DEBUG:teuthology.orchestra.run.smithi192:> sudo dd if=/var/lib/ceph/osd/ceph-4/fsid of=/dev/stdout 2024-06-03T08:35:28.687 DEBUG:teuthology.orchestra.run.smithi192:> set -ex 2024-06-03T08:35:28.687 DEBUG:teuthology.orchestra.run.smithi192:> sudo dd if=/var/lib/ceph/osd/ceph-5/fsid of=/dev/stdout 2024-06-03T08:35:28.698 INFO:tasks.ceph.mgr.x.smithi192.stderr:2024-06-03T08:35:28.697+0000 7f0e6be43040 -1 mgr[py] Module progress has missing NOTIFY_TYPES member 2024-06-03T08:35:28.721 DEBUG:teuthology.orchestra.run.smithi192:> set -ex 2024-06-03T08:35:28.721 DEBUG:teuthology.orchestra.run.smithi192:> sudo dd if=/var/lib/ceph/osd/ceph-6/fsid of=/dev/stdout 2024-06-03T08:35:28.747 INFO:tasks.ceph.mgr.x.smithi192.stderr:2024-06-03T08:35:28.746+0000 7f0e6be43040 -1 mgr[py] Module influx has missing NOTIFY_TYPES member 2024-06-03T08:35:28.771 DEBUG:teuthology.orchestra.run.smithi192:> set -ex 2024-06-03T08:35:28.771 DEBUG:teuthology.orchestra.run.smithi192:> sudo dd if=/var/lib/ceph/osd/ceph-7/fsid of=/dev/stdout 2024-06-03T08:35:28.835 DEBUG:teuthology.orchestra.run.smithi192:> sudo ceph --cluster ceph osd new b25ab3bb-3587-45d0-991a-6d41bb529b37 0 2024-06-03T08:35:28.842 INFO:tasks.ceph.mgr.y.smithi088.stderr:2024-06-03T08:35:28.842+0000 7ff0aea1d040 -1 mgr[py] Module progress has missing NOTIFY_TYPES member 2024-06-03T08:35:28.888 INFO:tasks.ceph.mgr.y.smithi088.stderr:2024-06-03T08:35:28.888+0000 7ff0aea1d040 -1 mgr[py] Module influx has missing NOTIFY_TYPES member 2024-06-03T08:35:28.974 INFO:teuthology.orchestra.run.smithi192.stderr:2024-06-03T08:35:28.973+0000 7fe2d54e2640 10 monclient: get_monmap_and_config 2024-06-03T08:35:28.974 INFO:teuthology.orchestra.run.smithi192.stderr:2024-06-03T08:35:28.973+0000 7fe2d54e2640 10 monclient: build_initial_monmap 2024-06-03T08:35:28.974 INFO:teuthology.orchestra.run.smithi192.stderr:2024-06-03T08:35:28.973+0000 7fe2d54e2640 10 monclient: monmap: 2024-06-03T08:35:28.974 INFO:teuthology.orchestra.run.smithi192.stderr:epoch 0 2024-06-03T08:35:28.974 INFO:teuthology.orchestra.run.smithi192.stderr:fsid ec3c3fdd-3ac6-4d7d-9412-683130ed8b04 2024-06-03T08:35:28.975 INFO:teuthology.orchestra.run.smithi192.stderr:last_changed 2024-06-03T08:35:28.974567+0000 2024-06-03T08:35:28.975 INFO:teuthology.orchestra.run.smithi192.stderr:created 2024-06-03T08:35:28.974567+0000 2024-06-03T08:35:28.975 INFO:teuthology.orchestra.run.smithi192.stderr:min_mon_release 0 (unknown) 2024-06-03T08:35:28.975 INFO:teuthology.orchestra.run.smithi192.stderr:election_strategy: 3 2024-06-03T08:35:28.975 INFO:teuthology.orchestra.run.smithi192.stderr:0: v1:172.21.15.88:6789/0 mon.noname-a 2024-06-03T08:35:28.975 INFO:teuthology.orchestra.run.smithi192.stderr:1: v1:172.21.15.192:6789/0 mon.noname-c 2024-06-03T08:35:28.975 INFO:teuthology.orchestra.run.smithi192.stderr:2: v1:172.21.15.88:6790/0 mon.noname-b 2024-06-03T08:35:28.975 INFO:teuthology.orchestra.run.smithi192.stderr: 2024-06-03T08:35:28.975 INFO:teuthology.orchestra.run.smithi192.stderr:2024-06-03T08:35:28.974+0000 7fe2d54e2640 1 Processor -- start 2024-06-03T08:35:28.975 INFO:teuthology.orchestra.run.smithi192.stderr:2024-06-03T08:35:28.974+0000 7fe2d54e2640 1 -- start start 2024-06-03T08:35:28.975 INFO:teuthology.orchestra.run.smithi192.stderr:2024-06-03T08:35:28.974+0000 7fe2d54e2640 10 monclient: init 2024-06-03T08:35:28.976 INFO:teuthology.orchestra.run.smithi192.stderr:2024-06-03T08:35:28.974+0000 7fe2d54e2640 10 monclient: _reopen_session rank -1 2024-06-03T08:35:28.976 INFO:teuthology.orchestra.run.smithi192.stderr:2024-06-03T08:35:28.974+0000 7fe2d54e2640 10 monclient: _add_conns ranks=[2,0,1] 2024-06-03T08:35:28.977 INFO:teuthology.orchestra.run.smithi192.stderr:2024-06-03T08:35:28.974+0000 7fe2d54e2640 10 monclient(hunting): picked mon.noname-b con 0x7fe2d00b4120 addr v1:172.21.15.88:6790/0 2024-06-03T08:35:28.977 INFO:teuthology.orchestra.run.smithi192.stderr:2024-06-03T08:35:28.974+0000 7fe2d54e2640 10 monclient(hunting): picked mon.noname-a con 0x7fe2d00ae7f0 addr v1:172.21.15.88:6789/0 2024-06-03T08:35:28.977 INFO:teuthology.orchestra.run.smithi192.stderr:2024-06-03T08:35:28.974+0000 7fe2d54e2640 10 monclient(hunting): picked mon.noname-c con 0x7fe2d00af490 addr v1:172.21.15.192:6789/0 2024-06-03T08:35:28.977 INFO:teuthology.orchestra.run.smithi192.stderr:2024-06-03T08:35:28.974+0000 7fe2d54e2640 1 -- --> v1:172.21.15.88:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fe2d00714c0 con 0x7fe2d00ae7f0 2024-06-03T08:35:28.977 INFO:teuthology.orchestra.run.smithi192.stderr:2024-06-03T08:35:28.974+0000 7fe2d54e2640 1 -- --> v1:172.21.15.192:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fe2d006e020 con 0x7fe2d00af490 2024-06-03T08:35:28.977 INFO:teuthology.orchestra.run.smithi192.stderr:2024-06-03T08:35:28.974+0000 7fe2d54e2640 1 -- --> v1:172.21.15.88:6790/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fe2d006e270 con 0x7fe2d00b4120 2024-06-03T08:35:28.977 INFO:teuthology.orchestra.run.smithi192.stderr:2024-06-03T08:35:28.974+0000 7fe2d54e2640 10 monclient(hunting): _renew_subs 2024-06-03T08:35:28.977 INFO:teuthology.orchestra.run.smithi192.stderr:2024-06-03T08:35:28.974+0000 7fe2d54e2640 10 monclient(hunting): authenticate will time out at 1194.786029s 2024-06-03T08:35:28.977 INFO:teuthology.orchestra.run.smithi192.stderr:2024-06-03T08:35:28.975+0000 7fe2cf7fe640 1 --1- >> v1:172.21.15.192:6789/0 conn(0x7fe2d00af490 0x7fe2d014ce70 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.192:6789/0 says I am v1:172.21.15.192:43806/0 (socket says 172.21.15.192:43806) 2024-06-03T08:35:28.977 INFO:teuthology.orchestra.run.smithi192.stderr:2024-06-03T08:35:28.975+0000 7fe2cf7fe640 1 -- 172.21.15.192:0/3809358033 learned_addr learned my addr 172.21.15.192:0/3809358033 (peer_addr_for_me v1:172.21.15.192:0/0) 2024-06-03T08:35:28.978 INFO:teuthology.orchestra.run.smithi192.stderr:2024-06-03T08:35:28.977+0000 7fe2cdffb640 1 -- 172.21.15.192:0/3809358033 <== mon.1 v1:172.21.15.192:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 813632647 0 0) 0x7fe2d006e020 con 0x7fe2d00af490 2024-06-03T08:35:28.978 INFO:teuthology.orchestra.run.smithi192.stderr:2024-06-03T08:35:28.977+0000 7fe2cdffb640 10 monclient(hunting): _negotiate 2024-06-03T08:35:28.978 INFO:teuthology.orchestra.run.smithi192.stderr:2024-06-03T08:35:28.977+0000 7fe2cdffb640 10 monclient(hunting): _init_auth method 2 2024-06-03T08:35:28.978 INFO:teuthology.orchestra.run.smithi192.stderr:2024-06-03T08:35:28.977+0000 7fe2cdffb640 10 monclient(hunting): _init_auth creating new auth 2024-06-03T08:35:28.978 INFO:teuthology.orchestra.run.smithi192.stderr:2024-06-03T08:35:28.977+0000 7fe2cdffb640 10 monclient(hunting): my global_id is 4117 2024-06-03T08:35:28.978 INFO:teuthology.orchestra.run.smithi192.stderr:2024-06-03T08:35:28.977+0000 7fe2cdffb640 1 -- 172.21.15.192:0/3809358033 --> v1:172.21.15.192:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7fe2b8003520 con 0x7fe2d00af490 2024-06-03T08:35:28.978 INFO:teuthology.orchestra.run.smithi192.stderr:2024-06-03T08:35:28.978+0000 7fe2cdffb640 1 -- 172.21.15.192:0/3809358033 <== mon.2 v1:172.21.15.88:6790/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 3505782853 0 0) 0x7fe2d006e270 con 0x7fe2d00b4120 2024-06-03T08:35:28.979 INFO:teuthology.orchestra.run.smithi192.stderr:2024-06-03T08:35:28.978+0000 7fe2cdffb640 10 monclient(hunting): _negotiate 2024-06-03T08:35:28.979 INFO:teuthology.orchestra.run.smithi192.stderr:2024-06-03T08:35:28.978+0000 7fe2cdffb640 10 monclient(hunting): _init_auth method 2 2024-06-03T08:35:28.979 INFO:teuthology.orchestra.run.smithi192.stderr:2024-06-03T08:35:28.978+0000 7fe2cdffb640 10 monclient(hunting): _init_auth creating new auth 2024-06-03T08:35:28.979 INFO:teuthology.orchestra.run.smithi192.stderr:2024-06-03T08:35:28.978+0000 7fe2cdffb640 10 monclient(hunting): my global_id is 4118 2024-06-03T08:35:28.979 INFO:teuthology.orchestra.run.smithi192.stderr:2024-06-03T08:35:28.978+0000 7fe2cdffb640 1 -- 172.21.15.192:0/3809358033 --> v1:172.21.15.88:6790/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7fe2d006e020 con 0x7fe2d00b4120 2024-06-03T08:35:28.979 INFO:teuthology.orchestra.run.smithi192.stderr:2024-06-03T08:35:28.978+0000 7fe2cdffb640 1 -- 172.21.15.192:0/3809358033 <== mon.0 v1:172.21.15.88:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 729239683 0 0) 0x7fe2c4001110 con 0x7fe2d00ae7f0 2024-06-03T08:35:28.979 INFO:teuthology.orchestra.run.smithi192.stderr:2024-06-03T08:35:28.978+0000 7fe2cdffb640 10 monclient(hunting): _negotiate 2024-06-03T08:35:28.979 INFO:teuthology.orchestra.run.smithi192.stderr:2024-06-03T08:35:28.978+0000 7fe2cdffb640 10 monclient(hunting): _init_auth method 2 2024-06-03T08:35:28.980 INFO:teuthology.orchestra.run.smithi192.stderr:2024-06-03T08:35:28.978+0000 7fe2cdffb640 10 monclient(hunting): _init_auth creating new auth 2024-06-03T08:35:28.980 INFO:teuthology.orchestra.run.smithi192.stderr:2024-06-03T08:35:28.978+0000 7fe2cdffb640 10 monclient(hunting): my global_id is 4116 2024-06-03T08:35:28.980 INFO:teuthology.orchestra.run.smithi192.stderr:2024-06-03T08:35:28.978+0000 7fe2cdffb640 1 -- 172.21.15.192:0/3809358033 --> v1:172.21.15.88:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7fe2d006e270 con 0x7fe2d00ae7f0 2024-06-03T08:35:28.982 INFO:teuthology.orchestra.run.smithi192.stderr:2024-06-03T08:35:28.981+0000 7fe2cdffb640 1 -- 172.21.15.192:0/3809358033 <== mon.1 v1:172.21.15.192:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 214+0+0 (unknown 3260872716 0 0) 0x7fe2b8003520 con 0x7fe2d00af490 2024-06-03T08:35:28.982 INFO:teuthology.orchestra.run.smithi192.stderr:2024-06-03T08:35:28.981+0000 7fe2cdffb640 1 -- 172.21.15.192:0/3809358033 >> v1:172.21.15.88:6790/0 conn(0x7fe2d00b4120 legacy=0x7fe2d013e7e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-06-03T08:35:28.982 INFO:teuthology.orchestra.run.smithi192.stderr:2024-06-03T08:35:28.981+0000 7fe2cdffb640 1 -- 172.21.15.192:0/3809358033 >> v1:172.21.15.88:6789/0 conn(0x7fe2d00ae7f0 legacy=0x7fe2d0144ad0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-06-03T08:35:28.982 INFO:teuthology.orchestra.run.smithi192.stderr:2024-06-03T08:35:28.981+0000 7fe2cdffb640 10 monclient: _finish_hunting 0 2024-06-03T08:35:28.982 INFO:teuthology.orchestra.run.smithi192.stderr:2024-06-03T08:35:28.981+0000 7fe2cdffb640 1 monclient: found mon.noname-c 2024-06-03T08:35:28.982 INFO:teuthology.orchestra.run.smithi192.stderr:2024-06-03T08:35:28.981+0000 7fe2cdffb640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-06-03T08:35:28.982 INFO:teuthology.orchestra.run.smithi192.stderr:2024-06-03T08:35:28.981+0000 7fe2cdffb640 10 monclient: _send_mon_message to mon.noname-c at v1:172.21.15.192:6789/0 2024-06-03T08:35:28.983 INFO:teuthology.orchestra.run.smithi192.stderr:2024-06-03T08:35:28.981+0000 7fe2cdffb640 1 -- 172.21.15.192:0/3809358033 --> v1:172.21.15.192:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fe2d00ae2d0 con 0x7fe2d00af490 2024-06-03T08:35:28.983 INFO:teuthology.orchestra.run.smithi192.stderr:2024-06-03T08:35:28.981+0000 7fe2cdffb640 10 monclient: _finish_auth 0 2024-06-03T08:35:28.983 INFO:teuthology.orchestra.run.smithi192.stderr:2024-06-03T08:35:28.981+0000 7fe2cdffb640 10 monclient: _check_auth_tickets 2024-06-03T08:35:28.983 INFO:teuthology.orchestra.run.smithi192.stderr:2024-06-03T08:35:28.981+0000 7fe2cdffb640 20 monclient: _check_auth_rotating not needed by client.admin 2024-06-03T08:35:28.983 INFO:teuthology.orchestra.run.smithi192.stderr:2024-06-03T08:35:28.981+0000 7fe2cdffb640 1 -- 172.21.15.192:0/3809358033 <== mon.1 v1:172.21.15.192:6789/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 2234374065 0 0) 0x7fe2c8001990 con 0x7fe2d00af490 2024-06-03T08:35:28.983 INFO:teuthology.orchestra.run.smithi192.stderr:2024-06-03T08:35:28.981+0000 7fe2cdffb640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-06-03T08:35:28.983 INFO:teuthology.orchestra.run.smithi192.stderr:2024-06-03T08:35:28.981+0000 7fe2cdffb640 10 monclient: got monmap 1 from mon.noname-c (according to old e1) 2024-06-03T08:35:28.983 INFO:teuthology.orchestra.run.smithi192.stderr:2024-06-03T08:35:28.981+0000 7fe2cdffb640 10 monclient: dump: 2024-06-03T08:35:28.983 INFO:teuthology.orchestra.run.smithi192.stderr:epoch 1 2024-06-03T08:35:28.983 INFO:teuthology.orchestra.run.smithi192.stderr:fsid ec3c3fdd-3ac6-4d7d-9412-683130ed8b04 2024-06-03T08:35:28.983 INFO:teuthology.orchestra.run.smithi192.stderr:last_changed 2024-06-03T08:34:31.885125+0000 2024-06-03T08:35:28.983 INFO:teuthology.orchestra.run.smithi192.stderr:created 2024-06-03T08:34:31.885125+0000 2024-06-03T08:35:28.983 INFO:teuthology.orchestra.run.smithi192.stderr:min_mon_release 18 (reef) 2024-06-03T08:35:28.983 INFO:teuthology.orchestra.run.smithi192.stderr:election_strategy: 3 2024-06-03T08:35:28.983 INFO:teuthology.orchestra.run.smithi192.stderr:0: v1:172.21.15.88:6789/0 mon.a 2024-06-03T08:35:28.984 INFO:teuthology.orchestra.run.smithi192.stderr:1: v1:172.21.15.192:6789/0 mon.b 2024-06-03T08:35:28.984 INFO:teuthology.orchestra.run.smithi192.stderr:2: v1:172.21.15.88:6790/0 mon.c 2024-06-03T08:35:28.984 INFO:teuthology.orchestra.run.smithi192.stderr: 2024-06-03T08:35:28.984 INFO:teuthology.orchestra.run.smithi192.stderr:2024-06-03T08:35:28.981+0000 7fe2d54e2640 5 monclient: authenticate success, global_id 4117 2024-06-03T08:35:28.984 INFO:teuthology.orchestra.run.smithi192.stderr:2024-06-03T08:35:28.981+0000 7fe2d54e2640 20 monclient: get_monmap_and_config waiting for monmap|config 2024-06-03T08:35:29.007 INFO:teuthology.orchestra.run.smithi192.stderr:2024-06-03T08:35:29.007+0000 7fe2cdffb640 1 -- 172.21.15.192:0/3809358033 <== mon.1 v1:172.21.15.192:6789/0 4 ==== config(0 keys) v1 ==== 4+0+0 (unknown 0 0 0) 0x7fe2c8001bb0 con 0x7fe2d00af490 2024-06-03T08:35:29.007 INFO:teuthology.orchestra.run.smithi192.stderr:2024-06-03T08:35:29.007+0000 7fe2cdffb640 10 monclient: handle_config config(0 keys) v1 2024-06-03T08:35:29.007 INFO:teuthology.orchestra.run.smithi192.stderr:2024-06-03T08:35:29.007+0000 7fe2cdffb640 1 -- 172.21.15.192:0/3809358033 <== mon.1 v1:172.21.15.192:6789/0 5 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 2234374065 0 0) 0x7fe2c8001f40 con 0x7fe2d00af490 2024-06-03T08:35:29.007 INFO:teuthology.orchestra.run.smithi192.stderr:2024-06-03T08:35:29.007+0000 7fe2cdffb640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-06-03T08:35:29.007 INFO:teuthology.orchestra.run.smithi192.stderr:2024-06-03T08:35:29.007+0000 7fe2cdffb640 10 monclient: got monmap 1 from mon.b (according to old e1) 2024-06-03T08:35:29.008 INFO:teuthology.orchestra.run.smithi192.stderr:2024-06-03T08:35:29.007+0000 7fe2cdffb640 10 monclient: dump: 2024-06-03T08:35:29.008 INFO:teuthology.orchestra.run.smithi192.stderr:epoch 1 2024-06-03T08:35:29.008 INFO:teuthology.orchestra.run.smithi192.stderr:fsid ec3c3fdd-3ac6-4d7d-9412-683130ed8b04 2024-06-03T08:35:29.008 INFO:teuthology.orchestra.run.smithi192.stderr:last_changed 2024-06-03T08:34:31.885125+0000 2024-06-03T08:35:29.008 INFO:teuthology.orchestra.run.smithi192.stderr:created 2024-06-03T08:34:31.885125+0000 2024-06-03T08:35:29.008 INFO:teuthology.orchestra.run.smithi192.stderr:min_mon_release 18 (reef) 2024-06-03T08:35:29.008 INFO:teuthology.orchestra.run.smithi192.stderr:election_strategy: 3 2024-06-03T08:35:29.008 INFO:teuthology.orchestra.run.smithi192.stderr:0: v1:172.21.15.88:6789/0 mon.a 2024-06-03T08:35:29.008 INFO:teuthology.orchestra.run.smithi192.stderr:1: v1:172.21.15.192:6789/0 mon.b 2024-06-03T08:35:29.008 INFO:teuthology.orchestra.run.smithi192.stderr:2: v1:172.21.15.88:6790/0 mon.c 2024-06-03T08:35:29.008 INFO:teuthology.orchestra.run.smithi192.stderr: 2024-06-03T08:35:29.008 INFO:teuthology.orchestra.run.smithi192.stderr:2024-06-03T08:35:29.007+0000 7fe2d54e2640 10 monclient: get_monmap_and_config success 2024-06-03T08:35:29.009 INFO:teuthology.orchestra.run.smithi192.stderr:2024-06-03T08:35:29.007+0000 7fe2d54e2640 4 set_mon_vals no callback set 2024-06-03T08:35:29.011 INFO:teuthology.orchestra.run.smithi192.stderr:2024-06-03T08:35:29.007+0000 7fe2d54e2640 10 monclient: shutdown 2024-06-03T08:35:29.011 INFO:teuthology.orchestra.run.smithi192.stderr:2024-06-03T08:35:29.007+0000 7fe2d54e2640 20 monclient: shutdown discarding 0 pending message(s) 2024-06-03T08:35:29.011 INFO:teuthology.orchestra.run.smithi192.stderr:2024-06-03T08:35:29.007+0000 7fe2d54e2640 1 -- 172.21.15.192:0/3809358033 >> v1:172.21.15.192:6789/0 conn(0x7fe2d00af490 legacy=0x7fe2d014ce70 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-06-03T08:35:29.011 INFO:teuthology.orchestra.run.smithi192.stderr:2024-06-03T08:35:29.008+0000 7fe2d54e2640 1 -- 172.21.15.192:0/3809358033 shutdown_connections 2024-06-03T08:35:29.011 INFO:teuthology.orchestra.run.smithi192.stderr:2024-06-03T08:35:29.008+0000 7fe2d54e2640 1 -- 172.21.15.192:0/3809358033 >> 172.21.15.192:0/3809358033 conn(0x7fe2d006a600 msgr2=0x7fe2d00b37a0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-06-03T08:35:29.011 INFO:teuthology.orchestra.run.smithi192.stderr:2024-06-03T08:35:29.008+0000 7fe2d54e2640 1 -- 172.21.15.192:0/3809358033 shutdown_connections 2024-06-03T08:35:29.011 INFO:teuthology.orchestra.run.smithi192.stderr:2024-06-03T08:35:29.008+0000 7fe2d54e2640 1 -- 172.21.15.192:0/3809358033 wait complete. 2024-06-03T08:35:29.012 INFO:teuthology.orchestra.run.smithi192.stderr:2024-06-03T08:35:29.008+0000 7fe2d54e2640 10 monclient: build_initial_monmap 2024-06-03T08:35:29.012 INFO:teuthology.orchestra.run.smithi192.stderr:2024-06-03T08:35:29.008+0000 7fe2d54e2640 10 monclient: monmap: 2024-06-03T08:35:29.012 INFO:teuthology.orchestra.run.smithi192.stderr:epoch 0 2024-06-03T08:35:29.012 INFO:teuthology.orchestra.run.smithi192.stderr:fsid 00000000-0000-0000-0000-000000000000 2024-06-03T08:35:29.012 INFO:teuthology.orchestra.run.smithi192.stderr:last_changed 0.000000 2024-06-03T08:35:29.012 INFO:teuthology.orchestra.run.smithi192.stderr:created 0.000000 2024-06-03T08:35:29.012 INFO:teuthology.orchestra.run.smithi192.stderr:min_mon_release 0 (unknown) 2024-06-03T08:35:29.012 INFO:teuthology.orchestra.run.smithi192.stderr:election_strategy: 1 2024-06-03T08:35:29.013 INFO:teuthology.orchestra.run.smithi192.stderr:0: v1:172.21.15.88:6789/0 mon.noname-a 2024-06-03T08:35:29.013 INFO:teuthology.orchestra.run.smithi192.stderr:1: v1:172.21.15.192:6789/0 mon.noname-b 2024-06-03T08:35:29.013 INFO:teuthology.orchestra.run.smithi192.stderr:2: v1:172.21.15.88:6790/0 mon.noname-c 2024-06-03T08:35:29.013 INFO:teuthology.orchestra.run.smithi192.stderr: 2024-06-03T08:35:29.013 INFO:teuthology.orchestra.run.smithi192.stderr:2024-06-03T08:35:29.009+0000 7fe2d54e2640 1 librados: starting msgr at 2024-06-03T08:35:29.013 INFO:teuthology.orchestra.run.smithi192.stderr:2024-06-03T08:35:29.009+0000 7fe2d54e2640 1 librados: starting objecter 2024-06-03T08:35:29.013 INFO:teuthology.orchestra.run.smithi192.stderr:2024-06-03T08:35:29.009+0000 7fe2d54e2640 1 Processor -- start 2024-06-03T08:35:29.013 INFO:teuthology.orchestra.run.smithi192.stderr:2024-06-03T08:35:29.009+0000 7fe2d54e2640 1 -- start start 2024-06-03T08:35:29.014 INFO:teuthology.orchestra.run.smithi192.stderr:2024-06-03T08:35:29.009+0000 7fe2d54e2640 1 librados: setting wanted keys 2024-06-03T08:35:29.014 INFO:teuthology.orchestra.run.smithi192.stderr:2024-06-03T08:35:29.009+0000 7fe2d54e2640 1 librados: calling monclient init 2024-06-03T08:35:29.014 INFO:teuthology.orchestra.run.smithi192.stderr:2024-06-03T08:35:29.009+0000 7fe2d54e2640 10 monclient: init 2024-06-03T08:35:29.014 INFO:teuthology.orchestra.run.smithi192.stderr:2024-06-03T08:35:29.009+0000 7fe2d54e2640 10 monclient: _reopen_session rank -1 2024-06-03T08:35:29.014 INFO:teuthology.orchestra.run.smithi192.stderr:2024-06-03T08:35:29.009+0000 7fe2d54e2640 10 monclient: _add_conns ranks=[1,2,0] 2024-06-03T08:35:29.014 INFO:teuthology.orchestra.run.smithi192.stderr:2024-06-03T08:35:29.009+0000 7fe2d54e2640 10 monclient(hunting): picked mon.noname-b con 0x7fe2d00ae7f0 addr v1:172.21.15.192:6789/0 2024-06-03T08:35:29.014 INFO:teuthology.orchestra.run.smithi192.stderr:2024-06-03T08:35:29.010+0000 7fe2d54e2640 10 monclient(hunting): picked mon.noname-c con 0x7fe2d00af490 addr v1:172.21.15.88:6790/0 2024-06-03T08:35:29.014 INFO:teuthology.orchestra.run.smithi192.stderr:2024-06-03T08:35:29.010+0000 7fe2d54e2640 10 monclient(hunting): picked mon.noname-a con 0x7fe2d00b4120 addr v1:172.21.15.88:6789/0 2024-06-03T08:35:29.014 INFO:teuthology.orchestra.run.smithi192.stderr:2024-06-03T08:35:29.010+0000 7fe2d54e2640 1 -- --> v1:172.21.15.88:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fe2d01cd130 con 0x7fe2d00b4120 2024-06-03T08:35:29.014 INFO:teuthology.orchestra.run.smithi192.stderr:2024-06-03T08:35:29.010+0000 7fe2d54e2640 1 -- --> v1:172.21.15.192:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fe2d01cd3a0 con 0x7fe2d00ae7f0 2024-06-03T08:35:29.014 INFO:teuthology.orchestra.run.smithi192.stderr:2024-06-03T08:35:29.010+0000 7fe2d54e2640 1 -- --> v1:172.21.15.88:6790/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fe2d01d6a00 con 0x7fe2d00af490 2024-06-03T08:35:29.014 INFO:teuthology.orchestra.run.smithi192.stderr:2024-06-03T08:35:29.010+0000 7fe2d54e2640 10 monclient(hunting): _renew_subs 2024-06-03T08:35:29.014 INFO:teuthology.orchestra.run.smithi192.stderr:2024-06-03T08:35:29.010+0000 7fe2d54e2640 10 monclient(hunting): authenticate will time out at 1194.821123s 2024-06-03T08:35:29.014 INFO:teuthology.orchestra.run.smithi192.stderr:2024-06-03T08:35:29.010+0000 7fe2ceffd640 1 --1- >> v1:172.21.15.192:6789/0 conn(0x7fe2d00ae7f0 0x7fe2d01cde10 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=1).handle_server_banner_and_identify peer v1:172.21.15.192:6789/0 says I am v1:172.21.15.192:43808/0 (socket says 172.21.15.192:43808) 2024-06-03T08:35:29.015 INFO:teuthology.orchestra.run.smithi192.stderr:2024-06-03T08:35:29.010+0000 7fe2ceffd640 1 -- 172.21.15.192:0/3991703371 learned_addr learned my addr 172.21.15.192:0/3991703371 (peer_addr_for_me v1:172.21.15.192:0/0) 2024-06-03T08:35:29.015 INFO:teuthology.orchestra.run.smithi192.stderr:2024-06-03T08:35:29.010+0000 7fe2b77fe640 10 client.?.objecter ms_handle_connect 0x7fe2d00ae7f0 2024-06-03T08:35:29.015 INFO:teuthology.orchestra.run.smithi192.stderr:2024-06-03T08:35:29.010+0000 7fe2b77fe640 10 client.?.objecter resend_mon_ops 2024-06-03T08:35:29.015 INFO:teuthology.orchestra.run.smithi192.stderr:2024-06-03T08:35:29.010+0000 7fe2b77fe640 10 client.?.objecter ms_handle_connect 0x7fe2d00b4120 2024-06-03T08:35:29.015 INFO:teuthology.orchestra.run.smithi192.stderr:2024-06-03T08:35:29.011+0000 7fe2b77fe640 10 client.?.objecter resend_mon_ops 2024-06-03T08:35:29.015 INFO:teuthology.orchestra.run.smithi192.stderr:2024-06-03T08:35:29.013+0000 7fe2b77fe640 1 -- 172.21.15.192:0/3991703371 <== mon.1 v1:172.21.15.192:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 3723923791 0 0) 0x7fe2d01cd3a0 con 0x7fe2d00ae7f0 2024-06-03T08:35:29.016 INFO:teuthology.orchestra.run.smithi192.stderr:2024-06-03T08:35:29.013+0000 7fe2b77fe640 10 monclient(hunting): _negotiate 2024-06-03T08:35:29.016 INFO:teuthology.orchestra.run.smithi192.stderr:2024-06-03T08:35:29.013+0000 7fe2b77fe640 10 monclient(hunting): _init_auth method 2 2024-06-03T08:35:29.016 INFO:teuthology.orchestra.run.smithi192.stderr:2024-06-03T08:35:29.013+0000 7fe2b77fe640 10 monclient(hunting): _init_auth creating new auth 2024-06-03T08:35:29.016 INFO:teuthology.orchestra.run.smithi192.stderr:2024-06-03T08:35:29.013+0000 7fe2b77fe640 10 monclient(hunting): my global_id is 4120 2024-06-03T08:35:29.016 INFO:teuthology.orchestra.run.smithi192.stderr:2024-06-03T08:35:29.013+0000 7fe2b77fe640 1 -- 172.21.15.192:0/3991703371 --> v1:172.21.15.192:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7fe2bc004eb0 con 0x7fe2d00ae7f0 2024-06-03T08:35:29.016 INFO:teuthology.orchestra.run.smithi192.stderr:2024-06-03T08:35:29.013+0000 7fe2b77fe640 1 -- 172.21.15.192:0/3991703371 <== mon.0 v1:172.21.15.88:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 1899257243 0 0) 0x7fe2d01cd130 con 0x7fe2d00b4120 2024-06-03T08:35:29.016 INFO:teuthology.orchestra.run.smithi192.stderr:2024-06-03T08:35:29.013+0000 7fe2b77fe640 10 monclient(hunting): _negotiate 2024-06-03T08:35:29.016 INFO:teuthology.orchestra.run.smithi192.stderr:2024-06-03T08:35:29.013+0000 7fe2b77fe640 10 monclient(hunting): _init_auth method 2 2024-06-03T08:35:29.016 INFO:teuthology.orchestra.run.smithi192.stderr:2024-06-03T08:35:29.013+0000 7fe2b77fe640 10 monclient(hunting): _init_auth creating new auth 2024-06-03T08:35:29.016 INFO:teuthology.orchestra.run.smithi192.stderr:2024-06-03T08:35:29.013+0000 7fe2b77fe640 10 monclient(hunting): my global_id is 4119 2024-06-03T08:35:29.016 INFO:teuthology.orchestra.run.smithi192.stderr:2024-06-03T08:35:29.013+0000 7fe2b77fe640 1 -- 172.21.15.192:0/3991703371 --> v1:172.21.15.88:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7fe2d01cd3a0 con 0x7fe2d00b4120 2024-06-03T08:35:29.019 INFO:teuthology.orchestra.run.smithi192.stderr:2024-06-03T08:35:29.018+0000 7fe2b77fe640 1 -- 172.21.15.192:0/3991703371 <== mon.1 v1:172.21.15.192:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 764+0+0 (unknown 2952769841 0 0) 0x7fe2bc004eb0 con 0x7fe2d00ae7f0 2024-06-03T08:35:29.019 INFO:teuthology.orchestra.run.smithi192.stderr:2024-06-03T08:35:29.018+0000 7fe2b77fe640 1 -- 172.21.15.192:0/3991703371 --> v1:172.21.15.192:6789/0 -- auth(proto 2 165 bytes epoch 0) v1 -- 0x7fe2d01cd130 con 0x7fe2d00ae7f0 2024-06-03T08:35:29.019 INFO:teuthology.orchestra.run.smithi192.stderr:2024-06-03T08:35:29.018+0000 7fe2b77fe640 1 -- 172.21.15.192:0/3991703371 <== mon.1 v1:172.21.15.192:6789/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 2234374065 0 0) 0x7fe2c0001970 con 0x7fe2d00ae7f0 2024-06-03T08:35:29.019 INFO:teuthology.orchestra.run.smithi192.stderr:2024-06-03T08:35:29.018+0000 7fe2b77fe640 10 monclient(hunting): handle_monmap mon_map magic: 0 v1 2024-06-03T08:35:29.019 INFO:teuthology.orchestra.run.smithi192.stderr:2024-06-03T08:35:29.018+0000 7fe2b77fe640 10 monclient(hunting): got monmap 1 from mon.noname-b (according to old e1) 2024-06-03T08:35:29.019 INFO:teuthology.orchestra.run.smithi192.stderr:2024-06-03T08:35:29.018+0000 7fe2b77fe640 10 monclient(hunting): dump: 2024-06-03T08:35:29.019 INFO:teuthology.orchestra.run.smithi192.stderr:epoch 1 2024-06-03T08:35:29.019 INFO:teuthology.orchestra.run.smithi192.stderr:fsid ec3c3fdd-3ac6-4d7d-9412-683130ed8b04 2024-06-03T08:35:29.019 INFO:teuthology.orchestra.run.smithi192.stderr:last_changed 2024-06-03T08:34:31.885125+0000 2024-06-03T08:35:29.019 INFO:teuthology.orchestra.run.smithi192.stderr:created 2024-06-03T08:34:31.885125+0000 2024-06-03T08:35:29.019 INFO:teuthology.orchestra.run.smithi192.stderr:min_mon_release 18 (reef) 2024-06-03T08:35:29.020 INFO:teuthology.orchestra.run.smithi192.stderr:election_strategy: 3 2024-06-03T08:35:29.020 INFO:teuthology.orchestra.run.smithi192.stderr:0: v1:172.21.15.88:6789/0 mon.a 2024-06-03T08:35:29.020 INFO:teuthology.orchestra.run.smithi192.stderr:1: v1:172.21.15.192:6789/0 mon.b 2024-06-03T08:35:29.020 INFO:teuthology.orchestra.run.smithi192.stderr:2: v1:172.21.15.88:6790/0 mon.c 2024-06-03T08:35:29.020 INFO:teuthology.orchestra.run.smithi192.stderr: 2024-06-03T08:35:29.020 INFO:teuthology.orchestra.run.smithi192.stderr:2024-06-03T08:35:29.018+0000 7fe2b77fe640 10 monclient(hunting): _finish_auth 0 2024-06-03T08:35:29.020 INFO:teuthology.orchestra.run.smithi192.stderr:2024-06-03T08:35:29.018+0000 7fe2b77fe640 1 -- 172.21.15.192:0/3991703371 <== mon.0 v1:172.21.15.88:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 764+0+0 (unknown 2957299646 0 0) 0x7fe2d01cd3a0 con 0x7fe2d00b4120 2024-06-03T08:35:29.020 INFO:teuthology.orchestra.run.smithi192.stderr:2024-06-03T08:35:29.018+0000 7fe2b77fe640 1 -- 172.21.15.192:0/3991703371 --> v1:172.21.15.88:6789/0 -- auth(proto 2 165 bytes epoch 0) v1 -- 0x7fe2bc004eb0 con 0x7fe2d00b4120 2024-06-03T08:35:29.020 INFO:teuthology.orchestra.run.smithi192.stderr:2024-06-03T08:35:29.018+0000 7fe2b77fe640 1 -- 172.21.15.192:0/3991703371 <== mon.0 v1:172.21.15.88:6789/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 2234374065 0 0) 0x7fe2c8001ec0 con 0x7fe2d00b4120 2024-06-03T08:35:29.020 INFO:teuthology.orchestra.run.smithi192.stderr:2024-06-03T08:35:29.018+0000 7fe2b77fe640 10 monclient(hunting): handle_monmap mon_map magic: 0 v1 2024-06-03T08:35:29.020 INFO:teuthology.orchestra.run.smithi192.stderr:2024-06-03T08:35:29.018+0000 7fe2b77fe640 10 monclient(hunting): got monmap 1 from mon.a (according to old e1) 2024-06-03T08:35:29.020 INFO:teuthology.orchestra.run.smithi192.stderr:2024-06-03T08:35:29.018+0000 7fe2b77fe640 10 monclient(hunting): dump: 2024-06-03T08:35:29.020 INFO:teuthology.orchestra.run.smithi192.stderr:epoch 1 2024-06-03T08:35:29.020 INFO:teuthology.orchestra.run.smithi192.stderr:fsid ec3c3fdd-3ac6-4d7d-9412-683130ed8b04 2024-06-03T08:35:29.020 INFO:teuthology.orchestra.run.smithi192.stderr:last_changed 2024-06-03T08:34:31.885125+0000 2024-06-03T08:35:29.021 INFO:teuthology.orchestra.run.smithi192.stderr:created 2024-06-03T08:34:31.885125+0000 2024-06-03T08:35:29.021 INFO:teuthology.orchestra.run.smithi192.stderr:min_mon_release 18 (reef) 2024-06-03T08:35:29.021 INFO:teuthology.orchestra.run.smithi192.stderr:election_strategy: 3 2024-06-03T08:35:29.021 INFO:teuthology.orchestra.run.smithi192.stderr:0: v1:172.21.15.88:6789/0 mon.a 2024-06-03T08:35:29.021 INFO:teuthology.orchestra.run.smithi192.stderr:1: v1:172.21.15.192:6789/0 mon.b 2024-06-03T08:35:29.021 INFO:teuthology.orchestra.run.smithi192.stderr:2: v1:172.21.15.88:6790/0 mon.c 2024-06-03T08:35:29.021 INFO:teuthology.orchestra.run.smithi192.stderr: 2024-06-03T08:35:29.024 INFO:teuthology.orchestra.run.smithi192.stderr:2024-06-03T08:35:29.022+0000 7fe2b77fe640 10 client.?.objecter ms_handle_connect 0x7fe2d00af490 2024-06-03T08:35:29.024 INFO:teuthology.orchestra.run.smithi192.stderr:2024-06-03T08:35:29.022+0000 7fe2b77fe640 10 client.?.objecter resend_mon_ops 2024-06-03T08:35:29.024 INFO:teuthology.orchestra.run.smithi192.stderr:2024-06-03T08:35:29.022+0000 7fe2b77fe640 1 -- 172.21.15.192:0/3991703371 <== mon.0 v1:172.21.15.88:6789/0 4 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 580+0+0 (unknown 3358479300 0 0) 0x7fe2bc004eb0 con 0x7fe2d00b4120 2024-06-03T08:35:29.024 INFO:teuthology.orchestra.run.smithi192.stderr:2024-06-03T08:35:29.022+0000 7fe2b77fe640 1 -- 172.21.15.192:0/3991703371 >> v1:172.21.15.88:6790/0 conn(0x7fe2d00af490 legacy=0x7fe2d01d6f20 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-06-03T08:35:29.024 INFO:teuthology.orchestra.run.smithi192.stderr:2024-06-03T08:35:29.022+0000 7fe2b77fe640 1 -- 172.21.15.192:0/3991703371 >> v1:172.21.15.192:6789/0 conn(0x7fe2d00ae7f0 legacy=0x7fe2d01cde10 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-06-03T08:35:29.024 INFO:teuthology.orchestra.run.smithi192.stderr:2024-06-03T08:35:29.022+0000 7fe2b77fe640 10 monclient: _finish_hunting 0 2024-06-03T08:35:29.024 INFO:teuthology.orchestra.run.smithi192.stderr:2024-06-03T08:35:29.022+0000 7fe2b77fe640 1 monclient: found mon.a 2024-06-03T08:35:29.024 INFO:teuthology.orchestra.run.smithi192.stderr:2024-06-03T08:35:29.022+0000 7fe2b77fe640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-06-03T08:35:29.024 INFO:teuthology.orchestra.run.smithi192.stderr:2024-06-03T08:35:29.022+0000 7fe2b77fe640 10 monclient: _send_mon_message to mon.a at v1:172.21.15.88:6789/0 2024-06-03T08:35:29.024 INFO:teuthology.orchestra.run.smithi192.stderr:2024-06-03T08:35:29.022+0000 7fe2b77fe640 1 -- 172.21.15.192:0/3991703371 --> v1:172.21.15.88:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fe2d01d7630 con 0x7fe2d00b4120 2024-06-03T08:35:29.025 INFO:teuthology.orchestra.run.smithi192.stderr:2024-06-03T08:35:29.022+0000 7fe2b77fe640 10 monclient: _finish_auth 0 2024-06-03T08:35:29.025 INFO:teuthology.orchestra.run.smithi192.stderr:2024-06-03T08:35:29.022+0000 7fe2b77fe640 10 monclient: _check_auth_tickets 2024-06-03T08:35:29.025 INFO:teuthology.orchestra.run.smithi192.stderr:2024-06-03T08:35:29.022+0000 7fe2b77fe640 20 monclient: _check_auth_rotating not needed by client.admin 2024-06-03T08:35:29.025 INFO:teuthology.orchestra.run.smithi192.stderr:2024-06-03T08:35:29.022+0000 7fe2d54e2640 5 monclient: authenticate success, global_id 4119 2024-06-03T08:35:29.025 INFO:teuthology.orchestra.run.smithi192.stderr:2024-06-03T08:35:29.022+0000 7fe2d54e2640 10 monclient: _renew_subs 2024-06-03T08:35:29.025 INFO:teuthology.orchestra.run.smithi192.stderr:2024-06-03T08:35:29.022+0000 7fe2d54e2640 10 monclient: _send_mon_message to mon.a at v1:172.21.15.88:6789/0 2024-06-03T08:35:29.025 INFO:teuthology.orchestra.run.smithi192.stderr:2024-06-03T08:35:29.022+0000 7fe2d54e2640 1 -- 172.21.15.192:0/3991703371 --> v1:172.21.15.88:6789/0 -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fe2d01cafb0 con 0x7fe2d00b4120 2024-06-03T08:35:29.025 INFO:teuthology.orchestra.run.smithi192.stderr:2024-06-03T08:35:29.022+0000 7fe2d54e2640 10 client.4119.objecter _maybe_request_map subscribing (onetime) to next osd map 2024-06-03T08:35:29.025 INFO:teuthology.orchestra.run.smithi192.stderr:2024-06-03T08:35:29.022+0000 7fe2d54e2640 10 monclient: _renew_subs 2024-06-03T08:35:29.026 INFO:teuthology.orchestra.run.smithi192.stderr:2024-06-03T08:35:29.022+0000 7fe2d54e2640 10 monclient: _send_mon_message to mon.a at v1:172.21.15.88:6789/0 2024-06-03T08:35:29.026 INFO:teuthology.orchestra.run.smithi192.stderr:2024-06-03T08:35:29.022+0000 7fe2d54e2640 1 -- 172.21.15.192:0/3991703371 --> v1:172.21.15.88:6789/0 -- mon_subscribe({osdmap=0}) v3 -- 0x7fe2d013ade0 con 0x7fe2d00b4120 2024-06-03T08:35:29.026 INFO:teuthology.orchestra.run.smithi192.stderr:2024-06-03T08:35:29.022+0000 7fe2d54e2640 1 librados: init done 2024-06-03T08:35:29.026 INFO:teuthology.orchestra.run.smithi192.stderr:2024-06-03T08:35:29.023+0000 7fe2d54e2640 10 start_mon_command cmd=[{"prefix": "get_command_descriptions"}] 2024-06-03T08:35:29.026 INFO:teuthology.orchestra.run.smithi192.stderr:2024-06-03T08:35:29.023+0000 7fe2d54e2640 10 monclient: _send_command 1 [{"prefix": "get_command_descriptions"}] 2024-06-03T08:35:29.026 INFO:teuthology.orchestra.run.smithi192.stderr:2024-06-03T08:35:29.023+0000 7fe2d54e2640 10 monclient: _send_mon_message to mon.a at v1:172.21.15.88:6789/0 2024-06-03T08:35:29.026 INFO:teuthology.orchestra.run.smithi192.stderr:2024-06-03T08:35:29.023+0000 7fe2d54e2640 1 -- 172.21.15.192:0/3991703371 --> v1:172.21.15.88:6789/0 -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fe2d01d7820 con 0x7fe2d00b4120 2024-06-03T08:35:29.027 INFO:teuthology.orchestra.run.smithi192.stderr:2024-06-03T08:35:29.027+0000 7fe2b77fe640 1 -- 172.21.15.192:0/3991703371 <== mon.0 v1:172.21.15.88:6789/0 5 ==== config(0 keys) v1 ==== 4+0+0 (unknown 0 0 0) 0x7fe2c8001480 con 0x7fe2d00b4120 2024-06-03T08:35:29.027 INFO:teuthology.orchestra.run.smithi192.stderr:2024-06-03T08:35:29.027+0000 7fe2b77fe640 10 monclient: handle_config config(0 keys) v1 2024-06-03T08:35:29.028 INFO:teuthology.orchestra.run.smithi192.stderr:2024-06-03T08:35:29.027+0000 7fe2b77fe640 1 -- 172.21.15.192:0/3991703371 <== mon.0 v1:172.21.15.88:6789/0 6 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 2234374065 0 0) 0x7fe2c8004e10 con 0x7fe2d00b4120 2024-06-03T08:35:29.028 INFO:teuthology.orchestra.run.smithi192.stderr:2024-06-03T08:35:29.027+0000 7fe2b77fe640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-06-03T08:35:29.028 INFO:teuthology.orchestra.run.smithi192.stderr:2024-06-03T08:35:29.027+0000 7fe2b77fe640 10 monclient: got monmap 1 from mon.a (according to old e1) 2024-06-03T08:35:29.028 INFO:teuthology.orchestra.run.smithi192.stderr:2024-06-03T08:35:29.027+0000 7fe2b77fe640 10 monclient: dump: 2024-06-03T08:35:29.028 INFO:teuthology.orchestra.run.smithi192.stderr:epoch 1 2024-06-03T08:35:29.028 INFO:teuthology.orchestra.run.smithi192.stderr:fsid ec3c3fdd-3ac6-4d7d-9412-683130ed8b04 2024-06-03T08:35:29.028 INFO:teuthology.orchestra.run.smithi192.stderr:last_changed 2024-06-03T08:34:31.885125+0000 2024-06-03T08:35:29.028 INFO:teuthology.orchestra.run.smithi192.stderr:created 2024-06-03T08:34:31.885125+0000 2024-06-03T08:35:29.028 INFO:teuthology.orchestra.run.smithi192.stderr:min_mon_release 18 (reef) 2024-06-03T08:35:29.028 INFO:teuthology.orchestra.run.smithi192.stderr:election_strategy: 3 2024-06-03T08:35:29.028 INFO:teuthology.orchestra.run.smithi192.stderr:0: v1:172.21.15.88:6789/0 mon.a 2024-06-03T08:35:29.028 INFO:teuthology.orchestra.run.smithi192.stderr:1: v1:172.21.15.192:6789/0 mon.b 2024-06-03T08:35:29.028 INFO:teuthology.orchestra.run.smithi192.stderr:2: v1:172.21.15.88:6790/0 mon.c 2024-06-03T08:35:29.028 INFO:teuthology.orchestra.run.smithi192.stderr: 2024-06-03T08:35:29.028 INFO:teuthology.orchestra.run.smithi192.stderr:2024-06-03T08:35:29.027+0000 7fe2b77fe640 1 -- 172.21.15.192:0/3991703371 <== mon.0 v1:172.21.15.88:6789/0 7 ==== mgrmap(e 1) v1 ==== 668+0+0 (unknown 3836259515 0 0) 0x7fe2c8005270 con 0x7fe2d00b4120 2024-06-03T08:35:29.029 INFO:teuthology.orchestra.run.smithi192.stderr:2024-06-03T08:35:29.027+0000 7fe2b77fe640 1 -- 172.21.15.192:0/3991703371 <== mon.0 v1:172.21.15.88:6789/0 8 ==== osd_map(2..2 src has 1..2) v4 ==== 725+0+0 (unknown 2300978042 0 0) 0x7fe2c8006460 con 0x7fe2d00b4120 2024-06-03T08:35:29.029 INFO:teuthology.orchestra.run.smithi192.stderr:2024-06-03T08:35:29.027+0000 7fe2cd7fa640 4 set_mon_vals no callback set 2024-06-03T08:35:29.029 INFO:teuthology.orchestra.run.smithi192.stderr:2024-06-03T08:35:29.027+0000 7fe2b77fe640 10 client.4119.objecter ms_dispatch 0x7fe2d0000b90 osd_map(2..2 src has 1..2) v4 2024-06-03T08:35:29.029 INFO:teuthology.orchestra.run.smithi192.stderr:2024-06-03T08:35:29.027+0000 7fe2b77fe640 3 client.4119.objecter handle_osd_map got epochs [2,2] > 0 2024-06-03T08:35:29.029 INFO:teuthology.orchestra.run.smithi192.stderr:2024-06-03T08:35:29.027+0000 7fe2b77fe640 3 client.4119.objecter handle_osd_map decoding full epoch 2 2024-06-03T08:35:29.029 INFO:teuthology.orchestra.run.smithi192.stderr:2024-06-03T08:35:29.027+0000 7fe2b77fe640 20 client.4119.objecter dump_active .. 0 homeless 2024-06-03T08:35:29.123 INFO:teuthology.orchestra.run.smithi192.stderr:2024-06-03T08:35:29.123+0000 7fe2b77fe640 1 -- 172.21.15.192:0/3991703371 <== mon.0 v1:172.21.15.88:6789/0 9 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+73121 (unknown 1092875540 0 1417653156) 0x7fe2c80067e0 con 0x7fe2d00b4120 2024-06-03T08:35:29.123 INFO:teuthology.orchestra.run.smithi192.stderr:2024-06-03T08:35:29.123+0000 7fe2b77fe640 10 monclient: handle_mon_command_ack 1 [{"prefix": "get_command_descriptions"}] 2024-06-03T08:35:29.123 INFO:teuthology.orchestra.run.smithi192.stderr:2024-06-03T08:35:29.123+0000 7fe2b77fe640 10 monclient: _finish_command 1 = system:0 2024-06-03T08:35:29.195 INFO:teuthology.orchestra.run.smithi192.stderr:2024-06-03T08:35:29.195+0000 7fe2d54e2640 10 start_mon_command cmd=[{"prefix": "osd new", "uuid": "b25ab3bb-3587-45d0-991a-6d41bb529b37", "id": 0}] 2024-06-03T08:35:29.195 INFO:teuthology.orchestra.run.smithi192.stderr:2024-06-03T08:35:29.195+0000 7fe2d54e2640 10 monclient: _send_command 2 [{"prefix": "osd new", "uuid": "b25ab3bb-3587-45d0-991a-6d41bb529b37", "id": 0}] 2024-06-03T08:35:29.195 INFO:teuthology.orchestra.run.smithi192.stderr:2024-06-03T08:35:29.195+0000 7fe2d54e2640 10 monclient: _send_mon_message to mon.a at v1:172.21.15.88:6789/0 2024-06-03T08:35:29.195 INFO:teuthology.orchestra.run.smithi192.stderr:2024-06-03T08:35:29.195+0000 7fe2d54e2640 1 -- 172.21.15.192:0/3991703371 --> v1:172.21.15.88:6789/0 -- mon_command({"prefix": "osd new", "uuid": "b25ab3bb-3587-45d0-991a-6d41bb529b37", "id": 0} v 0) v1 -- 0x7fe2d0071a70 con 0x7fe2d00b4120 2024-06-03T08:35:29.310 INFO:teuthology.orchestra.run.smithi192.stderr:2024-06-03T08:35:29.310+0000 7fe2b77fe640 1 -- 172.21.15.192:0/3991703371 <== mon.0 v1:172.21.15.88:6789/0 10 ==== mon_command_ack([{"prefix": "osd new", "uuid": "b25ab3bb-3587-45d0-991a-6d41bb529b37", "id": 0}]=0 v3) v1 ==== 112+0+2 (unknown 1189976671 0 732707515) 0x7fe2c80069d0 con 0x7fe2d00b4120 2024-06-03T08:35:29.310 INFO:teuthology.orchestra.run.smithi192.stderr:2024-06-03T08:35:29.310+0000 7fe2b77fe640 10 monclient: handle_mon_command_ack 2 [{"prefix": "osd new", "uuid": "b25ab3bb-3587-45d0-991a-6d41bb529b37", "id": 0}] 2024-06-03T08:35:29.310 INFO:teuthology.orchestra.run.smithi192.stderr:2024-06-03T08:35:29.310+0000 7fe2b77fe640 10 monclient: _finish_command 2 = system:0 2024-06-03T08:35:29.311 INFO:teuthology.orchestra.run.smithi192.stdout:0 2024-06-03T08:35:29.312 INFO:teuthology.orchestra.run.smithi192.stderr:2024-06-03T08:35:29.310+0000 7fe2d54e2640 10 librados: watch_flush enter 2024-06-03T08:35:29.312 INFO:teuthology.orchestra.run.smithi192.stderr:2024-06-03T08:35:29.310+0000 7fe2d54e2640 10 librados: watch_flush exit 2024-06-03T08:35:29.312 INFO:teuthology.orchestra.run.smithi192.stderr:2024-06-03T08:35:29.310+0000 7fe2d54e2640 20 client.4119.objecter shutdown clearing up homeless session... 2024-06-03T08:35:29.312 INFO:teuthology.orchestra.run.smithi192.stderr:2024-06-03T08:35:29.310+0000 7fe2d54e2640 10 client.4119.objecter successfully canceled tick 2024-06-03T08:35:29.312 INFO:teuthology.orchestra.run.smithi192.stderr:2024-06-03T08:35:29.310+0000 7fe2d54e2640 10 monclient: shutdown 2024-06-03T08:35:29.312 INFO:teuthology.orchestra.run.smithi192.stderr:2024-06-03T08:35:29.310+0000 7fe2d54e2640 20 monclient: shutdown discarding 0 pending message(s) 2024-06-03T08:35:29.313 INFO:teuthology.orchestra.run.smithi192.stderr:2024-06-03T08:35:29.310+0000 7fe2d54e2640 1 -- 172.21.15.192:0/3991703371 >> v1:172.21.15.88:6789/0 conn(0x7fe2d00b4120 legacy=0x7fe2d01e1310 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-06-03T08:35:29.313 INFO:teuthology.orchestra.run.smithi192.stderr:2024-06-03T08:35:29.311+0000 7fe2d54e2640 1 -- 172.21.15.192:0/3991703371 shutdown_connections 2024-06-03T08:35:29.313 INFO:teuthology.orchestra.run.smithi192.stderr:2024-06-03T08:35:29.311+0000 7fe2d54e2640 1 -- 172.21.15.192:0/3991703371 >> 172.21.15.192:0/3991703371 conn(0x7fe2d006a600 msgr2=0x7fe2d006f650 unknown :-1 s=STATE_NONE l=0).mark_down 2024-06-03T08:35:29.313 INFO:teuthology.orchestra.run.smithi192.stderr:2024-06-03T08:35:29.311+0000 7fe2d54e2640 1 -- 172.21.15.192:0/3991703371 shutdown_connections 2024-06-03T08:35:29.313 INFO:teuthology.orchestra.run.smithi192.stderr:2024-06-03T08:35:29.311+0000 7fe2d54e2640 1 -- 172.21.15.192:0/3991703371 wait complete. 2024-06-03T08:35:29.313 INFO:teuthology.orchestra.run.smithi192.stderr:2024-06-03T08:35:29.311+0000 7fe2d54e2640 1 librados: shutdown 2024-06-03T08:35:29.322 DEBUG:teuthology.orchestra.run.smithi192:> sudo ceph --cluster ceph osd new 00d160e7-9773-41d8-8d17-21e70620a0fd 1 2024-06-03T08:35:29.421 INFO:teuthology.orchestra.run.smithi192.stderr:2024-06-03T08:35:29.421+0000 7fb711816640 10 monclient: get_monmap_and_config 2024-06-03T08:35:29.421 INFO:teuthology.orchestra.run.smithi192.stderr:2024-06-03T08:35:29.421+0000 7fb711816640 10 monclient: build_initial_monmap 2024-06-03T08:35:29.421 INFO:teuthology.orchestra.run.smithi192.stderr:2024-06-03T08:35:29.421+0000 7fb711816640 10 monclient: monmap: 2024-06-03T08:35:29.421 INFO:teuthology.orchestra.run.smithi192.stderr:epoch 0 2024-06-03T08:35:29.422 INFO:teuthology.orchestra.run.smithi192.stderr:fsid ec3c3fdd-3ac6-4d7d-9412-683130ed8b04 2024-06-03T08:35:29.422 INFO:teuthology.orchestra.run.smithi192.stderr:last_changed 2024-06-03T08:35:29.422085+0000 2024-06-03T08:35:29.422 INFO:teuthology.orchestra.run.smithi192.stderr:created 2024-06-03T08:35:29.422085+0000 2024-06-03T08:35:29.422 INFO:teuthology.orchestra.run.smithi192.stderr:min_mon_release 0 (unknown) 2024-06-03T08:35:29.422 INFO:teuthology.orchestra.run.smithi192.stderr:election_strategy: 3 2024-06-03T08:35:29.422 INFO:teuthology.orchestra.run.smithi192.stderr:0: v1:172.21.15.88:6789/0 mon.noname-a 2024-06-03T08:35:29.422 INFO:teuthology.orchestra.run.smithi192.stderr:1: v1:172.21.15.192:6789/0 mon.noname-c 2024-06-03T08:35:29.422 INFO:teuthology.orchestra.run.smithi192.stderr:2: v1:172.21.15.88:6790/0 mon.noname-b 2024-06-03T08:35:29.422 INFO:teuthology.orchestra.run.smithi192.stderr: 2024-06-03T08:35:29.423 INFO:teuthology.orchestra.run.smithi192.stderr:2024-06-03T08:35:29.421+0000 7fb711816640 1 Processor -- start 2024-06-03T08:35:29.423 INFO:teuthology.orchestra.run.smithi192.stderr:2024-06-03T08:35:29.421+0000 7fb711816640 1 -- start start 2024-06-03T08:35:29.424 INFO:teuthology.orchestra.run.smithi192.stderr:2024-06-03T08:35:29.421+0000 7fb711816640 10 monclient: init 2024-06-03T08:35:29.424 INFO:teuthology.orchestra.run.smithi192.stderr:2024-06-03T08:35:29.422+0000 7fb711816640 10 monclient: _reopen_session rank -1 2024-06-03T08:35:29.424 INFO:teuthology.orchestra.run.smithi192.stderr:2024-06-03T08:35:29.422+0000 7fb711816640 10 monclient: _add_conns ranks=[2,0,1] 2024-06-03T08:35:29.424 INFO:teuthology.orchestra.run.smithi192.stderr:2024-06-03T08:35:29.422+0000 7fb711816640 10 monclient(hunting): picked mon.noname-b con 0x7fb70c0b43c0 addr v1:172.21.15.88:6790/0 2024-06-03T08:35:29.424 INFO:teuthology.orchestra.run.smithi192.stderr:2024-06-03T08:35:29.422+0000 7fb711816640 10 monclient(hunting): picked mon.noname-a con 0x7fb70c13e6c0 addr v1:172.21.15.88:6789/0 2024-06-03T08:35:29.424 INFO:teuthology.orchestra.run.smithi192.stderr:2024-06-03T08:35:29.422+0000 7fb711816640 10 monclient(hunting): picked mon.noname-c con 0x7fb70c14ae20 addr v1:172.21.15.192:6789/0 2024-06-03T08:35:29.424 INFO:teuthology.orchestra.run.smithi192.stderr:2024-06-03T08:35:29.422+0000 7fb711816640 1 -- --> v1:172.21.15.88:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fb70c0a8ed0 con 0x7fb70c13e6c0 2024-06-03T08:35:29.424 INFO:teuthology.orchestra.run.smithi192.stderr:2024-06-03T08:35:29.422+0000 7fb711816640 1 -- --> v1:172.21.15.192:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fb70c0aa420 con 0x7fb70c14ae20 2024-06-03T08:35:29.424 INFO:teuthology.orchestra.run.smithi192.stderr:2024-06-03T08:35:29.422+0000 7fb711816640 1 -- --> v1:172.21.15.88:6790/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fb70c0aa690 con 0x7fb70c0b43c0 2024-06-03T08:35:29.424 INFO:teuthology.orchestra.run.smithi192.stderr:2024-06-03T08:35:29.422+0000 7fb711816640 10 monclient(hunting): _renew_subs 2024-06-03T08:35:29.424 INFO:teuthology.orchestra.run.smithi192.stderr:2024-06-03T08:35:29.422+0000 7fb711816640 10 monclient(hunting): authenticate will time out at 1195.233654s 2024-06-03T08:35:29.424 INFO:teuthology.orchestra.run.smithi192.stderr:2024-06-03T08:35:29.423+0000 7fb70b7fe640 1 --1- >> v1:172.21.15.192:6789/0 conn(0x7fb70c14ae20 0x7fb70c14d210 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.192:6789/0 says I am v1:172.21.15.192:43818/0 (socket says 172.21.15.192:43818) 2024-06-03T08:35:29.425 INFO:teuthology.orchestra.run.smithi192.stderr:2024-06-03T08:35:29.423+0000 7fb70b7fe640 1 -- 172.21.15.192:0/3082137170 learned_addr learned my addr 172.21.15.192:0/3082137170 (peer_addr_for_me v1:172.21.15.192:0/0) 2024-06-03T08:35:29.426 INFO:teuthology.orchestra.run.smithi192.stderr:2024-06-03T08:35:29.426+0000 7fb709ffb640 1 -- 172.21.15.192:0/3082137170 <== mon.2 v1:172.21.15.88:6790/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 1406535820 0 0) 0x7fb70c0aa690 con 0x7fb70c0b43c0 2024-06-03T08:35:29.426 INFO:teuthology.orchestra.run.smithi192.stderr:2024-06-03T08:35:29.426+0000 7fb709ffb640 10 monclient(hunting): _negotiate 2024-06-03T08:35:29.426 INFO:teuthology.orchestra.run.smithi192.stderr:2024-06-03T08:35:29.426+0000 7fb709ffb640 10 monclient(hunting): _init_auth method 2 2024-06-03T08:35:29.426 INFO:teuthology.orchestra.run.smithi192.stderr:2024-06-03T08:35:29.426+0000 7fb709ffb640 10 monclient(hunting): _init_auth creating new auth 2024-06-03T08:35:29.427 INFO:teuthology.orchestra.run.smithi192.stderr:2024-06-03T08:35:29.426+0000 7fb709ffb640 10 monclient(hunting): my global_id is 4124 2024-06-03T08:35:29.427 INFO:teuthology.orchestra.run.smithi192.stderr:2024-06-03T08:35:29.426+0000 7fb709ffb640 1 -- 172.21.15.192:0/3082137170 --> v1:172.21.15.88:6790/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7fb6f4003520 con 0x7fb70c0b43c0 2024-06-03T08:35:29.427 INFO:teuthology.orchestra.run.smithi192.stderr:2024-06-03T08:35:29.426+0000 7fb709ffb640 1 -- 172.21.15.192:0/3082137170 <== mon.0 v1:172.21.15.88:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 2767930413 0 0) 0x7fb70c0a8ed0 con 0x7fb70c13e6c0 2024-06-03T08:35:29.427 INFO:teuthology.orchestra.run.smithi192.stderr:2024-06-03T08:35:29.426+0000 7fb709ffb640 10 monclient(hunting): _negotiate 2024-06-03T08:35:29.427 INFO:teuthology.orchestra.run.smithi192.stderr:2024-06-03T08:35:29.426+0000 7fb709ffb640 10 monclient(hunting): _init_auth method 2 2024-06-03T08:35:29.427 INFO:teuthology.orchestra.run.smithi192.stderr:2024-06-03T08:35:29.426+0000 7fb709ffb640 10 monclient(hunting): _init_auth creating new auth 2024-06-03T08:35:29.427 INFO:teuthology.orchestra.run.smithi192.stderr:2024-06-03T08:35:29.426+0000 7fb709ffb640 10 monclient(hunting): my global_id is 4122 2024-06-03T08:35:29.427 INFO:teuthology.orchestra.run.smithi192.stderr:2024-06-03T08:35:29.426+0000 7fb709ffb640 1 -- 172.21.15.192:0/3082137170 --> v1:172.21.15.88:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7fb70c0aa690 con 0x7fb70c13e6c0 2024-06-03T08:35:29.427 INFO:teuthology.orchestra.run.smithi192.stderr:2024-06-03T08:35:29.426+0000 7fb709ffb640 1 -- 172.21.15.192:0/3082137170 <== mon.1 v1:172.21.15.192:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 2581768473 0 0) 0x7fb70c0aa420 con 0x7fb70c14ae20 2024-06-03T08:35:29.427 INFO:teuthology.orchestra.run.smithi192.stderr:2024-06-03T08:35:29.426+0000 7fb709ffb640 10 monclient(hunting): _negotiate 2024-06-03T08:35:29.427 INFO:teuthology.orchestra.run.smithi192.stderr:2024-06-03T08:35:29.426+0000 7fb709ffb640 10 monclient(hunting): _init_auth method 2 2024-06-03T08:35:29.427 INFO:teuthology.orchestra.run.smithi192.stderr:2024-06-03T08:35:29.426+0000 7fb709ffb640 10 monclient(hunting): _init_auth creating new auth 2024-06-03T08:35:29.427 INFO:teuthology.orchestra.run.smithi192.stderr:2024-06-03T08:35:29.426+0000 7fb709ffb640 10 monclient(hunting): my global_id is 4123 2024-06-03T08:35:29.427 INFO:teuthology.orchestra.run.smithi192.stderr:2024-06-03T08:35:29.426+0000 7fb709ffb640 1 -- 172.21.15.192:0/3082137170 --> v1:172.21.15.192:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7fb70c0a8ed0 con 0x7fb70c14ae20 2024-06-03T08:35:29.430 INFO:teuthology.orchestra.run.smithi192.stderr:2024-06-03T08:35:29.429+0000 7fb709ffb640 1 -- 172.21.15.192:0/3082137170 <== mon.0 v1:172.21.15.88:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 214+0+0 (unknown 4240006035 0 0) 0x7fb70c0aa690 con 0x7fb70c13e6c0 2024-06-03T08:35:29.430 INFO:teuthology.orchestra.run.smithi192.stderr:2024-06-03T08:35:29.429+0000 7fb709ffb640 1 -- 172.21.15.192:0/3082137170 >> v1:172.21.15.88:6790/0 conn(0x7fb70c0b43c0 legacy=0x7fb70c1415b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-06-03T08:35:29.430 INFO:teuthology.orchestra.run.smithi192.stderr:2024-06-03T08:35:29.429+0000 7fb709ffb640 1 -- 172.21.15.192:0/3082137170 >> v1:172.21.15.192:6789/0 conn(0x7fb70c14ae20 legacy=0x7fb70c14d210 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-06-03T08:35:29.430 INFO:teuthology.orchestra.run.smithi192.stderr:2024-06-03T08:35:29.429+0000 7fb709ffb640 10 monclient: _finish_hunting 0 2024-06-03T08:35:29.430 INFO:teuthology.orchestra.run.smithi192.stderr:2024-06-03T08:35:29.429+0000 7fb709ffb640 1 monclient: found mon.noname-a 2024-06-03T08:35:29.430 INFO:teuthology.orchestra.run.smithi192.stderr:2024-06-03T08:35:29.429+0000 7fb709ffb640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-06-03T08:35:29.430 INFO:teuthology.orchestra.run.smithi192.stderr:2024-06-03T08:35:29.429+0000 7fb709ffb640 10 monclient: _send_mon_message to mon.noname-a at v1:172.21.15.88:6789/0 2024-06-03T08:35:29.430 INFO:teuthology.orchestra.run.smithi192.stderr:2024-06-03T08:35:29.429+0000 7fb709ffb640 1 -- 172.21.15.192:0/3082137170 --> v1:172.21.15.88:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fb70c0a87f0 con 0x7fb70c13e6c0 2024-06-03T08:35:29.430 INFO:teuthology.orchestra.run.smithi192.stderr:2024-06-03T08:35:29.429+0000 7fb709ffb640 10 monclient: _finish_auth 0 2024-06-03T08:35:29.430 INFO:teuthology.orchestra.run.smithi192.stderr:2024-06-03T08:35:29.429+0000 7fb709ffb640 10 monclient: _check_auth_tickets 2024-06-03T08:35:29.430 INFO:teuthology.orchestra.run.smithi192.stderr:2024-06-03T08:35:29.429+0000 7fb709ffb640 20 monclient: _check_auth_rotating not needed by client.admin 2024-06-03T08:35:29.431 INFO:teuthology.orchestra.run.smithi192.stderr:2024-06-03T08:35:29.429+0000 7fb709ffb640 1 -- 172.21.15.192:0/3082137170 <== mon.0 v1:172.21.15.88:6789/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 2234374065 0 0) 0x7fb7000015f0 con 0x7fb70c13e6c0 2024-06-03T08:35:29.431 INFO:teuthology.orchestra.run.smithi192.stderr:2024-06-03T08:35:29.429+0000 7fb709ffb640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-06-03T08:35:29.431 INFO:teuthology.orchestra.run.smithi192.stderr:2024-06-03T08:35:29.429+0000 7fb709ffb640 10 monclient: got monmap 1 from mon.noname-a (according to old e1) 2024-06-03T08:35:29.431 INFO:teuthology.orchestra.run.smithi192.stderr:2024-06-03T08:35:29.429+0000 7fb709ffb640 10 monclient: dump: 2024-06-03T08:35:29.431 INFO:teuthology.orchestra.run.smithi192.stderr:epoch 1 2024-06-03T08:35:29.431 INFO:teuthology.orchestra.run.smithi192.stderr:fsid ec3c3fdd-3ac6-4d7d-9412-683130ed8b04 2024-06-03T08:35:29.431 INFO:teuthology.orchestra.run.smithi192.stderr:last_changed 2024-06-03T08:34:31.885125+0000 2024-06-03T08:35:29.431 INFO:teuthology.orchestra.run.smithi192.stderr:created 2024-06-03T08:34:31.885125+0000 2024-06-03T08:35:29.431 INFO:teuthology.orchestra.run.smithi192.stderr:min_mon_release 18 (reef) 2024-06-03T08:35:29.431 INFO:teuthology.orchestra.run.smithi192.stderr:election_strategy: 3 2024-06-03T08:35:29.431 INFO:teuthology.orchestra.run.smithi192.stderr:0: v1:172.21.15.88:6789/0 mon.a 2024-06-03T08:35:29.431 INFO:teuthology.orchestra.run.smithi192.stderr:1: v1:172.21.15.192:6789/0 mon.b 2024-06-03T08:35:29.431 INFO:teuthology.orchestra.run.smithi192.stderr:2: v1:172.21.15.88:6790/0 mon.c 2024-06-03T08:35:29.432 INFO:teuthology.orchestra.run.smithi192.stderr: 2024-06-03T08:35:29.432 INFO:teuthology.orchestra.run.smithi192.stderr:2024-06-03T08:35:29.429+0000 7fb711816640 5 monclient: authenticate success, global_id 4122 2024-06-03T08:35:29.432 INFO:teuthology.orchestra.run.smithi192.stderr:2024-06-03T08:35:29.429+0000 7fb711816640 20 monclient: get_monmap_and_config waiting for monmap|config 2024-06-03T08:35:29.432 INFO:teuthology.orchestra.run.smithi192.stderr:2024-06-03T08:35:29.431+0000 7fb709ffb640 1 -- 172.21.15.192:0/3082137170 <== mon.0 v1:172.21.15.88:6789/0 4 ==== config(0 keys) v1 ==== 4+0+0 (unknown 0 0 0) 0x7fb700001810 con 0x7fb70c13e6c0 2024-06-03T08:35:29.432 INFO:teuthology.orchestra.run.smithi192.stderr:2024-06-03T08:35:29.431+0000 7fb709ffb640 10 monclient: handle_config config(0 keys) v1 2024-06-03T08:35:29.432 INFO:teuthology.orchestra.run.smithi192.stderr:2024-06-03T08:35:29.431+0000 7fb709ffb640 1 -- 172.21.15.192:0/3082137170 <== mon.0 v1:172.21.15.88:6789/0 5 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 2234374065 0 0) 0x7fb700001be0 con 0x7fb70c13e6c0 2024-06-03T08:35:29.432 INFO:teuthology.orchestra.run.smithi192.stderr:2024-06-03T08:35:29.431+0000 7fb709ffb640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-06-03T08:35:29.432 INFO:teuthology.orchestra.run.smithi192.stderr:2024-06-03T08:35:29.431+0000 7fb709ffb640 10 monclient: got monmap 1 from mon.a (according to old e1) 2024-06-03T08:35:29.433 INFO:teuthology.orchestra.run.smithi192.stderr:2024-06-03T08:35:29.431+0000 7fb709ffb640 10 monclient: dump: 2024-06-03T08:35:29.433 INFO:teuthology.orchestra.run.smithi192.stderr:epoch 1 2024-06-03T08:35:29.433 INFO:teuthology.orchestra.run.smithi192.stderr:fsid ec3c3fdd-3ac6-4d7d-9412-683130ed8b04 2024-06-03T08:35:29.433 INFO:teuthology.orchestra.run.smithi192.stderr:last_changed 2024-06-03T08:34:31.885125+0000 2024-06-03T08:35:29.433 INFO:teuthology.orchestra.run.smithi192.stderr:created 2024-06-03T08:34:31.885125+0000 2024-06-03T08:35:29.433 INFO:teuthology.orchestra.run.smithi192.stderr:min_mon_release 18 (reef) 2024-06-03T08:35:29.433 INFO:teuthology.orchestra.run.smithi192.stderr:election_strategy: 3 2024-06-03T08:35:29.433 INFO:teuthology.orchestra.run.smithi192.stderr:0: v1:172.21.15.88:6789/0 mon.a 2024-06-03T08:35:29.433 INFO:teuthology.orchestra.run.smithi192.stderr:1: v1:172.21.15.192:6789/0 mon.b 2024-06-03T08:35:29.433 INFO:teuthology.orchestra.run.smithi192.stderr:2: v1:172.21.15.88:6790/0 mon.c 2024-06-03T08:35:29.433 INFO:teuthology.orchestra.run.smithi192.stderr: 2024-06-03T08:35:29.433 INFO:teuthology.orchestra.run.smithi192.stderr:2024-06-03T08:35:29.431+0000 7fb711816640 10 monclient: get_monmap_and_config success 2024-06-03T08:35:29.433 INFO:teuthology.orchestra.run.smithi192.stderr:2024-06-03T08:35:29.431+0000 7fb711816640 4 set_mon_vals no callback set 2024-06-03T08:35:29.435 INFO:teuthology.orchestra.run.smithi192.stderr:2024-06-03T08:35:29.432+0000 7fb711816640 10 monclient: shutdown 2024-06-03T08:35:29.435 INFO:teuthology.orchestra.run.smithi192.stderr:2024-06-03T08:35:29.432+0000 7fb711816640 20 monclient: shutdown discarding 0 pending message(s) 2024-06-03T08:35:29.435 INFO:teuthology.orchestra.run.smithi192.stderr:2024-06-03T08:35:29.432+0000 7fb711816640 1 -- 172.21.15.192:0/3082137170 >> v1:172.21.15.88:6789/0 conn(0x7fb70c13e6c0 legacy=0x7fb70c13eb40 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-06-03T08:35:29.435 INFO:teuthology.orchestra.run.smithi192.stderr:2024-06-03T08:35:29.432+0000 7fb711816640 1 -- 172.21.15.192:0/3082137170 shutdown_connections 2024-06-03T08:35:29.435 INFO:teuthology.orchestra.run.smithi192.stderr:2024-06-03T08:35:29.432+0000 7fb711816640 1 -- 172.21.15.192:0/3082137170 >> 172.21.15.192:0/3082137170 conn(0x7fb70c0719d0 msgr2=0x7fb70c071de0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-06-03T08:35:29.435 INFO:teuthology.orchestra.run.smithi192.stderr:2024-06-03T08:35:29.433+0000 7fb711816640 1 -- 172.21.15.192:0/3082137170 shutdown_connections 2024-06-03T08:35:29.435 INFO:teuthology.orchestra.run.smithi192.stderr:2024-06-03T08:35:29.433+0000 7fb711816640 1 -- 172.21.15.192:0/3082137170 wait complete. 2024-06-03T08:35:29.436 INFO:teuthology.orchestra.run.smithi192.stderr:2024-06-03T08:35:29.434+0000 7fb711816640 10 monclient: build_initial_monmap 2024-06-03T08:35:29.436 INFO:teuthology.orchestra.run.smithi192.stderr:2024-06-03T08:35:29.434+0000 7fb711816640 10 monclient: monmap: 2024-06-03T08:35:29.436 INFO:teuthology.orchestra.run.smithi192.stderr:epoch 0 2024-06-03T08:35:29.436 INFO:teuthology.orchestra.run.smithi192.stderr:fsid 00000000-0000-0000-0000-000000000000 2024-06-03T08:35:29.436 INFO:teuthology.orchestra.run.smithi192.stderr:last_changed 0.000000 2024-06-03T08:35:29.436 INFO:teuthology.orchestra.run.smithi192.stderr:created 0.000000 2024-06-03T08:35:29.436 INFO:teuthology.orchestra.run.smithi192.stderr:min_mon_release 0 (unknown) 2024-06-03T08:35:29.436 INFO:teuthology.orchestra.run.smithi192.stderr:election_strategy: 1 2024-06-03T08:35:29.436 INFO:teuthology.orchestra.run.smithi192.stderr:0: v1:172.21.15.88:6789/0 mon.noname-a 2024-06-03T08:35:29.436 INFO:teuthology.orchestra.run.smithi192.stderr:1: v1:172.21.15.192:6789/0 mon.noname-b 2024-06-03T08:35:29.436 INFO:teuthology.orchestra.run.smithi192.stderr:2: v1:172.21.15.88:6790/0 mon.noname-c 2024-06-03T08:35:29.436 INFO:teuthology.orchestra.run.smithi192.stderr: 2024-06-03T08:35:29.436 INFO:teuthology.orchestra.run.smithi192.stderr:2024-06-03T08:35:29.434+0000 7fb711816640 1 librados: starting msgr at 2024-06-03T08:35:29.437 INFO:teuthology.orchestra.run.smithi192.stderr:2024-06-03T08:35:29.434+0000 7fb711816640 1 librados: starting objecter 2024-06-03T08:35:29.437 INFO:teuthology.orchestra.run.smithi192.stderr:2024-06-03T08:35:29.434+0000 7fb711816640 1 Processor -- start 2024-06-03T08:35:29.437 INFO:teuthology.orchestra.run.smithi192.stderr:2024-06-03T08:35:29.434+0000 7fb711816640 1 -- start start 2024-06-03T08:35:29.437 INFO:teuthology.orchestra.run.smithi192.stderr:2024-06-03T08:35:29.434+0000 7fb711816640 1 librados: setting wanted keys 2024-06-03T08:35:29.437 INFO:teuthology.orchestra.run.smithi192.stderr:2024-06-03T08:35:29.434+0000 7fb711816640 1 librados: calling monclient init 2024-06-03T08:35:29.437 INFO:teuthology.orchestra.run.smithi192.stderr:2024-06-03T08:35:29.434+0000 7fb711816640 10 monclient: init 2024-06-03T08:35:29.438 INFO:teuthology.orchestra.run.smithi192.stderr:2024-06-03T08:35:29.435+0000 7fb711816640 10 monclient: _reopen_session rank -1 2024-06-03T08:35:29.438 INFO:teuthology.orchestra.run.smithi192.stderr:2024-06-03T08:35:29.435+0000 7fb711816640 10 monclient: _add_conns ranks=[0,2,1] 2024-06-03T08:35:29.438 INFO:teuthology.orchestra.run.smithi192.stderr:2024-06-03T08:35:29.435+0000 7fb711816640 10 monclient(hunting): picked mon.noname-a con 0x7fb70c0b43c0 addr v1:172.21.15.88:6789/0 2024-06-03T08:35:29.438 INFO:teuthology.orchestra.run.smithi192.stderr:2024-06-03T08:35:29.435+0000 7fb711816640 10 monclient(hunting): picked mon.noname-c con 0x7fb70c13e6c0 addr v1:172.21.15.88:6790/0 2024-06-03T08:35:29.438 INFO:teuthology.orchestra.run.smithi192.stderr:2024-06-03T08:35:29.435+0000 7fb711816640 10 monclient(hunting): picked mon.noname-b con 0x7fb70c14ae20 addr v1:172.21.15.192:6789/0 2024-06-03T08:35:29.439 INFO:teuthology.orchestra.run.smithi192.stderr:2024-06-03T08:35:29.435+0000 7fb711816640 1 -- --> v1:172.21.15.88:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fb70c1d9a20 con 0x7fb70c0b43c0 2024-06-03T08:35:29.439 INFO:teuthology.orchestra.run.smithi192.stderr:2024-06-03T08:35:29.435+0000 7fb711816640 1 -- --> v1:172.21.15.192:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fb70c1da650 con 0x7fb70c14ae20 2024-06-03T08:35:29.439 INFO:teuthology.orchestra.run.smithi192.stderr:2024-06-03T08:35:29.435+0000 7fb711816640 1 -- --> v1:172.21.15.88:6790/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fb70c144020 con 0x7fb70c13e6c0 2024-06-03T08:35:29.439 INFO:teuthology.orchestra.run.smithi192.stderr:2024-06-03T08:35:29.435+0000 7fb711816640 10 monclient(hunting): _renew_subs 2024-06-03T08:35:29.439 INFO:teuthology.orchestra.run.smithi192.stderr:2024-06-03T08:35:29.435+0000 7fb711816640 10 monclient(hunting): authenticate will time out at 1195.246964s 2024-06-03T08:35:29.439 INFO:teuthology.orchestra.run.smithi192.stderr:2024-06-03T08:35:29.436+0000 7fb70b7fe640 1 --1- >> v1:172.21.15.192:6789/0 conn(0x7fb70c14ae20 0x7fb70c1d6c70 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=1).handle_server_banner_and_identify peer v1:172.21.15.192:6789/0 says I am v1:172.21.15.192:43832/0 (socket says 172.21.15.192:43832) 2024-06-03T08:35:29.439 INFO:teuthology.orchestra.run.smithi192.stderr:2024-06-03T08:35:29.436+0000 7fb70b7fe640 1 -- 172.21.15.192:0/3998583559 learned_addr learned my addr 172.21.15.192:0/3998583559 (peer_addr_for_me v1:172.21.15.192:0/0) 2024-06-03T08:35:29.439 INFO:teuthology.orchestra.run.smithi192.stderr:2024-06-03T08:35:29.436+0000 7fb6f3fff640 10 client.?.objecter ms_handle_connect 0x7fb70c14ae20 2024-06-03T08:35:29.439 INFO:teuthology.orchestra.run.smithi192.stderr:2024-06-03T08:35:29.436+0000 7fb6f3fff640 10 client.?.objecter resend_mon_ops 2024-06-03T08:35:29.439 INFO:teuthology.orchestra.run.smithi192.stderr:2024-06-03T08:35:29.436+0000 7fb6f3fff640 10 client.?.objecter ms_handle_connect 0x7fb70c13e6c0 2024-06-03T08:35:29.439 INFO:teuthology.orchestra.run.smithi192.stderr:2024-06-03T08:35:29.436+0000 7fb6f3fff640 10 client.?.objecter resend_mon_ops 2024-06-03T08:35:29.439 INFO:teuthology.orchestra.run.smithi192.stderr:2024-06-03T08:35:29.436+0000 7fb6f3fff640 10 client.?.objecter ms_handle_connect 0x7fb70c0b43c0 2024-06-03T08:35:29.439 INFO:teuthology.orchestra.run.smithi192.stderr:2024-06-03T08:35:29.436+0000 7fb6f3fff640 10 client.?.objecter resend_mon_ops 2024-06-03T08:35:29.441 INFO:teuthology.orchestra.run.smithi192.stderr:2024-06-03T08:35:29.438+0000 7fb6f3fff640 1 -- 172.21.15.192:0/3998583559 <== mon.0 v1:172.21.15.88:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 1365028818 0 0) 0x7fb70c1d9a20 con 0x7fb70c0b43c0 2024-06-03T08:35:29.441 INFO:teuthology.orchestra.run.smithi192.stderr:2024-06-03T08:35:29.438+0000 7fb6f3fff640 10 monclient(hunting): _negotiate 2024-06-03T08:35:29.441 INFO:teuthology.orchestra.run.smithi192.stderr:2024-06-03T08:35:29.438+0000 7fb6f3fff640 10 monclient(hunting): _init_auth method 2 2024-06-03T08:35:29.441 INFO:teuthology.orchestra.run.smithi192.stderr:2024-06-03T08:35:29.438+0000 7fb6f3fff640 10 monclient(hunting): _init_auth creating new auth 2024-06-03T08:35:29.441 INFO:teuthology.orchestra.run.smithi192.stderr:2024-06-03T08:35:29.438+0000 7fb6f3fff640 10 monclient(hunting): my global_id is 4125 2024-06-03T08:35:29.441 INFO:teuthology.orchestra.run.smithi192.stderr:2024-06-03T08:35:29.438+0000 7fb6f3fff640 1 -- 172.21.15.192:0/3998583559 --> v1:172.21.15.88:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7fb6f8004eb0 con 0x7fb70c0b43c0 2024-06-03T08:35:29.441 INFO:teuthology.orchestra.run.smithi192.stderr:2024-06-03T08:35:29.439+0000 7fb6f3fff640 1 -- 172.21.15.192:0/3998583559 <== mon.2 v1:172.21.15.88:6790/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 332056980 0 0) 0x7fb70c144020 con 0x7fb70c13e6c0 2024-06-03T08:35:29.441 INFO:teuthology.orchestra.run.smithi192.stderr:2024-06-03T08:35:29.439+0000 7fb6f3fff640 10 monclient(hunting): _negotiate 2024-06-03T08:35:29.441 INFO:teuthology.orchestra.run.smithi192.stderr:2024-06-03T08:35:29.439+0000 7fb6f3fff640 10 monclient(hunting): _init_auth method 2 2024-06-03T08:35:29.441 INFO:teuthology.orchestra.run.smithi192.stderr:2024-06-03T08:35:29.439+0000 7fb6f3fff640 10 monclient(hunting): _init_auth creating new auth 2024-06-03T08:35:29.441 INFO:teuthology.orchestra.run.smithi192.stderr:2024-06-03T08:35:29.439+0000 7fb6f3fff640 10 monclient(hunting): my global_id is 4127 2024-06-03T08:35:29.441 INFO:teuthology.orchestra.run.smithi192.stderr:2024-06-03T08:35:29.439+0000 7fb6f3fff640 1 -- 172.21.15.192:0/3998583559 --> v1:172.21.15.88:6790/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7fb70c1d9a20 con 0x7fb70c13e6c0 2024-06-03T08:35:29.441 INFO:teuthology.orchestra.run.smithi192.stderr:2024-06-03T08:35:29.439+0000 7fb6f3fff640 1 -- 172.21.15.192:0/3998583559 <== mon.1 v1:172.21.15.192:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 1671906338 0 0) 0x7fb70c1da650 con 0x7fb70c14ae20 2024-06-03T08:35:29.442 INFO:teuthology.orchestra.run.smithi192.stderr:2024-06-03T08:35:29.439+0000 7fb6f3fff640 10 monclient(hunting): _negotiate 2024-06-03T08:35:29.442 INFO:teuthology.orchestra.run.smithi192.stderr:2024-06-03T08:35:29.439+0000 7fb6f3fff640 10 monclient(hunting): _init_auth method 2 2024-06-03T08:35:29.442 INFO:teuthology.orchestra.run.smithi192.stderr:2024-06-03T08:35:29.439+0000 7fb6f3fff640 10 monclient(hunting): _init_auth creating new auth 2024-06-03T08:35:29.442 INFO:teuthology.orchestra.run.smithi192.stderr:2024-06-03T08:35:29.439+0000 7fb6f3fff640 10 monclient(hunting): my global_id is 4126 2024-06-03T08:35:29.442 INFO:teuthology.orchestra.run.smithi192.stderr:2024-06-03T08:35:29.439+0000 7fb6f3fff640 1 -- 172.21.15.192:0/3998583559 --> v1:172.21.15.192:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7fb70c144020 con 0x7fb70c14ae20 2024-06-03T08:35:29.445 INFO:teuthology.orchestra.run.smithi192.stderr:2024-06-03T08:35:29.444+0000 7fb6f3fff640 1 -- 172.21.15.192:0/3998583559 <== mon.0 v1:172.21.15.88:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 764+0+0 (unknown 2546118490 0 0) 0x7fb6f8004eb0 con 0x7fb70c0b43c0 2024-06-03T08:35:29.445 INFO:teuthology.orchestra.run.smithi192.stderr:2024-06-03T08:35:29.444+0000 7fb6f3fff640 1 -- 172.21.15.192:0/3998583559 --> v1:172.21.15.88:6789/0 -- auth(proto 2 165 bytes epoch 0) v1 -- 0x7fb70c1da650 con 0x7fb70c0b43c0 2024-06-03T08:35:29.445 INFO:teuthology.orchestra.run.smithi192.stderr:2024-06-03T08:35:29.444+0000 7fb6f3fff640 1 -- 172.21.15.192:0/3998583559 <== mon.0 v1:172.21.15.88:6789/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 2234374065 0 0) 0x7fb6fc0018d0 con 0x7fb70c0b43c0 2024-06-03T08:35:29.445 INFO:teuthology.orchestra.run.smithi192.stderr:2024-06-03T08:35:29.444+0000 7fb6f3fff640 10 monclient(hunting): handle_monmap mon_map magic: 0 v1 2024-06-03T08:35:29.445 INFO:teuthology.orchestra.run.smithi192.stderr:2024-06-03T08:35:29.444+0000 7fb6f3fff640 10 monclient(hunting): got monmap 1 from mon.noname-a (according to old e1) 2024-06-03T08:35:29.445 INFO:teuthology.orchestra.run.smithi192.stderr:2024-06-03T08:35:29.444+0000 7fb6f3fff640 10 monclient(hunting): dump: 2024-06-03T08:35:29.445 INFO:teuthology.orchestra.run.smithi192.stderr:epoch 1 2024-06-03T08:35:29.445 INFO:teuthology.orchestra.run.smithi192.stderr:fsid ec3c3fdd-3ac6-4d7d-9412-683130ed8b04 2024-06-03T08:35:29.445 INFO:teuthology.orchestra.run.smithi192.stderr:last_changed 2024-06-03T08:34:31.885125+0000 2024-06-03T08:35:29.445 INFO:teuthology.orchestra.run.smithi192.stderr:created 2024-06-03T08:34:31.885125+0000 2024-06-03T08:35:29.445 INFO:teuthology.orchestra.run.smithi192.stderr:min_mon_release 18 (reef) 2024-06-03T08:35:29.445 INFO:teuthology.orchestra.run.smithi192.stderr:election_strategy: 3 2024-06-03T08:35:29.446 INFO:teuthology.orchestra.run.smithi192.stderr:0: v1:172.21.15.88:6789/0 mon.a 2024-06-03T08:35:29.446 INFO:teuthology.orchestra.run.smithi192.stderr:1: v1:172.21.15.192:6789/0 mon.b 2024-06-03T08:35:29.446 INFO:teuthology.orchestra.run.smithi192.stderr:2: v1:172.21.15.88:6790/0 mon.c 2024-06-03T08:35:29.446 INFO:teuthology.orchestra.run.smithi192.stderr: 2024-06-03T08:35:29.446 INFO:teuthology.orchestra.run.smithi192.stderr:2024-06-03T08:35:29.444+0000 7fb6f3fff640 10 monclient(hunting): _finish_auth 0 2024-06-03T08:35:29.446 INFO:teuthology.orchestra.run.smithi192.stderr:2024-06-03T08:35:29.444+0000 7fb6f3fff640 1 -- 172.21.15.192:0/3998583559 <== mon.1 v1:172.21.15.192:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 764+0+0 (unknown 14079493 0 0) 0x7fb70c144020 con 0x7fb70c14ae20 2024-06-03T08:35:29.446 INFO:teuthology.orchestra.run.smithi192.stderr:2024-06-03T08:35:29.444+0000 7fb6f3fff640 1 -- 172.21.15.192:0/3998583559 --> v1:172.21.15.192:6789/0 -- auth(proto 2 165 bytes epoch 0) v1 -- 0x7fb6f8004eb0 con 0x7fb70c14ae20 2024-06-03T08:35:29.446 INFO:teuthology.orchestra.run.smithi192.stderr:2024-06-03T08:35:29.444+0000 7fb6f3fff640 1 -- 172.21.15.192:0/3998583559 <== mon.1 v1:172.21.15.192:6789/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 2234374065 0 0) 0x7fb704001b90 con 0x7fb70c14ae20 2024-06-03T08:35:29.446 INFO:teuthology.orchestra.run.smithi192.stderr:2024-06-03T08:35:29.444+0000 7fb6f3fff640 10 monclient(hunting): handle_monmap mon_map magic: 0 v1 2024-06-03T08:35:29.446 INFO:teuthology.orchestra.run.smithi192.stderr:2024-06-03T08:35:29.444+0000 7fb6f3fff640 10 monclient(hunting): got monmap 1 from mon.b (according to old e1) 2024-06-03T08:35:29.446 INFO:teuthology.orchestra.run.smithi192.stderr:2024-06-03T08:35:29.444+0000 7fb6f3fff640 10 monclient(hunting): dump: 2024-06-03T08:35:29.446 INFO:teuthology.orchestra.run.smithi192.stderr:epoch 1 2024-06-03T08:35:29.446 INFO:teuthology.orchestra.run.smithi192.stderr:fsid ec3c3fdd-3ac6-4d7d-9412-683130ed8b04 2024-06-03T08:35:29.447 INFO:teuthology.orchestra.run.smithi192.stderr:last_changed 2024-06-03T08:34:31.885125+0000 2024-06-03T08:35:29.447 INFO:teuthology.orchestra.run.smithi192.stderr:created 2024-06-03T08:34:31.885125+0000 2024-06-03T08:35:29.447 INFO:teuthology.orchestra.run.smithi192.stderr:min_mon_release 18 (reef) 2024-06-03T08:35:29.447 INFO:teuthology.orchestra.run.smithi192.stderr:election_strategy: 3 2024-06-03T08:35:29.447 INFO:teuthology.orchestra.run.smithi192.stderr:0: v1:172.21.15.88:6789/0 mon.a 2024-06-03T08:35:29.447 INFO:teuthology.orchestra.run.smithi192.stderr:1: v1:172.21.15.192:6789/0 mon.b 2024-06-03T08:35:29.447 INFO:teuthology.orchestra.run.smithi192.stderr:2: v1:172.21.15.88:6790/0 mon.c 2024-06-03T08:35:29.447 INFO:teuthology.orchestra.run.smithi192.stderr: 2024-06-03T08:35:29.449 INFO:teuthology.orchestra.run.smithi192.stderr:2024-06-03T08:35:29.447+0000 7fb6f3fff640 1 -- 172.21.15.192:0/3998583559 <== mon.0 v1:172.21.15.88:6789/0 4 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 580+0+0 (unknown 728778700 0 0) 0x7fb70c1da650 con 0x7fb70c0b43c0 2024-06-03T08:35:29.449 INFO:teuthology.orchestra.run.smithi192.stderr:2024-06-03T08:35:29.447+0000 7fb6f3fff640 1 -- 172.21.15.192:0/3998583559 >> v1:172.21.15.88:6790/0 conn(0x7fb70c13e6c0 legacy=0x7fb70c1d6560 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-06-03T08:35:29.449 INFO:teuthology.orchestra.run.smithi192.stderr:2024-06-03T08:35:29.447+0000 7fb6f3fff640 1 -- 172.21.15.192:0/3998583559 >> v1:172.21.15.192:6789/0 conn(0x7fb70c14ae20 legacy=0x7fb70c1d6c70 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-06-03T08:35:29.449 INFO:teuthology.orchestra.run.smithi192.stderr:2024-06-03T08:35:29.447+0000 7fb6f3fff640 10 monclient: _finish_hunting 0 2024-06-03T08:35:29.449 INFO:teuthology.orchestra.run.smithi192.stderr:2024-06-03T08:35:29.447+0000 7fb6f3fff640 1 monclient: found mon.a 2024-06-03T08:35:29.449 INFO:teuthology.orchestra.run.smithi192.stderr:2024-06-03T08:35:29.447+0000 7fb6f3fff640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-06-03T08:35:29.449 INFO:teuthology.orchestra.run.smithi192.stderr:2024-06-03T08:35:29.447+0000 7fb6f3fff640 10 monclient: _send_mon_message to mon.a at v1:172.21.15.88:6789/0 2024-06-03T08:35:29.449 INFO:teuthology.orchestra.run.smithi192.stderr:2024-06-03T08:35:29.447+0000 7fb6f3fff640 1 -- 172.21.15.192:0/3998583559 --> v1:172.21.15.88:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fb70c1441b0 con 0x7fb70c0b43c0 2024-06-03T08:35:29.449 INFO:teuthology.orchestra.run.smithi192.stderr:2024-06-03T08:35:29.447+0000 7fb6f3fff640 10 monclient: _finish_auth 0 2024-06-03T08:35:29.449 INFO:teuthology.orchestra.run.smithi192.stderr:2024-06-03T08:35:29.447+0000 7fb6f3fff640 10 monclient: _check_auth_tickets 2024-06-03T08:35:29.450 INFO:teuthology.orchestra.run.smithi192.stderr:2024-06-03T08:35:29.447+0000 7fb6f3fff640 20 monclient: _check_auth_rotating not needed by client.admin 2024-06-03T08:35:29.450 INFO:teuthology.orchestra.run.smithi192.stderr:2024-06-03T08:35:29.447+0000 7fb711816640 5 monclient: authenticate success, global_id 4125 2024-06-03T08:35:29.450 INFO:teuthology.orchestra.run.smithi192.stderr:2024-06-03T08:35:29.447+0000 7fb711816640 10 monclient: _renew_subs 2024-06-03T08:35:29.450 INFO:teuthology.orchestra.run.smithi192.stderr:2024-06-03T08:35:29.447+0000 7fb711816640 10 monclient: _send_mon_message to mon.a at v1:172.21.15.88:6789/0 2024-06-03T08:35:29.450 INFO:teuthology.orchestra.run.smithi192.stderr:2024-06-03T08:35:29.447+0000 7fb711816640 1 -- 172.21.15.192:0/3998583559 --> v1:172.21.15.88:6789/0 -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fb70c1e1dc0 con 0x7fb70c0b43c0 2024-06-03T08:35:29.450 INFO:teuthology.orchestra.run.smithi192.stderr:2024-06-03T08:35:29.447+0000 7fb711816640 10 client.4125.objecter _maybe_request_map subscribing (onetime) to next osd map 2024-06-03T08:35:29.450 INFO:teuthology.orchestra.run.smithi192.stderr:2024-06-03T08:35:29.447+0000 7fb711816640 10 monclient: _renew_subs 2024-06-03T08:35:29.450 INFO:teuthology.orchestra.run.smithi192.stderr:2024-06-03T08:35:29.447+0000 7fb711816640 10 monclient: _send_mon_message to mon.a at v1:172.21.15.88:6789/0 2024-06-03T08:35:29.450 INFO:teuthology.orchestra.run.smithi192.stderr:2024-06-03T08:35:29.447+0000 7fb711816640 1 -- 172.21.15.192:0/3998583559 --> v1:172.21.15.88:6789/0 -- mon_subscribe({osdmap=0}) v3 -- 0x7fb70c1e2130 con 0x7fb70c0b43c0 2024-06-03T08:35:29.450 INFO:teuthology.orchestra.run.smithi192.stderr:2024-06-03T08:35:29.447+0000 7fb711816640 1 librados: init done 2024-06-03T08:35:29.450 INFO:teuthology.orchestra.run.smithi192.stderr:2024-06-03T08:35:29.448+0000 7fb711816640 10 start_mon_command cmd=[{"prefix": "get_command_descriptions"}] 2024-06-03T08:35:29.450 INFO:teuthology.orchestra.run.smithi192.stderr:2024-06-03T08:35:29.448+0000 7fb711816640 10 monclient: _send_command 1 [{"prefix": "get_command_descriptions"}] 2024-06-03T08:35:29.450 INFO:teuthology.orchestra.run.smithi192.stderr:2024-06-03T08:35:29.448+0000 7fb711816640 10 monclient: _send_mon_message to mon.a at v1:172.21.15.88:6789/0 2024-06-03T08:35:29.451 INFO:teuthology.orchestra.run.smithi192.stderr:2024-06-03T08:35:29.448+0000 7fb711816640 1 -- 172.21.15.192:0/3998583559 --> v1:172.21.15.88:6789/0 -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fb70c1d9810 con 0x7fb70c0b43c0 2024-06-03T08:35:29.496 INFO:tasks.ceph.mgr.x.smithi192.stderr:2024-06-03T08:35:29.496+0000 7f0e6be43040 -1 mgr[py] Module telegraf has missing NOTIFY_TYPES member 2024-06-03T08:35:29.550 INFO:teuthology.orchestra.run.smithi192.stderr:2024-06-03T08:35:29.549+0000 7fb6f3fff640 1 -- 172.21.15.192:0/3998583559 <== mon.0 v1:172.21.15.88:6789/0 5 ==== config(0 keys) v1 ==== 4+0+0 (unknown 0 0 0) 0x7fb6fc003610 con 0x7fb70c0b43c0 2024-06-03T08:35:29.550 INFO:teuthology.orchestra.run.smithi192.stderr:2024-06-03T08:35:29.549+0000 7fb6f3fff640 10 monclient: handle_config config(0 keys) v1 2024-06-03T08:35:29.550 INFO:teuthology.orchestra.run.smithi192.stderr:2024-06-03T08:35:29.549+0000 7fb6f3fff640 1 -- 172.21.15.192:0/3998583559 <== mon.0 v1:172.21.15.88:6789/0 6 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 2234374065 0 0) 0x7fb6fc003920 con 0x7fb70c0b43c0 2024-06-03T08:35:29.550 INFO:teuthology.orchestra.run.smithi192.stderr:2024-06-03T08:35:29.549+0000 7fb6f3fff640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-06-03T08:35:29.550 INFO:teuthology.orchestra.run.smithi192.stderr:2024-06-03T08:35:29.549+0000 7fb6f3fff640 10 monclient: got monmap 1 from mon.a (according to old e1) 2024-06-03T08:35:29.550 INFO:teuthology.orchestra.run.smithi192.stderr:2024-06-03T08:35:29.549+0000 7fb6f3fff640 10 monclient: dump: 2024-06-03T08:35:29.550 INFO:teuthology.orchestra.run.smithi192.stderr:epoch 1 2024-06-03T08:35:29.550 INFO:teuthology.orchestra.run.smithi192.stderr:fsid ec3c3fdd-3ac6-4d7d-9412-683130ed8b04 2024-06-03T08:35:29.550 INFO:teuthology.orchestra.run.smithi192.stderr:last_changed 2024-06-03T08:34:31.885125+0000 2024-06-03T08:35:29.551 INFO:teuthology.orchestra.run.smithi192.stderr:created 2024-06-03T08:34:31.885125+0000 2024-06-03T08:35:29.551 INFO:teuthology.orchestra.run.smithi192.stderr:min_mon_release 18 (reef) 2024-06-03T08:35:29.551 INFO:teuthology.orchestra.run.smithi192.stderr:election_strategy: 3 2024-06-03T08:35:29.551 INFO:teuthology.orchestra.run.smithi192.stderr:0: v1:172.21.15.88:6789/0 mon.a 2024-06-03T08:35:29.551 INFO:teuthology.orchestra.run.smithi192.stderr:1: v1:172.21.15.192:6789/0 mon.b 2024-06-03T08:35:29.551 INFO:teuthology.orchestra.run.smithi192.stderr:2: v1:172.21.15.88:6790/0 mon.c 2024-06-03T08:35:29.551 INFO:teuthology.orchestra.run.smithi192.stderr: 2024-06-03T08:35:29.551 INFO:teuthology.orchestra.run.smithi192.stderr:2024-06-03T08:35:29.549+0000 7fb6f3fff640 1 -- 172.21.15.192:0/3998583559 <== mon.0 v1:172.21.15.88:6789/0 7 ==== mgrmap(e 1) v1 ==== 668+0+0 (unknown 3836259515 0 0) 0x7fb6fc003d80 con 0x7fb70c0b43c0 2024-06-03T08:35:29.551 INFO:teuthology.orchestra.run.smithi192.stderr:2024-06-03T08:35:29.549+0000 7fb7097fa640 4 set_mon_vals no callback set 2024-06-03T08:35:29.552 INFO:teuthology.orchestra.run.smithi192.stderr:2024-06-03T08:35:29.551+0000 7fb6f3fff640 1 -- 172.21.15.192:0/3998583559 <== mon.0 v1:172.21.15.88:6789/0 8 ==== osd_map(3..3 src has 1..3) v4 ==== 840+0+0 (unknown 1168880041 0 0) 0x7fb6fc004f80 con 0x7fb70c0b43c0 2024-06-03T08:35:29.552 INFO:teuthology.orchestra.run.smithi192.stderr:2024-06-03T08:35:29.551+0000 7fb6f3fff640 10 client.4125.objecter ms_dispatch 0x7fb70c000b90 osd_map(3..3 src has 1..3) v4 2024-06-03T08:35:29.552 INFO:teuthology.orchestra.run.smithi192.stderr:2024-06-03T08:35:29.551+0000 7fb6f3fff640 3 client.4125.objecter handle_osd_map got epochs [3,3] > 0 2024-06-03T08:35:29.552 INFO:teuthology.orchestra.run.smithi192.stderr:2024-06-03T08:35:29.551+0000 7fb6f3fff640 3 client.4125.objecter handle_osd_map decoding full epoch 3 2024-06-03T08:35:29.552 INFO:teuthology.orchestra.run.smithi192.stderr:2024-06-03T08:35:29.551+0000 7fb6f3fff640 20 client.4125.objecter dump_active .. 0 homeless 2024-06-03T08:35:29.552 INFO:teuthology.orchestra.run.smithi192.stderr:2024-06-03T08:35:29.551+0000 7fb6f3fff640 1 -- 172.21.15.192:0/3998583559 <== mon.0 v1:172.21.15.88:6789/0 9 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+73121 (unknown 1092875540 0 1417653156) 0x7fb6fc004640 con 0x7fb70c0b43c0 2024-06-03T08:35:29.552 INFO:teuthology.orchestra.run.smithi192.stderr:2024-06-03T08:35:29.551+0000 7fb6f3fff640 10 monclient: handle_mon_command_ack 1 [{"prefix": "get_command_descriptions"}] 2024-06-03T08:35:29.552 INFO:teuthology.orchestra.run.smithi192.stderr:2024-06-03T08:35:29.551+0000 7fb6f3fff640 10 monclient: _finish_command 1 = system:0 2024-06-03T08:35:29.587 INFO:tasks.ceph.mgr.x.smithi192.stderr:2024-06-03T08:35:29.587+0000 7f0e6be43040 -1 mgr[py] Module osd_support has missing NOTIFY_TYPES member 2024-06-03T08:35:29.625 INFO:teuthology.orchestra.run.smithi192.stderr:2024-06-03T08:35:29.624+0000 7fb711816640 10 start_mon_command cmd=[{"prefix": "osd new", "uuid": "00d160e7-9773-41d8-8d17-21e70620a0fd", "id": 1}] 2024-06-03T08:35:29.625 INFO:teuthology.orchestra.run.smithi192.stderr:2024-06-03T08:35:29.624+0000 7fb711816640 10 monclient: _send_command 2 [{"prefix": "osd new", "uuid": "00d160e7-9773-41d8-8d17-21e70620a0fd", "id": 1}] 2024-06-03T08:35:29.625 INFO:teuthology.orchestra.run.smithi192.stderr:2024-06-03T08:35:29.624+0000 7fb711816640 10 monclient: _send_mon_message to mon.a at v1:172.21.15.88:6789/0 2024-06-03T08:35:29.625 INFO:teuthology.orchestra.run.smithi192.stderr:2024-06-03T08:35:29.624+0000 7fb711816640 1 -- 172.21.15.192:0/3998583559 --> v1:172.21.15.88:6789/0 -- mon_command({"prefix": "osd new", "uuid": "00d160e7-9773-41d8-8d17-21e70620a0fd", "id": 1} v 0) v1 -- 0x7fb70c04ea60 con 0x7fb70c0b43c0 2024-06-03T08:35:29.632 INFO:tasks.ceph.mgr.x.smithi192.stderr:2024-06-03T08:35:29.631+0000 7f0e6be43040 -1 mgr[py] Module iostat has missing NOTIFY_TYPES member 2024-06-03T08:35:29.641 INFO:tasks.ceph.mgr.y.smithi088.stderr:2024-06-03T08:35:29.641+0000 7ff0aea1d040 -1 mgr[py] Module telegraf has missing NOTIFY_TYPES member 2024-06-03T08:35:29.691 INFO:teuthology.orchestra.run.smithi192.stderr:2024-06-03T08:35:29.690+0000 7fb6f3fff640 1 -- 172.21.15.192:0/3998583559 <== mon.0 v1:172.21.15.88:6789/0 10 ==== mon_command_ack([{"prefix": "osd new", "uuid": "00d160e7-9773-41d8-8d17-21e70620a0fd", "id": 1}]=0 v4) v1 ==== 112+0+2 (unknown 2193802477 0 940483276) 0x7fb6fc004210 con 0x7fb70c0b43c0 2024-06-03T08:35:29.691 INFO:teuthology.orchestra.run.smithi192.stderr:2024-06-03T08:35:29.690+0000 7fb6f3fff640 10 monclient: handle_mon_command_ack 2 [{"prefix": "osd new", "uuid": "00d160e7-9773-41d8-8d17-21e70620a0fd", "id": 1}] 2024-06-03T08:35:29.691 INFO:teuthology.orchestra.run.smithi192.stderr:2024-06-03T08:35:29.690+0000 7fb6f3fff640 10 monclient: _finish_command 2 = system:0 2024-06-03T08:35:29.691 INFO:teuthology.orchestra.run.smithi192.stdout:1 2024-06-03T08:35:29.692 INFO:teuthology.orchestra.run.smithi192.stderr:2024-06-03T08:35:29.691+0000 7fb711816640 10 librados: watch_flush enter 2024-06-03T08:35:29.693 INFO:teuthology.orchestra.run.smithi192.stderr:2024-06-03T08:35:29.691+0000 7fb711816640 10 librados: watch_flush exit 2024-06-03T08:35:29.693 INFO:teuthology.orchestra.run.smithi192.stderr:2024-06-03T08:35:29.691+0000 7fb711816640 20 client.4125.objecter shutdown clearing up homeless session... 2024-06-03T08:35:29.693 INFO:teuthology.orchestra.run.smithi192.stderr:2024-06-03T08:35:29.691+0000 7fb711816640 10 client.4125.objecter successfully canceled tick 2024-06-03T08:35:29.693 INFO:teuthology.orchestra.run.smithi192.stderr:2024-06-03T08:35:29.692+0000 7fb711816640 10 monclient: shutdown 2024-06-03T08:35:29.693 INFO:teuthology.orchestra.run.smithi192.stderr:2024-06-03T08:35:29.692+0000 7fb711816640 20 monclient: shutdown discarding 0 pending message(s) 2024-06-03T08:35:29.693 INFO:teuthology.orchestra.run.smithi192.stderr:2024-06-03T08:35:29.692+0000 7fb711816640 1 -- 172.21.15.192:0/3998583559 >> v1:172.21.15.88:6789/0 conn(0x7fb70c0b43c0 legacy=0x7fb70c1d9f40 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-06-03T08:35:29.693 INFO:teuthology.orchestra.run.smithi192.stderr:2024-06-03T08:35:29.692+0000 7fb711816640 1 -- 172.21.15.192:0/3998583559 shutdown_connections 2024-06-03T08:35:29.693 INFO:teuthology.orchestra.run.smithi192.stderr:2024-06-03T08:35:29.692+0000 7fb711816640 1 -- 172.21.15.192:0/3998583559 >> 172.21.15.192:0/3998583559 conn(0x7fb70c0719d0 msgr2=0x7fb70c0b1a70 unknown :-1 s=STATE_NONE l=0).mark_down 2024-06-03T08:35:29.693 INFO:teuthology.orchestra.run.smithi192.stderr:2024-06-03T08:35:29.692+0000 7fb711816640 1 -- 172.21.15.192:0/3998583559 shutdown_connections 2024-06-03T08:35:29.693 INFO:teuthology.orchestra.run.smithi192.stderr:2024-06-03T08:35:29.692+0000 7fb711816640 1 -- 172.21.15.192:0/3998583559 wait complete. 2024-06-03T08:35:29.693 INFO:teuthology.orchestra.run.smithi192.stderr:2024-06-03T08:35:29.692+0000 7fb711816640 1 librados: shutdown 2024-06-03T08:35:29.704 DEBUG:teuthology.orchestra.run.smithi192:> sudo ceph --cluster ceph osd new addfb2a7-2bde-4cc3-98cf-17fd2bdfa3a9 2 2024-06-03T08:35:29.747 INFO:tasks.ceph.mgr.y.smithi088.stderr:2024-06-03T08:35:29.746+0000 7ff0aea1d040 -1 mgr[py] Module osd_support has missing NOTIFY_TYPES member 2024-06-03T08:35:29.794 INFO:tasks.ceph.mgr.y.smithi088.stderr:2024-06-03T08:35:29.793+0000 7ff0aea1d040 -1 mgr[py] Module iostat has missing NOTIFY_TYPES member 2024-06-03T08:35:29.808 INFO:teuthology.orchestra.run.smithi192.stderr:2024-06-03T08:35:29.807+0000 7f75f6daa640 10 monclient: get_monmap_and_config 2024-06-03T08:35:29.808 INFO:teuthology.orchestra.run.smithi192.stderr:2024-06-03T08:35:29.807+0000 7f75f6daa640 10 monclient: build_initial_monmap 2024-06-03T08:35:29.808 INFO:teuthology.orchestra.run.smithi192.stderr:2024-06-03T08:35:29.807+0000 7f75f6daa640 10 monclient: monmap: 2024-06-03T08:35:29.808 INFO:teuthology.orchestra.run.smithi192.stderr:epoch 0 2024-06-03T08:35:29.808 INFO:teuthology.orchestra.run.smithi192.stderr:fsid ec3c3fdd-3ac6-4d7d-9412-683130ed8b04 2024-06-03T08:35:29.808 INFO:teuthology.orchestra.run.smithi192.stderr:last_changed 2024-06-03T08:35:29.808633+0000 2024-06-03T08:35:29.808 INFO:teuthology.orchestra.run.smithi192.stderr:created 2024-06-03T08:35:29.808633+0000 2024-06-03T08:35:29.808 INFO:teuthology.orchestra.run.smithi192.stderr:min_mon_release 0 (unknown) 2024-06-03T08:35:29.808 INFO:teuthology.orchestra.run.smithi192.stderr:election_strategy: 3 2024-06-03T08:35:29.808 INFO:teuthology.orchestra.run.smithi192.stderr:0: v1:172.21.15.88:6789/0 mon.noname-a 2024-06-03T08:35:29.809 INFO:teuthology.orchestra.run.smithi192.stderr:1: v1:172.21.15.192:6789/0 mon.noname-c 2024-06-03T08:35:29.809 INFO:teuthology.orchestra.run.smithi192.stderr:2: v1:172.21.15.88:6790/0 mon.noname-b 2024-06-03T08:35:29.809 INFO:teuthology.orchestra.run.smithi192.stderr: 2024-06-03T08:35:29.810 INFO:teuthology.orchestra.run.smithi192.stderr:2024-06-03T08:35:29.808+0000 7f75f6daa640 1 Processor -- start 2024-06-03T08:35:29.810 INFO:teuthology.orchestra.run.smithi192.stderr:2024-06-03T08:35:29.808+0000 7f75f6daa640 1 -- start start 2024-06-03T08:35:29.810 INFO:teuthology.orchestra.run.smithi192.stderr:2024-06-03T08:35:29.808+0000 7f75f6daa640 10 monclient: init 2024-06-03T08:35:29.810 INFO:teuthology.orchestra.run.smithi192.stderr:2024-06-03T08:35:29.809+0000 7f75f6daa640 10 monclient: _reopen_session rank -1 2024-06-03T08:35:29.810 INFO:teuthology.orchestra.run.smithi192.stderr:2024-06-03T08:35:29.809+0000 7f75f6daa640 10 monclient: _add_conns ranks=[2,0,1] 2024-06-03T08:35:29.810 INFO:teuthology.orchestra.run.smithi192.stderr:2024-06-03T08:35:29.809+0000 7f75f6daa640 10 monclient(hunting): picked mon.noname-b con 0x7f75f00b43c0 addr v1:172.21.15.88:6790/0 2024-06-03T08:35:29.810 INFO:teuthology.orchestra.run.smithi192.stderr:2024-06-03T08:35:29.809+0000 7f75f6daa640 10 monclient(hunting): picked mon.noname-a con 0x7f75f013e6c0 addr v1:172.21.15.88:6789/0 2024-06-03T08:35:29.810 INFO:teuthology.orchestra.run.smithi192.stderr:2024-06-03T08:35:29.809+0000 7f75f6daa640 10 monclient(hunting): picked mon.noname-c con 0x7f75f014ae20 addr v1:172.21.15.192:6789/0 2024-06-03T08:35:29.811 INFO:teuthology.orchestra.run.smithi192.stderr:2024-06-03T08:35:29.809+0000 7f75f6daa640 1 -- --> v1:172.21.15.88:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f75f00a8ed0 con 0x7f75f013e6c0 2024-06-03T08:35:29.811 INFO:teuthology.orchestra.run.smithi192.stderr:2024-06-03T08:35:29.809+0000 7f75f6daa640 1 -- --> v1:172.21.15.192:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f75f00aa420 con 0x7f75f014ae20 2024-06-03T08:35:29.811 INFO:teuthology.orchestra.run.smithi192.stderr:2024-06-03T08:35:29.809+0000 7f75f6daa640 1 -- --> v1:172.21.15.88:6790/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f75f00aa690 con 0x7f75f00b43c0 2024-06-03T08:35:29.811 INFO:teuthology.orchestra.run.smithi192.stderr:2024-06-03T08:35:29.809+0000 7f75f6daa640 10 monclient(hunting): _renew_subs 2024-06-03T08:35:29.811 INFO:teuthology.orchestra.run.smithi192.stderr:2024-06-03T08:35:29.809+0000 7f75f6daa640 10 monclient(hunting): authenticate will time out at 1195.620380s 2024-06-03T08:35:29.811 INFO:tasks.ceph.mgr.x.smithi192.stderr:2024-06-03T08:35:29.809+0000 7f0e6be43040 -1 mgr[py] Module alerts has missing NOTIFY_TYPES member 2024-06-03T08:35:29.811 INFO:teuthology.orchestra.run.smithi192.stderr:2024-06-03T08:35:29.809+0000 7f75f4b46640 1 --1- >> v1:172.21.15.88:6790/0 conn(0x7f75f00b43c0 0x7f75f01415b0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.88:6790/0 says I am v1:172.21.15.192:60128/0 (socket says 172.21.15.192:60128) 2024-06-03T08:35:29.812 INFO:teuthology.orchestra.run.smithi192.stderr:2024-06-03T08:35:29.809+0000 7f75f4b46640 1 -- 172.21.15.192:0/2417188045 learned_addr learned my addr 172.21.15.192:0/2417188045 (peer_addr_for_me v1:172.21.15.192:0/0) 2024-06-03T08:35:29.813 INFO:teuthology.orchestra.run.smithi192.stderr:2024-06-03T08:35:29.812+0000 7f75ef7fe640 1 -- 172.21.15.192:0/2417188045 <== mon.0 v1:172.21.15.88:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 2853881362 0 0) 0x7f75f00a8ed0 con 0x7f75f013e6c0 2024-06-03T08:35:29.813 INFO:teuthology.orchestra.run.smithi192.stderr:2024-06-03T08:35:29.812+0000 7f75ef7fe640 10 monclient(hunting): _negotiate 2024-06-03T08:35:29.813 INFO:teuthology.orchestra.run.smithi192.stderr:2024-06-03T08:35:29.812+0000 7f75ef7fe640 10 monclient(hunting): _init_auth method 2 2024-06-03T08:35:29.813 INFO:teuthology.orchestra.run.smithi192.stderr:2024-06-03T08:35:29.812+0000 7f75ef7fe640 10 monclient(hunting): _init_auth creating new auth 2024-06-03T08:35:29.813 INFO:teuthology.orchestra.run.smithi192.stderr:2024-06-03T08:35:29.812+0000 7f75ef7fe640 10 monclient(hunting): my global_id is 4128 2024-06-03T08:35:29.813 INFO:teuthology.orchestra.run.smithi192.stderr:2024-06-03T08:35:29.812+0000 7f75ef7fe640 1 -- 172.21.15.192:0/2417188045 --> v1:172.21.15.88:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f75e0003520 con 0x7f75f013e6c0 2024-06-03T08:35:29.813 INFO:teuthology.orchestra.run.smithi192.stderr:2024-06-03T08:35:29.812+0000 7f75ef7fe640 1 -- 172.21.15.192:0/2417188045 <== mon.1 v1:172.21.15.192:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 3472413684 0 0) 0x7f75f00aa420 con 0x7f75f014ae20 2024-06-03T08:35:29.813 INFO:teuthology.orchestra.run.smithi192.stderr:2024-06-03T08:35:29.812+0000 7f75ef7fe640 10 monclient(hunting): _negotiate 2024-06-03T08:35:29.813 INFO:teuthology.orchestra.run.smithi192.stderr:2024-06-03T08:35:29.812+0000 7f75ef7fe640 10 monclient(hunting): _init_auth method 2 2024-06-03T08:35:29.813 INFO:teuthology.orchestra.run.smithi192.stderr:2024-06-03T08:35:29.812+0000 7f75ef7fe640 10 monclient(hunting): _init_auth creating new auth 2024-06-03T08:35:29.813 INFO:teuthology.orchestra.run.smithi192.stderr:2024-06-03T08:35:29.812+0000 7f75ef7fe640 10 monclient(hunting): my global_id is 4129 2024-06-03T08:35:29.814 INFO:teuthology.orchestra.run.smithi192.stderr:2024-06-03T08:35:29.812+0000 7f75ef7fe640 1 -- 172.21.15.192:0/2417188045 --> v1:172.21.15.192:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f75f00a8ed0 con 0x7f75f014ae20 2024-06-03T08:35:29.814 INFO:teuthology.orchestra.run.smithi192.stderr:2024-06-03T08:35:29.812+0000 7f75ef7fe640 1 -- 172.21.15.192:0/2417188045 <== mon.2 v1:172.21.15.88:6790/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 708917398 0 0) 0x7f75f00aa690 con 0x7f75f00b43c0 2024-06-03T08:35:29.814 INFO:teuthology.orchestra.run.smithi192.stderr:2024-06-03T08:35:29.812+0000 7f75ef7fe640 10 monclient(hunting): _negotiate 2024-06-03T08:35:29.814 INFO:teuthology.orchestra.run.smithi192.stderr:2024-06-03T08:35:29.812+0000 7f75ef7fe640 10 monclient(hunting): _init_auth method 2 2024-06-03T08:35:29.814 INFO:teuthology.orchestra.run.smithi192.stderr:2024-06-03T08:35:29.812+0000 7f75ef7fe640 10 monclient(hunting): _init_auth creating new auth 2024-06-03T08:35:29.814 INFO:teuthology.orchestra.run.smithi192.stderr:2024-06-03T08:35:29.812+0000 7f75ef7fe640 10 monclient(hunting): my global_id is 4130 2024-06-03T08:35:29.814 INFO:teuthology.orchestra.run.smithi192.stderr:2024-06-03T08:35:29.812+0000 7f75ef7fe640 1 -- 172.21.15.192:0/2417188045 --> v1:172.21.15.88:6790/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f75f00aa420 con 0x7f75f00b43c0 2024-06-03T08:35:29.816 INFO:teuthology.orchestra.run.smithi192.stderr:2024-06-03T08:35:29.815+0000 7f75ef7fe640 1 -- 172.21.15.192:0/2417188045 <== mon.2 v1:172.21.15.88:6790/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 214+0+0 (unknown 661913087 0 0) 0x7f75f00aa420 con 0x7f75f00b43c0 2024-06-03T08:35:29.816 INFO:teuthology.orchestra.run.smithi192.stderr:2024-06-03T08:35:29.815+0000 7f75ef7fe640 1 -- 172.21.15.192:0/2417188045 >> v1:172.21.15.192:6789/0 conn(0x7f75f014ae20 legacy=0x7f75f014d210 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-06-03T08:35:29.816 INFO:teuthology.orchestra.run.smithi192.stderr:2024-06-03T08:35:29.815+0000 7f75ef7fe640 1 -- 172.21.15.192:0/2417188045 >> v1:172.21.15.88:6789/0 conn(0x7f75f013e6c0 legacy=0x7f75f013eb40 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-06-03T08:35:29.816 INFO:teuthology.orchestra.run.smithi192.stderr:2024-06-03T08:35:29.815+0000 7f75ef7fe640 10 monclient: _finish_hunting 0 2024-06-03T08:35:29.817 INFO:teuthology.orchestra.run.smithi192.stderr:2024-06-03T08:35:29.815+0000 7f75ef7fe640 1 monclient: found mon.noname-b 2024-06-03T08:35:29.817 INFO:teuthology.orchestra.run.smithi192.stderr:2024-06-03T08:35:29.815+0000 7f75ef7fe640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-06-03T08:35:29.817 INFO:teuthology.orchestra.run.smithi192.stderr:2024-06-03T08:35:29.815+0000 7f75ef7fe640 10 monclient: _send_mon_message to mon.noname-b at v1:172.21.15.88:6790/0 2024-06-03T08:35:29.817 INFO:teuthology.orchestra.run.smithi192.stderr:2024-06-03T08:35:29.815+0000 7f75ef7fe640 1 -- 172.21.15.192:0/2417188045 --> v1:172.21.15.88:6790/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f75f00a87f0 con 0x7f75f00b43c0 2024-06-03T08:35:29.817 INFO:teuthology.orchestra.run.smithi192.stderr:2024-06-03T08:35:29.815+0000 7f75ef7fe640 10 monclient: _finish_auth 0 2024-06-03T08:35:29.817 INFO:teuthology.orchestra.run.smithi192.stderr:2024-06-03T08:35:29.815+0000 7f75ef7fe640 10 monclient: _check_auth_tickets 2024-06-03T08:35:29.817 INFO:teuthology.orchestra.run.smithi192.stderr:2024-06-03T08:35:29.815+0000 7f75ef7fe640 20 monclient: _check_auth_rotating not needed by client.admin 2024-06-03T08:35:29.817 INFO:teuthology.orchestra.run.smithi192.stderr:2024-06-03T08:35:29.816+0000 7f75f6daa640 5 monclient: authenticate success, global_id 4130 2024-06-03T08:35:29.817 INFO:teuthology.orchestra.run.smithi192.stderr:2024-06-03T08:35:29.816+0000 7f75f6daa640 20 monclient: get_monmap_and_config waiting for monmap|config 2024-06-03T08:35:29.817 INFO:teuthology.orchestra.run.smithi192.stderr:2024-06-03T08:35:29.816+0000 7f75ef7fe640 1 -- 172.21.15.192:0/2417188045 <== mon.2 v1:172.21.15.88:6790/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 2234374065 0 0) 0x7f75e40015b0 con 0x7f75f00b43c0 2024-06-03T08:35:29.817 INFO:teuthology.orchestra.run.smithi192.stderr:2024-06-03T08:35:29.816+0000 7f75ef7fe640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-06-03T08:35:29.817 INFO:teuthology.orchestra.run.smithi192.stderr:2024-06-03T08:35:29.816+0000 7f75ef7fe640 10 monclient: got monmap 1 from mon.noname-b (according to old e1) 2024-06-03T08:35:29.817 INFO:teuthology.orchestra.run.smithi192.stderr:2024-06-03T08:35:29.816+0000 7f75ef7fe640 10 monclient: dump: 2024-06-03T08:35:29.818 INFO:teuthology.orchestra.run.smithi192.stderr:epoch 1 2024-06-03T08:35:29.818 INFO:teuthology.orchestra.run.smithi192.stderr:fsid ec3c3fdd-3ac6-4d7d-9412-683130ed8b04 2024-06-03T08:35:29.818 INFO:teuthology.orchestra.run.smithi192.stderr:last_changed 2024-06-03T08:34:31.885125+0000 2024-06-03T08:35:29.818 INFO:teuthology.orchestra.run.smithi192.stderr:created 2024-06-03T08:34:31.885125+0000 2024-06-03T08:35:29.818 INFO:teuthology.orchestra.run.smithi192.stderr:min_mon_release 18 (reef) 2024-06-03T08:35:29.818 INFO:teuthology.orchestra.run.smithi192.stderr:election_strategy: 3 2024-06-03T08:35:29.818 INFO:teuthology.orchestra.run.smithi192.stderr:0: v1:172.21.15.88:6789/0 mon.a 2024-06-03T08:35:29.818 INFO:teuthology.orchestra.run.smithi192.stderr:1: v1:172.21.15.192:6789/0 mon.b 2024-06-03T08:35:29.818 INFO:teuthology.orchestra.run.smithi192.stderr:2: v1:172.21.15.88:6790/0 mon.c 2024-06-03T08:35:29.818 INFO:teuthology.orchestra.run.smithi192.stderr: 2024-06-03T08:35:29.818 INFO:teuthology.orchestra.run.smithi192.stderr:2024-06-03T08:35:29.816+0000 7f75f6daa640 20 monclient: get_monmap_and_config waiting for monmap|config 2024-06-03T08:35:29.842 INFO:teuthology.orchestra.run.smithi192.stderr:2024-06-03T08:35:29.842+0000 7f75ef7fe640 1 -- 172.21.15.192:0/2417188045 <== mon.2 v1:172.21.15.88:6790/0 4 ==== config(0 keys) v1 ==== 4+0+0 (unknown 0 0 0) 0x7f75e4001800 con 0x7f75f00b43c0 2024-06-03T08:35:29.843 INFO:teuthology.orchestra.run.smithi192.stderr:2024-06-03T08:35:29.842+0000 7f75ef7fe640 10 monclient: handle_config config(0 keys) v1 2024-06-03T08:35:29.843 INFO:teuthology.orchestra.run.smithi192.stderr:2024-06-03T08:35:29.842+0000 7f75ef7fe640 1 -- 172.21.15.192:0/2417188045 <== mon.2 v1:172.21.15.88:6790/0 5 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 2234374065 0 0) 0x7f75e4001b90 con 0x7f75f00b43c0 2024-06-03T08:35:29.843 INFO:teuthology.orchestra.run.smithi192.stderr:2024-06-03T08:35:29.842+0000 7f75ef7fe640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-06-03T08:35:29.843 INFO:teuthology.orchestra.run.smithi192.stderr:2024-06-03T08:35:29.842+0000 7f75ef7fe640 10 monclient: got monmap 1 from mon.c (according to old e1) 2024-06-03T08:35:29.843 INFO:teuthology.orchestra.run.smithi192.stderr:2024-06-03T08:35:29.842+0000 7f75ef7fe640 10 monclient: dump: 2024-06-03T08:35:29.843 INFO:teuthology.orchestra.run.smithi192.stderr:epoch 1 2024-06-03T08:35:29.843 INFO:teuthology.orchestra.run.smithi192.stderr:fsid ec3c3fdd-3ac6-4d7d-9412-683130ed8b04 2024-06-03T08:35:29.843 INFO:teuthology.orchestra.run.smithi192.stderr:last_changed 2024-06-03T08:34:31.885125+0000 2024-06-03T08:35:29.843 INFO:teuthology.orchestra.run.smithi192.stderr:created 2024-06-03T08:34:31.885125+0000 2024-06-03T08:35:29.843 INFO:teuthology.orchestra.run.smithi192.stderr:min_mon_release 18 (reef) 2024-06-03T08:35:29.843 INFO:teuthology.orchestra.run.smithi192.stderr:election_strategy: 3 2024-06-03T08:35:29.843 INFO:teuthology.orchestra.run.smithi192.stderr:0: v1:172.21.15.88:6789/0 mon.a 2024-06-03T08:35:29.843 INFO:teuthology.orchestra.run.smithi192.stderr:1: v1:172.21.15.192:6789/0 mon.b 2024-06-03T08:35:29.844 INFO:teuthology.orchestra.run.smithi192.stderr:2: v1:172.21.15.88:6790/0 mon.c 2024-06-03T08:35:29.844 INFO:teuthology.orchestra.run.smithi192.stderr: 2024-06-03T08:35:29.844 INFO:teuthology.orchestra.run.smithi192.stderr:2024-06-03T08:35:29.842+0000 7f75f6daa640 10 monclient: get_monmap_and_config success 2024-06-03T08:35:29.844 INFO:teuthology.orchestra.run.smithi192.stderr:2024-06-03T08:35:29.842+0000 7f75f6daa640 4 set_mon_vals no callback set 2024-06-03T08:35:29.847 INFO:teuthology.orchestra.run.smithi192.stderr:2024-06-03T08:35:29.843+0000 7f75f6daa640 10 monclient: shutdown 2024-06-03T08:35:29.847 INFO:teuthology.orchestra.run.smithi192.stderr:2024-06-03T08:35:29.843+0000 7f75f6daa640 20 monclient: shutdown discarding 0 pending message(s) 2024-06-03T08:35:29.847 INFO:teuthology.orchestra.run.smithi192.stderr:2024-06-03T08:35:29.843+0000 7f75f6daa640 1 -- 172.21.15.192:0/2417188045 >> v1:172.21.15.88:6790/0 conn(0x7f75f00b43c0 legacy=0x7f75f01415b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-06-03T08:35:29.847 INFO:teuthology.orchestra.run.smithi192.stderr:2024-06-03T08:35:29.843+0000 7f75f6daa640 1 -- 172.21.15.192:0/2417188045 shutdown_connections 2024-06-03T08:35:29.847 INFO:teuthology.orchestra.run.smithi192.stderr:2024-06-03T08:35:29.843+0000 7f75f6daa640 1 -- 172.21.15.192:0/2417188045 >> 172.21.15.192:0/2417188045 conn(0x7f75f00719d0 msgr2=0x7f75f0071de0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-06-03T08:35:29.847 INFO:teuthology.orchestra.run.smithi192.stderr:2024-06-03T08:35:29.843+0000 7f75f6daa640 1 -- 172.21.15.192:0/2417188045 shutdown_connections 2024-06-03T08:35:29.847 INFO:teuthology.orchestra.run.smithi192.stderr:2024-06-03T08:35:29.843+0000 7f75f6daa640 1 -- 172.21.15.192:0/2417188045 wait complete. 2024-06-03T08:35:29.847 INFO:teuthology.orchestra.run.smithi192.stderr:2024-06-03T08:35:29.843+0000 7f75f6daa640 10 monclient: build_initial_monmap 2024-06-03T08:35:29.847 INFO:teuthology.orchestra.run.smithi192.stderr:2024-06-03T08:35:29.843+0000 7f75f6daa640 10 monclient: monmap: 2024-06-03T08:35:29.847 INFO:teuthology.orchestra.run.smithi192.stderr:epoch 0 2024-06-03T08:35:29.847 INFO:teuthology.orchestra.run.smithi192.stderr:fsid 00000000-0000-0000-0000-000000000000 2024-06-03T08:35:29.848 INFO:teuthology.orchestra.run.smithi192.stderr:last_changed 0.000000 2024-06-03T08:35:29.848 INFO:teuthology.orchestra.run.smithi192.stderr:created 0.000000 2024-06-03T08:35:29.848 INFO:teuthology.orchestra.run.smithi192.stderr:min_mon_release 0 (unknown) 2024-06-03T08:35:29.848 INFO:teuthology.orchestra.run.smithi192.stderr:election_strategy: 1 2024-06-03T08:35:29.848 INFO:teuthology.orchestra.run.smithi192.stderr:0: v1:172.21.15.88:6789/0 mon.noname-a 2024-06-03T08:35:29.848 INFO:teuthology.orchestra.run.smithi192.stderr:1: v1:172.21.15.192:6789/0 mon.noname-b 2024-06-03T08:35:29.848 INFO:teuthology.orchestra.run.smithi192.stderr:2: v1:172.21.15.88:6790/0 mon.noname-c 2024-06-03T08:35:29.848 INFO:teuthology.orchestra.run.smithi192.stderr: 2024-06-03T08:35:29.848 INFO:teuthology.orchestra.run.smithi192.stderr:2024-06-03T08:35:29.844+0000 7f75f6daa640 1 librados: starting msgr at 2024-06-03T08:35:29.848 INFO:teuthology.orchestra.run.smithi192.stderr:2024-06-03T08:35:29.844+0000 7f75f6daa640 1 librados: starting objecter 2024-06-03T08:35:29.848 INFO:teuthology.orchestra.run.smithi192.stderr:2024-06-03T08:35:29.844+0000 7f75f6daa640 1 Processor -- start 2024-06-03T08:35:29.848 INFO:teuthology.orchestra.run.smithi192.stderr:2024-06-03T08:35:29.844+0000 7f75f6daa640 1 -- start start 2024-06-03T08:35:29.848 INFO:teuthology.orchestra.run.smithi192.stderr:2024-06-03T08:35:29.844+0000 7f75f6daa640 1 librados: setting wanted keys 2024-06-03T08:35:29.848 INFO:teuthology.orchestra.run.smithi192.stderr:2024-06-03T08:35:29.844+0000 7f75f6daa640 1 librados: calling monclient init 2024-06-03T08:35:29.849 INFO:teuthology.orchestra.run.smithi192.stderr:2024-06-03T08:35:29.844+0000 7f75f6daa640 10 monclient: init 2024-06-03T08:35:29.849 INFO:teuthology.orchestra.run.smithi192.stderr:2024-06-03T08:35:29.845+0000 7f75f6daa640 10 monclient: _reopen_session rank -1 2024-06-03T08:35:29.849 INFO:teuthology.orchestra.run.smithi192.stderr:2024-06-03T08:35:29.845+0000 7f75f6daa640 10 monclient: _add_conns ranks=[0,1,2] 2024-06-03T08:35:29.849 INFO:teuthology.orchestra.run.smithi192.stderr:2024-06-03T08:35:29.845+0000 7f75f6daa640 10 monclient(hunting): picked mon.noname-a con 0x7f75f00b43c0 addr v1:172.21.15.88:6789/0 2024-06-03T08:35:29.849 INFO:teuthology.orchestra.run.smithi192.stderr:2024-06-03T08:35:29.845+0000 7f75f6daa640 10 monclient(hunting): picked mon.noname-b con 0x7f75f013e6c0 addr v1:172.21.15.192:6789/0 2024-06-03T08:35:29.849 INFO:teuthology.orchestra.run.smithi192.stderr:2024-06-03T08:35:29.845+0000 7f75f6daa640 10 monclient(hunting): picked mon.noname-c con 0x7f75f014ae20 addr v1:172.21.15.88:6790/0 2024-06-03T08:35:29.849 INFO:teuthology.orchestra.run.smithi192.stderr:2024-06-03T08:35:29.845+0000 7f75f6daa640 1 -- --> v1:172.21.15.88:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f75f0144b60 con 0x7f75f00b43c0 2024-06-03T08:35:29.849 INFO:teuthology.orchestra.run.smithi192.stderr:2024-06-03T08:35:29.845+0000 7f75f6daa640 1 -- --> v1:172.21.15.192:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f75f01d9a50 con 0x7f75f013e6c0 2024-06-03T08:35:29.849 INFO:teuthology.orchestra.run.smithi192.stderr:2024-06-03T08:35:29.845+0000 7f75f6daa640 1 -- --> v1:172.21.15.88:6790/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f75f01da680 con 0x7f75f014ae20 2024-06-03T08:35:29.849 INFO:teuthology.orchestra.run.smithi192.stderr:2024-06-03T08:35:29.845+0000 7f75f6daa640 10 monclient(hunting): _renew_subs 2024-06-03T08:35:29.849 INFO:teuthology.orchestra.run.smithi192.stderr:2024-06-03T08:35:29.845+0000 7f75f6daa640 10 monclient(hunting): authenticate will time out at 1195.656157s 2024-06-03T08:35:29.849 INFO:teuthology.orchestra.run.smithi192.stderr:2024-06-03T08:35:29.845+0000 7f75f5347640 1 --1- >> v1:172.21.15.88:6790/0 conn(0x7f75f014ae20 0x7f75f01e1c50 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=1).handle_server_banner_and_identify peer v1:172.21.15.88:6790/0 says I am v1:172.21.15.192:60138/0 (socket says 172.21.15.192:60138) 2024-06-03T08:35:29.849 INFO:teuthology.orchestra.run.smithi192.stderr:2024-06-03T08:35:29.845+0000 7f75f5347640 1 -- 172.21.15.192:0/1811979606 learned_addr learned my addr 172.21.15.192:0/1811979606 (peer_addr_for_me v1:172.21.15.192:0/0) 2024-06-03T08:35:29.850 INFO:teuthology.orchestra.run.smithi192.stderr:2024-06-03T08:35:29.845+0000 7f75ed7fa640 10 client.?.objecter ms_handle_connect 0x7f75f014ae20 2024-06-03T08:35:29.850 INFO:teuthology.orchestra.run.smithi192.stderr:2024-06-03T08:35:29.845+0000 7f75ed7fa640 10 client.?.objecter resend_mon_ops 2024-06-03T08:35:29.850 INFO:teuthology.orchestra.run.smithi192.stderr:2024-06-03T08:35:29.846+0000 7f75ed7fa640 10 client.?.objecter ms_handle_connect 0x7f75f013e6c0 2024-06-03T08:35:29.850 INFO:teuthology.orchestra.run.smithi192.stderr:2024-06-03T08:35:29.846+0000 7f75ed7fa640 10 client.?.objecter resend_mon_ops 2024-06-03T08:35:29.850 INFO:teuthology.orchestra.run.smithi192.stderr:2024-06-03T08:35:29.846+0000 7f75ed7fa640 10 client.?.objecter ms_handle_connect 0x7f75f00b43c0 2024-06-03T08:35:29.850 INFO:teuthology.orchestra.run.smithi192.stderr:2024-06-03T08:35:29.846+0000 7f75ed7fa640 10 client.?.objecter resend_mon_ops 2024-06-03T08:35:29.851 INFO:teuthology.orchestra.run.smithi192.stderr:2024-06-03T08:35:29.847+0000 7f75ed7fa640 1 -- 172.21.15.192:0/1811979606 <== mon.2 v1:172.21.15.88:6790/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 1490076478 0 0) 0x7f75f01da680 con 0x7f75f014ae20 2024-06-03T08:35:29.851 INFO:teuthology.orchestra.run.smithi192.stderr:2024-06-03T08:35:29.847+0000 7f75ed7fa640 10 monclient(hunting): _negotiate 2024-06-03T08:35:29.851 INFO:teuthology.orchestra.run.smithi192.stderr:2024-06-03T08:35:29.847+0000 7f75ed7fa640 10 monclient(hunting): _init_auth method 2 2024-06-03T08:35:29.851 INFO:teuthology.orchestra.run.smithi192.stderr:2024-06-03T08:35:29.847+0000 7f75ed7fa640 10 monclient(hunting): _init_auth creating new auth 2024-06-03T08:35:29.851 INFO:teuthology.orchestra.run.smithi192.stderr:2024-06-03T08:35:29.847+0000 7f75ed7fa640 10 monclient(hunting): my global_id is 4133 2024-06-03T08:35:29.851 INFO:teuthology.orchestra.run.smithi192.stderr:2024-06-03T08:35:29.847+0000 7f75ed7fa640 1 -- 172.21.15.192:0/1811979606 --> v1:172.21.15.88:6790/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f75d4004eb0 con 0x7f75f014ae20 2024-06-03T08:35:29.851 INFO:teuthology.orchestra.run.smithi192.stderr:2024-06-03T08:35:29.848+0000 7f75ed7fa640 1 -- 172.21.15.192:0/1811979606 <== mon.0 v1:172.21.15.88:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 4222407964 0 0) 0x7f75f0144b60 con 0x7f75f00b43c0 2024-06-03T08:35:29.851 INFO:teuthology.orchestra.run.smithi192.stderr:2024-06-03T08:35:29.848+0000 7f75ed7fa640 10 monclient(hunting): _negotiate 2024-06-03T08:35:29.851 INFO:teuthology.orchestra.run.smithi192.stderr:2024-06-03T08:35:29.848+0000 7f75ed7fa640 10 monclient(hunting): _init_auth method 2 2024-06-03T08:35:29.851 INFO:teuthology.orchestra.run.smithi192.stderr:2024-06-03T08:35:29.848+0000 7f75ed7fa640 10 monclient(hunting): _init_auth creating new auth 2024-06-03T08:35:29.852 INFO:teuthology.orchestra.run.smithi192.stderr:2024-06-03T08:35:29.848+0000 7f75ed7fa640 10 monclient(hunting): my global_id is 4131 2024-06-03T08:35:29.852 INFO:teuthology.orchestra.run.smithi192.stderr:2024-06-03T08:35:29.848+0000 7f75ed7fa640 1 -- 172.21.15.192:0/1811979606 --> v1:172.21.15.88:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f75f01da680 con 0x7f75f00b43c0 2024-06-03T08:35:29.852 INFO:teuthology.orchestra.run.smithi192.stderr:2024-06-03T08:35:29.848+0000 7f75ed7fa640 1 -- 172.21.15.192:0/1811979606 <== mon.1 v1:172.21.15.192:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 2121038376 0 0) 0x7f75f01d9a50 con 0x7f75f013e6c0 2024-06-03T08:35:29.852 INFO:teuthology.orchestra.run.smithi192.stderr:2024-06-03T08:35:29.848+0000 7f75ed7fa640 10 monclient(hunting): _negotiate 2024-06-03T08:35:29.852 INFO:teuthology.orchestra.run.smithi192.stderr:2024-06-03T08:35:29.848+0000 7f75ed7fa640 10 monclient(hunting): _init_auth method 2 2024-06-03T08:35:29.852 INFO:teuthology.orchestra.run.smithi192.stderr:2024-06-03T08:35:29.848+0000 7f75ed7fa640 10 monclient(hunting): _init_auth creating new auth 2024-06-03T08:35:29.852 INFO:teuthology.orchestra.run.smithi192.stderr:2024-06-03T08:35:29.848+0000 7f75ed7fa640 10 monclient(hunting): my global_id is 4132 2024-06-03T08:35:29.852 INFO:teuthology.orchestra.run.smithi192.stderr:2024-06-03T08:35:29.848+0000 7f75ed7fa640 1 -- 172.21.15.192:0/1811979606 --> v1:172.21.15.192:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f75f0144b60 con 0x7f75f013e6c0 2024-06-03T08:35:29.853 INFO:teuthology.orchestra.run.smithi192.stderr:2024-06-03T08:35:29.852+0000 7f75ed7fa640 1 -- 172.21.15.192:0/1811979606 <== mon.2 v1:172.21.15.88:6790/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 764+0+0 (unknown 1860941772 0 0) 0x7f75d4004eb0 con 0x7f75f014ae20 2024-06-03T08:35:29.853 INFO:teuthology.orchestra.run.smithi192.stderr:2024-06-03T08:35:29.852+0000 7f75ed7fa640 1 -- 172.21.15.192:0/1811979606 --> v1:172.21.15.88:6790/0 -- auth(proto 2 165 bytes epoch 0) v1 -- 0x7f75f01d9a50 con 0x7f75f014ae20 2024-06-03T08:35:29.853 INFO:teuthology.orchestra.run.smithi192.stderr:2024-06-03T08:35:29.852+0000 7f75ed7fa640 1 -- 172.21.15.192:0/1811979606 <== mon.2 v1:172.21.15.88:6790/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 2234374065 0 0) 0x7f75e8001b30 con 0x7f75f014ae20 2024-06-03T08:35:29.853 INFO:teuthology.orchestra.run.smithi192.stderr:2024-06-03T08:35:29.852+0000 7f75ed7fa640 10 monclient(hunting): handle_monmap mon_map magic: 0 v1 2024-06-03T08:35:29.853 INFO:teuthology.orchestra.run.smithi192.stderr:2024-06-03T08:35:29.852+0000 7f75ed7fa640 10 monclient(hunting): got monmap 1 from mon.noname-c (according to old e1) 2024-06-03T08:35:29.853 INFO:teuthology.orchestra.run.smithi192.stderr:2024-06-03T08:35:29.852+0000 7f75ed7fa640 10 monclient(hunting): dump: 2024-06-03T08:35:29.853 INFO:teuthology.orchestra.run.smithi192.stderr:epoch 1 2024-06-03T08:35:29.853 INFO:teuthology.orchestra.run.smithi192.stderr:fsid ec3c3fdd-3ac6-4d7d-9412-683130ed8b04 2024-06-03T08:35:29.853 INFO:teuthology.orchestra.run.smithi192.stderr:last_changed 2024-06-03T08:34:31.885125+0000 2024-06-03T08:35:29.853 INFO:teuthology.orchestra.run.smithi192.stderr:created 2024-06-03T08:34:31.885125+0000 2024-06-03T08:35:29.853 INFO:teuthology.orchestra.run.smithi192.stderr:min_mon_release 18 (reef) 2024-06-03T08:35:29.853 INFO:teuthology.orchestra.run.smithi192.stderr:election_strategy: 3 2024-06-03T08:35:29.853 INFO:teuthology.orchestra.run.smithi192.stderr:0: v1:172.21.15.88:6789/0 mon.a 2024-06-03T08:35:29.854 INFO:teuthology.orchestra.run.smithi192.stderr:1: v1:172.21.15.192:6789/0 mon.b 2024-06-03T08:35:29.854 INFO:teuthology.orchestra.run.smithi192.stderr:2: v1:172.21.15.88:6790/0 mon.c 2024-06-03T08:35:29.854 INFO:teuthology.orchestra.run.smithi192.stderr: 2024-06-03T08:35:29.854 INFO:teuthology.orchestra.run.smithi192.stderr:2024-06-03T08:35:29.852+0000 7f75ed7fa640 10 monclient(hunting): _finish_auth 0 2024-06-03T08:35:29.854 INFO:teuthology.orchestra.run.smithi192.stderr:2024-06-03T08:35:29.853+0000 7f75ed7fa640 1 -- 172.21.15.192:0/1811979606 <== mon.0 v1:172.21.15.88:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 764+0+0 (unknown 936613274 0 0) 0x7f75f01da680 con 0x7f75f00b43c0 2024-06-03T08:35:29.855 INFO:teuthology.orchestra.run.smithi192.stderr:2024-06-03T08:35:29.853+0000 7f75ed7fa640 1 -- 172.21.15.192:0/1811979606 --> v1:172.21.15.88:6789/0 -- auth(proto 2 165 bytes epoch 0) v1 -- 0x7f75d4004eb0 con 0x7f75f00b43c0 2024-06-03T08:35:29.855 INFO:teuthology.orchestra.run.smithi192.stderr:2024-06-03T08:35:29.853+0000 7f75ed7fa640 1 -- 172.21.15.192:0/1811979606 <== mon.0 v1:172.21.15.88:6789/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 2234374065 0 0) 0x7f75e4001b50 con 0x7f75f00b43c0 2024-06-03T08:35:29.855 INFO:teuthology.orchestra.run.smithi192.stderr:2024-06-03T08:35:29.853+0000 7f75ed7fa640 10 monclient(hunting): handle_monmap mon_map magic: 0 v1 2024-06-03T08:35:29.855 INFO:teuthology.orchestra.run.smithi192.stderr:2024-06-03T08:35:29.853+0000 7f75ed7fa640 10 monclient(hunting): got monmap 1 from mon.a (according to old e1) 2024-06-03T08:35:29.855 INFO:teuthology.orchestra.run.smithi192.stderr:2024-06-03T08:35:29.853+0000 7f75ed7fa640 10 monclient(hunting): dump: 2024-06-03T08:35:29.855 INFO:teuthology.orchestra.run.smithi192.stderr:epoch 1 2024-06-03T08:35:29.855 INFO:teuthology.orchestra.run.smithi192.stderr:fsid ec3c3fdd-3ac6-4d7d-9412-683130ed8b04 2024-06-03T08:35:29.855 INFO:teuthology.orchestra.run.smithi192.stderr:last_changed 2024-06-03T08:34:31.885125+0000 2024-06-03T08:35:29.855 INFO:teuthology.orchestra.run.smithi192.stderr:created 2024-06-03T08:34:31.885125+0000 2024-06-03T08:35:29.855 INFO:teuthology.orchestra.run.smithi192.stderr:min_mon_release 18 (reef) 2024-06-03T08:35:29.855 INFO:teuthology.orchestra.run.smithi192.stderr:election_strategy: 3 2024-06-03T08:35:29.855 INFO:teuthology.orchestra.run.smithi192.stderr:0: v1:172.21.15.88:6789/0 mon.a 2024-06-03T08:35:29.855 INFO:teuthology.orchestra.run.smithi192.stderr:1: v1:172.21.15.192:6789/0 mon.b 2024-06-03T08:35:29.855 INFO:teuthology.orchestra.run.smithi192.stderr:2: v1:172.21.15.88:6790/0 mon.c 2024-06-03T08:35:29.856 INFO:teuthology.orchestra.run.smithi192.stderr: 2024-06-03T08:35:29.856 INFO:teuthology.orchestra.run.smithi192.stderr:2024-06-03T08:35:29.855+0000 7f75ed7fa640 1 -- 172.21.15.192:0/1811979606 <== mon.1 v1:172.21.15.192:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 764+0+0 (unknown 4294038669 0 0) 0x7f75f0144b60 con 0x7f75f013e6c0 2024-06-03T08:35:29.856 INFO:teuthology.orchestra.run.smithi192.stderr:2024-06-03T08:35:29.855+0000 7f75ed7fa640 1 -- 172.21.15.192:0/1811979606 --> v1:172.21.15.192:6789/0 -- auth(proto 2 165 bytes epoch 0) v1 -- 0x7f75f01da680 con 0x7f75f013e6c0 2024-06-03T08:35:29.856 INFO:teuthology.orchestra.run.smithi192.stderr:2024-06-03T08:35:29.855+0000 7f75ed7fa640 1 -- 172.21.15.192:0/1811979606 <== mon.1 v1:172.21.15.192:6789/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 2234374065 0 0) 0x7f75dc0018d0 con 0x7f75f013e6c0 2024-06-03T08:35:29.856 INFO:teuthology.orchestra.run.smithi192.stderr:2024-06-03T08:35:29.855+0000 7f75ed7fa640 10 monclient(hunting): handle_monmap mon_map magic: 0 v1 2024-06-03T08:35:29.857 INFO:teuthology.orchestra.run.smithi192.stderr:2024-06-03T08:35:29.855+0000 7f75ed7fa640 10 monclient(hunting): got monmap 1 from mon.b (according to old e1) 2024-06-03T08:35:29.857 INFO:teuthology.orchestra.run.smithi192.stderr:2024-06-03T08:35:29.855+0000 7f75ed7fa640 10 monclient(hunting): dump: 2024-06-03T08:35:29.857 INFO:teuthology.orchestra.run.smithi192.stderr:epoch 1 2024-06-03T08:35:29.857 INFO:teuthology.orchestra.run.smithi192.stderr:fsid ec3c3fdd-3ac6-4d7d-9412-683130ed8b04 2024-06-03T08:35:29.857 INFO:teuthology.orchestra.run.smithi192.stderr:last_changed 2024-06-03T08:34:31.885125+0000 2024-06-03T08:35:29.857 INFO:teuthology.orchestra.run.smithi192.stderr:created 2024-06-03T08:34:31.885125+0000 2024-06-03T08:35:29.857 INFO:teuthology.orchestra.run.smithi192.stderr:min_mon_release 18 (reef) 2024-06-03T08:35:29.857 INFO:teuthology.orchestra.run.smithi192.stderr:election_strategy: 3 2024-06-03T08:35:29.857 INFO:teuthology.orchestra.run.smithi192.stderr:0: v1:172.21.15.88:6789/0 mon.a 2024-06-03T08:35:29.857 INFO:teuthology.orchestra.run.smithi192.stderr:1: v1:172.21.15.192:6789/0 mon.b 2024-06-03T08:35:29.857 INFO:teuthology.orchestra.run.smithi192.stderr:2: v1:172.21.15.88:6790/0 mon.c 2024-06-03T08:35:29.857 INFO:teuthology.orchestra.run.smithi192.stderr: 2024-06-03T08:35:29.859 INFO:teuthology.orchestra.run.smithi192.stderr:2024-06-03T08:35:29.857+0000 7f75ed7fa640 1 -- 172.21.15.192:0/1811979606 <== mon.0 v1:172.21.15.88:6789/0 4 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 580+0+0 (unknown 2082218364 0 0) 0x7f75d4004eb0 con 0x7f75f00b43c0 2024-06-03T08:35:29.859 INFO:teuthology.orchestra.run.smithi192.stderr:2024-06-03T08:35:29.857+0000 7f75ed7fa640 1 -- 172.21.15.192:0/1811979606 >> v1:172.21.15.88:6790/0 conn(0x7f75f014ae20 legacy=0x7f75f01e1c50 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-06-03T08:35:29.859 INFO:teuthology.orchestra.run.smithi192.stderr:2024-06-03T08:35:29.857+0000 7f75ed7fa640 1 -- 172.21.15.192:0/1811979606 >> v1:172.21.15.192:6789/0 conn(0x7f75f013e6c0 legacy=0x7f75f01382c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-06-03T08:35:29.859 INFO:teuthology.orchestra.run.smithi192.stderr:2024-06-03T08:35:29.857+0000 7f75ed7fa640 10 monclient: _finish_hunting 0 2024-06-03T08:35:29.859 INFO:teuthology.orchestra.run.smithi192.stderr:2024-06-03T08:35:29.857+0000 7f75ed7fa640 1 monclient: found mon.a 2024-06-03T08:35:29.859 INFO:teuthology.orchestra.run.smithi192.stderr:2024-06-03T08:35:29.857+0000 7f75ed7fa640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-06-03T08:35:29.859 INFO:teuthology.orchestra.run.smithi192.stderr:2024-06-03T08:35:29.857+0000 7f75ed7fa640 10 monclient: _send_mon_message to mon.a at v1:172.21.15.88:6789/0 2024-06-03T08:35:29.859 INFO:teuthology.orchestra.run.smithi192.stderr:2024-06-03T08:35:29.857+0000 7f75ed7fa640 1 -- 172.21.15.192:0/1811979606 --> v1:172.21.15.88:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f75f01da810 con 0x7f75f00b43c0 2024-06-03T08:35:29.859 INFO:teuthology.orchestra.run.smithi192.stderr:2024-06-03T08:35:29.857+0000 7f75ed7fa640 10 monclient: _finish_auth 0 2024-06-03T08:35:29.859 INFO:teuthology.orchestra.run.smithi192.stderr:2024-06-03T08:35:29.857+0000 7f75ed7fa640 10 monclient: _check_auth_tickets 2024-06-03T08:35:29.859 INFO:teuthology.orchestra.run.smithi192.stderr:2024-06-03T08:35:29.857+0000 7f75ed7fa640 20 monclient: _check_auth_rotating not needed by client.admin 2024-06-03T08:35:29.859 INFO:teuthology.orchestra.run.smithi192.stderr:2024-06-03T08:35:29.857+0000 7f75f6daa640 5 monclient: authenticate success, global_id 4131 2024-06-03T08:35:29.860 INFO:teuthology.orchestra.run.smithi192.stderr:2024-06-03T08:35:29.857+0000 7f75f6daa640 10 monclient: _renew_subs 2024-06-03T08:35:29.860 INFO:teuthology.orchestra.run.smithi192.stderr:2024-06-03T08:35:29.857+0000 7f75f6daa640 10 monclient: _send_mon_message to mon.a at v1:172.21.15.88:6789/0 2024-06-03T08:35:29.860 INFO:teuthology.orchestra.run.smithi192.stderr:2024-06-03T08:35:29.857+0000 7f75f6daa640 1 -- 172.21.15.192:0/1811979606 --> v1:172.21.15.88:6789/0 -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f75f0142d10 con 0x7f75f00b43c0 2024-06-03T08:35:29.860 INFO:teuthology.orchestra.run.smithi192.stderr:2024-06-03T08:35:29.857+0000 7f75f6daa640 10 client.4131.objecter _maybe_request_map subscribing (onetime) to next osd map 2024-06-03T08:35:29.860 INFO:teuthology.orchestra.run.smithi192.stderr:2024-06-03T08:35:29.857+0000 7f75f6daa640 10 monclient: _renew_subs 2024-06-03T08:35:29.860 INFO:teuthology.orchestra.run.smithi192.stderr:2024-06-03T08:35:29.857+0000 7f75f6daa640 10 monclient: _send_mon_message to mon.a at v1:172.21.15.88:6789/0 2024-06-03T08:35:29.860 INFO:teuthology.orchestra.run.smithi192.stderr:2024-06-03T08:35:29.857+0000 7f75f6daa640 1 -- 172.21.15.192:0/1811979606 --> v1:172.21.15.88:6789/0 -- mon_subscribe({osdmap=0}) v3 -- 0x7f75f01458a0 con 0x7f75f00b43c0 2024-06-03T08:35:29.860 INFO:teuthology.orchestra.run.smithi192.stderr:2024-06-03T08:35:29.857+0000 7f75f6daa640 1 librados: init done 2024-06-03T08:35:29.860 INFO:teuthology.orchestra.run.smithi192.stderr:2024-06-03T08:35:29.858+0000 7f75f6daa640 10 start_mon_command cmd=[{"prefix": "get_command_descriptions"}] 2024-06-03T08:35:29.860 INFO:teuthology.orchestra.run.smithi192.stderr:2024-06-03T08:35:29.858+0000 7f75f6daa640 10 monclient: _send_command 1 [{"prefix": "get_command_descriptions"}] 2024-06-03T08:35:29.861 INFO:teuthology.orchestra.run.smithi192.stderr:2024-06-03T08:35:29.858+0000 7f75f6daa640 10 monclient: _send_mon_message to mon.a at v1:172.21.15.88:6789/0 2024-06-03T08:35:29.861 INFO:teuthology.orchestra.run.smithi192.stderr:2024-06-03T08:35:29.858+0000 7f75f6daa640 1 -- 172.21.15.192:0/1811979606 --> v1:172.21.15.88:6789/0 -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f75f004ea60 con 0x7f75f00b43c0 2024-06-03T08:35:29.861 INFO:teuthology.orchestra.run.smithi192.stderr:2024-06-03T08:35:29.861+0000 7f75ed7fa640 1 -- 172.21.15.192:0/1811979606 <== mon.0 v1:172.21.15.88:6789/0 5 ==== config(0 keys) v1 ==== 4+0+0 (unknown 0 0 0) 0x7f75e40011d0 con 0x7f75f00b43c0 2024-06-03T08:35:29.862 INFO:teuthology.orchestra.run.smithi192.stderr:2024-06-03T08:35:29.861+0000 7f75ed7fa640 10 monclient: handle_config config(0 keys) v1 2024-06-03T08:35:29.862 INFO:teuthology.orchestra.run.smithi192.stderr:2024-06-03T08:35:29.861+0000 7f75ed7fa640 1 -- 172.21.15.192:0/1811979606 <== mon.0 v1:172.21.15.88:6789/0 6 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 2234374065 0 0) 0x7f75e4004b10 con 0x7f75f00b43c0 2024-06-03T08:35:29.862 INFO:teuthology.orchestra.run.smithi192.stderr:2024-06-03T08:35:29.861+0000 7f75ed7fa640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-06-03T08:35:29.862 INFO:teuthology.orchestra.run.smithi192.stderr:2024-06-03T08:35:29.861+0000 7f75ed7fa640 10 monclient: got monmap 1 from mon.a (according to old e1) 2024-06-03T08:35:29.862 INFO:teuthology.orchestra.run.smithi192.stderr:2024-06-03T08:35:29.861+0000 7f75ed7fa640 10 monclient: dump: 2024-06-03T08:35:29.862 INFO:teuthology.orchestra.run.smithi192.stderr:epoch 1 2024-06-03T08:35:29.862 INFO:teuthology.orchestra.run.smithi192.stderr:fsid ec3c3fdd-3ac6-4d7d-9412-683130ed8b04 2024-06-03T08:35:29.862 INFO:teuthology.orchestra.run.smithi192.stderr:last_changed 2024-06-03T08:34:31.885125+0000 2024-06-03T08:35:29.862 INFO:teuthology.orchestra.run.smithi192.stderr:created 2024-06-03T08:34:31.885125+0000 2024-06-03T08:35:29.862 INFO:teuthology.orchestra.run.smithi192.stderr:min_mon_release 18 (reef) 2024-06-03T08:35:29.862 INFO:teuthology.orchestra.run.smithi192.stderr:election_strategy: 3 2024-06-03T08:35:29.862 INFO:teuthology.orchestra.run.smithi192.stderr:0: v1:172.21.15.88:6789/0 mon.a 2024-06-03T08:35:29.863 INFO:teuthology.orchestra.run.smithi192.stderr:1: v1:172.21.15.192:6789/0 mon.b 2024-06-03T08:35:29.863 INFO:teuthology.orchestra.run.smithi192.stderr:2: v1:172.21.15.88:6790/0 mon.c 2024-06-03T08:35:29.863 INFO:teuthology.orchestra.run.smithi192.stderr: 2024-06-03T08:35:29.863 INFO:teuthology.orchestra.run.smithi192.stderr:2024-06-03T08:35:29.861+0000 7f75ed7fa640 1 -- 172.21.15.192:0/1811979606 <== mon.0 v1:172.21.15.88:6789/0 7 ==== mgrmap(e 1) v1 ==== 668+0+0 (unknown 3836259515 0 0) 0x7f75e4004f70 con 0x7f75f00b43c0 2024-06-03T08:35:29.863 INFO:teuthology.orchestra.run.smithi192.stderr:2024-06-03T08:35:29.861+0000 7f75ed7fa640 1 -- 172.21.15.192:0/1811979606 <== mon.0 v1:172.21.15.88:6789/0 8 ==== osd_map(4..4 src has 1..4) v4 ==== 955+0+0 (unknown 990039770 0 0) 0x7f75e40062b0 con 0x7f75f00b43c0 2024-06-03T08:35:29.863 INFO:teuthology.orchestra.run.smithi192.stderr:2024-06-03T08:35:29.861+0000 7f75eeffd640 4 set_mon_vals no callback set 2024-06-03T08:35:29.863 INFO:teuthology.orchestra.run.smithi192.stderr:2024-06-03T08:35:29.861+0000 7f75ed7fa640 10 client.4131.objecter ms_dispatch 0x7f75f0000b90 osd_map(4..4 src has 1..4) v4 2024-06-03T08:35:29.863 INFO:teuthology.orchestra.run.smithi192.stderr:2024-06-03T08:35:29.861+0000 7f75ed7fa640 3 client.4131.objecter handle_osd_map got epochs [4,4] > 0 2024-06-03T08:35:29.863 INFO:teuthology.orchestra.run.smithi192.stderr:2024-06-03T08:35:29.861+0000 7f75ed7fa640 3 client.4131.objecter handle_osd_map decoding full epoch 4 2024-06-03T08:35:29.863 INFO:teuthology.orchestra.run.smithi192.stderr:2024-06-03T08:35:29.861+0000 7f75ed7fa640 20 client.4131.objecter dump_active .. 0 homeless 2024-06-03T08:35:29.864 INFO:tasks.ceph.mgr.x.smithi192.stderr:2024-06-03T08:35:29.862+0000 7f0e6be43040 -1 mgr[py] Module selftest has missing NOTIFY_TYPES member 2024-06-03T08:35:29.917 INFO:tasks.ceph.mgr.x.smithi192.stderr:2024-06-03T08:35:29.916+0000 7f0e6be43040 -1 mgr[py] Module pg_autoscaler has missing NOTIFY_TYPES member 2024-06-03T08:35:29.958 INFO:teuthology.orchestra.run.smithi192.stderr:2024-06-03T08:35:29.958+0000 7f75ed7fa640 1 -- 172.21.15.192:0/1811979606 <== mon.0 v1:172.21.15.88:6789/0 9 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+73121 (unknown 1092875540 0 1417653156) 0x7f75e4006650 con 0x7f75f00b43c0 2024-06-03T08:35:29.958 INFO:teuthology.orchestra.run.smithi192.stderr:2024-06-03T08:35:29.958+0000 7f75ed7fa640 10 monclient: handle_mon_command_ack 1 [{"prefix": "get_command_descriptions"}] 2024-06-03T08:35:29.958 INFO:teuthology.orchestra.run.smithi192.stderr:2024-06-03T08:35:29.958+0000 7f75ed7fa640 10 monclient: _finish_command 1 = system:0 2024-06-03T08:35:29.965 INFO:tasks.ceph.mgr.y.smithi088.stderr:2024-06-03T08:35:29.965+0000 7ff0aea1d040 -1 mgr[py] Module alerts has missing NOTIFY_TYPES member 2024-06-03T08:35:30.015 INFO:tasks.ceph.mgr.y.smithi088.stderr:2024-06-03T08:35:30.015+0000 7ff0aea1d040 -1 mgr[py] Module selftest has missing NOTIFY_TYPES member 2024-06-03T08:35:30.031 INFO:teuthology.orchestra.run.smithi192.stderr:2024-06-03T08:35:30.030+0000 7f75f6daa640 10 start_mon_command cmd=[{"prefix": "osd new", "uuid": "addfb2a7-2bde-4cc3-98cf-17fd2bdfa3a9", "id": 2}] 2024-06-03T08:35:30.031 INFO:teuthology.orchestra.run.smithi192.stderr:2024-06-03T08:35:30.030+0000 7f75f6daa640 10 monclient: _send_command 2 [{"prefix": "osd new", "uuid": "addfb2a7-2bde-4cc3-98cf-17fd2bdfa3a9", "id": 2}] 2024-06-03T08:35:30.031 INFO:teuthology.orchestra.run.smithi192.stderr:2024-06-03T08:35:30.030+0000 7f75f6daa640 10 monclient: _send_mon_message to mon.a at v1:172.21.15.88:6789/0 2024-06-03T08:35:30.031 INFO:teuthology.orchestra.run.smithi192.stderr:2024-06-03T08:35:30.030+0000 7f75f6daa640 1 -- 172.21.15.192:0/1811979606 --> v1:172.21.15.88:6789/0 -- mon_command({"prefix": "osd new", "uuid": "addfb2a7-2bde-4cc3-98cf-17fd2bdfa3a9", "id": 2} v 0) v1 -- 0x7f75f0142f00 con 0x7f75f00b43c0 2024-06-03T08:35:30.034 INFO:tasks.ceph.mgr.x.smithi192.stderr:2024-06-03T08:35:30.034+0000 7f0e6be43040 -1 mgr[py] Module test_orchestrator has missing NOTIFY_TYPES member 2024-06-03T08:35:30.073 INFO:tasks.ceph.mgr.y.smithi088.stderr:2024-06-03T08:35:30.072+0000 7ff0aea1d040 -1 mgr[py] Module pg_autoscaler has missing NOTIFY_TYPES member 2024-06-03T08:35:30.078 INFO:teuthology.orchestra.run.smithi192.stderr:2024-06-03T08:35:30.077+0000 7f75ed7fa640 1 -- 172.21.15.192:0/1811979606 <== mon.0 v1:172.21.15.88:6789/0 10 ==== mon_command_ack([{"prefix": "osd new", "uuid": "addfb2a7-2bde-4cc3-98cf-17fd2bdfa3a9", "id": 2}]=0 v5) v1 ==== 112+0+2 (unknown 894196225 0 216599125) 0x7f75e4005f70 con 0x7f75f00b43c0 2024-06-03T08:35:30.078 INFO:teuthology.orchestra.run.smithi192.stderr:2024-06-03T08:35:30.077+0000 7f75ed7fa640 10 monclient: handle_mon_command_ack 2 [{"prefix": "osd new", "uuid": "addfb2a7-2bde-4cc3-98cf-17fd2bdfa3a9", "id": 2}] 2024-06-03T08:35:30.078 INFO:teuthology.orchestra.run.smithi192.stderr:2024-06-03T08:35:30.077+0000 7f75ed7fa640 10 monclient: _finish_command 2 = system:0 2024-06-03T08:35:30.079 INFO:teuthology.orchestra.run.smithi192.stdout:2 2024-06-03T08:35:30.081 INFO:teuthology.orchestra.run.smithi192.stderr:2024-06-03T08:35:30.078+0000 7f75f6daa640 10 librados: watch_flush enter 2024-06-03T08:35:30.081 INFO:teuthology.orchestra.run.smithi192.stderr:2024-06-03T08:35:30.078+0000 7f75f6daa640 10 librados: watch_flush exit 2024-06-03T08:35:30.081 INFO:teuthology.orchestra.run.smithi192.stderr:2024-06-03T08:35:30.078+0000 7f75f6daa640 20 client.4131.objecter shutdown clearing up homeless session... 2024-06-03T08:35:30.081 INFO:teuthology.orchestra.run.smithi192.stderr:2024-06-03T08:35:30.078+0000 7f75f6daa640 10 client.4131.objecter successfully canceled tick 2024-06-03T08:35:30.082 INFO:teuthology.orchestra.run.smithi192.stderr:2024-06-03T08:35:30.078+0000 7f75f6daa640 10 monclient: shutdown 2024-06-03T08:35:30.082 INFO:teuthology.orchestra.run.smithi192.stderr:2024-06-03T08:35:30.078+0000 7f75f6daa640 20 monclient: shutdown discarding 0 pending message(s) 2024-06-03T08:35:30.082 INFO:teuthology.orchestra.run.smithi192.stderr:2024-06-03T08:35:30.078+0000 7f75f6daa640 1 -- 172.21.15.192:0/1811979606 >> v1:172.21.15.88:6789/0 conn(0x7f75f00b43c0 legacy=0x7f75f01d9f70 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-06-03T08:35:30.082 INFO:teuthology.orchestra.run.smithi192.stderr:2024-06-03T08:35:30.078+0000 7f75f6daa640 1 -- 172.21.15.192:0/1811979606 shutdown_connections 2024-06-03T08:35:30.082 INFO:teuthology.orchestra.run.smithi192.stderr:2024-06-03T08:35:30.078+0000 7f75f6daa640 1 -- 172.21.15.192:0/1811979606 >> 172.21.15.192:0/1811979606 conn(0x7f75f00719d0 msgr2=0x7f75f0071db0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-06-03T08:35:30.082 INFO:teuthology.orchestra.run.smithi192.stderr:2024-06-03T08:35:30.078+0000 7f75f6daa640 1 -- 172.21.15.192:0/1811979606 shutdown_connections 2024-06-03T08:35:30.083 INFO:teuthology.orchestra.run.smithi192.stderr:2024-06-03T08:35:30.078+0000 7f75f6daa640 1 -- 172.21.15.192:0/1811979606 wait complete. 2024-06-03T08:35:30.083 INFO:teuthology.orchestra.run.smithi192.stderr:2024-06-03T08:35:30.078+0000 7f75f6daa640 1 librados: shutdown 2024-06-03T08:35:30.089 DEBUG:teuthology.orchestra.run.smithi192:> sudo ceph --cluster ceph osd new 6c5770b0-c93c-4101-8c53-bd761b8abcde 3 2024-06-03T08:35:30.188 INFO:tasks.ceph.mgr.y.smithi088.stderr:2024-06-03T08:35:30.187+0000 7ff0aea1d040 -1 mgr[py] Module test_orchestrator has missing NOTIFY_TYPES member 2024-06-03T08:35:30.188 INFO:teuthology.orchestra.run.smithi192.stderr:2024-06-03T08:35:30.187+0000 7f34c9a45640 10 monclient: get_monmap_and_config 2024-06-03T08:35:30.188 INFO:teuthology.orchestra.run.smithi192.stderr:2024-06-03T08:35:30.187+0000 7f34c9a45640 10 monclient: build_initial_monmap 2024-06-03T08:35:30.188 INFO:teuthology.orchestra.run.smithi192.stderr:2024-06-03T08:35:30.188+0000 7f34c9a45640 10 monclient: monmap: 2024-06-03T08:35:30.189 INFO:teuthology.orchestra.run.smithi192.stderr:epoch 0 2024-06-03T08:35:30.189 INFO:teuthology.orchestra.run.smithi192.stderr:fsid ec3c3fdd-3ac6-4d7d-9412-683130ed8b04 2024-06-03T08:35:30.189 INFO:teuthology.orchestra.run.smithi192.stderr:last_changed 2024-06-03T08:35:30.188873+0000 2024-06-03T08:35:30.189 INFO:teuthology.orchestra.run.smithi192.stderr:created 2024-06-03T08:35:30.188873+0000 2024-06-03T08:35:30.189 INFO:teuthology.orchestra.run.smithi192.stderr:min_mon_release 0 (unknown) 2024-06-03T08:35:30.189 INFO:teuthology.orchestra.run.smithi192.stderr:election_strategy: 3 2024-06-03T08:35:30.189 INFO:teuthology.orchestra.run.smithi192.stderr:0: v1:172.21.15.88:6789/0 mon.noname-a 2024-06-03T08:35:30.189 INFO:teuthology.orchestra.run.smithi192.stderr:1: v1:172.21.15.192:6789/0 mon.noname-c 2024-06-03T08:35:30.189 INFO:teuthology.orchestra.run.smithi192.stderr:2: v1:172.21.15.88:6790/0 mon.noname-b 2024-06-03T08:35:30.189 INFO:teuthology.orchestra.run.smithi192.stderr: 2024-06-03T08:35:30.189 INFO:teuthology.orchestra.run.smithi192.stderr:2024-06-03T08:35:30.188+0000 7f34c9a45640 1 Processor -- start 2024-06-03T08:35:30.189 INFO:teuthology.orchestra.run.smithi192.stderr:2024-06-03T08:35:30.188+0000 7f34c9a45640 1 -- start start 2024-06-03T08:35:30.189 INFO:teuthology.orchestra.run.smithi192.stderr:2024-06-03T08:35:30.188+0000 7f34c9a45640 10 monclient: init 2024-06-03T08:35:30.191 INFO:teuthology.orchestra.run.smithi192.stderr:2024-06-03T08:35:30.189+0000 7f34c9a45640 10 monclient: _reopen_session rank -1 2024-06-03T08:35:30.191 INFO:teuthology.orchestra.run.smithi192.stderr:2024-06-03T08:35:30.189+0000 7f34c9a45640 10 monclient: _add_conns ranks=[2,0,1] 2024-06-03T08:35:30.191 INFO:teuthology.orchestra.run.smithi192.stderr:2024-06-03T08:35:30.189+0000 7f34c9a45640 10 monclient(hunting): picked mon.noname-b con 0x7f34c40a8960 addr v1:172.21.15.88:6790/0 2024-06-03T08:35:30.191 INFO:teuthology.orchestra.run.smithi192.stderr:2024-06-03T08:35:30.189+0000 7f34c9a45640 10 monclient(hunting): picked mon.noname-a con 0x7f34c41450f0 addr v1:172.21.15.88:6789/0 2024-06-03T08:35:30.191 INFO:teuthology.orchestra.run.smithi192.stderr:2024-06-03T08:35:30.189+0000 7f34c9a45640 10 monclient(hunting): picked mon.noname-c con 0x7f34c413eec0 addr v1:172.21.15.192:6789/0 2024-06-03T08:35:30.191 INFO:teuthology.orchestra.run.smithi192.stderr:2024-06-03T08:35:30.189+0000 7f34c9a45640 1 -- --> v1:172.21.15.88:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f34c4144c60 con 0x7f34c41450f0 2024-06-03T08:35:30.191 INFO:teuthology.orchestra.run.smithi192.stderr:2024-06-03T08:35:30.189+0000 7f34c9a45640 1 -- --> v1:172.21.15.192:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f34c40b3930 con 0x7f34c413eec0 2024-06-03T08:35:30.191 INFO:teuthology.orchestra.run.smithi192.stderr:2024-06-03T08:35:30.189+0000 7f34c9a45640 1 -- --> v1:172.21.15.88:6790/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f34c40b1ad0 con 0x7f34c40a8960 2024-06-03T08:35:30.191 INFO:teuthology.orchestra.run.smithi192.stderr:2024-06-03T08:35:30.189+0000 7f34c9a45640 10 monclient(hunting): _renew_subs 2024-06-03T08:35:30.191 INFO:teuthology.orchestra.run.smithi192.stderr:2024-06-03T08:35:30.189+0000 7f34c9a45640 10 monclient(hunting): authenticate will time out at 1196.000368s 2024-06-03T08:35:30.192 INFO:teuthology.orchestra.run.smithi192.stderr:2024-06-03T08:35:30.189+0000 7f34c37fe640 1 --1- >> v1:172.21.15.192:6789/0 conn(0x7f34c413eec0 0x7f34c414ce60 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.192:6789/0 says I am v1:172.21.15.192:43868/0 (socket says 172.21.15.192:43868) 2024-06-03T08:35:30.192 INFO:teuthology.orchestra.run.smithi192.stderr:2024-06-03T08:35:30.189+0000 7f34c37fe640 1 -- 172.21.15.192:0/2995183643 learned_addr learned my addr 172.21.15.192:0/2995183643 (peer_addr_for_me v1:172.21.15.192:0/0) 2024-06-03T08:35:30.193 INFO:teuthology.orchestra.run.smithi192.stderr:2024-06-03T08:35:30.192+0000 7f34c1ffb640 1 -- 172.21.15.192:0/2995183643 <== mon.2 v1:172.21.15.88:6790/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 2404204027 0 0) 0x7f34c40b1ad0 con 0x7f34c40a8960 2024-06-03T08:35:30.193 INFO:teuthology.orchestra.run.smithi192.stderr:2024-06-03T08:35:30.192+0000 7f34c1ffb640 10 monclient(hunting): _negotiate 2024-06-03T08:35:30.193 INFO:teuthology.orchestra.run.smithi192.stderr:2024-06-03T08:35:30.192+0000 7f34c1ffb640 10 monclient(hunting): _init_auth method 2 2024-06-03T08:35:30.193 INFO:teuthology.orchestra.run.smithi192.stderr:2024-06-03T08:35:30.192+0000 7f34c1ffb640 10 monclient(hunting): _init_auth creating new auth 2024-06-03T08:35:30.193 INFO:teuthology.orchestra.run.smithi192.stderr:2024-06-03T08:35:30.192+0000 7f34c1ffb640 10 monclient(hunting): my global_id is 4136 2024-06-03T08:35:30.193 INFO:teuthology.orchestra.run.smithi192.stderr:2024-06-03T08:35:30.192+0000 7f34c1ffb640 1 -- 172.21.15.192:0/2995183643 --> v1:172.21.15.88:6790/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f34ac003520 con 0x7f34c40a8960 2024-06-03T08:35:30.193 INFO:teuthology.orchestra.run.smithi192.stderr:2024-06-03T08:35:30.192+0000 7f34c1ffb640 1 -- 172.21.15.192:0/2995183643 <== mon.1 v1:172.21.15.192:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 3467831664 0 0) 0x7f34c40b3930 con 0x7f34c413eec0 2024-06-03T08:35:30.193 INFO:teuthology.orchestra.run.smithi192.stderr:2024-06-03T08:35:30.192+0000 7f34c1ffb640 10 monclient(hunting): _negotiate 2024-06-03T08:35:30.193 INFO:teuthology.orchestra.run.smithi192.stderr:2024-06-03T08:35:30.192+0000 7f34c1ffb640 10 monclient(hunting): _init_auth method 2 2024-06-03T08:35:30.193 INFO:teuthology.orchestra.run.smithi192.stderr:2024-06-03T08:35:30.192+0000 7f34c1ffb640 10 monclient(hunting): _init_auth creating new auth 2024-06-03T08:35:30.193 INFO:teuthology.orchestra.run.smithi192.stderr:2024-06-03T08:35:30.192+0000 7f34c1ffb640 10 monclient(hunting): my global_id is 4135 2024-06-03T08:35:30.193 INFO:teuthology.orchestra.run.smithi192.stderr:2024-06-03T08:35:30.192+0000 7f34c1ffb640 1 -- 172.21.15.192:0/2995183643 --> v1:172.21.15.192:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f34c40b1ad0 con 0x7f34c413eec0 2024-06-03T08:35:30.193 INFO:teuthology.orchestra.run.smithi192.stderr:2024-06-03T08:35:30.192+0000 7f34c1ffb640 1 -- 172.21.15.192:0/2995183643 <== mon.0 v1:172.21.15.88:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 142774149 0 0) 0x7f34c4144c60 con 0x7f34c41450f0 2024-06-03T08:35:30.193 INFO:teuthology.orchestra.run.smithi192.stderr:2024-06-03T08:35:30.192+0000 7f34c1ffb640 10 monclient(hunting): _negotiate 2024-06-03T08:35:30.194 INFO:teuthology.orchestra.run.smithi192.stderr:2024-06-03T08:35:30.192+0000 7f34c1ffb640 10 monclient(hunting): _init_auth method 2 2024-06-03T08:35:30.194 INFO:teuthology.orchestra.run.smithi192.stderr:2024-06-03T08:35:30.192+0000 7f34c1ffb640 10 monclient(hunting): _init_auth creating new auth 2024-06-03T08:35:30.194 INFO:teuthology.orchestra.run.smithi192.stderr:2024-06-03T08:35:30.192+0000 7f34c1ffb640 10 monclient(hunting): my global_id is 4134 2024-06-03T08:35:30.194 INFO:teuthology.orchestra.run.smithi192.stderr:2024-06-03T08:35:30.192+0000 7f34c1ffb640 1 -- 172.21.15.192:0/2995183643 --> v1:172.21.15.88:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f34c40b3930 con 0x7f34c41450f0 2024-06-03T08:35:30.196 INFO:teuthology.orchestra.run.smithi192.stderr:2024-06-03T08:35:30.195+0000 7f34c1ffb640 1 -- 172.21.15.192:0/2995183643 <== mon.2 v1:172.21.15.88:6790/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 214+0+0 (unknown 2196291995 0 0) 0x7f34ac003520 con 0x7f34c40a8960 2024-06-03T08:35:30.196 INFO:teuthology.orchestra.run.smithi192.stderr:2024-06-03T08:35:30.195+0000 7f34c1ffb640 1 -- 172.21.15.192:0/2995183643 >> v1:172.21.15.192:6789/0 conn(0x7f34c413eec0 legacy=0x7f34c414ce60 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-06-03T08:35:30.196 INFO:teuthology.orchestra.run.smithi192.stderr:2024-06-03T08:35:30.195+0000 7f34c1ffb640 1 -- 172.21.15.192:0/2995183643 >> v1:172.21.15.88:6789/0 conn(0x7f34c41450f0 legacy=0x7f34c413e7b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-06-03T08:35:30.196 INFO:teuthology.orchestra.run.smithi192.stderr:2024-06-03T08:35:30.195+0000 7f34c1ffb640 10 monclient: _finish_hunting 0 2024-06-03T08:35:30.196 INFO:teuthology.orchestra.run.smithi192.stderr:2024-06-03T08:35:30.195+0000 7f34c1ffb640 1 monclient: found mon.noname-b 2024-06-03T08:35:30.196 INFO:teuthology.orchestra.run.smithi192.stderr:2024-06-03T08:35:30.195+0000 7f34c1ffb640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-06-03T08:35:30.196 INFO:teuthology.orchestra.run.smithi192.stderr:2024-06-03T08:35:30.195+0000 7f34c1ffb640 10 monclient: _send_mon_message to mon.noname-b at v1:172.21.15.88:6790/0 2024-06-03T08:35:30.197 INFO:teuthology.orchestra.run.smithi192.stderr:2024-06-03T08:35:30.195+0000 7f34c1ffb640 1 -- 172.21.15.192:0/2995183643 --> v1:172.21.15.88:6790/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f34c40b1c60 con 0x7f34c40a8960 2024-06-03T08:35:30.197 INFO:teuthology.orchestra.run.smithi192.stderr:2024-06-03T08:35:30.195+0000 7f34c1ffb640 10 monclient: _finish_auth 0 2024-06-03T08:35:30.197 INFO:teuthology.orchestra.run.smithi192.stderr:2024-06-03T08:35:30.195+0000 7f34c1ffb640 10 monclient: _check_auth_tickets 2024-06-03T08:35:30.197 INFO:teuthology.orchestra.run.smithi192.stderr:2024-06-03T08:35:30.195+0000 7f34c1ffb640 20 monclient: _check_auth_rotating not needed by client.admin 2024-06-03T08:35:30.197 INFO:teuthology.orchestra.run.smithi192.stderr:2024-06-03T08:35:30.195+0000 7f34c1ffb640 1 -- 172.21.15.192:0/2995183643 <== mon.2 v1:172.21.15.88:6790/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 2234374065 0 0) 0x7f34b4001540 con 0x7f34c40a8960 2024-06-03T08:35:30.197 INFO:teuthology.orchestra.run.smithi192.stderr:2024-06-03T08:35:30.195+0000 7f34c1ffb640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-06-03T08:35:30.197 INFO:teuthology.orchestra.run.smithi192.stderr:2024-06-03T08:35:30.195+0000 7f34c1ffb640 10 monclient: got monmap 1 from mon.noname-b (according to old e1) 2024-06-03T08:35:30.197 INFO:teuthology.orchestra.run.smithi192.stderr:2024-06-03T08:35:30.195+0000 7f34c1ffb640 10 monclient: dump: 2024-06-03T08:35:30.197 INFO:teuthology.orchestra.run.smithi192.stderr:epoch 1 2024-06-03T08:35:30.197 INFO:teuthology.orchestra.run.smithi192.stderr:fsid ec3c3fdd-3ac6-4d7d-9412-683130ed8b04 2024-06-03T08:35:30.197 INFO:teuthology.orchestra.run.smithi192.stderr:last_changed 2024-06-03T08:34:31.885125+0000 2024-06-03T08:35:30.197 INFO:teuthology.orchestra.run.smithi192.stderr:created 2024-06-03T08:34:31.885125+0000 2024-06-03T08:35:30.197 INFO:teuthology.orchestra.run.smithi192.stderr:min_mon_release 18 (reef) 2024-06-03T08:35:30.197 INFO:teuthology.orchestra.run.smithi192.stderr:election_strategy: 3 2024-06-03T08:35:30.198 INFO:teuthology.orchestra.run.smithi192.stderr:0: v1:172.21.15.88:6789/0 mon.a 2024-06-03T08:35:30.198 INFO:teuthology.orchestra.run.smithi192.stderr:1: v1:172.21.15.192:6789/0 mon.b 2024-06-03T08:35:30.198 INFO:teuthology.orchestra.run.smithi192.stderr:2: v1:172.21.15.88:6790/0 mon.c 2024-06-03T08:35:30.198 INFO:teuthology.orchestra.run.smithi192.stderr: 2024-06-03T08:35:30.198 INFO:teuthology.orchestra.run.smithi192.stderr:2024-06-03T08:35:30.195+0000 7f34c9a45640 5 monclient: authenticate success, global_id 4136 2024-06-03T08:35:30.198 INFO:teuthology.orchestra.run.smithi192.stderr:2024-06-03T08:35:30.195+0000 7f34c9a45640 20 monclient: get_monmap_and_config waiting for monmap|config 2024-06-03T08:35:30.205 INFO:teuthology.orchestra.run.smithi192.stderr:2024-06-03T08:35:30.198+0000 7f34c1ffb640 1 -- 172.21.15.192:0/2995183643 <== mon.2 v1:172.21.15.88:6790/0 4 ==== config(0 keys) v1 ==== 4+0+0 (unknown 0 0 0) 0x7f34b4001760 con 0x7f34c40a8960 2024-06-03T08:35:30.206 INFO:teuthology.orchestra.run.smithi192.stderr:2024-06-03T08:35:30.198+0000 7f34c1ffb640 10 monclient: handle_config config(0 keys) v1 2024-06-03T08:35:30.206 INFO:teuthology.orchestra.run.smithi192.stderr:2024-06-03T08:35:30.198+0000 7f34c1ffb640 1 -- 172.21.15.192:0/2995183643 <== mon.2 v1:172.21.15.88:6790/0 5 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 2234374065 0 0) 0x7f34b4001a70 con 0x7f34c40a8960 2024-06-03T08:35:30.206 INFO:teuthology.orchestra.run.smithi192.stderr:2024-06-03T08:35:30.198+0000 7f34c9a45640 10 monclient: get_monmap_and_config success 2024-06-03T08:35:30.206 INFO:teuthology.orchestra.run.smithi192.stderr:2024-06-03T08:35:30.198+0000 7f34c9a45640 4 set_mon_vals no callback set 2024-06-03T08:35:30.206 INFO:teuthology.orchestra.run.smithi192.stderr:2024-06-03T08:35:30.198+0000 7f34c1ffb640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-06-03T08:35:30.206 INFO:teuthology.orchestra.run.smithi192.stderr:2024-06-03T08:35:30.198+0000 7f34c1ffb640 10 monclient: got monmap 1 from mon.c (according to old e1) 2024-06-03T08:35:30.206 INFO:teuthology.orchestra.run.smithi192.stderr:2024-06-03T08:35:30.198+0000 7f34c1ffb640 10 monclient: dump: 2024-06-03T08:35:30.207 INFO:teuthology.orchestra.run.smithi192.stderr:epoch 1 2024-06-03T08:35:30.207 INFO:teuthology.orchestra.run.smithi192.stderr:fsid ec3c3fdd-3ac6-4d7d-9412-683130ed8b04 2024-06-03T08:35:30.207 INFO:teuthology.orchestra.run.smithi192.stderr:last_changed 2024-06-03T08:34:31.885125+0000 2024-06-03T08:35:30.207 INFO:teuthology.orchestra.run.smithi192.stderr:created 2024-06-03T08:34:31.885125+0000 2024-06-03T08:35:30.207 INFO:teuthology.orchestra.run.smithi192.stderr:min_mon_release 18 (reef) 2024-06-03T08:35:30.207 INFO:teuthology.orchestra.run.smithi192.stderr:election_strategy: 3 2024-06-03T08:35:30.207 INFO:teuthology.orchestra.run.smithi192.stderr:0: v1:172.21.15.88:6789/0 mon.a 2024-06-03T08:35:30.207 INFO:teuthology.orchestra.run.smithi192.stderr:1: v1:172.21.15.192:6789/0 mon.b 2024-06-03T08:35:30.207 INFO:teuthology.orchestra.run.smithi192.stderr:2: v1:172.21.15.88:6790/0 mon.c 2024-06-03T08:35:30.207 INFO:teuthology.orchestra.run.smithi192.stderr: 2024-06-03T08:35:30.207 INFO:teuthology.orchestra.run.smithi192.stderr:2024-06-03T08:35:30.198+0000 7f34c9a45640 10 monclient: shutdown 2024-06-03T08:35:30.207 INFO:teuthology.orchestra.run.smithi192.stderr:2024-06-03T08:35:30.198+0000 7f34c9a45640 20 monclient: shutdown discarding 0 pending message(s) 2024-06-03T08:35:30.207 INFO:teuthology.orchestra.run.smithi192.stderr:2024-06-03T08:35:30.198+0000 7f34c9a45640 1 -- 172.21.15.192:0/2995183643 >> v1:172.21.15.88:6790/0 conn(0x7f34c40a8960 legacy=0x7f34c40a8d40 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-06-03T08:35:30.208 INFO:teuthology.orchestra.run.smithi192.stderr:2024-06-03T08:35:30.198+0000 7f34c9a45640 1 -- 172.21.15.192:0/2995183643 shutdown_connections 2024-06-03T08:35:30.208 INFO:teuthology.orchestra.run.smithi192.stderr:2024-06-03T08:35:30.198+0000 7f34c9a45640 1 -- 172.21.15.192:0/2995183643 >> 172.21.15.192:0/2995183643 conn(0x7f34c40b70d0 msgr2=0x7f34c40b74e0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-06-03T08:35:30.208 INFO:teuthology.orchestra.run.smithi192.stderr:2024-06-03T08:35:30.199+0000 7f34c9a45640 1 -- 172.21.15.192:0/2995183643 shutdown_connections 2024-06-03T08:35:30.208 INFO:teuthology.orchestra.run.smithi192.stderr:2024-06-03T08:35:30.199+0000 7f34c9a45640 1 -- 172.21.15.192:0/2995183643 wait complete. 2024-06-03T08:35:30.208 INFO:teuthology.orchestra.run.smithi192.stderr:2024-06-03T08:35:30.199+0000 7f34c9a45640 10 monclient: build_initial_monmap 2024-06-03T08:35:30.208 INFO:teuthology.orchestra.run.smithi192.stderr:2024-06-03T08:35:30.199+0000 7f34c9a45640 10 monclient: monmap: 2024-06-03T08:35:30.208 INFO:teuthology.orchestra.run.smithi192.stderr:epoch 0 2024-06-03T08:35:30.208 INFO:teuthology.orchestra.run.smithi192.stderr:fsid 00000000-0000-0000-0000-000000000000 2024-06-03T08:35:30.208 INFO:teuthology.orchestra.run.smithi192.stderr:last_changed 0.000000 2024-06-03T08:35:30.208 INFO:teuthology.orchestra.run.smithi192.stderr:created 0.000000 2024-06-03T08:35:30.208 INFO:teuthology.orchestra.run.smithi192.stderr:min_mon_release 0 (unknown) 2024-06-03T08:35:30.208 INFO:teuthology.orchestra.run.smithi192.stderr:election_strategy: 1 2024-06-03T08:35:30.208 INFO:teuthology.orchestra.run.smithi192.stderr:0: v1:172.21.15.88:6789/0 mon.noname-a 2024-06-03T08:35:30.208 INFO:teuthology.orchestra.run.smithi192.stderr:1: v1:172.21.15.192:6789/0 mon.noname-b 2024-06-03T08:35:30.209 INFO:teuthology.orchestra.run.smithi192.stderr:2: v1:172.21.15.88:6790/0 mon.noname-c 2024-06-03T08:35:30.209 INFO:teuthology.orchestra.run.smithi192.stderr: 2024-06-03T08:35:30.209 INFO:teuthology.orchestra.run.smithi192.stderr:2024-06-03T08:35:30.199+0000 7f34c9a45640 1 librados: starting msgr at 2024-06-03T08:35:30.209 INFO:teuthology.orchestra.run.smithi192.stderr:2024-06-03T08:35:30.199+0000 7f34c9a45640 1 librados: starting objecter 2024-06-03T08:35:30.209 INFO:teuthology.orchestra.run.smithi192.stderr:2024-06-03T08:35:30.200+0000 7f34c9a45640 1 Processor -- start 2024-06-03T08:35:30.209 INFO:teuthology.orchestra.run.smithi192.stderr:2024-06-03T08:35:30.200+0000 7f34c9a45640 1 -- start start 2024-06-03T08:35:30.209 INFO:teuthology.orchestra.run.smithi192.stderr:2024-06-03T08:35:30.200+0000 7f34c9a45640 1 librados: setting wanted keys 2024-06-03T08:35:30.209 INFO:teuthology.orchestra.run.smithi192.stderr:2024-06-03T08:35:30.200+0000 7f34c9a45640 1 librados: calling monclient init 2024-06-03T08:35:30.209 INFO:teuthology.orchestra.run.smithi192.stderr:2024-06-03T08:35:30.200+0000 7f34c9a45640 10 monclient: init 2024-06-03T08:35:30.209 INFO:teuthology.orchestra.run.smithi192.stderr:2024-06-03T08:35:30.200+0000 7f34c9a45640 10 monclient: _reopen_session rank -1 2024-06-03T08:35:30.209 INFO:teuthology.orchestra.run.smithi192.stderr:2024-06-03T08:35:30.200+0000 7f34c9a45640 10 monclient: _add_conns ranks=[1,0,2] 2024-06-03T08:35:30.209 INFO:teuthology.orchestra.run.smithi192.stderr:2024-06-03T08:35:30.200+0000 7f34c9a45640 10 monclient(hunting): picked mon.noname-b con 0x7f34c40a8960 addr v1:172.21.15.192:6789/0 2024-06-03T08:35:30.209 INFO:teuthology.orchestra.run.smithi192.stderr:2024-06-03T08:35:30.200+0000 7f34c9a45640 10 monclient(hunting): picked mon.noname-a con 0x7f34c413eec0 addr v1:172.21.15.88:6789/0 2024-06-03T08:35:30.209 INFO:teuthology.orchestra.run.smithi192.stderr:2024-06-03T08:35:30.200+0000 7f34c9a45640 10 monclient(hunting): picked mon.noname-c con 0x7f34c41450f0 addr v1:172.21.15.88:6790/0 2024-06-03T08:35:30.210 INFO:teuthology.orchestra.run.smithi192.stderr:2024-06-03T08:35:30.200+0000 7f34c9a45640 1 -- --> v1:172.21.15.88:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f34c4143c60 con 0x7f34c413eec0 2024-06-03T08:35:30.210 INFO:teuthology.orchestra.run.smithi192.stderr:2024-06-03T08:35:30.200+0000 7f34c9a45640 1 -- --> v1:172.21.15.192:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f34c4142710 con 0x7f34c40a8960 2024-06-03T08:35:30.210 INFO:teuthology.orchestra.run.smithi192.stderr:2024-06-03T08:35:30.200+0000 7f34c9a45640 1 -- --> v1:172.21.15.88:6790/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f34c41d6820 con 0x7f34c41450f0 2024-06-03T08:35:30.210 INFO:teuthology.orchestra.run.smithi192.stderr:2024-06-03T08:35:30.200+0000 7f34c9a45640 10 monclient(hunting): _renew_subs 2024-06-03T08:35:30.210 INFO:teuthology.orchestra.run.smithi192.stderr:2024-06-03T08:35:30.200+0000 7f34c9a45640 10 monclient(hunting): authenticate will time out at 1196.011939s 2024-06-03T08:35:30.210 INFO:teuthology.orchestra.run.smithi192.stderr:2024-06-03T08:35:30.201+0000 7f34c2ffd640 1 --1- >> v1:172.21.15.192:6789/0 conn(0x7f34c40a8960 0x7f34c4143550 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=1).handle_server_banner_and_identify peer v1:172.21.15.192:6789/0 says I am v1:172.21.15.192:43872/0 (socket says 172.21.15.192:43872) 2024-06-03T08:35:30.210 INFO:teuthology.orchestra.run.smithi192.stderr:2024-06-03T08:35:30.201+0000 7f34c2ffd640 1 -- 172.21.15.192:0/2771649427 learned_addr learned my addr 172.21.15.192:0/2771649427 (peer_addr_for_me v1:172.21.15.192:0/0) 2024-06-03T08:35:30.210 INFO:teuthology.orchestra.run.smithi192.stderr:2024-06-03T08:35:30.201+0000 7f34abfff640 10 client.?.objecter ms_handle_connect 0x7f34c40a8960 2024-06-03T08:35:30.210 INFO:teuthology.orchestra.run.smithi192.stderr:2024-06-03T08:35:30.201+0000 7f34abfff640 10 client.?.objecter resend_mon_ops 2024-06-03T08:35:30.210 INFO:teuthology.orchestra.run.smithi192.stderr:2024-06-03T08:35:30.201+0000 7f34abfff640 10 client.?.objecter ms_handle_connect 0x7f34c41450f0 2024-06-03T08:35:30.210 INFO:teuthology.orchestra.run.smithi192.stderr:2024-06-03T08:35:30.201+0000 7f34abfff640 10 client.?.objecter resend_mon_ops 2024-06-03T08:35:30.210 INFO:teuthology.orchestra.run.smithi192.stderr:2024-06-03T08:35:30.201+0000 7f34abfff640 10 client.?.objecter ms_handle_connect 0x7f34c413eec0 2024-06-03T08:35:30.210 INFO:teuthology.orchestra.run.smithi192.stderr:2024-06-03T08:35:30.201+0000 7f34abfff640 10 client.?.objecter resend_mon_ops 2024-06-03T08:35:30.211 INFO:teuthology.orchestra.run.smithi192.stderr:2024-06-03T08:35:30.203+0000 7f34abfff640 1 -- 172.21.15.192:0/2771649427 <== mon.2 v1:172.21.15.88:6790/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 2560269165 0 0) 0x7f34c41d6820 con 0x7f34c41450f0 2024-06-03T08:35:30.211 INFO:teuthology.orchestra.run.smithi192.stderr:2024-06-03T08:35:30.203+0000 7f34abfff640 10 monclient(hunting): _negotiate 2024-06-03T08:35:30.211 INFO:teuthology.orchestra.run.smithi192.stderr:2024-06-03T08:35:30.203+0000 7f34abfff640 10 monclient(hunting): _init_auth method 2 2024-06-03T08:35:30.211 INFO:teuthology.orchestra.run.smithi192.stderr:2024-06-03T08:35:30.203+0000 7f34abfff640 10 monclient(hunting): _init_auth creating new auth 2024-06-03T08:35:30.211 INFO:teuthology.orchestra.run.smithi192.stderr:2024-06-03T08:35:30.203+0000 7f34abfff640 10 monclient(hunting): my global_id is 4139 2024-06-03T08:35:30.211 INFO:teuthology.orchestra.run.smithi192.stderr:2024-06-03T08:35:30.203+0000 7f34abfff640 1 -- 172.21.15.192:0/2771649427 --> v1:172.21.15.88:6790/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f34b0004eb0 con 0x7f34c41450f0 2024-06-03T08:35:30.211 INFO:teuthology.orchestra.run.smithi192.stderr:2024-06-03T08:35:30.203+0000 7f34abfff640 1 -- 172.21.15.192:0/2771649427 <== mon.1 v1:172.21.15.192:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 769877247 0 0) 0x7f34c4142710 con 0x7f34c40a8960 2024-06-03T08:35:30.211 INFO:teuthology.orchestra.run.smithi192.stderr:2024-06-03T08:35:30.203+0000 7f34abfff640 10 monclient(hunting): _negotiate 2024-06-03T08:35:30.211 INFO:teuthology.orchestra.run.smithi192.stderr:2024-06-03T08:35:30.203+0000 7f34abfff640 10 monclient(hunting): _init_auth method 2 2024-06-03T08:35:30.212 INFO:teuthology.orchestra.run.smithi192.stderr:2024-06-03T08:35:30.203+0000 7f34abfff640 10 monclient(hunting): _init_auth creating new auth 2024-06-03T08:35:30.212 INFO:teuthology.orchestra.run.smithi192.stderr:2024-06-03T08:35:30.203+0000 7f34abfff640 10 monclient(hunting): my global_id is 4138 2024-06-03T08:35:30.212 INFO:teuthology.orchestra.run.smithi192.stderr:2024-06-03T08:35:30.203+0000 7f34abfff640 1 -- 172.21.15.192:0/2771649427 --> v1:172.21.15.192:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f34c41d6820 con 0x7f34c40a8960 2024-06-03T08:35:30.212 INFO:teuthology.orchestra.run.smithi192.stderr:2024-06-03T08:35:30.204+0000 7f34abfff640 1 -- 172.21.15.192:0/2771649427 <== mon.0 v1:172.21.15.88:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 430238619 0 0) 0x7f34c4143c60 con 0x7f34c413eec0 2024-06-03T08:35:30.212 INFO:teuthology.orchestra.run.smithi192.stderr:2024-06-03T08:35:30.204+0000 7f34abfff640 10 monclient(hunting): _negotiate 2024-06-03T08:35:30.212 INFO:teuthology.orchestra.run.smithi192.stderr:2024-06-03T08:35:30.204+0000 7f34abfff640 10 monclient(hunting): _init_auth method 2 2024-06-03T08:35:30.212 INFO:teuthology.orchestra.run.smithi192.stderr:2024-06-03T08:35:30.204+0000 7f34abfff640 10 monclient(hunting): _init_auth creating new auth 2024-06-03T08:35:30.212 INFO:teuthology.orchestra.run.smithi192.stderr:2024-06-03T08:35:30.204+0000 7f34abfff640 10 monclient(hunting): my global_id is 4137 2024-06-03T08:35:30.212 INFO:teuthology.orchestra.run.smithi192.stderr:2024-06-03T08:35:30.204+0000 7f34abfff640 1 -- 172.21.15.192:0/2771649427 --> v1:172.21.15.88:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f34c4142710 con 0x7f34c413eec0 2024-06-03T08:35:30.215 INFO:teuthology.orchestra.run.smithi192.stderr:2024-06-03T08:35:30.208+0000 7f34abfff640 1 -- 172.21.15.192:0/2771649427 <== mon.2 v1:172.21.15.88:6790/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 764+0+0 (unknown 1259696721 0 0) 0x7f34b0004eb0 con 0x7f34c41450f0 2024-06-03T08:35:30.215 INFO:teuthology.orchestra.run.smithi192.stderr:2024-06-03T08:35:30.208+0000 7f34abfff640 1 -- 172.21.15.192:0/2771649427 --> v1:172.21.15.88:6790/0 -- auth(proto 2 165 bytes epoch 0) v1 -- 0x7f34c4143c60 con 0x7f34c41450f0 2024-06-03T08:35:30.215 INFO:teuthology.orchestra.run.smithi192.stderr:2024-06-03T08:35:30.208+0000 7f34abfff640 1 -- 172.21.15.192:0/2771649427 <== mon.2 v1:172.21.15.88:6790/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 2234374065 0 0) 0x7f34bc001c30 con 0x7f34c41450f0 2024-06-03T08:35:30.215 INFO:teuthology.orchestra.run.smithi192.stderr:2024-06-03T08:35:30.208+0000 7f34abfff640 10 monclient(hunting): handle_monmap mon_map magic: 0 v1 2024-06-03T08:35:30.215 INFO:teuthology.orchestra.run.smithi192.stderr:2024-06-03T08:35:30.208+0000 7f34abfff640 10 monclient(hunting): got monmap 1 from mon.noname-c (according to old e1) 2024-06-03T08:35:30.215 INFO:teuthology.orchestra.run.smithi192.stderr:2024-06-03T08:35:30.208+0000 7f34abfff640 10 monclient(hunting): dump: 2024-06-03T08:35:30.215 INFO:teuthology.orchestra.run.smithi192.stderr:epoch 1 2024-06-03T08:35:30.215 INFO:teuthology.orchestra.run.smithi192.stderr:fsid ec3c3fdd-3ac6-4d7d-9412-683130ed8b04 2024-06-03T08:35:30.215 INFO:teuthology.orchestra.run.smithi192.stderr:last_changed 2024-06-03T08:34:31.885125+0000 2024-06-03T08:35:30.215 INFO:teuthology.orchestra.run.smithi192.stderr:created 2024-06-03T08:34:31.885125+0000 2024-06-03T08:35:30.215 INFO:teuthology.orchestra.run.smithi192.stderr:min_mon_release 18 (reef) 2024-06-03T08:35:30.216 INFO:teuthology.orchestra.run.smithi192.stderr:election_strategy: 3 2024-06-03T08:35:30.216 INFO:teuthology.orchestra.run.smithi192.stderr:0: v1:172.21.15.88:6789/0 mon.a 2024-06-03T08:35:30.216 INFO:teuthology.orchestra.run.smithi192.stderr:1: v1:172.21.15.192:6789/0 mon.b 2024-06-03T08:35:30.216 INFO:teuthology.orchestra.run.smithi192.stderr:2: v1:172.21.15.88:6790/0 mon.c 2024-06-03T08:35:30.216 INFO:teuthology.orchestra.run.smithi192.stderr: 2024-06-03T08:35:30.216 INFO:teuthology.orchestra.run.smithi192.stderr:2024-06-03T08:35:30.208+0000 7f34abfff640 10 monclient(hunting): _finish_auth 0 2024-06-03T08:35:30.216 INFO:teuthology.orchestra.run.smithi192.stderr:2024-06-03T08:35:30.209+0000 7f34abfff640 1 -- 172.21.15.192:0/2771649427 <== mon.0 v1:172.21.15.88:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 764+0+0 (unknown 1001951759 0 0) 0x7f34c4142710 con 0x7f34c413eec0 2024-06-03T08:35:30.216 INFO:teuthology.orchestra.run.smithi192.stderr:2024-06-03T08:35:30.209+0000 7f34abfff640 1 -- 172.21.15.192:0/2771649427 --> v1:172.21.15.88:6789/0 -- auth(proto 2 165 bytes epoch 0) v1 -- 0x7f34b0004eb0 con 0x7f34c413eec0 2024-06-03T08:35:30.216 INFO:teuthology.orchestra.run.smithi192.stderr:2024-06-03T08:35:30.209+0000 7f34abfff640 1 -- 172.21.15.192:0/2771649427 <== mon.0 v1:172.21.15.88:6789/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 2234374065 0 0) 0x7f34b8001810 con 0x7f34c413eec0 2024-06-03T08:35:30.216 INFO:teuthology.orchestra.run.smithi192.stderr:2024-06-03T08:35:30.209+0000 7f34abfff640 10 monclient(hunting): handle_monmap mon_map magic: 0 v1 2024-06-03T08:35:30.217 INFO:teuthology.orchestra.run.smithi192.stderr:2024-06-03T08:35:30.209+0000 7f34abfff640 10 monclient(hunting): got monmap 1 from mon.a (according to old e1) 2024-06-03T08:35:30.217 INFO:teuthology.orchestra.run.smithi192.stderr:2024-06-03T08:35:30.209+0000 7f34abfff640 10 monclient(hunting): dump: 2024-06-03T08:35:30.217 INFO:teuthology.orchestra.run.smithi192.stderr:epoch 1 2024-06-03T08:35:30.217 INFO:teuthology.orchestra.run.smithi192.stderr:fsid ec3c3fdd-3ac6-4d7d-9412-683130ed8b04 2024-06-03T08:35:30.217 INFO:teuthology.orchestra.run.smithi192.stderr:last_changed 2024-06-03T08:34:31.885125+0000 2024-06-03T08:35:30.217 INFO:teuthology.orchestra.run.smithi192.stderr:created 2024-06-03T08:34:31.885125+0000 2024-06-03T08:35:30.217 INFO:teuthology.orchestra.run.smithi192.stderr:min_mon_release 18 (reef) 2024-06-03T08:35:30.217 INFO:teuthology.orchestra.run.smithi192.stderr:election_strategy: 3 2024-06-03T08:35:30.217 INFO:teuthology.orchestra.run.smithi192.stderr:0: v1:172.21.15.88:6789/0 mon.a 2024-06-03T08:35:30.217 INFO:teuthology.orchestra.run.smithi192.stderr:1: v1:172.21.15.192:6789/0 mon.b 2024-06-03T08:35:30.217 INFO:teuthology.orchestra.run.smithi192.stderr:2: v1:172.21.15.88:6790/0 mon.c 2024-06-03T08:35:30.217 INFO:teuthology.orchestra.run.smithi192.stderr: 2024-06-03T08:35:30.217 INFO:teuthology.orchestra.run.smithi192.stderr:2024-06-03T08:35:30.210+0000 7f34abfff640 1 -- 172.21.15.192:0/2771649427 <== mon.1 v1:172.21.15.192:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 764+0+0 (unknown 550666304 0 0) 0x7f34c41d6820 con 0x7f34c40a8960 2024-06-03T08:35:30.217 INFO:teuthology.orchestra.run.smithi192.stderr:2024-06-03T08:35:30.210+0000 7f34abfff640 1 -- 172.21.15.192:0/2771649427 --> v1:172.21.15.192:6789/0 -- auth(proto 2 165 bytes epoch 0) v1 -- 0x7f34c4142710 con 0x7f34c40a8960 2024-06-03T08:35:30.218 INFO:teuthology.orchestra.run.smithi192.stderr:2024-06-03T08:35:30.210+0000 7f34abfff640 1 -- 172.21.15.192:0/2771649427 <== mon.1 v1:172.21.15.192:6789/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 2234374065 0 0) 0x7f34b4001440 con 0x7f34c40a8960 2024-06-03T08:35:30.218 INFO:teuthology.orchestra.run.smithi192.stderr:2024-06-03T08:35:30.210+0000 7f34abfff640 10 monclient(hunting): handle_monmap mon_map magic: 0 v1 2024-06-03T08:35:30.218 INFO:teuthology.orchestra.run.smithi192.stderr:2024-06-03T08:35:30.210+0000 7f34abfff640 10 monclient(hunting): got monmap 1 from mon.b (according to old e1) 2024-06-03T08:35:30.218 INFO:teuthology.orchestra.run.smithi192.stderr:2024-06-03T08:35:30.210+0000 7f34abfff640 10 monclient(hunting): dump: 2024-06-03T08:35:30.218 INFO:teuthology.orchestra.run.smithi192.stderr:epoch 1 2024-06-03T08:35:30.218 INFO:teuthology.orchestra.run.smithi192.stderr:fsid ec3c3fdd-3ac6-4d7d-9412-683130ed8b04 2024-06-03T08:35:30.218 INFO:teuthology.orchestra.run.smithi192.stderr:last_changed 2024-06-03T08:34:31.885125+0000 2024-06-03T08:35:30.218 INFO:teuthology.orchestra.run.smithi192.stderr:created 2024-06-03T08:34:31.885125+0000 2024-06-03T08:35:30.218 INFO:teuthology.orchestra.run.smithi192.stderr:min_mon_release 18 (reef) 2024-06-03T08:35:30.218 INFO:teuthology.orchestra.run.smithi192.stderr:election_strategy: 3 2024-06-03T08:35:30.218 INFO:teuthology.orchestra.run.smithi192.stderr:0: v1:172.21.15.88:6789/0 mon.a 2024-06-03T08:35:30.218 INFO:teuthology.orchestra.run.smithi192.stderr:1: v1:172.21.15.192:6789/0 mon.b 2024-06-03T08:35:30.218 INFO:teuthology.orchestra.run.smithi192.stderr:2: v1:172.21.15.88:6790/0 mon.c 2024-06-03T08:35:30.218 INFO:teuthology.orchestra.run.smithi192.stderr: 2024-06-03T08:35:30.219 INFO:teuthology.orchestra.run.smithi192.stderr:2024-06-03T08:35:30.211+0000 7f34abfff640 1 -- 172.21.15.192:0/2771649427 <== mon.2 v1:172.21.15.88:6790/0 4 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 580+0+0 (unknown 1849422748 0 0) 0x7f34c4143c60 con 0x7f34c41450f0 2024-06-03T08:35:30.219 INFO:teuthology.orchestra.run.smithi192.stderr:2024-06-03T08:35:30.211+0000 7f34abfff640 1 -- 172.21.15.192:0/2771649427 >> v1:172.21.15.192:6789/0 conn(0x7f34c40a8960 legacy=0x7f34c4143550 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-06-03T08:35:30.219 INFO:teuthology.orchestra.run.smithi192.stderr:2024-06-03T08:35:30.211+0000 7f34abfff640 1 -- 172.21.15.192:0/2771649427 >> v1:172.21.15.88:6789/0 conn(0x7f34c413eec0 legacy=0x7f34c40b0290 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-06-03T08:35:30.219 INFO:teuthology.orchestra.run.smithi192.stderr:2024-06-03T08:35:30.211+0000 7f34abfff640 10 monclient: _finish_hunting 0 2024-06-03T08:35:30.219 INFO:teuthology.orchestra.run.smithi192.stderr:2024-06-03T08:35:30.211+0000 7f34abfff640 1 monclient: found mon.c 2024-06-03T08:35:30.219 INFO:teuthology.orchestra.run.smithi192.stderr:2024-06-03T08:35:30.212+0000 7f34abfff640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-06-03T08:35:30.219 INFO:teuthology.orchestra.run.smithi192.stderr:2024-06-03T08:35:30.212+0000 7f34abfff640 10 monclient: _send_mon_message to mon.c at v1:172.21.15.88:6790/0 2024-06-03T08:35:30.219 INFO:teuthology.orchestra.run.smithi192.stderr:2024-06-03T08:35:30.212+0000 7f34abfff640 1 -- 172.21.15.192:0/2771649427 --> v1:172.21.15.88:6790/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f34c4140ba0 con 0x7f34c41450f0 2024-06-03T08:35:30.219 INFO:teuthology.orchestra.run.smithi192.stderr:2024-06-03T08:35:30.212+0000 7f34abfff640 10 monclient: _finish_auth 0 2024-06-03T08:35:30.219 INFO:teuthology.orchestra.run.smithi192.stderr:2024-06-03T08:35:30.212+0000 7f34abfff640 10 monclient: _check_auth_tickets 2024-06-03T08:35:30.219 INFO:teuthology.orchestra.run.smithi192.stderr:2024-06-03T08:35:30.212+0000 7f34abfff640 20 monclient: _check_auth_rotating not needed by client.admin 2024-06-03T08:35:30.219 INFO:teuthology.orchestra.run.smithi192.stderr:2024-06-03T08:35:30.212+0000 7f34c9a45640 5 monclient: authenticate success, global_id 4139 2024-06-03T08:35:30.219 INFO:teuthology.orchestra.run.smithi192.stderr:2024-06-03T08:35:30.212+0000 7f34c9a45640 10 monclient: _renew_subs 2024-06-03T08:35:30.219 INFO:teuthology.orchestra.run.smithi192.stderr:2024-06-03T08:35:30.212+0000 7f34c9a45640 10 monclient: _send_mon_message to mon.c at v1:172.21.15.88:6790/0 2024-06-03T08:35:30.220 INFO:teuthology.orchestra.run.smithi192.stderr:2024-06-03T08:35:30.212+0000 7f34c9a45640 1 -- 172.21.15.192:0/2771649427 --> v1:172.21.15.88:6790/0 -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f34c40b09a0 con 0x7f34c41450f0 2024-06-03T08:35:30.220 INFO:teuthology.orchestra.run.smithi192.stderr:2024-06-03T08:35:30.212+0000 7f34c9a45640 10 client.4139.objecter _maybe_request_map subscribing (onetime) to next osd map 2024-06-03T08:35:30.220 INFO:teuthology.orchestra.run.smithi192.stderr:2024-06-03T08:35:30.212+0000 7f34c9a45640 10 monclient: _renew_subs 2024-06-03T08:35:30.220 INFO:teuthology.orchestra.run.smithi192.stderr:2024-06-03T08:35:30.212+0000 7f34c9a45640 10 monclient: _send_mon_message to mon.c at v1:172.21.15.88:6790/0 2024-06-03T08:35:30.220 INFO:teuthology.orchestra.run.smithi192.stderr:2024-06-03T08:35:30.212+0000 7f34c9a45640 1 -- 172.21.15.192:0/2771649427 --> v1:172.21.15.88:6790/0 -- mon_subscribe({osdmap=0}) v3 -- 0x7f34c41d7630 con 0x7f34c41450f0 2024-06-03T08:35:30.220 INFO:teuthology.orchestra.run.smithi192.stderr:2024-06-03T08:35:30.212+0000 7f34c9a45640 1 librados: init done 2024-06-03T08:35:30.220 INFO:teuthology.orchestra.run.smithi192.stderr:2024-06-03T08:35:30.212+0000 7f34c9a45640 10 start_mon_command cmd=[{"prefix": "get_command_descriptions"}] 2024-06-03T08:35:30.220 INFO:teuthology.orchestra.run.smithi192.stderr:2024-06-03T08:35:30.212+0000 7f34c9a45640 10 monclient: _send_command 1 [{"prefix": "get_command_descriptions"}] 2024-06-03T08:35:30.220 INFO:teuthology.orchestra.run.smithi192.stderr:2024-06-03T08:35:30.212+0000 7f34c9a45640 10 monclient: _send_mon_message to mon.c at v1:172.21.15.88:6790/0 2024-06-03T08:35:30.220 INFO:teuthology.orchestra.run.smithi192.stderr:2024-06-03T08:35:30.212+0000 7f34c9a45640 1 -- 172.21.15.192:0/2771649427 --> v1:172.21.15.88:6790/0 -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f34c40a9e60 con 0x7f34c41450f0 2024-06-03T08:35:30.235 INFO:teuthology.orchestra.run.smithi192.stderr:2024-06-03T08:35:30.234+0000 7f34abfff640 1 -- 172.21.15.192:0/2771649427 <== mon.2 v1:172.21.15.88:6790/0 5 ==== config(0 keys) v1 ==== 4+0+0 (unknown 0 0 0) 0x7f34bc004b60 con 0x7f34c41450f0 2024-06-03T08:35:30.235 INFO:teuthology.orchestra.run.smithi192.stderr:2024-06-03T08:35:30.234+0000 7f34abfff640 10 monclient: handle_config config(0 keys) v1 2024-06-03T08:35:30.235 INFO:teuthology.orchestra.run.smithi192.stderr:2024-06-03T08:35:30.234+0000 7f34abfff640 1 -- 172.21.15.192:0/2771649427 <== mon.2 v1:172.21.15.88:6790/0 6 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 2234374065 0 0) 0x7f34bc004e70 con 0x7f34c41450f0 2024-06-03T08:35:30.235 INFO:teuthology.orchestra.run.smithi192.stderr:2024-06-03T08:35:30.234+0000 7f34abfff640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-06-03T08:35:30.235 INFO:teuthology.orchestra.run.smithi192.stderr:2024-06-03T08:35:30.234+0000 7f34abfff640 10 monclient: got monmap 1 from mon.c (according to old e1) 2024-06-03T08:35:30.235 INFO:teuthology.orchestra.run.smithi192.stderr:2024-06-03T08:35:30.234+0000 7f34abfff640 10 monclient: dump: 2024-06-03T08:35:30.235 INFO:teuthology.orchestra.run.smithi192.stderr:epoch 1 2024-06-03T08:35:30.235 INFO:teuthology.orchestra.run.smithi192.stderr:fsid ec3c3fdd-3ac6-4d7d-9412-683130ed8b04 2024-06-03T08:35:30.235 INFO:teuthology.orchestra.run.smithi192.stderr:last_changed 2024-06-03T08:34:31.885125+0000 2024-06-03T08:35:30.235 INFO:teuthology.orchestra.run.smithi192.stderr:created 2024-06-03T08:34:31.885125+0000 2024-06-03T08:35:30.235 INFO:teuthology.orchestra.run.smithi192.stderr:min_mon_release 18 (reef) 2024-06-03T08:35:30.236 INFO:teuthology.orchestra.run.smithi192.stderr:election_strategy: 3 2024-06-03T08:35:30.236 INFO:teuthology.orchestra.run.smithi192.stderr:0: v1:172.21.15.88:6789/0 mon.a 2024-06-03T08:35:30.236 INFO:teuthology.orchestra.run.smithi192.stderr:1: v1:172.21.15.192:6789/0 mon.b 2024-06-03T08:35:30.236 INFO:teuthology.orchestra.run.smithi192.stderr:2: v1:172.21.15.88:6790/0 mon.c 2024-06-03T08:35:30.236 INFO:teuthology.orchestra.run.smithi192.stderr: 2024-06-03T08:35:30.236 INFO:teuthology.orchestra.run.smithi192.stderr:2024-06-03T08:35:30.234+0000 7f34abfff640 1 -- 172.21.15.192:0/2771649427 <== mon.2 v1:172.21.15.88:6790/0 7 ==== mgrmap(e 1) v1 ==== 668+0+0 (unknown 3836259515 0 0) 0x7f34bc0052d0 con 0x7f34c41450f0 2024-06-03T08:35:30.236 INFO:teuthology.orchestra.run.smithi192.stderr:2024-06-03T08:35:30.234+0000 7f34abfff640 1 -- 172.21.15.192:0/2771649427 <== mon.2 v1:172.21.15.88:6790/0 8 ==== osd_map(5..5 src has 1..5) v4 ==== 1070+0+0 (unknown 3112267295 0 0) 0x7f34bc006580 con 0x7f34c41450f0 2024-06-03T08:35:30.236 INFO:teuthology.orchestra.run.smithi192.stderr:2024-06-03T08:35:30.234+0000 7f34abfff640 10 client.4139.objecter ms_dispatch 0x7f34c4000b90 osd_map(5..5 src has 1..5) v4 2024-06-03T08:35:30.236 INFO:teuthology.orchestra.run.smithi192.stderr:2024-06-03T08:35:30.234+0000 7f34abfff640 3 client.4139.objecter handle_osd_map got epochs [5,5] > 0 2024-06-03T08:35:30.236 INFO:teuthology.orchestra.run.smithi192.stderr:2024-06-03T08:35:30.234+0000 7f34abfff640 3 client.4139.objecter handle_osd_map decoding full epoch 5 2024-06-03T08:35:30.236 INFO:teuthology.orchestra.run.smithi192.stderr:2024-06-03T08:35:30.234+0000 7f34c17fa640 4 set_mon_vals no callback set 2024-06-03T08:35:30.236 INFO:teuthology.orchestra.run.smithi192.stderr:2024-06-03T08:35:30.234+0000 7f34abfff640 20 client.4139.objecter dump_active .. 0 homeless 2024-06-03T08:35:30.399 INFO:teuthology.orchestra.run.smithi192.stderr:2024-06-03T08:35:30.399+0000 7f34abfff640 1 -- 172.21.15.192:0/2771649427 <== mon.2 v1:172.21.15.88:6790/0 9 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+73121 (unknown 1092875540 0 1417653156) 0x7f34bc006200 con 0x7f34c41450f0 2024-06-03T08:35:30.400 INFO:teuthology.orchestra.run.smithi192.stderr:2024-06-03T08:35:30.399+0000 7f34abfff640 10 monclient: handle_mon_command_ack 1 [{"prefix": "get_command_descriptions"}] 2024-06-03T08:35:30.400 INFO:teuthology.orchestra.run.smithi192.stderr:2024-06-03T08:35:30.399+0000 7f34abfff640 10 monclient: _finish_command 1 = system:0 2024-06-03T08:35:30.447 INFO:tasks.ceph.mgr.x.smithi192.stderr:2024-06-03T08:35:30.447+0000 7f0e6be43040 -1 mgr[py] Module rook has missing NOTIFY_TYPES member 2024-06-03T08:35:30.472 INFO:teuthology.orchestra.run.smithi192.stderr:2024-06-03T08:35:30.471+0000 7f34c9a45640 10 start_mon_command cmd=[{"prefix": "osd new", "uuid": "6c5770b0-c93c-4101-8c53-bd761b8abcde", "id": 3}] 2024-06-03T08:35:30.472 INFO:teuthology.orchestra.run.smithi192.stderr:2024-06-03T08:35:30.471+0000 7f34c9a45640 10 monclient: _send_command 2 [{"prefix": "osd new", "uuid": "6c5770b0-c93c-4101-8c53-bd761b8abcde", "id": 3}] 2024-06-03T08:35:30.472 INFO:teuthology.orchestra.run.smithi192.stderr:2024-06-03T08:35:30.471+0000 7f34c9a45640 10 monclient: _send_mon_message to mon.c at v1:172.21.15.88:6790/0 2024-06-03T08:35:30.472 INFO:teuthology.orchestra.run.smithi192.stderr:2024-06-03T08:35:30.471+0000 7f34c9a45640 1 -- 172.21.15.192:0/2771649427 --> v1:172.21.15.88:6790/0 -- mon_command({"prefix": "osd new", "uuid": "6c5770b0-c93c-4101-8c53-bd761b8abcde", "id": 3} v 0) v1 -- 0x7f34c41d69b0 con 0x7f34c41450f0 2024-06-03T08:35:30.580 INFO:teuthology.orchestra.run.smithi192.stderr:2024-06-03T08:35:30.580+0000 7f34abfff640 1 -- 172.21.15.192:0/2771649427 <== mon.2 v1:172.21.15.88:6790/0 10 ==== mon_command_ack([{"prefix": "osd new", "uuid": "6c5770b0-c93c-4101-8c53-bd761b8abcde", "id": 3}]=0 v6) v1 ==== 112+0+2 (unknown 325948440 0 525046306) 0x7f34bc006790 con 0x7f34c41450f0 2024-06-03T08:35:30.580 INFO:teuthology.orchestra.run.smithi192.stderr:2024-06-03T08:35:30.580+0000 7f34abfff640 10 monclient: handle_mon_command_ack 2 [{"prefix": "osd new", "uuid": "6c5770b0-c93c-4101-8c53-bd761b8abcde", "id": 3}] 2024-06-03T08:35:30.580 INFO:teuthology.orchestra.run.smithi192.stderr:2024-06-03T08:35:30.580+0000 7f34abfff640 10 monclient: _finish_command 2 = system:0 2024-06-03T08:35:30.581 INFO:teuthology.orchestra.run.smithi192.stdout:3 2024-06-03T08:35:30.582 INFO:teuthology.orchestra.run.smithi192.stderr:2024-06-03T08:35:30.581+0000 7f34c9a45640 10 librados: watch_flush enter 2024-06-03T08:35:30.582 INFO:teuthology.orchestra.run.smithi192.stderr:2024-06-03T08:35:30.581+0000 7f34c9a45640 10 librados: watch_flush exit 2024-06-03T08:35:30.582 INFO:teuthology.orchestra.run.smithi192.stderr:2024-06-03T08:35:30.581+0000 7f34c9a45640 20 client.4139.objecter shutdown clearing up homeless session... 2024-06-03T08:35:30.582 INFO:teuthology.orchestra.run.smithi192.stderr:2024-06-03T08:35:30.581+0000 7f34c9a45640 10 client.4139.objecter successfully canceled tick 2024-06-03T08:35:30.582 INFO:teuthology.orchestra.run.smithi192.stderr:2024-06-03T08:35:30.581+0000 7f34c9a45640 10 monclient: shutdown 2024-06-03T08:35:30.582 INFO:teuthology.orchestra.run.smithi192.stderr:2024-06-03T08:35:30.581+0000 7f34c9a45640 20 monclient: shutdown discarding 0 pending message(s) 2024-06-03T08:35:30.583 INFO:teuthology.orchestra.run.smithi192.stderr:2024-06-03T08:35:30.581+0000 7f34c9a45640 1 -- 172.21.15.192:0/2771649427 >> v1:172.21.15.88:6790/0 conn(0x7f34c41450f0 legacy=0x7f34c41d6d40 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-06-03T08:35:30.583 INFO:teuthology.orchestra.run.smithi192.stderr:2024-06-03T08:35:30.581+0000 7f34c9a45640 1 -- 172.21.15.192:0/2771649427 shutdown_connections 2024-06-03T08:35:30.583 INFO:teuthology.orchestra.run.smithi192.stderr:2024-06-03T08:35:30.581+0000 7f34c9a45640 1 -- 172.21.15.192:0/2771649427 >> 172.21.15.192:0/2771649427 conn(0x7f34c40b70d0 msgr2=0x7f34c413e580 unknown :-1 s=STATE_NONE l=0).mark_down 2024-06-03T08:35:30.583 INFO:teuthology.orchestra.run.smithi192.stderr:2024-06-03T08:35:30.581+0000 7f34c9a45640 1 -- 172.21.15.192:0/2771649427 shutdown_connections 2024-06-03T08:35:30.583 INFO:teuthology.orchestra.run.smithi192.stderr:2024-06-03T08:35:30.581+0000 7f34c9a45640 1 -- 172.21.15.192:0/2771649427 wait complete. 2024-06-03T08:35:30.583 INFO:teuthology.orchestra.run.smithi192.stderr:2024-06-03T08:35:30.581+0000 7f34c9a45640 1 librados: shutdown 2024-06-03T08:35:30.583 INFO:tasks.ceph.mgr.y.smithi088.stderr:2024-06-03T08:35:30.581+0000 7ff0aea1d040 -1 mgr[py] Module rook has missing NOTIFY_TYPES member 2024-06-03T08:35:30.592 DEBUG:teuthology.orchestra.run.smithi192:> sudo ceph --cluster ceph osd new f001b3cb-2081-428b-bac3-b3df3c19b978 4 2024-06-03T08:35:30.690 INFO:teuthology.orchestra.run.smithi192.stderr:2024-06-03T08:35:30.690+0000 7fb9fb677640 10 monclient: get_monmap_and_config 2024-06-03T08:35:30.690 INFO:teuthology.orchestra.run.smithi192.stderr:2024-06-03T08:35:30.690+0000 7fb9fb677640 10 monclient: build_initial_monmap 2024-06-03T08:35:30.691 INFO:teuthology.orchestra.run.smithi192.stderr:2024-06-03T08:35:30.690+0000 7fb9fb677640 10 monclient: monmap: 2024-06-03T08:35:30.691 INFO:teuthology.orchestra.run.smithi192.stderr:epoch 0 2024-06-03T08:35:30.691 INFO:teuthology.orchestra.run.smithi192.stderr:fsid ec3c3fdd-3ac6-4d7d-9412-683130ed8b04 2024-06-03T08:35:30.691 INFO:teuthology.orchestra.run.smithi192.stderr:last_changed 2024-06-03T08:35:30.691209+0000 2024-06-03T08:35:30.691 INFO:teuthology.orchestra.run.smithi192.stderr:created 2024-06-03T08:35:30.691209+0000 2024-06-03T08:35:30.691 INFO:teuthology.orchestra.run.smithi192.stderr:min_mon_release 0 (unknown) 2024-06-03T08:35:30.691 INFO:teuthology.orchestra.run.smithi192.stderr:election_strategy: 3 2024-06-03T08:35:30.691 INFO:teuthology.orchestra.run.smithi192.stderr:0: v1:172.21.15.88:6789/0 mon.noname-a 2024-06-03T08:35:30.691 INFO:teuthology.orchestra.run.smithi192.stderr:1: v1:172.21.15.192:6789/0 mon.noname-c 2024-06-03T08:35:30.691 INFO:teuthology.orchestra.run.smithi192.stderr:2: v1:172.21.15.88:6790/0 mon.noname-b 2024-06-03T08:35:30.692 INFO:teuthology.orchestra.run.smithi192.stderr: 2024-06-03T08:35:30.693 INFO:teuthology.orchestra.run.smithi192.stderr:2024-06-03T08:35:30.690+0000 7fb9fb677640 1 Processor -- start 2024-06-03T08:35:30.693 INFO:teuthology.orchestra.run.smithi192.stderr:2024-06-03T08:35:30.691+0000 7fb9fb677640 1 -- start start 2024-06-03T08:35:30.693 INFO:teuthology.orchestra.run.smithi192.stderr:2024-06-03T08:35:30.691+0000 7fb9fb677640 10 monclient: init 2024-06-03T08:35:30.693 INFO:teuthology.orchestra.run.smithi192.stderr:2024-06-03T08:35:30.691+0000 7fb9fb677640 10 monclient: _reopen_session rank -1 2024-06-03T08:35:30.693 INFO:teuthology.orchestra.run.smithi192.stderr:2024-06-03T08:35:30.691+0000 7fb9fb677640 10 monclient: _add_conns ranks=[2,1,0] 2024-06-03T08:35:30.693 INFO:teuthology.orchestra.run.smithi192.stderr:2024-06-03T08:35:30.691+0000 7fb9fb677640 10 monclient(hunting): picked mon.noname-b con 0x7fb9f40b43c0 addr v1:172.21.15.88:6790/0 2024-06-03T08:35:30.694 INFO:teuthology.orchestra.run.smithi192.stderr:2024-06-03T08:35:30.691+0000 7fb9fb677640 10 monclient(hunting): picked mon.noname-c con 0x7fb9f413e6c0 addr v1:172.21.15.192:6789/0 2024-06-03T08:35:30.694 INFO:teuthology.orchestra.run.smithi192.stderr:2024-06-03T08:35:30.691+0000 7fb9fb677640 10 monclient(hunting): picked mon.noname-a con 0x7fb9f414ae20 addr v1:172.21.15.88:6789/0 2024-06-03T08:35:30.694 INFO:teuthology.orchestra.run.smithi192.stderr:2024-06-03T08:35:30.691+0000 7fb9fb677640 1 -- --> v1:172.21.15.88:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fb9f40a8ed0 con 0x7fb9f414ae20 2024-06-03T08:35:30.694 INFO:teuthology.orchestra.run.smithi192.stderr:2024-06-03T08:35:30.691+0000 7fb9fb677640 1 -- --> v1:172.21.15.192:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fb9f40aa420 con 0x7fb9f413e6c0 2024-06-03T08:35:30.694 INFO:teuthology.orchestra.run.smithi192.stderr:2024-06-03T08:35:30.691+0000 7fb9fb677640 1 -- --> v1:172.21.15.88:6790/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fb9f40aa690 con 0x7fb9f40b43c0 2024-06-03T08:35:30.694 INFO:teuthology.orchestra.run.smithi192.stderr:2024-06-03T08:35:30.691+0000 7fb9fb677640 10 monclient(hunting): _renew_subs 2024-06-03T08:35:30.694 INFO:teuthology.orchestra.run.smithi192.stderr:2024-06-03T08:35:30.691+0000 7fb9fb677640 10 monclient(hunting): authenticate will time out at 1196.502815s 2024-06-03T08:35:30.694 INFO:teuthology.orchestra.run.smithi192.stderr:2024-06-03T08:35:30.692+0000 7fb9f8c12640 1 --1- >> v1:172.21.15.192:6789/0 conn(0x7fb9f413e6c0 0x7fb9f413eb40 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.192:6789/0 says I am v1:172.21.15.192:43874/0 (socket says 172.21.15.192:43874) 2024-06-03T08:35:30.694 INFO:teuthology.orchestra.run.smithi192.stderr:2024-06-03T08:35:30.692+0000 7fb9f8c12640 1 -- 172.21.15.192:0/115720852 learned_addr learned my addr 172.21.15.192:0/115720852 (peer_addr_for_me v1:172.21.15.192:0/0) 2024-06-03T08:35:30.695 INFO:teuthology.orchestra.run.smithi192.stderr:2024-06-03T08:35:30.694+0000 7fb9ebfff640 1 -- 172.21.15.192:0/115720852 <== mon.1 v1:172.21.15.192:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 3891835689 0 0) 0x7fb9f40aa420 con 0x7fb9f413e6c0 2024-06-03T08:35:30.695 INFO:teuthology.orchestra.run.smithi192.stderr:2024-06-03T08:35:30.694+0000 7fb9ebfff640 10 monclient(hunting): _negotiate 2024-06-03T08:35:30.695 INFO:teuthology.orchestra.run.smithi192.stderr:2024-06-03T08:35:30.694+0000 7fb9ebfff640 10 monclient(hunting): _init_auth method 2 2024-06-03T08:35:30.696 INFO:teuthology.orchestra.run.smithi192.stderr:2024-06-03T08:35:30.694+0000 7fb9ebfff640 10 monclient(hunting): _init_auth creating new auth 2024-06-03T08:35:30.696 INFO:teuthology.orchestra.run.smithi192.stderr:2024-06-03T08:35:30.694+0000 7fb9ebfff640 10 monclient(hunting): my global_id is 4141 2024-06-03T08:35:30.696 INFO:teuthology.orchestra.run.smithi192.stderr:2024-06-03T08:35:30.694+0000 7fb9ebfff640 1 -- 172.21.15.192:0/115720852 --> v1:172.21.15.192:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7fb9e0003520 con 0x7fb9f413e6c0 2024-06-03T08:35:30.696 INFO:teuthology.orchestra.run.smithi192.stderr:2024-06-03T08:35:30.695+0000 7fb9ebfff640 1 -- 172.21.15.192:0/115720852 <== mon.0 v1:172.21.15.88:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 1008397220 0 0) 0x7fb9f40a8ed0 con 0x7fb9f414ae20 2024-06-03T08:35:30.696 INFO:teuthology.orchestra.run.smithi192.stderr:2024-06-03T08:35:30.695+0000 7fb9ebfff640 10 monclient(hunting): _negotiate 2024-06-03T08:35:30.696 INFO:teuthology.orchestra.run.smithi192.stderr:2024-06-03T08:35:30.695+0000 7fb9ebfff640 10 monclient(hunting): _init_auth method 2 2024-06-03T08:35:30.696 INFO:teuthology.orchestra.run.smithi192.stderr:2024-06-03T08:35:30.695+0000 7fb9ebfff640 10 monclient(hunting): _init_auth creating new auth 2024-06-03T08:35:30.696 INFO:teuthology.orchestra.run.smithi192.stderr:2024-06-03T08:35:30.695+0000 7fb9ebfff640 10 monclient(hunting): my global_id is 4140 2024-06-03T08:35:30.696 INFO:teuthology.orchestra.run.smithi192.stderr:2024-06-03T08:35:30.695+0000 7fb9ebfff640 1 -- 172.21.15.192:0/115720852 --> v1:172.21.15.88:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7fb9f40aa420 con 0x7fb9f414ae20 2024-06-03T08:35:30.696 INFO:teuthology.orchestra.run.smithi192.stderr:2024-06-03T08:35:30.695+0000 7fb9ebfff640 1 -- 172.21.15.192:0/115720852 <== mon.2 v1:172.21.15.88:6790/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 3767601404 0 0) 0x7fb9f40aa690 con 0x7fb9f40b43c0 2024-06-03T08:35:30.696 INFO:teuthology.orchestra.run.smithi192.stderr:2024-06-03T08:35:30.695+0000 7fb9ebfff640 10 monclient(hunting): _negotiate 2024-06-03T08:35:30.697 INFO:teuthology.orchestra.run.smithi192.stderr:2024-06-03T08:35:30.695+0000 7fb9ebfff640 10 monclient(hunting): _init_auth method 2 2024-06-03T08:35:30.697 INFO:teuthology.orchestra.run.smithi192.stderr:2024-06-03T08:35:30.695+0000 7fb9ebfff640 10 monclient(hunting): _init_auth creating new auth 2024-06-03T08:35:30.697 INFO:teuthology.orchestra.run.smithi192.stderr:2024-06-03T08:35:30.695+0000 7fb9ebfff640 10 monclient(hunting): my global_id is 4142 2024-06-03T08:35:30.697 INFO:teuthology.orchestra.run.smithi192.stderr:2024-06-03T08:35:30.695+0000 7fb9ebfff640 1 -- 172.21.15.192:0/115720852 --> v1:172.21.15.88:6790/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7fb9f40a8ed0 con 0x7fb9f40b43c0 2024-06-03T08:35:30.698 INFO:teuthology.orchestra.run.smithi192.stderr:2024-06-03T08:35:30.697+0000 7fb9ebfff640 1 -- 172.21.15.192:0/115720852 <== mon.1 v1:172.21.15.192:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 214+0+0 (unknown 1564730341 0 0) 0x7fb9e0003520 con 0x7fb9f413e6c0 2024-06-03T08:35:30.698 INFO:teuthology.orchestra.run.smithi192.stderr:2024-06-03T08:35:30.697+0000 7fb9ebfff640 1 -- 172.21.15.192:0/115720852 >> v1:172.21.15.88:6790/0 conn(0x7fb9f40b43c0 legacy=0x7fb9f41415b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-06-03T08:35:30.698 INFO:teuthology.orchestra.run.smithi192.stderr:2024-06-03T08:35:30.697+0000 7fb9ebfff640 1 -- 172.21.15.192:0/115720852 >> v1:172.21.15.88:6789/0 conn(0x7fb9f414ae20 legacy=0x7fb9f414d210 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-06-03T08:35:30.698 INFO:teuthology.orchestra.run.smithi192.stderr:2024-06-03T08:35:30.697+0000 7fb9ebfff640 10 monclient: _finish_hunting 0 2024-06-03T08:35:30.698 INFO:teuthology.orchestra.run.smithi192.stderr:2024-06-03T08:35:30.697+0000 7fb9ebfff640 1 monclient: found mon.noname-c 2024-06-03T08:35:30.698 INFO:teuthology.orchestra.run.smithi192.stderr:2024-06-03T08:35:30.697+0000 7fb9ebfff640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-06-03T08:35:30.698 INFO:teuthology.orchestra.run.smithi192.stderr:2024-06-03T08:35:30.697+0000 7fb9ebfff640 10 monclient: _send_mon_message to mon.noname-c at v1:172.21.15.192:6789/0 2024-06-03T08:35:30.698 INFO:teuthology.orchestra.run.smithi192.stderr:2024-06-03T08:35:30.697+0000 7fb9ebfff640 1 -- 172.21.15.192:0/115720852 --> v1:172.21.15.192:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fb9f40a87f0 con 0x7fb9f413e6c0 2024-06-03T08:35:30.698 INFO:teuthology.orchestra.run.smithi192.stderr:2024-06-03T08:35:30.697+0000 7fb9ebfff640 10 monclient: _finish_auth 0 2024-06-03T08:35:30.699 INFO:teuthology.orchestra.run.smithi192.stderr:2024-06-03T08:35:30.697+0000 7fb9ebfff640 10 monclient: _check_auth_tickets 2024-06-03T08:35:30.699 INFO:teuthology.orchestra.run.smithi192.stderr:2024-06-03T08:35:30.697+0000 7fb9ebfff640 20 monclient: _check_auth_rotating not needed by client.admin 2024-06-03T08:35:30.699 INFO:teuthology.orchestra.run.smithi192.stderr:2024-06-03T08:35:30.697+0000 7fb9ebfff640 1 -- 172.21.15.192:0/115720852 <== mon.1 v1:172.21.15.192:6789/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 2234374065 0 0) 0x7fb9e4001610 con 0x7fb9f413e6c0 2024-06-03T08:35:30.700 INFO:teuthology.orchestra.run.smithi192.stderr:2024-06-03T08:35:30.697+0000 7fb9fb677640 5 monclient: authenticate success, global_id 4141 2024-06-03T08:35:30.700 INFO:teuthology.orchestra.run.smithi192.stderr:2024-06-03T08:35:30.697+0000 7fb9fb677640 20 monclient: get_monmap_and_config waiting for monmap|config 2024-06-03T08:35:30.700 INFO:teuthology.orchestra.run.smithi192.stderr:2024-06-03T08:35:30.697+0000 7fb9ebfff640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-06-03T08:35:30.700 INFO:teuthology.orchestra.run.smithi192.stderr:2024-06-03T08:35:30.697+0000 7fb9ebfff640 10 monclient: got monmap 1 from mon.noname-c (according to old e1) 2024-06-03T08:35:30.700 INFO:teuthology.orchestra.run.smithi192.stderr:2024-06-03T08:35:30.697+0000 7fb9ebfff640 10 monclient: dump: 2024-06-03T08:35:30.700 INFO:teuthology.orchestra.run.smithi192.stderr:epoch 1 2024-06-03T08:35:30.700 INFO:teuthology.orchestra.run.smithi192.stderr:fsid ec3c3fdd-3ac6-4d7d-9412-683130ed8b04 2024-06-03T08:35:30.700 INFO:teuthology.orchestra.run.smithi192.stderr:last_changed 2024-06-03T08:34:31.885125+0000 2024-06-03T08:35:30.700 INFO:teuthology.orchestra.run.smithi192.stderr:created 2024-06-03T08:34:31.885125+0000 2024-06-03T08:35:30.700 INFO:teuthology.orchestra.run.smithi192.stderr:min_mon_release 18 (reef) 2024-06-03T08:35:30.700 INFO:teuthology.orchestra.run.smithi192.stderr:election_strategy: 3 2024-06-03T08:35:30.700 INFO:teuthology.orchestra.run.smithi192.stderr:0: v1:172.21.15.88:6789/0 mon.a 2024-06-03T08:35:30.700 INFO:teuthology.orchestra.run.smithi192.stderr:1: v1:172.21.15.192:6789/0 mon.b 2024-06-03T08:35:30.701 INFO:teuthology.orchestra.run.smithi192.stderr:2: v1:172.21.15.88:6790/0 mon.c 2024-06-03T08:35:30.701 INFO:teuthology.orchestra.run.smithi192.stderr: 2024-06-03T08:35:30.701 INFO:teuthology.orchestra.run.smithi192.stderr:2024-06-03T08:35:30.697+0000 7fb9fb677640 20 monclient: get_monmap_and_config waiting for monmap|config 2024-06-03T08:35:30.704 INFO:teuthology.orchestra.run.smithi192.stderr:2024-06-03T08:35:30.700+0000 7fb9ebfff640 1 -- 172.21.15.192:0/115720852 <== mon.1 v1:172.21.15.192:6789/0 4 ==== config(0 keys) v1 ==== 4+0+0 (unknown 0 0 0) 0x7fb9e4001830 con 0x7fb9f413e6c0 2024-06-03T08:35:30.704 INFO:teuthology.orchestra.run.smithi192.stderr:2024-06-03T08:35:30.700+0000 7fb9ebfff640 10 monclient: handle_config config(0 keys) v1 2024-06-03T08:35:30.704 INFO:teuthology.orchestra.run.smithi192.stderr:2024-06-03T08:35:30.700+0000 7fb9ebfff640 1 -- 172.21.15.192:0/115720852 <== mon.1 v1:172.21.15.192:6789/0 5 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 2234374065 0 0) 0x7fb9e4001bc0 con 0x7fb9f413e6c0 2024-06-03T08:35:30.704 INFO:teuthology.orchestra.run.smithi192.stderr:2024-06-03T08:35:30.700+0000 7fb9ebfff640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-06-03T08:35:30.704 INFO:teuthology.orchestra.run.smithi192.stderr:2024-06-03T08:35:30.700+0000 7fb9ebfff640 10 monclient: got monmap 1 from mon.b (according to old e1) 2024-06-03T08:35:30.704 INFO:teuthology.orchestra.run.smithi192.stderr:2024-06-03T08:35:30.700+0000 7fb9ebfff640 10 monclient: dump: 2024-06-03T08:35:30.704 INFO:teuthology.orchestra.run.smithi192.stderr:epoch 1 2024-06-03T08:35:30.704 INFO:teuthology.orchestra.run.smithi192.stderr:fsid ec3c3fdd-3ac6-4d7d-9412-683130ed8b04 2024-06-03T08:35:30.704 INFO:teuthology.orchestra.run.smithi192.stderr:last_changed 2024-06-03T08:34:31.885125+0000 2024-06-03T08:35:30.704 INFO:teuthology.orchestra.run.smithi192.stderr:created 2024-06-03T08:34:31.885125+0000 2024-06-03T08:35:30.705 INFO:teuthology.orchestra.run.smithi192.stderr:min_mon_release 18 (reef) 2024-06-03T08:35:30.705 INFO:teuthology.orchestra.run.smithi192.stderr:election_strategy: 3 2024-06-03T08:35:30.705 INFO:teuthology.orchestra.run.smithi192.stderr:0: v1:172.21.15.88:6789/0 mon.a 2024-06-03T08:35:30.705 INFO:teuthology.orchestra.run.smithi192.stderr:1: v1:172.21.15.192:6789/0 mon.b 2024-06-03T08:35:30.705 INFO:teuthology.orchestra.run.smithi192.stderr:2: v1:172.21.15.88:6790/0 mon.c 2024-06-03T08:35:30.705 INFO:teuthology.orchestra.run.smithi192.stderr: 2024-06-03T08:35:30.705 INFO:teuthology.orchestra.run.smithi192.stderr:2024-06-03T08:35:30.700+0000 7fb9fb677640 10 monclient: get_monmap_and_config success 2024-06-03T08:35:30.705 INFO:teuthology.orchestra.run.smithi192.stderr:2024-06-03T08:35:30.700+0000 7fb9fb677640 4 set_mon_vals no callback set 2024-06-03T08:35:30.705 INFO:teuthology.orchestra.run.smithi192.stderr:2024-06-03T08:35:30.700+0000 7fb9fb677640 10 monclient: shutdown 2024-06-03T08:35:30.705 INFO:teuthology.orchestra.run.smithi192.stderr:2024-06-03T08:35:30.700+0000 7fb9fb677640 20 monclient: shutdown discarding 0 pending message(s) 2024-06-03T08:35:30.705 INFO:teuthology.orchestra.run.smithi192.stderr:2024-06-03T08:35:30.700+0000 7fb9fb677640 1 -- 172.21.15.192:0/115720852 >> v1:172.21.15.192:6789/0 conn(0x7fb9f413e6c0 legacy=0x7fb9f413eb40 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-06-03T08:35:30.705 INFO:teuthology.orchestra.run.smithi192.stderr:2024-06-03T08:35:30.700+0000 7fb9fb677640 1 -- 172.21.15.192:0/115720852 shutdown_connections 2024-06-03T08:35:30.705 INFO:teuthology.orchestra.run.smithi192.stderr:2024-06-03T08:35:30.700+0000 7fb9fb677640 1 -- 172.21.15.192:0/115720852 >> 172.21.15.192:0/115720852 conn(0x7fb9f40719d0 msgr2=0x7fb9f4071de0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-06-03T08:35:30.705 INFO:teuthology.orchestra.run.smithi192.stderr:2024-06-03T08:35:30.700+0000 7fb9fb677640 1 -- 172.21.15.192:0/115720852 shutdown_connections 2024-06-03T08:35:30.706 INFO:teuthology.orchestra.run.smithi192.stderr:2024-06-03T08:35:30.700+0000 7fb9fb677640 1 -- 172.21.15.192:0/115720852 wait complete. 2024-06-03T08:35:30.706 INFO:teuthology.orchestra.run.smithi192.stderr:2024-06-03T08:35:30.701+0000 7fb9fb677640 10 monclient: build_initial_monmap 2024-06-03T08:35:30.706 INFO:teuthology.orchestra.run.smithi192.stderr:2024-06-03T08:35:30.701+0000 7fb9fb677640 10 monclient: monmap: 2024-06-03T08:35:30.706 INFO:teuthology.orchestra.run.smithi192.stderr:epoch 0 2024-06-03T08:35:30.706 INFO:teuthology.orchestra.run.smithi192.stderr:fsid 00000000-0000-0000-0000-000000000000 2024-06-03T08:35:30.706 INFO:teuthology.orchestra.run.smithi192.stderr:last_changed 0.000000 2024-06-03T08:35:30.706 INFO:teuthology.orchestra.run.smithi192.stderr:created 0.000000 2024-06-03T08:35:30.706 INFO:teuthology.orchestra.run.smithi192.stderr:min_mon_release 0 (unknown) 2024-06-03T08:35:30.706 INFO:teuthology.orchestra.run.smithi192.stderr:election_strategy: 1 2024-06-03T08:35:30.706 INFO:teuthology.orchestra.run.smithi192.stderr:0: v1:172.21.15.88:6789/0 mon.noname-a 2024-06-03T08:35:30.706 INFO:teuthology.orchestra.run.smithi192.stderr:1: v1:172.21.15.192:6789/0 mon.noname-b 2024-06-03T08:35:30.706 INFO:teuthology.orchestra.run.smithi192.stderr:2: v1:172.21.15.88:6790/0 mon.noname-c 2024-06-03T08:35:30.706 INFO:teuthology.orchestra.run.smithi192.stderr: 2024-06-03T08:35:30.706 INFO:teuthology.orchestra.run.smithi192.stderr:2024-06-03T08:35:30.701+0000 7fb9fb677640 1 librados: starting msgr at 2024-06-03T08:35:30.707 INFO:teuthology.orchestra.run.smithi192.stderr:2024-06-03T08:35:30.701+0000 7fb9fb677640 1 librados: starting objecter 2024-06-03T08:35:30.707 INFO:teuthology.orchestra.run.smithi192.stderr:2024-06-03T08:35:30.701+0000 7fb9fb677640 1 Processor -- start 2024-06-03T08:35:30.707 INFO:teuthology.orchestra.run.smithi192.stderr:2024-06-03T08:35:30.701+0000 7fb9fb677640 1 -- start start 2024-06-03T08:35:30.707 INFO:teuthology.orchestra.run.smithi192.stderr:2024-06-03T08:35:30.701+0000 7fb9fb677640 1 librados: setting wanted keys 2024-06-03T08:35:30.707 INFO:teuthology.orchestra.run.smithi192.stderr:2024-06-03T08:35:30.701+0000 7fb9fb677640 1 librados: calling monclient init 2024-06-03T08:35:30.707 INFO:teuthology.orchestra.run.smithi192.stderr:2024-06-03T08:35:30.701+0000 7fb9fb677640 10 monclient: init 2024-06-03T08:35:30.707 INFO:teuthology.orchestra.run.smithi192.stderr:2024-06-03T08:35:30.702+0000 7fb9fb677640 10 monclient: _reopen_session rank -1 2024-06-03T08:35:30.707 INFO:teuthology.orchestra.run.smithi192.stderr:2024-06-03T08:35:30.702+0000 7fb9fb677640 10 monclient: _add_conns ranks=[2,1,0] 2024-06-03T08:35:30.707 INFO:teuthology.orchestra.run.smithi192.stderr:2024-06-03T08:35:30.702+0000 7fb9fb677640 10 monclient(hunting): picked mon.noname-c con 0x7fb9f40b43c0 addr v1:172.21.15.88:6790/0 2024-06-03T08:35:30.707 INFO:teuthology.orchestra.run.smithi192.stderr:2024-06-03T08:35:30.702+0000 7fb9fb677640 10 monclient(hunting): picked mon.noname-b con 0x7fb9f413e6c0 addr v1:172.21.15.192:6789/0 2024-06-03T08:35:30.707 INFO:teuthology.orchestra.run.smithi192.stderr:2024-06-03T08:35:30.702+0000 7fb9fb677640 10 monclient(hunting): picked mon.noname-a con 0x7fb9f414ae20 addr v1:172.21.15.88:6789/0 2024-06-03T08:35:30.707 INFO:teuthology.orchestra.run.smithi192.stderr:2024-06-03T08:35:30.702+0000 7fb9fb677640 1 -- --> v1:172.21.15.88:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fb9f41360d0 con 0x7fb9f414ae20 2024-06-03T08:35:30.707 INFO:teuthology.orchestra.run.smithi192.stderr:2024-06-03T08:35:30.702+0000 7fb9fb677640 1 -- --> v1:172.21.15.192:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fb9f4134dc0 con 0x7fb9f413e6c0 2024-06-03T08:35:30.707 INFO:teuthology.orchestra.run.smithi192.stderr:2024-06-03T08:35:30.702+0000 7fb9fb677640 1 -- --> v1:172.21.15.88:6790/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fb9f4140ef0 con 0x7fb9f40b43c0 2024-06-03T08:35:30.708 INFO:teuthology.orchestra.run.smithi192.stderr:2024-06-03T08:35:30.702+0000 7fb9fb677640 10 monclient(hunting): _renew_subs 2024-06-03T08:35:30.708 INFO:teuthology.orchestra.run.smithi192.stderr:2024-06-03T08:35:30.702+0000 7fb9fb677640 10 monclient(hunting): authenticate will time out at 1196.513610s 2024-06-03T08:35:30.708 INFO:teuthology.orchestra.run.smithi192.stderr:2024-06-03T08:35:30.702+0000 7fb9f8c12640 1 --1- >> v1:172.21.15.192:6789/0 conn(0x7fb9f413e6c0 0x7fb9f4133fa0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=1).handle_server_banner_and_identify peer v1:172.21.15.192:6789/0 says I am v1:172.21.15.192:43878/0 (socket says 172.21.15.192:43878) 2024-06-03T08:35:30.708 INFO:teuthology.orchestra.run.smithi192.stderr:2024-06-03T08:35:30.702+0000 7fb9f8c12640 1 -- 172.21.15.192:0/4041815792 learned_addr learned my addr 172.21.15.192:0/4041815792 (peer_addr_for_me v1:172.21.15.192:0/0) 2024-06-03T08:35:30.708 INFO:teuthology.orchestra.run.smithi192.stderr:2024-06-03T08:35:30.703+0000 7fb9e9ffb640 10 client.?.objecter ms_handle_connect 0x7fb9f413e6c0 2024-06-03T08:35:30.708 INFO:teuthology.orchestra.run.smithi192.stderr:2024-06-03T08:35:30.703+0000 7fb9e9ffb640 10 client.?.objecter resend_mon_ops 2024-06-03T08:35:30.708 INFO:teuthology.orchestra.run.smithi192.stderr:2024-06-03T08:35:30.703+0000 7fb9e9ffb640 10 client.?.objecter ms_handle_connect 0x7fb9f40b43c0 2024-06-03T08:35:30.708 INFO:teuthology.orchestra.run.smithi192.stderr:2024-06-03T08:35:30.703+0000 7fb9e9ffb640 10 client.?.objecter resend_mon_ops 2024-06-03T08:35:30.708 INFO:teuthology.orchestra.run.smithi192.stderr:2024-06-03T08:35:30.703+0000 7fb9e9ffb640 10 client.?.objecter ms_handle_connect 0x7fb9f414ae20 2024-06-03T08:35:30.708 INFO:teuthology.orchestra.run.smithi192.stderr:2024-06-03T08:35:30.703+0000 7fb9e9ffb640 10 client.?.objecter resend_mon_ops 2024-06-03T08:35:30.709 INFO:teuthology.orchestra.run.smithi192.stderr:2024-06-03T08:35:30.705+0000 7fb9e9ffb640 1 -- 172.21.15.192:0/4041815792 <== mon.1 v1:172.21.15.192:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 909801467 0 0) 0x7fb9f4134dc0 con 0x7fb9f413e6c0 2024-06-03T08:35:30.709 INFO:teuthology.orchestra.run.smithi192.stderr:2024-06-03T08:35:30.705+0000 7fb9e9ffb640 10 monclient(hunting): _negotiate 2024-06-03T08:35:30.709 INFO:teuthology.orchestra.run.smithi192.stderr:2024-06-03T08:35:30.705+0000 7fb9e9ffb640 10 monclient(hunting): _init_auth method 2 2024-06-03T08:35:30.709 INFO:teuthology.orchestra.run.smithi192.stderr:2024-06-03T08:35:30.705+0000 7fb9e9ffb640 10 monclient(hunting): _init_auth creating new auth 2024-06-03T08:35:30.709 INFO:teuthology.orchestra.run.smithi192.stderr:2024-06-03T08:35:30.705+0000 7fb9e9ffb640 10 monclient(hunting): my global_id is 4144 2024-06-03T08:35:30.709 INFO:teuthology.orchestra.run.smithi192.stderr:2024-06-03T08:35:30.705+0000 7fb9e9ffb640 1 -- 172.21.15.192:0/4041815792 --> v1:172.21.15.192:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7fb9d8004eb0 con 0x7fb9f413e6c0 2024-06-03T08:35:30.709 INFO:teuthology.orchestra.run.smithi192.stderr:2024-06-03T08:35:30.705+0000 7fb9e9ffb640 1 -- 172.21.15.192:0/4041815792 <== mon.2 v1:172.21.15.88:6790/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 4022534608 0 0) 0x7fb9f4140ef0 con 0x7fb9f40b43c0 2024-06-03T08:35:30.710 INFO:teuthology.orchestra.run.smithi192.stderr:2024-06-03T08:35:30.705+0000 7fb9e9ffb640 10 monclient(hunting): _negotiate 2024-06-03T08:35:30.710 INFO:teuthology.orchestra.run.smithi192.stderr:2024-06-03T08:35:30.705+0000 7fb9e9ffb640 10 monclient(hunting): _init_auth method 2 2024-06-03T08:35:30.710 INFO:teuthology.orchestra.run.smithi192.stderr:2024-06-03T08:35:30.705+0000 7fb9e9ffb640 10 monclient(hunting): _init_auth creating new auth 2024-06-03T08:35:30.710 INFO:teuthology.orchestra.run.smithi192.stderr:2024-06-03T08:35:30.705+0000 7fb9e9ffb640 10 monclient(hunting): my global_id is 4145 2024-06-03T08:35:30.710 INFO:teuthology.orchestra.run.smithi192.stderr:2024-06-03T08:35:30.705+0000 7fb9e9ffb640 1 -- 172.21.15.192:0/4041815792 --> v1:172.21.15.88:6790/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7fb9f4134dc0 con 0x7fb9f40b43c0 2024-06-03T08:35:30.710 INFO:teuthology.orchestra.run.smithi192.stderr:2024-06-03T08:35:30.705+0000 7fb9e9ffb640 1 -- 172.21.15.192:0/4041815792 <== mon.0 v1:172.21.15.88:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 2312009880 0 0) 0x7fb9f41360d0 con 0x7fb9f414ae20 2024-06-03T08:35:30.710 INFO:teuthology.orchestra.run.smithi192.stderr:2024-06-03T08:35:30.705+0000 7fb9e9ffb640 10 monclient(hunting): _negotiate 2024-06-03T08:35:30.710 INFO:teuthology.orchestra.run.smithi192.stderr:2024-06-03T08:35:30.705+0000 7fb9e9ffb640 10 monclient(hunting): _init_auth method 2 2024-06-03T08:35:30.710 INFO:teuthology.orchestra.run.smithi192.stderr:2024-06-03T08:35:30.705+0000 7fb9e9ffb640 10 monclient(hunting): _init_auth creating new auth 2024-06-03T08:35:30.710 INFO:teuthology.orchestra.run.smithi192.stderr:2024-06-03T08:35:30.705+0000 7fb9e9ffb640 10 monclient(hunting): my global_id is 4143 2024-06-03T08:35:30.710 INFO:teuthology.orchestra.run.smithi192.stderr:2024-06-03T08:35:30.705+0000 7fb9e9ffb640 1 -- 172.21.15.192:0/4041815792 --> v1:172.21.15.88:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7fb9f4140ef0 con 0x7fb9f414ae20 2024-06-03T08:35:30.711 INFO:teuthology.orchestra.run.smithi192.stderr:2024-06-03T08:35:30.709+0000 7fb9e9ffb640 1 -- 172.21.15.192:0/4041815792 <== mon.1 v1:172.21.15.192:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 764+0+0 (unknown 1091027088 0 0) 0x7fb9d8004eb0 con 0x7fb9f413e6c0 2024-06-03T08:35:30.711 INFO:teuthology.orchestra.run.smithi192.stderr:2024-06-03T08:35:30.709+0000 7fb9e9ffb640 1 -- 172.21.15.192:0/4041815792 --> v1:172.21.15.192:6789/0 -- auth(proto 2 165 bytes epoch 0) v1 -- 0x7fb9f41360d0 con 0x7fb9f413e6c0 2024-06-03T08:35:30.712 INFO:teuthology.orchestra.run.smithi192.stderr:2024-06-03T08:35:30.709+0000 7fb9e9ffb640 1 -- 172.21.15.192:0/4041815792 <== mon.1 v1:172.21.15.192:6789/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 2234374065 0 0) 0x7fb9e4001b40 con 0x7fb9f413e6c0 2024-06-03T08:35:30.712 INFO:teuthology.orchestra.run.smithi192.stderr:2024-06-03T08:35:30.709+0000 7fb9e9ffb640 10 monclient(hunting): handle_monmap mon_map magic: 0 v1 2024-06-03T08:35:30.712 INFO:teuthology.orchestra.run.smithi192.stderr:2024-06-03T08:35:30.709+0000 7fb9e9ffb640 10 monclient(hunting): got monmap 1 from mon.noname-b (according to old e1) 2024-06-03T08:35:30.712 INFO:teuthology.orchestra.run.smithi192.stderr:2024-06-03T08:35:30.709+0000 7fb9e9ffb640 10 monclient(hunting): dump: 2024-06-03T08:35:30.712 INFO:teuthology.orchestra.run.smithi192.stderr:epoch 1 2024-06-03T08:35:30.712 INFO:teuthology.orchestra.run.smithi192.stderr:fsid ec3c3fdd-3ac6-4d7d-9412-683130ed8b04 2024-06-03T08:35:30.712 INFO:teuthology.orchestra.run.smithi192.stderr:last_changed 2024-06-03T08:34:31.885125+0000 2024-06-03T08:35:30.712 INFO:teuthology.orchestra.run.smithi192.stderr:created 2024-06-03T08:34:31.885125+0000 2024-06-03T08:35:30.712 INFO:teuthology.orchestra.run.smithi192.stderr:min_mon_release 18 (reef) 2024-06-03T08:35:30.712 INFO:teuthology.orchestra.run.smithi192.stderr:election_strategy: 3 2024-06-03T08:35:30.712 INFO:teuthology.orchestra.run.smithi192.stderr:0: v1:172.21.15.88:6789/0 mon.a 2024-06-03T08:35:30.712 INFO:teuthology.orchestra.run.smithi192.stderr:1: v1:172.21.15.192:6789/0 mon.b 2024-06-03T08:35:30.712 INFO:teuthology.orchestra.run.smithi192.stderr:2: v1:172.21.15.88:6790/0 mon.c 2024-06-03T08:35:30.712 INFO:teuthology.orchestra.run.smithi192.stderr: 2024-06-03T08:35:30.713 INFO:teuthology.orchestra.run.smithi192.stderr:2024-06-03T08:35:30.709+0000 7fb9e9ffb640 10 monclient(hunting): _finish_auth 0 2024-06-03T08:35:30.713 INFO:teuthology.orchestra.run.smithi192.stderr:2024-06-03T08:35:30.710+0000 7fb9e9ffb640 1 -- 172.21.15.192:0/4041815792 <== mon.2 v1:172.21.15.88:6790/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 764+0+0 (unknown 81210123 0 0) 0x7fb9f4134dc0 con 0x7fb9f40b43c0 2024-06-03T08:35:30.713 INFO:teuthology.orchestra.run.smithi192.stderr:2024-06-03T08:35:30.710+0000 7fb9e9ffb640 1 -- 172.21.15.192:0/4041815792 --> v1:172.21.15.88:6790/0 -- auth(proto 2 165 bytes epoch 0) v1 -- 0x7fb9d8004eb0 con 0x7fb9f40b43c0 2024-06-03T08:35:30.713 INFO:teuthology.orchestra.run.smithi192.stderr:2024-06-03T08:35:30.710+0000 7fb9e9ffb640 1 -- 172.21.15.192:0/4041815792 <== mon.2 v1:172.21.15.88:6790/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 2234374065 0 0) 0x7fb9f00018b0 con 0x7fb9f40b43c0 2024-06-03T08:35:30.713 INFO:teuthology.orchestra.run.smithi192.stderr:2024-06-03T08:35:30.710+0000 7fb9e9ffb640 10 monclient(hunting): handle_monmap mon_map magic: 0 v1 2024-06-03T08:35:30.713 INFO:teuthology.orchestra.run.smithi192.stderr:2024-06-03T08:35:30.710+0000 7fb9e9ffb640 10 monclient(hunting): got monmap 1 from mon.c (according to old e1) 2024-06-03T08:35:30.713 INFO:teuthology.orchestra.run.smithi192.stderr:2024-06-03T08:35:30.710+0000 7fb9e9ffb640 10 monclient(hunting): dump: 2024-06-03T08:35:30.713 INFO:teuthology.orchestra.run.smithi192.stderr:epoch 1 2024-06-03T08:35:30.713 INFO:teuthology.orchestra.run.smithi192.stderr:fsid ec3c3fdd-3ac6-4d7d-9412-683130ed8b04 2024-06-03T08:35:30.713 INFO:teuthology.orchestra.run.smithi192.stderr:last_changed 2024-06-03T08:34:31.885125+0000 2024-06-03T08:35:30.713 INFO:teuthology.orchestra.run.smithi192.stderr:created 2024-06-03T08:34:31.885125+0000 2024-06-03T08:35:30.713 INFO:teuthology.orchestra.run.smithi192.stderr:min_mon_release 18 (reef) 2024-06-03T08:35:30.713 INFO:teuthology.orchestra.run.smithi192.stderr:election_strategy: 3 2024-06-03T08:35:30.713 INFO:teuthology.orchestra.run.smithi192.stderr:0: v1:172.21.15.88:6789/0 mon.a 2024-06-03T08:35:30.714 INFO:teuthology.orchestra.run.smithi192.stderr:1: v1:172.21.15.192:6789/0 mon.b 2024-06-03T08:35:30.714 INFO:teuthology.orchestra.run.smithi192.stderr:2: v1:172.21.15.88:6790/0 mon.c 2024-06-03T08:35:30.714 INFO:teuthology.orchestra.run.smithi192.stderr: 2024-06-03T08:35:30.714 INFO:teuthology.orchestra.run.smithi192.stderr:2024-06-03T08:35:30.711+0000 7fb9e9ffb640 1 -- 172.21.15.192:0/4041815792 <== mon.0 v1:172.21.15.88:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 764+0+0 (unknown 3810168465 0 0) 0x7fb9f4140ef0 con 0x7fb9f414ae20 2024-06-03T08:35:30.714 INFO:teuthology.orchestra.run.smithi192.stderr:2024-06-03T08:35:30.711+0000 7fb9e9ffb640 1 -- 172.21.15.192:0/4041815792 --> v1:172.21.15.88:6789/0 -- auth(proto 2 165 bytes epoch 0) v1 -- 0x7fb9f4134dc0 con 0x7fb9f414ae20 2024-06-03T08:35:30.714 INFO:teuthology.orchestra.run.smithi192.stderr:2024-06-03T08:35:30.711+0000 7fb9e9ffb640 1 -- 172.21.15.192:0/4041815792 <== mon.0 v1:172.21.15.88:6789/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 2234374065 0 0) 0x7fb9ec001b50 con 0x7fb9f414ae20 2024-06-03T08:35:30.714 INFO:teuthology.orchestra.run.smithi192.stderr:2024-06-03T08:35:30.711+0000 7fb9e9ffb640 10 monclient(hunting): handle_monmap mon_map magic: 0 v1 2024-06-03T08:35:30.714 INFO:teuthology.orchestra.run.smithi192.stderr:2024-06-03T08:35:30.711+0000 7fb9e9ffb640 10 monclient(hunting): got monmap 1 from mon.a (according to old e1) 2024-06-03T08:35:30.714 INFO:teuthology.orchestra.run.smithi192.stderr:2024-06-03T08:35:30.711+0000 7fb9e9ffb640 10 monclient(hunting): dump: 2024-06-03T08:35:30.714 INFO:teuthology.orchestra.run.smithi192.stderr:epoch 1 2024-06-03T08:35:30.714 INFO:teuthology.orchestra.run.smithi192.stderr:fsid ec3c3fdd-3ac6-4d7d-9412-683130ed8b04 2024-06-03T08:35:30.714 INFO:teuthology.orchestra.run.smithi192.stderr:last_changed 2024-06-03T08:34:31.885125+0000 2024-06-03T08:35:30.714 INFO:teuthology.orchestra.run.smithi192.stderr:created 2024-06-03T08:34:31.885125+0000 2024-06-03T08:35:30.714 INFO:teuthology.orchestra.run.smithi192.stderr:min_mon_release 18 (reef) 2024-06-03T08:35:30.715 INFO:teuthology.orchestra.run.smithi192.stderr:election_strategy: 3 2024-06-03T08:35:30.715 INFO:teuthology.orchestra.run.smithi192.stderr:0: v1:172.21.15.88:6789/0 mon.a 2024-06-03T08:35:30.715 INFO:teuthology.orchestra.run.smithi192.stderr:1: v1:172.21.15.192:6789/0 mon.b 2024-06-03T08:35:30.715 INFO:teuthology.orchestra.run.smithi192.stderr:2: v1:172.21.15.88:6790/0 mon.c 2024-06-03T08:35:30.715 INFO:teuthology.orchestra.run.smithi192.stderr: 2024-06-03T08:35:30.716 INFO:teuthology.orchestra.run.smithi192.stderr:2024-06-03T08:35:30.713+0000 7fb9e9ffb640 1 -- 172.21.15.192:0/4041815792 <== mon.1 v1:172.21.15.192:6789/0 4 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 580+0+0 (unknown 2358315146 0 0) 0x7fb9f41360d0 con 0x7fb9f413e6c0 2024-06-03T08:35:30.716 INFO:teuthology.orchestra.run.smithi192.stderr:2024-06-03T08:35:30.713+0000 7fb9e9ffb640 1 -- 172.21.15.192:0/4041815792 >> v1:172.21.15.88:6790/0 conn(0x7fb9f40b43c0 legacy=0x7fb9f41d9bc0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-06-03T08:35:30.716 INFO:teuthology.orchestra.run.smithi192.stderr:2024-06-03T08:35:30.713+0000 7fb9e9ffb640 1 -- 172.21.15.192:0/4041815792 >> v1:172.21.15.88:6789/0 conn(0x7fb9f414ae20 legacy=0x7fb9f41346b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-06-03T08:35:30.717 INFO:teuthology.orchestra.run.smithi192.stderr:2024-06-03T08:35:30.713+0000 7fb9e9ffb640 10 monclient: _finish_hunting 0 2024-06-03T08:35:30.717 INFO:teuthology.orchestra.run.smithi192.stderr:2024-06-03T08:35:30.713+0000 7fb9e9ffb640 1 monclient: found mon.b 2024-06-03T08:35:30.717 INFO:teuthology.orchestra.run.smithi192.stderr:2024-06-03T08:35:30.713+0000 7fb9e9ffb640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-06-03T08:35:30.717 INFO:teuthology.orchestra.run.smithi192.stderr:2024-06-03T08:35:30.713+0000 7fb9e9ffb640 10 monclient: _send_mon_message to mon.b at v1:172.21.15.192:6789/0 2024-06-03T08:35:30.717 INFO:teuthology.orchestra.run.smithi192.stderr:2024-06-03T08:35:30.713+0000 7fb9e9ffb640 1 -- 172.21.15.192:0/4041815792 --> v1:172.21.15.192:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fb9f4135c60 con 0x7fb9f413e6c0 2024-06-03T08:35:30.717 INFO:teuthology.orchestra.run.smithi192.stderr:2024-06-03T08:35:30.713+0000 7fb9e9ffb640 10 monclient: _finish_auth 0 2024-06-03T08:35:30.717 INFO:teuthology.orchestra.run.smithi192.stderr:2024-06-03T08:35:30.713+0000 7fb9e9ffb640 10 monclient: _check_auth_tickets 2024-06-03T08:35:30.717 INFO:teuthology.orchestra.run.smithi192.stderr:2024-06-03T08:35:30.713+0000 7fb9e9ffb640 20 monclient: _check_auth_rotating not needed by client.admin 2024-06-03T08:35:30.717 INFO:teuthology.orchestra.run.smithi192.stderr:2024-06-03T08:35:30.713+0000 7fb9fb677640 5 monclient: authenticate success, global_id 4144 2024-06-03T08:35:30.717 INFO:teuthology.orchestra.run.smithi192.stderr:2024-06-03T08:35:30.713+0000 7fb9fb677640 10 monclient: _renew_subs 2024-06-03T08:35:30.717 INFO:teuthology.orchestra.run.smithi192.stderr:2024-06-03T08:35:30.713+0000 7fb9fb677640 10 monclient: _send_mon_message to mon.b at v1:172.21.15.192:6789/0 2024-06-03T08:35:30.717 INFO:teuthology.orchestra.run.smithi192.stderr:2024-06-03T08:35:30.713+0000 7fb9fb677640 1 -- 172.21.15.192:0/4041815792 --> v1:172.21.15.192:6789/0 -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fb9f4136560 con 0x7fb9f413e6c0 2024-06-03T08:35:30.717 INFO:teuthology.orchestra.run.smithi192.stderr:2024-06-03T08:35:30.713+0000 7fb9fb677640 10 client.4144.objecter _maybe_request_map subscribing (onetime) to next osd map 2024-06-03T08:35:30.717 INFO:teuthology.orchestra.run.smithi192.stderr:2024-06-03T08:35:30.713+0000 7fb9fb677640 10 monclient: _renew_subs 2024-06-03T08:35:30.718 INFO:teuthology.orchestra.run.smithi192.stderr:2024-06-03T08:35:30.713+0000 7fb9fb677640 10 monclient: _send_mon_message to mon.b at v1:172.21.15.192:6789/0 2024-06-03T08:35:30.718 INFO:teuthology.orchestra.run.smithi192.stderr:2024-06-03T08:35:30.713+0000 7fb9fb677640 1 -- 172.21.15.192:0/4041815792 --> v1:172.21.15.192:6789/0 -- mon_subscribe({osdmap=0}) v3 -- 0x7fb9f41da4b0 con 0x7fb9f413e6c0 2024-06-03T08:35:30.718 INFO:teuthology.orchestra.run.smithi192.stderr:2024-06-03T08:35:30.713+0000 7fb9fb677640 1 librados: init done 2024-06-03T08:35:30.718 INFO:teuthology.orchestra.run.smithi192.stderr:2024-06-03T08:35:30.714+0000 7fb9fb677640 10 start_mon_command cmd=[{"prefix": "get_command_descriptions"}] 2024-06-03T08:35:30.718 INFO:teuthology.orchestra.run.smithi192.stderr:2024-06-03T08:35:30.714+0000 7fb9fb677640 10 monclient: _send_command 1 [{"prefix": "get_command_descriptions"}] 2024-06-03T08:35:30.718 INFO:teuthology.orchestra.run.smithi192.stderr:2024-06-03T08:35:30.714+0000 7fb9fb677640 10 monclient: _send_mon_message to mon.b at v1:172.21.15.192:6789/0 2024-06-03T08:35:30.718 INFO:teuthology.orchestra.run.smithi192.stderr:2024-06-03T08:35:30.714+0000 7fb9fb677640 1 -- 172.21.15.192:0/4041815792 --> v1:172.21.15.192:6789/0 -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fb9f4136750 con 0x7fb9f413e6c0 2024-06-03T08:35:30.721 INFO:tasks.ceph.mgr.x.smithi192.stderr:2024-06-03T08:35:30.721+0000 7f0e6be43040 -1 mgr[py] Module volumes has missing NOTIFY_TYPES member 2024-06-03T08:35:30.735 INFO:teuthology.orchestra.run.smithi192.stderr:2024-06-03T08:35:30.734+0000 7fb9e9ffb640 1 -- 172.21.15.192:0/4041815792 <== mon.1 v1:172.21.15.192:6789/0 5 ==== config(0 keys) v1 ==== 4+0+0 (unknown 0 0 0) 0x7fb9e4004a90 con 0x7fb9f413e6c0 2024-06-03T08:35:30.735 INFO:teuthology.orchestra.run.smithi192.stderr:2024-06-03T08:35:30.734+0000 7fb9e9ffb640 10 monclient: handle_config config(0 keys) v1 2024-06-03T08:35:30.735 INFO:teuthology.orchestra.run.smithi192.stderr:2024-06-03T08:35:30.734+0000 7fb9e9ffb640 1 -- 172.21.15.192:0/4041815792 <== mon.1 v1:172.21.15.192:6789/0 6 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 2234374065 0 0) 0x7fb9e4004da0 con 0x7fb9f413e6c0 2024-06-03T08:35:30.735 INFO:teuthology.orchestra.run.smithi192.stderr:2024-06-03T08:35:30.734+0000 7fb9e9ffb640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-06-03T08:35:30.735 INFO:teuthology.orchestra.run.smithi192.stderr:2024-06-03T08:35:30.734+0000 7fb9e9ffb640 10 monclient: got monmap 1 from mon.b (according to old e1) 2024-06-03T08:35:30.735 INFO:teuthology.orchestra.run.smithi192.stderr:2024-06-03T08:35:30.734+0000 7fb9e9ffb640 10 monclient: dump: 2024-06-03T08:35:30.735 INFO:teuthology.orchestra.run.smithi192.stderr:epoch 1 2024-06-03T08:35:30.735 INFO:teuthology.orchestra.run.smithi192.stderr:fsid ec3c3fdd-3ac6-4d7d-9412-683130ed8b04 2024-06-03T08:35:30.735 INFO:teuthology.orchestra.run.smithi192.stderr:last_changed 2024-06-03T08:34:31.885125+0000 2024-06-03T08:35:30.735 INFO:teuthology.orchestra.run.smithi192.stderr:created 2024-06-03T08:34:31.885125+0000 2024-06-03T08:35:30.735 INFO:teuthology.orchestra.run.smithi192.stderr:min_mon_release 18 (reef) 2024-06-03T08:35:30.735 INFO:teuthology.orchestra.run.smithi192.stderr:election_strategy: 3 2024-06-03T08:35:30.736 INFO:teuthology.orchestra.run.smithi192.stderr:0: v1:172.21.15.88:6789/0 mon.a 2024-06-03T08:35:30.736 INFO:teuthology.orchestra.run.smithi192.stderr:1: v1:172.21.15.192:6789/0 mon.b 2024-06-03T08:35:30.736 INFO:teuthology.orchestra.run.smithi192.stderr:2: v1:172.21.15.88:6790/0 mon.c 2024-06-03T08:35:30.736 INFO:teuthology.orchestra.run.smithi192.stderr: 2024-06-03T08:35:30.736 INFO:teuthology.orchestra.run.smithi192.stderr:2024-06-03T08:35:30.734+0000 7fb9e9ffb640 1 -- 172.21.15.192:0/4041815792 <== mon.1 v1:172.21.15.192:6789/0 7 ==== mgrmap(e 1) v1 ==== 668+0+0 (unknown 3836259515 0 0) 0x7fb9e4005200 con 0x7fb9f413e6c0 2024-06-03T08:35:30.736 INFO:teuthology.orchestra.run.smithi192.stderr:2024-06-03T08:35:30.734+0000 7fb9eb7fe640 4 set_mon_vals no callback set 2024-06-03T08:35:30.736 INFO:teuthology.orchestra.run.smithi192.stderr:2024-06-03T08:35:30.734+0000 7fb9e9ffb640 1 -- 172.21.15.192:0/4041815792 <== mon.1 v1:172.21.15.192:6789/0 8 ==== osd_map(6..6 src has 1..6) v4 ==== 1185+0+0 (unknown 4285659265 0 0) 0x7fb9e4006520 con 0x7fb9f413e6c0 2024-06-03T08:35:30.736 INFO:teuthology.orchestra.run.smithi192.stderr:2024-06-03T08:35:30.734+0000 7fb9e9ffb640 10 client.4144.objecter ms_dispatch 0x7fb9f4000b90 osd_map(6..6 src has 1..6) v4 2024-06-03T08:35:30.736 INFO:teuthology.orchestra.run.smithi192.stderr:2024-06-03T08:35:30.734+0000 7fb9e9ffb640 3 client.4144.objecter handle_osd_map got epochs [6,6] > 0 2024-06-03T08:35:30.736 INFO:teuthology.orchestra.run.smithi192.stderr:2024-06-03T08:35:30.734+0000 7fb9e9ffb640 3 client.4144.objecter handle_osd_map decoding full epoch 6 2024-06-03T08:35:30.736 INFO:teuthology.orchestra.run.smithi192.stderr:2024-06-03T08:35:30.734+0000 7fb9e9ffb640 20 client.4144.objecter dump_active .. 0 homeless 2024-06-03T08:35:30.776 INFO:tasks.ceph.mgr.x.smithi192.stderr:2024-06-03T08:35:30.776+0000 7f0e6be43040 -1 mgr[py] Module crash has missing NOTIFY_TYPES member 2024-06-03T08:35:30.844 INFO:tasks.ceph.mgr.y.smithi088.stderr:2024-06-03T08:35:30.844+0000 7ff0aea1d040 -1 mgr[py] Module volumes has missing NOTIFY_TYPES member 2024-06-03T08:35:30.895 INFO:tasks.ceph.mgr.x.smithi192.stderr:2024-06-03T08:35:30.895+0000 7f0e6be43040 -1 mgr[py] Module rgw has missing NOTIFY_TYPES member 2024-06-03T08:35:30.897 INFO:tasks.ceph.mgr.y.smithi088.stderr:2024-06-03T08:35:30.897+0000 7ff0aea1d040 -1 mgr[py] Module crash has missing NOTIFY_TYPES member 2024-06-03T08:35:30.906 INFO:teuthology.orchestra.run.smithi192.stderr:2024-06-03T08:35:30.906+0000 7fb9e9ffb640 1 -- 172.21.15.192:0/4041815792 <== mon.1 v1:172.21.15.192:6789/0 9 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+73121 (unknown 1092875540 0 1417653156) 0x7fb9e40060c0 con 0x7fb9f413e6c0 2024-06-03T08:35:30.906 INFO:teuthology.orchestra.run.smithi192.stderr:2024-06-03T08:35:30.906+0000 7fb9e9ffb640 10 monclient: handle_mon_command_ack 1 [{"prefix": "get_command_descriptions"}] 2024-06-03T08:35:30.906 INFO:teuthology.orchestra.run.smithi192.stderr:2024-06-03T08:35:30.906+0000 7fb9e9ffb640 10 monclient: _finish_command 1 = system:0 2024-06-03T08:35:30.997 INFO:teuthology.orchestra.run.smithi192.stderr:2024-06-03T08:35:30.997+0000 7fb9fb677640 10 start_mon_command cmd=[{"prefix": "osd new", "uuid": "f001b3cb-2081-428b-bac3-b3df3c19b978", "id": 4}] 2024-06-03T08:35:30.998 INFO:teuthology.orchestra.run.smithi192.stderr:2024-06-03T08:35:30.997+0000 7fb9fb677640 10 monclient: _send_command 2 [{"prefix": "osd new", "uuid": "f001b3cb-2081-428b-bac3-b3df3c19b978", "id": 4}] 2024-06-03T08:35:30.998 INFO:teuthology.orchestra.run.smithi192.stderr:2024-06-03T08:35:30.997+0000 7fb9fb677640 10 monclient: _send_mon_message to mon.b at v1:172.21.15.192:6789/0 2024-06-03T08:35:30.998 INFO:teuthology.orchestra.run.smithi192.stderr:2024-06-03T08:35:30.997+0000 7fb9fb677640 1 -- 172.21.15.192:0/4041815792 --> v1:172.21.15.192:6789/0 -- mon_command({"prefix": "osd new", "uuid": "f001b3cb-2081-428b-bac3-b3df3c19b978", "id": 4} v 0) v1 -- 0x7fb9f4142e10 con 0x7fb9f413e6c0 2024-06-03T08:35:31.017 INFO:tasks.ceph.mgr.y.smithi088.stderr:2024-06-03T08:35:31.017+0000 7ff0aea1d040 -1 mgr[py] Module rgw has missing NOTIFY_TYPES member 2024-06-03T08:35:31.107 INFO:teuthology.orchestra.run.smithi192.stderr:2024-06-03T08:35:31.106+0000 7fb9e9ffb640 1 -- 172.21.15.192:0/4041815792 <== mon.1 v1:172.21.15.192:6789/0 10 ==== mon_command_ack([{"prefix": "osd new", "uuid": "f001b3cb-2081-428b-bac3-b3df3c19b978", "id": 4}]=0 v7) v1 ==== 112+0+2 (unknown 4285746601 0 1697012583) 0x7fb9e4006370 con 0x7fb9f413e6c0 2024-06-03T08:35:31.107 INFO:teuthology.orchestra.run.smithi192.stderr:2024-06-03T08:35:31.106+0000 7fb9e9ffb640 10 monclient: handle_mon_command_ack 2 [{"prefix": "osd new", "uuid": "f001b3cb-2081-428b-bac3-b3df3c19b978", "id": 4}] 2024-06-03T08:35:31.107 INFO:teuthology.orchestra.run.smithi192.stderr:2024-06-03T08:35:31.106+0000 7fb9e9ffb640 10 monclient: _finish_command 2 = system:0 2024-06-03T08:35:31.107 INFO:teuthology.orchestra.run.smithi192.stdout:4 2024-06-03T08:35:31.108 INFO:teuthology.orchestra.run.smithi192.stderr:2024-06-03T08:35:31.107+0000 7fb9fb677640 10 librados: watch_flush enter 2024-06-03T08:35:31.108 INFO:teuthology.orchestra.run.smithi192.stderr:2024-06-03T08:35:31.107+0000 7fb9fb677640 10 librados: watch_flush exit 2024-06-03T08:35:31.109 INFO:teuthology.orchestra.run.smithi192.stderr:2024-06-03T08:35:31.107+0000 7fb9fb677640 20 client.4144.objecter shutdown clearing up homeless session... 2024-06-03T08:35:31.109 INFO:teuthology.orchestra.run.smithi192.stderr:2024-06-03T08:35:31.107+0000 7fb9fb677640 10 client.4144.objecter successfully canceled tick 2024-06-03T08:35:31.109 INFO:teuthology.orchestra.run.smithi192.stderr:2024-06-03T08:35:31.107+0000 7fb9fb677640 10 monclient: shutdown 2024-06-03T08:35:31.109 INFO:teuthology.orchestra.run.smithi192.stderr:2024-06-03T08:35:31.107+0000 7fb9fb677640 20 monclient: shutdown discarding 0 pending message(s) 2024-06-03T08:35:31.109 INFO:teuthology.orchestra.run.smithi192.stderr:2024-06-03T08:35:31.107+0000 7fb9fb677640 1 -- 172.21.15.192:0/4041815792 >> v1:172.21.15.192:6789/0 conn(0x7fb9f413e6c0 legacy=0x7fb9f4133fa0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-06-03T08:35:31.109 INFO:teuthology.orchestra.run.smithi192.stderr:2024-06-03T08:35:31.107+0000 7fb9fb677640 1 -- 172.21.15.192:0/4041815792 shutdown_connections 2024-06-03T08:35:31.109 INFO:teuthology.orchestra.run.smithi192.stderr:2024-06-03T08:35:31.107+0000 7fb9fb677640 1 -- 172.21.15.192:0/4041815792 >> 172.21.15.192:0/4041815792 conn(0x7fb9f40719d0 msgr2=0x7fb9f40b1a70 unknown :-1 s=STATE_NONE l=0).mark_down 2024-06-03T08:35:31.109 INFO:teuthology.orchestra.run.smithi192.stderr:2024-06-03T08:35:31.107+0000 7fb9fb677640 1 -- 172.21.15.192:0/4041815792 shutdown_connections 2024-06-03T08:35:31.109 INFO:teuthology.orchestra.run.smithi192.stderr:2024-06-03T08:35:31.108+0000 7fb9fb677640 1 -- 172.21.15.192:0/4041815792 wait complete. 2024-06-03T08:35:31.109 INFO:teuthology.orchestra.run.smithi192.stderr:2024-06-03T08:35:31.108+0000 7fb9fb677640 1 librados: shutdown 2024-06-03T08:35:31.118 DEBUG:teuthology.orchestra.run.smithi192:> sudo ceph --cluster ceph osd new 3cc263e9-e9bf-45e0-a3b7-83a76469eb70 5 2024-06-03T08:35:31.200 INFO:tasks.ceph.mgr.x.smithi192.stderr:2024-06-03T08:35:31.200+0000 7f0e6be43040 -1 mgr[py] Module prometheus has missing NOTIFY_TYPES member 2024-06-03T08:35:31.215 INFO:teuthology.orchestra.run.smithi192.stderr:2024-06-03T08:35:31.215+0000 7f781c5eb640 10 monclient: get_monmap_and_config 2024-06-03T08:35:31.215 INFO:teuthology.orchestra.run.smithi192.stderr:2024-06-03T08:35:31.215+0000 7f781c5eb640 10 monclient: build_initial_monmap 2024-06-03T08:35:31.215 INFO:teuthology.orchestra.run.smithi192.stderr:2024-06-03T08:35:31.215+0000 7f781c5eb640 10 monclient: monmap: 2024-06-03T08:35:31.215 INFO:teuthology.orchestra.run.smithi192.stderr:epoch 0 2024-06-03T08:35:31.215 INFO:teuthology.orchestra.run.smithi192.stderr:fsid ec3c3fdd-3ac6-4d7d-9412-683130ed8b04 2024-06-03T08:35:31.215 INFO:teuthology.orchestra.run.smithi192.stderr:last_changed 2024-06-03T08:35:31.215913+0000 2024-06-03T08:35:31.215 INFO:teuthology.orchestra.run.smithi192.stderr:created 2024-06-03T08:35:31.215913+0000 2024-06-03T08:35:31.216 INFO:teuthology.orchestra.run.smithi192.stderr:min_mon_release 0 (unknown) 2024-06-03T08:35:31.216 INFO:teuthology.orchestra.run.smithi192.stderr:election_strategy: 3 2024-06-03T08:35:31.216 INFO:teuthology.orchestra.run.smithi192.stderr:0: v1:172.21.15.88:6789/0 mon.noname-a 2024-06-03T08:35:31.216 INFO:teuthology.orchestra.run.smithi192.stderr:1: v1:172.21.15.192:6789/0 mon.noname-c 2024-06-03T08:35:31.216 INFO:teuthology.orchestra.run.smithi192.stderr:2: v1:172.21.15.88:6790/0 mon.noname-b 2024-06-03T08:35:31.216 INFO:teuthology.orchestra.run.smithi192.stderr: 2024-06-03T08:35:31.217 INFO:teuthology.orchestra.run.smithi192.stderr:2024-06-03T08:35:31.215+0000 7f781c5eb640 1 Processor -- start 2024-06-03T08:35:31.217 INFO:teuthology.orchestra.run.smithi192.stderr:2024-06-03T08:35:31.215+0000 7f781c5eb640 1 -- start start 2024-06-03T08:35:31.217 INFO:teuthology.orchestra.run.smithi192.stderr:2024-06-03T08:35:31.215+0000 7f781c5eb640 10 monclient: init 2024-06-03T08:35:31.217 INFO:teuthology.orchestra.run.smithi192.stderr:2024-06-03T08:35:31.216+0000 7f781c5eb640 10 monclient: _reopen_session rank -1 2024-06-03T08:35:31.218 INFO:teuthology.orchestra.run.smithi192.stderr:2024-06-03T08:35:31.216+0000 7f781c5eb640 10 monclient: _add_conns ranks=[1,0,2] 2024-06-03T08:35:31.218 INFO:teuthology.orchestra.run.smithi192.stderr:2024-06-03T08:35:31.216+0000 7f781c5eb640 10 monclient(hunting): picked mon.noname-c con 0x7f78140a8960 addr v1:172.21.15.192:6789/0 2024-06-03T08:35:31.218 INFO:teuthology.orchestra.run.smithi192.stderr:2024-06-03T08:35:31.216+0000 7f781c5eb640 10 monclient(hunting): picked mon.noname-a con 0x7f78141450f0 addr v1:172.21.15.88:6789/0 2024-06-03T08:35:31.218 INFO:teuthology.orchestra.run.smithi192.stderr:2024-06-03T08:35:31.216+0000 7f781c5eb640 10 monclient(hunting): picked mon.noname-b con 0x7f781413eec0 addr v1:172.21.15.88:6790/0 2024-06-03T08:35:31.218 INFO:teuthology.orchestra.run.smithi192.stderr:2024-06-03T08:35:31.216+0000 7f781c5eb640 1 -- --> v1:172.21.15.88:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f7814144c60 con 0x7f78141450f0 2024-06-03T08:35:31.218 INFO:teuthology.orchestra.run.smithi192.stderr:2024-06-03T08:35:31.216+0000 7f781c5eb640 1 -- --> v1:172.21.15.192:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f78140b3930 con 0x7f78140a8960 2024-06-03T08:35:31.218 INFO:teuthology.orchestra.run.smithi192.stderr:2024-06-03T08:35:31.216+0000 7f781c5eb640 1 -- --> v1:172.21.15.88:6790/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f78140b1ad0 con 0x7f781413eec0 2024-06-03T08:35:31.218 INFO:teuthology.orchestra.run.smithi192.stderr:2024-06-03T08:35:31.216+0000 7f781c5eb640 10 monclient(hunting): _renew_subs 2024-06-03T08:35:31.218 INFO:teuthology.orchestra.run.smithi192.stderr:2024-06-03T08:35:31.216+0000 7f781c5eb640 10 monclient(hunting): authenticate will time out at 1197.027525s 2024-06-03T08:35:31.218 INFO:teuthology.orchestra.run.smithi192.stderr:2024-06-03T08:35:31.216+0000 7f781a387640 1 --1- >> v1:172.21.15.192:6789/0 conn(0x7f78140a8960 0x7f78140a8d40 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.192:6789/0 says I am v1:172.21.15.192:43882/0 (socket says 172.21.15.192:43882) 2024-06-03T08:35:31.218 INFO:teuthology.orchestra.run.smithi192.stderr:2024-06-03T08:35:31.216+0000 7f781a387640 1 -- 172.21.15.192:0/2000380941 learned_addr learned my addr 172.21.15.192:0/2000380941 (peer_addr_for_me v1:172.21.15.192:0/0) 2024-06-03T08:35:31.219 INFO:teuthology.orchestra.run.smithi192.stderr:2024-06-03T08:35:31.219+0000 7f7819385640 1 -- 172.21.15.192:0/2000380941 <== mon.1 v1:172.21.15.192:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 3031644543 0 0) 0x7f78140b3930 con 0x7f78140a8960 2024-06-03T08:35:31.219 INFO:teuthology.orchestra.run.smithi192.stderr:2024-06-03T08:35:31.219+0000 7f7819385640 10 monclient(hunting): _negotiate 2024-06-03T08:35:31.219 INFO:teuthology.orchestra.run.smithi192.stderr:2024-06-03T08:35:31.219+0000 7f7819385640 10 monclient(hunting): _init_auth method 2 2024-06-03T08:35:31.219 INFO:teuthology.orchestra.run.smithi192.stderr:2024-06-03T08:35:31.219+0000 7f7819385640 10 monclient(hunting): _init_auth creating new auth 2024-06-03T08:35:31.219 INFO:teuthology.orchestra.run.smithi192.stderr:2024-06-03T08:35:31.219+0000 7f7819385640 10 monclient(hunting): my global_id is 4147 2024-06-03T08:35:31.220 INFO:teuthology.orchestra.run.smithi192.stderr:2024-06-03T08:35:31.219+0000 7f7819385640 1 -- 172.21.15.192:0/2000380941 --> v1:172.21.15.192:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f7800003520 con 0x7f78140a8960 2024-06-03T08:35:31.220 INFO:teuthology.orchestra.run.smithi192.stderr:2024-06-03T08:35:31.219+0000 7f7819385640 1 -- 172.21.15.192:0/2000380941 <== mon.2 v1:172.21.15.88:6790/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 3346844098 0 0) 0x7f78140b1ad0 con 0x7f781413eec0 2024-06-03T08:35:31.220 INFO:teuthology.orchestra.run.smithi192.stderr:2024-06-03T08:35:31.219+0000 7f7819385640 10 monclient(hunting): _negotiate 2024-06-03T08:35:31.220 INFO:teuthology.orchestra.run.smithi192.stderr:2024-06-03T08:35:31.219+0000 7f7819385640 10 monclient(hunting): _init_auth method 2 2024-06-03T08:35:31.220 INFO:teuthology.orchestra.run.smithi192.stderr:2024-06-03T08:35:31.219+0000 7f7819385640 10 monclient(hunting): _init_auth creating new auth 2024-06-03T08:35:31.220 INFO:teuthology.orchestra.run.smithi192.stderr:2024-06-03T08:35:31.219+0000 7f7819385640 10 monclient(hunting): my global_id is 4148 2024-06-03T08:35:31.220 INFO:teuthology.orchestra.run.smithi192.stderr:2024-06-03T08:35:31.219+0000 7f7819385640 1 -- 172.21.15.192:0/2000380941 --> v1:172.21.15.88:6790/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f78140b3930 con 0x7f781413eec0 2024-06-03T08:35:31.220 INFO:teuthology.orchestra.run.smithi192.stderr:2024-06-03T08:35:31.219+0000 7f7819385640 1 -- 172.21.15.192:0/2000380941 <== mon.0 v1:172.21.15.88:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 1481866157 0 0) 0x7f7814144c60 con 0x7f78141450f0 2024-06-03T08:35:31.221 INFO:teuthology.orchestra.run.smithi192.stderr:2024-06-03T08:35:31.219+0000 7f7819385640 10 monclient(hunting): _negotiate 2024-06-03T08:35:31.221 INFO:teuthology.orchestra.run.smithi192.stderr:2024-06-03T08:35:31.219+0000 7f7819385640 10 monclient(hunting): _init_auth method 2 2024-06-03T08:35:31.221 INFO:teuthology.orchestra.run.smithi192.stderr:2024-06-03T08:35:31.219+0000 7f7819385640 10 monclient(hunting): _init_auth creating new auth 2024-06-03T08:35:31.221 INFO:teuthology.orchestra.run.smithi192.stderr:2024-06-03T08:35:31.219+0000 7f7819385640 10 monclient(hunting): my global_id is 4146 2024-06-03T08:35:31.221 INFO:teuthology.orchestra.run.smithi192.stderr:2024-06-03T08:35:31.219+0000 7f7819385640 1 -- 172.21.15.192:0/2000380941 --> v1:172.21.15.88:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f78140b1ad0 con 0x7f78141450f0 2024-06-03T08:35:31.223 INFO:teuthology.orchestra.run.smithi192.stderr:2024-06-03T08:35:31.222+0000 7f7819385640 1 -- 172.21.15.192:0/2000380941 <== mon.1 v1:172.21.15.192:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 214+0+0 (unknown 402187267 0 0) 0x7f7800003520 con 0x7f78140a8960 2024-06-03T08:35:31.223 INFO:teuthology.orchestra.run.smithi192.stderr:2024-06-03T08:35:31.222+0000 7f7819385640 1 -- 172.21.15.192:0/2000380941 >> v1:172.21.15.88:6790/0 conn(0x7f781413eec0 legacy=0x7f781414ce60 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-06-03T08:35:31.223 INFO:teuthology.orchestra.run.smithi192.stderr:2024-06-03T08:35:31.222+0000 7f7819385640 1 -- 172.21.15.192:0/2000380941 >> v1:172.21.15.88:6789/0 conn(0x7f78141450f0 legacy=0x7f781413e7b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-06-03T08:35:31.223 INFO:teuthology.orchestra.run.smithi192.stderr:2024-06-03T08:35:31.222+0000 7f7819385640 10 monclient: _finish_hunting 0 2024-06-03T08:35:31.223 INFO:teuthology.orchestra.run.smithi192.stderr:2024-06-03T08:35:31.222+0000 7f7819385640 1 monclient: found mon.noname-c 2024-06-03T08:35:31.223 INFO:teuthology.orchestra.run.smithi192.stderr:2024-06-03T08:35:31.222+0000 7f7819385640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-06-03T08:35:31.223 INFO:teuthology.orchestra.run.smithi192.stderr:2024-06-03T08:35:31.222+0000 7f7819385640 10 monclient: _send_mon_message to mon.noname-c at v1:172.21.15.192:6789/0 2024-06-03T08:35:31.223 INFO:teuthology.orchestra.run.smithi192.stderr:2024-06-03T08:35:31.222+0000 7f7819385640 1 -- 172.21.15.192:0/2000380941 --> v1:172.21.15.192:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f78140b1c60 con 0x7f78140a8960 2024-06-03T08:35:31.223 INFO:teuthology.orchestra.run.smithi192.stderr:2024-06-03T08:35:31.222+0000 7f7819385640 10 monclient: _finish_auth 0 2024-06-03T08:35:31.223 INFO:teuthology.orchestra.run.smithi192.stderr:2024-06-03T08:35:31.222+0000 7f7819385640 10 monclient: _check_auth_tickets 2024-06-03T08:35:31.223 INFO:teuthology.orchestra.run.smithi192.stderr:2024-06-03T08:35:31.222+0000 7f7819385640 20 monclient: _check_auth_rotating not needed by client.admin 2024-06-03T08:35:31.223 INFO:teuthology.orchestra.run.smithi192.stderr:2024-06-03T08:35:31.222+0000 7f7819385640 1 -- 172.21.15.192:0/2000380941 <== mon.1 v1:172.21.15.192:6789/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 2234374065 0 0) 0x7f7810001610 con 0x7f78140a8960 2024-06-03T08:35:31.224 INFO:teuthology.orchestra.run.smithi192.stderr:2024-06-03T08:35:31.222+0000 7f7819385640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-06-03T08:35:31.224 INFO:teuthology.orchestra.run.smithi192.stderr:2024-06-03T08:35:31.222+0000 7f7819385640 10 monclient: got monmap 1 from mon.noname-c (according to old e1) 2024-06-03T08:35:31.224 INFO:teuthology.orchestra.run.smithi192.stderr:2024-06-03T08:35:31.222+0000 7f7819385640 10 monclient: dump: 2024-06-03T08:35:31.224 INFO:teuthology.orchestra.run.smithi192.stderr:epoch 1 2024-06-03T08:35:31.224 INFO:teuthology.orchestra.run.smithi192.stderr:fsid ec3c3fdd-3ac6-4d7d-9412-683130ed8b04 2024-06-03T08:35:31.224 INFO:teuthology.orchestra.run.smithi192.stderr:last_changed 2024-06-03T08:34:31.885125+0000 2024-06-03T08:35:31.224 INFO:teuthology.orchestra.run.smithi192.stderr:created 2024-06-03T08:34:31.885125+0000 2024-06-03T08:35:31.224 INFO:teuthology.orchestra.run.smithi192.stderr:min_mon_release 18 (reef) 2024-06-03T08:35:31.224 INFO:teuthology.orchestra.run.smithi192.stderr:election_strategy: 3 2024-06-03T08:35:31.224 INFO:teuthology.orchestra.run.smithi192.stderr:0: v1:172.21.15.88:6789/0 mon.a 2024-06-03T08:35:31.224 INFO:teuthology.orchestra.run.smithi192.stderr:1: v1:172.21.15.192:6789/0 mon.b 2024-06-03T08:35:31.224 INFO:teuthology.orchestra.run.smithi192.stderr:2: v1:172.21.15.88:6790/0 mon.c 2024-06-03T08:35:31.224 INFO:teuthology.orchestra.run.smithi192.stderr: 2024-06-03T08:35:31.225 INFO:teuthology.orchestra.run.smithi192.stderr:2024-06-03T08:35:31.222+0000 7f781c5eb640 5 monclient: authenticate success, global_id 4147 2024-06-03T08:35:31.225 INFO:teuthology.orchestra.run.smithi192.stderr:2024-06-03T08:35:31.222+0000 7f781c5eb640 20 monclient: get_monmap_and_config waiting for monmap|config 2024-06-03T08:35:31.231 INFO:teuthology.orchestra.run.smithi192.stderr:2024-06-03T08:35:31.224+0000 7f7819385640 1 -- 172.21.15.192:0/2000380941 <== mon.1 v1:172.21.15.192:6789/0 4 ==== config(0 keys) v1 ==== 4+0+0 (unknown 0 0 0) 0x7f7810001830 con 0x7f78140a8960 2024-06-03T08:35:31.231 INFO:teuthology.orchestra.run.smithi192.stderr:2024-06-03T08:35:31.224+0000 7f7819385640 10 monclient: handle_config config(0 keys) v1 2024-06-03T08:35:31.231 INFO:teuthology.orchestra.run.smithi192.stderr:2024-06-03T08:35:31.224+0000 7f7819385640 1 -- 172.21.15.192:0/2000380941 <== mon.1 v1:172.21.15.192:6789/0 5 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 2234374065 0 0) 0x7f7810001b40 con 0x7f78140a8960 2024-06-03T08:35:31.231 INFO:teuthology.orchestra.run.smithi192.stderr:2024-06-03T08:35:31.224+0000 7f781c5eb640 10 monclient: get_monmap_and_config success 2024-06-03T08:35:31.231 INFO:teuthology.orchestra.run.smithi192.stderr:2024-06-03T08:35:31.224+0000 7f7819385640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-06-03T08:35:31.231 INFO:teuthology.orchestra.run.smithi192.stderr:2024-06-03T08:35:31.224+0000 7f781c5eb640 4 set_mon_vals no callback set 2024-06-03T08:35:31.231 INFO:teuthology.orchestra.run.smithi192.stderr:2024-06-03T08:35:31.224+0000 7f7819385640 10 monclient: got monmap 1 from mon.b (according to old e1) 2024-06-03T08:35:31.232 INFO:teuthology.orchestra.run.smithi192.stderr:2024-06-03T08:35:31.224+0000 7f7819385640 10 monclient: dump: 2024-06-03T08:35:31.232 INFO:teuthology.orchestra.run.smithi192.stderr:epoch 1 2024-06-03T08:35:31.232 INFO:teuthology.orchestra.run.smithi192.stderr:fsid ec3c3fdd-3ac6-4d7d-9412-683130ed8b04 2024-06-03T08:35:31.232 INFO:teuthology.orchestra.run.smithi192.stderr:last_changed 2024-06-03T08:34:31.885125+0000 2024-06-03T08:35:31.232 INFO:teuthology.orchestra.run.smithi192.stderr:created 2024-06-03T08:34:31.885125+0000 2024-06-03T08:35:31.232 INFO:teuthology.orchestra.run.smithi192.stderr:min_mon_release 18 (reef) 2024-06-03T08:35:31.232 INFO:teuthology.orchestra.run.smithi192.stderr:election_strategy: 3 2024-06-03T08:35:31.232 INFO:teuthology.orchestra.run.smithi192.stderr:0: v1:172.21.15.88:6789/0 mon.a 2024-06-03T08:35:31.232 INFO:teuthology.orchestra.run.smithi192.stderr:1: v1:172.21.15.192:6789/0 mon.b 2024-06-03T08:35:31.232 INFO:teuthology.orchestra.run.smithi192.stderr:2: v1:172.21.15.88:6790/0 mon.c 2024-06-03T08:35:31.232 INFO:teuthology.orchestra.run.smithi192.stderr: 2024-06-03T08:35:31.232 INFO:teuthology.orchestra.run.smithi192.stderr:2024-06-03T08:35:31.224+0000 7f781c5eb640 10 monclient: shutdown 2024-06-03T08:35:31.232 INFO:teuthology.orchestra.run.smithi192.stderr:2024-06-03T08:35:31.224+0000 7f781c5eb640 20 monclient: shutdown discarding 0 pending message(s) 2024-06-03T08:35:31.232 INFO:teuthology.orchestra.run.smithi192.stderr:2024-06-03T08:35:31.224+0000 7f781c5eb640 1 -- 172.21.15.192:0/2000380941 >> v1:172.21.15.192:6789/0 conn(0x7f78140a8960 legacy=0x7f78140a8d40 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-06-03T08:35:31.233 INFO:teuthology.orchestra.run.smithi192.stderr:2024-06-03T08:35:31.224+0000 7f781c5eb640 1 -- 172.21.15.192:0/2000380941 shutdown_connections 2024-06-03T08:35:31.233 INFO:teuthology.orchestra.run.smithi192.stderr:2024-06-03T08:35:31.224+0000 7f781c5eb640 1 -- 172.21.15.192:0/2000380941 >> 172.21.15.192:0/2000380941 conn(0x7f78140b70d0 msgr2=0x7f78140b74e0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-06-03T08:35:31.233 INFO:teuthology.orchestra.run.smithi192.stderr:2024-06-03T08:35:31.225+0000 7f781c5eb640 1 -- 172.21.15.192:0/2000380941 shutdown_connections 2024-06-03T08:35:31.233 INFO:teuthology.orchestra.run.smithi192.stderr:2024-06-03T08:35:31.225+0000 7f781c5eb640 1 -- 172.21.15.192:0/2000380941 wait complete. 2024-06-03T08:35:31.233 INFO:teuthology.orchestra.run.smithi192.stderr:2024-06-03T08:35:31.225+0000 7f781c5eb640 10 monclient: build_initial_monmap 2024-06-03T08:35:31.233 INFO:teuthology.orchestra.run.smithi192.stderr:2024-06-03T08:35:31.225+0000 7f781c5eb640 10 monclient: monmap: 2024-06-03T08:35:31.233 INFO:teuthology.orchestra.run.smithi192.stderr:epoch 0 2024-06-03T08:35:31.233 INFO:teuthology.orchestra.run.smithi192.stderr:fsid 00000000-0000-0000-0000-000000000000 2024-06-03T08:35:31.233 INFO:teuthology.orchestra.run.smithi192.stderr:last_changed 0.000000 2024-06-03T08:35:31.233 INFO:teuthology.orchestra.run.smithi192.stderr:created 0.000000 2024-06-03T08:35:31.233 INFO:teuthology.orchestra.run.smithi192.stderr:min_mon_release 0 (unknown) 2024-06-03T08:35:31.233 INFO:teuthology.orchestra.run.smithi192.stderr:election_strategy: 1 2024-06-03T08:35:31.233 INFO:teuthology.orchestra.run.smithi192.stderr:0: v1:172.21.15.88:6789/0 mon.noname-a 2024-06-03T08:35:31.234 INFO:teuthology.orchestra.run.smithi192.stderr:1: v1:172.21.15.192:6789/0 mon.noname-b 2024-06-03T08:35:31.234 INFO:teuthology.orchestra.run.smithi192.stderr:2: v1:172.21.15.88:6790/0 mon.noname-c 2024-06-03T08:35:31.234 INFO:teuthology.orchestra.run.smithi192.stderr: 2024-06-03T08:35:31.234 INFO:teuthology.orchestra.run.smithi192.stderr:2024-06-03T08:35:31.225+0000 7f781c5eb640 1 librados: starting msgr at 2024-06-03T08:35:31.234 INFO:teuthology.orchestra.run.smithi192.stderr:2024-06-03T08:35:31.225+0000 7f781c5eb640 1 librados: starting objecter 2024-06-03T08:35:31.234 INFO:teuthology.orchestra.run.smithi192.stderr:2024-06-03T08:35:31.226+0000 7f781c5eb640 1 Processor -- start 2024-06-03T08:35:31.234 INFO:teuthology.orchestra.run.smithi192.stderr:2024-06-03T08:35:31.226+0000 7f781c5eb640 1 -- start start 2024-06-03T08:35:31.234 INFO:teuthology.orchestra.run.smithi192.stderr:2024-06-03T08:35:31.226+0000 7f781c5eb640 1 librados: setting wanted keys 2024-06-03T08:35:31.234 INFO:teuthology.orchestra.run.smithi192.stderr:2024-06-03T08:35:31.226+0000 7f781c5eb640 1 librados: calling monclient init 2024-06-03T08:35:31.234 INFO:teuthology.orchestra.run.smithi192.stderr:2024-06-03T08:35:31.226+0000 7f781c5eb640 10 monclient: init 2024-06-03T08:35:31.234 INFO:teuthology.orchestra.run.smithi192.stderr:2024-06-03T08:35:31.226+0000 7f781c5eb640 10 monclient: _reopen_session rank -1 2024-06-03T08:35:31.234 INFO:teuthology.orchestra.run.smithi192.stderr:2024-06-03T08:35:31.226+0000 7f781c5eb640 10 monclient: _add_conns ranks=[1,0,2] 2024-06-03T08:35:31.234 INFO:teuthology.orchestra.run.smithi192.stderr:2024-06-03T08:35:31.226+0000 7f781c5eb640 10 monclient(hunting): picked mon.noname-b con 0x7f78140a8960 addr v1:172.21.15.192:6789/0 2024-06-03T08:35:31.235 INFO:teuthology.orchestra.run.smithi192.stderr:2024-06-03T08:35:31.226+0000 7f781c5eb640 10 monclient(hunting): picked mon.noname-a con 0x7f781413eec0 addr v1:172.21.15.88:6789/0 2024-06-03T08:35:31.235 INFO:teuthology.orchestra.run.smithi192.stderr:2024-06-03T08:35:31.226+0000 7f781c5eb640 10 monclient(hunting): picked mon.noname-c con 0x7f78141450f0 addr v1:172.21.15.88:6790/0 2024-06-03T08:35:31.235 INFO:teuthology.orchestra.run.smithi192.stderr:2024-06-03T08:35:31.226+0000 7f781c5eb640 1 -- --> v1:172.21.15.88:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f78141d68e0 con 0x7f781413eec0 2024-06-03T08:35:31.235 INFO:teuthology.orchestra.run.smithi192.stderr:2024-06-03T08:35:31.226+0000 7f781c5eb640 1 -- --> v1:172.21.15.192:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f78140ac220 con 0x7f78140a8960 2024-06-03T08:35:31.235 INFO:teuthology.orchestra.run.smithi192.stderr:2024-06-03T08:35:31.226+0000 7f781c5eb640 1 -- --> v1:172.21.15.88:6790/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f78140ac3b0 con 0x7f78141450f0 2024-06-03T08:35:31.235 INFO:teuthology.orchestra.run.smithi192.stderr:2024-06-03T08:35:31.226+0000 7f781c5eb640 10 monclient(hunting): _renew_subs 2024-06-03T08:35:31.235 INFO:teuthology.orchestra.run.smithi192.stderr:2024-06-03T08:35:31.226+0000 7f781c5eb640 10 monclient(hunting): authenticate will time out at 1197.037737s 2024-06-03T08:35:31.235 INFO:teuthology.orchestra.run.smithi192.stderr:2024-06-03T08:35:31.226+0000 7f781a387640 1 --1- >> v1:172.21.15.192:6789/0 conn(0x7f78140a8960 0x7f78141d6e00 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=1).handle_server_banner_and_identify peer v1:172.21.15.192:6789/0 says I am v1:172.21.15.192:43892/0 (socket says 172.21.15.192:43892) 2024-06-03T08:35:31.235 INFO:teuthology.orchestra.run.smithi192.stderr:2024-06-03T08:35:31.226+0000 7f781a387640 1 -- 172.21.15.192:0/4039535999 learned_addr learned my addr 172.21.15.192:0/4039535999 (peer_addr_for_me v1:172.21.15.192:0/0) 2024-06-03T08:35:31.235 INFO:teuthology.orchestra.run.smithi192.stderr:2024-06-03T08:35:31.227+0000 7f780affd640 10 client.?.objecter ms_handle_connect 0x7f78140a8960 2024-06-03T08:35:31.235 INFO:teuthology.orchestra.run.smithi192.stderr:2024-06-03T08:35:31.227+0000 7f780affd640 10 client.?.objecter resend_mon_ops 2024-06-03T08:35:31.235 INFO:teuthology.orchestra.run.smithi192.stderr:2024-06-03T08:35:31.227+0000 7f780affd640 10 client.?.objecter ms_handle_connect 0x7f78141450f0 2024-06-03T08:35:31.235 INFO:teuthology.orchestra.run.smithi192.stderr:2024-06-03T08:35:31.227+0000 7f780affd640 10 client.?.objecter resend_mon_ops 2024-06-03T08:35:31.235 INFO:teuthology.orchestra.run.smithi192.stderr:2024-06-03T08:35:31.227+0000 7f780affd640 10 client.?.objecter ms_handle_connect 0x7f781413eec0 2024-06-03T08:35:31.236 INFO:teuthology.orchestra.run.smithi192.stderr:2024-06-03T08:35:31.227+0000 7f780affd640 10 client.?.objecter resend_mon_ops 2024-06-03T08:35:31.236 INFO:teuthology.orchestra.run.smithi192.stderr:2024-06-03T08:35:31.229+0000 7f780affd640 1 -- 172.21.15.192:0/4039535999 <== mon.1 v1:172.21.15.192:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 4165171281 0 0) 0x7f78140ac220 con 0x7f78140a8960 2024-06-03T08:35:31.236 INFO:teuthology.orchestra.run.smithi192.stderr:2024-06-03T08:35:31.229+0000 7f780affd640 10 monclient(hunting): _negotiate 2024-06-03T08:35:31.236 INFO:teuthology.orchestra.run.smithi192.stderr:2024-06-03T08:35:31.229+0000 7f780affd640 10 monclient(hunting): _init_auth method 2 2024-06-03T08:35:31.236 INFO:teuthology.orchestra.run.smithi192.stderr:2024-06-03T08:35:31.229+0000 7f780affd640 10 monclient(hunting): _init_auth creating new auth 2024-06-03T08:35:31.236 INFO:teuthology.orchestra.run.smithi192.stderr:2024-06-03T08:35:31.229+0000 7f780affd640 10 monclient(hunting): my global_id is 4150 2024-06-03T08:35:31.236 INFO:teuthology.orchestra.run.smithi192.stderr:2024-06-03T08:35:31.229+0000 7f780affd640 1 -- 172.21.15.192:0/4039535999 --> v1:172.21.15.192:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f77f8004eb0 con 0x7f78140a8960 2024-06-03T08:35:31.236 INFO:teuthology.orchestra.run.smithi192.stderr:2024-06-03T08:35:31.229+0000 7f780affd640 1 -- 172.21.15.192:0/4039535999 <== mon.2 v1:172.21.15.88:6790/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 1597981418 0 0) 0x7f78140ac3b0 con 0x7f78141450f0 2024-06-03T08:35:31.236 INFO:teuthology.orchestra.run.smithi192.stderr:2024-06-03T08:35:31.229+0000 7f780affd640 10 monclient(hunting): _negotiate 2024-06-03T08:35:31.236 INFO:teuthology.orchestra.run.smithi192.stderr:2024-06-03T08:35:31.229+0000 7f780affd640 10 monclient(hunting): _init_auth method 2 2024-06-03T08:35:31.236 INFO:teuthology.orchestra.run.smithi192.stderr:2024-06-03T08:35:31.229+0000 7f780affd640 10 monclient(hunting): _init_auth creating new auth 2024-06-03T08:35:31.236 INFO:teuthology.orchestra.run.smithi192.stderr:2024-06-03T08:35:31.229+0000 7f780affd640 10 monclient(hunting): my global_id is 4151 2024-06-03T08:35:31.236 INFO:teuthology.orchestra.run.smithi192.stderr:2024-06-03T08:35:31.229+0000 7f780affd640 1 -- 172.21.15.192:0/4039535999 --> v1:172.21.15.88:6790/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f78140ac220 con 0x7f78141450f0 2024-06-03T08:35:31.236 INFO:teuthology.orchestra.run.smithi192.stderr:2024-06-03T08:35:31.229+0000 7f780affd640 1 -- 172.21.15.192:0/4039535999 <== mon.0 v1:172.21.15.88:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 1797169109 0 0) 0x7f78141d68e0 con 0x7f781413eec0 2024-06-03T08:35:31.237 INFO:teuthology.orchestra.run.smithi192.stderr:2024-06-03T08:35:31.229+0000 7f780affd640 10 monclient(hunting): _negotiate 2024-06-03T08:35:31.237 INFO:teuthology.orchestra.run.smithi192.stderr:2024-06-03T08:35:31.229+0000 7f780affd640 10 monclient(hunting): _init_auth method 2 2024-06-03T08:35:31.237 INFO:teuthology.orchestra.run.smithi192.stderr:2024-06-03T08:35:31.229+0000 7f780affd640 10 monclient(hunting): _init_auth creating new auth 2024-06-03T08:35:31.237 INFO:teuthology.orchestra.run.smithi192.stderr:2024-06-03T08:35:31.229+0000 7f780affd640 10 monclient(hunting): my global_id is 4149 2024-06-03T08:35:31.237 INFO:teuthology.orchestra.run.smithi192.stderr:2024-06-03T08:35:31.229+0000 7f780affd640 1 -- 172.21.15.192:0/4039535999 --> v1:172.21.15.88:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f78140ac3b0 con 0x7f781413eec0 2024-06-03T08:35:31.239 INFO:teuthology.orchestra.run.smithi192.stderr:2024-06-03T08:35:31.233+0000 7f780affd640 1 -- 172.21.15.192:0/4039535999 <== mon.1 v1:172.21.15.192:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 764+0+0 (unknown 232417757 0 0) 0x7f77f8004eb0 con 0x7f78140a8960 2024-06-03T08:35:31.239 INFO:teuthology.orchestra.run.smithi192.stderr:2024-06-03T08:35:31.234+0000 7f780affd640 1 -- 172.21.15.192:0/4039535999 --> v1:172.21.15.192:6789/0 -- auth(proto 2 165 bytes epoch 0) v1 -- 0x7f78141d68e0 con 0x7f78140a8960 2024-06-03T08:35:31.239 INFO:teuthology.orchestra.run.smithi192.stderr:2024-06-03T08:35:31.234+0000 7f780affd640 1 -- 172.21.15.192:0/4039535999 <== mon.1 v1:172.21.15.192:6789/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 2234374065 0 0) 0x7f7810001510 con 0x7f78140a8960 2024-06-03T08:35:31.239 INFO:teuthology.orchestra.run.smithi192.stderr:2024-06-03T08:35:31.234+0000 7f780affd640 10 monclient(hunting): handle_monmap mon_map magic: 0 v1 2024-06-03T08:35:31.239 INFO:teuthology.orchestra.run.smithi192.stderr:2024-06-03T08:35:31.234+0000 7f780affd640 10 monclient(hunting): got monmap 1 from mon.noname-b (according to old e1) 2024-06-03T08:35:31.239 INFO:teuthology.orchestra.run.smithi192.stderr:2024-06-03T08:35:31.234+0000 7f780affd640 10 monclient(hunting): dump: 2024-06-03T08:35:31.239 INFO:teuthology.orchestra.run.smithi192.stderr:epoch 1 2024-06-03T08:35:31.239 INFO:teuthology.orchestra.run.smithi192.stderr:fsid ec3c3fdd-3ac6-4d7d-9412-683130ed8b04 2024-06-03T08:35:31.239 INFO:teuthology.orchestra.run.smithi192.stderr:last_changed 2024-06-03T08:34:31.885125+0000 2024-06-03T08:35:31.239 INFO:teuthology.orchestra.run.smithi192.stderr:created 2024-06-03T08:34:31.885125+0000 2024-06-03T08:35:31.239 INFO:teuthology.orchestra.run.smithi192.stderr:min_mon_release 18 (reef) 2024-06-03T08:35:31.240 INFO:teuthology.orchestra.run.smithi192.stderr:election_strategy: 3 2024-06-03T08:35:31.240 INFO:teuthology.orchestra.run.smithi192.stderr:0: v1:172.21.15.88:6789/0 mon.a 2024-06-03T08:35:31.240 INFO:teuthology.orchestra.run.smithi192.stderr:1: v1:172.21.15.192:6789/0 mon.b 2024-06-03T08:35:31.240 INFO:teuthology.orchestra.run.smithi192.stderr:2: v1:172.21.15.88:6790/0 mon.c 2024-06-03T08:35:31.240 INFO:teuthology.orchestra.run.smithi192.stderr: 2024-06-03T08:35:31.240 INFO:teuthology.orchestra.run.smithi192.stderr:2024-06-03T08:35:31.234+0000 7f780affd640 10 monclient(hunting): _finish_auth 0 2024-06-03T08:35:31.240 INFO:teuthology.orchestra.run.smithi192.stderr:2024-06-03T08:35:31.234+0000 7f780affd640 1 -- 172.21.15.192:0/4039535999 <== mon.2 v1:172.21.15.88:6790/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 764+0+0 (unknown 2921157363 0 0) 0x7f78140ac220 con 0x7f78141450f0 2024-06-03T08:35:31.240 INFO:teuthology.orchestra.run.smithi192.stderr:2024-06-03T08:35:31.234+0000 7f780affd640 1 -- 172.21.15.192:0/4039535999 --> v1:172.21.15.88:6790/0 -- auth(proto 2 165 bytes epoch 0) v1 -- 0x7f77f8004eb0 con 0x7f78141450f0 2024-06-03T08:35:31.240 INFO:teuthology.orchestra.run.smithi192.stderr:2024-06-03T08:35:31.234+0000 7f780affd640 1 -- 172.21.15.192:0/4039535999 <== mon.2 v1:172.21.15.88:6790/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 2234374065 0 0) 0x7f780c001bb0 con 0x7f78141450f0 2024-06-03T08:35:31.240 INFO:teuthology.orchestra.run.smithi192.stderr:2024-06-03T08:35:31.234+0000 7f780affd640 10 monclient(hunting): handle_monmap mon_map magic: 0 v1 2024-06-03T08:35:31.240 INFO:teuthology.orchestra.run.smithi192.stderr:2024-06-03T08:35:31.234+0000 7f780affd640 10 monclient(hunting): got monmap 1 from mon.c (according to old e1) 2024-06-03T08:35:31.240 INFO:teuthology.orchestra.run.smithi192.stderr:2024-06-03T08:35:31.234+0000 7f780affd640 10 monclient(hunting): dump: 2024-06-03T08:35:31.240 INFO:teuthology.orchestra.run.smithi192.stderr:epoch 1 2024-06-03T08:35:31.241 INFO:teuthology.orchestra.run.smithi192.stderr:fsid ec3c3fdd-3ac6-4d7d-9412-683130ed8b04 2024-06-03T08:35:31.241 INFO:teuthology.orchestra.run.smithi192.stderr:last_changed 2024-06-03T08:34:31.885125+0000 2024-06-03T08:35:31.241 INFO:teuthology.orchestra.run.smithi192.stderr:created 2024-06-03T08:34:31.885125+0000 2024-06-03T08:35:31.241 INFO:teuthology.orchestra.run.smithi192.stderr:min_mon_release 18 (reef) 2024-06-03T08:35:31.241 INFO:teuthology.orchestra.run.smithi192.stderr:election_strategy: 3 2024-06-03T08:35:31.241 INFO:teuthology.orchestra.run.smithi192.stderr:0: v1:172.21.15.88:6789/0 mon.a 2024-06-03T08:35:31.241 INFO:teuthology.orchestra.run.smithi192.stderr:1: v1:172.21.15.192:6789/0 mon.b 2024-06-03T08:35:31.241 INFO:teuthology.orchestra.run.smithi192.stderr:2: v1:172.21.15.88:6790/0 mon.c 2024-06-03T08:35:31.241 INFO:teuthology.orchestra.run.smithi192.stderr: 2024-06-03T08:35:31.241 INFO:teuthology.orchestra.run.smithi192.stderr:2024-06-03T08:35:31.234+0000 7f780affd640 1 -- 172.21.15.192:0/4039535999 <== mon.0 v1:172.21.15.88:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 764+0+0 (unknown 2292747746 0 0) 0x7f78140ac3b0 con 0x7f781413eec0 2024-06-03T08:35:31.241 INFO:teuthology.orchestra.run.smithi192.stderr:2024-06-03T08:35:31.234+0000 7f780affd640 1 -- 172.21.15.192:0/4039535999 --> v1:172.21.15.88:6789/0 -- auth(proto 2 165 bytes epoch 0) v1 -- 0x7f78140ac220 con 0x7f781413eec0 2024-06-03T08:35:31.241 INFO:teuthology.orchestra.run.smithi192.stderr:2024-06-03T08:35:31.234+0000 7f780affd640 1 -- 172.21.15.192:0/4039535999 <== mon.0 v1:172.21.15.88:6789/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 2234374065 0 0) 0x7f7804001810 con 0x7f781413eec0 2024-06-03T08:35:31.241 INFO:teuthology.orchestra.run.smithi192.stderr:2024-06-03T08:35:31.234+0000 7f780affd640 10 monclient(hunting): handle_monmap mon_map magic: 0 v1 2024-06-03T08:35:31.241 INFO:teuthology.orchestra.run.smithi192.stderr:2024-06-03T08:35:31.234+0000 7f780affd640 10 monclient(hunting): got monmap 1 from mon.a (according to old e1) 2024-06-03T08:35:31.242 INFO:teuthology.orchestra.run.smithi192.stderr:2024-06-03T08:35:31.234+0000 7f780affd640 10 monclient(hunting): dump: 2024-06-03T08:35:31.242 INFO:teuthology.orchestra.run.smithi192.stderr:epoch 1 2024-06-03T08:35:31.242 INFO:teuthology.orchestra.run.smithi192.stderr:fsid ec3c3fdd-3ac6-4d7d-9412-683130ed8b04 2024-06-03T08:35:31.242 INFO:teuthology.orchestra.run.smithi192.stderr:last_changed 2024-06-03T08:34:31.885125+0000 2024-06-03T08:35:31.242 INFO:teuthology.orchestra.run.smithi192.stderr:created 2024-06-03T08:34:31.885125+0000 2024-06-03T08:35:31.242 INFO:teuthology.orchestra.run.smithi192.stderr:min_mon_release 18 (reef) 2024-06-03T08:35:31.242 INFO:teuthology.orchestra.run.smithi192.stderr:election_strategy: 3 2024-06-03T08:35:31.242 INFO:teuthology.orchestra.run.smithi192.stderr:0: v1:172.21.15.88:6789/0 mon.a 2024-06-03T08:35:31.242 INFO:teuthology.orchestra.run.smithi192.stderr:1: v1:172.21.15.192:6789/0 mon.b 2024-06-03T08:35:31.242 INFO:teuthology.orchestra.run.smithi192.stderr:2: v1:172.21.15.88:6790/0 mon.c 2024-06-03T08:35:31.242 INFO:teuthology.orchestra.run.smithi192.stderr: 2024-06-03T08:35:31.242 INFO:teuthology.orchestra.run.smithi192.stderr:2024-06-03T08:35:31.237+0000 7f780affd640 1 -- 172.21.15.192:0/4039535999 <== mon.1 v1:172.21.15.192:6789/0 4 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 580+0+0 (unknown 3997128948 0 0) 0x7f78141d68e0 con 0x7f78140a8960 2024-06-03T08:35:31.242 INFO:teuthology.orchestra.run.smithi192.stderr:2024-06-03T08:35:31.237+0000 7f780affd640 1 -- 172.21.15.192:0/4039535999 >> v1:172.21.15.88:6790/0 conn(0x7f78141450f0 legacy=0x7f78141da280 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-06-03T08:35:31.243 INFO:teuthology.orchestra.run.smithi192.stderr:2024-06-03T08:35:31.237+0000 7f780affd640 1 -- 172.21.15.192:0/4039535999 >> v1:172.21.15.88:6789/0 conn(0x7f781413eec0 legacy=0x7f78140b0170 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-06-03T08:35:31.243 INFO:teuthology.orchestra.run.smithi192.stderr:2024-06-03T08:35:31.237+0000 7f780affd640 10 monclient: _finish_hunting 0 2024-06-03T08:35:31.243 INFO:teuthology.orchestra.run.smithi192.stderr:2024-06-03T08:35:31.237+0000 7f780affd640 1 monclient: found mon.b 2024-06-03T08:35:31.243 INFO:teuthology.orchestra.run.smithi192.stderr:2024-06-03T08:35:31.237+0000 7f780affd640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-06-03T08:35:31.243 INFO:teuthology.orchestra.run.smithi192.stderr:2024-06-03T08:35:31.237+0000 7f780affd640 10 monclient: _send_mon_message to mon.b at v1:172.21.15.192:6789/0 2024-06-03T08:35:31.243 INFO:teuthology.orchestra.run.smithi192.stderr:2024-06-03T08:35:31.237+0000 7f780affd640 1 -- 172.21.15.192:0/4039535999 --> v1:172.21.15.192:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f78141d7510 con 0x7f78140a8960 2024-06-03T08:35:31.243 INFO:teuthology.orchestra.run.smithi192.stderr:2024-06-03T08:35:31.237+0000 7f780affd640 10 monclient: _finish_auth 0 2024-06-03T08:35:31.243 INFO:teuthology.orchestra.run.smithi192.stderr:2024-06-03T08:35:31.237+0000 7f780affd640 10 monclient: _check_auth_tickets 2024-06-03T08:35:31.243 INFO:teuthology.orchestra.run.smithi192.stderr:2024-06-03T08:35:31.237+0000 7f780affd640 20 monclient: _check_auth_rotating not needed by client.admin 2024-06-03T08:35:31.243 INFO:teuthology.orchestra.run.smithi192.stderr:2024-06-03T08:35:31.237+0000 7f781c5eb640 5 monclient: authenticate success, global_id 4150 2024-06-03T08:35:31.243 INFO:teuthology.orchestra.run.smithi192.stderr:2024-06-03T08:35:31.237+0000 7f781c5eb640 10 monclient: _renew_subs 2024-06-03T08:35:31.243 INFO:teuthology.orchestra.run.smithi192.stderr:2024-06-03T08:35:31.237+0000 7f781c5eb640 10 monclient: _send_mon_message to mon.b at v1:172.21.15.192:6789/0 2024-06-03T08:35:31.243 INFO:teuthology.orchestra.run.smithi192.stderr:2024-06-03T08:35:31.237+0000 7f781c5eb640 1 -- 172.21.15.192:0/4039535999 --> v1:172.21.15.192:6789/0 -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f78141d77c0 con 0x7f78140a8960 2024-06-03T08:35:31.243 INFO:teuthology.orchestra.run.smithi192.stderr:2024-06-03T08:35:31.237+0000 7f781c5eb640 10 client.4150.objecter _maybe_request_map subscribing (onetime) to next osd map 2024-06-03T08:35:31.244 INFO:teuthology.orchestra.run.smithi192.stderr:2024-06-03T08:35:31.237+0000 7f781c5eb640 10 monclient: _renew_subs 2024-06-03T08:35:31.244 INFO:teuthology.orchestra.run.smithi192.stderr:2024-06-03T08:35:31.237+0000 7f781c5eb640 10 monclient: _send_mon_message to mon.b at v1:172.21.15.192:6789/0 2024-06-03T08:35:31.244 INFO:teuthology.orchestra.run.smithi192.stderr:2024-06-03T08:35:31.237+0000 7f781c5eb640 1 -- 172.21.15.192:0/4039535999 --> v1:172.21.15.192:6789/0 -- mon_subscribe({osdmap=0}) v3 -- 0x7f78140b0a60 con 0x7f78140a8960 2024-06-03T08:35:31.244 INFO:teuthology.orchestra.run.smithi192.stderr:2024-06-03T08:35:31.237+0000 7f781c5eb640 1 librados: init done 2024-06-03T08:35:31.244 INFO:teuthology.orchestra.run.smithi192.stderr:2024-06-03T08:35:31.238+0000 7f781c5eb640 10 start_mon_command cmd=[{"prefix": "get_command_descriptions"}] 2024-06-03T08:35:31.244 INFO:teuthology.orchestra.run.smithi192.stderr:2024-06-03T08:35:31.238+0000 7f781c5eb640 10 monclient: _send_command 1 [{"prefix": "get_command_descriptions"}] 2024-06-03T08:35:31.244 INFO:teuthology.orchestra.run.smithi192.stderr:2024-06-03T08:35:31.238+0000 7f781c5eb640 10 monclient: _send_mon_message to mon.b at v1:172.21.15.192:6789/0 2024-06-03T08:35:31.244 INFO:teuthology.orchestra.run.smithi192.stderr:2024-06-03T08:35:31.238+0000 7f781c5eb640 1 -- 172.21.15.192:0/4039535999 --> v1:172.21.15.192:6789/0 -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f78140a9e60 con 0x7f78140a8960 2024-06-03T08:35:31.251 INFO:tasks.ceph.mgr.x.smithi192.stderr:2024-06-03T08:35:31.251+0000 7f0e6be43040 -1 mgr[py] Module devicehealth has missing NOTIFY_TYPES member 2024-06-03T08:35:31.330 INFO:tasks.ceph.mgr.y.smithi088.stderr:2024-06-03T08:35:31.330+0000 7ff0aea1d040 -1 mgr[py] Module prometheus has missing NOTIFY_TYPES member 2024-06-03T08:35:31.337 INFO:teuthology.orchestra.run.smithi192.stderr:2024-06-03T08:35:31.337+0000 7f780affd640 1 -- 172.21.15.192:0/4039535999 <== mon.1 v1:172.21.15.192:6789/0 5 ==== config(0 keys) v1 ==== 4+0+0 (unknown 0 0 0) 0x7f7810004a90 con 0x7f78140a8960 2024-06-03T08:35:31.337 INFO:teuthology.orchestra.run.smithi192.stderr:2024-06-03T08:35:31.337+0000 7f780affd640 10 monclient: handle_config config(0 keys) v1 2024-06-03T08:35:31.337 INFO:teuthology.orchestra.run.smithi192.stderr:2024-06-03T08:35:31.337+0000 7f780affd640 1 -- 172.21.15.192:0/4039535999 <== mon.1 v1:172.21.15.192:6789/0 6 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 2234374065 0 0) 0x7f7810004da0 con 0x7f78140a8960 2024-06-03T08:35:31.338 INFO:teuthology.orchestra.run.smithi192.stderr:2024-06-03T08:35:31.337+0000 7f780affd640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-06-03T08:35:31.338 INFO:teuthology.orchestra.run.smithi192.stderr:2024-06-03T08:35:31.337+0000 7f780affd640 10 monclient: got monmap 1 from mon.b (according to old e1) 2024-06-03T08:35:31.338 INFO:teuthology.orchestra.run.smithi192.stderr:2024-06-03T08:35:31.337+0000 7f780affd640 10 monclient: dump: 2024-06-03T08:35:31.338 INFO:teuthology.orchestra.run.smithi192.stderr:epoch 1 2024-06-03T08:35:31.338 INFO:teuthology.orchestra.run.smithi192.stderr:fsid ec3c3fdd-3ac6-4d7d-9412-683130ed8b04 2024-06-03T08:35:31.338 INFO:teuthology.orchestra.run.smithi192.stderr:last_changed 2024-06-03T08:34:31.885125+0000 2024-06-03T08:35:31.338 INFO:teuthology.orchestra.run.smithi192.stderr:created 2024-06-03T08:34:31.885125+0000 2024-06-03T08:35:31.338 INFO:teuthology.orchestra.run.smithi192.stderr:min_mon_release 18 (reef) 2024-06-03T08:35:31.338 INFO:teuthology.orchestra.run.smithi192.stderr:election_strategy: 3 2024-06-03T08:35:31.338 INFO:teuthology.orchestra.run.smithi192.stderr:0: v1:172.21.15.88:6789/0 mon.a 2024-06-03T08:35:31.338 INFO:teuthology.orchestra.run.smithi192.stderr:1: v1:172.21.15.192:6789/0 mon.b 2024-06-03T08:35:31.338 INFO:teuthology.orchestra.run.smithi192.stderr:2: v1:172.21.15.88:6790/0 mon.c 2024-06-03T08:35:31.338 INFO:teuthology.orchestra.run.smithi192.stderr: 2024-06-03T08:35:31.339 INFO:teuthology.orchestra.run.smithi192.stderr:2024-06-03T08:35:31.337+0000 7f780affd640 1 -- 172.21.15.192:0/4039535999 <== mon.1 v1:172.21.15.192:6789/0 7 ==== mgrmap(e 1) v1 ==== 668+0+0 (unknown 3836259515 0 0) 0x7f7810005240 con 0x7f78140a8960 2024-06-03T08:35:31.339 INFO:teuthology.orchestra.run.smithi192.stderr:2024-06-03T08:35:31.337+0000 7f7818b84640 4 set_mon_vals no callback set 2024-06-03T08:35:31.339 INFO:teuthology.orchestra.run.smithi192.stderr:2024-06-03T08:35:31.337+0000 7f780affd640 1 -- 172.21.15.192:0/4039535999 <== mon.1 v1:172.21.15.192:6789/0 8 ==== osd_map(7..7 src has 1..7) v4 ==== 1300+0+0 (unknown 513688909 0 0) 0x7f7810006820 con 0x7f78140a8960 2024-06-03T08:35:31.339 INFO:teuthology.orchestra.run.smithi192.stderr:2024-06-03T08:35:31.337+0000 7f780affd640 10 client.4150.objecter ms_dispatch 0x7f7814000b90 osd_map(7..7 src has 1..7) v4 2024-06-03T08:35:31.339 INFO:teuthology.orchestra.run.smithi192.stderr:2024-06-03T08:35:31.337+0000 7f780affd640 3 client.4150.objecter handle_osd_map got epochs [7,7] > 0 2024-06-03T08:35:31.339 INFO:teuthology.orchestra.run.smithi192.stderr:2024-06-03T08:35:31.337+0000 7f780affd640 3 client.4150.objecter handle_osd_map decoding full epoch 7 2024-06-03T08:35:31.339 INFO:teuthology.orchestra.run.smithi192.stderr:2024-06-03T08:35:31.337+0000 7f780affd640 20 client.4150.objecter dump_active .. 0 homeless 2024-06-03T08:35:31.339 INFO:teuthology.orchestra.run.smithi192.stderr:2024-06-03T08:35:31.337+0000 7f780affd640 1 -- 172.21.15.192:0/4039535999 <== mon.1 v1:172.21.15.192:6789/0 9 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+73121 (unknown 1092875540 0 1417653156) 0x7f78100069e0 con 0x7f78140a8960 2024-06-03T08:35:31.339 INFO:teuthology.orchestra.run.smithi192.stderr:2024-06-03T08:35:31.337+0000 7f780affd640 10 monclient: handle_mon_command_ack 1 [{"prefix": "get_command_descriptions"}] 2024-06-03T08:35:31.339 INFO:teuthology.orchestra.run.smithi192.stderr:2024-06-03T08:35:31.337+0000 7f780affd640 10 monclient: _finish_command 1 = system:0 2024-06-03T08:35:31.379 INFO:tasks.ceph.mgr.y.smithi088.stderr:2024-06-03T08:35:31.378+0000 7ff0aea1d040 -1 mgr[py] Module devicehealth has missing NOTIFY_TYPES member 2024-06-03T08:35:31.410 INFO:teuthology.orchestra.run.smithi192.stderr:2024-06-03T08:35:31.410+0000 7f781c5eb640 10 start_mon_command cmd=[{"prefix": "osd new", "uuid": "3cc263e9-e9bf-45e0-a3b7-83a76469eb70", "id": 5}] 2024-06-03T08:35:31.411 INFO:teuthology.orchestra.run.smithi192.stderr:2024-06-03T08:35:31.410+0000 7f781c5eb640 10 monclient: _send_command 2 [{"prefix": "osd new", "uuid": "3cc263e9-e9bf-45e0-a3b7-83a76469eb70", "id": 5}] 2024-06-03T08:35:31.411 INFO:teuthology.orchestra.run.smithi192.stderr:2024-06-03T08:35:31.410+0000 7f781c5eb640 10 monclient: _send_mon_message to mon.b at v1:172.21.15.192:6789/0 2024-06-03T08:35:31.411 INFO:teuthology.orchestra.run.smithi192.stderr:2024-06-03T08:35:31.410+0000 7f781c5eb640 1 -- 172.21.15.192:0/4039535999 --> v1:172.21.15.192:6789/0 -- mon_command({"prefix": "osd new", "uuid": "3cc263e9-e9bf-45e0-a3b7-83a76469eb70", "id": 5} v 0) v1 -- 0x7f78141d6a70 con 0x7f78140a8960 2024-06-03T08:35:31.411 INFO:tasks.ceph.mgr.x.smithi192.stderr:2024-06-03T08:35:31.411+0000 7f0e6be43040 -1 mgr[py] Module telemetry has missing NOTIFY_TYPES member 2024-06-03T08:35:31.474 INFO:teuthology.orchestra.run.smithi192.stderr:2024-06-03T08:35:31.473+0000 7f780affd640 1 -- 172.21.15.192:0/4039535999 <== mon.1 v1:172.21.15.192:6789/0 10 ==== mon_command_ack([{"prefix": "osd new", "uuid": "3cc263e9-e9bf-45e0-a3b7-83a76469eb70", "id": 5}]=0 v8) v1 ==== 112+0+2 (unknown 1589224411 0 1988412176) 0x7f78100062b0 con 0x7f78140a8960 2024-06-03T08:35:31.474 INFO:teuthology.orchestra.run.smithi192.stderr:2024-06-03T08:35:31.473+0000 7f780affd640 10 monclient: handle_mon_command_ack 2 [{"prefix": "osd new", "uuid": "3cc263e9-e9bf-45e0-a3b7-83a76469eb70", "id": 5}] 2024-06-03T08:35:31.474 INFO:teuthology.orchestra.run.smithi192.stderr:2024-06-03T08:35:31.473+0000 7f780affd640 10 monclient: _finish_command 2 = system:0 2024-06-03T08:35:31.474 INFO:teuthology.orchestra.run.smithi192.stdout:5 2024-06-03T08:35:31.475 INFO:teuthology.orchestra.run.smithi192.stderr:2024-06-03T08:35:31.474+0000 7f781c5eb640 10 librados: watch_flush enter 2024-06-03T08:35:31.476 INFO:teuthology.orchestra.run.smithi192.stderr:2024-06-03T08:35:31.474+0000 7f781c5eb640 10 librados: watch_flush exit 2024-06-03T08:35:31.476 INFO:teuthology.orchestra.run.smithi192.stderr:2024-06-03T08:35:31.474+0000 7f781c5eb640 20 client.4150.objecter shutdown clearing up homeless session... 2024-06-03T08:35:31.476 INFO:teuthology.orchestra.run.smithi192.stderr:2024-06-03T08:35:31.474+0000 7f781c5eb640 10 client.4150.objecter successfully canceled tick 2024-06-03T08:35:31.476 INFO:teuthology.orchestra.run.smithi192.stderr:2024-06-03T08:35:31.474+0000 7f781c5eb640 10 monclient: shutdown 2024-06-03T08:35:31.476 INFO:teuthology.orchestra.run.smithi192.stderr:2024-06-03T08:35:31.474+0000 7f781c5eb640 20 monclient: shutdown discarding 0 pending message(s) 2024-06-03T08:35:31.476 INFO:teuthology.orchestra.run.smithi192.stderr:2024-06-03T08:35:31.474+0000 7f781c5eb640 1 -- 172.21.15.192:0/4039535999 >> v1:172.21.15.192:6789/0 conn(0x7f78140a8960 legacy=0x7f78141d6e00 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-06-03T08:35:31.476 INFO:teuthology.orchestra.run.smithi192.stderr:2024-06-03T08:35:31.475+0000 7f781c5eb640 1 -- 172.21.15.192:0/4039535999 shutdown_connections 2024-06-03T08:35:31.476 INFO:teuthology.orchestra.run.smithi192.stderr:2024-06-03T08:35:31.475+0000 7f781c5eb640 1 -- 172.21.15.192:0/4039535999 >> 172.21.15.192:0/4039535999 conn(0x7f78140b70d0 msgr2=0x7f781413e580 unknown :-1 s=STATE_NONE l=0).mark_down 2024-06-03T08:35:31.476 INFO:teuthology.orchestra.run.smithi192.stderr:2024-06-03T08:35:31.475+0000 7f781c5eb640 1 -- 172.21.15.192:0/4039535999 shutdown_connections 2024-06-03T08:35:31.476 INFO:teuthology.orchestra.run.smithi192.stderr:2024-06-03T08:35:31.475+0000 7f781c5eb640 1 -- 172.21.15.192:0/4039535999 wait complete. 2024-06-03T08:35:31.476 INFO:teuthology.orchestra.run.smithi192.stderr:2024-06-03T08:35:31.475+0000 7f781c5eb640 1 librados: shutdown 2024-06-03T08:35:31.485 DEBUG:teuthology.orchestra.run.smithi192:> sudo ceph --cluster ceph osd new 82c83316-809f-4901-a5ed-34b6a075b393 6 2024-06-03T08:35:31.545 INFO:tasks.ceph.mgr.y.smithi088.stderr:2024-06-03T08:35:31.544+0000 7ff0aea1d040 -1 mgr[py] Module telemetry has missing NOTIFY_TYPES member 2024-06-03T08:35:31.556 INFO:tasks.ceph.mgr.x.smithi192.stderr:2024-06-03T08:35:31.556+0000 7f0e6be43040 -1 mgr[py] Module diskprediction_local has missing NOTIFY_TYPES member 2024-06-03T08:35:31.584 INFO:teuthology.orchestra.run.smithi192.stderr:2024-06-03T08:35:31.584+0000 7f6ff92a8640 10 monclient: get_monmap_and_config 2024-06-03T08:35:31.584 INFO:teuthology.orchestra.run.smithi192.stderr:2024-06-03T08:35:31.584+0000 7f6ff92a8640 10 monclient: build_initial_monmap 2024-06-03T08:35:31.584 INFO:teuthology.orchestra.run.smithi192.stderr:2024-06-03T08:35:31.584+0000 7f6ff92a8640 10 monclient: monmap: 2024-06-03T08:35:31.585 INFO:teuthology.orchestra.run.smithi192.stderr:epoch 0 2024-06-03T08:35:31.585 INFO:teuthology.orchestra.run.smithi192.stderr:fsid ec3c3fdd-3ac6-4d7d-9412-683130ed8b04 2024-06-03T08:35:31.585 INFO:teuthology.orchestra.run.smithi192.stderr:last_changed 2024-06-03T08:35:31.584974+0000 2024-06-03T08:35:31.585 INFO:teuthology.orchestra.run.smithi192.stderr:created 2024-06-03T08:35:31.584974+0000 2024-06-03T08:35:31.585 INFO:teuthology.orchestra.run.smithi192.stderr:min_mon_release 0 (unknown) 2024-06-03T08:35:31.585 INFO:teuthology.orchestra.run.smithi192.stderr:election_strategy: 3 2024-06-03T08:35:31.585 INFO:teuthology.orchestra.run.smithi192.stderr:0: v1:172.21.15.88:6789/0 mon.noname-a 2024-06-03T08:35:31.585 INFO:teuthology.orchestra.run.smithi192.stderr:1: v1:172.21.15.192:6789/0 mon.noname-c 2024-06-03T08:35:31.585 INFO:teuthology.orchestra.run.smithi192.stderr:2: v1:172.21.15.88:6790/0 mon.noname-b 2024-06-03T08:35:31.585 INFO:teuthology.orchestra.run.smithi192.stderr: 2024-06-03T08:35:31.585 INFO:teuthology.orchestra.run.smithi192.stderr:2024-06-03T08:35:31.584+0000 7f6ff92a8640 1 Processor -- start 2024-06-03T08:35:31.585 INFO:teuthology.orchestra.run.smithi192.stderr:2024-06-03T08:35:31.584+0000 7f6ff92a8640 1 -- start start 2024-06-03T08:35:31.585 INFO:teuthology.orchestra.run.smithi192.stderr:2024-06-03T08:35:31.584+0000 7f6ff92a8640 10 monclient: init 2024-06-03T08:35:31.587 INFO:teuthology.orchestra.run.smithi192.stderr:2024-06-03T08:35:31.585+0000 7f6ff92a8640 10 monclient: _reopen_session rank -1 2024-06-03T08:35:31.587 INFO:teuthology.orchestra.run.smithi192.stderr:2024-06-03T08:35:31.585+0000 7f6ff92a8640 10 monclient: _add_conns ranks=[1,0,2] 2024-06-03T08:35:31.587 INFO:teuthology.orchestra.run.smithi192.stderr:2024-06-03T08:35:31.585+0000 7f6ff92a8640 10 monclient(hunting): picked mon.noname-c con 0x7f6ff40b43c0 addr v1:172.21.15.192:6789/0 2024-06-03T08:35:31.587 INFO:teuthology.orchestra.run.smithi192.stderr:2024-06-03T08:35:31.585+0000 7f6ff92a8640 10 monclient(hunting): picked mon.noname-a con 0x7f6ff413e6c0 addr v1:172.21.15.88:6789/0 2024-06-03T08:35:31.587 INFO:teuthology.orchestra.run.smithi192.stderr:2024-06-03T08:35:31.585+0000 7f6ff92a8640 10 monclient(hunting): picked mon.noname-b con 0x7f6ff414ae20 addr v1:172.21.15.88:6790/0 2024-06-03T08:35:31.587 INFO:teuthology.orchestra.run.smithi192.stderr:2024-06-03T08:35:31.585+0000 7f6ff92a8640 1 -- --> v1:172.21.15.88:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f6ff40a8ed0 con 0x7f6ff413e6c0 2024-06-03T08:35:31.587 INFO:teuthology.orchestra.run.smithi192.stderr:2024-06-03T08:35:31.585+0000 7f6ff92a8640 1 -- --> v1:172.21.15.192:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f6ff40aa420 con 0x7f6ff40b43c0 2024-06-03T08:35:31.587 INFO:teuthology.orchestra.run.smithi192.stderr:2024-06-03T08:35:31.585+0000 7f6ff92a8640 1 -- --> v1:172.21.15.88:6790/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f6ff40aa690 con 0x7f6ff414ae20 2024-06-03T08:35:31.587 INFO:teuthology.orchestra.run.smithi192.stderr:2024-06-03T08:35:31.585+0000 7f6ff92a8640 10 monclient(hunting): _renew_subs 2024-06-03T08:35:31.587 INFO:teuthology.orchestra.run.smithi192.stderr:2024-06-03T08:35:31.585+0000 7f6ff92a8640 10 monclient(hunting): authenticate will time out at 1197.396482s 2024-06-03T08:35:31.588 INFO:teuthology.orchestra.run.smithi192.stderr:2024-06-03T08:35:31.585+0000 7f6ff2ffd640 1 --1- >> v1:172.21.15.192:6789/0 conn(0x7f6ff40b43c0 0x7f6ff41415b0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.192:6789/0 says I am v1:172.21.15.192:43900/0 (socket says 172.21.15.192:43900) 2024-06-03T08:35:31.588 INFO:teuthology.orchestra.run.smithi192.stderr:2024-06-03T08:35:31.585+0000 7f6ff2ffd640 1 -- 172.21.15.192:0/4076141853 learned_addr learned my addr 172.21.15.192:0/4076141853 (peer_addr_for_me v1:172.21.15.192:0/0) 2024-06-03T08:35:31.588 INFO:teuthology.orchestra.run.smithi192.stderr:2024-06-03T08:35:31.588+0000 7f6ff1ffb640 1 -- 172.21.15.192:0/4076141853 <== mon.2 v1:172.21.15.88:6790/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 3999259811 0 0) 0x7f6ff40aa690 con 0x7f6ff414ae20 2024-06-03T08:35:31.588 INFO:teuthology.orchestra.run.smithi192.stderr:2024-06-03T08:35:31.588+0000 7f6ff1ffb640 10 monclient(hunting): _negotiate 2024-06-03T08:35:31.588 INFO:teuthology.orchestra.run.smithi192.stderr:2024-06-03T08:35:31.588+0000 7f6ff1ffb640 10 monclient(hunting): _init_auth method 2 2024-06-03T08:35:31.589 INFO:teuthology.orchestra.run.smithi192.stderr:2024-06-03T08:35:31.588+0000 7f6ff1ffb640 10 monclient(hunting): _init_auth creating new auth 2024-06-03T08:35:31.589 INFO:teuthology.orchestra.run.smithi192.stderr:2024-06-03T08:35:31.588+0000 7f6ff1ffb640 10 monclient(hunting): my global_id is 4154 2024-06-03T08:35:31.589 INFO:teuthology.orchestra.run.smithi192.stderr:2024-06-03T08:35:31.588+0000 7f6ff1ffb640 1 -- 172.21.15.192:0/4076141853 --> v1:172.21.15.88:6790/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f6fe0003520 con 0x7f6ff414ae20 2024-06-03T08:35:31.589 INFO:teuthology.orchestra.run.smithi192.stderr:2024-06-03T08:35:31.588+0000 7f6ff1ffb640 1 -- 172.21.15.192:0/4076141853 <== mon.1 v1:172.21.15.192:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 1857872286 0 0) 0x7f6ff40aa420 con 0x7f6ff40b43c0 2024-06-03T08:35:31.589 INFO:teuthology.orchestra.run.smithi192.stderr:2024-06-03T08:35:31.588+0000 7f6ff1ffb640 10 monclient(hunting): _negotiate 2024-06-03T08:35:31.589 INFO:teuthology.orchestra.run.smithi192.stderr:2024-06-03T08:35:31.588+0000 7f6ff1ffb640 10 monclient(hunting): _init_auth method 2 2024-06-03T08:35:31.589 INFO:teuthology.orchestra.run.smithi192.stderr:2024-06-03T08:35:31.588+0000 7f6ff1ffb640 10 monclient(hunting): _init_auth creating new auth 2024-06-03T08:35:31.589 INFO:teuthology.orchestra.run.smithi192.stderr:2024-06-03T08:35:31.588+0000 7f6ff1ffb640 10 monclient(hunting): my global_id is 4153 2024-06-03T08:35:31.589 INFO:teuthology.orchestra.run.smithi192.stderr:2024-06-03T08:35:31.588+0000 7f6ff1ffb640 1 -- 172.21.15.192:0/4076141853 --> v1:172.21.15.192:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f6ff40aa690 con 0x7f6ff40b43c0 2024-06-03T08:35:31.590 INFO:teuthology.orchestra.run.smithi192.stderr:2024-06-03T08:35:31.589+0000 7f6ff1ffb640 1 -- 172.21.15.192:0/4076141853 <== mon.0 v1:172.21.15.88:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 1570694119 0 0) 0x7f6ff40a8ed0 con 0x7f6ff413e6c0 2024-06-03T08:35:31.590 INFO:teuthology.orchestra.run.smithi192.stderr:2024-06-03T08:35:31.589+0000 7f6ff1ffb640 10 monclient(hunting): _negotiate 2024-06-03T08:35:31.590 INFO:teuthology.orchestra.run.smithi192.stderr:2024-06-03T08:35:31.589+0000 7f6ff1ffb640 10 monclient(hunting): _init_auth method 2 2024-06-03T08:35:31.590 INFO:teuthology.orchestra.run.smithi192.stderr:2024-06-03T08:35:31.589+0000 7f6ff1ffb640 10 monclient(hunting): _init_auth creating new auth 2024-06-03T08:35:31.590 INFO:teuthology.orchestra.run.smithi192.stderr:2024-06-03T08:35:31.589+0000 7f6ff1ffb640 10 monclient(hunting): my global_id is 4152 2024-06-03T08:35:31.590 INFO:teuthology.orchestra.run.smithi192.stderr:2024-06-03T08:35:31.589+0000 7f6ff1ffb640 1 -- 172.21.15.192:0/4076141853 --> v1:172.21.15.88:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f6ff40aa420 con 0x7f6ff413e6c0 2024-06-03T08:35:31.592 INFO:teuthology.orchestra.run.smithi192.stderr:2024-06-03T08:35:31.592+0000 7f6ff1ffb640 1 -- 172.21.15.192:0/4076141853 <== mon.2 v1:172.21.15.88:6790/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 214+0+0 (unknown 907592524 0 0) 0x7f6fe0003520 con 0x7f6ff414ae20 2024-06-03T08:35:31.592 INFO:teuthology.orchestra.run.smithi192.stderr:2024-06-03T08:35:31.592+0000 7f6ff1ffb640 1 -- 172.21.15.192:0/4076141853 >> v1:172.21.15.192:6789/0 conn(0x7f6ff40b43c0 legacy=0x7f6ff41415b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-06-03T08:35:31.592 INFO:teuthology.orchestra.run.smithi192.stderr:2024-06-03T08:35:31.592+0000 7f6ff1ffb640 1 -- 172.21.15.192:0/4076141853 >> v1:172.21.15.88:6789/0 conn(0x7f6ff413e6c0 legacy=0x7f6ff413eb40 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-06-03T08:35:31.592 INFO:teuthology.orchestra.run.smithi192.stderr:2024-06-03T08:35:31.592+0000 7f6ff1ffb640 10 monclient: _finish_hunting 0 2024-06-03T08:35:31.592 INFO:teuthology.orchestra.run.smithi192.stderr:2024-06-03T08:35:31.592+0000 7f6ff1ffb640 1 monclient: found mon.noname-b 2024-06-03T08:35:31.592 INFO:teuthology.orchestra.run.smithi192.stderr:2024-06-03T08:35:31.592+0000 7f6ff1ffb640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-06-03T08:35:31.592 INFO:teuthology.orchestra.run.smithi192.stderr:2024-06-03T08:35:31.592+0000 7f6ff1ffb640 10 monclient: _send_mon_message to mon.noname-b at v1:172.21.15.88:6790/0 2024-06-03T08:35:31.592 INFO:teuthology.orchestra.run.smithi192.stderr:2024-06-03T08:35:31.592+0000 7f6ff1ffb640 1 -- 172.21.15.192:0/4076141853 --> v1:172.21.15.88:6790/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f6ff40a87f0 con 0x7f6ff414ae20 2024-06-03T08:35:31.593 INFO:teuthology.orchestra.run.smithi192.stderr:2024-06-03T08:35:31.592+0000 7f6ff1ffb640 10 monclient: _finish_auth 0 2024-06-03T08:35:31.593 INFO:teuthology.orchestra.run.smithi192.stderr:2024-06-03T08:35:31.592+0000 7f6ff1ffb640 10 monclient: _check_auth_tickets 2024-06-03T08:35:31.593 INFO:teuthology.orchestra.run.smithi192.stderr:2024-06-03T08:35:31.592+0000 7f6ff1ffb640 20 monclient: _check_auth_rotating not needed by client.admin 2024-06-03T08:35:31.593 INFO:teuthology.orchestra.run.smithi192.stderr:2024-06-03T08:35:31.592+0000 7f6ff1ffb640 1 -- 172.21.15.192:0/4076141853 <== mon.2 v1:172.21.15.88:6790/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 2234374065 0 0) 0x7f6fec001990 con 0x7f6ff414ae20 2024-06-03T08:35:31.593 INFO:teuthology.orchestra.run.smithi192.stderr:2024-06-03T08:35:31.592+0000 7f6ff1ffb640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-06-03T08:35:31.593 INFO:teuthology.orchestra.run.smithi192.stderr:2024-06-03T08:35:31.592+0000 7f6ff1ffb640 10 monclient: got monmap 1 from mon.noname-b (according to old e1) 2024-06-03T08:35:31.593 INFO:teuthology.orchestra.run.smithi192.stderr:2024-06-03T08:35:31.592+0000 7f6ff1ffb640 10 monclient: dump: 2024-06-03T08:35:31.593 INFO:teuthology.orchestra.run.smithi192.stderr:epoch 1 2024-06-03T08:35:31.593 INFO:teuthology.orchestra.run.smithi192.stderr:fsid ec3c3fdd-3ac6-4d7d-9412-683130ed8b04 2024-06-03T08:35:31.593 INFO:teuthology.orchestra.run.smithi192.stderr:last_changed 2024-06-03T08:34:31.885125+0000 2024-06-03T08:35:31.593 INFO:teuthology.orchestra.run.smithi192.stderr:created 2024-06-03T08:34:31.885125+0000 2024-06-03T08:35:31.593 INFO:teuthology.orchestra.run.smithi192.stderr:min_mon_release 18 (reef) 2024-06-03T08:35:31.593 INFO:teuthology.orchestra.run.smithi192.stderr:election_strategy: 3 2024-06-03T08:35:31.593 INFO:teuthology.orchestra.run.smithi192.stderr:0: v1:172.21.15.88:6789/0 mon.a 2024-06-03T08:35:31.594 INFO:teuthology.orchestra.run.smithi192.stderr:1: v1:172.21.15.192:6789/0 mon.b 2024-06-03T08:35:31.594 INFO:teuthology.orchestra.run.smithi192.stderr:2: v1:172.21.15.88:6790/0 mon.c 2024-06-03T08:35:31.594 INFO:teuthology.orchestra.run.smithi192.stderr: 2024-06-03T08:35:31.594 INFO:teuthology.orchestra.run.smithi192.stderr:2024-06-03T08:35:31.592+0000 7f6ff92a8640 5 monclient: authenticate success, global_id 4154 2024-06-03T08:35:31.594 INFO:teuthology.orchestra.run.smithi192.stderr:2024-06-03T08:35:31.592+0000 7f6ff92a8640 20 monclient: get_monmap_and_config waiting for monmap|config 2024-06-03T08:35:31.595 INFO:teuthology.orchestra.run.smithi192.stderr:2024-06-03T08:35:31.593+0000 7f6ff1ffb640 1 -- 172.21.15.192:0/4076141853 <== mon.2 v1:172.21.15.88:6790/0 4 ==== config(0 keys) v1 ==== 4+0+0 (unknown 0 0 0) 0x7f6fec001bb0 con 0x7f6ff414ae20 2024-06-03T08:35:31.595 INFO:teuthology.orchestra.run.smithi192.stderr:2024-06-03T08:35:31.593+0000 7f6ff1ffb640 10 monclient: handle_config config(0 keys) v1 2024-06-03T08:35:31.595 INFO:teuthology.orchestra.run.smithi192.stderr:2024-06-03T08:35:31.593+0000 7f6ff1ffb640 1 -- 172.21.15.192:0/4076141853 <== mon.2 v1:172.21.15.88:6790/0 5 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 2234374065 0 0) 0x7f6fec001f40 con 0x7f6ff414ae20 2024-06-03T08:35:31.596 INFO:teuthology.orchestra.run.smithi192.stderr:2024-06-03T08:35:31.593+0000 7f6ff1ffb640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-06-03T08:35:31.596 INFO:teuthology.orchestra.run.smithi192.stderr:2024-06-03T08:35:31.593+0000 7f6ff1ffb640 10 monclient: got monmap 1 from mon.c (according to old e1) 2024-06-03T08:35:31.596 INFO:teuthology.orchestra.run.smithi192.stderr:2024-06-03T08:35:31.593+0000 7f6ff1ffb640 10 monclient: dump: 2024-06-03T08:35:31.596 INFO:teuthology.orchestra.run.smithi192.stderr:epoch 1 2024-06-03T08:35:31.596 INFO:teuthology.orchestra.run.smithi192.stderr:fsid ec3c3fdd-3ac6-4d7d-9412-683130ed8b04 2024-06-03T08:35:31.596 INFO:teuthology.orchestra.run.smithi192.stderr:last_changed 2024-06-03T08:34:31.885125+0000 2024-06-03T08:35:31.596 INFO:teuthology.orchestra.run.smithi192.stderr:created 2024-06-03T08:34:31.885125+0000 2024-06-03T08:35:31.596 INFO:teuthology.orchestra.run.smithi192.stderr:min_mon_release 18 (reef) 2024-06-03T08:35:31.596 INFO:teuthology.orchestra.run.smithi192.stderr:election_strategy: 3 2024-06-03T08:35:31.596 INFO:teuthology.orchestra.run.smithi192.stderr:0: v1:172.21.15.88:6789/0 mon.a 2024-06-03T08:35:31.596 INFO:teuthology.orchestra.run.smithi192.stderr:1: v1:172.21.15.192:6789/0 mon.b 2024-06-03T08:35:31.596 INFO:teuthology.orchestra.run.smithi192.stderr:2: v1:172.21.15.88:6790/0 mon.c 2024-06-03T08:35:31.596 INFO:teuthology.orchestra.run.smithi192.stderr: 2024-06-03T08:35:31.597 INFO:teuthology.orchestra.run.smithi192.stderr:2024-06-03T08:35:31.593+0000 7f6ff92a8640 10 monclient: get_monmap_and_config success 2024-06-03T08:35:31.597 INFO:teuthology.orchestra.run.smithi192.stderr:2024-06-03T08:35:31.593+0000 7f6ff92a8640 4 set_mon_vals no callback set 2024-06-03T08:35:31.597 INFO:teuthology.orchestra.run.smithi192.stderr:2024-06-03T08:35:31.594+0000 7f6ff92a8640 10 monclient: shutdown 2024-06-03T08:35:31.597 INFO:teuthology.orchestra.run.smithi192.stderr:2024-06-03T08:35:31.594+0000 7f6ff92a8640 20 monclient: shutdown discarding 0 pending message(s) 2024-06-03T08:35:31.597 INFO:teuthology.orchestra.run.smithi192.stderr:2024-06-03T08:35:31.594+0000 7f6ff92a8640 1 -- 172.21.15.192:0/4076141853 >> v1:172.21.15.88:6790/0 conn(0x7f6ff414ae20 legacy=0x7f6ff414d210 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-06-03T08:35:31.597 INFO:teuthology.orchestra.run.smithi192.stderr:2024-06-03T08:35:31.594+0000 7f6ff92a8640 1 -- 172.21.15.192:0/4076141853 shutdown_connections 2024-06-03T08:35:31.597 INFO:teuthology.orchestra.run.smithi192.stderr:2024-06-03T08:35:31.594+0000 7f6ff92a8640 1 -- 172.21.15.192:0/4076141853 >> 172.21.15.192:0/4076141853 conn(0x7f6ff40719d0 msgr2=0x7f6ff4071de0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-06-03T08:35:31.597 INFO:teuthology.orchestra.run.smithi192.stderr:2024-06-03T08:35:31.594+0000 7f6ff92a8640 1 -- 172.21.15.192:0/4076141853 shutdown_connections 2024-06-03T08:35:31.597 INFO:teuthology.orchestra.run.smithi192.stderr:2024-06-03T08:35:31.594+0000 7f6ff92a8640 1 -- 172.21.15.192:0/4076141853 wait complete. 2024-06-03T08:35:31.597 INFO:teuthology.orchestra.run.smithi192.stderr:2024-06-03T08:35:31.595+0000 7f6ff92a8640 10 monclient: build_initial_monmap 2024-06-03T08:35:31.597 INFO:teuthology.orchestra.run.smithi192.stderr:2024-06-03T08:35:31.595+0000 7f6ff92a8640 10 monclient: monmap: 2024-06-03T08:35:31.597 INFO:teuthology.orchestra.run.smithi192.stderr:epoch 0 2024-06-03T08:35:31.597 INFO:teuthology.orchestra.run.smithi192.stderr:fsid 00000000-0000-0000-0000-000000000000 2024-06-03T08:35:31.597 INFO:teuthology.orchestra.run.smithi192.stderr:last_changed 0.000000 2024-06-03T08:35:31.597 INFO:teuthology.orchestra.run.smithi192.stderr:created 0.000000 2024-06-03T08:35:31.598 INFO:teuthology.orchestra.run.smithi192.stderr:min_mon_release 0 (unknown) 2024-06-03T08:35:31.598 INFO:teuthology.orchestra.run.smithi192.stderr:election_strategy: 1 2024-06-03T08:35:31.598 INFO:teuthology.orchestra.run.smithi192.stderr:0: v1:172.21.15.88:6789/0 mon.noname-a 2024-06-03T08:35:31.598 INFO:teuthology.orchestra.run.smithi192.stderr:1: v1:172.21.15.192:6789/0 mon.noname-b 2024-06-03T08:35:31.598 INFO:teuthology.orchestra.run.smithi192.stderr:2: v1:172.21.15.88:6790/0 mon.noname-c 2024-06-03T08:35:31.598 INFO:teuthology.orchestra.run.smithi192.stderr: 2024-06-03T08:35:31.598 INFO:teuthology.orchestra.run.smithi192.stderr:2024-06-03T08:35:31.595+0000 7f6ff92a8640 1 librados: starting msgr at 2024-06-03T08:35:31.598 INFO:teuthology.orchestra.run.smithi192.stderr:2024-06-03T08:35:31.595+0000 7f6ff92a8640 1 librados: starting objecter 2024-06-03T08:35:31.598 INFO:teuthology.orchestra.run.smithi192.stderr:2024-06-03T08:35:31.595+0000 7f6ff92a8640 1 Processor -- start 2024-06-03T08:35:31.598 INFO:teuthology.orchestra.run.smithi192.stderr:2024-06-03T08:35:31.595+0000 7f6ff92a8640 1 -- start start 2024-06-03T08:35:31.598 INFO:teuthology.orchestra.run.smithi192.stderr:2024-06-03T08:35:31.595+0000 7f6ff92a8640 1 librados: setting wanted keys 2024-06-03T08:35:31.598 INFO:teuthology.orchestra.run.smithi192.stderr:2024-06-03T08:35:31.595+0000 7f6ff92a8640 1 librados: calling monclient init 2024-06-03T08:35:31.598 INFO:teuthology.orchestra.run.smithi192.stderr:2024-06-03T08:35:31.595+0000 7f6ff92a8640 10 monclient: init 2024-06-03T08:35:31.600 INFO:teuthology.orchestra.run.smithi192.stderr:2024-06-03T08:35:31.596+0000 7f6ff92a8640 10 monclient: _reopen_session rank -1 2024-06-03T08:35:31.600 INFO:teuthology.orchestra.run.smithi192.stderr:2024-06-03T08:35:31.596+0000 7f6ff92a8640 10 monclient: _add_conns ranks=[2,0,1] 2024-06-03T08:35:31.600 INFO:teuthology.orchestra.run.smithi192.stderr:2024-06-03T08:35:31.596+0000 7f6ff92a8640 10 monclient(hunting): picked mon.noname-c con 0x7f6ff40b43c0 addr v1:172.21.15.88:6790/0 2024-06-03T08:35:31.601 INFO:teuthology.orchestra.run.smithi192.stderr:2024-06-03T08:35:31.596+0000 7f6ff92a8640 10 monclient(hunting): picked mon.noname-a con 0x7f6ff413e6c0 addr v1:172.21.15.88:6789/0 2024-06-03T08:35:31.601 INFO:teuthology.orchestra.run.smithi192.stderr:2024-06-03T08:35:31.596+0000 7f6ff92a8640 10 monclient(hunting): picked mon.noname-b con 0x7f6ff414ae20 addr v1:172.21.15.192:6789/0 2024-06-03T08:35:31.601 INFO:teuthology.orchestra.run.smithi192.stderr:2024-06-03T08:35:31.596+0000 7f6ff92a8640 1 -- --> v1:172.21.15.88:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f6ff41453e0 con 0x7f6ff413e6c0 2024-06-03T08:35:31.601 INFO:teuthology.orchestra.run.smithi192.stderr:2024-06-03T08:35:31.596+0000 7f6ff92a8640 1 -- --> v1:172.21.15.192:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f6ff41385d0 con 0x7f6ff414ae20 2024-06-03T08:35:31.601 INFO:teuthology.orchestra.run.smithi192.stderr:2024-06-03T08:35:31.596+0000 7f6ff92a8640 1 -- --> v1:172.21.15.88:6790/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f6ff4145680 con 0x7f6ff40b43c0 2024-06-03T08:35:31.601 INFO:teuthology.orchestra.run.smithi192.stderr:2024-06-03T08:35:31.596+0000 7f6ff92a8640 10 monclient(hunting): _renew_subs 2024-06-03T08:35:31.601 INFO:teuthology.orchestra.run.smithi192.stderr:2024-06-03T08:35:31.596+0000 7f6ff92a8640 10 monclient(hunting): authenticate will time out at 1197.407384s 2024-06-03T08:35:31.601 INFO:teuthology.orchestra.run.smithi192.stderr:2024-06-03T08:35:31.596+0000 7f6ff37fe640 1 --1- >> v1:172.21.15.192:6789/0 conn(0x7f6ff414ae20 0x7f6ff41d9ab0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=1).handle_server_banner_and_identify peer v1:172.21.15.192:6789/0 says I am v1:172.21.15.192:43912/0 (socket says 172.21.15.192:43912) 2024-06-03T08:35:31.601 INFO:teuthology.orchestra.run.smithi192.stderr:2024-06-03T08:35:31.596+0000 7f6ff37fe640 1 -- 172.21.15.192:0/2274270257 learned_addr learned my addr 172.21.15.192:0/2274270257 (peer_addr_for_me v1:172.21.15.192:0/0) 2024-06-03T08:35:31.601 INFO:teuthology.orchestra.run.smithi192.stderr:2024-06-03T08:35:31.597+0000 7f6feb7fe640 10 client.?.objecter ms_handle_connect 0x7f6ff414ae20 2024-06-03T08:35:31.601 INFO:teuthology.orchestra.run.smithi192.stderr:2024-06-03T08:35:31.597+0000 7f6feb7fe640 10 client.?.objecter resend_mon_ops 2024-06-03T08:35:31.601 INFO:teuthology.orchestra.run.smithi192.stderr:2024-06-03T08:35:31.597+0000 7f6feb7fe640 10 client.?.objecter ms_handle_connect 0x7f6ff40b43c0 2024-06-03T08:35:31.601 INFO:teuthology.orchestra.run.smithi192.stderr:2024-06-03T08:35:31.597+0000 7f6feb7fe640 10 client.?.objecter resend_mon_ops 2024-06-03T08:35:31.602 INFO:teuthology.orchestra.run.smithi192.stderr:2024-06-03T08:35:31.597+0000 7f6feb7fe640 10 client.?.objecter ms_handle_connect 0x7f6ff413e6c0 2024-06-03T08:35:31.602 INFO:teuthology.orchestra.run.smithi192.stderr:2024-06-03T08:35:31.597+0000 7f6feb7fe640 10 client.?.objecter resend_mon_ops 2024-06-03T08:35:31.602 INFO:teuthology.orchestra.run.smithi192.stderr:2024-06-03T08:35:31.599+0000 7f6feb7fe640 1 -- 172.21.15.192:0/2274270257 <== mon.2 v1:172.21.15.88:6790/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 2913037259 0 0) 0x7f6ff4145680 con 0x7f6ff40b43c0 2024-06-03T08:35:31.602 INFO:teuthology.orchestra.run.smithi192.stderr:2024-06-03T08:35:31.599+0000 7f6feb7fe640 10 monclient(hunting): _negotiate 2024-06-03T08:35:31.602 INFO:teuthology.orchestra.run.smithi192.stderr:2024-06-03T08:35:31.599+0000 7f6feb7fe640 10 monclient(hunting): _init_auth method 2 2024-06-03T08:35:31.602 INFO:teuthology.orchestra.run.smithi192.stderr:2024-06-03T08:35:31.599+0000 7f6feb7fe640 10 monclient(hunting): _init_auth creating new auth 2024-06-03T08:35:31.602 INFO:teuthology.orchestra.run.smithi192.stderr:2024-06-03T08:35:31.599+0000 7f6feb7fe640 10 monclient(hunting): my global_id is 4157 2024-06-03T08:35:31.602 INFO:teuthology.orchestra.run.smithi192.stderr:2024-06-03T08:35:31.599+0000 7f6feb7fe640 1 -- 172.21.15.192:0/2274270257 --> v1:172.21.15.88:6790/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f6fd4004eb0 con 0x7f6ff40b43c0 2024-06-03T08:35:31.602 INFO:teuthology.orchestra.run.smithi192.stderr:2024-06-03T08:35:31.599+0000 7f6feb7fe640 1 -- 172.21.15.192:0/2274270257 <== mon.1 v1:172.21.15.192:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 3963107664 0 0) 0x7f6ff41385d0 con 0x7f6ff414ae20 2024-06-03T08:35:31.602 INFO:teuthology.orchestra.run.smithi192.stderr:2024-06-03T08:35:31.599+0000 7f6feb7fe640 10 monclient(hunting): _negotiate 2024-06-03T08:35:31.602 INFO:teuthology.orchestra.run.smithi192.stderr:2024-06-03T08:35:31.599+0000 7f6feb7fe640 10 monclient(hunting): _init_auth method 2 2024-06-03T08:35:31.603 INFO:teuthology.orchestra.run.smithi192.stderr:2024-06-03T08:35:31.599+0000 7f6feb7fe640 10 monclient(hunting): _init_auth creating new auth 2024-06-03T08:35:31.603 INFO:teuthology.orchestra.run.smithi192.stderr:2024-06-03T08:35:31.599+0000 7f6feb7fe640 10 monclient(hunting): my global_id is 4156 2024-06-03T08:35:31.603 INFO:teuthology.orchestra.run.smithi192.stderr:2024-06-03T08:35:31.599+0000 7f6feb7fe640 1 -- 172.21.15.192:0/2274270257 --> v1:172.21.15.192:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f6ff4145680 con 0x7f6ff414ae20 2024-06-03T08:35:31.603 INFO:teuthology.orchestra.run.smithi192.stderr:2024-06-03T08:35:31.599+0000 7f6feb7fe640 1 -- 172.21.15.192:0/2274270257 <== mon.0 v1:172.21.15.88:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 746317117 0 0) 0x7f6ff41453e0 con 0x7f6ff413e6c0 2024-06-03T08:35:31.603 INFO:teuthology.orchestra.run.smithi192.stderr:2024-06-03T08:35:31.599+0000 7f6feb7fe640 10 monclient(hunting): _negotiate 2024-06-03T08:35:31.603 INFO:teuthology.orchestra.run.smithi192.stderr:2024-06-03T08:35:31.599+0000 7f6feb7fe640 10 monclient(hunting): _init_auth method 2 2024-06-03T08:35:31.603 INFO:teuthology.orchestra.run.smithi192.stderr:2024-06-03T08:35:31.599+0000 7f6feb7fe640 10 monclient(hunting): _init_auth creating new auth 2024-06-03T08:35:31.603 INFO:teuthology.orchestra.run.smithi192.stderr:2024-06-03T08:35:31.599+0000 7f6feb7fe640 10 monclient(hunting): my global_id is 4155 2024-06-03T08:35:31.603 INFO:teuthology.orchestra.run.smithi192.stderr:2024-06-03T08:35:31.599+0000 7f6feb7fe640 1 -- 172.21.15.192:0/2274270257 --> v1:172.21.15.88:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f6ff41385d0 con 0x7f6ff413e6c0 2024-06-03T08:35:31.604 INFO:teuthology.orchestra.run.smithi192.stderr:2024-06-03T08:35:31.603+0000 7f6feb7fe640 1 -- 172.21.15.192:0/2274270257 <== mon.2 v1:172.21.15.88:6790/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 764+0+0 (unknown 281540922 0 0) 0x7f6fd4004eb0 con 0x7f6ff40b43c0 2024-06-03T08:35:31.604 INFO:teuthology.orchestra.run.smithi192.stderr:2024-06-03T08:35:31.603+0000 7f6feb7fe640 1 -- 172.21.15.192:0/2274270257 --> v1:172.21.15.88:6790/0 -- auth(proto 2 165 bytes epoch 0) v1 -- 0x7f6ff41453e0 con 0x7f6ff40b43c0 2024-06-03T08:35:31.604 INFO:teuthology.orchestra.run.smithi192.stderr:2024-06-03T08:35:31.603+0000 7f6feb7fe640 1 -- 172.21.15.192:0/2274270257 <== mon.2 v1:172.21.15.88:6790/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 2234374065 0 0) 0x7f6fe40018b0 con 0x7f6ff40b43c0 2024-06-03T08:35:31.604 INFO:teuthology.orchestra.run.smithi192.stderr:2024-06-03T08:35:31.603+0000 7f6feb7fe640 10 monclient(hunting): handle_monmap mon_map magic: 0 v1 2024-06-03T08:35:31.604 INFO:teuthology.orchestra.run.smithi192.stderr:2024-06-03T08:35:31.603+0000 7f6feb7fe640 10 monclient(hunting): got monmap 1 from mon.noname-c (according to old e1) 2024-06-03T08:35:31.605 INFO:teuthology.orchestra.run.smithi192.stderr:2024-06-03T08:35:31.603+0000 7f6feb7fe640 10 monclient(hunting): dump: 2024-06-03T08:35:31.605 INFO:teuthology.orchestra.run.smithi192.stderr:epoch 1 2024-06-03T08:35:31.605 INFO:teuthology.orchestra.run.smithi192.stderr:fsid ec3c3fdd-3ac6-4d7d-9412-683130ed8b04 2024-06-03T08:35:31.605 INFO:teuthology.orchestra.run.smithi192.stderr:last_changed 2024-06-03T08:34:31.885125+0000 2024-06-03T08:35:31.605 INFO:teuthology.orchestra.run.smithi192.stderr:created 2024-06-03T08:34:31.885125+0000 2024-06-03T08:35:31.605 INFO:teuthology.orchestra.run.smithi192.stderr:min_mon_release 18 (reef) 2024-06-03T08:35:31.605 INFO:teuthology.orchestra.run.smithi192.stderr:election_strategy: 3 2024-06-03T08:35:31.605 INFO:teuthology.orchestra.run.smithi192.stderr:0: v1:172.21.15.88:6789/0 mon.a 2024-06-03T08:35:31.605 INFO:teuthology.orchestra.run.smithi192.stderr:1: v1:172.21.15.192:6789/0 mon.b 2024-06-03T08:35:31.605 INFO:teuthology.orchestra.run.smithi192.stderr:2: v1:172.21.15.88:6790/0 mon.c 2024-06-03T08:35:31.606 INFO:teuthology.orchestra.run.smithi192.stderr: 2024-06-03T08:35:31.606 INFO:teuthology.orchestra.run.smithi192.stderr:2024-06-03T08:35:31.603+0000 7f6feb7fe640 10 monclient(hunting): _finish_auth 0 2024-06-03T08:35:31.606 INFO:teuthology.orchestra.run.smithi192.stderr:2024-06-03T08:35:31.604+0000 7f6feb7fe640 1 -- 172.21.15.192:0/2274270257 <== mon.1 v1:172.21.15.192:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 764+0+0 (unknown 670674272 0 0) 0x7f6ff4145680 con 0x7f6ff414ae20 2024-06-03T08:35:31.606 INFO:teuthology.orchestra.run.smithi192.stderr:2024-06-03T08:35:31.604+0000 7f6feb7fe640 1 -- 172.21.15.192:0/2274270257 --> v1:172.21.15.192:6789/0 -- auth(proto 2 165 bytes epoch 0) v1 -- 0x7f6fd4004eb0 con 0x7f6ff414ae20 2024-06-03T08:35:31.606 INFO:teuthology.orchestra.run.smithi192.stderr:2024-06-03T08:35:31.604+0000 7f6feb7fe640 1 -- 172.21.15.192:0/2274270257 <== mon.1 v1:172.21.15.192:6789/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 2234374065 0 0) 0x7f6fec001ec0 con 0x7f6ff414ae20 2024-06-03T08:35:31.606 INFO:teuthology.orchestra.run.smithi192.stderr:2024-06-03T08:35:31.604+0000 7f6feb7fe640 10 monclient(hunting): handle_monmap mon_map magic: 0 v1 2024-06-03T08:35:31.606 INFO:teuthology.orchestra.run.smithi192.stderr:2024-06-03T08:35:31.604+0000 7f6feb7fe640 10 monclient(hunting): got monmap 1 from mon.b (according to old e1) 2024-06-03T08:35:31.606 INFO:teuthology.orchestra.run.smithi192.stderr:2024-06-03T08:35:31.604+0000 7f6feb7fe640 10 monclient(hunting): dump: 2024-06-03T08:35:31.606 INFO:teuthology.orchestra.run.smithi192.stderr:epoch 1 2024-06-03T08:35:31.606 INFO:teuthology.orchestra.run.smithi192.stderr:fsid ec3c3fdd-3ac6-4d7d-9412-683130ed8b04 2024-06-03T08:35:31.606 INFO:teuthology.orchestra.run.smithi192.stderr:last_changed 2024-06-03T08:34:31.885125+0000 2024-06-03T08:35:31.606 INFO:teuthology.orchestra.run.smithi192.stderr:created 2024-06-03T08:34:31.885125+0000 2024-06-03T08:35:31.606 INFO:teuthology.orchestra.run.smithi192.stderr:min_mon_release 18 (reef) 2024-06-03T08:35:31.606 INFO:teuthology.orchestra.run.smithi192.stderr:election_strategy: 3 2024-06-03T08:35:31.607 INFO:teuthology.orchestra.run.smithi192.stderr:0: v1:172.21.15.88:6789/0 mon.a 2024-06-03T08:35:31.607 INFO:teuthology.orchestra.run.smithi192.stderr:1: v1:172.21.15.192:6789/0 mon.b 2024-06-03T08:35:31.607 INFO:teuthology.orchestra.run.smithi192.stderr:2: v1:172.21.15.88:6790/0 mon.c 2024-06-03T08:35:31.607 INFO:teuthology.orchestra.run.smithi192.stderr: 2024-06-03T08:35:31.607 INFO:teuthology.orchestra.run.smithi192.stderr:2024-06-03T08:35:31.604+0000 7f6feb7fe640 1 -- 172.21.15.192:0/2274270257 <== mon.0 v1:172.21.15.88:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 764+0+0 (unknown 4145505215 0 0) 0x7f6ff41385d0 con 0x7f6ff413e6c0 2024-06-03T08:35:31.607 INFO:teuthology.orchestra.run.smithi192.stderr:2024-06-03T08:35:31.604+0000 7f6feb7fe640 1 -- 172.21.15.192:0/2274270257 --> v1:172.21.15.88:6789/0 -- auth(proto 2 165 bytes epoch 0) v1 -- 0x7f6ff4145680 con 0x7f6ff413e6c0 2024-06-03T08:35:31.607 INFO:teuthology.orchestra.run.smithi192.stderr:2024-06-03T08:35:31.604+0000 7f6feb7fe640 1 -- 172.21.15.192:0/2274270257 <== mon.0 v1:172.21.15.88:6789/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 2234374065 0 0) 0x7f6fdc0018b0 con 0x7f6ff413e6c0 2024-06-03T08:35:31.607 INFO:teuthology.orchestra.run.smithi192.stderr:2024-06-03T08:35:31.604+0000 7f6feb7fe640 10 monclient(hunting): handle_monmap mon_map magic: 0 v1 2024-06-03T08:35:31.607 INFO:teuthology.orchestra.run.smithi192.stderr:2024-06-03T08:35:31.604+0000 7f6feb7fe640 10 monclient(hunting): got monmap 1 from mon.a (according to old e1) 2024-06-03T08:35:31.607 INFO:teuthology.orchestra.run.smithi192.stderr:2024-06-03T08:35:31.604+0000 7f6feb7fe640 10 monclient(hunting): dump: 2024-06-03T08:35:31.607 INFO:teuthology.orchestra.run.smithi192.stderr:epoch 1 2024-06-03T08:35:31.607 INFO:teuthology.orchestra.run.smithi192.stderr:fsid ec3c3fdd-3ac6-4d7d-9412-683130ed8b04 2024-06-03T08:35:31.607 INFO:teuthology.orchestra.run.smithi192.stderr:last_changed 2024-06-03T08:34:31.885125+0000 2024-06-03T08:35:31.607 INFO:teuthology.orchestra.run.smithi192.stderr:created 2024-06-03T08:34:31.885125+0000 2024-06-03T08:35:31.608 INFO:teuthology.orchestra.run.smithi192.stderr:min_mon_release 18 (reef) 2024-06-03T08:35:31.608 INFO:teuthology.orchestra.run.smithi192.stderr:election_strategy: 3 2024-06-03T08:35:31.608 INFO:teuthology.orchestra.run.smithi192.stderr:0: v1:172.21.15.88:6789/0 mon.a 2024-06-03T08:35:31.608 INFO:teuthology.orchestra.run.smithi192.stderr:1: v1:172.21.15.192:6789/0 mon.b 2024-06-03T08:35:31.608 INFO:teuthology.orchestra.run.smithi192.stderr:2: v1:172.21.15.88:6790/0 mon.c 2024-06-03T08:35:31.608 INFO:teuthology.orchestra.run.smithi192.stderr: 2024-06-03T08:35:31.609 INFO:teuthology.orchestra.run.smithi192.stderr:2024-06-03T08:35:31.607+0000 7f6feb7fe640 1 -- 172.21.15.192:0/2274270257 <== mon.2 v1:172.21.15.88:6790/0 4 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 580+0+0 (unknown 1733827657 0 0) 0x7f6ff41453e0 con 0x7f6ff40b43c0 2024-06-03T08:35:31.609 INFO:teuthology.orchestra.run.smithi192.stderr:2024-06-03T08:35:31.607+0000 7f6feb7fe640 1 -- 172.21.15.192:0/2274270257 >> v1:172.21.15.192:6789/0 conn(0x7f6ff414ae20 legacy=0x7f6ff41d9ab0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-06-03T08:35:31.609 INFO:teuthology.orchestra.run.smithi192.stderr:2024-06-03T08:35:31.607+0000 7f6feb7fe640 1 -- 172.21.15.192:0/2274270257 >> v1:172.21.15.88:6789/0 conn(0x7f6ff413e6c0 legacy=0x7f6ff4135200 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-06-03T08:35:31.609 INFO:teuthology.orchestra.run.smithi192.stderr:2024-06-03T08:35:31.607+0000 7f6feb7fe640 10 monclient: _finish_hunting 0 2024-06-03T08:35:31.609 INFO:teuthology.orchestra.run.smithi192.stderr:2024-06-03T08:35:31.607+0000 7f6feb7fe640 1 monclient: found mon.c 2024-06-03T08:35:31.609 INFO:teuthology.orchestra.run.smithi192.stderr:2024-06-03T08:35:31.607+0000 7f6feb7fe640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-06-03T08:35:31.609 INFO:teuthology.orchestra.run.smithi192.stderr:2024-06-03T08:35:31.607+0000 7f6feb7fe640 10 monclient: _send_mon_message to mon.c at v1:172.21.15.88:6790/0 2024-06-03T08:35:31.610 INFO:teuthology.orchestra.run.smithi192.stderr:2024-06-03T08:35:31.607+0000 7f6feb7fe640 1 -- 172.21.15.192:0/2274270257 --> v1:172.21.15.88:6790/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f6ff4143e10 con 0x7f6ff40b43c0 2024-06-03T08:35:31.610 INFO:teuthology.orchestra.run.smithi192.stderr:2024-06-03T08:35:31.607+0000 7f6feb7fe640 10 monclient: _finish_auth 0 2024-06-03T08:35:31.610 INFO:teuthology.orchestra.run.smithi192.stderr:2024-06-03T08:35:31.607+0000 7f6feb7fe640 10 monclient: _check_auth_tickets 2024-06-03T08:35:31.610 INFO:teuthology.orchestra.run.smithi192.stderr:2024-06-03T08:35:31.607+0000 7f6feb7fe640 20 monclient: _check_auth_rotating not needed by client.admin 2024-06-03T08:35:31.610 INFO:teuthology.orchestra.run.smithi192.stderr:2024-06-03T08:35:31.607+0000 7f6ff92a8640 5 monclient: authenticate success, global_id 4157 2024-06-03T08:35:31.610 INFO:teuthology.orchestra.run.smithi192.stderr:2024-06-03T08:35:31.607+0000 7f6ff92a8640 10 monclient: _renew_subs 2024-06-03T08:35:31.610 INFO:teuthology.orchestra.run.smithi192.stderr:2024-06-03T08:35:31.607+0000 7f6ff92a8640 10 monclient: _send_mon_message to mon.c at v1:172.21.15.88:6790/0 2024-06-03T08:35:31.610 INFO:teuthology.orchestra.run.smithi192.stderr:2024-06-03T08:35:31.607+0000 7f6ff92a8640 1 -- 172.21.15.192:0/2274270257 --> v1:172.21.15.88:6790/0 -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f6ff41430d0 con 0x7f6ff40b43c0 2024-06-03T08:35:31.610 INFO:teuthology.orchestra.run.smithi192.stderr:2024-06-03T08:35:31.607+0000 7f6ff92a8640 10 client.4157.objecter _maybe_request_map subscribing (onetime) to next osd map 2024-06-03T08:35:31.610 INFO:teuthology.orchestra.run.smithi192.stderr:2024-06-03T08:35:31.607+0000 7f6ff92a8640 10 monclient: _renew_subs 2024-06-03T08:35:31.610 INFO:teuthology.orchestra.run.smithi192.stderr:2024-06-03T08:35:31.607+0000 7f6ff92a8640 10 monclient: _send_mon_message to mon.c at v1:172.21.15.88:6790/0 2024-06-03T08:35:31.610 INFO:teuthology.orchestra.run.smithi192.stderr:2024-06-03T08:35:31.607+0000 7f6ff92a8640 1 -- 172.21.15.192:0/2274270257 --> v1:172.21.15.88:6790/0 -- mon_subscribe({osdmap=0}) v3 -- 0x7f6ff4135af0 con 0x7f6ff40b43c0 2024-06-03T08:35:31.610 INFO:teuthology.orchestra.run.smithi192.stderr:2024-06-03T08:35:31.607+0000 7f6ff92a8640 1 librados: init done 2024-06-03T08:35:31.611 INFO:teuthology.orchestra.run.smithi192.stderr:2024-06-03T08:35:31.608+0000 7f6ff92a8640 10 start_mon_command cmd=[{"prefix": "get_command_descriptions"}] 2024-06-03T08:35:31.611 INFO:teuthology.orchestra.run.smithi192.stderr:2024-06-03T08:35:31.608+0000 7f6ff92a8640 10 monclient: _send_command 1 [{"prefix": "get_command_descriptions"}] 2024-06-03T08:35:31.611 INFO:teuthology.orchestra.run.smithi192.stderr:2024-06-03T08:35:31.608+0000 7f6ff92a8640 10 monclient: _send_mon_message to mon.c at v1:172.21.15.88:6790/0 2024-06-03T08:35:31.611 INFO:teuthology.orchestra.run.smithi192.stderr:2024-06-03T08:35:31.608+0000 7f6ff92a8640 1 -- 172.21.15.192:0/2274270257 --> v1:172.21.15.88:6790/0 -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f6ff41432c0 con 0x7f6ff40b43c0 2024-06-03T08:35:31.611 INFO:tasks.ceph.mgr.x.smithi192.stderr:2024-06-03T08:35:31.610+0000 7f0e6be43040 -1 mgr[py] Module osd_perf_query has missing NOTIFY_TYPES member 2024-06-03T08:35:31.612 INFO:teuthology.orchestra.run.smithi192.stderr:2024-06-03T08:35:31.611+0000 7f6feb7fe640 1 -- 172.21.15.192:0/2274270257 <== mon.2 v1:172.21.15.88:6790/0 5 ==== config(0 keys) v1 ==== 4+0+0 (unknown 0 0 0) 0x7f6fe40047e0 con 0x7f6ff40b43c0 2024-06-03T08:35:31.612 INFO:teuthology.orchestra.run.smithi192.stderr:2024-06-03T08:35:31.611+0000 7f6feb7fe640 10 monclient: handle_config config(0 keys) v1 2024-06-03T08:35:31.612 INFO:teuthology.orchestra.run.smithi192.stderr:2024-06-03T08:35:31.611+0000 7f6feb7fe640 1 -- 172.21.15.192:0/2274270257 <== mon.2 v1:172.21.15.88:6790/0 6 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 2234374065 0 0) 0x7f6fe4001d80 con 0x7f6ff40b43c0 2024-06-03T08:35:31.612 INFO:teuthology.orchestra.run.smithi192.stderr:2024-06-03T08:35:31.611+0000 7f6feb7fe640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-06-03T08:35:31.612 INFO:teuthology.orchestra.run.smithi192.stderr:2024-06-03T08:35:31.611+0000 7f6feb7fe640 10 monclient: got monmap 1 from mon.c (according to old e1) 2024-06-03T08:35:31.612 INFO:teuthology.orchestra.run.smithi192.stderr:2024-06-03T08:35:31.611+0000 7f6feb7fe640 10 monclient: dump: 2024-06-03T08:35:31.612 INFO:teuthology.orchestra.run.smithi192.stderr:epoch 1 2024-06-03T08:35:31.612 INFO:teuthology.orchestra.run.smithi192.stderr:fsid ec3c3fdd-3ac6-4d7d-9412-683130ed8b04 2024-06-03T08:35:31.612 INFO:teuthology.orchestra.run.smithi192.stderr:last_changed 2024-06-03T08:34:31.885125+0000 2024-06-03T08:35:31.612 INFO:teuthology.orchestra.run.smithi192.stderr:created 2024-06-03T08:34:31.885125+0000 2024-06-03T08:35:31.613 INFO:teuthology.orchestra.run.smithi192.stderr:min_mon_release 18 (reef) 2024-06-03T08:35:31.613 INFO:teuthology.orchestra.run.smithi192.stderr:election_strategy: 3 2024-06-03T08:35:31.613 INFO:teuthology.orchestra.run.smithi192.stderr:0: v1:172.21.15.88:6789/0 mon.a 2024-06-03T08:35:31.613 INFO:teuthology.orchestra.run.smithi192.stderr:1: v1:172.21.15.192:6789/0 mon.b 2024-06-03T08:35:31.613 INFO:teuthology.orchestra.run.smithi192.stderr:2: v1:172.21.15.88:6790/0 mon.c 2024-06-03T08:35:31.613 INFO:teuthology.orchestra.run.smithi192.stderr: 2024-06-03T08:35:31.613 INFO:teuthology.orchestra.run.smithi192.stderr:2024-06-03T08:35:31.611+0000 7f6feb7fe640 1 -- 172.21.15.192:0/2274270257 <== mon.2 v1:172.21.15.88:6790/0 7 ==== mgrmap(e 1) v1 ==== 668+0+0 (unknown 3836259515 0 0) 0x7f6fe4004e70 con 0x7f6ff40b43c0 2024-06-03T08:35:31.613 INFO:teuthology.orchestra.run.smithi192.stderr:2024-06-03T08:35:31.611+0000 7f6feb7fe640 1 -- 172.21.15.192:0/2274270257 <== mon.2 v1:172.21.15.88:6790/0 8 ==== osd_map(8..8 src has 1..8) v4 ==== 1415+0+0 (unknown 2855714161 0 0) 0x7f6fe4006390 con 0x7f6ff40b43c0 2024-06-03T08:35:31.613 INFO:teuthology.orchestra.run.smithi192.stderr:2024-06-03T08:35:31.611+0000 7f6feb7fe640 10 client.4157.objecter ms_dispatch 0x7f6ff4000b90 osd_map(8..8 src has 1..8) v4 2024-06-03T08:35:31.613 INFO:teuthology.orchestra.run.smithi192.stderr:2024-06-03T08:35:31.611+0000 7f6feb7fe640 3 client.4157.objecter handle_osd_map got epochs [8,8] > 0 2024-06-03T08:35:31.613 INFO:teuthology.orchestra.run.smithi192.stderr:2024-06-03T08:35:31.611+0000 7f6feb7fe640 3 client.4157.objecter handle_osd_map decoding full epoch 8 2024-06-03T08:35:31.613 INFO:teuthology.orchestra.run.smithi192.stderr:2024-06-03T08:35:31.611+0000 7f6ff17fa640 4 set_mon_vals no callback set 2024-06-03T08:35:31.613 INFO:teuthology.orchestra.run.smithi192.stderr:2024-06-03T08:35:31.611+0000 7f6feb7fe640 20 client.4157.objecter dump_active .. 0 homeless 2024-06-03T08:35:31.680 INFO:tasks.ceph.mgr.x.smithi192.stderr:2024-06-03T08:35:31.680+0000 7f0e6be43040 -1 mgr[py] Module rbd_support has missing NOTIFY_TYPES member 2024-06-03T08:35:31.688 INFO:tasks.ceph.mgr.y.smithi088.stderr:2024-06-03T08:35:31.687+0000 7ff0aea1d040 -1 mgr[py] Module diskprediction_local has missing NOTIFY_TYPES member 2024-06-03T08:35:31.706 INFO:teuthology.orchestra.run.smithi192.stderr:2024-06-03T08:35:31.706+0000 7f6feb7fe640 1 -- 172.21.15.192:0/2274270257 <== mon.2 v1:172.21.15.88:6790/0 9 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+73121 (unknown 1092875540 0 1417653156) 0x7f6fe4005850 con 0x7f6ff40b43c0 2024-06-03T08:35:31.707 INFO:teuthology.orchestra.run.smithi192.stderr:2024-06-03T08:35:31.706+0000 7f6feb7fe640 10 monclient: handle_mon_command_ack 1 [{"prefix": "get_command_descriptions"}] 2024-06-03T08:35:31.707 INFO:teuthology.orchestra.run.smithi192.stderr:2024-06-03T08:35:31.706+0000 7f6feb7fe640 10 monclient: _finish_command 1 = system:0 2024-06-03T08:35:31.725 INFO:tasks.ceph.mgr.x.smithi192.stderr:2024-06-03T08:35:31.725+0000 7f0e6be43040 -1 mgr[py] Module zabbix has missing NOTIFY_TYPES member 2024-06-03T08:35:31.738 INFO:tasks.ceph.mgr.y.smithi088.stderr:2024-06-03T08:35:31.738+0000 7ff0aea1d040 -1 mgr[py] Module osd_perf_query has missing NOTIFY_TYPES member 2024-06-03T08:35:31.778 INFO:teuthology.orchestra.run.smithi192.stderr:2024-06-03T08:35:31.778+0000 7f6ff92a8640 10 start_mon_command cmd=[{"prefix": "osd new", "uuid": "82c83316-809f-4901-a5ed-34b6a075b393", "id": 6}] 2024-06-03T08:35:31.778 INFO:teuthology.orchestra.run.smithi192.stderr:2024-06-03T08:35:31.778+0000 7f6ff92a8640 10 monclient: _send_command 2 [{"prefix": "osd new", "uuid": "82c83316-809f-4901-a5ed-34b6a075b393", "id": 6}] 2024-06-03T08:35:31.778 INFO:teuthology.orchestra.run.smithi192.stderr:2024-06-03T08:35:31.778+0000 7f6ff92a8640 10 monclient: _send_mon_message to mon.c at v1:172.21.15.88:6790/0 2024-06-03T08:35:31.778 INFO:teuthology.orchestra.run.smithi192.stderr:2024-06-03T08:35:31.778+0000 7f6ff92a8640 1 -- 172.21.15.192:0/2274270257 --> v1:172.21.15.88:6790/0 -- mon_command({"prefix": "osd new", "uuid": "82c83316-809f-4901-a5ed-34b6a075b393", "id": 6} v 0) v1 -- 0x7f6ff4145810 con 0x7f6ff40b43c0 2024-06-03T08:35:31.809 INFO:tasks.ceph.mgr.y.smithi088.stderr:2024-06-03T08:35:31.809+0000 7ff0aea1d040 -1 mgr[py] Module rbd_support has missing NOTIFY_TYPES member 2024-06-03T08:35:31.844 INFO:teuthology.orchestra.run.smithi192.stderr:2024-06-03T08:35:31.844+0000 7f6feb7fe640 1 -- 172.21.15.192:0/2274270257 <== mon.2 v1:172.21.15.88:6790/0 10 ==== mon_command_ack([{"prefix": "osd new", "uuid": "82c83316-809f-4901-a5ed-34b6a075b393", "id": 6}]=0 v9) v1 ==== 112+0+2 (unknown 3562586486 0 1113811849) 0x7f6fe4005f20 con 0x7f6ff40b43c0 2024-06-03T08:35:31.844 INFO:teuthology.orchestra.run.smithi192.stderr:2024-06-03T08:35:31.844+0000 7f6feb7fe640 10 monclient: handle_mon_command_ack 2 [{"prefix": "osd new", "uuid": "82c83316-809f-4901-a5ed-34b6a075b393", "id": 6}] 2024-06-03T08:35:31.844 INFO:teuthology.orchestra.run.smithi192.stderr:2024-06-03T08:35:31.844+0000 7f6feb7fe640 10 monclient: _finish_command 2 = system:0 2024-06-03T08:35:31.845 INFO:teuthology.orchestra.run.smithi192.stdout:6 2024-06-03T08:35:31.846 INFO:teuthology.orchestra.run.smithi192.stderr:2024-06-03T08:35:31.845+0000 7f6ff92a8640 10 librados: watch_flush enter 2024-06-03T08:35:31.846 INFO:teuthology.orchestra.run.smithi192.stderr:2024-06-03T08:35:31.845+0000 7f6ff92a8640 10 librados: watch_flush exit 2024-06-03T08:35:31.846 INFO:teuthology.orchestra.run.smithi192.stderr:2024-06-03T08:35:31.845+0000 7f6ff92a8640 20 client.4157.objecter shutdown clearing up homeless session... 2024-06-03T08:35:31.846 INFO:teuthology.orchestra.run.smithi192.stderr:2024-06-03T08:35:31.845+0000 7f6ff92a8640 10 client.4157.objecter successfully canceled tick 2024-06-03T08:35:31.846 INFO:teuthology.orchestra.run.smithi192.stderr:2024-06-03T08:35:31.845+0000 7f6ff92a8640 10 monclient: shutdown 2024-06-03T08:35:31.846 INFO:teuthology.orchestra.run.smithi192.stderr:2024-06-03T08:35:31.845+0000 7f6ff92a8640 20 monclient: shutdown discarding 0 pending message(s) 2024-06-03T08:35:31.846 INFO:teuthology.orchestra.run.smithi192.stderr:2024-06-03T08:35:31.845+0000 7f6ff92a8640 1 -- 172.21.15.192:0/2274270257 >> v1:172.21.15.88:6790/0 conn(0x7f6ff40b43c0 legacy=0x7f6ff4143700 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-06-03T08:35:31.846 INFO:teuthology.orchestra.run.smithi192.stderr:2024-06-03T08:35:31.845+0000 7f6ff92a8640 1 -- 172.21.15.192:0/2274270257 shutdown_connections 2024-06-03T08:35:31.846 INFO:teuthology.orchestra.run.smithi192.stderr:2024-06-03T08:35:31.845+0000 7f6ff92a8640 1 -- 172.21.15.192:0/2274270257 >> 172.21.15.192:0/2274270257 conn(0x7f6ff40719d0 msgr2=0x7f6ff4071db0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-06-03T08:35:31.847 INFO:teuthology.orchestra.run.smithi192.stderr:2024-06-03T08:35:31.845+0000 7f6ff92a8640 1 -- 172.21.15.192:0/2274270257 shutdown_connections 2024-06-03T08:35:31.847 INFO:teuthology.orchestra.run.smithi192.stderr:2024-06-03T08:35:31.845+0000 7f6ff92a8640 1 -- 172.21.15.192:0/2274270257 wait complete. 2024-06-03T08:35:31.847 INFO:teuthology.orchestra.run.smithi192.stderr:2024-06-03T08:35:31.845+0000 7f6ff92a8640 1 librados: shutdown 2024-06-03T08:35:31.855 DEBUG:teuthology.orchestra.run.smithi192:> sudo ceph --cluster ceph osd new aafe0716-1102-4eae-aac6-0f81bb154e7d 7 2024-06-03T08:35:31.857 INFO:tasks.ceph.mgr.y.smithi088.stderr:2024-06-03T08:35:31.857+0000 7ff0aea1d040 -1 mgr[py] Module zabbix has missing NOTIFY_TYPES member 2024-06-03T08:35:31.955 INFO:teuthology.orchestra.run.smithi192.stderr:2024-06-03T08:35:31.955+0000 7fc3a0470640 10 monclient: get_monmap_and_config 2024-06-03T08:35:31.955 INFO:teuthology.orchestra.run.smithi192.stderr:2024-06-03T08:35:31.955+0000 7fc3a0470640 10 monclient: build_initial_monmap 2024-06-03T08:35:31.955 INFO:teuthology.orchestra.run.smithi192.stderr:2024-06-03T08:35:31.955+0000 7fc3a0470640 10 monclient: monmap: 2024-06-03T08:35:31.956 INFO:teuthology.orchestra.run.smithi192.stderr:epoch 0 2024-06-03T08:35:31.956 INFO:teuthology.orchestra.run.smithi192.stderr:fsid ec3c3fdd-3ac6-4d7d-9412-683130ed8b04 2024-06-03T08:35:31.956 INFO:teuthology.orchestra.run.smithi192.stderr:last_changed 2024-06-03T08:35:31.956145+0000 2024-06-03T08:35:31.956 INFO:teuthology.orchestra.run.smithi192.stderr:created 2024-06-03T08:35:31.956145+0000 2024-06-03T08:35:31.956 INFO:teuthology.orchestra.run.smithi192.stderr:min_mon_release 0 (unknown) 2024-06-03T08:35:31.956 INFO:teuthology.orchestra.run.smithi192.stderr:election_strategy: 3 2024-06-03T08:35:31.956 INFO:teuthology.orchestra.run.smithi192.stderr:0: v1:172.21.15.88:6789/0 mon.noname-a 2024-06-03T08:35:31.956 INFO:teuthology.orchestra.run.smithi192.stderr:1: v1:172.21.15.192:6789/0 mon.noname-c 2024-06-03T08:35:31.956 INFO:teuthology.orchestra.run.smithi192.stderr:2: v1:172.21.15.88:6790/0 mon.noname-b 2024-06-03T08:35:31.956 INFO:teuthology.orchestra.run.smithi192.stderr: 2024-06-03T08:35:31.957 INFO:teuthology.orchestra.run.smithi192.stderr:2024-06-03T08:35:31.955+0000 7fc3a0470640 1 Processor -- start 2024-06-03T08:35:31.957 INFO:teuthology.orchestra.run.smithi192.stderr:2024-06-03T08:35:31.955+0000 7fc3a0470640 1 -- start start 2024-06-03T08:35:31.958 INFO:teuthology.orchestra.run.smithi192.stderr:2024-06-03T08:35:31.955+0000 7fc3a0470640 10 monclient: init 2024-06-03T08:35:31.958 INFO:teuthology.orchestra.run.smithi192.stderr:2024-06-03T08:35:31.956+0000 7fc3a0470640 10 monclient: _reopen_session rank -1 2024-06-03T08:35:31.958 INFO:teuthology.orchestra.run.smithi192.stderr:2024-06-03T08:35:31.956+0000 7fc3a0470640 10 monclient: _add_conns ranks=[2,1,0] 2024-06-03T08:35:31.958 INFO:teuthology.orchestra.run.smithi192.stderr:2024-06-03T08:35:31.956+0000 7fc3a0470640 10 monclient(hunting): picked mon.noname-b con 0x7fc3980b43c0 addr v1:172.21.15.88:6790/0 2024-06-03T08:35:31.958 INFO:teuthology.orchestra.run.smithi192.stderr:2024-06-03T08:35:31.956+0000 7fc3a0470640 10 monclient(hunting): picked mon.noname-c con 0x7fc39813e6c0 addr v1:172.21.15.192:6789/0 2024-06-03T08:35:31.958 INFO:teuthology.orchestra.run.smithi192.stderr:2024-06-03T08:35:31.956+0000 7fc3a0470640 10 monclient(hunting): picked mon.noname-a con 0x7fc39814ae20 addr v1:172.21.15.88:6789/0 2024-06-03T08:35:31.958 INFO:teuthology.orchestra.run.smithi192.stderr:2024-06-03T08:35:31.956+0000 7fc3a0470640 1 -- --> v1:172.21.15.88:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fc3980a8ed0 con 0x7fc39814ae20 2024-06-03T08:35:31.958 INFO:teuthology.orchestra.run.smithi192.stderr:2024-06-03T08:35:31.956+0000 7fc3a0470640 1 -- --> v1:172.21.15.192:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fc3980aa420 con 0x7fc39813e6c0 2024-06-03T08:35:31.958 INFO:teuthology.orchestra.run.smithi192.stderr:2024-06-03T08:35:31.956+0000 7fc3a0470640 1 -- --> v1:172.21.15.88:6790/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fc3980aa690 con 0x7fc3980b43c0 2024-06-03T08:35:31.958 INFO:teuthology.orchestra.run.smithi192.stderr:2024-06-03T08:35:31.956+0000 7fc3a0470640 10 monclient(hunting): _renew_subs 2024-06-03T08:35:31.958 INFO:teuthology.orchestra.run.smithi192.stderr:2024-06-03T08:35:31.956+0000 7fc3a0470640 10 monclient(hunting): authenticate will time out at 1197.767844s 2024-06-03T08:35:31.966 INFO:teuthology.orchestra.run.smithi192.stderr:2024-06-03T08:35:31.965+0000 7fc39e20c640 1 --1- >> v1:172.21.15.88:6790/0 conn(0x7fc3980b43c0 0x7fc3981415b0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.88:6790/0 says I am v1:172.21.15.192:60216/0 (socket says 172.21.15.192:60216) 2024-06-03T08:35:31.966 INFO:teuthology.orchestra.run.smithi192.stderr:2024-06-03T08:35:31.965+0000 7fc39e20c640 1 -- 172.21.15.192:0/3410776488 learned_addr learned my addr 172.21.15.192:0/3410776488 (peer_addr_for_me v1:172.21.15.192:0/0) 2024-06-03T08:35:31.977 INFO:teuthology.orchestra.run.smithi192.stderr:2024-06-03T08:35:31.977+0000 7fc39d20a640 1 -- 172.21.15.192:0/3410776488 <== mon.0 v1:172.21.15.88:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 3807416182 0 0) 0x7fc3980a8ed0 con 0x7fc39814ae20 2024-06-03T08:35:31.978 INFO:teuthology.orchestra.run.smithi192.stderr:2024-06-03T08:35:31.977+0000 7fc39d20a640 10 monclient(hunting): _negotiate 2024-06-03T08:35:31.978 INFO:teuthology.orchestra.run.smithi192.stderr:2024-06-03T08:35:31.977+0000 7fc39d20a640 10 monclient(hunting): _init_auth method 2 2024-06-03T08:35:31.978 INFO:teuthology.orchestra.run.smithi192.stderr:2024-06-03T08:35:31.977+0000 7fc39d20a640 10 monclient(hunting): _init_auth creating new auth 2024-06-03T08:35:31.978 INFO:teuthology.orchestra.run.smithi192.stderr:2024-06-03T08:35:31.977+0000 7fc39d20a640 10 monclient(hunting): my global_id is 4158 2024-06-03T08:35:31.978 INFO:teuthology.orchestra.run.smithi192.stderr:2024-06-03T08:35:31.977+0000 7fc39d20a640 1 -- 172.21.15.192:0/3410776488 --> v1:172.21.15.88:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7fc384003520 con 0x7fc39814ae20 2024-06-03T08:35:31.978 INFO:teuthology.orchestra.run.smithi192.stderr:2024-06-03T08:35:31.977+0000 7fc39d20a640 1 -- 172.21.15.192:0/3410776488 <== mon.1 v1:172.21.15.192:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 1566231143 0 0) 0x7fc3980aa420 con 0x7fc39813e6c0 2024-06-03T08:35:31.978 INFO:teuthology.orchestra.run.smithi192.stderr:2024-06-03T08:35:31.977+0000 7fc39d20a640 10 monclient(hunting): _negotiate 2024-06-03T08:35:31.978 INFO:teuthology.orchestra.run.smithi192.stderr:2024-06-03T08:35:31.977+0000 7fc39d20a640 10 monclient(hunting): _init_auth method 2 2024-06-03T08:35:31.978 INFO:teuthology.orchestra.run.smithi192.stderr:2024-06-03T08:35:31.977+0000 7fc39d20a640 10 monclient(hunting): _init_auth creating new auth 2024-06-03T08:35:31.978 INFO:teuthology.orchestra.run.smithi192.stderr:2024-06-03T08:35:31.977+0000 7fc39d20a640 10 monclient(hunting): my global_id is 4159 2024-06-03T08:35:31.978 INFO:teuthology.orchestra.run.smithi192.stderr:2024-06-03T08:35:31.977+0000 7fc39d20a640 1 -- 172.21.15.192:0/3410776488 --> v1:172.21.15.192:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7fc3980a8ed0 con 0x7fc39813e6c0 2024-06-03T08:35:31.979 INFO:teuthology.orchestra.run.smithi192.stderr:2024-06-03T08:35:31.978+0000 7fc39d20a640 1 -- 172.21.15.192:0/3410776488 <== mon.2 v1:172.21.15.88:6790/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 2398487543 0 0) 0x7fc3980aa690 con 0x7fc3980b43c0 2024-06-03T08:35:31.979 INFO:teuthology.orchestra.run.smithi192.stderr:2024-06-03T08:35:31.978+0000 7fc39d20a640 10 monclient(hunting): _negotiate 2024-06-03T08:35:31.979 INFO:teuthology.orchestra.run.smithi192.stderr:2024-06-03T08:35:31.978+0000 7fc39d20a640 10 monclient(hunting): _init_auth method 2 2024-06-03T08:35:31.979 INFO:teuthology.orchestra.run.smithi192.stderr:2024-06-03T08:35:31.978+0000 7fc39d20a640 10 monclient(hunting): _init_auth creating new auth 2024-06-03T08:35:31.979 INFO:teuthology.orchestra.run.smithi192.stderr:2024-06-03T08:35:31.978+0000 7fc39d20a640 10 monclient(hunting): my global_id is 4160 2024-06-03T08:35:31.979 INFO:teuthology.orchestra.run.smithi192.stderr:2024-06-03T08:35:31.978+0000 7fc39d20a640 1 -- 172.21.15.192:0/3410776488 --> v1:172.21.15.88:6790/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7fc3980aa420 con 0x7fc3980b43c0 2024-06-03T08:35:31.982 INFO:teuthology.orchestra.run.smithi192.stderr:2024-06-03T08:35:31.981+0000 7fc39d20a640 1 -- 172.21.15.192:0/3410776488 <== mon.0 v1:172.21.15.88:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 214+0+0 (unknown 4241627956 0 0) 0x7fc384003520 con 0x7fc39814ae20 2024-06-03T08:35:31.982 INFO:teuthology.orchestra.run.smithi192.stderr:2024-06-03T08:35:31.981+0000 7fc39d20a640 1 -- 172.21.15.192:0/3410776488 >> v1:172.21.15.88:6790/0 conn(0x7fc3980b43c0 legacy=0x7fc3981415b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-06-03T08:35:31.982 INFO:teuthology.orchestra.run.smithi192.stderr:2024-06-03T08:35:31.981+0000 7fc39d20a640 1 -- 172.21.15.192:0/3410776488 >> v1:172.21.15.192:6789/0 conn(0x7fc39813e6c0 legacy=0x7fc39813eb40 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-06-03T08:35:31.982 INFO:teuthology.orchestra.run.smithi192.stderr:2024-06-03T08:35:31.981+0000 7fc39d20a640 10 monclient: _finish_hunting 0 2024-06-03T08:35:31.982 INFO:teuthology.orchestra.run.smithi192.stderr:2024-06-03T08:35:31.981+0000 7fc39d20a640 1 monclient: found mon.noname-a 2024-06-03T08:35:31.982 INFO:teuthology.orchestra.run.smithi192.stderr:2024-06-03T08:35:31.981+0000 7fc39d20a640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-06-03T08:35:31.982 INFO:teuthology.orchestra.run.smithi192.stderr:2024-06-03T08:35:31.981+0000 7fc39d20a640 10 monclient: _send_mon_message to mon.noname-a at v1:172.21.15.88:6789/0 2024-06-03T08:35:31.982 INFO:teuthology.orchestra.run.smithi192.stderr:2024-06-03T08:35:31.981+0000 7fc39d20a640 1 -- 172.21.15.192:0/3410776488 --> v1:172.21.15.88:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fc3980a87f0 con 0x7fc39814ae20 2024-06-03T08:35:31.982 INFO:teuthology.orchestra.run.smithi192.stderr:2024-06-03T08:35:31.981+0000 7fc39d20a640 10 monclient: _finish_auth 0 2024-06-03T08:35:31.982 INFO:teuthology.orchestra.run.smithi192.stderr:2024-06-03T08:35:31.981+0000 7fc39d20a640 10 monclient: _check_auth_tickets 2024-06-03T08:35:31.983 INFO:teuthology.orchestra.run.smithi192.stderr:2024-06-03T08:35:31.981+0000 7fc39d20a640 20 monclient: _check_auth_rotating not needed by client.admin 2024-06-03T08:35:31.983 INFO:teuthology.orchestra.run.smithi192.stderr:2024-06-03T08:35:31.981+0000 7fc39d20a640 1 -- 172.21.15.192:0/3410776488 <== mon.0 v1:172.21.15.88:6789/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 2234374065 0 0) 0x7fc390001990 con 0x7fc39814ae20 2024-06-03T08:35:31.983 INFO:teuthology.orchestra.run.smithi192.stderr:2024-06-03T08:35:31.981+0000 7fc3a0470640 5 monclient: authenticate success, global_id 4158 2024-06-03T08:35:31.983 INFO:teuthology.orchestra.run.smithi192.stderr:2024-06-03T08:35:31.981+0000 7fc3a0470640 20 monclient: get_monmap_and_config waiting for monmap|config 2024-06-03T08:35:31.983 INFO:teuthology.orchestra.run.smithi192.stderr:2024-06-03T08:35:31.981+0000 7fc39d20a640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-06-03T08:35:31.983 INFO:teuthology.orchestra.run.smithi192.stderr:2024-06-03T08:35:31.981+0000 7fc39d20a640 10 monclient: got monmap 1 from mon.noname-a (according to old e1) 2024-06-03T08:35:31.983 INFO:teuthology.orchestra.run.smithi192.stderr:2024-06-03T08:35:31.981+0000 7fc39d20a640 10 monclient: dump: 2024-06-03T08:35:31.983 INFO:teuthology.orchestra.run.smithi192.stderr:epoch 1 2024-06-03T08:35:31.983 INFO:teuthology.orchestra.run.smithi192.stderr:fsid ec3c3fdd-3ac6-4d7d-9412-683130ed8b04 2024-06-03T08:35:31.983 INFO:teuthology.orchestra.run.smithi192.stderr:last_changed 2024-06-03T08:34:31.885125+0000 2024-06-03T08:35:31.983 INFO:teuthology.orchestra.run.smithi192.stderr:created 2024-06-03T08:34:31.885125+0000 2024-06-03T08:35:31.983 INFO:teuthology.orchestra.run.smithi192.stderr:min_mon_release 18 (reef) 2024-06-03T08:35:31.983 INFO:teuthology.orchestra.run.smithi192.stderr:election_strategy: 3 2024-06-03T08:35:31.983 INFO:teuthology.orchestra.run.smithi192.stderr:0: v1:172.21.15.88:6789/0 mon.a 2024-06-03T08:35:31.984 INFO:teuthology.orchestra.run.smithi192.stderr:1: v1:172.21.15.192:6789/0 mon.b 2024-06-03T08:35:31.984 INFO:teuthology.orchestra.run.smithi192.stderr:2: v1:172.21.15.88:6790/0 mon.c 2024-06-03T08:35:31.984 INFO:teuthology.orchestra.run.smithi192.stderr: 2024-06-03T08:35:31.984 INFO:teuthology.orchestra.run.smithi192.stderr:2024-06-03T08:35:31.981+0000 7fc3a0470640 20 monclient: get_monmap_and_config waiting for monmap|config 2024-06-03T08:35:31.985 INFO:teuthology.orchestra.run.smithi192.stderr:2024-06-03T08:35:31.985+0000 7fc39d20a640 1 -- 172.21.15.192:0/3410776488 <== mon.0 v1:172.21.15.88:6789/0 4 ==== config(0 keys) v1 ==== 4+0+0 (unknown 0 0 0) 0x7fc390001bb0 con 0x7fc39814ae20 2024-06-03T08:35:31.985 INFO:teuthology.orchestra.run.smithi192.stderr:2024-06-03T08:35:31.985+0000 7fc39d20a640 10 monclient: handle_config config(0 keys) v1 2024-06-03T08:35:31.985 INFO:teuthology.orchestra.run.smithi192.stderr:2024-06-03T08:35:31.985+0000 7fc39d20a640 1 -- 172.21.15.192:0/3410776488 <== mon.0 v1:172.21.15.88:6789/0 5 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 2234374065 0 0) 0x7fc390001f40 con 0x7fc39814ae20 2024-06-03T08:35:31.985 INFO:teuthology.orchestra.run.smithi192.stderr:2024-06-03T08:35:31.985+0000 7fc3a0470640 10 monclient: get_monmap_and_config success 2024-06-03T08:35:31.985 INFO:teuthology.orchestra.run.smithi192.stderr:2024-06-03T08:35:31.985+0000 7fc3a0470640 4 set_mon_vals no callback set 2024-06-03T08:35:31.985 INFO:teuthology.orchestra.run.smithi192.stderr:2024-06-03T08:35:31.985+0000 7fc39d20a640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-06-03T08:35:31.986 INFO:teuthology.orchestra.run.smithi192.stderr:2024-06-03T08:35:31.985+0000 7fc39d20a640 10 monclient: got monmap 1 from mon.a (according to old e1) 2024-06-03T08:35:31.986 INFO:teuthology.orchestra.run.smithi192.stderr:2024-06-03T08:35:31.985+0000 7fc39d20a640 10 monclient: dump: 2024-06-03T08:35:31.986 INFO:teuthology.orchestra.run.smithi192.stderr:epoch 1 2024-06-03T08:35:31.986 INFO:teuthology.orchestra.run.smithi192.stderr:fsid ec3c3fdd-3ac6-4d7d-9412-683130ed8b04 2024-06-03T08:35:31.986 INFO:teuthology.orchestra.run.smithi192.stderr:last_changed 2024-06-03T08:34:31.885125+0000 2024-06-03T08:35:31.986 INFO:teuthology.orchestra.run.smithi192.stderr:created 2024-06-03T08:34:31.885125+0000 2024-06-03T08:35:31.986 INFO:teuthology.orchestra.run.smithi192.stderr:min_mon_release 18 (reef) 2024-06-03T08:35:31.986 INFO:teuthology.orchestra.run.smithi192.stderr:election_strategy: 3 2024-06-03T08:35:31.986 INFO:teuthology.orchestra.run.smithi192.stderr:0: v1:172.21.15.88:6789/0 mon.a 2024-06-03T08:35:31.986 INFO:teuthology.orchestra.run.smithi192.stderr:1: v1:172.21.15.192:6789/0 mon.b 2024-06-03T08:35:31.986 INFO:teuthology.orchestra.run.smithi192.stderr:2: v1:172.21.15.88:6790/0 mon.c 2024-06-03T08:35:31.986 INFO:teuthology.orchestra.run.smithi192.stderr: 2024-06-03T08:35:31.988 INFO:teuthology.orchestra.run.smithi192.stderr:2024-06-03T08:35:31.985+0000 7fc3a0470640 10 monclient: shutdown 2024-06-03T08:35:31.989 INFO:teuthology.orchestra.run.smithi192.stderr:2024-06-03T08:35:31.985+0000 7fc3a0470640 20 monclient: shutdown discarding 0 pending message(s) 2024-06-03T08:35:31.989 INFO:teuthology.orchestra.run.smithi192.stderr:2024-06-03T08:35:31.985+0000 7fc3a0470640 1 -- 172.21.15.192:0/3410776488 >> v1:172.21.15.88:6789/0 conn(0x7fc39814ae20 legacy=0x7fc39814d210 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-06-03T08:35:31.989 INFO:teuthology.orchestra.run.smithi192.stderr:2024-06-03T08:35:31.986+0000 7fc3a0470640 1 -- 172.21.15.192:0/3410776488 shutdown_connections 2024-06-03T08:35:31.989 INFO:teuthology.orchestra.run.smithi192.stderr:2024-06-03T08:35:31.986+0000 7fc3a0470640 1 -- 172.21.15.192:0/3410776488 >> 172.21.15.192:0/3410776488 conn(0x7fc3980719d0 msgr2=0x7fc398071de0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-06-03T08:35:31.989 INFO:teuthology.orchestra.run.smithi192.stderr:2024-06-03T08:35:31.986+0000 7fc3a0470640 1 -- 172.21.15.192:0/3410776488 shutdown_connections 2024-06-03T08:35:31.989 INFO:teuthology.orchestra.run.smithi192.stderr:2024-06-03T08:35:31.986+0000 7fc3a0470640 1 -- 172.21.15.192:0/3410776488 wait complete. 2024-06-03T08:35:31.989 INFO:teuthology.orchestra.run.smithi192.stderr:2024-06-03T08:35:31.986+0000 7fc3a0470640 10 monclient: build_initial_monmap 2024-06-03T08:35:31.989 INFO:teuthology.orchestra.run.smithi192.stderr:2024-06-03T08:35:31.986+0000 7fc3a0470640 10 monclient: monmap: 2024-06-03T08:35:31.989 INFO:teuthology.orchestra.run.smithi192.stderr:epoch 0 2024-06-03T08:35:31.989 INFO:teuthology.orchestra.run.smithi192.stderr:fsid 00000000-0000-0000-0000-000000000000 2024-06-03T08:35:31.989 INFO:teuthology.orchestra.run.smithi192.stderr:last_changed 0.000000 2024-06-03T08:35:31.989 INFO:teuthology.orchestra.run.smithi192.stderr:created 0.000000 2024-06-03T08:35:31.989 INFO:teuthology.orchestra.run.smithi192.stderr:min_mon_release 0 (unknown) 2024-06-03T08:35:31.989 INFO:teuthology.orchestra.run.smithi192.stderr:election_strategy: 1 2024-06-03T08:35:31.990 INFO:teuthology.orchestra.run.smithi192.stderr:0: v1:172.21.15.88:6789/0 mon.noname-a 2024-06-03T08:35:31.990 INFO:teuthology.orchestra.run.smithi192.stderr:1: v1:172.21.15.192:6789/0 mon.noname-b 2024-06-03T08:35:31.990 INFO:teuthology.orchestra.run.smithi192.stderr:2: v1:172.21.15.88:6790/0 mon.noname-c 2024-06-03T08:35:31.990 INFO:teuthology.orchestra.run.smithi192.stderr: 2024-06-03T08:35:31.990 INFO:teuthology.orchestra.run.smithi192.stderr:2024-06-03T08:35:31.986+0000 7fc3a0470640 1 librados: starting msgr at 2024-06-03T08:35:31.990 INFO:teuthology.orchestra.run.smithi192.stderr:2024-06-03T08:35:31.987+0000 7fc3a0470640 1 librados: starting objecter 2024-06-03T08:35:31.990 INFO:teuthology.orchestra.run.smithi192.stderr:2024-06-03T08:35:31.987+0000 7fc3a0470640 1 Processor -- start 2024-06-03T08:35:31.990 INFO:teuthology.orchestra.run.smithi192.stderr:2024-06-03T08:35:31.987+0000 7fc3a0470640 1 -- start start 2024-06-03T08:35:31.990 INFO:teuthology.orchestra.run.smithi192.stderr:2024-06-03T08:35:31.987+0000 7fc3a0470640 1 librados: setting wanted keys 2024-06-03T08:35:31.990 INFO:teuthology.orchestra.run.smithi192.stderr:2024-06-03T08:35:31.987+0000 7fc3a0470640 1 librados: calling monclient init 2024-06-03T08:35:31.990 INFO:teuthology.orchestra.run.smithi192.stderr:2024-06-03T08:35:31.987+0000 7fc3a0470640 10 monclient: init 2024-06-03T08:35:31.990 INFO:teuthology.orchestra.run.smithi192.stderr:2024-06-03T08:35:31.987+0000 7fc3a0470640 10 monclient: _reopen_session rank -1 2024-06-03T08:35:31.990 INFO:teuthology.orchestra.run.smithi192.stderr:2024-06-03T08:35:31.987+0000 7fc3a0470640 10 monclient: _add_conns ranks=[0,1,2] 2024-06-03T08:35:31.991 INFO:teuthology.orchestra.run.smithi192.stderr:2024-06-03T08:35:31.987+0000 7fc3a0470640 10 monclient(hunting): picked mon.noname-a con 0x7fc3980b43c0 addr v1:172.21.15.88:6789/0 2024-06-03T08:35:31.991 INFO:teuthology.orchestra.run.smithi192.stderr:2024-06-03T08:35:31.987+0000 7fc3a0470640 10 monclient(hunting): picked mon.noname-b con 0x7fc39813e6c0 addr v1:172.21.15.192:6789/0 2024-06-03T08:35:31.991 INFO:teuthology.orchestra.run.smithi192.stderr:2024-06-03T08:35:31.987+0000 7fc3a0470640 10 monclient(hunting): picked mon.noname-c con 0x7fc39814ae20 addr v1:172.21.15.88:6790/0 2024-06-03T08:35:31.991 INFO:teuthology.orchestra.run.smithi192.stderr:2024-06-03T08:35:31.987+0000 7fc3a0470640 1 -- --> v1:172.21.15.88:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fc39813f3b0 con 0x7fc3980b43c0 2024-06-03T08:35:31.991 INFO:teuthology.orchestra.run.smithi192.stderr:2024-06-03T08:35:31.987+0000 7fc3a0470640 1 -- --> v1:172.21.15.192:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fc398143e30 con 0x7fc39813e6c0 2024-06-03T08:35:31.991 INFO:teuthology.orchestra.run.smithi192.stderr:2024-06-03T08:35:31.987+0000 7fc3a0470640 1 -- --> v1:172.21.15.88:6790/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fc398142b30 con 0x7fc39814ae20 2024-06-03T08:35:31.991 INFO:teuthology.orchestra.run.smithi192.stderr:2024-06-03T08:35:31.987+0000 7fc3a0470640 10 monclient(hunting): _renew_subs 2024-06-03T08:35:31.991 INFO:teuthology.orchestra.run.smithi192.stderr:2024-06-03T08:35:31.987+0000 7fc3a0470640 10 monclient(hunting): authenticate will time out at 1197.798811s 2024-06-03T08:35:31.991 INFO:teuthology.orchestra.run.smithi192.stderr:2024-06-03T08:35:31.988+0000 7fc39da0b640 1 --1- >> v1:172.21.15.192:6789/0 conn(0x7fc39813e6c0 0x7fc398135300 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=1).handle_server_banner_and_identify peer v1:172.21.15.192:6789/0 says I am v1:172.21.15.192:43938/0 (socket says 172.21.15.192:43938) 2024-06-03T08:35:31.991 INFO:teuthology.orchestra.run.smithi192.stderr:2024-06-03T08:35:31.988+0000 7fc39da0b640 1 -- 172.21.15.192:0/3424013783 learned_addr learned my addr 172.21.15.192:0/3424013783 (peer_addr_for_me v1:172.21.15.192:0/0) 2024-06-03T08:35:31.991 INFO:teuthology.orchestra.run.smithi192.stderr:2024-06-03T08:35:31.988+0000 7fc38effd640 10 client.?.objecter ms_handle_connect 0x7fc39813e6c0 2024-06-03T08:35:31.991 INFO:teuthology.orchestra.run.smithi192.stderr:2024-06-03T08:35:31.988+0000 7fc38effd640 10 client.?.objecter resend_mon_ops 2024-06-03T08:35:31.991 INFO:teuthology.orchestra.run.smithi192.stderr:2024-06-03T08:35:31.988+0000 7fc38effd640 10 client.?.objecter ms_handle_connect 0x7fc39814ae20 2024-06-03T08:35:31.991 INFO:teuthology.orchestra.run.smithi192.stderr:2024-06-03T08:35:31.988+0000 7fc38effd640 10 client.?.objecter resend_mon_ops 2024-06-03T08:35:31.991 INFO:teuthology.orchestra.run.smithi192.stderr:2024-06-03T08:35:31.988+0000 7fc38effd640 10 client.?.objecter ms_handle_connect 0x7fc3980b43c0 2024-06-03T08:35:31.992 INFO:teuthology.orchestra.run.smithi192.stderr:2024-06-03T08:35:31.988+0000 7fc38effd640 10 client.?.objecter resend_mon_ops 2024-06-03T08:35:31.992 INFO:teuthology.orchestra.run.smithi192.stderr:2024-06-03T08:35:31.990+0000 7fc38effd640 1 -- 172.21.15.192:0/3424013783 <== mon.1 v1:172.21.15.192:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 4070241717 0 0) 0x7fc398143e30 con 0x7fc39813e6c0 2024-06-03T08:35:31.992 INFO:teuthology.orchestra.run.smithi192.stderr:2024-06-03T08:35:31.990+0000 7fc38effd640 10 monclient(hunting): _negotiate 2024-06-03T08:35:31.993 INFO:teuthology.orchestra.run.smithi192.stderr:2024-06-03T08:35:31.990+0000 7fc38effd640 10 monclient(hunting): _init_auth method 2 2024-06-03T08:35:31.993 INFO:teuthology.orchestra.run.smithi192.stderr:2024-06-03T08:35:31.990+0000 7fc38effd640 10 monclient(hunting): _init_auth creating new auth 2024-06-03T08:35:31.993 INFO:teuthology.orchestra.run.smithi192.stderr:2024-06-03T08:35:31.990+0000 7fc38effd640 10 monclient(hunting): my global_id is 4162 2024-06-03T08:35:31.993 INFO:teuthology.orchestra.run.smithi192.stderr:2024-06-03T08:35:31.990+0000 7fc38effd640 1 -- 172.21.15.192:0/3424013783 --> v1:172.21.15.192:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7fc37c004eb0 con 0x7fc39813e6c0 2024-06-03T08:35:31.993 INFO:teuthology.orchestra.run.smithi192.stderr:2024-06-03T08:35:31.991+0000 7fc38effd640 1 -- 172.21.15.192:0/3424013783 <== mon.0 v1:172.21.15.88:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 1382177245 0 0) 0x7fc39813f3b0 con 0x7fc3980b43c0 2024-06-03T08:35:31.993 INFO:teuthology.orchestra.run.smithi192.stderr:2024-06-03T08:35:31.991+0000 7fc38effd640 10 monclient(hunting): _negotiate 2024-06-03T08:35:31.993 INFO:teuthology.orchestra.run.smithi192.stderr:2024-06-03T08:35:31.991+0000 7fc38effd640 10 monclient(hunting): _init_auth method 2 2024-06-03T08:35:31.993 INFO:teuthology.orchestra.run.smithi192.stderr:2024-06-03T08:35:31.991+0000 7fc38effd640 10 monclient(hunting): _init_auth creating new auth 2024-06-03T08:35:31.993 INFO:teuthology.orchestra.run.smithi192.stderr:2024-06-03T08:35:31.991+0000 7fc38effd640 10 monclient(hunting): my global_id is 4161 2024-06-03T08:35:31.993 INFO:teuthology.orchestra.run.smithi192.stderr:2024-06-03T08:35:31.991+0000 7fc38effd640 1 -- 172.21.15.192:0/3424013783 --> v1:172.21.15.88:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7fc398143e30 con 0x7fc3980b43c0 2024-06-03T08:35:31.993 INFO:teuthology.orchestra.run.smithi192.stderr:2024-06-03T08:35:31.992+0000 7fc38effd640 1 -- 172.21.15.192:0/3424013783 <== mon.2 v1:172.21.15.88:6790/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 3764348773 0 0) 0x7fc398142b30 con 0x7fc39814ae20 2024-06-03T08:35:31.993 INFO:teuthology.orchestra.run.smithi192.stderr:2024-06-03T08:35:31.992+0000 7fc38effd640 10 monclient(hunting): _negotiate 2024-06-03T08:35:31.993 INFO:teuthology.orchestra.run.smithi192.stderr:2024-06-03T08:35:31.992+0000 7fc38effd640 10 monclient(hunting): _init_auth method 2 2024-06-03T08:35:31.993 INFO:teuthology.orchestra.run.smithi192.stderr:2024-06-03T08:35:31.992+0000 7fc38effd640 10 monclient(hunting): _init_auth creating new auth 2024-06-03T08:35:31.994 INFO:teuthology.orchestra.run.smithi192.stderr:2024-06-03T08:35:31.992+0000 7fc38effd640 10 monclient(hunting): my global_id is 4163 2024-06-03T08:35:31.994 INFO:teuthology.orchestra.run.smithi192.stderr:2024-06-03T08:35:31.992+0000 7fc38effd640 1 -- 172.21.15.192:0/3424013783 --> v1:172.21.15.88:6790/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7fc39813f3b0 con 0x7fc39814ae20 2024-06-03T08:35:31.995 INFO:teuthology.orchestra.run.smithi192.stderr:2024-06-03T08:35:31.995+0000 7fc38effd640 1 -- 172.21.15.192:0/3424013783 <== mon.1 v1:172.21.15.192:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 764+0+0 (unknown 3931936255 0 0) 0x7fc37c004eb0 con 0x7fc39813e6c0 2024-06-03T08:35:31.995 INFO:teuthology.orchestra.run.smithi192.stderr:2024-06-03T08:35:31.995+0000 7fc38effd640 1 -- 172.21.15.192:0/3424013783 --> v1:172.21.15.192:6789/0 -- auth(proto 2 165 bytes epoch 0) v1 -- 0x7fc398142b30 con 0x7fc39813e6c0 2024-06-03T08:35:31.996 INFO:teuthology.orchestra.run.smithi192.stderr:2024-06-03T08:35:31.995+0000 7fc38effd640 1 -- 172.21.15.192:0/3424013783 <== mon.1 v1:172.21.15.192:6789/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 2234374065 0 0) 0x7fc388001970 con 0x7fc39813e6c0 2024-06-03T08:35:31.996 INFO:teuthology.orchestra.run.smithi192.stderr:2024-06-03T08:35:31.995+0000 7fc38effd640 10 monclient(hunting): handle_monmap mon_map magic: 0 v1 2024-06-03T08:35:31.996 INFO:teuthology.orchestra.run.smithi192.stderr:2024-06-03T08:35:31.995+0000 7fc38effd640 10 monclient(hunting): got monmap 1 from mon.noname-b (according to old e1) 2024-06-03T08:35:31.996 INFO:teuthology.orchestra.run.smithi192.stderr:2024-06-03T08:35:31.995+0000 7fc38effd640 10 monclient(hunting): dump: 2024-06-03T08:35:31.996 INFO:teuthology.orchestra.run.smithi192.stderr:epoch 1 2024-06-03T08:35:31.996 INFO:teuthology.orchestra.run.smithi192.stderr:fsid ec3c3fdd-3ac6-4d7d-9412-683130ed8b04 2024-06-03T08:35:31.996 INFO:teuthology.orchestra.run.smithi192.stderr:last_changed 2024-06-03T08:34:31.885125+0000 2024-06-03T08:35:31.996 INFO:teuthology.orchestra.run.smithi192.stderr:created 2024-06-03T08:34:31.885125+0000 2024-06-03T08:35:31.996 INFO:teuthology.orchestra.run.smithi192.stderr:min_mon_release 18 (reef) 2024-06-03T08:35:31.996 INFO:teuthology.orchestra.run.smithi192.stderr:election_strategy: 3 2024-06-03T08:35:31.996 INFO:teuthology.orchestra.run.smithi192.stderr:0: v1:172.21.15.88:6789/0 mon.a 2024-06-03T08:35:31.996 INFO:teuthology.orchestra.run.smithi192.stderr:1: v1:172.21.15.192:6789/0 mon.b 2024-06-03T08:35:31.996 INFO:teuthology.orchestra.run.smithi192.stderr:2: v1:172.21.15.88:6790/0 mon.c 2024-06-03T08:35:31.996 INFO:teuthology.orchestra.run.smithi192.stderr: 2024-06-03T08:35:31.997 INFO:teuthology.orchestra.run.smithi192.stderr:2024-06-03T08:35:31.995+0000 7fc38effd640 10 monclient(hunting): _finish_auth 0 2024-06-03T08:35:31.997 INFO:teuthology.orchestra.run.smithi192.stderr:2024-06-03T08:35:31.996+0000 7fc38effd640 1 -- 172.21.15.192:0/3424013783 <== mon.0 v1:172.21.15.88:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 764+0+0 (unknown 1220279696 0 0) 0x7fc398143e30 con 0x7fc3980b43c0 2024-06-03T08:35:31.998 INFO:teuthology.orchestra.run.smithi192.stderr:2024-06-03T08:35:31.996+0000 7fc38effd640 1 -- 172.21.15.192:0/3424013783 --> v1:172.21.15.88:6789/0 -- auth(proto 2 165 bytes epoch 0) v1 -- 0x7fc37c004eb0 con 0x7fc3980b43c0 2024-06-03T08:35:31.998 INFO:teuthology.orchestra.run.smithi192.stderr:2024-06-03T08:35:31.996+0000 7fc38effd640 1 -- 172.21.15.192:0/3424013783 <== mon.0 v1:172.21.15.88:6789/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 2234374065 0 0) 0x7fc3940018b0 con 0x7fc3980b43c0 2024-06-03T08:35:31.998 INFO:teuthology.orchestra.run.smithi192.stderr:2024-06-03T08:35:31.996+0000 7fc38effd640 10 monclient(hunting): handle_monmap mon_map magic: 0 v1 2024-06-03T08:35:31.998 INFO:teuthology.orchestra.run.smithi192.stderr:2024-06-03T08:35:31.996+0000 7fc38effd640 10 monclient(hunting): got monmap 1 from mon.a (according to old e1) 2024-06-03T08:35:31.998 INFO:teuthology.orchestra.run.smithi192.stderr:2024-06-03T08:35:31.996+0000 7fc38effd640 10 monclient(hunting): dump: 2024-06-03T08:35:31.998 INFO:teuthology.orchestra.run.smithi192.stderr:epoch 1 2024-06-03T08:35:31.998 INFO:teuthology.orchestra.run.smithi192.stderr:fsid ec3c3fdd-3ac6-4d7d-9412-683130ed8b04 2024-06-03T08:35:31.998 INFO:teuthology.orchestra.run.smithi192.stderr:last_changed 2024-06-03T08:34:31.885125+0000 2024-06-03T08:35:31.998 INFO:teuthology.orchestra.run.smithi192.stderr:created 2024-06-03T08:34:31.885125+0000 2024-06-03T08:35:31.998 INFO:teuthology.orchestra.run.smithi192.stderr:min_mon_release 18 (reef) 2024-06-03T08:35:31.998 INFO:teuthology.orchestra.run.smithi192.stderr:election_strategy: 3 2024-06-03T08:35:31.998 INFO:teuthology.orchestra.run.smithi192.stderr:0: v1:172.21.15.88:6789/0 mon.a 2024-06-03T08:35:31.998 INFO:teuthology.orchestra.run.smithi192.stderr:1: v1:172.21.15.192:6789/0 mon.b 2024-06-03T08:35:31.999 INFO:teuthology.orchestra.run.smithi192.stderr:2: v1:172.21.15.88:6790/0 mon.c 2024-06-03T08:35:31.999 INFO:teuthology.orchestra.run.smithi192.stderr: 2024-06-03T08:35:31.999 INFO:teuthology.orchestra.run.smithi192.stderr:2024-06-03T08:35:31.997+0000 7fc38effd640 1 -- 172.21.15.192:0/3424013783 <== mon.2 v1:172.21.15.88:6790/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 764+0+0 (unknown 2172481098 0 0) 0x7fc39813f3b0 con 0x7fc39814ae20 2024-06-03T08:35:31.999 INFO:teuthology.orchestra.run.smithi192.stderr:2024-06-03T08:35:31.997+0000 7fc38effd640 1 -- 172.21.15.192:0/3424013783 --> v1:172.21.15.88:6790/0 -- auth(proto 2 165 bytes epoch 0) v1 -- 0x7fc398143e30 con 0x7fc39814ae20 2024-06-03T08:35:31.999 INFO:teuthology.orchestra.run.smithi192.stderr:2024-06-03T08:35:31.997+0000 7fc38effd640 1 -- 172.21.15.192:0/3424013783 <== mon.2 v1:172.21.15.88:6790/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 2234374065 0 0) 0x7fc390001ec0 con 0x7fc39814ae20 2024-06-03T08:35:31.999 INFO:teuthology.orchestra.run.smithi192.stderr:2024-06-03T08:35:31.997+0000 7fc38effd640 10 monclient(hunting): handle_monmap mon_map magic: 0 v1 2024-06-03T08:35:31.999 INFO:teuthology.orchestra.run.smithi192.stderr:2024-06-03T08:35:31.997+0000 7fc38effd640 10 monclient(hunting): got monmap 1 from mon.c (according to old e1) 2024-06-03T08:35:31.999 INFO:teuthology.orchestra.run.smithi192.stderr:2024-06-03T08:35:31.997+0000 7fc38effd640 10 monclient(hunting): dump: 2024-06-03T08:35:31.999 INFO:teuthology.orchestra.run.smithi192.stderr:epoch 1 2024-06-03T08:35:31.999 INFO:teuthology.orchestra.run.smithi192.stderr:fsid ec3c3fdd-3ac6-4d7d-9412-683130ed8b04 2024-06-03T08:35:31.999 INFO:teuthology.orchestra.run.smithi192.stderr:last_changed 2024-06-03T08:34:31.885125+0000 2024-06-03T08:35:31.999 INFO:teuthology.orchestra.run.smithi192.stderr:created 2024-06-03T08:34:31.885125+0000 2024-06-03T08:35:32.000 INFO:teuthology.orchestra.run.smithi192.stderr:min_mon_release 18 (reef) 2024-06-03T08:35:32.000 INFO:teuthology.orchestra.run.smithi192.stderr:election_strategy: 3 2024-06-03T08:35:32.000 INFO:teuthology.orchestra.run.smithi192.stderr:0: v1:172.21.15.88:6789/0 mon.a 2024-06-03T08:35:32.000 INFO:teuthology.orchestra.run.smithi192.stderr:1: v1:172.21.15.192:6789/0 mon.b 2024-06-03T08:35:32.000 INFO:teuthology.orchestra.run.smithi192.stderr:2: v1:172.21.15.88:6790/0 mon.c 2024-06-03T08:35:32.000 INFO:teuthology.orchestra.run.smithi192.stderr: 2024-06-03T08:35:32.001 INFO:teuthology.orchestra.run.smithi192.stderr:2024-06-03T08:35:31.998+0000 7fc38effd640 1 -- 172.21.15.192:0/3424013783 <== mon.1 v1:172.21.15.192:6789/0 4 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 580+0+0 (unknown 3043241915 0 0) 0x7fc398142b30 con 0x7fc39813e6c0 2024-06-03T08:35:32.001 INFO:teuthology.orchestra.run.smithi192.stderr:2024-06-03T08:35:31.998+0000 7fc38effd640 1 -- 172.21.15.192:0/3424013783 >> v1:172.21.15.88:6790/0 conn(0x7fc39814ae20 legacy=0x7fc3981d6a20 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-06-03T08:35:32.001 INFO:teuthology.orchestra.run.smithi192.stderr:2024-06-03T08:35:31.998+0000 7fc38effd640 1 -- 172.21.15.192:0/3424013783 >> v1:172.21.15.88:6789/0 conn(0x7fc3980b43c0 legacy=0x7fc398143720 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-06-03T08:35:32.001 INFO:teuthology.orchestra.run.smithi192.stderr:2024-06-03T08:35:31.998+0000 7fc38effd640 10 monclient: _finish_hunting 0 2024-06-03T08:35:32.001 INFO:teuthology.orchestra.run.smithi192.stderr:2024-06-03T08:35:31.998+0000 7fc38effd640 1 monclient: found mon.b 2024-06-03T08:35:32.002 INFO:teuthology.orchestra.run.smithi192.stderr:2024-06-03T08:35:31.998+0000 7fc38effd640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-06-03T08:35:32.002 INFO:teuthology.orchestra.run.smithi192.stderr:2024-06-03T08:35:31.998+0000 7fc38effd640 10 monclient: _send_mon_message to mon.b at v1:172.21.15.192:6789/0 2024-06-03T08:35:32.002 INFO:teuthology.orchestra.run.smithi192.stderr:2024-06-03T08:35:31.998+0000 7fc38effd640 1 -- 172.21.15.192:0/3424013783 --> v1:172.21.15.192:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fc398142cc0 con 0x7fc39813e6c0 2024-06-03T08:35:32.002 INFO:teuthology.orchestra.run.smithi192.stderr:2024-06-03T08:35:31.998+0000 7fc38effd640 10 monclient: _finish_auth 0 2024-06-03T08:35:32.002 INFO:teuthology.orchestra.run.smithi192.stderr:2024-06-03T08:35:31.998+0000 7fc38effd640 10 monclient: _check_auth_tickets 2024-06-03T08:35:32.002 INFO:teuthology.orchestra.run.smithi192.stderr:2024-06-03T08:35:31.998+0000 7fc38effd640 20 monclient: _check_auth_rotating not needed by client.admin 2024-06-03T08:35:32.002 INFO:teuthology.orchestra.run.smithi192.stderr:2024-06-03T08:35:31.998+0000 7fc3a0470640 5 monclient: authenticate success, global_id 4162 2024-06-03T08:35:32.002 INFO:teuthology.orchestra.run.smithi192.stderr:2024-06-03T08:35:31.998+0000 7fc3a0470640 10 monclient: _renew_subs 2024-06-03T08:35:32.002 INFO:teuthology.orchestra.run.smithi192.stderr:2024-06-03T08:35:31.998+0000 7fc3a0470640 10 monclient: _send_mon_message to mon.b at v1:172.21.15.192:6789/0 2024-06-03T08:35:32.002 INFO:teuthology.orchestra.run.smithi192.stderr:2024-06-03T08:35:31.998+0000 7fc3a0470640 1 -- 172.21.15.192:0/3424013783 --> v1:172.21.15.192:6789/0 -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fc398145620 con 0x7fc39813e6c0 2024-06-03T08:35:32.002 INFO:teuthology.orchestra.run.smithi192.stderr:2024-06-03T08:35:31.998+0000 7fc3a0470640 10 client.4162.objecter _maybe_request_map subscribing (onetime) to next osd map 2024-06-03T08:35:32.002 INFO:teuthology.orchestra.run.smithi192.stderr:2024-06-03T08:35:31.998+0000 7fc3a0470640 10 monclient: _renew_subs 2024-06-03T08:35:32.002 INFO:teuthology.orchestra.run.smithi192.stderr:2024-06-03T08:35:31.998+0000 7fc3a0470640 10 monclient: _send_mon_message to mon.b at v1:172.21.15.192:6789/0 2024-06-03T08:35:32.003 INFO:teuthology.orchestra.run.smithi192.stderr:2024-06-03T08:35:31.998+0000 7fc3a0470640 1 -- 172.21.15.192:0/3424013783 --> v1:172.21.15.192:6789/0 -- mon_subscribe({osdmap=0}) v3 -- 0x7fc398135bf0 con 0x7fc39813e6c0 2024-06-03T08:35:32.003 INFO:teuthology.orchestra.run.smithi192.stderr:2024-06-03T08:35:31.998+0000 7fc3a0470640 1 librados: init done 2024-06-03T08:35:32.003 INFO:teuthology.orchestra.run.smithi192.stderr:2024-06-03T08:35:31.999+0000 7fc3a0470640 10 start_mon_command cmd=[{"prefix": "get_command_descriptions"}] 2024-06-03T08:35:32.003 INFO:teuthology.orchestra.run.smithi192.stderr:2024-06-03T08:35:31.999+0000 7fc3a0470640 10 monclient: _send_command 1 [{"prefix": "get_command_descriptions"}] 2024-06-03T08:35:32.003 INFO:teuthology.orchestra.run.smithi192.stderr:2024-06-03T08:35:31.999+0000 7fc3a0470640 10 monclient: _send_mon_message to mon.b at v1:172.21.15.192:6789/0 2024-06-03T08:35:32.003 INFO:teuthology.orchestra.run.smithi192.stderr:2024-06-03T08:35:31.999+0000 7fc3a0470640 1 -- 172.21.15.192:0/3424013783 --> v1:172.21.15.192:6789/0 -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fc398143fc0 con 0x7fc39813e6c0 2024-06-03T08:35:32.098 INFO:teuthology.orchestra.run.smithi192.stderr:2024-06-03T08:35:32.097+0000 7fc38effd640 1 -- 172.21.15.192:0/3424013783 <== mon.1 v1:172.21.15.192:6789/0 5 ==== config(0 keys) v1 ==== 4+0+0 (unknown 0 0 0) 0x7fc388003610 con 0x7fc39813e6c0 2024-06-03T08:35:32.098 INFO:teuthology.orchestra.run.smithi192.stderr:2024-06-03T08:35:32.097+0000 7fc38effd640 10 monclient: handle_config config(0 keys) v1 2024-06-03T08:35:32.098 INFO:teuthology.orchestra.run.smithi192.stderr:2024-06-03T08:35:32.097+0000 7fc38effd640 1 -- 172.21.15.192:0/3424013783 <== mon.1 v1:172.21.15.192:6789/0 6 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 2234374065 0 0) 0x7fc388003920 con 0x7fc39813e6c0 2024-06-03T08:35:32.098 INFO:teuthology.orchestra.run.smithi192.stderr:2024-06-03T08:35:32.097+0000 7fc38effd640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-06-03T08:35:32.098 INFO:teuthology.orchestra.run.smithi192.stderr:2024-06-03T08:35:32.097+0000 7fc38effd640 10 monclient: got monmap 1 from mon.b (according to old e1) 2024-06-03T08:35:32.098 INFO:teuthology.orchestra.run.smithi192.stderr:2024-06-03T08:35:32.097+0000 7fc38effd640 10 monclient: dump: 2024-06-03T08:35:32.098 INFO:teuthology.orchestra.run.smithi192.stderr:epoch 1 2024-06-03T08:35:32.098 INFO:teuthology.orchestra.run.smithi192.stderr:fsid ec3c3fdd-3ac6-4d7d-9412-683130ed8b04 2024-06-03T08:35:32.098 INFO:teuthology.orchestra.run.smithi192.stderr:last_changed 2024-06-03T08:34:31.885125+0000 2024-06-03T08:35:32.099 INFO:teuthology.orchestra.run.smithi192.stderr:created 2024-06-03T08:34:31.885125+0000 2024-06-03T08:35:32.099 INFO:teuthology.orchestra.run.smithi192.stderr:min_mon_release 18 (reef) 2024-06-03T08:35:32.099 INFO:teuthology.orchestra.run.smithi192.stderr:election_strategy: 3 2024-06-03T08:35:32.099 INFO:teuthology.orchestra.run.smithi192.stderr:0: v1:172.21.15.88:6789/0 mon.a 2024-06-03T08:35:32.099 INFO:teuthology.orchestra.run.smithi192.stderr:1: v1:172.21.15.192:6789/0 mon.b 2024-06-03T08:35:32.099 INFO:teuthology.orchestra.run.smithi192.stderr:2: v1:172.21.15.88:6790/0 mon.c 2024-06-03T08:35:32.099 INFO:teuthology.orchestra.run.smithi192.stderr: 2024-06-03T08:35:32.099 INFO:teuthology.orchestra.run.smithi192.stderr:2024-06-03T08:35:32.097+0000 7fc38effd640 1 -- 172.21.15.192:0/3424013783 <== mon.1 v1:172.21.15.192:6789/0 7 ==== mgrmap(e 1) v1 ==== 668+0+0 (unknown 3836259515 0 0) 0x7fc388003d80 con 0x7fc39813e6c0 2024-06-03T08:35:32.099 INFO:teuthology.orchestra.run.smithi192.stderr:2024-06-03T08:35:32.097+0000 7fc39ca09640 4 set_mon_vals no callback set 2024-06-03T08:35:32.099 INFO:teuthology.orchestra.run.smithi192.stderr:2024-06-03T08:35:32.097+0000 7fc38effd640 1 -- 172.21.15.192:0/3424013783 <== mon.1 v1:172.21.15.192:6789/0 8 ==== osd_map(9..9 src has 1..9) v4 ==== 1530+0+0 (unknown 3929892782 0 0) 0x7fc388006480 con 0x7fc39813e6c0 2024-06-03T08:35:32.099 INFO:teuthology.orchestra.run.smithi192.stderr:2024-06-03T08:35:32.097+0000 7fc38effd640 10 client.4162.objecter ms_dispatch 0x7fc398000b90 osd_map(9..9 src has 1..9) v4 2024-06-03T08:35:32.099 INFO:teuthology.orchestra.run.smithi192.stderr:2024-06-03T08:35:32.097+0000 7fc38effd640 3 client.4162.objecter handle_osd_map got epochs [9,9] > 0 2024-06-03T08:35:32.099 INFO:teuthology.orchestra.run.smithi192.stderr:2024-06-03T08:35:32.097+0000 7fc38effd640 3 client.4162.objecter handle_osd_map decoding full epoch 9 2024-06-03T08:35:32.100 INFO:teuthology.orchestra.run.smithi192.stderr:2024-06-03T08:35:32.097+0000 7fc38effd640 20 client.4162.objecter dump_active .. 0 homeless 2024-06-03T08:35:32.100 INFO:teuthology.orchestra.run.smithi192.stderr:2024-06-03T08:35:32.097+0000 7fc38effd640 1 -- 172.21.15.192:0/3424013783 <== mon.1 v1:172.21.15.192:6789/0 9 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+73121 (unknown 1092875540 0 1417653156) 0x7fc3880068a0 con 0x7fc39813e6c0 2024-06-03T08:35:32.100 INFO:teuthology.orchestra.run.smithi192.stderr:2024-06-03T08:35:32.097+0000 7fc38effd640 10 monclient: handle_mon_command_ack 1 [{"prefix": "get_command_descriptions"}] 2024-06-03T08:35:32.100 INFO:teuthology.orchestra.run.smithi192.stderr:2024-06-03T08:35:32.097+0000 7fc38effd640 10 monclient: _finish_command 1 = system:0 2024-06-03T08:35:32.170 INFO:teuthology.orchestra.run.smithi192.stderr:2024-06-03T08:35:32.170+0000 7fc3a0470640 10 start_mon_command cmd=[{"prefix": "osd new", "uuid": "aafe0716-1102-4eae-aac6-0f81bb154e7d", "id": 7}] 2024-06-03T08:35:32.170 INFO:teuthology.orchestra.run.smithi192.stderr:2024-06-03T08:35:32.170+0000 7fc3a0470640 10 monclient: _send_command 2 [{"prefix": "osd new", "uuid": "aafe0716-1102-4eae-aac6-0f81bb154e7d", "id": 7}] 2024-06-03T08:35:32.170 INFO:teuthology.orchestra.run.smithi192.stderr:2024-06-03T08:35:32.170+0000 7fc3a0470640 10 monclient: _send_mon_message to mon.b at v1:172.21.15.192:6789/0 2024-06-03T08:35:32.171 INFO:teuthology.orchestra.run.smithi192.stderr:2024-06-03T08:35:32.170+0000 7fc3a0470640 1 -- 172.21.15.192:0/3424013783 --> v1:172.21.15.192:6789/0 -- mon_command({"prefix": "osd new", "uuid": "aafe0716-1102-4eae-aac6-0f81bb154e7d", "id": 7} v 0) v1 -- 0x7fc39804ea60 con 0x7fc39813e6c0 2024-06-03T08:35:32.212 INFO:tasks.ceph.mgr.x.smithi192.stderr:2024-06-03T08:35:32.211+0000 7f0e6be43040 -1 mgr[py] Module status has missing NOTIFY_TYPES member 2024-06-03T08:35:32.226 INFO:teuthology.orchestra.run.smithi192.stderr:2024-06-03T08:35:32.225+0000 7fc38effd640 1 -- 172.21.15.192:0/3424013783 <== mon.1 v1:172.21.15.192:6789/0 10 ==== mon_command_ack([{"prefix": "osd new", "uuid": "aafe0716-1102-4eae-aac6-0f81bb154e7d", "id": 7}]=0 v10) v1 ==== 112+0+2 (unknown 501043043 0 1371665406) 0x7fc388005890 con 0x7fc39813e6c0 2024-06-03T08:35:32.226 INFO:teuthology.orchestra.run.smithi192.stderr:2024-06-03T08:35:32.225+0000 7fc38effd640 10 monclient: handle_mon_command_ack 2 [{"prefix": "osd new", "uuid": "aafe0716-1102-4eae-aac6-0f81bb154e7d", "id": 7}] 2024-06-03T08:35:32.226 INFO:teuthology.orchestra.run.smithi192.stderr:2024-06-03T08:35:32.225+0000 7fc38effd640 10 monclient: _finish_command 2 = system:0 2024-06-03T08:35:32.226 INFO:teuthology.orchestra.run.smithi192.stdout:7 2024-06-03T08:35:32.227 INFO:teuthology.orchestra.run.smithi192.stderr:2024-06-03T08:35:32.226+0000 7fc3a0470640 10 librados: watch_flush enter 2024-06-03T08:35:32.227 INFO:teuthology.orchestra.run.smithi192.stderr:2024-06-03T08:35:32.226+0000 7fc3a0470640 10 librados: watch_flush exit 2024-06-03T08:35:32.227 INFO:teuthology.orchestra.run.smithi192.stderr:2024-06-03T08:35:32.226+0000 7fc3a0470640 20 client.4162.objecter shutdown clearing up homeless session... 2024-06-03T08:35:32.227 INFO:teuthology.orchestra.run.smithi192.stderr:2024-06-03T08:35:32.226+0000 7fc3a0470640 10 client.4162.objecter successfully canceled tick 2024-06-03T08:35:32.228 INFO:teuthology.orchestra.run.smithi192.stderr:2024-06-03T08:35:32.226+0000 7fc3a0470640 10 monclient: shutdown 2024-06-03T08:35:32.228 INFO:teuthology.orchestra.run.smithi192.stderr:2024-06-03T08:35:32.226+0000 7fc3a0470640 20 monclient: shutdown discarding 0 pending message(s) 2024-06-03T08:35:32.228 INFO:teuthology.orchestra.run.smithi192.stderr:2024-06-03T08:35:32.226+0000 7fc3a0470640 1 -- 172.21.15.192:0/3424013783 >> v1:172.21.15.192:6789/0 conn(0x7fc39813e6c0 legacy=0x7fc398135300 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-06-03T08:35:32.228 INFO:teuthology.orchestra.run.smithi192.stderr:2024-06-03T08:35:32.226+0000 7fc3a0470640 1 -- 172.21.15.192:0/3424013783 shutdown_connections 2024-06-03T08:35:32.228 INFO:teuthology.orchestra.run.smithi192.stderr:2024-06-03T08:35:32.226+0000 7fc3a0470640 1 -- 172.21.15.192:0/3424013783 >> 172.21.15.192:0/3424013783 conn(0x7fc3980719d0 msgr2=0x7fc3980b25b0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-06-03T08:35:32.228 INFO:teuthology.orchestra.run.smithi192.stderr:2024-06-03T08:35:32.226+0000 7fc3a0470640 1 -- 172.21.15.192:0/3424013783 shutdown_connections 2024-06-03T08:35:32.228 INFO:teuthology.orchestra.run.smithi192.stderr:2024-06-03T08:35:32.226+0000 7fc3a0470640 1 -- 172.21.15.192:0/3424013783 wait complete. 2024-06-03T08:35:32.228 INFO:teuthology.orchestra.run.smithi192.stderr:2024-06-03T08:35:32.227+0000 7fc3a0470640 1 librados: shutdown 2024-06-03T08:35:32.237 DEBUG:tasks.ceph_manager:running osd.0 under valgrind with args ['cd', '/home/ubuntu/cephtest', Raw('&&'), 'sudo', 'adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'daemon-helper', 'term', 'env', 'OPENSSL_ia32cap=~0x1000000000000000', 'valgrind', '--trace-children=no', '--child-silent-after-fork=yes', '--soname-synonyms=somalloc=*tcmalloc*', '--num-callers=50', '--suppressions=/home/ubuntu/cephtest/valgrind.supp', '--xml=yes', '--xml-file=/var/log/ceph/valgrind/osd.0.log', '--time-stamp=yes', '--vgdb=yes', '--exit-on-first-error=yes', '--error-exitcode=42', '--tool=memcheck'] 2024-06-03T08:35:32.237 INFO:tasks.ceph.osd.0:Restarting daemon 2024-06-03T08:35:32.238 DEBUG:teuthology.orchestra.run.smithi088:> cd /home/ubuntu/cephtest && sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage daemon-helper term env 'OPENSSL_ia32cap=~0x1000000000000000' valgrind --trace-children=no --child-silent-after-fork=yes '--soname-synonyms=somalloc=*tcmalloc*' --num-callers=50 --suppressions=/home/ubuntu/cephtest/valgrind.supp --xml=yes --xml-file=/var/log/ceph/valgrind/osd.0.log --time-stamp=yes --vgdb=yes --exit-on-first-error=yes --error-exitcode=42 --tool=memcheck ceph-osd -f --cluster ceph -i 0 2024-06-03T08:35:32.240 INFO:tasks.ceph.osd.0:Started 2024-06-03T08:35:32.240 DEBUG:tasks.ceph_manager:running osd.1 under valgrind with args ['cd', '/home/ubuntu/cephtest', Raw('&&'), 'sudo', 'adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'daemon-helper', 'term', 'env', 'OPENSSL_ia32cap=~0x1000000000000000', 'valgrind', '--trace-children=no', '--child-silent-after-fork=yes', '--soname-synonyms=somalloc=*tcmalloc*', '--num-callers=50', '--suppressions=/home/ubuntu/cephtest/valgrind.supp', '--xml=yes', '--xml-file=/var/log/ceph/valgrind/osd.1.log', '--time-stamp=yes', '--vgdb=yes', '--exit-on-first-error=yes', '--error-exitcode=42', '--tool=memcheck'] 2024-06-03T08:35:32.240 INFO:tasks.ceph.osd.1:Restarting daemon 2024-06-03T08:35:32.240 DEBUG:teuthology.orchestra.run.smithi088:> cd /home/ubuntu/cephtest && sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage daemon-helper term env 'OPENSSL_ia32cap=~0x1000000000000000' valgrind --trace-children=no --child-silent-after-fork=yes '--soname-synonyms=somalloc=*tcmalloc*' --num-callers=50 --suppressions=/home/ubuntu/cephtest/valgrind.supp --xml=yes --xml-file=/var/log/ceph/valgrind/osd.1.log --time-stamp=yes --vgdb=yes --exit-on-first-error=yes --error-exitcode=42 --tool=memcheck ceph-osd -f --cluster ceph -i 1 2024-06-03T08:35:32.242 INFO:tasks.ceph.osd.1:Started 2024-06-03T08:35:32.242 DEBUG:tasks.ceph_manager:running osd.2 under valgrind with args ['cd', '/home/ubuntu/cephtest', Raw('&&'), 'sudo', 'adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'daemon-helper', 'term', 'env', 'OPENSSL_ia32cap=~0x1000000000000000', 'valgrind', '--trace-children=no', '--child-silent-after-fork=yes', '--soname-synonyms=somalloc=*tcmalloc*', '--num-callers=50', '--suppressions=/home/ubuntu/cephtest/valgrind.supp', '--xml=yes', '--xml-file=/var/log/ceph/valgrind/osd.2.log', '--time-stamp=yes', '--vgdb=yes', '--exit-on-first-error=yes', '--error-exitcode=42', '--tool=memcheck'] 2024-06-03T08:35:32.243 INFO:tasks.ceph.osd.2:Restarting daemon 2024-06-03T08:35:32.243 DEBUG:teuthology.orchestra.run.smithi088:> cd /home/ubuntu/cephtest && sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage daemon-helper term env 'OPENSSL_ia32cap=~0x1000000000000000' valgrind --trace-children=no --child-silent-after-fork=yes '--soname-synonyms=somalloc=*tcmalloc*' --num-callers=50 --suppressions=/home/ubuntu/cephtest/valgrind.supp --xml=yes --xml-file=/var/log/ceph/valgrind/osd.2.log --time-stamp=yes --vgdb=yes --exit-on-first-error=yes --error-exitcode=42 --tool=memcheck ceph-osd -f --cluster ceph -i 2 2024-06-03T08:35:32.245 INFO:tasks.ceph.osd.2:Started 2024-06-03T08:35:32.245 DEBUG:tasks.ceph_manager:running osd.3 under valgrind with args ['cd', '/home/ubuntu/cephtest', Raw('&&'), 'sudo', 'adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'daemon-helper', 'term', 'env', 'OPENSSL_ia32cap=~0x1000000000000000', 'valgrind', '--trace-children=no', '--child-silent-after-fork=yes', '--soname-synonyms=somalloc=*tcmalloc*', '--num-callers=50', '--suppressions=/home/ubuntu/cephtest/valgrind.supp', '--xml=yes', '--xml-file=/var/log/ceph/valgrind/osd.3.log', '--time-stamp=yes', '--vgdb=yes', '--exit-on-first-error=yes', '--error-exitcode=42', '--tool=memcheck'] 2024-06-03T08:35:32.245 INFO:tasks.ceph.osd.3:Restarting daemon 2024-06-03T08:35:32.245 DEBUG:teuthology.orchestra.run.smithi088:> cd /home/ubuntu/cephtest && sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage daemon-helper term env 'OPENSSL_ia32cap=~0x1000000000000000' valgrind --trace-children=no --child-silent-after-fork=yes '--soname-synonyms=somalloc=*tcmalloc*' --num-callers=50 --suppressions=/home/ubuntu/cephtest/valgrind.supp --xml=yes --xml-file=/var/log/ceph/valgrind/osd.3.log --time-stamp=yes --vgdb=yes --exit-on-first-error=yes --error-exitcode=42 --tool=memcheck ceph-osd -f --cluster ceph -i 3 2024-06-03T08:35:32.250 INFO:tasks.ceph.osd.3:Started 2024-06-03T08:35:32.250 DEBUG:tasks.ceph_manager:running osd.4 under valgrind with args ['cd', '/home/ubuntu/cephtest', Raw('&&'), 'sudo', 'adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'daemon-helper', 'term', 'env', 'OPENSSL_ia32cap=~0x1000000000000000', 'valgrind', '--trace-children=no', '--child-silent-after-fork=yes', '--soname-synonyms=somalloc=*tcmalloc*', '--num-callers=50', '--suppressions=/home/ubuntu/cephtest/valgrind.supp', '--xml=yes', '--xml-file=/var/log/ceph/valgrind/osd.4.log', '--time-stamp=yes', '--vgdb=yes', '--exit-on-first-error=yes', '--error-exitcode=42', '--tool=memcheck'] 2024-06-03T08:35:32.251 INFO:tasks.ceph.osd.4:Restarting daemon 2024-06-03T08:35:32.251 DEBUG:teuthology.orchestra.run.smithi192:> cd /home/ubuntu/cephtest && sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage daemon-helper term env 'OPENSSL_ia32cap=~0x1000000000000000' valgrind --trace-children=no --child-silent-after-fork=yes '--soname-synonyms=somalloc=*tcmalloc*' --num-callers=50 --suppressions=/home/ubuntu/cephtest/valgrind.supp --xml=yes --xml-file=/var/log/ceph/valgrind/osd.4.log --time-stamp=yes --vgdb=yes --exit-on-first-error=yes --error-exitcode=42 --tool=memcheck ceph-osd -f --cluster ceph -i 4 2024-06-03T08:35:32.253 INFO:tasks.ceph.osd.4:Started 2024-06-03T08:35:32.253 DEBUG:tasks.ceph_manager:running osd.5 under valgrind with args ['cd', '/home/ubuntu/cephtest', Raw('&&'), 'sudo', 'adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'daemon-helper', 'term', 'env', 'OPENSSL_ia32cap=~0x1000000000000000', 'valgrind', '--trace-children=no', '--child-silent-after-fork=yes', '--soname-synonyms=somalloc=*tcmalloc*', '--num-callers=50', '--suppressions=/home/ubuntu/cephtest/valgrind.supp', '--xml=yes', '--xml-file=/var/log/ceph/valgrind/osd.5.log', '--time-stamp=yes', '--vgdb=yes', '--exit-on-first-error=yes', '--error-exitcode=42', '--tool=memcheck'] 2024-06-03T08:35:32.253 INFO:tasks.ceph.osd.5:Restarting daemon 2024-06-03T08:35:32.253 DEBUG:teuthology.orchestra.run.smithi192:> cd /home/ubuntu/cephtest && sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage daemon-helper term env 'OPENSSL_ia32cap=~0x1000000000000000' valgrind --trace-children=no --child-silent-after-fork=yes '--soname-synonyms=somalloc=*tcmalloc*' --num-callers=50 --suppressions=/home/ubuntu/cephtest/valgrind.supp --xml=yes --xml-file=/var/log/ceph/valgrind/osd.5.log --time-stamp=yes --vgdb=yes --exit-on-first-error=yes --error-exitcode=42 --tool=memcheck ceph-osd -f --cluster ceph -i 5 2024-06-03T08:35:32.255 INFO:tasks.ceph.osd.5:Started 2024-06-03T08:35:32.255 DEBUG:tasks.ceph_manager:running osd.6 under valgrind with args ['cd', '/home/ubuntu/cephtest', Raw('&&'), 'sudo', 'adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'daemon-helper', 'term', 'env', 'OPENSSL_ia32cap=~0x1000000000000000', 'valgrind', '--trace-children=no', '--child-silent-after-fork=yes', '--soname-synonyms=somalloc=*tcmalloc*', '--num-callers=50', '--suppressions=/home/ubuntu/cephtest/valgrind.supp', '--xml=yes', '--xml-file=/var/log/ceph/valgrind/osd.6.log', '--time-stamp=yes', '--vgdb=yes', '--exit-on-first-error=yes', '--error-exitcode=42', '--tool=memcheck'] 2024-06-03T08:35:32.256 INFO:tasks.ceph.osd.6:Restarting daemon 2024-06-03T08:35:32.256 DEBUG:teuthology.orchestra.run.smithi192:> cd /home/ubuntu/cephtest && sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage daemon-helper term env 'OPENSSL_ia32cap=~0x1000000000000000' valgrind --trace-children=no --child-silent-after-fork=yes '--soname-synonyms=somalloc=*tcmalloc*' --num-callers=50 --suppressions=/home/ubuntu/cephtest/valgrind.supp --xml=yes --xml-file=/var/log/ceph/valgrind/osd.6.log --time-stamp=yes --vgdb=yes --exit-on-first-error=yes --error-exitcode=42 --tool=memcheck ceph-osd -f --cluster ceph -i 6 2024-06-03T08:35:32.258 INFO:tasks.ceph.osd.6:Started 2024-06-03T08:35:32.258 DEBUG:tasks.ceph_manager:running osd.7 under valgrind with args ['cd', '/home/ubuntu/cephtest', Raw('&&'), 'sudo', 'adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'daemon-helper', 'term', 'env', 'OPENSSL_ia32cap=~0x1000000000000000', 'valgrind', '--trace-children=no', '--child-silent-after-fork=yes', '--soname-synonyms=somalloc=*tcmalloc*', '--num-callers=50', '--suppressions=/home/ubuntu/cephtest/valgrind.supp', '--xml=yes', '--xml-file=/var/log/ceph/valgrind/osd.7.log', '--time-stamp=yes', '--vgdb=yes', '--exit-on-first-error=yes', '--error-exitcode=42', '--tool=memcheck'] 2024-06-03T08:35:32.258 INFO:tasks.ceph.osd.7:Restarting daemon 2024-06-03T08:35:32.258 DEBUG:teuthology.orchestra.run.smithi192:> cd /home/ubuntu/cephtest && sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage daemon-helper term env 'OPENSSL_ia32cap=~0x1000000000000000' valgrind --trace-children=no --child-silent-after-fork=yes '--soname-synonyms=somalloc=*tcmalloc*' --num-callers=50 --suppressions=/home/ubuntu/cephtest/valgrind.supp --xml=yes --xml-file=/var/log/ceph/valgrind/osd.7.log --time-stamp=yes --vgdb=yes --exit-on-first-error=yes --error-exitcode=42 --tool=memcheck ceph-osd -f --cluster ceph -i 7 2024-06-03T08:35:32.260 INFO:tasks.ceph.osd.7:Started 2024-06-03T08:35:32.261 DEBUG:teuthology.orchestra.run.smithi088:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd dump --format=json 2024-06-03T08:35:32.366 INFO:tasks.ceph.mgr.x.smithi192.stderr:2024-06-03T08:35:32.365+0000 7f0e6be43040 -1 mgr[py] Module nfs has missing NOTIFY_TYPES member 2024-06-03T08:35:32.371 INFO:teuthology.orchestra.run.smithi088.stderr:2024-06-03T08:35:32.370+0000 7fdda885f640 10 monclient: get_monmap_and_config 2024-06-03T08:35:32.371 INFO:teuthology.orchestra.run.smithi088.stderr:2024-06-03T08:35:32.370+0000 7fdda885f640 10 monclient: build_initial_monmap 2024-06-03T08:35:32.371 INFO:teuthology.orchestra.run.smithi088.stderr:2024-06-03T08:35:32.371+0000 7fdda885f640 10 monclient: monmap: 2024-06-03T08:35:32.371 INFO:teuthology.orchestra.run.smithi088.stderr:epoch 0 2024-06-03T08:35:32.372 INFO:teuthology.orchestra.run.smithi088.stderr:fsid ec3c3fdd-3ac6-4d7d-9412-683130ed8b04 2024-06-03T08:35:32.372 INFO:teuthology.orchestra.run.smithi088.stderr:last_changed 2024-06-03T08:35:32.372020+0000 2024-06-03T08:35:32.372 INFO:teuthology.orchestra.run.smithi088.stderr:created 2024-06-03T08:35:32.372020+0000 2024-06-03T08:35:32.372 INFO:teuthology.orchestra.run.smithi088.stderr:min_mon_release 0 (unknown) 2024-06-03T08:35:32.372 INFO:teuthology.orchestra.run.smithi088.stderr:election_strategy: 3 2024-06-03T08:35:32.372 INFO:teuthology.orchestra.run.smithi088.stderr:0: v1:172.21.15.88:6789/0 mon.noname-a 2024-06-03T08:35:32.372 INFO:teuthology.orchestra.run.smithi088.stderr:1: v1:172.21.15.192:6789/0 mon.noname-c 2024-06-03T08:35:32.372 INFO:teuthology.orchestra.run.smithi088.stderr:2: v1:172.21.15.88:6790/0 mon.noname-b 2024-06-03T08:35:32.372 INFO:teuthology.orchestra.run.smithi088.stderr: 2024-06-03T08:35:32.373 INFO:teuthology.orchestra.run.smithi088.stderr:2024-06-03T08:35:32.371+0000 7fdda885f640 1 Processor -- start 2024-06-03T08:35:32.373 INFO:teuthology.orchestra.run.smithi088.stderr:2024-06-03T08:35:32.371+0000 7fdda885f640 1 -- start start 2024-06-03T08:35:32.373 INFO:teuthology.orchestra.run.smithi088.stderr:2024-06-03T08:35:32.372+0000 7fdda885f640 10 monclient: init 2024-06-03T08:35:32.373 INFO:teuthology.orchestra.run.smithi088.stderr:2024-06-03T08:35:32.373+0000 7fdda885f640 10 monclient: _reopen_session rank -1 2024-06-03T08:35:32.373 INFO:teuthology.orchestra.run.smithi088.stderr:2024-06-03T08:35:32.373+0000 7fdda885f640 10 monclient: _add_conns ranks=[1,2,0] 2024-06-03T08:35:32.373 INFO:teuthology.orchestra.run.smithi088.stderr:2024-06-03T08:35:32.373+0000 7fdda885f640 10 monclient(hunting): picked mon.noname-c con 0x7fdda40a8960 addr v1:172.21.15.192:6789/0 2024-06-03T08:35:32.374 INFO:teuthology.orchestra.run.smithi088.stderr:2024-06-03T08:35:32.373+0000 7fdda885f640 10 monclient(hunting): picked mon.noname-b con 0x7fdda41450f0 addr v1:172.21.15.88:6790/0 2024-06-03T08:35:32.374 INFO:teuthology.orchestra.run.smithi088.stderr:2024-06-03T08:35:32.373+0000 7fdda885f640 10 monclient(hunting): picked mon.noname-a con 0x7fdda413eec0 addr v1:172.21.15.88:6789/0 2024-06-03T08:35:32.374 INFO:teuthology.orchestra.run.smithi088.stderr:2024-06-03T08:35:32.373+0000 7fdda885f640 1 -- --> v1:172.21.15.88:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fdda4144c60 con 0x7fdda413eec0 2024-06-03T08:35:32.374 INFO:teuthology.orchestra.run.smithi088.stderr:2024-06-03T08:35:32.373+0000 7fdda885f640 1 -- --> v1:172.21.15.192:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fdda40b3930 con 0x7fdda40a8960 2024-06-03T08:35:32.374 INFO:teuthology.orchestra.run.smithi088.stderr:2024-06-03T08:35:32.373+0000 7fdda885f640 1 -- --> v1:172.21.15.88:6790/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fdda40b1ad0 con 0x7fdda41450f0 2024-06-03T08:35:32.374 INFO:teuthology.orchestra.run.smithi088.stderr:2024-06-03T08:35:32.373+0000 7fdda885f640 10 monclient(hunting): _renew_subs 2024-06-03T08:35:32.374 INFO:teuthology.orchestra.run.smithi088.stderr:2024-06-03T08:35:32.373+0000 7fdda885f640 10 monclient(hunting): authenticate will time out at 1188.183917s 2024-06-03T08:35:32.375 INFO:teuthology.orchestra.run.smithi088.stderr:2024-06-03T08:35:32.374+0000 7fdda259c640 1 --1- >> v1:172.21.15.192:6789/0 conn(0x7fdda40a8960 0x7fdda40a8d40 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.192:6789/0 says I am v1:172.21.15.88:33488/0 (socket says 172.21.15.88:33488) 2024-06-03T08:35:32.375 INFO:teuthology.orchestra.run.smithi088.stderr:2024-06-03T08:35:32.374+0000 7fdda259c640 1 -- 172.21.15.88:0/3052403824 learned_addr learned my addr 172.21.15.88:0/3052403824 (peer_addr_for_me v1:172.21.15.88:0/0) 2024-06-03T08:35:32.379 INFO:teuthology.orchestra.run.smithi088.stderr:2024-06-03T08:35:32.379+0000 7fdda159a640 1 -- 172.21.15.88:0/3052403824 <== mon.1 v1:172.21.15.192:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 1888155537 0 0) 0x7fdda40b3930 con 0x7fdda40a8960 2024-06-03T08:35:32.379 INFO:teuthology.orchestra.run.smithi088.stderr:2024-06-03T08:35:32.379+0000 7fdda159a640 10 monclient(hunting): _negotiate 2024-06-03T08:35:32.380 INFO:teuthology.orchestra.run.smithi088.stderr:2024-06-03T08:35:32.379+0000 7fdda159a640 10 monclient(hunting): _init_auth method 2 2024-06-03T08:35:32.380 INFO:teuthology.orchestra.run.smithi088.stderr:2024-06-03T08:35:32.379+0000 7fdda159a640 10 monclient(hunting): _init_auth creating new auth 2024-06-03T08:35:32.380 INFO:teuthology.orchestra.run.smithi088.stderr:2024-06-03T08:35:32.379+0000 7fdda159a640 10 monclient(hunting): my global_id is 4165 2024-06-03T08:35:32.380 INFO:teuthology.orchestra.run.smithi088.stderr:2024-06-03T08:35:32.379+0000 7fdda159a640 1 -- 172.21.15.88:0/3052403824 --> v1:172.21.15.192:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7fdd8c003520 con 0x7fdda40a8960 2024-06-03T08:35:32.380 INFO:teuthology.orchestra.run.smithi088.stderr:2024-06-03T08:35:32.379+0000 7fdda159a640 1 -- 172.21.15.88:0/3052403824 <== mon.2 v1:172.21.15.88:6790/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 3525215380 0 0) 0x7fdda40b1ad0 con 0x7fdda41450f0 2024-06-03T08:35:32.380 INFO:teuthology.orchestra.run.smithi088.stderr:2024-06-03T08:35:32.379+0000 7fdda159a640 10 monclient(hunting): _negotiate 2024-06-03T08:35:32.380 INFO:teuthology.orchestra.run.smithi088.stderr:2024-06-03T08:35:32.379+0000 7fdda159a640 10 monclient(hunting): _init_auth method 2 2024-06-03T08:35:32.380 INFO:teuthology.orchestra.run.smithi088.stderr:2024-06-03T08:35:32.379+0000 7fdda159a640 10 monclient(hunting): _init_auth creating new auth 2024-06-03T08:35:32.380 INFO:teuthology.orchestra.run.smithi088.stderr:2024-06-03T08:35:32.379+0000 7fdda159a640 10 monclient(hunting): my global_id is 4166 2024-06-03T08:35:32.380 INFO:teuthology.orchestra.run.smithi088.stderr:2024-06-03T08:35:32.379+0000 7fdda159a640 1 -- 172.21.15.88:0/3052403824 --> v1:172.21.15.88:6790/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7fdda40b3930 con 0x7fdda41450f0 2024-06-03T08:35:32.384 INFO:teuthology.orchestra.run.smithi088.stderr:2024-06-03T08:35:32.384+0000 7fdda159a640 1 -- 172.21.15.88:0/3052403824 <== mon.1 v1:172.21.15.192:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 214+0+0 (unknown 1868887593 0 0) 0x7fdd8c003520 con 0x7fdda40a8960 2024-06-03T08:35:32.384 INFO:teuthology.orchestra.run.smithi088.stderr:2024-06-03T08:35:32.384+0000 7fdda159a640 1 -- 172.21.15.88:0/3052403824 >> v1:172.21.15.88:6790/0 conn(0x7fdda41450f0 legacy=0x7fdda413e7b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-06-03T08:35:32.385 INFO:teuthology.orchestra.run.smithi088.stderr:2024-06-03T08:35:32.384+0000 7fdda159a640 1 -- 172.21.15.88:0/3052403824 >> v1:172.21.15.88:6789/0 conn(0x7fdda413eec0 legacy=0x7fdda414ce60 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-06-03T08:35:32.385 INFO:teuthology.orchestra.run.smithi088.stderr:2024-06-03T08:35:32.384+0000 7fdda159a640 10 monclient: _finish_hunting 0 2024-06-03T08:35:32.385 INFO:teuthology.orchestra.run.smithi088.stderr:2024-06-03T08:35:32.384+0000 7fdda159a640 1 monclient: found mon.noname-c 2024-06-03T08:35:32.385 INFO:teuthology.orchestra.run.smithi088.stderr:2024-06-03T08:35:32.384+0000 7fdda159a640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-06-03T08:35:32.385 INFO:teuthology.orchestra.run.smithi088.stderr:2024-06-03T08:35:32.384+0000 7fdda159a640 10 monclient: _send_mon_message to mon.noname-c at v1:172.21.15.192:6789/0 2024-06-03T08:35:32.385 INFO:teuthology.orchestra.run.smithi088.stderr:2024-06-03T08:35:32.384+0000 7fdda159a640 1 -- 172.21.15.88:0/3052403824 --> v1:172.21.15.192:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fdda40b1c60 con 0x7fdda40a8960 2024-06-03T08:35:32.385 INFO:teuthology.orchestra.run.smithi088.stderr:2024-06-03T08:35:32.384+0000 7fdda159a640 10 monclient: _finish_auth 0 2024-06-03T08:35:32.385 INFO:teuthology.orchestra.run.smithi088.stderr:2024-06-03T08:35:32.384+0000 7fdda159a640 10 monclient: _check_auth_tickets 2024-06-03T08:35:32.385 INFO:teuthology.orchestra.run.smithi088.stderr:2024-06-03T08:35:32.384+0000 7fdda159a640 20 monclient: _check_auth_rotating not needed by client.admin 2024-06-03T08:35:32.385 INFO:teuthology.orchestra.run.smithi088.stderr:2024-06-03T08:35:32.384+0000 7fdda159a640 1 -- 172.21.15.88:0/3052403824 <== mon.1 v1:172.21.15.192:6789/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 2234374065 0 0) 0x7fdd94001610 con 0x7fdda40a8960 2024-06-03T08:35:32.385 INFO:teuthology.orchestra.run.smithi088.stderr:2024-06-03T08:35:32.384+0000 7fdda159a640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-06-03T08:35:32.385 INFO:teuthology.orchestra.run.smithi088.stderr:2024-06-03T08:35:32.384+0000 7fdda159a640 10 monclient: got monmap 1 from mon.noname-c (according to old e1) 2024-06-03T08:35:32.385 INFO:teuthology.orchestra.run.smithi088.stderr:2024-06-03T08:35:32.384+0000 7fdda159a640 10 monclient: dump: 2024-06-03T08:35:32.385 INFO:teuthology.orchestra.run.smithi088.stderr:epoch 1 2024-06-03T08:35:32.386 INFO:teuthology.orchestra.run.smithi088.stderr:fsid ec3c3fdd-3ac6-4d7d-9412-683130ed8b04 2024-06-03T08:35:32.386 INFO:teuthology.orchestra.run.smithi088.stderr:last_changed 2024-06-03T08:34:31.885125+0000 2024-06-03T08:35:32.386 INFO:teuthology.orchestra.run.smithi088.stderr:created 2024-06-03T08:34:31.885125+0000 2024-06-03T08:35:32.386 INFO:teuthology.orchestra.run.smithi088.stderr:min_mon_release 18 (reef) 2024-06-03T08:35:32.386 INFO:teuthology.orchestra.run.smithi088.stderr:election_strategy: 3 2024-06-03T08:35:32.386 INFO:teuthology.orchestra.run.smithi088.stderr:0: v1:172.21.15.88:6789/0 mon.a 2024-06-03T08:35:32.386 INFO:teuthology.orchestra.run.smithi088.stderr:1: v1:172.21.15.192:6789/0 mon.b 2024-06-03T08:35:32.386 INFO:teuthology.orchestra.run.smithi088.stderr:2: v1:172.21.15.88:6790/0 mon.c 2024-06-03T08:35:32.386 INFO:teuthology.orchestra.run.smithi088.stderr: 2024-06-03T08:35:32.386 INFO:teuthology.orchestra.run.smithi088.stderr:2024-06-03T08:35:32.384+0000 7fdda885f640 5 monclient: authenticate success, global_id 4165 2024-06-03T08:35:32.386 INFO:teuthology.orchestra.run.smithi088.stderr:2024-06-03T08:35:32.384+0000 7fdda885f640 20 monclient: get_monmap_and_config waiting for monmap|config 2024-06-03T08:35:32.387 INFO:teuthology.orchestra.run.smithi088.stderr:2024-06-03T08:35:32.386+0000 7fdda159a640 1 -- 172.21.15.88:0/3052403824 <== mon.1 v1:172.21.15.192:6789/0 4 ==== config(0 keys) v1 ==== 4+0+0 (unknown 0 0 0) 0x7fdd94001830 con 0x7fdda40a8960 2024-06-03T08:35:32.387 INFO:teuthology.orchestra.run.smithi088.stderr:2024-06-03T08:35:32.386+0000 7fdda159a640 10 monclient: handle_config config(0 keys) v1 2024-06-03T08:35:32.387 INFO:teuthology.orchestra.run.smithi088.stderr:2024-06-03T08:35:32.386+0000 7fdda159a640 1 -- 172.21.15.88:0/3052403824 <== mon.1 v1:172.21.15.192:6789/0 5 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 2234374065 0 0) 0x7fdd94001b40 con 0x7fdda40a8960 2024-06-03T08:35:32.387 INFO:teuthology.orchestra.run.smithi088.stderr:2024-06-03T08:35:32.386+0000 7fdda159a640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-06-03T08:35:32.387 INFO:teuthology.orchestra.run.smithi088.stderr:2024-06-03T08:35:32.386+0000 7fdda159a640 10 monclient: got monmap 1 from mon.b (according to old e1) 2024-06-03T08:35:32.387 INFO:teuthology.orchestra.run.smithi088.stderr:2024-06-03T08:35:32.386+0000 7fdda159a640 10 monclient: dump: 2024-06-03T08:35:32.387 INFO:teuthology.orchestra.run.smithi088.stderr:epoch 1 2024-06-03T08:35:32.388 INFO:teuthology.orchestra.run.smithi088.stderr:fsid ec3c3fdd-3ac6-4d7d-9412-683130ed8b04 2024-06-03T08:35:32.388 INFO:teuthology.orchestra.run.smithi088.stderr:last_changed 2024-06-03T08:34:31.885125+0000 2024-06-03T08:35:32.388 INFO:teuthology.orchestra.run.smithi088.stderr:created 2024-06-03T08:34:31.885125+0000 2024-06-03T08:35:32.388 INFO:teuthology.orchestra.run.smithi088.stderr:min_mon_release 18 (reef) 2024-06-03T08:35:32.388 INFO:teuthology.orchestra.run.smithi088.stderr:election_strategy: 3 2024-06-03T08:35:32.388 INFO:teuthology.orchestra.run.smithi088.stderr:0: v1:172.21.15.88:6789/0 mon.a 2024-06-03T08:35:32.388 INFO:teuthology.orchestra.run.smithi088.stderr:1: v1:172.21.15.192:6789/0 mon.b 2024-06-03T08:35:32.388 INFO:teuthology.orchestra.run.smithi088.stderr:2: v1:172.21.15.88:6790/0 mon.c 2024-06-03T08:35:32.388 INFO:teuthology.orchestra.run.smithi088.stderr: 2024-06-03T08:35:32.388 INFO:teuthology.orchestra.run.smithi088.stderr:2024-06-03T08:35:32.386+0000 7fdda885f640 10 monclient: get_monmap_and_config success 2024-06-03T08:35:32.388 INFO:teuthology.orchestra.run.smithi088.stderr:2024-06-03T08:35:32.386+0000 7fdda885f640 4 set_mon_vals no callback set 2024-06-03T08:35:32.389 INFO:teuthology.orchestra.run.smithi088.stderr:2024-06-03T08:35:32.387+0000 7fdda885f640 10 monclient: shutdown 2024-06-03T08:35:32.389 INFO:teuthology.orchestra.run.smithi088.stderr:2024-06-03T08:35:32.387+0000 7fdda885f640 20 monclient: shutdown discarding 0 pending message(s) 2024-06-03T08:35:32.389 INFO:teuthology.orchestra.run.smithi088.stderr:2024-06-03T08:35:32.387+0000 7fdda885f640 1 -- 172.21.15.88:0/3052403824 >> v1:172.21.15.192:6789/0 conn(0x7fdda40a8960 legacy=0x7fdda40a8d40 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-06-03T08:35:32.391 INFO:teuthology.orchestra.run.smithi088.stderr:2024-06-03T08:35:32.391+0000 7fdda885f640 1 -- 172.21.15.88:0/3052403824 shutdown_connections 2024-06-03T08:35:32.391 INFO:teuthology.orchestra.run.smithi088.stderr:2024-06-03T08:35:32.391+0000 7fdda885f640 1 -- 172.21.15.88:0/3052403824 >> 172.21.15.88:0/3052403824 conn(0x7fdda40b70d0 msgr2=0x7fdda40b74e0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-06-03T08:35:32.393 INFO:teuthology.orchestra.run.smithi088.stderr:2024-06-03T08:35:32.393+0000 7fdda885f640 1 -- 172.21.15.88:0/3052403824 shutdown_connections 2024-06-03T08:35:32.393 INFO:teuthology.orchestra.run.smithi088.stderr:2024-06-03T08:35:32.393+0000 7fdda885f640 1 -- 172.21.15.88:0/3052403824 wait complete. 2024-06-03T08:35:32.394 INFO:teuthology.orchestra.run.smithi088.stderr:2024-06-03T08:35:32.394+0000 7fdda885f640 10 monclient: build_initial_monmap 2024-06-03T08:35:32.394 INFO:teuthology.orchestra.run.smithi088.stderr:2024-06-03T08:35:32.394+0000 7fdda885f640 10 monclient: monmap: 2024-06-03T08:35:32.395 INFO:teuthology.orchestra.run.smithi088.stderr:epoch 0 2024-06-03T08:35:32.395 INFO:teuthology.orchestra.run.smithi088.stderr:fsid 00000000-0000-0000-0000-000000000000 2024-06-03T08:35:32.395 INFO:teuthology.orchestra.run.smithi088.stderr:last_changed 0.000000 2024-06-03T08:35:32.395 INFO:teuthology.orchestra.run.smithi088.stderr:created 0.000000 2024-06-03T08:35:32.395 INFO:teuthology.orchestra.run.smithi088.stderr:min_mon_release 0 (unknown) 2024-06-03T08:35:32.395 INFO:teuthology.orchestra.run.smithi088.stderr:election_strategy: 1 2024-06-03T08:35:32.395 INFO:teuthology.orchestra.run.smithi088.stderr:0: v1:172.21.15.88:6789/0 mon.noname-a 2024-06-03T08:35:32.395 INFO:teuthology.orchestra.run.smithi088.stderr:1: v1:172.21.15.192:6789/0 mon.noname-b 2024-06-03T08:35:32.395 INFO:teuthology.orchestra.run.smithi088.stderr:2: v1:172.21.15.88:6790/0 mon.noname-c 2024-06-03T08:35:32.395 INFO:teuthology.orchestra.run.smithi088.stderr: 2024-06-03T08:35:32.396 INFO:teuthology.orchestra.run.smithi088.stderr:2024-06-03T08:35:32.395+0000 7fdda885f640 1 librados: starting msgr at 2024-06-03T08:35:32.396 INFO:teuthology.orchestra.run.smithi088.stderr:2024-06-03T08:35:32.395+0000 7fdda885f640 1 librados: starting objecter 2024-06-03T08:35:32.396 INFO:teuthology.orchestra.run.smithi088.stderr:2024-06-03T08:35:32.395+0000 7fdda885f640 1 Processor -- start 2024-06-03T08:35:32.396 INFO:teuthology.orchestra.run.smithi088.stderr:2024-06-03T08:35:32.395+0000 7fdda885f640 1 -- start start 2024-06-03T08:35:32.396 INFO:teuthology.orchestra.run.smithi088.stderr:2024-06-03T08:35:32.395+0000 7fdda885f640 1 librados: setting wanted keys 2024-06-03T08:35:32.396 INFO:teuthology.orchestra.run.smithi088.stderr:2024-06-03T08:35:32.395+0000 7fdda885f640 1 librados: calling monclient init 2024-06-03T08:35:32.397 INFO:teuthology.orchestra.run.smithi088.stderr:2024-06-03T08:35:32.395+0000 7fdda885f640 10 monclient: init 2024-06-03T08:35:32.398 INFO:teuthology.orchestra.run.smithi088.stderr:2024-06-03T08:35:32.396+0000 7fdda885f640 10 monclient: _reopen_session rank -1 2024-06-03T08:35:32.398 INFO:teuthology.orchestra.run.smithi088.stderr:2024-06-03T08:35:32.396+0000 7fdda885f640 10 monclient: _add_conns ranks=[1,2,0] 2024-06-03T08:35:32.398 INFO:teuthology.orchestra.run.smithi088.stderr:2024-06-03T08:35:32.397+0000 7fdda885f640 10 monclient(hunting): picked mon.noname-b con 0x7fdda40a8960 addr v1:172.21.15.192:6789/0 2024-06-03T08:35:32.398 INFO:teuthology.orchestra.run.smithi088.stderr:2024-06-03T08:35:32.397+0000 7fdda885f640 10 monclient(hunting): picked mon.noname-c con 0x7fdda413eec0 addr v1:172.21.15.88:6790/0 2024-06-03T08:35:32.398 INFO:teuthology.orchestra.run.smithi088.stderr:2024-06-03T08:35:32.397+0000 7fdda885f640 10 monclient(hunting): picked mon.noname-a con 0x7fdda41450f0 addr v1:172.21.15.88:6789/0 2024-06-03T08:35:32.398 INFO:teuthology.orchestra.run.smithi088.stderr:2024-06-03T08:35:32.397+0000 7fdda885f640 1 -- --> v1:172.21.15.88:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fdda40ac8f0 con 0x7fdda41450f0 2024-06-03T08:35:32.398 INFO:teuthology.orchestra.run.smithi088.stderr:2024-06-03T08:35:32.397+0000 7fdda885f640 1 -- --> v1:172.21.15.192:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fdda41d9c20 con 0x7fdda40a8960 2024-06-03T08:35:32.398 INFO:teuthology.orchestra.run.smithi088.stderr:2024-06-03T08:35:32.397+0000 7fdda885f640 1 -- --> v1:172.21.15.88:6790/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fdda41414b0 con 0x7fdda413eec0 2024-06-03T08:35:32.399 INFO:teuthology.orchestra.run.smithi088.stderr:2024-06-03T08:35:32.397+0000 7fdda885f640 10 monclient(hunting): _renew_subs 2024-06-03T08:35:32.399 INFO:teuthology.orchestra.run.smithi088.stderr:2024-06-03T08:35:32.397+0000 7fdda1d9b640 1 --1- >> v1:172.21.15.88:6790/0 conn(0x7fdda413eec0 0x7fdda41da140 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=1).handle_server_banner_and_identify peer v1:172.21.15.88:6790/0 says I am v1:172.21.15.88:42634/0 (socket says 172.21.15.88:42634) 2024-06-03T08:35:32.399 INFO:teuthology.orchestra.run.smithi088.stderr:2024-06-03T08:35:32.397+0000 7fdda1d9b640 1 -- 172.21.15.88:0/1890752434 learned_addr learned my addr 172.21.15.88:0/1890752434 (peer_addr_for_me v1:172.21.15.88:0/0) 2024-06-03T08:35:32.399 INFO:teuthology.orchestra.run.smithi088.stderr:2024-06-03T08:35:32.397+0000 7fdda885f640 10 monclient(hunting): authenticate will time out at 1188.207919s 2024-06-03T08:35:32.399 INFO:teuthology.orchestra.run.smithi088.stderr:2024-06-03T08:35:32.398+0000 7fdd92ffd640 10 client.?.objecter ms_handle_connect 0x7fdda40a8960 2024-06-03T08:35:32.399 INFO:teuthology.orchestra.run.smithi088.stderr:2024-06-03T08:35:32.398+0000 7fdd92ffd640 10 client.?.objecter resend_mon_ops 2024-06-03T08:35:32.400 INFO:teuthology.orchestra.run.smithi088.stderr:2024-06-03T08:35:32.398+0000 7fdd92ffd640 10 client.?.objecter ms_handle_connect 0x7fdda413eec0 2024-06-03T08:35:32.400 INFO:teuthology.orchestra.run.smithi088.stderr:2024-06-03T08:35:32.398+0000 7fdd92ffd640 10 client.?.objecter resend_mon_ops 2024-06-03T08:35:32.401 INFO:teuthology.orchestra.run.smithi088.stderr:2024-06-03T08:35:32.400+0000 7fdd92ffd640 1 -- 172.21.15.88:0/1890752434 <== mon.2 v1:172.21.15.88:6790/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 398376404 0 0) 0x7fdda41414b0 con 0x7fdda413eec0 2024-06-03T08:35:32.401 INFO:teuthology.orchestra.run.smithi088.stderr:2024-06-03T08:35:32.400+0000 7fdd92ffd640 10 monclient(hunting): _negotiate 2024-06-03T08:35:32.401 INFO:teuthology.orchestra.run.smithi088.stderr:2024-06-03T08:35:32.400+0000 7fdd92ffd640 10 monclient(hunting): _init_auth method 2 2024-06-03T08:35:32.401 INFO:teuthology.orchestra.run.smithi088.stderr:2024-06-03T08:35:32.400+0000 7fdd92ffd640 10 monclient(hunting): _init_auth creating new auth 2024-06-03T08:35:32.401 INFO:teuthology.orchestra.run.smithi088.stderr:2024-06-03T08:35:32.400+0000 7fdd92ffd640 10 monclient(hunting): my global_id is 4169 2024-06-03T08:35:32.401 INFO:teuthology.orchestra.run.smithi088.stderr:2024-06-03T08:35:32.400+0000 7fdd92ffd640 1 -- 172.21.15.88:0/1890752434 --> v1:172.21.15.88:6790/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7fdd84004eb0 con 0x7fdda413eec0 2024-06-03T08:35:32.402 INFO:teuthology.orchestra.run.smithi088.stderr:2024-06-03T08:35:32.401+0000 7fdd92ffd640 1 -- 172.21.15.88:0/1890752434 <== mon.1 v1:172.21.15.192:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 1265088760 0 0) 0x7fdda41d9c20 con 0x7fdda40a8960 2024-06-03T08:35:32.402 INFO:teuthology.orchestra.run.smithi088.stderr:2024-06-03T08:35:32.401+0000 7fdd92ffd640 10 monclient(hunting): _negotiate 2024-06-03T08:35:32.402 INFO:teuthology.orchestra.run.smithi088.stderr:2024-06-03T08:35:32.401+0000 7fdd92ffd640 10 monclient(hunting): _init_auth method 2 2024-06-03T08:35:32.402 INFO:teuthology.orchestra.run.smithi088.stderr:2024-06-03T08:35:32.401+0000 7fdd92ffd640 10 monclient(hunting): _init_auth creating new auth 2024-06-03T08:35:32.402 INFO:teuthology.orchestra.run.smithi088.stderr:2024-06-03T08:35:32.401+0000 7fdd92ffd640 10 monclient(hunting): my global_id is 4168 2024-06-03T08:35:32.403 INFO:teuthology.orchestra.run.smithi088.stderr:2024-06-03T08:35:32.401+0000 7fdd92ffd640 1 -- 172.21.15.88:0/1890752434 --> v1:172.21.15.192:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7fdda41414b0 con 0x7fdda40a8960 2024-06-03T08:35:32.406 INFO:teuthology.orchestra.run.smithi088.stderr:2024-06-03T08:35:32.405+0000 7fdd92ffd640 1 -- 172.21.15.88:0/1890752434 <== mon.2 v1:172.21.15.88:6790/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 764+0+0 (unknown 1386005932 0 0) 0x7fdd84004eb0 con 0x7fdda413eec0 2024-06-03T08:35:32.406 INFO:teuthology.orchestra.run.smithi088.stderr:2024-06-03T08:35:32.405+0000 7fdd92ffd640 1 -- 172.21.15.88:0/1890752434 --> v1:172.21.15.88:6790/0 -- auth(proto 2 165 bytes epoch 0) v1 -- 0x7fdda41d9c20 con 0x7fdda413eec0 2024-06-03T08:35:32.406 INFO:teuthology.orchestra.run.smithi088.stderr:2024-06-03T08:35:32.405+0000 7fdd92ffd640 1 -- 172.21.15.88:0/1890752434 <== mon.2 v1:172.21.15.88:6790/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 2234374065 0 0) 0x7fdd980018f0 con 0x7fdda413eec0 2024-06-03T08:35:32.406 INFO:teuthology.orchestra.run.smithi088.stderr:2024-06-03T08:35:32.405+0000 7fdd92ffd640 10 monclient(hunting): handle_monmap mon_map magic: 0 v1 2024-06-03T08:35:32.406 INFO:teuthology.orchestra.run.smithi088.stderr:2024-06-03T08:35:32.406+0000 7fdd92ffd640 10 monclient(hunting): got monmap 1 from mon.noname-c (according to old e1) 2024-06-03T08:35:32.406 INFO:teuthology.orchestra.run.smithi088.stderr:2024-06-03T08:35:32.406+0000 7fdd92ffd640 10 monclient(hunting): dump: 2024-06-03T08:35:32.406 INFO:teuthology.orchestra.run.smithi088.stderr:epoch 1 2024-06-03T08:35:32.406 INFO:teuthology.orchestra.run.smithi088.stderr:fsid ec3c3fdd-3ac6-4d7d-9412-683130ed8b04 2024-06-03T08:35:32.406 INFO:teuthology.orchestra.run.smithi088.stderr:last_changed 2024-06-03T08:34:31.885125+0000 2024-06-03T08:35:32.407 INFO:teuthology.orchestra.run.smithi088.stderr:created 2024-06-03T08:34:31.885125+0000 2024-06-03T08:35:32.407 INFO:teuthology.orchestra.run.smithi088.stderr:min_mon_release 18 (reef) 2024-06-03T08:35:32.407 INFO:teuthology.orchestra.run.smithi088.stderr:election_strategy: 3 2024-06-03T08:35:32.407 INFO:teuthology.orchestra.run.smithi088.stderr:0: v1:172.21.15.88:6789/0 mon.a 2024-06-03T08:35:32.407 INFO:teuthology.orchestra.run.smithi088.stderr:1: v1:172.21.15.192:6789/0 mon.b 2024-06-03T08:35:32.407 INFO:teuthology.orchestra.run.smithi088.stderr:2: v1:172.21.15.88:6790/0 mon.c 2024-06-03T08:35:32.407 INFO:teuthology.orchestra.run.smithi088.stderr: 2024-06-03T08:35:32.407 INFO:teuthology.orchestra.run.smithi088.stderr:2024-06-03T08:35:32.406+0000 7fdd92ffd640 10 monclient(hunting): _finish_auth 0 2024-06-03T08:35:32.409 INFO:teuthology.orchestra.run.smithi088.stderr:2024-06-03T08:35:32.408+0000 7fdd92ffd640 1 -- 172.21.15.88:0/1890752434 <== mon.1 v1:172.21.15.192:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 764+0+0 (unknown 1662513621 0 0) 0x7fdda41414b0 con 0x7fdda40a8960 2024-06-03T08:35:32.409 INFO:teuthology.orchestra.run.smithi088.stderr:2024-06-03T08:35:32.408+0000 7fdd92ffd640 1 -- 172.21.15.88:0/1890752434 --> v1:172.21.15.192:6789/0 -- auth(proto 2 165 bytes epoch 0) v1 -- 0x7fdd84004eb0 con 0x7fdda40a8960 2024-06-03T08:35:32.409 INFO:teuthology.orchestra.run.smithi088.stderr:2024-06-03T08:35:32.408+0000 7fdd92ffd640 1 -- 172.21.15.88:0/1890752434 <== mon.1 v1:172.21.15.192:6789/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 2234374065 0 0) 0x7fdd94001510 con 0x7fdda40a8960 2024-06-03T08:35:32.409 INFO:teuthology.orchestra.run.smithi088.stderr:2024-06-03T08:35:32.408+0000 7fdd92ffd640 10 monclient(hunting): handle_monmap mon_map magic: 0 v1 2024-06-03T08:35:32.410 INFO:teuthology.orchestra.run.smithi088.stderr:2024-06-03T08:35:32.408+0000 7fdd92ffd640 10 monclient(hunting): got monmap 1 from mon.b (according to old e1) 2024-06-03T08:35:32.410 INFO:teuthology.orchestra.run.smithi088.stderr:2024-06-03T08:35:32.408+0000 7fdd92ffd640 10 monclient(hunting): dump: 2024-06-03T08:35:32.410 INFO:teuthology.orchestra.run.smithi088.stderr:epoch 1 2024-06-03T08:35:32.410 INFO:teuthology.orchestra.run.smithi088.stderr:fsid ec3c3fdd-3ac6-4d7d-9412-683130ed8b04 2024-06-03T08:35:32.410 INFO:teuthology.orchestra.run.smithi088.stderr:last_changed 2024-06-03T08:34:31.885125+0000 2024-06-03T08:35:32.410 INFO:teuthology.orchestra.run.smithi088.stderr:created 2024-06-03T08:34:31.885125+0000 2024-06-03T08:35:32.410 INFO:teuthology.orchestra.run.smithi088.stderr:min_mon_release 18 (reef) 2024-06-03T08:35:32.410 INFO:teuthology.orchestra.run.smithi088.stderr:election_strategy: 3 2024-06-03T08:35:32.410 INFO:teuthology.orchestra.run.smithi088.stderr:0: v1:172.21.15.88:6789/0 mon.a 2024-06-03T08:35:32.410 INFO:teuthology.orchestra.run.smithi088.stderr:1: v1:172.21.15.192:6789/0 mon.b 2024-06-03T08:35:32.411 INFO:teuthology.orchestra.run.smithi088.stderr:2: v1:172.21.15.88:6790/0 mon.c 2024-06-03T08:35:32.411 INFO:teuthology.orchestra.run.smithi088.stderr: 2024-06-03T08:35:32.413 INFO:teuthology.orchestra.run.smithi088.stderr:2024-06-03T08:35:32.410+0000 7fdd92ffd640 1 -- 172.21.15.88:0/1890752434 <== mon.2 v1:172.21.15.88:6790/0 4 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 580+0+0 (unknown 2660357289 0 0) 0x7fdda41d9c20 con 0x7fdda413eec0 2024-06-03T08:35:32.413 INFO:teuthology.orchestra.run.smithi088.stderr:2024-06-03T08:35:32.410+0000 7fdd92ffd640 1 -- 172.21.15.88:0/1890752434 >> v1:172.21.15.192:6789/0 conn(0x7fdda40a8960 legacy=0x7fdda40aaff0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-06-03T08:35:32.413 INFO:teuthology.orchestra.run.smithi088.stderr:2024-06-03T08:35:32.410+0000 7fdd92ffd640 1 -- 172.21.15.88:0/1890752434 >> v1:172.21.15.88:6789/0 conn(0x7fdda41450f0 legacy=0x7fdda40b0090 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-06-03T08:35:32.413 INFO:teuthology.orchestra.run.smithi088.stderr:2024-06-03T08:35:32.410+0000 7fdd92ffd640 10 monclient: _finish_hunting 0 2024-06-03T08:35:32.413 INFO:teuthology.orchestra.run.smithi088.stderr:2024-06-03T08:35:32.410+0000 7fdd92ffd640 1 monclient: found mon.c 2024-06-03T08:35:32.413 INFO:teuthology.orchestra.run.smithi088.stderr:2024-06-03T08:35:32.410+0000 7fdd92ffd640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-06-03T08:35:32.413 INFO:teuthology.orchestra.run.smithi088.stderr:2024-06-03T08:35:32.410+0000 7fdd92ffd640 10 monclient: _send_mon_message to mon.c at v1:172.21.15.88:6790/0 2024-06-03T08:35:32.413 INFO:teuthology.orchestra.run.smithi088.stderr:2024-06-03T08:35:32.410+0000 7fdd92ffd640 1 -- 172.21.15.88:0/1890752434 --> v1:172.21.15.88:6790/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fdda413f6c0 con 0x7fdda413eec0 2024-06-03T08:35:32.413 INFO:teuthology.orchestra.run.smithi088.stderr:2024-06-03T08:35:32.410+0000 7fdd92ffd640 10 monclient: _finish_auth 0 2024-06-03T08:35:32.413 INFO:teuthology.orchestra.run.smithi088.stderr:2024-06-03T08:35:32.410+0000 7fdd92ffd640 10 monclient: _check_auth_tickets 2024-06-03T08:35:32.413 INFO:teuthology.orchestra.run.smithi088.stderr:2024-06-03T08:35:32.410+0000 7fdd92ffd640 20 monclient: _check_auth_rotating not needed by client.admin 2024-06-03T08:35:32.414 INFO:teuthology.orchestra.run.smithi088.stderr:2024-06-03T08:35:32.410+0000 7fdda885f640 5 monclient: authenticate success, global_id 4169 2024-06-03T08:35:32.414 INFO:teuthology.orchestra.run.smithi088.stderr:2024-06-03T08:35:32.410+0000 7fdda885f640 10 monclient: _renew_subs 2024-06-03T08:35:32.414 INFO:teuthology.orchestra.run.smithi088.stderr:2024-06-03T08:35:32.410+0000 7fdda885f640 10 monclient: _send_mon_message to mon.c at v1:172.21.15.88:6790/0 2024-06-03T08:35:32.414 INFO:teuthology.orchestra.run.smithi088.stderr:2024-06-03T08:35:32.410+0000 7fdda885f640 1 -- 172.21.15.88:0/1890752434 --> v1:172.21.15.88:6790/0 -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fdda41da850 con 0x7fdda413eec0 2024-06-03T08:35:32.414 INFO:teuthology.orchestra.run.smithi088.stderr:2024-06-03T08:35:32.410+0000 7fdda885f640 10 client.4169.objecter _maybe_request_map subscribing (onetime) to next osd map 2024-06-03T08:35:32.414 INFO:teuthology.orchestra.run.smithi088.stderr:2024-06-03T08:35:32.410+0000 7fdda885f640 10 monclient: _renew_subs 2024-06-03T08:35:32.414 INFO:teuthology.orchestra.run.smithi088.stderr:2024-06-03T08:35:32.410+0000 7fdda885f640 10 monclient: _send_mon_message to mon.c at v1:172.21.15.88:6790/0 2024-06-03T08:35:32.414 INFO:teuthology.orchestra.run.smithi088.stderr:2024-06-03T08:35:32.410+0000 7fdda885f640 1 -- 172.21.15.88:0/1890752434 --> v1:172.21.15.88:6790/0 -- mon_subscribe({osdmap=0}) v3 -- 0x7fdda41424d0 con 0x7fdda413eec0 2024-06-03T08:35:32.414 INFO:teuthology.orchestra.run.smithi088.stderr:2024-06-03T08:35:32.410+0000 7fdda885f640 1 librados: init done 2024-06-03T08:35:32.414 INFO:teuthology.orchestra.run.smithi088.stderr:2024-06-03T08:35:32.411+0000 7fdd90ff9640 10 start_mon_command cmd=[{"prefix": "get_command_descriptions"}] 2024-06-03T08:35:32.414 INFO:teuthology.orchestra.run.smithi088.stderr:2024-06-03T08:35:32.411+0000 7fdd90ff9640 10 monclient: _send_command 1 [{"prefix": "get_command_descriptions"}] 2024-06-03T08:35:32.414 INFO:teuthology.orchestra.run.smithi088.stderr:2024-06-03T08:35:32.411+0000 7fdd90ff9640 10 monclient: _send_mon_message to mon.c at v1:172.21.15.88:6790/0 2024-06-03T08:35:32.414 INFO:teuthology.orchestra.run.smithi088.stderr:2024-06-03T08:35:32.411+0000 7fdd90ff9640 1 -- 172.21.15.88:0/1890752434 --> v1:172.21.15.88:6790/0 -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fdda40a9e60 con 0x7fdda413eec0 2024-06-03T08:35:32.416 INFO:teuthology.orchestra.run.smithi088.stderr:2024-06-03T08:35:32.415+0000 7fdd92ffd640 1 -- 172.21.15.88:0/1890752434 <== mon.2 v1:172.21.15.88:6790/0 5 ==== config(0 keys) v1 ==== 4+0+0 (unknown 0 0 0) 0x7fdd98003610 con 0x7fdda413eec0 2024-06-03T08:35:32.416 INFO:teuthology.orchestra.run.smithi088.stderr:2024-06-03T08:35:32.415+0000 7fdd92ffd640 10 monclient: handle_config config(0 keys) v1 2024-06-03T08:35:32.416 INFO:teuthology.orchestra.run.smithi088.stderr:2024-06-03T08:35:32.415+0000 7fdd92ffd640 1 -- 172.21.15.88:0/1890752434 <== mon.2 v1:172.21.15.88:6790/0 6 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 2234374065 0 0) 0x7fdd98003920 con 0x7fdda413eec0 2024-06-03T08:35:32.416 INFO:teuthology.orchestra.run.smithi088.stderr:2024-06-03T08:35:32.415+0000 7fdd92ffd640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-06-03T08:35:32.416 INFO:teuthology.orchestra.run.smithi088.stderr:2024-06-03T08:35:32.415+0000 7fdd92ffd640 10 monclient: got monmap 1 from mon.c (according to old e1) 2024-06-03T08:35:32.416 INFO:teuthology.orchestra.run.smithi088.stderr:2024-06-03T08:35:32.415+0000 7fdd92ffd640 10 monclient: dump: 2024-06-03T08:35:32.417 INFO:teuthology.orchestra.run.smithi088.stderr:epoch 1 2024-06-03T08:35:32.417 INFO:teuthology.orchestra.run.smithi088.stderr:fsid ec3c3fdd-3ac6-4d7d-9412-683130ed8b04 2024-06-03T08:35:32.417 INFO:teuthology.orchestra.run.smithi088.stderr:last_changed 2024-06-03T08:34:31.885125+0000 2024-06-03T08:35:32.417 INFO:teuthology.orchestra.run.smithi088.stderr:created 2024-06-03T08:34:31.885125+0000 2024-06-03T08:35:32.417 INFO:teuthology.orchestra.run.smithi088.stderr:min_mon_release 18 (reef) 2024-06-03T08:35:32.417 INFO:teuthology.orchestra.run.smithi088.stderr:election_strategy: 3 2024-06-03T08:35:32.417 INFO:teuthology.orchestra.run.smithi088.stderr:0: v1:172.21.15.88:6789/0 mon.a 2024-06-03T08:35:32.417 INFO:teuthology.orchestra.run.smithi088.stderr:1: v1:172.21.15.192:6789/0 mon.b 2024-06-03T08:35:32.417 INFO:teuthology.orchestra.run.smithi088.stderr:2: v1:172.21.15.88:6790/0 mon.c 2024-06-03T08:35:32.417 INFO:teuthology.orchestra.run.smithi088.stderr: 2024-06-03T08:35:32.418 INFO:teuthology.orchestra.run.smithi088.stderr:2024-06-03T08:35:32.415+0000 7fdd92ffd640 1 -- 172.21.15.88:0/1890752434 <== mon.2 v1:172.21.15.88:6790/0 7 ==== mgrmap(e 1) v1 ==== 668+0+0 (unknown 3836259515 0 0) 0x7fdd98003d80 con 0x7fdda413eec0 2024-06-03T08:35:32.418 INFO:teuthology.orchestra.run.smithi088.stderr:2024-06-03T08:35:32.415+0000 7fdd92ffd640 1 -- 172.21.15.88:0/1890752434 <== mon.2 v1:172.21.15.88:6790/0 8 ==== osd_map(10..10 src has 1..10) v4 ==== 1645+0+0 (unknown 309997979 0 0) 0x7fdd98006500 con 0x7fdda413eec0 2024-06-03T08:35:32.418 INFO:teuthology.orchestra.run.smithi088.stderr:2024-06-03T08:35:32.415+0000 7fdd92ffd640 10 client.4169.objecter ms_dispatch 0x7fdda4000b90 osd_map(10..10 src has 1..10) v4 2024-06-03T08:35:32.418 INFO:teuthology.orchestra.run.smithi088.stderr:2024-06-03T08:35:32.415+0000 7fdd92ffd640 3 client.4169.objecter handle_osd_map got epochs [10,10] > 0 2024-06-03T08:35:32.418 INFO:teuthology.orchestra.run.smithi088.stderr:2024-06-03T08:35:32.415+0000 7fdd92ffd640 3 client.4169.objecter handle_osd_map decoding full epoch 10 2024-06-03T08:35:32.418 INFO:teuthology.orchestra.run.smithi088.stderr:2024-06-03T08:35:32.415+0000 7fdd92ffd640 20 client.4169.objecter dump_active .. 0 homeless 2024-06-03T08:35:32.418 INFO:teuthology.orchestra.run.smithi088.stderr:2024-06-03T08:35:32.415+0000 7fdda0d99640 4 set_mon_vals no callback set 2024-06-03T08:35:32.419 INFO:tasks.ceph.mgr.y.smithi088.stderr:2024-06-03T08:35:32.419+0000 7ff0aea1d040 -1 mgr[py] Module status has missing NOTIFY_TYPES member 2024-06-03T08:35:32.522 INFO:teuthology.orchestra.run.smithi088.stderr:2024-06-03T08:35:32.521+0000 7fdd92ffd640 1 -- 172.21.15.88:0/1890752434 <== mon.2 v1:172.21.15.88:6790/0 9 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+73121 (unknown 1092875540 0 1417653156) 0x7fdd980057c0 con 0x7fdda413eec0 2024-06-03T08:35:32.522 INFO:teuthology.orchestra.run.smithi088.stderr:2024-06-03T08:35:32.521+0000 7fdd92ffd640 10 monclient: handle_mon_command_ack 1 [{"prefix": "get_command_descriptions"}] 2024-06-03T08:35:32.522 INFO:teuthology.orchestra.run.smithi088.stderr:2024-06-03T08:35:32.521+0000 7fdd92ffd640 10 monclient: _finish_command 1 = system:0 2024-06-03T08:35:32.632 INFO:tasks.ceph.mgr.y.smithi088.stderr:2024-06-03T08:35:32.631+0000 7ff0aea1d040 -1 mgr[py] Module nfs has missing NOTIFY_TYPES member 2024-06-03T08:35:32.643 INFO:teuthology.orchestra.run.smithi088.stderr:2024-06-03T08:35:32.642+0000 7fdd90ff9640 10 start_mon_command cmd=[{"prefix": "osd dump", "format": "json"}] 2024-06-03T08:35:32.643 INFO:teuthology.orchestra.run.smithi088.stderr:2024-06-03T08:35:32.642+0000 7fdd90ff9640 10 monclient: _send_command 2 [{"prefix": "osd dump", "format": "json"}] 2024-06-03T08:35:32.643 INFO:teuthology.orchestra.run.smithi088.stderr:2024-06-03T08:35:32.642+0000 7fdd90ff9640 10 monclient: _send_mon_message to mon.c at v1:172.21.15.88:6790/0 2024-06-03T08:35:32.643 INFO:teuthology.orchestra.run.smithi088.stderr:2024-06-03T08:35:32.642+0000 7fdd90ff9640 1 -- 172.21.15.88:0/1890752434 --> v1:172.21.15.88:6790/0 -- mon_command({"prefix": "osd dump", "format": "json"} v 0) v1 -- 0x7fdda404ea60 con 0x7fdda413eec0 2024-06-03T08:35:32.649 INFO:teuthology.orchestra.run.smithi088.stderr:2024-06-03T08:35:32.649+0000 7fdd92ffd640 1 -- 172.21.15.88:0/1890752434 <== mon.2 v1:172.21.15.88:6790/0 10 ==== mgrmap(e 2) v1 ==== 44528+0+0 (unknown 38132633 0 0) 0x7fdd98004030 con 0x7fdda413eec0 2024-06-03T08:35:32.786 INFO:teuthology.orchestra.run.smithi088.stderr:2024-06-03T08:35:32.784+0000 7fdd92ffd640 1 -- 172.21.15.88:0/1890752434 <== mon.2 v1:172.21.15.88:6790/0 11 ==== mon_command_ack([{"prefix": "osd dump", "format": "json"}]=0 v10) v1 ==== 74+0+7057 (unknown 4040575075 0 1067785724) 0x7fdd98015b30 con 0x7fdda413eec0 2024-06-03T08:35:32.786 INFO:teuthology.orchestra.run.smithi088.stderr:2024-06-03T08:35:32.785+0000 7fdd92ffd640 10 monclient: handle_mon_command_ack 2 [{"prefix": "osd dump", "format": "json"}] 2024-06-03T08:35:32.786 INFO:teuthology.orchestra.run.smithi088.stderr:2024-06-03T08:35:32.785+0000 7fdd92ffd640 10 monclient: _finish_command 2 = system:0 2024-06-03T08:35:32.787 INFO:teuthology.orchestra.run.smithi088.stdout: 2024-06-03T08:35:32.787 INFO:teuthology.orchestra.run.smithi088.stdout:{"epoch":10,"fsid":"ec3c3fdd-3ac6-4d7d-9412-683130ed8b04","created":"2024-06-03T08:35:26.799638+0000","modified":"2024-06-03T08:35:32.183068+0000","last_up_change":"0.000000","last_in_change":"2024-06-03T08:35:32.183068+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":2,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":0,"max_osd":8,"require_min_compat_client":"luminous","min_compat_client":"jewel","require_osd_release":"reef","allow_crimson":false,"pools":[],"osds":[{"osd":0,"uuid":"b25ab3bb-3587-45d0-991a-6d41bb529b37","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":1,"uuid":"00d160e7-9773-41d8-8d17-21e70620a0fd","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":2,"uuid":"addfb2a7-2bde-4cc3-98cf-17fd2bdfa3a9","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":3,"uuid":"6c5770b0-c93c-4101-8c53-bd761b8abcde","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":4,"uuid":"f001b3cb-2081-428b-bac3-b3df3c19b978","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":5,"uuid":"3cc263e9-e9bf-45e0-a3b7-83a76469eb70","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":6,"uuid":"82c83316-809f-4901-a5ed-34b6a075b393","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":7,"uuid":"aafe0716-1102-4eae-aac6-0f81bb154e7d","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":4,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":5,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":6,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":7,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[],"pg_upmap_primaries":[],"pg_temp":[],"primary_temp":[],"blocklist":{},"range_blocklist":{},"erasure_code_profiles":{"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","technique":"reed_sol_van"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2024-06-03T08:35:32.789 INFO:teuthology.orchestra.run.smithi088.stderr:2024-06-03T08:35:32.786+0000 7fdd90ff9640 10 librados: watch_flush enter 2024-06-03T08:35:32.789 INFO:teuthology.orchestra.run.smithi088.stderr:2024-06-03T08:35:32.786+0000 7fdd90ff9640 10 librados: watch_flush exit 2024-06-03T08:35:32.790 INFO:teuthology.orchestra.run.smithi088.stderr:2024-06-03T08:35:32.786+0000 7fdd90ff9640 20 client.4169.objecter shutdown clearing up homeless session... 2024-06-03T08:35:32.790 INFO:teuthology.orchestra.run.smithi088.stderr:2024-06-03T08:35:32.786+0000 7fdd90ff9640 10 client.4169.objecter successfully canceled tick 2024-06-03T08:35:32.790 INFO:teuthology.orchestra.run.smithi088.stderr:2024-06-03T08:35:32.786+0000 7fdd90ff9640 10 monclient: shutdown 2024-06-03T08:35:32.790 INFO:teuthology.orchestra.run.smithi088.stderr:2024-06-03T08:35:32.786+0000 7fdd90ff9640 20 monclient: shutdown discarding 0 pending message(s) 2024-06-03T08:35:32.790 INFO:teuthology.orchestra.run.smithi088.stderr:2024-06-03T08:35:32.786+0000 7fdd90ff9640 1 -- 172.21.15.88:0/1890752434 >> v1:172.21.15.88:6790/0 conn(0x7fdda413eec0 legacy=0x7fdda41da140 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-06-03T08:35:32.790 INFO:teuthology.orchestra.run.smithi088.stderr:2024-06-03T08:35:32.786+0000 7fdd90ff9640 1 -- 172.21.15.88:0/1890752434 shutdown_connections 2024-06-03T08:35:32.791 INFO:teuthology.orchestra.run.smithi088.stderr:2024-06-03T08:35:32.786+0000 7fdd90ff9640 1 -- 172.21.15.88:0/1890752434 >> 172.21.15.88:0/1890752434 conn(0x7fdda40b70d0 msgr2=0x7fdda40b3b80 unknown :-1 s=STATE_NONE l=0).mark_down 2024-06-03T08:35:32.791 INFO:teuthology.orchestra.run.smithi088.stderr:2024-06-03T08:35:32.786+0000 7fdd90ff9640 1 -- 172.21.15.88:0/1890752434 shutdown_connections 2024-06-03T08:35:32.791 INFO:teuthology.orchestra.run.smithi088.stderr:2024-06-03T08:35:32.786+0000 7fdd90ff9640 1 -- 172.21.15.88:0/1890752434 wait complete. 2024-06-03T08:35:32.791 INFO:teuthology.orchestra.run.smithi088.stderr:2024-06-03T08:35:32.786+0000 7fdd90ff9640 1 librados: shutdown 2024-06-03T08:35:32.801 INFO:tasks.ceph.ceph_manager.ceph:[] 2024-06-03T08:35:32.801 INFO:tasks.ceph:Waiting for OSDs to come up 2024-06-03T08:35:33.608 DEBUG:teuthology.orchestra.run.smithi088:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph --cluster ceph osd dump --format=json 2024-06-03T08:35:33.751 INFO:teuthology.misc.health.smithi088.stderr:2024-06-03T08:35:33.750+0000 7fefeb576640 10 monclient: get_monmap_and_config 2024-06-03T08:35:33.751 INFO:teuthology.misc.health.smithi088.stderr:2024-06-03T08:35:33.750+0000 7fefeb576640 10 monclient: build_initial_monmap 2024-06-03T08:35:33.751 INFO:teuthology.misc.health.smithi088.stderr:2024-06-03T08:35:33.750+0000 7fefeb576640 10 monclient: monmap: 2024-06-03T08:35:33.751 INFO:teuthology.misc.health.smithi088.stderr:epoch 0 2024-06-03T08:35:33.751 INFO:teuthology.misc.health.smithi088.stderr:fsid ec3c3fdd-3ac6-4d7d-9412-683130ed8b04 2024-06-03T08:35:33.751 INFO:teuthology.misc.health.smithi088.stderr:last_changed 2024-06-03T08:35:33.751465+0000 2024-06-03T08:35:33.751 INFO:teuthology.misc.health.smithi088.stderr:created 2024-06-03T08:35:33.751465+0000 2024-06-03T08:35:33.751 INFO:teuthology.misc.health.smithi088.stderr:min_mon_release 0 (unknown) 2024-06-03T08:35:33.751 INFO:teuthology.misc.health.smithi088.stderr:election_strategy: 3 2024-06-03T08:35:33.752 INFO:teuthology.misc.health.smithi088.stderr:0: v1:172.21.15.88:6789/0 mon.noname-a 2024-06-03T08:35:33.752 INFO:teuthology.misc.health.smithi088.stderr:1: v1:172.21.15.192:6789/0 mon.noname-c 2024-06-03T08:35:33.752 INFO:teuthology.misc.health.smithi088.stderr:2: v1:172.21.15.88:6790/0 mon.noname-b 2024-06-03T08:35:33.752 INFO:teuthology.misc.health.smithi088.stderr: 2024-06-03T08:35:33.753 INFO:teuthology.misc.health.smithi088.stderr:2024-06-03T08:35:33.751+0000 7fefeb576640 1 Processor -- start 2024-06-03T08:35:33.753 INFO:teuthology.misc.health.smithi088.stderr:2024-06-03T08:35:33.751+0000 7fefeb576640 1 -- start start 2024-06-03T08:35:33.753 INFO:teuthology.misc.health.smithi088.stderr:2024-06-03T08:35:33.751+0000 7fefeb576640 10 monclient: init 2024-06-03T08:35:33.753 INFO:teuthology.misc.health.smithi088.stderr:2024-06-03T08:35:33.752+0000 7fefeb576640 10 monclient: _reopen_session rank -1 2024-06-03T08:35:33.753 INFO:teuthology.misc.health.smithi088.stderr:2024-06-03T08:35:33.752+0000 7fefeb576640 10 monclient: _add_conns ranks=[0,1,2] 2024-06-03T08:35:33.754 INFO:teuthology.misc.health.smithi088.stderr:2024-06-03T08:35:33.752+0000 7fefeb576640 10 monclient(hunting): picked mon.noname-a con 0x7fefe40b1070 addr v1:172.21.15.88:6789/0 2024-06-03T08:35:33.754 INFO:teuthology.misc.health.smithi088.stderr:2024-06-03T08:35:33.752+0000 7fefeb576640 10 monclient(hunting): picked mon.noname-c con 0x7fefe4144b40 addr v1:172.21.15.192:6789/0 2024-06-03T08:35:33.754 INFO:teuthology.misc.health.smithi088.stderr:2024-06-03T08:35:33.752+0000 7fefeb576640 10 monclient(hunting): picked mon.noname-b con 0x7fefe40af690 addr v1:172.21.15.88:6790/0 2024-06-03T08:35:33.754 INFO:teuthology.misc.health.smithi088.stderr:2024-06-03T08:35:33.752+0000 7fefeb576640 1 -- --> v1:172.21.15.88:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fefe40b1820 con 0x7fefe40b1070 2024-06-03T08:35:33.754 INFO:teuthology.misc.health.smithi088.stderr:2024-06-03T08:35:33.752+0000 7fefeb576640 1 -- --> v1:172.21.15.192:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fefe40b7d00 con 0x7fefe4144b40 2024-06-03T08:35:33.754 INFO:teuthology.misc.health.smithi088.stderr:2024-06-03T08:35:33.752+0000 7fefeb576640 1 -- --> v1:172.21.15.88:6790/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fefe40ae2d0 con 0x7fefe40af690 2024-06-03T08:35:33.754 INFO:teuthology.misc.health.smithi088.stderr:2024-06-03T08:35:33.753+0000 7fefeb576640 10 monclient(hunting): _renew_subs 2024-06-03T08:35:33.754 INFO:teuthology.misc.health.smithi088.stderr:2024-06-03T08:35:33.753+0000 7fefeb576640 10 monclient(hunting): authenticate will time out at 1189.562987s 2024-06-03T08:35:33.754 INFO:teuthology.misc.health.smithi088.stderr:2024-06-03T08:35:33.754+0000 7fefe9312640 1 --1- >> v1:172.21.15.88:6789/0 conn(0x7fefe40b1070 0x7fefe41451c0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.88:6789/0 says I am v1:172.21.15.88:46390/0 (socket says 172.21.15.88:46390) 2024-06-03T08:35:33.755 INFO:teuthology.misc.health.smithi088.stderr:2024-06-03T08:35:33.754+0000 7fefe9312640 1 -- 172.21.15.88:0/1064936869 learned_addr learned my addr 172.21.15.88:0/1064936869 (peer_addr_for_me v1:172.21.15.88:0/0) 2024-06-03T08:35:33.758 INFO:teuthology.misc.health.smithi088.stderr:2024-06-03T08:35:33.757+0000 7fefdbfff640 1 -- 172.21.15.88:0/1064936869 <== mon.0 v1:172.21.15.88:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 4183113864 0 0) 0x7fefe40b1820 con 0x7fefe40b1070 2024-06-03T08:35:33.758 INFO:teuthology.misc.health.smithi088.stderr:2024-06-03T08:35:33.757+0000 7fefdbfff640 10 monclient(hunting): _negotiate 2024-06-03T08:35:33.758 INFO:teuthology.misc.health.smithi088.stderr:2024-06-03T08:35:33.757+0000 7fefdbfff640 10 monclient(hunting): _init_auth method 2 2024-06-03T08:35:33.758 INFO:teuthology.misc.health.smithi088.stderr:2024-06-03T08:35:33.757+0000 7fefdbfff640 10 monclient(hunting): _init_auth creating new auth 2024-06-03T08:35:33.758 INFO:teuthology.misc.health.smithi088.stderr:2024-06-03T08:35:33.757+0000 7fefdbfff640 10 monclient(hunting): my global_id is 4176 2024-06-03T08:35:33.759 INFO:teuthology.misc.health.smithi088.stderr:2024-06-03T08:35:33.757+0000 7fefdbfff640 1 -- 172.21.15.88:0/1064936869 --> v1:172.21.15.88:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7fefd0003520 con 0x7fefe40b1070 2024-06-03T08:35:33.763 INFO:teuthology.misc.health.smithi088.stderr:2024-06-03T08:35:33.762+0000 7fefdbfff640 1 -- 172.21.15.88:0/1064936869 <== mon.0 v1:172.21.15.88:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 214+0+0 (unknown 2503227473 0 0) 0x7fefd0003520 con 0x7fefe40b1070 2024-06-03T08:35:33.763 INFO:teuthology.misc.health.smithi088.stderr:2024-06-03T08:35:33.763+0000 7fefdbfff640 1 -- 172.21.15.88:0/1064936869 >> v1:172.21.15.88:6790/0 conn(0x7fefe40af690 legacy=0x7fefe414cee0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-06-03T08:35:33.763 INFO:teuthology.misc.health.smithi088.stderr:2024-06-03T08:35:33.763+0000 7fefdbfff640 1 -- 172.21.15.88:0/1064936869 >> v1:172.21.15.192:6789/0 conn(0x7fefe4144b40 legacy=0x7fefe40aef80 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-06-03T08:35:33.764 INFO:teuthology.misc.health.smithi088.stderr:2024-06-03T08:35:33.763+0000 7fefdbfff640 10 monclient: _finish_hunting 0 2024-06-03T08:35:33.764 INFO:teuthology.misc.health.smithi088.stderr:2024-06-03T08:35:33.763+0000 7fefdbfff640 1 monclient: found mon.noname-a 2024-06-03T08:35:33.764 INFO:teuthology.misc.health.smithi088.stderr:2024-06-03T08:35:33.763+0000 7fefdbfff640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-06-03T08:35:33.764 INFO:teuthology.misc.health.smithi088.stderr:2024-06-03T08:35:33.763+0000 7fefdbfff640 10 monclient: _send_mon_message to mon.noname-a at v1:172.21.15.88:6789/0 2024-06-03T08:35:33.764 INFO:teuthology.misc.health.smithi088.stderr:2024-06-03T08:35:33.763+0000 7fefdbfff640 1 -- 172.21.15.88:0/1064936869 --> v1:172.21.15.88:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fefe40b1450 con 0x7fefe40b1070 2024-06-03T08:35:33.764 INFO:teuthology.misc.health.smithi088.stderr:2024-06-03T08:35:33.763+0000 7fefdbfff640 10 monclient: _finish_auth 0 2024-06-03T08:35:33.764 INFO:teuthology.misc.health.smithi088.stderr:2024-06-03T08:35:33.763+0000 7fefdbfff640 10 monclient: _check_auth_tickets 2024-06-03T08:35:33.764 INFO:teuthology.misc.health.smithi088.stderr:2024-06-03T08:35:33.763+0000 7fefdbfff640 20 monclient: _check_auth_rotating not needed by client.admin 2024-06-03T08:35:33.765 INFO:teuthology.misc.health.smithi088.stderr:2024-06-03T08:35:33.763+0000 7fefdbfff640 1 -- 172.21.15.88:0/1064936869 <== mon.0 v1:172.21.15.88:6789/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 2234374065 0 0) 0x7fefe0001610 con 0x7fefe40b1070 2024-06-03T08:35:33.765 INFO:teuthology.misc.health.smithi088.stderr:2024-06-03T08:35:33.763+0000 7fefdbfff640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-06-03T08:35:33.765 INFO:teuthology.misc.health.smithi088.stderr:2024-06-03T08:35:33.763+0000 7fefdbfff640 10 monclient: got monmap 1 from mon.noname-a (according to old e1) 2024-06-03T08:35:33.765 INFO:teuthology.misc.health.smithi088.stderr:2024-06-03T08:35:33.763+0000 7fefdbfff640 10 monclient: dump: 2024-06-03T08:35:33.765 INFO:teuthology.misc.health.smithi088.stderr:epoch 1 2024-06-03T08:35:33.765 INFO:teuthology.misc.health.smithi088.stderr:fsid ec3c3fdd-3ac6-4d7d-9412-683130ed8b04 2024-06-03T08:35:33.765 INFO:teuthology.misc.health.smithi088.stderr:last_changed 2024-06-03T08:34:31.885125+0000 2024-06-03T08:35:33.765 INFO:teuthology.misc.health.smithi088.stderr:created 2024-06-03T08:34:31.885125+0000 2024-06-03T08:35:33.765 INFO:teuthology.misc.health.smithi088.stderr:min_mon_release 18 (reef) 2024-06-03T08:35:33.765 INFO:teuthology.misc.health.smithi088.stderr:election_strategy: 3 2024-06-03T08:35:33.765 INFO:teuthology.misc.health.smithi088.stderr:0: v1:172.21.15.88:6789/0 mon.a 2024-06-03T08:35:33.765 INFO:teuthology.misc.health.smithi088.stderr:1: v1:172.21.15.192:6789/0 mon.b 2024-06-03T08:35:33.765 INFO:teuthology.misc.health.smithi088.stderr:2: v1:172.21.15.88:6790/0 mon.c 2024-06-03T08:35:33.765 INFO:teuthology.misc.health.smithi088.stderr: 2024-06-03T08:35:33.766 INFO:teuthology.misc.health.smithi088.stderr:2024-06-03T08:35:33.763+0000 7fefeb576640 5 monclient: authenticate success, global_id 4176 2024-06-03T08:35:33.766 INFO:teuthology.misc.health.smithi088.stderr:2024-06-03T08:35:33.763+0000 7fefeb576640 20 monclient: get_monmap_and_config waiting for monmap|config 2024-06-03T08:35:33.767 INFO:teuthology.misc.health.smithi088.stderr:2024-06-03T08:35:33.765+0000 7fefdbfff640 1 -- 172.21.15.88:0/1064936869 <== mon.0 v1:172.21.15.88:6789/0 4 ==== config(0 keys) v1 ==== 4+0+0 (unknown 0 0 0) 0x7fefe0001830 con 0x7fefe40b1070 2024-06-03T08:35:33.767 INFO:teuthology.misc.health.smithi088.stderr:2024-06-03T08:35:33.765+0000 7fefdbfff640 10 monclient: handle_config config(0 keys) v1 2024-06-03T08:35:33.767 INFO:teuthology.misc.health.smithi088.stderr:2024-06-03T08:35:33.765+0000 7fefdbfff640 1 -- 172.21.15.88:0/1064936869 <== mon.0 v1:172.21.15.88:6789/0 5 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 2234374065 0 0) 0x7fefe0001c40 con 0x7fefe40b1070 2024-06-03T08:35:33.767 INFO:teuthology.misc.health.smithi088.stderr:2024-06-03T08:35:33.765+0000 7fefdbfff640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-06-03T08:35:33.767 INFO:teuthology.misc.health.smithi088.stderr:2024-06-03T08:35:33.765+0000 7fefdbfff640 10 monclient: got monmap 1 from mon.a (according to old e1) 2024-06-03T08:35:33.767 INFO:teuthology.misc.health.smithi088.stderr:2024-06-03T08:35:33.765+0000 7fefdbfff640 10 monclient: dump: 2024-06-03T08:35:33.768 INFO:teuthology.misc.health.smithi088.stderr:epoch 1 2024-06-03T08:35:33.768 INFO:teuthology.misc.health.smithi088.stderr:fsid ec3c3fdd-3ac6-4d7d-9412-683130ed8b04 2024-06-03T08:35:33.768 INFO:teuthology.misc.health.smithi088.stderr:last_changed 2024-06-03T08:34:31.885125+0000 2024-06-03T08:35:33.768 INFO:teuthology.misc.health.smithi088.stderr:created 2024-06-03T08:34:31.885125+0000 2024-06-03T08:35:33.768 INFO:teuthology.misc.health.smithi088.stderr:min_mon_release 18 (reef) 2024-06-03T08:35:33.768 INFO:teuthology.misc.health.smithi088.stderr:election_strategy: 3 2024-06-03T08:35:33.768 INFO:teuthology.misc.health.smithi088.stderr:0: v1:172.21.15.88:6789/0 mon.a 2024-06-03T08:35:33.768 INFO:teuthology.misc.health.smithi088.stderr:1: v1:172.21.15.192:6789/0 mon.b 2024-06-03T08:35:33.768 INFO:teuthology.misc.health.smithi088.stderr:2: v1:172.21.15.88:6790/0 mon.c 2024-06-03T08:35:33.768 INFO:teuthology.misc.health.smithi088.stderr: 2024-06-03T08:35:33.769 INFO:teuthology.misc.health.smithi088.stderr:2024-06-03T08:35:33.765+0000 7fefeb576640 10 monclient: get_monmap_and_config success 2024-06-03T08:35:33.769 INFO:teuthology.misc.health.smithi088.stderr:2024-06-03T08:35:33.765+0000 7fefeb576640 4 set_mon_vals no callback set 2024-06-03T08:35:33.769 INFO:teuthology.misc.health.smithi088.stderr:2024-06-03T08:35:33.766+0000 7fefeb576640 10 monclient: shutdown 2024-06-03T08:35:33.769 INFO:teuthology.misc.health.smithi088.stderr:2024-06-03T08:35:33.766+0000 7fefeb576640 20 monclient: shutdown discarding 0 pending message(s) 2024-06-03T08:35:33.769 INFO:teuthology.misc.health.smithi088.stderr:2024-06-03T08:35:33.766+0000 7fefeb576640 1 -- 172.21.15.88:0/1064936869 >> v1:172.21.15.88:6789/0 conn(0x7fefe40b1070 legacy=0x7fefe41451c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-06-03T08:35:33.769 INFO:teuthology.misc.health.smithi088.stderr:2024-06-03T08:35:33.766+0000 7fefeb576640 1 -- 172.21.15.88:0/1064936869 shutdown_connections 2024-06-03T08:35:33.769 INFO:teuthology.misc.health.smithi088.stderr:2024-06-03T08:35:33.766+0000 7fefeb576640 1 -- 172.21.15.88:0/1064936869 >> 172.21.15.88:0/1064936869 conn(0x7fefe40b71e0 msgr2=0x7fefe40b75f0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-06-03T08:35:33.769 INFO:teuthology.misc.health.smithi088.stderr:2024-06-03T08:35:33.767+0000 7fefeb576640 1 -- 172.21.15.88:0/1064936869 shutdown_connections 2024-06-03T08:35:33.769 INFO:teuthology.misc.health.smithi088.stderr:2024-06-03T08:35:33.767+0000 7fefeb576640 1 -- 172.21.15.88:0/1064936869 wait complete. 2024-06-03T08:35:33.772 INFO:teuthology.misc.health.smithi088.stderr:2024-06-03T08:35:33.768+0000 7fefeb576640 10 monclient: build_initial_monmap 2024-06-03T08:35:33.772 INFO:teuthology.misc.health.smithi088.stderr:2024-06-03T08:35:33.768+0000 7fefeb576640 10 monclient: monmap: 2024-06-03T08:35:33.772 INFO:teuthology.misc.health.smithi088.stderr:epoch 0 2024-06-03T08:35:33.772 INFO:teuthology.misc.health.smithi088.stderr:fsid 00000000-0000-0000-0000-000000000000 2024-06-03T08:35:33.772 INFO:teuthology.misc.health.smithi088.stderr:last_changed 0.000000 2024-06-03T08:35:33.772 INFO:teuthology.misc.health.smithi088.stderr:created 0.000000 2024-06-03T08:35:33.772 INFO:teuthology.misc.health.smithi088.stderr:min_mon_release 0 (unknown) 2024-06-03T08:35:33.772 INFO:teuthology.misc.health.smithi088.stderr:election_strategy: 1 2024-06-03T08:35:33.772 INFO:teuthology.misc.health.smithi088.stderr:0: v1:172.21.15.88:6789/0 mon.noname-a 2024-06-03T08:35:33.773 INFO:teuthology.misc.health.smithi088.stderr:1: v1:172.21.15.192:6789/0 mon.noname-b 2024-06-03T08:35:33.773 INFO:teuthology.misc.health.smithi088.stderr:2: v1:172.21.15.88:6790/0 mon.noname-c 2024-06-03T08:35:33.773 INFO:teuthology.misc.health.smithi088.stderr: 2024-06-03T08:35:33.773 INFO:teuthology.misc.health.smithi088.stderr:2024-06-03T08:35:33.768+0000 7fefeb576640 1 librados: starting msgr at 2024-06-03T08:35:33.773 INFO:teuthology.misc.health.smithi088.stderr:2024-06-03T08:35:33.768+0000 7fefeb576640 1 librados: starting objecter 2024-06-03T08:35:33.773 INFO:teuthology.misc.health.smithi088.stderr:2024-06-03T08:35:33.768+0000 7fefeb576640 1 Processor -- start 2024-06-03T08:35:33.773 INFO:teuthology.misc.health.smithi088.stderr:2024-06-03T08:35:33.769+0000 7fefeb576640 1 -- start start 2024-06-03T08:35:33.773 INFO:teuthology.misc.health.smithi088.stderr:2024-06-03T08:35:33.769+0000 7fefeb576640 1 librados: setting wanted keys 2024-06-03T08:35:33.773 INFO:teuthology.misc.health.smithi088.stderr:2024-06-03T08:35:33.769+0000 7fefeb576640 1 librados: calling monclient init 2024-06-03T08:35:33.773 INFO:teuthology.misc.health.smithi088.stderr:2024-06-03T08:35:33.769+0000 7fefeb576640 10 monclient: init 2024-06-03T08:35:33.773 INFO:teuthology.misc.health.smithi088.stderr:2024-06-03T08:35:33.770+0000 7fefeb576640 10 monclient: _reopen_session rank -1 2024-06-03T08:35:33.773 INFO:teuthology.misc.health.smithi088.stderr:2024-06-03T08:35:33.770+0000 7fefeb576640 10 monclient: _add_conns ranks=[1,0,2] 2024-06-03T08:35:33.773 INFO:teuthology.misc.health.smithi088.stderr:2024-06-03T08:35:33.770+0000 7fefeb576640 10 monclient(hunting): picked mon.noname-b con 0x7fefe40b1070 addr v1:172.21.15.192:6789/0 2024-06-03T08:35:33.773 INFO:teuthology.misc.health.smithi088.stderr:2024-06-03T08:35:33.770+0000 7fefeb576640 10 monclient(hunting): picked mon.noname-a con 0x7fefe4144b40 addr v1:172.21.15.88:6789/0 2024-06-03T08:35:33.774 INFO:teuthology.misc.health.smithi088.stderr:2024-06-03T08:35:33.770+0000 7fefeb576640 10 monclient(hunting): picked mon.noname-c con 0x7fefe4141070 addr v1:172.21.15.88:6790/0 2024-06-03T08:35:33.774 INFO:teuthology.misc.health.smithi088.stderr:2024-06-03T08:35:33.770+0000 7fefeb576640 1 -- --> v1:172.21.15.88:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fefe41df970 con 0x7fefe4144b40 2024-06-03T08:35:33.774 INFO:teuthology.misc.health.smithi088.stderr:2024-06-03T08:35:33.770+0000 7fefeb576640 1 -- --> v1:172.21.15.192:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fefe4140b90 con 0x7fefe40b1070 2024-06-03T08:35:33.774 INFO:teuthology.misc.health.smithi088.stderr:2024-06-03T08:35:33.770+0000 7fefeb576640 1 -- --> v1:172.21.15.88:6790/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fefe412ee20 con 0x7fefe4141070 2024-06-03T08:35:33.774 INFO:teuthology.misc.health.smithi088.stderr:2024-06-03T08:35:33.770+0000 7fefeb576640 10 monclient(hunting): _renew_subs 2024-06-03T08:35:33.774 INFO:teuthology.misc.health.smithi088.stderr:2024-06-03T08:35:33.770+0000 7fefeb576640 10 monclient(hunting): authenticate will time out at 1189.580801s 2024-06-03T08:35:33.774 INFO:teuthology.misc.health.smithi088.stderr:2024-06-03T08:35:33.771+0000 7fefe8b11640 1 --1- >> v1:172.21.15.88:6789/0 conn(0x7fefe4144b40 0x7fefe41deb50 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=1).handle_server_banner_and_identify peer v1:172.21.15.88:6789/0 says I am v1:172.21.15.88:46400/0 (socket says 172.21.15.88:46400) 2024-06-03T08:35:33.774 INFO:teuthology.misc.health.smithi088.stderr:2024-06-03T08:35:33.771+0000 7fefe8b11640 1 -- 172.21.15.88:0/944120537 learned_addr learned my addr 172.21.15.88:0/944120537 (peer_addr_for_me v1:172.21.15.88:0/0) 2024-06-03T08:35:33.774 INFO:teuthology.misc.health.smithi088.stderr:2024-06-03T08:35:33.771+0000 7fefd9ffb640 10 client.?.objecter ms_handle_connect 0x7fefe4144b40 2024-06-03T08:35:33.774 INFO:teuthology.misc.health.smithi088.stderr:2024-06-03T08:35:33.771+0000 7fefd9ffb640 10 client.?.objecter resend_mon_ops 2024-06-03T08:35:33.775 INFO:teuthology.misc.health.smithi088.stderr:2024-06-03T08:35:33.774+0000 7fefd9ffb640 1 -- 172.21.15.88:0/944120537 <== mon.0 v1:172.21.15.88:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 2122279652 0 0) 0x7fefe41df970 con 0x7fefe4144b40 2024-06-03T08:35:33.775 INFO:teuthology.misc.health.smithi088.stderr:2024-06-03T08:35:33.774+0000 7fefd9ffb640 10 monclient(hunting): _negotiate 2024-06-03T08:35:33.775 INFO:teuthology.misc.health.smithi088.stderr:2024-06-03T08:35:33.774+0000 7fefd9ffb640 10 monclient(hunting): _init_auth method 2 2024-06-03T08:35:33.775 INFO:teuthology.misc.health.smithi088.stderr:2024-06-03T08:35:33.774+0000 7fefd9ffb640 10 monclient(hunting): _init_auth creating new auth 2024-06-03T08:35:33.775 INFO:teuthology.misc.health.smithi088.stderr:2024-06-03T08:35:33.774+0000 7fefd9ffb640 10 monclient(hunting): my global_id is 4179 2024-06-03T08:35:33.775 INFO:teuthology.misc.health.smithi088.stderr:2024-06-03T08:35:33.774+0000 7fefd9ffb640 1 -- 172.21.15.88:0/944120537 --> v1:172.21.15.88:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7fefc8004eb0 con 0x7fefe4144b40 2024-06-03T08:35:33.778 INFO:teuthology.misc.health.smithi088.stderr:2024-06-03T08:35:33.778+0000 7fefd9ffb640 10 client.?.objecter ms_handle_connect 0x7fefe4141070 2024-06-03T08:35:33.778 INFO:teuthology.misc.health.smithi088.stderr:2024-06-03T08:35:33.778+0000 7fefd9ffb640 10 client.?.objecter resend_mon_ops 2024-06-03T08:35:33.786 INFO:teuthology.misc.health.smithi088.stderr:2024-06-03T08:35:33.785+0000 7fefd9ffb640 1 -- 172.21.15.88:0/944120537 <== mon.0 v1:172.21.15.88:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 764+0+0 (unknown 3313445064 0 0) 0x7fefc8004eb0 con 0x7fefe4144b40 2024-06-03T08:35:33.786 INFO:teuthology.misc.health.smithi088.stderr:2024-06-03T08:35:33.785+0000 7fefd9ffb640 1 -- 172.21.15.88:0/944120537 --> v1:172.21.15.88:6789/0 -- auth(proto 2 165 bytes epoch 0) v1 -- 0x7fefe41df970 con 0x7fefe4144b40 2024-06-03T08:35:33.786 INFO:teuthology.misc.health.smithi088.stderr:2024-06-03T08:35:33.785+0000 7fefd9ffb640 1 -- 172.21.15.88:0/944120537 <== mon.0 v1:172.21.15.88:6789/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 2234374065 0 0) 0x7fefd4001770 con 0x7fefe4144b40 2024-06-03T08:35:33.786 INFO:teuthology.misc.health.smithi088.stderr:2024-06-03T08:35:33.785+0000 7fefd9ffb640 10 monclient(hunting): handle_monmap mon_map magic: 0 v1 2024-06-03T08:35:33.786 INFO:teuthology.misc.health.smithi088.stderr:2024-06-03T08:35:33.785+0000 7fefd9ffb640 10 monclient(hunting): got monmap 1 from mon.noname-a (according to old e1) 2024-06-03T08:35:33.786 INFO:teuthology.misc.health.smithi088.stderr:2024-06-03T08:35:33.785+0000 7fefd9ffb640 10 monclient(hunting): dump: 2024-06-03T08:35:33.786 INFO:teuthology.misc.health.smithi088.stderr:epoch 1 2024-06-03T08:35:33.786 INFO:teuthology.misc.health.smithi088.stderr:fsid ec3c3fdd-3ac6-4d7d-9412-683130ed8b04 2024-06-03T08:35:33.786 INFO:teuthology.misc.health.smithi088.stderr:last_changed 2024-06-03T08:34:31.885125+0000 2024-06-03T08:35:33.787 INFO:teuthology.misc.health.smithi088.stderr:created 2024-06-03T08:34:31.885125+0000 2024-06-03T08:35:33.787 INFO:teuthology.misc.health.smithi088.stderr:min_mon_release 18 (reef) 2024-06-03T08:35:33.787 INFO:teuthology.misc.health.smithi088.stderr:election_strategy: 3 2024-06-03T08:35:33.787 INFO:teuthology.misc.health.smithi088.stderr:0: v1:172.21.15.88:6789/0 mon.a 2024-06-03T08:35:33.787 INFO:teuthology.misc.health.smithi088.stderr:1: v1:172.21.15.192:6789/0 mon.b 2024-06-03T08:35:33.787 INFO:teuthology.misc.health.smithi088.stderr:2: v1:172.21.15.88:6790/0 mon.c 2024-06-03T08:35:33.787 INFO:teuthology.misc.health.smithi088.stderr: 2024-06-03T08:35:33.787 INFO:teuthology.misc.health.smithi088.stderr:2024-06-03T08:35:33.785+0000 7fefd9ffb640 10 monclient(hunting): _finish_auth 0 2024-06-03T08:35:33.793 INFO:teuthology.misc.health.smithi088.stderr:2024-06-03T08:35:33.791+0000 7fefd9ffb640 1 -- 172.21.15.88:0/944120537 <== mon.0 v1:172.21.15.88:6789/0 4 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 580+0+0 (unknown 2928044622 0 0) 0x7fefe41df970 con 0x7fefe4144b40 2024-06-03T08:35:33.793 INFO:teuthology.misc.health.smithi088.stderr:2024-06-03T08:35:33.791+0000 7fefd9ffb640 1 -- 172.21.15.88:0/944120537 >> v1:172.21.15.88:6790/0 conn(0x7fefe4141070 legacy=0x7fefe41df260 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-06-03T08:35:33.793 INFO:teuthology.misc.health.smithi088.stderr:2024-06-03T08:35:33.791+0000 7fefd9ffb640 1 -- 172.21.15.88:0/944120537 >> v1:172.21.15.192:6789/0 conn(0x7fefe40b1070 legacy=0x7fefe412c4c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-06-03T08:35:33.793 INFO:teuthology.misc.health.smithi088.stderr:2024-06-03T08:35:33.791+0000 7fefd9ffb640 10 monclient: _finish_hunting 0 2024-06-03T08:35:33.794 INFO:teuthology.misc.health.smithi088.stderr:2024-06-03T08:35:33.791+0000 7fefd9ffb640 1 monclient: found mon.a 2024-06-03T08:35:33.794 INFO:teuthology.misc.health.smithi088.stderr:2024-06-03T08:35:33.791+0000 7fefd9ffb640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-06-03T08:35:33.794 INFO:teuthology.misc.health.smithi088.stderr:2024-06-03T08:35:33.791+0000 7fefd9ffb640 10 monclient: _send_mon_message to mon.a at v1:172.21.15.88:6789/0 2024-06-03T08:35:33.794 INFO:teuthology.misc.health.smithi088.stderr:2024-06-03T08:35:33.791+0000 7fefd9ffb640 1 -- 172.21.15.88:0/944120537 --> v1:172.21.15.88:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fefe412cbd0 con 0x7fefe4144b40 2024-06-03T08:35:33.794 INFO:teuthology.misc.health.smithi088.stderr:2024-06-03T08:35:33.791+0000 7fefd9ffb640 10 monclient: _finish_auth 0 2024-06-03T08:35:33.794 INFO:teuthology.misc.health.smithi088.stderr:2024-06-03T08:35:33.791+0000 7fefd9ffb640 10 monclient: _check_auth_tickets 2024-06-03T08:35:33.794 INFO:teuthology.misc.health.smithi088.stderr:2024-06-03T08:35:33.791+0000 7fefd9ffb640 20 monclient: _check_auth_rotating not needed by client.admin 2024-06-03T08:35:33.794 INFO:teuthology.misc.health.smithi088.stderr:2024-06-03T08:35:33.791+0000 7fefeb576640 5 monclient: authenticate success, global_id 4179 2024-06-03T08:35:33.794 INFO:teuthology.misc.health.smithi088.stderr:2024-06-03T08:35:33.791+0000 7fefeb576640 10 monclient: _renew_subs 2024-06-03T08:35:33.794 INFO:teuthology.misc.health.smithi088.stderr:2024-06-03T08:35:33.791+0000 7fefeb576640 10 monclient: _send_mon_message to mon.a at v1:172.21.15.88:6789/0 2024-06-03T08:35:33.794 INFO:teuthology.misc.health.smithi088.stderr:2024-06-03T08:35:33.791+0000 7fefeb576640 1 -- 172.21.15.88:0/944120537 --> v1:172.21.15.88:6789/0 -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fefe4130e10 con 0x7fefe4144b40 2024-06-03T08:35:33.794 INFO:teuthology.misc.health.smithi088.stderr:2024-06-03T08:35:33.791+0000 7fefeb576640 10 client.4179.objecter _maybe_request_map subscribing (onetime) to next osd map 2024-06-03T08:35:33.795 INFO:teuthology.misc.health.smithi088.stderr:2024-06-03T08:35:33.791+0000 7fefeb576640 10 monclient: _renew_subs 2024-06-03T08:35:33.795 INFO:teuthology.misc.health.smithi088.stderr:2024-06-03T08:35:33.791+0000 7fefeb576640 10 monclient: _send_mon_message to mon.a at v1:172.21.15.88:6789/0 2024-06-03T08:35:33.795 INFO:teuthology.misc.health.smithi088.stderr:2024-06-03T08:35:33.791+0000 7fefeb576640 1 -- 172.21.15.88:0/944120537 --> v1:172.21.15.88:6789/0 -- mon_subscribe({osdmap=0}) v3 -- 0x7fefe41308b0 con 0x7fefe4144b40 2024-06-03T08:35:33.795 INFO:teuthology.misc.health.smithi088.stderr:2024-06-03T08:35:33.791+0000 7fefeb576640 1 librados: init done 2024-06-03T08:35:33.795 INFO:teuthology.misc.health.smithi088.stderr:2024-06-03T08:35:33.792+0000 7fefeb576640 10 start_mon_command cmd=[{"prefix": "get_command_descriptions"}] 2024-06-03T08:35:33.795 INFO:teuthology.misc.health.smithi088.stderr:2024-06-03T08:35:33.792+0000 7fefeb576640 10 monclient: _send_command 1 [{"prefix": "get_command_descriptions"}] 2024-06-03T08:35:33.795 INFO:teuthology.misc.health.smithi088.stderr:2024-06-03T08:35:33.792+0000 7fefeb576640 10 monclient: _send_mon_message to mon.a at v1:172.21.15.88:6789/0 2024-06-03T08:35:33.795 INFO:teuthology.misc.health.smithi088.stderr:2024-06-03T08:35:33.792+0000 7fefeb576640 1 -- 172.21.15.88:0/944120537 --> v1:172.21.15.88:6789/0 -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fefe406dd60 con 0x7fefe4144b40 2024-06-03T08:35:33.795 INFO:teuthology.misc.health.smithi088.stderr:2024-06-03T08:35:33.795+0000 7fefd9ffb640 1 -- 172.21.15.88:0/944120537 <== mon.0 v1:172.21.15.88:6789/0 5 ==== config(0 keys) v1 ==== 4+0+0 (unknown 0 0 0) 0x7fefd4001d60 con 0x7fefe4144b40 2024-06-03T08:35:33.796 INFO:teuthology.misc.health.smithi088.stderr:2024-06-03T08:35:33.795+0000 7fefd9ffb640 10 monclient: handle_config config(0 keys) v1 2024-06-03T08:35:33.796 INFO:teuthology.misc.health.smithi088.stderr:2024-06-03T08:35:33.795+0000 7fefdb7fe640 4 set_mon_vals no callback set 2024-06-03T08:35:33.797 INFO:teuthology.misc.health.smithi088.stderr:2024-06-03T08:35:33.797+0000 7fefd9ffb640 1 -- 172.21.15.88:0/944120537 <== mon.0 v1:172.21.15.88:6789/0 6 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 2234374065 0 0) 0x7fefd40049a0 con 0x7fefe4144b40 2024-06-03T08:35:33.797 INFO:teuthology.misc.health.smithi088.stderr:2024-06-03T08:35:33.797+0000 7fefd9ffb640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-06-03T08:35:33.798 INFO:teuthology.misc.health.smithi088.stderr:2024-06-03T08:35:33.797+0000 7fefd9ffb640 10 monclient: got monmap 1 from mon.a (according to old e1) 2024-06-03T08:35:33.798 INFO:teuthology.misc.health.smithi088.stderr:2024-06-03T08:35:33.797+0000 7fefd9ffb640 10 monclient: dump: 2024-06-03T08:35:33.798 INFO:teuthology.misc.health.smithi088.stderr:epoch 1 2024-06-03T08:35:33.798 INFO:teuthology.misc.health.smithi088.stderr:fsid ec3c3fdd-3ac6-4d7d-9412-683130ed8b04 2024-06-03T08:35:33.798 INFO:teuthology.misc.health.smithi088.stderr:last_changed 2024-06-03T08:34:31.885125+0000 2024-06-03T08:35:33.798 INFO:teuthology.misc.health.smithi088.stderr:created 2024-06-03T08:34:31.885125+0000 2024-06-03T08:35:33.798 INFO:teuthology.misc.health.smithi088.stderr:min_mon_release 18 (reef) 2024-06-03T08:35:33.798 INFO:teuthology.misc.health.smithi088.stderr:election_strategy: 3 2024-06-03T08:35:33.798 INFO:teuthology.misc.health.smithi088.stderr:0: v1:172.21.15.88:6789/0 mon.a 2024-06-03T08:35:33.798 INFO:teuthology.misc.health.smithi088.stderr:1: v1:172.21.15.192:6789/0 mon.b 2024-06-03T08:35:33.798 INFO:teuthology.misc.health.smithi088.stderr:2: v1:172.21.15.88:6790/0 mon.c 2024-06-03T08:35:33.798 INFO:teuthology.misc.health.smithi088.stderr: 2024-06-03T08:35:33.980 INFO:teuthology.misc.health.smithi088.stderr:2024-06-03T08:35:33.980+0000 7fefd9ffb640 1 -- 172.21.15.88:0/944120537 <== mon.0 v1:172.21.15.88:6789/0 7 ==== mgrmap(e 2) v1 ==== 44528+0+0 (unknown 38132633 0 0) 0x7fefd400f900 con 0x7fefe4144b40 2024-06-03T08:35:33.994 INFO:teuthology.misc.health.smithi088.stderr:2024-06-03T08:35:33.994+0000 7fefd9ffb640 1 -- 172.21.15.88:0/944120537 <== mon.0 v1:172.21.15.88:6789/0 8 ==== osd_map(10..10 src has 1..10) v4 ==== 1645+0+0 (unknown 309997979 0 0) 0x7fefd4015540 con 0x7fefe4144b40 2024-06-03T08:35:33.995 INFO:teuthology.misc.health.smithi088.stderr:2024-06-03T08:35:33.994+0000 7fefd9ffb640 10 client.4179.objecter ms_dispatch 0x7fefe4000b90 osd_map(10..10 src has 1..10) v4 2024-06-03T08:35:33.995 INFO:teuthology.misc.health.smithi088.stderr:2024-06-03T08:35:33.994+0000 7fefd9ffb640 3 client.4179.objecter handle_osd_map got epochs [10,10] > 0 2024-06-03T08:35:33.996 INFO:teuthology.misc.health.smithi088.stderr:2024-06-03T08:35:33.994+0000 7fefd9ffb640 3 client.4179.objecter handle_osd_map decoding full epoch 10 2024-06-03T08:35:33.996 INFO:teuthology.misc.health.smithi088.stderr:2024-06-03T08:35:33.994+0000 7fefd9ffb640 20 client.4179.objecter dump_active .. 0 homeless 2024-06-03T08:35:33.996 INFO:teuthology.misc.health.smithi088.stderr:2024-06-03T08:35:33.994+0000 7fefd9ffb640 1 -- 172.21.15.88:0/944120537 <== mon.0 v1:172.21.15.88:6789/0 9 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+73121 (unknown 1092875540 0 1417653156) 0x7fefd4014800 con 0x7fefe4144b40 2024-06-03T08:35:33.996 INFO:teuthology.misc.health.smithi088.stderr:2024-06-03T08:35:33.994+0000 7fefd9ffb640 10 monclient: handle_mon_command_ack 1 [{"prefix": "get_command_descriptions"}] 2024-06-03T08:35:33.996 INFO:teuthology.misc.health.smithi088.stderr:2024-06-03T08:35:33.994+0000 7fefd9ffb640 10 monclient: _finish_command 1 = system:0 2024-06-03T08:35:34.118 INFO:teuthology.misc.health.smithi088.stderr:2024-06-03T08:35:34.117+0000 7fefeb576640 10 start_mon_command cmd=[{"prefix": "osd dump", "format": "json"}] 2024-06-03T08:35:34.118 INFO:teuthology.misc.health.smithi088.stderr:2024-06-03T08:35:34.117+0000 7fefeb576640 10 monclient: _send_command 2 [{"prefix": "osd dump", "format": "json"}] 2024-06-03T08:35:34.118 INFO:teuthology.misc.health.smithi088.stderr:2024-06-03T08:35:34.117+0000 7fefeb576640 10 monclient: _send_mon_message to mon.a at v1:172.21.15.88:6789/0 2024-06-03T08:35:34.118 INFO:teuthology.misc.health.smithi088.stderr:2024-06-03T08:35:34.117+0000 7fefeb576640 1 -- 172.21.15.88:0/944120537 --> v1:172.21.15.88:6789/0 -- mon_command({"prefix": "osd dump", "format": "json"} v 0) v1 -- 0x7fefe40af130 con 0x7fefe4144b40 2024-06-03T08:35:34.223 INFO:teuthology.misc.health.smithi088.stderr:2024-06-03T08:35:34.222+0000 7fefd9ffb640 1 -- 172.21.15.88:0/944120537 <== mon.0 v1:172.21.15.88:6789/0 10 ==== mgrmap(e 3) v1 ==== 88948+0+0 (unknown 619726803 0 0) 0x7fefd40115c0 con 0x7fefe4144b40 2024-06-03T08:35:34.224 INFO:teuthology.misc.health.smithi088.stderr:2024-06-03T08:35:34.223+0000 7fefe9312640 10 monclient: get_auth_request con 0x7fefc806d780 auth_method 0 2024-06-03T08:35:34.224 INFO:teuthology.misc.health.smithi088.stderr:2024-06-03T08:35:34.224+0000 7fefd9ffb640 10 client.4179.objecter ms_handle_connect 0x7fefc806d780 2024-06-03T08:35:34.342 INFO:tasks.ceph.mgr.x.smithi192.stderr:2024-06-03T08:35:34.342+0000 7f0e0cc9b640 -1 mgr.server handle_report got status from non-daemon mon.b 2024-06-03T08:35:34.401 INFO:tasks.ceph.mgr.x.smithi192.stderr:2024-06-03T08:35:34.401+0000 7f0e0cc9b640 -1 mgr.server handle_report got status from non-daemon mon.c 2024-06-03T08:35:34.444 INFO:teuthology.misc.health.smithi088.stderr:2024-06-03T08:35:34.443+0000 7fefd9ffb640 1 -- 172.21.15.88:0/944120537 <== mon.0 v1:172.21.15.88:6789/0 11 ==== mon_command_ack([{"prefix": "osd dump", "format": "json"}]=0 v10) v1 ==== 74+0+7057 (unknown 4040575075 0 1067785724) 0x7fefd4021360 con 0x7fefe4144b40 2024-06-03T08:35:34.444 INFO:teuthology.misc.health.smithi088.stderr:2024-06-03T08:35:34.443+0000 7fefd9ffb640 10 monclient: handle_mon_command_ack 2 [{"prefix": "osd dump", "format": "json"}] 2024-06-03T08:35:34.444 INFO:teuthology.misc.health.smithi088.stderr:2024-06-03T08:35:34.443+0000 7fefd9ffb640 10 monclient: _finish_command 2 = system:0 2024-06-03T08:35:34.444 INFO:teuthology.misc.health.smithi088.stdout: 2024-06-03T08:35:34.444 INFO:teuthology.misc.health.smithi088.stdout:{"epoch":10,"fsid":"ec3c3fdd-3ac6-4d7d-9412-683130ed8b04","created":"2024-06-03T08:35:26.799638+0000","modified":"2024-06-03T08:35:32.183068+0000","last_up_change":"0.000000","last_in_change":"2024-06-03T08:35:32.183068+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":2,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":0,"max_osd":8,"require_min_compat_client":"luminous","min_compat_client":"jewel","require_osd_release":"reef","allow_crimson":false,"pools":[],"osds":[{"osd":0,"uuid":"b25ab3bb-3587-45d0-991a-6d41bb529b37","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":1,"uuid":"00d160e7-9773-41d8-8d17-21e70620a0fd","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":2,"uuid":"addfb2a7-2bde-4cc3-98cf-17fd2bdfa3a9","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":3,"uuid":"6c5770b0-c93c-4101-8c53-bd761b8abcde","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":4,"uuid":"f001b3cb-2081-428b-bac3-b3df3c19b978","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":5,"uuid":"3cc263e9-e9bf-45e0-a3b7-83a76469eb70","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":6,"uuid":"82c83316-809f-4901-a5ed-34b6a075b393","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":7,"uuid":"aafe0716-1102-4eae-aac6-0f81bb154e7d","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":4,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":5,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":6,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":7,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[],"pg_upmap_primaries":[],"pg_temp":[],"primary_temp":[],"blocklist":{},"range_blocklist":{},"erasure_code_profiles":{"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","technique":"reed_sol_van"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2024-06-03T08:35:34.445 INFO:teuthology.misc.health.smithi088.stderr:2024-06-03T08:35:34.444+0000 7fefeb576640 10 librados: watch_flush enter 2024-06-03T08:35:34.445 INFO:teuthology.misc.health.smithi088.stderr:2024-06-03T08:35:34.444+0000 7fefeb576640 10 librados: watch_flush exit 2024-06-03T08:35:34.445 INFO:teuthology.misc.health.smithi088.stderr:2024-06-03T08:35:34.444+0000 7fefeb576640 20 client.4179.objecter shutdown clearing up homeless session... 2024-06-03T08:35:34.446 INFO:teuthology.misc.health.smithi088.stderr:2024-06-03T08:35:34.444+0000 7fefeb576640 10 client.4179.objecter successfully canceled tick 2024-06-03T08:35:34.446 INFO:teuthology.misc.health.smithi088.stderr:2024-06-03T08:35:34.444+0000 7fefeb576640 1 -- 172.21.15.88:0/944120537 >> v1:172.21.15.192:6800/34229 conn(0x7fefc806d780 legacy=0x7fefc806fb70 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-06-03T08:35:34.446 INFO:teuthology.misc.health.smithi088.stderr:2024-06-03T08:35:34.444+0000 7fefeb576640 10 monclient: shutdown 2024-06-03T08:35:34.446 INFO:teuthology.misc.health.smithi088.stderr:2024-06-03T08:35:34.444+0000 7fefeb576640 20 monclient: shutdown discarding 0 pending message(s) 2024-06-03T08:35:34.446 INFO:teuthology.misc.health.smithi088.stderr:2024-06-03T08:35:34.444+0000 7fefeb576640 1 -- 172.21.15.88:0/944120537 >> v1:172.21.15.88:6789/0 conn(0x7fefe4144b40 legacy=0x7fefe41deb50 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-06-03T08:35:34.446 INFO:teuthology.misc.health.smithi088.stderr:2024-06-03T08:35:34.445+0000 7fefeb576640 1 -- 172.21.15.88:0/944120537 shutdown_connections 2024-06-03T08:35:34.446 INFO:teuthology.misc.health.smithi088.stderr:2024-06-03T08:35:34.445+0000 7fefeb576640 1 -- 172.21.15.88:0/944120537 >> 172.21.15.88:0/944120537 conn(0x7fefe40b71e0 msgr2=0x7fefe4144f90 unknown :-1 s=STATE_NONE l=0).mark_down 2024-06-03T08:35:34.446 INFO:teuthology.misc.health.smithi088.stderr:2024-06-03T08:35:34.445+0000 7fefeb576640 1 -- 172.21.15.88:0/944120537 shutdown_connections 2024-06-03T08:35:34.446 INFO:teuthology.misc.health.smithi088.stderr:2024-06-03T08:35:34.445+0000 7fefeb576640 1 -- 172.21.15.88:0/944120537 wait complete. 2024-06-03T08:35:34.446 INFO:teuthology.misc.health.smithi088.stderr:2024-06-03T08:35:34.445+0000 7fefeb576640 1 librados: shutdown 2024-06-03T08:35:34.458 DEBUG:teuthology.misc:0 of 8 OSDs are up 2024-06-03T08:35:39.086 INFO:tasks.ceph.osd.7.smithi192.stderr:==00:00:00:06.762 34787== Warning: unimplemented fcntl command: 1038 2024-06-03T08:35:39.148 INFO:tasks.ceph.osd.7.smithi192.stderr:==00:00:00:06.823 34787== Warning: unimplemented fcntl command: 1038 2024-06-03T08:35:39.319 INFO:tasks.ceph.osd.5.smithi192.stderr:==00:00:00:06.999 34786== Warning: unimplemented fcntl command: 1038 2024-06-03T08:35:39.373 INFO:tasks.ceph.osd.4.smithi192.stderr:==00:00:00:07.055 34785== Warning: unimplemented fcntl command: 1038 2024-06-03T08:35:39.382 INFO:tasks.ceph.osd.5.smithi192.stderr:==00:00:00:07.063 34786== Warning: unimplemented fcntl command: 1038 2024-06-03T08:35:39.432 INFO:tasks.ceph.osd.4.smithi192.stderr:==00:00:00:07.114 34785== Warning: unimplemented fcntl command: 1038 2024-06-03T08:35:39.535 INFO:tasks.ceph.osd.6.smithi192.stderr:==00:00:00:07.194 34788== Warning: unimplemented fcntl command: 1038 2024-06-03T08:35:39.592 INFO:tasks.ceph.osd.6.smithi192.stderr:==00:00:00:07.251 34788== Warning: unimplemented fcntl command: 1038 2024-06-03T08:35:39.673 INFO:tasks.ceph.osd.1.smithi088.stderr:==00:00:00:07.351 35067== Warning: unimplemented fcntl command: 1038 2024-06-03T08:35:39.678 INFO:tasks.ceph.osd.2.smithi088.stderr:==00:00:00:07.367 35065== Warning: unimplemented fcntl command: 1038 2024-06-03T08:35:39.692 INFO:tasks.ceph.osd.7.smithi192.stderr:2024-06-03T08:35:39.684+0000 5dcd500 -1 Falling back to public interface 2024-06-03T08:35:39.692 INFO:tasks.ceph.osd.0.smithi088.stderr:==00:00:00:07.369 35066== Warning: unimplemented fcntl command: 1038 2024-06-03T08:35:39.717 INFO:tasks.ceph.osd.3.smithi088.stderr:==00:00:00:07.385 35068== Warning: unimplemented fcntl command: 1038 2024-06-03T08:35:39.733 INFO:tasks.ceph.osd.1.smithi088.stderr:==00:00:00:07.411 35067== Warning: unimplemented fcntl command: 1038 2024-06-03T08:35:39.739 INFO:tasks.ceph.osd.2.smithi088.stderr:==00:00:00:07.428 35065== Warning: unimplemented fcntl command: 1038 2024-06-03T08:35:39.753 INFO:tasks.ceph.osd.0.smithi088.stderr:==00:00:00:07.429 35066== Warning: unimplemented fcntl command: 1038 2024-06-03T08:35:39.777 INFO:tasks.ceph.osd.3.smithi088.stderr:==00:00:00:07.445 35068== Warning: unimplemented fcntl command: 1038 2024-06-03T08:35:39.973 INFO:tasks.ceph.osd.5.smithi192.stderr:2024-06-03T08:35:39.966+0000 5dcd500 -1 Falling back to public interface 2024-06-03T08:35:40.003 INFO:tasks.ceph.osd.4.smithi192.stderr:2024-06-03T08:35:39.997+0000 5dcd500 -1 Falling back to public interface 2024-06-03T08:35:40.132 INFO:tasks.ceph.osd.7.smithi192.stderr:==00:00:00:07.807 34787== Warning: unimplemented fcntl command: 1038 2024-06-03T08:35:40.179 INFO:tasks.ceph.osd.6.smithi192.stderr:2024-06-03T08:35:40.173+0000 5dcd500 -1 Falling back to public interface 2024-06-03T08:35:40.288 INFO:tasks.ceph.osd.1.smithi088.stderr:2024-06-03T08:35:40.278+0000 5dcd500 -1 Falling back to public interface 2024-06-03T08:35:40.292 INFO:tasks.ceph.osd.2.smithi088.stderr:2024-06-03T08:35:40.283+0000 5dcd500 -1 Falling back to public interface 2024-06-03T08:35:40.318 INFO:tasks.ceph.osd.0.smithi088.stderr:2024-06-03T08:35:40.311+0000 5dcd500 -1 Falling back to public interface 2024-06-03T08:35:40.346 INFO:tasks.ceph.osd.3.smithi088.stderr:2024-06-03T08:35:40.335+0000 5dcd500 -1 Falling back to public interface 2024-06-03T08:35:40.445 INFO:tasks.ceph.osd.5.smithi192.stderr:==00:00:00:08.125 34786== Warning: unimplemented fcntl command: 1038 2024-06-03T08:35:40.459 INFO:tasks.ceph.osd.4.smithi192.stderr:==00:00:00:08.141 34785== Warning: unimplemented fcntl command: 1038 2024-06-03T08:35:40.484 INFO:tasks.ceph.osd.7.smithi192.stderr:==00:00:00:08.160 34787== Warning: unimplemented fcntl command: 1038 2024-06-03T08:35:40.633 INFO:tasks.ceph.osd.6.smithi192.stderr:==00:00:00:08.292 34788== Warning: unimplemented fcntl command: 1038 2024-06-03T08:35:40.725 INFO:tasks.ceph.osd.1.smithi088.stderr:==00:00:00:08.403 35067== Warning: unimplemented fcntl command: 1038 2024-06-03T08:35:40.741 INFO:tasks.ceph.osd.2.smithi088.stderr:==00:00:00:08.430 35065== Warning: unimplemented fcntl command: 1038 2024-06-03T08:35:40.757 INFO:tasks.ceph.osd.0.smithi088.stderr:==00:00:00:08.433 35066== Warning: unimplemented fcntl command: 1038 2024-06-03T08:35:40.785 INFO:tasks.ceph.osd.3.smithi088.stderr:==00:00:00:08.452 35068== Warning: unimplemented fcntl command: 1038 2024-06-03T08:35:40.797 INFO:tasks.ceph.osd.5.smithi192.stderr:==00:00:00:08.478 34786== Warning: unimplemented fcntl command: 1038 2024-06-03T08:35:40.822 INFO:tasks.ceph.osd.4.smithi192.stderr:==00:00:00:08.505 34785== Warning: unimplemented fcntl command: 1038 2024-06-03T08:35:40.836 INFO:tasks.ceph.osd.1.smithi088.stderr:==00:00:00:08.514 35067== Warning: unimplemented fcntl command: 1038 2024-06-03T08:35:40.845 INFO:tasks.ceph.osd.7.smithi192.stderr:==00:00:00:08.521 34787== Warning: unimplemented fcntl command: 1038 2024-06-03T08:35:40.869 INFO:tasks.ceph.osd.0.smithi088.stderr:==00:00:00:08.545 35066== Warning: unimplemented fcntl command: 1038 2024-06-03T08:35:40.983 INFO:tasks.ceph.osd.6.smithi192.stderr:==00:00:00:08.643 34788== Warning: unimplemented fcntl command: 1038 2024-06-03T08:35:41.093 INFO:tasks.ceph.osd.2.smithi088.stderr:==00:00:00:08.782 35065== Warning: unimplemented fcntl command: 1038 2024-06-03T08:35:41.148 INFO:tasks.ceph.osd.1.smithi088.stderr:==00:00:00:08.826 35067== Warning: unimplemented fcntl command: 1038 2024-06-03T08:35:41.150 INFO:tasks.ceph.osd.7.smithi192.stderr:==00:00:00:08.826 34787== Warning: unimplemented fcntl command: 1038 2024-06-03T08:35:41.153 INFO:tasks.ceph.osd.3.smithi088.stderr:==00:00:00:08.821 35068== Warning: unimplemented fcntl command: 1038 2024-06-03T08:35:41.160 INFO:tasks.ceph.osd.5.smithi192.stderr:==00:00:00:08.840 34786== Warning: unimplemented fcntl command: 1038 2024-06-03T08:35:41.181 INFO:tasks.ceph.osd.0.smithi088.stderr:==00:00:00:08.857 35066== Warning: unimplemented fcntl command: 1038 2024-06-03T08:35:41.184 INFO:tasks.ceph.osd.4.smithi192.stderr:==00:00:00:08.867 34785== Warning: unimplemented fcntl command: 1038 2024-06-03T08:35:41.262 DEBUG:teuthology.orchestra.run.smithi088:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph --cluster ceph osd dump --format=json 2024-06-03T08:35:41.334 INFO:tasks.ceph.osd.6.smithi192.stderr:==00:00:00:08.994 34788== Warning: unimplemented fcntl command: 1038 2024-06-03T08:35:41.351 INFO:teuthology.misc.health.smithi088.stderr:2024-06-03T08:35:41.351+0000 7fbea5616640 10 monclient: get_monmap_and_config 2024-06-03T08:35:41.351 INFO:teuthology.misc.health.smithi088.stderr:2024-06-03T08:35:41.351+0000 7fbea5616640 10 monclient: build_initial_monmap 2024-06-03T08:35:41.351 INFO:teuthology.misc.health.smithi088.stderr:2024-06-03T08:35:41.351+0000 7fbea5616640 10 monclient: monmap: 2024-06-03T08:35:41.351 INFO:teuthology.misc.health.smithi088.stderr:epoch 0 2024-06-03T08:35:41.351 INFO:teuthology.misc.health.smithi088.stderr:fsid ec3c3fdd-3ac6-4d7d-9412-683130ed8b04 2024-06-03T08:35:41.351 INFO:teuthology.misc.health.smithi088.stderr:last_changed 2024-06-03T08:35:41.352130+0000 2024-06-03T08:35:41.352 INFO:teuthology.misc.health.smithi088.stderr:created 2024-06-03T08:35:41.352130+0000 2024-06-03T08:35:41.352 INFO:teuthology.misc.health.smithi088.stderr:min_mon_release 0 (unknown) 2024-06-03T08:35:41.352 INFO:teuthology.misc.health.smithi088.stderr:election_strategy: 3 2024-06-03T08:35:41.352 INFO:teuthology.misc.health.smithi088.stderr:0: v1:172.21.15.88:6789/0 mon.noname-a 2024-06-03T08:35:41.352 INFO:teuthology.misc.health.smithi088.stderr:1: v1:172.21.15.192:6789/0 mon.noname-c 2024-06-03T08:35:41.352 INFO:teuthology.misc.health.smithi088.stderr:2: v1:172.21.15.88:6790/0 mon.noname-b 2024-06-03T08:35:41.352 INFO:teuthology.misc.health.smithi088.stderr: 2024-06-03T08:35:41.353 INFO:teuthology.misc.health.smithi088.stderr:2024-06-03T08:35:41.351+0000 7fbea5616640 1 Processor -- start 2024-06-03T08:35:41.353 INFO:teuthology.misc.health.smithi088.stderr:2024-06-03T08:35:41.351+0000 7fbea5616640 1 -- start start 2024-06-03T08:35:41.353 INFO:teuthology.misc.health.smithi088.stderr:2024-06-03T08:35:41.351+0000 7fbea5616640 10 monclient: init 2024-06-03T08:35:41.354 INFO:teuthology.misc.health.smithi088.stderr:2024-06-03T08:35:41.352+0000 7fbea5616640 10 monclient: _reopen_session rank -1 2024-06-03T08:35:41.354 INFO:teuthology.misc.health.smithi088.stderr:2024-06-03T08:35:41.352+0000 7fbea5616640 10 monclient: _add_conns ranks=[1,0,2] 2024-06-03T08:35:41.354 INFO:teuthology.misc.health.smithi088.stderr:2024-06-03T08:35:41.352+0000 7fbea5616640 10 monclient(hunting): picked mon.noname-c con 0x7fbea0144ca0 addr v1:172.21.15.192:6789/0 2024-06-03T08:35:41.354 INFO:teuthology.misc.health.smithi088.stderr:2024-06-03T08:35:41.352+0000 7fbea5616640 10 monclient(hunting): picked mon.noname-a con 0x7fbea0145720 addr v1:172.21.15.88:6789/0 2024-06-03T08:35:41.354 INFO:teuthology.misc.health.smithi088.stderr:2024-06-03T08:35:41.352+0000 7fbea5616640 10 monclient(hunting): picked mon.noname-b con 0x7fbea013eed0 addr v1:172.21.15.88:6790/0 2024-06-03T08:35:41.354 INFO:teuthology.misc.health.smithi088.stderr:2024-06-03T08:35:41.352+0000 7fbea5616640 1 -- --> v1:172.21.15.88:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fbea00aa330 con 0x7fbea0145720 2024-06-03T08:35:41.354 INFO:teuthology.misc.health.smithi088.stderr:2024-06-03T08:35:41.352+0000 7fbea5616640 1 -- --> v1:172.21.15.192:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fbea0145290 con 0x7fbea0144ca0 2024-06-03T08:35:41.354 INFO:teuthology.misc.health.smithi088.stderr:2024-06-03T08:35:41.352+0000 7fbea5616640 1 -- --> v1:172.21.15.88:6790/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fbea0145080 con 0x7fbea013eed0 2024-06-03T08:35:41.354 INFO:teuthology.misc.health.smithi088.stderr:2024-06-03T08:35:41.352+0000 7fbea5616640 10 monclient(hunting): _renew_subs 2024-06-03T08:35:41.354 INFO:teuthology.misc.health.smithi088.stderr:2024-06-03T08:35:41.352+0000 7fbea5616640 10 monclient(hunting): authenticate will time out at 1197.162363s 2024-06-03T08:35:41.354 INFO:teuthology.misc.health.smithi088.stderr:2024-06-03T08:35:41.352+0000 7fbe9effd640 1 --1- >> v1:172.21.15.192:6789/0 conn(0x7fbea0144ca0 0x7fbea00b1c70 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.192:6789/0 says I am v1:172.21.15.88:58530/0 (socket says 172.21.15.88:58530) 2024-06-03T08:35:41.354 INFO:teuthology.misc.health.smithi088.stderr:2024-06-03T08:35:41.352+0000 7fbe9effd640 1 -- 172.21.15.88:0/2250836690 learned_addr learned my addr 172.21.15.88:0/2250836690 (peer_addr_for_me v1:172.21.15.88:0/0) 2024-06-03T08:35:41.355 INFO:teuthology.misc.health.smithi088.stderr:2024-06-03T08:35:41.355+0000 7fbe9dffb640 1 -- 172.21.15.88:0/2250836690 <== mon.2 v1:172.21.15.88:6790/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 2077755905 0 0) 0x7fbea0145080 con 0x7fbea013eed0 2024-06-03T08:35:41.356 INFO:teuthology.misc.health.smithi088.stderr:2024-06-03T08:35:41.355+0000 7fbe9dffb640 10 monclient(hunting): _negotiate 2024-06-03T08:35:41.356 INFO:teuthology.misc.health.smithi088.stderr:2024-06-03T08:35:41.355+0000 7fbe9dffb640 10 monclient(hunting): _init_auth method 2 2024-06-03T08:35:41.356 INFO:teuthology.misc.health.smithi088.stderr:2024-06-03T08:35:41.355+0000 7fbe9dffb640 10 monclient(hunting): _init_auth creating new auth 2024-06-03T08:35:41.356 INFO:teuthology.misc.health.smithi088.stderr:2024-06-03T08:35:41.355+0000 7fbe9dffb640 10 monclient(hunting): my global_id is 4208 2024-06-03T08:35:41.356 INFO:teuthology.misc.health.smithi088.stderr:2024-06-03T08:35:41.355+0000 7fbe9dffb640 1 -- 172.21.15.88:0/2250836690 --> v1:172.21.15.88:6790/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7fbe8c003520 con 0x7fbea013eed0 2024-06-03T08:35:41.356 INFO:teuthology.misc.health.smithi088.stderr:2024-06-03T08:35:41.355+0000 7fbe9dffb640 1 -- 172.21.15.88:0/2250836690 <== mon.0 v1:172.21.15.88:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 1381527750 0 0) 0x7fbea00aa330 con 0x7fbea0145720 2024-06-03T08:35:41.356 INFO:teuthology.misc.health.smithi088.stderr:2024-06-03T08:35:41.355+0000 7fbe9dffb640 10 monclient(hunting): _negotiate 2024-06-03T08:35:41.356 INFO:teuthology.misc.health.smithi088.stderr:2024-06-03T08:35:41.355+0000 7fbe9dffb640 10 monclient(hunting): _init_auth method 2 2024-06-03T08:35:41.356 INFO:teuthology.misc.health.smithi088.stderr:2024-06-03T08:35:41.355+0000 7fbe9dffb640 10 monclient(hunting): _init_auth creating new auth 2024-06-03T08:35:41.356 INFO:teuthology.misc.health.smithi088.stderr:2024-06-03T08:35:41.355+0000 7fbe9dffb640 10 monclient(hunting): my global_id is 4206 2024-06-03T08:35:41.356 INFO:teuthology.misc.health.smithi088.stderr:2024-06-03T08:35:41.355+0000 7fbe9dffb640 1 -- 172.21.15.88:0/2250836690 --> v1:172.21.15.88:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7fbea0145080 con 0x7fbea0145720 2024-06-03T08:35:41.356 INFO:teuthology.misc.health.smithi088.stderr:2024-06-03T08:35:41.355+0000 7fbe9dffb640 1 -- 172.21.15.88:0/2250836690 <== mon.1 v1:172.21.15.192:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 2267489959 0 0) 0x7fbea0145290 con 0x7fbea0144ca0 2024-06-03T08:35:41.356 INFO:teuthology.misc.health.smithi088.stderr:2024-06-03T08:35:41.355+0000 7fbe9dffb640 10 monclient(hunting): _negotiate 2024-06-03T08:35:41.357 INFO:teuthology.misc.health.smithi088.stderr:2024-06-03T08:35:41.355+0000 7fbe9dffb640 10 monclient(hunting): _init_auth method 2 2024-06-03T08:35:41.357 INFO:teuthology.misc.health.smithi088.stderr:2024-06-03T08:35:41.355+0000 7fbe9dffb640 10 monclient(hunting): _init_auth creating new auth 2024-06-03T08:35:41.357 INFO:teuthology.misc.health.smithi088.stderr:2024-06-03T08:35:41.355+0000 7fbe9dffb640 10 monclient(hunting): my global_id is 4195 2024-06-03T08:35:41.357 INFO:teuthology.misc.health.smithi088.stderr:2024-06-03T08:35:41.355+0000 7fbe9dffb640 1 -- 172.21.15.88:0/2250836690 --> v1:172.21.15.192:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7fbea00aa330 con 0x7fbea0144ca0 2024-06-03T08:35:41.359 INFO:teuthology.misc.health.smithi088.stderr:2024-06-03T08:35:41.358+0000 7fbe9dffb640 1 -- 172.21.15.88:0/2250836690 <== mon.2 v1:172.21.15.88:6790/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 214+0+0 (unknown 7983010 0 0) 0x7fbe8c003520 con 0x7fbea013eed0 2024-06-03T08:35:41.359 INFO:teuthology.misc.health.smithi088.stderr:2024-06-03T08:35:41.358+0000 7fbe9dffb640 1 -- 172.21.15.88:0/2250836690 >> v1:172.21.15.192:6789/0 conn(0x7fbea0144ca0 legacy=0x7fbea00b1c70 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-06-03T08:35:41.359 INFO:teuthology.misc.health.smithi088.stderr:2024-06-03T08:35:41.358+0000 7fbe9dffb640 1 -- 172.21.15.88:0/2250836690 >> v1:172.21.15.88:6789/0 conn(0x7fbea0145720 legacy=0x7fbea013e7c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-06-03T08:35:41.359 INFO:teuthology.misc.health.smithi088.stderr:2024-06-03T08:35:41.358+0000 7fbe9dffb640 10 monclient: _finish_hunting 0 2024-06-03T08:35:41.359 INFO:teuthology.misc.health.smithi088.stderr:2024-06-03T08:35:41.358+0000 7fbe9dffb640 1 monclient: found mon.noname-b 2024-06-03T08:35:41.359 INFO:teuthology.misc.health.smithi088.stderr:2024-06-03T08:35:41.358+0000 7fbe9dffb640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-06-03T08:35:41.359 INFO:teuthology.misc.health.smithi088.stderr:2024-06-03T08:35:41.358+0000 7fbe9dffb640 10 monclient: _send_mon_message to mon.noname-b at v1:172.21.15.88:6790/0 2024-06-03T08:35:41.359 INFO:teuthology.misc.health.smithi088.stderr:2024-06-03T08:35:41.358+0000 7fbe9dffb640 1 -- 172.21.15.88:0/2250836690 --> v1:172.21.15.88:6790/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fbea00b2380 con 0x7fbea013eed0 2024-06-03T08:35:41.359 INFO:teuthology.misc.health.smithi088.stderr:2024-06-03T08:35:41.358+0000 7fbe9dffb640 10 monclient: _finish_auth 0 2024-06-03T08:35:41.359 INFO:teuthology.misc.health.smithi088.stderr:2024-06-03T08:35:41.358+0000 7fbe9dffb640 10 monclient: _check_auth_tickets 2024-06-03T08:35:41.360 INFO:teuthology.misc.health.smithi088.stderr:2024-06-03T08:35:41.358+0000 7fbe9dffb640 20 monclient: _check_auth_rotating not needed by client.admin 2024-06-03T08:35:41.360 INFO:teuthology.misc.health.smithi088.stderr:2024-06-03T08:35:41.358+0000 7fbea5616640 5 monclient: authenticate success, global_id 4208 2024-06-03T08:35:41.360 INFO:teuthology.misc.health.smithi088.stderr:2024-06-03T08:35:41.358+0000 7fbea5616640 20 monclient: get_monmap_and_config waiting for monmap|config 2024-06-03T08:35:41.360 INFO:teuthology.misc.health.smithi088.stderr:2024-06-03T08:35:41.358+0000 7fbe9dffb640 1 -- 172.21.15.88:0/2250836690 <== mon.2 v1:172.21.15.88:6790/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 2234374065 0 0) 0x7fbe90001990 con 0x7fbea013eed0 2024-06-03T08:35:41.360 INFO:teuthology.misc.health.smithi088.stderr:2024-06-03T08:35:41.358+0000 7fbe9dffb640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-06-03T08:35:41.360 INFO:teuthology.misc.health.smithi088.stderr:2024-06-03T08:35:41.358+0000 7fbe9dffb640 10 monclient: got monmap 1 from mon.noname-b (according to old e1) 2024-06-03T08:35:41.360 INFO:teuthology.misc.health.smithi088.stderr:2024-06-03T08:35:41.358+0000 7fbe9dffb640 10 monclient: dump: 2024-06-03T08:35:41.360 INFO:teuthology.misc.health.smithi088.stderr:epoch 1 2024-06-03T08:35:41.360 INFO:teuthology.misc.health.smithi088.stderr:fsid ec3c3fdd-3ac6-4d7d-9412-683130ed8b04 2024-06-03T08:35:41.360 INFO:teuthology.misc.health.smithi088.stderr:last_changed 2024-06-03T08:34:31.885125+0000 2024-06-03T08:35:41.360 INFO:teuthology.misc.health.smithi088.stderr:created 2024-06-03T08:34:31.885125+0000 2024-06-03T08:35:41.360 INFO:teuthology.misc.health.smithi088.stderr:min_mon_release 18 (reef) 2024-06-03T08:35:41.360 INFO:teuthology.misc.health.smithi088.stderr:election_strategy: 3 2024-06-03T08:35:41.361 INFO:teuthology.misc.health.smithi088.stderr:0: v1:172.21.15.88:6789/0 mon.a 2024-06-03T08:35:41.361 INFO:teuthology.misc.health.smithi088.stderr:1: v1:172.21.15.192:6789/0 mon.b 2024-06-03T08:35:41.361 INFO:teuthology.misc.health.smithi088.stderr:2: v1:172.21.15.88:6790/0 mon.c 2024-06-03T08:35:41.361 INFO:teuthology.misc.health.smithi088.stderr: 2024-06-03T08:35:41.361 INFO:teuthology.misc.health.smithi088.stderr:2024-06-03T08:35:41.358+0000 7fbea5616640 20 monclient: get_monmap_and_config waiting for monmap|config 2024-06-03T08:35:41.364 INFO:teuthology.misc.health.smithi088.stderr:2024-06-03T08:35:41.360+0000 7fbe9dffb640 1 -- 172.21.15.88:0/2250836690 <== mon.2 v1:172.21.15.88:6790/0 4 ==== config(0 keys) v1 ==== 4+0+0 (unknown 0 0 0) 0x7fbe90001be0 con 0x7fbea013eed0 2024-06-03T08:35:41.364 INFO:teuthology.misc.health.smithi088.stderr:2024-06-03T08:35:41.360+0000 7fbe9dffb640 10 monclient: handle_config config(0 keys) v1 2024-06-03T08:35:41.364 INFO:teuthology.misc.health.smithi088.stderr:2024-06-03T08:35:41.360+0000 7fbea5616640 10 monclient: get_monmap_and_config success 2024-06-03T08:35:41.364 INFO:teuthology.misc.health.smithi088.stderr:2024-06-03T08:35:41.360+0000 7fbea5616640 4 set_mon_vals no callback set 2024-06-03T08:35:41.364 INFO:teuthology.misc.health.smithi088.stderr:2024-06-03T08:35:41.360+0000 7fbe9dffb640 1 -- 172.21.15.88:0/2250836690 <== mon.2 v1:172.21.15.88:6790/0 5 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 2234374065 0 0) 0x7fbe90001f70 con 0x7fbea013eed0 2024-06-03T08:35:41.364 INFO:teuthology.misc.health.smithi088.stderr:2024-06-03T08:35:41.360+0000 7fbe9dffb640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-06-03T08:35:41.365 INFO:teuthology.misc.health.smithi088.stderr:2024-06-03T08:35:41.360+0000 7fbe9dffb640 10 monclient: got monmap 1 from mon.c (according to old e1) 2024-06-03T08:35:41.365 INFO:teuthology.misc.health.smithi088.stderr:2024-06-03T08:35:41.360+0000 7fbe9dffb640 10 monclient: dump: 2024-06-03T08:35:41.365 INFO:teuthology.misc.health.smithi088.stderr:epoch 1 2024-06-03T08:35:41.365 INFO:teuthology.misc.health.smithi088.stderr:fsid ec3c3fdd-3ac6-4d7d-9412-683130ed8b04 2024-06-03T08:35:41.365 INFO:teuthology.misc.health.smithi088.stderr:last_changed 2024-06-03T08:34:31.885125+0000 2024-06-03T08:35:41.365 INFO:teuthology.misc.health.smithi088.stderr:created 2024-06-03T08:34:31.885125+0000 2024-06-03T08:35:41.365 INFO:teuthology.misc.health.smithi088.stderr:min_mon_release 18 (reef) 2024-06-03T08:35:41.365 INFO:teuthology.misc.health.smithi088.stderr:election_strategy: 3 2024-06-03T08:35:41.365 INFO:teuthology.misc.health.smithi088.stderr:0: v1:172.21.15.88:6789/0 mon.a 2024-06-03T08:35:41.365 INFO:teuthology.misc.health.smithi088.stderr:1: v1:172.21.15.192:6789/0 mon.b 2024-06-03T08:35:41.366 INFO:teuthology.misc.health.smithi088.stderr:2: v1:172.21.15.88:6790/0 mon.c 2024-06-03T08:35:41.366 INFO:teuthology.misc.health.smithi088.stderr: 2024-06-03T08:35:41.366 INFO:teuthology.misc.health.smithi088.stderr:2024-06-03T08:35:41.360+0000 7fbea5616640 10 monclient: shutdown 2024-06-03T08:35:41.366 INFO:teuthology.misc.health.smithi088.stderr:2024-06-03T08:35:41.360+0000 7fbea5616640 20 monclient: shutdown discarding 0 pending message(s) 2024-06-03T08:35:41.366 INFO:teuthology.misc.health.smithi088.stderr:2024-06-03T08:35:41.360+0000 7fbea5616640 1 -- 172.21.15.88:0/2250836690 >> v1:172.21.15.88:6790/0 conn(0x7fbea013eed0 legacy=0x7fbea014cdd0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-06-03T08:35:41.366 INFO:teuthology.misc.health.smithi088.stderr:2024-06-03T08:35:41.360+0000 7fbea5616640 1 -- 172.21.15.88:0/2250836690 shutdown_connections 2024-06-03T08:35:41.366 INFO:teuthology.misc.health.smithi088.stderr:2024-06-03T08:35:41.360+0000 7fbea5616640 1 -- 172.21.15.88:0/2250836690 >> 172.21.15.88:0/2250836690 conn(0x7fbea00b7270 msgr2=0x7fbea00b7680 unknown :-1 s=STATE_NONE l=0).mark_down 2024-06-03T08:35:41.366 INFO:teuthology.misc.health.smithi088.stderr:2024-06-03T08:35:41.360+0000 7fbea5616640 1 -- 172.21.15.88:0/2250836690 shutdown_connections 2024-06-03T08:35:41.366 INFO:teuthology.misc.health.smithi088.stderr:2024-06-03T08:35:41.360+0000 7fbea5616640 1 -- 172.21.15.88:0/2250836690 wait complete. 2024-06-03T08:35:41.366 INFO:teuthology.misc.health.smithi088.stderr:2024-06-03T08:35:41.361+0000 7fbea5616640 10 monclient: build_initial_monmap 2024-06-03T08:35:41.366 INFO:teuthology.misc.health.smithi088.stderr:2024-06-03T08:35:41.361+0000 7fbea5616640 10 monclient: monmap: 2024-06-03T08:35:41.366 INFO:teuthology.misc.health.smithi088.stderr:epoch 0 2024-06-03T08:35:41.367 INFO:teuthology.misc.health.smithi088.stderr:fsid 00000000-0000-0000-0000-000000000000 2024-06-03T08:35:41.367 INFO:teuthology.misc.health.smithi088.stderr:last_changed 0.000000 2024-06-03T08:35:41.367 INFO:teuthology.misc.health.smithi088.stderr:created 0.000000 2024-06-03T08:35:41.367 INFO:teuthology.misc.health.smithi088.stderr:min_mon_release 0 (unknown) 2024-06-03T08:35:41.367 INFO:teuthology.misc.health.smithi088.stderr:election_strategy: 1 2024-06-03T08:35:41.367 INFO:teuthology.misc.health.smithi088.stderr:0: v1:172.21.15.88:6789/0 mon.noname-a 2024-06-03T08:35:41.367 INFO:teuthology.misc.health.smithi088.stderr:1: v1:172.21.15.192:6789/0 mon.noname-b 2024-06-03T08:35:41.367 INFO:teuthology.misc.health.smithi088.stderr:2: v1:172.21.15.88:6790/0 mon.noname-c 2024-06-03T08:35:41.367 INFO:teuthology.misc.health.smithi088.stderr: 2024-06-03T08:35:41.367 INFO:teuthology.misc.health.smithi088.stderr:2024-06-03T08:35:41.361+0000 7fbea5616640 1 librados: starting msgr at 2024-06-03T08:35:41.367 INFO:teuthology.misc.health.smithi088.stderr:2024-06-03T08:35:41.361+0000 7fbea5616640 1 librados: starting objecter 2024-06-03T08:35:41.367 INFO:teuthology.misc.health.smithi088.stderr:2024-06-03T08:35:41.361+0000 7fbea5616640 1 Processor -- start 2024-06-03T08:35:41.367 INFO:teuthology.misc.health.smithi088.stderr:2024-06-03T08:35:41.362+0000 7fbea5616640 1 -- start start 2024-06-03T08:35:41.368 INFO:teuthology.misc.health.smithi088.stderr:2024-06-03T08:35:41.362+0000 7fbea5616640 1 librados: setting wanted keys 2024-06-03T08:35:41.368 INFO:teuthology.misc.health.smithi088.stderr:2024-06-03T08:35:41.362+0000 7fbea5616640 1 librados: calling monclient init 2024-06-03T08:35:41.368 INFO:teuthology.misc.health.smithi088.stderr:2024-06-03T08:35:41.362+0000 7fbea5616640 10 monclient: init 2024-06-03T08:35:41.368 INFO:teuthology.misc.health.smithi088.stderr:2024-06-03T08:35:41.362+0000 7fbea5616640 10 monclient: _reopen_session rank -1 2024-06-03T08:35:41.368 INFO:teuthology.misc.health.smithi088.stderr:2024-06-03T08:35:41.362+0000 7fbea5616640 10 monclient: _add_conns ranks=[1,0,2] 2024-06-03T08:35:41.368 INFO:teuthology.misc.health.smithi088.stderr:2024-06-03T08:35:41.362+0000 7fbea5616640 10 monclient(hunting): picked mon.noname-b con 0x7fbea013eed0 addr v1:172.21.15.192:6789/0 2024-06-03T08:35:41.368 INFO:teuthology.misc.health.smithi088.stderr:2024-06-03T08:35:41.362+0000 7fbea5616640 10 monclient(hunting): picked mon.noname-a con 0x7fbea0144ca0 addr v1:172.21.15.88:6789/0 2024-06-03T08:35:41.368 INFO:teuthology.misc.health.smithi088.stderr:2024-06-03T08:35:41.362+0000 7fbea5616640 10 monclient(hunting): picked mon.noname-c con 0x7fbea0145720 addr v1:172.21.15.88:6790/0 2024-06-03T08:35:41.368 INFO:teuthology.misc.health.smithi088.stderr:2024-06-03T08:35:41.362+0000 7fbea5616640 1 -- --> v1:172.21.15.88:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fbea01d9bb0 con 0x7fbea0144ca0 2024-06-03T08:35:41.368 INFO:teuthology.misc.health.smithi088.stderr:2024-06-03T08:35:41.362+0000 7fbea5616640 1 -- --> v1:172.21.15.192:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fbea00ac440 con 0x7fbea013eed0 2024-06-03T08:35:41.368 INFO:teuthology.misc.health.smithi088.stderr:2024-06-03T08:35:41.362+0000 7fbea5616640 1 -- --> v1:172.21.15.88:6790/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fbea0140dc0 con 0x7fbea0145720 2024-06-03T08:35:41.368 INFO:teuthology.misc.health.smithi088.stderr:2024-06-03T08:35:41.362+0000 7fbea5616640 10 monclient(hunting): _renew_subs 2024-06-03T08:35:41.368 INFO:teuthology.misc.health.smithi088.stderr:2024-06-03T08:35:41.362+0000 7fbea5616640 10 monclient(hunting): authenticate will time out at 1197.172563s 2024-06-03T08:35:41.369 INFO:teuthology.misc.health.smithi088.stderr:2024-06-03T08:35:41.362+0000 7fbe9f7fe640 1 --1- >> v1:172.21.15.88:6790/0 conn(0x7fbea0145720 0x7fbea01366b0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=1).handle_server_banner_and_identify peer v1:172.21.15.88:6790/0 says I am v1:172.21.15.88:48410/0 (socket says 172.21.15.88:48410) 2024-06-03T08:35:41.369 INFO:teuthology.misc.health.smithi088.stderr:2024-06-03T08:35:41.362+0000 7fbe9f7fe640 1 -- 172.21.15.88:0/3816889373 learned_addr learned my addr 172.21.15.88:0/3816889373 (peer_addr_for_me v1:172.21.15.88:0/0) 2024-06-03T08:35:41.369 INFO:teuthology.misc.health.smithi088.stderr:2024-06-03T08:35:41.363+0000 7fbe977fe640 10 client.?.objecter ms_handle_connect 0x7fbea0145720 2024-06-03T08:35:41.369 INFO:teuthology.misc.health.smithi088.stderr:2024-06-03T08:35:41.363+0000 7fbe977fe640 10 client.?.objecter resend_mon_ops 2024-06-03T08:35:41.369 INFO:teuthology.misc.health.smithi088.stderr:2024-06-03T08:35:41.363+0000 7fbe977fe640 10 client.?.objecter ms_handle_connect 0x7fbea013eed0 2024-06-03T08:35:41.369 INFO:teuthology.misc.health.smithi088.stderr:2024-06-03T08:35:41.363+0000 7fbe977fe640 10 client.?.objecter resend_mon_ops 2024-06-03T08:35:41.369 INFO:teuthology.misc.health.smithi088.stderr:2024-06-03T08:35:41.363+0000 7fbe977fe640 10 client.?.objecter ms_handle_connect 0x7fbea0144ca0 2024-06-03T08:35:41.369 INFO:teuthology.misc.health.smithi088.stderr:2024-06-03T08:35:41.363+0000 7fbe977fe640 10 client.?.objecter resend_mon_ops 2024-06-03T08:35:41.371 INFO:teuthology.misc.health.smithi088.stderr:2024-06-03T08:35:41.365+0000 7fbe977fe640 1 -- 172.21.15.88:0/3816889373 <== mon.1 v1:172.21.15.192:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 2748994714 0 0) 0x7fbea00ac440 con 0x7fbea013eed0 2024-06-03T08:35:41.371 INFO:teuthology.misc.health.smithi088.stderr:2024-06-03T08:35:41.365+0000 7fbe977fe640 10 monclient(hunting): _negotiate 2024-06-03T08:35:41.371 INFO:teuthology.misc.health.smithi088.stderr:2024-06-03T08:35:41.365+0000 7fbe977fe640 10 monclient(hunting): _init_auth method 2 2024-06-03T08:35:41.371 INFO:teuthology.misc.health.smithi088.stderr:2024-06-03T08:35:41.365+0000 7fbe977fe640 10 monclient(hunting): _init_auth creating new auth 2024-06-03T08:35:41.371 INFO:teuthology.misc.health.smithi088.stderr:2024-06-03T08:35:41.365+0000 7fbe977fe640 10 monclient(hunting): my global_id is 4198 2024-06-03T08:35:41.371 INFO:teuthology.misc.health.smithi088.stderr:2024-06-03T08:35:41.365+0000 7fbe977fe640 1 -- 172.21.15.88:0/3816889373 --> v1:172.21.15.192:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7fbe80004eb0 con 0x7fbea013eed0 2024-06-03T08:35:41.372 INFO:teuthology.misc.health.smithi088.stderr:2024-06-03T08:35:41.365+0000 7fbe977fe640 1 -- 172.21.15.88:0/3816889373 <== mon.2 v1:172.21.15.88:6790/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 535129454 0 0) 0x7fbea0140dc0 con 0x7fbea0145720 2024-06-03T08:35:41.372 INFO:teuthology.misc.health.smithi088.stderr:2024-06-03T08:35:41.365+0000 7fbe977fe640 10 monclient(hunting): _negotiate 2024-06-03T08:35:41.372 INFO:teuthology.misc.health.smithi088.stderr:2024-06-03T08:35:41.365+0000 7fbe977fe640 10 monclient(hunting): _init_auth method 2 2024-06-03T08:35:41.372 INFO:teuthology.misc.health.smithi088.stderr:2024-06-03T08:35:41.365+0000 7fbe977fe640 10 monclient(hunting): _init_auth creating new auth 2024-06-03T08:35:41.372 INFO:teuthology.misc.health.smithi088.stderr:2024-06-03T08:35:41.365+0000 7fbe977fe640 10 monclient(hunting): my global_id is 4211 2024-06-03T08:35:41.372 INFO:teuthology.misc.health.smithi088.stderr:2024-06-03T08:35:41.365+0000 7fbe977fe640 1 -- 172.21.15.88:0/3816889373 --> v1:172.21.15.88:6790/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7fbea00ac440 con 0x7fbea0145720 2024-06-03T08:35:41.372 INFO:teuthology.misc.health.smithi088.stderr:2024-06-03T08:35:41.365+0000 7fbe977fe640 1 -- 172.21.15.88:0/3816889373 <== mon.0 v1:172.21.15.88:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 3410872143 0 0) 0x7fbea01d9bb0 con 0x7fbea0144ca0 2024-06-03T08:35:41.372 INFO:teuthology.misc.health.smithi088.stderr:2024-06-03T08:35:41.365+0000 7fbe977fe640 10 monclient(hunting): _negotiate 2024-06-03T08:35:41.372 INFO:teuthology.misc.health.smithi088.stderr:2024-06-03T08:35:41.365+0000 7fbe977fe640 10 monclient(hunting): _init_auth method 2 2024-06-03T08:35:41.372 INFO:teuthology.misc.health.smithi088.stderr:2024-06-03T08:35:41.365+0000 7fbe977fe640 10 monclient(hunting): _init_auth creating new auth 2024-06-03T08:35:41.372 INFO:teuthology.misc.health.smithi088.stderr:2024-06-03T08:35:41.365+0000 7fbe977fe640 10 monclient(hunting): my global_id is 4209 2024-06-03T08:35:41.372 INFO:teuthology.misc.health.smithi088.stderr:2024-06-03T08:35:41.365+0000 7fbe977fe640 1 -- 172.21.15.88:0/3816889373 --> v1:172.21.15.88:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7fbea0140dc0 con 0x7fbea0144ca0 2024-06-03T08:35:41.372 INFO:teuthology.misc.health.smithi088.stderr:2024-06-03T08:35:41.369+0000 7fbe977fe640 1 -- 172.21.15.88:0/3816889373 <== mon.1 v1:172.21.15.192:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 764+0+0 (unknown 1448903332 0 0) 0x7fbe80004eb0 con 0x7fbea013eed0 2024-06-03T08:35:41.373 INFO:teuthology.misc.health.smithi088.stderr:2024-06-03T08:35:41.369+0000 7fbe977fe640 1 -- 172.21.15.88:0/3816889373 --> v1:172.21.15.192:6789/0 -- auth(proto 2 165 bytes epoch 0) v1 -- 0x7fbea01d9bb0 con 0x7fbea013eed0 2024-06-03T08:35:41.373 INFO:teuthology.misc.health.smithi088.stderr:2024-06-03T08:35:41.369+0000 7fbe977fe640 1 -- 172.21.15.88:0/3816889373 <== mon.1 v1:172.21.15.192:6789/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 2234374065 0 0) 0x7fbe980018b0 con 0x7fbea013eed0 2024-06-03T08:35:41.373 INFO:teuthology.misc.health.smithi088.stderr:2024-06-03T08:35:41.369+0000 7fbe977fe640 10 monclient(hunting): handle_monmap mon_map magic: 0 v1 2024-06-03T08:35:41.373 INFO:teuthology.misc.health.smithi088.stderr:2024-06-03T08:35:41.369+0000 7fbe977fe640 10 monclient(hunting): got monmap 1 from mon.noname-b (according to old e1) 2024-06-03T08:35:41.373 INFO:teuthology.misc.health.smithi088.stderr:2024-06-03T08:35:41.369+0000 7fbe977fe640 10 monclient(hunting): dump: 2024-06-03T08:35:41.373 INFO:teuthology.misc.health.smithi088.stderr:epoch 1 2024-06-03T08:35:41.373 INFO:teuthology.misc.health.smithi088.stderr:fsid ec3c3fdd-3ac6-4d7d-9412-683130ed8b04 2024-06-03T08:35:41.373 INFO:teuthology.misc.health.smithi088.stderr:last_changed 2024-06-03T08:34:31.885125+0000 2024-06-03T08:35:41.373 INFO:teuthology.misc.health.smithi088.stderr:created 2024-06-03T08:34:31.885125+0000 2024-06-03T08:35:41.373 INFO:teuthology.misc.health.smithi088.stderr:min_mon_release 18 (reef) 2024-06-03T08:35:41.373 INFO:teuthology.misc.health.smithi088.stderr:election_strategy: 3 2024-06-03T08:35:41.373 INFO:teuthology.misc.health.smithi088.stderr:0: v1:172.21.15.88:6789/0 mon.a 2024-06-03T08:35:41.373 INFO:teuthology.misc.health.smithi088.stderr:1: v1:172.21.15.192:6789/0 mon.b 2024-06-03T08:35:41.373 INFO:teuthology.misc.health.smithi088.stderr:2: v1:172.21.15.88:6790/0 mon.c 2024-06-03T08:35:41.374 INFO:teuthology.misc.health.smithi088.stderr: 2024-06-03T08:35:41.374 INFO:teuthology.misc.health.smithi088.stderr:2024-06-03T08:35:41.369+0000 7fbe977fe640 10 monclient(hunting): _finish_auth 0 2024-06-03T08:35:41.374 INFO:teuthology.misc.health.smithi088.stderr:2024-06-03T08:35:41.370+0000 7fbe977fe640 1 -- 172.21.15.88:0/3816889373 <== mon.2 v1:172.21.15.88:6790/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 764+0+0 (unknown 288172980 0 0) 0x7fbea00ac440 con 0x7fbea0145720 2024-06-03T08:35:41.374 INFO:teuthology.misc.health.smithi088.stderr:2024-06-03T08:35:41.370+0000 7fbe977fe640 1 -- 172.21.15.88:0/3816889373 --> v1:172.21.15.88:6790/0 -- auth(proto 2 165 bytes epoch 0) v1 -- 0x7fbe80004eb0 con 0x7fbea0145720 2024-06-03T08:35:41.374 INFO:teuthology.misc.health.smithi088.stderr:2024-06-03T08:35:41.370+0000 7fbe977fe640 1 -- 172.21.15.88:0/3816889373 <== mon.2 v1:172.21.15.88:6790/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 2234374065 0 0) 0x7fbe90001950 con 0x7fbea0145720 2024-06-03T08:35:41.374 INFO:teuthology.misc.health.smithi088.stderr:2024-06-03T08:35:41.370+0000 7fbe977fe640 10 monclient(hunting): handle_monmap mon_map magic: 0 v1 2024-06-03T08:35:41.374 INFO:teuthology.misc.health.smithi088.stderr:2024-06-03T08:35:41.370+0000 7fbe977fe640 10 monclient(hunting): got monmap 1 from mon.c (according to old e1) 2024-06-03T08:35:41.374 INFO:teuthology.misc.health.smithi088.stderr:2024-06-03T08:35:41.370+0000 7fbe977fe640 10 monclient(hunting): dump: 2024-06-03T08:35:41.374 INFO:teuthology.misc.health.smithi088.stderr:epoch 1 2024-06-03T08:35:41.374 INFO:teuthology.misc.health.smithi088.stderr:fsid ec3c3fdd-3ac6-4d7d-9412-683130ed8b04 2024-06-03T08:35:41.374 INFO:teuthology.misc.health.smithi088.stderr:last_changed 2024-06-03T08:34:31.885125+0000 2024-06-03T08:35:41.374 INFO:teuthology.misc.health.smithi088.stderr:created 2024-06-03T08:34:31.885125+0000 2024-06-03T08:35:41.374 INFO:teuthology.misc.health.smithi088.stderr:min_mon_release 18 (reef) 2024-06-03T08:35:41.374 INFO:teuthology.misc.health.smithi088.stderr:election_strategy: 3 2024-06-03T08:35:41.375 INFO:teuthology.misc.health.smithi088.stderr:0: v1:172.21.15.88:6789/0 mon.a 2024-06-03T08:35:41.375 INFO:teuthology.misc.health.smithi088.stderr:1: v1:172.21.15.192:6789/0 mon.b 2024-06-03T08:35:41.375 INFO:teuthology.misc.health.smithi088.stderr:2: v1:172.21.15.88:6790/0 mon.c 2024-06-03T08:35:41.375 INFO:teuthology.misc.health.smithi088.stderr: 2024-06-03T08:35:41.375 INFO:teuthology.misc.health.smithi088.stderr:2024-06-03T08:35:41.370+0000 7fbe977fe640 1 -- 172.21.15.88:0/3816889373 <== mon.0 v1:172.21.15.88:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 764+0+0 (unknown 3350404689 0 0) 0x7fbea0140dc0 con 0x7fbea0144ca0 2024-06-03T08:35:41.375 INFO:teuthology.misc.health.smithi088.stderr:2024-06-03T08:35:41.370+0000 7fbe977fe640 1 -- 172.21.15.88:0/3816889373 --> v1:172.21.15.88:6789/0 -- auth(proto 2 165 bytes epoch 0) v1 -- 0x7fbea00ac440 con 0x7fbea0144ca0 2024-06-03T08:35:41.375 INFO:teuthology.misc.health.smithi088.stderr:2024-06-03T08:35:41.370+0000 7fbe977fe640 1 -- 172.21.15.88:0/3816889373 <== mon.0 v1:172.21.15.88:6789/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 2234374065 0 0) 0x7fbe880016e0 con 0x7fbea0144ca0 2024-06-03T08:35:41.375 INFO:teuthology.misc.health.smithi088.stderr:2024-06-03T08:35:41.370+0000 7fbe977fe640 10 monclient(hunting): handle_monmap mon_map magic: 0 v1 2024-06-03T08:35:41.375 INFO:teuthology.misc.health.smithi088.stderr:2024-06-03T08:35:41.370+0000 7fbe977fe640 10 monclient(hunting): got monmap 1 from mon.a (according to old e1) 2024-06-03T08:35:41.375 INFO:teuthology.misc.health.smithi088.stderr:2024-06-03T08:35:41.370+0000 7fbe977fe640 10 monclient(hunting): dump: 2024-06-03T08:35:41.375 INFO:teuthology.misc.health.smithi088.stderr:epoch 1 2024-06-03T08:35:41.375 INFO:teuthology.misc.health.smithi088.stderr:fsid ec3c3fdd-3ac6-4d7d-9412-683130ed8b04 2024-06-03T08:35:41.375 INFO:teuthology.misc.health.smithi088.stderr:last_changed 2024-06-03T08:34:31.885125+0000 2024-06-03T08:35:41.375 INFO:teuthology.misc.health.smithi088.stderr:created 2024-06-03T08:34:31.885125+0000 2024-06-03T08:35:41.376 INFO:teuthology.misc.health.smithi088.stderr:min_mon_release 18 (reef) 2024-06-03T08:35:41.376 INFO:teuthology.misc.health.smithi088.stderr:election_strategy: 3 2024-06-03T08:35:41.376 INFO:teuthology.misc.health.smithi088.stderr:0: v1:172.21.15.88:6789/0 mon.a 2024-06-03T08:35:41.376 INFO:teuthology.misc.health.smithi088.stderr:1: v1:172.21.15.192:6789/0 mon.b 2024-06-03T08:35:41.376 INFO:teuthology.misc.health.smithi088.stderr:2: v1:172.21.15.88:6790/0 mon.c 2024-06-03T08:35:41.376 INFO:teuthology.misc.health.smithi088.stderr: 2024-06-03T08:35:41.377 INFO:teuthology.misc.health.smithi088.stderr:2024-06-03T08:35:41.372+0000 7fbe977fe640 1 -- 172.21.15.88:0/3816889373 <== mon.1 v1:172.21.15.192:6789/0 4 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 580+0+0 (unknown 4258443722 0 0) 0x7fbea01d9bb0 con 0x7fbea013eed0 2024-06-03T08:35:41.377 INFO:teuthology.misc.health.smithi088.stderr:2024-06-03T08:35:41.372+0000 7fbe977fe640 1 -- 172.21.15.88:0/3816889373 >> v1:172.21.15.88:6790/0 conn(0x7fbea0145720 legacy=0x7fbea01366b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-06-03T08:35:41.377 INFO:teuthology.misc.health.smithi088.stderr:2024-06-03T08:35:41.372+0000 7fbe977fe640 1 -- 172.21.15.88:0/3816889373 >> v1:172.21.15.88:6789/0 conn(0x7fbea0144ca0 legacy=0x7fbea01da040 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-06-03T08:35:41.378 INFO:teuthology.misc.health.smithi088.stderr:2024-06-03T08:35:41.372+0000 7fbe977fe640 10 monclient: _finish_hunting 0 2024-06-03T08:35:41.378 INFO:teuthology.misc.health.smithi088.stderr:2024-06-03T08:35:41.372+0000 7fbe977fe640 1 monclient: found mon.b 2024-06-03T08:35:41.378 INFO:teuthology.misc.health.smithi088.stderr:2024-06-03T08:35:41.372+0000 7fbe977fe640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-06-03T08:35:41.378 INFO:teuthology.misc.health.smithi088.stderr:2024-06-03T08:35:41.372+0000 7fbe977fe640 10 monclient: _send_mon_message to mon.b at v1:172.21.15.192:6789/0 2024-06-03T08:35:41.378 INFO:teuthology.misc.health.smithi088.stderr:2024-06-03T08:35:41.372+0000 7fbe977fe640 1 -- 172.21.15.88:0/3816889373 --> v1:172.21.15.192:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fbea01da750 con 0x7fbea013eed0 2024-06-03T08:35:41.378 INFO:teuthology.misc.health.smithi088.stderr:2024-06-03T08:35:41.372+0000 7fbe977fe640 10 monclient: _finish_auth 0 2024-06-03T08:35:41.378 INFO:teuthology.misc.health.smithi088.stderr:2024-06-03T08:35:41.372+0000 7fbe977fe640 10 monclient: _check_auth_tickets 2024-06-03T08:35:41.378 INFO:teuthology.misc.health.smithi088.stderr:2024-06-03T08:35:41.372+0000 7fbe977fe640 20 monclient: _check_auth_rotating not needed by client.admin 2024-06-03T08:35:41.378 INFO:teuthology.misc.health.smithi088.stderr:2024-06-03T08:35:41.372+0000 7fbea5616640 5 monclient: authenticate success, global_id 4198 2024-06-03T08:35:41.378 INFO:teuthology.misc.health.smithi088.stderr:2024-06-03T08:35:41.372+0000 7fbea5616640 10 monclient: _renew_subs 2024-06-03T08:35:41.378 INFO:teuthology.misc.health.smithi088.stderr:2024-06-03T08:35:41.372+0000 7fbea5616640 10 monclient: _send_mon_message to mon.b at v1:172.21.15.192:6789/0 2024-06-03T08:35:41.378 INFO:teuthology.misc.health.smithi088.stderr:2024-06-03T08:35:41.372+0000 7fbea5616640 1 -- 172.21.15.88:0/3816889373 --> v1:172.21.15.192:6789/0 -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fbea013f2b0 con 0x7fbea013eed0 2024-06-03T08:35:41.378 INFO:teuthology.misc.health.smithi088.stderr:2024-06-03T08:35:41.372+0000 7fbea5616640 10 client.4198.objecter _maybe_request_map subscribing (onetime) to next osd map 2024-06-03T08:35:41.379 INFO:teuthology.misc.health.smithi088.stderr:2024-06-03T08:35:41.372+0000 7fbea5616640 10 monclient: _renew_subs 2024-06-03T08:35:41.379 INFO:teuthology.misc.health.smithi088.stderr:2024-06-03T08:35:41.372+0000 7fbea5616640 10 monclient: _send_mon_message to mon.b at v1:172.21.15.192:6789/0 2024-06-03T08:35:41.379 INFO:teuthology.misc.health.smithi088.stderr:2024-06-03T08:35:41.372+0000 7fbea5616640 1 -- 172.21.15.88:0/3816889373 --> v1:172.21.15.192:6789/0 -- mon_subscribe({osdmap=0}) v3 -- 0x7fbea0136fa0 con 0x7fbea013eed0 2024-06-03T08:35:41.379 INFO:teuthology.misc.health.smithi088.stderr:2024-06-03T08:35:41.372+0000 7fbea5616640 1 librados: init done 2024-06-03T08:35:41.379 INFO:teuthology.misc.health.smithi088.stderr:2024-06-03T08:35:41.374+0000 7fbea5616640 10 start_mon_command cmd=[{"prefix": "get_command_descriptions"}] 2024-06-03T08:35:41.379 INFO:teuthology.misc.health.smithi088.stderr:2024-06-03T08:35:41.374+0000 7fbea5616640 10 monclient: _send_command 1 [{"prefix": "get_command_descriptions"}] 2024-06-03T08:35:41.379 INFO:teuthology.misc.health.smithi088.stderr:2024-06-03T08:35:41.374+0000 7fbea5616640 10 monclient: _send_mon_message to mon.b at v1:172.21.15.192:6789/0 2024-06-03T08:35:41.379 INFO:teuthology.misc.health.smithi088.stderr:2024-06-03T08:35:41.374+0000 7fbea5616640 1 -- 172.21.15.88:0/3816889373 --> v1:172.21.15.192:6789/0 -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fbea00a9a90 con 0x7fbea013eed0 2024-06-03T08:35:41.388 INFO:teuthology.misc.health.smithi088.stderr:2024-06-03T08:35:41.388+0000 7fbe977fe640 1 -- 172.21.15.88:0/3816889373 <== mon.1 v1:172.21.15.192:6789/0 5 ==== config(0 keys) v1 ==== 4+0+0 (unknown 0 0 0) 0x7fbe980047e0 con 0x7fbea013eed0 2024-06-03T08:35:41.388 INFO:teuthology.misc.health.smithi088.stderr:2024-06-03T08:35:41.388+0000 7fbe977fe640 10 monclient: handle_config config(0 keys) v1 2024-06-03T08:35:41.388 INFO:teuthology.misc.health.smithi088.stderr:2024-06-03T08:35:41.388+0000 7fbe977fe640 1 -- 172.21.15.88:0/3816889373 <== mon.1 v1:172.21.15.192:6789/0 6 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 2234374065 0 0) 0x7fbe98001d80 con 0x7fbea013eed0 2024-06-03T08:35:41.388 INFO:teuthology.misc.health.smithi088.stderr:2024-06-03T08:35:41.388+0000 7fbe977fe640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-06-03T08:35:41.388 INFO:teuthology.misc.health.smithi088.stderr:2024-06-03T08:35:41.388+0000 7fbe977fe640 10 monclient: got monmap 1 from mon.b (according to old e1) 2024-06-03T08:35:41.388 INFO:teuthology.misc.health.smithi088.stderr:2024-06-03T08:35:41.388+0000 7fbe977fe640 10 monclient: dump: 2024-06-03T08:35:41.388 INFO:teuthology.misc.health.smithi088.stderr:epoch 1 2024-06-03T08:35:41.388 INFO:teuthology.misc.health.smithi088.stderr:fsid ec3c3fdd-3ac6-4d7d-9412-683130ed8b04 2024-06-03T08:35:41.389 INFO:teuthology.misc.health.smithi088.stderr:last_changed 2024-06-03T08:34:31.885125+0000 2024-06-03T08:35:41.389 INFO:teuthology.misc.health.smithi088.stderr:created 2024-06-03T08:34:31.885125+0000 2024-06-03T08:35:41.389 INFO:teuthology.misc.health.smithi088.stderr:min_mon_release 18 (reef) 2024-06-03T08:35:41.389 INFO:teuthology.misc.health.smithi088.stderr:election_strategy: 3 2024-06-03T08:35:41.389 INFO:teuthology.misc.health.smithi088.stderr:0: v1:172.21.15.88:6789/0 mon.a 2024-06-03T08:35:41.389 INFO:teuthology.misc.health.smithi088.stderr:1: v1:172.21.15.192:6789/0 mon.b 2024-06-03T08:35:41.389 INFO:teuthology.misc.health.smithi088.stderr:2: v1:172.21.15.88:6790/0 mon.c 2024-06-03T08:35:41.389 INFO:teuthology.misc.health.smithi088.stderr: 2024-06-03T08:35:41.389 INFO:teuthology.misc.health.smithi088.stderr:2024-06-03T08:35:41.388+0000 7fbe9d7fa640 4 set_mon_vals no callback set 2024-06-03T08:35:41.390 INFO:teuthology.misc.health.smithi088.stderr:2024-06-03T08:35:41.388+0000 7fbe977fe640 1 -- 172.21.15.88:0/3816889373 <== mon.1 v1:172.21.15.192:6789/0 7 ==== mgrmap(e 4) v1 ==== 89054+0+0 (unknown 914547237 0 0) 0x7fbe98004b70 con 0x7fbea013eed0 2024-06-03T08:35:41.390 INFO:teuthology.misc.health.smithi088.stderr:2024-06-03T08:35:41.389+0000 7fbe9e7fc640 10 monclient: get_auth_request con 0x7fbe8006d310 auth_method 0 2024-06-03T08:35:41.390 INFO:teuthology.misc.health.smithi088.stderr:2024-06-03T08:35:41.389+0000 7fbe977fe640 10 client.4198.objecter ms_handle_connect 0x7fbe8006d310 2024-06-03T08:35:41.391 INFO:teuthology.misc.health.smithi088.stderr:2024-06-03T08:35:41.391+0000 7fbe977fe640 1 -- 172.21.15.88:0/3816889373 <== mon.1 v1:172.21.15.192:6789/0 8 ==== osd_map(10..10 src has 1..10) v4 ==== 1645+0+0 (unknown 309997979 0 0) 0x7fbe98054080 con 0x7fbea013eed0 2024-06-03T08:35:41.391 INFO:teuthology.misc.health.smithi088.stderr:2024-06-03T08:35:41.391+0000 7fbe977fe640 10 client.4198.objecter ms_dispatch 0x7fbea0000b90 osd_map(10..10 src has 1..10) v4 2024-06-03T08:35:41.391 INFO:teuthology.misc.health.smithi088.stderr:2024-06-03T08:35:41.391+0000 7fbe977fe640 3 client.4198.objecter handle_osd_map got epochs [10,10] > 0 2024-06-03T08:35:41.391 INFO:teuthology.misc.health.smithi088.stderr:2024-06-03T08:35:41.391+0000 7fbe977fe640 3 client.4198.objecter handle_osd_map decoding full epoch 10 2024-06-03T08:35:41.391 INFO:teuthology.misc.health.smithi088.stderr:2024-06-03T08:35:41.391+0000 7fbe977fe640 20 client.4198.objecter dump_active .. 0 homeless 2024-06-03T08:35:41.426 INFO:tasks.ceph.osd.1.smithi088.stderr:==00:00:00:09.104 35067== Warning: unimplemented fcntl command: 1038 2024-06-03T08:35:41.447 INFO:tasks.ceph.osd.7.smithi192.stderr:==00:00:00:09.123 34787== Warning: unimplemented fcntl command: 1038 2024-06-03T08:35:41.452 INFO:tasks.ceph.osd.5.smithi192.stderr:==00:00:00:09.132 34786== Warning: unimplemented fcntl command: 1038 2024-06-03T08:35:41.454 INFO:tasks.ceph.osd.0.smithi088.stderr:==00:00:00:09.131 35066== Warning: unimplemented fcntl command: 1038 2024-06-03T08:35:41.464 INFO:tasks.ceph.osd.2.smithi088.stderr:==00:00:00:09.152 35065== Warning: unimplemented fcntl command: 1038 2024-06-03T08:35:41.479 INFO:tasks.ceph.osd.3.smithi088.stderr:==00:00:00:09.147 35068== Warning: unimplemented fcntl command: 1038 2024-06-03T08:35:41.491 INFO:tasks.ceph.osd.4.smithi192.stderr:==00:00:00:09.174 34785== Warning: unimplemented fcntl command: 1038 2024-06-03T08:35:41.631 INFO:teuthology.misc.health.smithi088.stderr:2024-06-03T08:35:41.630+0000 7fbe977fe640 1 -- 172.21.15.88:0/3816889373 <== mon.1 v1:172.21.15.192:6789/0 9 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+182498 (unknown 1092875540 0 3016760700) 0x7fbe9804fa60 con 0x7fbea013eed0 2024-06-03T08:35:41.631 INFO:teuthology.misc.health.smithi088.stderr:2024-06-03T08:35:41.630+0000 7fbe977fe640 10 monclient: handle_mon_command_ack 1 [{"prefix": "get_command_descriptions"}] 2024-06-03T08:35:41.631 INFO:teuthology.misc.health.smithi088.stderr:2024-06-03T08:35:41.630+0000 7fbe977fe640 10 monclient: _finish_command 1 = system:0 2024-06-03T08:35:41.642 INFO:tasks.ceph.osd.6.smithi192.stderr:==00:00:00:09.302 34788== Warning: unimplemented fcntl command: 1038 2024-06-03T08:35:41.695 INFO:tasks.ceph.osd.1.smithi088.stderr:==00:00:00:09.373 35067== Warning: unimplemented fcntl command: 1038 2024-06-03T08:35:41.721 INFO:tasks.ceph.osd.0.smithi088.stderr:==00:00:00:09.397 35066== Warning: unimplemented fcntl command: 1038 2024-06-03T08:35:41.738 INFO:tasks.ceph.osd.7.smithi192.stderr:==00:00:00:09.414 34787== Warning: unimplemented fcntl command: 1038 2024-06-03T08:35:41.743 INFO:tasks.ceph.osd.5.smithi192.stderr:==00:00:00:09.423 34786== Warning: unimplemented fcntl command: 1038 2024-06-03T08:35:41.756 INFO:tasks.ceph.osd.3.smithi088.stderr:==00:00:00:09.424 35068== Warning: unimplemented fcntl command: 1038 2024-06-03T08:35:41.764 INFO:tasks.ceph.osd.2.smithi088.stderr:==00:00:00:09.453 35065== Warning: unimplemented fcntl command: 1038 2024-06-03T08:35:41.791 INFO:tasks.ceph.osd.4.smithi192.stderr:==00:00:00:09.474 34785== Warning: unimplemented fcntl command: 1038 2024-06-03T08:35:41.809 INFO:teuthology.misc.health.smithi088.stderr:2024-06-03T08:35:41.808+0000 7fbea5616640 10 start_mon_command cmd=[{"prefix": "osd dump", "format": "json"}] 2024-06-03T08:35:41.809 INFO:teuthology.misc.health.smithi088.stderr:2024-06-03T08:35:41.808+0000 7fbea5616640 10 monclient: _send_command 2 [{"prefix": "osd dump", "format": "json"}] 2024-06-03T08:35:41.809 INFO:teuthology.misc.health.smithi088.stderr:2024-06-03T08:35:41.808+0000 7fbea5616640 10 monclient: _send_mon_message to mon.b at v1:172.21.15.192:6789/0 2024-06-03T08:35:41.809 INFO:teuthology.misc.health.smithi088.stderr:2024-06-03T08:35:41.808+0000 7fbea5616640 1 -- 172.21.15.88:0/3816889373 --> v1:172.21.15.192:6789/0 -- mon_command({"prefix": "osd dump", "format": "json"} v 0) v1 -- 0x7fbea01da8e0 con 0x7fbea013eed0 2024-06-03T08:35:41.853 INFO:teuthology.misc.health.smithi088.stderr:2024-06-03T08:35:41.852+0000 7fbe977fe640 1 -- 172.21.15.88:0/3816889373 <== mon.1 v1:172.21.15.192:6789/0 10 ==== mon_command_ack([{"prefix": "osd dump", "format": "json"}]=0 v10) v1 ==== 74+0+7057 (unknown 4040575075 0 1067785724) 0x7fbe98051b60 con 0x7fbea013eed0 2024-06-03T08:35:41.853 INFO:teuthology.misc.health.smithi088.stderr:2024-06-03T08:35:41.852+0000 7fbe977fe640 10 monclient: handle_mon_command_ack 2 [{"prefix": "osd dump", "format": "json"}] 2024-06-03T08:35:41.853 INFO:teuthology.misc.health.smithi088.stderr:2024-06-03T08:35:41.852+0000 7fbe977fe640 10 monclient: _finish_command 2 = system:0 2024-06-03T08:35:41.853 INFO:teuthology.misc.health.smithi088.stdout: 2024-06-03T08:35:41.853 INFO:teuthology.misc.health.smithi088.stdout:{"epoch":10,"fsid":"ec3c3fdd-3ac6-4d7d-9412-683130ed8b04","created":"2024-06-03T08:35:26.799638+0000","modified":"2024-06-03T08:35:32.183068+0000","last_up_change":"0.000000","last_in_change":"2024-06-03T08:35:32.183068+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":2,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":0,"max_osd":8,"require_min_compat_client":"luminous","min_compat_client":"jewel","require_osd_release":"reef","allow_crimson":false,"pools":[],"osds":[{"osd":0,"uuid":"b25ab3bb-3587-45d0-991a-6d41bb529b37","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":1,"uuid":"00d160e7-9773-41d8-8d17-21e70620a0fd","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":2,"uuid":"addfb2a7-2bde-4cc3-98cf-17fd2bdfa3a9","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":3,"uuid":"6c5770b0-c93c-4101-8c53-bd761b8abcde","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":4,"uuid":"f001b3cb-2081-428b-bac3-b3df3c19b978","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":5,"uuid":"3cc263e9-e9bf-45e0-a3b7-83a76469eb70","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":6,"uuid":"82c83316-809f-4901-a5ed-34b6a075b393","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":7,"uuid":"aafe0716-1102-4eae-aac6-0f81bb154e7d","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":4,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":5,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":6,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":7,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[],"pg_upmap_primaries":[],"pg_temp":[],"primary_temp":[],"blocklist":{},"range_blocklist":{},"erasure_code_profiles":{"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","technique":"reed_sol_van"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2024-06-03T08:35:41.855 INFO:teuthology.misc.health.smithi088.stderr:2024-06-03T08:35:41.854+0000 7fbea5616640 10 librados: watch_flush enter 2024-06-03T08:35:41.855 INFO:teuthology.misc.health.smithi088.stderr:2024-06-03T08:35:41.854+0000 7fbea5616640 10 librados: watch_flush exit 2024-06-03T08:35:41.855 INFO:teuthology.misc.health.smithi088.stderr:2024-06-03T08:35:41.854+0000 7fbea5616640 20 client.4198.objecter shutdown clearing up homeless session... 2024-06-03T08:35:41.855 INFO:teuthology.misc.health.smithi088.stderr:2024-06-03T08:35:41.854+0000 7fbea5616640 10 client.4198.objecter successfully canceled tick 2024-06-03T08:35:41.856 INFO:teuthology.misc.health.smithi088.stderr:2024-06-03T08:35:41.854+0000 7fbea5616640 1 -- 172.21.15.88:0/3816889373 >> v1:172.21.15.192:6800/34229 conn(0x7fbe8006d310 legacy=0x7fbe8006f7d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-06-03T08:35:41.856 INFO:teuthology.misc.health.smithi088.stderr:2024-06-03T08:35:41.854+0000 7fbea5616640 10 monclient: shutdown 2024-06-03T08:35:41.856 INFO:teuthology.misc.health.smithi088.stderr:2024-06-03T08:35:41.854+0000 7fbea5616640 20 monclient: shutdown discarding 0 pending message(s) 2024-06-03T08:35:41.856 INFO:teuthology.misc.health.smithi088.stderr:2024-06-03T08:35:41.854+0000 7fbea5616640 1 -- 172.21.15.88:0/3816889373 >> v1:172.21.15.192:6789/0 conn(0x7fbea013eed0 legacy=0x7fbea00aad90 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-06-03T08:35:41.856 INFO:teuthology.misc.health.smithi088.stderr:2024-06-03T08:35:41.854+0000 7fbea5616640 1 -- 172.21.15.88:0/3816889373 shutdown_connections 2024-06-03T08:35:41.856 INFO:teuthology.misc.health.smithi088.stderr:2024-06-03T08:35:41.854+0000 7fbea5616640 1 -- 172.21.15.88:0/3816889373 >> 172.21.15.88:0/3816889373 conn(0x7fbea00b7270 msgr2=0x7fbea00b7650 unknown :-1 s=STATE_NONE l=0).mark_down 2024-06-03T08:35:41.856 INFO:teuthology.misc.health.smithi088.stderr:2024-06-03T08:35:41.854+0000 7fbea5616640 1 -- 172.21.15.88:0/3816889373 shutdown_connections 2024-06-03T08:35:41.856 INFO:teuthology.misc.health.smithi088.stderr:2024-06-03T08:35:41.854+0000 7fbea5616640 1 -- 172.21.15.88:0/3816889373 wait complete. 2024-06-03T08:35:41.856 INFO:teuthology.misc.health.smithi088.stderr:2024-06-03T08:35:41.854+0000 7fbea5616640 1 librados: shutdown 2024-06-03T08:35:41.864 DEBUG:teuthology.misc:0 of 8 OSDs are up 2024-06-03T08:35:41.930 INFO:tasks.ceph.osd.6.smithi192.stderr:==00:00:00:09.589 34788== Warning: unimplemented fcntl command: 1038 2024-06-03T08:35:41.962 INFO:tasks.ceph.osd.1.smithi088.stderr:==00:00:00:09.640 35067== Warning: unimplemented fcntl command: 1038 2024-06-03T08:35:41.985 INFO:tasks.ceph.osd.0.smithi088.stderr:==00:00:00:09.661 35066== Warning: unimplemented fcntl command: 1038 2024-06-03T08:35:42.025 INFO:tasks.ceph.osd.3.smithi088.stderr:==00:00:00:09.692 35068== Warning: unimplemented fcntl command: 1038 2024-06-03T08:35:42.030 INFO:tasks.ceph.osd.7.smithi192.stderr:==00:00:00:09.706 34787== Warning: unimplemented fcntl command: 1038 2024-06-03T08:35:42.037 INFO:tasks.ceph.osd.5.smithi192.stderr:==00:00:00:09.718 34786== Warning: unimplemented fcntl command: 1038 2024-06-03T08:35:42.066 INFO:tasks.ceph.osd.2.smithi088.stderr:==00:00:00:09.755 35065== Warning: unimplemented fcntl command: 1038 2024-06-03T08:35:42.085 INFO:tasks.ceph.osd.4.smithi192.stderr:==00:00:00:09.768 34785== Warning: unimplemented fcntl command: 1038 2024-06-03T08:35:42.221 INFO:tasks.ceph.osd.6.smithi192.stderr:==00:00:00:09.881 34788== Warning: unimplemented fcntl command: 1038 2024-06-03T08:35:42.231 INFO:tasks.ceph.osd.1.smithi088.stderr:==00:00:00:09.909 35067== Warning: unimplemented fcntl command: 1038 2024-06-03T08:35:42.254 INFO:tasks.ceph.osd.0.smithi088.stderr:==00:00:00:09.930 35066== Warning: unimplemented fcntl command: 1038 2024-06-03T08:35:42.293 INFO:tasks.ceph.osd.3.smithi088.stderr:==00:00:00:09.961 35068== Warning: unimplemented fcntl command: 1038 2024-06-03T08:35:42.323 INFO:tasks.ceph.osd.7.smithi192.stderr:==00:00:00:09.999 34787== Warning: unimplemented fcntl command: 1038 2024-06-03T08:35:42.328 INFO:tasks.ceph.osd.5.smithi192.stderr:==00:00:00:10.008 34786== Warning: unimplemented fcntl command: 1038 2024-06-03T08:35:42.353 INFO:tasks.ceph.osd.2.smithi088.stderr:==00:00:00:10.042 35065== Warning: unimplemented fcntl command: 1038 2024-06-03T08:35:42.375 INFO:tasks.ceph.osd.4.smithi192.stderr:==00:00:00:10.058 34785== Warning: unimplemented fcntl command: 1038 2024-06-03T08:35:42.502 INFO:tasks.ceph.osd.1.smithi088.stderr:==00:00:00:10.180 35067== Warning: unimplemented fcntl command: 1038 2024-06-03T08:35:42.512 INFO:tasks.ceph.osd.6.smithi192.stderr:==00:00:00:10.172 34788== Warning: unimplemented fcntl command: 1038 2024-06-03T08:35:42.524 INFO:tasks.ceph.osd.0.smithi088.stderr:==00:00:00:10.201 35066== Warning: unimplemented fcntl command: 1038 2024-06-03T08:35:42.558 INFO:tasks.ceph.osd.3.smithi088.stderr:==00:00:00:10.225 35068== Warning: unimplemented fcntl command: 1038 2024-06-03T08:35:42.623 INFO:tasks.ceph.osd.2.smithi088.stderr:==00:00:00:10.312 35065== Warning: unimplemented fcntl command: 1038 2024-06-03T08:35:42.625 INFO:tasks.ceph.osd.7.smithi192.stderr:==00:00:00:10.301 34787== Warning: unimplemented fcntl command: 1038 2024-06-03T08:35:42.633 INFO:tasks.ceph.osd.5.smithi192.stderr:==00:00:00:10.313 34786== Warning: unimplemented fcntl command: 1038 2024-06-03T08:35:42.665 INFO:tasks.ceph.osd.4.smithi192.stderr:==00:00:00:10.348 34785== Warning: unimplemented fcntl command: 1038 2024-06-03T08:35:42.770 INFO:tasks.ceph.osd.1.smithi088.stderr:==00:00:00:10.448 35067== Warning: unimplemented fcntl command: 1038 2024-06-03T08:35:42.795 INFO:tasks.ceph.osd.0.smithi088.stderr:==00:00:00:10.472 35066== Warning: unimplemented fcntl command: 1038 2024-06-03T08:35:42.800 INFO:tasks.ceph.osd.6.smithi192.stderr:==00:00:00:10.460 34788== Warning: unimplemented fcntl command: 1038 2024-06-03T08:35:42.824 INFO:tasks.ceph.osd.3.smithi088.stderr:==00:00:00:10.491 35068== Warning: unimplemented fcntl command: 1038 2024-06-03T08:35:42.918 INFO:tasks.ceph.osd.2.smithi088.stderr:==00:00:00:10.607 35065== Warning: unimplemented fcntl command: 1038 2024-06-03T08:35:42.930 INFO:tasks.ceph.osd.5.smithi192.stderr:==00:00:00:10.610 34786== Warning: unimplemented fcntl command: 1038 2024-06-03T08:35:42.973 INFO:tasks.ceph.osd.4.smithi192.stderr:==00:00:00:10.655 34785== Warning: unimplemented fcntl command: 1038 2024-06-03T08:35:43.087 INFO:tasks.ceph.osd.6.smithi192.stderr:==00:00:00:10.747 34788== Warning: unimplemented fcntl command: 1038 2024-06-03T08:35:43.109 INFO:tasks.ceph.osd.3.smithi088.stderr:==00:00:00:10.776 35068== Warning: unimplemented fcntl command: 1038 2024-06-03T08:35:43.153 INFO:tasks.ceph.osd.7.smithi192.stderr:==00:00:00:10.828 34787== Warning: unimplemented fcntl command: 1038 2024-06-03T08:35:43.205 INFO:tasks.ceph.osd.7.smithi192.stderr:==00:00:00:10.880 34787== Warning: unimplemented fcntl command: 1038 2024-06-03T08:35:43.221 INFO:tasks.ceph.osd.2.smithi088.stderr:==00:00:00:10.909 35065== Warning: unimplemented fcntl command: 1038 2024-06-03T08:35:43.238 INFO:tasks.ceph.osd.1.smithi088.stderr:==00:00:00:10.916 35067== Warning: unimplemented fcntl command: 1038 2024-06-03T08:35:43.293 INFO:tasks.ceph.osd.1.smithi088.stderr:==00:00:00:10.971 35067== Warning: unimplemented fcntl command: 1038 2024-06-03T08:35:43.299 INFO:tasks.ceph.osd.0.smithi088.stderr:==00:00:00:10.976 35066== Warning: unimplemented fcntl command: 1038 2024-06-03T08:35:43.354 INFO:tasks.ceph.osd.0.smithi088.stderr:==00:00:00:11.031 35066== Warning: unimplemented fcntl command: 1038 2024-06-03T08:35:43.450 INFO:tasks.ceph.osd.5.smithi192.stderr:==00:00:00:11.130 34786== Warning: unimplemented fcntl command: 1038 2024-06-03T08:35:43.505 INFO:tasks.ceph.osd.5.smithi192.stderr:==00:00:00:11.186 34786== Warning: unimplemented fcntl command: 1038 2024-06-03T08:35:43.522 INFO:tasks.ceph.osd.4.smithi192.stderr:==00:00:00:11.205 34785== Warning: unimplemented fcntl command: 1038 2024-06-03T08:35:43.581 INFO:tasks.ceph.osd.3.smithi088.stderr:==00:00:00:11.249 35068== Warning: unimplemented fcntl command: 1038 2024-06-03T08:35:43.601 INFO:tasks.ceph.osd.4.smithi192.stderr:==00:00:00:11.284 34785== Warning: unimplemented fcntl command: 1038 2024-06-03T08:35:43.637 INFO:tasks.ceph.osd.3.smithi088.stderr:==00:00:00:11.305 35068== Warning: unimplemented fcntl command: 1038 2024-06-03T08:35:43.658 INFO:tasks.ceph.osd.6.smithi192.stderr:==00:00:00:11.317 34788== Warning: unimplemented fcntl command: 1038 2024-06-03T08:35:43.760 INFO:tasks.ceph.osd.6.smithi192.stderr:==00:00:00:11.420 34788== Warning: unimplemented fcntl command: 1038 2024-06-03T08:35:43.817 INFO:tasks.ceph.osd.2.smithi088.stderr:==00:00:00:11.506 35065== Warning: unimplemented fcntl command: 1038 2024-06-03T08:35:43.872 INFO:tasks.ceph.osd.2.smithi088.stderr:==00:00:00:11.561 35065== Warning: unimplemented fcntl command: 1038 2024-06-03T08:35:48.630 INFO:tasks.ceph.osd.0.smithi088.stderr:==00:00:00:16.306 35066== Warning: unimplemented fcntl command: 1038 2024-06-03T08:35:48.671 DEBUG:teuthology.orchestra.run.smithi088:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph --cluster ceph osd dump --format=json 2024-06-03T08:35:48.689 INFO:tasks.ceph.osd.7.smithi192.stderr:==00:00:00:16.365 34787== Warning: unimplemented fcntl command: 1038 2024-06-03T08:35:48.763 INFO:teuthology.misc.health.smithi088.stderr:2024-06-03T08:35:48.762+0000 7f68a55fb640 10 monclient: get_monmap_and_config 2024-06-03T08:35:48.763 INFO:teuthology.misc.health.smithi088.stderr:2024-06-03T08:35:48.762+0000 7f68a55fb640 10 monclient: build_initial_monmap 2024-06-03T08:35:48.763 INFO:teuthology.misc.health.smithi088.stderr:2024-06-03T08:35:48.762+0000 7f68a55fb640 10 monclient: monmap: 2024-06-03T08:35:48.763 INFO:teuthology.misc.health.smithi088.stderr:epoch 0 2024-06-03T08:35:48.763 INFO:teuthology.misc.health.smithi088.stderr:fsid ec3c3fdd-3ac6-4d7d-9412-683130ed8b04 2024-06-03T08:35:48.763 INFO:teuthology.misc.health.smithi088.stderr:last_changed 2024-06-03T08:35:48.763484+0000 2024-06-03T08:35:48.763 INFO:teuthology.misc.health.smithi088.stderr:created 2024-06-03T08:35:48.763484+0000 2024-06-03T08:35:48.763 INFO:teuthology.misc.health.smithi088.stderr:min_mon_release 0 (unknown) 2024-06-03T08:35:48.763 INFO:teuthology.misc.health.smithi088.stderr:election_strategy: 3 2024-06-03T08:35:48.763 INFO:teuthology.misc.health.smithi088.stderr:0: v1:172.21.15.88:6789/0 mon.noname-a 2024-06-03T08:35:48.764 INFO:teuthology.misc.health.smithi088.stderr:1: v1:172.21.15.192:6789/0 mon.noname-c 2024-06-03T08:35:48.764 INFO:teuthology.misc.health.smithi088.stderr:2: v1:172.21.15.88:6790/0 mon.noname-b 2024-06-03T08:35:48.764 INFO:teuthology.misc.health.smithi088.stderr: 2024-06-03T08:35:48.765 INFO:teuthology.misc.health.smithi088.stderr:2024-06-03T08:35:48.763+0000 7f68a55fb640 1 Processor -- start 2024-06-03T08:35:48.765 INFO:teuthology.misc.health.smithi088.stderr:2024-06-03T08:35:48.763+0000 7f68a55fb640 1 -- start start 2024-06-03T08:35:48.765 INFO:teuthology.misc.health.smithi088.stderr:2024-06-03T08:35:48.763+0000 7f68a55fb640 10 monclient: init 2024-06-03T08:35:48.765 INFO:teuthology.misc.health.smithi088.stderr:2024-06-03T08:35:48.763+0000 7f68a55fb640 10 monclient: _reopen_session rank -1 2024-06-03T08:35:48.765 INFO:teuthology.misc.health.smithi088.stderr:2024-06-03T08:35:48.763+0000 7f68a55fb640 10 monclient: _add_conns ranks=[2,0,1] 2024-06-03T08:35:48.765 INFO:teuthology.misc.health.smithi088.stderr:2024-06-03T08:35:48.763+0000 7f68a55fb640 10 monclient(hunting): picked mon.noname-b con 0x7f68a0144ca0 addr v1:172.21.15.88:6790/0 2024-06-03T08:35:48.765 INFO:teuthology.misc.health.smithi088.stderr:2024-06-03T08:35:48.763+0000 7f68a55fb640 10 monclient(hunting): picked mon.noname-a con 0x7f68a0145720 addr v1:172.21.15.88:6789/0 2024-06-03T08:35:48.765 INFO:teuthology.misc.health.smithi088.stderr:2024-06-03T08:35:48.764+0000 7f68a55fb640 10 monclient(hunting): picked mon.noname-c con 0x7f68a013eed0 addr v1:172.21.15.192:6789/0 2024-06-03T08:35:48.766 INFO:teuthology.misc.health.smithi088.stderr:2024-06-03T08:35:48.764+0000 7f68a55fb640 1 -- --> v1:172.21.15.88:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f68a00aa330 con 0x7f68a0145720 2024-06-03T08:35:48.766 INFO:teuthology.misc.health.smithi088.stderr:2024-06-03T08:35:48.764+0000 7f68a55fb640 1 -- --> v1:172.21.15.192:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f68a0145290 con 0x7f68a013eed0 2024-06-03T08:35:48.766 INFO:teuthology.misc.health.smithi088.stderr:2024-06-03T08:35:48.764+0000 7f68a55fb640 1 -- --> v1:172.21.15.88:6790/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f68a0145080 con 0x7f68a0144ca0 2024-06-03T08:35:48.766 INFO:teuthology.misc.health.smithi088.stderr:2024-06-03T08:35:48.764+0000 7f68a55fb640 10 monclient(hunting): _renew_subs 2024-06-03T08:35:48.766 INFO:teuthology.misc.health.smithi088.stderr:2024-06-03T08:35:48.764+0000 7f68a55fb640 10 monclient(hunting): authenticate will time out at 1204.574274s 2024-06-03T08:35:48.766 INFO:teuthology.misc.health.smithi088.stderr:2024-06-03T08:35:48.764+0000 7f689effd640 1 --1- >> v1:172.21.15.88:6790/0 conn(0x7f68a0144ca0 0x7f68a00b1c70 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.88:6790/0 says I am v1:172.21.15.88:47700/0 (socket says 172.21.15.88:47700) 2024-06-03T08:35:48.766 INFO:teuthology.misc.health.smithi088.stderr:2024-06-03T08:35:48.764+0000 7f689effd640 1 -- 172.21.15.88:0/3843889073 learned_addr learned my addr 172.21.15.88:0/3843889073 (peer_addr_for_me v1:172.21.15.88:0/0) 2024-06-03T08:35:48.768 INFO:teuthology.misc.health.smithi088.stderr:2024-06-03T08:35:48.767+0000 7f689dffb640 1 -- 172.21.15.88:0/3843889073 <== mon.1 v1:172.21.15.192:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 967002026 0 0) 0x7f68a0145290 con 0x7f68a013eed0 2024-06-03T08:35:48.768 INFO:teuthology.misc.health.smithi088.stderr:2024-06-03T08:35:48.767+0000 7f689dffb640 10 monclient(hunting): _negotiate 2024-06-03T08:35:48.768 INFO:teuthology.misc.health.smithi088.stderr:2024-06-03T08:35:48.767+0000 7f689dffb640 10 monclient(hunting): _init_auth method 2 2024-06-03T08:35:48.768 INFO:teuthology.misc.health.smithi088.stderr:2024-06-03T08:35:48.767+0000 7f689dffb640 10 monclient(hunting): _init_auth creating new auth 2024-06-03T08:35:48.768 INFO:teuthology.misc.health.smithi088.stderr:2024-06-03T08:35:48.767+0000 7f689dffb640 10 monclient(hunting): my global_id is 4201 2024-06-03T08:35:48.768 INFO:teuthology.misc.health.smithi088.stderr:2024-06-03T08:35:48.767+0000 7f689dffb640 1 -- 172.21.15.88:0/3843889073 --> v1:172.21.15.192:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f6888003520 con 0x7f68a013eed0 2024-06-03T08:35:48.768 INFO:teuthology.misc.health.smithi088.stderr:2024-06-03T08:35:48.767+0000 7f689dffb640 1 -- 172.21.15.88:0/3843889073 <== mon.2 v1:172.21.15.88:6790/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 1239542150 0 0) 0x7f68a0145080 con 0x7f68a0144ca0 2024-06-03T08:35:48.768 INFO:teuthology.misc.health.smithi088.stderr:2024-06-03T08:35:48.767+0000 7f689dffb640 10 monclient(hunting): _negotiate 2024-06-03T08:35:48.768 INFO:teuthology.misc.health.smithi088.stderr:2024-06-03T08:35:48.767+0000 7f689dffb640 10 monclient(hunting): _init_auth method 2 2024-06-03T08:35:48.768 INFO:teuthology.misc.health.smithi088.stderr:2024-06-03T08:35:48.767+0000 7f689dffb640 10 monclient(hunting): _init_auth creating new auth 2024-06-03T08:35:48.768 INFO:teuthology.misc.health.smithi088.stderr:2024-06-03T08:35:48.767+0000 7f689dffb640 10 monclient(hunting): my global_id is 4214 2024-06-03T08:35:48.768 INFO:teuthology.misc.health.smithi088.stderr:2024-06-03T08:35:48.767+0000 7f689dffb640 1 -- 172.21.15.88:0/3843889073 --> v1:172.21.15.88:6790/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f68a0145290 con 0x7f68a0144ca0 2024-06-03T08:35:48.769 INFO:teuthology.misc.health.smithi088.stderr:2024-06-03T08:35:48.768+0000 7f689dffb640 1 -- 172.21.15.88:0/3843889073 <== mon.0 v1:172.21.15.88:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 2214165444 0 0) 0x7f68a00aa330 con 0x7f68a0145720 2024-06-03T08:35:48.769 INFO:teuthology.misc.health.smithi088.stderr:2024-06-03T08:35:48.768+0000 7f689dffb640 10 monclient(hunting): _negotiate 2024-06-03T08:35:48.769 INFO:teuthology.misc.health.smithi088.stderr:2024-06-03T08:35:48.768+0000 7f689dffb640 10 monclient(hunting): _init_auth method 2 2024-06-03T08:35:48.769 INFO:teuthology.misc.health.smithi088.stderr:2024-06-03T08:35:48.768+0000 7f689dffb640 10 monclient(hunting): _init_auth creating new auth 2024-06-03T08:35:48.769 INFO:teuthology.misc.health.smithi088.stderr:2024-06-03T08:35:48.768+0000 7f689dffb640 10 monclient(hunting): my global_id is 4212 2024-06-03T08:35:48.769 INFO:teuthology.misc.health.smithi088.stderr:2024-06-03T08:35:48.768+0000 7f689dffb640 1 -- 172.21.15.88:0/3843889073 --> v1:172.21.15.88:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f68880039d0 con 0x7f68a0145720 2024-06-03T08:35:48.771 INFO:teuthology.misc.health.smithi088.stderr:2024-06-03T08:35:48.771+0000 7f689dffb640 1 -- 172.21.15.88:0/3843889073 <== mon.1 v1:172.21.15.192:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 214+0+0 (unknown 1411140797 0 0) 0x7f6888003520 con 0x7f68a013eed0 2024-06-03T08:35:48.772 INFO:teuthology.misc.health.smithi088.stderr:2024-06-03T08:35:48.771+0000 7f689dffb640 1 -- 172.21.15.88:0/3843889073 >> v1:172.21.15.88:6790/0 conn(0x7f68a0144ca0 legacy=0x7f68a00b1c70 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-06-03T08:35:48.772 INFO:teuthology.misc.health.smithi088.stderr:2024-06-03T08:35:48.771+0000 7f689dffb640 1 -- 172.21.15.88:0/3843889073 >> v1:172.21.15.88:6789/0 conn(0x7f68a0145720 legacy=0x7f68a013e7c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-06-03T08:35:48.772 INFO:teuthology.misc.health.smithi088.stderr:2024-06-03T08:35:48.771+0000 7f689dffb640 10 monclient: _finish_hunting 0 2024-06-03T08:35:48.772 INFO:teuthology.misc.health.smithi088.stderr:2024-06-03T08:35:48.771+0000 7f689dffb640 1 monclient: found mon.noname-c 2024-06-03T08:35:48.772 INFO:teuthology.misc.health.smithi088.stderr:2024-06-03T08:35:48.771+0000 7f689dffb640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-06-03T08:35:48.772 INFO:teuthology.misc.health.smithi088.stderr:2024-06-03T08:35:48.771+0000 7f689dffb640 10 monclient: _send_mon_message to mon.noname-c at v1:172.21.15.192:6789/0 2024-06-03T08:35:48.772 INFO:teuthology.misc.health.smithi088.stderr:2024-06-03T08:35:48.771+0000 7f689dffb640 1 -- 172.21.15.88:0/3843889073 --> v1:172.21.15.192:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f68a00b2380 con 0x7f68a013eed0 2024-06-03T08:35:48.772 INFO:teuthology.misc.health.smithi088.stderr:2024-06-03T08:35:48.771+0000 7f689dffb640 10 monclient: _finish_auth 0 2024-06-03T08:35:48.772 INFO:teuthology.misc.health.smithi088.stderr:2024-06-03T08:35:48.771+0000 7f689dffb640 10 monclient: _check_auth_tickets 2024-06-03T08:35:48.772 INFO:teuthology.misc.health.smithi088.stderr:2024-06-03T08:35:48.771+0000 7f689dffb640 20 monclient: _check_auth_rotating not needed by client.admin 2024-06-03T08:35:48.772 INFO:teuthology.misc.health.smithi088.stderr:2024-06-03T08:35:48.771+0000 7f689dffb640 1 -- 172.21.15.88:0/3843889073 <== mon.1 v1:172.21.15.192:6789/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 2234374065 0 0) 0x7f6898001990 con 0x7f68a013eed0 2024-06-03T08:35:48.772 INFO:teuthology.misc.health.smithi088.stderr:2024-06-03T08:35:48.771+0000 7f689dffb640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-06-03T08:35:48.772 INFO:teuthology.misc.health.smithi088.stderr:2024-06-03T08:35:48.771+0000 7f689dffb640 10 monclient: got monmap 1 from mon.noname-c (according to old e1) 2024-06-03T08:35:48.773 INFO:teuthology.misc.health.smithi088.stderr:2024-06-03T08:35:48.771+0000 7f689dffb640 10 monclient: dump: 2024-06-03T08:35:48.773 INFO:teuthology.misc.health.smithi088.stderr:epoch 1 2024-06-03T08:35:48.773 INFO:teuthology.misc.health.smithi088.stderr:fsid ec3c3fdd-3ac6-4d7d-9412-683130ed8b04 2024-06-03T08:35:48.773 INFO:teuthology.misc.health.smithi088.stderr:last_changed 2024-06-03T08:34:31.885125+0000 2024-06-03T08:35:48.773 INFO:teuthology.misc.health.smithi088.stderr:created 2024-06-03T08:34:31.885125+0000 2024-06-03T08:35:48.773 INFO:teuthology.misc.health.smithi088.stderr:min_mon_release 18 (reef) 2024-06-03T08:35:48.773 INFO:teuthology.misc.health.smithi088.stderr:election_strategy: 3 2024-06-03T08:35:48.773 INFO:teuthology.misc.health.smithi088.stderr:0: v1:172.21.15.88:6789/0 mon.a 2024-06-03T08:35:48.773 INFO:teuthology.misc.health.smithi088.stderr:1: v1:172.21.15.192:6789/0 mon.b 2024-06-03T08:35:48.773 INFO:teuthology.misc.health.smithi088.stderr:2: v1:172.21.15.88:6790/0 mon.c 2024-06-03T08:35:48.773 INFO:teuthology.misc.health.smithi088.stderr: 2024-06-03T08:35:48.773 INFO:teuthology.misc.health.smithi088.stderr:2024-06-03T08:35:48.771+0000 7f68a55fb640 5 monclient: authenticate success, global_id 4201 2024-06-03T08:35:48.773 INFO:teuthology.misc.health.smithi088.stderr:2024-06-03T08:35:48.771+0000 7f68a55fb640 20 monclient: get_monmap_and_config waiting for monmap|config 2024-06-03T08:35:48.775 INFO:teuthology.misc.health.smithi088.stderr:2024-06-03T08:35:48.773+0000 7f689dffb640 1 -- 172.21.15.88:0/3843889073 <== mon.1 v1:172.21.15.192:6789/0 4 ==== config(0 keys) v1 ==== 4+0+0 (unknown 0 0 0) 0x7f6898001bb0 con 0x7f68a013eed0 2024-06-03T08:35:48.775 INFO:teuthology.misc.health.smithi088.stderr:2024-06-03T08:35:48.773+0000 7f689dffb640 10 monclient: handle_config config(0 keys) v1 2024-06-03T08:35:48.775 INFO:teuthology.misc.health.smithi088.stderr:2024-06-03T08:35:48.773+0000 7f689dffb640 1 -- 172.21.15.88:0/3843889073 <== mon.1 v1:172.21.15.192:6789/0 5 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 2234374065 0 0) 0x7f6898001f40 con 0x7f68a013eed0 2024-06-03T08:35:48.775 INFO:teuthology.misc.health.smithi088.stderr:2024-06-03T08:35:48.773+0000 7f689dffb640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-06-03T08:35:48.775 INFO:teuthology.misc.health.smithi088.stderr:2024-06-03T08:35:48.773+0000 7f689dffb640 10 monclient: got monmap 1 from mon.b (according to old e1) 2024-06-03T08:35:48.775 INFO:teuthology.misc.health.smithi088.stderr:2024-06-03T08:35:48.773+0000 7f689dffb640 10 monclient: dump: 2024-06-03T08:35:48.775 INFO:teuthology.misc.health.smithi088.stderr:epoch 1 2024-06-03T08:35:48.775 INFO:teuthology.misc.health.smithi088.stderr:fsid ec3c3fdd-3ac6-4d7d-9412-683130ed8b04 2024-06-03T08:35:48.775 INFO:teuthology.misc.health.smithi088.stderr:last_changed 2024-06-03T08:34:31.885125+0000 2024-06-03T08:35:48.775 INFO:teuthology.misc.health.smithi088.stderr:created 2024-06-03T08:34:31.885125+0000 2024-06-03T08:35:48.776 INFO:teuthology.misc.health.smithi088.stderr:min_mon_release 18 (reef) 2024-06-03T08:35:48.776 INFO:teuthology.misc.health.smithi088.stderr:election_strategy: 3 2024-06-03T08:35:48.776 INFO:teuthology.misc.health.smithi088.stderr:0: v1:172.21.15.88:6789/0 mon.a 2024-06-03T08:35:48.776 INFO:teuthology.misc.health.smithi088.stderr:1: v1:172.21.15.192:6789/0 mon.b 2024-06-03T08:35:48.776 INFO:teuthology.misc.health.smithi088.stderr:2: v1:172.21.15.88:6790/0 mon.c 2024-06-03T08:35:48.776 INFO:teuthology.misc.health.smithi088.stderr: 2024-06-03T08:35:48.776 INFO:teuthology.misc.health.smithi088.stderr:2024-06-03T08:35:48.773+0000 7f68a55fb640 10 monclient: get_monmap_and_config success 2024-06-03T08:35:48.776 INFO:teuthology.misc.health.smithi088.stderr:2024-06-03T08:35:48.773+0000 7f68a55fb640 4 set_mon_vals no callback set 2024-06-03T08:35:48.776 INFO:teuthology.misc.health.smithi088.stderr:2024-06-03T08:35:48.774+0000 7f68a55fb640 10 monclient: shutdown 2024-06-03T08:35:48.776 INFO:teuthology.misc.health.smithi088.stderr:2024-06-03T08:35:48.774+0000 7f68a55fb640 20 monclient: shutdown discarding 0 pending message(s) 2024-06-03T08:35:48.776 INFO:teuthology.misc.health.smithi088.stderr:2024-06-03T08:35:48.774+0000 7f68a55fb640 1 -- 172.21.15.88:0/3843889073 >> v1:172.21.15.192:6789/0 conn(0x7f68a013eed0 legacy=0x7f68a014cdd0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-06-03T08:35:48.776 INFO:teuthology.misc.health.smithi088.stderr:2024-06-03T08:35:48.774+0000 7f68a55fb640 1 -- 172.21.15.88:0/3843889073 shutdown_connections 2024-06-03T08:35:48.776 INFO:teuthology.misc.health.smithi088.stderr:2024-06-03T08:35:48.774+0000 7f68a55fb640 1 -- 172.21.15.88:0/3843889073 >> 172.21.15.88:0/3843889073 conn(0x7f68a00b7270 msgr2=0x7f68a00b7680 unknown :-1 s=STATE_NONE l=0).mark_down 2024-06-03T08:35:48.776 INFO:teuthology.misc.health.smithi088.stderr:2024-06-03T08:35:48.774+0000 7f68a55fb640 1 -- 172.21.15.88:0/3843889073 shutdown_connections 2024-06-03T08:35:48.777 INFO:teuthology.misc.health.smithi088.stderr:2024-06-03T08:35:48.774+0000 7f68a55fb640 1 -- 172.21.15.88:0/3843889073 wait complete. 2024-06-03T08:35:48.779 INFO:teuthology.misc.health.smithi088.stderr:2024-06-03T08:35:48.775+0000 7f68a55fb640 10 monclient: build_initial_monmap 2024-06-03T08:35:48.779 INFO:teuthology.misc.health.smithi088.stderr:2024-06-03T08:35:48.775+0000 7f68a55fb640 10 monclient: monmap: 2024-06-03T08:35:48.779 INFO:teuthology.misc.health.smithi088.stderr:epoch 0 2024-06-03T08:35:48.780 INFO:teuthology.misc.health.smithi088.stderr:fsid 00000000-0000-0000-0000-000000000000 2024-06-03T08:35:48.780 INFO:teuthology.misc.health.smithi088.stderr:last_changed 0.000000 2024-06-03T08:35:48.780 INFO:teuthology.misc.health.smithi088.stderr:created 0.000000 2024-06-03T08:35:48.780 INFO:teuthology.misc.health.smithi088.stderr:min_mon_release 0 (unknown) 2024-06-03T08:35:48.780 INFO:teuthology.misc.health.smithi088.stderr:election_strategy: 1 2024-06-03T08:35:48.780 INFO:teuthology.misc.health.smithi088.stderr:0: v1:172.21.15.88:6789/0 mon.noname-a 2024-06-03T08:35:48.780 INFO:teuthology.misc.health.smithi088.stderr:1: v1:172.21.15.192:6789/0 mon.noname-b 2024-06-03T08:35:48.780 INFO:teuthology.misc.health.smithi088.stderr:2: v1:172.21.15.88:6790/0 mon.noname-c 2024-06-03T08:35:48.780 INFO:teuthology.misc.health.smithi088.stderr: 2024-06-03T08:35:48.780 INFO:teuthology.misc.health.smithi088.stderr:2024-06-03T08:35:48.776+0000 7f68a55fb640 1 librados: starting msgr at 2024-06-03T08:35:48.780 INFO:teuthology.misc.health.smithi088.stderr:2024-06-03T08:35:48.776+0000 7f68a55fb640 1 librados: starting objecter 2024-06-03T08:35:48.780 INFO:teuthology.misc.health.smithi088.stderr:2024-06-03T08:35:48.776+0000 7f68a55fb640 1 Processor -- start 2024-06-03T08:35:48.780 INFO:teuthology.misc.health.smithi088.stderr:2024-06-03T08:35:48.776+0000 7f68a55fb640 1 -- start start 2024-06-03T08:35:48.781 INFO:teuthology.misc.health.smithi088.stderr:2024-06-03T08:35:48.776+0000 7f68a55fb640 1 librados: setting wanted keys 2024-06-03T08:35:48.781 INFO:teuthology.misc.health.smithi088.stderr:2024-06-03T08:35:48.776+0000 7f68a55fb640 1 librados: calling monclient init 2024-06-03T08:35:48.781 INFO:teuthology.misc.health.smithi088.stderr:2024-06-03T08:35:48.776+0000 7f68a55fb640 10 monclient: init 2024-06-03T08:35:48.781 INFO:teuthology.misc.health.smithi088.stderr:2024-06-03T08:35:48.777+0000 7f68a55fb640 10 monclient: _reopen_session rank -1 2024-06-03T08:35:48.781 INFO:teuthology.misc.health.smithi088.stderr:2024-06-03T08:35:48.777+0000 7f68a55fb640 10 monclient: _add_conns ranks=[0,2,1] 2024-06-03T08:35:48.781 INFO:teuthology.misc.health.smithi088.stderr:2024-06-03T08:35:48.778+0000 7f68a55fb640 10 monclient(hunting): picked mon.noname-a con 0x7f68a013eed0 addr v1:172.21.15.88:6789/0 2024-06-03T08:35:48.781 INFO:teuthology.misc.health.smithi088.stderr:2024-06-03T08:35:48.778+0000 7f68a55fb640 10 monclient(hunting): picked mon.noname-c con 0x7f68a0144ca0 addr v1:172.21.15.88:6790/0 2024-06-03T08:35:48.781 INFO:teuthology.misc.health.smithi088.stderr:2024-06-03T08:35:48.778+0000 7f68a55fb640 10 monclient(hunting): picked mon.noname-b con 0x7f68a0145720 addr v1:172.21.15.192:6789/0 2024-06-03T08:35:48.781 INFO:teuthology.misc.health.smithi088.stderr:2024-06-03T08:35:48.778+0000 7f68a55fb640 1 -- --> v1:172.21.15.88:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f68a00ac580 con 0x7f68a013eed0 2024-06-03T08:35:48.781 INFO:teuthology.misc.health.smithi088.stderr:2024-06-03T08:35:48.778+0000 7f68a55fb640 1 -- --> v1:172.21.15.192:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f68a0140ee0 con 0x7f68a0145720 2024-06-03T08:35:48.781 INFO:teuthology.misc.health.smithi088.stderr:2024-06-03T08:35:48.778+0000 7f68a55fb640 1 -- --> v1:172.21.15.88:6790/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f68a01d6650 con 0x7f68a0144ca0 2024-06-03T08:35:48.781 INFO:teuthology.misc.health.smithi088.stderr:2024-06-03T08:35:48.778+0000 7f68a55fb640 10 monclient(hunting): _renew_subs 2024-06-03T08:35:48.781 INFO:teuthology.misc.health.smithi088.stderr:2024-06-03T08:35:48.778+0000 7f68a55fb640 10 monclient(hunting): authenticate will time out at 1204.588381s 2024-06-03T08:35:48.782 INFO:teuthology.misc.health.smithi088.stderr:2024-06-03T08:35:48.778+0000 7f689e7fc640 1 --1- >> v1:172.21.15.88:6790/0 conn(0x7f68a0144ca0 0x7f68a01d6b70 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=1).handle_server_banner_and_identify peer v1:172.21.15.88:6790/0 says I am v1:172.21.15.88:47702/0 (socket says 172.21.15.88:47702) 2024-06-03T08:35:48.782 INFO:teuthology.misc.health.smithi088.stderr:2024-06-03T08:35:48.778+0000 7f689e7fc640 1 -- 172.21.15.88:0/374680176 learned_addr learned my addr 172.21.15.88:0/374680176 (peer_addr_for_me v1:172.21.15.88:0/0) 2024-06-03T08:35:48.782 INFO:teuthology.misc.health.smithi088.stderr:2024-06-03T08:35:48.779+0000 7f68877fe640 10 client.?.objecter ms_handle_connect 0x7f68a0144ca0 2024-06-03T08:35:48.782 INFO:teuthology.misc.health.smithi088.stderr:2024-06-03T08:35:48.779+0000 7f68877fe640 10 client.?.objecter resend_mon_ops 2024-06-03T08:35:48.782 INFO:teuthology.misc.health.smithi088.stderr:2024-06-03T08:35:48.779+0000 7f68877fe640 10 client.?.objecter ms_handle_connect 0x7f68a013eed0 2024-06-03T08:35:48.782 INFO:teuthology.misc.health.smithi088.stderr:2024-06-03T08:35:48.779+0000 7f68877fe640 10 client.?.objecter resend_mon_ops 2024-06-03T08:35:48.782 INFO:teuthology.misc.health.smithi088.stderr:2024-06-03T08:35:48.779+0000 7f68877fe640 10 client.?.objecter ms_handle_connect 0x7f68a0145720 2024-06-03T08:35:48.782 INFO:teuthology.misc.health.smithi088.stderr:2024-06-03T08:35:48.779+0000 7f68877fe640 10 client.?.objecter resend_mon_ops 2024-06-03T08:35:48.784 INFO:teuthology.misc.health.smithi088.stderr:2024-06-03T08:35:48.780+0000 7f68877fe640 1 -- 172.21.15.88:0/374680176 <== mon.1 v1:172.21.15.192:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 88975378 0 0) 0x7f68a0140ee0 con 0x7f68a0145720 2024-06-03T08:35:48.784 INFO:teuthology.misc.health.smithi088.stderr:2024-06-03T08:35:48.780+0000 7f68877fe640 10 monclient(hunting): _negotiate 2024-06-03T08:35:48.784 INFO:teuthology.misc.health.smithi088.stderr:2024-06-03T08:35:48.780+0000 7f68877fe640 10 monclient(hunting): _init_auth method 2 2024-06-03T08:35:48.784 INFO:teuthology.misc.health.smithi088.stderr:2024-06-03T08:35:48.780+0000 7f68877fe640 10 monclient(hunting): _init_auth creating new auth 2024-06-03T08:35:48.784 INFO:teuthology.misc.health.smithi088.stderr:2024-06-03T08:35:48.780+0000 7f68877fe640 10 monclient(hunting): my global_id is 4204 2024-06-03T08:35:48.784 INFO:teuthology.misc.health.smithi088.stderr:2024-06-03T08:35:48.780+0000 7f68877fe640 1 -- 172.21.15.88:0/374680176 --> v1:172.21.15.192:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f688c004eb0 con 0x7f68a0145720 2024-06-03T08:35:48.784 INFO:teuthology.misc.health.smithi088.stderr:2024-06-03T08:35:48.781+0000 7f68877fe640 1 -- 172.21.15.88:0/374680176 <== mon.2 v1:172.21.15.88:6790/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 4005591385 0 0) 0x7f68a01d6650 con 0x7f68a0144ca0 2024-06-03T08:35:48.785 INFO:teuthology.misc.health.smithi088.stderr:2024-06-03T08:35:48.781+0000 7f68877fe640 10 monclient(hunting): _negotiate 2024-06-03T08:35:48.785 INFO:teuthology.misc.health.smithi088.stderr:2024-06-03T08:35:48.781+0000 7f68877fe640 10 monclient(hunting): _init_auth method 2 2024-06-03T08:35:48.785 INFO:teuthology.misc.health.smithi088.stderr:2024-06-03T08:35:48.781+0000 7f68877fe640 10 monclient(hunting): _init_auth creating new auth 2024-06-03T08:35:48.785 INFO:teuthology.misc.health.smithi088.stderr:2024-06-03T08:35:48.781+0000 7f68877fe640 10 monclient(hunting): my global_id is 4217 2024-06-03T08:35:48.785 INFO:teuthology.misc.health.smithi088.stderr:2024-06-03T08:35:48.781+0000 7f68877fe640 1 -- 172.21.15.88:0/374680176 --> v1:172.21.15.88:6790/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f68a0140ee0 con 0x7f68a0144ca0 2024-06-03T08:35:48.785 INFO:teuthology.misc.health.smithi088.stderr:2024-06-03T08:35:48.782+0000 7f68877fe640 1 -- 172.21.15.88:0/374680176 <== mon.0 v1:172.21.15.88:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 569104443 0 0) 0x7f68a00ac580 con 0x7f68a013eed0 2024-06-03T08:35:48.785 INFO:teuthology.misc.health.smithi088.stderr:2024-06-03T08:35:48.782+0000 7f68877fe640 10 monclient(hunting): _negotiate 2024-06-03T08:35:48.785 INFO:teuthology.misc.health.smithi088.stderr:2024-06-03T08:35:48.782+0000 7f68877fe640 10 monclient(hunting): _init_auth method 2 2024-06-03T08:35:48.785 INFO:teuthology.misc.health.smithi088.stderr:2024-06-03T08:35:48.782+0000 7f68877fe640 10 monclient(hunting): _init_auth creating new auth 2024-06-03T08:35:48.785 INFO:teuthology.misc.health.smithi088.stderr:2024-06-03T08:35:48.782+0000 7f68877fe640 10 monclient(hunting): my global_id is 4215 2024-06-03T08:35:48.785 INFO:teuthology.misc.health.smithi088.stderr:2024-06-03T08:35:48.782+0000 7f68877fe640 1 -- 172.21.15.88:0/374680176 --> v1:172.21.15.88:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f68a01d6650 con 0x7f68a013eed0 2024-06-03T08:35:48.787 INFO:teuthology.misc.health.smithi088.stderr:2024-06-03T08:35:48.786+0000 7f68877fe640 1 -- 172.21.15.88:0/374680176 <== mon.1 v1:172.21.15.192:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 764+0+0 (unknown 3902194898 0 0) 0x7f688c004eb0 con 0x7f68a0145720 2024-06-03T08:35:48.787 INFO:teuthology.misc.health.smithi088.stderr:2024-06-03T08:35:48.786+0000 7f68877fe640 1 -- 172.21.15.88:0/374680176 --> v1:172.21.15.192:6789/0 -- auth(proto 2 165 bytes epoch 0) v1 -- 0x7f68a00ac580 con 0x7f68a0145720 2024-06-03T08:35:48.787 INFO:teuthology.misc.health.smithi088.stderr:2024-06-03T08:35:48.786+0000 7f68877fe640 1 -- 172.21.15.88:0/374680176 <== mon.1 v1:172.21.15.192:6789/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 2234374065 0 0) 0x7f6898001ec0 con 0x7f68a0145720 2024-06-03T08:35:48.787 INFO:teuthology.misc.health.smithi088.stderr:2024-06-03T08:35:48.786+0000 7f68877fe640 10 monclient(hunting): handle_monmap mon_map magic: 0 v1 2024-06-03T08:35:48.787 INFO:teuthology.misc.health.smithi088.stderr:2024-06-03T08:35:48.786+0000 7f68877fe640 10 monclient(hunting): got monmap 1 from mon.noname-b (according to old e1) 2024-06-03T08:35:48.787 INFO:teuthology.misc.health.smithi088.stderr:2024-06-03T08:35:48.786+0000 7f68877fe640 10 monclient(hunting): dump: 2024-06-03T08:35:48.787 INFO:teuthology.misc.health.smithi088.stderr:epoch 1 2024-06-03T08:35:48.787 INFO:teuthology.misc.health.smithi088.stderr:fsid ec3c3fdd-3ac6-4d7d-9412-683130ed8b04 2024-06-03T08:35:48.787 INFO:teuthology.misc.health.smithi088.stderr:last_changed 2024-06-03T08:34:31.885125+0000 2024-06-03T08:35:48.787 INFO:teuthology.misc.health.smithi088.stderr:created 2024-06-03T08:34:31.885125+0000 2024-06-03T08:35:48.788 INFO:teuthology.misc.health.smithi088.stderr:min_mon_release 18 (reef) 2024-06-03T08:35:48.788 INFO:teuthology.misc.health.smithi088.stderr:election_strategy: 3 2024-06-03T08:35:48.788 INFO:teuthology.misc.health.smithi088.stderr:0: v1:172.21.15.88:6789/0 mon.a 2024-06-03T08:35:48.788 INFO:teuthology.misc.health.smithi088.stderr:1: v1:172.21.15.192:6789/0 mon.b 2024-06-03T08:35:48.788 INFO:teuthology.misc.health.smithi088.stderr:2: v1:172.21.15.88:6790/0 mon.c 2024-06-03T08:35:48.788 INFO:teuthology.misc.health.smithi088.stderr: 2024-06-03T08:35:48.788 INFO:teuthology.misc.health.smithi088.stderr:2024-06-03T08:35:48.786+0000 7f68877fe640 10 monclient(hunting): _finish_auth 0 2024-06-03T08:35:48.790 INFO:teuthology.misc.health.smithi088.stderr:2024-06-03T08:35:48.789+0000 7f68877fe640 1 -- 172.21.15.88:0/374680176 <== mon.2 v1:172.21.15.88:6790/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 764+0+0 (unknown 3247414067 0 0) 0x7f68a0140ee0 con 0x7f68a0144ca0 2024-06-03T08:35:48.790 INFO:teuthology.misc.health.smithi088.stderr:2024-06-03T08:35:48.789+0000 7f68877fe640 1 -- 172.21.15.88:0/374680176 --> v1:172.21.15.88:6790/0 -- auth(proto 2 165 bytes epoch 0) v1 -- 0x7f688c004eb0 con 0x7f68a0144ca0 2024-06-03T08:35:48.790 INFO:teuthology.misc.health.smithi088.stderr:2024-06-03T08:35:48.789+0000 7f68877fe640 1 -- 172.21.15.88:0/374680176 <== mon.2 v1:172.21.15.88:6790/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 2234374065 0 0) 0x7f68940018d0 con 0x7f68a0144ca0 2024-06-03T08:35:48.790 INFO:teuthology.misc.health.smithi088.stderr:2024-06-03T08:35:48.789+0000 7f68877fe640 10 monclient(hunting): handle_monmap mon_map magic: 0 v1 2024-06-03T08:35:48.790 INFO:teuthology.misc.health.smithi088.stderr:2024-06-03T08:35:48.789+0000 7f68877fe640 10 monclient(hunting): got monmap 1 from mon.c (according to old e1) 2024-06-03T08:35:48.790 INFO:teuthology.misc.health.smithi088.stderr:2024-06-03T08:35:48.789+0000 7f68877fe640 10 monclient(hunting): dump: 2024-06-03T08:35:48.790 INFO:teuthology.misc.health.smithi088.stderr:epoch 1 2024-06-03T08:35:48.790 INFO:teuthology.misc.health.smithi088.stderr:fsid ec3c3fdd-3ac6-4d7d-9412-683130ed8b04 2024-06-03T08:35:48.790 INFO:teuthology.misc.health.smithi088.stderr:last_changed 2024-06-03T08:34:31.885125+0000 2024-06-03T08:35:48.790 INFO:teuthology.misc.health.smithi088.stderr:created 2024-06-03T08:34:31.885125+0000 2024-06-03T08:35:48.790 INFO:teuthology.misc.health.smithi088.stderr:min_mon_release 18 (reef) 2024-06-03T08:35:48.790 INFO:teuthology.misc.health.smithi088.stderr:election_strategy: 3 2024-06-03T08:35:48.791 INFO:teuthology.misc.health.smithi088.stderr:0: v1:172.21.15.88:6789/0 mon.a 2024-06-03T08:35:48.791 INFO:teuthology.misc.health.smithi088.stderr:1: v1:172.21.15.192:6789/0 mon.b 2024-06-03T08:35:48.791 INFO:teuthology.misc.health.smithi088.stderr:2: v1:172.21.15.88:6790/0 mon.c 2024-06-03T08:35:48.791 INFO:teuthology.misc.health.smithi088.stderr: 2024-06-03T08:35:48.793 INFO:teuthology.misc.health.smithi088.stderr:2024-06-03T08:35:48.789+0000 7f68877fe640 1 -- 172.21.15.88:0/374680176 <== mon.1 v1:172.21.15.192:6789/0 4 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 580+0+0 (unknown 654282866 0 0) 0x7f68a00ac580 con 0x7f68a0145720 2024-06-03T08:35:48.793 INFO:teuthology.misc.health.smithi088.stderr:2024-06-03T08:35:48.790+0000 7f68877fe640 1 -- 172.21.15.88:0/374680176 >> v1:172.21.15.88:6790/0 conn(0x7f68a0144ca0 legacy=0x7f68a01d6b70 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-06-03T08:35:48.794 INFO:teuthology.misc.health.smithi088.stderr:2024-06-03T08:35:48.790+0000 7f68877fe640 1 -- 172.21.15.88:0/374680176 >> v1:172.21.15.88:6789/0 conn(0x7f68a013eed0 legacy=0x7f68a00aade0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-06-03T08:35:48.794 INFO:teuthology.misc.health.smithi088.stderr:2024-06-03T08:35:48.790+0000 7f68877fe640 10 monclient: _finish_hunting 0 2024-06-03T08:35:48.794 INFO:teuthology.misc.health.smithi088.stderr:2024-06-03T08:35:48.790+0000 7f68877fe640 1 monclient: found mon.b 2024-06-03T08:35:48.794 INFO:teuthology.misc.health.smithi088.stderr:2024-06-03T08:35:48.790+0000 7f68877fe640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-06-03T08:35:48.794 INFO:teuthology.misc.health.smithi088.stderr:2024-06-03T08:35:48.790+0000 7f68877fe640 10 monclient: _send_mon_message to mon.b at v1:172.21.15.192:6789/0 2024-06-03T08:35:48.794 INFO:teuthology.misc.health.smithi088.stderr:2024-06-03T08:35:48.790+0000 7f68877fe640 1 -- 172.21.15.88:0/374680176 --> v1:172.21.15.192:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f68a013f2b0 con 0x7f68a0145720 2024-06-03T08:35:48.794 INFO:teuthology.misc.health.smithi088.stderr:2024-06-03T08:35:48.790+0000 7f68877fe640 10 monclient: _finish_auth 0 2024-06-03T08:35:48.794 INFO:teuthology.misc.health.smithi088.stderr:2024-06-03T08:35:48.790+0000 7f68877fe640 10 monclient: _check_auth_tickets 2024-06-03T08:35:48.794 INFO:teuthology.misc.health.smithi088.stderr:2024-06-03T08:35:48.790+0000 7f68877fe640 20 monclient: _check_auth_rotating not needed by client.admin 2024-06-03T08:35:48.794 INFO:teuthology.misc.health.smithi088.stderr:2024-06-03T08:35:48.790+0000 7f68a55fb640 5 monclient: authenticate success, global_id 4204 2024-06-03T08:35:48.794 INFO:teuthology.misc.health.smithi088.stderr:2024-06-03T08:35:48.790+0000 7f68a55fb640 10 monclient: _renew_subs 2024-06-03T08:35:48.794 INFO:teuthology.misc.health.smithi088.stderr:2024-06-03T08:35:48.790+0000 7f68a55fb640 10 monclient: _send_mon_message to mon.b at v1:172.21.15.192:6789/0 2024-06-03T08:35:48.794 INFO:teuthology.misc.health.smithi088.stderr:2024-06-03T08:35:48.790+0000 7f68a55fb640 1 -- 172.21.15.88:0/374680176 --> v1:172.21.15.192:6789/0 -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f68a01d7280 con 0x7f68a0145720 2024-06-03T08:35:48.795 INFO:teuthology.misc.health.smithi088.stderr:2024-06-03T08:35:48.790+0000 7f68a55fb640 10 client.4204.objecter _maybe_request_map subscribing (onetime) to next osd map 2024-06-03T08:35:48.795 INFO:teuthology.misc.health.smithi088.stderr:2024-06-03T08:35:48.790+0000 7f68a55fb640 10 monclient: _renew_subs 2024-06-03T08:35:48.795 INFO:teuthology.misc.health.smithi088.stderr:2024-06-03T08:35:48.790+0000 7f68a55fb640 10 monclient: _send_mon_message to mon.b at v1:172.21.15.192:6789/0 2024-06-03T08:35:48.795 INFO:teuthology.misc.health.smithi088.stderr:2024-06-03T08:35:48.790+0000 7f68a55fb640 1 -- 172.21.15.88:0/374680176 --> v1:172.21.15.192:6789/0 -- mon_subscribe({osdmap=0}) v3 -- 0x7f68a00b1670 con 0x7f68a0145720 2024-06-03T08:35:48.795 INFO:teuthology.misc.health.smithi088.stderr:2024-06-03T08:35:48.790+0000 7f68a55fb640 1 librados: init done 2024-06-03T08:35:48.795 INFO:teuthology.misc.health.smithi088.stderr:2024-06-03T08:35:48.791+0000 7f68a55fb640 10 start_mon_command cmd=[{"prefix": "get_command_descriptions"}] 2024-06-03T08:35:48.795 INFO:teuthology.misc.health.smithi088.stderr:2024-06-03T08:35:48.791+0000 7f68a55fb640 10 monclient: _send_command 1 [{"prefix": "get_command_descriptions"}] 2024-06-03T08:35:48.795 INFO:teuthology.misc.health.smithi088.stderr:2024-06-03T08:35:48.791+0000 7f68a55fb640 10 monclient: _send_mon_message to mon.b at v1:172.21.15.192:6789/0 2024-06-03T08:35:48.795 INFO:teuthology.misc.health.smithi088.stderr:2024-06-03T08:35:48.791+0000 7f68a55fb640 1 -- 172.21.15.88:0/374680176 --> v1:172.21.15.192:6789/0 -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f68a00a9a90 con 0x7f68a0145720 2024-06-03T08:35:48.815 INFO:teuthology.misc.health.smithi088.stderr:2024-06-03T08:35:48.814+0000 7f68877fe640 1 -- 172.21.15.88:0/374680176 <== mon.1 v1:172.21.15.192:6789/0 5 ==== config(0 keys) v1 ==== 4+0+0 (unknown 0 0 0) 0x7f68980014c0 con 0x7f68a0145720 2024-06-03T08:35:48.815 INFO:teuthology.misc.health.smithi088.stderr:2024-06-03T08:35:48.814+0000 7f68877fe640 10 monclient: handle_config config(0 keys) v1 2024-06-03T08:35:48.815 INFO:teuthology.misc.health.smithi088.stderr:2024-06-03T08:35:48.814+0000 7f68877fe640 1 -- 172.21.15.88:0/374680176 <== mon.1 v1:172.21.15.192:6789/0 6 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 2234374065 0 0) 0x7f6898003dc0 con 0x7f68a0145720 2024-06-03T08:35:48.815 INFO:teuthology.misc.health.smithi088.stderr:2024-06-03T08:35:48.814+0000 7f68877fe640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-06-03T08:35:48.815 INFO:teuthology.misc.health.smithi088.stderr:2024-06-03T08:35:48.814+0000 7f68877fe640 10 monclient: got monmap 1 from mon.b (according to old e1) 2024-06-03T08:35:48.815 INFO:teuthology.misc.health.smithi088.stderr:2024-06-03T08:35:48.814+0000 7f68877fe640 10 monclient: dump: 2024-06-03T08:35:48.815 INFO:teuthology.misc.health.smithi088.stderr:epoch 1 2024-06-03T08:35:48.815 INFO:teuthology.misc.health.smithi088.stderr:fsid ec3c3fdd-3ac6-4d7d-9412-683130ed8b04 2024-06-03T08:35:48.815 INFO:teuthology.misc.health.smithi088.stderr:last_changed 2024-06-03T08:34:31.885125+0000 2024-06-03T08:35:48.815 INFO:teuthology.misc.health.smithi088.stderr:created 2024-06-03T08:34:31.885125+0000 2024-06-03T08:35:48.815 INFO:teuthology.misc.health.smithi088.stderr:min_mon_release 18 (reef) 2024-06-03T08:35:48.816 INFO:teuthology.misc.health.smithi088.stderr:election_strategy: 3 2024-06-03T08:35:48.816 INFO:teuthology.misc.health.smithi088.stderr:0: v1:172.21.15.88:6789/0 mon.a 2024-06-03T08:35:48.816 INFO:teuthology.misc.health.smithi088.stderr:1: v1:172.21.15.192:6789/0 mon.b 2024-06-03T08:35:48.816 INFO:teuthology.misc.health.smithi088.stderr:2: v1:172.21.15.88:6790/0 mon.c 2024-06-03T08:35:48.816 INFO:teuthology.misc.health.smithi088.stderr: 2024-06-03T08:35:48.816 INFO:teuthology.misc.health.smithi088.stderr:2024-06-03T08:35:48.814+0000 7f689d7fa640 4 set_mon_vals no callback set 2024-06-03T08:35:48.816 INFO:teuthology.misc.health.smithi088.stderr:2024-06-03T08:35:48.815+0000 7f68877fe640 1 -- 172.21.15.88:0/374680176 <== mon.1 v1:172.21.15.192:6789/0 7 ==== mgrmap(e 4) v1 ==== 89054+0+0 (unknown 914547237 0 0) 0x7f6898019b10 con 0x7f68a0145720 2024-06-03T08:35:48.816 INFO:teuthology.misc.health.smithi088.stderr:2024-06-03T08:35:48.816+0000 7f689effd640 10 monclient: get_auth_request con 0x7f688c06d260 auth_method 0 2024-06-03T08:35:48.817 INFO:teuthology.misc.health.smithi088.stderr:2024-06-03T08:35:48.816+0000 7f68877fe640 10 client.4204.objecter ms_handle_connect 0x7f688c06d260 2024-06-03T08:35:48.819 INFO:teuthology.misc.health.smithi088.stderr:2024-06-03T08:35:48.819+0000 7f68877fe640 1 -- 172.21.15.88:0/374680176 <== mon.1 v1:172.21.15.192:6789/0 8 ==== osd_map(10..10 src has 1..10) v4 ==== 1645+0+0 (unknown 309997979 0 0) 0x7f6898053200 con 0x7f68a0145720 2024-06-03T08:35:48.819 INFO:teuthology.misc.health.smithi088.stderr:2024-06-03T08:35:48.819+0000 7f68877fe640 10 client.4204.objecter ms_dispatch 0x7f68a0000b90 osd_map(10..10 src has 1..10) v4 2024-06-03T08:35:48.819 INFO:teuthology.misc.health.smithi088.stderr:2024-06-03T08:35:48.819+0000 7f68877fe640 3 client.4204.objecter handle_osd_map got epochs [10,10] > 0 2024-06-03T08:35:48.819 INFO:teuthology.misc.health.smithi088.stderr:2024-06-03T08:35:48.819+0000 7f68877fe640 3 client.4204.objecter handle_osd_map decoding full epoch 10 2024-06-03T08:35:48.819 INFO:teuthology.misc.health.smithi088.stderr:2024-06-03T08:35:48.819+0000 7f68877fe640 20 client.4204.objecter dump_active .. 0 homeless 2024-06-03T08:35:48.965 INFO:tasks.ceph.osd.1.smithi088.stderr:==00:00:00:16.643 35067== Warning: unimplemented fcntl command: 1038 2024-06-03T08:35:49.153 INFO:tasks.ceph.osd.3.smithi088.stderr:==00:00:00:16.820 35068== Warning: unimplemented fcntl command: 1038 2024-06-03T08:35:49.169 INFO:teuthology.misc.health.smithi088.stderr:2024-06-03T08:35:49.168+0000 7f68877fe640 1 -- 172.21.15.88:0/374680176 <== mon.1 v1:172.21.15.192:6789/0 9 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+182498 (unknown 1092875540 0 3016760700) 0x7f689804ebe0 con 0x7f68a0145720 2024-06-03T08:35:49.170 INFO:teuthology.misc.health.smithi088.stderr:2024-06-03T08:35:49.168+0000 7f68877fe640 10 monclient: handle_mon_command_ack 1 [{"prefix": "get_command_descriptions"}] 2024-06-03T08:35:49.170 INFO:teuthology.misc.health.smithi088.stderr:2024-06-03T08:35:49.168+0000 7f68877fe640 10 monclient: _finish_command 1 = system:0 2024-06-03T08:35:49.385 INFO:teuthology.misc.health.smithi088.stderr:2024-06-03T08:35:49.384+0000 7f68a55fb640 10 start_mon_command cmd=[{"prefix": "osd dump", "format": "json"}] 2024-06-03T08:35:49.385 INFO:teuthology.misc.health.smithi088.stderr:2024-06-03T08:35:49.385+0000 7f68a55fb640 10 monclient: _send_command 2 [{"prefix": "osd dump", "format": "json"}] 2024-06-03T08:35:49.385 INFO:teuthology.misc.health.smithi088.stderr:2024-06-03T08:35:49.385+0000 7f68a55fb640 10 monclient: _send_mon_message to mon.b at v1:172.21.15.192:6789/0 2024-06-03T08:35:49.385 INFO:teuthology.misc.health.smithi088.stderr:2024-06-03T08:35:49.385+0000 7f68a55fb640 1 -- 172.21.15.88:0/374680176 --> v1:172.21.15.192:6789/0 -- mon_command({"prefix": "osd dump", "format": "json"} v 0) v1 -- 0x7f68a01d67e0 con 0x7f68a0145720 2024-06-03T08:35:49.406 INFO:teuthology.misc.health.smithi088.stderr:2024-06-03T08:35:49.405+0000 7f68877fe640 1 -- 172.21.15.88:0/374680176 <== mon.1 v1:172.21.15.192:6789/0 10 ==== mon_command_ack([{"prefix": "osd dump", "format": "json"}]=0 v10) v1 ==== 74+0+7057 (unknown 4040575075 0 1067785724) 0x7f6898050ce0 con 0x7f68a0145720 2024-06-03T08:35:49.406 INFO:teuthology.misc.health.smithi088.stderr:2024-06-03T08:35:49.405+0000 7f68877fe640 10 monclient: handle_mon_command_ack 2 [{"prefix": "osd dump", "format": "json"}] 2024-06-03T08:35:49.406 INFO:teuthology.misc.health.smithi088.stderr:2024-06-03T08:35:49.405+0000 7f68877fe640 10 monclient: _finish_command 2 = system:0 2024-06-03T08:35:49.406 INFO:teuthology.misc.health.smithi088.stdout: 2024-06-03T08:35:49.406 INFO:teuthology.misc.health.smithi088.stdout:{"epoch":10,"fsid":"ec3c3fdd-3ac6-4d7d-9412-683130ed8b04","created":"2024-06-03T08:35:26.799638+0000","modified":"2024-06-03T08:35:32.183068+0000","last_up_change":"0.000000","last_in_change":"2024-06-03T08:35:32.183068+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":2,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":0,"max_osd":8,"require_min_compat_client":"luminous","min_compat_client":"jewel","require_osd_release":"reef","allow_crimson":false,"pools":[],"osds":[{"osd":0,"uuid":"b25ab3bb-3587-45d0-991a-6d41bb529b37","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":1,"uuid":"00d160e7-9773-41d8-8d17-21e70620a0fd","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":2,"uuid":"addfb2a7-2bde-4cc3-98cf-17fd2bdfa3a9","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":3,"uuid":"6c5770b0-c93c-4101-8c53-bd761b8abcde","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":4,"uuid":"f001b3cb-2081-428b-bac3-b3df3c19b978","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":5,"uuid":"3cc263e9-e9bf-45e0-a3b7-83a76469eb70","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":6,"uuid":"82c83316-809f-4901-a5ed-34b6a075b393","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":7,"uuid":"aafe0716-1102-4eae-aac6-0f81bb154e7d","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":4,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":5,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":6,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":7,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[],"pg_upmap_primaries":[],"pg_temp":[],"primary_temp":[],"blocklist":{},"range_blocklist":{},"erasure_code_profiles":{"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","technique":"reed_sol_van"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2024-06-03T08:35:49.409 INFO:teuthology.misc.health.smithi088.stderr:2024-06-03T08:35:49.408+0000 7f68a55fb640 10 librados: watch_flush enter 2024-06-03T08:35:49.410 INFO:teuthology.misc.health.smithi088.stderr:2024-06-03T08:35:49.408+0000 7f68a55fb640 10 librados: watch_flush exit 2024-06-03T08:35:49.410 INFO:teuthology.misc.health.smithi088.stderr:2024-06-03T08:35:49.408+0000 7f68a55fb640 20 client.4204.objecter shutdown clearing up homeless session... 2024-06-03T08:35:49.410 INFO:teuthology.misc.health.smithi088.stderr:2024-06-03T08:35:49.408+0000 7f68a55fb640 10 client.4204.objecter successfully canceled tick 2024-06-03T08:35:49.410 INFO:teuthology.misc.health.smithi088.stderr:2024-06-03T08:35:49.409+0000 7f68a55fb640 1 -- 172.21.15.88:0/374680176 >> v1:172.21.15.192:6800/34229 conn(0x7f688c06d260 legacy=0x7f688c06f720 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-06-03T08:35:49.410 INFO:teuthology.misc.health.smithi088.stderr:2024-06-03T08:35:49.409+0000 7f68a55fb640 10 monclient: shutdown 2024-06-03T08:35:49.410 INFO:teuthology.misc.health.smithi088.stderr:2024-06-03T08:35:49.409+0000 7f68a55fb640 20 monclient: shutdown discarding 0 pending message(s) 2024-06-03T08:35:49.410 INFO:teuthology.misc.health.smithi088.stderr:2024-06-03T08:35:49.409+0000 7f68a55fb640 1 -- 172.21.15.88:0/374680176 >> v1:172.21.15.192:6789/0 conn(0x7f68a0145720 legacy=0x7f68a00b0d80 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-06-03T08:35:49.410 INFO:teuthology.misc.health.smithi088.stderr:2024-06-03T08:35:49.409+0000 7f68a55fb640 1 -- 172.21.15.88:0/374680176 shutdown_connections 2024-06-03T08:35:49.410 INFO:teuthology.misc.health.smithi088.stderr:2024-06-03T08:35:49.409+0000 7f68a55fb640 1 -- 172.21.15.88:0/374680176 >> 172.21.15.88:0/374680176 conn(0x7f68a00b7270 msgr2=0x7f68a013bb30 unknown :-1 s=STATE_NONE l=0).mark_down 2024-06-03T08:35:49.410 INFO:teuthology.misc.health.smithi088.stderr:2024-06-03T08:35:49.409+0000 7f68a55fb640 1 -- 172.21.15.88:0/374680176 shutdown_connections 2024-06-03T08:35:49.410 INFO:teuthology.misc.health.smithi088.stderr:2024-06-03T08:35:49.409+0000 7f68a55fb640 1 -- 172.21.15.88:0/374680176 wait complete. 2024-06-03T08:35:49.411 INFO:teuthology.misc.health.smithi088.stderr:2024-06-03T08:35:49.409+0000 7f68a55fb640 1 librados: shutdown 2024-06-03T08:35:49.420 DEBUG:teuthology.misc:0 of 8 OSDs are up 2024-06-03T08:35:49.447 INFO:tasks.ceph.osd.4.smithi192.stderr:==00:00:00:17.129 34785== Warning: unimplemented fcntl command: 1038 2024-06-03T08:35:49.534 INFO:tasks.ceph.osd.2.smithi088.stderr:==00:00:00:17.223 35065== Warning: unimplemented fcntl command: 1038 2024-06-03T08:35:49.749 INFO:tasks.ceph.osd.5.smithi192.stderr:==00:00:00:17.429 34786== Warning: unimplemented fcntl command: 1038 2024-06-03T08:35:49.769 INFO:tasks.ceph.osd.6.smithi192.stderr:==00:00:00:17.428 34788== Warning: unimplemented fcntl command: 1038 2024-06-03T08:35:54.506 INFO:tasks.ceph.osd.7.smithi192.stderr:==00:00:00:22.182 34787== Warning: unimplemented fcntl command: 1038 2024-06-03T08:35:54.513 INFO:tasks.ceph.osd.7.smithi192.stderr:==00:00:00:22.189 34787== Warning: unimplemented fcntl command: 1038 2024-06-03T08:35:54.630 INFO:tasks.ceph.osd.0.smithi088.stderr:==00:00:00:22.306 35066== Warning: unimplemented fcntl command: 1038 2024-06-03T08:35:54.634 INFO:tasks.ceph.osd.0.smithi088.stderr:==00:00:00:22.310 35066== Warning: unimplemented fcntl command: 1038 2024-06-03T08:35:54.942 INFO:tasks.ceph.osd.1.smithi088.stderr:==00:00:00:22.620 35067== Warning: unimplemented fcntl command: 1038 2024-06-03T08:35:54.945 INFO:tasks.ceph.osd.1.smithi088.stderr:==00:00:00:22.624 35067== Warning: unimplemented fcntl command: 1038 2024-06-03T08:35:54.962 INFO:tasks.ceph.osd.3.smithi088.stderr:==00:00:00:22.630 35068== Warning: unimplemented fcntl command: 1038 2024-06-03T08:35:54.966 INFO:tasks.ceph.osd.3.smithi088.stderr:==00:00:00:22.634 35068== Warning: unimplemented fcntl command: 1038 2024-06-03T08:35:55.336 INFO:tasks.ceph.osd.4.smithi192.stderr:==00:00:00:23.019 34785== Warning: unimplemented fcntl command: 1038 2024-06-03T08:35:55.340 INFO:tasks.ceph.osd.4.smithi192.stderr:==00:00:00:23.023 34785== Warning: unimplemented fcntl command: 1038 2024-06-03T08:35:55.527 INFO:tasks.ceph.osd.2.smithi088.stderr:==00:00:00:23.216 35065== Warning: unimplemented fcntl command: 1038 2024-06-03T08:35:55.533 INFO:tasks.ceph.osd.2.smithi088.stderr:==00:00:00:23.222 35065== Warning: unimplemented fcntl command: 1038 2024-06-03T08:35:55.601 INFO:tasks.ceph.osd.5.smithi192.stderr:==00:00:00:23.281 34786== Warning: unimplemented fcntl command: 1038 2024-06-03T08:35:55.604 INFO:tasks.ceph.osd.5.smithi192.stderr:==00:00:00:23.285 34786== Warning: unimplemented fcntl command: 1038 2024-06-03T08:35:55.635 INFO:tasks.ceph.osd.6.smithi192.stderr:==00:00:00:23.295 34788== Warning: unimplemented fcntl command: 1038 2024-06-03T08:35:55.639 INFO:tasks.ceph.osd.6.smithi192.stderr:==00:00:00:23.299 34788== Warning: unimplemented fcntl command: 1038 2024-06-03T08:35:56.227 DEBUG:teuthology.orchestra.run.smithi088:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph --cluster ceph osd dump --format=json 2024-06-03T08:35:56.355 INFO:teuthology.misc.health.smithi088.stderr:2024-06-03T08:35:56.355+0000 7f9e152e7640 10 monclient: get_monmap_and_config 2024-06-03T08:35:56.355 INFO:teuthology.misc.health.smithi088.stderr:2024-06-03T08:35:56.355+0000 7f9e152e7640 10 monclient: build_initial_monmap 2024-06-03T08:35:56.356 INFO:teuthology.misc.health.smithi088.stderr:2024-06-03T08:35:56.355+0000 7f9e152e7640 10 monclient: monmap: 2024-06-03T08:35:56.356 INFO:teuthology.misc.health.smithi088.stderr:epoch 0 2024-06-03T08:35:56.356 INFO:teuthology.misc.health.smithi088.stderr:fsid ec3c3fdd-3ac6-4d7d-9412-683130ed8b04 2024-06-03T08:35:56.356 INFO:teuthology.misc.health.smithi088.stderr:last_changed 2024-06-03T08:35:56.356314+0000 2024-06-03T08:35:56.356 INFO:teuthology.misc.health.smithi088.stderr:created 2024-06-03T08:35:56.356314+0000 2024-06-03T08:35:56.356 INFO:teuthology.misc.health.smithi088.stderr:min_mon_release 0 (unknown) 2024-06-03T08:35:56.356 INFO:teuthology.misc.health.smithi088.stderr:election_strategy: 3 2024-06-03T08:35:56.356 INFO:teuthology.misc.health.smithi088.stderr:0: v1:172.21.15.88:6789/0 mon.noname-a 2024-06-03T08:35:56.356 INFO:teuthology.misc.health.smithi088.stderr:1: v1:172.21.15.192:6789/0 mon.noname-c 2024-06-03T08:35:56.356 INFO:teuthology.misc.health.smithi088.stderr:2: v1:172.21.15.88:6790/0 mon.noname-b 2024-06-03T08:35:56.356 INFO:teuthology.misc.health.smithi088.stderr: 2024-06-03T08:35:56.358 INFO:teuthology.misc.health.smithi088.stderr:2024-06-03T08:35:56.355+0000 7f9e152e7640 1 Processor -- start 2024-06-03T08:35:56.358 INFO:teuthology.misc.health.smithi088.stderr:2024-06-03T08:35:56.355+0000 7f9e152e7640 1 -- start start 2024-06-03T08:35:56.358 INFO:teuthology.misc.health.smithi088.stderr:2024-06-03T08:35:56.355+0000 7f9e152e7640 10 monclient: init 2024-06-03T08:35:56.358 INFO:teuthology.misc.health.smithi088.stderr:2024-06-03T08:35:56.356+0000 7f9e152e7640 10 monclient: _reopen_session rank -1 2024-06-03T08:35:56.358 INFO:teuthology.misc.health.smithi088.stderr:2024-06-03T08:35:56.356+0000 7f9e152e7640 10 monclient: _add_conns ranks=[1,0,2] 2024-06-03T08:35:56.358 INFO:teuthology.misc.health.smithi088.stderr:2024-06-03T08:35:56.356+0000 7f9e152e7640 10 monclient(hunting): picked mon.noname-c con 0x7f9e100a8c60 addr v1:172.21.15.192:6789/0 2024-06-03T08:35:56.358 INFO:teuthology.misc.health.smithi088.stderr:2024-06-03T08:35:56.356+0000 7f9e152e7640 10 monclient(hunting): picked mon.noname-a con 0x7f9e100b1be0 addr v1:172.21.15.88:6789/0 2024-06-03T08:35:56.358 INFO:teuthology.misc.health.smithi088.stderr:2024-06-03T08:35:56.356+0000 7f9e152e7640 10 monclient(hunting): picked mon.noname-b con 0x7f9e1013eba0 addr v1:172.21.15.88:6790/0 2024-06-03T08:35:56.358 INFO:teuthology.misc.health.smithi088.stderr:2024-06-03T08:35:56.356+0000 7f9e152e7640 1 -- --> v1:172.21.15.88:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f9e100a8780 con 0x7f9e100b1be0 2024-06-03T08:35:56.359 INFO:teuthology.misc.health.smithi088.stderr:2024-06-03T08:35:56.356+0000 7f9e152e7640 1 -- --> v1:172.21.15.192:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f9e100aa630 con 0x7f9e100a8c60 2024-06-03T08:35:56.359 INFO:teuthology.misc.health.smithi088.stderr:2024-06-03T08:35:56.356+0000 7f9e152e7640 1 -- --> v1:172.21.15.88:6790/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f9e1013f0b0 con 0x7f9e1013eba0 2024-06-03T08:35:56.359 INFO:teuthology.misc.health.smithi088.stderr:2024-06-03T08:35:56.356+0000 7f9e152e7640 10 monclient(hunting): _renew_subs 2024-06-03T08:35:56.359 INFO:teuthology.misc.health.smithi088.stderr:2024-06-03T08:35:56.356+0000 7f9e152e7640 10 monclient(hunting): authenticate will time out at 1212.166807s 2024-06-03T08:35:56.359 INFO:teuthology.misc.health.smithi088.stderr:2024-06-03T08:35:56.357+0000 7f9e0f7fe640 1 --1- >> v1:172.21.15.88:6790/0 conn(0x7f9e1013eba0 0x7f9e1014ce90 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.88:6790/0 says I am v1:172.21.15.88:47708/0 (socket says 172.21.15.88:47708) 2024-06-03T08:35:56.359 INFO:teuthology.misc.health.smithi088.stderr:2024-06-03T08:35:56.357+0000 7f9e0f7fe640 1 -- 172.21.15.88:0/3831716783 learned_addr learned my addr 172.21.15.88:0/3831716783 (peer_addr_for_me v1:172.21.15.88:0/0) 2024-06-03T08:35:56.362 INFO:teuthology.misc.health.smithi088.stderr:2024-06-03T08:35:56.360+0000 7f9e0dffb640 1 -- 172.21.15.88:0/3831716783 <== mon.1 v1:172.21.15.192:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 4214971647 0 0) 0x7f9e100aa630 con 0x7f9e100a8c60 2024-06-03T08:35:56.362 INFO:teuthology.misc.health.smithi088.stderr:2024-06-03T08:35:56.360+0000 7f9e0dffb640 10 monclient(hunting): _negotiate 2024-06-03T08:35:56.362 INFO:teuthology.misc.health.smithi088.stderr:2024-06-03T08:35:56.360+0000 7f9e0dffb640 10 monclient(hunting): _init_auth method 2 2024-06-03T08:35:56.362 INFO:teuthology.misc.health.smithi088.stderr:2024-06-03T08:35:56.360+0000 7f9e0dffb640 10 monclient(hunting): _init_auth creating new auth 2024-06-03T08:35:56.362 INFO:teuthology.misc.health.smithi088.stderr:2024-06-03T08:35:56.360+0000 7f9e0dffb640 10 monclient(hunting): my global_id is 4207 2024-06-03T08:35:56.362 INFO:teuthology.misc.health.smithi088.stderr:2024-06-03T08:35:56.360+0000 7f9e0dffb640 1 -- 172.21.15.88:0/3831716783 --> v1:172.21.15.192:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f9df8003520 con 0x7f9e100a8c60 2024-06-03T08:35:56.362 INFO:teuthology.misc.health.smithi088.stderr:2024-06-03T08:35:56.361+0000 7f9e0dffb640 1 -- 172.21.15.88:0/3831716783 <== mon.2 v1:172.21.15.88:6790/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 4075715359 0 0) 0x7f9e1013f0b0 con 0x7f9e1013eba0 2024-06-03T08:35:56.362 INFO:teuthology.misc.health.smithi088.stderr:2024-06-03T08:35:56.361+0000 7f9e0dffb640 10 monclient(hunting): _negotiate 2024-06-03T08:35:56.362 INFO:teuthology.misc.health.smithi088.stderr:2024-06-03T08:35:56.361+0000 7f9e0dffb640 10 monclient(hunting): _init_auth method 2 2024-06-03T08:35:56.362 INFO:teuthology.misc.health.smithi088.stderr:2024-06-03T08:35:56.361+0000 7f9e0dffb640 10 monclient(hunting): _init_auth creating new auth 2024-06-03T08:35:56.362 INFO:teuthology.misc.health.smithi088.stderr:2024-06-03T08:35:56.361+0000 7f9e0dffb640 10 monclient(hunting): my global_id is 4220 2024-06-03T08:35:56.363 INFO:teuthology.misc.health.smithi088.stderr:2024-06-03T08:35:56.361+0000 7f9e0dffb640 1 -- 172.21.15.88:0/3831716783 --> v1:172.21.15.88:6790/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f9e100aa630 con 0x7f9e1013eba0 2024-06-03T08:35:56.363 INFO:teuthology.misc.health.smithi088.stderr:2024-06-03T08:35:56.361+0000 7f9e0dffb640 1 -- 172.21.15.88:0/3831716783 <== mon.0 v1:172.21.15.88:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 4246985559 0 0) 0x7f9e100a8780 con 0x7f9e100b1be0 2024-06-03T08:35:56.363 INFO:teuthology.misc.health.smithi088.stderr:2024-06-03T08:35:56.361+0000 7f9e0dffb640 10 monclient(hunting): _negotiate 2024-06-03T08:35:56.363 INFO:teuthology.misc.health.smithi088.stderr:2024-06-03T08:35:56.361+0000 7f9e0dffb640 10 monclient(hunting): _init_auth method 2 2024-06-03T08:35:56.363 INFO:teuthology.misc.health.smithi088.stderr:2024-06-03T08:35:56.361+0000 7f9e0dffb640 10 monclient(hunting): _init_auth creating new auth 2024-06-03T08:35:56.363 INFO:teuthology.misc.health.smithi088.stderr:2024-06-03T08:35:56.361+0000 7f9e0dffb640 10 monclient(hunting): my global_id is 4218 2024-06-03T08:35:56.363 INFO:teuthology.misc.health.smithi088.stderr:2024-06-03T08:35:56.361+0000 7f9e0dffb640 1 -- 172.21.15.88:0/3831716783 --> v1:172.21.15.88:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f9e1013f0b0 con 0x7f9e100b1be0 2024-06-03T08:35:56.366 INFO:teuthology.misc.health.smithi088.stderr:2024-06-03T08:35:56.365+0000 7f9e0dffb640 1 -- 172.21.15.88:0/3831716783 <== mon.0 v1:172.21.15.88:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 214+0+0 (unknown 2299616343 0 0) 0x7f9e1013f0b0 con 0x7f9e100b1be0 2024-06-03T08:35:56.367 INFO:teuthology.misc.health.smithi088.stderr:2024-06-03T08:35:56.365+0000 7f9e0dffb640 1 -- 172.21.15.88:0/3831716783 >> v1:172.21.15.88:6790/0 conn(0x7f9e1013eba0 legacy=0x7f9e1014ce90 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-06-03T08:35:56.367 INFO:teuthology.misc.health.smithi088.stderr:2024-06-03T08:35:56.365+0000 7f9e0dffb640 1 -- 172.21.15.88:0/3831716783 >> v1:172.21.15.192:6789/0 conn(0x7f9e100a8c60 legacy=0x7f9e10141e50 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-06-03T08:35:56.367 INFO:teuthology.misc.health.smithi088.stderr:2024-06-03T08:35:56.365+0000 7f9e0dffb640 10 monclient: _finish_hunting 0 2024-06-03T08:35:56.367 INFO:teuthology.misc.health.smithi088.stderr:2024-06-03T08:35:56.365+0000 7f9e0dffb640 1 monclient: found mon.noname-a 2024-06-03T08:35:56.367 INFO:teuthology.misc.health.smithi088.stderr:2024-06-03T08:35:56.365+0000 7f9e0dffb640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-06-03T08:35:56.367 INFO:teuthology.misc.health.smithi088.stderr:2024-06-03T08:35:56.365+0000 7f9e0dffb640 10 monclient: _send_mon_message to mon.noname-a at v1:172.21.15.88:6789/0 2024-06-03T08:35:56.367 INFO:teuthology.misc.health.smithi088.stderr:2024-06-03T08:35:56.365+0000 7f9e0dffb640 1 -- 172.21.15.88:0/3831716783 --> v1:172.21.15.88:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f9e10141930 con 0x7f9e100b1be0 2024-06-03T08:35:56.367 INFO:teuthology.misc.health.smithi088.stderr:2024-06-03T08:35:56.365+0000 7f9e0dffb640 10 monclient: _finish_auth 0 2024-06-03T08:35:56.367 INFO:teuthology.misc.health.smithi088.stderr:2024-06-03T08:35:56.365+0000 7f9e0dffb640 10 monclient: _check_auth_tickets 2024-06-03T08:35:56.367 INFO:teuthology.misc.health.smithi088.stderr:2024-06-03T08:35:56.365+0000 7f9e0dffb640 20 monclient: _check_auth_rotating not needed by client.admin 2024-06-03T08:35:56.367 INFO:teuthology.misc.health.smithi088.stderr:2024-06-03T08:35:56.365+0000 7f9e0dffb640 1 -- 172.21.15.88:0/3831716783 <== mon.0 v1:172.21.15.88:6789/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 2234374065 0 0) 0x7f9e040015b0 con 0x7f9e100b1be0 2024-06-03T08:35:56.367 INFO:teuthology.misc.health.smithi088.stderr:2024-06-03T08:35:56.365+0000 7f9e0dffb640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-06-03T08:35:56.367 INFO:teuthology.misc.health.smithi088.stderr:2024-06-03T08:35:56.365+0000 7f9e0dffb640 10 monclient: got monmap 1 from mon.noname-a (according to old e1) 2024-06-03T08:35:56.368 INFO:teuthology.misc.health.smithi088.stderr:2024-06-03T08:35:56.365+0000 7f9e0dffb640 10 monclient: dump: 2024-06-03T08:35:56.368 INFO:teuthology.misc.health.smithi088.stderr:epoch 1 2024-06-03T08:35:56.368 INFO:teuthology.misc.health.smithi088.stderr:fsid ec3c3fdd-3ac6-4d7d-9412-683130ed8b04 2024-06-03T08:35:56.368 INFO:teuthology.misc.health.smithi088.stderr:last_changed 2024-06-03T08:34:31.885125+0000 2024-06-03T08:35:56.368 INFO:teuthology.misc.health.smithi088.stderr:created 2024-06-03T08:34:31.885125+0000 2024-06-03T08:35:56.368 INFO:teuthology.misc.health.smithi088.stderr:min_mon_release 18 (reef) 2024-06-03T08:35:56.368 INFO:teuthology.misc.health.smithi088.stderr:election_strategy: 3 2024-06-03T08:35:56.368 INFO:teuthology.misc.health.smithi088.stderr:0: v1:172.21.15.88:6789/0 mon.a 2024-06-03T08:35:56.368 INFO:teuthology.misc.health.smithi088.stderr:1: v1:172.21.15.192:6789/0 mon.b 2024-06-03T08:35:56.368 INFO:teuthology.misc.health.smithi088.stderr:2: v1:172.21.15.88:6790/0 mon.c 2024-06-03T08:35:56.368 INFO:teuthology.misc.health.smithi088.stderr: 2024-06-03T08:35:56.368 INFO:teuthology.misc.health.smithi088.stderr:2024-06-03T08:35:56.365+0000 7f9e152e7640 5 monclient: authenticate success, global_id 4218 2024-06-03T08:35:56.368 INFO:teuthology.misc.health.smithi088.stderr:2024-06-03T08:35:56.365+0000 7f9e152e7640 20 monclient: get_monmap_and_config waiting for monmap|config 2024-06-03T08:35:56.370 INFO:teuthology.misc.health.smithi088.stderr:2024-06-03T08:35:56.368+0000 7f9e0dffb640 1 -- 172.21.15.88:0/3831716783 <== mon.0 v1:172.21.15.88:6789/0 4 ==== config(0 keys) v1 ==== 4+0+0 (unknown 0 0 0) 0x7f9e040017d0 con 0x7f9e100b1be0 2024-06-03T08:35:56.370 INFO:teuthology.misc.health.smithi088.stderr:2024-06-03T08:35:56.369+0000 7f9e0dffb640 10 monclient: handle_config config(0 keys) v1 2024-06-03T08:35:56.370 INFO:teuthology.misc.health.smithi088.stderr:2024-06-03T08:35:56.369+0000 7f9e0dffb640 1 -- 172.21.15.88:0/3831716783 <== mon.0 v1:172.21.15.88:6789/0 5 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 2234374065 0 0) 0x7f9e04001b60 con 0x7f9e100b1be0 2024-06-03T08:35:56.370 INFO:teuthology.misc.health.smithi088.stderr:2024-06-03T08:35:56.369+0000 7f9e0dffb640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-06-03T08:35:56.370 INFO:teuthology.misc.health.smithi088.stderr:2024-06-03T08:35:56.369+0000 7f9e0dffb640 10 monclient: got monmap 1 from mon.a (according to old e1) 2024-06-03T08:35:56.370 INFO:teuthology.misc.health.smithi088.stderr:2024-06-03T08:35:56.369+0000 7f9e0dffb640 10 monclient: dump: 2024-06-03T08:35:56.371 INFO:teuthology.misc.health.smithi088.stderr:epoch 1 2024-06-03T08:35:56.371 INFO:teuthology.misc.health.smithi088.stderr:fsid ec3c3fdd-3ac6-4d7d-9412-683130ed8b04 2024-06-03T08:35:56.371 INFO:teuthology.misc.health.smithi088.stderr:last_changed 2024-06-03T08:34:31.885125+0000 2024-06-03T08:35:56.371 INFO:teuthology.misc.health.smithi088.stderr:created 2024-06-03T08:34:31.885125+0000 2024-06-03T08:35:56.371 INFO:teuthology.misc.health.smithi088.stderr:min_mon_release 18 (reef) 2024-06-03T08:35:56.371 INFO:teuthology.misc.health.smithi088.stderr:election_strategy: 3 2024-06-03T08:35:56.371 INFO:teuthology.misc.health.smithi088.stderr:0: v1:172.21.15.88:6789/0 mon.a 2024-06-03T08:35:56.371 INFO:teuthology.misc.health.smithi088.stderr:1: v1:172.21.15.192:6789/0 mon.b 2024-06-03T08:35:56.371 INFO:teuthology.misc.health.smithi088.stderr:2: v1:172.21.15.88:6790/0 mon.c 2024-06-03T08:35:56.371 INFO:teuthology.misc.health.smithi088.stderr: 2024-06-03T08:35:56.371 INFO:teuthology.misc.health.smithi088.stderr:2024-06-03T08:35:56.369+0000 7f9e152e7640 10 monclient: get_monmap_and_config success 2024-06-03T08:35:56.371 INFO:teuthology.misc.health.smithi088.stderr:2024-06-03T08:35:56.369+0000 7f9e152e7640 4 set_mon_vals no callback set 2024-06-03T08:35:56.371 INFO:teuthology.misc.health.smithi088.stderr:2024-06-03T08:35:56.369+0000 7f9e152e7640 10 monclient: shutdown 2024-06-03T08:35:56.372 INFO:teuthology.misc.health.smithi088.stderr:2024-06-03T08:35:56.369+0000 7f9e152e7640 20 monclient: shutdown discarding 0 pending message(s) 2024-06-03T08:35:56.372 INFO:teuthology.misc.health.smithi088.stderr:2024-06-03T08:35:56.369+0000 7f9e152e7640 1 -- 172.21.15.88:0/3831716783 >> v1:172.21.15.88:6789/0 conn(0x7f9e100b1be0 legacy=0x7f9e1013e490 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-06-03T08:35:56.372 INFO:teuthology.misc.health.smithi088.stderr:2024-06-03T08:35:56.370+0000 7f9e152e7640 1 -- 172.21.15.88:0/3831716783 shutdown_connections 2024-06-03T08:35:56.372 INFO:teuthology.misc.health.smithi088.stderr:2024-06-03T08:35:56.370+0000 7f9e152e7640 1 -- 172.21.15.88:0/3831716783 >> 172.21.15.88:0/3831716783 conn(0x7f9e100b71e0 msgr2=0x7f9e100b75f0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-06-03T08:35:56.372 INFO:teuthology.misc.health.smithi088.stderr:2024-06-03T08:35:56.370+0000 7f9e152e7640 1 -- 172.21.15.88:0/3831716783 shutdown_connections 2024-06-03T08:35:56.372 INFO:teuthology.misc.health.smithi088.stderr:2024-06-03T08:35:56.370+0000 7f9e152e7640 1 -- 172.21.15.88:0/3831716783 wait complete. 2024-06-03T08:35:56.374 INFO:teuthology.misc.health.smithi088.stderr:2024-06-03T08:35:56.371+0000 7f9e152e7640 10 monclient: build_initial_monmap 2024-06-03T08:35:56.374 INFO:teuthology.misc.health.smithi088.stderr:2024-06-03T08:35:56.371+0000 7f9e152e7640 10 monclient: monmap: 2024-06-03T08:35:56.374 INFO:teuthology.misc.health.smithi088.stderr:epoch 0 2024-06-03T08:35:56.375 INFO:teuthology.misc.health.smithi088.stderr:fsid 00000000-0000-0000-0000-000000000000 2024-06-03T08:35:56.375 INFO:teuthology.misc.health.smithi088.stderr:last_changed 0.000000 2024-06-03T08:35:56.375 INFO:teuthology.misc.health.smithi088.stderr:created 0.000000 2024-06-03T08:35:56.375 INFO:teuthology.misc.health.smithi088.stderr:min_mon_release 0 (unknown) 2024-06-03T08:35:56.375 INFO:teuthology.misc.health.smithi088.stderr:election_strategy: 1 2024-06-03T08:35:56.375 INFO:teuthology.misc.health.smithi088.stderr:0: v1:172.21.15.88:6789/0 mon.noname-a 2024-06-03T08:35:56.375 INFO:teuthology.misc.health.smithi088.stderr:1: v1:172.21.15.192:6789/0 mon.noname-b 2024-06-03T08:35:56.375 INFO:teuthology.misc.health.smithi088.stderr:2: v1:172.21.15.88:6790/0 mon.noname-c 2024-06-03T08:35:56.375 INFO:teuthology.misc.health.smithi088.stderr: 2024-06-03T08:35:56.375 INFO:teuthology.misc.health.smithi088.stderr:2024-06-03T08:35:56.371+0000 7f9e152e7640 1 librados: starting msgr at 2024-06-03T08:35:56.375 INFO:teuthology.misc.health.smithi088.stderr:2024-06-03T08:35:56.371+0000 7f9e152e7640 1 librados: starting objecter 2024-06-03T08:35:56.375 INFO:teuthology.misc.health.smithi088.stderr:2024-06-03T08:35:56.371+0000 7f9e152e7640 1 Processor -- start 2024-06-03T08:35:56.375 INFO:teuthology.misc.health.smithi088.stderr:2024-06-03T08:35:56.371+0000 7f9e152e7640 1 -- start start 2024-06-03T08:35:56.376 INFO:teuthology.misc.health.smithi088.stderr:2024-06-03T08:35:56.371+0000 7f9e152e7640 1 librados: setting wanted keys 2024-06-03T08:35:56.376 INFO:teuthology.misc.health.smithi088.stderr:2024-06-03T08:35:56.371+0000 7f9e152e7640 1 librados: calling monclient init 2024-06-03T08:35:56.376 INFO:teuthology.misc.health.smithi088.stderr:2024-06-03T08:35:56.371+0000 7f9e152e7640 10 monclient: init 2024-06-03T08:35:56.376 INFO:teuthology.misc.health.smithi088.stderr:2024-06-03T08:35:56.372+0000 7f9e152e7640 10 monclient: _reopen_session rank -1 2024-06-03T08:35:56.376 INFO:teuthology.misc.health.smithi088.stderr:2024-06-03T08:35:56.372+0000 7f9e152e7640 10 monclient: _add_conns ranks=[2,1,0] 2024-06-03T08:35:56.376 INFO:teuthology.misc.health.smithi088.stderr:2024-06-03T08:35:56.372+0000 7f9e152e7640 10 monclient(hunting): picked mon.noname-c con 0x7f9e100a8c60 addr v1:172.21.15.88:6790/0 2024-06-03T08:35:56.376 INFO:teuthology.misc.health.smithi088.stderr:2024-06-03T08:35:56.372+0000 7f9e152e7640 10 monclient(hunting): picked mon.noname-b con 0x7f9e100b1be0 addr v1:172.21.15.192:6789/0 2024-06-03T08:35:56.376 INFO:teuthology.misc.health.smithi088.stderr:2024-06-03T08:35:56.372+0000 7f9e152e7640 10 monclient(hunting): picked mon.noname-a con 0x7f9e1013eba0 addr v1:172.21.15.88:6789/0 2024-06-03T08:35:56.376 INFO:teuthology.misc.health.smithi088.stderr:2024-06-03T08:35:56.372+0000 7f9e152e7640 1 -- --> v1:172.21.15.88:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f9e10144400 con 0x7f9e1013eba0 2024-06-03T08:35:56.376 INFO:teuthology.misc.health.smithi088.stderr:2024-06-03T08:35:56.373+0000 7f9e152e7640 1 -- --> v1:172.21.15.192:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f9e100b68b0 con 0x7f9e100b1be0 2024-06-03T08:35:56.376 INFO:teuthology.misc.health.smithi088.stderr:2024-06-03T08:35:56.373+0000 7f9e152e7640 1 -- --> v1:172.21.15.88:6790/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f9e10144fd0 con 0x7f9e100a8c60 2024-06-03T08:35:56.376 INFO:teuthology.misc.health.smithi088.stderr:2024-06-03T08:35:56.373+0000 7f9e152e7640 10 monclient(hunting): _renew_subs 2024-06-03T08:35:56.376 INFO:teuthology.misc.health.smithi088.stderr:2024-06-03T08:35:56.373+0000 7f9e152e7640 10 monclient(hunting): authenticate will time out at 1212.182975s 2024-06-03T08:35:56.377 INFO:teuthology.misc.health.smithi088.stderr:2024-06-03T08:35:56.373+0000 7f9e0effd640 1 --1- >> v1:172.21.15.88:6790/0 conn(0x7f9e100a8c60 0x7f9e10143b80 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=1).handle_server_banner_and_identify peer v1:172.21.15.88:6790/0 says I am v1:172.21.15.88:47724/0 (socket says 172.21.15.88:47724) 2024-06-03T08:35:56.377 INFO:teuthology.misc.health.smithi088.stderr:2024-06-03T08:35:56.373+0000 7f9e0e7fc640 1 --1- >> v1:172.21.15.192:6789/0 conn(0x7f9e100b1be0 0x7f9e101d6440 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=1).handle_server_banner_and_identify peer v1:172.21.15.192:6789/0 says I am v1:172.21.15.88:33346/0 (socket says 172.21.15.88:33346) 2024-06-03T08:35:56.377 INFO:teuthology.misc.health.smithi088.stderr:2024-06-03T08:35:56.373+0000 7f9e0e7fc640 1 -- 172.21.15.88:0/915799954 learned_addr learned my addr 172.21.15.88:0/915799954 (peer_addr_for_me v1:172.21.15.88:0/0) 2024-06-03T08:35:56.377 INFO:teuthology.misc.health.smithi088.stderr:2024-06-03T08:35:56.373+0000 7f9df77fe640 10 client.?.objecter ms_handle_connect 0x7f9e100b1be0 2024-06-03T08:35:56.377 INFO:teuthology.misc.health.smithi088.stderr:2024-06-03T08:35:56.373+0000 7f9df77fe640 10 client.?.objecter resend_mon_ops 2024-06-03T08:35:56.377 INFO:teuthology.misc.health.smithi088.stderr:2024-06-03T08:35:56.373+0000 7f9df77fe640 10 client.?.objecter ms_handle_connect 0x7f9e1013eba0 2024-06-03T08:35:56.377 INFO:teuthology.misc.health.smithi088.stderr:2024-06-03T08:35:56.373+0000 7f9df77fe640 10 client.?.objecter resend_mon_ops 2024-06-03T08:35:56.377 INFO:teuthology.misc.health.smithi088.stderr:2024-06-03T08:35:56.374+0000 7f9df77fe640 10 client.?.objecter ms_handle_connect 0x7f9e100a8c60 2024-06-03T08:35:56.377 INFO:teuthology.misc.health.smithi088.stderr:2024-06-03T08:35:56.374+0000 7f9df77fe640 10 client.?.objecter resend_mon_ops 2024-06-03T08:35:56.378 INFO:teuthology.misc.health.smithi088.stderr:2024-06-03T08:35:56.376+0000 7f9df77fe640 1 -- 172.21.15.88:0/915799954 <== mon.2 v1:172.21.15.88:6790/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 2542115292 0 0) 0x7f9e10144fd0 con 0x7f9e100a8c60 2024-06-03T08:35:56.378 INFO:teuthology.misc.health.smithi088.stderr:2024-06-03T08:35:56.376+0000 7f9df77fe640 10 monclient(hunting): _negotiate 2024-06-03T08:35:56.378 INFO:teuthology.misc.health.smithi088.stderr:2024-06-03T08:35:56.376+0000 7f9df77fe640 10 monclient(hunting): _init_auth method 2 2024-06-03T08:35:56.378 INFO:teuthology.misc.health.smithi088.stderr:2024-06-03T08:35:56.376+0000 7f9df77fe640 10 monclient(hunting): _init_auth creating new auth 2024-06-03T08:35:56.378 INFO:teuthology.misc.health.smithi088.stderr:2024-06-03T08:35:56.376+0000 7f9df77fe640 10 monclient(hunting): my global_id is 4223 2024-06-03T08:35:56.378 INFO:teuthology.misc.health.smithi088.stderr:2024-06-03T08:35:56.376+0000 7f9df77fe640 1 -- 172.21.15.88:0/915799954 --> v1:172.21.15.88:6790/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f9dfc004eb0 con 0x7f9e100a8c60 2024-06-03T08:35:56.378 INFO:teuthology.misc.health.smithi088.stderr:2024-06-03T08:35:56.377+0000 7f9df77fe640 1 -- 172.21.15.88:0/915799954 <== mon.0 v1:172.21.15.88:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 2931215638 0 0) 0x7f9e10144400 con 0x7f9e1013eba0 2024-06-03T08:35:56.378 INFO:teuthology.misc.health.smithi088.stderr:2024-06-03T08:35:56.377+0000 7f9df77fe640 10 monclient(hunting): _negotiate 2024-06-03T08:35:56.378 INFO:teuthology.misc.health.smithi088.stderr:2024-06-03T08:35:56.377+0000 7f9df77fe640 10 monclient(hunting): _init_auth method 2 2024-06-03T08:35:56.378 INFO:teuthology.misc.health.smithi088.stderr:2024-06-03T08:35:56.377+0000 7f9df77fe640 10 monclient(hunting): _init_auth creating new auth 2024-06-03T08:35:56.378 INFO:teuthology.misc.health.smithi088.stderr:2024-06-03T08:35:56.377+0000 7f9df77fe640 10 monclient(hunting): my global_id is 4221 2024-06-03T08:35:56.379 INFO:teuthology.misc.health.smithi088.stderr:2024-06-03T08:35:56.377+0000 7f9df77fe640 1 -- 172.21.15.88:0/915799954 --> v1:172.21.15.88:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f9e10144fd0 con 0x7f9e1013eba0 2024-06-03T08:35:56.385 INFO:teuthology.misc.health.smithi088.stderr:2024-06-03T08:35:56.384+0000 7f9df77fe640 1 -- 172.21.15.88:0/915799954 <== mon.0 v1:172.21.15.88:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 764+0+0 (unknown 2180273577 0 0) 0x7f9e10144fd0 con 0x7f9e1013eba0 2024-06-03T08:35:56.385 INFO:teuthology.misc.health.smithi088.stderr:2024-06-03T08:35:56.384+0000 7f9df77fe640 1 -- 172.21.15.88:0/915799954 --> v1:172.21.15.88:6789/0 -- auth(proto 2 165 bytes epoch 0) v1 -- 0x7f9e10144400 con 0x7f9e1013eba0 2024-06-03T08:35:56.385 INFO:teuthology.misc.health.smithi088.stderr:2024-06-03T08:35:56.384+0000 7f9df77fe640 1 -- 172.21.15.88:0/915799954 <== mon.0 v1:172.21.15.88:6789/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 2234374065 0 0) 0x7f9e08001c30 con 0x7f9e1013eba0 2024-06-03T08:35:56.385 INFO:teuthology.misc.health.smithi088.stderr:2024-06-03T08:35:56.384+0000 7f9df77fe640 10 monclient(hunting): handle_monmap mon_map magic: 0 v1 2024-06-03T08:35:56.385 INFO:teuthology.misc.health.smithi088.stderr:2024-06-03T08:35:56.384+0000 7f9df77fe640 10 monclient(hunting): got monmap 1 from mon.noname-a (according to old e1) 2024-06-03T08:35:56.385 INFO:teuthology.misc.health.smithi088.stderr:2024-06-03T08:35:56.384+0000 7f9df77fe640 10 monclient(hunting): dump: 2024-06-03T08:35:56.385 INFO:teuthology.misc.health.smithi088.stderr:epoch 1 2024-06-03T08:35:56.385 INFO:teuthology.misc.health.smithi088.stderr:fsid ec3c3fdd-3ac6-4d7d-9412-683130ed8b04 2024-06-03T08:35:56.385 INFO:teuthology.misc.health.smithi088.stderr:last_changed 2024-06-03T08:34:31.885125+0000 2024-06-03T08:35:56.385 INFO:teuthology.misc.health.smithi088.stderr:created 2024-06-03T08:34:31.885125+0000 2024-06-03T08:35:56.386 INFO:teuthology.misc.health.smithi088.stderr:min_mon_release 18 (reef) 2024-06-03T08:35:56.386 INFO:teuthology.misc.health.smithi088.stderr:election_strategy: 3 2024-06-03T08:35:56.386 INFO:teuthology.misc.health.smithi088.stderr:0: v1:172.21.15.88:6789/0 mon.a 2024-06-03T08:35:56.386 INFO:teuthology.misc.health.smithi088.stderr:1: v1:172.21.15.192:6789/0 mon.b 2024-06-03T08:35:56.386 INFO:teuthology.misc.health.smithi088.stderr:2: v1:172.21.15.88:6790/0 mon.c 2024-06-03T08:35:56.386 INFO:teuthology.misc.health.smithi088.stderr: 2024-06-03T08:35:56.386 INFO:teuthology.misc.health.smithi088.stderr:2024-06-03T08:35:56.384+0000 7f9df77fe640 10 monclient(hunting): _finish_auth 0 2024-06-03T08:35:56.387 INFO:teuthology.misc.health.smithi088.stderr:2024-06-03T08:35:56.387+0000 7f9df77fe640 1 -- 172.21.15.88:0/915799954 <== mon.2 v1:172.21.15.88:6790/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 764+0+0 (unknown 2499932295 0 0) 0x7f9dfc004eb0 con 0x7f9e100a8c60 2024-06-03T08:35:56.387 INFO:teuthology.misc.health.smithi088.stderr:2024-06-03T08:35:56.387+0000 7f9df77fe640 1 -- 172.21.15.88:0/915799954 --> v1:172.21.15.88:6790/0 -- auth(proto 2 165 bytes epoch 0) v1 -- 0x7f9e10144fd0 con 0x7f9e100a8c60 2024-06-03T08:35:56.387 INFO:teuthology.misc.health.smithi088.stderr:2024-06-03T08:35:56.387+0000 7f9df77fe640 1 -- 172.21.15.88:0/915799954 <== mon.2 v1:172.21.15.88:6790/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 2234374065 0 0) 0x7f9e00001830 con 0x7f9e100a8c60 2024-06-03T08:35:56.387 INFO:teuthology.misc.health.smithi088.stderr:2024-06-03T08:35:56.387+0000 7f9df77fe640 10 monclient(hunting): handle_monmap mon_map magic: 0 v1 2024-06-03T08:35:56.387 INFO:teuthology.misc.health.smithi088.stderr:2024-06-03T08:35:56.387+0000 7f9df77fe640 10 monclient(hunting): got monmap 1 from mon.c (according to old e1) 2024-06-03T08:35:56.387 INFO:teuthology.misc.health.smithi088.stderr:2024-06-03T08:35:56.387+0000 7f9df77fe640 10 monclient(hunting): dump: 2024-06-03T08:35:56.387 INFO:teuthology.misc.health.smithi088.stderr:epoch 1 2024-06-03T08:35:56.388 INFO:teuthology.misc.health.smithi088.stderr:fsid ec3c3fdd-3ac6-4d7d-9412-683130ed8b04 2024-06-03T08:35:56.388 INFO:teuthology.misc.health.smithi088.stderr:last_changed 2024-06-03T08:34:31.885125+0000 2024-06-03T08:35:56.388 INFO:teuthology.misc.health.smithi088.stderr:created 2024-06-03T08:34:31.885125+0000 2024-06-03T08:35:56.388 INFO:teuthology.misc.health.smithi088.stderr:min_mon_release 18 (reef) 2024-06-03T08:35:56.388 INFO:teuthology.misc.health.smithi088.stderr:election_strategy: 3 2024-06-03T08:35:56.388 INFO:teuthology.misc.health.smithi088.stderr:0: v1:172.21.15.88:6789/0 mon.a 2024-06-03T08:35:56.388 INFO:teuthology.misc.health.smithi088.stderr:1: v1:172.21.15.192:6789/0 mon.b 2024-06-03T08:35:56.388 INFO:teuthology.misc.health.smithi088.stderr:2: v1:172.21.15.88:6790/0 mon.c 2024-06-03T08:35:56.388 INFO:teuthology.misc.health.smithi088.stderr: 2024-06-03T08:35:56.393 INFO:teuthology.misc.health.smithi088.stderr:2024-06-03T08:35:56.391+0000 7f9df77fe640 1 -- 172.21.15.88:0/915799954 <== mon.0 v1:172.21.15.88:6789/0 4 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 580+0+0 (unknown 3890646854 0 0) 0x7f9e10144400 con 0x7f9e1013eba0 2024-06-03T08:35:56.393 INFO:teuthology.misc.health.smithi088.stderr:2024-06-03T08:35:56.391+0000 7f9df77fe640 1 -- 172.21.15.88:0/915799954 >> v1:172.21.15.88:6790/0 conn(0x7f9e100a8c60 legacy=0x7f9e10143b80 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-06-03T08:35:56.393 INFO:teuthology.misc.health.smithi088.stderr:2024-06-03T08:35:56.391+0000 7f9df77fe640 1 -- 172.21.15.88:0/915799954 >> v1:172.21.15.192:6789/0 conn(0x7f9e100b1be0 legacy=0x7f9e101d6440 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-06-03T08:35:56.393 INFO:teuthology.misc.health.smithi088.stderr:2024-06-03T08:35:56.391+0000 7f9df77fe640 10 monclient: _finish_hunting 0 2024-06-03T08:35:56.393 INFO:teuthology.misc.health.smithi088.stderr:2024-06-03T08:35:56.391+0000 7f9df77fe640 1 monclient: found mon.a 2024-06-03T08:35:56.393 INFO:teuthology.misc.health.smithi088.stderr:2024-06-03T08:35:56.391+0000 7f9df77fe640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-06-03T08:35:56.393 INFO:teuthology.misc.health.smithi088.stderr:2024-06-03T08:35:56.391+0000 7f9df77fe640 10 monclient: _send_mon_message to mon.a at v1:172.21.15.88:6789/0 2024-06-03T08:35:56.393 INFO:teuthology.misc.health.smithi088.stderr:2024-06-03T08:35:56.391+0000 7f9df77fe640 1 -- 172.21.15.88:0/915799954 --> v1:172.21.15.88:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f9e100b6090 con 0x7f9e1013eba0 2024-06-03T08:35:56.394 INFO:teuthology.misc.health.smithi088.stderr:2024-06-03T08:35:56.391+0000 7f9df77fe640 10 monclient: _finish_auth 0 2024-06-03T08:35:56.394 INFO:teuthology.misc.health.smithi088.stderr:2024-06-03T08:35:56.391+0000 7f9df77fe640 10 monclient: _check_auth_tickets 2024-06-03T08:35:56.394 INFO:teuthology.misc.health.smithi088.stderr:2024-06-03T08:35:56.391+0000 7f9df77fe640 20 monclient: _check_auth_rotating not needed by client.admin 2024-06-03T08:35:56.394 INFO:teuthology.misc.health.smithi088.stderr:2024-06-03T08:35:56.391+0000 7f9e152e7640 5 monclient: authenticate success, global_id 4221 2024-06-03T08:35:56.394 INFO:teuthology.misc.health.smithi088.stderr:2024-06-03T08:35:56.391+0000 7f9e152e7640 10 monclient: _renew_subs 2024-06-03T08:35:56.394 INFO:teuthology.misc.health.smithi088.stderr:2024-06-03T08:35:56.391+0000 7f9e152e7640 10 monclient: _send_mon_message to mon.a at v1:172.21.15.88:6789/0 2024-06-03T08:35:56.394 INFO:teuthology.misc.health.smithi088.stderr:2024-06-03T08:35:56.391+0000 7f9e152e7640 1 -- 172.21.15.88:0/915799954 --> v1:172.21.15.88:6789/0 -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f9e100b62e0 con 0x7f9e1013eba0 2024-06-03T08:35:56.394 INFO:teuthology.misc.health.smithi088.stderr:2024-06-03T08:35:56.391+0000 7f9e152e7640 10 client.4221.objecter _maybe_request_map subscribing (onetime) to next osd map 2024-06-03T08:35:56.394 INFO:teuthology.misc.health.smithi088.stderr:2024-06-03T08:35:56.391+0000 7f9e152e7640 10 monclient: _renew_subs 2024-06-03T08:35:56.394 INFO:teuthology.misc.health.smithi088.stderr:2024-06-03T08:35:56.391+0000 7f9e152e7640 10 monclient: _send_mon_message to mon.a at v1:172.21.15.88:6789/0 2024-06-03T08:35:56.394 INFO:teuthology.misc.health.smithi088.stderr:2024-06-03T08:35:56.391+0000 7f9e152e7640 1 -- 172.21.15.88:0/915799954 --> v1:172.21.15.88:6789/0 -- mon_subscribe({osdmap=0}) v3 -- 0x7f9e101d6d30 con 0x7f9e1013eba0 2024-06-03T08:35:56.394 INFO:teuthology.misc.health.smithi088.stderr:2024-06-03T08:35:56.391+0000 7f9e152e7640 1 librados: init done 2024-06-03T08:35:56.394 INFO:teuthology.misc.health.smithi088.stderr:2024-06-03T08:35:56.392+0000 7f9e152e7640 10 start_mon_command cmd=[{"prefix": "get_command_descriptions"}] 2024-06-03T08:35:56.395 INFO:teuthology.misc.health.smithi088.stderr:2024-06-03T08:35:56.392+0000 7f9e152e7640 10 monclient: _send_command 1 [{"prefix": "get_command_descriptions"}] 2024-06-03T08:35:56.395 INFO:teuthology.misc.health.smithi088.stderr:2024-06-03T08:35:56.392+0000 7f9e152e7640 10 monclient: _send_mon_message to mon.a at v1:172.21.15.88:6789/0 2024-06-03T08:35:56.395 INFO:teuthology.misc.health.smithi088.stderr:2024-06-03T08:35:56.392+0000 7f9e152e7640 1 -- 172.21.15.88:0/915799954 --> v1:172.21.15.88:6789/0 -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f9e101d6f80 con 0x7f9e1013eba0 2024-06-03T08:35:56.455 INFO:teuthology.misc.health.smithi088.stderr:2024-06-03T08:35:56.455+0000 7f9df77fe640 1 -- 172.21.15.88:0/915799954 <== mon.0 v1:172.21.15.88:6789/0 5 ==== config(0 keys) v1 ==== 4+0+0 (unknown 0 0 0) 0x7f9e08004b60 con 0x7f9e1013eba0 2024-06-03T08:35:56.455 INFO:teuthology.misc.health.smithi088.stderr:2024-06-03T08:35:56.455+0000 7f9df77fe640 10 monclient: handle_config config(0 keys) v1 2024-06-03T08:35:56.455 INFO:teuthology.misc.health.smithi088.stderr:2024-06-03T08:35:56.455+0000 7f9df77fe640 1 -- 172.21.15.88:0/915799954 <== mon.0 v1:172.21.15.88:6789/0 6 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 2234374065 0 0) 0x7f9e08002100 con 0x7f9e1013eba0 2024-06-03T08:35:56.455 INFO:teuthology.misc.health.smithi088.stderr:2024-06-03T08:35:56.455+0000 7f9df77fe640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-06-03T08:35:56.456 INFO:teuthology.misc.health.smithi088.stderr:2024-06-03T08:35:56.455+0000 7f9df77fe640 10 monclient: got monmap 1 from mon.a (according to old e1) 2024-06-03T08:35:56.456 INFO:teuthology.misc.health.smithi088.stderr:2024-06-03T08:35:56.455+0000 7f9df77fe640 10 monclient: dump: 2024-06-03T08:35:56.456 INFO:teuthology.misc.health.smithi088.stderr:epoch 1 2024-06-03T08:35:56.456 INFO:teuthology.misc.health.smithi088.stderr:fsid ec3c3fdd-3ac6-4d7d-9412-683130ed8b04 2024-06-03T08:35:56.456 INFO:teuthology.misc.health.smithi088.stderr:last_changed 2024-06-03T08:34:31.885125+0000 2024-06-03T08:35:56.456 INFO:teuthology.misc.health.smithi088.stderr:created 2024-06-03T08:34:31.885125+0000 2024-06-03T08:35:56.456 INFO:teuthology.misc.health.smithi088.stderr:min_mon_release 18 (reef) 2024-06-03T08:35:56.456 INFO:teuthology.misc.health.smithi088.stderr:election_strategy: 3 2024-06-03T08:35:56.456 INFO:teuthology.misc.health.smithi088.stderr:0: v1:172.21.15.88:6789/0 mon.a 2024-06-03T08:35:56.456 INFO:teuthology.misc.health.smithi088.stderr:1: v1:172.21.15.192:6789/0 mon.b 2024-06-03T08:35:56.456 INFO:teuthology.misc.health.smithi088.stderr:2: v1:172.21.15.88:6790/0 mon.c 2024-06-03T08:35:56.456 INFO:teuthology.misc.health.smithi088.stderr: 2024-06-03T08:35:56.456 INFO:teuthology.misc.health.smithi088.stderr:2024-06-03T08:35:56.455+0000 7f9e0d7fa640 4 set_mon_vals no callback set 2024-06-03T08:35:56.457 INFO:teuthology.misc.health.smithi088.stderr:2024-06-03T08:35:56.456+0000 7f9df77fe640 1 -- 172.21.15.88:0/915799954 <== mon.0 v1:172.21.15.88:6789/0 7 ==== mgrmap(e 4) v1 ==== 89054+0+0 (unknown 914547237 0 0) 0x7f9e08004ef0 con 0x7f9e1013eba0 2024-06-03T08:35:56.457 INFO:teuthology.misc.health.smithi088.stderr:2024-06-03T08:35:56.457+0000 7f9e0effd640 10 monclient: get_auth_request con 0x7f9dfc06d340 auth_method 0 2024-06-03T08:35:56.457 INFO:teuthology.misc.health.smithi088.stderr:2024-06-03T08:35:56.457+0000 7f9df77fe640 10 client.4221.objecter ms_handle_connect 0x7f9dfc06d340 2024-06-03T08:35:56.462 INFO:teuthology.misc.health.smithi088.stderr:2024-06-03T08:35:56.461+0000 7f9df77fe640 1 -- 172.21.15.88:0/915799954 <== mon.0 v1:172.21.15.88:6789/0 8 ==== osd_map(10..10 src has 1..10) v4 ==== 1645+0+0 (unknown 309997979 0 0) 0x7f9e08054370 con 0x7f9e1013eba0 2024-06-03T08:35:56.462 INFO:teuthology.misc.health.smithi088.stderr:2024-06-03T08:35:56.461+0000 7f9df77fe640 10 client.4221.objecter ms_dispatch 0x7f9e10000b90 osd_map(10..10 src has 1..10) v4 2024-06-03T08:35:56.462 INFO:teuthology.misc.health.smithi088.stderr:2024-06-03T08:35:56.461+0000 7f9df77fe640 3 client.4221.objecter handle_osd_map got epochs [10,10] > 0 2024-06-03T08:35:56.462 INFO:teuthology.misc.health.smithi088.stderr:2024-06-03T08:35:56.461+0000 7f9df77fe640 3 client.4221.objecter handle_osd_map decoding full epoch 10 2024-06-03T08:35:56.462 INFO:teuthology.misc.health.smithi088.stderr:2024-06-03T08:35:56.461+0000 7f9df77fe640 20 client.4221.objecter dump_active .. 0 homeless 2024-06-03T08:35:56.838 INFO:teuthology.misc.health.smithi088.stderr:2024-06-03T08:35:56.837+0000 7f9df77fe640 1 -- 172.21.15.88:0/915799954 <== mon.0 v1:172.21.15.88:6789/0 9 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+182498 (unknown 1092875540 0 3016760700) 0x7f9e0804fd50 con 0x7f9e1013eba0 2024-06-03T08:35:56.838 INFO:teuthology.misc.health.smithi088.stderr:2024-06-03T08:35:56.837+0000 7f9df77fe640 10 monclient: handle_mon_command_ack 1 [{"prefix": "get_command_descriptions"}] 2024-06-03T08:35:56.838 INFO:teuthology.misc.health.smithi088.stderr:2024-06-03T08:35:56.837+0000 7f9df77fe640 10 monclient: _finish_command 1 = system:0 2024-06-03T08:35:57.171 INFO:teuthology.misc.health.smithi088.stderr:2024-06-03T08:35:57.171+0000 7f9e152e7640 10 start_mon_command cmd=[{"prefix": "osd dump", "format": "json"}] 2024-06-03T08:35:57.172 INFO:teuthology.misc.health.smithi088.stderr:2024-06-03T08:35:57.171+0000 7f9e152e7640 10 monclient: _send_command 2 [{"prefix": "osd dump", "format": "json"}] 2024-06-03T08:35:57.172 INFO:teuthology.misc.health.smithi088.stderr:2024-06-03T08:35:57.171+0000 7f9e152e7640 10 monclient: _send_mon_message to mon.a at v1:172.21.15.88:6789/0 2024-06-03T08:35:57.172 INFO:teuthology.misc.health.smithi088.stderr:2024-06-03T08:35:57.171+0000 7f9e152e7640 1 -- 172.21.15.88:0/915799954 --> v1:172.21.15.88:6789/0 -- mon_command({"prefix": "osd dump", "format": "json"} v 0) v1 -- 0x7f9e10144870 con 0x7f9e1013eba0 2024-06-03T08:35:57.187 INFO:teuthology.misc.health.smithi088.stderr:2024-06-03T08:35:57.187+0000 7f9df77fe640 1 -- 172.21.15.88:0/915799954 <== mon.0 v1:172.21.15.88:6789/0 10 ==== mon_command_ack([{"prefix": "osd dump", "format": "json"}]=0 v10) v1 ==== 74+0+7057 (unknown 4040575075 0 1067785724) 0x7f9e10144870 con 0x7f9e1013eba0 2024-06-03T08:35:57.187 INFO:teuthology.misc.health.smithi088.stderr:2024-06-03T08:35:57.187+0000 7f9df77fe640 10 monclient: handle_mon_command_ack 2 [{"prefix": "osd dump", "format": "json"}] 2024-06-03T08:35:57.187 INFO:teuthology.misc.health.smithi088.stderr:2024-06-03T08:35:57.187+0000 7f9df77fe640 10 monclient: _finish_command 2 = system:0 2024-06-03T08:35:57.188 INFO:teuthology.misc.health.smithi088.stdout: 2024-06-03T08:35:57.188 INFO:teuthology.misc.health.smithi088.stdout:{"epoch":10,"fsid":"ec3c3fdd-3ac6-4d7d-9412-683130ed8b04","created":"2024-06-03T08:35:26.799638+0000","modified":"2024-06-03T08:35:32.183068+0000","last_up_change":"0.000000","last_in_change":"2024-06-03T08:35:32.183068+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":2,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":0,"max_osd":8,"require_min_compat_client":"luminous","min_compat_client":"jewel","require_osd_release":"reef","allow_crimson":false,"pools":[],"osds":[{"osd":0,"uuid":"b25ab3bb-3587-45d0-991a-6d41bb529b37","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":1,"uuid":"00d160e7-9773-41d8-8d17-21e70620a0fd","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":2,"uuid":"addfb2a7-2bde-4cc3-98cf-17fd2bdfa3a9","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":3,"uuid":"6c5770b0-c93c-4101-8c53-bd761b8abcde","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":4,"uuid":"f001b3cb-2081-428b-bac3-b3df3c19b978","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":5,"uuid":"3cc263e9-e9bf-45e0-a3b7-83a76469eb70","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":6,"uuid":"82c83316-809f-4901-a5ed-34b6a075b393","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":7,"uuid":"aafe0716-1102-4eae-aac6-0f81bb154e7d","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":4,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":5,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":6,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":7,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[],"pg_upmap_primaries":[],"pg_temp":[],"primary_temp":[],"blocklist":{},"range_blocklist":{},"erasure_code_profiles":{"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","technique":"reed_sol_van"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2024-06-03T08:35:57.190 INFO:teuthology.misc.health.smithi088.stderr:2024-06-03T08:35:57.189+0000 7f9e152e7640 10 librados: watch_flush enter 2024-06-03T08:35:57.190 INFO:teuthology.misc.health.smithi088.stderr:2024-06-03T08:35:57.189+0000 7f9e152e7640 10 librados: watch_flush exit 2024-06-03T08:35:57.190 INFO:teuthology.misc.health.smithi088.stderr:2024-06-03T08:35:57.189+0000 7f9e152e7640 20 client.4221.objecter shutdown clearing up homeless session... 2024-06-03T08:35:57.190 INFO:teuthology.misc.health.smithi088.stderr:2024-06-03T08:35:57.189+0000 7f9e152e7640 10 client.4221.objecter successfully canceled tick 2024-06-03T08:35:57.190 INFO:teuthology.misc.health.smithi088.stderr:2024-06-03T08:35:57.189+0000 7f9e152e7640 1 -- 172.21.15.88:0/915799954 >> v1:172.21.15.192:6800/34229 conn(0x7f9dfc06d340 legacy=0x7f9dfc06f800 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-06-03T08:35:57.190 INFO:teuthology.misc.health.smithi088.stderr:2024-06-03T08:35:57.189+0000 7f9e152e7640 10 monclient: shutdown 2024-06-03T08:35:57.190 INFO:teuthology.misc.health.smithi088.stderr:2024-06-03T08:35:57.189+0000 7f9e152e7640 20 monclient: shutdown discarding 0 pending message(s) 2024-06-03T08:35:57.190 INFO:teuthology.misc.health.smithi088.stderr:2024-06-03T08:35:57.189+0000 7f9e152e7640 1 -- 172.21.15.88:0/915799954 >> v1:172.21.15.88:6789/0 conn(0x7f9e1013eba0 legacy=0x7f9e101e0f70 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-06-03T08:35:57.190 INFO:teuthology.misc.health.smithi088.stderr:2024-06-03T08:35:57.189+0000 7f9e152e7640 1 -- 172.21.15.88:0/915799954 shutdown_connections 2024-06-03T08:35:57.190 INFO:teuthology.misc.health.smithi088.stderr:2024-06-03T08:35:57.189+0000 7f9e152e7640 1 -- 172.21.15.88:0/915799954 >> 172.21.15.88:0/915799954 conn(0x7f9e100b71e0 msgr2=0x7f9e100b75c0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-06-03T08:35:57.191 INFO:teuthology.misc.health.smithi088.stderr:2024-06-03T08:35:57.191+0000 7f9e152e7640 1 -- 172.21.15.88:0/915799954 shutdown_connections 2024-06-03T08:35:57.191 INFO:teuthology.misc.health.smithi088.stderr:2024-06-03T08:35:57.191+0000 7f9e152e7640 1 -- 172.21.15.88:0/915799954 wait complete. 2024-06-03T08:35:57.191 INFO:teuthology.misc.health.smithi088.stderr:2024-06-03T08:35:57.191+0000 7f9e152e7640 1 librados: shutdown 2024-06-03T08:35:57.201 DEBUG:teuthology.misc:0 of 8 OSDs are up 2024-06-03T08:35:58.904 INFO:tasks.ceph.osd.7.smithi192.stderr:==00:00:00:26.579 34787== Warning: unimplemented fcntl command: 1038 2024-06-03T08:35:59.120 INFO:tasks.ceph.osd.0.smithi088.stderr:==00:00:00:26.796 35066== Warning: unimplemented fcntl command: 1038 2024-06-03T08:35:59.383 INFO:tasks.ceph.osd.3.smithi088.stderr:==00:00:00:27.050 35068== Warning: unimplemented fcntl command: 1038 2024-06-03T08:35:59.384 INFO:tasks.ceph.osd.1.smithi088.stderr:==00:00:00:27.063 35067== Warning: unimplemented fcntl command: 1038 2024-06-03T08:35:59.788 INFO:tasks.ceph.osd.4.smithi192.stderr:==00:00:00:27.471 34785== Warning: unimplemented fcntl command: 1038 2024-06-03T08:35:59.981 INFO:tasks.ceph.osd.2.smithi088.stderr:==00:00:00:27.670 35065== Warning: unimplemented fcntl command: 1038 2024-06-03T08:36:00.059 INFO:tasks.ceph.osd.5.smithi192.stderr:==00:00:00:27.740 34786== Warning: unimplemented fcntl command: 1038 2024-06-03T08:36:00.613 INFO:tasks.ceph.osd.6.smithi192.stderr:==00:00:00:28.272 34788== Warning: unimplemented fcntl command: 1038 2024-06-03T08:36:04.008 DEBUG:teuthology.orchestra.run.smithi088:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph --cluster ceph osd dump --format=json 2024-06-03T08:36:04.155 INFO:teuthology.misc.health.smithi088.stderr:2024-06-03T08:36:04.155+0000 7f3027beb640 10 monclient: get_monmap_and_config 2024-06-03T08:36:04.156 INFO:teuthology.misc.health.smithi088.stderr:2024-06-03T08:36:04.155+0000 7f3027beb640 10 monclient: build_initial_monmap 2024-06-03T08:36:04.156 INFO:teuthology.misc.health.smithi088.stderr:2024-06-03T08:36:04.155+0000 7f3027beb640 10 monclient: monmap: 2024-06-03T08:36:04.156 INFO:teuthology.misc.health.smithi088.stderr:epoch 0 2024-06-03T08:36:04.156 INFO:teuthology.misc.health.smithi088.stderr:fsid ec3c3fdd-3ac6-4d7d-9412-683130ed8b04 2024-06-03T08:36:04.156 INFO:teuthology.misc.health.smithi088.stderr:last_changed 2024-06-03T08:36:04.156374+0000 2024-06-03T08:36:04.156 INFO:teuthology.misc.health.smithi088.stderr:created 2024-06-03T08:36:04.156374+0000 2024-06-03T08:36:04.156 INFO:teuthology.misc.health.smithi088.stderr:min_mon_release 0 (unknown) 2024-06-03T08:36:04.156 INFO:teuthology.misc.health.smithi088.stderr:election_strategy: 3 2024-06-03T08:36:04.156 INFO:teuthology.misc.health.smithi088.stderr:0: v1:172.21.15.88:6789/0 mon.noname-a 2024-06-03T08:36:04.156 INFO:teuthology.misc.health.smithi088.stderr:1: v1:172.21.15.192:6789/0 mon.noname-c 2024-06-03T08:36:04.156 INFO:teuthology.misc.health.smithi088.stderr:2: v1:172.21.15.88:6790/0 mon.noname-b 2024-06-03T08:36:04.157 INFO:teuthology.misc.health.smithi088.stderr: 2024-06-03T08:36:04.158 INFO:teuthology.misc.health.smithi088.stderr:2024-06-03T08:36:04.156+0000 7f3027beb640 1 Processor -- start 2024-06-03T08:36:04.158 INFO:teuthology.misc.health.smithi088.stderr:2024-06-03T08:36:04.156+0000 7f3027beb640 1 -- start start 2024-06-03T08:36:04.159 INFO:teuthology.misc.health.smithi088.stderr:2024-06-03T08:36:04.156+0000 7f3027beb640 10 monclient: init 2024-06-03T08:36:04.159 INFO:teuthology.misc.health.smithi088.stderr:2024-06-03T08:36:04.157+0000 7f3027beb640 10 monclient: _reopen_session rank -1 2024-06-03T08:36:04.159 INFO:teuthology.misc.health.smithi088.stderr:2024-06-03T08:36:04.157+0000 7f3027beb640 10 monclient: _add_conns ranks=[2,0,1] 2024-06-03T08:36:04.159 INFO:teuthology.misc.health.smithi088.stderr:2024-06-03T08:36:04.157+0000 7f3027beb640 10 monclient(hunting): picked mon.noname-b con 0x7f3020144ca0 addr v1:172.21.15.88:6790/0 2024-06-03T08:36:04.159 INFO:teuthology.misc.health.smithi088.stderr:2024-06-03T08:36:04.157+0000 7f3027beb640 10 monclient(hunting): picked mon.noname-a con 0x7f3020145720 addr v1:172.21.15.88:6789/0 2024-06-03T08:36:04.159 INFO:teuthology.misc.health.smithi088.stderr:2024-06-03T08:36:04.157+0000 7f3027beb640 10 monclient(hunting): picked mon.noname-c con 0x7f302013eed0 addr v1:172.21.15.192:6789/0 2024-06-03T08:36:04.159 INFO:teuthology.misc.health.smithi088.stderr:2024-06-03T08:36:04.157+0000 7f3027beb640 1 -- --> v1:172.21.15.88:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f30200aa330 con 0x7f3020145720 2024-06-03T08:36:04.159 INFO:teuthology.misc.health.smithi088.stderr:2024-06-03T08:36:04.157+0000 7f3027beb640 1 -- --> v1:172.21.15.192:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f3020145290 con 0x7f302013eed0 2024-06-03T08:36:04.159 INFO:teuthology.misc.health.smithi088.stderr:2024-06-03T08:36:04.157+0000 7f3027beb640 1 -- --> v1:172.21.15.88:6790/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f3020145080 con 0x7f3020144ca0 2024-06-03T08:36:04.159 INFO:teuthology.misc.health.smithi088.stderr:2024-06-03T08:36:04.157+0000 7f3027beb640 10 monclient(hunting): _renew_subs 2024-06-03T08:36:04.159 INFO:teuthology.misc.health.smithi088.stderr:2024-06-03T08:36:04.157+0000 7f3027beb640 10 monclient(hunting): authenticate will time out at 1219.967906s 2024-06-03T08:36:04.159 INFO:teuthology.misc.health.smithi088.stderr:2024-06-03T08:36:04.158+0000 7f3025987640 1 --1- >> v1:172.21.15.88:6790/0 conn(0x7f3020144ca0 0x7f30200b1c70 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.88:6790/0 says I am v1:172.21.15.88:45466/0 (socket says 172.21.15.88:45466) 2024-06-03T08:36:04.159 INFO:teuthology.misc.health.smithi088.stderr:2024-06-03T08:36:04.158+0000 7f3025987640 1 -- 172.21.15.88:0/326992825 learned_addr learned my addr 172.21.15.88:0/326992825 (peer_addr_for_me v1:172.21.15.88:0/0) 2024-06-03T08:36:04.162 INFO:teuthology.misc.health.smithi088.stderr:2024-06-03T08:36:04.161+0000 7f3024985640 1 -- 172.21.15.88:0/326992825 <== mon.1 v1:172.21.15.192:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 4257476920 0 0) 0x7f3020145290 con 0x7f302013eed0 2024-06-03T08:36:04.162 INFO:teuthology.misc.health.smithi088.stderr:2024-06-03T08:36:04.161+0000 7f3024985640 10 monclient(hunting): _negotiate 2024-06-03T08:36:04.163 INFO:teuthology.misc.health.smithi088.stderr:2024-06-03T08:36:04.161+0000 7f3024985640 10 monclient(hunting): _init_auth method 2 2024-06-03T08:36:04.163 INFO:teuthology.misc.health.smithi088.stderr:2024-06-03T08:36:04.161+0000 7f3024985640 10 monclient(hunting): _init_auth creating new auth 2024-06-03T08:36:04.163 INFO:teuthology.misc.health.smithi088.stderr:2024-06-03T08:36:04.161+0000 7f3024985640 10 monclient(hunting): my global_id is 4213 2024-06-03T08:36:04.163 INFO:teuthology.misc.health.smithi088.stderr:2024-06-03T08:36:04.161+0000 7f3024985640 1 -- 172.21.15.88:0/326992825 --> v1:172.21.15.192:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f300c003520 con 0x7f302013eed0 2024-06-03T08:36:04.163 INFO:teuthology.misc.health.smithi088.stderr:2024-06-03T08:36:04.161+0000 7f3024985640 1 -- 172.21.15.88:0/326992825 <== mon.0 v1:172.21.15.88:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 1648252291 0 0) 0x7f30200aa330 con 0x7f3020145720 2024-06-03T08:36:04.163 INFO:teuthology.misc.health.smithi088.stderr:2024-06-03T08:36:04.161+0000 7f3024985640 10 monclient(hunting): _negotiate 2024-06-03T08:36:04.163 INFO:teuthology.misc.health.smithi088.stderr:2024-06-03T08:36:04.161+0000 7f3024985640 10 monclient(hunting): _init_auth method 2 2024-06-03T08:36:04.163 INFO:teuthology.misc.health.smithi088.stderr:2024-06-03T08:36:04.162+0000 7f3024985640 10 monclient(hunting): _init_auth creating new auth 2024-06-03T08:36:04.163 INFO:teuthology.misc.health.smithi088.stderr:2024-06-03T08:36:04.162+0000 7f3024985640 10 monclient(hunting): my global_id is 4224 2024-06-03T08:36:04.163 INFO:teuthology.misc.health.smithi088.stderr:2024-06-03T08:36:04.162+0000 7f3024985640 1 -- 172.21.15.88:0/326992825 --> v1:172.21.15.88:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f3020145290 con 0x7f3020145720 2024-06-03T08:36:04.163 INFO:teuthology.misc.health.smithi088.stderr:2024-06-03T08:36:04.162+0000 7f3024985640 1 -- 172.21.15.88:0/326992825 <== mon.2 v1:172.21.15.88:6790/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 1303243070 0 0) 0x7f3020145080 con 0x7f3020144ca0 2024-06-03T08:36:04.163 INFO:teuthology.misc.health.smithi088.stderr:2024-06-03T08:36:04.162+0000 7f3024985640 10 monclient(hunting): _negotiate 2024-06-03T08:36:04.163 INFO:teuthology.misc.health.smithi088.stderr:2024-06-03T08:36:04.162+0000 7f3024985640 10 monclient(hunting): _init_auth method 2 2024-06-03T08:36:04.164 INFO:teuthology.misc.health.smithi088.stderr:2024-06-03T08:36:04.162+0000 7f3024985640 10 monclient(hunting): _init_auth creating new auth 2024-06-03T08:36:04.164 INFO:teuthology.misc.health.smithi088.stderr:2024-06-03T08:36:04.162+0000 7f3024985640 10 monclient(hunting): my global_id is 4226 2024-06-03T08:36:04.164 INFO:teuthology.misc.health.smithi088.stderr:2024-06-03T08:36:04.162+0000 7f3024985640 1 -- 172.21.15.88:0/326992825 --> v1:172.21.15.88:6790/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f30200aa330 con 0x7f3020144ca0 2024-06-03T08:36:04.166 INFO:teuthology.misc.health.smithi088.stderr:2024-06-03T08:36:04.164+0000 7f3024985640 1 -- 172.21.15.88:0/326992825 <== mon.1 v1:172.21.15.192:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 214+0+0 (unknown 1040235565 0 0) 0x7f300c003520 con 0x7f302013eed0 2024-06-03T08:36:04.166 INFO:teuthology.misc.health.smithi088.stderr:2024-06-03T08:36:04.164+0000 7f3024985640 1 -- 172.21.15.88:0/326992825 >> v1:172.21.15.88:6790/0 conn(0x7f3020144ca0 legacy=0x7f30200b1c70 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-06-03T08:36:04.166 INFO:teuthology.misc.health.smithi088.stderr:2024-06-03T08:36:04.164+0000 7f3024985640 1 -- 172.21.15.88:0/326992825 >> v1:172.21.15.88:6789/0 conn(0x7f3020145720 legacy=0x7f302013e7c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-06-03T08:36:04.166 INFO:teuthology.misc.health.smithi088.stderr:2024-06-03T08:36:04.164+0000 7f3024985640 10 monclient: _finish_hunting 0 2024-06-03T08:36:04.166 INFO:teuthology.misc.health.smithi088.stderr:2024-06-03T08:36:04.164+0000 7f3024985640 1 monclient: found mon.noname-c 2024-06-03T08:36:04.166 INFO:teuthology.misc.health.smithi088.stderr:2024-06-03T08:36:04.164+0000 7f3024985640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-06-03T08:36:04.166 INFO:teuthology.misc.health.smithi088.stderr:2024-06-03T08:36:04.164+0000 7f3024985640 10 monclient: _send_mon_message to mon.noname-c at v1:172.21.15.192:6789/0 2024-06-03T08:36:04.166 INFO:teuthology.misc.health.smithi088.stderr:2024-06-03T08:36:04.164+0000 7f3024985640 1 -- 172.21.15.88:0/326992825 --> v1:172.21.15.192:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f30200b2380 con 0x7f302013eed0 2024-06-03T08:36:04.166 INFO:teuthology.misc.health.smithi088.stderr:2024-06-03T08:36:04.164+0000 7f3024985640 10 monclient: _finish_auth 0 2024-06-03T08:36:04.166 INFO:teuthology.misc.health.smithi088.stderr:2024-06-03T08:36:04.164+0000 7f3024985640 10 monclient: _check_auth_tickets 2024-06-03T08:36:04.166 INFO:teuthology.misc.health.smithi088.stderr:2024-06-03T08:36:04.164+0000 7f3024985640 20 monclient: _check_auth_rotating not needed by client.admin 2024-06-03T08:36:04.166 INFO:teuthology.misc.health.smithi088.stderr:2024-06-03T08:36:04.164+0000 7f3024985640 1 -- 172.21.15.88:0/326992825 <== mon.1 v1:172.21.15.192:6789/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 2234374065 0 0) 0x7f3018001990 con 0x7f302013eed0 2024-06-03T08:36:04.167 INFO:teuthology.misc.health.smithi088.stderr:2024-06-03T08:36:04.164+0000 7f3024985640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-06-03T08:36:04.167 INFO:teuthology.misc.health.smithi088.stderr:2024-06-03T08:36:04.164+0000 7f3024985640 10 monclient: got monmap 1 from mon.noname-c (according to old e1) 2024-06-03T08:36:04.167 INFO:teuthology.misc.health.smithi088.stderr:2024-06-03T08:36:04.164+0000 7f3024985640 10 monclient: dump: 2024-06-03T08:36:04.167 INFO:teuthology.misc.health.smithi088.stderr:epoch 1 2024-06-03T08:36:04.167 INFO:teuthology.misc.health.smithi088.stderr:fsid ec3c3fdd-3ac6-4d7d-9412-683130ed8b04 2024-06-03T08:36:04.167 INFO:teuthology.misc.health.smithi088.stderr:last_changed 2024-06-03T08:34:31.885125+0000 2024-06-03T08:36:04.167 INFO:teuthology.misc.health.smithi088.stderr:created 2024-06-03T08:34:31.885125+0000 2024-06-03T08:36:04.167 INFO:teuthology.misc.health.smithi088.stderr:min_mon_release 18 (reef) 2024-06-03T08:36:04.167 INFO:teuthology.misc.health.smithi088.stderr:election_strategy: 3 2024-06-03T08:36:04.167 INFO:teuthology.misc.health.smithi088.stderr:0: v1:172.21.15.88:6789/0 mon.a 2024-06-03T08:36:04.167 INFO:teuthology.misc.health.smithi088.stderr:1: v1:172.21.15.192:6789/0 mon.b 2024-06-03T08:36:04.167 INFO:teuthology.misc.health.smithi088.stderr:2: v1:172.21.15.88:6790/0 mon.c 2024-06-03T08:36:04.167 INFO:teuthology.misc.health.smithi088.stderr: 2024-06-03T08:36:04.167 INFO:teuthology.misc.health.smithi088.stderr:2024-06-03T08:36:04.165+0000 7f3027beb640 5 monclient: authenticate success, global_id 4213 2024-06-03T08:36:04.168 INFO:teuthology.misc.health.smithi088.stderr:2024-06-03T08:36:04.165+0000 7f3027beb640 20 monclient: get_monmap_and_config waiting for monmap|config 2024-06-03T08:36:04.169 INFO:teuthology.misc.health.smithi088.stderr:2024-06-03T08:36:04.166+0000 7f3024985640 1 -- 172.21.15.88:0/326992825 <== mon.1 v1:172.21.15.192:6789/0 4 ==== config(0 keys) v1 ==== 4+0+0 (unknown 0 0 0) 0x7f3018001be0 con 0x7f302013eed0 2024-06-03T08:36:04.169 INFO:teuthology.misc.health.smithi088.stderr:2024-06-03T08:36:04.166+0000 7f3024985640 10 monclient: handle_config config(0 keys) v1 2024-06-03T08:36:04.169 INFO:teuthology.misc.health.smithi088.stderr:2024-06-03T08:36:04.166+0000 7f3024985640 1 -- 172.21.15.88:0/326992825 <== mon.1 v1:172.21.15.192:6789/0 5 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 2234374065 0 0) 0x7f3018001f70 con 0x7f302013eed0 2024-06-03T08:36:04.169 INFO:teuthology.misc.health.smithi088.stderr:2024-06-03T08:36:04.166+0000 7f3027beb640 10 monclient: get_monmap_and_config success 2024-06-03T08:36:04.169 INFO:teuthology.misc.health.smithi088.stderr:2024-06-03T08:36:04.166+0000 7f3027beb640 4 set_mon_vals no callback set 2024-06-03T08:36:04.169 INFO:teuthology.misc.health.smithi088.stderr:2024-06-03T08:36:04.167+0000 7f3027beb640 10 monclient: shutdown 2024-06-03T08:36:04.169 INFO:teuthology.misc.health.smithi088.stderr:2024-06-03T08:36:04.167+0000 7f3024985640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-06-03T08:36:04.170 INFO:teuthology.misc.health.smithi088.stderr:2024-06-03T08:36:04.167+0000 7f3024985640 10 monclient: got monmap 1 from mon.b (according to old e1) 2024-06-03T08:36:04.170 INFO:teuthology.misc.health.smithi088.stderr:2024-06-03T08:36:04.167+0000 7f3024985640 10 monclient: dump: 2024-06-03T08:36:04.170 INFO:teuthology.misc.health.smithi088.stderr:epoch 1 2024-06-03T08:36:04.170 INFO:teuthology.misc.health.smithi088.stderr:fsid ec3c3fdd-3ac6-4d7d-9412-683130ed8b04 2024-06-03T08:36:04.170 INFO:teuthology.misc.health.smithi088.stderr:last_changed 2024-06-03T08:34:31.885125+0000 2024-06-03T08:36:04.170 INFO:teuthology.misc.health.smithi088.stderr:created 2024-06-03T08:34:31.885125+0000 2024-06-03T08:36:04.170 INFO:teuthology.misc.health.smithi088.stderr:min_mon_release 18 (reef) 2024-06-03T08:36:04.170 INFO:teuthology.misc.health.smithi088.stderr:election_strategy: 3 2024-06-03T08:36:04.170 INFO:teuthology.misc.health.smithi088.stderr:0: v1:172.21.15.88:6789/0 mon.a 2024-06-03T08:36:04.170 INFO:teuthology.misc.health.smithi088.stderr:1: v1:172.21.15.192:6789/0 mon.b 2024-06-03T08:36:04.170 INFO:teuthology.misc.health.smithi088.stderr:2: v1:172.21.15.88:6790/0 mon.c 2024-06-03T08:36:04.170 INFO:teuthology.misc.health.smithi088.stderr: 2024-06-03T08:36:04.171 INFO:teuthology.misc.health.smithi088.stderr:2024-06-03T08:36:04.167+0000 7f3027beb640 20 monclient: shutdown discarding 0 pending message(s) 2024-06-03T08:36:04.171 INFO:teuthology.misc.health.smithi088.stderr:2024-06-03T08:36:04.167+0000 7f3027beb640 1 -- 172.21.15.88:0/326992825 >> v1:172.21.15.192:6789/0 conn(0x7f302013eed0 legacy=0x7f302014cdd0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-06-03T08:36:04.171 INFO:teuthology.misc.health.smithi088.stderr:2024-06-03T08:36:04.167+0000 7f3027beb640 1 -- 172.21.15.88:0/326992825 shutdown_connections 2024-06-03T08:36:04.171 INFO:teuthology.misc.health.smithi088.stderr:2024-06-03T08:36:04.167+0000 7f3027beb640 1 -- 172.21.15.88:0/326992825 >> 172.21.15.88:0/326992825 conn(0x7f30200b7270 msgr2=0x7f30200b7680 unknown :-1 s=STATE_NONE l=0).mark_down 2024-06-03T08:36:04.171 INFO:teuthology.misc.health.smithi088.stderr:2024-06-03T08:36:04.167+0000 7f3027beb640 1 -- 172.21.15.88:0/326992825 shutdown_connections 2024-06-03T08:36:04.171 INFO:teuthology.misc.health.smithi088.stderr:2024-06-03T08:36:04.167+0000 7f3027beb640 1 -- 172.21.15.88:0/326992825 wait complete. 2024-06-03T08:36:04.171 INFO:teuthology.misc.health.smithi088.stderr:2024-06-03T08:36:04.168+0000 7f3027beb640 10 monclient: build_initial_monmap 2024-06-03T08:36:04.171 INFO:teuthology.misc.health.smithi088.stderr:2024-06-03T08:36:04.168+0000 7f3027beb640 10 monclient: monmap: 2024-06-03T08:36:04.171 INFO:teuthology.misc.health.smithi088.stderr:epoch 0 2024-06-03T08:36:04.171 INFO:teuthology.misc.health.smithi088.stderr:fsid 00000000-0000-0000-0000-000000000000 2024-06-03T08:36:04.171 INFO:teuthology.misc.health.smithi088.stderr:last_changed 0.000000 2024-06-03T08:36:04.171 INFO:teuthology.misc.health.smithi088.stderr:created 0.000000 2024-06-03T08:36:04.171 INFO:teuthology.misc.health.smithi088.stderr:min_mon_release 0 (unknown) 2024-06-03T08:36:04.171 INFO:teuthology.misc.health.smithi088.stderr:election_strategy: 1 2024-06-03T08:36:04.171 INFO:teuthology.misc.health.smithi088.stderr:0: v1:172.21.15.88:6789/0 mon.noname-a 2024-06-03T08:36:04.172 INFO:teuthology.misc.health.smithi088.stderr:1: v1:172.21.15.192:6789/0 mon.noname-b 2024-06-03T08:36:04.172 INFO:teuthology.misc.health.smithi088.stderr:2: v1:172.21.15.88:6790/0 mon.noname-c 2024-06-03T08:36:04.172 INFO:teuthology.misc.health.smithi088.stderr: 2024-06-03T08:36:04.172 INFO:teuthology.misc.health.smithi088.stderr:2024-06-03T08:36:04.168+0000 7f3027beb640 1 librados: starting msgr at 2024-06-03T08:36:04.172 INFO:teuthology.misc.health.smithi088.stderr:2024-06-03T08:36:04.168+0000 7f3027beb640 1 librados: starting objecter 2024-06-03T08:36:04.172 INFO:teuthology.misc.health.smithi088.stderr:2024-06-03T08:36:04.169+0000 7f3027beb640 1 Processor -- start 2024-06-03T08:36:04.172 INFO:teuthology.misc.health.smithi088.stderr:2024-06-03T08:36:04.169+0000 7f3027beb640 1 -- start start 2024-06-03T08:36:04.172 INFO:teuthology.misc.health.smithi088.stderr:2024-06-03T08:36:04.169+0000 7f3027beb640 1 librados: setting wanted keys 2024-06-03T08:36:04.172 INFO:teuthology.misc.health.smithi088.stderr:2024-06-03T08:36:04.169+0000 7f3027beb640 1 librados: calling monclient init 2024-06-03T08:36:04.172 INFO:teuthology.misc.health.smithi088.stderr:2024-06-03T08:36:04.169+0000 7f3027beb640 10 monclient: init 2024-06-03T08:36:04.175 INFO:teuthology.misc.health.smithi088.stderr:2024-06-03T08:36:04.170+0000 7f3027beb640 10 monclient: _reopen_session rank -1 2024-06-03T08:36:04.175 INFO:teuthology.misc.health.smithi088.stderr:2024-06-03T08:36:04.170+0000 7f3027beb640 10 monclient: _add_conns ranks=[1,2,0] 2024-06-03T08:36:04.175 INFO:teuthology.misc.health.smithi088.stderr:2024-06-03T08:36:04.170+0000 7f3027beb640 10 monclient(hunting): picked mon.noname-b con 0x7f302013eed0 addr v1:172.21.15.192:6789/0 2024-06-03T08:36:04.175 INFO:teuthology.misc.health.smithi088.stderr:2024-06-03T08:36:04.170+0000 7f3027beb640 10 monclient(hunting): picked mon.noname-c con 0x7f3020144ca0 addr v1:172.21.15.88:6790/0 2024-06-03T08:36:04.175 INFO:teuthology.misc.health.smithi088.stderr:2024-06-03T08:36:04.170+0000 7f3027beb640 10 monclient(hunting): picked mon.noname-a con 0x7f3020145720 addr v1:172.21.15.88:6789/0 2024-06-03T08:36:04.175 INFO:teuthology.misc.health.smithi088.stderr:2024-06-03T08:36:04.170+0000 7f3027beb640 1 -- --> v1:172.21.15.88:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f30200ab250 con 0x7f3020145720 2024-06-03T08:36:04.176 INFO:teuthology.misc.health.smithi088.stderr:2024-06-03T08:36:04.170+0000 7f3027beb640 1 -- --> v1:172.21.15.192:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f30200ab480 con 0x7f302013eed0 2024-06-03T08:36:04.176 INFO:teuthology.misc.health.smithi088.stderr:2024-06-03T08:36:04.170+0000 7f3027beb640 1 -- --> v1:172.21.15.88:6790/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f30200ac8e0 con 0x7f3020144ca0 2024-06-03T08:36:04.176 INFO:teuthology.misc.health.smithi088.stderr:2024-06-03T08:36:04.170+0000 7f3027beb640 10 monclient(hunting): _renew_subs 2024-06-03T08:36:04.176 INFO:teuthology.misc.health.smithi088.stderr:2024-06-03T08:36:04.170+0000 7f3027beb640 10 monclient(hunting): authenticate will time out at 1219.980378s 2024-06-03T08:36:04.176 INFO:teuthology.misc.health.smithi088.stderr:2024-06-03T08:36:04.170+0000 7f3025987640 1 --1- >> v1:172.21.15.192:6789/0 conn(0x7f302013eed0 0x7f30200aab40 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=1).handle_server_banner_and_identify peer v1:172.21.15.192:6789/0 says I am v1:172.21.15.88:44048/0 (socket says 172.21.15.88:44048) 2024-06-03T08:36:04.176 INFO:teuthology.misc.health.smithi088.stderr:2024-06-03T08:36:04.170+0000 7f3025987640 1 -- 172.21.15.88:0/143279954 learned_addr learned my addr 172.21.15.88:0/143279954 (peer_addr_for_me v1:172.21.15.88:0/0) 2024-06-03T08:36:04.176 INFO:teuthology.misc.health.smithi088.stderr:2024-06-03T08:36:04.171+0000 7f30167fc640 10 client.?.objecter ms_handle_connect 0x7f302013eed0 2024-06-03T08:36:04.176 INFO:teuthology.misc.health.smithi088.stderr:2024-06-03T08:36:04.171+0000 7f30167fc640 10 client.?.objecter resend_mon_ops 2024-06-03T08:36:04.176 INFO:teuthology.misc.health.smithi088.stderr:2024-06-03T08:36:04.171+0000 7f30167fc640 10 client.?.objecter ms_handle_connect 0x7f3020144ca0 2024-06-03T08:36:04.176 INFO:teuthology.misc.health.smithi088.stderr:2024-06-03T08:36:04.171+0000 7f30167fc640 10 client.?.objecter resend_mon_ops 2024-06-03T08:36:04.176 INFO:teuthology.misc.health.smithi088.stderr:2024-06-03T08:36:04.171+0000 7f30167fc640 10 client.?.objecter ms_handle_connect 0x7f3020145720 2024-06-03T08:36:04.176 INFO:teuthology.misc.health.smithi088.stderr:2024-06-03T08:36:04.171+0000 7f30167fc640 10 client.?.objecter resend_mon_ops 2024-06-03T08:36:04.176 INFO:teuthology.misc.health.smithi088.stderr:2024-06-03T08:36:04.172+0000 7f30167fc640 1 -- 172.21.15.88:0/143279954 <== mon.1 v1:172.21.15.192:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 558143415 0 0) 0x7f30200ab480 con 0x7f302013eed0 2024-06-03T08:36:04.177 INFO:teuthology.misc.health.smithi088.stderr:2024-06-03T08:36:04.172+0000 7f30167fc640 10 monclient(hunting): _negotiate 2024-06-03T08:36:04.177 INFO:teuthology.misc.health.smithi088.stderr:2024-06-03T08:36:04.172+0000 7f30167fc640 10 monclient(hunting): _init_auth method 2 2024-06-03T08:36:04.177 INFO:teuthology.misc.health.smithi088.stderr:2024-06-03T08:36:04.172+0000 7f30167fc640 10 monclient(hunting): _init_auth creating new auth 2024-06-03T08:36:04.177 INFO:teuthology.misc.health.smithi088.stderr:2024-06-03T08:36:04.172+0000 7f30167fc640 10 monclient(hunting): my global_id is 4216 2024-06-03T08:36:04.177 INFO:teuthology.misc.health.smithi088.stderr:2024-06-03T08:36:04.172+0000 7f30167fc640 1 -- 172.21.15.88:0/143279954 --> v1:172.21.15.192:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f3004004eb0 con 0x7f302013eed0 2024-06-03T08:36:04.177 INFO:teuthology.misc.health.smithi088.stderr:2024-06-03T08:36:04.174+0000 7f30167fc640 1 -- 172.21.15.88:0/143279954 <== mon.2 v1:172.21.15.88:6790/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 926456456 0 0) 0x7f30200ac8e0 con 0x7f3020144ca0 2024-06-03T08:36:04.177 INFO:teuthology.misc.health.smithi088.stderr:2024-06-03T08:36:04.174+0000 7f30167fc640 10 monclient(hunting): _negotiate 2024-06-03T08:36:04.177 INFO:teuthology.misc.health.smithi088.stderr:2024-06-03T08:36:04.174+0000 7f30167fc640 10 monclient(hunting): _init_auth method 2 2024-06-03T08:36:04.177 INFO:teuthology.misc.health.smithi088.stderr:2024-06-03T08:36:04.174+0000 7f30167fc640 10 monclient(hunting): _init_auth creating new auth 2024-06-03T08:36:04.177 INFO:teuthology.misc.health.smithi088.stderr:2024-06-03T08:36:04.174+0000 7f30167fc640 10 monclient(hunting): my global_id is 4229 2024-06-03T08:36:04.177 INFO:teuthology.misc.health.smithi088.stderr:2024-06-03T08:36:04.174+0000 7f30167fc640 1 -- 172.21.15.88:0/143279954 --> v1:172.21.15.88:6790/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f30200ab480 con 0x7f3020144ca0 2024-06-03T08:36:04.177 INFO:teuthology.misc.health.smithi088.stderr:2024-06-03T08:36:04.174+0000 7f30167fc640 1 -- 172.21.15.88:0/143279954 <== mon.0 v1:172.21.15.88:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 3296814278 0 0) 0x7f30200ab250 con 0x7f3020145720 2024-06-03T08:36:04.177 INFO:teuthology.misc.health.smithi088.stderr:2024-06-03T08:36:04.174+0000 7f30167fc640 10 monclient(hunting): _negotiate 2024-06-03T08:36:04.178 INFO:teuthology.misc.health.smithi088.stderr:2024-06-03T08:36:04.174+0000 7f30167fc640 10 monclient(hunting): _init_auth method 2 2024-06-03T08:36:04.178 INFO:teuthology.misc.health.smithi088.stderr:2024-06-03T08:36:04.174+0000 7f30167fc640 10 monclient(hunting): _init_auth creating new auth 2024-06-03T08:36:04.178 INFO:teuthology.misc.health.smithi088.stderr:2024-06-03T08:36:04.174+0000 7f30167fc640 10 monclient(hunting): my global_id is 4227 2024-06-03T08:36:04.178 INFO:teuthology.misc.health.smithi088.stderr:2024-06-03T08:36:04.174+0000 7f30167fc640 1 -- 172.21.15.88:0/143279954 --> v1:172.21.15.88:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f30200ac8e0 con 0x7f3020145720 2024-06-03T08:36:04.179 INFO:teuthology.misc.health.smithi088.stderr:2024-06-03T08:36:04.179+0000 7f30167fc640 1 -- 172.21.15.88:0/143279954 <== mon.1 v1:172.21.15.192:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 764+0+0 (unknown 697480416 0 0) 0x7f3004004eb0 con 0x7f302013eed0 2024-06-03T08:36:04.179 INFO:teuthology.misc.health.smithi088.stderr:2024-06-03T08:36:04.179+0000 7f30167fc640 1 -- 172.21.15.88:0/143279954 --> v1:172.21.15.192:6789/0 -- auth(proto 2 165 bytes epoch 0) v1 -- 0x7f30200ab250 con 0x7f302013eed0 2024-06-03T08:36:04.179 INFO:teuthology.misc.health.smithi088.stderr:2024-06-03T08:36:04.179+0000 7f30167fc640 1 -- 172.21.15.88:0/143279954 <== mon.1 v1:172.21.15.192:6789/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 2234374065 0 0) 0x7f301c0018b0 con 0x7f302013eed0 2024-06-03T08:36:04.179 INFO:teuthology.misc.health.smithi088.stderr:2024-06-03T08:36:04.179+0000 7f30167fc640 10 monclient(hunting): handle_monmap mon_map magic: 0 v1 2024-06-03T08:36:04.179 INFO:teuthology.misc.health.smithi088.stderr:2024-06-03T08:36:04.179+0000 7f30167fc640 10 monclient(hunting): got monmap 1 from mon.noname-b (according to old e1) 2024-06-03T08:36:04.180 INFO:teuthology.misc.health.smithi088.stderr:2024-06-03T08:36:04.179+0000 7f30167fc640 10 monclient(hunting): dump: 2024-06-03T08:36:04.180 INFO:teuthology.misc.health.smithi088.stderr:epoch 1 2024-06-03T08:36:04.180 INFO:teuthology.misc.health.smithi088.stderr:fsid ec3c3fdd-3ac6-4d7d-9412-683130ed8b04 2024-06-03T08:36:04.180 INFO:teuthology.misc.health.smithi088.stderr:last_changed 2024-06-03T08:34:31.885125+0000 2024-06-03T08:36:04.180 INFO:teuthology.misc.health.smithi088.stderr:created 2024-06-03T08:34:31.885125+0000 2024-06-03T08:36:04.180 INFO:teuthology.misc.health.smithi088.stderr:min_mon_release 18 (reef) 2024-06-03T08:36:04.180 INFO:teuthology.misc.health.smithi088.stderr:election_strategy: 3 2024-06-03T08:36:04.180 INFO:teuthology.misc.health.smithi088.stderr:0: v1:172.21.15.88:6789/0 mon.a 2024-06-03T08:36:04.180 INFO:teuthology.misc.health.smithi088.stderr:1: v1:172.21.15.192:6789/0 mon.b 2024-06-03T08:36:04.180 INFO:teuthology.misc.health.smithi088.stderr:2: v1:172.21.15.88:6790/0 mon.c 2024-06-03T08:36:04.180 INFO:teuthology.misc.health.smithi088.stderr: 2024-06-03T08:36:04.181 INFO:teuthology.misc.health.smithi088.stderr:2024-06-03T08:36:04.179+0000 7f30167fc640 10 monclient(hunting): _finish_auth 0 2024-06-03T08:36:04.182 INFO:teuthology.misc.health.smithi088.stderr:2024-06-03T08:36:04.181+0000 7f30167fc640 1 -- 172.21.15.88:0/143279954 <== mon.2 v1:172.21.15.88:6790/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 764+0+0 (unknown 3812499347 0 0) 0x7f30200ab480 con 0x7f3020144ca0 2024-06-03T08:36:04.182 INFO:teuthology.misc.health.smithi088.stderr:2024-06-03T08:36:04.181+0000 7f30167fc640 1 -- 172.21.15.88:0/143279954 --> v1:172.21.15.88:6790/0 -- auth(proto 2 165 bytes epoch 0) v1 -- 0x7f3004004eb0 con 0x7f3020144ca0 2024-06-03T08:36:04.185 INFO:teuthology.misc.health.smithi088.stderr:2024-06-03T08:36:04.182+0000 7f30167fc640 1 -- 172.21.15.88:0/143279954 <== mon.1 v1:172.21.15.192:6789/0 4 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 580+0+0 (unknown 3297230285 0 0) 0x7f30200ab250 con 0x7f302013eed0 2024-06-03T08:36:04.185 INFO:teuthology.misc.health.smithi088.stderr:2024-06-03T08:36:04.182+0000 7f30167fc640 1 -- 172.21.15.88:0/143279954 >> v1:172.21.15.88:6790/0 conn(0x7f3020144ca0 legacy=0x7f302013f610 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-06-03T08:36:04.185 INFO:teuthology.misc.health.smithi088.stderr:2024-06-03T08:36:04.182+0000 7f30167fc640 1 -- 172.21.15.88:0/143279954 >> v1:172.21.15.88:6789/0 conn(0x7f3020145720 legacy=0x7f30201366d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-06-03T08:36:04.185 INFO:teuthology.misc.health.smithi088.stderr:2024-06-03T08:36:04.182+0000 7f30167fc640 10 monclient: _finish_hunting 0 2024-06-03T08:36:04.185 INFO:teuthology.misc.health.smithi088.stderr:2024-06-03T08:36:04.182+0000 7f30167fc640 1 monclient: found mon.b 2024-06-03T08:36:04.185 INFO:teuthology.misc.health.smithi088.stderr:2024-06-03T08:36:04.182+0000 7f30167fc640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-06-03T08:36:04.185 INFO:teuthology.misc.health.smithi088.stderr:2024-06-03T08:36:04.182+0000 7f30167fc640 10 monclient: _send_mon_message to mon.b at v1:172.21.15.192:6789/0 2024-06-03T08:36:04.185 INFO:teuthology.misc.health.smithi088.stderr:2024-06-03T08:36:04.182+0000 7f30167fc640 1 -- 172.21.15.88:0/143279954 --> v1:172.21.15.192:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f3020141300 con 0x7f302013eed0 2024-06-03T08:36:04.185 INFO:teuthology.misc.health.smithi088.stderr:2024-06-03T08:36:04.182+0000 7f30167fc640 10 monclient: _finish_auth 0 2024-06-03T08:36:04.185 INFO:teuthology.misc.health.smithi088.stderr:2024-06-03T08:36:04.182+0000 7f30167fc640 10 monclient: _check_auth_tickets 2024-06-03T08:36:04.186 INFO:teuthology.misc.health.smithi088.stderr:2024-06-03T08:36:04.182+0000 7f30167fc640 20 monclient: _check_auth_rotating not needed by client.admin 2024-06-03T08:36:04.186 INFO:teuthology.misc.health.smithi088.stderr:2024-06-03T08:36:04.182+0000 7f3027beb640 5 monclient: authenticate success, global_id 4216 2024-06-03T08:36:04.186 INFO:teuthology.misc.health.smithi088.stderr:2024-06-03T08:36:04.182+0000 7f3027beb640 10 monclient: _renew_subs 2024-06-03T08:36:04.186 INFO:teuthology.misc.health.smithi088.stderr:2024-06-03T08:36:04.182+0000 7f3027beb640 10 monclient: _send_mon_message to mon.b at v1:172.21.15.192:6789/0 2024-06-03T08:36:04.186 INFO:teuthology.misc.health.smithi088.stderr:2024-06-03T08:36:04.182+0000 7f3027beb640 1 -- 172.21.15.88:0/143279954 --> v1:172.21.15.192:6789/0 -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f3020142220 con 0x7f302013eed0 2024-06-03T08:36:04.186 INFO:teuthology.misc.health.smithi088.stderr:2024-06-03T08:36:04.182+0000 7f3027beb640 10 client.4216.objecter _maybe_request_map subscribing (onetime) to next osd map 2024-06-03T08:36:04.186 INFO:teuthology.misc.health.smithi088.stderr:2024-06-03T08:36:04.182+0000 7f3027beb640 10 monclient: _renew_subs 2024-06-03T08:36:04.186 INFO:teuthology.misc.health.smithi088.stderr:2024-06-03T08:36:04.182+0000 7f3027beb640 10 monclient: _send_mon_message to mon.b at v1:172.21.15.192:6789/0 2024-06-03T08:36:04.186 INFO:teuthology.misc.health.smithi088.stderr:2024-06-03T08:36:04.182+0000 7f3027beb640 1 -- 172.21.15.88:0/143279954 --> v1:172.21.15.192:6789/0 -- mon_subscribe({osdmap=0}) v3 -- 0x7f30200ab830 con 0x7f302013eed0 2024-06-03T08:36:04.186 INFO:teuthology.misc.health.smithi088.stderr:2024-06-03T08:36:04.182+0000 7f3027beb640 1 librados: init done 2024-06-03T08:36:04.186 INFO:teuthology.misc.health.smithi088.stderr:2024-06-03T08:36:04.184+0000 7f3027beb640 10 start_mon_command cmd=[{"prefix": "get_command_descriptions"}] 2024-06-03T08:36:04.186 INFO:teuthology.misc.health.smithi088.stderr:2024-06-03T08:36:04.184+0000 7f3027beb640 10 monclient: _send_command 1 [{"prefix": "get_command_descriptions"}] 2024-06-03T08:36:04.187 INFO:teuthology.misc.health.smithi088.stderr:2024-06-03T08:36:04.184+0000 7f3027beb640 10 monclient: _send_mon_message to mon.b at v1:172.21.15.192:6789/0 2024-06-03T08:36:04.187 INFO:teuthology.misc.health.smithi088.stderr:2024-06-03T08:36:04.184+0000 7f3027beb640 1 -- 172.21.15.88:0/143279954 --> v1:172.21.15.192:6789/0 -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f30200a9a90 con 0x7f302013eed0 2024-06-03T08:36:04.204 INFO:teuthology.misc.health.smithi088.stderr:2024-06-03T08:36:04.203+0000 7f30167fc640 1 -- 172.21.15.88:0/143279954 <== mon.1 v1:172.21.15.192:6789/0 5 ==== config(0 keys) v1 ==== 4+0+0 (unknown 0 0 0) 0x7f301c0047e0 con 0x7f302013eed0 2024-06-03T08:36:04.205 INFO:teuthology.misc.health.smithi088.stderr:2024-06-03T08:36:04.203+0000 7f30167fc640 10 monclient: handle_config config(0 keys) v1 2024-06-03T08:36:04.205 INFO:teuthology.misc.health.smithi088.stderr:2024-06-03T08:36:04.203+0000 7f30167fc640 1 -- 172.21.15.88:0/143279954 <== mon.1 v1:172.21.15.192:6789/0 6 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 2234374065 0 0) 0x7f301c004af0 con 0x7f302013eed0 2024-06-03T08:36:04.205 INFO:teuthology.misc.health.smithi088.stderr:2024-06-03T08:36:04.203+0000 7f30167fc640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-06-03T08:36:04.205 INFO:teuthology.misc.health.smithi088.stderr:2024-06-03T08:36:04.203+0000 7f30167fc640 10 monclient: got monmap 1 from mon.b (according to old e1) 2024-06-03T08:36:04.205 INFO:teuthology.misc.health.smithi088.stderr:2024-06-03T08:36:04.203+0000 7f30167fc640 10 monclient: dump: 2024-06-03T08:36:04.205 INFO:teuthology.misc.health.smithi088.stderr:epoch 1 2024-06-03T08:36:04.205 INFO:teuthology.misc.health.smithi088.stderr:fsid ec3c3fdd-3ac6-4d7d-9412-683130ed8b04 2024-06-03T08:36:04.205 INFO:teuthology.misc.health.smithi088.stderr:last_changed 2024-06-03T08:34:31.885125+0000 2024-06-03T08:36:04.205 INFO:teuthology.misc.health.smithi088.stderr:created 2024-06-03T08:34:31.885125+0000 2024-06-03T08:36:04.205 INFO:teuthology.misc.health.smithi088.stderr:min_mon_release 18 (reef) 2024-06-03T08:36:04.205 INFO:teuthology.misc.health.smithi088.stderr:election_strategy: 3 2024-06-03T08:36:04.205 INFO:teuthology.misc.health.smithi088.stderr:0: v1:172.21.15.88:6789/0 mon.a 2024-06-03T08:36:04.206 INFO:teuthology.misc.health.smithi088.stderr:1: v1:172.21.15.192:6789/0 mon.b 2024-06-03T08:36:04.206 INFO:teuthology.misc.health.smithi088.stderr:2: v1:172.21.15.88:6790/0 mon.c 2024-06-03T08:36:04.206 INFO:teuthology.misc.health.smithi088.stderr: 2024-06-03T08:36:04.206 INFO:teuthology.misc.health.smithi088.stderr:2024-06-03T08:36:04.203+0000 7f3017fff640 4 set_mon_vals no callback set 2024-06-03T08:36:04.207 INFO:teuthology.misc.health.smithi088.stderr:2024-06-03T08:36:04.205+0000 7f30167fc640 1 -- 172.21.15.88:0/143279954 <== mon.1 v1:172.21.15.192:6789/0 7 ==== mgrmap(e 4) v1 ==== 89054+0+0 (unknown 914547237 0 0) 0x7f301c004c50 con 0x7f302013eed0 2024-06-03T08:36:04.207 INFO:teuthology.misc.health.smithi088.stderr:2024-06-03T08:36:04.205+0000 7f3025186640 10 monclient: get_auth_request con 0x7f300406d3b0 auth_method 0 2024-06-03T08:36:04.207 INFO:teuthology.misc.health.smithi088.stderr:2024-06-03T08:36:04.206+0000 7f30167fc640 10 client.4216.objecter ms_handle_connect 0x7f300406d3b0 2024-06-03T08:36:04.208 INFO:teuthology.misc.health.smithi088.stderr:2024-06-03T08:36:04.207+0000 7f30167fc640 1 -- 172.21.15.88:0/143279954 <== mon.1 v1:172.21.15.192:6789/0 8 ==== osd_map(10..10 src has 1..10) v4 ==== 1645+0+0 (unknown 309997979 0 0) 0x7f301c054110 con 0x7f302013eed0 2024-06-03T08:36:04.208 INFO:teuthology.misc.health.smithi088.stderr:2024-06-03T08:36:04.207+0000 7f30167fc640 10 client.4216.objecter ms_dispatch 0x7f3020000b90 osd_map(10..10 src has 1..10) v4 2024-06-03T08:36:04.208 INFO:teuthology.misc.health.smithi088.stderr:2024-06-03T08:36:04.207+0000 7f30167fc640 3 client.4216.objecter handle_osd_map got epochs [10,10] > 0 2024-06-03T08:36:04.208 INFO:teuthology.misc.health.smithi088.stderr:2024-06-03T08:36:04.207+0000 7f30167fc640 3 client.4216.objecter handle_osd_map decoding full epoch 10 2024-06-03T08:36:04.208 INFO:teuthology.misc.health.smithi088.stderr:2024-06-03T08:36:04.207+0000 7f30167fc640 20 client.4216.objecter dump_active .. 0 homeless 2024-06-03T08:36:04.507 INFO:teuthology.misc.health.smithi088.stderr:2024-06-03T08:36:04.507+0000 7f30167fc640 1 -- 172.21.15.88:0/143279954 <== mon.1 v1:172.21.15.192:6789/0 9 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+182498 (unknown 1092875540 0 3016760700) 0x7f301c04faf0 con 0x7f302013eed0 2024-06-03T08:36:04.507 INFO:teuthology.misc.health.smithi088.stderr:2024-06-03T08:36:04.507+0000 7f30167fc640 10 monclient: handle_mon_command_ack 1 [{"prefix": "get_command_descriptions"}] 2024-06-03T08:36:04.508 INFO:teuthology.misc.health.smithi088.stderr:2024-06-03T08:36:04.507+0000 7f30167fc640 10 monclient: _finish_command 1 = system:0 2024-06-03T08:36:04.718 INFO:teuthology.misc.health.smithi088.stderr:2024-06-03T08:36:04.718+0000 7f3027beb640 10 start_mon_command cmd=[{"prefix": "osd dump", "format": "json"}] 2024-06-03T08:36:04.718 INFO:teuthology.misc.health.smithi088.stderr:2024-06-03T08:36:04.718+0000 7f3027beb640 10 monclient: _send_command 2 [{"prefix": "osd dump", "format": "json"}] 2024-06-03T08:36:04.718 INFO:teuthology.misc.health.smithi088.stderr:2024-06-03T08:36:04.718+0000 7f3027beb640 10 monclient: _send_mon_message to mon.b at v1:172.21.15.192:6789/0 2024-06-03T08:36:04.719 INFO:teuthology.misc.health.smithi088.stderr:2024-06-03T08:36:04.718+0000 7f3027beb640 1 -- 172.21.15.88:0/143279954 --> v1:172.21.15.192:6789/0 -- mon_command({"prefix": "osd dump", "format": "json"} v 0) v1 -- 0x7f30200b3b20 con 0x7f302013eed0 2024-06-03T08:36:04.839 INFO:teuthology.misc.health.smithi088.stderr:2024-06-03T08:36:04.838+0000 7f30167fc640 1 -- 172.21.15.88:0/143279954 <== mon.1 v1:172.21.15.192:6789/0 10 ==== mon_command_ack([{"prefix": "osd dump", "format": "json"}]=0 v10) v1 ==== 74+0+7057 (unknown 4040575075 0 1067785724) 0x7f301c051bf0 con 0x7f302013eed0 2024-06-03T08:36:04.839 INFO:teuthology.misc.health.smithi088.stderr:2024-06-03T08:36:04.838+0000 7f30167fc640 10 monclient: handle_mon_command_ack 2 [{"prefix": "osd dump", "format": "json"}] 2024-06-03T08:36:04.839 INFO:teuthology.misc.health.smithi088.stderr:2024-06-03T08:36:04.838+0000 7f30167fc640 10 monclient: _finish_command 2 = system:0 2024-06-03T08:36:04.839 INFO:teuthology.misc.health.smithi088.stdout: 2024-06-03T08:36:04.839 INFO:teuthology.misc.health.smithi088.stdout:{"epoch":10,"fsid":"ec3c3fdd-3ac6-4d7d-9412-683130ed8b04","created":"2024-06-03T08:35:26.799638+0000","modified":"2024-06-03T08:35:32.183068+0000","last_up_change":"0.000000","last_in_change":"2024-06-03T08:35:32.183068+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":2,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":0,"max_osd":8,"require_min_compat_client":"luminous","min_compat_client":"jewel","require_osd_release":"reef","allow_crimson":false,"pools":[],"osds":[{"osd":0,"uuid":"b25ab3bb-3587-45d0-991a-6d41bb529b37","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":1,"uuid":"00d160e7-9773-41d8-8d17-21e70620a0fd","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":2,"uuid":"addfb2a7-2bde-4cc3-98cf-17fd2bdfa3a9","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":3,"uuid":"6c5770b0-c93c-4101-8c53-bd761b8abcde","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":4,"uuid":"f001b3cb-2081-428b-bac3-b3df3c19b978","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":5,"uuid":"3cc263e9-e9bf-45e0-a3b7-83a76469eb70","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":6,"uuid":"82c83316-809f-4901-a5ed-34b6a075b393","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":7,"uuid":"aafe0716-1102-4eae-aac6-0f81bb154e7d","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":4,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":5,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":6,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":7,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[],"pg_upmap_primaries":[],"pg_temp":[],"primary_temp":[],"blocklist":{},"range_blocklist":{},"erasure_code_profiles":{"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","technique":"reed_sol_van"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2024-06-03T08:36:04.841 INFO:teuthology.misc.health.smithi088.stderr:2024-06-03T08:36:04.840+0000 7f3027beb640 10 librados: watch_flush enter 2024-06-03T08:36:04.841 INFO:teuthology.misc.health.smithi088.stderr:2024-06-03T08:36:04.840+0000 7f3027beb640 10 librados: watch_flush exit 2024-06-03T08:36:04.841 INFO:teuthology.misc.health.smithi088.stderr:2024-06-03T08:36:04.840+0000 7f3027beb640 20 client.4216.objecter shutdown clearing up homeless session... 2024-06-03T08:36:04.841 INFO:teuthology.misc.health.smithi088.stderr:2024-06-03T08:36:04.840+0000 7f3027beb640 10 client.4216.objecter successfully canceled tick 2024-06-03T08:36:04.841 INFO:teuthology.misc.health.smithi088.stderr:2024-06-03T08:36:04.840+0000 7f3027beb640 1 -- 172.21.15.88:0/143279954 >> v1:172.21.15.192:6800/34229 conn(0x7f300406d3b0 legacy=0x7f300406f870 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-06-03T08:36:04.841 INFO:teuthology.misc.health.smithi088.stderr:2024-06-03T08:36:04.840+0000 7f3027beb640 10 monclient: shutdown 2024-06-03T08:36:04.841 INFO:teuthology.misc.health.smithi088.stderr:2024-06-03T08:36:04.840+0000 7f3027beb640 20 monclient: shutdown discarding 0 pending message(s) 2024-06-03T08:36:04.841 INFO:teuthology.misc.health.smithi088.stderr:2024-06-03T08:36:04.840+0000 7f3027beb640 1 -- 172.21.15.88:0/143279954 >> v1:172.21.15.192:6789/0 conn(0x7f302013eed0 legacy=0x7f30200aab40 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-06-03T08:36:04.841 INFO:teuthology.misc.health.smithi088.stderr:2024-06-03T08:36:04.840+0000 7f3027beb640 1 -- 172.21.15.88:0/143279954 shutdown_connections 2024-06-03T08:36:04.841 INFO:teuthology.misc.health.smithi088.stderr:2024-06-03T08:36:04.840+0000 7f3027beb640 1 -- 172.21.15.88:0/143279954 >> 172.21.15.88:0/143279954 conn(0x7f30200b7270 msgr2=0x7f30200b65f0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-06-03T08:36:04.841 INFO:teuthology.misc.health.smithi088.stderr:2024-06-03T08:36:04.840+0000 7f3027beb640 1 -- 172.21.15.88:0/143279954 shutdown_connections 2024-06-03T08:36:04.841 INFO:teuthology.misc.health.smithi088.stderr:2024-06-03T08:36:04.840+0000 7f3027beb640 1 -- 172.21.15.88:0/143279954 wait complete. 2024-06-03T08:36:04.842 INFO:teuthology.misc.health.smithi088.stderr:2024-06-03T08:36:04.840+0000 7f3027beb640 1 librados: shutdown 2024-06-03T08:36:04.852 DEBUG:teuthology.misc:0 of 8 OSDs are up 2024-06-03T08:36:07.656 INFO:tasks.ceph.osd.7.smithi192.stderr:2024-06-03T08:36:07.652+0000 5dcd500 -1 osd.7 0 log_to_monitors true 2024-06-03T08:36:07.659 INFO:tasks.ceph.osd.3.smithi088.stderr:2024-06-03T08:36:07.656+0000 5dcd500 -1 osd.3 0 log_to_monitors true 2024-06-03T08:36:07.669 INFO:tasks.ceph.osd.1.smithi088.stderr:2024-06-03T08:36:07.666+0000 5dcd500 -1 osd.1 0 log_to_monitors true 2024-06-03T08:36:07.921 INFO:tasks.ceph.osd.0.smithi088.stderr:2024-06-03T08:36:07.911+0000 5dcd500 -1 osd.0 0 log_to_monitors true 2024-06-03T08:36:08.121 INFO:tasks.ceph.osd.4.smithi192.stderr:2024-06-03T08:36:08.116+0000 5dcd500 -1 osd.4 0 log_to_monitors true 2024-06-03T08:36:08.557 INFO:tasks.ceph.osd.5.smithi192.stderr:2024-06-03T08:36:08.554+0000 5dcd500 -1 osd.5 0 log_to_monitors true 2024-06-03T08:36:08.854 INFO:tasks.ceph.osd.2.smithi088.stderr:2024-06-03T08:36:08.852+0000 5dcd500 -1 osd.2 0 log_to_monitors true 2024-06-03T08:36:08.900 INFO:tasks.ceph.osd.6.smithi192.stderr:2024-06-03T08:36:08.897+0000 5dcd500 -1 osd.6 0 log_to_monitors true 2024-06-03T08:36:11.396 INFO:tasks.ceph.osd.0.smithi088.stderr:2024-06-03T08:36:11.390+0000 c4a2640 -1 osd.0 0 waiting for initial osdmap 2024-06-03T08:36:11.432 INFO:tasks.ceph.osd.1.smithi088.stderr:2024-06-03T08:36:11.428+0000 c4a2640 -1 osd.1 0 waiting for initial osdmap 2024-06-03T08:36:11.462 INFO:tasks.ceph.osd.3.smithi088.stderr:2024-06-03T08:36:11.446+0000 c4a2640 -1 osd.3 0 waiting for initial osdmap 2024-06-03T08:36:11.658 DEBUG:teuthology.orchestra.run.smithi088:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph --cluster ceph osd dump --format=json 2024-06-03T08:36:11.753 INFO:teuthology.misc.health.smithi088.stderr:2024-06-03T08:36:11.752+0000 7f84631f5640 10 monclient: get_monmap_and_config 2024-06-03T08:36:11.753 INFO:teuthology.misc.health.smithi088.stderr:2024-06-03T08:36:11.752+0000 7f84631f5640 10 monclient: build_initial_monmap 2024-06-03T08:36:11.753 INFO:teuthology.misc.health.smithi088.stderr:2024-06-03T08:36:11.752+0000 7f84631f5640 10 monclient: monmap: 2024-06-03T08:36:11.753 INFO:teuthology.misc.health.smithi088.stderr:epoch 0 2024-06-03T08:36:11.754 INFO:teuthology.misc.health.smithi088.stderr:fsid ec3c3fdd-3ac6-4d7d-9412-683130ed8b04 2024-06-03T08:36:11.754 INFO:teuthology.misc.health.smithi088.stderr:last_changed 2024-06-03T08:36:11.753964+0000 2024-06-03T08:36:11.754 INFO:teuthology.misc.health.smithi088.stderr:created 2024-06-03T08:36:11.753964+0000 2024-06-03T08:36:11.754 INFO:teuthology.misc.health.smithi088.stderr:min_mon_release 0 (unknown) 2024-06-03T08:36:11.754 INFO:teuthology.misc.health.smithi088.stderr:election_strategy: 3 2024-06-03T08:36:11.754 INFO:teuthology.misc.health.smithi088.stderr:0: v1:172.21.15.88:6789/0 mon.noname-a 2024-06-03T08:36:11.754 INFO:teuthology.misc.health.smithi088.stderr:1: v1:172.21.15.192:6789/0 mon.noname-c 2024-06-03T08:36:11.754 INFO:teuthology.misc.health.smithi088.stderr:2: v1:172.21.15.88:6790/0 mon.noname-b 2024-06-03T08:36:11.754 INFO:teuthology.misc.health.smithi088.stderr: 2024-06-03T08:36:11.756 INFO:teuthology.misc.health.smithi088.stderr:2024-06-03T08:36:11.753+0000 7f84631f5640 1 Processor -- start 2024-06-03T08:36:11.756 INFO:teuthology.misc.health.smithi088.stderr:2024-06-03T08:36:11.753+0000 7f84631f5640 1 -- start start 2024-06-03T08:36:11.756 INFO:teuthology.misc.health.smithi088.stderr:2024-06-03T08:36:11.753+0000 7f84631f5640 10 monclient: init 2024-06-03T08:36:11.756 INFO:teuthology.misc.health.smithi088.stderr:2024-06-03T08:36:11.754+0000 7f84631f5640 10 monclient: _reopen_session rank -1 2024-06-03T08:36:11.756 INFO:teuthology.misc.health.smithi088.stderr:2024-06-03T08:36:11.754+0000 7f84631f5640 10 monclient: _add_conns ranks=[1,0,2] 2024-06-03T08:36:11.756 INFO:teuthology.misc.health.smithi088.stderr:2024-06-03T08:36:11.754+0000 7f84631f5640 10 monclient(hunting): picked mon.noname-c con 0x7f845c06fce0 addr v1:172.21.15.192:6789/0 2024-06-03T08:36:11.756 INFO:teuthology.misc.health.smithi088.stderr:2024-06-03T08:36:11.754+0000 7f84631f5640 10 monclient(hunting): picked mon.noname-a con 0x7f845c0ad920 addr v1:172.21.15.88:6789/0 2024-06-03T08:36:11.757 INFO:teuthology.misc.health.smithi088.stderr:2024-06-03T08:36:11.754+0000 7f84631f5640 10 monclient(hunting): picked mon.noname-b con 0x7f845c13c8e0 addr v1:172.21.15.88:6790/0 2024-06-03T08:36:11.757 INFO:teuthology.misc.health.smithi088.stderr:2024-06-03T08:36:11.754+0000 7f84631f5640 1 -- --> v1:172.21.15.88:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f845c071c50 con 0x7f845c0ad920 2024-06-03T08:36:11.757 INFO:teuthology.misc.health.smithi088.stderr:2024-06-03T08:36:11.754+0000 7f84631f5640 1 -- --> v1:172.21.15.192:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f845c071f30 con 0x7f845c06fce0 2024-06-03T08:36:11.757 INFO:teuthology.misc.health.smithi088.stderr:2024-06-03T08:36:11.754+0000 7f84631f5640 1 -- --> v1:172.21.15.88:6790/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f845c13cdf0 con 0x7f845c13c8e0 2024-06-03T08:36:11.757 INFO:teuthology.misc.health.smithi088.stderr:2024-06-03T08:36:11.754+0000 7f84631f5640 10 monclient(hunting): _renew_subs 2024-06-03T08:36:11.757 INFO:teuthology.misc.health.smithi088.stderr:2024-06-03T08:36:11.754+0000 7f84631f5640 10 monclient(hunting): authenticate will time out at 1227.564388s 2024-06-03T08:36:11.757 INFO:teuthology.misc.health.smithi088.stderr:2024-06-03T08:36:11.754+0000 7f845bfff640 1 --1- >> v1:172.21.15.88:6789/0 conn(0x7f845c0ad920 0x7f845c13c1d0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.88:6789/0 says I am v1:172.21.15.88:36632/0 (socket says 172.21.15.88:36632) 2024-06-03T08:36:11.757 INFO:teuthology.misc.health.smithi088.stderr:2024-06-03T08:36:11.755+0000 7f845bfff640 1 -- 172.21.15.88:0/1134228046 learned_addr learned my addr 172.21.15.88:0/1134228046 (peer_addr_for_me v1:172.21.15.88:0/0) 2024-06-03T08:36:11.759 INFO:teuthology.misc.health.smithi088.stderr:2024-06-03T08:36:11.757+0000 7f845b7fe640 1 -- 172.21.15.88:0/1134228046 <== mon.2 v1:172.21.15.88:6790/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 917145637 0 0) 0x7f845c13cdf0 con 0x7f845c13c8e0 2024-06-03T08:36:11.759 INFO:teuthology.misc.health.smithi088.stderr:2024-06-03T08:36:11.757+0000 7f845b7fe640 10 monclient(hunting): _negotiate 2024-06-03T08:36:11.759 INFO:teuthology.misc.health.smithi088.stderr:2024-06-03T08:36:11.757+0000 7f845b7fe640 10 monclient(hunting): _init_auth method 2 2024-06-03T08:36:11.760 INFO:teuthology.misc.health.smithi088.stderr:2024-06-03T08:36:11.757+0000 7f845b7fe640 10 monclient(hunting): _init_auth creating new auth 2024-06-03T08:36:11.760 INFO:teuthology.misc.health.smithi088.stderr:2024-06-03T08:36:11.757+0000 7f845b7fe640 10 monclient(hunting): my global_id is 4256 2024-06-03T08:36:11.760 INFO:teuthology.misc.health.smithi088.stderr:2024-06-03T08:36:11.757+0000 7f845b7fe640 1 -- 172.21.15.88:0/1134228046 --> v1:172.21.15.88:6790/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f844c003520 con 0x7f845c13c8e0 2024-06-03T08:36:11.760 INFO:teuthology.misc.health.smithi088.stderr:2024-06-03T08:36:11.757+0000 7f845b7fe640 1 -- 172.21.15.88:0/1134228046 <== mon.0 v1:172.21.15.88:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 4049762435 0 0) 0x7f845c071c50 con 0x7f845c0ad920 2024-06-03T08:36:11.760 INFO:teuthology.misc.health.smithi088.stderr:2024-06-03T08:36:11.757+0000 7f845b7fe640 10 monclient(hunting): _negotiate 2024-06-03T08:36:11.760 INFO:teuthology.misc.health.smithi088.stderr:2024-06-03T08:36:11.757+0000 7f845b7fe640 10 monclient(hunting): _init_auth method 2 2024-06-03T08:36:11.760 INFO:teuthology.misc.health.smithi088.stderr:2024-06-03T08:36:11.757+0000 7f845b7fe640 10 monclient(hunting): _init_auth creating new auth 2024-06-03T08:36:11.760 INFO:teuthology.misc.health.smithi088.stderr:2024-06-03T08:36:11.757+0000 7f845b7fe640 10 monclient(hunting): my global_id is 4251 2024-06-03T08:36:11.760 INFO:teuthology.misc.health.smithi088.stderr:2024-06-03T08:36:11.757+0000 7f845b7fe640 1 -- 172.21.15.88:0/1134228046 --> v1:172.21.15.88:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f845c13cdf0 con 0x7f845c0ad920 2024-06-03T08:36:11.760 INFO:teuthology.misc.health.smithi088.stderr:2024-06-03T08:36:11.758+0000 7f845b7fe640 1 -- 172.21.15.88:0/1134228046 <== mon.1 v1:172.21.15.192:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 802038307 0 0) 0x7f845c071f30 con 0x7f845c06fce0 2024-06-03T08:36:11.760 INFO:teuthology.misc.health.smithi088.stderr:2024-06-03T08:36:11.758+0000 7f845b7fe640 10 monclient(hunting): _negotiate 2024-06-03T08:36:11.761 INFO:teuthology.misc.health.smithi088.stderr:2024-06-03T08:36:11.758+0000 7f845b7fe640 10 monclient(hunting): _init_auth method 2 2024-06-03T08:36:11.761 INFO:teuthology.misc.health.smithi088.stderr:2024-06-03T08:36:11.758+0000 7f845b7fe640 10 monclient(hunting): _init_auth creating new auth 2024-06-03T08:36:11.761 INFO:teuthology.misc.health.smithi088.stderr:2024-06-03T08:36:11.758+0000 7f845b7fe640 10 monclient(hunting): my global_id is 4243 2024-06-03T08:36:11.761 INFO:teuthology.misc.health.smithi088.stderr:2024-06-03T08:36:11.758+0000 7f845b7fe640 1 -- 172.21.15.88:0/1134228046 --> v1:172.21.15.192:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f845c071c50 con 0x7f845c06fce0 2024-06-03T08:36:11.761 INFO:teuthology.misc.health.smithi088.stderr:2024-06-03T08:36:11.761+0000 7f845b7fe640 1 -- 172.21.15.88:0/1134228046 <== mon.0 v1:172.21.15.88:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 214+0+0 (unknown 1845318953 0 0) 0x7f845c13cdf0 con 0x7f845c0ad920 2024-06-03T08:36:11.761 INFO:teuthology.misc.health.smithi088.stderr:2024-06-03T08:36:11.761+0000 7f845b7fe640 1 -- 172.21.15.88:0/1134228046 >> v1:172.21.15.88:6790/0 conn(0x7f845c13c8e0 legacy=0x7f845c14abd0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-06-03T08:36:11.761 INFO:teuthology.misc.health.smithi088.stderr:2024-06-03T08:36:11.761+0000 7f845b7fe640 1 -- 172.21.15.88:0/1134228046 >> v1:172.21.15.192:6789/0 conn(0x7f845c06fce0 legacy=0x7f845c13fb30 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-06-03T08:36:11.762 INFO:teuthology.misc.health.smithi088.stderr:2024-06-03T08:36:11.761+0000 7f845b7fe640 10 monclient: _finish_hunting 0 2024-06-03T08:36:11.762 INFO:teuthology.misc.health.smithi088.stderr:2024-06-03T08:36:11.761+0000 7f845b7fe640 1 monclient: found mon.noname-a 2024-06-03T08:36:11.762 INFO:teuthology.misc.health.smithi088.stderr:2024-06-03T08:36:11.761+0000 7f845b7fe640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-06-03T08:36:11.762 INFO:teuthology.misc.health.smithi088.stderr:2024-06-03T08:36:11.761+0000 7f845b7fe640 10 monclient: _send_mon_message to mon.noname-a at v1:172.21.15.88:6789/0 2024-06-03T08:36:11.762 INFO:teuthology.misc.health.smithi088.stderr:2024-06-03T08:36:11.761+0000 7f845b7fe640 1 -- 172.21.15.88:0/1134228046 --> v1:172.21.15.88:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f845c0add40 con 0x7f845c0ad920 2024-06-03T08:36:11.762 INFO:teuthology.misc.health.smithi088.stderr:2024-06-03T08:36:11.761+0000 7f845b7fe640 10 monclient: _finish_auth 0 2024-06-03T08:36:11.762 INFO:teuthology.misc.health.smithi088.stderr:2024-06-03T08:36:11.761+0000 7f845b7fe640 10 monclient: _check_auth_tickets 2024-06-03T08:36:11.762 INFO:teuthology.misc.health.smithi088.stderr:2024-06-03T08:36:11.761+0000 7f845b7fe640 20 monclient: _check_auth_rotating not needed by client.admin 2024-06-03T08:36:11.762 INFO:teuthology.misc.health.smithi088.stderr:2024-06-03T08:36:11.761+0000 7f845b7fe640 1 -- 172.21.15.88:0/1134228046 <== mon.0 v1:172.21.15.88:6789/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 2234374065 0 0) 0x7f84480015f0 con 0x7f845c0ad920 2024-06-03T08:36:11.762 INFO:teuthology.misc.health.smithi088.stderr:2024-06-03T08:36:11.761+0000 7f845b7fe640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-06-03T08:36:11.762 INFO:teuthology.misc.health.smithi088.stderr:2024-06-03T08:36:11.761+0000 7f845b7fe640 10 monclient: got monmap 1 from mon.noname-a (according to old e1) 2024-06-03T08:36:11.763 INFO:teuthology.misc.health.smithi088.stderr:2024-06-03T08:36:11.761+0000 7f845b7fe640 10 monclient: dump: 2024-06-03T08:36:11.763 INFO:teuthology.misc.health.smithi088.stderr:epoch 1 2024-06-03T08:36:11.763 INFO:teuthology.misc.health.smithi088.stderr:fsid ec3c3fdd-3ac6-4d7d-9412-683130ed8b04 2024-06-03T08:36:11.763 INFO:teuthology.misc.health.smithi088.stderr:last_changed 2024-06-03T08:34:31.885125+0000 2024-06-03T08:36:11.763 INFO:teuthology.misc.health.smithi088.stderr:created 2024-06-03T08:34:31.885125+0000 2024-06-03T08:36:11.763 INFO:teuthology.misc.health.smithi088.stderr:min_mon_release 18 (reef) 2024-06-03T08:36:11.763 INFO:teuthology.misc.health.smithi088.stderr:election_strategy: 3 2024-06-03T08:36:11.763 INFO:teuthology.misc.health.smithi088.stderr:0: v1:172.21.15.88:6789/0 mon.a 2024-06-03T08:36:11.763 INFO:teuthology.misc.health.smithi088.stderr:1: v1:172.21.15.192:6789/0 mon.b 2024-06-03T08:36:11.763 INFO:teuthology.misc.health.smithi088.stderr:2: v1:172.21.15.88:6790/0 mon.c 2024-06-03T08:36:11.763 INFO:teuthology.misc.health.smithi088.stderr: 2024-06-03T08:36:11.764 INFO:teuthology.misc.health.smithi088.stderr:2024-06-03T08:36:11.761+0000 7f84631f5640 5 monclient: authenticate success, global_id 4251 2024-06-03T08:36:11.764 INFO:teuthology.misc.health.smithi088.stderr:2024-06-03T08:36:11.761+0000 7f84631f5640 20 monclient: get_monmap_and_config waiting for monmap|config 2024-06-03T08:36:11.765 INFO:teuthology.misc.health.smithi088.stderr:2024-06-03T08:36:11.762+0000 7f845b7fe640 1 -- 172.21.15.88:0/1134228046 <== mon.0 v1:172.21.15.88:6789/0 4 ==== config(0 keys) v1 ==== 4+0+0 (unknown 0 0 0) 0x7f8448001810 con 0x7f845c0ad920 2024-06-03T08:36:11.765 INFO:teuthology.misc.health.smithi088.stderr:2024-06-03T08:36:11.762+0000 7f845b7fe640 10 monclient: handle_config config(0 keys) v1 2024-06-03T08:36:11.765 INFO:teuthology.misc.health.smithi088.stderr:2024-06-03T08:36:11.762+0000 7f84631f5640 10 monclient: get_monmap_and_config success 2024-06-03T08:36:11.765 INFO:teuthology.misc.health.smithi088.stderr:2024-06-03T08:36:11.762+0000 7f84631f5640 4 set_mon_vals no callback set 2024-06-03T08:36:11.765 INFO:teuthology.misc.health.smithi088.stderr:2024-06-03T08:36:11.763+0000 7f845b7fe640 1 -- 172.21.15.88:0/1134228046 <== mon.0 v1:172.21.15.88:6789/0 5 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 2234374065 0 0) 0x7f8448001ba0 con 0x7f845c0ad920 2024-06-03T08:36:11.765 INFO:teuthology.misc.health.smithi088.stderr:2024-06-03T08:36:11.763+0000 7f845b7fe640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-06-03T08:36:11.766 INFO:teuthology.misc.health.smithi088.stderr:2024-06-03T08:36:11.763+0000 7f845b7fe640 10 monclient: got monmap 1 from mon.a (according to old e1) 2024-06-03T08:36:11.766 INFO:teuthology.misc.health.smithi088.stderr:2024-06-03T08:36:11.763+0000 7f845b7fe640 10 monclient: dump: 2024-06-03T08:36:11.766 INFO:teuthology.misc.health.smithi088.stderr:epoch 1 2024-06-03T08:36:11.766 INFO:teuthology.misc.health.smithi088.stderr:fsid ec3c3fdd-3ac6-4d7d-9412-683130ed8b04 2024-06-03T08:36:11.766 INFO:teuthology.misc.health.smithi088.stderr:last_changed 2024-06-03T08:34:31.885125+0000 2024-06-03T08:36:11.766 INFO:teuthology.misc.health.smithi088.stderr:created 2024-06-03T08:34:31.885125+0000 2024-06-03T08:36:11.766 INFO:teuthology.misc.health.smithi088.stderr:min_mon_release 18 (reef) 2024-06-03T08:36:11.766 INFO:teuthology.misc.health.smithi088.stderr:election_strategy: 3 2024-06-03T08:36:11.766 INFO:teuthology.misc.health.smithi088.stderr:0: v1:172.21.15.88:6789/0 mon.a 2024-06-03T08:36:11.767 INFO:teuthology.misc.health.smithi088.stderr:1: v1:172.21.15.192:6789/0 mon.b 2024-06-03T08:36:11.767 INFO:teuthology.misc.health.smithi088.stderr:2: v1:172.21.15.88:6790/0 mon.c 2024-06-03T08:36:11.767 INFO:teuthology.misc.health.smithi088.stderr: 2024-06-03T08:36:11.767 INFO:teuthology.misc.health.smithi088.stderr:2024-06-03T08:36:11.763+0000 7f84631f5640 10 monclient: shutdown 2024-06-03T08:36:11.767 INFO:teuthology.misc.health.smithi088.stderr:2024-06-03T08:36:11.763+0000 7f84631f5640 20 monclient: shutdown discarding 0 pending message(s) 2024-06-03T08:36:11.767 INFO:teuthology.misc.health.smithi088.stderr:2024-06-03T08:36:11.763+0000 7f84631f5640 1 -- 172.21.15.88:0/1134228046 >> v1:172.21.15.88:6789/0 conn(0x7f845c0ad920 legacy=0x7f845c13c1d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-06-03T08:36:11.767 INFO:teuthology.misc.health.smithi088.stderr:2024-06-03T08:36:11.763+0000 7f84631f5640 1 -- 172.21.15.88:0/1134228046 shutdown_connections 2024-06-03T08:36:11.767 INFO:teuthology.misc.health.smithi088.stderr:2024-06-03T08:36:11.763+0000 7f84631f5640 1 -- 172.21.15.88:0/1134228046 >> 172.21.15.88:0/1134228046 conn(0x7f845c0b2f20 msgr2=0x7f845c0b3300 unknown :-1 s=STATE_NONE l=0).mark_down 2024-06-03T08:36:11.768 INFO:teuthology.misc.health.smithi088.stderr:2024-06-03T08:36:11.763+0000 7f84631f5640 1 -- 172.21.15.88:0/1134228046 shutdown_connections 2024-06-03T08:36:11.768 INFO:teuthology.misc.health.smithi088.stderr:2024-06-03T08:36:11.763+0000 7f84631f5640 1 -- 172.21.15.88:0/1134228046 wait complete. 2024-06-03T08:36:11.768 INFO:teuthology.misc.health.smithi088.stderr:2024-06-03T08:36:11.764+0000 7f84631f5640 10 monclient: build_initial_monmap 2024-06-03T08:36:11.768 INFO:teuthology.misc.health.smithi088.stderr:2024-06-03T08:36:11.764+0000 7f84631f5640 10 monclient: monmap: 2024-06-03T08:36:11.768 INFO:teuthology.misc.health.smithi088.stderr:epoch 0 2024-06-03T08:36:11.768 INFO:teuthology.misc.health.smithi088.stderr:fsid 00000000-0000-0000-0000-000000000000 2024-06-03T08:36:11.768 INFO:teuthology.misc.health.smithi088.stderr:last_changed 0.000000 2024-06-03T08:36:11.768 INFO:teuthology.misc.health.smithi088.stderr:created 0.000000 2024-06-03T08:36:11.768 INFO:teuthology.misc.health.smithi088.stderr:min_mon_release 0 (unknown) 2024-06-03T08:36:11.768 INFO:teuthology.misc.health.smithi088.stderr:election_strategy: 1 2024-06-03T08:36:11.768 INFO:teuthology.misc.health.smithi088.stderr:0: v1:172.21.15.88:6789/0 mon.noname-a 2024-06-03T08:36:11.768 INFO:teuthology.misc.health.smithi088.stderr:1: v1:172.21.15.192:6789/0 mon.noname-b 2024-06-03T08:36:11.769 INFO:teuthology.misc.health.smithi088.stderr:2: v1:172.21.15.88:6790/0 mon.noname-c 2024-06-03T08:36:11.769 INFO:teuthology.misc.health.smithi088.stderr: 2024-06-03T08:36:11.769 INFO:teuthology.misc.health.smithi088.stderr:2024-06-03T08:36:11.764+0000 7f84631f5640 1 librados: starting msgr at 2024-06-03T08:36:11.769 INFO:teuthology.misc.health.smithi088.stderr:2024-06-03T08:36:11.764+0000 7f84631f5640 1 librados: starting objecter 2024-06-03T08:36:11.769 INFO:teuthology.misc.health.smithi088.stderr:2024-06-03T08:36:11.764+0000 7f84631f5640 1 Processor -- start 2024-06-03T08:36:11.769 INFO:teuthology.misc.health.smithi088.stderr:2024-06-03T08:36:11.764+0000 7f84631f5640 1 -- start start 2024-06-03T08:36:11.769 INFO:teuthology.misc.health.smithi088.stderr:2024-06-03T08:36:11.764+0000 7f84631f5640 1 librados: setting wanted keys 2024-06-03T08:36:11.769 INFO:teuthology.misc.health.smithi088.stderr:2024-06-03T08:36:11.764+0000 7f84631f5640 1 librados: calling monclient init 2024-06-03T08:36:11.769 INFO:teuthology.misc.health.smithi088.stderr:2024-06-03T08:36:11.764+0000 7f84631f5640 10 monclient: init 2024-06-03T08:36:11.772 INFO:teuthology.misc.health.smithi088.stderr:2024-06-03T08:36:11.765+0000 7f84631f5640 10 monclient: _reopen_session rank -1 2024-06-03T08:36:11.772 INFO:teuthology.misc.health.smithi088.stderr:2024-06-03T08:36:11.765+0000 7f84631f5640 10 monclient: _add_conns ranks=[0,1,2] 2024-06-03T08:36:11.772 INFO:teuthology.misc.health.smithi088.stderr:2024-06-03T08:36:11.765+0000 7f84631f5640 10 monclient(hunting): picked mon.noname-a con 0x7f845c06fce0 addr v1:172.21.15.88:6789/0 2024-06-03T08:36:11.772 INFO:teuthology.misc.health.smithi088.stderr:2024-06-03T08:36:11.765+0000 7f84631f5640 10 monclient(hunting): picked mon.noname-b con 0x7f845c0ad920 addr v1:172.21.15.192:6789/0 2024-06-03T08:36:11.773 INFO:teuthology.misc.health.smithi088.stderr:2024-06-03T08:36:11.765+0000 7f84631f5640 10 monclient(hunting): picked mon.noname-c con 0x7f845c13c8e0 addr v1:172.21.15.88:6790/0 2024-06-03T08:36:11.773 INFO:teuthology.misc.health.smithi088.stderr:2024-06-03T08:36:11.765+0000 7f84631f5640 1 -- --> v1:172.21.15.88:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f845c06dc80 con 0x7f845c06fce0 2024-06-03T08:36:11.773 INFO:teuthology.misc.health.smithi088.stderr:2024-06-03T08:36:11.765+0000 7f84631f5640 1 -- --> v1:172.21.15.192:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f845c12aa10 con 0x7f845c0ad920 2024-06-03T08:36:11.773 INFO:teuthology.misc.health.smithi088.stderr:2024-06-03T08:36:11.765+0000 7f84631f5640 1 -- --> v1:172.21.15.88:6790/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f845c13e4a0 con 0x7f845c13c8e0 2024-06-03T08:36:11.773 INFO:teuthology.misc.health.smithi088.stderr:2024-06-03T08:36:11.765+0000 7f84631f5640 10 monclient(hunting): _renew_subs 2024-06-03T08:36:11.773 INFO:teuthology.misc.health.smithi088.stderr:2024-06-03T08:36:11.765+0000 7f84631f5640 10 monclient(hunting): authenticate will time out at 1227.575915s 2024-06-03T08:36:11.773 INFO:teuthology.misc.health.smithi088.stderr:2024-06-03T08:36:11.766+0000 7f8460f91640 1 --1- >> v1:172.21.15.88:6789/0 conn(0x7f845c06fce0 0x7f845c1d9930 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=1).handle_server_banner_and_identify peer v1:172.21.15.88:6789/0 says I am v1:172.21.15.88:36648/0 (socket says 172.21.15.88:36648) 2024-06-03T08:36:11.773 INFO:teuthology.misc.health.smithi088.stderr:2024-06-03T08:36:11.766+0000 7f8460f91640 1 -- 172.21.15.88:0/2877681936 learned_addr learned my addr 172.21.15.88:0/2877681936 (peer_addr_for_me v1:172.21.15.88:0/0) 2024-06-03T08:36:11.773 INFO:teuthology.misc.health.smithi088.stderr:2024-06-03T08:36:11.766+0000 7f84597fa640 10 client.?.objecter ms_handle_connect 0x7f845c06fce0 2024-06-03T08:36:11.773 INFO:teuthology.misc.health.smithi088.stderr:2024-06-03T08:36:11.766+0000 7f84597fa640 10 client.?.objecter resend_mon_ops 2024-06-03T08:36:11.773 INFO:teuthology.misc.health.smithi088.stderr:2024-06-03T08:36:11.766+0000 7f84597fa640 10 client.?.objecter ms_handle_connect 0x7f845c0ad920 2024-06-03T08:36:11.774 INFO:teuthology.misc.health.smithi088.stderr:2024-06-03T08:36:11.766+0000 7f84597fa640 10 client.?.objecter resend_mon_ops 2024-06-03T08:36:11.774 INFO:teuthology.misc.health.smithi088.stderr:2024-06-03T08:36:11.767+0000 7f84597fa640 10 client.?.objecter ms_handle_connect 0x7f845c13c8e0 2024-06-03T08:36:11.774 INFO:teuthology.misc.health.smithi088.stderr:2024-06-03T08:36:11.767+0000 7f84597fa640 10 client.?.objecter resend_mon_ops 2024-06-03T08:36:11.774 INFO:teuthology.misc.health.smithi088.stderr:2024-06-03T08:36:11.768+0000 7f84597fa640 1 -- 172.21.15.88:0/2877681936 <== mon.0 v1:172.21.15.88:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 1695211810 0 0) 0x7f845c06dc80 con 0x7f845c06fce0 2024-06-03T08:36:11.774 INFO:teuthology.misc.health.smithi088.stderr:2024-06-03T08:36:11.768+0000 7f84597fa640 10 monclient(hunting): _negotiate 2024-06-03T08:36:11.774 INFO:teuthology.misc.health.smithi088.stderr:2024-06-03T08:36:11.768+0000 7f84597fa640 10 monclient(hunting): _init_auth method 2 2024-06-03T08:36:11.774 INFO:teuthology.misc.health.smithi088.stderr:2024-06-03T08:36:11.768+0000 7f84597fa640 10 monclient(hunting): _init_auth creating new auth 2024-06-03T08:36:11.774 INFO:teuthology.misc.health.smithi088.stderr:2024-06-03T08:36:11.768+0000 7f84597fa640 10 monclient(hunting): my global_id is 4254 2024-06-03T08:36:11.774 INFO:teuthology.misc.health.smithi088.stderr:2024-06-03T08:36:11.768+0000 7f84597fa640 1 -- 172.21.15.88:0/2877681936 --> v1:172.21.15.88:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f8440004eb0 con 0x7f845c06fce0 2024-06-03T08:36:11.774 INFO:teuthology.misc.health.smithi088.stderr:2024-06-03T08:36:11.769+0000 7f84597fa640 1 -- 172.21.15.88:0/2877681936 <== mon.2 v1:172.21.15.88:6790/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 3887331458 0 0) 0x7f845c13e4a0 con 0x7f845c13c8e0 2024-06-03T08:36:11.774 INFO:teuthology.misc.health.smithi088.stderr:2024-06-03T08:36:11.769+0000 7f84597fa640 10 monclient(hunting): _negotiate 2024-06-03T08:36:11.774 INFO:teuthology.misc.health.smithi088.stderr:2024-06-03T08:36:11.769+0000 7f84597fa640 10 monclient(hunting): _init_auth method 2 2024-06-03T08:36:11.775 INFO:teuthology.misc.health.smithi088.stderr:2024-06-03T08:36:11.769+0000 7f84597fa640 10 monclient(hunting): _init_auth creating new auth 2024-06-03T08:36:11.775 INFO:teuthology.misc.health.smithi088.stderr:2024-06-03T08:36:11.769+0000 7f84597fa640 10 monclient(hunting): my global_id is 4259 2024-06-03T08:36:11.775 INFO:teuthology.misc.health.smithi088.stderr:2024-06-03T08:36:11.769+0000 7f84597fa640 1 -- 172.21.15.88:0/2877681936 --> v1:172.21.15.88:6790/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f845c06dc80 con 0x7f845c13c8e0 2024-06-03T08:36:11.775 INFO:teuthology.misc.health.smithi088.stderr:2024-06-03T08:36:11.769+0000 7f84597fa640 1 -- 172.21.15.88:0/2877681936 <== mon.1 v1:172.21.15.192:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 2850860429 0 0) 0x7f845c12aa10 con 0x7f845c0ad920 2024-06-03T08:36:11.775 INFO:teuthology.misc.health.smithi088.stderr:2024-06-03T08:36:11.769+0000 7f84597fa640 10 monclient(hunting): _negotiate 2024-06-03T08:36:11.775 INFO:teuthology.misc.health.smithi088.stderr:2024-06-03T08:36:11.769+0000 7f84597fa640 10 monclient(hunting): _init_auth method 2 2024-06-03T08:36:11.775 INFO:teuthology.misc.health.smithi088.stderr:2024-06-03T08:36:11.769+0000 7f84597fa640 10 monclient(hunting): _init_auth creating new auth 2024-06-03T08:36:11.775 INFO:teuthology.misc.health.smithi088.stderr:2024-06-03T08:36:11.769+0000 7f84597fa640 10 monclient(hunting): my global_id is 4246 2024-06-03T08:36:11.775 INFO:teuthology.misc.health.smithi088.stderr:2024-06-03T08:36:11.769+0000 7f84597fa640 1 -- 172.21.15.88:0/2877681936 --> v1:172.21.15.192:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f845c13e4a0 con 0x7f845c0ad920 2024-06-03T08:36:11.777 INFO:teuthology.misc.health.smithi088.stderr:2024-06-03T08:36:11.774+0000 7f84597fa640 1 -- 172.21.15.88:0/2877681936 <== mon.2 v1:172.21.15.88:6790/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 764+0+0 (unknown 3829287316 0 0) 0x7f845c06dc80 con 0x7f845c13c8e0 2024-06-03T08:36:11.777 INFO:teuthology.misc.health.smithi088.stderr:2024-06-03T08:36:11.774+0000 7f84597fa640 1 -- 172.21.15.88:0/2877681936 --> v1:172.21.15.88:6790/0 -- auth(proto 2 165 bytes epoch 0) v1 -- 0x7f845c12aa10 con 0x7f845c13c8e0 2024-06-03T08:36:11.777 INFO:teuthology.misc.health.smithi088.stderr:2024-06-03T08:36:11.774+0000 7f84597fa640 1 -- 172.21.15.88:0/2877681936 <== mon.2 v1:172.21.15.88:6790/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 2234374065 0 0) 0x7f8454001b50 con 0x7f845c13c8e0 2024-06-03T08:36:11.777 INFO:teuthology.misc.health.smithi088.stderr:2024-06-03T08:36:11.774+0000 7f84597fa640 10 monclient(hunting): handle_monmap mon_map magic: 0 v1 2024-06-03T08:36:11.777 INFO:teuthology.misc.health.smithi088.stderr:2024-06-03T08:36:11.774+0000 7f84597fa640 10 monclient(hunting): got monmap 1 from mon.noname-c (according to old e1) 2024-06-03T08:36:11.777 INFO:teuthology.misc.health.smithi088.stderr:2024-06-03T08:36:11.774+0000 7f84597fa640 10 monclient(hunting): dump: 2024-06-03T08:36:11.777 INFO:teuthology.misc.health.smithi088.stderr:epoch 1 2024-06-03T08:36:11.777 INFO:teuthology.misc.health.smithi088.stderr:fsid ec3c3fdd-3ac6-4d7d-9412-683130ed8b04 2024-06-03T08:36:11.777 INFO:teuthology.misc.health.smithi088.stderr:last_changed 2024-06-03T08:34:31.885125+0000 2024-06-03T08:36:11.777 INFO:teuthology.misc.health.smithi088.stderr:created 2024-06-03T08:34:31.885125+0000 2024-06-03T08:36:11.778 INFO:teuthology.misc.health.smithi088.stderr:min_mon_release 18 (reef) 2024-06-03T08:36:11.778 INFO:teuthology.misc.health.smithi088.stderr:election_strategy: 3 2024-06-03T08:36:11.778 INFO:teuthology.misc.health.smithi088.stderr:0: v1:172.21.15.88:6789/0 mon.a 2024-06-03T08:36:11.778 INFO:teuthology.misc.health.smithi088.stderr:1: v1:172.21.15.192:6789/0 mon.b 2024-06-03T08:36:11.778 INFO:teuthology.misc.health.smithi088.stderr:2: v1:172.21.15.88:6790/0 mon.c 2024-06-03T08:36:11.778 INFO:teuthology.misc.health.smithi088.stderr: 2024-06-03T08:36:11.778 INFO:teuthology.misc.health.smithi088.stderr:2024-06-03T08:36:11.774+0000 7f84597fa640 10 monclient(hunting): _finish_auth 0 2024-06-03T08:36:11.778 INFO:teuthology.misc.health.smithi088.stderr:2024-06-03T08:36:11.775+0000 7f84597fa640 1 -- 172.21.15.88:0/2877681936 <== mon.0 v1:172.21.15.88:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 764+0+0 (unknown 3932081624 0 0) 0x7f8440004eb0 con 0x7f845c06fce0 2024-06-03T08:36:11.778 INFO:teuthology.misc.health.smithi088.stderr:2024-06-03T08:36:11.775+0000 7f84597fa640 1 -- 172.21.15.88:0/2877681936 --> v1:172.21.15.88:6789/0 -- auth(proto 2 165 bytes epoch 0) v1 -- 0x7f845c06dc80 con 0x7f845c06fce0 2024-06-03T08:36:11.778 INFO:teuthology.misc.health.smithi088.stderr:2024-06-03T08:36:11.775+0000 7f84597fa640 1 -- 172.21.15.88:0/2877681936 <== mon.0 v1:172.21.15.88:6789/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 2234374065 0 0) 0x7f8450001990 con 0x7f845c06fce0 2024-06-03T08:36:11.778 INFO:teuthology.misc.health.smithi088.stderr:2024-06-03T08:36:11.775+0000 7f84597fa640 10 monclient(hunting): handle_monmap mon_map magic: 0 v1 2024-06-03T08:36:11.779 INFO:teuthology.misc.health.smithi088.stderr:2024-06-03T08:36:11.775+0000 7f84597fa640 10 monclient(hunting): got monmap 1 from mon.a (according to old e1) 2024-06-03T08:36:11.779 INFO:teuthology.misc.health.smithi088.stderr:2024-06-03T08:36:11.775+0000 7f84597fa640 10 monclient(hunting): dump: 2024-06-03T08:36:11.779 INFO:teuthology.misc.health.smithi088.stderr:epoch 1 2024-06-03T08:36:11.779 INFO:teuthology.misc.health.smithi088.stderr:fsid ec3c3fdd-3ac6-4d7d-9412-683130ed8b04 2024-06-03T08:36:11.779 INFO:teuthology.misc.health.smithi088.stderr:last_changed 2024-06-03T08:34:31.885125+0000 2024-06-03T08:36:11.779 INFO:teuthology.misc.health.smithi088.stderr:created 2024-06-03T08:34:31.885125+0000 2024-06-03T08:36:11.779 INFO:teuthology.misc.health.smithi088.stderr:min_mon_release 18 (reef) 2024-06-03T08:36:11.779 INFO:teuthology.misc.health.smithi088.stderr:election_strategy: 3 2024-06-03T08:36:11.779 INFO:teuthology.misc.health.smithi088.stderr:0: v1:172.21.15.88:6789/0 mon.a 2024-06-03T08:36:11.779 INFO:teuthology.misc.health.smithi088.stderr:1: v1:172.21.15.192:6789/0 mon.b 2024-06-03T08:36:11.779 INFO:teuthology.misc.health.smithi088.stderr:2: v1:172.21.15.88:6790/0 mon.c 2024-06-03T08:36:11.779 INFO:teuthology.misc.health.smithi088.stderr: 2024-06-03T08:36:11.780 INFO:teuthology.misc.health.smithi088.stderr:2024-06-03T08:36:11.776+0000 7f84597fa640 1 -- 172.21.15.88:0/2877681936 <== mon.1 v1:172.21.15.192:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 764+0+0 (unknown 714128234 0 0) 0x7f845c13e4a0 con 0x7f845c0ad920 2024-06-03T08:36:11.780 INFO:teuthology.misc.health.smithi088.stderr:2024-06-03T08:36:11.776+0000 7f84597fa640 1 -- 172.21.15.88:0/2877681936 --> v1:172.21.15.192:6789/0 -- auth(proto 2 165 bytes epoch 0) v1 -- 0x7f8440004eb0 con 0x7f845c0ad920 2024-06-03T08:36:11.780 INFO:teuthology.misc.health.smithi088.stderr:2024-06-03T08:36:11.776+0000 7f84597fa640 1 -- 172.21.15.88:0/2877681936 <== mon.1 v1:172.21.15.192:6789/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 2234374065 0 0) 0x7f84480014f0 con 0x7f845c0ad920 2024-06-03T08:36:11.780 INFO:teuthology.misc.health.smithi088.stderr:2024-06-03T08:36:11.776+0000 7f84597fa640 10 monclient(hunting): handle_monmap mon_map magic: 0 v1 2024-06-03T08:36:11.780 INFO:teuthology.misc.health.smithi088.stderr:2024-06-03T08:36:11.776+0000 7f84597fa640 10 monclient(hunting): got monmap 1 from mon.b (according to old e1) 2024-06-03T08:36:11.780 INFO:teuthology.misc.health.smithi088.stderr:2024-06-03T08:36:11.776+0000 7f84597fa640 10 monclient(hunting): dump: 2024-06-03T08:36:11.780 INFO:teuthology.misc.health.smithi088.stderr:epoch 1 2024-06-03T08:36:11.780 INFO:teuthology.misc.health.smithi088.stderr:fsid ec3c3fdd-3ac6-4d7d-9412-683130ed8b04 2024-06-03T08:36:11.780 INFO:teuthology.misc.health.smithi088.stderr:last_changed 2024-06-03T08:34:31.885125+0000 2024-06-03T08:36:11.780 INFO:teuthology.misc.health.smithi088.stderr:created 2024-06-03T08:34:31.885125+0000 2024-06-03T08:36:11.780 INFO:teuthology.misc.health.smithi088.stderr:min_mon_release 18 (reef) 2024-06-03T08:36:11.781 INFO:teuthology.misc.health.smithi088.stderr:election_strategy: 3 2024-06-03T08:36:11.781 INFO:teuthology.misc.health.smithi088.stderr:0: v1:172.21.15.88:6789/0 mon.a 2024-06-03T08:36:11.781 INFO:teuthology.misc.health.smithi088.stderr:1: v1:172.21.15.192:6789/0 mon.b 2024-06-03T08:36:11.781 INFO:teuthology.misc.health.smithi088.stderr:2: v1:172.21.15.88:6790/0 mon.c 2024-06-03T08:36:11.781 INFO:teuthology.misc.health.smithi088.stderr: 2024-06-03T08:36:11.782 INFO:teuthology.misc.health.smithi088.stderr:2024-06-03T08:36:11.779+0000 7f84597fa640 1 -- 172.21.15.88:0/2877681936 <== mon.2 v1:172.21.15.88:6790/0 4 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 580+0+0 (unknown 3595925136 0 0) 0x7f845c12aa10 con 0x7f845c13c8e0 2024-06-03T08:36:11.782 INFO:teuthology.misc.health.smithi088.stderr:2024-06-03T08:36:11.779+0000 7f84597fa640 1 -- 172.21.15.88:0/2877681936 >> v1:172.21.15.192:6789/0 conn(0x7f845c0ad920 legacy=0x7f845c129bf0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-06-03T08:36:11.782 INFO:teuthology.misc.health.smithi088.stderr:2024-06-03T08:36:11.779+0000 7f84597fa640 1 -- 172.21.15.88:0/2877681936 >> v1:172.21.15.88:6789/0 conn(0x7f845c06fce0 legacy=0x7f845c1d9930 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-06-03T08:36:11.782 INFO:teuthology.misc.health.smithi088.stderr:2024-06-03T08:36:11.779+0000 7f84597fa640 10 monclient: _finish_hunting 0 2024-06-03T08:36:11.782 INFO:teuthology.misc.health.smithi088.stderr:2024-06-03T08:36:11.779+0000 7f84597fa640 1 monclient: found mon.c 2024-06-03T08:36:11.782 INFO:teuthology.misc.health.smithi088.stderr:2024-06-03T08:36:11.779+0000 7f84597fa640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-06-03T08:36:11.782 INFO:teuthology.misc.health.smithi088.stderr:2024-06-03T08:36:11.779+0000 7f84597fa640 10 monclient: _send_mon_message to mon.c at v1:172.21.15.88:6790/0 2024-06-03T08:36:11.782 INFO:teuthology.misc.health.smithi088.stderr:2024-06-03T08:36:11.779+0000 7f84597fa640 1 -- 172.21.15.88:0/2877681936 --> v1:172.21.15.88:6790/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f845c13d430 con 0x7f845c13c8e0 2024-06-03T08:36:11.783 INFO:teuthology.misc.health.smithi088.stderr:2024-06-03T08:36:11.779+0000 7f84597fa640 10 monclient: _finish_auth 0 2024-06-03T08:36:11.783 INFO:teuthology.misc.health.smithi088.stderr:2024-06-03T08:36:11.779+0000 7f84597fa640 10 monclient: _check_auth_tickets 2024-06-03T08:36:11.783 INFO:teuthology.misc.health.smithi088.stderr:2024-06-03T08:36:11.779+0000 7f84597fa640 20 monclient: _check_auth_rotating not needed by client.admin 2024-06-03T08:36:11.783 INFO:teuthology.misc.health.smithi088.stderr:2024-06-03T08:36:11.779+0000 7f84631f5640 5 monclient: authenticate success, global_id 4259 2024-06-03T08:36:11.783 INFO:teuthology.misc.health.smithi088.stderr:2024-06-03T08:36:11.779+0000 7f84631f5640 10 monclient: _renew_subs 2024-06-03T08:36:11.783 INFO:teuthology.misc.health.smithi088.stderr:2024-06-03T08:36:11.779+0000 7f84631f5640 10 monclient: _send_mon_message to mon.c at v1:172.21.15.88:6790/0 2024-06-03T08:36:11.783 INFO:teuthology.misc.health.smithi088.stderr:2024-06-03T08:36:11.779+0000 7f84631f5640 1 -- 172.21.15.88:0/2877681936 --> v1:172.21.15.88:6790/0 -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f845c1395a0 con 0x7f845c13c8e0 2024-06-03T08:36:11.783 INFO:teuthology.misc.health.smithi088.stderr:2024-06-03T08:36:11.779+0000 7f84631f5640 10 client.4259.objecter _maybe_request_map subscribing (onetime) to next osd map 2024-06-03T08:36:11.783 INFO:teuthology.misc.health.smithi088.stderr:2024-06-03T08:36:11.779+0000 7f84631f5640 10 monclient: _renew_subs 2024-06-03T08:36:11.783 INFO:teuthology.misc.health.smithi088.stderr:2024-06-03T08:36:11.779+0000 7f84631f5640 10 monclient: _send_mon_message to mon.c at v1:172.21.15.88:6790/0 2024-06-03T08:36:11.783 INFO:teuthology.misc.health.smithi088.stderr:2024-06-03T08:36:11.779+0000 7f84631f5640 1 -- 172.21.15.88:0/2877681936 --> v1:172.21.15.88:6790/0 -- mon_subscribe({osdmap=0}) v3 -- 0x7f845c13ddd0 con 0x7f845c13c8e0 2024-06-03T08:36:11.783 INFO:teuthology.misc.health.smithi088.stderr:2024-06-03T08:36:11.779+0000 7f84631f5640 1 librados: init done 2024-06-03T08:36:11.784 INFO:teuthology.misc.health.smithi088.stderr:2024-06-03T08:36:11.780+0000 7f84631f5640 10 start_mon_command cmd=[{"prefix": "get_command_descriptions"}] 2024-06-03T08:36:11.784 INFO:teuthology.misc.health.smithi088.stderr:2024-06-03T08:36:11.780+0000 7f84631f5640 10 monclient: _send_command 1 [{"prefix": "get_command_descriptions"}] 2024-06-03T08:36:11.784 INFO:teuthology.misc.health.smithi088.stderr:2024-06-03T08:36:11.780+0000 7f84631f5640 10 monclient: _send_mon_message to mon.c at v1:172.21.15.88:6790/0 2024-06-03T08:36:11.784 INFO:teuthology.misc.health.smithi088.stderr:2024-06-03T08:36:11.780+0000 7f84631f5640 1 -- 172.21.15.88:0/2877681936 --> v1:172.21.15.88:6790/0 -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f845c13e630 con 0x7f845c13c8e0 2024-06-03T08:36:11.851 INFO:tasks.ceph.osd.0.smithi088.stderr:2024-06-03T08:36:11.832+0000 10267640 -1 osd.0 12 set_numa_affinity unable to identify public interface '' numa node: (2) No such file or directory 2024-06-03T08:36:11.885 INFO:tasks.ceph.osd.1.smithi088.stderr:2024-06-03T08:36:11.872+0000 10267640 -1 osd.1 12 set_numa_affinity unable to identify public interface '' numa node: (2) No such file or directory 2024-06-03T08:36:11.932 INFO:tasks.ceph.osd.3.smithi088.stderr:2024-06-03T08:36:11.919+0000 10267640 -1 osd.3 12 set_numa_affinity unable to identify public interface '' numa node: (2) No such file or directory 2024-06-03T08:36:12.095 INFO:teuthology.misc.health.smithi088.stderr:2024-06-03T08:36:12.095+0000 7f84597fa640 1 -- 172.21.15.88:0/2877681936 <== mon.2 v1:172.21.15.88:6790/0 5 ==== config(0 keys) v1 ==== 4+0+0 (unknown 0 0 0) 0x7f8454002020 con 0x7f845c13c8e0 2024-06-03T08:36:12.095 INFO:teuthology.misc.health.smithi088.stderr:2024-06-03T08:36:12.095+0000 7f84597fa640 10 monclient: handle_config config(0 keys) v1 2024-06-03T08:36:12.096 INFO:teuthology.misc.health.smithi088.stderr:2024-06-03T08:36:12.095+0000 7f84597fa640 1 -- 172.21.15.88:0/2877681936 <== mon.2 v1:172.21.15.88:6790/0 6 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 2234374065 0 0) 0x7f8454003b30 con 0x7f845c13c8e0 2024-06-03T08:36:12.096 INFO:teuthology.misc.health.smithi088.stderr:2024-06-03T08:36:12.095+0000 7f84597fa640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-06-03T08:36:12.096 INFO:teuthology.misc.health.smithi088.stderr:2024-06-03T08:36:12.095+0000 7f84597fa640 10 monclient: got monmap 1 from mon.c (according to old e1) 2024-06-03T08:36:12.096 INFO:teuthology.misc.health.smithi088.stderr:2024-06-03T08:36:12.095+0000 7f84597fa640 10 monclient: dump: 2024-06-03T08:36:12.096 INFO:teuthology.misc.health.smithi088.stderr:epoch 1 2024-06-03T08:36:12.096 INFO:teuthology.misc.health.smithi088.stderr:fsid ec3c3fdd-3ac6-4d7d-9412-683130ed8b04 2024-06-03T08:36:12.096 INFO:teuthology.misc.health.smithi088.stderr:last_changed 2024-06-03T08:34:31.885125+0000 2024-06-03T08:36:12.096 INFO:teuthology.misc.health.smithi088.stderr:created 2024-06-03T08:34:31.885125+0000 2024-06-03T08:36:12.096 INFO:teuthology.misc.health.smithi088.stderr:min_mon_release 18 (reef) 2024-06-03T08:36:12.096 INFO:teuthology.misc.health.smithi088.stderr:election_strategy: 3 2024-06-03T08:36:12.096 INFO:teuthology.misc.health.smithi088.stderr:0: v1:172.21.15.88:6789/0 mon.a 2024-06-03T08:36:12.096 INFO:teuthology.misc.health.smithi088.stderr:1: v1:172.21.15.192:6789/0 mon.b 2024-06-03T08:36:12.097 INFO:teuthology.misc.health.smithi088.stderr:2: v1:172.21.15.88:6790/0 mon.c 2024-06-03T08:36:12.097 INFO:teuthology.misc.health.smithi088.stderr: 2024-06-03T08:36:12.097 INFO:teuthology.misc.health.smithi088.stderr:2024-06-03T08:36:12.095+0000 7f845affd640 4 set_mon_vals no callback set 2024-06-03T08:36:12.097 INFO:teuthology.misc.health.smithi088.stderr:2024-06-03T08:36:12.095+0000 7f84597fa640 1 -- 172.21.15.88:0/2877681936 <== mon.2 v1:172.21.15.88:6790/0 7 ==== mgrmap(e 4) v1 ==== 89054+0+0 (unknown 914547237 0 0) 0x7f8454019880 con 0x7f845c13c8e0 2024-06-03T08:36:12.097 INFO:teuthology.misc.health.smithi088.stderr:2024-06-03T08:36:12.096+0000 7f8460f91640 10 monclient: get_auth_request con 0x7f844006d400 auth_method 0 2024-06-03T08:36:12.097 INFO:teuthology.misc.health.smithi088.stderr:2024-06-03T08:36:12.096+0000 7f84597fa640 10 client.4259.objecter ms_handle_connect 0x7f844006d400 2024-06-03T08:36:12.101 INFO:teuthology.misc.health.smithi088.stderr:2024-06-03T08:36:12.100+0000 7f84597fa640 1 -- 172.21.15.88:0/2877681936 <== mon.2 v1:172.21.15.88:6790/0 8 ==== osd_map(12..12 src has 1..12) v4 ==== 2199+0+0 (unknown 715337950 0 0) 0x7f8454053070 con 0x7f845c13c8e0 2024-06-03T08:36:12.101 INFO:teuthology.misc.health.smithi088.stderr:2024-06-03T08:36:12.100+0000 7f84597fa640 10 client.4259.objecter ms_dispatch 0x7f845c000b90 osd_map(12..12 src has 1..12) v4 2024-06-03T08:36:12.101 INFO:teuthology.misc.health.smithi088.stderr:2024-06-03T08:36:12.100+0000 7f84597fa640 3 client.4259.objecter handle_osd_map got epochs [12,12] > 0 2024-06-03T08:36:12.101 INFO:teuthology.misc.health.smithi088.stderr:2024-06-03T08:36:12.100+0000 7f84597fa640 3 client.4259.objecter handle_osd_map decoding full epoch 12 2024-06-03T08:36:12.101 INFO:teuthology.misc.health.smithi088.stderr:2024-06-03T08:36:12.100+0000 7f84597fa640 20 client.4259.objecter dump_active .. 0 homeless 2024-06-03T08:36:12.101 INFO:teuthology.misc.health.smithi088.stderr:2024-06-03T08:36:12.100+0000 7f84597fa640 1 -- 172.21.15.88:0/2877681936 <== mon.2 v1:172.21.15.88:6790/0 9 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+182498 (unknown 1092875540 0 3016760700) 0x7f845404ec20 con 0x7f845c13c8e0 2024-06-03T08:36:12.101 INFO:teuthology.misc.health.smithi088.stderr:2024-06-03T08:36:12.100+0000 7f84597fa640 10 monclient: handle_mon_command_ack 1 [{"prefix": "get_command_descriptions"}] 2024-06-03T08:36:12.101 INFO:teuthology.misc.health.smithi088.stderr:2024-06-03T08:36:12.100+0000 7f84597fa640 10 monclient: _finish_command 1 = system:0 2024-06-03T08:36:12.286 INFO:teuthology.misc.health.smithi088.stderr:2024-06-03T08:36:12.286+0000 7f84631f5640 10 start_mon_command cmd=[{"prefix": "osd dump", "format": "json"}] 2024-06-03T08:36:12.286 INFO:teuthology.misc.health.smithi088.stderr:2024-06-03T08:36:12.286+0000 7f84631f5640 10 monclient: _send_command 2 [{"prefix": "osd dump", "format": "json"}] 2024-06-03T08:36:12.287 INFO:teuthology.misc.health.smithi088.stderr:2024-06-03T08:36:12.286+0000 7f84631f5640 10 monclient: _send_mon_message to mon.c at v1:172.21.15.88:6790/0 2024-06-03T08:36:12.287 INFO:teuthology.misc.health.smithi088.stderr:2024-06-03T08:36:12.286+0000 7f84631f5640 1 -- 172.21.15.88:0/2877681936 --> v1:172.21.15.88:6790/0 -- mon_command({"prefix": "osd dump", "format": "json"} v 0) v1 -- 0x7f845c04e730 con 0x7f845c13c8e0 2024-06-03T08:36:12.425 INFO:teuthology.misc.health.smithi088.stderr:2024-06-03T08:36:12.425+0000 7f84597fa640 1 -- 172.21.15.88:0/2877681936 <== mon.2 v1:172.21.15.88:6790/0 10 ==== mon_command_ack([{"prefix": "osd dump", "format": "json"}]=0 v13) v1 ==== 74+0+7057 (unknown 62554612 0 1449335198) 0x7f8454050b50 con 0x7f845c13c8e0 2024-06-03T08:36:12.425 INFO:teuthology.misc.health.smithi088.stderr:2024-06-03T08:36:12.425+0000 7f84597fa640 10 monclient: handle_mon_command_ack 2 [{"prefix": "osd dump", "format": "json"}] 2024-06-03T08:36:12.426 INFO:teuthology.misc.health.smithi088.stderr:2024-06-03T08:36:12.425+0000 7f84597fa640 10 monclient: _finish_command 2 = system:0 2024-06-03T08:36:12.426 INFO:teuthology.misc.health.smithi088.stdout: 2024-06-03T08:36:12.426 INFO:teuthology.misc.health.smithi088.stdout:{"epoch":13,"fsid":"ec3c3fdd-3ac6-4d7d-9412-683130ed8b04","created":"2024-06-03T08:35:26.799638+0000","modified":"2024-06-03T08:36:11.997309+0000","last_up_change":"0.000000","last_in_change":"2024-06-03T08:35:32.183068+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":5,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":0,"max_osd":8,"require_min_compat_client":"luminous","min_compat_client":"jewel","require_osd_release":"reef","allow_crimson":false,"pools":[],"osds":[{"osd":0,"uuid":"b25ab3bb-3587-45d0-991a-6d41bb529b37","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":1,"uuid":"00d160e7-9773-41d8-8d17-21e70620a0fd","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":2,"uuid":"addfb2a7-2bde-4cc3-98cf-17fd2bdfa3a9","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":3,"uuid":"6c5770b0-c93c-4101-8c53-bd761b8abcde","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":4,"uuid":"f001b3cb-2081-428b-bac3-b3df3c19b978","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":5,"uuid":"3cc263e9-e9bf-45e0-a3b7-83a76469eb70","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":6,"uuid":"82c83316-809f-4901-a5ed-34b6a075b393","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":7,"uuid":"aafe0716-1102-4eae-aac6-0f81bb154e7d","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":4,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":5,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":6,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":7,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[],"pg_upmap_primaries":[],"pg_temp":[],"primary_temp":[],"blocklist":{},"range_blocklist":{},"erasure_code_profiles":{"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","technique":"reed_sol_van"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2024-06-03T08:36:12.428 INFO:teuthology.misc.health.smithi088.stderr:2024-06-03T08:36:12.427+0000 7f84631f5640 10 librados: watch_flush enter 2024-06-03T08:36:12.429 INFO:teuthology.misc.health.smithi088.stderr:2024-06-03T08:36:12.427+0000 7f84631f5640 10 librados: watch_flush exit 2024-06-03T08:36:12.429 INFO:teuthology.misc.health.smithi088.stderr:2024-06-03T08:36:12.427+0000 7f84631f5640 20 client.4259.objecter shutdown clearing up homeless session... 2024-06-03T08:36:12.429 INFO:teuthology.misc.health.smithi088.stderr:2024-06-03T08:36:12.427+0000 7f84631f5640 10 client.4259.objecter successfully canceled tick 2024-06-03T08:36:12.429 INFO:teuthology.misc.health.smithi088.stderr:2024-06-03T08:36:12.428+0000 7f84631f5640 1 -- 172.21.15.88:0/2877681936 >> v1:172.21.15.192:6800/34229 conn(0x7f844006d400 legacy=0x7f844006f8c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-06-03T08:36:12.429 INFO:teuthology.misc.health.smithi088.stderr:2024-06-03T08:36:12.428+0000 7f84631f5640 10 monclient: shutdown 2024-06-03T08:36:12.429 INFO:teuthology.misc.health.smithi088.stderr:2024-06-03T08:36:12.428+0000 7f84631f5640 20 monclient: shutdown discarding 0 pending message(s) 2024-06-03T08:36:12.429 INFO:teuthology.misc.health.smithi088.stderr:2024-06-03T08:36:12.428+0000 7f84631f5640 1 -- 172.21.15.88:0/2877681936 >> v1:172.21.15.88:6790/0 conn(0x7f845c13c8e0 legacy=0x7f845c12a300 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-06-03T08:36:12.429 INFO:teuthology.misc.health.smithi088.stderr:2024-06-03T08:36:12.428+0000 7f84631f5640 1 -- 172.21.15.88:0/2877681936 shutdown_connections 2024-06-03T08:36:12.429 INFO:teuthology.misc.health.smithi088.stderr:2024-06-03T08:36:12.428+0000 7f84631f5640 1 -- 172.21.15.88:0/2877681936 >> 172.21.15.88:0/2877681936 conn(0x7f845c0b2f20 msgr2=0x7f845c0737f0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-06-03T08:36:12.429 INFO:teuthology.misc.health.smithi088.stderr:2024-06-03T08:36:12.428+0000 7f84631f5640 1 -- 172.21.15.88:0/2877681936 shutdown_connections 2024-06-03T08:36:12.429 INFO:teuthology.misc.health.smithi088.stderr:2024-06-03T08:36:12.428+0000 7f84631f5640 1 -- 172.21.15.88:0/2877681936 wait complete. 2024-06-03T08:36:12.429 INFO:teuthology.misc.health.smithi088.stderr:2024-06-03T08:36:12.428+0000 7f84631f5640 1 librados: shutdown 2024-06-03T08:36:12.438 DEBUG:teuthology.misc:0 of 8 OSDs are up 2024-06-03T08:36:18.436 INFO:tasks.ceph.osd.4.smithi192.stderr:2024-06-03T08:36:18.434+0000 5dcd500 -1 bdev(0x16959420 /var/lib/ceph/osd/ceph-4/block) aio_submit retries 2 2024-06-03T08:36:18.625 INFO:tasks.ceph.osd.5.smithi192.stderr:2024-06-03T08:36:18.624+0000 5dcd500 -1 bdev(0x16952e70 /var/lib/ceph/osd/ceph-5/block) aio_submit retries 1 2024-06-03T08:36:18.634 INFO:tasks.ceph.osd.4.smithi192.stderr:2024-06-03T08:36:18.524+0000 5dcd500 -1 bdev(0x16959420 /var/lib/ceph/osd/ceph-4/block) aio_submit retries 1 2024-06-03T08:36:18.643 INFO:tasks.ceph.osd.4.smithi192.stderr:2024-06-03T08:36:18.578+0000 5dcd500 -1 bdev(0x16959420 /var/lib/ceph/osd/ceph-4/block) aio_submit retries 1 2024-06-03T08:36:18.743 INFO:tasks.ceph.osd.5.smithi192.stderr:2024-06-03T08:36:18.690+0000 5dcd500 -1 bdev(0x16952e70 /var/lib/ceph/osd/ceph-5/block) aio_submit retries 2 2024-06-03T08:36:18.883 INFO:tasks.ceph.osd.7.smithi192.stderr:2024-06-03T08:36:18.881+0000 5dcd500 -1 bdev(0x1694d260 /var/lib/ceph/osd/ceph-7/block) aio_submit retries 2 2024-06-03T08:36:18.992 INFO:tasks.ceph.osd.5.smithi192.stderr:2024-06-03T08:36:18.748+0000 5dcd500 -1 bdev(0x16952e70 /var/lib/ceph/osd/ceph-5/block) aio_submit retries 1 2024-06-03T08:36:19.037 INFO:tasks.ceph.osd.5.smithi192.stderr:2024-06-03T08:36:18.981+0000 5dcd500 -1 bdev(0x16952e70 /var/lib/ceph/osd/ceph-5/block) aio_submit retries 1 2024-06-03T08:36:19.037 INFO:tasks.ceph.osd.5.smithi192.stderr:2024-06-03T08:36:18.993+0000 5dcd500 -1 bdev(0x16952e70 /var/lib/ceph/osd/ceph-5/block) aio_submit retries 1 2024-06-03T08:36:19.060 INFO:tasks.ceph.osd.5.smithi192.stderr:2024-06-03T08:36:19.038+0000 5dcd500 -1 bdev(0x16952e70 /var/lib/ceph/osd/ceph-5/block) aio_submit retries 2 2024-06-03T08:36:19.112 INFO:tasks.ceph.osd.5.smithi192.stderr:2024-06-03T08:36:19.061+0000 5dcd500 -1 bdev(0x16952e70 /var/lib/ceph/osd/ceph-5/block) aio_submit retries 1 2024-06-03T08:36:19.243 DEBUG:teuthology.orchestra.run.smithi088:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph --cluster ceph osd dump --format=json 2024-06-03T08:36:19.275 INFO:tasks.ceph.osd.2.smithi088.stderr:2024-06-03T08:36:19.240+0000 5dcd500 -1 bdev(0x18a920e0 /var/lib/ceph/osd/ceph-2/block) aio_submit retries 1 2024-06-03T08:36:19.350 INFO:teuthology.misc.health.smithi088.stderr:2024-06-03T08:36:19.349+0000 7ff2489d7640 10 monclient: get_monmap_and_config 2024-06-03T08:36:19.350 INFO:teuthology.misc.health.smithi088.stderr:2024-06-03T08:36:19.349+0000 7ff2489d7640 10 monclient: build_initial_monmap 2024-06-03T08:36:19.351 INFO:teuthology.misc.health.smithi088.stderr:2024-06-03T08:36:19.350+0000 7ff2489d7640 10 monclient: monmap: 2024-06-03T08:36:19.351 INFO:teuthology.misc.health.smithi088.stderr:epoch 0 2024-06-03T08:36:19.351 INFO:teuthology.misc.health.smithi088.stderr:fsid ec3c3fdd-3ac6-4d7d-9412-683130ed8b04 2024-06-03T08:36:19.351 INFO:teuthology.misc.health.smithi088.stderr:last_changed 2024-06-03T08:36:19.351004+0000 2024-06-03T08:36:19.351 INFO:teuthology.misc.health.smithi088.stderr:created 2024-06-03T08:36:19.351004+0000 2024-06-03T08:36:19.351 INFO:teuthology.misc.health.smithi088.stderr:min_mon_release 0 (unknown) 2024-06-03T08:36:19.351 INFO:teuthology.misc.health.smithi088.stderr:election_strategy: 3 2024-06-03T08:36:19.351 INFO:teuthology.misc.health.smithi088.stderr:0: v1:172.21.15.88:6789/0 mon.noname-a 2024-06-03T08:36:19.351 INFO:teuthology.misc.health.smithi088.stderr:1: v1:172.21.15.192:6789/0 mon.noname-c 2024-06-03T08:36:19.351 INFO:teuthology.misc.health.smithi088.stderr:2: v1:172.21.15.88:6790/0 mon.noname-b 2024-06-03T08:36:19.351 INFO:teuthology.misc.health.smithi088.stderr: 2024-06-03T08:36:19.353 INFO:teuthology.misc.health.smithi088.stderr:2024-06-03T08:36:19.350+0000 7ff2489d7640 1 Processor -- start 2024-06-03T08:36:19.353 INFO:teuthology.misc.health.smithi088.stderr:2024-06-03T08:36:19.351+0000 7ff2489d7640 1 -- start start 2024-06-03T08:36:19.353 INFO:teuthology.misc.health.smithi088.stderr:2024-06-03T08:36:19.351+0000 7ff2489d7640 10 monclient: init 2024-06-03T08:36:19.353 INFO:teuthology.misc.health.smithi088.stderr:2024-06-03T08:36:19.352+0000 7ff2489d7640 10 monclient: _reopen_session rank -1 2024-06-03T08:36:19.353 INFO:teuthology.misc.health.smithi088.stderr:2024-06-03T08:36:19.352+0000 7ff2489d7640 10 monclient: _add_conns ranks=[0,1,2] 2024-06-03T08:36:19.353 INFO:teuthology.misc.health.smithi088.stderr:2024-06-03T08:36:19.352+0000 7ff2489d7640 10 monclient(hunting): picked mon.noname-a con 0x7ff2440b1070 addr v1:172.21.15.88:6789/0 2024-06-03T08:36:19.353 INFO:teuthology.misc.health.smithi088.stderr:2024-06-03T08:36:19.352+0000 7ff2489d7640 10 monclient(hunting): picked mon.noname-c con 0x7ff244144b40 addr v1:172.21.15.192:6789/0 2024-06-03T08:36:19.353 INFO:teuthology.misc.health.smithi088.stderr:2024-06-03T08:36:19.352+0000 7ff2489d7640 10 monclient(hunting): picked mon.noname-b con 0x7ff2440af690 addr v1:172.21.15.88:6790/0 2024-06-03T08:36:19.353 INFO:teuthology.misc.health.smithi088.stderr:2024-06-03T08:36:19.352+0000 7ff2489d7640 1 -- --> v1:172.21.15.88:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7ff2440b1820 con 0x7ff2440b1070 2024-06-03T08:36:19.353 INFO:teuthology.misc.health.smithi088.stderr:2024-06-03T08:36:19.352+0000 7ff2489d7640 1 -- --> v1:172.21.15.192:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7ff2440b7d00 con 0x7ff244144b40 2024-06-03T08:36:19.353 INFO:teuthology.misc.health.smithi088.stderr:2024-06-03T08:36:19.352+0000 7ff2489d7640 1 -- --> v1:172.21.15.88:6790/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7ff2440ae2d0 con 0x7ff2440af690 2024-06-03T08:36:19.354 INFO:teuthology.misc.health.smithi088.stderr:2024-06-03T08:36:19.352+0000 7ff2489d7640 10 monclient(hunting): _renew_subs 2024-06-03T08:36:19.354 INFO:teuthology.misc.health.smithi088.stderr:2024-06-03T08:36:19.352+0000 7ff2489d7640 10 monclient(hunting): authenticate will time out at 1235.162449s 2024-06-03T08:36:19.365 INFO:tasks.ceph.osd.4.smithi192.stderr:2024-06-03T08:36:18.643+0000 5dcd500 -1 bdev(0x16959420 /var/lib/ceph/osd/ceph-4/block) aio_submit retries 1 2024-06-03T08:36:19.387 INFO:teuthology.misc.health.smithi088.stderr:2024-06-03T08:36:19.387+0000 7ff242d9d640 1 --1- >> v1:172.21.15.88:6790/0 conn(0x7ff2440af690 0x7ff24414cee0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.88:6790/0 says I am v1:172.21.15.88:50396/0 (socket says 172.21.15.88:50396) 2024-06-03T08:36:19.387 INFO:teuthology.misc.health.smithi088.stderr:2024-06-03T08:36:19.387+0000 7ff242d9d640 1 -- 172.21.15.88:0/77836364 learned_addr learned my addr 172.21.15.88:0/77836364 (peer_addr_for_me v1:172.21.15.88:0/0) 2024-06-03T08:36:19.396 INFO:teuthology.misc.health.smithi088.stderr:2024-06-03T08:36:19.396+0000 7ff24159a640 1 -- 172.21.15.88:0/77836364 <== mon.2 v1:172.21.15.88:6790/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 1956108912 0 0) 0x7ff2440ae2d0 con 0x7ff2440af690 2024-06-03T08:36:19.396 INFO:teuthology.misc.health.smithi088.stderr:2024-06-03T08:36:19.396+0000 7ff24159a640 10 monclient(hunting): _negotiate 2024-06-03T08:36:19.396 INFO:teuthology.misc.health.smithi088.stderr:2024-06-03T08:36:19.396+0000 7ff24159a640 10 monclient(hunting): _init_auth method 2 2024-06-03T08:36:19.396 INFO:teuthology.misc.health.smithi088.stderr:2024-06-03T08:36:19.396+0000 7ff24159a640 10 monclient(hunting): _init_auth creating new auth 2024-06-03T08:36:19.396 INFO:teuthology.misc.health.smithi088.stderr:2024-06-03T08:36:19.396+0000 7ff24159a640 10 monclient(hunting): my global_id is 4262 2024-06-03T08:36:19.396 INFO:teuthology.misc.health.smithi088.stderr:2024-06-03T08:36:19.396+0000 7ff24159a640 1 -- 172.21.15.88:0/77836364 --> v1:172.21.15.88:6790/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7ff22c003520 con 0x7ff2440af690 2024-06-03T08:36:19.402 INFO:teuthology.misc.health.smithi088.stderr:2024-06-03T08:36:19.401+0000 7ff24159a640 1 -- 172.21.15.88:0/77836364 <== mon.2 v1:172.21.15.88:6790/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 214+0+0 (unknown 2013396453 0 0) 0x7ff22c003520 con 0x7ff2440af690 2024-06-03T08:36:19.402 INFO:teuthology.misc.health.smithi088.stderr:2024-06-03T08:36:19.401+0000 7ff24159a640 1 -- 172.21.15.88:0/77836364 >> v1:172.21.15.192:6789/0 conn(0x7ff244144b40 legacy=0x7ff2440aef80 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-06-03T08:36:19.402 INFO:teuthology.misc.health.smithi088.stderr:2024-06-03T08:36:19.401+0000 7ff24159a640 1 -- 172.21.15.88:0/77836364 >> v1:172.21.15.88:6789/0 conn(0x7ff2440b1070 legacy=0x7ff2441451c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-06-03T08:36:19.402 INFO:teuthology.misc.health.smithi088.stderr:2024-06-03T08:36:19.401+0000 7ff24159a640 10 monclient: _finish_hunting 0 2024-06-03T08:36:19.402 INFO:teuthology.misc.health.smithi088.stderr:2024-06-03T08:36:19.401+0000 7ff24159a640 1 monclient: found mon.noname-b 2024-06-03T08:36:19.403 INFO:teuthology.misc.health.smithi088.stderr:2024-06-03T08:36:19.401+0000 7ff24159a640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-06-03T08:36:19.403 INFO:teuthology.misc.health.smithi088.stderr:2024-06-03T08:36:19.401+0000 7ff24159a640 10 monclient: _send_mon_message to mon.noname-b at v1:172.21.15.88:6790/0 2024-06-03T08:36:19.403 INFO:teuthology.misc.health.smithi088.stderr:2024-06-03T08:36:19.401+0000 7ff24159a640 1 -- 172.21.15.88:0/77836364 --> v1:172.21.15.88:6790/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7ff2440b1450 con 0x7ff2440af690 2024-06-03T08:36:19.403 INFO:teuthology.misc.health.smithi088.stderr:2024-06-03T08:36:19.401+0000 7ff24159a640 10 monclient: _finish_auth 0 2024-06-03T08:36:19.403 INFO:teuthology.misc.health.smithi088.stderr:2024-06-03T08:36:19.401+0000 7ff24159a640 10 monclient: _check_auth_tickets 2024-06-03T08:36:19.403 INFO:teuthology.misc.health.smithi088.stderr:2024-06-03T08:36:19.401+0000 7ff24159a640 20 monclient: _check_auth_rotating not needed by client.admin 2024-06-03T08:36:19.403 INFO:teuthology.misc.health.smithi088.stderr:2024-06-03T08:36:19.401+0000 7ff24159a640 1 -- 172.21.15.88:0/77836364 <== mon.2 v1:172.21.15.88:6790/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 2234374065 0 0) 0x7ff23c001990 con 0x7ff2440af690 2024-06-03T08:36:19.403 INFO:teuthology.misc.health.smithi088.stderr:2024-06-03T08:36:19.401+0000 7ff24159a640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-06-03T08:36:19.403 INFO:teuthology.misc.health.smithi088.stderr:2024-06-03T08:36:19.401+0000 7ff24159a640 10 monclient: got monmap 1 from mon.noname-b (according to old e1) 2024-06-03T08:36:19.403 INFO:teuthology.misc.health.smithi088.stderr:2024-06-03T08:36:19.401+0000 7ff24159a640 10 monclient: dump: 2024-06-03T08:36:19.403 INFO:teuthology.misc.health.smithi088.stderr:epoch 1 2024-06-03T08:36:19.403 INFO:teuthology.misc.health.smithi088.stderr:fsid ec3c3fdd-3ac6-4d7d-9412-683130ed8b04 2024-06-03T08:36:19.403 INFO:teuthology.misc.health.smithi088.stderr:last_changed 2024-06-03T08:34:31.885125+0000 2024-06-03T08:36:19.404 INFO:teuthology.misc.health.smithi088.stderr:created 2024-06-03T08:34:31.885125+0000 2024-06-03T08:36:19.404 INFO:teuthology.misc.health.smithi088.stderr:min_mon_release 18 (reef) 2024-06-03T08:36:19.404 INFO:teuthology.misc.health.smithi088.stderr:election_strategy: 3 2024-06-03T08:36:19.404 INFO:teuthology.misc.health.smithi088.stderr:0: v1:172.21.15.88:6789/0 mon.a 2024-06-03T08:36:19.404 INFO:teuthology.misc.health.smithi088.stderr:1: v1:172.21.15.192:6789/0 mon.b 2024-06-03T08:36:19.404 INFO:teuthology.misc.health.smithi088.stderr:2: v1:172.21.15.88:6790/0 mon.c 2024-06-03T08:36:19.404 INFO:teuthology.misc.health.smithi088.stderr: 2024-06-03T08:36:19.404 INFO:teuthology.misc.health.smithi088.stderr:2024-06-03T08:36:19.401+0000 7ff2489d7640 5 monclient: authenticate success, global_id 4262 2024-06-03T08:36:19.404 INFO:teuthology.misc.health.smithi088.stderr:2024-06-03T08:36:19.401+0000 7ff2489d7640 20 monclient: get_monmap_and_config waiting for monmap|config 2024-06-03T08:36:19.405 INFO:teuthology.misc.health.smithi088.stderr:2024-06-03T08:36:19.403+0000 7ff24159a640 1 -- 172.21.15.88:0/77836364 <== mon.2 v1:172.21.15.88:6790/0 4 ==== config(0 keys) v1 ==== 4+0+0 (unknown 0 0 0) 0x7ff23c001bb0 con 0x7ff2440af690 2024-06-03T08:36:19.405 INFO:teuthology.misc.health.smithi088.stderr:2024-06-03T08:36:19.403+0000 7ff24159a640 10 monclient: handle_config config(0 keys) v1 2024-06-03T08:36:19.405 INFO:teuthology.misc.health.smithi088.stderr:2024-06-03T08:36:19.403+0000 7ff24159a640 1 -- 172.21.15.88:0/77836364 <== mon.2 v1:172.21.15.88:6790/0 5 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 2234374065 0 0) 0x7ff23c001fc0 con 0x7ff2440af690 2024-06-03T08:36:19.405 INFO:teuthology.misc.health.smithi088.stderr:2024-06-03T08:36:19.403+0000 7ff2489d7640 10 monclient: get_monmap_and_config success 2024-06-03T08:36:19.405 INFO:teuthology.misc.health.smithi088.stderr:2024-06-03T08:36:19.403+0000 7ff2489d7640 4 set_mon_vals no callback set 2024-06-03T08:36:19.405 INFO:teuthology.misc.health.smithi088.stderr:2024-06-03T08:36:19.403+0000 7ff24159a640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-06-03T08:36:19.405 INFO:teuthology.misc.health.smithi088.stderr:2024-06-03T08:36:19.403+0000 7ff24159a640 10 monclient: got monmap 1 from mon.c (according to old e1) 2024-06-03T08:36:19.406 INFO:teuthology.misc.health.smithi088.stderr:2024-06-03T08:36:19.403+0000 7ff24159a640 10 monclient: dump: 2024-06-03T08:36:19.406 INFO:teuthology.misc.health.smithi088.stderr:epoch 1 2024-06-03T08:36:19.406 INFO:teuthology.misc.health.smithi088.stderr:fsid ec3c3fdd-3ac6-4d7d-9412-683130ed8b04 2024-06-03T08:36:19.406 INFO:teuthology.misc.health.smithi088.stderr:last_changed 2024-06-03T08:34:31.885125+0000 2024-06-03T08:36:19.406 INFO:teuthology.misc.health.smithi088.stderr:created 2024-06-03T08:34:31.885125+0000 2024-06-03T08:36:19.406 INFO:teuthology.misc.health.smithi088.stderr:min_mon_release 18 (reef) 2024-06-03T08:36:19.406 INFO:teuthology.misc.health.smithi088.stderr:election_strategy: 3 2024-06-03T08:36:19.406 INFO:teuthology.misc.health.smithi088.stderr:0: v1:172.21.15.88:6789/0 mon.a 2024-06-03T08:36:19.406 INFO:teuthology.misc.health.smithi088.stderr:1: v1:172.21.15.192:6789/0 mon.b 2024-06-03T08:36:19.406 INFO:teuthology.misc.health.smithi088.stderr:2: v1:172.21.15.88:6790/0 mon.c 2024-06-03T08:36:19.406 INFO:teuthology.misc.health.smithi088.stderr: 2024-06-03T08:36:19.406 INFO:teuthology.misc.health.smithi088.stderr:2024-06-03T08:36:19.404+0000 7ff2489d7640 10 monclient: shutdown 2024-06-03T08:36:19.406 INFO:teuthology.misc.health.smithi088.stderr:2024-06-03T08:36:19.404+0000 7ff2489d7640 20 monclient: shutdown discarding 0 pending message(s) 2024-06-03T08:36:19.407 INFO:teuthology.misc.health.smithi088.stderr:2024-06-03T08:36:19.404+0000 7ff2489d7640 1 -- 172.21.15.88:0/77836364 >> v1:172.21.15.88:6790/0 conn(0x7ff2440af690 legacy=0x7ff24414cee0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-06-03T08:36:19.407 INFO:teuthology.misc.health.smithi088.stderr:2024-06-03T08:36:19.404+0000 7ff2489d7640 1 -- 172.21.15.88:0/77836364 shutdown_connections 2024-06-03T08:36:19.407 INFO:teuthology.misc.health.smithi088.stderr:2024-06-03T08:36:19.404+0000 7ff2489d7640 1 -- 172.21.15.88:0/77836364 >> 172.21.15.88:0/77836364 conn(0x7ff2440b71e0 msgr2=0x7ff2440b75f0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-06-03T08:36:19.407 INFO:teuthology.misc.health.smithi088.stderr:2024-06-03T08:36:19.404+0000 7ff2489d7640 1 -- 172.21.15.88:0/77836364 shutdown_connections 2024-06-03T08:36:19.407 INFO:teuthology.misc.health.smithi088.stderr:2024-06-03T08:36:19.404+0000 7ff2489d7640 1 -- 172.21.15.88:0/77836364 wait complete. 2024-06-03T08:36:19.409 INFO:teuthology.misc.health.smithi088.stderr:2024-06-03T08:36:19.405+0000 7ff2489d7640 10 monclient: build_initial_monmap 2024-06-03T08:36:19.409 INFO:teuthology.misc.health.smithi088.stderr:2024-06-03T08:36:19.405+0000 7ff2489d7640 10 monclient: monmap: 2024-06-03T08:36:19.409 INFO:teuthology.misc.health.smithi088.stderr:epoch 0 2024-06-03T08:36:19.410 INFO:teuthology.misc.health.smithi088.stderr:fsid 00000000-0000-0000-0000-000000000000 2024-06-03T08:36:19.410 INFO:teuthology.misc.health.smithi088.stderr:last_changed 0.000000 2024-06-03T08:36:19.410 INFO:teuthology.misc.health.smithi088.stderr:created 0.000000 2024-06-03T08:36:19.410 INFO:teuthology.misc.health.smithi088.stderr:min_mon_release 0 (unknown) 2024-06-03T08:36:19.410 INFO:teuthology.misc.health.smithi088.stderr:election_strategy: 1 2024-06-03T08:36:19.410 INFO:teuthology.misc.health.smithi088.stderr:0: v1:172.21.15.88:6789/0 mon.noname-a 2024-06-03T08:36:19.410 INFO:teuthology.misc.health.smithi088.stderr:1: v1:172.21.15.192:6789/0 mon.noname-b 2024-06-03T08:36:19.410 INFO:teuthology.misc.health.smithi088.stderr:2: v1:172.21.15.88:6790/0 mon.noname-c 2024-06-03T08:36:19.410 INFO:teuthology.misc.health.smithi088.stderr: 2024-06-03T08:36:19.410 INFO:teuthology.misc.health.smithi088.stderr:2024-06-03T08:36:19.406+0000 7ff2489d7640 1 librados: starting msgr at 2024-06-03T08:36:19.410 INFO:teuthology.misc.health.smithi088.stderr:2024-06-03T08:36:19.406+0000 7ff2489d7640 1 librados: starting objecter 2024-06-03T08:36:19.410 INFO:teuthology.misc.health.smithi088.stderr:2024-06-03T08:36:19.406+0000 7ff2489d7640 1 Processor -- start 2024-06-03T08:36:19.410 INFO:teuthology.misc.health.smithi088.stderr:2024-06-03T08:36:19.406+0000 7ff2489d7640 1 -- start start 2024-06-03T08:36:19.411 INFO:teuthology.misc.health.smithi088.stderr:2024-06-03T08:36:19.406+0000 7ff2489d7640 1 librados: setting wanted keys 2024-06-03T08:36:19.411 INFO:teuthology.misc.health.smithi088.stderr:2024-06-03T08:36:19.406+0000 7ff2489d7640 1 librados: calling monclient init 2024-06-03T08:36:19.411 INFO:teuthology.misc.health.smithi088.stderr:2024-06-03T08:36:19.406+0000 7ff2489d7640 10 monclient: init 2024-06-03T08:36:19.411 INFO:teuthology.misc.health.smithi088.stderr:2024-06-03T08:36:19.407+0000 7ff2489d7640 10 monclient: _reopen_session rank -1 2024-06-03T08:36:19.411 INFO:teuthology.misc.health.smithi088.stderr:2024-06-03T08:36:19.407+0000 7ff2489d7640 10 monclient: _add_conns ranks=[2,1,0] 2024-06-03T08:36:19.411 INFO:teuthology.misc.health.smithi088.stderr:2024-06-03T08:36:19.407+0000 7ff2489d7640 10 monclient(hunting): picked mon.noname-c con 0x7ff2440b1070 addr v1:172.21.15.88:6790/0 2024-06-03T08:36:19.411 INFO:teuthology.misc.health.smithi088.stderr:2024-06-03T08:36:19.407+0000 7ff2489d7640 10 monclient(hunting): picked mon.noname-b con 0x7ff244144b40 addr v1:172.21.15.192:6789/0 2024-06-03T08:36:19.411 INFO:teuthology.misc.health.smithi088.stderr:2024-06-03T08:36:19.407+0000 7ff2489d7640 10 monclient(hunting): picked mon.noname-a con 0x7ff2440aa510 addr v1:172.21.15.88:6789/0 2024-06-03T08:36:19.411 INFO:teuthology.misc.health.smithi088.stderr:2024-06-03T08:36:19.407+0000 7ff2489d7640 1 -- --> v1:172.21.15.88:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7ff2440abf40 con 0x7ff2440aa510 2024-06-03T08:36:19.411 INFO:teuthology.misc.health.smithi088.stderr:2024-06-03T08:36:19.407+0000 7ff2489d7640 1 -- --> v1:172.21.15.192:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7ff24413ad90 con 0x7ff244144b40 2024-06-03T08:36:19.411 INFO:teuthology.misc.health.smithi088.stderr:2024-06-03T08:36:19.407+0000 7ff2489d7640 1 -- --> v1:172.21.15.88:6790/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7ff244142110 con 0x7ff2440b1070 2024-06-03T08:36:19.411 INFO:teuthology.misc.health.smithi088.stderr:2024-06-03T08:36:19.407+0000 7ff2489d7640 10 monclient(hunting): _renew_subs 2024-06-03T08:36:19.411 INFO:teuthology.misc.health.smithi088.stderr:2024-06-03T08:36:19.407+0000 7ff2489d7640 10 monclient(hunting): authenticate will time out at 1235.217295s 2024-06-03T08:36:19.411 INFO:teuthology.misc.health.smithi088.stderr:2024-06-03T08:36:19.407+0000 7ff24259c640 1 --1- >> v1:172.21.15.88:6790/0 conn(0x7ff2440b1070 0x7ff2440ac5d0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=1).handle_server_banner_and_identify peer v1:172.21.15.88:6790/0 says I am v1:172.21.15.88:50398/0 (socket says 172.21.15.88:50398) 2024-06-03T08:36:19.412 INFO:teuthology.misc.health.smithi088.stderr:2024-06-03T08:36:19.407+0000 7ff24259c640 1 -- 172.21.15.88:0/2406649008 learned_addr learned my addr 172.21.15.88:0/2406649008 (peer_addr_for_me v1:172.21.15.88:0/0) 2024-06-03T08:36:19.412 INFO:teuthology.misc.health.smithi088.stderr:2024-06-03T08:36:19.408+0000 7ff232ffd640 10 client.?.objecter ms_handle_connect 0x7ff2440b1070 2024-06-03T08:36:19.412 INFO:teuthology.misc.health.smithi088.stderr:2024-06-03T08:36:19.408+0000 7ff232ffd640 10 client.?.objecter resend_mon_ops 2024-06-03T08:36:19.412 INFO:teuthology.misc.health.smithi088.stderr:2024-06-03T08:36:19.408+0000 7ff232ffd640 10 client.?.objecter ms_handle_connect 0x7ff2440aa510 2024-06-03T08:36:19.412 INFO:teuthology.misc.health.smithi088.stderr:2024-06-03T08:36:19.408+0000 7ff232ffd640 10 client.?.objecter resend_mon_ops 2024-06-03T08:36:19.412 INFO:teuthology.misc.health.smithi088.stderr:2024-06-03T08:36:19.411+0000 7ff232ffd640 1 -- 172.21.15.88:0/2406649008 <== mon.0 v1:172.21.15.88:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 751081259 0 0) 0x7ff2440abf40 con 0x7ff2440aa510 2024-06-03T08:36:19.412 INFO:teuthology.misc.health.smithi088.stderr:2024-06-03T08:36:19.411+0000 7ff232ffd640 10 monclient(hunting): _negotiate 2024-06-03T08:36:19.413 INFO:teuthology.misc.health.smithi088.stderr:2024-06-03T08:36:19.411+0000 7ff232ffd640 10 monclient(hunting): _init_auth method 2 2024-06-03T08:36:19.413 INFO:teuthology.misc.health.smithi088.stderr:2024-06-03T08:36:19.411+0000 7ff232ffd640 10 monclient(hunting): _init_auth creating new auth 2024-06-03T08:36:19.413 INFO:teuthology.misc.health.smithi088.stderr:2024-06-03T08:36:19.411+0000 7ff232ffd640 10 monclient(hunting): my global_id is 4257 2024-06-03T08:36:19.413 INFO:teuthology.misc.health.smithi088.stderr:2024-06-03T08:36:19.411+0000 7ff232ffd640 1 -- 172.21.15.88:0/2406649008 --> v1:172.21.15.88:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7ff224004eb0 con 0x7ff2440aa510 2024-06-03T08:36:19.413 INFO:teuthology.misc.health.smithi088.stderr:2024-06-03T08:36:19.413+0000 7ff232ffd640 1 -- 172.21.15.88:0/2406649008 <== mon.2 v1:172.21.15.88:6790/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 3767315706 0 0) 0x7ff244142110 con 0x7ff2440b1070 2024-06-03T08:36:19.413 INFO:teuthology.misc.health.smithi088.stderr:2024-06-03T08:36:19.413+0000 7ff232ffd640 10 monclient(hunting): _negotiate 2024-06-03T08:36:19.413 INFO:teuthology.misc.health.smithi088.stderr:2024-06-03T08:36:19.413+0000 7ff232ffd640 10 monclient(hunting): _init_auth method 2 2024-06-03T08:36:19.414 INFO:teuthology.misc.health.smithi088.stderr:2024-06-03T08:36:19.413+0000 7ff232ffd640 10 monclient(hunting): _init_auth creating new auth 2024-06-03T08:36:19.414 INFO:teuthology.misc.health.smithi088.stderr:2024-06-03T08:36:19.413+0000 7ff232ffd640 10 monclient(hunting): my global_id is 4265 2024-06-03T08:36:19.414 INFO:teuthology.misc.health.smithi088.stderr:2024-06-03T08:36:19.413+0000 7ff232ffd640 1 -- 172.21.15.88:0/2406649008 --> v1:172.21.15.88:6790/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7ff2440abf40 con 0x7ff2440b1070 2024-06-03T08:36:19.419 INFO:teuthology.misc.health.smithi088.stderr:2024-06-03T08:36:19.418+0000 7ff232ffd640 1 -- 172.21.15.88:0/2406649008 <== mon.0 v1:172.21.15.88:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 764+0+0 (unknown 1500498182 0 0) 0x7ff224004eb0 con 0x7ff2440aa510 2024-06-03T08:36:19.419 INFO:teuthology.misc.health.smithi088.stderr:2024-06-03T08:36:19.418+0000 7ff232ffd640 1 -- 172.21.15.88:0/2406649008 --> v1:172.21.15.88:6789/0 -- auth(proto 2 165 bytes epoch 0) v1 -- 0x7ff244142110 con 0x7ff2440aa510 2024-06-03T08:36:19.420 INFO:teuthology.misc.health.smithi088.stderr:2024-06-03T08:36:19.418+0000 7ff232ffd640 1 -- 172.21.15.88:0/2406649008 <== mon.0 v1:172.21.15.88:6789/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 2234374065 0 0) 0x7ff23c001ec0 con 0x7ff2440aa510 2024-06-03T08:36:19.420 INFO:teuthology.misc.health.smithi088.stderr:2024-06-03T08:36:19.418+0000 7ff232ffd640 10 monclient(hunting): handle_monmap mon_map magic: 0 v1 2024-06-03T08:36:19.420 INFO:teuthology.misc.health.smithi088.stderr:2024-06-03T08:36:19.418+0000 7ff232ffd640 10 monclient(hunting): got monmap 1 from mon.noname-a (according to old e1) 2024-06-03T08:36:19.420 INFO:teuthology.misc.health.smithi088.stderr:2024-06-03T08:36:19.418+0000 7ff232ffd640 10 monclient(hunting): dump: 2024-06-03T08:36:19.420 INFO:teuthology.misc.health.smithi088.stderr:epoch 1 2024-06-03T08:36:19.420 INFO:teuthology.misc.health.smithi088.stderr:fsid ec3c3fdd-3ac6-4d7d-9412-683130ed8b04 2024-06-03T08:36:19.420 INFO:teuthology.misc.health.smithi088.stderr:last_changed 2024-06-03T08:34:31.885125+0000 2024-06-03T08:36:19.420 INFO:teuthology.misc.health.smithi088.stderr:created 2024-06-03T08:34:31.885125+0000 2024-06-03T08:36:19.420 INFO:teuthology.misc.health.smithi088.stderr:min_mon_release 18 (reef) 2024-06-03T08:36:19.420 INFO:teuthology.misc.health.smithi088.stderr:election_strategy: 3 2024-06-03T08:36:19.420 INFO:teuthology.misc.health.smithi088.stderr:0: v1:172.21.15.88:6789/0 mon.a 2024-06-03T08:36:19.421 INFO:teuthology.misc.health.smithi088.stderr:1: v1:172.21.15.192:6789/0 mon.b 2024-06-03T08:36:19.421 INFO:teuthology.misc.health.smithi088.stderr:2: v1:172.21.15.88:6790/0 mon.c 2024-06-03T08:36:19.421 INFO:teuthology.misc.health.smithi088.stderr: 2024-06-03T08:36:19.421 INFO:teuthology.misc.health.smithi088.stderr:2024-06-03T08:36:19.418+0000 7ff232ffd640 10 monclient(hunting): _finish_auth 0 2024-06-03T08:36:19.421 INFO:teuthology.misc.health.smithi088.stderr:2024-06-03T08:36:19.419+0000 7ff232ffd640 1 -- 172.21.15.88:0/2406649008 <== mon.2 v1:172.21.15.88:6790/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 764+0+0 (unknown 1649310806 0 0) 0x7ff2440abf40 con 0x7ff2440b1070 2024-06-03T08:36:19.421 INFO:teuthology.misc.health.smithi088.stderr:2024-06-03T08:36:19.419+0000 7ff232ffd640 1 -- 172.21.15.88:0/2406649008 --> v1:172.21.15.88:6790/0 -- auth(proto 2 165 bytes epoch 0) v1 -- 0x7ff224004eb0 con 0x7ff2440b1070 2024-06-03T08:36:19.421 INFO:teuthology.misc.health.smithi088.stderr:2024-06-03T08:36:19.419+0000 7ff232ffd640 1 -- 172.21.15.88:0/2406649008 <== mon.2 v1:172.21.15.88:6790/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 2234374065 0 0) 0x7ff2340018e0 con 0x7ff2440b1070 2024-06-03T08:36:19.421 INFO:teuthology.misc.health.smithi088.stderr:2024-06-03T08:36:19.419+0000 7ff232ffd640 10 monclient(hunting): handle_monmap mon_map magic: 0 v1 2024-06-03T08:36:19.421 INFO:teuthology.misc.health.smithi088.stderr:2024-06-03T08:36:19.419+0000 7ff232ffd640 10 monclient(hunting): got monmap 1 from mon.c (according to old e1) 2024-06-03T08:36:19.421 INFO:teuthology.misc.health.smithi088.stderr:2024-06-03T08:36:19.419+0000 7ff232ffd640 10 monclient(hunting): dump: 2024-06-03T08:36:19.421 INFO:teuthology.misc.health.smithi088.stderr:epoch 1 2024-06-03T08:36:19.422 INFO:teuthology.misc.health.smithi088.stderr:fsid ec3c3fdd-3ac6-4d7d-9412-683130ed8b04 2024-06-03T08:36:19.422 INFO:teuthology.misc.health.smithi088.stderr:last_changed 2024-06-03T08:34:31.885125+0000 2024-06-03T08:36:19.422 INFO:teuthology.misc.health.smithi088.stderr:created 2024-06-03T08:34:31.885125+0000 2024-06-03T08:36:19.422 INFO:teuthology.misc.health.smithi088.stderr:min_mon_release 18 (reef) 2024-06-03T08:36:19.422 INFO:teuthology.misc.health.smithi088.stderr:election_strategy: 3 2024-06-03T08:36:19.422 INFO:teuthology.misc.health.smithi088.stderr:0: v1:172.21.15.88:6789/0 mon.a 2024-06-03T08:36:19.422 INFO:teuthology.misc.health.smithi088.stderr:1: v1:172.21.15.192:6789/0 mon.b 2024-06-03T08:36:19.422 INFO:teuthology.misc.health.smithi088.stderr:2: v1:172.21.15.88:6790/0 mon.c 2024-06-03T08:36:19.422 INFO:teuthology.misc.health.smithi088.stderr: 2024-06-03T08:36:19.424 INFO:teuthology.misc.health.smithi088.stderr:2024-06-03T08:36:19.423+0000 7ff232ffd640 1 -- 172.21.15.88:0/2406649008 <== mon.2 v1:172.21.15.88:6790/0 4 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 580+0+0 (unknown 830268934 0 0) 0x7ff224004eb0 con 0x7ff2440b1070 2024-06-03T08:36:19.424 INFO:teuthology.misc.health.smithi088.stderr:2024-06-03T08:36:19.424+0000 7ff232ffd640 1 -- 172.21.15.88:0/2406649008 >> v1:172.21.15.192:6789/0 conn(0x7ff244144b40 legacy=0x7ff244139f70 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-06-03T08:36:19.424 INFO:teuthology.misc.health.smithi088.stderr:2024-06-03T08:36:19.424+0000 7ff232ffd640 1 -- 172.21.15.88:0/2406649008 >> v1:172.21.15.88:6789/0 conn(0x7ff2440aa510 legacy=0x7ff24413a680 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-06-03T08:36:19.424 INFO:teuthology.misc.health.smithi088.stderr:2024-06-03T08:36:19.424+0000 7ff232ffd640 10 monclient: _finish_hunting 0 2024-06-03T08:36:19.424 INFO:teuthology.misc.health.smithi088.stderr:2024-06-03T08:36:19.424+0000 7ff232ffd640 1 monclient: found mon.c 2024-06-03T08:36:19.424 INFO:teuthology.misc.health.smithi088.stderr:2024-06-03T08:36:19.424+0000 7ff232ffd640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-06-03T08:36:19.425 INFO:teuthology.misc.health.smithi088.stderr:2024-06-03T08:36:19.424+0000 7ff232ffd640 10 monclient: _send_mon_message to mon.c at v1:172.21.15.88:6790/0 2024-06-03T08:36:19.425 INFO:teuthology.misc.health.smithi088.stderr:2024-06-03T08:36:19.424+0000 7ff232ffd640 1 -- 172.21.15.88:0/2406649008 --> v1:172.21.15.88:6790/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7ff2441418a0 con 0x7ff2440b1070 2024-06-03T08:36:19.425 INFO:teuthology.misc.health.smithi088.stderr:2024-06-03T08:36:19.424+0000 7ff232ffd640 10 monclient: _finish_auth 0 2024-06-03T08:36:19.425 INFO:teuthology.misc.health.smithi088.stderr:2024-06-03T08:36:19.424+0000 7ff232ffd640 10 monclient: _check_auth_tickets 2024-06-03T08:36:19.425 INFO:teuthology.misc.health.smithi088.stderr:2024-06-03T08:36:19.424+0000 7ff232ffd640 20 monclient: _check_auth_rotating not needed by client.admin 2024-06-03T08:36:19.425 INFO:teuthology.misc.health.smithi088.stderr:2024-06-03T08:36:19.424+0000 7ff2489d7640 5 monclient: authenticate success, global_id 4265 2024-06-03T08:36:19.425 INFO:teuthology.misc.health.smithi088.stderr:2024-06-03T08:36:19.424+0000 7ff2489d7640 10 monclient: _renew_subs 2024-06-03T08:36:19.425 INFO:teuthology.misc.health.smithi088.stderr:2024-06-03T08:36:19.424+0000 7ff2489d7640 10 monclient: _send_mon_message to mon.c at v1:172.21.15.88:6790/0 2024-06-03T08:36:19.425 INFO:teuthology.misc.health.smithi088.stderr:2024-06-03T08:36:19.424+0000 7ff2489d7640 1 -- 172.21.15.88:0/2406649008 --> v1:172.21.15.88:6790/0 -- mon_subscribe({mgrmap=0+}) v3 -- 0x7ff2440aaec0 con 0x7ff2440b1070 2024-06-03T08:36:19.425 INFO:teuthology.misc.health.smithi088.stderr:2024-06-03T08:36:19.424+0000 7ff2489d7640 10 client.4265.objecter _maybe_request_map subscribing (onetime) to next osd map 2024-06-03T08:36:19.425 INFO:teuthology.misc.health.smithi088.stderr:2024-06-03T08:36:19.424+0000 7ff2489d7640 10 monclient: _renew_subs 2024-06-03T08:36:19.425 INFO:teuthology.misc.health.smithi088.stderr:2024-06-03T08:36:19.424+0000 7ff2489d7640 10 monclient: _send_mon_message to mon.c at v1:172.21.15.88:6790/0 2024-06-03T08:36:19.425 INFO:teuthology.misc.health.smithi088.stderr:2024-06-03T08:36:19.424+0000 7ff2489d7640 1 -- 172.21.15.88:0/2406649008 --> v1:172.21.15.88:6790/0 -- mon_subscribe({osdmap=0}) v3 -- 0x7ff2440acce0 con 0x7ff2440b1070 2024-06-03T08:36:19.425 INFO:teuthology.misc.health.smithi088.stderr:2024-06-03T08:36:19.424+0000 7ff2489d7640 1 librados: init done 2024-06-03T08:36:19.426 INFO:teuthology.misc.health.smithi088.stderr:2024-06-03T08:36:19.425+0000 7ff2489d7640 10 start_mon_command cmd=[{"prefix": "get_command_descriptions"}] 2024-06-03T08:36:19.426 INFO:teuthology.misc.health.smithi088.stderr:2024-06-03T08:36:19.425+0000 7ff2489d7640 10 monclient: _send_command 1 [{"prefix": "get_command_descriptions"}] 2024-06-03T08:36:19.426 INFO:teuthology.misc.health.smithi088.stderr:2024-06-03T08:36:19.425+0000 7ff2489d7640 10 monclient: _send_mon_message to mon.c at v1:172.21.15.88:6790/0 2024-06-03T08:36:19.426 INFO:teuthology.misc.health.smithi088.stderr:2024-06-03T08:36:19.425+0000 7ff2489d7640 1 -- 172.21.15.88:0/2406649008 --> v1:172.21.15.88:6790/0 -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7ff244141af0 con 0x7ff2440b1070 2024-06-03T08:36:19.597 INFO:tasks.ceph.osd.5.smithi192.stderr:2024-06-03T08:36:19.214+0000 5dcd500 -1 bdev(0x16952e70 /var/lib/ceph/osd/ceph-5/block) aio_submit retries 1 2024-06-03T08:36:19.631 INFO:tasks.ceph.osd.2.smithi088.stderr:2024-06-03T08:36:19.275+0000 5dcd500 -1 bdev(0x18a920e0 /var/lib/ceph/osd/ceph-2/block) aio_submit retries 2 2024-06-03T08:36:19.632 INFO:tasks.ceph.osd.2.smithi088.stderr:2024-06-03T08:36:19.316+0000 5dcd500 -1 bdev(0x18a920e0 /var/lib/ceph/osd/ceph-2/block) aio_submit retries 1 2024-06-03T08:36:19.669 INFO:tasks.ceph.osd.2.smithi088.stderr:2024-06-03T08:36:19.347+0000 5dcd500 -1 bdev(0x18a920e0 /var/lib/ceph/osd/ceph-2/block) aio_submit retries 1 2024-06-03T08:36:19.669 INFO:tasks.ceph.osd.2.smithi088.stderr:2024-06-03T08:36:19.633+0000 5dcd500 -1 bdev(0x18a920e0 /var/lib/ceph/osd/ceph-2/block) aio_submit retries 2 2024-06-03T08:36:19.683 INFO:tasks.ceph.osd.2.smithi088.stderr:2024-06-03T08:36:19.670+0000 5dcd500 -1 bdev(0x18a920e0 /var/lib/ceph/osd/ceph-2/block) aio_submit retries 2 2024-06-03T08:36:19.705 INFO:tasks.ceph.osd.2.smithi088.stderr:2024-06-03T08:36:19.683+0000 5dcd500 -1 bdev(0x18a920e0 /var/lib/ceph/osd/ceph-2/block) aio_submit retries 1 2024-06-03T08:36:19.716 INFO:tasks.ceph.osd.2.smithi088.stderr:2024-06-03T08:36:19.705+0000 5dcd500 -1 bdev(0x18a920e0 /var/lib/ceph/osd/ceph-2/block) aio_submit retries 1 2024-06-03T08:36:19.728 INFO:tasks.ceph.osd.2.smithi088.stderr:2024-06-03T08:36:19.716+0000 5dcd500 -1 bdev(0x18a920e0 /var/lib/ceph/osd/ceph-2/block) aio_submit retries 1 2024-06-03T08:36:19.744 INFO:teuthology.misc.health.smithi088.stderr:2024-06-03T08:36:19.744+0000 7ff232ffd640 1 -- 172.21.15.88:0/2406649008 <== mon.2 v1:172.21.15.88:6790/0 5 ==== config(0 keys) v1 ==== 4+0+0 (unknown 0 0 0) 0x7ff2340035f0 con 0x7ff2440b1070 2024-06-03T08:36:19.744 INFO:teuthology.misc.health.smithi088.stderr:2024-06-03T08:36:19.744+0000 7ff232ffd640 10 monclient: handle_config config(0 keys) v1 2024-06-03T08:36:19.745 INFO:teuthology.misc.health.smithi088.stderr:2024-06-03T08:36:19.744+0000 7ff232ffd640 1 -- 172.21.15.88:0/2406649008 <== mon.2 v1:172.21.15.88:6790/0 6 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 2234374065 0 0) 0x7ff234003980 con 0x7ff2440b1070 2024-06-03T08:36:19.745 INFO:teuthology.misc.health.smithi088.stderr:2024-06-03T08:36:19.744+0000 7ff232ffd640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-06-03T08:36:19.745 INFO:teuthology.misc.health.smithi088.stderr:2024-06-03T08:36:19.744+0000 7ff232ffd640 10 monclient: got monmap 1 from mon.c (according to old e1) 2024-06-03T08:36:19.745 INFO:teuthology.misc.health.smithi088.stderr:2024-06-03T08:36:19.744+0000 7ff232ffd640 10 monclient: dump: 2024-06-03T08:36:19.745 INFO:teuthology.misc.health.smithi088.stderr:epoch 1 2024-06-03T08:36:19.745 INFO:teuthology.misc.health.smithi088.stderr:fsid ec3c3fdd-3ac6-4d7d-9412-683130ed8b04 2024-06-03T08:36:19.745 INFO:teuthology.misc.health.smithi088.stderr:last_changed 2024-06-03T08:34:31.885125+0000 2024-06-03T08:36:19.745 INFO:teuthology.misc.health.smithi088.stderr:created 2024-06-03T08:34:31.885125+0000 2024-06-03T08:36:19.745 INFO:teuthology.misc.health.smithi088.stderr:min_mon_release 18 (reef) 2024-06-03T08:36:19.745 INFO:teuthology.misc.health.smithi088.stderr:election_strategy: 3 2024-06-03T08:36:19.745 INFO:teuthology.misc.health.smithi088.stderr:0: v1:172.21.15.88:6789/0 mon.a 2024-06-03T08:36:19.745 INFO:teuthology.misc.health.smithi088.stderr:1: v1:172.21.15.192:6789/0 mon.b 2024-06-03T08:36:19.745 INFO:teuthology.misc.health.smithi088.stderr:2: v1:172.21.15.88:6790/0 mon.c 2024-06-03T08:36:19.746 INFO:teuthology.misc.health.smithi088.stderr: 2024-06-03T08:36:19.746 INFO:teuthology.misc.health.smithi088.stderr:2024-06-03T08:36:19.744+0000 7ff240d99640 4 set_mon_vals no callback set 2024-06-03T08:36:19.746 INFO:teuthology.misc.health.smithi088.stderr:2024-06-03T08:36:19.745+0000 7ff232ffd640 1 -- 172.21.15.88:0/2406649008 <== mon.2 v1:172.21.15.88:6790/0 7 ==== mgrmap(e 4) v1 ==== 89054+0+0 (unknown 914547237 0 0) 0x7ff234003ae0 con 0x7ff2440b1070 2024-06-03T08:36:19.746 INFO:teuthology.misc.health.smithi088.stderr:2024-06-03T08:36:19.746+0000 7ff241d9b640 10 monclient: get_auth_request con 0x7ff22406d160 auth_method 0 2024-06-03T08:36:19.746 INFO:teuthology.misc.health.smithi088.stderr:2024-06-03T08:36:19.746+0000 7ff232ffd640 10 client.4265.objecter ms_handle_connect 0x7ff22406d160 2024-06-03T08:36:19.751 INFO:teuthology.misc.health.smithi088.stderr:2024-06-03T08:36:19.751+0000 7ff232ffd640 1 -- 172.21.15.88:0/2406649008 <== mon.2 v1:172.21.15.88:6790/0 8 ==== osd_map(19..19 src has 1..19) v4 ==== 3037+0+0 (unknown 2512651453 0 0) 0x7ff234054060 con 0x7ff2440b1070 2024-06-03T08:36:19.751 INFO:teuthology.misc.health.smithi088.stderr:2024-06-03T08:36:19.751+0000 7ff232ffd640 10 client.4265.objecter ms_dispatch 0x7ff244000b90 osd_map(19..19 src has 1..19) v4 2024-06-03T08:36:19.751 INFO:teuthology.misc.health.smithi088.stderr:2024-06-03T08:36:19.751+0000 7ff232ffd640 3 client.4265.objecter handle_osd_map got epochs [19,19] > 0 2024-06-03T08:36:19.751 INFO:teuthology.misc.health.smithi088.stderr:2024-06-03T08:36:19.751+0000 7ff232ffd640 3 client.4265.objecter handle_osd_map decoding full epoch 19 2024-06-03T08:36:19.751 INFO:teuthology.misc.health.smithi088.stderr:2024-06-03T08:36:19.751+0000 7ff232ffd640 20 client.4265.objecter dump_active .. 0 homeless 2024-06-03T08:36:19.752 INFO:teuthology.misc.health.smithi088.stderr:2024-06-03T08:36:19.751+0000 7ff232ffd640 1 -- 172.21.15.88:0/2406649008 <== mon.2 v1:172.21.15.88:6790/0 9 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+182498 (unknown 1092875540 0 3016760700) 0x7ff23404fc10 con 0x7ff2440b1070 2024-06-03T08:36:19.752 INFO:teuthology.misc.health.smithi088.stderr:2024-06-03T08:36:19.751+0000 7ff232ffd640 10 monclient: handle_mon_command_ack 1 [{"prefix": "get_command_descriptions"}] 2024-06-03T08:36:19.752 INFO:teuthology.misc.health.smithi088.stderr:2024-06-03T08:36:19.751+0000 7ff232ffd640 10 monclient: _finish_command 1 = system:0 2024-06-03T08:36:19.759 INFO:tasks.ceph.osd.2.smithi088.stderr:2024-06-03T08:36:19.728+0000 5dcd500 -1 bdev(0x18a920e0 /var/lib/ceph/osd/ceph-2/block) aio_submit retries 1 2024-06-03T08:36:19.865 INFO:tasks.ceph.osd.2.smithi088.stderr:2024-06-03T08:36:19.739+0000 5dcd500 -1 bdev(0x18a920e0 /var/lib/ceph/osd/ceph-2/block) aio_submit retries 1 2024-06-03T08:36:19.865 INFO:tasks.ceph.osd.2.smithi088.stderr:2024-06-03T08:36:19.750+0000 5dcd500 -1 bdev(0x18a920e0 /var/lib/ceph/osd/ceph-2/block) aio_submit retries 1 2024-06-03T08:36:19.876 INFO:tasks.ceph.osd.2.smithi088.stderr:2024-06-03T08:36:19.865+0000 5dcd500 -1 bdev(0x18a920e0 /var/lib/ceph/osd/ceph-2/block) aio_submit retries 1 2024-06-03T08:36:19.961 INFO:teuthology.misc.health.smithi088.stderr:2024-06-03T08:36:19.960+0000 7ff2489d7640 10 start_mon_command cmd=[{"prefix": "osd dump", "format": "json"}] 2024-06-03T08:36:19.961 INFO:teuthology.misc.health.smithi088.stderr:2024-06-03T08:36:19.960+0000 7ff2489d7640 10 monclient: _send_command 2 [{"prefix": "osd dump", "format": "json"}] 2024-06-03T08:36:19.961 INFO:teuthology.misc.health.smithi088.stderr:2024-06-03T08:36:19.960+0000 7ff2489d7640 10 monclient: _send_mon_message to mon.c at v1:172.21.15.88:6790/0 2024-06-03T08:36:19.961 INFO:teuthology.misc.health.smithi088.stderr:2024-06-03T08:36:19.960+0000 7ff2489d7640 1 -- 172.21.15.88:0/2406649008 --> v1:172.21.15.88:6790/0 -- mon_command({"prefix": "osd dump", "format": "json"} v 0) v1 -- 0x7ff2440acfd0 con 0x7ff2440b1070 2024-06-03T08:36:19.962 INFO:tasks.ceph.osd.2.smithi088.stderr:2024-06-03T08:36:19.878+0000 5dcd500 -1 bdev(0x18a920e0 /var/lib/ceph/osd/ceph-2/block) aio_submit retries 1 2024-06-03T08:36:19.965 INFO:tasks.ceph.osd.2.smithi088.stderr:2024-06-03T08:36:19.962+0000 5dcd500 -1 bdev(0x18a920e0 /var/lib/ceph/osd/ceph-2/block) aio_submit retries 2 2024-06-03T08:36:20.021 INFO:tasks.ceph.osd.2.smithi088.stderr:2024-06-03T08:36:19.990+0000 5dcd500 -1 bdev(0x18a920e0 /var/lib/ceph/osd/ceph-2/block) aio_submit retries 1 2024-06-03T08:36:20.028 INFO:tasks.ceph.osd.2.smithi088.stderr:2024-06-03T08:36:20.027+0000 5dcd500 -1 bdev(0x18a920e0 /var/lib/ceph/osd/ceph-2/block) aio_submit retries 3 2024-06-03T08:36:20.048 INFO:teuthology.misc.health.smithi088.stderr:2024-06-03T08:36:20.047+0000 7ff232ffd640 1 -- 172.21.15.88:0/2406649008 <== mon.2 v1:172.21.15.88:6790/0 10 ==== mon_command_ack([{"prefix": "osd dump", "format": "json"}]=0 v19) v1 ==== 74+0+9536 (unknown 941590788 0 3159818570) 0x7ff234051b40 con 0x7ff2440b1070 2024-06-03T08:36:20.048 INFO:teuthology.misc.health.smithi088.stderr:2024-06-03T08:36:20.047+0000 7ff232ffd640 10 monclient: handle_mon_command_ack 2 [{"prefix": "osd dump", "format": "json"}] 2024-06-03T08:36:20.048 INFO:teuthology.misc.health.smithi088.stderr:2024-06-03T08:36:20.047+0000 7ff232ffd640 10 monclient: _finish_command 2 = system:0 2024-06-03T08:36:20.049 INFO:teuthology.misc.health.smithi088.stdout: 2024-06-03T08:36:20.049 INFO:teuthology.misc.health.smithi088.stdout:{"epoch":19,"fsid":"ec3c3fdd-3ac6-4d7d-9412-683130ed8b04","created":"2024-06-03T08:35:26.799638+0000","modified":"2024-06-03T08:36:19.126082+0000","last_up_change":"2024-06-03T08:36:13.284059+0000","last_in_change":"2024-06-03T08:35:32.183068+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":5,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":1,"max_osd":8,"require_min_compat_client":"luminous","min_compat_client":"jewel","require_osd_release":"reef","allow_crimson":false,"pools":[{"pool":1,"pool_name":".mgr","create_time":"2024-06-03T08:36:15.192388+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":1,"pg_placement_num":1,"pg_placement_num_target":1,"pg_num_target":1,"pg_num_pending":1,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"19","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_max":32,"pg_num_min":1},"application_metadata":{"mgr":{}},"read_balance":{"score_acting":7.690000057220459,"score_stable":7.690000057220459,"optimal_score":0.12999999523162842,"raw_score_acting":1,"raw_score_stable":1,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}}],"osds":[{"osd":0,"uuid":"b25ab3bb-3587-45d0-991a-6d41bb529b37","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":14,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.88:6808","nonce":35066}]},"cluster_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.88:6809","nonce":35066}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.88:6811","nonce":35066}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.88:6810","nonce":35066}]},"public_addr":"172.21.15.88:6808/35066","cluster_addr":"172.21.15.88:6809/35066","heartbeat_back_addr":"172.21.15.88:6811/35066","heartbeat_front_addr":"172.21.15.88:6810/35066","state":["exists","up"]},{"osd":1,"uuid":"00d160e7-9773-41d8-8d17-21e70620a0fd","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":14,"up_thru":16,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.88:6800","nonce":35067}]},"cluster_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.88:6801","nonce":35067}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.88:6804","nonce":35067}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.88:6803","nonce":35067}]},"public_addr":"172.21.15.88:6800/35067","cluster_addr":"172.21.15.88:6801/35067","heartbeat_back_addr":"172.21.15.88:6804/35067","heartbeat_front_addr":"172.21.15.88:6803/35067","state":["exists","up"]},{"osd":2,"uuid":"addfb2a7-2bde-4cc3-98cf-17fd2bdfa3a9","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":3,"uuid":"6c5770b0-c93c-4101-8c53-bd761b8abcde","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":14,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.88:6812","nonce":35068}]},"cluster_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.88:6813","nonce":35068}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.88:6815","nonce":35068}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.88:6814","nonce":35068}]},"public_addr":"172.21.15.88:6812/35068","cluster_addr":"172.21.15.88:6813/35068","heartbeat_back_addr":"172.21.15.88:6815/35068","heartbeat_front_addr":"172.21.15.88:6814/35068","state":["exists","up"]},{"osd":4,"uuid":"f001b3cb-2081-428b-bac3-b3df3c19b978","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":5,"uuid":"3cc263e9-e9bf-45e0-a3b7-83a76469eb70","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":6,"uuid":"82c83316-809f-4901-a5ed-34b6a075b393","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":7,"uuid":"aafe0716-1102-4eae-aac6-0f81bb154e7d","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2024-06-03T08:36:10.027533+0000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2024-06-03T08:36:09.853335+0000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2024-06-03T08:36:09.824468+0000","dead_epoch":0},{"osd":4,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":5,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":6,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":7,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[],"pg_upmap_primaries":[],"pg_temp":[],"primary_temp":[],"blocklist":{},"range_blocklist":{},"erasure_code_profiles":{"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","technique":"reed_sol_van"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2024-06-03T08:36:20.051 INFO:teuthology.misc.health.smithi088.stderr:2024-06-03T08:36:20.050+0000 7ff2489d7640 10 librados: watch_flush enter 2024-06-03T08:36:20.051 INFO:teuthology.misc.health.smithi088.stderr:2024-06-03T08:36:20.050+0000 7ff2489d7640 10 librados: watch_flush exit 2024-06-03T08:36:20.051 INFO:teuthology.misc.health.smithi088.stderr:2024-06-03T08:36:20.050+0000 7ff2489d7640 20 client.4265.objecter shutdown clearing up homeless session... 2024-06-03T08:36:20.051 INFO:teuthology.misc.health.smithi088.stderr:2024-06-03T08:36:20.050+0000 7ff2489d7640 10 client.4265.objecter successfully canceled tick 2024-06-03T08:36:20.051 INFO:teuthology.misc.health.smithi088.stderr:2024-06-03T08:36:20.050+0000 7ff2489d7640 1 -- 172.21.15.88:0/2406649008 >> v1:172.21.15.192:6800/34229 conn(0x7ff22406d160 legacy=0x7ff22406f620 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-06-03T08:36:20.051 INFO:teuthology.misc.health.smithi088.stderr:2024-06-03T08:36:20.050+0000 7ff2489d7640 10 monclient: shutdown 2024-06-03T08:36:20.051 INFO:teuthology.misc.health.smithi088.stderr:2024-06-03T08:36:20.050+0000 7ff2489d7640 20 monclient: shutdown discarding 0 pending message(s) 2024-06-03T08:36:20.052 INFO:teuthology.misc.health.smithi088.stderr:2024-06-03T08:36:20.050+0000 7ff2489d7640 1 -- 172.21.15.88:0/2406649008 >> v1:172.21.15.88:6790/0 conn(0x7ff2440b1070 legacy=0x7ff2440ac5d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-06-03T08:36:20.052 INFO:teuthology.misc.health.smithi088.stderr:2024-06-03T08:36:20.050+0000 7ff2489d7640 1 -- 172.21.15.88:0/2406649008 shutdown_connections 2024-06-03T08:36:20.052 INFO:teuthology.misc.health.smithi088.stderr:2024-06-03T08:36:20.050+0000 7ff2489d7640 1 -- 172.21.15.88:0/2406649008 >> 172.21.15.88:0/2406649008 conn(0x7ff2440b71e0 msgr2=0x7ff244144f90 unknown :-1 s=STATE_NONE l=0).mark_down 2024-06-03T08:36:20.052 INFO:teuthology.misc.health.smithi088.stderr:2024-06-03T08:36:20.050+0000 7ff2489d7640 1 -- 172.21.15.88:0/2406649008 shutdown_connections 2024-06-03T08:36:20.052 INFO:teuthology.misc.health.smithi088.stderr:2024-06-03T08:36:20.050+0000 7ff2489d7640 1 -- 172.21.15.88:0/2406649008 wait complete. 2024-06-03T08:36:20.052 INFO:teuthology.misc.health.smithi088.stderr:2024-06-03T08:36:20.051+0000 7ff2489d7640 1 librados: shutdown 2024-06-03T08:36:20.061 DEBUG:teuthology.misc:3 of 8 OSDs are up 2024-06-03T08:36:20.114 INFO:tasks.ceph.osd.2.smithi088.stderr:2024-06-03T08:36:20.034+0000 5dcd500 -1 bdev(0x18a920e0 /var/lib/ceph/osd/ceph-2/block) aio_submit retries 2 2024-06-03T08:36:20.209 INFO:tasks.ceph.osd.2.smithi088.stderr:2024-06-03T08:36:20.158+0000 5dcd500 -1 bdev(0x18a920e0 /var/lib/ceph/osd/ceph-2/block) aio_submit retries 2 2024-06-03T08:36:20.209 INFO:tasks.ceph.osd.2.smithi088.stderr:2024-06-03T08:36:20.197+0000 5dcd500 -1 bdev(0x18a920e0 /var/lib/ceph/osd/ceph-2/block) aio_submit retries 1 2024-06-03T08:36:20.228 INFO:tasks.ceph.osd.2.smithi088.stderr:2024-06-03T08:36:20.208+0000 5dcd500 -1 bdev(0x18a920e0 /var/lib/ceph/osd/ceph-2/block) aio_submit retries 1 2024-06-03T08:36:20.692 INFO:tasks.ceph.osd.4.smithi192.stderr:2024-06-03T08:36:20.690+0000 c4a2640 -1 osd.4 0 waiting for initial osdmap 2024-06-03T08:36:20.779 INFO:tasks.ceph.osd.5.smithi192.stderr:2024-06-03T08:36:20.774+0000 c4a2640 -1 osd.5 0 waiting for initial osdmap 2024-06-03T08:36:20.869 INFO:tasks.ceph.osd.7.smithi192.stderr:2024-06-03T08:36:20.863+0000 ac89640 -1 osd.7 0 waiting for initial osdmap 2024-06-03T08:36:20.928 INFO:tasks.ceph.osd.6.smithi192.stderr:2024-06-03T08:36:20.024+0000 5dcd500 -1 bdev(0x18ab8280 /var/lib/ceph/osd/ceph-6/block) aio_submit retries 1 2024-06-03T08:36:21.183 INFO:tasks.ceph.osd.2.smithi088.stderr:2024-06-03T08:36:21.178+0000 c4a2640 -1 osd.2 0 waiting for initial osdmap 2024-06-03T08:36:21.202 INFO:tasks.ceph.osd.4.smithi192.stderr:2024-06-03T08:36:21.189+0000 1426f640 -1 osd.4 19 set_numa_affinity unable to identify public interface '' numa node: (2) No such file or directory 2024-06-03T08:36:21.350 INFO:tasks.ceph.osd.5.smithi192.stderr:2024-06-03T08:36:21.334+0000 1426f640 -1 osd.5 19 set_numa_affinity unable to identify public interface '' numa node: (2) No such file or directory 2024-06-03T08:36:21.397 INFO:tasks.ceph.osd.7.smithi192.stderr:2024-06-03T08:36:21.384+0000 1426f640 -1 osd.7 19 set_numa_affinity unable to identify public interface '' numa node: (2) No such file or directory 2024-06-03T08:36:21.672 INFO:tasks.ceph.osd.2.smithi088.stderr:2024-06-03T08:36:21.659+0000 1426f640 -1 osd.2 19 set_numa_affinity unable to identify public interface '' numa node: (2) No such file or directory 2024-06-03T08:36:22.131 INFO:tasks.ceph.osd.6.smithi192.stderr:2024-06-03T08:36:22.127+0000 c4a2640 -1 osd.6 0 waiting for initial osdmap 2024-06-03T08:36:22.810 INFO:tasks.ceph.osd.6.smithi192.stderr:2024-06-03T08:36:22.794+0000 1426f640 -1 osd.6 19 set_numa_affinity unable to identify public interface '' numa node: (2) No such file or directory 2024-06-03T08:36:23.921 INFO:tasks.ceph.mon.a.smithi088.stderr:==00:00:01:19.246 34743== Warning: unimplemented fcntl command: 1036 2024-06-03T08:36:23.980 INFO:tasks.ceph.mon.b.smithi192.stderr:==00:00:01:19.282 34228== Warning: unimplemented fcntl command: 1036 2024-06-03T08:36:23.985 INFO:tasks.ceph.mon.a.smithi088.stderr:==00:00:01:19.311 34743== Warning: unimplemented fcntl command: 1036 2024-06-03T08:36:24.024 INFO:tasks.ceph.mon.c.smithi088.stderr:==00:00:01:19.350 34749== Warning: unimplemented fcntl command: 1036 2024-06-03T08:36:24.040 INFO:tasks.ceph.mon.b.smithi192.stderr:==00:00:01:19.342 34228== Warning: unimplemented fcntl command: 1036 2024-06-03T08:36:24.099 INFO:tasks.ceph.mon.c.smithi088.stderr:==00:00:01:19.424 34749== Warning: unimplemented fcntl command: 1036 2024-06-03T08:36:24.285 INFO:tasks.ceph.mon.b.smithi192.stderr:==00:00:01:19.587 34228== Warning: unimplemented fcntl command: 1036 2024-06-03T08:36:24.289 INFO:tasks.ceph.mon.a.smithi088.stderr:==00:00:01:19.614 34743== Warning: unimplemented fcntl command: 1036 2024-06-03T08:36:24.383 INFO:tasks.ceph.mon.c.smithi088.stderr:==00:00:01:19.708 34749== Warning: unimplemented fcntl command: 1036 2024-06-03T08:36:26.868 DEBUG:teuthology.orchestra.run.smithi088:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph --cluster ceph osd dump --format=json 2024-06-03T08:36:26.957 INFO:teuthology.misc.health.smithi088.stderr:2024-06-03T08:36:26.957+0000 7f56d8595640 10 monclient: get_monmap_and_config 2024-06-03T08:36:26.958 INFO:teuthology.misc.health.smithi088.stderr:2024-06-03T08:36:26.957+0000 7f56d8595640 10 monclient: build_initial_monmap 2024-06-03T08:36:26.958 INFO:teuthology.misc.health.smithi088.stderr:2024-06-03T08:36:26.957+0000 7f56d8595640 10 monclient: monmap: 2024-06-03T08:36:26.958 INFO:teuthology.misc.health.smithi088.stderr:epoch 0 2024-06-03T08:36:26.958 INFO:teuthology.misc.health.smithi088.stderr:fsid ec3c3fdd-3ac6-4d7d-9412-683130ed8b04 2024-06-03T08:36:26.958 INFO:teuthology.misc.health.smithi088.stderr:last_changed 2024-06-03T08:36:26.958546+0000 2024-06-03T08:36:26.958 INFO:teuthology.misc.health.smithi088.stderr:created 2024-06-03T08:36:26.958546+0000 2024-06-03T08:36:26.958 INFO:teuthology.misc.health.smithi088.stderr:min_mon_release 0 (unknown) 2024-06-03T08:36:26.958 INFO:teuthology.misc.health.smithi088.stderr:election_strategy: 3 2024-06-03T08:36:26.958 INFO:teuthology.misc.health.smithi088.stderr:0: v1:172.21.15.88:6789/0 mon.noname-a 2024-06-03T08:36:26.958 INFO:teuthology.misc.health.smithi088.stderr:1: v1:172.21.15.192:6789/0 mon.noname-c 2024-06-03T08:36:26.958 INFO:teuthology.misc.health.smithi088.stderr:2: v1:172.21.15.88:6790/0 mon.noname-b 2024-06-03T08:36:26.958 INFO:teuthology.misc.health.smithi088.stderr: 2024-06-03T08:36:26.959 INFO:teuthology.misc.health.smithi088.stderr:2024-06-03T08:36:26.958+0000 7f56d8595640 1 Processor -- start 2024-06-03T08:36:26.960 INFO:teuthology.misc.health.smithi088.stderr:2024-06-03T08:36:26.958+0000 7f56d8595640 1 -- start start 2024-06-03T08:36:26.960 INFO:teuthology.misc.health.smithi088.stderr:2024-06-03T08:36:26.958+0000 7f56d8595640 10 monclient: init 2024-06-03T08:36:26.960 INFO:teuthology.misc.health.smithi088.stderr:2024-06-03T08:36:26.958+0000 7f56d8595640 10 monclient: _reopen_session rank -1 2024-06-03T08:36:26.960 INFO:teuthology.misc.health.smithi088.stderr:2024-06-03T08:36:26.958+0000 7f56d8595640 10 monclient: _add_conns ranks=[1,0,2] 2024-06-03T08:36:26.960 INFO:teuthology.misc.health.smithi088.stderr:2024-06-03T08:36:26.958+0000 7f56d8595640 10 monclient(hunting): picked mon.noname-c con 0x7f56d0070db0 addr v1:172.21.15.192:6789/0 2024-06-03T08:36:26.960 INFO:teuthology.misc.health.smithi088.stderr:2024-06-03T08:36:26.958+0000 7f56d8595640 10 monclient(hunting): picked mon.noname-a con 0x7f56d0071190 addr v1:172.21.15.88:6789/0 2024-06-03T08:36:26.960 INFO:teuthology.misc.health.smithi088.stderr:2024-06-03T08:36:26.958+0000 7f56d8595640 10 monclient(hunting): picked mon.noname-b con 0x7f56d013d940 addr v1:172.21.15.88:6790/0 2024-06-03T08:36:26.960 INFO:teuthology.misc.health.smithi088.stderr:2024-06-03T08:36:26.958+0000 7f56d8595640 1 -- --> v1:172.21.15.88:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f56d00af860 con 0x7f56d0071190 2024-06-03T08:36:26.960 INFO:teuthology.misc.health.smithi088.stderr:2024-06-03T08:36:26.958+0000 7f56d8595640 1 -- --> v1:172.21.15.192:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f56d0070060 con 0x7f56d0070db0 2024-06-03T08:36:26.960 INFO:teuthology.misc.health.smithi088.stderr:2024-06-03T08:36:26.958+0000 7f56d8595640 1 -- --> v1:172.21.15.88:6790/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f56d00702d0 con 0x7f56d013d940 2024-06-03T08:36:26.960 INFO:teuthology.misc.health.smithi088.stderr:2024-06-03T08:36:26.958+0000 7f56d8595640 10 monclient(hunting): _renew_subs 2024-06-03T08:36:26.960 INFO:teuthology.misc.health.smithi088.stderr:2024-06-03T08:36:26.958+0000 7f56d8595640 10 monclient(hunting): authenticate will time out at 1242.768840s 2024-06-03T08:36:26.961 INFO:teuthology.misc.health.smithi088.stderr:2024-06-03T08:36:26.959+0000 7f56d6b32640 1 --1- >> v1:172.21.15.88:6790/0 conn(0x7f56d013d940 0x7f56d0149260 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.88:6790/0 says I am v1:172.21.15.88:50408/0 (socket says 172.21.15.88:50408) 2024-06-03T08:36:26.961 INFO:teuthology.misc.health.smithi088.stderr:2024-06-03T08:36:26.959+0000 7f56d6331640 1 --1- >> v1:172.21.15.192:6789/0 conn(0x7f56d0070db0 0x7f56d00ae1b0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.192:6789/0 says I am v1:172.21.15.88:38420/0 (socket says 172.21.15.88:38420) 2024-06-03T08:36:26.961 INFO:teuthology.misc.health.smithi088.stderr:2024-06-03T08:36:26.959+0000 7f56d6b32640 1 -- 172.21.15.88:0/4084105133 learned_addr learned my addr 172.21.15.88:0/4084105133 (peer_addr_for_me v1:172.21.15.88:0/0) 2024-06-03T08:36:26.962 INFO:teuthology.misc.health.smithi088.stderr:2024-06-03T08:36:26.961+0000 7f56d532f640 1 -- 172.21.15.88:0/4084105133 <== mon.1 v1:172.21.15.192:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 2503230172 0 0) 0x7f56d0070060 con 0x7f56d0070db0 2024-06-03T08:36:26.962 INFO:teuthology.misc.health.smithi088.stderr:2024-06-03T08:36:26.961+0000 7f56d532f640 10 monclient(hunting): _negotiate 2024-06-03T08:36:26.962 INFO:teuthology.misc.health.smithi088.stderr:2024-06-03T08:36:26.961+0000 7f56d532f640 10 monclient(hunting): _init_auth method 2 2024-06-03T08:36:26.962 INFO:teuthology.misc.health.smithi088.stderr:2024-06-03T08:36:26.961+0000 7f56d532f640 10 monclient(hunting): _init_auth creating new auth 2024-06-03T08:36:26.962 INFO:teuthology.misc.health.smithi088.stderr:2024-06-03T08:36:26.961+0000 7f56d532f640 10 monclient(hunting): my global_id is 4258 2024-06-03T08:36:26.962 INFO:teuthology.misc.health.smithi088.stderr:2024-06-03T08:36:26.961+0000 7f56d532f640 1 -- 172.21.15.88:0/4084105133 --> v1:172.21.15.192:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f56bc004630 con 0x7f56d0070db0 2024-06-03T08:36:26.963 INFO:teuthology.misc.health.smithi088.stderr:2024-06-03T08:36:26.962+0000 7f56d532f640 1 -- 172.21.15.88:0/4084105133 <== mon.2 v1:172.21.15.88:6790/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 2511826490 0 0) 0x7f56d00702d0 con 0x7f56d013d940 2024-06-03T08:36:26.963 INFO:teuthology.misc.health.smithi088.stderr:2024-06-03T08:36:26.962+0000 7f56d532f640 10 monclient(hunting): _negotiate 2024-06-03T08:36:26.963 INFO:teuthology.misc.health.smithi088.stderr:2024-06-03T08:36:26.962+0000 7f56d532f640 10 monclient(hunting): _init_auth method 2 2024-06-03T08:36:26.963 INFO:teuthology.misc.health.smithi088.stderr:2024-06-03T08:36:26.962+0000 7f56d532f640 10 monclient(hunting): _init_auth creating new auth 2024-06-03T08:36:26.963 INFO:teuthology.misc.health.smithi088.stderr:2024-06-03T08:36:26.962+0000 7f56d532f640 10 monclient(hunting): my global_id is 4277 2024-06-03T08:36:26.963 INFO:teuthology.misc.health.smithi088.stderr:2024-06-03T08:36:26.962+0000 7f56d532f640 1 -- 172.21.15.88:0/4084105133 --> v1:172.21.15.88:6790/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f56d0070060 con 0x7f56d013d940 2024-06-03T08:36:26.963 INFO:teuthology.misc.health.smithi088.stderr:2024-06-03T08:36:26.962+0000 7f56d532f640 1 -- 172.21.15.88:0/4084105133 <== mon.0 v1:172.21.15.88:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 2403764520 0 0) 0x7f56d00af860 con 0x7f56d0071190 2024-06-03T08:36:26.963 INFO:teuthology.misc.health.smithi088.stderr:2024-06-03T08:36:26.962+0000 7f56d532f640 10 monclient(hunting): _negotiate 2024-06-03T08:36:26.963 INFO:teuthology.misc.health.smithi088.stderr:2024-06-03T08:36:26.962+0000 7f56d532f640 10 monclient(hunting): _init_auth method 2 2024-06-03T08:36:26.963 INFO:teuthology.misc.health.smithi088.stderr:2024-06-03T08:36:26.962+0000 7f56d532f640 10 monclient(hunting): _init_auth creating new auth 2024-06-03T08:36:26.964 INFO:teuthology.misc.health.smithi088.stderr:2024-06-03T08:36:26.962+0000 7f56d532f640 10 monclient(hunting): my global_id is 4263 2024-06-03T08:36:26.964 INFO:teuthology.misc.health.smithi088.stderr:2024-06-03T08:36:26.962+0000 7f56d532f640 1 -- 172.21.15.88:0/4084105133 --> v1:172.21.15.88:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f56d00702d0 con 0x7f56d0071190 2024-06-03T08:36:26.965 INFO:teuthology.misc.health.smithi088.stderr:2024-06-03T08:36:26.964+0000 7f56d532f640 1 -- 172.21.15.88:0/4084105133 <== mon.2 v1:172.21.15.88:6790/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 214+0+0 (unknown 1509499271 0 0) 0x7f56d0070060 con 0x7f56d013d940 2024-06-03T08:36:26.965 INFO:teuthology.misc.health.smithi088.stderr:2024-06-03T08:36:26.964+0000 7f56d532f640 1 -- 172.21.15.88:0/4084105133 >> v1:172.21.15.192:6789/0 conn(0x7f56d0070db0 legacy=0x7f56d00ae1b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-06-03T08:36:26.965 INFO:teuthology.misc.health.smithi088.stderr:2024-06-03T08:36:26.964+0000 7f56d532f640 1 -- 172.21.15.88:0/4084105133 >> v1:172.21.15.88:6789/0 conn(0x7f56d0071190 legacy=0x7f56d013ad00 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-06-03T08:36:26.965 INFO:teuthology.misc.health.smithi088.stderr:2024-06-03T08:36:26.964+0000 7f56d532f640 10 monclient: _finish_hunting 0 2024-06-03T08:36:26.965 INFO:teuthology.misc.health.smithi088.stderr:2024-06-03T08:36:26.964+0000 7f56d532f640 1 monclient: found mon.noname-b 2024-06-03T08:36:26.966 INFO:teuthology.misc.health.smithi088.stderr:2024-06-03T08:36:26.964+0000 7f56d532f640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-06-03T08:36:26.966 INFO:teuthology.misc.health.smithi088.stderr:2024-06-03T08:36:26.964+0000 7f56d532f640 10 monclient: _send_mon_message to mon.noname-b at v1:172.21.15.88:6790/0 2024-06-03T08:36:26.966 INFO:teuthology.misc.health.smithi088.stderr:2024-06-03T08:36:26.964+0000 7f56d532f640 1 -- 172.21.15.88:0/4084105133 --> v1:172.21.15.88:6790/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f56d00ae8c0 con 0x7f56d013d940 2024-06-03T08:36:26.966 INFO:teuthology.misc.health.smithi088.stderr:2024-06-03T08:36:26.964+0000 7f56d532f640 10 monclient: _finish_auth 0 2024-06-03T08:36:26.966 INFO:teuthology.misc.health.smithi088.stderr:2024-06-03T08:36:26.964+0000 7f56d532f640 10 monclient: _check_auth_tickets 2024-06-03T08:36:26.966 INFO:teuthology.misc.health.smithi088.stderr:2024-06-03T08:36:26.964+0000 7f56d532f640 20 monclient: _check_auth_rotating not needed by client.admin 2024-06-03T08:36:26.966 INFO:teuthology.misc.health.smithi088.stderr:2024-06-03T08:36:26.964+0000 7f56d532f640 1 -- 172.21.15.88:0/4084105133 <== mon.2 v1:172.21.15.88:6790/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 2234374065 0 0) 0x7f56c8001970 con 0x7f56d013d940 2024-06-03T08:36:26.970 INFO:teuthology.misc.health.smithi088.stderr:2024-06-03T08:36:26.964+0000 7f56d8595640 5 monclient: authenticate success, global_id 4277 2024-06-03T08:36:26.970 INFO:teuthology.misc.health.smithi088.stderr:2024-06-03T08:36:26.964+0000 7f56d8595640 20 monclient: get_monmap_and_config waiting for monmap|config 2024-06-03T08:36:26.970 INFO:teuthology.misc.health.smithi088.stderr:2024-06-03T08:36:26.964+0000 7f56d532f640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-06-03T08:36:26.970 INFO:teuthology.misc.health.smithi088.stderr:2024-06-03T08:36:26.965+0000 7f56d532f640 10 monclient: got monmap 1 from mon.noname-b (according to old e1) 2024-06-03T08:36:26.970 INFO:teuthology.misc.health.smithi088.stderr:2024-06-03T08:36:26.965+0000 7f56d532f640 10 monclient: dump: 2024-06-03T08:36:26.970 INFO:teuthology.misc.health.smithi088.stderr:epoch 1 2024-06-03T08:36:26.970 INFO:teuthology.misc.health.smithi088.stderr:fsid ec3c3fdd-3ac6-4d7d-9412-683130ed8b04 2024-06-03T08:36:26.970 INFO:teuthology.misc.health.smithi088.stderr:last_changed 2024-06-03T08:34:31.885125+0000 2024-06-03T08:36:26.970 INFO:teuthology.misc.health.smithi088.stderr:created 2024-06-03T08:34:31.885125+0000 2024-06-03T08:36:26.970 INFO:teuthology.misc.health.smithi088.stderr:min_mon_release 18 (reef) 2024-06-03T08:36:26.970 INFO:teuthology.misc.health.smithi088.stderr:election_strategy: 3 2024-06-03T08:36:26.970 INFO:teuthology.misc.health.smithi088.stderr:0: v1:172.21.15.88:6789/0 mon.a 2024-06-03T08:36:26.971 INFO:teuthology.misc.health.smithi088.stderr:1: v1:172.21.15.192:6789/0 mon.b 2024-06-03T08:36:26.971 INFO:teuthology.misc.health.smithi088.stderr:2: v1:172.21.15.88:6790/0 mon.c 2024-06-03T08:36:26.971 INFO:teuthology.misc.health.smithi088.stderr: 2024-06-03T08:36:26.971 INFO:teuthology.misc.health.smithi088.stderr:2024-06-03T08:36:26.965+0000 7f56d8595640 20 monclient: get_monmap_and_config waiting for monmap|config 2024-06-03T08:36:26.974 INFO:teuthology.misc.health.smithi088.stderr:2024-06-03T08:36:26.967+0000 7f56d532f640 1 -- 172.21.15.88:0/4084105133 <== mon.2 v1:172.21.15.88:6790/0 4 ==== config(0 keys) v1 ==== 4+0+0 (unknown 0 0 0) 0x7f56c8001b90 con 0x7f56d013d940 2024-06-03T08:36:26.975 INFO:teuthology.misc.health.smithi088.stderr:2024-06-03T08:36:26.967+0000 7f56d532f640 10 monclient: handle_config config(0 keys) v1 2024-06-03T08:36:26.975 INFO:teuthology.misc.health.smithi088.stderr:2024-06-03T08:36:26.967+0000 7f56d532f640 1 -- 172.21.15.88:0/4084105133 <== mon.2 v1:172.21.15.88:6790/0 5 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 2234374065 0 0) 0x7f56c8001ee0 con 0x7f56d013d940 2024-06-03T08:36:26.975 INFO:teuthology.misc.health.smithi088.stderr:2024-06-03T08:36:26.967+0000 7f56d8595640 10 monclient: get_monmap_and_config success 2024-06-03T08:36:26.975 INFO:teuthology.misc.health.smithi088.stderr:2024-06-03T08:36:26.967+0000 7f56d8595640 4 set_mon_vals no callback set 2024-06-03T08:36:26.975 INFO:teuthology.misc.health.smithi088.stderr:2024-06-03T08:36:26.967+0000 7f56d532f640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-06-03T08:36:26.975 INFO:teuthology.misc.health.smithi088.stderr:2024-06-03T08:36:26.967+0000 7f56d532f640 10 monclient: got monmap 1 from mon.c (according to old e1) 2024-06-03T08:36:26.975 INFO:teuthology.misc.health.smithi088.stderr:2024-06-03T08:36:26.967+0000 7f56d532f640 10 monclient: dump: 2024-06-03T08:36:26.975 INFO:teuthology.misc.health.smithi088.stderr:epoch 1 2024-06-03T08:36:26.975 INFO:teuthology.misc.health.smithi088.stderr:fsid ec3c3fdd-3ac6-4d7d-9412-683130ed8b04 2024-06-03T08:36:26.975 INFO:teuthology.misc.health.smithi088.stderr:last_changed 2024-06-03T08:34:31.885125+0000 2024-06-03T08:36:26.975 INFO:teuthology.misc.health.smithi088.stderr:created 2024-06-03T08:34:31.885125+0000 2024-06-03T08:36:26.975 INFO:teuthology.misc.health.smithi088.stderr:min_mon_release 18 (reef) 2024-06-03T08:36:26.975 INFO:teuthology.misc.health.smithi088.stderr:election_strategy: 3 2024-06-03T08:36:26.976 INFO:teuthology.misc.health.smithi088.stderr:0: v1:172.21.15.88:6789/0 mon.a 2024-06-03T08:36:26.976 INFO:teuthology.misc.health.smithi088.stderr:1: v1:172.21.15.192:6789/0 mon.b 2024-06-03T08:36:26.976 INFO:teuthology.misc.health.smithi088.stderr:2: v1:172.21.15.88:6790/0 mon.c 2024-06-03T08:36:26.976 INFO:teuthology.misc.health.smithi088.stderr: 2024-06-03T08:36:26.976 INFO:teuthology.misc.health.smithi088.stderr:2024-06-03T08:36:26.968+0000 7f56d8595640 10 monclient: shutdown 2024-06-03T08:36:26.976 INFO:teuthology.misc.health.smithi088.stderr:2024-06-03T08:36:26.968+0000 7f56d8595640 20 monclient: shutdown discarding 0 pending message(s) 2024-06-03T08:36:26.976 INFO:teuthology.misc.health.smithi088.stderr:2024-06-03T08:36:26.968+0000 7f56d8595640 1 -- 172.21.15.88:0/4084105133 >> v1:172.21.15.88:6790/0 conn(0x7f56d013d940 legacy=0x7f56d0149260 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-06-03T08:36:26.976 INFO:teuthology.misc.health.smithi088.stderr:2024-06-03T08:36:26.968+0000 7f56d8595640 1 -- 172.21.15.88:0/4084105133 shutdown_connections 2024-06-03T08:36:26.976 INFO:teuthology.misc.health.smithi088.stderr:2024-06-03T08:36:26.968+0000 7f56d8595640 1 -- 172.21.15.88:0/4084105133 >> 172.21.15.88:0/4084105133 conn(0x7f56d00b2f20 msgr2=0x7f56d00b3300 unknown :-1 s=STATE_NONE l=0).mark_down 2024-06-03T08:36:26.976 INFO:teuthology.misc.health.smithi088.stderr:2024-06-03T08:36:26.968+0000 7f56d8595640 1 -- 172.21.15.88:0/4084105133 shutdown_connections 2024-06-03T08:36:26.976 INFO:teuthology.misc.health.smithi088.stderr:2024-06-03T08:36:26.968+0000 7f56d8595640 1 -- 172.21.15.88:0/4084105133 wait complete. 2024-06-03T08:36:26.976 INFO:teuthology.misc.health.smithi088.stderr:2024-06-03T08:36:26.969+0000 7f56d8595640 10 monclient: build_initial_monmap 2024-06-03T08:36:26.977 INFO:teuthology.misc.health.smithi088.stderr:2024-06-03T08:36:26.969+0000 7f56d8595640 10 monclient: monmap: 2024-06-03T08:36:26.977 INFO:teuthology.misc.health.smithi088.stderr:epoch 0 2024-06-03T08:36:26.977 INFO:teuthology.misc.health.smithi088.stderr:fsid 00000000-0000-0000-0000-000000000000 2024-06-03T08:36:26.977 INFO:teuthology.misc.health.smithi088.stderr:last_changed 0.000000 2024-06-03T08:36:26.977 INFO:teuthology.misc.health.smithi088.stderr:created 0.000000 2024-06-03T08:36:26.977 INFO:teuthology.misc.health.smithi088.stderr:min_mon_release 0 (unknown) 2024-06-03T08:36:26.977 INFO:teuthology.misc.health.smithi088.stderr:election_strategy: 1 2024-06-03T08:36:26.977 INFO:teuthology.misc.health.smithi088.stderr:0: v1:172.21.15.88:6789/0 mon.noname-a 2024-06-03T08:36:26.977 INFO:teuthology.misc.health.smithi088.stderr:1: v1:172.21.15.192:6789/0 mon.noname-b 2024-06-03T08:36:26.977 INFO:teuthology.misc.health.smithi088.stderr:2: v1:172.21.15.88:6790/0 mon.noname-c 2024-06-03T08:36:26.977 INFO:teuthology.misc.health.smithi088.stderr: 2024-06-03T08:36:26.977 INFO:teuthology.misc.health.smithi088.stderr:2024-06-03T08:36:26.969+0000 7f56d8595640 1 librados: starting msgr at 2024-06-03T08:36:26.977 INFO:teuthology.misc.health.smithi088.stderr:2024-06-03T08:36:26.969+0000 7f56d8595640 1 librados: starting objecter 2024-06-03T08:36:26.978 INFO:teuthology.misc.health.smithi088.stderr:2024-06-03T08:36:26.969+0000 7f56d8595640 1 Processor -- start 2024-06-03T08:36:26.978 INFO:teuthology.misc.health.smithi088.stderr:2024-06-03T08:36:26.969+0000 7f56d8595640 1 -- start start 2024-06-03T08:36:26.978 INFO:teuthology.misc.health.smithi088.stderr:2024-06-03T08:36:26.969+0000 7f56d8595640 1 librados: setting wanted keys 2024-06-03T08:36:26.978 INFO:teuthology.misc.health.smithi088.stderr:2024-06-03T08:36:26.969+0000 7f56d8595640 1 librados: calling monclient init 2024-06-03T08:36:26.978 INFO:teuthology.misc.health.smithi088.stderr:2024-06-03T08:36:26.969+0000 7f56d8595640 10 monclient: init 2024-06-03T08:36:26.978 INFO:teuthology.misc.health.smithi088.stderr:2024-06-03T08:36:26.969+0000 7f56d8595640 10 monclient: _reopen_session rank -1 2024-06-03T08:36:26.978 INFO:teuthology.misc.health.smithi088.stderr:2024-06-03T08:36:26.970+0000 7f56d8595640 10 monclient: _add_conns ranks=[2,1,0] 2024-06-03T08:36:26.978 INFO:teuthology.misc.health.smithi088.stderr:2024-06-03T08:36:26.970+0000 7f56d8595640 10 monclient(hunting): picked mon.noname-c con 0x7f56d0070db0 addr v1:172.21.15.88:6790/0 2024-06-03T08:36:26.978 INFO:teuthology.misc.health.smithi088.stderr:2024-06-03T08:36:26.970+0000 7f56d8595640 10 monclient(hunting): picked mon.noname-b con 0x7f56d013d940 addr v1:172.21.15.192:6789/0 2024-06-03T08:36:26.978 INFO:teuthology.misc.health.smithi088.stderr:2024-06-03T08:36:26.970+0000 7f56d8595640 10 monclient(hunting): picked mon.noname-a con 0x7f56d01c5540 addr v1:172.21.15.88:6789/0 2024-06-03T08:36:26.978 INFO:teuthology.misc.health.smithi088.stderr:2024-06-03T08:36:26.970+0000 7f56d8595640 1 -- --> v1:172.21.15.88:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f56d0139820 con 0x7f56d01c5540 2024-06-03T08:36:26.978 INFO:teuthology.misc.health.smithi088.stderr:2024-06-03T08:36:26.970+0000 7f56d8595640 1 -- --> v1:172.21.15.192:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f56d0136d50 con 0x7f56d013d940 2024-06-03T08:36:26.979 INFO:teuthology.misc.health.smithi088.stderr:2024-06-03T08:36:26.970+0000 7f56d8595640 1 -- --> v1:172.21.15.88:6790/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f56d0136ee0 con 0x7f56d0070db0 2024-06-03T08:36:26.979 INFO:teuthology.misc.health.smithi088.stderr:2024-06-03T08:36:26.970+0000 7f56d8595640 10 monclient(hunting): _renew_subs 2024-06-03T08:36:26.979 INFO:teuthology.misc.health.smithi088.stderr:2024-06-03T08:36:26.970+0000 7f56d8595640 10 monclient(hunting): authenticate will time out at 1242.780041s 2024-06-03T08:36:26.979 INFO:teuthology.misc.health.smithi088.stderr:2024-06-03T08:36:26.970+0000 7f56d6331640 1 --1- >> v1:172.21.15.88:6790/0 conn(0x7f56d0070db0 0x7f56d0139110 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=1).handle_server_banner_and_identify peer v1:172.21.15.88:6790/0 says I am v1:172.21.15.88:53990/0 (socket says 172.21.15.88:53990) 2024-06-03T08:36:26.979 INFO:teuthology.misc.health.smithi088.stderr:2024-06-03T08:36:26.970+0000 7f56d6331640 1 -- 172.21.15.88:0/1882124989 learned_addr learned my addr 172.21.15.88:0/1882124989 (peer_addr_for_me v1:172.21.15.88:0/0) 2024-06-03T08:36:26.979 INFO:teuthology.misc.health.smithi088.stderr:2024-06-03T08:36:26.970+0000 7f56d5b30640 1 --1- >> v1:172.21.15.192:6789/0 conn(0x7f56d013d940 0x7f56d00abe20 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=1).handle_server_banner_and_identify peer v1:172.21.15.192:6789/0 says I am v1:172.21.15.88:52204/0 (socket says 172.21.15.88:52204) 2024-06-03T08:36:26.979 INFO:teuthology.misc.health.smithi088.stderr:2024-06-03T08:36:26.970+0000 7f56c6ffd640 10 client.?.objecter ms_handle_connect 0x7f56d0070db0 2024-06-03T08:36:26.979 INFO:teuthology.misc.health.smithi088.stderr:2024-06-03T08:36:26.970+0000 7f56c6ffd640 10 client.?.objecter resend_mon_ops 2024-06-03T08:36:26.979 INFO:teuthology.misc.health.smithi088.stderr:2024-06-03T08:36:26.970+0000 7f56c6ffd640 10 client.?.objecter ms_handle_connect 0x7f56d01c5540 2024-06-03T08:36:26.979 INFO:teuthology.misc.health.smithi088.stderr:2024-06-03T08:36:26.970+0000 7f56c6ffd640 10 client.?.objecter resend_mon_ops 2024-06-03T08:36:26.979 INFO:teuthology.misc.health.smithi088.stderr:2024-06-03T08:36:26.970+0000 7f56c6ffd640 10 client.?.objecter ms_handle_connect 0x7f56d013d940 2024-06-03T08:36:26.979 INFO:teuthology.misc.health.smithi088.stderr:2024-06-03T08:36:26.970+0000 7f56c6ffd640 10 client.?.objecter resend_mon_ops 2024-06-03T08:36:26.979 INFO:teuthology.misc.health.smithi088.stderr:2024-06-03T08:36:26.972+0000 7f56c6ffd640 1 -- 172.21.15.88:0/1882124989 <== mon.0 v1:172.21.15.88:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 1906934538 0 0) 0x7f56d0139820 con 0x7f56d01c5540 2024-06-03T08:36:26.979 INFO:teuthology.misc.health.smithi088.stderr:2024-06-03T08:36:26.972+0000 7f56c6ffd640 10 monclient(hunting): _negotiate 2024-06-03T08:36:26.980 INFO:teuthology.misc.health.smithi088.stderr:2024-06-03T08:36:26.972+0000 7f56c6ffd640 10 monclient(hunting): _init_auth method 2 2024-06-03T08:36:26.980 INFO:teuthology.misc.health.smithi088.stderr:2024-06-03T08:36:26.972+0000 7f56c6ffd640 10 monclient(hunting): _init_auth creating new auth 2024-06-03T08:36:26.980 INFO:teuthology.misc.health.smithi088.stderr:2024-06-03T08:36:26.972+0000 7f56c6ffd640 10 monclient(hunting): my global_id is 4266 2024-06-03T08:36:26.980 INFO:teuthology.misc.health.smithi088.stderr:2024-06-03T08:36:26.972+0000 7f56c6ffd640 1 -- 172.21.15.88:0/1882124989 --> v1:172.21.15.88:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f56b4004eb0 con 0x7f56d01c5540 2024-06-03T08:36:26.980 INFO:teuthology.misc.health.smithi088.stderr:2024-06-03T08:36:26.972+0000 7f56c6ffd640 1 -- 172.21.15.88:0/1882124989 <== mon.2 v1:172.21.15.88:6790/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 1541296130 0 0) 0x7f56d0136ee0 con 0x7f56d0070db0 2024-06-03T08:36:26.980 INFO:teuthology.misc.health.smithi088.stderr:2024-06-03T08:36:26.972+0000 7f56c6ffd640 10 monclient(hunting): _negotiate 2024-06-03T08:36:26.980 INFO:teuthology.misc.health.smithi088.stderr:2024-06-03T08:36:26.972+0000 7f56c6ffd640 10 monclient(hunting): _init_auth method 2 2024-06-03T08:36:26.980 INFO:teuthology.misc.health.smithi088.stderr:2024-06-03T08:36:26.972+0000 7f56c6ffd640 10 monclient(hunting): _init_auth creating new auth 2024-06-03T08:36:26.980 INFO:teuthology.misc.health.smithi088.stderr:2024-06-03T08:36:26.972+0000 7f56c6ffd640 10 monclient(hunting): my global_id is 4280 2024-06-03T08:36:26.980 INFO:teuthology.misc.health.smithi088.stderr:2024-06-03T08:36:26.972+0000 7f56c6ffd640 1 -- 172.21.15.88:0/1882124989 --> v1:172.21.15.88:6790/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f56d0139820 con 0x7f56d0070db0 2024-06-03T08:36:26.980 INFO:teuthology.misc.health.smithi088.stderr:2024-06-03T08:36:26.972+0000 7f56c6ffd640 1 -- 172.21.15.88:0/1882124989 <== mon.1 v1:172.21.15.192:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 1606572926 0 0) 0x7f56d0136d50 con 0x7f56d013d940 2024-06-03T08:36:26.980 INFO:teuthology.misc.health.smithi088.stderr:2024-06-03T08:36:26.972+0000 7f56c6ffd640 10 monclient(hunting): _negotiate 2024-06-03T08:36:26.980 INFO:teuthology.misc.health.smithi088.stderr:2024-06-03T08:36:26.972+0000 7f56c6ffd640 10 monclient(hunting): _init_auth method 2 2024-06-03T08:36:26.981 INFO:teuthology.misc.health.smithi088.stderr:2024-06-03T08:36:26.972+0000 7f56c6ffd640 10 monclient(hunting): _init_auth creating new auth 2024-06-03T08:36:26.981 INFO:teuthology.misc.health.smithi088.stderr:2024-06-03T08:36:26.972+0000 7f56c6ffd640 10 monclient(hunting): my global_id is 4261 2024-06-03T08:36:26.981 INFO:teuthology.misc.health.smithi088.stderr:2024-06-03T08:36:26.972+0000 7f56c6ffd640 1 -- 172.21.15.88:0/1882124989 --> v1:172.21.15.192:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f56d0136ee0 con 0x7f56d013d940 2024-06-03T08:36:26.983 INFO:teuthology.misc.health.smithi088.stderr:2024-06-03T08:36:26.977+0000 7f56c6ffd640 1 -- 172.21.15.88:0/1882124989 <== mon.1 v1:172.21.15.192:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 764+0+0 (unknown 1529067686 0 0) 0x7f56d0136ee0 con 0x7f56d013d940 2024-06-03T08:36:26.983 INFO:teuthology.misc.health.smithi088.stderr:2024-06-03T08:36:26.977+0000 7f56c6ffd640 1 -- 172.21.15.88:0/1882124989 --> v1:172.21.15.192:6789/0 -- auth(proto 2 165 bytes epoch 0) v1 -- 0x7f56d0136d50 con 0x7f56d013d940 2024-06-03T08:36:26.983 INFO:teuthology.misc.health.smithi088.stderr:2024-06-03T08:36:26.977+0000 7f56c6ffd640 1 -- 172.21.15.88:0/1882124989 <== mon.1 v1:172.21.15.192:6789/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 2234374065 0 0) 0x7f56c00018b0 con 0x7f56d013d940 2024-06-03T08:36:26.983 INFO:teuthology.misc.health.smithi088.stderr:2024-06-03T08:36:26.977+0000 7f56c6ffd640 10 monclient(hunting): handle_monmap mon_map magic: 0 v1 2024-06-03T08:36:26.983 INFO:teuthology.misc.health.smithi088.stderr:2024-06-03T08:36:26.977+0000 7f56c6ffd640 10 monclient(hunting): got monmap 1 from mon.noname-b (according to old e1) 2024-06-03T08:36:26.983 INFO:teuthology.misc.health.smithi088.stderr:2024-06-03T08:36:26.977+0000 7f56c6ffd640 10 monclient(hunting): dump: 2024-06-03T08:36:26.983 INFO:teuthology.misc.health.smithi088.stderr:epoch 1 2024-06-03T08:36:26.983 INFO:teuthology.misc.health.smithi088.stderr:fsid ec3c3fdd-3ac6-4d7d-9412-683130ed8b04 2024-06-03T08:36:26.984 INFO:teuthology.misc.health.smithi088.stderr:last_changed 2024-06-03T08:34:31.885125+0000 2024-06-03T08:36:26.984 INFO:teuthology.misc.health.smithi088.stderr:created 2024-06-03T08:34:31.885125+0000 2024-06-03T08:36:26.984 INFO:teuthology.misc.health.smithi088.stderr:min_mon_release 18 (reef) 2024-06-03T08:36:26.984 INFO:teuthology.misc.health.smithi088.stderr:election_strategy: 3 2024-06-03T08:36:26.984 INFO:teuthology.misc.health.smithi088.stderr:0: v1:172.21.15.88:6789/0 mon.a 2024-06-03T08:36:26.984 INFO:teuthology.misc.health.smithi088.stderr:1: v1:172.21.15.192:6789/0 mon.b 2024-06-03T08:36:26.984 INFO:teuthology.misc.health.smithi088.stderr:2: v1:172.21.15.88:6790/0 mon.c 2024-06-03T08:36:26.984 INFO:teuthology.misc.health.smithi088.stderr: 2024-06-03T08:36:26.984 INFO:teuthology.misc.health.smithi088.stderr:2024-06-03T08:36:26.977+0000 7f56c6ffd640 10 monclient(hunting): _finish_auth 0 2024-06-03T08:36:26.984 INFO:teuthology.misc.health.smithi088.stderr:2024-06-03T08:36:26.978+0000 7f56c6ffd640 1 -- 172.21.15.88:0/1882124989 <== mon.2 v1:172.21.15.88:6790/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 764+0+0 (unknown 2010511847 0 0) 0x7f56d0139820 con 0x7f56d0070db0 2024-06-03T08:36:26.984 INFO:teuthology.misc.health.smithi088.stderr:2024-06-03T08:36:26.978+0000 7f56c6ffd640 1 -- 172.21.15.88:0/1882124989 --> v1:172.21.15.88:6790/0 -- auth(proto 2 165 bytes epoch 0) v1 -- 0x7f56d0136ee0 con 0x7f56d0070db0 2024-06-03T08:36:26.984 INFO:teuthology.misc.health.smithi088.stderr:2024-06-03T08:36:26.978+0000 7f56c6ffd640 1 -- 172.21.15.88:0/1882124989 <== mon.2 v1:172.21.15.88:6790/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 2234374065 0 0) 0x7f56cc0018d0 con 0x7f56d0070db0 2024-06-03T08:36:26.984 INFO:teuthology.misc.health.smithi088.stderr:2024-06-03T08:36:26.978+0000 7f56c6ffd640 10 monclient(hunting): handle_monmap mon_map magic: 0 v1 2024-06-03T08:36:26.984 INFO:teuthology.misc.health.smithi088.stderr:2024-06-03T08:36:26.978+0000 7f56c6ffd640 10 monclient(hunting): got monmap 1 from mon.c (according to old e1) 2024-06-03T08:36:26.985 INFO:teuthology.misc.health.smithi088.stderr:2024-06-03T08:36:26.978+0000 7f56c6ffd640 10 monclient(hunting): dump: 2024-06-03T08:36:26.985 INFO:teuthology.misc.health.smithi088.stderr:epoch 1 2024-06-03T08:36:26.985 INFO:teuthology.misc.health.smithi088.stderr:fsid ec3c3fdd-3ac6-4d7d-9412-683130ed8b04 2024-06-03T08:36:26.985 INFO:teuthology.misc.health.smithi088.stderr:last_changed 2024-06-03T08:34:31.885125+0000 2024-06-03T08:36:26.985 INFO:teuthology.misc.health.smithi088.stderr:created 2024-06-03T08:34:31.885125+0000 2024-06-03T08:36:26.985 INFO:teuthology.misc.health.smithi088.stderr:min_mon_release 18 (reef) 2024-06-03T08:36:26.985 INFO:teuthology.misc.health.smithi088.stderr:election_strategy: 3 2024-06-03T08:36:26.985 INFO:teuthology.misc.health.smithi088.stderr:0: v1:172.21.15.88:6789/0 mon.a 2024-06-03T08:36:26.985 INFO:teuthology.misc.health.smithi088.stderr:1: v1:172.21.15.192:6789/0 mon.b 2024-06-03T08:36:26.985 INFO:teuthology.misc.health.smithi088.stderr:2: v1:172.21.15.88:6790/0 mon.c 2024-06-03T08:36:26.985 INFO:teuthology.misc.health.smithi088.stderr: 2024-06-03T08:36:26.985 INFO:teuthology.misc.health.smithi088.stderr:2024-06-03T08:36:26.978+0000 7f56c6ffd640 1 -- 172.21.15.88:0/1882124989 <== mon.0 v1:172.21.15.88:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 764+0+0 (unknown 2558115105 0 0) 0x7f56b4004eb0 con 0x7f56d01c5540 2024-06-03T08:36:26.985 INFO:teuthology.misc.health.smithi088.stderr:2024-06-03T08:36:26.978+0000 7f56c6ffd640 1 -- 172.21.15.88:0/1882124989 --> v1:172.21.15.88:6789/0 -- auth(proto 2 165 bytes epoch 0) v1 -- 0x7f56d0139820 con 0x7f56d01c5540 2024-06-03T08:36:26.985 INFO:teuthology.misc.health.smithi088.stderr:2024-06-03T08:36:26.978+0000 7f56c6ffd640 1 -- 172.21.15.88:0/1882124989 <== mon.0 v1:172.21.15.88:6789/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 2234374065 0 0) 0x7f56c8001ea0 con 0x7f56d01c5540 2024-06-03T08:36:26.986 INFO:teuthology.misc.health.smithi088.stderr:2024-06-03T08:36:26.978+0000 7f56c6ffd640 10 monclient(hunting): handle_monmap mon_map magic: 0 v1 2024-06-03T08:36:26.986 INFO:teuthology.misc.health.smithi088.stderr:2024-06-03T08:36:26.978+0000 7f56c6ffd640 10 monclient(hunting): got monmap 1 from mon.a (according to old e1) 2024-06-03T08:36:26.986 INFO:teuthology.misc.health.smithi088.stderr:2024-06-03T08:36:26.978+0000 7f56c6ffd640 10 monclient(hunting): dump: 2024-06-03T08:36:26.986 INFO:teuthology.misc.health.smithi088.stderr:epoch 1 2024-06-03T08:36:26.986 INFO:teuthology.misc.health.smithi088.stderr:fsid ec3c3fdd-3ac6-4d7d-9412-683130ed8b04 2024-06-03T08:36:26.986 INFO:teuthology.misc.health.smithi088.stderr:last_changed 2024-06-03T08:34:31.885125+0000 2024-06-03T08:36:26.986 INFO:teuthology.misc.health.smithi088.stderr:created 2024-06-03T08:34:31.885125+0000 2024-06-03T08:36:26.986 INFO:teuthology.misc.health.smithi088.stderr:min_mon_release 18 (reef) 2024-06-03T08:36:26.986 INFO:teuthology.misc.health.smithi088.stderr:election_strategy: 3 2024-06-03T08:36:26.986 INFO:teuthology.misc.health.smithi088.stderr:0: v1:172.21.15.88:6789/0 mon.a 2024-06-03T08:36:26.986 INFO:teuthology.misc.health.smithi088.stderr:1: v1:172.21.15.192:6789/0 mon.b 2024-06-03T08:36:26.986 INFO:teuthology.misc.health.smithi088.stderr:2: v1:172.21.15.88:6790/0 mon.c 2024-06-03T08:36:26.986 INFO:teuthology.misc.health.smithi088.stderr: 2024-06-03T08:36:26.987 INFO:teuthology.misc.health.smithi088.stderr:2024-06-03T08:36:26.981+0000 7f56c6ffd640 1 -- 172.21.15.88:0/1882124989 <== mon.1 v1:172.21.15.192:6789/0 4 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 580+0+0 (unknown 700672032 0 0) 0x7f56d0136d50 con 0x7f56d013d940 2024-06-03T08:36:26.987 INFO:teuthology.misc.health.smithi088.stderr:2024-06-03T08:36:26.981+0000 7f56c6ffd640 1 -- 172.21.15.88:0/1882124989 >> v1:172.21.15.88:6790/0 conn(0x7f56d0070db0 legacy=0x7f56d0139110 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-06-03T08:36:26.987 INFO:teuthology.misc.health.smithi088.stderr:2024-06-03T08:36:26.981+0000 7f56c6ffd640 1 -- 172.21.15.88:0/1882124989 >> v1:172.21.15.88:6789/0 conn(0x7f56d01c5540 legacy=0x7f56d00ac530 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-06-03T08:36:26.987 INFO:teuthology.misc.health.smithi088.stderr:2024-06-03T08:36:26.981+0000 7f56c6ffd640 10 monclient: _finish_hunting 0 2024-06-03T08:36:26.987 INFO:teuthology.misc.health.smithi088.stderr:2024-06-03T08:36:26.981+0000 7f56c6ffd640 1 monclient: found mon.b 2024-06-03T08:36:26.987 INFO:teuthology.misc.health.smithi088.stderr:2024-06-03T08:36:26.981+0000 7f56c6ffd640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-06-03T08:36:26.987 INFO:teuthology.misc.health.smithi088.stderr:2024-06-03T08:36:26.981+0000 7f56c6ffd640 10 monclient: _send_mon_message to mon.b at v1:172.21.15.192:6789/0 2024-06-03T08:36:26.987 INFO:teuthology.misc.health.smithi088.stderr:2024-06-03T08:36:26.981+0000 7f56c6ffd640 1 -- 172.21.15.88:0/1882124989 --> v1:172.21.15.192:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f56d01d2480 con 0x7f56d013d940 2024-06-03T08:36:26.987 INFO:teuthology.misc.health.smithi088.stderr:2024-06-03T08:36:26.981+0000 7f56c6ffd640 10 monclient: _finish_auth 0 2024-06-03T08:36:26.987 INFO:teuthology.misc.health.smithi088.stderr:2024-06-03T08:36:26.981+0000 7f56c6ffd640 10 monclient: _check_auth_tickets 2024-06-03T08:36:26.987 INFO:teuthology.misc.health.smithi088.stderr:2024-06-03T08:36:26.981+0000 7f56c6ffd640 20 monclient: _check_auth_rotating not needed by client.admin 2024-06-03T08:36:26.987 INFO:teuthology.misc.health.smithi088.stderr:2024-06-03T08:36:26.981+0000 7f56d8595640 5 monclient: authenticate success, global_id 4261 2024-06-03T08:36:26.988 INFO:teuthology.misc.health.smithi088.stderr:2024-06-03T08:36:26.981+0000 7f56d8595640 10 monclient: _renew_subs 2024-06-03T08:36:26.988 INFO:teuthology.misc.health.smithi088.stderr:2024-06-03T08:36:26.981+0000 7f56d8595640 10 monclient: _send_mon_message to mon.b at v1:172.21.15.192:6789/0 2024-06-03T08:36:26.988 INFO:teuthology.misc.health.smithi088.stderr:2024-06-03T08:36:26.981+0000 7f56d8595640 1 -- 172.21.15.88:0/1882124989 --> v1:172.21.15.192:6789/0 -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f56d01d2730 con 0x7f56d013d940 2024-06-03T08:36:26.988 INFO:teuthology.misc.health.smithi088.stderr:2024-06-03T08:36:26.981+0000 7f56d8595640 10 client.4261.objecter _maybe_request_map subscribing (onetime) to next osd map 2024-06-03T08:36:26.988 INFO:teuthology.misc.health.smithi088.stderr:2024-06-03T08:36:26.981+0000 7f56d8595640 10 monclient: _renew_subs 2024-06-03T08:36:26.988 INFO:teuthology.misc.health.smithi088.stderr:2024-06-03T08:36:26.981+0000 7f56d8595640 10 monclient: _send_mon_message to mon.b at v1:172.21.15.192:6789/0 2024-06-03T08:36:26.988 INFO:teuthology.misc.health.smithi088.stderr:2024-06-03T08:36:26.981+0000 7f56d8595640 1 -- 172.21.15.88:0/1882124989 --> v1:172.21.15.192:6789/0 -- mon_subscribe({osdmap=0}) v3 -- 0x7f56d00ace20 con 0x7f56d013d940 2024-06-03T08:36:26.988 INFO:teuthology.misc.health.smithi088.stderr:2024-06-03T08:36:26.981+0000 7f56d8595640 1 librados: init done 2024-06-03T08:36:26.988 INFO:teuthology.misc.health.smithi088.stderr:2024-06-03T08:36:26.982+0000 7f56d8595640 10 start_mon_command cmd=[{"prefix": "get_command_descriptions"}] 2024-06-03T08:36:26.988 INFO:teuthology.misc.health.smithi088.stderr:2024-06-03T08:36:26.982+0000 7f56d8595640 10 monclient: _send_command 1 [{"prefix": "get_command_descriptions"}] 2024-06-03T08:36:26.988 INFO:teuthology.misc.health.smithi088.stderr:2024-06-03T08:36:26.982+0000 7f56d8595640 10 monclient: _send_mon_message to mon.b at v1:172.21.15.192:6789/0 2024-06-03T08:36:26.988 INFO:teuthology.misc.health.smithi088.stderr:2024-06-03T08:36:26.982+0000 7f56d8595640 1 -- 172.21.15.88:0/1882124989 --> v1:172.21.15.192:6789/0 -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f56d004e730 con 0x7f56d013d940 2024-06-03T08:36:26.997 INFO:teuthology.misc.health.smithi088.stderr:2024-06-03T08:36:26.997+0000 7f56c6ffd640 1 -- 172.21.15.88:0/1882124989 <== mon.1 v1:172.21.15.192:6789/0 5 ==== config(0 keys) v1 ==== 4+0+0 (unknown 0 0 0) 0x7f56c0003610 con 0x7f56d013d940 2024-06-03T08:36:26.997 INFO:teuthology.misc.health.smithi088.stderr:2024-06-03T08:36:26.997+0000 7f56c6ffd640 10 monclient: handle_config config(0 keys) v1 2024-06-03T08:36:26.997 INFO:teuthology.misc.health.smithi088.stderr:2024-06-03T08:36:26.997+0000 7f56c6ffd640 1 -- 172.21.15.88:0/1882124989 <== mon.1 v1:172.21.15.192:6789/0 6 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 2234374065 0 0) 0x7f56c00039a0 con 0x7f56d013d940 2024-06-03T08:36:26.997 INFO:teuthology.misc.health.smithi088.stderr:2024-06-03T08:36:26.997+0000 7f56c6ffd640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-06-03T08:36:26.997 INFO:teuthology.misc.health.smithi088.stderr:2024-06-03T08:36:26.997+0000 7f56c6ffd640 10 monclient: got monmap 1 from mon.b (according to old e1) 2024-06-03T08:36:26.997 INFO:teuthology.misc.health.smithi088.stderr:2024-06-03T08:36:26.997+0000 7f56c6ffd640 10 monclient: dump: 2024-06-03T08:36:26.997 INFO:teuthology.misc.health.smithi088.stderr:epoch 1 2024-06-03T08:36:26.998 INFO:teuthology.misc.health.smithi088.stderr:fsid ec3c3fdd-3ac6-4d7d-9412-683130ed8b04 2024-06-03T08:36:26.998 INFO:teuthology.misc.health.smithi088.stderr:last_changed 2024-06-03T08:34:31.885125+0000 2024-06-03T08:36:26.998 INFO:teuthology.misc.health.smithi088.stderr:created 2024-06-03T08:34:31.885125+0000 2024-06-03T08:36:26.998 INFO:teuthology.misc.health.smithi088.stderr:min_mon_release 18 (reef) 2024-06-03T08:36:26.998 INFO:teuthology.misc.health.smithi088.stderr:election_strategy: 3 2024-06-03T08:36:26.998 INFO:teuthology.misc.health.smithi088.stderr:0: v1:172.21.15.88:6789/0 mon.a 2024-06-03T08:36:26.998 INFO:teuthology.misc.health.smithi088.stderr:1: v1:172.21.15.192:6789/0 mon.b 2024-06-03T08:36:26.998 INFO:teuthology.misc.health.smithi088.stderr:2: v1:172.21.15.88:6790/0 mon.c 2024-06-03T08:36:26.998 INFO:teuthology.misc.health.smithi088.stderr: 2024-06-03T08:36:26.998 INFO:teuthology.misc.health.smithi088.stderr:2024-06-03T08:36:26.997+0000 7f56d4b2e640 4 set_mon_vals no callback set 2024-06-03T08:36:26.999 INFO:teuthology.misc.health.smithi088.stderr:2024-06-03T08:36:26.997+0000 7f56c6ffd640 1 -- 172.21.15.88:0/1882124989 <== mon.1 v1:172.21.15.192:6789/0 7 ==== mgrmap(e 5) v1 ==== 89110+0+0 (unknown 873533973 0 0) 0x7f56c0003b00 con 0x7f56d013d940 2024-06-03T08:36:26.999 INFO:teuthology.misc.health.smithi088.stderr:2024-06-03T08:36:26.998+0000 7f56d6331640 10 monclient: get_auth_request con 0x7f56b406d390 auth_method 0 2024-06-03T08:36:26.999 INFO:teuthology.misc.health.smithi088.stderr:2024-06-03T08:36:26.998+0000 7f56c6ffd640 10 client.4261.objecter ms_handle_connect 0x7f56b406d390 2024-06-03T08:36:27.000 INFO:teuthology.misc.health.smithi088.stderr:2024-06-03T08:36:26.999+0000 7f56c6ffd640 1 -- 172.21.15.88:0/1882124989 <== mon.1 v1:172.21.15.192:6789/0 8 ==== osd_map(22..22 src has 1..22) v4 ==== 3737+0+0 (unknown 2958406083 0 0) 0x7f56c0054080 con 0x7f56d013d940 2024-06-03T08:36:27.000 INFO:teuthology.misc.health.smithi088.stderr:2024-06-03T08:36:26.999+0000 7f56c6ffd640 10 client.4261.objecter ms_dispatch 0x7f56d0000b90 osd_map(22..22 src has 1..22) v4 2024-06-03T08:36:27.000 INFO:teuthology.misc.health.smithi088.stderr:2024-06-03T08:36:26.999+0000 7f56c6ffd640 3 client.4261.objecter handle_osd_map got epochs [22,22] > 0 2024-06-03T08:36:27.000 INFO:teuthology.misc.health.smithi088.stderr:2024-06-03T08:36:26.999+0000 7f56c6ffd640 3 client.4261.objecter handle_osd_map decoding full epoch 22 2024-06-03T08:36:27.000 INFO:teuthology.misc.health.smithi088.stderr:2024-06-03T08:36:26.999+0000 7f56c6ffd640 20 client.4261.objecter dump_active .. 0 homeless 2024-06-03T08:36:27.240 INFO:teuthology.misc.health.smithi088.stderr:2024-06-03T08:36:27.240+0000 7f56c6ffd640 1 -- 172.21.15.88:0/1882124989 <== mon.1 v1:172.21.15.192:6789/0 9 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+182498 (unknown 1092875540 0 3016760700) 0x7f56c004fc30 con 0x7f56d013d940 2024-06-03T08:36:27.240 INFO:teuthology.misc.health.smithi088.stderr:2024-06-03T08:36:27.240+0000 7f56c6ffd640 10 monclient: handle_mon_command_ack 1 [{"prefix": "get_command_descriptions"}] 2024-06-03T08:36:27.240 INFO:teuthology.misc.health.smithi088.stderr:2024-06-03T08:36:27.240+0000 7f56c6ffd640 10 monclient: _finish_command 1 = system:0 2024-06-03T08:36:27.417 INFO:teuthology.misc.health.smithi088.stderr:2024-06-03T08:36:27.416+0000 7f56d8595640 10 start_mon_command cmd=[{"prefix": "osd dump", "format": "json"}] 2024-06-03T08:36:27.417 INFO:teuthology.misc.health.smithi088.stderr:2024-06-03T08:36:27.416+0000 7f56d8595640 10 monclient: _send_command 2 [{"prefix": "osd dump", "format": "json"}] 2024-06-03T08:36:27.417 INFO:teuthology.misc.health.smithi088.stderr:2024-06-03T08:36:27.416+0000 7f56d8595640 10 monclient: _send_mon_message to mon.b at v1:172.21.15.192:6789/0 2024-06-03T08:36:27.417 INFO:teuthology.misc.health.smithi088.stderr:2024-06-03T08:36:27.416+0000 7f56d8595640 1 -- 172.21.15.88:0/1882124989 --> v1:172.21.15.192:6789/0 -- mon_command({"prefix": "osd dump", "format": "json"} v 0) v1 -- 0x7f56d01d28c0 con 0x7f56d013d940 2024-06-03T08:36:27.577 INFO:teuthology.misc.health.smithi088.stderr:2024-06-03T08:36:27.576+0000 7f56c6ffd640 1 -- 172.21.15.88:0/1882124989 <== mon.1 v1:172.21.15.192:6789/0 10 ==== mon_command_ack([{"prefix": "osd dump", "format": "json"}]=0 v22) v1 ==== 74+0+10606 (unknown 861009196 0 1477752898) 0x7f56c0051b60 con 0x7f56d013d940 2024-06-03T08:36:27.577 INFO:teuthology.misc.health.smithi088.stderr:2024-06-03T08:36:27.576+0000 7f56c6ffd640 10 monclient: handle_mon_command_ack 2 [{"prefix": "osd dump", "format": "json"}] 2024-06-03T08:36:27.577 INFO:teuthology.misc.health.smithi088.stderr:2024-06-03T08:36:27.576+0000 7f56c6ffd640 10 monclient: _finish_command 2 = system:0 2024-06-03T08:36:27.577 INFO:teuthology.misc.health.smithi088.stdout: 2024-06-03T08:36:27.577 INFO:teuthology.misc.health.smithi088.stdout:{"epoch":22,"fsid":"ec3c3fdd-3ac6-4d7d-9412-683130ed8b04","created":"2024-06-03T08:35:26.799638+0000","modified":"2024-06-03T08:36:25.306788+0000","last_up_change":"2024-06-03T08:36:23.712633+0000","last_in_change":"2024-06-03T08:35:32.183068+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":5,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":1,"max_osd":8,"require_min_compat_client":"luminous","min_compat_client":"jewel","require_osd_release":"reef","allow_crimson":false,"pools":[{"pool":1,"pool_name":".mgr","create_time":"2024-06-03T08:36:15.192388+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":1,"pg_placement_num":1,"pg_placement_num_target":1,"pg_num_target":1,"pg_num_pending":1,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"19","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_max":32,"pg_num_min":1},"application_metadata":{"mgr":{}},"read_balance":{"score_acting":8,"score_stable":8,"optimal_score":0.25,"raw_score_acting":2,"raw_score_stable":2,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}}],"osds":[{"osd":0,"uuid":"b25ab3bb-3587-45d0-991a-6d41bb529b37","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":14,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.88:6808","nonce":35066}]},"cluster_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.88:6809","nonce":35066}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.88:6811","nonce":35066}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.88:6810","nonce":35066}]},"public_addr":"172.21.15.88:6808/35066","cluster_addr":"172.21.15.88:6809/35066","heartbeat_back_addr":"172.21.15.88:6811/35066","heartbeat_front_addr":"172.21.15.88:6810/35066","state":["exists","up"]},{"osd":1,"uuid":"00d160e7-9773-41d8-8d17-21e70620a0fd","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":14,"up_thru":20,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.88:6800","nonce":35067}]},"cluster_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.88:6801","nonce":35067}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.88:6804","nonce":35067}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.88:6803","nonce":35067}]},"public_addr":"172.21.15.88:6800/35067","cluster_addr":"172.21.15.88:6801/35067","heartbeat_back_addr":"172.21.15.88:6804/35067","heartbeat_front_addr":"172.21.15.88:6803/35067","state":["exists","up"]},{"osd":2,"uuid":"addfb2a7-2bde-4cc3-98cf-17fd2bdfa3a9","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":20,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.88:6802","nonce":35065}]},"cluster_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.88:6805","nonce":35065}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.88:6807","nonce":35065}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.88:6806","nonce":35065}]},"public_addr":"172.21.15.88:6802/35065","cluster_addr":"172.21.15.88:6805/35065","heartbeat_back_addr":"172.21.15.88:6807/35065","heartbeat_front_addr":"172.21.15.88:6806/35065","state":["exists","up"]},{"osd":3,"uuid":"6c5770b0-c93c-4101-8c53-bd761b8abcde","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":14,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.88:6812","nonce":35068}]},"cluster_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.88:6813","nonce":35068}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.88:6815","nonce":35068}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.88:6814","nonce":35068}]},"public_addr":"172.21.15.88:6812/35068","cluster_addr":"172.21.15.88:6813/35068","heartbeat_back_addr":"172.21.15.88:6815/35068","heartbeat_front_addr":"172.21.15.88:6814/35068","state":["exists","up"]},{"osd":4,"uuid":"f001b3cb-2081-428b-bac3-b3df3c19b978","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":20,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.192:6809","nonce":34785}]},"cluster_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.192:6810","nonce":34785}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.192:6812","nonce":34785}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.192:6811","nonce":34785}]},"public_addr":"172.21.15.192:6809/34785","cluster_addr":"172.21.15.192:6810/34785","heartbeat_back_addr":"172.21.15.192:6812/34785","heartbeat_front_addr":"172.21.15.192:6811/34785","state":["exists","up"]},{"osd":5,"uuid":"3cc263e9-e9bf-45e0-a3b7-83a76469eb70","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":20,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.192:6805","nonce":34786}]},"cluster_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.192:6806","nonce":34786}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.192:6808","nonce":34786}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.192:6807","nonce":34786}]},"public_addr":"172.21.15.192:6805/34786","cluster_addr":"172.21.15.192:6806/34786","heartbeat_back_addr":"172.21.15.192:6808/34786","heartbeat_front_addr":"172.21.15.192:6807/34786","state":["exists","up"]},{"osd":6,"uuid":"82c83316-809f-4901-a5ed-34b6a075b393","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":21,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.192:6813","nonce":34788}]},"cluster_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.192:6814","nonce":34788}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.192:6816","nonce":34788}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.192:6815","nonce":34788}]},"public_addr":"172.21.15.192:6813/34788","cluster_addr":"172.21.15.192:6814/34788","heartbeat_back_addr":"172.21.15.192:6816/34788","heartbeat_front_addr":"172.21.15.192:6815/34788","state":["exists","up"]},{"osd":7,"uuid":"aafe0716-1102-4eae-aac6-0f81bb154e7d","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":20,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.192:6801","nonce":34787}]},"cluster_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.192:6802","nonce":34787}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.192:6804","nonce":34787}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.192:6803","nonce":34787}]},"public_addr":"172.21.15.192:6801/34787","cluster_addr":"172.21.15.192:6802/34787","heartbeat_back_addr":"172.21.15.192:6804/34787","heartbeat_front_addr":"172.21.15.192:6803/34787","state":["exists","up"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2024-06-03T08:36:10.027533+0000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2024-06-03T08:36:09.853335+0000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2024-06-03T08:36:10.789609+0000","dead_epoch":0},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2024-06-03T08:36:09.824468+0000","dead_epoch":0},{"osd":4,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2024-06-03T08:36:10.117232+0000","dead_epoch":0},{"osd":5,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2024-06-03T08:36:10.469993+0000","dead_epoch":0},{"osd":6,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2024-06-03T08:36:10.789874+0000","dead_epoch":0},{"osd":7,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2024-06-03T08:36:09.541793+0000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[],"pg_upmap_primaries":[],"pg_temp":[],"primary_temp":[],"blocklist":{},"range_blocklist":{},"erasure_code_profiles":{"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","technique":"reed_sol_van"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2024-06-03T08:36:27.579 INFO:teuthology.misc.health.smithi088.stderr:2024-06-03T08:36:27.578+0000 7f56d8595640 10 librados: watch_flush enter 2024-06-03T08:36:27.579 INFO:teuthology.misc.health.smithi088.stderr:2024-06-03T08:36:27.578+0000 7f56d8595640 10 librados: watch_flush exit 2024-06-03T08:36:27.580 INFO:teuthology.misc.health.smithi088.stderr:2024-06-03T08:36:27.578+0000 7f56d8595640 20 client.4261.objecter shutdown clearing up homeless session... 2024-06-03T08:36:27.580 INFO:teuthology.misc.health.smithi088.stderr:2024-06-03T08:36:27.578+0000 7f56d8595640 10 client.4261.objecter successfully canceled tick 2024-06-03T08:36:27.580 INFO:teuthology.misc.health.smithi088.stderr:2024-06-03T08:36:27.578+0000 7f56d8595640 1 -- 172.21.15.88:0/1882124989 >> v1:172.21.15.192:6800/34229 conn(0x7f56b406d390 legacy=0x7f56b406f850 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-06-03T08:36:27.580 INFO:teuthology.misc.health.smithi088.stderr:2024-06-03T08:36:27.578+0000 7f56d8595640 10 monclient: shutdown 2024-06-03T08:36:27.580 INFO:teuthology.misc.health.smithi088.stderr:2024-06-03T08:36:27.578+0000 7f56d8595640 20 monclient: shutdown discarding 0 pending message(s) 2024-06-03T08:36:27.580 INFO:teuthology.misc.health.smithi088.stderr:2024-06-03T08:36:27.578+0000 7f56d8595640 1 -- 172.21.15.88:0/1882124989 >> v1:172.21.15.192:6789/0 conn(0x7f56d013d940 legacy=0x7f56d00abe20 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-06-03T08:36:27.580 INFO:teuthology.misc.health.smithi088.stderr:2024-06-03T08:36:27.578+0000 7f56d8595640 1 -- 172.21.15.88:0/1882124989 shutdown_connections 2024-06-03T08:36:27.580 INFO:teuthology.misc.health.smithi088.stderr:2024-06-03T08:36:27.578+0000 7f56d8595640 1 -- 172.21.15.88:0/1882124989 >> 172.21.15.88:0/1882124989 conn(0x7f56d00b2f20 msgr2=0x7f56d00ae070 unknown :-1 s=STATE_NONE l=0).mark_down 2024-06-03T08:36:27.580 INFO:teuthology.misc.health.smithi088.stderr:2024-06-03T08:36:27.579+0000 7f56d8595640 1 -- 172.21.15.88:0/1882124989 shutdown_connections 2024-06-03T08:36:27.580 INFO:teuthology.misc.health.smithi088.stderr:2024-06-03T08:36:27.579+0000 7f56d8595640 1 -- 172.21.15.88:0/1882124989 wait complete. 2024-06-03T08:36:27.580 INFO:teuthology.misc.health.smithi088.stderr:2024-06-03T08:36:27.579+0000 7f56d8595640 1 librados: shutdown 2024-06-03T08:36:27.588 DEBUG:teuthology.misc:8 of 8 OSDs are up 2024-06-03T08:36:27.589 INFO:tasks.ceph:Creating RBD pool 2024-06-03T08:36:27.589 DEBUG:teuthology.orchestra.run.smithi088:> sudo ceph --cluster ceph osd pool create rbd 8 2024-06-03T08:36:27.687 INFO:teuthology.orchestra.run.smithi088.stderr:2024-06-03T08:36:27.687+0000 7fa3447b2640 10 monclient: get_monmap_and_config 2024-06-03T08:36:27.687 INFO:teuthology.orchestra.run.smithi088.stderr:2024-06-03T08:36:27.687+0000 7fa3447b2640 10 monclient: build_initial_monmap 2024-06-03T08:36:27.687 INFO:teuthology.orchestra.run.smithi088.stderr:2024-06-03T08:36:27.687+0000 7fa3447b2640 10 monclient: monmap: 2024-06-03T08:36:27.687 INFO:teuthology.orchestra.run.smithi088.stderr:epoch 0 2024-06-03T08:36:27.687 INFO:teuthology.orchestra.run.smithi088.stderr:fsid ec3c3fdd-3ac6-4d7d-9412-683130ed8b04 2024-06-03T08:36:27.687 INFO:teuthology.orchestra.run.smithi088.stderr:last_changed 2024-06-03T08:36:27.687345+0000 2024-06-03T08:36:27.687 INFO:teuthology.orchestra.run.smithi088.stderr:created 2024-06-03T08:36:27.687345+0000 2024-06-03T08:36:27.687 INFO:teuthology.orchestra.run.smithi088.stderr:min_mon_release 0 (unknown) 2024-06-03T08:36:27.687 INFO:teuthology.orchestra.run.smithi088.stderr:election_strategy: 3 2024-06-03T08:36:27.688 INFO:teuthology.orchestra.run.smithi088.stderr:0: v1:172.21.15.88:6789/0 mon.noname-a 2024-06-03T08:36:27.688 INFO:teuthology.orchestra.run.smithi088.stderr:1: v1:172.21.15.192:6789/0 mon.noname-c 2024-06-03T08:36:27.688 INFO:teuthology.orchestra.run.smithi088.stderr:2: v1:172.21.15.88:6790/0 mon.noname-b 2024-06-03T08:36:27.688 INFO:teuthology.orchestra.run.smithi088.stderr: 2024-06-03T08:36:27.688 INFO:teuthology.orchestra.run.smithi088.stderr:2024-06-03T08:36:27.687+0000 7fa3447b2640 1 Processor -- start 2024-06-03T08:36:27.688 INFO:teuthology.orchestra.run.smithi088.stderr:2024-06-03T08:36:27.687+0000 7fa3447b2640 1 -- start start 2024-06-03T08:36:27.688 INFO:teuthology.orchestra.run.smithi088.stderr:2024-06-03T08:36:27.687+0000 7fa3447b2640 10 monclient: init 2024-06-03T08:36:27.689 INFO:teuthology.orchestra.run.smithi088.stderr:2024-06-03T08:36:27.687+0000 7fa3447b2640 10 monclient: _reopen_session rank -1 2024-06-03T08:36:27.689 INFO:teuthology.orchestra.run.smithi088.stderr:2024-06-03T08:36:27.687+0000 7fa3447b2640 10 monclient: _add_conns ranks=[2,0,1] 2024-06-03T08:36:27.689 INFO:teuthology.orchestra.run.smithi088.stderr:2024-06-03T08:36:27.687+0000 7fa3447b2640 10 monclient(hunting): picked mon.noname-b con 0x7fa33c06fc70 addr v1:172.21.15.88:6790/0 2024-06-03T08:36:27.689 INFO:teuthology.orchestra.run.smithi088.stderr:2024-06-03T08:36:27.687+0000 7fa3447b2640 10 monclient(hunting): picked mon.noname-a con 0x7fa33c0ad920 addr v1:172.21.15.88:6789/0 2024-06-03T08:36:27.689 INFO:teuthology.orchestra.run.smithi088.stderr:2024-06-03T08:36:27.687+0000 7fa3447b2640 10 monclient(hunting): picked mon.noname-c con 0x7fa33c1393f0 addr v1:172.21.15.192:6789/0 2024-06-03T08:36:27.689 INFO:teuthology.orchestra.run.smithi088.stderr:2024-06-03T08:36:27.687+0000 7fa3447b2640 1 -- --> v1:172.21.15.88:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fa33c070420 con 0x7fa33c0ad920 2024-06-03T08:36:27.689 INFO:teuthology.orchestra.run.smithi088.stderr:2024-06-03T08:36:27.687+0000 7fa3447b2640 1 -- --> v1:172.21.15.192:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fa33c071050 con 0x7fa33c1393f0 2024-06-03T08:36:27.689 INFO:teuthology.orchestra.run.smithi088.stderr:2024-06-03T08:36:27.687+0000 7fa3447b2640 1 -- --> v1:172.21.15.88:6790/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fa33c0add60 con 0x7fa33c06fc70 2024-06-03T08:36:27.689 INFO:teuthology.orchestra.run.smithi088.stderr:2024-06-03T08:36:27.687+0000 7fa3447b2640 10 monclient(hunting): _renew_subs 2024-06-03T08:36:27.689 INFO:teuthology.orchestra.run.smithi088.stderr:2024-06-03T08:36:27.687+0000 7fa3447b2640 10 monclient(hunting): authenticate will time out at 1243.497665s 2024-06-03T08:36:27.690 INFO:teuthology.orchestra.run.smithi088.stderr:2024-06-03T08:36:27.688+0000 7fa341d4d640 1 --1- >> v1:172.21.15.88:6789/0 conn(0x7fa33c0ad920 0x7fa33c138ce0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.88:6789/0 says I am v1:172.21.15.88:60190/0 (socket says 172.21.15.88:60190) 2024-06-03T08:36:27.690 INFO:teuthology.orchestra.run.smithi088.stderr:2024-06-03T08:36:27.688+0000 7fa341d4d640 1 -- 172.21.15.88:0/3835153564 learned_addr learned my addr 172.21.15.88:0/3835153564 (peer_addr_for_me v1:172.21.15.88:0/0) 2024-06-03T08:36:27.690 INFO:teuthology.orchestra.run.smithi088.stderr:2024-06-03T08:36:27.688+0000 7fa34254e640 1 --1- >> v1:172.21.15.88:6790/0 conn(0x7fa33c06fc70 0x7fa33c13fbb0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.88:6790/0 says I am v1:172.21.15.88:54006/0 (socket says 172.21.15.88:54006) 2024-06-03T08:36:27.691 INFO:teuthology.orchestra.run.smithi088.stderr:2024-06-03T08:36:27.691+0000 7fa34154c640 1 -- 172.21.15.88:0/3835153564 <== mon.2 v1:172.21.15.88:6790/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 3654571498 0 0) 0x7fa33c0add60 con 0x7fa33c06fc70 2024-06-03T08:36:27.691 INFO:teuthology.orchestra.run.smithi088.stderr:2024-06-03T08:36:27.691+0000 7fa34154c640 10 monclient(hunting): _negotiate 2024-06-03T08:36:27.692 INFO:teuthology.orchestra.run.smithi088.stderr:2024-06-03T08:36:27.691+0000 7fa34154c640 10 monclient(hunting): _init_auth method 2 2024-06-03T08:36:27.692 INFO:teuthology.orchestra.run.smithi088.stderr:2024-06-03T08:36:27.691+0000 7fa34154c640 10 monclient(hunting): _init_auth creating new auth 2024-06-03T08:36:27.692 INFO:teuthology.orchestra.run.smithi088.stderr:2024-06-03T08:36:27.691+0000 7fa34154c640 10 monclient(hunting): my global_id is 4283 2024-06-03T08:36:27.692 INFO:teuthology.orchestra.run.smithi088.stderr:2024-06-03T08:36:27.691+0000 7fa34154c640 1 -- 172.21.15.88:0/3835153564 --> v1:172.21.15.88:6790/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7fa324003520 con 0x7fa33c06fc70 2024-06-03T08:36:27.692 INFO:teuthology.orchestra.run.smithi088.stderr:2024-06-03T08:36:27.691+0000 7fa34154c640 1 -- 172.21.15.88:0/3835153564 <== mon.1 v1:172.21.15.192:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 1786268254 0 0) 0x7fa33c071050 con 0x7fa33c1393f0 2024-06-03T08:36:27.692 INFO:teuthology.orchestra.run.smithi088.stderr:2024-06-03T08:36:27.691+0000 7fa34154c640 10 monclient(hunting): _negotiate 2024-06-03T08:36:27.692 INFO:teuthology.orchestra.run.smithi088.stderr:2024-06-03T08:36:27.691+0000 7fa34154c640 10 monclient(hunting): _init_auth method 2 2024-06-03T08:36:27.692 INFO:teuthology.orchestra.run.smithi088.stderr:2024-06-03T08:36:27.691+0000 7fa34154c640 10 monclient(hunting): _init_auth creating new auth 2024-06-03T08:36:27.692 INFO:teuthology.orchestra.run.smithi088.stderr:2024-06-03T08:36:27.691+0000 7fa34154c640 10 monclient(hunting): my global_id is 4264 2024-06-03T08:36:27.692 INFO:teuthology.orchestra.run.smithi088.stderr:2024-06-03T08:36:27.691+0000 7fa34154c640 1 -- 172.21.15.88:0/3835153564 --> v1:172.21.15.192:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7fa33c0add60 con 0x7fa33c1393f0 2024-06-03T08:36:27.693 INFO:teuthology.orchestra.run.smithi088.stderr:2024-06-03T08:36:27.692+0000 7fa34154c640 1 -- 172.21.15.88:0/3835153564 <== mon.0 v1:172.21.15.88:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 2333377619 0 0) 0x7fa33c070420 con 0x7fa33c0ad920 2024-06-03T08:36:27.693 INFO:teuthology.orchestra.run.smithi088.stderr:2024-06-03T08:36:27.692+0000 7fa34154c640 10 monclient(hunting): _negotiate 2024-06-03T08:36:27.693 INFO:teuthology.orchestra.run.smithi088.stderr:2024-06-03T08:36:27.692+0000 7fa34154c640 10 monclient(hunting): _init_auth method 2 2024-06-03T08:36:27.693 INFO:teuthology.orchestra.run.smithi088.stderr:2024-06-03T08:36:27.692+0000 7fa34154c640 10 monclient(hunting): _init_auth creating new auth 2024-06-03T08:36:27.693 INFO:teuthology.orchestra.run.smithi088.stderr:2024-06-03T08:36:27.692+0000 7fa34154c640 10 monclient(hunting): my global_id is 4269 2024-06-03T08:36:27.693 INFO:teuthology.orchestra.run.smithi088.stderr:2024-06-03T08:36:27.692+0000 7fa34154c640 1 -- 172.21.15.88:0/3835153564 --> v1:172.21.15.88:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7fa33c071050 con 0x7fa33c0ad920 2024-06-03T08:36:27.695 INFO:teuthology.orchestra.run.smithi088.stderr:2024-06-03T08:36:27.694+0000 7fa34154c640 1 -- 172.21.15.88:0/3835153564 <== mon.1 v1:172.21.15.192:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 214+0+0 (unknown 2363673194 0 0) 0x7fa33c0add60 con 0x7fa33c1393f0 2024-06-03T08:36:27.695 INFO:teuthology.orchestra.run.smithi088.stderr:2024-06-03T08:36:27.694+0000 7fa34154c640 1 -- 172.21.15.88:0/3835153564 >> v1:172.21.15.88:6790/0 conn(0x7fa33c06fc70 legacy=0x7fa33c13fbb0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-06-03T08:36:27.695 INFO:teuthology.orchestra.run.smithi088.stderr:2024-06-03T08:36:27.694+0000 7fa34154c640 1 -- 172.21.15.88:0/3835153564 >> v1:172.21.15.88:6789/0 conn(0x7fa33c0ad920 legacy=0x7fa33c138ce0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-06-03T08:36:27.695 INFO:teuthology.orchestra.run.smithi088.stderr:2024-06-03T08:36:27.694+0000 7fa34154c640 10 monclient: _finish_hunting 0 2024-06-03T08:36:27.695 INFO:teuthology.orchestra.run.smithi088.stderr:2024-06-03T08:36:27.694+0000 7fa34154c640 1 monclient: found mon.noname-c 2024-06-03T08:36:27.696 INFO:teuthology.orchestra.run.smithi088.stderr:2024-06-03T08:36:27.694+0000 7fa34154c640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-06-03T08:36:27.696 INFO:teuthology.orchestra.run.smithi088.stderr:2024-06-03T08:36:27.694+0000 7fa34154c640 10 monclient: _send_mon_message to mon.noname-c at v1:172.21.15.192:6789/0 2024-06-03T08:36:27.696 INFO:teuthology.orchestra.run.smithi088.stderr:2024-06-03T08:36:27.694+0000 7fa34154c640 1 -- 172.21.15.88:0/3835153564 --> v1:172.21.15.192:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fa33c071be0 con 0x7fa33c1393f0 2024-06-03T08:36:27.696 INFO:teuthology.orchestra.run.smithi088.stderr:2024-06-03T08:36:27.694+0000 7fa34154c640 10 monclient: _finish_auth 0 2024-06-03T08:36:27.696 INFO:teuthology.orchestra.run.smithi088.stderr:2024-06-03T08:36:27.694+0000 7fa34154c640 10 monclient: _check_auth_tickets 2024-06-03T08:36:27.696 INFO:teuthology.orchestra.run.smithi088.stderr:2024-06-03T08:36:27.694+0000 7fa34154c640 20 monclient: _check_auth_rotating not needed by client.admin 2024-06-03T08:36:27.696 INFO:teuthology.orchestra.run.smithi088.stderr:2024-06-03T08:36:27.694+0000 7fa3447b2640 5 monclient: authenticate success, global_id 4264 2024-06-03T08:36:27.696 INFO:teuthology.orchestra.run.smithi088.stderr:2024-06-03T08:36:27.694+0000 7fa3447b2640 20 monclient: get_monmap_and_config waiting for monmap|config 2024-06-03T08:36:27.696 INFO:teuthology.orchestra.run.smithi088.stderr:2024-06-03T08:36:27.694+0000 7fa34154c640 1 -- 172.21.15.88:0/3835153564 <== mon.1 v1:172.21.15.192:6789/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 2234374065 0 0) 0x7fa32c001970 con 0x7fa33c1393f0 2024-06-03T08:36:27.696 INFO:teuthology.orchestra.run.smithi088.stderr:2024-06-03T08:36:27.695+0000 7fa34154c640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-06-03T08:36:27.696 INFO:teuthology.orchestra.run.smithi088.stderr:2024-06-03T08:36:27.695+0000 7fa34154c640 10 monclient: got monmap 1 from mon.noname-c (according to old e1) 2024-06-03T08:36:27.696 INFO:teuthology.orchestra.run.smithi088.stderr:2024-06-03T08:36:27.695+0000 7fa34154c640 10 monclient: dump: 2024-06-03T08:36:27.696 INFO:teuthology.orchestra.run.smithi088.stderr:epoch 1 2024-06-03T08:36:27.697 INFO:teuthology.orchestra.run.smithi088.stderr:fsid ec3c3fdd-3ac6-4d7d-9412-683130ed8b04 2024-06-03T08:36:27.697 INFO:teuthology.orchestra.run.smithi088.stderr:last_changed 2024-06-03T08:34:31.885125+0000 2024-06-03T08:36:27.697 INFO:teuthology.orchestra.run.smithi088.stderr:created 2024-06-03T08:34:31.885125+0000 2024-06-03T08:36:27.697 INFO:teuthology.orchestra.run.smithi088.stderr:min_mon_release 18 (reef) 2024-06-03T08:36:27.697 INFO:teuthology.orchestra.run.smithi088.stderr:election_strategy: 3 2024-06-03T08:36:27.697 INFO:teuthology.orchestra.run.smithi088.stderr:0: v1:172.21.15.88:6789/0 mon.a 2024-06-03T08:36:27.697 INFO:teuthology.orchestra.run.smithi088.stderr:1: v1:172.21.15.192:6789/0 mon.b 2024-06-03T08:36:27.697 INFO:teuthology.orchestra.run.smithi088.stderr:2: v1:172.21.15.88:6790/0 mon.c 2024-06-03T08:36:27.697 INFO:teuthology.orchestra.run.smithi088.stderr: 2024-06-03T08:36:27.697 INFO:teuthology.orchestra.run.smithi088.stderr:2024-06-03T08:36:27.695+0000 7fa3447b2640 20 monclient: get_monmap_and_config waiting for monmap|config 2024-06-03T08:36:27.698 INFO:teuthology.orchestra.run.smithi088.stderr:2024-06-03T08:36:27.697+0000 7fa34154c640 1 -- 172.21.15.88:0/3835153564 <== mon.1 v1:172.21.15.192:6789/0 4 ==== config(0 keys) v1 ==== 4+0+0 (unknown 0 0 0) 0x7fa32c001b90 con 0x7fa33c1393f0 2024-06-03T08:36:27.698 INFO:teuthology.orchestra.run.smithi088.stderr:2024-06-03T08:36:27.697+0000 7fa34154c640 10 monclient: handle_config config(0 keys) v1 2024-06-03T08:36:27.698 INFO:teuthology.orchestra.run.smithi088.stderr:2024-06-03T08:36:27.697+0000 7fa34154c640 1 -- 172.21.15.88:0/3835153564 <== mon.1 v1:172.21.15.192:6789/0 5 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 2234374065 0 0) 0x7fa32c001ee0 con 0x7fa33c1393f0 2024-06-03T08:36:27.698 INFO:teuthology.orchestra.run.smithi088.stderr:2024-06-03T08:36:27.697+0000 7fa34154c640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-06-03T08:36:27.698 INFO:teuthology.orchestra.run.smithi088.stderr:2024-06-03T08:36:27.697+0000 7fa34154c640 10 monclient: got monmap 1 from mon.b (according to old e1) 2024-06-03T08:36:27.698 INFO:teuthology.orchestra.run.smithi088.stderr:2024-06-03T08:36:27.697+0000 7fa34154c640 10 monclient: dump: 2024-06-03T08:36:27.698 INFO:teuthology.orchestra.run.smithi088.stderr:epoch 1 2024-06-03T08:36:27.698 INFO:teuthology.orchestra.run.smithi088.stderr:fsid ec3c3fdd-3ac6-4d7d-9412-683130ed8b04 2024-06-03T08:36:27.698 INFO:teuthology.orchestra.run.smithi088.stderr:last_changed 2024-06-03T08:34:31.885125+0000 2024-06-03T08:36:27.699 INFO:teuthology.orchestra.run.smithi088.stderr:created 2024-06-03T08:34:31.885125+0000 2024-06-03T08:36:27.699 INFO:teuthology.orchestra.run.smithi088.stderr:min_mon_release 18 (reef) 2024-06-03T08:36:27.699 INFO:teuthology.orchestra.run.smithi088.stderr:election_strategy: 3 2024-06-03T08:36:27.699 INFO:teuthology.orchestra.run.smithi088.stderr:0: v1:172.21.15.88:6789/0 mon.a 2024-06-03T08:36:27.699 INFO:teuthology.orchestra.run.smithi088.stderr:1: v1:172.21.15.192:6789/0 mon.b 2024-06-03T08:36:27.699 INFO:teuthology.orchestra.run.smithi088.stderr:2: v1:172.21.15.88:6790/0 mon.c 2024-06-03T08:36:27.699 INFO:teuthology.orchestra.run.smithi088.stderr: 2024-06-03T08:36:27.699 INFO:teuthology.orchestra.run.smithi088.stderr:2024-06-03T08:36:27.697+0000 7fa3447b2640 10 monclient: get_monmap_and_config success 2024-06-03T08:36:27.699 INFO:teuthology.orchestra.run.smithi088.stderr:2024-06-03T08:36:27.697+0000 7fa3447b2640 4 set_mon_vals no callback set 2024-06-03T08:36:27.701 INFO:teuthology.orchestra.run.smithi088.stderr:2024-06-03T08:36:27.698+0000 7fa3447b2640 10 monclient: shutdown 2024-06-03T08:36:27.702 INFO:teuthology.orchestra.run.smithi088.stderr:2024-06-03T08:36:27.698+0000 7fa3447b2640 20 monclient: shutdown discarding 0 pending message(s) 2024-06-03T08:36:27.702 INFO:teuthology.orchestra.run.smithi088.stderr:2024-06-03T08:36:27.698+0000 7fa3447b2640 1 -- 172.21.15.88:0/3835153564 >> v1:172.21.15.192:6789/0 conn(0x7fa33c1393f0 legacy=0x7fa33c14ac50 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-06-03T08:36:27.702 INFO:teuthology.orchestra.run.smithi088.stderr:2024-06-03T08:36:27.698+0000 7fa3447b2640 1 -- 172.21.15.88:0/3835153564 shutdown_connections 2024-06-03T08:36:27.702 INFO:teuthology.orchestra.run.smithi088.stderr:2024-06-03T08:36:27.698+0000 7fa3447b2640 1 -- 172.21.15.88:0/3835153564 >> 172.21.15.88:0/3835153564 conn(0x7fa33c0b2f20 msgr2=0x7fa33c0b3300 unknown :-1 s=STATE_NONE l=0).mark_down 2024-06-03T08:36:27.702 INFO:teuthology.orchestra.run.smithi088.stderr:2024-06-03T08:36:27.698+0000 7fa3447b2640 1 -- 172.21.15.88:0/3835153564 shutdown_connections 2024-06-03T08:36:27.702 INFO:teuthology.orchestra.run.smithi088.stderr:2024-06-03T08:36:27.698+0000 7fa3447b2640 1 -- 172.21.15.88:0/3835153564 wait complete. 2024-06-03T08:36:27.702 INFO:teuthology.orchestra.run.smithi088.stderr:2024-06-03T08:36:27.699+0000 7fa3447b2640 10 monclient: build_initial_monmap 2024-06-03T08:36:27.702 INFO:teuthology.orchestra.run.smithi088.stderr:2024-06-03T08:36:27.699+0000 7fa3447b2640 10 monclient: monmap: 2024-06-03T08:36:27.702 INFO:teuthology.orchestra.run.smithi088.stderr:epoch 0 2024-06-03T08:36:27.702 INFO:teuthology.orchestra.run.smithi088.stderr:fsid 00000000-0000-0000-0000-000000000000 2024-06-03T08:36:27.702 INFO:teuthology.orchestra.run.smithi088.stderr:last_changed 0.000000 2024-06-03T08:36:27.702 INFO:teuthology.orchestra.run.smithi088.stderr:created 0.000000 2024-06-03T08:36:27.702 INFO:teuthology.orchestra.run.smithi088.stderr:min_mon_release 0 (unknown) 2024-06-03T08:36:27.703 INFO:teuthology.orchestra.run.smithi088.stderr:election_strategy: 1 2024-06-03T08:36:27.703 INFO:teuthology.orchestra.run.smithi088.stderr:0: v1:172.21.15.88:6789/0 mon.noname-a 2024-06-03T08:36:27.703 INFO:teuthology.orchestra.run.smithi088.stderr:1: v1:172.21.15.192:6789/0 mon.noname-b 2024-06-03T08:36:27.703 INFO:teuthology.orchestra.run.smithi088.stderr:2: v1:172.21.15.88:6790/0 mon.noname-c 2024-06-03T08:36:27.703 INFO:teuthology.orchestra.run.smithi088.stderr: 2024-06-03T08:36:27.703 INFO:teuthology.orchestra.run.smithi088.stderr:2024-06-03T08:36:27.699+0000 7fa3447b2640 1 librados: starting msgr at 2024-06-03T08:36:27.703 INFO:teuthology.orchestra.run.smithi088.stderr:2024-06-03T08:36:27.699+0000 7fa3447b2640 1 librados: starting objecter 2024-06-03T08:36:27.703 INFO:teuthology.orchestra.run.smithi088.stderr:2024-06-03T08:36:27.699+0000 7fa3447b2640 1 Processor -- start 2024-06-03T08:36:27.703 INFO:teuthology.orchestra.run.smithi088.stderr:2024-06-03T08:36:27.699+0000 7fa3447b2640 1 -- start start 2024-06-03T08:36:27.703 INFO:teuthology.orchestra.run.smithi088.stderr:2024-06-03T08:36:27.699+0000 7fa3447b2640 1 librados: setting wanted keys 2024-06-03T08:36:27.703 INFO:teuthology.orchestra.run.smithi088.stderr:2024-06-03T08:36:27.699+0000 7fa3447b2640 1 librados: calling monclient init 2024-06-03T08:36:27.703 INFO:teuthology.orchestra.run.smithi088.stderr:2024-06-03T08:36:27.699+0000 7fa3447b2640 10 monclient: init 2024-06-03T08:36:27.704 INFO:teuthology.orchestra.run.smithi088.stderr:2024-06-03T08:36:27.700+0000 7fa3447b2640 10 monclient: _reopen_session rank -1 2024-06-03T08:36:27.704 INFO:teuthology.orchestra.run.smithi088.stderr:2024-06-03T08:36:27.700+0000 7fa3447b2640 10 monclient: _add_conns ranks=[0,2,1] 2024-06-03T08:36:27.704 INFO:teuthology.orchestra.run.smithi088.stderr:2024-06-03T08:36:27.700+0000 7fa3447b2640 10 monclient(hunting): picked mon.noname-a con 0x7fa33c06fc70 addr v1:172.21.15.88:6789/0 2024-06-03T08:36:27.704 INFO:teuthology.orchestra.run.smithi088.stderr:2024-06-03T08:36:27.700+0000 7fa3447b2640 10 monclient(hunting): picked mon.noname-c con 0x7fa33c0ad920 addr v1:172.21.15.88:6790/0 2024-06-03T08:36:27.704 INFO:teuthology.orchestra.run.smithi088.stderr:2024-06-03T08:36:27.700+0000 7fa3447b2640 10 monclient(hunting): picked mon.noname-b con 0x7fa33c1393f0 addr v1:172.21.15.192:6789/0 2024-06-03T08:36:27.704 INFO:teuthology.orchestra.run.smithi088.stderr:2024-06-03T08:36:27.700+0000 7fa3447b2640 1 -- --> v1:172.21.15.88:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fa33c1422c0 con 0x7fa33c06fc70 2024-06-03T08:36:27.704 INFO:teuthology.orchestra.run.smithi088.stderr:2024-06-03T08:36:27.700+0000 7fa3447b2640 1 -- --> v1:172.21.15.192:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fa33c0a9440 con 0x7fa33c1393f0 2024-06-03T08:36:27.704 INFO:teuthology.orchestra.run.smithi088.stderr:2024-06-03T08:36:27.700+0000 7fa3447b2640 1 -- --> v1:172.21.15.88:6790/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fa33c1d5210 con 0x7fa33c0ad920 2024-06-03T08:36:27.704 INFO:teuthology.orchestra.run.smithi088.stderr:2024-06-03T08:36:27.700+0000 7fa3447b2640 10 monclient(hunting): _renew_subs 2024-06-03T08:36:27.704 INFO:teuthology.orchestra.run.smithi088.stderr:2024-06-03T08:36:27.700+0000 7fa3447b2640 10 monclient(hunting): authenticate will time out at 1243.510511s 2024-06-03T08:36:27.704 INFO:teuthology.orchestra.run.smithi088.stderr:2024-06-03T08:36:27.701+0000 7fa34254e640 1 --1- >> v1:172.21.15.88:6789/0 conn(0x7fa33c06fc70 0x7fa33c0a8d30 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=1).handle_server_banner_and_identify peer v1:172.21.15.88:6789/0 says I am v1:172.21.15.88:60192/0 (socket says 172.21.15.88:60192) 2024-06-03T08:36:27.704 INFO:teuthology.orchestra.run.smithi088.stderr:2024-06-03T08:36:27.701+0000 7fa34254e640 1 -- 172.21.15.88:0/749058204 learned_addr learned my addr 172.21.15.88:0/749058204 (peer_addr_for_me v1:172.21.15.88:0/0) 2024-06-03T08:36:27.704 INFO:teuthology.orchestra.run.smithi088.stderr:2024-06-03T08:36:27.701+0000 7fa33affd640 10 client.?.objecter ms_handle_connect 0x7fa33c06fc70 2024-06-03T08:36:27.704 INFO:teuthology.orchestra.run.smithi088.stderr:2024-06-03T08:36:27.701+0000 7fa33affd640 10 client.?.objecter resend_mon_ops 2024-06-03T08:36:27.705 INFO:teuthology.orchestra.run.smithi088.stderr:2024-06-03T08:36:27.701+0000 7fa33affd640 10 client.?.objecter ms_handle_connect 0x7fa33c0ad920 2024-06-03T08:36:27.705 INFO:teuthology.orchestra.run.smithi088.stderr:2024-06-03T08:36:27.701+0000 7fa33affd640 10 client.?.objecter resend_mon_ops 2024-06-03T08:36:27.705 INFO:teuthology.orchestra.run.smithi088.stderr:2024-06-03T08:36:27.701+0000 7fa33affd640 10 client.?.objecter ms_handle_connect 0x7fa33c1393f0 2024-06-03T08:36:27.705 INFO:teuthology.orchestra.run.smithi088.stderr:2024-06-03T08:36:27.701+0000 7fa33affd640 10 client.?.objecter resend_mon_ops 2024-06-03T08:36:27.705 INFO:teuthology.orchestra.run.smithi088.stderr:2024-06-03T08:36:27.703+0000 7fa33affd640 1 -- 172.21.15.88:0/749058204 <== mon.1 v1:172.21.15.192:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 112300825 0 0) 0x7fa33c0a9440 con 0x7fa33c1393f0 2024-06-03T08:36:27.706 INFO:teuthology.orchestra.run.smithi088.stderr:2024-06-03T08:36:27.703+0000 7fa33affd640 10 monclient(hunting): _negotiate 2024-06-03T08:36:27.706 INFO:teuthology.orchestra.run.smithi088.stderr:2024-06-03T08:36:27.703+0000 7fa33affd640 10 monclient(hunting): _init_auth method 2 2024-06-03T08:36:27.706 INFO:teuthology.orchestra.run.smithi088.stderr:2024-06-03T08:36:27.703+0000 7fa33affd640 10 monclient(hunting): _init_auth creating new auth 2024-06-03T08:36:27.706 INFO:teuthology.orchestra.run.smithi088.stderr:2024-06-03T08:36:27.703+0000 7fa33affd640 10 monclient(hunting): my global_id is 4267 2024-06-03T08:36:27.706 INFO:teuthology.orchestra.run.smithi088.stderr:2024-06-03T08:36:27.703+0000 7fa33affd640 1 -- 172.21.15.88:0/749058204 --> v1:172.21.15.192:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7fa328004eb0 con 0x7fa33c1393f0 2024-06-03T08:36:27.706 INFO:teuthology.orchestra.run.smithi088.stderr:2024-06-03T08:36:27.703+0000 7fa33affd640 1 -- 172.21.15.88:0/749058204 <== mon.0 v1:172.21.15.88:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 1755628503 0 0) 0x7fa33c1422c0 con 0x7fa33c06fc70 2024-06-03T08:36:27.706 INFO:teuthology.orchestra.run.smithi088.stderr:2024-06-03T08:36:27.703+0000 7fa33affd640 10 monclient(hunting): _negotiate 2024-06-03T08:36:27.706 INFO:teuthology.orchestra.run.smithi088.stderr:2024-06-03T08:36:27.703+0000 7fa33affd640 10 monclient(hunting): _init_auth method 2 2024-06-03T08:36:27.706 INFO:teuthology.orchestra.run.smithi088.stderr:2024-06-03T08:36:27.703+0000 7fa33affd640 10 monclient(hunting): _init_auth creating new auth 2024-06-03T08:36:27.706 INFO:teuthology.orchestra.run.smithi088.stderr:2024-06-03T08:36:27.703+0000 7fa33affd640 10 monclient(hunting): my global_id is 4272 2024-06-03T08:36:27.706 INFO:teuthology.orchestra.run.smithi088.stderr:2024-06-03T08:36:27.703+0000 7fa33affd640 1 -- 172.21.15.88:0/749058204 --> v1:172.21.15.88:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7fa33c0a9440 con 0x7fa33c06fc70 2024-06-03T08:36:27.707 INFO:teuthology.orchestra.run.smithi088.stderr:2024-06-03T08:36:27.703+0000 7fa33affd640 1 -- 172.21.15.88:0/749058204 <== mon.2 v1:172.21.15.88:6790/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 2910384170 0 0) 0x7fa33c1d5210 con 0x7fa33c0ad920 2024-06-03T08:36:27.707 INFO:teuthology.orchestra.run.smithi088.stderr:2024-06-03T08:36:27.703+0000 7fa33affd640 10 monclient(hunting): _negotiate 2024-06-03T08:36:27.707 INFO:teuthology.orchestra.run.smithi088.stderr:2024-06-03T08:36:27.703+0000 7fa33affd640 10 monclient(hunting): _init_auth method 2 2024-06-03T08:36:27.707 INFO:teuthology.orchestra.run.smithi088.stderr:2024-06-03T08:36:27.703+0000 7fa33affd640 10 monclient(hunting): _init_auth creating new auth 2024-06-03T08:36:27.707 INFO:teuthology.orchestra.run.smithi088.stderr:2024-06-03T08:36:27.703+0000 7fa33affd640 10 monclient(hunting): my global_id is 4286 2024-06-03T08:36:27.707 INFO:teuthology.orchestra.run.smithi088.stderr:2024-06-03T08:36:27.703+0000 7fa33affd640 1 -- 172.21.15.88:0/749058204 --> v1:172.21.15.88:6790/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7fa33c1422c0 con 0x7fa33c0ad920 2024-06-03T08:36:27.708 INFO:teuthology.orchestra.run.smithi088.stderr:2024-06-03T08:36:27.707+0000 7fa33affd640 1 -- 172.21.15.88:0/749058204 <== mon.1 v1:172.21.15.192:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 764+0+0 (unknown 1743586441 0 0) 0x7fa328004eb0 con 0x7fa33c1393f0 2024-06-03T08:36:27.708 INFO:teuthology.orchestra.run.smithi088.stderr:2024-06-03T08:36:27.707+0000 7fa33affd640 1 -- 172.21.15.88:0/749058204 --> v1:172.21.15.192:6789/0 -- auth(proto 2 165 bytes epoch 0) v1 -- 0x7fa33c1d5210 con 0x7fa33c1393f0 2024-06-03T08:36:27.708 INFO:teuthology.orchestra.run.smithi088.stderr:2024-06-03T08:36:27.707+0000 7fa33affd640 1 -- 172.21.15.88:0/749058204 <== mon.1 v1:172.21.15.192:6789/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 2234374065 0 0) 0x7fa32c001ea0 con 0x7fa33c1393f0 2024-06-03T08:36:27.708 INFO:teuthology.orchestra.run.smithi088.stderr:2024-06-03T08:36:27.707+0000 7fa33affd640 10 monclient(hunting): handle_monmap mon_map magic: 0 v1 2024-06-03T08:36:27.708 INFO:teuthology.orchestra.run.smithi088.stderr:2024-06-03T08:36:27.707+0000 7fa33affd640 10 monclient(hunting): got monmap 1 from mon.noname-b (according to old e1) 2024-06-03T08:36:27.708 INFO:teuthology.orchestra.run.smithi088.stderr:2024-06-03T08:36:27.707+0000 7fa33affd640 10 monclient(hunting): dump: 2024-06-03T08:36:27.708 INFO:teuthology.orchestra.run.smithi088.stderr:epoch 1 2024-06-03T08:36:27.708 INFO:teuthology.orchestra.run.smithi088.stderr:fsid ec3c3fdd-3ac6-4d7d-9412-683130ed8b04 2024-06-03T08:36:27.708 INFO:teuthology.orchestra.run.smithi088.stderr:last_changed 2024-06-03T08:34:31.885125+0000 2024-06-03T08:36:27.709 INFO:teuthology.orchestra.run.smithi088.stderr:created 2024-06-03T08:34:31.885125+0000 2024-06-03T08:36:27.709 INFO:teuthology.orchestra.run.smithi088.stderr:min_mon_release 18 (reef) 2024-06-03T08:36:27.709 INFO:teuthology.orchestra.run.smithi088.stderr:election_strategy: 3 2024-06-03T08:36:27.709 INFO:teuthology.orchestra.run.smithi088.stderr:0: v1:172.21.15.88:6789/0 mon.a 2024-06-03T08:36:27.709 INFO:teuthology.orchestra.run.smithi088.stderr:1: v1:172.21.15.192:6789/0 mon.b 2024-06-03T08:36:27.709 INFO:teuthology.orchestra.run.smithi088.stderr:2: v1:172.21.15.88:6790/0 mon.c 2024-06-03T08:36:27.709 INFO:teuthology.orchestra.run.smithi088.stderr: 2024-06-03T08:36:27.709 INFO:teuthology.orchestra.run.smithi088.stderr:2024-06-03T08:36:27.707+0000 7fa33affd640 10 monclient(hunting): _finish_auth 0 2024-06-03T08:36:27.712 INFO:teuthology.orchestra.run.smithi088.stderr:2024-06-03T08:36:27.710+0000 7fa33affd640 1 -- 172.21.15.88:0/749058204 <== mon.2 v1:172.21.15.88:6790/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 764+0+0 (unknown 418099927 0 0) 0x7fa33c1422c0 con 0x7fa33c0ad920 2024-06-03T08:36:27.712 INFO:teuthology.orchestra.run.smithi088.stderr:2024-06-03T08:36:27.710+0000 7fa33affd640 1 -- 172.21.15.88:0/749058204 --> v1:172.21.15.88:6790/0 -- auth(proto 2 165 bytes epoch 0) v1 -- 0x7fa328004eb0 con 0x7fa33c0ad920 2024-06-03T08:36:27.712 INFO:teuthology.orchestra.run.smithi088.stderr:2024-06-03T08:36:27.710+0000 7fa33affd640 1 -- 172.21.15.88:0/749058204 <== mon.2 v1:172.21.15.88:6790/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 2234374065 0 0) 0x7fa330001870 con 0x7fa33c0ad920 2024-06-03T08:36:27.712 INFO:teuthology.orchestra.run.smithi088.stderr:2024-06-03T08:36:27.710+0000 7fa33affd640 10 monclient(hunting): handle_monmap mon_map magic: 0 v1 2024-06-03T08:36:27.712 INFO:teuthology.orchestra.run.smithi088.stderr:2024-06-03T08:36:27.710+0000 7fa33affd640 10 monclient(hunting): got monmap 1 from mon.c (according to old e1) 2024-06-03T08:36:27.712 INFO:teuthology.orchestra.run.smithi088.stderr:2024-06-03T08:36:27.710+0000 7fa33affd640 10 monclient(hunting): dump: 2024-06-03T08:36:27.712 INFO:teuthology.orchestra.run.smithi088.stderr:epoch 1 2024-06-03T08:36:27.712 INFO:teuthology.orchestra.run.smithi088.stderr:fsid ec3c3fdd-3ac6-4d7d-9412-683130ed8b04 2024-06-03T08:36:27.712 INFO:teuthology.orchestra.run.smithi088.stderr:last_changed 2024-06-03T08:34:31.885125+0000 2024-06-03T08:36:27.712 INFO:teuthology.orchestra.run.smithi088.stderr:created 2024-06-03T08:34:31.885125+0000 2024-06-03T08:36:27.712 INFO:teuthology.orchestra.run.smithi088.stderr:min_mon_release 18 (reef) 2024-06-03T08:36:27.712 INFO:teuthology.orchestra.run.smithi088.stderr:election_strategy: 3 2024-06-03T08:36:27.712 INFO:teuthology.orchestra.run.smithi088.stderr:0: v1:172.21.15.88:6789/0 mon.a 2024-06-03T08:36:27.713 INFO:teuthology.orchestra.run.smithi088.stderr:1: v1:172.21.15.192:6789/0 mon.b 2024-06-03T08:36:27.713 INFO:teuthology.orchestra.run.smithi088.stderr:2: v1:172.21.15.88:6790/0 mon.c 2024-06-03T08:36:27.713 INFO:teuthology.orchestra.run.smithi088.stderr: 2024-06-03T08:36:27.713 INFO:teuthology.orchestra.run.smithi088.stderr:2024-06-03T08:36:27.711+0000 7fa33affd640 1 -- 172.21.15.88:0/749058204 <== mon.1 v1:172.21.15.192:6789/0 4 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 580+0+0 (unknown 3067999530 0 0) 0x7fa33c1d5210 con 0x7fa33c1393f0 2024-06-03T08:36:27.713 INFO:teuthology.orchestra.run.smithi088.stderr:2024-06-03T08:36:27.711+0000 7fa33affd640 1 -- 172.21.15.88:0/749058204 >> v1:172.21.15.88:6790/0 conn(0x7fa33c0ad920 legacy=0x7fa33c1d4b00 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-06-03T08:36:27.713 INFO:teuthology.orchestra.run.smithi088.stderr:2024-06-03T08:36:27.711+0000 7fa33affd640 1 -- 172.21.15.88:0/749058204 >> v1:172.21.15.88:6789/0 conn(0x7fa33c06fc70 legacy=0x7fa33c0a8d30 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-06-03T08:36:27.713 INFO:teuthology.orchestra.run.smithi088.stderr:2024-06-03T08:36:27.711+0000 7fa33affd640 10 monclient: _finish_hunting 0 2024-06-03T08:36:27.713 INFO:teuthology.orchestra.run.smithi088.stderr:2024-06-03T08:36:27.711+0000 7fa33affd640 1 monclient: found mon.b 2024-06-03T08:36:27.713 INFO:teuthology.orchestra.run.smithi088.stderr:2024-06-03T08:36:27.711+0000 7fa33affd640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-06-03T08:36:27.713 INFO:teuthology.orchestra.run.smithi088.stderr:2024-06-03T08:36:27.711+0000 7fa33affd640 10 monclient: _send_mon_message to mon.b at v1:172.21.15.192:6789/0 2024-06-03T08:36:27.713 INFO:teuthology.orchestra.run.smithi088.stderr:2024-06-03T08:36:27.711+0000 7fa33affd640 1 -- 172.21.15.88:0/749058204 --> v1:172.21.15.192:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fa33c1d4300 con 0x7fa33c1393f0 2024-06-03T08:36:27.713 INFO:teuthology.orchestra.run.smithi088.stderr:2024-06-03T08:36:27.711+0000 7fa33affd640 10 monclient: _finish_auth 0 2024-06-03T08:36:27.713 INFO:teuthology.orchestra.run.smithi088.stderr:2024-06-03T08:36:27.711+0000 7fa33affd640 10 monclient: _check_auth_tickets 2024-06-03T08:36:27.713 INFO:teuthology.orchestra.run.smithi088.stderr:2024-06-03T08:36:27.711+0000 7fa33affd640 20 monclient: _check_auth_rotating not needed by client.admin 2024-06-03T08:36:27.714 INFO:teuthology.orchestra.run.smithi088.stderr:2024-06-03T08:36:27.711+0000 7fa3447b2640 5 monclient: authenticate success, global_id 4267 2024-06-03T08:36:27.714 INFO:teuthology.orchestra.run.smithi088.stderr:2024-06-03T08:36:27.711+0000 7fa3447b2640 10 monclient: _renew_subs 2024-06-03T08:36:27.714 INFO:teuthology.orchestra.run.smithi088.stderr:2024-06-03T08:36:27.711+0000 7fa3447b2640 10 monclient: _send_mon_message to mon.b at v1:172.21.15.192:6789/0 2024-06-03T08:36:27.714 INFO:teuthology.orchestra.run.smithi088.stderr:2024-06-03T08:36:27.711+0000 7fa3447b2640 1 -- 172.21.15.88:0/749058204 --> v1:172.21.15.192:6789/0 -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fa33c1d45e0 con 0x7fa33c1393f0 2024-06-03T08:36:27.714 INFO:teuthology.orchestra.run.smithi088.stderr:2024-06-03T08:36:27.711+0000 7fa3447b2640 10 client.4267.objecter _maybe_request_map subscribing (onetime) to next osd map 2024-06-03T08:36:27.714 INFO:teuthology.orchestra.run.smithi088.stderr:2024-06-03T08:36:27.711+0000 7fa3447b2640 10 monclient: _renew_subs 2024-06-03T08:36:27.714 INFO:teuthology.orchestra.run.smithi088.stderr:2024-06-03T08:36:27.711+0000 7fa3447b2640 10 monclient: _send_mon_message to mon.b at v1:172.21.15.192:6789/0 2024-06-03T08:36:27.714 INFO:teuthology.orchestra.run.smithi088.stderr:2024-06-03T08:36:27.711+0000 7fa3447b2640 1 -- 172.21.15.88:0/749058204 --> v1:172.21.15.192:6789/0 -- mon_subscribe({osdmap=0}) v3 -- 0x7fa33c143090 con 0x7fa33c1393f0 2024-06-03T08:36:27.714 INFO:teuthology.orchestra.run.smithi088.stderr:2024-06-03T08:36:27.711+0000 7fa3447b2640 1 librados: init done 2024-06-03T08:36:27.715 INFO:teuthology.orchestra.run.smithi088.stderr:2024-06-03T08:36:27.712+0000 7fa3447b2640 10 start_mon_command cmd=[{"prefix": "get_command_descriptions"}] 2024-06-03T08:36:27.715 INFO:teuthology.orchestra.run.smithi088.stderr:2024-06-03T08:36:27.712+0000 7fa3447b2640 10 monclient: _send_command 1 [{"prefix": "get_command_descriptions"}] 2024-06-03T08:36:27.715 INFO:teuthology.orchestra.run.smithi088.stderr:2024-06-03T08:36:27.712+0000 7fa3447b2640 10 monclient: _send_mon_message to mon.b at v1:172.21.15.192:6789/0 2024-06-03T08:36:27.715 INFO:teuthology.orchestra.run.smithi088.stderr:2024-06-03T08:36:27.712+0000 7fa3447b2640 1 -- 172.21.15.88:0/749058204 --> v1:172.21.15.192:6789/0 -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fa33c04ea60 con 0x7fa33c1393f0 2024-06-03T08:36:27.726 INFO:teuthology.orchestra.run.smithi088.stderr:2024-06-03T08:36:27.727+0000 7fa33affd640 1 -- 172.21.15.88:0/749058204 <== mon.1 v1:172.21.15.192:6789/0 5 ==== config(0 keys) v1 ==== 4+0+0 (unknown 0 0 0) 0x7fa32c0020e0 con 0x7fa33c1393f0 2024-06-03T08:36:27.726 INFO:teuthology.orchestra.run.smithi088.stderr:2024-06-03T08:36:27.727+0000 7fa33affd640 10 monclient: handle_config config(0 keys) v1 2024-06-03T08:36:27.727 INFO:teuthology.orchestra.run.smithi088.stderr:2024-06-03T08:36:27.727+0000 7fa33affd640 1 -- 172.21.15.88:0/749058204 <== mon.1 v1:172.21.15.192:6789/0 6 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 2234374065 0 0) 0x7fa32c003dc0 con 0x7fa33c1393f0 2024-06-03T08:36:27.727 INFO:teuthology.orchestra.run.smithi088.stderr:2024-06-03T08:36:27.727+0000 7fa33affd640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-06-03T08:36:27.727 INFO:teuthology.orchestra.run.smithi088.stderr:2024-06-03T08:36:27.727+0000 7fa33affd640 10 monclient: got monmap 1 from mon.b (according to old e1) 2024-06-03T08:36:27.727 INFO:teuthology.orchestra.run.smithi088.stderr:2024-06-03T08:36:27.727+0000 7fa33affd640 10 monclient: dump: 2024-06-03T08:36:27.727 INFO:teuthology.orchestra.run.smithi088.stderr:epoch 1 2024-06-03T08:36:27.727 INFO:teuthology.orchestra.run.smithi088.stderr:fsid ec3c3fdd-3ac6-4d7d-9412-683130ed8b04 2024-06-03T08:36:27.727 INFO:teuthology.orchestra.run.smithi088.stderr:last_changed 2024-06-03T08:34:31.885125+0000 2024-06-03T08:36:27.727 INFO:teuthology.orchestra.run.smithi088.stderr:created 2024-06-03T08:34:31.885125+0000 2024-06-03T08:36:27.727 INFO:teuthology.orchestra.run.smithi088.stderr:min_mon_release 18 (reef) 2024-06-03T08:36:27.727 INFO:teuthology.orchestra.run.smithi088.stderr:election_strategy: 3 2024-06-03T08:36:27.727 INFO:teuthology.orchestra.run.smithi088.stderr:0: v1:172.21.15.88:6789/0 mon.a 2024-06-03T08:36:27.728 INFO:teuthology.orchestra.run.smithi088.stderr:1: v1:172.21.15.192:6789/0 mon.b 2024-06-03T08:36:27.728 INFO:teuthology.orchestra.run.smithi088.stderr:2: v1:172.21.15.88:6790/0 mon.c 2024-06-03T08:36:27.728 INFO:teuthology.orchestra.run.smithi088.stderr: 2024-06-03T08:36:27.728 INFO:teuthology.orchestra.run.smithi088.stderr:2024-06-03T08:36:27.727+0000 7fa340d4b640 4 set_mon_vals no callback set 2024-06-03T08:36:27.729 INFO:teuthology.orchestra.run.smithi088.stderr:2024-06-03T08:36:27.727+0000 7fa33affd640 1 -- 172.21.15.88:0/749058204 <== mon.1 v1:172.21.15.192:6789/0 7 ==== mgrmap(e 5) v1 ==== 89110+0+0 (unknown 873533973 0 0) 0x7fa32c019b40 con 0x7fa33c1393f0 2024-06-03T08:36:27.729 INFO:teuthology.orchestra.run.smithi088.stderr:2024-06-03T08:36:27.727+0000 7fa34254e640 10 monclient: get_auth_request con 0x7fa32806d2a0 auth_method 0 2024-06-03T08:36:27.729 INFO:teuthology.orchestra.run.smithi088.stderr:2024-06-03T08:36:27.728+0000 7fa33affd640 10 client.4267.objecter ms_handle_connect 0x7fa32806d2a0 2024-06-03T08:36:27.729 INFO:teuthology.orchestra.run.smithi088.stderr:2024-06-03T08:36:27.728+0000 7fa33affd640 1 -- 172.21.15.88:0/749058204 <== mon.1 v1:172.21.15.192:6789/0 8 ==== osd_map(22..22 src has 1..22) v4 ==== 3737+0+0 (unknown 2958406083 0 0) 0x7fa32c053230 con 0x7fa33c1393f0 2024-06-03T08:36:27.729 INFO:teuthology.orchestra.run.smithi088.stderr:2024-06-03T08:36:27.728+0000 7fa33affd640 10 client.4267.objecter ms_dispatch 0x7fa33c000b90 osd_map(22..22 src has 1..22) v4 2024-06-03T08:36:27.729 INFO:teuthology.orchestra.run.smithi088.stderr:2024-06-03T08:36:27.728+0000 7fa33affd640 3 client.4267.objecter handle_osd_map got epochs [22,22] > 0 2024-06-03T08:36:27.729 INFO:teuthology.orchestra.run.smithi088.stderr:2024-06-03T08:36:27.728+0000 7fa33affd640 3 client.4267.objecter handle_osd_map decoding full epoch 22 2024-06-03T08:36:27.729 INFO:teuthology.orchestra.run.smithi088.stderr:2024-06-03T08:36:27.729+0000 7fa33affd640 20 client.4267.objecter dump_active .. 0 homeless 2024-06-03T08:36:27.964 INFO:teuthology.orchestra.run.smithi088.stderr:2024-06-03T08:36:27.964+0000 7fa33affd640 1 -- 172.21.15.88:0/749058204 <== mon.1 v1:172.21.15.192:6789/0 9 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+182498 (unknown 1092875540 0 3016760700) 0x7fa32c04ede0 con 0x7fa33c1393f0 2024-06-03T08:36:27.964 INFO:teuthology.orchestra.run.smithi088.stderr:2024-06-03T08:36:27.964+0000 7fa33affd640 10 monclient: handle_mon_command_ack 1 [{"prefix": "get_command_descriptions"}] 2024-06-03T08:36:27.964 INFO:teuthology.orchestra.run.smithi088.stderr:2024-06-03T08:36:27.964+0000 7fa33affd640 10 monclient: _finish_command 1 = system:0 2024-06-03T08:36:28.138 INFO:teuthology.orchestra.run.smithi088.stderr:2024-06-03T08:36:28.138+0000 7fa3447b2640 10 start_mon_command cmd=[{"prefix": "osd pool create", "pool": "rbd", "pg_num": 8}] 2024-06-03T08:36:28.138 INFO:teuthology.orchestra.run.smithi088.stderr:2024-06-03T08:36:28.138+0000 7fa3447b2640 10 monclient: _send_command 2 [{"prefix": "osd pool create", "pool": "rbd", "pg_num": 8}] 2024-06-03T08:36:28.138 INFO:teuthology.orchestra.run.smithi088.stderr:2024-06-03T08:36:28.138+0000 7fa3447b2640 10 monclient: _send_mon_message to mon.b at v1:172.21.15.192:6789/0 2024-06-03T08:36:28.138 INFO:teuthology.orchestra.run.smithi088.stderr:2024-06-03T08:36:28.138+0000 7fa3447b2640 1 -- 172.21.15.88:0/749058204 --> v1:172.21.15.192:6789/0 -- mon_command({"prefix": "osd pool create", "pool": "rbd", "pg_num": 8} v 0) v1 -- 0x7fa33c1d4770 con 0x7fa33c1393f0 2024-06-03T08:36:28.588 INFO:teuthology.orchestra.run.smithi088.stderr:2024-06-03T08:36:28.589+0000 7fa33affd640 1 -- 172.21.15.88:0/749058204 <== mon.1 v1:172.21.15.192:6789/0 10 ==== mon_command_ack([{"prefix": "osd pool create", "pool": "rbd", "pg_num": 8}]=0 pool 'rbd' created v23) v1 ==== 109+0+0 (unknown 2091248979 0 0) 0x7fa32c050d10 con 0x7fa33c1393f0 2024-06-03T08:36:28.588 INFO:teuthology.orchestra.run.smithi088.stderr:2024-06-03T08:36:28.589+0000 7fa33affd640 10 monclient: handle_mon_command_ack 2 [{"prefix": "osd pool create", "pool": "rbd", "pg_num": 8}] 2024-06-03T08:36:28.589 INFO:teuthology.orchestra.run.smithi088.stderr:2024-06-03T08:36:28.589+0000 7fa33affd640 10 monclient: _finish_command 2 = system:0 pool 'rbd' created 2024-06-03T08:36:28.589 INFO:teuthology.orchestra.run.smithi088.stderr:pool 'rbd' created 2024-06-03T08:36:28.591 INFO:teuthology.orchestra.run.smithi088.stderr:2024-06-03T08:36:28.590+0000 7fa3447b2640 10 librados: watch_flush enter 2024-06-03T08:36:28.591 INFO:teuthology.orchestra.run.smithi088.stderr:2024-06-03T08:36:28.590+0000 7fa3447b2640 10 librados: watch_flush exit 2024-06-03T08:36:28.591 INFO:teuthology.orchestra.run.smithi088.stderr:2024-06-03T08:36:28.590+0000 7fa3447b2640 20 client.4267.objecter shutdown clearing up homeless session... 2024-06-03T08:36:28.591 INFO:teuthology.orchestra.run.smithi088.stderr:2024-06-03T08:36:28.590+0000 7fa3447b2640 10 client.4267.objecter successfully canceled tick 2024-06-03T08:36:28.591 INFO:teuthology.orchestra.run.smithi088.stderr:2024-06-03T08:36:28.590+0000 7fa3447b2640 1 -- 172.21.15.88:0/749058204 >> v1:172.21.15.192:6800/34229 conn(0x7fa32806d2a0 legacy=0x7fa32806f760 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-06-03T08:36:28.591 INFO:teuthology.orchestra.run.smithi088.stderr:2024-06-03T08:36:28.590+0000 7fa3447b2640 10 monclient: shutdown 2024-06-03T08:36:28.591 INFO:teuthology.orchestra.run.smithi088.stderr:2024-06-03T08:36:28.590+0000 7fa3447b2640 20 monclient: shutdown discarding 0 pending message(s) 2024-06-03T08:36:28.591 INFO:teuthology.orchestra.run.smithi088.stderr:2024-06-03T08:36:28.590+0000 7fa3447b2640 1 -- 172.21.15.88:0/749058204 >> v1:172.21.15.192:6789/0 conn(0x7fa33c1393f0 legacy=0x7fa33c135320 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-06-03T08:36:28.591 INFO:teuthology.orchestra.run.smithi088.stderr:2024-06-03T08:36:28.590+0000 7fa3447b2640 1 -- 172.21.15.88:0/749058204 shutdown_connections 2024-06-03T08:36:28.591 INFO:teuthology.orchestra.run.smithi088.stderr:2024-06-03T08:36:28.590+0000 7fa3447b2640 1 -- 172.21.15.88:0/749058204 >> 172.21.15.88:0/749058204 conn(0x7fa33c0b2f20 msgr2=0x7fa33c13f100 unknown :-1 s=STATE_NONE l=0).mark_down 2024-06-03T08:36:28.591 INFO:teuthology.orchestra.run.smithi088.stderr:2024-06-03T08:36:28.590+0000 7fa3447b2640 1 -- 172.21.15.88:0/749058204 shutdown_connections 2024-06-03T08:36:28.591 INFO:teuthology.orchestra.run.smithi088.stderr:2024-06-03T08:36:28.590+0000 7fa3447b2640 1 -- 172.21.15.88:0/749058204 wait complete. 2024-06-03T08:36:28.592 INFO:teuthology.orchestra.run.smithi088.stderr:2024-06-03T08:36:28.590+0000 7fa3447b2640 1 librados: shutdown 2024-06-03T08:36:28.601 DEBUG:teuthology.orchestra.run.smithi088:> sudo ceph --cluster ceph osd pool application enable rbd rbd --yes-i-really-mean-it 2024-06-03T08:36:28.704 INFO:teuthology.orchestra.run.smithi088.stderr:2024-06-03T08:36:28.705+0000 7fd6049d7640 10 monclient: get_monmap_and_config 2024-06-03T08:36:28.704 INFO:teuthology.orchestra.run.smithi088.stderr:2024-06-03T08:36:28.705+0000 7fd6049d7640 10 monclient: build_initial_monmap 2024-06-03T08:36:28.704 INFO:teuthology.orchestra.run.smithi088.stderr:2024-06-03T08:36:28.705+0000 7fd6049d7640 10 monclient: monmap: 2024-06-03T08:36:28.704 INFO:teuthology.orchestra.run.smithi088.stderr:epoch 0 2024-06-03T08:36:28.704 INFO:teuthology.orchestra.run.smithi088.stderr:fsid ec3c3fdd-3ac6-4d7d-9412-683130ed8b04 2024-06-03T08:36:28.704 INFO:teuthology.orchestra.run.smithi088.stderr:last_changed 2024-06-03T08:36:28.705143+0000 2024-06-03T08:36:28.705 INFO:teuthology.orchestra.run.smithi088.stderr:created 2024-06-03T08:36:28.705143+0000 2024-06-03T08:36:28.705 INFO:teuthology.orchestra.run.smithi088.stderr:min_mon_release 0 (unknown) 2024-06-03T08:36:28.705 INFO:teuthology.orchestra.run.smithi088.stderr:election_strategy: 3 2024-06-03T08:36:28.705 INFO:teuthology.orchestra.run.smithi088.stderr:0: v1:172.21.15.88:6789/0 mon.noname-a 2024-06-03T08:36:28.705 INFO:teuthology.orchestra.run.smithi088.stderr:1: v1:172.21.15.192:6789/0 mon.noname-c 2024-06-03T08:36:28.705 INFO:teuthology.orchestra.run.smithi088.stderr:2: v1:172.21.15.88:6790/0 mon.noname-b 2024-06-03T08:36:28.705 INFO:teuthology.orchestra.run.smithi088.stderr: 2024-06-03T08:36:28.706 INFO:teuthology.orchestra.run.smithi088.stderr:2024-06-03T08:36:28.705+0000 7fd6049d7640 1 Processor -- start 2024-06-03T08:36:28.706 INFO:teuthology.orchestra.run.smithi088.stderr:2024-06-03T08:36:28.705+0000 7fd6049d7640 1 -- start start 2024-06-03T08:36:28.706 INFO:teuthology.orchestra.run.smithi088.stderr:2024-06-03T08:36:28.705+0000 7fd6049d7640 10 monclient: init 2024-06-03T08:36:28.707 INFO:teuthology.orchestra.run.smithi088.stderr:2024-06-03T08:36:28.706+0000 7fd6049d7640 10 monclient: _reopen_session rank -1 2024-06-03T08:36:28.707 INFO:teuthology.orchestra.run.smithi088.stderr:2024-06-03T08:36:28.706+0000 7fd6049d7640 10 monclient: _add_conns ranks=[0,2,1] 2024-06-03T08:36:28.707 INFO:teuthology.orchestra.run.smithi088.stderr:2024-06-03T08:36:28.706+0000 7fd6049d7640 10 monclient(hunting): picked mon.noname-a con 0x7fd6000a8a60 addr v1:172.21.15.88:6789/0 2024-06-03T08:36:28.707 INFO:teuthology.orchestra.run.smithi088.stderr:2024-06-03T08:36:28.706+0000 7fd6049d7640 10 monclient(hunting): picked mon.noname-b con 0x7fd6000b1b70 addr v1:172.21.15.88:6790/0 2024-06-03T08:36:28.707 INFO:teuthology.orchestra.run.smithi088.stderr:2024-06-03T08:36:28.706+0000 7fd6049d7640 10 monclient(hunting): picked mon.noname-c con 0x7fd60014ae00 addr v1:172.21.15.192:6789/0 2024-06-03T08:36:28.707 INFO:teuthology.orchestra.run.smithi088.stderr:2024-06-03T08:36:28.706+0000 7fd6049d7640 1 -- --> v1:172.21.15.88:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fd6000b18c0 con 0x7fd6000a8a60 2024-06-03T08:36:28.707 INFO:teuthology.orchestra.run.smithi088.stderr:2024-06-03T08:36:28.706+0000 7fd6049d7640 1 -- --> v1:172.21.15.192:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fd6000a92a0 con 0x7fd60014ae00 2024-06-03T08:36:28.707 INFO:teuthology.orchestra.run.smithi088.stderr:2024-06-03T08:36:28.706+0000 7fd6049d7640 1 -- --> v1:172.21.15.88:6790/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fd6000b1fb0 con 0x7fd6000b1b70 2024-06-03T08:36:28.707 INFO:teuthology.orchestra.run.smithi088.stderr:2024-06-03T08:36:28.706+0000 7fd6049d7640 10 monclient(hunting): _renew_subs 2024-06-03T08:36:28.707 INFO:teuthology.orchestra.run.smithi088.stderr:2024-06-03T08:36:28.706+0000 7fd6049d7640 10 monclient(hunting): authenticate will time out at 1244.515605s 2024-06-03T08:36:28.707 INFO:teuthology.orchestra.run.smithi088.stderr:2024-06-03T08:36:28.707+0000 7fd5fdd9b640 1 --1- >> v1:172.21.15.88:6790/0 conn(0x7fd6000b1b70 0x7fd60013b140 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.88:6790/0 says I am v1:172.21.15.88:54024/0 (socket says 172.21.15.88:54024) 2024-06-03T08:36:28.707 INFO:teuthology.orchestra.run.smithi088.stderr:2024-06-03T08:36:28.707+0000 7fd5fdd9b640 1 -- 172.21.15.88:0/3951537229 learned_addr learned my addr 172.21.15.88:0/3951537229 (peer_addr_for_me v1:172.21.15.88:0/0) 2024-06-03T08:36:28.708 INFO:teuthology.orchestra.run.smithi088.stderr:2024-06-03T08:36:28.708+0000 7fd5fd59a640 1 -- 172.21.15.88:0/3951537229 <== mon.1 v1:172.21.15.192:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 1294532780 0 0) 0x7fd6000a92a0 con 0x7fd60014ae00 2024-06-03T08:36:28.708 INFO:teuthology.orchestra.run.smithi088.stderr:2024-06-03T08:36:28.708+0000 7fd5fd59a640 10 monclient(hunting): _negotiate 2024-06-03T08:36:28.708 INFO:teuthology.orchestra.run.smithi088.stderr:2024-06-03T08:36:28.708+0000 7fd5fd59a640 10 monclient(hunting): _init_auth method 2 2024-06-03T08:36:28.708 INFO:teuthology.orchestra.run.smithi088.stderr:2024-06-03T08:36:28.708+0000 7fd5fd59a640 10 monclient(hunting): _init_auth creating new auth 2024-06-03T08:36:28.709 INFO:teuthology.orchestra.run.smithi088.stderr:2024-06-03T08:36:28.708+0000 7fd5fd59a640 10 monclient(hunting): my global_id is 4270 2024-06-03T08:36:28.709 INFO:teuthology.orchestra.run.smithi088.stderr:2024-06-03T08:36:28.708+0000 7fd5fd59a640 1 -- 172.21.15.88:0/3951537229 --> v1:172.21.15.192:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7fd5e8003520 con 0x7fd60014ae00 2024-06-03T08:36:28.709 INFO:teuthology.orchestra.run.smithi088.stderr:2024-06-03T08:36:28.710+0000 7fd5fd59a640 1 -- 172.21.15.88:0/3951537229 <== mon.2 v1:172.21.15.88:6790/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 1354352743 0 0) 0x7fd6000b1fb0 con 0x7fd6000b1b70 2024-06-03T08:36:28.709 INFO:teuthology.orchestra.run.smithi088.stderr:2024-06-03T08:36:28.710+0000 7fd5fd59a640 10 monclient(hunting): _negotiate 2024-06-03T08:36:28.709 INFO:teuthology.orchestra.run.smithi088.stderr:2024-06-03T08:36:28.710+0000 7fd5fd59a640 10 monclient(hunting): _init_auth method 2 2024-06-03T08:36:28.709 INFO:teuthology.orchestra.run.smithi088.stderr:2024-06-03T08:36:28.710+0000 7fd5fd59a640 10 monclient(hunting): _init_auth creating new auth 2024-06-03T08:36:28.709 INFO:teuthology.orchestra.run.smithi088.stderr:2024-06-03T08:36:28.710+0000 7fd5fd59a640 10 monclient(hunting): my global_id is 4289 2024-06-03T08:36:28.710 INFO:teuthology.orchestra.run.smithi088.stderr:2024-06-03T08:36:28.710+0000 7fd5fd59a640 1 -- 172.21.15.88:0/3951537229 --> v1:172.21.15.88:6790/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7fd6000a92a0 con 0x7fd6000b1b70 2024-06-03T08:36:28.712 INFO:teuthology.orchestra.run.smithi088.stderr:2024-06-03T08:36:28.712+0000 7fd5fd59a640 1 -- 172.21.15.88:0/3951537229 <== mon.1 v1:172.21.15.192:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 214+0+0 (unknown 738113845 0 0) 0x7fd5e8003520 con 0x7fd60014ae00 2024-06-03T08:36:28.712 INFO:teuthology.orchestra.run.smithi088.stderr:2024-06-03T08:36:28.712+0000 7fd5fd59a640 1 -- 172.21.15.88:0/3951537229 >> v1:172.21.15.88:6790/0 conn(0x7fd6000b1b70 legacy=0x7fd60013b140 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-06-03T08:36:28.712 INFO:teuthology.orchestra.run.smithi088.stderr:2024-06-03T08:36:28.712+0000 7fd5fd59a640 1 -- 172.21.15.88:0/3951537229 >> v1:172.21.15.88:6789/0 conn(0x7fd6000a8a60 legacy=0x7fd600141d70 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-06-03T08:36:28.712 INFO:teuthology.orchestra.run.smithi088.stderr:2024-06-03T08:36:28.712+0000 7fd5fd59a640 10 monclient: _finish_hunting 0 2024-06-03T08:36:28.712 INFO:teuthology.orchestra.run.smithi088.stderr:2024-06-03T08:36:28.712+0000 7fd5fd59a640 1 monclient: found mon.noname-c 2024-06-03T08:36:28.712 INFO:teuthology.orchestra.run.smithi088.stderr:2024-06-03T08:36:28.712+0000 7fd5fd59a640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-06-03T08:36:28.712 INFO:teuthology.orchestra.run.smithi088.stderr:2024-06-03T08:36:28.712+0000 7fd5fd59a640 10 monclient: _send_mon_message to mon.noname-c at v1:172.21.15.192:6789/0 2024-06-03T08:36:28.713 INFO:teuthology.orchestra.run.smithi088.stderr:2024-06-03T08:36:28.712+0000 7fd5fd59a640 1 -- 172.21.15.88:0/3951537229 --> v1:172.21.15.192:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fd6000aa070 con 0x7fd60014ae00 2024-06-03T08:36:28.713 INFO:teuthology.orchestra.run.smithi088.stderr:2024-06-03T08:36:28.712+0000 7fd5fd59a640 10 monclient: _finish_auth 0 2024-06-03T08:36:28.713 INFO:teuthology.orchestra.run.smithi088.stderr:2024-06-03T08:36:28.712+0000 7fd5fd59a640 10 monclient: _check_auth_tickets 2024-06-03T08:36:28.713 INFO:teuthology.orchestra.run.smithi088.stderr:2024-06-03T08:36:28.712+0000 7fd5fd59a640 20 monclient: _check_auth_rotating not needed by client.admin 2024-06-03T08:36:28.713 INFO:teuthology.orchestra.run.smithi088.stderr:2024-06-03T08:36:28.712+0000 7fd5fd59a640 1 -- 172.21.15.88:0/3951537229 <== mon.1 v1:172.21.15.192:6789/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 2234374065 0 0) 0x7fd5f8001990 con 0x7fd60014ae00 2024-06-03T08:36:28.713 INFO:teuthology.orchestra.run.smithi088.stderr:2024-06-03T08:36:28.712+0000 7fd5fd59a640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-06-03T08:36:28.713 INFO:teuthology.orchestra.run.smithi088.stderr:2024-06-03T08:36:28.712+0000 7fd5fd59a640 10 monclient: got monmap 1 from mon.noname-c (according to old e1) 2024-06-03T08:36:28.713 INFO:teuthology.orchestra.run.smithi088.stderr:2024-06-03T08:36:28.712+0000 7fd5fd59a640 10 monclient: dump: 2024-06-03T08:36:28.713 INFO:teuthology.orchestra.run.smithi088.stderr:epoch 1 2024-06-03T08:36:28.713 INFO:teuthology.orchestra.run.smithi088.stderr:fsid ec3c3fdd-3ac6-4d7d-9412-683130ed8b04 2024-06-03T08:36:28.713 INFO:teuthology.orchestra.run.smithi088.stderr:last_changed 2024-06-03T08:34:31.885125+0000 2024-06-03T08:36:28.713 INFO:teuthology.orchestra.run.smithi088.stderr:created 2024-06-03T08:34:31.885125+0000 2024-06-03T08:36:28.713 INFO:teuthology.orchestra.run.smithi088.stderr:min_mon_release 18 (reef) 2024-06-03T08:36:28.713 INFO:teuthology.orchestra.run.smithi088.stderr:election_strategy: 3 2024-06-03T08:36:28.713 INFO:teuthology.orchestra.run.smithi088.stderr:0: v1:172.21.15.88:6789/0 mon.a 2024-06-03T08:36:28.714 INFO:teuthology.orchestra.run.smithi088.stderr:1: v1:172.21.15.192:6789/0 mon.b 2024-06-03T08:36:28.714 INFO:teuthology.orchestra.run.smithi088.stderr:2: v1:172.21.15.88:6790/0 mon.c 2024-06-03T08:36:28.714 INFO:teuthology.orchestra.run.smithi088.stderr: 2024-06-03T08:36:28.714 INFO:teuthology.orchestra.run.smithi088.stderr:2024-06-03T08:36:28.712+0000 7fd6049d7640 5 monclient: authenticate success, global_id 4270 2024-06-03T08:36:28.714 INFO:teuthology.orchestra.run.smithi088.stderr:2024-06-03T08:36:28.712+0000 7fd6049d7640 20 monclient: get_monmap_and_config waiting for monmap|config 2024-06-03T08:36:28.716 INFO:teuthology.orchestra.run.smithi088.stderr:2024-06-03T08:36:28.714+0000 7fd5fd59a640 1 -- 172.21.15.88:0/3951537229 <== mon.1 v1:172.21.15.192:6789/0 4 ==== config(0 keys) v1 ==== 4+0+0 (unknown 0 0 0) 0x7fd5f8001bb0 con 0x7fd60014ae00 2024-06-03T08:36:28.716 INFO:teuthology.orchestra.run.smithi088.stderr:2024-06-03T08:36:28.714+0000 7fd5fd59a640 10 monclient: handle_config config(0 keys) v1 2024-06-03T08:36:28.716 INFO:teuthology.orchestra.run.smithi088.stderr:2024-06-03T08:36:28.714+0000 7fd5fd59a640 1 -- 172.21.15.88:0/3951537229 <== mon.1 v1:172.21.15.192:6789/0 5 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 2234374065 0 0) 0x7fd5f8001ec0 con 0x7fd60014ae00 2024-06-03T08:36:28.716 INFO:teuthology.orchestra.run.smithi088.stderr:2024-06-03T08:36:28.714+0000 7fd5fd59a640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-06-03T08:36:28.716 INFO:teuthology.orchestra.run.smithi088.stderr:2024-06-03T08:36:28.714+0000 7fd5fd59a640 10 monclient: got monmap 1 from mon.b (according to old e1) 2024-06-03T08:36:28.716 INFO:teuthology.orchestra.run.smithi088.stderr:2024-06-03T08:36:28.714+0000 7fd5fd59a640 10 monclient: dump: 2024-06-03T08:36:28.716 INFO:teuthology.orchestra.run.smithi088.stderr:epoch 1 2024-06-03T08:36:28.716 INFO:teuthology.orchestra.run.smithi088.stderr:fsid ec3c3fdd-3ac6-4d7d-9412-683130ed8b04 2024-06-03T08:36:28.716 INFO:teuthology.orchestra.run.smithi088.stderr:last_changed 2024-06-03T08:34:31.885125+0000 2024-06-03T08:36:28.716 INFO:teuthology.orchestra.run.smithi088.stderr:created 2024-06-03T08:34:31.885125+0000 2024-06-03T08:36:28.716 INFO:teuthology.orchestra.run.smithi088.stderr:min_mon_release 18 (reef) 2024-06-03T08:36:28.716 INFO:teuthology.orchestra.run.smithi088.stderr:election_strategy: 3 2024-06-03T08:36:28.716 INFO:teuthology.orchestra.run.smithi088.stderr:0: v1:172.21.15.88:6789/0 mon.a 2024-06-03T08:36:28.717 INFO:teuthology.orchestra.run.smithi088.stderr:1: v1:172.21.15.192:6789/0 mon.b 2024-06-03T08:36:28.717 INFO:teuthology.orchestra.run.smithi088.stderr:2: v1:172.21.15.88:6790/0 mon.c 2024-06-03T08:36:28.717 INFO:teuthology.orchestra.run.smithi088.stderr: 2024-06-03T08:36:28.717 INFO:teuthology.orchestra.run.smithi088.stderr:2024-06-03T08:36:28.714+0000 7fd6049d7640 10 monclient: get_monmap_and_config success 2024-06-03T08:36:28.717 INFO:teuthology.orchestra.run.smithi088.stderr:2024-06-03T08:36:28.714+0000 7fd6049d7640 4 set_mon_vals no callback set 2024-06-03T08:36:28.717 INFO:teuthology.orchestra.run.smithi088.stderr:2024-06-03T08:36:28.715+0000 7fd6049d7640 10 monclient: shutdown 2024-06-03T08:36:28.717 INFO:teuthology.orchestra.run.smithi088.stderr:2024-06-03T08:36:28.715+0000 7fd6049d7640 20 monclient: shutdown discarding 0 pending message(s) 2024-06-03T08:36:28.717 INFO:teuthology.orchestra.run.smithi088.stderr:2024-06-03T08:36:28.715+0000 7fd6049d7640 1 -- 172.21.15.88:0/3951537229 >> v1:172.21.15.192:6789/0 conn(0x7fd60014ae00 legacy=0x7fd60014d1f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-06-03T08:36:28.717 INFO:teuthology.orchestra.run.smithi088.stderr:2024-06-03T08:36:28.715+0000 7fd6049d7640 1 -- 172.21.15.88:0/3951537229 shutdown_connections 2024-06-03T08:36:28.717 INFO:teuthology.orchestra.run.smithi088.stderr:2024-06-03T08:36:28.715+0000 7fd6049d7640 1 -- 172.21.15.88:0/3951537229 >> 172.21.15.88:0/3951537229 conn(0x7fd6000b7170 msgr2=0x7fd6000b7580 unknown :-1 s=STATE_NONE l=0).mark_down 2024-06-03T08:36:28.717 INFO:teuthology.orchestra.run.smithi088.stderr:2024-06-03T08:36:28.715+0000 7fd6049d7640 1 -- 172.21.15.88:0/3951537229 shutdown_connections 2024-06-03T08:36:28.717 INFO:teuthology.orchestra.run.smithi088.stderr:2024-06-03T08:36:28.715+0000 7fd6049d7640 1 -- 172.21.15.88:0/3951537229 wait complete. 2024-06-03T08:36:28.717 INFO:teuthology.orchestra.run.smithi088.stderr:2024-06-03T08:36:28.716+0000 7fd6049d7640 10 monclient: build_initial_monmap 2024-06-03T08:36:28.718 INFO:teuthology.orchestra.run.smithi088.stderr:2024-06-03T08:36:28.716+0000 7fd6049d7640 10 monclient: monmap: 2024-06-03T08:36:28.718 INFO:teuthology.orchestra.run.smithi088.stderr:epoch 0 2024-06-03T08:36:28.718 INFO:teuthology.orchestra.run.smithi088.stderr:fsid 00000000-0000-0000-0000-000000000000 2024-06-03T08:36:28.718 INFO:teuthology.orchestra.run.smithi088.stderr:last_changed 0.000000 2024-06-03T08:36:28.718 INFO:teuthology.orchestra.run.smithi088.stderr:created 0.000000 2024-06-03T08:36:28.718 INFO:teuthology.orchestra.run.smithi088.stderr:min_mon_release 0 (unknown) 2024-06-03T08:36:28.718 INFO:teuthology.orchestra.run.smithi088.stderr:election_strategy: 1 2024-06-03T08:36:28.718 INFO:teuthology.orchestra.run.smithi088.stderr:0: v1:172.21.15.88:6789/0 mon.noname-a 2024-06-03T08:36:28.718 INFO:teuthology.orchestra.run.smithi088.stderr:1: v1:172.21.15.192:6789/0 mon.noname-b 2024-06-03T08:36:28.718 INFO:teuthology.orchestra.run.smithi088.stderr:2: v1:172.21.15.88:6790/0 mon.noname-c 2024-06-03T08:36:28.718 INFO:teuthology.orchestra.run.smithi088.stderr: 2024-06-03T08:36:28.718 INFO:teuthology.orchestra.run.smithi088.stderr:2024-06-03T08:36:28.716+0000 7fd6049d7640 1 librados: starting msgr at 2024-06-03T08:36:28.719 INFO:teuthology.orchestra.run.smithi088.stderr:2024-06-03T08:36:28.716+0000 7fd6049d7640 1 librados: starting objecter 2024-06-03T08:36:28.719 INFO:teuthology.orchestra.run.smithi088.stderr:2024-06-03T08:36:28.716+0000 7fd6049d7640 1 Processor -- start 2024-06-03T08:36:28.719 INFO:teuthology.orchestra.run.smithi088.stderr:2024-06-03T08:36:28.716+0000 7fd6049d7640 1 -- start start 2024-06-03T08:36:28.719 INFO:teuthology.orchestra.run.smithi088.stderr:2024-06-03T08:36:28.716+0000 7fd6049d7640 1 librados: setting wanted keys 2024-06-03T08:36:28.719 INFO:teuthology.orchestra.run.smithi088.stderr:2024-06-03T08:36:28.716+0000 7fd6049d7640 1 librados: calling monclient init 2024-06-03T08:36:28.719 INFO:teuthology.orchestra.run.smithi088.stderr:2024-06-03T08:36:28.716+0000 7fd6049d7640 10 monclient: init 2024-06-03T08:36:28.721 INFO:teuthology.orchestra.run.smithi088.stderr:2024-06-03T08:36:28.717+0000 7fd6049d7640 10 monclient: _reopen_session rank -1 2024-06-03T08:36:28.721 INFO:teuthology.orchestra.run.smithi088.stderr:2024-06-03T08:36:28.717+0000 7fd6049d7640 10 monclient: _add_conns ranks=[0,1,2] 2024-06-03T08:36:28.721 INFO:teuthology.orchestra.run.smithi088.stderr:2024-06-03T08:36:28.717+0000 7fd6049d7640 10 monclient(hunting): picked mon.noname-a con 0x7fd6000a8a60 addr v1:172.21.15.88:6789/0 2024-06-03T08:36:28.721 INFO:teuthology.orchestra.run.smithi088.stderr:2024-06-03T08:36:28.717+0000 7fd6049d7640 10 monclient(hunting): picked mon.noname-b con 0x7fd6000b1b70 addr v1:172.21.15.192:6789/0 2024-06-03T08:36:28.721 INFO:teuthology.orchestra.run.smithi088.stderr:2024-06-03T08:36:28.717+0000 7fd6049d7640 10 monclient(hunting): picked mon.noname-c con 0x7fd60014ae00 addr v1:172.21.15.88:6790/0 2024-06-03T08:36:28.721 INFO:teuthology.orchestra.run.smithi088.stderr:2024-06-03T08:36:28.717+0000 7fd6049d7640 1 -- --> v1:172.21.15.88:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fd60013dfa0 con 0x7fd6000a8a60 2024-06-03T08:36:28.721 INFO:teuthology.orchestra.run.smithi088.stderr:2024-06-03T08:36:28.717+0000 7fd6049d7640 1 -- --> v1:172.21.15.192:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fd60013d050 con 0x7fd6000b1b70 2024-06-03T08:36:28.721 INFO:teuthology.orchestra.run.smithi088.stderr:2024-06-03T08:36:28.717+0000 7fd6049d7640 1 -- --> v1:172.21.15.88:6790/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fd60013d380 con 0x7fd60014ae00 2024-06-03T08:36:28.721 INFO:teuthology.orchestra.run.smithi088.stderr:2024-06-03T08:36:28.717+0000 7fd6049d7640 10 monclient(hunting): _renew_subs 2024-06-03T08:36:28.721 INFO:teuthology.orchestra.run.smithi088.stderr:2024-06-03T08:36:28.717+0000 7fd6049d7640 10 monclient(hunting): authenticate will time out at 1244.526432s 2024-06-03T08:36:28.721 INFO:teuthology.orchestra.run.smithi088.stderr:2024-06-03T08:36:28.717+0000 7fd5fdd9b640 1 --1- >> v1:172.21.15.192:6789/0 conn(0x7fd6000b1b70 0x7fd6001da050 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=1).handle_server_banner_and_identify peer v1:172.21.15.192:6789/0 says I am v1:172.21.15.88:52248/0 (socket says 172.21.15.88:52248) 2024-06-03T08:36:28.721 INFO:teuthology.orchestra.run.smithi088.stderr:2024-06-03T08:36:28.717+0000 7fd5fdd9b640 1 -- 172.21.15.88:0/2232032709 learned_addr learned my addr 172.21.15.88:0/2232032709 (peer_addr_for_me v1:172.21.15.88:0/0) 2024-06-03T08:36:28.721 INFO:teuthology.orchestra.run.smithi088.stderr:2024-06-03T08:36:28.718+0000 7fd5eeffd640 10 client.?.objecter ms_handle_connect 0x7fd6000b1b70 2024-06-03T08:36:28.722 INFO:teuthology.orchestra.run.smithi088.stderr:2024-06-03T08:36:28.718+0000 7fd5eeffd640 10 client.?.objecter resend_mon_ops 2024-06-03T08:36:28.722 INFO:teuthology.orchestra.run.smithi088.stderr:2024-06-03T08:36:28.718+0000 7fd5eeffd640 10 client.?.objecter ms_handle_connect 0x7fd60014ae00 2024-06-03T08:36:28.722 INFO:teuthology.orchestra.run.smithi088.stderr:2024-06-03T08:36:28.718+0000 7fd5eeffd640 10 client.?.objecter resend_mon_ops 2024-06-03T08:36:28.722 INFO:teuthology.orchestra.run.smithi088.stderr:2024-06-03T08:36:28.720+0000 7fd5eeffd640 1 -- 172.21.15.88:0/2232032709 <== mon.1 v1:172.21.15.192:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 3050415238 0 0) 0x7fd60013d050 con 0x7fd6000b1b70 2024-06-03T08:36:28.722 INFO:teuthology.orchestra.run.smithi088.stderr:2024-06-03T08:36:28.720+0000 7fd5eeffd640 10 monclient(hunting): _negotiate 2024-06-03T08:36:28.722 INFO:teuthology.orchestra.run.smithi088.stderr:2024-06-03T08:36:28.720+0000 7fd5eeffd640 10 monclient(hunting): _init_auth method 2 2024-06-03T08:36:28.722 INFO:teuthology.orchestra.run.smithi088.stderr:2024-06-03T08:36:28.720+0000 7fd5eeffd640 10 monclient(hunting): _init_auth creating new auth 2024-06-03T08:36:28.722 INFO:teuthology.orchestra.run.smithi088.stderr:2024-06-03T08:36:28.720+0000 7fd5eeffd640 10 monclient(hunting): my global_id is 4273 2024-06-03T08:36:28.722 INFO:teuthology.orchestra.run.smithi088.stderr:2024-06-03T08:36:28.720+0000 7fd5eeffd640 1 -- 172.21.15.88:0/2232032709 --> v1:172.21.15.192:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7fd5e0004eb0 con 0x7fd6000b1b70 2024-06-03T08:36:28.722 INFO:teuthology.orchestra.run.smithi088.stderr:2024-06-03T08:36:28.721+0000 7fd5eeffd640 1 -- 172.21.15.88:0/2232032709 <== mon.2 v1:172.21.15.88:6790/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 3013662760 0 0) 0x7fd60013d380 con 0x7fd60014ae00 2024-06-03T08:36:28.722 INFO:teuthology.orchestra.run.smithi088.stderr:2024-06-03T08:36:28.721+0000 7fd5eeffd640 10 monclient(hunting): _negotiate 2024-06-03T08:36:28.722 INFO:teuthology.orchestra.run.smithi088.stderr:2024-06-03T08:36:28.721+0000 7fd5eeffd640 10 monclient(hunting): _init_auth method 2 2024-06-03T08:36:28.722 INFO:teuthology.orchestra.run.smithi088.stderr:2024-06-03T08:36:28.721+0000 7fd5eeffd640 10 monclient(hunting): _init_auth creating new auth 2024-06-03T08:36:28.723 INFO:teuthology.orchestra.run.smithi088.stderr:2024-06-03T08:36:28.721+0000 7fd5eeffd640 10 monclient(hunting): my global_id is 4292 2024-06-03T08:36:28.723 INFO:teuthology.orchestra.run.smithi088.stderr:2024-06-03T08:36:28.721+0000 7fd5eeffd640 1 -- 172.21.15.88:0/2232032709 --> v1:172.21.15.88:6790/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7fd60013d050 con 0x7fd60014ae00 2024-06-03T08:36:28.724 INFO:teuthology.orchestra.run.smithi088.stderr:2024-06-03T08:36:28.724+0000 7fd5eeffd640 1 -- 172.21.15.88:0/2232032709 <== mon.1 v1:172.21.15.192:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 764+0+0 (unknown 1235383809 0 0) 0x7fd5e0004eb0 con 0x7fd6000b1b70 2024-06-03T08:36:28.724 INFO:teuthology.orchestra.run.smithi088.stderr:2024-06-03T08:36:28.724+0000 7fd5eeffd640 1 -- 172.21.15.88:0/2232032709 --> v1:172.21.15.192:6789/0 -- auth(proto 2 165 bytes epoch 0) v1 -- 0x7fd60013d380 con 0x7fd6000b1b70 2024-06-03T08:36:28.724 INFO:teuthology.orchestra.run.smithi088.stderr:2024-06-03T08:36:28.724+0000 7fd5eeffd640 1 -- 172.21.15.88:0/2232032709 <== mon.1 v1:172.21.15.192:6789/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 2234374065 0 0) 0x7fd5f4001910 con 0x7fd6000b1b70 2024-06-03T08:36:28.724 INFO:teuthology.orchestra.run.smithi088.stderr:2024-06-03T08:36:28.724+0000 7fd5eeffd640 10 monclient(hunting): handle_monmap mon_map magic: 0 v1 2024-06-03T08:36:28.724 INFO:teuthology.orchestra.run.smithi088.stderr:2024-06-03T08:36:28.724+0000 7fd5eeffd640 10 monclient(hunting): got monmap 1 from mon.noname-b (according to old e1) 2024-06-03T08:36:28.724 INFO:teuthology.orchestra.run.smithi088.stderr:2024-06-03T08:36:28.724+0000 7fd5eeffd640 10 monclient(hunting): dump: 2024-06-03T08:36:28.724 INFO:teuthology.orchestra.run.smithi088.stderr:epoch 1 2024-06-03T08:36:28.724 INFO:teuthology.orchestra.run.smithi088.stderr:fsid ec3c3fdd-3ac6-4d7d-9412-683130ed8b04 2024-06-03T08:36:28.724 INFO:teuthology.orchestra.run.smithi088.stderr:last_changed 2024-06-03T08:34:31.885125+0000 2024-06-03T08:36:28.725 INFO:teuthology.orchestra.run.smithi088.stderr:created 2024-06-03T08:34:31.885125+0000 2024-06-03T08:36:28.725 INFO:teuthology.orchestra.run.smithi088.stderr:min_mon_release 18 (reef) 2024-06-03T08:36:28.725 INFO:teuthology.orchestra.run.smithi088.stderr:election_strategy: 3 2024-06-03T08:36:28.725 INFO:teuthology.orchestra.run.smithi088.stderr:0: v1:172.21.15.88:6789/0 mon.a 2024-06-03T08:36:28.725 INFO:teuthology.orchestra.run.smithi088.stderr:1: v1:172.21.15.192:6789/0 mon.b 2024-06-03T08:36:28.725 INFO:teuthology.orchestra.run.smithi088.stderr:2: v1:172.21.15.88:6790/0 mon.c 2024-06-03T08:36:28.725 INFO:teuthology.orchestra.run.smithi088.stderr: 2024-06-03T08:36:28.725 INFO:teuthology.orchestra.run.smithi088.stderr:2024-06-03T08:36:28.724+0000 7fd5eeffd640 10 monclient(hunting): _finish_auth 0 2024-06-03T08:36:28.727 INFO:teuthology.orchestra.run.smithi088.stderr:2024-06-03T08:36:28.727+0000 7fd5eeffd640 1 -- 172.21.15.88:0/2232032709 <== mon.2 v1:172.21.15.88:6790/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 764+0+0 (unknown 2618483312 0 0) 0x7fd60013d050 con 0x7fd60014ae00 2024-06-03T08:36:28.727 INFO:teuthology.orchestra.run.smithi088.stderr:2024-06-03T08:36:28.727+0000 7fd5eeffd640 1 -- 172.21.15.88:0/2232032709 --> v1:172.21.15.88:6790/0 -- auth(proto 2 165 bytes epoch 0) v1 -- 0x7fd5e0004eb0 con 0x7fd60014ae00 2024-06-03T08:36:28.727 INFO:teuthology.orchestra.run.smithi088.stderr:2024-06-03T08:36:28.727+0000 7fd5eeffd640 1 -- 172.21.15.88:0/2232032709 <== mon.2 v1:172.21.15.88:6790/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 2234374065 0 0) 0x7fd5f8001ec0 con 0x7fd60014ae00 2024-06-03T08:36:28.727 INFO:teuthology.orchestra.run.smithi088.stderr:2024-06-03T08:36:28.727+0000 7fd5eeffd640 10 monclient(hunting): handle_monmap mon_map magic: 0 v1 2024-06-03T08:36:28.727 INFO:teuthology.orchestra.run.smithi088.stderr:2024-06-03T08:36:28.727+0000 7fd5eeffd640 10 monclient(hunting): got monmap 1 from mon.c (according to old e1) 2024-06-03T08:36:28.727 INFO:teuthology.orchestra.run.smithi088.stderr:2024-06-03T08:36:28.727+0000 7fd5eeffd640 10 monclient(hunting): dump: 2024-06-03T08:36:28.727 INFO:teuthology.orchestra.run.smithi088.stderr:epoch 1 2024-06-03T08:36:28.727 INFO:teuthology.orchestra.run.smithi088.stderr:fsid ec3c3fdd-3ac6-4d7d-9412-683130ed8b04 2024-06-03T08:36:28.727 INFO:teuthology.orchestra.run.smithi088.stderr:last_changed 2024-06-03T08:34:31.885125+0000 2024-06-03T08:36:28.728 INFO:teuthology.orchestra.run.smithi088.stderr:created 2024-06-03T08:34:31.885125+0000 2024-06-03T08:36:28.728 INFO:teuthology.orchestra.run.smithi088.stderr:min_mon_release 18 (reef) 2024-06-03T08:36:28.728 INFO:teuthology.orchestra.run.smithi088.stderr:election_strategy: 3 2024-06-03T08:36:28.728 INFO:teuthology.orchestra.run.smithi088.stderr:0: v1:172.21.15.88:6789/0 mon.a 2024-06-03T08:36:28.728 INFO:teuthology.orchestra.run.smithi088.stderr:1: v1:172.21.15.192:6789/0 mon.b 2024-06-03T08:36:28.728 INFO:teuthology.orchestra.run.smithi088.stderr:2: v1:172.21.15.88:6790/0 mon.c 2024-06-03T08:36:28.728 INFO:teuthology.orchestra.run.smithi088.stderr: 2024-06-03T08:36:28.729 INFO:teuthology.orchestra.run.smithi088.stderr:2024-06-03T08:36:28.729+0000 7fd5eeffd640 1 -- 172.21.15.88:0/2232032709 <== mon.1 v1:172.21.15.192:6789/0 4 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 580+0+0 (unknown 2594602851 0 0) 0x7fd60013d380 con 0x7fd6000b1b70 2024-06-03T08:36:28.730 INFO:teuthology.orchestra.run.smithi088.stderr:2024-06-03T08:36:28.729+0000 7fd5eeffd640 1 -- 172.21.15.88:0/2232032709 >> v1:172.21.15.88:6790/0 conn(0x7fd60014ae00 legacy=0x7fd6001e0cc0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-06-03T08:36:28.730 INFO:teuthology.orchestra.run.smithi088.stderr:2024-06-03T08:36:28.729+0000 7fd5eeffd640 1 -- 172.21.15.88:0/2232032709 >> v1:172.21.15.88:6789/0 conn(0x7fd6000a8a60 legacy=0x7fd6001321f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-06-03T08:36:28.730 INFO:teuthology.orchestra.run.smithi088.stderr:2024-06-03T08:36:28.729+0000 7fd5eeffd640 10 monclient: _finish_hunting 0 2024-06-03T08:36:28.730 INFO:teuthology.orchestra.run.smithi088.stderr:2024-06-03T08:36:28.729+0000 7fd5eeffd640 1 monclient: found mon.b 2024-06-03T08:36:28.730 INFO:teuthology.orchestra.run.smithi088.stderr:2024-06-03T08:36:28.729+0000 7fd5eeffd640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-06-03T08:36:28.730 INFO:teuthology.orchestra.run.smithi088.stderr:2024-06-03T08:36:28.729+0000 7fd5eeffd640 10 monclient: _send_mon_message to mon.b at v1:172.21.15.192:6789/0 2024-06-03T08:36:28.730 INFO:teuthology.orchestra.run.smithi088.stderr:2024-06-03T08:36:28.729+0000 7fd5eeffd640 1 -- 172.21.15.88:0/2232032709 --> v1:172.21.15.192:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fd6001d9b30 con 0x7fd6000b1b70 2024-06-03T08:36:28.730 INFO:teuthology.orchestra.run.smithi088.stderr:2024-06-03T08:36:28.729+0000 7fd5eeffd640 10 monclient: _finish_auth 0 2024-06-03T08:36:28.730 INFO:teuthology.orchestra.run.smithi088.stderr:2024-06-03T08:36:28.729+0000 7fd5eeffd640 10 monclient: _check_auth_tickets 2024-06-03T08:36:28.730 INFO:teuthology.orchestra.run.smithi088.stderr:2024-06-03T08:36:28.729+0000 7fd5eeffd640 20 monclient: _check_auth_rotating not needed by client.admin 2024-06-03T08:36:28.730 INFO:teuthology.orchestra.run.smithi088.stderr:2024-06-03T08:36:28.729+0000 7fd6049d7640 5 monclient: authenticate success, global_id 4273 2024-06-03T08:36:28.730 INFO:teuthology.orchestra.run.smithi088.stderr:2024-06-03T08:36:28.729+0000 7fd6049d7640 10 monclient: _renew_subs 2024-06-03T08:36:28.730 INFO:teuthology.orchestra.run.smithi088.stderr:2024-06-03T08:36:28.729+0000 7fd6049d7640 10 monclient: _send_mon_message to mon.b at v1:172.21.15.192:6789/0 2024-06-03T08:36:28.731 INFO:teuthology.orchestra.run.smithi088.stderr:2024-06-03T08:36:28.729+0000 7fd6049d7640 1 -- 172.21.15.88:0/2232032709 --> v1:172.21.15.192:6789/0 -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fd6001da760 con 0x7fd6000b1b70 2024-06-03T08:36:28.731 INFO:teuthology.orchestra.run.smithi088.stderr:2024-06-03T08:36:28.729+0000 7fd6049d7640 10 client.4273.objecter _maybe_request_map subscribing (onetime) to next osd map 2024-06-03T08:36:28.731 INFO:teuthology.orchestra.run.smithi088.stderr:2024-06-03T08:36:28.729+0000 7fd6049d7640 10 monclient: _renew_subs 2024-06-03T08:36:28.731 INFO:teuthology.orchestra.run.smithi088.stderr:2024-06-03T08:36:28.729+0000 7fd6049d7640 10 monclient: _send_mon_message to mon.b at v1:172.21.15.192:6789/0 2024-06-03T08:36:28.731 INFO:teuthology.orchestra.run.smithi088.stderr:2024-06-03T08:36:28.729+0000 7fd6049d7640 1 -- 172.21.15.88:0/2232032709 --> v1:172.21.15.192:6789/0 -- mon_subscribe({osdmap=0}) v3 -- 0x7fd60013e880 con 0x7fd6000b1b70 2024-06-03T08:36:28.731 INFO:teuthology.orchestra.run.smithi088.stderr:2024-06-03T08:36:28.729+0000 7fd6049d7640 1 librados: init done 2024-06-03T08:36:28.731 INFO:teuthology.orchestra.run.smithi088.stderr:2024-06-03T08:36:28.730+0000 7fd6049d7640 10 start_mon_command cmd=[{"prefix": "get_command_descriptions"}] 2024-06-03T08:36:28.731 INFO:teuthology.orchestra.run.smithi088.stderr:2024-06-03T08:36:28.730+0000 7fd6049d7640 10 monclient: _send_command 1 [{"prefix": "get_command_descriptions"}] 2024-06-03T08:36:28.731 INFO:teuthology.orchestra.run.smithi088.stderr:2024-06-03T08:36:28.730+0000 7fd6049d7640 10 monclient: _send_mon_message to mon.b at v1:172.21.15.192:6789/0 2024-06-03T08:36:28.731 INFO:teuthology.orchestra.run.smithi088.stderr:2024-06-03T08:36:28.730+0000 7fd6049d7640 1 -- 172.21.15.88:0/2232032709 --> v1:172.21.15.192:6789/0 -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fd6001d9d40 con 0x7fd6000b1b70 2024-06-03T08:36:28.939 INFO:teuthology.orchestra.run.smithi088.stderr:2024-06-03T08:36:28.939+0000 7fd5eeffd640 1 -- 172.21.15.88:0/2232032709 <== mon.1 v1:172.21.15.192:6789/0 5 ==== config(0 keys) v1 ==== 4+0+0 (unknown 0 0 0) 0x7fd5f40047e0 con 0x7fd6000b1b70 2024-06-03T08:36:28.939 INFO:teuthology.orchestra.run.smithi088.stderr:2024-06-03T08:36:28.940+0000 7fd5eeffd640 10 monclient: handle_config config(0 keys) v1 2024-06-03T08:36:28.940 INFO:teuthology.orchestra.run.smithi088.stderr:2024-06-03T08:36:28.940+0000 7fd5fcd99640 4 set_mon_vals no callback set 2024-06-03T08:36:28.973 INFO:teuthology.orchestra.run.smithi088.stderr:2024-06-03T08:36:28.974+0000 7fd5eeffd640 1 -- 172.21.15.88:0/2232032709 <== mon.1 v1:172.21.15.192:6789/0 6 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 2234374065 0 0) 0x7fd5f4004af0 con 0x7fd6000b1b70 2024-06-03T08:36:28.973 INFO:teuthology.orchestra.run.smithi088.stderr:2024-06-03T08:36:28.974+0000 7fd5eeffd640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-06-03T08:36:28.973 INFO:teuthology.orchestra.run.smithi088.stderr:2024-06-03T08:36:28.974+0000 7fd5eeffd640 10 monclient: got monmap 1 from mon.b (according to old e1) 2024-06-03T08:36:28.973 INFO:teuthology.orchestra.run.smithi088.stderr:2024-06-03T08:36:28.974+0000 7fd5eeffd640 10 monclient: dump: 2024-06-03T08:36:28.974 INFO:teuthology.orchestra.run.smithi088.stderr:epoch 1 2024-06-03T08:36:28.974 INFO:teuthology.orchestra.run.smithi088.stderr:fsid ec3c3fdd-3ac6-4d7d-9412-683130ed8b04 2024-06-03T08:36:28.974 INFO:teuthology.orchestra.run.smithi088.stderr:last_changed 2024-06-03T08:34:31.885125+0000 2024-06-03T08:36:28.974 INFO:teuthology.orchestra.run.smithi088.stderr:created 2024-06-03T08:34:31.885125+0000 2024-06-03T08:36:28.974 INFO:teuthology.orchestra.run.smithi088.stderr:min_mon_release 18 (reef) 2024-06-03T08:36:28.974 INFO:teuthology.orchestra.run.smithi088.stderr:election_strategy: 3 2024-06-03T08:36:28.974 INFO:teuthology.orchestra.run.smithi088.stderr:0: v1:172.21.15.88:6789/0 mon.a 2024-06-03T08:36:28.974 INFO:teuthology.orchestra.run.smithi088.stderr:1: v1:172.21.15.192:6789/0 mon.b 2024-06-03T08:36:28.974 INFO:teuthology.orchestra.run.smithi088.stderr:2: v1:172.21.15.88:6790/0 mon.c 2024-06-03T08:36:28.974 INFO:teuthology.orchestra.run.smithi088.stderr: 2024-06-03T08:36:28.975 INFO:teuthology.orchestra.run.smithi088.stderr:2024-06-03T08:36:28.975+0000 7fd5eeffd640 1 -- 172.21.15.88:0/2232032709 <== mon.1 v1:172.21.15.192:6789/0 7 ==== mgrmap(e 5) v1 ==== 89110+0+0 (unknown 873533973 0 0) 0x7fd5f401a870 con 0x7fd6000b1b70 2024-06-03T08:36:28.975 INFO:teuthology.orchestra.run.smithi088.stderr:2024-06-03T08:36:28.976+0000 7fd5fe59c640 10 monclient: get_auth_request con 0x7fd5e006d1d0 auth_method 0 2024-06-03T08:36:28.976 INFO:teuthology.orchestra.run.smithi088.stderr:2024-06-03T08:36:28.976+0000 7fd5eeffd640 10 client.4273.objecter ms_handle_connect 0x7fd5e006d1d0 2024-06-03T08:36:28.979 INFO:teuthology.orchestra.run.smithi088.stderr:2024-06-03T08:36:28.980+0000 7fd5eeffd640 1 -- 172.21.15.88:0/2232032709 <== mon.1 v1:172.21.15.192:6789/0 8 ==== osd_map(23..23 src has 1..23) v4 ==== 4079+0+0 (unknown 1390638890 0 0) 0x7fd5f40541a0 con 0x7fd6000b1b70 2024-06-03T08:36:28.979 INFO:teuthology.orchestra.run.smithi088.stderr:2024-06-03T08:36:28.980+0000 7fd5eeffd640 10 client.4273.objecter ms_dispatch 0x7fd600000b90 osd_map(23..23 src has 1..23) v4 2024-06-03T08:36:28.979 INFO:teuthology.orchestra.run.smithi088.stderr:2024-06-03T08:36:28.980+0000 7fd5eeffd640 3 client.4273.objecter handle_osd_map got epochs [23,23] > 0 2024-06-03T08:36:28.979 INFO:teuthology.orchestra.run.smithi088.stderr:2024-06-03T08:36:28.980+0000 7fd5eeffd640 3 client.4273.objecter handle_osd_map decoding full epoch 23 2024-06-03T08:36:28.980 INFO:teuthology.orchestra.run.smithi088.stderr:2024-06-03T08:36:28.980+0000 7fd5eeffd640 20 client.4273.objecter dump_active .. 0 homeless 2024-06-03T08:36:28.981 INFO:teuthology.orchestra.run.smithi088.stderr:2024-06-03T08:36:28.982+0000 7fd5eeffd640 1 -- 172.21.15.88:0/2232032709 <== mon.1 v1:172.21.15.192:6789/0 9 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+182498 (unknown 1092875540 0 3016760700) 0x7fd5f404fb80 con 0x7fd6000b1b70 2024-06-03T08:36:28.981 INFO:teuthology.orchestra.run.smithi088.stderr:2024-06-03T08:36:28.982+0000 7fd5eeffd640 10 monclient: handle_mon_command_ack 1 [{"prefix": "get_command_descriptions"}] 2024-06-03T08:36:28.982 INFO:teuthology.orchestra.run.smithi088.stderr:2024-06-03T08:36:28.982+0000 7fd5eeffd640 10 monclient: _finish_command 1 = system:0 2024-06-03T08:36:29.163 INFO:teuthology.orchestra.run.smithi088.stderr:2024-06-03T08:36:29.163+0000 7fd6049d7640 10 start_mon_command cmd=[{"prefix": "osd pool application enable", "pool": "rbd", "app": "rbd", "yes_i_really_mean_it": true}] 2024-06-03T08:36:29.163 INFO:teuthology.orchestra.run.smithi088.stderr:2024-06-03T08:36:29.163+0000 7fd6049d7640 10 monclient: _send_command 2 [{"prefix": "osd pool application enable", "pool": "rbd", "app": "rbd", "yes_i_really_mean_it": true}] 2024-06-03T08:36:29.163 INFO:teuthology.orchestra.run.smithi088.stderr:2024-06-03T08:36:29.163+0000 7fd6049d7640 10 monclient: _send_mon_message to mon.b at v1:172.21.15.192:6789/0 2024-06-03T08:36:29.164 INFO:teuthology.orchestra.run.smithi088.stderr:2024-06-03T08:36:29.163+0000 7fd6049d7640 1 -- 172.21.15.88:0/2232032709 --> v1:172.21.15.192:6789/0 -- mon_command({"prefix": "osd pool application enable", "pool": "rbd", "app": "rbd", "yes_i_really_mean_it": true} v 0) v1 -- 0x7fd60004ea60 con 0x7fd6000b1b70 2024-06-03T08:36:29.696 INFO:teuthology.orchestra.run.smithi088.stderr:2024-06-03T08:36:29.695+0000 7fd5eeffd640 1 -- 172.21.15.88:0/2232032709 <== mon.1 v1:172.21.15.192:6789/0 10 ==== mon_command_ack([{"prefix": "osd pool application enable", "pool": "rbd", "app": "rbd", "yes_i_really_mean_it": true}]=0 enabled application 'rbd' on pool 'rbd' v24) v1 ==== 173+0+0 (unknown 388435365 0 0) 0x7fd5f4051c80 con 0x7fd6000b1b70 2024-06-03T08:36:29.696 INFO:teuthology.orchestra.run.smithi088.stderr:2024-06-03T08:36:29.695+0000 7fd5eeffd640 10 monclient: handle_mon_command_ack 2 [{"prefix": "osd pool application enable", "pool": "rbd", "app": "rbd", "yes_i_really_mean_it": true}] 2024-06-03T08:36:29.696 INFO:teuthology.orchestra.run.smithi088.stderr:2024-06-03T08:36:29.695+0000 7fd5eeffd640 10 monclient: _finish_command 2 = system:0 enabled application 'rbd' on pool 'rbd' 2024-06-03T08:36:29.696 INFO:teuthology.orchestra.run.smithi088.stderr:enabled application 'rbd' on pool 'rbd' 2024-06-03T08:36:29.698 INFO:teuthology.orchestra.run.smithi088.stderr:2024-06-03T08:36:29.697+0000 7fd6049d7640 10 librados: watch_flush enter 2024-06-03T08:36:29.698 INFO:teuthology.orchestra.run.smithi088.stderr:2024-06-03T08:36:29.697+0000 7fd6049d7640 10 librados: watch_flush exit 2024-06-03T08:36:29.698 INFO:teuthology.orchestra.run.smithi088.stderr:2024-06-03T08:36:29.697+0000 7fd6049d7640 20 client.4273.objecter shutdown clearing up homeless session... 2024-06-03T08:36:29.698 INFO:teuthology.orchestra.run.smithi088.stderr:2024-06-03T08:36:29.697+0000 7fd6049d7640 10 client.4273.objecter successfully canceled tick 2024-06-03T08:36:29.699 INFO:teuthology.orchestra.run.smithi088.stderr:2024-06-03T08:36:29.697+0000 7fd6049d7640 1 -- 172.21.15.88:0/2232032709 >> v1:172.21.15.192:6800/34229 conn(0x7fd5e006d1d0 legacy=0x7fd5e006f690 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-06-03T08:36:29.699 INFO:teuthology.orchestra.run.smithi088.stderr:2024-06-03T08:36:29.697+0000 7fd6049d7640 10 monclient: shutdown 2024-06-03T08:36:29.699 INFO:teuthology.orchestra.run.smithi088.stderr:2024-06-03T08:36:29.697+0000 7fd6049d7640 20 monclient: shutdown discarding 0 pending message(s) 2024-06-03T08:36:29.699 INFO:teuthology.orchestra.run.smithi088.stderr:2024-06-03T08:36:29.697+0000 7fd6049d7640 1 -- 172.21.15.88:0/2232032709 >> v1:172.21.15.192:6789/0 conn(0x7fd6000b1b70 legacy=0x7fd6001da050 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-06-03T08:36:29.699 INFO:teuthology.orchestra.run.smithi088.stderr:2024-06-03T08:36:29.697+0000 7fd6049d7640 1 -- 172.21.15.88:0/2232032709 shutdown_connections 2024-06-03T08:36:29.699 INFO:teuthology.orchestra.run.smithi088.stderr:2024-06-03T08:36:29.697+0000 7fd6049d7640 1 -- 172.21.15.88:0/2232032709 >> 172.21.15.88:0/2232032709 conn(0x7fd6000b7170 msgr2=0x7fd6000b7550 unknown :-1 s=STATE_NONE l=0).mark_down 2024-06-03T08:36:29.699 INFO:teuthology.orchestra.run.smithi088.stderr:2024-06-03T08:36:29.697+0000 7fd6049d7640 1 -- 172.21.15.88:0/2232032709 shutdown_connections 2024-06-03T08:36:29.699 INFO:teuthology.orchestra.run.smithi088.stderr:2024-06-03T08:36:29.697+0000 7fd6049d7640 1 -- 172.21.15.88:0/2232032709 wait complete. 2024-06-03T08:36:29.699 INFO:teuthology.orchestra.run.smithi088.stderr:2024-06-03T08:36:29.698+0000 7fd6049d7640 1 librados: shutdown 2024-06-03T08:36:29.709 INFO:tasks.ceph:Starting mds daemons in cluster ceph... 2024-06-03T08:36:29.710 INFO:tasks.ceph:Waiting until ceph daemons up and pgs clean... 2024-06-03T08:36:29.710 INFO:tasks.ceph.ceph_manager.ceph:waiting for mgr available 2024-06-03T08:36:29.710 DEBUG:teuthology.orchestra.run.smithi088:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph mgr dump --format=json 2024-06-03T08:36:29.710 INFO:tasks.daemonwatchdog.daemon_watchdog:watchdog starting 2024-06-03T08:36:29.812 INFO:teuthology.orchestra.run.smithi088.stderr:2024-06-03T08:36:29.812+0000 7fa597bb9640 10 monclient: get_monmap_and_config 2024-06-03T08:36:29.812 INFO:teuthology.orchestra.run.smithi088.stderr:2024-06-03T08:36:29.812+0000 7fa597bb9640 10 monclient: build_initial_monmap 2024-06-03T08:36:29.813 INFO:teuthology.orchestra.run.smithi088.stderr:2024-06-03T08:36:29.812+0000 7fa597bb9640 10 monclient: monmap: 2024-06-03T08:36:29.813 INFO:teuthology.orchestra.run.smithi088.stderr:epoch 0 2024-06-03T08:36:29.813 INFO:teuthology.orchestra.run.smithi088.stderr:fsid ec3c3fdd-3ac6-4d7d-9412-683130ed8b04 2024-06-03T08:36:29.813 INFO:teuthology.orchestra.run.smithi088.stderr:last_changed 2024-06-03T08:36:29.813324+0000 2024-06-03T08:36:29.813 INFO:teuthology.orchestra.run.smithi088.stderr:created 2024-06-03T08:36:29.813324+0000 2024-06-03T08:36:29.813 INFO:teuthology.orchestra.run.smithi088.stderr:min_mon_release 0 (unknown) 2024-06-03T08:36:29.813 INFO:teuthology.orchestra.run.smithi088.stderr:election_strategy: 3 2024-06-03T08:36:29.813 INFO:teuthology.orchestra.run.smithi088.stderr:0: v1:172.21.15.88:6789/0 mon.noname-a 2024-06-03T08:36:29.813 INFO:teuthology.orchestra.run.smithi088.stderr:1: v1:172.21.15.192:6789/0 mon.noname-c 2024-06-03T08:36:29.813 INFO:teuthology.orchestra.run.smithi088.stderr:2: v1:172.21.15.88:6790/0 mon.noname-b 2024-06-03T08:36:29.813 INFO:teuthology.orchestra.run.smithi088.stderr: 2024-06-03T08:36:29.815 INFO:teuthology.orchestra.run.smithi088.stderr:2024-06-03T08:36:29.812+0000 7fa597bb9640 1 Processor -- start 2024-06-03T08:36:29.815 INFO:teuthology.orchestra.run.smithi088.stderr:2024-06-03T08:36:29.813+0000 7fa597bb9640 1 -- start start 2024-06-03T08:36:29.815 INFO:teuthology.orchestra.run.smithi088.stderr:2024-06-03T08:36:29.813+0000 7fa597bb9640 10 monclient: init 2024-06-03T08:36:29.815 INFO:teuthology.orchestra.run.smithi088.stderr:2024-06-03T08:36:29.813+0000 7fa597bb9640 10 monclient: _reopen_session rank -1 2024-06-03T08:36:29.815 INFO:teuthology.orchestra.run.smithi088.stderr:2024-06-03T08:36:29.813+0000 7fa597bb9640 10 monclient: _add_conns ranks=[1,0,2] 2024-06-03T08:36:29.815 INFO:teuthology.orchestra.run.smithi088.stderr:2024-06-03T08:36:29.813+0000 7fa597bb9640 10 monclient(hunting): picked mon.noname-c con 0x7fa5900adb90 addr v1:172.21.15.192:6789/0 2024-06-03T08:36:29.815 INFO:teuthology.orchestra.run.smithi088.stderr:2024-06-03T08:36:29.813+0000 7fa597bb9640 10 monclient(hunting): picked mon.noname-a con 0x7fa59006ee00 addr v1:172.21.15.88:6789/0 2024-06-03T08:36:29.815 INFO:teuthology.orchestra.run.smithi088.stderr:2024-06-03T08:36:29.813+0000 7fa597bb9640 10 monclient(hunting): picked mon.noname-b con 0x7fa5901383e0 addr v1:172.21.15.88:6790/0 2024-06-03T08:36:29.815 INFO:teuthology.orchestra.run.smithi088.stderr:2024-06-03T08:36:29.813+0000 7fa597bb9640 1 -- --> v1:172.21.15.88:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fa59006f2c0 con 0x7fa59006ee00 2024-06-03T08:36:29.816 INFO:teuthology.orchestra.run.smithi088.stderr:2024-06-03T08:36:29.813+0000 7fa597bb9640 1 -- --> v1:172.21.15.192:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fa5900710c0 con 0x7fa5900adb90 2024-06-03T08:36:29.816 INFO:teuthology.orchestra.run.smithi088.stderr:2024-06-03T08:36:29.813+0000 7fa597bb9640 1 -- --> v1:172.21.15.88:6790/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fa590071fd0 con 0x7fa5901383e0 2024-06-03T08:36:29.816 INFO:teuthology.orchestra.run.smithi088.stderr:2024-06-03T08:36:29.813+0000 7fa597bb9640 10 monclient(hunting): _renew_subs 2024-06-03T08:36:29.816 INFO:teuthology.orchestra.run.smithi088.stderr:2024-06-03T08:36:29.813+0000 7fa597bb9640 10 monclient(hunting): authenticate will time out at 1245.623684s 2024-06-03T08:36:29.816 INFO:teuthology.orchestra.run.smithi088.stderr:2024-06-03T08:36:29.814+0000 7fa595955640 1 --1- >> v1:172.21.15.192:6789/0 conn(0x7fa5900adb90 0x7fa5900718c0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.192:6789/0 says I am v1:172.21.15.88:52262/0 (socket says 172.21.15.88:52262) 2024-06-03T08:36:29.816 INFO:teuthology.orchestra.run.smithi088.stderr:2024-06-03T08:36:29.814+0000 7fa595955640 1 -- 172.21.15.88:0/3460351213 learned_addr learned my addr 172.21.15.88:0/3460351213 (peer_addr_for_me v1:172.21.15.88:0/0) 2024-06-03T08:36:29.817 INFO:teuthology.orchestra.run.smithi088.stderr:2024-06-03T08:36:29.816+0000 7fa594953640 1 -- 172.21.15.88:0/3460351213 <== mon.1 v1:172.21.15.192:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 667762519 0 0) 0x7fa5900710c0 con 0x7fa5900adb90 2024-06-03T08:36:29.817 INFO:teuthology.orchestra.run.smithi088.stderr:2024-06-03T08:36:29.816+0000 7fa594953640 10 monclient(hunting): _negotiate 2024-06-03T08:36:29.817 INFO:teuthology.orchestra.run.smithi088.stderr:2024-06-03T08:36:29.816+0000 7fa594953640 10 monclient(hunting): _init_auth method 2 2024-06-03T08:36:29.817 INFO:teuthology.orchestra.run.smithi088.stderr:2024-06-03T08:36:29.816+0000 7fa594953640 10 monclient(hunting): _init_auth creating new auth 2024-06-03T08:36:29.817 INFO:teuthology.orchestra.run.smithi088.stderr:2024-06-03T08:36:29.816+0000 7fa594953640 10 monclient(hunting): my global_id is 4276 2024-06-03T08:36:29.817 INFO:teuthology.orchestra.run.smithi088.stderr:2024-06-03T08:36:29.816+0000 7fa594953640 1 -- 172.21.15.88:0/3460351213 --> v1:172.21.15.192:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7fa57c005730 con 0x7fa5900adb90 2024-06-03T08:36:29.817 INFO:teuthology.orchestra.run.smithi088.stderr:2024-06-03T08:36:29.816+0000 7fa594953640 1 -- 172.21.15.88:0/3460351213 <== mon.0 v1:172.21.15.88:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 2734806953 0 0) 0x7fa59006f2c0 con 0x7fa59006ee00 2024-06-03T08:36:29.817 INFO:teuthology.orchestra.run.smithi088.stderr:2024-06-03T08:36:29.816+0000 7fa594953640 10 monclient(hunting): _negotiate 2024-06-03T08:36:29.817 INFO:teuthology.orchestra.run.smithi088.stderr:2024-06-03T08:36:29.816+0000 7fa594953640 10 monclient(hunting): _init_auth method 2 2024-06-03T08:36:29.817 INFO:teuthology.orchestra.run.smithi088.stderr:2024-06-03T08:36:29.816+0000 7fa594953640 10 monclient(hunting): _init_auth creating new auth 2024-06-03T08:36:29.818 INFO:teuthology.orchestra.run.smithi088.stderr:2024-06-03T08:36:29.816+0000 7fa594953640 10 monclient(hunting): my global_id is 4275 2024-06-03T08:36:29.818 INFO:teuthology.orchestra.run.smithi088.stderr:2024-06-03T08:36:29.816+0000 7fa594953640 1 -- 172.21.15.88:0/3460351213 --> v1:172.21.15.88:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7fa5900710c0 con 0x7fa59006ee00 2024-06-03T08:36:29.818 INFO:teuthology.orchestra.run.smithi088.stderr:2024-06-03T08:36:29.816+0000 7fa594953640 1 -- 172.21.15.88:0/3460351213 <== mon.2 v1:172.21.15.88:6790/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 1007067384 0 0) 0x7fa590071fd0 con 0x7fa5901383e0 2024-06-03T08:36:29.818 INFO:teuthology.orchestra.run.smithi088.stderr:2024-06-03T08:36:29.816+0000 7fa594953640 10 monclient(hunting): _negotiate 2024-06-03T08:36:29.818 INFO:teuthology.orchestra.run.smithi088.stderr:2024-06-03T08:36:29.816+0000 7fa594953640 10 monclient(hunting): _init_auth method 2 2024-06-03T08:36:29.818 INFO:teuthology.orchestra.run.smithi088.stderr:2024-06-03T08:36:29.816+0000 7fa594953640 10 monclient(hunting): _init_auth creating new auth 2024-06-03T08:36:29.818 INFO:teuthology.orchestra.run.smithi088.stderr:2024-06-03T08:36:29.816+0000 7fa594953640 10 monclient(hunting): my global_id is 4295 2024-06-03T08:36:29.818 INFO:teuthology.orchestra.run.smithi088.stderr:2024-06-03T08:36:29.816+0000 7fa594953640 1 -- 172.21.15.88:0/3460351213 --> v1:172.21.15.88:6790/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7fa59006f2c0 con 0x7fa5901383e0 2024-06-03T08:36:29.820 INFO:teuthology.orchestra.run.smithi088.stderr:2024-06-03T08:36:29.819+0000 7fa594953640 1 -- 172.21.15.88:0/3460351213 <== mon.1 v1:172.21.15.192:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 214+0+0 (unknown 2343718644 0 0) 0x7fa57c005730 con 0x7fa5900adb90 2024-06-03T08:36:29.820 INFO:teuthology.orchestra.run.smithi088.stderr:2024-06-03T08:36:29.819+0000 7fa594953640 1 -- 172.21.15.88:0/3460351213 >> v1:172.21.15.88:6790/0 conn(0x7fa5901383e0 legacy=0x7fa590149af0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-06-03T08:36:29.820 INFO:teuthology.orchestra.run.smithi088.stderr:2024-06-03T08:36:29.819+0000 7fa594953640 1 -- 172.21.15.88:0/3460351213 >> v1:172.21.15.88:6789/0 conn(0x7fa59006ee00 legacy=0x7fa590137cd0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-06-03T08:36:29.820 INFO:teuthology.orchestra.run.smithi088.stderr:2024-06-03T08:36:29.819+0000 7fa594953640 10 monclient: _finish_hunting 0 2024-06-03T08:36:29.820 INFO:teuthology.orchestra.run.smithi088.stderr:2024-06-03T08:36:29.819+0000 7fa594953640 1 monclient: found mon.noname-c 2024-06-03T08:36:29.820 INFO:teuthology.orchestra.run.smithi088.stderr:2024-06-03T08:36:29.819+0000 7fa594953640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-06-03T08:36:29.820 INFO:teuthology.orchestra.run.smithi088.stderr:2024-06-03T08:36:29.819+0000 7fa594953640 10 monclient: _send_mon_message to mon.noname-c at v1:172.21.15.192:6789/0 2024-06-03T08:36:29.821 INFO:teuthology.orchestra.run.smithi088.stderr:2024-06-03T08:36:29.819+0000 7fa594953640 1 -- 172.21.15.88:0/3460351213 --> v1:172.21.15.192:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fa590141750 con 0x7fa5900adb90 2024-06-03T08:36:29.821 INFO:teuthology.orchestra.run.smithi088.stderr:2024-06-03T08:36:29.819+0000 7fa594953640 10 monclient: _finish_auth 0 2024-06-03T08:36:29.821 INFO:teuthology.orchestra.run.smithi088.stderr:2024-06-03T08:36:29.819+0000 7fa594953640 10 monclient: _check_auth_tickets 2024-06-03T08:36:29.821 INFO:teuthology.orchestra.run.smithi088.stderr:2024-06-03T08:36:29.819+0000 7fa594953640 20 monclient: _check_auth_rotating not needed by client.admin 2024-06-03T08:36:29.821 INFO:teuthology.orchestra.run.smithi088.stderr:2024-06-03T08:36:29.819+0000 7fa594953640 1 -- 172.21.15.88:0/3460351213 <== mon.1 v1:172.21.15.192:6789/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 2234374065 0 0) 0x7fa58c001610 con 0x7fa5900adb90 2024-06-03T08:36:29.821 INFO:teuthology.orchestra.run.smithi088.stderr:2024-06-03T08:36:29.819+0000 7fa597bb9640 5 monclient: authenticate success, global_id 4276 2024-06-03T08:36:29.821 INFO:teuthology.orchestra.run.smithi088.stderr:2024-06-03T08:36:29.819+0000 7fa597bb9640 20 monclient: get_monmap_and_config waiting for monmap|config 2024-06-03T08:36:29.821 INFO:teuthology.orchestra.run.smithi088.stderr:2024-06-03T08:36:29.819+0000 7fa594953640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-06-03T08:36:29.821 INFO:teuthology.orchestra.run.smithi088.stderr:2024-06-03T08:36:29.819+0000 7fa594953640 10 monclient: got monmap 1 from mon.noname-c (according to old e1) 2024-06-03T08:36:29.821 INFO:teuthology.orchestra.run.smithi088.stderr:2024-06-03T08:36:29.819+0000 7fa594953640 10 monclient: dump: 2024-06-03T08:36:29.821 INFO:teuthology.orchestra.run.smithi088.stderr:epoch 1 2024-06-03T08:36:29.821 INFO:teuthology.orchestra.run.smithi088.stderr:fsid ec3c3fdd-3ac6-4d7d-9412-683130ed8b04 2024-06-03T08:36:29.821 INFO:teuthology.orchestra.run.smithi088.stderr:last_changed 2024-06-03T08:34:31.885125+0000 2024-06-03T08:36:29.822 INFO:teuthology.orchestra.run.smithi088.stderr:created 2024-06-03T08:34:31.885125+0000 2024-06-03T08:36:29.822 INFO:teuthology.orchestra.run.smithi088.stderr:min_mon_release 18 (reef) 2024-06-03T08:36:29.822 INFO:teuthology.orchestra.run.smithi088.stderr:election_strategy: 3 2024-06-03T08:36:29.822 INFO:teuthology.orchestra.run.smithi088.stderr:0: v1:172.21.15.88:6789/0 mon.a 2024-06-03T08:36:29.822 INFO:teuthology.orchestra.run.smithi088.stderr:1: v1:172.21.15.192:6789/0 mon.b 2024-06-03T08:36:29.822 INFO:teuthology.orchestra.run.smithi088.stderr:2: v1:172.21.15.88:6790/0 mon.c 2024-06-03T08:36:29.822 INFO:teuthology.orchestra.run.smithi088.stderr: 2024-06-03T08:36:29.822 INFO:teuthology.orchestra.run.smithi088.stderr:2024-06-03T08:36:29.819+0000 7fa597bb9640 20 monclient: get_monmap_and_config waiting for monmap|config 2024-06-03T08:36:29.825 INFO:teuthology.orchestra.run.smithi088.stderr:2024-06-03T08:36:29.821+0000 7fa594953640 1 -- 172.21.15.88:0/3460351213 <== mon.1 v1:172.21.15.192:6789/0 4 ==== config(0 keys) v1 ==== 4+0+0 (unknown 0 0 0) 0x7fa58c0018a0 con 0x7fa5900adb90 2024-06-03T08:36:29.825 INFO:teuthology.orchestra.run.smithi088.stderr:2024-06-03T08:36:29.821+0000 7fa594953640 10 monclient: handle_config config(0 keys) v1 2024-06-03T08:36:29.825 INFO:teuthology.orchestra.run.smithi088.stderr:2024-06-03T08:36:29.821+0000 7fa594953640 1 -- 172.21.15.88:0/3460351213 <== mon.1 v1:172.21.15.192:6789/0 5 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 2234374065 0 0) 0x7fa58c001c30 con 0x7fa5900adb90 2024-06-03T08:36:29.825 INFO:teuthology.orchestra.run.smithi088.stderr:2024-06-03T08:36:29.821+0000 7fa597bb9640 10 monclient: get_monmap_and_config success 2024-06-03T08:36:29.825 INFO:teuthology.orchestra.run.smithi088.stderr:2024-06-03T08:36:29.821+0000 7fa597bb9640 4 set_mon_vals no callback set 2024-06-03T08:36:29.825 INFO:teuthology.orchestra.run.smithi088.stderr:2024-06-03T08:36:29.821+0000 7fa594953640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-06-03T08:36:29.826 INFO:teuthology.orchestra.run.smithi088.stderr:2024-06-03T08:36:29.821+0000 7fa594953640 10 monclient: got monmap 1 from mon.b (according to old e1) 2024-06-03T08:36:29.826 INFO:teuthology.orchestra.run.smithi088.stderr:2024-06-03T08:36:29.821+0000 7fa594953640 10 monclient: dump: 2024-06-03T08:36:29.826 INFO:teuthology.orchestra.run.smithi088.stderr:epoch 1 2024-06-03T08:36:29.826 INFO:teuthology.orchestra.run.smithi088.stderr:fsid ec3c3fdd-3ac6-4d7d-9412-683130ed8b04 2024-06-03T08:36:29.826 INFO:teuthology.orchestra.run.smithi088.stderr:last_changed 2024-06-03T08:34:31.885125+0000 2024-06-03T08:36:29.826 INFO:teuthology.orchestra.run.smithi088.stderr:created 2024-06-03T08:34:31.885125+0000 2024-06-03T08:36:29.826 INFO:teuthology.orchestra.run.smithi088.stderr:min_mon_release 18 (reef) 2024-06-03T08:36:29.826 INFO:teuthology.orchestra.run.smithi088.stderr:election_strategy: 3 2024-06-03T08:36:29.826 INFO:teuthology.orchestra.run.smithi088.stderr:0: v1:172.21.15.88:6789/0 mon.a 2024-06-03T08:36:29.826 INFO:teuthology.orchestra.run.smithi088.stderr:1: v1:172.21.15.192:6789/0 mon.b 2024-06-03T08:36:29.826 INFO:teuthology.orchestra.run.smithi088.stderr:2: v1:172.21.15.88:6790/0 mon.c 2024-06-03T08:36:29.826 INFO:teuthology.orchestra.run.smithi088.stderr: 2024-06-03T08:36:29.826 INFO:teuthology.orchestra.run.smithi088.stderr:2024-06-03T08:36:29.822+0000 7fa597bb9640 10 monclient: shutdown 2024-06-03T08:36:29.827 INFO:teuthology.orchestra.run.smithi088.stderr:2024-06-03T08:36:29.822+0000 7fa597bb9640 20 monclient: shutdown discarding 0 pending message(s) 2024-06-03T08:36:29.827 INFO:teuthology.orchestra.run.smithi088.stderr:2024-06-03T08:36:29.822+0000 7fa597bb9640 1 -- 172.21.15.88:0/3460351213 >> v1:172.21.15.192:6789/0 conn(0x7fa5900adb90 legacy=0x7fa5900718c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-06-03T08:36:29.827 INFO:teuthology.orchestra.run.smithi088.stderr:2024-06-03T08:36:29.822+0000 7fa597bb9640 1 -- 172.21.15.88:0/3460351213 shutdown_connections 2024-06-03T08:36:29.827 INFO:teuthology.orchestra.run.smithi088.stderr:2024-06-03T08:36:29.822+0000 7fa597bb9640 1 -- 172.21.15.88:0/3460351213 >> 172.21.15.88:0/3460351213 conn(0x7fa5900b2f20 msgr2=0x7fa5900b3300 unknown :-1 s=STATE_NONE l=0).mark_down 2024-06-03T08:36:29.827 INFO:teuthology.orchestra.run.smithi088.stderr:2024-06-03T08:36:29.822+0000 7fa597bb9640 1 -- 172.21.15.88:0/3460351213 shutdown_connections 2024-06-03T08:36:29.827 INFO:teuthology.orchestra.run.smithi088.stderr:2024-06-03T08:36:29.822+0000 7fa597bb9640 1 -- 172.21.15.88:0/3460351213 wait complete. 2024-06-03T08:36:29.827 INFO:teuthology.orchestra.run.smithi088.stderr:2024-06-03T08:36:29.823+0000 7fa597bb9640 10 monclient: build_initial_monmap 2024-06-03T08:36:29.827 INFO:teuthology.orchestra.run.smithi088.stderr:2024-06-03T08:36:29.823+0000 7fa597bb9640 10 monclient: monmap: 2024-06-03T08:36:29.827 INFO:teuthology.orchestra.run.smithi088.stderr:epoch 0 2024-06-03T08:36:29.827 INFO:teuthology.orchestra.run.smithi088.stderr:fsid 00000000-0000-0000-0000-000000000000 2024-06-03T08:36:29.827 INFO:teuthology.orchestra.run.smithi088.stderr:last_changed 0.000000 2024-06-03T08:36:29.827 INFO:teuthology.orchestra.run.smithi088.stderr:created 0.000000 2024-06-03T08:36:29.827 INFO:teuthology.orchestra.run.smithi088.stderr:min_mon_release 0 (unknown) 2024-06-03T08:36:29.828 INFO:teuthology.orchestra.run.smithi088.stderr:election_strategy: 1 2024-06-03T08:36:29.828 INFO:teuthology.orchestra.run.smithi088.stderr:0: v1:172.21.15.88:6789/0 mon.noname-a 2024-06-03T08:36:29.828 INFO:teuthology.orchestra.run.smithi088.stderr:1: v1:172.21.15.192:6789/0 mon.noname-b 2024-06-03T08:36:29.828 INFO:teuthology.orchestra.run.smithi088.stderr:2: v1:172.21.15.88:6790/0 mon.noname-c 2024-06-03T08:36:29.828 INFO:teuthology.orchestra.run.smithi088.stderr: 2024-06-03T08:36:29.828 INFO:teuthology.orchestra.run.smithi088.stderr:2024-06-03T08:36:29.823+0000 7fa597bb9640 1 librados: starting msgr at 2024-06-03T08:36:29.828 INFO:teuthology.orchestra.run.smithi088.stderr:2024-06-03T08:36:29.823+0000 7fa597bb9640 1 librados: starting objecter 2024-06-03T08:36:29.828 INFO:teuthology.orchestra.run.smithi088.stderr:2024-06-03T08:36:29.823+0000 7fa597bb9640 1 Processor -- start 2024-06-03T08:36:29.828 INFO:teuthology.orchestra.run.smithi088.stderr:2024-06-03T08:36:29.823+0000 7fa597bb9640 1 -- start start 2024-06-03T08:36:29.828 INFO:teuthology.orchestra.run.smithi088.stderr:2024-06-03T08:36:29.823+0000 7fa597bb9640 1 librados: setting wanted keys 2024-06-03T08:36:29.828 INFO:teuthology.orchestra.run.smithi088.stderr:2024-06-03T08:36:29.823+0000 7fa597bb9640 1 librados: calling monclient init 2024-06-03T08:36:29.828 INFO:teuthology.orchestra.run.smithi088.stderr:2024-06-03T08:36:29.823+0000 7fa597bb9640 10 monclient: init 2024-06-03T08:36:29.828 INFO:teuthology.orchestra.run.smithi088.stderr:2024-06-03T08:36:29.823+0000 7fa597bb9640 10 monclient: _reopen_session rank -1 2024-06-03T08:36:29.829 INFO:teuthology.orchestra.run.smithi088.stderr:2024-06-03T08:36:29.823+0000 7fa597bb9640 10 monclient: _add_conns ranks=[2,0,1] 2024-06-03T08:36:29.829 INFO:teuthology.orchestra.run.smithi088.stderr:2024-06-03T08:36:29.823+0000 7fa597bb9640 10 monclient(hunting): picked mon.noname-c con 0x7fa59006ee00 addr v1:172.21.15.88:6790/0 2024-06-03T08:36:29.829 INFO:teuthology.orchestra.run.smithi088.stderr:2024-06-03T08:36:29.824+0000 7fa597bb9640 10 monclient(hunting): picked mon.noname-a con 0x7fa5900adb90 addr v1:172.21.15.88:6789/0 2024-06-03T08:36:29.829 INFO:teuthology.orchestra.run.smithi088.stderr:2024-06-03T08:36:29.824+0000 7fa597bb9640 10 monclient(hunting): picked mon.noname-b con 0x7fa5901383e0 addr v1:172.21.15.192:6789/0 2024-06-03T08:36:29.829 INFO:teuthology.orchestra.run.smithi088.stderr:2024-06-03T08:36:29.824+0000 7fa597bb9640 1 -- --> v1:172.21.15.88:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fa590140fe0 con 0x7fa5900adb90 2024-06-03T08:36:29.829 INFO:teuthology.orchestra.run.smithi088.stderr:2024-06-03T08:36:29.824+0000 7fa597bb9640 1 -- --> v1:172.21.15.192:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fa590140cc0 con 0x7fa5901383e0 2024-06-03T08:36:29.829 INFO:teuthology.orchestra.run.smithi088.stderr:2024-06-03T08:36:29.824+0000 7fa597bb9640 1 -- --> v1:172.21.15.88:6790/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fa590133320 con 0x7fa59006ee00 2024-06-03T08:36:29.829 INFO:teuthology.orchestra.run.smithi088.stderr:2024-06-03T08:36:29.824+0000 7fa597bb9640 10 monclient(hunting): _renew_subs 2024-06-03T08:36:29.829 INFO:teuthology.orchestra.run.smithi088.stderr:2024-06-03T08:36:29.824+0000 7fa597bb9640 10 monclient(hunting): authenticate will time out at 1245.634007s 2024-06-03T08:36:29.829 INFO:teuthology.orchestra.run.smithi088.stderr:2024-06-03T08:36:29.824+0000 7fa595154640 1 --1- >> v1:172.21.15.88:6789/0 conn(0x7fa5900adb90 0x7fa5901d6a30 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=1).handle_server_banner_and_identify peer v1:172.21.15.88:6789/0 says I am v1:172.21.15.88:60216/0 (socket says 172.21.15.88:60216) 2024-06-03T08:36:29.829 INFO:teuthology.orchestra.run.smithi088.stderr:2024-06-03T08:36:29.824+0000 7fa595154640 1 -- 172.21.15.88:0/1922385508 learned_addr learned my addr 172.21.15.88:0/1922385508 (peer_addr_for_me v1:172.21.15.88:0/0) 2024-06-03T08:36:29.829 INFO:teuthology.orchestra.run.smithi088.stderr:2024-06-03T08:36:29.824+0000 7fa596156640 1 --1- 172.21.15.88:0/1922385508 >> v1:172.21.15.192:6789/0 conn(0x7fa5901383e0 0x7fa5901d7140 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=1).handle_server_banner_and_identify peer v1:172.21.15.192:6789/0 says I am v1:172.21.15.88:52274/0 (socket says 172.21.15.88:52274) 2024-06-03T08:36:29.829 INFO:teuthology.orchestra.run.smithi088.stderr:2024-06-03T08:36:29.824+0000 7fa5867fc640 10 client.?.objecter ms_handle_connect 0x7fa5901383e0 2024-06-03T08:36:29.829 INFO:teuthology.orchestra.run.smithi088.stderr:2024-06-03T08:36:29.824+0000 7fa5867fc640 10 client.?.objecter resend_mon_ops 2024-06-03T08:36:29.830 INFO:teuthology.orchestra.run.smithi088.stderr:2024-06-03T08:36:29.824+0000 7fa5867fc640 10 client.?.objecter ms_handle_connect 0x7fa5900adb90 2024-06-03T08:36:29.830 INFO:teuthology.orchestra.run.smithi088.stderr:2024-06-03T08:36:29.824+0000 7fa5867fc640 10 client.?.objecter resend_mon_ops 2024-06-03T08:36:29.830 INFO:teuthology.orchestra.run.smithi088.stderr:2024-06-03T08:36:29.824+0000 7fa5867fc640 10 client.?.objecter ms_handle_connect 0x7fa59006ee00 2024-06-03T08:36:29.830 INFO:teuthology.orchestra.run.smithi088.stderr:2024-06-03T08:36:29.824+0000 7fa5867fc640 10 client.?.objecter resend_mon_ops 2024-06-03T08:36:29.831 INFO:teuthology.orchestra.run.smithi088.stderr:2024-06-03T08:36:29.826+0000 7fa5867fc640 1 -- 172.21.15.88:0/1922385508 <== mon.1 v1:172.21.15.192:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 236681846 0 0) 0x7fa590140cc0 con 0x7fa5901383e0 2024-06-03T08:36:29.831 INFO:teuthology.orchestra.run.smithi088.stderr:2024-06-03T08:36:29.826+0000 7fa5867fc640 10 monclient(hunting): _negotiate 2024-06-03T08:36:29.831 INFO:teuthology.orchestra.run.smithi088.stderr:2024-06-03T08:36:29.826+0000 7fa5867fc640 10 monclient(hunting): _init_auth method 2 2024-06-03T08:36:29.831 INFO:teuthology.orchestra.run.smithi088.stderr:2024-06-03T08:36:29.826+0000 7fa5867fc640 10 monclient(hunting): _init_auth creating new auth 2024-06-03T08:36:29.834 INFO:teuthology.orchestra.run.smithi088.stderr:2024-06-03T08:36:29.826+0000 7fa5867fc640 10 monclient(hunting): my global_id is 4279 2024-06-03T08:36:29.834 INFO:teuthology.orchestra.run.smithi088.stderr:2024-06-03T08:36:29.826+0000 7fa5867fc640 1 -- 172.21.15.88:0/1922385508 --> v1:172.21.15.192:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7fa574004eb0 con 0x7fa5901383e0 2024-06-03T08:36:29.834 INFO:teuthology.orchestra.run.smithi088.stderr:2024-06-03T08:36:29.827+0000 7fa5867fc640 1 -- 172.21.15.88:0/1922385508 <== mon.2 v1:172.21.15.88:6790/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 911277 0 0) 0x7fa590133320 con 0x7fa59006ee00 2024-06-03T08:36:29.834 INFO:teuthology.orchestra.run.smithi088.stderr:2024-06-03T08:36:29.827+0000 7fa5867fc640 10 monclient(hunting): _negotiate 2024-06-03T08:36:29.834 INFO:teuthology.orchestra.run.smithi088.stderr:2024-06-03T08:36:29.827+0000 7fa5867fc640 10 monclient(hunting): _init_auth method 2 2024-06-03T08:36:29.834 INFO:teuthology.orchestra.run.smithi088.stderr:2024-06-03T08:36:29.827+0000 7fa5867fc640 10 monclient(hunting): _init_auth creating new auth 2024-06-03T08:36:29.834 INFO:teuthology.orchestra.run.smithi088.stderr:2024-06-03T08:36:29.827+0000 7fa5867fc640 10 monclient(hunting): my global_id is 4298 2024-06-03T08:36:29.834 INFO:teuthology.orchestra.run.smithi088.stderr:2024-06-03T08:36:29.827+0000 7fa5867fc640 1 -- 172.21.15.88:0/1922385508 --> v1:172.21.15.88:6790/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7fa590140cc0 con 0x7fa59006ee00 2024-06-03T08:36:29.834 INFO:teuthology.orchestra.run.smithi088.stderr:2024-06-03T08:36:29.827+0000 7fa5867fc640 1 -- 172.21.15.88:0/1922385508 <== mon.0 v1:172.21.15.88:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 882379104 0 0) 0x7fa590140fe0 con 0x7fa5900adb90 2024-06-03T08:36:29.835 INFO:teuthology.orchestra.run.smithi088.stderr:2024-06-03T08:36:29.827+0000 7fa5867fc640 10 monclient(hunting): _negotiate 2024-06-03T08:36:29.835 INFO:teuthology.orchestra.run.smithi088.stderr:2024-06-03T08:36:29.827+0000 7fa5867fc640 10 monclient(hunting): _init_auth method 2 2024-06-03T08:36:29.835 INFO:teuthology.orchestra.run.smithi088.stderr:2024-06-03T08:36:29.827+0000 7fa5867fc640 10 monclient(hunting): _init_auth creating new auth 2024-06-03T08:36:29.835 INFO:teuthology.orchestra.run.smithi088.stderr:2024-06-03T08:36:29.827+0000 7fa5867fc640 10 monclient(hunting): my global_id is 4278 2024-06-03T08:36:29.835 INFO:teuthology.orchestra.run.smithi088.stderr:2024-06-03T08:36:29.827+0000 7fa5867fc640 1 -- 172.21.15.88:0/1922385508 --> v1:172.21.15.88:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7fa590133320 con 0x7fa5900adb90 2024-06-03T08:36:29.837 INFO:teuthology.orchestra.run.smithi088.stderr:2024-06-03T08:36:29.831+0000 7fa5867fc640 1 -- 172.21.15.88:0/1922385508 <== mon.1 v1:172.21.15.192:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 764+0+0 (unknown 659242880 0 0) 0x7fa574004eb0 con 0x7fa5901383e0 2024-06-03T08:36:29.837 INFO:teuthology.orchestra.run.smithi088.stderr:2024-06-03T08:36:29.831+0000 7fa5867fc640 1 -- 172.21.15.88:0/1922385508 --> v1:172.21.15.192:6789/0 -- auth(proto 2 165 bytes epoch 0) v1 -- 0x7fa590140fe0 con 0x7fa5901383e0 2024-06-03T08:36:29.837 INFO:teuthology.orchestra.run.smithi088.stderr:2024-06-03T08:36:29.831+0000 7fa5867fc640 1 -- 172.21.15.88:0/1922385508 <== mon.1 v1:172.21.15.192:6789/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 2234374065 0 0) 0x7fa588001c30 con 0x7fa5901383e0 2024-06-03T08:36:29.837 INFO:teuthology.orchestra.run.smithi088.stderr:2024-06-03T08:36:29.831+0000 7fa5867fc640 10 monclient(hunting): handle_monmap mon_map magic: 0 v1 2024-06-03T08:36:29.837 INFO:teuthology.orchestra.run.smithi088.stderr:2024-06-03T08:36:29.831+0000 7fa5867fc640 10 monclient(hunting): got monmap 1 from mon.noname-b (according to old e1) 2024-06-03T08:36:29.838 INFO:teuthology.orchestra.run.smithi088.stderr:2024-06-03T08:36:29.831+0000 7fa5867fc640 10 monclient(hunting): dump: 2024-06-03T08:36:29.838 INFO:teuthology.orchestra.run.smithi088.stderr:epoch 1 2024-06-03T08:36:29.838 INFO:teuthology.orchestra.run.smithi088.stderr:fsid ec3c3fdd-3ac6-4d7d-9412-683130ed8b04 2024-06-03T08:36:29.838 INFO:teuthology.orchestra.run.smithi088.stderr:last_changed 2024-06-03T08:34:31.885125+0000 2024-06-03T08:36:29.838 INFO:teuthology.orchestra.run.smithi088.stderr:created 2024-06-03T08:34:31.885125+0000 2024-06-03T08:36:29.838 INFO:teuthology.orchestra.run.smithi088.stderr:min_mon_release 18 (reef) 2024-06-03T08:36:29.838 INFO:teuthology.orchestra.run.smithi088.stderr:election_strategy: 3 2024-06-03T08:36:29.838 INFO:teuthology.orchestra.run.smithi088.stderr:0: v1:172.21.15.88:6789/0 mon.a 2024-06-03T08:36:29.838 INFO:teuthology.orchestra.run.smithi088.stderr:1: v1:172.21.15.192:6789/0 mon.b 2024-06-03T08:36:29.838 INFO:teuthology.orchestra.run.smithi088.stderr:2: v1:172.21.15.88:6790/0 mon.c 2024-06-03T08:36:29.838 INFO:teuthology.orchestra.run.smithi088.stderr: 2024-06-03T08:36:29.838 INFO:teuthology.orchestra.run.smithi088.stderr:2024-06-03T08:36:29.831+0000 7fa5867fc640 10 monclient(hunting): _finish_auth 0 2024-06-03T08:36:29.838 INFO:teuthology.orchestra.run.smithi088.stderr:2024-06-03T08:36:29.831+0000 7fa5867fc640 1 -- 172.21.15.88:0/1922385508 <== mon.2 v1:172.21.15.88:6790/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 764+0+0 (unknown 990638516 0 0) 0x7fa590140cc0 con 0x7fa59006ee00 2024-06-03T08:36:29.839 INFO:teuthology.orchestra.run.smithi088.stderr:2024-06-03T08:36:29.832+0000 7fa5867fc640 1 -- 172.21.15.88:0/1922385508 --> v1:172.21.15.88:6790/0 -- auth(proto 2 165 bytes epoch 0) v1 -- 0x7fa574004eb0 con 0x7fa59006ee00 2024-06-03T08:36:29.839 INFO:teuthology.orchestra.run.smithi088.stderr:2024-06-03T08:36:29.832+0000 7fa5867fc640 1 -- 172.21.15.88:0/1922385508 <== mon.2 v1:172.21.15.88:6790/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 2234374065 0 0) 0x7fa58c001510 con 0x7fa59006ee00 2024-06-03T08:36:29.839 INFO:teuthology.orchestra.run.smithi088.stderr:2024-06-03T08:36:29.832+0000 7fa5867fc640 10 monclient(hunting): handle_monmap mon_map magic: 0 v1 2024-06-03T08:36:29.839 INFO:teuthology.orchestra.run.smithi088.stderr:2024-06-03T08:36:29.832+0000 7fa5867fc640 10 monclient(hunting): got monmap 1 from mon.c (according to old e1) 2024-06-03T08:36:29.839 INFO:teuthology.orchestra.run.smithi088.stderr:2024-06-03T08:36:29.832+0000 7fa5867fc640 10 monclient(hunting): dump: 2024-06-03T08:36:29.839 INFO:teuthology.orchestra.run.smithi088.stderr:epoch 1 2024-06-03T08:36:29.839 INFO:teuthology.orchestra.run.smithi088.stderr:fsid ec3c3fdd-3ac6-4d7d-9412-683130ed8b04 2024-06-03T08:36:29.839 INFO:teuthology.orchestra.run.smithi088.stderr:last_changed 2024-06-03T08:34:31.885125+0000 2024-06-03T08:36:29.839 INFO:teuthology.orchestra.run.smithi088.stderr:created 2024-06-03T08:34:31.885125+0000 2024-06-03T08:36:29.839 INFO:teuthology.orchestra.run.smithi088.stderr:min_mon_release 18 (reef) 2024-06-03T08:36:29.839 INFO:teuthology.orchestra.run.smithi088.stderr:election_strategy: 3 2024-06-03T08:36:29.839 INFO:teuthology.orchestra.run.smithi088.stderr:0: v1:172.21.15.88:6789/0 mon.a 2024-06-03T08:36:29.840 INFO:teuthology.orchestra.run.smithi088.stderr:1: v1:172.21.15.192:6789/0 mon.b 2024-06-03T08:36:29.840 INFO:teuthology.orchestra.run.smithi088.stderr:2: v1:172.21.15.88:6790/0 mon.c 2024-06-03T08:36:29.840 INFO:teuthology.orchestra.run.smithi088.stderr: 2024-06-03T08:36:29.840 INFO:teuthology.orchestra.run.smithi088.stderr:2024-06-03T08:36:29.832+0000 7fa5867fc640 1 -- 172.21.15.88:0/1922385508 <== mon.0 v1:172.21.15.88:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 764+0+0 (unknown 1107552011 0 0) 0x7fa590133320 con 0x7fa5900adb90 2024-06-03T08:36:29.840 INFO:teuthology.orchestra.run.smithi088.stderr:2024-06-03T08:36:29.832+0000 7fa5867fc640 1 -- 172.21.15.88:0/1922385508 --> v1:172.21.15.88:6789/0 -- auth(proto 2 165 bytes epoch 0) v1 -- 0x7fa590140cc0 con 0x7fa5900adb90 2024-06-03T08:36:29.840 INFO:teuthology.orchestra.run.smithi088.stderr:2024-06-03T08:36:29.832+0000 7fa5867fc640 1 -- 172.21.15.88:0/1922385508 <== mon.0 v1:172.21.15.88:6789/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 2234374065 0 0) 0x7fa580001970 con 0x7fa5900adb90 2024-06-03T08:36:29.840 INFO:teuthology.orchestra.run.smithi088.stderr:2024-06-03T08:36:29.832+0000 7fa5867fc640 10 monclient(hunting): handle_monmap mon_map magic: 0 v1 2024-06-03T08:36:29.840 INFO:teuthology.orchestra.run.smithi088.stderr:2024-06-03T08:36:29.832+0000 7fa5867fc640 10 monclient(hunting): got monmap 1 from mon.a (according to old e1) 2024-06-03T08:36:29.840 INFO:teuthology.orchestra.run.smithi088.stderr:2024-06-03T08:36:29.832+0000 7fa5867fc640 10 monclient(hunting): dump: 2024-06-03T08:36:29.840 INFO:teuthology.orchestra.run.smithi088.stderr:epoch 1 2024-06-03T08:36:29.840 INFO:teuthology.orchestra.run.smithi088.stderr:fsid ec3c3fdd-3ac6-4d7d-9412-683130ed8b04 2024-06-03T08:36:29.840 INFO:teuthology.orchestra.run.smithi088.stderr:last_changed 2024-06-03T08:34:31.885125+0000 2024-06-03T08:36:29.840 INFO:teuthology.orchestra.run.smithi088.stderr:created 2024-06-03T08:34:31.885125+0000 2024-06-03T08:36:29.841 INFO:teuthology.orchestra.run.smithi088.stderr:min_mon_release 18 (reef) 2024-06-03T08:36:29.841 INFO:teuthology.orchestra.run.smithi088.stderr:election_strategy: 3 2024-06-03T08:36:29.841 INFO:teuthology.orchestra.run.smithi088.stderr:0: v1:172.21.15.88:6789/0 mon.a 2024-06-03T08:36:29.841 INFO:teuthology.orchestra.run.smithi088.stderr:1: v1:172.21.15.192:6789/0 mon.b 2024-06-03T08:36:29.841 INFO:teuthology.orchestra.run.smithi088.stderr:2: v1:172.21.15.88:6790/0 mon.c 2024-06-03T08:36:29.841 INFO:teuthology.orchestra.run.smithi088.stderr: 2024-06-03T08:36:29.841 INFO:teuthology.orchestra.run.smithi088.stderr:2024-06-03T08:36:29.835+0000 7fa5867fc640 1 -- 172.21.15.88:0/1922385508 <== mon.2 v1:172.21.15.88:6790/0 4 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 580+0+0 (unknown 3569307991 0 0) 0x7fa574004eb0 con 0x7fa59006ee00 2024-06-03T08:36:29.841 INFO:teuthology.orchestra.run.smithi088.stderr:2024-06-03T08:36:29.835+0000 7fa5867fc640 1 -- 172.21.15.88:0/1922385508 >> v1:172.21.15.192:6789/0 conn(0x7fa5901383e0 legacy=0x7fa5901d7140 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-06-03T08:36:29.841 INFO:teuthology.orchestra.run.smithi088.stderr:2024-06-03T08:36:29.835+0000 7fa5867fc640 1 -- 172.21.15.88:0/1922385508 >> v1:172.21.15.88:6789/0 conn(0x7fa5900adb90 legacy=0x7fa5901d6a30 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-06-03T08:36:29.841 INFO:teuthology.orchestra.run.smithi088.stderr:2024-06-03T08:36:29.835+0000 7fa5867fc640 10 monclient: _finish_hunting 0 2024-06-03T08:36:29.841 INFO:teuthology.orchestra.run.smithi088.stderr:2024-06-03T08:36:29.835+0000 7fa5867fc640 1 monclient: found mon.c 2024-06-03T08:36:29.841 INFO:teuthology.orchestra.run.smithi088.stderr:2024-06-03T08:36:29.835+0000 7fa5867fc640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-06-03T08:36:29.841 INFO:teuthology.orchestra.run.smithi088.stderr:2024-06-03T08:36:29.835+0000 7fa5867fc640 10 monclient: _send_mon_message to mon.c at v1:172.21.15.88:6790/0 2024-06-03T08:36:29.841 INFO:teuthology.orchestra.run.smithi088.stderr:2024-06-03T08:36:29.835+0000 7fa5867fc640 1 -- 172.21.15.88:0/1922385508 --> v1:172.21.15.88:6790/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fa5901334b0 con 0x7fa59006ee00 2024-06-03T08:36:29.842 INFO:teuthology.orchestra.run.smithi088.stderr:2024-06-03T08:36:29.835+0000 7fa5867fc640 10 monclient: _finish_auth 0 2024-06-03T08:36:29.842 INFO:teuthology.orchestra.run.smithi088.stderr:2024-06-03T08:36:29.835+0000 7fa5867fc640 10 monclient: _check_auth_tickets 2024-06-03T08:36:29.842 INFO:teuthology.orchestra.run.smithi088.stderr:2024-06-03T08:36:29.835+0000 7fa5867fc640 20 monclient: _check_auth_rotating not needed by client.admin 2024-06-03T08:36:29.842 INFO:teuthology.orchestra.run.smithi088.stderr:2024-06-03T08:36:29.835+0000 7fa597bb9640 5 monclient: authenticate success, global_id 4298 2024-06-03T08:36:29.842 INFO:teuthology.orchestra.run.smithi088.stderr:2024-06-03T08:36:29.835+0000 7fa597bb9640 10 monclient: _renew_subs 2024-06-03T08:36:29.842 INFO:teuthology.orchestra.run.smithi088.stderr:2024-06-03T08:36:29.835+0000 7fa597bb9640 10 monclient: _send_mon_message to mon.c at v1:172.21.15.88:6790/0 2024-06-03T08:36:29.842 INFO:teuthology.orchestra.run.smithi088.stderr:2024-06-03T08:36:29.835+0000 7fa597bb9640 1 -- 172.21.15.88:0/1922385508 --> v1:172.21.15.88:6790/0 -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fa5900acad0 con 0x7fa59006ee00 2024-06-03T08:36:29.842 INFO:teuthology.orchestra.run.smithi088.stderr:2024-06-03T08:36:29.835+0000 7fa597bb9640 10 client.4298.objecter _maybe_request_map subscribing (onetime) to next osd map 2024-06-03T08:36:29.842 INFO:teuthology.orchestra.run.smithi088.stderr:2024-06-03T08:36:29.835+0000 7fa597bb9640 10 monclient: _renew_subs 2024-06-03T08:36:29.842 INFO:teuthology.orchestra.run.smithi088.stderr:2024-06-03T08:36:29.835+0000 7fa597bb9640 10 monclient: _send_mon_message to mon.c at v1:172.21.15.88:6790/0 2024-06-03T08:36:29.842 INFO:teuthology.orchestra.run.smithi088.stderr:2024-06-03T08:36:29.835+0000 7fa597bb9640 1 -- 172.21.15.88:0/1922385508 --> v1:172.21.15.88:6790/0 -- mon_subscribe({osdmap=0}) v3 -- 0x7fa5901e2260 con 0x7fa59006ee00 2024-06-03T08:36:29.842 INFO:teuthology.orchestra.run.smithi088.stderr:2024-06-03T08:36:29.835+0000 7fa597bb9640 1 librados: init done 2024-06-03T08:36:29.842 INFO:teuthology.orchestra.run.smithi088.stderr:2024-06-03T08:36:29.837+0000 7fa597bb9640 10 start_mon_command cmd=[{"prefix": "get_command_descriptions"}] 2024-06-03T08:36:29.843 INFO:teuthology.orchestra.run.smithi088.stderr:2024-06-03T08:36:29.837+0000 7fa597bb9640 10 monclient: _send_command 1 [{"prefix": "get_command_descriptions"}] 2024-06-03T08:36:29.843 INFO:teuthology.orchestra.run.smithi088.stderr:2024-06-03T08:36:29.837+0000 7fa597bb9640 10 monclient: _send_mon_message to mon.c at v1:172.21.15.88:6790/0 2024-06-03T08:36:29.843 INFO:teuthology.orchestra.run.smithi088.stderr:2024-06-03T08:36:29.837+0000 7fa597bb9640 1 -- 172.21.15.88:0/1922385508 --> v1:172.21.15.88:6790/0 -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fa59004ea60 con 0x7fa59006ee00 2024-06-03T08:36:29.844 INFO:teuthology.orchestra.run.smithi088.stderr:2024-06-03T08:36:29.844+0000 7fa5867fc640 1 -- 172.21.15.88:0/1922385508 <== mon.2 v1:172.21.15.88:6790/0 5 ==== config(0 keys) v1 ==== 4+0+0 (unknown 0 0 0) 0x7fa58c004a90 con 0x7fa59006ee00 2024-06-03T08:36:29.844 INFO:teuthology.orchestra.run.smithi088.stderr:2024-06-03T08:36:29.844+0000 7fa5867fc640 10 monclient: handle_config config(0 keys) v1 2024-06-03T08:36:29.844 INFO:teuthology.orchestra.run.smithi088.stderr:2024-06-03T08:36:29.844+0000 7fa587fff640 4 set_mon_vals no callback set 2024-06-03T08:36:29.855 INFO:teuthology.orchestra.run.smithi088.stderr:2024-06-03T08:36:29.854+0000 7fa5867fc640 1 -- 172.21.15.88:0/1922385508 <== mon.2 v1:172.21.15.88:6790/0 6 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 2234374065 0 0) 0x7fa58c004da0 con 0x7fa59006ee00 2024-06-03T08:36:29.855 INFO:teuthology.orchestra.run.smithi088.stderr:2024-06-03T08:36:29.854+0000 7fa5867fc640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-06-03T08:36:29.855 INFO:teuthology.orchestra.run.smithi088.stderr:2024-06-03T08:36:29.854+0000 7fa5867fc640 10 monclient: got monmap 1 from mon.c (according to old e1) 2024-06-03T08:36:29.855 INFO:teuthology.orchestra.run.smithi088.stderr:2024-06-03T08:36:29.854+0000 7fa5867fc640 10 monclient: dump: 2024-06-03T08:36:29.855 INFO:teuthology.orchestra.run.smithi088.stderr:epoch 1 2024-06-03T08:36:29.855 INFO:teuthology.orchestra.run.smithi088.stderr:fsid ec3c3fdd-3ac6-4d7d-9412-683130ed8b04 2024-06-03T08:36:29.855 INFO:teuthology.orchestra.run.smithi088.stderr:last_changed 2024-06-03T08:34:31.885125+0000 2024-06-03T08:36:29.855 INFO:teuthology.orchestra.run.smithi088.stderr:created 2024-06-03T08:34:31.885125+0000 2024-06-03T08:36:29.855 INFO:teuthology.orchestra.run.smithi088.stderr:min_mon_release 18 (reef) 2024-06-03T08:36:29.856 INFO:teuthology.orchestra.run.smithi088.stderr:election_strategy: 3 2024-06-03T08:36:29.856 INFO:teuthology.orchestra.run.smithi088.stderr:0: v1:172.21.15.88:6789/0 mon.a 2024-06-03T08:36:29.856 INFO:teuthology.orchestra.run.smithi088.stderr:1: v1:172.21.15.192:6789/0 mon.b 2024-06-03T08:36:29.856 INFO:teuthology.orchestra.run.smithi088.stderr:2: v1:172.21.15.88:6790/0 mon.c 2024-06-03T08:36:29.856 INFO:teuthology.orchestra.run.smithi088.stderr: 2024-06-03T08:36:29.856 INFO:teuthology.orchestra.run.smithi088.stderr:2024-06-03T08:36:29.855+0000 7fa5867fc640 1 -- 172.21.15.88:0/1922385508 <== mon.2 v1:172.21.15.88:6790/0 7 ==== mgrmap(e 5) v1 ==== 89110+0+0 (unknown 873533973 0 0) 0x7fa58c01ab20 con 0x7fa59006ee00 2024-06-03T08:36:29.856 INFO:teuthology.orchestra.run.smithi088.stderr:2024-06-03T08:36:29.856+0000 7fa595154640 10 monclient: get_auth_request con 0x7fa57406d300 auth_method 0 2024-06-03T08:36:29.857 INFO:teuthology.orchestra.run.smithi088.stderr:2024-06-03T08:36:29.856+0000 7fa5867fc640 10 client.4298.objecter ms_handle_connect 0x7fa57406d300 2024-06-03T08:36:29.860 INFO:teuthology.orchestra.run.smithi088.stderr:2024-06-03T08:36:29.859+0000 7fa5867fc640 1 -- 172.21.15.88:0/1922385508 <== mon.2 v1:172.21.15.88:6790/0 8 ==== osd_map(24..24 src has 1..24) v4 ==== 4090+0+0 (unknown 1321958051 0 0) 0x7fa58c0540f0 con 0x7fa59006ee00 2024-06-03T08:36:29.860 INFO:teuthology.orchestra.run.smithi088.stderr:2024-06-03T08:36:29.859+0000 7fa5867fc640 10 client.4298.objecter ms_dispatch 0x7fa590000b90 osd_map(24..24 src has 1..24) v4 2024-06-03T08:36:29.860 INFO:teuthology.orchestra.run.smithi088.stderr:2024-06-03T08:36:29.859+0000 7fa5867fc640 3 client.4298.objecter handle_osd_map got epochs [24,24] > 0 2024-06-03T08:36:29.860 INFO:teuthology.orchestra.run.smithi088.stderr:2024-06-03T08:36:29.859+0000 7fa5867fc640 3 client.4298.objecter handle_osd_map decoding full epoch 24 2024-06-03T08:36:29.860 INFO:teuthology.orchestra.run.smithi088.stderr:2024-06-03T08:36:29.859+0000 7fa5867fc640 20 client.4298.objecter dump_active .. 0 homeless 2024-06-03T08:36:30.166 INFO:teuthology.orchestra.run.smithi088.stderr:2024-06-03T08:36:30.165+0000 7fa5867fc640 1 -- 172.21.15.88:0/1922385508 <== mon.2 v1:172.21.15.88:6790/0 9 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+182498 (unknown 1092875540 0 3016760700) 0x7fa58c04fad0 con 0x7fa59006ee00 2024-06-03T08:36:30.166 INFO:teuthology.orchestra.run.smithi088.stderr:2024-06-03T08:36:30.165+0000 7fa5867fc640 10 monclient: handle_mon_command_ack 1 [{"prefix": "get_command_descriptions"}] 2024-06-03T08:36:30.166 INFO:teuthology.orchestra.run.smithi088.stderr:2024-06-03T08:36:30.165+0000 7fa5867fc640 10 monclient: _finish_command 1 = system:0 2024-06-03T08:36:30.419 INFO:teuthology.orchestra.run.smithi088.stderr:2024-06-03T08:36:30.418+0000 7fa597bb9640 10 start_mon_command cmd=[{"prefix": "mgr dump", "format": "json"}] 2024-06-03T08:36:30.419 INFO:teuthology.orchestra.run.smithi088.stderr:2024-06-03T08:36:30.418+0000 7fa597bb9640 10 monclient: _send_command 2 [{"prefix": "mgr dump", "format": "json"}] 2024-06-03T08:36:30.419 INFO:teuthology.orchestra.run.smithi088.stderr:2024-06-03T08:36:30.418+0000 7fa597bb9640 10 monclient: _send_mon_message to mon.c at v1:172.21.15.88:6790/0 2024-06-03T08:36:30.419 INFO:teuthology.orchestra.run.smithi088.stderr:2024-06-03T08:36:30.418+0000 7fa597bb9640 1 -- 172.21.15.88:0/1922385508 --> v1:172.21.15.88:6790/0 -- mon_command({"prefix": "mgr dump", "format": "json"} v 0) v1 -- 0x7fa5900afa80 con 0x7fa59006ee00 2024-06-03T08:36:30.555 INFO:teuthology.orchestra.run.smithi088.stderr:2024-06-03T08:36:30.554+0000 7fa5867fc640 1 -- 172.21.15.88:0/1922385508 <== mon.2 v1:172.21.15.88:6790/0 10 ==== mon_command_ack([{"prefix": "mgr dump", "format": "json"}]=0 v5) v1 ==== 74+0+170887 (unknown 481731318 0 1087410199) 0x7fa58c051bd0 con 0x7fa59006ee00 2024-06-03T08:36:30.555 INFO:teuthology.orchestra.run.smithi088.stderr:2024-06-03T08:36:30.554+0000 7fa5867fc640 10 monclient: handle_mon_command_ack 2 [{"prefix": "mgr dump", "format": "json"}] 2024-06-03T08:36:30.555 INFO:teuthology.orchestra.run.smithi088.stderr:2024-06-03T08:36:30.554+0000 7fa5867fc640 10 monclient: _finish_command 2 = system:0 2024-06-03T08:36:30.559 INFO:teuthology.orchestra.run.smithi088.stdout: 2024-06-03T08:36:30.560 INFO:teuthology.orchestra.run.smithi088.stderr:2024-06-03T08:36:30.557+0000 7fa597bb9640 10 librados: watch_flush enter 2024-06-03T08:36:30.561 INFO:teuthology.orchestra.run.smithi088.stderr:2024-06-03T08:36:30.557+0000 7fa597bb9640 10 librados: watch_flush exit 2024-06-03T08:36:30.561 INFO:teuthology.orchestra.run.smithi088.stderr:2024-06-03T08:36:30.557+0000 7fa597bb9640 20 client.4298.objecter shutdown clearing up homeless session... 2024-06-03T08:36:30.561 INFO:teuthology.orchestra.run.smithi088.stderr:2024-06-03T08:36:30.557+0000 7fa597bb9640 10 client.4298.objecter successfully canceled tick 2024-06-03T08:36:30.561 INFO:teuthology.orchestra.run.smithi088.stderr:2024-06-03T08:36:30.557+0000 7fa597bb9640 1 -- 172.21.15.88:0/1922385508 >> v1:172.21.15.192:6800/34229 conn(0x7fa57406d300 legacy=0x7fa57406f7c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-06-03T08:36:30.561 INFO:teuthology.orchestra.run.smithi088.stderr:2024-06-03T08:36:30.557+0000 7fa597bb9640 10 monclient: shutdown 2024-06-03T08:36:30.561 INFO:teuthology.orchestra.run.smithi088.stderr:2024-06-03T08:36:30.557+0000 7fa597bb9640 20 monclient: shutdown discarding 0 pending message(s) 2024-06-03T08:36:30.561 INFO:teuthology.orchestra.run.smithi088.stderr:2024-06-03T08:36:30.557+0000 7fa597bb9640 1 -- 172.21.15.88:0/1922385508 >> v1:172.21.15.88:6790/0 conn(0x7fa59006ee00 legacy=0x7fa5901405b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-06-03T08:36:30.561 INFO:teuthology.orchestra.run.smithi088.stderr:2024-06-03T08:36:30.557+0000 7fa597bb9640 1 -- 172.21.15.88:0/1922385508 shutdown_connections 2024-06-03T08:36:30.561 INFO:teuthology.orchestra.run.smithi088.stderr:2024-06-03T08:36:30.557+0000 7fa597bb9640 1 -- 172.21.15.88:0/1922385508 >> 172.21.15.88:0/1922385508 conn(0x7fa5900b2f20 msgr2=0x7fa5900b14e0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-06-03T08:36:30.561 INFO:teuthology.orchestra.run.smithi088.stderr:2024-06-03T08:36:30.557+0000 7fa597bb9640 1 -- 172.21.15.88:0/1922385508 shutdown_connections 2024-06-03T08:36:30.561 INFO:teuthology.orchestra.run.smithi088.stderr:2024-06-03T08:36:30.557+0000 7fa597bb9640 1 -- 172.21.15.88:0/1922385508 wait complete. 2024-06-03T08:36:30.561 INFO:teuthology.orchestra.run.smithi088.stderr:2024-06-03T08:36:30.557+0000 7fa597bb9640 1 librados: shutdown 2024-06-03T08:36:30.568 INFO:teuthology.orchestra.run.smithi088.stdout:{"epoch":5,"flags":0,"active_gid":4110,"active_name":"x","active_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.192:6800","nonce":34229}]},"active_addr":"172.21.15.192:6800/34229","active_change":"2024-06-03T08:35:32.415772+0000","active_mgr_features":4540138322906710015,"available":true,"standbys":[{"gid":4113,"name":"y","mgr_features":4540138322906710015,"available_modules":[{"name":"alerts","can_run":true,"error_string":"","module_options":{"interval":{"name":"interval","type":"secs","level":"advanced","flags":1,"default_value":"60","min":"","max":"","enum_allowed":[],"desc":"How frequently to reexamine health status","long_desc":"","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"smtp_destination":{"name":"smtp_destination","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"Email address to send alerts to","long_desc":"","tags":[],"see_also":[]},"smtp_from_name":{"name":"smtp_from_name","type":"str","level":"advanced","flags":1,"default_value":"Ceph","min":"","max":"","enum_allowed":[],"desc":"Email From: name","long_desc":"","tags":[],"see_also":[]},"smtp_host":{"name":"smtp_host","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"SMTP server","long_desc":"","tags":[],"see_also":[]},"smtp_password":{"name":"smtp_password","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"Password to authenticate with","long_desc":"","tags":[],"see_also":[]},"smtp_port":{"name":"smtp_port","type":"int","level":"advanced","flags":1,"default_value":"465","min":"","max":"","enum_allowed":[],"desc":"SMTP port","long_desc":"","tags":[],"see_also":[]},"smtp_sender":{"name":"smtp_sender","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"SMTP envelope sender","long_desc":"","tags":[],"see_also":[]},"smtp_ssl":{"name":"smtp_ssl","type":"bool","level":"advanced","flags":1,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"Use SSL to connect to SMTP server","long_desc":"","tags":[],"see_also":[]},"smtp_user":{"name":"smtp_user","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"User to authenticate as","long_desc":"","tags":[],"see_also":[]}}},{"name":"balancer","can_run":true,"error_string":"","module_options":{"active":{"name":"active","type":"bool","level":"advanced","flags":1,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"automatically balance PGs across cluster","long_desc":"","tags":[],"see_also":[]},"begin_time":{"name":"begin_time","type":"str","level":"advanced","flags":1,"default_value":"0000","min":"","max":"","enum_allowed":[],"desc":"beginning time of day to automatically balance","long_desc":"This is a time of day in the format HHMM.","tags":[],"see_also":[]},"begin_weekday":{"name":"begin_weekday","type":"uint","level":"advanced","flags":1,"default_value":"0","min":"0","max":"6","enum_allowed":[],"desc":"Restrict automatic balancing to this day of the week or later","long_desc":"0 = Sunday, 1 = Monday, etc.","tags":[],"see_also":[]},"crush_compat_max_iterations":{"name":"crush_compat_max_iterations","type":"uint","level":"advanced","flags":1,"default_value":"25","min":"1","max":"250","enum_allowed":[],"desc":"maximum number of iterations to attempt optimization","long_desc":"","tags":[],"see_also":[]},"crush_compat_metrics":{"name":"crush_compat_metrics","type":"str","level":"advanced","flags":1,"default_value":"pgs,objects,bytes","min":"","max":"","enum_allowed":[],"desc":"metrics with which to calculate OSD utilization","long_desc":"Value is a list of one or more of \"pgs\", \"objects\", or \"bytes\", and indicates which metrics to use to balance utilization.","tags":[],"see_also":[]},"crush_compat_step":{"name":"crush_compat_step","type":"float","level":"advanced","flags":1,"default_value":"0.5","min":"0.001","max":"0.999","enum_allowed":[],"desc":"aggressiveness of optimization","long_desc":".99 is very aggressive, .01 is less aggressive","tags":[],"see_also":[]},"end_time":{"name":"end_time","type":"str","level":"advanced","flags":1,"default_value":"2359","min":"","max":"","enum_allowed":[],"desc":"ending time of day to automatically balance","long_desc":"This is a time of day in the format HHMM.","tags":[],"see_also":[]},"end_weekday":{"name":"end_weekday","type":"uint","level":"advanced","flags":1,"default_value":"0","min":"0","max":"6","enum_allowed":[],"desc":"Restrict automatic balancing to days of the week earlier than this","long_desc":"0 = Sunday, 1 = Monday, etc.","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"min_score":{"name":"min_score","type":"float","level":"advanced","flags":1,"default_value":"0","min":"","max":"","enum_allowed":[],"desc":"minimum score, below which no optimization is attempted","long_desc":"","tags":[],"see_also":[]},"mode":{"name":"mode","type":"str","level":"advanced","flags":1,"default_value":"upmap","min":"","max":"","enum_allowed":["crush-compat","none","upmap"],"desc":"Balancer mode","long_desc":"","tags":[],"see_also":[]},"pool_ids":{"name":"pool_ids","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"pools which the automatic balancing will be limited to","long_desc":"","tags":[],"see_also":[]},"sleep_interval":{"name":"sleep_interval","type":"secs","level":"advanced","flags":1,"default_value":"60","min":"","max":"","enum_allowed":[],"desc":"how frequently to wake up and attempt optimization","long_desc":"","tags":[],"see_also":[]},"upmap_max_deviation":{"name":"upmap_max_deviation","type":"int","level":"advanced","flags":1,"default_value":"5","min":"1","max":"","enum_allowed":[],"desc":"deviation below which no optimization is attempted","long_desc":"If the number of PGs are within this count then no optimization is attempted","tags":[],"see_also":[]},"upmap_max_optimizations":{"name":"upmap_max_optimizations","type":"uint","level":"advanced","flags":1,"default_value":"10","min":"","max":"","enum_allowed":[],"desc":"maximum upmap optimizations to make per attempt","long_desc":"","tags":[],"see_also":[]}}},{"name":"cephadm","can_run":true,"error_string":"","module_options":{"agent_down_multiplier":{"name":"agent_down_multiplier","type":"float","level":"advanced","flags":0,"default_value":"3.0","min":"","max":"","enum_allowed":[],"desc":"Multiplied by agent refresh rate to calculate how long agent must not report before being marked down","long_desc":"","tags":[],"see_also":[]},"agent_refresh_rate":{"name":"agent_refresh_rate","type":"secs","level":"advanced","flags":0,"default_value":"20","min":"","max":"","enum_allowed":[],"desc":"How often agent on each host will try to gather and send metadata","long_desc":"","tags":[],"see_also":[]},"agent_starting_port":{"name":"agent_starting_port","type":"int","level":"advanced","flags":0,"default_value":"4721","min":"","max":"","enum_allowed":[],"desc":"First port agent will try to bind to (will also try up to next 1000 subsequent ports if blocked)","long_desc":"","tags":[],"see_also":[]},"allow_ptrace":{"name":"allow_ptrace","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"allow SYS_PTRACE capability on ceph containers","long_desc":"The SYS_PTRACE capability is needed to attach to a process with gdb or strace. Enabling this options can allow debugging daemons that encounter problems at runtime.","tags":[],"see_also":[]},"autotune_interval":{"name":"autotune_interval","type":"secs","level":"advanced","flags":0,"default_value":"600","min":"","max":"","enum_allowed":[],"desc":"how frequently to autotune daemon memory","long_desc":"","tags":[],"see_also":[]},"autotune_memory_target_ratio":{"name":"autotune_memory_target_ratio","type":"float","level":"advanced","flags":0,"default_value":"0.7","min":"","max":"","enum_allowed":[],"desc":"ratio of total system memory to divide amongst autotuned daemons","long_desc":"","tags":[],"see_also":[]},"cgroups_split":{"name":"cgroups_split","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"Pass --cgroups=split when cephadm creates containers (currently podman only)","long_desc":"","tags":[],"see_also":[]},"config_checks_enabled":{"name":"config_checks_enabled","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"Enable or disable the cephadm configuration analysis","long_desc":"","tags":[],"see_also":[]},"config_dashboard":{"name":"config_dashboard","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"manage configs like API endpoints in Dashboard.","long_desc":"","tags":[],"see_also":[]},"container_image_alertmanager":{"name":"container_image_alertmanager","type":"str","level":"advanced","flags":0,"default_value":"quay.io/prometheus/alertmanager:v0.25.0","min":"","max":"","enum_allowed":[],"desc":"Prometheus container image","long_desc":"","tags":[],"see_also":[]},"container_image_base":{"name":"container_image_base","type":"str","level":"advanced","flags":1,"default_value":"quay.io/ceph/ceph","min":"","max":"","enum_allowed":[],"desc":"Container image name, without the tag","long_desc":"","tags":[],"see_also":[]},"container_image_elasticsearch":{"name":"container_image_elasticsearch","type":"str","level":"advanced","flags":0,"default_value":"quay.io/omrizeneva/elasticsearch:6.8.23","min":"","max":"","enum_allowed":[],"desc":"elasticsearch container image","long_desc":"","tags":[],"see_also":[]},"container_image_grafana":{"name":"container_image_grafana","type":"str","level":"advanced","flags":0,"default_value":"quay.io/ceph/ceph-grafana:9.4.7","min":"","max":"","enum_allowed":[],"desc":"Prometheus container image","long_desc":"","tags":[],"see_also":[]},"container_image_haproxy":{"name":"container_image_haproxy","type":"str","level":"advanced","flags":0,"default_value":"quay.io/ceph/haproxy:2.3","min":"","max":"","enum_allowed":[],"desc":"HAproxy container image","long_desc":"","tags":[],"see_also":[]},"container_image_jaeger_agent":{"name":"container_image_jaeger_agent","type":"str","level":"advanced","flags":0,"default_value":"quay.io/jaegertracing/jaeger-agent:1.29","min":"","max":"","enum_allowed":[],"desc":"Jaeger agent container image","long_desc":"","tags":[],"see_also":[]},"container_image_jaeger_collector":{"name":"container_image_jaeger_collector","type":"str","level":"advanced","flags":0,"default_value":"quay.io/jaegertracing/jaeger-collector:1.29","min":"","max":"","enum_allowed":[],"desc":"Jaeger collector container image","long_desc":"","tags":[],"see_also":[]},"container_image_jaeger_query":{"name":"container_image_jaeger_query","type":"str","level":"advanced","flags":0,"default_value":"quay.io/jaegertracing/jaeger-query:1.29","min":"","max":"","enum_allowed":[],"desc":"Jaeger query container image","long_desc":"","tags":[],"see_also":[]},"container_image_keepalived":{"name":"container_image_keepalived","type":"str","level":"advanced","flags":0,"default_value":"quay.io/ceph/keepalived:2.2.4","min":"","max":"","enum_allowed":[],"desc":"Keepalived container image","long_desc":"","tags":[],"see_also":[]},"container_image_loki":{"name":"container_image_loki","type":"str","level":"advanced","flags":0,"default_value":"docker.io/grafana/loki:2.4.0","min":"","max":"","enum_allowed":[],"desc":"Loki container image","long_desc":"","tags":[],"see_also":[]},"container_image_node_exporter":{"name":"container_image_node_exporter","type":"str","level":"advanced","flags":0,"default_value":"quay.io/prometheus/node-exporter:v1.5.0","min":"","max":"","enum_allowed":[],"desc":"Prometheus container image","long_desc":"","tags":[],"see_also":[]},"container_image_nvmeof":{"name":"container_image_nvmeof","type":"str","level":"advanced","flags":0,"default_value":"quay.io/ceph/nvmeof:1.0.0","min":"","max":"","enum_allowed":[],"desc":"Nvme-of container image","long_desc":"","tags":[],"see_also":[]},"container_image_prometheus":{"name":"container_image_prometheus","type":"str","level":"advanced","flags":0,"default_value":"quay.io/prometheus/prometheus:v2.43.0","min":"","max":"","enum_allowed":[],"desc":"Prometheus container image","long_desc":"","tags":[],"see_also":[]},"container_image_promtail":{"name":"container_image_promtail","type":"str","level":"advanced","flags":0,"default_value":"docker.io/grafana/promtail:2.4.0","min":"","max":"","enum_allowed":[],"desc":"Promtail container image","long_desc":"","tags":[],"see_also":[]},"container_image_snmp_gateway":{"name":"container_image_snmp_gateway","type":"str","level":"advanced","flags":0,"default_value":"docker.io/maxwo/snmp-notifier:v1.2.1","min":"","max":"","enum_allowed":[],"desc":"SNMP Gateway container image","long_desc":"","tags":[],"see_also":[]},"container_init":{"name":"container_init","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"Run podman/docker with `--init`","long_desc":"","tags":[],"see_also":[]},"daemon_cache_timeout":{"name":"daemon_cache_timeout","type":"secs","level":"advanced","flags":0,"default_value":"600","min":"","max":"","enum_allowed":[],"desc":"seconds to cache service (daemon) inventory","long_desc":"","tags":[],"see_also":[]},"default_cephadm_command_timeout":{"name":"default_cephadm_command_timeout","type":"int","level":"advanced","flags":0,"default_value":"900","min":"","max":"","enum_allowed":[],"desc":"Default timeout applied to cephadm commands run directly on the host (in seconds)","long_desc":"","tags":[],"see_also":[]},"default_registry":{"name":"default_registry","type":"str","level":"advanced","flags":0,"default_value":"docker.io","min":"","max":"","enum_allowed":[],"desc":"Search-registry to which we should normalize unqualified image names. This is not the default registry","long_desc":"","tags":[],"see_also":[]},"device_cache_timeout":{"name":"device_cache_timeout","type":"secs","level":"advanced","flags":0,"default_value":"1800","min":"","max":"","enum_allowed":[],"desc":"seconds to cache device inventory","long_desc":"","tags":[],"see_also":[]},"device_enhanced_scan":{"name":"device_enhanced_scan","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"Use libstoragemgmt during device scans","long_desc":"","tags":[],"see_also":[]},"facts_cache_timeout":{"name":"facts_cache_timeout","type":"secs","level":"advanced","flags":0,"default_value":"60","min":"","max":"","enum_allowed":[],"desc":"seconds to cache host facts data","long_desc":"","tags":[],"see_also":[]},"host_check_interval":{"name":"host_check_interval","type":"secs","level":"advanced","flags":0,"default_value":"600","min":"","max":"","enum_allowed":[],"desc":"how frequently to perform a host check","long_desc":"","tags":[],"see_also":[]},"hw_monitoring":{"name":"hw_monitoring","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"Deploy hw monitoring daemon on every host.","long_desc":"","tags":[],"see_also":[]},"inventory_list_all":{"name":"inventory_list_all","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"Whether ceph-volume inventory should report more devices (mostly mappers (LVs / mpaths), partitions...)","long_desc":"","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_refresh_metadata":{"name":"log_refresh_metadata","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"Log all refresh metadata. Includes daemon, device, and host info collected regularly. Only has effect if logging at debug level","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"log to the \"cephadm\" cluster log channel\"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"manage_etc_ceph_ceph_conf":{"name":"manage_etc_ceph_ceph_conf","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"Manage and own /etc/ceph/ceph.conf on the hosts.","long_desc":"","tags":[],"see_also":[]},"manage_etc_ceph_ceph_conf_hosts":{"name":"manage_etc_ceph_ceph_conf_hosts","type":"str","level":"advanced","flags":0,"default_value":"*","min":"","max":"","enum_allowed":[],"desc":"PlacementSpec describing on which hosts to manage /etc/ceph/ceph.conf","long_desc":"","tags":[],"see_also":[]},"max_count_per_host":{"name":"max_count_per_host","type":"int","level":"advanced","flags":0,"default_value":"10","min":"","max":"","enum_allowed":[],"desc":"max number of daemons per service per host","long_desc":"","tags":[],"see_also":[]},"max_osd_draining_count":{"name":"max_osd_draining_count","type":"int","level":"advanced","flags":0,"default_value":"10","min":"","max":"","enum_allowed":[],"desc":"max number of osds that will be drained simultaneously when osds are removed","long_desc":"","tags":[],"see_also":[]},"migration_current":{"name":"migration_current","type":"int","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"internal - do not modify","long_desc":"","tags":[],"see_also":[]},"mode":{"name":"mode","type":"str","level":"advanced","flags":0,"default_value":"root","min":"","max":"","enum_allowed":["cephadm-package","root"],"desc":"mode for remote execution of cephadm","long_desc":"","tags":[],"see_also":[]},"oob_default_addr":{"name":"oob_default_addr","type":"str","level":"advanced","flags":0,"default_value":"169.254.1.1","min":"","max":"","enum_allowed":[],"desc":"Default address for RedFish API (oob management).","long_desc":"","tags":[],"see_also":[]},"prometheus_alerts_path":{"name":"prometheus_alerts_path","type":"str","level":"advanced","flags":0,"default_value":"/etc/prometheus/ceph/ceph_default_alerts.yml","min":"","max":"","enum_allowed":[],"desc":"location of alerts to include in prometheus deployments","long_desc":"","tags":[],"see_also":[]},"registry_insecure":{"name":"registry_insecure","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"Registry is to be considered insecure (no TLS available). Only for development purposes.","long_desc":"","tags":[],"see_also":[]},"registry_password":{"name":"registry_password","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"Custom repository password. Only used for logging into a registry.","long_desc":"","tags":[],"see_also":[]},"registry_url":{"name":"registry_url","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"Registry url for login purposes. This is not the default registry","long_desc":"","tags":[],"see_also":[]},"registry_username":{"name":"registry_username","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"Custom repository username. Only used for logging into a registry.","long_desc":"","tags":[],"see_also":[]},"secure_monitoring_stack":{"name":"secure_monitoring_stack","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"Enable TLS security for all the monitoring stack daemons","long_desc":"","tags":[],"see_also":[]},"service_discovery_port":{"name":"service_discovery_port","type":"int","level":"advanced","flags":0,"default_value":"8765","min":"","max":"","enum_allowed":[],"desc":"cephadm service discovery port","long_desc":"","tags":[],"see_also":[]},"ssh_config_file":{"name":"ssh_config_file","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"customized SSH config file to connect to managed hosts","long_desc":"","tags":[],"see_also":[]},"use_agent":{"name":"use_agent","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"Use cephadm agent on each host to gather and send metadata","long_desc":"","tags":[],"see_also":[]},"use_repo_digest":{"name":"use_repo_digest","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"Automatically convert image tags to image digest. Make sure all daemons use the same image","long_desc":"","tags":[],"see_also":[]},"warn_on_failed_host_check":{"name":"warn_on_failed_host_check","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"raise a health warning if the host check fails","long_desc":"","tags":[],"see_also":[]},"warn_on_stray_daemons":{"name":"warn_on_stray_daemons","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"raise a health warning if daemons are detected that are not managed by cephadm","long_desc":"","tags":[],"see_also":[]},"warn_on_stray_hosts":{"name":"warn_on_stray_hosts","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"raise a health warning if daemons are detected on a host that is not managed by cephadm","long_desc":"","tags":[],"see_also":[]}}},{"name":"crash","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"retain_interval":{"name":"retain_interval","type":"secs","level":"advanced","flags":1,"default_value":"31536000","min":"","max":"","enum_allowed":[],"desc":"how long to retain crashes before pruning them","long_desc":"","tags":[],"see_also":[]},"warn_recent_interval":{"name":"warn_recent_interval","type":"secs","level":"advanced","flags":1,"default_value":"1209600","min":"","max":"","enum_allowed":[],"desc":"time interval in which to warn about recent crashes","long_desc":"","tags":[],"see_also":[]}}},{"name":"dashboard","can_run":true,"error_string":"","module_options":{"ACCOUNT_LOCKOUT_ATTEMPTS":{"name":"ACCOUNT_LOCKOUT_ATTEMPTS","type":"int","level":"advanced","flags":0,"default_value":"10","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"ALERTMANAGER_API_HOST":{"name":"ALERTMANAGER_API_HOST","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"ALERTMANAGER_API_SSL_VERIFY":{"name":"ALERTMANAGER_API_SSL_VERIFY","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"AUDIT_API_ENABLED":{"name":"AUDIT_API_ENABLED","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"AUDIT_API_LOG_PAYLOAD":{"name":"AUDIT_API_LOG_PAYLOAD","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"ENABLE_BROWSABLE_API":{"name":"ENABLE_BROWSABLE_API","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"FEATURE_TOGGLE_CEPHFS":{"name":"FEATURE_TOGGLE_CEPHFS","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"FEATURE_TOGGLE_DASHBOARD":{"name":"FEATURE_TOGGLE_DASHBOARD","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"FEATURE_TOGGLE_ISCSI":{"name":"FEATURE_TOGGLE_ISCSI","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"FEATURE_TOGGLE_MIRRORING":{"name":"FEATURE_TOGGLE_MIRRORING","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"FEATURE_TOGGLE_NFS":{"name":"FEATURE_TOGGLE_NFS","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"FEATURE_TOGGLE_RBD":{"name":"FEATURE_TOGGLE_RBD","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"FEATURE_TOGGLE_RGW":{"name":"FEATURE_TOGGLE_RGW","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"GANESHA_CLUSTERS_RADOS_POOL_NAMESPACE":{"name":"GANESHA_CLUSTERS_RADOS_POOL_NAMESPACE","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"GRAFANA_API_PASSWORD":{"name":"GRAFANA_API_PASSWORD","type":"str","level":"advanced","flags":0,"default_value":"admin","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"GRAFANA_API_SSL_VERIFY":{"name":"GRAFANA_API_SSL_VERIFY","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"GRAFANA_API_URL":{"name":"GRAFANA_API_URL","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"GRAFANA_API_USERNAME":{"name":"GRAFANA_API_USERNAME","type":"str","level":"advanced","flags":0,"default_value":"admin","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"GRAFANA_FRONTEND_API_URL":{"name":"GRAFANA_FRONTEND_API_URL","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"GRAFANA_UPDATE_DASHBOARDS":{"name":"GRAFANA_UPDATE_DASHBOARDS","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"ISCSI_API_SSL_VERIFICATION":{"name":"ISCSI_API_SSL_VERIFICATION","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"ISSUE_TRACKER_API_KEY":{"name":"ISSUE_TRACKER_API_KEY","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"PROMETHEUS_API_HOST":{"name":"PROMETHEUS_API_HOST","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"PROMETHEUS_API_SSL_VERIFY":{"name":"PROMETHEUS_API_SSL_VERIFY","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"PWD_POLICY_CHECK_COMPLEXITY_ENABLED":{"name":"PWD_POLICY_CHECK_COMPLEXITY_ENABLED","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"PWD_POLICY_CHECK_EXCLUSION_LIST_ENABLED":{"name":"PWD_POLICY_CHECK_EXCLUSION_LIST_ENABLED","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"PWD_POLICY_CHECK_LENGTH_ENABLED":{"name":"PWD_POLICY_CHECK_LENGTH_ENABLED","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"PWD_POLICY_CHECK_OLDPWD_ENABLED":{"name":"PWD_POLICY_CHECK_OLDPWD_ENABLED","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"PWD_POLICY_CHECK_REPETITIVE_CHARS_ENABLED":{"name":"PWD_POLICY_CHECK_REPETITIVE_CHARS_ENABLED","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"PWD_POLICY_CHECK_SEQUENTIAL_CHARS_ENABLED":{"name":"PWD_POLICY_CHECK_SEQUENTIAL_CHARS_ENABLED","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"PWD_POLICY_CHECK_USERNAME_ENABLED":{"name":"PWD_POLICY_CHECK_USERNAME_ENABLED","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"PWD_POLICY_ENABLED":{"name":"PWD_POLICY_ENABLED","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"PWD_POLICY_EXCLUSION_LIST":{"name":"PWD_POLICY_EXCLUSION_LIST","type":"str","level":"advanced","flags":0,"default_value":"osd,host,dashboard,pool,block,nfs,ceph,monitors,gateway,logs,crush,maps","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"PWD_POLICY_MIN_COMPLEXITY":{"name":"PWD_POLICY_MIN_COMPLEXITY","type":"int","level":"advanced","flags":0,"default_value":"10","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"PWD_POLICY_MIN_LENGTH":{"name":"PWD_POLICY_MIN_LENGTH","type":"int","level":"advanced","flags":0,"default_value":"8","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"REST_REQUESTS_TIMEOUT":{"name":"REST_REQUESTS_TIMEOUT","type":"int","level":"advanced","flags":0,"default_value":"45","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"RGW_API_ACCESS_KEY":{"name":"RGW_API_ACCESS_KEY","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"RGW_API_ADMIN_RESOURCE":{"name":"RGW_API_ADMIN_RESOURCE","type":"str","level":"advanced","flags":0,"default_value":"admin","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"RGW_API_SECRET_KEY":{"name":"RGW_API_SECRET_KEY","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"RGW_API_SSL_VERIFY":{"name":"RGW_API_SSL_VERIFY","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"UNSAFE_TLS_v1_2":{"name":"UNSAFE_TLS_v1_2","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"USER_PWD_EXPIRATION_SPAN":{"name":"USER_PWD_EXPIRATION_SPAN","type":"int","level":"advanced","flags":0,"default_value":"0","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"USER_PWD_EXPIRATION_WARNING_1":{"name":"USER_PWD_EXPIRATION_WARNING_1","type":"int","level":"advanced","flags":0,"default_value":"10","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"USER_PWD_EXPIRATION_WARNING_2":{"name":"USER_PWD_EXPIRATION_WARNING_2","type":"int","level":"advanced","flags":0,"default_value":"5","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"cross_origin_url":{"name":"cross_origin_url","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"crt_file":{"name":"crt_file","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"debug":{"name":"debug","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"Enable/disable debug options","long_desc":"","tags":[],"see_also":[]},"jwt_token_ttl":{"name":"jwt_token_ttl","type":"int","level":"advanced","flags":0,"default_value":"28800","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"key_file":{"name":"key_file","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"motd":{"name":"motd","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"The message of the day","long_desc":"","tags":[],"see_also":[]},"redirect_resolve_ip_addr":{"name":"redirect_resolve_ip_addr","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"server_addr":{"name":"server_addr","type":"str","level":"advanced","flags":0,"default_value":"::","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"server_port":{"name":"server_port","type":"int","level":"advanced","flags":0,"default_value":"8080","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"ssl":{"name":"ssl","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"ssl_server_port":{"name":"ssl_server_port","type":"int","level":"advanced","flags":0,"default_value":"8443","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"standby_behaviour":{"name":"standby_behaviour","type":"str","level":"advanced","flags":0,"default_value":"redirect","min":"","max":"","enum_allowed":["error","redirect"],"desc":"","long_desc":"","tags":[],"see_also":[]},"standby_error_status_code":{"name":"standby_error_status_code","type":"int","level":"advanced","flags":0,"default_value":"500","min":"400","max":"599","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"url_prefix":{"name":"url_prefix","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"devicehealth","can_run":true,"error_string":"","module_options":{"enable_monitoring":{"name":"enable_monitoring","type":"bool","level":"advanced","flags":1,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"monitor device health metrics","long_desc":"","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"mark_out_threshold":{"name":"mark_out_threshold","type":"secs","level":"advanced","flags":1,"default_value":"2419200","min":"","max":"","enum_allowed":[],"desc":"automatically mark OSD if it may fail before this long","long_desc":"","tags":[],"see_also":[]},"pool_name":{"name":"pool_name","type":"str","level":"advanced","flags":1,"default_value":"device_health_metrics","min":"","max":"","enum_allowed":[],"desc":"name of pool in which to store device health metrics","long_desc":"","tags":[],"see_also":[]},"retention_period":{"name":"retention_period","type":"secs","level":"advanced","flags":1,"default_value":"15552000","min":"","max":"","enum_allowed":[],"desc":"how long to retain device health metrics","long_desc":"","tags":[],"see_also":[]},"scrape_frequency":{"name":"scrape_frequency","type":"secs","level":"advanced","flags":1,"default_value":"86400","min":"","max":"","enum_allowed":[],"desc":"how frequently to scrape device health metrics","long_desc":"","tags":[],"see_also":[]},"self_heal":{"name":"self_heal","type":"bool","level":"advanced","flags":1,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"preemptively heal cluster around devices that may fail","long_desc":"","tags":[],"see_also":[]},"sleep_interval":{"name":"sleep_interval","type":"secs","level":"advanced","flags":1,"default_value":"600","min":"","max":"","enum_allowed":[],"desc":"how frequently to wake up and check device health","long_desc":"","tags":[],"see_also":[]},"warn_threshold":{"name":"warn_threshold","type":"secs","level":"advanced","flags":1,"default_value":"7257600","min":"","max":"","enum_allowed":[],"desc":"raise health warning if OSD may fail before this long","long_desc":"","tags":[],"see_also":[]}}},{"name":"diskprediction_local","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"predict_interval":{"name":"predict_interval","type":"str","level":"advanced","flags":0,"default_value":"86400","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"predictor_model":{"name":"predictor_model","type":"str","level":"advanced","flags":0,"default_value":"prophetstor","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"sleep_interval":{"name":"sleep_interval","type":"str","level":"advanced","flags":0,"default_value":"600","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"influx","can_run":false,"error_string":"influxdb python module not found","module_options":{"batch_size":{"name":"batch_size","type":"int","level":"advanced","flags":0,"default_value":"5000","min":"","max":"","enum_allowed":[],"desc":"How big batches of data points should be when sending to InfluxDB.","long_desc":"","tags":[],"see_also":[]},"database":{"name":"database","type":"str","level":"advanced","flags":0,"default_value":"ceph","min":"","max":"","enum_allowed":[],"desc":"InfluxDB database name. You will need to create this database and grant write privileges to the configured username or the username must have admin privileges to create it.","long_desc":"","tags":[],"see_also":[]},"hostname":{"name":"hostname","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"InfluxDB server hostname","long_desc":"","tags":[],"see_also":[]},"interval":{"name":"interval","type":"secs","level":"advanced","flags":0,"default_value":"30","min":"5","max":"","enum_allowed":[],"desc":"Time between reports to InfluxDB. Default 30 seconds.","long_desc":"","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"password":{"name":"password","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"password of InfluxDB server user","long_desc":"","tags":[],"see_also":[]},"port":{"name":"port","type":"int","level":"advanced","flags":0,"default_value":"8086","min":"","max":"","enum_allowed":[],"desc":"InfluxDB server port","long_desc":"","tags":[],"see_also":[]},"ssl":{"name":"ssl","type":"str","level":"advanced","flags":0,"default_value":"false","min":"","max":"","enum_allowed":[],"desc":"Use https connection for InfluxDB server. Use \"true\" or \"false\".","long_desc":"","tags":[],"see_also":[]},"threads":{"name":"threads","type":"int","level":"advanced","flags":0,"default_value":"5","min":"1","max":"32","enum_allowed":[],"desc":"How many worker threads should be spawned for sending data to InfluxDB.","long_desc":"","tags":[],"see_also":[]},"username":{"name":"username","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"username of InfluxDB server user","long_desc":"","tags":[],"see_also":[]},"verify_ssl":{"name":"verify_ssl","type":"str","level":"advanced","flags":0,"default_value":"true","min":"","max":"","enum_allowed":[],"desc":"Verify https cert for InfluxDB server. Use \"true\" or \"false\".","long_desc":"","tags":[],"see_also":[]}}},{"name":"insights","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"iostat","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"localpool","can_run":true,"error_string":"","module_options":{"failure_domain":{"name":"failure_domain","type":"str","level":"advanced","flags":1,"default_value":"host","min":"","max":"","enum_allowed":[],"desc":"failure domain for any created local pool","long_desc":"what failure domain we should separate data replicas across.","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"min_size":{"name":"min_size","type":"int","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"default min_size for any created local pool","long_desc":"value to set min_size to (unchanged from Ceph's default if this option is not set)","tags":[],"see_also":[]},"num_rep":{"name":"num_rep","type":"int","level":"advanced","flags":1,"default_value":"3","min":"","max":"","enum_allowed":[],"desc":"default replica count for any created local pool","long_desc":"","tags":[],"see_also":[]},"pg_num":{"name":"pg_num","type":"int","level":"advanced","flags":1,"default_value":"128","min":"","max":"","enum_allowed":[],"desc":"default pg_num for any created local pool","long_desc":"","tags":[],"see_also":[]},"prefix":{"name":"prefix","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"name prefix for any created local pool","long_desc":"","tags":[],"see_also":[]},"subtree":{"name":"subtree","type":"str","level":"advanced","flags":1,"default_value":"rack","min":"","max":"","enum_allowed":[],"desc":"CRUSH level for which to create a local pool","long_desc":"which CRUSH subtree type the module should create a pool for.","tags":[],"see_also":[]}}},{"name":"mds_autoscaler","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"mirroring","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"nfs","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"orchestrator","can_run":true,"error_string":"","module_options":{"fail_fs":{"name":"fail_fs","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"Fail filesystem for rapid multi-rank mds upgrade","long_desc":"","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"orchestrator":{"name":"orchestrator","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["cephadm","rook","test_orchestrator"],"desc":"Orchestrator backend","long_desc":"","tags":[],"see_also":[]}}},{"name":"osd_perf_query","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"osd_support","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"pg_autoscaler","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"sleep_interval":{"name":"sleep_interval","type":"secs","level":"advanced","flags":0,"default_value":"60","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"threshold":{"name":"threshold","type":"float","level":"advanced","flags":0,"default_value":"3.0","min":"1.0","max":"","enum_allowed":[],"desc":"scaling threshold","long_desc":"The factor by which the `NEW PG_NUM` must vary from the current`PG_NUM` before being accepted. Cannot be less than 1.0","tags":[],"see_also":[]}}},{"name":"progress","can_run":true,"error_string":"","module_options":{"allow_pg_recovery_event":{"name":"allow_pg_recovery_event","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"allow the module to show pg recovery progress","long_desc":"","tags":[],"see_also":[]},"enabled":{"name":"enabled","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"max_completed_events":{"name":"max_completed_events","type":"int","level":"advanced","flags":1,"default_value":"50","min":"","max":"","enum_allowed":[],"desc":"number of past completed events to remember","long_desc":"","tags":[],"see_also":[]},"sleep_interval":{"name":"sleep_interval","type":"secs","level":"advanced","flags":1,"default_value":"5","min":"","max":"","enum_allowed":[],"desc":"how long the module is going to sleep","long_desc":"","tags":[],"see_also":[]}}},{"name":"prometheus","can_run":true,"error_string":"","module_options":{"cache":{"name":"cache","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"exclude_perf_counters":{"name":"exclude_perf_counters","type":"bool","level":"advanced","flags":1,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"Do not include perf-counters in the metrics output","long_desc":"Gathering perf-counters from a single Prometheus exporter can degrade ceph-mgr performance, especially in large clusters. Instead, Ceph-exporter daemons are now used by default for perf-counter gathering. This should only be disabled when no ceph-exporters are deployed.","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"rbd_stats_pools":{"name":"rbd_stats_pools","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"rbd_stats_pools_refresh_interval":{"name":"rbd_stats_pools_refresh_interval","type":"int","level":"advanced","flags":0,"default_value":"300","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"scrape_interval":{"name":"scrape_interval","type":"float","level":"advanced","flags":0,"default_value":"15.0","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"server_addr":{"name":"server_addr","type":"str","level":"advanced","flags":0,"default_value":"::","min":"","max":"","enum_allowed":[],"desc":"the IPv4 or IPv6 address on which the module listens for HTTP requests","long_desc":"","tags":[],"see_also":[]},"server_port":{"name":"server_port","type":"int","level":"advanced","flags":1,"default_value":"9283","min":"","max":"","enum_allowed":[],"desc":"the port on which the module listens for HTTP requests","long_desc":"","tags":[],"see_also":[]},"stale_cache_strategy":{"name":"stale_cache_strategy","type":"str","level":"advanced","flags":0,"default_value":"log","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"standby_behaviour":{"name":"standby_behaviour","type":"str","level":"advanced","flags":1,"default_value":"default","min":"","max":"","enum_allowed":["default","error"],"desc":"","long_desc":"","tags":[],"see_also":[]},"standby_error_status_code":{"name":"standby_error_status_code","type":"int","level":"advanced","flags":1,"default_value":"500","min":"400","max":"599","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"rbd_support","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"max_concurrent_snap_create":{"name":"max_concurrent_snap_create","type":"int","level":"advanced","flags":0,"default_value":"10","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"mirror_snapshot_schedule":{"name":"mirror_snapshot_schedule","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"trash_purge_schedule":{"name":"trash_purge_schedule","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"restful","can_run":true,"error_string":"","module_options":{"enable_auth":{"name":"enable_auth","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"key_file":{"name":"key_file","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"server_addr":{"name":"server_addr","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"server_port":{"name":"server_port","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"rgw","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"rook","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"storage_class":{"name":"storage_class","type":"str","level":"advanced","flags":0,"default_value":"local","min":"","max":"","enum_allowed":[],"desc":"storage class name for LSO-discovered PVs","long_desc":"","tags":[],"see_also":[]}}},{"name":"selftest","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"roption1":{"name":"roption1","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"roption2":{"name":"roption2","type":"str","level":"advanced","flags":0,"default_value":"xyz","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"rwoption1":{"name":"rwoption1","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"rwoption2":{"name":"rwoption2","type":"int","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"rwoption3":{"name":"rwoption3","type":"float","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"rwoption4":{"name":"rwoption4","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"rwoption5":{"name":"rwoption5","type":"bool","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"rwoption6":{"name":"rwoption6","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"rwoption7":{"name":"rwoption7","type":"int","level":"advanced","flags":0,"default_value":"","min":"1","max":"42","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"testkey":{"name":"testkey","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"testlkey":{"name":"testlkey","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"testnewline":{"name":"testnewline","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"snap_schedule","can_run":true,"error_string":"","module_options":{"allow_m_granularity":{"name":"allow_m_granularity","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"allow minute scheduled snapshots","long_desc":"","tags":[],"see_also":[]},"dump_on_update":{"name":"dump_on_update","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"dump database to debug log on update","long_desc":"","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"stats","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"status","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"telegraf","can_run":true,"error_string":"","module_options":{"address":{"name":"address","type":"str","level":"advanced","flags":0,"default_value":"unixgram:///tmp/telegraf.sock","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"interval":{"name":"interval","type":"secs","level":"advanced","flags":0,"default_value":"15","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"telemetry","can_run":true,"error_string":"","module_options":{"channel_basic":{"name":"channel_basic","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"Share basic cluster information (size, version)","long_desc":"","tags":[],"see_also":[]},"channel_crash":{"name":"channel_crash","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"Share metadata about Ceph daemon crashes (version, stack straces, etc)","long_desc":"","tags":[],"see_also":[]},"channel_device":{"name":"channel_device","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"Share device health metrics (e.g., SMART data, minus potentially identifying info like serial numbers)","long_desc":"","tags":[],"see_also":[]},"channel_ident":{"name":"channel_ident","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"Share a user-provided description and/or contact email for the cluster","long_desc":"","tags":[],"see_also":[]},"channel_perf":{"name":"channel_perf","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"Share various performance metrics of a cluster","long_desc":"","tags":[],"see_also":[]},"contact":{"name":"contact","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"description":{"name":"description","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"device_url":{"name":"device_url","type":"str","level":"advanced","flags":0,"default_value":"https://telemetry.ceph.com/device","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"enabled":{"name":"enabled","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"interval":{"name":"interval","type":"int","level":"advanced","flags":0,"default_value":"24","min":"8","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"last_opt_revision":{"name":"last_opt_revision","type":"int","level":"advanced","flags":0,"default_value":"1","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"leaderboard":{"name":"leaderboard","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"leaderboard_description":{"name":"leaderboard_description","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"organization":{"name":"organization","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"proxy":{"name":"proxy","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"url":{"name":"url","type":"str","level":"advanced","flags":0,"default_value":"https://telemetry.ceph.com/report","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"test_orchestrator","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"volumes","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"max_concurrent_clones":{"name":"max_concurrent_clones","type":"int","level":"advanced","flags":0,"default_value":"4","min":"","max":"","enum_allowed":[],"desc":"Number of asynchronous cloner threads","long_desc":"","tags":[],"see_also":[]},"snapshot_clone_delay":{"name":"snapshot_clone_delay","type":"int","level":"advanced","flags":0,"default_value":"0","min":"","max":"","enum_allowed":[],"desc":"Delay clone begin operation by snapshot_clone_delay seconds","long_desc":"","tags":[],"see_also":[]},"snapshot_clone_no_wait":{"name":"snapshot_clone_no_wait","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"Reject subvolume clone request when cloner threads are busy","long_desc":"","tags":[],"see_also":[]}}},{"name":"zabbix","can_run":true,"error_string":"","module_options":{"discovery_interval":{"name":"discovery_interval","type":"uint","level":"advanced","flags":0,"default_value":"100","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"identifier":{"name":"identifier","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"interval":{"name":"interval","type":"secs","level":"advanced","flags":0,"default_value":"60","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"zabbix_host":{"name":"zabbix_host","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"zabbix_port":{"name":"zabbix_port","type":"int","level":"advanced","flags":0,"default_value":"10051","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"zabbix_sender":{"name":"zabbix_sender","type":"str","level":"advanced","flags":0,"default_value":"/usr/bin/zabbix_sender","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}}]}],"modules":["iostat","nfs","restful"],"available_modules":[{"name":"alerts","can_run":true,"error_string":"","module_options":{"interval":{"name":"interval","type":"secs","level":"advanced","flags":1,"default_value":"60","min":"","max":"","enum_allowed":[],"desc":"How frequently to reexamine health status","long_desc":"","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"smtp_destination":{"name":"smtp_destination","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"Email address to send alerts to","long_desc":"","tags":[],"see_also":[]},"smtp_from_name":{"name":"smtp_from_name","type":"str","level":"advanced","flags":1,"default_value":"Ceph","min":"","max":"","enum_allowed":[],"desc":"Email From: name","long_desc":"","tags":[],"see_also":[]},"smtp_host":{"name":"smtp_host","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"SMTP server","long_desc":"","tags":[],"see_also":[]},"smtp_password":{"name":"smtp_password","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"Password to authenticate with","long_desc":"","tags":[],"see_also":[]},"smtp_port":{"name":"smtp_port","type":"int","level":"advanced","flags":1,"default_value":"465","min":"","max":"","enum_allowed":[],"desc":"SMTP port","long_desc":"","tags":[],"see_also":[]},"smtp_sender":{"name":"smtp_sender","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"SMTP envelope sender","long_desc":"","tags":[],"see_also":[]},"smtp_ssl":{"name":"smtp_ssl","type":"bool","level":"advanced","flags":1,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"Use SSL to connect to SMTP server","long_desc":"","tags":[],"see_also":[]},"smtp_user":{"name":"smtp_user","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"User to authenticate as","long_desc":"","tags":[],"see_also":[]}}},{"name":"balancer","can_run":true,"error_string":"","module_options":{"active":{"name":"active","type":"bool","level":"advanced","flags":1,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"automatically balance PGs across cluster","long_desc":"","tags":[],"see_also":[]},"begin_time":{"name":"begin_time","type":"str","level":"advanced","flags":1,"default_value":"0000","min":"","max":"","enum_allowed":[],"desc":"beginning time of day to automatically balance","long_desc":"This is a time of day in the format HHMM.","tags":[],"see_also":[]},"begin_weekday":{"name":"begin_weekday","type":"uint","level":"advanced","flags":1,"default_value":"0","min":"0","max":"6","enum_allowed":[],"desc":"Restrict automatic balancing to this day of the week or later","long_desc":"0 = Sunday, 1 = Monday, etc.","tags":[],"see_also":[]},"crush_compat_max_iterations":{"name":"crush_compat_max_iterations","type":"uint","level":"advanced","flags":1,"default_value":"25","min":"1","max":"250","enum_allowed":[],"desc":"maximum number of iterations to attempt optimization","long_desc":"","tags":[],"see_also":[]},"crush_compat_metrics":{"name":"crush_compat_metrics","type":"str","level":"advanced","flags":1,"default_value":"pgs,objects,bytes","min":"","max":"","enum_allowed":[],"desc":"metrics with which to calculate OSD utilization","long_desc":"Value is a list of one or more of \"pgs\", \"objects\", or \"bytes\", and indicates which metrics to use to balance utilization.","tags":[],"see_also":[]},"crush_compat_step":{"name":"crush_compat_step","type":"float","level":"advanced","flags":1,"default_value":"0.5","min":"0.001","max":"0.999","enum_allowed":[],"desc":"aggressiveness of optimization","long_desc":".99 is very aggressive, .01 is less aggressive","tags":[],"see_also":[]},"end_time":{"name":"end_time","type":"str","level":"advanced","flags":1,"default_value":"2359","min":"","max":"","enum_allowed":[],"desc":"ending time of day to automatically balance","long_desc":"This is a time of day in the format HHMM.","tags":[],"see_also":[]},"end_weekday":{"name":"end_weekday","type":"uint","level":"advanced","flags":1,"default_value":"0","min":"0","max":"6","enum_allowed":[],"desc":"Restrict automatic balancing to days of the week earlier than this","long_desc":"0 = Sunday, 1 = Monday, etc.","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"min_score":{"name":"min_score","type":"float","level":"advanced","flags":1,"default_value":"0","min":"","max":"","enum_allowed":[],"desc":"minimum score, below which no optimization is attempted","long_desc":"","tags":[],"see_also":[]},"mode":{"name":"mode","type":"str","level":"advanced","flags":1,"default_value":"upmap","min":"","max":"","enum_allowed":["crush-compat","none","upmap"],"desc":"Balancer mode","long_desc":"","tags":[],"see_also":[]},"pool_ids":{"name":"pool_ids","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"pools which the automatic balancing will be limited to","long_desc":"","tags":[],"see_also":[]},"sleep_interval":{"name":"sleep_interval","type":"secs","level":"advanced","flags":1,"default_value":"60","min":"","max":"","enum_allowed":[],"desc":"how frequently to wake up and attempt optimization","long_desc":"","tags":[],"see_also":[]},"upmap_max_deviation":{"name":"upmap_max_deviation","type":"int","level":"advanced","flags":1,"default_value":"5","min":"1","max":"","enum_allowed":[],"desc":"deviation below which no optimization is attempted","long_desc":"If the number of PGs are within this count then no optimization is attempted","tags":[],"see_also":[]},"upmap_max_optimizations":{"name":"upmap_max_optimizations","type":"uint","level":"advanced","flags":1,"default_value":"10","min":"","max":"","enum_allowed":[],"desc":"maximum upmap optimizations to make per attempt","long_desc":"","tags":[],"see_also":[]}}},{"name":"cephadm","can_run":true,"error_string":"","module_options":{"agent_down_multiplier":{"name":"agent_down_multiplier","type":"float","level":"advanced","flags":0,"default_value":"3.0","min":"","max":"","enum_allowed":[],"desc":"Multiplied by agent refresh rate to calculate how long agent must not report before being marked down","long_desc":"","tags":[],"see_also":[]},"agent_refresh_rate":{"name":"agent_refresh_rate","type":"secs","level":"advanced","flags":0,"default_value":"20","min":"","max":"","enum_allowed":[],"desc":"How often agent on each host will try to gather and send metadata","long_desc":"","tags":[],"see_also":[]},"agent_starting_port":{"name":"agent_starting_port","type":"int","level":"advanced","flags":0,"default_value":"4721","min":"","max":"","enum_allowed":[],"desc":"First port agent will try to bind to (will also try up to next 1000 subsequent ports if blocked)","long_desc":"","tags":[],"see_also":[]},"allow_ptrace":{"name":"allow_ptrace","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"allow SYS_PTRACE capability on ceph containers","long_desc":"The SYS_PTRACE capability is needed to attach to a process with gdb or strace. Enabling this options can allow debugging daemons that encounter problems at runtime.","tags":[],"see_also":[]},"autotune_interval":{"name":"autotune_interval","type":"secs","level":"advanced","flags":0,"default_value":"600","min":"","max":"","enum_allowed":[],"desc":"how frequently to autotune daemon memory","long_desc":"","tags":[],"see_also":[]},"autotune_memory_target_ratio":{"name":"autotune_memory_target_ratio","type":"float","level":"advanced","flags":0,"default_value":"0.7","min":"","max":"","enum_allowed":[],"desc":"ratio of total system memory to divide amongst autotuned daemons","long_desc":"","tags":[],"see_also":[]},"cgroups_split":{"name":"cgroups_split","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"Pass --cgroups=split when cephadm creates containers (currently podman only)","long_desc":"","tags":[],"see_also":[]},"config_checks_enabled":{"name":"config_checks_enabled","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"Enable or disable the cephadm configuration analysis","long_desc":"","tags":[],"see_also":[]},"config_dashboard":{"name":"config_dashboard","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"manage configs like API endpoints in Dashboard.","long_desc":"","tags":[],"see_also":[]},"container_image_alertmanager":{"name":"container_image_alertmanager","type":"str","level":"advanced","flags":0,"default_value":"quay.io/prometheus/alertmanager:v0.25.0","min":"","max":"","enum_allowed":[],"desc":"Prometheus container image","long_desc":"","tags":[],"see_also":[]},"container_image_base":{"name":"container_image_base","type":"str","level":"advanced","flags":1,"default_value":"quay.io/ceph/ceph","min":"","max":"","enum_allowed":[],"desc":"Container image name, without the tag","long_desc":"","tags":[],"see_also":[]},"container_image_elasticsearch":{"name":"container_image_elasticsearch","type":"str","level":"advanced","flags":0,"default_value":"quay.io/omrizeneva/elasticsearch:6.8.23","min":"","max":"","enum_allowed":[],"desc":"elasticsearch container image","long_desc":"","tags":[],"see_also":[]},"container_image_grafana":{"name":"container_image_grafana","type":"str","level":"advanced","flags":0,"default_value":"quay.io/ceph/ceph-grafana:9.4.7","min":"","max":"","enum_allowed":[],"desc":"Prometheus container image","long_desc":"","tags":[],"see_also":[]},"container_image_haproxy":{"name":"container_image_haproxy","type":"str","level":"advanced","flags":0,"default_value":"quay.io/ceph/haproxy:2.3","min":"","max":"","enum_allowed":[],"desc":"HAproxy container image","long_desc":"","tags":[],"see_also":[]},"container_image_jaeger_agent":{"name":"container_image_jaeger_agent","type":"str","level":"advanced","flags":0,"default_value":"quay.io/jaegertracing/jaeger-agent:1.29","min":"","max":"","enum_allowed":[],"desc":"Jaeger agent container image","long_desc":"","tags":[],"see_also":[]},"container_image_jaeger_collector":{"name":"container_image_jaeger_collector","type":"str","level":"advanced","flags":0,"default_value":"quay.io/jaegertracing/jaeger-collector:1.29","min":"","max":"","enum_allowed":[],"desc":"Jaeger collector container image","long_desc":"","tags":[],"see_also":[]},"container_image_jaeger_query":{"name":"container_image_jaeger_query","type":"str","level":"advanced","flags":0,"default_value":"quay.io/jaegertracing/jaeger-query:1.29","min":"","max":"","enum_allowed":[],"desc":"Jaeger query container image","long_desc":"","tags":[],"see_also":[]},"container_image_keepalived":{"name":"container_image_keepalived","type":"str","level":"advanced","flags":0,"default_value":"quay.io/ceph/keepalived:2.2.4","min":"","max":"","enum_allowed":[],"desc":"Keepalived container image","long_desc":"","tags":[],"see_also":[]},"container_image_loki":{"name":"container_image_loki","type":"str","level":"advanced","flags":0,"default_value":"docker.io/grafana/loki:2.4.0","min":"","max":"","enum_allowed":[],"desc":"Loki container image","long_desc":"","tags":[],"see_also":[]},"container_image_node_exporter":{"name":"container_image_node_exporter","type":"str","level":"advanced","flags":0,"default_value":"quay.io/prometheus/node-exporter:v1.5.0","min":"","max":"","enum_allowed":[],"desc":"Prometheus container image","long_desc":"","tags":[],"see_also":[]},"container_image_nvmeof":{"name":"container_image_nvmeof","type":"str","level":"advanced","flags":0,"default_value":"quay.io/ceph/nvmeof:1.0.0","min":"","max":"","enum_allowed":[],"desc":"Nvme-of container image","long_desc":"","tags":[],"see_also":[]},"container_image_prometheus":{"name":"container_image_prometheus","type":"str","level":"advanced","flags":0,"default_value":"quay.io/prometheus/prometheus:v2.43.0","min":"","max":"","enum_allowed":[],"desc":"Prometheus container image","long_desc":"","tags":[],"see_also":[]},"container_image_promtail":{"name":"container_image_promtail","type":"str","level":"advanced","flags":0,"default_value":"docker.io/grafana/promtail:2.4.0","min":"","max":"","enum_allowed":[],"desc":"Promtail container image","long_desc":"","tags":[],"see_also":[]},"container_image_snmp_gateway":{"name":"container_image_snmp_gateway","type":"str","level":"advanced","flags":0,"default_value":"docker.io/maxwo/snmp-notifier:v1.2.1","min":"","max":"","enum_allowed":[],"desc":"SNMP Gateway container image","long_desc":"","tags":[],"see_also":[]},"container_init":{"name":"container_init","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"Run podman/docker with `--init`","long_desc":"","tags":[],"see_also":[]},"daemon_cache_timeout":{"name":"daemon_cache_timeout","type":"secs","level":"advanced","flags":0,"default_value":"600","min":"","max":"","enum_allowed":[],"desc":"seconds to cache service (daemon) inventory","long_desc":"","tags":[],"see_also":[]},"default_cephadm_command_timeout":{"name":"default_cephadm_command_timeout","type":"int","level":"advanced","flags":0,"default_value":"900","min":"","max":"","enum_allowed":[],"desc":"Default timeout applied to cephadm commands run directly on the host (in seconds)","long_desc":"","tags":[],"see_also":[]},"default_registry":{"name":"default_registry","type":"str","level":"advanced","flags":0,"default_value":"docker.io","min":"","max":"","enum_allowed":[],"desc":"Search-registry to which we should normalize unqualified image names. This is not the default registry","long_desc":"","tags":[],"see_also":[]},"device_cache_timeout":{"name":"device_cache_timeout","type":"secs","level":"advanced","flags":0,"default_value":"1800","min":"","max":"","enum_allowed":[],"desc":"seconds to cache device inventory","long_desc":"","tags":[],"see_also":[]},"device_enhanced_scan":{"name":"device_enhanced_scan","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"Use libstoragemgmt during device scans","long_desc":"","tags":[],"see_also":[]},"facts_cache_timeout":{"name":"facts_cache_timeout","type":"secs","level":"advanced","flags":0,"default_value":"60","min":"","max":"","enum_allowed":[],"desc":"seconds to cache host facts data","long_desc":"","tags":[],"see_also":[]},"host_check_interval":{"name":"host_check_interval","type":"secs","level":"advanced","flags":0,"default_value":"600","min":"","max":"","enum_allowed":[],"desc":"how frequently to perform a host check","long_desc":"","tags":[],"see_also":[]},"hw_monitoring":{"name":"hw_monitoring","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"Deploy hw monitoring daemon on every host.","long_desc":"","tags":[],"see_also":[]},"inventory_list_all":{"name":"inventory_list_all","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"Whether ceph-volume inventory should report more devices (mostly mappers (LVs / mpaths), partitions...)","long_desc":"","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_refresh_metadata":{"name":"log_refresh_metadata","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"Log all refresh metadata. Includes daemon, device, and host info collected regularly. Only has effect if logging at debug level","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"log to the \"cephadm\" cluster log channel\"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"manage_etc_ceph_ceph_conf":{"name":"manage_etc_ceph_ceph_conf","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"Manage and own /etc/ceph/ceph.conf on the hosts.","long_desc":"","tags":[],"see_also":[]},"manage_etc_ceph_ceph_conf_hosts":{"name":"manage_etc_ceph_ceph_conf_hosts","type":"str","level":"advanced","flags":0,"default_value":"*","min":"","max":"","enum_allowed":[],"desc":"PlacementSpec describing on which hosts to manage /etc/ceph/ceph.conf","long_desc":"","tags":[],"see_also":[]},"max_count_per_host":{"name":"max_count_per_host","type":"int","level":"advanced","flags":0,"default_value":"10","min":"","max":"","enum_allowed":[],"desc":"max number of daemons per service per host","long_desc":"","tags":[],"see_also":[]},"max_osd_draining_count":{"name":"max_osd_draining_count","type":"int","level":"advanced","flags":0,"default_value":"10","min":"","max":"","enum_allowed":[],"desc":"max number of osds that will be drained simultaneously when osds are removed","long_desc":"","tags":[],"see_also":[]},"migration_current":{"name":"migration_current","type":"int","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"internal - do not modify","long_desc":"","tags":[],"see_also":[]},"mode":{"name":"mode","type":"str","level":"advanced","flags":0,"default_value":"root","min":"","max":"","enum_allowed":["cephadm-package","root"],"desc":"mode for remote execution of cephadm","long_desc":"","tags":[],"see_also":[]},"oob_default_addr":{"name":"oob_default_addr","type":"str","level":"advanced","flags":0,"default_value":"169.254.1.1","min":"","max":"","enum_allowed":[],"desc":"Default address for RedFish API (oob management).","long_desc":"","tags":[],"see_also":[]},"prometheus_alerts_path":{"name":"prometheus_alerts_path","type":"str","level":"advanced","flags":0,"default_value":"/etc/prometheus/ceph/ceph_default_alerts.yml","min":"","max":"","enum_allowed":[],"desc":"location of alerts to include in prometheus deployments","long_desc":"","tags":[],"see_also":[]},"registry_insecure":{"name":"registry_insecure","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"Registry is to be considered insecure (no TLS available). Only for development purposes.","long_desc":"","tags":[],"see_also":[]},"registry_password":{"name":"registry_password","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"Custom repository password. Only used for logging into a registry.","long_desc":"","tags":[],"see_also":[]},"registry_url":{"name":"registry_url","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"Registry url for login purposes. This is not the default registry","long_desc":"","tags":[],"see_also":[]},"registry_username":{"name":"registry_username","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"Custom repository username. Only used for logging into a registry.","long_desc":"","tags":[],"see_also":[]},"secure_monitoring_stack":{"name":"secure_monitoring_stack","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"Enable TLS security for all the monitoring stack daemons","long_desc":"","tags":[],"see_also":[]},"service_discovery_port":{"name":"service_discovery_port","type":"int","level":"advanced","flags":0,"default_value":"8765","min":"","max":"","enum_allowed":[],"desc":"cephadm service discovery port","long_desc":"","tags":[],"see_also":[]},"ssh_config_file":{"name":"ssh_config_file","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"customized SSH config file to connect to managed hosts","long_desc":"","tags":[],"see_also":[]},"use_agent":{"name":"use_agent","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"Use cephadm agent on each host to gather and send metadata","long_desc":"","tags":[],"see_also":[]},"use_repo_digest":{"name":"use_repo_digest","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"Automatically convert image tags to image digest. Make sure all daemons use the same image","long_desc":"","tags":[],"see_also":[]},"warn_on_failed_host_check":{"name":"warn_on_failed_host_check","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"raise a health warning if the host check fails","long_desc":"","tags":[],"see_also":[]},"warn_on_stray_daemons":{"name":"warn_on_stray_daemons","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"raise a health warning if daemons are detected that are not managed by cephadm","long_desc":"","tags":[],"see_also":[]},"warn_on_stray_hosts":{"name":"warn_on_stray_hosts","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"raise a health warning if daemons are detected on a host that is not managed by cephadm","long_desc":"","tags":[],"see_also":[]}}},{"name":"crash","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"retain_interval":{"name":"retain_interval","type":"secs","level":"advanced","flags":1,"default_value":"31536000","min":"","max":"","enum_allowed":[],"desc":"how long to retain crashes before pruning them","long_desc":"","tags":[],"see_also":[]},"warn_recent_interval":{"name":"warn_recent_interval","type":"secs","level":"advanced","flags":1,"default_value":"1209600","min":"","max":"","enum_allowed":[],"desc":"time interval in which to warn about recent crashes","long_desc":"","tags":[],"see_also":[]}}},{"name":"dashboard","can_run":true,"error_string":"","module_options":{"ACCOUNT_LOCKOUT_ATTEMPTS":{"name":"ACCOUNT_LOCKOUT_ATTEMPTS","type":"int","level":"advanced","flags":0,"default_value":"10","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"ALERTMANAGER_API_HOST":{"name":"ALERTMANAGER_API_HOST","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"ALERTMANAGER_API_SSL_VERIFY":{"name":"ALERTMANAGER_API_SSL_VERIFY","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"AUDIT_API_ENABLED":{"name":"AUDIT_API_ENABLED","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"AUDIT_API_LOG_PAYLOAD":{"name":"AUDIT_API_LOG_PAYLOAD","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"ENABLE_BROWSABLE_API":{"name":"ENABLE_BROWSABLE_API","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"FEATURE_TOGGLE_CEPHFS":{"name":"FEATURE_TOGGLE_CEPHFS","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"FEATURE_TOGGLE_DASHBOARD":{"name":"FEATURE_TOGGLE_DASHBOARD","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"FEATURE_TOGGLE_ISCSI":{"name":"FEATURE_TOGGLE_ISCSI","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"FEATURE_TOGGLE_MIRRORING":{"name":"FEATURE_TOGGLE_MIRRORING","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"FEATURE_TOGGLE_NFS":{"name":"FEATURE_TOGGLE_NFS","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"FEATURE_TOGGLE_RBD":{"name":"FEATURE_TOGGLE_RBD","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"FEATURE_TOGGLE_RGW":{"name":"FEATURE_TOGGLE_RGW","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"GANESHA_CLUSTERS_RADOS_POOL_NAMESPACE":{"name":"GANESHA_CLUSTERS_RADOS_POOL_NAMESPACE","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"GRAFANA_API_PASSWORD":{"name":"GRAFANA_API_PASSWORD","type":"str","level":"advanced","flags":0,"default_value":"admin","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"GRAFANA_API_SSL_VERIFY":{"name":"GRAFANA_API_SSL_VERIFY","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"GRAFANA_API_URL":{"name":"GRAFANA_API_URL","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"GRAFANA_API_USERNAME":{"name":"GRAFANA_API_USERNAME","type":"str","level":"advanced","flags":0,"default_value":"admin","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"GRAFANA_FRONTEND_API_URL":{"name":"GRAFANA_FRONTEND_API_URL","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"GRAFANA_UPDATE_DASHBOARDS":{"name":"GRAFANA_UPDATE_DASHBOARDS","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"ISCSI_API_SSL_VERIFICATION":{"name":"ISCSI_API_SSL_VERIFICATION","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"ISSUE_TRACKER_API_KEY":{"name":"ISSUE_TRACKER_API_KEY","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"PROMETHEUS_API_HOST":{"name":"PROMETHEUS_API_HOST","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"PROMETHEUS_API_SSL_VERIFY":{"name":"PROMETHEUS_API_SSL_VERIFY","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"PWD_POLICY_CHECK_COMPLEXITY_ENABLED":{"name":"PWD_POLICY_CHECK_COMPLEXITY_ENABLED","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"PWD_POLICY_CHECK_EXCLUSION_LIST_ENABLED":{"name":"PWD_POLICY_CHECK_EXCLUSION_LIST_ENABLED","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"PWD_POLICY_CHECK_LENGTH_ENABLED":{"name":"PWD_POLICY_CHECK_LENGTH_ENABLED","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"PWD_POLICY_CHECK_OLDPWD_ENABLED":{"name":"PWD_POLICY_CHECK_OLDPWD_ENABLED","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"PWD_POLICY_CHECK_REPETITIVE_CHARS_ENABLED":{"name":"PWD_POLICY_CHECK_REPETITIVE_CHARS_ENABLED","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"PWD_POLICY_CHECK_SEQUENTIAL_CHARS_ENABLED":{"name":"PWD_POLICY_CHECK_SEQUENTIAL_CHARS_ENABLED","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"PWD_POLICY_CHECK_USERNAME_ENABLED":{"name":"PWD_POLICY_CHECK_USERNAME_ENABLED","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"PWD_POLICY_ENABLED":{"name":"PWD_POLICY_ENABLED","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"PWD_POLICY_EXCLUSION_LIST":{"name":"PWD_POLICY_EXCLUSION_LIST","type":"str","level":"advanced","flags":0,"default_value":"osd,host,dashboard,pool,block,nfs,ceph,monitors,gateway,logs,crush,maps","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"PWD_POLICY_MIN_COMPLEXITY":{"name":"PWD_POLICY_MIN_COMPLEXITY","type":"int","level":"advanced","flags":0,"default_value":"10","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"PWD_POLICY_MIN_LENGTH":{"name":"PWD_POLICY_MIN_LENGTH","type":"int","level":"advanced","flags":0,"default_value":"8","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"REST_REQUESTS_TIMEOUT":{"name":"REST_REQUESTS_TIMEOUT","type":"int","level":"advanced","flags":0,"default_value":"45","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"RGW_API_ACCESS_KEY":{"name":"RGW_API_ACCESS_KEY","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"RGW_API_ADMIN_RESOURCE":{"name":"RGW_API_ADMIN_RESOURCE","type":"str","level":"advanced","flags":0,"default_value":"admin","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"RGW_API_SECRET_KEY":{"name":"RGW_API_SECRET_KEY","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"RGW_API_SSL_VERIFY":{"name":"RGW_API_SSL_VERIFY","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"UNSAFE_TLS_v1_2":{"name":"UNSAFE_TLS_v1_2","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"USER_PWD_EXPIRATION_SPAN":{"name":"USER_PWD_EXPIRATION_SPAN","type":"int","level":"advanced","flags":0,"default_value":"0","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"USER_PWD_EXPIRATION_WARNING_1":{"name":"USER_PWD_EXPIRATION_WARNING_1","type":"int","level":"advanced","flags":0,"default_value":"10","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"USER_PWD_EXPIRATION_WARNING_2":{"name":"USER_PWD_EXPIRATION_WARNING_2","type":"int","level":"advanced","flags":0,"default_value":"5","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"cross_origin_url":{"name":"cross_origin_url","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"crt_file":{"name":"crt_file","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"debug":{"name":"debug","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"Enable/disable debug options","long_desc":"","tags":[],"see_also":[]},"jwt_token_ttl":{"name":"jwt_token_ttl","type":"int","level":"advanced","flags":0,"default_value":"28800","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"key_file":{"name":"key_file","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"motd":{"name":"motd","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"The message of the day","long_desc":"","tags":[],"see_also":[]},"redirect_resolve_ip_addr":{"name":"redirect_resolve_ip_addr","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"server_addr":{"name":"server_addr","type":"str","level":"advanced","flags":0,"default_value":"::","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"server_port":{"name":"server_port","type":"int","level":"advanced","flags":0,"default_value":"8080","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"ssl":{"name":"ssl","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"ssl_server_port":{"name":"ssl_server_port","type":"int","level":"advanced","flags":0,"default_value":"8443","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"standby_behaviour":{"name":"standby_behaviour","type":"str","level":"advanced","flags":0,"default_value":"redirect","min":"","max":"","enum_allowed":["error","redirect"],"desc":"","long_desc":"","tags":[],"see_also":[]},"standby_error_status_code":{"name":"standby_error_status_code","type":"int","level":"advanced","flags":0,"default_value":"500","min":"400","max":"599","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"url_prefix":{"name":"url_prefix","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"devicehealth","can_run":true,"error_string":"","module_options":{"enable_monitoring":{"name":"enable_monitoring","type":"bool","level":"advanced","flags":1,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"monitor device health metrics","long_desc":"","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"mark_out_threshold":{"name":"mark_out_threshold","type":"secs","level":"advanced","flags":1,"default_value":"2419200","min":"","max":"","enum_allowed":[],"desc":"automatically mark OSD if it may fail before this long","long_desc":"","tags":[],"see_also":[]},"pool_name":{"name":"pool_name","type":"str","level":"advanced","flags":1,"default_value":"device_health_metrics","min":"","max":"","enum_allowed":[],"desc":"name of pool in which to store device health metrics","long_desc":"","tags":[],"see_also":[]},"retention_period":{"name":"retention_period","type":"secs","level":"advanced","flags":1,"default_value":"15552000","min":"","max":"","enum_allowed":[],"desc":"how long to retain device health metrics","long_desc":"","tags":[],"see_also":[]},"scrape_frequency":{"name":"scrape_frequency","type":"secs","level":"advanced","flags":1,"default_value":"86400","min":"","max":"","enum_allowed":[],"desc":"how frequently to scrape device health metrics","long_desc":"","tags":[],"see_also":[]},"self_heal":{"name":"self_heal","type":"bool","level":"advanced","flags":1,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"preemptively heal cluster around devices that may fail","long_desc":"","tags":[],"see_also":[]},"sleep_interval":{"name":"sleep_interval","type":"secs","level":"advanced","flags":1,"default_value":"600","min":"","max":"","enum_allowed":[],"desc":"how frequently to wake up and check device health","long_desc":"","tags":[],"see_also":[]},"warn_threshold":{"name":"warn_threshold","type":"secs","level":"advanced","flags":1,"default_value":"7257600","min":"","max":"","enum_allowed":[],"desc":"raise health warning if OSD may fail before this long","long_desc":"","tags":[],"see_also":[]}}},{"name":"diskprediction_local","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"predict_interval":{"name":"predict_interval","type":"str","level":"advanced","flags":0,"default_value":"86400","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"predictor_model":{"name":"predictor_model","type":"str","level":"advanced","flags":0,"default_value":"prophetstor","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"sleep_interval":{"name":"sleep_interval","type":"str","level":"advanced","flags":0,"default_value":"600","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"influx","can_run":false,"error_string":"influxdb python module not found","module_options":{"batch_size":{"name":"batch_size","type":"int","level":"advanced","flags":0,"default_value":"5000","min":"","max":"","enum_allowed":[],"desc":"How big batches of data points should be when sending to InfluxDB.","long_desc":"","tags":[],"see_also":[]},"database":{"name":"database","type":"str","level":"advanced","flags":0,"default_value":"ceph","min":"","max":"","enum_allowed":[],"desc":"InfluxDB database name. You will need to create this database and grant write privileges to the configured username or the username must have admin privileges to create it.","long_desc":"","tags":[],"see_also":[]},"hostname":{"name":"hostname","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"InfluxDB server hostname","long_desc":"","tags":[],"see_also":[]},"interval":{"name":"interval","type":"secs","level":"advanced","flags":0,"default_value":"30","min":"5","max":"","enum_allowed":[],"desc":"Time between reports to InfluxDB. Default 30 seconds.","long_desc":"","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"password":{"name":"password","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"password of InfluxDB server user","long_desc":"","tags":[],"see_also":[]},"port":{"name":"port","type":"int","level":"advanced","flags":0,"default_value":"8086","min":"","max":"","enum_allowed":[],"desc":"InfluxDB server port","long_desc":"","tags":[],"see_also":[]},"ssl":{"name":"ssl","type":"str","level":"advanced","flags":0,"default_value":"false","min":"","max":"","enum_allowed":[],"desc":"Use https connection for InfluxDB server. Use \"true\" or \"false\".","long_desc":"","tags":[],"see_also":[]},"threads":{"name":"threads","type":"int","level":"advanced","flags":0,"default_value":"5","min":"1","max":"32","enum_allowed":[],"desc":"How many worker threads should be spawned for sending data to InfluxDB.","long_desc":"","tags":[],"see_also":[]},"username":{"name":"username","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"username of InfluxDB server user","long_desc":"","tags":[],"see_also":[]},"verify_ssl":{"name":"verify_ssl","type":"str","level":"advanced","flags":0,"default_value":"true","min":"","max":"","enum_allowed":[],"desc":"Verify https cert for InfluxDB server. Use \"true\" or \"false\".","long_desc":"","tags":[],"see_also":[]}}},{"name":"insights","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"iostat","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"localpool","can_run":true,"error_string":"","module_options":{"failure_domain":{"name":"failure_domain","type":"str","level":"advanced","flags":1,"default_value":"host","min":"","max":"","enum_allowed":[],"desc":"failure domain for any created local pool","long_desc":"what failure domain we should separate data replicas across.","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"min_size":{"name":"min_size","type":"int","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"default min_size for any created local pool","long_desc":"value to set min_size to (unchanged from Ceph's default if this option is not set)","tags":[],"see_also":[]},"num_rep":{"name":"num_rep","type":"int","level":"advanced","flags":1,"default_value":"3","min":"","max":"","enum_allowed":[],"desc":"default replica count for any created local pool","long_desc":"","tags":[],"see_also":[]},"pg_num":{"name":"pg_num","type":"int","level":"advanced","flags":1,"default_value":"128","min":"","max":"","enum_allowed":[],"desc":"default pg_num for any created local pool","long_desc":"","tags":[],"see_also":[]},"prefix":{"name":"prefix","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"name prefix for any created local pool","long_desc":"","tags":[],"see_also":[]},"subtree":{"name":"subtree","type":"str","level":"advanced","flags":1,"default_value":"rack","min":"","max":"","enum_allowed":[],"desc":"CRUSH level for which to create a local pool","long_desc":"which CRUSH subtree type the module should create a pool for.","tags":[],"see_also":[]}}},{"name":"mds_autoscaler","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"mirroring","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"nfs","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"orchestrator","can_run":true,"error_string":"","module_options":{"fail_fs":{"name":"fail_fs","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"Fail filesystem for rapid multi-rank mds upgrade","long_desc":"","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"orchestrator":{"name":"orchestrator","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["cephadm","rook","test_orchestrator"],"desc":"Orchestrator backend","long_desc":"","tags":[],"see_also":[]}}},{"name":"osd_perf_query","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"osd_support","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"pg_autoscaler","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"sleep_interval":{"name":"sleep_interval","type":"secs","level":"advanced","flags":0,"default_value":"60","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"threshold":{"name":"threshold","type":"float","level":"advanced","flags":0,"default_value":"3.0","min":"1.0","max":"","enum_allowed":[],"desc":"scaling threshold","long_desc":"The factor by which the `NEW PG_NUM` must vary from the current`PG_NUM` before being accepted. Cannot be less than 1.0","tags":[],"see_also":[]}}},{"name":"progress","can_run":true,"error_string":"","module_options":{"allow_pg_recovery_event":{"name":"allow_pg_recovery_event","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"allow the module to show pg recovery progress","long_desc":"","tags":[],"see_also":[]},"enabled":{"name":"enabled","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"max_completed_events":{"name":"max_completed_events","type":"int","level":"advanced","flags":1,"default_value":"50","min":"","max":"","enum_allowed":[],"desc":"number of past completed events to remember","long_desc":"","tags":[],"see_also":[]},"sleep_interval":{"name":"sleep_interval","type":"secs","level":"advanced","flags":1,"default_value":"5","min":"","max":"","enum_allowed":[],"desc":"how long the module is going to sleep","long_desc":"","tags":[],"see_also":[]}}},{"name":"prometheus","can_run":true,"error_string":"","module_options":{"cache":{"name":"cache","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"exclude_perf_counters":{"name":"exclude_perf_counters","type":"bool","level":"advanced","flags":1,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"Do not include perf-counters in the metrics output","long_desc":"Gathering perf-counters from a single Prometheus exporter can degrade ceph-mgr performance, especially in large clusters. Instead, Ceph-exporter daemons are now used by default for perf-counter gathering. This should only be disabled when no ceph-exporters are deployed.","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"rbd_stats_pools":{"name":"rbd_stats_pools","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"rbd_stats_pools_refresh_interval":{"name":"rbd_stats_pools_refresh_interval","type":"int","level":"advanced","flags":0,"default_value":"300","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"scrape_interval":{"name":"scrape_interval","type":"float","level":"advanced","flags":0,"default_value":"15.0","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"server_addr":{"name":"server_addr","type":"str","level":"advanced","flags":0,"default_value":"::","min":"","max":"","enum_allowed":[],"desc":"the IPv4 or IPv6 address on which the module listens for HTTP requests","long_desc":"","tags":[],"see_also":[]},"server_port":{"name":"server_port","type":"int","level":"advanced","flags":1,"default_value":"9283","min":"","max":"","enum_allowed":[],"desc":"the port on which the module listens for HTTP requests","long_desc":"","tags":[],"see_also":[]},"stale_cache_strategy":{"name":"stale_cache_strategy","type":"str","level":"advanced","flags":0,"default_value":"log","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"standby_behaviour":{"name":"standby_behaviour","type":"str","level":"advanced","flags":1,"default_value":"default","min":"","max":"","enum_allowed":["default","error"],"desc":"","long_desc":"","tags":[],"see_also":[]},"standby_error_status_code":{"name":"standby_error_status_code","type":"int","level":"advanced","flags":1,"default_value":"500","min":"400","max":"599","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"rbd_support","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"max_concurrent_snap_create":{"name":"max_concurrent_snap_create","type":"int","level":"advanced","flags":0,"default_value":"10","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"mirror_snapshot_schedule":{"name":"mirror_snapshot_schedule","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"trash_purge_schedule":{"name":"trash_purge_schedule","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"restful","can_run":true,"error_string":"","module_options":{"enable_auth":{"name":"enable_auth","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"key_file":{"name":"key_file","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"server_addr":{"name":"server_addr","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"server_port":{"name":"server_port","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"rgw","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"rook","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"storage_class":{"name":"storage_class","type":"str","level":"advanced","flags":0,"default_value":"local","min":"","max":"","enum_allowed":[],"desc":"storage class name for LSO-discovered PVs","long_desc":"","tags":[],"see_also":[]}}},{"name":"selftest","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"roption1":{"name":"roption1","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"roption2":{"name":"roption2","type":"str","level":"advanced","flags":0,"default_value":"xyz","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"rwoption1":{"name":"rwoption1","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"rwoption2":{"name":"rwoption2","type":"int","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"rwoption3":{"name":"rwoption3","type":"float","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"rwoption4":{"name":"rwoption4","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"rwoption5":{"name":"rwoption5","type":"bool","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"rwoption6":{"name":"rwoption6","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"rwoption7":{"name":"rwoption7","type":"int","level":"advanced","flags":0,"default_value":"","min":"1","max":"42","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"testkey":{"name":"testkey","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"testlkey":{"name":"testlkey","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"testnewline":{"name":"testnewline","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"snap_schedule","can_run":true,"error_string":"","module_options":{"allow_m_granularity":{"name":"allow_m_granularity","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"allow minute scheduled snapshots","long_desc":"","tags":[],"see_also":[]},"dump_on_update":{"name":"dump_on_update","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"dump database to debug log on update","long_desc":"","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"stats","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"status","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"telegraf","can_run":true,"error_string":"","module_options":{"address":{"name":"address","type":"str","level":"advanced","flags":0,"default_value":"unixgram:///tmp/telegraf.sock","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"interval":{"name":"interval","type":"secs","level":"advanced","flags":0,"default_value":"15","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"telemetry","can_run":true,"error_string":"","module_options":{"channel_basic":{"name":"channel_basic","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"Share basic cluster information (size, version)","long_desc":"","tags":[],"see_also":[]},"channel_crash":{"name":"channel_crash","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"Share metadata about Ceph daemon crashes (version, stack straces, etc)","long_desc":"","tags":[],"see_also":[]},"channel_device":{"name":"channel_device","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"Share device health metrics (e.g., SMART data, minus potentially identifying info like serial numbers)","long_desc":"","tags":[],"see_also":[]},"channel_ident":{"name":"channel_ident","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"Share a user-provided description and/or contact email for the cluster","long_desc":"","tags":[],"see_also":[]},"channel_perf":{"name":"channel_perf","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"Share various performance metrics of a cluster","long_desc":"","tags":[],"see_also":[]},"contact":{"name":"contact","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"description":{"name":"description","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"device_url":{"name":"device_url","type":"str","level":"advanced","flags":0,"default_value":"https://telemetry.ceph.com/device","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"enabled":{"name":"enabled","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"interval":{"name":"interval","type":"int","level":"advanced","flags":0,"default_value":"24","min":"8","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"last_opt_revision":{"name":"last_opt_revision","type":"int","level":"advanced","flags":0,"default_value":"1","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"leaderboard":{"name":"leaderboard","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"leaderboard_description":{"name":"leaderboard_description","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"organization":{"name":"organization","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"proxy":{"name":"proxy","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"url":{"name":"url","type":"str","level":"advanced","flags":0,"default_value":"https://telemetry.ceph.com/report","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"test_orchestrator","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"volumes","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"max_concurrent_clones":{"name":"max_concurrent_clones","type":"int","level":"advanced","flags":0,"default_value":"4","min":"","max":"","enum_allowed":[],"desc":"Number of asynchronous cloner threads","long_desc":"","tags":[],"see_also":[]},"snapshot_clone_delay":{"name":"snapshot_clone_delay","type":"int","level":"advanced","flags":0,"default_value":"0","min":"","max":"","enum_allowed":[],"desc":"Delay clone begin operation by snapshot_clone_delay seconds","long_desc":"","tags":[],"see_also":[]},"snapshot_clone_no_wait":{"name":"snapshot_clone_no_wait","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"Reject subvolume clone request when cloner threads are busy","long_desc":"","tags":[],"see_also":[]}}},{"name":"zabbix","can_run":true,"error_string":"","module_options":{"discovery_interval":{"name":"discovery_interval","type":"uint","level":"advanced","flags":0,"default_value":"100","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"identifier":{"name":"identifier","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"interval":{"name":"interval","type":"secs","level":"advanced","flags":0,"default_value":"60","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"zabbix_host":{"name":"zabbix_host","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"zabbix_port":{"name":"zabbix_port","type":"int","level":"advanced","flags":0,"default_value":"10051","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"zabbix_sender":{"name":"zabbix_sender","type":"str","level":"advanced","flags":0,"default_value":"/usr/bin/zabbix_sender","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}}],"services":{},"always_on_modules":{"octopus":["balancer","crash","devicehealth","orchestrator","pg_autoscaler","progress","rbd_support","status","telemetry","volumes"],"pacific":["balancer","crash","devicehealth","orchestrator","pg_autoscaler","progress","rbd_support","status","telemetry","volumes"],"quincy":["balancer","crash","devicehealth","orchestrator","pg_autoscaler","progress","rbd_support","status","telemetry","volumes"],"reef":["balancer","crash","devicehealth","orchestrator","pg_autoscaler","progress","rbd_support","status","telemetry","volumes"]},"last_failure_osd_epoch":0,"active_clients":[{"name":"devicehealth","addrvec":[{"type":"v2","addr":"172.21.15.192:0","nonce":2718546593}]},{"name":"libcephsqlite","addrvec":[{"type":"v2","addr":"172.21.15.192:0","nonce":3853185339}]},{"name":"rbd_support","addrvec":[{"type":"v2","addr":"172.21.15.192:0","nonce":729193348}]},{"name":"volumes","addrvec":[{"type":"v2","addr":"172.21.15.192:0","nonce":1867340613}]}]} 2024-06-03T08:36:30.572 INFO:tasks.ceph.ceph_manager.ceph:mgr available! 2024-06-03T08:36:30.572 INFO:tasks.ceph.ceph_manager.ceph:waiting for all up 2024-06-03T08:36:30.573 DEBUG:teuthology.orchestra.run.smithi088:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd dump --format=json 2024-06-03T08:36:30.678 INFO:teuthology.orchestra.run.smithi088.stderr:2024-06-03T08:36:30.678+0000 7fa99b669640 10 monclient: get_monmap_and_config 2024-06-03T08:36:30.678 INFO:teuthology.orchestra.run.smithi088.stderr:2024-06-03T08:36:30.678+0000 7fa99b669640 10 monclient: build_initial_monmap 2024-06-03T08:36:30.678 INFO:teuthology.orchestra.run.smithi088.stderr:2024-06-03T08:36:30.678+0000 7fa99b669640 10 monclient: monmap: 2024-06-03T08:36:30.679 INFO:teuthology.orchestra.run.smithi088.stderr:epoch 0 2024-06-03T08:36:30.679 INFO:teuthology.orchestra.run.smithi088.stderr:fsid ec3c3fdd-3ac6-4d7d-9412-683130ed8b04 2024-06-03T08:36:30.679 INFO:teuthology.orchestra.run.smithi088.stderr:last_changed 2024-06-03T08:36:30.679231+0000 2024-06-03T08:36:30.679 INFO:teuthology.orchestra.run.smithi088.stderr:created 2024-06-03T08:36:30.679231+0000 2024-06-03T08:36:30.679 INFO:teuthology.orchestra.run.smithi088.stderr:min_mon_release 0 (unknown) 2024-06-03T08:36:30.679 INFO:teuthology.orchestra.run.smithi088.stderr:election_strategy: 3 2024-06-03T08:36:30.679 INFO:teuthology.orchestra.run.smithi088.stderr:0: v1:172.21.15.88:6789/0 mon.noname-a 2024-06-03T08:36:30.679 INFO:teuthology.orchestra.run.smithi088.stderr:1: v1:172.21.15.192:6789/0 mon.noname-c 2024-06-03T08:36:30.679 INFO:teuthology.orchestra.run.smithi088.stderr:2: v1:172.21.15.88:6790/0 mon.noname-b 2024-06-03T08:36:30.679 INFO:teuthology.orchestra.run.smithi088.stderr: 2024-06-03T08:36:30.681 INFO:teuthology.orchestra.run.smithi088.stderr:2024-06-03T08:36:30.678+0000 7fa99b669640 1 Processor -- start 2024-06-03T08:36:30.681 INFO:teuthology.orchestra.run.smithi088.stderr:2024-06-03T08:36:30.678+0000 7fa99b669640 1 -- start start 2024-06-03T08:36:30.681 INFO:teuthology.orchestra.run.smithi088.stderr:2024-06-03T08:36:30.678+0000 7fa99b669640 10 monclient: init 2024-06-03T08:36:30.681 INFO:teuthology.orchestra.run.smithi088.stderr:2024-06-03T08:36:30.679+0000 7fa99b669640 10 monclient: _reopen_session rank -1 2024-06-03T08:36:30.681 INFO:teuthology.orchestra.run.smithi088.stderr:2024-06-03T08:36:30.679+0000 7fa99b669640 10 monclient: _add_conns ranks=[1,2,0] 2024-06-03T08:36:30.681 INFO:teuthology.orchestra.run.smithi088.stderr:2024-06-03T08:36:30.679+0000 7fa99b669640 10 monclient(hunting): picked mon.noname-c con 0x7fa9940a8960 addr v1:172.21.15.192:6789/0 2024-06-03T08:36:30.681 INFO:teuthology.orchestra.run.smithi088.stderr:2024-06-03T08:36:30.679+0000 7fa99b669640 10 monclient(hunting): picked mon.noname-b con 0x7fa9941450f0 addr v1:172.21.15.88:6790/0 2024-06-03T08:36:30.681 INFO:teuthology.orchestra.run.smithi088.stderr:2024-06-03T08:36:30.679+0000 7fa99b669640 10 monclient(hunting): picked mon.noname-a con 0x7fa99413eec0 addr v1:172.21.15.88:6789/0 2024-06-03T08:36:30.681 INFO:teuthology.orchestra.run.smithi088.stderr:2024-06-03T08:36:30.679+0000 7fa99b669640 1 -- --> v1:172.21.15.88:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fa994144c60 con 0x7fa99413eec0 2024-06-03T08:36:30.681 INFO:teuthology.orchestra.run.smithi088.stderr:2024-06-03T08:36:30.679+0000 7fa99b669640 1 -- --> v1:172.21.15.192:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fa9940b3930 con 0x7fa9940a8960 2024-06-03T08:36:30.681 INFO:teuthology.orchestra.run.smithi088.stderr:2024-06-03T08:36:30.679+0000 7fa99b669640 1 -- --> v1:172.21.15.88:6790/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fa9940b1ad0 con 0x7fa9941450f0 2024-06-03T08:36:30.681 INFO:teuthology.orchestra.run.smithi088.stderr:2024-06-03T08:36:30.679+0000 7fa99b669640 10 monclient(hunting): _renew_subs 2024-06-03T08:36:30.681 INFO:teuthology.orchestra.run.smithi088.stderr:2024-06-03T08:36:30.679+0000 7fa99b669640 10 monclient(hunting): authenticate will time out at 1246.489536s 2024-06-03T08:36:30.682 INFO:teuthology.orchestra.run.smithi088.stderr:2024-06-03T08:36:30.680+0000 7fa998c04640 1 --1- >> v1:172.21.15.88:6790/0 conn(0x7fa9941450f0 0x7fa99413e7b0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.88:6790/0 says I am v1:172.21.15.88:54050/0 (socket says 172.21.15.88:54050) 2024-06-03T08:36:30.682 INFO:teuthology.orchestra.run.smithi088.stderr:2024-06-03T08:36:30.680+0000 7fa998c04640 1 -- 172.21.15.88:0/2677097666 learned_addr learned my addr 172.21.15.88:0/2677097666 (peer_addr_for_me v1:172.21.15.88:0/0) 2024-06-03T08:36:30.698 INFO:teuthology.orchestra.run.smithi088.stderr:2024-06-03T08:36:30.698+0000 7fa98bfff640 1 -- 172.21.15.88:0/2677097666 <== mon.1 v1:172.21.15.192:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 1773748021 0 0) 0x7fa9940b3930 con 0x7fa9940a8960 2024-06-03T08:36:30.699 INFO:teuthology.orchestra.run.smithi088.stderr:2024-06-03T08:36:30.698+0000 7fa98bfff640 10 monclient(hunting): _negotiate 2024-06-03T08:36:30.699 INFO:teuthology.orchestra.run.smithi088.stderr:2024-06-03T08:36:30.698+0000 7fa98bfff640 10 monclient(hunting): _init_auth method 2 2024-06-03T08:36:30.699 INFO:teuthology.orchestra.run.smithi088.stderr:2024-06-03T08:36:30.698+0000 7fa98bfff640 10 monclient(hunting): _init_auth creating new auth 2024-06-03T08:36:30.699 INFO:teuthology.orchestra.run.smithi088.stderr:2024-06-03T08:36:30.698+0000 7fa98bfff640 10 monclient(hunting): my global_id is 4282 2024-06-03T08:36:30.699 INFO:teuthology.orchestra.run.smithi088.stderr:2024-06-03T08:36:30.698+0000 7fa98bfff640 1 -- 172.21.15.88:0/2677097666 --> v1:172.21.15.192:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7fa980003520 con 0x7fa9940a8960 2024-06-03T08:36:30.704 INFO:teuthology.orchestra.run.smithi088.stderr:2024-06-03T08:36:30.703+0000 7fa98bfff640 1 -- 172.21.15.88:0/2677097666 <== mon.1 v1:172.21.15.192:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 214+0+0 (unknown 432984632 0 0) 0x7fa980003520 con 0x7fa9940a8960 2024-06-03T08:36:30.704 INFO:teuthology.orchestra.run.smithi088.stderr:2024-06-03T08:36:30.703+0000 7fa98bfff640 1 -- 172.21.15.88:0/2677097666 >> v1:172.21.15.88:6790/0 conn(0x7fa9941450f0 legacy=0x7fa99413e7b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-06-03T08:36:30.704 INFO:teuthology.orchestra.run.smithi088.stderr:2024-06-03T08:36:30.703+0000 7fa98bfff640 1 -- 172.21.15.88:0/2677097666 >> v1:172.21.15.88:6789/0 conn(0x7fa99413eec0 legacy=0x7fa99414ce60 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-06-03T08:36:30.704 INFO:teuthology.orchestra.run.smithi088.stderr:2024-06-03T08:36:30.703+0000 7fa98bfff640 10 monclient: _finish_hunting 0 2024-06-03T08:36:30.704 INFO:teuthology.orchestra.run.smithi088.stderr:2024-06-03T08:36:30.703+0000 7fa98bfff640 1 monclient: found mon.noname-c 2024-06-03T08:36:30.704 INFO:teuthology.orchestra.run.smithi088.stderr:2024-06-03T08:36:30.703+0000 7fa98bfff640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-06-03T08:36:30.704 INFO:teuthology.orchestra.run.smithi088.stderr:2024-06-03T08:36:30.703+0000 7fa98bfff640 10 monclient: _send_mon_message to mon.noname-c at v1:172.21.15.192:6789/0 2024-06-03T08:36:30.704 INFO:teuthology.orchestra.run.smithi088.stderr:2024-06-03T08:36:30.703+0000 7fa98bfff640 1 -- 172.21.15.88:0/2677097666 --> v1:172.21.15.192:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fa9940b1c60 con 0x7fa9940a8960 2024-06-03T08:36:30.704 INFO:teuthology.orchestra.run.smithi088.stderr:2024-06-03T08:36:30.703+0000 7fa98bfff640 10 monclient: _finish_auth 0 2024-06-03T08:36:30.704 INFO:teuthology.orchestra.run.smithi088.stderr:2024-06-03T08:36:30.703+0000 7fa98bfff640 10 monclient: _check_auth_tickets 2024-06-03T08:36:30.705 INFO:teuthology.orchestra.run.smithi088.stderr:2024-06-03T08:36:30.703+0000 7fa98bfff640 20 monclient: _check_auth_rotating not needed by client.admin 2024-06-03T08:36:30.705 INFO:teuthology.orchestra.run.smithi088.stderr:2024-06-03T08:36:30.703+0000 7fa98bfff640 1 -- 172.21.15.88:0/2677097666 <== mon.1 v1:172.21.15.192:6789/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 2234374065 0 0) 0x7fa990001610 con 0x7fa9940a8960 2024-06-03T08:36:30.705 INFO:teuthology.orchestra.run.smithi088.stderr:2024-06-03T08:36:30.703+0000 7fa98bfff640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-06-03T08:36:30.705 INFO:teuthology.orchestra.run.smithi088.stderr:2024-06-03T08:36:30.703+0000 7fa98bfff640 10 monclient: got monmap 1 from mon.noname-c (according to old e1) 2024-06-03T08:36:30.705 INFO:teuthology.orchestra.run.smithi088.stderr:2024-06-03T08:36:30.703+0000 7fa98bfff640 10 monclient: dump: 2024-06-03T08:36:30.705 INFO:teuthology.orchestra.run.smithi088.stderr:epoch 1 2024-06-03T08:36:30.705 INFO:teuthology.orchestra.run.smithi088.stderr:fsid ec3c3fdd-3ac6-4d7d-9412-683130ed8b04 2024-06-03T08:36:30.705 INFO:teuthology.orchestra.run.smithi088.stderr:last_changed 2024-06-03T08:34:31.885125+0000 2024-06-03T08:36:30.705 INFO:teuthology.orchestra.run.smithi088.stderr:created 2024-06-03T08:34:31.885125+0000 2024-06-03T08:36:30.705 INFO:teuthology.orchestra.run.smithi088.stderr:min_mon_release 18 (reef) 2024-06-03T08:36:30.705 INFO:teuthology.orchestra.run.smithi088.stderr:election_strategy: 3 2024-06-03T08:36:30.705 INFO:teuthology.orchestra.run.smithi088.stderr:0: v1:172.21.15.88:6789/0 mon.a 2024-06-03T08:36:30.705 INFO:teuthology.orchestra.run.smithi088.stderr:1: v1:172.21.15.192:6789/0 mon.b 2024-06-03T08:36:30.706 INFO:teuthology.orchestra.run.smithi088.stderr:2: v1:172.21.15.88:6790/0 mon.c 2024-06-03T08:36:30.706 INFO:teuthology.orchestra.run.smithi088.stderr: 2024-06-03T08:36:30.706 INFO:teuthology.orchestra.run.smithi088.stderr:2024-06-03T08:36:30.703+0000 7fa99b669640 5 monclient: authenticate success, global_id 4282 2024-06-03T08:36:30.706 INFO:teuthology.orchestra.run.smithi088.stderr:2024-06-03T08:36:30.703+0000 7fa99b669640 20 monclient: get_monmap_and_config waiting for monmap|config 2024-06-03T08:36:30.707 INFO:teuthology.orchestra.run.smithi088.stderr:2024-06-03T08:36:30.705+0000 7fa98bfff640 1 -- 172.21.15.88:0/2677097666 <== mon.1 v1:172.21.15.192:6789/0 4 ==== config(0 keys) v1 ==== 4+0+0 (unknown 0 0 0) 0x7fa990001830 con 0x7fa9940a8960 2024-06-03T08:36:30.707 INFO:teuthology.orchestra.run.smithi088.stderr:2024-06-03T08:36:30.705+0000 7fa98bfff640 10 monclient: handle_config config(0 keys) v1 2024-06-03T08:36:30.708 INFO:teuthology.orchestra.run.smithi088.stderr:2024-06-03T08:36:30.705+0000 7fa98bfff640 1 -- 172.21.15.88:0/2677097666 <== mon.1 v1:172.21.15.192:6789/0 5 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 2234374065 0 0) 0x7fa990001bc0 con 0x7fa9940a8960 2024-06-03T08:36:30.708 INFO:teuthology.orchestra.run.smithi088.stderr:2024-06-03T08:36:30.705+0000 7fa98bfff640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-06-03T08:36:30.708 INFO:teuthology.orchestra.run.smithi088.stderr:2024-06-03T08:36:30.705+0000 7fa98bfff640 10 monclient: got monmap 1 from mon.b (according to old e1) 2024-06-03T08:36:30.708 INFO:teuthology.orchestra.run.smithi088.stderr:2024-06-03T08:36:30.705+0000 7fa98bfff640 10 monclient: dump: 2024-06-03T08:36:30.708 INFO:teuthology.orchestra.run.smithi088.stderr:epoch 1 2024-06-03T08:36:30.708 INFO:teuthology.orchestra.run.smithi088.stderr:fsid ec3c3fdd-3ac6-4d7d-9412-683130ed8b04 2024-06-03T08:36:30.708 INFO:teuthology.orchestra.run.smithi088.stderr:last_changed 2024-06-03T08:34:31.885125+0000 2024-06-03T08:36:30.708 INFO:teuthology.orchestra.run.smithi088.stderr:created 2024-06-03T08:34:31.885125+0000 2024-06-03T08:36:30.708 INFO:teuthology.orchestra.run.smithi088.stderr:min_mon_release 18 (reef) 2024-06-03T08:36:30.708 INFO:teuthology.orchestra.run.smithi088.stderr:election_strategy: 3 2024-06-03T08:36:30.708 INFO:teuthology.orchestra.run.smithi088.stderr:0: v1:172.21.15.88:6789/0 mon.a 2024-06-03T08:36:30.709 INFO:teuthology.orchestra.run.smithi088.stderr:1: v1:172.21.15.192:6789/0 mon.b 2024-06-03T08:36:30.709 INFO:teuthology.orchestra.run.smithi088.stderr:2: v1:172.21.15.88:6790/0 mon.c 2024-06-03T08:36:30.709 INFO:teuthology.orchestra.run.smithi088.stderr: 2024-06-03T08:36:30.709 INFO:teuthology.orchestra.run.smithi088.stderr:2024-06-03T08:36:30.705+0000 7fa99b669640 10 monclient: get_monmap_and_config success 2024-06-03T08:36:30.709 INFO:teuthology.orchestra.run.smithi088.stderr:2024-06-03T08:36:30.705+0000 7fa99b669640 4 set_mon_vals no callback set 2024-06-03T08:36:30.709 INFO:teuthology.orchestra.run.smithi088.stderr:2024-06-03T08:36:30.706+0000 7fa99b669640 10 monclient: shutdown 2024-06-03T08:36:30.709 INFO:teuthology.orchestra.run.smithi088.stderr:2024-06-03T08:36:30.706+0000 7fa99b669640 20 monclient: shutdown discarding 0 pending message(s) 2024-06-03T08:36:30.709 INFO:teuthology.orchestra.run.smithi088.stderr:2024-06-03T08:36:30.706+0000 7fa99b669640 1 -- 172.21.15.88:0/2677097666 >> v1:172.21.15.192:6789/0 conn(0x7fa9940a8960 legacy=0x7fa9940a8d40 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-06-03T08:36:30.709 INFO:teuthology.orchestra.run.smithi088.stderr:2024-06-03T08:36:30.706+0000 7fa99b669640 1 -- 172.21.15.88:0/2677097666 shutdown_connections 2024-06-03T08:36:30.709 INFO:teuthology.orchestra.run.smithi088.stderr:2024-06-03T08:36:30.706+0000 7fa99b669640 1 -- 172.21.15.88:0/2677097666 >> 172.21.15.88:0/2677097666 conn(0x7fa9940b70d0 msgr2=0x7fa9940b74e0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-06-03T08:36:30.709 INFO:teuthology.orchestra.run.smithi088.stderr:2024-06-03T08:36:30.706+0000 7fa99b669640 1 -- 172.21.15.88:0/2677097666 shutdown_connections 2024-06-03T08:36:30.709 INFO:teuthology.orchestra.run.smithi088.stderr:2024-06-03T08:36:30.706+0000 7fa99b669640 1 -- 172.21.15.88:0/2677097666 wait complete. 2024-06-03T08:36:30.709 INFO:teuthology.orchestra.run.smithi088.stderr:2024-06-03T08:36:30.706+0000 7fa99b669640 10 monclient: build_initial_monmap 2024-06-03T08:36:30.709 INFO:teuthology.orchestra.run.smithi088.stderr:2024-06-03T08:36:30.706+0000 7fa99b669640 10 monclient: monmap: 2024-06-03T08:36:30.710 INFO:teuthology.orchestra.run.smithi088.stderr:epoch 0 2024-06-03T08:36:30.710 INFO:teuthology.orchestra.run.smithi088.stderr:fsid 00000000-0000-0000-0000-000000000000 2024-06-03T08:36:30.710 INFO:teuthology.orchestra.run.smithi088.stderr:last_changed 0.000000 2024-06-03T08:36:30.710 INFO:teuthology.orchestra.run.smithi088.stderr:created 0.000000 2024-06-03T08:36:30.710 INFO:teuthology.orchestra.run.smithi088.stderr:min_mon_release 0 (unknown) 2024-06-03T08:36:30.710 INFO:teuthology.orchestra.run.smithi088.stderr:election_strategy: 1 2024-06-03T08:36:30.710 INFO:teuthology.orchestra.run.smithi088.stderr:0: v1:172.21.15.88:6789/0 mon.noname-a 2024-06-03T08:36:30.710 INFO:teuthology.orchestra.run.smithi088.stderr:1: v1:172.21.15.192:6789/0 mon.noname-b 2024-06-03T08:36:30.710 INFO:teuthology.orchestra.run.smithi088.stderr:2: v1:172.21.15.88:6790/0 mon.noname-c 2024-06-03T08:36:30.710 INFO:teuthology.orchestra.run.smithi088.stderr: 2024-06-03T08:36:30.710 INFO:teuthology.orchestra.run.smithi088.stderr:2024-06-03T08:36:30.707+0000 7fa99b669640 1 librados: starting msgr at 2024-06-03T08:36:30.711 INFO:teuthology.orchestra.run.smithi088.stderr:2024-06-03T08:36:30.707+0000 7fa99b669640 1 librados: starting objecter 2024-06-03T08:36:30.711 INFO:teuthology.orchestra.run.smithi088.stderr:2024-06-03T08:36:30.707+0000 7fa99b669640 1 Processor -- start 2024-06-03T08:36:30.711 INFO:teuthology.orchestra.run.smithi088.stderr:2024-06-03T08:36:30.707+0000 7fa99b669640 1 -- start start 2024-06-03T08:36:30.711 INFO:teuthology.orchestra.run.smithi088.stderr:2024-06-03T08:36:30.707+0000 7fa99b669640 1 librados: setting wanted keys 2024-06-03T08:36:30.711 INFO:teuthology.orchestra.run.smithi088.stderr:2024-06-03T08:36:30.707+0000 7fa99b669640 1 librados: calling monclient init 2024-06-03T08:36:30.711 INFO:teuthology.orchestra.run.smithi088.stderr:2024-06-03T08:36:30.707+0000 7fa99b669640 10 monclient: init 2024-06-03T08:36:30.712 INFO:teuthology.orchestra.run.smithi088.stderr:2024-06-03T08:36:30.708+0000 7fa99b669640 10 monclient: _reopen_session rank -1 2024-06-03T08:36:30.712 INFO:teuthology.orchestra.run.smithi088.stderr:2024-06-03T08:36:30.708+0000 7fa99b669640 10 monclient: _add_conns ranks=[2,0,1] 2024-06-03T08:36:30.713 INFO:teuthology.orchestra.run.smithi088.stderr:2024-06-03T08:36:30.708+0000 7fa99b669640 10 monclient(hunting): picked mon.noname-c con 0x7fa9940a8960 addr v1:172.21.15.88:6790/0 2024-06-03T08:36:30.713 INFO:teuthology.orchestra.run.smithi088.stderr:2024-06-03T08:36:30.708+0000 7fa99b669640 10 monclient(hunting): picked mon.noname-a con 0x7fa99413eec0 addr v1:172.21.15.88:6789/0 2024-06-03T08:36:30.713 INFO:teuthology.orchestra.run.smithi088.stderr:2024-06-03T08:36:30.708+0000 7fa99b669640 10 monclient(hunting): picked mon.noname-b con 0x7fa9941450f0 addr v1:172.21.15.192:6789/0 2024-06-03T08:36:30.713 INFO:teuthology.orchestra.run.smithi088.stderr:2024-06-03T08:36:30.708+0000 7fa99b669640 1 -- --> v1:172.21.15.88:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fa9941d4350 con 0x7fa99413eec0 2024-06-03T08:36:30.713 INFO:teuthology.orchestra.run.smithi088.stderr:2024-06-03T08:36:30.708+0000 7fa99b669640 1 -- --> v1:172.21.15.192:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fa9940ac530 con 0x7fa9941450f0 2024-06-03T08:36:30.713 INFO:teuthology.orchestra.run.smithi088.stderr:2024-06-03T08:36:30.708+0000 7fa99b669640 1 -- --> v1:172.21.15.88:6790/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fa9940ae590 con 0x7fa9940a8960 2024-06-03T08:36:30.713 INFO:teuthology.orchestra.run.smithi088.stderr:2024-06-03T08:36:30.708+0000 7fa99b669640 10 monclient(hunting): _renew_subs 2024-06-03T08:36:30.713 INFO:teuthology.orchestra.run.smithi088.stderr:2024-06-03T08:36:30.708+0000 7fa99b669640 10 monclient(hunting): authenticate will time out at 1246.518079s 2024-06-03T08:36:30.713 INFO:teuthology.orchestra.run.smithi088.stderr:2024-06-03T08:36:30.708+0000 7fa999c06640 1 --1- >> v1:172.21.15.192:6789/0 conn(0x7fa9941450f0 0x7fa9941d6c90 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=1).handle_server_banner_and_identify peer v1:172.21.15.192:6789/0 says I am v1:172.21.15.88:52298/0 (socket says 172.21.15.88:52298) 2024-06-03T08:36:30.713 INFO:teuthology.orchestra.run.smithi088.stderr:2024-06-03T08:36:30.708+0000 7fa999c06640 1 -- 172.21.15.88:0/4052359507 learned_addr learned my addr 172.21.15.88:0/4052359507 (peer_addr_for_me v1:172.21.15.88:0/0) 2024-06-03T08:36:30.713 INFO:teuthology.orchestra.run.smithi088.stderr:2024-06-03T08:36:30.708+0000 7fa989ffb640 10 client.?.objecter ms_handle_connect 0x7fa9941450f0 2024-06-03T08:36:30.713 INFO:teuthology.orchestra.run.smithi088.stderr:2024-06-03T08:36:30.708+0000 7fa989ffb640 10 client.?.objecter resend_mon_ops 2024-06-03T08:36:30.713 INFO:teuthology.orchestra.run.smithi088.stderr:2024-06-03T08:36:30.710+0000 7fa989ffb640 1 -- 172.21.15.88:0/4052359507 <== mon.1 v1:172.21.15.192:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 2080266535 0 0) 0x7fa9940ac530 con 0x7fa9941450f0 2024-06-03T08:36:30.714 INFO:teuthology.orchestra.run.smithi088.stderr:2024-06-03T08:36:30.710+0000 7fa989ffb640 10 monclient(hunting): _negotiate 2024-06-03T08:36:30.714 INFO:teuthology.orchestra.run.smithi088.stderr:2024-06-03T08:36:30.710+0000 7fa989ffb640 10 monclient(hunting): _init_auth method 2 2024-06-03T08:36:30.714 INFO:teuthology.orchestra.run.smithi088.stderr:2024-06-03T08:36:30.710+0000 7fa989ffb640 10 monclient(hunting): _init_auth creating new auth 2024-06-03T08:36:30.714 INFO:teuthology.orchestra.run.smithi088.stderr:2024-06-03T08:36:30.710+0000 7fa989ffb640 10 monclient(hunting): my global_id is 4285 2024-06-03T08:36:30.714 INFO:teuthology.orchestra.run.smithi088.stderr:2024-06-03T08:36:30.710+0000 7fa989ffb640 1 -- 172.21.15.88:0/4052359507 --> v1:172.21.15.192:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7fa978004eb0 con 0x7fa9941450f0 2024-06-03T08:36:30.714 INFO:teuthology.orchestra.run.smithi088.stderr:2024-06-03T08:36:30.711+0000 7fa989ffb640 10 client.?.objecter ms_handle_connect 0x7fa99413eec0 2024-06-03T08:36:30.714 INFO:teuthology.orchestra.run.smithi088.stderr:2024-06-03T08:36:30.711+0000 7fa989ffb640 10 client.?.objecter resend_mon_ops 2024-06-03T08:36:30.716 INFO:teuthology.orchestra.run.smithi088.stderr:2024-06-03T08:36:30.716+0000 7fa989ffb640 1 -- 172.21.15.88:0/4052359507 <== mon.1 v1:172.21.15.192:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 764+0+0 (unknown 1784489435 0 0) 0x7fa978004eb0 con 0x7fa9941450f0 2024-06-03T08:36:30.717 INFO:teuthology.orchestra.run.smithi088.stderr:2024-06-03T08:36:30.716+0000 7fa989ffb640 1 -- 172.21.15.88:0/4052359507 --> v1:172.21.15.192:6789/0 -- auth(proto 2 165 bytes epoch 0) v1 -- 0x7fa9940ac530 con 0x7fa9941450f0 2024-06-03T08:36:30.717 INFO:teuthology.orchestra.run.smithi088.stderr:2024-06-03T08:36:30.716+0000 7fa989ffb640 1 -- 172.21.15.88:0/4052359507 <== mon.1 v1:172.21.15.192:6789/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 2234374065 0 0) 0x7fa98c001bb0 con 0x7fa9941450f0 2024-06-03T08:36:30.717 INFO:teuthology.orchestra.run.smithi088.stderr:2024-06-03T08:36:30.716+0000 7fa989ffb640 10 monclient(hunting): handle_monmap mon_map magic: 0 v1 2024-06-03T08:36:30.717 INFO:teuthology.orchestra.run.smithi088.stderr:2024-06-03T08:36:30.716+0000 7fa989ffb640 10 monclient(hunting): got monmap 1 from mon.noname-b (according to old e1) 2024-06-03T08:36:30.717 INFO:teuthology.orchestra.run.smithi088.stderr:2024-06-03T08:36:30.716+0000 7fa989ffb640 10 monclient(hunting): dump: 2024-06-03T08:36:30.717 INFO:teuthology.orchestra.run.smithi088.stderr:epoch 1 2024-06-03T08:36:30.717 INFO:teuthology.orchestra.run.smithi088.stderr:fsid ec3c3fdd-3ac6-4d7d-9412-683130ed8b04 2024-06-03T08:36:30.717 INFO:teuthology.orchestra.run.smithi088.stderr:last_changed 2024-06-03T08:34:31.885125+0000 2024-06-03T08:36:30.717 INFO:teuthology.orchestra.run.smithi088.stderr:created 2024-06-03T08:34:31.885125+0000 2024-06-03T08:36:30.717 INFO:teuthology.orchestra.run.smithi088.stderr:min_mon_release 18 (reef) 2024-06-03T08:36:30.717 INFO:teuthology.orchestra.run.smithi088.stderr:election_strategy: 3 2024-06-03T08:36:30.717 INFO:teuthology.orchestra.run.smithi088.stderr:0: v1:172.21.15.88:6789/0 mon.a 2024-06-03T08:36:30.718 INFO:teuthology.orchestra.run.smithi088.stderr:1: v1:172.21.15.192:6789/0 mon.b 2024-06-03T08:36:30.718 INFO:teuthology.orchestra.run.smithi088.stderr:2: v1:172.21.15.88:6790/0 mon.c 2024-06-03T08:36:30.718 INFO:teuthology.orchestra.run.smithi088.stderr: 2024-06-03T08:36:30.718 INFO:teuthology.orchestra.run.smithi088.stderr:2024-06-03T08:36:30.716+0000 7fa989ffb640 10 monclient(hunting): _finish_auth 0 2024-06-03T08:36:30.721 INFO:teuthology.orchestra.run.smithi088.stderr:2024-06-03T08:36:30.720+0000 7fa989ffb640 1 -- 172.21.15.88:0/4052359507 <== mon.1 v1:172.21.15.192:6789/0 4 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 580+0+0 (unknown 931314798 0 0) 0x7fa9940ac530 con 0x7fa9941450f0 2024-06-03T08:36:30.721 INFO:teuthology.orchestra.run.smithi088.stderr:2024-06-03T08:36:30.720+0000 7fa989ffb640 1 -- 172.21.15.88:0/4052359507 >> v1:172.21.15.88:6790/0 conn(0x7fa9940a8960 legacy=0x7fa9940ade80 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-06-03T08:36:30.721 INFO:teuthology.orchestra.run.smithi088.stderr:2024-06-03T08:36:30.720+0000 7fa989ffb640 1 -- 172.21.15.88:0/4052359507 >> v1:172.21.15.88:6789/0 conn(0x7fa99413eec0 legacy=0x7fa9941da210 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-06-03T08:36:30.722 INFO:teuthology.orchestra.run.smithi088.stderr:2024-06-03T08:36:30.720+0000 7fa989ffb640 10 monclient: _finish_hunting 0 2024-06-03T08:36:30.722 INFO:teuthology.orchestra.run.smithi088.stderr:2024-06-03T08:36:30.720+0000 7fa989ffb640 1 monclient: found mon.b 2024-06-03T08:36:30.722 INFO:teuthology.orchestra.run.smithi088.stderr:2024-06-03T08:36:30.720+0000 7fa989ffb640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-06-03T08:36:30.722 INFO:teuthology.orchestra.run.smithi088.stderr:2024-06-03T08:36:30.720+0000 7fa989ffb640 10 monclient: _send_mon_message to mon.b at v1:172.21.15.192:6789/0 2024-06-03T08:36:30.722 INFO:teuthology.orchestra.run.smithi088.stderr:2024-06-03T08:36:30.720+0000 7fa989ffb640 1 -- 172.21.15.88:0/4052359507 --> v1:172.21.15.192:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fa9941d6730 con 0x7fa9941450f0 2024-06-03T08:36:30.722 INFO:teuthology.orchestra.run.smithi088.stderr:2024-06-03T08:36:30.720+0000 7fa989ffb640 10 monclient: _finish_auth 0 2024-06-03T08:36:30.722 INFO:teuthology.orchestra.run.smithi088.stderr:2024-06-03T08:36:30.720+0000 7fa989ffb640 10 monclient: _check_auth_tickets 2024-06-03T08:36:30.722 INFO:teuthology.orchestra.run.smithi088.stderr:2024-06-03T08:36:30.720+0000 7fa989ffb640 20 monclient: _check_auth_rotating not needed by client.admin 2024-06-03T08:36:30.722 INFO:teuthology.orchestra.run.smithi088.stderr:2024-06-03T08:36:30.720+0000 7fa99b669640 5 monclient: authenticate success, global_id 4285 2024-06-03T08:36:30.722 INFO:teuthology.orchestra.run.smithi088.stderr:2024-06-03T08:36:30.720+0000 7fa99b669640 10 monclient: _renew_subs 2024-06-03T08:36:30.722 INFO:teuthology.orchestra.run.smithi088.stderr:2024-06-03T08:36:30.720+0000 7fa99b669640 10 monclient: _send_mon_message to mon.b at v1:172.21.15.192:6789/0 2024-06-03T08:36:30.722 INFO:teuthology.orchestra.run.smithi088.stderr:2024-06-03T08:36:30.720+0000 7fa99b669640 1 -- 172.21.15.88:0/4052359507 --> v1:172.21.15.192:6789/0 -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fa9941da920 con 0x7fa9941450f0 2024-06-03T08:36:30.723 INFO:teuthology.orchestra.run.smithi088.stderr:2024-06-03T08:36:30.721+0000 7fa99b669640 10 client.4285.objecter _maybe_request_map subscribing (onetime) to next osd map 2024-06-03T08:36:30.723 INFO:teuthology.orchestra.run.smithi088.stderr:2024-06-03T08:36:30.721+0000 7fa99b669640 10 monclient: _renew_subs 2024-06-03T08:36:30.723 INFO:teuthology.orchestra.run.smithi088.stderr:2024-06-03T08:36:30.721+0000 7fa99b669640 10 monclient: _send_mon_message to mon.b at v1:172.21.15.192:6789/0 2024-06-03T08:36:30.723 INFO:teuthology.orchestra.run.smithi088.stderr:2024-06-03T08:36:30.721+0000 7fa99b669640 1 -- 172.21.15.88:0/4052359507 --> v1:172.21.15.192:6789/0 -- mon_subscribe({osdmap=0}) v3 -- 0x7fa9941d7580 con 0x7fa9941450f0 2024-06-03T08:36:30.723 INFO:teuthology.orchestra.run.smithi088.stderr:2024-06-03T08:36:30.721+0000 7fa99b669640 1 librados: init done 2024-06-03T08:36:30.723 INFO:teuthology.orchestra.run.smithi088.stderr:2024-06-03T08:36:30.721+0000 7fa99b669640 10 start_mon_command cmd=[{"prefix": "get_command_descriptions"}] 2024-06-03T08:36:30.723 INFO:teuthology.orchestra.run.smithi088.stderr:2024-06-03T08:36:30.721+0000 7fa99b669640 10 monclient: _send_command 1 [{"prefix": "get_command_descriptions"}] 2024-06-03T08:36:30.724 INFO:teuthology.orchestra.run.smithi088.stderr:2024-06-03T08:36:30.721+0000 7fa99b669640 10 monclient: _send_mon_message to mon.b at v1:172.21.15.192:6789/0 2024-06-03T08:36:30.724 INFO:teuthology.orchestra.run.smithi088.stderr:2024-06-03T08:36:30.721+0000 7fa99b669640 1 -- 172.21.15.88:0/4052359507 --> v1:172.21.15.192:6789/0 -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fa9941da060 con 0x7fa9941450f0 2024-06-03T08:36:30.737 INFO:teuthology.orchestra.run.smithi088.stderr:2024-06-03T08:36:30.736+0000 7fa989ffb640 1 -- 172.21.15.88:0/4052359507 <== mon.1 v1:172.21.15.192:6789/0 5 ==== config(0 keys) v1 ==== 4+0+0 (unknown 0 0 0) 0x7fa98c003970 con 0x7fa9941450f0 2024-06-03T08:36:30.737 INFO:teuthology.orchestra.run.smithi088.stderr:2024-06-03T08:36:30.736+0000 7fa989ffb640 10 monclient: handle_config config(0 keys) v1 2024-06-03T08:36:30.737 INFO:teuthology.orchestra.run.smithi088.stderr:2024-06-03T08:36:30.736+0000 7fa989ffb640 1 -- 172.21.15.88:0/4052359507 <== mon.1 v1:172.21.15.192:6789/0 6 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 2234374065 0 0) 0x7fa98c003d00 con 0x7fa9941450f0 2024-06-03T08:36:30.737 INFO:teuthology.orchestra.run.smithi088.stderr:2024-06-03T08:36:30.736+0000 7fa989ffb640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-06-03T08:36:30.737 INFO:teuthology.orchestra.run.smithi088.stderr:2024-06-03T08:36:30.736+0000 7fa989ffb640 10 monclient: got monmap 1 from mon.b (according to old e1) 2024-06-03T08:36:30.737 INFO:teuthology.orchestra.run.smithi088.stderr:2024-06-03T08:36:30.736+0000 7fa989ffb640 10 monclient: dump: 2024-06-03T08:36:30.737 INFO:teuthology.orchestra.run.smithi088.stderr:epoch 1 2024-06-03T08:36:30.737 INFO:teuthology.orchestra.run.smithi088.stderr:fsid ec3c3fdd-3ac6-4d7d-9412-683130ed8b04 2024-06-03T08:36:30.738 INFO:teuthology.orchestra.run.smithi088.stderr:last_changed 2024-06-03T08:34:31.885125+0000 2024-06-03T08:36:30.738 INFO:teuthology.orchestra.run.smithi088.stderr:created 2024-06-03T08:34:31.885125+0000 2024-06-03T08:36:30.738 INFO:teuthology.orchestra.run.smithi088.stderr:min_mon_release 18 (reef) 2024-06-03T08:36:30.738 INFO:teuthology.orchestra.run.smithi088.stderr:election_strategy: 3 2024-06-03T08:36:30.738 INFO:teuthology.orchestra.run.smithi088.stderr:0: v1:172.21.15.88:6789/0 mon.a 2024-06-03T08:36:30.738 INFO:teuthology.orchestra.run.smithi088.stderr:1: v1:172.21.15.192:6789/0 mon.b 2024-06-03T08:36:30.738 INFO:teuthology.orchestra.run.smithi088.stderr:2: v1:172.21.15.88:6790/0 mon.c 2024-06-03T08:36:30.738 INFO:teuthology.orchestra.run.smithi088.stderr: 2024-06-03T08:36:30.738 INFO:teuthology.orchestra.run.smithi088.stderr:2024-06-03T08:36:30.736+0000 7fa98b7fe640 4 set_mon_vals no callback set 2024-06-03T08:36:30.739 INFO:teuthology.orchestra.run.smithi088.stderr:2024-06-03T08:36:30.737+0000 7fa989ffb640 1 -- 172.21.15.88:0/4052359507 <== mon.1 v1:172.21.15.192:6789/0 7 ==== mgrmap(e 5) v1 ==== 89110+0+0 (unknown 873533973 0 0) 0x7fa98c003e60 con 0x7fa9941450f0 2024-06-03T08:36:30.739 INFO:teuthology.orchestra.run.smithi088.stderr:2024-06-03T08:36:30.738+0000 7fa999405640 10 monclient: get_auth_request con 0x7fa97806d010 auth_method 0 2024-06-03T08:36:30.739 INFO:teuthology.orchestra.run.smithi088.stderr:2024-06-03T08:36:30.738+0000 7fa989ffb640 10 client.4285.objecter ms_handle_connect 0x7fa97806d010 2024-06-03T08:36:30.978 INFO:teuthology.orchestra.run.smithi088.stderr:2024-06-03T08:36:30.977+0000 7fa989ffb640 1 -- 172.21.15.88:0/4052359507 <== mon.1 v1:172.21.15.192:6789/0 8 ==== osd_map(25..25 src has 1..25) v4 ==== 4090+0+0 (unknown 2269083415 0 0) 0x7fa98c0543b0 con 0x7fa9941450f0 2024-06-03T08:36:30.978 INFO:teuthology.orchestra.run.smithi088.stderr:2024-06-03T08:36:30.977+0000 7fa989ffb640 10 client.4285.objecter ms_dispatch 0x7fa994000b90 osd_map(25..25 src has 1..25) v4 2024-06-03T08:36:30.978 INFO:teuthology.orchestra.run.smithi088.stderr:2024-06-03T08:36:30.977+0000 7fa989ffb640 3 client.4285.objecter handle_osd_map got epochs [25,25] > 0 2024-06-03T08:36:30.978 INFO:teuthology.orchestra.run.smithi088.stderr:2024-06-03T08:36:30.977+0000 7fa989ffb640 3 client.4285.objecter handle_osd_map decoding full epoch 25 2024-06-03T08:36:30.978 INFO:teuthology.orchestra.run.smithi088.stderr:2024-06-03T08:36:30.977+0000 7fa989ffb640 20 client.4285.objecter dump_active .. 0 homeless 2024-06-03T08:36:30.980 INFO:teuthology.orchestra.run.smithi088.stderr:2024-06-03T08:36:30.980+0000 7fa989ffb640 1 -- 172.21.15.88:0/4052359507 <== mon.1 v1:172.21.15.192:6789/0 9 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+182498 (unknown 1092875540 0 3016760700) 0x7fa98c04ff60 con 0x7fa9941450f0 2024-06-03T08:36:30.980 INFO:teuthology.orchestra.run.smithi088.stderr:2024-06-03T08:36:30.980+0000 7fa989ffb640 10 monclient: handle_mon_command_ack 1 [{"prefix": "get_command_descriptions"}] 2024-06-03T08:36:30.980 INFO:teuthology.orchestra.run.smithi088.stderr:2024-06-03T08:36:30.980+0000 7fa989ffb640 10 monclient: _finish_command 1 = system:0 2024-06-03T08:36:31.163 INFO:teuthology.orchestra.run.smithi088.stderr:2024-06-03T08:36:31.163+0000 7fa99b669640 10 start_mon_command cmd=[{"prefix": "osd dump", "format": "json"}] 2024-06-03T08:36:31.164 INFO:teuthology.orchestra.run.smithi088.stderr:2024-06-03T08:36:31.163+0000 7fa99b669640 10 monclient: _send_command 2 [{"prefix": "osd dump", "format": "json"}] 2024-06-03T08:36:31.164 INFO:teuthology.orchestra.run.smithi088.stderr:2024-06-03T08:36:31.163+0000 7fa99b669640 10 monclient: _send_mon_message to mon.b at v1:172.21.15.192:6789/0 2024-06-03T08:36:31.164 INFO:teuthology.orchestra.run.smithi088.stderr:2024-06-03T08:36:31.163+0000 7fa99b669640 1 -- 172.21.15.88:0/4052359507 --> v1:172.21.15.192:6789/0 -- mon_command({"prefix": "osd dump", "format": "json"} v 0) v1 -- 0x7fa9940a9e60 con 0x7fa9941450f0 2024-06-03T08:36:31.194 INFO:teuthology.orchestra.run.smithi088.stderr:2024-06-03T08:36:31.194+0000 7fa989ffb640 1 -- 172.21.15.88:0/4052359507 <== mon.1 v1:172.21.15.192:6789/0 10 ==== mon_command_ack([{"prefix": "osd dump", "format": "json"}]=0 v25) v1 ==== 74+0+12431 (unknown 780366164 0 1565932004) 0x7fa98c051e90 con 0x7fa9941450f0 2024-06-03T08:36:31.194 INFO:teuthology.orchestra.run.smithi088.stderr:2024-06-03T08:36:31.194+0000 7fa989ffb640 10 monclient: handle_mon_command_ack 2 [{"prefix": "osd dump", "format": "json"}] 2024-06-03T08:36:31.194 INFO:teuthology.orchestra.run.smithi088.stderr:2024-06-03T08:36:31.194+0000 7fa989ffb640 10 monclient: _finish_command 2 = system:0 2024-06-03T08:36:31.195 INFO:teuthology.orchestra.run.smithi088.stdout: 2024-06-03T08:36:31.195 INFO:teuthology.orchestra.run.smithi088.stdout:{"epoch":25,"fsid":"ec3c3fdd-3ac6-4d7d-9412-683130ed8b04","created":"2024-06-03T08:35:26.799638+0000","modified":"2024-06-03T08:36:30.620322+0000","last_up_change":"2024-06-03T08:36:23.712633+0000","last_in_change":"2024-06-03T08:35:32.183068+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":5,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":2,"max_osd":8,"require_min_compat_client":"luminous","min_compat_client":"jewel","require_osd_release":"reef","allow_crimson":false,"pools":[{"pool":1,"pool_name":".mgr","create_time":"2024-06-03T08:36:15.192388+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":1,"pg_placement_num":1,"pg_placement_num_target":1,"pg_num_target":1,"pg_num_pending":1,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"19","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_max":32,"pg_num_min":1},"application_metadata":{"mgr":{}},"read_balance":{"score_acting":8,"score_stable":8,"optimal_score":0.25,"raw_score_acting":2,"raw_score_stable":2,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":2,"pool_name":"rbd","create_time":"2024-06-03T08:36:28.156336+0000","flags":32769,"flags_names":"hashpspool,creating","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":8,"pg_placement_num":8,"pg_placement_num_target":8,"pg_num_target":8,"pg_num_pending":8,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"24","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{},"application_metadata":{"rbd":{}},"read_balance":{"score_acting":4.9800000190734863,"score_stable":4.9800000190734863,"optimal_score":0.87999999523162842,"raw_score_acting":4.380000114440918,"raw_score_stable":4.380000114440918,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}}],"osds":[{"osd":0,"uuid":"b25ab3bb-3587-45d0-991a-6d41bb529b37","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":14,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.88:6808","nonce":35066}]},"cluster_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.88:6809","nonce":35066}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.88:6811","nonce":35066}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.88:6810","nonce":35066}]},"public_addr":"172.21.15.88:6808/35066","cluster_addr":"172.21.15.88:6809/35066","heartbeat_back_addr":"172.21.15.88:6811/35066","heartbeat_front_addr":"172.21.15.88:6810/35066","state":["exists","up"]},{"osd":1,"uuid":"00d160e7-9773-41d8-8d17-21e70620a0fd","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":14,"up_thru":20,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.88:6800","nonce":35067}]},"cluster_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.88:6801","nonce":35067}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.88:6804","nonce":35067}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.88:6803","nonce":35067}]},"public_addr":"172.21.15.88:6800/35067","cluster_addr":"172.21.15.88:6801/35067","heartbeat_back_addr":"172.21.15.88:6804/35067","heartbeat_front_addr":"172.21.15.88:6803/35067","state":["exists","up"]},{"osd":2,"uuid":"addfb2a7-2bde-4cc3-98cf-17fd2bdfa3a9","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":20,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.88:6802","nonce":35065}]},"cluster_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.88:6805","nonce":35065}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.88:6807","nonce":35065}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.88:6806","nonce":35065}]},"public_addr":"172.21.15.88:6802/35065","cluster_addr":"172.21.15.88:6805/35065","heartbeat_back_addr":"172.21.15.88:6807/35065","heartbeat_front_addr":"172.21.15.88:6806/35065","state":["exists","up"]},{"osd":3,"uuid":"6c5770b0-c93c-4101-8c53-bd761b8abcde","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":14,"up_thru":23,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.88:6812","nonce":35068}]},"cluster_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.88:6813","nonce":35068}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.88:6815","nonce":35068}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.88:6814","nonce":35068}]},"public_addr":"172.21.15.88:6812/35068","cluster_addr":"172.21.15.88:6813/35068","heartbeat_back_addr":"172.21.15.88:6815/35068","heartbeat_front_addr":"172.21.15.88:6814/35068","state":["exists","up"]},{"osd":4,"uuid":"f001b3cb-2081-428b-bac3-b3df3c19b978","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":20,"up_thru":23,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.192:6809","nonce":34785}]},"cluster_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.192:6810","nonce":34785}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.192:6812","nonce":34785}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.192:6811","nonce":34785}]},"public_addr":"172.21.15.192:6809/34785","cluster_addr":"172.21.15.192:6810/34785","heartbeat_back_addr":"172.21.15.192:6812/34785","heartbeat_front_addr":"172.21.15.192:6811/34785","state":["exists","up"]},{"osd":5,"uuid":"3cc263e9-e9bf-45e0-a3b7-83a76469eb70","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":20,"up_thru":23,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.192:6805","nonce":34786}]},"cluster_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.192:6806","nonce":34786}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.192:6808","nonce":34786}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.192:6807","nonce":34786}]},"public_addr":"172.21.15.192:6805/34786","cluster_addr":"172.21.15.192:6806/34786","heartbeat_back_addr":"172.21.15.192:6808/34786","heartbeat_front_addr":"172.21.15.192:6807/34786","state":["exists","up"]},{"osd":6,"uuid":"82c83316-809f-4901-a5ed-34b6a075b393","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":21,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.192:6813","nonce":34788}]},"cluster_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.192:6814","nonce":34788}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.192:6816","nonce":34788}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.192:6815","nonce":34788}]},"public_addr":"172.21.15.192:6813/34788","cluster_addr":"172.21.15.192:6814/34788","heartbeat_back_addr":"172.21.15.192:6816/34788","heartbeat_front_addr":"172.21.15.192:6815/34788","state":["exists","up"]},{"osd":7,"uuid":"aafe0716-1102-4eae-aac6-0f81bb154e7d","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":20,"up_thru":23,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.192:6801","nonce":34787}]},"cluster_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.192:6802","nonce":34787}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.192:6804","nonce":34787}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.192:6803","nonce":34787}]},"public_addr":"172.21.15.192:6801/34787","cluster_addr":"172.21.15.192:6802/34787","heartbeat_back_addr":"172.21.15.192:6804/34787","heartbeat_front_addr":"172.21.15.192:6803/34787","state":["exists","up"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2024-06-03T08:36:10.027533+0000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2024-06-03T08:36:09.853335+0000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2024-06-03T08:36:10.789609+0000","dead_epoch":0},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2024-06-03T08:36:09.824468+0000","dead_epoch":0},{"osd":4,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2024-06-03T08:36:10.117232+0000","dead_epoch":0},{"osd":5,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2024-06-03T08:36:10.469993+0000","dead_epoch":0},{"osd":6,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2024-06-03T08:36:10.789874+0000","dead_epoch":0},{"osd":7,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2024-06-03T08:36:09.541793+0000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[],"pg_upmap_primaries":[],"pg_temp":[],"primary_temp":[],"blocklist":{},"range_blocklist":{},"erasure_code_profiles":{"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","technique":"reed_sol_van"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2024-06-03T08:36:31.196 INFO:teuthology.orchestra.run.smithi088.stderr:2024-06-03T08:36:31.195+0000 7fa99b669640 10 librados: watch_flush enter 2024-06-03T08:36:31.197 INFO:teuthology.orchestra.run.smithi088.stderr:2024-06-03T08:36:31.195+0000 7fa99b669640 10 librados: watch_flush exit 2024-06-03T08:36:31.197 INFO:teuthology.orchestra.run.smithi088.stderr:2024-06-03T08:36:31.195+0000 7fa99b669640 20 client.4285.objecter shutdown clearing up homeless session... 2024-06-03T08:36:31.197 INFO:teuthology.orchestra.run.smithi088.stderr:2024-06-03T08:36:31.195+0000 7fa99b669640 10 client.4285.objecter successfully canceled tick 2024-06-03T08:36:31.197 INFO:teuthology.orchestra.run.smithi088.stderr:2024-06-03T08:36:31.196+0000 7fa99b669640 1 -- 172.21.15.88:0/4052359507 >> v1:172.21.15.192:6800/34229 conn(0x7fa97806d010 legacy=0x7fa97806f4d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-06-03T08:36:31.197 INFO:teuthology.orchestra.run.smithi088.stderr:2024-06-03T08:36:31.196+0000 7fa99b669640 10 monclient: shutdown 2024-06-03T08:36:31.197 INFO:teuthology.orchestra.run.smithi088.stderr:2024-06-03T08:36:31.196+0000 7fa99b669640 20 monclient: shutdown discarding 0 pending message(s) 2024-06-03T08:36:31.197 INFO:teuthology.orchestra.run.smithi088.stderr:2024-06-03T08:36:31.196+0000 7fa99b669640 1 -- 172.21.15.88:0/4052359507 >> v1:172.21.15.192:6789/0 conn(0x7fa9941450f0 legacy=0x7fa9941d6c90 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-06-03T08:36:31.197 INFO:teuthology.orchestra.run.smithi088.stderr:2024-06-03T08:36:31.196+0000 7fa99b669640 1 -- 172.21.15.88:0/4052359507 shutdown_connections 2024-06-03T08:36:31.197 INFO:teuthology.orchestra.run.smithi088.stderr:2024-06-03T08:36:31.196+0000 7fa99b669640 1 -- 172.21.15.88:0/4052359507 >> 172.21.15.88:0/4052359507 conn(0x7fa9940b70d0 msgr2=0x7fa99413e580 unknown :-1 s=STATE_NONE l=0).mark_down 2024-06-03T08:36:31.197 INFO:teuthology.orchestra.run.smithi088.stderr:2024-06-03T08:36:31.196+0000 7fa99b669640 1 -- 172.21.15.88:0/4052359507 shutdown_connections 2024-06-03T08:36:31.197 INFO:teuthology.orchestra.run.smithi088.stderr:2024-06-03T08:36:31.196+0000 7fa99b669640 1 -- 172.21.15.88:0/4052359507 wait complete. 2024-06-03T08:36:31.197 INFO:teuthology.orchestra.run.smithi088.stderr:2024-06-03T08:36:31.196+0000 7fa99b669640 1 librados: shutdown 2024-06-03T08:36:31.208 INFO:tasks.ceph.ceph_manager.ceph:all up! 2024-06-03T08:36:31.208 DEBUG:teuthology.orchestra.run.smithi088:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd dump --format=json 2024-06-03T08:36:31.306 INFO:teuthology.orchestra.run.smithi088.stderr:2024-06-03T08:36:31.306+0000 7f3783099640 10 monclient: get_monmap_and_config 2024-06-03T08:36:31.306 INFO:teuthology.orchestra.run.smithi088.stderr:2024-06-03T08:36:31.306+0000 7f3783099640 10 monclient: build_initial_monmap 2024-06-03T08:36:31.306 INFO:teuthology.orchestra.run.smithi088.stderr:2024-06-03T08:36:31.306+0000 7f3783099640 10 monclient: monmap: 2024-06-03T08:36:31.307 INFO:teuthology.orchestra.run.smithi088.stderr:epoch 0 2024-06-03T08:36:31.307 INFO:teuthology.orchestra.run.smithi088.stderr:fsid ec3c3fdd-3ac6-4d7d-9412-683130ed8b04 2024-06-03T08:36:31.307 INFO:teuthology.orchestra.run.smithi088.stderr:last_changed 2024-06-03T08:36:31.307237+0000 2024-06-03T08:36:31.307 INFO:teuthology.orchestra.run.smithi088.stderr:created 2024-06-03T08:36:31.307237+0000 2024-06-03T08:36:31.307 INFO:teuthology.orchestra.run.smithi088.stderr:min_mon_release 0 (unknown) 2024-06-03T08:36:31.307 INFO:teuthology.orchestra.run.smithi088.stderr:election_strategy: 3 2024-06-03T08:36:31.307 INFO:teuthology.orchestra.run.smithi088.stderr:0: v1:172.21.15.88:6789/0 mon.noname-a 2024-06-03T08:36:31.307 INFO:teuthology.orchestra.run.smithi088.stderr:1: v1:172.21.15.192:6789/0 mon.noname-c 2024-06-03T08:36:31.307 INFO:teuthology.orchestra.run.smithi088.stderr:2: v1:172.21.15.88:6790/0 mon.noname-b 2024-06-03T08:36:31.307 INFO:teuthology.orchestra.run.smithi088.stderr: 2024-06-03T08:36:31.309 INFO:teuthology.orchestra.run.smithi088.stderr:2024-06-03T08:36:31.306+0000 7f3783099640 1 Processor -- start 2024-06-03T08:36:31.309 INFO:teuthology.orchestra.run.smithi088.stderr:2024-06-03T08:36:31.307+0000 7f3783099640 1 -- start start 2024-06-03T08:36:31.309 INFO:teuthology.orchestra.run.smithi088.stderr:2024-06-03T08:36:31.307+0000 7f3783099640 10 monclient: init 2024-06-03T08:36:31.309 INFO:teuthology.orchestra.run.smithi088.stderr:2024-06-03T08:36:31.307+0000 7f3783099640 10 monclient: _reopen_session rank -1 2024-06-03T08:36:31.309 INFO:teuthology.orchestra.run.smithi088.stderr:2024-06-03T08:36:31.307+0000 7f3783099640 10 monclient: _add_conns ranks=[1,2,0] 2024-06-03T08:36:31.309 INFO:teuthology.orchestra.run.smithi088.stderr:2024-06-03T08:36:31.307+0000 7f3783099640 10 monclient(hunting): picked mon.noname-c con 0x7f377c0a8960 addr v1:172.21.15.192:6789/0 2024-06-03T08:36:31.309 INFO:teuthology.orchestra.run.smithi088.stderr:2024-06-03T08:36:31.307+0000 7f3783099640 10 monclient(hunting): picked mon.noname-b con 0x7f377c1450f0 addr v1:172.21.15.88:6790/0 2024-06-03T08:36:31.309 INFO:teuthology.orchestra.run.smithi088.stderr:2024-06-03T08:36:31.307+0000 7f3783099640 10 monclient(hunting): picked mon.noname-a con 0x7f377c13eec0 addr v1:172.21.15.88:6789/0 2024-06-03T08:36:31.309 INFO:teuthology.orchestra.run.smithi088.stderr:2024-06-03T08:36:31.307+0000 7f3783099640 1 -- --> v1:172.21.15.88:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f377c144c60 con 0x7f377c13eec0 2024-06-03T08:36:31.309 INFO:teuthology.orchestra.run.smithi088.stderr:2024-06-03T08:36:31.307+0000 7f3783099640 1 -- --> v1:172.21.15.192:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f377c0b3930 con 0x7f377c0a8960 2024-06-03T08:36:31.309 INFO:teuthology.orchestra.run.smithi088.stderr:2024-06-03T08:36:31.307+0000 7f3783099640 1 -- --> v1:172.21.15.88:6790/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f377c0b1ad0 con 0x7f377c1450f0 2024-06-03T08:36:31.309 INFO:teuthology.orchestra.run.smithi088.stderr:2024-06-03T08:36:31.307+0000 7f3783099640 10 monclient(hunting): _renew_subs 2024-06-03T08:36:31.310 INFO:teuthology.orchestra.run.smithi088.stderr:2024-06-03T08:36:31.307+0000 7f3783099640 10 monclient(hunting): authenticate will time out at 1247.117592s 2024-06-03T08:36:31.310 INFO:teuthology.orchestra.run.smithi088.stderr:2024-06-03T08:36:31.308+0000 7f3780e35640 1 --1- >> v1:172.21.15.192:6789/0 conn(0x7f377c0a8960 0x7f377c0a8d40 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.192:6789/0 says I am v1:172.21.15.88:52306/0 (socket says 172.21.15.88:52306) 2024-06-03T08:36:31.310 INFO:teuthology.orchestra.run.smithi088.stderr:2024-06-03T08:36:31.308+0000 7f3780e35640 1 -- 172.21.15.88:0/2923013247 learned_addr learned my addr 172.21.15.88:0/2923013247 (peer_addr_for_me v1:172.21.15.88:0/0) 2024-06-03T08:36:31.310 INFO:teuthology.orchestra.run.smithi088.stderr:2024-06-03T08:36:31.308+0000 7f3781636640 1 --1- >> v1:172.21.15.88:6789/0 conn(0x7f377c13eec0 0x7f377c14ce60 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.88:6789/0 says I am v1:172.21.15.88:60246/0 (socket says 172.21.15.88:60246) 2024-06-03T08:36:31.310 INFO:teuthology.orchestra.run.smithi088.stderr:2024-06-03T08:36:31.310+0000 7f377b7fe640 1 -- 172.21.15.88:0/2923013247 <== mon.1 v1:172.21.15.192:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 3896493918 0 0) 0x7f377c0b3930 con 0x7f377c0a8960 2024-06-03T08:36:31.311 INFO:teuthology.orchestra.run.smithi088.stderr:2024-06-03T08:36:31.310+0000 7f377b7fe640 10 monclient(hunting): _negotiate 2024-06-03T08:36:31.311 INFO:teuthology.orchestra.run.smithi088.stderr:2024-06-03T08:36:31.310+0000 7f377b7fe640 10 monclient(hunting): _init_auth method 2 2024-06-03T08:36:31.311 INFO:teuthology.orchestra.run.smithi088.stderr:2024-06-03T08:36:31.310+0000 7f377b7fe640 10 monclient(hunting): _init_auth creating new auth 2024-06-03T08:36:31.311 INFO:teuthology.orchestra.run.smithi088.stderr:2024-06-03T08:36:31.310+0000 7f377b7fe640 10 monclient(hunting): my global_id is 4288 2024-06-03T08:36:31.311 INFO:teuthology.orchestra.run.smithi088.stderr:2024-06-03T08:36:31.310+0000 7f377b7fe640 1 -- 172.21.15.88:0/2923013247 --> v1:172.21.15.192:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f3768003520 con 0x7f377c0a8960 2024-06-03T08:36:31.311 INFO:teuthology.orchestra.run.smithi088.stderr:2024-06-03T08:36:31.311+0000 7f377b7fe640 1 -- 172.21.15.88:0/2923013247 <== mon.2 v1:172.21.15.88:6790/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 132709408 0 0) 0x7f377c0b1ad0 con 0x7f377c1450f0 2024-06-03T08:36:31.312 INFO:teuthology.orchestra.run.smithi088.stderr:2024-06-03T08:36:31.311+0000 7f377b7fe640 10 monclient(hunting): _negotiate 2024-06-03T08:36:31.312 INFO:teuthology.orchestra.run.smithi088.stderr:2024-06-03T08:36:31.311+0000 7f377b7fe640 10 monclient(hunting): _init_auth method 2 2024-06-03T08:36:31.312 INFO:teuthology.orchestra.run.smithi088.stderr:2024-06-03T08:36:31.311+0000 7f377b7fe640 10 monclient(hunting): _init_auth creating new auth 2024-06-03T08:36:31.312 INFO:teuthology.orchestra.run.smithi088.stderr:2024-06-03T08:36:31.311+0000 7f377b7fe640 10 monclient(hunting): my global_id is 4304 2024-06-03T08:36:31.312 INFO:teuthology.orchestra.run.smithi088.stderr:2024-06-03T08:36:31.311+0000 7f377b7fe640 1 -- 172.21.15.88:0/2923013247 --> v1:172.21.15.88:6790/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f377c0b3930 con 0x7f377c1450f0 2024-06-03T08:36:31.312 INFO:teuthology.orchestra.run.smithi088.stderr:2024-06-03T08:36:31.311+0000 7f377b7fe640 1 -- 172.21.15.88:0/2923013247 <== mon.0 v1:172.21.15.88:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 942569028 0 0) 0x7f377c144c60 con 0x7f377c13eec0 2024-06-03T08:36:31.312 INFO:teuthology.orchestra.run.smithi088.stderr:2024-06-03T08:36:31.311+0000 7f377b7fe640 10 monclient(hunting): _negotiate 2024-06-03T08:36:31.312 INFO:teuthology.orchestra.run.smithi088.stderr:2024-06-03T08:36:31.311+0000 7f377b7fe640 10 monclient(hunting): _init_auth method 2 2024-06-03T08:36:31.312 INFO:teuthology.orchestra.run.smithi088.stderr:2024-06-03T08:36:31.311+0000 7f377b7fe640 10 monclient(hunting): _init_auth creating new auth 2024-06-03T08:36:31.312 INFO:teuthology.orchestra.run.smithi088.stderr:2024-06-03T08:36:31.311+0000 7f377b7fe640 10 monclient(hunting): my global_id is 4284 2024-06-03T08:36:31.312 INFO:teuthology.orchestra.run.smithi088.stderr:2024-06-03T08:36:31.311+0000 7f377b7fe640 1 -- 172.21.15.88:0/2923013247 --> v1:172.21.15.88:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f377c0b1ad0 con 0x7f377c13eec0 2024-06-03T08:36:31.314 INFO:teuthology.orchestra.run.smithi088.stderr:2024-06-03T08:36:31.313+0000 7f377b7fe640 1 -- 172.21.15.88:0/2923013247 <== mon.1 v1:172.21.15.192:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 214+0+0 (unknown 4063153370 0 0) 0x7f3768003520 con 0x7f377c0a8960 2024-06-03T08:36:31.314 INFO:teuthology.orchestra.run.smithi088.stderr:2024-06-03T08:36:31.313+0000 7f377b7fe640 1 -- 172.21.15.88:0/2923013247 >> v1:172.21.15.88:6790/0 conn(0x7f377c1450f0 legacy=0x7f377c13e7b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-06-03T08:36:31.314 INFO:teuthology.orchestra.run.smithi088.stderr:2024-06-03T08:36:31.313+0000 7f377b7fe640 1 -- 172.21.15.88:0/2923013247 >> v1:172.21.15.88:6789/0 conn(0x7f377c13eec0 legacy=0x7f377c14ce60 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-06-03T08:36:31.314 INFO:teuthology.orchestra.run.smithi088.stderr:2024-06-03T08:36:31.313+0000 7f377b7fe640 10 monclient: _finish_hunting 0 2024-06-03T08:36:31.314 INFO:teuthology.orchestra.run.smithi088.stderr:2024-06-03T08:36:31.313+0000 7f377b7fe640 1 monclient: found mon.noname-c 2024-06-03T08:36:31.314 INFO:teuthology.orchestra.run.smithi088.stderr:2024-06-03T08:36:31.313+0000 7f377b7fe640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-06-03T08:36:31.314 INFO:teuthology.orchestra.run.smithi088.stderr:2024-06-03T08:36:31.313+0000 7f377b7fe640 10 monclient: _send_mon_message to mon.noname-c at v1:172.21.15.192:6789/0 2024-06-03T08:36:31.314 INFO:teuthology.orchestra.run.smithi088.stderr:2024-06-03T08:36:31.313+0000 7f377b7fe640 1 -- 172.21.15.88:0/2923013247 --> v1:172.21.15.192:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f377c0b1c60 con 0x7f377c0a8960 2024-06-03T08:36:31.314 INFO:teuthology.orchestra.run.smithi088.stderr:2024-06-03T08:36:31.313+0000 7f377b7fe640 10 monclient: _finish_auth 0 2024-06-03T08:36:31.315 INFO:teuthology.orchestra.run.smithi088.stderr:2024-06-03T08:36:31.313+0000 7f377b7fe640 10 monclient: _check_auth_tickets 2024-06-03T08:36:31.315 INFO:teuthology.orchestra.run.smithi088.stderr:2024-06-03T08:36:31.313+0000 7f377b7fe640 20 monclient: _check_auth_rotating not needed by client.admin 2024-06-03T08:36:31.315 INFO:teuthology.orchestra.run.smithi088.stderr:2024-06-03T08:36:31.313+0000 7f377b7fe640 1 -- 172.21.15.88:0/2923013247 <== mon.1 v1:172.21.15.192:6789/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 2234374065 0 0) 0x7f3764001610 con 0x7f377c0a8960 2024-06-03T08:36:31.315 INFO:teuthology.orchestra.run.smithi088.stderr:2024-06-03T08:36:31.313+0000 7f3783099640 5 monclient: authenticate success, global_id 4288 2024-06-03T08:36:31.315 INFO:teuthology.orchestra.run.smithi088.stderr:2024-06-03T08:36:31.313+0000 7f3783099640 20 monclient: get_monmap_and_config waiting for monmap|config 2024-06-03T08:36:31.315 INFO:teuthology.orchestra.run.smithi088.stderr:2024-06-03T08:36:31.313+0000 7f377b7fe640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-06-03T08:36:31.315 INFO:teuthology.orchestra.run.smithi088.stderr:2024-06-03T08:36:31.313+0000 7f377b7fe640 10 monclient: got monmap 1 from mon.noname-c (according to old e1) 2024-06-03T08:36:31.315 INFO:teuthology.orchestra.run.smithi088.stderr:2024-06-03T08:36:31.313+0000 7f377b7fe640 10 monclient: dump: 2024-06-03T08:36:31.315 INFO:teuthology.orchestra.run.smithi088.stderr:epoch 1 2024-06-03T08:36:31.315 INFO:teuthology.orchestra.run.smithi088.stderr:fsid ec3c3fdd-3ac6-4d7d-9412-683130ed8b04 2024-06-03T08:36:31.315 INFO:teuthology.orchestra.run.smithi088.stderr:last_changed 2024-06-03T08:34:31.885125+0000 2024-06-03T08:36:31.315 INFO:teuthology.orchestra.run.smithi088.stderr:created 2024-06-03T08:34:31.885125+0000 2024-06-03T08:36:31.315 INFO:teuthology.orchestra.run.smithi088.stderr:min_mon_release 18 (reef) 2024-06-03T08:36:31.316 INFO:teuthology.orchestra.run.smithi088.stderr:election_strategy: 3 2024-06-03T08:36:31.316 INFO:teuthology.orchestra.run.smithi088.stderr:0: v1:172.21.15.88:6789/0 mon.a 2024-06-03T08:36:31.316 INFO:teuthology.orchestra.run.smithi088.stderr:1: v1:172.21.15.192:6789/0 mon.b 2024-06-03T08:36:31.316 INFO:teuthology.orchestra.run.smithi088.stderr:2: v1:172.21.15.88:6790/0 mon.c 2024-06-03T08:36:31.316 INFO:teuthology.orchestra.run.smithi088.stderr: 2024-06-03T08:36:31.316 INFO:teuthology.orchestra.run.smithi088.stderr:2024-06-03T08:36:31.313+0000 7f3783099640 20 monclient: get_monmap_and_config waiting for monmap|config 2024-06-03T08:36:31.318 INFO:teuthology.orchestra.run.smithi088.stderr:2024-06-03T08:36:31.315+0000 7f377b7fe640 1 -- 172.21.15.88:0/2923013247 <== mon.1 v1:172.21.15.192:6789/0 4 ==== config(0 keys) v1 ==== 4+0+0 (unknown 0 0 0) 0x7f3764001830 con 0x7f377c0a8960 2024-06-03T08:36:31.319 INFO:teuthology.orchestra.run.smithi088.stderr:2024-06-03T08:36:31.315+0000 7f377b7fe640 10 monclient: handle_config config(0 keys) v1 2024-06-03T08:36:31.319 INFO:teuthology.orchestra.run.smithi088.stderr:2024-06-03T08:36:31.315+0000 7f377b7fe640 1 -- 172.21.15.88:0/2923013247 <== mon.1 v1:172.21.15.192:6789/0 5 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 2234374065 0 0) 0x7f3764001b40 con 0x7f377c0a8960 2024-06-03T08:36:31.319 INFO:teuthology.orchestra.run.smithi088.stderr:2024-06-03T08:36:31.315+0000 7f377b7fe640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-06-03T08:36:31.319 INFO:teuthology.orchestra.run.smithi088.stderr:2024-06-03T08:36:31.315+0000 7f377b7fe640 10 monclient: got monmap 1 from mon.b (according to old e1) 2024-06-03T08:36:31.319 INFO:teuthology.orchestra.run.smithi088.stderr:2024-06-03T08:36:31.315+0000 7f377b7fe640 10 monclient: dump: 2024-06-03T08:36:31.319 INFO:teuthology.orchestra.run.smithi088.stderr:epoch 1 2024-06-03T08:36:31.319 INFO:teuthology.orchestra.run.smithi088.stderr:fsid ec3c3fdd-3ac6-4d7d-9412-683130ed8b04 2024-06-03T08:36:31.319 INFO:teuthology.orchestra.run.smithi088.stderr:last_changed 2024-06-03T08:34:31.885125+0000 2024-06-03T08:36:31.319 INFO:teuthology.orchestra.run.smithi088.stderr:created 2024-06-03T08:34:31.885125+0000 2024-06-03T08:36:31.319 INFO:teuthology.orchestra.run.smithi088.stderr:min_mon_release 18 (reef) 2024-06-03T08:36:31.319 INFO:teuthology.orchestra.run.smithi088.stderr:election_strategy: 3 2024-06-03T08:36:31.319 INFO:teuthology.orchestra.run.smithi088.stderr:0: v1:172.21.15.88:6789/0 mon.a 2024-06-03T08:36:31.319 INFO:teuthology.orchestra.run.smithi088.stderr:1: v1:172.21.15.192:6789/0 mon.b 2024-06-03T08:36:31.320 INFO:teuthology.orchestra.run.smithi088.stderr:2: v1:172.21.15.88:6790/0 mon.c 2024-06-03T08:36:31.320 INFO:teuthology.orchestra.run.smithi088.stderr: 2024-06-03T08:36:31.320 INFO:teuthology.orchestra.run.smithi088.stderr:2024-06-03T08:36:31.315+0000 7f3783099640 10 monclient: get_monmap_and_config success 2024-06-03T08:36:31.320 INFO:teuthology.orchestra.run.smithi088.stderr:2024-06-03T08:36:31.315+0000 7f3783099640 4 set_mon_vals no callback set 2024-06-03T08:36:31.320 INFO:teuthology.orchestra.run.smithi088.stderr:2024-06-03T08:36:31.316+0000 7f3783099640 10 monclient: shutdown 2024-06-03T08:36:31.320 INFO:teuthology.orchestra.run.smithi088.stderr:2024-06-03T08:36:31.316+0000 7f3783099640 20 monclient: shutdown discarding 0 pending message(s) 2024-06-03T08:36:31.320 INFO:teuthology.orchestra.run.smithi088.stderr:2024-06-03T08:36:31.316+0000 7f3783099640 1 -- 172.21.15.88:0/2923013247 >> v1:172.21.15.192:6789/0 conn(0x7f377c0a8960 legacy=0x7f377c0a8d40 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-06-03T08:36:31.320 INFO:teuthology.orchestra.run.smithi088.stderr:2024-06-03T08:36:31.316+0000 7f3783099640 1 -- 172.21.15.88:0/2923013247 shutdown_connections 2024-06-03T08:36:31.320 INFO:teuthology.orchestra.run.smithi088.stderr:2024-06-03T08:36:31.316+0000 7f3783099640 1 -- 172.21.15.88:0/2923013247 >> 172.21.15.88:0/2923013247 conn(0x7f377c0b70d0 msgr2=0x7f377c0b74e0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-06-03T08:36:31.320 INFO:teuthology.orchestra.run.smithi088.stderr:2024-06-03T08:36:31.316+0000 7f3783099640 1 -- 172.21.15.88:0/2923013247 shutdown_connections 2024-06-03T08:36:31.320 INFO:teuthology.orchestra.run.smithi088.stderr:2024-06-03T08:36:31.316+0000 7f3783099640 1 -- 172.21.15.88:0/2923013247 wait complete. 2024-06-03T08:36:31.320 INFO:teuthology.orchestra.run.smithi088.stderr:2024-06-03T08:36:31.316+0000 7f3783099640 10 monclient: build_initial_monmap 2024-06-03T08:36:31.320 INFO:teuthology.orchestra.run.smithi088.stderr:2024-06-03T08:36:31.316+0000 7f3783099640 10 monclient: monmap: 2024-06-03T08:36:31.321 INFO:teuthology.orchestra.run.smithi088.stderr:epoch 0 2024-06-03T08:36:31.321 INFO:teuthology.orchestra.run.smithi088.stderr:fsid 00000000-0000-0000-0000-000000000000 2024-06-03T08:36:31.321 INFO:teuthology.orchestra.run.smithi088.stderr:last_changed 0.000000 2024-06-03T08:36:31.321 INFO:teuthology.orchestra.run.smithi088.stderr:created 0.000000 2024-06-03T08:36:31.321 INFO:teuthology.orchestra.run.smithi088.stderr:min_mon_release 0 (unknown) 2024-06-03T08:36:31.321 INFO:teuthology.orchestra.run.smithi088.stderr:election_strategy: 1 2024-06-03T08:36:31.321 INFO:teuthology.orchestra.run.smithi088.stderr:0: v1:172.21.15.88:6789/0 mon.noname-a 2024-06-03T08:36:31.321 INFO:teuthology.orchestra.run.smithi088.stderr:1: v1:172.21.15.192:6789/0 mon.noname-b 2024-06-03T08:36:31.321 INFO:teuthology.orchestra.run.smithi088.stderr:2: v1:172.21.15.88:6790/0 mon.noname-c 2024-06-03T08:36:31.321 INFO:teuthology.orchestra.run.smithi088.stderr: 2024-06-03T08:36:31.321 INFO:teuthology.orchestra.run.smithi088.stderr:2024-06-03T08:36:31.317+0000 7f3783099640 1 librados: starting msgr at 2024-06-03T08:36:31.321 INFO:teuthology.orchestra.run.smithi088.stderr:2024-06-03T08:36:31.317+0000 7f3783099640 1 librados: starting objecter 2024-06-03T08:36:31.321 INFO:teuthology.orchestra.run.smithi088.stderr:2024-06-03T08:36:31.317+0000 7f3783099640 1 Processor -- start 2024-06-03T08:36:31.322 INFO:teuthology.orchestra.run.smithi088.stderr:2024-06-03T08:36:31.317+0000 7f3783099640 1 -- start start 2024-06-03T08:36:31.322 INFO:teuthology.orchestra.run.smithi088.stderr:2024-06-03T08:36:31.317+0000 7f3783099640 1 librados: setting wanted keys 2024-06-03T08:36:31.322 INFO:teuthology.orchestra.run.smithi088.stderr:2024-06-03T08:36:31.317+0000 7f3783099640 1 librados: calling monclient init 2024-06-03T08:36:31.322 INFO:teuthology.orchestra.run.smithi088.stderr:2024-06-03T08:36:31.317+0000 7f3783099640 10 monclient: init 2024-06-03T08:36:31.322 INFO:teuthology.orchestra.run.smithi088.stderr:2024-06-03T08:36:31.318+0000 7f3783099640 10 monclient: _reopen_session rank -1 2024-06-03T08:36:31.322 INFO:teuthology.orchestra.run.smithi088.stderr:2024-06-03T08:36:31.318+0000 7f3783099640 10 monclient: _add_conns ranks=[2,0,1] 2024-06-03T08:36:31.322 INFO:teuthology.orchestra.run.smithi088.stderr:2024-06-03T08:36:31.318+0000 7f3783099640 10 monclient(hunting): picked mon.noname-c con 0x7f377c0a8960 addr v1:172.21.15.88:6790/0 2024-06-03T08:36:31.322 INFO:teuthology.orchestra.run.smithi088.stderr:2024-06-03T08:36:31.318+0000 7f3783099640 10 monclient(hunting): picked mon.noname-a con 0x7f377c13eec0 addr v1:172.21.15.88:6789/0 2024-06-03T08:36:31.322 INFO:teuthology.orchestra.run.smithi088.stderr:2024-06-03T08:36:31.318+0000 7f3783099640 10 monclient(hunting): picked mon.noname-b con 0x7f377c1450f0 addr v1:172.21.15.192:6789/0 2024-06-03T08:36:31.322 INFO:teuthology.orchestra.run.smithi088.stderr:2024-06-03T08:36:31.318+0000 7f3783099640 1 -- --> v1:172.21.15.88:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f377c1d9dd0 con 0x7f377c13eec0 2024-06-03T08:36:31.322 INFO:teuthology.orchestra.run.smithi088.stderr:2024-06-03T08:36:31.318+0000 7f3783099640 1 -- --> v1:172.21.15.192:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f377c1411c0 con 0x7f377c1450f0 2024-06-03T08:36:31.322 INFO:teuthology.orchestra.run.smithi088.stderr:2024-06-03T08:36:31.318+0000 7f3783099640 1 -- --> v1:172.21.15.88:6790/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f377c13f380 con 0x7f377c0a8960 2024-06-03T08:36:31.322 INFO:teuthology.orchestra.run.smithi088.stderr:2024-06-03T08:36:31.318+0000 7f3783099640 10 monclient(hunting): _renew_subs 2024-06-03T08:36:31.322 INFO:teuthology.orchestra.run.smithi088.stderr:2024-06-03T08:36:31.318+0000 7f3783099640 10 monclient(hunting): authenticate will time out at 1247.128044s 2024-06-03T08:36:31.322 INFO:teuthology.orchestra.run.smithi088.stderr:2024-06-03T08:36:31.318+0000 7f3781636640 1 --1- >> v1:172.21.15.192:6789/0 conn(0x7f377c1450f0 0x7f377c1e1240 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=1).handle_server_banner_and_identify peer v1:172.21.15.192:6789/0 says I am v1:172.21.15.88:52308/0 (socket says 172.21.15.88:52308) 2024-06-03T08:36:31.323 INFO:teuthology.orchestra.run.smithi088.stderr:2024-06-03T08:36:31.318+0000 7f3781636640 1 -- 172.21.15.88:0/3701686329 learned_addr learned my addr 172.21.15.88:0/3701686329 (peer_addr_for_me v1:172.21.15.88:0/0) 2024-06-03T08:36:31.323 INFO:teuthology.orchestra.run.smithi088.stderr:2024-06-03T08:36:31.318+0000 7f37797fa640 10 client.?.objecter ms_handle_connect 0x7f377c1450f0 2024-06-03T08:36:31.323 INFO:teuthology.orchestra.run.smithi088.stderr:2024-06-03T08:36:31.318+0000 7f37797fa640 10 client.?.objecter resend_mon_ops 2024-06-03T08:36:31.324 INFO:teuthology.orchestra.run.smithi088.stderr:2024-06-03T08:36:31.319+0000 7f37797fa640 10 client.?.objecter ms_handle_connect 0x7f377c13eec0 2024-06-03T08:36:31.324 INFO:teuthology.orchestra.run.smithi088.stderr:2024-06-03T08:36:31.319+0000 7f37797fa640 10 client.?.objecter resend_mon_ops 2024-06-03T08:36:31.324 INFO:teuthology.orchestra.run.smithi088.stderr:2024-06-03T08:36:31.319+0000 7f37797fa640 10 client.?.objecter ms_handle_connect 0x7f377c0a8960 2024-06-03T08:36:31.324 INFO:teuthology.orchestra.run.smithi088.stderr:2024-06-03T08:36:31.319+0000 7f37797fa640 10 client.?.objecter resend_mon_ops 2024-06-03T08:36:31.324 INFO:teuthology.orchestra.run.smithi088.stderr:2024-06-03T08:36:31.320+0000 7f37797fa640 1 -- 172.21.15.88:0/3701686329 <== mon.1 v1:172.21.15.192:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 3877349029 0 0) 0x7f377c1411c0 con 0x7f377c1450f0 2024-06-03T08:36:31.324 INFO:teuthology.orchestra.run.smithi088.stderr:2024-06-03T08:36:31.320+0000 7f37797fa640 10 monclient(hunting): _negotiate 2024-06-03T08:36:31.324 INFO:teuthology.orchestra.run.smithi088.stderr:2024-06-03T08:36:31.320+0000 7f37797fa640 10 monclient(hunting): _init_auth method 2 2024-06-03T08:36:31.324 INFO:teuthology.orchestra.run.smithi088.stderr:2024-06-03T08:36:31.320+0000 7f37797fa640 10 monclient(hunting): _init_auth creating new auth 2024-06-03T08:36:31.324 INFO:teuthology.orchestra.run.smithi088.stderr:2024-06-03T08:36:31.320+0000 7f37797fa640 10 monclient(hunting): my global_id is 4291 2024-06-03T08:36:31.324 INFO:teuthology.orchestra.run.smithi088.stderr:2024-06-03T08:36:31.320+0000 7f37797fa640 1 -- 172.21.15.88:0/3701686329 --> v1:172.21.15.192:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f375c004eb0 con 0x7f377c1450f0 2024-06-03T08:36:31.324 INFO:teuthology.orchestra.run.smithi088.stderr:2024-06-03T08:36:31.321+0000 7f37797fa640 1 -- 172.21.15.88:0/3701686329 <== mon.0 v1:172.21.15.88:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 398278103 0 0) 0x7f377c1d9dd0 con 0x7f377c13eec0 2024-06-03T08:36:31.324 INFO:teuthology.orchestra.run.smithi088.stderr:2024-06-03T08:36:31.321+0000 7f37797fa640 10 monclient(hunting): _negotiate 2024-06-03T08:36:31.324 INFO:teuthology.orchestra.run.smithi088.stderr:2024-06-03T08:36:31.321+0000 7f37797fa640 10 monclient(hunting): _init_auth method 2 2024-06-03T08:36:31.325 INFO:teuthology.orchestra.run.smithi088.stderr:2024-06-03T08:36:31.321+0000 7f37797fa640 10 monclient(hunting): _init_auth creating new auth 2024-06-03T08:36:31.325 INFO:teuthology.orchestra.run.smithi088.stderr:2024-06-03T08:36:31.321+0000 7f37797fa640 10 monclient(hunting): my global_id is 4287 2024-06-03T08:36:31.325 INFO:teuthology.orchestra.run.smithi088.stderr:2024-06-03T08:36:31.321+0000 7f37797fa640 1 -- 172.21.15.88:0/3701686329 --> v1:172.21.15.88:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f377c1411c0 con 0x7f377c13eec0 2024-06-03T08:36:31.325 INFO:teuthology.orchestra.run.smithi088.stderr:2024-06-03T08:36:31.321+0000 7f37797fa640 1 -- 172.21.15.88:0/3701686329 <== mon.2 v1:172.21.15.88:6790/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 1545731264 0 0) 0x7f377c13f380 con 0x7f377c0a8960 2024-06-03T08:36:31.325 INFO:teuthology.orchestra.run.smithi088.stderr:2024-06-03T08:36:31.321+0000 7f37797fa640 10 monclient(hunting): _negotiate 2024-06-03T08:36:31.325 INFO:teuthology.orchestra.run.smithi088.stderr:2024-06-03T08:36:31.321+0000 7f37797fa640 10 monclient(hunting): _init_auth method 2 2024-06-03T08:36:31.325 INFO:teuthology.orchestra.run.smithi088.stderr:2024-06-03T08:36:31.321+0000 7f37797fa640 10 monclient(hunting): _init_auth creating new auth 2024-06-03T08:36:31.325 INFO:teuthology.orchestra.run.smithi088.stderr:2024-06-03T08:36:31.321+0000 7f37797fa640 10 monclient(hunting): my global_id is 4307 2024-06-03T08:36:31.325 INFO:teuthology.orchestra.run.smithi088.stderr:2024-06-03T08:36:31.321+0000 7f37797fa640 1 -- 172.21.15.88:0/3701686329 --> v1:172.21.15.88:6790/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f377c1d9dd0 con 0x7f377c0a8960 2024-06-03T08:36:31.326 INFO:teuthology.orchestra.run.smithi088.stderr:2024-06-03T08:36:31.325+0000 7f37797fa640 1 -- 172.21.15.88:0/3701686329 <== mon.1 v1:172.21.15.192:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 764+0+0 (unknown 3149516261 0 0) 0x7f375c004eb0 con 0x7f377c1450f0 2024-06-03T08:36:31.326 INFO:teuthology.orchestra.run.smithi088.stderr:2024-06-03T08:36:31.325+0000 7f37797fa640 1 -- 172.21.15.88:0/3701686329 --> v1:172.21.15.192:6789/0 -- auth(proto 2 165 bytes epoch 0) v1 -- 0x7f377c13f380 con 0x7f377c1450f0 2024-06-03T08:36:31.326 INFO:teuthology.orchestra.run.smithi088.stderr:2024-06-03T08:36:31.325+0000 7f37797fa640 1 -- 172.21.15.88:0/3701686329 <== mon.1 v1:172.21.15.192:6789/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 2234374065 0 0) 0x7f3774001840 con 0x7f377c1450f0 2024-06-03T08:36:31.326 INFO:teuthology.orchestra.run.smithi088.stderr:2024-06-03T08:36:31.325+0000 7f37797fa640 10 monclient(hunting): handle_monmap mon_map magic: 0 v1 2024-06-03T08:36:31.326 INFO:teuthology.orchestra.run.smithi088.stderr:2024-06-03T08:36:31.325+0000 7f37797fa640 10 monclient(hunting): got monmap 1 from mon.noname-b (according to old e1) 2024-06-03T08:36:31.326 INFO:teuthology.orchestra.run.smithi088.stderr:2024-06-03T08:36:31.325+0000 7f37797fa640 10 monclient(hunting): dump: 2024-06-03T08:36:31.326 INFO:teuthology.orchestra.run.smithi088.stderr:epoch 1 2024-06-03T08:36:31.327 INFO:teuthology.orchestra.run.smithi088.stderr:fsid ec3c3fdd-3ac6-4d7d-9412-683130ed8b04 2024-06-03T08:36:31.327 INFO:teuthology.orchestra.run.smithi088.stderr:last_changed 2024-06-03T08:34:31.885125+0000 2024-06-03T08:36:31.327 INFO:teuthology.orchestra.run.smithi088.stderr:created 2024-06-03T08:34:31.885125+0000 2024-06-03T08:36:31.327 INFO:teuthology.orchestra.run.smithi088.stderr:min_mon_release 18 (reef) 2024-06-03T08:36:31.327 INFO:teuthology.orchestra.run.smithi088.stderr:election_strategy: 3 2024-06-03T08:36:31.327 INFO:teuthology.orchestra.run.smithi088.stderr:0: v1:172.21.15.88:6789/0 mon.a 2024-06-03T08:36:31.327 INFO:teuthology.orchestra.run.smithi088.stderr:1: v1:172.21.15.192:6789/0 mon.b 2024-06-03T08:36:31.327 INFO:teuthology.orchestra.run.smithi088.stderr:2: v1:172.21.15.88:6790/0 mon.c 2024-06-03T08:36:31.327 INFO:teuthology.orchestra.run.smithi088.stderr: 2024-06-03T08:36:31.327 INFO:teuthology.orchestra.run.smithi088.stderr:2024-06-03T08:36:31.325+0000 7f37797fa640 10 monclient(hunting): _finish_auth 0 2024-06-03T08:36:31.329 INFO:teuthology.orchestra.run.smithi088.stderr:2024-06-03T08:36:31.326+0000 7f37797fa640 1 -- 172.21.15.88:0/3701686329 <== mon.0 v1:172.21.15.88:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 764+0+0 (unknown 3122117074 0 0) 0x7f377c1411c0 con 0x7f377c13eec0 2024-06-03T08:36:31.329 INFO:teuthology.orchestra.run.smithi088.stderr:2024-06-03T08:36:31.326+0000 7f37797fa640 1 -- 172.21.15.88:0/3701686329 --> v1:172.21.15.88:6789/0 -- auth(proto 2 165 bytes epoch 0) v1 -- 0x7f375c004eb0 con 0x7f377c13eec0 2024-06-03T08:36:31.329 INFO:teuthology.orchestra.run.smithi088.stderr:2024-06-03T08:36:31.326+0000 7f37797fa640 1 -- 172.21.15.88:0/3701686329 <== mon.0 v1:172.21.15.88:6789/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 2234374065 0 0) 0x7f376c001b90 con 0x7f377c13eec0 2024-06-03T08:36:31.329 INFO:teuthology.orchestra.run.smithi088.stderr:2024-06-03T08:36:31.326+0000 7f37797fa640 10 monclient(hunting): handle_monmap mon_map magic: 0 v1 2024-06-03T08:36:31.329 INFO:teuthology.orchestra.run.smithi088.stderr:2024-06-03T08:36:31.326+0000 7f37797fa640 10 monclient(hunting): got monmap 1 from mon.a (according to old e1) 2024-06-03T08:36:31.329 INFO:teuthology.orchestra.run.smithi088.stderr:2024-06-03T08:36:31.326+0000 7f37797fa640 10 monclient(hunting): dump: 2024-06-03T08:36:31.329 INFO:teuthology.orchestra.run.smithi088.stderr:epoch 1 2024-06-03T08:36:31.329 INFO:teuthology.orchestra.run.smithi088.stderr:fsid ec3c3fdd-3ac6-4d7d-9412-683130ed8b04 2024-06-03T08:36:31.329 INFO:teuthology.orchestra.run.smithi088.stderr:last_changed 2024-06-03T08:34:31.885125+0000 2024-06-03T08:36:31.330 INFO:teuthology.orchestra.run.smithi088.stderr:created 2024-06-03T08:34:31.885125+0000 2024-06-03T08:36:31.330 INFO:teuthology.orchestra.run.smithi088.stderr:min_mon_release 18 (reef) 2024-06-03T08:36:31.330 INFO:teuthology.orchestra.run.smithi088.stderr:election_strategy: 3 2024-06-03T08:36:31.330 INFO:teuthology.orchestra.run.smithi088.stderr:0: v1:172.21.15.88:6789/0 mon.a 2024-06-03T08:36:31.330 INFO:teuthology.orchestra.run.smithi088.stderr:1: v1:172.21.15.192:6789/0 mon.b 2024-06-03T08:36:31.330 INFO:teuthology.orchestra.run.smithi088.stderr:2: v1:172.21.15.88:6790/0 mon.c 2024-06-03T08:36:31.330 INFO:teuthology.orchestra.run.smithi088.stderr: 2024-06-03T08:36:31.330 INFO:teuthology.orchestra.run.smithi088.stderr:2024-06-03T08:36:31.326+0000 7f37797fa640 1 -- 172.21.15.88:0/3701686329 <== mon.2 v1:172.21.15.88:6790/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 764+0+0 (unknown 4049317922 0 0) 0x7f377c1d9dd0 con 0x7f377c0a8960 2024-06-03T08:36:31.330 INFO:teuthology.orchestra.run.smithi088.stderr:2024-06-03T08:36:31.326+0000 7f37797fa640 1 -- 172.21.15.88:0/3701686329 --> v1:172.21.15.88:6790/0 -- auth(proto 2 165 bytes epoch 0) v1 -- 0x7f377c1411c0 con 0x7f377c0a8960 2024-06-03T08:36:31.330 INFO:teuthology.orchestra.run.smithi088.stderr:2024-06-03T08:36:31.326+0000 7f37797fa640 1 -- 172.21.15.88:0/3701686329 <== mon.2 v1:172.21.15.88:6790/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 2234374065 0 0) 0x7f3764001510 con 0x7f377c0a8960 2024-06-03T08:36:31.330 INFO:teuthology.orchestra.run.smithi088.stderr:2024-06-03T08:36:31.326+0000 7f37797fa640 10 monclient(hunting): handle_monmap mon_map magic: 0 v1 2024-06-03T08:36:31.330 INFO:teuthology.orchestra.run.smithi088.stderr:2024-06-03T08:36:31.326+0000 7f37797fa640 10 monclient(hunting): got monmap 1 from mon.c (according to old e1) 2024-06-03T08:36:31.330 INFO:teuthology.orchestra.run.smithi088.stderr:2024-06-03T08:36:31.326+0000 7f37797fa640 10 monclient(hunting): dump: 2024-06-03T08:36:31.331 INFO:teuthology.orchestra.run.smithi088.stderr:epoch 1 2024-06-03T08:36:31.331 INFO:teuthology.orchestra.run.smithi088.stderr:fsid ec3c3fdd-3ac6-4d7d-9412-683130ed8b04 2024-06-03T08:36:31.331 INFO:teuthology.orchestra.run.smithi088.stderr:last_changed 2024-06-03T08:34:31.885125+0000 2024-06-03T08:36:31.331 INFO:teuthology.orchestra.run.smithi088.stderr:created 2024-06-03T08:34:31.885125+0000 2024-06-03T08:36:31.331 INFO:teuthology.orchestra.run.smithi088.stderr:min_mon_release 18 (reef) 2024-06-03T08:36:31.331 INFO:teuthology.orchestra.run.smithi088.stderr:election_strategy: 3 2024-06-03T08:36:31.331 INFO:teuthology.orchestra.run.smithi088.stderr:0: v1:172.21.15.88:6789/0 mon.a 2024-06-03T08:36:31.331 INFO:teuthology.orchestra.run.smithi088.stderr:1: v1:172.21.15.192:6789/0 mon.b 2024-06-03T08:36:31.331 INFO:teuthology.orchestra.run.smithi088.stderr:2: v1:172.21.15.88:6790/0 mon.c 2024-06-03T08:36:31.331 INFO:teuthology.orchestra.run.smithi088.stderr: 2024-06-03T08:36:31.331 INFO:teuthology.orchestra.run.smithi088.stderr:2024-06-03T08:36:31.328+0000 7f37797fa640 1 -- 172.21.15.88:0/3701686329 <== mon.1 v1:172.21.15.192:6789/0 4 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 580+0+0 (unknown 2360033557 0 0) 0x7f377c13f380 con 0x7f377c1450f0 2024-06-03T08:36:31.331 INFO:teuthology.orchestra.run.smithi088.stderr:2024-06-03T08:36:31.328+0000 7f37797fa640 1 -- 172.21.15.88:0/3701686329 >> v1:172.21.15.88:6790/0 conn(0x7f377c0a8960 legacy=0x7f377c142000 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-06-03T08:36:31.331 INFO:teuthology.orchestra.run.smithi088.stderr:2024-06-03T08:36:31.328+0000 7f37797fa640 1 -- 172.21.15.88:0/3701686329 >> v1:172.21.15.88:6789/0 conn(0x7f377c13eec0 legacy=0x7f377c1da2f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-06-03T08:36:31.331 INFO:teuthology.orchestra.run.smithi088.stderr:2024-06-03T08:36:31.328+0000 7f37797fa640 10 monclient: _finish_hunting 0 2024-06-03T08:36:31.331 INFO:teuthology.orchestra.run.smithi088.stderr:2024-06-03T08:36:31.328+0000 7f37797fa640 1 monclient: found mon.b 2024-06-03T08:36:31.332 INFO:teuthology.orchestra.run.smithi088.stderr:2024-06-03T08:36:31.328+0000 7f37797fa640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-06-03T08:36:31.332 INFO:teuthology.orchestra.run.smithi088.stderr:2024-06-03T08:36:31.328+0000 7f37797fa640 10 monclient: _send_mon_message to mon.b at v1:172.21.15.192:6789/0 2024-06-03T08:36:31.332 INFO:teuthology.orchestra.run.smithi088.stderr:2024-06-03T08:36:31.328+0000 7f37797fa640 1 -- 172.21.15.88:0/3701686329 --> v1:172.21.15.192:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f377c13f510 con 0x7f377c1450f0 2024-06-03T08:36:31.332 INFO:teuthology.orchestra.run.smithi088.stderr:2024-06-03T08:36:31.328+0000 7f37797fa640 10 monclient: _finish_auth 0 2024-06-03T08:36:31.332 INFO:teuthology.orchestra.run.smithi088.stderr:2024-06-03T08:36:31.328+0000 7f37797fa640 10 monclient: _check_auth_tickets 2024-06-03T08:36:31.332 INFO:teuthology.orchestra.run.smithi088.stderr:2024-06-03T08:36:31.328+0000 7f37797fa640 20 monclient: _check_auth_rotating not needed by client.admin 2024-06-03T08:36:31.332 INFO:teuthology.orchestra.run.smithi088.stderr:2024-06-03T08:36:31.328+0000 7f3783099640 5 monclient: authenticate success, global_id 4291 2024-06-03T08:36:31.332 INFO:teuthology.orchestra.run.smithi088.stderr:2024-06-03T08:36:31.328+0000 7f3783099640 10 monclient: _renew_subs 2024-06-03T08:36:31.332 INFO:teuthology.orchestra.run.smithi088.stderr:2024-06-03T08:36:31.328+0000 7f3783099640 10 monclient: _send_mon_message to mon.b at v1:172.21.15.192:6789/0 2024-06-03T08:36:31.332 INFO:teuthology.orchestra.run.smithi088.stderr:2024-06-03T08:36:31.328+0000 7f3783099640 1 -- 172.21.15.88:0/3701686329 --> v1:172.21.15.192:6789/0 -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f377c1daa00 con 0x7f377c1450f0 2024-06-03T08:36:31.332 INFO:teuthology.orchestra.run.smithi088.stderr:2024-06-03T08:36:31.328+0000 7f3783099640 10 client.4291.objecter _maybe_request_map subscribing (onetime) to next osd map 2024-06-03T08:36:31.332 INFO:teuthology.orchestra.run.smithi088.stderr:2024-06-03T08:36:31.328+0000 7f3783099640 10 monclient: _renew_subs 2024-06-03T08:36:31.332 INFO:teuthology.orchestra.run.smithi088.stderr:2024-06-03T08:36:31.328+0000 7f3783099640 10 monclient: _send_mon_message to mon.b at v1:172.21.15.192:6789/0 2024-06-03T08:36:31.332 INFO:teuthology.orchestra.run.smithi088.stderr:2024-06-03T08:36:31.328+0000 7f3783099640 1 -- 172.21.15.88:0/3701686329 --> v1:172.21.15.192:6789/0 -- mon_subscribe({osdmap=0}) v3 -- 0x7f377c0ab240 con 0x7f377c1450f0 2024-06-03T08:36:31.332 INFO:teuthology.orchestra.run.smithi088.stderr:2024-06-03T08:36:31.328+0000 7f3783099640 1 librados: init done 2024-06-03T08:36:31.333 INFO:teuthology.orchestra.run.smithi088.stderr:2024-06-03T08:36:31.329+0000 7f3783099640 10 start_mon_command cmd=[{"prefix": "get_command_descriptions"}] 2024-06-03T08:36:31.333 INFO:teuthology.orchestra.run.smithi088.stderr:2024-06-03T08:36:31.329+0000 7f3783099640 10 monclient: _send_command 1 [{"prefix": "get_command_descriptions"}] 2024-06-03T08:36:31.333 INFO:teuthology.orchestra.run.smithi088.stderr:2024-06-03T08:36:31.329+0000 7f3783099640 10 monclient: _send_mon_message to mon.b at v1:172.21.15.192:6789/0 2024-06-03T08:36:31.333 INFO:teuthology.orchestra.run.smithi088.stderr:2024-06-03T08:36:31.329+0000 7f3783099640 1 -- 172.21.15.88:0/3701686329 --> v1:172.21.15.192:6789/0 -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f377c0a9e60 con 0x7f377c1450f0 2024-06-03T08:36:31.569 INFO:teuthology.orchestra.run.smithi088.stderr:2024-06-03T08:36:31.568+0000 7f37797fa640 1 -- 172.21.15.88:0/3701686329 <== mon.1 v1:172.21.15.192:6789/0 5 ==== config(0 keys) v1 ==== 4+0+0 (unknown 0 0 0) 0x7f37740047e0 con 0x7f377c1450f0 2024-06-03T08:36:31.569 INFO:teuthology.orchestra.run.smithi088.stderr:2024-06-03T08:36:31.568+0000 7f37797fa640 10 monclient: handle_config config(0 keys) v1 2024-06-03T08:36:31.569 INFO:teuthology.orchestra.run.smithi088.stderr:2024-06-03T08:36:31.568+0000 7f37797fa640 1 -- 172.21.15.88:0/3701686329 <== mon.1 v1:172.21.15.192:6789/0 6 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 2234374065 0 0) 0x7f3774004af0 con 0x7f377c1450f0 2024-06-03T08:36:31.569 INFO:teuthology.orchestra.run.smithi088.stderr:2024-06-03T08:36:31.568+0000 7f37797fa640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-06-03T08:36:31.569 INFO:teuthology.orchestra.run.smithi088.stderr:2024-06-03T08:36:31.568+0000 7f37797fa640 10 monclient: got monmap 1 from mon.b (according to old e1) 2024-06-03T08:36:31.569 INFO:teuthology.orchestra.run.smithi088.stderr:2024-06-03T08:36:31.568+0000 7f37797fa640 10 monclient: dump: 2024-06-03T08:36:31.569 INFO:teuthology.orchestra.run.smithi088.stderr:epoch 1 2024-06-03T08:36:31.569 INFO:teuthology.orchestra.run.smithi088.stderr:fsid ec3c3fdd-3ac6-4d7d-9412-683130ed8b04 2024-06-03T08:36:31.570 INFO:teuthology.orchestra.run.smithi088.stderr:last_changed 2024-06-03T08:34:31.885125+0000 2024-06-03T08:36:31.570 INFO:teuthology.orchestra.run.smithi088.stderr:created 2024-06-03T08:34:31.885125+0000 2024-06-03T08:36:31.570 INFO:teuthology.orchestra.run.smithi088.stderr:min_mon_release 18 (reef) 2024-06-03T08:36:31.570 INFO:teuthology.orchestra.run.smithi088.stderr:election_strategy: 3 2024-06-03T08:36:31.570 INFO:teuthology.orchestra.run.smithi088.stderr:0: v1:172.21.15.88:6789/0 mon.a 2024-06-03T08:36:31.570 INFO:teuthology.orchestra.run.smithi088.stderr:1: v1:172.21.15.192:6789/0 mon.b 2024-06-03T08:36:31.570 INFO:teuthology.orchestra.run.smithi088.stderr:2: v1:172.21.15.88:6790/0 mon.c 2024-06-03T08:36:31.570 INFO:teuthology.orchestra.run.smithi088.stderr: 2024-06-03T08:36:31.570 INFO:teuthology.orchestra.run.smithi088.stderr:2024-06-03T08:36:31.568+0000 7f377affd640 4 set_mon_vals no callback set 2024-06-03T08:36:31.571 INFO:teuthology.orchestra.run.smithi088.stderr:2024-06-03T08:36:31.569+0000 7f37797fa640 1 -- 172.21.15.88:0/3701686329 <== mon.1 v1:172.21.15.192:6789/0 7 ==== mgrmap(e 5) v1 ==== 89110+0+0 (unknown 873533973 0 0) 0x7f377401a870 con 0x7f377c1450f0 2024-06-03T08:36:31.571 INFO:teuthology.orchestra.run.smithi088.stderr:2024-06-03T08:36:31.570+0000 7f3780e35640 10 monclient: get_auth_request con 0x7f375c06d450 auth_method 0 2024-06-03T08:36:31.571 INFO:teuthology.orchestra.run.smithi088.stderr:2024-06-03T08:36:31.570+0000 7f37797fa640 10 client.4291.objecter ms_handle_connect 0x7f375c06d450 2024-06-03T08:36:31.571 INFO:teuthology.orchestra.run.smithi088.stderr:2024-06-03T08:36:31.571+0000 7f37797fa640 1 -- 172.21.15.88:0/3701686329 <== mon.1 v1:172.21.15.192:6789/0 8 ==== osd_map(25..25 src has 1..25) v4 ==== 4090+0+0 (unknown 2269083415 0 0) 0x7f3774054060 con 0x7f377c1450f0 2024-06-03T08:36:31.571 INFO:teuthology.orchestra.run.smithi088.stderr:2024-06-03T08:36:31.571+0000 7f37797fa640 10 client.4291.objecter ms_dispatch 0x7f377c000b90 osd_map(25..25 src has 1..25) v4 2024-06-03T08:36:31.572 INFO:teuthology.orchestra.run.smithi088.stderr:2024-06-03T08:36:31.571+0000 7f37797fa640 3 client.4291.objecter handle_osd_map got epochs [25,25] > 0 2024-06-03T08:36:31.572 INFO:teuthology.orchestra.run.smithi088.stderr:2024-06-03T08:36:31.571+0000 7f37797fa640 3 client.4291.objecter handle_osd_map decoding full epoch 25 2024-06-03T08:36:31.572 INFO:teuthology.orchestra.run.smithi088.stderr:2024-06-03T08:36:31.571+0000 7f37797fa640 20 client.4291.objecter dump_active .. 0 homeless 2024-06-03T08:36:31.573 INFO:teuthology.orchestra.run.smithi088.stderr:2024-06-03T08:36:31.573+0000 7f37797fa640 1 -- 172.21.15.88:0/3701686329 <== mon.1 v1:172.21.15.192:6789/0 9 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+182498 (unknown 1092875540 0 3016760700) 0x7f377404fa40 con 0x7f377c1450f0 2024-06-03T08:36:31.574 INFO:teuthology.orchestra.run.smithi088.stderr:2024-06-03T08:36:31.573+0000 7f37797fa640 10 monclient: handle_mon_command_ack 1 [{"prefix": "get_command_descriptions"}] 2024-06-03T08:36:31.574 INFO:teuthology.orchestra.run.smithi088.stderr:2024-06-03T08:36:31.573+0000 7f37797fa640 10 monclient: _finish_command 1 = system:0 2024-06-03T08:36:31.744 INFO:teuthology.orchestra.run.smithi088.stderr:2024-06-03T08:36:31.743+0000 7f3783099640 10 start_mon_command cmd=[{"prefix": "osd dump", "format": "json"}] 2024-06-03T08:36:31.744 INFO:teuthology.orchestra.run.smithi088.stderr:2024-06-03T08:36:31.743+0000 7f3783099640 10 monclient: _send_command 2 [{"prefix": "osd dump", "format": "json"}] 2024-06-03T08:36:31.744 INFO:teuthology.orchestra.run.smithi088.stderr:2024-06-03T08:36:31.743+0000 7f3783099640 10 monclient: _send_mon_message to mon.b at v1:172.21.15.192:6789/0 2024-06-03T08:36:31.744 INFO:teuthology.orchestra.run.smithi088.stderr:2024-06-03T08:36:31.743+0000 7f3783099640 1 -- 172.21.15.88:0/3701686329 --> v1:172.21.15.192:6789/0 -- mon_command({"prefix": "osd dump", "format": "json"} v 0) v1 -- 0x7f377c0ab4f0 con 0x7f377c1450f0 2024-06-03T08:36:31.767 INFO:teuthology.orchestra.run.smithi088.stderr:2024-06-03T08:36:31.766+0000 7f37797fa640 1 -- 172.21.15.88:0/3701686329 <== mon.1 v1:172.21.15.192:6789/0 10 ==== mon_command_ack([{"prefix": "osd dump", "format": "json"}]=0 v25) v1 ==== 74+0+12431 (unknown 780366164 0 1565932004) 0x7f3774051b40 con 0x7f377c1450f0 2024-06-03T08:36:31.767 INFO:teuthology.orchestra.run.smithi088.stderr:2024-06-03T08:36:31.766+0000 7f37797fa640 10 monclient: handle_mon_command_ack 2 [{"prefix": "osd dump", "format": "json"}] 2024-06-03T08:36:31.767 INFO:teuthology.orchestra.run.smithi088.stderr:2024-06-03T08:36:31.766+0000 7f37797fa640 10 monclient: _finish_command 2 = system:0 2024-06-03T08:36:31.767 INFO:teuthology.orchestra.run.smithi088.stdout: 2024-06-03T08:36:31.767 INFO:teuthology.orchestra.run.smithi088.stdout:{"epoch":25,"fsid":"ec3c3fdd-3ac6-4d7d-9412-683130ed8b04","created":"2024-06-03T08:35:26.799638+0000","modified":"2024-06-03T08:36:30.620322+0000","last_up_change":"2024-06-03T08:36:23.712633+0000","last_in_change":"2024-06-03T08:35:32.183068+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":5,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":2,"max_osd":8,"require_min_compat_client":"luminous","min_compat_client":"jewel","require_osd_release":"reef","allow_crimson":false,"pools":[{"pool":1,"pool_name":".mgr","create_time":"2024-06-03T08:36:15.192388+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":1,"pg_placement_num":1,"pg_placement_num_target":1,"pg_num_target":1,"pg_num_pending":1,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"19","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_max":32,"pg_num_min":1},"application_metadata":{"mgr":{}},"read_balance":{"score_acting":8,"score_stable":8,"optimal_score":0.25,"raw_score_acting":2,"raw_score_stable":2,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":2,"pool_name":"rbd","create_time":"2024-06-03T08:36:28.156336+0000","flags":32769,"flags_names":"hashpspool,creating","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":8,"pg_placement_num":8,"pg_placement_num_target":8,"pg_num_target":8,"pg_num_pending":8,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"24","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{},"application_metadata":{"rbd":{}},"read_balance":{"score_acting":4.9800000190734863,"score_stable":4.9800000190734863,"optimal_score":0.87999999523162842,"raw_score_acting":4.380000114440918,"raw_score_stable":4.380000114440918,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}}],"osds":[{"osd":0,"uuid":"b25ab3bb-3587-45d0-991a-6d41bb529b37","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":14,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.88:6808","nonce":35066}]},"cluster_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.88:6809","nonce":35066}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.88:6811","nonce":35066}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.88:6810","nonce":35066}]},"public_addr":"172.21.15.88:6808/35066","cluster_addr":"172.21.15.88:6809/35066","heartbeat_back_addr":"172.21.15.88:6811/35066","heartbeat_front_addr":"172.21.15.88:6810/35066","state":["exists","up"]},{"osd":1,"uuid":"00d160e7-9773-41d8-8d17-21e70620a0fd","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":14,"up_thru":20,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.88:6800","nonce":35067}]},"cluster_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.88:6801","nonce":35067}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.88:6804","nonce":35067}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.88:6803","nonce":35067}]},"public_addr":"172.21.15.88:6800/35067","cluster_addr":"172.21.15.88:6801/35067","heartbeat_back_addr":"172.21.15.88:6804/35067","heartbeat_front_addr":"172.21.15.88:6803/35067","state":["exists","up"]},{"osd":2,"uuid":"addfb2a7-2bde-4cc3-98cf-17fd2bdfa3a9","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":20,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.88:6802","nonce":35065}]},"cluster_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.88:6805","nonce":35065}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.88:6807","nonce":35065}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.88:6806","nonce":35065}]},"public_addr":"172.21.15.88:6802/35065","cluster_addr":"172.21.15.88:6805/35065","heartbeat_back_addr":"172.21.15.88:6807/35065","heartbeat_front_addr":"172.21.15.88:6806/35065","state":["exists","up"]},{"osd":3,"uuid":"6c5770b0-c93c-4101-8c53-bd761b8abcde","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":14,"up_thru":23,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.88:6812","nonce":35068}]},"cluster_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.88:6813","nonce":35068}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.88:6815","nonce":35068}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.88:6814","nonce":35068}]},"public_addr":"172.21.15.88:6812/35068","cluster_addr":"172.21.15.88:6813/35068","heartbeat_back_addr":"172.21.15.88:6815/35068","heartbeat_front_addr":"172.21.15.88:6814/35068","state":["exists","up"]},{"osd":4,"uuid":"f001b3cb-2081-428b-bac3-b3df3c19b978","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":20,"up_thru":23,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.192:6809","nonce":34785}]},"cluster_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.192:6810","nonce":34785}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.192:6812","nonce":34785}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.192:6811","nonce":34785}]},"public_addr":"172.21.15.192:6809/34785","cluster_addr":"172.21.15.192:6810/34785","heartbeat_back_addr":"172.21.15.192:6812/34785","heartbeat_front_addr":"172.21.15.192:6811/34785","state":["exists","up"]},{"osd":5,"uuid":"3cc263e9-e9bf-45e0-a3b7-83a76469eb70","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":20,"up_thru":23,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.192:6805","nonce":34786}]},"cluster_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.192:6806","nonce":34786}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.192:6808","nonce":34786}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.192:6807","nonce":34786}]},"public_addr":"172.21.15.192:6805/34786","cluster_addr":"172.21.15.192:6806/34786","heartbeat_back_addr":"172.21.15.192:6808/34786","heartbeat_front_addr":"172.21.15.192:6807/34786","state":["exists","up"]},{"osd":6,"uuid":"82c83316-809f-4901-a5ed-34b6a075b393","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":21,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.192:6813","nonce":34788}]},"cluster_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.192:6814","nonce":34788}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.192:6816","nonce":34788}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.192:6815","nonce":34788}]},"public_addr":"172.21.15.192:6813/34788","cluster_addr":"172.21.15.192:6814/34788","heartbeat_back_addr":"172.21.15.192:6816/34788","heartbeat_front_addr":"172.21.15.192:6815/34788","state":["exists","up"]},{"osd":7,"uuid":"aafe0716-1102-4eae-aac6-0f81bb154e7d","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":20,"up_thru":23,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.192:6801","nonce":34787}]},"cluster_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.192:6802","nonce":34787}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.192:6804","nonce":34787}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.192:6803","nonce":34787}]},"public_addr":"172.21.15.192:6801/34787","cluster_addr":"172.21.15.192:6802/34787","heartbeat_back_addr":"172.21.15.192:6804/34787","heartbeat_front_addr":"172.21.15.192:6803/34787","state":["exists","up"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2024-06-03T08:36:10.027533+0000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2024-06-03T08:36:09.853335+0000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2024-06-03T08:36:10.789609+0000","dead_epoch":0},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2024-06-03T08:36:09.824468+0000","dead_epoch":0},{"osd":4,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2024-06-03T08:36:10.117232+0000","dead_epoch":0},{"osd":5,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2024-06-03T08:36:10.469993+0000","dead_epoch":0},{"osd":6,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2024-06-03T08:36:10.789874+0000","dead_epoch":0},{"osd":7,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2024-06-03T08:36:09.541793+0000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[],"pg_upmap_primaries":[],"pg_temp":[],"primary_temp":[],"blocklist":{},"range_blocklist":{},"erasure_code_profiles":{"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","technique":"reed_sol_van"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2024-06-03T08:36:31.769 INFO:teuthology.orchestra.run.smithi088.stderr:2024-06-03T08:36:31.768+0000 7f3783099640 10 librados: watch_flush enter 2024-06-03T08:36:31.769 INFO:teuthology.orchestra.run.smithi088.stderr:2024-06-03T08:36:31.768+0000 7f3783099640 10 librados: watch_flush exit 2024-06-03T08:36:31.769 INFO:teuthology.orchestra.run.smithi088.stderr:2024-06-03T08:36:31.768+0000 7f3783099640 20 client.4291.objecter shutdown clearing up homeless session... 2024-06-03T08:36:31.769 INFO:teuthology.orchestra.run.smithi088.stderr:2024-06-03T08:36:31.768+0000 7f3783099640 10 client.4291.objecter successfully canceled tick 2024-06-03T08:36:31.769 INFO:teuthology.orchestra.run.smithi088.stderr:2024-06-03T08:36:31.768+0000 7f3783099640 1 -- 172.21.15.88:0/3701686329 >> v1:172.21.15.192:6800/34229 conn(0x7f375c06d450 legacy=0x7f375c06f910 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-06-03T08:36:31.769 INFO:teuthology.orchestra.run.smithi088.stderr:2024-06-03T08:36:31.768+0000 7f3783099640 10 monclient: shutdown 2024-06-03T08:36:31.770 INFO:teuthology.orchestra.run.smithi088.stderr:2024-06-03T08:36:31.768+0000 7f3783099640 20 monclient: shutdown discarding 0 pending message(s) 2024-06-03T08:36:31.770 INFO:teuthology.orchestra.run.smithi088.stderr:2024-06-03T08:36:31.768+0000 7f3783099640 1 -- 172.21.15.88:0/3701686329 >> v1:172.21.15.192:6789/0 conn(0x7f377c1450f0 legacy=0x7f377c1e1240 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-06-03T08:36:31.770 INFO:teuthology.orchestra.run.smithi088.stderr:2024-06-03T08:36:31.768+0000 7f3783099640 1 -- 172.21.15.88:0/3701686329 shutdown_connections 2024-06-03T08:36:31.770 INFO:teuthology.orchestra.run.smithi088.stderr:2024-06-03T08:36:31.768+0000 7f3783099640 1 -- 172.21.15.88:0/3701686329 >> 172.21.15.88:0/3701686329 conn(0x7f377c0b70d0 msgr2=0x7f377c13e580 unknown :-1 s=STATE_NONE l=0).mark_down 2024-06-03T08:36:31.770 INFO:teuthology.orchestra.run.smithi088.stderr:2024-06-03T08:36:31.768+0000 7f3783099640 1 -- 172.21.15.88:0/3701686329 shutdown_connections 2024-06-03T08:36:31.770 INFO:teuthology.orchestra.run.smithi088.stderr:2024-06-03T08:36:31.768+0000 7f3783099640 1 -- 172.21.15.88:0/3701686329 wait complete. 2024-06-03T08:36:31.770 INFO:teuthology.orchestra.run.smithi088.stderr:2024-06-03T08:36:31.768+0000 7f3783099640 1 librados: shutdown 2024-06-03T08:36:31.781 DEBUG:teuthology.orchestra.run.smithi088:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph tell osd.0 flush_pg_stats 2024-06-03T08:36:31.781 DEBUG:teuthology.orchestra.run.smithi088:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph tell osd.1 flush_pg_stats 2024-06-03T08:36:31.782 DEBUG:teuthology.orchestra.run.smithi088:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph tell osd.2 flush_pg_stats 2024-06-03T08:36:31.782 DEBUG:teuthology.orchestra.run.smithi088:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph tell osd.3 flush_pg_stats 2024-06-03T08:36:31.782 DEBUG:teuthology.orchestra.run.smithi088:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph tell osd.4 flush_pg_stats 2024-06-03T08:36:31.783 DEBUG:teuthology.orchestra.run.smithi088:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph tell osd.5 flush_pg_stats 2024-06-03T08:36:31.783 DEBUG:teuthology.orchestra.run.smithi088:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph tell osd.6 flush_pg_stats 2024-06-03T08:36:31.784 DEBUG:teuthology.orchestra.run.smithi088:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph tell osd.7 flush_pg_stats 2024-06-03T08:36:31.964 INFO:teuthology.orchestra.run.smithi088.stderr:2024-06-03T08:36:31.962+0000 7f199da81640 10 monclient: get_monmap_and_config 2024-06-03T08:36:31.964 INFO:teuthology.orchestra.run.smithi088.stderr:2024-06-03T08:36:31.962+0000 7f199da81640 10 monclient: build_initial_monmap 2024-06-03T08:36:31.964 INFO:teuthology.orchestra.run.smithi088.stderr:2024-06-03T08:36:31.962+0000 7f199da81640 10 monclient: monmap: 2024-06-03T08:36:31.964 INFO:teuthology.orchestra.run.smithi088.stderr:epoch 0 2024-06-03T08:36:31.964 INFO:teuthology.orchestra.run.smithi088.stderr:fsid ec3c3fdd-3ac6-4d7d-9412-683130ed8b04 2024-06-03T08:36:31.964 INFO:teuthology.orchestra.run.smithi088.stderr:last_changed 2024-06-03T08:36:31.963481+0000 2024-06-03T08:36:31.965 INFO:teuthology.orchestra.run.smithi088.stderr:created 2024-06-03T08:36:31.963481+0000 2024-06-03T08:36:31.965 INFO:teuthology.orchestra.run.smithi088.stderr:min_mon_release 0 (unknown) 2024-06-03T08:36:31.965 INFO:teuthology.orchestra.run.smithi088.stderr:election_strategy: 3 2024-06-03T08:36:31.965 INFO:teuthology.orchestra.run.smithi088.stderr:0: v1:172.21.15.88:6789/0 mon.noname-a 2024-06-03T08:36:31.965 INFO:teuthology.orchestra.run.smithi088.stderr:1: v1:172.21.15.192:6789/0 mon.noname-c 2024-06-03T08:36:31.965 INFO:teuthology.orchestra.run.smithi088.stderr:2: v1:172.21.15.88:6790/0 mon.noname-b 2024-06-03T08:36:31.965 INFO:teuthology.orchestra.run.smithi088.stderr: 2024-06-03T08:36:31.965 INFO:teuthology.orchestra.run.smithi088.stderr:2024-06-03T08:36:31.964+0000 7f199da81640 1 Processor -- start 2024-06-03T08:36:31.966 INFO:teuthology.orchestra.run.smithi088.stderr:2024-06-03T08:36:31.964+0000 7f199da81640 1 -- start start 2024-06-03T08:36:31.967 INFO:teuthology.orchestra.run.smithi088.stderr:2024-06-03T08:36:31.964+0000 7f199da81640 10 monclient: init 2024-06-03T08:36:31.967 INFO:teuthology.orchestra.run.smithi088.stderr:2024-06-03T08:36:31.965+0000 7f199da81640 10 monclient: _reopen_session rank -1 2024-06-03T08:36:31.967 INFO:teuthology.orchestra.run.smithi088.stderr:2024-06-03T08:36:31.965+0000 7f199da81640 10 monclient: _add_conns ranks=[2,0,1] 2024-06-03T08:36:31.967 INFO:teuthology.orchestra.run.smithi088.stderr:2024-06-03T08:36:31.965+0000 7f199da81640 10 monclient(hunting): picked mon.noname-b con 0x7f1990006410 addr v1:172.21.15.88:6790/0 2024-06-03T08:36:31.967 INFO:teuthology.orchestra.run.smithi088.stderr:2024-06-03T08:36:31.965+0000 7f199da81640 10 monclient(hunting): picked mon.noname-a con 0x7f199002a640 addr v1:172.21.15.88:6789/0 2024-06-03T08:36:31.967 INFO:teuthology.orchestra.run.smithi088.stderr:2024-06-03T08:36:31.965+0000 7f199da81640 10 monclient(hunting): picked mon.noname-c con 0x7f19900327d0 addr v1:172.21.15.192:6789/0 2024-06-03T08:36:31.967 INFO:teuthology.orchestra.run.smithi088.stderr:2024-06-03T08:36:31.965+0000 7f199da81640 1 -- --> v1:172.21.15.88:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f199002c820 con 0x7f199002a640 2024-06-03T08:36:31.967 INFO:teuthology.orchestra.run.smithi088.stderr:2024-06-03T08:36:31.965+0000 7f199da81640 1 -- --> v1:172.21.15.192:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f199002a120 con 0x7f19900327d0 2024-06-03T08:36:31.967 INFO:teuthology.orchestra.run.smithi088.stderr:2024-06-03T08:36:31.965+0000 7f199da81640 1 -- --> v1:172.21.15.88:6790/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f19900c75b0 con 0x7f1990006410 2024-06-03T08:36:31.967 INFO:teuthology.orchestra.run.smithi088.stderr:2024-06-03T08:36:31.965+0000 7f199da81640 10 monclient(hunting): _renew_subs 2024-06-03T08:36:31.967 INFO:teuthology.orchestra.run.smithi088.stderr:2024-06-03T08:36:31.965+0000 7f199da81640 10 monclient(hunting): authenticate will time out at 1247.775383s 2024-06-03T08:36:31.968 INFO:teuthology.orchestra.run.smithi088.stderr:2024-06-03T08:36:31.967+0000 7f1996ffd640 1 --1- >> v1:172.21.15.88:6789/0 conn(0x7f199002a640 0x7f19900320c0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.88:6789/0 says I am v1:172.21.15.88:60252/0 (socket says 172.21.15.88:60252) 2024-06-03T08:36:31.968 INFO:teuthology.orchestra.run.smithi088.stderr:2024-06-03T08:36:31.967+0000 7f1996ffd640 1 -- 172.21.15.88:0/402424495 learned_addr learned my addr 172.21.15.88:0/402424495 (peer_addr_for_me v1:172.21.15.88:0/0) 2024-06-03T08:36:31.972 INFO:teuthology.orchestra.run.smithi088.stderr:2024-06-03T08:36:31.972+0000 7f19967fc640 1 -- 172.21.15.88:0/402424495 <== mon.2 v1:172.21.15.88:6790/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 3513741398 0 0) 0x7f19900c75b0 con 0x7f1990006410 2024-06-03T08:36:31.972 INFO:teuthology.orchestra.run.smithi088.stderr:2024-06-03T08:36:31.972+0000 7f19967fc640 10 monclient(hunting): _negotiate 2024-06-03T08:36:31.972 INFO:teuthology.orchestra.run.smithi088.stderr:2024-06-03T08:36:31.972+0000 7f19967fc640 10 monclient(hunting): _init_auth method 2 2024-06-03T08:36:31.972 INFO:teuthology.orchestra.run.smithi088.stderr:2024-06-03T08:36:31.972+0000 7f19967fc640 10 monclient(hunting): _init_auth creating new auth 2024-06-03T08:36:31.972 INFO:teuthology.orchestra.run.smithi088.stderr:2024-06-03T08:36:31.972+0000 7f19967fc640 10 monclient(hunting): my global_id is 4310 2024-06-03T08:36:31.972 INFO:teuthology.orchestra.run.smithi088.stderr:2024-06-03T08:36:31.972+0000 7f19967fc640 1 -- 172.21.15.88:0/402424495 --> v1:172.21.15.88:6790/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f1988003520 con 0x7f1990006410 2024-06-03T08:36:31.975 INFO:teuthology.orchestra.run.smithi088.stderr:2024-06-03T08:36:31.974+0000 7f19967fc640 1 -- 172.21.15.88:0/402424495 <== mon.0 v1:172.21.15.88:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 1169733636 0 0) 0x7f199002c820 con 0x7f199002a640 2024-06-03T08:36:31.975 INFO:teuthology.orchestra.run.smithi088.stderr:2024-06-03T08:36:31.974+0000 7f19967fc640 10 monclient(hunting): _negotiate 2024-06-03T08:36:31.975 INFO:teuthology.orchestra.run.smithi088.stderr:2024-06-03T08:36:31.974+0000 7f19967fc640 10 monclient(hunting): _init_auth method 2 2024-06-03T08:36:31.975 INFO:teuthology.orchestra.run.smithi088.stderr:2024-06-03T08:36:31.974+0000 7f19967fc640 10 monclient(hunting): _init_auth creating new auth 2024-06-03T08:36:31.975 INFO:teuthology.orchestra.run.smithi088.stderr:2024-06-03T08:36:31.974+0000 7f19967fc640 10 monclient(hunting): my global_id is 4290 2024-06-03T08:36:31.975 INFO:teuthology.orchestra.run.smithi088.stderr:2024-06-03T08:36:31.974+0000 7f19967fc640 1 -- 172.21.15.88:0/402424495 --> v1:172.21.15.88:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f19900c75b0 con 0x7f199002a640 2024-06-03T08:36:31.983 INFO:teuthology.orchestra.run.smithi088.stderr:2024-06-03T08:36:31.982+0000 7f19967fc640 1 -- 172.21.15.88:0/402424495 <== mon.1 v1:172.21.15.192:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 3380577181 0 0) 0x7f199002a120 con 0x7f19900327d0 2024-06-03T08:36:31.983 INFO:teuthology.orchestra.run.smithi088.stderr:2024-06-03T08:36:31.982+0000 7f19967fc640 10 monclient(hunting): _negotiate 2024-06-03T08:36:31.983 INFO:teuthology.orchestra.run.smithi088.stderr:2024-06-03T08:36:31.982+0000 7f19967fc640 10 monclient(hunting): _init_auth method 2 2024-06-03T08:36:31.983 INFO:teuthology.orchestra.run.smithi088.stderr:2024-06-03T08:36:31.982+0000 7f19967fc640 10 monclient(hunting): _init_auth creating new auth 2024-06-03T08:36:31.983 INFO:teuthology.orchestra.run.smithi088.stderr:2024-06-03T08:36:31.982+0000 7f19967fc640 10 monclient(hunting): my global_id is 4294 2024-06-03T08:36:31.983 INFO:teuthology.orchestra.run.smithi088.stderr:2024-06-03T08:36:31.982+0000 7f19967fc640 1 -- 172.21.15.88:0/402424495 --> v1:172.21.15.192:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f199002c820 con 0x7f19900327d0 2024-06-03T08:36:31.983 INFO:teuthology.orchestra.run.smithi088.stderr:2024-06-03T08:36:31.982+0000 7f19967fc640 1 -- 172.21.15.88:0/402424495 <== mon.2 v1:172.21.15.88:6790/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 214+0+0 (unknown 1839342984 0 0) 0x7f1988003520 con 0x7f1990006410 2024-06-03T08:36:31.984 INFO:teuthology.orchestra.run.smithi088.stderr:2024-06-03T08:36:31.982+0000 7f19967fc640 1 -- 172.21.15.88:0/402424495 >> v1:172.21.15.192:6789/0 conn(0x7f19900327d0 legacy=0x7f19900d1550 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-06-03T08:36:31.984 INFO:teuthology.orchestra.run.smithi088.stderr:2024-06-03T08:36:31.982+0000 7f19967fc640 1 -- 172.21.15.88:0/402424495 >> v1:172.21.15.88:6789/0 conn(0x7f199002a640 legacy=0x7f19900320c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-06-03T08:36:31.984 INFO:teuthology.orchestra.run.smithi088.stderr:2024-06-03T08:36:31.982+0000 7f19967fc640 10 monclient: _finish_hunting 0 2024-06-03T08:36:31.984 INFO:teuthology.orchestra.run.smithi088.stderr:2024-06-03T08:36:31.982+0000 7f19967fc640 1 monclient: found mon.noname-b 2024-06-03T08:36:31.984 INFO:teuthology.orchestra.run.smithi088.stderr:2024-06-03T08:36:31.982+0000 7f19967fc640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-06-03T08:36:31.984 INFO:teuthology.orchestra.run.smithi088.stderr:2024-06-03T08:36:31.982+0000 7f19967fc640 10 monclient: _send_mon_message to mon.noname-b at v1:172.21.15.88:6790/0 2024-06-03T08:36:31.984 INFO:teuthology.orchestra.run.smithi088.stderr:2024-06-03T08:36:31.982+0000 7f19967fc640 1 -- 172.21.15.88:0/402424495 --> v1:172.21.15.88:6790/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f1990033110 con 0x7f1990006410 2024-06-03T08:36:31.984 INFO:teuthology.orchestra.run.smithi088.stderr:2024-06-03T08:36:31.983+0000 7f19967fc640 10 monclient: _finish_auth 0 2024-06-03T08:36:31.985 INFO:teuthology.orchestra.run.smithi088.stderr:2024-06-03T08:36:31.983+0000 7f19967fc640 10 monclient: _check_auth_tickets 2024-06-03T08:36:31.985 INFO:teuthology.orchestra.run.smithi088.stderr:2024-06-03T08:36:31.983+0000 7f19967fc640 20 monclient: _check_auth_rotating not needed by client.admin 2024-06-03T08:36:31.985 INFO:teuthology.orchestra.run.smithi088.stderr:2024-06-03T08:36:31.983+0000 7f199da81640 5 monclient: authenticate success, global_id 4310 2024-06-03T08:36:31.985 INFO:teuthology.orchestra.run.smithi088.stderr:2024-06-03T08:36:31.983+0000 7f199da81640 20 monclient: get_monmap_and_config waiting for monmap|config 2024-06-03T08:36:31.985 INFO:teuthology.orchestra.run.smithi088.stderr:2024-06-03T08:36:31.983+0000 7f19967fc640 1 -- 172.21.15.88:0/402424495 <== mon.2 v1:172.21.15.88:6790/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 2234374065 0 0) 0x7f198c001590 con 0x7f1990006410 2024-06-03T08:36:31.986 INFO:teuthology.orchestra.run.smithi088.stderr:2024-06-03T08:36:31.984+0000 7f19967fc640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-06-03T08:36:31.986 INFO:teuthology.orchestra.run.smithi088.stderr:2024-06-03T08:36:31.984+0000 7f19967fc640 10 monclient: got monmap 1 from mon.noname-b (according to old e1) 2024-06-03T08:36:31.986 INFO:teuthology.orchestra.run.smithi088.stderr:2024-06-03T08:36:31.984+0000 7f19967fc640 10 monclient: dump: 2024-06-03T08:36:31.986 INFO:teuthology.orchestra.run.smithi088.stderr:epoch 1 2024-06-03T08:36:31.986 INFO:teuthology.orchestra.run.smithi088.stderr:fsid ec3c3fdd-3ac6-4d7d-9412-683130ed8b04 2024-06-03T08:36:31.986 INFO:teuthology.orchestra.run.smithi088.stderr:last_changed 2024-06-03T08:34:31.885125+0000 2024-06-03T08:36:31.986 INFO:teuthology.orchestra.run.smithi088.stderr:created 2024-06-03T08:34:31.885125+0000 2024-06-03T08:36:31.986 INFO:teuthology.orchestra.run.smithi088.stderr:min_mon_release 18 (reef) 2024-06-03T08:36:31.986 INFO:teuthology.orchestra.run.smithi088.stderr:election_strategy: 3 2024-06-03T08:36:31.986 INFO:teuthology.orchestra.run.smithi088.stderr:0: v1:172.21.15.88:6789/0 mon.a 2024-06-03T08:36:31.986 INFO:teuthology.orchestra.run.smithi088.stderr:1: v1:172.21.15.192:6789/0 mon.b 2024-06-03T08:36:31.986 INFO:teuthology.orchestra.run.smithi088.stderr:2: v1:172.21.15.88:6790/0 mon.c 2024-06-03T08:36:31.987 INFO:teuthology.orchestra.run.smithi088.stderr: 2024-06-03T08:36:31.987 INFO:teuthology.orchestra.run.smithi088.stderr:2024-06-03T08:36:31.984+0000 7f199da81640 20 monclient: get_monmap_and_config waiting for monmap|config 2024-06-03T08:36:31.988 INFO:teuthology.orchestra.run.smithi088.stderr:2024-06-03T08:36:31.988+0000 7f19967fc640 1 -- 172.21.15.88:0/402424495 <== mon.2 v1:172.21.15.88:6790/0 4 ==== config(0 keys) v1 ==== 4+0+0 (unknown 0 0 0) 0x7f198c001820 con 0x7f1990006410 2024-06-03T08:36:31.989 INFO:teuthology.orchestra.run.smithi088.stderr:2024-06-03T08:36:31.988+0000 7f19967fc640 10 monclient: handle_config config(0 keys) v1 2024-06-03T08:36:31.989 INFO:teuthology.orchestra.run.smithi088.stderr:2024-06-03T08:36:31.987+0000 7ff49a002640 10 monclient: get_monmap_and_config 2024-06-03T08:36:31.989 INFO:teuthology.orchestra.run.smithi088.stderr:2024-06-03T08:36:31.987+0000 7ff49a002640 10 monclient: build_initial_monmap 2024-06-03T08:36:31.989 INFO:teuthology.orchestra.run.smithi088.stderr:2024-06-03T08:36:31.987+0000 7ff49a002640 10 monclient: monmap: 2024-06-03T08:36:31.989 INFO:teuthology.orchestra.run.smithi088.stderr:epoch 0 2024-06-03T08:36:31.989 INFO:teuthology.orchestra.run.smithi088.stderr:fsid ec3c3fdd-3ac6-4d7d-9412-683130ed8b04 2024-06-03T08:36:31.989 INFO:teuthology.orchestra.run.smithi088.stderr:last_changed 2024-06-03T08:36:31.988116+0000 2024-06-03T08:36:31.989 INFO:teuthology.orchestra.run.smithi088.stderr:created 2024-06-03T08:36:31.988116+0000 2024-06-03T08:36:31.990 INFO:teuthology.orchestra.run.smithi088.stderr:min_mon_release 0 (unknown) 2024-06-03T08:36:31.990 INFO:teuthology.orchestra.run.smithi088.stderr:election_strategy: 3 2024-06-03T08:36:31.990 INFO:teuthology.orchestra.run.smithi088.stderr:0: v1:172.21.15.88:6789/0 mon.noname-a 2024-06-03T08:36:31.990 INFO:teuthology.orchestra.run.smithi088.stderr:1: v1:172.21.15.192:6789/0 mon.noname-c 2024-06-03T08:36:31.990 INFO:teuthology.orchestra.run.smithi088.stderr:2: v1:172.21.15.88:6790/0 mon.noname-b 2024-06-03T08:36:31.990 INFO:teuthology.orchestra.run.smithi088.stderr: 2024-06-03T08:36:31.990 INFO:teuthology.orchestra.run.smithi088.stderr:2024-06-03T08:36:31.989+0000 7ff49a002640 1 Processor -- start 2024-06-03T08:36:31.990 INFO:teuthology.orchestra.run.smithi088.stderr:2024-06-03T08:36:31.989+0000 7f19967fc640 1 -- 172.21.15.88:0/402424495 <== mon.2 v1:172.21.15.88:6790/0 5 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 2234374065 0 0) 0x7f198c001bb0 con 0x7f1990006410 2024-06-03T08:36:31.990 INFO:teuthology.orchestra.run.smithi088.stderr:2024-06-03T08:36:31.989+0000 7f19967fc640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-06-03T08:36:31.991 INFO:teuthology.orchestra.run.smithi088.stderr:2024-06-03T08:36:31.989+0000 7f19967fc640 10 monclient: got monmap 1 from mon.c (according to old e1) 2024-06-03T08:36:31.991 INFO:teuthology.orchestra.run.smithi088.stderr:2024-06-03T08:36:31.989+0000 7f19967fc640 10 monclient: dump: 2024-06-03T08:36:31.991 INFO:teuthology.orchestra.run.smithi088.stderr:epoch 1 2024-06-03T08:36:31.991 INFO:teuthology.orchestra.run.smithi088.stderr:fsid ec3c3fdd-3ac6-4d7d-9412-683130ed8b04 2024-06-03T08:36:31.991 INFO:teuthology.orchestra.run.smithi088.stderr:last_changed 2024-06-03T08:34:31.885125+0000 2024-06-03T08:36:31.991 INFO:teuthology.orchestra.run.smithi088.stderr:created 2024-06-03T08:34:31.885125+0000 2024-06-03T08:36:31.991 INFO:teuthology.orchestra.run.smithi088.stderr:min_mon_release 18 (reef) 2024-06-03T08:36:31.991 INFO:teuthology.orchestra.run.smithi088.stderr:election_strategy: 3 2024-06-03T08:36:31.991 INFO:teuthology.orchestra.run.smithi088.stderr:0: v1:172.21.15.88:6789/0 mon.a 2024-06-03T08:36:31.991 INFO:teuthology.orchestra.run.smithi088.stderr:1: v1:172.21.15.192:6789/0 mon.b 2024-06-03T08:36:31.991 INFO:teuthology.orchestra.run.smithi088.stderr:2: v1:172.21.15.88:6790/0 mon.c 2024-06-03T08:36:31.991 INFO:teuthology.orchestra.run.smithi088.stderr: 2024-06-03T08:36:31.992 INFO:teuthology.orchestra.run.smithi088.stderr:2024-06-03T08:36:31.989+0000 7f199da81640 10 monclient: get_monmap_and_config success 2024-06-03T08:36:31.992 INFO:teuthology.orchestra.run.smithi088.stderr:2024-06-03T08:36:31.989+0000 7f199da81640 4 set_mon_vals no callback set 2024-06-03T08:36:31.992 INFO:teuthology.orchestra.run.smithi088.stderr:2024-06-03T08:36:31.990+0000 7f199da81640 10 monclient: shutdown 2024-06-03T08:36:31.992 INFO:teuthology.orchestra.run.smithi088.stderr:2024-06-03T08:36:31.990+0000 7f199da81640 20 monclient: shutdown discarding 0 pending message(s) 2024-06-03T08:36:31.992 INFO:teuthology.orchestra.run.smithi088.stderr:2024-06-03T08:36:31.990+0000 7f199da81640 1 -- 172.21.15.88:0/402424495 >> v1:172.21.15.88:6790/0 conn(0x7f1990006410 legacy=0x7f19900c6ea0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-06-03T08:36:31.992 INFO:teuthology.orchestra.run.smithi088.stderr:2024-06-03T08:36:31.989+0000 7ff49a002640 1 -- start start 2024-06-03T08:36:31.992 INFO:teuthology.orchestra.run.smithi088.stderr:2024-06-03T08:36:31.989+0000 7ff49a002640 10 monclient: init 2024-06-03T08:36:31.992 INFO:teuthology.orchestra.run.smithi088.stderr:2024-06-03T08:36:31.990+0000 7ff49a002640 10 monclient: _reopen_session rank -1 2024-06-03T08:36:31.992 INFO:teuthology.orchestra.run.smithi088.stderr:2024-06-03T08:36:31.990+0000 7ff49a002640 10 monclient: _add_conns ranks=[0,1,2] 2024-06-03T08:36:31.992 INFO:teuthology.orchestra.run.smithi088.stderr:2024-06-03T08:36:31.990+0000 7ff49a002640 10 monclient(hunting): picked mon.noname-a con 0x7ff49406dc10 addr v1:172.21.15.88:6789/0 2024-06-03T08:36:31.992 INFO:teuthology.orchestra.run.smithi088.stderr:2024-06-03T08:36:31.990+0000 7ff49a002640 10 monclient(hunting): picked mon.noname-c con 0x7ff49406f070 addr v1:172.21.15.192:6789/0 2024-06-03T08:36:31.993 INFO:teuthology.orchestra.run.smithi088.stderr:2024-06-03T08:36:31.990+0000 7ff49a002640 10 monclient(hunting): picked mon.noname-b con 0x7ff494141540 addr v1:172.21.15.88:6790/0 2024-06-03T08:36:31.993 INFO:teuthology.orchestra.run.smithi088.stderr:2024-06-03T08:36:31.990+0000 7ff49a002640 1 -- --> v1:172.21.15.88:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7ff494070140 con 0x7ff49406dc10 2024-06-03T08:36:31.993 INFO:teuthology.orchestra.run.smithi088.stderr:2024-06-03T08:36:31.990+0000 7ff49a002640 1 -- --> v1:172.21.15.192:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7ff4941419d0 con 0x7ff49406f070 2024-06-03T08:36:31.993 INFO:teuthology.orchestra.run.smithi088.stderr:2024-06-03T08:36:31.990+0000 7ff49a002640 1 -- --> v1:172.21.15.88:6790/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7ff494141020 con 0x7ff494141540 2024-06-03T08:36:31.993 INFO:teuthology.orchestra.run.smithi088.stderr:2024-06-03T08:36:31.990+0000 7ff49a002640 10 monclient(hunting): _renew_subs 2024-06-03T08:36:31.993 INFO:teuthology.orchestra.run.smithi088.stderr:2024-06-03T08:36:31.990+0000 7ff49a002640 10 monclient(hunting): authenticate will time out at 1247.800302s 2024-06-03T08:36:31.993 INFO:teuthology.orchestra.run.smithi088.stderr:2024-06-03T08:36:31.990+0000 7f199da81640 1 -- 172.21.15.88:0/402424495 shutdown_connections 2024-06-03T08:36:31.993 INFO:teuthology.orchestra.run.smithi088.stderr:2024-06-03T08:36:31.990+0000 7f199da81640 1 -- 172.21.15.88:0/402424495 >> 172.21.15.88:0/402424495 conn(0x7f19900058f0 msgr2=0x7f1990005d00 unknown :-1 s=STATE_NONE l=0).mark_down 2024-06-03T08:36:31.994 INFO:teuthology.orchestra.run.smithi088.stderr:2024-06-03T08:36:31.990+0000 7f199da81640 1 -- 172.21.15.88:0/402424495 shutdown_connections 2024-06-03T08:36:31.994 INFO:teuthology.orchestra.run.smithi088.stderr:2024-06-03T08:36:31.991+0000 7ff492ffd640 1 --1- >> v1:172.21.15.192:6789/0 conn(0x7ff49406f070 0x7ff4940aded0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.192:6789/0 says I am v1:172.21.15.88:52326/0 (socket says 172.21.15.88:52326) 2024-06-03T08:36:31.994 INFO:teuthology.orchestra.run.smithi088.stderr:2024-06-03T08:36:31.991+0000 7ff492ffd640 1 -- 172.21.15.88:0/2655868321 learned_addr learned my addr 172.21.15.88:0/2655868321 (peer_addr_for_me v1:172.21.15.88:0/0) 2024-06-03T08:36:31.995 INFO:teuthology.orchestra.run.smithi088.stderr:2024-06-03T08:36:31.990+0000 7f199da81640 1 -- 172.21.15.88:0/402424495 wait complete. 2024-06-03T08:36:31.995 INFO:teuthology.orchestra.run.smithi088.stderr:2024-06-03T08:36:31.991+0000 7f199da81640 10 monclient: build_initial_monmap 2024-06-03T08:36:31.995 INFO:teuthology.orchestra.run.smithi088.stderr:2024-06-03T08:36:31.991+0000 7f199da81640 10 monclient: monmap: 2024-06-03T08:36:31.996 INFO:teuthology.orchestra.run.smithi088.stderr:epoch 0 2024-06-03T08:36:31.996 INFO:teuthology.orchestra.run.smithi088.stderr:fsid 00000000-0000-0000-0000-000000000000 2024-06-03T08:36:31.996 INFO:teuthology.orchestra.run.smithi088.stderr:last_changed 0.000000 2024-06-03T08:36:31.996 INFO:teuthology.orchestra.run.smithi088.stderr:created 0.000000 2024-06-03T08:36:31.996 INFO:teuthology.orchestra.run.smithi088.stderr:min_mon_release 0 (unknown) 2024-06-03T08:36:31.996 INFO:teuthology.orchestra.run.smithi088.stderr:election_strategy: 1 2024-06-03T08:36:31.996 INFO:teuthology.orchestra.run.smithi088.stderr:0: v1:172.21.15.88:6789/0 mon.noname-a 2024-06-03T08:36:31.996 INFO:teuthology.orchestra.run.smithi088.stderr:1: v1:172.21.15.192:6789/0 mon.noname-b 2024-06-03T08:36:31.996 INFO:teuthology.orchestra.run.smithi088.stderr:2: v1:172.21.15.88:6790/0 mon.noname-c 2024-06-03T08:36:31.996 INFO:teuthology.orchestra.run.smithi088.stderr: 2024-06-03T08:36:31.996 INFO:teuthology.orchestra.run.smithi088.stderr:2024-06-03T08:36:31.992+0000 7f199da81640 1 librados: starting msgr at 2024-06-03T08:36:31.996 INFO:teuthology.orchestra.run.smithi088.stderr:2024-06-03T08:36:31.992+0000 7f199da81640 1 librados: starting objecter 2024-06-03T08:36:31.996 INFO:teuthology.orchestra.run.smithi088.stderr:2024-06-03T08:36:31.992+0000 7f199da81640 1 Processor -- start 2024-06-03T08:36:31.997 INFO:teuthology.orchestra.run.smithi088.stderr:2024-06-03T08:36:31.993+0000 7f199da81640 1 -- start start 2024-06-03T08:36:31.997 INFO:teuthology.orchestra.run.smithi088.stderr:2024-06-03T08:36:31.993+0000 7f199da81640 1 librados: setting wanted keys 2024-06-03T08:36:31.997 INFO:teuthology.orchestra.run.smithi088.stderr:2024-06-03T08:36:31.993+0000 7f199da81640 1 librados: calling monclient init 2024-06-03T08:36:31.997 INFO:teuthology.orchestra.run.smithi088.stderr:2024-06-03T08:36:31.993+0000 7f199da81640 10 monclient: init 2024-06-03T08:36:31.997 INFO:teuthology.orchestra.run.smithi088.stderr:2024-06-03T08:36:31.993+0000 7f199da81640 10 monclient: _reopen_session rank -1 2024-06-03T08:36:31.997 INFO:teuthology.orchestra.run.smithi088.stderr:2024-06-03T08:36:31.993+0000 7f199da81640 10 monclient: _add_conns ranks=[0,2,1] 2024-06-03T08:36:31.997 INFO:teuthology.orchestra.run.smithi088.stderr:2024-06-03T08:36:31.993+0000 7f199da81640 10 monclient(hunting): picked mon.noname-a con 0x7f1990006410 addr v1:172.21.15.88:6789/0 2024-06-03T08:36:31.997 INFO:teuthology.orchestra.run.smithi088.stderr:2024-06-03T08:36:31.995+0000 7f199da81640 10 monclient(hunting): picked mon.noname-c con 0x7f199002a640 addr v1:172.21.15.88:6790/0 2024-06-03T08:36:31.998 INFO:teuthology.orchestra.run.smithi088.stderr:2024-06-03T08:36:31.995+0000 7f199da81640 10 monclient(hunting): picked mon.noname-b con 0x7f19900327d0 addr v1:172.21.15.192:6789/0 2024-06-03T08:36:31.998 INFO:teuthology.orchestra.run.smithi088.stderr:2024-06-03T08:36:31.995+0000 7f199da81640 1 -- --> v1:172.21.15.88:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f19900d3880 con 0x7f1990006410 2024-06-03T08:36:31.998 INFO:teuthology.orchestra.run.smithi088.stderr:2024-06-03T08:36:31.996+0000 7f199da81640 1 -- --> v1:172.21.15.192:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f19900d3ad0 con 0x7f19900327d0 2024-06-03T08:36:31.999 INFO:teuthology.orchestra.run.smithi088.stderr:2024-06-03T08:36:31.996+0000 7f19977fe640 1 --1- >> v1:172.21.15.88:6789/0 conn(0x7f1990006410 0x7f19901641f0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=1).handle_server_banner_and_identify peer v1:172.21.15.88:6789/0 says I am v1:172.21.15.88:60280/0 (socket says 172.21.15.88:60280) 2024-06-03T08:36:31.999 INFO:teuthology.orchestra.run.smithi088.stderr:2024-06-03T08:36:31.996+0000 7f19977fe640 1 -- 172.21.15.88:0/1438849704 learned_addr learned my addr 172.21.15.88:0/1438849704 (peer_addr_for_me v1:172.21.15.88:0/0) 2024-06-03T08:36:31.999 INFO:teuthology.orchestra.run.smithi088.stderr:2024-06-03T08:36:31.996+0000 7f199da81640 1 -- 172.21.15.88:0/1438849704 --> v1:172.21.15.88:6790/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f1990164c70 con 0x7f199002a640 2024-06-03T08:36:31.999 INFO:teuthology.orchestra.run.smithi088.stderr:2024-06-03T08:36:31.996+0000 7f199da81640 10 monclient(hunting): _renew_subs 2024-06-03T08:36:31.999 INFO:teuthology.orchestra.run.smithi088.stderr:2024-06-03T08:36:31.996+0000 7f199da81640 10 monclient(hunting): authenticate will time out at 1247.806601s 2024-06-03T08:36:31.999 INFO:teuthology.orchestra.run.smithi088.stderr:2024-06-03T08:36:31.996+0000 7f1983fff640 10 client.?.objecter ms_handle_connect 0x7f1990006410 2024-06-03T08:36:31.999 INFO:teuthology.orchestra.run.smithi088.stderr:2024-06-03T08:36:31.996+0000 7f1983fff640 10 client.?.objecter resend_mon_ops 2024-06-03T08:36:32.000 INFO:teuthology.orchestra.run.smithi088.stderr:2024-06-03T08:36:31.996+0000 7fd65b934640 10 monclient: get_monmap_and_config 2024-06-03T08:36:32.000 INFO:teuthology.orchestra.run.smithi088.stderr:2024-06-03T08:36:31.996+0000 7fd65b934640 10 monclient: build_initial_monmap 2024-06-03T08:36:32.000 INFO:teuthology.orchestra.run.smithi088.stderr:2024-06-03T08:36:31.996+0000 7fd65b934640 10 monclient: monmap: 2024-06-03T08:36:32.000 INFO:teuthology.orchestra.run.smithi088.stderr:epoch 0 2024-06-03T08:36:32.000 INFO:teuthology.orchestra.run.smithi088.stderr:fsid ec3c3fdd-3ac6-4d7d-9412-683130ed8b04 2024-06-03T08:36:32.000 INFO:teuthology.orchestra.run.smithi088.stderr:last_changed 2024-06-03T08:36:31.997363+0000 2024-06-03T08:36:32.000 INFO:teuthology.orchestra.run.smithi088.stderr:created 2024-06-03T08:36:31.997363+0000 2024-06-03T08:36:32.000 INFO:teuthology.orchestra.run.smithi088.stderr:min_mon_release 0 (unknown) 2024-06-03T08:36:32.000 INFO:teuthology.orchestra.run.smithi088.stderr:election_strategy: 3 2024-06-03T08:36:32.000 INFO:teuthology.orchestra.run.smithi088.stderr:0: v1:172.21.15.88:6789/0 mon.noname-a 2024-06-03T08:36:32.000 INFO:teuthology.orchestra.run.smithi088.stderr:1: v1:172.21.15.192:6789/0 mon.noname-c 2024-06-03T08:36:32.001 INFO:teuthology.orchestra.run.smithi088.stderr:2: v1:172.21.15.88:6790/0 mon.noname-b 2024-06-03T08:36:32.001 INFO:teuthology.orchestra.run.smithi088.stderr: 2024-06-03T08:36:32.001 INFO:teuthology.orchestra.run.smithi088.stderr:2024-06-03T08:36:31.998+0000 7fd65b934640 1 Processor -- start 2024-06-03T08:36:32.001 INFO:teuthology.orchestra.run.smithi088.stderr:2024-06-03T08:36:31.998+0000 7fd65b934640 1 -- start start 2024-06-03T08:36:32.001 INFO:teuthology.orchestra.run.smithi088.stderr:2024-06-03T08:36:31.998+0000 7fd65b934640 10 monclient: init 2024-06-03T08:36:32.001 INFO:teuthology.orchestra.run.smithi088.stderr:2024-06-03T08:36:31.999+0000 7fd65b934640 10 monclient: _reopen_session rank -1 2024-06-03T08:36:32.001 INFO:teuthology.orchestra.run.smithi088.stderr:2024-06-03T08:36:31.999+0000 7fd65b934640 10 monclient: _add_conns ranks=[1,2,0] 2024-06-03T08:36:32.001 INFO:teuthology.orchestra.run.smithi088.stderr:2024-06-03T08:36:31.999+0000 7fd65b934640 10 monclient(hunting): picked mon.noname-c con 0x7fd65406dc10 addr v1:172.21.15.192:6789/0 2024-06-03T08:36:32.001 INFO:teuthology.orchestra.run.smithi088.stderr:2024-06-03T08:36:31.999+0000 7fd65b934640 10 monclient(hunting): picked mon.noname-b con 0x7fd65406f070 addr v1:172.21.15.88:6790/0 2024-06-03T08:36:32.001 INFO:teuthology.orchestra.run.smithi088.stderr:2024-06-03T08:36:31.999+0000 7fd65b934640 10 monclient(hunting): picked mon.noname-a con 0x7fd654141540 addr v1:172.21.15.88:6789/0 2024-06-03T08:36:32.001 INFO:teuthology.orchestra.run.smithi088.stderr:2024-06-03T08:36:31.999+0000 7fd65b934640 1 -- --> v1:172.21.15.88:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fd654070140 con 0x7fd654141540 2024-06-03T08:36:32.001 INFO:teuthology.orchestra.run.smithi088.stderr:2024-06-03T08:36:31.999+0000 7fd65b934640 1 -- --> v1:172.21.15.192:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fd6541419d0 con 0x7fd65406dc10 2024-06-03T08:36:32.001 INFO:teuthology.orchestra.run.smithi088.stderr:2024-06-03T08:36:31.999+0000 7fd65b934640 1 -- --> v1:172.21.15.88:6790/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fd654141020 con 0x7fd65406f070 2024-06-03T08:36:32.002 INFO:teuthology.orchestra.run.smithi088.stderr:2024-06-03T08:36:31.999+0000 7fd65b934640 10 monclient(hunting): _renew_subs 2024-06-03T08:36:32.002 INFO:teuthology.orchestra.run.smithi088.stderr:2024-06-03T08:36:31.999+0000 7fd65b934640 10 monclient(hunting): authenticate will time out at 1247.809270s 2024-06-03T08:36:32.002 INFO:teuthology.orchestra.run.smithi088.stderr:2024-06-03T08:36:31.999+0000 7f1983fff640 10 client.?.objecter ms_handle_connect 0x7f199002a640 2024-06-03T08:36:32.002 INFO:teuthology.orchestra.run.smithi088.stderr:2024-06-03T08:36:31.999+0000 7f1983fff640 10 client.?.objecter resend_mon_ops 2024-06-03T08:36:32.002 INFO:teuthology.orchestra.run.smithi088.stderr:2024-06-03T08:36:31.999+0000 7f1983fff640 10 client.?.objecter ms_handle_connect 0x7f19900327d0 2024-06-03T08:36:32.002 INFO:teuthology.orchestra.run.smithi088.stderr:2024-06-03T08:36:31.999+0000 7f1983fff640 10 client.?.objecter resend_mon_ops 2024-06-03T08:36:32.002 INFO:teuthology.orchestra.run.smithi088.stderr:2024-06-03T08:36:31.998+0000 7ff4927fc640 1 -- 172.21.15.88:0/2655868321 <== mon.1 v1:172.21.15.192:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 3175278565 0 0) 0x7ff4941419d0 con 0x7ff49406f070 2024-06-03T08:36:32.003 INFO:teuthology.orchestra.run.smithi088.stderr:2024-06-03T08:36:31.998+0000 7ff4927fc640 10 monclient(hunting): _negotiate 2024-06-03T08:36:32.003 INFO:teuthology.orchestra.run.smithi088.stderr:2024-06-03T08:36:31.998+0000 7ff4927fc640 10 monclient(hunting): _init_auth method 2 2024-06-03T08:36:32.003 INFO:teuthology.orchestra.run.smithi088.stderr:2024-06-03T08:36:31.998+0000 7ff4927fc640 10 monclient(hunting): _init_auth creating new auth 2024-06-03T08:36:32.003 INFO:teuthology.orchestra.run.smithi088.stderr:2024-06-03T08:36:31.998+0000 7ff4927fc640 10 monclient(hunting): my global_id is 4297 2024-06-03T08:36:32.003 INFO:teuthology.orchestra.run.smithi088.stderr:2024-06-03T08:36:31.998+0000 7ff4927fc640 1 -- 172.21.15.88:0/2655868321 --> v1:172.21.15.192:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7ff47c003520 con 0x7ff49406f070 2024-06-03T08:36:32.003 INFO:teuthology.orchestra.run.smithi088.stderr:2024-06-03T08:36:31.999+0000 7ff4927fc640 1 -- 172.21.15.88:0/2655868321 <== mon.2 v1:172.21.15.88:6790/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 3122653976 0 0) 0x7ff494141020 con 0x7ff494141540 2024-06-03T08:36:32.003 INFO:teuthology.orchestra.run.smithi088.stderr:2024-06-03T08:36:31.999+0000 7ff4927fc640 10 monclient(hunting): _negotiate 2024-06-03T08:36:32.003 INFO:teuthology.orchestra.run.smithi088.stderr:2024-06-03T08:36:31.999+0000 7ff4927fc640 10 monclient(hunting): _init_auth method 2 2024-06-03T08:36:32.003 INFO:teuthology.orchestra.run.smithi088.stderr:2024-06-03T08:36:31.999+0000 7ff4927fc640 10 monclient(hunting): _init_auth creating new auth 2024-06-03T08:36:32.003 INFO:teuthology.orchestra.run.smithi088.stderr:2024-06-03T08:36:31.999+0000 7ff4927fc640 10 monclient(hunting): my global_id is 4313 2024-06-03T08:36:32.003 INFO:teuthology.orchestra.run.smithi088.stderr:2024-06-03T08:36:31.999+0000 7ff4927fc640 1 -- 172.21.15.88:0/2655868321 --> v1:172.21.15.88:6790/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7ff4941419d0 con 0x7ff494141540 2024-06-03T08:36:32.004 INFO:teuthology.orchestra.run.smithi088.stderr:2024-06-03T08:36:31.997+0000 7fd347fb9640 10 monclient: get_monmap_and_config 2024-06-03T08:36:32.004 INFO:teuthology.orchestra.run.smithi088.stderr:2024-06-03T08:36:31.997+0000 7fd347fb9640 10 monclient: build_initial_monmap 2024-06-03T08:36:32.004 INFO:teuthology.orchestra.run.smithi088.stderr:2024-06-03T08:36:31.997+0000 7fd347fb9640 10 monclient: monmap: 2024-06-03T08:36:32.004 INFO:teuthology.orchestra.run.smithi088.stderr:epoch 0 2024-06-03T08:36:32.004 INFO:teuthology.orchestra.run.smithi088.stderr:fsid ec3c3fdd-3ac6-4d7d-9412-683130ed8b04 2024-06-03T08:36:32.004 INFO:teuthology.orchestra.run.smithi088.stderr:last_changed 2024-06-03T08:36:31.998338+0000 2024-06-03T08:36:32.004 INFO:teuthology.orchestra.run.smithi088.stderr:created 2024-06-03T08:36:31.998338+0000 2024-06-03T08:36:32.004 INFO:teuthology.orchestra.run.smithi088.stderr:min_mon_release 0 (unknown) 2024-06-03T08:36:32.004 INFO:teuthology.orchestra.run.smithi088.stderr:election_strategy: 3 2024-06-03T08:36:32.004 INFO:teuthology.orchestra.run.smithi088.stderr:0: v1:172.21.15.88:6789/0 mon.noname-a 2024-06-03T08:36:32.004 INFO:teuthology.orchestra.run.smithi088.stderr:1: v1:172.21.15.192:6789/0 mon.noname-c 2024-06-03T08:36:32.004 INFO:teuthology.orchestra.run.smithi088.stderr:2: v1:172.21.15.88:6790/0 mon.noname-b 2024-06-03T08:36:32.004 INFO:teuthology.orchestra.run.smithi088.stderr: 2024-06-03T08:36:32.005 INFO:teuthology.orchestra.run.smithi088.stderr:2024-06-03T08:36:31.999+0000 7ff4927fc640 1 -- 172.21.15.88:0/2655868321 <== mon.0 v1:172.21.15.88:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 3272147651 0 0) 0x7ff494070140 con 0x7ff49406dc10 2024-06-03T08:36:32.005 INFO:teuthology.orchestra.run.smithi088.stderr:2024-06-03T08:36:31.999+0000 7ff4927fc640 10 monclient(hunting): _negotiate 2024-06-03T08:36:32.005 INFO:teuthology.orchestra.run.smithi088.stderr:2024-06-03T08:36:31.999+0000 7ff4927fc640 10 monclient(hunting): _init_auth method 2 2024-06-03T08:36:32.005 INFO:teuthology.orchestra.run.smithi088.stderr:2024-06-03T08:36:32.000+0000 7ff4927fc640 10 monclient(hunting): _init_auth creating new auth 2024-06-03T08:36:32.006 INFO:teuthology.orchestra.run.smithi088.stderr:2024-06-03T08:36:32.000+0000 7ff4927fc640 10 monclient(hunting): my global_id is 4293 2024-06-03T08:36:32.006 INFO:teuthology.orchestra.run.smithi088.stderr:2024-06-03T08:36:32.000+0000 7ff4927fc640 1 -- 172.21.15.88:0/2655868321 --> v1:172.21.15.88:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7ff494141020 con 0x7ff49406dc10 2024-06-03T08:36:32.006 INFO:teuthology.orchestra.run.smithi088.stderr:2024-06-03T08:36:32.000+0000 7fd347fb9640 1 Processor -- start 2024-06-03T08:36:32.006 INFO:teuthology.orchestra.run.smithi088.stderr:2024-06-03T08:36:32.000+0000 7fd658ecf640 1 --1- >> v1:172.21.15.88:6790/0 conn(0x7fd65406f070 0x7fd6540aded0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.88:6790/0 says I am v1:172.21.15.88:54088/0 (socket says 172.21.15.88:54088) 2024-06-03T08:36:32.006 INFO:teuthology.orchestra.run.smithi088.stderr:2024-06-03T08:36:32.000+0000 7fd658ecf640 1 -- 172.21.15.88:0/3008984679 learned_addr learned my addr 172.21.15.88:0/3008984679 (peer_addr_for_me v1:172.21.15.88:0/0) 2024-06-03T08:36:32.007 INFO:teuthology.orchestra.run.smithi088.stderr:2024-06-03T08:36:32.000+0000 7fd347fb9640 1 -- start start 2024-06-03T08:36:32.007 INFO:teuthology.orchestra.run.smithi088.stderr:2024-06-03T08:36:32.000+0000 7fd347fb9640 10 monclient: init 2024-06-03T08:36:32.007 INFO:teuthology.orchestra.run.smithi088.stderr:2024-06-03T08:36:32.001+0000 7fd347fb9640 10 monclient: _reopen_session rank -1 2024-06-03T08:36:32.007 INFO:teuthology.orchestra.run.smithi088.stderr:2024-06-03T08:36:32.001+0000 7fd347fb9640 10 monclient: _add_conns ranks=[2,0,1] 2024-06-03T08:36:32.007 INFO:teuthology.orchestra.run.smithi088.stderr:2024-06-03T08:36:32.001+0000 7fd347fb9640 10 monclient(hunting): picked mon.noname-b con 0x7fd338006410 addr v1:172.21.15.88:6790/0 2024-06-03T08:36:32.007 INFO:teuthology.orchestra.run.smithi088.stderr:2024-06-03T08:36:32.001+0000 7fd347fb9640 10 monclient(hunting): picked mon.noname-a con 0x7fd33802a640 addr v1:172.21.15.88:6789/0 2024-06-03T08:36:32.007 INFO:teuthology.orchestra.run.smithi088.stderr:2024-06-03T08:36:32.001+0000 7fd347fb9640 10 monclient(hunting): picked mon.noname-c con 0x7fd3380327d0 addr v1:172.21.15.192:6789/0 2024-06-03T08:36:32.007 INFO:teuthology.orchestra.run.smithi088.stderr:2024-06-03T08:36:32.001+0000 7fd347fb9640 1 -- --> v1:172.21.15.88:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fd33802c820 con 0x7fd33802a640 2024-06-03T08:36:32.007 INFO:teuthology.orchestra.run.smithi088.stderr:2024-06-03T08:36:32.001+0000 7fd347fb9640 1 -- --> v1:172.21.15.192:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fd33802a120 con 0x7fd3380327d0 2024-06-03T08:36:32.008 INFO:teuthology.orchestra.run.smithi088.stderr:2024-06-03T08:36:32.001+0000 7fd347fb9640 1 -- --> v1:172.21.15.88:6790/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fd3380c75b0 con 0x7fd338006410 2024-06-03T08:36:32.008 INFO:teuthology.orchestra.run.smithi088.stderr:2024-06-03T08:36:32.001+0000 7fd347fb9640 10 monclient(hunting): _renew_subs 2024-06-03T08:36:32.008 INFO:teuthology.orchestra.run.smithi088.stderr:2024-06-03T08:36:32.001+0000 7fd347fb9640 10 monclient(hunting): authenticate will time out at 1247.811759s 2024-06-03T08:36:32.008 INFO:teuthology.orchestra.run.smithi088.stderr:2024-06-03T08:36:32.003+0000 7fd345554640 1 --1- >> v1:172.21.15.88:6789/0 conn(0x7fd33802a640 0x7fd3380320c0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.88:6789/0 says I am v1:172.21.15.88:60288/0 (socket says 172.21.15.88:60288) 2024-06-03T08:36:32.008 INFO:teuthology.orchestra.run.smithi088.stderr:2024-06-03T08:36:32.003+0000 7fd345554640 1 -- 172.21.15.88:0/2256678336 learned_addr learned my addr 172.21.15.88:0/2256678336 (peer_addr_for_me v1:172.21.15.88:0/0) 2024-06-03T08:36:32.008 INFO:teuthology.orchestra.run.smithi088.stderr:2024-06-03T08:36:32.004+0000 7ff4927fc640 1 -- 172.21.15.88:0/2655868321 <== mon.1 v1:172.21.15.192:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 214+0+0 (unknown 1756644259 0 0) 0x7ff47c003520 con 0x7ff49406f070 2024-06-03T08:36:32.008 INFO:teuthology.orchestra.run.smithi088.stderr:2024-06-03T08:36:32.004+0000 7ff4927fc640 1 -- 172.21.15.88:0/2655868321 >> v1:172.21.15.88:6790/0 conn(0x7ff494141540 legacy=0x7ff4940aeec0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-06-03T08:36:32.008 INFO:teuthology.orchestra.run.smithi088.stderr:2024-06-03T08:36:32.004+0000 7ff4927fc640 1 -- 172.21.15.88:0/2655868321 >> v1:172.21.15.88:6789/0 conn(0x7ff49406dc10 legacy=0x7ff49406dff0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-06-03T08:36:32.009 INFO:teuthology.orchestra.run.smithi088.stderr:2024-06-03T08:36:32.004+0000 7ff4927fc640 10 monclient: _finish_hunting 0 2024-06-03T08:36:32.009 INFO:teuthology.orchestra.run.smithi088.stderr:2024-06-03T08:36:32.004+0000 7ff4927fc640 1 monclient: found mon.noname-c 2024-06-03T08:36:32.009 INFO:teuthology.orchestra.run.smithi088.stderr:2024-06-03T08:36:32.004+0000 7ff4927fc640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-06-03T08:36:32.009 INFO:teuthology.orchestra.run.smithi088.stderr:2024-06-03T08:36:32.004+0000 7ff4927fc640 10 monclient: _send_mon_message to mon.noname-c at v1:172.21.15.192:6789/0 2024-06-03T08:36:32.009 INFO:teuthology.orchestra.run.smithi088.stderr:2024-06-03T08:36:32.004+0000 7ff4927fc640 1 -- 172.21.15.88:0/2655868321 --> v1:172.21.15.192:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7ff49406edb0 con 0x7ff49406f070 2024-06-03T08:36:32.009 INFO:teuthology.orchestra.run.smithi088.stderr:2024-06-03T08:36:32.004+0000 7ff4927fc640 10 monclient: _finish_auth 0 2024-06-03T08:36:32.009 INFO:teuthology.orchestra.run.smithi088.stderr:2024-06-03T08:36:32.004+0000 7ff4927fc640 10 monclient: _check_auth_tickets 2024-06-03T08:36:32.009 INFO:teuthology.orchestra.run.smithi088.stderr:2024-06-03T08:36:32.004+0000 7ff4927fc640 20 monclient: _check_auth_rotating not needed by client.admin 2024-06-03T08:36:32.009 INFO:teuthology.orchestra.run.smithi088.stderr:2024-06-03T08:36:32.004+0000 7ff4927fc640 1 -- 172.21.15.88:0/2655868321 <== mon.1 v1:172.21.15.192:6789/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 2234374065 0 0) 0x7ff484001540 con 0x7ff49406f070 2024-06-03T08:36:32.009 INFO:teuthology.orchestra.run.smithi088.stderr:2024-06-03T08:36:32.004+0000 7ff4927fc640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-06-03T08:36:32.009 INFO:teuthology.orchestra.run.smithi088.stderr:2024-06-03T08:36:32.004+0000 7ff4927fc640 10 monclient: got monmap 1 from mon.noname-c (according to old e1) 2024-06-03T08:36:32.009 INFO:teuthology.orchestra.run.smithi088.stderr:2024-06-03T08:36:32.004+0000 7ff4927fc640 10 monclient: dump: 2024-06-03T08:36:32.009 INFO:teuthology.orchestra.run.smithi088.stderr:epoch 1 2024-06-03T08:36:32.010 INFO:teuthology.orchestra.run.smithi088.stderr:fsid ec3c3fdd-3ac6-4d7d-9412-683130ed8b04 2024-06-03T08:36:32.010 INFO:teuthology.orchestra.run.smithi088.stderr:last_changed 2024-06-03T08:34:31.885125+0000 2024-06-03T08:36:32.010 INFO:teuthology.orchestra.run.smithi088.stderr:created 2024-06-03T08:34:31.885125+0000 2024-06-03T08:36:32.010 INFO:teuthology.orchestra.run.smithi088.stderr:min_mon_release 18 (reef) 2024-06-03T08:36:32.010 INFO:teuthology.orchestra.run.smithi088.stderr:election_strategy: 3 2024-06-03T08:36:32.010 INFO:teuthology.orchestra.run.smithi088.stderr:0: v1:172.21.15.88:6789/0 mon.a 2024-06-03T08:36:32.010 INFO:teuthology.orchestra.run.smithi088.stderr:1: v1:172.21.15.192:6789/0 mon.b 2024-06-03T08:36:32.010 INFO:teuthology.orchestra.run.smithi088.stderr:2: v1:172.21.15.88:6790/0 mon.c 2024-06-03T08:36:32.010 INFO:teuthology.orchestra.run.smithi088.stderr: 2024-06-03T08:36:32.010 INFO:teuthology.orchestra.run.smithi088.stderr:2024-06-03T08:36:32.004+0000 7ff49a002640 5 monclient: authenticate success, global_id 4297 2024-06-03T08:36:32.010 INFO:teuthology.orchestra.run.smithi088.stderr:2024-06-03T08:36:32.004+0000 7ff49a002640 20 monclient: get_monmap_and_config waiting for monmap|config 2024-06-03T08:36:32.011 INFO:teuthology.orchestra.run.smithi088.stderr:2024-06-03T08:36:32.006+0000 7f1983fff640 1 -- 172.21.15.88:0/1438849704 <== mon.0 v1:172.21.15.88:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 1801313688 0 0) 0x7f19900d3880 con 0x7f1990006410 2024-06-03T08:36:32.011 INFO:teuthology.orchestra.run.smithi088.stderr:2024-06-03T08:36:32.006+0000 7f1983fff640 10 monclient(hunting): _negotiate 2024-06-03T08:36:32.011 INFO:teuthology.orchestra.run.smithi088.stderr:2024-06-03T08:36:32.006+0000 7f1983fff640 10 monclient(hunting): _init_auth method 2 2024-06-03T08:36:32.011 INFO:teuthology.orchestra.run.smithi088.stderr:2024-06-03T08:36:32.006+0000 7f1983fff640 10 monclient(hunting): _init_auth creating new auth 2024-06-03T08:36:32.011 INFO:teuthology.orchestra.run.smithi088.stderr:2024-06-03T08:36:32.006+0000 7f1983fff640 10 monclient(hunting): my global_id is 4296 2024-06-03T08:36:32.011 INFO:teuthology.orchestra.run.smithi088.stderr:2024-06-03T08:36:32.006+0000 7f1983fff640 1 -- 172.21.15.88:0/1438849704 --> v1:172.21.15.88:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f197c004eb0 con 0x7f1990006410 2024-06-03T08:36:32.011 INFO:teuthology.orchestra.run.smithi088.stderr:2024-06-03T08:36:32.006+0000 7f1983fff640 1 -- 172.21.15.88:0/1438849704 <== mon.1 v1:172.21.15.192:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 2709851297 0 0) 0x7f19900d3ad0 con 0x7f19900327d0 2024-06-03T08:36:32.012 INFO:teuthology.orchestra.run.smithi088.stderr:2024-06-03T08:36:32.006+0000 7f1983fff640 10 monclient(hunting): _negotiate 2024-06-03T08:36:32.012 INFO:teuthology.orchestra.run.smithi088.stderr:2024-06-03T08:36:32.006+0000 7f1983fff640 10 monclient(hunting): _init_auth method 2 2024-06-03T08:36:32.012 INFO:teuthology.orchestra.run.smithi088.stderr:2024-06-03T08:36:32.006+0000 7f1983fff640 10 monclient(hunting): _init_auth creating new auth 2024-06-03T08:36:32.012 INFO:teuthology.orchestra.run.smithi088.stderr:2024-06-03T08:36:32.006+0000 7f1983fff640 10 monclient(hunting): my global_id is 4300 2024-06-03T08:36:32.012 INFO:teuthology.orchestra.run.smithi088.stderr:2024-06-03T08:36:32.006+0000 7f1983fff640 1 -- 172.21.15.88:0/1438849704 --> v1:172.21.15.192:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f19900d3880 con 0x7f19900327d0 2024-06-03T08:36:32.012 INFO:teuthology.orchestra.run.smithi088.stderr:2024-06-03T08:36:32.007+0000 7f4c36155640 10 monclient: get_monmap_and_config 2024-06-03T08:36:32.013 INFO:teuthology.orchestra.run.smithi088.stderr:2024-06-03T08:36:32.007+0000 7f4c36155640 10 monclient: build_initial_monmap 2024-06-03T08:36:32.013 INFO:teuthology.orchestra.run.smithi088.stderr:2024-06-03T08:36:32.007+0000 7f4c36155640 10 monclient: monmap: 2024-06-03T08:36:32.013 INFO:teuthology.orchestra.run.smithi088.stderr:epoch 0 2024-06-03T08:36:32.013 INFO:teuthology.orchestra.run.smithi088.stderr:fsid ec3c3fdd-3ac6-4d7d-9412-683130ed8b04 2024-06-03T08:36:32.013 INFO:teuthology.orchestra.run.smithi088.stderr:last_changed 2024-06-03T08:36:32.008681+0000 2024-06-03T08:36:32.013 INFO:teuthology.orchestra.run.smithi088.stderr:created 2024-06-03T08:36:32.008681+0000 2024-06-03T08:36:32.013 INFO:teuthology.orchestra.run.smithi088.stderr:min_mon_release 0 (unknown) 2024-06-03T08:36:32.013 INFO:teuthology.orchestra.run.smithi088.stderr:election_strategy: 3 2024-06-03T08:36:32.013 INFO:teuthology.orchestra.run.smithi088.stderr:0: v1:172.21.15.88:6789/0 mon.noname-a 2024-06-03T08:36:32.013 INFO:teuthology.orchestra.run.smithi088.stderr:1: v1:172.21.15.192:6789/0 mon.noname-c 2024-06-03T08:36:32.013 INFO:teuthology.orchestra.run.smithi088.stderr:2: v1:172.21.15.88:6790/0 mon.noname-b 2024-06-03T08:36:32.014 INFO:teuthology.orchestra.run.smithi088.stderr: 2024-06-03T08:36:32.014 INFO:teuthology.orchestra.run.smithi088.stderr:2024-06-03T08:36:32.008+0000 7f4c36155640 1 Processor -- start 2024-06-03T08:36:32.014 INFO:teuthology.orchestra.run.smithi088.stderr:2024-06-03T08:36:32.008+0000 7f4c36155640 1 -- start start 2024-06-03T08:36:32.014 INFO:teuthology.orchestra.run.smithi088.stderr:2024-06-03T08:36:32.008+0000 7f4c36155640 10 monclient: init 2024-06-03T08:36:32.014 INFO:teuthology.orchestra.run.smithi088.stderr:2024-06-03T08:36:32.009+0000 7fd344d53640 1 -- 172.21.15.88:0/2256678336 <== mon.1 v1:172.21.15.192:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 1327398702 0 0) 0x7fd33802a120 con 0x7fd3380327d0 2024-06-03T08:36:32.014 INFO:teuthology.orchestra.run.smithi088.stderr:2024-06-03T08:36:32.009+0000 7fd344d53640 10 monclient(hunting): _negotiate 2024-06-03T08:36:32.014 INFO:teuthology.orchestra.run.smithi088.stderr:2024-06-03T08:36:32.009+0000 7fd344d53640 10 monclient(hunting): _init_auth method 2 2024-06-03T08:36:32.014 INFO:teuthology.orchestra.run.smithi088.stderr:2024-06-03T08:36:32.009+0000 7fd344d53640 10 monclient(hunting): _init_auth creating new auth 2024-06-03T08:36:32.014 INFO:teuthology.orchestra.run.smithi088.stderr:2024-06-03T08:36:32.009+0000 7fd344d53640 10 monclient(hunting): my global_id is 4306 2024-06-03T08:36:32.014 INFO:teuthology.orchestra.run.smithi088.stderr:2024-06-03T08:36:32.009+0000 7fd344d53640 1 -- 172.21.15.88:0/2256678336 --> v1:172.21.15.192:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7fd32c003520 con 0x7fd3380327d0 2024-06-03T08:36:32.015 INFO:teuthology.orchestra.run.smithi088.stderr:2024-06-03T08:36:32.009+0000 7f4c36155640 10 monclient: _reopen_session rank -1 2024-06-03T08:36:32.015 INFO:teuthology.orchestra.run.smithi088.stderr:2024-06-03T08:36:32.009+0000 7f4c36155640 10 monclient: _add_conns ranks=[2,0,1] 2024-06-03T08:36:32.015 INFO:teuthology.orchestra.run.smithi088.stderr:2024-06-03T08:36:32.010+0000 7f4c36155640 10 monclient(hunting): picked mon.noname-b con 0x7f4c300b4060 addr v1:172.21.15.88:6790/0 2024-06-03T08:36:32.016 INFO:teuthology.orchestra.run.smithi088.stderr:2024-06-03T08:36:32.010+0000 7f4c36155640 10 monclient(hunting): picked mon.noname-a con 0x7f4c3013ee50 addr v1:172.21.15.88:6789/0 2024-06-03T08:36:32.016 INFO:teuthology.orchestra.run.smithi088.stderr:2024-06-03T08:36:32.010+0000 7f4c36155640 10 monclient(hunting): picked mon.noname-c con 0x7f4c300a86f0 addr v1:172.21.15.192:6789/0 2024-06-03T08:36:32.016 INFO:teuthology.orchestra.run.smithi088.stderr:2024-06-03T08:36:32.010+0000 7f4c36155640 1 -- --> v1:172.21.15.88:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f4c300a8ca0 con 0x7f4c3013ee50 2024-06-03T08:36:32.016 INFO:teuthology.orchestra.run.smithi088.stderr:2024-06-03T08:36:32.010+0000 7fd64bfff640 1 -- 172.21.15.88:0/3008984679 <== mon.1 v1:172.21.15.192:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 2339175227 0 0) 0x7fd6541419d0 con 0x7fd65406dc10 2024-06-03T08:36:32.016 INFO:teuthology.orchestra.run.smithi088.stderr:2024-06-03T08:36:32.010+0000 7fd64bfff640 10 monclient(hunting): _negotiate 2024-06-03T08:36:32.017 INFO:teuthology.orchestra.run.smithi088.stderr:2024-06-03T08:36:32.010+0000 7fd64bfff640 10 monclient(hunting): _init_auth method 2 2024-06-03T08:36:32.017 INFO:teuthology.orchestra.run.smithi088.stderr:2024-06-03T08:36:32.010+0000 7fd64bfff640 10 monclient(hunting): _init_auth creating new auth 2024-06-03T08:36:32.017 INFO:teuthology.orchestra.run.smithi088.stderr:2024-06-03T08:36:32.010+0000 7fd64bfff640 10 monclient(hunting): my global_id is 4303 2024-06-03T08:36:32.017 INFO:teuthology.orchestra.run.smithi088.stderr:2024-06-03T08:36:32.010+0000 7fd64bfff640 1 -- 172.21.15.88:0/3008984679 --> v1:172.21.15.192:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7fd640003520 con 0x7fd65406dc10 2024-06-03T08:36:32.017 INFO:teuthology.orchestra.run.smithi088.stderr:2024-06-03T08:36:32.010+0000 7f4c36155640 1 -- --> v1:172.21.15.192:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f4c300a9130 con 0x7f4c300a86f0 2024-06-03T08:36:32.018 INFO:teuthology.orchestra.run.smithi088.stderr:2024-06-03T08:36:32.010+0000 7f4c36155640 1 -- --> v1:172.21.15.88:6790/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f4c300a9f00 con 0x7f4c300b4060 2024-06-03T08:36:32.018 INFO:teuthology.orchestra.run.smithi088.stderr:2024-06-03T08:36:32.010+0000 7f4c36155640 10 monclient(hunting): _renew_subs 2024-06-03T08:36:32.018 INFO:teuthology.orchestra.run.smithi088.stderr:2024-06-03T08:36:32.010+0000 7f4c36155640 10 monclient(hunting): authenticate will time out at 1247.820808s 2024-06-03T08:36:32.018 INFO:teuthology.orchestra.run.smithi088.stderr:2024-06-03T08:36:32.014+0000 7f4c2f7fe640 1 --1- >> v1:172.21.15.88:6790/0 conn(0x7f4c300b4060 0x7f4c3013e740 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.88:6790/0 says I am v1:172.21.15.88:54118/0 (socket says 172.21.15.88:54118) 2024-06-03T08:36:32.018 INFO:teuthology.orchestra.run.smithi088.stderr:2024-06-03T08:36:32.014+0000 7f4c2f7fe640 1 -- 172.21.15.88:0/795329302 learned_addr learned my addr 172.21.15.88:0/795329302 (peer_addr_for_me v1:172.21.15.88:0/0) 2024-06-03T08:36:32.019 INFO:teuthology.orchestra.run.smithi088.stderr:2024-06-03T08:36:32.014+0000 7fd64bfff640 1 -- 172.21.15.88:0/3008984679 <== mon.0 v1:172.21.15.88:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 3462988026 0 0) 0x7fd654070140 con 0x7fd654141540 2024-06-03T08:36:32.019 INFO:teuthology.orchestra.run.smithi088.stderr:2024-06-03T08:36:32.014+0000 7fd64bfff640 10 monclient(hunting): _negotiate 2024-06-03T08:36:32.019 INFO:teuthology.orchestra.run.smithi088.stderr:2024-06-03T08:36:32.014+0000 7fd64bfff640 10 monclient(hunting): _init_auth method 2 2024-06-03T08:36:32.019 INFO:teuthology.orchestra.run.smithi088.stderr:2024-06-03T08:36:32.014+0000 7fd64bfff640 10 monclient(hunting): _init_auth creating new auth 2024-06-03T08:36:32.019 INFO:teuthology.orchestra.run.smithi088.stderr:2024-06-03T08:36:32.014+0000 7fd64bfff640 10 monclient(hunting): my global_id is 4299 2024-06-03T08:36:32.019 INFO:teuthology.orchestra.run.smithi088.stderr:2024-06-03T08:36:32.014+0000 7fd64bfff640 1 -- 172.21.15.88:0/3008984679 --> v1:172.21.15.88:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7fd6541419d0 con 0x7fd654141540 2024-06-03T08:36:32.019 INFO:teuthology.orchestra.run.smithi088.stderr:2024-06-03T08:36:32.014+0000 7fd64bfff640 1 -- 172.21.15.88:0/3008984679 <== mon.2 v1:172.21.15.88:6790/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 3530669382 0 0) 0x7fd654141020 con 0x7fd65406f070 2024-06-03T08:36:32.019 INFO:teuthology.orchestra.run.smithi088.stderr:2024-06-03T08:36:32.014+0000 7fd64bfff640 10 monclient(hunting): _negotiate 2024-06-03T08:36:32.020 INFO:teuthology.orchestra.run.smithi088.stderr:2024-06-03T08:36:32.014+0000 7fd64bfff640 10 monclient(hunting): _init_auth method 2 2024-06-03T08:36:32.020 INFO:teuthology.orchestra.run.smithi088.stderr:2024-06-03T08:36:32.014+0000 7fd64bfff640 10 monclient(hunting): _init_auth creating new auth 2024-06-03T08:36:32.020 INFO:teuthology.orchestra.run.smithi088.stderr:2024-06-03T08:36:32.014+0000 7fd64bfff640 10 monclient(hunting): my global_id is 4319 2024-06-03T08:36:32.020 INFO:teuthology.orchestra.run.smithi088.stderr:2024-06-03T08:36:32.014+0000 7fd64bfff640 1 -- 172.21.15.88:0/3008984679 --> v1:172.21.15.88:6790/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7fd654070140 con 0x7fd65406f070 2024-06-03T08:36:32.020 INFO:teuthology.orchestra.run.smithi088.stderr:2024-06-03T08:36:32.015+0000 7f1983fff640 1 -- 172.21.15.88:0/1438849704 <== mon.1 v1:172.21.15.192:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 764+0+0 (unknown 2634087336 0 0) 0x7f19900d3880 con 0x7f19900327d0 2024-06-03T08:36:32.020 INFO:teuthology.orchestra.run.smithi088.stderr:2024-06-03T08:36:32.015+0000 7f1983fff640 1 -- 172.21.15.88:0/1438849704 --> v1:172.21.15.192:6789/0 -- auth(proto 2 165 bytes epoch 0) v1 -- 0x7f19900d3ad0 con 0x7f19900327d0 2024-06-03T08:36:32.020 INFO:teuthology.orchestra.run.smithi088.stderr:2024-06-03T08:36:32.015+0000 7f1983fff640 1 -- 172.21.15.88:0/1438849704 <== mon.1 v1:172.21.15.192:6789/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 2234374065 0 0) 0x7f19840018b0 con 0x7f19900327d0 2024-06-03T08:36:32.020 INFO:teuthology.orchestra.run.smithi088.stderr:2024-06-03T08:36:32.015+0000 7f1983fff640 10 monclient(hunting): handle_monmap mon_map magic: 0 v1 2024-06-03T08:36:32.020 INFO:teuthology.orchestra.run.smithi088.stderr:2024-06-03T08:36:32.015+0000 7f1983fff640 10 monclient(hunting): got monmap 1 from mon.noname-b (according to old e1) 2024-06-03T08:36:32.021 INFO:teuthology.orchestra.run.smithi088.stderr:2024-06-03T08:36:32.015+0000 7f1983fff640 10 monclient(hunting): dump: 2024-06-03T08:36:32.021 INFO:teuthology.orchestra.run.smithi088.stderr:epoch 1 2024-06-03T08:36:32.021 INFO:teuthology.orchestra.run.smithi088.stderr:fsid ec3c3fdd-3ac6-4d7d-9412-683130ed8b04 2024-06-03T08:36:32.021 INFO:teuthology.orchestra.run.smithi088.stderr:last_changed 2024-06-03T08:34:31.885125+0000 2024-06-03T08:36:32.021 INFO:teuthology.orchestra.run.smithi088.stderr:created 2024-06-03T08:34:31.885125+0000 2024-06-03T08:36:32.021 INFO:teuthology.orchestra.run.smithi088.stderr:min_mon_release 18 (reef) 2024-06-03T08:36:32.021 INFO:teuthology.orchestra.run.smithi088.stderr:election_strategy: 3 2024-06-03T08:36:32.021 INFO:teuthology.orchestra.run.smithi088.stderr:0: v1:172.21.15.88:6789/0 mon.a 2024-06-03T08:36:32.021 INFO:teuthology.orchestra.run.smithi088.stderr:1: v1:172.21.15.192:6789/0 mon.b 2024-06-03T08:36:32.021 INFO:teuthology.orchestra.run.smithi088.stderr:2: v1:172.21.15.88:6790/0 mon.c 2024-06-03T08:36:32.021 INFO:teuthology.orchestra.run.smithi088.stderr: 2024-06-03T08:36:32.021 INFO:teuthology.orchestra.run.smithi088.stderr:2024-06-03T08:36:32.015+0000 7f1983fff640 10 monclient(hunting): _finish_auth 0 2024-06-03T08:36:32.022 INFO:teuthology.orchestra.run.smithi088.stderr:2024-06-03T08:36:32.019+0000 7fa0f1016640 10 monclient: get_monmap_and_config 2024-06-03T08:36:32.022 INFO:teuthology.orchestra.run.smithi088.stderr:2024-06-03T08:36:32.019+0000 7fa0f1016640 10 monclient: build_initial_monmap 2024-06-03T08:36:32.022 INFO:teuthology.orchestra.run.smithi088.stderr:2024-06-03T08:36:32.019+0000 7fa0f1016640 10 monclient: monmap: 2024-06-03T08:36:32.022 INFO:teuthology.orchestra.run.smithi088.stderr:epoch 0 2024-06-03T08:36:32.022 INFO:teuthology.orchestra.run.smithi088.stderr:fsid ec3c3fdd-3ac6-4d7d-9412-683130ed8b04 2024-06-03T08:36:32.022 INFO:teuthology.orchestra.run.smithi088.stderr:last_changed 2024-06-03T08:36:32.020239+0000 2024-06-03T08:36:32.022 INFO:teuthology.orchestra.run.smithi088.stderr:created 2024-06-03T08:36:32.020239+0000 2024-06-03T08:36:32.022 INFO:teuthology.orchestra.run.smithi088.stderr:min_mon_release 0 (unknown) 2024-06-03T08:36:32.022 INFO:teuthology.orchestra.run.smithi088.stderr:election_strategy: 3 2024-06-03T08:36:32.022 INFO:teuthology.orchestra.run.smithi088.stderr:0: v1:172.21.15.88:6789/0 mon.noname-a 2024-06-03T08:36:32.023 INFO:teuthology.orchestra.run.smithi088.stderr:1: v1:172.21.15.192:6789/0 mon.noname-c 2024-06-03T08:36:32.023 INFO:teuthology.orchestra.run.smithi088.stderr:2: v1:172.21.15.88:6790/0 mon.noname-b 2024-06-03T08:36:32.023 INFO:teuthology.orchestra.run.smithi088.stderr: 2024-06-03T08:36:32.023 INFO:teuthology.orchestra.run.smithi088.stderr:2024-06-03T08:36:32.019+0000 7f9bafd56640 10 monclient: get_monmap_and_config 2024-06-03T08:36:32.023 INFO:teuthology.orchestra.run.smithi088.stderr:2024-06-03T08:36:32.019+0000 7f9bafd56640 10 monclient: build_initial_monmap 2024-06-03T08:36:32.023 INFO:teuthology.orchestra.run.smithi088.stderr:2024-06-03T08:36:32.019+0000 7f9bafd56640 10 monclient: monmap: 2024-06-03T08:36:32.023 INFO:teuthology.orchestra.run.smithi088.stderr:epoch 0 2024-06-03T08:36:32.023 INFO:teuthology.orchestra.run.smithi088.stderr:fsid ec3c3fdd-3ac6-4d7d-9412-683130ed8b04 2024-06-03T08:36:32.023 INFO:teuthology.orchestra.run.smithi088.stderr:last_changed 2024-06-03T08:36:32.020732+0000 2024-06-03T08:36:32.023 INFO:teuthology.orchestra.run.smithi088.stderr:created 2024-06-03T08:36:32.020732+0000 2024-06-03T08:36:32.024 INFO:teuthology.orchestra.run.smithi088.stderr:min_mon_release 0 (unknown) 2024-06-03T08:36:32.024 INFO:teuthology.orchestra.run.smithi088.stderr:election_strategy: 3 2024-06-03T08:36:32.024 INFO:teuthology.orchestra.run.smithi088.stderr:0: v1:172.21.15.88:6789/0 mon.noname-a 2024-06-03T08:36:32.024 INFO:teuthology.orchestra.run.smithi088.stderr:1: v1:172.21.15.192:6789/0 mon.noname-c 2024-06-03T08:36:32.024 INFO:teuthology.orchestra.run.smithi088.stderr:2: v1:172.21.15.88:6790/0 mon.noname-b 2024-06-03T08:36:32.024 INFO:teuthology.orchestra.run.smithi088.stderr: 2024-06-03T08:36:32.024 INFO:teuthology.orchestra.run.smithi088.stderr:2024-06-03T08:36:32.020+0000 7fa0f1016640 1 Processor -- start 2024-06-03T08:36:32.024 INFO:teuthology.orchestra.run.smithi088.stderr:2024-06-03T08:36:32.020+0000 7fa0f1016640 1 -- start start 2024-06-03T08:36:32.024 INFO:teuthology.orchestra.run.smithi088.stderr:2024-06-03T08:36:32.020+0000 7fa0f1016640 10 monclient: init 2024-06-03T08:36:32.025 INFO:teuthology.orchestra.run.smithi088.stderr:2024-06-03T08:36:32.020+0000 7f9bafd56640 1 Processor -- start 2024-06-03T08:36:32.026 INFO:teuthology.orchestra.run.smithi088.stderr:2024-06-03T08:36:32.020+0000 7f1983fff640 1 -- 172.21.15.88:0/1438849704 <== mon.2 v1:172.21.15.88:6790/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 3890029046 0 0) 0x7f1990164c70 con 0x7f199002a640 2024-06-03T08:36:32.026 INFO:teuthology.orchestra.run.smithi088.stderr:2024-06-03T08:36:32.020+0000 7f1983fff640 10 monclient(hunting): _negotiate 2024-06-03T08:36:32.026 INFO:teuthology.orchestra.run.smithi088.stderr:2024-06-03T08:36:32.020+0000 7f1983fff640 10 monclient(hunting): _init_auth method 2 2024-06-03T08:36:32.026 INFO:teuthology.orchestra.run.smithi088.stderr:2024-06-03T08:36:32.020+0000 7f1983fff640 10 monclient(hunting): _init_auth creating new auth 2024-06-03T08:36:32.026 INFO:teuthology.orchestra.run.smithi088.stderr:2024-06-03T08:36:32.020+0000 7f1983fff640 10 monclient(hunting): my global_id is 4316 2024-06-03T08:36:32.026 INFO:teuthology.orchestra.run.smithi088.stderr:2024-06-03T08:36:32.020+0000 7f1983fff640 1 -- 172.21.15.88:0/1438849704 --> v1:172.21.15.88:6790/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f19900d3880 con 0x7f199002a640 2024-06-03T08:36:32.027 INFO:teuthology.orchestra.run.smithi088.stderr:2024-06-03T08:36:32.020+0000 7fa0f1016640 10 monclient: _reopen_session rank -1 2024-06-03T08:36:32.027 INFO:teuthology.orchestra.run.smithi088.stderr:2024-06-03T08:36:32.020+0000 7fa0f1016640 10 monclient: _add_conns ranks=[1,0,2] 2024-06-03T08:36:32.027 INFO:teuthology.orchestra.run.smithi088.stderr:2024-06-03T08:36:32.020+0000 7fa0f1016640 10 monclient(hunting): picked mon.noname-c con 0x7fa0ec0a8960 addr v1:172.21.15.192:6789/0 2024-06-03T08:36:32.027 INFO:teuthology.orchestra.run.smithi088.stderr:2024-06-03T08:36:32.021+0000 7fa0f1016640 10 monclient(hunting): picked mon.noname-a con 0x7fa0ec1450f0 addr v1:172.21.15.88:6789/0 2024-06-03T08:36:32.027 INFO:teuthology.orchestra.run.smithi088.stderr:2024-06-03T08:36:32.021+0000 7fa0f1016640 10 monclient(hunting): picked mon.noname-b con 0x7fa0ec13eec0 addr v1:172.21.15.88:6790/0 2024-06-03T08:36:32.027 INFO:teuthology.orchestra.run.smithi088.stderr:2024-06-03T08:36:32.021+0000 7fa0f1016640 1 -- --> v1:172.21.15.88:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fa0ec144c60 con 0x7fa0ec1450f0 2024-06-03T08:36:32.027 INFO:teuthology.orchestra.run.smithi088.stderr:2024-06-03T08:36:32.021+0000 7fa0f1016640 1 -- --> v1:172.21.15.192:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fa0ec0b3930 con 0x7fa0ec0a8960 2024-06-03T08:36:32.027 INFO:teuthology.orchestra.run.smithi088.stderr:2024-06-03T08:36:32.021+0000 7fa0f1016640 1 -- --> v1:172.21.15.88:6790/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fa0ec0b1ad0 con 0x7fa0ec13eec0 2024-06-03T08:36:32.027 INFO:teuthology.orchestra.run.smithi088.stderr:2024-06-03T08:36:32.021+0000 7fa0f1016640 10 monclient(hunting): _renew_subs 2024-06-03T08:36:32.027 INFO:teuthology.orchestra.run.smithi088.stderr:2024-06-03T08:36:32.021+0000 7fa0f1016640 10 monclient(hunting): authenticate will time out at 1247.831025s 2024-06-03T08:36:32.027 INFO:teuthology.orchestra.run.smithi088.stderr:2024-06-03T08:36:32.021+0000 7f9bafd56640 1 -- start start 2024-06-03T08:36:32.028 INFO:teuthology.orchestra.run.smithi088.stderr:2024-06-03T08:36:32.021+0000 7f9bafd56640 10 monclient: init 2024-06-03T08:36:32.028 INFO:teuthology.orchestra.run.smithi088.stderr:2024-06-03T08:36:32.021+0000 7f1983fff640 1 -- 172.21.15.88:0/1438849704 <== mon.0 v1:172.21.15.88:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 764+0+0 (unknown 1610592729 0 0) 0x7f197c004eb0 con 0x7f1990006410 2024-06-03T08:36:32.028 INFO:teuthology.orchestra.run.smithi088.stderr:2024-06-03T08:36:32.021+0000 7f1983fff640 1 -- 172.21.15.88:0/1438849704 --> v1:172.21.15.88:6789/0 -- auth(proto 2 165 bytes epoch 0) v1 -- 0x7f1990164c70 con 0x7f1990006410 2024-06-03T08:36:32.028 INFO:teuthology.orchestra.run.smithi088.stderr:2024-06-03T08:36:32.021+0000 7f1983fff640 1 -- 172.21.15.88:0/1438849704 <== mon.0 v1:172.21.15.88:6789/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 2234374065 0 0) 0x7f198c0015d0 con 0x7f1990006410 2024-06-03T08:36:32.028 INFO:teuthology.orchestra.run.smithi088.stderr:2024-06-03T08:36:32.021+0000 7f1983fff640 10 monclient(hunting): handle_monmap mon_map magic: 0 v1 2024-06-03T08:36:32.028 INFO:teuthology.orchestra.run.smithi088.stderr:2024-06-03T08:36:32.021+0000 7f1983fff640 10 monclient(hunting): got monmap 1 from mon.a (according to old e1) 2024-06-03T08:36:32.028 INFO:teuthology.orchestra.run.smithi088.stderr:2024-06-03T08:36:32.021+0000 7f1983fff640 10 monclient(hunting): dump: 2024-06-03T08:36:32.029 INFO:teuthology.orchestra.run.smithi088.stderr:epoch 1 2024-06-03T08:36:32.029 INFO:teuthology.orchestra.run.smithi088.stderr:fsid ec3c3fdd-3ac6-4d7d-9412-683130ed8b04 2024-06-03T08:36:32.029 INFO:teuthology.orchestra.run.smithi088.stderr:last_changed 2024-06-03T08:34:31.885125+0000 2024-06-03T08:36:32.029 INFO:teuthology.orchestra.run.smithi088.stderr:created 2024-06-03T08:34:31.885125+0000 2024-06-03T08:36:32.029 INFO:teuthology.orchestra.run.smithi088.stderr:min_mon_release 18 (reef) 2024-06-03T08:36:32.029 INFO:teuthology.orchestra.run.smithi088.stderr:election_strategy: 3 2024-06-03T08:36:32.029 INFO:teuthology.orchestra.run.smithi088.stderr:0: v1:172.21.15.88:6789/0 mon.a 2024-06-03T08:36:32.029 INFO:teuthology.orchestra.run.smithi088.stderr:1: v1:172.21.15.192:6789/0 mon.b 2024-06-03T08:36:32.029 INFO:teuthology.orchestra.run.smithi088.stderr:2: v1:172.21.15.88:6790/0 mon.c 2024-06-03T08:36:32.029 INFO:teuthology.orchestra.run.smithi088.stderr: 2024-06-03T08:36:32.030 INFO:teuthology.orchestra.run.smithi088.stderr:2024-06-03T08:36:32.021+0000 7f9bafd56640 10 monclient: _reopen_session rank -1 2024-06-03T08:36:32.030 INFO:teuthology.orchestra.run.smithi088.stderr:2024-06-03T08:36:32.022+0000 7f9bafd56640 10 monclient: _add_conns ranks=[2,0,1] 2024-06-03T08:36:32.030 INFO:teuthology.orchestra.run.smithi088.stderr:2024-06-03T08:36:32.022+0000 7f9bafd56640 10 monclient(hunting): picked mon.noname-b con 0x7f9ba80b43c0 addr v1:172.21.15.88:6790/0 2024-06-03T08:36:32.030 INFO:teuthology.orchestra.run.smithi088.stderr:2024-06-03T08:36:32.022+0000 7f1983fff640 1 -- 172.21.15.88:0/1438849704 <== mon.1 v1:172.21.15.192:6789/0 4 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 580+0+0 (unknown 1535685072 0 0) 0x7f19900d3ad0 con 0x7f19900327d0 2024-06-03T08:36:32.031 INFO:teuthology.orchestra.run.smithi088.stderr:2024-06-03T08:36:32.022+0000 7f1983fff640 1 -- 172.21.15.88:0/1438849704 >> v1:172.21.15.88:6790/0 conn(0x7f199002a640 legacy=0x7f19900bf950 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-06-03T08:36:32.031 INFO:teuthology.orchestra.run.smithi088.stderr:2024-06-03T08:36:32.022+0000 7f1983fff640 1 -- 172.21.15.88:0/1438849704 >> v1:172.21.15.88:6789/0 conn(0x7f1990006410 legacy=0x7f19901641f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-06-03T08:36:32.031 INFO:teuthology.orchestra.run.smithi088.stderr:2024-06-03T08:36:32.022+0000 7f1983fff640 10 monclient: _finish_hunting 0 2024-06-03T08:36:32.031 INFO:teuthology.orchestra.run.smithi088.stderr:2024-06-03T08:36:32.022+0000 7f1983fff640 1 monclient: found mon.b 2024-06-03T08:36:32.031 INFO:teuthology.orchestra.run.smithi088.stderr:2024-06-03T08:36:32.022+0000 7f1983fff640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-06-03T08:36:32.031 INFO:teuthology.orchestra.run.smithi088.stderr:2024-06-03T08:36:32.022+0000 7f1983fff640 10 monclient: _send_mon_message to mon.b at v1:172.21.15.192:6789/0 2024-06-03T08:36:32.031 INFO:teuthology.orchestra.run.smithi088.stderr:2024-06-03T08:36:32.022+0000 7f1983fff640 1 -- 172.21.15.88:0/1438849704 --> v1:172.21.15.192:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f1990164900 con 0x7f19900327d0 2024-06-03T08:36:32.031 INFO:teuthology.orchestra.run.smithi088.stderr:2024-06-03T08:36:32.022+0000 7f9bafd56640 10 monclient(hunting): picked mon.noname-a con 0x7f9ba813e6c0 addr v1:172.21.15.88:6789/0 2024-06-03T08:36:32.032 INFO:teuthology.orchestra.run.smithi088.stderr:2024-06-03T08:36:32.022+0000 7f1983fff640 10 monclient: _finish_auth 0 2024-06-03T08:36:32.032 INFO:teuthology.orchestra.run.smithi088.stderr:2024-06-03T08:36:32.022+0000 7f1983fff640 10 monclient: _check_auth_tickets 2024-06-03T08:36:32.032 INFO:teuthology.orchestra.run.smithi088.stderr:2024-06-03T08:36:32.022+0000 7f1983fff640 20 monclient: _check_auth_rotating not needed by client.admin 2024-06-03T08:36:32.032 INFO:teuthology.orchestra.run.smithi088.stderr:2024-06-03T08:36:32.022+0000 7f9bafd56640 10 monclient(hunting): picked mon.noname-c con 0x7f9ba814ae20 addr v1:172.21.15.192:6789/0 2024-06-03T08:36:32.032 INFO:teuthology.orchestra.run.smithi088.stderr:2024-06-03T08:36:32.022+0000 7f199da81640 5 monclient: authenticate success, global_id 4300 2024-06-03T08:36:32.032 INFO:teuthology.orchestra.run.smithi088.stderr:2024-06-03T08:36:32.022+0000 7f199da81640 10 monclient: _renew_subs 2024-06-03T08:36:32.032 INFO:teuthology.orchestra.run.smithi088.stderr:2024-06-03T08:36:32.022+0000 7f199da81640 10 monclient: _send_mon_message to mon.b at v1:172.21.15.192:6789/0 2024-06-03T08:36:32.033 INFO:teuthology.orchestra.run.smithi088.stderr:2024-06-03T08:36:32.022+0000 7f199da81640 1 -- 172.21.15.88:0/1438849704 --> v1:172.21.15.192:6789/0 -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f19900d2bd0 con 0x7f19900327d0 2024-06-03T08:36:32.033 INFO:teuthology.orchestra.run.smithi088.stderr:2024-06-03T08:36:32.022+0000 7f199da81640 10 client.4300.objecter _maybe_request_map subscribing (onetime) to next osd map 2024-06-03T08:36:32.033 INFO:teuthology.orchestra.run.smithi088.stderr:2024-06-03T08:36:32.022+0000 7f199da81640 10 monclient: _renew_subs 2024-06-03T08:36:32.033 INFO:teuthology.orchestra.run.smithi088.stderr:2024-06-03T08:36:32.022+0000 7f199da81640 10 monclient: _send_mon_message to mon.b at v1:172.21.15.192:6789/0 2024-06-03T08:36:32.033 INFO:teuthology.orchestra.run.smithi088.stderr:2024-06-03T08:36:32.022+0000 7f199da81640 1 -- 172.21.15.88:0/1438849704 --> v1:172.21.15.192:6789/0 -- mon_subscribe({osdmap=0}) v3 -- 0x7f199016c4a0 con 0x7f19900327d0 2024-06-03T08:36:32.033 INFO:teuthology.orchestra.run.smithi088.stderr:2024-06-03T08:36:32.022+0000 7f199da81640 1 librados: init done 2024-06-03T08:36:32.033 INFO:teuthology.orchestra.run.smithi088.stderr:2024-06-03T08:36:32.022+0000 7f9bafd56640 1 -- --> v1:172.21.15.88:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f9ba80a8ed0 con 0x7f9ba813e6c0 2024-06-03T08:36:32.033 INFO:teuthology.orchestra.run.smithi088.stderr:2024-06-03T08:36:32.022+0000 7f9bafd56640 1 -- --> v1:172.21.15.192:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f9ba80aa420 con 0x7f9ba814ae20 2024-06-03T08:36:32.034 INFO:teuthology.orchestra.run.smithi088.stderr:2024-06-03T08:36:32.022+0000 7f9bafd56640 1 -- --> v1:172.21.15.88:6790/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f9ba80aa690 con 0x7f9ba80b43c0 2024-06-03T08:36:32.034 INFO:teuthology.orchestra.run.smithi088.stderr:2024-06-03T08:36:32.022+0000 7f9bafd56640 10 monclient(hunting): _renew_subs 2024-06-03T08:36:32.034 INFO:teuthology.orchestra.run.smithi088.stderr:2024-06-03T08:36:32.022+0000 7f9bafd56640 10 monclient(hunting): authenticate will time out at 1247.832373s 2024-06-03T08:36:32.034 INFO:teuthology.orchestra.run.smithi088.stderr:2024-06-03T08:36:32.022+0000 7fd344d53640 1 -- 172.21.15.88:0/2256678336 <== mon.0 v1:172.21.15.88:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 1066001383 0 0) 0x7fd33802c820 con 0x7fd33802a640 2024-06-03T08:36:32.034 INFO:teuthology.orchestra.run.smithi088.stderr:2024-06-03T08:36:32.022+0000 7fd344d53640 10 monclient(hunting): _negotiate 2024-06-03T08:36:32.034 INFO:teuthology.orchestra.run.smithi088.stderr:2024-06-03T08:36:32.022+0000 7fd344d53640 10 monclient(hunting): _init_auth method 2 2024-06-03T08:36:32.034 INFO:teuthology.orchestra.run.smithi088.stderr:2024-06-03T08:36:32.022+0000 7fd344d53640 10 monclient(hunting): _init_auth creating new auth 2024-06-03T08:36:32.035 INFO:teuthology.orchestra.run.smithi088.stderr:2024-06-03T08:36:32.022+0000 7fd344d53640 10 monclient(hunting): my global_id is 4302 2024-06-03T08:36:32.035 INFO:teuthology.orchestra.run.smithi088.stderr:2024-06-03T08:36:32.022+0000 7fd344d53640 1 -- 172.21.15.88:0/2256678336 --> v1:172.21.15.88:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7fd33802a120 con 0x7fd33802a640 2024-06-03T08:36:32.035 INFO:teuthology.orchestra.run.smithi088.stderr:2024-06-03T08:36:32.023+0000 7f199da81640 10 client.4300.objecter _submit_command 1 [{"prefix": "get_command_descriptions"}] 2024-06-03T08:36:32.036 INFO:teuthology.orchestra.run.smithi088.stderr:2024-06-03T08:36:32.023+0000 7f199da81640 15 client.4300.objecter _session_command_op_assign -1 1 2024-06-03T08:36:32.036 INFO:teuthology.orchestra.run.smithi088.stderr:2024-06-03T08:36:32.023+0000 7f199da81640 20 client.4300.objecter _get_session osd=-1 returning homeless 2024-06-03T08:36:32.036 INFO:teuthology.orchestra.run.smithi088.stderr:2024-06-03T08:36:32.023+0000 7f199da81640 10 client.4300.objecter _maybe_request_map subscribing (onetime) to next osd map 2024-06-03T08:36:32.036 INFO:teuthology.orchestra.run.smithi088.stderr:2024-06-03T08:36:32.023+0000 7f199da81640 10 monclient: _send_mon_message to mon.b at v1:172.21.15.192:6789/0 2024-06-03T08:36:32.036 INFO:teuthology.orchestra.run.smithi088.stderr:2024-06-03T08:36:32.023+0000 7f199da81640 1 -- 172.21.15.88:0/1438849704 --> v1:172.21.15.192:6789/0 -- mon_get_version(what=osdmap handle=1) v1 -- 0x7f19900347c0 con 0x7f19900327d0 2024-06-03T08:36:32.036 INFO:teuthology.orchestra.run.smithi088.stderr:2024-06-03T08:36:32.023+0000 7f791cb92640 10 monclient: get_monmap_and_config 2024-06-03T08:36:32.036 INFO:teuthology.orchestra.run.smithi088.stderr:2024-06-03T08:36:32.023+0000 7f791cb92640 10 monclient: build_initial_monmap 2024-06-03T08:36:32.037 INFO:teuthology.orchestra.run.smithi088.stderr:2024-06-03T08:36:32.023+0000 7f791cb92640 10 monclient: monmap: 2024-06-03T08:36:32.037 INFO:teuthology.orchestra.run.smithi088.stderr:epoch 0 2024-06-03T08:36:32.037 INFO:teuthology.orchestra.run.smithi088.stderr:fsid ec3c3fdd-3ac6-4d7d-9412-683130ed8b04 2024-06-03T08:36:32.037 INFO:teuthology.orchestra.run.smithi088.stderr:last_changed 2024-06-03T08:36:32.024288+0000 2024-06-03T08:36:32.037 INFO:teuthology.orchestra.run.smithi088.stderr:created 2024-06-03T08:36:32.024288+0000 2024-06-03T08:36:32.037 INFO:teuthology.orchestra.run.smithi088.stderr:min_mon_release 0 (unknown) 2024-06-03T08:36:32.037 INFO:teuthology.orchestra.run.smithi088.stderr:election_strategy: 3 2024-06-03T08:36:32.037 INFO:teuthology.orchestra.run.smithi088.stderr:0: v1:172.21.15.88:6789/0 mon.noname-a 2024-06-03T08:36:32.037 INFO:teuthology.orchestra.run.smithi088.stderr:1: v1:172.21.15.192:6789/0 mon.noname-c 2024-06-03T08:36:32.037 INFO:teuthology.orchestra.run.smithi088.stderr:2: v1:172.21.15.88:6790/0 mon.noname-b 2024-06-03T08:36:32.037 INFO:teuthology.orchestra.run.smithi088.stderr: 2024-06-03T08:36:32.037 INFO:teuthology.orchestra.run.smithi088.stderr:2024-06-03T08:36:32.023+0000 7f791cb92640 1 Processor -- start 2024-06-03T08:36:32.039 INFO:teuthology.orchestra.run.smithi088.stderr:2024-06-03T08:36:32.024+0000 7fd64bfff640 1 -- 172.21.15.88:0/3008984679 <== mon.2 v1:172.21.15.88:6790/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 214+0+0 (unknown 1707592752 0 0) 0x7fd654070140 con 0x7fd65406f070 2024-06-03T08:36:32.039 INFO:teuthology.orchestra.run.smithi088.stderr:2024-06-03T08:36:32.024+0000 7fd64bfff640 1 -- 172.21.15.88:0/3008984679 >> v1:172.21.15.192:6789/0 conn(0x7fd65406dc10 legacy=0x7fd65406dff0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-06-03T08:36:32.039 INFO:teuthology.orchestra.run.smithi088.stderr:2024-06-03T08:36:32.024+0000 7fd64bfff640 1 -- 172.21.15.88:0/3008984679 >> v1:172.21.15.88:6789/0 conn(0x7fd654141540 legacy=0x7fd6540aeec0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-06-03T08:36:32.039 INFO:teuthology.orchestra.run.smithi088.stderr:2024-06-03T08:36:32.024+0000 7fd64bfff640 10 monclient: _finish_hunting 0 2024-06-03T08:36:32.039 INFO:teuthology.orchestra.run.smithi088.stderr:2024-06-03T08:36:32.024+0000 7fd64bfff640 1 monclient: found mon.noname-b 2024-06-03T08:36:32.040 INFO:teuthology.orchestra.run.smithi088.stderr:2024-06-03T08:36:32.024+0000 7fd64bfff640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-06-03T08:36:32.040 INFO:teuthology.orchestra.run.smithi088.stderr:2024-06-03T08:36:32.024+0000 7fd64bfff640 10 monclient: _send_mon_message to mon.noname-b at v1:172.21.15.88:6790/0 2024-06-03T08:36:32.040 INFO:teuthology.orchestra.run.smithi088.stderr:2024-06-03T08:36:32.024+0000 7fd64bfff640 1 -- 172.21.15.88:0/3008984679 --> v1:172.21.15.88:6790/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fd65406edb0 con 0x7fd65406f070 2024-06-03T08:36:32.040 INFO:teuthology.orchestra.run.smithi088.stderr:2024-06-03T08:36:32.024+0000 7fd64bfff640 10 monclient: _finish_auth 0 2024-06-03T08:36:32.040 INFO:teuthology.orchestra.run.smithi088.stderr:2024-06-03T08:36:32.024+0000 7fd64bfff640 10 monclient: _check_auth_tickets 2024-06-03T08:36:32.040 INFO:teuthology.orchestra.run.smithi088.stderr:2024-06-03T08:36:32.024+0000 7fd64bfff640 20 monclient: _check_auth_rotating not needed by client.admin 2024-06-03T08:36:32.040 INFO:teuthology.orchestra.run.smithi088.stderr:2024-06-03T08:36:32.024+0000 7fd64bfff640 1 -- 172.21.15.88:0/3008984679 <== mon.2 v1:172.21.15.88:6790/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 2234374065 0 0) 0x7fd6500015b0 con 0x7fd65406f070 2024-06-03T08:36:32.040 INFO:teuthology.orchestra.run.smithi088.stderr:2024-06-03T08:36:32.024+0000 7fd65b934640 5 monclient: authenticate success, global_id 4319 2024-06-03T08:36:32.040 INFO:teuthology.orchestra.run.smithi088.stderr:2024-06-03T08:36:32.024+0000 7fd65b934640 20 monclient: get_monmap_and_config waiting for monmap|config 2024-06-03T08:36:32.040 INFO:teuthology.orchestra.run.smithi088.stderr:2024-06-03T08:36:32.024+0000 7fd64bfff640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-06-03T08:36:32.040 INFO:teuthology.orchestra.run.smithi088.stderr:2024-06-03T08:36:32.024+0000 7fd64bfff640 10 monclient: got monmap 1 from mon.noname-b (according to old e1) 2024-06-03T08:36:32.040 INFO:teuthology.orchestra.run.smithi088.stderr:2024-06-03T08:36:32.024+0000 7fd64bfff640 10 monclient: dump: 2024-06-03T08:36:32.040 INFO:teuthology.orchestra.run.smithi088.stderr:epoch 1 2024-06-03T08:36:32.040 INFO:teuthology.orchestra.run.smithi088.stderr:fsid ec3c3fdd-3ac6-4d7d-9412-683130ed8b04 2024-06-03T08:36:32.041 INFO:teuthology.orchestra.run.smithi088.stderr:last_changed 2024-06-03T08:34:31.885125+0000 2024-06-03T08:36:32.041 INFO:teuthology.orchestra.run.smithi088.stderr:created 2024-06-03T08:34:31.885125+0000 2024-06-03T08:36:32.041 INFO:teuthology.orchestra.run.smithi088.stderr:min_mon_release 18 (reef) 2024-06-03T08:36:32.041 INFO:teuthology.orchestra.run.smithi088.stderr:election_strategy: 3 2024-06-03T08:36:32.041 INFO:teuthology.orchestra.run.smithi088.stderr:0: v1:172.21.15.88:6789/0 mon.a 2024-06-03T08:36:32.041 INFO:teuthology.orchestra.run.smithi088.stderr:1: v1:172.21.15.192:6789/0 mon.b 2024-06-03T08:36:32.041 INFO:teuthology.orchestra.run.smithi088.stderr:2: v1:172.21.15.88:6790/0 mon.c 2024-06-03T08:36:32.041 INFO:teuthology.orchestra.run.smithi088.stderr: 2024-06-03T08:36:32.041 INFO:teuthology.orchestra.run.smithi088.stderr:2024-06-03T08:36:32.024+0000 7fd65b934640 20 monclient: get_monmap_and_config waiting for monmap|config 2024-06-03T08:36:32.042 INFO:teuthology.orchestra.run.smithi088.stderr:2024-06-03T08:36:32.024+0000 7f791cb92640 1 -- start start 2024-06-03T08:36:32.042 INFO:teuthology.orchestra.run.smithi088.stderr:2024-06-03T08:36:32.024+0000 7f791cb92640 10 monclient: init 2024-06-03T08:36:32.042 INFO:teuthology.orchestra.run.smithi088.stderr:2024-06-03T08:36:32.024+0000 7f791cb92640 10 monclient: _reopen_session rank -1 2024-06-03T08:36:32.042 INFO:teuthology.orchestra.run.smithi088.stderr:2024-06-03T08:36:32.025+0000 7f791cb92640 10 monclient: _add_conns ranks=[2,0,1] 2024-06-03T08:36:32.042 INFO:teuthology.orchestra.run.smithi088.stderr:2024-06-03T08:36:32.025+0000 7f791cb92640 10 monclient(hunting): picked mon.noname-b con 0x7f79180b43c0 addr v1:172.21.15.88:6790/0 2024-06-03T08:36:32.042 INFO:teuthology.orchestra.run.smithi088.stderr:2024-06-03T08:36:32.025+0000 7f791cb92640 10 monclient(hunting): picked mon.noname-a con 0x7f791813e6c0 addr v1:172.21.15.88:6789/0 2024-06-03T08:36:32.042 INFO:teuthology.orchestra.run.smithi088.stderr:2024-06-03T08:36:32.025+0000 7f791cb92640 10 monclient(hunting): picked mon.noname-c con 0x7f791814ae20 addr v1:172.21.15.192:6789/0 2024-06-03T08:36:32.043 INFO:teuthology.orchestra.run.smithi088.stderr:2024-06-03T08:36:32.025+0000 7f791cb92640 1 -- --> v1:172.21.15.88:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f79180a8ed0 con 0x7f791813e6c0 2024-06-03T08:36:32.043 INFO:teuthology.orchestra.run.smithi088.stderr:2024-06-03T08:36:32.025+0000 7f791cb92640 1 -- --> v1:172.21.15.192:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f79180aa420 con 0x7f791814ae20 2024-06-03T08:36:32.043 INFO:teuthology.orchestra.run.smithi088.stderr:2024-06-03T08:36:32.025+0000 7f791cb92640 1 -- --> v1:172.21.15.88:6790/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f79180aa690 con 0x7f79180b43c0 2024-06-03T08:36:32.043 INFO:teuthology.orchestra.run.smithi088.stderr:2024-06-03T08:36:32.025+0000 7f791cb92640 10 monclient(hunting): _renew_subs 2024-06-03T08:36:32.043 INFO:teuthology.orchestra.run.smithi088.stderr:2024-06-03T08:36:32.025+0000 7f791cb92640 10 monclient(hunting): authenticate will time out at 1247.835248s 2024-06-03T08:36:32.043 INFO:teuthology.orchestra.run.smithi088.stderr:2024-06-03T08:36:32.025+0000 7f4c2e7fc640 1 -- 172.21.15.88:0/795329302 <== mon.2 v1:172.21.15.88:6790/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 4133664870 0 0) 0x7f4c300a9f00 con 0x7f4c300b4060 2024-06-03T08:36:32.043 INFO:teuthology.orchestra.run.smithi088.stderr:2024-06-03T08:36:32.025+0000 7f4c2e7fc640 10 monclient(hunting): _negotiate 2024-06-03T08:36:32.043 INFO:teuthology.orchestra.run.smithi088.stderr:2024-06-03T08:36:32.025+0000 7f4c2e7fc640 10 monclient(hunting): _init_auth method 2 2024-06-03T08:36:32.043 INFO:teuthology.orchestra.run.smithi088.stderr:2024-06-03T08:36:32.025+0000 7f4c2e7fc640 10 monclient(hunting): _init_auth creating new auth 2024-06-03T08:36:32.043 INFO:teuthology.orchestra.run.smithi088.stderr:2024-06-03T08:36:32.025+0000 7f4c2e7fc640 10 monclient(hunting): my global_id is 4322 2024-06-03T08:36:32.044 INFO:teuthology.orchestra.run.smithi088.stderr:2024-06-03T08:36:32.025+0000 7f4c2e7fc640 1 -- 172.21.15.88:0/795329302 --> v1:172.21.15.88:6790/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f4c20003520 con 0x7f4c300b4060 2024-06-03T08:36:32.044 INFO:teuthology.orchestra.run.smithi088.stderr:2024-06-03T08:36:32.032+0000 7fa0ead9d640 1 --1- >> v1:172.21.15.192:6789/0 conn(0x7fa0ec0a8960 0x7fa0ec0a8d40 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.192:6789/0 says I am v1:172.21.15.88:52370/0 (socket says 172.21.15.88:52370) 2024-06-03T08:36:32.044 INFO:teuthology.orchestra.run.smithi088.stderr:2024-06-03T08:36:32.032+0000 7fa0ead9d640 1 -- 172.21.15.88:0/424874043 learned_addr learned my addr 172.21.15.88:0/424874043 (peer_addr_for_me v1:172.21.15.88:0/0) 2024-06-03T08:36:32.044 INFO:teuthology.orchestra.run.smithi088.stderr:2024-06-03T08:36:32.033+0000 7f9badaf2640 1 --1- >> v1:172.21.15.88:6790/0 conn(0x7f9ba80b43c0 0x7f9ba81415b0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.88:6790/0 says I am v1:172.21.15.88:54146/0 (socket says 172.21.15.88:54146) 2024-06-03T08:36:32.044 INFO:teuthology.orchestra.run.smithi088.stderr:2024-06-03T08:36:32.033+0000 7f9badaf2640 1 -- 172.21.15.88:0/3753766473 learned_addr learned my addr 172.21.15.88:0/3753766473 (peer_addr_for_me v1:172.21.15.88:0/0) 2024-06-03T08:36:32.046 INFO:teuthology.orchestra.run.smithi088.stderr:2024-06-03T08:36:32.034+0000 7ff4927fc640 1 -- 172.21.15.88:0/2655868321 <== mon.1 v1:172.21.15.192:6789/0 4 ==== config(0 keys) v1 ==== 4+0+0 (unknown 0 0 0) 0x7ff4840017d0 con 0x7ff49406f070 2024-06-03T08:36:32.046 INFO:teuthology.orchestra.run.smithi088.stderr:2024-06-03T08:36:32.034+0000 7ff4927fc640 10 monclient: handle_config config(0 keys) v1 2024-06-03T08:36:32.046 INFO:teuthology.orchestra.run.smithi088.stderr:2024-06-03T08:36:32.034+0000 7ff4927fc640 1 -- 172.21.15.88:0/2655868321 <== mon.1 v1:172.21.15.192:6789/0 5 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 2234374065 0 0) 0x7ff484001b60 con 0x7ff49406f070 2024-06-03T08:36:32.046 INFO:teuthology.orchestra.run.smithi088.stderr:2024-06-03T08:36:32.034+0000 7ff4927fc640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-06-03T08:36:32.046 INFO:teuthology.orchestra.run.smithi088.stderr:2024-06-03T08:36:32.034+0000 7ff4927fc640 10 monclient: got monmap 1 from mon.b (according to old e1) 2024-06-03T08:36:32.046 INFO:teuthology.orchestra.run.smithi088.stderr:2024-06-03T08:36:32.034+0000 7ff4927fc640 10 monclient: dump: 2024-06-03T08:36:32.046 INFO:teuthology.orchestra.run.smithi088.stderr:epoch 1 2024-06-03T08:36:32.046 INFO:teuthology.orchestra.run.smithi088.stderr:fsid ec3c3fdd-3ac6-4d7d-9412-683130ed8b04 2024-06-03T08:36:32.046 INFO:teuthology.orchestra.run.smithi088.stderr:last_changed 2024-06-03T08:34:31.885125+0000 2024-06-03T08:36:32.046 INFO:teuthology.orchestra.run.smithi088.stderr:created 2024-06-03T08:34:31.885125+0000 2024-06-03T08:36:32.046 INFO:teuthology.orchestra.run.smithi088.stderr:min_mon_release 18 (reef) 2024-06-03T08:36:32.047 INFO:teuthology.orchestra.run.smithi088.stderr:election_strategy: 3 2024-06-03T08:36:32.047 INFO:teuthology.orchestra.run.smithi088.stderr:0: v1:172.21.15.88:6789/0 mon.a 2024-06-03T08:36:32.047 INFO:teuthology.orchestra.run.smithi088.stderr:1: v1:172.21.15.192:6789/0 mon.b 2024-06-03T08:36:32.047 INFO:teuthology.orchestra.run.smithi088.stderr:2: v1:172.21.15.88:6790/0 mon.c 2024-06-03T08:36:32.047 INFO:teuthology.orchestra.run.smithi088.stderr: 2024-06-03T08:36:32.047 INFO:teuthology.orchestra.run.smithi088.stderr:2024-06-03T08:36:32.034+0000 7ff49a002640 10 monclient: get_monmap_and_config success 2024-06-03T08:36:32.047 INFO:teuthology.orchestra.run.smithi088.stderr:2024-06-03T08:36:32.034+0000 7ff49a002640 4 set_mon_vals no callback set 2024-06-03T08:36:32.047 INFO:teuthology.orchestra.run.smithi088.stderr:2024-06-03T08:36:32.035+0000 7ff49a002640 10 monclient: shutdown 2024-06-03T08:36:32.047 INFO:teuthology.orchestra.run.smithi088.stderr:2024-06-03T08:36:32.035+0000 7ff49a002640 20 monclient: shutdown discarding 0 pending message(s) 2024-06-03T08:36:32.047 INFO:teuthology.orchestra.run.smithi088.stderr:2024-06-03T08:36:32.035+0000 7ff49a002640 1 -- 172.21.15.88:0/2655868321 >> v1:172.21.15.192:6789/0 conn(0x7ff49406f070 legacy=0x7ff4940aded0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-06-03T08:36:32.047 INFO:teuthology.orchestra.run.smithi088.stderr:2024-06-03T08:36:32.035+0000 7ff49a002640 1 -- 172.21.15.88:0/2655868321 shutdown_connections 2024-06-03T08:36:32.047 INFO:teuthology.orchestra.run.smithi088.stderr:2024-06-03T08:36:32.035+0000 7ff49a002640 1 -- 172.21.15.88:0/2655868321 >> 172.21.15.88:0/2655868321 conn(0x7ff4940b2f20 msgr2=0x7ff4940b3300 unknown :-1 s=STATE_NONE l=0).mark_down 2024-06-03T08:36:32.048 INFO:teuthology.orchestra.run.smithi088.stderr:2024-06-03T08:36:32.035+0000 7fd344d53640 1 -- 172.21.15.88:0/2256678336 <== mon.1 v1:172.21.15.192:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 214+0+0 (unknown 4247855655 0 0) 0x7fd32c003520 con 0x7fd3380327d0 2024-06-03T08:36:32.048 INFO:teuthology.orchestra.run.smithi088.stderr:2024-06-03T08:36:32.035+0000 7fd344d53640 1 -- 172.21.15.88:0/2256678336 >> v1:172.21.15.88:6790/0 conn(0x7fd338006410 legacy=0x7fd3380c6ea0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-06-03T08:36:32.048 INFO:teuthology.orchestra.run.smithi088.stderr:2024-06-03T08:36:32.035+0000 7fd344d53640 1 -- 172.21.15.88:0/2256678336 >> v1:172.21.15.88:6789/0 conn(0x7fd33802a640 legacy=0x7fd3380320c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-06-03T08:36:32.048 INFO:teuthology.orchestra.run.smithi088.stderr:2024-06-03T08:36:32.035+0000 7fd344d53640 10 monclient: _finish_hunting 0 2024-06-03T08:36:32.049 INFO:teuthology.orchestra.run.smithi088.stderr:2024-06-03T08:36:32.035+0000 7fd344d53640 1 monclient: found mon.noname-c 2024-06-03T08:36:32.049 INFO:teuthology.orchestra.run.smithi088.stderr:2024-06-03T08:36:32.035+0000 7fd344d53640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-06-03T08:36:32.049 INFO:teuthology.orchestra.run.smithi088.stderr:2024-06-03T08:36:32.035+0000 7fd344d53640 10 monclient: _send_mon_message to mon.noname-c at v1:172.21.15.192:6789/0 2024-06-03T08:36:32.049 INFO:teuthology.orchestra.run.smithi088.stderr:2024-06-03T08:36:32.035+0000 7fd344d53640 1 -- 172.21.15.88:0/2256678336 --> v1:172.21.15.192:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fd338033110 con 0x7fd3380327d0 2024-06-03T08:36:32.049 INFO:teuthology.orchestra.run.smithi088.stderr:2024-06-03T08:36:32.035+0000 7fd344d53640 10 monclient: _finish_auth 0 2024-06-03T08:36:32.049 INFO:teuthology.orchestra.run.smithi088.stderr:2024-06-03T08:36:32.035+0000 7ff49a002640 1 -- 172.21.15.88:0/2655868321 shutdown_connections 2024-06-03T08:36:32.050 INFO:teuthology.orchestra.run.smithi088.stderr:2024-06-03T08:36:32.035+0000 7fd344d53640 10 monclient: _check_auth_tickets 2024-06-03T08:36:32.050 INFO:teuthology.orchestra.run.smithi088.stderr:2024-06-03T08:36:32.035+0000 7fd344d53640 20 monclient: _check_auth_rotating not needed by client.admin 2024-06-03T08:36:32.050 INFO:teuthology.orchestra.run.smithi088.stderr:2024-06-03T08:36:32.035+0000 7fd344d53640 1 -- 172.21.15.88:0/2256678336 <== mon.1 v1:172.21.15.192:6789/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 2234374065 0 0) 0x7fd3400693d0 con 0x7fd3380327d0 2024-06-03T08:36:32.050 INFO:teuthology.orchestra.run.smithi088.stderr:2024-06-03T08:36:32.035+0000 7fd344d53640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-06-03T08:36:32.050 INFO:teuthology.orchestra.run.smithi088.stderr:2024-06-03T08:36:32.035+0000 7fd344d53640 10 monclient: got monmap 1 from mon.noname-c (according to old e1) 2024-06-03T08:36:32.050 INFO:teuthology.orchestra.run.smithi088.stderr:2024-06-03T08:36:32.035+0000 7fd344d53640 10 monclient: dump: 2024-06-03T08:36:32.050 INFO:teuthology.orchestra.run.smithi088.stderr:epoch 1 2024-06-03T08:36:32.050 INFO:teuthology.orchestra.run.smithi088.stderr:fsid ec3c3fdd-3ac6-4d7d-9412-683130ed8b04 2024-06-03T08:36:32.050 INFO:teuthology.orchestra.run.smithi088.stderr:last_changed 2024-06-03T08:34:31.885125+0000 2024-06-03T08:36:32.050 INFO:teuthology.orchestra.run.smithi088.stderr:created 2024-06-03T08:34:31.885125+0000 2024-06-03T08:36:32.050 INFO:teuthology.orchestra.run.smithi088.stderr:min_mon_release 18 (reef) 2024-06-03T08:36:32.051 INFO:teuthology.orchestra.run.smithi088.stderr:election_strategy: 3 2024-06-03T08:36:32.051 INFO:teuthology.orchestra.run.smithi088.stderr:0: v1:172.21.15.88:6789/0 mon.a 2024-06-03T08:36:32.051 INFO:teuthology.orchestra.run.smithi088.stderr:1: v1:172.21.15.192:6789/0 mon.b 2024-06-03T08:36:32.051 INFO:teuthology.orchestra.run.smithi088.stderr:2: v1:172.21.15.88:6790/0 mon.c 2024-06-03T08:36:32.051 INFO:teuthology.orchestra.run.smithi088.stderr: 2024-06-03T08:36:32.051 INFO:teuthology.orchestra.run.smithi088.stderr:2024-06-03T08:36:32.036+0000 7fd347fb9640 5 monclient: authenticate success, global_id 4306 2024-06-03T08:36:32.051 INFO:teuthology.orchestra.run.smithi088.stderr:2024-06-03T08:36:32.036+0000 7fd347fb9640 20 monclient: get_monmap_and_config waiting for monmap|config 2024-06-03T08:36:32.051 INFO:teuthology.orchestra.run.smithi088.stderr:2024-06-03T08:36:32.036+0000 7ff49a002640 1 -- 172.21.15.88:0/2655868321 wait complete. 2024-06-03T08:36:32.052 INFO:teuthology.orchestra.run.smithi088.stderr:2024-06-03T08:36:32.036+0000 7f4c2e7fc640 1 -- 172.21.15.88:0/795329302 <== mon.0 v1:172.21.15.88:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 4053226866 0 0) 0x7f4c300a8ca0 con 0x7f4c3013ee50 2024-06-03T08:36:32.052 INFO:teuthology.orchestra.run.smithi088.stderr:2024-06-03T08:36:32.036+0000 7f4c2e7fc640 10 monclient(hunting): _negotiate 2024-06-03T08:36:32.052 INFO:teuthology.orchestra.run.smithi088.stderr:2024-06-03T08:36:32.036+0000 7f4c2e7fc640 10 monclient(hunting): _init_auth method 2 2024-06-03T08:36:32.052 INFO:teuthology.orchestra.run.smithi088.stderr:2024-06-03T08:36:32.036+0000 7f4c2e7fc640 10 monclient(hunting): _init_auth creating new auth 2024-06-03T08:36:32.052 INFO:teuthology.orchestra.run.smithi088.stderr:2024-06-03T08:36:32.036+0000 7f4c2e7fc640 10 monclient(hunting): my global_id is 4305 2024-06-03T08:36:32.052 INFO:teuthology.orchestra.run.smithi088.stderr:2024-06-03T08:36:32.036+0000 7f4c2e7fc640 1 -- 172.21.15.88:0/795329302 --> v1:172.21.15.88:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f4c300a9f00 con 0x7f4c3013ee50 2024-06-03T08:36:32.053 INFO:teuthology.orchestra.run.smithi088.stderr:2024-06-03T08:36:32.036+0000 7ff49a002640 10 monclient: build_initial_monmap 2024-06-03T08:36:32.054 INFO:teuthology.orchestra.run.smithi088.stderr:2024-06-03T08:36:32.036+0000 7ff49a002640 10 monclient: monmap: 2024-06-03T08:36:32.054 INFO:teuthology.orchestra.run.smithi088.stderr:epoch 0 2024-06-03T08:36:32.054 INFO:teuthology.orchestra.run.smithi088.stderr:fsid 00000000-0000-0000-0000-000000000000 2024-06-03T08:36:32.054 INFO:teuthology.orchestra.run.smithi088.stderr:last_changed 0.000000 2024-06-03T08:36:32.054 INFO:teuthology.orchestra.run.smithi088.stderr:created 0.000000 2024-06-03T08:36:32.054 INFO:teuthology.orchestra.run.smithi088.stderr:min_mon_release 0 (unknown) 2024-06-03T08:36:32.054 INFO:teuthology.orchestra.run.smithi088.stderr:election_strategy: 1 2024-06-03T08:36:32.054 INFO:teuthology.orchestra.run.smithi088.stderr:0: v1:172.21.15.88:6789/0 mon.noname-a 2024-06-03T08:36:32.054 INFO:teuthology.orchestra.run.smithi088.stderr:1: v1:172.21.15.192:6789/0 mon.noname-b 2024-06-03T08:36:32.054 INFO:teuthology.orchestra.run.smithi088.stderr:2: v1:172.21.15.88:6790/0 mon.noname-c 2024-06-03T08:36:32.054 INFO:teuthology.orchestra.run.smithi088.stderr: 2024-06-03T08:36:32.054 INFO:teuthology.orchestra.run.smithi088.stderr:2024-06-03T08:36:32.037+0000 7ff49a002640 1 librados: starting msgr at 2024-06-03T08:36:32.054 INFO:teuthology.orchestra.run.smithi088.stderr:2024-06-03T08:36:32.037+0000 7ff49a002640 1 librados: starting objecter 2024-06-03T08:36:32.055 INFO:teuthology.orchestra.run.smithi088.stderr:2024-06-03T08:36:32.037+0000 7ff49a002640 1 Processor -- start 2024-06-03T08:36:32.055 INFO:teuthology.orchestra.run.smithi088.stderr:2024-06-03T08:36:32.037+0000 7ff49a002640 1 -- start start 2024-06-03T08:36:32.055 INFO:teuthology.orchestra.run.smithi088.stderr:2024-06-03T08:36:32.037+0000 7ff49a002640 1 librados: setting wanted keys 2024-06-03T08:36:32.055 INFO:teuthology.orchestra.run.smithi088.stderr:2024-06-03T08:36:32.037+0000 7ff49a002640 1 librados: calling monclient init 2024-06-03T08:36:32.055 INFO:teuthology.orchestra.run.smithi088.stderr:2024-06-03T08:36:32.037+0000 7ff49a002640 10 monclient: init 2024-06-03T08:36:32.055 INFO:teuthology.orchestra.run.smithi088.stderr:2024-06-03T08:36:32.038+0000 7ff49a002640 10 monclient: _reopen_session rank -1 2024-06-03T08:36:32.055 INFO:teuthology.orchestra.run.smithi088.stderr:2024-06-03T08:36:32.038+0000 7ff49a002640 10 monclient: _add_conns ranks=[1,2,0] 2024-06-03T08:36:32.055 INFO:teuthology.orchestra.run.smithi088.stderr:2024-06-03T08:36:32.038+0000 7ff49a002640 10 monclient(hunting): picked mon.noname-b con 0x7ff49406dc10 addr v1:172.21.15.192:6789/0 2024-06-03T08:36:32.055 INFO:teuthology.orchestra.run.smithi088.stderr:2024-06-03T08:36:32.038+0000 7ff49a002640 10 monclient(hunting): picked mon.noname-c con 0x7ff494141540 addr v1:172.21.15.88:6790/0 2024-06-03T08:36:32.055 INFO:teuthology.orchestra.run.smithi088.stderr:2024-06-03T08:36:32.038+0000 7ff49a002640 10 monclient(hunting): picked mon.noname-a con 0x7ff49412fd80 addr v1:172.21.15.88:6789/0 2024-06-03T08:36:32.056 INFO:teuthology.orchestra.run.smithi088.stderr:2024-06-03T08:36:32.038+0000 7fa0e9d9b640 1 -- 172.21.15.88:0/424874043 <== mon.1 v1:172.21.15.192:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 733160555 0 0) 0x7fa0ec0b3930 con 0x7fa0ec0a8960 2024-06-03T08:36:32.056 INFO:teuthology.orchestra.run.smithi088.stderr:2024-06-03T08:36:32.038+0000 7fa0e9d9b640 10 monclient(hunting): _negotiate 2024-06-03T08:36:32.056 INFO:teuthology.orchestra.run.smithi088.stderr:2024-06-03T08:36:32.038+0000 7fa0e9d9b640 10 monclient(hunting): _init_auth method 2 2024-06-03T08:36:32.056 INFO:teuthology.orchestra.run.smithi088.stderr:2024-06-03T08:36:32.038+0000 7fa0e9d9b640 10 monclient(hunting): _init_auth creating new auth 2024-06-03T08:36:32.056 INFO:teuthology.orchestra.run.smithi088.stderr:2024-06-03T08:36:32.038+0000 7fa0e9d9b640 10 monclient(hunting): my global_id is 4309 2024-06-03T08:36:32.056 INFO:teuthology.orchestra.run.smithi088.stderr:2024-06-03T08:36:32.038+0000 7fa0e9d9b640 1 -- 172.21.15.88:0/424874043 --> v1:172.21.15.192:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7fa0d4003520 con 0x7fa0ec0a8960 2024-06-03T08:36:32.057 INFO:teuthology.orchestra.run.smithi088.stderr:2024-06-03T08:36:32.038+0000 7ff49a002640 1 -- --> v1:172.21.15.88:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7ff49412ab90 con 0x7ff49412fd80 2024-06-03T08:36:32.057 INFO:teuthology.orchestra.run.smithi088.stderr:2024-06-03T08:36:32.038+0000 7ff49a002640 1 -- --> v1:172.21.15.192:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7ff494130f50 con 0x7ff49406dc10 2024-06-03T08:36:32.057 INFO:teuthology.orchestra.run.smithi088.stderr:2024-06-03T08:36:32.039+0000 7ff49a002640 1 -- --> v1:172.21.15.88:6790/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7ff49412bb80 con 0x7ff494141540 2024-06-03T08:36:32.057 INFO:teuthology.orchestra.run.smithi088.stderr:2024-06-03T08:36:32.039+0000 7ff49a002640 10 monclient(hunting): _renew_subs 2024-06-03T08:36:32.057 INFO:teuthology.orchestra.run.smithi088.stderr:2024-06-03T08:36:32.039+0000 7ff49a002640 10 monclient(hunting): authenticate will time out at 1247.849014s 2024-06-03T08:36:32.057 INFO:teuthology.orchestra.run.smithi088.stderr:2024-06-03T08:36:32.040+0000 7f791659c640 1 --1- >> v1:172.21.15.88:6790/0 conn(0x7f79180b43c0 0x7f79181415b0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.88:6790/0 says I am v1:172.21.15.88:54162/0 (socket says 172.21.15.88:54162) 2024-06-03T08:36:32.058 INFO:teuthology.orchestra.run.smithi088.stderr:2024-06-03T08:36:32.040+0000 7f791659c640 1 -- 172.21.15.88:0/1994185464 learned_addr learned my addr 172.21.15.88:0/1994185464 (peer_addr_for_me v1:172.21.15.88:0/0) 2024-06-03T08:36:32.058 INFO:teuthology.orchestra.run.smithi088.stderr:2024-06-03T08:36:32.041+0000 7f4c2e7fc640 1 -- 172.21.15.88:0/795329302 <== mon.2 v1:172.21.15.88:6790/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 214+0+0 (unknown 685576362 0 0) 0x7f4c20003520 con 0x7f4c300b4060 2024-06-03T08:36:32.059 INFO:teuthology.orchestra.run.smithi088.stderr:2024-06-03T08:36:32.041+0000 7f4c2e7fc640 1 -- 172.21.15.88:0/795329302 >> v1:172.21.15.192:6789/0 conn(0x7f4c300a86f0 legacy=0x7f4c3014ccf0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-06-03T08:36:32.059 INFO:teuthology.orchestra.run.smithi088.stderr:2024-06-03T08:36:32.041+0000 7f4c2e7fc640 1 -- 172.21.15.88:0/795329302 >> v1:172.21.15.88:6789/0 conn(0x7f4c3013ee50 legacy=0x7f4c301449b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-06-03T08:36:32.059 INFO:teuthology.orchestra.run.smithi088.stderr:2024-06-03T08:36:32.041+0000 7f4c2e7fc640 10 monclient: _finish_hunting 0 2024-06-03T08:36:32.059 INFO:teuthology.orchestra.run.smithi088.stderr:2024-06-03T08:36:32.041+0000 7f4c2e7fc640 1 monclient: found mon.noname-b 2024-06-03T08:36:32.059 INFO:teuthology.orchestra.run.smithi088.stderr:2024-06-03T08:36:32.041+0000 7f4c2e7fc640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-06-03T08:36:32.059 INFO:teuthology.orchestra.run.smithi088.stderr:2024-06-03T08:36:32.041+0000 7f4c2e7fc640 10 monclient: _send_mon_message to mon.noname-b at v1:172.21.15.88:6790/0 2024-06-03T08:36:32.059 INFO:teuthology.orchestra.run.smithi088.stderr:2024-06-03T08:36:32.041+0000 7f4c2e7fc640 1 -- 172.21.15.88:0/795329302 --> v1:172.21.15.88:6790/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f4c300aa1e0 con 0x7f4c300b4060 2024-06-03T08:36:32.059 INFO:teuthology.orchestra.run.smithi088.stderr:2024-06-03T08:36:32.041+0000 7f4c2e7fc640 10 monclient: _finish_auth 0 2024-06-03T08:36:32.059 INFO:teuthology.orchestra.run.smithi088.stderr:2024-06-03T08:36:32.041+0000 7f4c2e7fc640 10 monclient: _check_auth_tickets 2024-06-03T08:36:32.059 INFO:teuthology.orchestra.run.smithi088.stderr:2024-06-03T08:36:32.041+0000 7f4c2e7fc640 20 monclient: _check_auth_rotating not needed by client.admin 2024-06-03T08:36:32.059 INFO:teuthology.orchestra.run.smithi088.stderr:2024-06-03T08:36:32.041+0000 7f4c2e7fc640 1 -- 172.21.15.88:0/795329302 <== mon.2 v1:172.21.15.88:6790/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 2234374065 0 0) 0x7f4c24001610 con 0x7f4c300b4060 2024-06-03T08:36:32.060 INFO:teuthology.orchestra.run.smithi088.stderr:2024-06-03T08:36:32.041+0000 7f4c36155640 5 monclient: authenticate success, global_id 4322 2024-06-03T08:36:32.060 INFO:teuthology.orchestra.run.smithi088.stderr:2024-06-03T08:36:32.041+0000 7f4c36155640 20 monclient: get_monmap_and_config waiting for monmap|config 2024-06-03T08:36:32.060 INFO:teuthology.orchestra.run.smithi088.stderr:2024-06-03T08:36:32.041+0000 7f4c2e7fc640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-06-03T08:36:32.060 INFO:teuthology.orchestra.run.smithi088.stderr:2024-06-03T08:36:32.041+0000 7f4c2e7fc640 10 monclient: got monmap 1 from mon.noname-b (according to old e1) 2024-06-03T08:36:32.060 INFO:teuthology.orchestra.run.smithi088.stderr:2024-06-03T08:36:32.041+0000 7f4c2e7fc640 10 monclient: dump: 2024-06-03T08:36:32.060 INFO:teuthology.orchestra.run.smithi088.stderr:epoch 1 2024-06-03T08:36:32.060 INFO:teuthology.orchestra.run.smithi088.stderr:fsid ec3c3fdd-3ac6-4d7d-9412-683130ed8b04 2024-06-03T08:36:32.060 INFO:teuthology.orchestra.run.smithi088.stderr:last_changed 2024-06-03T08:34:31.885125+0000 2024-06-03T08:36:32.060 INFO:teuthology.orchestra.run.smithi088.stderr:created 2024-06-03T08:34:31.885125+0000 2024-06-03T08:36:32.060 INFO:teuthology.orchestra.run.smithi088.stderr:min_mon_release 18 (reef) 2024-06-03T08:36:32.060 INFO:teuthology.orchestra.run.smithi088.stderr:election_strategy: 3 2024-06-03T08:36:32.060 INFO:teuthology.orchestra.run.smithi088.stderr:0: v1:172.21.15.88:6789/0 mon.a 2024-06-03T08:36:32.060 INFO:teuthology.orchestra.run.smithi088.stderr:1: v1:172.21.15.192:6789/0 mon.b 2024-06-03T08:36:32.061 INFO:teuthology.orchestra.run.smithi088.stderr:2: v1:172.21.15.88:6790/0 mon.c 2024-06-03T08:36:32.061 INFO:teuthology.orchestra.run.smithi088.stderr: 2024-06-03T08:36:32.061 INFO:teuthology.orchestra.run.smithi088.stderr:2024-06-03T08:36:32.041+0000 7f4c36155640 20 monclient: get_monmap_and_config waiting for monmap|config 2024-06-03T08:36:32.061 INFO:teuthology.orchestra.run.smithi088.stderr:2024-06-03T08:36:32.042+0000 7f9bacaf0640 1 -- 172.21.15.88:0/3753766473 <== mon.2 v1:172.21.15.88:6790/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 290377424 0 0) 0x7f9ba80aa690 con 0x7f9ba80b43c0 2024-06-03T08:36:32.061 INFO:teuthology.orchestra.run.smithi088.stderr:2024-06-03T08:36:32.042+0000 7f9bacaf0640 10 monclient(hunting): _negotiate 2024-06-03T08:36:32.061 INFO:teuthology.orchestra.run.smithi088.stderr:2024-06-03T08:36:32.042+0000 7f9bacaf0640 10 monclient(hunting): _init_auth method 2 2024-06-03T08:36:32.061 INFO:teuthology.orchestra.run.smithi088.stderr:2024-06-03T08:36:32.042+0000 7f9bacaf0640 10 monclient(hunting): _init_auth creating new auth 2024-06-03T08:36:32.061 INFO:teuthology.orchestra.run.smithi088.stderr:2024-06-03T08:36:32.042+0000 7f9bacaf0640 10 monclient(hunting): my global_id is 4331 2024-06-03T08:36:32.062 INFO:teuthology.orchestra.run.smithi088.stderr:2024-06-03T08:36:32.042+0000 7f9bacaf0640 1 -- 172.21.15.88:0/3753766473 --> v1:172.21.15.88:6790/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f9b94003520 con 0x7f9ba80b43c0 2024-06-03T08:36:32.062 INFO:teuthology.orchestra.run.smithi088.stderr:2024-06-03T08:36:32.044+0000 7fa0e9d9b640 1 -- 172.21.15.88:0/424874043 <== mon.0 v1:172.21.15.88:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 2698697106 0 0) 0x7fa0ec144c60 con 0x7fa0ec1450f0 2024-06-03T08:36:32.062 INFO:teuthology.orchestra.run.smithi088.stderr:2024-06-03T08:36:32.044+0000 7fa0e9d9b640 10 monclient(hunting): _negotiate 2024-06-03T08:36:32.062 INFO:teuthology.orchestra.run.smithi088.stderr:2024-06-03T08:36:32.044+0000 7fa0e9d9b640 10 monclient(hunting): _init_auth method 2 2024-06-03T08:36:32.062 INFO:teuthology.orchestra.run.smithi088.stderr:2024-06-03T08:36:32.044+0000 7fa0e9d9b640 10 monclient(hunting): _init_auth creating new auth 2024-06-03T08:36:32.063 INFO:teuthology.orchestra.run.smithi088.stderr:2024-06-03T08:36:32.044+0000 7fa0e9d9b640 10 monclient(hunting): my global_id is 4308 2024-06-03T08:36:32.063 INFO:teuthology.orchestra.run.smithi088.stderr:2024-06-03T08:36:32.044+0000 7fa0e9d9b640 1 -- 172.21.15.88:0/424874043 --> v1:172.21.15.88:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7fa0ec0b3930 con 0x7fa0ec1450f0 2024-06-03T08:36:32.063 INFO:teuthology.orchestra.run.smithi088.stderr:2024-06-03T08:36:32.045+0000 7fa0e9d9b640 1 -- 172.21.15.88:0/424874043 <== mon.2 v1:172.21.15.88:6790/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 2075363570 0 0) 0x7fa0ec0b1ad0 con 0x7fa0ec13eec0 2024-06-03T08:36:32.063 INFO:teuthology.orchestra.run.smithi088.stderr:2024-06-03T08:36:32.045+0000 7fa0e9d9b640 10 monclient(hunting): _negotiate 2024-06-03T08:36:32.063 INFO:teuthology.orchestra.run.smithi088.stderr:2024-06-03T08:36:32.045+0000 7fa0e9d9b640 10 monclient(hunting): _init_auth method 2 2024-06-03T08:36:32.063 INFO:teuthology.orchestra.run.smithi088.stderr:2024-06-03T08:36:32.045+0000 7fa0e9d9b640 10 monclient(hunting): _init_auth creating new auth 2024-06-03T08:36:32.063 INFO:teuthology.orchestra.run.smithi088.stderr:2024-06-03T08:36:32.045+0000 7fa0e9d9b640 10 monclient(hunting): my global_id is 4328 2024-06-03T08:36:32.063 INFO:teuthology.orchestra.run.smithi088.stderr:2024-06-03T08:36:32.045+0000 7fa0e9d9b640 1 -- 172.21.15.88:0/424874043 --> v1:172.21.15.88:6790/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7fa0ec144c60 con 0x7fa0ec13eec0 2024-06-03T08:36:32.063 INFO:teuthology.orchestra.run.smithi088.stderr:2024-06-03T08:36:32.046+0000 7ff493fff640 1 --1- >> v1:172.21.15.88:6789/0 conn(0x7ff49412fd80 0x7ff4941dd0f0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=1).handle_server_banner_and_identify peer v1:172.21.15.88:6789/0 says I am v1:172.21.15.88:60336/0 (socket says 172.21.15.88:60336) 2024-06-03T08:36:32.064 INFO:teuthology.orchestra.run.smithi088.stderr:2024-06-03T08:36:32.046+0000 7ff493fff640 1 -- 172.21.15.88:0/2378444824 learned_addr learned my addr 172.21.15.88:0/2378444824 (peer_addr_for_me v1:172.21.15.88:0/0) 2024-06-03T08:36:32.064 INFO:teuthology.orchestra.run.smithi088.stderr:2024-06-03T08:36:32.046+0000 7f4c2e7fc640 1 -- 172.21.15.88:0/795329302 <== mon.2 v1:172.21.15.88:6790/0 4 ==== config(0 keys) v1 ==== 4+0+0 (unknown 0 0 0) 0x7f4c24001830 con 0x7f4c300b4060 2024-06-03T08:36:32.064 INFO:teuthology.orchestra.run.smithi088.stderr:2024-06-03T08:36:32.046+0000 7f4c2e7fc640 10 monclient: handle_config config(0 keys) v1 2024-06-03T08:36:32.064 INFO:teuthology.orchestra.run.smithi088.stderr:2024-06-03T08:36:32.046+0000 7f4c2e7fc640 1 -- 172.21.15.88:0/795329302 <== mon.2 v1:172.21.15.88:6790/0 5 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 2234374065 0 0) 0x7f4c24001bc0 con 0x7f4c300b4060 2024-06-03T08:36:32.064 INFO:teuthology.orchestra.run.smithi088.stderr:2024-06-03T08:36:32.046+0000 7f4c36155640 10 monclient: get_monmap_and_config success 2024-06-03T08:36:32.064 INFO:teuthology.orchestra.run.smithi088.stderr:2024-06-03T08:36:32.046+0000 7f4c36155640 4 set_mon_vals no callback set 2024-06-03T08:36:32.064 INFO:teuthology.orchestra.run.smithi088.stderr:2024-06-03T08:36:32.046+0000 7f4c2e7fc640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-06-03T08:36:32.064 INFO:teuthology.orchestra.run.smithi088.stderr:2024-06-03T08:36:32.046+0000 7f4c2e7fc640 10 monclient: got monmap 1 from mon.c (according to old e1) 2024-06-03T08:36:32.065 INFO:teuthology.orchestra.run.smithi088.stderr:2024-06-03T08:36:32.046+0000 7f4c2e7fc640 10 monclient: dump: 2024-06-03T08:36:32.065 INFO:teuthology.orchestra.run.smithi088.stderr:epoch 1 2024-06-03T08:36:32.065 INFO:teuthology.orchestra.run.smithi088.stderr:fsid ec3c3fdd-3ac6-4d7d-9412-683130ed8b04 2024-06-03T08:36:32.065 INFO:teuthology.orchestra.run.smithi088.stderr:last_changed 2024-06-03T08:34:31.885125+0000 2024-06-03T08:36:32.065 INFO:teuthology.orchestra.run.smithi088.stderr:created 2024-06-03T08:34:31.885125+0000 2024-06-03T08:36:32.065 INFO:teuthology.orchestra.run.smithi088.stderr:min_mon_release 18 (reef) 2024-06-03T08:36:32.065 INFO:teuthology.orchestra.run.smithi088.stderr:election_strategy: 3 2024-06-03T08:36:32.065 INFO:teuthology.orchestra.run.smithi088.stderr:0: v1:172.21.15.88:6789/0 mon.a 2024-06-03T08:36:32.065 INFO:teuthology.orchestra.run.smithi088.stderr:1: v1:172.21.15.192:6789/0 mon.b 2024-06-03T08:36:32.065 INFO:teuthology.orchestra.run.smithi088.stderr:2: v1:172.21.15.88:6790/0 mon.c 2024-06-03T08:36:32.065 INFO:teuthology.orchestra.run.smithi088.stderr: 2024-06-03T08:36:32.066 INFO:teuthology.orchestra.run.smithi088.stderr:2024-06-03T08:36:32.046+0000 7fd64bfff640 1 -- 172.21.15.88:0/3008984679 <== mon.2 v1:172.21.15.88:6790/0 4 ==== config(0 keys) v1 ==== 4+0+0 (unknown 0 0 0) 0x7fd650001840 con 0x7fd65406f070 2024-06-03T08:36:32.066 INFO:teuthology.orchestra.run.smithi088.stderr:2024-06-03T08:36:32.046+0000 7fd64bfff640 10 monclient: handle_config config(0 keys) v1 2024-06-03T08:36:32.066 INFO:teuthology.orchestra.run.smithi088.stderr:2024-06-03T08:36:32.046+0000 7fd65b934640 10 monclient: get_monmap_and_config success 2024-06-03T08:36:32.066 INFO:teuthology.orchestra.run.smithi088.stderr:2024-06-03T08:36:32.046+0000 7fd65b934640 4 set_mon_vals no callback set 2024-06-03T08:36:32.066 INFO:teuthology.orchestra.run.smithi088.stderr:2024-06-03T08:36:32.047+0000 7fd64bfff640 1 -- 172.21.15.88:0/3008984679 <== mon.2 v1:172.21.15.88:6790/0 5 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 2234374065 0 0) 0x7fd650001bd0 con 0x7fd65406f070 2024-06-03T08:36:32.066 INFO:teuthology.orchestra.run.smithi088.stderr:2024-06-03T08:36:32.047+0000 7fd64bfff640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-06-03T08:36:32.066 INFO:teuthology.orchestra.run.smithi088.stderr:2024-06-03T08:36:32.047+0000 7fd64bfff640 10 monclient: got monmap 1 from mon.c (according to old e1) 2024-06-03T08:36:32.066 INFO:teuthology.orchestra.run.smithi088.stderr:2024-06-03T08:36:32.047+0000 7fd64bfff640 10 monclient: dump: 2024-06-03T08:36:32.067 INFO:teuthology.orchestra.run.smithi088.stderr:epoch 1 2024-06-03T08:36:32.067 INFO:teuthology.orchestra.run.smithi088.stderr:fsid ec3c3fdd-3ac6-4d7d-9412-683130ed8b04 2024-06-03T08:36:32.067 INFO:teuthology.orchestra.run.smithi088.stderr:last_changed 2024-06-03T08:34:31.885125+0000 2024-06-03T08:36:32.067 INFO:teuthology.orchestra.run.smithi088.stderr:created 2024-06-03T08:34:31.885125+0000 2024-06-03T08:36:32.067 INFO:teuthology.orchestra.run.smithi088.stderr:min_mon_release 18 (reef) 2024-06-03T08:36:32.067 INFO:teuthology.orchestra.run.smithi088.stderr:election_strategy: 3 2024-06-03T08:36:32.067 INFO:teuthology.orchestra.run.smithi088.stderr:0: v1:172.21.15.88:6789/0 mon.a 2024-06-03T08:36:32.067 INFO:teuthology.orchestra.run.smithi088.stderr:1: v1:172.21.15.192:6789/0 mon.b 2024-06-03T08:36:32.067 INFO:teuthology.orchestra.run.smithi088.stderr:2: v1:172.21.15.88:6790/0 mon.c 2024-06-03T08:36:32.067 INFO:teuthology.orchestra.run.smithi088.stderr: 2024-06-03T08:36:32.068 INFO:teuthology.orchestra.run.smithi088.stderr:2024-06-03T08:36:32.047+0000 7f4c36155640 10 monclient: shutdown 2024-06-03T08:36:32.068 INFO:teuthology.orchestra.run.smithi088.stderr:2024-06-03T08:36:32.047+0000 7f4c36155640 20 monclient: shutdown discarding 0 pending message(s) 2024-06-03T08:36:32.068 INFO:teuthology.orchestra.run.smithi088.stderr:2024-06-03T08:36:32.047+0000 7f4c36155640 1 -- 172.21.15.88:0/795329302 >> v1:172.21.15.88:6790/0 conn(0x7f4c300b4060 legacy=0x7f4c3013e740 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-06-03T08:36:32.068 INFO:teuthology.orchestra.run.smithi088.stderr:2024-06-03T08:36:32.047+0000 7f4c36155640 1 -- 172.21.15.88:0/795329302 shutdown_connections 2024-06-03T08:36:32.068 INFO:teuthology.orchestra.run.smithi088.stderr:2024-06-03T08:36:32.047+0000 7f4c36155640 1 -- 172.21.15.88:0/795329302 >> 172.21.15.88:0/795329302 conn(0x7f4c300717c0 msgr2=0x7f4c30071ba0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-06-03T08:36:32.068 INFO:teuthology.orchestra.run.smithi088.stderr:2024-06-03T08:36:32.047+0000 7f4c36155640 1 -- 172.21.15.88:0/795329302 shutdown_connections 2024-06-03T08:36:32.068 INFO:teuthology.orchestra.run.smithi088.stderr:2024-06-03T08:36:32.047+0000 7f4c36155640 1 -- 172.21.15.88:0/795329302 wait complete. 2024-06-03T08:36:32.069 INFO:teuthology.orchestra.run.smithi088.stderr:2024-06-03T08:36:32.047+0000 7fd65b934640 10 monclient: shutdown 2024-06-03T08:36:32.069 INFO:teuthology.orchestra.run.smithi088.stderr:2024-06-03T08:36:32.047+0000 7fd65b934640 20 monclient: shutdown discarding 0 pending message(s) 2024-06-03T08:36:32.069 INFO:teuthology.orchestra.run.smithi088.stderr:2024-06-03T08:36:32.047+0000 7fd65b934640 1 -- 172.21.15.88:0/3008984679 >> v1:172.21.15.88:6790/0 conn(0x7fd65406f070 legacy=0x7fd6540aded0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-06-03T08:36:32.069 INFO:teuthology.orchestra.run.smithi088.stderr:2024-06-03T08:36:32.047+0000 7fd65b934640 1 -- 172.21.15.88:0/3008984679 shutdown_connections 2024-06-03T08:36:32.069 INFO:teuthology.orchestra.run.smithi088.stderr:2024-06-03T08:36:32.047+0000 7fd65b934640 1 -- 172.21.15.88:0/3008984679 >> 172.21.15.88:0/3008984679 conn(0x7fd6540b2f20 msgr2=0x7fd6540b3300 unknown :-1 s=STATE_NONE l=0).mark_down 2024-06-03T08:36:32.069 INFO:teuthology.orchestra.run.smithi088.stderr:2024-06-03T08:36:32.047+0000 7f9bacaf0640 1 -- 172.21.15.88:0/3753766473 <== mon.0 v1:172.21.15.88:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 652313641 0 0) 0x7f9ba80a8ed0 con 0x7f9ba813e6c0 2024-06-03T08:36:32.069 INFO:teuthology.orchestra.run.smithi088.stderr:2024-06-03T08:36:32.047+0000 7f9bacaf0640 10 monclient(hunting): _negotiate 2024-06-03T08:36:32.069 INFO:teuthology.orchestra.run.smithi088.stderr:2024-06-03T08:36:32.047+0000 7f9bacaf0640 10 monclient(hunting): _init_auth method 2 2024-06-03T08:36:32.069 INFO:teuthology.orchestra.run.smithi088.stderr:2024-06-03T08:36:32.047+0000 7f9bacaf0640 10 monclient(hunting): _init_auth creating new auth 2024-06-03T08:36:32.070 INFO:teuthology.orchestra.run.smithi088.stderr:2024-06-03T08:36:32.047+0000 7f9bacaf0640 10 monclient(hunting): my global_id is 4311 2024-06-03T08:36:32.070 INFO:teuthology.orchestra.run.smithi088.stderr:2024-06-03T08:36:32.047+0000 7f9bacaf0640 1 -- 172.21.15.88:0/3753766473 --> v1:172.21.15.88:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f9ba80aa690 con 0x7f9ba813e6c0 2024-06-03T08:36:32.070 INFO:teuthology.orchestra.run.smithi088.stderr:2024-06-03T08:36:32.047+0000 7f1983fff640 1 -- 172.21.15.88:0/1438849704 <== mon.1 v1:172.21.15.192:6789/0 5 ==== config(0 keys) v1 ==== 4+0+0 (unknown 0 0 0) 0x7f1984003610 con 0x7f19900327d0 2024-06-03T08:36:32.070 INFO:teuthology.orchestra.run.smithi088.stderr:2024-06-03T08:36:32.047+0000 7f1983fff640 10 monclient: handle_config config(0 keys) v1 2024-06-03T08:36:32.070 INFO:teuthology.orchestra.run.smithi088.stderr:2024-06-03T08:36:32.047+0000 7f1983fff640 1 -- 172.21.15.88:0/1438849704 <== mon.1 v1:172.21.15.192:6789/0 6 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 2234374065 0 0) 0x7f1984001d80 con 0x7f19900327d0 2024-06-03T08:36:32.070 INFO:teuthology.orchestra.run.smithi088.stderr:2024-06-03T08:36:32.047+0000 7f1983fff640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-06-03T08:36:32.070 INFO:teuthology.orchestra.run.smithi088.stderr:2024-06-03T08:36:32.047+0000 7fd65b934640 1 -- 172.21.15.88:0/3008984679 shutdown_connections 2024-06-03T08:36:32.071 INFO:teuthology.orchestra.run.smithi088.stderr:2024-06-03T08:36:32.047+0000 7fd65b934640 1 -- 172.21.15.88:0/3008984679 wait complete. 2024-06-03T08:36:32.071 INFO:teuthology.orchestra.run.smithi088.stderr:2024-06-03T08:36:32.047+0000 7f1983fff640 10 monclient: got monmap 1 from mon.b (according to old e1) 2024-06-03T08:36:32.071 INFO:teuthology.orchestra.run.smithi088.stderr:2024-06-03T08:36:32.047+0000 7f1983fff640 10 monclient: dump: 2024-06-03T08:36:32.071 INFO:teuthology.orchestra.run.smithi088.stderr:epoch 1 2024-06-03T08:36:32.071 INFO:teuthology.orchestra.run.smithi088.stderr:fsid ec3c3fdd-3ac6-4d7d-9412-683130ed8b04 2024-06-03T08:36:32.071 INFO:teuthology.orchestra.run.smithi088.stderr:last_changed 2024-06-03T08:34:31.885125+0000 2024-06-03T08:36:32.071 INFO:teuthology.orchestra.run.smithi088.stderr:created 2024-06-03T08:34:31.885125+0000 2024-06-03T08:36:32.072 INFO:teuthology.orchestra.run.smithi088.stderr:min_mon_release 18 (reef) 2024-06-03T08:36:32.072 INFO:teuthology.orchestra.run.smithi088.stderr:election_strategy: 3 2024-06-03T08:36:32.072 INFO:teuthology.orchestra.run.smithi088.stderr:0: v1:172.21.15.88:6789/0 mon.a 2024-06-03T08:36:32.072 INFO:teuthology.orchestra.run.smithi088.stderr:1: v1:172.21.15.192:6789/0 mon.b 2024-06-03T08:36:32.072 INFO:teuthology.orchestra.run.smithi088.stderr:2: v1:172.21.15.88:6790/0 mon.c 2024-06-03T08:36:32.072 INFO:teuthology.orchestra.run.smithi088.stderr: 2024-06-03T08:36:32.072 INFO:teuthology.orchestra.run.smithi088.stderr:2024-06-03T08:36:32.047+0000 7f1995ffb640 4 set_mon_vals no callback set 2024-06-03T08:36:32.072 INFO:teuthology.orchestra.run.smithi088.stderr:2024-06-03T08:36:32.048+0000 7f4c36155640 10 monclient: build_initial_monmap 2024-06-03T08:36:32.073 INFO:teuthology.orchestra.run.smithi088.stderr:2024-06-03T08:36:32.048+0000 7ff47bfff640 10 client.?.objecter ms_handle_connect 0x7ff49412fd80 2024-06-03T08:36:32.073 INFO:teuthology.orchestra.run.smithi088.stderr:2024-06-03T08:36:32.048+0000 7ff47bfff640 10 client.?.objecter resend_mon_ops 2024-06-03T08:36:32.073 INFO:teuthology.orchestra.run.smithi088.stderr:2024-06-03T08:36:32.048+0000 7fd65b934640 10 monclient: build_initial_monmap 2024-06-03T08:36:32.073 INFO:teuthology.orchestra.run.smithi088.stderr:2024-06-03T08:36:32.048+0000 7fd65b934640 10 monclient: monmap: 2024-06-03T08:36:32.073 INFO:teuthology.orchestra.run.smithi088.stderr:epoch 0 2024-06-03T08:36:32.073 INFO:teuthology.orchestra.run.smithi088.stderr:fsid 00000000-0000-0000-0000-000000000000 2024-06-03T08:36:32.073 INFO:teuthology.orchestra.run.smithi088.stderr:last_changed 0.000000 2024-06-03T08:36:32.073 INFO:teuthology.orchestra.run.smithi088.stderr:created 0.000000 2024-06-03T08:36:32.073 INFO:teuthology.orchestra.run.smithi088.stderr:min_mon_release 0 (unknown) 2024-06-03T08:36:32.074 INFO:teuthology.orchestra.run.smithi088.stderr:election_strategy: 1 2024-06-03T08:36:32.074 INFO:teuthology.orchestra.run.smithi088.stderr:0: v1:172.21.15.88:6789/0 mon.noname-a 2024-06-03T08:36:32.074 INFO:teuthology.orchestra.run.smithi088.stderr:1: v1:172.21.15.192:6789/0 mon.noname-b 2024-06-03T08:36:32.074 INFO:teuthology.orchestra.run.smithi088.stderr:2: v1:172.21.15.88:6790/0 mon.noname-c 2024-06-03T08:36:32.074 INFO:teuthology.orchestra.run.smithi088.stderr: 2024-06-03T08:36:32.074 INFO:teuthology.orchestra.run.smithi088.stderr:2024-06-03T08:36:32.048+0000 7f4c36155640 10 monclient: monmap: 2024-06-03T08:36:32.074 INFO:teuthology.orchestra.run.smithi088.stderr:epoch 0 2024-06-03T08:36:32.074 INFO:teuthology.orchestra.run.smithi088.stderr:fsid 00000000-0000-0000-0000-000000000000 2024-06-03T08:36:32.074 INFO:teuthology.orchestra.run.smithi088.stderr:last_changed 0.000000 2024-06-03T08:36:32.074 INFO:teuthology.orchestra.run.smithi088.stderr:created 0.000000 2024-06-03T08:36:32.075 INFO:teuthology.orchestra.run.smithi088.stderr:min_mon_release 0 (unknown) 2024-06-03T08:36:32.075 INFO:teuthology.orchestra.run.smithi088.stderr:election_strategy: 1 2024-06-03T08:36:32.075 INFO:teuthology.orchestra.run.smithi088.stderr:0: v1:172.21.15.88:6789/0 mon.noname-a 2024-06-03T08:36:32.075 INFO:teuthology.orchestra.run.smithi088.stderr:1: v1:172.21.15.192:6789/0 mon.noname-b 2024-06-03T08:36:32.075 INFO:teuthology.orchestra.run.smithi088.stderr:2: v1:172.21.15.88:6790/0 mon.noname-c 2024-06-03T08:36:32.075 INFO:teuthology.orchestra.run.smithi088.stderr: 2024-06-03T08:36:32.075 INFO:teuthology.orchestra.run.smithi088.stderr:2024-06-03T08:36:32.049+0000 7f1983fff640 1 -- 172.21.15.88:0/1438849704 <== mon.1 v1:172.21.15.192:6789/0 7 ==== mgrmap(e 5) v1 ==== 89110+0+0 (unknown 873533973 0 0) 0x7f1984019540 con 0x7f19900327d0 2024-06-03T08:36:32.076 INFO:teuthology.orchestra.run.smithi088.stderr:2024-06-03T08:36:32.049+0000 7fd65b934640 1 librados: starting msgr at 2024-06-03T08:36:32.076 INFO:teuthology.orchestra.run.smithi088.stderr:2024-06-03T08:36:32.049+0000 7fd65b934640 1 librados: starting objecter 2024-06-03T08:36:32.076 INFO:teuthology.orchestra.run.smithi088.stderr:2024-06-03T08:36:32.049+0000 7fd65b934640 1 Processor -- start 2024-06-03T08:36:32.076 INFO:teuthology.orchestra.run.smithi088.stderr:2024-06-03T08:36:32.049+0000 7fd65b934640 1 -- start start 2024-06-03T08:36:32.076 INFO:teuthology.orchestra.run.smithi088.stderr:2024-06-03T08:36:32.049+0000 7fd65b934640 1 librados: setting wanted keys 2024-06-03T08:36:32.076 INFO:teuthology.orchestra.run.smithi088.stderr:2024-06-03T08:36:32.049+0000 7fd65b934640 1 librados: calling monclient init 2024-06-03T08:36:32.076 INFO:teuthology.orchestra.run.smithi088.stderr:2024-06-03T08:36:32.049+0000 7fd65b934640 10 monclient: init 2024-06-03T08:36:32.077 INFO:teuthology.orchestra.run.smithi088.stderr:2024-06-03T08:36:32.049+0000 7f4c36155640 1 librados: starting msgr at 2024-06-03T08:36:32.077 INFO:teuthology.orchestra.run.smithi088.stderr:2024-06-03T08:36:32.049+0000 7f4c36155640 1 librados: starting objecter 2024-06-03T08:36:32.077 INFO:teuthology.orchestra.run.smithi088.stderr:2024-06-03T08:36:32.049+0000 7f4c36155640 1 Processor -- start 2024-06-03T08:36:32.077 INFO:teuthology.orchestra.run.smithi088.stderr:2024-06-03T08:36:32.049+0000 7f4c36155640 1 -- start start 2024-06-03T08:36:32.077 INFO:teuthology.orchestra.run.smithi088.stderr:2024-06-03T08:36:32.049+0000 7f4c36155640 1 librados: setting wanted keys 2024-06-03T08:36:32.078 INFO:teuthology.orchestra.run.smithi088.stderr:2024-06-03T08:36:32.049+0000 7f4c36155640 1 librados: calling monclient init 2024-06-03T08:36:32.078 INFO:teuthology.orchestra.run.smithi088.stderr:2024-06-03T08:36:32.050+0000 7f4c36155640 10 monclient: init 2024-06-03T08:36:32.078 INFO:teuthology.orchestra.run.smithi088.stderr:2024-06-03T08:36:32.050+0000 7f19977fe640 10 monclient: get_auth_request con 0x7f197c06d250 auth_method 0 2024-06-03T08:36:32.079 INFO:teuthology.orchestra.run.smithi088.stderr:2024-06-03T08:36:32.050+0000 7fd65b934640 10 monclient: _reopen_session rank -1 2024-06-03T08:36:32.079 INFO:teuthology.orchestra.run.smithi088.stderr:2024-06-03T08:36:32.050+0000 7fd65b934640 10 monclient: _add_conns ranks=[0,2,1] 2024-06-03T08:36:32.079 INFO:teuthology.orchestra.run.smithi088.stderr:2024-06-03T08:36:32.050+0000 7fd65b934640 10 monclient(hunting): picked mon.noname-a con 0x7fd65406dc10 addr v1:172.21.15.88:6789/0 2024-06-03T08:36:32.079 INFO:teuthology.orchestra.run.smithi088.stderr:2024-06-03T08:36:32.050+0000 7fd65b934640 10 monclient(hunting): picked mon.noname-c con 0x7fd654141540 addr v1:172.21.15.88:6790/0 2024-06-03T08:36:32.079 INFO:teuthology.orchestra.run.smithi088.stderr:2024-06-03T08:36:32.050+0000 7fd65b934640 10 monclient(hunting): picked mon.noname-b con 0x7fd65413b700 addr v1:172.21.15.192:6789/0 2024-06-03T08:36:32.079 INFO:teuthology.orchestra.run.smithi088.stderr:2024-06-03T08:36:32.050+0000 7fd65b934640 1 -- --> v1:172.21.15.88:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fd65413bcb0 con 0x7fd65406dc10 2024-06-03T08:36:32.079 INFO:teuthology.orchestra.run.smithi088.stderr:2024-06-03T08:36:32.050+0000 7f1983fff640 10 client.4300.objecter ms_handle_connect 0x7f197c06d250 2024-06-03T08:36:32.080 INFO:teuthology.orchestra.run.smithi088.stderr:2024-06-03T08:36:32.050+0000 7fd65b934640 1 -- --> v1:172.21.15.192:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fd65413b4c0 con 0x7fd65413b700 2024-06-03T08:36:32.080 INFO:teuthology.orchestra.run.smithi088.stderr:2024-06-03T08:36:32.050+0000 7fd65b934640 1 -- --> v1:172.21.15.88:6790/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fd654133f70 con 0x7fd654141540 2024-06-03T08:36:32.080 INFO:teuthology.orchestra.run.smithi088.stderr:2024-06-03T08:36:32.050+0000 7fd65b934640 10 monclient(hunting): _renew_subs 2024-06-03T08:36:32.080 INFO:teuthology.orchestra.run.smithi088.stderr:2024-06-03T08:36:32.050+0000 7fd65b934640 10 monclient(hunting): authenticate will time out at 1247.860316s 2024-06-03T08:36:32.081 INFO:teuthology.orchestra.run.smithi088.stderr:2024-06-03T08:36:32.050+0000 7f4c36155640 10 monclient: _reopen_session rank -1 2024-06-03T08:36:32.081 INFO:teuthology.orchestra.run.smithi088.stderr:2024-06-03T08:36:32.050+0000 7f4c36155640 10 monclient: _add_conns ranks=[1,2,0] 2024-06-03T08:36:32.081 INFO:teuthology.orchestra.run.smithi088.stderr:2024-06-03T08:36:32.050+0000 7f4c36155640 10 monclient(hunting): picked mon.noname-b con 0x7f4c300a86f0 addr v1:172.21.15.192:6789/0 2024-06-03T08:36:32.081 INFO:teuthology.orchestra.run.smithi088.stderr:2024-06-03T08:36:32.050+0000 7f4c36155640 10 monclient(hunting): picked mon.noname-c con 0x7f4c300b4060 addr v1:172.21.15.88:6790/0 2024-06-03T08:36:32.081 INFO:teuthology.orchestra.run.smithi088.stderr:2024-06-03T08:36:32.050+0000 7f4c36155640 10 monclient(hunting): picked mon.noname-a con 0x7f4c3013ee50 addr v1:172.21.15.88:6789/0 2024-06-03T08:36:32.081 INFO:teuthology.orchestra.run.smithi088.stderr:2024-06-03T08:36:32.050+0000 7f4c36155640 1 -- --> v1:172.21.15.88:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f4c30140630 con 0x7f4c3013ee50 2024-06-03T08:36:32.081 INFO:teuthology.orchestra.run.smithi088.stderr:2024-06-03T08:36:32.050+0000 7f4c36155640 1 -- --> v1:172.21.15.192:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f4c3013e160 con 0x7f4c300a86f0 2024-06-03T08:36:32.082 INFO:teuthology.orchestra.run.smithi088.stderr:2024-06-03T08:36:32.051+0000 7f4c36155640 1 -- --> v1:172.21.15.88:6790/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f4c30141200 con 0x7f4c300b4060 2024-06-03T08:36:32.082 INFO:teuthology.orchestra.run.smithi088.stderr:2024-06-03T08:36:32.051+0000 7f4c36155640 10 monclient(hunting): _renew_subs 2024-06-03T08:36:32.082 INFO:teuthology.orchestra.run.smithi088.stderr:2024-06-03T08:36:32.051+0000 7f1983fff640 1 -- 172.21.15.88:0/1438849704 <== mon.1 v1:172.21.15.192:6789/0 8 ==== osd_map(26..26 src has 1..26) v4 ==== 4090+0+0 (unknown 3637450554 0 0) 0x7f1984052d80 con 0x7f19900327d0 2024-06-03T08:36:32.082 INFO:teuthology.orchestra.run.smithi088.stderr:2024-06-03T08:36:32.051+0000 7f4c36155640 10 monclient(hunting): authenticate will time out at 1247.860994s 2024-06-03T08:36:32.088 INFO:teuthology.orchestra.run.smithi088.stderr:2024-06-03T08:36:32.051+0000 7f1983fff640 10 client.4300.objecter ms_dispatch 0x7f1998000b90 osd_map(26..26 src has 1..26) v4 2024-06-03T08:36:32.088 INFO:teuthology.orchestra.run.smithi088.stderr:2024-06-03T08:36:32.051+0000 7f1983fff640 3 client.4300.objecter handle_osd_map got epochs [26,26] > 0 2024-06-03T08:36:32.088 INFO:teuthology.orchestra.run.smithi088.stderr:2024-06-03T08:36:32.051+0000 7f1983fff640 3 client.4300.objecter handle_osd_map decoding full epoch 26 2024-06-03T08:36:32.088 INFO:teuthology.orchestra.run.smithi088.stderr:2024-06-03T08:36:32.051+0000 7f1983fff640 10 client.4300.objecter checking command 1 2024-06-03T08:36:32.088 INFO:teuthology.orchestra.run.smithi088.stderr:2024-06-03T08:36:32.051+0000 7f1983fff640 20 client.4300.objecter _get_session s=0x7f197c073bd0 osd=2 3 2024-06-03T08:36:32.088 INFO:teuthology.orchestra.run.smithi088.stderr:2024-06-03T08:36:32.051+0000 7f1983fff640 20 client.4300.objecter put_session s=0x7f197c073bd0 osd=2 3 2024-06-03T08:36:32.088 INFO:teuthology.orchestra.run.smithi088.stderr:2024-06-03T08:36:32.051+0000 7f1983fff640 15 client.4300.objecter _session_command_op_remove -1 1 2024-06-03T08:36:32.088 INFO:teuthology.orchestra.run.smithi088.stderr:2024-06-03T08:36:32.051+0000 7f1983fff640 20 client.4300.objecter _get_session s=0x7f197c073bd0 osd=2 3 2024-06-03T08:36:32.088 INFO:teuthology.orchestra.run.smithi088.stderr:2024-06-03T08:36:32.051+0000 7f1983fff640 20 client.4300.objecter get_session s=0x7f197c073bd0 osd=2 3 2024-06-03T08:36:32.088 INFO:teuthology.orchestra.run.smithi088.stderr:2024-06-03T08:36:32.051+0000 7f1983fff640 15 client.4300.objecter _session_command_op_assign 2 1 2024-06-03T08:36:32.088 INFO:teuthology.orchestra.run.smithi088.stderr:2024-06-03T08:36:32.051+0000 7f1983fff640 20 client.4300.objecter put_session s=0x7f197c073bd0 osd=2 4 2024-06-03T08:36:32.089 INFO:teuthology.orchestra.run.smithi088.stderr:2024-06-03T08:36:32.051+0000 7f1983fff640 10 client.4300.objecter _send_command 1 2024-06-03T08:36:32.089 INFO:teuthology.orchestra.run.smithi088.stderr:2024-06-03T08:36:32.051+0000 7f1983fff640 1 -- 172.21.15.88:0/1438849704 --> v1:172.21.15.88:6802/35065 -- command(tid 1: {"prefix": "get_command_descriptions"}) v1 -- 0x7f197c077ef0 con 0x7f197c074280 2024-06-03T08:36:32.089 INFO:teuthology.orchestra.run.smithi088.stderr:2024-06-03T08:36:32.051+0000 7f1983fff640 20 client.4300.objecter dump_active .. 0 homeless 2024-06-03T08:36:32.089 INFO:teuthology.orchestra.run.smithi088.stderr:2024-06-03T08:36:32.051+0000 7f1996ffd640 10 monclient: get_auth_request con 0x7f197c074280 auth_method 0 2024-06-03T08:36:32.089 INFO:teuthology.orchestra.run.smithi088.stderr:2024-06-03T08:36:32.052+0000 7f1983fff640 1 -- 172.21.15.88:0/1438849704 <== mon.1 v1:172.21.15.192:6789/0 9 ==== mon_get_version_reply(handle=1 version=26) v2 ==== 24+0+0 (unknown 2370098492 0 0) 0x7f198404e930 con 0x7f19900327d0 2024-06-03T08:36:32.089 INFO:teuthology.orchestra.run.smithi088.stderr:2024-06-03T08:36:32.052+0000 7f1983fff640 10 monclient: handle_get_version_reply finishing 1 version 26 2024-06-03T08:36:32.089 INFO:teuthology.orchestra.run.smithi088.stderr:2024-06-03T08:36:32.055+0000 7f1983fff640 10 client.4300.objecter ms_handle_connect 0x7f197c074280 2024-06-03T08:36:32.090 INFO:teuthology.orchestra.run.smithi088.stderr:2024-06-03T08:36:32.056+0000 7fa0e9d9b640 1 -- 172.21.15.88:0/424874043 <== mon.1 v1:172.21.15.192:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 214+0+0 (unknown 554456052 0 0) 0x7fa0d4003520 con 0x7fa0ec0a8960 2024-06-03T08:36:32.090 INFO:teuthology.orchestra.run.smithi088.stderr:2024-06-03T08:36:32.056+0000 7fa0e9d9b640 1 -- 172.21.15.88:0/424874043 >> v1:172.21.15.88:6790/0 conn(0x7fa0ec13eec0 legacy=0x7fa0ec14ce60 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-06-03T08:36:32.090 INFO:teuthology.orchestra.run.smithi088.stderr:2024-06-03T08:36:32.056+0000 7fa0e9d9b640 1 -- 172.21.15.88:0/424874043 >> v1:172.21.15.88:6789/0 conn(0x7fa0ec1450f0 legacy=0x7fa0ec13e7b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-06-03T08:36:32.090 INFO:teuthology.orchestra.run.smithi088.stderr:2024-06-03T08:36:32.056+0000 7fa0e9d9b640 10 monclient: _finish_hunting 0 2024-06-03T08:36:32.090 INFO:teuthology.orchestra.run.smithi088.stderr:2024-06-03T08:36:32.056+0000 7fa0e9d9b640 1 monclient: found mon.noname-c 2024-06-03T08:36:32.090 INFO:teuthology.orchestra.run.smithi088.stderr:2024-06-03T08:36:32.056+0000 7fa0e9d9b640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-06-03T08:36:32.090 INFO:teuthology.orchestra.run.smithi088.stderr:2024-06-03T08:36:32.056+0000 7fa0e9d9b640 10 monclient: _send_mon_message to mon.noname-c at v1:172.21.15.192:6789/0 2024-06-03T08:36:32.091 INFO:teuthology.orchestra.run.smithi088.stderr:2024-06-03T08:36:32.056+0000 7fa0e9d9b640 1 -- 172.21.15.88:0/424874043 --> v1:172.21.15.192:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fa0ec0b1c60 con 0x7fa0ec0a8960 2024-06-03T08:36:32.091 INFO:teuthology.orchestra.run.smithi088.stderr:2024-06-03T08:36:32.056+0000 7fa0e9d9b640 10 monclient: _finish_auth 0 2024-06-03T08:36:32.091 INFO:teuthology.orchestra.run.smithi088.stderr:2024-06-03T08:36:32.056+0000 7fa0e9d9b640 10 monclient: _check_auth_tickets 2024-06-03T08:36:32.091 INFO:teuthology.orchestra.run.smithi088.stderr:2024-06-03T08:36:32.056+0000 7fa0e9d9b640 20 monclient: _check_auth_rotating not needed by client.admin 2024-06-03T08:36:32.091 INFO:teuthology.orchestra.run.smithi088.stderr:2024-06-03T08:36:32.056+0000 7fa0f1016640 5 monclient: authenticate success, global_id 4309 2024-06-03T08:36:32.091 INFO:teuthology.orchestra.run.smithi088.stderr:2024-06-03T08:36:32.056+0000 7fa0f1016640 20 monclient: get_monmap_and_config waiting for monmap|config 2024-06-03T08:36:32.091 INFO:teuthology.orchestra.run.smithi088.stderr:2024-06-03T08:36:32.056+0000 7fa0e9d9b640 1 -- 172.21.15.88:0/424874043 <== mon.1 v1:172.21.15.192:6789/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 2234374065 0 0) 0x7fa0dc001610 con 0x7fa0ec0a8960 2024-06-03T08:36:32.091 INFO:teuthology.orchestra.run.smithi088.stderr:2024-06-03T08:36:32.056+0000 7fa0e9d9b640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-06-03T08:36:32.091 INFO:teuthology.orchestra.run.smithi088.stderr:2024-06-03T08:36:32.056+0000 7fa0e9d9b640 10 monclient: got monmap 1 from mon.noname-c (according to old e1) 2024-06-03T08:36:32.091 INFO:teuthology.orchestra.run.smithi088.stderr:2024-06-03T08:36:32.056+0000 7fa0e9d9b640 10 monclient: dump: 2024-06-03T08:36:32.091 INFO:teuthology.orchestra.run.smithi088.stderr:epoch 1 2024-06-03T08:36:32.091 INFO:teuthology.orchestra.run.smithi088.stderr:fsid ec3c3fdd-3ac6-4d7d-9412-683130ed8b04 2024-06-03T08:36:32.091 INFO:teuthology.orchestra.run.smithi088.stderr:last_changed 2024-06-03T08:34:31.885125+0000 2024-06-03T08:36:32.091 INFO:teuthology.orchestra.run.smithi088.stderr:created 2024-06-03T08:34:31.885125+0000 2024-06-03T08:36:32.092 INFO:teuthology.orchestra.run.smithi088.stderr:min_mon_release 18 (reef) 2024-06-03T08:36:32.092 INFO:teuthology.orchestra.run.smithi088.stderr:election_strategy: 3 2024-06-03T08:36:32.092 INFO:teuthology.orchestra.run.smithi088.stderr:0: v1:172.21.15.88:6789/0 mon.a 2024-06-03T08:36:32.092 INFO:teuthology.orchestra.run.smithi088.stderr:1: v1:172.21.15.192:6789/0 mon.b 2024-06-03T08:36:32.092 INFO:teuthology.orchestra.run.smithi088.stderr:2: v1:172.21.15.88:6790/0 mon.c 2024-06-03T08:36:32.092 INFO:teuthology.orchestra.run.smithi088.stderr: 2024-06-03T08:36:32.092 INFO:teuthology.orchestra.run.smithi088.stderr:2024-06-03T08:36:32.056+0000 7fa0f1016640 20 monclient: get_monmap_and_config waiting for monmap|config 2024-06-03T08:36:32.092 INFO:teuthology.orchestra.run.smithi088.stderr:2024-06-03T08:36:32.057+0000 7f791559a640 1 -- 172.21.15.88:0/1994185464 <== mon.0 v1:172.21.15.88:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 4283497263 0 0) 0x7f79180a8ed0 con 0x7f791813e6c0 2024-06-03T08:36:32.093 INFO:teuthology.orchestra.run.smithi088.stderr:2024-06-03T08:36:32.057+0000 7f791559a640 10 monclient(hunting): _negotiate 2024-06-03T08:36:32.093 INFO:teuthology.orchestra.run.smithi088.stderr:2024-06-03T08:36:32.057+0000 7f791559a640 10 monclient(hunting): _init_auth method 2 2024-06-03T08:36:32.093 INFO:teuthology.orchestra.run.smithi088.stderr:2024-06-03T08:36:32.057+0000 7f791559a640 10 monclient(hunting): _init_auth creating new auth 2024-06-03T08:36:32.093 INFO:teuthology.orchestra.run.smithi088.stderr:2024-06-03T08:36:32.057+0000 7f791559a640 10 monclient(hunting): my global_id is 4314 2024-06-03T08:36:32.093 INFO:teuthology.orchestra.run.smithi088.stderr:2024-06-03T08:36:32.057+0000 7f791559a640 1 -- 172.21.15.88:0/1994185464 --> v1:172.21.15.88:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f7900003520 con 0x7f791813e6c0 2024-06-03T08:36:32.093 INFO:teuthology.orchestra.run.smithi088.stderr:2024-06-03T08:36:32.058+0000 7fd6596d0640 1 --1- >> v1:172.21.15.88:6789/0 conn(0x7fd65406dc10 0x7fd65412d770 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=1).handle_server_banner_and_identify peer v1:172.21.15.88:6789/0 says I am v1:172.21.15.88:60344/0 (socket says 172.21.15.88:60344) 2024-06-03T08:36:32.093 INFO:teuthology.orchestra.run.smithi088.stderr:2024-06-03T08:36:32.058+0000 7fd6596d0640 1 -- 172.21.15.88:0/2839181686 learned_addr learned my addr 172.21.15.88:0/2839181686 (peer_addr_for_me v1:172.21.15.88:0/0) 2024-06-03T08:36:32.095 INFO:teuthology.orchestra.run.smithi088.stderr:2024-06-03T08:36:32.058+0000 7f9bacaf0640 1 -- 172.21.15.88:0/3753766473 <== mon.2 v1:172.21.15.88:6790/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 214+0+0 (unknown 439122060 0 0) 0x7f9b94003520 con 0x7f9ba80b43c0 2024-06-03T08:36:32.095 INFO:teuthology.orchestra.run.smithi088.stderr:2024-06-03T08:36:32.058+0000 7f9bacaf0640 1 -- 172.21.15.88:0/3753766473 >> v1:172.21.15.192:6789/0 conn(0x7f9ba814ae20 legacy=0x7f9ba814d210 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-06-03T08:36:32.095 INFO:teuthology.orchestra.run.smithi088.stderr:2024-06-03T08:36:32.058+0000 7f9bacaf0640 1 -- 172.21.15.88:0/3753766473 >> v1:172.21.15.88:6789/0 conn(0x7f9ba813e6c0 legacy=0x7f9ba813eb40 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-06-03T08:36:32.095 INFO:teuthology.orchestra.run.smithi088.stderr:2024-06-03T08:36:32.058+0000 7f9bacaf0640 10 monclient: _finish_hunting 0 2024-06-03T08:36:32.095 INFO:teuthology.orchestra.run.smithi088.stderr:2024-06-03T08:36:32.058+0000 7f9bacaf0640 1 monclient: found mon.noname-b 2024-06-03T08:36:32.095 INFO:teuthology.orchestra.run.smithi088.stderr:2024-06-03T08:36:32.058+0000 7f9bacaf0640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-06-03T08:36:32.096 INFO:teuthology.orchestra.run.smithi088.stderr:2024-06-03T08:36:32.058+0000 7f9bacaf0640 10 monclient: _send_mon_message to mon.noname-b at v1:172.21.15.88:6790/0 2024-06-03T08:36:32.096 INFO:teuthology.orchestra.run.smithi088.stderr:2024-06-03T08:36:32.058+0000 7f9bacaf0640 1 -- 172.21.15.88:0/3753766473 --> v1:172.21.15.88:6790/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f9ba80a87f0 con 0x7f9ba80b43c0 2024-06-03T08:36:32.096 INFO:teuthology.orchestra.run.smithi088.stderr:2024-06-03T08:36:32.058+0000 7f9bacaf0640 10 monclient: _finish_auth 0 2024-06-03T08:36:32.096 INFO:teuthology.orchestra.run.smithi088.stderr:2024-06-03T08:36:32.058+0000 7f9bacaf0640 10 monclient: _check_auth_tickets 2024-06-03T08:36:32.096 INFO:teuthology.orchestra.run.smithi088.stderr:2024-06-03T08:36:32.058+0000 7f9bacaf0640 20 monclient: _check_auth_rotating not needed by client.admin 2024-06-03T08:36:32.096 INFO:teuthology.orchestra.run.smithi088.stderr:2024-06-03T08:36:32.058+0000 7f9bacaf0640 1 -- 172.21.15.88:0/3753766473 <== mon.2 v1:172.21.15.88:6790/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 2234374065 0 0) 0x7f9ba4001610 con 0x7f9ba80b43c0 2024-06-03T08:36:32.096 INFO:teuthology.orchestra.run.smithi088.stderr:2024-06-03T08:36:32.058+0000 7f9bafd56640 5 monclient: authenticate success, global_id 4331 2024-06-03T08:36:32.096 INFO:teuthology.orchestra.run.smithi088.stderr:2024-06-03T08:36:32.058+0000 7f9bafd56640 20 monclient: get_monmap_and_config waiting for monmap|config 2024-06-03T08:36:32.096 INFO:teuthology.orchestra.run.smithi088.stderr:2024-06-03T08:36:32.058+0000 7f9bacaf0640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-06-03T08:36:32.096 INFO:teuthology.orchestra.run.smithi088.stderr:2024-06-03T08:36:32.058+0000 7f9bacaf0640 10 monclient: got monmap 1 from mon.noname-b (according to old e1) 2024-06-03T08:36:32.096 INFO:teuthology.orchestra.run.smithi088.stderr:2024-06-03T08:36:32.058+0000 7f9bacaf0640 10 monclient: dump: 2024-06-03T08:36:32.096 INFO:teuthology.orchestra.run.smithi088.stderr:epoch 1 2024-06-03T08:36:32.096 INFO:teuthology.orchestra.run.smithi088.stderr:fsid ec3c3fdd-3ac6-4d7d-9412-683130ed8b04 2024-06-03T08:36:32.097 INFO:teuthology.orchestra.run.smithi088.stderr:last_changed 2024-06-03T08:34:31.885125+0000 2024-06-03T08:36:32.097 INFO:teuthology.orchestra.run.smithi088.stderr:created 2024-06-03T08:34:31.885125+0000 2024-06-03T08:36:32.097 INFO:teuthology.orchestra.run.smithi088.stderr:min_mon_release 18 (reef) 2024-06-03T08:36:32.097 INFO:teuthology.orchestra.run.smithi088.stderr:election_strategy: 3 2024-06-03T08:36:32.097 INFO:teuthology.orchestra.run.smithi088.stderr:0: v1:172.21.15.88:6789/0 mon.a 2024-06-03T08:36:32.097 INFO:teuthology.orchestra.run.smithi088.stderr:1: v1:172.21.15.192:6789/0 mon.b 2024-06-03T08:36:32.097 INFO:teuthology.orchestra.run.smithi088.stderr:2: v1:172.21.15.88:6790/0 mon.c 2024-06-03T08:36:32.097 INFO:teuthology.orchestra.run.smithi088.stderr: 2024-06-03T08:36:32.097 INFO:teuthology.orchestra.run.smithi088.stderr:2024-06-03T08:36:32.058+0000 7f9bafd56640 20 monclient: get_monmap_and_config waiting for monmap|config 2024-06-03T08:36:32.098 INFO:teuthology.orchestra.run.smithi088.stderr:2024-06-03T08:36:32.058+0000 7ff47bfff640 10 client.?.objecter ms_handle_connect 0x7ff49406dc10 2024-06-03T08:36:32.098 INFO:teuthology.orchestra.run.smithi088.stderr:2024-06-03T08:36:32.058+0000 7ff47bfff640 10 client.?.objecter resend_mon_ops 2024-06-03T08:36:32.098 INFO:teuthology.orchestra.run.smithi088.stderr:2024-06-03T08:36:32.059+0000 7ff47bfff640 1 -- 172.21.15.88:0/2378444824 <== mon.0 v1:172.21.15.88:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 3913964992 0 0) 0x7ff49412ab90 con 0x7ff49412fd80 2024-06-03T08:36:32.098 INFO:teuthology.orchestra.run.smithi088.stderr:2024-06-03T08:36:32.059+0000 7ff47bfff640 10 monclient(hunting): _negotiate 2024-06-03T08:36:32.099 INFO:teuthology.orchestra.run.smithi088.stderr:2024-06-03T08:36:32.059+0000 7ff47bfff640 10 monclient(hunting): _init_auth method 2 2024-06-03T08:36:32.099 INFO:teuthology.orchestra.run.smithi088.stderr:2024-06-03T08:36:32.059+0000 7ff47bfff640 10 monclient(hunting): _init_auth creating new auth 2024-06-03T08:36:32.099 INFO:teuthology.orchestra.run.smithi088.stderr:2024-06-03T08:36:32.059+0000 7ff47bfff640 10 monclient(hunting): my global_id is 4317 2024-06-03T08:36:32.099 INFO:teuthology.orchestra.run.smithi088.stderr:2024-06-03T08:36:32.059+0000 7ff47bfff640 1 -- 172.21.15.88:0/2378444824 --> v1:172.21.15.88:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7ff480004eb0 con 0x7ff49412fd80 2024-06-03T08:36:32.099 INFO:teuthology.orchestra.run.smithi088.stderr:2024-06-03T08:36:32.060+0000 7ff47bfff640 10 client.?.objecter ms_handle_connect 0x7ff494141540 2024-06-03T08:36:32.099 INFO:teuthology.orchestra.run.smithi088.stderr:2024-06-03T08:36:32.060+0000 7ff47bfff640 10 client.?.objecter resend_mon_ops 2024-06-03T08:36:32.099 INFO:teuthology.orchestra.run.smithi088.stderr:2024-06-03T08:36:32.060+0000 7fd649ffb640 10 client.?.objecter ms_handle_connect 0x7fd65406dc10 2024-06-03T08:36:32.099 INFO:teuthology.orchestra.run.smithi088.stderr:2024-06-03T08:36:32.060+0000 7fd649ffb640 10 client.?.objecter resend_mon_ops 2024-06-03T08:36:32.100 INFO:teuthology.orchestra.run.smithi088.stderr:2024-06-03T08:36:32.061+0000 7f791559a640 1 -- 172.21.15.88:0/1994185464 <== mon.1 v1:172.21.15.192:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 3595209210 0 0) 0x7f79180aa420 con 0x7f791814ae20 2024-06-03T08:36:32.100 INFO:teuthology.orchestra.run.smithi088.stderr:2024-06-03T08:36:32.061+0000 7f791559a640 10 monclient(hunting): _negotiate 2024-06-03T08:36:32.100 INFO:teuthology.orchestra.run.smithi088.stderr:2024-06-03T08:36:32.061+0000 7f791559a640 10 monclient(hunting): _init_auth method 2 2024-06-03T08:36:32.100 INFO:teuthology.orchestra.run.smithi088.stderr:2024-06-03T08:36:32.061+0000 7f791559a640 10 monclient(hunting): _init_auth creating new auth 2024-06-03T08:36:32.100 INFO:teuthology.orchestra.run.smithi088.stderr:2024-06-03T08:36:32.061+0000 7f791559a640 10 monclient(hunting): my global_id is 4318 2024-06-03T08:36:32.100 INFO:teuthology.orchestra.run.smithi088.stderr:2024-06-03T08:36:32.061+0000 7f791559a640 1 -- 172.21.15.88:0/1994185464 --> v1:172.21.15.192:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f79180a8ed0 con 0x7f791814ae20 2024-06-03T08:36:32.101 INFO:teuthology.orchestra.run.smithi088.stderr:2024-06-03T08:36:32.063+0000 7f791559a640 1 -- 172.21.15.88:0/1994185464 <== mon.2 v1:172.21.15.88:6790/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 3062085149 0 0) 0x7f79180aa690 con 0x7f79180b43c0 2024-06-03T08:36:32.101 INFO:teuthology.orchestra.run.smithi088.stderr:2024-06-03T08:36:32.063+0000 7f791559a640 10 monclient(hunting): _negotiate 2024-06-03T08:36:32.101 INFO:teuthology.orchestra.run.smithi088.stderr:2024-06-03T08:36:32.063+0000 7f791559a640 10 monclient(hunting): _init_auth method 2 2024-06-03T08:36:32.101 INFO:teuthology.orchestra.run.smithi088.stderr:2024-06-03T08:36:32.063+0000 7f791559a640 10 monclient(hunting): _init_auth creating new auth 2024-06-03T08:36:32.101 INFO:teuthology.orchestra.run.smithi088.stderr:2024-06-03T08:36:32.063+0000 7f791559a640 10 monclient(hunting): my global_id is 4334 2024-06-03T08:36:32.101 INFO:teuthology.orchestra.run.smithi088.stderr:2024-06-03T08:36:32.063+0000 7f791559a640 1 -- 172.21.15.88:0/1994185464 --> v1:172.21.15.88:6790/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f79180aa420 con 0x7f79180b43c0 2024-06-03T08:36:32.101 INFO:teuthology.orchestra.run.smithi088.stderr:2024-06-03T08:36:32.065+0000 7f4c2f7fe640 1 --1- >> v1:172.21.15.192:6789/0 conn(0x7f4c300a86f0 0x7f4c3013ff20 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=1).handle_server_banner_and_identify peer v1:172.21.15.192:6789/0 says I am v1:172.21.15.88:52426/0 (socket says 172.21.15.88:52426) 2024-06-03T08:36:32.101 INFO:teuthology.orchestra.run.smithi088.stderr:2024-06-03T08:36:32.065+0000 7f4c2f7fe640 1 -- 172.21.15.88:0/2099487910 learned_addr learned my addr 172.21.15.88:0/2099487910 (peer_addr_for_me v1:172.21.15.88:0/0) 2024-06-03T08:36:32.102 INFO:teuthology.orchestra.run.smithi088.stderr:2024-06-03T08:36:32.065+0000 7f791559a640 1 -- 172.21.15.88:0/1994185464 <== mon.1 v1:172.21.15.192:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 214+0+0 (unknown 2118132583 0 0) 0x7f79180a8ed0 con 0x7f791814ae20 2024-06-03T08:36:32.102 INFO:teuthology.orchestra.run.smithi088.stderr:2024-06-03T08:36:32.065+0000 7f791559a640 1 -- 172.21.15.88:0/1994185464 >> v1:172.21.15.88:6790/0 conn(0x7f79180b43c0 legacy=0x7f79181415b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-06-03T08:36:32.102 INFO:teuthology.orchestra.run.smithi088.stderr:2024-06-03T08:36:32.065+0000 7f791559a640 1 -- 172.21.15.88:0/1994185464 >> v1:172.21.15.88:6789/0 conn(0x7f791813e6c0 legacy=0x7f791813eb40 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-06-03T08:36:32.102 INFO:teuthology.orchestra.run.smithi088.stderr:2024-06-03T08:36:32.065+0000 7f791559a640 10 monclient: _finish_hunting 0 2024-06-03T08:36:32.102 INFO:teuthology.orchestra.run.smithi088.stderr:2024-06-03T08:36:32.065+0000 7f791559a640 1 monclient: found mon.noname-c 2024-06-03T08:36:32.102 INFO:teuthology.orchestra.run.smithi088.stderr:2024-06-03T08:36:32.065+0000 7f791559a640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-06-03T08:36:32.103 INFO:teuthology.orchestra.run.smithi088.stderr:2024-06-03T08:36:32.065+0000 7f791559a640 10 monclient: _send_mon_message to mon.noname-c at v1:172.21.15.192:6789/0 2024-06-03T08:36:32.103 INFO:teuthology.orchestra.run.smithi088.stderr:2024-06-03T08:36:32.065+0000 7f791559a640 1 -- 172.21.15.88:0/1994185464 --> v1:172.21.15.192:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f79180a87f0 con 0x7f791814ae20 2024-06-03T08:36:32.103 INFO:teuthology.orchestra.run.smithi088.stderr:2024-06-03T08:36:32.065+0000 7f791559a640 10 monclient: _finish_auth 0 2024-06-03T08:36:32.103 INFO:teuthology.orchestra.run.smithi088.stderr:2024-06-03T08:36:32.065+0000 7f791559a640 10 monclient: _check_auth_tickets 2024-06-03T08:36:32.103 INFO:teuthology.orchestra.run.smithi088.stderr:2024-06-03T08:36:32.065+0000 7f791559a640 20 monclient: _check_auth_rotating not needed by client.admin 2024-06-03T08:36:32.103 INFO:teuthology.orchestra.run.smithi088.stderr:2024-06-03T08:36:32.065+0000 7f791559a640 1 -- 172.21.15.88:0/1994185464 <== mon.1 v1:172.21.15.192:6789/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 2234374065 0 0) 0x7f7910001970 con 0x7f791814ae20 2024-06-03T08:36:32.103 INFO:teuthology.orchestra.run.smithi088.stderr:2024-06-03T08:36:32.065+0000 7f791559a640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-06-03T08:36:32.103 INFO:teuthology.orchestra.run.smithi088.stderr:2024-06-03T08:36:32.065+0000 7f791559a640 10 monclient: got monmap 1 from mon.noname-c (according to old e1) 2024-06-03T08:36:32.103 INFO:teuthology.orchestra.run.smithi088.stderr:2024-06-03T08:36:32.065+0000 7f791559a640 10 monclient: dump: 2024-06-03T08:36:32.103 INFO:teuthology.orchestra.run.smithi088.stderr:epoch 1 2024-06-03T08:36:32.103 INFO:teuthology.orchestra.run.smithi088.stderr:fsid ec3c3fdd-3ac6-4d7d-9412-683130ed8b04 2024-06-03T08:36:32.103 INFO:teuthology.orchestra.run.smithi088.stderr:last_changed 2024-06-03T08:34:31.885125+0000 2024-06-03T08:36:32.104 INFO:teuthology.orchestra.run.smithi088.stderr:created 2024-06-03T08:34:31.885125+0000 2024-06-03T08:36:32.104 INFO:teuthology.orchestra.run.smithi088.stderr:min_mon_release 18 (reef) 2024-06-03T08:36:32.104 INFO:teuthology.orchestra.run.smithi088.stderr:election_strategy: 3 2024-06-03T08:36:32.104 INFO:teuthology.orchestra.run.smithi088.stderr:0: v1:172.21.15.88:6789/0 mon.a 2024-06-03T08:36:32.104 INFO:teuthology.orchestra.run.smithi088.stderr:1: v1:172.21.15.192:6789/0 mon.b 2024-06-03T08:36:32.104 INFO:teuthology.orchestra.run.smithi088.stderr:2: v1:172.21.15.88:6790/0 mon.c 2024-06-03T08:36:32.104 INFO:teuthology.orchestra.run.smithi088.stderr: 2024-06-03T08:36:32.104 INFO:teuthology.orchestra.run.smithi088.stderr:2024-06-03T08:36:32.065+0000 7f791cb92640 5 monclient: authenticate success, global_id 4318 2024-06-03T08:36:32.104 INFO:teuthology.orchestra.run.smithi088.stderr:2024-06-03T08:36:32.065+0000 7f791cb92640 20 monclient: get_monmap_and_config waiting for monmap|config 2024-06-03T08:36:32.104 INFO:teuthology.orchestra.run.smithi088.stderr:2024-06-03T08:36:32.066+0000 7fd649ffb640 10 client.?.objecter ms_handle_connect 0x7fd654141540 2024-06-03T08:36:32.104 INFO:teuthology.orchestra.run.smithi088.stderr:2024-06-03T08:36:32.066+0000 7fd649ffb640 10 client.?.objecter resend_mon_ops 2024-06-03T08:36:32.105 INFO:teuthology.orchestra.run.smithi088.stderr:2024-06-03T08:36:32.066+0000 7f4c1bfff640 10 client.?.objecter ms_handle_connect 0x7f4c300a86f0 2024-06-03T08:36:32.105 INFO:teuthology.orchestra.run.smithi088.stderr:2024-06-03T08:36:32.066+0000 7f4c1bfff640 10 client.?.objecter resend_mon_ops 2024-06-03T08:36:32.105 INFO:teuthology.orchestra.run.smithi088.stderr:2024-06-03T08:36:32.066+0000 7f4c1bfff640 10 client.?.objecter ms_handle_connect 0x7f4c300b4060 2024-06-03T08:36:32.105 INFO:teuthology.orchestra.run.smithi088.stderr:2024-06-03T08:36:32.067+0000 7f4c1bfff640 10 client.?.objecter resend_mon_ops 2024-06-03T08:36:32.106 INFO:teuthology.orchestra.run.smithi088.stderr:2024-06-03T08:36:32.067+0000 7fd649ffb640 10 client.?.objecter ms_handle_connect 0x7fd65413b700 2024-06-03T08:36:32.106 INFO:teuthology.orchestra.run.smithi088.stderr:2024-06-03T08:36:32.067+0000 7fd649ffb640 10 client.?.objecter resend_mon_ops 2024-06-03T08:36:32.107 INFO:teuthology.orchestra.run.smithi088.stderr:2024-06-03T08:36:32.068+0000 7f9bacaf0640 1 -- 172.21.15.88:0/3753766473 <== mon.2 v1:172.21.15.88:6790/0 4 ==== config(0 keys) v1 ==== 4+0+0 (unknown 0 0 0) 0x7f9ba40018a0 con 0x7f9ba80b43c0 2024-06-03T08:36:32.107 INFO:teuthology.orchestra.run.smithi088.stderr:2024-06-03T08:36:32.068+0000 7f9bacaf0640 10 monclient: handle_config config(0 keys) v1 2024-06-03T08:36:32.107 INFO:teuthology.orchestra.run.smithi088.stderr:2024-06-03T08:36:32.068+0000 7f9bacaf0640 1 -- 172.21.15.88:0/3753766473 <== mon.2 v1:172.21.15.88:6790/0 5 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 2234374065 0 0) 0x7f9ba4001c30 con 0x7f9ba80b43c0 2024-06-03T08:36:32.107 INFO:teuthology.orchestra.run.smithi088.stderr:2024-06-03T08:36:32.068+0000 7f9bafd56640 10 monclient: get_monmap_and_config success 2024-06-03T08:36:32.108 INFO:teuthology.orchestra.run.smithi088.stderr:2024-06-03T08:36:32.068+0000 7f9bafd56640 4 set_mon_vals no callback set 2024-06-03T08:36:32.108 INFO:teuthology.orchestra.run.smithi088.stderr:2024-06-03T08:36:32.068+0000 7f9bacaf0640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-06-03T08:36:32.108 INFO:teuthology.orchestra.run.smithi088.stderr:2024-06-03T08:36:32.068+0000 7f9bacaf0640 10 monclient: got monmap 1 from mon.c (according to old e1) 2024-06-03T08:36:32.108 INFO:teuthology.orchestra.run.smithi088.stderr:2024-06-03T08:36:32.068+0000 7f9bacaf0640 10 monclient: dump: 2024-06-03T08:36:32.108 INFO:teuthology.orchestra.run.smithi088.stderr:epoch 1 2024-06-03T08:36:32.108 INFO:teuthology.orchestra.run.smithi088.stderr:fsid ec3c3fdd-3ac6-4d7d-9412-683130ed8b04 2024-06-03T08:36:32.108 INFO:teuthology.orchestra.run.smithi088.stderr:last_changed 2024-06-03T08:34:31.885125+0000 2024-06-03T08:36:32.108 INFO:teuthology.orchestra.run.smithi088.stderr:created 2024-06-03T08:34:31.885125+0000 2024-06-03T08:36:32.108 INFO:teuthology.orchestra.run.smithi088.stderr:min_mon_release 18 (reef) 2024-06-03T08:36:32.108 INFO:teuthology.orchestra.run.smithi088.stderr:election_strategy: 3 2024-06-03T08:36:32.108 INFO:teuthology.orchestra.run.smithi088.stderr:0: v1:172.21.15.88:6789/0 mon.a 2024-06-03T08:36:32.108 INFO:teuthology.orchestra.run.smithi088.stderr:1: v1:172.21.15.192:6789/0 mon.b 2024-06-03T08:36:32.109 INFO:teuthology.orchestra.run.smithi088.stderr:2: v1:172.21.15.88:6790/0 mon.c 2024-06-03T08:36:32.109 INFO:teuthology.orchestra.run.smithi088.stderr: 2024-06-03T08:36:32.109 INFO:teuthology.orchestra.run.smithi088.stderr:2024-06-03T08:36:32.069+0000 7f9bafd56640 10 monclient: shutdown 2024-06-03T08:36:32.109 INFO:teuthology.orchestra.run.smithi088.stderr:2024-06-03T08:36:32.069+0000 7f9bafd56640 20 monclient: shutdown discarding 0 pending message(s) 2024-06-03T08:36:32.109 INFO:teuthology.orchestra.run.smithi088.stderr:2024-06-03T08:36:32.069+0000 7f9bafd56640 1 -- 172.21.15.88:0/3753766473 >> v1:172.21.15.88:6790/0 conn(0x7f9ba80b43c0 legacy=0x7f9ba81415b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-06-03T08:36:32.109 INFO:teuthology.orchestra.run.smithi088.stderr:2024-06-03T08:36:32.069+0000 7f9bafd56640 1 -- 172.21.15.88:0/3753766473 shutdown_connections 2024-06-03T08:36:32.109 INFO:teuthology.orchestra.run.smithi088.stderr:2024-06-03T08:36:32.069+0000 7f9bafd56640 1 -- 172.21.15.88:0/3753766473 >> 172.21.15.88:0/3753766473 conn(0x7f9ba80719d0 msgr2=0x7f9ba8071de0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-06-03T08:36:32.109 INFO:teuthology.orchestra.run.smithi088.stderr:2024-06-03T08:36:32.070+0000 7f9bafd56640 1 -- 172.21.15.88:0/3753766473 shutdown_connections 2024-06-03T08:36:32.109 INFO:teuthology.orchestra.run.smithi088.stderr:2024-06-03T08:36:32.070+0000 7f9bafd56640 1 -- 172.21.15.88:0/3753766473 wait complete. 2024-06-03T08:36:32.109 INFO:teuthology.orchestra.run.smithi088.stderr:2024-06-03T08:36:32.071+0000 7f9bafd56640 10 monclient: build_initial_monmap 2024-06-03T08:36:32.109 INFO:teuthology.orchestra.run.smithi088.stderr:2024-06-03T08:36:32.071+0000 7f9bafd56640 10 monclient: monmap: 2024-06-03T08:36:32.109 INFO:teuthology.orchestra.run.smithi088.stderr:epoch 0 2024-06-03T08:36:32.109 INFO:teuthology.orchestra.run.smithi088.stderr:fsid 00000000-0000-0000-0000-000000000000 2024-06-03T08:36:32.109 INFO:teuthology.orchestra.run.smithi088.stderr:last_changed 0.000000 2024-06-03T08:36:32.110 INFO:teuthology.orchestra.run.smithi088.stderr:created 0.000000 2024-06-03T08:36:32.110 INFO:teuthology.orchestra.run.smithi088.stderr:min_mon_release 0 (unknown) 2024-06-03T08:36:32.110 INFO:teuthology.orchestra.run.smithi088.stderr:election_strategy: 1 2024-06-03T08:36:32.110 INFO:teuthology.orchestra.run.smithi088.stderr:0: v1:172.21.15.88:6789/0 mon.noname-a 2024-06-03T08:36:32.110 INFO:teuthology.orchestra.run.smithi088.stderr:1: v1:172.21.15.192:6789/0 mon.noname-b 2024-06-03T08:36:32.110 INFO:teuthology.orchestra.run.smithi088.stderr:2: v1:172.21.15.88:6790/0 mon.noname-c 2024-06-03T08:36:32.110 INFO:teuthology.orchestra.run.smithi088.stderr: 2024-06-03T08:36:32.110 INFO:teuthology.orchestra.run.smithi088.stderr:2024-06-03T08:36:32.071+0000 7f9bafd56640 1 librados: starting msgr at 2024-06-03T08:36:32.110 INFO:teuthology.orchestra.run.smithi088.stderr:2024-06-03T08:36:32.071+0000 7f9bafd56640 1 librados: starting objecter 2024-06-03T08:36:32.111 INFO:teuthology.orchestra.run.smithi088.stderr:2024-06-03T08:36:32.071+0000 7ff47bfff640 1 -- 172.21.15.88:0/2378444824 <== mon.2 v1:172.21.15.88:6790/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 1223292007 0 0) 0x7ff49412bb80 con 0x7ff494141540 2024-06-03T08:36:32.111 INFO:teuthology.orchestra.run.smithi088.stderr:2024-06-03T08:36:32.071+0000 7ff47bfff640 10 monclient(hunting): _negotiate 2024-06-03T08:36:32.111 INFO:teuthology.orchestra.run.smithi088.stderr:2024-06-03T08:36:32.071+0000 7ff47bfff640 10 monclient(hunting): _init_auth method 2 2024-06-03T08:36:32.111 INFO:teuthology.orchestra.run.smithi088.stderr:2024-06-03T08:36:32.071+0000 7ff47bfff640 10 monclient(hunting): _init_auth creating new auth 2024-06-03T08:36:32.111 INFO:teuthology.orchestra.run.smithi088.stderr:2024-06-03T08:36:32.071+0000 7ff47bfff640 10 monclient(hunting): my global_id is 4337 2024-06-03T08:36:32.111 INFO:teuthology.orchestra.run.smithi088.stderr:2024-06-03T08:36:32.071+0000 7f9bafd56640 1 Processor -- start 2024-06-03T08:36:32.112 INFO:teuthology.orchestra.run.smithi088.stderr:2024-06-03T08:36:32.071+0000 7ff47bfff640 1 -- 172.21.15.88:0/2378444824 --> v1:172.21.15.88:6790/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7ff49412ab90 con 0x7ff494141540 2024-06-03T08:36:32.112 INFO:teuthology.orchestra.run.smithi088.stderr:2024-06-03T08:36:32.071+0000 7ff47bfff640 1 -- 172.21.15.88:0/2378444824 <== mon.1 v1:172.21.15.192:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 430417605 0 0) 0x7ff494141020 con 0x7ff49406dc10 2024-06-03T08:36:32.112 INFO:teuthology.orchestra.run.smithi088.stderr:2024-06-03T08:36:32.071+0000 7ff47bfff640 10 monclient(hunting): _negotiate 2024-06-03T08:36:32.112 INFO:teuthology.orchestra.run.smithi088.stderr:2024-06-03T08:36:32.071+0000 7ff47bfff640 10 monclient(hunting): _init_auth method 2 2024-06-03T08:36:32.112 INFO:teuthology.orchestra.run.smithi088.stderr:2024-06-03T08:36:32.071+0000 7ff47bfff640 10 monclient(hunting): _init_auth creating new auth 2024-06-03T08:36:32.112 INFO:teuthology.orchestra.run.smithi088.stderr:2024-06-03T08:36:32.071+0000 7ff47bfff640 10 monclient(hunting): my global_id is 4321 2024-06-03T08:36:32.113 INFO:teuthology.orchestra.run.smithi088.stderr:2024-06-03T08:36:32.071+0000 7f9bafd56640 1 -- start start 2024-06-03T08:36:32.113 INFO:teuthology.orchestra.run.smithi088.stderr:2024-06-03T08:36:32.071+0000 7ff47bfff640 1 -- 172.21.15.88:0/2378444824 --> v1:172.21.15.192:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7ff49412bb80 con 0x7ff49406dc10 2024-06-03T08:36:32.114 INFO:teuthology.orchestra.run.smithi088.stderr:2024-06-03T08:36:32.071+0000 7f9bafd56640 1 librados: setting wanted keys 2024-06-03T08:36:32.114 INFO:teuthology.orchestra.run.smithi088.stderr:2024-06-03T08:36:32.072+0000 7f9bafd56640 1 librados: calling monclient init 2024-06-03T08:36:32.114 INFO:teuthology.orchestra.run.smithi088.stderr:2024-06-03T08:36:32.072+0000 7f9bafd56640 10 monclient: init 2024-06-03T08:36:32.115 INFO:teuthology.orchestra.run.smithi088.stderr:2024-06-03T08:36:32.073+0000 7f9bafd56640 10 monclient: _reopen_session rank -1 2024-06-03T08:36:32.115 INFO:teuthology.orchestra.run.smithi088.stderr:2024-06-03T08:36:32.073+0000 7f9bafd56640 10 monclient: _add_conns ranks=[0,1,2] 2024-06-03T08:36:32.115 INFO:teuthology.orchestra.run.smithi088.stderr:2024-06-03T08:36:32.073+0000 7f9bafd56640 10 monclient(hunting): picked mon.noname-a con 0x7f9ba80b43c0 addr v1:172.21.15.88:6789/0 2024-06-03T08:36:32.115 INFO:teuthology.orchestra.run.smithi088.stderr:2024-06-03T08:36:32.073+0000 7f9bafd56640 10 monclient(hunting): picked mon.noname-b con 0x7f9ba813e6c0 addr v1:172.21.15.192:6789/0 2024-06-03T08:36:32.115 INFO:teuthology.orchestra.run.smithi088.stderr:2024-06-03T08:36:32.073+0000 7f9bafd56640 10 monclient(hunting): picked mon.noname-c con 0x7f9ba814ae20 addr v1:172.21.15.88:6790/0 2024-06-03T08:36:32.115 INFO:teuthology.orchestra.run.smithi088.stderr:2024-06-03T08:36:32.073+0000 7f9bafd56640 1 -- --> v1:172.21.15.88:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f9ba81d9de0 con 0x7f9ba80b43c0 2024-06-03T08:36:32.115 INFO:teuthology.orchestra.run.smithi088.stderr:2024-06-03T08:36:32.073+0000 7f9bafd56640 1 -- --> v1:172.21.15.192:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f9ba8145850 con 0x7f9ba813e6c0 2024-06-03T08:36:32.115 INFO:teuthology.orchestra.run.smithi088.stderr:2024-06-03T08:36:32.073+0000 7f9bafd56640 1 -- --> v1:172.21.15.88:6790/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f9ba8139530 con 0x7f9ba814ae20 2024-06-03T08:36:32.115 INFO:teuthology.orchestra.run.smithi088.stderr:2024-06-03T08:36:32.073+0000 7f9bafd56640 10 monclient(hunting): _renew_subs 2024-06-03T08:36:32.115 INFO:teuthology.orchestra.run.smithi088.stderr:2024-06-03T08:36:32.073+0000 7f9bafd56640 10 monclient(hunting): authenticate will time out at 1247.883542s 2024-06-03T08:36:32.116 INFO:teuthology.orchestra.run.smithi088.stderr:2024-06-03T08:36:32.073+0000 7fd649ffb640 1 -- 172.21.15.88:0/2839181686 <== mon.2 v1:172.21.15.88:6790/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 3237230919 0 0) 0x7fd654133f70 con 0x7fd654141540 2024-06-03T08:36:32.116 INFO:teuthology.orchestra.run.smithi088.stderr:2024-06-03T08:36:32.073+0000 7fd649ffb640 10 monclient(hunting): _negotiate 2024-06-03T08:36:32.116 INFO:teuthology.orchestra.run.smithi088.stderr:2024-06-03T08:36:32.073+0000 7fd649ffb640 10 monclient(hunting): _init_auth method 2 2024-06-03T08:36:32.116 INFO:teuthology.orchestra.run.smithi088.stderr:2024-06-03T08:36:32.073+0000 7fd649ffb640 10 monclient(hunting): _init_auth creating new auth 2024-06-03T08:36:32.116 INFO:teuthology.orchestra.run.smithi088.stderr:2024-06-03T08:36:32.073+0000 7fd649ffb640 10 monclient(hunting): my global_id is 4340 2024-06-03T08:36:32.116 INFO:teuthology.orchestra.run.smithi088.stderr:2024-06-03T08:36:32.073+0000 7fd649ffb640 1 -- 172.21.15.88:0/2839181686 --> v1:172.21.15.88:6790/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7fd638004eb0 con 0x7fd654141540 2024-06-03T08:36:32.117 INFO:teuthology.orchestra.run.smithi088.stderr:2024-06-03T08:36:32.074+0000 7f791559a640 1 -- 172.21.15.88:0/1994185464 <== mon.1 v1:172.21.15.192:6789/0 4 ==== config(0 keys) v1 ==== 4+0+0 (unknown 0 0 0) 0x7f7910001b90 con 0x7f791814ae20 2024-06-03T08:36:32.117 INFO:teuthology.orchestra.run.smithi088.stderr:2024-06-03T08:36:32.074+0000 7f791559a640 10 monclient: handle_config config(0 keys) v1 2024-06-03T08:36:32.117 INFO:teuthology.orchestra.run.smithi088.stderr:2024-06-03T08:36:32.074+0000 7f791559a640 1 -- 172.21.15.88:0/1994185464 <== mon.1 v1:172.21.15.192:6789/0 5 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 2234374065 0 0) 0x7f7910001f20 con 0x7f791814ae20 2024-06-03T08:36:32.117 INFO:teuthology.orchestra.run.smithi088.stderr:2024-06-03T08:36:32.074+0000 7f791559a640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-06-03T08:36:32.117 INFO:teuthology.orchestra.run.smithi088.stderr:2024-06-03T08:36:32.074+0000 7f791559a640 10 monclient: got monmap 1 from mon.b (according to old e1) 2024-06-03T08:36:32.117 INFO:teuthology.orchestra.run.smithi088.stderr:2024-06-03T08:36:32.074+0000 7f791559a640 10 monclient: dump: 2024-06-03T08:36:32.117 INFO:teuthology.orchestra.run.smithi088.stderr:epoch 1 2024-06-03T08:36:32.117 INFO:teuthology.orchestra.run.smithi088.stderr:fsid ec3c3fdd-3ac6-4d7d-9412-683130ed8b04 2024-06-03T08:36:32.117 INFO:teuthology.orchestra.run.smithi088.stderr:last_changed 2024-06-03T08:34:31.885125+0000 2024-06-03T08:36:32.117 INFO:teuthology.orchestra.run.smithi088.stderr:created 2024-06-03T08:34:31.885125+0000 2024-06-03T08:36:32.117 INFO:teuthology.orchestra.run.smithi088.stderr:min_mon_release 18 (reef) 2024-06-03T08:36:32.117 INFO:teuthology.orchestra.run.smithi088.stderr:election_strategy: 3 2024-06-03T08:36:32.118 INFO:teuthology.orchestra.run.smithi088.stderr:0: v1:172.21.15.88:6789/0 mon.a 2024-06-03T08:36:32.118 INFO:teuthology.orchestra.run.smithi088.stderr:1: v1:172.21.15.192:6789/0 mon.b 2024-06-03T08:36:32.118 INFO:teuthology.orchestra.run.smithi088.stderr:2: v1:172.21.15.88:6790/0 mon.c 2024-06-03T08:36:32.118 INFO:teuthology.orchestra.run.smithi088.stderr: 2024-06-03T08:36:32.118 INFO:teuthology.orchestra.run.smithi088.stderr:2024-06-03T08:36:32.074+0000 7f791cb92640 10 monclient: get_monmap_and_config success 2024-06-03T08:36:32.118 INFO:teuthology.orchestra.run.smithi088.stderr:2024-06-03T08:36:32.074+0000 7f791cb92640 4 set_mon_vals no callback set 2024-06-03T08:36:32.118 INFO:teuthology.orchestra.run.smithi088.stderr:2024-06-03T08:36:32.074+0000 7fd344d53640 1 -- 172.21.15.88:0/2256678336 <== mon.1 v1:172.21.15.192:6789/0 4 ==== config(0 keys) v1 ==== 4+0+0 (unknown 0 0 0) 0x7fd34004e730 con 0x7fd3380327d0 2024-06-03T08:36:32.118 INFO:teuthology.orchestra.run.smithi088.stderr:2024-06-03T08:36:32.074+0000 7fd344d53640 10 monclient: handle_config config(0 keys) v1 2024-06-03T08:36:32.119 INFO:teuthology.orchestra.run.smithi088.stderr:2024-06-03T08:36:32.074+0000 7fd344d53640 1 -- 172.21.15.88:0/2256678336 <== mon.1 v1:172.21.15.192:6789/0 5 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 2234374065 0 0) 0x7fd3400008d0 con 0x7fd3380327d0 2024-06-03T08:36:32.119 INFO:teuthology.orchestra.run.smithi088.stderr:2024-06-03T08:36:32.074+0000 7fd344d53640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-06-03T08:36:32.119 INFO:teuthology.orchestra.run.smithi088.stderr:2024-06-03T08:36:32.074+0000 7fd344d53640 10 monclient: got monmap 1 from mon.b (according to old e1) 2024-06-03T08:36:32.119 INFO:teuthology.orchestra.run.smithi088.stderr:2024-06-03T08:36:32.074+0000 7fd344d53640 10 monclient: dump: 2024-06-03T08:36:32.119 INFO:teuthology.orchestra.run.smithi088.stderr:epoch 1 2024-06-03T08:36:32.119 INFO:teuthology.orchestra.run.smithi088.stderr:fsid ec3c3fdd-3ac6-4d7d-9412-683130ed8b04 2024-06-03T08:36:32.119 INFO:teuthology.orchestra.run.smithi088.stderr:last_changed 2024-06-03T08:34:31.885125+0000 2024-06-03T08:36:32.119 INFO:teuthology.orchestra.run.smithi088.stderr:created 2024-06-03T08:34:31.885125+0000 2024-06-03T08:36:32.119 INFO:teuthology.orchestra.run.smithi088.stderr:min_mon_release 18 (reef) 2024-06-03T08:36:32.119 INFO:teuthology.orchestra.run.smithi088.stderr:election_strategy: 3 2024-06-03T08:36:32.119 INFO:teuthology.orchestra.run.smithi088.stderr:0: v1:172.21.15.88:6789/0 mon.a 2024-06-03T08:36:32.119 INFO:teuthology.orchestra.run.smithi088.stderr:1: v1:172.21.15.192:6789/0 mon.b 2024-06-03T08:36:32.119 INFO:teuthology.orchestra.run.smithi088.stderr:2: v1:172.21.15.88:6790/0 mon.c 2024-06-03T08:36:32.120 INFO:teuthology.orchestra.run.smithi088.stderr: 2024-06-03T08:36:32.120 INFO:teuthology.orchestra.run.smithi088.stderr:2024-06-03T08:36:32.074+0000 7fd347fb9640 10 monclient: get_monmap_and_config success 2024-06-03T08:36:32.120 INFO:teuthology.orchestra.run.smithi088.stderr:2024-06-03T08:36:32.074+0000 7fd347fb9640 4 set_mon_vals no callback set 2024-06-03T08:36:32.120 INFO:teuthology.orchestra.run.smithi088.stderr:2024-06-03T08:36:32.074+0000 7fd649ffb640 1 -- 172.21.15.88:0/2839181686 <== mon.1 v1:172.21.15.192:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 1586614981 0 0) 0x7fd65413b4c0 con 0x7fd65413b700 2024-06-03T08:36:32.120 INFO:teuthology.orchestra.run.smithi088.stderr:2024-06-03T08:36:32.074+0000 7fd649ffb640 10 monclient(hunting): _negotiate 2024-06-03T08:36:32.120 INFO:teuthology.orchestra.run.smithi088.stderr:2024-06-03T08:36:32.074+0000 7fd649ffb640 10 monclient(hunting): _init_auth method 2 2024-06-03T08:36:32.120 INFO:teuthology.orchestra.run.smithi088.stderr:2024-06-03T08:36:32.074+0000 7fd649ffb640 10 monclient(hunting): _init_auth creating new auth 2024-06-03T08:36:32.120 INFO:teuthology.orchestra.run.smithi088.stderr:2024-06-03T08:36:32.074+0000 7fd649ffb640 10 monclient(hunting): my global_id is 4324 2024-06-03T08:36:32.120 INFO:teuthology.orchestra.run.smithi088.stderr:2024-06-03T08:36:32.074+0000 7fd649ffb640 1 -- 172.21.15.88:0/2839181686 --> v1:172.21.15.192:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7fd654133f70 con 0x7fd65413b700 2024-06-03T08:36:32.121 INFO:teuthology.orchestra.run.smithi088.stderr:2024-06-03T08:36:32.074+0000 7f4c1bfff640 1 -- 172.21.15.88:0/2099487910 <== mon.1 v1:172.21.15.192:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 1605422374 0 0) 0x7f4c3013e160 con 0x7f4c300a86f0 2024-06-03T08:36:32.121 INFO:teuthology.orchestra.run.smithi088.stderr:2024-06-03T08:36:32.074+0000 7f4c1bfff640 10 monclient(hunting): _negotiate 2024-06-03T08:36:32.121 INFO:teuthology.orchestra.run.smithi088.stderr:2024-06-03T08:36:32.074+0000 7f4c1bfff640 10 monclient(hunting): _init_auth method 2 2024-06-03T08:36:32.121 INFO:teuthology.orchestra.run.smithi088.stderr:2024-06-03T08:36:32.074+0000 7f4c1bfff640 10 monclient(hunting): _init_auth creating new auth 2024-06-03T08:36:32.121 INFO:teuthology.orchestra.run.smithi088.stderr:2024-06-03T08:36:32.074+0000 7f4c1bfff640 10 monclient(hunting): my global_id is 4327 2024-06-03T08:36:32.121 INFO:teuthology.orchestra.run.smithi088.stderr:2024-06-03T08:36:32.075+0000 7f4c1bfff640 1 -- 172.21.15.88:0/2099487910 --> v1:172.21.15.192:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f4c14004eb0 con 0x7f4c300a86f0 2024-06-03T08:36:32.121 INFO:teuthology.orchestra.run.smithi088.stderr:2024-06-03T08:36:32.075+0000 7fa0e9d9b640 1 -- 172.21.15.88:0/424874043 <== mon.1 v1:172.21.15.192:6789/0 4 ==== config(0 keys) v1 ==== 4+0+0 (unknown 0 0 0) 0x7fa0dc001830 con 0x7fa0ec0a8960 2024-06-03T08:36:32.122 INFO:teuthology.orchestra.run.smithi088.stderr:2024-06-03T08:36:32.075+0000 7fa0e9d9b640 10 monclient: handle_config config(0 keys) v1 2024-06-03T08:36:32.122 INFO:teuthology.orchestra.run.smithi088.stderr:2024-06-03T08:36:32.075+0000 7fa0f1016640 10 monclient: get_monmap_and_config success 2024-06-03T08:36:32.122 INFO:teuthology.orchestra.run.smithi088.stderr:2024-06-03T08:36:32.075+0000 7fa0f1016640 4 set_mon_vals no callback set 2024-06-03T08:36:32.122 INFO:teuthology.orchestra.run.smithi088.stderr:2024-06-03T08:36:32.075+0000 7f791cb92640 10 monclient: shutdown 2024-06-03T08:36:32.122 INFO:teuthology.orchestra.run.smithi088.stderr:2024-06-03T08:36:32.075+0000 7f791cb92640 20 monclient: shutdown discarding 0 pending message(s) 2024-06-03T08:36:32.122 INFO:teuthology.orchestra.run.smithi088.stderr:2024-06-03T08:36:32.075+0000 7f791cb92640 1 -- 172.21.15.88:0/1994185464 >> v1:172.21.15.192:6789/0 conn(0x7f791814ae20 legacy=0x7f791814d210 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-06-03T08:36:32.123 INFO:teuthology.orchestra.run.smithi088.stderr:2024-06-03T08:36:32.075+0000 7f791cb92640 1 -- 172.21.15.88:0/1994185464 shutdown_connections 2024-06-03T08:36:32.123 INFO:teuthology.orchestra.run.smithi088.stderr:2024-06-03T08:36:32.075+0000 7f791cb92640 1 -- 172.21.15.88:0/1994185464 >> 172.21.15.88:0/1994185464 conn(0x7f79180719d0 msgr2=0x7f7918071de0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-06-03T08:36:32.123 INFO:teuthology.orchestra.run.smithi088.stderr:2024-06-03T08:36:32.075+0000 7f791cb92640 1 -- 172.21.15.88:0/1994185464 shutdown_connections 2024-06-03T08:36:32.123 INFO:teuthology.orchestra.run.smithi088.stderr:2024-06-03T08:36:32.075+0000 7fd347fb9640 10 monclient: shutdown 2024-06-03T08:36:32.123 INFO:teuthology.orchestra.run.smithi088.stderr:2024-06-03T08:36:32.075+0000 7fd347fb9640 20 monclient: shutdown discarding 0 pending message(s) 2024-06-03T08:36:32.123 INFO:teuthology.orchestra.run.smithi088.stderr:2024-06-03T08:36:32.075+0000 7fd347fb9640 1 -- 172.21.15.88:0/2256678336 >> v1:172.21.15.192:6789/0 conn(0x7fd3380327d0 legacy=0x7fd3380c8200 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-06-03T08:36:32.123 INFO:teuthology.orchestra.run.smithi088.stderr:2024-06-03T08:36:32.075+0000 7f791cb92640 1 -- 172.21.15.88:0/1994185464 wait complete. 2024-06-03T08:36:32.124 INFO:teuthology.orchestra.run.smithi088.stderr:2024-06-03T08:36:32.075+0000 7fa0e9d9b640 1 -- 172.21.15.88:0/424874043 <== mon.1 v1:172.21.15.192:6789/0 5 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 2234374065 0 0) 0x7fa0dc001b40 con 0x7fa0ec0a8960 2024-06-03T08:36:32.124 INFO:teuthology.orchestra.run.smithi088.stderr:2024-06-03T08:36:32.075+0000 7fa0e9d9b640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-06-03T08:36:32.124 INFO:teuthology.orchestra.run.smithi088.stderr:2024-06-03T08:36:32.075+0000 7fa0e9d9b640 10 monclient: got monmap 1 from mon.b (according to old e1) 2024-06-03T08:36:32.124 INFO:teuthology.orchestra.run.smithi088.stderr:2024-06-03T08:36:32.075+0000 7fa0e9d9b640 10 monclient: dump: 2024-06-03T08:36:32.124 INFO:teuthology.orchestra.run.smithi088.stderr:epoch 1 2024-06-03T08:36:32.124 INFO:teuthology.orchestra.run.smithi088.stderr:fsid ec3c3fdd-3ac6-4d7d-9412-683130ed8b04 2024-06-03T08:36:32.124 INFO:teuthology.orchestra.run.smithi088.stderr:last_changed 2024-06-03T08:34:31.885125+0000 2024-06-03T08:36:32.124 INFO:teuthology.orchestra.run.smithi088.stderr:created 2024-06-03T08:34:31.885125+0000 2024-06-03T08:36:32.124 INFO:teuthology.orchestra.run.smithi088.stderr:min_mon_release 18 (reef) 2024-06-03T08:36:32.125 INFO:teuthology.orchestra.run.smithi088.stderr:election_strategy: 3 2024-06-03T08:36:32.125 INFO:teuthology.orchestra.run.smithi088.stderr:0: v1:172.21.15.88:6789/0 mon.a 2024-06-03T08:36:32.125 INFO:teuthology.orchestra.run.smithi088.stderr:1: v1:172.21.15.192:6789/0 mon.b 2024-06-03T08:36:32.125 INFO:teuthology.orchestra.run.smithi088.stderr:2: v1:172.21.15.88:6790/0 mon.c 2024-06-03T08:36:32.125 INFO:teuthology.orchestra.run.smithi088.stderr: 2024-06-03T08:36:32.125 INFO:teuthology.orchestra.run.smithi088.stderr:2024-06-03T08:36:32.075+0000 7fd347fb9640 1 -- 172.21.15.88:0/2256678336 shutdown_connections 2024-06-03T08:36:32.125 INFO:teuthology.orchestra.run.smithi088.stderr:2024-06-03T08:36:32.075+0000 7fd347fb9640 1 -- 172.21.15.88:0/2256678336 >> 172.21.15.88:0/2256678336 conn(0x7fd3380058f0 msgr2=0x7fd338005d00 unknown :-1 s=STATE_NONE l=0).mark_down 2024-06-03T08:36:32.125 INFO:teuthology.orchestra.run.smithi088.stderr:2024-06-03T08:36:32.075+0000 7fd347fb9640 1 -- 172.21.15.88:0/2256678336 shutdown_connections 2024-06-03T08:36:32.126 INFO:teuthology.orchestra.run.smithi088.stderr:2024-06-03T08:36:32.075+0000 7fd347fb9640 1 -- 172.21.15.88:0/2256678336 wait complete. 2024-06-03T08:36:32.126 INFO:teuthology.orchestra.run.smithi088.stderr:2024-06-03T08:36:32.075+0000 7fa0f1016640 10 monclient: shutdown 2024-06-03T08:36:32.126 INFO:teuthology.orchestra.run.smithi088.stderr:2024-06-03T08:36:32.075+0000 7fa0f1016640 20 monclient: shutdown discarding 0 pending message(s) 2024-06-03T08:36:32.126 INFO:teuthology.orchestra.run.smithi088.stderr:2024-06-03T08:36:32.075+0000 7fa0f1016640 1 -- 172.21.15.88:0/424874043 >> v1:172.21.15.192:6789/0 conn(0x7fa0ec0a8960 legacy=0x7fa0ec0a8d40 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-06-03T08:36:32.127 INFO:teuthology.orchestra.run.smithi088.stderr:2024-06-03T08:36:32.076+0000 7fa0f1016640 1 -- 172.21.15.88:0/424874043 shutdown_connections 2024-06-03T08:36:32.127 INFO:teuthology.orchestra.run.smithi088.stderr:2024-06-03T08:36:32.076+0000 7fa0f1016640 1 -- 172.21.15.88:0/424874043 >> 172.21.15.88:0/424874043 conn(0x7fa0ec0b70d0 msgr2=0x7fa0ec0b74e0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-06-03T08:36:32.127 INFO:teuthology.orchestra.run.smithi088.stderr:2024-06-03T08:36:32.076+0000 7fa0f1016640 1 -- 172.21.15.88:0/424874043 shutdown_connections 2024-06-03T08:36:32.127 INFO:teuthology.orchestra.run.smithi088.stderr:2024-06-03T08:36:32.076+0000 7fa0f1016640 1 -- 172.21.15.88:0/424874043 wait complete. 2024-06-03T08:36:32.127 INFO:teuthology.orchestra.run.smithi088.stderr:2024-06-03T08:36:32.076+0000 7fd347fb9640 10 monclient: build_initial_monmap 2024-06-03T08:36:32.127 INFO:teuthology.orchestra.run.smithi088.stderr:2024-06-03T08:36:32.076+0000 7fd347fb9640 10 monclient: monmap: 2024-06-03T08:36:32.127 INFO:teuthology.orchestra.run.smithi088.stderr:epoch 0 2024-06-03T08:36:32.127 INFO:teuthology.orchestra.run.smithi088.stderr:fsid 00000000-0000-0000-0000-000000000000 2024-06-03T08:36:32.127 INFO:teuthology.orchestra.run.smithi088.stderr:last_changed 0.000000 2024-06-03T08:36:32.128 INFO:teuthology.orchestra.run.smithi088.stderr:created 0.000000 2024-06-03T08:36:32.128 INFO:teuthology.orchestra.run.smithi088.stderr:min_mon_release 0 (unknown) 2024-06-03T08:36:32.128 INFO:teuthology.orchestra.run.smithi088.stderr:election_strategy: 1 2024-06-03T08:36:32.128 INFO:teuthology.orchestra.run.smithi088.stderr:0: v1:172.21.15.88:6789/0 mon.noname-a 2024-06-03T08:36:32.128 INFO:teuthology.orchestra.run.smithi088.stderr:1: v1:172.21.15.192:6789/0 mon.noname-b 2024-06-03T08:36:32.128 INFO:teuthology.orchestra.run.smithi088.stderr:2: v1:172.21.15.88:6790/0 mon.noname-c 2024-06-03T08:36:32.128 INFO:teuthology.orchestra.run.smithi088.stderr: 2024-06-03T08:36:32.128 INFO:teuthology.orchestra.run.smithi088.stderr:2024-06-03T08:36:32.076+0000 7f791cb92640 10 monclient: build_initial_monmap 2024-06-03T08:36:32.128 INFO:teuthology.orchestra.run.smithi088.stderr:2024-06-03T08:36:32.076+0000 7f791cb92640 10 monclient: monmap: 2024-06-03T08:36:32.128 INFO:teuthology.orchestra.run.smithi088.stderr:epoch 0 2024-06-03T08:36:32.128 INFO:teuthology.orchestra.run.smithi088.stderr:fsid 00000000-0000-0000-0000-000000000000 2024-06-03T08:36:32.129 INFO:teuthology.orchestra.run.smithi088.stderr:last_changed 0.000000 2024-06-03T08:36:32.129 INFO:teuthology.orchestra.run.smithi088.stderr:created 0.000000 2024-06-03T08:36:32.129 INFO:teuthology.orchestra.run.smithi088.stderr:min_mon_release 0 (unknown) 2024-06-03T08:36:32.129 INFO:teuthology.orchestra.run.smithi088.stderr:election_strategy: 1 2024-06-03T08:36:32.129 INFO:teuthology.orchestra.run.smithi088.stderr:0: v1:172.21.15.88:6789/0 mon.noname-a 2024-06-03T08:36:32.129 INFO:teuthology.orchestra.run.smithi088.stderr:1: v1:172.21.15.192:6789/0 mon.noname-b 2024-06-03T08:36:32.129 INFO:teuthology.orchestra.run.smithi088.stderr:2: v1:172.21.15.88:6790/0 mon.noname-c 2024-06-03T08:36:32.129 INFO:teuthology.orchestra.run.smithi088.stderr: 2024-06-03T08:36:32.130 INFO:teuthology.orchestra.run.smithi088.stderr:2024-06-03T08:36:32.076+0000 7fd347fb9640 1 librados: starting msgr at 2024-06-03T08:36:32.130 INFO:teuthology.orchestra.run.smithi088.stderr:2024-06-03T08:36:32.076+0000 7fd347fb9640 1 librados: starting objecter 2024-06-03T08:36:32.130 INFO:teuthology.orchestra.run.smithi088.stderr:2024-06-03T08:36:32.077+0000 7fd347fb9640 1 Processor -- start 2024-06-03T08:36:32.130 INFO:teuthology.orchestra.run.smithi088.stderr:2024-06-03T08:36:32.077+0000 7fd347fb9640 1 -- start start 2024-06-03T08:36:32.130 INFO:teuthology.orchestra.run.smithi088.stderr:2024-06-03T08:36:32.077+0000 7fd347fb9640 1 librados: setting wanted keys 2024-06-03T08:36:32.130 INFO:teuthology.orchestra.run.smithi088.stderr:2024-06-03T08:36:32.077+0000 7fd347fb9640 1 librados: calling monclient init 2024-06-03T08:36:32.130 INFO:teuthology.orchestra.run.smithi088.stderr:2024-06-03T08:36:32.077+0000 7fd347fb9640 10 monclient: init 2024-06-03T08:36:32.130 INFO:teuthology.orchestra.run.smithi088.stderr:2024-06-03T08:36:32.077+0000 7f791cb92640 1 librados: starting msgr at 2024-06-03T08:36:32.130 INFO:teuthology.orchestra.run.smithi088.stderr:2024-06-03T08:36:32.077+0000 7f791cb92640 1 librados: starting objecter 2024-06-03T08:36:32.131 INFO:teuthology.orchestra.run.smithi088.stderr:2024-06-03T08:36:32.077+0000 7fa0f1016640 10 monclient: build_initial_monmap 2024-06-03T08:36:32.131 INFO:teuthology.orchestra.run.smithi088.stderr:2024-06-03T08:36:32.077+0000 7fa0f1016640 10 monclient: monmap: 2024-06-03T08:36:32.131 INFO:teuthology.orchestra.run.smithi088.stderr:epoch 0 2024-06-03T08:36:32.131 INFO:teuthology.orchestra.run.smithi088.stderr:fsid 00000000-0000-0000-0000-000000000000 2024-06-03T08:36:32.131 INFO:teuthology.orchestra.run.smithi088.stderr:last_changed 0.000000 2024-06-03T08:36:32.131 INFO:teuthology.orchestra.run.smithi088.stderr:created 0.000000 2024-06-03T08:36:32.131 INFO:teuthology.orchestra.run.smithi088.stderr:min_mon_release 0 (unknown) 2024-06-03T08:36:32.131 INFO:teuthology.orchestra.run.smithi088.stderr:election_strategy: 1 2024-06-03T08:36:32.131 INFO:teuthology.orchestra.run.smithi088.stderr:0: v1:172.21.15.88:6789/0 mon.noname-a 2024-06-03T08:36:32.131 INFO:teuthology.orchestra.run.smithi088.stderr:1: v1:172.21.15.192:6789/0 mon.noname-b 2024-06-03T08:36:32.131 INFO:teuthology.orchestra.run.smithi088.stderr:2: v1:172.21.15.88:6790/0 mon.noname-c 2024-06-03T08:36:32.131 INFO:teuthology.orchestra.run.smithi088.stderr: 2024-06-03T08:36:32.132 INFO:teuthology.orchestra.run.smithi088.stderr:2024-06-03T08:36:32.077+0000 7f791cb92640 1 Processor -- start 2024-06-03T08:36:32.132 INFO:teuthology.orchestra.run.smithi088.stderr:2024-06-03T08:36:32.077+0000 7f791cb92640 1 -- start start 2024-06-03T08:36:32.132 INFO:teuthology.orchestra.run.smithi088.stderr:2024-06-03T08:36:32.077+0000 7f791cb92640 1 librados: setting wanted keys 2024-06-03T08:36:32.132 INFO:teuthology.orchestra.run.smithi088.stderr:2024-06-03T08:36:32.077+0000 7f791cb92640 1 librados: calling monclient init 2024-06-03T08:36:32.132 INFO:teuthology.orchestra.run.smithi088.stderr:2024-06-03T08:36:32.077+0000 7f791cb92640 10 monclient: init 2024-06-03T08:36:32.132 INFO:teuthology.orchestra.run.smithi088.stderr:2024-06-03T08:36:32.077+0000 7fa0f1016640 1 librados: starting msgr at 2024-06-03T08:36:32.133 INFO:teuthology.orchestra.run.smithi088.stderr:2024-06-03T08:36:32.077+0000 7fa0f1016640 1 librados: starting objecter 2024-06-03T08:36:32.133 INFO:teuthology.orchestra.run.smithi088.stderr:2024-06-03T08:36:32.077+0000 7fd347fb9640 10 monclient: _reopen_session rank -1 2024-06-03T08:36:32.133 INFO:teuthology.orchestra.run.smithi088.stderr:2024-06-03T08:36:32.077+0000 7fd347fb9640 10 monclient: _add_conns ranks=[0,2,1] 2024-06-03T08:36:32.133 INFO:teuthology.orchestra.run.smithi088.stderr:2024-06-03T08:36:32.077+0000 7fd347fb9640 10 monclient(hunting): picked mon.noname-a con 0x7fd338006410 addr v1:172.21.15.88:6789/0 2024-06-03T08:36:32.133 INFO:teuthology.orchestra.run.smithi088.stderr:2024-06-03T08:36:32.077+0000 7fd347fb9640 10 monclient(hunting): picked mon.noname-c con 0x7fd33802a640 addr v1:172.21.15.88:6790/0 2024-06-03T08:36:32.133 INFO:teuthology.orchestra.run.smithi088.stderr:2024-06-03T08:36:32.077+0000 7fd347fb9640 10 monclient(hunting): picked mon.noname-b con 0x7fd3380327d0 addr v1:172.21.15.192:6789/0 2024-06-03T08:36:32.133 INFO:teuthology.orchestra.run.smithi088.stderr:2024-06-03T08:36:32.077+0000 7fd347fb9640 1 -- --> v1:172.21.15.88:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fd338031470 con 0x7fd338006410 2024-06-03T08:36:32.134 INFO:teuthology.orchestra.run.smithi088.stderr:2024-06-03T08:36:32.077+0000 7fd347fb9640 1 -- --> v1:172.21.15.192:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fd338031160 con 0x7fd3380327d0 2024-06-03T08:36:32.134 INFO:teuthology.orchestra.run.smithi088.stderr:2024-06-03T08:36:32.077+0000 7fd347fb9640 1 -- --> v1:172.21.15.88:6790/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fd3380b2650 con 0x7fd33802a640 2024-06-03T08:36:32.134 INFO:teuthology.orchestra.run.smithi088.stderr:2024-06-03T08:36:32.077+0000 7fd347fb9640 10 monclient(hunting): _renew_subs 2024-06-03T08:36:32.134 INFO:teuthology.orchestra.run.smithi088.stderr:2024-06-03T08:36:32.077+0000 7fd347fb9640 10 monclient(hunting): authenticate will time out at 1247.887821s 2024-06-03T08:36:32.134 INFO:teuthology.orchestra.run.smithi088.stderr:2024-06-03T08:36:32.078+0000 7fa0f1016640 1 Processor -- start 2024-06-03T08:36:32.134 INFO:teuthology.orchestra.run.smithi088.stderr:2024-06-03T08:36:32.078+0000 7f9bae2f3640 1 --1- >> v1:172.21.15.88:6790/0 conn(0x7f9ba814ae20 0x7f9ba81e1380 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=1).handle_server_banner_and_identify peer v1:172.21.15.88:6790/0 says I am v1:172.21.15.88:54206/0 (socket says 172.21.15.88:54206) 2024-06-03T08:36:32.134 INFO:teuthology.orchestra.run.smithi088.stderr:2024-06-03T08:36:32.078+0000 7f9bae2f3640 1 -- 172.21.15.88:0/496575353 learned_addr learned my addr 172.21.15.88:0/496575353 (peer_addr_for_me v1:172.21.15.88:0/0) 2024-06-03T08:36:32.135 INFO:teuthology.orchestra.run.smithi088.stderr:2024-06-03T08:36:32.078+0000 7f791cb92640 10 monclient: _reopen_session rank -1 2024-06-03T08:36:32.135 INFO:teuthology.orchestra.run.smithi088.stderr:2024-06-03T08:36:32.078+0000 7f791cb92640 10 monclient: _add_conns ranks=[0,2,1] 2024-06-03T08:36:32.135 INFO:teuthology.orchestra.run.smithi088.stderr:2024-06-03T08:36:32.078+0000 7f791cb92640 10 monclient(hunting): picked mon.noname-a con 0x7f79180b43c0 addr v1:172.21.15.88:6789/0 2024-06-03T08:36:32.135 INFO:teuthology.orchestra.run.smithi088.stderr:2024-06-03T08:36:32.078+0000 7f791cb92640 10 monclient(hunting): picked mon.noname-c con 0x7f791813e6c0 addr v1:172.21.15.88:6790/0 2024-06-03T08:36:32.136 INFO:teuthology.orchestra.run.smithi088.stderr:2024-06-03T08:36:32.078+0000 7f791cb92640 10 monclient(hunting): picked mon.noname-b con 0x7f791814ae20 addr v1:172.21.15.192:6789/0 2024-06-03T08:36:32.136 INFO:teuthology.orchestra.run.smithi088.stderr:2024-06-03T08:36:32.078+0000 7f791cb92640 1 -- --> v1:172.21.15.88:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f79181442a0 con 0x7f79180b43c0 2024-06-03T08:36:32.136 INFO:teuthology.orchestra.run.smithi088.stderr:2024-06-03T08:36:32.078+0000 7f791cb92640 1 -- --> v1:172.21.15.192:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f7918140f70 con 0x7f791814ae20 2024-06-03T08:36:32.136 INFO:teuthology.orchestra.run.smithi088.stderr:2024-06-03T08:36:32.078+0000 7f791cb92640 1 -- --> v1:172.21.15.88:6790/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f7918135a10 con 0x7f791813e6c0 2024-06-03T08:36:32.136 INFO:teuthology.orchestra.run.smithi088.stderr:2024-06-03T08:36:32.078+0000 7f791cb92640 10 monclient(hunting): _renew_subs 2024-06-03T08:36:32.136 INFO:teuthology.orchestra.run.smithi088.stderr:2024-06-03T08:36:32.078+0000 7f791cb92640 10 monclient(hunting): authenticate will time out at 1247.888466s 2024-06-03T08:36:32.136 INFO:teuthology.orchestra.run.smithi088.stderr:2024-06-03T08:36:32.078+0000 7f9b9e7fc640 10 client.?.objecter ms_handle_connect 0x7f9ba814ae20 2024-06-03T08:36:32.136 INFO:teuthology.orchestra.run.smithi088.stderr:2024-06-03T08:36:32.078+0000 7f9b9e7fc640 10 client.?.objecter resend_mon_ops 2024-06-03T08:36:32.137 INFO:teuthology.orchestra.run.smithi088.stderr:2024-06-03T08:36:32.078+0000 7fa0f1016640 1 -- start start 2024-06-03T08:36:32.137 INFO:teuthology.orchestra.run.smithi088.stderr:2024-06-03T08:36:32.078+0000 7fa0f1016640 1 librados: setting wanted keys 2024-06-03T08:36:32.137 INFO:teuthology.orchestra.run.smithi088.stderr:2024-06-03T08:36:32.078+0000 7fa0f1016640 1 librados: calling monclient init 2024-06-03T08:36:32.137 INFO:teuthology.orchestra.run.smithi088.stderr:2024-06-03T08:36:32.078+0000 7fa0f1016640 10 monclient: init 2024-06-03T08:36:32.137 INFO:teuthology.orchestra.run.smithi088.stderr:2024-06-03T08:36:32.079+0000 7fa0f1016640 10 monclient: _reopen_session rank -1 2024-06-03T08:36:32.137 INFO:teuthology.orchestra.run.smithi088.stderr:2024-06-03T08:36:32.079+0000 7fa0f1016640 10 monclient: _add_conns ranks=[0,2,1] 2024-06-03T08:36:32.137 INFO:teuthology.orchestra.run.smithi088.stderr:2024-06-03T08:36:32.079+0000 7fd345554640 1 --1- >> v1:172.21.15.88:6790/0 conn(0x7fd33802a640 0x7fd338031990 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=1).handle_server_banner_and_identify peer v1:172.21.15.88:6790/0 says I am v1:172.21.15.88:54222/0 (socket says 172.21.15.88:54222) 2024-06-03T08:36:32.137 INFO:teuthology.orchestra.run.smithi088.stderr:2024-06-03T08:36:32.079+0000 7fd345554640 1 -- 172.21.15.88:0/3500826820 learned_addr learned my addr 172.21.15.88:0/3500826820 (peer_addr_for_me v1:172.21.15.88:0/0) 2024-06-03T08:36:32.138 INFO:teuthology.orchestra.run.smithi088.stderr:2024-06-03T08:36:32.079+0000 7fa0f1016640 10 monclient(hunting): picked mon.noname-a con 0x7fa0ec0a8960 addr v1:172.21.15.88:6789/0 2024-06-03T08:36:32.138 INFO:teuthology.orchestra.run.smithi088.stderr:2024-06-03T08:36:32.079+0000 7f4c1bfff640 10 client.?.objecter ms_handle_connect 0x7f4c3013ee50 2024-06-03T08:36:32.138 INFO:teuthology.orchestra.run.smithi088.stderr:2024-06-03T08:36:32.079+0000 7f4c1bfff640 10 client.?.objecter resend_mon_ops 2024-06-03T08:36:32.139 INFO:teuthology.orchestra.run.smithi088.stderr:2024-06-03T08:36:32.079+0000 7fa0f1016640 10 monclient(hunting): picked mon.noname-c con 0x7fa0ec13eec0 addr v1:172.21.15.88:6790/0 2024-06-03T08:36:32.139 INFO:teuthology.orchestra.run.smithi088.stderr:2024-06-03T08:36:32.079+0000 7fa0f1016640 10 monclient(hunting): picked mon.noname-b con 0x7fa0ec1450f0 addr v1:172.21.15.192:6789/0 2024-06-03T08:36:32.139 INFO:teuthology.orchestra.run.smithi088.stderr:2024-06-03T08:36:32.079+0000 7fa0f1016640 1 -- --> v1:172.21.15.88:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fa0ec1d6630 con 0x7fa0ec0a8960 2024-06-03T08:36:32.139 INFO:teuthology.orchestra.run.smithi088.stderr:2024-06-03T08:36:32.079+0000 7fa0f1016640 1 -- --> v1:172.21.15.192:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fa0ec13ff40 con 0x7fa0ec1450f0 2024-06-03T08:36:32.139 INFO:teuthology.orchestra.run.smithi088.stderr:2024-06-03T08:36:32.079+0000 7fa0f1016640 1 -- --> v1:172.21.15.88:6790/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fa0ec139c30 con 0x7fa0ec13eec0 2024-06-03T08:36:32.140 INFO:teuthology.orchestra.run.smithi088.stderr:2024-06-03T08:36:32.079+0000 7fa0f1016640 10 monclient(hunting): _renew_subs 2024-06-03T08:36:32.140 INFO:teuthology.orchestra.run.smithi088.stderr:2024-06-03T08:36:32.079+0000 7fa0f1016640 10 monclient(hunting): authenticate will time out at 1247.889411s 2024-06-03T08:36:32.140 INFO:teuthology.orchestra.run.smithi088.stderr:2024-06-03T08:36:32.079+0000 7f9b9e7fc640 10 client.?.objecter ms_handle_connect 0x7f9ba80b43c0 2024-06-03T08:36:32.140 INFO:teuthology.orchestra.run.smithi088.stderr:2024-06-03T08:36:32.079+0000 7f9b9e7fc640 10 client.?.objecter resend_mon_ops 2024-06-03T08:36:32.140 INFO:teuthology.orchestra.run.smithi088.stderr:2024-06-03T08:36:32.079+0000 7f7915d9b640 1 --1- >> v1:172.21.15.88:6790/0 conn(0x7f791813e6c0 0x7f7918135300 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=1).handle_server_banner_and_identify peer v1:172.21.15.88:6790/0 says I am v1:172.21.15.88:54226/0 (socket says 172.21.15.88:54226) 2024-06-03T08:36:32.140 INFO:teuthology.orchestra.run.smithi088.stderr:2024-06-03T08:36:32.079+0000 7f7915d9b640 1 -- 172.21.15.88:0/996211931 learned_addr learned my addr 172.21.15.88:0/996211931 (peer_addr_for_me v1:172.21.15.88:0/0) 2024-06-03T08:36:32.141 INFO:teuthology.orchestra.run.smithi088.stderr:2024-06-03T08:36:32.080+0000 7ff47bfff640 1 -- 172.21.15.88:0/2378444824 <== mon.0 v1:172.21.15.88:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 764+0+0 (unknown 348018694 0 0) 0x7ff480004eb0 con 0x7ff49412fd80 2024-06-03T08:36:32.141 INFO:teuthology.orchestra.run.smithi088.stderr:2024-06-03T08:36:32.080+0000 7ff47bfff640 1 -- 172.21.15.88:0/2378444824 --> v1:172.21.15.88:6789/0 -- auth(proto 2 165 bytes epoch 0) v1 -- 0x7ff494141020 con 0x7ff49412fd80 2024-06-03T08:36:32.141 INFO:teuthology.orchestra.run.smithi088.stderr:2024-06-03T08:36:32.081+0000 7ff47bfff640 1 -- 172.21.15.88:0/2378444824 <== mon.1 v1:172.21.15.192:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 764+0+0 (unknown 3263314587 0 0) 0x7ff49412bb80 con 0x7ff49406dc10 2024-06-03T08:36:32.141 INFO:teuthology.orchestra.run.smithi088.stderr:2024-06-03T08:36:32.081+0000 7ff47bfff640 1 -- 172.21.15.88:0/2378444824 --> v1:172.21.15.192:6789/0 -- auth(proto 2 165 bytes epoch 0) v1 -- 0x7ff480004eb0 con 0x7ff49406dc10 2024-06-03T08:36:32.141 INFO:teuthology.orchestra.run.smithi088.stderr:2024-06-03T08:36:32.081+0000 7fd3367fc640 10 client.?.objecter ms_handle_connect 0x7fd33802a640 2024-06-03T08:36:32.142 INFO:teuthology.orchestra.run.smithi088.stderr:2024-06-03T08:36:32.081+0000 7fd3367fc640 10 client.?.objecter resend_mon_ops 2024-06-03T08:36:32.142 INFO:teuthology.orchestra.run.smithi088.stderr:2024-06-03T08:36:32.081+0000 7f7906ffd640 10 client.?.objecter ms_handle_connect 0x7f791813e6c0 2024-06-03T08:36:32.142 INFO:teuthology.orchestra.run.smithi088.stderr:2024-06-03T08:36:32.081+0000 7f7906ffd640 10 client.?.objecter resend_mon_ops 2024-06-03T08:36:32.142 INFO:teuthology.orchestra.run.smithi088.stderr:2024-06-03T08:36:32.082+0000 7ff47bfff640 1 -- 172.21.15.88:0/2378444824 <== mon.0 v1:172.21.15.88:6789/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 2234374065 0 0) 0x7ff48c001cb0 con 0x7ff49412fd80 2024-06-03T08:36:32.142 INFO:teuthology.orchestra.run.smithi088.stderr:2024-06-03T08:36:32.082+0000 7ff47bfff640 10 monclient(hunting): handle_monmap mon_map magic: 0 v1 2024-06-03T08:36:32.142 INFO:teuthology.orchestra.run.smithi088.stderr:2024-06-03T08:36:32.082+0000 7ff47bfff640 10 monclient(hunting): got monmap 1 from mon.noname-a (according to old e1) 2024-06-03T08:36:32.142 INFO:teuthology.orchestra.run.smithi088.stderr:2024-06-03T08:36:32.082+0000 7ff47bfff640 10 monclient(hunting): dump: 2024-06-03T08:36:32.142 INFO:teuthology.orchestra.run.smithi088.stderr:epoch 1 2024-06-03T08:36:32.143 INFO:teuthology.orchestra.run.smithi088.stderr:fsid ec3c3fdd-3ac6-4d7d-9412-683130ed8b04 2024-06-03T08:36:32.143 INFO:teuthology.orchestra.run.smithi088.stderr:last_changed 2024-06-03T08:34:31.885125+0000 2024-06-03T08:36:32.143 INFO:teuthology.orchestra.run.smithi088.stderr:created 2024-06-03T08:34:31.885125+0000 2024-06-03T08:36:32.143 INFO:teuthology.orchestra.run.smithi088.stderr:min_mon_release 18 (reef) 2024-06-03T08:36:32.143 INFO:teuthology.orchestra.run.smithi088.stderr:election_strategy: 3 2024-06-03T08:36:32.143 INFO:teuthology.orchestra.run.smithi088.stderr:0: v1:172.21.15.88:6789/0 mon.a 2024-06-03T08:36:32.143 INFO:teuthology.orchestra.run.smithi088.stderr:1: v1:172.21.15.192:6789/0 mon.b 2024-06-03T08:36:32.143 INFO:teuthology.orchestra.run.smithi088.stderr:2: v1:172.21.15.88:6790/0 mon.c 2024-06-03T08:36:32.143 INFO:teuthology.orchestra.run.smithi088.stderr: 2024-06-03T08:36:32.143 INFO:teuthology.orchestra.run.smithi088.stderr:2024-06-03T08:36:32.082+0000 7ff47bfff640 10 monclient(hunting): _finish_auth 0 2024-06-03T08:36:32.143 INFO:teuthology.orchestra.run.smithi088.stderr:2024-06-03T08:36:32.085+0000 7f7906ffd640 10 client.?.objecter ms_handle_connect 0x7f79180b43c0 2024-06-03T08:36:32.143 INFO:teuthology.orchestra.run.smithi088.stderr:2024-06-03T08:36:32.085+0000 7f7906ffd640 10 client.?.objecter resend_mon_ops 2024-06-03T08:36:32.144 INFO:teuthology.orchestra.run.smithi088.stderr:2024-06-03T08:36:32.085+0000 7fd649ffb640 1 -- 172.21.15.88:0/2839181686 <== mon.0 v1:172.21.15.88:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 2931992326 0 0) 0x7fd65413bcb0 con 0x7fd65406dc10 2024-06-03T08:36:32.144 INFO:teuthology.orchestra.run.smithi088.stderr:2024-06-03T08:36:32.085+0000 7fd649ffb640 10 monclient(hunting): _negotiate 2024-06-03T08:36:32.144 INFO:teuthology.orchestra.run.smithi088.stderr:2024-06-03T08:36:32.085+0000 7fd649ffb640 10 monclient(hunting): _init_auth method 2 2024-06-03T08:36:32.144 INFO:teuthology.orchestra.run.smithi088.stderr:2024-06-03T08:36:32.085+0000 7fd649ffb640 10 monclient(hunting): _init_auth creating new auth 2024-06-03T08:36:32.144 INFO:teuthology.orchestra.run.smithi088.stderr:2024-06-03T08:36:32.085+0000 7fd649ffb640 10 monclient(hunting): my global_id is 4320 2024-06-03T08:36:32.144 INFO:teuthology.orchestra.run.smithi088.stderr:2024-06-03T08:36:32.085+0000 7fd649ffb640 1 -- 172.21.15.88:0/2839181686 --> v1:172.21.15.88:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7fd65413b4c0 con 0x7fd65406dc10 2024-06-03T08:36:32.145 INFO:teuthology.orchestra.run.smithi088.stderr:2024-06-03T08:36:32.086+0000 7fa0ea59c640 1 --1- >> v1:172.21.15.88:6790/0 conn(0x7fa0ec13eec0 0x7fa0ec141b70 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=1).handle_server_banner_and_identify peer v1:172.21.15.88:6790/0 says I am v1:172.21.15.88:54242/0 (socket says 172.21.15.88:54242) 2024-06-03T08:36:32.145 INFO:teuthology.orchestra.run.smithi088.stderr:2024-06-03T08:36:32.086+0000 7fa0ea59c640 1 -- 172.21.15.88:0/1282623662 learned_addr learned my addr 172.21.15.88:0/1282623662 (peer_addr_for_me v1:172.21.15.88:0/0) 2024-06-03T08:36:32.145 INFO:teuthology.orchestra.run.smithi088.stderr:2024-06-03T08:36:32.086+0000 7fa0d37fe640 10 client.?.objecter ms_handle_connect 0x7fa0ec13eec0 2024-06-03T08:36:32.145 INFO:teuthology.orchestra.run.smithi088.stderr:2024-06-03T08:36:32.086+0000 7fa0d37fe640 10 client.?.objecter resend_mon_ops 2024-06-03T08:36:32.145 INFO:teuthology.orchestra.run.smithi088.stderr:2024-06-03T08:36:32.089+0000 7f9b9e7fc640 10 client.?.objecter ms_handle_connect 0x7f9ba813e6c0 2024-06-03T08:36:32.145 INFO:teuthology.orchestra.run.smithi088.stderr:2024-06-03T08:36:32.089+0000 7f9b9e7fc640 10 client.?.objecter resend_mon_ops 2024-06-03T08:36:32.146 INFO:teuthology.orchestra.run.smithi088.stderr:2024-06-03T08:36:32.089+0000 7ff47bfff640 1 -- 172.21.15.88:0/2378444824 <== mon.1 v1:172.21.15.192:6789/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 2234374065 0 0) 0x7ff4880017d0 con 0x7ff49406dc10 2024-06-03T08:36:32.146 INFO:teuthology.orchestra.run.smithi088.stderr:2024-06-03T08:36:32.089+0000 7ff47bfff640 10 monclient(hunting): handle_monmap mon_map magic: 0 v1 2024-06-03T08:36:32.146 INFO:teuthology.orchestra.run.smithi088.stderr:2024-06-03T08:36:32.089+0000 7ff47bfff640 10 monclient(hunting): got monmap 1 from mon.b (according to old e1) 2024-06-03T08:36:32.146 INFO:teuthology.orchestra.run.smithi088.stderr:2024-06-03T08:36:32.089+0000 7ff47bfff640 10 monclient(hunting): dump: 2024-06-03T08:36:32.146 INFO:teuthology.orchestra.run.smithi088.stderr:epoch 1 2024-06-03T08:36:32.146 INFO:teuthology.orchestra.run.smithi088.stderr:fsid ec3c3fdd-3ac6-4d7d-9412-683130ed8b04 2024-06-03T08:36:32.146 INFO:teuthology.orchestra.run.smithi088.stderr:last_changed 2024-06-03T08:34:31.885125+0000 2024-06-03T08:36:32.146 INFO:teuthology.orchestra.run.smithi088.stderr:created 2024-06-03T08:34:31.885125+0000 2024-06-03T08:36:32.146 INFO:teuthology.orchestra.run.smithi088.stderr:min_mon_release 18 (reef) 2024-06-03T08:36:32.146 INFO:teuthology.orchestra.run.smithi088.stderr:election_strategy: 3 2024-06-03T08:36:32.147 INFO:teuthology.orchestra.run.smithi088.stderr:0: v1:172.21.15.88:6789/0 mon.a 2024-06-03T08:36:32.147 INFO:teuthology.orchestra.run.smithi088.stderr:1: v1:172.21.15.192:6789/0 mon.b 2024-06-03T08:36:32.147 INFO:teuthology.orchestra.run.smithi088.stderr:2: v1:172.21.15.88:6790/0 mon.c 2024-06-03T08:36:32.147 INFO:teuthology.orchestra.run.smithi088.stderr: 2024-06-03T08:36:32.147 INFO:teuthology.orchestra.run.smithi088.stderr:2024-06-03T08:36:32.091+0000 7f4c1bfff640 1 -- 172.21.15.88:0/2099487910 <== mon.0 v1:172.21.15.88:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 3904319345 0 0) 0x7f4c30140630 con 0x7f4c3013ee50 2024-06-03T08:36:32.148 INFO:teuthology.orchestra.run.smithi088.stderr:2024-06-03T08:36:32.091+0000 7f4c1bfff640 10 monclient(hunting): _negotiate 2024-06-03T08:36:32.148 INFO:teuthology.orchestra.run.smithi088.stderr:2024-06-03T08:36:32.091+0000 7f4c1bfff640 10 monclient(hunting): _init_auth method 2 2024-06-03T08:36:32.148 INFO:teuthology.orchestra.run.smithi088.stderr:2024-06-03T08:36:32.091+0000 7f4c1bfff640 10 monclient(hunting): _init_auth creating new auth 2024-06-03T08:36:32.148 INFO:teuthology.orchestra.run.smithi088.stderr:2024-06-03T08:36:32.091+0000 7f4c1bfff640 10 monclient(hunting): my global_id is 4323 2024-06-03T08:36:32.148 INFO:teuthology.orchestra.run.smithi088.stderr:2024-06-03T08:36:32.091+0000 7f4c1bfff640 1 -- 172.21.15.88:0/2099487910 --> v1:172.21.15.88:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f4c3013e160 con 0x7f4c3013ee50 2024-06-03T08:36:32.148 INFO:teuthology.orchestra.run.smithi088.stderr:2024-06-03T08:36:32.093+0000 7fd3367fc640 10 client.?.objecter ms_handle_connect 0x7fd338006410 2024-06-03T08:36:32.148 INFO:teuthology.orchestra.run.smithi088.stderr:2024-06-03T08:36:32.093+0000 7fd3367fc640 10 client.?.objecter resend_mon_ops 2024-06-03T08:36:32.149 INFO:teuthology.orchestra.run.smithi088.stderr:2024-06-03T08:36:32.093+0000 7fd649ffb640 1 -- 172.21.15.88:0/2839181686 <== mon.1 v1:172.21.15.192:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 764+0+0 (unknown 1813676988 0 0) 0x7fd654133f70 con 0x7fd65413b700 2024-06-03T08:36:32.149 INFO:teuthology.orchestra.run.smithi088.stderr:2024-06-03T08:36:32.093+0000 7fd649ffb640 1 -- 172.21.15.88:0/2839181686 --> v1:172.21.15.192:6789/0 -- auth(proto 2 165 bytes epoch 0) v1 -- 0x7fd65413bcb0 con 0x7fd65413b700 2024-06-03T08:36:32.149 INFO:teuthology.orchestra.run.smithi088.stderr:2024-06-03T08:36:32.093+0000 7fd649ffb640 1 -- 172.21.15.88:0/2839181686 <== mon.1 v1:172.21.15.192:6789/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 2234374065 0 0) 0x7fd64c001b50 con 0x7fd65413b700 2024-06-03T08:36:32.149 INFO:teuthology.orchestra.run.smithi088.stderr:2024-06-03T08:36:32.093+0000 7fd649ffb640 10 monclient(hunting): handle_monmap mon_map magic: 0 v1 2024-06-03T08:36:32.149 INFO:teuthology.orchestra.run.smithi088.stderr:2024-06-03T08:36:32.093+0000 7fd649ffb640 10 monclient(hunting): got monmap 1 from mon.noname-b (according to old e1) 2024-06-03T08:36:32.150 INFO:teuthology.orchestra.run.smithi088.stderr:2024-06-03T08:36:32.093+0000 7fd649ffb640 10 monclient(hunting): dump: 2024-06-03T08:36:32.150 INFO:teuthology.orchestra.run.smithi088.stderr:epoch 1 2024-06-03T08:36:32.150 INFO:teuthology.orchestra.run.smithi088.stderr:fsid ec3c3fdd-3ac6-4d7d-9412-683130ed8b04 2024-06-03T08:36:32.150 INFO:teuthology.orchestra.run.smithi088.stderr:last_changed 2024-06-03T08:34:31.885125+0000 2024-06-03T08:36:32.150 INFO:teuthology.orchestra.run.smithi088.stderr:created 2024-06-03T08:34:31.885125+0000 2024-06-03T08:36:32.150 INFO:teuthology.orchestra.run.smithi088.stderr:min_mon_release 18 (reef) 2024-06-03T08:36:32.150 INFO:teuthology.orchestra.run.smithi088.stderr:election_strategy: 3 2024-06-03T08:36:32.150 INFO:teuthology.orchestra.run.smithi088.stderr:0: v1:172.21.15.88:6789/0 mon.a 2024-06-03T08:36:32.150 INFO:teuthology.orchestra.run.smithi088.stderr:1: v1:172.21.15.192:6789/0 mon.b 2024-06-03T08:36:32.150 INFO:teuthology.orchestra.run.smithi088.stderr:2: v1:172.21.15.88:6790/0 mon.c 2024-06-03T08:36:32.150 INFO:teuthology.orchestra.run.smithi088.stderr: 2024-06-03T08:36:32.150 INFO:teuthology.orchestra.run.smithi088.stderr:2024-06-03T08:36:32.093+0000 7fd649ffb640 10 monclient(hunting): _finish_auth 0 2024-06-03T08:36:32.150 INFO:teuthology.orchestra.run.smithi088.stderr:2024-06-03T08:36:32.094+0000 7fd649ffb640 1 -- 172.21.15.88:0/2839181686 <== mon.0 v1:172.21.15.88:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 764+0+0 (unknown 3468774513 0 0) 0x7fd65413b4c0 con 0x7fd65406dc10 2024-06-03T08:36:32.150 INFO:teuthology.orchestra.run.smithi088.stderr:2024-06-03T08:36:32.094+0000 7fd649ffb640 1 -- 172.21.15.88:0/2839181686 --> v1:172.21.15.88:6789/0 -- auth(proto 2 165 bytes epoch 0) v1 -- 0x7fd654133f70 con 0x7fd65406dc10 2024-06-03T08:36:32.151 INFO:teuthology.orchestra.run.smithi088.stderr:2024-06-03T08:36:32.094+0000 7fd649ffb640 1 -- 172.21.15.88:0/2839181686 <== mon.0 v1:172.21.15.88:6789/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 2234374065 0 0) 0x7fd6440018d0 con 0x7fd65406dc10 2024-06-03T08:36:32.151 INFO:teuthology.orchestra.run.smithi088.stderr:2024-06-03T08:36:32.094+0000 7fd649ffb640 10 monclient(hunting): handle_monmap mon_map magic: 0 v1 2024-06-03T08:36:32.151 INFO:teuthology.orchestra.run.smithi088.stderr:2024-06-03T08:36:32.094+0000 7fd649ffb640 10 monclient(hunting): got monmap 1 from mon.a (according to old e1) 2024-06-03T08:36:32.151 INFO:teuthology.orchestra.run.smithi088.stderr:2024-06-03T08:36:32.094+0000 7fd649ffb640 10 monclient(hunting): dump: 2024-06-03T08:36:32.151 INFO:teuthology.orchestra.run.smithi088.stderr:epoch 1 2024-06-03T08:36:32.151 INFO:teuthology.orchestra.run.smithi088.stderr:fsid ec3c3fdd-3ac6-4d7d-9412-683130ed8b04 2024-06-03T08:36:32.151 INFO:teuthology.orchestra.run.smithi088.stderr:last_changed 2024-06-03T08:34:31.885125+0000 2024-06-03T08:36:32.151 INFO:teuthology.orchestra.run.smithi088.stderr:created 2024-06-03T08:34:31.885125+0000 2024-06-03T08:36:32.151 INFO:teuthology.orchestra.run.smithi088.stderr:min_mon_release 18 (reef) 2024-06-03T08:36:32.151 INFO:teuthology.orchestra.run.smithi088.stderr:election_strategy: 3 2024-06-03T08:36:32.151 INFO:teuthology.orchestra.run.smithi088.stderr:0: v1:172.21.15.88:6789/0 mon.a 2024-06-03T08:36:32.151 INFO:teuthology.orchestra.run.smithi088.stderr:1: v1:172.21.15.192:6789/0 mon.b 2024-06-03T08:36:32.151 INFO:teuthology.orchestra.run.smithi088.stderr:2: v1:172.21.15.88:6790/0 mon.c 2024-06-03T08:36:32.152 INFO:teuthology.orchestra.run.smithi088.stderr: 2024-06-03T08:36:32.152 INFO:teuthology.orchestra.run.smithi088.stderr:2024-06-03T08:36:32.096+0000 7f9b9e7fc640 1 -- 172.21.15.88:0/496575353 <== mon.0 v1:172.21.15.88:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 2308443776 0 0) 0x7f9ba81d9de0 con 0x7f9ba80b43c0 2024-06-03T08:36:32.152 INFO:teuthology.orchestra.run.smithi088.stderr:2024-06-03T08:36:32.096+0000 7f9b9e7fc640 10 monclient(hunting): _negotiate 2024-06-03T08:36:32.152 INFO:teuthology.orchestra.run.smithi088.stderr:2024-06-03T08:36:32.096+0000 7f9b9e7fc640 10 monclient(hunting): _init_auth method 2 2024-06-03T08:36:32.152 INFO:teuthology.orchestra.run.smithi088.stderr:2024-06-03T08:36:32.096+0000 7f9b9e7fc640 10 monclient(hunting): _init_auth creating new auth 2024-06-03T08:36:32.152 INFO:teuthology.orchestra.run.smithi088.stderr:2024-06-03T08:36:32.096+0000 7f9b9e7fc640 10 monclient(hunting): my global_id is 4326 2024-06-03T08:36:32.152 INFO:teuthology.orchestra.run.smithi088.stderr:2024-06-03T08:36:32.096+0000 7f9b9e7fc640 1 -- 172.21.15.88:0/496575353 --> v1:172.21.15.88:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f9b8c004eb0 con 0x7f9ba80b43c0 2024-06-03T08:36:32.152 INFO:teuthology.orchestra.run.smithi088.stderr:2024-06-03T08:36:32.098+0000 7fd3367fc640 10 client.?.objecter ms_handle_connect 0x7fd3380327d0 2024-06-03T08:36:32.153 INFO:teuthology.orchestra.run.smithi088.stderr:2024-06-03T08:36:32.098+0000 7fd3367fc640 10 client.?.objecter resend_mon_ops 2024-06-03T08:36:32.153 INFO:teuthology.orchestra.run.smithi088.stderr:2024-06-03T08:36:32.099+0000 7f4c1bfff640 1 -- 172.21.15.88:0/2099487910 <== mon.1 v1:172.21.15.192:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 764+0+0 (unknown 2599058848 0 0) 0x7f4c14004eb0 con 0x7f4c300a86f0 2024-06-03T08:36:32.153 INFO:teuthology.orchestra.run.smithi088.stderr:2024-06-03T08:36:32.099+0000 7f4c1bfff640 1 -- 172.21.15.88:0/2099487910 --> v1:172.21.15.192:6789/0 -- auth(proto 2 165 bytes epoch 0) v1 -- 0x7f4c30140630 con 0x7f4c300a86f0 2024-06-03T08:36:32.153 INFO:teuthology.orchestra.run.smithi088.stderr:2024-06-03T08:36:32.099+0000 7f4c1bfff640 1 -- 172.21.15.88:0/2099487910 <== mon.1 v1:172.21.15.192:6789/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 2234374065 0 0) 0x7f4c24001b40 con 0x7f4c300a86f0 2024-06-03T08:36:32.154 INFO:teuthology.orchestra.run.smithi088.stderr:2024-06-03T08:36:32.099+0000 7f4c1bfff640 10 monclient(hunting): handle_monmap mon_map magic: 0 v1 2024-06-03T08:36:32.154 INFO:teuthology.orchestra.run.smithi088.stderr:2024-06-03T08:36:32.099+0000 7f4c1bfff640 10 monclient(hunting): got monmap 1 from mon.noname-b (according to old e1) 2024-06-03T08:36:32.154 INFO:teuthology.orchestra.run.smithi088.stderr:2024-06-03T08:36:32.099+0000 7f4c1bfff640 10 monclient(hunting): dump: 2024-06-03T08:36:32.154 INFO:teuthology.orchestra.run.smithi088.stderr:epoch 1 2024-06-03T08:36:32.154 INFO:teuthology.orchestra.run.smithi088.stderr:fsid ec3c3fdd-3ac6-4d7d-9412-683130ed8b04 2024-06-03T08:36:32.154 INFO:teuthology.orchestra.run.smithi088.stderr:last_changed 2024-06-03T08:34:31.885125+0000 2024-06-03T08:36:32.154 INFO:teuthology.orchestra.run.smithi088.stderr:created 2024-06-03T08:34:31.885125+0000 2024-06-03T08:36:32.154 INFO:teuthology.orchestra.run.smithi088.stderr:min_mon_release 18 (reef) 2024-06-03T08:36:32.154 INFO:teuthology.orchestra.run.smithi088.stderr:election_strategy: 3 2024-06-03T08:36:32.154 INFO:teuthology.orchestra.run.smithi088.stderr:0: v1:172.21.15.88:6789/0 mon.a 2024-06-03T08:36:32.154 INFO:teuthology.orchestra.run.smithi088.stderr:1: v1:172.21.15.192:6789/0 mon.b 2024-06-03T08:36:32.154 INFO:teuthology.orchestra.run.smithi088.stderr:2: v1:172.21.15.88:6790/0 mon.c 2024-06-03T08:36:32.154 INFO:teuthology.orchestra.run.smithi088.stderr: 2024-06-03T08:36:32.155 INFO:teuthology.orchestra.run.smithi088.stderr:2024-06-03T08:36:32.099+0000 7f4c1bfff640 10 monclient(hunting): _finish_auth 0 2024-06-03T08:36:32.155 INFO:teuthology.orchestra.run.smithi088.stderr:2024-06-03T08:36:32.100+0000 7f9b9e7fc640 1 -- 172.21.15.88:0/496575353 <== mon.1 v1:172.21.15.192:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 351425125 0 0) 0x7f9ba8145850 con 0x7f9ba813e6c0 2024-06-03T08:36:32.155 INFO:teuthology.orchestra.run.smithi088.stderr:2024-06-03T08:36:32.100+0000 7f9b9e7fc640 10 monclient(hunting): _negotiate 2024-06-03T08:36:32.155 INFO:teuthology.orchestra.run.smithi088.stderr:2024-06-03T08:36:32.100+0000 7f9b9e7fc640 10 monclient(hunting): _init_auth method 2 2024-06-03T08:36:32.155 INFO:teuthology.orchestra.run.smithi088.stderr:2024-06-03T08:36:32.100+0000 7f9b9e7fc640 10 monclient(hunting): _init_auth creating new auth 2024-06-03T08:36:32.155 INFO:teuthology.orchestra.run.smithi088.stderr:2024-06-03T08:36:32.100+0000 7f9b9e7fc640 10 monclient(hunting): my global_id is 4330 2024-06-03T08:36:32.155 INFO:teuthology.orchestra.run.smithi088.stderr:2024-06-03T08:36:32.100+0000 7f9b9e7fc640 1 -- 172.21.15.88:0/496575353 --> v1:172.21.15.192:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f9ba81d9de0 con 0x7f9ba813e6c0 2024-06-03T08:36:32.156 INFO:teuthology.orchestra.run.smithi088.stderr:2024-06-03T08:36:32.101+0000 7fa0d37fe640 10 client.?.objecter ms_handle_connect 0x7fa0ec1450f0 2024-06-03T08:36:32.156 INFO:teuthology.orchestra.run.smithi088.stderr:2024-06-03T08:36:32.101+0000 7fa0d37fe640 10 client.?.objecter resend_mon_ops 2024-06-03T08:36:32.156 INFO:teuthology.orchestra.run.smithi088.stderr:2024-06-03T08:36:32.102+0000 7f7906ffd640 10 client.?.objecter ms_handle_connect 0x7f791814ae20 2024-06-03T08:36:32.156 INFO:teuthology.orchestra.run.smithi088.stderr:2024-06-03T08:36:32.102+0000 7f7906ffd640 10 client.?.objecter resend_mon_ops 2024-06-03T08:36:32.156 INFO:teuthology.orchestra.run.smithi088.stderr:2024-06-03T08:36:32.105+0000 7f7906ffd640 1 -- 172.21.15.88:0/996211931 <== mon.0 v1:172.21.15.88:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 1814926836 0 0) 0x7f79181442a0 con 0x7f79180b43c0 2024-06-03T08:36:32.156 INFO:teuthology.orchestra.run.smithi088.stderr:2024-06-03T08:36:32.105+0000 7f7906ffd640 10 monclient(hunting): _negotiate 2024-06-03T08:36:32.157 INFO:teuthology.orchestra.run.smithi088.stderr:2024-06-03T08:36:32.105+0000 7f7906ffd640 10 monclient(hunting): _init_auth method 2 2024-06-03T08:36:32.157 INFO:teuthology.orchestra.run.smithi088.stderr:2024-06-03T08:36:32.105+0000 7f7906ffd640 10 monclient(hunting): _init_auth creating new auth 2024-06-03T08:36:32.157 INFO:teuthology.orchestra.run.smithi088.stderr:2024-06-03T08:36:32.105+0000 7f7906ffd640 10 monclient(hunting): my global_id is 4329 2024-06-03T08:36:32.157 INFO:teuthology.orchestra.run.smithi088.stderr:2024-06-03T08:36:32.105+0000 7f7906ffd640 1 -- 172.21.15.88:0/996211931 --> v1:172.21.15.88:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f78f8004eb0 con 0x7f79180b43c0 2024-06-03T08:36:32.157 INFO:teuthology.orchestra.run.smithi088.stderr:2024-06-03T08:36:32.108+0000 7fd3367fc640 1 -- 172.21.15.88:0/3500826820 <== mon.0 v1:172.21.15.88:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 3700535625 0 0) 0x7fd338031470 con 0x7fd338006410 2024-06-03T08:36:32.157 INFO:teuthology.orchestra.run.smithi088.stderr:2024-06-03T08:36:32.108+0000 7fd3367fc640 10 monclient(hunting): _negotiate 2024-06-03T08:36:32.157 INFO:teuthology.orchestra.run.smithi088.stderr:2024-06-03T08:36:32.108+0000 7fd3367fc640 10 monclient(hunting): _init_auth method 2 2024-06-03T08:36:32.157 INFO:teuthology.orchestra.run.smithi088.stderr:2024-06-03T08:36:32.108+0000 7fd3367fc640 10 monclient(hunting): _init_auth creating new auth 2024-06-03T08:36:32.158 INFO:teuthology.orchestra.run.smithi088.stderr:2024-06-03T08:36:32.108+0000 7fd3367fc640 10 monclient(hunting): my global_id is 4332 2024-06-03T08:36:32.158 INFO:teuthology.orchestra.run.smithi088.stderr:2024-06-03T08:36:32.108+0000 7fd3367fc640 1 -- 172.21.15.88:0/3500826820 --> v1:172.21.15.88:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7fd324004eb0 con 0x7fd338006410 2024-06-03T08:36:32.158 INFO:teuthology.orchestra.run.smithi088.stderr:2024-06-03T08:36:32.112+0000 7fa0d37fe640 10 client.?.objecter ms_handle_connect 0x7fa0ec0a8960 2024-06-03T08:36:32.158 INFO:teuthology.orchestra.run.smithi088.stderr:2024-06-03T08:36:32.112+0000 7fa0d37fe640 10 client.?.objecter resend_mon_ops 2024-06-03T08:36:32.159 INFO:teuthology.orchestra.run.smithi088.stderr:2024-06-03T08:36:32.112+0000 7f9b9e7fc640 1 -- 172.21.15.88:0/496575353 <== mon.0 v1:172.21.15.88:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 764+0+0 (unknown 990137747 0 0) 0x7f9b8c004eb0 con 0x7f9ba80b43c0 2024-06-03T08:36:32.159 INFO:teuthology.orchestra.run.smithi088.stderr:2024-06-03T08:36:32.112+0000 7f9b9e7fc640 1 -- 172.21.15.88:0/496575353 --> v1:172.21.15.88:6789/0 -- auth(proto 2 165 bytes epoch 0) v1 -- 0x7f9ba8145850 con 0x7f9ba80b43c0 2024-06-03T08:36:32.159 INFO:teuthology.orchestra.run.smithi088.stderr:2024-06-03T08:36:32.112+0000 7f9b9e7fc640 1 -- 172.21.15.88:0/496575353 <== mon.0 v1:172.21.15.88:6789/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 2234374065 0 0) 0x7f9ba4001550 con 0x7f9ba80b43c0 2024-06-03T08:36:32.159 INFO:teuthology.orchestra.run.smithi088.stderr:2024-06-03T08:36:32.112+0000 7f9b9e7fc640 10 monclient(hunting): handle_monmap mon_map magic: 0 v1 2024-06-03T08:36:32.159 INFO:teuthology.orchestra.run.smithi088.stderr:2024-06-03T08:36:32.112+0000 7f9b9e7fc640 10 monclient(hunting): got monmap 1 from mon.noname-a (according to old e1) 2024-06-03T08:36:32.159 INFO:teuthology.orchestra.run.smithi088.stderr:2024-06-03T08:36:32.112+0000 7f9b9e7fc640 10 monclient(hunting): dump: 2024-06-03T08:36:32.159 INFO:teuthology.orchestra.run.smithi088.stderr:epoch 1 2024-06-03T08:36:32.159 INFO:teuthology.orchestra.run.smithi088.stderr:fsid ec3c3fdd-3ac6-4d7d-9412-683130ed8b04 2024-06-03T08:36:32.159 INFO:teuthology.orchestra.run.smithi088.stderr:last_changed 2024-06-03T08:34:31.885125+0000 2024-06-03T08:36:32.159 INFO:teuthology.orchestra.run.smithi088.stderr:created 2024-06-03T08:34:31.885125+0000 2024-06-03T08:36:32.159 INFO:teuthology.orchestra.run.smithi088.stderr:min_mon_release 18 (reef) 2024-06-03T08:36:32.159 INFO:teuthology.orchestra.run.smithi088.stderr:election_strategy: 3 2024-06-03T08:36:32.159 INFO:teuthology.orchestra.run.smithi088.stderr:0: v1:172.21.15.88:6789/0 mon.a 2024-06-03T08:36:32.160 INFO:teuthology.orchestra.run.smithi088.stderr:1: v1:172.21.15.192:6789/0 mon.b 2024-06-03T08:36:32.160 INFO:teuthology.orchestra.run.smithi088.stderr:2: v1:172.21.15.88:6790/0 mon.c 2024-06-03T08:36:32.160 INFO:teuthology.orchestra.run.smithi088.stderr: 2024-06-03T08:36:32.160 INFO:teuthology.orchestra.run.smithi088.stderr:2024-06-03T08:36:32.112+0000 7f9b9e7fc640 10 monclient(hunting): _finish_auth 0 2024-06-03T08:36:32.162 INFO:teuthology.orchestra.run.smithi088.stderr:2024-06-03T08:36:32.113+0000 7ff47bfff640 1 -- 172.21.15.88:0/2378444824 <== mon.0 v1:172.21.15.88:6789/0 4 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 580+0+0 (unknown 3115627372 0 0) 0x7ff494141020 con 0x7ff49412fd80 2024-06-03T08:36:32.162 INFO:teuthology.orchestra.run.smithi088.stderr:2024-06-03T08:36:32.113+0000 7ff47bfff640 1 -- 172.21.15.88:0/2378444824 >> v1:172.21.15.88:6790/0 conn(0x7ff494141540 legacy=0x7ff49412f670 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-06-03T08:36:32.162 INFO:teuthology.orchestra.run.smithi088.stderr:2024-06-03T08:36:32.113+0000 7ff47bfff640 1 -- 172.21.15.88:0/2378444824 >> v1:172.21.15.192:6789/0 conn(0x7ff49406dc10 legacy=0x7ff49412a030 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-06-03T08:36:32.162 INFO:teuthology.orchestra.run.smithi088.stderr:2024-06-03T08:36:32.113+0000 7ff47bfff640 10 monclient: _finish_hunting 0 2024-06-03T08:36:32.162 INFO:teuthology.orchestra.run.smithi088.stderr:2024-06-03T08:36:32.113+0000 7ff47bfff640 1 monclient: found mon.a 2024-06-03T08:36:32.162 INFO:teuthology.orchestra.run.smithi088.stderr:2024-06-03T08:36:32.113+0000 7ff47bfff640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-06-03T08:36:32.162 INFO:teuthology.orchestra.run.smithi088.stderr:2024-06-03T08:36:32.113+0000 7ff47bfff640 10 monclient: _send_mon_message to mon.a at v1:172.21.15.88:6789/0 2024-06-03T08:36:32.162 INFO:teuthology.orchestra.run.smithi088.stderr:2024-06-03T08:36:32.113+0000 7ff47bfff640 1 -- 172.21.15.88:0/2378444824 --> v1:172.21.15.88:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7ff494130230 con 0x7ff49412fd80 2024-06-03T08:36:32.162 INFO:teuthology.orchestra.run.smithi088.stderr:2024-06-03T08:36:32.113+0000 7ff47bfff640 10 monclient: _finish_auth 0 2024-06-03T08:36:32.163 INFO:teuthology.orchestra.run.smithi088.stderr:2024-06-03T08:36:32.113+0000 7ff47bfff640 10 monclient: _check_auth_tickets 2024-06-03T08:36:32.163 INFO:teuthology.orchestra.run.smithi088.stderr:2024-06-03T08:36:32.113+0000 7ff47bfff640 20 monclient: _check_auth_rotating not needed by client.admin 2024-06-03T08:36:32.163 INFO:teuthology.orchestra.run.smithi088.stderr:2024-06-03T08:36:32.113+0000 7ff49a002640 5 monclient: authenticate success, global_id 4317 2024-06-03T08:36:32.163 INFO:teuthology.orchestra.run.smithi088.stderr:2024-06-03T08:36:32.113+0000 7ff49a002640 10 monclient: _renew_subs 2024-06-03T08:36:32.163 INFO:teuthology.orchestra.run.smithi088.stderr:2024-06-03T08:36:32.113+0000 7ff49a002640 10 monclient: _send_mon_message to mon.a at v1:172.21.15.88:6789/0 2024-06-03T08:36:32.163 INFO:teuthology.orchestra.run.smithi088.stderr:2024-06-03T08:36:32.113+0000 7ff49a002640 1 -- 172.21.15.88:0/2378444824 --> v1:172.21.15.88:6789/0 -- mon_subscribe({mgrmap=0+}) v3 -- 0x7ff49412a740 con 0x7ff49412fd80 2024-06-03T08:36:32.163 INFO:teuthology.orchestra.run.smithi088.stderr:2024-06-03T08:36:32.113+0000 7ff49a002640 10 client.4317.objecter _maybe_request_map subscribing (onetime) to next osd map 2024-06-03T08:36:32.163 INFO:teuthology.orchestra.run.smithi088.stderr:2024-06-03T08:36:32.113+0000 7ff49a002640 10 monclient: _renew_subs 2024-06-03T08:36:32.163 INFO:teuthology.orchestra.run.smithi088.stderr:2024-06-03T08:36:32.113+0000 7ff49a002640 10 monclient: _send_mon_message to mon.a at v1:172.21.15.88:6789/0 2024-06-03T08:36:32.163 INFO:teuthology.orchestra.run.smithi088.stderr:2024-06-03T08:36:32.113+0000 7ff49a002640 1 -- 172.21.15.88:0/2378444824 --> v1:172.21.15.88:6789/0 -- mon_subscribe({osdmap=0}) v3 -- 0x7ff494129690 con 0x7ff49412fd80 2024-06-03T08:36:32.163 INFO:teuthology.orchestra.run.smithi088.stderr:2024-06-03T08:36:32.113+0000 7ff49a002640 1 librados: init done 2024-06-03T08:36:32.163 INFO:teuthology.orchestra.run.smithi088.stderr:2024-06-03T08:36:32.114+0000 7ff49a002640 10 client.4317.objecter _submit_command 1 [{"prefix": "get_command_descriptions"}] 2024-06-03T08:36:32.163 INFO:teuthology.orchestra.run.smithi088.stderr:2024-06-03T08:36:32.114+0000 7ff49a002640 15 client.4317.objecter _session_command_op_assign -1 1 2024-06-03T08:36:32.163 INFO:teuthology.orchestra.run.smithi088.stderr:2024-06-03T08:36:32.114+0000 7ff49a002640 20 client.4317.objecter _get_session osd=-1 returning homeless 2024-06-03T08:36:32.163 INFO:teuthology.orchestra.run.smithi088.stderr:2024-06-03T08:36:32.114+0000 7ff49a002640 10 client.4317.objecter _maybe_request_map subscribing (onetime) to next osd map 2024-06-03T08:36:32.164 INFO:teuthology.orchestra.run.smithi088.stderr:2024-06-03T08:36:32.114+0000 7ff49a002640 10 monclient: _send_mon_message to mon.a at v1:172.21.15.88:6789/0 2024-06-03T08:36:32.164 INFO:teuthology.orchestra.run.smithi088.stderr:2024-06-03T08:36:32.114+0000 7ff49a002640 1 -- 172.21.15.88:0/2378444824 --> v1:172.21.15.88:6789/0 -- mon_get_version(what=osdmap handle=1) v1 -- 0x7ff4940b3c20 con 0x7ff49412fd80 2024-06-03T08:36:32.164 INFO:teuthology.orchestra.run.smithi088.stderr:2024-06-03T08:36:32.122+0000 7f1983fff640 1 -- 172.21.15.88:0/1438849704 <== osd.2 v1:172.21.15.88:6802/35065 1 ==== command_reply(tid 1: 0 ) v1 ==== 8+0+26118 (unknown 0 0 1686565167) 0x7f197c077ef0 con 0x7f197c074280 2024-06-03T08:36:32.164 INFO:teuthology.orchestra.run.smithi088.stderr:2024-06-03T08:36:32.122+0000 7f1983fff640 10 client.4300.objecter ms_dispatch 0x7f1998000b90 command_reply(tid 1: 0 ) v1 2024-06-03T08:36:32.164 INFO:teuthology.orchestra.run.smithi088.stderr:2024-06-03T08:36:32.122+0000 7f1983fff640 10 client.4300.objecter _finish_command 1 = system:0 2024-06-03T08:36:32.164 INFO:teuthology.orchestra.run.smithi088.stderr:2024-06-03T08:36:32.122+0000 7f1983fff640 20 client.4300.objecter put_session s=0x7f197c073bd0 osd=2 4 2024-06-03T08:36:32.164 INFO:teuthology.orchestra.run.smithi088.stderr:2024-06-03T08:36:32.122+0000 7f1983fff640 15 client.4300.objecter _session_command_op_remove 2 1 2024-06-03T08:36:32.165 INFO:teuthology.orchestra.run.smithi088.stderr:2024-06-03T08:36:32.134+0000 7f4c1bfff640 1 -- 172.21.15.88:0/2099487910 <== mon.0 v1:172.21.15.88:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 764+0+0 (unknown 3879522733 0 0) 0x7f4c3013e160 con 0x7f4c3013ee50 2024-06-03T08:36:32.165 INFO:teuthology.orchestra.run.smithi088.stderr:2024-06-03T08:36:32.134+0000 7f4c1bfff640 1 -- 172.21.15.88:0/2099487910 --> v1:172.21.15.88:6789/0 -- auth(proto 2 165 bytes epoch 0) v1 -- 0x7f4c14004eb0 con 0x7f4c3013ee50 2024-06-03T08:36:32.165 INFO:teuthology.orchestra.run.smithi088.stderr:2024-06-03T08:36:32.134+0000 7f4c1bfff640 1 -- 172.21.15.88:0/2099487910 <== mon.0 v1:172.21.15.88:6789/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 2234374065 0 0) 0x7f4c28001ba0 con 0x7f4c3013ee50 2024-06-03T08:36:32.165 INFO:teuthology.orchestra.run.smithi088.stderr:2024-06-03T08:36:32.134+0000 7f4c1bfff640 10 monclient(hunting): handle_monmap mon_map magic: 0 v1 2024-06-03T08:36:32.165 INFO:teuthology.orchestra.run.smithi088.stderr:2024-06-03T08:36:32.134+0000 7f4c1bfff640 10 monclient(hunting): got monmap 1 from mon.a (according to old e1) 2024-06-03T08:36:32.165 INFO:teuthology.orchestra.run.smithi088.stderr:2024-06-03T08:36:32.134+0000 7f4c1bfff640 10 monclient(hunting): dump: 2024-06-03T08:36:32.165 INFO:teuthology.orchestra.run.smithi088.stderr:epoch 1 2024-06-03T08:36:32.165 INFO:teuthology.orchestra.run.smithi088.stderr:fsid ec3c3fdd-3ac6-4d7d-9412-683130ed8b04 2024-06-03T08:36:32.165 INFO:teuthology.orchestra.run.smithi088.stderr:last_changed 2024-06-03T08:34:31.885125+0000 2024-06-03T08:36:32.165 INFO:teuthology.orchestra.run.smithi088.stderr:created 2024-06-03T08:34:31.885125+0000 2024-06-03T08:36:32.165 INFO:teuthology.orchestra.run.smithi088.stderr:min_mon_release 18 (reef) 2024-06-03T08:36:32.165 INFO:teuthology.orchestra.run.smithi088.stderr:election_strategy: 3 2024-06-03T08:36:32.166 INFO:teuthology.orchestra.run.smithi088.stderr:0: v1:172.21.15.88:6789/0 mon.a 2024-06-03T08:36:32.166 INFO:teuthology.orchestra.run.smithi088.stderr:1: v1:172.21.15.192:6789/0 mon.b 2024-06-03T08:36:32.166 INFO:teuthology.orchestra.run.smithi088.stderr:2: v1:172.21.15.88:6790/0 mon.c 2024-06-03T08:36:32.166 INFO:teuthology.orchestra.run.smithi088.stderr: 2024-06-03T08:36:32.166 INFO:teuthology.orchestra.run.smithi088.stderr:2024-06-03T08:36:32.134+0000 7fd3367fc640 1 -- 172.21.15.88:0/3500826820 <== mon.0 v1:172.21.15.88:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 764+0+0 (unknown 587133584 0 0) 0x7fd324004eb0 con 0x7fd338006410 2024-06-03T08:36:32.166 INFO:teuthology.orchestra.run.smithi088.stderr:2024-06-03T08:36:32.134+0000 7fd3367fc640 1 -- 172.21.15.88:0/3500826820 --> v1:172.21.15.88:6789/0 -- auth(proto 2 165 bytes epoch 0) v1 -- 0x7fd338031470 con 0x7fd338006410 2024-06-03T08:36:32.166 INFO:teuthology.orchestra.run.smithi088.stderr:2024-06-03T08:36:32.134+0000 7fd3367fc640 1 -- 172.21.15.88:0/3500826820 <== mon.0 v1:172.21.15.88:6789/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 2234374065 0 0) 0x7fd330001870 con 0x7fd338006410 2024-06-03T08:36:32.166 INFO:teuthology.orchestra.run.smithi088.stderr:2024-06-03T08:36:32.134+0000 7fd3367fc640 10 monclient(hunting): handle_monmap mon_map magic: 0 v1 2024-06-03T08:36:32.167 INFO:teuthology.orchestra.run.smithi088.stderr:2024-06-03T08:36:32.134+0000 7fd3367fc640 10 monclient(hunting): got monmap 1 from mon.noname-a (according to old e1) 2024-06-03T08:36:32.167 INFO:teuthology.orchestra.run.smithi088.stderr:2024-06-03T08:36:32.134+0000 7fd3367fc640 10 monclient(hunting): dump: 2024-06-03T08:36:32.167 INFO:teuthology.orchestra.run.smithi088.stderr:epoch 1 2024-06-03T08:36:32.167 INFO:teuthology.orchestra.run.smithi088.stderr:fsid ec3c3fdd-3ac6-4d7d-9412-683130ed8b04 2024-06-03T08:36:32.167 INFO:teuthology.orchestra.run.smithi088.stderr:last_changed 2024-06-03T08:34:31.885125+0000 2024-06-03T08:36:32.167 INFO:teuthology.orchestra.run.smithi088.stderr:created 2024-06-03T08:34:31.885125+0000 2024-06-03T08:36:32.167 INFO:teuthology.orchestra.run.smithi088.stderr:min_mon_release 18 (reef) 2024-06-03T08:36:32.167 INFO:teuthology.orchestra.run.smithi088.stderr:election_strategy: 3 2024-06-03T08:36:32.167 INFO:teuthology.orchestra.run.smithi088.stderr:0: v1:172.21.15.88:6789/0 mon.a 2024-06-03T08:36:32.167 INFO:teuthology.orchestra.run.smithi088.stderr:1: v1:172.21.15.192:6789/0 mon.b 2024-06-03T08:36:32.167 INFO:teuthology.orchestra.run.smithi088.stderr:2: v1:172.21.15.88:6790/0 mon.c 2024-06-03T08:36:32.167 INFO:teuthology.orchestra.run.smithi088.stderr: 2024-06-03T08:36:32.167 INFO:teuthology.orchestra.run.smithi088.stderr:2024-06-03T08:36:32.134+0000 7fd3367fc640 10 monclient(hunting): _finish_auth 0 2024-06-03T08:36:32.168 INFO:teuthology.orchestra.run.smithi088.stderr:2024-06-03T08:36:32.136+0000 7f199da81640 10 client.4300.objecter _submit_command 2 [{"prefix": "flush_pg_stats"}] 2024-06-03T08:36:32.168 INFO:teuthology.orchestra.run.smithi088.stderr:2024-06-03T08:36:32.136+0000 7f199da81640 15 client.4300.objecter _session_command_op_assign -1 2 2024-06-03T08:36:32.168 INFO:teuthology.orchestra.run.smithi088.stderr:2024-06-03T08:36:32.136+0000 7f199da81640 20 client.4300.objecter _get_session s=0x7f197c073bd0 osd=2 3 2024-06-03T08:36:32.168 INFO:teuthology.orchestra.run.smithi088.stderr:2024-06-03T08:36:32.136+0000 7f199da81640 20 client.4300.objecter put_session s=0x7f197c073bd0 osd=2 3 2024-06-03T08:36:32.168 INFO:teuthology.orchestra.run.smithi088.stderr:2024-06-03T08:36:32.136+0000 7f199da81640 20 client.4300.objecter _get_session s=0x7f197c073bd0 osd=2 3 2024-06-03T08:36:32.168 INFO:teuthology.orchestra.run.smithi088.stderr:2024-06-03T08:36:32.136+0000 7f199da81640 15 client.4300.objecter _session_command_op_remove -1 2 2024-06-03T08:36:32.168 INFO:teuthology.orchestra.run.smithi088.stderr:2024-06-03T08:36:32.136+0000 7f199da81640 20 client.4300.objecter get_session s=0x7f197c073bd0 osd=2 3 2024-06-03T08:36:32.168 INFO:teuthology.orchestra.run.smithi088.stderr:2024-06-03T08:36:32.136+0000 7f199da81640 15 client.4300.objecter _session_command_op_assign 2 2 2024-06-03T08:36:32.168 INFO:teuthology.orchestra.run.smithi088.stderr:2024-06-03T08:36:32.136+0000 7f199da81640 20 client.4300.objecter put_session s=0x7f197c073bd0 osd=2 4 2024-06-03T08:36:32.168 INFO:teuthology.orchestra.run.smithi088.stderr:2024-06-03T08:36:32.136+0000 7f199da81640 10 client.4300.objecter _send_command 2 2024-06-03T08:36:32.168 INFO:teuthology.orchestra.run.smithi088.stderr:2024-06-03T08:36:32.136+0000 7f199da81640 1 -- 172.21.15.88:0/1438849704 --> v1:172.21.15.88:6802/35065 -- command(tid 2: {"prefix": "flush_pg_stats"}) v1 -- 0x7f1990004cd0 con 0x7f197c074280 2024-06-03T08:36:32.169 INFO:teuthology.orchestra.run.smithi088.stderr:2024-06-03T08:36:32.137+0000 7f7906ffd640 1 -- 172.21.15.88:0/996211931 <== mon.0 v1:172.21.15.88:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 764+0+0 (unknown 989591624 0 0) 0x7f78f8004eb0 con 0x7f79180b43c0 2024-06-03T08:36:32.169 INFO:teuthology.orchestra.run.smithi088.stderr:2024-06-03T08:36:32.137+0000 7f7906ffd640 1 -- 172.21.15.88:0/996211931 --> v1:172.21.15.88:6789/0 -- auth(proto 2 165 bytes epoch 0) v1 -- 0x7f79181442a0 con 0x7f79180b43c0 2024-06-03T08:36:32.169 INFO:teuthology.orchestra.run.smithi088.stderr:2024-06-03T08:36:32.137+0000 7f7906ffd640 1 -- 172.21.15.88:0/996211931 <== mon.0 v1:172.21.15.88:6789/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 2234374065 0 0) 0x7f79080018b0 con 0x7f79180b43c0 2024-06-03T08:36:32.169 INFO:teuthology.orchestra.run.smithi088.stderr:2024-06-03T08:36:32.137+0000 7f7906ffd640 10 monclient(hunting): handle_monmap mon_map magic: 0 v1 2024-06-03T08:36:32.169 INFO:teuthology.orchestra.run.smithi088.stderr:2024-06-03T08:36:32.137+0000 7f7906ffd640 10 monclient(hunting): got monmap 1 from mon.noname-a (according to old e1) 2024-06-03T08:36:32.169 INFO:teuthology.orchestra.run.smithi088.stderr:2024-06-03T08:36:32.137+0000 7f7906ffd640 10 monclient(hunting): dump: 2024-06-03T08:36:32.170 INFO:teuthology.orchestra.run.smithi088.stderr:epoch 1 2024-06-03T08:36:32.170 INFO:teuthology.orchestra.run.smithi088.stderr:fsid ec3c3fdd-3ac6-4d7d-9412-683130ed8b04 2024-06-03T08:36:32.170 INFO:teuthology.orchestra.run.smithi088.stderr:last_changed 2024-06-03T08:34:31.885125+0000 2024-06-03T08:36:32.170 INFO:teuthology.orchestra.run.smithi088.stderr:created 2024-06-03T08:34:31.885125+0000 2024-06-03T08:36:32.170 INFO:teuthology.orchestra.run.smithi088.stderr:min_mon_release 18 (reef) 2024-06-03T08:36:32.170 INFO:teuthology.orchestra.run.smithi088.stderr:election_strategy: 3 2024-06-03T08:36:32.170 INFO:teuthology.orchestra.run.smithi088.stderr:0: v1:172.21.15.88:6789/0 mon.a 2024-06-03T08:36:32.170 INFO:teuthology.orchestra.run.smithi088.stderr:1: v1:172.21.15.192:6789/0 mon.b 2024-06-03T08:36:32.170 INFO:teuthology.orchestra.run.smithi088.stderr:2: v1:172.21.15.88:6790/0 mon.c 2024-06-03T08:36:32.170 INFO:teuthology.orchestra.run.smithi088.stderr: 2024-06-03T08:36:32.170 INFO:teuthology.orchestra.run.smithi088.stderr:2024-06-03T08:36:32.137+0000 7f7906ffd640 10 monclient(hunting): _finish_auth 0 2024-06-03T08:36:32.171 INFO:teuthology.orchestra.run.smithi088.stderr:2024-06-03T08:36:32.146+0000 7ff47bfff640 1 -- 172.21.15.88:0/2378444824 <== mon.0 v1:172.21.15.88:6789/0 5 ==== config(0 keys) v1 ==== 4+0+0 (unknown 0 0 0) 0x7ff48c003990 con 0x7ff49412fd80 2024-06-03T08:36:32.171 INFO:teuthology.orchestra.run.smithi088.stderr:2024-06-03T08:36:32.146+0000 7ff47bfff640 10 monclient: handle_config config(0 keys) v1 2024-06-03T08:36:32.171 INFO:teuthology.orchestra.run.smithi088.stderr:2024-06-03T08:36:32.146+0000 7ff47bfff640 1 -- 172.21.15.88:0/2378444824 <== mon.0 v1:172.21.15.88:6789/0 6 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 2234374065 0 0) 0x7ff48c003d20 con 0x7ff49412fd80 2024-06-03T08:36:32.171 INFO:teuthology.orchestra.run.smithi088.stderr:2024-06-03T08:36:32.146+0000 7ff47bfff640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-06-03T08:36:32.172 INFO:teuthology.orchestra.run.smithi088.stderr:2024-06-03T08:36:32.146+0000 7ff47bfff640 10 monclient: got monmap 1 from mon.a (according to old e1) 2024-06-03T08:36:32.172 INFO:teuthology.orchestra.run.smithi088.stderr:2024-06-03T08:36:32.146+0000 7ff47bfff640 10 monclient: dump: 2024-06-03T08:36:32.172 INFO:teuthology.orchestra.run.smithi088.stderr:epoch 1 2024-06-03T08:36:32.172 INFO:teuthology.orchestra.run.smithi088.stderr:fsid ec3c3fdd-3ac6-4d7d-9412-683130ed8b04 2024-06-03T08:36:32.172 INFO:teuthology.orchestra.run.smithi088.stderr:last_changed 2024-06-03T08:34:31.885125+0000 2024-06-03T08:36:32.172 INFO:teuthology.orchestra.run.smithi088.stderr:created 2024-06-03T08:34:31.885125+0000 2024-06-03T08:36:32.172 INFO:teuthology.orchestra.run.smithi088.stderr:min_mon_release 18 (reef) 2024-06-03T08:36:32.172 INFO:teuthology.orchestra.run.smithi088.stderr:election_strategy: 3 2024-06-03T08:36:32.172 INFO:teuthology.orchestra.run.smithi088.stderr:0: v1:172.21.15.88:6789/0 mon.a 2024-06-03T08:36:32.172 INFO:teuthology.orchestra.run.smithi088.stderr:1: v1:172.21.15.192:6789/0 mon.b 2024-06-03T08:36:32.172 INFO:teuthology.orchestra.run.smithi088.stderr:2: v1:172.21.15.88:6790/0 mon.c 2024-06-03T08:36:32.172 INFO:teuthology.orchestra.run.smithi088.stderr: 2024-06-03T08:36:32.172 INFO:teuthology.orchestra.run.smithi088.stderr:2024-06-03T08:36:32.146+0000 7ff491ffb640 4 set_mon_vals no callback set 2024-06-03T08:36:32.173 INFO:teuthology.orchestra.run.smithi088.stderr:2024-06-03T08:36:32.147+0000 7ff47bfff640 1 -- 172.21.15.88:0/2378444824 <== mon.0 v1:172.21.15.88:6789/0 7 ==== mgrmap(e 5) v1 ==== 89110+0+0 (unknown 873533973 0 0) 0x7ff48c003e80 con 0x7ff49412fd80 2024-06-03T08:36:32.173 INFO:teuthology.orchestra.run.smithi088.stderr:2024-06-03T08:36:32.148+0000 7ff4937fe640 10 monclient: get_auth_request con 0x7ff48006d230 auth_method 0 2024-06-03T08:36:32.173 INFO:teuthology.orchestra.run.smithi088.stderr:2024-06-03T08:36:32.148+0000 7ff47bfff640 10 client.4317.objecter ms_handle_connect 0x7ff48006d230 2024-06-03T08:36:32.173 INFO:teuthology.orchestra.run.smithi088.stderr:2024-06-03T08:36:32.151+0000 7ff47bfff640 1 -- 172.21.15.88:0/2378444824 <== mon.0 v1:172.21.15.88:6789/0 8 ==== osd_map(26..26 src has 1..26) v4 ==== 4090+0+0 (unknown 3637450554 0 0) 0x7ff48c0543f0 con 0x7ff49412fd80 2024-06-03T08:36:32.173 INFO:teuthology.orchestra.run.smithi088.stderr:2024-06-03T08:36:32.151+0000 7ff47bfff640 10 client.4317.objecter ms_dispatch 0x7ff494000b90 osd_map(26..26 src has 1..26) v4 2024-06-03T08:36:32.173 INFO:teuthology.orchestra.run.smithi088.stderr:2024-06-03T08:36:32.151+0000 7ff47bfff640 3 client.4317.objecter handle_osd_map got epochs [26,26] > 0 2024-06-03T08:36:32.173 INFO:teuthology.orchestra.run.smithi088.stderr:2024-06-03T08:36:32.151+0000 7ff47bfff640 3 client.4317.objecter handle_osd_map decoding full epoch 26 2024-06-03T08:36:32.174 INFO:teuthology.orchestra.run.smithi088.stderr:2024-06-03T08:36:32.151+0000 7f1983fff640 1 -- 172.21.15.88:0/1438849704 <== osd.2 v1:172.21.15.88:6802/35065 2 ==== command_reply(tid 2: 0 ) v1 ==== 8+0+11 (unknown 0 0 2914482355) 0x7f1990004cd0 con 0x7f197c074280 2024-06-03T08:36:32.174 INFO:teuthology.orchestra.run.smithi088.stderr:2024-06-03T08:36:32.151+0000 7f1983fff640 10 client.4300.objecter ms_dispatch 0x7f1998000b90 command_reply(tid 2: 0 ) v1 2024-06-03T08:36:32.174 INFO:teuthology.orchestra.run.smithi088.stderr:2024-06-03T08:36:32.151+0000 7f1983fff640 10 client.4300.objecter _finish_command 2 = system:0 2024-06-03T08:36:32.174 INFO:teuthology.orchestra.run.smithi088.stderr:2024-06-03T08:36:32.151+0000 7f1983fff640 20 client.4300.objecter put_session s=0x7f197c073bd0 osd=2 4 2024-06-03T08:36:32.174 INFO:teuthology.orchestra.run.smithi088.stderr:2024-06-03T08:36:32.151+0000 7f1983fff640 15 client.4300.objecter _session_command_op_remove 2 2 2024-06-03T08:36:32.174 INFO:teuthology.orchestra.run.smithi088.stderr:2024-06-03T08:36:32.151+0000 7ff47bfff640 10 client.4317.objecter checking command 1 2024-06-03T08:36:32.175 INFO:teuthology.orchestra.run.smithi088.stderr:2024-06-03T08:36:32.151+0000 7ff47bfff640 20 client.4317.objecter _get_session s=0x7ff480073bb0 osd=3 3 2024-06-03T08:36:32.175 INFO:teuthology.orchestra.run.smithi088.stderr:2024-06-03T08:36:32.151+0000 7ff47bfff640 20 client.4317.objecter put_session s=0x7ff480073bb0 osd=3 3 2024-06-03T08:36:32.175 INFO:teuthology.orchestra.run.smithi088.stderr:2024-06-03T08:36:32.151+0000 7ff47bfff640 15 client.4317.objecter _session_command_op_remove -1 1 2024-06-03T08:36:32.175 INFO:teuthology.orchestra.run.smithi088.stderr:2024-06-03T08:36:32.151+0000 7ff47bfff640 20 client.4317.objecter _get_session s=0x7ff480073bb0 osd=3 3 2024-06-03T08:36:32.175 INFO:teuthology.orchestra.run.smithi088.stderr:2024-06-03T08:36:32.151+0000 7ff47bfff640 20 client.4317.objecter get_session s=0x7ff480073bb0 osd=3 3 2024-06-03T08:36:32.175 INFO:teuthology.orchestra.run.smithi088.stderr:2024-06-03T08:36:32.151+0000 7ff47bfff640 15 client.4317.objecter _session_command_op_assign 3 1 2024-06-03T08:36:32.175 INFO:teuthology.orchestra.run.smithi088.stderr:2024-06-03T08:36:32.151+0000 7ff47bfff640 20 client.4317.objecter put_session s=0x7ff480073bb0 osd=3 4 2024-06-03T08:36:32.175 INFO:teuthology.orchestra.run.smithi088.stderr:2024-06-03T08:36:32.151+0000 7ff47bfff640 10 client.4317.objecter _send_command 1 2024-06-03T08:36:32.175 INFO:teuthology.orchestra.run.smithi088.stderr:2024-06-03T08:36:32.151+0000 7ff47bfff640 1 -- 172.21.15.88:0/2378444824 --> v1:172.21.15.88:6812/35068 -- command(tid 1: {"prefix": "get_command_descriptions"}) v1 -- 0x7ff480077ed0 con 0x7ff480074260 2024-06-03T08:36:32.175 INFO:teuthology.orchestra.run.smithi088.stderr:2024-06-03T08:36:32.151+0000 7ff47bfff640 20 client.4317.objecter dump_active .. 0 homeless 2024-06-03T08:36:32.175 INFO:teuthology.orchestra.run.smithi088.stderr:2024-06-03T08:36:32.151+0000 7ff47bfff640 1 -- 172.21.15.88:0/2378444824 <== mon.0 v1:172.21.15.88:6789/0 9 ==== mon_get_version_reply(handle=1 version=26) v2 ==== 24+0+0 (unknown 2370098492 0 0) 0x7ff48c04ffa0 con 0x7ff49412fd80 2024-06-03T08:36:32.175 INFO:teuthology.orchestra.run.smithi088.stderr:2024-06-03T08:36:32.151+0000 7ff47bfff640 10 monclient: handle_get_version_reply finishing 1 version 26 2024-06-03T08:36:32.176 INFO:teuthology.orchestra.run.smithi088.stderr:2024-06-03T08:36:32.152+0000 7f199da81640 10 librados: watch_flush enter 2024-06-03T08:36:32.176 INFO:teuthology.orchestra.run.smithi088.stderr:2024-06-03T08:36:32.152+0000 7f199da81640 10 librados: watch_flush exit 2024-06-03T08:36:32.176 INFO:teuthology.orchestra.run.smithi088.stderr:2024-06-03T08:36:32.152+0000 7f199da81640 10 client.4300.objecter close_session for osd.2 2024-06-03T08:36:32.176 INFO:teuthology.orchestra.run.smithi088.stderr:2024-06-03T08:36:32.152+0000 7f199da81640 1 -- 172.21.15.88:0/1438849704 >> v1:172.21.15.88:6802/35065 conn(0x7f197c074280 legacy=0x7f197c0766a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-06-03T08:36:32.177 INFO:teuthology.orchestra.run.smithi088.stderr:2024-06-03T08:36:32.152+0000 7f199da81640 20 client.4300.objecter put_session s=0x7f197c073bd0 osd=2 1 2024-06-03T08:36:32.177 INFO:teuthology.orchestra.run.smithi088.stderr:2024-06-03T08:36:32.152+0000 7f199da81640 20 client.4300.objecter shutdown clearing up homeless session... 2024-06-03T08:36:32.177 INFO:teuthology.orchestra.run.smithi088.stderr:2024-06-03T08:36:32.152+0000 7f199da81640 10 client.4300.objecter successfully canceled tick 2024-06-03T08:36:32.177 INFO:teuthology.orchestra.run.smithi088.stderr:2024-06-03T08:36:32.152+0000 7f199da81640 1 -- 172.21.15.88:0/1438849704 >> v1:172.21.15.192:6800/34229 conn(0x7f197c06d250 legacy=0x7f197c06f710 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-06-03T08:36:32.177 INFO:teuthology.orchestra.run.smithi088.stderr:2024-06-03T08:36:32.152+0000 7f199da81640 10 monclient: shutdown 2024-06-03T08:36:32.177 INFO:teuthology.orchestra.run.smithi088.stderr:2024-06-03T08:36:32.152+0000 7f199da81640 20 monclient: shutdown discarding 0 pending message(s) 2024-06-03T08:36:32.177 INFO:teuthology.orchestra.run.smithi088.stderr:2024-06-03T08:36:32.152+0000 7f199da81640 1 -- 172.21.15.88:0/1438849704 >> v1:172.21.15.192:6789/0 conn(0x7f19900327d0 legacy=0x7f19900c0060 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-06-03T08:36:32.177 INFO:teuthology.orchestra.run.smithi088.stderr:2024-06-03T08:36:32.152+0000 7f1997fff640 1 -- 172.21.15.88:0/1438849704 reap_dead start 2024-06-03T08:36:32.177 INFO:teuthology.orchestra.run.smithi088.stderr:2024-06-03T08:36:32.152+0000 7f199da81640 1 -- 172.21.15.88:0/1438849704 shutdown_connections 2024-06-03T08:36:32.177 INFO:teuthology.orchestra.run.smithi088.stderr:2024-06-03T08:36:32.152+0000 7f199da81640 1 -- 172.21.15.88:0/1438849704 >> 172.21.15.88:0/1438849704 conn(0x7f19900058f0 msgr2=0x7f1990029610 unknown :-1 s=STATE_NONE l=0).mark_down 2024-06-03T08:36:32.177 INFO:teuthology.orchestra.run.smithi088.stderr:2024-06-03T08:36:32.152+0000 7ff492ffd640 10 monclient: get_auth_request con 0x7ff480074260 auth_method 0 2024-06-03T08:36:32.178 INFO:teuthology.orchestra.run.smithi088.stderr:2024-06-03T08:36:32.152+0000 7f199da81640 1 -- 172.21.15.88:0/1438849704 shutdown_connections 2024-06-03T08:36:32.178 INFO:teuthology.orchestra.run.smithi088.stderr:2024-06-03T08:36:32.152+0000 7f199da81640 1 -- 172.21.15.88:0/1438849704 wait complete. 2024-06-03T08:36:32.178 INFO:teuthology.orchestra.run.smithi088.stderr:2024-06-03T08:36:32.152+0000 7f199da81640 1 librados: shutdown 2024-06-03T08:36:32.178 INFO:teuthology.orchestra.run.smithi088.stderr:2024-06-03T08:36:32.154+0000 7ff47bfff640 10 client.4317.objecter ms_handle_connect 0x7ff480074260 2024-06-03T08:36:32.179 INFO:teuthology.orchestra.run.smithi088.stderr:2024-06-03T08:36:32.157+0000 7f9b9e7fc640 1 -- 172.21.15.88:0/496575353 <== mon.0 v1:172.21.15.88:6789/0 4 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 580+0+0 (unknown 3977881309 0 0) 0x7f9ba8145850 con 0x7f9ba80b43c0 2024-06-03T08:36:32.179 INFO:teuthology.orchestra.run.smithi088.stderr:2024-06-03T08:36:32.157+0000 7f9b9e7fc640 1 -- 172.21.15.88:0/496575353 >> v1:172.21.15.88:6790/0 conn(0x7f9ba814ae20 legacy=0x7f9ba81e1380 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-06-03T08:36:32.179 INFO:teuthology.orchestra.run.smithi088.stderr:2024-06-03T08:36:32.157+0000 7f9b9e7fc640 1 -- 172.21.15.88:0/496575353 >> v1:172.21.15.192:6789/0 conn(0x7f9ba813e6c0 legacy=0x7f9ba8143590 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-06-03T08:36:32.179 INFO:teuthology.orchestra.run.smithi088.stderr:2024-06-03T08:36:32.157+0000 7f9b9e7fc640 10 monclient: _finish_hunting 0 2024-06-03T08:36:32.179 INFO:teuthology.orchestra.run.smithi088.stderr:2024-06-03T08:36:32.157+0000 7f9b9e7fc640 1 monclient: found mon.a 2024-06-03T08:36:32.179 INFO:teuthology.orchestra.run.smithi088.stderr:2024-06-03T08:36:32.157+0000 7f9b9e7fc640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-06-03T08:36:32.179 INFO:teuthology.orchestra.run.smithi088.stderr:2024-06-03T08:36:32.157+0000 7f9b9e7fc640 10 monclient: _send_mon_message to mon.a at v1:172.21.15.88:6789/0 2024-06-03T08:36:32.179 INFO:teuthology.orchestra.run.smithi088.stderr:2024-06-03T08:36:32.157+0000 7f9b9e7fc640 1 -- 172.21.15.88:0/496575353 --> v1:172.21.15.88:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f9ba813b2d0 con 0x7f9ba80b43c0 2024-06-03T08:36:32.180 INFO:teuthology.orchestra.run.smithi088.stderr:2024-06-03T08:36:32.157+0000 7f9b9e7fc640 10 monclient: _finish_auth 0 2024-06-03T08:36:32.180 INFO:teuthology.orchestra.run.smithi088.stderr:2024-06-03T08:36:32.157+0000 7f9b9e7fc640 10 monclient: _check_auth_tickets 2024-06-03T08:36:32.180 INFO:teuthology.orchestra.run.smithi088.stderr:2024-06-03T08:36:32.157+0000 7f9b9e7fc640 20 monclient: _check_auth_rotating not needed by client.admin 2024-06-03T08:36:32.180 INFO:teuthology.orchestra.run.smithi088.stderr:2024-06-03T08:36:32.157+0000 7f9bafd56640 5 monclient: authenticate success, global_id 4326 2024-06-03T08:36:32.180 INFO:teuthology.orchestra.run.smithi088.stderr:2024-06-03T08:36:32.157+0000 7f9bafd56640 10 monclient: _renew_subs 2024-06-03T08:36:32.180 INFO:teuthology.orchestra.run.smithi088.stderr:2024-06-03T08:36:32.157+0000 7f9bafd56640 10 monclient: _send_mon_message to mon.a at v1:172.21.15.88:6789/0 2024-06-03T08:36:32.180 INFO:teuthology.orchestra.run.smithi088.stderr:2024-06-03T08:36:32.157+0000 7f9bafd56640 1 -- 172.21.15.88:0/496575353 --> v1:172.21.15.88:6789/0 -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f9ba81da9e0 con 0x7f9ba80b43c0 2024-06-03T08:36:32.180 INFO:teuthology.orchestra.run.smithi088.stderr:2024-06-03T08:36:32.157+0000 7f9bafd56640 10 client.4326.objecter _maybe_request_map subscribing (onetime) to next osd map 2024-06-03T08:36:32.180 INFO:teuthology.orchestra.run.smithi088.stderr:2024-06-03T08:36:32.157+0000 7f9bafd56640 10 monclient: _renew_subs 2024-06-03T08:36:32.180 INFO:teuthology.orchestra.run.smithi088.stderr:2024-06-03T08:36:32.157+0000 7f9bafd56640 10 monclient: _send_mon_message to mon.a at v1:172.21.15.88:6789/0 2024-06-03T08:36:32.180 INFO:teuthology.orchestra.run.smithi088.stderr:2024-06-03T08:36:32.157+0000 7f9bafd56640 1 -- 172.21.15.88:0/496575353 --> v1:172.21.15.88:6789/0 -- mon_subscribe({osdmap=0}) v3 -- 0x7f9ba8143e80 con 0x7f9ba80b43c0 2024-06-03T08:36:32.180 INFO:teuthology.orchestra.run.smithi088.stderr:2024-06-03T08:36:32.157+0000 7f9bafd56640 1 librados: init done 2024-06-03T08:36:32.180 INFO:teuthology.orchestra.run.smithi088.stderr:2024-06-03T08:36:32.158+0000 7f9bafd56640 10 client.4326.objecter _submit_command 1 [{"prefix": "get_command_descriptions"}] 2024-06-03T08:36:32.180 INFO:teuthology.orchestra.run.smithi088.stderr:2024-06-03T08:36:32.158+0000 7f9bafd56640 15 client.4326.objecter _session_command_op_assign -1 1 2024-06-03T08:36:32.180 INFO:teuthology.orchestra.run.smithi088.stderr:2024-06-03T08:36:32.158+0000 7f9bafd56640 20 client.4326.objecter _get_session osd=-1 returning homeless 2024-06-03T08:36:32.181 INFO:teuthology.orchestra.run.smithi088.stderr:2024-06-03T08:36:32.158+0000 7f9bafd56640 10 client.4326.objecter _maybe_request_map subscribing (onetime) to next osd map 2024-06-03T08:36:32.181 INFO:teuthology.orchestra.run.smithi088.stderr:2024-06-03T08:36:32.158+0000 7f9bafd56640 10 monclient: _send_mon_message to mon.a at v1:172.21.15.88:6789/0 2024-06-03T08:36:32.181 INFO:teuthology.orchestra.run.smithi088.stderr:2024-06-03T08:36:32.158+0000 7f9bafd56640 1 -- 172.21.15.88:0/496575353 --> v1:172.21.15.88:6789/0 -- mon_get_version(what=osdmap handle=1) v1 -- 0x7f9ba8144170 con 0x7f9ba80b43c0 2024-06-03T08:36:32.182 INFO:teuthology.orchestra.run.smithi088.stderr:2024-06-03T08:36:32.159+0000 7fd649ffb640 1 -- 172.21.15.88:0/2839181686 <== mon.0 v1:172.21.15.88:6789/0 4 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 580+0+0 (unknown 2890033596 0 0) 0x7fd654133f70 con 0x7fd65406dc10 2024-06-03T08:36:32.182 INFO:teuthology.orchestra.run.smithi088.stderr:2024-06-03T08:36:32.159+0000 7fd649ffb640 1 -- 172.21.15.88:0/2839181686 >> v1:172.21.15.88:6790/0 conn(0x7fd654141540 legacy=0x7fd65413d9c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-06-03T08:36:32.182 INFO:teuthology.orchestra.run.smithi088.stderr:2024-06-03T08:36:32.159+0000 7fd649ffb640 1 -- 172.21.15.88:0/2839181686 >> v1:172.21.15.192:6789/0 conn(0x7fd65413b700 legacy=0x7fd65413e0d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-06-03T08:36:32.182 INFO:teuthology.orchestra.run.smithi088.stderr:2024-06-03T08:36:32.159+0000 7fd649ffb640 10 monclient: _finish_hunting 0 2024-06-03T08:36:32.182 INFO:teuthology.orchestra.run.smithi088.stderr:2024-06-03T08:36:32.159+0000 7fd649ffb640 1 monclient: found mon.a 2024-06-03T08:36:32.182 INFO:teuthology.orchestra.run.smithi088.stderr:2024-06-03T08:36:32.159+0000 7fd649ffb640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-06-03T08:36:32.182 INFO:teuthology.orchestra.run.smithi088.stderr:2024-06-03T08:36:32.159+0000 7fd649ffb640 10 monclient: _send_mon_message to mon.a at v1:172.21.15.88:6789/0 2024-06-03T08:36:32.182 INFO:teuthology.orchestra.run.smithi088.stderr:2024-06-03T08:36:32.159+0000 7fd649ffb640 1 -- 172.21.15.88:0/2839181686 --> v1:172.21.15.88:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fd654134100 con 0x7fd65406dc10 2024-06-03T08:36:32.182 INFO:teuthology.orchestra.run.smithi088.stderr:2024-06-03T08:36:32.159+0000 7fd649ffb640 10 monclient: _finish_auth 0 2024-06-03T08:36:32.182 INFO:teuthology.orchestra.run.smithi088.stderr:2024-06-03T08:36:32.159+0000 7fd649ffb640 10 monclient: _check_auth_tickets 2024-06-03T08:36:32.183 INFO:teuthology.orchestra.run.smithi088.stderr:2024-06-03T08:36:32.159+0000 7fd649ffb640 20 monclient: _check_auth_rotating not needed by client.admin 2024-06-03T08:36:32.183 INFO:teuthology.orchestra.run.smithi088.stderr:2024-06-03T08:36:32.159+0000 7fd65b934640 5 monclient: authenticate success, global_id 4320 2024-06-03T08:36:32.183 INFO:teuthology.orchestra.run.smithi088.stderr:2024-06-03T08:36:32.159+0000 7fd65b934640 10 monclient: _renew_subs 2024-06-03T08:36:32.183 INFO:teuthology.orchestra.run.smithi088.stderr:2024-06-03T08:36:32.159+0000 7fd65b934640 10 monclient: _send_mon_message to mon.a at v1:172.21.15.88:6789/0 2024-06-03T08:36:32.183 INFO:teuthology.orchestra.run.smithi088.stderr:2024-06-03T08:36:32.159+0000 7fd65b934640 1 -- 172.21.15.88:0/2839181686 --> v1:172.21.15.88:6789/0 -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fd654130e00 con 0x7fd65406dc10 2024-06-03T08:36:32.183 INFO:teuthology.orchestra.run.smithi088.stderr:2024-06-03T08:36:32.159+0000 7fd65b934640 10 client.4320.objecter _maybe_request_map subscribing (onetime) to next osd map 2024-06-03T08:36:32.183 INFO:teuthology.orchestra.run.smithi088.stderr:2024-06-03T08:36:32.159+0000 7fd65b934640 10 monclient: _renew_subs 2024-06-03T08:36:32.183 INFO:teuthology.orchestra.run.smithi088.stderr:2024-06-03T08:36:32.159+0000 7fd65b934640 10 monclient: _send_mon_message to mon.a at v1:172.21.15.88:6789/0 2024-06-03T08:36:32.183 INFO:teuthology.orchestra.run.smithi088.stderr:2024-06-03T08:36:32.159+0000 7fd65b934640 1 -- 172.21.15.88:0/2839181686 --> v1:172.21.15.88:6789/0 -- mon_subscribe({osdmap=0}) v3 -- 0x7fd65412de80 con 0x7fd65406dc10 2024-06-03T08:36:32.183 INFO:teuthology.orchestra.run.smithi088.stderr:2024-06-03T08:36:32.159+0000 7fd65b934640 1 librados: init done 2024-06-03T08:36:32.184 INFO:teuthology.orchestra.run.smithi088.stderr:2024-06-03T08:36:32.160+0000 7fd3367fc640 1 -- 172.21.15.88:0/3500826820 <== mon.0 v1:172.21.15.88:6789/0 4 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 580+0+0 (unknown 235795555 0 0) 0x7fd338031470 con 0x7fd338006410 2024-06-03T08:36:32.184 INFO:teuthology.orchestra.run.smithi088.stderr:2024-06-03T08:36:32.160+0000 7fd3367fc640 1 -- 172.21.15.88:0/3500826820 >> v1:172.21.15.88:6790/0 conn(0x7fd33802a640 legacy=0x7fd338031990 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-06-03T08:36:32.184 INFO:teuthology.orchestra.run.smithi088.stderr:2024-06-03T08:36:32.160+0000 7fd3367fc640 1 -- 172.21.15.88:0/3500826820 >> v1:172.21.15.192:6789/0 conn(0x7fd3380327d0 legacy=0x7fd3380c3ba0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-06-03T08:36:32.184 INFO:teuthology.orchestra.run.smithi088.stderr:2024-06-03T08:36:32.160+0000 7fd3367fc640 10 monclient: _finish_hunting 0 2024-06-03T08:36:32.184 INFO:teuthology.orchestra.run.smithi088.stderr:2024-06-03T08:36:32.160+0000 7fd3367fc640 1 monclient: found mon.a 2024-06-03T08:36:32.184 INFO:teuthology.orchestra.run.smithi088.stderr:2024-06-03T08:36:32.160+0000 7fd3367fc640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-06-03T08:36:32.184 INFO:teuthology.orchestra.run.smithi088.stderr:2024-06-03T08:36:32.160+0000 7fd3367fc640 10 monclient: _send_mon_message to mon.a at v1:172.21.15.88:6789/0 2024-06-03T08:36:32.184 INFO:teuthology.orchestra.run.smithi088.stderr:2024-06-03T08:36:32.160+0000 7fd3367fc640 1 -- 172.21.15.88:0/3500826820 --> v1:172.21.15.88:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fd3380b27e0 con 0x7fd338006410 2024-06-03T08:36:32.184 INFO:teuthology.orchestra.run.smithi088.stderr:2024-06-03T08:36:32.160+0000 7fd3367fc640 10 monclient: _finish_auth 0 2024-06-03T08:36:32.184 INFO:teuthology.orchestra.run.smithi088.stderr:2024-06-03T08:36:32.160+0000 7fd3367fc640 10 monclient: _check_auth_tickets 2024-06-03T08:36:32.184 INFO:teuthology.orchestra.run.smithi088.stderr:2024-06-03T08:36:32.160+0000 7fd3367fc640 20 monclient: _check_auth_rotating not needed by client.admin 2024-06-03T08:36:32.184 INFO:teuthology.orchestra.run.smithi088.stderr:2024-06-03T08:36:32.160+0000 7fd347fb9640 5 monclient: authenticate success, global_id 4332 2024-06-03T08:36:32.184 INFO:teuthology.orchestra.run.smithi088.stderr:2024-06-03T08:36:32.160+0000 7fd347fb9640 10 monclient: _renew_subs 2024-06-03T08:36:32.184 INFO:teuthology.orchestra.run.smithi088.stderr:2024-06-03T08:36:32.160+0000 7fd347fb9640 10 monclient: _send_mon_message to mon.a at v1:172.21.15.88:6789/0 2024-06-03T08:36:32.185 INFO:teuthology.orchestra.run.smithi088.stderr:2024-06-03T08:36:32.160+0000 7fd347fb9640 1 -- 172.21.15.88:0/3500826820 --> v1:172.21.15.88:6789/0 -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fd3380bcff0 con 0x7fd338006410 2024-06-03T08:36:32.185 INFO:teuthology.orchestra.run.smithi088.stderr:2024-06-03T08:36:32.160+0000 7fd65b934640 10 client.4320.objecter _submit_command 1 [{"prefix": "get_command_descriptions"}] 2024-06-03T08:36:32.185 INFO:teuthology.orchestra.run.smithi088.stderr:2024-06-03T08:36:32.160+0000 7fd65b934640 15 client.4320.objecter _session_command_op_assign -1 1 2024-06-03T08:36:32.185 INFO:teuthology.orchestra.run.smithi088.stderr:2024-06-03T08:36:32.160+0000 7fd65b934640 20 client.4320.objecter _get_session osd=-1 returning homeless 2024-06-03T08:36:32.185 INFO:teuthology.orchestra.run.smithi088.stderr:2024-06-03T08:36:32.160+0000 7fd65b934640 10 client.4320.objecter _maybe_request_map subscribing (onetime) to next osd map 2024-06-03T08:36:32.185 INFO:teuthology.orchestra.run.smithi088.stderr:2024-06-03T08:36:32.160+0000 7fd65b934640 10 monclient: _send_mon_message to mon.a at v1:172.21.15.88:6789/0 2024-06-03T08:36:32.185 INFO:teuthology.orchestra.run.smithi088.stderr:2024-06-03T08:36:32.160+0000 7fd65b934640 1 -- 172.21.15.88:0/2839181686 --> v1:172.21.15.88:6789/0 -- mon_get_version(what=osdmap handle=1) v1 -- 0x7fd6540b3c20 con 0x7fd65406dc10 2024-06-03T08:36:32.186 INFO:teuthology.orchestra.run.smithi088.stderr:2024-06-03T08:36:32.160+0000 7fd347fb9640 10 client.4332.objecter _maybe_request_map subscribing (onetime) to next osd map 2024-06-03T08:36:32.186 INFO:teuthology.orchestra.run.smithi088.stderr:2024-06-03T08:36:32.160+0000 7fd347fb9640 10 monclient: _renew_subs 2024-06-03T08:36:32.186 INFO:teuthology.orchestra.run.smithi088.stderr:2024-06-03T08:36:32.160+0000 7fd347fb9640 10 monclient: _send_mon_message to mon.a at v1:172.21.15.88:6789/0 2024-06-03T08:36:32.186 INFO:teuthology.orchestra.run.smithi088.stderr:2024-06-03T08:36:32.160+0000 7fd347fb9640 1 -- 172.21.15.88:0/3500826820 --> v1:172.21.15.88:6789/0 -- mon_subscribe({osdmap=0}) v3 -- 0x7fd338168720 con 0x7fd338006410 2024-06-03T08:36:32.186 INFO:teuthology.orchestra.run.smithi088.stderr:2024-06-03T08:36:32.160+0000 7fd347fb9640 1 librados: init done 2024-06-03T08:36:32.186 INFO:teuthology.orchestra.run.smithi088.stderr:2024-06-03T08:36:32.161+0000 7fd347fb9640 10 client.4332.objecter _submit_command 1 [{"prefix": "get_command_descriptions"}] 2024-06-03T08:36:32.186 INFO:teuthology.orchestra.run.smithi088.stderr:2024-06-03T08:36:32.161+0000 7fd347fb9640 15 client.4332.objecter _session_command_op_assign -1 1 2024-06-03T08:36:32.186 INFO:teuthology.orchestra.run.smithi088.stderr:2024-06-03T08:36:32.161+0000 7fd347fb9640 20 client.4332.objecter _get_session osd=-1 returning homeless 2024-06-03T08:36:32.186 INFO:teuthology.orchestra.run.smithi088.stderr:2024-06-03T08:36:32.161+0000 7fd347fb9640 10 client.4332.objecter _maybe_request_map subscribing (onetime) to next osd map 2024-06-03T08:36:32.186 INFO:teuthology.orchestra.run.smithi088.stderr:2024-06-03T08:36:32.161+0000 7fd347fb9640 10 monclient: _send_mon_message to mon.a at v1:172.21.15.88:6789/0 2024-06-03T08:36:32.186 INFO:teuthology.orchestra.run.smithi088.stderr:2024-06-03T08:36:32.161+0000 7fd347fb9640 1 -- 172.21.15.88:0/3500826820 --> v1:172.21.15.88:6789/0 -- mon_get_version(what=osdmap handle=1) v1 -- 0x7fd3380312f0 con 0x7fd338006410 2024-06-03T08:36:32.187 INFO:teuthology.orchestra.run.smithi088.stdout:85899345929 2024-06-03T08:36:32.188 DEBUG:teuthology.orchestra.run.smithi088:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd last-stat-seq osd.2 2024-06-03T08:36:32.188 INFO:teuthology.orchestra.run.smithi088.stderr:2024-06-03T08:36:32.163+0000 7fa0d37fe640 1 -- 172.21.15.88:0/1282623662 <== mon.0 v1:172.21.15.88:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 2255811554 0 0) 0x7fa0ec1d6630 con 0x7fa0ec0a8960 2024-06-03T08:36:32.188 INFO:teuthology.orchestra.run.smithi088.stderr:2024-06-03T08:36:32.163+0000 7fa0d37fe640 10 monclient(hunting): _negotiate 2024-06-03T08:36:32.188 INFO:teuthology.orchestra.run.smithi088.stderr:2024-06-03T08:36:32.163+0000 7fa0d37fe640 10 monclient(hunting): _init_auth method 2 2024-06-03T08:36:32.188 INFO:teuthology.orchestra.run.smithi088.stderr:2024-06-03T08:36:32.163+0000 7fa0d37fe640 10 monclient(hunting): _init_auth creating new auth 2024-06-03T08:36:32.188 INFO:teuthology.orchestra.run.smithi088.stderr:2024-06-03T08:36:32.163+0000 7fa0d37fe640 10 monclient(hunting): my global_id is 4335 2024-06-03T08:36:32.189 INFO:teuthology.orchestra.run.smithi088.stderr:2024-06-03T08:36:32.163+0000 7fa0d37fe640 1 -- 172.21.15.88:0/1282623662 --> v1:172.21.15.88:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7fa0d8004eb0 con 0x7fa0ec0a8960 2024-06-03T08:36:32.190 INFO:teuthology.orchestra.run.smithi088.stderr:2024-06-03T08:36:32.168+0000 7f4c1bfff640 1 -- 172.21.15.88:0/2099487910 <== mon.0 v1:172.21.15.88:6789/0 4 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 580+0+0 (unknown 575564751 0 0) 0x7f4c14004eb0 con 0x7f4c3013ee50 2024-06-03T08:36:32.190 INFO:teuthology.orchestra.run.smithi088.stderr:2024-06-03T08:36:32.168+0000 7f4c1bfff640 1 -- 172.21.15.88:0/2099487910 >> v1:172.21.15.88:6790/0 conn(0x7f4c300b4060 legacy=0x7f4c3013c570 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-06-03T08:36:32.190 INFO:teuthology.orchestra.run.smithi088.stderr:2024-06-03T08:36:32.168+0000 7f4c1bfff640 1 -- 172.21.15.88:0/2099487910 >> v1:172.21.15.192:6789/0 conn(0x7f4c300a86f0 legacy=0x7f4c3013ff20 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-06-03T08:36:32.190 INFO:teuthology.orchestra.run.smithi088.stderr:2024-06-03T08:36:32.168+0000 7f4c1bfff640 10 monclient: _finish_hunting 0 2024-06-03T08:36:32.190 INFO:teuthology.orchestra.run.smithi088.stderr:2024-06-03T08:36:32.168+0000 7f4c1bfff640 1 monclient: found mon.a 2024-06-03T08:36:32.190 INFO:teuthology.orchestra.run.smithi088.stderr:2024-06-03T08:36:32.168+0000 7f4c1bfff640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-06-03T08:36:32.190 INFO:teuthology.orchestra.run.smithi088.stderr:2024-06-03T08:36:32.168+0000 7f4c1bfff640 10 monclient: _send_mon_message to mon.a at v1:172.21.15.88:6789/0 2024-06-03T08:36:32.190 INFO:teuthology.orchestra.run.smithi088.stderr:2024-06-03T08:36:32.168+0000 7f4c1bfff640 1 -- 172.21.15.88:0/2099487910 --> v1:172.21.15.88:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f4c30140ce0 con 0x7f4c3013ee50 2024-06-03T08:36:32.191 INFO:teuthology.orchestra.run.smithi088.stderr:2024-06-03T08:36:32.168+0000 7f4c1bfff640 10 monclient: _finish_auth 0 2024-06-03T08:36:32.191 INFO:teuthology.orchestra.run.smithi088.stderr:2024-06-03T08:36:32.168+0000 7f4c1bfff640 10 monclient: _check_auth_tickets 2024-06-03T08:36:32.191 INFO:teuthology.orchestra.run.smithi088.stderr:2024-06-03T08:36:32.168+0000 7f4c1bfff640 20 monclient: _check_auth_rotating not needed by client.admin 2024-06-03T08:36:32.191 INFO:teuthology.orchestra.run.smithi088.stderr:2024-06-03T08:36:32.168+0000 7f4c36155640 5 monclient: authenticate success, global_id 4323 2024-06-03T08:36:32.191 INFO:teuthology.orchestra.run.smithi088.stderr:2024-06-03T08:36:32.168+0000 7f4c36155640 10 monclient: _renew_subs 2024-06-03T08:36:32.191 INFO:teuthology.orchestra.run.smithi088.stderr:2024-06-03T08:36:32.168+0000 7f4c36155640 10 monclient: _send_mon_message to mon.a at v1:172.21.15.88:6789/0 2024-06-03T08:36:32.191 INFO:teuthology.orchestra.run.smithi088.stderr:2024-06-03T08:36:32.168+0000 7f4c36155640 1 -- 172.21.15.88:0/2099487910 --> v1:172.21.15.88:6789/0 -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f4c3013f2b0 con 0x7f4c3013ee50 2024-06-03T08:36:32.191 INFO:teuthology.orchestra.run.smithi088.stderr:2024-06-03T08:36:32.168+0000 7f4c36155640 10 client.4323.objecter _maybe_request_map subscribing (onetime) to next osd map 2024-06-03T08:36:32.191 INFO:teuthology.orchestra.run.smithi088.stderr:2024-06-03T08:36:32.168+0000 7f4c36155640 10 monclient: _renew_subs 2024-06-03T08:36:32.191 INFO:teuthology.orchestra.run.smithi088.stderr:2024-06-03T08:36:32.168+0000 7f4c36155640 10 monclient: _send_mon_message to mon.a at v1:172.21.15.88:6789/0 2024-06-03T08:36:32.191 INFO:teuthology.orchestra.run.smithi088.stderr:2024-06-03T08:36:32.168+0000 7f4c36155640 1 -- 172.21.15.88:0/2099487910 --> v1:172.21.15.88:6789/0 -- mon_subscribe({osdmap=0}) v3 -- 0x7f4c30140a30 con 0x7f4c3013ee50 2024-06-03T08:36:32.191 INFO:teuthology.orchestra.run.smithi088.stderr:2024-06-03T08:36:32.168+0000 7f4c36155640 1 librados: init done 2024-06-03T08:36:32.192 INFO:teuthology.orchestra.run.smithi088.stderr:2024-06-03T08:36:32.169+0000 7f4c36155640 10 client.4323.objecter _submit_command 1 [{"prefix": "get_command_descriptions"}] 2024-06-03T08:36:32.192 INFO:teuthology.orchestra.run.smithi088.stderr:2024-06-03T08:36:32.169+0000 7f4c36155640 15 client.4323.objecter _session_command_op_assign -1 1 2024-06-03T08:36:32.192 INFO:teuthology.orchestra.run.smithi088.stderr:2024-06-03T08:36:32.169+0000 7f4c36155640 20 client.4323.objecter _get_session osd=-1 returning homeless 2024-06-03T08:36:32.192 INFO:teuthology.orchestra.run.smithi088.stderr:2024-06-03T08:36:32.169+0000 7f4c36155640 10 client.4323.objecter _maybe_request_map subscribing (onetime) to next osd map 2024-06-03T08:36:32.192 INFO:teuthology.orchestra.run.smithi088.stderr:2024-06-03T08:36:32.169+0000 7f4c36155640 10 monclient: _send_mon_message to mon.a at v1:172.21.15.88:6789/0 2024-06-03T08:36:32.192 INFO:teuthology.orchestra.run.smithi088.stderr:2024-06-03T08:36:32.169+0000 7f4c36155640 1 -- 172.21.15.88:0/2099487910 --> v1:172.21.15.88:6789/0 -- mon_get_version(what=osdmap handle=1) v1 -- 0x7f4c300b4440 con 0x7f4c3013ee50 2024-06-03T08:36:32.200 INFO:teuthology.orchestra.run.smithi088.stderr:2024-06-03T08:36:32.199+0000 7ff47bfff640 1 -- 172.21.15.88:0/2378444824 <== osd.3 v1:172.21.15.88:6812/35068 1 ==== command_reply(tid 1: 0 ) v1 ==== 8+0+26118 (unknown 0 0 1686565167) 0x7ff480077ed0 con 0x7ff480074260 2024-06-03T08:36:32.200 INFO:teuthology.orchestra.run.smithi088.stderr:2024-06-03T08:36:32.199+0000 7ff47bfff640 10 client.4317.objecter ms_dispatch 0x7ff494000b90 command_reply(tid 1: 0 ) v1 2024-06-03T08:36:32.200 INFO:teuthology.orchestra.run.smithi088.stderr:2024-06-03T08:36:32.199+0000 7ff47bfff640 10 client.4317.objecter _finish_command 1 = system:0 2024-06-03T08:36:32.200 INFO:teuthology.orchestra.run.smithi088.stderr:2024-06-03T08:36:32.199+0000 7ff47bfff640 20 client.4317.objecter put_session s=0x7ff480073bb0 osd=3 4 2024-06-03T08:36:32.200 INFO:teuthology.orchestra.run.smithi088.stderr:2024-06-03T08:36:32.199+0000 7ff47bfff640 15 client.4317.objecter _session_command_op_remove 3 1 2024-06-03T08:36:32.228 INFO:teuthology.orchestra.run.smithi088.stderr:2024-06-03T08:36:32.228+0000 7ff49a002640 10 client.4317.objecter _submit_command 2 [{"prefix": "flush_pg_stats"}] 2024-06-03T08:36:32.228 INFO:teuthology.orchestra.run.smithi088.stderr:2024-06-03T08:36:32.228+0000 7ff49a002640 15 client.4317.objecter _session_command_op_assign -1 2 2024-06-03T08:36:32.228 INFO:teuthology.orchestra.run.smithi088.stderr:2024-06-03T08:36:32.228+0000 7ff49a002640 20 client.4317.objecter _get_session s=0x7ff480073bb0 osd=3 3 2024-06-03T08:36:32.229 INFO:teuthology.orchestra.run.smithi088.stderr:2024-06-03T08:36:32.228+0000 7ff49a002640 20 client.4317.objecter put_session s=0x7ff480073bb0 osd=3 3 2024-06-03T08:36:32.229 INFO:teuthology.orchestra.run.smithi088.stderr:2024-06-03T08:36:32.228+0000 7ff49a002640 20 client.4317.objecter _get_session s=0x7ff480073bb0 osd=3 3 2024-06-03T08:36:32.229 INFO:teuthology.orchestra.run.smithi088.stderr:2024-06-03T08:36:32.228+0000 7ff49a002640 15 client.4317.objecter _session_command_op_remove -1 2 2024-06-03T08:36:32.229 INFO:teuthology.orchestra.run.smithi088.stderr:2024-06-03T08:36:32.228+0000 7ff49a002640 20 client.4317.objecter get_session s=0x7ff480073bb0 osd=3 3 2024-06-03T08:36:32.229 INFO:teuthology.orchestra.run.smithi088.stderr:2024-06-03T08:36:32.228+0000 7ff49a002640 15 client.4317.objecter _session_command_op_assign 3 2 2024-06-03T08:36:32.229 INFO:teuthology.orchestra.run.smithi088.stderr:2024-06-03T08:36:32.228+0000 7ff49a002640 20 client.4317.objecter put_session s=0x7ff480073bb0 osd=3 4 2024-06-03T08:36:32.229 INFO:teuthology.orchestra.run.smithi088.stderr:2024-06-03T08:36:32.228+0000 7ff49a002640 10 client.4317.objecter _send_command 2 2024-06-03T08:36:32.229 INFO:teuthology.orchestra.run.smithi088.stderr:2024-06-03T08:36:32.228+0000 7ff49a002640 1 -- 172.21.15.88:0/2378444824 --> v1:172.21.15.88:6812/35068 -- command(tid 2: {"prefix": "flush_pg_stats"}) v1 -- 0x7ff49412b980 con 0x7ff480074260 2024-06-03T08:36:32.235 INFO:teuthology.orchestra.run.smithi088.stderr:2024-06-03T08:36:32.234+0000 7ff47bfff640 1 -- 172.21.15.88:0/2378444824 <== osd.3 v1:172.21.15.88:6812/35068 2 ==== command_reply(tid 2: 0 ) v1 ==== 8+0+11 (unknown 0 0 3931925601) 0x7ff49412b980 con 0x7ff480074260 2024-06-03T08:36:32.235 INFO:teuthology.orchestra.run.smithi088.stderr:2024-06-03T08:36:32.234+0000 7ff47bfff640 10 client.4317.objecter ms_dispatch 0x7ff494000b90 command_reply(tid 2: 0 ) v1 2024-06-03T08:36:32.235 INFO:teuthology.orchestra.run.smithi088.stderr:2024-06-03T08:36:32.234+0000 7ff47bfff640 10 client.4317.objecter _finish_command 2 = system:0 2024-06-03T08:36:32.235 INFO:teuthology.orchestra.run.smithi088.stderr:2024-06-03T08:36:32.234+0000 7ff47bfff640 20 client.4317.objecter put_session s=0x7ff480073bb0 osd=3 4 2024-06-03T08:36:32.235 INFO:teuthology.orchestra.run.smithi088.stderr:2024-06-03T08:36:32.234+0000 7ff47bfff640 15 client.4317.objecter _session_command_op_remove 3 2 2024-06-03T08:36:32.236 INFO:teuthology.orchestra.run.smithi088.stderr:2024-06-03T08:36:32.235+0000 7ff49a002640 10 librados: watch_flush enter 2024-06-03T08:36:32.237 INFO:teuthology.orchestra.run.smithi088.stderr:2024-06-03T08:36:32.235+0000 7ff49a002640 10 librados: watch_flush exit 2024-06-03T08:36:32.237 INFO:teuthology.orchestra.run.smithi088.stderr:2024-06-03T08:36:32.235+0000 7ff49a002640 10 client.4317.objecter close_session for osd.3 2024-06-03T08:36:32.237 INFO:teuthology.orchestra.run.smithi088.stderr:2024-06-03T08:36:32.235+0000 7ff49a002640 1 -- 172.21.15.88:0/2378444824 >> v1:172.21.15.88:6812/35068 conn(0x7ff480074260 legacy=0x7ff480076680 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-06-03T08:36:32.237 INFO:teuthology.orchestra.run.smithi088.stderr:2024-06-03T08:36:32.235+0000 7ff49a002640 20 client.4317.objecter put_session s=0x7ff480073bb0 osd=3 1 2024-06-03T08:36:32.237 INFO:teuthology.orchestra.run.smithi088.stderr:2024-06-03T08:36:32.235+0000 7ff49a002640 20 client.4317.objecter shutdown clearing up homeless session... 2024-06-03T08:36:32.237 INFO:teuthology.orchestra.run.smithi088.stderr:2024-06-03T08:36:32.235+0000 7ff49a002640 10 client.4317.objecter successfully canceled tick 2024-06-03T08:36:32.237 INFO:teuthology.orchestra.run.smithi088.stderr:2024-06-03T08:36:32.235+0000 7ff49a002640 1 -- 172.21.15.88:0/2378444824 >> v1:172.21.15.192:6800/34229 conn(0x7ff48006d230 legacy=0x7ff48006f6f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-06-03T08:36:32.237 INFO:teuthology.orchestra.run.smithi088.stderr:2024-06-03T08:36:32.235+0000 7ff49a002640 10 monclient: shutdown 2024-06-03T08:36:32.237 INFO:teuthology.orchestra.run.smithi088.stderr:2024-06-03T08:36:32.235+0000 7ff49a002640 20 monclient: shutdown discarding 0 pending message(s) 2024-06-03T08:36:32.237 INFO:teuthology.orchestra.run.smithi088.stderr:2024-06-03T08:36:32.235+0000 7ff49a002640 1 -- 172.21.15.88:0/2378444824 >> v1:172.21.15.88:6789/0 conn(0x7ff49412fd80 legacy=0x7ff4941dd0f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-06-03T08:36:32.237 INFO:teuthology.orchestra.run.smithi088.stderr:2024-06-03T08:36:32.235+0000 7ff493fff640 1 -- 172.21.15.88:0/2378444824 reap_dead start 2024-06-03T08:36:32.238 INFO:teuthology.orchestra.run.smithi088.stderr:2024-06-03T08:36:32.235+0000 7ff49a002640 1 -- 172.21.15.88:0/2378444824 shutdown_connections 2024-06-03T08:36:32.238 INFO:teuthology.orchestra.run.smithi088.stderr:2024-06-03T08:36:32.235+0000 7ff49a002640 1 -- 172.21.15.88:0/2378444824 >> 172.21.15.88:0/2378444824 conn(0x7ff4940b2f20 msgr2=0x7ff4940aee30 unknown :-1 s=STATE_NONE l=0).mark_down 2024-06-03T08:36:32.238 INFO:teuthology.orchestra.run.smithi088.stderr:2024-06-03T08:36:32.235+0000 7ff49a002640 1 -- 172.21.15.88:0/2378444824 shutdown_connections 2024-06-03T08:36:32.238 INFO:teuthology.orchestra.run.smithi088.stderr:2024-06-03T08:36:32.235+0000 7ff49a002640 1 -- 172.21.15.88:0/2378444824 wait complete. 2024-06-03T08:36:32.238 INFO:teuthology.orchestra.run.smithi088.stderr:2024-06-03T08:36:32.235+0000 7ff49a002640 1 librados: shutdown 2024-06-03T08:36:32.250 INFO:teuthology.orchestra.run.smithi088.stdout:60129542152 2024-06-03T08:36:32.250 DEBUG:teuthology.orchestra.run.smithi088:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd last-stat-seq osd.3 2024-06-03T08:36:32.250 INFO:teuthology.orchestra.run.smithi088.stderr:2024-06-03T08:36:32.249+0000 7f7906ffd640 1 -- 172.21.15.88:0/996211931 <== mon.1 v1:172.21.15.192:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 806751900 0 0) 0x7f7918140f70 con 0x7f791814ae20 2024-06-03T08:36:32.250 INFO:teuthology.orchestra.run.smithi088.stderr:2024-06-03T08:36:32.249+0000 7f7906ffd640 10 monclient(hunting): _negotiate 2024-06-03T08:36:32.251 INFO:teuthology.orchestra.run.smithi088.stderr:2024-06-03T08:36:32.249+0000 7f7906ffd640 10 monclient(hunting): _init_auth method 2 2024-06-03T08:36:32.251 INFO:teuthology.orchestra.run.smithi088.stderr:2024-06-03T08:36:32.249+0000 7f7906ffd640 10 monclient(hunting): _init_auth creating new auth 2024-06-03T08:36:32.251 INFO:teuthology.orchestra.run.smithi088.stderr:2024-06-03T08:36:32.249+0000 7f7906ffd640 10 monclient(hunting): my global_id is 4336 2024-06-03T08:36:32.251 INFO:teuthology.orchestra.run.smithi088.stderr:2024-06-03T08:36:32.249+0000 7f7906ffd640 1 -- 172.21.15.88:0/996211931 --> v1:172.21.15.192:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f78f8004eb0 con 0x7f791814ae20 2024-06-03T08:36:32.251 INFO:teuthology.orchestra.run.smithi088.stderr:2024-06-03T08:36:32.251+0000 7fa0d37fe640 1 -- 172.21.15.88:0/1282623662 <== mon.1 v1:172.21.15.192:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 1632107066 0 0) 0x7fa0ec13ff40 con 0x7fa0ec1450f0 2024-06-03T08:36:32.251 INFO:teuthology.orchestra.run.smithi088.stderr:2024-06-03T08:36:32.251+0000 7fa0d37fe640 10 monclient(hunting): _negotiate 2024-06-03T08:36:32.252 INFO:teuthology.orchestra.run.smithi088.stderr:2024-06-03T08:36:32.251+0000 7fa0d37fe640 10 monclient(hunting): _init_auth method 2 2024-06-03T08:36:32.252 INFO:teuthology.orchestra.run.smithi088.stderr:2024-06-03T08:36:32.251+0000 7fa0d37fe640 10 monclient(hunting): _init_auth creating new auth 2024-06-03T08:36:32.252 INFO:teuthology.orchestra.run.smithi088.stderr:2024-06-03T08:36:32.251+0000 7fa0d37fe640 10 monclient(hunting): my global_id is 4333 2024-06-03T08:36:32.252 INFO:teuthology.orchestra.run.smithi088.stderr:2024-06-03T08:36:32.251+0000 7fa0d37fe640 1 -- 172.21.15.88:0/1282623662 --> v1:172.21.15.192:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7fa0ec1d6630 con 0x7fa0ec1450f0 2024-06-03T08:36:32.260 INFO:teuthology.orchestra.run.smithi088.stderr:2024-06-03T08:36:32.260+0000 7f7906ffd640 1 -- 172.21.15.88:0/996211931 <== mon.1 v1:172.21.15.192:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 764+0+0 (unknown 4216587544 0 0) 0x7f78f8004eb0 con 0x7f791814ae20 2024-06-03T08:36:32.260 INFO:teuthology.orchestra.run.smithi088.stderr:2024-06-03T08:36:32.260+0000 7f7906ffd640 1 -- 172.21.15.88:0/996211931 --> v1:172.21.15.192:6789/0 -- auth(proto 2 165 bytes epoch 0) v1 -- 0x7f7918140f70 con 0x7f791814ae20 2024-06-03T08:36:32.260 INFO:teuthology.orchestra.run.smithi088.stderr:2024-06-03T08:36:32.260+0000 7f7906ffd640 1 -- 172.21.15.88:0/996211931 <== mon.1 v1:172.21.15.192:6789/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 2234374065 0 0) 0x7f7910001ea0 con 0x7f791814ae20 2024-06-03T08:36:32.260 INFO:teuthology.orchestra.run.smithi088.stderr:2024-06-03T08:36:32.260+0000 7f7906ffd640 10 monclient(hunting): handle_monmap mon_map magic: 0 v1 2024-06-03T08:36:32.261 INFO:teuthology.orchestra.run.smithi088.stderr:2024-06-03T08:36:32.260+0000 7f7906ffd640 10 monclient(hunting): got monmap 1 from mon.b (according to old e1) 2024-06-03T08:36:32.261 INFO:teuthology.orchestra.run.smithi088.stderr:2024-06-03T08:36:32.260+0000 7f7906ffd640 10 monclient(hunting): dump: 2024-06-03T08:36:32.261 INFO:teuthology.orchestra.run.smithi088.stderr:epoch 1 2024-06-03T08:36:32.261 INFO:teuthology.orchestra.run.smithi088.stderr:fsid ec3c3fdd-3ac6-4d7d-9412-683130ed8b04 2024-06-03T08:36:32.261 INFO:teuthology.orchestra.run.smithi088.stderr:last_changed 2024-06-03T08:34:31.885125+0000 2024-06-03T08:36:32.261 INFO:teuthology.orchestra.run.smithi088.stderr:created 2024-06-03T08:34:31.885125+0000 2024-06-03T08:36:32.261 INFO:teuthology.orchestra.run.smithi088.stderr:min_mon_release 18 (reef) 2024-06-03T08:36:32.261 INFO:teuthology.orchestra.run.smithi088.stderr:election_strategy: 3 2024-06-03T08:36:32.261 INFO:teuthology.orchestra.run.smithi088.stderr:0: v1:172.21.15.88:6789/0 mon.a 2024-06-03T08:36:32.261 INFO:teuthology.orchestra.run.smithi088.stderr:1: v1:172.21.15.192:6789/0 mon.b 2024-06-03T08:36:32.261 INFO:teuthology.orchestra.run.smithi088.stderr:2: v1:172.21.15.88:6790/0 mon.c 2024-06-03T08:36:32.261 INFO:teuthology.orchestra.run.smithi088.stderr: 2024-06-03T08:36:32.262 INFO:teuthology.orchestra.run.smithi088.stderr:2024-06-03T08:36:32.260+0000 7fa0d37fe640 1 -- 172.21.15.88:0/1282623662 <== mon.1 v1:172.21.15.192:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 764+0+0 (unknown 686841730 0 0) 0x7fa0ec1d6630 con 0x7fa0ec1450f0 2024-06-03T08:36:32.262 INFO:teuthology.orchestra.run.smithi088.stderr:2024-06-03T08:36:32.260+0000 7fa0d37fe640 1 -- 172.21.15.88:0/1282623662 --> v1:172.21.15.192:6789/0 -- auth(proto 2 165 bytes epoch 0) v1 -- 0x7fa0ec13ff40 con 0x7fa0ec1450f0 2024-06-03T08:36:32.262 INFO:teuthology.orchestra.run.smithi088.stderr:2024-06-03T08:36:32.260+0000 7fa0d37fe640 1 -- 172.21.15.88:0/1282623662 <== mon.1 v1:172.21.15.192:6789/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 2234374065 0 0) 0x7fa0e4001bf0 con 0x7fa0ec1450f0 2024-06-03T08:36:32.262 INFO:teuthology.orchestra.run.smithi088.stderr:2024-06-03T08:36:32.260+0000 7fa0d37fe640 10 monclient(hunting): handle_monmap mon_map magic: 0 v1 2024-06-03T08:36:32.262 INFO:teuthology.orchestra.run.smithi088.stderr:2024-06-03T08:36:32.260+0000 7fa0d37fe640 10 monclient(hunting): got monmap 1 from mon.noname-b (according to old e1) 2024-06-03T08:36:32.263 INFO:teuthology.orchestra.run.smithi088.stderr:2024-06-03T08:36:32.260+0000 7fa0d37fe640 10 monclient(hunting): dump: 2024-06-03T08:36:32.263 INFO:teuthology.orchestra.run.smithi088.stderr:epoch 1 2024-06-03T08:36:32.263 INFO:teuthology.orchestra.run.smithi088.stderr:fsid ec3c3fdd-3ac6-4d7d-9412-683130ed8b04 2024-06-03T08:36:32.263 INFO:teuthology.orchestra.run.smithi088.stderr:last_changed 2024-06-03T08:34:31.885125+0000 2024-06-03T08:36:32.263 INFO:teuthology.orchestra.run.smithi088.stderr:created 2024-06-03T08:34:31.885125+0000 2024-06-03T08:36:32.263 INFO:teuthology.orchestra.run.smithi088.stderr:min_mon_release 18 (reef) 2024-06-03T08:36:32.263 INFO:teuthology.orchestra.run.smithi088.stderr:election_strategy: 3 2024-06-03T08:36:32.263 INFO:teuthology.orchestra.run.smithi088.stderr:0: v1:172.21.15.88:6789/0 mon.a 2024-06-03T08:36:32.263 INFO:teuthology.orchestra.run.smithi088.stderr:1: v1:172.21.15.192:6789/0 mon.b 2024-06-03T08:36:32.263 INFO:teuthology.orchestra.run.smithi088.stderr:2: v1:172.21.15.88:6790/0 mon.c 2024-06-03T08:36:32.263 INFO:teuthology.orchestra.run.smithi088.stderr: 2024-06-03T08:36:32.263 INFO:teuthology.orchestra.run.smithi088.stderr:2024-06-03T08:36:32.260+0000 7fa0d37fe640 10 monclient(hunting): _finish_auth 0 2024-06-03T08:36:32.269 INFO:teuthology.orchestra.run.smithi088.stderr:2024-06-03T08:36:32.268+0000 7f7906ffd640 1 -- 172.21.15.88:0/996211931 <== mon.1 v1:172.21.15.192:6789/0 4 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 580+0+0 (unknown 1982793138 0 0) 0x7f7918140f70 con 0x7f791814ae20 2024-06-03T08:36:32.269 INFO:teuthology.orchestra.run.smithi088.stderr:2024-06-03T08:36:32.269+0000 7f7906ffd640 1 -- 172.21.15.88:0/996211931 >> v1:172.21.15.88:6790/0 conn(0x7f791813e6c0 legacy=0x7f7918135300 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-06-03T08:36:32.269 INFO:teuthology.orchestra.run.smithi088.stderr:2024-06-03T08:36:32.269+0000 7f7906ffd640 1 -- 172.21.15.88:0/996211931 >> v1:172.21.15.88:6789/0 conn(0x7f79180b43c0 legacy=0x7f7918142e80 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-06-03T08:36:32.269 INFO:teuthology.orchestra.run.smithi088.stderr:2024-06-03T08:36:32.269+0000 7f7906ffd640 10 monclient: _finish_hunting 0 2024-06-03T08:36:32.269 INFO:teuthology.orchestra.run.smithi088.stderr:2024-06-03T08:36:32.269+0000 7fa0d37fe640 1 -- 172.21.15.88:0/1282623662 <== mon.1 v1:172.21.15.192:6789/0 4 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 580+0+0 (unknown 3964915859 0 0) 0x7fa0ec13ff40 con 0x7fa0ec1450f0 2024-06-03T08:36:32.270 INFO:teuthology.orchestra.run.smithi088.stderr:2024-06-03T08:36:32.269+0000 7f7906ffd640 1 monclient: found mon.b 2024-06-03T08:36:32.270 INFO:teuthology.orchestra.run.smithi088.stderr:2024-06-03T08:36:32.269+0000 7f7906ffd640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-06-03T08:36:32.270 INFO:teuthology.orchestra.run.smithi088.stderr:2024-06-03T08:36:32.269+0000 7f7906ffd640 10 monclient: _send_mon_message to mon.b at v1:172.21.15.192:6789/0 2024-06-03T08:36:32.270 INFO:teuthology.orchestra.run.smithi088.stderr:2024-06-03T08:36:32.269+0000 7f7906ffd640 1 -- 172.21.15.88:0/996211931 --> v1:172.21.15.192:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f7918135ba0 con 0x7f791814ae20 2024-06-03T08:36:32.270 INFO:teuthology.orchestra.run.smithi088.stderr:2024-06-03T08:36:32.269+0000 7f7906ffd640 10 monclient: _finish_auth 0 2024-06-03T08:36:32.270 INFO:teuthology.orchestra.run.smithi088.stderr:2024-06-03T08:36:32.269+0000 7f7906ffd640 10 monclient: _check_auth_tickets 2024-06-03T08:36:32.270 INFO:teuthology.orchestra.run.smithi088.stderr:2024-06-03T08:36:32.269+0000 7f7906ffd640 20 monclient: _check_auth_rotating not needed by client.admin 2024-06-03T08:36:32.270 INFO:teuthology.orchestra.run.smithi088.stderr:2024-06-03T08:36:32.269+0000 7f791cb92640 5 monclient: authenticate success, global_id 4336 2024-06-03T08:36:32.271 INFO:teuthology.orchestra.run.smithi088.stderr:2024-06-03T08:36:32.269+0000 7fa0d37fe640 1 -- 172.21.15.88:0/1282623662 >> v1:172.21.15.88:6790/0 conn(0x7fa0ec13eec0 legacy=0x7fa0ec141b70 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-06-03T08:36:32.271 INFO:teuthology.orchestra.run.smithi088.stderr:2024-06-03T08:36:32.269+0000 7f791cb92640 10 monclient: _renew_subs 2024-06-03T08:36:32.271 INFO:teuthology.orchestra.run.smithi088.stderr:2024-06-03T08:36:32.269+0000 7f791cb92640 10 monclient: _send_mon_message to mon.b at v1:172.21.15.192:6789/0 2024-06-03T08:36:32.271 INFO:teuthology.orchestra.run.smithi088.stderr:2024-06-03T08:36:32.269+0000 7f791cb92640 1 -- 172.21.15.88:0/996211931 --> v1:172.21.15.192:6789/0 -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f7918143bc0 con 0x7f791814ae20 2024-06-03T08:36:32.272 INFO:teuthology.orchestra.run.smithi088.stderr:2024-06-03T08:36:32.269+0000 7fa0d37fe640 1 -- 172.21.15.88:0/1282623662 >> v1:172.21.15.88:6789/0 conn(0x7fa0ec0a8960 legacy=0x7fa0ec1d6b60 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-06-03T08:36:32.272 INFO:teuthology.orchestra.run.smithi088.stderr:2024-06-03T08:36:32.269+0000 7fa0d37fe640 10 monclient: _finish_hunting 0 2024-06-03T08:36:32.272 INFO:teuthology.orchestra.run.smithi088.stderr:2024-06-03T08:36:32.269+0000 7fa0d37fe640 1 monclient: found mon.b 2024-06-03T08:36:32.272 INFO:teuthology.orchestra.run.smithi088.stderr:2024-06-03T08:36:32.269+0000 7fa0d37fe640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-06-03T08:36:32.272 INFO:teuthology.orchestra.run.smithi088.stderr:2024-06-03T08:36:32.269+0000 7fa0d37fe640 10 monclient: _send_mon_message to mon.b at v1:172.21.15.192:6789/0 2024-06-03T08:36:32.272 INFO:teuthology.orchestra.run.smithi088.stderr:2024-06-03T08:36:32.269+0000 7fa0d37fe640 1 -- 172.21.15.88:0/1282623662 --> v1:172.21.15.192:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fa0ec140200 con 0x7fa0ec1450f0 2024-06-03T08:36:32.272 INFO:teuthology.orchestra.run.smithi088.stderr:2024-06-03T08:36:32.269+0000 7f791cb92640 10 client.4336.objecter _maybe_request_map subscribing (onetime) to next osd map 2024-06-03T08:36:32.272 INFO:teuthology.orchestra.run.smithi088.stderr:2024-06-03T08:36:32.269+0000 7f791cb92640 10 monclient: _renew_subs 2024-06-03T08:36:32.273 INFO:teuthology.orchestra.run.smithi088.stderr:2024-06-03T08:36:32.269+0000 7fa0d37fe640 10 monclient: _finish_auth 0 2024-06-03T08:36:32.273 INFO:teuthology.orchestra.run.smithi088.stderr:2024-06-03T08:36:32.269+0000 7f791cb92640 10 monclient: _send_mon_message to mon.b at v1:172.21.15.192:6789/0 2024-06-03T08:36:32.273 INFO:teuthology.orchestra.run.smithi088.stderr:2024-06-03T08:36:32.269+0000 7f791cb92640 1 -- 172.21.15.88:0/996211931 --> v1:172.21.15.192:6789/0 -- mon_subscribe({osdmap=0}) v3 -- 0x7f79181d71d0 con 0x7f791814ae20 2024-06-03T08:36:32.273 INFO:teuthology.orchestra.run.smithi088.stderr:2024-06-03T08:36:32.269+0000 7fa0d37fe640 10 monclient: _check_auth_tickets 2024-06-03T08:36:32.273 INFO:teuthology.orchestra.run.smithi088.stderr:2024-06-03T08:36:32.269+0000 7fa0d37fe640 20 monclient: _check_auth_rotating not needed by client.admin 2024-06-03T08:36:32.274 INFO:teuthology.orchestra.run.smithi088.stderr:2024-06-03T08:36:32.269+0000 7f791cb92640 1 librados: init done 2024-06-03T08:36:32.274 INFO:teuthology.orchestra.run.smithi088.stderr:2024-06-03T08:36:32.269+0000 7fa0f1016640 5 monclient: authenticate success, global_id 4333 2024-06-03T08:36:32.274 INFO:teuthology.orchestra.run.smithi088.stderr:2024-06-03T08:36:32.269+0000 7fa0f1016640 10 monclient: _renew_subs 2024-06-03T08:36:32.275 INFO:teuthology.orchestra.run.smithi088.stderr:2024-06-03T08:36:32.269+0000 7fa0f1016640 10 monclient: _send_mon_message to mon.b at v1:172.21.15.192:6789/0 2024-06-03T08:36:32.275 INFO:teuthology.orchestra.run.smithi088.stderr:2024-06-03T08:36:32.269+0000 7fa0f1016640 1 -- 172.21.15.88:0/1282623662 --> v1:172.21.15.192:6789/0 -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fa0ec1d7270 con 0x7fa0ec1450f0 2024-06-03T08:36:32.275 INFO:teuthology.orchestra.run.smithi088.stderr:2024-06-03T08:36:32.269+0000 7fa0f1016640 10 client.4333.objecter _maybe_request_map subscribing (onetime) to next osd map 2024-06-03T08:36:32.275 INFO:teuthology.orchestra.run.smithi088.stderr:2024-06-03T08:36:32.269+0000 7fa0f1016640 10 monclient: _renew_subs 2024-06-03T08:36:32.275 INFO:teuthology.orchestra.run.smithi088.stderr:2024-06-03T08:36:32.269+0000 7fa0f1016640 10 monclient: _send_mon_message to mon.b at v1:172.21.15.192:6789/0 2024-06-03T08:36:32.275 INFO:teuthology.orchestra.run.smithi088.stderr:2024-06-03T08:36:32.269+0000 7fa0f1016640 1 -- 172.21.15.88:0/1282623662 --> v1:172.21.15.192:6789/0 -- mon_subscribe({osdmap=0}) v3 -- 0x7fa0ec13f960 con 0x7fa0ec1450f0 2024-06-03T08:36:32.275 INFO:teuthology.orchestra.run.smithi088.stderr:2024-06-03T08:36:32.269+0000 7fa0f1016640 1 librados: init done 2024-06-03T08:36:32.275 INFO:teuthology.orchestra.run.smithi088.stderr:2024-06-03T08:36:32.270+0000 7fa0f1016640 10 client.4333.objecter _submit_command 1 [{"prefix": "get_command_descriptions"}] 2024-06-03T08:36:32.275 INFO:teuthology.orchestra.run.smithi088.stderr:2024-06-03T08:36:32.270+0000 7fa0f1016640 15 client.4333.objecter _session_command_op_assign -1 1 2024-06-03T08:36:32.275 INFO:teuthology.orchestra.run.smithi088.stderr:2024-06-03T08:36:32.270+0000 7fa0f1016640 20 client.4333.objecter _get_session osd=-1 returning homeless 2024-06-03T08:36:32.275 INFO:teuthology.orchestra.run.smithi088.stderr:2024-06-03T08:36:32.270+0000 7fa0f1016640 10 client.4333.objecter _maybe_request_map subscribing (onetime) to next osd map 2024-06-03T08:36:32.275 INFO:teuthology.orchestra.run.smithi088.stderr:2024-06-03T08:36:32.270+0000 7fa0f1016640 10 monclient: _send_mon_message to mon.b at v1:172.21.15.192:6789/0 2024-06-03T08:36:32.275 INFO:teuthology.orchestra.run.smithi088.stderr:2024-06-03T08:36:32.270+0000 7fa0f1016640 1 -- 172.21.15.88:0/1282623662 --> v1:172.21.15.192:6789/0 -- mon_get_version(what=osdmap handle=1) v1 -- 0x7fa0ec0a9e60 con 0x7fa0ec1450f0 2024-06-03T08:36:32.276 INFO:teuthology.orchestra.run.smithi088.stderr:2024-06-03T08:36:32.270+0000 7f791cb92640 10 client.4336.objecter _submit_command 1 [{"prefix": "get_command_descriptions"}] 2024-06-03T08:36:32.276 INFO:teuthology.orchestra.run.smithi088.stderr:2024-06-03T08:36:32.270+0000 7f791cb92640 15 client.4336.objecter _session_command_op_assign -1 1 2024-06-03T08:36:32.276 INFO:teuthology.orchestra.run.smithi088.stderr:2024-06-03T08:36:32.270+0000 7f791cb92640 20 client.4336.objecter _get_session osd=-1 returning homeless 2024-06-03T08:36:32.276 INFO:teuthology.orchestra.run.smithi088.stderr:2024-06-03T08:36:32.270+0000 7f791cb92640 10 client.4336.objecter _maybe_request_map subscribing (onetime) to next osd map 2024-06-03T08:36:32.276 INFO:teuthology.orchestra.run.smithi088.stderr:2024-06-03T08:36:32.270+0000 7f791cb92640 10 monclient: _send_mon_message to mon.b at v1:172.21.15.192:6789/0 2024-06-03T08:36:32.276 INFO:teuthology.orchestra.run.smithi088.stderr:2024-06-03T08:36:32.270+0000 7f791cb92640 1 -- 172.21.15.88:0/996211931 --> v1:172.21.15.192:6789/0 -- mon_get_version(what=osdmap handle=1) v1 -- 0x7f79181458b0 con 0x7f791814ae20 2024-06-03T08:36:32.293 INFO:teuthology.orchestra.run.smithi088.stderr:2024-06-03T08:36:32.293+0000 7fa0d37fe640 1 -- 172.21.15.88:0/1282623662 <== mon.1 v1:172.21.15.192:6789/0 5 ==== config(0 keys) v1 ==== 4+0+0 (unknown 0 0 0) 0x7fa0ec0a9e60 con 0x7fa0ec1450f0 2024-06-03T08:36:32.293 INFO:teuthology.orchestra.run.smithi088.stderr:2024-06-03T08:36:32.293+0000 7fa0d37fe640 10 monclient: handle_config config(0 keys) v1 2024-06-03T08:36:32.293 INFO:teuthology.orchestra.run.smithi088.stderr:2024-06-03T08:36:32.293+0000 7fa0d37fe640 1 -- 172.21.15.88:0/1282623662 <== mon.1 v1:172.21.15.192:6789/0 6 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 2234374065 0 0) 0x7fa0e4002100 con 0x7fa0ec1450f0 2024-06-03T08:36:32.293 INFO:teuthology.orchestra.run.smithi088.stderr:2024-06-03T08:36:32.293+0000 7fa0d37fe640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-06-03T08:36:32.293 INFO:teuthology.orchestra.run.smithi088.stderr:2024-06-03T08:36:32.293+0000 7fa0d37fe640 10 monclient: got monmap 1 from mon.b (according to old e1) 2024-06-03T08:36:32.293 INFO:teuthology.orchestra.run.smithi088.stderr:2024-06-03T08:36:32.293+0000 7fa0d37fe640 10 monclient: dump: 2024-06-03T08:36:32.293 INFO:teuthology.orchestra.run.smithi088.stderr:epoch 1 2024-06-03T08:36:32.294 INFO:teuthology.orchestra.run.smithi088.stderr:fsid ec3c3fdd-3ac6-4d7d-9412-683130ed8b04 2024-06-03T08:36:32.294 INFO:teuthology.orchestra.run.smithi088.stderr:last_changed 2024-06-03T08:34:31.885125+0000 2024-06-03T08:36:32.294 INFO:teuthology.orchestra.run.smithi088.stderr:created 2024-06-03T08:34:31.885125+0000 2024-06-03T08:36:32.294 INFO:teuthology.orchestra.run.smithi088.stderr:min_mon_release 18 (reef) 2024-06-03T08:36:32.294 INFO:teuthology.orchestra.run.smithi088.stderr:election_strategy: 3 2024-06-03T08:36:32.294 INFO:teuthology.orchestra.run.smithi088.stderr:0: v1:172.21.15.88:6789/0 mon.a 2024-06-03T08:36:32.294 INFO:teuthology.orchestra.run.smithi088.stderr:1: v1:172.21.15.192:6789/0 mon.b 2024-06-03T08:36:32.294 INFO:teuthology.orchestra.run.smithi088.stderr:2: v1:172.21.15.88:6790/0 mon.c 2024-06-03T08:36:32.294 INFO:teuthology.orchestra.run.smithi088.stderr: 2024-06-03T08:36:32.295 INFO:teuthology.orchestra.run.smithi088.stderr:2024-06-03T08:36:32.293+0000 7fa0e959a640 4 set_mon_vals no callback set 2024-06-03T08:36:32.295 INFO:teuthology.orchestra.run.smithi088.stderr:2024-06-03T08:36:32.293+0000 7fa0d37fe640 1 -- 172.21.15.88:0/1282623662 <== mon.1 v1:172.21.15.192:6789/0 7 ==== mgrmap(e 5) v1 ==== 89110+0+0 (unknown 873533973 0 0) 0x7fa0e4003bb0 con 0x7fa0ec1450f0 2024-06-03T08:36:32.295 INFO:teuthology.orchestra.run.smithi088.stderr:2024-06-03T08:36:32.294+0000 7fa0ead9d640 10 monclient: get_auth_request con 0x7fa0d806d0d0 auth_method 0 2024-06-03T08:36:32.295 INFO:teuthology.orchestra.run.smithi088.stderr:2024-06-03T08:36:32.294+0000 7fa0d37fe640 10 client.4333.objecter ms_handle_connect 0x7fa0d806d0d0 2024-06-03T08:36:32.297 INFO:teuthology.orchestra.run.smithi088.stderr:2024-06-03T08:36:32.296+0000 7f9b9e7fc640 1 -- 172.21.15.88:0/496575353 <== mon.0 v1:172.21.15.88:6789/0 5 ==== config(0 keys) v1 ==== 4+0+0 (unknown 0 0 0) 0x7f9ba8144170 con 0x7f9ba80b43c0 2024-06-03T08:36:32.297 INFO:teuthology.orchestra.run.smithi088.stderr:2024-06-03T08:36:32.296+0000 7f9b9e7fc640 10 monclient: handle_config config(0 keys) v1 2024-06-03T08:36:32.297 INFO:teuthology.orchestra.run.smithi088.stderr:2024-06-03T08:36:32.296+0000 7f9b9e7fc640 1 -- 172.21.15.88:0/496575353 <== mon.0 v1:172.21.15.88:6789/0 6 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 2234374065 0 0) 0x7f9ba4001230 con 0x7f9ba80b43c0 2024-06-03T08:36:32.297 INFO:teuthology.orchestra.run.smithi088.stderr:2024-06-03T08:36:32.296+0000 7f9b9e7fc640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-06-03T08:36:32.297 INFO:teuthology.orchestra.run.smithi088.stderr:2024-06-03T08:36:32.296+0000 7f9b9e7fc640 10 monclient: got monmap 1 from mon.a (according to old e1) 2024-06-03T08:36:32.297 INFO:teuthology.orchestra.run.smithi088.stderr:2024-06-03T08:36:32.296+0000 7f9b9e7fc640 10 monclient: dump: 2024-06-03T08:36:32.297 INFO:teuthology.orchestra.run.smithi088.stderr:epoch 1 2024-06-03T08:36:32.297 INFO:teuthology.orchestra.run.smithi088.stderr:fsid ec3c3fdd-3ac6-4d7d-9412-683130ed8b04 2024-06-03T08:36:32.297 INFO:teuthology.orchestra.run.smithi088.stderr:last_changed 2024-06-03T08:34:31.885125+0000 2024-06-03T08:36:32.297 INFO:teuthology.orchestra.run.smithi088.stderr:created 2024-06-03T08:34:31.885125+0000 2024-06-03T08:36:32.297 INFO:teuthology.orchestra.run.smithi088.stderr:min_mon_release 18 (reef) 2024-06-03T08:36:32.298 INFO:teuthology.orchestra.run.smithi088.stderr:election_strategy: 3 2024-06-03T08:36:32.298 INFO:teuthology.orchestra.run.smithi088.stderr:0: v1:172.21.15.88:6789/0 mon.a 2024-06-03T08:36:32.298 INFO:teuthology.orchestra.run.smithi088.stderr:1: v1:172.21.15.192:6789/0 mon.b 2024-06-03T08:36:32.298 INFO:teuthology.orchestra.run.smithi088.stderr:2: v1:172.21.15.88:6790/0 mon.c 2024-06-03T08:36:32.298 INFO:teuthology.orchestra.run.smithi088.stderr: 2024-06-03T08:36:32.298 INFO:teuthology.orchestra.run.smithi088.stderr:2024-06-03T08:36:32.296+0000 7f9b9ffff640 4 set_mon_vals no callback set 2024-06-03T08:36:32.298 INFO:teuthology.orchestra.run.smithi088.stderr:2024-06-03T08:36:32.297+0000 7f9b9e7fc640 1 -- 172.21.15.88:0/496575353 <== mon.0 v1:172.21.15.88:6789/0 7 ==== mgrmap(e 5) v1 ==== 89110+0+0 (unknown 873533973 0 0) 0x7f9ba401a850 con 0x7f9ba80b43c0 2024-06-03T08:36:32.299 INFO:teuthology.orchestra.run.smithi088.stderr:2024-06-03T08:36:32.298+0000 7f9bad2f1640 10 monclient: get_auth_request con 0x7f9b8c06d120 auth_method 0 2024-06-03T08:36:32.299 INFO:teuthology.orchestra.run.smithi088.stderr:2024-06-03T08:36:32.298+0000 7f9b9e7fc640 10 client.4326.objecter ms_handle_connect 0x7f9b8c06d120 2024-06-03T08:36:32.301 INFO:teuthology.orchestra.run.smithi088.stderr:2024-06-03T08:36:32.300+0000 7f7906ffd640 1 -- 172.21.15.88:0/996211931 <== mon.1 v1:172.21.15.192:6789/0 5 ==== config(0 keys) v1 ==== 4+0+0 (unknown 0 0 0) 0x7f79181458b0 con 0x7f791814ae20 2024-06-03T08:36:32.301 INFO:teuthology.orchestra.run.smithi088.stderr:2024-06-03T08:36:32.300+0000 7f7906ffd640 10 monclient: handle_config config(0 keys) v1 2024-06-03T08:36:32.301 INFO:teuthology.orchestra.run.smithi088.stderr:2024-06-03T08:36:32.300+0000 7f7906ffd640 1 -- 172.21.15.88:0/996211931 <== mon.1 v1:172.21.15.192:6789/0 6 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 2234374065 0 0) 0x7f7910002100 con 0x7f791814ae20 2024-06-03T08:36:32.301 INFO:teuthology.orchestra.run.smithi088.stderr:2024-06-03T08:36:32.300+0000 7f7906ffd640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-06-03T08:36:32.301 INFO:teuthology.orchestra.run.smithi088.stderr:2024-06-03T08:36:32.300+0000 7f7906ffd640 10 monclient: got monmap 1 from mon.b (according to old e1) 2024-06-03T08:36:32.301 INFO:teuthology.orchestra.run.smithi088.stderr:2024-06-03T08:36:32.300+0000 7f7906ffd640 10 monclient: dump: 2024-06-03T08:36:32.301 INFO:teuthology.orchestra.run.smithi088.stderr:epoch 1 2024-06-03T08:36:32.301 INFO:teuthology.orchestra.run.smithi088.stderr:fsid ec3c3fdd-3ac6-4d7d-9412-683130ed8b04 2024-06-03T08:36:32.301 INFO:teuthology.orchestra.run.smithi088.stderr:last_changed 2024-06-03T08:34:31.885125+0000 2024-06-03T08:36:32.301 INFO:teuthology.orchestra.run.smithi088.stderr:created 2024-06-03T08:34:31.885125+0000 2024-06-03T08:36:32.301 INFO:teuthology.orchestra.run.smithi088.stderr:min_mon_release 18 (reef) 2024-06-03T08:36:32.302 INFO:teuthology.orchestra.run.smithi088.stderr:election_strategy: 3 2024-06-03T08:36:32.302 INFO:teuthology.orchestra.run.smithi088.stderr:0: v1:172.21.15.88:6789/0 mon.a 2024-06-03T08:36:32.302 INFO:teuthology.orchestra.run.smithi088.stderr:1: v1:172.21.15.192:6789/0 mon.b 2024-06-03T08:36:32.302 INFO:teuthology.orchestra.run.smithi088.stderr:2: v1:172.21.15.88:6790/0 mon.c 2024-06-03T08:36:32.302 INFO:teuthology.orchestra.run.smithi088.stderr: 2024-06-03T08:36:32.302 INFO:teuthology.orchestra.run.smithi088.stderr:2024-06-03T08:36:32.300+0000 7f7914d99640 4 set_mon_vals no callback set 2024-06-03T08:36:32.302 INFO:teuthology.orchestra.run.smithi088.stderr:2024-06-03T08:36:32.300+0000 7fb10c947640 10 monclient: get_monmap_and_config 2024-06-03T08:36:32.302 INFO:teuthology.orchestra.run.smithi088.stderr:2024-06-03T08:36:32.300+0000 7fb10c947640 10 monclient: build_initial_monmap 2024-06-03T08:36:32.302 INFO:teuthology.orchestra.run.smithi088.stderr:2024-06-03T08:36:32.300+0000 7fb10c947640 10 monclient: monmap: 2024-06-03T08:36:32.302 INFO:teuthology.orchestra.run.smithi088.stderr:epoch 0 2024-06-03T08:36:32.303 INFO:teuthology.orchestra.run.smithi088.stderr:fsid ec3c3fdd-3ac6-4d7d-9412-683130ed8b04 2024-06-03T08:36:32.303 INFO:teuthology.orchestra.run.smithi088.stderr:last_changed 2024-06-03T08:36:32.302003+0000 2024-06-03T08:36:32.303 INFO:teuthology.orchestra.run.smithi088.stderr:created 2024-06-03T08:36:32.302003+0000 2024-06-03T08:36:32.303 INFO:teuthology.orchestra.run.smithi088.stderr:min_mon_release 0 (unknown) 2024-06-03T08:36:32.303 INFO:teuthology.orchestra.run.smithi088.stderr:election_strategy: 3 2024-06-03T08:36:32.303 INFO:teuthology.orchestra.run.smithi088.stderr:0: v1:172.21.15.88:6789/0 mon.noname-a 2024-06-03T08:36:32.303 INFO:teuthology.orchestra.run.smithi088.stderr:1: v1:172.21.15.192:6789/0 mon.noname-c 2024-06-03T08:36:32.303 INFO:teuthology.orchestra.run.smithi088.stderr:2: v1:172.21.15.88:6790/0 mon.noname-b 2024-06-03T08:36:32.303 INFO:teuthology.orchestra.run.smithi088.stderr: 2024-06-03T08:36:32.303 INFO:teuthology.orchestra.run.smithi088.stderr:2024-06-03T08:36:32.301+0000 7f7906ffd640 1 -- 172.21.15.88:0/996211931 <== mon.1 v1:172.21.15.192:6789/0 7 ==== mgrmap(e 5) v1 ==== 89110+0+0 (unknown 873533973 0 0) 0x7f79100199e0 con 0x7f791814ae20 2024-06-03T08:36:32.304 INFO:teuthology.orchestra.run.smithi088.stderr:2024-06-03T08:36:32.301+0000 7fb10c947640 1 Processor -- start 2024-06-03T08:36:32.304 INFO:teuthology.orchestra.run.smithi088.stderr:2024-06-03T08:36:32.301+0000 7fb10c947640 1 -- start start 2024-06-03T08:36:32.304 INFO:teuthology.orchestra.run.smithi088.stderr:2024-06-03T08:36:32.301+0000 7fb10c947640 10 monclient: init 2024-06-03T08:36:32.304 INFO:teuthology.orchestra.run.smithi088.stderr:2024-06-03T08:36:32.302+0000 7f791659c640 10 monclient: get_auth_request con 0x7f78f806d100 auth_method 0 2024-06-03T08:36:32.305 INFO:teuthology.orchestra.run.smithi088.stderr:2024-06-03T08:36:32.302+0000 7fb10c947640 10 monclient: _reopen_session rank -1 2024-06-03T08:36:32.305 INFO:teuthology.orchestra.run.smithi088.stderr:2024-06-03T08:36:32.302+0000 7fb10c947640 10 monclient: _add_conns ranks=[1,2,0] 2024-06-03T08:36:32.305 INFO:teuthology.orchestra.run.smithi088.stderr:2024-06-03T08:36:32.302+0000 7fb10c947640 10 monclient(hunting): picked mon.noname-c con 0x7fb1080a8960 addr v1:172.21.15.192:6789/0 2024-06-03T08:36:32.305 INFO:teuthology.orchestra.run.smithi088.stderr:2024-06-03T08:36:32.302+0000 7fb10c947640 10 monclient(hunting): picked mon.noname-b con 0x7fb1081450f0 addr v1:172.21.15.88:6790/0 2024-06-03T08:36:32.305 INFO:teuthology.orchestra.run.smithi088.stderr:2024-06-03T08:36:32.302+0000 7fb10c947640 10 monclient(hunting): picked mon.noname-a con 0x7fb10813eec0 addr v1:172.21.15.88:6789/0 2024-06-03T08:36:32.305 INFO:teuthology.orchestra.run.smithi088.stderr:2024-06-03T08:36:32.302+0000 7fb10c947640 1 -- --> v1:172.21.15.88:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fb108144c60 con 0x7fb10813eec0 2024-06-03T08:36:32.306 INFO:teuthology.orchestra.run.smithi088.stderr:2024-06-03T08:36:32.302+0000 7fb10c947640 1 -- --> v1:172.21.15.192:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fb1080b3930 con 0x7fb1080a8960 2024-06-03T08:36:32.306 INFO:teuthology.orchestra.run.smithi088.stderr:2024-06-03T08:36:32.302+0000 7fb10c947640 1 -- --> v1:172.21.15.88:6790/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fb1080b1ad0 con 0x7fb1081450f0 2024-06-03T08:36:32.306 INFO:teuthology.orchestra.run.smithi088.stderr:2024-06-03T08:36:32.302+0000 7fb10c947640 10 monclient(hunting): _renew_subs 2024-06-03T08:36:32.306 INFO:teuthology.orchestra.run.smithi088.stderr:2024-06-03T08:36:32.302+0000 7fb10c947640 10 monclient(hunting): authenticate will time out at 1248.112593s 2024-06-03T08:36:32.306 INFO:teuthology.orchestra.run.smithi088.stderr:2024-06-03T08:36:32.302+0000 7f7906ffd640 10 client.4336.objecter ms_handle_connect 0x7f78f806d100 2024-06-03T08:36:32.306 INFO:teuthology.orchestra.run.smithi088.stderr:2024-06-03T08:36:32.303+0000 7fb0fffff640 1 --1- >> v1:172.21.15.88:6790/0 conn(0x7fb1081450f0 0x7fb10813e7b0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.88:6790/0 says I am v1:172.21.15.88:54252/0 (socket says 172.21.15.88:54252) 2024-06-03T08:36:32.306 INFO:teuthology.orchestra.run.smithi088.stderr:2024-06-03T08:36:32.303+0000 7fb0fffff640 1 -- 172.21.15.88:0/2407293285 learned_addr learned my addr 172.21.15.88:0/2407293285 (peer_addr_for_me v1:172.21.15.88:0/0) 2024-06-03T08:36:32.307 INFO:teuthology.orchestra.run.smithi088.stderr:2024-06-03T08:36:32.303+0000 7fa0d37fe640 1 -- 172.21.15.88:0/1282623662 <== mon.1 v1:172.21.15.192:6789/0 8 ==== osd_map(26..26 src has 1..26) v4 ==== 4090+0+0 (unknown 3637450554 0 0) 0x7fa0e40530f0 con 0x7fa0ec1450f0 2024-06-03T08:36:32.307 INFO:teuthology.orchestra.run.smithi088.stderr:2024-06-03T08:36:32.303+0000 7fa0d37fe640 10 client.4333.objecter ms_dispatch 0x7fa0ec000b90 osd_map(26..26 src has 1..26) v4 2024-06-03T08:36:32.307 INFO:teuthology.orchestra.run.smithi088.stderr:2024-06-03T08:36:32.303+0000 7fa0d37fe640 3 client.4333.objecter handle_osd_map got epochs [26,26] > 0 2024-06-03T08:36:32.307 INFO:teuthology.orchestra.run.smithi088.stderr:2024-06-03T08:36:32.303+0000 7fa0d37fe640 3 client.4333.objecter handle_osd_map decoding full epoch 26 2024-06-03T08:36:32.308 INFO:teuthology.orchestra.run.smithi088.stderr:2024-06-03T08:36:32.303+0000 7f9b9e7fc640 1 -- 172.21.15.88:0/496575353 <== mon.0 v1:172.21.15.88:6789/0 8 ==== osd_map(26..26 src has 1..26) v4 ==== 4090+0+0 (unknown 3637450554 0 0) 0x7f9ba4054010 con 0x7f9ba80b43c0 2024-06-03T08:36:32.308 INFO:teuthology.orchestra.run.smithi088.stderr:2024-06-03T08:36:32.303+0000 7f9b9e7fc640 10 client.4326.objecter ms_dispatch 0x7f9ba8000b90 osd_map(26..26 src has 1..26) v4 2024-06-03T08:36:32.308 INFO:teuthology.orchestra.run.smithi088.stderr:2024-06-03T08:36:32.303+0000 7f9b9e7fc640 3 client.4326.objecter handle_osd_map got epochs [26,26] > 0 2024-06-03T08:36:32.308 INFO:teuthology.orchestra.run.smithi088.stderr:2024-06-03T08:36:32.303+0000 7f9b9e7fc640 3 client.4326.objecter handle_osd_map decoding full epoch 26 2024-06-03T08:36:32.308 INFO:teuthology.orchestra.run.smithi088.stderr:2024-06-03T08:36:32.303+0000 7f9b9e7fc640 10 client.4326.objecter checking command 1 2024-06-03T08:36:32.308 INFO:teuthology.orchestra.run.smithi088.stderr:2024-06-03T08:36:32.303+0000 7f9b9e7fc640 20 client.4326.objecter _get_session s=0x7f9b8c073aa0 osd=6 3 2024-06-03T08:36:32.308 INFO:teuthology.orchestra.run.smithi088.stderr:2024-06-03T08:36:32.303+0000 7f9b9e7fc640 20 client.4326.objecter put_session s=0x7f9b8c073aa0 osd=6 3 2024-06-03T08:36:32.308 INFO:teuthology.orchestra.run.smithi088.stderr:2024-06-03T08:36:32.303+0000 7f9b9e7fc640 15 client.4326.objecter _session_command_op_remove -1 1 2024-06-03T08:36:32.308 INFO:teuthology.orchestra.run.smithi088.stderr:2024-06-03T08:36:32.303+0000 7f9b9e7fc640 20 client.4326.objecter _get_session s=0x7f9b8c073aa0 osd=6 3 2024-06-03T08:36:32.308 INFO:teuthology.orchestra.run.smithi088.stderr:2024-06-03T08:36:32.303+0000 7fa0d37fe640 10 client.4333.objecter checking command 1 2024-06-03T08:36:32.309 INFO:teuthology.orchestra.run.smithi088.stderr:2024-06-03T08:36:32.303+0000 7f9b9e7fc640 20 client.4326.objecter get_session s=0x7f9b8c073aa0 osd=6 3 2024-06-03T08:36:32.309 INFO:teuthology.orchestra.run.smithi088.stderr:2024-06-03T08:36:32.303+0000 7f9b9e7fc640 15 client.4326.objecter _session_command_op_assign 6 1 2024-06-03T08:36:32.309 INFO:teuthology.orchestra.run.smithi088.stderr:2024-06-03T08:36:32.303+0000 7f9b9e7fc640 20 client.4326.objecter put_session s=0x7f9b8c073aa0 osd=6 4 2024-06-03T08:36:32.309 INFO:teuthology.orchestra.run.smithi088.stderr:2024-06-03T08:36:32.303+0000 7f9b9e7fc640 10 client.4326.objecter _send_command 1 2024-06-03T08:36:32.310 INFO:teuthology.orchestra.run.smithi088.stderr:2024-06-03T08:36:32.303+0000 7f9b9e7fc640 1 -- 172.21.15.88:0/496575353 --> v1:172.21.15.192:6813/34788 -- command(tid 1: {"prefix": "get_command_descriptions"}) v1 -- 0x7f9b8c077d20 con 0x7f9b8c074150 2024-06-03T08:36:32.310 INFO:teuthology.orchestra.run.smithi088.stderr:2024-06-03T08:36:32.303+0000 7f9b9e7fc640 20 client.4326.objecter dump_active .. 0 homeless 2024-06-03T08:36:32.310 INFO:teuthology.orchestra.run.smithi088.stderr:2024-06-03T08:36:32.303+0000 7f9b9e7fc640 1 -- 172.21.15.88:0/496575353 <== mon.0 v1:172.21.15.88:6789/0 9 ==== mon_get_version_reply(handle=1 version=26) v2 ==== 24+0+0 (unknown 2370098492 0 0) 0x7f9ba404f9f0 con 0x7f9ba80b43c0 2024-06-03T08:36:32.310 INFO:teuthology.orchestra.run.smithi088.stderr:2024-06-03T08:36:32.303+0000 7f9b9e7fc640 10 monclient: handle_get_version_reply finishing 1 version 26 2024-06-03T08:36:32.311 INFO:teuthology.orchestra.run.smithi088.stderr:2024-06-03T08:36:32.303+0000 7fa0d37fe640 20 client.4333.objecter _get_session s=0x7fa0d8073ac0 osd=5 3 2024-06-03T08:36:32.311 INFO:teuthology.orchestra.run.smithi088.stderr:2024-06-03T08:36:32.303+0000 7fa0d37fe640 20 client.4333.objecter put_session s=0x7fa0d8073ac0 osd=5 3 2024-06-03T08:36:32.311 INFO:teuthology.orchestra.run.smithi088.stderr:2024-06-03T08:36:32.304+0000 7fa0d37fe640 15 client.4333.objecter _session_command_op_remove -1 1 2024-06-03T08:36:32.311 INFO:teuthology.orchestra.run.smithi088.stderr:2024-06-03T08:36:32.304+0000 7fa0d37fe640 20 client.4333.objecter _get_session s=0x7fa0d8073ac0 osd=5 3 2024-06-03T08:36:32.311 INFO:teuthology.orchestra.run.smithi088.stderr:2024-06-03T08:36:32.304+0000 7fa0d37fe640 20 client.4333.objecter get_session s=0x7fa0d8073ac0 osd=5 3 2024-06-03T08:36:32.311 INFO:teuthology.orchestra.run.smithi088.stderr:2024-06-03T08:36:32.304+0000 7fa0d37fe640 15 client.4333.objecter _session_command_op_assign 5 1 2024-06-03T08:36:32.311 INFO:teuthology.orchestra.run.smithi088.stderr:2024-06-03T08:36:32.304+0000 7fa0d37fe640 20 client.4333.objecter put_session s=0x7fa0d8073ac0 osd=5 4 2024-06-03T08:36:32.312 INFO:teuthology.orchestra.run.smithi088.stderr:2024-06-03T08:36:32.304+0000 7fa0d37fe640 10 client.4333.objecter _send_command 1 2024-06-03T08:36:32.312 INFO:teuthology.orchestra.run.smithi088.stderr:2024-06-03T08:36:32.304+0000 7fa0d37fe640 1 -- 172.21.15.88:0/1282623662 --> v1:172.21.15.192:6805/34786 -- command(tid 1: {"prefix": "get_command_descriptions"}) v1 -- 0x7fa0d8077d40 con 0x7fa0d8074170 2024-06-03T08:36:32.312 INFO:teuthology.orchestra.run.smithi088.stderr:2024-06-03T08:36:32.304+0000 7fa0d37fe640 20 client.4333.objecter dump_active .. 0 homeless 2024-06-03T08:36:32.312 INFO:teuthology.orchestra.run.smithi088.stderr:2024-06-03T08:36:32.304+0000 7fa0d37fe640 1 -- 172.21.15.88:0/1282623662 <== mon.1 v1:172.21.15.192:6789/0 9 ==== mon_get_version_reply(handle=1 version=26) v2 ==== 24+0+0 (unknown 2370098492 0 0) 0x7fa0e404ece0 con 0x7fa0ec1450f0 2024-06-03T08:36:32.312 INFO:teuthology.orchestra.run.smithi088.stderr:2024-06-03T08:36:32.304+0000 7fa0d37fe640 10 monclient: handle_get_version_reply finishing 1 version 26 2024-06-03T08:36:32.312 INFO:teuthology.orchestra.run.smithi088.stderr:2024-06-03T08:36:32.304+0000 7f9bae2f3640 10 monclient: get_auth_request con 0x7f9b8c074150 auth_method 0 2024-06-03T08:36:32.312 INFO:teuthology.orchestra.run.smithi088.stderr:2024-06-03T08:36:32.304+0000 7fa0ea59c640 10 monclient: get_auth_request con 0x7fa0d8074170 auth_method 0 2024-06-03T08:36:32.313 INFO:teuthology.orchestra.run.smithi088.stderr:2024-06-03T08:36:32.306+0000 7fb105d9b640 1 -- 172.21.15.88:0/2407293285 <== mon.2 v1:172.21.15.88:6790/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 3150479026 0 0) 0x7fb1080b1ad0 con 0x7fb1081450f0 2024-06-03T08:36:32.313 INFO:teuthology.orchestra.run.smithi088.stderr:2024-06-03T08:36:32.306+0000 7fb105d9b640 10 monclient(hunting): _negotiate 2024-06-03T08:36:32.313 INFO:teuthology.orchestra.run.smithi088.stderr:2024-06-03T08:36:32.306+0000 7fb105d9b640 10 monclient(hunting): _init_auth method 2 2024-06-03T08:36:32.313 INFO:teuthology.orchestra.run.smithi088.stderr:2024-06-03T08:36:32.306+0000 7fb105d9b640 10 monclient(hunting): _init_auth creating new auth 2024-06-03T08:36:32.313 INFO:teuthology.orchestra.run.smithi088.stderr:2024-06-03T08:36:32.306+0000 7fb105d9b640 10 monclient(hunting): my global_id is 4343 2024-06-03T08:36:32.313 INFO:teuthology.orchestra.run.smithi088.stderr:2024-06-03T08:36:32.306+0000 7fb105d9b640 1 -- 172.21.15.88:0/2407293285 --> v1:172.21.15.88:6790/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7fb0f4003520 con 0x7fb1081450f0 2024-06-03T08:36:32.313 INFO:teuthology.orchestra.run.smithi088.stderr:2024-06-03T08:36:32.307+0000 7f9b9e7fc640 10 client.4326.objecter ms_handle_connect 0x7f9b8c074150 2024-06-03T08:36:32.314 INFO:teuthology.orchestra.run.smithi088.stderr:2024-06-03T08:36:32.307+0000 7fa0d37fe640 10 client.4333.objecter ms_handle_connect 0x7fa0d8074170 2024-06-03T08:36:32.314 INFO:teuthology.orchestra.run.smithi088.stderr:2024-06-03T08:36:32.309+0000 7f7906ffd640 1 -- 172.21.15.88:0/996211931 <== mon.1 v1:172.21.15.192:6789/0 8 ==== osd_map(26..26 src has 1..26) v4 ==== 4090+0+0 (unknown 3637450554 0 0) 0x7f7910053190 con 0x7f791814ae20 2024-06-03T08:36:32.314 INFO:teuthology.orchestra.run.smithi088.stderr:2024-06-03T08:36:32.309+0000 7f7906ffd640 10 client.4336.objecter ms_dispatch 0x7f7918000b90 osd_map(26..26 src has 1..26) v4 2024-06-03T08:36:32.315 INFO:teuthology.orchestra.run.smithi088.stderr:2024-06-03T08:36:32.309+0000 7f7906ffd640 3 client.4336.objecter handle_osd_map got epochs [26,26] > 0 2024-06-03T08:36:32.315 INFO:teuthology.orchestra.run.smithi088.stderr:2024-06-03T08:36:32.309+0000 7f7906ffd640 3 client.4336.objecter handle_osd_map decoding full epoch 26 2024-06-03T08:36:32.315 INFO:teuthology.orchestra.run.smithi088.stderr:2024-06-03T08:36:32.309+0000 7f7906ffd640 10 client.4336.objecter checking command 1 2024-06-03T08:36:32.315 INFO:teuthology.orchestra.run.smithi088.stderr:2024-06-03T08:36:32.309+0000 7f7906ffd640 20 client.4336.objecter _get_session s=0x7f78f8073b10 osd=7 3 2024-06-03T08:36:32.315 INFO:teuthology.orchestra.run.smithi088.stderr:2024-06-03T08:36:32.309+0000 7f7906ffd640 20 client.4336.objecter put_session s=0x7f78f8073b10 osd=7 3 2024-06-03T08:36:32.315 INFO:teuthology.orchestra.run.smithi088.stderr:2024-06-03T08:36:32.309+0000 7f7906ffd640 15 client.4336.objecter _session_command_op_remove -1 1 2024-06-03T08:36:32.315 INFO:teuthology.orchestra.run.smithi088.stderr:2024-06-03T08:36:32.309+0000 7f7906ffd640 20 client.4336.objecter _get_session s=0x7f78f8073b10 osd=7 3 2024-06-03T08:36:32.315 INFO:teuthology.orchestra.run.smithi088.stderr:2024-06-03T08:36:32.309+0000 7f7906ffd640 20 client.4336.objecter get_session s=0x7f78f8073b10 osd=7 3 2024-06-03T08:36:32.315 INFO:teuthology.orchestra.run.smithi088.stderr:2024-06-03T08:36:32.309+0000 7f7906ffd640 15 client.4336.objecter _session_command_op_assign 7 1 2024-06-03T08:36:32.315 INFO:teuthology.orchestra.run.smithi088.stderr:2024-06-03T08:36:32.309+0000 7f7906ffd640 20 client.4336.objecter put_session s=0x7f78f8073b10 osd=7 4 2024-06-03T08:36:32.315 INFO:teuthology.orchestra.run.smithi088.stderr:2024-06-03T08:36:32.309+0000 7f7906ffd640 10 client.4336.objecter _send_command 1 2024-06-03T08:36:32.315 INFO:teuthology.orchestra.run.smithi088.stderr:2024-06-03T08:36:32.309+0000 7f7906ffd640 1 -- 172.21.15.88:0/996211931 --> v1:172.21.15.192:6801/34787 -- command(tid 1: {"prefix": "get_command_descriptions"}) v1 -- 0x7f78f8077e30 con 0x7f78f80741c0 2024-06-03T08:36:32.315 INFO:teuthology.orchestra.run.smithi088.stderr:2024-06-03T08:36:32.309+0000 7f7906ffd640 20 client.4336.objecter dump_active .. 0 homeless 2024-06-03T08:36:32.316 INFO:teuthology.orchestra.run.smithi088.stderr:2024-06-03T08:36:32.309+0000 7f7906ffd640 1 -- 172.21.15.88:0/996211931 <== mon.1 v1:172.21.15.192:6789/0 9 ==== mon_get_version_reply(handle=1 version=26) v2 ==== 24+0+0 (unknown 2370098492 0 0) 0x7f791004ed40 con 0x7f791814ae20 2024-06-03T08:36:32.316 INFO:teuthology.orchestra.run.smithi088.stderr:2024-06-03T08:36:32.309+0000 7f7906ffd640 10 monclient: handle_get_version_reply finishing 1 version 26 2024-06-03T08:36:32.316 INFO:teuthology.orchestra.run.smithi088.stderr:2024-06-03T08:36:32.310+0000 7fb105d9b640 1 -- 172.21.15.88:0/2407293285 <== mon.2 v1:172.21.15.88:6790/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 214+0+0 (unknown 3208779843 0 0) 0x7fb0f4003520 con 0x7fb1081450f0 2024-06-03T08:36:32.316 INFO:teuthology.orchestra.run.smithi088.stderr:2024-06-03T08:36:32.310+0000 7fb105d9b640 1 -- 172.21.15.88:0/2407293285 >> v1:172.21.15.192:6789/0 conn(0x7fb1080a8960 legacy=0x7fb1080a8d40 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-06-03T08:36:32.316 INFO:teuthology.orchestra.run.smithi088.stderr:2024-06-03T08:36:32.310+0000 7fb105d9b640 1 -- 172.21.15.88:0/2407293285 >> v1:172.21.15.88:6789/0 conn(0x7fb10813eec0 legacy=0x7fb10814ce60 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-06-03T08:36:32.316 INFO:teuthology.orchestra.run.smithi088.stderr:2024-06-03T08:36:32.310+0000 7fb105d9b640 10 monclient: _finish_hunting 0 2024-06-03T08:36:32.316 INFO:teuthology.orchestra.run.smithi088.stderr:2024-06-03T08:36:32.310+0000 7fb105d9b640 1 monclient: found mon.noname-b 2024-06-03T08:36:32.316 INFO:teuthology.orchestra.run.smithi088.stderr:2024-06-03T08:36:32.310+0000 7fb105d9b640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-06-03T08:36:32.317 INFO:teuthology.orchestra.run.smithi088.stderr:2024-06-03T08:36:32.310+0000 7fb105d9b640 10 monclient: _send_mon_message to mon.noname-b at v1:172.21.15.88:6790/0 2024-06-03T08:36:32.317 INFO:teuthology.orchestra.run.smithi088.stderr:2024-06-03T08:36:32.310+0000 7fb105d9b640 1 -- 172.21.15.88:0/2407293285 --> v1:172.21.15.88:6790/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fb1080b1c60 con 0x7fb1081450f0 2024-06-03T08:36:32.317 INFO:teuthology.orchestra.run.smithi088.stderr:2024-06-03T08:36:32.310+0000 7fb105d9b640 10 monclient: _finish_auth 0 2024-06-03T08:36:32.317 INFO:teuthology.orchestra.run.smithi088.stderr:2024-06-03T08:36:32.310+0000 7fb105d9b640 10 monclient: _check_auth_tickets 2024-06-03T08:36:32.317 INFO:teuthology.orchestra.run.smithi088.stderr:2024-06-03T08:36:32.310+0000 7fb105d9b640 20 monclient: _check_auth_rotating not needed by client.admin 2024-06-03T08:36:32.317 INFO:teuthology.orchestra.run.smithi088.stderr:2024-06-03T08:36:32.310+0000 7fb10c947640 5 monclient: authenticate success, global_id 4343 2024-06-03T08:36:32.317 INFO:teuthology.orchestra.run.smithi088.stderr:2024-06-03T08:36:32.310+0000 7fb10c947640 20 monclient: get_monmap_and_config waiting for monmap|config 2024-06-03T08:36:32.317 INFO:teuthology.orchestra.run.smithi088.stderr:2024-06-03T08:36:32.310+0000 7fb105d9b640 1 -- 172.21.15.88:0/2407293285 <== mon.2 v1:172.21.15.88:6790/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 2234374065 0 0) 0x7fb0f0001610 con 0x7fb1081450f0 2024-06-03T08:36:32.317 INFO:teuthology.orchestra.run.smithi088.stderr:2024-06-03T08:36:32.310+0000 7fb105d9b640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-06-03T08:36:32.317 INFO:teuthology.orchestra.run.smithi088.stderr:2024-06-03T08:36:32.310+0000 7fb105d9b640 10 monclient: got monmap 1 from mon.noname-b (according to old e1) 2024-06-03T08:36:32.317 INFO:teuthology.orchestra.run.smithi088.stderr:2024-06-03T08:36:32.310+0000 7fb105d9b640 10 monclient: dump: 2024-06-03T08:36:32.317 INFO:teuthology.orchestra.run.smithi088.stderr:epoch 1 2024-06-03T08:36:32.317 INFO:teuthology.orchestra.run.smithi088.stderr:fsid ec3c3fdd-3ac6-4d7d-9412-683130ed8b04 2024-06-03T08:36:32.317 INFO:teuthology.orchestra.run.smithi088.stderr:last_changed 2024-06-03T08:34:31.885125+0000 2024-06-03T08:36:32.318 INFO:teuthology.orchestra.run.smithi088.stderr:created 2024-06-03T08:34:31.885125+0000 2024-06-03T08:36:32.318 INFO:teuthology.orchestra.run.smithi088.stderr:min_mon_release 18 (reef) 2024-06-03T08:36:32.318 INFO:teuthology.orchestra.run.smithi088.stderr:election_strategy: 3 2024-06-03T08:36:32.318 INFO:teuthology.orchestra.run.smithi088.stderr:0: v1:172.21.15.88:6789/0 mon.a 2024-06-03T08:36:32.318 INFO:teuthology.orchestra.run.smithi088.stderr:1: v1:172.21.15.192:6789/0 mon.b 2024-06-03T08:36:32.318 INFO:teuthology.orchestra.run.smithi088.stderr:2: v1:172.21.15.88:6790/0 mon.c 2024-06-03T08:36:32.318 INFO:teuthology.orchestra.run.smithi088.stderr: 2024-06-03T08:36:32.318 INFO:teuthology.orchestra.run.smithi088.stderr:2024-06-03T08:36:32.310+0000 7fb10c947640 20 monclient: get_monmap_and_config waiting for monmap|config 2024-06-03T08:36:32.318 INFO:teuthology.orchestra.run.smithi088.stderr:2024-06-03T08:36:32.311+0000 7f7915d9b640 10 monclient: get_auth_request con 0x7f78f80741c0 auth_method 0 2024-06-03T08:36:32.319 INFO:teuthology.orchestra.run.smithi088.stderr:2024-06-03T08:36:32.312+0000 7fb105d9b640 1 -- 172.21.15.88:0/2407293285 <== mon.2 v1:172.21.15.88:6790/0 4 ==== config(0 keys) v1 ==== 4+0+0 (unknown 0 0 0) 0x7fb0f0001900 con 0x7fb1081450f0 2024-06-03T08:36:32.319 INFO:teuthology.orchestra.run.smithi088.stderr:2024-06-03T08:36:32.312+0000 7fb105d9b640 10 monclient: handle_config config(0 keys) v1 2024-06-03T08:36:32.319 INFO:teuthology.orchestra.run.smithi088.stderr:2024-06-03T08:36:32.312+0000 7fb10c947640 10 monclient: get_monmap_and_config success 2024-06-03T08:36:32.319 INFO:teuthology.orchestra.run.smithi088.stderr:2024-06-03T08:36:32.312+0000 7fb10c947640 4 set_mon_vals no callback set 2024-06-03T08:36:32.319 INFO:teuthology.orchestra.run.smithi088.stderr:2024-06-03T08:36:32.312+0000 7fb105d9b640 1 -- 172.21.15.88:0/2407293285 <== mon.2 v1:172.21.15.88:6790/0 5 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 2234374065 0 0) 0x7fb0f0001610 con 0x7fb1081450f0 2024-06-03T08:36:32.320 INFO:teuthology.orchestra.run.smithi088.stderr:2024-06-03T08:36:32.312+0000 7fb105d9b640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-06-03T08:36:32.320 INFO:teuthology.orchestra.run.smithi088.stderr:2024-06-03T08:36:32.312+0000 7fb105d9b640 10 monclient: got monmap 1 from mon.c (according to old e1) 2024-06-03T08:36:32.320 INFO:teuthology.orchestra.run.smithi088.stderr:2024-06-03T08:36:32.312+0000 7fb105d9b640 10 monclient: dump: 2024-06-03T08:36:32.320 INFO:teuthology.orchestra.run.smithi088.stderr:epoch 1 2024-06-03T08:36:32.320 INFO:teuthology.orchestra.run.smithi088.stderr:fsid ec3c3fdd-3ac6-4d7d-9412-683130ed8b04 2024-06-03T08:36:32.320 INFO:teuthology.orchestra.run.smithi088.stderr:last_changed 2024-06-03T08:34:31.885125+0000 2024-06-03T08:36:32.320 INFO:teuthology.orchestra.run.smithi088.stderr:created 2024-06-03T08:34:31.885125+0000 2024-06-03T08:36:32.320 INFO:teuthology.orchestra.run.smithi088.stderr:min_mon_release 18 (reef) 2024-06-03T08:36:32.320 INFO:teuthology.orchestra.run.smithi088.stderr:election_strategy: 3 2024-06-03T08:36:32.320 INFO:teuthology.orchestra.run.smithi088.stderr:0: v1:172.21.15.88:6789/0 mon.a 2024-06-03T08:36:32.320 INFO:teuthology.orchestra.run.smithi088.stderr:1: v1:172.21.15.192:6789/0 mon.b 2024-06-03T08:36:32.320 INFO:teuthology.orchestra.run.smithi088.stderr:2: v1:172.21.15.88:6790/0 mon.c 2024-06-03T08:36:32.321 INFO:teuthology.orchestra.run.smithi088.stderr: 2024-06-03T08:36:32.321 INFO:teuthology.orchestra.run.smithi088.stderr:2024-06-03T08:36:32.312+0000 7fb10c947640 10 monclient: shutdown 2024-06-03T08:36:32.321 INFO:teuthology.orchestra.run.smithi088.stderr:2024-06-03T08:36:32.312+0000 7fb10c947640 20 monclient: shutdown discarding 0 pending message(s) 2024-06-03T08:36:32.321 INFO:teuthology.orchestra.run.smithi088.stderr:2024-06-03T08:36:32.312+0000 7fb10c947640 1 -- 172.21.15.88:0/2407293285 >> v1:172.21.15.88:6790/0 conn(0x7fb1081450f0 legacy=0x7fb10813e7b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-06-03T08:36:32.321 INFO:teuthology.orchestra.run.smithi088.stderr:2024-06-03T08:36:32.313+0000 7fb10c947640 1 -- 172.21.15.88:0/2407293285 shutdown_connections 2024-06-03T08:36:32.321 INFO:teuthology.orchestra.run.smithi088.stderr:2024-06-03T08:36:32.313+0000 7fb10c947640 1 -- 172.21.15.88:0/2407293285 >> 172.21.15.88:0/2407293285 conn(0x7fb1080b70d0 msgr2=0x7fb1080b74e0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-06-03T08:36:32.321 INFO:teuthology.orchestra.run.smithi088.stderr:2024-06-03T08:36:32.313+0000 7fb10c947640 1 -- 172.21.15.88:0/2407293285 shutdown_connections 2024-06-03T08:36:32.321 INFO:teuthology.orchestra.run.smithi088.stderr:2024-06-03T08:36:32.313+0000 7fb10c947640 1 -- 172.21.15.88:0/2407293285 wait complete. 2024-06-03T08:36:32.321 INFO:teuthology.orchestra.run.smithi088.stderr:2024-06-03T08:36:32.313+0000 7f7906ffd640 10 client.4336.objecter ms_handle_connect 0x7f78f80741c0 2024-06-03T08:36:32.322 INFO:teuthology.orchestra.run.smithi088.stderr:2024-06-03T08:36:32.313+0000 7fb10c947640 10 monclient: build_initial_monmap 2024-06-03T08:36:32.322 INFO:teuthology.orchestra.run.smithi088.stderr:2024-06-03T08:36:32.314+0000 7fb10c947640 10 monclient: monmap: 2024-06-03T08:36:32.323 INFO:teuthology.orchestra.run.smithi088.stderr:epoch 0 2024-06-03T08:36:32.323 INFO:teuthology.orchestra.run.smithi088.stderr:fsid 00000000-0000-0000-0000-000000000000 2024-06-03T08:36:32.323 INFO:teuthology.orchestra.run.smithi088.stderr:last_changed 0.000000 2024-06-03T08:36:32.323 INFO:teuthology.orchestra.run.smithi088.stderr:created 0.000000 2024-06-03T08:36:32.323 INFO:teuthology.orchestra.run.smithi088.stderr:min_mon_release 0 (unknown) 2024-06-03T08:36:32.323 INFO:teuthology.orchestra.run.smithi088.stderr:election_strategy: 1 2024-06-03T08:36:32.323 INFO:teuthology.orchestra.run.smithi088.stderr:0: v1:172.21.15.88:6789/0 mon.noname-a 2024-06-03T08:36:32.323 INFO:teuthology.orchestra.run.smithi088.stderr:1: v1:172.21.15.192:6789/0 mon.noname-b 2024-06-03T08:36:32.323 INFO:teuthology.orchestra.run.smithi088.stderr:2: v1:172.21.15.88:6790/0 mon.noname-c 2024-06-03T08:36:32.323 INFO:teuthology.orchestra.run.smithi088.stderr: 2024-06-03T08:36:32.323 INFO:teuthology.orchestra.run.smithi088.stderr:2024-06-03T08:36:32.314+0000 7fb10c947640 1 librados: starting msgr at 2024-06-03T08:36:32.323 INFO:teuthology.orchestra.run.smithi088.stderr:2024-06-03T08:36:32.314+0000 7fb10c947640 1 librados: starting objecter 2024-06-03T08:36:32.324 INFO:teuthology.orchestra.run.smithi088.stderr:2024-06-03T08:36:32.314+0000 7fb10c947640 1 Processor -- start 2024-06-03T08:36:32.324 INFO:teuthology.orchestra.run.smithi088.stderr:2024-06-03T08:36:32.314+0000 7fb10c947640 1 -- start start 2024-06-03T08:36:32.324 INFO:teuthology.orchestra.run.smithi088.stderr:2024-06-03T08:36:32.314+0000 7fb10c947640 1 librados: setting wanted keys 2024-06-03T08:36:32.324 INFO:teuthology.orchestra.run.smithi088.stderr:2024-06-03T08:36:32.314+0000 7fb10c947640 1 librados: calling monclient init 2024-06-03T08:36:32.324 INFO:teuthology.orchestra.run.smithi088.stderr:2024-06-03T08:36:32.314+0000 7fb10c947640 10 monclient: init 2024-06-03T08:36:32.324 INFO:teuthology.orchestra.run.smithi088.stderr:2024-06-03T08:36:32.315+0000 7fb10c947640 10 monclient: _reopen_session rank -1 2024-06-03T08:36:32.324 INFO:teuthology.orchestra.run.smithi088.stderr:2024-06-03T08:36:32.315+0000 7fb10c947640 10 monclient: _add_conns ranks=[2,1,0] 2024-06-03T08:36:32.324 INFO:teuthology.orchestra.run.smithi088.stderr:2024-06-03T08:36:32.315+0000 7fb10c947640 10 monclient(hunting): picked mon.noname-c con 0x7fb1080a8960 addr v1:172.21.15.88:6790/0 2024-06-03T08:36:32.324 INFO:teuthology.orchestra.run.smithi088.stderr:2024-06-03T08:36:32.315+0000 7fb10c947640 10 monclient(hunting): picked mon.noname-b con 0x7fb10813eec0 addr v1:172.21.15.192:6789/0 2024-06-03T08:36:32.324 INFO:teuthology.orchestra.run.smithi088.stderr:2024-06-03T08:36:32.315+0000 7fd3367fc640 1 -- 172.21.15.88:0/3500826820 <== mon.0 v1:172.21.15.88:6789/0 5 ==== config(0 keys) v1 ==== 4+0+0 (unknown 0 0 0) 0x7fd330003610 con 0x7fd338006410 2024-06-03T08:36:32.325 INFO:teuthology.orchestra.run.smithi088.stderr:2024-06-03T08:36:32.315+0000 7fd3367fc640 10 monclient: handle_config config(0 keys) v1 2024-06-03T08:36:32.325 INFO:teuthology.orchestra.run.smithi088.stderr:2024-06-03T08:36:32.315+0000 7fd3367fc640 1 -- 172.21.15.88:0/3500826820 <== mon.0 v1:172.21.15.88:6789/0 6 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 2234374065 0 0) 0x7fd3300039a0 con 0x7fd338006410 2024-06-03T08:36:32.325 INFO:teuthology.orchestra.run.smithi088.stderr:2024-06-03T08:36:32.315+0000 7fd3367fc640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-06-03T08:36:32.325 INFO:teuthology.orchestra.run.smithi088.stderr:2024-06-03T08:36:32.315+0000 7fd3367fc640 10 monclient: got monmap 1 from mon.a (according to old e1) 2024-06-03T08:36:32.325 INFO:teuthology.orchestra.run.smithi088.stderr:2024-06-03T08:36:32.315+0000 7fd3367fc640 10 monclient: dump: 2024-06-03T08:36:32.325 INFO:teuthology.orchestra.run.smithi088.stderr:epoch 1 2024-06-03T08:36:32.325 INFO:teuthology.orchestra.run.smithi088.stderr:fsid ec3c3fdd-3ac6-4d7d-9412-683130ed8b04 2024-06-03T08:36:32.325 INFO:teuthology.orchestra.run.smithi088.stderr:last_changed 2024-06-03T08:34:31.885125+0000 2024-06-03T08:36:32.325 INFO:teuthology.orchestra.run.smithi088.stderr:created 2024-06-03T08:34:31.885125+0000 2024-06-03T08:36:32.325 INFO:teuthology.orchestra.run.smithi088.stderr:min_mon_release 18 (reef) 2024-06-03T08:36:32.325 INFO:teuthology.orchestra.run.smithi088.stderr:election_strategy: 3 2024-06-03T08:36:32.325 INFO:teuthology.orchestra.run.smithi088.stderr:0: v1:172.21.15.88:6789/0 mon.a 2024-06-03T08:36:32.325 INFO:teuthology.orchestra.run.smithi088.stderr:1: v1:172.21.15.192:6789/0 mon.b 2024-06-03T08:36:32.326 INFO:teuthology.orchestra.run.smithi088.stderr:2: v1:172.21.15.88:6790/0 mon.c 2024-06-03T08:36:32.326 INFO:teuthology.orchestra.run.smithi088.stderr: 2024-06-03T08:36:32.326 INFO:teuthology.orchestra.run.smithi088.stderr:2024-06-03T08:36:32.315+0000 7fb10c947640 10 monclient(hunting): picked mon.noname-a con 0x7fb1081450f0 addr v1:172.21.15.88:6789/0 2024-06-03T08:36:32.326 INFO:teuthology.orchestra.run.smithi088.stderr:2024-06-03T08:36:32.315+0000 7fb10c947640 1 -- --> v1:172.21.15.88:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fb1080a9660 con 0x7fb1081450f0 2024-06-03T08:36:32.326 INFO:teuthology.orchestra.run.smithi088.stderr:2024-06-03T08:36:32.315+0000 7fb10c947640 1 -- --> v1:172.21.15.192:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fb108144200 con 0x7fb10813eec0 2024-06-03T08:36:32.326 INFO:teuthology.orchestra.run.smithi088.stderr:2024-06-03T08:36:32.315+0000 7fb10c947640 1 -- --> v1:172.21.15.88:6790/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fb108140520 con 0x7fb1080a8960 2024-06-03T08:36:32.326 INFO:teuthology.orchestra.run.smithi088.stderr:2024-06-03T08:36:32.315+0000 7fb10c947640 10 monclient(hunting): _renew_subs 2024-06-03T08:36:32.326 INFO:teuthology.orchestra.run.smithi088.stderr:2024-06-03T08:36:32.315+0000 7fb10c947640 10 monclient(hunting): authenticate will time out at 1248.125191s 2024-06-03T08:36:32.327 INFO:teuthology.orchestra.run.smithi088.stderr:2024-06-03T08:36:32.315+0000 7fd337fff640 4 set_mon_vals no callback set 2024-06-03T08:36:32.327 INFO:teuthology.orchestra.run.smithi088.stderr:2024-06-03T08:36:32.315+0000 7fb0fffff640 1 --1- >> v1:172.21.15.192:6789/0 conn(0x7fb10813eec0 0x7fb1081d9ee0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=1).handle_server_banner_and_identify peer v1:172.21.15.192:6789/0 says I am v1:172.21.15.88:52486/0 (socket says 172.21.15.88:52486) 2024-06-03T08:36:32.327 INFO:teuthology.orchestra.run.smithi088.stderr:2024-06-03T08:36:32.315+0000 7fb10659c640 1 --1- >> v1:172.21.15.88:6790/0 conn(0x7fb1080a8960 0x7fb1081351d0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=1).handle_server_banner_and_identify peer v1:172.21.15.88:6790/0 says I am v1:172.21.15.88:54264/0 (socket says 172.21.15.88:54264) 2024-06-03T08:36:32.327 INFO:teuthology.orchestra.run.smithi088.stderr:2024-06-03T08:36:32.315+0000 7fb0fffff640 1 -- 172.21.15.88:0/4168539993 learned_addr learned my addr 172.21.15.88:0/4168539993 (peer_addr_for_me v1:172.21.15.88:0/0) 2024-06-03T08:36:32.328 INFO:teuthology.orchestra.run.smithi088.stderr:2024-06-03T08:36:32.316+0000 7fb0feffd640 10 client.?.objecter ms_handle_connect 0x7fb10813eec0 2024-06-03T08:36:32.328 INFO:teuthology.orchestra.run.smithi088.stderr:2024-06-03T08:36:32.316+0000 7fb0feffd640 10 client.?.objecter resend_mon_ops 2024-06-03T08:36:32.328 INFO:teuthology.orchestra.run.smithi088.stderr:2024-06-03T08:36:32.316+0000 7fb0feffd640 10 client.?.objecter ms_handle_connect 0x7fb1080a8960 2024-06-03T08:36:32.328 INFO:teuthology.orchestra.run.smithi088.stderr:2024-06-03T08:36:32.316+0000 7fb0feffd640 10 client.?.objecter resend_mon_ops 2024-06-03T08:36:32.328 INFO:teuthology.orchestra.run.smithi088.stderr:2024-06-03T08:36:32.316+0000 7fd3367fc640 1 -- 172.21.15.88:0/3500826820 <== mon.0 v1:172.21.15.88:6789/0 7 ==== mgrmap(e 5) v1 ==== 89110+0+0 (unknown 873533973 0 0) 0x7fd330003b00 con 0x7fd338006410 2024-06-03T08:36:32.328 INFO:teuthology.orchestra.run.smithi088.stderr:2024-06-03T08:36:32.317+0000 7fd345554640 10 monclient: get_auth_request con 0x7fd32406d0d0 auth_method 0 2024-06-03T08:36:32.328 INFO:teuthology.orchestra.run.smithi088.stderr:2024-06-03T08:36:32.317+0000 7fd3367fc640 10 client.4332.objecter ms_handle_connect 0x7fd32406d0d0 2024-06-03T08:36:32.330 INFO:teuthology.orchestra.run.smithi088.stderr:2024-06-03T08:36:32.318+0000 7fb0feffd640 1 -- 172.21.15.88:0/4168539993 <== mon.1 v1:172.21.15.192:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 2381799098 0 0) 0x7fb108144200 con 0x7fb10813eec0 2024-06-03T08:36:32.330 INFO:teuthology.orchestra.run.smithi088.stderr:2024-06-03T08:36:32.318+0000 7fb0feffd640 10 monclient(hunting): _negotiate 2024-06-03T08:36:32.330 INFO:teuthology.orchestra.run.smithi088.stderr:2024-06-03T08:36:32.318+0000 7fb0feffd640 10 monclient(hunting): _init_auth method 2 2024-06-03T08:36:32.330 INFO:teuthology.orchestra.run.smithi088.stderr:2024-06-03T08:36:32.318+0000 7fb0feffd640 10 monclient(hunting): _init_auth creating new auth 2024-06-03T08:36:32.331 INFO:teuthology.orchestra.run.smithi088.stderr:2024-06-03T08:36:32.318+0000 7fb0feffd640 10 monclient(hunting): my global_id is 4339 2024-06-03T08:36:32.331 INFO:teuthology.orchestra.run.smithi088.stderr:2024-06-03T08:36:32.318+0000 7fb0feffd640 1 -- 172.21.15.88:0/4168539993 --> v1:172.21.15.192:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7fb0e8004eb0 con 0x7fb10813eec0 2024-06-03T08:36:32.331 INFO:teuthology.orchestra.run.smithi088.stderr:2024-06-03T08:36:32.318+0000 7fb0feffd640 1 -- 172.21.15.88:0/4168539993 <== mon.2 v1:172.21.15.88:6790/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 1494829918 0 0) 0x7fb108140520 con 0x7fb1080a8960 2024-06-03T08:36:32.331 INFO:teuthology.orchestra.run.smithi088.stderr:2024-06-03T08:36:32.318+0000 7fb0feffd640 10 monclient(hunting): _negotiate 2024-06-03T08:36:32.331 INFO:teuthology.orchestra.run.smithi088.stderr:2024-06-03T08:36:32.318+0000 7fb0feffd640 10 monclient(hunting): _init_auth method 2 2024-06-03T08:36:32.331 INFO:teuthology.orchestra.run.smithi088.stderr:2024-06-03T08:36:32.318+0000 7fb0feffd640 10 monclient(hunting): _init_auth creating new auth 2024-06-03T08:36:32.331 INFO:teuthology.orchestra.run.smithi088.stderr:2024-06-03T08:36:32.318+0000 7fb0feffd640 10 monclient(hunting): my global_id is 4346 2024-06-03T08:36:32.331 INFO:teuthology.orchestra.run.smithi088.stderr:2024-06-03T08:36:32.318+0000 7fb0feffd640 1 -- 172.21.15.88:0/4168539993 --> v1:172.21.15.88:6790/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7fb108144200 con 0x7fb1080a8960 2024-06-03T08:36:32.331 INFO:teuthology.orchestra.run.smithi088.stderr:2024-06-03T08:36:32.323+0000 7fb0feffd640 1 -- 172.21.15.88:0/4168539993 <== mon.1 v1:172.21.15.192:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 764+0+0 (unknown 4243261954 0 0) 0x7fb0e8004eb0 con 0x7fb10813eec0 2024-06-03T08:36:32.331 INFO:teuthology.orchestra.run.smithi088.stderr:2024-06-03T08:36:32.323+0000 7fb0feffd640 1 -- 172.21.15.88:0/4168539993 --> v1:172.21.15.192:6789/0 -- auth(proto 2 165 bytes epoch 0) v1 -- 0x7fb108140520 con 0x7fb10813eec0 2024-06-03T08:36:32.331 INFO:teuthology.orchestra.run.smithi088.stderr:2024-06-03T08:36:32.323+0000 7fb0feffd640 1 -- 172.21.15.88:0/4168539993 <== mon.2 v1:172.21.15.88:6790/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 764+0+0 (unknown 726720387 0 0) 0x7fb108144200 con 0x7fb1080a8960 2024-06-03T08:36:32.331 INFO:teuthology.orchestra.run.smithi088.stderr:2024-06-03T08:36:32.323+0000 7fb0feffd640 1 -- 172.21.15.88:0/4168539993 --> v1:172.21.15.88:6790/0 -- auth(proto 2 165 bytes epoch 0) v1 -- 0x7fb0e8004eb0 con 0x7fb1080a8960 2024-06-03T08:36:32.332 INFO:teuthology.orchestra.run.smithi088.stderr:2024-06-03T08:36:32.323+0000 7fb0feffd640 1 -- 172.21.15.88:0/4168539993 <== mon.1 v1:172.21.15.192:6789/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 2234374065 0 0) 0x7fb0f0001610 con 0x7fb10813eec0 2024-06-03T08:36:32.332 INFO:teuthology.orchestra.run.smithi088.stderr:2024-06-03T08:36:32.323+0000 7fb0feffd640 10 monclient(hunting): handle_monmap mon_map magic: 0 v1 2024-06-03T08:36:32.332 INFO:teuthology.orchestra.run.smithi088.stderr:2024-06-03T08:36:32.323+0000 7fb0feffd640 10 monclient(hunting): got monmap 1 from mon.noname-b (according to old e1) 2024-06-03T08:36:32.332 INFO:teuthology.orchestra.run.smithi088.stderr:2024-06-03T08:36:32.323+0000 7fb0feffd640 10 monclient(hunting): dump: 2024-06-03T08:36:32.332 INFO:teuthology.orchestra.run.smithi088.stderr:epoch 1 2024-06-03T08:36:32.332 INFO:teuthology.orchestra.run.smithi088.stderr:fsid ec3c3fdd-3ac6-4d7d-9412-683130ed8b04 2024-06-03T08:36:32.332 INFO:teuthology.orchestra.run.smithi088.stderr:last_changed 2024-06-03T08:34:31.885125+0000 2024-06-03T08:36:32.332 INFO:teuthology.orchestra.run.smithi088.stderr:created 2024-06-03T08:34:31.885125+0000 2024-06-03T08:36:32.332 INFO:teuthology.orchestra.run.smithi088.stderr:min_mon_release 18 (reef) 2024-06-03T08:36:32.332 INFO:teuthology.orchestra.run.smithi088.stderr:election_strategy: 3 2024-06-03T08:36:32.332 INFO:teuthology.orchestra.run.smithi088.stderr:0: v1:172.21.15.88:6789/0 mon.a 2024-06-03T08:36:32.332 INFO:teuthology.orchestra.run.smithi088.stderr:1: v1:172.21.15.192:6789/0 mon.b 2024-06-03T08:36:32.333 INFO:teuthology.orchestra.run.smithi088.stderr:2: v1:172.21.15.88:6790/0 mon.c 2024-06-03T08:36:32.333 INFO:teuthology.orchestra.run.smithi088.stderr: 2024-06-03T08:36:32.333 INFO:teuthology.orchestra.run.smithi088.stderr:2024-06-03T08:36:32.323+0000 7fb0feffd640 10 monclient(hunting): _finish_auth 0 2024-06-03T08:36:32.333 INFO:teuthology.orchestra.run.smithi088.stderr:2024-06-03T08:36:32.323+0000 7fb0feffd640 1 -- 172.21.15.88:0/4168539993 <== mon.2 v1:172.21.15.88:6790/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 2234374065 0 0) 0x7fb0f80017a0 con 0x7fb1080a8960 2024-06-03T08:36:32.333 INFO:teuthology.orchestra.run.smithi088.stderr:2024-06-03T08:36:32.323+0000 7fb0feffd640 10 monclient(hunting): handle_monmap mon_map magic: 0 v1 2024-06-03T08:36:32.333 INFO:teuthology.orchestra.run.smithi088.stderr:2024-06-03T08:36:32.323+0000 7fb0feffd640 10 monclient(hunting): got monmap 1 from mon.c (according to old e1) 2024-06-03T08:36:32.333 INFO:teuthology.orchestra.run.smithi088.stderr:2024-06-03T08:36:32.323+0000 7fb0feffd640 10 monclient(hunting): dump: 2024-06-03T08:36:32.333 INFO:teuthology.orchestra.run.smithi088.stderr:epoch 1 2024-06-03T08:36:32.333 INFO:teuthology.orchestra.run.smithi088.stderr:fsid ec3c3fdd-3ac6-4d7d-9412-683130ed8b04 2024-06-03T08:36:32.333 INFO:teuthology.orchestra.run.smithi088.stderr:last_changed 2024-06-03T08:34:31.885125+0000 2024-06-03T08:36:32.333 INFO:teuthology.orchestra.run.smithi088.stderr:created 2024-06-03T08:34:31.885125+0000 2024-06-03T08:36:32.333 INFO:teuthology.orchestra.run.smithi088.stderr:min_mon_release 18 (reef) 2024-06-03T08:36:32.333 INFO:teuthology.orchestra.run.smithi088.stderr:election_strategy: 3 2024-06-03T08:36:32.334 INFO:teuthology.orchestra.run.smithi088.stderr:0: v1:172.21.15.88:6789/0 mon.a 2024-06-03T08:36:32.334 INFO:teuthology.orchestra.run.smithi088.stderr:1: v1:172.21.15.192:6789/0 mon.b 2024-06-03T08:36:32.334 INFO:teuthology.orchestra.run.smithi088.stderr:2: v1:172.21.15.88:6790/0 mon.c 2024-06-03T08:36:32.334 INFO:teuthology.orchestra.run.smithi088.stderr: 2024-06-03T08:36:32.334 INFO:teuthology.orchestra.run.smithi088.stderr:2024-06-03T08:36:32.326+0000 7fb0feffd640 1 -- 172.21.15.88:0/4168539993 <== mon.1 v1:172.21.15.192:6789/0 4 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 580+0+0 (unknown 2982079488 0 0) 0x7fb108140520 con 0x7fb10813eec0 2024-06-03T08:36:32.334 INFO:teuthology.orchestra.run.smithi088.stderr:2024-06-03T08:36:32.326+0000 7fb0feffd640 1 -- 172.21.15.88:0/4168539993 >> v1:172.21.15.88:6790/0 conn(0x7fb1080a8960 legacy=0x7fb1081351d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-06-03T08:36:32.334 INFO:teuthology.orchestra.run.smithi088.stderr:2024-06-03T08:36:32.326+0000 7fb0feffd640 1 -- 172.21.15.88:0/4168539993 >> v1:172.21.15.88:6789/0 conn(0x7fb1081450f0 legacy=0x7fb1081da5f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-06-03T08:36:32.334 INFO:teuthology.orchestra.run.smithi088.stderr:2024-06-03T08:36:32.326+0000 7fb0feffd640 10 monclient: _finish_hunting 0 2024-06-03T08:36:32.334 INFO:teuthology.orchestra.run.smithi088.stderr:2024-06-03T08:36:32.326+0000 7fb0feffd640 1 monclient: found mon.b 2024-06-03T08:36:32.334 INFO:teuthology.orchestra.run.smithi088.stderr:2024-06-03T08:36:32.326+0000 7fb0feffd640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-06-03T08:36:32.334 INFO:teuthology.orchestra.run.smithi088.stderr:2024-06-03T08:36:32.326+0000 7fb0feffd640 10 monclient: _send_mon_message to mon.b at v1:172.21.15.192:6789/0 2024-06-03T08:36:32.334 INFO:teuthology.orchestra.run.smithi088.stderr:2024-06-03T08:36:32.326+0000 7fb0feffd640 1 -- 172.21.15.88:0/4168539993 --> v1:172.21.15.192:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fb1080b09e0 con 0x7fb10813eec0 2024-06-03T08:36:32.334 INFO:teuthology.orchestra.run.smithi088.stderr:2024-06-03T08:36:32.326+0000 7fb0feffd640 10 monclient: _finish_auth 0 2024-06-03T08:36:32.335 INFO:teuthology.orchestra.run.smithi088.stderr:2024-06-03T08:36:32.326+0000 7fb0feffd640 10 monclient: _check_auth_tickets 2024-06-03T08:36:32.335 INFO:teuthology.orchestra.run.smithi088.stderr:2024-06-03T08:36:32.326+0000 7fb0feffd640 20 monclient: _check_auth_rotating not needed by client.admin 2024-06-03T08:36:32.335 INFO:teuthology.orchestra.run.smithi088.stderr:2024-06-03T08:36:32.326+0000 7fb10c947640 5 monclient: authenticate success, global_id 4339 2024-06-03T08:36:32.335 INFO:teuthology.orchestra.run.smithi088.stderr:2024-06-03T08:36:32.326+0000 7fb10c947640 10 monclient: _renew_subs 2024-06-03T08:36:32.335 INFO:teuthology.orchestra.run.smithi088.stderr:2024-06-03T08:36:32.326+0000 7fb10c947640 10 monclient: _send_mon_message to mon.b at v1:172.21.15.192:6789/0 2024-06-03T08:36:32.335 INFO:teuthology.orchestra.run.smithi088.stderr:2024-06-03T08:36:32.326+0000 7fb10c947640 1 -- 172.21.15.88:0/4168539993 --> v1:172.21.15.192:6789/0 -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fb10813b500 con 0x7fb10813eec0 2024-06-03T08:36:32.335 INFO:teuthology.orchestra.run.smithi088.stderr:2024-06-03T08:36:32.326+0000 7fb10c947640 10 client.4339.objecter _maybe_request_map subscribing (onetime) to next osd map 2024-06-03T08:36:32.335 INFO:teuthology.orchestra.run.smithi088.stderr:2024-06-03T08:36:32.326+0000 7fb10c947640 10 monclient: _renew_subs 2024-06-03T08:36:32.335 INFO:teuthology.orchestra.run.smithi088.stderr:2024-06-03T08:36:32.326+0000 7fb10c947640 10 monclient: _send_mon_message to mon.b at v1:172.21.15.192:6789/0 2024-06-03T08:36:32.335 INFO:teuthology.orchestra.run.smithi088.stderr:2024-06-03T08:36:32.326+0000 7fb10c947640 1 -- 172.21.15.88:0/4168539993 --> v1:172.21.15.192:6789/0 -- mon_subscribe({osdmap=0}) v3 -- 0x7fb108143be0 con 0x7fb10813eec0 2024-06-03T08:36:32.335 INFO:teuthology.orchestra.run.smithi088.stderr:2024-06-03T08:36:32.326+0000 7fb10c947640 1 librados: init done 2024-06-03T08:36:32.335 INFO:teuthology.orchestra.run.smithi088.stderr:2024-06-03T08:36:32.327+0000 7fb10c947640 10 start_mon_command cmd=[{"prefix": "get_command_descriptions"}] 2024-06-03T08:36:32.335 INFO:teuthology.orchestra.run.smithi088.stderr:2024-06-03T08:36:32.327+0000 7fb10c947640 10 monclient: _send_command 1 [{"prefix": "get_command_descriptions"}] 2024-06-03T08:36:32.335 INFO:teuthology.orchestra.run.smithi088.stderr:2024-06-03T08:36:32.327+0000 7fb10c947640 10 monclient: _send_mon_message to mon.b at v1:172.21.15.192:6789/0 2024-06-03T08:36:32.336 INFO:teuthology.orchestra.run.smithi088.stderr:2024-06-03T08:36:32.327+0000 7fb10c947640 1 -- 172.21.15.88:0/4168539993 --> v1:172.21.15.192:6789/0 -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fb1080a9e60 con 0x7fb10813eec0 2024-06-03T08:36:32.336 INFO:teuthology.orchestra.run.smithi088.stderr:2024-06-03T08:36:32.330+0000 7fd649ffb640 1 -- 172.21.15.88:0/2839181686 <== mon.0 v1:172.21.15.88:6789/0 5 ==== config(0 keys) v1 ==== 4+0+0 (unknown 0 0 0) 0x7fd6440035f0 con 0x7fd65406dc10 2024-06-03T08:36:32.336 INFO:teuthology.orchestra.run.smithi088.stderr:2024-06-03T08:36:32.330+0000 7fd649ffb640 10 monclient: handle_config config(0 keys) v1 2024-06-03T08:36:32.336 INFO:teuthology.orchestra.run.smithi088.stderr:2024-06-03T08:36:32.330+0000 7fd649ffb640 1 -- 172.21.15.88:0/2839181686 <== mon.0 v1:172.21.15.88:6789/0 6 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 2234374065 0 0) 0x7fd644003900 con 0x7fd65406dc10 2024-06-03T08:36:32.337 INFO:teuthology.orchestra.run.smithi088.stderr:2024-06-03T08:36:32.330+0000 7fd649ffb640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-06-03T08:36:32.337 INFO:teuthology.orchestra.run.smithi088.stderr:2024-06-03T08:36:32.330+0000 7fd649ffb640 10 monclient: got monmap 1 from mon.a (according to old e1) 2024-06-03T08:36:32.337 INFO:teuthology.orchestra.run.smithi088.stderr:2024-06-03T08:36:32.330+0000 7fd649ffb640 10 monclient: dump: 2024-06-03T08:36:32.337 INFO:teuthology.orchestra.run.smithi088.stderr:epoch 1 2024-06-03T08:36:32.337 INFO:teuthology.orchestra.run.smithi088.stderr:fsid ec3c3fdd-3ac6-4d7d-9412-683130ed8b04 2024-06-03T08:36:32.337 INFO:teuthology.orchestra.run.smithi088.stderr:last_changed 2024-06-03T08:34:31.885125+0000 2024-06-03T08:36:32.337 INFO:teuthology.orchestra.run.smithi088.stderr:created 2024-06-03T08:34:31.885125+0000 2024-06-03T08:36:32.337 INFO:teuthology.orchestra.run.smithi088.stderr:min_mon_release 18 (reef) 2024-06-03T08:36:32.337 INFO:teuthology.orchestra.run.smithi088.stderr:election_strategy: 3 2024-06-03T08:36:32.337 INFO:teuthology.orchestra.run.smithi088.stderr:0: v1:172.21.15.88:6789/0 mon.a 2024-06-03T08:36:32.337 INFO:teuthology.orchestra.run.smithi088.stderr:1: v1:172.21.15.192:6789/0 mon.b 2024-06-03T08:36:32.337 INFO:teuthology.orchestra.run.smithi088.stderr:2: v1:172.21.15.88:6790/0 mon.c 2024-06-03T08:36:32.337 INFO:teuthology.orchestra.run.smithi088.stderr: 2024-06-03T08:36:32.338 INFO:teuthology.orchestra.run.smithi088.stderr:2024-06-03T08:36:32.330+0000 7fd64b7fe640 4 set_mon_vals no callback set 2024-06-03T08:36:32.338 INFO:teuthology.orchestra.run.smithi088.stderr:2024-06-03T08:36:32.330+0000 7fd649ffb640 1 -- 172.21.15.88:0/2839181686 <== mon.0 v1:172.21.15.88:6789/0 7 ==== mgrmap(e 5) v1 ==== 89110+0+0 (unknown 873533973 0 0) 0x7fd644003a60 con 0x7fd65406dc10 2024-06-03T08:36:32.338 INFO:teuthology.orchestra.run.smithi088.stderr:2024-06-03T08:36:32.331+0000 7fd658ecf640 10 monclient: get_auth_request con 0x7fd63806d340 auth_method 0 2024-06-03T08:36:32.338 INFO:teuthology.orchestra.run.smithi088.stderr:2024-06-03T08:36:32.331+0000 7fd649ffb640 10 client.4320.objecter ms_handle_connect 0x7fd63806d340 2024-06-03T08:36:32.342 INFO:teuthology.orchestra.run.smithi088.stderr:2024-06-03T08:36:32.341+0000 7fd649ffb640 1 -- 172.21.15.88:0/2839181686 <== mon.0 v1:172.21.15.88:6789/0 8 ==== osd_map(26..26 src has 1..26) v4 ==== 4090+0+0 (unknown 3637450554 0 0) 0x7fd6440540d0 con 0x7fd65406dc10 2024-06-03T08:36:32.342 INFO:teuthology.orchestra.run.smithi088.stderr:2024-06-03T08:36:32.341+0000 7fd649ffb640 10 client.4320.objecter ms_dispatch 0x7fd654000b90 osd_map(26..26 src has 1..26) v4 2024-06-03T08:36:32.342 INFO:teuthology.orchestra.run.smithi088.stderr:2024-06-03T08:36:32.341+0000 7fd649ffb640 3 client.4320.objecter handle_osd_map got epochs [26,26] > 0 2024-06-03T08:36:32.342 INFO:teuthology.orchestra.run.smithi088.stderr:2024-06-03T08:36:32.341+0000 7fd649ffb640 3 client.4320.objecter handle_osd_map decoding full epoch 26 2024-06-03T08:36:32.342 INFO:teuthology.orchestra.run.smithi088.stderr:2024-06-03T08:36:32.341+0000 7fd649ffb640 10 client.4320.objecter checking command 1 2024-06-03T08:36:32.342 INFO:teuthology.orchestra.run.smithi088.stderr:2024-06-03T08:36:32.341+0000 7fd649ffb640 20 client.4320.objecter _get_session s=0x7fd638073ca0 osd=0 3 2024-06-03T08:36:32.342 INFO:teuthology.orchestra.run.smithi088.stderr:2024-06-03T08:36:32.341+0000 7fd649ffb640 20 client.4320.objecter put_session s=0x7fd638073ca0 osd=0 3 2024-06-03T08:36:32.343 INFO:teuthology.orchestra.run.smithi088.stderr:2024-06-03T08:36:32.341+0000 7fd649ffb640 15 client.4320.objecter _session_command_op_remove -1 1 2024-06-03T08:36:32.343 INFO:teuthology.orchestra.run.smithi088.stderr:2024-06-03T08:36:32.341+0000 7fd649ffb640 20 client.4320.objecter _get_session s=0x7fd638073ca0 osd=0 3 2024-06-03T08:36:32.343 INFO:teuthology.orchestra.run.smithi088.stderr:2024-06-03T08:36:32.341+0000 7fd649ffb640 20 client.4320.objecter get_session s=0x7fd638073ca0 osd=0 3 2024-06-03T08:36:32.343 INFO:teuthology.orchestra.run.smithi088.stderr:2024-06-03T08:36:32.341+0000 7fd649ffb640 15 client.4320.objecter _session_command_op_assign 0 1 2024-06-03T08:36:32.343 INFO:teuthology.orchestra.run.smithi088.stderr:2024-06-03T08:36:32.341+0000 7fd649ffb640 20 client.4320.objecter put_session s=0x7fd638073ca0 osd=0 4 2024-06-03T08:36:32.343 INFO:teuthology.orchestra.run.smithi088.stderr:2024-06-03T08:36:32.341+0000 7fd649ffb640 10 client.4320.objecter _send_command 1 2024-06-03T08:36:32.343 INFO:teuthology.orchestra.run.smithi088.stderr:2024-06-03T08:36:32.341+0000 7fd649ffb640 1 -- 172.21.15.88:0/2839181686 --> v1:172.21.15.88:6808/35066 -- command(tid 1: {"prefix": "get_command_descriptions"}) v1 -- 0x7fd638077f20 con 0x7fd638074350 2024-06-03T08:36:32.343 INFO:teuthology.orchestra.run.smithi088.stderr:2024-06-03T08:36:32.341+0000 7fd649ffb640 20 client.4320.objecter dump_active .. 0 homeless 2024-06-03T08:36:32.343 INFO:teuthology.orchestra.run.smithi088.stderr:2024-06-03T08:36:32.341+0000 7fd649ffb640 1 -- 172.21.15.88:0/2839181686 <== mon.0 v1:172.21.15.88:6789/0 9 ==== mon_get_version_reply(handle=1 version=26) v2 ==== 24+0+0 (unknown 2370098492 0 0) 0x7fd64404fc80 con 0x7fd65406dc10 2024-06-03T08:36:32.343 INFO:teuthology.orchestra.run.smithi088.stderr:2024-06-03T08:36:32.341+0000 7fd649ffb640 10 monclient: handle_get_version_reply finishing 1 version 26 2024-06-03T08:36:32.344 INFO:teuthology.orchestra.run.smithi088.stderr:2024-06-03T08:36:32.342+0000 7fd3367fc640 1 -- 172.21.15.88:0/3500826820 <== mon.0 v1:172.21.15.88:6789/0 8 ==== osd_map(26..26 src has 1..26) v4 ==== 4090+0+0 (unknown 3637450554 0 0) 0x7fd330052f00 con 0x7fd338006410 2024-06-03T08:36:32.344 INFO:teuthology.orchestra.run.smithi088.stderr:2024-06-03T08:36:32.342+0000 7fd3367fc640 10 client.4332.objecter ms_dispatch 0x7fd340000b90 osd_map(26..26 src has 1..26) v4 2024-06-03T08:36:32.344 INFO:teuthology.orchestra.run.smithi088.stderr:2024-06-03T08:36:32.342+0000 7fd3367fc640 3 client.4332.objecter handle_osd_map got epochs [26,26] > 0 2024-06-03T08:36:32.344 INFO:teuthology.orchestra.run.smithi088.stderr:2024-06-03T08:36:32.342+0000 7fd3367fc640 3 client.4332.objecter handle_osd_map decoding full epoch 26 2024-06-03T08:36:32.344 INFO:teuthology.orchestra.run.smithi088.stderr:2024-06-03T08:36:32.342+0000 7fd3367fc640 10 client.4332.objecter checking command 1 2024-06-03T08:36:32.344 INFO:teuthology.orchestra.run.smithi088.stderr:2024-06-03T08:36:32.342+0000 7fd3367fc640 20 client.4332.objecter _get_session s=0x7fd324073ae0 osd=4 3 2024-06-03T08:36:32.344 INFO:teuthology.orchestra.run.smithi088.stderr:2024-06-03T08:36:32.342+0000 7fd3367fc640 20 client.4332.objecter put_session s=0x7fd324073ae0 osd=4 3 2024-06-03T08:36:32.344 INFO:teuthology.orchestra.run.smithi088.stderr:2024-06-03T08:36:32.342+0000 7fd3367fc640 15 client.4332.objecter _session_command_op_remove -1 1 2024-06-03T08:36:32.344 INFO:teuthology.orchestra.run.smithi088.stderr:2024-06-03T08:36:32.342+0000 7fd3367fc640 20 client.4332.objecter _get_session s=0x7fd324073ae0 osd=4 3 2024-06-03T08:36:32.345 INFO:teuthology.orchestra.run.smithi088.stderr:2024-06-03T08:36:32.342+0000 7fd3367fc640 20 client.4332.objecter get_session s=0x7fd324073ae0 osd=4 3 2024-06-03T08:36:32.345 INFO:teuthology.orchestra.run.smithi088.stderr:2024-06-03T08:36:32.342+0000 7fd3367fc640 15 client.4332.objecter _session_command_op_assign 4 1 2024-06-03T08:36:32.345 INFO:teuthology.orchestra.run.smithi088.stderr:2024-06-03T08:36:32.342+0000 7fd3367fc640 20 client.4332.objecter put_session s=0x7fd324073ae0 osd=4 4 2024-06-03T08:36:32.345 INFO:teuthology.orchestra.run.smithi088.stderr:2024-06-03T08:36:32.342+0000 7fd3367fc640 10 client.4332.objecter _send_command 1 2024-06-03T08:36:32.345 INFO:teuthology.orchestra.run.smithi088.stderr:2024-06-03T08:36:32.342+0000 7fd3367fc640 1 -- 172.21.15.88:0/3500826820 --> v1:172.21.15.192:6809/34785 -- command(tid 1: {"prefix": "get_command_descriptions"}) v1 -- 0x7fd324077d60 con 0x7fd324074190 2024-06-03T08:36:32.345 INFO:teuthology.orchestra.run.smithi088.stderr:2024-06-03T08:36:32.342+0000 7fd3367fc640 20 client.4332.objecter dump_active .. 0 homeless 2024-06-03T08:36:32.345 INFO:teuthology.orchestra.run.smithi088.stderr:2024-06-03T08:36:32.342+0000 7fd3367fc640 1 -- 172.21.15.88:0/3500826820 <== mon.0 v1:172.21.15.88:6789/0 9 ==== mon_get_version_reply(handle=1 version=26) v2 ==== 24+0+0 (unknown 2370098492 0 0) 0x7fd33004eab0 con 0x7fd338006410 2024-06-03T08:36:32.345 INFO:teuthology.orchestra.run.smithi088.stderr:2024-06-03T08:36:32.342+0000 7fd3367fc640 10 monclient: handle_get_version_reply finishing 1 version 26 2024-06-03T08:36:32.345 INFO:teuthology.orchestra.run.smithi088.stderr:2024-06-03T08:36:32.342+0000 7fd659ed1640 10 monclient: get_auth_request con 0x7fd638074350 auth_method 0 2024-06-03T08:36:32.346 INFO:teuthology.orchestra.run.smithi088.stderr:2024-06-03T08:36:32.343+0000 7fd346556640 10 monclient: get_auth_request con 0x7fd324074190 auth_method 0 2024-06-03T08:36:32.346 INFO:teuthology.orchestra.run.smithi088.stderr:2024-06-03T08:36:32.344+0000 7fd649ffb640 10 client.4320.objecter ms_handle_connect 0x7fd638074350 2024-06-03T08:36:32.346 INFO:teuthology.orchestra.run.smithi088.stderr:2024-06-03T08:36:32.346+0000 7fd3367fc640 10 client.4332.objecter ms_handle_connect 0x7fd324074190 2024-06-03T08:36:32.352 INFO:teuthology.orchestra.run.smithi088.stderr:2024-06-03T08:36:32.352+0000 7f4c1bfff640 1 -- 172.21.15.88:0/2099487910 <== mon.0 v1:172.21.15.88:6789/0 5 ==== config(0 keys) v1 ==== 4+0+0 (unknown 0 0 0) 0x7f4c280020d0 con 0x7f4c3013ee50 2024-06-03T08:36:32.352 INFO:teuthology.orchestra.run.smithi088.stderr:2024-06-03T08:36:32.352+0000 7f4c1bfff640 10 monclient: handle_config config(0 keys) v1 2024-06-03T08:36:32.353 INFO:teuthology.orchestra.run.smithi088.stderr:2024-06-03T08:36:32.352+0000 7f4c1bfff640 1 -- 172.21.15.88:0/2099487910 <== mon.0 v1:172.21.15.88:6789/0 6 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 2234374065 0 0) 0x7f4c28003b60 con 0x7f4c3013ee50 2024-06-03T08:36:32.353 INFO:teuthology.orchestra.run.smithi088.stderr:2024-06-03T08:36:32.352+0000 7f4c1bfff640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-06-03T08:36:32.353 INFO:teuthology.orchestra.run.smithi088.stderr:2024-06-03T08:36:32.352+0000 7f4c1bfff640 10 monclient: got monmap 1 from mon.a (according to old e1) 2024-06-03T08:36:32.353 INFO:teuthology.orchestra.run.smithi088.stderr:2024-06-03T08:36:32.352+0000 7f4c1bfff640 10 monclient: dump: 2024-06-03T08:36:32.353 INFO:teuthology.orchestra.run.smithi088.stderr:epoch 1 2024-06-03T08:36:32.353 INFO:teuthology.orchestra.run.smithi088.stderr:fsid ec3c3fdd-3ac6-4d7d-9412-683130ed8b04 2024-06-03T08:36:32.353 INFO:teuthology.orchestra.run.smithi088.stderr:last_changed 2024-06-03T08:34:31.885125+0000 2024-06-03T08:36:32.353 INFO:teuthology.orchestra.run.smithi088.stderr:created 2024-06-03T08:34:31.885125+0000 2024-06-03T08:36:32.353 INFO:teuthology.orchestra.run.smithi088.stderr:min_mon_release 18 (reef) 2024-06-03T08:36:32.353 INFO:teuthology.orchestra.run.smithi088.stderr:election_strategy: 3 2024-06-03T08:36:32.353 INFO:teuthology.orchestra.run.smithi088.stderr:0: v1:172.21.15.88:6789/0 mon.a 2024-06-03T08:36:32.353 INFO:teuthology.orchestra.run.smithi088.stderr:1: v1:172.21.15.192:6789/0 mon.b 2024-06-03T08:36:32.353 INFO:teuthology.orchestra.run.smithi088.stderr:2: v1:172.21.15.88:6790/0 mon.c 2024-06-03T08:36:32.354 INFO:teuthology.orchestra.run.smithi088.stderr: 2024-06-03T08:36:32.354 INFO:teuthology.orchestra.run.smithi088.stderr:2024-06-03T08:36:32.352+0000 7f4c2dffb640 4 set_mon_vals no callback set 2024-06-03T08:36:32.354 INFO:teuthology.orchestra.run.smithi088.stderr:2024-06-03T08:36:32.353+0000 7f4c1bfff640 1 -- 172.21.15.88:0/2099487910 <== mon.0 v1:172.21.15.88:6789/0 7 ==== mgrmap(e 5) v1 ==== 89110+0+0 (unknown 873533973 0 0) 0x7f4c28003cc0 con 0x7f4c3013ee50 2024-06-03T08:36:32.354 INFO:teuthology.orchestra.run.smithi088.stderr:2024-06-03T08:36:32.354+0000 7f4c2f7fe640 10 monclient: get_auth_request con 0x7f4c1406d200 auth_method 0 2024-06-03T08:36:32.355 INFO:teuthology.orchestra.run.smithi088.stderr:2024-06-03T08:36:32.354+0000 7f4c1bfff640 10 client.4323.objecter ms_handle_connect 0x7f4c1406d200 2024-06-03T08:36:32.358 INFO:teuthology.orchestra.run.smithi088.stderr:2024-06-03T08:36:32.357+0000 7f4c1bfff640 1 -- 172.21.15.88:0/2099487910 <== mon.0 v1:172.21.15.88:6789/0 8 ==== osd_map(26..26 src has 1..26) v4 ==== 4090+0+0 (unknown 3637450554 0 0) 0x7f4c28053110 con 0x7f4c3013ee50 2024-06-03T08:36:32.358 INFO:teuthology.orchestra.run.smithi088.stderr:2024-06-03T08:36:32.357+0000 7f4c1bfff640 10 client.4323.objecter ms_dispatch 0x7f4c30000b90 osd_map(26..26 src has 1..26) v4 2024-06-03T08:36:32.358 INFO:teuthology.orchestra.run.smithi088.stderr:2024-06-03T08:36:32.357+0000 7f4c1bfff640 3 client.4323.objecter handle_osd_map got epochs [26,26] > 0 2024-06-03T08:36:32.358 INFO:teuthology.orchestra.run.smithi088.stderr:2024-06-03T08:36:32.357+0000 7f4c1bfff640 3 client.4323.objecter handle_osd_map decoding full epoch 26 2024-06-03T08:36:32.358 INFO:teuthology.orchestra.run.smithi088.stderr:2024-06-03T08:36:32.357+0000 7f4c1bfff640 10 client.4323.objecter checking command 1 2024-06-03T08:36:32.358 INFO:teuthology.orchestra.run.smithi088.stderr:2024-06-03T08:36:32.357+0000 7f4c1bfff640 20 client.4323.objecter _get_session s=0x7f4c14073b80 osd=1 3 2024-06-03T08:36:32.358 INFO:teuthology.orchestra.run.smithi088.stderr:2024-06-03T08:36:32.357+0000 7f4c1bfff640 20 client.4323.objecter put_session s=0x7f4c14073b80 osd=1 3 2024-06-03T08:36:32.358 INFO:teuthology.orchestra.run.smithi088.stderr:2024-06-03T08:36:32.357+0000 7f4c1bfff640 15 client.4323.objecter _session_command_op_remove -1 1 2024-06-03T08:36:32.358 INFO:teuthology.orchestra.run.smithi088.stderr:2024-06-03T08:36:32.357+0000 7f4c1bfff640 20 client.4323.objecter _get_session s=0x7f4c14073b80 osd=1 3 2024-06-03T08:36:32.359 INFO:teuthology.orchestra.run.smithi088.stderr:2024-06-03T08:36:32.357+0000 7f4c1bfff640 20 client.4323.objecter get_session s=0x7f4c14073b80 osd=1 3 2024-06-03T08:36:32.359 INFO:teuthology.orchestra.run.smithi088.stderr:2024-06-03T08:36:32.357+0000 7f4c1bfff640 15 client.4323.objecter _session_command_op_assign 1 1 2024-06-03T08:36:32.359 INFO:teuthology.orchestra.run.smithi088.stderr:2024-06-03T08:36:32.357+0000 7f4c1bfff640 20 client.4323.objecter put_session s=0x7f4c14073b80 osd=1 4 2024-06-03T08:36:32.359 INFO:teuthology.orchestra.run.smithi088.stderr:2024-06-03T08:36:32.357+0000 7f4c1bfff640 10 client.4323.objecter _send_command 1 2024-06-03T08:36:32.359 INFO:teuthology.orchestra.run.smithi088.stderr:2024-06-03T08:36:32.357+0000 7f4c1bfff640 1 -- 172.21.15.88:0/2099487910 --> v1:172.21.15.88:6800/35067 -- command(tid 1: {"prefix": "get_command_descriptions"}) v1 -- 0x7f4c14077ea0 con 0x7f4c14074230 2024-06-03T08:36:32.359 INFO:teuthology.orchestra.run.smithi088.stderr:2024-06-03T08:36:32.357+0000 7f4c1bfff640 20 client.4323.objecter dump_active .. 0 homeless 2024-06-03T08:36:32.359 INFO:teuthology.orchestra.run.smithi088.stderr:2024-06-03T08:36:32.357+0000 7f4c1bfff640 1 -- 172.21.15.88:0/2099487910 <== mon.0 v1:172.21.15.88:6789/0 9 ==== mon_get_version_reply(handle=1 version=26) v2 ==== 24+0+0 (unknown 2370098492 0 0) 0x7f4c2804eaf0 con 0x7f4c3013ee50 2024-06-03T08:36:32.359 INFO:teuthology.orchestra.run.smithi088.stderr:2024-06-03T08:36:32.357+0000 7f4c1bfff640 10 monclient: handle_get_version_reply finishing 1 version 26 2024-06-03T08:36:32.359 INFO:teuthology.orchestra.run.smithi088.stderr:2024-06-03T08:36:32.358+0000 7f4c2effd640 10 monclient: get_auth_request con 0x7f4c14074230 auth_method 0 2024-06-03T08:36:32.360 INFO:teuthology.orchestra.run.smithi088.stderr:2024-06-03T08:36:32.360+0000 7f4c1bfff640 10 client.4323.objecter ms_handle_connect 0x7f4c14074230 2024-06-03T08:36:32.367 INFO:teuthology.orchestra.run.smithi088.stderr:2024-06-03T08:36:32.366+0000 7fccc7ad1640 10 monclient: get_monmap_and_config 2024-06-03T08:36:32.367 INFO:teuthology.orchestra.run.smithi088.stderr:2024-06-03T08:36:32.366+0000 7fccc7ad1640 10 monclient: build_initial_monmap 2024-06-03T08:36:32.367 INFO:teuthology.orchestra.run.smithi088.stderr:2024-06-03T08:36:32.366+0000 7fccc7ad1640 10 monclient: monmap: 2024-06-03T08:36:32.367 INFO:teuthology.orchestra.run.smithi088.stderr:epoch 0 2024-06-03T08:36:32.367 INFO:teuthology.orchestra.run.smithi088.stderr:fsid ec3c3fdd-3ac6-4d7d-9412-683130ed8b04 2024-06-03T08:36:32.367 INFO:teuthology.orchestra.run.smithi088.stderr:last_changed 2024-06-03T08:36:32.367893+0000 2024-06-03T08:36:32.367 INFO:teuthology.orchestra.run.smithi088.stderr:created 2024-06-03T08:36:32.367893+0000 2024-06-03T08:36:32.367 INFO:teuthology.orchestra.run.smithi088.stderr:min_mon_release 0 (unknown) 2024-06-03T08:36:32.367 INFO:teuthology.orchestra.run.smithi088.stderr:election_strategy: 3 2024-06-03T08:36:32.367 INFO:teuthology.orchestra.run.smithi088.stderr:0: v1:172.21.15.88:6789/0 mon.noname-a 2024-06-03T08:36:32.367 INFO:teuthology.orchestra.run.smithi088.stderr:1: v1:172.21.15.192:6789/0 mon.noname-c 2024-06-03T08:36:32.367 INFO:teuthology.orchestra.run.smithi088.stderr:2: v1:172.21.15.88:6790/0 mon.noname-b 2024-06-03T08:36:32.368 INFO:teuthology.orchestra.run.smithi088.stderr: 2024-06-03T08:36:32.369 INFO:teuthology.orchestra.run.smithi088.stderr:2024-06-03T08:36:32.367+0000 7fccc7ad1640 1 Processor -- start 2024-06-03T08:36:32.369 INFO:teuthology.orchestra.run.smithi088.stderr:2024-06-03T08:36:32.367+0000 7fccc7ad1640 1 -- start start 2024-06-03T08:36:32.369 INFO:teuthology.orchestra.run.smithi088.stderr:2024-06-03T08:36:32.367+0000 7fccc7ad1640 10 monclient: init 2024-06-03T08:36:32.369 INFO:teuthology.orchestra.run.smithi088.stderr:2024-06-03T08:36:32.368+0000 7fccc7ad1640 10 monclient: _reopen_session rank -1 2024-06-03T08:36:32.369 INFO:teuthology.orchestra.run.smithi088.stderr:2024-06-03T08:36:32.368+0000 7fccc7ad1640 10 monclient: _add_conns ranks=[0,1,2] 2024-06-03T08:36:32.369 INFO:teuthology.orchestra.run.smithi088.stderr:2024-06-03T08:36:32.368+0000 7fccc7ad1640 10 monclient(hunting): picked mon.noname-a con 0x7fccc00a8a60 addr v1:172.21.15.88:6789/0 2024-06-03T08:36:32.369 INFO:teuthology.orchestra.run.smithi088.stderr:2024-06-03T08:36:32.368+0000 7fccc7ad1640 10 monclient(hunting): picked mon.noname-c con 0x7fccc00b0d00 addr v1:172.21.15.192:6789/0 2024-06-03T08:36:32.369 INFO:teuthology.orchestra.run.smithi088.stderr:2024-06-03T08:36:32.368+0000 7fccc7ad1640 10 monclient(hunting): picked mon.noname-b con 0x7fccc01451d0 addr v1:172.21.15.88:6790/0 2024-06-03T08:36:32.369 INFO:teuthology.orchestra.run.smithi088.stderr:2024-06-03T08:36:32.368+0000 7fccc7ad1640 1 -- --> v1:172.21.15.88:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fccc00a9050 con 0x7fccc00a8a60 2024-06-03T08:36:32.369 INFO:teuthology.orchestra.run.smithi088.stderr:2024-06-03T08:36:32.368+0000 7fccc7ad1640 1 -- --> v1:172.21.15.192:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fccc00a9d90 con 0x7fccc00b0d00 2024-06-03T08:36:32.370 INFO:teuthology.orchestra.run.smithi088.stderr:2024-06-03T08:36:32.368+0000 7fccc7ad1640 1 -- --> v1:172.21.15.88:6790/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fccc00a8e40 con 0x7fccc01451d0 2024-06-03T08:36:32.370 INFO:teuthology.orchestra.run.smithi088.stderr:2024-06-03T08:36:32.368+0000 7fccc7ad1640 10 monclient(hunting): _renew_subs 2024-06-03T08:36:32.370 INFO:teuthology.orchestra.run.smithi088.stderr:2024-06-03T08:36:32.368+0000 7fccc7ad1640 10 monclient(hunting): authenticate will time out at 1248.178251s 2024-06-03T08:36:32.370 INFO:teuthology.orchestra.run.smithi088.stderr:2024-06-03T08:36:32.368+0000 7fccc586d640 1 --1- >> v1:172.21.15.88:6789/0 conn(0x7fccc00a8a60 0x7fccc0144ac0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.88:6789/0 says I am v1:172.21.15.88:60408/0 (socket says 172.21.15.88:60408) 2024-06-03T08:36:32.370 INFO:teuthology.orchestra.run.smithi088.stderr:2024-06-03T08:36:32.368+0000 7fccc586d640 1 -- 172.21.15.88:0/3715962079 learned_addr learned my addr 172.21.15.88:0/3715962079 (peer_addr_for_me v1:172.21.15.88:0/0) 2024-06-03T08:36:32.372 INFO:teuthology.orchestra.run.smithi088.stderr:2024-06-03T08:36:32.371+0000 7fccc486b640 1 -- 172.21.15.88:0/3715962079 <== mon.2 v1:172.21.15.88:6790/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 1194370322 0 0) 0x7fccc00a8e40 con 0x7fccc01451d0 2024-06-03T08:36:32.372 INFO:teuthology.orchestra.run.smithi088.stderr:2024-06-03T08:36:32.371+0000 7fccc486b640 10 monclient(hunting): _negotiate 2024-06-03T08:36:32.372 INFO:teuthology.orchestra.run.smithi088.stderr:2024-06-03T08:36:32.371+0000 7fccc486b640 10 monclient(hunting): _init_auth method 2 2024-06-03T08:36:32.372 INFO:teuthology.orchestra.run.smithi088.stderr:2024-06-03T08:36:32.371+0000 7fccc486b640 10 monclient(hunting): _init_auth creating new auth 2024-06-03T08:36:32.372 INFO:teuthology.orchestra.run.smithi088.stderr:2024-06-03T08:36:32.371+0000 7fccc486b640 10 monclient(hunting): my global_id is 4349 2024-06-03T08:36:32.372 INFO:teuthology.orchestra.run.smithi088.stderr:2024-06-03T08:36:32.371+0000 7fccc486b640 1 -- 172.21.15.88:0/3715962079 --> v1:172.21.15.88:6790/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7fccac003520 con 0x7fccc01451d0 2024-06-03T08:36:32.372 INFO:teuthology.orchestra.run.smithi088.stderr:2024-06-03T08:36:32.371+0000 7fccc486b640 1 -- 172.21.15.88:0/3715962079 <== mon.0 v1:172.21.15.88:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 1457589229 0 0) 0x7fccc00a9050 con 0x7fccc00a8a60 2024-06-03T08:36:32.372 INFO:teuthology.orchestra.run.smithi088.stderr:2024-06-03T08:36:32.371+0000 7fccc486b640 10 monclient(hunting): _negotiate 2024-06-03T08:36:32.372 INFO:teuthology.orchestra.run.smithi088.stderr:2024-06-03T08:36:32.371+0000 7fccc486b640 10 monclient(hunting): _init_auth method 2 2024-06-03T08:36:32.372 INFO:teuthology.orchestra.run.smithi088.stderr:2024-06-03T08:36:32.371+0000 7fccc486b640 10 monclient(hunting): _init_auth creating new auth 2024-06-03T08:36:32.372 INFO:teuthology.orchestra.run.smithi088.stderr:2024-06-03T08:36:32.371+0000 7fccc486b640 10 monclient(hunting): my global_id is 4338 2024-06-03T08:36:32.372 INFO:teuthology.orchestra.run.smithi088.stderr:2024-06-03T08:36:32.371+0000 7fccc486b640 1 -- 172.21.15.88:0/3715962079 --> v1:172.21.15.88:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7fccc00a8e40 con 0x7fccc00a8a60 2024-06-03T08:36:32.375 INFO:teuthology.orchestra.run.smithi088.stderr:2024-06-03T08:36:32.375+0000 7fccc486b640 1 -- 172.21.15.88:0/3715962079 <== mon.2 v1:172.21.15.88:6790/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 214+0+0 (unknown 1904716018 0 0) 0x7fccac003520 con 0x7fccc01451d0 2024-06-03T08:36:32.376 INFO:teuthology.orchestra.run.smithi088.stderr:2024-06-03T08:36:32.375+0000 7fccc486b640 1 -- 172.21.15.88:0/3715962079 >> v1:172.21.15.192:6789/0 conn(0x7fccc00b0d00 legacy=0x7fccc013e360 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-06-03T08:36:32.376 INFO:teuthology.orchestra.run.smithi088.stderr:2024-06-03T08:36:32.375+0000 7fccc486b640 1 -- 172.21.15.88:0/3715962079 >> v1:172.21.15.88:6789/0 conn(0x7fccc00a8a60 legacy=0x7fccc0144ac0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-06-03T08:36:32.376 INFO:teuthology.orchestra.run.smithi088.stderr:2024-06-03T08:36:32.375+0000 7fccc486b640 10 monclient: _finish_hunting 0 2024-06-03T08:36:32.376 INFO:teuthology.orchestra.run.smithi088.stderr:2024-06-03T08:36:32.375+0000 7fccc486b640 1 monclient: found mon.noname-b 2024-06-03T08:36:32.376 INFO:teuthology.orchestra.run.smithi088.stderr:2024-06-03T08:36:32.375+0000 7fccc486b640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-06-03T08:36:32.376 INFO:teuthology.orchestra.run.smithi088.stderr:2024-06-03T08:36:32.375+0000 7fccc486b640 10 monclient: _send_mon_message to mon.noname-b at v1:172.21.15.88:6790/0 2024-06-03T08:36:32.376 INFO:teuthology.orchestra.run.smithi088.stderr:2024-06-03T08:36:32.375+0000 7fccc486b640 1 -- 172.21.15.88:0/3715962079 --> v1:172.21.15.88:6790/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fccc013f180 con 0x7fccc01451d0 2024-06-03T08:36:32.376 INFO:teuthology.orchestra.run.smithi088.stderr:2024-06-03T08:36:32.375+0000 7fccc486b640 10 monclient: _finish_auth 0 2024-06-03T08:36:32.376 INFO:teuthology.orchestra.run.smithi088.stderr:2024-06-03T08:36:32.375+0000 7fccc486b640 10 monclient: _check_auth_tickets 2024-06-03T08:36:32.376 INFO:teuthology.orchestra.run.smithi088.stderr:2024-06-03T08:36:32.375+0000 7fccc486b640 20 monclient: _check_auth_rotating not needed by client.admin 2024-06-03T08:36:32.376 INFO:teuthology.orchestra.run.smithi088.stderr:2024-06-03T08:36:32.375+0000 7fccc486b640 1 -- 172.21.15.88:0/3715962079 <== mon.2 v1:172.21.15.88:6790/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 2234374065 0 0) 0x7fccb80019b0 con 0x7fccc01451d0 2024-06-03T08:36:32.376 INFO:teuthology.orchestra.run.smithi088.stderr:2024-06-03T08:36:32.375+0000 7fccc486b640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-06-03T08:36:32.376 INFO:teuthology.orchestra.run.smithi088.stderr:2024-06-03T08:36:32.375+0000 7fccc486b640 10 monclient: got monmap 1 from mon.noname-b (according to old e1) 2024-06-03T08:36:32.377 INFO:teuthology.orchestra.run.smithi088.stderr:2024-06-03T08:36:32.375+0000 7fccc486b640 10 monclient: dump: 2024-06-03T08:36:32.377 INFO:teuthology.orchestra.run.smithi088.stderr:epoch 1 2024-06-03T08:36:32.377 INFO:teuthology.orchestra.run.smithi088.stderr:fsid ec3c3fdd-3ac6-4d7d-9412-683130ed8b04 2024-06-03T08:36:32.377 INFO:teuthology.orchestra.run.smithi088.stderr:last_changed 2024-06-03T08:34:31.885125+0000 2024-06-03T08:36:32.377 INFO:teuthology.orchestra.run.smithi088.stderr:created 2024-06-03T08:34:31.885125+0000 2024-06-03T08:36:32.377 INFO:teuthology.orchestra.run.smithi088.stderr:min_mon_release 18 (reef) 2024-06-03T08:36:32.377 INFO:teuthology.orchestra.run.smithi088.stderr:election_strategy: 3 2024-06-03T08:36:32.377 INFO:teuthology.orchestra.run.smithi088.stderr:0: v1:172.21.15.88:6789/0 mon.a 2024-06-03T08:36:32.377 INFO:teuthology.orchestra.run.smithi088.stderr:1: v1:172.21.15.192:6789/0 mon.b 2024-06-03T08:36:32.377 INFO:teuthology.orchestra.run.smithi088.stderr:2: v1:172.21.15.88:6790/0 mon.c 2024-06-03T08:36:32.377 INFO:teuthology.orchestra.run.smithi088.stderr: 2024-06-03T08:36:32.377 INFO:teuthology.orchestra.run.smithi088.stderr:2024-06-03T08:36:32.375+0000 7fccc7ad1640 5 monclient: authenticate success, global_id 4349 2024-06-03T08:36:32.377 INFO:teuthology.orchestra.run.smithi088.stderr:2024-06-03T08:36:32.375+0000 7fccc7ad1640 20 monclient: get_monmap_and_config waiting for monmap|config 2024-06-03T08:36:32.379 INFO:teuthology.orchestra.run.smithi088.stderr:2024-06-03T08:36:32.377+0000 7fccc486b640 1 -- 172.21.15.88:0/3715962079 <== mon.2 v1:172.21.15.88:6790/0 4 ==== config(0 keys) v1 ==== 4+0+0 (unknown 0 0 0) 0x7fccb8001bd0 con 0x7fccc01451d0 2024-06-03T08:36:32.379 INFO:teuthology.orchestra.run.smithi088.stderr:2024-06-03T08:36:32.377+0000 7fccc486b640 10 monclient: handle_config config(0 keys) v1 2024-06-03T08:36:32.379 INFO:teuthology.orchestra.run.smithi088.stderr:2024-06-03T08:36:32.377+0000 7fccc486b640 1 -- 172.21.15.88:0/3715962079 <== mon.2 v1:172.21.15.88:6790/0 5 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 2234374065 0 0) 0x7fccb8001ee0 con 0x7fccc01451d0 2024-06-03T08:36:32.379 INFO:teuthology.orchestra.run.smithi088.stderr:2024-06-03T08:36:32.377+0000 7fccc7ad1640 10 monclient: get_monmap_and_config success 2024-06-03T08:36:32.379 INFO:teuthology.orchestra.run.smithi088.stderr:2024-06-03T08:36:32.377+0000 7fccc7ad1640 4 set_mon_vals no callback set 2024-06-03T08:36:32.379 INFO:teuthology.orchestra.run.smithi088.stderr:2024-06-03T08:36:32.377+0000 7fccc486b640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-06-03T08:36:32.379 INFO:teuthology.orchestra.run.smithi088.stderr:2024-06-03T08:36:32.377+0000 7fccc486b640 10 monclient: got monmap 1 from mon.c (according to old e1) 2024-06-03T08:36:32.379 INFO:teuthology.orchestra.run.smithi088.stderr:2024-06-03T08:36:32.377+0000 7fccc486b640 10 monclient: dump: 2024-06-03T08:36:32.379 INFO:teuthology.orchestra.run.smithi088.stderr:epoch 1 2024-06-03T08:36:32.379 INFO:teuthology.orchestra.run.smithi088.stderr:fsid ec3c3fdd-3ac6-4d7d-9412-683130ed8b04 2024-06-03T08:36:32.379 INFO:teuthology.orchestra.run.smithi088.stderr:last_changed 2024-06-03T08:34:31.885125+0000 2024-06-03T08:36:32.380 INFO:teuthology.orchestra.run.smithi088.stderr:created 2024-06-03T08:34:31.885125+0000 2024-06-03T08:36:32.380 INFO:teuthology.orchestra.run.smithi088.stderr:min_mon_release 18 (reef) 2024-06-03T08:36:32.380 INFO:teuthology.orchestra.run.smithi088.stderr:election_strategy: 3 2024-06-03T08:36:32.380 INFO:teuthology.orchestra.run.smithi088.stderr:0: v1:172.21.15.88:6789/0 mon.a 2024-06-03T08:36:32.380 INFO:teuthology.orchestra.run.smithi088.stderr:1: v1:172.21.15.192:6789/0 mon.b 2024-06-03T08:36:32.380 INFO:teuthology.orchestra.run.smithi088.stderr:2: v1:172.21.15.88:6790/0 mon.c 2024-06-03T08:36:32.380 INFO:teuthology.orchestra.run.smithi088.stderr: 2024-06-03T08:36:32.380 INFO:teuthology.orchestra.run.smithi088.stderr:2024-06-03T08:36:32.378+0000 7fccc7ad1640 10 monclient: shutdown 2024-06-03T08:36:32.380 INFO:teuthology.orchestra.run.smithi088.stderr:2024-06-03T08:36:32.378+0000 7fccc7ad1640 20 monclient: shutdown discarding 0 pending message(s) 2024-06-03T08:36:32.380 INFO:teuthology.orchestra.run.smithi088.stderr:2024-06-03T08:36:32.378+0000 7fccc7ad1640 1 -- 172.21.15.88:0/3715962079 >> v1:172.21.15.88:6790/0 conn(0x7fccc01451d0 legacy=0x7fccc013ea70 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-06-03T08:36:32.380 INFO:teuthology.orchestra.run.smithi088.stderr:2024-06-03T08:36:32.378+0000 7fccc7ad1640 1 -- 172.21.15.88:0/3715962079 shutdown_connections 2024-06-03T08:36:32.380 INFO:teuthology.orchestra.run.smithi088.stderr:2024-06-03T08:36:32.378+0000 7fccc7ad1640 1 -- 172.21.15.88:0/3715962079 >> 172.21.15.88:0/3715962079 conn(0x7fccc00b70d0 msgr2=0x7fccc00b74e0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-06-03T08:36:32.380 INFO:teuthology.orchestra.run.smithi088.stderr:2024-06-03T08:36:32.378+0000 7fccc7ad1640 1 -- 172.21.15.88:0/3715962079 shutdown_connections 2024-06-03T08:36:32.381 INFO:teuthology.orchestra.run.smithi088.stderr:2024-06-03T08:36:32.378+0000 7fccc7ad1640 1 -- 172.21.15.88:0/3715962079 wait complete. 2024-06-03T08:36:32.381 INFO:teuthology.orchestra.run.smithi088.stderr:2024-06-03T08:36:32.378+0000 7f7906ffd640 1 -- 172.21.15.88:0/996211931 <== osd.7 v1:172.21.15.192:6801/34787 1 ==== command_reply(tid 1: 0 ) v1 ==== 8+0+26118 (unknown 0 0 1686565167) 0x7f78f8077e30 con 0x7f78f80741c0 2024-06-03T08:36:32.381 INFO:teuthology.orchestra.run.smithi088.stderr:2024-06-03T08:36:32.378+0000 7f7906ffd640 10 client.4336.objecter ms_dispatch 0x7f7918000b90 command_reply(tid 1: 0 ) v1 2024-06-03T08:36:32.381 INFO:teuthology.orchestra.run.smithi088.stderr:2024-06-03T08:36:32.378+0000 7f7906ffd640 10 client.4336.objecter _finish_command 1 = system:0 2024-06-03T08:36:32.381 INFO:teuthology.orchestra.run.smithi088.stderr:2024-06-03T08:36:32.378+0000 7f7906ffd640 20 client.4336.objecter put_session s=0x7f78f8073b10 osd=7 4 2024-06-03T08:36:32.381 INFO:teuthology.orchestra.run.smithi088.stderr:2024-06-03T08:36:32.378+0000 7f7906ffd640 15 client.4336.objecter _session_command_op_remove 7 1 2024-06-03T08:36:32.383 INFO:teuthology.orchestra.run.smithi088.stderr:2024-06-03T08:36:32.379+0000 7fccc7ad1640 10 monclient: build_initial_monmap 2024-06-03T08:36:32.383 INFO:teuthology.orchestra.run.smithi088.stderr:2024-06-03T08:36:32.379+0000 7fccc7ad1640 10 monclient: monmap: 2024-06-03T08:36:32.383 INFO:teuthology.orchestra.run.smithi088.stderr:epoch 0 2024-06-03T08:36:32.383 INFO:teuthology.orchestra.run.smithi088.stderr:fsid 00000000-0000-0000-0000-000000000000 2024-06-03T08:36:32.383 INFO:teuthology.orchestra.run.smithi088.stderr:last_changed 0.000000 2024-06-03T08:36:32.383 INFO:teuthology.orchestra.run.smithi088.stderr:created 0.000000 2024-06-03T08:36:32.383 INFO:teuthology.orchestra.run.smithi088.stderr:min_mon_release 0 (unknown) 2024-06-03T08:36:32.383 INFO:teuthology.orchestra.run.smithi088.stderr:election_strategy: 1 2024-06-03T08:36:32.384 INFO:teuthology.orchestra.run.smithi088.stderr:0: v1:172.21.15.88:6789/0 mon.noname-a 2024-06-03T08:36:32.384 INFO:teuthology.orchestra.run.smithi088.stderr:1: v1:172.21.15.192:6789/0 mon.noname-b 2024-06-03T08:36:32.384 INFO:teuthology.orchestra.run.smithi088.stderr:2: v1:172.21.15.88:6790/0 mon.noname-c 2024-06-03T08:36:32.384 INFO:teuthology.orchestra.run.smithi088.stderr: 2024-06-03T08:36:32.384 INFO:teuthology.orchestra.run.smithi088.stderr:2024-06-03T08:36:32.379+0000 7fccc7ad1640 1 librados: starting msgr at 2024-06-03T08:36:32.384 INFO:teuthology.orchestra.run.smithi088.stderr:2024-06-03T08:36:32.379+0000 7fccc7ad1640 1 librados: starting objecter 2024-06-03T08:36:32.384 INFO:teuthology.orchestra.run.smithi088.stderr:2024-06-03T08:36:32.379+0000 7fccc7ad1640 1 Processor -- start 2024-06-03T08:36:32.384 INFO:teuthology.orchestra.run.smithi088.stderr:2024-06-03T08:36:32.379+0000 7fccc7ad1640 1 -- start start 2024-06-03T08:36:32.384 INFO:teuthology.orchestra.run.smithi088.stderr:2024-06-03T08:36:32.379+0000 7fccc7ad1640 1 librados: setting wanted keys 2024-06-03T08:36:32.384 INFO:teuthology.orchestra.run.smithi088.stderr:2024-06-03T08:36:32.379+0000 7fccc7ad1640 1 librados: calling monclient init 2024-06-03T08:36:32.384 INFO:teuthology.orchestra.run.smithi088.stderr:2024-06-03T08:36:32.379+0000 7fccc7ad1640 10 monclient: init 2024-06-03T08:36:32.384 INFO:teuthology.orchestra.run.smithi088.stderr:2024-06-03T08:36:32.380+0000 7fccc7ad1640 10 monclient: _reopen_session rank -1 2024-06-03T08:36:32.384 INFO:teuthology.orchestra.run.smithi088.stderr:2024-06-03T08:36:32.380+0000 7fccc7ad1640 10 monclient: _add_conns ranks=[2,0,1] 2024-06-03T08:36:32.384 INFO:teuthology.orchestra.run.smithi088.stderr:2024-06-03T08:36:32.380+0000 7fccc7ad1640 10 monclient(hunting): picked mon.noname-c con 0x7fccc00a8a60 addr v1:172.21.15.88:6790/0 2024-06-03T08:36:32.385 INFO:teuthology.orchestra.run.smithi088.stderr:2024-06-03T08:36:32.380+0000 7fccc7ad1640 10 monclient(hunting): picked mon.noname-a con 0x7fccc00b0d00 addr v1:172.21.15.88:6789/0 2024-06-03T08:36:32.385 INFO:teuthology.orchestra.run.smithi088.stderr:2024-06-03T08:36:32.380+0000 7fccc7ad1640 10 monclient(hunting): picked mon.noname-b con 0x7fccc01451d0 addr v1:172.21.15.192:6789/0 2024-06-03T08:36:32.385 INFO:teuthology.orchestra.run.smithi088.stderr:2024-06-03T08:36:32.380+0000 7fccc7ad1640 1 -- --> v1:172.21.15.88:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fccc01347e0 con 0x7fccc00b0d00 2024-06-03T08:36:32.385 INFO:teuthology.orchestra.run.smithi088.stderr:2024-06-03T08:36:32.380+0000 7fccc7ad1640 1 -- --> v1:172.21.15.192:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fccc012ebc0 con 0x7fccc01451d0 2024-06-03T08:36:32.385 INFO:teuthology.orchestra.run.smithi088.stderr:2024-06-03T08:36:32.380+0000 7fccc7ad1640 1 -- --> v1:172.21.15.88:6790/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fccc0135500 con 0x7fccc00a8a60 2024-06-03T08:36:32.385 INFO:teuthology.orchestra.run.smithi088.stderr:2024-06-03T08:36:32.380+0000 7fccc7ad1640 10 monclient(hunting): _renew_subs 2024-06-03T08:36:32.385 INFO:teuthology.orchestra.run.smithi088.stderr:2024-06-03T08:36:32.380+0000 7fccc7ad1640 10 monclient(hunting): authenticate will time out at 1248.190599s 2024-06-03T08:36:32.385 INFO:teuthology.orchestra.run.smithi088.stderr:2024-06-03T08:36:32.380+0000 7fccc506c640 1 --1- >> v1:172.21.15.88:6789/0 conn(0x7fccc00b0d00 0x7fccc012dda0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=1).handle_server_banner_and_identify peer v1:172.21.15.88:6789/0 says I am v1:172.21.15.88:60416/0 (socket says 172.21.15.88:60416) 2024-06-03T08:36:32.385 INFO:teuthology.orchestra.run.smithi088.stderr:2024-06-03T08:36:32.380+0000 7fccc506c640 1 -- 172.21.15.88:0/1456172716 learned_addr learned my addr 172.21.15.88:0/1456172716 (peer_addr_for_me v1:172.21.15.88:0/0) 2024-06-03T08:36:32.385 INFO:teuthology.orchestra.run.smithi088.stderr:2024-06-03T08:36:32.381+0000 7fccb67fc640 10 client.?.objecter ms_handle_connect 0x7fccc00b0d00 2024-06-03T08:36:32.385 INFO:teuthology.orchestra.run.smithi088.stderr:2024-06-03T08:36:32.381+0000 7fccb67fc640 10 client.?.objecter resend_mon_ops 2024-06-03T08:36:32.385 INFO:teuthology.orchestra.run.smithi088.stderr:2024-06-03T08:36:32.381+0000 7fccb67fc640 10 client.?.objecter ms_handle_connect 0x7fccc00a8a60 2024-06-03T08:36:32.385 INFO:teuthology.orchestra.run.smithi088.stderr:2024-06-03T08:36:32.381+0000 7fccb67fc640 10 client.?.objecter resend_mon_ops 2024-06-03T08:36:32.386 INFO:teuthology.orchestra.run.smithi088.stderr:2024-06-03T08:36:32.384+0000 7fccb67fc640 1 -- 172.21.15.88:0/1456172716 <== mon.0 v1:172.21.15.88:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 4161563933 0 0) 0x7fccb00011a0 con 0x7fccc00b0d00 2024-06-03T08:36:32.386 INFO:teuthology.orchestra.run.smithi088.stderr:2024-06-03T08:36:32.384+0000 7fccb67fc640 10 monclient(hunting): _negotiate 2024-06-03T08:36:32.386 INFO:teuthology.orchestra.run.smithi088.stderr:2024-06-03T08:36:32.384+0000 7fccb67fc640 10 monclient(hunting): _init_auth method 2 2024-06-03T08:36:32.386 INFO:teuthology.orchestra.run.smithi088.stderr:2024-06-03T08:36:32.384+0000 7fccb67fc640 10 monclient(hunting): _init_auth creating new auth 2024-06-03T08:36:32.386 INFO:teuthology.orchestra.run.smithi088.stderr:2024-06-03T08:36:32.384+0000 7fccb67fc640 10 monclient(hunting): my global_id is 4341 2024-06-03T08:36:32.387 INFO:teuthology.orchestra.run.smithi088.stderr:2024-06-03T08:36:32.384+0000 7fccb67fc640 1 -- 172.21.15.88:0/1456172716 --> v1:172.21.15.88:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7fcca4004eb0 con 0x7fccc00b0d00 2024-06-03T08:36:32.387 INFO:teuthology.orchestra.run.smithi088.stderr:2024-06-03T08:36:32.384+0000 7fccb67fc640 1 -- 172.21.15.88:0/1456172716 <== mon.2 v1:172.21.15.88:6790/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 867626716 0 0) 0x7fccc0135500 con 0x7fccc00a8a60 2024-06-03T08:36:32.387 INFO:teuthology.orchestra.run.smithi088.stderr:2024-06-03T08:36:32.384+0000 7fccb67fc640 10 monclient(hunting): _negotiate 2024-06-03T08:36:32.387 INFO:teuthology.orchestra.run.smithi088.stderr:2024-06-03T08:36:32.384+0000 7fccb67fc640 10 monclient(hunting): _init_auth method 2 2024-06-03T08:36:32.387 INFO:teuthology.orchestra.run.smithi088.stderr:2024-06-03T08:36:32.384+0000 7fccb67fc640 10 monclient(hunting): _init_auth creating new auth 2024-06-03T08:36:32.387 INFO:teuthology.orchestra.run.smithi088.stderr:2024-06-03T08:36:32.384+0000 7fccb67fc640 10 monclient(hunting): my global_id is 4352 2024-06-03T08:36:32.387 INFO:teuthology.orchestra.run.smithi088.stderr:2024-06-03T08:36:32.384+0000 7fccb67fc640 1 -- 172.21.15.88:0/1456172716 --> v1:172.21.15.88:6790/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7fccb00011a0 con 0x7fccc00a8a60 2024-06-03T08:36:32.390 INFO:teuthology.orchestra.run.smithi088.stderr:2024-06-03T08:36:32.390+0000 7fccb67fc640 1 -- 172.21.15.88:0/1456172716 <== mon.2 v1:172.21.15.88:6790/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 764+0+0 (unknown 1359759990 0 0) 0x7fccb00011a0 con 0x7fccc00a8a60 2024-06-03T08:36:32.390 INFO:teuthology.orchestra.run.smithi088.stderr:2024-06-03T08:36:32.390+0000 7fccb67fc640 1 -- 172.21.15.88:0/1456172716 --> v1:172.21.15.88:6790/0 -- auth(proto 2 165 bytes epoch 0) v1 -- 0x7fccc0135500 con 0x7fccc00a8a60 2024-06-03T08:36:32.390 INFO:teuthology.orchestra.run.smithi088.stderr:2024-06-03T08:36:32.390+0000 7fccb67fc640 1 -- 172.21.15.88:0/1456172716 <== mon.2 v1:172.21.15.88:6790/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 2234374065 0 0) 0x7fccbc0018d0 con 0x7fccc00a8a60 2024-06-03T08:36:32.390 INFO:teuthology.orchestra.run.smithi088.stderr:2024-06-03T08:36:32.390+0000 7fccb67fc640 10 monclient(hunting): handle_monmap mon_map magic: 0 v1 2024-06-03T08:36:32.390 INFO:teuthology.orchestra.run.smithi088.stderr:2024-06-03T08:36:32.390+0000 7fccb67fc640 10 monclient(hunting): got monmap 1 from mon.noname-c (according to old e1) 2024-06-03T08:36:32.391 INFO:teuthology.orchestra.run.smithi088.stderr:2024-06-03T08:36:32.390+0000 7fccb67fc640 10 monclient(hunting): dump: 2024-06-03T08:36:32.391 INFO:teuthology.orchestra.run.smithi088.stderr:epoch 1 2024-06-03T08:36:32.391 INFO:teuthology.orchestra.run.smithi088.stderr:fsid ec3c3fdd-3ac6-4d7d-9412-683130ed8b04 2024-06-03T08:36:32.391 INFO:teuthology.orchestra.run.smithi088.stderr:last_changed 2024-06-03T08:34:31.885125+0000 2024-06-03T08:36:32.391 INFO:teuthology.orchestra.run.smithi088.stderr:created 2024-06-03T08:34:31.885125+0000 2024-06-03T08:36:32.391 INFO:teuthology.orchestra.run.smithi088.stderr:min_mon_release 18 (reef) 2024-06-03T08:36:32.391 INFO:teuthology.orchestra.run.smithi088.stderr:election_strategy: 3 2024-06-03T08:36:32.391 INFO:teuthology.orchestra.run.smithi088.stderr:0: v1:172.21.15.88:6789/0 mon.a 2024-06-03T08:36:32.391 INFO:teuthology.orchestra.run.smithi088.stderr:1: v1:172.21.15.192:6789/0 mon.b 2024-06-03T08:36:32.391 INFO:teuthology.orchestra.run.smithi088.stderr:2: v1:172.21.15.88:6790/0 mon.c 2024-06-03T08:36:32.391 INFO:teuthology.orchestra.run.smithi088.stderr: 2024-06-03T08:36:32.391 INFO:teuthology.orchestra.run.smithi088.stderr:2024-06-03T08:36:32.390+0000 7fccb67fc640 10 monclient(hunting): _finish_auth 0 2024-06-03T08:36:32.392 INFO:teuthology.orchestra.run.smithi088.stderr:2024-06-03T08:36:32.390+0000 7f9b9e7fc640 1 -- 172.21.15.88:0/496575353 <== osd.6 v1:172.21.15.192:6813/34788 1 ==== command_reply(tid 1: 0 ) v1 ==== 8+0+26118 (unknown 0 0 1686565167) 0x7f9b8c077d20 con 0x7f9b8c074150 2024-06-03T08:36:32.392 INFO:teuthology.orchestra.run.smithi088.stderr:2024-06-03T08:36:32.390+0000 7f9b9e7fc640 10 client.4326.objecter ms_dispatch 0x7f9ba8000b90 command_reply(tid 1: 0 ) v1 2024-06-03T08:36:32.392 INFO:teuthology.orchestra.run.smithi088.stderr:2024-06-03T08:36:32.390+0000 7f9b9e7fc640 10 client.4326.objecter _finish_command 1 = system:0 2024-06-03T08:36:32.392 INFO:teuthology.orchestra.run.smithi088.stderr:2024-06-03T08:36:32.390+0000 7f9b9e7fc640 20 client.4326.objecter put_session s=0x7f9b8c073aa0 osd=6 4 2024-06-03T08:36:32.392 INFO:teuthology.orchestra.run.smithi088.stderr:2024-06-03T08:36:32.390+0000 7f9b9e7fc640 15 client.4326.objecter _session_command_op_remove 6 1 2024-06-03T08:36:32.393 INFO:teuthology.orchestra.run.smithi088.stderr:2024-06-03T08:36:32.392+0000 7fa0d37fe640 1 -- 172.21.15.88:0/1282623662 <== osd.5 v1:172.21.15.192:6805/34786 1 ==== command_reply(tid 1: 0 ) v1 ==== 8+0+26118 (unknown 0 0 1686565167) 0x7fa0d8077d40 con 0x7fa0d8074170 2024-06-03T08:36:32.393 INFO:teuthology.orchestra.run.smithi088.stderr:2024-06-03T08:36:32.392+0000 7fa0d37fe640 10 client.4333.objecter ms_dispatch 0x7fa0ec000b90 command_reply(tid 1: 0 ) v1 2024-06-03T08:36:32.393 INFO:teuthology.orchestra.run.smithi088.stderr:2024-06-03T08:36:32.392+0000 7fa0d37fe640 10 client.4333.objecter _finish_command 1 = system:0 2024-06-03T08:36:32.393 INFO:teuthology.orchestra.run.smithi088.stderr:2024-06-03T08:36:32.392+0000 7fa0d37fe640 20 client.4333.objecter put_session s=0x7fa0d8073ac0 osd=5 4 2024-06-03T08:36:32.393 INFO:teuthology.orchestra.run.smithi088.stderr:2024-06-03T08:36:32.392+0000 7fa0d37fe640 15 client.4333.objecter _session_command_op_remove 5 1 2024-06-03T08:36:32.393 INFO:teuthology.orchestra.run.smithi088.stderr:2024-06-03T08:36:32.392+0000 7fccb67fc640 1 -- 172.21.15.88:0/1456172716 <== mon.0 v1:172.21.15.88:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 764+0+0 (unknown 2111116870 0 0) 0x7fcca4004eb0 con 0x7fccc00b0d00 2024-06-03T08:36:32.393 INFO:teuthology.orchestra.run.smithi088.stderr:2024-06-03T08:36:32.392+0000 7fccb67fc640 1 -- 172.21.15.88:0/1456172716 --> v1:172.21.15.88:6789/0 -- auth(proto 2 165 bytes epoch 0) v1 -- 0x7fccb00011a0 con 0x7fccc00b0d00 2024-06-03T08:36:32.394 INFO:teuthology.orchestra.run.smithi088.stderr:2024-06-03T08:36:32.392+0000 7fccb67fc640 1 -- 172.21.15.88:0/1456172716 <== mon.0 v1:172.21.15.88:6789/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 2234374065 0 0) 0x7fccb00018d0 con 0x7fccc00b0d00 2024-06-03T08:36:32.394 INFO:teuthology.orchestra.run.smithi088.stderr:2024-06-03T08:36:32.392+0000 7fccb67fc640 10 monclient(hunting): handle_monmap mon_map magic: 0 v1 2024-06-03T08:36:32.394 INFO:teuthology.orchestra.run.smithi088.stderr:2024-06-03T08:36:32.392+0000 7fccb67fc640 10 monclient(hunting): got monmap 1 from mon.a (according to old e1) 2024-06-03T08:36:32.394 INFO:teuthology.orchestra.run.smithi088.stderr:2024-06-03T08:36:32.392+0000 7fccb67fc640 10 monclient(hunting): dump: 2024-06-03T08:36:32.394 INFO:teuthology.orchestra.run.smithi088.stderr:epoch 1 2024-06-03T08:36:32.394 INFO:teuthology.orchestra.run.smithi088.stderr:fsid ec3c3fdd-3ac6-4d7d-9412-683130ed8b04 2024-06-03T08:36:32.394 INFO:teuthology.orchestra.run.smithi088.stderr:last_changed 2024-06-03T08:34:31.885125+0000 2024-06-03T08:36:32.394 INFO:teuthology.orchestra.run.smithi088.stderr:created 2024-06-03T08:34:31.885125+0000 2024-06-03T08:36:32.394 INFO:teuthology.orchestra.run.smithi088.stderr:min_mon_release 18 (reef) 2024-06-03T08:36:32.394 INFO:teuthology.orchestra.run.smithi088.stderr:election_strategy: 3 2024-06-03T08:36:32.394 INFO:teuthology.orchestra.run.smithi088.stderr:0: v1:172.21.15.88:6789/0 mon.a 2024-06-03T08:36:32.394 INFO:teuthology.orchestra.run.smithi088.stderr:1: v1:172.21.15.192:6789/0 mon.b 2024-06-03T08:36:32.394 INFO:teuthology.orchestra.run.smithi088.stderr:2: v1:172.21.15.88:6790/0 mon.c 2024-06-03T08:36:32.394 INFO:teuthology.orchestra.run.smithi088.stderr: 2024-06-03T08:36:32.396 INFO:teuthology.orchestra.run.smithi088.stderr:2024-06-03T08:36:32.396+0000 7fccb67fc640 1 -- 172.21.15.88:0/1456172716 <== mon.2 v1:172.21.15.88:6790/0 4 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 580+0+0 (unknown 2368149431 0 0) 0x7fccc0135500 con 0x7fccc00a8a60 2024-06-03T08:36:32.396 INFO:teuthology.orchestra.run.smithi088.stderr:2024-06-03T08:36:32.396+0000 7fccb67fc640 1 -- 172.21.15.88:0/1456172716 >> v1:172.21.15.192:6789/0 conn(0x7fccc01451d0 legacy=0x7fccc012e4b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-06-03T08:36:32.397 INFO:teuthology.orchestra.run.smithi088.stderr:2024-06-03T08:36:32.396+0000 7fccb67fc640 1 -- 172.21.15.88:0/1456172716 >> v1:172.21.15.88:6789/0 conn(0x7fccc00b0d00 legacy=0x7fccc012dda0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-06-03T08:36:32.397 INFO:teuthology.orchestra.run.smithi088.stderr:2024-06-03T08:36:32.396+0000 7fccb67fc640 10 monclient: _finish_hunting 0 2024-06-03T08:36:32.397 INFO:teuthology.orchestra.run.smithi088.stderr:2024-06-03T08:36:32.396+0000 7fccb67fc640 1 monclient: found mon.c 2024-06-03T08:36:32.397 INFO:teuthology.orchestra.run.smithi088.stderr:2024-06-03T08:36:32.396+0000 7fccb67fc640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-06-03T08:36:32.397 INFO:teuthology.orchestra.run.smithi088.stderr:2024-06-03T08:36:32.396+0000 7fccb67fc640 10 monclient: _send_mon_message to mon.c at v1:172.21.15.88:6790/0 2024-06-03T08:36:32.397 INFO:teuthology.orchestra.run.smithi088.stderr:2024-06-03T08:36:32.396+0000 7fccb67fc640 1 -- 172.21.15.88:0/1456172716 --> v1:172.21.15.88:6790/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fccc01357e0 con 0x7fccc00a8a60 2024-06-03T08:36:32.397 INFO:teuthology.orchestra.run.smithi088.stderr:2024-06-03T08:36:32.396+0000 7fccb67fc640 10 monclient: _finish_auth 0 2024-06-03T08:36:32.397 INFO:teuthology.orchestra.run.smithi088.stderr:2024-06-03T08:36:32.396+0000 7fccb67fc640 10 monclient: _check_auth_tickets 2024-06-03T08:36:32.397 INFO:teuthology.orchestra.run.smithi088.stderr:2024-06-03T08:36:32.396+0000 7fccb67fc640 20 monclient: _check_auth_rotating not needed by client.admin 2024-06-03T08:36:32.397 INFO:teuthology.orchestra.run.smithi088.stderr:2024-06-03T08:36:32.396+0000 7fccc7ad1640 5 monclient: authenticate success, global_id 4352 2024-06-03T08:36:32.397 INFO:teuthology.orchestra.run.smithi088.stderr:2024-06-03T08:36:32.396+0000 7fccc7ad1640 10 monclient: _renew_subs 2024-06-03T08:36:32.397 INFO:teuthology.orchestra.run.smithi088.stderr:2024-06-03T08:36:32.396+0000 7fccc7ad1640 10 monclient: _send_mon_message to mon.c at v1:172.21.15.88:6790/0 2024-06-03T08:36:32.397 INFO:teuthology.orchestra.run.smithi088.stderr:2024-06-03T08:36:32.396+0000 7fccc7ad1640 1 -- 172.21.15.88:0/1456172716 --> v1:172.21.15.88:6790/0 -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fccc0133b90 con 0x7fccc00a8a60 2024-06-03T08:36:32.398 INFO:teuthology.orchestra.run.smithi088.stderr:2024-06-03T08:36:32.396+0000 7fccc7ad1640 10 client.4352.objecter _maybe_request_map subscribing (onetime) to next osd map 2024-06-03T08:36:32.398 INFO:teuthology.orchestra.run.smithi088.stderr:2024-06-03T08:36:32.396+0000 7fccc7ad1640 10 monclient: _renew_subs 2024-06-03T08:36:32.398 INFO:teuthology.orchestra.run.smithi088.stderr:2024-06-03T08:36:32.396+0000 7fccc7ad1640 10 monclient: _send_mon_message to mon.c at v1:172.21.15.88:6790/0 2024-06-03T08:36:32.398 INFO:teuthology.orchestra.run.smithi088.stderr:2024-06-03T08:36:32.396+0000 7fccc7ad1640 1 -- 172.21.15.88:0/1456172716 --> v1:172.21.15.88:6790/0 -- mon_subscribe({osdmap=0}) v3 -- 0x7fccc0133180 con 0x7fccc00a8a60 2024-06-03T08:36:32.398 INFO:teuthology.orchestra.run.smithi088.stderr:2024-06-03T08:36:32.396+0000 7fccc7ad1640 1 librados: init done 2024-06-03T08:36:32.398 INFO:teuthology.orchestra.run.smithi088.stderr:2024-06-03T08:36:32.397+0000 7fd649ffb640 1 -- 172.21.15.88:0/2839181686 <== osd.0 v1:172.21.15.88:6808/35066 1 ==== command_reply(tid 1: 0 ) v1 ==== 8+0+26118 (unknown 0 0 1686565167) 0x7fd638077f20 con 0x7fd638074350 2024-06-03T08:36:32.398 INFO:teuthology.orchestra.run.smithi088.stderr:2024-06-03T08:36:32.397+0000 7fd649ffb640 10 client.4320.objecter ms_dispatch 0x7fd654000b90 command_reply(tid 1: 0 ) v1 2024-06-03T08:36:32.399 INFO:teuthology.orchestra.run.smithi088.stderr:2024-06-03T08:36:32.397+0000 7fd649ffb640 10 client.4320.objecter _finish_command 1 = system:0 2024-06-03T08:36:32.399 INFO:teuthology.orchestra.run.smithi088.stderr:2024-06-03T08:36:32.397+0000 7fd649ffb640 20 client.4320.objecter put_session s=0x7fd638073ca0 osd=0 4 2024-06-03T08:36:32.399 INFO:teuthology.orchestra.run.smithi088.stderr:2024-06-03T08:36:32.397+0000 7fd649ffb640 15 client.4320.objecter _session_command_op_remove 0 1 2024-06-03T08:36:32.399 INFO:teuthology.orchestra.run.smithi088.stderr:2024-06-03T08:36:32.397+0000 7f791cb92640 10 client.4336.objecter _submit_command 2 [{"prefix": "flush_pg_stats"}] 2024-06-03T08:36:32.399 INFO:teuthology.orchestra.run.smithi088.stderr:2024-06-03T08:36:32.397+0000 7f791cb92640 15 client.4336.objecter _session_command_op_assign -1 2 2024-06-03T08:36:32.399 INFO:teuthology.orchestra.run.smithi088.stderr:2024-06-03T08:36:32.397+0000 7f791cb92640 20 client.4336.objecter _get_session s=0x7f78f8073b10 osd=7 3 2024-06-03T08:36:32.399 INFO:teuthology.orchestra.run.smithi088.stderr:2024-06-03T08:36:32.397+0000 7f791cb92640 20 client.4336.objecter put_session s=0x7f78f8073b10 osd=7 3 2024-06-03T08:36:32.399 INFO:teuthology.orchestra.run.smithi088.stderr:2024-06-03T08:36:32.397+0000 7f791cb92640 20 client.4336.objecter _get_session s=0x7f78f8073b10 osd=7 3 2024-06-03T08:36:32.399 INFO:teuthology.orchestra.run.smithi088.stderr:2024-06-03T08:36:32.397+0000 7f791cb92640 15 client.4336.objecter _session_command_op_remove -1 2 2024-06-03T08:36:32.399 INFO:teuthology.orchestra.run.smithi088.stderr:2024-06-03T08:36:32.397+0000 7f791cb92640 20 client.4336.objecter get_session s=0x7f78f8073b10 osd=7 3 2024-06-03T08:36:32.399 INFO:teuthology.orchestra.run.smithi088.stderr:2024-06-03T08:36:32.397+0000 7f791cb92640 15 client.4336.objecter _session_command_op_assign 7 2 2024-06-03T08:36:32.400 INFO:teuthology.orchestra.run.smithi088.stderr:2024-06-03T08:36:32.397+0000 7f791cb92640 20 client.4336.objecter put_session s=0x7f78f8073b10 osd=7 4 2024-06-03T08:36:32.400 INFO:teuthology.orchestra.run.smithi088.stderr:2024-06-03T08:36:32.397+0000 7f791cb92640 10 client.4336.objecter _send_command 2 2024-06-03T08:36:32.400 INFO:teuthology.orchestra.run.smithi088.stderr:2024-06-03T08:36:32.397+0000 7f791cb92640 1 -- 172.21.15.88:0/996211931 --> v1:172.21.15.192:6801/34787 -- command(tid 2: {"prefix": "flush_pg_stats"}) v1 -- 0x7f7918071db0 con 0x7f78f80741c0 2024-06-03T08:36:32.400 INFO:teuthology.orchestra.run.smithi088.stderr:2024-06-03T08:36:32.397+0000 7fccc7ad1640 10 start_mon_command cmd=[{"prefix": "get_command_descriptions"}] 2024-06-03T08:36:32.400 INFO:teuthology.orchestra.run.smithi088.stderr:2024-06-03T08:36:32.397+0000 7fccc7ad1640 10 monclient: _send_command 1 [{"prefix": "get_command_descriptions"}] 2024-06-03T08:36:32.400 INFO:teuthology.orchestra.run.smithi088.stderr:2024-06-03T08:36:32.397+0000 7fccc7ad1640 10 monclient: _send_mon_message to mon.c at v1:172.21.15.88:6790/0 2024-06-03T08:36:32.401 INFO:teuthology.orchestra.run.smithi088.stderr:2024-06-03T08:36:32.397+0000 7fccc7ad1640 1 -- 172.21.15.88:0/1456172716 --> v1:172.21.15.88:6790/0 -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fccc0133470 con 0x7fccc00a8a60 2024-06-03T08:36:32.414 INFO:teuthology.orchestra.run.smithi088.stderr:2024-06-03T08:36:32.414+0000 7f4c1bfff640 1 -- 172.21.15.88:0/2099487910 <== osd.1 v1:172.21.15.88:6800/35067 1 ==== command_reply(tid 1: 0 ) v1 ==== 8+0+26118 (unknown 0 0 1686565167) 0x7f4c14077ea0 con 0x7f4c14074230 2024-06-03T08:36:32.414 INFO:teuthology.orchestra.run.smithi088.stderr:2024-06-03T08:36:32.414+0000 7f4c1bfff640 10 client.4323.objecter ms_dispatch 0x7f4c30000b90 command_reply(tid 1: 0 ) v1 2024-06-03T08:36:32.415 INFO:teuthology.orchestra.run.smithi088.stderr:2024-06-03T08:36:32.414+0000 7f4c1bfff640 10 client.4323.objecter _finish_command 1 = system:0 2024-06-03T08:36:32.415 INFO:teuthology.orchestra.run.smithi088.stderr:2024-06-03T08:36:32.414+0000 7f4c1bfff640 20 client.4323.objecter put_session s=0x7f4c14073b80 osd=1 4 2024-06-03T08:36:32.415 INFO:teuthology.orchestra.run.smithi088.stderr:2024-06-03T08:36:32.414+0000 7f4c1bfff640 15 client.4323.objecter _session_command_op_remove 1 1 2024-06-03T08:36:32.419 INFO:teuthology.orchestra.run.smithi088.stderr:2024-06-03T08:36:32.419+0000 7fd65b934640 10 client.4320.objecter _submit_command 2 [{"prefix": "flush_pg_stats"}] 2024-06-03T08:36:32.419 INFO:teuthology.orchestra.run.smithi088.stderr:2024-06-03T08:36:32.419+0000 7fd65b934640 15 client.4320.objecter _session_command_op_assign -1 2 2024-06-03T08:36:32.420 INFO:teuthology.orchestra.run.smithi088.stderr:2024-06-03T08:36:32.419+0000 7fd65b934640 20 client.4320.objecter _get_session s=0x7fd638073ca0 osd=0 3 2024-06-03T08:36:32.420 INFO:teuthology.orchestra.run.smithi088.stderr:2024-06-03T08:36:32.419+0000 7fd65b934640 20 client.4320.objecter put_session s=0x7fd638073ca0 osd=0 3 2024-06-03T08:36:32.420 INFO:teuthology.orchestra.run.smithi088.stderr:2024-06-03T08:36:32.419+0000 7fd65b934640 20 client.4320.objecter _get_session s=0x7fd638073ca0 osd=0 3 2024-06-03T08:36:32.420 INFO:teuthology.orchestra.run.smithi088.stderr:2024-06-03T08:36:32.419+0000 7fd65b934640 15 client.4320.objecter _session_command_op_remove -1 2 2024-06-03T08:36:32.420 INFO:teuthology.orchestra.run.smithi088.stderr:2024-06-03T08:36:32.419+0000 7fd65b934640 20 client.4320.objecter get_session s=0x7fd638073ca0 osd=0 3 2024-06-03T08:36:32.420 INFO:teuthology.orchestra.run.smithi088.stderr:2024-06-03T08:36:32.419+0000 7fd65b934640 15 client.4320.objecter _session_command_op_assign 0 2 2024-06-03T08:36:32.420 INFO:teuthology.orchestra.run.smithi088.stderr:2024-06-03T08:36:32.419+0000 7fd65b934640 20 client.4320.objecter put_session s=0x7fd638073ca0 osd=0 4 2024-06-03T08:36:32.420 INFO:teuthology.orchestra.run.smithi088.stderr:2024-06-03T08:36:32.419+0000 7fd65b934640 10 client.4320.objecter _send_command 2 2024-06-03T08:36:32.420 INFO:teuthology.orchestra.run.smithi088.stderr:2024-06-03T08:36:32.419+0000 7fd65b934640 1 -- 172.21.15.88:0/2839181686 --> v1:172.21.15.88:6808/35066 -- command(tid 2: {"prefix": "flush_pg_stats"}) v1 -- 0x7fd65404ea60 con 0x7fd638074350 2024-06-03T08:36:32.422 INFO:teuthology.orchestra.run.smithi088.stderr:2024-06-03T08:36:32.421+0000 7f7906ffd640 1 -- 172.21.15.88:0/996211931 <== osd.7 v1:172.21.15.192:6801/34787 2 ==== command_reply(tid 2: 0 ) v1 ==== 8+0+11 (unknown 0 0 1608317872) 0x7f7918071db0 con 0x7f78f80741c0 2024-06-03T08:36:32.422 INFO:teuthology.orchestra.run.smithi088.stderr:2024-06-03T08:36:32.421+0000 7f7906ffd640 10 client.4336.objecter ms_dispatch 0x7f7918000b90 command_reply(tid 2: 0 ) v1 2024-06-03T08:36:32.422 INFO:teuthology.orchestra.run.smithi088.stderr:2024-06-03T08:36:32.421+0000 7f7906ffd640 10 client.4336.objecter _finish_command 2 = system:0 2024-06-03T08:36:32.422 INFO:teuthology.orchestra.run.smithi088.stderr:2024-06-03T08:36:32.421+0000 7f7906ffd640 20 client.4336.objecter put_session s=0x7f78f8073b10 osd=7 4 2024-06-03T08:36:32.422 INFO:teuthology.orchestra.run.smithi088.stderr:2024-06-03T08:36:32.421+0000 7f7906ffd640 15 client.4336.objecter _session_command_op_remove 7 2 2024-06-03T08:36:32.423 INFO:teuthology.orchestra.run.smithi088.stderr:2024-06-03T08:36:32.422+0000 7f7904ff9640 10 librados: watch_flush enter 2024-06-03T08:36:32.423 INFO:teuthology.orchestra.run.smithi088.stderr:2024-06-03T08:36:32.422+0000 7f7904ff9640 10 librados: watch_flush exit 2024-06-03T08:36:32.423 INFO:teuthology.orchestra.run.smithi088.stderr:2024-06-03T08:36:32.422+0000 7f7904ff9640 10 client.4336.objecter close_session for osd.7 2024-06-03T08:36:32.423 INFO:teuthology.orchestra.run.smithi088.stderr:2024-06-03T08:36:32.422+0000 7f7904ff9640 1 -- 172.21.15.88:0/996211931 >> v1:172.21.15.192:6801/34787 conn(0x7f78f80741c0 legacy=0x7f78f80765e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-06-03T08:36:32.424 INFO:teuthology.orchestra.run.smithi088.stderr:2024-06-03T08:36:32.422+0000 7f7904ff9640 20 client.4336.objecter put_session s=0x7f78f8073b10 osd=7 1 2024-06-03T08:36:32.424 INFO:teuthology.orchestra.run.smithi088.stderr:2024-06-03T08:36:32.422+0000 7f7904ff9640 20 client.4336.objecter shutdown clearing up homeless session... 2024-06-03T08:36:32.424 INFO:teuthology.orchestra.run.smithi088.stderr:2024-06-03T08:36:32.422+0000 7f7904ff9640 10 client.4336.objecter successfully canceled tick 2024-06-03T08:36:32.424 INFO:teuthology.orchestra.run.smithi088.stderr:2024-06-03T08:36:32.422+0000 7f7904ff9640 1 -- 172.21.15.88:0/996211931 >> v1:172.21.15.192:6800/34229 conn(0x7f78f806d100 legacy=0x7f78f806f5c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-06-03T08:36:32.424 INFO:teuthology.orchestra.run.smithi088.stderr:2024-06-03T08:36:32.422+0000 7f7904ff9640 10 monclient: shutdown 2024-06-03T08:36:32.424 INFO:teuthology.orchestra.run.smithi088.stderr:2024-06-03T08:36:32.422+0000 7f7904ff9640 20 monclient: shutdown discarding 0 pending message(s) 2024-06-03T08:36:32.424 INFO:teuthology.orchestra.run.smithi088.stderr:2024-06-03T08:36:32.422+0000 7f7904ff9640 1 -- 172.21.15.88:0/996211931 >> v1:172.21.15.192:6789/0 conn(0x7f791814ae20 legacy=0x7f79181d6ac0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-06-03T08:36:32.424 INFO:teuthology.orchestra.run.smithi088.stderr:2024-06-03T08:36:32.422+0000 7f9bafd56640 10 client.4326.objecter _submit_command 2 [{"prefix": "flush_pg_stats"}] 2024-06-03T08:36:32.425 INFO:teuthology.orchestra.run.smithi088.stderr:2024-06-03T08:36:32.422+0000 7f9bafd56640 15 client.4326.objecter _session_command_op_assign -1 2 2024-06-03T08:36:32.425 INFO:teuthology.orchestra.run.smithi088.stderr:2024-06-03T08:36:32.422+0000 7f9bafd56640 20 client.4326.objecter _get_session s=0x7f9b8c073aa0 osd=6 3 2024-06-03T08:36:32.425 INFO:teuthology.orchestra.run.smithi088.stderr:2024-06-03T08:36:32.422+0000 7f9bafd56640 20 client.4326.objecter put_session s=0x7f9b8c073aa0 osd=6 3 2024-06-03T08:36:32.425 INFO:teuthology.orchestra.run.smithi088.stderr:2024-06-03T08:36:32.422+0000 7f9bafd56640 20 client.4326.objecter _get_session s=0x7f9b8c073aa0 osd=6 3 2024-06-03T08:36:32.425 INFO:teuthology.orchestra.run.smithi088.stderr:2024-06-03T08:36:32.422+0000 7f9bafd56640 15 client.4326.objecter _session_command_op_remove -1 2 2024-06-03T08:36:32.425 INFO:teuthology.orchestra.run.smithi088.stderr:2024-06-03T08:36:32.422+0000 7f9bafd56640 20 client.4326.objecter get_session s=0x7f9b8c073aa0 osd=6 3 2024-06-03T08:36:32.425 INFO:teuthology.orchestra.run.smithi088.stderr:2024-06-03T08:36:32.422+0000 7f9bafd56640 15 client.4326.objecter _session_command_op_assign 6 2 2024-06-03T08:36:32.425 INFO:teuthology.orchestra.run.smithi088.stderr:2024-06-03T08:36:32.422+0000 7f9bafd56640 20 client.4326.objecter put_session s=0x7f9b8c073aa0 osd=6 4 2024-06-03T08:36:32.425 INFO:teuthology.orchestra.run.smithi088.stderr:2024-06-03T08:36:32.422+0000 7f9bafd56640 10 client.4326.objecter _send_command 2 2024-06-03T08:36:32.425 INFO:teuthology.orchestra.run.smithi088.stderr:2024-06-03T08:36:32.422+0000 7f7916d9d640 1 -- 172.21.15.88:0/996211931 reap_dead start 2024-06-03T08:36:32.426 INFO:teuthology.orchestra.run.smithi088.stderr:2024-06-03T08:36:32.422+0000 7f7904ff9640 1 -- 172.21.15.88:0/996211931 shutdown_connections 2024-06-03T08:36:32.426 INFO:teuthology.orchestra.run.smithi088.stderr:2024-06-03T08:36:32.422+0000 7f9bafd56640 1 -- 172.21.15.88:0/496575353 --> v1:172.21.15.192:6813/34788 -- command(tid 2: {"prefix": "flush_pg_stats"}) v1 -- 0x7f9ba804ea60 con 0x7f9b8c074150 2024-06-03T08:36:32.426 INFO:teuthology.orchestra.run.smithi088.stderr:2024-06-03T08:36:32.422+0000 7f7904ff9640 1 -- 172.21.15.88:0/996211931 >> 172.21.15.88:0/996211931 conn(0x7f79180719d0 msgr2=0x7f79180b1a70 unknown :-1 s=STATE_NONE l=0).mark_down 2024-06-03T08:36:32.427 INFO:teuthology.orchestra.run.smithi088.stderr:2024-06-03T08:36:32.422+0000 7f7904ff9640 1 -- 172.21.15.88:0/996211931 shutdown_connections 2024-06-03T08:36:32.427 INFO:teuthology.orchestra.run.smithi088.stderr:2024-06-03T08:36:32.422+0000 7f7904ff9640 1 -- 172.21.15.88:0/996211931 wait complete. 2024-06-03T08:36:32.427 INFO:teuthology.orchestra.run.smithi088.stderr:2024-06-03T08:36:32.423+0000 7f7904ff9640 1 librados: shutdown 2024-06-03T08:36:32.428 INFO:teuthology.orchestra.run.smithi088.stderr:2024-06-03T08:36:32.423+0000 7fa0f1016640 10 client.4333.objecter _submit_command 2 [{"prefix": "flush_pg_stats"}] 2024-06-03T08:36:32.428 INFO:teuthology.orchestra.run.smithi088.stderr:2024-06-03T08:36:32.423+0000 7fa0f1016640 15 client.4333.objecter _session_command_op_assign -1 2 2024-06-03T08:36:32.428 INFO:teuthology.orchestra.run.smithi088.stderr:2024-06-03T08:36:32.423+0000 7fa0f1016640 20 client.4333.objecter _get_session s=0x7fa0d8073ac0 osd=5 3 2024-06-03T08:36:32.428 INFO:teuthology.orchestra.run.smithi088.stderr:2024-06-03T08:36:32.423+0000 7fa0f1016640 20 client.4333.objecter put_session s=0x7fa0d8073ac0 osd=5 3 2024-06-03T08:36:32.428 INFO:teuthology.orchestra.run.smithi088.stderr:2024-06-03T08:36:32.423+0000 7fa0f1016640 20 client.4333.objecter _get_session s=0x7fa0d8073ac0 osd=5 3 2024-06-03T08:36:32.428 INFO:teuthology.orchestra.run.smithi088.stderr:2024-06-03T08:36:32.423+0000 7fa0f1016640 15 client.4333.objecter _session_command_op_remove -1 2 2024-06-03T08:36:32.428 INFO:teuthology.orchestra.run.smithi088.stderr:2024-06-03T08:36:32.423+0000 7fa0f1016640 20 client.4333.objecter get_session s=0x7fa0d8073ac0 osd=5 3 2024-06-03T08:36:32.428 INFO:teuthology.orchestra.run.smithi088.stderr:2024-06-03T08:36:32.423+0000 7fa0f1016640 15 client.4333.objecter _session_command_op_assign 5 2 2024-06-03T08:36:32.428 INFO:teuthology.orchestra.run.smithi088.stderr:2024-06-03T08:36:32.423+0000 7fa0f1016640 20 client.4333.objecter put_session s=0x7fa0d8073ac0 osd=5 4 2024-06-03T08:36:32.429 INFO:teuthology.orchestra.run.smithi088.stderr:2024-06-03T08:36:32.423+0000 7fa0f1016640 10 client.4333.objecter _send_command 2 2024-06-03T08:36:32.429 INFO:teuthology.orchestra.run.smithi088.stderr:2024-06-03T08:36:32.423+0000 7fa0f1016640 1 -- 172.21.15.88:0/1282623662 --> v1:172.21.15.192:6805/34786 -- command(tid 2: {"prefix": "flush_pg_stats"}) v1 -- 0x7fa0ec141190 con 0x7fa0d8074170 2024-06-03T08:36:32.430 INFO:teuthology.orchestra.run.smithi088.stderr:2024-06-03T08:36:32.430+0000 7fd649ffb640 1 -- 172.21.15.88:0/2839181686 <== osd.0 v1:172.21.15.88:6808/35066 2 ==== command_reply(tid 2: 0 ) v1 ==== 8+0+11 (unknown 0 0 191306902) 0x7fd65404ea60 con 0x7fd638074350 2024-06-03T08:36:32.430 INFO:teuthology.orchestra.run.smithi088.stderr:2024-06-03T08:36:32.430+0000 7fd649ffb640 10 client.4320.objecter ms_dispatch 0x7fd654000b90 command_reply(tid 2: 0 ) v1 2024-06-03T08:36:32.430 INFO:teuthology.orchestra.run.smithi088.stderr:2024-06-03T08:36:32.430+0000 7fd649ffb640 10 client.4320.objecter _finish_command 2 = system:0 2024-06-03T08:36:32.431 INFO:teuthology.orchestra.run.smithi088.stderr:2024-06-03T08:36:32.430+0000 7fd649ffb640 20 client.4320.objecter put_session s=0x7fd638073ca0 osd=0 4 2024-06-03T08:36:32.431 INFO:teuthology.orchestra.run.smithi088.stderr:2024-06-03T08:36:32.430+0000 7fd649ffb640 15 client.4320.objecter _session_command_op_remove 0 2 2024-06-03T08:36:32.432 INFO:teuthology.orchestra.run.smithi088.stderr:2024-06-03T08:36:32.430+0000 7fd65b934640 10 librados: watch_flush enter 2024-06-03T08:36:32.432 INFO:teuthology.orchestra.run.smithi088.stderr:2024-06-03T08:36:32.430+0000 7fd65b934640 10 librados: watch_flush exit 2024-06-03T08:36:32.432 INFO:teuthology.orchestra.run.smithi088.stderr:2024-06-03T08:36:32.430+0000 7fd65b934640 10 client.4320.objecter close_session for osd.0 2024-06-03T08:36:32.432 INFO:teuthology.orchestra.run.smithi088.stderr:2024-06-03T08:36:32.430+0000 7fd65b934640 1 -- 172.21.15.88:0/2839181686 >> v1:172.21.15.88:6808/35066 conn(0x7fd638074350 legacy=0x7fd638076770 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-06-03T08:36:32.433 INFO:teuthology.orchestra.run.smithi088.stderr:2024-06-03T08:36:32.430+0000 7fd65b934640 20 client.4320.objecter put_session s=0x7fd638073ca0 osd=0 1 2024-06-03T08:36:32.433 INFO:teuthology.orchestra.run.smithi088.stderr:2024-06-03T08:36:32.430+0000 7fd65b934640 20 client.4320.objecter shutdown clearing up homeless session... 2024-06-03T08:36:32.433 INFO:teuthology.orchestra.run.smithi088.stderr:2024-06-03T08:36:32.430+0000 7fd65b934640 10 client.4320.objecter successfully canceled tick 2024-06-03T08:36:32.433 INFO:teuthology.orchestra.run.smithi088.stderr:2024-06-03T08:36:32.431+0000 7fd65b934640 1 -- 172.21.15.88:0/2839181686 >> v1:172.21.15.192:6800/34229 conn(0x7fd63806d340 legacy=0x7fd63806f800 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-06-03T08:36:32.433 INFO:teuthology.orchestra.run.smithi088.stderr:2024-06-03T08:36:32.431+0000 7fd65b934640 10 monclient: shutdown 2024-06-03T08:36:32.433 INFO:teuthology.orchestra.run.smithi088.stderr:2024-06-03T08:36:32.431+0000 7fd65b934640 20 monclient: shutdown discarding 0 pending message(s) 2024-06-03T08:36:32.433 INFO:teuthology.orchestra.run.smithi088.stderr:2024-06-03T08:36:32.431+0000 7fd65b934640 1 -- 172.21.15.88:0/2839181686 >> v1:172.21.15.88:6789/0 conn(0x7fd65406dc10 legacy=0x7fd65412d770 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-06-03T08:36:32.433 INFO:teuthology.orchestra.run.smithi088.stderr:2024-06-03T08:36:32.431+0000 7fd659ed1640 1 -- 172.21.15.88:0/2839181686 reap_dead start 2024-06-03T08:36:32.433 INFO:teuthology.orchestra.run.smithi088.stderr:2024-06-03T08:36:32.431+0000 7fd65b934640 1 -- 172.21.15.88:0/2839181686 shutdown_connections 2024-06-03T08:36:32.433 INFO:teuthology.orchestra.run.smithi088.stderr:2024-06-03T08:36:32.431+0000 7fd65b934640 1 -- 172.21.15.88:0/2839181686 >> 172.21.15.88:0/2839181686 conn(0x7fd6540b2f20 msgr2=0x7fd6540aee30 unknown :-1 s=STATE_NONE l=0).mark_down 2024-06-03T08:36:32.433 INFO:teuthology.orchestra.run.smithi088.stderr:2024-06-03T08:36:32.431+0000 7fd65b934640 1 -- 172.21.15.88:0/2839181686 shutdown_connections 2024-06-03T08:36:32.433 INFO:teuthology.orchestra.run.smithi088.stderr:2024-06-03T08:36:32.431+0000 7fd65b934640 1 -- 172.21.15.88:0/2839181686 wait complete. 2024-06-03T08:36:32.434 INFO:teuthology.orchestra.run.smithi088.stderr:2024-06-03T08:36:32.431+0000 7fd65b934640 1 librados: shutdown 2024-06-03T08:36:32.434 INFO:teuthology.orchestra.run.smithi088.stdout:85899345928 2024-06-03T08:36:32.435 DEBUG:teuthology.orchestra.run.smithi088:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd last-stat-seq osd.7 2024-06-03T08:36:32.438 INFO:teuthology.orchestra.run.smithi088.stderr:2024-06-03T08:36:32.438+0000 7f9b9e7fc640 1 -- 172.21.15.88:0/496575353 <== osd.6 v1:172.21.15.192:6813/34788 2 ==== command_reply(tid 2: 0 ) v1 ==== 8+0+11 (unknown 0 0 2365829177) 0x7f9ba804ea60 con 0x7f9b8c074150 2024-06-03T08:36:32.439 INFO:teuthology.orchestra.run.smithi088.stderr:2024-06-03T08:36:32.438+0000 7f9b9e7fc640 10 client.4326.objecter ms_dispatch 0x7f9ba8000b90 command_reply(tid 2: 0 ) v1 2024-06-03T08:36:32.439 INFO:teuthology.orchestra.run.smithi088.stderr:2024-06-03T08:36:32.438+0000 7f9b9e7fc640 10 client.4326.objecter _finish_command 2 = system:0 2024-06-03T08:36:32.439 INFO:teuthology.orchestra.run.smithi088.stderr:2024-06-03T08:36:32.438+0000 7f9b9e7fc640 20 client.4326.objecter put_session s=0x7f9b8c073aa0 osd=6 4 2024-06-03T08:36:32.439 INFO:teuthology.orchestra.run.smithi088.stderr:2024-06-03T08:36:32.438+0000 7f9b9e7fc640 15 client.4326.objecter _session_command_op_remove 6 2 2024-06-03T08:36:32.440 INFO:teuthology.orchestra.run.smithi088.stderr:2024-06-03T08:36:32.438+0000 7f9bafd56640 10 librados: watch_flush enter 2024-06-03T08:36:32.440 INFO:teuthology.orchestra.run.smithi088.stderr:2024-06-03T08:36:32.438+0000 7f9bafd56640 10 librados: watch_flush exit 2024-06-03T08:36:32.440 INFO:teuthology.orchestra.run.smithi088.stderr:2024-06-03T08:36:32.438+0000 7f9bafd56640 10 client.4326.objecter close_session for osd.6 2024-06-03T08:36:32.440 INFO:teuthology.orchestra.run.smithi088.stderr:2024-06-03T08:36:32.438+0000 7f9bafd56640 1 -- 172.21.15.88:0/496575353 >> v1:172.21.15.192:6813/34788 conn(0x7f9b8c074150 legacy=0x7f9b8c076570 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-06-03T08:36:32.441 INFO:teuthology.orchestra.run.smithi088.stderr:2024-06-03T08:36:32.438+0000 7f9bafd56640 20 client.4326.objecter put_session s=0x7f9b8c073aa0 osd=6 1 2024-06-03T08:36:32.441 INFO:teuthology.orchestra.run.smithi088.stderr:2024-06-03T08:36:32.438+0000 7f9bafd56640 20 client.4326.objecter shutdown clearing up homeless session... 2024-06-03T08:36:32.441 INFO:teuthology.orchestra.run.smithi088.stderr:2024-06-03T08:36:32.438+0000 7f9bafd56640 10 client.4326.objecter successfully canceled tick 2024-06-03T08:36:32.441 INFO:teuthology.orchestra.run.smithi088.stderr:2024-06-03T08:36:32.438+0000 7f9bafd56640 1 -- 172.21.15.88:0/496575353 >> v1:172.21.15.192:6800/34229 conn(0x7f9b8c06d120 legacy=0x7f9b8c06f5e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-06-03T08:36:32.441 INFO:teuthology.orchestra.run.smithi088.stderr:2024-06-03T08:36:32.438+0000 7f9bafd56640 10 monclient: shutdown 2024-06-03T08:36:32.441 INFO:teuthology.orchestra.run.smithi088.stderr:2024-06-03T08:36:32.438+0000 7f9bafd56640 20 monclient: shutdown discarding 0 pending message(s) 2024-06-03T08:36:32.441 INFO:teuthology.orchestra.run.smithi088.stderr:2024-06-03T08:36:32.438+0000 7f9bafd56640 1 -- 172.21.15.88:0/496575353 >> v1:172.21.15.88:6789/0 conn(0x7f9ba80b43c0 legacy=0x7f9ba81da270 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-06-03T08:36:32.441 INFO:teuthology.orchestra.run.smithi088.stderr:2024-06-03T08:36:32.439+0000 7f9bae2f3640 1 -- 172.21.15.88:0/496575353 reap_dead start 2024-06-03T08:36:32.441 INFO:teuthology.orchestra.run.smithi088.stderr:2024-06-03T08:36:32.439+0000 7f9bafd56640 1 -- 172.21.15.88:0/496575353 shutdown_connections 2024-06-03T08:36:32.441 INFO:teuthology.orchestra.run.smithi088.stderr:2024-06-03T08:36:32.439+0000 7f9bafd56640 1 -- 172.21.15.88:0/496575353 >> 172.21.15.88:0/496575353 conn(0x7f9ba80719d0 msgr2=0x7f9ba813cef0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-06-03T08:36:32.442 INFO:teuthology.orchestra.run.smithi088.stderr:2024-06-03T08:36:32.439+0000 7fd3367fc640 1 -- 172.21.15.88:0/3500826820 <== osd.4 v1:172.21.15.192:6809/34785 1 ==== command_reply(tid 1: 0 ) v1 ==== 8+0+26118 (unknown 0 0 1686565167) 0x7fd324077d60 con 0x7fd324074190 2024-06-03T08:36:32.442 INFO:teuthology.orchestra.run.smithi088.stderr:2024-06-03T08:36:32.439+0000 7fd3367fc640 10 client.4332.objecter ms_dispatch 0x7fd340000b90 command_reply(tid 1: 0 ) v1 2024-06-03T08:36:32.442 INFO:teuthology.orchestra.run.smithi088.stderr:2024-06-03T08:36:32.439+0000 7fd3367fc640 10 client.4332.objecter _finish_command 1 = system:0 2024-06-03T08:36:32.442 INFO:teuthology.orchestra.run.smithi088.stderr:2024-06-03T08:36:32.439+0000 7fd3367fc640 20 client.4332.objecter put_session s=0x7fd324073ae0 osd=4 4 2024-06-03T08:36:32.442 INFO:teuthology.orchestra.run.smithi088.stderr:2024-06-03T08:36:32.439+0000 7fd3367fc640 15 client.4332.objecter _session_command_op_remove 4 1 2024-06-03T08:36:32.443 INFO:teuthology.orchestra.run.smithi088.stderr:2024-06-03T08:36:32.439+0000 7f9bafd56640 1 -- 172.21.15.88:0/496575353 shutdown_connections 2024-06-03T08:36:32.443 INFO:teuthology.orchestra.run.smithi088.stderr:2024-06-03T08:36:32.439+0000 7f9bafd56640 1 -- 172.21.15.88:0/496575353 wait complete. 2024-06-03T08:36:32.443 INFO:teuthology.orchestra.run.smithi088.stderr:2024-06-03T08:36:32.439+0000 7f9bafd56640 1 librados: shutdown 2024-06-03T08:36:32.444 INFO:teuthology.orchestra.run.smithi088.stdout:60129542150 2024-06-03T08:36:32.444 DEBUG:teuthology.orchestra.run.smithi088:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd last-stat-seq osd.0 2024-06-03T08:36:32.446 INFO:teuthology.orchestra.run.smithi088.stderr:2024-06-03T08:36:32.444+0000 7f4c36155640 10 client.4323.objecter _submit_command 2 [{"prefix": "flush_pg_stats"}] 2024-06-03T08:36:32.446 INFO:teuthology.orchestra.run.smithi088.stderr:2024-06-03T08:36:32.444+0000 7f4c36155640 15 client.4323.objecter _session_command_op_assign -1 2 2024-06-03T08:36:32.446 INFO:teuthology.orchestra.run.smithi088.stderr:2024-06-03T08:36:32.444+0000 7f4c36155640 20 client.4323.objecter _get_session s=0x7f4c14073b80 osd=1 3 2024-06-03T08:36:32.446 INFO:teuthology.orchestra.run.smithi088.stderr:2024-06-03T08:36:32.444+0000 7f4c36155640 20 client.4323.objecter put_session s=0x7f4c14073b80 osd=1 3 2024-06-03T08:36:32.446 INFO:teuthology.orchestra.run.smithi088.stderr:2024-06-03T08:36:32.444+0000 7f4c36155640 20 client.4323.objecter _get_session s=0x7f4c14073b80 osd=1 3 2024-06-03T08:36:32.446 INFO:teuthology.orchestra.run.smithi088.stderr:2024-06-03T08:36:32.444+0000 7f4c36155640 15 client.4323.objecter _session_command_op_remove -1 2 2024-06-03T08:36:32.446 INFO:teuthology.orchestra.run.smithi088.stderr:2024-06-03T08:36:32.444+0000 7f4c36155640 20 client.4323.objecter get_session s=0x7f4c14073b80 osd=1 3 2024-06-03T08:36:32.446 INFO:teuthology.orchestra.run.smithi088.stderr:2024-06-03T08:36:32.444+0000 7f4c36155640 15 client.4323.objecter _session_command_op_assign 1 2 2024-06-03T08:36:32.446 INFO:teuthology.orchestra.run.smithi088.stderr:2024-06-03T08:36:32.444+0000 7f4c36155640 20 client.4323.objecter put_session s=0x7f4c14073b80 osd=1 4 2024-06-03T08:36:32.446 INFO:teuthology.orchestra.run.smithi088.stderr:2024-06-03T08:36:32.444+0000 7f4c36155640 10 client.4323.objecter _send_command 2 2024-06-03T08:36:32.447 INFO:teuthology.orchestra.run.smithi088.stderr:2024-06-03T08:36:32.444+0000 7f4c36155640 1 -- 172.21.15.88:0/2099487910 --> v1:172.21.15.88:6800/35067 -- command(tid 2: {"prefix": "flush_pg_stats"}) v1 -- 0x7f4c300674b0 con 0x7f4c14074230 2024-06-03T08:36:32.450 INFO:teuthology.orchestra.run.smithi088.stderr:2024-06-03T08:36:32.449+0000 7fa0d37fe640 1 -- 172.21.15.88:0/1282623662 <== osd.5 v1:172.21.15.192:6805/34786 2 ==== command_reply(tid 2: 0 ) v1 ==== 8+0+11 (unknown 0 0 3332848392) 0x7fa0ec141190 con 0x7fa0d8074170 2024-06-03T08:36:32.450 INFO:teuthology.orchestra.run.smithi088.stderr:2024-06-03T08:36:32.449+0000 7fa0d37fe640 10 client.4333.objecter ms_dispatch 0x7fa0ec000b90 command_reply(tid 2: 0 ) v1 2024-06-03T08:36:32.450 INFO:teuthology.orchestra.run.smithi088.stderr:2024-06-03T08:36:32.449+0000 7fa0d37fe640 10 client.4333.objecter _finish_command 2 = system:0 2024-06-03T08:36:32.450 INFO:teuthology.orchestra.run.smithi088.stderr:2024-06-03T08:36:32.449+0000 7fa0d37fe640 20 client.4333.objecter put_session s=0x7fa0d8073ac0 osd=5 4 2024-06-03T08:36:32.450 INFO:teuthology.orchestra.run.smithi088.stderr:2024-06-03T08:36:32.449+0000 7fa0d37fe640 15 client.4333.objecter _session_command_op_remove 5 2 2024-06-03T08:36:32.451 INFO:teuthology.orchestra.run.smithi088.stderr:2024-06-03T08:36:32.449+0000 7fa0f1016640 10 librados: watch_flush enter 2024-06-03T08:36:32.451 INFO:teuthology.orchestra.run.smithi088.stderr:2024-06-03T08:36:32.449+0000 7fa0f1016640 10 librados: watch_flush exit 2024-06-03T08:36:32.451 INFO:teuthology.orchestra.run.smithi088.stderr:2024-06-03T08:36:32.449+0000 7fa0f1016640 10 client.4333.objecter close_session for osd.5 2024-06-03T08:36:32.452 INFO:teuthology.orchestra.run.smithi088.stderr:2024-06-03T08:36:32.449+0000 7fa0f1016640 1 -- 172.21.15.88:0/1282623662 >> v1:172.21.15.192:6805/34786 conn(0x7fa0d8074170 legacy=0x7fa0d8076590 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-06-03T08:36:32.452 INFO:teuthology.orchestra.run.smithi088.stderr:2024-06-03T08:36:32.449+0000 7fa0f1016640 20 client.4333.objecter put_session s=0x7fa0d8073ac0 osd=5 1 2024-06-03T08:36:32.452 INFO:teuthology.orchestra.run.smithi088.stderr:2024-06-03T08:36:32.449+0000 7fa0f1016640 20 client.4333.objecter shutdown clearing up homeless session... 2024-06-03T08:36:32.452 INFO:teuthology.orchestra.run.smithi088.stderr:2024-06-03T08:36:32.449+0000 7fa0f1016640 10 client.4333.objecter successfully canceled tick 2024-06-03T08:36:32.452 INFO:teuthology.orchestra.run.smithi088.stderr:2024-06-03T08:36:32.449+0000 7fa0f1016640 1 -- 172.21.15.88:0/1282623662 >> v1:172.21.15.192:6800/34229 conn(0x7fa0d806d0d0 legacy=0x7fa0d806f590 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-06-03T08:36:32.452 INFO:teuthology.orchestra.run.smithi088.stderr:2024-06-03T08:36:32.449+0000 7fa0f1016640 10 monclient: shutdown 2024-06-03T08:36:32.452 INFO:teuthology.orchestra.run.smithi088.stderr:2024-06-03T08:36:32.449+0000 7fa0f1016640 20 monclient: shutdown discarding 0 pending message(s) 2024-06-03T08:36:32.452 INFO:teuthology.orchestra.run.smithi088.stderr:2024-06-03T08:36:32.449+0000 7fa0f1016640 1 -- 172.21.15.88:0/1282623662 >> v1:172.21.15.192:6789/0 conn(0x7fa0ec1450f0 legacy=0x7fa0ec1e1f30 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-06-03T08:36:32.452 INFO:teuthology.orchestra.run.smithi088.stderr:2024-06-03T08:36:32.449+0000 7fa0eb59e640 1 -- 172.21.15.88:0/1282623662 reap_dead start 2024-06-03T08:36:32.452 INFO:teuthology.orchestra.run.smithi088.stderr:2024-06-03T08:36:32.449+0000 7fa0f1016640 1 -- 172.21.15.88:0/1282623662 shutdown_connections 2024-06-03T08:36:32.452 INFO:teuthology.orchestra.run.smithi088.stderr:2024-06-03T08:36:32.449+0000 7fa0f1016640 1 -- 172.21.15.88:0/1282623662 >> 172.21.15.88:0/1282623662 conn(0x7fa0ec0b70d0 msgr2=0x7fa0ec13b860 unknown :-1 s=STATE_NONE l=0).mark_down 2024-06-03T08:36:32.452 INFO:teuthology.orchestra.run.smithi088.stderr:2024-06-03T08:36:32.449+0000 7fa0f1016640 1 -- 172.21.15.88:0/1282623662 shutdown_connections 2024-06-03T08:36:32.452 INFO:teuthology.orchestra.run.smithi088.stderr:2024-06-03T08:36:32.450+0000 7fa0f1016640 1 -- 172.21.15.88:0/1282623662 wait complete. 2024-06-03T08:36:32.453 INFO:teuthology.orchestra.run.smithi088.stderr:2024-06-03T08:36:32.450+0000 7fa0f1016640 1 librados: shutdown 2024-06-03T08:36:32.453 INFO:teuthology.orchestra.run.smithi088.stderr:2024-06-03T08:36:32.451+0000 7f4c1bfff640 1 -- 172.21.15.88:0/2099487910 <== osd.1 v1:172.21.15.88:6800/35067 2 ==== command_reply(tid 2: 0 ) v1 ==== 8+0+11 (unknown 0 0 3931925601) 0x7f4c300674b0 con 0x7f4c14074230 2024-06-03T08:36:32.454 INFO:teuthology.orchestra.run.smithi088.stderr:2024-06-03T08:36:32.451+0000 7f4c1bfff640 10 client.4323.objecter ms_dispatch 0x7f4c30000b90 command_reply(tid 2: 0 ) v1 2024-06-03T08:36:32.454 INFO:teuthology.orchestra.run.smithi088.stderr:2024-06-03T08:36:32.451+0000 7f4c1bfff640 10 client.4323.objecter _finish_command 2 = system:0 2024-06-03T08:36:32.454 INFO:teuthology.orchestra.run.smithi088.stderr:2024-06-03T08:36:32.451+0000 7f4c1bfff640 20 client.4323.objecter put_session s=0x7f4c14073b80 osd=1 4 2024-06-03T08:36:32.454 INFO:teuthology.orchestra.run.smithi088.stderr:2024-06-03T08:36:32.451+0000 7f4c1bfff640 15 client.4323.objecter _session_command_op_remove 1 2 2024-06-03T08:36:32.454 INFO:teuthology.orchestra.run.smithi088.stderr:2024-06-03T08:36:32.452+0000 7f4c36155640 10 librados: watch_flush enter 2024-06-03T08:36:32.454 INFO:teuthology.orchestra.run.smithi088.stderr:2024-06-03T08:36:32.452+0000 7f4c36155640 10 librados: watch_flush exit 2024-06-03T08:36:32.454 INFO:teuthology.orchestra.run.smithi088.stderr:2024-06-03T08:36:32.452+0000 7f4c36155640 10 client.4323.objecter close_session for osd.1 2024-06-03T08:36:32.455 INFO:teuthology.orchestra.run.smithi088.stderr:2024-06-03T08:36:32.452+0000 7f4c36155640 1 -- 172.21.15.88:0/2099487910 >> v1:172.21.15.88:6800/35067 conn(0x7f4c14074230 legacy=0x7f4c14076650 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-06-03T08:36:32.455 INFO:teuthology.orchestra.run.smithi088.stderr:2024-06-03T08:36:32.452+0000 7f4c36155640 20 client.4323.objecter put_session s=0x7f4c14073b80 osd=1 1 2024-06-03T08:36:32.455 INFO:teuthology.orchestra.run.smithi088.stderr:2024-06-03T08:36:32.452+0000 7f4c36155640 20 client.4323.objecter shutdown clearing up homeless session... 2024-06-03T08:36:32.455 INFO:teuthology.orchestra.run.smithi088.stderr:2024-06-03T08:36:32.452+0000 7f4c36155640 10 client.4323.objecter successfully canceled tick 2024-06-03T08:36:32.455 INFO:teuthology.orchestra.run.smithi088.stdout:90194313226 2024-06-03T08:36:32.456 DEBUG:teuthology.orchestra.run.smithi088:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd last-stat-seq osd.6 2024-06-03T08:36:32.457 INFO:teuthology.orchestra.run.smithi088.stderr:2024-06-03T08:36:32.453+0000 7f4c36155640 1 -- 172.21.15.88:0/2099487910 >> v1:172.21.15.192:6800/34229 conn(0x7f4c1406d200 legacy=0x7f4c1406f6c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-06-03T08:36:32.457 INFO:teuthology.orchestra.run.smithi088.stderr:2024-06-03T08:36:32.453+0000 7f4c36155640 10 monclient: shutdown 2024-06-03T08:36:32.457 INFO:teuthology.orchestra.run.smithi088.stderr:2024-06-03T08:36:32.453+0000 7f4c36155640 20 monclient: shutdown discarding 0 pending message(s) 2024-06-03T08:36:32.457 INFO:teuthology.orchestra.run.smithi088.stderr:2024-06-03T08:36:32.453+0000 7f4c36155640 1 -- 172.21.15.88:0/2099487910 >> v1:172.21.15.88:6789/0 conn(0x7f4c3013ee50 legacy=0x7f4c301ccb00 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-06-03T08:36:32.457 INFO:teuthology.orchestra.run.smithi088.stderr:2024-06-03T08:36:32.453+0000 7f4c2ffff640 1 -- 172.21.15.88:0/2099487910 reap_dead start 2024-06-03T08:36:32.457 INFO:teuthology.orchestra.run.smithi088.stderr:2024-06-03T08:36:32.454+0000 7f4c36155640 1 -- 172.21.15.88:0/2099487910 shutdown_connections 2024-06-03T08:36:32.457 INFO:teuthology.orchestra.run.smithi088.stderr:2024-06-03T08:36:32.454+0000 7f4c36155640 1 -- 172.21.15.88:0/2099487910 >> 172.21.15.88:0/2099487910 conn(0x7f4c300717c0 msgr2=0x7f4c3013b940 unknown :-1 s=STATE_NONE l=0).mark_down 2024-06-03T08:36:32.457 INFO:teuthology.orchestra.run.smithi088.stderr:2024-06-03T08:36:32.454+0000 7f4c36155640 1 -- 172.21.15.88:0/2099487910 shutdown_connections 2024-06-03T08:36:32.457 INFO:teuthology.orchestra.run.smithi088.stderr:2024-06-03T08:36:32.454+0000 7f4c36155640 1 -- 172.21.15.88:0/2099487910 wait complete. 2024-06-03T08:36:32.457 INFO:teuthology.orchestra.run.smithi088.stderr:2024-06-03T08:36:32.455+0000 7f4c36155640 1 librados: shutdown 2024-06-03T08:36:32.460 INFO:teuthology.orchestra.run.smithi088.stderr:2024-06-03T08:36:32.459+0000 7fd347fb9640 10 client.4332.objecter _submit_command 2 [{"prefix": "flush_pg_stats"}] 2024-06-03T08:36:32.460 INFO:teuthology.orchestra.run.smithi088.stderr:2024-06-03T08:36:32.459+0000 7fd347fb9640 15 client.4332.objecter _session_command_op_assign -1 2 2024-06-03T08:36:32.460 INFO:teuthology.orchestra.run.smithi088.stderr:2024-06-03T08:36:32.459+0000 7fd347fb9640 20 client.4332.objecter _get_session s=0x7fd324073ae0 osd=4 3 2024-06-03T08:36:32.460 INFO:teuthology.orchestra.run.smithi088.stderr:2024-06-03T08:36:32.459+0000 7fd347fb9640 20 client.4332.objecter put_session s=0x7fd324073ae0 osd=4 3 2024-06-03T08:36:32.460 INFO:teuthology.orchestra.run.smithi088.stderr:2024-06-03T08:36:32.459+0000 7fd347fb9640 20 client.4332.objecter _get_session s=0x7fd324073ae0 osd=4 3 2024-06-03T08:36:32.460 INFO:teuthology.orchestra.run.smithi088.stderr:2024-06-03T08:36:32.459+0000 7fd347fb9640 15 client.4332.objecter _session_command_op_remove -1 2 2024-06-03T08:36:32.460 INFO:teuthology.orchestra.run.smithi088.stderr:2024-06-03T08:36:32.459+0000 7fd347fb9640 20 client.4332.objecter get_session s=0x7fd324073ae0 osd=4 3 2024-06-03T08:36:32.460 INFO:teuthology.orchestra.run.smithi088.stderr:2024-06-03T08:36:32.459+0000 7fd347fb9640 15 client.4332.objecter _session_command_op_assign 4 2 2024-06-03T08:36:32.460 INFO:teuthology.orchestra.run.smithi088.stderr:2024-06-03T08:36:32.459+0000 7fd347fb9640 20 client.4332.objecter put_session s=0x7fd324073ae0 osd=4 4 2024-06-03T08:36:32.461 INFO:teuthology.orchestra.run.smithi088.stderr:2024-06-03T08:36:32.459+0000 7fd347fb9640 10 client.4332.objecter _send_command 2 2024-06-03T08:36:32.461 INFO:teuthology.orchestra.run.smithi088.stderr:2024-06-03T08:36:32.459+0000 7fd347fb9640 1 -- 172.21.15.88:0/3500826820 --> v1:172.21.15.192:6809/34785 -- command(tid 2: {"prefix": "flush_pg_stats"}) v1 -- 0x7fd338034b20 con 0x7fd324074190 2024-06-03T08:36:32.467 INFO:teuthology.orchestra.run.smithi088.stdout:85899345930 2024-06-03T08:36:32.467 DEBUG:teuthology.orchestra.run.smithi088:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd last-stat-seq osd.5 2024-06-03T08:36:32.474 INFO:teuthology.orchestra.run.smithi088.stdout:60129542152 2024-06-03T08:36:32.474 DEBUG:teuthology.orchestra.run.smithi088:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd last-stat-seq osd.1 2024-06-03T08:36:32.502 INFO:teuthology.orchestra.run.smithi088.stderr:2024-06-03T08:36:32.502+0000 7fd3367fc640 1 -- 172.21.15.88:0/3500826820 <== osd.4 v1:172.21.15.192:6809/34785 2 ==== command_reply(tid 2: 0 ) v1 ==== 8+0+11 (unknown 0 0 1608317872) 0x7fd3400693d0 con 0x7fd324074190 2024-06-03T08:36:32.502 INFO:teuthology.orchestra.run.smithi088.stderr:2024-06-03T08:36:32.502+0000 7fd3367fc640 10 client.4332.objecter ms_dispatch 0x7fd340000b90 command_reply(tid 2: 0 ) v1 2024-06-03T08:36:32.502 INFO:teuthology.orchestra.run.smithi088.stderr:2024-06-03T08:36:32.502+0000 7fd3367fc640 10 client.4332.objecter _finish_command 2 = system:0 2024-06-03T08:36:32.503 INFO:teuthology.orchestra.run.smithi088.stderr:2024-06-03T08:36:32.502+0000 7fd3367fc640 20 client.4332.objecter put_session s=0x7fd324073ae0 osd=4 4 2024-06-03T08:36:32.503 INFO:teuthology.orchestra.run.smithi088.stderr:2024-06-03T08:36:32.502+0000 7fd3367fc640 15 client.4332.objecter _session_command_op_remove 4 2 2024-06-03T08:36:32.503 INFO:teuthology.orchestra.run.smithi088.stderr:2024-06-03T08:36:32.502+0000 7fd323fff640 10 librados: watch_flush enter 2024-06-03T08:36:32.503 INFO:teuthology.orchestra.run.smithi088.stderr:2024-06-03T08:36:32.503+0000 7fd323fff640 10 librados: watch_flush exit 2024-06-03T08:36:32.503 INFO:teuthology.orchestra.run.smithi088.stderr:2024-06-03T08:36:32.503+0000 7fd323fff640 10 client.4332.objecter close_session for osd.4 2024-06-03T08:36:32.504 INFO:teuthology.orchestra.run.smithi088.stderr:2024-06-03T08:36:32.503+0000 7fd323fff640 1 -- 172.21.15.88:0/3500826820 >> v1:172.21.15.192:6809/34785 conn(0x7fd324074190 legacy=0x7fd3240765b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-06-03T08:36:32.504 INFO:teuthology.orchestra.run.smithi088.stderr:2024-06-03T08:36:32.503+0000 7fd323fff640 20 client.4332.objecter put_session s=0x7fd324073ae0 osd=4 1 2024-06-03T08:36:32.504 INFO:teuthology.orchestra.run.smithi088.stderr:2024-06-03T08:36:32.503+0000 7fd323fff640 20 client.4332.objecter shutdown clearing up homeless session... 2024-06-03T08:36:32.504 INFO:teuthology.orchestra.run.smithi088.stderr:2024-06-03T08:36:32.503+0000 7fd323fff640 10 client.4332.objecter successfully canceled tick 2024-06-03T08:36:32.504 INFO:teuthology.orchestra.run.smithi088.stderr:2024-06-03T08:36:32.503+0000 7fd323fff640 1 -- 172.21.15.88:0/3500826820 >> v1:172.21.15.192:6800/34229 conn(0x7fd32406d0d0 legacy=0x7fd32406f590 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-06-03T08:36:32.504 INFO:teuthology.orchestra.run.smithi088.stderr:2024-06-03T08:36:32.503+0000 7fd323fff640 10 monclient: shutdown 2024-06-03T08:36:32.504 INFO:teuthology.orchestra.run.smithi088.stderr:2024-06-03T08:36:32.503+0000 7fd323fff640 20 monclient: shutdown discarding 0 pending message(s) 2024-06-03T08:36:32.504 INFO:teuthology.orchestra.run.smithi088.stderr:2024-06-03T08:36:32.503+0000 7fd323fff640 1 -- 172.21.15.88:0/3500826820 >> v1:172.21.15.88:6789/0 conn(0x7fd338006410 legacy=0x7fd3380afdb0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-06-03T08:36:32.504 INFO:teuthology.orchestra.run.smithi088.stderr:2024-06-03T08:36:32.503+0000 7fd346556640 1 -- 172.21.15.88:0/3500826820 reap_dead start 2024-06-03T08:36:32.505 INFO:teuthology.orchestra.run.smithi088.stderr:2024-06-03T08:36:32.503+0000 7fd323fff640 1 -- 172.21.15.88:0/3500826820 shutdown_connections 2024-06-03T08:36:32.505 INFO:teuthology.orchestra.run.smithi088.stderr:2024-06-03T08:36:32.503+0000 7fd323fff640 1 -- 172.21.15.88:0/3500826820 >> 172.21.15.88:0/3500826820 conn(0x7fd3380058f0 msgr2=0x7fd3380c52c0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-06-03T08:36:32.505 INFO:teuthology.orchestra.run.smithi088.stderr:2024-06-03T08:36:32.503+0000 7fd323fff640 1 -- 172.21.15.88:0/3500826820 shutdown_connections 2024-06-03T08:36:32.505 INFO:teuthology.orchestra.run.smithi088.stderr:2024-06-03T08:36:32.503+0000 7fd323fff640 1 -- 172.21.15.88:0/3500826820 wait complete. 2024-06-03T08:36:32.505 INFO:teuthology.orchestra.run.smithi088.stderr:2024-06-03T08:36:32.503+0000 7fd323fff640 1 librados: shutdown 2024-06-03T08:36:32.519 INFO:teuthology.orchestra.run.smithi088.stdout:85899345928 2024-06-03T08:36:32.519 DEBUG:teuthology.orchestra.run.smithi088:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd last-stat-seq osd.4 2024-06-03T08:36:32.539 INFO:teuthology.orchestra.run.smithi088.stderr:2024-06-03T08:36:32.538+0000 7fb0feffd640 1 -- 172.21.15.88:0/4168539993 <== mon.1 v1:172.21.15.192:6789/0 5 ==== config(0 keys) v1 ==== 4+0+0 (unknown 0 0 0) 0x7fb0f0001d00 con 0x7fb10813eec0 2024-06-03T08:36:32.539 INFO:teuthology.orchestra.run.smithi088.stderr:2024-06-03T08:36:32.539+0000 7fb0feffd640 10 monclient: handle_config config(0 keys) v1 2024-06-03T08:36:32.540 INFO:teuthology.orchestra.run.smithi088.stderr:2024-06-03T08:36:32.539+0000 7fb10559a640 4 set_mon_vals no callback set 2024-06-03T08:36:32.577 INFO:teuthology.orchestra.run.smithi088.stderr:2024-06-03T08:36:32.577+0000 7f224f31d640 10 monclient: get_monmap_and_config 2024-06-03T08:36:32.578 INFO:teuthology.orchestra.run.smithi088.stderr:2024-06-03T08:36:32.577+0000 7f224f31d640 10 monclient: build_initial_monmap 2024-06-03T08:36:32.578 INFO:teuthology.orchestra.run.smithi088.stderr:2024-06-03T08:36:32.577+0000 7f224f31d640 10 monclient: monmap: 2024-06-03T08:36:32.578 INFO:teuthology.orchestra.run.smithi088.stderr:epoch 0 2024-06-03T08:36:32.578 INFO:teuthology.orchestra.run.smithi088.stderr:fsid ec3c3fdd-3ac6-4d7d-9412-683130ed8b04 2024-06-03T08:36:32.578 INFO:teuthology.orchestra.run.smithi088.stderr:last_changed 2024-06-03T08:36:32.578437+0000 2024-06-03T08:36:32.578 INFO:teuthology.orchestra.run.smithi088.stderr:created 2024-06-03T08:36:32.578437+0000 2024-06-03T08:36:32.578 INFO:teuthology.orchestra.run.smithi088.stderr:min_mon_release 0 (unknown) 2024-06-03T08:36:32.578 INFO:teuthology.orchestra.run.smithi088.stderr:election_strategy: 3 2024-06-03T08:36:32.578 INFO:teuthology.orchestra.run.smithi088.stderr:0: v1:172.21.15.88:6789/0 mon.noname-a 2024-06-03T08:36:32.578 INFO:teuthology.orchestra.run.smithi088.stderr:1: v1:172.21.15.192:6789/0 mon.noname-c 2024-06-03T08:36:32.578 INFO:teuthology.orchestra.run.smithi088.stderr:2: v1:172.21.15.88:6790/0 mon.noname-b 2024-06-03T08:36:32.578 INFO:teuthology.orchestra.run.smithi088.stderr: 2024-06-03T08:36:32.579 INFO:teuthology.orchestra.run.smithi088.stderr:2024-06-03T08:36:32.577+0000 7fccb67fc640 1 -- 172.21.15.88:0/1456172716 <== mon.2 v1:172.21.15.88:6790/0 5 ==== config(0 keys) v1 ==== 4+0+0 (unknown 0 0 0) 0x7fccbc0047e0 con 0x7fccc00a8a60 2024-06-03T08:36:32.579 INFO:teuthology.orchestra.run.smithi088.stderr:2024-06-03T08:36:32.577+0000 7fccb67fc640 10 monclient: handle_config config(0 keys) v1 2024-06-03T08:36:32.579 INFO:teuthology.orchestra.run.smithi088.stderr:2024-06-03T08:36:32.577+0000 7fccb67fc640 1 -- 172.21.15.88:0/1456172716 <== mon.2 v1:172.21.15.88:6790/0 6 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 2234374065 0 0) 0x7fccbc004950 con 0x7fccc00a8a60 2024-06-03T08:36:32.579 INFO:teuthology.orchestra.run.smithi088.stderr:2024-06-03T08:36:32.577+0000 7fccb67fc640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-06-03T08:36:32.579 INFO:teuthology.orchestra.run.smithi088.stderr:2024-06-03T08:36:32.577+0000 7fccb67fc640 10 monclient: got monmap 1 from mon.c (according to old e1) 2024-06-03T08:36:32.580 INFO:teuthology.orchestra.run.smithi088.stderr:2024-06-03T08:36:32.577+0000 7fccb67fc640 10 monclient: dump: 2024-06-03T08:36:32.580 INFO:teuthology.orchestra.run.smithi088.stderr:epoch 1 2024-06-03T08:36:32.580 INFO:teuthology.orchestra.run.smithi088.stderr:fsid ec3c3fdd-3ac6-4d7d-9412-683130ed8b04 2024-06-03T08:36:32.580 INFO:teuthology.orchestra.run.smithi088.stderr:last_changed 2024-06-03T08:34:31.885125+0000 2024-06-03T08:36:32.580 INFO:teuthology.orchestra.run.smithi088.stderr:created 2024-06-03T08:34:31.885125+0000 2024-06-03T08:36:32.580 INFO:teuthology.orchestra.run.smithi088.stderr:min_mon_release 18 (reef) 2024-06-03T08:36:32.580 INFO:teuthology.orchestra.run.smithi088.stderr:election_strategy: 3 2024-06-03T08:36:32.580 INFO:teuthology.orchestra.run.smithi088.stderr:0: v1:172.21.15.88:6789/0 mon.a 2024-06-03T08:36:32.580 INFO:teuthology.orchestra.run.smithi088.stderr:1: v1:172.21.15.192:6789/0 mon.b 2024-06-03T08:36:32.580 INFO:teuthology.orchestra.run.smithi088.stderr:2: v1:172.21.15.88:6790/0 mon.c 2024-06-03T08:36:32.580 INFO:teuthology.orchestra.run.smithi088.stderr: 2024-06-03T08:36:32.580 INFO:teuthology.orchestra.run.smithi088.stderr:2024-06-03T08:36:32.577+0000 7fccb7fff640 4 set_mon_vals no callback set 2024-06-03T08:36:32.581 INFO:teuthology.orchestra.run.smithi088.stderr:2024-06-03T08:36:32.578+0000 7f224f31d640 1 Processor -- start 2024-06-03T08:36:32.581 INFO:teuthology.orchestra.run.smithi088.stderr:2024-06-03T08:36:32.579+0000 7f224f31d640 1 -- start start 2024-06-03T08:36:32.581 INFO:teuthology.orchestra.run.smithi088.stderr:2024-06-03T08:36:32.579+0000 7fccb67fc640 1 -- 172.21.15.88:0/1456172716 <== mon.2 v1:172.21.15.88:6790/0 7 ==== mgrmap(e 5) v1 ==== 89110+0+0 (unknown 873533973 0 0) 0x7fccbc004ab0 con 0x7fccc00a8a60 2024-06-03T08:36:32.582 INFO:teuthology.orchestra.run.smithi088.stderr:2024-06-03T08:36:32.579+0000 7f224f31d640 10 monclient: init 2024-06-03T08:36:32.582 INFO:teuthology.orchestra.run.smithi088.stderr:2024-06-03T08:36:32.580+0000 7f224f31d640 10 monclient: _reopen_session rank -1 2024-06-03T08:36:32.582 INFO:teuthology.orchestra.run.smithi088.stderr:2024-06-03T08:36:32.580+0000 7f224f31d640 10 monclient: _add_conns ranks=[2,1,0] 2024-06-03T08:36:32.582 INFO:teuthology.orchestra.run.smithi088.stderr:2024-06-03T08:36:32.580+0000 7f224f31d640 10 monclient(hunting): picked mon.noname-b con 0x7f22480b4250 addr v1:172.21.15.88:6790/0 2024-06-03T08:36:32.582 INFO:teuthology.orchestra.run.smithi088.stderr:2024-06-03T08:36:32.580+0000 7f224f31d640 10 monclient(hunting): picked mon.noname-c con 0x7f22480a89f0 addr v1:172.21.15.192:6789/0 2024-06-03T08:36:32.582 INFO:teuthology.orchestra.run.smithi088.stderr:2024-06-03T08:36:32.580+0000 7f224f31d640 10 monclient(hunting): picked mon.noname-a con 0x7f22481451c0 addr v1:172.21.15.88:6789/0 2024-06-03T08:36:32.582 INFO:teuthology.orchestra.run.smithi088.stderr:2024-06-03T08:36:32.580+0000 7f224f31d640 1 -- --> v1:172.21.15.88:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f22480aa6c0 con 0x7f22481451c0 2024-06-03T08:36:32.582 INFO:teuthology.orchestra.run.smithi088.stderr:2024-06-03T08:36:32.580+0000 7f224f31d640 1 -- --> v1:172.21.15.192:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f22480a8ed0 con 0x7f22480a89f0 2024-06-03T08:36:32.582 INFO:teuthology.orchestra.run.smithi088.stderr:2024-06-03T08:36:32.580+0000 7f224f31d640 1 -- --> v1:172.21.15.88:6790/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f224813f0f0 con 0x7f22480b4250 2024-06-03T08:36:32.583 INFO:teuthology.orchestra.run.smithi088.stderr:2024-06-03T08:36:32.580+0000 7f224f31d640 10 monclient(hunting): _renew_subs 2024-06-03T08:36:32.583 INFO:teuthology.orchestra.run.smithi088.stderr:2024-06-03T08:36:32.580+0000 7f224f31d640 10 monclient(hunting): authenticate will time out at 1248.390325s 2024-06-03T08:36:32.583 INFO:teuthology.orchestra.run.smithi088.stderr:2024-06-03T08:36:32.581+0000 7fccc506c640 10 monclient: get_auth_request con 0x7fcca406d290 auth_method 0 2024-06-03T08:36:32.583 INFO:teuthology.orchestra.run.smithi088.stderr:2024-06-03T08:36:32.582+0000 7f224d8ba640 1 --1- >> v1:172.21.15.88:6789/0 conn(0x7f22481451c0 0x7f224814ce50 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.88:6789/0 says I am v1:172.21.15.88:60430/0 (socket says 172.21.15.88:60430) 2024-06-03T08:36:32.583 INFO:teuthology.orchestra.run.smithi088.stderr:2024-06-03T08:36:32.582+0000 7f224d8ba640 1 -- 172.21.15.88:0/4258495103 learned_addr learned my addr 172.21.15.88:0/4258495103 (peer_addr_for_me v1:172.21.15.88:0/0) 2024-06-03T08:36:32.584 INFO:teuthology.orchestra.run.smithi088.stderr:2024-06-03T08:36:32.582+0000 7fccb67fc640 10 client.4352.objecter ms_handle_connect 0x7fcca406d290 2024-06-03T08:36:32.586 INFO:teuthology.orchestra.run.smithi088.stderr:2024-06-03T08:36:32.585+0000 7f223ffff640 1 -- 172.21.15.88:0/4258495103 <== mon.0 v1:172.21.15.88:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 251287277 0 0) 0x7f22480aa6c0 con 0x7f22481451c0 2024-06-03T08:36:32.586 INFO:teuthology.orchestra.run.smithi088.stderr:2024-06-03T08:36:32.585+0000 7f223ffff640 10 monclient(hunting): _negotiate 2024-06-03T08:36:32.586 INFO:teuthology.orchestra.run.smithi088.stderr:2024-06-03T08:36:32.585+0000 7f223ffff640 10 monclient(hunting): _init_auth method 2 2024-06-03T08:36:32.586 INFO:teuthology.orchestra.run.smithi088.stderr:2024-06-03T08:36:32.585+0000 7f223ffff640 10 monclient(hunting): _init_auth creating new auth 2024-06-03T08:36:32.587 INFO:teuthology.orchestra.run.smithi088.stderr:2024-06-03T08:36:32.585+0000 7f223ffff640 10 monclient(hunting): my global_id is 4344 2024-06-03T08:36:32.587 INFO:teuthology.orchestra.run.smithi088.stderr:2024-06-03T08:36:32.585+0000 7f223ffff640 1 -- 172.21.15.88:0/4258495103 --> v1:172.21.15.88:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f2234003520 con 0x7f22481451c0 2024-06-03T08:36:32.595 INFO:teuthology.orchestra.run.smithi088.stderr:2024-06-03T08:36:32.594+0000 7f223ffff640 1 -- 172.21.15.88:0/4258495103 <== mon.0 v1:172.21.15.88:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 214+0+0 (unknown 520459190 0 0) 0x7f2234003520 con 0x7f22481451c0 2024-06-03T08:36:32.596 INFO:teuthology.orchestra.run.smithi088.stderr:2024-06-03T08:36:32.594+0000 7f223ffff640 1 -- 172.21.15.88:0/4258495103 >> v1:172.21.15.88:6790/0 conn(0x7f22480b4250 legacy=0x7f224813e9e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-06-03T08:36:32.596 INFO:teuthology.orchestra.run.smithi088.stderr:2024-06-03T08:36:32.594+0000 7f223ffff640 1 -- 172.21.15.88:0/4258495103 >> v1:172.21.15.192:6789/0 conn(0x7f22480a89f0 legacy=0x7f2248144ab0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-06-03T08:36:32.596 INFO:teuthology.orchestra.run.smithi088.stderr:2024-06-03T08:36:32.594+0000 7f223ffff640 10 monclient: _finish_hunting 0 2024-06-03T08:36:32.596 INFO:teuthology.orchestra.run.smithi088.stderr:2024-06-03T08:36:32.594+0000 7f223ffff640 1 monclient: found mon.noname-a 2024-06-03T08:36:32.596 INFO:teuthology.orchestra.run.smithi088.stderr:2024-06-03T08:36:32.594+0000 7f223ffff640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-06-03T08:36:32.596 INFO:teuthology.orchestra.run.smithi088.stderr:2024-06-03T08:36:32.594+0000 7f223ffff640 10 monclient: _send_mon_message to mon.noname-a at v1:172.21.15.88:6789/0 2024-06-03T08:36:32.596 INFO:teuthology.orchestra.run.smithi088.stderr:2024-06-03T08:36:32.594+0000 7f223ffff640 1 -- 172.21.15.88:0/4258495103 --> v1:172.21.15.88:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f22480b39a0 con 0x7f22481451c0 2024-06-03T08:36:32.597 INFO:teuthology.orchestra.run.smithi088.stderr:2024-06-03T08:36:32.594+0000 7f223ffff640 10 monclient: _finish_auth 0 2024-06-03T08:36:32.597 INFO:teuthology.orchestra.run.smithi088.stderr:2024-06-03T08:36:32.594+0000 7f223ffff640 10 monclient: _check_auth_tickets 2024-06-03T08:36:32.597 INFO:teuthology.orchestra.run.smithi088.stderr:2024-06-03T08:36:32.594+0000 7f223ffff640 20 monclient: _check_auth_rotating not needed by client.admin 2024-06-03T08:36:32.597 INFO:teuthology.orchestra.run.smithi088.stderr:2024-06-03T08:36:32.594+0000 7f224f31d640 5 monclient: authenticate success, global_id 4344 2024-06-03T08:36:32.597 INFO:teuthology.orchestra.run.smithi088.stderr:2024-06-03T08:36:32.594+0000 7f224f31d640 20 monclient: get_monmap_and_config waiting for monmap|config 2024-06-03T08:36:32.597 INFO:teuthology.orchestra.run.smithi088.stderr:2024-06-03T08:36:32.595+0000 7f223ffff640 1 -- 172.21.15.88:0/4258495103 <== mon.0 v1:172.21.15.88:6789/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 2234374065 0 0) 0x7f2240001990 con 0x7f22481451c0 2024-06-03T08:36:32.597 INFO:teuthology.orchestra.run.smithi088.stderr:2024-06-03T08:36:32.595+0000 7f223ffff640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-06-03T08:36:32.597 INFO:teuthology.orchestra.run.smithi088.stderr:2024-06-03T08:36:32.595+0000 7f223ffff640 10 monclient: got monmap 1 from mon.noname-a (according to old e1) 2024-06-03T08:36:32.597 INFO:teuthology.orchestra.run.smithi088.stderr:2024-06-03T08:36:32.595+0000 7f223ffff640 10 monclient: dump: 2024-06-03T08:36:32.597 INFO:teuthology.orchestra.run.smithi088.stderr:epoch 1 2024-06-03T08:36:32.597 INFO:teuthology.orchestra.run.smithi088.stderr:fsid ec3c3fdd-3ac6-4d7d-9412-683130ed8b04 2024-06-03T08:36:32.597 INFO:teuthology.orchestra.run.smithi088.stderr:last_changed 2024-06-03T08:34:31.885125+0000 2024-06-03T08:36:32.597 INFO:teuthology.orchestra.run.smithi088.stderr:created 2024-06-03T08:34:31.885125+0000 2024-06-03T08:36:32.598 INFO:teuthology.orchestra.run.smithi088.stderr:min_mon_release 18 (reef) 2024-06-03T08:36:32.598 INFO:teuthology.orchestra.run.smithi088.stderr:election_strategy: 3 2024-06-03T08:36:32.598 INFO:teuthology.orchestra.run.smithi088.stderr:0: v1:172.21.15.88:6789/0 mon.a 2024-06-03T08:36:32.598 INFO:teuthology.orchestra.run.smithi088.stderr:1: v1:172.21.15.192:6789/0 mon.b 2024-06-03T08:36:32.598 INFO:teuthology.orchestra.run.smithi088.stderr:2: v1:172.21.15.88:6790/0 mon.c 2024-06-03T08:36:32.598 INFO:teuthology.orchestra.run.smithi088.stderr: 2024-06-03T08:36:32.598 INFO:teuthology.orchestra.run.smithi088.stderr:2024-06-03T08:36:32.595+0000 7f224f31d640 20 monclient: get_monmap_and_config waiting for monmap|config 2024-06-03T08:36:32.600 INFO:teuthology.orchestra.run.smithi088.stderr:2024-06-03T08:36:32.598+0000 7f223ffff640 1 -- 172.21.15.88:0/4258495103 <== mon.0 v1:172.21.15.88:6789/0 4 ==== config(0 keys) v1 ==== 4+0+0 (unknown 0 0 0) 0x7f2240001c20 con 0x7f22481451c0 2024-06-03T08:36:32.600 INFO:teuthology.orchestra.run.smithi088.stderr:2024-06-03T08:36:32.598+0000 7f223ffff640 10 monclient: handle_config config(0 keys) v1 2024-06-03T08:36:32.600 INFO:teuthology.orchestra.run.smithi088.stderr:2024-06-03T08:36:32.598+0000 7f224f31d640 10 monclient: get_monmap_and_config success 2024-06-03T08:36:32.600 INFO:teuthology.orchestra.run.smithi088.stderr:2024-06-03T08:36:32.598+0000 7f224f31d640 4 set_mon_vals no callback set 2024-06-03T08:36:32.600 INFO:teuthology.orchestra.run.smithi088.stderr:2024-06-03T08:36:32.599+0000 7f224f31d640 10 monclient: shutdown 2024-06-03T08:36:32.600 INFO:teuthology.orchestra.run.smithi088.stderr:2024-06-03T08:36:32.599+0000 7f224f31d640 20 monclient: shutdown discarding 0 pending message(s) 2024-06-03T08:36:32.600 INFO:teuthology.orchestra.run.smithi088.stderr:2024-06-03T08:36:32.599+0000 7f224f31d640 1 -- 172.21.15.88:0/4258495103 >> v1:172.21.15.88:6789/0 conn(0x7f22481451c0 legacy=0x7f224814ce50 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-06-03T08:36:32.600 INFO:teuthology.orchestra.run.smithi088.stderr:2024-06-03T08:36:32.599+0000 7f224f31d640 1 -- 172.21.15.88:0/4258495103 shutdown_connections 2024-06-03T08:36:32.600 INFO:teuthology.orchestra.run.smithi088.stderr:2024-06-03T08:36:32.599+0000 7f224f31d640 1 -- 172.21.15.88:0/4258495103 >> 172.21.15.88:0/4258495103 conn(0x7f22480717f0 msgr2=0x7f2248071c00 unknown :-1 s=STATE_NONE l=0).mark_down 2024-06-03T08:36:32.601 INFO:teuthology.orchestra.run.smithi088.stderr:2024-06-03T08:36:32.600+0000 7f224f31d640 1 -- 172.21.15.88:0/4258495103 shutdown_connections 2024-06-03T08:36:32.602 INFO:teuthology.orchestra.run.smithi088.stderr:2024-06-03T08:36:32.601+0000 7fccb67fc640 1 -- 172.21.15.88:0/1456172716 <== mon.2 v1:172.21.15.88:6790/0 8 ==== osd_map(26..26 src has 1..26) v4 ==== 4090+0+0 (unknown 3637450554 0 0) 0x7fccbc053ed0 con 0x7fccc00a8a60 2024-06-03T08:36:32.602 INFO:teuthology.orchestra.run.smithi088.stderr:2024-06-03T08:36:32.601+0000 7fccb67fc640 10 client.4352.objecter ms_dispatch 0x7fccc0000b90 osd_map(26..26 src has 1..26) v4 2024-06-03T08:36:32.602 INFO:teuthology.orchestra.run.smithi088.stderr:2024-06-03T08:36:32.601+0000 7fccb67fc640 3 client.4352.objecter handle_osd_map got epochs [26,26] > 0 2024-06-03T08:36:32.602 INFO:teuthology.orchestra.run.smithi088.stderr:2024-06-03T08:36:32.601+0000 7fccb67fc640 3 client.4352.objecter handle_osd_map decoding full epoch 26 2024-06-03T08:36:32.602 INFO:teuthology.orchestra.run.smithi088.stderr:2024-06-03T08:36:32.601+0000 7fccb67fc640 20 client.4352.objecter dump_active .. 0 homeless 2024-06-03T08:36:32.603 INFO:teuthology.orchestra.run.smithi088.stderr:2024-06-03T08:36:32.600+0000 7f224f31d640 1 -- 172.21.15.88:0/4258495103 wait complete. 2024-06-03T08:36:32.603 INFO:teuthology.orchestra.run.smithi088.stderr:2024-06-03T08:36:32.601+0000 7f224f31d640 10 monclient: build_initial_monmap 2024-06-03T08:36:32.603 INFO:teuthology.orchestra.run.smithi088.stderr:2024-06-03T08:36:32.601+0000 7f224f31d640 10 monclient: monmap: 2024-06-03T08:36:32.603 INFO:teuthology.orchestra.run.smithi088.stderr:epoch 0 2024-06-03T08:36:32.603 INFO:teuthology.orchestra.run.smithi088.stderr:fsid 00000000-0000-0000-0000-000000000000 2024-06-03T08:36:32.603 INFO:teuthology.orchestra.run.smithi088.stderr:last_changed 0.000000 2024-06-03T08:36:32.603 INFO:teuthology.orchestra.run.smithi088.stderr:created 0.000000 2024-06-03T08:36:32.603 INFO:teuthology.orchestra.run.smithi088.stderr:min_mon_release 0 (unknown) 2024-06-03T08:36:32.603 INFO:teuthology.orchestra.run.smithi088.stderr:election_strategy: 1 2024-06-03T08:36:32.603 INFO:teuthology.orchestra.run.smithi088.stderr:0: v1:172.21.15.88:6789/0 mon.noname-a 2024-06-03T08:36:32.604 INFO:teuthology.orchestra.run.smithi088.stderr:1: v1:172.21.15.192:6789/0 mon.noname-b 2024-06-03T08:36:32.604 INFO:teuthology.orchestra.run.smithi088.stderr:2: v1:172.21.15.88:6790/0 mon.noname-c 2024-06-03T08:36:32.604 INFO:teuthology.orchestra.run.smithi088.stderr: 2024-06-03T08:36:32.604 INFO:teuthology.orchestra.run.smithi088.stderr:2024-06-03T08:36:32.602+0000 7f224f31d640 1 librados: starting msgr at 2024-06-03T08:36:32.604 INFO:teuthology.orchestra.run.smithi088.stderr:2024-06-03T08:36:32.602+0000 7f224f31d640 1 librados: starting objecter 2024-06-03T08:36:32.604 INFO:teuthology.orchestra.run.smithi088.stderr:2024-06-03T08:36:32.602+0000 7f224f31d640 1 Processor -- start 2024-06-03T08:36:32.604 INFO:teuthology.orchestra.run.smithi088.stderr:2024-06-03T08:36:32.602+0000 7f224f31d640 1 -- start start 2024-06-03T08:36:32.604 INFO:teuthology.orchestra.run.smithi088.stderr:2024-06-03T08:36:32.602+0000 7f224f31d640 1 librados: setting wanted keys 2024-06-03T08:36:32.604 INFO:teuthology.orchestra.run.smithi088.stderr:2024-06-03T08:36:32.602+0000 7f224f31d640 1 librados: calling monclient init 2024-06-03T08:36:32.604 INFO:teuthology.orchestra.run.smithi088.stderr:2024-06-03T08:36:32.602+0000 7f224f31d640 10 monclient: init 2024-06-03T08:36:32.604 INFO:teuthology.orchestra.run.smithi088.stderr:2024-06-03T08:36:32.603+0000 7f224f31d640 10 monclient: _reopen_session rank -1 2024-06-03T08:36:32.604 INFO:teuthology.orchestra.run.smithi088.stderr:2024-06-03T08:36:32.603+0000 7f224f31d640 10 monclient: _add_conns ranks=[0,1,2] 2024-06-03T08:36:32.605 INFO:teuthology.orchestra.run.smithi088.stderr:2024-06-03T08:36:32.603+0000 7f224f31d640 10 monclient(hunting): picked mon.noname-a con 0x7f22480a89f0 addr v1:172.21.15.88:6789/0 2024-06-03T08:36:32.605 INFO:teuthology.orchestra.run.smithi088.stderr:2024-06-03T08:36:32.603+0000 7f224f31d640 10 monclient(hunting): picked mon.noname-b con 0x7f22480b4250 addr v1:172.21.15.192:6789/0 2024-06-03T08:36:32.605 INFO:teuthology.orchestra.run.smithi088.stderr:2024-06-03T08:36:32.603+0000 7f224f31d640 10 monclient(hunting): picked mon.noname-c con 0x7f22481451c0 addr v1:172.21.15.88:6790/0 2024-06-03T08:36:32.605 INFO:teuthology.orchestra.run.smithi088.stderr:2024-06-03T08:36:32.603+0000 7f224f31d640 1 -- --> v1:172.21.15.88:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f22480b0770 con 0x7f22480a89f0 2024-06-03T08:36:32.605 INFO:teuthology.orchestra.run.smithi088.stderr:2024-06-03T08:36:32.603+0000 7f224f31d640 1 -- --> v1:172.21.15.192:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f22480b0a50 con 0x7f22480b4250 2024-06-03T08:36:32.605 INFO:teuthology.orchestra.run.smithi088.stderr:2024-06-03T08:36:32.603+0000 7f224f31d640 1 -- --> v1:172.21.15.88:6790/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f22480ab4e0 con 0x7f22481451c0 2024-06-03T08:36:32.606 INFO:teuthology.orchestra.run.smithi088.stderr:2024-06-03T08:36:32.603+0000 7f224f31d640 10 monclient(hunting): _renew_subs 2024-06-03T08:36:32.606 INFO:teuthology.orchestra.run.smithi088.stderr:2024-06-03T08:36:32.603+0000 7f224f31d640 10 monclient(hunting): authenticate will time out at 1248.413647s 2024-06-03T08:36:32.606 INFO:teuthology.orchestra.run.smithi088.stderr:2024-06-03T08:36:32.604+0000 7f224d0b9640 1 --1- >> v1:172.21.15.88:6789/0 conn(0x7f22480a89f0 0x7f22481d6730 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=1).handle_server_banner_and_identify peer v1:172.21.15.88:6789/0 says I am v1:172.21.15.88:60436/0 (socket says 172.21.15.88:60436) 2024-06-03T08:36:32.606 INFO:teuthology.orchestra.run.smithi088.stderr:2024-06-03T08:36:32.604+0000 7f224d0b9640 1 -- 172.21.15.88:0/598291210 learned_addr learned my addr 172.21.15.88:0/598291210 (peer_addr_for_me v1:172.21.15.88:0/0) 2024-06-03T08:36:32.606 INFO:teuthology.orchestra.run.smithi088.stderr:2024-06-03T08:36:32.605+0000 7f223dffb640 10 client.?.objecter ms_handle_connect 0x7f22480a89f0 2024-06-03T08:36:32.606 INFO:teuthology.orchestra.run.smithi088.stderr:2024-06-03T08:36:32.605+0000 7f223dffb640 10 client.?.objecter resend_mon_ops 2024-06-03T08:36:32.608 INFO:teuthology.orchestra.run.smithi088.stderr:2024-06-03T08:36:32.608+0000 7f50e2d7d640 10 monclient: get_monmap_and_config 2024-06-03T08:36:32.608 INFO:teuthology.orchestra.run.smithi088.stderr:2024-06-03T08:36:32.608+0000 7f50e2d7d640 10 monclient: build_initial_monmap 2024-06-03T08:36:32.608 INFO:teuthology.orchestra.run.smithi088.stderr:2024-06-03T08:36:32.608+0000 7f50e2d7d640 10 monclient: monmap: 2024-06-03T08:36:32.609 INFO:teuthology.orchestra.run.smithi088.stderr:epoch 0 2024-06-03T08:36:32.609 INFO:teuthology.orchestra.run.smithi088.stderr:fsid ec3c3fdd-3ac6-4d7d-9412-683130ed8b04 2024-06-03T08:36:32.609 INFO:teuthology.orchestra.run.smithi088.stderr:last_changed 2024-06-03T08:36:32.609351+0000 2024-06-03T08:36:32.609 INFO:teuthology.orchestra.run.smithi088.stderr:created 2024-06-03T08:36:32.609351+0000 2024-06-03T08:36:32.609 INFO:teuthology.orchestra.run.smithi088.stderr:min_mon_release 0 (unknown) 2024-06-03T08:36:32.609 INFO:teuthology.orchestra.run.smithi088.stderr:election_strategy: 3 2024-06-03T08:36:32.609 INFO:teuthology.orchestra.run.smithi088.stderr:0: v1:172.21.15.88:6789/0 mon.noname-a 2024-06-03T08:36:32.609 INFO:teuthology.orchestra.run.smithi088.stderr:1: v1:172.21.15.192:6789/0 mon.noname-c 2024-06-03T08:36:32.609 INFO:teuthology.orchestra.run.smithi088.stderr:2: v1:172.21.15.88:6790/0 mon.noname-b 2024-06-03T08:36:32.609 INFO:teuthology.orchestra.run.smithi088.stderr: 2024-06-03T08:36:32.610 INFO:teuthology.orchestra.run.smithi088.stderr:2024-06-03T08:36:32.609+0000 7f223dffb640 1 -- 172.21.15.88:0/598291210 <== mon.0 v1:172.21.15.88:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 3802295496 0 0) 0x7f22480b0770 con 0x7f22480a89f0 2024-06-03T08:36:32.610 INFO:teuthology.orchestra.run.smithi088.stderr:2024-06-03T08:36:32.609+0000 7f223dffb640 10 monclient(hunting): _negotiate 2024-06-03T08:36:32.610 INFO:teuthology.orchestra.run.smithi088.stderr:2024-06-03T08:36:32.609+0000 7f223dffb640 10 monclient(hunting): _init_auth method 2 2024-06-03T08:36:32.610 INFO:teuthology.orchestra.run.smithi088.stderr:2024-06-03T08:36:32.609+0000 7f223dffb640 10 monclient(hunting): _init_auth creating new auth 2024-06-03T08:36:32.610 INFO:teuthology.orchestra.run.smithi088.stderr:2024-06-03T08:36:32.609+0000 7f223dffb640 10 monclient(hunting): my global_id is 4347 2024-06-03T08:36:32.610 INFO:teuthology.orchestra.run.smithi088.stderr:2024-06-03T08:36:32.609+0000 7f223dffb640 1 -- 172.21.15.88:0/598291210 --> v1:172.21.15.88:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f222c004eb0 con 0x7f22480a89f0 2024-06-03T08:36:32.611 INFO:teuthology.orchestra.run.smithi088.stderr:2024-06-03T08:36:32.609+0000 7f50e2d7d640 1 Processor -- start 2024-06-03T08:36:32.611 INFO:teuthology.orchestra.run.smithi088.stderr:2024-06-03T08:36:32.609+0000 7f50e2d7d640 1 -- start start 2024-06-03T08:36:32.612 INFO:teuthology.orchestra.run.smithi088.stderr:2024-06-03T08:36:32.609+0000 7f50e2d7d640 10 monclient: init 2024-06-03T08:36:32.612 INFO:teuthology.orchestra.run.smithi088.stderr:2024-06-03T08:36:32.610+0000 7f50e2d7d640 10 monclient: _reopen_session rank -1 2024-06-03T08:36:32.612 INFO:teuthology.orchestra.run.smithi088.stderr:2024-06-03T08:36:32.610+0000 7f50e2d7d640 10 monclient: _add_conns ranks=[1,0,2] 2024-06-03T08:36:32.612 INFO:teuthology.orchestra.run.smithi088.stderr:2024-06-03T08:36:32.610+0000 7f50e2d7d640 10 monclient(hunting): picked mon.noname-c con 0x7f50dc0b4250 addr v1:172.21.15.192:6789/0 2024-06-03T08:36:32.612 INFO:teuthology.orchestra.run.smithi088.stderr:2024-06-03T08:36:32.610+0000 7f50e2d7d640 10 monclient(hunting): picked mon.noname-a con 0x7f50dc0a89f0 addr v1:172.21.15.88:6789/0 2024-06-03T08:36:32.612 INFO:teuthology.orchestra.run.smithi088.stderr:2024-06-03T08:36:32.610+0000 7f50e2d7d640 10 monclient(hunting): picked mon.noname-b con 0x7f50dc1451c0 addr v1:172.21.15.88:6790/0 2024-06-03T08:36:32.612 INFO:teuthology.orchestra.run.smithi088.stderr:2024-06-03T08:36:32.610+0000 7f50e2d7d640 1 -- --> v1:172.21.15.88:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f50dc0aa6c0 con 0x7f50dc0a89f0 2024-06-03T08:36:32.612 INFO:teuthology.orchestra.run.smithi088.stderr:2024-06-03T08:36:32.610+0000 7f50e2d7d640 1 -- --> v1:172.21.15.192:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f50dc0a8ed0 con 0x7f50dc0b4250 2024-06-03T08:36:32.612 INFO:teuthology.orchestra.run.smithi088.stderr:2024-06-03T08:36:32.610+0000 7f50e2d7d640 1 -- --> v1:172.21.15.88:6790/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f50dc13f0f0 con 0x7f50dc1451c0 2024-06-03T08:36:32.612 INFO:teuthology.orchestra.run.smithi088.stderr:2024-06-03T08:36:32.610+0000 7f50e2d7d640 10 monclient(hunting): _renew_subs 2024-06-03T08:36:32.612 INFO:teuthology.orchestra.run.smithi088.stderr:2024-06-03T08:36:32.610+0000 7f50e2d7d640 10 monclient(hunting): authenticate will time out at 1248.420263s 2024-06-03T08:36:32.615 INFO:teuthology.orchestra.run.smithi088.stderr:2024-06-03T08:36:32.615+0000 7f50e131a640 1 --1- >> v1:172.21.15.88:6790/0 conn(0x7f50dc1451c0 0x7f50dc14ce50 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.88:6790/0 says I am v1:172.21.15.88:54296/0 (socket says 172.21.15.88:54296) 2024-06-03T08:36:32.615 INFO:teuthology.orchestra.run.smithi088.stderr:2024-06-03T08:36:32.615+0000 7f50e131a640 1 -- 172.21.15.88:0/680985712 learned_addr learned my addr 172.21.15.88:0/680985712 (peer_addr_for_me v1:172.21.15.88:0/0) 2024-06-03T08:36:32.618 INFO:teuthology.orchestra.run.smithi088.stderr:2024-06-03T08:36:32.617+0000 7f223dffb640 1 -- 172.21.15.88:0/598291210 <== mon.0 v1:172.21.15.88:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 764+0+0 (unknown 1480082562 0 0) 0x7f222c004eb0 con 0x7f22480a89f0 2024-06-03T08:36:32.618 INFO:teuthology.orchestra.run.smithi088.stderr:2024-06-03T08:36:32.617+0000 7f223dffb640 1 -- 172.21.15.88:0/598291210 --> v1:172.21.15.88:6789/0 -- auth(proto 2 165 bytes epoch 0) v1 -- 0x7f22480b0770 con 0x7f22480a89f0 2024-06-03T08:36:32.618 INFO:teuthology.orchestra.run.smithi088.stderr:2024-06-03T08:36:32.617+0000 7f223dffb640 1 -- 172.21.15.88:0/598291210 <== mon.0 v1:172.21.15.88:6789/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 2234374065 0 0) 0x7f22380018e0 con 0x7f22480a89f0 2024-06-03T08:36:32.618 INFO:teuthology.orchestra.run.smithi088.stderr:2024-06-03T08:36:32.617+0000 7f223dffb640 10 monclient(hunting): handle_monmap mon_map magic: 0 v1 2024-06-03T08:36:32.618 INFO:teuthology.orchestra.run.smithi088.stderr:2024-06-03T08:36:32.617+0000 7f223dffb640 10 monclient(hunting): got monmap 1 from mon.noname-a (according to old e1) 2024-06-03T08:36:32.618 INFO:teuthology.orchestra.run.smithi088.stderr:2024-06-03T08:36:32.617+0000 7f223dffb640 10 monclient(hunting): dump: 2024-06-03T08:36:32.618 INFO:teuthology.orchestra.run.smithi088.stderr:epoch 1 2024-06-03T08:36:32.618 INFO:teuthology.orchestra.run.smithi088.stderr:fsid ec3c3fdd-3ac6-4d7d-9412-683130ed8b04 2024-06-03T08:36:32.618 INFO:teuthology.orchestra.run.smithi088.stderr:last_changed 2024-06-03T08:34:31.885125+0000 2024-06-03T08:36:32.619 INFO:teuthology.orchestra.run.smithi088.stderr:created 2024-06-03T08:34:31.885125+0000 2024-06-03T08:36:32.619 INFO:teuthology.orchestra.run.smithi088.stderr:min_mon_release 18 (reef) 2024-06-03T08:36:32.619 INFO:teuthology.orchestra.run.smithi088.stderr:election_strategy: 3 2024-06-03T08:36:32.619 INFO:teuthology.orchestra.run.smithi088.stderr:0: v1:172.21.15.88:6789/0 mon.a 2024-06-03T08:36:32.619 INFO:teuthology.orchestra.run.smithi088.stderr:1: v1:172.21.15.192:6789/0 mon.b 2024-06-03T08:36:32.619 INFO:teuthology.orchestra.run.smithi088.stderr:2: v1:172.21.15.88:6790/0 mon.c 2024-06-03T08:36:32.619 INFO:teuthology.orchestra.run.smithi088.stderr: 2024-06-03T08:36:32.619 INFO:teuthology.orchestra.run.smithi088.stderr:2024-06-03T08:36:32.617+0000 7f223dffb640 10 monclient(hunting): _finish_auth 0 2024-06-03T08:36:32.624 INFO:teuthology.orchestra.run.smithi088.stderr:2024-06-03T08:36:32.623+0000 7f223dffb640 1 -- 172.21.15.88:0/598291210 <== mon.0 v1:172.21.15.88:6789/0 4 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 580+0+0 (unknown 331358500 0 0) 0x7f22480b0770 con 0x7f22480a89f0 2024-06-03T08:36:32.624 INFO:teuthology.orchestra.run.smithi088.stderr:2024-06-03T08:36:32.623+0000 7f223dffb640 1 -- 172.21.15.88:0/598291210 >> v1:172.21.15.88:6790/0 conn(0x7f22481451c0 legacy=0x7f22481e1f70 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-06-03T08:36:32.625 INFO:teuthology.orchestra.run.smithi088.stderr:2024-06-03T08:36:32.623+0000 7f223dffb640 1 -- 172.21.15.88:0/598291210 >> v1:172.21.15.192:6789/0 conn(0x7f22480b4250 legacy=0x7f2248138230 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-06-03T08:36:32.625 INFO:teuthology.orchestra.run.smithi088.stderr:2024-06-03T08:36:32.623+0000 7f223dffb640 10 monclient: _finish_hunting 0 2024-06-03T08:36:32.625 INFO:teuthology.orchestra.run.smithi088.stderr:2024-06-03T08:36:32.623+0000 7f223dffb640 1 monclient: found mon.a 2024-06-03T08:36:32.625 INFO:teuthology.orchestra.run.smithi088.stderr:2024-06-03T08:36:32.623+0000 7f223dffb640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-06-03T08:36:32.625 INFO:teuthology.orchestra.run.smithi088.stderr:2024-06-03T08:36:32.623+0000 7f223dffb640 10 monclient: _send_mon_message to mon.a at v1:172.21.15.88:6789/0 2024-06-03T08:36:32.625 INFO:teuthology.orchestra.run.smithi088.stderr:2024-06-03T08:36:32.623+0000 7f223dffb640 1 -- 172.21.15.88:0/598291210 --> v1:172.21.15.88:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f22480ab670 con 0x7f22480a89f0 2024-06-03T08:36:32.625 INFO:teuthology.orchestra.run.smithi088.stderr:2024-06-03T08:36:32.623+0000 7f223dffb640 10 monclient: _finish_auth 0 2024-06-03T08:36:32.625 INFO:teuthology.orchestra.run.smithi088.stderr:2024-06-03T08:36:32.623+0000 7f223dffb640 10 monclient: _check_auth_tickets 2024-06-03T08:36:32.625 INFO:teuthology.orchestra.run.smithi088.stderr:2024-06-03T08:36:32.623+0000 7f223dffb640 20 monclient: _check_auth_rotating not needed by client.admin 2024-06-03T08:36:32.625 INFO:teuthology.orchestra.run.smithi088.stderr:2024-06-03T08:36:32.624+0000 7f224f31d640 5 monclient: authenticate success, global_id 4347 2024-06-03T08:36:32.625 INFO:teuthology.orchestra.run.smithi088.stderr:2024-06-03T08:36:32.624+0000 7f224f31d640 10 monclient: _renew_subs 2024-06-03T08:36:32.625 INFO:teuthology.orchestra.run.smithi088.stderr:2024-06-03T08:36:32.624+0000 7f224f31d640 10 monclient: _send_mon_message to mon.a at v1:172.21.15.88:6789/0 2024-06-03T08:36:32.625 INFO:teuthology.orchestra.run.smithi088.stderr:2024-06-03T08:36:32.624+0000 7f224f31d640 1 -- 172.21.15.88:0/598291210 --> v1:172.21.15.88:6789/0 -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f22481d6e40 con 0x7f22480a89f0 2024-06-03T08:36:32.625 INFO:teuthology.orchestra.run.smithi088.stderr:2024-06-03T08:36:32.624+0000 7f224f31d640 10 client.4347.objecter _maybe_request_map subscribing (onetime) to next osd map 2024-06-03T08:36:32.626 INFO:teuthology.orchestra.run.smithi088.stderr:2024-06-03T08:36:32.624+0000 7f224f31d640 10 monclient: _renew_subs 2024-06-03T08:36:32.626 INFO:teuthology.orchestra.run.smithi088.stderr:2024-06-03T08:36:32.624+0000 7f224f31d640 10 monclient: _send_mon_message to mon.a at v1:172.21.15.88:6789/0 2024-06-03T08:36:32.626 INFO:teuthology.orchestra.run.smithi088.stderr:2024-06-03T08:36:32.624+0000 7f224f31d640 1 -- 172.21.15.88:0/598291210 --> v1:172.21.15.88:6789/0 -- mon_subscribe({osdmap=0}) v3 -- 0x7f2248142260 con 0x7f22480a89f0 2024-06-03T08:36:32.626 INFO:teuthology.orchestra.run.smithi088.stderr:2024-06-03T08:36:32.624+0000 7f224f31d640 1 librados: init done 2024-06-03T08:36:32.626 INFO:teuthology.orchestra.run.smithi088.stderr:2024-06-03T08:36:32.624+0000 7f3d5015c640 10 monclient: get_monmap_and_config 2024-06-03T08:36:32.626 INFO:teuthology.orchestra.run.smithi088.stderr:2024-06-03T08:36:32.624+0000 7f3d5015c640 10 monclient: build_initial_monmap 2024-06-03T08:36:32.626 INFO:teuthology.orchestra.run.smithi088.stderr:2024-06-03T08:36:32.624+0000 7f3d5015c640 10 monclient: monmap: 2024-06-03T08:36:32.626 INFO:teuthology.orchestra.run.smithi088.stderr:epoch 0 2024-06-03T08:36:32.627 INFO:teuthology.orchestra.run.smithi088.stderr:fsid ec3c3fdd-3ac6-4d7d-9412-683130ed8b04 2024-06-03T08:36:32.627 INFO:teuthology.orchestra.run.smithi088.stderr:last_changed 2024-06-03T08:36:32.625713+0000 2024-06-03T08:36:32.627 INFO:teuthology.orchestra.run.smithi088.stderr:created 2024-06-03T08:36:32.625713+0000 2024-06-03T08:36:32.627 INFO:teuthology.orchestra.run.smithi088.stderr:min_mon_release 0 (unknown) 2024-06-03T08:36:32.627 INFO:teuthology.orchestra.run.smithi088.stderr:election_strategy: 3 2024-06-03T08:36:32.627 INFO:teuthology.orchestra.run.smithi088.stderr:0: v1:172.21.15.88:6789/0 mon.noname-a 2024-06-03T08:36:32.627 INFO:teuthology.orchestra.run.smithi088.stderr:1: v1:172.21.15.192:6789/0 mon.noname-c 2024-06-03T08:36:32.627 INFO:teuthology.orchestra.run.smithi088.stderr:2: v1:172.21.15.88:6790/0 mon.noname-b 2024-06-03T08:36:32.627 INFO:teuthology.orchestra.run.smithi088.stderr: 2024-06-03T08:36:32.627 INFO:teuthology.orchestra.run.smithi088.stderr:2024-06-03T08:36:32.625+0000 7f3d5015c640 1 Processor -- start 2024-06-03T08:36:32.627 INFO:teuthology.orchestra.run.smithi088.stderr:2024-06-03T08:36:32.625+0000 7f224f31d640 10 start_mon_command cmd=[{"prefix": "get_command_descriptions"}] 2024-06-03T08:36:32.628 INFO:teuthology.orchestra.run.smithi088.stderr:2024-06-03T08:36:32.625+0000 7f3d5015c640 1 -- start start 2024-06-03T08:36:32.628 INFO:teuthology.orchestra.run.smithi088.stderr:2024-06-03T08:36:32.625+0000 7f3d5015c640 10 monclient: init 2024-06-03T08:36:32.628 INFO:teuthology.orchestra.run.smithi088.stderr:2024-06-03T08:36:32.625+0000 7f224f31d640 10 monclient: _send_command 1 [{"prefix": "get_command_descriptions"}] 2024-06-03T08:36:32.628 INFO:teuthology.orchestra.run.smithi088.stderr:2024-06-03T08:36:32.625+0000 7f224f31d640 10 monclient: _send_mon_message to mon.a at v1:172.21.15.88:6789/0 2024-06-03T08:36:32.628 INFO:teuthology.orchestra.run.smithi088.stderr:2024-06-03T08:36:32.625+0000 7f224f31d640 1 -- 172.21.15.88:0/598291210 --> v1:172.21.15.88:6789/0 -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f22480671b0 con 0x7f22480a89f0 2024-06-03T08:36:32.629 INFO:teuthology.orchestra.run.smithi088.stderr:2024-06-03T08:36:32.626+0000 7f3d5015c640 10 monclient: _reopen_session rank -1 2024-06-03T08:36:32.629 INFO:teuthology.orchestra.run.smithi088.stderr:2024-06-03T08:36:32.626+0000 7f3d5015c640 10 monclient: _add_conns ranks=[1,0,2] 2024-06-03T08:36:32.629 INFO:teuthology.orchestra.run.smithi088.stderr:2024-06-03T08:36:32.626+0000 7f3d5015c640 10 monclient(hunting): picked mon.noname-c con 0x7f3d480b4250 addr v1:172.21.15.192:6789/0 2024-06-03T08:36:32.629 INFO:teuthology.orchestra.run.smithi088.stderr:2024-06-03T08:36:32.626+0000 7f3d5015c640 10 monclient(hunting): picked mon.noname-a con 0x7f3d480a89f0 addr v1:172.21.15.88:6789/0 2024-06-03T08:36:32.629 INFO:teuthology.orchestra.run.smithi088.stderr:2024-06-03T08:36:32.626+0000 7f3d5015c640 10 monclient(hunting): picked mon.noname-b con 0x7f3d481451c0 addr v1:172.21.15.88:6790/0 2024-06-03T08:36:32.629 INFO:teuthology.orchestra.run.smithi088.stderr:2024-06-03T08:36:32.626+0000 7f3d5015c640 1 -- --> v1:172.21.15.88:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f3d480aa6c0 con 0x7f3d480a89f0 2024-06-03T08:36:32.629 INFO:teuthology.orchestra.run.smithi088.stderr:2024-06-03T08:36:32.626+0000 7f3d5015c640 1 -- --> v1:172.21.15.192:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f3d480a8ed0 con 0x7f3d480b4250 2024-06-03T08:36:32.629 INFO:teuthology.orchestra.run.smithi088.stderr:2024-06-03T08:36:32.626+0000 7f3d5015c640 1 -- --> v1:172.21.15.88:6790/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f3d4813f0f0 con 0x7f3d481451c0 2024-06-03T08:36:32.629 INFO:teuthology.orchestra.run.smithi088.stderr:2024-06-03T08:36:32.626+0000 7f3d5015c640 10 monclient(hunting): _renew_subs 2024-06-03T08:36:32.629 INFO:teuthology.orchestra.run.smithi088.stderr:2024-06-03T08:36:32.626+0000 7f3d5015c640 10 monclient(hunting): authenticate will time out at 1248.436437s 2024-06-03T08:36:32.629 INFO:teuthology.orchestra.run.smithi088.stderr:2024-06-03T08:36:32.627+0000 7f3d4d6f7640 1 --1- >> v1:172.21.15.88:6789/0 conn(0x7f3d480a89f0 0x7f3d48144ab0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.88:6789/0 says I am v1:172.21.15.88:60454/0 (socket says 172.21.15.88:60454) 2024-06-03T08:36:32.630 INFO:teuthology.orchestra.run.smithi088.stderr:2024-06-03T08:36:32.627+0000 7f3d4d6f7640 1 -- 172.21.15.88:0/1229108545 learned_addr learned my addr 172.21.15.88:0/1229108545 (peer_addr_for_me v1:172.21.15.88:0/0) 2024-06-03T08:36:32.638 INFO:teuthology.orchestra.run.smithi088.stderr:2024-06-03T08:36:32.638+0000 7fb0feffd640 1 -- 172.21.15.88:0/4168539993 <== mon.1 v1:172.21.15.192:6789/0 6 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 2234374065 0 0) 0x7fb0f0001210 con 0x7fb10813eec0 2024-06-03T08:36:32.638 INFO:teuthology.orchestra.run.smithi088.stderr:2024-06-03T08:36:32.638+0000 7fb0feffd640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-06-03T08:36:32.638 INFO:teuthology.orchestra.run.smithi088.stderr:2024-06-03T08:36:32.638+0000 7fb0feffd640 10 monclient: got monmap 1 from mon.b (according to old e1) 2024-06-03T08:36:32.639 INFO:teuthology.orchestra.run.smithi088.stderr:2024-06-03T08:36:32.638+0000 7fb0feffd640 10 monclient: dump: 2024-06-03T08:36:32.639 INFO:teuthology.orchestra.run.smithi088.stderr:epoch 1 2024-06-03T08:36:32.639 INFO:teuthology.orchestra.run.smithi088.stderr:fsid ec3c3fdd-3ac6-4d7d-9412-683130ed8b04 2024-06-03T08:36:32.639 INFO:teuthology.orchestra.run.smithi088.stderr:last_changed 2024-06-03T08:34:31.885125+0000 2024-06-03T08:36:32.639 INFO:teuthology.orchestra.run.smithi088.stderr:created 2024-06-03T08:34:31.885125+0000 2024-06-03T08:36:32.639 INFO:teuthology.orchestra.run.smithi088.stderr:min_mon_release 18 (reef) 2024-06-03T08:36:32.639 INFO:teuthology.orchestra.run.smithi088.stderr:election_strategy: 3 2024-06-03T08:36:32.639 INFO:teuthology.orchestra.run.smithi088.stderr:0: v1:172.21.15.88:6789/0 mon.a 2024-06-03T08:36:32.639 INFO:teuthology.orchestra.run.smithi088.stderr:1: v1:172.21.15.192:6789/0 mon.b 2024-06-03T08:36:32.639 INFO:teuthology.orchestra.run.smithi088.stderr:2: v1:172.21.15.88:6790/0 mon.c 2024-06-03T08:36:32.639 INFO:teuthology.orchestra.run.smithi088.stderr: 2024-06-03T08:36:32.640 INFO:teuthology.orchestra.run.smithi088.stderr:2024-06-03T08:36:32.639+0000 7fb0feffd640 1 -- 172.21.15.88:0/4168539993 <== mon.1 v1:172.21.15.192:6789/0 7 ==== mgrmap(e 5) v1 ==== 89110+0+0 (unknown 873533973 0 0) 0x7fb0f001abb0 con 0x7fb10813eec0 2024-06-03T08:36:32.641 INFO:teuthology.orchestra.run.smithi088.stderr:2024-06-03T08:36:32.639+0000 7f84063aa640 10 monclient: get_monmap_and_config 2024-06-03T08:36:32.641 INFO:teuthology.orchestra.run.smithi088.stderr:2024-06-03T08:36:32.639+0000 7f84063aa640 10 monclient: build_initial_monmap 2024-06-03T08:36:32.641 INFO:teuthology.orchestra.run.smithi088.stderr:2024-06-03T08:36:32.639+0000 7f84063aa640 10 monclient: monmap: 2024-06-03T08:36:32.641 INFO:teuthology.orchestra.run.smithi088.stderr:epoch 0 2024-06-03T08:36:32.641 INFO:teuthology.orchestra.run.smithi088.stderr:fsid ec3c3fdd-3ac6-4d7d-9412-683130ed8b04 2024-06-03T08:36:32.641 INFO:teuthology.orchestra.run.smithi088.stderr:last_changed 2024-06-03T08:36:32.640509+0000 2024-06-03T08:36:32.641 INFO:teuthology.orchestra.run.smithi088.stderr:created 2024-06-03T08:36:32.640509+0000 2024-06-03T08:36:32.641 INFO:teuthology.orchestra.run.smithi088.stderr:min_mon_release 0 (unknown) 2024-06-03T08:36:32.641 INFO:teuthology.orchestra.run.smithi088.stderr:election_strategy: 3 2024-06-03T08:36:32.642 INFO:teuthology.orchestra.run.smithi088.stderr:0: v1:172.21.15.88:6789/0 mon.noname-a 2024-06-03T08:36:32.642 INFO:teuthology.orchestra.run.smithi088.stderr:1: v1:172.21.15.192:6789/0 mon.noname-c 2024-06-03T08:36:32.642 INFO:teuthology.orchestra.run.smithi088.stderr:2: v1:172.21.15.88:6790/0 mon.noname-b 2024-06-03T08:36:32.642 INFO:teuthology.orchestra.run.smithi088.stderr: 2024-06-03T08:36:32.642 INFO:teuthology.orchestra.run.smithi088.stderr:2024-06-03T08:36:32.640+0000 7f84063aa640 1 Processor -- start 2024-06-03T08:36:32.642 INFO:teuthology.orchestra.run.smithi088.stderr:2024-06-03T08:36:32.640+0000 7f84063aa640 1 -- start start 2024-06-03T08:36:32.642 INFO:teuthology.orchestra.run.smithi088.stderr:2024-06-03T08:36:32.640+0000 7f84063aa640 10 monclient: init 2024-06-03T08:36:32.642 INFO:teuthology.orchestra.run.smithi088.stderr:2024-06-03T08:36:32.640+0000 7fb10659c640 10 monclient: get_auth_request con 0x7fb0e806d200 auth_method 0 2024-06-03T08:36:32.643 INFO:teuthology.orchestra.run.smithi088.stderr:2024-06-03T08:36:32.641+0000 7fb0feffd640 10 client.4339.objecter ms_handle_connect 0x7fb0e806d200 2024-06-03T08:36:32.643 INFO:teuthology.orchestra.run.smithi088.stderr:2024-06-03T08:36:32.641+0000 7fb0feffd640 1 -- 172.21.15.88:0/4168539993 <== mon.1 v1:172.21.15.192:6789/0 8 ==== osd_map(26..26 src has 1..26) v4 ==== 4090+0+0 (unknown 3637450554 0 0) 0x7fb0f0054350 con 0x7fb10813eec0 2024-06-03T08:36:32.643 INFO:teuthology.orchestra.run.smithi088.stderr:2024-06-03T08:36:32.641+0000 7fb0feffd640 10 client.4339.objecter ms_dispatch 0x7fb108000b90 osd_map(26..26 src has 1..26) v4 2024-06-03T08:36:32.643 INFO:teuthology.orchestra.run.smithi088.stderr:2024-06-03T08:36:32.641+0000 7fb0feffd640 3 client.4339.objecter handle_osd_map got epochs [26,26] > 0 2024-06-03T08:36:32.643 INFO:teuthology.orchestra.run.smithi088.stderr:2024-06-03T08:36:32.641+0000 7fb0feffd640 3 client.4339.objecter handle_osd_map decoding full epoch 26 2024-06-03T08:36:32.643 INFO:teuthology.orchestra.run.smithi088.stderr:2024-06-03T08:36:32.641+0000 7fb0feffd640 20 client.4339.objecter dump_active .. 0 homeless 2024-06-03T08:36:32.644 INFO:teuthology.orchestra.run.smithi088.stderr:2024-06-03T08:36:32.641+0000 7f84063aa640 10 monclient: _reopen_session rank -1 2024-06-03T08:36:32.644 INFO:teuthology.orchestra.run.smithi088.stderr:2024-06-03T08:36:32.641+0000 7f84063aa640 10 monclient: _add_conns ranks=[1,0,2] 2024-06-03T08:36:32.644 INFO:teuthology.orchestra.run.smithi088.stderr:2024-06-03T08:36:32.641+0000 7f84063aa640 10 monclient(hunting): picked mon.noname-c con 0x7f84000ae8c0 addr v1:172.21.15.192:6789/0 2024-06-03T08:36:32.644 INFO:teuthology.orchestra.run.smithi088.stderr:2024-06-03T08:36:32.641+0000 7f84063aa640 10 monclient(hunting): picked mon.noname-a con 0x7f84000712a0 addr v1:172.21.15.88:6789/0 2024-06-03T08:36:32.644 INFO:teuthology.orchestra.run.smithi088.stderr:2024-06-03T08:36:32.641+0000 7f84063aa640 10 monclient(hunting): picked mon.noname-b con 0x7f840013b2f0 addr v1:172.21.15.88:6790/0 2024-06-03T08:36:32.644 INFO:teuthology.orchestra.run.smithi088.stderr:2024-06-03T08:36:32.641+0000 7f84063aa640 1 -- --> v1:172.21.15.88:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f8400070910 con 0x7f84000712a0 2024-06-03T08:36:32.644 INFO:teuthology.orchestra.run.smithi088.stderr:2024-06-03T08:36:32.641+0000 7f84063aa640 1 -- --> v1:172.21.15.192:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f8400071fa0 con 0x7f84000ae8c0 2024-06-03T08:36:32.645 INFO:teuthology.orchestra.run.smithi088.stderr:2024-06-03T08:36:32.641+0000 7f84063aa640 1 -- --> v1:172.21.15.88:6790/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f8400071720 con 0x7f840013b2f0 2024-06-03T08:36:32.645 INFO:teuthology.orchestra.run.smithi088.stderr:2024-06-03T08:36:32.641+0000 7f84063aa640 10 monclient(hunting): _renew_subs 2024-06-03T08:36:32.645 INFO:teuthology.orchestra.run.smithi088.stderr:2024-06-03T08:36:32.641+0000 7f84063aa640 10 monclient(hunting): authenticate will time out at 1248.451773s 2024-06-03T08:36:32.645 INFO:teuthology.orchestra.run.smithi088.stderr:2024-06-03T08:36:32.643+0000 7fb0feffd640 1 -- 172.21.15.88:0/4168539993 <== mon.1 v1:172.21.15.192:6789/0 9 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+182498 (unknown 1092875540 0 3016760700) 0x7fb0f004fd30 con 0x7fb10813eec0 2024-06-03T08:36:32.645 INFO:teuthology.orchestra.run.smithi088.stderr:2024-06-03T08:36:32.643+0000 7fb0feffd640 10 monclient: handle_mon_command_ack 1 [{"prefix": "get_command_descriptions"}] 2024-06-03T08:36:32.645 INFO:teuthology.orchestra.run.smithi088.stderr:2024-06-03T08:36:32.643+0000 7fb0feffd640 10 monclient: _finish_command 1 = system:0 2024-06-03T08:36:32.646 INFO:teuthology.orchestra.run.smithi088.stderr:2024-06-03T08:36:32.645+0000 7f16b9c8f640 10 monclient: get_monmap_and_config 2024-06-03T08:36:32.646 INFO:teuthology.orchestra.run.smithi088.stderr:2024-06-03T08:36:32.645+0000 7f16b9c8f640 10 monclient: build_initial_monmap 2024-06-03T08:36:32.646 INFO:teuthology.orchestra.run.smithi088.stderr:2024-06-03T08:36:32.645+0000 7f16b9c8f640 10 monclient: monmap: 2024-06-03T08:36:32.646 INFO:teuthology.orchestra.run.smithi088.stderr:epoch 0 2024-06-03T08:36:32.646 INFO:teuthology.orchestra.run.smithi088.stderr:fsid ec3c3fdd-3ac6-4d7d-9412-683130ed8b04 2024-06-03T08:36:32.646 INFO:teuthology.orchestra.run.smithi088.stderr:last_changed 2024-06-03T08:36:32.646326+0000 2024-06-03T08:36:32.646 INFO:teuthology.orchestra.run.smithi088.stderr:created 2024-06-03T08:36:32.646326+0000 2024-06-03T08:36:32.646 INFO:teuthology.orchestra.run.smithi088.stderr:min_mon_release 0 (unknown) 2024-06-03T08:36:32.646 INFO:teuthology.orchestra.run.smithi088.stderr:election_strategy: 3 2024-06-03T08:36:32.646 INFO:teuthology.orchestra.run.smithi088.stderr:0: v1:172.21.15.88:6789/0 mon.noname-a 2024-06-03T08:36:32.646 INFO:teuthology.orchestra.run.smithi088.stderr:1: v1:172.21.15.192:6789/0 mon.noname-c 2024-06-03T08:36:32.646 INFO:teuthology.orchestra.run.smithi088.stderr:2: v1:172.21.15.88:6790/0 mon.noname-b 2024-06-03T08:36:32.646 INFO:teuthology.orchestra.run.smithi088.stderr: 2024-06-03T08:36:32.647 INFO:teuthology.orchestra.run.smithi088.stderr:2024-06-03T08:36:32.646+0000 7f16b9c8f640 1 Processor -- start 2024-06-03T08:36:32.647 INFO:teuthology.orchestra.run.smithi088.stderr:2024-06-03T08:36:32.646+0000 7f16b9c8f640 1 -- start start 2024-06-03T08:36:32.647 INFO:teuthology.orchestra.run.smithi088.stderr:2024-06-03T08:36:32.646+0000 7f16b9c8f640 10 monclient: init 2024-06-03T08:36:32.647 INFO:teuthology.orchestra.run.smithi088.stderr:2024-06-03T08:36:32.646+0000 7f83fffff640 1 --1- >> v1:172.21.15.192:6789/0 conn(0x7f84000ae8c0 0x7f840013abe0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.192:6789/0 says I am v1:172.21.15.88:52540/0 (socket says 172.21.15.88:52540) 2024-06-03T08:36:32.648 INFO:teuthology.orchestra.run.smithi088.stderr:2024-06-03T08:36:32.646+0000 7f83fffff640 1 -- 172.21.15.88:0/3468491357 learned_addr learned my addr 172.21.15.88:0/3468491357 (peer_addr_for_me v1:172.21.15.88:0/0) 2024-06-03T08:36:32.649 INFO:teuthology.orchestra.run.smithi088.stderr:2024-06-03T08:36:32.647+0000 7f16b9c8f640 10 monclient: _reopen_session rank -1 2024-06-03T08:36:32.649 INFO:teuthology.orchestra.run.smithi088.stderr:2024-06-03T08:36:32.647+0000 7f16b9c8f640 10 monclient: _add_conns ranks=[1,2,0] 2024-06-03T08:36:32.649 INFO:teuthology.orchestra.run.smithi088.stderr:2024-06-03T08:36:32.647+0000 7f16b9c8f640 10 monclient(hunting): picked mon.noname-c con 0x7f16b40b4250 addr v1:172.21.15.192:6789/0 2024-06-03T08:36:32.649 INFO:teuthology.orchestra.run.smithi088.stderr:2024-06-03T08:36:32.647+0000 7f16b9c8f640 10 monclient(hunting): picked mon.noname-b con 0x7f16b40a89f0 addr v1:172.21.15.88:6790/0 2024-06-03T08:36:32.649 INFO:teuthology.orchestra.run.smithi088.stderr:2024-06-03T08:36:32.647+0000 7f16b9c8f640 10 monclient(hunting): picked mon.noname-a con 0x7f16b41451c0 addr v1:172.21.15.88:6789/0 2024-06-03T08:36:32.649 INFO:teuthology.orchestra.run.smithi088.stderr:2024-06-03T08:36:32.647+0000 7f16b9c8f640 1 -- --> v1:172.21.15.88:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f16b40aa6c0 con 0x7f16b41451c0 2024-06-03T08:36:32.649 INFO:teuthology.orchestra.run.smithi088.stderr:2024-06-03T08:36:32.647+0000 7f16b9c8f640 1 -- --> v1:172.21.15.192:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f16b40a8ed0 con 0x7f16b40b4250 2024-06-03T08:36:32.649 INFO:teuthology.orchestra.run.smithi088.stderr:2024-06-03T08:36:32.647+0000 7f16b9c8f640 1 -- --> v1:172.21.15.88:6790/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f16b413f0f0 con 0x7f16b40a89f0 2024-06-03T08:36:32.649 INFO:teuthology.orchestra.run.smithi088.stderr:2024-06-03T08:36:32.647+0000 7f16b9c8f640 10 monclient(hunting): _renew_subs 2024-06-03T08:36:32.649 INFO:teuthology.orchestra.run.smithi088.stderr:2024-06-03T08:36:32.647+0000 7f16b9c8f640 10 monclient(hunting): authenticate will time out at 1248.457837s 2024-06-03T08:36:32.651 INFO:teuthology.orchestra.run.smithi088.stderr:2024-06-03T08:36:32.651+0000 7f223dffb640 1 -- 172.21.15.88:0/598291210 <== mon.0 v1:172.21.15.88:6789/0 5 ==== config(0 keys) v1 ==== 4+0+0 (unknown 0 0 0) 0x7f2238001cd0 con 0x7f22480a89f0 2024-06-03T08:36:32.651 INFO:teuthology.orchestra.run.smithi088.stderr:2024-06-03T08:36:32.651+0000 7f223dffb640 10 monclient: handle_config config(0 keys) v1 2024-06-03T08:36:32.651 INFO:teuthology.orchestra.run.smithi088.stderr:2024-06-03T08:36:32.651+0000 7f223dffb640 1 -- 172.21.15.88:0/598291210 <== mon.0 v1:172.21.15.88:6789/0 6 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 2234374065 0 0) 0x7f22380018e0 con 0x7f22480a89f0 2024-06-03T08:36:32.651 INFO:teuthology.orchestra.run.smithi088.stderr:2024-06-03T08:36:32.651+0000 7f223dffb640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-06-03T08:36:32.651 INFO:teuthology.orchestra.run.smithi088.stderr:2024-06-03T08:36:32.651+0000 7f223dffb640 10 monclient: got monmap 1 from mon.a (according to old e1) 2024-06-03T08:36:32.651 INFO:teuthology.orchestra.run.smithi088.stderr:2024-06-03T08:36:32.651+0000 7f223dffb640 10 monclient: dump: 2024-06-03T08:36:32.651 INFO:teuthology.orchestra.run.smithi088.stderr:epoch 1 2024-06-03T08:36:32.651 INFO:teuthology.orchestra.run.smithi088.stderr:fsid ec3c3fdd-3ac6-4d7d-9412-683130ed8b04 2024-06-03T08:36:32.652 INFO:teuthology.orchestra.run.smithi088.stderr:last_changed 2024-06-03T08:34:31.885125+0000 2024-06-03T08:36:32.652 INFO:teuthology.orchestra.run.smithi088.stderr:created 2024-06-03T08:34:31.885125+0000 2024-06-03T08:36:32.652 INFO:teuthology.orchestra.run.smithi088.stderr:min_mon_release 18 (reef) 2024-06-03T08:36:32.652 INFO:teuthology.orchestra.run.smithi088.stderr:election_strategy: 3 2024-06-03T08:36:32.652 INFO:teuthology.orchestra.run.smithi088.stderr:0: v1:172.21.15.88:6789/0 mon.a 2024-06-03T08:36:32.652 INFO:teuthology.orchestra.run.smithi088.stderr:1: v1:172.21.15.192:6789/0 mon.b 2024-06-03T08:36:32.652 INFO:teuthology.orchestra.run.smithi088.stderr:2: v1:172.21.15.88:6790/0 mon.c 2024-06-03T08:36:32.652 INFO:teuthology.orchestra.run.smithi088.stderr: 2024-06-03T08:36:32.652 INFO:teuthology.orchestra.run.smithi088.stderr:2024-06-03T08:36:32.651+0000 7f223f7fe640 4 set_mon_vals no callback set 2024-06-03T08:36:32.652 INFO:teuthology.orchestra.run.smithi088.stderr:2024-06-03T08:36:32.652+0000 7f223dffb640 1 -- 172.21.15.88:0/598291210 <== mon.0 v1:172.21.15.88:6789/0 7 ==== mgrmap(e 5) v1 ==== 89110+0+0 (unknown 873533973 0 0) 0x7f22380038a0 con 0x7f22480a89f0 2024-06-03T08:36:32.653 INFO:teuthology.orchestra.run.smithi088.stderr:2024-06-03T08:36:32.653+0000 7f83feffd640 1 -- 172.21.15.88:0/3468491357 <== mon.1 v1:172.21.15.192:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 1863382353 0 0) 0x7f8400071fa0 con 0x7f84000ae8c0 2024-06-03T08:36:32.653 INFO:teuthology.orchestra.run.smithi088.stderr:2024-06-03T08:36:32.653+0000 7f83feffd640 10 monclient(hunting): _negotiate 2024-06-03T08:36:32.653 INFO:teuthology.orchestra.run.smithi088.stderr:2024-06-03T08:36:32.653+0000 7f83feffd640 10 monclient(hunting): _init_auth method 2 2024-06-03T08:36:32.653 INFO:teuthology.orchestra.run.smithi088.stderr:2024-06-03T08:36:32.653+0000 7f83feffd640 10 monclient(hunting): _init_auth creating new auth 2024-06-03T08:36:32.653 INFO:teuthology.orchestra.run.smithi088.stderr:2024-06-03T08:36:32.653+0000 7f83feffd640 10 monclient(hunting): my global_id is 4345 2024-06-03T08:36:32.653 INFO:teuthology.orchestra.run.smithi088.stderr:2024-06-03T08:36:32.653+0000 7f83feffd640 1 -- 172.21.15.88:0/3468491357 --> v1:172.21.15.192:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f83e8003520 con 0x7f84000ae8c0 2024-06-03T08:36:32.654 INFO:teuthology.orchestra.run.smithi088.stderr:2024-06-03T08:36:32.653+0000 7f224c8b8640 10 monclient: get_auth_request con 0x7f222c06d200 auth_method 0 2024-06-03T08:36:32.654 INFO:teuthology.orchestra.run.smithi088.stderr:2024-06-03T08:36:32.653+0000 7f223dffb640 10 client.4347.objecter ms_handle_connect 0x7f222c06d200 2024-06-03T08:36:32.671 INFO:teuthology.orchestra.run.smithi088.stderr:2024-06-03T08:36:32.670+0000 7f16b37fe640 1 --1- >> v1:172.21.15.192:6789/0 conn(0x7f16b40b4250 0x7f16b413e9e0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.192:6789/0 says I am v1:172.21.15.88:52554/0 (socket says 172.21.15.88:52554) 2024-06-03T08:36:32.671 INFO:teuthology.orchestra.run.smithi088.stderr:2024-06-03T08:36:32.670+0000 7f16b37fe640 1 -- 172.21.15.88:0/1383755685 learned_addr learned my addr 172.21.15.88:0/1383755685 (peer_addr_for_me v1:172.21.15.88:0/0) 2024-06-03T08:36:32.672 INFO:teuthology.orchestra.run.smithi088.stderr:2024-06-03T08:36:32.672+0000 7f83feffd640 1 -- 172.21.15.88:0/3468491357 <== mon.1 v1:172.21.15.192:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 214+0+0 (unknown 2140111358 0 0) 0x7f83e8003520 con 0x7f84000ae8c0 2024-06-03T08:36:32.673 INFO:teuthology.orchestra.run.smithi088.stderr:2024-06-03T08:36:32.672+0000 7f83feffd640 1 -- 172.21.15.88:0/3468491357 >> v1:172.21.15.88:6790/0 conn(0x7f840013b2f0 legacy=0x7f8400149260 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-06-03T08:36:32.673 INFO:teuthology.orchestra.run.smithi088.stderr:2024-06-03T08:36:32.672+0000 7f83feffd640 1 -- 172.21.15.88:0/3468491357 >> v1:172.21.15.88:6789/0 conn(0x7f84000712a0 legacy=0x7f840013d940 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-06-03T08:36:32.673 INFO:teuthology.orchestra.run.smithi088.stderr:2024-06-03T08:36:32.672+0000 7f83feffd640 10 monclient: _finish_hunting 0 2024-06-03T08:36:32.673 INFO:teuthology.orchestra.run.smithi088.stderr:2024-06-03T08:36:32.672+0000 7f83feffd640 1 monclient: found mon.noname-c 2024-06-03T08:36:32.673 INFO:teuthology.orchestra.run.smithi088.stderr:2024-06-03T08:36:32.672+0000 7f83feffd640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-06-03T08:36:32.673 INFO:teuthology.orchestra.run.smithi088.stderr:2024-06-03T08:36:32.672+0000 7f83feffd640 10 monclient: _send_mon_message to mon.noname-c at v1:172.21.15.192:6789/0 2024-06-03T08:36:32.673 INFO:teuthology.orchestra.run.smithi088.stderr:2024-06-03T08:36:32.672+0000 7f83feffd640 1 -- 172.21.15.88:0/3468491357 --> v1:172.21.15.192:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f840006f600 con 0x7f84000ae8c0 2024-06-03T08:36:32.673 INFO:teuthology.orchestra.run.smithi088.stderr:2024-06-03T08:36:32.672+0000 7f83feffd640 10 monclient: _finish_auth 0 2024-06-03T08:36:32.673 INFO:teuthology.orchestra.run.smithi088.stderr:2024-06-03T08:36:32.672+0000 7f83feffd640 10 monclient: _check_auth_tickets 2024-06-03T08:36:32.673 INFO:teuthology.orchestra.run.smithi088.stderr:2024-06-03T08:36:32.672+0000 7f83feffd640 20 monclient: _check_auth_rotating not needed by client.admin 2024-06-03T08:36:32.673 INFO:teuthology.orchestra.run.smithi088.stderr:2024-06-03T08:36:32.672+0000 7f84063aa640 5 monclient: authenticate success, global_id 4345 2024-06-03T08:36:32.673 INFO:teuthology.orchestra.run.smithi088.stderr:2024-06-03T08:36:32.672+0000 7f84063aa640 20 monclient: get_monmap_and_config waiting for monmap|config 2024-06-03T08:36:32.673 INFO:teuthology.orchestra.run.smithi088.stderr:2024-06-03T08:36:32.672+0000 7f83feffd640 1 -- 172.21.15.88:0/3468491357 <== mon.1 v1:172.21.15.192:6789/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 2234374065 0 0) 0x7f83f8001990 con 0x7f84000ae8c0 2024-06-03T08:36:32.673 INFO:teuthology.orchestra.run.smithi088.stderr:2024-06-03T08:36:32.672+0000 7f83feffd640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-06-03T08:36:32.674 INFO:teuthology.orchestra.run.smithi088.stderr:2024-06-03T08:36:32.672+0000 7f83feffd640 10 monclient: got monmap 1 from mon.noname-c (according to old e1) 2024-06-03T08:36:32.674 INFO:teuthology.orchestra.run.smithi088.stderr:2024-06-03T08:36:32.672+0000 7f83feffd640 10 monclient: dump: 2024-06-03T08:36:32.674 INFO:teuthology.orchestra.run.smithi088.stderr:epoch 1 2024-06-03T08:36:32.674 INFO:teuthology.orchestra.run.smithi088.stderr:fsid ec3c3fdd-3ac6-4d7d-9412-683130ed8b04 2024-06-03T08:36:32.674 INFO:teuthology.orchestra.run.smithi088.stderr:last_changed 2024-06-03T08:34:31.885125+0000 2024-06-03T08:36:32.674 INFO:teuthology.orchestra.run.smithi088.stderr:created 2024-06-03T08:34:31.885125+0000 2024-06-03T08:36:32.674 INFO:teuthology.orchestra.run.smithi088.stderr:min_mon_release 18 (reef) 2024-06-03T08:36:32.674 INFO:teuthology.orchestra.run.smithi088.stderr:election_strategy: 3 2024-06-03T08:36:32.674 INFO:teuthology.orchestra.run.smithi088.stderr:0: v1:172.21.15.88:6789/0 mon.a 2024-06-03T08:36:32.674 INFO:teuthology.orchestra.run.smithi088.stderr:1: v1:172.21.15.192:6789/0 mon.b 2024-06-03T08:36:32.674 INFO:teuthology.orchestra.run.smithi088.stderr:2: v1:172.21.15.88:6790/0 mon.c 2024-06-03T08:36:32.674 INFO:teuthology.orchestra.run.smithi088.stderr: 2024-06-03T08:36:32.674 INFO:teuthology.orchestra.run.smithi088.stderr:2024-06-03T08:36:32.672+0000 7f84063aa640 20 monclient: get_monmap_and_config waiting for monmap|config 2024-06-03T08:36:32.675 INFO:teuthology.orchestra.run.smithi088.stderr:2024-06-03T08:36:32.673+0000 7f50db7fe640 1 -- 172.21.15.88:0/680985712 <== mon.1 v1:172.21.15.192:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 3414530441 0 0) 0x7f50dc0a8ed0 con 0x7f50dc0b4250 2024-06-03T08:36:32.675 INFO:teuthology.orchestra.run.smithi088.stderr:2024-06-03T08:36:32.673+0000 7f50db7fe640 10 monclient(hunting): _negotiate 2024-06-03T08:36:32.675 INFO:teuthology.orchestra.run.smithi088.stderr:2024-06-03T08:36:32.673+0000 7f50db7fe640 10 monclient(hunting): _init_auth method 2 2024-06-03T08:36:32.675 INFO:teuthology.orchestra.run.smithi088.stderr:2024-06-03T08:36:32.673+0000 7f50db7fe640 10 monclient(hunting): _init_auth creating new auth 2024-06-03T08:36:32.675 INFO:teuthology.orchestra.run.smithi088.stderr:2024-06-03T08:36:32.673+0000 7f50db7fe640 10 monclient(hunting): my global_id is 4342 2024-06-03T08:36:32.675 INFO:teuthology.orchestra.run.smithi088.stderr:2024-06-03T08:36:32.673+0000 7f50db7fe640 1 -- 172.21.15.88:0/680985712 --> v1:172.21.15.192:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f50cc003520 con 0x7f50dc0b4250 2024-06-03T08:36:32.677 INFO:teuthology.orchestra.run.smithi088.stderr:2024-06-03T08:36:32.676+0000 7f3d4cef6640 1 -- 172.21.15.88:0/1229108545 <== mon.1 v1:172.21.15.192:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 1234702129 0 0) 0x7f3d480a8ed0 con 0x7f3d480b4250 2024-06-03T08:36:32.677 INFO:teuthology.orchestra.run.smithi088.stderr:2024-06-03T08:36:32.676+0000 7f3d4cef6640 10 monclient(hunting): _negotiate 2024-06-03T08:36:32.677 INFO:teuthology.orchestra.run.smithi088.stderr:2024-06-03T08:36:32.676+0000 7f3d4cef6640 10 monclient(hunting): _init_auth method 2 2024-06-03T08:36:32.677 INFO:teuthology.orchestra.run.smithi088.stderr:2024-06-03T08:36:32.676+0000 7f3d4cef6640 10 monclient(hunting): _init_auth creating new auth 2024-06-03T08:36:32.677 INFO:teuthology.orchestra.run.smithi088.stderr:2024-06-03T08:36:32.676+0000 7f3d4cef6640 10 monclient(hunting): my global_id is 4348 2024-06-03T08:36:32.677 INFO:teuthology.orchestra.run.smithi088.stderr:2024-06-03T08:36:32.676+0000 7f3d4cef6640 1 -- 172.21.15.88:0/1229108545 --> v1:172.21.15.192:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f3d34003520 con 0x7f3d480b4250 2024-06-03T08:36:32.677 INFO:teuthology.orchestra.run.smithi088.stderr:2024-06-03T08:36:32.677+0000 7f16b27fc640 1 -- 172.21.15.88:0/1383755685 <== mon.1 v1:172.21.15.192:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 1217577990 0 0) 0x7f16b40a8ed0 con 0x7f16b40b4250 2024-06-03T08:36:32.677 INFO:teuthology.orchestra.run.smithi088.stderr:2024-06-03T08:36:32.677+0000 7f16b27fc640 10 monclient(hunting): _negotiate 2024-06-03T08:36:32.678 INFO:teuthology.orchestra.run.smithi088.stderr:2024-06-03T08:36:32.677+0000 7f16b27fc640 10 monclient(hunting): _init_auth method 2 2024-06-03T08:36:32.678 INFO:teuthology.orchestra.run.smithi088.stderr:2024-06-03T08:36:32.677+0000 7f16b27fc640 10 monclient(hunting): _init_auth creating new auth 2024-06-03T08:36:32.678 INFO:teuthology.orchestra.run.smithi088.stderr:2024-06-03T08:36:32.677+0000 7f16b27fc640 10 monclient(hunting): my global_id is 4351 2024-06-03T08:36:32.678 INFO:teuthology.orchestra.run.smithi088.stderr:2024-06-03T08:36:32.677+0000 7f16b27fc640 1 -- 172.21.15.88:0/1383755685 --> v1:172.21.15.192:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f169c003520 con 0x7f16b40b4250 2024-06-03T08:36:32.682 INFO:teuthology.orchestra.run.smithi088.stderr:2024-06-03T08:36:32.681+0000 7f83feffd640 1 -- 172.21.15.88:0/3468491357 <== mon.1 v1:172.21.15.192:6789/0 4 ==== config(0 keys) v1 ==== 4+0+0 (unknown 0 0 0) 0x7f83f8001c20 con 0x7f84000ae8c0 2024-06-03T08:36:32.682 INFO:teuthology.orchestra.run.smithi088.stderr:2024-06-03T08:36:32.681+0000 7f83feffd640 10 monclient: handle_config config(0 keys) v1 2024-06-03T08:36:32.682 INFO:teuthology.orchestra.run.smithi088.stderr:2024-06-03T08:36:32.681+0000 7feb16ff6640 10 monclient: get_monmap_and_config 2024-06-03T08:36:32.682 INFO:teuthology.orchestra.run.smithi088.stderr:2024-06-03T08:36:32.681+0000 7feb16ff6640 10 monclient: build_initial_monmap 2024-06-03T08:36:32.682 INFO:teuthology.orchestra.run.smithi088.stderr:2024-06-03T08:36:32.681+0000 7feb16ff6640 10 monclient: monmap: 2024-06-03T08:36:32.682 INFO:teuthology.orchestra.run.smithi088.stderr:epoch 0 2024-06-03T08:36:32.683 INFO:teuthology.orchestra.run.smithi088.stderr:fsid ec3c3fdd-3ac6-4d7d-9412-683130ed8b04 2024-06-03T08:36:32.683 INFO:teuthology.orchestra.run.smithi088.stderr:last_changed 2024-06-03T08:36:32.682993+0000 2024-06-03T08:36:32.683 INFO:teuthology.orchestra.run.smithi088.stderr:created 2024-06-03T08:36:32.682993+0000 2024-06-03T08:36:32.683 INFO:teuthology.orchestra.run.smithi088.stderr:min_mon_release 0 (unknown) 2024-06-03T08:36:32.683 INFO:teuthology.orchestra.run.smithi088.stderr:election_strategy: 3 2024-06-03T08:36:32.683 INFO:teuthology.orchestra.run.smithi088.stderr:0: v1:172.21.15.88:6789/0 mon.noname-a 2024-06-03T08:36:32.683 INFO:teuthology.orchestra.run.smithi088.stderr:1: v1:172.21.15.192:6789/0 mon.noname-c 2024-06-03T08:36:32.683 INFO:teuthology.orchestra.run.smithi088.stderr:2: v1:172.21.15.88:6790/0 mon.noname-b 2024-06-03T08:36:32.683 INFO:teuthology.orchestra.run.smithi088.stderr: 2024-06-03T08:36:32.683 INFO:teuthology.orchestra.run.smithi088.stderr:2024-06-03T08:36:32.681+0000 7f84063aa640 10 monclient: get_monmap_and_config success 2024-06-03T08:36:32.683 INFO:teuthology.orchestra.run.smithi088.stderr:2024-06-03T08:36:32.681+0000 7f84063aa640 4 set_mon_vals no callback set 2024-06-03T08:36:32.684 INFO:teuthology.orchestra.run.smithi088.stderr:2024-06-03T08:36:32.682+0000 7f83feffd640 1 -- 172.21.15.88:0/3468491357 <== mon.1 v1:172.21.15.192:6789/0 5 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 2234374065 0 0) 0x7f83f8001fb0 con 0x7f84000ae8c0 2024-06-03T08:36:32.684 INFO:teuthology.orchestra.run.smithi088.stderr:2024-06-03T08:36:32.682+0000 7f83feffd640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-06-03T08:36:32.684 INFO:teuthology.orchestra.run.smithi088.stderr:2024-06-03T08:36:32.682+0000 7f83feffd640 10 monclient: got monmap 1 from mon.b (according to old e1) 2024-06-03T08:36:32.684 INFO:teuthology.orchestra.run.smithi088.stderr:2024-06-03T08:36:32.682+0000 7f83feffd640 10 monclient: dump: 2024-06-03T08:36:32.684 INFO:teuthology.orchestra.run.smithi088.stderr:epoch 1 2024-06-03T08:36:32.684 INFO:teuthology.orchestra.run.smithi088.stderr:fsid ec3c3fdd-3ac6-4d7d-9412-683130ed8b04 2024-06-03T08:36:32.684 INFO:teuthology.orchestra.run.smithi088.stderr:last_changed 2024-06-03T08:34:31.885125+0000 2024-06-03T08:36:32.684 INFO:teuthology.orchestra.run.smithi088.stderr:created 2024-06-03T08:34:31.885125+0000 2024-06-03T08:36:32.684 INFO:teuthology.orchestra.run.smithi088.stderr:min_mon_release 18 (reef) 2024-06-03T08:36:32.684 INFO:teuthology.orchestra.run.smithi088.stderr:election_strategy: 3 2024-06-03T08:36:32.684 INFO:teuthology.orchestra.run.smithi088.stderr:0: v1:172.21.15.88:6789/0 mon.a 2024-06-03T08:36:32.684 INFO:teuthology.orchestra.run.smithi088.stderr:1: v1:172.21.15.192:6789/0 mon.b 2024-06-03T08:36:32.684 INFO:teuthology.orchestra.run.smithi088.stderr:2: v1:172.21.15.88:6790/0 mon.c 2024-06-03T08:36:32.685 INFO:teuthology.orchestra.run.smithi088.stderr: 2024-06-03T08:36:32.685 INFO:teuthology.orchestra.run.smithi088.stderr:2024-06-03T08:36:32.682+0000 7feb16ff6640 1 Processor -- start 2024-06-03T08:36:32.685 INFO:teuthology.orchestra.run.smithi088.stderr:2024-06-03T08:36:32.682+0000 7f84063aa640 10 monclient: shutdown 2024-06-03T08:36:32.685 INFO:teuthology.orchestra.run.smithi088.stderr:2024-06-03T08:36:32.682+0000 7f84063aa640 20 monclient: shutdown discarding 0 pending message(s) 2024-06-03T08:36:32.685 INFO:teuthology.orchestra.run.smithi088.stderr:2024-06-03T08:36:32.682+0000 7f84063aa640 1 -- 172.21.15.88:0/3468491357 >> v1:172.21.15.192:6789/0 conn(0x7f84000ae8c0 legacy=0x7f840013abe0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-06-03T08:36:32.686 INFO:teuthology.orchestra.run.smithi088.stderr:2024-06-03T08:36:32.682+0000 7feb16ff6640 1 -- start start 2024-06-03T08:36:32.686 INFO:teuthology.orchestra.run.smithi088.stderr:2024-06-03T08:36:32.682+0000 7feb16ff6640 10 monclient: init 2024-06-03T08:36:32.686 INFO:teuthology.orchestra.run.smithi088.stderr:2024-06-03T08:36:32.682+0000 7f84063aa640 1 -- 172.21.15.88:0/3468491357 shutdown_connections 2024-06-03T08:36:32.686 INFO:teuthology.orchestra.run.smithi088.stderr:2024-06-03T08:36:32.683+0000 7f84063aa640 1 -- 172.21.15.88:0/3468491357 >> 172.21.15.88:0/3468491357 conn(0x7f84000b2f20 msgr2=0x7f84000b3300 unknown :-1 s=STATE_NONE l=0).mark_down 2024-06-03T08:36:32.687 INFO:teuthology.orchestra.run.smithi088.stderr:2024-06-03T08:36:32.683+0000 7f84063aa640 1 -- 172.21.15.88:0/3468491357 shutdown_connections 2024-06-03T08:36:32.687 INFO:teuthology.orchestra.run.smithi088.stderr:2024-06-03T08:36:32.683+0000 7f84063aa640 1 -- 172.21.15.88:0/3468491357 wait complete. 2024-06-03T08:36:32.687 INFO:teuthology.orchestra.run.smithi088.stderr:2024-06-03T08:36:32.683+0000 7feb16ff6640 10 monclient: _reopen_session rank -1 2024-06-03T08:36:32.687 INFO:teuthology.orchestra.run.smithi088.stderr:2024-06-03T08:36:32.683+0000 7feb16ff6640 10 monclient: _add_conns ranks=[1,0,2] 2024-06-03T08:36:32.687 INFO:teuthology.orchestra.run.smithi088.stderr:2024-06-03T08:36:32.683+0000 7feb16ff6640 10 monclient(hunting): picked mon.noname-c con 0x7feb100a8960 addr v1:172.21.15.192:6789/0 2024-06-03T08:36:32.687 INFO:teuthology.orchestra.run.smithi088.stderr:2024-06-03T08:36:32.683+0000 7feb16ff6640 10 monclient(hunting): picked mon.noname-a con 0x7feb101450f0 addr v1:172.21.15.88:6789/0 2024-06-03T08:36:32.688 INFO:teuthology.orchestra.run.smithi088.stderr:2024-06-03T08:36:32.683+0000 7f84063aa640 10 monclient: build_initial_monmap 2024-06-03T08:36:32.688 INFO:teuthology.orchestra.run.smithi088.stderr:2024-06-03T08:36:32.683+0000 7f84063aa640 10 monclient: monmap: 2024-06-03T08:36:32.688 INFO:teuthology.orchestra.run.smithi088.stderr:epoch 0 2024-06-03T08:36:32.688 INFO:teuthology.orchestra.run.smithi088.stderr:fsid 00000000-0000-0000-0000-000000000000 2024-06-03T08:36:32.688 INFO:teuthology.orchestra.run.smithi088.stderr:last_changed 0.000000 2024-06-03T08:36:32.688 INFO:teuthology.orchestra.run.smithi088.stderr:created 0.000000 2024-06-03T08:36:32.688 INFO:teuthology.orchestra.run.smithi088.stderr:min_mon_release 0 (unknown) 2024-06-03T08:36:32.688 INFO:teuthology.orchestra.run.smithi088.stderr:election_strategy: 1 2024-06-03T08:36:32.688 INFO:teuthology.orchestra.run.smithi088.stderr:0: v1:172.21.15.88:6789/0 mon.noname-a 2024-06-03T08:36:32.688 INFO:teuthology.orchestra.run.smithi088.stderr:1: v1:172.21.15.192:6789/0 mon.noname-b 2024-06-03T08:36:32.688 INFO:teuthology.orchestra.run.smithi088.stderr:2: v1:172.21.15.88:6790/0 mon.noname-c 2024-06-03T08:36:32.689 INFO:teuthology.orchestra.run.smithi088.stderr: 2024-06-03T08:36:32.689 INFO:teuthology.orchestra.run.smithi088.stderr:2024-06-03T08:36:32.683+0000 7feb16ff6640 10 monclient(hunting): picked mon.noname-b con 0x7feb1013eec0 addr v1:172.21.15.88:6790/0 2024-06-03T08:36:32.689 INFO:teuthology.orchestra.run.smithi088.stderr:2024-06-03T08:36:32.683+0000 7feb16ff6640 1 -- --> v1:172.21.15.88:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7feb10144c60 con 0x7feb101450f0 2024-06-03T08:36:32.689 INFO:teuthology.orchestra.run.smithi088.stderr:2024-06-03T08:36:32.684+0000 7feb16ff6640 1 -- --> v1:172.21.15.192:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7feb100b3930 con 0x7feb100a8960 2024-06-03T08:36:32.689 INFO:teuthology.orchestra.run.smithi088.stderr:2024-06-03T08:36:32.684+0000 7feb16ff6640 1 -- --> v1:172.21.15.88:6790/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7feb100b1ad0 con 0x7feb1013eec0 2024-06-03T08:36:32.689 INFO:teuthology.orchestra.run.smithi088.stderr:2024-06-03T08:36:32.684+0000 7feb16ff6640 10 monclient(hunting): _renew_subs 2024-06-03T08:36:32.689 INFO:teuthology.orchestra.run.smithi088.stderr:2024-06-03T08:36:32.684+0000 7feb16ff6640 10 monclient(hunting): authenticate will time out at 1248.493989s 2024-06-03T08:36:32.690 INFO:teuthology.orchestra.run.smithi088.stderr:2024-06-03T08:36:32.684+0000 7f84063aa640 1 librados: starting msgr at 2024-06-03T08:36:32.690 INFO:teuthology.orchestra.run.smithi088.stderr:2024-06-03T08:36:32.684+0000 7f84063aa640 1 librados: starting objecter 2024-06-03T08:36:32.690 INFO:teuthology.orchestra.run.smithi088.stderr:2024-06-03T08:36:32.684+0000 7f84063aa640 1 Processor -- start 2024-06-03T08:36:32.690 INFO:teuthology.orchestra.run.smithi088.stderr:2024-06-03T08:36:32.684+0000 7f84063aa640 1 -- start start 2024-06-03T08:36:32.690 INFO:teuthology.orchestra.run.smithi088.stderr:2024-06-03T08:36:32.684+0000 7f84063aa640 1 librados: setting wanted keys 2024-06-03T08:36:32.690 INFO:teuthology.orchestra.run.smithi088.stderr:2024-06-03T08:36:32.684+0000 7f84063aa640 1 librados: calling monclient init 2024-06-03T08:36:32.691 INFO:teuthology.orchestra.run.smithi088.stderr:2024-06-03T08:36:32.684+0000 7f84063aa640 10 monclient: init 2024-06-03T08:36:32.691 INFO:teuthology.orchestra.run.smithi088.stderr:2024-06-03T08:36:32.684+0000 7f3d4cef6640 1 -- 172.21.15.88:0/1229108545 <== mon.1 v1:172.21.15.192:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 214+0+0 (unknown 258355808 0 0) 0x7f3d34003520 con 0x7f3d480b4250 2024-06-03T08:36:32.691 INFO:teuthology.orchestra.run.smithi088.stderr:2024-06-03T08:36:32.684+0000 7f3d4cef6640 1 -- 172.21.15.88:0/1229108545 >> v1:172.21.15.88:6790/0 conn(0x7f3d481451c0 legacy=0x7f3d4814ce50 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-06-03T08:36:32.691 INFO:teuthology.orchestra.run.smithi088.stderr:2024-06-03T08:36:32.684+0000 7f3d4cef6640 1 -- 172.21.15.88:0/1229108545 >> v1:172.21.15.88:6789/0 conn(0x7f3d480a89f0 legacy=0x7f3d48144ab0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-06-03T08:36:32.691 INFO:teuthology.orchestra.run.smithi088.stderr:2024-06-03T08:36:32.684+0000 7f3d4cef6640 10 monclient: _finish_hunting 0 2024-06-03T08:36:32.691 INFO:teuthology.orchestra.run.smithi088.stderr:2024-06-03T08:36:32.684+0000 7f3d4cef6640 1 monclient: found mon.noname-c 2024-06-03T08:36:32.691 INFO:teuthology.orchestra.run.smithi088.stderr:2024-06-03T08:36:32.684+0000 7f3d4cef6640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-06-03T08:36:32.691 INFO:teuthology.orchestra.run.smithi088.stderr:2024-06-03T08:36:32.684+0000 7f3d4cef6640 10 monclient: _send_mon_message to mon.noname-c at v1:172.21.15.192:6789/0 2024-06-03T08:36:32.691 INFO:teuthology.orchestra.run.smithi088.stderr:2024-06-03T08:36:32.684+0000 7f3d4cef6640 1 -- 172.21.15.88:0/1229108545 --> v1:172.21.15.192:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f3d480b39a0 con 0x7f3d480b4250 2024-06-03T08:36:32.692 INFO:teuthology.orchestra.run.smithi088.stderr:2024-06-03T08:36:32.684+0000 7f3d4cef6640 10 monclient: _finish_auth 0 2024-06-03T08:36:32.692 INFO:teuthology.orchestra.run.smithi088.stderr:2024-06-03T08:36:32.684+0000 7f3d4cef6640 10 monclient: _check_auth_tickets 2024-06-03T08:36:32.692 INFO:teuthology.orchestra.run.smithi088.stderr:2024-06-03T08:36:32.684+0000 7f3d4cef6640 20 monclient: _check_auth_rotating not needed by client.admin 2024-06-03T08:36:32.692 INFO:teuthology.orchestra.run.smithi088.stderr:2024-06-03T08:36:32.684+0000 7f3d4cef6640 1 -- 172.21.15.88:0/1229108545 <== mon.1 v1:172.21.15.192:6789/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 2234374065 0 0) 0x7f3d44001610 con 0x7f3d480b4250 2024-06-03T08:36:32.692 INFO:teuthology.orchestra.run.smithi088.stderr:2024-06-03T08:36:32.684+0000 7f3d5015c640 5 monclient: authenticate success, global_id 4348 2024-06-03T08:36:32.692 INFO:teuthology.orchestra.run.smithi088.stderr:2024-06-03T08:36:32.684+0000 7f3d5015c640 20 monclient: get_monmap_and_config waiting for monmap|config 2024-06-03T08:36:32.692 INFO:teuthology.orchestra.run.smithi088.stderr:2024-06-03T08:36:32.684+0000 7f3d4cef6640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-06-03T08:36:32.692 INFO:teuthology.orchestra.run.smithi088.stderr:2024-06-03T08:36:32.685+0000 7f3d4cef6640 10 monclient: got monmap 1 from mon.noname-c (according to old e1) 2024-06-03T08:36:32.692 INFO:teuthology.orchestra.run.smithi088.stderr:2024-06-03T08:36:32.685+0000 7f3d4cef6640 10 monclient: dump: 2024-06-03T08:36:32.692 INFO:teuthology.orchestra.run.smithi088.stderr:epoch 1 2024-06-03T08:36:32.692 INFO:teuthology.orchestra.run.smithi088.stderr:fsid ec3c3fdd-3ac6-4d7d-9412-683130ed8b04 2024-06-03T08:36:32.692 INFO:teuthology.orchestra.run.smithi088.stderr:last_changed 2024-06-03T08:34:31.885125+0000 2024-06-03T08:36:32.692 INFO:teuthology.orchestra.run.smithi088.stderr:created 2024-06-03T08:34:31.885125+0000 2024-06-03T08:36:32.693 INFO:teuthology.orchestra.run.smithi088.stderr:min_mon_release 18 (reef) 2024-06-03T08:36:32.693 INFO:teuthology.orchestra.run.smithi088.stderr:election_strategy: 3 2024-06-03T08:36:32.693 INFO:teuthology.orchestra.run.smithi088.stderr:0: v1:172.21.15.88:6789/0 mon.a 2024-06-03T08:36:32.693 INFO:teuthology.orchestra.run.smithi088.stderr:1: v1:172.21.15.192:6789/0 mon.b 2024-06-03T08:36:32.693 INFO:teuthology.orchestra.run.smithi088.stderr:2: v1:172.21.15.88:6790/0 mon.c 2024-06-03T08:36:32.693 INFO:teuthology.orchestra.run.smithi088.stderr: 2024-06-03T08:36:32.693 INFO:teuthology.orchestra.run.smithi088.stderr:2024-06-03T08:36:32.685+0000 7f3d5015c640 20 monclient: get_monmap_and_config waiting for monmap|config 2024-06-03T08:36:32.694 INFO:teuthology.orchestra.run.smithi088.stderr:2024-06-03T08:36:32.685+0000 7f84063aa640 10 monclient: _reopen_session rank -1 2024-06-03T08:36:32.694 INFO:teuthology.orchestra.run.smithi088.stderr:2024-06-03T08:36:32.685+0000 7f84063aa640 10 monclient: _add_conns ranks=[0,1,2] 2024-06-03T08:36:32.694 INFO:teuthology.orchestra.run.smithi088.stderr:2024-06-03T08:36:32.685+0000 7f84063aa640 10 monclient(hunting): picked mon.noname-a con 0x7f84000712a0 addr v1:172.21.15.88:6789/0 2024-06-03T08:36:32.694 INFO:teuthology.orchestra.run.smithi088.stderr:2024-06-03T08:36:32.685+0000 7f84063aa640 10 monclient(hunting): picked mon.noname-b con 0x7f84000ae8c0 addr v1:172.21.15.192:6789/0 2024-06-03T08:36:32.694 INFO:teuthology.orchestra.run.smithi088.stderr:2024-06-03T08:36:32.685+0000 7f84063aa640 10 monclient(hunting): picked mon.noname-c con 0x7f840013b2f0 addr v1:172.21.15.88:6790/0 2024-06-03T08:36:32.694 INFO:teuthology.orchestra.run.smithi088.stderr:2024-06-03T08:36:32.685+0000 7f84063aa640 1 -- --> v1:172.21.15.88:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f8400150630 con 0x7f84000712a0 2024-06-03T08:36:32.694 INFO:teuthology.orchestra.run.smithi088.stderr:2024-06-03T08:36:32.685+0000 7f84063aa640 1 -- --> v1:172.21.15.192:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f840013f480 con 0x7f84000ae8c0 2024-06-03T08:36:32.694 INFO:teuthology.orchestra.run.smithi088.stderr:2024-06-03T08:36:32.685+0000 7f84063aa640 1 -- --> v1:172.21.15.88:6790/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f840014fe70 con 0x7f840013b2f0 2024-06-03T08:36:32.694 INFO:teuthology.orchestra.run.smithi088.stderr:2024-06-03T08:36:32.685+0000 7f84063aa640 10 monclient(hunting): _renew_subs 2024-06-03T08:36:32.694 INFO:teuthology.orchestra.run.smithi088.stderr:2024-06-03T08:36:32.685+0000 7f84063aa640 10 monclient(hunting): authenticate will time out at 1248.495249s 2024-06-03T08:36:32.695 INFO:teuthology.orchestra.run.smithi088.stderr:2024-06-03T08:36:32.687+0000 7f50db7fe640 1 -- 172.21.15.88:0/680985712 <== mon.1 v1:172.21.15.192:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 214+0+0 (unknown 3578320378 0 0) 0x7f50cc003520 con 0x7f50dc0b4250 2024-06-03T08:36:32.695 INFO:teuthology.orchestra.run.smithi088.stderr:2024-06-03T08:36:32.687+0000 7f50db7fe640 1 -- 172.21.15.88:0/680985712 >> v1:172.21.15.88:6790/0 conn(0x7f50dc1451c0 legacy=0x7f50dc14ce50 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-06-03T08:36:32.695 INFO:teuthology.orchestra.run.smithi088.stderr:2024-06-03T08:36:32.687+0000 7f50db7fe640 1 -- 172.21.15.88:0/680985712 >> v1:172.21.15.88:6789/0 conn(0x7f50dc0a89f0 legacy=0x7f50dc144ab0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-06-03T08:36:32.696 INFO:teuthology.orchestra.run.smithi088.stderr:2024-06-03T08:36:32.687+0000 7f50db7fe640 10 monclient: _finish_hunting 0 2024-06-03T08:36:32.696 INFO:teuthology.orchestra.run.smithi088.stderr:2024-06-03T08:36:32.687+0000 7f50db7fe640 1 monclient: found mon.noname-c 2024-06-03T08:36:32.696 INFO:teuthology.orchestra.run.smithi088.stderr:2024-06-03T08:36:32.687+0000 7f50db7fe640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-06-03T08:36:32.696 INFO:teuthology.orchestra.run.smithi088.stderr:2024-06-03T08:36:32.687+0000 7f50db7fe640 10 monclient: _send_mon_message to mon.noname-c at v1:172.21.15.192:6789/0 2024-06-03T08:36:32.696 INFO:teuthology.orchestra.run.smithi088.stderr:2024-06-03T08:36:32.687+0000 7f50db7fe640 1 -- 172.21.15.88:0/680985712 --> v1:172.21.15.192:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f50dc0b39a0 con 0x7f50dc0b4250 2024-06-03T08:36:32.696 INFO:teuthology.orchestra.run.smithi088.stderr:2024-06-03T08:36:32.687+0000 7f50db7fe640 10 monclient: _finish_auth 0 2024-06-03T08:36:32.696 INFO:teuthology.orchestra.run.smithi088.stderr:2024-06-03T08:36:32.687+0000 7f50db7fe640 10 monclient: _check_auth_tickets 2024-06-03T08:36:32.696 INFO:teuthology.orchestra.run.smithi088.stderr:2024-06-03T08:36:32.687+0000 7f50db7fe640 20 monclient: _check_auth_rotating not needed by client.admin 2024-06-03T08:36:32.696 INFO:teuthology.orchestra.run.smithi088.stderr:2024-06-03T08:36:32.687+0000 7f50e2d7d640 5 monclient: authenticate success, global_id 4342 2024-06-03T08:36:32.696 INFO:teuthology.orchestra.run.smithi088.stderr:2024-06-03T08:36:32.687+0000 7f50db7fe640 1 -- 172.21.15.88:0/680985712 <== mon.1 v1:172.21.15.192:6789/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 2234374065 0 0) 0x7f50d00015f0 con 0x7f50dc0b4250 2024-06-03T08:36:32.697 INFO:teuthology.orchestra.run.smithi088.stderr:2024-06-03T08:36:32.687+0000 7f50e2d7d640 20 monclient: get_monmap_and_config waiting for monmap|config 2024-06-03T08:36:32.697 INFO:teuthology.orchestra.run.smithi088.stderr:2024-06-03T08:36:32.687+0000 7f50db7fe640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-06-03T08:36:32.697 INFO:teuthology.orchestra.run.smithi088.stderr:2024-06-03T08:36:32.687+0000 7f50db7fe640 10 monclient: got monmap 1 from mon.noname-c (according to old e1) 2024-06-03T08:36:32.697 INFO:teuthology.orchestra.run.smithi088.stderr:2024-06-03T08:36:32.687+0000 7f50db7fe640 10 monclient: dump: 2024-06-03T08:36:32.697 INFO:teuthology.orchestra.run.smithi088.stderr:epoch 1 2024-06-03T08:36:32.697 INFO:teuthology.orchestra.run.smithi088.stderr:fsid ec3c3fdd-3ac6-4d7d-9412-683130ed8b04 2024-06-03T08:36:32.697 INFO:teuthology.orchestra.run.smithi088.stderr:last_changed 2024-06-03T08:34:31.885125+0000 2024-06-03T08:36:32.697 INFO:teuthology.orchestra.run.smithi088.stderr:created 2024-06-03T08:34:31.885125+0000 2024-06-03T08:36:32.697 INFO:teuthology.orchestra.run.smithi088.stderr:min_mon_release 18 (reef) 2024-06-03T08:36:32.697 INFO:teuthology.orchestra.run.smithi088.stderr:election_strategy: 3 2024-06-03T08:36:32.697 INFO:teuthology.orchestra.run.smithi088.stderr:0: v1:172.21.15.88:6789/0 mon.a 2024-06-03T08:36:32.697 INFO:teuthology.orchestra.run.smithi088.stderr:1: v1:172.21.15.192:6789/0 mon.b 2024-06-03T08:36:32.697 INFO:teuthology.orchestra.run.smithi088.stderr:2: v1:172.21.15.88:6790/0 mon.c 2024-06-03T08:36:32.697 INFO:teuthology.orchestra.run.smithi088.stderr: 2024-06-03T08:36:32.698 INFO:teuthology.orchestra.run.smithi088.stderr:2024-06-03T08:36:32.687+0000 7f50e2d7d640 20 monclient: get_monmap_and_config waiting for monmap|config 2024-06-03T08:36:32.702 INFO:teuthology.orchestra.run.smithi088.stderr:2024-06-03T08:36:32.689+0000 7f16b27fc640 1 -- 172.21.15.88:0/1383755685 <== mon.1 v1:172.21.15.192:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 214+0+0 (unknown 1197551152 0 0) 0x7f169c003520 con 0x7f16b40b4250 2024-06-03T08:36:32.702 INFO:teuthology.orchestra.run.smithi088.stderr:2024-06-03T08:36:32.689+0000 7f16b27fc640 1 -- 172.21.15.88:0/1383755685 >> v1:172.21.15.88:6790/0 conn(0x7f16b40a89f0 legacy=0x7f16b4144ab0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-06-03T08:36:32.702 INFO:teuthology.orchestra.run.smithi088.stderr:2024-06-03T08:36:32.689+0000 7f16b27fc640 1 -- 172.21.15.88:0/1383755685 >> v1:172.21.15.88:6789/0 conn(0x7f16b41451c0 legacy=0x7f16b414ce50 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-06-03T08:36:32.703 INFO:teuthology.orchestra.run.smithi088.stderr:2024-06-03T08:36:32.689+0000 7f16b27fc640 10 monclient: _finish_hunting 0 2024-06-03T08:36:32.703 INFO:teuthology.orchestra.run.smithi088.stderr:2024-06-03T08:36:32.689+0000 7f16b27fc640 1 monclient: found mon.noname-c 2024-06-03T08:36:32.703 INFO:teuthology.orchestra.run.smithi088.stderr:2024-06-03T08:36:32.689+0000 7f16b27fc640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-06-03T08:36:32.703 INFO:teuthology.orchestra.run.smithi088.stderr:2024-06-03T08:36:32.689+0000 7f16b27fc640 10 monclient: _send_mon_message to mon.noname-c at v1:172.21.15.192:6789/0 2024-06-03T08:36:32.703 INFO:teuthology.orchestra.run.smithi088.stderr:2024-06-03T08:36:32.689+0000 7f16b27fc640 1 -- 172.21.15.88:0/1383755685 --> v1:172.21.15.192:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f16b40b39a0 con 0x7f16b40b4250 2024-06-03T08:36:32.703 INFO:teuthology.orchestra.run.smithi088.stderr:2024-06-03T08:36:32.689+0000 7f16b27fc640 10 monclient: _finish_auth 0 2024-06-03T08:36:32.703 INFO:teuthology.orchestra.run.smithi088.stderr:2024-06-03T08:36:32.689+0000 7f16b27fc640 10 monclient: _check_auth_tickets 2024-06-03T08:36:32.703 INFO:teuthology.orchestra.run.smithi088.stderr:2024-06-03T08:36:32.689+0000 7f16b27fc640 20 monclient: _check_auth_rotating not needed by client.admin 2024-06-03T08:36:32.703 INFO:teuthology.orchestra.run.smithi088.stderr:2024-06-03T08:36:32.689+0000 7f16b27fc640 1 -- 172.21.15.88:0/1383755685 <== mon.1 v1:172.21.15.192:6789/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 2234374065 0 0) 0x7f16a4001610 con 0x7f16b40b4250 2024-06-03T08:36:32.703 INFO:teuthology.orchestra.run.smithi088.stderr:2024-06-03T08:36:32.689+0000 7f16b27fc640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-06-03T08:36:32.703 INFO:teuthology.orchestra.run.smithi088.stderr:2024-06-03T08:36:32.689+0000 7f16b27fc640 10 monclient: got monmap 1 from mon.noname-c (according to old e1) 2024-06-03T08:36:32.703 INFO:teuthology.orchestra.run.smithi088.stderr:2024-06-03T08:36:32.689+0000 7f16b27fc640 10 monclient: dump: 2024-06-03T08:36:32.703 INFO:teuthology.orchestra.run.smithi088.stderr:epoch 1 2024-06-03T08:36:32.704 INFO:teuthology.orchestra.run.smithi088.stderr:fsid ec3c3fdd-3ac6-4d7d-9412-683130ed8b04 2024-06-03T08:36:32.704 INFO:teuthology.orchestra.run.smithi088.stderr:last_changed 2024-06-03T08:34:31.885125+0000 2024-06-03T08:36:32.704 INFO:teuthology.orchestra.run.smithi088.stderr:created 2024-06-03T08:34:31.885125+0000 2024-06-03T08:36:32.704 INFO:teuthology.orchestra.run.smithi088.stderr:min_mon_release 18 (reef) 2024-06-03T08:36:32.704 INFO:teuthology.orchestra.run.smithi088.stderr:election_strategy: 3 2024-06-03T08:36:32.704 INFO:teuthology.orchestra.run.smithi088.stderr:0: v1:172.21.15.88:6789/0 mon.a 2024-06-03T08:36:32.704 INFO:teuthology.orchestra.run.smithi088.stderr:1: v1:172.21.15.192:6789/0 mon.b 2024-06-03T08:36:32.704 INFO:teuthology.orchestra.run.smithi088.stderr:2: v1:172.21.15.88:6790/0 mon.c 2024-06-03T08:36:32.704 INFO:teuthology.orchestra.run.smithi088.stderr: 2024-06-03T08:36:32.704 INFO:teuthology.orchestra.run.smithi088.stderr:2024-06-03T08:36:32.689+0000 7f16b9c8f640 5 monclient: authenticate success, global_id 4351 2024-06-03T08:36:32.704 INFO:teuthology.orchestra.run.smithi088.stderr:2024-06-03T08:36:32.689+0000 7f16b9c8f640 20 monclient: get_monmap_and_config waiting for monmap|config 2024-06-03T08:36:32.705 INFO:teuthology.orchestra.run.smithi088.stderr:2024-06-03T08:36:32.690+0000 7f83ff7fe640 1 --1- >> v1:172.21.15.192:6789/0 conn(0x7f84000ae8c0 0x7f84001d9b00 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=1).handle_server_banner_and_identify peer v1:172.21.15.192:6789/0 says I am v1:172.21.15.88:52576/0 (socket says 172.21.15.88:52576) 2024-06-03T08:36:32.705 INFO:teuthology.orchestra.run.smithi088.stderr:2024-06-03T08:36:32.690+0000 7f83ff7fe640 1 -- 172.21.15.88:0/4283667430 learned_addr learned my addr 172.21.15.88:0/4283667430 (peer_addr_for_me v1:172.21.15.88:0/0) 2024-06-03T08:36:32.705 INFO:teuthology.orchestra.run.smithi088.stderr:2024-06-03T08:36:32.690+0000 7feb14d92640 1 --1- >> v1:172.21.15.192:6789/0 conn(0x7feb100a8960 0x7feb100a8d40 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.192:6789/0 says I am v1:172.21.15.88:52566/0 (socket says 172.21.15.88:52566) 2024-06-03T08:36:32.705 INFO:teuthology.orchestra.run.smithi088.stderr:2024-06-03T08:36:32.691+0000 7feb14d92640 1 -- 172.21.15.88:0/978336162 learned_addr learned my addr 172.21.15.88:0/978336162 (peer_addr_for_me v1:172.21.15.88:0/0) 2024-06-03T08:36:32.705 INFO:teuthology.orchestra.run.smithi088.stderr:2024-06-03T08:36:32.692+0000 7f83fcff9640 10 client.?.objecter ms_handle_connect 0x7f84000ae8c0 2024-06-03T08:36:32.705 INFO:teuthology.orchestra.run.smithi088.stderr:2024-06-03T08:36:32.692+0000 7f83fcff9640 10 client.?.objecter resend_mon_ops 2024-06-03T08:36:32.706 INFO:teuthology.orchestra.run.smithi088.stderr:2024-06-03T08:36:32.693+0000 7f3d4cef6640 1 -- 172.21.15.88:0/1229108545 <== mon.1 v1:172.21.15.192:6789/0 4 ==== config(0 keys) v1 ==== 4+0+0 (unknown 0 0 0) 0x7f3d44001830 con 0x7f3d480b4250 2024-06-03T08:36:32.706 INFO:teuthology.orchestra.run.smithi088.stderr:2024-06-03T08:36:32.693+0000 7f3d4cef6640 10 monclient: handle_config config(0 keys) v1 2024-06-03T08:36:32.706 INFO:teuthology.orchestra.run.smithi088.stderr:2024-06-03T08:36:32.693+0000 7f3d4cef6640 1 -- 172.21.15.88:0/1229108545 <== mon.1 v1:172.21.15.192:6789/0 5 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 2234374065 0 0) 0x7f3d44001bc0 con 0x7f3d480b4250 2024-06-03T08:36:32.706 INFO:teuthology.orchestra.run.smithi088.stderr:2024-06-03T08:36:32.693+0000 7f3d4cef6640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-06-03T08:36:32.706 INFO:teuthology.orchestra.run.smithi088.stderr:2024-06-03T08:36:32.693+0000 7f3d4cef6640 10 monclient: got monmap 1 from mon.b (according to old e1) 2024-06-03T08:36:32.706 INFO:teuthology.orchestra.run.smithi088.stderr:2024-06-03T08:36:32.693+0000 7f3d4cef6640 10 monclient: dump: 2024-06-03T08:36:32.707 INFO:teuthology.orchestra.run.smithi088.stderr:epoch 1 2024-06-03T08:36:32.707 INFO:teuthology.orchestra.run.smithi088.stderr:fsid ec3c3fdd-3ac6-4d7d-9412-683130ed8b04 2024-06-03T08:36:32.707 INFO:teuthology.orchestra.run.smithi088.stderr:last_changed 2024-06-03T08:34:31.885125+0000 2024-06-03T08:36:32.707 INFO:teuthology.orchestra.run.smithi088.stderr:created 2024-06-03T08:34:31.885125+0000 2024-06-03T08:36:32.707 INFO:teuthology.orchestra.run.smithi088.stderr:min_mon_release 18 (reef) 2024-06-03T08:36:32.707 INFO:teuthology.orchestra.run.smithi088.stderr:election_strategy: 3 2024-06-03T08:36:32.707 INFO:teuthology.orchestra.run.smithi088.stderr:0: v1:172.21.15.88:6789/0 mon.a 2024-06-03T08:36:32.707 INFO:teuthology.orchestra.run.smithi088.stderr:1: v1:172.21.15.192:6789/0 mon.b 2024-06-03T08:36:32.707 INFO:teuthology.orchestra.run.smithi088.stderr:2: v1:172.21.15.88:6790/0 mon.c 2024-06-03T08:36:32.707 INFO:teuthology.orchestra.run.smithi088.stderr: 2024-06-03T08:36:32.707 INFO:teuthology.orchestra.run.smithi088.stderr:2024-06-03T08:36:32.693+0000 7f3d5015c640 10 monclient: get_monmap_and_config success 2024-06-03T08:36:32.707 INFO:teuthology.orchestra.run.smithi088.stderr:2024-06-03T08:36:32.693+0000 7f3d5015c640 4 set_mon_vals no callback set 2024-06-03T08:36:32.708 INFO:teuthology.orchestra.run.smithi088.stderr:2024-06-03T08:36:32.693+0000 7f16b27fc640 1 -- 172.21.15.88:0/1383755685 <== mon.1 v1:172.21.15.192:6789/0 4 ==== config(0 keys) v1 ==== 4+0+0 (unknown 0 0 0) 0x7f16a40018a0 con 0x7f16b40b4250 2024-06-03T08:36:32.708 INFO:teuthology.orchestra.run.smithi088.stderr:2024-06-03T08:36:32.693+0000 7f16b27fc640 10 monclient: handle_config config(0 keys) v1 2024-06-03T08:36:32.708 INFO:teuthology.orchestra.run.smithi088.stderr:2024-06-03T08:36:32.693+0000 7f16b27fc640 1 -- 172.21.15.88:0/1383755685 <== mon.1 v1:172.21.15.192:6789/0 5 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 2234374065 0 0) 0x7f16a4001c30 con 0x7f16b40b4250 2024-06-03T08:36:32.708 INFO:teuthology.orchestra.run.smithi088.stderr:2024-06-03T08:36:32.693+0000 7f16b27fc640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-06-03T08:36:32.708 INFO:teuthology.orchestra.run.smithi088.stderr:2024-06-03T08:36:32.693+0000 7f16b27fc640 10 monclient: got monmap 1 from mon.b (according to old e1) 2024-06-03T08:36:32.708 INFO:teuthology.orchestra.run.smithi088.stderr:2024-06-03T08:36:32.693+0000 7f16b27fc640 10 monclient: dump: 2024-06-03T08:36:32.709 INFO:teuthology.orchestra.run.smithi088.stderr:epoch 1 2024-06-03T08:36:32.709 INFO:teuthology.orchestra.run.smithi088.stderr:fsid ec3c3fdd-3ac6-4d7d-9412-683130ed8b04 2024-06-03T08:36:32.709 INFO:teuthology.orchestra.run.smithi088.stderr:last_changed 2024-06-03T08:34:31.885125+0000 2024-06-03T08:36:32.709 INFO:teuthology.orchestra.run.smithi088.stderr:created 2024-06-03T08:34:31.885125+0000 2024-06-03T08:36:32.709 INFO:teuthology.orchestra.run.smithi088.stderr:min_mon_release 18 (reef) 2024-06-03T08:36:32.709 INFO:teuthology.orchestra.run.smithi088.stderr:election_strategy: 3 2024-06-03T08:36:32.709 INFO:teuthology.orchestra.run.smithi088.stderr:0: v1:172.21.15.88:6789/0 mon.a 2024-06-03T08:36:32.709 INFO:teuthology.orchestra.run.smithi088.stderr:1: v1:172.21.15.192:6789/0 mon.b 2024-06-03T08:36:32.709 INFO:teuthology.orchestra.run.smithi088.stderr:2: v1:172.21.15.88:6790/0 mon.c 2024-06-03T08:36:32.709 INFO:teuthology.orchestra.run.smithi088.stderr: 2024-06-03T08:36:32.709 INFO:teuthology.orchestra.run.smithi088.stderr:2024-06-03T08:36:32.693+0000 7f16b9c8f640 10 monclient: get_monmap_and_config success 2024-06-03T08:36:32.709 INFO:teuthology.orchestra.run.smithi088.stderr:2024-06-03T08:36:32.693+0000 7f16b9c8f640 4 set_mon_vals no callback set 2024-06-03T08:36:32.710 INFO:teuthology.orchestra.run.smithi088.stderr:2024-06-03T08:36:32.694+0000 7f3d5015c640 10 monclient: shutdown 2024-06-03T08:36:32.710 INFO:teuthology.orchestra.run.smithi088.stderr:2024-06-03T08:36:32.694+0000 7f3d5015c640 20 monclient: shutdown discarding 0 pending message(s) 2024-06-03T08:36:32.710 INFO:teuthology.orchestra.run.smithi088.stderr:2024-06-03T08:36:32.694+0000 7f3d5015c640 1 -- 172.21.15.88:0/1229108545 >> v1:172.21.15.192:6789/0 conn(0x7f3d480b4250 legacy=0x7f3d4813e9e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-06-03T08:36:32.710 INFO:teuthology.orchestra.run.smithi088.stderr:2024-06-03T08:36:32.694+0000 7f3d5015c640 1 -- 172.21.15.88:0/1229108545 shutdown_connections 2024-06-03T08:36:32.710 INFO:teuthology.orchestra.run.smithi088.stderr:2024-06-03T08:36:32.694+0000 7f3d5015c640 1 -- 172.21.15.88:0/1229108545 >> 172.21.15.88:0/1229108545 conn(0x7f3d480717f0 msgr2=0x7f3d48071c00 unknown :-1 s=STATE_NONE l=0).mark_down 2024-06-03T08:36:32.710 INFO:teuthology.orchestra.run.smithi088.stderr:2024-06-03T08:36:32.694+0000 7f3d5015c640 1 -- 172.21.15.88:0/1229108545 shutdown_connections 2024-06-03T08:36:32.711 INFO:teuthology.orchestra.run.smithi088.stderr:2024-06-03T08:36:32.694+0000 7f3d5015c640 1 -- 172.21.15.88:0/1229108545 wait complete. 2024-06-03T08:36:32.711 INFO:teuthology.orchestra.run.smithi088.stderr:2024-06-03T08:36:32.694+0000 7f16b9c8f640 10 monclient: shutdown 2024-06-03T08:36:32.711 INFO:teuthology.orchestra.run.smithi088.stderr:2024-06-03T08:36:32.694+0000 7f16b9c8f640 20 monclient: shutdown discarding 0 pending message(s) 2024-06-03T08:36:32.711 INFO:teuthology.orchestra.run.smithi088.stderr:2024-06-03T08:36:32.694+0000 7f16b9c8f640 1 -- 172.21.15.88:0/1383755685 >> v1:172.21.15.192:6789/0 conn(0x7f16b40b4250 legacy=0x7f16b413e9e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-06-03T08:36:32.711 INFO:teuthology.orchestra.run.smithi088.stderr:2024-06-03T08:36:32.694+0000 7f16b9c8f640 1 -- 172.21.15.88:0/1383755685 shutdown_connections 2024-06-03T08:36:32.712 INFO:teuthology.orchestra.run.smithi088.stderr:2024-06-03T08:36:32.694+0000 7f3d5015c640 10 monclient: build_initial_monmap 2024-06-03T08:36:32.712 INFO:teuthology.orchestra.run.smithi088.stderr:2024-06-03T08:36:32.694+0000 7f3d5015c640 10 monclient: monmap: 2024-06-03T08:36:32.712 INFO:teuthology.orchestra.run.smithi088.stderr:epoch 0 2024-06-03T08:36:32.712 INFO:teuthology.orchestra.run.smithi088.stderr:fsid 00000000-0000-0000-0000-000000000000 2024-06-03T08:36:32.712 INFO:teuthology.orchestra.run.smithi088.stderr:last_changed 0.000000 2024-06-03T08:36:32.712 INFO:teuthology.orchestra.run.smithi088.stderr:created 0.000000 2024-06-03T08:36:32.712 INFO:teuthology.orchestra.run.smithi088.stderr:min_mon_release 0 (unknown) 2024-06-03T08:36:32.712 INFO:teuthology.orchestra.run.smithi088.stderr:election_strategy: 1 2024-06-03T08:36:32.712 INFO:teuthology.orchestra.run.smithi088.stderr:0: v1:172.21.15.88:6789/0 mon.noname-a 2024-06-03T08:36:32.712 INFO:teuthology.orchestra.run.smithi088.stderr:1: v1:172.21.15.192:6789/0 mon.noname-b 2024-06-03T08:36:32.712 INFO:teuthology.orchestra.run.smithi088.stderr:2: v1:172.21.15.88:6790/0 mon.noname-c 2024-06-03T08:36:32.713 INFO:teuthology.orchestra.run.smithi088.stderr: 2024-06-03T08:36:32.713 INFO:teuthology.orchestra.run.smithi088.stderr:2024-06-03T08:36:32.694+0000 7f16b9c8f640 1 -- 172.21.15.88:0/1383755685 >> 172.21.15.88:0/1383755685 conn(0x7f16b40717f0 msgr2=0x7f16b4071c00 unknown :-1 s=STATE_NONE l=0).mark_down 2024-06-03T08:36:32.713 INFO:teuthology.orchestra.run.smithi088.stderr:2024-06-03T08:36:32.695+0000 7f16b9c8f640 1 -- 172.21.15.88:0/1383755685 shutdown_connections 2024-06-03T08:36:32.713 INFO:teuthology.orchestra.run.smithi088.stderr:2024-06-03T08:36:32.695+0000 7f16b9c8f640 1 -- 172.21.15.88:0/1383755685 wait complete. 2024-06-03T08:36:32.714 INFO:teuthology.orchestra.run.smithi088.stderr:2024-06-03T08:36:32.695+0000 7f3d5015c640 1 librados: starting msgr at 2024-06-03T08:36:32.714 INFO:teuthology.orchestra.run.smithi088.stderr:2024-06-03T08:36:32.695+0000 7f3d5015c640 1 librados: starting objecter 2024-06-03T08:36:32.714 INFO:teuthology.orchestra.run.smithi088.stderr:2024-06-03T08:36:32.695+0000 7f3d5015c640 1 Processor -- start 2024-06-03T08:36:32.714 INFO:teuthology.orchestra.run.smithi088.stderr:2024-06-03T08:36:32.695+0000 7f3d5015c640 1 -- start start 2024-06-03T08:36:32.714 INFO:teuthology.orchestra.run.smithi088.stderr:2024-06-03T08:36:32.695+0000 7f3d5015c640 1 librados: setting wanted keys 2024-06-03T08:36:32.715 INFO:teuthology.orchestra.run.smithi088.stderr:2024-06-03T08:36:32.695+0000 7f3d5015c640 1 librados: calling monclient init 2024-06-03T08:36:32.715 INFO:teuthology.orchestra.run.smithi088.stderr:2024-06-03T08:36:32.695+0000 7f3d5015c640 10 monclient: init 2024-06-03T08:36:32.715 INFO:teuthology.orchestra.run.smithi088.stderr:2024-06-03T08:36:32.696+0000 7f3d5015c640 10 monclient: _reopen_session rank -1 2024-06-03T08:36:32.715 INFO:teuthology.orchestra.run.smithi088.stderr:2024-06-03T08:36:32.696+0000 7f3d5015c640 10 monclient: _add_conns ranks=[1,2,0] 2024-06-03T08:36:32.715 INFO:teuthology.orchestra.run.smithi088.stderr:2024-06-03T08:36:32.696+0000 7f3d5015c640 10 monclient(hunting): picked mon.noname-b con 0x7f3d480a89f0 addr v1:172.21.15.192:6789/0 2024-06-03T08:36:32.715 INFO:teuthology.orchestra.run.smithi088.stderr:2024-06-03T08:36:32.696+0000 7f3d5015c640 10 monclient(hunting): picked mon.noname-c con 0x7f3d480b4250 addr v1:172.21.15.88:6790/0 2024-06-03T08:36:32.715 INFO:teuthology.orchestra.run.smithi088.stderr:2024-06-03T08:36:32.696+0000 7f3d5015c640 10 monclient(hunting): picked mon.noname-a con 0x7f3d481451c0 addr v1:172.21.15.88:6789/0 2024-06-03T08:36:32.715 INFO:teuthology.orchestra.run.smithi088.stderr:2024-06-03T08:36:32.696+0000 7f3d5015c640 1 -- --> v1:172.21.15.88:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f3d48143c70 con 0x7f3d481451c0 2024-06-03T08:36:32.715 INFO:teuthology.orchestra.run.smithi088.stderr:2024-06-03T08:36:32.696+0000 7f3d5015c640 1 -- --> v1:172.21.15.192:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f3d481da3c0 con 0x7f3d480a89f0 2024-06-03T08:36:32.715 INFO:teuthology.orchestra.run.smithi088.stderr:2024-06-03T08:36:32.696+0000 7f3d5015c640 1 -- --> v1:172.21.15.88:6790/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f3d481da8e0 con 0x7f3d480b4250 2024-06-03T08:36:32.715 INFO:teuthology.orchestra.run.smithi088.stderr:2024-06-03T08:36:32.696+0000 7f3d5015c640 10 monclient(hunting): _renew_subs 2024-06-03T08:36:32.715 INFO:teuthology.orchestra.run.smithi088.stderr:2024-06-03T08:36:32.696+0000 7f3d5015c640 10 monclient(hunting): authenticate will time out at 1248.506083s 2024-06-03T08:36:32.716 INFO:teuthology.orchestra.run.smithi088.stderr:2024-06-03T08:36:32.696+0000 7f16b9c8f640 10 monclient: build_initial_monmap 2024-06-03T08:36:32.716 INFO:teuthology.orchestra.run.smithi088.stderr:2024-06-03T08:36:32.696+0000 7f16b9c8f640 10 monclient: monmap: 2024-06-03T08:36:32.716 INFO:teuthology.orchestra.run.smithi088.stderr:epoch 0 2024-06-03T08:36:32.717 INFO:teuthology.orchestra.run.smithi088.stderr:fsid 00000000-0000-0000-0000-000000000000 2024-06-03T08:36:32.717 INFO:teuthology.orchestra.run.smithi088.stderr:last_changed 0.000000 2024-06-03T08:36:32.717 INFO:teuthology.orchestra.run.smithi088.stderr:created 0.000000 2024-06-03T08:36:32.717 INFO:teuthology.orchestra.run.smithi088.stderr:min_mon_release 0 (unknown) 2024-06-03T08:36:32.717 INFO:teuthology.orchestra.run.smithi088.stderr:election_strategy: 1 2024-06-03T08:36:32.717 INFO:teuthology.orchestra.run.smithi088.stderr:0: v1:172.21.15.88:6789/0 mon.noname-a 2024-06-03T08:36:32.717 INFO:teuthology.orchestra.run.smithi088.stderr:1: v1:172.21.15.192:6789/0 mon.noname-b 2024-06-03T08:36:32.717 INFO:teuthology.orchestra.run.smithi088.stderr:2: v1:172.21.15.88:6790/0 mon.noname-c 2024-06-03T08:36:32.717 INFO:teuthology.orchestra.run.smithi088.stderr: 2024-06-03T08:36:32.717 INFO:teuthology.orchestra.run.smithi088.stderr:2024-06-03T08:36:32.696+0000 7f16b9c8f640 1 librados: starting msgr at 2024-06-03T08:36:32.717 INFO:teuthology.orchestra.run.smithi088.stderr:2024-06-03T08:36:32.696+0000 7f16b9c8f640 1 librados: starting objecter 2024-06-03T08:36:32.717 INFO:teuthology.orchestra.run.smithi088.stderr:2024-06-03T08:36:32.696+0000 7f16b9c8f640 1 Processor -- start 2024-06-03T08:36:32.717 INFO:teuthology.orchestra.run.smithi088.stderr:2024-06-03T08:36:32.697+0000 7f16b9c8f640 1 -- start start 2024-06-03T08:36:32.717 INFO:teuthology.orchestra.run.smithi088.stderr:2024-06-03T08:36:32.697+0000 7f16b9c8f640 1 librados: setting wanted keys 2024-06-03T08:36:32.718 INFO:teuthology.orchestra.run.smithi088.stderr:2024-06-03T08:36:32.697+0000 7f16b9c8f640 1 librados: calling monclient init 2024-06-03T08:36:32.718 INFO:teuthology.orchestra.run.smithi088.stderr:2024-06-03T08:36:32.697+0000 7f16b9c8f640 10 monclient: init 2024-06-03T08:36:32.718 INFO:teuthology.orchestra.run.smithi088.stderr:2024-06-03T08:36:32.697+0000 7f83fcff9640 1 -- 172.21.15.88:0/4283667430 <== mon.1 v1:172.21.15.192:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 392351767 0 0) 0x7f840013f480 con 0x7f84000ae8c0 2024-06-03T08:36:32.718 INFO:teuthology.orchestra.run.smithi088.stderr:2024-06-03T08:36:32.697+0000 7f83fcff9640 10 monclient(hunting): _negotiate 2024-06-03T08:36:32.718 INFO:teuthology.orchestra.run.smithi088.stderr:2024-06-03T08:36:32.697+0000 7f83fcff9640 10 monclient(hunting): _init_auth method 2 2024-06-03T08:36:32.718 INFO:teuthology.orchestra.run.smithi088.stderr:2024-06-03T08:36:32.697+0000 7f83fcff9640 10 monclient(hunting): _init_auth creating new auth 2024-06-03T08:36:32.718 INFO:teuthology.orchestra.run.smithi088.stderr:2024-06-03T08:36:32.697+0000 7f83fcff9640 10 monclient(hunting): my global_id is 4354 2024-06-03T08:36:32.718 INFO:teuthology.orchestra.run.smithi088.stderr:2024-06-03T08:36:32.697+0000 7f83fcff9640 1 -- 172.21.15.88:0/4283667430 --> v1:172.21.15.192:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f83ec004eb0 con 0x7f84000ae8c0 2024-06-03T08:36:32.719 INFO:teuthology.orchestra.run.smithi088.stderr:2024-06-03T08:36:32.698+0000 7f16b9c8f640 10 monclient: _reopen_session rank -1 2024-06-03T08:36:32.719 INFO:teuthology.orchestra.run.smithi088.stderr:2024-06-03T08:36:32.698+0000 7f16b9c8f640 10 monclient: _add_conns ranks=[1,2,0] 2024-06-03T08:36:32.719 INFO:teuthology.orchestra.run.smithi088.stderr:2024-06-03T08:36:32.698+0000 7f16b9c8f640 10 monclient(hunting): picked mon.noname-b con 0x7f16b40a89f0 addr v1:172.21.15.192:6789/0 2024-06-03T08:36:32.720 INFO:teuthology.orchestra.run.smithi088.stderr:2024-06-03T08:36:32.698+0000 7f16b9c8f640 10 monclient(hunting): picked mon.noname-c con 0x7f16b40b4250 addr v1:172.21.15.88:6790/0 2024-06-03T08:36:32.720 INFO:teuthology.orchestra.run.smithi088.stderr:2024-06-03T08:36:32.698+0000 7f16b9c8f640 10 monclient(hunting): picked mon.noname-a con 0x7f16b41451c0 addr v1:172.21.15.88:6789/0 2024-06-03T08:36:32.720 INFO:teuthology.orchestra.run.smithi088.stderr:2024-06-03T08:36:32.698+0000 7f16b9c8f640 1 -- --> v1:172.21.15.88:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f16b4133cb0 con 0x7f16b41451c0 2024-06-03T08:36:32.720 INFO:teuthology.orchestra.run.smithi088.stderr:2024-06-03T08:36:32.698+0000 7f16b9c8f640 1 -- --> v1:172.21.15.192:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f16b41da190 con 0x7f16b40a89f0 2024-06-03T08:36:32.720 INFO:teuthology.orchestra.run.smithi088.stderr:2024-06-03T08:36:32.699+0000 7f16b9c8f640 1 -- --> v1:172.21.15.88:6790/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f16b4137e80 con 0x7f16b40b4250 2024-06-03T08:36:32.720 INFO:teuthology.orchestra.run.smithi088.stderr:2024-06-03T08:36:32.699+0000 7f16b9c8f640 10 monclient(hunting): _renew_subs 2024-06-03T08:36:32.720 INFO:teuthology.orchestra.run.smithi088.stderr:2024-06-03T08:36:32.699+0000 7f16b9c8f640 10 monclient(hunting): authenticate will time out at 1248.509103s 2024-06-03T08:36:32.721 INFO:teuthology.orchestra.run.smithi088.stderr:2024-06-03T08:36:32.705+0000 7feb0f7fe640 1 -- 172.21.15.88:0/978336162 <== mon.1 v1:172.21.15.192:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 1669032316 0 0) 0x7feb100b3930 con 0x7feb100a8960 2024-06-03T08:36:32.721 INFO:teuthology.orchestra.run.smithi088.stderr:2024-06-03T08:36:32.705+0000 7feb0f7fe640 10 monclient(hunting): _negotiate 2024-06-03T08:36:32.721 INFO:teuthology.orchestra.run.smithi088.stderr:2024-06-03T08:36:32.705+0000 7feb0f7fe640 10 monclient(hunting): _init_auth method 2 2024-06-03T08:36:32.721 INFO:teuthology.orchestra.run.smithi088.stderr:2024-06-03T08:36:32.705+0000 7feb0f7fe640 10 monclient(hunting): _init_auth creating new auth 2024-06-03T08:36:32.721 INFO:teuthology.orchestra.run.smithi088.stderr:2024-06-03T08:36:32.705+0000 7feb0f7fe640 10 monclient(hunting): my global_id is 4357 2024-06-03T08:36:32.721 INFO:teuthology.orchestra.run.smithi088.stderr:2024-06-03T08:36:32.705+0000 7feb0f7fe640 1 -- 172.21.15.88:0/978336162 --> v1:172.21.15.192:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7feb00003520 con 0x7feb100a8960 2024-06-03T08:36:32.721 INFO:teuthology.orchestra.run.smithi088.stderr:2024-06-03T08:36:32.706+0000 7f83fcff9640 1 -- 172.21.15.88:0/4283667430 <== mon.1 v1:172.21.15.192:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 764+0+0 (unknown 2177826221 0 0) 0x7f83ec004eb0 con 0x7f84000ae8c0 2024-06-03T08:36:32.721 INFO:teuthology.orchestra.run.smithi088.stderr:2024-06-03T08:36:32.706+0000 7f83fcff9640 1 -- 172.21.15.88:0/4283667430 --> v1:172.21.15.192:6789/0 -- auth(proto 2 165 bytes epoch 0) v1 -- 0x7f840013f480 con 0x7f84000ae8c0 2024-06-03T08:36:32.722 INFO:teuthology.orchestra.run.smithi088.stderr:2024-06-03T08:36:32.706+0000 7f83fcff9640 1 -- 172.21.15.88:0/4283667430 <== mon.1 v1:172.21.15.192:6789/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 2234374065 0 0) 0x7f83f4001970 con 0x7f84000ae8c0 2024-06-03T08:36:32.722 INFO:teuthology.orchestra.run.smithi088.stderr:2024-06-03T08:36:32.706+0000 7f83fcff9640 10 monclient(hunting): handle_monmap mon_map magic: 0 v1 2024-06-03T08:36:32.722 INFO:teuthology.orchestra.run.smithi088.stderr:2024-06-03T08:36:32.706+0000 7f83fcff9640 10 monclient(hunting): got monmap 1 from mon.noname-b (according to old e1) 2024-06-03T08:36:32.722 INFO:teuthology.orchestra.run.smithi088.stderr:2024-06-03T08:36:32.706+0000 7f83fcff9640 10 monclient(hunting): dump: 2024-06-03T08:36:32.722 INFO:teuthology.orchestra.run.smithi088.stderr:epoch 1 2024-06-03T08:36:32.722 INFO:teuthology.orchestra.run.smithi088.stderr:fsid ec3c3fdd-3ac6-4d7d-9412-683130ed8b04 2024-06-03T08:36:32.722 INFO:teuthology.orchestra.run.smithi088.stderr:last_changed 2024-06-03T08:34:31.885125+0000 2024-06-03T08:36:32.722 INFO:teuthology.orchestra.run.smithi088.stderr:created 2024-06-03T08:34:31.885125+0000 2024-06-03T08:36:32.722 INFO:teuthology.orchestra.run.smithi088.stderr:min_mon_release 18 (reef) 2024-06-03T08:36:32.722 INFO:teuthology.orchestra.run.smithi088.stderr:election_strategy: 3 2024-06-03T08:36:32.722 INFO:teuthology.orchestra.run.smithi088.stderr:0: v1:172.21.15.88:6789/0 mon.a 2024-06-03T08:36:32.722 INFO:teuthology.orchestra.run.smithi088.stderr:1: v1:172.21.15.192:6789/0 mon.b 2024-06-03T08:36:32.722 INFO:teuthology.orchestra.run.smithi088.stderr:2: v1:172.21.15.88:6790/0 mon.c 2024-06-03T08:36:32.723 INFO:teuthology.orchestra.run.smithi088.stderr: 2024-06-03T08:36:32.723 INFO:teuthology.orchestra.run.smithi088.stderr:2024-06-03T08:36:32.706+0000 7f83fcff9640 10 monclient(hunting): _finish_auth 0 2024-06-03T08:36:32.725 INFO:teuthology.orchestra.run.smithi088.stderr:2024-06-03T08:36:32.706+0000 7f50db7fe640 1 -- 172.21.15.88:0/680985712 <== mon.1 v1:172.21.15.192:6789/0 4 ==== config(0 keys) v1 ==== 4+0+0 (unknown 0 0 0) 0x7f50d0001910 con 0x7f50dc0b4250 2024-06-03T08:36:32.725 INFO:teuthology.orchestra.run.smithi088.stderr:2024-06-03T08:36:32.706+0000 7f50db7fe640 10 monclient: handle_config config(0 keys) v1 2024-06-03T08:36:32.725 INFO:teuthology.orchestra.run.smithi088.stderr:2024-06-03T08:36:32.706+0000 7f50e2d7d640 10 monclient: get_monmap_and_config success 2024-06-03T08:36:32.725 INFO:teuthology.orchestra.run.smithi088.stderr:2024-06-03T08:36:32.706+0000 7f50e2d7d640 4 set_mon_vals no callback set 2024-06-03T08:36:32.725 INFO:teuthology.orchestra.run.smithi088.stderr:2024-06-03T08:36:32.706+0000 7f50db7fe640 1 -- 172.21.15.88:0/680985712 <== mon.1 v1:172.21.15.192:6789/0 5 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 2234374065 0 0) 0x7f50d0001ca0 con 0x7f50dc0b4250 2024-06-03T08:36:32.725 INFO:teuthology.orchestra.run.smithi088.stderr:2024-06-03T08:36:32.706+0000 7f50db7fe640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-06-03T08:36:32.725 INFO:teuthology.orchestra.run.smithi088.stderr:2024-06-03T08:36:32.706+0000 7f50db7fe640 10 monclient: got monmap 1 from mon.b (according to old e1) 2024-06-03T08:36:32.725 INFO:teuthology.orchestra.run.smithi088.stderr:2024-06-03T08:36:32.706+0000 7f50db7fe640 10 monclient: dump: 2024-06-03T08:36:32.726 INFO:teuthology.orchestra.run.smithi088.stderr:epoch 1 2024-06-03T08:36:32.726 INFO:teuthology.orchestra.run.smithi088.stderr:fsid ec3c3fdd-3ac6-4d7d-9412-683130ed8b04 2024-06-03T08:36:32.726 INFO:teuthology.orchestra.run.smithi088.stderr:last_changed 2024-06-03T08:34:31.885125+0000 2024-06-03T08:36:32.726 INFO:teuthology.orchestra.run.smithi088.stderr:created 2024-06-03T08:34:31.885125+0000 2024-06-03T08:36:32.726 INFO:teuthology.orchestra.run.smithi088.stderr:min_mon_release 18 (reef) 2024-06-03T08:36:32.726 INFO:teuthology.orchestra.run.smithi088.stderr:election_strategy: 3 2024-06-03T08:36:32.726 INFO:teuthology.orchestra.run.smithi088.stderr:0: v1:172.21.15.88:6789/0 mon.a 2024-06-03T08:36:32.726 INFO:teuthology.orchestra.run.smithi088.stderr:1: v1:172.21.15.192:6789/0 mon.b 2024-06-03T08:36:32.726 INFO:teuthology.orchestra.run.smithi088.stderr:2: v1:172.21.15.88:6790/0 mon.c 2024-06-03T08:36:32.726 INFO:teuthology.orchestra.run.smithi088.stderr: 2024-06-03T08:36:32.726 INFO:teuthology.orchestra.run.smithi088.stderr:2024-06-03T08:36:32.707+0000 7f50e2d7d640 10 monclient: shutdown 2024-06-03T08:36:32.726 INFO:teuthology.orchestra.run.smithi088.stderr:2024-06-03T08:36:32.707+0000 7f50e2d7d640 20 monclient: shutdown discarding 0 pending message(s) 2024-06-03T08:36:32.726 INFO:teuthology.orchestra.run.smithi088.stderr:2024-06-03T08:36:32.707+0000 7f50e2d7d640 1 -- 172.21.15.88:0/680985712 >> v1:172.21.15.192:6789/0 conn(0x7f50dc0b4250 legacy=0x7f50dc13e9e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-06-03T08:36:32.727 INFO:teuthology.orchestra.run.smithi088.stderr:2024-06-03T08:36:32.707+0000 7f50e2d7d640 1 -- 172.21.15.88:0/680985712 shutdown_connections 2024-06-03T08:36:32.727 INFO:teuthology.orchestra.run.smithi088.stderr:2024-06-03T08:36:32.707+0000 7f50e2d7d640 1 -- 172.21.15.88:0/680985712 >> 172.21.15.88:0/680985712 conn(0x7f50dc0717f0 msgr2=0x7f50dc071c00 unknown :-1 s=STATE_NONE l=0).mark_down 2024-06-03T08:36:32.727 INFO:teuthology.orchestra.run.smithi088.stderr:2024-06-03T08:36:32.707+0000 7f50e2d7d640 1 -- 172.21.15.88:0/680985712 shutdown_connections 2024-06-03T08:36:32.727 INFO:teuthology.orchestra.run.smithi088.stderr:2024-06-03T08:36:32.707+0000 7f50e2d7d640 1 -- 172.21.15.88:0/680985712 wait complete. 2024-06-03T08:36:32.727 INFO:teuthology.orchestra.run.smithi088.stderr:2024-06-03T08:36:32.707+0000 7f50e2d7d640 10 monclient: build_initial_monmap 2024-06-03T08:36:32.727 INFO:teuthology.orchestra.run.smithi088.stderr:2024-06-03T08:36:32.707+0000 7f50e2d7d640 10 monclient: monmap: 2024-06-03T08:36:32.727 INFO:teuthology.orchestra.run.smithi088.stderr:epoch 0 2024-06-03T08:36:32.727 INFO:teuthology.orchestra.run.smithi088.stderr:fsid 00000000-0000-0000-0000-000000000000 2024-06-03T08:36:32.727 INFO:teuthology.orchestra.run.smithi088.stderr:last_changed 0.000000 2024-06-03T08:36:32.727 INFO:teuthology.orchestra.run.smithi088.stderr:created 0.000000 2024-06-03T08:36:32.727 INFO:teuthology.orchestra.run.smithi088.stderr:min_mon_release 0 (unknown) 2024-06-03T08:36:32.727 INFO:teuthology.orchestra.run.smithi088.stderr:election_strategy: 1 2024-06-03T08:36:32.727 INFO:teuthology.orchestra.run.smithi088.stderr:0: v1:172.21.15.88:6789/0 mon.noname-a 2024-06-03T08:36:32.728 INFO:teuthology.orchestra.run.smithi088.stderr:1: v1:172.21.15.192:6789/0 mon.noname-b 2024-06-03T08:36:32.728 INFO:teuthology.orchestra.run.smithi088.stderr:2: v1:172.21.15.88:6790/0 mon.noname-c 2024-06-03T08:36:32.728 INFO:teuthology.orchestra.run.smithi088.stderr: 2024-06-03T08:36:32.728 INFO:teuthology.orchestra.run.smithi088.stderr:2024-06-03T08:36:32.708+0000 7f50e2d7d640 1 librados: starting msgr at 2024-06-03T08:36:32.728 INFO:teuthology.orchestra.run.smithi088.stderr:2024-06-03T08:36:32.708+0000 7f50e2d7d640 1 librados: starting objecter 2024-06-03T08:36:32.728 INFO:teuthology.orchestra.run.smithi088.stderr:2024-06-03T08:36:32.708+0000 7f50e2d7d640 1 Processor -- start 2024-06-03T08:36:32.728 INFO:teuthology.orchestra.run.smithi088.stderr:2024-06-03T08:36:32.708+0000 7f50e2d7d640 1 -- start start 2024-06-03T08:36:32.728 INFO:teuthology.orchestra.run.smithi088.stderr:2024-06-03T08:36:32.708+0000 7f50e2d7d640 1 librados: setting wanted keys 2024-06-03T08:36:32.728 INFO:teuthology.orchestra.run.smithi088.stderr:2024-06-03T08:36:32.708+0000 7f50e2d7d640 1 librados: calling monclient init 2024-06-03T08:36:32.728 INFO:teuthology.orchestra.run.smithi088.stderr:2024-06-03T08:36:32.708+0000 7f50e2d7d640 10 monclient: init 2024-06-03T08:36:32.728 INFO:teuthology.orchestra.run.smithi088.stderr:2024-06-03T08:36:32.708+0000 7f50e2d7d640 10 monclient: _reopen_session rank -1 2024-06-03T08:36:32.728 INFO:teuthology.orchestra.run.smithi088.stderr:2024-06-03T08:36:32.708+0000 7f50e2d7d640 10 monclient: _add_conns ranks=[1,2,0] 2024-06-03T08:36:32.728 INFO:teuthology.orchestra.run.smithi088.stderr:2024-06-03T08:36:32.709+0000 7f50e2d7d640 10 monclient(hunting): picked mon.noname-b con 0x7f50dc0a89f0 addr v1:172.21.15.192:6789/0 2024-06-03T08:36:32.728 INFO:teuthology.orchestra.run.smithi088.stderr:2024-06-03T08:36:32.709+0000 7f50e2d7d640 10 monclient(hunting): picked mon.noname-c con 0x7f50dc0b4250 addr v1:172.21.15.88:6790/0 2024-06-03T08:36:32.729 INFO:teuthology.orchestra.run.smithi088.stderr:2024-06-03T08:36:32.709+0000 7f50e2d7d640 10 monclient(hunting): picked mon.noname-a con 0x7f50dc1451c0 addr v1:172.21.15.88:6789/0 2024-06-03T08:36:32.729 INFO:teuthology.orchestra.run.smithi088.stderr:2024-06-03T08:36:32.709+0000 7f50e2d7d640 1 -- --> v1:172.21.15.88:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f50dc1dac20 con 0x7f50dc1451c0 2024-06-03T08:36:32.729 INFO:teuthology.orchestra.run.smithi088.stderr:2024-06-03T08:36:32.709+0000 7f50e2d7d640 1 -- --> v1:172.21.15.192:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f50dc13f3b0 con 0x7f50dc0a89f0 2024-06-03T08:36:32.729 INFO:teuthology.orchestra.run.smithi088.stderr:2024-06-03T08:36:32.709+0000 7f50e2d7d640 1 -- --> v1:172.21.15.88:6790/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f50dc13f540 con 0x7f50dc0b4250 2024-06-03T08:36:32.729 INFO:teuthology.orchestra.run.smithi088.stderr:2024-06-03T08:36:32.709+0000 7f50e2d7d640 10 monclient(hunting): _renew_subs 2024-06-03T08:36:32.729 INFO:teuthology.orchestra.run.smithi088.stderr:2024-06-03T08:36:32.709+0000 7f50e2d7d640 10 monclient(hunting): authenticate will time out at 1248.519190s 2024-06-03T08:36:32.730 INFO:teuthology.orchestra.run.smithi088.stderr:2024-06-03T08:36:32.710+0000 7f83fcff9640 1 -- 172.21.15.88:0/4283667430 <== mon.1 v1:172.21.15.192:6789/0 4 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 580+0+0 (unknown 3941536123 0 0) 0x7f840013f480 con 0x7f84000ae8c0 2024-06-03T08:36:32.730 INFO:teuthology.orchestra.run.smithi088.stderr:2024-06-03T08:36:32.710+0000 7f83fcff9640 1 -- 172.21.15.88:0/4283667430 >> v1:172.21.15.88:6790/0 conn(0x7f840013b2f0 legacy=0x7f84001e0e30 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-06-03T08:36:32.730 INFO:teuthology.orchestra.run.smithi088.stderr:2024-06-03T08:36:32.710+0000 7f83fcff9640 1 -- 172.21.15.88:0/4283667430 >> v1:172.21.15.88:6789/0 conn(0x7f84000712a0 legacy=0x7f840014f1f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-06-03T08:36:32.730 INFO:teuthology.orchestra.run.smithi088.stderr:2024-06-03T08:36:32.710+0000 7f83fcff9640 10 monclient: _finish_hunting 0 2024-06-03T08:36:32.730 INFO:teuthology.orchestra.run.smithi088.stderr:2024-06-03T08:36:32.710+0000 7f83fcff9640 1 monclient: found mon.b 2024-06-03T08:36:32.730 INFO:teuthology.orchestra.run.smithi088.stderr:2024-06-03T08:36:32.710+0000 7f83fcff9640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-06-03T08:36:32.730 INFO:teuthology.orchestra.run.smithi088.stderr:2024-06-03T08:36:32.710+0000 7f83fcff9640 10 monclient: _send_mon_message to mon.b at v1:172.21.15.192:6789/0 2024-06-03T08:36:32.730 INFO:teuthology.orchestra.run.smithi088.stderr:2024-06-03T08:36:32.710+0000 7f83fcff9640 1 -- 172.21.15.88:0/4283667430 --> v1:172.21.15.192:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f8400150110 con 0x7f84000ae8c0 2024-06-03T08:36:32.731 INFO:teuthology.orchestra.run.smithi088.stderr:2024-06-03T08:36:32.710+0000 7f83fcff9640 10 monclient: _finish_auth 0 2024-06-03T08:36:32.731 INFO:teuthology.orchestra.run.smithi088.stderr:2024-06-03T08:36:32.710+0000 7f83fcff9640 10 monclient: _check_auth_tickets 2024-06-03T08:36:32.731 INFO:teuthology.orchestra.run.smithi088.stderr:2024-06-03T08:36:32.710+0000 7f83fcff9640 20 monclient: _check_auth_rotating not needed by client.admin 2024-06-03T08:36:32.731 INFO:teuthology.orchestra.run.smithi088.stderr:2024-06-03T08:36:32.710+0000 7f84063aa640 5 monclient: authenticate success, global_id 4354 2024-06-03T08:36:32.731 INFO:teuthology.orchestra.run.smithi088.stderr:2024-06-03T08:36:32.710+0000 7f84063aa640 10 monclient: _renew_subs 2024-06-03T08:36:32.731 INFO:teuthology.orchestra.run.smithi088.stderr:2024-06-03T08:36:32.710+0000 7f84063aa640 10 monclient: _send_mon_message to mon.b at v1:172.21.15.192:6789/0 2024-06-03T08:36:32.731 INFO:teuthology.orchestra.run.smithi088.stderr:2024-06-03T08:36:32.710+0000 7f84063aa640 1 -- 172.21.15.88:0/4283667430 --> v1:172.21.15.192:6789/0 -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f840014dcf0 con 0x7f84000ae8c0 2024-06-03T08:36:32.731 INFO:teuthology.orchestra.run.smithi088.stderr:2024-06-03T08:36:32.710+0000 7f84063aa640 10 client.4354.objecter _maybe_request_map subscribing (onetime) to next osd map 2024-06-03T08:36:32.731 INFO:teuthology.orchestra.run.smithi088.stderr:2024-06-03T08:36:32.710+0000 7f84063aa640 10 monclient: _renew_subs 2024-06-03T08:36:32.731 INFO:teuthology.orchestra.run.smithi088.stderr:2024-06-03T08:36:32.710+0000 7f84063aa640 10 monclient: _send_mon_message to mon.b at v1:172.21.15.192:6789/0 2024-06-03T08:36:32.731 INFO:teuthology.orchestra.run.smithi088.stderr:2024-06-03T08:36:32.710+0000 7f84063aa640 1 -- 172.21.15.88:0/4283667430 --> v1:172.21.15.192:6789/0 -- mon_subscribe({osdmap=0}) v3 -- 0x7f84001da210 con 0x7f84000ae8c0 2024-06-03T08:36:32.731 INFO:teuthology.orchestra.run.smithi088.stderr:2024-06-03T08:36:32.710+0000 7f84063aa640 1 librados: init done 2024-06-03T08:36:32.732 INFO:teuthology.orchestra.run.smithi088.stderr:2024-06-03T08:36:32.710+0000 7f3d4def8640 1 --1- >> v1:172.21.15.192:6789/0 conn(0x7f3d480a89f0 0x7f3d480aab70 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=1).handle_server_banner_and_identify peer v1:172.21.15.192:6789/0 says I am v1:172.21.15.88:52590/0 (socket says 172.21.15.88:52590) 2024-06-03T08:36:32.732 INFO:teuthology.orchestra.run.smithi088.stderr:2024-06-03T08:36:32.710+0000 7f3d4def8640 1 -- 172.21.15.88:0/1768052638 learned_addr learned my addr 172.21.15.88:0/1768052638 (peer_addr_for_me v1:172.21.15.88:0/0) 2024-06-03T08:36:32.732 INFO:teuthology.orchestra.run.smithi088.stderr:2024-06-03T08:36:32.710+0000 7f50e0b19640 1 --1- >> v1:172.21.15.192:6789/0 conn(0x7f50dc0a89f0 0x7f50dc141af0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=1).handle_server_banner_and_identify peer v1:172.21.15.192:6789/0 says I am v1:172.21.15.88:52600/0 (socket says 172.21.15.88:52600) 2024-06-03T08:36:32.732 INFO:teuthology.orchestra.run.smithi088.stderr:2024-06-03T08:36:32.710+0000 7f50e0b19640 1 -- 172.21.15.88:0/1176708298 learned_addr learned my addr 172.21.15.88:0/1176708298 (peer_addr_for_me v1:172.21.15.88:0/0) 2024-06-03T08:36:32.733 INFO:teuthology.orchestra.run.smithi088.stderr:2024-06-03T08:36:32.711+0000 7f83e67fc640 10 start_mon_command cmd=[{"prefix": "get_command_descriptions"}] 2024-06-03T08:36:32.733 INFO:teuthology.orchestra.run.smithi088.stderr:2024-06-03T08:36:32.711+0000 7f83e67fc640 10 monclient: _send_command 1 [{"prefix": "get_command_descriptions"}] 2024-06-03T08:36:32.733 INFO:teuthology.orchestra.run.smithi088.stderr:2024-06-03T08:36:32.711+0000 7f83e67fc640 10 monclient: _send_mon_message to mon.b at v1:172.21.15.192:6789/0 2024-06-03T08:36:32.733 INFO:teuthology.orchestra.run.smithi088.stderr:2024-06-03T08:36:32.711+0000 7f83e67fc640 1 -- 172.21.15.88:0/4283667430 --> v1:172.21.15.192:6789/0 -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f83e0005040 con 0x7f84000ae8c0 2024-06-03T08:36:32.765 INFO:teuthology.orchestra.run.smithi088.stderr:2024-06-03T08:36:32.764+0000 7fccb67fc640 1 -- 172.21.15.88:0/1456172716 <== mon.2 v1:172.21.15.88:6790/0 9 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+182498 (unknown 1092875540 0 3016760700) 0x7fccbc04fa80 con 0x7fccc00a8a60 2024-06-03T08:36:32.765 INFO:teuthology.orchestra.run.smithi088.stderr:2024-06-03T08:36:32.764+0000 7fccb67fc640 10 monclient: handle_mon_command_ack 1 [{"prefix": "get_command_descriptions"}] 2024-06-03T08:36:32.765 INFO:teuthology.orchestra.run.smithi088.stderr:2024-06-03T08:36:32.764+0000 7fccb67fc640 10 monclient: _finish_command 1 = system:0 2024-06-03T08:36:32.768 INFO:teuthology.orchestra.run.smithi088.stderr:2024-06-03T08:36:32.768+0000 7f16b2ffd640 1 --1- >> v1:172.21.15.88:6790/0 conn(0x7f16b40b4250 0x7f16b4132e90 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=1).handle_server_banner_and_identify peer v1:172.21.15.88:6790/0 says I am v1:172.21.15.88:54380/0 (socket says 172.21.15.88:54380) 2024-06-03T08:36:32.768 INFO:teuthology.orchestra.run.smithi088.stderr:2024-06-03T08:36:32.768+0000 7f16b2ffd640 1 -- 172.21.15.88:0/2965169668 learned_addr learned my addr 172.21.15.88:0/2965169668 (peer_addr_for_me v1:172.21.15.88:0/0) 2024-06-03T08:36:32.771 INFO:teuthology.orchestra.run.smithi088.stderr:2024-06-03T08:36:32.771+0000 7f169bfff640 10 client.?.objecter ms_handle_connect 0x7f16b40b4250 2024-06-03T08:36:32.771 INFO:teuthology.orchestra.run.smithi088.stderr:2024-06-03T08:36:32.771+0000 7f169bfff640 10 client.?.objecter resend_mon_ops 2024-06-03T08:36:32.773 INFO:teuthology.orchestra.run.smithi088.stderr:2024-06-03T08:36:32.772+0000 7feb0f7fe640 1 -- 172.21.15.88:0/978336162 <== mon.2 v1:172.21.15.88:6790/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 849550438 0 0) 0x7feb100b1ad0 con 0x7feb1013eec0 2024-06-03T08:36:32.773 INFO:teuthology.orchestra.run.smithi088.stderr:2024-06-03T08:36:32.773+0000 7feb0f7fe640 10 monclient(hunting): _negotiate 2024-06-03T08:36:32.773 INFO:teuthology.orchestra.run.smithi088.stderr:2024-06-03T08:36:32.773+0000 7feb0f7fe640 10 monclient(hunting): _init_auth method 2 2024-06-03T08:36:32.773 INFO:teuthology.orchestra.run.smithi088.stderr:2024-06-03T08:36:32.773+0000 7feb0f7fe640 10 monclient(hunting): _init_auth creating new auth 2024-06-03T08:36:32.773 INFO:teuthology.orchestra.run.smithi088.stderr:2024-06-03T08:36:32.773+0000 7feb0f7fe640 10 monclient(hunting): my global_id is 4355 2024-06-03T08:36:32.773 INFO:teuthology.orchestra.run.smithi088.stderr:2024-06-03T08:36:32.773+0000 7feb0f7fe640 1 -- 172.21.15.88:0/978336162 --> v1:172.21.15.88:6790/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7feb100b3930 con 0x7feb1013eec0 2024-06-03T08:36:32.774 INFO:teuthology.orchestra.run.smithi088.stderr:2024-06-03T08:36:32.774+0000 7f50d97fa640 10 client.?.objecter ms_handle_connect 0x7f50dc0b4250 2024-06-03T08:36:32.774 INFO:teuthology.orchestra.run.smithi088.stderr:2024-06-03T08:36:32.774+0000 7f50d97fa640 10 client.?.objecter resend_mon_ops 2024-06-03T08:36:32.778 INFO:teuthology.orchestra.run.smithi088.stderr:2024-06-03T08:36:32.778+0000 7f3d3e7fc640 10 client.?.objecter ms_handle_connect 0x7f3d480b4250 2024-06-03T08:36:32.779 INFO:teuthology.orchestra.run.smithi088.stderr:2024-06-03T08:36:32.778+0000 7f3d3e7fc640 10 client.?.objecter resend_mon_ops 2024-06-03T08:36:32.781 INFO:teuthology.orchestra.run.smithi088.stderr:2024-06-03T08:36:32.781+0000 7f169bfff640 1 -- 172.21.15.88:0/2965169668 <== mon.2 v1:172.21.15.88:6790/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 2816465471 0 0) 0x7f16b4137e80 con 0x7f16b40b4250 2024-06-03T08:36:32.781 INFO:teuthology.orchestra.run.smithi088.stderr:2024-06-03T08:36:32.781+0000 7f169bfff640 10 monclient(hunting): _negotiate 2024-06-03T08:36:32.781 INFO:teuthology.orchestra.run.smithi088.stderr:2024-06-03T08:36:32.781+0000 7f169bfff640 10 monclient(hunting): _init_auth method 2 2024-06-03T08:36:32.781 INFO:teuthology.orchestra.run.smithi088.stderr:2024-06-03T08:36:32.781+0000 7f169bfff640 10 monclient(hunting): _init_auth creating new auth 2024-06-03T08:36:32.781 INFO:teuthology.orchestra.run.smithi088.stderr:2024-06-03T08:36:32.781+0000 7f169bfff640 10 monclient(hunting): my global_id is 4358 2024-06-03T08:36:32.781 INFO:teuthology.orchestra.run.smithi088.stderr:2024-06-03T08:36:32.781+0000 7f169bfff640 1 -- 172.21.15.88:0/2965169668 --> v1:172.21.15.88:6790/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f16a0004eb0 con 0x7f16b40b4250 2024-06-03T08:36:32.782 INFO:teuthology.orchestra.run.smithi088.stderr:2024-06-03T08:36:32.781+0000 7f50d97fa640 1 -- 172.21.15.88:0/1176708298 <== mon.2 v1:172.21.15.88:6790/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 1708838702 0 0) 0x7f50dc13f540 con 0x7f50dc0b4250 2024-06-03T08:36:32.782 INFO:teuthology.orchestra.run.smithi088.stderr:2024-06-03T08:36:32.781+0000 7f50d97fa640 10 monclient(hunting): _negotiate 2024-06-03T08:36:32.782 INFO:teuthology.orchestra.run.smithi088.stderr:2024-06-03T08:36:32.781+0000 7f50d97fa640 10 monclient(hunting): _init_auth method 2 2024-06-03T08:36:32.782 INFO:teuthology.orchestra.run.smithi088.stderr:2024-06-03T08:36:32.781+0000 7f50d97fa640 10 monclient(hunting): _init_auth creating new auth 2024-06-03T08:36:32.782 INFO:teuthology.orchestra.run.smithi088.stderr:2024-06-03T08:36:32.781+0000 7f50d97fa640 10 monclient(hunting): my global_id is 4361 2024-06-03T08:36:32.782 INFO:teuthology.orchestra.run.smithi088.stderr:2024-06-03T08:36:32.781+0000 7f50d97fa640 1 -- 172.21.15.88:0/1176708298 --> v1:172.21.15.88:6790/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f50c0004eb0 con 0x7f50dc0b4250 2024-06-03T08:36:32.783 INFO:teuthology.orchestra.run.smithi088.stderr:2024-06-03T08:36:32.782+0000 7feb0f7fe640 1 -- 172.21.15.88:0/978336162 <== mon.2 v1:172.21.15.88:6790/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 214+0+0 (unknown 1442715287 0 0) 0x7feb100b3930 con 0x7feb1013eec0 2024-06-03T08:36:32.783 INFO:teuthology.orchestra.run.smithi088.stderr:2024-06-03T08:36:32.782+0000 7feb0f7fe640 1 -- 172.21.15.88:0/978336162 >> v1:172.21.15.192:6789/0 conn(0x7feb100a8960 legacy=0x7feb100a8d40 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-06-03T08:36:32.783 INFO:teuthology.orchestra.run.smithi088.stderr:2024-06-03T08:36:32.782+0000 7feb0f7fe640 1 -- 172.21.15.88:0/978336162 >> v1:172.21.15.88:6789/0 conn(0x7feb101450f0 legacy=0x7feb1013e7b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-06-03T08:36:32.783 INFO:teuthology.orchestra.run.smithi088.stderr:2024-06-03T08:36:32.782+0000 7feb0f7fe640 10 monclient: _finish_hunting 0 2024-06-03T08:36:32.783 INFO:teuthology.orchestra.run.smithi088.stderr:2024-06-03T08:36:32.782+0000 7feb0f7fe640 1 monclient: found mon.noname-b 2024-06-03T08:36:32.784 INFO:teuthology.orchestra.run.smithi088.stderr:2024-06-03T08:36:32.782+0000 7feb0f7fe640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-06-03T08:36:32.784 INFO:teuthology.orchestra.run.smithi088.stderr:2024-06-03T08:36:32.782+0000 7feb0f7fe640 10 monclient: _send_mon_message to mon.noname-b at v1:172.21.15.88:6790/0 2024-06-03T08:36:32.784 INFO:teuthology.orchestra.run.smithi088.stderr:2024-06-03T08:36:32.782+0000 7feb0f7fe640 1 -- 172.21.15.88:0/978336162 --> v1:172.21.15.88:6790/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7feb100b1c60 con 0x7feb1013eec0 2024-06-03T08:36:32.784 INFO:teuthology.orchestra.run.smithi088.stderr:2024-06-03T08:36:32.782+0000 7feb0f7fe640 10 monclient: _finish_auth 0 2024-06-03T08:36:32.784 INFO:teuthology.orchestra.run.smithi088.stderr:2024-06-03T08:36:32.782+0000 7feb0f7fe640 10 monclient: _check_auth_tickets 2024-06-03T08:36:32.784 INFO:teuthology.orchestra.run.smithi088.stderr:2024-06-03T08:36:32.782+0000 7feb0f7fe640 20 monclient: _check_auth_rotating not needed by client.admin 2024-06-03T08:36:32.784 INFO:teuthology.orchestra.run.smithi088.stderr:2024-06-03T08:36:32.782+0000 7feb0f7fe640 1 -- 172.21.15.88:0/978336162 <== mon.2 v1:172.21.15.88:6790/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 2234374065 0 0) 0x7feb08001990 con 0x7feb1013eec0 2024-06-03T08:36:32.784 INFO:teuthology.orchestra.run.smithi088.stderr:2024-06-03T08:36:32.782+0000 7feb0f7fe640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-06-03T08:36:32.784 INFO:teuthology.orchestra.run.smithi088.stderr:2024-06-03T08:36:32.782+0000 7feb0f7fe640 10 monclient: got monmap 1 from mon.noname-b (according to old e1) 2024-06-03T08:36:32.784 INFO:teuthology.orchestra.run.smithi088.stderr:2024-06-03T08:36:32.782+0000 7feb0f7fe640 10 monclient: dump: 2024-06-03T08:36:32.784 INFO:teuthology.orchestra.run.smithi088.stderr:epoch 1 2024-06-03T08:36:32.784 INFO:teuthology.orchestra.run.smithi088.stderr:fsid ec3c3fdd-3ac6-4d7d-9412-683130ed8b04 2024-06-03T08:36:32.784 INFO:teuthology.orchestra.run.smithi088.stderr:last_changed 2024-06-03T08:34:31.885125+0000 2024-06-03T08:36:32.784 INFO:teuthology.orchestra.run.smithi088.stderr:created 2024-06-03T08:34:31.885125+0000 2024-06-03T08:36:32.785 INFO:teuthology.orchestra.run.smithi088.stderr:min_mon_release 18 (reef) 2024-06-03T08:36:32.785 INFO:teuthology.orchestra.run.smithi088.stderr:election_strategy: 3 2024-06-03T08:36:32.785 INFO:teuthology.orchestra.run.smithi088.stderr:0: v1:172.21.15.88:6789/0 mon.a 2024-06-03T08:36:32.785 INFO:teuthology.orchestra.run.smithi088.stderr:1: v1:172.21.15.192:6789/0 mon.b 2024-06-03T08:36:32.785 INFO:teuthology.orchestra.run.smithi088.stderr:2: v1:172.21.15.88:6790/0 mon.c 2024-06-03T08:36:32.785 INFO:teuthology.orchestra.run.smithi088.stderr: 2024-06-03T08:36:32.785 INFO:teuthology.orchestra.run.smithi088.stderr:2024-06-03T08:36:32.782+0000 7feb16ff6640 5 monclient: authenticate success, global_id 4355 2024-06-03T08:36:32.785 INFO:teuthology.orchestra.run.smithi088.stderr:2024-06-03T08:36:32.782+0000 7feb16ff6640 20 monclient: get_monmap_and_config waiting for monmap|config 2024-06-03T08:36:32.788 INFO:teuthology.orchestra.run.smithi088.stderr:2024-06-03T08:36:32.788+0000 7f3d3e7fc640 1 -- 172.21.15.88:0/1768052638 <== mon.2 v1:172.21.15.88:6790/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 1614629358 0 0) 0x7f3d481da8e0 con 0x7f3d480b4250 2024-06-03T08:36:32.788 INFO:teuthology.orchestra.run.smithi088.stderr:2024-06-03T08:36:32.788+0000 7f3d3e7fc640 10 monclient(hunting): _negotiate 2024-06-03T08:36:32.788 INFO:teuthology.orchestra.run.smithi088.stderr:2024-06-03T08:36:32.788+0000 7f3d3e7fc640 10 monclient(hunting): _init_auth method 2 2024-06-03T08:36:32.789 INFO:teuthology.orchestra.run.smithi088.stderr:2024-06-03T08:36:32.788+0000 7f3d3e7fc640 10 monclient(hunting): _init_auth creating new auth 2024-06-03T08:36:32.789 INFO:teuthology.orchestra.run.smithi088.stderr:2024-06-03T08:36:32.788+0000 7f3d3e7fc640 10 monclient(hunting): my global_id is 4364 2024-06-03T08:36:32.789 INFO:teuthology.orchestra.run.smithi088.stderr:2024-06-03T08:36:32.788+0000 7f3d3e7fc640 1 -- 172.21.15.88:0/1768052638 --> v1:172.21.15.88:6790/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f3d2c004eb0 con 0x7f3d480b4250 2024-06-03T08:36:32.791 INFO:teuthology.orchestra.run.smithi088.stderr:2024-06-03T08:36:32.791+0000 7f50d97fa640 1 -- 172.21.15.88:0/1176708298 <== mon.2 v1:172.21.15.88:6790/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 764+0+0 (unknown 501265066 0 0) 0x7f50c0004eb0 con 0x7f50dc0b4250 2024-06-03T08:36:32.791 INFO:teuthology.orchestra.run.smithi088.stderr:2024-06-03T08:36:32.791+0000 7f50d97fa640 1 -- 172.21.15.88:0/1176708298 --> v1:172.21.15.88:6790/0 -- auth(proto 2 165 bytes epoch 0) v1 -- 0x7f50dc13f540 con 0x7f50dc0b4250 2024-06-03T08:36:32.791 INFO:teuthology.orchestra.run.smithi088.stderr:2024-06-03T08:36:32.791+0000 7f50d97fa640 1 -- 172.21.15.88:0/1176708298 <== mon.2 v1:172.21.15.88:6790/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 2234374065 0 0) 0x7f50c8001740 con 0x7f50dc0b4250 2024-06-03T08:36:32.791 INFO:teuthology.orchestra.run.smithi088.stderr:2024-06-03T08:36:32.791+0000 7f50d97fa640 10 monclient(hunting): handle_monmap mon_map magic: 0 v1 2024-06-03T08:36:32.791 INFO:teuthology.orchestra.run.smithi088.stderr:2024-06-03T08:36:32.791+0000 7f50d97fa640 10 monclient(hunting): got monmap 1 from mon.noname-c (according to old e1) 2024-06-03T08:36:32.792 INFO:teuthology.orchestra.run.smithi088.stderr:2024-06-03T08:36:32.791+0000 7f50d97fa640 10 monclient(hunting): dump: 2024-06-03T08:36:32.792 INFO:teuthology.orchestra.run.smithi088.stderr:epoch 1 2024-06-03T08:36:32.792 INFO:teuthology.orchestra.run.smithi088.stderr:fsid ec3c3fdd-3ac6-4d7d-9412-683130ed8b04 2024-06-03T08:36:32.792 INFO:teuthology.orchestra.run.smithi088.stderr:last_changed 2024-06-03T08:34:31.885125+0000 2024-06-03T08:36:32.792 INFO:teuthology.orchestra.run.smithi088.stderr:created 2024-06-03T08:34:31.885125+0000 2024-06-03T08:36:32.792 INFO:teuthology.orchestra.run.smithi088.stderr:min_mon_release 18 (reef) 2024-06-03T08:36:32.792 INFO:teuthology.orchestra.run.smithi088.stderr:election_strategy: 3 2024-06-03T08:36:32.792 INFO:teuthology.orchestra.run.smithi088.stderr:0: v1:172.21.15.88:6789/0 mon.a 2024-06-03T08:36:32.792 INFO:teuthology.orchestra.run.smithi088.stderr:1: v1:172.21.15.192:6789/0 mon.b 2024-06-03T08:36:32.792 INFO:teuthology.orchestra.run.smithi088.stderr:2: v1:172.21.15.88:6790/0 mon.c 2024-06-03T08:36:32.792 INFO:teuthology.orchestra.run.smithi088.stderr: 2024-06-03T08:36:32.792 INFO:teuthology.orchestra.run.smithi088.stderr:2024-06-03T08:36:32.791+0000 7f50d97fa640 10 monclient(hunting): _finish_auth 0 2024-06-03T08:36:32.803 INFO:teuthology.orchestra.run.smithi088.stderr:2024-06-03T08:36:32.803+0000 7f169bfff640 1 -- 172.21.15.88:0/2965169668 <== mon.2 v1:172.21.15.88:6790/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 764+0+0 (unknown 2766778650 0 0) 0x7f16a0004eb0 con 0x7f16b40b4250 2024-06-03T08:36:32.803 INFO:teuthology.orchestra.run.smithi088.stderr:2024-06-03T08:36:32.803+0000 7f169bfff640 1 -- 172.21.15.88:0/2965169668 --> v1:172.21.15.88:6790/0 -- auth(proto 2 165 bytes epoch 0) v1 -- 0x7f16b4137e80 con 0x7f16b40b4250 2024-06-03T08:36:32.804 INFO:teuthology.orchestra.run.smithi088.stderr:2024-06-03T08:36:32.803+0000 7f169bfff640 1 -- 172.21.15.88:0/2965169668 <== mon.2 v1:172.21.15.88:6790/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 2234374065 0 0) 0x7f16a8001750 con 0x7f16b40b4250 2024-06-03T08:36:32.804 INFO:teuthology.orchestra.run.smithi088.stderr:2024-06-03T08:36:32.803+0000 7f169bfff640 10 monclient(hunting): handle_monmap mon_map magic: 0 v1 2024-06-03T08:36:32.804 INFO:teuthology.orchestra.run.smithi088.stderr:2024-06-03T08:36:32.803+0000 7f169bfff640 10 monclient(hunting): got monmap 1 from mon.noname-c (according to old e1) 2024-06-03T08:36:32.804 INFO:teuthology.orchestra.run.smithi088.stderr:2024-06-03T08:36:32.803+0000 7f169bfff640 10 monclient(hunting): dump: 2024-06-03T08:36:32.804 INFO:teuthology.orchestra.run.smithi088.stderr:epoch 1 2024-06-03T08:36:32.804 INFO:teuthology.orchestra.run.smithi088.stderr:fsid ec3c3fdd-3ac6-4d7d-9412-683130ed8b04 2024-06-03T08:36:32.804 INFO:teuthology.orchestra.run.smithi088.stderr:last_changed 2024-06-03T08:34:31.885125+0000 2024-06-03T08:36:32.804 INFO:teuthology.orchestra.run.smithi088.stderr:created 2024-06-03T08:34:31.885125+0000 2024-06-03T08:36:32.804 INFO:teuthology.orchestra.run.smithi088.stderr:min_mon_release 18 (reef) 2024-06-03T08:36:32.804 INFO:teuthology.orchestra.run.smithi088.stderr:election_strategy: 3 2024-06-03T08:36:32.804 INFO:teuthology.orchestra.run.smithi088.stderr:0: v1:172.21.15.88:6789/0 mon.a 2024-06-03T08:36:32.804 INFO:teuthology.orchestra.run.smithi088.stderr:1: v1:172.21.15.192:6789/0 mon.b 2024-06-03T08:36:32.805 INFO:teuthology.orchestra.run.smithi088.stderr:2: v1:172.21.15.88:6790/0 mon.c 2024-06-03T08:36:32.805 INFO:teuthology.orchestra.run.smithi088.stderr: 2024-06-03T08:36:32.805 INFO:teuthology.orchestra.run.smithi088.stderr:2024-06-03T08:36:32.803+0000 7f169bfff640 10 monclient(hunting): _finish_auth 0 2024-06-03T08:36:32.805 INFO:teuthology.orchestra.run.smithi088.stderr:2024-06-03T08:36:32.804+0000 7f3d3e7fc640 1 -- 172.21.15.88:0/1768052638 <== mon.2 v1:172.21.15.88:6790/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 764+0+0 (unknown 1903354561 0 0) 0x7f3d2c004eb0 con 0x7f3d480b4250 2024-06-03T08:36:32.806 INFO:teuthology.orchestra.run.smithi088.stderr:2024-06-03T08:36:32.804+0000 7f3d3e7fc640 1 -- 172.21.15.88:0/1768052638 --> v1:172.21.15.88:6790/0 -- auth(proto 2 165 bytes epoch 0) v1 -- 0x7f3d481da8e0 con 0x7f3d480b4250 2024-06-03T08:36:32.806 INFO:teuthology.orchestra.run.smithi088.stderr:2024-06-03T08:36:32.804+0000 7f3d3e7fc640 1 -- 172.21.15.88:0/1768052638 <== mon.2 v1:172.21.15.88:6790/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 2234374065 0 0) 0x7f3d38001740 con 0x7f3d480b4250 2024-06-03T08:36:32.806 INFO:teuthology.orchestra.run.smithi088.stderr:2024-06-03T08:36:32.804+0000 7f3d3e7fc640 10 monclient(hunting): handle_monmap mon_map magic: 0 v1 2024-06-03T08:36:32.806 INFO:teuthology.orchestra.run.smithi088.stderr:2024-06-03T08:36:32.804+0000 7f3d3e7fc640 10 monclient(hunting): got monmap 1 from mon.noname-c (according to old e1) 2024-06-03T08:36:32.806 INFO:teuthology.orchestra.run.smithi088.stderr:2024-06-03T08:36:32.804+0000 7f3d3e7fc640 10 monclient(hunting): dump: 2024-06-03T08:36:32.806 INFO:teuthology.orchestra.run.smithi088.stderr:epoch 1 2024-06-03T08:36:32.806 INFO:teuthology.orchestra.run.smithi088.stderr:fsid ec3c3fdd-3ac6-4d7d-9412-683130ed8b04 2024-06-03T08:36:32.806 INFO:teuthology.orchestra.run.smithi088.stderr:last_changed 2024-06-03T08:34:31.885125+0000 2024-06-03T08:36:32.806 INFO:teuthology.orchestra.run.smithi088.stderr:created 2024-06-03T08:34:31.885125+0000 2024-06-03T08:36:32.806 INFO:teuthology.orchestra.run.smithi088.stderr:min_mon_release 18 (reef) 2024-06-03T08:36:32.806 INFO:teuthology.orchestra.run.smithi088.stderr:election_strategy: 3 2024-06-03T08:36:32.806 INFO:teuthology.orchestra.run.smithi088.stderr:0: v1:172.21.15.88:6789/0 mon.a 2024-06-03T08:36:32.807 INFO:teuthology.orchestra.run.smithi088.stderr:1: v1:172.21.15.192:6789/0 mon.b 2024-06-03T08:36:32.807 INFO:teuthology.orchestra.run.smithi088.stderr:2: v1:172.21.15.88:6790/0 mon.c 2024-06-03T08:36:32.807 INFO:teuthology.orchestra.run.smithi088.stderr: 2024-06-03T08:36:32.807 INFO:teuthology.orchestra.run.smithi088.stderr:2024-06-03T08:36:32.804+0000 7f3d3e7fc640 10 monclient(hunting): _finish_auth 0 2024-06-03T08:36:32.808 INFO:teuthology.orchestra.run.smithi088.stderr:2024-06-03T08:36:32.807+0000 7f50d97fa640 1 -- 172.21.15.88:0/1176708298 <== mon.2 v1:172.21.15.88:6790/0 4 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 580+0+0 (unknown 3511127750 0 0) 0x7f50dc13f540 con 0x7f50dc0b4250 2024-06-03T08:36:32.808 INFO:teuthology.orchestra.run.smithi088.stderr:2024-06-03T08:36:32.807+0000 7f50d97fa640 1 -- 172.21.15.88:0/1176708298 >> v1:172.21.15.192:6789/0 conn(0x7f50dc0a89f0 legacy=0x7f50dc141af0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-06-03T08:36:32.808 INFO:teuthology.orchestra.run.smithi088.stderr:2024-06-03T08:36:32.807+0000 7f50d97fa640 1 -- 172.21.15.88:0/1176708298 >> v1:172.21.15.88:6789/0 conn(0x7f50dc1451c0 legacy=0x7f50dc1da510 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-06-03T08:36:32.808 INFO:teuthology.orchestra.run.smithi088.stderr:2024-06-03T08:36:32.808+0000 7f50d97fa640 10 monclient: _finish_hunting 0 2024-06-03T08:36:32.809 INFO:teuthology.orchestra.run.smithi088.stderr:2024-06-03T08:36:32.808+0000 7f50d97fa640 1 monclient: found mon.c 2024-06-03T08:36:32.809 INFO:teuthology.orchestra.run.smithi088.stderr:2024-06-03T08:36:32.808+0000 7f50d97fa640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-06-03T08:36:32.809 INFO:teuthology.orchestra.run.smithi088.stderr:2024-06-03T08:36:32.808+0000 7f50d97fa640 10 monclient: _send_mon_message to mon.c at v1:172.21.15.88:6790/0 2024-06-03T08:36:32.809 INFO:teuthology.orchestra.run.smithi088.stderr:2024-06-03T08:36:32.808+0000 7f50d97fa640 1 -- 172.21.15.88:0/1176708298 --> v1:172.21.15.88:6790/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f50dc1447b0 con 0x7f50dc0b4250 2024-06-03T08:36:32.809 INFO:teuthology.orchestra.run.smithi088.stderr:2024-06-03T08:36:32.808+0000 7f50d97fa640 10 monclient: _finish_auth 0 2024-06-03T08:36:32.809 INFO:teuthology.orchestra.run.smithi088.stderr:2024-06-03T08:36:32.808+0000 7f50d97fa640 10 monclient: _check_auth_tickets 2024-06-03T08:36:32.809 INFO:teuthology.orchestra.run.smithi088.stderr:2024-06-03T08:36:32.808+0000 7f50d97fa640 20 monclient: _check_auth_rotating not needed by client.admin 2024-06-03T08:36:32.809 INFO:teuthology.orchestra.run.smithi088.stderr:2024-06-03T08:36:32.808+0000 7f50e2d7d640 5 monclient: authenticate success, global_id 4361 2024-06-03T08:36:32.809 INFO:teuthology.orchestra.run.smithi088.stderr:2024-06-03T08:36:32.808+0000 7f50e2d7d640 10 monclient: _renew_subs 2024-06-03T08:36:32.809 INFO:teuthology.orchestra.run.smithi088.stderr:2024-06-03T08:36:32.808+0000 7f50e2d7d640 10 monclient: _send_mon_message to mon.c at v1:172.21.15.88:6790/0 2024-06-03T08:36:32.809 INFO:teuthology.orchestra.run.smithi088.stderr:2024-06-03T08:36:32.808+0000 7f50e2d7d640 1 -- 172.21.15.88:0/1176708298 --> v1:172.21.15.88:6790/0 -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f50dc144a00 con 0x7f50dc0b4250 2024-06-03T08:36:32.809 INFO:teuthology.orchestra.run.smithi088.stderr:2024-06-03T08:36:32.808+0000 7f50e2d7d640 10 client.4361.objecter _maybe_request_map subscribing (onetime) to next osd map 2024-06-03T08:36:32.809 INFO:teuthology.orchestra.run.smithi088.stderr:2024-06-03T08:36:32.808+0000 7f50e2d7d640 10 monclient: _renew_subs 2024-06-03T08:36:32.809 INFO:teuthology.orchestra.run.smithi088.stderr:2024-06-03T08:36:32.808+0000 7f50e2d7d640 10 monclient: _send_mon_message to mon.c at v1:172.21.15.88:6790/0 2024-06-03T08:36:32.809 INFO:teuthology.orchestra.run.smithi088.stderr:2024-06-03T08:36:32.808+0000 7f50e2d7d640 1 -- 172.21.15.88:0/1176708298 --> v1:172.21.15.88:6790/0 -- mon_subscribe({osdmap=0}) v3 -- 0x7f50dc140500 con 0x7f50dc0b4250 2024-06-03T08:36:32.810 INFO:teuthology.orchestra.run.smithi088.stderr:2024-06-03T08:36:32.808+0000 7f50e2d7d640 1 librados: init done 2024-06-03T08:36:32.810 INFO:teuthology.orchestra.run.smithi088.stderr:2024-06-03T08:36:32.808+0000 7f50e2d7d640 10 start_mon_command cmd=[{"prefix": "get_command_descriptions"}] 2024-06-03T08:36:32.810 INFO:teuthology.orchestra.run.smithi088.stderr:2024-06-03T08:36:32.809+0000 7f50e2d7d640 10 monclient: _send_command 1 [{"prefix": "get_command_descriptions"}] 2024-06-03T08:36:32.810 INFO:teuthology.orchestra.run.smithi088.stderr:2024-06-03T08:36:32.809+0000 7f50e2d7d640 10 monclient: _send_mon_message to mon.c at v1:172.21.15.88:6790/0 2024-06-03T08:36:32.810 INFO:teuthology.orchestra.run.smithi088.stderr:2024-06-03T08:36:32.809+0000 7f50e2d7d640 1 -- 172.21.15.88:0/1176708298 --> v1:172.21.15.88:6790/0 -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f50dc13f910 con 0x7f50dc0b4250 2024-06-03T08:36:32.815 INFO:teuthology.orchestra.run.smithi088.stderr:2024-06-03T08:36:32.815+0000 7feb0f7fe640 1 -- 172.21.15.88:0/978336162 <== mon.2 v1:172.21.15.88:6790/0 4 ==== config(0 keys) v1 ==== 4+0+0 (unknown 0 0 0) 0x7feb08001bb0 con 0x7feb1013eec0 2024-06-03T08:36:32.815 INFO:teuthology.orchestra.run.smithi088.stderr:2024-06-03T08:36:32.815+0000 7feb0f7fe640 10 monclient: handle_config config(0 keys) v1 2024-06-03T08:36:32.815 INFO:teuthology.orchestra.run.smithi088.stderr:2024-06-03T08:36:32.815+0000 7feb0f7fe640 1 -- 172.21.15.88:0/978336162 <== mon.2 v1:172.21.15.88:6790/0 5 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 2234374065 0 0) 0x7feb08001f00 con 0x7feb1013eec0 2024-06-03T08:36:32.815 INFO:teuthology.orchestra.run.smithi088.stderr:2024-06-03T08:36:32.815+0000 7feb16ff6640 10 monclient: get_monmap_and_config success 2024-06-03T08:36:32.816 INFO:teuthology.orchestra.run.smithi088.stderr:2024-06-03T08:36:32.815+0000 7feb16ff6640 4 set_mon_vals no callback set 2024-06-03T08:36:32.816 INFO:teuthology.orchestra.run.smithi088.stderr:2024-06-03T08:36:32.815+0000 7feb0f7fe640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-06-03T08:36:32.816 INFO:teuthology.orchestra.run.smithi088.stderr:2024-06-03T08:36:32.815+0000 7feb0f7fe640 10 monclient: got monmap 1 from mon.c (according to old e1) 2024-06-03T08:36:32.816 INFO:teuthology.orchestra.run.smithi088.stderr:2024-06-03T08:36:32.815+0000 7feb0f7fe640 10 monclient: dump: 2024-06-03T08:36:32.816 INFO:teuthology.orchestra.run.smithi088.stderr:epoch 1 2024-06-03T08:36:32.816 INFO:teuthology.orchestra.run.smithi088.stderr:fsid ec3c3fdd-3ac6-4d7d-9412-683130ed8b04 2024-06-03T08:36:32.816 INFO:teuthology.orchestra.run.smithi088.stderr:last_changed 2024-06-03T08:34:31.885125+0000 2024-06-03T08:36:32.816 INFO:teuthology.orchestra.run.smithi088.stderr:created 2024-06-03T08:34:31.885125+0000 2024-06-03T08:36:32.816 INFO:teuthology.orchestra.run.smithi088.stderr:min_mon_release 18 (reef) 2024-06-03T08:36:32.816 INFO:teuthology.orchestra.run.smithi088.stderr:election_strategy: 3 2024-06-03T08:36:32.816 INFO:teuthology.orchestra.run.smithi088.stderr:0: v1:172.21.15.88:6789/0 mon.a 2024-06-03T08:36:32.816 INFO:teuthology.orchestra.run.smithi088.stderr:1: v1:172.21.15.192:6789/0 mon.b 2024-06-03T08:36:32.816 INFO:teuthology.orchestra.run.smithi088.stderr:2: v1:172.21.15.88:6790/0 mon.c 2024-06-03T08:36:32.816 INFO:teuthology.orchestra.run.smithi088.stderr: 2024-06-03T08:36:32.818 INFO:teuthology.orchestra.run.smithi088.stderr:2024-06-03T08:36:32.816+0000 7feb16ff6640 10 monclient: shutdown 2024-06-03T08:36:32.818 INFO:teuthology.orchestra.run.smithi088.stderr:2024-06-03T08:36:32.816+0000 7feb16ff6640 20 monclient: shutdown discarding 0 pending message(s) 2024-06-03T08:36:32.818 INFO:teuthology.orchestra.run.smithi088.stderr:2024-06-03T08:36:32.816+0000 7feb16ff6640 1 -- 172.21.15.88:0/978336162 >> v1:172.21.15.88:6790/0 conn(0x7feb1013eec0 legacy=0x7feb1014ce60 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-06-03T08:36:32.818 INFO:teuthology.orchestra.run.smithi088.stderr:2024-06-03T08:36:32.816+0000 7feb16ff6640 1 -- 172.21.15.88:0/978336162 shutdown_connections 2024-06-03T08:36:32.818 INFO:teuthology.orchestra.run.smithi088.stderr:2024-06-03T08:36:32.816+0000 7feb16ff6640 1 -- 172.21.15.88:0/978336162 >> 172.21.15.88:0/978336162 conn(0x7feb100b70d0 msgr2=0x7feb100b74e0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-06-03T08:36:32.818 INFO:teuthology.orchestra.run.smithi088.stderr:2024-06-03T08:36:32.816+0000 7feb16ff6640 1 -- 172.21.15.88:0/978336162 shutdown_connections 2024-06-03T08:36:32.818 INFO:teuthology.orchestra.run.smithi088.stderr:2024-06-03T08:36:32.816+0000 7feb16ff6640 1 -- 172.21.15.88:0/978336162 wait complete. 2024-06-03T08:36:32.818 INFO:teuthology.orchestra.run.smithi088.stderr:2024-06-03T08:36:32.817+0000 7feb16ff6640 10 monclient: build_initial_monmap 2024-06-03T08:36:32.818 INFO:teuthology.orchestra.run.smithi088.stderr:2024-06-03T08:36:32.817+0000 7feb16ff6640 10 monclient: monmap: 2024-06-03T08:36:32.818 INFO:teuthology.orchestra.run.smithi088.stderr:epoch 0 2024-06-03T08:36:32.818 INFO:teuthology.orchestra.run.smithi088.stderr:fsid 00000000-0000-0000-0000-000000000000 2024-06-03T08:36:32.818 INFO:teuthology.orchestra.run.smithi088.stderr:last_changed 0.000000 2024-06-03T08:36:32.819 INFO:teuthology.orchestra.run.smithi088.stderr:created 0.000000 2024-06-03T08:36:32.819 INFO:teuthology.orchestra.run.smithi088.stderr:min_mon_release 0 (unknown) 2024-06-03T08:36:32.819 INFO:teuthology.orchestra.run.smithi088.stderr:election_strategy: 1 2024-06-03T08:36:32.819 INFO:teuthology.orchestra.run.smithi088.stderr:0: v1:172.21.15.88:6789/0 mon.noname-a 2024-06-03T08:36:32.819 INFO:teuthology.orchestra.run.smithi088.stderr:1: v1:172.21.15.192:6789/0 mon.noname-b 2024-06-03T08:36:32.819 INFO:teuthology.orchestra.run.smithi088.stderr:2: v1:172.21.15.88:6790/0 mon.noname-c 2024-06-03T08:36:32.819 INFO:teuthology.orchestra.run.smithi088.stderr: 2024-06-03T08:36:32.819 INFO:teuthology.orchestra.run.smithi088.stderr:2024-06-03T08:36:32.817+0000 7feb16ff6640 1 librados: starting msgr at 2024-06-03T08:36:32.819 INFO:teuthology.orchestra.run.smithi088.stderr:2024-06-03T08:36:32.818+0000 7feb16ff6640 1 librados: starting objecter 2024-06-03T08:36:32.821 INFO:teuthology.orchestra.run.smithi088.stderr:2024-06-03T08:36:32.818+0000 7feb16ff6640 1 Processor -- start 2024-06-03T08:36:32.821 INFO:teuthology.orchestra.run.smithi088.stderr:2024-06-03T08:36:32.818+0000 7feb16ff6640 1 -- start start 2024-06-03T08:36:32.821 INFO:teuthology.orchestra.run.smithi088.stderr:2024-06-03T08:36:32.818+0000 7feb16ff6640 1 librados: setting wanted keys 2024-06-03T08:36:32.821 INFO:teuthology.orchestra.run.smithi088.stderr:2024-06-03T08:36:32.818+0000 7feb16ff6640 1 librados: calling monclient init 2024-06-03T08:36:32.821 INFO:teuthology.orchestra.run.smithi088.stderr:2024-06-03T08:36:32.818+0000 7feb16ff6640 10 monclient: init 2024-06-03T08:36:32.821 INFO:teuthology.orchestra.run.smithi088.stderr:2024-06-03T08:36:32.819+0000 7feb16ff6640 10 monclient: _reopen_session rank -1 2024-06-03T08:36:32.821 INFO:teuthology.orchestra.run.smithi088.stderr:2024-06-03T08:36:32.819+0000 7feb16ff6640 10 monclient: _add_conns ranks=[1,0,2] 2024-06-03T08:36:32.821 INFO:teuthology.orchestra.run.smithi088.stderr:2024-06-03T08:36:32.819+0000 7feb16ff6640 10 monclient(hunting): picked mon.noname-b con 0x7feb100a8960 addr v1:172.21.15.192:6789/0 2024-06-03T08:36:32.821 INFO:teuthology.orchestra.run.smithi088.stderr:2024-06-03T08:36:32.819+0000 7feb16ff6640 10 monclient(hunting): picked mon.noname-a con 0x7feb1013eec0 addr v1:172.21.15.88:6789/0 2024-06-03T08:36:32.821 INFO:teuthology.orchestra.run.smithi088.stderr:2024-06-03T08:36:32.819+0000 7feb16ff6640 10 monclient(hunting): picked mon.noname-c con 0x7feb101450f0 addr v1:172.21.15.88:6790/0 2024-06-03T08:36:32.821 INFO:teuthology.orchestra.run.smithi088.stderr:2024-06-03T08:36:32.819+0000 7feb16ff6640 1 -- --> v1:172.21.15.88:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7feb10139d00 con 0x7feb1013eec0 2024-06-03T08:36:32.821 INFO:teuthology.orchestra.run.smithi088.stderr:2024-06-03T08:36:32.819+0000 7feb16ff6640 1 -- --> v1:172.21.15.192:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7feb100af340 con 0x7feb100a8960 2024-06-03T08:36:32.821 INFO:teuthology.orchestra.run.smithi088.stderr:2024-06-03T08:36:32.819+0000 7feb16ff6640 1 -- --> v1:172.21.15.88:6790/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7feb100ab220 con 0x7feb101450f0 2024-06-03T08:36:32.822 INFO:teuthology.orchestra.run.smithi088.stderr:2024-06-03T08:36:32.819+0000 7feb16ff6640 10 monclient(hunting): _renew_subs 2024-06-03T08:36:32.822 INFO:teuthology.orchestra.run.smithi088.stderr:2024-06-03T08:36:32.819+0000 7feb16ff6640 10 monclient(hunting): authenticate will time out at 1248.629800s 2024-06-03T08:36:32.839 INFO:teuthology.orchestra.run.smithi088.stderr:2024-06-03T08:36:32.839+0000 7fb10c947640 10 start_mon_command cmd=[{"prefix": "osd last-stat-seq", "id": 2}] 2024-06-03T08:36:32.840 INFO:teuthology.orchestra.run.smithi088.stderr:2024-06-03T08:36:32.839+0000 7fb10c947640 10 monclient: _send_command 2 [{"prefix": "osd last-stat-seq", "id": 2}] 2024-06-03T08:36:32.840 INFO:teuthology.orchestra.run.smithi088.stderr:2024-06-03T08:36:32.839+0000 7fb10c947640 10 monclient: _send_mon_message to mon.b at v1:172.21.15.192:6789/0 2024-06-03T08:36:32.840 INFO:teuthology.orchestra.run.smithi088.stderr:2024-06-03T08:36:32.839+0000 7fb10c947640 1 -- 172.21.15.88:0/4168539993 --> v1:172.21.15.192:6789/0 -- mon_command({"prefix": "osd last-stat-seq", "id": 2} v 0) v1 -- 0x7fb108143600 con 0x7fb10813eec0 2024-06-03T08:36:32.896 INFO:teuthology.orchestra.run.smithi088.stderr:2024-06-03T08:36:32.895+0000 7f3d3e7fc640 1 -- 172.21.15.88:0/1768052638 <== mon.2 v1:172.21.15.88:6790/0 4 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 580+0+0 (unknown 3797539162 0 0) 0x7f3d481da8e0 con 0x7f3d480b4250 2024-06-03T08:36:32.896 INFO:teuthology.orchestra.run.smithi088.stderr:2024-06-03T08:36:32.895+0000 7f3d3e7fc640 1 -- 172.21.15.88:0/1768052638 >> v1:172.21.15.192:6789/0 conn(0x7f3d480a89f0 legacy=0x7f3d480aab70 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-06-03T08:36:32.896 INFO:teuthology.orchestra.run.smithi088.stderr:2024-06-03T08:36:32.895+0000 7f3d3e7fc640 1 -- 172.21.15.88:0/1768052638 >> v1:172.21.15.88:6789/0 conn(0x7f3d481451c0 legacy=0x7f3d481e1e00 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-06-03T08:36:32.897 INFO:teuthology.orchestra.run.smithi088.stderr:2024-06-03T08:36:32.895+0000 7f3d3e7fc640 10 monclient: _finish_hunting 0 2024-06-03T08:36:32.897 INFO:teuthology.orchestra.run.smithi088.stderr:2024-06-03T08:36:32.895+0000 7f3d3e7fc640 1 monclient: found mon.c 2024-06-03T08:36:32.897 INFO:teuthology.orchestra.run.smithi088.stderr:2024-06-03T08:36:32.895+0000 7f3d3e7fc640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-06-03T08:36:32.897 INFO:teuthology.orchestra.run.smithi088.stderr:2024-06-03T08:36:32.895+0000 7f3d3e7fc640 10 monclient: _send_mon_message to mon.c at v1:172.21.15.88:6790/0 2024-06-03T08:36:32.897 INFO:teuthology.orchestra.run.smithi088.stderr:2024-06-03T08:36:32.895+0000 7f3d3e7fc640 1 -- 172.21.15.88:0/1768052638 --> v1:172.21.15.88:6790/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f3d481447b0 con 0x7f3d480b4250 2024-06-03T08:36:32.897 INFO:teuthology.orchestra.run.smithi088.stderr:2024-06-03T08:36:32.895+0000 7f3d3e7fc640 10 monclient: _finish_auth 0 2024-06-03T08:36:32.897 INFO:teuthology.orchestra.run.smithi088.stderr:2024-06-03T08:36:32.895+0000 7f3d3e7fc640 10 monclient: _check_auth_tickets 2024-06-03T08:36:32.897 INFO:teuthology.orchestra.run.smithi088.stderr:2024-06-03T08:36:32.895+0000 7f3d3e7fc640 20 monclient: _check_auth_rotating not needed by client.admin 2024-06-03T08:36:32.897 INFO:teuthology.orchestra.run.smithi088.stderr:2024-06-03T08:36:32.895+0000 7f3d5015c640 5 monclient: authenticate success, global_id 4364 2024-06-03T08:36:32.897 INFO:teuthology.orchestra.run.smithi088.stderr:2024-06-03T08:36:32.895+0000 7f3d5015c640 10 monclient: _renew_subs 2024-06-03T08:36:32.897 INFO:teuthology.orchestra.run.smithi088.stderr:2024-06-03T08:36:32.895+0000 7f3d5015c640 10 monclient: _send_mon_message to mon.c at v1:172.21.15.88:6790/0 2024-06-03T08:36:32.897 INFO:teuthology.orchestra.run.smithi088.stderr:2024-06-03T08:36:32.895+0000 7f3d5015c640 1 -- 172.21.15.88:0/1768052638 --> v1:172.21.15.88:6790/0 -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f3d480ab280 con 0x7f3d480b4250 2024-06-03T08:36:32.897 INFO:teuthology.orchestra.run.smithi088.stderr:2024-06-03T08:36:32.895+0000 7f3d5015c640 10 client.4364.objecter _maybe_request_map subscribing (onetime) to next osd map 2024-06-03T08:36:32.897 INFO:teuthology.orchestra.run.smithi088.stderr:2024-06-03T08:36:32.895+0000 7f3d5015c640 10 monclient: _renew_subs 2024-06-03T08:36:32.898 INFO:teuthology.orchestra.run.smithi088.stderr:2024-06-03T08:36:32.895+0000 7f3d5015c640 10 monclient: _send_mon_message to mon.c at v1:172.21.15.88:6790/0 2024-06-03T08:36:32.898 INFO:teuthology.orchestra.run.smithi088.stderr:2024-06-03T08:36:32.895+0000 7f3d5015c640 1 -- 172.21.15.88:0/1768052638 --> v1:172.21.15.88:6790/0 -- mon_subscribe({osdmap=0}) v3 -- 0x7f3d480b03a0 con 0x7f3d480b4250 2024-06-03T08:36:32.898 INFO:teuthology.orchestra.run.smithi088.stderr:2024-06-03T08:36:32.895+0000 7f3d5015c640 1 librados: init done 2024-06-03T08:36:32.898 INFO:teuthology.orchestra.run.smithi088.stderr:2024-06-03T08:36:32.897+0000 7f3d5015c640 10 start_mon_command cmd=[{"prefix": "get_command_descriptions"}] 2024-06-03T08:36:32.898 INFO:teuthology.orchestra.run.smithi088.stderr:2024-06-03T08:36:32.897+0000 7f3d5015c640 10 monclient: _send_command 1 [{"prefix": "get_command_descriptions"}] 2024-06-03T08:36:32.898 INFO:teuthology.orchestra.run.smithi088.stderr:2024-06-03T08:36:32.897+0000 7f3d5015c640 10 monclient: _send_mon_message to mon.c at v1:172.21.15.88:6790/0 2024-06-03T08:36:32.899 INFO:teuthology.orchestra.run.smithi088.stderr:2024-06-03T08:36:32.897+0000 7f3d5015c640 1 -- 172.21.15.88:0/1768052638 --> v1:172.21.15.88:6790/0 -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f3d480671b0 con 0x7f3d480b4250 2024-06-03T08:36:32.951 INFO:teuthology.orchestra.run.smithi088.stderr:2024-06-03T08:36:32.951+0000 7fccc7ad1640 10 start_mon_command cmd=[{"prefix": "osd last-stat-seq", "id": 3}] 2024-06-03T08:36:32.952 INFO:teuthology.orchestra.run.smithi088.stderr:2024-06-03T08:36:32.951+0000 7fccc7ad1640 10 monclient: _send_command 2 [{"prefix": "osd last-stat-seq", "id": 3}] 2024-06-03T08:36:32.952 INFO:teuthology.orchestra.run.smithi088.stderr:2024-06-03T08:36:32.951+0000 7fccc7ad1640 10 monclient: _send_mon_message to mon.c at v1:172.21.15.88:6790/0 2024-06-03T08:36:32.952 INFO:teuthology.orchestra.run.smithi088.stderr:2024-06-03T08:36:32.951+0000 7fccc7ad1640 1 -- 172.21.15.88:0/1456172716 --> v1:172.21.15.88:6790/0 -- mon_command({"prefix": "osd last-stat-seq", "id": 3} v 0) v1 -- 0x7fccc0133d20 con 0x7fccc00a8a60 2024-06-03T08:36:32.957 INFO:teuthology.orchestra.run.smithi088.stderr:2024-06-03T08:36:32.956+0000 7f223dffb640 1 -- 172.21.15.88:0/598291210 <== mon.0 v1:172.21.15.88:6789/0 8 ==== osd_map(26..26 src has 1..26) v4 ==== 4090+0+0 (unknown 3637450554 0 0) 0x7f2238053e90 con 0x7f22480a89f0 2024-06-03T08:36:32.957 INFO:teuthology.orchestra.run.smithi088.stderr:2024-06-03T08:36:32.956+0000 7f223dffb640 10 client.4347.objecter ms_dispatch 0x7f2248000b90 osd_map(26..26 src has 1..26) v4 2024-06-03T08:36:32.957 INFO:teuthology.orchestra.run.smithi088.stderr:2024-06-03T08:36:32.956+0000 7f223dffb640 3 client.4347.objecter handle_osd_map got epochs [26,26] > 0 2024-06-03T08:36:32.957 INFO:teuthology.orchestra.run.smithi088.stderr:2024-06-03T08:36:32.956+0000 7f223dffb640 3 client.4347.objecter handle_osd_map decoding full epoch 26 2024-06-03T08:36:32.957 INFO:teuthology.orchestra.run.smithi088.stderr:2024-06-03T08:36:32.956+0000 7f223dffb640 20 client.4347.objecter dump_active .. 0 homeless 2024-06-03T08:36:32.957 INFO:teuthology.orchestra.run.smithi088.stderr:2024-06-03T08:36:32.956+0000 7f223dffb640 1 -- 172.21.15.88:0/598291210 <== mon.0 v1:172.21.15.88:6789/0 9 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+182498 (unknown 1092875540 0 3016760700) 0x7f223804f870 con 0x7f22480a89f0 2024-06-03T08:36:32.957 INFO:teuthology.orchestra.run.smithi088.stderr:2024-06-03T08:36:32.956+0000 7f223dffb640 10 monclient: handle_mon_command_ack 1 [{"prefix": "get_command_descriptions"}] 2024-06-03T08:36:32.957 INFO:teuthology.orchestra.run.smithi088.stderr:2024-06-03T08:36:32.956+0000 7f223dffb640 10 monclient: _finish_command 1 = system:0 2024-06-03T08:36:32.958 INFO:teuthology.orchestra.run.smithi088.stderr:2024-06-03T08:36:32.957+0000 7f83fcff9640 1 -- 172.21.15.88:0/4283667430 <== mon.1 v1:172.21.15.192:6789/0 5 ==== config(0 keys) v1 ==== 4+0+0 (unknown 0 0 0) 0x7f83f4001cf0 con 0x7f84000ae8c0 2024-06-03T08:36:32.958 INFO:teuthology.orchestra.run.smithi088.stderr:2024-06-03T08:36:32.957+0000 7f83fcff9640 10 monclient: handle_config config(0 keys) v1 2024-06-03T08:36:32.958 INFO:teuthology.orchestra.run.smithi088.stderr:2024-06-03T08:36:32.957+0000 7f83fcff9640 1 -- 172.21.15.88:0/4283667430 <== mon.1 v1:172.21.15.192:6789/0 6 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 2234374065 0 0) 0x7f83f4003a60 con 0x7f84000ae8c0 2024-06-03T08:36:32.958 INFO:teuthology.orchestra.run.smithi088.stderr:2024-06-03T08:36:32.957+0000 7f83fcff9640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-06-03T08:36:32.958 INFO:teuthology.orchestra.run.smithi088.stderr:2024-06-03T08:36:32.957+0000 7f83fcff9640 10 monclient: got monmap 1 from mon.b (according to old e1) 2024-06-03T08:36:32.958 INFO:teuthology.orchestra.run.smithi088.stderr:2024-06-03T08:36:32.957+0000 7f83fcff9640 10 monclient: dump: 2024-06-03T08:36:32.958 INFO:teuthology.orchestra.run.smithi088.stderr:epoch 1 2024-06-03T08:36:32.958 INFO:teuthology.orchestra.run.smithi088.stderr:fsid ec3c3fdd-3ac6-4d7d-9412-683130ed8b04 2024-06-03T08:36:32.959 INFO:teuthology.orchestra.run.smithi088.stderr:last_changed 2024-06-03T08:34:31.885125+0000 2024-06-03T08:36:32.959 INFO:teuthology.orchestra.run.smithi088.stderr:created 2024-06-03T08:34:31.885125+0000 2024-06-03T08:36:32.959 INFO:teuthology.orchestra.run.smithi088.stderr:min_mon_release 18 (reef) 2024-06-03T08:36:32.959 INFO:teuthology.orchestra.run.smithi088.stderr:election_strategy: 3 2024-06-03T08:36:32.959 INFO:teuthology.orchestra.run.smithi088.stderr:0: v1:172.21.15.88:6789/0 mon.a 2024-06-03T08:36:32.959 INFO:teuthology.orchestra.run.smithi088.stderr:1: v1:172.21.15.192:6789/0 mon.b 2024-06-03T08:36:32.959 INFO:teuthology.orchestra.run.smithi088.stderr:2: v1:172.21.15.88:6790/0 mon.c 2024-06-03T08:36:32.959 INFO:teuthology.orchestra.run.smithi088.stderr: 2024-06-03T08:36:32.959 INFO:teuthology.orchestra.run.smithi088.stderr:2024-06-03T08:36:32.957+0000 7f83fe7fc640 4 set_mon_vals no callback set 2024-06-03T08:36:32.960 INFO:teuthology.orchestra.run.smithi088.stderr:2024-06-03T08:36:32.958+0000 7f83fcff9640 1 -- 172.21.15.88:0/4283667430 <== mon.1 v1:172.21.15.192:6789/0 7 ==== mgrmap(e 5) v1 ==== 89110+0+0 (unknown 873533973 0 0) 0x7f83f4003bc0 con 0x7f84000ae8c0 2024-06-03T08:36:32.960 INFO:teuthology.orchestra.run.smithi088.stderr:2024-06-03T08:36:32.959+0000 7f83fffff640 10 monclient: get_auth_request con 0x7f83ec06d080 auth_method 0 2024-06-03T08:36:32.960 INFO:teuthology.orchestra.run.smithi088.stderr:2024-06-03T08:36:32.959+0000 7f83fcff9640 10 client.4354.objecter ms_handle_connect 0x7f83ec06d080 2024-06-03T08:36:32.961 INFO:teuthology.orchestra.run.smithi088.stderr:2024-06-03T08:36:32.961+0000 7f83fcff9640 1 -- 172.21.15.88:0/4283667430 <== mon.1 v1:172.21.15.192:6789/0 8 ==== osd_map(26..26 src has 1..26) v4 ==== 4090+0+0 (unknown 3637450554 0 0) 0x7f83f40541d0 con 0x7f84000ae8c0 2024-06-03T08:36:32.961 INFO:teuthology.orchestra.run.smithi088.stderr:2024-06-03T08:36:32.961+0000 7f83fcff9640 10 client.4354.objecter ms_dispatch 0x7f8400000b90 osd_map(26..26 src has 1..26) v4 2024-06-03T08:36:32.961 INFO:teuthology.orchestra.run.smithi088.stderr:2024-06-03T08:36:32.961+0000 7f83fcff9640 3 client.4354.objecter handle_osd_map got epochs [26,26] > 0 2024-06-03T08:36:32.961 INFO:teuthology.orchestra.run.smithi088.stderr:2024-06-03T08:36:32.961+0000 7f83fcff9640 3 client.4354.objecter handle_osd_map decoding full epoch 26 2024-06-03T08:36:32.962 INFO:teuthology.orchestra.run.smithi088.stderr:2024-06-03T08:36:32.961+0000 7f83fcff9640 20 client.4354.objecter dump_active .. 0 homeless 2024-06-03T08:36:32.965 INFO:teuthology.orchestra.run.smithi088.stderr:2024-06-03T08:36:32.964+0000 7f83fcff9640 1 -- 172.21.15.88:0/4283667430 <== mon.1 v1:172.21.15.192:6789/0 9 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+182498 (unknown 1092875540 0 3016760700) 0x7f83f404fbb0 con 0x7f84000ae8c0 2024-06-03T08:36:32.965 INFO:teuthology.orchestra.run.smithi088.stderr:2024-06-03T08:36:32.964+0000 7f83fcff9640 10 monclient: handle_mon_command_ack 1 [{"prefix": "get_command_descriptions"}] 2024-06-03T08:36:32.965 INFO:teuthology.orchestra.run.smithi088.stderr:2024-06-03T08:36:32.964+0000 7f83fcff9640 10 monclient: _finish_command 1 = system:0 2024-06-03T08:36:32.966 INFO:teuthology.orchestra.run.smithi088.stderr:2024-06-03T08:36:32.966+0000 7f169bfff640 10 client.?.objecter ms_handle_connect 0x7f16b41451c0 2024-06-03T08:36:32.966 INFO:teuthology.orchestra.run.smithi088.stderr:2024-06-03T08:36:32.966+0000 7f169bfff640 10 client.?.objecter resend_mon_ops 2024-06-03T08:36:32.967 INFO:teuthology.orchestra.run.smithi088.stderr:2024-06-03T08:36:32.966+0000 7feb14d92640 1 --1- >> v1:172.21.15.192:6789/0 conn(0x7feb100a8960 0x7feb100aab10 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=1).handle_server_banner_and_identify peer v1:172.21.15.192:6789/0 says I am v1:172.21.15.88:52608/0 (socket says 172.21.15.88:52608) 2024-06-03T08:36:32.967 INFO:teuthology.orchestra.run.smithi088.stderr:2024-06-03T08:36:32.966+0000 7feb14d92640 1 -- 172.21.15.88:0/1366328441 learned_addr learned my addr 172.21.15.88:0/1366328441 (peer_addr_for_me v1:172.21.15.88:0/0) 2024-06-03T08:36:32.967 INFO:teuthology.orchestra.run.smithi088.stderr:2024-06-03T08:36:32.967+0000 7feb0d7fa640 10 client.?.objecter ms_handle_connect 0x7feb100a8960 2024-06-03T08:36:32.967 INFO:teuthology.orchestra.run.smithi088.stderr:2024-06-03T08:36:32.967+0000 7feb0d7fa640 10 client.?.objecter resend_mon_ops 2024-06-03T08:36:32.973 INFO:teuthology.orchestra.run.smithi088.stderr:2024-06-03T08:36:32.972+0000 7feb0d7fa640 10 client.?.objecter ms_handle_connect 0x7feb1013eec0 2024-06-03T08:36:32.973 INFO:teuthology.orchestra.run.smithi088.stderr:2024-06-03T08:36:32.972+0000 7feb0d7fa640 10 client.?.objecter resend_mon_ops 2024-06-03T08:36:32.974 INFO:teuthology.orchestra.run.smithi088.stderr:2024-06-03T08:36:32.973+0000 7f169bfff640 1 -- 172.21.15.88:0/2965169668 <== mon.0 v1:172.21.15.88:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 2696716242 0 0) 0x7f16b4133cb0 con 0x7f16b41451c0 2024-06-03T08:36:32.974 INFO:teuthology.orchestra.run.smithi088.stderr:2024-06-03T08:36:32.973+0000 7f169bfff640 10 monclient(hunting): _negotiate 2024-06-03T08:36:32.974 INFO:teuthology.orchestra.run.smithi088.stderr:2024-06-03T08:36:32.973+0000 7f169bfff640 10 monclient(hunting): _init_auth method 2 2024-06-03T08:36:32.974 INFO:teuthology.orchestra.run.smithi088.stderr:2024-06-03T08:36:32.973+0000 7f169bfff640 10 monclient(hunting): _init_auth creating new auth 2024-06-03T08:36:32.974 INFO:teuthology.orchestra.run.smithi088.stderr:2024-06-03T08:36:32.973+0000 7f169bfff640 10 monclient(hunting): my global_id is 4356 2024-06-03T08:36:32.974 INFO:teuthology.orchestra.run.smithi088.stderr:2024-06-03T08:36:32.973+0000 7f169bfff640 1 -- 172.21.15.88:0/2965169668 --> v1:172.21.15.88:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f16a0004eb0 con 0x7f16b41451c0 2024-06-03T08:36:32.974 INFO:teuthology.orchestra.run.smithi088.stderr:2024-06-03T08:36:32.974+0000 7f169bfff640 10 client.?.objecter ms_handle_connect 0x7f16b40a89f0 2024-06-03T08:36:32.975 INFO:teuthology.orchestra.run.smithi088.stderr:2024-06-03T08:36:32.974+0000 7f169bfff640 10 client.?.objecter resend_mon_ops 2024-06-03T08:36:32.980 INFO:teuthology.orchestra.run.smithi088.stderr:2024-06-03T08:36:32.979+0000 7f169bfff640 1 -- 172.21.15.88:0/2965169668 <== mon.0 v1:172.21.15.88:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 764+0+0 (unknown 3214681167 0 0) 0x7f16a0004eb0 con 0x7f16b41451c0 2024-06-03T08:36:32.980 INFO:teuthology.orchestra.run.smithi088.stderr:2024-06-03T08:36:32.979+0000 7f169bfff640 1 -- 172.21.15.88:0/2965169668 --> v1:172.21.15.88:6789/0 -- auth(proto 2 165 bytes epoch 0) v1 -- 0x7f16b4133cb0 con 0x7f16b41451c0 2024-06-03T08:36:32.982 INFO:teuthology.orchestra.run.smithi088.stderr:2024-06-03T08:36:32.982+0000 7f169bfff640 1 -- 172.21.15.88:0/2965169668 <== mon.0 v1:172.21.15.88:6789/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 2234374065 0 0) 0x7f16ac001b30 con 0x7f16b41451c0 2024-06-03T08:36:32.982 INFO:teuthology.orchestra.run.smithi088.stderr:2024-06-03T08:36:32.982+0000 7f169bfff640 10 monclient(hunting): handle_monmap mon_map magic: 0 v1 2024-06-03T08:36:32.982 INFO:teuthology.orchestra.run.smithi088.stderr:2024-06-03T08:36:32.982+0000 7f169bfff640 10 monclient(hunting): got monmap 1 from mon.a (according to old e1) 2024-06-03T08:36:32.982 INFO:teuthology.orchestra.run.smithi088.stderr:2024-06-03T08:36:32.982+0000 7f169bfff640 10 monclient(hunting): dump: 2024-06-03T08:36:32.982 INFO:teuthology.orchestra.run.smithi088.stderr:epoch 1 2024-06-03T08:36:32.982 INFO:teuthology.orchestra.run.smithi088.stderr:fsid ec3c3fdd-3ac6-4d7d-9412-683130ed8b04 2024-06-03T08:36:32.983 INFO:teuthology.orchestra.run.smithi088.stderr:last_changed 2024-06-03T08:34:31.885125+0000 2024-06-03T08:36:32.983 INFO:teuthology.orchestra.run.smithi088.stderr:created 2024-06-03T08:34:31.885125+0000 2024-06-03T08:36:32.983 INFO:teuthology.orchestra.run.smithi088.stderr:min_mon_release 18 (reef) 2024-06-03T08:36:32.983 INFO:teuthology.orchestra.run.smithi088.stderr:election_strategy: 3 2024-06-03T08:36:32.983 INFO:teuthology.orchestra.run.smithi088.stderr:0: v1:172.21.15.88:6789/0 mon.a 2024-06-03T08:36:32.983 INFO:teuthology.orchestra.run.smithi088.stderr:1: v1:172.21.15.192:6789/0 mon.b 2024-06-03T08:36:32.983 INFO:teuthology.orchestra.run.smithi088.stderr:2: v1:172.21.15.88:6790/0 mon.c 2024-06-03T08:36:32.983 INFO:teuthology.orchestra.run.smithi088.stderr: 2024-06-03T08:36:32.986 INFO:teuthology.orchestra.run.smithi088.stderr:2024-06-03T08:36:32.985+0000 7f169bfff640 1 -- 172.21.15.88:0/2965169668 <== mon.1 v1:172.21.15.192:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 828082400 0 0) 0x7f16b41da190 con 0x7f16b40a89f0 2024-06-03T08:36:32.986 INFO:teuthology.orchestra.run.smithi088.stderr:2024-06-03T08:36:32.985+0000 7f169bfff640 10 monclient(hunting): _negotiate 2024-06-03T08:36:32.986 INFO:teuthology.orchestra.run.smithi088.stderr:2024-06-03T08:36:32.985+0000 7f169bfff640 10 monclient(hunting): _init_auth method 2 2024-06-03T08:36:32.986 INFO:teuthology.orchestra.run.smithi088.stderr:2024-06-03T08:36:32.985+0000 7f169bfff640 10 monclient(hunting): _init_auth creating new auth 2024-06-03T08:36:32.986 INFO:teuthology.orchestra.run.smithi088.stderr:2024-06-03T08:36:32.985+0000 7f169bfff640 10 monclient(hunting): my global_id is 4363 2024-06-03T08:36:32.986 INFO:teuthology.orchestra.run.smithi088.stderr:2024-06-03T08:36:32.985+0000 7f169bfff640 1 -- 172.21.15.88:0/2965169668 --> v1:172.21.15.192:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f16a0004eb0 con 0x7f16b40a89f0 2024-06-03T08:36:32.986 INFO:teuthology.orchestra.run.smithi088.stderr:2024-06-03T08:36:32.986+0000 7feb0d7fa640 1 -- 172.21.15.88:0/1366328441 <== mon.1 v1:172.21.15.192:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 1772707976 0 0) 0x7feb100af340 con 0x7feb100a8960 2024-06-03T08:36:32.987 INFO:teuthology.orchestra.run.smithi088.stderr:2024-06-03T08:36:32.986+0000 7feb0d7fa640 10 monclient(hunting): _negotiate 2024-06-03T08:36:32.987 INFO:teuthology.orchestra.run.smithi088.stderr:2024-06-03T08:36:32.986+0000 7feb0d7fa640 10 monclient(hunting): _init_auth method 2 2024-06-03T08:36:32.987 INFO:teuthology.orchestra.run.smithi088.stderr:2024-06-03T08:36:32.986+0000 7feb0d7fa640 10 monclient(hunting): _init_auth creating new auth 2024-06-03T08:36:32.987 INFO:teuthology.orchestra.run.smithi088.stderr:2024-06-03T08:36:32.986+0000 7feb0d7fa640 10 monclient(hunting): my global_id is 4360 2024-06-03T08:36:32.987 INFO:teuthology.orchestra.run.smithi088.stderr:2024-06-03T08:36:32.986+0000 7feb0d7fa640 1 -- 172.21.15.88:0/1366328441 --> v1:172.21.15.192:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7feaf4004eb0 con 0x7feb100a8960 2024-06-03T08:36:32.987 INFO:teuthology.orchestra.run.smithi088.stderr:2024-06-03T08:36:32.986+0000 7fb0feffd640 1 -- 172.21.15.88:0/4168539993 <== mon.1 v1:172.21.15.192:6789/0 10 ==== mon_command_ack([{"prefix": "osd last-stat-seq", "id": 2}]=0 v0) v1 ==== 74+0+12 (unknown 92182286 0 3651700234) 0x7fb0f0051e30 con 0x7fb10813eec0 2024-06-03T08:36:32.987 INFO:teuthology.orchestra.run.smithi088.stderr:2024-06-03T08:36:32.986+0000 7fb0feffd640 10 monclient: handle_mon_command_ack 2 [{"prefix": "osd last-stat-seq", "id": 2}] 2024-06-03T08:36:32.987 INFO:teuthology.orchestra.run.smithi088.stderr:2024-06-03T08:36:32.986+0000 7fb0feffd640 10 monclient: _finish_command 2 = system:0 2024-06-03T08:36:32.988 INFO:teuthology.orchestra.run.smithi088.stdout:85899345928 2024-06-03T08:36:32.988 INFO:teuthology.orchestra.run.smithi088.stderr:2024-06-03T08:36:32.988+0000 7fb10c947640 10 librados: watch_flush enter 2024-06-03T08:36:32.988 INFO:teuthology.orchestra.run.smithi088.stderr:2024-06-03T08:36:32.988+0000 7fb10c947640 10 librados: watch_flush exit 2024-06-03T08:36:32.988 INFO:teuthology.orchestra.run.smithi088.stderr:2024-06-03T08:36:32.988+0000 7fb10c947640 20 client.4339.objecter shutdown clearing up homeless session... 2024-06-03T08:36:32.988 INFO:teuthology.orchestra.run.smithi088.stderr:2024-06-03T08:36:32.988+0000 7fb10c947640 10 client.4339.objecter successfully canceled tick 2024-06-03T08:36:32.989 INFO:teuthology.orchestra.run.smithi088.stderr:2024-06-03T08:36:32.988+0000 7feb0d7fa640 1 -- 172.21.15.88:0/1366328441 <== mon.0 v1:172.21.15.88:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 2392931288 0 0) 0x7feb10139d00 con 0x7feb1013eec0 2024-06-03T08:36:32.989 INFO:teuthology.orchestra.run.smithi088.stderr:2024-06-03T08:36:32.988+0000 7feb0d7fa640 10 monclient(hunting): _negotiate 2024-06-03T08:36:32.989 INFO:teuthology.orchestra.run.smithi088.stderr:2024-06-03T08:36:32.988+0000 7feb0d7fa640 10 monclient(hunting): _init_auth method 2 2024-06-03T08:36:32.989 INFO:teuthology.orchestra.run.smithi088.stderr:2024-06-03T08:36:32.988+0000 7feb0d7fa640 10 monclient(hunting): _init_auth creating new auth 2024-06-03T08:36:32.989 INFO:teuthology.orchestra.run.smithi088.stderr:2024-06-03T08:36:32.988+0000 7feb0d7fa640 10 monclient(hunting): my global_id is 4359 2024-06-03T08:36:32.989 INFO:teuthology.orchestra.run.smithi088.stderr:2024-06-03T08:36:32.988+0000 7feb0d7fa640 1 -- 172.21.15.88:0/1366328441 --> v1:172.21.15.88:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7feb100af340 con 0x7feb1013eec0 2024-06-03T08:36:32.990 INFO:teuthology.orchestra.run.smithi088.stderr:2024-06-03T08:36:32.988+0000 7fb10c947640 1 -- 172.21.15.88:0/4168539993 >> v1:172.21.15.192:6800/34229 conn(0x7fb0e806d200 legacy=0x7fb0e806f6c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-06-03T08:36:32.990 INFO:teuthology.orchestra.run.smithi088.stderr:2024-06-03T08:36:32.988+0000 7fb10c947640 10 monclient: shutdown 2024-06-03T08:36:32.990 INFO:teuthology.orchestra.run.smithi088.stderr:2024-06-03T08:36:32.988+0000 7fb10c947640 20 monclient: shutdown discarding 0 pending message(s) 2024-06-03T08:36:32.990 INFO:teuthology.orchestra.run.smithi088.stderr:2024-06-03T08:36:32.988+0000 7fb10c947640 1 -- 172.21.15.88:0/4168539993 >> v1:172.21.15.192:6789/0 conn(0x7fb10813eec0 legacy=0x7fb1081d9ee0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-06-03T08:36:32.990 INFO:teuthology.orchestra.run.smithi088.stderr:2024-06-03T08:36:32.988+0000 7fb10c947640 1 -- 172.21.15.88:0/4168539993 shutdown_connections 2024-06-03T08:36:32.990 INFO:teuthology.orchestra.run.smithi088.stderr:2024-06-03T08:36:32.988+0000 7fb10c947640 1 -- 172.21.15.88:0/4168539993 >> 172.21.15.88:0/4168539993 conn(0x7fb1080b70d0 msgr2=0x7fb10813e580 unknown :-1 s=STATE_NONE l=0).mark_down 2024-06-03T08:36:32.991 INFO:teuthology.orchestra.run.smithi088.stderr:2024-06-03T08:36:32.988+0000 7fb10c947640 1 -- 172.21.15.88:0/4168539993 shutdown_connections 2024-06-03T08:36:32.991 INFO:teuthology.orchestra.run.smithi088.stderr:2024-06-03T08:36:32.988+0000 7fb10c947640 1 -- 172.21.15.88:0/4168539993 wait complete. 2024-06-03T08:36:32.991 INFO:teuthology.orchestra.run.smithi088.stderr:2024-06-03T08:36:32.988+0000 7fb10c947640 1 librados: shutdown 2024-06-03T08:36:32.999 INFO:teuthology.orchestra.run.smithi088.stderr:2024-06-03T08:36:32.998+0000 7feb0d7fa640 1 -- 172.21.15.88:0/1366328441 <== mon.1 v1:172.21.15.192:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 764+0+0 (unknown 1062950190 0 0) 0x7feaf4004eb0 con 0x7feb100a8960 2024-06-03T08:36:32.999 INFO:teuthology.orchestra.run.smithi088.stderr:2024-06-03T08:36:32.998+0000 7feb0d7fa640 1 -- 172.21.15.88:0/1366328441 --> v1:172.21.15.192:6789/0 -- auth(proto 2 165 bytes epoch 0) v1 -- 0x7feb10139d00 con 0x7feb100a8960 2024-06-03T08:36:32.999 INFO:teuthology.orchestra.run.smithi088.stderr:2024-06-03T08:36:32.998+0000 7feb0d7fa640 1 -- 172.21.15.88:0/1366328441 <== mon.1 v1:172.21.15.192:6789/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 2234374065 0 0) 0x7feb040018d0 con 0x7feb100a8960 2024-06-03T08:36:32.999 INFO:teuthology.orchestra.run.smithi088.stderr:2024-06-03T08:36:32.998+0000 7feb0d7fa640 10 monclient(hunting): handle_monmap mon_map magic: 0 v1 2024-06-03T08:36:32.999 INFO:teuthology.orchestra.run.smithi088.stderr:2024-06-03T08:36:32.998+0000 7feb0d7fa640 10 monclient(hunting): got monmap 1 from mon.noname-b (according to old e1) 2024-06-03T08:36:32.999 INFO:teuthology.orchestra.run.smithi088.stderr:2024-06-03T08:36:32.998+0000 7feb0d7fa640 10 monclient(hunting): dump: 2024-06-03T08:36:32.999 INFO:teuthology.orchestra.run.smithi088.stderr:epoch 1 2024-06-03T08:36:32.999 INFO:teuthology.orchestra.run.smithi088.stderr:fsid ec3c3fdd-3ac6-4d7d-9412-683130ed8b04 2024-06-03T08:36:32.999 INFO:teuthology.orchestra.run.smithi088.stderr:last_changed 2024-06-03T08:34:31.885125+0000 2024-06-03T08:36:32.999 INFO:teuthology.orchestra.run.smithi088.stderr:created 2024-06-03T08:34:31.885125+0000 2024-06-03T08:36:32.999 INFO:teuthology.orchestra.run.smithi088.stderr:min_mon_release 18 (reef) 2024-06-03T08:36:32.999 INFO:teuthology.orchestra.run.smithi088.stderr:election_strategy: 3 2024-06-03T08:36:32.999 INFO:teuthology.orchestra.run.smithi088.stderr:0: v1:172.21.15.88:6789/0 mon.a 2024-06-03T08:36:32.999 INFO:teuthology.orchestra.run.smithi088.stderr:1: v1:172.21.15.192:6789/0 mon.b 2024-06-03T08:36:33.000 INFO:teuthology.orchestra.run.smithi088.stderr:2: v1:172.21.15.88:6790/0 mon.c 2024-06-03T08:36:33.000 INFO:teuthology.orchestra.run.smithi088.stderr: 2024-06-03T08:36:33.000 INFO:teuthology.orchestra.run.smithi088.stderr:2024-06-03T08:36:32.998+0000 7f169bfff640 1 -- 172.21.15.88:0/2965169668 <== mon.0 v1:172.21.15.88:6789/0 4 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 580+0+0 (unknown 2704038626 0 0) 0x7f16b4133cb0 con 0x7f16b41451c0 2024-06-03T08:36:33.000 INFO:teuthology.orchestra.run.smithi088.stderr:2024-06-03T08:36:32.998+0000 7f169bfff640 1 -- 172.21.15.88:0/2965169668 >> v1:172.21.15.88:6790/0 conn(0x7f16b40b4250 legacy=0x7f16b4132e90 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-06-03T08:36:33.000 INFO:teuthology.orchestra.run.smithi088.stderr:2024-06-03T08:36:32.998+0000 7f169bfff640 1 -- 172.21.15.88:0/2965169668 >> v1:172.21.15.192:6789/0 conn(0x7f16b40a89f0 legacy=0x7f16b413f3b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-06-03T08:36:33.000 INFO:teuthology.orchestra.run.smithi088.stderr:2024-06-03T08:36:32.998+0000 7f169bfff640 10 monclient: _finish_hunting 0 2024-06-03T08:36:33.000 INFO:teuthology.orchestra.run.smithi088.stderr:2024-06-03T08:36:32.998+0000 7f169bfff640 1 monclient: found mon.a 2024-06-03T08:36:33.000 INFO:teuthology.orchestra.run.smithi088.stderr:2024-06-03T08:36:32.998+0000 7feb0d7fa640 10 monclient(hunting): _finish_auth 0 2024-06-03T08:36:33.002 INFO:teuthology.orchestra.run.smithi088.stderr:2024-06-03T08:36:32.998+0000 7f169bfff640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-06-03T08:36:33.002 INFO:teuthology.orchestra.run.smithi088.stderr:2024-06-03T08:36:32.998+0000 7f169bfff640 10 monclient: _send_mon_message to mon.a at v1:172.21.15.88:6789/0 2024-06-03T08:36:33.002 INFO:teuthology.orchestra.run.smithi088.stderr:2024-06-03T08:36:32.998+0000 7f169bfff640 1 -- 172.21.15.88:0/2965169668 --> v1:172.21.15.88:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f16b41da670 con 0x7f16b41451c0 2024-06-03T08:36:33.002 INFO:teuthology.orchestra.run.smithi088.stderr:2024-06-03T08:36:32.998+0000 7f169bfff640 10 monclient: _finish_auth 0 2024-06-03T08:36:33.002 INFO:teuthology.orchestra.run.smithi088.stderr:2024-06-03T08:36:32.998+0000 7f169bfff640 10 monclient: _check_auth_tickets 2024-06-03T08:36:33.002 INFO:teuthology.orchestra.run.smithi088.stderr:2024-06-03T08:36:32.998+0000 7f169bfff640 20 monclient: _check_auth_rotating not needed by client.admin 2024-06-03T08:36:33.002 INFO:teuthology.orchestra.run.smithi088.stderr:2024-06-03T08:36:32.998+0000 7f16b9c8f640 5 monclient: authenticate success, global_id 4356 2024-06-03T08:36:33.002 INFO:teuthology.orchestra.run.smithi088.stderr:2024-06-03T08:36:32.998+0000 7f16b9c8f640 10 monclient: _renew_subs 2024-06-03T08:36:33.002 INFO:teuthology.orchestra.run.smithi088.stderr:2024-06-03T08:36:32.998+0000 7f16b9c8f640 10 monclient: _send_mon_message to mon.a at v1:172.21.15.88:6789/0 2024-06-03T08:36:33.002 INFO:teuthology.orchestra.run.smithi088.stderr:2024-06-03T08:36:32.998+0000 7f16b9c8f640 1 -- 172.21.15.88:0/2965169668 --> v1:172.21.15.88:6789/0 -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f16b413fac0 con 0x7f16b41451c0 2024-06-03T08:36:33.002 INFO:teuthology.orchestra.run.smithi088.stderr:2024-06-03T08:36:32.998+0000 7f16b9c8f640 10 client.4356.objecter _maybe_request_map subscribing (onetime) to next osd map 2024-06-03T08:36:33.003 INFO:teuthology.orchestra.run.smithi088.stderr:2024-06-03T08:36:32.998+0000 7f16b9c8f640 10 monclient: _renew_subs 2024-06-03T08:36:33.003 INFO:teuthology.orchestra.run.smithi088.stderr:2024-06-03T08:36:32.998+0000 7f16b9c8f640 10 monclient: _send_mon_message to mon.a at v1:172.21.15.88:6789/0 2024-06-03T08:36:33.003 INFO:teuthology.orchestra.run.smithi088.stderr:2024-06-03T08:36:32.998+0000 7f16b9c8f640 1 -- 172.21.15.88:0/2965169668 --> v1:172.21.15.88:6789/0 -- mon_subscribe({osdmap=0}) v3 -- 0x7f16b4138a50 con 0x7f16b41451c0 2024-06-03T08:36:33.003 INFO:teuthology.orchestra.run.smithi088.stderr:2024-06-03T08:36:32.998+0000 7f16b9c8f640 1 librados: init done 2024-06-03T08:36:33.003 INFO:teuthology.orchestra.run.smithi088.stderr:2024-06-03T08:36:32.999+0000 7f16b9c8f640 10 start_mon_command cmd=[{"prefix": "get_command_descriptions"}] 2024-06-03T08:36:33.003 INFO:teuthology.orchestra.run.smithi088.stderr:2024-06-03T08:36:33.000+0000 7f16b9c8f640 10 monclient: _send_command 1 [{"prefix": "get_command_descriptions"}] 2024-06-03T08:36:33.003 INFO:teuthology.orchestra.run.smithi088.stderr:2024-06-03T08:36:33.000+0000 7f16b9c8f640 10 monclient: _send_mon_message to mon.a at v1:172.21.15.88:6789/0 2024-06-03T08:36:33.003 INFO:teuthology.orchestra.run.smithi088.stderr:2024-06-03T08:36:33.000+0000 7f16b9c8f640 1 -- 172.21.15.88:0/2965169668 --> v1:172.21.15.88:6789/0 -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f16b40671b0 con 0x7f16b41451c0 2024-06-03T08:36:33.004 INFO:tasks.ceph.ceph_manager.ceph:need seq 85899345929 got 85899345928 for osd.2 2024-06-03T08:36:33.005 INFO:teuthology.orchestra.run.smithi088.stderr:2024-06-03T08:36:33.004+0000 7feb0d7fa640 1 -- 172.21.15.88:0/1366328441 <== mon.1 v1:172.21.15.192:6789/0 4 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 580+0+0 (unknown 3705361709 0 0) 0x7feb10139d00 con 0x7feb100a8960 2024-06-03T08:36:33.005 INFO:teuthology.orchestra.run.smithi088.stderr:2024-06-03T08:36:33.004+0000 7feb0d7fa640 1 -- 172.21.15.88:0/1366328441 >> v1:172.21.15.88:6790/0 conn(0x7feb101450f0 legacy=0x7feb101395f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-06-03T08:36:33.005 INFO:teuthology.orchestra.run.smithi088.stderr:2024-06-03T08:36:33.004+0000 7feb0d7fa640 1 -- 172.21.15.88:0/1366328441 >> v1:172.21.15.88:6789/0 conn(0x7feb1013eec0 legacy=0x7feb10138ee0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-06-03T08:36:33.005 INFO:teuthology.orchestra.run.smithi088.stderr:2024-06-03T08:36:33.004+0000 7feb0d7fa640 10 monclient: _finish_hunting 0 2024-06-03T08:36:33.005 INFO:teuthology.orchestra.run.smithi088.stderr:2024-06-03T08:36:33.004+0000 7feb0d7fa640 1 monclient: found mon.b 2024-06-03T08:36:33.005 INFO:teuthology.orchestra.run.smithi088.stderr:2024-06-03T08:36:33.004+0000 7feb0d7fa640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-06-03T08:36:33.006 INFO:teuthology.orchestra.run.smithi088.stderr:2024-06-03T08:36:33.004+0000 7feb0d7fa640 10 monclient: _send_mon_message to mon.b at v1:172.21.15.192:6789/0 2024-06-03T08:36:33.006 INFO:teuthology.orchestra.run.smithi088.stderr:2024-06-03T08:36:33.004+0000 7feb0d7fa640 1 -- 172.21.15.88:0/1366328441 --> v1:172.21.15.192:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7feb100af860 con 0x7feb100a8960 2024-06-03T08:36:33.006 INFO:teuthology.orchestra.run.smithi088.stderr:2024-06-03T08:36:33.004+0000 7feb0d7fa640 10 monclient: _finish_auth 0 2024-06-03T08:36:33.006 INFO:teuthology.orchestra.run.smithi088.stderr:2024-06-03T08:36:33.004+0000 7feb0d7fa640 10 monclient: _check_auth_tickets 2024-06-03T08:36:33.006 INFO:teuthology.orchestra.run.smithi088.stderr:2024-06-03T08:36:33.004+0000 7feb0d7fa640 20 monclient: _check_auth_rotating not needed by client.admin 2024-06-03T08:36:33.006 INFO:teuthology.orchestra.run.smithi088.stderr:2024-06-03T08:36:33.004+0000 7feb16ff6640 5 monclient: authenticate success, global_id 4360 2024-06-03T08:36:33.006 INFO:teuthology.orchestra.run.smithi088.stderr:2024-06-03T08:36:33.004+0000 7feb16ff6640 10 monclient: _renew_subs 2024-06-03T08:36:33.006 INFO:teuthology.orchestra.run.smithi088.stderr:2024-06-03T08:36:33.004+0000 7feb16ff6640 10 monclient: _send_mon_message to mon.b at v1:172.21.15.192:6789/0 2024-06-03T08:36:33.006 INFO:teuthology.orchestra.run.smithi088.stderr:2024-06-03T08:36:33.004+0000 7feb16ff6640 1 -- 172.21.15.88:0/1366328441 --> v1:172.21.15.192:6789/0 -- mon_subscribe({mgrmap=0+}) v3 -- 0x7feb100afb10 con 0x7feb100a8960 2024-06-03T08:36:33.007 INFO:teuthology.orchestra.run.smithi088.stderr:2024-06-03T08:36:33.005+0000 7feb16ff6640 10 client.4360.objecter _maybe_request_map subscribing (onetime) to next osd map 2024-06-03T08:36:33.007 INFO:teuthology.orchestra.run.smithi088.stderr:2024-06-03T08:36:33.005+0000 7feb16ff6640 10 monclient: _renew_subs 2024-06-03T08:36:33.007 INFO:teuthology.orchestra.run.smithi088.stderr:2024-06-03T08:36:33.005+0000 7feb16ff6640 10 monclient: _send_mon_message to mon.b at v1:172.21.15.192:6789/0 2024-06-03T08:36:33.007 INFO:teuthology.orchestra.run.smithi088.stderr:2024-06-03T08:36:33.005+0000 7feb16ff6640 1 -- 172.21.15.88:0/1366328441 --> v1:172.21.15.192:6789/0 -- mon_subscribe({osdmap=0}) v3 -- 0x7feb100aff80 con 0x7feb100a8960 2024-06-03T08:36:33.007 INFO:teuthology.orchestra.run.smithi088.stderr:2024-06-03T08:36:33.005+0000 7feb16ff6640 1 librados: init done 2024-06-03T08:36:33.007 INFO:teuthology.orchestra.run.smithi088.stderr:2024-06-03T08:36:33.006+0000 7feb16ff6640 10 start_mon_command cmd=[{"prefix": "get_command_descriptions"}] 2024-06-03T08:36:33.008 INFO:teuthology.orchestra.run.smithi088.stderr:2024-06-03T08:36:33.006+0000 7feb16ff6640 10 monclient: _send_command 1 [{"prefix": "get_command_descriptions"}] 2024-06-03T08:36:33.008 INFO:teuthology.orchestra.run.smithi088.stderr:2024-06-03T08:36:33.006+0000 7feb16ff6640 10 monclient: _send_mon_message to mon.b at v1:172.21.15.192:6789/0 2024-06-03T08:36:33.008 INFO:teuthology.orchestra.run.smithi088.stderr:2024-06-03T08:36:33.006+0000 7feb16ff6640 1 -- 172.21.15.88:0/1366328441 --> v1:172.21.15.192:6789/0 -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7feb100a9e60 con 0x7feb100a8960 2024-06-03T08:36:33.090 INFO:teuthology.orchestra.run.smithi088.stderr:2024-06-03T08:36:33.089+0000 7f50d97fa640 1 -- 172.21.15.88:0/1176708298 <== mon.2 v1:172.21.15.88:6790/0 5 ==== config(0 keys) v1 ==== 4+0+0 (unknown 0 0 0) 0x7f50c8001d70 con 0x7f50dc0b4250 2024-06-03T08:36:33.090 INFO:teuthology.orchestra.run.smithi088.stderr:2024-06-03T08:36:33.089+0000 7f50d97fa640 10 monclient: handle_config config(0 keys) v1 2024-06-03T08:36:33.090 INFO:teuthology.orchestra.run.smithi088.stderr:2024-06-03T08:36:33.089+0000 7f50d97fa640 1 -- 172.21.15.88:0/1176708298 <== mon.2 v1:172.21.15.88:6790/0 6 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 2234374065 0 0) 0x7f50c80037b0 con 0x7f50dc0b4250 2024-06-03T08:36:33.090 INFO:teuthology.orchestra.run.smithi088.stderr:2024-06-03T08:36:33.089+0000 7f50d97fa640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-06-03T08:36:33.090 INFO:teuthology.orchestra.run.smithi088.stderr:2024-06-03T08:36:33.089+0000 7f50d97fa640 10 monclient: got monmap 1 from mon.c (according to old e1) 2024-06-03T08:36:33.090 INFO:teuthology.orchestra.run.smithi088.stderr:2024-06-03T08:36:33.089+0000 7f50d97fa640 10 monclient: dump: 2024-06-03T08:36:33.090 INFO:teuthology.orchestra.run.smithi088.stderr:epoch 1 2024-06-03T08:36:33.090 INFO:teuthology.orchestra.run.smithi088.stderr:fsid ec3c3fdd-3ac6-4d7d-9412-683130ed8b04 2024-06-03T08:36:33.090 INFO:teuthology.orchestra.run.smithi088.stderr:last_changed 2024-06-03T08:34:31.885125+0000 2024-06-03T08:36:33.091 INFO:teuthology.orchestra.run.smithi088.stderr:created 2024-06-03T08:34:31.885125+0000 2024-06-03T08:36:33.091 INFO:teuthology.orchestra.run.smithi088.stderr:min_mon_release 18 (reef) 2024-06-03T08:36:33.091 INFO:teuthology.orchestra.run.smithi088.stderr:election_strategy: 3 2024-06-03T08:36:33.091 INFO:teuthology.orchestra.run.smithi088.stderr:0: v1:172.21.15.88:6789/0 mon.a 2024-06-03T08:36:33.091 INFO:teuthology.orchestra.run.smithi088.stderr:1: v1:172.21.15.192:6789/0 mon.b 2024-06-03T08:36:33.091 INFO:teuthology.orchestra.run.smithi088.stderr:2: v1:172.21.15.88:6790/0 mon.c 2024-06-03T08:36:33.091 INFO:teuthology.orchestra.run.smithi088.stderr: 2024-06-03T08:36:33.092 INFO:teuthology.orchestra.run.smithi088.stderr:2024-06-03T08:36:33.089+0000 7f50daffd640 4 set_mon_vals no callback set 2024-06-03T08:36:33.092 INFO:teuthology.orchestra.run.smithi088.stderr:2024-06-03T08:36:33.090+0000 7f50d97fa640 1 -- 172.21.15.88:0/1176708298 <== mon.2 v1:172.21.15.88:6790/0 7 ==== mgrmap(e 5) v1 ==== 89110+0+0 (unknown 873533973 0 0) 0x7f50c8003910 con 0x7f50dc0b4250 2024-06-03T08:36:33.092 INFO:teuthology.orchestra.run.smithi088.stderr:2024-06-03T08:36:33.091+0000 7f50e0b19640 10 monclient: get_auth_request con 0x7f50c006d110 auth_method 0 2024-06-03T08:36:33.092 INFO:teuthology.orchestra.run.smithi088.stderr:2024-06-03T08:36:33.091+0000 7f50d97fa640 10 client.4361.objecter ms_handle_connect 0x7f50c006d110 2024-06-03T08:36:33.102 INFO:teuthology.orchestra.run.smithi088.stderr:2024-06-03T08:36:33.101+0000 7f50d97fa640 1 -- 172.21.15.88:0/1176708298 <== mon.2 v1:172.21.15.88:6790/0 8 ==== osd_map(26..26 src has 1..26) v4 ==== 4090+0+0 (unknown 3637450554 0 0) 0x7f50c8052dc0 con 0x7f50dc0b4250 2024-06-03T08:36:33.102 INFO:teuthology.orchestra.run.smithi088.stderr:2024-06-03T08:36:33.101+0000 7f50d97fa640 10 client.4361.objecter ms_dispatch 0x7f50dc000b90 osd_map(26..26 src has 1..26) v4 2024-06-03T08:36:33.102 INFO:teuthology.orchestra.run.smithi088.stderr:2024-06-03T08:36:33.101+0000 7f50d97fa640 3 client.4361.objecter handle_osd_map got epochs [26,26] > 0 2024-06-03T08:36:33.102 INFO:teuthology.orchestra.run.smithi088.stderr:2024-06-03T08:36:33.101+0000 7f50d97fa640 3 client.4361.objecter handle_osd_map decoding full epoch 26 2024-06-03T08:36:33.102 INFO:teuthology.orchestra.run.smithi088.stderr:2024-06-03T08:36:33.101+0000 7f50d97fa640 20 client.4361.objecter dump_active .. 0 homeless 2024-06-03T08:36:33.102 INFO:teuthology.orchestra.run.smithi088.stderr:2024-06-03T08:36:33.101+0000 7f50d97fa640 1 -- 172.21.15.88:0/1176708298 <== mon.2 v1:172.21.15.88:6790/0 9 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+182498 (unknown 1092875540 0 3016760700) 0x7f50c804e7a0 con 0x7f50dc0b4250 2024-06-03T08:36:33.102 INFO:teuthology.orchestra.run.smithi088.stderr:2024-06-03T08:36:33.101+0000 7f50d97fa640 10 monclient: handle_mon_command_ack 1 [{"prefix": "get_command_descriptions"}] 2024-06-03T08:36:33.102 INFO:teuthology.orchestra.run.smithi088.stderr:2024-06-03T08:36:33.101+0000 7f50d97fa640 10 monclient: _finish_command 1 = system:0 2024-06-03T08:36:33.153 INFO:teuthology.orchestra.run.smithi088.stderr:2024-06-03T08:36:33.153+0000 7f83e67fc640 10 start_mon_command cmd=[{"prefix": "osd last-stat-seq", "id": 1}] 2024-06-03T08:36:33.154 INFO:teuthology.orchestra.run.smithi088.stderr:2024-06-03T08:36:33.153+0000 7f83e67fc640 10 monclient: _send_command 2 [{"prefix": "osd last-stat-seq", "id": 1}] 2024-06-03T08:36:33.154 INFO:teuthology.orchestra.run.smithi088.stderr:2024-06-03T08:36:33.153+0000 7f83e67fc640 10 monclient: _send_mon_message to mon.b at v1:172.21.15.192:6789/0 2024-06-03T08:36:33.154 INFO:teuthology.orchestra.run.smithi088.stderr:2024-06-03T08:36:33.153+0000 7f83e67fc640 1 -- 172.21.15.88:0/4283667430 --> v1:172.21.15.192:6789/0 -- mon_command({"prefix": "osd last-stat-seq", "id": 1} v 0) v1 -- 0x7f83e00052f0 con 0x7f84000ae8c0 2024-06-03T08:36:33.195 INFO:teuthology.orchestra.run.smithi088.stderr:2024-06-03T08:36:33.195+0000 7f224f31d640 10 start_mon_command cmd=[{"prefix": "osd last-stat-seq", "id": 7}] 2024-06-03T08:36:33.195 INFO:teuthology.orchestra.run.smithi088.stderr:2024-06-03T08:36:33.195+0000 7f224f31d640 10 monclient: _send_command 2 [{"prefix": "osd last-stat-seq", "id": 7}] 2024-06-03T08:36:33.196 INFO:teuthology.orchestra.run.smithi088.stderr:2024-06-03T08:36:33.195+0000 7f224f31d640 10 monclient: _send_mon_message to mon.a at v1:172.21.15.88:6789/0 2024-06-03T08:36:33.196 INFO:teuthology.orchestra.run.smithi088.stderr:2024-06-03T08:36:33.195+0000 7f224f31d640 1 -- 172.21.15.88:0/598291210 --> v1:172.21.15.88:6789/0 -- mon_command({"prefix": "osd last-stat-seq", "id": 7} v 0) v1 -- 0x7f224804ea60 con 0x7f22480a89f0 2024-06-03T08:36:33.219 INFO:teuthology.orchestra.run.smithi088.stderr:2024-06-03T08:36:33.219+0000 7feb0d7fa640 1 -- 172.21.15.88:0/1366328441 <== mon.1 v1:172.21.15.192:6789/0 5 ==== config(0 keys) v1 ==== 4+0+0 (unknown 0 0 0) 0x7feb040047e0 con 0x7feb100a8960 2024-06-03T08:36:33.219 INFO:teuthology.orchestra.run.smithi088.stderr:2024-06-03T08:36:33.219+0000 7feb0d7fa640 10 monclient: handle_config config(0 keys) v1 2024-06-03T08:36:33.219 INFO:teuthology.orchestra.run.smithi088.stderr:2024-06-03T08:36:33.219+0000 7feb0effd640 4 set_mon_vals no callback set 2024-06-03T08:36:33.253 INFO:teuthology.orchestra.run.smithi088.stderr:2024-06-03T08:36:33.252+0000 7feb0d7fa640 1 -- 172.21.15.88:0/1366328441 <== mon.1 v1:172.21.15.192:6789/0 6 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 2234374065 0 0) 0x7feb04004b70 con 0x7feb100a8960 2024-06-03T08:36:33.253 INFO:teuthology.orchestra.run.smithi088.stderr:2024-06-03T08:36:33.252+0000 7feb0d7fa640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-06-03T08:36:33.253 INFO:teuthology.orchestra.run.smithi088.stderr:2024-06-03T08:36:33.252+0000 7feb0d7fa640 10 monclient: got monmap 1 from mon.b (according to old e1) 2024-06-03T08:36:33.253 INFO:teuthology.orchestra.run.smithi088.stderr:2024-06-03T08:36:33.252+0000 7feb0d7fa640 10 monclient: dump: 2024-06-03T08:36:33.253 INFO:teuthology.orchestra.run.smithi088.stderr:epoch 1 2024-06-03T08:36:33.253 INFO:teuthology.orchestra.run.smithi088.stderr:fsid ec3c3fdd-3ac6-4d7d-9412-683130ed8b04 2024-06-03T08:36:33.253 INFO:teuthology.orchestra.run.smithi088.stderr:last_changed 2024-06-03T08:34:31.885125+0000 2024-06-03T08:36:33.253 INFO:teuthology.orchestra.run.smithi088.stderr:created 2024-06-03T08:34:31.885125+0000 2024-06-03T08:36:33.254 INFO:teuthology.orchestra.run.smithi088.stderr:min_mon_release 18 (reef) 2024-06-03T08:36:33.254 INFO:teuthology.orchestra.run.smithi088.stderr:election_strategy: 3 2024-06-03T08:36:33.254 INFO:teuthology.orchestra.run.smithi088.stderr:0: v1:172.21.15.88:6789/0 mon.a 2024-06-03T08:36:33.254 INFO:teuthology.orchestra.run.smithi088.stderr:1: v1:172.21.15.192:6789/0 mon.b 2024-06-03T08:36:33.254 INFO:teuthology.orchestra.run.smithi088.stderr:2: v1:172.21.15.88:6790/0 mon.c 2024-06-03T08:36:33.254 INFO:teuthology.orchestra.run.smithi088.stderr: 2024-06-03T08:36:33.255 INFO:teuthology.orchestra.run.smithi088.stderr:2024-06-03T08:36:33.253+0000 7feb0d7fa640 1 -- 172.21.15.88:0/1366328441 <== mon.1 v1:172.21.15.192:6789/0 7 ==== mgrmap(e 5) v1 ==== 89110+0+0 (unknown 873533973 0 0) 0x7feb04004d50 con 0x7feb100a8960 2024-06-03T08:36:33.255 INFO:teuthology.orchestra.run.smithi088.stderr:2024-06-03T08:36:33.254+0000 7feb0ffff640 10 monclient: get_auth_request con 0x7feaf406d0b0 auth_method 0 2024-06-03T08:36:33.255 INFO:teuthology.orchestra.run.smithi088.stderr:2024-06-03T08:36:33.254+0000 7feb0d7fa640 10 client.4360.objecter ms_handle_connect 0x7feaf406d0b0 2024-06-03T08:36:33.258 INFO:teuthology.orchestra.run.smithi088.stderr:2024-06-03T08:36:33.257+0000 7feb0d7fa640 1 -- 172.21.15.88:0/1366328441 <== mon.1 v1:172.21.15.192:6789/0 8 ==== osd_map(26..26 src has 1..26) v4 ==== 4090+0+0 (unknown 3637450554 0 0) 0x7feb040541a0 con 0x7feb100a8960 2024-06-03T08:36:33.258 INFO:teuthology.orchestra.run.smithi088.stderr:2024-06-03T08:36:33.257+0000 7feb0d7fa640 10 client.4360.objecter ms_dispatch 0x7feb10000b90 osd_map(26..26 src has 1..26) v4 2024-06-03T08:36:33.258 INFO:teuthology.orchestra.run.smithi088.stderr:2024-06-03T08:36:33.257+0000 7feb0d7fa640 3 client.4360.objecter handle_osd_map got epochs [26,26] > 0 2024-06-03T08:36:33.258 INFO:teuthology.orchestra.run.smithi088.stderr:2024-06-03T08:36:33.257+0000 7feb0d7fa640 3 client.4360.objecter handle_osd_map decoding full epoch 26 2024-06-03T08:36:33.258 INFO:teuthology.orchestra.run.smithi088.stderr:2024-06-03T08:36:33.258+0000 7feb0d7fa640 20 client.4360.objecter dump_active .. 0 homeless 2024-06-03T08:36:33.260 INFO:teuthology.orchestra.run.smithi088.stderr:2024-06-03T08:36:33.260+0000 7feb0d7fa640 1 -- 172.21.15.88:0/1366328441 <== mon.1 v1:172.21.15.192:6789/0 9 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+182498 (unknown 1092875540 0 3016760700) 0x7feb0404fb80 con 0x7feb100a8960 2024-06-03T08:36:33.260 INFO:teuthology.orchestra.run.smithi088.stderr:2024-06-03T08:36:33.260+0000 7feb0d7fa640 10 monclient: handle_mon_command_ack 1 [{"prefix": "get_command_descriptions"}] 2024-06-03T08:36:33.260 INFO:teuthology.orchestra.run.smithi088.stderr:2024-06-03T08:36:33.260+0000 7feb0d7fa640 10 monclient: _finish_command 1 = system:0 2024-06-03T08:36:33.262 INFO:teuthology.orchestra.run.smithi088.stderr:2024-06-03T08:36:33.262+0000 7f83fcff9640 1 -- 172.21.15.88:0/4283667430 <== mon.1 v1:172.21.15.192:6789/0 10 ==== mon_command_ack([{"prefix": "osd last-stat-seq", "id": 1}]=0 v0) v1 ==== 74+0+12 (unknown 832126871 0 3426929836) 0x7f83f4051cb0 con 0x7f84000ae8c0 2024-06-03T08:36:33.262 INFO:teuthology.orchestra.run.smithi088.stderr:2024-06-03T08:36:33.262+0000 7f83fcff9640 10 monclient: handle_mon_command_ack 2 [{"prefix": "osd last-stat-seq", "id": 1}] 2024-06-03T08:36:33.262 INFO:teuthology.orchestra.run.smithi088.stderr:2024-06-03T08:36:33.262+0000 7f83fcff9640 10 monclient: _finish_command 2 = system:0 2024-06-03T08:36:33.262 INFO:teuthology.orchestra.run.smithi088.stdout:60129542151 2024-06-03T08:36:33.265 INFO:teuthology.orchestra.run.smithi088.stderr:2024-06-03T08:36:33.264+0000 7f83e67fc640 10 librados: watch_flush enter 2024-06-03T08:36:33.265 INFO:teuthology.orchestra.run.smithi088.stderr:2024-06-03T08:36:33.264+0000 7f83e67fc640 10 librados: watch_flush exit 2024-06-03T08:36:33.265 INFO:teuthology.orchestra.run.smithi088.stderr:2024-06-03T08:36:33.264+0000 7f83e67fc640 20 client.4354.objecter shutdown clearing up homeless session... 2024-06-03T08:36:33.265 INFO:teuthology.orchestra.run.smithi088.stderr:2024-06-03T08:36:33.264+0000 7f83e67fc640 10 client.4354.objecter successfully canceled tick 2024-06-03T08:36:33.266 INFO:teuthology.orchestra.run.smithi088.stderr:2024-06-03T08:36:33.264+0000 7f83e67fc640 1 -- 172.21.15.88:0/4283667430 >> v1:172.21.15.192:6800/34229 conn(0x7f83ec06d080 legacy=0x7f83ec06f540 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-06-03T08:36:33.266 INFO:teuthology.orchestra.run.smithi088.stderr:2024-06-03T08:36:33.265+0000 7f83e67fc640 10 monclient: shutdown 2024-06-03T08:36:33.266 INFO:teuthology.orchestra.run.smithi088.stderr:2024-06-03T08:36:33.265+0000 7f83e67fc640 20 monclient: shutdown discarding 0 pending message(s) 2024-06-03T08:36:33.266 INFO:teuthology.orchestra.run.smithi088.stderr:2024-06-03T08:36:33.265+0000 7f83e67fc640 1 -- 172.21.15.88:0/4283667430 >> v1:172.21.15.192:6789/0 conn(0x7f84000ae8c0 legacy=0x7f84001d9b00 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-06-03T08:36:33.266 INFO:teuthology.orchestra.run.smithi088.stderr:2024-06-03T08:36:33.265+0000 7f83e67fc640 1 -- 172.21.15.88:0/4283667430 shutdown_connections 2024-06-03T08:36:33.266 INFO:teuthology.orchestra.run.smithi088.stderr:2024-06-03T08:36:33.265+0000 7f83e67fc640 1 -- 172.21.15.88:0/4283667430 >> 172.21.15.88:0/4283667430 conn(0x7f84000b2f20 msgr2=0x7f84000b1730 unknown :-1 s=STATE_NONE l=0).mark_down 2024-06-03T08:36:33.266 INFO:teuthology.orchestra.run.smithi088.stderr:2024-06-03T08:36:33.265+0000 7f83e67fc640 1 -- 172.21.15.88:0/4283667430 shutdown_connections 2024-06-03T08:36:33.266 INFO:teuthology.orchestra.run.smithi088.stderr:2024-06-03T08:36:33.265+0000 7f83e67fc640 1 -- 172.21.15.88:0/4283667430 wait complete. 2024-06-03T08:36:33.266 INFO:teuthology.orchestra.run.smithi088.stderr:2024-06-03T08:36:33.265+0000 7f83e67fc640 1 librados: shutdown 2024-06-03T08:36:33.272 INFO:teuthology.orchestra.run.smithi088.stderr:2024-06-03T08:36:33.272+0000 7f169bfff640 1 -- 172.21.15.88:0/2965169668 <== mon.0 v1:172.21.15.88:6789/0 5 ==== config(0 keys) v1 ==== 4+0+0 (unknown 0 0 0) 0x7f16ac0020d0 con 0x7f16b41451c0 2024-06-03T08:36:33.273 INFO:teuthology.orchestra.run.smithi088.stderr:2024-06-03T08:36:33.272+0000 7f169bfff640 10 monclient: handle_config config(0 keys) v1 2024-06-03T08:36:33.273 INFO:teuthology.orchestra.run.smithi088.stderr:2024-06-03T08:36:33.272+0000 7f169bfff640 1 -- 172.21.15.88:0/2965169668 <== mon.0 v1:172.21.15.88:6789/0 6 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 2234374065 0 0) 0x7f16ac003b10 con 0x7f16b41451c0 2024-06-03T08:36:33.273 INFO:teuthology.orchestra.run.smithi088.stderr:2024-06-03T08:36:33.272+0000 7f169bfff640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-06-03T08:36:33.273 INFO:teuthology.orchestra.run.smithi088.stderr:2024-06-03T08:36:33.272+0000 7f169bfff640 10 monclient: got monmap 1 from mon.a (according to old e1) 2024-06-03T08:36:33.273 INFO:teuthology.orchestra.run.smithi088.stderr:2024-06-03T08:36:33.272+0000 7f169bfff640 10 monclient: dump: 2024-06-03T08:36:33.273 INFO:teuthology.orchestra.run.smithi088.stderr:epoch 1 2024-06-03T08:36:33.273 INFO:teuthology.orchestra.run.smithi088.stderr:fsid ec3c3fdd-3ac6-4d7d-9412-683130ed8b04 2024-06-03T08:36:33.273 INFO:teuthology.orchestra.run.smithi088.stderr:last_changed 2024-06-03T08:34:31.885125+0000 2024-06-03T08:36:33.273 INFO:teuthology.orchestra.run.smithi088.stderr:created 2024-06-03T08:34:31.885125+0000 2024-06-03T08:36:33.273 INFO:teuthology.orchestra.run.smithi088.stderr:min_mon_release 18 (reef) 2024-06-03T08:36:33.273 INFO:teuthology.orchestra.run.smithi088.stderr:election_strategy: 3 2024-06-03T08:36:33.273 INFO:teuthology.orchestra.run.smithi088.stderr:0: v1:172.21.15.88:6789/0 mon.a 2024-06-03T08:36:33.274 INFO:teuthology.orchestra.run.smithi088.stderr:1: v1:172.21.15.192:6789/0 mon.b 2024-06-03T08:36:33.274 INFO:teuthology.orchestra.run.smithi088.stderr:2: v1:172.21.15.88:6790/0 mon.c 2024-06-03T08:36:33.274 INFO:teuthology.orchestra.run.smithi088.stderr: 2024-06-03T08:36:33.274 INFO:teuthology.orchestra.run.smithi088.stderr:2024-06-03T08:36:33.272+0000 7f16b1ffb640 4 set_mon_vals no callback set 2024-06-03T08:36:33.274 INFO:teuthology.orchestra.run.smithi088.stderr:2024-06-03T08:36:33.273+0000 7f169bfff640 1 -- 172.21.15.88:0/2965169668 <== mon.0 v1:172.21.15.88:6789/0 7 ==== mgrmap(e 5) v1 ==== 89110+0+0 (unknown 873533973 0 0) 0x7f16ac019890 con 0x7f16b41451c0 2024-06-03T08:36:33.274 INFO:teuthology.orchestra.run.smithi088.stderr:2024-06-03T08:36:33.274+0000 7f16b37fe640 10 monclient: get_auth_request con 0x7f16a006d230 auth_method 0 2024-06-03T08:36:33.275 INFO:teuthology.orchestra.run.smithi088.stderr:2024-06-03T08:36:33.274+0000 7f169bfff640 10 client.4356.objecter ms_handle_connect 0x7f16a006d230 2024-06-03T08:36:33.280 INFO:tasks.ceph.ceph_manager.ceph:need seq 60129542152 got 60129542151 for osd.1 2024-06-03T08:36:33.284 INFO:teuthology.orchestra.run.smithi088.stderr:2024-06-03T08:36:33.283+0000 7f169bfff640 1 -- 172.21.15.88:0/2965169668 <== mon.0 v1:172.21.15.88:6789/0 8 ==== osd_map(26..26 src has 1..26) v4 ==== 4090+0+0 (unknown 3637450554 0 0) 0x7f16ac053180 con 0x7f16b41451c0 2024-06-03T08:36:33.284 INFO:teuthology.orchestra.run.smithi088.stderr:2024-06-03T08:36:33.283+0000 7f169bfff640 10 client.4356.objecter ms_dispatch 0x7f16b4000b90 osd_map(26..26 src has 1..26) v4 2024-06-03T08:36:33.284 INFO:teuthology.orchestra.run.smithi088.stderr:2024-06-03T08:36:33.283+0000 7f169bfff640 3 client.4356.objecter handle_osd_map got epochs [26,26] > 0 2024-06-03T08:36:33.284 INFO:teuthology.orchestra.run.smithi088.stderr:2024-06-03T08:36:33.283+0000 7f169bfff640 3 client.4356.objecter handle_osd_map decoding full epoch 26 2024-06-03T08:36:33.284 INFO:teuthology.orchestra.run.smithi088.stderr:2024-06-03T08:36:33.283+0000 7f169bfff640 20 client.4356.objecter dump_active .. 0 homeless 2024-06-03T08:36:33.284 INFO:teuthology.orchestra.run.smithi088.stderr:2024-06-03T08:36:33.283+0000 7f169bfff640 1 -- 172.21.15.88:0/2965169668 <== mon.0 v1:172.21.15.88:6789/0 9 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+182498 (unknown 1092875540 0 3016760700) 0x7f16ac04eb60 con 0x7f16b41451c0 2024-06-03T08:36:33.284 INFO:teuthology.orchestra.run.smithi088.stderr:2024-06-03T08:36:33.283+0000 7f169bfff640 10 monclient: handle_mon_command_ack 1 [{"prefix": "get_command_descriptions"}] 2024-06-03T08:36:33.284 INFO:teuthology.orchestra.run.smithi088.stderr:2024-06-03T08:36:33.283+0000 7f169bfff640 10 monclient: _finish_command 1 = system:0 2024-06-03T08:36:33.285 INFO:teuthology.orchestra.run.smithi088.stderr:2024-06-03T08:36:33.284+0000 7f50e2d7d640 10 start_mon_command cmd=[{"prefix": "osd last-stat-seq", "id": 0}] 2024-06-03T08:36:33.285 INFO:teuthology.orchestra.run.smithi088.stderr:2024-06-03T08:36:33.284+0000 7f50e2d7d640 10 monclient: _send_command 2 [{"prefix": "osd last-stat-seq", "id": 0}] 2024-06-03T08:36:33.285 INFO:teuthology.orchestra.run.smithi088.stderr:2024-06-03T08:36:33.284+0000 7f50e2d7d640 10 monclient: _send_mon_message to mon.c at v1:172.21.15.88:6790/0 2024-06-03T08:36:33.285 INFO:teuthology.orchestra.run.smithi088.stderr:2024-06-03T08:36:33.284+0000 7f50e2d7d640 1 -- 172.21.15.88:0/1176708298 --> v1:172.21.15.88:6790/0 -- mon_command({"prefix": "osd last-stat-seq", "id": 0} v 0) v1 -- 0x7f50dc0671b0 con 0x7f50dc0b4250 2024-06-03T08:36:33.300 INFO:teuthology.orchestra.run.smithi088.stderr:2024-06-03T08:36:33.300+0000 7f223dffb640 1 -- 172.21.15.88:0/598291210 <== mon.0 v1:172.21.15.88:6789/0 10 ==== mon_command_ack([{"prefix": "osd last-stat-seq", "id": 7}]=0 v0) v1 ==== 74+0+12 (unknown 1482059429 0 1053920503) 0x7f2238051970 con 0x7f22480a89f0 2024-06-03T08:36:33.300 INFO:teuthology.orchestra.run.smithi088.stderr:2024-06-03T08:36:33.300+0000 7f223dffb640 10 monclient: handle_mon_command_ack 2 [{"prefix": "osd last-stat-seq", "id": 7}] 2024-06-03T08:36:33.300 INFO:teuthology.orchestra.run.smithi088.stderr:2024-06-03T08:36:33.300+0000 7f223dffb640 10 monclient: _finish_command 2 = system:0 2024-06-03T08:36:33.300 INFO:teuthology.orchestra.run.smithi088.stdout:85899345927 2024-06-03T08:36:33.303 INFO:teuthology.orchestra.run.smithi088.stderr:2024-06-03T08:36:33.302+0000 7f224f31d640 10 librados: watch_flush enter 2024-06-03T08:36:33.303 INFO:teuthology.orchestra.run.smithi088.stderr:2024-06-03T08:36:33.302+0000 7f224f31d640 10 librados: watch_flush exit 2024-06-03T08:36:33.303 INFO:teuthology.orchestra.run.smithi088.stderr:2024-06-03T08:36:33.302+0000 7f224f31d640 20 client.4347.objecter shutdown clearing up homeless session... 2024-06-03T08:36:33.303 INFO:teuthology.orchestra.run.smithi088.stderr:2024-06-03T08:36:33.302+0000 7f224f31d640 10 client.4347.objecter successfully canceled tick 2024-06-03T08:36:33.303 INFO:teuthology.orchestra.run.smithi088.stderr:2024-06-03T08:36:33.302+0000 7f224f31d640 1 -- 172.21.15.88:0/598291210 >> v1:172.21.15.192:6800/34229 conn(0x7f222c06d200 legacy=0x7f222c06f6c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-06-03T08:36:33.303 INFO:teuthology.orchestra.run.smithi088.stderr:2024-06-03T08:36:33.302+0000 7f224f31d640 10 monclient: shutdown 2024-06-03T08:36:33.304 INFO:teuthology.orchestra.run.smithi088.stderr:2024-06-03T08:36:33.302+0000 7f224f31d640 20 monclient: shutdown discarding 0 pending message(s) 2024-06-03T08:36:33.304 INFO:teuthology.orchestra.run.smithi088.stderr:2024-06-03T08:36:33.302+0000 7f224f31d640 1 -- 172.21.15.88:0/598291210 >> v1:172.21.15.88:6789/0 conn(0x7f22480a89f0 legacy=0x7f22481d6730 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-06-03T08:36:33.304 INFO:teuthology.orchestra.run.smithi088.stderr:2024-06-03T08:36:33.302+0000 7f224f31d640 1 -- 172.21.15.88:0/598291210 shutdown_connections 2024-06-03T08:36:33.304 INFO:teuthology.orchestra.run.smithi088.stderr:2024-06-03T08:36:33.302+0000 7f224f31d640 1 -- 172.21.15.88:0/598291210 >> 172.21.15.88:0/598291210 conn(0x7f22480717f0 msgr2=0x7f22480b63b0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-06-03T08:36:33.304 INFO:teuthology.orchestra.run.smithi088.stderr:2024-06-03T08:36:33.302+0000 7f224f31d640 1 -- 172.21.15.88:0/598291210 shutdown_connections 2024-06-03T08:36:33.304 INFO:teuthology.orchestra.run.smithi088.stderr:2024-06-03T08:36:33.302+0000 7f224f31d640 1 -- 172.21.15.88:0/598291210 wait complete. 2024-06-03T08:36:33.304 INFO:teuthology.orchestra.run.smithi088.stderr:2024-06-03T08:36:33.302+0000 7f224f31d640 1 librados: shutdown 2024-06-03T08:36:33.317 INFO:tasks.ceph.ceph_manager.ceph:need seq 85899345928 got 85899345927 for osd.7 2024-06-03T08:36:33.394 INFO:teuthology.orchestra.run.smithi088.stderr:2024-06-03T08:36:33.394+0000 7fccb67fc640 1 -- 172.21.15.88:0/1456172716 <== mon.2 v1:172.21.15.88:6790/0 10 ==== mon_command_ack([{"prefix": "osd last-stat-seq", "id": 3}]=0 v0) v1 ==== 74+0+12 (unknown 383520633 0 3426929836) 0x7fccbc0519b0 con 0x7fccc00a8a60 2024-06-03T08:36:33.394 INFO:teuthology.orchestra.run.smithi088.stderr:2024-06-03T08:36:33.394+0000 7fccb67fc640 10 monclient: handle_mon_command_ack 2 [{"prefix": "osd last-stat-seq", "id": 3}] 2024-06-03T08:36:33.394 INFO:teuthology.orchestra.run.smithi088.stderr:2024-06-03T08:36:33.394+0000 7fccb67fc640 10 monclient: _finish_command 2 = system:0 2024-06-03T08:36:33.395 INFO:teuthology.orchestra.run.smithi088.stdout:60129542151 2024-06-03T08:36:33.398 INFO:teuthology.orchestra.run.smithi088.stderr:2024-06-03T08:36:33.397+0000 7fccc7ad1640 10 librados: watch_flush enter 2024-06-03T08:36:33.398 INFO:teuthology.orchestra.run.smithi088.stderr:2024-06-03T08:36:33.397+0000 7fccc7ad1640 10 librados: watch_flush exit 2024-06-03T08:36:33.398 INFO:teuthology.orchestra.run.smithi088.stderr:2024-06-03T08:36:33.397+0000 7fccc7ad1640 20 client.4352.objecter shutdown clearing up homeless session... 2024-06-03T08:36:33.398 INFO:teuthology.orchestra.run.smithi088.stderr:2024-06-03T08:36:33.397+0000 7fccc7ad1640 10 client.4352.objecter successfully canceled tick 2024-06-03T08:36:33.398 INFO:teuthology.orchestra.run.smithi088.stderr:2024-06-03T08:36:33.397+0000 7fccc7ad1640 1 -- 172.21.15.88:0/1456172716 >> v1:172.21.15.192:6800/34229 conn(0x7fcca406d290 legacy=0x7fcca406f750 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-06-03T08:36:33.398 INFO:teuthology.orchestra.run.smithi088.stderr:2024-06-03T08:36:33.397+0000 7fccc7ad1640 10 monclient: shutdown 2024-06-03T08:36:33.398 INFO:teuthology.orchestra.run.smithi088.stderr:2024-06-03T08:36:33.397+0000 7fccc7ad1640 20 monclient: shutdown discarding 0 pending message(s) 2024-06-03T08:36:33.398 INFO:teuthology.orchestra.run.smithi088.stderr:2024-06-03T08:36:33.397+0000 7fccc7ad1640 1 -- 172.21.15.88:0/1456172716 >> v1:172.21.15.88:6790/0 conn(0x7fccc00a8a60 legacy=0x7fccc01340d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-06-03T08:36:33.398 INFO:teuthology.orchestra.run.smithi088.stderr:2024-06-03T08:36:33.397+0000 7fccc7ad1640 1 -- 172.21.15.88:0/1456172716 shutdown_connections 2024-06-03T08:36:33.398 INFO:teuthology.orchestra.run.smithi088.stderr:2024-06-03T08:36:33.397+0000 7fccc7ad1640 1 -- 172.21.15.88:0/1456172716 >> 172.21.15.88:0/1456172716 conn(0x7fccc00b70d0 msgr2=0x7fccc00b4c90 unknown :-1 s=STATE_NONE l=0).mark_down 2024-06-03T08:36:33.399 INFO:teuthology.orchestra.run.smithi088.stderr:2024-06-03T08:36:33.397+0000 7fccc7ad1640 1 -- 172.21.15.88:0/1456172716 shutdown_connections 2024-06-03T08:36:33.399 INFO:teuthology.orchestra.run.smithi088.stderr:2024-06-03T08:36:33.397+0000 7fccc7ad1640 1 -- 172.21.15.88:0/1456172716 wait complete. 2024-06-03T08:36:33.399 INFO:teuthology.orchestra.run.smithi088.stderr:2024-06-03T08:36:33.397+0000 7fccc7ad1640 1 librados: shutdown 2024-06-03T08:36:33.403 INFO:teuthology.orchestra.run.smithi088.stderr:2024-06-03T08:36:33.403+0000 7f3d3e7fc640 1 -- 172.21.15.88:0/1768052638 <== mon.2 v1:172.21.15.88:6790/0 5 ==== config(0 keys) v1 ==== 4+0+0 (unknown 0 0 0) 0x7f3d38001cf0 con 0x7f3d480b4250 2024-06-03T08:36:33.403 INFO:teuthology.orchestra.run.smithi088.stderr:2024-06-03T08:36:33.403+0000 7f3d3e7fc640 10 monclient: handle_config config(0 keys) v1 2024-06-03T08:36:33.403 INFO:teuthology.orchestra.run.smithi088.stderr:2024-06-03T08:36:33.403+0000 7f3d3e7fc640 1 -- 172.21.15.88:0/1768052638 <== mon.2 v1:172.21.15.88:6790/0 6 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 2234374065 0 0) 0x7f3d38004980 con 0x7f3d480b4250 2024-06-03T08:36:33.403 INFO:teuthology.orchestra.run.smithi088.stderr:2024-06-03T08:36:33.403+0000 7f3d3e7fc640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-06-03T08:36:33.403 INFO:teuthology.orchestra.run.smithi088.stderr:2024-06-03T08:36:33.403+0000 7f3d3e7fc640 10 monclient: got monmap 1 from mon.c (according to old e1) 2024-06-03T08:36:33.403 INFO:teuthology.orchestra.run.smithi088.stderr:2024-06-03T08:36:33.403+0000 7f3d3e7fc640 10 monclient: dump: 2024-06-03T08:36:33.403 INFO:teuthology.orchestra.run.smithi088.stderr:epoch 1 2024-06-03T08:36:33.404 INFO:teuthology.orchestra.run.smithi088.stderr:fsid ec3c3fdd-3ac6-4d7d-9412-683130ed8b04 2024-06-03T08:36:33.404 INFO:teuthology.orchestra.run.smithi088.stderr:last_changed 2024-06-03T08:34:31.885125+0000 2024-06-03T08:36:33.404 INFO:teuthology.orchestra.run.smithi088.stderr:created 2024-06-03T08:34:31.885125+0000 2024-06-03T08:36:33.404 INFO:teuthology.orchestra.run.smithi088.stderr:min_mon_release 18 (reef) 2024-06-03T08:36:33.404 INFO:teuthology.orchestra.run.smithi088.stderr:election_strategy: 3 2024-06-03T08:36:33.404 INFO:teuthology.orchestra.run.smithi088.stderr:0: v1:172.21.15.88:6789/0 mon.a 2024-06-03T08:36:33.404 INFO:teuthology.orchestra.run.smithi088.stderr:1: v1:172.21.15.192:6789/0 mon.b 2024-06-03T08:36:33.404 INFO:teuthology.orchestra.run.smithi088.stderr:2: v1:172.21.15.88:6790/0 mon.c 2024-06-03T08:36:33.404 INFO:teuthology.orchestra.run.smithi088.stderr: 2024-06-03T08:36:33.404 INFO:teuthology.orchestra.run.smithi088.stderr:2024-06-03T08:36:33.403+0000 7f3d3ffff640 4 set_mon_vals no callback set 2024-06-03T08:36:33.405 INFO:teuthology.orchestra.run.smithi088.stderr:2024-06-03T08:36:33.404+0000 7f3d3e7fc640 1 -- 172.21.15.88:0/1768052638 <== mon.2 v1:172.21.15.88:6790/0 7 ==== mgrmap(e 5) v1 ==== 89110+0+0 (unknown 873533973 0 0) 0x7f3d38004ae0 con 0x7f3d480b4250 2024-06-03T08:36:33.405 INFO:teuthology.orchestra.run.smithi088.stderr:2024-06-03T08:36:33.404+0000 7f3d4def8640 10 monclient: get_auth_request con 0x7f3d2c06d080 auth_method 0 2024-06-03T08:36:33.405 INFO:teuthology.orchestra.run.smithi088.stderr:2024-06-03T08:36:33.405+0000 7f3d3e7fc640 10 client.4364.objecter ms_handle_connect 0x7f3d2c06d080 2024-06-03T08:36:33.409 INFO:teuthology.orchestra.run.smithi088.stderr:2024-06-03T08:36:33.409+0000 7f3d3e7fc640 1 -- 172.21.15.88:0/1768052638 <== mon.2 v1:172.21.15.88:6790/0 8 ==== osd_map(26..26 src has 1..26) v4 ==== 4090+0+0 (unknown 3637450554 0 0) 0x7f3d38053ff0 con 0x7f3d480b4250 2024-06-03T08:36:33.409 INFO:teuthology.orchestra.run.smithi088.stderr:2024-06-03T08:36:33.409+0000 7f3d3e7fc640 10 client.4364.objecter ms_dispatch 0x7f3d48000b90 osd_map(26..26 src has 1..26) v4 2024-06-03T08:36:33.410 INFO:teuthology.orchestra.run.smithi088.stderr:2024-06-03T08:36:33.409+0000 7f3d3e7fc640 3 client.4364.objecter handle_osd_map got epochs [26,26] > 0 2024-06-03T08:36:33.410 INFO:teuthology.orchestra.run.smithi088.stderr:2024-06-03T08:36:33.409+0000 7f3d3e7fc640 3 client.4364.objecter handle_osd_map decoding full epoch 26 2024-06-03T08:36:33.410 INFO:teuthology.orchestra.run.smithi088.stderr:2024-06-03T08:36:33.409+0000 7f3d3e7fc640 20 client.4364.objecter dump_active .. 0 homeless 2024-06-03T08:36:33.410 INFO:teuthology.orchestra.run.smithi088.stderr:2024-06-03T08:36:33.409+0000 7f3d3e7fc640 1 -- 172.21.15.88:0/1768052638 <== mon.2 v1:172.21.15.88:6790/0 9 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+182498 (unknown 1092875540 0 3016760700) 0x7f3d3804f9d0 con 0x7f3d480b4250 2024-06-03T08:36:33.410 INFO:teuthology.orchestra.run.smithi088.stderr:2024-06-03T08:36:33.409+0000 7f3d3e7fc640 10 monclient: handle_mon_command_ack 1 [{"prefix": "get_command_descriptions"}] 2024-06-03T08:36:33.410 INFO:teuthology.orchestra.run.smithi088.stderr:2024-06-03T08:36:33.409+0000 7f3d3e7fc640 10 monclient: _finish_command 1 = system:0 2024-06-03T08:36:33.412 INFO:tasks.ceph.ceph_manager.ceph:need seq 60129542152 got 60129542151 for osd.3 2024-06-03T08:36:33.427 INFO:teuthology.orchestra.run.smithi088.stderr:2024-06-03T08:36:33.426+0000 7f50d97fa640 1 -- 172.21.15.88:0/1176708298 <== mon.2 v1:172.21.15.88:6790/0 10 ==== mon_command_ack([{"prefix": "osd last-stat-seq", "id": 0}]=0 v0) v1 ==== 74+0+12 (unknown 574334944 0 4095188330) 0x7f50c80508a0 con 0x7f50dc0b4250 2024-06-03T08:36:33.427 INFO:teuthology.orchestra.run.smithi088.stderr:2024-06-03T08:36:33.426+0000 7f50d97fa640 10 monclient: handle_mon_command_ack 2 [{"prefix": "osd last-stat-seq", "id": 0}] 2024-06-03T08:36:33.427 INFO:teuthology.orchestra.run.smithi088.stderr:2024-06-03T08:36:33.426+0000 7f50d97fa640 10 monclient: _finish_command 2 = system:0 2024-06-03T08:36:33.427 INFO:teuthology.orchestra.run.smithi088.stdout:60129542149 2024-06-03T08:36:33.429 INFO:teuthology.orchestra.run.smithi088.stderr:2024-06-03T08:36:33.428+0000 7f50e2d7d640 10 librados: watch_flush enter 2024-06-03T08:36:33.429 INFO:teuthology.orchestra.run.smithi088.stderr:2024-06-03T08:36:33.428+0000 7f50e2d7d640 10 librados: watch_flush exit 2024-06-03T08:36:33.429 INFO:teuthology.orchestra.run.smithi088.stderr:2024-06-03T08:36:33.428+0000 7f50e2d7d640 20 client.4361.objecter shutdown clearing up homeless session... 2024-06-03T08:36:33.429 INFO:teuthology.orchestra.run.smithi088.stderr:2024-06-03T08:36:33.428+0000 7f50e2d7d640 10 client.4361.objecter successfully canceled tick 2024-06-03T08:36:33.429 INFO:teuthology.orchestra.run.smithi088.stderr:2024-06-03T08:36:33.428+0000 7f50e2d7d640 1 -- 172.21.15.88:0/1176708298 >> v1:172.21.15.192:6800/34229 conn(0x7f50c006d110 legacy=0x7f50c006f5d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-06-03T08:36:33.430 INFO:teuthology.orchestra.run.smithi088.stderr:2024-06-03T08:36:33.428+0000 7f50e2d7d640 10 monclient: shutdown 2024-06-03T08:36:33.430 INFO:teuthology.orchestra.run.smithi088.stderr:2024-06-03T08:36:33.428+0000 7f50e2d7d640 20 monclient: shutdown discarding 0 pending message(s) 2024-06-03T08:36:33.430 INFO:teuthology.orchestra.run.smithi088.stderr:2024-06-03T08:36:33.428+0000 7f50e2d7d640 1 -- 172.21.15.88:0/1176708298 >> v1:172.21.15.88:6790/0 conn(0x7f50dc0b4250 legacy=0x7f50dc1d9e00 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-06-03T08:36:33.430 INFO:teuthology.orchestra.run.smithi088.stderr:2024-06-03T08:36:33.428+0000 7f50e2d7d640 1 -- 172.21.15.88:0/1176708298 shutdown_connections 2024-06-03T08:36:33.430 INFO:teuthology.orchestra.run.smithi088.stderr:2024-06-03T08:36:33.428+0000 7f50e2d7d640 1 -- 172.21.15.88:0/1176708298 >> 172.21.15.88:0/1176708298 conn(0x7f50dc0717f0 msgr2=0x7f50dc0b4d60 unknown :-1 s=STATE_NONE l=0).mark_down 2024-06-03T08:36:33.430 INFO:teuthology.orchestra.run.smithi088.stderr:2024-06-03T08:36:33.429+0000 7f50e2d7d640 1 -- 172.21.15.88:0/1176708298 shutdown_connections 2024-06-03T08:36:33.431 INFO:teuthology.orchestra.run.smithi088.stderr:2024-06-03T08:36:33.429+0000 7f50e2d7d640 1 -- 172.21.15.88:0/1176708298 wait complete. 2024-06-03T08:36:33.431 INFO:teuthology.orchestra.run.smithi088.stderr:2024-06-03T08:36:33.429+0000 7f50e2d7d640 1 librados: shutdown 2024-06-03T08:36:33.441 INFO:tasks.ceph.ceph_manager.ceph:need seq 60129542150 got 60129542149 for osd.0 2024-06-03T08:36:33.461 INFO:teuthology.orchestra.run.smithi088.stderr:2024-06-03T08:36:33.460+0000 7feb16ff6640 10 start_mon_command cmd=[{"prefix": "osd last-stat-seq", "id": 4}] 2024-06-03T08:36:33.461 INFO:teuthology.orchestra.run.smithi088.stderr:2024-06-03T08:36:33.461+0000 7feb16ff6640 10 monclient: _send_command 2 [{"prefix": "osd last-stat-seq", "id": 4}] 2024-06-03T08:36:33.461 INFO:teuthology.orchestra.run.smithi088.stderr:2024-06-03T08:36:33.461+0000 7feb16ff6640 10 monclient: _send_mon_message to mon.b at v1:172.21.15.192:6789/0 2024-06-03T08:36:33.461 INFO:teuthology.orchestra.run.smithi088.stderr:2024-06-03T08:36:33.461+0000 7feb16ff6640 1 -- 172.21.15.88:0/1366328441 --> v1:172.21.15.192:6789/0 -- mon_command({"prefix": "osd last-stat-seq", "id": 4} v 0) v1 -- 0x7feb100af4d0 con 0x7feb100a8960 2024-06-03T08:36:33.465 INFO:teuthology.orchestra.run.smithi088.stderr:2024-06-03T08:36:33.464+0000 7feb0d7fa640 1 -- 172.21.15.88:0/1366328441 <== mon.1 v1:172.21.15.192:6789/0 10 ==== mon_command_ack([{"prefix": "osd last-stat-seq", "id": 4}]=0 v0) v1 ==== 74+0+12 (unknown 1823589948 0 1053920503) 0x7feb04051c80 con 0x7feb100a8960 2024-06-03T08:36:33.465 INFO:teuthology.orchestra.run.smithi088.stderr:2024-06-03T08:36:33.464+0000 7feb0d7fa640 10 monclient: handle_mon_command_ack 2 [{"prefix": "osd last-stat-seq", "id": 4}] 2024-06-03T08:36:33.465 INFO:teuthology.orchestra.run.smithi088.stderr:2024-06-03T08:36:33.464+0000 7feb0d7fa640 10 monclient: _finish_command 2 = system:0 2024-06-03T08:36:33.465 INFO:teuthology.orchestra.run.smithi088.stdout:85899345927 2024-06-03T08:36:33.467 INFO:teuthology.orchestra.run.smithi088.stderr:2024-06-03T08:36:33.466+0000 7feb16ff6640 10 librados: watch_flush enter 2024-06-03T08:36:33.467 INFO:teuthology.orchestra.run.smithi088.stderr:2024-06-03T08:36:33.466+0000 7feb16ff6640 10 librados: watch_flush exit 2024-06-03T08:36:33.467 INFO:teuthology.orchestra.run.smithi088.stderr:2024-06-03T08:36:33.466+0000 7feb16ff6640 20 client.4360.objecter shutdown clearing up homeless session... 2024-06-03T08:36:33.467 INFO:teuthology.orchestra.run.smithi088.stderr:2024-06-03T08:36:33.466+0000 7feb16ff6640 10 client.4360.objecter successfully canceled tick 2024-06-03T08:36:33.467 INFO:teuthology.orchestra.run.smithi088.stderr:2024-06-03T08:36:33.466+0000 7feb16ff6640 1 -- 172.21.15.88:0/1366328441 >> v1:172.21.15.192:6800/34229 conn(0x7feaf406d0b0 legacy=0x7feaf406f570 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-06-03T08:36:33.468 INFO:teuthology.orchestra.run.smithi088.stderr:2024-06-03T08:36:33.466+0000 7feb16ff6640 10 monclient: shutdown 2024-06-03T08:36:33.468 INFO:teuthology.orchestra.run.smithi088.stderr:2024-06-03T08:36:33.466+0000 7feb16ff6640 20 monclient: shutdown discarding 0 pending message(s) 2024-06-03T08:36:33.468 INFO:teuthology.orchestra.run.smithi088.stderr:2024-06-03T08:36:33.466+0000 7feb16ff6640 1 -- 172.21.15.88:0/1366328441 >> v1:172.21.15.192:6789/0 conn(0x7feb100a8960 legacy=0x7feb100aab10 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-06-03T08:36:33.468 INFO:teuthology.orchestra.run.smithi088.stderr:2024-06-03T08:36:33.466+0000 7feb16ff6640 1 -- 172.21.15.88:0/1366328441 shutdown_connections 2024-06-03T08:36:33.468 INFO:teuthology.orchestra.run.smithi088.stderr:2024-06-03T08:36:33.466+0000 7feb16ff6640 1 -- 172.21.15.88:0/1366328441 >> 172.21.15.88:0/1366328441 conn(0x7feb100b70d0 msgr2=0x7feb100a8d40 unknown :-1 s=STATE_NONE l=0).mark_down 2024-06-03T08:36:33.468 INFO:teuthology.orchestra.run.smithi088.stderr:2024-06-03T08:36:33.466+0000 7feb16ff6640 1 -- 172.21.15.88:0/1366328441 shutdown_connections 2024-06-03T08:36:33.468 INFO:teuthology.orchestra.run.smithi088.stderr:2024-06-03T08:36:33.467+0000 7feb16ff6640 1 -- 172.21.15.88:0/1366328441 wait complete. 2024-06-03T08:36:33.468 INFO:teuthology.orchestra.run.smithi088.stderr:2024-06-03T08:36:33.467+0000 7feb16ff6640 1 librados: shutdown 2024-06-03T08:36:33.479 INFO:tasks.ceph.ceph_manager.ceph:need seq 85899345928 got 85899345927 for osd.4 2024-06-03T08:36:33.517 INFO:teuthology.orchestra.run.smithi088.stderr:2024-06-03T08:36:33.517+0000 7f16b9c8f640 10 start_mon_command cmd=[{"prefix": "osd last-stat-seq", "id": 5}] 2024-06-03T08:36:33.517 INFO:teuthology.orchestra.run.smithi088.stderr:2024-06-03T08:36:33.517+0000 7f16b9c8f640 10 monclient: _send_command 2 [{"prefix": "osd last-stat-seq", "id": 5}] 2024-06-03T08:36:33.517 INFO:teuthology.orchestra.run.smithi088.stderr:2024-06-03T08:36:33.517+0000 7f16b9c8f640 10 monclient: _send_mon_message to mon.a at v1:172.21.15.88:6789/0 2024-06-03T08:36:33.517 INFO:teuthology.orchestra.run.smithi088.stderr:2024-06-03T08:36:33.517+0000 7f16b9c8f640 1 -- 172.21.15.88:0/2965169668 --> v1:172.21.15.88:6789/0 -- mon_command({"prefix": "osd last-stat-seq", "id": 5} v 0) v1 -- 0x7f16b404ea60 con 0x7f16b41451c0 2024-06-03T08:36:33.522 INFO:teuthology.orchestra.run.smithi088.stderr:2024-06-03T08:36:33.521+0000 7f169bfff640 1 -- 172.21.15.88:0/2965169668 <== mon.0 v1:172.21.15.88:6789/0 10 ==== mon_command_ack([{"prefix": "osd last-stat-seq", "id": 5}]=0 v0) v1 ==== 74+0+12 (unknown 2131975755 0 3791509452) 0x7f16ac050c60 con 0x7f16b41451c0 2024-06-03T08:36:33.522 INFO:teuthology.orchestra.run.smithi088.stderr:2024-06-03T08:36:33.521+0000 7f169bfff640 10 monclient: handle_mon_command_ack 2 [{"prefix": "osd last-stat-seq", "id": 5}] 2024-06-03T08:36:33.522 INFO:teuthology.orchestra.run.smithi088.stderr:2024-06-03T08:36:33.521+0000 7f169bfff640 10 monclient: _finish_command 2 = system:0 2024-06-03T08:36:33.523 INFO:teuthology.orchestra.run.smithi088.stdout:85899345930 2024-06-03T08:36:33.524 INFO:teuthology.orchestra.run.smithi088.stderr:2024-06-03T08:36:33.522+0000 7f16b9c8f640 10 librados: watch_flush enter 2024-06-03T08:36:33.524 INFO:teuthology.orchestra.run.smithi088.stderr:2024-06-03T08:36:33.522+0000 7f16b9c8f640 10 librados: watch_flush exit 2024-06-03T08:36:33.524 INFO:teuthology.orchestra.run.smithi088.stderr:2024-06-03T08:36:33.522+0000 7f16b9c8f640 20 client.4356.objecter shutdown clearing up homeless session... 2024-06-03T08:36:33.524 INFO:teuthology.orchestra.run.smithi088.stderr:2024-06-03T08:36:33.522+0000 7f16b9c8f640 10 client.4356.objecter successfully canceled tick 2024-06-03T08:36:33.524 INFO:teuthology.orchestra.run.smithi088.stderr:2024-06-03T08:36:33.523+0000 7f16b9c8f640 1 -- 172.21.15.88:0/2965169668 >> v1:172.21.15.192:6800/34229 conn(0x7f16a006d230 legacy=0x7f16a006f6f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-06-03T08:36:33.524 INFO:teuthology.orchestra.run.smithi088.stderr:2024-06-03T08:36:33.523+0000 7f16b9c8f640 10 monclient: shutdown 2024-06-03T08:36:33.524 INFO:teuthology.orchestra.run.smithi088.stderr:2024-06-03T08:36:33.523+0000 7f16b9c8f640 20 monclient: shutdown discarding 0 pending message(s) 2024-06-03T08:36:33.524 INFO:teuthology.orchestra.run.smithi088.stderr:2024-06-03T08:36:33.523+0000 7f16b9c8f640 1 -- 172.21.15.88:0/2965169668 >> v1:172.21.15.88:6789/0 conn(0x7f16b41451c0 legacy=0x7f16b41335a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-06-03T08:36:33.524 INFO:teuthology.orchestra.run.smithi088.stderr:2024-06-03T08:36:33.523+0000 7f16b9c8f640 1 -- 172.21.15.88:0/2965169668 shutdown_connections 2024-06-03T08:36:33.524 INFO:teuthology.orchestra.run.smithi088.stderr:2024-06-03T08:36:33.523+0000 7f16b9c8f640 1 -- 172.21.15.88:0/2965169668 >> 172.21.15.88:0/2965169668 conn(0x7f16b40717f0 msgr2=0x7f16b40b63b0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-06-03T08:36:33.524 INFO:teuthology.orchestra.run.smithi088.stderr:2024-06-03T08:36:33.523+0000 7f16b9c8f640 1 -- 172.21.15.88:0/2965169668 shutdown_connections 2024-06-03T08:36:33.525 INFO:teuthology.orchestra.run.smithi088.stderr:2024-06-03T08:36:33.523+0000 7f16b9c8f640 1 -- 172.21.15.88:0/2965169668 wait complete. 2024-06-03T08:36:33.525 INFO:teuthology.orchestra.run.smithi088.stderr:2024-06-03T08:36:33.523+0000 7f16b9c8f640 1 librados: shutdown 2024-06-03T08:36:33.534 INFO:tasks.ceph.ceph_manager.ceph:need seq 85899345930 got 85899345930 for osd.5 2024-06-03T08:36:33.534 DEBUG:teuthology.parallel:result is None 2024-06-03T08:36:33.602 INFO:teuthology.orchestra.run.smithi088.stderr:2024-06-03T08:36:33.602+0000 7f3d5015c640 10 start_mon_command cmd=[{"prefix": "osd last-stat-seq", "id": 6}] 2024-06-03T08:36:33.602 INFO:teuthology.orchestra.run.smithi088.stderr:2024-06-03T08:36:33.602+0000 7f3d5015c640 10 monclient: _send_command 2 [{"prefix": "osd last-stat-seq", "id": 6}] 2024-06-03T08:36:33.603 INFO:teuthology.orchestra.run.smithi088.stderr:2024-06-03T08:36:33.602+0000 7f3d5015c640 10 monclient: _send_mon_message to mon.c at v1:172.21.15.88:6790/0 2024-06-03T08:36:33.603 INFO:teuthology.orchestra.run.smithi088.stderr:2024-06-03T08:36:33.602+0000 7f3d5015c640 1 -- 172.21.15.88:0/1768052638 --> v1:172.21.15.88:6790/0 -- mon_command({"prefix": "osd last-stat-seq", "id": 6} v 0) v1 -- 0x7f3d4804ea60 con 0x7f3d480b4250 2024-06-03T08:36:33.634 INFO:teuthology.orchestra.run.smithi088.stderr:2024-06-03T08:36:33.633+0000 7f3d3e7fc640 1 -- 172.21.15.88:0/1768052638 <== mon.2 v1:172.21.15.88:6790/0 10 ==== mon_command_ack([{"prefix": "osd last-stat-seq", "id": 6}]=0 v0) v1 ==== 74+0+12 (unknown 1274345170 0 591313397) 0x7f3d38051ad0 con 0x7f3d480b4250 2024-06-03T08:36:33.634 INFO:teuthology.orchestra.run.smithi088.stderr:2024-06-03T08:36:33.633+0000 7f3d3e7fc640 10 monclient: handle_mon_command_ack 2 [{"prefix": "osd last-stat-seq", "id": 6}] 2024-06-03T08:36:33.634 INFO:teuthology.orchestra.run.smithi088.stderr:2024-06-03T08:36:33.633+0000 7f3d3e7fc640 10 monclient: _finish_command 2 = system:0 2024-06-03T08:36:33.634 INFO:teuthology.orchestra.run.smithi088.stdout:90194313226 2024-06-03T08:36:33.636 INFO:teuthology.orchestra.run.smithi088.stderr:2024-06-03T08:36:33.635+0000 7f3d5015c640 10 librados: watch_flush enter 2024-06-03T08:36:33.636 INFO:teuthology.orchestra.run.smithi088.stderr:2024-06-03T08:36:33.635+0000 7f3d5015c640 10 librados: watch_flush exit 2024-06-03T08:36:33.636 INFO:teuthology.orchestra.run.smithi088.stderr:2024-06-03T08:36:33.635+0000 7f3d5015c640 20 client.4364.objecter shutdown clearing up homeless session... 2024-06-03T08:36:33.637 INFO:teuthology.orchestra.run.smithi088.stderr:2024-06-03T08:36:33.635+0000 7f3d5015c640 10 client.4364.objecter successfully canceled tick 2024-06-03T08:36:33.637 INFO:teuthology.orchestra.run.smithi088.stderr:2024-06-03T08:36:33.635+0000 7f3d5015c640 1 -- 172.21.15.88:0/1768052638 >> v1:172.21.15.192:6800/34229 conn(0x7f3d2c06d080 legacy=0x7f3d2c06f540 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-06-03T08:36:33.637 INFO:teuthology.orchestra.run.smithi088.stderr:2024-06-03T08:36:33.635+0000 7f3d5015c640 10 monclient: shutdown 2024-06-03T08:36:33.637 INFO:teuthology.orchestra.run.smithi088.stderr:2024-06-03T08:36:33.635+0000 7f3d5015c640 20 monclient: shutdown discarding 0 pending message(s) 2024-06-03T08:36:33.637 INFO:teuthology.orchestra.run.smithi088.stderr:2024-06-03T08:36:33.635+0000 7f3d5015c640 1 -- 172.21.15.88:0/1768052638 >> v1:172.21.15.88:6790/0 conn(0x7f3d480b4250 legacy=0x7f3d48138230 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-06-03T08:36:33.637 INFO:teuthology.orchestra.run.smithi088.stderr:2024-06-03T08:36:33.636+0000 7f3d5015c640 1 -- 172.21.15.88:0/1768052638 shutdown_connections 2024-06-03T08:36:33.637 INFO:teuthology.orchestra.run.smithi088.stderr:2024-06-03T08:36:33.636+0000 7f3d5015c640 1 -- 172.21.15.88:0/1768052638 >> 172.21.15.88:0/1768052638 conn(0x7f3d480717f0 msgr2=0x7f3d480b4d60 unknown :-1 s=STATE_NONE l=0).mark_down 2024-06-03T08:36:33.637 INFO:teuthology.orchestra.run.smithi088.stderr:2024-06-03T08:36:33.636+0000 7f3d5015c640 1 -- 172.21.15.88:0/1768052638 shutdown_connections 2024-06-03T08:36:33.637 INFO:teuthology.orchestra.run.smithi088.stderr:2024-06-03T08:36:33.636+0000 7f3d5015c640 1 -- 172.21.15.88:0/1768052638 wait complete. 2024-06-03T08:36:33.637 INFO:teuthology.orchestra.run.smithi088.stderr:2024-06-03T08:36:33.636+0000 7f3d5015c640 1 librados: shutdown 2024-06-03T08:36:33.647 INFO:tasks.ceph.ceph_manager.ceph:need seq 90194313226 got 90194313226 for osd.6 2024-06-03T08:36:33.647 DEBUG:teuthology.parallel:result is None 2024-06-03T08:36:34.005 DEBUG:teuthology.orchestra.run.smithi088:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd last-stat-seq osd.2 2024-06-03T08:36:34.105 INFO:teuthology.orchestra.run.smithi088.stderr:2024-06-03T08:36:34.104+0000 7fd769ee5640 10 monclient: get_monmap_and_config 2024-06-03T08:36:34.105 INFO:teuthology.orchestra.run.smithi088.stderr:2024-06-03T08:36:34.104+0000 7fd769ee5640 10 monclient: build_initial_monmap 2024-06-03T08:36:34.105 INFO:teuthology.orchestra.run.smithi088.stderr:2024-06-03T08:36:34.104+0000 7fd769ee5640 10 monclient: monmap: 2024-06-03T08:36:34.105 INFO:teuthology.orchestra.run.smithi088.stderr:epoch 0 2024-06-03T08:36:34.105 INFO:teuthology.orchestra.run.smithi088.stderr:fsid ec3c3fdd-3ac6-4d7d-9412-683130ed8b04 2024-06-03T08:36:34.105 INFO:teuthology.orchestra.run.smithi088.stderr:last_changed 2024-06-03T08:36:34.105620+0000 2024-06-03T08:36:34.105 INFO:teuthology.orchestra.run.smithi088.stderr:created 2024-06-03T08:36:34.105620+0000 2024-06-03T08:36:34.105 INFO:teuthology.orchestra.run.smithi088.stderr:min_mon_release 0 (unknown) 2024-06-03T08:36:34.105 INFO:teuthology.orchestra.run.smithi088.stderr:election_strategy: 3 2024-06-03T08:36:34.105 INFO:teuthology.orchestra.run.smithi088.stderr:0: v1:172.21.15.88:6789/0 mon.noname-a 2024-06-03T08:36:34.106 INFO:teuthology.orchestra.run.smithi088.stderr:1: v1:172.21.15.192:6789/0 mon.noname-c 2024-06-03T08:36:34.106 INFO:teuthology.orchestra.run.smithi088.stderr:2: v1:172.21.15.88:6790/0 mon.noname-b 2024-06-03T08:36:34.106 INFO:teuthology.orchestra.run.smithi088.stderr: 2024-06-03T08:36:34.107 INFO:teuthology.orchestra.run.smithi088.stderr:2024-06-03T08:36:34.105+0000 7fd769ee5640 1 Processor -- start 2024-06-03T08:36:34.107 INFO:teuthology.orchestra.run.smithi088.stderr:2024-06-03T08:36:34.105+0000 7fd769ee5640 1 -- start start 2024-06-03T08:36:34.107 INFO:teuthology.orchestra.run.smithi088.stderr:2024-06-03T08:36:34.105+0000 7fd769ee5640 10 monclient: init 2024-06-03T08:36:34.107 INFO:teuthology.orchestra.run.smithi088.stderr:2024-06-03T08:36:34.105+0000 7fd769ee5640 10 monclient: _reopen_session rank -1 2024-06-03T08:36:34.107 INFO:teuthology.orchestra.run.smithi088.stderr:2024-06-03T08:36:34.105+0000 7fd769ee5640 10 monclient: _add_conns ranks=[1,0,2] 2024-06-03T08:36:34.107 INFO:teuthology.orchestra.run.smithi088.stderr:2024-06-03T08:36:34.105+0000 7fd769ee5640 10 monclient(hunting): picked mon.noname-c con 0x7fd7640adb90 addr v1:172.21.15.192:6789/0 2024-06-03T08:36:34.107 INFO:teuthology.orchestra.run.smithi088.stderr:2024-06-03T08:36:34.105+0000 7fd769ee5640 10 monclient(hunting): picked mon.noname-a con 0x7fd76406ee00 addr v1:172.21.15.88:6789/0 2024-06-03T08:36:34.107 INFO:teuthology.orchestra.run.smithi088.stderr:2024-06-03T08:36:34.106+0000 7fd769ee5640 10 monclient(hunting): picked mon.noname-b con 0x7fd7641383e0 addr v1:172.21.15.88:6790/0 2024-06-03T08:36:34.107 INFO:teuthology.orchestra.run.smithi088.stderr:2024-06-03T08:36:34.106+0000 7fd769ee5640 1 -- --> v1:172.21.15.88:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fd76406f2c0 con 0x7fd76406ee00 2024-06-03T08:36:34.108 INFO:teuthology.orchestra.run.smithi088.stderr:2024-06-03T08:36:34.106+0000 7fd769ee5640 1 -- --> v1:172.21.15.192:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fd7640710c0 con 0x7fd7640adb90 2024-06-03T08:36:34.108 INFO:teuthology.orchestra.run.smithi088.stderr:2024-06-03T08:36:34.106+0000 7fd769ee5640 1 -- --> v1:172.21.15.88:6790/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fd764071fd0 con 0x7fd7641383e0 2024-06-03T08:36:34.108 INFO:teuthology.orchestra.run.smithi088.stderr:2024-06-03T08:36:34.106+0000 7fd769ee5640 10 monclient(hunting): _renew_subs 2024-06-03T08:36:34.108 INFO:teuthology.orchestra.run.smithi088.stderr:2024-06-03T08:36:34.106+0000 7fd769ee5640 10 monclient(hunting): authenticate will time out at 1249.915982s 2024-06-03T08:36:34.108 INFO:teuthology.orchestra.run.smithi088.stderr:2024-06-03T08:36:34.106+0000 7fd7637fe640 1 --1- >> v1:172.21.15.192:6789/0 conn(0x7fd7640adb90 0x7fd7640718c0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.192:6789/0 says I am v1:172.21.15.88:52622/0 (socket says 172.21.15.88:52622) 2024-06-03T08:36:34.108 INFO:teuthology.orchestra.run.smithi088.stderr:2024-06-03T08:36:34.106+0000 7fd7637fe640 1 -- 172.21.15.88:0/1911298615 learned_addr learned my addr 172.21.15.88:0/1911298615 (peer_addr_for_me v1:172.21.15.88:0/0) 2024-06-03T08:36:34.109 INFO:teuthology.orchestra.run.smithi088.stderr:2024-06-03T08:36:34.108+0000 7fd7627fc640 1 -- 172.21.15.88:0/1911298615 <== mon.1 v1:172.21.15.192:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 140618422 0 0) 0x7fd7640710c0 con 0x7fd7640adb90 2024-06-03T08:36:34.109 INFO:teuthology.orchestra.run.smithi088.stderr:2024-06-03T08:36:34.109+0000 7fd7627fc640 10 monclient(hunting): _negotiate 2024-06-03T08:36:34.109 INFO:teuthology.orchestra.run.smithi088.stderr:2024-06-03T08:36:34.109+0000 7fd7627fc640 10 monclient(hunting): _init_auth method 2 2024-06-03T08:36:34.109 INFO:teuthology.orchestra.run.smithi088.stderr:2024-06-03T08:36:34.109+0000 7fd7627fc640 10 monclient(hunting): _init_auth creating new auth 2024-06-03T08:36:34.109 INFO:teuthology.orchestra.run.smithi088.stderr:2024-06-03T08:36:34.109+0000 7fd7627fc640 10 monclient(hunting): my global_id is 4366 2024-06-03T08:36:34.109 INFO:teuthology.orchestra.run.smithi088.stderr:2024-06-03T08:36:34.109+0000 7fd7627fc640 1 -- 172.21.15.88:0/1911298615 --> v1:172.21.15.192:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7fd74c003520 con 0x7fd7640adb90 2024-06-03T08:36:34.110 INFO:teuthology.orchestra.run.smithi088.stderr:2024-06-03T08:36:34.109+0000 7fd7627fc640 1 -- 172.21.15.88:0/1911298615 <== mon.2 v1:172.21.15.88:6790/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 918541160 0 0) 0x7fd764071fd0 con 0x7fd7641383e0 2024-06-03T08:36:34.110 INFO:teuthology.orchestra.run.smithi088.stderr:2024-06-03T08:36:34.109+0000 7fd7627fc640 10 monclient(hunting): _negotiate 2024-06-03T08:36:34.110 INFO:teuthology.orchestra.run.smithi088.stderr:2024-06-03T08:36:34.109+0000 7fd7627fc640 10 monclient(hunting): _init_auth method 2 2024-06-03T08:36:34.110 INFO:teuthology.orchestra.run.smithi088.stderr:2024-06-03T08:36:34.109+0000 7fd7627fc640 10 monclient(hunting): _init_auth creating new auth 2024-06-03T08:36:34.110 INFO:teuthology.orchestra.run.smithi088.stderr:2024-06-03T08:36:34.109+0000 7fd7627fc640 10 monclient(hunting): my global_id is 4367 2024-06-03T08:36:34.110 INFO:teuthology.orchestra.run.smithi088.stderr:2024-06-03T08:36:34.109+0000 7fd7627fc640 1 -- 172.21.15.88:0/1911298615 --> v1:172.21.15.88:6790/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7fd7640710c0 con 0x7fd7641383e0 2024-06-03T08:36:34.110 INFO:teuthology.orchestra.run.smithi088.stderr:2024-06-03T08:36:34.109+0000 7fd7627fc640 1 -- 172.21.15.88:0/1911298615 <== mon.0 v1:172.21.15.88:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 3302346429 0 0) 0x7fd76406f2c0 con 0x7fd76406ee00 2024-06-03T08:36:34.110 INFO:teuthology.orchestra.run.smithi088.stderr:2024-06-03T08:36:34.109+0000 7fd7627fc640 10 monclient(hunting): _negotiate 2024-06-03T08:36:34.110 INFO:teuthology.orchestra.run.smithi088.stderr:2024-06-03T08:36:34.109+0000 7fd7627fc640 10 monclient(hunting): _init_auth method 2 2024-06-03T08:36:34.110 INFO:teuthology.orchestra.run.smithi088.stderr:2024-06-03T08:36:34.109+0000 7fd7627fc640 10 monclient(hunting): _init_auth creating new auth 2024-06-03T08:36:34.110 INFO:teuthology.orchestra.run.smithi088.stderr:2024-06-03T08:36:34.109+0000 7fd7627fc640 10 monclient(hunting): my global_id is 4362 2024-06-03T08:36:34.110 INFO:teuthology.orchestra.run.smithi088.stderr:2024-06-03T08:36:34.109+0000 7fd7627fc640 1 -- 172.21.15.88:0/1911298615 --> v1:172.21.15.88:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7fd764071fd0 con 0x7fd76406ee00 2024-06-03T08:36:34.112 INFO:teuthology.orchestra.run.smithi088.stderr:2024-06-03T08:36:34.112+0000 7fd7627fc640 1 -- 172.21.15.88:0/1911298615 <== mon.0 v1:172.21.15.88:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 214+0+0 (unknown 2000545228 0 0) 0x7fd764071fd0 con 0x7fd76406ee00 2024-06-03T08:36:34.112 INFO:teuthology.orchestra.run.smithi088.stderr:2024-06-03T08:36:34.112+0000 7fd7627fc640 1 -- 172.21.15.88:0/1911298615 >> v1:172.21.15.88:6790/0 conn(0x7fd7641383e0 legacy=0x7fd764149af0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-06-03T08:36:34.112 INFO:teuthology.orchestra.run.smithi088.stderr:2024-06-03T08:36:34.112+0000 7fd7627fc640 1 -- 172.21.15.88:0/1911298615 >> v1:172.21.15.192:6789/0 conn(0x7fd7640adb90 legacy=0x7fd7640718c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-06-03T08:36:34.112 INFO:teuthology.orchestra.run.smithi088.stderr:2024-06-03T08:36:34.112+0000 7fd7627fc640 10 monclient: _finish_hunting 0 2024-06-03T08:36:34.113 INFO:teuthology.orchestra.run.smithi088.stderr:2024-06-03T08:36:34.112+0000 7fd7627fc640 1 monclient: found mon.noname-a 2024-06-03T08:36:34.113 INFO:teuthology.orchestra.run.smithi088.stderr:2024-06-03T08:36:34.112+0000 7fd7627fc640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-06-03T08:36:34.113 INFO:teuthology.orchestra.run.smithi088.stderr:2024-06-03T08:36:34.112+0000 7fd7627fc640 10 monclient: _send_mon_message to mon.noname-a at v1:172.21.15.88:6789/0 2024-06-03T08:36:34.113 INFO:teuthology.orchestra.run.smithi088.stderr:2024-06-03T08:36:34.112+0000 7fd7627fc640 1 -- 172.21.15.88:0/1911298615 --> v1:172.21.15.88:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fd764141750 con 0x7fd76406ee00 2024-06-03T08:36:34.113 INFO:teuthology.orchestra.run.smithi088.stderr:2024-06-03T08:36:34.112+0000 7fd7627fc640 10 monclient: _finish_auth 0 2024-06-03T08:36:34.113 INFO:teuthology.orchestra.run.smithi088.stderr:2024-06-03T08:36:34.112+0000 7fd7627fc640 10 monclient: _check_auth_tickets 2024-06-03T08:36:34.113 INFO:teuthology.orchestra.run.smithi088.stderr:2024-06-03T08:36:34.112+0000 7fd7627fc640 20 monclient: _check_auth_rotating not needed by client.admin 2024-06-03T08:36:34.113 INFO:teuthology.orchestra.run.smithi088.stderr:2024-06-03T08:36:34.112+0000 7fd769ee5640 5 monclient: authenticate success, global_id 4362 2024-06-03T08:36:34.113 INFO:teuthology.orchestra.run.smithi088.stderr:2024-06-03T08:36:34.112+0000 7fd769ee5640 20 monclient: get_monmap_and_config waiting for monmap|config 2024-06-03T08:36:34.114 INFO:teuthology.orchestra.run.smithi088.stderr:2024-06-03T08:36:34.112+0000 7fd7627fc640 1 -- 172.21.15.88:0/1911298615 <== mon.0 v1:172.21.15.88:6789/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 2234374065 0 0) 0x7fd758001530 con 0x7fd76406ee00 2024-06-03T08:36:34.114 INFO:teuthology.orchestra.run.smithi088.stderr:2024-06-03T08:36:34.112+0000 7fd7627fc640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-06-03T08:36:34.114 INFO:teuthology.orchestra.run.smithi088.stderr:2024-06-03T08:36:34.112+0000 7fd7627fc640 10 monclient: got monmap 1 from mon.noname-a (according to old e1) 2024-06-03T08:36:34.114 INFO:teuthology.orchestra.run.smithi088.stderr:2024-06-03T08:36:34.112+0000 7fd7627fc640 10 monclient: dump: 2024-06-03T08:36:34.114 INFO:teuthology.orchestra.run.smithi088.stderr:epoch 1 2024-06-03T08:36:34.114 INFO:teuthology.orchestra.run.smithi088.stderr:fsid ec3c3fdd-3ac6-4d7d-9412-683130ed8b04 2024-06-03T08:36:34.114 INFO:teuthology.orchestra.run.smithi088.stderr:last_changed 2024-06-03T08:34:31.885125+0000 2024-06-03T08:36:34.114 INFO:teuthology.orchestra.run.smithi088.stderr:created 2024-06-03T08:34:31.885125+0000 2024-06-03T08:36:34.114 INFO:teuthology.orchestra.run.smithi088.stderr:min_mon_release 18 (reef) 2024-06-03T08:36:34.115 INFO:teuthology.orchestra.run.smithi088.stderr:election_strategy: 3 2024-06-03T08:36:34.115 INFO:teuthology.orchestra.run.smithi088.stderr:0: v1:172.21.15.88:6789/0 mon.a 2024-06-03T08:36:34.115 INFO:teuthology.orchestra.run.smithi088.stderr:1: v1:172.21.15.192:6789/0 mon.b 2024-06-03T08:36:34.115 INFO:teuthology.orchestra.run.smithi088.stderr:2: v1:172.21.15.88:6790/0 mon.c 2024-06-03T08:36:34.115 INFO:teuthology.orchestra.run.smithi088.stderr: 2024-06-03T08:36:34.115 INFO:teuthology.orchestra.run.smithi088.stderr:2024-06-03T08:36:34.112+0000 7fd769ee5640 20 monclient: get_monmap_and_config waiting for monmap|config 2024-06-03T08:36:34.116 INFO:teuthology.orchestra.run.smithi088.stderr:2024-06-03T08:36:34.114+0000 7fd7627fc640 1 -- 172.21.15.88:0/1911298615 <== mon.0 v1:172.21.15.88:6789/0 4 ==== config(0 keys) v1 ==== 4+0+0 (unknown 0 0 0) 0x7fd758001780 con 0x7fd76406ee00 2024-06-03T08:36:34.116 INFO:teuthology.orchestra.run.smithi088.stderr:2024-06-03T08:36:34.114+0000 7fd7627fc640 10 monclient: handle_config config(0 keys) v1 2024-06-03T08:36:34.117 INFO:teuthology.orchestra.run.smithi088.stderr:2024-06-03T08:36:34.114+0000 7fd7627fc640 1 -- 172.21.15.88:0/1911298615 <== mon.0 v1:172.21.15.88:6789/0 5 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 2234374065 0 0) 0x7fd758001b50 con 0x7fd76406ee00 2024-06-03T08:36:34.117 INFO:teuthology.orchestra.run.smithi088.stderr:2024-06-03T08:36:34.114+0000 7fd7627fc640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-06-03T08:36:34.117 INFO:teuthology.orchestra.run.smithi088.stderr:2024-06-03T08:36:34.114+0000 7fd7627fc640 10 monclient: got monmap 1 from mon.a (according to old e1) 2024-06-03T08:36:34.117 INFO:teuthology.orchestra.run.smithi088.stderr:2024-06-03T08:36:34.114+0000 7fd7627fc640 10 monclient: dump: 2024-06-03T08:36:34.117 INFO:teuthology.orchestra.run.smithi088.stderr:epoch 1 2024-06-03T08:36:34.117 INFO:teuthology.orchestra.run.smithi088.stderr:fsid ec3c3fdd-3ac6-4d7d-9412-683130ed8b04 2024-06-03T08:36:34.117 INFO:teuthology.orchestra.run.smithi088.stderr:last_changed 2024-06-03T08:34:31.885125+0000 2024-06-03T08:36:34.117 INFO:teuthology.orchestra.run.smithi088.stderr:created 2024-06-03T08:34:31.885125+0000 2024-06-03T08:36:34.117 INFO:teuthology.orchestra.run.smithi088.stderr:min_mon_release 18 (reef) 2024-06-03T08:36:34.117 INFO:teuthology.orchestra.run.smithi088.stderr:election_strategy: 3 2024-06-03T08:36:34.117 INFO:teuthology.orchestra.run.smithi088.stderr:0: v1:172.21.15.88:6789/0 mon.a 2024-06-03T08:36:34.117 INFO:teuthology.orchestra.run.smithi088.stderr:1: v1:172.21.15.192:6789/0 mon.b 2024-06-03T08:36:34.117 INFO:teuthology.orchestra.run.smithi088.stderr:2: v1:172.21.15.88:6790/0 mon.c 2024-06-03T08:36:34.118 INFO:teuthology.orchestra.run.smithi088.stderr: 2024-06-03T08:36:34.118 INFO:teuthology.orchestra.run.smithi088.stderr:2024-06-03T08:36:34.114+0000 7fd769ee5640 10 monclient: get_monmap_and_config success 2024-06-03T08:36:34.118 INFO:teuthology.orchestra.run.smithi088.stderr:2024-06-03T08:36:34.114+0000 7fd769ee5640 4 set_mon_vals no callback set 2024-06-03T08:36:34.118 INFO:teuthology.orchestra.run.smithi088.stderr:2024-06-03T08:36:34.115+0000 7fd769ee5640 10 monclient: shutdown 2024-06-03T08:36:34.118 INFO:teuthology.orchestra.run.smithi088.stderr:2024-06-03T08:36:34.115+0000 7fd769ee5640 20 monclient: shutdown discarding 0 pending message(s) 2024-06-03T08:36:34.118 INFO:teuthology.orchestra.run.smithi088.stderr:2024-06-03T08:36:34.115+0000 7fd769ee5640 1 -- 172.21.15.88:0/1911298615 >> v1:172.21.15.88:6789/0 conn(0x7fd76406ee00 legacy=0x7fd764137cd0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-06-03T08:36:34.118 INFO:teuthology.orchestra.run.smithi088.stderr:2024-06-03T08:36:34.115+0000 7fd769ee5640 1 -- 172.21.15.88:0/1911298615 shutdown_connections 2024-06-03T08:36:34.118 INFO:teuthology.orchestra.run.smithi088.stderr:2024-06-03T08:36:34.115+0000 7fd769ee5640 1 -- 172.21.15.88:0/1911298615 >> 172.21.15.88:0/1911298615 conn(0x7fd7640b2f20 msgr2=0x7fd7640b3300 unknown :-1 s=STATE_NONE l=0).mark_down 2024-06-03T08:36:34.118 INFO:teuthology.orchestra.run.smithi088.stderr:2024-06-03T08:36:34.115+0000 7fd769ee5640 1 -- 172.21.15.88:0/1911298615 shutdown_connections 2024-06-03T08:36:34.118 INFO:teuthology.orchestra.run.smithi088.stderr:2024-06-03T08:36:34.115+0000 7fd769ee5640 1 -- 172.21.15.88:0/1911298615 wait complete. 2024-06-03T08:36:34.118 INFO:teuthology.orchestra.run.smithi088.stderr:2024-06-03T08:36:34.115+0000 7fd769ee5640 10 monclient: build_initial_monmap 2024-06-03T08:36:34.118 INFO:teuthology.orchestra.run.smithi088.stderr:2024-06-03T08:36:34.115+0000 7fd769ee5640 10 monclient: monmap: 2024-06-03T08:36:34.118 INFO:teuthology.orchestra.run.smithi088.stderr:epoch 0 2024-06-03T08:36:34.118 INFO:teuthology.orchestra.run.smithi088.stderr:fsid 00000000-0000-0000-0000-000000000000 2024-06-03T08:36:34.119 INFO:teuthology.orchestra.run.smithi088.stderr:last_changed 0.000000 2024-06-03T08:36:34.119 INFO:teuthology.orchestra.run.smithi088.stderr:created 0.000000 2024-06-03T08:36:34.119 INFO:teuthology.orchestra.run.smithi088.stderr:min_mon_release 0 (unknown) 2024-06-03T08:36:34.119 INFO:teuthology.orchestra.run.smithi088.stderr:election_strategy: 1 2024-06-03T08:36:34.119 INFO:teuthology.orchestra.run.smithi088.stderr:0: v1:172.21.15.88:6789/0 mon.noname-a 2024-06-03T08:36:34.119 INFO:teuthology.orchestra.run.smithi088.stderr:1: v1:172.21.15.192:6789/0 mon.noname-b 2024-06-03T08:36:34.119 INFO:teuthology.orchestra.run.smithi088.stderr:2: v1:172.21.15.88:6790/0 mon.noname-c 2024-06-03T08:36:34.119 INFO:teuthology.orchestra.run.smithi088.stderr: 2024-06-03T08:36:34.119 INFO:teuthology.orchestra.run.smithi088.stderr:2024-06-03T08:36:34.116+0000 7fd769ee5640 1 librados: starting msgr at 2024-06-03T08:36:34.119 INFO:teuthology.orchestra.run.smithi088.stderr:2024-06-03T08:36:34.116+0000 7fd769ee5640 1 librados: starting objecter 2024-06-03T08:36:34.119 INFO:teuthology.orchestra.run.smithi088.stderr:2024-06-03T08:36:34.116+0000 7fd769ee5640 1 Processor -- start 2024-06-03T08:36:34.119 INFO:teuthology.orchestra.run.smithi088.stderr:2024-06-03T08:36:34.116+0000 7fd769ee5640 1 -- start start 2024-06-03T08:36:34.119 INFO:teuthology.orchestra.run.smithi088.stderr:2024-06-03T08:36:34.116+0000 7fd769ee5640 1 librados: setting wanted keys 2024-06-03T08:36:34.119 INFO:teuthology.orchestra.run.smithi088.stderr:2024-06-03T08:36:34.116+0000 7fd769ee5640 1 librados: calling monclient init 2024-06-03T08:36:34.120 INFO:teuthology.orchestra.run.smithi088.stderr:2024-06-03T08:36:34.116+0000 7fd769ee5640 10 monclient: init 2024-06-03T08:36:34.121 INFO:teuthology.orchestra.run.smithi088.stderr:2024-06-03T08:36:34.117+0000 7fd769ee5640 10 monclient: _reopen_session rank -1 2024-06-03T08:36:34.122 INFO:teuthology.orchestra.run.smithi088.stderr:2024-06-03T08:36:34.117+0000 7fd769ee5640 10 monclient: _add_conns ranks=[1,0,2] 2024-06-03T08:36:34.122 INFO:teuthology.orchestra.run.smithi088.stderr:2024-06-03T08:36:34.117+0000 7fd769ee5640 10 monclient(hunting): picked mon.noname-b con 0x7fd76406ee00 addr v1:172.21.15.192:6789/0 2024-06-03T08:36:34.122 INFO:teuthology.orchestra.run.smithi088.stderr:2024-06-03T08:36:34.117+0000 7fd769ee5640 10 monclient(hunting): picked mon.noname-a con 0x7fd7640adb90 addr v1:172.21.15.88:6789/0 2024-06-03T08:36:34.122 INFO:teuthology.orchestra.run.smithi088.stderr:2024-06-03T08:36:34.117+0000 7fd769ee5640 10 monclient(hunting): picked mon.noname-c con 0x7fd7641383e0 addr v1:172.21.15.88:6790/0 2024-06-03T08:36:34.122 INFO:teuthology.orchestra.run.smithi088.stderr:2024-06-03T08:36:34.117+0000 7fd769ee5640 1 -- --> v1:172.21.15.88:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fd76413c340 con 0x7fd7640adb90 2024-06-03T08:36:34.122 INFO:teuthology.orchestra.run.smithi088.stderr:2024-06-03T08:36:34.117+0000 7fd769ee5640 1 -- --> v1:172.21.15.192:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fd76413e120 con 0x7fd76406ee00 2024-06-03T08:36:34.122 INFO:teuthology.orchestra.run.smithi088.stderr:2024-06-03T08:36:34.117+0000 7fd769ee5640 1 -- --> v1:172.21.15.88:6790/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fd76413dc00 con 0x7fd7641383e0 2024-06-03T08:36:34.122 INFO:teuthology.orchestra.run.smithi088.stderr:2024-06-03T08:36:34.117+0000 7fd769ee5640 10 monclient(hunting): _renew_subs 2024-06-03T08:36:34.122 INFO:teuthology.orchestra.run.smithi088.stderr:2024-06-03T08:36:34.117+0000 7fd769ee5640 10 monclient(hunting): authenticate will time out at 1249.927165s 2024-06-03T08:36:34.122 INFO:teuthology.orchestra.run.smithi088.stderr:2024-06-03T08:36:34.117+0000 7fd762ffd640 1 --1- >> v1:172.21.15.88:6789/0 conn(0x7fd7640adb90 0x7fd764139d00 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=1).handle_server_banner_and_identify peer v1:172.21.15.88:6789/0 says I am v1:172.21.15.88:60520/0 (socket says 172.21.15.88:60520) 2024-06-03T08:36:34.122 INFO:teuthology.orchestra.run.smithi088.stderr:2024-06-03T08:36:34.117+0000 7fd762ffd640 1 -- 172.21.15.88:0/3191884890 learned_addr learned my addr 172.21.15.88:0/3191884890 (peer_addr_for_me v1:172.21.15.88:0/0) 2024-06-03T08:36:34.123 INFO:teuthology.orchestra.run.smithi088.stderr:2024-06-03T08:36:34.118+0000 7fd74bfff640 10 client.?.objecter ms_handle_connect 0x7fd7640adb90 2024-06-03T08:36:34.123 INFO:teuthology.orchestra.run.smithi088.stderr:2024-06-03T08:36:34.118+0000 7fd74bfff640 10 client.?.objecter resend_mon_ops 2024-06-03T08:36:34.123 INFO:teuthology.orchestra.run.smithi088.stderr:2024-06-03T08:36:34.118+0000 7fd74bfff640 10 client.?.objecter ms_handle_connect 0x7fd7641383e0 2024-06-03T08:36:34.123 INFO:teuthology.orchestra.run.smithi088.stderr:2024-06-03T08:36:34.118+0000 7fd74bfff640 10 client.?.objecter resend_mon_ops 2024-06-03T08:36:34.123 INFO:teuthology.orchestra.run.smithi088.stderr:2024-06-03T08:36:34.118+0000 7fd74bfff640 10 client.?.objecter ms_handle_connect 0x7fd76406ee00 2024-06-03T08:36:34.123 INFO:teuthology.orchestra.run.smithi088.stderr:2024-06-03T08:36:34.118+0000 7fd74bfff640 10 client.?.objecter resend_mon_ops 2024-06-03T08:36:34.123 INFO:teuthology.orchestra.run.smithi088.stderr:2024-06-03T08:36:34.119+0000 7fd74bfff640 1 -- 172.21.15.88:0/3191884890 <== mon.0 v1:172.21.15.88:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 261053147 0 0) 0x7fd76413c340 con 0x7fd7640adb90 2024-06-03T08:36:34.123 INFO:teuthology.orchestra.run.smithi088.stderr:2024-06-03T08:36:34.119+0000 7fd74bfff640 10 monclient(hunting): _negotiate 2024-06-03T08:36:34.123 INFO:teuthology.orchestra.run.smithi088.stderr:2024-06-03T08:36:34.119+0000 7fd74bfff640 10 monclient(hunting): _init_auth method 2 2024-06-03T08:36:34.123 INFO:teuthology.orchestra.run.smithi088.stderr:2024-06-03T08:36:34.119+0000 7fd74bfff640 10 monclient(hunting): _init_auth creating new auth 2024-06-03T08:36:34.123 INFO:teuthology.orchestra.run.smithi088.stderr:2024-06-03T08:36:34.119+0000 7fd74bfff640 10 monclient(hunting): my global_id is 4365 2024-06-03T08:36:34.123 INFO:teuthology.orchestra.run.smithi088.stderr:2024-06-03T08:36:34.119+0000 7fd74bfff640 1 -- 172.21.15.88:0/3191884890 --> v1:172.21.15.88:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7fd750004eb0 con 0x7fd7640adb90 2024-06-03T08:36:34.123 INFO:teuthology.orchestra.run.smithi088.stderr:2024-06-03T08:36:34.120+0000 7fd74bfff640 1 -- 172.21.15.88:0/3191884890 <== mon.2 v1:172.21.15.88:6790/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 2582845687 0 0) 0x7fd76413dc00 con 0x7fd7641383e0 2024-06-03T08:36:34.123 INFO:teuthology.orchestra.run.smithi088.stderr:2024-06-03T08:36:34.120+0000 7fd74bfff640 10 monclient(hunting): _negotiate 2024-06-03T08:36:34.124 INFO:teuthology.orchestra.run.smithi088.stderr:2024-06-03T08:36:34.120+0000 7fd74bfff640 10 monclient(hunting): _init_auth method 2 2024-06-03T08:36:34.124 INFO:teuthology.orchestra.run.smithi088.stderr:2024-06-03T08:36:34.120+0000 7fd74bfff640 10 monclient(hunting): _init_auth creating new auth 2024-06-03T08:36:34.124 INFO:teuthology.orchestra.run.smithi088.stderr:2024-06-03T08:36:34.120+0000 7fd74bfff640 10 monclient(hunting): my global_id is 4370 2024-06-03T08:36:34.124 INFO:teuthology.orchestra.run.smithi088.stderr:2024-06-03T08:36:34.120+0000 7fd74bfff640 1 -- 172.21.15.88:0/3191884890 --> v1:172.21.15.88:6790/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7fd76413c340 con 0x7fd7641383e0 2024-06-03T08:36:34.124 INFO:teuthology.orchestra.run.smithi088.stderr:2024-06-03T08:36:34.120+0000 7fd74bfff640 1 -- 172.21.15.88:0/3191884890 <== mon.1 v1:172.21.15.192:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 2588578332 0 0) 0x7fd76413e120 con 0x7fd76406ee00 2024-06-03T08:36:34.124 INFO:teuthology.orchestra.run.smithi088.stderr:2024-06-03T08:36:34.120+0000 7fd74bfff640 10 monclient(hunting): _negotiate 2024-06-03T08:36:34.124 INFO:teuthology.orchestra.run.smithi088.stderr:2024-06-03T08:36:34.120+0000 7fd74bfff640 10 monclient(hunting): _init_auth method 2 2024-06-03T08:36:34.124 INFO:teuthology.orchestra.run.smithi088.stderr:2024-06-03T08:36:34.120+0000 7fd74bfff640 10 monclient(hunting): _init_auth creating new auth 2024-06-03T08:36:34.124 INFO:teuthology.orchestra.run.smithi088.stderr:2024-06-03T08:36:34.120+0000 7fd74bfff640 10 monclient(hunting): my global_id is 4369 2024-06-03T08:36:34.124 INFO:teuthology.orchestra.run.smithi088.stderr:2024-06-03T08:36:34.120+0000 7fd74bfff640 1 -- 172.21.15.88:0/3191884890 --> v1:172.21.15.192:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7fd76413dc00 con 0x7fd76406ee00 2024-06-03T08:36:34.125 INFO:teuthology.orchestra.run.smithi088.stderr:2024-06-03T08:36:34.124+0000 7fd74bfff640 1 -- 172.21.15.88:0/3191884890 <== mon.1 v1:172.21.15.192:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 764+0+0 (unknown 2059206176 0 0) 0x7fd76413dc00 con 0x7fd76406ee00 2024-06-03T08:36:34.125 INFO:teuthology.orchestra.run.smithi088.stderr:2024-06-03T08:36:34.124+0000 7fd74bfff640 1 -- 172.21.15.88:0/3191884890 --> v1:172.21.15.192:6789/0 -- auth(proto 2 165 bytes epoch 0) v1 -- 0x7fd76413e120 con 0x7fd76406ee00 2024-06-03T08:36:34.126 INFO:teuthology.orchestra.run.smithi088.stderr:2024-06-03T08:36:34.124+0000 7fd74bfff640 1 -- 172.21.15.88:0/3191884890 <== mon.1 v1:172.21.15.192:6789/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 2234374065 0 0) 0x7fd754001210 con 0x7fd76406ee00 2024-06-03T08:36:34.126 INFO:teuthology.orchestra.run.smithi088.stderr:2024-06-03T08:36:34.124+0000 7fd74bfff640 10 monclient(hunting): handle_monmap mon_map magic: 0 v1 2024-06-03T08:36:34.126 INFO:teuthology.orchestra.run.smithi088.stderr:2024-06-03T08:36:34.124+0000 7fd74bfff640 10 monclient(hunting): got monmap 1 from mon.noname-b (according to old e1) 2024-06-03T08:36:34.126 INFO:teuthology.orchestra.run.smithi088.stderr:2024-06-03T08:36:34.124+0000 7fd74bfff640 10 monclient(hunting): dump: 2024-06-03T08:36:34.126 INFO:teuthology.orchestra.run.smithi088.stderr:epoch 1 2024-06-03T08:36:34.126 INFO:teuthology.orchestra.run.smithi088.stderr:fsid ec3c3fdd-3ac6-4d7d-9412-683130ed8b04 2024-06-03T08:36:34.126 INFO:teuthology.orchestra.run.smithi088.stderr:last_changed 2024-06-03T08:34:31.885125+0000 2024-06-03T08:36:34.126 INFO:teuthology.orchestra.run.smithi088.stderr:created 2024-06-03T08:34:31.885125+0000 2024-06-03T08:36:34.126 INFO:teuthology.orchestra.run.smithi088.stderr:min_mon_release 18 (reef) 2024-06-03T08:36:34.126 INFO:teuthology.orchestra.run.smithi088.stderr:election_strategy: 3 2024-06-03T08:36:34.126 INFO:teuthology.orchestra.run.smithi088.stderr:0: v1:172.21.15.88:6789/0 mon.a 2024-06-03T08:36:34.126 INFO:teuthology.orchestra.run.smithi088.stderr:1: v1:172.21.15.192:6789/0 mon.b 2024-06-03T08:36:34.126 INFO:teuthology.orchestra.run.smithi088.stderr:2: v1:172.21.15.88:6790/0 mon.c 2024-06-03T08:36:34.127 INFO:teuthology.orchestra.run.smithi088.stderr: 2024-06-03T08:36:34.127 INFO:teuthology.orchestra.run.smithi088.stderr:2024-06-03T08:36:34.124+0000 7fd74bfff640 10 monclient(hunting): _finish_auth 0 2024-06-03T08:36:34.127 INFO:teuthology.orchestra.run.smithi088.stderr:2024-06-03T08:36:34.124+0000 7fd74bfff640 1 -- 172.21.15.88:0/3191884890 <== mon.0 v1:172.21.15.88:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 764+0+0 (unknown 2086703933 0 0) 0x7fd750004eb0 con 0x7fd7640adb90 2024-06-03T08:36:34.127 INFO:teuthology.orchestra.run.smithi088.stderr:2024-06-03T08:36:34.124+0000 7fd74bfff640 1 -- 172.21.15.88:0/3191884890 --> v1:172.21.15.88:6789/0 -- auth(proto 2 165 bytes epoch 0) v1 -- 0x7fd76413dc00 con 0x7fd7640adb90 2024-06-03T08:36:34.127 INFO:teuthology.orchestra.run.smithi088.stderr:2024-06-03T08:36:34.124+0000 7fd74bfff640 1 -- 172.21.15.88:0/3191884890 <== mon.0 v1:172.21.15.88:6789/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 2234374065 0 0) 0x7fd758001b50 con 0x7fd7640adb90 2024-06-03T08:36:34.127 INFO:teuthology.orchestra.run.smithi088.stderr:2024-06-03T08:36:34.125+0000 7fd74bfff640 10 monclient(hunting): handle_monmap mon_map magic: 0 v1 2024-06-03T08:36:34.127 INFO:teuthology.orchestra.run.smithi088.stderr:2024-06-03T08:36:34.125+0000 7fd74bfff640 10 monclient(hunting): got monmap 1 from mon.a (according to old e1) 2024-06-03T08:36:34.127 INFO:teuthology.orchestra.run.smithi088.stderr:2024-06-03T08:36:34.125+0000 7fd74bfff640 10 monclient(hunting): dump: 2024-06-03T08:36:34.127 INFO:teuthology.orchestra.run.smithi088.stderr:epoch 1 2024-06-03T08:36:34.127 INFO:teuthology.orchestra.run.smithi088.stderr:fsid ec3c3fdd-3ac6-4d7d-9412-683130ed8b04 2024-06-03T08:36:34.127 INFO:teuthology.orchestra.run.smithi088.stderr:last_changed 2024-06-03T08:34:31.885125+0000 2024-06-03T08:36:34.127 INFO:teuthology.orchestra.run.smithi088.stderr:created 2024-06-03T08:34:31.885125+0000 2024-06-03T08:36:34.127 INFO:teuthology.orchestra.run.smithi088.stderr:min_mon_release 18 (reef) 2024-06-03T08:36:34.127 INFO:teuthology.orchestra.run.smithi088.stderr:election_strategy: 3 2024-06-03T08:36:34.128 INFO:teuthology.orchestra.run.smithi088.stderr:0: v1:172.21.15.88:6789/0 mon.a 2024-06-03T08:36:34.128 INFO:teuthology.orchestra.run.smithi088.stderr:1: v1:172.21.15.192:6789/0 mon.b 2024-06-03T08:36:34.128 INFO:teuthology.orchestra.run.smithi088.stderr:2: v1:172.21.15.88:6790/0 mon.c 2024-06-03T08:36:34.128 INFO:teuthology.orchestra.run.smithi088.stderr: 2024-06-03T08:36:34.128 INFO:teuthology.orchestra.run.smithi088.stderr:2024-06-03T08:36:34.125+0000 7fd74bfff640 1 -- 172.21.15.88:0/3191884890 <== mon.2 v1:172.21.15.88:6790/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 764+0+0 (unknown 823810916 0 0) 0x7fd76413c340 con 0x7fd7641383e0 2024-06-03T08:36:34.128 INFO:teuthology.orchestra.run.smithi088.stderr:2024-06-03T08:36:34.125+0000 7fd74bfff640 1 -- 172.21.15.88:0/3191884890 --> v1:172.21.15.88:6790/0 -- auth(proto 2 165 bytes epoch 0) v1 -- 0x7fd750004eb0 con 0x7fd7641383e0 2024-06-03T08:36:34.128 INFO:teuthology.orchestra.run.smithi088.stderr:2024-06-03T08:36:34.125+0000 7fd74bfff640 1 -- 172.21.15.88:0/3191884890 <== mon.2 v1:172.21.15.88:6790/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 2234374065 0 0) 0x7fd75c0019a0 con 0x7fd7641383e0 2024-06-03T08:36:34.128 INFO:teuthology.orchestra.run.smithi088.stderr:2024-06-03T08:36:34.125+0000 7fd74bfff640 10 monclient(hunting): handle_monmap mon_map magic: 0 v1 2024-06-03T08:36:34.128 INFO:teuthology.orchestra.run.smithi088.stderr:2024-06-03T08:36:34.125+0000 7fd74bfff640 10 monclient(hunting): got monmap 1 from mon.c (according to old e1) 2024-06-03T08:36:34.128 INFO:teuthology.orchestra.run.smithi088.stderr:2024-06-03T08:36:34.125+0000 7fd74bfff640 10 monclient(hunting): dump: 2024-06-03T08:36:34.128 INFO:teuthology.orchestra.run.smithi088.stderr:epoch 1 2024-06-03T08:36:34.128 INFO:teuthology.orchestra.run.smithi088.stderr:fsid ec3c3fdd-3ac6-4d7d-9412-683130ed8b04 2024-06-03T08:36:34.128 INFO:teuthology.orchestra.run.smithi088.stderr:last_changed 2024-06-03T08:34:31.885125+0000 2024-06-03T08:36:34.128 INFO:teuthology.orchestra.run.smithi088.stderr:created 2024-06-03T08:34:31.885125+0000 2024-06-03T08:36:34.129 INFO:teuthology.orchestra.run.smithi088.stderr:min_mon_release 18 (reef) 2024-06-03T08:36:34.129 INFO:teuthology.orchestra.run.smithi088.stderr:election_strategy: 3 2024-06-03T08:36:34.129 INFO:teuthology.orchestra.run.smithi088.stderr:0: v1:172.21.15.88:6789/0 mon.a 2024-06-03T08:36:34.129 INFO:teuthology.orchestra.run.smithi088.stderr:1: v1:172.21.15.192:6789/0 mon.b 2024-06-03T08:36:34.129 INFO:teuthology.orchestra.run.smithi088.stderr:2: v1:172.21.15.88:6790/0 mon.c 2024-06-03T08:36:34.129 INFO:teuthology.orchestra.run.smithi088.stderr: 2024-06-03T08:36:34.130 INFO:teuthology.orchestra.run.smithi088.stderr:2024-06-03T08:36:34.128+0000 7fd74bfff640 1 -- 172.21.15.88:0/3191884890 <== mon.1 v1:172.21.15.192:6789/0 4 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 580+0+0 (unknown 2622909263 0 0) 0x7fd76413e120 con 0x7fd76406ee00 2024-06-03T08:36:34.130 INFO:teuthology.orchestra.run.smithi088.stderr:2024-06-03T08:36:34.128+0000 7fd74bfff640 1 -- 172.21.15.88:0/3191884890 >> v1:172.21.15.88:6790/0 conn(0x7fd7641383e0 legacy=0x7fd7641dcdb0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-06-03T08:36:34.130 INFO:teuthology.orchestra.run.smithi088.stderr:2024-06-03T08:36:34.128+0000 7fd74bfff640 1 -- 172.21.15.88:0/3191884890 >> v1:172.21.15.88:6789/0 conn(0x7fd7640adb90 legacy=0x7fd764139d00 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-06-03T08:36:34.131 INFO:teuthology.orchestra.run.smithi088.stderr:2024-06-03T08:36:34.128+0000 7fd74bfff640 10 monclient: _finish_hunting 0 2024-06-03T08:36:34.131 INFO:teuthology.orchestra.run.smithi088.stderr:2024-06-03T08:36:34.128+0000 7fd74bfff640 1 monclient: found mon.b 2024-06-03T08:36:34.131 INFO:teuthology.orchestra.run.smithi088.stderr:2024-06-03T08:36:34.128+0000 7fd74bfff640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-06-03T08:36:34.131 INFO:teuthology.orchestra.run.smithi088.stderr:2024-06-03T08:36:34.128+0000 7fd74bfff640 10 monclient: _send_mon_message to mon.b at v1:172.21.15.192:6789/0 2024-06-03T08:36:34.131 INFO:teuthology.orchestra.run.smithi088.stderr:2024-06-03T08:36:34.128+0000 7fd74bfff640 1 -- 172.21.15.88:0/3191884890 --> v1:172.21.15.192:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fd76413bfd0 con 0x7fd76406ee00 2024-06-03T08:36:34.131 INFO:teuthology.orchestra.run.smithi088.stderr:2024-06-03T08:36:34.128+0000 7fd74bfff640 10 monclient: _finish_auth 0 2024-06-03T08:36:34.131 INFO:teuthology.orchestra.run.smithi088.stderr:2024-06-03T08:36:34.128+0000 7fd74bfff640 10 monclient: _check_auth_tickets 2024-06-03T08:36:34.131 INFO:teuthology.orchestra.run.smithi088.stderr:2024-06-03T08:36:34.128+0000 7fd74bfff640 20 monclient: _check_auth_rotating not needed by client.admin 2024-06-03T08:36:34.131 INFO:teuthology.orchestra.run.smithi088.stderr:2024-06-03T08:36:34.128+0000 7fd769ee5640 5 monclient: authenticate success, global_id 4369 2024-06-03T08:36:34.131 INFO:teuthology.orchestra.run.smithi088.stderr:2024-06-03T08:36:34.128+0000 7fd769ee5640 10 monclient: _renew_subs 2024-06-03T08:36:34.131 INFO:teuthology.orchestra.run.smithi088.stderr:2024-06-03T08:36:34.128+0000 7fd769ee5640 10 monclient: _send_mon_message to mon.b at v1:172.21.15.192:6789/0 2024-06-03T08:36:34.131 INFO:teuthology.orchestra.run.smithi088.stderr:2024-06-03T08:36:34.128+0000 7fd769ee5640 1 -- 172.21.15.88:0/3191884890 --> v1:172.21.15.192:6789/0 -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fd76413d6e0 con 0x7fd76406ee00 2024-06-03T08:36:34.131 INFO:teuthology.orchestra.run.smithi088.stderr:2024-06-03T08:36:34.128+0000 7fd769ee5640 10 client.4369.objecter _maybe_request_map subscribing (onetime) to next osd map 2024-06-03T08:36:34.132 INFO:teuthology.orchestra.run.smithi088.stderr:2024-06-03T08:36:34.128+0000 7fd769ee5640 10 monclient: _renew_subs 2024-06-03T08:36:34.132 INFO:teuthology.orchestra.run.smithi088.stderr:2024-06-03T08:36:34.128+0000 7fd769ee5640 10 monclient: _send_mon_message to mon.b at v1:172.21.15.192:6789/0 2024-06-03T08:36:34.132 INFO:teuthology.orchestra.run.smithi088.stderr:2024-06-03T08:36:34.128+0000 7fd769ee5640 1 -- 172.21.15.88:0/3191884890 --> v1:172.21.15.192:6789/0 -- mon_subscribe({osdmap=0}) v3 -- 0x7fd76413a5f0 con 0x7fd76406ee00 2024-06-03T08:36:34.132 INFO:teuthology.orchestra.run.smithi088.stderr:2024-06-03T08:36:34.128+0000 7fd769ee5640 1 librados: init done 2024-06-03T08:36:34.132 INFO:teuthology.orchestra.run.smithi088.stderr:2024-06-03T08:36:34.129+0000 7fd769ee5640 10 start_mon_command cmd=[{"prefix": "get_command_descriptions"}] 2024-06-03T08:36:34.132 INFO:teuthology.orchestra.run.smithi088.stderr:2024-06-03T08:36:34.129+0000 7fd769ee5640 10 monclient: _send_command 1 [{"prefix": "get_command_descriptions"}] 2024-06-03T08:36:34.132 INFO:teuthology.orchestra.run.smithi088.stderr:2024-06-03T08:36:34.129+0000 7fd769ee5640 10 monclient: _send_mon_message to mon.b at v1:172.21.15.192:6789/0 2024-06-03T08:36:34.132 INFO:teuthology.orchestra.run.smithi088.stderr:2024-06-03T08:36:34.129+0000 7fd769ee5640 1 -- 172.21.15.88:0/3191884890 --> v1:172.21.15.192:6789/0 -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fd76413dd90 con 0x7fd76406ee00 2024-06-03T08:36:34.132 INFO:teuthology.orchestra.run.smithi088.stderr:2024-06-03T08:36:34.130+0000 7fd74bfff640 1 -- 172.21.15.88:0/3191884890 <== mon.1 v1:172.21.15.192:6789/0 5 ==== config(0 keys) v1 ==== 4+0+0 (unknown 0 0 0) 0x7fd7540047e0 con 0x7fd76406ee00 2024-06-03T08:36:34.132 INFO:teuthology.orchestra.run.smithi088.stderr:2024-06-03T08:36:34.130+0000 7fd74bfff640 10 monclient: handle_config config(0 keys) v1 2024-06-03T08:36:34.132 INFO:teuthology.orchestra.run.smithi088.stderr:2024-06-03T08:36:34.130+0000 7fd761ffb640 4 set_mon_vals no callback set 2024-06-03T08:36:34.144 INFO:teuthology.orchestra.run.smithi088.stderr:2024-06-03T08:36:34.143+0000 7fd74bfff640 1 -- 172.21.15.88:0/3191884890 <== mon.1 v1:172.21.15.192:6789/0 6 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 2234374065 0 0) 0x7fd754004af0 con 0x7fd76406ee00 2024-06-03T08:36:34.144 INFO:teuthology.orchestra.run.smithi088.stderr:2024-06-03T08:36:34.143+0000 7fd74bfff640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-06-03T08:36:34.144 INFO:teuthology.orchestra.run.smithi088.stderr:2024-06-03T08:36:34.143+0000 7fd74bfff640 10 monclient: got monmap 1 from mon.b (according to old e1) 2024-06-03T08:36:34.144 INFO:teuthology.orchestra.run.smithi088.stderr:2024-06-03T08:36:34.143+0000 7fd74bfff640 10 monclient: dump: 2024-06-03T08:36:34.144 INFO:teuthology.orchestra.run.smithi088.stderr:epoch 1 2024-06-03T08:36:34.144 INFO:teuthology.orchestra.run.smithi088.stderr:fsid ec3c3fdd-3ac6-4d7d-9412-683130ed8b04 2024-06-03T08:36:34.144 INFO:teuthology.orchestra.run.smithi088.stderr:last_changed 2024-06-03T08:34:31.885125+0000 2024-06-03T08:36:34.144 INFO:teuthology.orchestra.run.smithi088.stderr:created 2024-06-03T08:34:31.885125+0000 2024-06-03T08:36:34.144 INFO:teuthology.orchestra.run.smithi088.stderr:min_mon_release 18 (reef) 2024-06-03T08:36:34.144 INFO:teuthology.orchestra.run.smithi088.stderr:election_strategy: 3 2024-06-03T08:36:34.144 INFO:teuthology.orchestra.run.smithi088.stderr:0: v1:172.21.15.88:6789/0 mon.a 2024-06-03T08:36:34.144 INFO:teuthology.orchestra.run.smithi088.stderr:1: v1:172.21.15.192:6789/0 mon.b 2024-06-03T08:36:34.145 INFO:teuthology.orchestra.run.smithi088.stderr:2: v1:172.21.15.88:6790/0 mon.c 2024-06-03T08:36:34.145 INFO:teuthology.orchestra.run.smithi088.stderr: 2024-06-03T08:36:34.145 INFO:teuthology.orchestra.run.smithi088.stderr:2024-06-03T08:36:34.144+0000 7fd74bfff640 1 -- 172.21.15.88:0/3191884890 <== mon.1 v1:172.21.15.192:6789/0 7 ==== mgrmap(e 5) v1 ==== 89110+0+0 (unknown 873533973 0 0) 0x7fd75401a870 con 0x7fd76406ee00 2024-06-03T08:36:34.145 INFO:teuthology.orchestra.run.smithi088.stderr:2024-06-03T08:36:34.145+0000 7fd762ffd640 10 monclient: get_auth_request con 0x7fd75006d3d0 auth_method 0 2024-06-03T08:36:34.145 INFO:teuthology.orchestra.run.smithi088.stderr:2024-06-03T08:36:34.145+0000 7fd74bfff640 10 client.4369.objecter ms_handle_connect 0x7fd75006d3d0 2024-06-03T08:36:34.147 INFO:teuthology.orchestra.run.smithi088.stderr:2024-06-03T08:36:34.147+0000 7fd74bfff640 1 -- 172.21.15.88:0/3191884890 <== mon.1 v1:172.21.15.192:6789/0 8 ==== osd_map(26..26 src has 1..26) v4 ==== 4090+0+0 (unknown 3637450554 0 0) 0x7fd7540540e0 con 0x7fd76406ee00 2024-06-03T08:36:34.147 INFO:teuthology.orchestra.run.smithi088.stderr:2024-06-03T08:36:34.147+0000 7fd74bfff640 10 client.4369.objecter ms_dispatch 0x7fd764000b90 osd_map(26..26 src has 1..26) v4 2024-06-03T08:36:34.147 INFO:teuthology.orchestra.run.smithi088.stderr:2024-06-03T08:36:34.147+0000 7fd74bfff640 3 client.4369.objecter handle_osd_map got epochs [26,26] > 0 2024-06-03T08:36:34.147 INFO:teuthology.orchestra.run.smithi088.stderr:2024-06-03T08:36:34.147+0000 7fd74bfff640 3 client.4369.objecter handle_osd_map decoding full epoch 26 2024-06-03T08:36:34.147 INFO:teuthology.orchestra.run.smithi088.stderr:2024-06-03T08:36:34.147+0000 7fd74bfff640 20 client.4369.objecter dump_active .. 0 homeless 2024-06-03T08:36:34.282 DEBUG:teuthology.orchestra.run.smithi088:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd last-stat-seq osd.1 2024-06-03T08:36:34.318 DEBUG:teuthology.orchestra.run.smithi088:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd last-stat-seq osd.7 2024-06-03T08:36:34.382 INFO:teuthology.orchestra.run.smithi088.stderr:2024-06-03T08:36:34.382+0000 7fd74bfff640 1 -- 172.21.15.88:0/3191884890 <== mon.1 v1:172.21.15.192:6789/0 9 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+182498 (unknown 1092875540 0 3016760700) 0x7fd75404fac0 con 0x7fd76406ee00 2024-06-03T08:36:34.383 INFO:teuthology.orchestra.run.smithi088.stderr:2024-06-03T08:36:34.382+0000 7fd74bfff640 10 monclient: handle_mon_command_ack 1 [{"prefix": "get_command_descriptions"}] 2024-06-03T08:36:34.383 INFO:teuthology.orchestra.run.smithi088.stderr:2024-06-03T08:36:34.382+0000 7fd74bfff640 10 monclient: _finish_command 1 = system:0 2024-06-03T08:36:34.385 INFO:teuthology.orchestra.run.smithi088.stderr:2024-06-03T08:36:34.385+0000 7f293a658640 10 monclient: get_monmap_and_config 2024-06-03T08:36:34.385 INFO:teuthology.orchestra.run.smithi088.stderr:2024-06-03T08:36:34.385+0000 7f293a658640 10 monclient: build_initial_monmap 2024-06-03T08:36:34.385 INFO:teuthology.orchestra.run.smithi088.stderr:2024-06-03T08:36:34.385+0000 7f293a658640 10 monclient: monmap: 2024-06-03T08:36:34.385 INFO:teuthology.orchestra.run.smithi088.stderr:epoch 0 2024-06-03T08:36:34.385 INFO:teuthology.orchestra.run.smithi088.stderr:fsid ec3c3fdd-3ac6-4d7d-9412-683130ed8b04 2024-06-03T08:36:34.386 INFO:teuthology.orchestra.run.smithi088.stderr:last_changed 2024-06-03T08:36:34.386074+0000 2024-06-03T08:36:34.386 INFO:teuthology.orchestra.run.smithi088.stderr:created 2024-06-03T08:36:34.386074+0000 2024-06-03T08:36:34.386 INFO:teuthology.orchestra.run.smithi088.stderr:min_mon_release 0 (unknown) 2024-06-03T08:36:34.386 INFO:teuthology.orchestra.run.smithi088.stderr:election_strategy: 3 2024-06-03T08:36:34.386 INFO:teuthology.orchestra.run.smithi088.stderr:0: v1:172.21.15.88:6789/0 mon.noname-a 2024-06-03T08:36:34.386 INFO:teuthology.orchestra.run.smithi088.stderr:1: v1:172.21.15.192:6789/0 mon.noname-c 2024-06-03T08:36:34.386 INFO:teuthology.orchestra.run.smithi088.stderr:2: v1:172.21.15.88:6790/0 mon.noname-b 2024-06-03T08:36:34.386 INFO:teuthology.orchestra.run.smithi088.stderr: 2024-06-03T08:36:34.387 INFO:teuthology.orchestra.run.smithi088.stderr:2024-06-03T08:36:34.386+0000 7f293a658640 1 Processor -- start 2024-06-03T08:36:34.387 INFO:teuthology.orchestra.run.smithi088.stderr:2024-06-03T08:36:34.386+0000 7f293a658640 1 -- start start 2024-06-03T08:36:34.387 INFO:teuthology.orchestra.run.smithi088.stderr:2024-06-03T08:36:34.386+0000 7f293a658640 10 monclient: init 2024-06-03T08:36:34.388 INFO:teuthology.orchestra.run.smithi088.stderr:2024-06-03T08:36:34.386+0000 7f293a658640 10 monclient: _reopen_session rank -1 2024-06-03T08:36:34.388 INFO:teuthology.orchestra.run.smithi088.stderr:2024-06-03T08:36:34.386+0000 7f293a658640 10 monclient: _add_conns ranks=[0,1,2] 2024-06-03T08:36:34.388 INFO:teuthology.orchestra.run.smithi088.stderr:2024-06-03T08:36:34.386+0000 7f293a658640 10 monclient(hunting): picked mon.noname-a con 0x7f29340b4250 addr v1:172.21.15.88:6789/0 2024-06-03T08:36:34.388 INFO:teuthology.orchestra.run.smithi088.stderr:2024-06-03T08:36:34.386+0000 7f293a658640 10 monclient(hunting): picked mon.noname-c con 0x7f29340a89f0 addr v1:172.21.15.192:6789/0 2024-06-03T08:36:34.388 INFO:teuthology.orchestra.run.smithi088.stderr:2024-06-03T08:36:34.386+0000 7f293a658640 10 monclient(hunting): picked mon.noname-b con 0x7f29341451c0 addr v1:172.21.15.88:6790/0 2024-06-03T08:36:34.388 INFO:teuthology.orchestra.run.smithi088.stderr:2024-06-03T08:36:34.386+0000 7f293a658640 1 -- --> v1:172.21.15.88:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f29340aa6c0 con 0x7f29340b4250 2024-06-03T08:36:34.388 INFO:teuthology.orchestra.run.smithi088.stderr:2024-06-03T08:36:34.386+0000 7f293a658640 1 -- --> v1:172.21.15.192:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f29340a8ed0 con 0x7f29340a89f0 2024-06-03T08:36:34.388 INFO:teuthology.orchestra.run.smithi088.stderr:2024-06-03T08:36:34.386+0000 7f293a658640 1 -- --> v1:172.21.15.88:6790/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f293413f0f0 con 0x7f29341451c0 2024-06-03T08:36:34.388 INFO:teuthology.orchestra.run.smithi088.stderr:2024-06-03T08:36:34.386+0000 7f293a658640 10 monclient(hunting): _renew_subs 2024-06-03T08:36:34.388 INFO:teuthology.orchestra.run.smithi088.stderr:2024-06-03T08:36:34.386+0000 7f293a658640 10 monclient(hunting): authenticate will time out at 1250.196789s 2024-06-03T08:36:34.388 INFO:teuthology.orchestra.run.smithi088.stderr:2024-06-03T08:36:34.387+0000 7f2933fff640 1 --1- >> v1:172.21.15.88:6789/0 conn(0x7f29340b4250 0x7f293413e9e0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.88:6789/0 says I am v1:172.21.15.88:60528/0 (socket says 172.21.15.88:60528) 2024-06-03T08:36:34.388 INFO:teuthology.orchestra.run.smithi088.stderr:2024-06-03T08:36:34.387+0000 7f2933fff640 1 -- 172.21.15.88:0/3570873325 learned_addr learned my addr 172.21.15.88:0/3570873325 (peer_addr_for_me v1:172.21.15.88:0/0) 2024-06-03T08:36:34.390 INFO:teuthology.orchestra.run.smithi088.stderr:2024-06-03T08:36:34.390+0000 7f29337fe640 1 -- 172.21.15.88:0/3570873325 <== mon.0 v1:172.21.15.88:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 890217411 0 0) 0x7f29340aa6c0 con 0x7f29340b4250 2024-06-03T08:36:34.390 INFO:teuthology.orchestra.run.smithi088.stderr:2024-06-03T08:36:34.390+0000 7f29337fe640 10 monclient(hunting): _negotiate 2024-06-03T08:36:34.390 INFO:teuthology.orchestra.run.smithi088.stderr:2024-06-03T08:36:34.390+0000 7f29337fe640 10 monclient(hunting): _init_auth method 2 2024-06-03T08:36:34.391 INFO:teuthology.orchestra.run.smithi088.stderr:2024-06-03T08:36:34.390+0000 7f29337fe640 10 monclient(hunting): _init_auth creating new auth 2024-06-03T08:36:34.391 INFO:teuthology.orchestra.run.smithi088.stderr:2024-06-03T08:36:34.390+0000 7f29337fe640 10 monclient(hunting): my global_id is 4368 2024-06-03T08:36:34.391 INFO:teuthology.orchestra.run.smithi088.stderr:2024-06-03T08:36:34.390+0000 7f29337fe640 1 -- 172.21.15.88:0/3570873325 --> v1:172.21.15.88:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f2920003520 con 0x7f29340b4250 2024-06-03T08:36:34.391 INFO:teuthology.orchestra.run.smithi088.stderr:2024-06-03T08:36:34.390+0000 7f29337fe640 1 -- 172.21.15.88:0/3570873325 <== mon.2 v1:172.21.15.88:6790/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 2176663035 0 0) 0x7f293413f0f0 con 0x7f29341451c0 2024-06-03T08:36:34.391 INFO:teuthology.orchestra.run.smithi088.stderr:2024-06-03T08:36:34.390+0000 7f29337fe640 10 monclient(hunting): _negotiate 2024-06-03T08:36:34.391 INFO:teuthology.orchestra.run.smithi088.stderr:2024-06-03T08:36:34.390+0000 7f29337fe640 10 monclient(hunting): _init_auth method 2 2024-06-03T08:36:34.391 INFO:teuthology.orchestra.run.smithi088.stderr:2024-06-03T08:36:34.390+0000 7f29337fe640 10 monclient(hunting): _init_auth creating new auth 2024-06-03T08:36:34.391 INFO:teuthology.orchestra.run.smithi088.stderr:2024-06-03T08:36:34.390+0000 7f29337fe640 10 monclient(hunting): my global_id is 4373 2024-06-03T08:36:34.391 INFO:teuthology.orchestra.run.smithi088.stderr:2024-06-03T08:36:34.390+0000 7f29337fe640 1 -- 172.21.15.88:0/3570873325 --> v1:172.21.15.88:6790/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f29340aa6c0 con 0x7f29341451c0 2024-06-03T08:36:34.394 INFO:teuthology.orchestra.run.smithi088.stderr:2024-06-03T08:36:34.393+0000 7f29337fe640 1 -- 172.21.15.88:0/3570873325 <== mon.0 v1:172.21.15.88:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 214+0+0 (unknown 3535638652 0 0) 0x7f2920003520 con 0x7f29340b4250 2024-06-03T08:36:34.394 INFO:teuthology.orchestra.run.smithi088.stderr:2024-06-03T08:36:34.394+0000 7f29337fe640 1 -- 172.21.15.88:0/3570873325 >> v1:172.21.15.88:6790/0 conn(0x7f29341451c0 legacy=0x7f293414ce50 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-06-03T08:36:34.394 INFO:teuthology.orchestra.run.smithi088.stderr:2024-06-03T08:36:34.394+0000 7f29337fe640 1 -- 172.21.15.88:0/3570873325 >> v1:172.21.15.192:6789/0 conn(0x7f29340a89f0 legacy=0x7f2934144ab0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-06-03T08:36:34.394 INFO:teuthology.orchestra.run.smithi088.stderr:2024-06-03T08:36:34.394+0000 7f29337fe640 10 monclient: _finish_hunting 0 2024-06-03T08:36:34.394 INFO:teuthology.orchestra.run.smithi088.stderr:2024-06-03T08:36:34.394+0000 7f29337fe640 1 monclient: found mon.noname-a 2024-06-03T08:36:34.395 INFO:teuthology.orchestra.run.smithi088.stderr:2024-06-03T08:36:34.394+0000 7f29337fe640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-06-03T08:36:34.395 INFO:teuthology.orchestra.run.smithi088.stderr:2024-06-03T08:36:34.394+0000 7f29337fe640 10 monclient: _send_mon_message to mon.noname-a at v1:172.21.15.88:6789/0 2024-06-03T08:36:34.395 INFO:teuthology.orchestra.run.smithi088.stderr:2024-06-03T08:36:34.394+0000 7f29337fe640 1 -- 172.21.15.88:0/3570873325 --> v1:172.21.15.88:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f29340b39a0 con 0x7f29340b4250 2024-06-03T08:36:34.395 INFO:teuthology.orchestra.run.smithi088.stderr:2024-06-03T08:36:34.394+0000 7f29337fe640 10 monclient: _finish_auth 0 2024-06-03T08:36:34.395 INFO:teuthology.orchestra.run.smithi088.stderr:2024-06-03T08:36:34.394+0000 7f29337fe640 10 monclient: _check_auth_tickets 2024-06-03T08:36:34.395 INFO:teuthology.orchestra.run.smithi088.stderr:2024-06-03T08:36:34.394+0000 7f29337fe640 20 monclient: _check_auth_rotating not needed by client.admin 2024-06-03T08:36:34.395 INFO:teuthology.orchestra.run.smithi088.stderr:2024-06-03T08:36:34.394+0000 7f29337fe640 1 -- 172.21.15.88:0/3570873325 <== mon.0 v1:172.21.15.88:6789/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 2234374065 0 0) 0x7f2924001610 con 0x7f29340b4250 2024-06-03T08:36:34.395 INFO:teuthology.orchestra.run.smithi088.stderr:2024-06-03T08:36:34.394+0000 7f293a658640 5 monclient: authenticate success, global_id 4368 2024-06-03T08:36:34.395 INFO:teuthology.orchestra.run.smithi088.stderr:2024-06-03T08:36:34.394+0000 7f293a658640 20 monclient: get_monmap_and_config waiting for monmap|config 2024-06-03T08:36:34.395 INFO:teuthology.orchestra.run.smithi088.stderr:2024-06-03T08:36:34.394+0000 7f29337fe640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-06-03T08:36:34.395 INFO:teuthology.orchestra.run.smithi088.stderr:2024-06-03T08:36:34.394+0000 7f29337fe640 10 monclient: got monmap 1 from mon.noname-a (according to old e1) 2024-06-03T08:36:34.395 INFO:teuthology.orchestra.run.smithi088.stderr:2024-06-03T08:36:34.394+0000 7f29337fe640 10 monclient: dump: 2024-06-03T08:36:34.395 INFO:teuthology.orchestra.run.smithi088.stderr:epoch 1 2024-06-03T08:36:34.395 INFO:teuthology.orchestra.run.smithi088.stderr:fsid ec3c3fdd-3ac6-4d7d-9412-683130ed8b04 2024-06-03T08:36:34.395 INFO:teuthology.orchestra.run.smithi088.stderr:last_changed 2024-06-03T08:34:31.885125+0000 2024-06-03T08:36:34.395 INFO:teuthology.orchestra.run.smithi088.stderr:created 2024-06-03T08:34:31.885125+0000 2024-06-03T08:36:34.396 INFO:teuthology.orchestra.run.smithi088.stderr:min_mon_release 18 (reef) 2024-06-03T08:36:34.396 INFO:teuthology.orchestra.run.smithi088.stderr:election_strategy: 3 2024-06-03T08:36:34.396 INFO:teuthology.orchestra.run.smithi088.stderr:0: v1:172.21.15.88:6789/0 mon.a 2024-06-03T08:36:34.396 INFO:teuthology.orchestra.run.smithi088.stderr:1: v1:172.21.15.192:6789/0 mon.b 2024-06-03T08:36:34.396 INFO:teuthology.orchestra.run.smithi088.stderr:2: v1:172.21.15.88:6790/0 mon.c 2024-06-03T08:36:34.396 INFO:teuthology.orchestra.run.smithi088.stderr: 2024-06-03T08:36:34.396 INFO:teuthology.orchestra.run.smithi088.stderr:2024-06-03T08:36:34.394+0000 7f293a658640 20 monclient: get_monmap_and_config waiting for monmap|config 2024-06-03T08:36:34.397 INFO:teuthology.orchestra.run.smithi088.stderr:2024-06-03T08:36:34.395+0000 7f29337fe640 1 -- 172.21.15.88:0/3570873325 <== mon.0 v1:172.21.15.88:6789/0 4 ==== config(0 keys) v1 ==== 4+0+0 (unknown 0 0 0) 0x7f2924001830 con 0x7f29340b4250 2024-06-03T08:36:34.397 INFO:teuthology.orchestra.run.smithi088.stderr:2024-06-03T08:36:34.395+0000 7f29337fe640 10 monclient: handle_config config(0 keys) v1 2024-06-03T08:36:34.397 INFO:teuthology.orchestra.run.smithi088.stderr:2024-06-03T08:36:34.395+0000 7f29337fe640 1 -- 172.21.15.88:0/3570873325 <== mon.0 v1:172.21.15.88:6789/0 5 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 2234374065 0 0) 0x7f2924001c00 con 0x7f29340b4250 2024-06-03T08:36:34.397 INFO:teuthology.orchestra.run.smithi088.stderr:2024-06-03T08:36:34.395+0000 7f29337fe640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-06-03T08:36:34.397 INFO:teuthology.orchestra.run.smithi088.stderr:2024-06-03T08:36:34.396+0000 7f29337fe640 10 monclient: got monmap 1 from mon.a (according to old e1) 2024-06-03T08:36:34.397 INFO:teuthology.orchestra.run.smithi088.stderr:2024-06-03T08:36:34.396+0000 7f29337fe640 10 monclient: dump: 2024-06-03T08:36:34.397 INFO:teuthology.orchestra.run.smithi088.stderr:epoch 1 2024-06-03T08:36:34.397 INFO:teuthology.orchestra.run.smithi088.stderr:fsid ec3c3fdd-3ac6-4d7d-9412-683130ed8b04 2024-06-03T08:36:34.397 INFO:teuthology.orchestra.run.smithi088.stderr:last_changed 2024-06-03T08:34:31.885125+0000 2024-06-03T08:36:34.397 INFO:teuthology.orchestra.run.smithi088.stderr:created 2024-06-03T08:34:31.885125+0000 2024-06-03T08:36:34.397 INFO:teuthology.orchestra.run.smithi088.stderr:min_mon_release 18 (reef) 2024-06-03T08:36:34.397 INFO:teuthology.orchestra.run.smithi088.stderr:election_strategy: 3 2024-06-03T08:36:34.397 INFO:teuthology.orchestra.run.smithi088.stderr:0: v1:172.21.15.88:6789/0 mon.a 2024-06-03T08:36:34.398 INFO:teuthology.orchestra.run.smithi088.stderr:1: v1:172.21.15.192:6789/0 mon.b 2024-06-03T08:36:34.398 INFO:teuthology.orchestra.run.smithi088.stderr:2: v1:172.21.15.88:6790/0 mon.c 2024-06-03T08:36:34.398 INFO:teuthology.orchestra.run.smithi088.stderr: 2024-06-03T08:36:34.398 INFO:teuthology.orchestra.run.smithi088.stderr:2024-06-03T08:36:34.396+0000 7f293a658640 10 monclient: get_monmap_and_config success 2024-06-03T08:36:34.398 INFO:teuthology.orchestra.run.smithi088.stderr:2024-06-03T08:36:34.396+0000 7f293a658640 4 set_mon_vals no callback set 2024-06-03T08:36:34.398 INFO:teuthology.orchestra.run.smithi088.stderr:2024-06-03T08:36:34.396+0000 7f293a658640 10 monclient: shutdown 2024-06-03T08:36:34.398 INFO:teuthology.orchestra.run.smithi088.stderr:2024-06-03T08:36:34.396+0000 7f293a658640 20 monclient: shutdown discarding 0 pending message(s) 2024-06-03T08:36:34.398 INFO:teuthology.orchestra.run.smithi088.stderr:2024-06-03T08:36:34.396+0000 7f293a658640 1 -- 172.21.15.88:0/3570873325 >> v1:172.21.15.88:6789/0 conn(0x7f29340b4250 legacy=0x7f293413e9e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-06-03T08:36:34.398 INFO:teuthology.orchestra.run.smithi088.stderr:2024-06-03T08:36:34.396+0000 7f293a658640 1 -- 172.21.15.88:0/3570873325 shutdown_connections 2024-06-03T08:36:34.398 INFO:teuthology.orchestra.run.smithi088.stderr:2024-06-03T08:36:34.396+0000 7f293a658640 1 -- 172.21.15.88:0/3570873325 >> 172.21.15.88:0/3570873325 conn(0x7f29340717f0 msgr2=0x7f2934071c00 unknown :-1 s=STATE_NONE l=0).mark_down 2024-06-03T08:36:34.398 INFO:teuthology.orchestra.run.smithi088.stderr:2024-06-03T08:36:34.396+0000 7f293a658640 1 -- 172.21.15.88:0/3570873325 shutdown_connections 2024-06-03T08:36:34.398 INFO:teuthology.orchestra.run.smithi088.stderr:2024-06-03T08:36:34.396+0000 7f293a658640 1 -- 172.21.15.88:0/3570873325 wait complete. 2024-06-03T08:36:34.400 INFO:teuthology.orchestra.run.smithi088.stderr:2024-06-03T08:36:34.397+0000 7f293a658640 10 monclient: build_initial_monmap 2024-06-03T08:36:34.400 INFO:teuthology.orchestra.run.smithi088.stderr:2024-06-03T08:36:34.397+0000 7f293a658640 10 monclient: monmap: 2024-06-03T08:36:34.400 INFO:teuthology.orchestra.run.smithi088.stderr:epoch 0 2024-06-03T08:36:34.400 INFO:teuthology.orchestra.run.smithi088.stderr:fsid 00000000-0000-0000-0000-000000000000 2024-06-03T08:36:34.401 INFO:teuthology.orchestra.run.smithi088.stderr:last_changed 0.000000 2024-06-03T08:36:34.401 INFO:teuthology.orchestra.run.smithi088.stderr:created 0.000000 2024-06-03T08:36:34.401 INFO:teuthology.orchestra.run.smithi088.stderr:min_mon_release 0 (unknown) 2024-06-03T08:36:34.401 INFO:teuthology.orchestra.run.smithi088.stderr:election_strategy: 1 2024-06-03T08:36:34.401 INFO:teuthology.orchestra.run.smithi088.stderr:0: v1:172.21.15.88:6789/0 mon.noname-a 2024-06-03T08:36:34.401 INFO:teuthology.orchestra.run.smithi088.stderr:1: v1:172.21.15.192:6789/0 mon.noname-b 2024-06-03T08:36:34.401 INFO:teuthology.orchestra.run.smithi088.stderr:2: v1:172.21.15.88:6790/0 mon.noname-c 2024-06-03T08:36:34.401 INFO:teuthology.orchestra.run.smithi088.stderr: 2024-06-03T08:36:34.401 INFO:teuthology.orchestra.run.smithi088.stderr:2024-06-03T08:36:34.397+0000 7f293a658640 1 librados: starting msgr at 2024-06-03T08:36:34.401 INFO:teuthology.orchestra.run.smithi088.stderr:2024-06-03T08:36:34.397+0000 7f293a658640 1 librados: starting objecter 2024-06-03T08:36:34.401 INFO:teuthology.orchestra.run.smithi088.stderr:2024-06-03T08:36:34.397+0000 7f293a658640 1 Processor -- start 2024-06-03T08:36:34.401 INFO:teuthology.orchestra.run.smithi088.stderr:2024-06-03T08:36:34.397+0000 7f293a658640 1 -- start start 2024-06-03T08:36:34.401 INFO:teuthology.orchestra.run.smithi088.stderr:2024-06-03T08:36:34.397+0000 7f293a658640 1 librados: setting wanted keys 2024-06-03T08:36:34.402 INFO:teuthology.orchestra.run.smithi088.stderr:2024-06-03T08:36:34.397+0000 7f293a658640 1 librados: calling monclient init 2024-06-03T08:36:34.402 INFO:teuthology.orchestra.run.smithi088.stderr:2024-06-03T08:36:34.397+0000 7f293a658640 10 monclient: init 2024-06-03T08:36:34.402 INFO:teuthology.orchestra.run.smithi088.stderr:2024-06-03T08:36:34.398+0000 7f293a658640 10 monclient: _reopen_session rank -1 2024-06-03T08:36:34.402 INFO:teuthology.orchestra.run.smithi088.stderr:2024-06-03T08:36:34.398+0000 7f293a658640 10 monclient: _add_conns ranks=[1,2,0] 2024-06-03T08:36:34.402 INFO:teuthology.orchestra.run.smithi088.stderr:2024-06-03T08:36:34.398+0000 7f293a658640 10 monclient(hunting): picked mon.noname-b con 0x7f29340a89f0 addr v1:172.21.15.192:6789/0 2024-06-03T08:36:34.402 INFO:teuthology.orchestra.run.smithi088.stderr:2024-06-03T08:36:34.398+0000 7f293a658640 10 monclient(hunting): picked mon.noname-c con 0x7f29340b4250 addr v1:172.21.15.88:6790/0 2024-06-03T08:36:34.402 INFO:teuthology.orchestra.run.smithi088.stderr:2024-06-03T08:36:34.398+0000 7f293a658640 10 monclient(hunting): picked mon.noname-a con 0x7f29341451c0 addr v1:172.21.15.88:6789/0 2024-06-03T08:36:34.402 INFO:teuthology.orchestra.run.smithi088.stderr:2024-06-03T08:36:34.398+0000 7f293a658640 1 -- --> v1:172.21.15.88:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f29341360c0 con 0x7f29341451c0 2024-06-03T08:36:34.402 INFO:teuthology.orchestra.run.smithi088.stderr:2024-06-03T08:36:34.398+0000 7f293a658640 1 -- --> v1:172.21.15.192:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f2934141590 con 0x7f29340a89f0 2024-06-03T08:36:34.402 INFO:teuthology.orchestra.run.smithi088.stderr:2024-06-03T08:36:34.398+0000 7f293a658640 1 -- --> v1:172.21.15.88:6790/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f293413f930 con 0x7f29340b4250 2024-06-03T08:36:34.402 INFO:teuthology.orchestra.run.smithi088.stderr:2024-06-03T08:36:34.398+0000 7f293a658640 10 monclient(hunting): _renew_subs 2024-06-03T08:36:34.402 INFO:teuthology.orchestra.run.smithi088.stderr:2024-06-03T08:36:34.398+0000 7f293a658640 10 monclient(hunting): authenticate will time out at 1250.208529s 2024-06-03T08:36:34.402 INFO:teuthology.orchestra.run.smithi088.stderr:2024-06-03T08:36:34.399+0000 7f292bfff640 1 --1- >> v1:172.21.15.88:6790/0 conn(0x7f29340b4250 0x7f29341d6b80 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=1).handle_server_banner_and_identify peer v1:172.21.15.88:6790/0 says I am v1:172.21.15.88:54418/0 (socket says 172.21.15.88:54418) 2024-06-03T08:36:34.402 INFO:teuthology.orchestra.run.smithi088.stderr:2024-06-03T08:36:34.399+0000 7f292bfff640 1 -- 172.21.15.88:0/248786529 learned_addr learned my addr 172.21.15.88:0/248786529 (peer_addr_for_me v1:172.21.15.88:0/0) 2024-06-03T08:36:34.403 INFO:teuthology.orchestra.run.smithi088.stderr:2024-06-03T08:36:34.399+0000 7f29317fa640 10 client.?.objecter ms_handle_connect 0x7f29340b4250 2024-06-03T08:36:34.403 INFO:teuthology.orchestra.run.smithi088.stderr:2024-06-03T08:36:34.399+0000 7f29317fa640 10 client.?.objecter resend_mon_ops 2024-06-03T08:36:34.403 INFO:teuthology.orchestra.run.smithi088.stderr:2024-06-03T08:36:34.399+0000 7f29317fa640 10 client.?.objecter ms_handle_connect 0x7f29340a89f0 2024-06-03T08:36:34.403 INFO:teuthology.orchestra.run.smithi088.stderr:2024-06-03T08:36:34.399+0000 7f29317fa640 10 client.?.objecter resend_mon_ops 2024-06-03T08:36:34.403 INFO:teuthology.orchestra.run.smithi088.stderr:2024-06-03T08:36:34.401+0000 7f29317fa640 1 -- 172.21.15.88:0/248786529 <== mon.2 v1:172.21.15.88:6790/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 3249459241 0 0) 0x7f293413f930 con 0x7f29340b4250 2024-06-03T08:36:34.403 INFO:teuthology.orchestra.run.smithi088.stderr:2024-06-03T08:36:34.401+0000 7f29317fa640 10 monclient(hunting): _negotiate 2024-06-03T08:36:34.403 INFO:teuthology.orchestra.run.smithi088.stderr:2024-06-03T08:36:34.401+0000 7f29317fa640 10 monclient(hunting): _init_auth method 2 2024-06-03T08:36:34.404 INFO:teuthology.orchestra.run.smithi088.stderr:2024-06-03T08:36:34.401+0000 7f29317fa640 10 monclient(hunting): _init_auth creating new auth 2024-06-03T08:36:34.404 INFO:teuthology.orchestra.run.smithi088.stderr:2024-06-03T08:36:34.401+0000 7f29317fa640 10 monclient(hunting): my global_id is 4376 2024-06-03T08:36:34.404 INFO:teuthology.orchestra.run.smithi088.stderr:2024-06-03T08:36:34.401+0000 7f29317fa640 1 -- 172.21.15.88:0/248786529 --> v1:172.21.15.88:6790/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f2914004eb0 con 0x7f29340b4250 2024-06-03T08:36:34.404 INFO:teuthology.orchestra.run.smithi088.stderr:2024-06-03T08:36:34.401+0000 7f29317fa640 1 -- 172.21.15.88:0/248786529 <== mon.1 v1:172.21.15.192:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 494240915 0 0) 0x7f2934141590 con 0x7f29340a89f0 2024-06-03T08:36:34.404 INFO:teuthology.orchestra.run.smithi088.stderr:2024-06-03T08:36:34.401+0000 7f29317fa640 10 monclient(hunting): _negotiate 2024-06-03T08:36:34.404 INFO:teuthology.orchestra.run.smithi088.stderr:2024-06-03T08:36:34.401+0000 7f29317fa640 10 monclient(hunting): _init_auth method 2 2024-06-03T08:36:34.404 INFO:teuthology.orchestra.run.smithi088.stderr:2024-06-03T08:36:34.401+0000 7f29317fa640 10 monclient(hunting): _init_auth creating new auth 2024-06-03T08:36:34.404 INFO:teuthology.orchestra.run.smithi088.stderr:2024-06-03T08:36:34.401+0000 7f29317fa640 10 monclient(hunting): my global_id is 4372 2024-06-03T08:36:34.404 INFO:teuthology.orchestra.run.smithi088.stderr:2024-06-03T08:36:34.401+0000 7f29317fa640 1 -- 172.21.15.88:0/248786529 --> v1:172.21.15.192:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f293413f930 con 0x7f29340a89f0 2024-06-03T08:36:34.406 INFO:teuthology.orchestra.run.smithi088.stderr:2024-06-03T08:36:34.406+0000 7f29317fa640 1 -- 172.21.15.88:0/248786529 <== mon.1 v1:172.21.15.192:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 764+0+0 (unknown 3873879136 0 0) 0x7f293413f930 con 0x7f29340a89f0 2024-06-03T08:36:34.407 INFO:teuthology.orchestra.run.smithi088.stderr:2024-06-03T08:36:34.406+0000 7f29317fa640 1 -- 172.21.15.88:0/248786529 --> v1:172.21.15.192:6789/0 -- auth(proto 2 165 bytes epoch 0) v1 -- 0x7f2934141590 con 0x7f29340a89f0 2024-06-03T08:36:34.407 INFO:teuthology.orchestra.run.smithi088.stderr:2024-06-03T08:36:34.406+0000 7f29317fa640 1 -- 172.21.15.88:0/248786529 <== mon.1 v1:172.21.15.192:6789/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 2234374065 0 0) 0x7f2924001b40 con 0x7f29340a89f0 2024-06-03T08:36:34.407 INFO:teuthology.orchestra.run.smithi088.stderr:2024-06-03T08:36:34.406+0000 7f29317fa640 10 monclient(hunting): handle_monmap mon_map magic: 0 v1 2024-06-03T08:36:34.407 INFO:teuthology.orchestra.run.smithi088.stderr:2024-06-03T08:36:34.406+0000 7f29317fa640 10 monclient(hunting): got monmap 1 from mon.noname-b (according to old e1) 2024-06-03T08:36:34.407 INFO:teuthology.orchestra.run.smithi088.stderr:2024-06-03T08:36:34.406+0000 7f29317fa640 10 monclient(hunting): dump: 2024-06-03T08:36:34.407 INFO:teuthology.orchestra.run.smithi088.stderr:epoch 1 2024-06-03T08:36:34.407 INFO:teuthology.orchestra.run.smithi088.stderr:fsid ec3c3fdd-3ac6-4d7d-9412-683130ed8b04 2024-06-03T08:36:34.407 INFO:teuthology.orchestra.run.smithi088.stderr:last_changed 2024-06-03T08:34:31.885125+0000 2024-06-03T08:36:34.407 INFO:teuthology.orchestra.run.smithi088.stderr:created 2024-06-03T08:34:31.885125+0000 2024-06-03T08:36:34.407 INFO:teuthology.orchestra.run.smithi088.stderr:min_mon_release 18 (reef) 2024-06-03T08:36:34.407 INFO:teuthology.orchestra.run.smithi088.stderr:election_strategy: 3 2024-06-03T08:36:34.407 INFO:teuthology.orchestra.run.smithi088.stderr:0: v1:172.21.15.88:6789/0 mon.a 2024-06-03T08:36:34.407 INFO:teuthology.orchestra.run.smithi088.stderr:1: v1:172.21.15.192:6789/0 mon.b 2024-06-03T08:36:34.408 INFO:teuthology.orchestra.run.smithi088.stderr:2: v1:172.21.15.88:6790/0 mon.c 2024-06-03T08:36:34.408 INFO:teuthology.orchestra.run.smithi088.stderr: 2024-06-03T08:36:34.408 INFO:teuthology.orchestra.run.smithi088.stderr:2024-06-03T08:36:34.406+0000 7f29317fa640 10 monclient(hunting): _finish_auth 0 2024-06-03T08:36:34.408 INFO:teuthology.orchestra.run.smithi088.stderr:2024-06-03T08:36:34.406+0000 7f29317fa640 1 -- 172.21.15.88:0/248786529 <== mon.2 v1:172.21.15.88:6790/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 764+0+0 (unknown 4054088800 0 0) 0x7f2914004eb0 con 0x7f29340b4250 2024-06-03T08:36:34.408 INFO:teuthology.orchestra.run.smithi088.stderr:2024-06-03T08:36:34.406+0000 7f29317fa640 1 -- 172.21.15.88:0/248786529 --> v1:172.21.15.88:6790/0 -- auth(proto 2 165 bytes epoch 0) v1 -- 0x7f293413f930 con 0x7f29340b4250 2024-06-03T08:36:34.408 INFO:teuthology.orchestra.run.smithi088.stderr:2024-06-03T08:36:34.406+0000 7f29317fa640 1 -- 172.21.15.88:0/248786529 <== mon.2 v1:172.21.15.88:6790/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 2234374065 0 0) 0x7f291c001750 con 0x7f29340b4250 2024-06-03T08:36:34.408 INFO:teuthology.orchestra.run.smithi088.stderr:2024-06-03T08:36:34.406+0000 7f29317fa640 10 monclient(hunting): handle_monmap mon_map magic: 0 v1 2024-06-03T08:36:34.408 INFO:teuthology.orchestra.run.smithi088.stderr:2024-06-03T08:36:34.406+0000 7f29317fa640 10 monclient(hunting): got monmap 1 from mon.c (according to old e1) 2024-06-03T08:36:34.408 INFO:teuthology.orchestra.run.smithi088.stderr:2024-06-03T08:36:34.406+0000 7f29317fa640 10 monclient(hunting): dump: 2024-06-03T08:36:34.408 INFO:teuthology.orchestra.run.smithi088.stderr:epoch 1 2024-06-03T08:36:34.408 INFO:teuthology.orchestra.run.smithi088.stderr:fsid ec3c3fdd-3ac6-4d7d-9412-683130ed8b04 2024-06-03T08:36:34.408 INFO:teuthology.orchestra.run.smithi088.stderr:last_changed 2024-06-03T08:34:31.885125+0000 2024-06-03T08:36:34.408 INFO:teuthology.orchestra.run.smithi088.stderr:created 2024-06-03T08:34:31.885125+0000 2024-06-03T08:36:34.408 INFO:teuthology.orchestra.run.smithi088.stderr:min_mon_release 18 (reef) 2024-06-03T08:36:34.409 INFO:teuthology.orchestra.run.smithi088.stderr:election_strategy: 3 2024-06-03T08:36:34.409 INFO:teuthology.orchestra.run.smithi088.stderr:0: v1:172.21.15.88:6789/0 mon.a 2024-06-03T08:36:34.409 INFO:teuthology.orchestra.run.smithi088.stderr:1: v1:172.21.15.192:6789/0 mon.b 2024-06-03T08:36:34.409 INFO:teuthology.orchestra.run.smithi088.stderr:2: v1:172.21.15.88:6790/0 mon.c 2024-06-03T08:36:34.409 INFO:teuthology.orchestra.run.smithi088.stderr: 2024-06-03T08:36:34.410 INFO:teuthology.orchestra.run.smithi088.stderr:2024-06-03T08:36:34.409+0000 7f29317fa640 1 -- 172.21.15.88:0/248786529 <== mon.1 v1:172.21.15.192:6789/0 4 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 580+0+0 (unknown 1534551178 0 0) 0x7f2934141590 con 0x7f29340a89f0 2024-06-03T08:36:34.410 INFO:teuthology.orchestra.run.smithi088.stderr:2024-06-03T08:36:34.409+0000 7f29317fa640 1 -- 172.21.15.88:0/248786529 >> v1:172.21.15.88:6790/0 conn(0x7f29340b4250 legacy=0x7f29341d6b80 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-06-03T08:36:34.410 INFO:teuthology.orchestra.run.smithi088.stderr:2024-06-03T08:36:34.409+0000 7f29317fa640 1 -- 172.21.15.88:0/248786529 >> v1:172.21.15.88:6789/0 conn(0x7f29341451c0 legacy=0x7f29341e1070 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-06-03T08:36:34.410 INFO:teuthology.orchestra.run.smithi088.stderr:2024-06-03T08:36:34.409+0000 7f29317fa640 10 monclient: _finish_hunting 0 2024-06-03T08:36:34.410 INFO:teuthology.orchestra.run.smithi088.stderr:2024-06-03T08:36:34.409+0000 7f29317fa640 1 monclient: found mon.b 2024-06-03T08:36:34.410 INFO:teuthology.orchestra.run.smithi088.stderr:2024-06-03T08:36:34.409+0000 7f29317fa640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-06-03T08:36:34.410 INFO:teuthology.orchestra.run.smithi088.stderr:2024-06-03T08:36:34.409+0000 7f29317fa640 10 monclient: _send_mon_message to mon.b at v1:172.21.15.192:6789/0 2024-06-03T08:36:34.410 INFO:teuthology.orchestra.run.smithi088.stderr:2024-06-03T08:36:34.409+0000 7f29317fa640 1 -- 172.21.15.88:0/248786529 --> v1:172.21.15.192:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f293413fac0 con 0x7f29340a89f0 2024-06-03T08:36:34.410 INFO:teuthology.orchestra.run.smithi088.stderr:2024-06-03T08:36:34.409+0000 7f29317fa640 10 monclient: _finish_auth 0 2024-06-03T08:36:34.411 INFO:teuthology.orchestra.run.smithi088.stderr:2024-06-03T08:36:34.409+0000 7f29317fa640 10 monclient: _check_auth_tickets 2024-06-03T08:36:34.411 INFO:teuthology.orchestra.run.smithi088.stderr:2024-06-03T08:36:34.409+0000 7f29317fa640 20 monclient: _check_auth_rotating not needed by client.admin 2024-06-03T08:36:34.411 INFO:teuthology.orchestra.run.smithi088.stderr:2024-06-03T08:36:34.409+0000 7f293a658640 5 monclient: authenticate success, global_id 4372 2024-06-03T08:36:34.411 INFO:teuthology.orchestra.run.smithi088.stderr:2024-06-03T08:36:34.409+0000 7f293a658640 10 monclient: _renew_subs 2024-06-03T08:36:34.411 INFO:teuthology.orchestra.run.smithi088.stderr:2024-06-03T08:36:34.409+0000 7f293a658640 10 monclient: _send_mon_message to mon.b at v1:172.21.15.192:6789/0 2024-06-03T08:36:34.411 INFO:teuthology.orchestra.run.smithi088.stderr:2024-06-03T08:36:34.409+0000 7f293a658640 1 -- 172.21.15.88:0/248786529 --> v1:172.21.15.192:6789/0 -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f29341d7290 con 0x7f29340a89f0 2024-06-03T08:36:34.411 INFO:teuthology.orchestra.run.smithi088.stderr:2024-06-03T08:36:34.409+0000 7f293a658640 10 client.4372.objecter _maybe_request_map subscribing (onetime) to next osd map 2024-06-03T08:36:34.411 INFO:teuthology.orchestra.run.smithi088.stderr:2024-06-03T08:36:34.409+0000 7f293a658640 10 monclient: _renew_subs 2024-06-03T08:36:34.411 INFO:teuthology.orchestra.run.smithi088.stderr:2024-06-03T08:36:34.409+0000 7f293a658640 10 monclient: _send_mon_message to mon.b at v1:172.21.15.192:6789/0 2024-06-03T08:36:34.411 INFO:teuthology.orchestra.run.smithi088.stderr:2024-06-03T08:36:34.409+0000 7f293a658640 1 -- 172.21.15.88:0/248786529 --> v1:172.21.15.192:6789/0 -- mon_subscribe({osdmap=0}) v3 -- 0x7f2934134500 con 0x7f29340a89f0 2024-06-03T08:36:34.411 INFO:teuthology.orchestra.run.smithi088.stderr:2024-06-03T08:36:34.409+0000 7f293a658640 1 librados: init done 2024-06-03T08:36:34.412 INFO:teuthology.orchestra.run.smithi088.stderr:2024-06-03T08:36:34.410+0000 7f293a658640 10 start_mon_command cmd=[{"prefix": "get_command_descriptions"}] 2024-06-03T08:36:34.412 INFO:teuthology.orchestra.run.smithi088.stderr:2024-06-03T08:36:34.410+0000 7f293a658640 10 monclient: _send_command 1 [{"prefix": "get_command_descriptions"}] 2024-06-03T08:36:34.412 INFO:teuthology.orchestra.run.smithi088.stderr:2024-06-03T08:36:34.410+0000 7f293a658640 10 monclient: _send_mon_message to mon.b at v1:172.21.15.192:6789/0 2024-06-03T08:36:34.412 INFO:teuthology.orchestra.run.smithi088.stderr:2024-06-03T08:36:34.410+0000 7f293a658640 1 -- 172.21.15.88:0/248786529 --> v1:172.21.15.192:6789/0 -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f29340671b0 con 0x7f29340a89f0 2024-06-03T08:36:34.412 DEBUG:teuthology.orchestra.run.smithi088:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd last-stat-seq osd.3 2024-06-03T08:36:34.424 INFO:teuthology.orchestra.run.smithi088.stderr:2024-06-03T08:36:34.423+0000 7f29317fa640 1 -- 172.21.15.88:0/248786529 <== mon.1 v1:172.21.15.192:6789/0 5 ==== config(0 keys) v1 ==== 4+0+0 (unknown 0 0 0) 0x7f2924004a90 con 0x7f29340a89f0 2024-06-03T08:36:34.424 INFO:teuthology.orchestra.run.smithi088.stderr:2024-06-03T08:36:34.424+0000 7f29317fa640 10 monclient: handle_config config(0 keys) v1 2024-06-03T08:36:34.424 INFO:teuthology.orchestra.run.smithi088.stderr:2024-06-03T08:36:34.424+0000 7f29317fa640 1 -- 172.21.15.88:0/248786529 <== mon.1 v1:172.21.15.192:6789/0 6 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 2234374065 0 0) 0x7f2924004da0 con 0x7f29340a89f0 2024-06-03T08:36:34.424 INFO:teuthology.orchestra.run.smithi088.stderr:2024-06-03T08:36:34.424+0000 7f29317fa640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-06-03T08:36:34.424 INFO:teuthology.orchestra.run.smithi088.stderr:2024-06-03T08:36:34.424+0000 7f29317fa640 10 monclient: got monmap 1 from mon.b (according to old e1) 2024-06-03T08:36:34.424 INFO:teuthology.orchestra.run.smithi088.stderr:2024-06-03T08:36:34.424+0000 7f29317fa640 10 monclient: dump: 2024-06-03T08:36:34.424 INFO:teuthology.orchestra.run.smithi088.stderr:epoch 1 2024-06-03T08:36:34.424 INFO:teuthology.orchestra.run.smithi088.stderr:fsid ec3c3fdd-3ac6-4d7d-9412-683130ed8b04 2024-06-03T08:36:34.425 INFO:teuthology.orchestra.run.smithi088.stderr:last_changed 2024-06-03T08:34:31.885125+0000 2024-06-03T08:36:34.425 INFO:teuthology.orchestra.run.smithi088.stderr:created 2024-06-03T08:34:31.885125+0000 2024-06-03T08:36:34.425 INFO:teuthology.orchestra.run.smithi088.stderr:min_mon_release 18 (reef) 2024-06-03T08:36:34.425 INFO:teuthology.orchestra.run.smithi088.stderr:election_strategy: 3 2024-06-03T08:36:34.425 INFO:teuthology.orchestra.run.smithi088.stderr:0: v1:172.21.15.88:6789/0 mon.a 2024-06-03T08:36:34.425 INFO:teuthology.orchestra.run.smithi088.stderr:1: v1:172.21.15.192:6789/0 mon.b 2024-06-03T08:36:34.425 INFO:teuthology.orchestra.run.smithi088.stderr:2: v1:172.21.15.88:6790/0 mon.c 2024-06-03T08:36:34.425 INFO:teuthology.orchestra.run.smithi088.stderr: 2024-06-03T08:36:34.425 INFO:teuthology.orchestra.run.smithi088.stderr:2024-06-03T08:36:34.424+0000 7f2932ffd640 4 set_mon_vals no callback set 2024-06-03T08:36:34.426 INFO:teuthology.orchestra.run.smithi088.stderr:2024-06-03T08:36:34.424+0000 7f29317fa640 1 -- 172.21.15.88:0/248786529 <== mon.1 v1:172.21.15.192:6789/0 7 ==== mgrmap(e 5) v1 ==== 89110+0+0 (unknown 873533973 0 0) 0x7f292401ab20 con 0x7f29340a89f0 2024-06-03T08:36:34.426 INFO:teuthology.orchestra.run.smithi088.stderr:2024-06-03T08:36:34.425+0000 7f292bfff640 10 monclient: get_auth_request con 0x7f291406d220 auth_method 0 2024-06-03T08:36:34.426 INFO:teuthology.orchestra.run.smithi088.stderr:2024-06-03T08:36:34.425+0000 7f29317fa640 10 client.4372.objecter ms_handle_connect 0x7f291406d220 2024-06-03T08:36:34.428 INFO:teuthology.orchestra.run.smithi088.stderr:2024-06-03T08:36:34.428+0000 7f29317fa640 1 -- 172.21.15.88:0/248786529 <== mon.1 v1:172.21.15.192:6789/0 8 ==== osd_map(26..26 src has 1..26) v4 ==== 4090+0+0 (unknown 3637450554 0 0) 0x7f29240541a0 con 0x7f29340a89f0 2024-06-03T08:36:34.428 INFO:teuthology.orchestra.run.smithi088.stderr:2024-06-03T08:36:34.428+0000 7f29317fa640 10 client.4372.objecter ms_dispatch 0x7f2934000b90 osd_map(26..26 src has 1..26) v4 2024-06-03T08:36:34.428 INFO:teuthology.orchestra.run.smithi088.stderr:2024-06-03T08:36:34.428+0000 7f29317fa640 3 client.4372.objecter handle_osd_map got epochs [26,26] > 0 2024-06-03T08:36:34.428 INFO:teuthology.orchestra.run.smithi088.stderr:2024-06-03T08:36:34.428+0000 7f29317fa640 3 client.4372.objecter handle_osd_map decoding full epoch 26 2024-06-03T08:36:34.428 INFO:teuthology.orchestra.run.smithi088.stderr:2024-06-03T08:36:34.428+0000 7f29317fa640 20 client.4372.objecter dump_active .. 0 homeless 2024-06-03T08:36:34.439 INFO:teuthology.orchestra.run.smithi088.stderr:2024-06-03T08:36:34.439+0000 7fbaefed1640 10 monclient: get_monmap_and_config 2024-06-03T08:36:34.439 INFO:teuthology.orchestra.run.smithi088.stderr:2024-06-03T08:36:34.439+0000 7fbaefed1640 10 monclient: build_initial_monmap 2024-06-03T08:36:34.439 INFO:teuthology.orchestra.run.smithi088.stderr:2024-06-03T08:36:34.439+0000 7fbaefed1640 10 monclient: monmap: 2024-06-03T08:36:34.439 INFO:teuthology.orchestra.run.smithi088.stderr:epoch 0 2024-06-03T08:36:34.439 INFO:teuthology.orchestra.run.smithi088.stderr:fsid ec3c3fdd-3ac6-4d7d-9412-683130ed8b04 2024-06-03T08:36:34.439 INFO:teuthology.orchestra.run.smithi088.stderr:last_changed 2024-06-03T08:36:34.440188+0000 2024-06-03T08:36:34.439 INFO:teuthology.orchestra.run.smithi088.stderr:created 2024-06-03T08:36:34.440188+0000 2024-06-03T08:36:34.439 INFO:teuthology.orchestra.run.smithi088.stderr:min_mon_release 0 (unknown) 2024-06-03T08:36:34.440 INFO:teuthology.orchestra.run.smithi088.stderr:election_strategy: 3 2024-06-03T08:36:34.440 INFO:teuthology.orchestra.run.smithi088.stderr:0: v1:172.21.15.88:6789/0 mon.noname-a 2024-06-03T08:36:34.440 INFO:teuthology.orchestra.run.smithi088.stderr:1: v1:172.21.15.192:6789/0 mon.noname-c 2024-06-03T08:36:34.440 INFO:teuthology.orchestra.run.smithi088.stderr:2: v1:172.21.15.88:6790/0 mon.noname-b 2024-06-03T08:36:34.440 INFO:teuthology.orchestra.run.smithi088.stderr: 2024-06-03T08:36:34.441 INFO:teuthology.orchestra.run.smithi088.stderr:2024-06-03T08:36:34.439+0000 7fbaefed1640 1 Processor -- start 2024-06-03T08:36:34.441 INFO:teuthology.orchestra.run.smithi088.stderr:2024-06-03T08:36:34.440+0000 7fbaefed1640 1 -- start start 2024-06-03T08:36:34.441 INFO:teuthology.orchestra.run.smithi088.stderr:2024-06-03T08:36:34.440+0000 7fbaefed1640 10 monclient: init 2024-06-03T08:36:34.442 INFO:teuthology.orchestra.run.smithi088.stderr:2024-06-03T08:36:34.440+0000 7fbaefed1640 10 monclient: _reopen_session rank -1 2024-06-03T08:36:34.442 INFO:teuthology.orchestra.run.smithi088.stderr:2024-06-03T08:36:34.440+0000 7fbaefed1640 10 monclient: _add_conns ranks=[2,0,1] 2024-06-03T08:36:34.442 INFO:teuthology.orchestra.run.smithi088.stderr:2024-06-03T08:36:34.440+0000 7fbaefed1640 10 monclient(hunting): picked mon.noname-b con 0x7fbae80b4250 addr v1:172.21.15.88:6790/0 2024-06-03T08:36:34.442 INFO:teuthology.orchestra.run.smithi088.stderr:2024-06-03T08:36:34.440+0000 7fbaefed1640 10 monclient(hunting): picked mon.noname-a con 0x7fbae80a89f0 addr v1:172.21.15.88:6789/0 2024-06-03T08:36:34.442 INFO:teuthology.orchestra.run.smithi088.stderr:2024-06-03T08:36:34.440+0000 7fbaefed1640 10 monclient(hunting): picked mon.noname-c con 0x7fbae81451c0 addr v1:172.21.15.192:6789/0 2024-06-03T08:36:34.442 INFO:teuthology.orchestra.run.smithi088.stderr:2024-06-03T08:36:34.440+0000 7fbaefed1640 1 -- --> v1:172.21.15.88:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fbae80aa6c0 con 0x7fbae80a89f0 2024-06-03T08:36:34.442 INFO:teuthology.orchestra.run.smithi088.stderr:2024-06-03T08:36:34.440+0000 7fbaefed1640 1 -- --> v1:172.21.15.192:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fbae80a8ed0 con 0x7fbae81451c0 2024-06-03T08:36:34.442 INFO:teuthology.orchestra.run.smithi088.stderr:2024-06-03T08:36:34.440+0000 7fbaefed1640 1 -- --> v1:172.21.15.88:6790/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fbae813f0f0 con 0x7fbae80b4250 2024-06-03T08:36:34.442 INFO:teuthology.orchestra.run.smithi088.stderr:2024-06-03T08:36:34.440+0000 7fbaefed1640 10 monclient(hunting): _renew_subs 2024-06-03T08:36:34.442 INFO:teuthology.orchestra.run.smithi088.stderr:2024-06-03T08:36:34.440+0000 7fbaefed1640 10 monclient(hunting): authenticate will time out at 1250.250702s 2024-06-03T08:36:34.442 INFO:teuthology.orchestra.run.smithi088.stderr:2024-06-03T08:36:34.441+0000 7fbaed46c640 1 --1- >> v1:172.21.15.88:6789/0 conn(0x7fbae80a89f0 0x7fbae8144ab0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.88:6789/0 says I am v1:172.21.15.88:60542/0 (socket says 172.21.15.88:60542) 2024-06-03T08:36:34.442 INFO:teuthology.orchestra.run.smithi088.stderr:2024-06-03T08:36:34.441+0000 7fbaed46c640 1 -- 172.21.15.88:0/3711412226 learned_addr learned my addr 172.21.15.88:0/3711412226 (peer_addr_for_me v1:172.21.15.88:0/0) 2024-06-03T08:36:34.443 DEBUG:teuthology.orchestra.run.smithi088:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd last-stat-seq osd.0 2024-06-03T08:36:34.444 INFO:teuthology.orchestra.run.smithi088.stderr:2024-06-03T08:36:34.444+0000 7fbaecc6b640 1 -- 172.21.15.88:0/3711412226 <== mon.2 v1:172.21.15.88:6790/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 2724223843 0 0) 0x7fbae813f0f0 con 0x7fbae80b4250 2024-06-03T08:36:34.444 INFO:teuthology.orchestra.run.smithi088.stderr:2024-06-03T08:36:34.444+0000 7fbaecc6b640 10 monclient(hunting): _negotiate 2024-06-03T08:36:34.444 INFO:teuthology.orchestra.run.smithi088.stderr:2024-06-03T08:36:34.444+0000 7fbaecc6b640 10 monclient(hunting): _init_auth method 2 2024-06-03T08:36:34.444 INFO:teuthology.orchestra.run.smithi088.stderr:2024-06-03T08:36:34.444+0000 7fbaecc6b640 10 monclient(hunting): _init_auth creating new auth 2024-06-03T08:36:34.444 INFO:teuthology.orchestra.run.smithi088.stderr:2024-06-03T08:36:34.444+0000 7fbaecc6b640 10 monclient(hunting): my global_id is 4379 2024-06-03T08:36:34.444 INFO:teuthology.orchestra.run.smithi088.stderr:2024-06-03T08:36:34.444+0000 7fbaecc6b640 1 -- 172.21.15.88:0/3711412226 --> v1:172.21.15.88:6790/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7fbad4003520 con 0x7fbae80b4250 2024-06-03T08:36:34.445 INFO:teuthology.orchestra.run.smithi088.stderr:2024-06-03T08:36:34.444+0000 7fbaecc6b640 1 -- 172.21.15.88:0/3711412226 <== mon.0 v1:172.21.15.88:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 966830236 0 0) 0x7fbae80aa6c0 con 0x7fbae80a89f0 2024-06-03T08:36:34.445 INFO:teuthology.orchestra.run.smithi088.stderr:2024-06-03T08:36:34.444+0000 7fbaecc6b640 10 monclient(hunting): _negotiate 2024-06-03T08:36:34.446 INFO:teuthology.orchestra.run.smithi088.stderr:2024-06-03T08:36:34.444+0000 7fbaecc6b640 10 monclient(hunting): _init_auth method 2 2024-06-03T08:36:34.446 INFO:teuthology.orchestra.run.smithi088.stderr:2024-06-03T08:36:34.444+0000 7fbaecc6b640 10 monclient(hunting): _init_auth creating new auth 2024-06-03T08:36:34.446 INFO:teuthology.orchestra.run.smithi088.stderr:2024-06-03T08:36:34.444+0000 7fbaecc6b640 10 monclient(hunting): my global_id is 4371 2024-06-03T08:36:34.446 INFO:teuthology.orchestra.run.smithi088.stderr:2024-06-03T08:36:34.444+0000 7fbaecc6b640 1 -- 172.21.15.88:0/3711412226 --> v1:172.21.15.88:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7fbae813f0f0 con 0x7fbae80a89f0 2024-06-03T08:36:34.449 INFO:teuthology.orchestra.run.smithi088.stderr:2024-06-03T08:36:34.448+0000 7fbaecc6b640 1 -- 172.21.15.88:0/3711412226 <== mon.2 v1:172.21.15.88:6790/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 214+0+0 (unknown 4036361152 0 0) 0x7fbad4003520 con 0x7fbae80b4250 2024-06-03T08:36:34.449 INFO:teuthology.orchestra.run.smithi088.stderr:2024-06-03T08:36:34.448+0000 7fbaecc6b640 1 -- 172.21.15.88:0/3711412226 >> v1:172.21.15.192:6789/0 conn(0x7fbae81451c0 legacy=0x7fbae814ce50 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-06-03T08:36:34.449 INFO:teuthology.orchestra.run.smithi088.stderr:2024-06-03T08:36:34.448+0000 7fbaecc6b640 1 -- 172.21.15.88:0/3711412226 >> v1:172.21.15.88:6789/0 conn(0x7fbae80a89f0 legacy=0x7fbae8144ab0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-06-03T08:36:34.449 INFO:teuthology.orchestra.run.smithi088.stderr:2024-06-03T08:36:34.448+0000 7fbaecc6b640 10 monclient: _finish_hunting 0 2024-06-03T08:36:34.449 INFO:teuthology.orchestra.run.smithi088.stderr:2024-06-03T08:36:34.448+0000 7fbaecc6b640 1 monclient: found mon.noname-b 2024-06-03T08:36:34.449 INFO:teuthology.orchestra.run.smithi088.stderr:2024-06-03T08:36:34.448+0000 7fbaecc6b640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-06-03T08:36:34.449 INFO:teuthology.orchestra.run.smithi088.stderr:2024-06-03T08:36:34.448+0000 7fbaecc6b640 10 monclient: _send_mon_message to mon.noname-b at v1:172.21.15.88:6790/0 2024-06-03T08:36:34.449 INFO:teuthology.orchestra.run.smithi088.stderr:2024-06-03T08:36:34.448+0000 7fbaecc6b640 1 -- 172.21.15.88:0/3711412226 --> v1:172.21.15.88:6790/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fbae80b39a0 con 0x7fbae80b4250 2024-06-03T08:36:34.449 INFO:teuthology.orchestra.run.smithi088.stderr:2024-06-03T08:36:34.448+0000 7fbaecc6b640 10 monclient: _finish_auth 0 2024-06-03T08:36:34.449 INFO:teuthology.orchestra.run.smithi088.stderr:2024-06-03T08:36:34.448+0000 7fbaecc6b640 10 monclient: _check_auth_tickets 2024-06-03T08:36:34.449 INFO:teuthology.orchestra.run.smithi088.stderr:2024-06-03T08:36:34.448+0000 7fbaecc6b640 20 monclient: _check_auth_rotating not needed by client.admin 2024-06-03T08:36:34.449 INFO:teuthology.orchestra.run.smithi088.stderr:2024-06-03T08:36:34.448+0000 7fbaecc6b640 1 -- 172.21.15.88:0/3711412226 <== mon.2 v1:172.21.15.88:6790/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 2234374065 0 0) 0x7fbae4001610 con 0x7fbae80b4250 2024-06-03T08:36:34.449 INFO:teuthology.orchestra.run.smithi088.stderr:2024-06-03T08:36:34.448+0000 7fbaecc6b640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-06-03T08:36:34.450 INFO:teuthology.orchestra.run.smithi088.stderr:2024-06-03T08:36:34.448+0000 7fbaecc6b640 10 monclient: got monmap 1 from mon.noname-b (according to old e1) 2024-06-03T08:36:34.450 INFO:teuthology.orchestra.run.smithi088.stderr:2024-06-03T08:36:34.448+0000 7fbaecc6b640 10 monclient: dump: 2024-06-03T08:36:34.450 INFO:teuthology.orchestra.run.smithi088.stderr:epoch 1 2024-06-03T08:36:34.450 INFO:teuthology.orchestra.run.smithi088.stderr:fsid ec3c3fdd-3ac6-4d7d-9412-683130ed8b04 2024-06-03T08:36:34.450 INFO:teuthology.orchestra.run.smithi088.stderr:last_changed 2024-06-03T08:34:31.885125+0000 2024-06-03T08:36:34.450 INFO:teuthology.orchestra.run.smithi088.stderr:created 2024-06-03T08:34:31.885125+0000 2024-06-03T08:36:34.450 INFO:teuthology.orchestra.run.smithi088.stderr:min_mon_release 18 (reef) 2024-06-03T08:36:34.450 INFO:teuthology.orchestra.run.smithi088.stderr:election_strategy: 3 2024-06-03T08:36:34.450 INFO:teuthology.orchestra.run.smithi088.stderr:0: v1:172.21.15.88:6789/0 mon.a 2024-06-03T08:36:34.450 INFO:teuthology.orchestra.run.smithi088.stderr:1: v1:172.21.15.192:6789/0 mon.b 2024-06-03T08:36:34.450 INFO:teuthology.orchestra.run.smithi088.stderr:2: v1:172.21.15.88:6790/0 mon.c 2024-06-03T08:36:34.450 INFO:teuthology.orchestra.run.smithi088.stderr: 2024-06-03T08:36:34.450 INFO:teuthology.orchestra.run.smithi088.stderr:2024-06-03T08:36:34.448+0000 7fbaefed1640 5 monclient: authenticate success, global_id 4379 2024-06-03T08:36:34.451 INFO:teuthology.orchestra.run.smithi088.stderr:2024-06-03T08:36:34.448+0000 7fbaefed1640 20 monclient: get_monmap_and_config waiting for monmap|config 2024-06-03T08:36:34.452 INFO:teuthology.orchestra.run.smithi088.stderr:2024-06-03T08:36:34.451+0000 7fbaecc6b640 1 -- 172.21.15.88:0/3711412226 <== mon.2 v1:172.21.15.88:6790/0 4 ==== config(0 keys) v1 ==== 4+0+0 (unknown 0 0 0) 0x7fbae4001830 con 0x7fbae80b4250 2024-06-03T08:36:34.452 INFO:teuthology.orchestra.run.smithi088.stderr:2024-06-03T08:36:34.451+0000 7fbaecc6b640 10 monclient: handle_config config(0 keys) v1 2024-06-03T08:36:34.452 INFO:teuthology.orchestra.run.smithi088.stderr:2024-06-03T08:36:34.451+0000 7fbaecc6b640 1 -- 172.21.15.88:0/3711412226 <== mon.2 v1:172.21.15.88:6790/0 5 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 2234374065 0 0) 0x7fbae4001bc0 con 0x7fbae80b4250 2024-06-03T08:36:34.452 INFO:teuthology.orchestra.run.smithi088.stderr:2024-06-03T08:36:34.451+0000 7fbaecc6b640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-06-03T08:36:34.452 INFO:teuthology.orchestra.run.smithi088.stderr:2024-06-03T08:36:34.451+0000 7fbaecc6b640 10 monclient: got monmap 1 from mon.c (according to old e1) 2024-06-03T08:36:34.453 INFO:teuthology.orchestra.run.smithi088.stderr:2024-06-03T08:36:34.451+0000 7fbaecc6b640 10 monclient: dump: 2024-06-03T08:36:34.453 INFO:teuthology.orchestra.run.smithi088.stderr:epoch 1 2024-06-03T08:36:34.453 INFO:teuthology.orchestra.run.smithi088.stderr:fsid ec3c3fdd-3ac6-4d7d-9412-683130ed8b04 2024-06-03T08:36:34.453 INFO:teuthology.orchestra.run.smithi088.stderr:last_changed 2024-06-03T08:34:31.885125+0000 2024-06-03T08:36:34.453 INFO:teuthology.orchestra.run.smithi088.stderr:created 2024-06-03T08:34:31.885125+0000 2024-06-03T08:36:34.453 INFO:teuthology.orchestra.run.smithi088.stderr:min_mon_release 18 (reef) 2024-06-03T08:36:34.453 INFO:teuthology.orchestra.run.smithi088.stderr:election_strategy: 3 2024-06-03T08:36:34.453 INFO:teuthology.orchestra.run.smithi088.stderr:0: v1:172.21.15.88:6789/0 mon.a 2024-06-03T08:36:34.453 INFO:teuthology.orchestra.run.smithi088.stderr:1: v1:172.21.15.192:6789/0 mon.b 2024-06-03T08:36:34.453 INFO:teuthology.orchestra.run.smithi088.stderr:2: v1:172.21.15.88:6790/0 mon.c 2024-06-03T08:36:34.453 INFO:teuthology.orchestra.run.smithi088.stderr: 2024-06-03T08:36:34.453 INFO:teuthology.orchestra.run.smithi088.stderr:2024-06-03T08:36:34.451+0000 7fbaefed1640 10 monclient: get_monmap_and_config success 2024-06-03T08:36:34.453 INFO:teuthology.orchestra.run.smithi088.stderr:2024-06-03T08:36:34.451+0000 7fbaefed1640 4 set_mon_vals no callback set 2024-06-03T08:36:34.454 INFO:teuthology.orchestra.run.smithi088.stderr:2024-06-03T08:36:34.452+0000 7fbaefed1640 10 monclient: shutdown 2024-06-03T08:36:34.455 INFO:teuthology.orchestra.run.smithi088.stderr:2024-06-03T08:36:34.452+0000 7fbaefed1640 20 monclient: shutdown discarding 0 pending message(s) 2024-06-03T08:36:34.455 INFO:teuthology.orchestra.run.smithi088.stderr:2024-06-03T08:36:34.452+0000 7fbaefed1640 1 -- 172.21.15.88:0/3711412226 >> v1:172.21.15.88:6790/0 conn(0x7fbae80b4250 legacy=0x7fbae813e9e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-06-03T08:36:34.455 INFO:teuthology.orchestra.run.smithi088.stderr:2024-06-03T08:36:34.452+0000 7fbaefed1640 1 -- 172.21.15.88:0/3711412226 shutdown_connections 2024-06-03T08:36:34.455 INFO:teuthology.orchestra.run.smithi088.stderr:2024-06-03T08:36:34.453+0000 7fbaefed1640 1 -- 172.21.15.88:0/3711412226 >> 172.21.15.88:0/3711412226 conn(0x7fbae80717f0 msgr2=0x7fbae8071c00 unknown :-1 s=STATE_NONE l=0).mark_down 2024-06-03T08:36:34.455 INFO:teuthology.orchestra.run.smithi088.stderr:2024-06-03T08:36:34.453+0000 7fbaefed1640 1 -- 172.21.15.88:0/3711412226 shutdown_connections 2024-06-03T08:36:34.455 INFO:teuthology.orchestra.run.smithi088.stderr:2024-06-03T08:36:34.453+0000 7fbaefed1640 1 -- 172.21.15.88:0/3711412226 wait complete. 2024-06-03T08:36:34.455 INFO:teuthology.orchestra.run.smithi088.stderr:2024-06-03T08:36:34.454+0000 7fbaefed1640 10 monclient: build_initial_monmap 2024-06-03T08:36:34.455 INFO:teuthology.orchestra.run.smithi088.stderr:2024-06-03T08:36:34.454+0000 7fbaefed1640 10 monclient: monmap: 2024-06-03T08:36:34.455 INFO:teuthology.orchestra.run.smithi088.stderr:epoch 0 2024-06-03T08:36:34.455 INFO:teuthology.orchestra.run.smithi088.stderr:fsid 00000000-0000-0000-0000-000000000000 2024-06-03T08:36:34.455 INFO:teuthology.orchestra.run.smithi088.stderr:last_changed 0.000000 2024-06-03T08:36:34.455 INFO:teuthology.orchestra.run.smithi088.stderr:created 0.000000 2024-06-03T08:36:34.455 INFO:teuthology.orchestra.run.smithi088.stderr:min_mon_release 0 (unknown) 2024-06-03T08:36:34.456 INFO:teuthology.orchestra.run.smithi088.stderr:election_strategy: 1 2024-06-03T08:36:34.456 INFO:teuthology.orchestra.run.smithi088.stderr:0: v1:172.21.15.88:6789/0 mon.noname-a 2024-06-03T08:36:34.456 INFO:teuthology.orchestra.run.smithi088.stderr:1: v1:172.21.15.192:6789/0 mon.noname-b 2024-06-03T08:36:34.456 INFO:teuthology.orchestra.run.smithi088.stderr:2: v1:172.21.15.88:6790/0 mon.noname-c 2024-06-03T08:36:34.456 INFO:teuthology.orchestra.run.smithi088.stderr: 2024-06-03T08:36:34.457 INFO:teuthology.orchestra.run.smithi088.stderr:2024-06-03T08:36:34.455+0000 7fbaefed1640 1 librados: starting msgr at 2024-06-03T08:36:34.457 INFO:teuthology.orchestra.run.smithi088.stderr:2024-06-03T08:36:34.455+0000 7fbaefed1640 1 librados: starting objecter 2024-06-03T08:36:34.457 INFO:teuthology.orchestra.run.smithi088.stderr:2024-06-03T08:36:34.455+0000 7fbaefed1640 1 Processor -- start 2024-06-03T08:36:34.457 INFO:teuthology.orchestra.run.smithi088.stderr:2024-06-03T08:36:34.455+0000 7fbaefed1640 1 -- start start 2024-06-03T08:36:34.457 INFO:teuthology.orchestra.run.smithi088.stderr:2024-06-03T08:36:34.455+0000 7fbaefed1640 1 librados: setting wanted keys 2024-06-03T08:36:34.457 INFO:teuthology.orchestra.run.smithi088.stderr:2024-06-03T08:36:34.455+0000 7fbaefed1640 1 librados: calling monclient init 2024-06-03T08:36:34.457 INFO:teuthology.orchestra.run.smithi088.stderr:2024-06-03T08:36:34.455+0000 7fbaefed1640 10 monclient: init 2024-06-03T08:36:34.457 INFO:teuthology.orchestra.run.smithi088.stderr:2024-06-03T08:36:34.456+0000 7fbaefed1640 10 monclient: _reopen_session rank -1 2024-06-03T08:36:34.457 INFO:teuthology.orchestra.run.smithi088.stderr:2024-06-03T08:36:34.456+0000 7fbaefed1640 10 monclient: _add_conns ranks=[2,0,1] 2024-06-03T08:36:34.458 INFO:teuthology.orchestra.run.smithi088.stderr:2024-06-03T08:36:34.457+0000 7fbaefed1640 10 monclient(hunting): picked mon.noname-c con 0x7fbae80a89f0 addr v1:172.21.15.88:6790/0 2024-06-03T08:36:34.459 INFO:teuthology.orchestra.run.smithi088.stderr:2024-06-03T08:36:34.457+0000 7fbaefed1640 10 monclient(hunting): picked mon.noname-a con 0x7fbae80b4250 addr v1:172.21.15.88:6789/0 2024-06-03T08:36:34.459 INFO:teuthology.orchestra.run.smithi088.stderr:2024-06-03T08:36:34.457+0000 7fbaefed1640 10 monclient(hunting): picked mon.noname-b con 0x7fbae81451c0 addr v1:172.21.15.192:6789/0 2024-06-03T08:36:34.459 INFO:teuthology.orchestra.run.smithi088.stderr:2024-06-03T08:36:34.457+0000 7fbaefed1640 1 -- --> v1:172.21.15.88:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fbae8141700 con 0x7fbae80b4250 2024-06-03T08:36:34.459 INFO:teuthology.orchestra.run.smithi088.stderr:2024-06-03T08:36:34.457+0000 7fbaefed1640 1 -- --> v1:172.21.15.192:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fbae81d4320 con 0x7fbae81451c0 2024-06-03T08:36:34.459 INFO:teuthology.orchestra.run.smithi088.stderr:2024-06-03T08:36:34.457+0000 7fbaedc6d640 1 --1- >> v1:172.21.15.88:6790/0 conn(0x7fbae80a89f0 0x7fbae81d6df0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=1).handle_server_banner_and_identify peer v1:172.21.15.88:6790/0 says I am v1:172.21.15.88:54436/0 (socket says 172.21.15.88:54436) 2024-06-03T08:36:34.459 INFO:teuthology.orchestra.run.smithi088.stderr:2024-06-03T08:36:34.457+0000 7fbaedc6d640 1 -- 172.21.15.88:0/1627903445 learned_addr learned my addr 172.21.15.88:0/1627903445 (peer_addr_for_me v1:172.21.15.88:0/0) 2024-06-03T08:36:34.459 INFO:teuthology.orchestra.run.smithi088.stderr:2024-06-03T08:36:34.458+0000 7fbaefed1640 1 -- 172.21.15.88:0/1627903445 --> v1:172.21.15.88:6790/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fbae81d68d0 con 0x7fbae80a89f0 2024-06-03T08:36:34.459 INFO:teuthology.orchestra.run.smithi088.stderr:2024-06-03T08:36:34.458+0000 7fbaefed1640 10 monclient(hunting): _renew_subs 2024-06-03T08:36:34.459 INFO:teuthology.orchestra.run.smithi088.stderr:2024-06-03T08:36:34.458+0000 7fbaefed1640 10 monclient(hunting): authenticate will time out at 1250.268237s 2024-06-03T08:36:34.459 INFO:teuthology.orchestra.run.smithi088.stderr:2024-06-03T08:36:34.458+0000 7fbade7fc640 10 client.?.objecter ms_handle_connect 0x7fbae80b4250 2024-06-03T08:36:34.459 INFO:teuthology.orchestra.run.smithi088.stderr:2024-06-03T08:36:34.458+0000 7fbade7fc640 10 client.?.objecter resend_mon_ops 2024-06-03T08:36:34.459 INFO:teuthology.orchestra.run.smithi088.stderr:2024-06-03T08:36:34.458+0000 7fbade7fc640 10 client.?.objecter ms_handle_connect 0x7fbae80a89f0 2024-06-03T08:36:34.460 INFO:teuthology.orchestra.run.smithi088.stderr:2024-06-03T08:36:34.458+0000 7fbade7fc640 10 client.?.objecter resend_mon_ops 2024-06-03T08:36:34.460 INFO:teuthology.orchestra.run.smithi088.stderr:2024-06-03T08:36:34.460+0000 7fbade7fc640 1 -- 172.21.15.88:0/1627903445 <== mon.0 v1:172.21.15.88:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 1029999545 0 0) 0x7fbae8141700 con 0x7fbae80b4250 2024-06-03T08:36:34.461 INFO:teuthology.orchestra.run.smithi088.stderr:2024-06-03T08:36:34.460+0000 7fbade7fc640 10 monclient(hunting): _negotiate 2024-06-03T08:36:34.461 INFO:teuthology.orchestra.run.smithi088.stderr:2024-06-03T08:36:34.460+0000 7fbade7fc640 10 monclient(hunting): _init_auth method 2 2024-06-03T08:36:34.461 INFO:teuthology.orchestra.run.smithi088.stderr:2024-06-03T08:36:34.460+0000 7fbade7fc640 10 monclient(hunting): _init_auth creating new auth 2024-06-03T08:36:34.461 INFO:teuthology.orchestra.run.smithi088.stderr:2024-06-03T08:36:34.460+0000 7fbade7fc640 10 monclient(hunting): my global_id is 4374 2024-06-03T08:36:34.461 INFO:teuthology.orchestra.run.smithi088.stderr:2024-06-03T08:36:34.460+0000 7fbade7fc640 1 -- 172.21.15.88:0/1627903445 --> v1:172.21.15.88:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7fbacc004eb0 con 0x7fbae80b4250 2024-06-03T08:36:34.462 INFO:teuthology.orchestra.run.smithi088.stderr:2024-06-03T08:36:34.461+0000 7fbade7fc640 1 -- 172.21.15.88:0/1627903445 <== mon.2 v1:172.21.15.88:6790/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 3225263171 0 0) 0x7fbae81d68d0 con 0x7fbae80a89f0 2024-06-03T08:36:34.462 INFO:teuthology.orchestra.run.smithi088.stderr:2024-06-03T08:36:34.461+0000 7fbade7fc640 10 monclient(hunting): _negotiate 2024-06-03T08:36:34.462 INFO:teuthology.orchestra.run.smithi088.stderr:2024-06-03T08:36:34.461+0000 7fbade7fc640 10 monclient(hunting): _init_auth method 2 2024-06-03T08:36:34.462 INFO:teuthology.orchestra.run.smithi088.stderr:2024-06-03T08:36:34.461+0000 7fbade7fc640 10 monclient(hunting): _init_auth creating new auth 2024-06-03T08:36:34.462 INFO:teuthology.orchestra.run.smithi088.stderr:2024-06-03T08:36:34.461+0000 7fbade7fc640 10 monclient(hunting): my global_id is 4382 2024-06-03T08:36:34.462 INFO:teuthology.orchestra.run.smithi088.stderr:2024-06-03T08:36:34.461+0000 7fbade7fc640 1 -- 172.21.15.88:0/1627903445 --> v1:172.21.15.88:6790/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7fbae8141700 con 0x7fbae80a89f0 2024-06-03T08:36:34.466 INFO:teuthology.orchestra.run.smithi088.stderr:2024-06-03T08:36:34.465+0000 7fbade7fc640 1 -- 172.21.15.88:0/1627903445 <== mon.0 v1:172.21.15.88:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 764+0+0 (unknown 3634430464 0 0) 0x7fbacc004eb0 con 0x7fbae80b4250 2024-06-03T08:36:34.466 INFO:teuthology.orchestra.run.smithi088.stderr:2024-06-03T08:36:34.465+0000 7fbade7fc640 1 -- 172.21.15.88:0/1627903445 --> v1:172.21.15.88:6789/0 -- auth(proto 2 165 bytes epoch 0) v1 -- 0x7fbae81d68d0 con 0x7fbae80b4250 2024-06-03T08:36:34.466 INFO:teuthology.orchestra.run.smithi088.stderr:2024-06-03T08:36:34.465+0000 7fbade7fc640 1 -- 172.21.15.88:0/1627903445 <== mon.0 v1:172.21.15.88:6789/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 2234374065 0 0) 0x7fbad80017f0 con 0x7fbae80b4250 2024-06-03T08:36:34.466 INFO:teuthology.orchestra.run.smithi088.stderr:2024-06-03T08:36:34.465+0000 7fbade7fc640 10 monclient(hunting): handle_monmap mon_map magic: 0 v1 2024-06-03T08:36:34.466 INFO:teuthology.orchestra.run.smithi088.stderr:2024-06-03T08:36:34.465+0000 7fbade7fc640 10 monclient(hunting): got monmap 1 from mon.noname-a (according to old e1) 2024-06-03T08:36:34.466 INFO:teuthology.orchestra.run.smithi088.stderr:2024-06-03T08:36:34.465+0000 7fbade7fc640 10 monclient(hunting): dump: 2024-06-03T08:36:34.466 INFO:teuthology.orchestra.run.smithi088.stderr:epoch 1 2024-06-03T08:36:34.466 INFO:teuthology.orchestra.run.smithi088.stderr:fsid ec3c3fdd-3ac6-4d7d-9412-683130ed8b04 2024-06-03T08:36:34.466 INFO:teuthology.orchestra.run.smithi088.stderr:last_changed 2024-06-03T08:34:31.885125+0000 2024-06-03T08:36:34.466 INFO:teuthology.orchestra.run.smithi088.stderr:created 2024-06-03T08:34:31.885125+0000 2024-06-03T08:36:34.466 INFO:teuthology.orchestra.run.smithi088.stderr:min_mon_release 18 (reef) 2024-06-03T08:36:34.466 INFO:teuthology.orchestra.run.smithi088.stderr:election_strategy: 3 2024-06-03T08:36:34.466 INFO:teuthology.orchestra.run.smithi088.stderr:0: v1:172.21.15.88:6789/0 mon.a 2024-06-03T08:36:34.467 INFO:teuthology.orchestra.run.smithi088.stderr:1: v1:172.21.15.192:6789/0 mon.b 2024-06-03T08:36:34.467 INFO:teuthology.orchestra.run.smithi088.stderr:2: v1:172.21.15.88:6790/0 mon.c 2024-06-03T08:36:34.467 INFO:teuthology.orchestra.run.smithi088.stderr: 2024-06-03T08:36:34.467 INFO:teuthology.orchestra.run.smithi088.stderr:2024-06-03T08:36:34.465+0000 7fbade7fc640 10 monclient(hunting): _finish_auth 0 2024-06-03T08:36:34.469 INFO:teuthology.orchestra.run.smithi088.stderr:2024-06-03T08:36:34.469+0000 7fbade7fc640 1 -- 172.21.15.88:0/1627903445 <== mon.2 v1:172.21.15.88:6790/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 764+0+0 (unknown 3483100185 0 0) 0x7fbae8141700 con 0x7fbae80a89f0 2024-06-03T08:36:34.470 INFO:teuthology.orchestra.run.smithi088.stderr:2024-06-03T08:36:34.469+0000 7fbade7fc640 1 -- 172.21.15.88:0/1627903445 --> v1:172.21.15.88:6790/0 -- auth(proto 2 165 bytes epoch 0) v1 -- 0x7fbacc004eb0 con 0x7fbae80a89f0 2024-06-03T08:36:34.470 INFO:teuthology.orchestra.run.smithi088.stderr:2024-06-03T08:36:34.469+0000 7fbade7fc640 1 -- 172.21.15.88:0/1627903445 <== mon.2 v1:172.21.15.88:6790/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 2234374065 0 0) 0x7fbae4001bc0 con 0x7fbae80a89f0 2024-06-03T08:36:34.470 INFO:teuthology.orchestra.run.smithi088.stderr:2024-06-03T08:36:34.469+0000 7fbade7fc640 10 monclient(hunting): handle_monmap mon_map magic: 0 v1 2024-06-03T08:36:34.470 INFO:teuthology.orchestra.run.smithi088.stderr:2024-06-03T08:36:34.469+0000 7fbade7fc640 10 monclient(hunting): got monmap 1 from mon.c (according to old e1) 2024-06-03T08:36:34.470 INFO:teuthology.orchestra.run.smithi088.stderr:2024-06-03T08:36:34.469+0000 7fbade7fc640 10 monclient(hunting): dump: 2024-06-03T08:36:34.470 INFO:teuthology.orchestra.run.smithi088.stderr:epoch 1 2024-06-03T08:36:34.470 INFO:teuthology.orchestra.run.smithi088.stderr:fsid ec3c3fdd-3ac6-4d7d-9412-683130ed8b04 2024-06-03T08:36:34.470 INFO:teuthology.orchestra.run.smithi088.stderr:last_changed 2024-06-03T08:34:31.885125+0000 2024-06-03T08:36:34.470 INFO:teuthology.orchestra.run.smithi088.stderr:created 2024-06-03T08:34:31.885125+0000 2024-06-03T08:36:34.470 INFO:teuthology.orchestra.run.smithi088.stderr:min_mon_release 18 (reef) 2024-06-03T08:36:34.470 INFO:teuthology.orchestra.run.smithi088.stderr:election_strategy: 3 2024-06-03T08:36:34.470 INFO:teuthology.orchestra.run.smithi088.stderr:0: v1:172.21.15.88:6789/0 mon.a 2024-06-03T08:36:34.470 INFO:teuthology.orchestra.run.smithi088.stderr:1: v1:172.21.15.192:6789/0 mon.b 2024-06-03T08:36:34.471 INFO:teuthology.orchestra.run.smithi088.stderr:2: v1:172.21.15.88:6790/0 mon.c 2024-06-03T08:36:34.471 INFO:teuthology.orchestra.run.smithi088.stderr: 2024-06-03T08:36:34.471 INFO:teuthology.orchestra.run.smithi088.stderr:2024-06-03T08:36:34.470+0000 7fbade7fc640 1 -- 172.21.15.88:0/1627903445 <== mon.0 v1:172.21.15.88:6789/0 4 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 580+0+0 (unknown 2488780360 0 0) 0x7fbae81d68d0 con 0x7fbae80b4250 2024-06-03T08:36:34.471 INFO:teuthology.orchestra.run.smithi088.stderr:2024-06-03T08:36:34.470+0000 7fbade7fc640 1 -- 172.21.15.88:0/1627903445 >> v1:172.21.15.88:6790/0 conn(0x7fbae80a89f0 legacy=0x7fbae81d6df0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-06-03T08:36:34.472 INFO:teuthology.orchestra.run.smithi088.stderr:2024-06-03T08:36:34.470+0000 7fbade7fc640 1 -- 172.21.15.88:0/1627903445 >> v1:172.21.15.192:6789/0 conn(0x7fbae81451c0 legacy=0x7fbae81e0f60 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-06-03T08:36:34.472 INFO:teuthology.orchestra.run.smithi088.stderr:2024-06-03T08:36:34.470+0000 7fbade7fc640 10 monclient: _finish_hunting 0 2024-06-03T08:36:34.472 INFO:teuthology.orchestra.run.smithi088.stderr:2024-06-03T08:36:34.470+0000 7fbade7fc640 1 monclient: found mon.a 2024-06-03T08:36:34.472 INFO:teuthology.orchestra.run.smithi088.stderr:2024-06-03T08:36:34.470+0000 7fbade7fc640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-06-03T08:36:34.472 INFO:teuthology.orchestra.run.smithi088.stderr:2024-06-03T08:36:34.470+0000 7fbade7fc640 10 monclient: _send_mon_message to mon.a at v1:172.21.15.88:6789/0 2024-06-03T08:36:34.472 INFO:teuthology.orchestra.run.smithi088.stderr:2024-06-03T08:36:34.470+0000 7fbade7fc640 1 -- 172.21.15.88:0/1627903445 --> v1:172.21.15.88:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fbae81d7500 con 0x7fbae80b4250 2024-06-03T08:36:34.472 INFO:teuthology.orchestra.run.smithi088.stderr:2024-06-03T08:36:34.470+0000 7fbade7fc640 10 monclient: _finish_auth 0 2024-06-03T08:36:34.472 INFO:teuthology.orchestra.run.smithi088.stderr:2024-06-03T08:36:34.470+0000 7fbade7fc640 10 monclient: _check_auth_tickets 2024-06-03T08:36:34.472 INFO:teuthology.orchestra.run.smithi088.stderr:2024-06-03T08:36:34.470+0000 7fbade7fc640 20 monclient: _check_auth_rotating not needed by client.admin 2024-06-03T08:36:34.472 INFO:teuthology.orchestra.run.smithi088.stderr:2024-06-03T08:36:34.470+0000 7fbaefed1640 5 monclient: authenticate success, global_id 4374 2024-06-03T08:36:34.472 INFO:teuthology.orchestra.run.smithi088.stderr:2024-06-03T08:36:34.470+0000 7fbaefed1640 10 monclient: _renew_subs 2024-06-03T08:36:34.472 INFO:teuthology.orchestra.run.smithi088.stderr:2024-06-03T08:36:34.470+0000 7fbaefed1640 10 monclient: _send_mon_message to mon.a at v1:172.21.15.88:6789/0 2024-06-03T08:36:34.472 INFO:teuthology.orchestra.run.smithi088.stderr:2024-06-03T08:36:34.470+0000 7fbaefed1640 1 -- 172.21.15.88:0/1627903445 --> v1:172.21.15.88:6789/0 -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fbae81349a0 con 0x7fbae80b4250 2024-06-03T08:36:34.472 INFO:teuthology.orchestra.run.smithi088.stderr:2024-06-03T08:36:34.470+0000 7fbaefed1640 10 client.4374.objecter _maybe_request_map subscribing (onetime) to next osd map 2024-06-03T08:36:34.473 INFO:teuthology.orchestra.run.smithi088.stderr:2024-06-03T08:36:34.470+0000 7fbaefed1640 10 monclient: _renew_subs 2024-06-03T08:36:34.473 INFO:teuthology.orchestra.run.smithi088.stderr:2024-06-03T08:36:34.470+0000 7fbaefed1640 10 monclient: _send_mon_message to mon.a at v1:172.21.15.88:6789/0 2024-06-03T08:36:34.473 INFO:teuthology.orchestra.run.smithi088.stderr:2024-06-03T08:36:34.470+0000 7fbaefed1640 1 -- 172.21.15.88:0/1627903445 --> v1:172.21.15.88:6789/0 -- mon_subscribe({osdmap=0}) v3 -- 0x7fbae81423b0 con 0x7fbae80b4250 2024-06-03T08:36:34.473 INFO:teuthology.orchestra.run.smithi088.stderr:2024-06-03T08:36:34.470+0000 7fbaefed1640 1 librados: init done 2024-06-03T08:36:34.473 INFO:teuthology.orchestra.run.smithi088.stderr:2024-06-03T08:36:34.472+0000 7fbaefed1640 10 start_mon_command cmd=[{"prefix": "get_command_descriptions"}] 2024-06-03T08:36:34.473 INFO:teuthology.orchestra.run.smithi088.stderr:2024-06-03T08:36:34.472+0000 7fbaefed1640 10 monclient: _send_command 1 [{"prefix": "get_command_descriptions"}] 2024-06-03T08:36:34.473 INFO:teuthology.orchestra.run.smithi088.stderr:2024-06-03T08:36:34.472+0000 7fbaefed1640 10 monclient: _send_mon_message to mon.a at v1:172.21.15.88:6789/0 2024-06-03T08:36:34.473 INFO:teuthology.orchestra.run.smithi088.stderr:2024-06-03T08:36:34.472+0000 7fbaefed1640 1 -- 172.21.15.88:0/1627903445 --> v1:172.21.15.88:6789/0 -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fbae81d6730 con 0x7fbae80b4250 2024-06-03T08:36:34.480 DEBUG:teuthology.orchestra.run.smithi088:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd last-stat-seq osd.4 2024-06-03T08:36:34.497 INFO:teuthology.orchestra.run.smithi088.stderr:2024-06-03T08:36:34.497+0000 7fbade7fc640 1 -- 172.21.15.88:0/1627903445 <== mon.0 v1:172.21.15.88:6789/0 5 ==== config(0 keys) v1 ==== 4+0+0 (unknown 0 0 0) 0x7fbad8001cc0 con 0x7fbae80b4250 2024-06-03T08:36:34.497 INFO:teuthology.orchestra.run.smithi088.stderr:2024-06-03T08:36:34.497+0000 7fbade7fc640 10 monclient: handle_config config(0 keys) v1 2024-06-03T08:36:34.497 INFO:teuthology.orchestra.run.smithi088.stderr:2024-06-03T08:36:34.497+0000 7fbade7fc640 1 -- 172.21.15.88:0/1627903445 <== mon.0 v1:172.21.15.88:6789/0 6 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 2234374065 0 0) 0x7fbad80047e0 con 0x7fbae80b4250 2024-06-03T08:36:34.497 INFO:teuthology.orchestra.run.smithi088.stderr:2024-06-03T08:36:34.497+0000 7fbade7fc640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-06-03T08:36:34.497 INFO:teuthology.orchestra.run.smithi088.stderr:2024-06-03T08:36:34.497+0000 7fbade7fc640 10 monclient: got monmap 1 from mon.a (according to old e1) 2024-06-03T08:36:34.498 INFO:teuthology.orchestra.run.smithi088.stderr:2024-06-03T08:36:34.497+0000 7fbade7fc640 10 monclient: dump: 2024-06-03T08:36:34.498 INFO:teuthology.orchestra.run.smithi088.stderr:epoch 1 2024-06-03T08:36:34.498 INFO:teuthology.orchestra.run.smithi088.stderr:fsid ec3c3fdd-3ac6-4d7d-9412-683130ed8b04 2024-06-03T08:36:34.498 INFO:teuthology.orchestra.run.smithi088.stderr:last_changed 2024-06-03T08:34:31.885125+0000 2024-06-03T08:36:34.498 INFO:teuthology.orchestra.run.smithi088.stderr:created 2024-06-03T08:34:31.885125+0000 2024-06-03T08:36:34.498 INFO:teuthology.orchestra.run.smithi088.stderr:min_mon_release 18 (reef) 2024-06-03T08:36:34.498 INFO:teuthology.orchestra.run.smithi088.stderr:election_strategy: 3 2024-06-03T08:36:34.498 INFO:teuthology.orchestra.run.smithi088.stderr:0: v1:172.21.15.88:6789/0 mon.a 2024-06-03T08:36:34.498 INFO:teuthology.orchestra.run.smithi088.stderr:1: v1:172.21.15.192:6789/0 mon.b 2024-06-03T08:36:34.498 INFO:teuthology.orchestra.run.smithi088.stderr:2: v1:172.21.15.88:6790/0 mon.c 2024-06-03T08:36:34.498 INFO:teuthology.orchestra.run.smithi088.stderr: 2024-06-03T08:36:34.499 INFO:teuthology.orchestra.run.smithi088.stderr:2024-06-03T08:36:34.497+0000 7fbadffff640 4 set_mon_vals no callback set 2024-06-03T08:36:34.499 INFO:teuthology.orchestra.run.smithi088.stderr:2024-06-03T08:36:34.498+0000 7fbade7fc640 1 -- 172.21.15.88:0/1627903445 <== mon.0 v1:172.21.15.88:6789/0 7 ==== mgrmap(e 5) v1 ==== 89110+0+0 (unknown 873533973 0 0) 0x7fbad8004940 con 0x7fbae80b4250 2024-06-03T08:36:34.499 INFO:teuthology.orchestra.run.smithi088.stderr:2024-06-03T08:36:34.499+0000 7fbaedc6d640 10 monclient: get_auth_request con 0x7fbacc06d220 auth_method 0 2024-06-03T08:36:34.500 INFO:teuthology.orchestra.run.smithi088.stderr:2024-06-03T08:36:34.499+0000 7fbade7fc640 10 client.4374.objecter ms_handle_connect 0x7fbacc06d220 2024-06-03T08:36:34.502 INFO:teuthology.orchestra.run.smithi088.stderr:2024-06-03T08:36:34.502+0000 7fbade7fc640 1 -- 172.21.15.88:0/1627903445 <== mon.0 v1:172.21.15.88:6789/0 8 ==== osd_map(26..26 src has 1..26) v4 ==== 4090+0+0 (unknown 3637450554 0 0) 0x7fbad8053e60 con 0x7fbae80b4250 2024-06-03T08:36:34.503 INFO:teuthology.orchestra.run.smithi088.stderr:2024-06-03T08:36:34.502+0000 7fbade7fc640 10 client.4374.objecter ms_dispatch 0x7fbae8000b90 osd_map(26..26 src has 1..26) v4 2024-06-03T08:36:34.503 INFO:teuthology.orchestra.run.smithi088.stderr:2024-06-03T08:36:34.502+0000 7fbade7fc640 3 client.4374.objecter handle_osd_map got epochs [26,26] > 0 2024-06-03T08:36:34.503 INFO:teuthology.orchestra.run.smithi088.stderr:2024-06-03T08:36:34.502+0000 7fbade7fc640 3 client.4374.objecter handle_osd_map decoding full epoch 26 2024-06-03T08:36:34.503 INFO:teuthology.orchestra.run.smithi088.stderr:2024-06-03T08:36:34.502+0000 7fbade7fc640 20 client.4374.objecter dump_active .. 0 homeless 2024-06-03T08:36:34.525 INFO:teuthology.orchestra.run.smithi088.stderr:2024-06-03T08:36:34.524+0000 7fef1dfb2640 10 monclient: get_monmap_and_config 2024-06-03T08:36:34.525 INFO:teuthology.orchestra.run.smithi088.stderr:2024-06-03T08:36:34.524+0000 7fef1dfb2640 10 monclient: build_initial_monmap 2024-06-03T08:36:34.525 INFO:teuthology.orchestra.run.smithi088.stderr:2024-06-03T08:36:34.524+0000 7fef1dfb2640 10 monclient: monmap: 2024-06-03T08:36:34.525 INFO:teuthology.orchestra.run.smithi088.stderr:epoch 0 2024-06-03T08:36:34.525 INFO:teuthology.orchestra.run.smithi088.stderr:fsid ec3c3fdd-3ac6-4d7d-9412-683130ed8b04 2024-06-03T08:36:34.525 INFO:teuthology.orchestra.run.smithi088.stderr:last_changed 2024-06-03T08:36:34.525613+0000 2024-06-03T08:36:34.525 INFO:teuthology.orchestra.run.smithi088.stderr:created 2024-06-03T08:36:34.525613+0000 2024-06-03T08:36:34.525 INFO:teuthology.orchestra.run.smithi088.stderr:min_mon_release 0 (unknown) 2024-06-03T08:36:34.525 INFO:teuthology.orchestra.run.smithi088.stderr:election_strategy: 3 2024-06-03T08:36:34.525 INFO:teuthology.orchestra.run.smithi088.stderr:0: v1:172.21.15.88:6789/0 mon.noname-a 2024-06-03T08:36:34.525 INFO:teuthology.orchestra.run.smithi088.stderr:1: v1:172.21.15.192:6789/0 mon.noname-c 2024-06-03T08:36:34.526 INFO:teuthology.orchestra.run.smithi088.stderr:2: v1:172.21.15.88:6790/0 mon.noname-b 2024-06-03T08:36:34.526 INFO:teuthology.orchestra.run.smithi088.stderr: 2024-06-03T08:36:34.527 INFO:teuthology.orchestra.run.smithi088.stderr:2024-06-03T08:36:34.525+0000 7fef1dfb2640 1 Processor -- start 2024-06-03T08:36:34.528 INFO:teuthology.orchestra.run.smithi088.stderr:2024-06-03T08:36:34.525+0000 7fef1dfb2640 1 -- start start 2024-06-03T08:36:34.528 INFO:teuthology.orchestra.run.smithi088.stderr:2024-06-03T08:36:34.525+0000 7fef1dfb2640 10 monclient: init 2024-06-03T08:36:34.528 INFO:teuthology.orchestra.run.smithi088.stderr:2024-06-03T08:36:34.526+0000 7fef1dfb2640 10 monclient: _reopen_session rank -1 2024-06-03T08:36:34.528 INFO:teuthology.orchestra.run.smithi088.stderr:2024-06-03T08:36:34.526+0000 7fef1dfb2640 10 monclient: _add_conns ranks=[0,1,2] 2024-06-03T08:36:34.528 INFO:teuthology.orchestra.run.smithi088.stderr:2024-06-03T08:36:34.526+0000 7fef1dfb2640 10 monclient(hunting): picked mon.noname-a con 0x7fef180a8960 addr v1:172.21.15.88:6789/0 2024-06-03T08:36:34.528 INFO:teuthology.orchestra.run.smithi088.stderr:2024-06-03T08:36:34.526+0000 7fef1dfb2640 10 monclient(hunting): picked mon.noname-c con 0x7fef181450f0 addr v1:172.21.15.192:6789/0 2024-06-03T08:36:34.528 INFO:teuthology.orchestra.run.smithi088.stderr:2024-06-03T08:36:34.526+0000 7fef1dfb2640 10 monclient(hunting): picked mon.noname-b con 0x7fef1813eec0 addr v1:172.21.15.88:6790/0 2024-06-03T08:36:34.528 INFO:teuthology.orchestra.run.smithi088.stderr:2024-06-03T08:36:34.526+0000 7fef1dfb2640 1 -- --> v1:172.21.15.88:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fef18144c60 con 0x7fef180a8960 2024-06-03T08:36:34.528 INFO:teuthology.orchestra.run.smithi088.stderr:2024-06-03T08:36:34.526+0000 7fef1dfb2640 1 -- --> v1:172.21.15.192:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fef180b3930 con 0x7fef181450f0 2024-06-03T08:36:34.528 INFO:teuthology.orchestra.run.smithi088.stderr:2024-06-03T08:36:34.526+0000 7fef1dfb2640 1 -- --> v1:172.21.15.88:6790/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fef180b1ad0 con 0x7fef1813eec0 2024-06-03T08:36:34.528 INFO:teuthology.orchestra.run.smithi088.stderr:2024-06-03T08:36:34.526+0000 7fef1dfb2640 10 monclient(hunting): _renew_subs 2024-06-03T08:36:34.528 INFO:teuthology.orchestra.run.smithi088.stderr:2024-06-03T08:36:34.526+0000 7fef1dfb2640 10 monclient(hunting): authenticate will time out at 1250.336349s 2024-06-03T08:36:34.528 INFO:teuthology.orchestra.run.smithi088.stderr:2024-06-03T08:36:34.527+0000 7fef17fff640 1 --1- >> v1:172.21.15.88:6790/0 conn(0x7fef1813eec0 0x7fef1814ce60 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.88:6790/0 says I am v1:172.21.15.88:54446/0 (socket says 172.21.15.88:54446) 2024-06-03T08:36:34.528 INFO:teuthology.orchestra.run.smithi088.stderr:2024-06-03T08:36:34.527+0000 7fef17fff640 1 -- 172.21.15.88:0/1810192344 learned_addr learned my addr 172.21.15.88:0/1810192344 (peer_addr_for_me v1:172.21.15.88:0/0) 2024-06-03T08:36:34.531 INFO:teuthology.orchestra.run.smithi088.stderr:2024-06-03T08:36:34.530+0000 7fef167fc640 1 -- 172.21.15.88:0/1810192344 <== mon.2 v1:172.21.15.88:6790/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 3996920819 0 0) 0x7fef180b1ad0 con 0x7fef1813eec0 2024-06-03T08:36:34.531 INFO:teuthology.orchestra.run.smithi088.stderr:2024-06-03T08:36:34.530+0000 7fef167fc640 10 monclient(hunting): _negotiate 2024-06-03T08:36:34.531 INFO:teuthology.orchestra.run.smithi088.stderr:2024-06-03T08:36:34.530+0000 7fef167fc640 10 monclient(hunting): _init_auth method 2 2024-06-03T08:36:34.531 INFO:teuthology.orchestra.run.smithi088.stderr:2024-06-03T08:36:34.530+0000 7fef167fc640 10 monclient(hunting): _init_auth creating new auth 2024-06-03T08:36:34.531 INFO:teuthology.orchestra.run.smithi088.stderr:2024-06-03T08:36:34.530+0000 7fef167fc640 10 monclient(hunting): my global_id is 4385 2024-06-03T08:36:34.531 INFO:teuthology.orchestra.run.smithi088.stderr:2024-06-03T08:36:34.530+0000 7fef167fc640 1 -- 172.21.15.88:0/1810192344 --> v1:172.21.15.88:6790/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7fef00003520 con 0x7fef1813eec0 2024-06-03T08:36:34.536 INFO:teuthology.orchestra.run.smithi088.stderr:2024-06-03T08:36:34.536+0000 7fef167fc640 1 -- 172.21.15.88:0/1810192344 <== mon.2 v1:172.21.15.88:6790/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 214+0+0 (unknown 2318792713 0 0) 0x7fef00003520 con 0x7fef1813eec0 2024-06-03T08:36:34.537 INFO:teuthology.orchestra.run.smithi088.stderr:2024-06-03T08:36:34.536+0000 7fef167fc640 1 -- 172.21.15.88:0/1810192344 >> v1:172.21.15.192:6789/0 conn(0x7fef181450f0 legacy=0x7fef1813e7b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-06-03T08:36:34.537 INFO:teuthology.orchestra.run.smithi088.stderr:2024-06-03T08:36:34.536+0000 7fef167fc640 1 -- 172.21.15.88:0/1810192344 >> v1:172.21.15.88:6789/0 conn(0x7fef180a8960 legacy=0x7fef180a8d40 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-06-03T08:36:34.537 INFO:teuthology.orchestra.run.smithi088.stderr:2024-06-03T08:36:34.536+0000 7fef167fc640 10 monclient: _finish_hunting 0 2024-06-03T08:36:34.537 INFO:teuthology.orchestra.run.smithi088.stderr:2024-06-03T08:36:34.536+0000 7fef167fc640 1 monclient: found mon.noname-b 2024-06-03T08:36:34.537 INFO:teuthology.orchestra.run.smithi088.stderr:2024-06-03T08:36:34.536+0000 7fef167fc640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-06-03T08:36:34.537 INFO:teuthology.orchestra.run.smithi088.stderr:2024-06-03T08:36:34.536+0000 7fef167fc640 10 monclient: _send_mon_message to mon.noname-b at v1:172.21.15.88:6790/0 2024-06-03T08:36:34.537 INFO:teuthology.orchestra.run.smithi088.stderr:2024-06-03T08:36:34.536+0000 7fef167fc640 1 -- 172.21.15.88:0/1810192344 --> v1:172.21.15.88:6790/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fef180b1c60 con 0x7fef1813eec0 2024-06-03T08:36:34.537 INFO:teuthology.orchestra.run.smithi088.stderr:2024-06-03T08:36:34.536+0000 7fef167fc640 10 monclient: _finish_auth 0 2024-06-03T08:36:34.537 INFO:teuthology.orchestra.run.smithi088.stderr:2024-06-03T08:36:34.536+0000 7fef167fc640 10 monclient: _check_auth_tickets 2024-06-03T08:36:34.537 INFO:teuthology.orchestra.run.smithi088.stderr:2024-06-03T08:36:34.536+0000 7fef167fc640 20 monclient: _check_auth_rotating not needed by client.admin 2024-06-03T08:36:34.537 INFO:teuthology.orchestra.run.smithi088.stderr:2024-06-03T08:36:34.536+0000 7fef167fc640 1 -- 172.21.15.88:0/1810192344 <== mon.2 v1:172.21.15.88:6790/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 2234374065 0 0) 0x7fef10001990 con 0x7fef1813eec0 2024-06-03T08:36:34.537 INFO:teuthology.orchestra.run.smithi088.stderr:2024-06-03T08:36:34.536+0000 7fef167fc640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-06-03T08:36:34.537 INFO:teuthology.orchestra.run.smithi088.stderr:2024-06-03T08:36:34.536+0000 7fef167fc640 10 monclient: got monmap 1 from mon.noname-b (according to old e1) 2024-06-03T08:36:34.538 INFO:teuthology.orchestra.run.smithi088.stderr:2024-06-03T08:36:34.536+0000 7fef167fc640 10 monclient: dump: 2024-06-03T08:36:34.538 INFO:teuthology.orchestra.run.smithi088.stderr:epoch 1 2024-06-03T08:36:34.538 INFO:teuthology.orchestra.run.smithi088.stderr:fsid ec3c3fdd-3ac6-4d7d-9412-683130ed8b04 2024-06-03T08:36:34.538 INFO:teuthology.orchestra.run.smithi088.stderr:last_changed 2024-06-03T08:34:31.885125+0000 2024-06-03T08:36:34.538 INFO:teuthology.orchestra.run.smithi088.stderr:created 2024-06-03T08:34:31.885125+0000 2024-06-03T08:36:34.538 INFO:teuthology.orchestra.run.smithi088.stderr:min_mon_release 18 (reef) 2024-06-03T08:36:34.538 INFO:teuthology.orchestra.run.smithi088.stderr:election_strategy: 3 2024-06-03T08:36:34.538 INFO:teuthology.orchestra.run.smithi088.stderr:0: v1:172.21.15.88:6789/0 mon.a 2024-06-03T08:36:34.538 INFO:teuthology.orchestra.run.smithi088.stderr:1: v1:172.21.15.192:6789/0 mon.b 2024-06-03T08:36:34.538 INFO:teuthology.orchestra.run.smithi088.stderr:2: v1:172.21.15.88:6790/0 mon.c 2024-06-03T08:36:34.538 INFO:teuthology.orchestra.run.smithi088.stderr: 2024-06-03T08:36:34.539 INFO:teuthology.orchestra.run.smithi088.stderr:2024-06-03T08:36:34.536+0000 7fef1dfb2640 5 monclient: authenticate success, global_id 4385 2024-06-03T08:36:34.539 INFO:teuthology.orchestra.run.smithi088.stderr:2024-06-03T08:36:34.536+0000 7fef1dfb2640 20 monclient: get_monmap_and_config waiting for monmap|config 2024-06-03T08:36:34.540 INFO:teuthology.orchestra.run.smithi088.stderr:2024-06-03T08:36:34.539+0000 7fef167fc640 1 -- 172.21.15.88:0/1810192344 <== mon.2 v1:172.21.15.88:6790/0 4 ==== config(0 keys) v1 ==== 4+0+0 (unknown 0 0 0) 0x7fef10001bb0 con 0x7fef1813eec0 2024-06-03T08:36:34.540 INFO:teuthology.orchestra.run.smithi088.stderr:2024-06-03T08:36:34.539+0000 7fef167fc640 10 monclient: handle_config config(0 keys) v1 2024-06-03T08:36:34.540 INFO:teuthology.orchestra.run.smithi088.stderr:2024-06-03T08:36:34.539+0000 7fef1dfb2640 10 monclient: get_monmap_and_config success 2024-06-03T08:36:34.540 INFO:teuthology.orchestra.run.smithi088.stderr:2024-06-03T08:36:34.539+0000 7fef167fc640 1 -- 172.21.15.88:0/1810192344 <== mon.2 v1:172.21.15.88:6790/0 5 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 2234374065 0 0) 0x7fef10001f40 con 0x7fef1813eec0 2024-06-03T08:36:34.541 INFO:teuthology.orchestra.run.smithi088.stderr:2024-06-03T08:36:34.539+0000 7fef167fc640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-06-03T08:36:34.541 INFO:teuthology.orchestra.run.smithi088.stderr:2024-06-03T08:36:34.539+0000 7fef1dfb2640 4 set_mon_vals no callback set 2024-06-03T08:36:34.541 INFO:teuthology.orchestra.run.smithi088.stderr:2024-06-03T08:36:34.539+0000 7fef167fc640 10 monclient: got monmap 1 from mon.c (according to old e1) 2024-06-03T08:36:34.541 INFO:teuthology.orchestra.run.smithi088.stderr:2024-06-03T08:36:34.539+0000 7fef167fc640 10 monclient: dump: 2024-06-03T08:36:34.541 INFO:teuthology.orchestra.run.smithi088.stderr:epoch 1 2024-06-03T08:36:34.541 INFO:teuthology.orchestra.run.smithi088.stderr:fsid ec3c3fdd-3ac6-4d7d-9412-683130ed8b04 2024-06-03T08:36:34.541 INFO:teuthology.orchestra.run.smithi088.stderr:last_changed 2024-06-03T08:34:31.885125+0000 2024-06-03T08:36:34.541 INFO:teuthology.orchestra.run.smithi088.stderr:created 2024-06-03T08:34:31.885125+0000 2024-06-03T08:36:34.541 INFO:teuthology.orchestra.run.smithi088.stderr:min_mon_release 18 (reef) 2024-06-03T08:36:34.541 INFO:teuthology.orchestra.run.smithi088.stderr:election_strategy: 3 2024-06-03T08:36:34.542 INFO:teuthology.orchestra.run.smithi088.stderr:0: v1:172.21.15.88:6789/0 mon.a 2024-06-03T08:36:34.542 INFO:teuthology.orchestra.run.smithi088.stderr:1: v1:172.21.15.192:6789/0 mon.b 2024-06-03T08:36:34.542 INFO:teuthology.orchestra.run.smithi088.stderr:2: v1:172.21.15.88:6790/0 mon.c 2024-06-03T08:36:34.542 INFO:teuthology.orchestra.run.smithi088.stderr: 2024-06-03T08:36:34.542 INFO:teuthology.orchestra.run.smithi088.stderr:2024-06-03T08:36:34.539+0000 7fef1dfb2640 10 monclient: shutdown 2024-06-03T08:36:34.542 INFO:teuthology.orchestra.run.smithi088.stderr:2024-06-03T08:36:34.539+0000 7fef1dfb2640 20 monclient: shutdown discarding 0 pending message(s) 2024-06-03T08:36:34.542 INFO:teuthology.orchestra.run.smithi088.stderr:2024-06-03T08:36:34.539+0000 7fef1dfb2640 1 -- 172.21.15.88:0/1810192344 >> v1:172.21.15.88:6790/0 conn(0x7fef1813eec0 legacy=0x7fef1814ce60 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-06-03T08:36:34.542 INFO:teuthology.orchestra.run.smithi088.stderr:2024-06-03T08:36:34.540+0000 7fef1dfb2640 1 -- 172.21.15.88:0/1810192344 shutdown_connections 2024-06-03T08:36:34.542 INFO:teuthology.orchestra.run.smithi088.stderr:2024-06-03T08:36:34.540+0000 7fef1dfb2640 1 -- 172.21.15.88:0/1810192344 >> 172.21.15.88:0/1810192344 conn(0x7fef180b70d0 msgr2=0x7fef180b74e0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-06-03T08:36:34.542 INFO:teuthology.orchestra.run.smithi088.stderr:2024-06-03T08:36:34.540+0000 7fef1dfb2640 1 -- 172.21.15.88:0/1810192344 shutdown_connections 2024-06-03T08:36:34.542 INFO:teuthology.orchestra.run.smithi088.stderr:2024-06-03T08:36:34.540+0000 7fef1dfb2640 1 -- 172.21.15.88:0/1810192344 wait complete. 2024-06-03T08:36:34.548 INFO:teuthology.orchestra.run.smithi088.stderr:2024-06-03T08:36:34.540+0000 7fef1dfb2640 10 monclient: build_initial_monmap 2024-06-03T08:36:34.549 INFO:teuthology.orchestra.run.smithi088.stderr:2024-06-03T08:36:34.541+0000 7fef1dfb2640 10 monclient: monmap: 2024-06-03T08:36:34.549 INFO:teuthology.orchestra.run.smithi088.stderr:epoch 0 2024-06-03T08:36:34.549 INFO:teuthology.orchestra.run.smithi088.stderr:fsid 00000000-0000-0000-0000-000000000000 2024-06-03T08:36:34.549 INFO:teuthology.orchestra.run.smithi088.stderr:last_changed 0.000000 2024-06-03T08:36:34.549 INFO:teuthology.orchestra.run.smithi088.stderr:created 0.000000 2024-06-03T08:36:34.549 INFO:teuthology.orchestra.run.smithi088.stderr:min_mon_release 0 (unknown) 2024-06-03T08:36:34.549 INFO:teuthology.orchestra.run.smithi088.stderr:election_strategy: 1 2024-06-03T08:36:34.549 INFO:teuthology.orchestra.run.smithi088.stderr:0: v1:172.21.15.88:6789/0 mon.noname-a 2024-06-03T08:36:34.549 INFO:teuthology.orchestra.run.smithi088.stderr:1: v1:172.21.15.192:6789/0 mon.noname-b 2024-06-03T08:36:34.549 INFO:teuthology.orchestra.run.smithi088.stderr:2: v1:172.21.15.88:6790/0 mon.noname-c 2024-06-03T08:36:34.549 INFO:teuthology.orchestra.run.smithi088.stderr: 2024-06-03T08:36:34.549 INFO:teuthology.orchestra.run.smithi088.stderr:2024-06-03T08:36:34.541+0000 7fef1dfb2640 1 librados: starting msgr at 2024-06-03T08:36:34.549 INFO:teuthology.orchestra.run.smithi088.stderr:2024-06-03T08:36:34.541+0000 7fef1dfb2640 1 librados: starting objecter 2024-06-03T08:36:34.549 INFO:teuthology.orchestra.run.smithi088.stderr:2024-06-03T08:36:34.541+0000 7fef1dfb2640 1 Processor -- start 2024-06-03T08:36:34.550 INFO:teuthology.orchestra.run.smithi088.stderr:2024-06-03T08:36:34.541+0000 7fef1dfb2640 1 -- start start 2024-06-03T08:36:34.550 INFO:teuthology.orchestra.run.smithi088.stderr:2024-06-03T08:36:34.541+0000 7fef1dfb2640 1 librados: setting wanted keys 2024-06-03T08:36:34.550 INFO:teuthology.orchestra.run.smithi088.stderr:2024-06-03T08:36:34.541+0000 7fef1dfb2640 1 librados: calling monclient init 2024-06-03T08:36:34.550 INFO:teuthology.orchestra.run.smithi088.stderr:2024-06-03T08:36:34.541+0000 7fef1dfb2640 10 monclient: init 2024-06-03T08:36:34.550 INFO:teuthology.orchestra.run.smithi088.stderr:2024-06-03T08:36:34.542+0000 7fef1dfb2640 10 monclient: _reopen_session rank -1 2024-06-03T08:36:34.550 INFO:teuthology.orchestra.run.smithi088.stderr:2024-06-03T08:36:34.542+0000 7fef1dfb2640 10 monclient: _add_conns ranks=[0,1,2] 2024-06-03T08:36:34.550 INFO:teuthology.orchestra.run.smithi088.stderr:2024-06-03T08:36:34.542+0000 7fef1dfb2640 10 monclient(hunting): picked mon.noname-a con 0x7fef180a8960 addr v1:172.21.15.88:6789/0 2024-06-03T08:36:34.550 INFO:teuthology.orchestra.run.smithi088.stderr:2024-06-03T08:36:34.542+0000 7fef1dfb2640 10 monclient(hunting): picked mon.noname-b con 0x7fef1813eec0 addr v1:172.21.15.192:6789/0 2024-06-03T08:36:34.550 INFO:teuthology.orchestra.run.smithi088.stderr:2024-06-03T08:36:34.542+0000 7fef1dfb2640 10 monclient(hunting): picked mon.noname-c con 0x7fef181450f0 addr v1:172.21.15.88:6790/0 2024-06-03T08:36:34.550 INFO:teuthology.orchestra.run.smithi088.stderr:2024-06-03T08:36:34.542+0000 7fef1dfb2640 1 -- --> v1:172.21.15.88:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fef180a9660 con 0x7fef180a8960 2024-06-03T08:36:34.550 INFO:teuthology.orchestra.run.smithi088.stderr:2024-06-03T08:36:34.542+0000 7fef1dfb2640 1 -- --> v1:172.21.15.192:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fef1813b810 con 0x7fef1813eec0 2024-06-03T08:36:34.550 INFO:teuthology.orchestra.run.smithi088.stderr:2024-06-03T08:36:34.542+0000 7fef1dfb2640 1 -- --> v1:172.21.15.88:6790/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fef180ab200 con 0x7fef181450f0 2024-06-03T08:36:34.550 INFO:teuthology.orchestra.run.smithi088.stderr:2024-06-03T08:36:34.542+0000 7fef1dfb2640 10 monclient(hunting): _renew_subs 2024-06-03T08:36:34.550 INFO:teuthology.orchestra.run.smithi088.stderr:2024-06-03T08:36:34.542+0000 7fef1dfb2640 10 monclient(hunting): authenticate will time out at 1250.352552s 2024-06-03T08:36:34.551 INFO:teuthology.orchestra.run.smithi088.stderr:2024-06-03T08:36:34.543+0000 7fef17fff640 1 --1- >> v1:172.21.15.88:6790/0 conn(0x7fef181450f0 0x7fef181e15e0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=1).handle_server_banner_and_identify peer v1:172.21.15.88:6790/0 says I am v1:172.21.15.88:54458/0 (socket says 172.21.15.88:54458) 2024-06-03T08:36:34.551 INFO:teuthology.orchestra.run.smithi088.stderr:2024-06-03T08:36:34.543+0000 7fef17fff640 1 -- 172.21.15.88:0/3086147656 learned_addr learned my addr 172.21.15.88:0/3086147656 (peer_addr_for_me v1:172.21.15.88:0/0) 2024-06-03T08:36:34.551 INFO:teuthology.orchestra.run.smithi088.stderr:2024-06-03T08:36:34.543+0000 7feefffff640 10 client.?.objecter ms_handle_connect 0x7fef181450f0 2024-06-03T08:36:34.551 INFO:teuthology.orchestra.run.smithi088.stderr:2024-06-03T08:36:34.543+0000 7feefffff640 10 client.?.objecter resend_mon_ops 2024-06-03T08:36:34.551 INFO:teuthology.orchestra.run.smithi088.stderr:2024-06-03T08:36:34.546+0000 7feefffff640 1 -- 172.21.15.88:0/3086147656 <== mon.2 v1:172.21.15.88:6790/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 999971559 0 0) 0x7fef180ab200 con 0x7fef181450f0 2024-06-03T08:36:34.551 INFO:teuthology.orchestra.run.smithi088.stderr:2024-06-03T08:36:34.546+0000 7feefffff640 10 monclient(hunting): _negotiate 2024-06-03T08:36:34.551 INFO:teuthology.orchestra.run.smithi088.stderr:2024-06-03T08:36:34.546+0000 7feefffff640 10 monclient(hunting): _init_auth method 2 2024-06-03T08:36:34.551 INFO:teuthology.orchestra.run.smithi088.stderr:2024-06-03T08:36:34.546+0000 7feefffff640 10 monclient(hunting): _init_auth creating new auth 2024-06-03T08:36:34.551 INFO:teuthology.orchestra.run.smithi088.stderr:2024-06-03T08:36:34.546+0000 7feefffff640 10 monclient(hunting): my global_id is 4388 2024-06-03T08:36:34.551 INFO:teuthology.orchestra.run.smithi088.stderr:2024-06-03T08:36:34.546+0000 7feefffff640 1 -- 172.21.15.88:0/3086147656 --> v1:172.21.15.88:6790/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7fef04004eb0 con 0x7fef181450f0 2024-06-03T08:36:34.555 INFO:teuthology.orchestra.run.smithi088.stderr:2024-06-03T08:36:34.554+0000 7feefffff640 1 -- 172.21.15.88:0/3086147656 <== mon.2 v1:172.21.15.88:6790/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 764+0+0 (unknown 925710392 0 0) 0x7fef04004eb0 con 0x7fef181450f0 2024-06-03T08:36:34.555 INFO:teuthology.orchestra.run.smithi088.stderr:2024-06-03T08:36:34.554+0000 7feefffff640 1 -- 172.21.15.88:0/3086147656 --> v1:172.21.15.88:6790/0 -- auth(proto 2 165 bytes epoch 0) v1 -- 0x7fef180ab200 con 0x7fef181450f0 2024-06-03T08:36:34.555 INFO:teuthology.orchestra.run.smithi088.stderr:2024-06-03T08:36:34.554+0000 7feefffff640 1 -- 172.21.15.88:0/3086147656 <== mon.2 v1:172.21.15.88:6790/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 2234374065 0 0) 0x7fef100018d0 con 0x7fef181450f0 2024-06-03T08:36:34.555 INFO:teuthology.orchestra.run.smithi088.stderr:2024-06-03T08:36:34.554+0000 7feefffff640 10 monclient(hunting): handle_monmap mon_map magic: 0 v1 2024-06-03T08:36:34.555 INFO:teuthology.orchestra.run.smithi088.stderr:2024-06-03T08:36:34.554+0000 7feefffff640 10 monclient(hunting): got monmap 1 from mon.noname-c (according to old e1) 2024-06-03T08:36:34.555 INFO:teuthology.orchestra.run.smithi088.stderr:2024-06-03T08:36:34.554+0000 7feefffff640 10 monclient(hunting): dump: 2024-06-03T08:36:34.555 INFO:teuthology.orchestra.run.smithi088.stderr:epoch 1 2024-06-03T08:36:34.556 INFO:teuthology.orchestra.run.smithi088.stderr:fsid ec3c3fdd-3ac6-4d7d-9412-683130ed8b04 2024-06-03T08:36:34.556 INFO:teuthology.orchestra.run.smithi088.stderr:last_changed 2024-06-03T08:34:31.885125+0000 2024-06-03T08:36:34.556 INFO:teuthology.orchestra.run.smithi088.stderr:created 2024-06-03T08:34:31.885125+0000 2024-06-03T08:36:34.556 INFO:teuthology.orchestra.run.smithi088.stderr:min_mon_release 18 (reef) 2024-06-03T08:36:34.556 INFO:teuthology.orchestra.run.smithi088.stderr:election_strategy: 3 2024-06-03T08:36:34.556 INFO:teuthology.orchestra.run.smithi088.stderr:0: v1:172.21.15.88:6789/0 mon.a 2024-06-03T08:36:34.556 INFO:teuthology.orchestra.run.smithi088.stderr:1: v1:172.21.15.192:6789/0 mon.b 2024-06-03T08:36:34.556 INFO:teuthology.orchestra.run.smithi088.stderr:2: v1:172.21.15.88:6790/0 mon.c 2024-06-03T08:36:34.556 INFO:teuthology.orchestra.run.smithi088.stderr: 2024-06-03T08:36:34.556 INFO:teuthology.orchestra.run.smithi088.stderr:2024-06-03T08:36:34.554+0000 7feefffff640 10 monclient(hunting): _finish_auth 0 2024-06-03T08:36:34.562 INFO:teuthology.orchestra.run.smithi088.stderr:2024-06-03T08:36:34.560+0000 7feefffff640 1 -- 172.21.15.88:0/3086147656 <== mon.2 v1:172.21.15.88:6790/0 4 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 580+0+0 (unknown 1641547415 0 0) 0x7fef180ab200 con 0x7fef181450f0 2024-06-03T08:36:34.562 INFO:teuthology.orchestra.run.smithi088.stderr:2024-06-03T08:36:34.560+0000 7feefffff640 1 -- 172.21.15.88:0/3086147656 >> v1:172.21.15.192:6789/0 conn(0x7fef1813eec0 legacy=0x7fef181379b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-06-03T08:36:34.562 INFO:teuthology.orchestra.run.smithi088.stderr:2024-06-03T08:36:34.560+0000 7feefffff640 1 -- 172.21.15.88:0/3086147656 >> v1:172.21.15.88:6789/0 conn(0x7fef180a8960 legacy=0x7fef1813b060 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-06-03T08:36:34.562 INFO:teuthology.orchestra.run.smithi088.stderr:2024-06-03T08:36:34.560+0000 7feefffff640 10 monclient: _finish_hunting 0 2024-06-03T08:36:34.563 INFO:teuthology.orchestra.run.smithi088.stderr:2024-06-03T08:36:34.560+0000 7feefffff640 1 monclient: found mon.c 2024-06-03T08:36:34.563 INFO:teuthology.orchestra.run.smithi088.stderr:2024-06-03T08:36:34.560+0000 7feefffff640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-06-03T08:36:34.563 INFO:teuthology.orchestra.run.smithi088.stderr:2024-06-03T08:36:34.560+0000 7feefffff640 10 monclient: _send_mon_message to mon.c at v1:172.21.15.88:6790/0 2024-06-03T08:36:34.563 INFO:teuthology.orchestra.run.smithi088.stderr:2024-06-03T08:36:34.560+0000 7feefffff640 1 -- 172.21.15.88:0/3086147656 --> v1:172.21.15.88:6790/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fef180ab4e0 con 0x7fef181450f0 2024-06-03T08:36:34.563 INFO:teuthology.orchestra.run.smithi088.stderr:2024-06-03T08:36:34.560+0000 7feefffff640 10 monclient: _finish_auth 0 2024-06-03T08:36:34.563 INFO:teuthology.orchestra.run.smithi088.stderr:2024-06-03T08:36:34.560+0000 7feefffff640 10 monclient: _check_auth_tickets 2024-06-03T08:36:34.563 INFO:teuthology.orchestra.run.smithi088.stderr:2024-06-03T08:36:34.560+0000 7feefffff640 20 monclient: _check_auth_rotating not needed by client.admin 2024-06-03T08:36:34.563 INFO:teuthology.orchestra.run.smithi088.stderr:2024-06-03T08:36:34.560+0000 7fef1dfb2640 5 monclient: authenticate success, global_id 4388 2024-06-03T08:36:34.563 INFO:teuthology.orchestra.run.smithi088.stderr:2024-06-03T08:36:34.560+0000 7fef1dfb2640 10 monclient: _renew_subs 2024-06-03T08:36:34.563 INFO:teuthology.orchestra.run.smithi088.stderr:2024-06-03T08:36:34.560+0000 7fef1dfb2640 10 monclient: _send_mon_message to mon.c at v1:172.21.15.88:6790/0 2024-06-03T08:36:34.563 INFO:teuthology.orchestra.run.smithi088.stderr:2024-06-03T08:36:34.560+0000 7fef1dfb2640 1 -- 172.21.15.88:0/3086147656 --> v1:172.21.15.88:6790/0 -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fef18139680 con 0x7fef181450f0 2024-06-03T08:36:34.563 INFO:teuthology.orchestra.run.smithi088.stderr:2024-06-03T08:36:34.560+0000 7fef1dfb2640 10 client.4388.objecter _maybe_request_map subscribing (onetime) to next osd map 2024-06-03T08:36:34.563 INFO:teuthology.orchestra.run.smithi088.stderr:2024-06-03T08:36:34.560+0000 7fef1dfb2640 10 monclient: _renew_subs 2024-06-03T08:36:34.563 INFO:teuthology.orchestra.run.smithi088.stderr:2024-06-03T08:36:34.560+0000 7fef1dfb2640 10 monclient: _send_mon_message to mon.c at v1:172.21.15.88:6790/0 2024-06-03T08:36:34.564 INFO:teuthology.orchestra.run.smithi088.stderr:2024-06-03T08:36:34.560+0000 7fef1dfb2640 1 -- 172.21.15.88:0/3086147656 --> v1:172.21.15.88:6790/0 -- mon_subscribe({osdmap=0}) v3 -- 0x7fef180b08a0 con 0x7fef181450f0 2024-06-03T08:36:34.564 INFO:teuthology.orchestra.run.smithi088.stderr:2024-06-03T08:36:34.560+0000 7fef1dfb2640 1 librados: init done 2024-06-03T08:36:34.564 INFO:teuthology.orchestra.run.smithi088.stderr:2024-06-03T08:36:34.561+0000 7fef1dfb2640 10 start_mon_command cmd=[{"prefix": "get_command_descriptions"}] 2024-06-03T08:36:34.564 INFO:teuthology.orchestra.run.smithi088.stderr:2024-06-03T08:36:34.561+0000 7fef1dfb2640 10 monclient: _send_command 1 [{"prefix": "get_command_descriptions"}] 2024-06-03T08:36:34.564 INFO:teuthology.orchestra.run.smithi088.stderr:2024-06-03T08:36:34.561+0000 7fef1dfb2640 10 monclient: _send_mon_message to mon.c at v1:172.21.15.88:6790/0 2024-06-03T08:36:34.564 INFO:teuthology.orchestra.run.smithi088.stderr:2024-06-03T08:36:34.561+0000 7fef1dfb2640 1 -- 172.21.15.88:0/3086147656 --> v1:172.21.15.88:6790/0 -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fef180a9e60 con 0x7fef181450f0 2024-06-03T08:36:34.582 INFO:teuthology.orchestra.run.smithi088.stderr:2024-06-03T08:36:34.582+0000 7feefffff640 1 -- 172.21.15.88:0/3086147656 <== mon.2 v1:172.21.15.88:6790/0 5 ==== config(0 keys) v1 ==== 4+0+0 (unknown 0 0 0) 0x7fef10004e10 con 0x7fef181450f0 2024-06-03T08:36:34.582 INFO:teuthology.orchestra.run.smithi088.stderr:2024-06-03T08:36:34.582+0000 7feefffff640 10 monclient: handle_config config(0 keys) v1 2024-06-03T08:36:34.582 INFO:teuthology.orchestra.run.smithi088.stderr:2024-06-03T08:36:34.582+0000 7feefffff640 1 -- 172.21.15.88:0/3086147656 <== mon.2 v1:172.21.15.88:6790/0 6 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 2234374065 0 0) 0x7fef100051a0 con 0x7fef181450f0 2024-06-03T08:36:34.582 INFO:teuthology.orchestra.run.smithi088.stderr:2024-06-03T08:36:34.582+0000 7feefffff640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-06-03T08:36:34.582 INFO:teuthology.orchestra.run.smithi088.stderr:2024-06-03T08:36:34.582+0000 7feefffff640 10 monclient: got monmap 1 from mon.c (according to old e1) 2024-06-03T08:36:34.582 INFO:teuthology.orchestra.run.smithi088.stderr:2024-06-03T08:36:34.582+0000 7feefffff640 10 monclient: dump: 2024-06-03T08:36:34.582 INFO:teuthology.orchestra.run.smithi088.stderr:epoch 1 2024-06-03T08:36:34.582 INFO:teuthology.orchestra.run.smithi088.stderr:fsid ec3c3fdd-3ac6-4d7d-9412-683130ed8b04 2024-06-03T08:36:34.583 INFO:teuthology.orchestra.run.smithi088.stderr:last_changed 2024-06-03T08:34:31.885125+0000 2024-06-03T08:36:34.583 INFO:teuthology.orchestra.run.smithi088.stderr:created 2024-06-03T08:34:31.885125+0000 2024-06-03T08:36:34.583 INFO:teuthology.orchestra.run.smithi088.stderr:min_mon_release 18 (reef) 2024-06-03T08:36:34.583 INFO:teuthology.orchestra.run.smithi088.stderr:election_strategy: 3 2024-06-03T08:36:34.583 INFO:teuthology.orchestra.run.smithi088.stderr:0: v1:172.21.15.88:6789/0 mon.a 2024-06-03T08:36:34.583 INFO:teuthology.orchestra.run.smithi088.stderr:1: v1:172.21.15.192:6789/0 mon.b 2024-06-03T08:36:34.583 INFO:teuthology.orchestra.run.smithi088.stderr:2: v1:172.21.15.88:6790/0 mon.c 2024-06-03T08:36:34.583 INFO:teuthology.orchestra.run.smithi088.stderr: 2024-06-03T08:36:34.584 INFO:teuthology.orchestra.run.smithi088.stderr:2024-06-03T08:36:34.582+0000 7fef15ffb640 4 set_mon_vals no callback set 2024-06-03T08:36:34.584 INFO:teuthology.orchestra.run.smithi088.stderr:2024-06-03T08:36:34.582+0000 7feefffff640 1 -- 172.21.15.88:0/3086147656 <== mon.2 v1:172.21.15.88:6790/0 7 ==== mgrmap(e 5) v1 ==== 89110+0+0 (unknown 873533973 0 0) 0x7fef10005300 con 0x7fef181450f0 2024-06-03T08:36:34.584 INFO:teuthology.orchestra.run.smithi088.stderr:2024-06-03T08:36:34.583+0000 7fef177fe640 10 monclient: get_auth_request con 0x7fef0406d080 auth_method 0 2024-06-03T08:36:34.584 INFO:teuthology.orchestra.run.smithi088.stderr:2024-06-03T08:36:34.583+0000 7feefffff640 10 client.4388.objecter ms_handle_connect 0x7fef0406d080 2024-06-03T08:36:34.585 INFO:teuthology.orchestra.run.smithi088.stderr:2024-06-03T08:36:34.584+0000 7feefffff640 1 -- 172.21.15.88:0/3086147656 <== mon.2 v1:172.21.15.88:6790/0 8 ==== osd_map(26..26 src has 1..26) v4 ==== 4090+0+0 (unknown 3637450554 0 0) 0x7fef100546e0 con 0x7fef181450f0 2024-06-03T08:36:34.585 INFO:teuthology.orchestra.run.smithi088.stderr:2024-06-03T08:36:34.584+0000 7feefffff640 10 client.4388.objecter ms_dispatch 0x7fef18000b90 osd_map(26..26 src has 1..26) v4 2024-06-03T08:36:34.585 INFO:teuthology.orchestra.run.smithi088.stderr:2024-06-03T08:36:34.584+0000 7feefffff640 3 client.4388.objecter handle_osd_map got epochs [26,26] > 0 2024-06-03T08:36:34.585 INFO:teuthology.orchestra.run.smithi088.stderr:2024-06-03T08:36:34.584+0000 7feefffff640 3 client.4388.objecter handle_osd_map decoding full epoch 26 2024-06-03T08:36:34.585 INFO:teuthology.orchestra.run.smithi088.stderr:2024-06-03T08:36:34.584+0000 7feefffff640 20 client.4388.objecter dump_active .. 0 homeless 2024-06-03T08:36:34.590 INFO:teuthology.orchestra.run.smithi088.stderr:2024-06-03T08:36:34.589+0000 7f3deb5b9640 10 monclient: get_monmap_and_config 2024-06-03T08:36:34.590 INFO:teuthology.orchestra.run.smithi088.stderr:2024-06-03T08:36:34.589+0000 7f3deb5b9640 10 monclient: build_initial_monmap 2024-06-03T08:36:34.590 INFO:teuthology.orchestra.run.smithi088.stderr:2024-06-03T08:36:34.589+0000 7f3deb5b9640 10 monclient: monmap: 2024-06-03T08:36:34.590 INFO:teuthology.orchestra.run.smithi088.stderr:epoch 0 2024-06-03T08:36:34.590 INFO:teuthology.orchestra.run.smithi088.stderr:fsid ec3c3fdd-3ac6-4d7d-9412-683130ed8b04 2024-06-03T08:36:34.590 INFO:teuthology.orchestra.run.smithi088.stderr:last_changed 2024-06-03T08:36:34.590918+0000 2024-06-03T08:36:34.590 INFO:teuthology.orchestra.run.smithi088.stderr:created 2024-06-03T08:36:34.590918+0000 2024-06-03T08:36:34.590 INFO:teuthology.orchestra.run.smithi088.stderr:min_mon_release 0 (unknown) 2024-06-03T08:36:34.590 INFO:teuthology.orchestra.run.smithi088.stderr:election_strategy: 3 2024-06-03T08:36:34.590 INFO:teuthology.orchestra.run.smithi088.stderr:0: v1:172.21.15.88:6789/0 mon.noname-a 2024-06-03T08:36:34.591 INFO:teuthology.orchestra.run.smithi088.stderr:1: v1:172.21.15.192:6789/0 mon.noname-c 2024-06-03T08:36:34.591 INFO:teuthology.orchestra.run.smithi088.stderr:2: v1:172.21.15.88:6790/0 mon.noname-b 2024-06-03T08:36:34.591 INFO:teuthology.orchestra.run.smithi088.stderr: 2024-06-03T08:36:34.591 INFO:teuthology.orchestra.run.smithi088.stderr:2024-06-03T08:36:34.590+0000 7f3deb5b9640 1 Processor -- start 2024-06-03T08:36:34.591 INFO:teuthology.orchestra.run.smithi088.stderr:2024-06-03T08:36:34.591+0000 7f3deb5b9640 1 -- start start 2024-06-03T08:36:34.591 INFO:teuthology.orchestra.run.smithi088.stderr:2024-06-03T08:36:34.591+0000 7f3deb5b9640 10 monclient: init 2024-06-03T08:36:34.592 INFO:teuthology.orchestra.run.smithi088.stderr:2024-06-03T08:36:34.592+0000 7f3deb5b9640 10 monclient: _reopen_session rank -1 2024-06-03T08:36:34.592 INFO:teuthology.orchestra.run.smithi088.stderr:2024-06-03T08:36:34.592+0000 7f3deb5b9640 10 monclient: _add_conns ranks=[2,0,1] 2024-06-03T08:36:34.592 INFO:teuthology.orchestra.run.smithi088.stderr:2024-06-03T08:36:34.592+0000 7f3deb5b9640 10 monclient(hunting): picked mon.noname-b con 0x7f3de40a8960 addr v1:172.21.15.88:6790/0 2024-06-03T08:36:34.592 INFO:teuthology.orchestra.run.smithi088.stderr:2024-06-03T08:36:34.592+0000 7f3deb5b9640 10 monclient(hunting): picked mon.noname-a con 0x7f3de41450f0 addr v1:172.21.15.88:6789/0 2024-06-03T08:36:34.592 INFO:teuthology.orchestra.run.smithi088.stderr:2024-06-03T08:36:34.592+0000 7f3deb5b9640 10 monclient(hunting): picked mon.noname-c con 0x7f3de413eec0 addr v1:172.21.15.192:6789/0 2024-06-03T08:36:34.592 INFO:teuthology.orchestra.run.smithi088.stderr:2024-06-03T08:36:34.592+0000 7f3deb5b9640 1 -- --> v1:172.21.15.88:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f3de4144c60 con 0x7f3de41450f0 2024-06-03T08:36:34.593 INFO:teuthology.orchestra.run.smithi088.stderr:2024-06-03T08:36:34.592+0000 7f3deb5b9640 1 -- --> v1:172.21.15.192:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f3de40b3930 con 0x7f3de413eec0 2024-06-03T08:36:34.593 INFO:teuthology.orchestra.run.smithi088.stderr:2024-06-03T08:36:34.592+0000 7f3deb5b9640 1 -- --> v1:172.21.15.88:6790/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f3de40b1ad0 con 0x7f3de40a8960 2024-06-03T08:36:34.593 INFO:teuthology.orchestra.run.smithi088.stderr:2024-06-03T08:36:34.592+0000 7f3deb5b9640 10 monclient(hunting): _renew_subs 2024-06-03T08:36:34.593 INFO:teuthology.orchestra.run.smithi088.stderr:2024-06-03T08:36:34.592+0000 7f3deb5b9640 10 monclient(hunting): authenticate will time out at 1250.402419s 2024-06-03T08:36:34.599 INFO:teuthology.orchestra.run.smithi088.stderr:2024-06-03T08:36:34.599+0000 7fd769ee5640 10 start_mon_command cmd=[{"prefix": "osd last-stat-seq", "id": 2}] 2024-06-03T08:36:34.599 INFO:teuthology.orchestra.run.smithi088.stderr:2024-06-03T08:36:34.599+0000 7fd769ee5640 10 monclient: _send_command 2 [{"prefix": "osd last-stat-seq", "id": 2}] 2024-06-03T08:36:34.599 INFO:teuthology.orchestra.run.smithi088.stderr:2024-06-03T08:36:34.599+0000 7fd769ee5640 10 monclient: _send_mon_message to mon.b at v1:172.21.15.192:6789/0 2024-06-03T08:36:34.599 INFO:teuthology.orchestra.run.smithi088.stderr:2024-06-03T08:36:34.599+0000 7fd769ee5640 1 -- 172.21.15.88:0/3191884890 --> v1:172.21.15.192:6789/0 -- mon_command({"prefix": "osd last-stat-seq", "id": 2} v 0) v1 -- 0x7fd764140be0 con 0x7fd76406ee00 2024-06-03T08:36:34.612 INFO:teuthology.orchestra.run.smithi088.stderr:2024-06-03T08:36:34.612+0000 7f0624b5f640 10 monclient: get_monmap_and_config 2024-06-03T08:36:34.612 INFO:teuthology.orchestra.run.smithi088.stderr:2024-06-03T08:36:34.612+0000 7f0624b5f640 10 monclient: build_initial_monmap 2024-06-03T08:36:34.613 INFO:teuthology.orchestra.run.smithi088.stderr:2024-06-03T08:36:34.612+0000 7f0624b5f640 10 monclient: monmap: 2024-06-03T08:36:34.613 INFO:teuthology.orchestra.run.smithi088.stderr:epoch 0 2024-06-03T08:36:34.613 INFO:teuthology.orchestra.run.smithi088.stderr:fsid ec3c3fdd-3ac6-4d7d-9412-683130ed8b04 2024-06-03T08:36:34.613 INFO:teuthology.orchestra.run.smithi088.stderr:last_changed 2024-06-03T08:36:34.613529+0000 2024-06-03T08:36:34.613 INFO:teuthology.orchestra.run.smithi088.stderr:created 2024-06-03T08:36:34.613529+0000 2024-06-03T08:36:34.613 INFO:teuthology.orchestra.run.smithi088.stderr:min_mon_release 0 (unknown) 2024-06-03T08:36:34.613 INFO:teuthology.orchestra.run.smithi088.stderr:election_strategy: 3 2024-06-03T08:36:34.613 INFO:teuthology.orchestra.run.smithi088.stderr:0: v1:172.21.15.88:6789/0 mon.noname-a 2024-06-03T08:36:34.613 INFO:teuthology.orchestra.run.smithi088.stderr:1: v1:172.21.15.192:6789/0 mon.noname-c 2024-06-03T08:36:34.613 INFO:teuthology.orchestra.run.smithi088.stderr:2: v1:172.21.15.88:6790/0 mon.noname-b 2024-06-03T08:36:34.613 INFO:teuthology.orchestra.run.smithi088.stderr: 2024-06-03T08:36:34.614 INFO:teuthology.orchestra.run.smithi088.stderr:2024-06-03T08:36:34.613+0000 7f0624b5f640 1 Processor -- start 2024-06-03T08:36:34.614 INFO:teuthology.orchestra.run.smithi088.stderr:2024-06-03T08:36:34.613+0000 7f0624b5f640 1 -- start start 2024-06-03T08:36:34.614 INFO:teuthology.orchestra.run.smithi088.stderr:2024-06-03T08:36:34.613+0000 7f0624b5f640 10 monclient: init 2024-06-03T08:36:34.614 INFO:teuthology.orchestra.run.smithi088.stderr:2024-06-03T08:36:34.613+0000 7f0624b5f640 10 monclient: _reopen_session rank -1 2024-06-03T08:36:34.614 INFO:teuthology.orchestra.run.smithi088.stderr:2024-06-03T08:36:34.613+0000 7f0624b5f640 10 monclient: _add_conns ranks=[2,0,1] 2024-06-03T08:36:34.614 INFO:teuthology.orchestra.run.smithi088.stderr:2024-06-03T08:36:34.613+0000 7f0624b5f640 10 monclient(hunting): picked mon.noname-b con 0x7f062006b9a0 addr v1:172.21.15.88:6790/0 2024-06-03T08:36:34.615 INFO:teuthology.orchestra.run.smithi088.stderr:2024-06-03T08:36:34.613+0000 7f0624b5f640 10 monclient(hunting): picked mon.noname-a con 0x7f06200ad980 addr v1:172.21.15.88:6789/0 2024-06-03T08:36:34.615 INFO:teuthology.orchestra.run.smithi088.stderr:2024-06-03T08:36:34.613+0000 7f0624b5f640 10 monclient(hunting): picked mon.noname-c con 0x7f06200ae540 addr v1:172.21.15.192:6789/0 2024-06-03T08:36:34.615 INFO:teuthology.orchestra.run.smithi088.stderr:2024-06-03T08:36:34.613+0000 7f0624b5f640 1 -- --> v1:172.21.15.88:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f062006c150 con 0x7f06200ad980 2024-06-03T08:36:34.615 INFO:teuthology.orchestra.run.smithi088.stderr:2024-06-03T08:36:34.613+0000 7f0624b5f640 1 -- --> v1:172.21.15.192:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f06200ad280 con 0x7f06200ae540 2024-06-03T08:36:34.615 INFO:teuthology.orchestra.run.smithi088.stderr:2024-06-03T08:36:34.613+0000 7f0624b5f640 1 -- --> v1:172.21.15.88:6790/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f06200ad560 con 0x7f062006b9a0 2024-06-03T08:36:34.615 INFO:teuthology.orchestra.run.smithi088.stderr:2024-06-03T08:36:34.613+0000 7f0624b5f640 10 monclient(hunting): _renew_subs 2024-06-03T08:36:34.615 INFO:teuthology.orchestra.run.smithi088.stderr:2024-06-03T08:36:34.613+0000 7f0624b5f640 10 monclient(hunting): authenticate will time out at 1250.423877s 2024-06-03T08:36:34.615 INFO:teuthology.orchestra.run.smithi088.stderr:2024-06-03T08:36:34.614+0000 7f061e59c640 1 --1- >> v1:172.21.15.88:6790/0 conn(0x7f062006b9a0 0x7f06200af940 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.88:6790/0 says I am v1:172.21.15.88:54470/0 (socket says 172.21.15.88:54470) 2024-06-03T08:36:34.615 INFO:teuthology.orchestra.run.smithi088.stderr:2024-06-03T08:36:34.614+0000 7f061e59c640 1 -- 172.21.15.88:0/2062031403 learned_addr learned my addr 172.21.15.88:0/2062031403 (peer_addr_for_me v1:172.21.15.88:0/0) 2024-06-03T08:36:34.616 INFO:teuthology.orchestra.run.smithi088.stderr:2024-06-03T08:36:34.615+0000 7f3de9355640 1 --1- >> v1:172.21.15.88:6790/0 conn(0x7f3de40a8960 0x7f3de40a8d40 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.88:6790/0 says I am v1:172.21.15.88:54468/0 (socket says 172.21.15.88:54468) 2024-06-03T08:36:34.616 INFO:teuthology.orchestra.run.smithi088.stderr:2024-06-03T08:36:34.615+0000 7f3de9355640 1 -- 172.21.15.88:0/3790771176 learned_addr learned my addr 172.21.15.88:0/3790771176 (peer_addr_for_me v1:172.21.15.88:0/0) 2024-06-03T08:36:34.671 INFO:teuthology.orchestra.run.smithi088.stderr:2024-06-03T08:36:34.670+0000 7f29317fa640 1 -- 172.21.15.88:0/248786529 <== mon.1 v1:172.21.15.192:6789/0 9 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+182498 (unknown 1092875540 0 3016760700) 0x7f292404fd50 con 0x7f29340a89f0 2024-06-03T08:36:34.671 INFO:teuthology.orchestra.run.smithi088.stderr:2024-06-03T08:36:34.670+0000 7f29317fa640 10 monclient: handle_mon_command_ack 1 [{"prefix": "get_command_descriptions"}] 2024-06-03T08:36:34.671 INFO:teuthology.orchestra.run.smithi088.stderr:2024-06-03T08:36:34.670+0000 7f29317fa640 10 monclient: _finish_command 1 = system:0 2024-06-03T08:36:34.677 INFO:teuthology.orchestra.run.smithi088.stderr:2024-06-03T08:36:34.677+0000 7f061dd9b640 1 -- 172.21.15.88:0/2062031403 <== mon.1 v1:172.21.15.192:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 2995955515 0 0) 0x7f06200ad280 con 0x7f06200ae540 2024-06-03T08:36:34.677 INFO:teuthology.orchestra.run.smithi088.stderr:2024-06-03T08:36:34.677+0000 7f061dd9b640 10 monclient(hunting): _negotiate 2024-06-03T08:36:34.677 INFO:teuthology.orchestra.run.smithi088.stderr:2024-06-03T08:36:34.677+0000 7f061dd9b640 10 monclient(hunting): _init_auth method 2 2024-06-03T08:36:34.677 INFO:teuthology.orchestra.run.smithi088.stderr:2024-06-03T08:36:34.677+0000 7f061dd9b640 10 monclient(hunting): _init_auth creating new auth 2024-06-03T08:36:34.677 INFO:teuthology.orchestra.run.smithi088.stderr:2024-06-03T08:36:34.677+0000 7f061dd9b640 10 monclient(hunting): my global_id is 4375 2024-06-03T08:36:34.678 INFO:teuthology.orchestra.run.smithi088.stderr:2024-06-03T08:36:34.677+0000 7f061dd9b640 1 -- 172.21.15.88:0/2062031403 --> v1:172.21.15.192:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f060c003520 con 0x7f06200ae540 2024-06-03T08:36:34.683 INFO:teuthology.orchestra.run.smithi088.stderr:2024-06-03T08:36:34.682+0000 7f061dd9b640 1 -- 172.21.15.88:0/2062031403 <== mon.1 v1:172.21.15.192:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 214+0+0 (unknown 2011262062 0 0) 0x7f060c003520 con 0x7f06200ae540 2024-06-03T08:36:34.683 INFO:teuthology.orchestra.run.smithi088.stderr:2024-06-03T08:36:34.682+0000 7f061dd9b640 1 -- 172.21.15.88:0/2062031403 >> v1:172.21.15.88:6790/0 conn(0x7f062006b9a0 legacy=0x7f06200af940 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-06-03T08:36:34.683 INFO:teuthology.orchestra.run.smithi088.stderr:2024-06-03T08:36:34.682+0000 7f061dd9b640 1 -- 172.21.15.88:0/2062031403 >> v1:172.21.15.88:6789/0 conn(0x7f06200ad980 legacy=0x7f06200ade30 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-06-03T08:36:34.683 INFO:teuthology.orchestra.run.smithi088.stderr:2024-06-03T08:36:34.682+0000 7f061dd9b640 10 monclient: _finish_hunting 0 2024-06-03T08:36:34.683 INFO:teuthology.orchestra.run.smithi088.stderr:2024-06-03T08:36:34.682+0000 7f061dd9b640 1 monclient: found mon.noname-c 2024-06-03T08:36:34.683 INFO:teuthology.orchestra.run.smithi088.stderr:2024-06-03T08:36:34.682+0000 7f061dd9b640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-06-03T08:36:34.683 INFO:teuthology.orchestra.run.smithi088.stderr:2024-06-03T08:36:34.682+0000 7f061dd9b640 10 monclient: _send_mon_message to mon.noname-c at v1:172.21.15.192:6789/0 2024-06-03T08:36:34.683 INFO:teuthology.orchestra.run.smithi088.stderr:2024-06-03T08:36:34.682+0000 7f061dd9b640 1 -- 172.21.15.88:0/2062031403 --> v1:172.21.15.192:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f062006c740 con 0x7f06200ae540 2024-06-03T08:36:34.683 INFO:teuthology.orchestra.run.smithi088.stderr:2024-06-03T08:36:34.682+0000 7f061dd9b640 10 monclient: _finish_auth 0 2024-06-03T08:36:34.683 INFO:teuthology.orchestra.run.smithi088.stderr:2024-06-03T08:36:34.682+0000 7f061dd9b640 10 monclient: _check_auth_tickets 2024-06-03T08:36:34.683 INFO:teuthology.orchestra.run.smithi088.stderr:2024-06-03T08:36:34.682+0000 7f061dd9b640 20 monclient: _check_auth_rotating not needed by client.admin 2024-06-03T08:36:34.683 INFO:teuthology.orchestra.run.smithi088.stderr:2024-06-03T08:36:34.682+0000 7f061dd9b640 1 -- 172.21.15.88:0/2062031403 <== mon.1 v1:172.21.15.192:6789/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 2234374065 0 0) 0x7f0618001990 con 0x7f06200ae540 2024-06-03T08:36:34.683 INFO:teuthology.orchestra.run.smithi088.stderr:2024-06-03T08:36:34.682+0000 7f061dd9b640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-06-03T08:36:34.683 INFO:teuthology.orchestra.run.smithi088.stderr:2024-06-03T08:36:34.682+0000 7f061dd9b640 10 monclient: got monmap 1 from mon.noname-c (according to old e1) 2024-06-03T08:36:34.684 INFO:teuthology.orchestra.run.smithi088.stderr:2024-06-03T08:36:34.682+0000 7f061dd9b640 10 monclient: dump: 2024-06-03T08:36:34.684 INFO:teuthology.orchestra.run.smithi088.stderr:epoch 1 2024-06-03T08:36:34.684 INFO:teuthology.orchestra.run.smithi088.stderr:fsid ec3c3fdd-3ac6-4d7d-9412-683130ed8b04 2024-06-03T08:36:34.684 INFO:teuthology.orchestra.run.smithi088.stderr:last_changed 2024-06-03T08:34:31.885125+0000 2024-06-03T08:36:34.684 INFO:teuthology.orchestra.run.smithi088.stderr:created 2024-06-03T08:34:31.885125+0000 2024-06-03T08:36:34.684 INFO:teuthology.orchestra.run.smithi088.stderr:min_mon_release 18 (reef) 2024-06-03T08:36:34.684 INFO:teuthology.orchestra.run.smithi088.stderr:election_strategy: 3 2024-06-03T08:36:34.684 INFO:teuthology.orchestra.run.smithi088.stderr:0: v1:172.21.15.88:6789/0 mon.a 2024-06-03T08:36:34.684 INFO:teuthology.orchestra.run.smithi088.stderr:1: v1:172.21.15.192:6789/0 mon.b 2024-06-03T08:36:34.684 INFO:teuthology.orchestra.run.smithi088.stderr:2: v1:172.21.15.88:6790/0 mon.c 2024-06-03T08:36:34.684 INFO:teuthology.orchestra.run.smithi088.stderr: 2024-06-03T08:36:34.684 INFO:teuthology.orchestra.run.smithi088.stderr:2024-06-03T08:36:34.682+0000 7f0624b5f640 5 monclient: authenticate success, global_id 4375 2024-06-03T08:36:34.684 INFO:teuthology.orchestra.run.smithi088.stderr:2024-06-03T08:36:34.682+0000 7f0624b5f640 20 monclient: get_monmap_and_config waiting for monmap|config 2024-06-03T08:36:34.693 INFO:teuthology.orchestra.run.smithi088.stderr:2024-06-03T08:36:34.692+0000 7f061dd9b640 1 -- 172.21.15.88:0/2062031403 <== mon.1 v1:172.21.15.192:6789/0 4 ==== config(0 keys) v1 ==== 4+0+0 (unknown 0 0 0) 0x7f0618001bb0 con 0x7f06200ae540 2024-06-03T08:36:34.693 INFO:teuthology.orchestra.run.smithi088.stderr:2024-06-03T08:36:34.692+0000 7f061dd9b640 10 monclient: handle_config config(0 keys) v1 2024-06-03T08:36:34.693 INFO:teuthology.orchestra.run.smithi088.stderr:2024-06-03T08:36:34.693+0000 7f0624b5f640 10 monclient: get_monmap_and_config success 2024-06-03T08:36:34.693 INFO:teuthology.orchestra.run.smithi088.stderr:2024-06-03T08:36:34.693+0000 7f0624b5f640 4 set_mon_vals no callback set 2024-06-03T08:36:34.693 INFO:teuthology.orchestra.run.smithi088.stderr:2024-06-03T08:36:34.693+0000 7f061dd9b640 1 -- 172.21.15.88:0/2062031403 <== mon.1 v1:172.21.15.192:6789/0 5 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 2234374065 0 0) 0x7f0618001fc0 con 0x7f06200ae540 2024-06-03T08:36:34.693 INFO:teuthology.orchestra.run.smithi088.stderr:2024-06-03T08:36:34.693+0000 7f061dd9b640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-06-03T08:36:34.693 INFO:teuthology.orchestra.run.smithi088.stderr:2024-06-03T08:36:34.693+0000 7f061dd9b640 10 monclient: got monmap 1 from mon.b (according to old e1) 2024-06-03T08:36:34.693 INFO:teuthology.orchestra.run.smithi088.stderr:2024-06-03T08:36:34.693+0000 7f061dd9b640 10 monclient: dump: 2024-06-03T08:36:34.694 INFO:teuthology.orchestra.run.smithi088.stderr:epoch 1 2024-06-03T08:36:34.694 INFO:teuthology.orchestra.run.smithi088.stderr:fsid ec3c3fdd-3ac6-4d7d-9412-683130ed8b04 2024-06-03T08:36:34.694 INFO:teuthology.orchestra.run.smithi088.stderr:last_changed 2024-06-03T08:34:31.885125+0000 2024-06-03T08:36:34.694 INFO:teuthology.orchestra.run.smithi088.stderr:created 2024-06-03T08:34:31.885125+0000 2024-06-03T08:36:34.694 INFO:teuthology.orchestra.run.smithi088.stderr:min_mon_release 18 (reef) 2024-06-03T08:36:34.694 INFO:teuthology.orchestra.run.smithi088.stderr:election_strategy: 3 2024-06-03T08:36:34.694 INFO:teuthology.orchestra.run.smithi088.stderr:0: v1:172.21.15.88:6789/0 mon.a 2024-06-03T08:36:34.694 INFO:teuthology.orchestra.run.smithi088.stderr:1: v1:172.21.15.192:6789/0 mon.b 2024-06-03T08:36:34.694 INFO:teuthology.orchestra.run.smithi088.stderr:2: v1:172.21.15.88:6790/0 mon.c 2024-06-03T08:36:34.694 INFO:teuthology.orchestra.run.smithi088.stderr: 2024-06-03T08:36:34.695 INFO:teuthology.orchestra.run.smithi088.stderr:2024-06-03T08:36:34.693+0000 7f3ddbfff640 1 -- 172.21.15.88:0/3790771176 <== mon.1 v1:172.21.15.192:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 1142973399 0 0) 0x7f3de40b3930 con 0x7f3de413eec0 2024-06-03T08:36:34.695 INFO:teuthology.orchestra.run.smithi088.stderr:2024-06-03T08:36:34.693+0000 7f3ddbfff640 10 monclient(hunting): _negotiate 2024-06-03T08:36:34.695 INFO:teuthology.orchestra.run.smithi088.stderr:2024-06-03T08:36:34.693+0000 7f3ddbfff640 10 monclient(hunting): _init_auth method 2 2024-06-03T08:36:34.695 INFO:teuthology.orchestra.run.smithi088.stderr:2024-06-03T08:36:34.693+0000 7f3ddbfff640 10 monclient(hunting): _init_auth creating new auth 2024-06-03T08:36:34.695 INFO:teuthology.orchestra.run.smithi088.stderr:2024-06-03T08:36:34.693+0000 7f3ddbfff640 10 monclient(hunting): my global_id is 4378 2024-06-03T08:36:34.695 INFO:teuthology.orchestra.run.smithi088.stderr:2024-06-03T08:36:34.693+0000 7f3ddbfff640 1 -- 172.21.15.88:0/3790771176 --> v1:172.21.15.192:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f3dd0003520 con 0x7f3de413eec0 2024-06-03T08:36:34.697 INFO:teuthology.orchestra.run.smithi088.stderr:2024-06-03T08:36:34.693+0000 7f0624b5f640 10 monclient: shutdown 2024-06-03T08:36:34.697 INFO:teuthology.orchestra.run.smithi088.stderr:2024-06-03T08:36:34.693+0000 7f0624b5f640 20 monclient: shutdown discarding 0 pending message(s) 2024-06-03T08:36:34.697 INFO:teuthology.orchestra.run.smithi088.stderr:2024-06-03T08:36:34.693+0000 7f0624b5f640 1 -- 172.21.15.88:0/2062031403 >> v1:172.21.15.192:6789/0 conn(0x7f06200ae540 legacy=0x7f06200ae960 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-06-03T08:36:34.697 INFO:teuthology.orchestra.run.smithi088.stderr:2024-06-03T08:36:34.693+0000 7f0624b5f640 1 -- 172.21.15.88:0/2062031403 shutdown_connections 2024-06-03T08:36:34.697 INFO:teuthology.orchestra.run.smithi088.stderr:2024-06-03T08:36:34.693+0000 7f0624b5f640 1 -- 172.21.15.88:0/2062031403 >> 172.21.15.88:0/2062031403 conn(0x7f06200b5120 msgr2=0x7f06200b5530 unknown :-1 s=STATE_NONE l=0).mark_down 2024-06-03T08:36:34.697 INFO:teuthology.orchestra.run.smithi088.stderr:2024-06-03T08:36:34.693+0000 7f0624b5f640 1 -- 172.21.15.88:0/2062031403 shutdown_connections 2024-06-03T08:36:34.697 INFO:teuthology.orchestra.run.smithi088.stderr:2024-06-03T08:36:34.693+0000 7f0624b5f640 1 -- 172.21.15.88:0/2062031403 wait complete. 2024-06-03T08:36:34.697 INFO:teuthology.orchestra.run.smithi088.stderr:2024-06-03T08:36:34.694+0000 7f0624b5f640 10 monclient: build_initial_monmap 2024-06-03T08:36:34.697 INFO:teuthology.orchestra.run.smithi088.stderr:2024-06-03T08:36:34.694+0000 7f0624b5f640 10 monclient: monmap: 2024-06-03T08:36:34.697 INFO:teuthology.orchestra.run.smithi088.stderr:epoch 0 2024-06-03T08:36:34.697 INFO:teuthology.orchestra.run.smithi088.stderr:fsid 00000000-0000-0000-0000-000000000000 2024-06-03T08:36:34.697 INFO:teuthology.orchestra.run.smithi088.stderr:last_changed 0.000000 2024-06-03T08:36:34.698 INFO:teuthology.orchestra.run.smithi088.stderr:created 0.000000 2024-06-03T08:36:34.698 INFO:teuthology.orchestra.run.smithi088.stderr:min_mon_release 0 (unknown) 2024-06-03T08:36:34.698 INFO:teuthology.orchestra.run.smithi088.stderr:election_strategy: 1 2024-06-03T08:36:34.698 INFO:teuthology.orchestra.run.smithi088.stderr:0: v1:172.21.15.88:6789/0 mon.noname-a 2024-06-03T08:36:34.698 INFO:teuthology.orchestra.run.smithi088.stderr:1: v1:172.21.15.192:6789/0 mon.noname-b 2024-06-03T08:36:34.698 INFO:teuthology.orchestra.run.smithi088.stderr:2: v1:172.21.15.88:6790/0 mon.noname-c 2024-06-03T08:36:34.698 INFO:teuthology.orchestra.run.smithi088.stderr: 2024-06-03T08:36:34.698 INFO:teuthology.orchestra.run.smithi088.stderr:2024-06-03T08:36:34.694+0000 7f0624b5f640 1 librados: starting msgr at 2024-06-03T08:36:34.698 INFO:teuthology.orchestra.run.smithi088.stderr:2024-06-03T08:36:34.694+0000 7f0624b5f640 1 librados: starting objecter 2024-06-03T08:36:34.698 INFO:teuthology.orchestra.run.smithi088.stderr:2024-06-03T08:36:34.695+0000 7f0624b5f640 1 Processor -- start 2024-06-03T08:36:34.698 INFO:teuthology.orchestra.run.smithi088.stderr:2024-06-03T08:36:34.695+0000 7f0624b5f640 1 -- start start 2024-06-03T08:36:34.698 INFO:teuthology.orchestra.run.smithi088.stderr:2024-06-03T08:36:34.695+0000 7f0624b5f640 1 librados: setting wanted keys 2024-06-03T08:36:34.698 INFO:teuthology.orchestra.run.smithi088.stderr:2024-06-03T08:36:34.695+0000 7f0624b5f640 1 librados: calling monclient init 2024-06-03T08:36:34.698 INFO:teuthology.orchestra.run.smithi088.stderr:2024-06-03T08:36:34.695+0000 7f0624b5f640 10 monclient: init 2024-06-03T08:36:34.699 INFO:teuthology.orchestra.run.smithi088.stderr:2024-06-03T08:36:34.695+0000 7f0624b5f640 10 monclient: _reopen_session rank -1 2024-06-03T08:36:34.699 INFO:teuthology.orchestra.run.smithi088.stderr:2024-06-03T08:36:34.695+0000 7f0624b5f640 10 monclient: _add_conns ranks=[0,1,2] 2024-06-03T08:36:34.699 INFO:teuthology.orchestra.run.smithi088.stderr:2024-06-03T08:36:34.695+0000 7f0624b5f640 10 monclient(hunting): picked mon.noname-a con 0x7f062006b9a0 addr v1:172.21.15.88:6789/0 2024-06-03T08:36:34.699 INFO:teuthology.orchestra.run.smithi088.stderr:2024-06-03T08:36:34.695+0000 7f0624b5f640 10 monclient(hunting): picked mon.noname-b con 0x7f06200ad980 addr v1:172.21.15.192:6789/0 2024-06-03T08:36:34.699 INFO:teuthology.orchestra.run.smithi088.stderr:2024-06-03T08:36:34.695+0000 7f0624b5f640 10 monclient(hunting): picked mon.noname-c con 0x7f06200ae540 addr v1:172.21.15.88:6790/0 2024-06-03T08:36:34.699 INFO:teuthology.orchestra.run.smithi088.stderr:2024-06-03T08:36:34.695+0000 7f0624b5f640 1 -- --> v1:172.21.15.88:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f062012ef10 con 0x7f062006b9a0 2024-06-03T08:36:34.699 INFO:teuthology.orchestra.run.smithi088.stderr:2024-06-03T08:36:34.695+0000 7f0624b5f640 1 -- --> v1:172.21.15.192:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f0620134550 con 0x7f06200ad980 2024-06-03T08:36:34.699 INFO:teuthology.orchestra.run.smithi088.stderr:2024-06-03T08:36:34.695+0000 7f0624b5f640 1 -- --> v1:172.21.15.88:6790/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f06201393b0 con 0x7f06200ae540 2024-06-03T08:36:34.699 INFO:teuthology.orchestra.run.smithi088.stderr:2024-06-03T08:36:34.695+0000 7f0624b5f640 10 monclient(hunting): _renew_subs 2024-06-03T08:36:34.699 INFO:teuthology.orchestra.run.smithi088.stderr:2024-06-03T08:36:34.695+0000 7f0624b5f640 10 monclient(hunting): authenticate will time out at 1250.505661s 2024-06-03T08:36:34.700 INFO:teuthology.orchestra.run.smithi088.stderr:2024-06-03T08:36:34.699+0000 7fd74bfff640 1 -- 172.21.15.88:0/3191884890 <== mon.1 v1:172.21.15.192:6789/0 10 ==== mon_command_ack([{"prefix": "osd last-stat-seq", "id": 2}]=0 v0) v1 ==== 74+0+12 (unknown 92182286 0 3389660797) 0x7fd754051bc0 con 0x7fd76406ee00 2024-06-03T08:36:34.700 INFO:teuthology.orchestra.run.smithi088.stderr:2024-06-03T08:36:34.699+0000 7fd74bfff640 10 monclient: handle_mon_command_ack 2 [{"prefix": "osd last-stat-seq", "id": 2}] 2024-06-03T08:36:34.700 INFO:teuthology.orchestra.run.smithi088.stderr:2024-06-03T08:36:34.699+0000 7fd74bfff640 10 monclient: _finish_command 2 = system:0 2024-06-03T08:36:34.700 INFO:teuthology.orchestra.run.smithi088.stdout:85899345929 2024-06-03T08:36:34.701 INFO:teuthology.orchestra.run.smithi088.stderr:2024-06-03T08:36:34.700+0000 7f3ddbfff640 1 -- 172.21.15.88:0/3790771176 <== mon.1 v1:172.21.15.192:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 214+0+0 (unknown 2945274170 0 0) 0x7f3dd0003520 con 0x7f3de413eec0 2024-06-03T08:36:34.701 INFO:teuthology.orchestra.run.smithi088.stderr:2024-06-03T08:36:34.700+0000 7f3ddbfff640 1 -- 172.21.15.88:0/3790771176 >> v1:172.21.15.88:6790/0 conn(0x7f3de40a8960 legacy=0x7f3de40a8d40 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-06-03T08:36:34.701 INFO:teuthology.orchestra.run.smithi088.stderr:2024-06-03T08:36:34.700+0000 7f3ddbfff640 1 -- 172.21.15.88:0/3790771176 >> v1:172.21.15.88:6789/0 conn(0x7f3de41450f0 legacy=0x7f3de413e7b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-06-03T08:36:34.701 INFO:teuthology.orchestra.run.smithi088.stderr:2024-06-03T08:36:34.700+0000 7f3ddbfff640 10 monclient: _finish_hunting 0 2024-06-03T08:36:34.701 INFO:teuthology.orchestra.run.smithi088.stderr:2024-06-03T08:36:34.700+0000 7f3ddbfff640 1 monclient: found mon.noname-c 2024-06-03T08:36:34.701 INFO:teuthology.orchestra.run.smithi088.stderr:2024-06-03T08:36:34.700+0000 7f3ddbfff640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-06-03T08:36:34.701 INFO:teuthology.orchestra.run.smithi088.stderr:2024-06-03T08:36:34.700+0000 7f3ddbfff640 10 monclient: _send_mon_message to mon.noname-c at v1:172.21.15.192:6789/0 2024-06-03T08:36:34.701 INFO:teuthology.orchestra.run.smithi088.stderr:2024-06-03T08:36:34.700+0000 7f3ddbfff640 1 -- 172.21.15.88:0/3790771176 --> v1:172.21.15.192:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f3de40b1c60 con 0x7f3de413eec0 2024-06-03T08:36:34.701 INFO:teuthology.orchestra.run.smithi088.stderr:2024-06-03T08:36:34.700+0000 7f3ddbfff640 10 monclient: _finish_auth 0 2024-06-03T08:36:34.702 INFO:teuthology.orchestra.run.smithi088.stderr:2024-06-03T08:36:34.700+0000 7f3ddbfff640 10 monclient: _check_auth_tickets 2024-06-03T08:36:34.702 INFO:teuthology.orchestra.run.smithi088.stderr:2024-06-03T08:36:34.700+0000 7f3ddbfff640 20 monclient: _check_auth_rotating not needed by client.admin 2024-06-03T08:36:34.702 INFO:teuthology.orchestra.run.smithi088.stderr:2024-06-03T08:36:34.700+0000 7f3ddbfff640 1 -- 172.21.15.88:0/3790771176 <== mon.1 v1:172.21.15.192:6789/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 2234374065 0 0) 0x7f3ddc001990 con 0x7f3de413eec0 2024-06-03T08:36:34.702 INFO:teuthology.orchestra.run.smithi088.stderr:2024-06-03T08:36:34.700+0000 7f3deb5b9640 5 monclient: authenticate success, global_id 4378 2024-06-03T08:36:34.702 INFO:teuthology.orchestra.run.smithi088.stderr:2024-06-03T08:36:34.700+0000 7f3deb5b9640 20 monclient: get_monmap_and_config waiting for monmap|config 2024-06-03T08:36:34.702 INFO:teuthology.orchestra.run.smithi088.stderr:2024-06-03T08:36:34.700+0000 7f3ddbfff640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-06-03T08:36:34.702 INFO:teuthology.orchestra.run.smithi088.stderr:2024-06-03T08:36:34.700+0000 7f3ddbfff640 10 monclient: got monmap 1 from mon.noname-c (according to old e1) 2024-06-03T08:36:34.702 INFO:teuthology.orchestra.run.smithi088.stderr:2024-06-03T08:36:34.700+0000 7f3ddbfff640 10 monclient: dump: 2024-06-03T08:36:34.702 INFO:teuthology.orchestra.run.smithi088.stderr:epoch 1 2024-06-03T08:36:34.702 INFO:teuthology.orchestra.run.smithi088.stderr:fsid ec3c3fdd-3ac6-4d7d-9412-683130ed8b04 2024-06-03T08:36:34.702 INFO:teuthology.orchestra.run.smithi088.stderr:last_changed 2024-06-03T08:34:31.885125+0000 2024-06-03T08:36:34.702 INFO:teuthology.orchestra.run.smithi088.stderr:created 2024-06-03T08:34:31.885125+0000 2024-06-03T08:36:34.702 INFO:teuthology.orchestra.run.smithi088.stderr:min_mon_release 18 (reef) 2024-06-03T08:36:34.703 INFO:teuthology.orchestra.run.smithi088.stderr:election_strategy: 3 2024-06-03T08:36:34.703 INFO:teuthology.orchestra.run.smithi088.stderr:0: v1:172.21.15.88:6789/0 mon.a 2024-06-03T08:36:34.703 INFO:teuthology.orchestra.run.smithi088.stderr:1: v1:172.21.15.192:6789/0 mon.b 2024-06-03T08:36:34.703 INFO:teuthology.orchestra.run.smithi088.stderr:2: v1:172.21.15.88:6790/0 mon.c 2024-06-03T08:36:34.703 INFO:teuthology.orchestra.run.smithi088.stderr: 2024-06-03T08:36:34.703 INFO:teuthology.orchestra.run.smithi088.stderr:2024-06-03T08:36:34.700+0000 7f3deb5b9640 20 monclient: get_monmap_and_config waiting for monmap|config 2024-06-03T08:36:34.703 INFO:teuthology.orchestra.run.smithi088.stderr:2024-06-03T08:36:34.700+0000 7f0617fff640 1 --1- >> v1:172.21.15.192:6789/0 conn(0x7f06200ad980 0x7f062012e0f0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=1).handle_server_banner_and_identify peer v1:172.21.15.192:6789/0 says I am v1:172.21.15.88:52720/0 (socket says 172.21.15.88:52720) 2024-06-03T08:36:34.703 INFO:teuthology.orchestra.run.smithi088.stderr:2024-06-03T08:36:34.700+0000 7f0617fff640 1 -- 172.21.15.88:0/228619988 learned_addr learned my addr 172.21.15.88:0/228619988 (peer_addr_for_me v1:172.21.15.88:0/0) 2024-06-03T08:36:34.704 INFO:teuthology.orchestra.run.smithi088.stderr:2024-06-03T08:36:34.701+0000 7fd769ee5640 10 librados: watch_flush enter 2024-06-03T08:36:34.704 INFO:teuthology.orchestra.run.smithi088.stderr:2024-06-03T08:36:34.701+0000 7fd769ee5640 10 librados: watch_flush exit 2024-06-03T08:36:34.704 INFO:teuthology.orchestra.run.smithi088.stderr:2024-06-03T08:36:34.701+0000 7fd769ee5640 20 client.4369.objecter shutdown clearing up homeless session... 2024-06-03T08:36:34.704 INFO:teuthology.orchestra.run.smithi088.stderr:2024-06-03T08:36:34.701+0000 7fd769ee5640 10 client.4369.objecter successfully canceled tick 2024-06-03T08:36:34.704 INFO:teuthology.orchestra.run.smithi088.stderr:2024-06-03T08:36:34.701+0000 7fd769ee5640 1 -- 172.21.15.88:0/3191884890 >> v1:172.21.15.192:6800/34229 conn(0x7fd75006d3d0 legacy=0x7fd75006f890 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-06-03T08:36:34.704 INFO:teuthology.orchestra.run.smithi088.stderr:2024-06-03T08:36:34.701+0000 7fd769ee5640 10 monclient: shutdown 2024-06-03T08:36:34.704 INFO:teuthology.orchestra.run.smithi088.stderr:2024-06-03T08:36:34.701+0000 7fd769ee5640 20 monclient: shutdown discarding 0 pending message(s) 2024-06-03T08:36:34.704 INFO:teuthology.orchestra.run.smithi088.stderr:2024-06-03T08:36:34.701+0000 7fd769ee5640 1 -- 172.21.15.88:0/3191884890 >> v1:172.21.15.192:6789/0 conn(0x7fd76406ee00 legacy=0x7fd76413ceb0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-06-03T08:36:34.704 INFO:teuthology.orchestra.run.smithi088.stderr:2024-06-03T08:36:34.702+0000 7fd769ee5640 1 -- 172.21.15.88:0/3191884890 shutdown_connections 2024-06-03T08:36:34.705 INFO:teuthology.orchestra.run.smithi088.stderr:2024-06-03T08:36:34.702+0000 7f0616ffd640 10 client.?.objecter ms_handle_connect 0x7f06200ad980 2024-06-03T08:36:34.705 INFO:teuthology.orchestra.run.smithi088.stderr:2024-06-03T08:36:34.702+0000 7f0616ffd640 10 client.?.objecter resend_mon_ops 2024-06-03T08:36:34.705 INFO:teuthology.orchestra.run.smithi088.stderr:2024-06-03T08:36:34.702+0000 7fd769ee5640 1 -- 172.21.15.88:0/3191884890 >> 172.21.15.88:0/3191884890 conn(0x7fd7640b2f20 msgr2=0x7fd7640b27a0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-06-03T08:36:34.706 INFO:teuthology.orchestra.run.smithi088.stderr:2024-06-03T08:36:34.702+0000 7fd769ee5640 1 -- 172.21.15.88:0/3191884890 shutdown_connections 2024-06-03T08:36:34.706 INFO:teuthology.orchestra.run.smithi088.stderr:2024-06-03T08:36:34.702+0000 7fd769ee5640 1 -- 172.21.15.88:0/3191884890 wait complete. 2024-06-03T08:36:34.706 INFO:teuthology.orchestra.run.smithi088.stderr:2024-06-03T08:36:34.702+0000 7fd769ee5640 1 librados: shutdown 2024-06-03T08:36:34.707 INFO:teuthology.orchestra.run.smithi088.stderr:2024-06-03T08:36:34.702+0000 7f3ddbfff640 1 -- 172.21.15.88:0/3790771176 <== mon.1 v1:172.21.15.192:6789/0 4 ==== config(0 keys) v1 ==== 4+0+0 (unknown 0 0 0) 0x7f3ddc001bb0 con 0x7f3de413eec0 2024-06-03T08:36:34.708 INFO:teuthology.orchestra.run.smithi088.stderr:2024-06-03T08:36:34.702+0000 7f3ddbfff640 10 monclient: handle_config config(0 keys) v1 2024-06-03T08:36:34.708 INFO:teuthology.orchestra.run.smithi088.stderr:2024-06-03T08:36:34.702+0000 7f3ddbfff640 1 -- 172.21.15.88:0/3790771176 <== mon.1 v1:172.21.15.192:6789/0 5 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 2234374065 0 0) 0x7f3ddc001f40 con 0x7f3de413eec0 2024-06-03T08:36:34.708 INFO:teuthology.orchestra.run.smithi088.stderr:2024-06-03T08:36:34.702+0000 7f3deb5b9640 10 monclient: get_monmap_and_config success 2024-06-03T08:36:34.708 INFO:teuthology.orchestra.run.smithi088.stderr:2024-06-03T08:36:34.702+0000 7f3ddbfff640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-06-03T08:36:34.708 INFO:teuthology.orchestra.run.smithi088.stderr:2024-06-03T08:36:34.702+0000 7f3ddbfff640 10 monclient: got monmap 1 from mon.b (according to old e1) 2024-06-03T08:36:34.708 INFO:teuthology.orchestra.run.smithi088.stderr:2024-06-03T08:36:34.702+0000 7f3ddbfff640 10 monclient: dump: 2024-06-03T08:36:34.708 INFO:teuthology.orchestra.run.smithi088.stderr:epoch 1 2024-06-03T08:36:34.708 INFO:teuthology.orchestra.run.smithi088.stderr:fsid ec3c3fdd-3ac6-4d7d-9412-683130ed8b04 2024-06-03T08:36:34.708 INFO:teuthology.orchestra.run.smithi088.stderr:last_changed 2024-06-03T08:34:31.885125+0000 2024-06-03T08:36:34.708 INFO:teuthology.orchestra.run.smithi088.stderr:created 2024-06-03T08:34:31.885125+0000 2024-06-03T08:36:34.708 INFO:teuthology.orchestra.run.smithi088.stderr:min_mon_release 18 (reef) 2024-06-03T08:36:34.708 INFO:teuthology.orchestra.run.smithi088.stderr:election_strategy: 3 2024-06-03T08:36:34.708 INFO:teuthology.orchestra.run.smithi088.stderr:0: v1:172.21.15.88:6789/0 mon.a 2024-06-03T08:36:34.709 INFO:teuthology.orchestra.run.smithi088.stderr:1: v1:172.21.15.192:6789/0 mon.b 2024-06-03T08:36:34.709 INFO:teuthology.orchestra.run.smithi088.stderr:2: v1:172.21.15.88:6790/0 mon.c 2024-06-03T08:36:34.709 INFO:teuthology.orchestra.run.smithi088.stderr: 2024-06-03T08:36:34.709 INFO:teuthology.orchestra.run.smithi088.stderr:2024-06-03T08:36:34.702+0000 7f3deb5b9640 4 set_mon_vals no callback set 2024-06-03T08:36:34.709 INFO:teuthology.orchestra.run.smithi088.stderr:2024-06-03T08:36:34.703+0000 7f3deb5b9640 10 monclient: shutdown 2024-06-03T08:36:34.709 INFO:teuthology.orchestra.run.smithi088.stderr:2024-06-03T08:36:34.703+0000 7f3deb5b9640 20 monclient: shutdown discarding 0 pending message(s) 2024-06-03T08:36:34.709 INFO:teuthology.orchestra.run.smithi088.stderr:2024-06-03T08:36:34.703+0000 7f3deb5b9640 1 -- 172.21.15.88:0/3790771176 >> v1:172.21.15.192:6789/0 conn(0x7f3de413eec0 legacy=0x7f3de414ce60 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-06-03T08:36:34.709 INFO:teuthology.orchestra.run.smithi088.stderr:2024-06-03T08:36:34.703+0000 7f3deb5b9640 1 -- 172.21.15.88:0/3790771176 shutdown_connections 2024-06-03T08:36:34.709 INFO:teuthology.orchestra.run.smithi088.stderr:2024-06-03T08:36:34.703+0000 7f3deb5b9640 1 -- 172.21.15.88:0/3790771176 >> 172.21.15.88:0/3790771176 conn(0x7f3de40b70d0 msgr2=0x7f3de40b74e0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-06-03T08:36:34.709 INFO:teuthology.orchestra.run.smithi088.stderr:2024-06-03T08:36:34.703+0000 7f3deb5b9640 1 -- 172.21.15.88:0/3790771176 shutdown_connections 2024-06-03T08:36:34.709 INFO:teuthology.orchestra.run.smithi088.stderr:2024-06-03T08:36:34.703+0000 7f3deb5b9640 1 -- 172.21.15.88:0/3790771176 wait complete. 2024-06-03T08:36:34.709 INFO:teuthology.orchestra.run.smithi088.stderr:2024-06-03T08:36:34.704+0000 7f3deb5b9640 10 monclient: build_initial_monmap 2024-06-03T08:36:34.709 INFO:teuthology.orchestra.run.smithi088.stderr:2024-06-03T08:36:34.704+0000 7f3deb5b9640 10 monclient: monmap: 2024-06-03T08:36:34.709 INFO:teuthology.orchestra.run.smithi088.stderr:epoch 0 2024-06-03T08:36:34.710 INFO:teuthology.orchestra.run.smithi088.stderr:fsid 00000000-0000-0000-0000-000000000000 2024-06-03T08:36:34.710 INFO:teuthology.orchestra.run.smithi088.stderr:last_changed 0.000000 2024-06-03T08:36:34.710 INFO:teuthology.orchestra.run.smithi088.stderr:created 0.000000 2024-06-03T08:36:34.710 INFO:teuthology.orchestra.run.smithi088.stderr:min_mon_release 0 (unknown) 2024-06-03T08:36:34.710 INFO:teuthology.orchestra.run.smithi088.stderr:election_strategy: 1 2024-06-03T08:36:34.710 INFO:teuthology.orchestra.run.smithi088.stderr:0: v1:172.21.15.88:6789/0 mon.noname-a 2024-06-03T08:36:34.710 INFO:teuthology.orchestra.run.smithi088.stderr:1: v1:172.21.15.192:6789/0 mon.noname-b 2024-06-03T08:36:34.710 INFO:teuthology.orchestra.run.smithi088.stderr:2: v1:172.21.15.88:6790/0 mon.noname-c 2024-06-03T08:36:34.710 INFO:teuthology.orchestra.run.smithi088.stderr: 2024-06-03T08:36:34.710 INFO:teuthology.orchestra.run.smithi088.stderr:2024-06-03T08:36:34.704+0000 7f3deb5b9640 1 librados: starting msgr at 2024-06-03T08:36:34.710 INFO:teuthology.orchestra.run.smithi088.stderr:2024-06-03T08:36:34.704+0000 7f3deb5b9640 1 librados: starting objecter 2024-06-03T08:36:34.710 INFO:teuthology.orchestra.run.smithi088.stderr:2024-06-03T08:36:34.704+0000 7f3deb5b9640 1 Processor -- start 2024-06-03T08:36:34.710 INFO:teuthology.orchestra.run.smithi088.stderr:2024-06-03T08:36:34.704+0000 7f3deb5b9640 1 -- start start 2024-06-03T08:36:34.711 INFO:teuthology.orchestra.run.smithi088.stderr:2024-06-03T08:36:34.705+0000 7f3deb5b9640 1 librados: setting wanted keys 2024-06-03T08:36:34.711 INFO:teuthology.orchestra.run.smithi088.stderr:2024-06-03T08:36:34.705+0000 7f3deb5b9640 1 librados: calling monclient init 2024-06-03T08:36:34.711 INFO:teuthology.orchestra.run.smithi088.stderr:2024-06-03T08:36:34.705+0000 7f3deb5b9640 10 monclient: init 2024-06-03T08:36:34.711 INFO:teuthology.orchestra.run.smithi088.stderr:2024-06-03T08:36:34.705+0000 7f3deb5b9640 10 monclient: _reopen_session rank -1 2024-06-03T08:36:34.711 INFO:teuthology.orchestra.run.smithi088.stderr:2024-06-03T08:36:34.705+0000 7f3deb5b9640 10 monclient: _add_conns ranks=[0,1,2] 2024-06-03T08:36:34.711 INFO:teuthology.orchestra.run.smithi088.stderr:2024-06-03T08:36:34.705+0000 7f3deb5b9640 10 monclient(hunting): picked mon.noname-a con 0x7f3de40a8960 addr v1:172.21.15.88:6789/0 2024-06-03T08:36:34.711 INFO:teuthology.orchestra.run.smithi088.stderr:2024-06-03T08:36:34.705+0000 7f3deb5b9640 10 monclient(hunting): picked mon.noname-b con 0x7f3de413eec0 addr v1:172.21.15.192:6789/0 2024-06-03T08:36:34.711 INFO:teuthology.orchestra.run.smithi088.stderr:2024-06-03T08:36:34.705+0000 7f3deb5b9640 10 monclient(hunting): picked mon.noname-c con 0x7f3de41450f0 addr v1:172.21.15.88:6790/0 2024-06-03T08:36:34.711 INFO:teuthology.orchestra.run.smithi088.stderr:2024-06-03T08:36:34.705+0000 7f3deb5b9640 1 -- --> v1:172.21.15.88:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f3de40aacb0 con 0x7f3de40a8960 2024-06-03T08:36:34.711 INFO:teuthology.orchestra.run.smithi088.stderr:2024-06-03T08:36:34.705+0000 7f3deb5b9640 1 -- --> v1:172.21.15.192:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f3de4141b20 con 0x7f3de413eec0 2024-06-03T08:36:34.711 INFO:teuthology.orchestra.run.smithi088.stderr:2024-06-03T08:36:34.705+0000 7f3deb5b9640 1 -- --> v1:172.21.15.88:6790/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f3de40ab1f0 con 0x7f3de41450f0 2024-06-03T08:36:34.711 INFO:teuthology.orchestra.run.smithi088.stderr:2024-06-03T08:36:34.705+0000 7f3deb5b9640 10 monclient(hunting): _renew_subs 2024-06-03T08:36:34.711 INFO:teuthology.orchestra.run.smithi088.stderr:2024-06-03T08:36:34.705+0000 7f3deb5b9640 10 monclient(hunting): authenticate will time out at 1250.515566s 2024-06-03T08:36:34.712 INFO:teuthology.orchestra.run.smithi088.stderr:2024-06-03T08:36:34.707+0000 7f0616ffd640 1 -- 172.21.15.88:0/228619988 <== mon.1 v1:172.21.15.192:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 1984726020 0 0) 0x7f0620134550 con 0x7f06200ad980 2024-06-03T08:36:34.712 INFO:teuthology.orchestra.run.smithi088.stderr:2024-06-03T08:36:34.707+0000 7f0616ffd640 10 monclient(hunting): _negotiate 2024-06-03T08:36:34.712 INFO:teuthology.orchestra.run.smithi088.stderr:2024-06-03T08:36:34.707+0000 7f0616ffd640 10 monclient(hunting): _init_auth method 2 2024-06-03T08:36:34.712 INFO:teuthology.orchestra.run.smithi088.stderr:2024-06-03T08:36:34.707+0000 7f0616ffd640 10 monclient(hunting): _init_auth creating new auth 2024-06-03T08:36:34.712 INFO:teuthology.orchestra.run.smithi088.stderr:2024-06-03T08:36:34.707+0000 7f0616ffd640 10 monclient(hunting): my global_id is 4381 2024-06-03T08:36:34.712 INFO:teuthology.orchestra.run.smithi088.stderr:2024-06-03T08:36:34.707+0000 7f0616ffd640 1 -- 172.21.15.88:0/228619988 --> v1:172.21.15.192:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f0600004eb0 con 0x7f06200ad980 2024-06-03T08:36:34.712 INFO:teuthology.orchestra.run.smithi088.stderr:2024-06-03T08:36:34.707+0000 7f3de8b54640 1 --1- >> v1:172.21.15.192:6789/0 conn(0x7f3de413eec0 0x7f3de41d6e60 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=1).handle_server_banner_and_identify peer v1:172.21.15.192:6789/0 says I am v1:172.21.15.88:52722/0 (socket says 172.21.15.88:52722) 2024-06-03T08:36:34.712 INFO:teuthology.orchestra.run.smithi088.stderr:2024-06-03T08:36:34.707+0000 7f3de8b54640 1 -- 172.21.15.88:0/1593134679 learned_addr learned my addr 172.21.15.88:0/1593134679 (peer_addr_for_me v1:172.21.15.88:0/0) 2024-06-03T08:36:34.713 INFO:teuthology.orchestra.run.smithi088.stderr:2024-06-03T08:36:34.708+0000 7f3dd9ffb640 10 client.?.objecter ms_handle_connect 0x7f3de413eec0 2024-06-03T08:36:34.713 INFO:teuthology.orchestra.run.smithi088.stderr:2024-06-03T08:36:34.708+0000 7f3dd9ffb640 10 client.?.objecter resend_mon_ops 2024-06-03T08:36:34.714 INFO:tasks.ceph.ceph_manager.ceph:need seq 85899345929 got 85899345929 for osd.2 2024-06-03T08:36:34.714 DEBUG:teuthology.parallel:result is None 2024-06-03T08:36:34.714 INFO:teuthology.orchestra.run.smithi088.stderr:2024-06-03T08:36:34.714+0000 7f3dd9ffb640 1 -- 172.21.15.88:0/1593134679 <== mon.1 v1:172.21.15.192:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 4143902868 0 0) 0x7f3de4141b20 con 0x7f3de413eec0 2024-06-03T08:36:34.714 INFO:teuthology.orchestra.run.smithi088.stderr:2024-06-03T08:36:34.714+0000 7f3dd9ffb640 10 monclient(hunting): _negotiate 2024-06-03T08:36:34.715 INFO:teuthology.orchestra.run.smithi088.stderr:2024-06-03T08:36:34.714+0000 7f3dd9ffb640 10 monclient(hunting): _init_auth method 2 2024-06-03T08:36:34.715 INFO:teuthology.orchestra.run.smithi088.stderr:2024-06-03T08:36:34.714+0000 7f3dd9ffb640 10 monclient(hunting): _init_auth creating new auth 2024-06-03T08:36:34.715 INFO:teuthology.orchestra.run.smithi088.stderr:2024-06-03T08:36:34.714+0000 7f3dd9ffb640 10 monclient(hunting): my global_id is 4384 2024-06-03T08:36:34.715 INFO:teuthology.orchestra.run.smithi088.stderr:2024-06-03T08:36:34.714+0000 7f3dd9ffb640 1 -- 172.21.15.88:0/1593134679 --> v1:172.21.15.192:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f3dc8004eb0 con 0x7f3de413eec0 2024-06-03T08:36:34.715 INFO:teuthology.orchestra.run.smithi088.stderr:2024-06-03T08:36:34.714+0000 7f0616ffd640 1 -- 172.21.15.88:0/228619988 <== mon.1 v1:172.21.15.192:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 764+0+0 (unknown 2902723095 0 0) 0x7f0600004eb0 con 0x7f06200ad980 2024-06-03T08:36:34.715 INFO:teuthology.orchestra.run.smithi088.stderr:2024-06-03T08:36:34.715+0000 7f0616ffd640 1 -- 172.21.15.88:0/228619988 --> v1:172.21.15.192:6789/0 -- auth(proto 2 165 bytes epoch 0) v1 -- 0x7f0620134550 con 0x7f06200ad980 2024-06-03T08:36:34.716 INFO:teuthology.orchestra.run.smithi088.stderr:2024-06-03T08:36:34.715+0000 7f0616ffd640 1 -- 172.21.15.88:0/228619988 <== mon.1 v1:172.21.15.192:6789/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 2234374065 0 0) 0x7f06080018e0 con 0x7f06200ad980 2024-06-03T08:36:34.716 INFO:teuthology.orchestra.run.smithi088.stderr:2024-06-03T08:36:34.715+0000 7f0616ffd640 10 monclient(hunting): handle_monmap mon_map magic: 0 v1 2024-06-03T08:36:34.716 INFO:teuthology.orchestra.run.smithi088.stderr:2024-06-03T08:36:34.715+0000 7f0616ffd640 10 monclient(hunting): got monmap 1 from mon.noname-b (according to old e1) 2024-06-03T08:36:34.716 INFO:teuthology.orchestra.run.smithi088.stderr:2024-06-03T08:36:34.715+0000 7f0616ffd640 10 monclient(hunting): dump: 2024-06-03T08:36:34.716 INFO:teuthology.orchestra.run.smithi088.stderr:epoch 1 2024-06-03T08:36:34.716 INFO:teuthology.orchestra.run.smithi088.stderr:fsid ec3c3fdd-3ac6-4d7d-9412-683130ed8b04 2024-06-03T08:36:34.716 INFO:teuthology.orchestra.run.smithi088.stderr:last_changed 2024-06-03T08:34:31.885125+0000 2024-06-03T08:36:34.716 INFO:teuthology.orchestra.run.smithi088.stderr:created 2024-06-03T08:34:31.885125+0000 2024-06-03T08:36:34.716 INFO:teuthology.orchestra.run.smithi088.stderr:min_mon_release 18 (reef) 2024-06-03T08:36:34.716 INFO:teuthology.orchestra.run.smithi088.stderr:election_strategy: 3 2024-06-03T08:36:34.716 INFO:teuthology.orchestra.run.smithi088.stderr:0: v1:172.21.15.88:6789/0 mon.a 2024-06-03T08:36:34.716 INFO:teuthology.orchestra.run.smithi088.stderr:1: v1:172.21.15.192:6789/0 mon.b 2024-06-03T08:36:34.716 INFO:teuthology.orchestra.run.smithi088.stderr:2: v1:172.21.15.88:6790/0 mon.c 2024-06-03T08:36:34.716 INFO:teuthology.orchestra.run.smithi088.stderr: 2024-06-03T08:36:34.717 INFO:teuthology.orchestra.run.smithi088.stderr:2024-06-03T08:36:34.715+0000 7f0616ffd640 10 monclient(hunting): _finish_auth 0 2024-06-03T08:36:34.721 INFO:teuthology.orchestra.run.smithi088.stderr:2024-06-03T08:36:34.720+0000 7f0616ffd640 1 -- 172.21.15.88:0/228619988 <== mon.1 v1:172.21.15.192:6789/0 4 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 580+0+0 (unknown 991262051 0 0) 0x7f0620134550 con 0x7f06200ad980 2024-06-03T08:36:34.722 INFO:teuthology.orchestra.run.smithi088.stderr:2024-06-03T08:36:34.720+0000 7f0616ffd640 1 -- 172.21.15.88:0/228619988 >> v1:172.21.15.88:6790/0 conn(0x7f06200ae540 legacy=0x7f062012e800 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-06-03T08:36:34.722 INFO:teuthology.orchestra.run.smithi088.stderr:2024-06-03T08:36:34.720+0000 7f0616ffd640 1 -- 172.21.15.88:0/228619988 >> v1:172.21.15.88:6789/0 conn(0x7f062006b9a0 legacy=0x7f06201349e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-06-03T08:36:34.722 INFO:teuthology.orchestra.run.smithi088.stderr:2024-06-03T08:36:34.720+0000 7f0616ffd640 10 monclient: _finish_hunting 0 2024-06-03T08:36:34.722 INFO:teuthology.orchestra.run.smithi088.stderr:2024-06-03T08:36:34.720+0000 7f0616ffd640 1 monclient: found mon.b 2024-06-03T08:36:34.722 INFO:teuthology.orchestra.run.smithi088.stderr:2024-06-03T08:36:34.720+0000 7f0616ffd640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-06-03T08:36:34.722 INFO:teuthology.orchestra.run.smithi088.stderr:2024-06-03T08:36:34.720+0000 7f0616ffd640 10 monclient: _send_mon_message to mon.b at v1:172.21.15.192:6789/0 2024-06-03T08:36:34.722 INFO:teuthology.orchestra.run.smithi088.stderr:2024-06-03T08:36:34.720+0000 7f0616ffd640 1 -- 172.21.15.88:0/228619988 --> v1:172.21.15.192:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f062013a7e0 con 0x7f06200ad980 2024-06-03T08:36:34.722 INFO:teuthology.orchestra.run.smithi088.stderr:2024-06-03T08:36:34.720+0000 7f0616ffd640 10 monclient: _finish_auth 0 2024-06-03T08:36:34.722 INFO:teuthology.orchestra.run.smithi088.stderr:2024-06-03T08:36:34.720+0000 7f0616ffd640 10 monclient: _check_auth_tickets 2024-06-03T08:36:34.722 INFO:teuthology.orchestra.run.smithi088.stderr:2024-06-03T08:36:34.720+0000 7f0616ffd640 20 monclient: _check_auth_rotating not needed by client.admin 2024-06-03T08:36:34.722 INFO:teuthology.orchestra.run.smithi088.stderr:2024-06-03T08:36:34.720+0000 7f0624b5f640 5 monclient: authenticate success, global_id 4381 2024-06-03T08:36:34.722 INFO:teuthology.orchestra.run.smithi088.stderr:2024-06-03T08:36:34.720+0000 7f0624b5f640 10 monclient: _renew_subs 2024-06-03T08:36:34.723 INFO:teuthology.orchestra.run.smithi088.stderr:2024-06-03T08:36:34.720+0000 7f0624b5f640 10 monclient: _send_mon_message to mon.b at v1:172.21.15.192:6789/0 2024-06-03T08:36:34.723 INFO:teuthology.orchestra.run.smithi088.stderr:2024-06-03T08:36:34.720+0000 7f0624b5f640 1 -- 172.21.15.88:0/228619988 --> v1:172.21.15.192:6789/0 -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f06201350f0 con 0x7f06200ad980 2024-06-03T08:36:34.723 INFO:teuthology.orchestra.run.smithi088.stderr:2024-06-03T08:36:34.721+0000 7f0624b5f640 10 client.4381.objecter _maybe_request_map subscribing (onetime) to next osd map 2024-06-03T08:36:34.723 INFO:teuthology.orchestra.run.smithi088.stderr:2024-06-03T08:36:34.721+0000 7f0624b5f640 10 monclient: _renew_subs 2024-06-03T08:36:34.723 INFO:teuthology.orchestra.run.smithi088.stderr:2024-06-03T08:36:34.721+0000 7f0624b5f640 10 monclient: _send_mon_message to mon.b at v1:172.21.15.192:6789/0 2024-06-03T08:36:34.723 INFO:teuthology.orchestra.run.smithi088.stderr:2024-06-03T08:36:34.721+0000 7f0624b5f640 1 -- 172.21.15.88:0/228619988 --> v1:172.21.15.192:6789/0 -- mon_subscribe({osdmap=0}) v3 -- 0x7f0620133250 con 0x7f06200ad980 2024-06-03T08:36:34.723 INFO:teuthology.orchestra.run.smithi088.stderr:2024-06-03T08:36:34.721+0000 7f0624b5f640 1 librados: init done 2024-06-03T08:36:34.723 INFO:teuthology.orchestra.run.smithi088.stderr:2024-06-03T08:36:34.721+0000 7f3dd9ffb640 1 -- 172.21.15.88:0/1593134679 <== mon.1 v1:172.21.15.192:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 764+0+0 (unknown 3750534716 0 0) 0x7f3dc8004eb0 con 0x7f3de413eec0 2024-06-03T08:36:34.723 INFO:teuthology.orchestra.run.smithi088.stderr:2024-06-03T08:36:34.721+0000 7f3dd9ffb640 1 -- 172.21.15.88:0/1593134679 --> v1:172.21.15.192:6789/0 -- auth(proto 2 165 bytes epoch 0) v1 -- 0x7f3de4141b20 con 0x7f3de413eec0 2024-06-03T08:36:34.724 INFO:teuthology.orchestra.run.smithi088.stderr:2024-06-03T08:36:34.721+0000 7f3dd9ffb640 1 -- 172.21.15.88:0/1593134679 <== mon.1 v1:172.21.15.192:6789/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 2234374065 0 0) 0x7f3dd4001750 con 0x7f3de413eec0 2024-06-03T08:36:34.724 INFO:teuthology.orchestra.run.smithi088.stderr:2024-06-03T08:36:34.721+0000 7f3dd9ffb640 10 monclient(hunting): handle_monmap mon_map magic: 0 v1 2024-06-03T08:36:34.724 INFO:teuthology.orchestra.run.smithi088.stderr:2024-06-03T08:36:34.721+0000 7f3dd9ffb640 10 monclient(hunting): got monmap 1 from mon.noname-b (according to old e1) 2024-06-03T08:36:34.724 INFO:teuthology.orchestra.run.smithi088.stderr:2024-06-03T08:36:34.721+0000 7f3dd9ffb640 10 monclient(hunting): dump: 2024-06-03T08:36:34.724 INFO:teuthology.orchestra.run.smithi088.stderr:epoch 1 2024-06-03T08:36:34.724 INFO:teuthology.orchestra.run.smithi088.stderr:fsid ec3c3fdd-3ac6-4d7d-9412-683130ed8b04 2024-06-03T08:36:34.724 INFO:teuthology.orchestra.run.smithi088.stderr:last_changed 2024-06-03T08:34:31.885125+0000 2024-06-03T08:36:34.724 INFO:teuthology.orchestra.run.smithi088.stderr:created 2024-06-03T08:34:31.885125+0000 2024-06-03T08:36:34.724 INFO:teuthology.orchestra.run.smithi088.stderr:min_mon_release 18 (reef) 2024-06-03T08:36:34.724 INFO:teuthology.orchestra.run.smithi088.stderr:election_strategy: 3 2024-06-03T08:36:34.724 INFO:teuthology.orchestra.run.smithi088.stderr:0: v1:172.21.15.88:6789/0 mon.a 2024-06-03T08:36:34.724 INFO:teuthology.orchestra.run.smithi088.stderr:1: v1:172.21.15.192:6789/0 mon.b 2024-06-03T08:36:34.724 INFO:teuthology.orchestra.run.smithi088.stderr:2: v1:172.21.15.88:6790/0 mon.c 2024-06-03T08:36:34.724 INFO:teuthology.orchestra.run.smithi088.stderr: 2024-06-03T08:36:34.725 INFO:teuthology.orchestra.run.smithi088.stderr:2024-06-03T08:36:34.721+0000 7f3dd9ffb640 10 monclient(hunting): _finish_auth 0 2024-06-03T08:36:34.725 INFO:teuthology.orchestra.run.smithi088.stderr:2024-06-03T08:36:34.721+0000 7f0624b5f640 10 start_mon_command cmd=[{"prefix": "get_command_descriptions"}] 2024-06-03T08:36:34.725 INFO:teuthology.orchestra.run.smithi088.stderr:2024-06-03T08:36:34.721+0000 7f0624b5f640 10 monclient: _send_command 1 [{"prefix": "get_command_descriptions"}] 2024-06-03T08:36:34.725 INFO:teuthology.orchestra.run.smithi088.stderr:2024-06-03T08:36:34.721+0000 7f0624b5f640 10 monclient: _send_mon_message to mon.b at v1:172.21.15.192:6789/0 2024-06-03T08:36:34.725 INFO:teuthology.orchestra.run.smithi088.stderr:2024-06-03T08:36:34.721+0000 7f0624b5f640 1 -- 172.21.15.88:0/228619988 --> v1:172.21.15.192:6789/0 -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f062013a620 con 0x7f06200ad980 2024-06-03T08:36:34.771 INFO:teuthology.orchestra.run.smithi088.stderr:2024-06-03T08:36:34.770+0000 7fbade7fc640 1 -- 172.21.15.88:0/1627903445 <== mon.0 v1:172.21.15.88:6789/0 9 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+182498 (unknown 1092875540 0 3016760700) 0x7fbad804f840 con 0x7fbae80b4250 2024-06-03T08:36:34.771 INFO:teuthology.orchestra.run.smithi088.stderr:2024-06-03T08:36:34.770+0000 7fbade7fc640 10 monclient: handle_mon_command_ack 1 [{"prefix": "get_command_descriptions"}] 2024-06-03T08:36:34.771 INFO:teuthology.orchestra.run.smithi088.stderr:2024-06-03T08:36:34.770+0000 7fbade7fc640 10 monclient: _finish_command 1 = system:0 2024-06-03T08:36:34.805 INFO:teuthology.orchestra.run.smithi088.stderr:2024-06-03T08:36:34.805+0000 7f3dd9ffb640 10 client.?.objecter ms_handle_connect 0x7f3de40a8960 2024-06-03T08:36:34.805 INFO:teuthology.orchestra.run.smithi088.stderr:2024-06-03T08:36:34.805+0000 7f3dd9ffb640 10 client.?.objecter resend_mon_ops 2024-06-03T08:36:34.809 INFO:teuthology.orchestra.run.smithi088.stderr:2024-06-03T08:36:34.808+0000 7f3dd9ffb640 1 -- 172.21.15.88:0/1593134679 <== mon.0 v1:172.21.15.88:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 1719161251 0 0) 0x7f3de40aacb0 con 0x7f3de40a8960 2024-06-03T08:36:34.809 INFO:teuthology.orchestra.run.smithi088.stderr:2024-06-03T08:36:34.808+0000 7f3dd9ffb640 10 monclient(hunting): _negotiate 2024-06-03T08:36:34.809 INFO:teuthology.orchestra.run.smithi088.stderr:2024-06-03T08:36:34.808+0000 7f3dd9ffb640 10 monclient(hunting): _init_auth method 2 2024-06-03T08:36:34.809 INFO:teuthology.orchestra.run.smithi088.stderr:2024-06-03T08:36:34.808+0000 7f3dd9ffb640 10 monclient(hunting): _init_auth creating new auth 2024-06-03T08:36:34.809 INFO:teuthology.orchestra.run.smithi088.stderr:2024-06-03T08:36:34.808+0000 7f3dd9ffb640 10 monclient(hunting): my global_id is 4377 2024-06-03T08:36:34.809 INFO:teuthology.orchestra.run.smithi088.stderr:2024-06-03T08:36:34.808+0000 7f3dd9ffb640 1 -- 172.21.15.88:0/1593134679 --> v1:172.21.15.88:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f3dc8004eb0 con 0x7f3de40a8960 2024-06-03T08:36:34.813 INFO:teuthology.orchestra.run.smithi088.stderr:2024-06-03T08:36:34.813+0000 7f3dd9ffb640 1 -- 172.21.15.88:0/1593134679 <== mon.0 v1:172.21.15.88:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 764+0+0 (unknown 2684247521 0 0) 0x7f3dc8004eb0 con 0x7f3de40a8960 2024-06-03T08:36:34.814 INFO:teuthology.orchestra.run.smithi088.stderr:2024-06-03T08:36:34.813+0000 7f3dd9ffb640 1 -- 172.21.15.88:0/1593134679 --> v1:172.21.15.88:6789/0 -- auth(proto 2 165 bytes epoch 0) v1 -- 0x7f3de40aacb0 con 0x7f3de40a8960 2024-06-03T08:36:34.814 INFO:teuthology.orchestra.run.smithi088.stderr:2024-06-03T08:36:34.813+0000 7f3dd9ffb640 1 -- 172.21.15.88:0/1593134679 <== mon.0 v1:172.21.15.88:6789/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 2234374065 0 0) 0x7f3de00014d0 con 0x7f3de40a8960 2024-06-03T08:36:34.814 INFO:teuthology.orchestra.run.smithi088.stderr:2024-06-03T08:36:34.813+0000 7f3dd9ffb640 10 monclient(hunting): handle_monmap mon_map magic: 0 v1 2024-06-03T08:36:34.814 INFO:teuthology.orchestra.run.smithi088.stderr:2024-06-03T08:36:34.813+0000 7f3dd9ffb640 10 monclient(hunting): got monmap 1 from mon.a (according to old e1) 2024-06-03T08:36:34.814 INFO:teuthology.orchestra.run.smithi088.stderr:2024-06-03T08:36:34.813+0000 7f3dd9ffb640 10 monclient(hunting): dump: 2024-06-03T08:36:34.814 INFO:teuthology.orchestra.run.smithi088.stderr:epoch 1 2024-06-03T08:36:34.814 INFO:teuthology.orchestra.run.smithi088.stderr:fsid ec3c3fdd-3ac6-4d7d-9412-683130ed8b04 2024-06-03T08:36:34.814 INFO:teuthology.orchestra.run.smithi088.stderr:last_changed 2024-06-03T08:34:31.885125+0000 2024-06-03T08:36:34.814 INFO:teuthology.orchestra.run.smithi088.stderr:created 2024-06-03T08:34:31.885125+0000 2024-06-03T08:36:34.814 INFO:teuthology.orchestra.run.smithi088.stderr:min_mon_release 18 (reef) 2024-06-03T08:36:34.814 INFO:teuthology.orchestra.run.smithi088.stderr:election_strategy: 3 2024-06-03T08:36:34.815 INFO:teuthology.orchestra.run.smithi088.stderr:0: v1:172.21.15.88:6789/0 mon.a 2024-06-03T08:36:34.815 INFO:teuthology.orchestra.run.smithi088.stderr:1: v1:172.21.15.192:6789/0 mon.b 2024-06-03T08:36:34.815 INFO:teuthology.orchestra.run.smithi088.stderr:2: v1:172.21.15.88:6790/0 mon.c 2024-06-03T08:36:34.815 INFO:teuthology.orchestra.run.smithi088.stderr: 2024-06-03T08:36:34.818 INFO:teuthology.orchestra.run.smithi088.stderr:2024-06-03T08:36:34.817+0000 7f3dd9ffb640 1 -- 172.21.15.88:0/1593134679 <== mon.0 v1:172.21.15.88:6789/0 4 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 580+0+0 (unknown 2322561180 0 0) 0x7f3de40aacb0 con 0x7f3de40a8960 2024-06-03T08:36:34.818 INFO:teuthology.orchestra.run.smithi088.stderr:2024-06-03T08:36:34.817+0000 7f3dd9ffb640 1 -- 172.21.15.88:0/1593134679 >> v1:172.21.15.88:6790/0 conn(0x7f3de41450f0 legacy=0x7f3de41e0f80 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-06-03T08:36:34.818 INFO:teuthology.orchestra.run.smithi088.stderr:2024-06-03T08:36:34.817+0000 7f3dd9ffb640 1 -- 172.21.15.88:0/1593134679 >> v1:172.21.15.192:6789/0 conn(0x7f3de413eec0 legacy=0x7f3de41d6e60 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-06-03T08:36:34.818 INFO:teuthology.orchestra.run.smithi088.stderr:2024-06-03T08:36:34.817+0000 7f3dd9ffb640 10 monclient: _finish_hunting 0 2024-06-03T08:36:34.818 INFO:teuthology.orchestra.run.smithi088.stderr:2024-06-03T08:36:34.817+0000 7f3dd9ffb640 1 monclient: found mon.a 2024-06-03T08:36:34.818 INFO:teuthology.orchestra.run.smithi088.stderr:2024-06-03T08:36:34.817+0000 7f3dd9ffb640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-06-03T08:36:34.818 INFO:teuthology.orchestra.run.smithi088.stderr:2024-06-03T08:36:34.817+0000 7f3dd9ffb640 10 monclient: _send_mon_message to mon.a at v1:172.21.15.88:6789/0 2024-06-03T08:36:34.818 INFO:teuthology.orchestra.run.smithi088.stderr:2024-06-03T08:36:34.817+0000 7f3dd9ffb640 1 -- 172.21.15.88:0/1593134679 --> v1:172.21.15.88:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f3de40ab380 con 0x7f3de40a8960 2024-06-03T08:36:34.818 INFO:teuthology.orchestra.run.smithi088.stderr:2024-06-03T08:36:34.817+0000 7f3dd9ffb640 10 monclient: _finish_auth 0 2024-06-03T08:36:34.818 INFO:teuthology.orchestra.run.smithi088.stderr:2024-06-03T08:36:34.817+0000 7f3dd9ffb640 10 monclient: _check_auth_tickets 2024-06-03T08:36:34.818 INFO:teuthology.orchestra.run.smithi088.stderr:2024-06-03T08:36:34.817+0000 7f3dd9ffb640 20 monclient: _check_auth_rotating not needed by client.admin 2024-06-03T08:36:34.819 INFO:teuthology.orchestra.run.smithi088.stderr:2024-06-03T08:36:34.817+0000 7f3deb5b9640 5 monclient: authenticate success, global_id 4377 2024-06-03T08:36:34.819 INFO:teuthology.orchestra.run.smithi088.stderr:2024-06-03T08:36:34.817+0000 7f3deb5b9640 10 monclient: _renew_subs 2024-06-03T08:36:34.819 INFO:teuthology.orchestra.run.smithi088.stderr:2024-06-03T08:36:34.817+0000 7f3deb5b9640 10 monclient: _send_mon_message to mon.a at v1:172.21.15.88:6789/0 2024-06-03T08:36:34.819 INFO:teuthology.orchestra.run.smithi088.stderr:2024-06-03T08:36:34.817+0000 7f3deb5b9640 1 -- 172.21.15.88:0/1593134679 --> v1:172.21.15.88:6789/0 -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f3de4142390 con 0x7f3de40a8960 2024-06-03T08:36:34.819 INFO:teuthology.orchestra.run.smithi088.stderr:2024-06-03T08:36:34.817+0000 7f3deb5b9640 10 client.4377.objecter _maybe_request_map subscribing (onetime) to next osd map 2024-06-03T08:36:34.819 INFO:teuthology.orchestra.run.smithi088.stderr:2024-06-03T08:36:34.817+0000 7f3deb5b9640 10 monclient: _renew_subs 2024-06-03T08:36:34.819 INFO:teuthology.orchestra.run.smithi088.stderr:2024-06-03T08:36:34.817+0000 7f3deb5b9640 10 monclient: _send_mon_message to mon.a at v1:172.21.15.88:6789/0 2024-06-03T08:36:34.819 INFO:teuthology.orchestra.run.smithi088.stderr:2024-06-03T08:36:34.817+0000 7f3deb5b9640 1 -- 172.21.15.88:0/1593134679 --> v1:172.21.15.88:6789/0 -- mon_subscribe({osdmap=0}) v3 -- 0x7f3de413f560 con 0x7f3de40a8960 2024-06-03T08:36:34.819 INFO:teuthology.orchestra.run.smithi088.stderr:2024-06-03T08:36:34.817+0000 7f3deb5b9640 1 librados: init done 2024-06-03T08:36:34.820 INFO:teuthology.orchestra.run.smithi088.stderr:2024-06-03T08:36:34.818+0000 7f3deb5b9640 10 start_mon_command cmd=[{"prefix": "get_command_descriptions"}] 2024-06-03T08:36:34.820 INFO:teuthology.orchestra.run.smithi088.stderr:2024-06-03T08:36:34.818+0000 7f3deb5b9640 10 monclient: _send_command 1 [{"prefix": "get_command_descriptions"}] 2024-06-03T08:36:34.820 INFO:teuthology.orchestra.run.smithi088.stderr:2024-06-03T08:36:34.818+0000 7f3deb5b9640 10 monclient: _send_mon_message to mon.a at v1:172.21.15.88:6789/0 2024-06-03T08:36:34.820 INFO:teuthology.orchestra.run.smithi088.stderr:2024-06-03T08:36:34.818+0000 7f3deb5b9640 1 -- 172.21.15.88:0/1593134679 --> v1:172.21.15.88:6789/0 -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f3de40a9e60 con 0x7f3de40a8960 2024-06-03T08:36:34.835 INFO:teuthology.orchestra.run.smithi088.stderr:2024-06-03T08:36:34.835+0000 7f3dd9ffb640 1 -- 172.21.15.88:0/1593134679 <== mon.0 v1:172.21.15.88:6789/0 5 ==== config(0 keys) v1 ==== 4+0+0 (unknown 0 0 0) 0x7f3de0001ac0 con 0x7f3de40a8960 2024-06-03T08:36:34.835 INFO:teuthology.orchestra.run.smithi088.stderr:2024-06-03T08:36:34.835+0000 7f3dd9ffb640 10 monclient: handle_config config(0 keys) v1 2024-06-03T08:36:34.836 INFO:teuthology.orchestra.run.smithi088.stderr:2024-06-03T08:36:34.835+0000 7f3dd9ffb640 1 -- 172.21.15.88:0/1593134679 <== mon.0 v1:172.21.15.88:6789/0 6 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 2234374065 0 0) 0x7f3de0004a50 con 0x7f3de40a8960 2024-06-03T08:36:34.837 INFO:teuthology.orchestra.run.smithi088.stderr:2024-06-03T08:36:34.835+0000 7f3dd9ffb640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-06-03T08:36:34.837 INFO:teuthology.orchestra.run.smithi088.stderr:2024-06-03T08:36:34.835+0000 7f3dd9ffb640 10 monclient: got monmap 1 from mon.a (according to old e1) 2024-06-03T08:36:34.837 INFO:teuthology.orchestra.run.smithi088.stderr:2024-06-03T08:36:34.835+0000 7f3dd9ffb640 10 monclient: dump: 2024-06-03T08:36:34.837 INFO:teuthology.orchestra.run.smithi088.stderr:epoch 1 2024-06-03T08:36:34.837 INFO:teuthology.orchestra.run.smithi088.stderr:fsid ec3c3fdd-3ac6-4d7d-9412-683130ed8b04 2024-06-03T08:36:34.837 INFO:teuthology.orchestra.run.smithi088.stderr:last_changed 2024-06-03T08:34:31.885125+0000 2024-06-03T08:36:34.837 INFO:teuthology.orchestra.run.smithi088.stderr:created 2024-06-03T08:34:31.885125+0000 2024-06-03T08:36:34.837 INFO:teuthology.orchestra.run.smithi088.stderr:min_mon_release 18 (reef) 2024-06-03T08:36:34.837 INFO:teuthology.orchestra.run.smithi088.stderr:election_strategy: 3 2024-06-03T08:36:34.837 INFO:teuthology.orchestra.run.smithi088.stderr:0: v1:172.21.15.88:6789/0 mon.a 2024-06-03T08:36:34.837 INFO:teuthology.orchestra.run.smithi088.stderr:1: v1:172.21.15.192:6789/0 mon.b 2024-06-03T08:36:34.838 INFO:teuthology.orchestra.run.smithi088.stderr:2: v1:172.21.15.88:6790/0 mon.c 2024-06-03T08:36:34.838 INFO:teuthology.orchestra.run.smithi088.stderr: 2024-06-03T08:36:34.838 INFO:teuthology.orchestra.run.smithi088.stderr:2024-06-03T08:36:34.835+0000 7f3ddb7fe640 4 set_mon_vals no callback set 2024-06-03T08:36:34.838 INFO:teuthology.orchestra.run.smithi088.stderr:2024-06-03T08:36:34.837+0000 7f3dd9ffb640 1 -- 172.21.15.88:0/1593134679 <== mon.0 v1:172.21.15.88:6789/0 7 ==== mgrmap(e 5) v1 ==== 89110+0+0 (unknown 873533973 0 0) 0x7f3de001a7d0 con 0x7f3de40a8960 2024-06-03T08:36:34.838 INFO:teuthology.orchestra.run.smithi088.stderr:2024-06-03T08:36:34.838+0000 7f3de8b54640 10 monclient: get_auth_request con 0x7f3dc806d250 auth_method 0 2024-06-03T08:36:34.838 INFO:teuthology.orchestra.run.smithi088.stderr:2024-06-03T08:36:34.838+0000 7f3dd9ffb640 10 client.4377.objecter ms_handle_connect 0x7f3dc806d250 2024-06-03T08:36:34.840 INFO:teuthology.orchestra.run.smithi088.stderr:2024-06-03T08:36:34.839+0000 7f3dd9ffb640 1 -- 172.21.15.88:0/1593134679 <== mon.0 v1:172.21.15.88:6789/0 8 ==== osd_map(26..26 src has 1..26) v4 ==== 4090+0+0 (unknown 3637450554 0 0) 0x7f3de0053f70 con 0x7f3de40a8960 2024-06-03T08:36:34.840 INFO:teuthology.orchestra.run.smithi088.stderr:2024-06-03T08:36:34.839+0000 7f3dd9ffb640 10 client.4377.objecter ms_dispatch 0x7f3de4000b90 osd_map(26..26 src has 1..26) v4 2024-06-03T08:36:34.840 INFO:teuthology.orchestra.run.smithi088.stderr:2024-06-03T08:36:34.839+0000 7f3dd9ffb640 3 client.4377.objecter handle_osd_map got epochs [26,26] > 0 2024-06-03T08:36:34.840 INFO:teuthology.orchestra.run.smithi088.stderr:2024-06-03T08:36:34.839+0000 7f3dd9ffb640 3 client.4377.objecter handle_osd_map decoding full epoch 26 2024-06-03T08:36:34.840 INFO:teuthology.orchestra.run.smithi088.stderr:2024-06-03T08:36:34.839+0000 7f3dd9ffb640 20 client.4377.objecter dump_active .. 0 homeless 2024-06-03T08:36:34.967 INFO:teuthology.orchestra.run.smithi088.stderr:2024-06-03T08:36:34.966+0000 7feefffff640 1 -- 172.21.15.88:0/3086147656 <== mon.2 v1:172.21.15.88:6790/0 9 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+182498 (unknown 1092875540 0 3016760700) 0x7fef10050290 con 0x7fef181450f0 2024-06-03T08:36:34.967 INFO:teuthology.orchestra.run.smithi088.stderr:2024-06-03T08:36:34.966+0000 7feefffff640 10 monclient: handle_mon_command_ack 1 [{"prefix": "get_command_descriptions"}] 2024-06-03T08:36:34.967 INFO:teuthology.orchestra.run.smithi088.stderr:2024-06-03T08:36:34.966+0000 7feefffff640 10 monclient: _finish_command 1 = system:0 2024-06-03T08:36:34.968 INFO:teuthology.orchestra.run.smithi088.stderr:2024-06-03T08:36:34.968+0000 7fbaefed1640 10 start_mon_command cmd=[{"prefix": "osd last-stat-seq", "id": 7}] 2024-06-03T08:36:34.968 INFO:teuthology.orchestra.run.smithi088.stderr:2024-06-03T08:36:34.968+0000 7fbaefed1640 10 monclient: _send_command 2 [{"prefix": "osd last-stat-seq", "id": 7}] 2024-06-03T08:36:34.968 INFO:teuthology.orchestra.run.smithi088.stderr:2024-06-03T08:36:34.968+0000 7fbaefed1640 10 monclient: _send_mon_message to mon.a at v1:172.21.15.88:6789/0 2024-06-03T08:36:34.968 INFO:teuthology.orchestra.run.smithi088.stderr:2024-06-03T08:36:34.968+0000 7fbaefed1640 1 -- 172.21.15.88:0/1627903445 --> v1:172.21.15.88:6789/0 -- mon_command({"prefix": "osd last-stat-seq", "id": 7} v 0) v1 -- 0x7fbae81d6a60 con 0x7fbae80b4250 2024-06-03T08:36:34.971 INFO:teuthology.orchestra.run.smithi088.stderr:2024-06-03T08:36:34.970+0000 7f0616ffd640 1 -- 172.21.15.88:0/228619988 <== mon.1 v1:172.21.15.192:6789/0 5 ==== config(0 keys) v1 ==== 4+0+0 (unknown 0 0 0) 0x7f0608001c80 con 0x7f06200ad980 2024-06-03T08:36:34.971 INFO:teuthology.orchestra.run.smithi088.stderr:2024-06-03T08:36:34.970+0000 7f0616ffd640 10 monclient: handle_config config(0 keys) v1 2024-06-03T08:36:34.971 INFO:teuthology.orchestra.run.smithi088.stderr:2024-06-03T08:36:34.970+0000 7f0616ffd640 1 -- 172.21.15.88:0/228619988 <== mon.1 v1:172.21.15.192:6789/0 6 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 2234374065 0 0) 0x7f0608003a40 con 0x7f06200ad980 2024-06-03T08:36:34.971 INFO:teuthology.orchestra.run.smithi088.stderr:2024-06-03T08:36:34.970+0000 7f0616ffd640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-06-03T08:36:34.971 INFO:teuthology.orchestra.run.smithi088.stderr:2024-06-03T08:36:34.970+0000 7f0616ffd640 10 monclient: got monmap 1 from mon.b (according to old e1) 2024-06-03T08:36:34.971 INFO:teuthology.orchestra.run.smithi088.stderr:2024-06-03T08:36:34.970+0000 7f0616ffd640 10 monclient: dump: 2024-06-03T08:36:34.971 INFO:teuthology.orchestra.run.smithi088.stderr:epoch 1 2024-06-03T08:36:34.971 INFO:teuthology.orchestra.run.smithi088.stderr:fsid ec3c3fdd-3ac6-4d7d-9412-683130ed8b04 2024-06-03T08:36:34.971 INFO:teuthology.orchestra.run.smithi088.stderr:last_changed 2024-06-03T08:34:31.885125+0000 2024-06-03T08:36:34.971 INFO:teuthology.orchestra.run.smithi088.stderr:created 2024-06-03T08:34:31.885125+0000 2024-06-03T08:36:34.971 INFO:teuthology.orchestra.run.smithi088.stderr:min_mon_release 18 (reef) 2024-06-03T08:36:34.971 INFO:teuthology.orchestra.run.smithi088.stderr:election_strategy: 3 2024-06-03T08:36:34.971 INFO:teuthology.orchestra.run.smithi088.stderr:0: v1:172.21.15.88:6789/0 mon.a 2024-06-03T08:36:34.971 INFO:teuthology.orchestra.run.smithi088.stderr:1: v1:172.21.15.192:6789/0 mon.b 2024-06-03T08:36:34.972 INFO:teuthology.orchestra.run.smithi088.stderr:2: v1:172.21.15.88:6790/0 mon.c 2024-06-03T08:36:34.972 INFO:teuthology.orchestra.run.smithi088.stderr: 2024-06-03T08:36:34.972 INFO:teuthology.orchestra.run.smithi088.stderr:2024-06-03T08:36:34.970+0000 7f061d59a640 4 set_mon_vals no callback set 2024-06-03T08:36:34.972 INFO:teuthology.orchestra.run.smithi088.stderr:2024-06-03T08:36:34.972+0000 7f0616ffd640 1 -- 172.21.15.88:0/228619988 <== mon.1 v1:172.21.15.192:6789/0 7 ==== mgrmap(e 5) v1 ==== 89110+0+0 (unknown 873533973 0 0) 0x7f0608003ba0 con 0x7f06200ad980 2024-06-03T08:36:34.973 INFO:teuthology.orchestra.run.smithi088.stderr:2024-06-03T08:36:34.972+0000 7f061e59c640 10 monclient: get_auth_request con 0x7f060006cff0 auth_method 0 2024-06-03T08:36:34.973 INFO:teuthology.orchestra.run.smithi088.stderr:2024-06-03T08:36:34.972+0000 7f0616ffd640 10 client.4381.objecter ms_handle_connect 0x7f060006cff0 2024-06-03T08:36:34.974 INFO:teuthology.orchestra.run.smithi088.stderr:2024-06-03T08:36:34.973+0000 7f0616ffd640 1 -- 172.21.15.88:0/228619988 <== mon.1 v1:172.21.15.192:6789/0 8 ==== osd_map(26..26 src has 1..26) v4 ==== 4090+0+0 (unknown 3637450554 0 0) 0x7f0608054120 con 0x7f06200ad980 2024-06-03T08:36:34.974 INFO:teuthology.orchestra.run.smithi088.stderr:2024-06-03T08:36:34.973+0000 7f0616ffd640 10 client.4381.objecter ms_dispatch 0x7f0620000b90 osd_map(26..26 src has 1..26) v4 2024-06-03T08:36:34.974 INFO:teuthology.orchestra.run.smithi088.stderr:2024-06-03T08:36:34.973+0000 7f0616ffd640 3 client.4381.objecter handle_osd_map got epochs [26,26] > 0 2024-06-03T08:36:34.974 INFO:teuthology.orchestra.run.smithi088.stderr:2024-06-03T08:36:34.973+0000 7f0616ffd640 3 client.4381.objecter handle_osd_map decoding full epoch 26 2024-06-03T08:36:34.974 INFO:teuthology.orchestra.run.smithi088.stderr:2024-06-03T08:36:34.973+0000 7f0616ffd640 20 client.4381.objecter dump_active .. 0 homeless 2024-06-03T08:36:34.977 INFO:teuthology.orchestra.run.smithi088.stderr:2024-06-03T08:36:34.977+0000 7f0616ffd640 1 -- 172.21.15.88:0/228619988 <== mon.1 v1:172.21.15.192:6789/0 9 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+182498 (unknown 1092875540 0 3016760700) 0x7f060804fb00 con 0x7f06200ad980 2024-06-03T08:36:34.977 INFO:teuthology.orchestra.run.smithi088.stderr:2024-06-03T08:36:34.977+0000 7f0616ffd640 10 monclient: handle_mon_command_ack 1 [{"prefix": "get_command_descriptions"}] 2024-06-03T08:36:34.978 INFO:teuthology.orchestra.run.smithi088.stderr:2024-06-03T08:36:34.977+0000 7f0616ffd640 10 monclient: _finish_command 1 = system:0 2024-06-03T08:36:34.999 INFO:teuthology.orchestra.run.smithi088.stderr:2024-06-03T08:36:34.999+0000 7f293a658640 10 start_mon_command cmd=[{"prefix": "osd last-stat-seq", "id": 1}] 2024-06-03T08:36:35.000 INFO:teuthology.orchestra.run.smithi088.stderr:2024-06-03T08:36:34.999+0000 7f293a658640 10 monclient: _send_command 2 [{"prefix": "osd last-stat-seq", "id": 1}] 2024-06-03T08:36:35.000 INFO:teuthology.orchestra.run.smithi088.stderr:2024-06-03T08:36:34.999+0000 7f293a658640 10 monclient: _send_mon_message to mon.b at v1:172.21.15.192:6789/0 2024-06-03T08:36:35.000 INFO:teuthology.orchestra.run.smithi088.stderr:2024-06-03T08:36:34.999+0000 7f293a658640 1 -- 172.21.15.88:0/248786529 --> v1:172.21.15.192:6789/0 -- mon_command({"prefix": "osd last-stat-seq", "id": 1} v 0) v1 -- 0x7f293404ea60 con 0x7f29340a89f0 2024-06-03T08:36:35.002 INFO:teuthology.orchestra.run.smithi088.stderr:2024-06-03T08:36:35.002+0000 7f29317fa640 1 -- 172.21.15.88:0/248786529 <== mon.1 v1:172.21.15.192:6789/0 10 ==== mon_command_ack([{"prefix": "osd last-stat-seq", "id": 1}]=0 v0) v1 ==== 74+0+12 (unknown 832126871 0 4171588661) 0x7f2924051c80 con 0x7f29340a89f0 2024-06-03T08:36:35.002 INFO:teuthology.orchestra.run.smithi088.stderr:2024-06-03T08:36:35.002+0000 7f29317fa640 10 monclient: handle_mon_command_ack 2 [{"prefix": "osd last-stat-seq", "id": 1}] 2024-06-03T08:36:35.002 INFO:teuthology.orchestra.run.smithi088.stderr:2024-06-03T08:36:35.002+0000 7f29317fa640 10 monclient: _finish_command 2 = system:0 2024-06-03T08:36:35.003 INFO:teuthology.orchestra.run.smithi088.stdout:60129542152 2024-06-03T08:36:35.005 INFO:teuthology.orchestra.run.smithi088.stderr:2024-06-03T08:36:35.004+0000 7f293a658640 10 librados: watch_flush enter 2024-06-03T08:36:35.005 INFO:teuthology.orchestra.run.smithi088.stderr:2024-06-03T08:36:35.004+0000 7f293a658640 10 librados: watch_flush exit 2024-06-03T08:36:35.005 INFO:teuthology.orchestra.run.smithi088.stderr:2024-06-03T08:36:35.004+0000 7f293a658640 20 client.4372.objecter shutdown clearing up homeless session... 2024-06-03T08:36:35.005 INFO:teuthology.orchestra.run.smithi088.stderr:2024-06-03T08:36:35.004+0000 7f293a658640 10 client.4372.objecter successfully canceled tick 2024-06-03T08:36:35.005 INFO:teuthology.orchestra.run.smithi088.stderr:2024-06-03T08:36:35.004+0000 7f293a658640 1 -- 172.21.15.88:0/248786529 >> v1:172.21.15.192:6800/34229 conn(0x7f291406d220 legacy=0x7f291406f6e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-06-03T08:36:35.005 INFO:teuthology.orchestra.run.smithi088.stderr:2024-06-03T08:36:35.004+0000 7f293a658640 10 monclient: shutdown 2024-06-03T08:36:35.005 INFO:teuthology.orchestra.run.smithi088.stderr:2024-06-03T08:36:35.004+0000 7f293a658640 20 monclient: shutdown discarding 0 pending message(s) 2024-06-03T08:36:35.006 INFO:teuthology.orchestra.run.smithi088.stderr:2024-06-03T08:36:35.004+0000 7f293a658640 1 -- 172.21.15.88:0/248786529 >> v1:172.21.15.192:6789/0 conn(0x7f29340a89f0 legacy=0x7f2934142160 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-06-03T08:36:35.006 INFO:teuthology.orchestra.run.smithi088.stderr:2024-06-03T08:36:35.004+0000 7f293a658640 1 -- 172.21.15.88:0/248786529 shutdown_connections 2024-06-03T08:36:35.006 INFO:teuthology.orchestra.run.smithi088.stderr:2024-06-03T08:36:35.004+0000 7f293a658640 1 -- 172.21.15.88:0/248786529 >> 172.21.15.88:0/248786529 conn(0x7f29340717f0 msgr2=0x7f29340b4d60 unknown :-1 s=STATE_NONE l=0).mark_down 2024-06-03T08:36:35.006 INFO:teuthology.orchestra.run.smithi088.stderr:2024-06-03T08:36:35.004+0000 7f293a658640 1 -- 172.21.15.88:0/248786529 shutdown_connections 2024-06-03T08:36:35.006 INFO:teuthology.orchestra.run.smithi088.stderr:2024-06-03T08:36:35.004+0000 7f293a658640 1 -- 172.21.15.88:0/248786529 wait complete. 2024-06-03T08:36:35.006 INFO:teuthology.orchestra.run.smithi088.stderr:2024-06-03T08:36:35.004+0000 7f293a658640 1 librados: shutdown 2024-06-03T08:36:35.016 INFO:tasks.ceph.ceph_manager.ceph:need seq 60129542152 got 60129542152 for osd.1 2024-06-03T08:36:35.016 DEBUG:teuthology.parallel:result is None 2024-06-03T08:36:35.149 INFO:teuthology.orchestra.run.smithi088.stderr:2024-06-03T08:36:35.149+0000 7fef1dfb2640 10 start_mon_command cmd=[{"prefix": "osd last-stat-seq", "id": 3}] 2024-06-03T08:36:35.149 INFO:teuthology.orchestra.run.smithi088.stderr:2024-06-03T08:36:35.149+0000 7fef1dfb2640 10 monclient: _send_command 2 [{"prefix": "osd last-stat-seq", "id": 3}] 2024-06-03T08:36:35.150 INFO:teuthology.orchestra.run.smithi088.stderr:2024-06-03T08:36:35.149+0000 7fef1dfb2640 10 monclient: _send_mon_message to mon.c at v1:172.21.15.88:6790/0 2024-06-03T08:36:35.150 INFO:teuthology.orchestra.run.smithi088.stderr:2024-06-03T08:36:35.149+0000 7fef1dfb2640 1 -- 172.21.15.88:0/3086147656 --> v1:172.21.15.88:6790/0 -- mon_command({"prefix": "osd last-stat-seq", "id": 3} v 0) v1 -- 0x7fef1804ea60 con 0x7fef181450f0 2024-06-03T08:36:35.152 INFO:teuthology.orchestra.run.smithi088.stderr:2024-06-03T08:36:35.152+0000 7feefffff640 1 -- 172.21.15.88:0/3086147656 <== mon.2 v1:172.21.15.88:6790/0 10 ==== mon_command_ack([{"prefix": "osd last-stat-seq", "id": 3}]=0 v0) v1 ==== 74+0+12 (unknown 383520633 0 4171588661) 0x7fef100521c0 con 0x7fef181450f0 2024-06-03T08:36:35.153 INFO:teuthology.orchestra.run.smithi088.stderr:2024-06-03T08:36:35.152+0000 7feefffff640 10 monclient: handle_mon_command_ack 2 [{"prefix": "osd last-stat-seq", "id": 3}] 2024-06-03T08:36:35.153 INFO:teuthology.orchestra.run.smithi088.stderr:2024-06-03T08:36:35.152+0000 7feefffff640 10 monclient: _finish_command 2 = system:0 2024-06-03T08:36:35.153 INFO:teuthology.orchestra.run.smithi088.stdout:60129542152 2024-06-03T08:36:35.155 INFO:teuthology.orchestra.run.smithi088.stderr:2024-06-03T08:36:35.154+0000 7fef1dfb2640 10 librados: watch_flush enter 2024-06-03T08:36:35.155 INFO:teuthology.orchestra.run.smithi088.stderr:2024-06-03T08:36:35.154+0000 7fef1dfb2640 10 librados: watch_flush exit 2024-06-03T08:36:35.155 INFO:teuthology.orchestra.run.smithi088.stderr:2024-06-03T08:36:35.154+0000 7fef1dfb2640 20 client.4388.objecter shutdown clearing up homeless session... 2024-06-03T08:36:35.155 INFO:teuthology.orchestra.run.smithi088.stderr:2024-06-03T08:36:35.154+0000 7fef1dfb2640 10 client.4388.objecter successfully canceled tick 2024-06-03T08:36:35.155 INFO:teuthology.orchestra.run.smithi088.stderr:2024-06-03T08:36:35.154+0000 7fef1dfb2640 1 -- 172.21.15.88:0/3086147656 >> v1:172.21.15.192:6800/34229 conn(0x7fef0406d080 legacy=0x7fef0406f540 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-06-03T08:36:35.156 INFO:teuthology.orchestra.run.smithi088.stderr:2024-06-03T08:36:35.154+0000 7fef1dfb2640 10 monclient: shutdown 2024-06-03T08:36:35.156 INFO:teuthology.orchestra.run.smithi088.stderr:2024-06-03T08:36:35.154+0000 7fef1dfb2640 20 monclient: shutdown discarding 0 pending message(s) 2024-06-03T08:36:35.156 INFO:teuthology.orchestra.run.smithi088.stderr:2024-06-03T08:36:35.154+0000 7fef1dfb2640 1 -- 172.21.15.88:0/3086147656 >> v1:172.21.15.88:6790/0 conn(0x7fef181450f0 legacy=0x7fef181e15e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-06-03T08:36:35.156 INFO:teuthology.orchestra.run.smithi088.stderr:2024-06-03T08:36:35.154+0000 7fef1dfb2640 1 -- 172.21.15.88:0/3086147656 shutdown_connections 2024-06-03T08:36:35.156 INFO:teuthology.orchestra.run.smithi088.stderr:2024-06-03T08:36:35.154+0000 7fef1dfb2640 1 -- 172.21.15.88:0/3086147656 >> 172.21.15.88:0/3086147656 conn(0x7fef180b70d0 msgr2=0x7fef1813e580 unknown :-1 s=STATE_NONE l=0).mark_down 2024-06-03T08:36:35.156 INFO:teuthology.orchestra.run.smithi088.stderr:2024-06-03T08:36:35.154+0000 7fef1dfb2640 1 -- 172.21.15.88:0/3086147656 shutdown_connections 2024-06-03T08:36:35.156 INFO:teuthology.orchestra.run.smithi088.stderr:2024-06-03T08:36:35.154+0000 7fef1dfb2640 1 -- 172.21.15.88:0/3086147656 wait complete. 2024-06-03T08:36:35.156 INFO:teuthology.orchestra.run.smithi088.stderr:2024-06-03T08:36:35.154+0000 7fef1dfb2640 1 librados: shutdown 2024-06-03T08:36:35.165 INFO:tasks.ceph.ceph_manager.ceph:need seq 60129542152 got 60129542152 for osd.3 2024-06-03T08:36:35.166 DEBUG:teuthology.parallel:result is None 2024-06-03T08:36:35.188 INFO:teuthology.orchestra.run.smithi088.stderr:2024-06-03T08:36:35.188+0000 7f3dd9ffb640 1 -- 172.21.15.88:0/1593134679 <== mon.0 v1:172.21.15.88:6789/0 9 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+182498 (unknown 1092875540 0 3016760700) 0x7f3de004f950 con 0x7f3de40a8960 2024-06-03T08:36:35.188 INFO:teuthology.orchestra.run.smithi088.stderr:2024-06-03T08:36:35.188+0000 7f3dd9ffb640 10 monclient: handle_mon_command_ack 1 [{"prefix": "get_command_descriptions"}] 2024-06-03T08:36:35.188 INFO:teuthology.orchestra.run.smithi088.stderr:2024-06-03T08:36:35.188+0000 7f3dd9ffb640 10 monclient: _finish_command 1 = system:0 2024-06-03T08:36:35.190 INFO:teuthology.orchestra.run.smithi088.stderr:2024-06-03T08:36:35.190+0000 7fbade7fc640 1 -- 172.21.15.88:0/1627903445 <== mon.0 v1:172.21.15.88:6789/0 10 ==== mon_command_ack([{"prefix": "osd last-stat-seq", "id": 7}]=0 v0) v1 ==== 74+0+12 (unknown 1482059429 0 3651700234) 0x7fbad8051940 con 0x7fbae80b4250 2024-06-03T08:36:35.190 INFO:teuthology.orchestra.run.smithi088.stderr:2024-06-03T08:36:35.190+0000 7fbade7fc640 10 monclient: handle_mon_command_ack 2 [{"prefix": "osd last-stat-seq", "id": 7}] 2024-06-03T08:36:35.190 INFO:teuthology.orchestra.run.smithi088.stderr:2024-06-03T08:36:35.190+0000 7fbade7fc640 10 monclient: _finish_command 2 = system:0 2024-06-03T08:36:35.191 INFO:teuthology.orchestra.run.smithi088.stdout:85899345928 2024-06-03T08:36:35.193 INFO:teuthology.orchestra.run.smithi088.stderr:2024-06-03T08:36:35.192+0000 7fbaefed1640 10 librados: watch_flush enter 2024-06-03T08:36:35.193 INFO:teuthology.orchestra.run.smithi088.stderr:2024-06-03T08:36:35.192+0000 7fbaefed1640 10 librados: watch_flush exit 2024-06-03T08:36:35.193 INFO:teuthology.orchestra.run.smithi088.stderr:2024-06-03T08:36:35.192+0000 7fbaefed1640 20 client.4374.objecter shutdown clearing up homeless session... 2024-06-03T08:36:35.193 INFO:teuthology.orchestra.run.smithi088.stderr:2024-06-03T08:36:35.192+0000 7fbaefed1640 10 client.4374.objecter successfully canceled tick 2024-06-03T08:36:35.193 INFO:teuthology.orchestra.run.smithi088.stderr:2024-06-03T08:36:35.192+0000 7fbaefed1640 1 -- 172.21.15.88:0/1627903445 >> v1:172.21.15.192:6800/34229 conn(0x7fbacc06d220 legacy=0x7fbacc06f6e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-06-03T08:36:35.193 INFO:teuthology.orchestra.run.smithi088.stderr:2024-06-03T08:36:35.192+0000 7fbaefed1640 10 monclient: shutdown 2024-06-03T08:36:35.193 INFO:teuthology.orchestra.run.smithi088.stderr:2024-06-03T08:36:35.192+0000 7fbaefed1640 20 monclient: shutdown discarding 0 pending message(s) 2024-06-03T08:36:35.193 INFO:teuthology.orchestra.run.smithi088.stderr:2024-06-03T08:36:35.192+0000 7fbaefed1640 1 -- 172.21.15.88:0/1627903445 >> v1:172.21.15.88:6789/0 conn(0x7fbae80b4250 legacy=0x7fbae8134290 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-06-03T08:36:35.194 INFO:teuthology.orchestra.run.smithi088.stderr:2024-06-03T08:36:35.192+0000 7fbaefed1640 1 -- 172.21.15.88:0/1627903445 shutdown_connections 2024-06-03T08:36:35.194 INFO:teuthology.orchestra.run.smithi088.stderr:2024-06-03T08:36:35.192+0000 7fbaefed1640 1 -- 172.21.15.88:0/1627903445 >> 172.21.15.88:0/1627903445 conn(0x7fbae80717f0 msgr2=0x7fbae80b4e60 unknown :-1 s=STATE_NONE l=0).mark_down 2024-06-03T08:36:35.194 INFO:teuthology.orchestra.run.smithi088.stderr:2024-06-03T08:36:35.192+0000 7fbaefed1640 1 -- 172.21.15.88:0/1627903445 shutdown_connections 2024-06-03T08:36:35.194 INFO:teuthology.orchestra.run.smithi088.stderr:2024-06-03T08:36:35.192+0000 7fbaefed1640 1 -- 172.21.15.88:0/1627903445 wait complete. 2024-06-03T08:36:35.194 INFO:teuthology.orchestra.run.smithi088.stderr:2024-06-03T08:36:35.192+0000 7fbaefed1640 1 librados: shutdown 2024-06-03T08:36:35.205 INFO:tasks.ceph.ceph_manager.ceph:need seq 85899345928 got 85899345928 for osd.7 2024-06-03T08:36:35.205 DEBUG:teuthology.parallel:result is None 2024-06-03T08:36:35.296 INFO:teuthology.orchestra.run.smithi088.stderr:2024-06-03T08:36:35.296+0000 7f0624b5f640 10 start_mon_command cmd=[{"prefix": "osd last-stat-seq", "id": 4}] 2024-06-03T08:36:35.297 INFO:teuthology.orchestra.run.smithi088.stderr:2024-06-03T08:36:35.296+0000 7f0624b5f640 10 monclient: _send_command 2 [{"prefix": "osd last-stat-seq", "id": 4}] 2024-06-03T08:36:35.297 INFO:teuthology.orchestra.run.smithi088.stderr:2024-06-03T08:36:35.296+0000 7f0624b5f640 10 monclient: _send_mon_message to mon.b at v1:172.21.15.192:6789/0 2024-06-03T08:36:35.297 INFO:teuthology.orchestra.run.smithi088.stderr:2024-06-03T08:36:35.296+0000 7f0624b5f640 1 -- 172.21.15.88:0/228619988 --> v1:172.21.15.192:6789/0 -- mon_command({"prefix": "osd last-stat-seq", "id": 4} v 0) v1 -- 0x7f062013a9f0 con 0x7f06200ad980 2024-06-03T08:36:35.299 INFO:teuthology.orchestra.run.smithi088.stderr:2024-06-03T08:36:35.299+0000 7f0616ffd640 1 -- 172.21.15.88:0/228619988 <== mon.1 v1:172.21.15.192:6789/0 10 ==== mon_command_ack([{"prefix": "osd last-stat-seq", "id": 4}]=0 v0) v1 ==== 74+0+12 (unknown 1823589948 0 3651700234) 0x7f0608051c00 con 0x7f06200ad980 2024-06-03T08:36:35.299 INFO:teuthology.orchestra.run.smithi088.stderr:2024-06-03T08:36:35.299+0000 7f0616ffd640 10 monclient: handle_mon_command_ack 2 [{"prefix": "osd last-stat-seq", "id": 4}] 2024-06-03T08:36:35.299 INFO:teuthology.orchestra.run.smithi088.stderr:2024-06-03T08:36:35.299+0000 7f0616ffd640 10 monclient: _finish_command 2 = system:0 2024-06-03T08:36:35.300 INFO:teuthology.orchestra.run.smithi088.stdout:85899345928 2024-06-03T08:36:35.302 INFO:teuthology.orchestra.run.smithi088.stderr:2024-06-03T08:36:35.301+0000 7f0624b5f640 10 librados: watch_flush enter 2024-06-03T08:36:35.302 INFO:teuthology.orchestra.run.smithi088.stderr:2024-06-03T08:36:35.301+0000 7f0624b5f640 10 librados: watch_flush exit 2024-06-03T08:36:35.302 INFO:teuthology.orchestra.run.smithi088.stderr:2024-06-03T08:36:35.301+0000 7f0624b5f640 20 client.4381.objecter shutdown clearing up homeless session... 2024-06-03T08:36:35.302 INFO:teuthology.orchestra.run.smithi088.stderr:2024-06-03T08:36:35.301+0000 7f0624b5f640 10 client.4381.objecter successfully canceled tick 2024-06-03T08:36:35.302 INFO:teuthology.orchestra.run.smithi088.stderr:2024-06-03T08:36:35.301+0000 7f0624b5f640 1 -- 172.21.15.88:0/228619988 >> v1:172.21.15.192:6800/34229 conn(0x7f060006cff0 legacy=0x7f060006f4b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-06-03T08:36:35.302 INFO:teuthology.orchestra.run.smithi088.stderr:2024-06-03T08:36:35.301+0000 7f0624b5f640 10 monclient: shutdown 2024-06-03T08:36:35.302 INFO:teuthology.orchestra.run.smithi088.stderr:2024-06-03T08:36:35.301+0000 7f0624b5f640 20 monclient: shutdown discarding 0 pending message(s) 2024-06-03T08:36:35.302 INFO:teuthology.orchestra.run.smithi088.stderr:2024-06-03T08:36:35.301+0000 7f0624b5f640 1 -- 172.21.15.88:0/228619988 >> v1:172.21.15.192:6789/0 conn(0x7f06200ad980 legacy=0x7f062012e0f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-06-03T08:36:35.302 INFO:teuthology.orchestra.run.smithi088.stderr:2024-06-03T08:36:35.301+0000 7f0624b5f640 1 -- 172.21.15.88:0/228619988 shutdown_connections 2024-06-03T08:36:35.303 INFO:teuthology.orchestra.run.smithi088.stderr:2024-06-03T08:36:35.301+0000 7f0624b5f640 1 -- 172.21.15.88:0/228619988 >> 172.21.15.88:0/228619988 conn(0x7f06200b5120 msgr2=0x7f06200b5500 unknown :-1 s=STATE_NONE l=0).mark_down 2024-06-03T08:36:35.303 INFO:teuthology.orchestra.run.smithi088.stderr:2024-06-03T08:36:35.301+0000 7f0624b5f640 1 -- 172.21.15.88:0/228619988 shutdown_connections 2024-06-03T08:36:35.303 INFO:teuthology.orchestra.run.smithi088.stderr:2024-06-03T08:36:35.301+0000 7f0624b5f640 1 -- 172.21.15.88:0/228619988 wait complete. 2024-06-03T08:36:35.303 INFO:teuthology.orchestra.run.smithi088.stderr:2024-06-03T08:36:35.301+0000 7f0624b5f640 1 librados: shutdown 2024-06-03T08:36:35.313 INFO:tasks.ceph.ceph_manager.ceph:need seq 85899345928 got 85899345928 for osd.4 2024-06-03T08:36:35.313 DEBUG:teuthology.parallel:result is None 2024-06-03T08:36:35.380 INFO:teuthology.orchestra.run.smithi088.stderr:2024-06-03T08:36:35.380+0000 7f3deb5b9640 10 start_mon_command cmd=[{"prefix": "osd last-stat-seq", "id": 0}] 2024-06-03T08:36:35.381 INFO:teuthology.orchestra.run.smithi088.stderr:2024-06-03T08:36:35.380+0000 7f3deb5b9640 10 monclient: _send_command 2 [{"prefix": "osd last-stat-seq", "id": 0}] 2024-06-03T08:36:35.381 INFO:teuthology.orchestra.run.smithi088.stderr:2024-06-03T08:36:35.380+0000 7f3deb5b9640 10 monclient: _send_mon_message to mon.a at v1:172.21.15.88:6789/0 2024-06-03T08:36:35.381 INFO:teuthology.orchestra.run.smithi088.stderr:2024-06-03T08:36:35.380+0000 7f3deb5b9640 1 -- 172.21.15.88:0/1593134679 --> v1:172.21.15.88:6789/0 -- mon_command({"prefix": "osd last-stat-seq", "id": 0} v 0) v1 -- 0x7f3de40aae40 con 0x7f3de40a8960 2024-06-03T08:36:35.392 INFO:teuthology.orchestra.run.smithi088.stderr:2024-06-03T08:36:35.392+0000 7f3dd9ffb640 1 -- 172.21.15.88:0/1593134679 <== mon.0 v1:172.21.15.88:6789/0 10 ==== mon_command_ack([{"prefix": "osd last-stat-seq", "id": 0}]=0 v0) v1 ==== 74+0+12 (unknown 574334944 0 3426929836) 0x7f3de0051a50 con 0x7f3de40a8960 2024-06-03T08:36:35.392 INFO:teuthology.orchestra.run.smithi088.stderr:2024-06-03T08:36:35.392+0000 7f3dd9ffb640 10 monclient: handle_mon_command_ack 2 [{"prefix": "osd last-stat-seq", "id": 0}] 2024-06-03T08:36:35.392 INFO:teuthology.orchestra.run.smithi088.stderr:2024-06-03T08:36:35.392+0000 7f3dd9ffb640 10 monclient: _finish_command 2 = system:0 2024-06-03T08:36:35.393 INFO:teuthology.orchestra.run.smithi088.stdout:60129542151 2024-06-03T08:36:35.395 INFO:teuthology.orchestra.run.smithi088.stderr:2024-06-03T08:36:35.394+0000 7f3deb5b9640 10 librados: watch_flush enter 2024-06-03T08:36:35.395 INFO:teuthology.orchestra.run.smithi088.stderr:2024-06-03T08:36:35.394+0000 7f3deb5b9640 10 librados: watch_flush exit 2024-06-03T08:36:35.395 INFO:teuthology.orchestra.run.smithi088.stderr:2024-06-03T08:36:35.394+0000 7f3deb5b9640 20 client.4377.objecter shutdown clearing up homeless session... 2024-06-03T08:36:35.395 INFO:teuthology.orchestra.run.smithi088.stderr:2024-06-03T08:36:35.394+0000 7f3deb5b9640 10 client.4377.objecter successfully canceled tick 2024-06-03T08:36:35.395 INFO:teuthology.orchestra.run.smithi088.stderr:2024-06-03T08:36:35.394+0000 7f3deb5b9640 1 -- 172.21.15.88:0/1593134679 >> v1:172.21.15.192:6800/34229 conn(0x7f3dc806d250 legacy=0x7f3dc806f710 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-06-03T08:36:35.395 INFO:teuthology.orchestra.run.smithi088.stderr:2024-06-03T08:36:35.394+0000 7f3deb5b9640 10 monclient: shutdown 2024-06-03T08:36:35.395 INFO:teuthology.orchestra.run.smithi088.stderr:2024-06-03T08:36:35.394+0000 7f3deb5b9640 20 monclient: shutdown discarding 0 pending message(s) 2024-06-03T08:36:35.396 INFO:teuthology.orchestra.run.smithi088.stderr:2024-06-03T08:36:35.394+0000 7f3deb5b9640 1 -- 172.21.15.88:0/1593134679 >> v1:172.21.15.88:6789/0 conn(0x7f3de40a8960 legacy=0x7f3de41d6750 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-06-03T08:36:35.396 INFO:teuthology.orchestra.run.smithi088.stderr:2024-06-03T08:36:35.394+0000 7f3deb5b9640 1 -- 172.21.15.88:0/1593134679 shutdown_connections 2024-06-03T08:36:35.396 INFO:teuthology.orchestra.run.smithi088.stderr:2024-06-03T08:36:35.394+0000 7f3deb5b9640 1 -- 172.21.15.88:0/1593134679 >> 172.21.15.88:0/1593134679 conn(0x7f3de40b70d0 msgr2=0x7f3de413e580 unknown :-1 s=STATE_NONE l=0).mark_down 2024-06-03T08:36:35.396 INFO:teuthology.orchestra.run.smithi088.stderr:2024-06-03T08:36:35.394+0000 7f3deb5b9640 1 -- 172.21.15.88:0/1593134679 shutdown_connections 2024-06-03T08:36:35.396 INFO:teuthology.orchestra.run.smithi088.stderr:2024-06-03T08:36:35.394+0000 7f3deb5b9640 1 -- 172.21.15.88:0/1593134679 wait complete. 2024-06-03T08:36:35.396 INFO:teuthology.orchestra.run.smithi088.stderr:2024-06-03T08:36:35.394+0000 7f3deb5b9640 1 librados: shutdown 2024-06-03T08:36:35.405 INFO:tasks.ceph.ceph_manager.ceph:need seq 60129542150 got 60129542151 for osd.0 2024-06-03T08:36:35.406 DEBUG:teuthology.parallel:result is None 2024-06-03T08:36:35.406 INFO:tasks.ceph.ceph_manager.ceph:waiting for clean 2024-06-03T08:36:35.406 DEBUG:teuthology.orchestra.run.smithi088:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph pg dump --format=json 2024-06-03T08:36:35.505 INFO:teuthology.orchestra.run.smithi088.stderr:2024-06-03T08:36:35.504+0000 7f46ba069640 10 monclient: get_monmap_and_config 2024-06-03T08:36:35.505 INFO:teuthology.orchestra.run.smithi088.stderr:2024-06-03T08:36:35.504+0000 7f46ba069640 10 monclient: build_initial_monmap 2024-06-03T08:36:35.505 INFO:teuthology.orchestra.run.smithi088.stderr:2024-06-03T08:36:35.504+0000 7f46ba069640 10 monclient: monmap: 2024-06-03T08:36:35.505 INFO:teuthology.orchestra.run.smithi088.stderr:epoch 0 2024-06-03T08:36:35.505 INFO:teuthology.orchestra.run.smithi088.stderr:fsid ec3c3fdd-3ac6-4d7d-9412-683130ed8b04 2024-06-03T08:36:35.505 INFO:teuthology.orchestra.run.smithi088.stderr:last_changed 2024-06-03T08:36:35.505644+0000 2024-06-03T08:36:35.505 INFO:teuthology.orchestra.run.smithi088.stderr:created 2024-06-03T08:36:35.505644+0000 2024-06-03T08:36:35.505 INFO:teuthology.orchestra.run.smithi088.stderr:min_mon_release 0 (unknown) 2024-06-03T08:36:35.505 INFO:teuthology.orchestra.run.smithi088.stderr:election_strategy: 3 2024-06-03T08:36:35.505 INFO:teuthology.orchestra.run.smithi088.stderr:0: v1:172.21.15.88:6789/0 mon.noname-a 2024-06-03T08:36:35.505 INFO:teuthology.orchestra.run.smithi088.stderr:1: v1:172.21.15.192:6789/0 mon.noname-c 2024-06-03T08:36:35.506 INFO:teuthology.orchestra.run.smithi088.stderr:2: v1:172.21.15.88:6790/0 mon.noname-b 2024-06-03T08:36:35.506 INFO:teuthology.orchestra.run.smithi088.stderr: 2024-06-03T08:36:35.507 INFO:teuthology.orchestra.run.smithi088.stderr:2024-06-03T08:36:35.505+0000 7f46ba069640 1 Processor -- start 2024-06-03T08:36:35.507 INFO:teuthology.orchestra.run.smithi088.stderr:2024-06-03T08:36:35.505+0000 7f46ba069640 1 -- start start 2024-06-03T08:36:35.507 INFO:teuthology.orchestra.run.smithi088.stderr:2024-06-03T08:36:35.505+0000 7f46ba069640 10 monclient: init 2024-06-03T08:36:35.507 INFO:teuthology.orchestra.run.smithi088.stderr:2024-06-03T08:36:35.505+0000 7f46ba069640 10 monclient: _reopen_session rank -1 2024-06-03T08:36:35.507 INFO:teuthology.orchestra.run.smithi088.stderr:2024-06-03T08:36:35.505+0000 7f46ba069640 10 monclient: _add_conns ranks=[2,0,1] 2024-06-03T08:36:35.507 INFO:teuthology.orchestra.run.smithi088.stderr:2024-06-03T08:36:35.505+0000 7f46ba069640 10 monclient(hunting): picked mon.noname-b con 0x7f46b40adb90 addr v1:172.21.15.88:6790/0 2024-06-03T08:36:35.507 INFO:teuthology.orchestra.run.smithi088.stderr:2024-06-03T08:36:35.505+0000 7f46ba069640 10 monclient(hunting): picked mon.noname-a con 0x7f46b406ee00 addr v1:172.21.15.88:6789/0 2024-06-03T08:36:35.507 INFO:teuthology.orchestra.run.smithi088.stderr:2024-06-03T08:36:35.506+0000 7f46ba069640 10 monclient(hunting): picked mon.noname-c con 0x7f46b41383e0 addr v1:172.21.15.192:6789/0 2024-06-03T08:36:35.507 INFO:teuthology.orchestra.run.smithi088.stderr:2024-06-03T08:36:35.506+0000 7f46ba069640 1 -- --> v1:172.21.15.88:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f46b406f2c0 con 0x7f46b406ee00 2024-06-03T08:36:35.508 INFO:teuthology.orchestra.run.smithi088.stderr:2024-06-03T08:36:35.506+0000 7f46ba069640 1 -- --> v1:172.21.15.192:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f46b40710c0 con 0x7f46b41383e0 2024-06-03T08:36:35.508 INFO:teuthology.orchestra.run.smithi088.stderr:2024-06-03T08:36:35.506+0000 7f46ba069640 1 -- --> v1:172.21.15.88:6790/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f46b4071fd0 con 0x7f46b40adb90 2024-06-03T08:36:35.508 INFO:teuthology.orchestra.run.smithi088.stderr:2024-06-03T08:36:35.506+0000 7f46ba069640 10 monclient(hunting): _renew_subs 2024-06-03T08:36:35.508 INFO:teuthology.orchestra.run.smithi088.stderr:2024-06-03T08:36:35.506+0000 7f46ba069640 10 monclient(hunting): authenticate will time out at 1251.315978s 2024-06-03T08:36:35.508 INFO:teuthology.orchestra.run.smithi088.stderr:2024-06-03T08:36:35.506+0000 7f46b2ffd640 1 --1- >> v1:172.21.15.88:6789/0 conn(0x7f46b406ee00 0x7f46b4137cd0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.88:6789/0 says I am v1:172.21.15.88:60616/0 (socket says 172.21.15.88:60616) 2024-06-03T08:36:35.508 INFO:teuthology.orchestra.run.smithi088.stderr:2024-06-03T08:36:35.506+0000 7f46b37fe640 1 --1- >> v1:172.21.15.88:6790/0 conn(0x7f46b40adb90 0x7f46b40718c0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.88:6790/0 says I am v1:172.21.15.88:54500/0 (socket says 172.21.15.88:54500) 2024-06-03T08:36:35.508 INFO:teuthology.orchestra.run.smithi088.stderr:2024-06-03T08:36:35.506+0000 7f46b2ffd640 1 -- 172.21.15.88:0/3620826712 learned_addr learned my addr 172.21.15.88:0/3620826712 (peer_addr_for_me v1:172.21.15.88:0/0) 2024-06-03T08:36:35.508 INFO:teuthology.orchestra.run.smithi088.stderr:2024-06-03T08:36:35.506+0000 7f46b3fff640 1 --1- 172.21.15.88:0/3620826712 >> v1:172.21.15.192:6789/0 conn(0x7f46b41383e0 0x7f46b4149af0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.192:6789/0 says I am v1:172.21.15.88:52734/0 (socket says 172.21.15.88:52734) 2024-06-03T08:36:35.509 INFO:teuthology.orchestra.run.smithi088.stderr:2024-06-03T08:36:35.508+0000 7f46b27fc640 1 -- 172.21.15.88:0/3620826712 <== mon.1 v1:172.21.15.192:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 164606769 0 0) 0x7f46b40710c0 con 0x7f46b41383e0 2024-06-03T08:36:35.509 INFO:teuthology.orchestra.run.smithi088.stderr:2024-06-03T08:36:35.508+0000 7f46b27fc640 10 monclient(hunting): _negotiate 2024-06-03T08:36:35.509 INFO:teuthology.orchestra.run.smithi088.stderr:2024-06-03T08:36:35.508+0000 7f46b27fc640 10 monclient(hunting): _init_auth method 2 2024-06-03T08:36:35.509 INFO:teuthology.orchestra.run.smithi088.stderr:2024-06-03T08:36:35.508+0000 7f46b27fc640 10 monclient(hunting): _init_auth creating new auth 2024-06-03T08:36:35.509 INFO:teuthology.orchestra.run.smithi088.stderr:2024-06-03T08:36:35.508+0000 7f46b27fc640 10 monclient(hunting): my global_id is 4387 2024-06-03T08:36:35.510 INFO:teuthology.orchestra.run.smithi088.stderr:2024-06-03T08:36:35.508+0000 7f46b27fc640 1 -- 172.21.15.88:0/3620826712 --> v1:172.21.15.192:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f46a0003520 con 0x7f46b41383e0 2024-06-03T08:36:35.510 INFO:teuthology.orchestra.run.smithi088.stderr:2024-06-03T08:36:35.509+0000 7f46b27fc640 1 -- 172.21.15.88:0/3620826712 <== mon.0 v1:172.21.15.88:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 2111600125 0 0) 0x7f46b406f2c0 con 0x7f46b406ee00 2024-06-03T08:36:35.510 INFO:teuthology.orchestra.run.smithi088.stderr:2024-06-03T08:36:35.509+0000 7f46b27fc640 10 monclient(hunting): _negotiate 2024-06-03T08:36:35.510 INFO:teuthology.orchestra.run.smithi088.stderr:2024-06-03T08:36:35.509+0000 7f46b27fc640 10 monclient(hunting): _init_auth method 2 2024-06-03T08:36:35.510 INFO:teuthology.orchestra.run.smithi088.stderr:2024-06-03T08:36:35.509+0000 7f46b27fc640 10 monclient(hunting): _init_auth creating new auth 2024-06-03T08:36:35.510 INFO:teuthology.orchestra.run.smithi088.stderr:2024-06-03T08:36:35.509+0000 7f46b27fc640 10 monclient(hunting): my global_id is 4380 2024-06-03T08:36:35.510 INFO:teuthology.orchestra.run.smithi088.stderr:2024-06-03T08:36:35.509+0000 7f46b27fc640 1 -- 172.21.15.88:0/3620826712 --> v1:172.21.15.88:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f46b40710c0 con 0x7f46b406ee00 2024-06-03T08:36:35.510 INFO:teuthology.orchestra.run.smithi088.stderr:2024-06-03T08:36:35.509+0000 7f46b27fc640 1 -- 172.21.15.88:0/3620826712 <== mon.2 v1:172.21.15.88:6790/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 2761036932 0 0) 0x7f46b4071fd0 con 0x7f46b40adb90 2024-06-03T08:36:35.510 INFO:teuthology.orchestra.run.smithi088.stderr:2024-06-03T08:36:35.509+0000 7f46b27fc640 10 monclient(hunting): _negotiate 2024-06-03T08:36:35.510 INFO:teuthology.orchestra.run.smithi088.stderr:2024-06-03T08:36:35.509+0000 7f46b27fc640 10 monclient(hunting): _init_auth method 2 2024-06-03T08:36:35.510 INFO:teuthology.orchestra.run.smithi088.stderr:2024-06-03T08:36:35.509+0000 7f46b27fc640 10 monclient(hunting): _init_auth creating new auth 2024-06-03T08:36:35.510 INFO:teuthology.orchestra.run.smithi088.stderr:2024-06-03T08:36:35.509+0000 7f46b27fc640 10 monclient(hunting): my global_id is 4391 2024-06-03T08:36:35.511 INFO:teuthology.orchestra.run.smithi088.stderr:2024-06-03T08:36:35.509+0000 7f46b27fc640 1 -- 172.21.15.88:0/3620826712 --> v1:172.21.15.88:6790/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f46b406f2c0 con 0x7f46b40adb90 2024-06-03T08:36:35.512 INFO:teuthology.orchestra.run.smithi088.stderr:2024-06-03T08:36:35.511+0000 7f46b27fc640 1 -- 172.21.15.88:0/3620826712 <== mon.1 v1:172.21.15.192:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 214+0+0 (unknown 1073354855 0 0) 0x7f46a0003520 con 0x7f46b41383e0 2024-06-03T08:36:35.512 INFO:teuthology.orchestra.run.smithi088.stderr:2024-06-03T08:36:35.511+0000 7f46b27fc640 1 -- 172.21.15.88:0/3620826712 >> v1:172.21.15.88:6790/0 conn(0x7f46b40adb90 legacy=0x7f46b40718c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-06-03T08:36:35.512 INFO:teuthology.orchestra.run.smithi088.stderr:2024-06-03T08:36:35.511+0000 7f46b27fc640 1 -- 172.21.15.88:0/3620826712 >> v1:172.21.15.88:6789/0 conn(0x7f46b406ee00 legacy=0x7f46b4137cd0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-06-03T08:36:35.512 INFO:teuthology.orchestra.run.smithi088.stderr:2024-06-03T08:36:35.511+0000 7f46b27fc640 10 monclient: _finish_hunting 0 2024-06-03T08:36:35.512 INFO:teuthology.orchestra.run.smithi088.stderr:2024-06-03T08:36:35.511+0000 7f46b27fc640 1 monclient: found mon.noname-c 2024-06-03T08:36:35.512 INFO:teuthology.orchestra.run.smithi088.stderr:2024-06-03T08:36:35.511+0000 7f46b27fc640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-06-03T08:36:35.512 INFO:teuthology.orchestra.run.smithi088.stderr:2024-06-03T08:36:35.511+0000 7f46b27fc640 10 monclient: _send_mon_message to mon.noname-c at v1:172.21.15.192:6789/0 2024-06-03T08:36:35.512 INFO:teuthology.orchestra.run.smithi088.stderr:2024-06-03T08:36:35.511+0000 7f46b27fc640 1 -- 172.21.15.88:0/3620826712 --> v1:172.21.15.192:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f46b4141750 con 0x7f46b41383e0 2024-06-03T08:36:35.513 INFO:teuthology.orchestra.run.smithi088.stderr:2024-06-03T08:36:35.511+0000 7f46b27fc640 10 monclient: _finish_auth 0 2024-06-03T08:36:35.513 INFO:teuthology.orchestra.run.smithi088.stderr:2024-06-03T08:36:35.511+0000 7f46b27fc640 10 monclient: _check_auth_tickets 2024-06-03T08:36:35.513 INFO:teuthology.orchestra.run.smithi088.stderr:2024-06-03T08:36:35.511+0000 7f46b27fc640 20 monclient: _check_auth_rotating not needed by client.admin 2024-06-03T08:36:35.513 INFO:teuthology.orchestra.run.smithi088.stderr:2024-06-03T08:36:35.512+0000 7f46ba069640 5 monclient: authenticate success, global_id 4387 2024-06-03T08:36:35.513 INFO:teuthology.orchestra.run.smithi088.stderr:2024-06-03T08:36:35.512+0000 7f46ba069640 20 monclient: get_monmap_and_config waiting for monmap|config 2024-06-03T08:36:35.514 INFO:teuthology.orchestra.run.smithi088.stderr:2024-06-03T08:36:35.512+0000 7f46b27fc640 1 -- 172.21.15.88:0/3620826712 <== mon.1 v1:172.21.15.192:6789/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 2234374065 0 0) 0x7f46ac001990 con 0x7f46b41383e0 2024-06-03T08:36:35.514 INFO:teuthology.orchestra.run.smithi088.stderr:2024-06-03T08:36:35.512+0000 7f46b27fc640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-06-03T08:36:35.514 INFO:teuthology.orchestra.run.smithi088.stderr:2024-06-03T08:36:35.512+0000 7f46b27fc640 10 monclient: got monmap 1 from mon.noname-c (according to old e1) 2024-06-03T08:36:35.514 INFO:teuthology.orchestra.run.smithi088.stderr:2024-06-03T08:36:35.512+0000 7f46b27fc640 10 monclient: dump: 2024-06-03T08:36:35.514 INFO:teuthology.orchestra.run.smithi088.stderr:epoch 1 2024-06-03T08:36:35.514 INFO:teuthology.orchestra.run.smithi088.stderr:fsid ec3c3fdd-3ac6-4d7d-9412-683130ed8b04 2024-06-03T08:36:35.514 INFO:teuthology.orchestra.run.smithi088.stderr:last_changed 2024-06-03T08:34:31.885125+0000 2024-06-03T08:36:35.514 INFO:teuthology.orchestra.run.smithi088.stderr:created 2024-06-03T08:34:31.885125+0000 2024-06-03T08:36:35.514 INFO:teuthology.orchestra.run.smithi088.stderr:min_mon_release 18 (reef) 2024-06-03T08:36:35.514 INFO:teuthology.orchestra.run.smithi088.stderr:election_strategy: 3 2024-06-03T08:36:35.514 INFO:teuthology.orchestra.run.smithi088.stderr:0: v1:172.21.15.88:6789/0 mon.a 2024-06-03T08:36:35.514 INFO:teuthology.orchestra.run.smithi088.stderr:1: v1:172.21.15.192:6789/0 mon.b 2024-06-03T08:36:35.515 INFO:teuthology.orchestra.run.smithi088.stderr:2: v1:172.21.15.88:6790/0 mon.c 2024-06-03T08:36:35.515 INFO:teuthology.orchestra.run.smithi088.stderr: 2024-06-03T08:36:35.515 INFO:teuthology.orchestra.run.smithi088.stderr:2024-06-03T08:36:35.512+0000 7f46ba069640 20 monclient: get_monmap_and_config waiting for monmap|config 2024-06-03T08:36:35.515 INFO:teuthology.orchestra.run.smithi088.stderr:2024-06-03T08:36:35.513+0000 7f46b27fc640 1 -- 172.21.15.88:0/3620826712 <== mon.1 v1:172.21.15.192:6789/0 4 ==== config(0 keys) v1 ==== 4+0+0 (unknown 0 0 0) 0x7f46ac001be0 con 0x7f46b41383e0 2024-06-03T08:36:35.515 INFO:teuthology.orchestra.run.smithi088.stderr:2024-06-03T08:36:35.513+0000 7f46b27fc640 10 monclient: handle_config config(0 keys) v1 2024-06-03T08:36:35.515 INFO:teuthology.orchestra.run.smithi088.stderr:2024-06-03T08:36:35.513+0000 7f46b27fc640 1 -- 172.21.15.88:0/3620826712 <== mon.1 v1:172.21.15.192:6789/0 5 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 2234374065 0 0) 0x7f46ac001f70 con 0x7f46b41383e0 2024-06-03T08:36:35.515 INFO:teuthology.orchestra.run.smithi088.stderr:2024-06-03T08:36:35.513+0000 7f46ba069640 10 monclient: get_monmap_and_config success 2024-06-03T08:36:35.515 INFO:teuthology.orchestra.run.smithi088.stderr:2024-06-03T08:36:35.513+0000 7f46ba069640 4 set_mon_vals no callback set 2024-06-03T08:36:35.515 INFO:teuthology.orchestra.run.smithi088.stderr:2024-06-03T08:36:35.513+0000 7f46b27fc640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-06-03T08:36:35.515 INFO:teuthology.orchestra.run.smithi088.stderr:2024-06-03T08:36:35.513+0000 7f46b27fc640 10 monclient: got monmap 1 from mon.b (according to old e1) 2024-06-03T08:36:35.515 INFO:teuthology.orchestra.run.smithi088.stderr:2024-06-03T08:36:35.513+0000 7f46b27fc640 10 monclient: dump: 2024-06-03T08:36:35.515 INFO:teuthology.orchestra.run.smithi088.stderr:epoch 1 2024-06-03T08:36:35.515 INFO:teuthology.orchestra.run.smithi088.stderr:fsid ec3c3fdd-3ac6-4d7d-9412-683130ed8b04 2024-06-03T08:36:35.515 INFO:teuthology.orchestra.run.smithi088.stderr:last_changed 2024-06-03T08:34:31.885125+0000 2024-06-03T08:36:35.516 INFO:teuthology.orchestra.run.smithi088.stderr:created 2024-06-03T08:34:31.885125+0000 2024-06-03T08:36:35.516 INFO:teuthology.orchestra.run.smithi088.stderr:min_mon_release 18 (reef) 2024-06-03T08:36:35.516 INFO:teuthology.orchestra.run.smithi088.stderr:election_strategy: 3 2024-06-03T08:36:35.516 INFO:teuthology.orchestra.run.smithi088.stderr:0: v1:172.21.15.88:6789/0 mon.a 2024-06-03T08:36:35.516 INFO:teuthology.orchestra.run.smithi088.stderr:1: v1:172.21.15.192:6789/0 mon.b 2024-06-03T08:36:35.516 INFO:teuthology.orchestra.run.smithi088.stderr:2: v1:172.21.15.88:6790/0 mon.c 2024-06-03T08:36:35.516 INFO:teuthology.orchestra.run.smithi088.stderr: 2024-06-03T08:36:35.519 INFO:teuthology.orchestra.run.smithi088.stderr:2024-06-03T08:36:35.514+0000 7f46ba069640 10 monclient: shutdown 2024-06-03T08:36:35.519 INFO:teuthology.orchestra.run.smithi088.stderr:2024-06-03T08:36:35.514+0000 7f46ba069640 20 monclient: shutdown discarding 0 pending message(s) 2024-06-03T08:36:35.519 INFO:teuthology.orchestra.run.smithi088.stderr:2024-06-03T08:36:35.514+0000 7f46ba069640 1 -- 172.21.15.88:0/3620826712 >> v1:172.21.15.192:6789/0 conn(0x7f46b41383e0 legacy=0x7f46b4149af0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-06-03T08:36:35.519 INFO:teuthology.orchestra.run.smithi088.stderr:2024-06-03T08:36:35.514+0000 7f46ba069640 1 -- 172.21.15.88:0/3620826712 shutdown_connections 2024-06-03T08:36:35.519 INFO:teuthology.orchestra.run.smithi088.stderr:2024-06-03T08:36:35.514+0000 7f46ba069640 1 -- 172.21.15.88:0/3620826712 >> 172.21.15.88:0/3620826712 conn(0x7f46b40b2f20 msgr2=0x7f46b40b3300 unknown :-1 s=STATE_NONE l=0).mark_down 2024-06-03T08:36:35.519 INFO:teuthology.orchestra.run.smithi088.stderr:2024-06-03T08:36:35.514+0000 7f46ba069640 1 -- 172.21.15.88:0/3620826712 shutdown_connections 2024-06-03T08:36:35.519 INFO:teuthology.orchestra.run.smithi088.stderr:2024-06-03T08:36:35.514+0000 7f46ba069640 1 -- 172.21.15.88:0/3620826712 wait complete. 2024-06-03T08:36:35.519 INFO:teuthology.orchestra.run.smithi088.stderr:2024-06-03T08:36:35.515+0000 7f46ba069640 10 monclient: build_initial_monmap 2024-06-03T08:36:35.519 INFO:teuthology.orchestra.run.smithi088.stderr:2024-06-03T08:36:35.515+0000 7f46ba069640 10 monclient: monmap: 2024-06-03T08:36:35.519 INFO:teuthology.orchestra.run.smithi088.stderr:epoch 0 2024-06-03T08:36:35.519 INFO:teuthology.orchestra.run.smithi088.stderr:fsid 00000000-0000-0000-0000-000000000000 2024-06-03T08:36:35.520 INFO:teuthology.orchestra.run.smithi088.stderr:last_changed 0.000000 2024-06-03T08:36:35.520 INFO:teuthology.orchestra.run.smithi088.stderr:created 0.000000 2024-06-03T08:36:35.520 INFO:teuthology.orchestra.run.smithi088.stderr:min_mon_release 0 (unknown) 2024-06-03T08:36:35.520 INFO:teuthology.orchestra.run.smithi088.stderr:election_strategy: 1 2024-06-03T08:36:35.520 INFO:teuthology.orchestra.run.smithi088.stderr:0: v1:172.21.15.88:6789/0 mon.noname-a 2024-06-03T08:36:35.520 INFO:teuthology.orchestra.run.smithi088.stderr:1: v1:172.21.15.192:6789/0 mon.noname-b 2024-06-03T08:36:35.520 INFO:teuthology.orchestra.run.smithi088.stderr:2: v1:172.21.15.88:6790/0 mon.noname-c 2024-06-03T08:36:35.520 INFO:teuthology.orchestra.run.smithi088.stderr: 2024-06-03T08:36:35.520 INFO:teuthology.orchestra.run.smithi088.stderr:2024-06-03T08:36:35.515+0000 7f46ba069640 1 librados: starting msgr at 2024-06-03T08:36:35.520 INFO:teuthology.orchestra.run.smithi088.stderr:2024-06-03T08:36:35.515+0000 7f46ba069640 1 librados: starting objecter 2024-06-03T08:36:35.520 INFO:teuthology.orchestra.run.smithi088.stderr:2024-06-03T08:36:35.515+0000 7f46ba069640 1 Processor -- start 2024-06-03T08:36:35.520 INFO:teuthology.orchestra.run.smithi088.stderr:2024-06-03T08:36:35.515+0000 7f46ba069640 1 -- start start 2024-06-03T08:36:35.520 INFO:teuthology.orchestra.run.smithi088.stderr:2024-06-03T08:36:35.515+0000 7f46ba069640 1 librados: setting wanted keys 2024-06-03T08:36:35.520 INFO:teuthology.orchestra.run.smithi088.stderr:2024-06-03T08:36:35.515+0000 7f46ba069640 1 librados: calling monclient init 2024-06-03T08:36:35.521 INFO:teuthology.orchestra.run.smithi088.stderr:2024-06-03T08:36:35.515+0000 7f46ba069640 10 monclient: init 2024-06-03T08:36:35.521 INFO:teuthology.orchestra.run.smithi088.stderr:2024-06-03T08:36:35.516+0000 7f46ba069640 10 monclient: _reopen_session rank -1 2024-06-03T08:36:35.521 INFO:teuthology.orchestra.run.smithi088.stderr:2024-06-03T08:36:35.516+0000 7f46ba069640 10 monclient: _add_conns ranks=[1,0,2] 2024-06-03T08:36:35.521 INFO:teuthology.orchestra.run.smithi088.stderr:2024-06-03T08:36:35.516+0000 7f46ba069640 10 monclient(hunting): picked mon.noname-b con 0x7f46b406ee00 addr v1:172.21.15.192:6789/0 2024-06-03T08:36:35.521 INFO:teuthology.orchestra.run.smithi088.stderr:2024-06-03T08:36:35.516+0000 7f46ba069640 10 monclient(hunting): picked mon.noname-a con 0x7f46b40adb90 addr v1:172.21.15.88:6789/0 2024-06-03T08:36:35.521 INFO:teuthology.orchestra.run.smithi088.stderr:2024-06-03T08:36:35.516+0000 7f46ba069640 10 monclient(hunting): picked mon.noname-c con 0x7f46b41383e0 addr v1:172.21.15.88:6790/0 2024-06-03T08:36:35.521 INFO:teuthology.orchestra.run.smithi088.stderr:2024-06-03T08:36:35.516+0000 7f46ba069640 1 -- --> v1:172.21.15.88:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f46b41d9ee0 con 0x7f46b40adb90 2024-06-03T08:36:35.521 INFO:teuthology.orchestra.run.smithi088.stderr:2024-06-03T08:36:35.516+0000 7f46ba069640 1 -- --> v1:172.21.15.192:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f46b4134c40 con 0x7f46b406ee00 2024-06-03T08:36:35.521 INFO:teuthology.orchestra.run.smithi088.stderr:2024-06-03T08:36:35.516+0000 7f46ba069640 1 -- --> v1:172.21.15.88:6790/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f46b41d9b00 con 0x7f46b41383e0 2024-06-03T08:36:35.521 INFO:teuthology.orchestra.run.smithi088.stderr:2024-06-03T08:36:35.516+0000 7f46ba069640 10 monclient(hunting): _renew_subs 2024-06-03T08:36:35.521 INFO:teuthology.orchestra.run.smithi088.stderr:2024-06-03T08:36:35.516+0000 7f46ba069640 10 monclient(hunting): authenticate will time out at 1251.326444s 2024-06-03T08:36:35.521 INFO:teuthology.orchestra.run.smithi088.stderr:2024-06-03T08:36:35.516+0000 7f46b37fe640 1 --1- >> v1:172.21.15.192:6789/0 conn(0x7f46b406ee00 0x7f46b41d6600 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=1).handle_server_banner_and_identify peer v1:172.21.15.192:6789/0 says I am v1:172.21.15.88:52750/0 (socket says 172.21.15.88:52750) 2024-06-03T08:36:35.521 INFO:teuthology.orchestra.run.smithi088.stderr:2024-06-03T08:36:35.516+0000 7f46b37fe640 1 -- 172.21.15.88:0/46137053 learned_addr learned my addr 172.21.15.88:0/46137053 (peer_addr_for_me v1:172.21.15.88:0/0) 2024-06-03T08:36:35.522 INFO:teuthology.orchestra.run.smithi088.stderr:2024-06-03T08:36:35.517+0000 7f46abfff640 10 client.?.objecter ms_handle_connect 0x7f46b406ee00 2024-06-03T08:36:35.522 INFO:teuthology.orchestra.run.smithi088.stderr:2024-06-03T08:36:35.517+0000 7f46abfff640 10 client.?.objecter resend_mon_ops 2024-06-03T08:36:35.522 INFO:teuthology.orchestra.run.smithi088.stderr:2024-06-03T08:36:35.517+0000 7f46abfff640 10 client.?.objecter ms_handle_connect 0x7f46b40adb90 2024-06-03T08:36:35.522 INFO:teuthology.orchestra.run.smithi088.stderr:2024-06-03T08:36:35.517+0000 7f46abfff640 10 client.?.objecter resend_mon_ops 2024-06-03T08:36:35.522 INFO:teuthology.orchestra.run.smithi088.stderr:2024-06-03T08:36:35.517+0000 7f46abfff640 10 client.?.objecter ms_handle_connect 0x7f46b41383e0 2024-06-03T08:36:35.522 INFO:teuthology.orchestra.run.smithi088.stderr:2024-06-03T08:36:35.517+0000 7f46abfff640 10 client.?.objecter resend_mon_ops 2024-06-03T08:36:35.522 INFO:teuthology.orchestra.run.smithi088.stderr:2024-06-03T08:36:35.518+0000 7f46abfff640 1 -- 172.21.15.88:0/46137053 <== mon.1 v1:172.21.15.192:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 1720617059 0 0) 0x7f46b4134c40 con 0x7f46b406ee00 2024-06-03T08:36:35.522 INFO:teuthology.orchestra.run.smithi088.stderr:2024-06-03T08:36:35.518+0000 7f46abfff640 10 monclient(hunting): _negotiate 2024-06-03T08:36:35.522 INFO:teuthology.orchestra.run.smithi088.stderr:2024-06-03T08:36:35.518+0000 7f46abfff640 10 monclient(hunting): _init_auth method 2 2024-06-03T08:36:35.522 INFO:teuthology.orchestra.run.smithi088.stderr:2024-06-03T08:36:35.518+0000 7f46abfff640 10 monclient(hunting): _init_auth creating new auth 2024-06-03T08:36:35.522 INFO:teuthology.orchestra.run.smithi088.stderr:2024-06-03T08:36:35.519+0000 7f46abfff640 10 monclient(hunting): my global_id is 4390 2024-06-03T08:36:35.522 INFO:teuthology.orchestra.run.smithi088.stderr:2024-06-03T08:36:35.519+0000 7f46abfff640 1 -- 172.21.15.88:0/46137053 --> v1:172.21.15.192:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f4694004eb0 con 0x7f46b406ee00 2024-06-03T08:36:35.524 INFO:teuthology.orchestra.run.smithi088.stderr:2024-06-03T08:36:35.519+0000 7f46abfff640 1 -- 172.21.15.88:0/46137053 <== mon.2 v1:172.21.15.88:6790/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 4108364843 0 0) 0x7f46b41d9b00 con 0x7f46b41383e0 2024-06-03T08:36:35.524 INFO:teuthology.orchestra.run.smithi088.stderr:2024-06-03T08:36:35.519+0000 7f46abfff640 10 monclient(hunting): _negotiate 2024-06-03T08:36:35.524 INFO:teuthology.orchestra.run.smithi088.stderr:2024-06-03T08:36:35.519+0000 7f46abfff640 10 monclient(hunting): _init_auth method 2 2024-06-03T08:36:35.524 INFO:teuthology.orchestra.run.smithi088.stderr:2024-06-03T08:36:35.519+0000 7f46abfff640 10 monclient(hunting): _init_auth creating new auth 2024-06-03T08:36:35.524 INFO:teuthology.orchestra.run.smithi088.stderr:2024-06-03T08:36:35.519+0000 7f46abfff640 10 monclient(hunting): my global_id is 4394 2024-06-03T08:36:35.524 INFO:teuthology.orchestra.run.smithi088.stderr:2024-06-03T08:36:35.519+0000 7f46abfff640 1 -- 172.21.15.88:0/46137053 --> v1:172.21.15.88:6790/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f46b4134c40 con 0x7f46b41383e0 2024-06-03T08:36:35.524 INFO:teuthology.orchestra.run.smithi088.stderr:2024-06-03T08:36:35.519+0000 7f46abfff640 1 -- 172.21.15.88:0/46137053 <== mon.0 v1:172.21.15.88:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 2402817947 0 0) 0x7f46b41d9ee0 con 0x7f46b40adb90 2024-06-03T08:36:35.524 INFO:teuthology.orchestra.run.smithi088.stderr:2024-06-03T08:36:35.519+0000 7f46abfff640 10 monclient(hunting): _negotiate 2024-06-03T08:36:35.524 INFO:teuthology.orchestra.run.smithi088.stderr:2024-06-03T08:36:35.519+0000 7f46abfff640 10 monclient(hunting): _init_auth method 2 2024-06-03T08:36:35.524 INFO:teuthology.orchestra.run.smithi088.stderr:2024-06-03T08:36:35.519+0000 7f46abfff640 10 monclient(hunting): _init_auth creating new auth 2024-06-03T08:36:35.524 INFO:teuthology.orchestra.run.smithi088.stderr:2024-06-03T08:36:35.519+0000 7f46abfff640 10 monclient(hunting): my global_id is 4383 2024-06-03T08:36:35.524 INFO:teuthology.orchestra.run.smithi088.stderr:2024-06-03T08:36:35.519+0000 7f46abfff640 1 -- 172.21.15.88:0/46137053 --> v1:172.21.15.88:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f46b41d9b00 con 0x7f46b40adb90 2024-06-03T08:36:35.525 INFO:teuthology.orchestra.run.smithi088.stderr:2024-06-03T08:36:35.523+0000 7f46abfff640 1 -- 172.21.15.88:0/46137053 <== mon.1 v1:172.21.15.192:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 764+0+0 (unknown 1477458473 0 0) 0x7f4694004eb0 con 0x7f46b406ee00 2024-06-03T08:36:35.525 INFO:teuthology.orchestra.run.smithi088.stderr:2024-06-03T08:36:35.523+0000 7f46abfff640 1 -- 172.21.15.88:0/46137053 --> v1:172.21.15.192:6789/0 -- auth(proto 2 165 bytes epoch 0) v1 -- 0x7f46b41d9ee0 con 0x7f46b406ee00 2024-06-03T08:36:35.525 INFO:teuthology.orchestra.run.smithi088.stderr:2024-06-03T08:36:35.523+0000 7f46abfff640 1 -- 172.21.15.88:0/46137053 <== mon.1 v1:172.21.15.192:6789/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 2234374065 0 0) 0x7f469c001a30 con 0x7f46b406ee00 2024-06-03T08:36:35.525 INFO:teuthology.orchestra.run.smithi088.stderr:2024-06-03T08:36:35.523+0000 7f46abfff640 10 monclient(hunting): handle_monmap mon_map magic: 0 v1 2024-06-03T08:36:35.525 INFO:teuthology.orchestra.run.smithi088.stderr:2024-06-03T08:36:35.523+0000 7f46abfff640 10 monclient(hunting): got monmap 1 from mon.noname-b (according to old e1) 2024-06-03T08:36:35.525 INFO:teuthology.orchestra.run.smithi088.stderr:2024-06-03T08:36:35.523+0000 7f46abfff640 10 monclient(hunting): dump: 2024-06-03T08:36:35.525 INFO:teuthology.orchestra.run.smithi088.stderr:epoch 1 2024-06-03T08:36:35.525 INFO:teuthology.orchestra.run.smithi088.stderr:fsid ec3c3fdd-3ac6-4d7d-9412-683130ed8b04 2024-06-03T08:36:35.525 INFO:teuthology.orchestra.run.smithi088.stderr:last_changed 2024-06-03T08:34:31.885125+0000 2024-06-03T08:36:35.525 INFO:teuthology.orchestra.run.smithi088.stderr:created 2024-06-03T08:34:31.885125+0000 2024-06-03T08:36:35.525 INFO:teuthology.orchestra.run.smithi088.stderr:min_mon_release 18 (reef) 2024-06-03T08:36:35.525 INFO:teuthology.orchestra.run.smithi088.stderr:election_strategy: 3 2024-06-03T08:36:35.525 INFO:teuthology.orchestra.run.smithi088.stderr:0: v1:172.21.15.88:6789/0 mon.a 2024-06-03T08:36:35.525 INFO:teuthology.orchestra.run.smithi088.stderr:1: v1:172.21.15.192:6789/0 mon.b 2024-06-03T08:36:35.526 INFO:teuthology.orchestra.run.smithi088.stderr:2: v1:172.21.15.88:6790/0 mon.c 2024-06-03T08:36:35.526 INFO:teuthology.orchestra.run.smithi088.stderr: 2024-06-03T08:36:35.526 INFO:teuthology.orchestra.run.smithi088.stderr:2024-06-03T08:36:35.523+0000 7f46abfff640 10 monclient(hunting): _finish_auth 0 2024-06-03T08:36:35.528 INFO:teuthology.orchestra.run.smithi088.stderr:2024-06-03T08:36:35.524+0000 7f46abfff640 1 -- 172.21.15.88:0/46137053 <== mon.2 v1:172.21.15.88:6790/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 764+0+0 (unknown 3282220010 0 0) 0x7f46b4134c40 con 0x7f46b41383e0 2024-06-03T08:36:35.528 INFO:teuthology.orchestra.run.smithi088.stderr:2024-06-03T08:36:35.524+0000 7f46abfff640 1 -- 172.21.15.88:0/46137053 --> v1:172.21.15.88:6790/0 -- auth(proto 2 165 bytes epoch 0) v1 -- 0x7f4694004eb0 con 0x7f46b41383e0 2024-06-03T08:36:35.528 INFO:teuthology.orchestra.run.smithi088.stderr:2024-06-03T08:36:35.524+0000 7f46abfff640 1 -- 172.21.15.88:0/46137053 <== mon.2 v1:172.21.15.88:6790/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 2234374065 0 0) 0x7f46ac001ef0 con 0x7f46b41383e0 2024-06-03T08:36:35.528 INFO:teuthology.orchestra.run.smithi088.stderr:2024-06-03T08:36:35.524+0000 7f46abfff640 10 monclient(hunting): handle_monmap mon_map magic: 0 v1 2024-06-03T08:36:35.528 INFO:teuthology.orchestra.run.smithi088.stderr:2024-06-03T08:36:35.524+0000 7f46abfff640 10 monclient(hunting): got monmap 1 from mon.c (according to old e1) 2024-06-03T08:36:35.528 INFO:teuthology.orchestra.run.smithi088.stderr:2024-06-03T08:36:35.524+0000 7f46abfff640 10 monclient(hunting): dump: 2024-06-03T08:36:35.528 INFO:teuthology.orchestra.run.smithi088.stderr:epoch 1 2024-06-03T08:36:35.528 INFO:teuthology.orchestra.run.smithi088.stderr:fsid ec3c3fdd-3ac6-4d7d-9412-683130ed8b04 2024-06-03T08:36:35.528 INFO:teuthology.orchestra.run.smithi088.stderr:last_changed 2024-06-03T08:34:31.885125+0000 2024-06-03T08:36:35.528 INFO:teuthology.orchestra.run.smithi088.stderr:created 2024-06-03T08:34:31.885125+0000 2024-06-03T08:36:35.529 INFO:teuthology.orchestra.run.smithi088.stderr:min_mon_release 18 (reef) 2024-06-03T08:36:35.529 INFO:teuthology.orchestra.run.smithi088.stderr:election_strategy: 3 2024-06-03T08:36:35.529 INFO:teuthology.orchestra.run.smithi088.stderr:0: v1:172.21.15.88:6789/0 mon.a 2024-06-03T08:36:35.529 INFO:teuthology.orchestra.run.smithi088.stderr:1: v1:172.21.15.192:6789/0 mon.b 2024-06-03T08:36:35.529 INFO:teuthology.orchestra.run.smithi088.stderr:2: v1:172.21.15.88:6790/0 mon.c 2024-06-03T08:36:35.529 INFO:teuthology.orchestra.run.smithi088.stderr: 2024-06-03T08:36:35.529 INFO:teuthology.orchestra.run.smithi088.stderr:2024-06-03T08:36:35.524+0000 7f46abfff640 1 -- 172.21.15.88:0/46137053 <== mon.0 v1:172.21.15.88:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 764+0+0 (unknown 3393306832 0 0) 0x7f46b41d9b00 con 0x7f46b40adb90 2024-06-03T08:36:35.529 INFO:teuthology.orchestra.run.smithi088.stderr:2024-06-03T08:36:35.524+0000 7f46abfff640 1 -- 172.21.15.88:0/46137053 --> v1:172.21.15.88:6789/0 -- auth(proto 2 165 bytes epoch 0) v1 -- 0x7f46b4134c40 con 0x7f46b40adb90 2024-06-03T08:36:35.529 INFO:teuthology.orchestra.run.smithi088.stderr:2024-06-03T08:36:35.524+0000 7f46abfff640 1 -- 172.21.15.88:0/46137053 <== mon.0 v1:172.21.15.88:6789/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 2234374065 0 0) 0x7f46a40018b0 con 0x7f46b40adb90 2024-06-03T08:36:35.529 INFO:teuthology.orchestra.run.smithi088.stderr:2024-06-03T08:36:35.524+0000 7f46abfff640 10 monclient(hunting): handle_monmap mon_map magic: 0 v1 2024-06-03T08:36:35.529 INFO:teuthology.orchestra.run.smithi088.stderr:2024-06-03T08:36:35.524+0000 7f46abfff640 10 monclient(hunting): got monmap 1 from mon.a (according to old e1) 2024-06-03T08:36:35.529 INFO:teuthology.orchestra.run.smithi088.stderr:2024-06-03T08:36:35.524+0000 7f46abfff640 10 monclient(hunting): dump: 2024-06-03T08:36:35.529 INFO:teuthology.orchestra.run.smithi088.stderr:epoch 1 2024-06-03T08:36:35.530 INFO:teuthology.orchestra.run.smithi088.stderr:fsid ec3c3fdd-3ac6-4d7d-9412-683130ed8b04 2024-06-03T08:36:35.530 INFO:teuthology.orchestra.run.smithi088.stderr:last_changed 2024-06-03T08:34:31.885125+0000 2024-06-03T08:36:35.530 INFO:teuthology.orchestra.run.smithi088.stderr:created 2024-06-03T08:34:31.885125+0000 2024-06-03T08:36:35.530 INFO:teuthology.orchestra.run.smithi088.stderr:min_mon_release 18 (reef) 2024-06-03T08:36:35.530 INFO:teuthology.orchestra.run.smithi088.stderr:election_strategy: 3 2024-06-03T08:36:35.530 INFO:teuthology.orchestra.run.smithi088.stderr:0: v1:172.21.15.88:6789/0 mon.a 2024-06-03T08:36:35.530 INFO:teuthology.orchestra.run.smithi088.stderr:1: v1:172.21.15.192:6789/0 mon.b 2024-06-03T08:36:35.530 INFO:teuthology.orchestra.run.smithi088.stderr:2: v1:172.21.15.88:6790/0 mon.c 2024-06-03T08:36:35.530 INFO:teuthology.orchestra.run.smithi088.stderr: 2024-06-03T08:36:35.530 INFO:teuthology.orchestra.run.smithi088.stderr:2024-06-03T08:36:35.526+0000 7f46abfff640 1 -- 172.21.15.88:0/46137053 <== mon.1 v1:172.21.15.192:6789/0 4 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 580+0+0 (unknown 774981237 0 0) 0x7f46b41d9ee0 con 0x7f46b406ee00 2024-06-03T08:36:35.530 INFO:teuthology.orchestra.run.smithi088.stderr:2024-06-03T08:36:35.526+0000 7f46abfff640 1 -- 172.21.15.88:0/46137053 >> v1:172.21.15.88:6790/0 conn(0x7f46b41383e0 legacy=0x7f46b4133520 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-06-03T08:36:35.530 INFO:teuthology.orchestra.run.smithi088.stderr:2024-06-03T08:36:35.526+0000 7f46abfff640 1 -- 172.21.15.88:0/46137053 >> v1:172.21.15.88:6789/0 conn(0x7f46b40adb90 legacy=0x7f46b41da400 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-06-03T08:36:35.530 INFO:teuthology.orchestra.run.smithi088.stderr:2024-06-03T08:36:35.526+0000 7f46abfff640 10 monclient: _finish_hunting 0 2024-06-03T08:36:35.531 INFO:teuthology.orchestra.run.smithi088.stderr:2024-06-03T08:36:35.526+0000 7f46abfff640 1 monclient: found mon.b 2024-06-03T08:36:35.531 INFO:teuthology.orchestra.run.smithi088.stderr:2024-06-03T08:36:35.526+0000 7f46abfff640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-06-03T08:36:35.531 INFO:teuthology.orchestra.run.smithi088.stderr:2024-06-03T08:36:35.526+0000 7f46abfff640 10 monclient: _send_mon_message to mon.b at v1:172.21.15.192:6789/0 2024-06-03T08:36:35.531 INFO:teuthology.orchestra.run.smithi088.stderr:2024-06-03T08:36:35.526+0000 7f46abfff640 1 -- 172.21.15.88:0/46137053 --> v1:172.21.15.192:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f46b41d9c90 con 0x7f46b406ee00 2024-06-03T08:36:35.531 INFO:teuthology.orchestra.run.smithi088.stderr:2024-06-03T08:36:35.526+0000 7f46abfff640 10 monclient: _finish_auth 0 2024-06-03T08:36:35.531 INFO:teuthology.orchestra.run.smithi088.stderr:2024-06-03T08:36:35.526+0000 7f46abfff640 10 monclient: _check_auth_tickets 2024-06-03T08:36:35.531 INFO:teuthology.orchestra.run.smithi088.stderr:2024-06-03T08:36:35.526+0000 7f46abfff640 20 monclient: _check_auth_rotating not needed by client.admin 2024-06-03T08:36:35.531 INFO:teuthology.orchestra.run.smithi088.stderr:2024-06-03T08:36:35.526+0000 7f46ba069640 5 monclient: authenticate success, global_id 4390 2024-06-03T08:36:35.531 INFO:teuthology.orchestra.run.smithi088.stderr:2024-06-03T08:36:35.526+0000 7f46ba069640 10 monclient: _renew_subs 2024-06-03T08:36:35.531 INFO:teuthology.orchestra.run.smithi088.stderr:2024-06-03T08:36:35.526+0000 7f46ba069640 10 monclient: _send_mon_message to mon.b at v1:172.21.15.192:6789/0 2024-06-03T08:36:35.531 INFO:teuthology.orchestra.run.smithi088.stderr:2024-06-03T08:36:35.526+0000 7f46ba069640 1 -- 172.21.15.88:0/46137053 --> v1:172.21.15.192:6789/0 -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f46b4133c30 con 0x7f46b406ee00 2024-06-03T08:36:35.531 INFO:teuthology.orchestra.run.smithi088.stderr:2024-06-03T08:36:35.526+0000 7f46ba069640 10 client.4390.objecter _maybe_request_map subscribing (onetime) to next osd map 2024-06-03T08:36:35.531 INFO:teuthology.orchestra.run.smithi088.stderr:2024-06-03T08:36:35.526+0000 7f46ba069640 10 monclient: _renew_subs 2024-06-03T08:36:35.532 INFO:teuthology.orchestra.run.smithi088.stderr:2024-06-03T08:36:35.526+0000 7f46ba069640 10 monclient: _send_mon_message to mon.b at v1:172.21.15.192:6789/0 2024-06-03T08:36:35.532 INFO:teuthology.orchestra.run.smithi088.stderr:2024-06-03T08:36:35.526+0000 7f46ba069640 1 -- 172.21.15.88:0/46137053 --> v1:172.21.15.192:6789/0 -- mon_subscribe({osdmap=0}) v3 -- 0x7f46b41341a0 con 0x7f46b406ee00 2024-06-03T08:36:35.532 INFO:teuthology.orchestra.run.smithi088.stderr:2024-06-03T08:36:35.526+0000 7f46ba069640 1 librados: init done 2024-06-03T08:36:35.532 INFO:teuthology.orchestra.run.smithi088.stderr:2024-06-03T08:36:35.527+0000 7f46ba069640 10 start_mon_command cmd=[{"prefix": "get_command_descriptions"}] 2024-06-03T08:36:35.532 INFO:teuthology.orchestra.run.smithi088.stderr:2024-06-03T08:36:35.527+0000 7f46ba069640 10 monclient: _send_command 1 [{"prefix": "get_command_descriptions"}] 2024-06-03T08:36:35.532 INFO:teuthology.orchestra.run.smithi088.stderr:2024-06-03T08:36:35.527+0000 7f46ba069640 10 monclient: _send_mon_message to mon.b at v1:172.21.15.192:6789/0 2024-06-03T08:36:35.532 INFO:teuthology.orchestra.run.smithi088.stderr:2024-06-03T08:36:35.527+0000 7f46ba069640 1 -- 172.21.15.88:0/46137053 --> v1:172.21.15.192:6789/0 -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f46b404ea60 con 0x7f46b406ee00 2024-06-03T08:36:35.541 INFO:teuthology.orchestra.run.smithi088.stderr:2024-06-03T08:36:35.541+0000 7f46abfff640 1 -- 172.21.15.88:0/46137053 <== mon.1 v1:172.21.15.192:6789/0 5 ==== config(0 keys) v1 ==== 4+0+0 (unknown 0 0 0) 0x7f469c0048f0 con 0x7f46b406ee00 2024-06-03T08:36:35.541 INFO:teuthology.orchestra.run.smithi088.stderr:2024-06-03T08:36:35.541+0000 7f46abfff640 10 monclient: handle_config config(0 keys) v1 2024-06-03T08:36:35.542 INFO:teuthology.orchestra.run.smithi088.stderr:2024-06-03T08:36:35.541+0000 7f46abfff640 1 -- 172.21.15.88:0/46137053 <== mon.1 v1:172.21.15.192:6789/0 6 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 2234374065 0 0) 0x7f469c004c00 con 0x7f46b406ee00 2024-06-03T08:36:35.542 INFO:teuthology.orchestra.run.smithi088.stderr:2024-06-03T08:36:35.541+0000 7f46abfff640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-06-03T08:36:35.542 INFO:teuthology.orchestra.run.smithi088.stderr:2024-06-03T08:36:35.541+0000 7f46abfff640 10 monclient: got monmap 1 from mon.b (according to old e1) 2024-06-03T08:36:35.542 INFO:teuthology.orchestra.run.smithi088.stderr:2024-06-03T08:36:35.541+0000 7f46abfff640 10 monclient: dump: 2024-06-03T08:36:35.542 INFO:teuthology.orchestra.run.smithi088.stderr:epoch 1 2024-06-03T08:36:35.542 INFO:teuthology.orchestra.run.smithi088.stderr:fsid ec3c3fdd-3ac6-4d7d-9412-683130ed8b04 2024-06-03T08:36:35.542 INFO:teuthology.orchestra.run.smithi088.stderr:last_changed 2024-06-03T08:34:31.885125+0000 2024-06-03T08:36:35.542 INFO:teuthology.orchestra.run.smithi088.stderr:created 2024-06-03T08:34:31.885125+0000 2024-06-03T08:36:35.542 INFO:teuthology.orchestra.run.smithi088.stderr:min_mon_release 18 (reef) 2024-06-03T08:36:35.542 INFO:teuthology.orchestra.run.smithi088.stderr:election_strategy: 3 2024-06-03T08:36:35.542 INFO:teuthology.orchestra.run.smithi088.stderr:0: v1:172.21.15.88:6789/0 mon.a 2024-06-03T08:36:35.542 INFO:teuthology.orchestra.run.smithi088.stderr:1: v1:172.21.15.192:6789/0 mon.b 2024-06-03T08:36:35.543 INFO:teuthology.orchestra.run.smithi088.stderr:2: v1:172.21.15.88:6790/0 mon.c 2024-06-03T08:36:35.543 INFO:teuthology.orchestra.run.smithi088.stderr: 2024-06-03T08:36:35.543 INFO:teuthology.orchestra.run.smithi088.stderr:2024-06-03T08:36:35.541+0000 7f46b1ffb640 4 set_mon_vals no callback set 2024-06-03T08:36:35.543 INFO:teuthology.orchestra.run.smithi088.stderr:2024-06-03T08:36:35.542+0000 7f46abfff640 1 -- 172.21.15.88:0/46137053 <== mon.1 v1:172.21.15.192:6789/0 7 ==== mgrmap(e 5) v1 ==== 89110+0+0 (unknown 873533973 0 0) 0x7f469c01a980 con 0x7f46b406ee00 2024-06-03T08:36:35.543 INFO:teuthology.orchestra.run.smithi088.stderr:2024-06-03T08:36:35.543+0000 7f46b2ffd640 10 monclient: get_auth_request con 0x7f469406d420 auth_method 0 2024-06-03T08:36:35.543 INFO:teuthology.orchestra.run.smithi088.stderr:2024-06-03T08:36:35.543+0000 7f46abfff640 10 client.4390.objecter ms_handle_connect 0x7f469406d420 2024-06-03T08:36:35.544 INFO:teuthology.orchestra.run.smithi088.stderr:2024-06-03T08:36:35.544+0000 7f46abfff640 1 -- 172.21.15.88:0/46137053 <== mon.1 v1:172.21.15.192:6789/0 8 ==== osd_map(26..26 src has 1..26) v4 ==== 4090+0+0 (unknown 3637450554 0 0) 0x7f469c054230 con 0x7f46b406ee00 2024-06-03T08:36:35.544 INFO:teuthology.orchestra.run.smithi088.stderr:2024-06-03T08:36:35.544+0000 7f46abfff640 10 client.4390.objecter ms_dispatch 0x7f46b4000b90 osd_map(26..26 src has 1..26) v4 2024-06-03T08:36:35.544 INFO:teuthology.orchestra.run.smithi088.stderr:2024-06-03T08:36:35.544+0000 7f46abfff640 3 client.4390.objecter handle_osd_map got epochs [26,26] > 0 2024-06-03T08:36:35.544 INFO:teuthology.orchestra.run.smithi088.stderr:2024-06-03T08:36:35.544+0000 7f46abfff640 3 client.4390.objecter handle_osd_map decoding full epoch 26 2024-06-03T08:36:35.544 INFO:teuthology.orchestra.run.smithi088.stderr:2024-06-03T08:36:35.544+0000 7f46abfff640 20 client.4390.objecter dump_active .. 0 homeless 2024-06-03T08:36:35.773 INFO:teuthology.orchestra.run.smithi088.stderr:2024-06-03T08:36:35.772+0000 7f46abfff640 1 -- 172.21.15.88:0/46137053 <== mon.1 v1:172.21.15.192:6789/0 9 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+182498 (unknown 1092875540 0 3016760700) 0x7f469c04fc10 con 0x7f46b406ee00 2024-06-03T08:36:35.773 INFO:teuthology.orchestra.run.smithi088.stderr:2024-06-03T08:36:35.772+0000 7f46abfff640 10 monclient: handle_mon_command_ack 1 [{"prefix": "get_command_descriptions"}] 2024-06-03T08:36:35.773 INFO:teuthology.orchestra.run.smithi088.stderr:2024-06-03T08:36:35.772+0000 7f46abfff640 10 monclient: _finish_command 1 = system:0 2024-06-03T08:36:35.942 INFO:teuthology.orchestra.run.smithi088.stderr:2024-06-03T08:36:35.942+0000 7f46ba069640 1 -- 172.21.15.88:0/46137053 --> v1:172.21.15.192:6800/34229 -- mgr_command(tid 0: {"prefix": "pg dump", "target": ["mon-mgr", ""], "format": "json"}) v1 -- 0x7f46b406ff80 con 0x7f469406d420 2024-06-03T08:36:35.944 INFO:teuthology.orchestra.run.smithi088.stderr:2024-06-03T08:36:35.944+0000 7f46abfff640 1 -- 172.21.15.88:0/46137053 <== mgr.4110 v1:172.21.15.192:6800/34229 1 ==== mgr_command_reply(tid 0: 0 dumped all) v1 ==== 18+0+48018 (unknown 2965378022 0 3644972437) 0x7f46a4001c30 con 0x7f469406d420 2024-06-03T08:36:35.945 INFO:teuthology.orchestra.run.smithi088.stdout: 2024-06-03T08:36:35.945 INFO:teuthology.orchestra.run.smithi088.stderr:dumped all 2024-06-03T08:36:35.947 INFO:teuthology.orchestra.run.smithi088.stderr:2024-06-03T08:36:35.946+0000 7f46ba069640 10 librados: watch_flush enter 2024-06-03T08:36:35.947 INFO:teuthology.orchestra.run.smithi088.stderr:2024-06-03T08:36:35.946+0000 7f46ba069640 10 librados: watch_flush exit 2024-06-03T08:36:35.947 INFO:teuthology.orchestra.run.smithi088.stderr:2024-06-03T08:36:35.946+0000 7f46ba069640 20 client.4390.objecter shutdown clearing up homeless session... 2024-06-03T08:36:35.947 INFO:teuthology.orchestra.run.smithi088.stderr:2024-06-03T08:36:35.946+0000 7f46ba069640 10 client.4390.objecter successfully canceled tick 2024-06-03T08:36:35.947 INFO:teuthology.orchestra.run.smithi088.stderr:2024-06-03T08:36:35.946+0000 7f46ba069640 1 -- 172.21.15.88:0/46137053 >> v1:172.21.15.192:6800/34229 conn(0x7f469406d420 legacy=0x7f469406f8e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-06-03T08:36:35.947 INFO:teuthology.orchestra.run.smithi088.stderr:2024-06-03T08:36:35.946+0000 7f46ba069640 10 monclient: shutdown 2024-06-03T08:36:35.947 INFO:teuthology.orchestra.run.smithi088.stderr:2024-06-03T08:36:35.946+0000 7f46ba069640 20 monclient: shutdown discarding 0 pending message(s) 2024-06-03T08:36:35.947 INFO:teuthology.orchestra.run.smithi088.stderr:2024-06-03T08:36:35.946+0000 7f46ba069640 1 -- 172.21.15.88:0/46137053 >> v1:172.21.15.192:6789/0 conn(0x7f46b406ee00 legacy=0x7f46b41d6600 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-06-03T08:36:35.947 INFO:teuthology.orchestra.run.smithi088.stderr:2024-06-03T08:36:35.946+0000 7f46ba069640 1 -- 172.21.15.88:0/46137053 shutdown_connections 2024-06-03T08:36:35.947 INFO:teuthology.orchestra.run.smithi088.stderr:2024-06-03T08:36:35.946+0000 7f46ba069640 1 -- 172.21.15.88:0/46137053 >> 172.21.15.88:0/46137053 conn(0x7f46b40b2f20 msgr2=0x7f46b40b2730 unknown :-1 s=STATE_NONE l=0).mark_down 2024-06-03T08:36:35.947 INFO:teuthology.orchestra.run.smithi088.stderr:2024-06-03T08:36:35.946+0000 7f46ba069640 1 -- 172.21.15.88:0/46137053 shutdown_connections 2024-06-03T08:36:35.947 INFO:teuthology.orchestra.run.smithi088.stderr:2024-06-03T08:36:35.946+0000 7f46ba069640 1 -- 172.21.15.88:0/46137053 wait complete. 2024-06-03T08:36:35.947 INFO:teuthology.orchestra.run.smithi088.stderr:2024-06-03T08:36:35.946+0000 7f46ba069640 1 librados: shutdown 2024-06-03T08:36:35.957 INFO:teuthology.orchestra.run.smithi088.stdout:{"pg_ready":true,"pg_map":{"version":40,"stamp":"2024-06-03T08:36:34.591503+0000","last_osdmap_epoch":0,"last_pg_scan":0,"pg_stats_sum":{"stat_sum":{"num_bytes":459280,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":96,"num_read_kb":82,"num_write":113,"num_write_kb":1372,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2,"num_bytes_recovered":397840,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":76,"ondisk_log_size":76,"up":18,"acting":18,"num_store_stats":0},"osd_stats_sum":{"up_from":0,"seq":0,"num_pgs":18,"num_osds":8,"num_per_pool_osds":8,"num_per_pool_omap_osds":8,"kb":754974720,"kb_used":218692,"kb_used_data":2504,"kb_used_omap":12,"kb_used_meta":216051,"kb_avail":754756028,"statfs":{"total":773094113280,"available":772870172672,"internally_reserved":0,"allocated":2564096,"data_stored":1403524,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":12744,"internal_metadata":221236792},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":48,"apply_latency_ms":48,"commit_latency_ns":48000000,"apply_latency_ns":48000000},"alerts":[],"network_ping_times":[]},"pg_stats_delta":{"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2,"num_bytes_recovered":397840,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":0,"acting":0,"num_store_stats":0,"stamp_delta":"6.001199"},"pg_stats":[{"pgid":"2.7","version":"0'0","reported_seq":11,"reported_epoch":25,"state":"active+clean","last_fresh":"2024-06-03T08:36:31.935126+0000","last_change":"2024-06-03T08:36:30.385405+0000","last_active":"2024-06-03T08:36:31.935126+0000","last_peered":"2024-06-03T08:36:31.935126+0000","last_clean":"2024-06-03T08:36:31.935126+0000","last_became_active":"2024-06-03T08:36:30.380302+0000","last_became_peered":"2024-06-03T08:36:30.380302+0000","last_unstale":"2024-06-03T08:36:31.935126+0000","last_undegraded":"2024-06-03T08:36:31.935126+0000","last_fullsized":"2024-06-03T08:36:31.935126+0000","mapping_epoch":23,"log_start":"0'0","ondisk_log_start":"0'0","created":23,"last_epoch_clean":24,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-06-03T08:36:28.444764+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-06-03T08:36:28.444764+0000","last_clean_scrub_stamp":"2024-06-03T08:36:28.444764+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-06-04T18:46:33.475181+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0],"acting":[3,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"2.6","version":"0'0","reported_seq":11,"reported_epoch":25,"state":"active+clean","last_fresh":"2024-06-03T08:36:32.484953+0000","last_change":"2024-06-03T08:36:30.430760+0000","last_active":"2024-06-03T08:36:32.484953+0000","last_peered":"2024-06-03T08:36:32.484953+0000","last_clean":"2024-06-03T08:36:32.484953+0000","last_became_active":"2024-06-03T08:36:30.404665+0000","last_became_peered":"2024-06-03T08:36:30.404665+0000","last_unstale":"2024-06-03T08:36:32.484953+0000","last_undegraded":"2024-06-03T08:36:32.484953+0000","last_fullsized":"2024-06-03T08:36:32.484953+0000","mapping_epoch":23,"log_start":"0'0","ondisk_log_start":"0'0","created":23,"last_epoch_clean":24,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-06-03T08:36:28.444764+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-06-03T08:36:28.444764+0000","last_clean_scrub_stamp":"2024-06-03T08:36:28.444764+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-06-04T10:37:37.746074+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,0],"acting":[4,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"2.5","version":"0'0","reported_seq":11,"reported_epoch":25,"state":"active+clean","last_fresh":"2024-06-03T08:36:31.913806+0000","last_change":"2024-06-03T08:36:30.024235+0000","last_active":"2024-06-03T08:36:31.913806+0000","last_peered":"2024-06-03T08:36:31.913806+0000","last_clean":"2024-06-03T08:36:31.913806+0000","last_became_active":"2024-06-03T08:36:30.019040+0000","last_became_peered":"2024-06-03T08:36:30.019040+0000","last_unstale":"2024-06-03T08:36:31.913806+0000","last_undegraded":"2024-06-03T08:36:31.913806+0000","last_fullsized":"2024-06-03T08:36:31.913806+0000","mapping_epoch":23,"log_start":"0'0","ondisk_log_start":"0'0","created":23,"last_epoch_clean":24,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-06-03T08:36:28.444764+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-06-03T08:36:28.444764+0000","last_clean_scrub_stamp":"2024-06-03T08:36:28.444764+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-06-04T15:46:47.109502+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,4],"acting":[3,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"2.4","version":"0'0","reported_seq":11,"reported_epoch":26,"state":"active+clean","last_fresh":"2024-06-03T08:36:31.819576+0000","last_change":"2024-06-03T08:36:30.264987+0000","last_active":"2024-06-03T08:36:31.819576+0000","last_peered":"2024-06-03T08:36:31.819576+0000","last_clean":"2024-06-03T08:36:31.819576+0000","last_became_active":"2024-06-03T08:36:30.237862+0000","last_became_peered":"2024-06-03T08:36:30.237862+0000","last_unstale":"2024-06-03T08:36:31.819576+0000","last_undegraded":"2024-06-03T08:36:31.819576+0000","last_fullsized":"2024-06-03T08:36:31.819576+0000","mapping_epoch":23,"log_start":"0'0","ondisk_log_start":"0'0","created":23,"last_epoch_clean":24,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-06-03T08:36:28.444764+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-06-03T08:36:28.444764+0000","last_clean_scrub_stamp":"2024-06-03T08:36:28.444764+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-06-04T17:35:23.776572+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,6],"acting":[5,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"2.2","version":"0'0","reported_seq":11,"reported_epoch":25,"state":"active+clean","last_fresh":"2024-06-03T08:36:31.918592+0000","last_change":"2024-06-03T08:36:30.230431+0000","last_active":"2024-06-03T08:36:31.918592+0000","last_peered":"2024-06-03T08:36:31.918592+0000","last_clean":"2024-06-03T08:36:31.918592+0000","last_became_active":"2024-06-03T08:36:30.221718+0000","last_became_peered":"2024-06-03T08:36:30.221718+0000","last_unstale":"2024-06-03T08:36:31.918592+0000","last_undegraded":"2024-06-03T08:36:31.918592+0000","last_fullsized":"2024-06-03T08:36:31.918592+0000","mapping_epoch":23,"log_start":"0'0","ondisk_log_start":"0'0","created":23,"last_epoch_clean":24,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-06-03T08:36:28.444764+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-06-03T08:36:28.444764+0000","last_clean_scrub_stamp":"2024-06-03T08:36:28.444764+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-06-04T15:23:37.659061+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,6],"acting":[3,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"2.1","version":"0'0","reported_seq":11,"reported_epoch":26,"state":"active+clean","last_fresh":"2024-06-03T08:36:31.900749+0000","last_change":"2024-06-03T08:36:30.883360+0000","last_active":"2024-06-03T08:36:31.900749+0000","last_peered":"2024-06-03T08:36:31.900749+0000","last_clean":"2024-06-03T08:36:31.900749+0000","last_became_active":"2024-06-03T08:36:30.855073+0000","last_became_peered":"2024-06-03T08:36:30.855073+0000","last_unstale":"2024-06-03T08:36:31.900749+0000","last_undegraded":"2024-06-03T08:36:31.900749+0000","last_fullsized":"2024-06-03T08:36:31.900749+0000","mapping_epoch":23,"log_start":"0'0","ondisk_log_start":"0'0","created":23,"last_epoch_clean":25,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-06-03T08:36:28.444764+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-06-03T08:36:28.444764+0000","last_clean_scrub_stamp":"2024-06-03T08:36:28.444764+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-06-04T18:12:22.363473+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,5],"acting":[7,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"2.0","version":"0'0","reported_seq":11,"reported_epoch":25,"state":"active+clean","last_fresh":"2024-06-03T08:36:31.924510+0000","last_change":"2024-06-03T08:36:29.984096+0000","last_active":"2024-06-03T08:36:31.924510+0000","last_peered":"2024-06-03T08:36:31.924510+0000","last_clean":"2024-06-03T08:36:31.924510+0000","last_became_active":"2024-06-03T08:36:29.957875+0000","last_became_peered":"2024-06-03T08:36:29.957875+0000","last_unstale":"2024-06-03T08:36:31.924510+0000","last_undegraded":"2024-06-03T08:36:31.924510+0000","last_fullsized":"2024-06-03T08:36:31.924510+0000","mapping_epoch":23,"log_start":"0'0","ondisk_log_start":"0'0","created":23,"last_epoch_clean":24,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-06-03T08:36:28.444764+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-06-03T08:36:28.444764+0000","last_clean_scrub_stamp":"2024-06-03T08:36:28.444764+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-06-04T08:41:33.679479+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,5],"acting":[3,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"2.3","version":"0'0","reported_seq":11,"reported_epoch":25,"state":"active+clean","last_fresh":"2024-06-03T08:36:31.921442+0000","last_change":"2024-06-03T08:36:30.351670+0000","last_active":"2024-06-03T08:36:31.921442+0000","last_peered":"2024-06-03T08:36:31.921442+0000","last_clean":"2024-06-03T08:36:31.921442+0000","last_became_active":"2024-06-03T08:36:30.343560+0000","last_became_peered":"2024-06-03T08:36:30.343560+0000","last_unstale":"2024-06-03T08:36:31.921442+0000","last_undegraded":"2024-06-03T08:36:31.921442+0000","last_fullsized":"2024-06-03T08:36:31.921442+0000","mapping_epoch":23,"log_start":"0'0","ondisk_log_start":"0'0","created":23,"last_epoch_clean":24,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-06-03T08:36:28.444764+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-06-03T08:36:28.444764+0000","last_clean_scrub_stamp":"2024-06-03T08:36:28.444764+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-06-04T12:02:24.765135+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,2],"acting":[3,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.0","version":"19'76","reported_seq":131,"reported_epoch":25,"state":"active+clean","last_fresh":"2024-06-03T08:36:31.901743+0000","last_change":"2024-06-03T08:36:26.702702+0000","last_active":"2024-06-03T08:36:31.901743+0000","last_peered":"2024-06-03T08:36:31.901743+0000","last_clean":"2024-06-03T08:36:31.901743+0000","last_became_active":"2024-06-03T08:36:26.128378+0000","last_became_peered":"2024-06-03T08:36:26.128378+0000","last_unstale":"2024-06-03T08:36:31.901743+0000","last_undegraded":"2024-06-03T08:36:31.901743+0000","last_fullsized":"2024-06-03T08:36:31.901743+0000","mapping_epoch":20,"log_start":"0'0","ondisk_log_start":"0'0","created":16,"last_epoch_clean":22,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-06-03T08:36:15.542635+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-06-03T08:36:15.542635+0000","last_clean_scrub_stamp":"2024-06-03T08:36:15.542635+0000","objects_scrubbed":0,"log_size":76,"log_dups_size":0,"ondisk_log_size":76,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-06-04T12:41:53.731360+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":459280,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":96,"num_read_kb":82,"num_write":113,"num_write_kb":1372,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2,"num_bytes_recovered":397840,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,5],"acting":[1,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]}],"pool_stats":[{"poolid":2,"num_pg":8,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":16,"acting":16,"num_store_stats":7},{"poolid":1,"num_pg":1,"stat_sum":{"num_bytes":459280,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":96,"num_read_kb":82,"num_write":113,"num_write_kb":1372,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2,"num_bytes_recovered":397840,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":802816,"data_stored":795680,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":76,"ondisk_log_size":76,"up":2,"acting":2,"num_store_stats":2}],"osd_stats":[{"osd":6,"up_from":21,"seq":90194313226,"num_pgs":2,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":27232,"kb_used_data":208,"kb_used_omap":1,"kb_used_meta":27006,"kb_avail":94344608,"statfs":{"total":96636764160,"available":96608878592,"internally_reserved":0,"allocated":212992,"data_stored":71931,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1593,"internal_metadata":27654599},"hb_peers":[0,1,2,3,4,5,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":30,"apply_latency_ms":30,"commit_latency_ns":30000000,"apply_latency_ns":30000000},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":9.282}]},{"osd":1,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":7.718}]},{"osd":2,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":12.675000000000001}]},{"osd":3,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":6.6449999999999996}]},{"osd":4,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":15.222}]},{"osd":5,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":13.324}]},{"osd":7,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":12.085000000000001}]}]},{"osd":7,"up_from":20,"seq":85899345929,"num_pgs":1,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":27236,"kb_used_data":216,"kb_used_omap":1,"kb_used_meta":27006,"kb_avail":94344604,"statfs":{"total":96636764160,"available":96608874496,"internally_reserved":0,"allocated":221184,"data_stored":76559,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1589,"internal_metadata":27654603},"hb_peers":[0,1,2,3,4,5,6],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":11.688000000000001}]},{"osd":1,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":7.5970000000000004}]},{"osd":2,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":13.432}]},{"osd":3,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":6.6369999999999996}]},{"osd":4,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":12.353999999999999}]},{"osd":5,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":10.159000000000001}]},{"osd":6,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":14.718}]}]},{"osd":5,"up_from":20,"seq":85899345931,"num_pgs":4,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":27636,"kb_used_data":608,"kb_used_omap":1,"kb_used_meta":27006,"kb_avail":94344204,"statfs":{"total":96636764160,"available":96608464896,"internally_reserved":0,"allocated":622592,"data_stored":474399,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1594,"internal_metadata":27654598},"hb_peers":[0,1,2,3,4,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":14.609999999999999}]},{"osd":1,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":17.68}]},{"osd":2,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":6.5190000000000001}]},{"osd":3,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":16.898}]},{"osd":4,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":7.601}]},{"osd":6,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":13.211}]},{"osd":7,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":11.711}]}]},{"osd":4,"up_from":20,"seq":85899345929,"num_pgs":2,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":27240,"kb_used_data":216,"kb_used_omap":1,"kb_used_meta":27006,"kb_avail":94344600,"statfs":{"total":96636764160,"available":96608870400,"internally_reserved":0,"allocated":221184,"data_stored":76559,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1594,"internal_metadata":27654598},"hb_peers":[0,1,2,3,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":8,"apply_latency_ms":8,"commit_latency_ns":8000000,"apply_latency_ns":8000000},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":7.7750000000000004}]},{"osd":1,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":9.2070000000000007}]},{"osd":2,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":21.491}]},{"osd":3,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":9.9920000000000009}]},{"osd":5,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":22.495999999999999}]},{"osd":6,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":10.614000000000001}]},{"osd":7,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":12.737}]}]},{"osd":2,"up_from":20,"seq":85899345930,"num_pgs":1,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":27236,"kb_used_data":216,"kb_used_omap":1,"kb_used_meta":27006,"kb_avail":94344604,"statfs":{"total":96636764160,"available":96608874496,"internally_reserved":0,"allocated":221184,"data_stored":76559,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1594,"internal_metadata":27654598},"hb_peers":[0,1,3,4,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":6.2350000000000003}]},{"osd":1,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":7.7670000000000003}]},{"osd":3,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":13.222}]},{"osd":4,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":17.765999999999998}]},{"osd":5,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":10.678000000000001}]},{"osd":6,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":11.221}]},{"osd":7,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":15.510999999999999}]}]},{"osd":3,"up_from":14,"seq":60129542153,"num_pgs":5,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":27236,"kb_used_data":216,"kb_used_omap":1,"kb_used_meta":27006,"kb_avail":94344604,"statfs":{"total":96636764160,"available":96608874496,"internally_reserved":0,"allocated":221184,"data_stored":76559,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1592,"internal_metadata":27654600},"hb_peers":[0,1,2,4,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":5,"apply_latency_ms":5,"commit_latency_ns":5000000,"apply_latency_ns":5000000},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":8.1679999999999993}]},{"osd":1,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":10.41}]},{"osd":2,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":9.3439999999999994}]},{"osd":4,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":12.706}]},{"osd":5,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":17.991}]},{"osd":6,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":14.965}]},{"osd":7,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":11.981}]}]},{"osd":1,"up_from":14,"seq":60129542153,"num_pgs":1,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":27632,"kb_used_data":608,"kb_used_omap":1,"kb_used_meta":27006,"kb_avail":94344208,"statfs":{"total":96636764160,"available":96608468992,"internally_reserved":0,"allocated":622592,"data_stored":474399,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":27654597},"hb_peers":[0,2,3,4,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":5,"apply_latency_ms":5,"commit_latency_ns":5000000,"apply_latency_ns":5000000},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":18.745000000000001}]},{"osd":2,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":16.329000000000001}]},{"osd":3,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":8.673}]},{"osd":4,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":18.344000000000001}]},{"osd":5,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":22.225999999999999}]},{"osd":6,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":21.404}]},{"osd":7,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":9.5229999999999997}]}]},{"osd":0,"up_from":14,"seq":60129542151,"num_pgs":2,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":27244,"kb_used_data":216,"kb_used_omap":1,"kb_used_meta":27006,"kb_avail":94344596,"statfs":{"total":96636764160,"available":96608866304,"internally_reserved":0,"allocated":221184,"data_stored":76559,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1593,"internal_metadata":27654599},"hb_peers":[1,2,3,4,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[{"osd":1,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":8.3710000000000004}]},{"osd":2,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":7.4710000000000001}]},{"osd":3,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":43.481999999999999}]},{"osd":4,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":11.834}]},{"osd":5,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":7.0220000000000002}]},{"osd":6,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":9.5090000000000003}]},{"osd":7,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":16.553999999999998}]}]}],"pool_statfs":[{"poolid":1,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":401408,"data_stored":397840,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":5,"total":0,"available":0,"internally_reserved":0,"allocated":401408,"data_stored":397840,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":4,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":5,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":6,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0}]}} 2024-06-03T08:36:35.958 DEBUG:teuthology.orchestra.run.smithi088:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph pg dump --format=json 2024-06-03T08:36:36.056 INFO:teuthology.orchestra.run.smithi088.stderr:2024-06-03T08:36:36.056+0000 7ff22643b640 10 monclient: get_monmap_and_config 2024-06-03T08:36:36.056 INFO:teuthology.orchestra.run.smithi088.stderr:2024-06-03T08:36:36.056+0000 7ff22643b640 10 monclient: build_initial_monmap 2024-06-03T08:36:36.056 INFO:teuthology.orchestra.run.smithi088.stderr:2024-06-03T08:36:36.056+0000 7ff22643b640 10 monclient: monmap: 2024-06-03T08:36:36.056 INFO:teuthology.orchestra.run.smithi088.stderr:epoch 0 2024-06-03T08:36:36.057 INFO:teuthology.orchestra.run.smithi088.stderr:fsid ec3c3fdd-3ac6-4d7d-9412-683130ed8b04 2024-06-03T08:36:36.057 INFO:teuthology.orchestra.run.smithi088.stderr:last_changed 2024-06-03T08:36:36.057201+0000 2024-06-03T08:36:36.057 INFO:teuthology.orchestra.run.smithi088.stderr:created 2024-06-03T08:36:36.057201+0000 2024-06-03T08:36:36.057 INFO:teuthology.orchestra.run.smithi088.stderr:min_mon_release 0 (unknown) 2024-06-03T08:36:36.057 INFO:teuthology.orchestra.run.smithi088.stderr:election_strategy: 3 2024-06-03T08:36:36.057 INFO:teuthology.orchestra.run.smithi088.stderr:0: v1:172.21.15.88:6789/0 mon.noname-a 2024-06-03T08:36:36.057 INFO:teuthology.orchestra.run.smithi088.stderr:1: v1:172.21.15.192:6789/0 mon.noname-c 2024-06-03T08:36:36.057 INFO:teuthology.orchestra.run.smithi088.stderr:2: v1:172.21.15.88:6790/0 mon.noname-b 2024-06-03T08:36:36.057 INFO:teuthology.orchestra.run.smithi088.stderr: 2024-06-03T08:36:36.058 INFO:teuthology.orchestra.run.smithi088.stderr:2024-06-03T08:36:36.056+0000 7ff22643b640 1 Processor -- start 2024-06-03T08:36:36.059 INFO:teuthology.orchestra.run.smithi088.stderr:2024-06-03T08:36:36.057+0000 7ff22643b640 1 -- start start 2024-06-03T08:36:36.059 INFO:teuthology.orchestra.run.smithi088.stderr:2024-06-03T08:36:36.057+0000 7ff22643b640 10 monclient: init 2024-06-03T08:36:36.059 INFO:teuthology.orchestra.run.smithi088.stderr:2024-06-03T08:36:36.057+0000 7ff22643b640 10 monclient: _reopen_session rank -1 2024-06-03T08:36:36.059 INFO:teuthology.orchestra.run.smithi088.stderr:2024-06-03T08:36:36.057+0000 7ff22643b640 10 monclient: _add_conns ranks=[2,0,1] 2024-06-03T08:36:36.059 INFO:teuthology.orchestra.run.smithi088.stderr:2024-06-03T08:36:36.057+0000 7ff22643b640 10 monclient(hunting): picked mon.noname-b con 0x7ff2200adb90 addr v1:172.21.15.88:6790/0 2024-06-03T08:36:36.059 INFO:teuthology.orchestra.run.smithi088.stderr:2024-06-03T08:36:36.057+0000 7ff22643b640 10 monclient(hunting): picked mon.noname-a con 0x7ff22006ee00 addr v1:172.21.15.88:6789/0 2024-06-03T08:36:36.059 INFO:teuthology.orchestra.run.smithi088.stderr:2024-06-03T08:36:36.057+0000 7ff22643b640 10 monclient(hunting): picked mon.noname-c con 0x7ff2201383e0 addr v1:172.21.15.192:6789/0 2024-06-03T08:36:36.059 INFO:teuthology.orchestra.run.smithi088.stderr:2024-06-03T08:36:36.057+0000 7ff22643b640 1 -- --> v1:172.21.15.88:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7ff22006f2c0 con 0x7ff22006ee00 2024-06-03T08:36:36.059 INFO:teuthology.orchestra.run.smithi088.stderr:2024-06-03T08:36:36.057+0000 7ff22643b640 1 -- --> v1:172.21.15.192:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7ff2200710c0 con 0x7ff2201383e0 2024-06-03T08:36:36.059 INFO:teuthology.orchestra.run.smithi088.stderr:2024-06-03T08:36:36.057+0000 7ff22643b640 1 -- --> v1:172.21.15.88:6790/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7ff220071fd0 con 0x7ff2200adb90 2024-06-03T08:36:36.059 INFO:teuthology.orchestra.run.smithi088.stderr:2024-06-03T08:36:36.057+0000 7ff22643b640 10 monclient(hunting): _renew_subs 2024-06-03T08:36:36.059 INFO:teuthology.orchestra.run.smithi088.stderr:2024-06-03T08:36:36.057+0000 7ff22643b640 10 monclient(hunting): authenticate will time out at 1251.867641s 2024-06-03T08:36:36.059 INFO:teuthology.orchestra.run.smithi088.stderr:2024-06-03T08:36:36.058+0000 7ff21ffff640 1 --1- >> v1:172.21.15.88:6790/0 conn(0x7ff2200adb90 0x7ff2200718c0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.88:6790/0 says I am v1:172.21.15.88:54516/0 (socket says 172.21.15.88:54516) 2024-06-03T08:36:36.060 INFO:teuthology.orchestra.run.smithi088.stderr:2024-06-03T08:36:36.058+0000 7ff21ffff640 1 -- 172.21.15.88:0/2650410273 learned_addr learned my addr 172.21.15.88:0/2650410273 (peer_addr_for_me v1:172.21.15.88:0/0) 2024-06-03T08:36:36.061 INFO:teuthology.orchestra.run.smithi088.stderr:2024-06-03T08:36:36.060+0000 7ff21effd640 1 -- 172.21.15.88:0/2650410273 <== mon.0 v1:172.21.15.88:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 3861565556 0 0) 0x7ff22006f2c0 con 0x7ff22006ee00 2024-06-03T08:36:36.061 INFO:teuthology.orchestra.run.smithi088.stderr:2024-06-03T08:36:36.060+0000 7ff21effd640 10 monclient(hunting): _negotiate 2024-06-03T08:36:36.061 INFO:teuthology.orchestra.run.smithi088.stderr:2024-06-03T08:36:36.060+0000 7ff21effd640 10 monclient(hunting): _init_auth method 2 2024-06-03T08:36:36.061 INFO:teuthology.orchestra.run.smithi088.stderr:2024-06-03T08:36:36.060+0000 7ff21effd640 10 monclient(hunting): _init_auth creating new auth 2024-06-03T08:36:36.061 INFO:teuthology.orchestra.run.smithi088.stderr:2024-06-03T08:36:36.060+0000 7ff21effd640 10 monclient(hunting): my global_id is 4386 2024-06-03T08:36:36.061 INFO:teuthology.orchestra.run.smithi088.stderr:2024-06-03T08:36:36.060+0000 7ff21effd640 1 -- 172.21.15.88:0/2650410273 --> v1:172.21.15.88:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7ff208005730 con 0x7ff22006ee00 2024-06-03T08:36:36.061 INFO:teuthology.orchestra.run.smithi088.stderr:2024-06-03T08:36:36.060+0000 7ff21effd640 1 -- 172.21.15.88:0/2650410273 <== mon.1 v1:172.21.15.192:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 2575298790 0 0) 0x7ff2200710c0 con 0x7ff2201383e0 2024-06-03T08:36:36.061 INFO:teuthology.orchestra.run.smithi088.stderr:2024-06-03T08:36:36.060+0000 7ff21effd640 10 monclient(hunting): _negotiate 2024-06-03T08:36:36.061 INFO:teuthology.orchestra.run.smithi088.stderr:2024-06-03T08:36:36.060+0000 7ff21effd640 10 monclient(hunting): _init_auth method 2 2024-06-03T08:36:36.061 INFO:teuthology.orchestra.run.smithi088.stderr:2024-06-03T08:36:36.060+0000 7ff21effd640 10 monclient(hunting): _init_auth creating new auth 2024-06-03T08:36:36.061 INFO:teuthology.orchestra.run.smithi088.stderr:2024-06-03T08:36:36.060+0000 7ff21effd640 10 monclient(hunting): my global_id is 4393 2024-06-03T08:36:36.062 INFO:teuthology.orchestra.run.smithi088.stderr:2024-06-03T08:36:36.060+0000 7ff21effd640 1 -- 172.21.15.88:0/2650410273 --> v1:172.21.15.192:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7ff22006f2c0 con 0x7ff2201383e0 2024-06-03T08:36:36.062 INFO:teuthology.orchestra.run.smithi088.stderr:2024-06-03T08:36:36.060+0000 7ff21effd640 1 -- 172.21.15.88:0/2650410273 <== mon.2 v1:172.21.15.88:6790/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 3834543231 0 0) 0x7ff220071fd0 con 0x7ff2200adb90 2024-06-03T08:36:36.062 INFO:teuthology.orchestra.run.smithi088.stderr:2024-06-03T08:36:36.060+0000 7ff21effd640 10 monclient(hunting): _negotiate 2024-06-03T08:36:36.062 INFO:teuthology.orchestra.run.smithi088.stderr:2024-06-03T08:36:36.060+0000 7ff21effd640 10 monclient(hunting): _init_auth method 2 2024-06-03T08:36:36.062 INFO:teuthology.orchestra.run.smithi088.stderr:2024-06-03T08:36:36.060+0000 7ff21effd640 10 monclient(hunting): _init_auth creating new auth 2024-06-03T08:36:36.062 INFO:teuthology.orchestra.run.smithi088.stderr:2024-06-03T08:36:36.060+0000 7ff21effd640 10 monclient(hunting): my global_id is 4397 2024-06-03T08:36:36.062 INFO:teuthology.orchestra.run.smithi088.stderr:2024-06-03T08:36:36.060+0000 7ff21effd640 1 -- 172.21.15.88:0/2650410273 --> v1:172.21.15.88:6790/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7ff2200710c0 con 0x7ff2200adb90 2024-06-03T08:36:36.064 INFO:teuthology.orchestra.run.smithi088.stderr:2024-06-03T08:36:36.063+0000 7ff21effd640 1 -- 172.21.15.88:0/2650410273 <== mon.2 v1:172.21.15.88:6790/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 214+0+0 (unknown 2433573506 0 0) 0x7ff2200710c0 con 0x7ff2200adb90 2024-06-03T08:36:36.064 INFO:teuthology.orchestra.run.smithi088.stderr:2024-06-03T08:36:36.063+0000 7ff21effd640 1 -- 172.21.15.88:0/2650410273 >> v1:172.21.15.192:6789/0 conn(0x7ff2201383e0 legacy=0x7ff220149af0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-06-03T08:36:36.064 INFO:teuthology.orchestra.run.smithi088.stderr:2024-06-03T08:36:36.063+0000 7ff21effd640 1 -- 172.21.15.88:0/2650410273 >> v1:172.21.15.88:6789/0 conn(0x7ff22006ee00 legacy=0x7ff220137cd0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-06-03T08:36:36.064 INFO:teuthology.orchestra.run.smithi088.stderr:2024-06-03T08:36:36.063+0000 7ff21effd640 10 monclient: _finish_hunting 0 2024-06-03T08:36:36.064 INFO:teuthology.orchestra.run.smithi088.stderr:2024-06-03T08:36:36.063+0000 7ff21effd640 1 monclient: found mon.noname-b 2024-06-03T08:36:36.064 INFO:teuthology.orchestra.run.smithi088.stderr:2024-06-03T08:36:36.063+0000 7ff21effd640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-06-03T08:36:36.064 INFO:teuthology.orchestra.run.smithi088.stderr:2024-06-03T08:36:36.063+0000 7ff21effd640 10 monclient: _send_mon_message to mon.noname-b at v1:172.21.15.88:6790/0 2024-06-03T08:36:36.064 INFO:teuthology.orchestra.run.smithi088.stderr:2024-06-03T08:36:36.063+0000 7ff21effd640 1 -- 172.21.15.88:0/2650410273 --> v1:172.21.15.88:6790/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7ff220141750 con 0x7ff2200adb90 2024-06-03T08:36:36.065 INFO:teuthology.orchestra.run.smithi088.stderr:2024-06-03T08:36:36.063+0000 7ff21effd640 10 monclient: _finish_auth 0 2024-06-03T08:36:36.065 INFO:teuthology.orchestra.run.smithi088.stderr:2024-06-03T08:36:36.063+0000 7ff21effd640 10 monclient: _check_auth_tickets 2024-06-03T08:36:36.065 INFO:teuthology.orchestra.run.smithi088.stderr:2024-06-03T08:36:36.063+0000 7ff21effd640 20 monclient: _check_auth_rotating not needed by client.admin 2024-06-03T08:36:36.065 INFO:teuthology.orchestra.run.smithi088.stderr:2024-06-03T08:36:36.063+0000 7ff21effd640 1 -- 172.21.15.88:0/2650410273 <== mon.2 v1:172.21.15.88:6790/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 2234374065 0 0) 0x7ff210001530 con 0x7ff2200adb90 2024-06-03T08:36:36.065 INFO:teuthology.orchestra.run.smithi088.stderr:2024-06-03T08:36:36.063+0000 7ff21effd640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-06-03T08:36:36.065 INFO:teuthology.orchestra.run.smithi088.stderr:2024-06-03T08:36:36.063+0000 7ff21effd640 10 monclient: got monmap 1 from mon.noname-b (according to old e1) 2024-06-03T08:36:36.065 INFO:teuthology.orchestra.run.smithi088.stderr:2024-06-03T08:36:36.063+0000 7ff21effd640 10 monclient: dump: 2024-06-03T08:36:36.065 INFO:teuthology.orchestra.run.smithi088.stderr:epoch 1 2024-06-03T08:36:36.065 INFO:teuthology.orchestra.run.smithi088.stderr:fsid ec3c3fdd-3ac6-4d7d-9412-683130ed8b04 2024-06-03T08:36:36.065 INFO:teuthology.orchestra.run.smithi088.stderr:last_changed 2024-06-03T08:34:31.885125+0000 2024-06-03T08:36:36.065 INFO:teuthology.orchestra.run.smithi088.stderr:created 2024-06-03T08:34:31.885125+0000 2024-06-03T08:36:36.065 INFO:teuthology.orchestra.run.smithi088.stderr:min_mon_release 18 (reef) 2024-06-03T08:36:36.065 INFO:teuthology.orchestra.run.smithi088.stderr:election_strategy: 3 2024-06-03T08:36:36.066 INFO:teuthology.orchestra.run.smithi088.stderr:0: v1:172.21.15.88:6789/0 mon.a 2024-06-03T08:36:36.066 INFO:teuthology.orchestra.run.smithi088.stderr:1: v1:172.21.15.192:6789/0 mon.b 2024-06-03T08:36:36.066 INFO:teuthology.orchestra.run.smithi088.stderr:2: v1:172.21.15.88:6790/0 mon.c 2024-06-03T08:36:36.066 INFO:teuthology.orchestra.run.smithi088.stderr: 2024-06-03T08:36:36.066 INFO:teuthology.orchestra.run.smithi088.stderr:2024-06-03T08:36:36.063+0000 7ff22643b640 5 monclient: authenticate success, global_id 4397 2024-06-03T08:36:36.066 INFO:teuthology.orchestra.run.smithi088.stderr:2024-06-03T08:36:36.063+0000 7ff22643b640 20 monclient: get_monmap_and_config waiting for monmap|config 2024-06-03T08:36:36.067 INFO:teuthology.orchestra.run.smithi088.stderr:2024-06-03T08:36:36.065+0000 7ff21effd640 1 -- 172.21.15.88:0/2650410273 <== mon.2 v1:172.21.15.88:6790/0 4 ==== config(0 keys) v1 ==== 4+0+0 (unknown 0 0 0) 0x7ff210001750 con 0x7ff2200adb90 2024-06-03T08:36:36.068 INFO:teuthology.orchestra.run.smithi088.stderr:2024-06-03T08:36:36.065+0000 7ff21effd640 10 monclient: handle_config config(0 keys) v1 2024-06-03T08:36:36.068 INFO:teuthology.orchestra.run.smithi088.stderr:2024-06-03T08:36:36.065+0000 7ff21effd640 1 -- 172.21.15.88:0/2650410273 <== mon.2 v1:172.21.15.88:6790/0 5 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 2234374065 0 0) 0x7ff210001ae0 con 0x7ff2200adb90 2024-06-03T08:36:36.068 INFO:teuthology.orchestra.run.smithi088.stderr:2024-06-03T08:36:36.065+0000 7ff22643b640 10 monclient: get_monmap_and_config success 2024-06-03T08:36:36.068 INFO:teuthology.orchestra.run.smithi088.stderr:2024-06-03T08:36:36.065+0000 7ff22643b640 4 set_mon_vals no callback set 2024-06-03T08:36:36.068 INFO:teuthology.orchestra.run.smithi088.stderr:2024-06-03T08:36:36.065+0000 7ff21effd640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-06-03T08:36:36.068 INFO:teuthology.orchestra.run.smithi088.stderr:2024-06-03T08:36:36.065+0000 7ff21effd640 10 monclient: got monmap 1 from mon.c (according to old e1) 2024-06-03T08:36:36.068 INFO:teuthology.orchestra.run.smithi088.stderr:2024-06-03T08:36:36.065+0000 7ff21effd640 10 monclient: dump: 2024-06-03T08:36:36.068 INFO:teuthology.orchestra.run.smithi088.stderr:epoch 1 2024-06-03T08:36:36.068 INFO:teuthology.orchestra.run.smithi088.stderr:fsid ec3c3fdd-3ac6-4d7d-9412-683130ed8b04 2024-06-03T08:36:36.068 INFO:teuthology.orchestra.run.smithi088.stderr:last_changed 2024-06-03T08:34:31.885125+0000 2024-06-03T08:36:36.068 INFO:teuthology.orchestra.run.smithi088.stderr:created 2024-06-03T08:34:31.885125+0000 2024-06-03T08:36:36.068 INFO:teuthology.orchestra.run.smithi088.stderr:min_mon_release 18 (reef) 2024-06-03T08:36:36.068 INFO:teuthology.orchestra.run.smithi088.stderr:election_strategy: 3 2024-06-03T08:36:36.069 INFO:teuthology.orchestra.run.smithi088.stderr:0: v1:172.21.15.88:6789/0 mon.a 2024-06-03T08:36:36.069 INFO:teuthology.orchestra.run.smithi088.stderr:1: v1:172.21.15.192:6789/0 mon.b 2024-06-03T08:36:36.069 INFO:teuthology.orchestra.run.smithi088.stderr:2: v1:172.21.15.88:6790/0 mon.c 2024-06-03T08:36:36.069 INFO:teuthology.orchestra.run.smithi088.stderr: 2024-06-03T08:36:36.069 INFO:teuthology.orchestra.run.smithi088.stderr:2024-06-03T08:36:36.066+0000 7ff22643b640 10 monclient: shutdown 2024-06-03T08:36:36.069 INFO:teuthology.orchestra.run.smithi088.stderr:2024-06-03T08:36:36.066+0000 7ff22643b640 20 monclient: shutdown discarding 0 pending message(s) 2024-06-03T08:36:36.069 INFO:teuthology.orchestra.run.smithi088.stderr:2024-06-03T08:36:36.066+0000 7ff22643b640 1 -- 172.21.15.88:0/2650410273 >> v1:172.21.15.88:6790/0 conn(0x7ff2200adb90 legacy=0x7ff2200718c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-06-03T08:36:36.069 INFO:teuthology.orchestra.run.smithi088.stderr:2024-06-03T08:36:36.066+0000 7ff22643b640 1 -- 172.21.15.88:0/2650410273 shutdown_connections 2024-06-03T08:36:36.069 INFO:teuthology.orchestra.run.smithi088.stderr:2024-06-03T08:36:36.066+0000 7ff22643b640 1 -- 172.21.15.88:0/2650410273 >> 172.21.15.88:0/2650410273 conn(0x7ff2200b2f20 msgr2=0x7ff2200b3300 unknown :-1 s=STATE_NONE l=0).mark_down 2024-06-03T08:36:36.069 INFO:teuthology.orchestra.run.smithi088.stderr:2024-06-03T08:36:36.066+0000 7ff22643b640 1 -- 172.21.15.88:0/2650410273 shutdown_connections 2024-06-03T08:36:36.069 INFO:teuthology.orchestra.run.smithi088.stderr:2024-06-03T08:36:36.066+0000 7ff22643b640 1 -- 172.21.15.88:0/2650410273 wait complete. 2024-06-03T08:36:36.069 INFO:teuthology.orchestra.run.smithi088.stderr:2024-06-03T08:36:36.067+0000 7ff22643b640 10 monclient: build_initial_monmap 2024-06-03T08:36:36.069 INFO:teuthology.orchestra.run.smithi088.stderr:2024-06-03T08:36:36.067+0000 7ff22643b640 10 monclient: monmap: 2024-06-03T08:36:36.069 INFO:teuthology.orchestra.run.smithi088.stderr:epoch 0 2024-06-03T08:36:36.070 INFO:teuthology.orchestra.run.smithi088.stderr:fsid 00000000-0000-0000-0000-000000000000 2024-06-03T08:36:36.070 INFO:teuthology.orchestra.run.smithi088.stderr:last_changed 0.000000 2024-06-03T08:36:36.070 INFO:teuthology.orchestra.run.smithi088.stderr:created 0.000000 2024-06-03T08:36:36.070 INFO:teuthology.orchestra.run.smithi088.stderr:min_mon_release 0 (unknown) 2024-06-03T08:36:36.070 INFO:teuthology.orchestra.run.smithi088.stderr:election_strategy: 1 2024-06-03T08:36:36.070 INFO:teuthology.orchestra.run.smithi088.stderr:0: v1:172.21.15.88:6789/0 mon.noname-a 2024-06-03T08:36:36.070 INFO:teuthology.orchestra.run.smithi088.stderr:1: v1:172.21.15.192:6789/0 mon.noname-b 2024-06-03T08:36:36.070 INFO:teuthology.orchestra.run.smithi088.stderr:2: v1:172.21.15.88:6790/0 mon.noname-c 2024-06-03T08:36:36.070 INFO:teuthology.orchestra.run.smithi088.stderr: 2024-06-03T08:36:36.070 INFO:teuthology.orchestra.run.smithi088.stderr:2024-06-03T08:36:36.067+0000 7ff22643b640 1 librados: starting msgr at 2024-06-03T08:36:36.070 INFO:teuthology.orchestra.run.smithi088.stderr:2024-06-03T08:36:36.067+0000 7ff22643b640 1 librados: starting objecter 2024-06-03T08:36:36.070 INFO:teuthology.orchestra.run.smithi088.stderr:2024-06-03T08:36:36.067+0000 7ff22643b640 1 Processor -- start 2024-06-03T08:36:36.070 INFO:teuthology.orchestra.run.smithi088.stderr:2024-06-03T08:36:36.067+0000 7ff22643b640 1 -- start start 2024-06-03T08:36:36.071 INFO:teuthology.orchestra.run.smithi088.stderr:2024-06-03T08:36:36.067+0000 7ff22643b640 1 librados: setting wanted keys 2024-06-03T08:36:36.071 INFO:teuthology.orchestra.run.smithi088.stderr:2024-06-03T08:36:36.067+0000 7ff22643b640 1 librados: calling monclient init 2024-06-03T08:36:36.071 INFO:teuthology.orchestra.run.smithi088.stderr:2024-06-03T08:36:36.067+0000 7ff22643b640 10 monclient: init 2024-06-03T08:36:36.073 INFO:teuthology.orchestra.run.smithi088.stderr:2024-06-03T08:36:36.068+0000 7ff22643b640 10 monclient: _reopen_session rank -1 2024-06-03T08:36:36.073 INFO:teuthology.orchestra.run.smithi088.stderr:2024-06-03T08:36:36.068+0000 7ff22643b640 10 monclient: _add_conns ranks=[2,0,1] 2024-06-03T08:36:36.073 INFO:teuthology.orchestra.run.smithi088.stderr:2024-06-03T08:36:36.068+0000 7ff22643b640 10 monclient(hunting): picked mon.noname-c con 0x7ff22006ee00 addr v1:172.21.15.88:6790/0 2024-06-03T08:36:36.073 INFO:teuthology.orchestra.run.smithi088.stderr:2024-06-03T08:36:36.068+0000 7ff22643b640 10 monclient(hunting): picked mon.noname-a con 0x7ff2200adb90 addr v1:172.21.15.88:6789/0 2024-06-03T08:36:36.073 INFO:teuthology.orchestra.run.smithi088.stderr:2024-06-03T08:36:36.068+0000 7ff22643b640 10 monclient(hunting): picked mon.noname-b con 0x7ff2201383e0 addr v1:172.21.15.192:6789/0 2024-06-03T08:36:36.073 INFO:teuthology.orchestra.run.smithi088.stderr:2024-06-03T08:36:36.068+0000 7ff22643b640 1 -- --> v1:172.21.15.88:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7ff22013ba00 con 0x7ff2200adb90 2024-06-03T08:36:36.073 INFO:teuthology.orchestra.run.smithi088.stderr:2024-06-03T08:36:36.068+0000 7ff22643b640 1 -- --> v1:172.21.15.192:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7ff220133ca0 con 0x7ff2201383e0 2024-06-03T08:36:36.073 INFO:teuthology.orchestra.run.smithi088.stderr:2024-06-03T08:36:36.068+0000 7ff22643b640 1 -- --> v1:172.21.15.88:6790/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7ff220138820 con 0x7ff22006ee00 2024-06-03T08:36:36.073 INFO:teuthology.orchestra.run.smithi088.stderr:2024-06-03T08:36:36.068+0000 7ff22643b640 10 monclient(hunting): _renew_subs 2024-06-03T08:36:36.074 INFO:teuthology.orchestra.run.smithi088.stderr:2024-06-03T08:36:36.068+0000 7ff22643b640 10 monclient(hunting): authenticate will time out at 1251.878114s 2024-06-03T08:36:36.074 INFO:teuthology.orchestra.run.smithi088.stderr:2024-06-03T08:36:36.068+0000 7ff21ffff640 1 --1- >> v1:172.21.15.88:6790/0 conn(0x7ff22006ee00 0x7ff220138e00 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=1).handle_server_banner_and_identify peer v1:172.21.15.88:6790/0 says I am v1:172.21.15.88:54528/0 (socket says 172.21.15.88:54528) 2024-06-03T08:36:36.074 INFO:teuthology.orchestra.run.smithi088.stderr:2024-06-03T08:36:36.068+0000 7ff21ffff640 1 -- 172.21.15.88:0/551408712 learned_addr learned my addr 172.21.15.88:0/551408712 (peer_addr_for_me v1:172.21.15.88:0/0) 2024-06-03T08:36:36.074 INFO:teuthology.orchestra.run.smithi088.stderr:2024-06-03T08:36:36.068+0000 7ff21cff9640 10 client.?.objecter ms_handle_connect 0x7ff22006ee00 2024-06-03T08:36:36.074 INFO:teuthology.orchestra.run.smithi088.stderr:2024-06-03T08:36:36.068+0000 7ff21cff9640 10 client.?.objecter resend_mon_ops 2024-06-03T08:36:36.074 INFO:teuthology.orchestra.run.smithi088.stderr:2024-06-03T08:36:36.069+0000 7ff21cff9640 10 client.?.objecter ms_handle_connect 0x7ff2200adb90 2024-06-03T08:36:36.074 INFO:teuthology.orchestra.run.smithi088.stderr:2024-06-03T08:36:36.069+0000 7ff21cff9640 10 client.?.objecter resend_mon_ops 2024-06-03T08:36:36.074 INFO:teuthology.orchestra.run.smithi088.stderr:2024-06-03T08:36:36.069+0000 7ff21cff9640 10 client.?.objecter ms_handle_connect 0x7ff2201383e0 2024-06-03T08:36:36.074 INFO:teuthology.orchestra.run.smithi088.stderr:2024-06-03T08:36:36.069+0000 7ff21cff9640 10 client.?.objecter resend_mon_ops 2024-06-03T08:36:36.074 INFO:teuthology.orchestra.run.smithi088.stderr:2024-06-03T08:36:36.070+0000 7ff21cff9640 1 -- 172.21.15.88:0/551408712 <== mon.2 v1:172.21.15.88:6790/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 3130165951 0 0) 0x7ff220138820 con 0x7ff22006ee00 2024-06-03T08:36:36.074 INFO:teuthology.orchestra.run.smithi088.stderr:2024-06-03T08:36:36.070+0000 7ff21cff9640 10 monclient(hunting): _negotiate 2024-06-03T08:36:36.074 INFO:teuthology.orchestra.run.smithi088.stderr:2024-06-03T08:36:36.070+0000 7ff21cff9640 10 monclient(hunting): _init_auth method 2 2024-06-03T08:36:36.074 INFO:teuthology.orchestra.run.smithi088.stderr:2024-06-03T08:36:36.070+0000 7ff21cff9640 10 monclient(hunting): _init_auth creating new auth 2024-06-03T08:36:36.075 INFO:teuthology.orchestra.run.smithi088.stderr:2024-06-03T08:36:36.070+0000 7ff21cff9640 10 monclient(hunting): my global_id is 4400 2024-06-03T08:36:36.075 INFO:teuthology.orchestra.run.smithi088.stderr:2024-06-03T08:36:36.070+0000 7ff21cff9640 1 -- 172.21.15.88:0/551408712 --> v1:172.21.15.88:6790/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7ff20c004eb0 con 0x7ff22006ee00 2024-06-03T08:36:36.075 INFO:teuthology.orchestra.run.smithi088.stderr:2024-06-03T08:36:36.071+0000 7ff21cff9640 1 -- 172.21.15.88:0/551408712 <== mon.1 v1:172.21.15.192:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 126017888 0 0) 0x7ff220133ca0 con 0x7ff2201383e0 2024-06-03T08:36:36.075 INFO:teuthology.orchestra.run.smithi088.stderr:2024-06-03T08:36:36.071+0000 7ff21cff9640 10 monclient(hunting): _negotiate 2024-06-03T08:36:36.075 INFO:teuthology.orchestra.run.smithi088.stderr:2024-06-03T08:36:36.071+0000 7ff21cff9640 10 monclient(hunting): _init_auth method 2 2024-06-03T08:36:36.075 INFO:teuthology.orchestra.run.smithi088.stderr:2024-06-03T08:36:36.071+0000 7ff21cff9640 10 monclient(hunting): _init_auth creating new auth 2024-06-03T08:36:36.075 INFO:teuthology.orchestra.run.smithi088.stderr:2024-06-03T08:36:36.071+0000 7ff21cff9640 10 monclient(hunting): my global_id is 4396 2024-06-03T08:36:36.075 INFO:teuthology.orchestra.run.smithi088.stderr:2024-06-03T08:36:36.071+0000 7ff21cff9640 1 -- 172.21.15.88:0/551408712 --> v1:172.21.15.192:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7ff220138820 con 0x7ff2201383e0 2024-06-03T08:36:36.075 INFO:teuthology.orchestra.run.smithi088.stderr:2024-06-03T08:36:36.071+0000 7ff21cff9640 1 -- 172.21.15.88:0/551408712 <== mon.0 v1:172.21.15.88:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 334667858 0 0) 0x7ff22013ba00 con 0x7ff2200adb90 2024-06-03T08:36:36.075 INFO:teuthology.orchestra.run.smithi088.stderr:2024-06-03T08:36:36.071+0000 7ff21cff9640 10 monclient(hunting): _negotiate 2024-06-03T08:36:36.075 INFO:teuthology.orchestra.run.smithi088.stderr:2024-06-03T08:36:36.071+0000 7ff21cff9640 10 monclient(hunting): _init_auth method 2 2024-06-03T08:36:36.075 INFO:teuthology.orchestra.run.smithi088.stderr:2024-06-03T08:36:36.071+0000 7ff21cff9640 10 monclient(hunting): _init_auth creating new auth 2024-06-03T08:36:36.075 INFO:teuthology.orchestra.run.smithi088.stderr:2024-06-03T08:36:36.071+0000 7ff21cff9640 10 monclient(hunting): my global_id is 4389 2024-06-03T08:36:36.076 INFO:teuthology.orchestra.run.smithi088.stderr:2024-06-03T08:36:36.071+0000 7ff21cff9640 1 -- 172.21.15.88:0/551408712 --> v1:172.21.15.88:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7ff220133ca0 con 0x7ff2200adb90 2024-06-03T08:36:36.077 INFO:teuthology.orchestra.run.smithi088.stderr:2024-06-03T08:36:36.075+0000 7ff21cff9640 1 -- 172.21.15.88:0/551408712 <== mon.1 v1:172.21.15.192:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 764+0+0 (unknown 1071746197 0 0) 0x7ff220138820 con 0x7ff2201383e0 2024-06-03T08:36:36.077 INFO:teuthology.orchestra.run.smithi088.stderr:2024-06-03T08:36:36.075+0000 7ff21cff9640 1 -- 172.21.15.88:0/551408712 --> v1:172.21.15.192:6789/0 -- auth(proto 2 165 bytes epoch 0) v1 -- 0x7ff22013ba00 con 0x7ff2201383e0 2024-06-03T08:36:36.077 INFO:teuthology.orchestra.run.smithi088.stderr:2024-06-03T08:36:36.075+0000 7ff21cff9640 1 -- 172.21.15.88:0/551408712 <== mon.1 v1:172.21.15.192:6789/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 2234374065 0 0) 0x7ff218001b50 con 0x7ff2201383e0 2024-06-03T08:36:36.077 INFO:teuthology.orchestra.run.smithi088.stderr:2024-06-03T08:36:36.075+0000 7ff21cff9640 10 monclient(hunting): handle_monmap mon_map magic: 0 v1 2024-06-03T08:36:36.077 INFO:teuthology.orchestra.run.smithi088.stderr:2024-06-03T08:36:36.075+0000 7ff21cff9640 10 monclient(hunting): got monmap 1 from mon.noname-b (according to old e1) 2024-06-03T08:36:36.077 INFO:teuthology.orchestra.run.smithi088.stderr:2024-06-03T08:36:36.075+0000 7ff21cff9640 10 monclient(hunting): dump: 2024-06-03T08:36:36.077 INFO:teuthology.orchestra.run.smithi088.stderr:epoch 1 2024-06-03T08:36:36.077 INFO:teuthology.orchestra.run.smithi088.stderr:fsid ec3c3fdd-3ac6-4d7d-9412-683130ed8b04 2024-06-03T08:36:36.077 INFO:teuthology.orchestra.run.smithi088.stderr:last_changed 2024-06-03T08:34:31.885125+0000 2024-06-03T08:36:36.078 INFO:teuthology.orchestra.run.smithi088.stderr:created 2024-06-03T08:34:31.885125+0000 2024-06-03T08:36:36.078 INFO:teuthology.orchestra.run.smithi088.stderr:min_mon_release 18 (reef) 2024-06-03T08:36:36.078 INFO:teuthology.orchestra.run.smithi088.stderr:election_strategy: 3 2024-06-03T08:36:36.078 INFO:teuthology.orchestra.run.smithi088.stderr:0: v1:172.21.15.88:6789/0 mon.a 2024-06-03T08:36:36.078 INFO:teuthology.orchestra.run.smithi088.stderr:1: v1:172.21.15.192:6789/0 mon.b 2024-06-03T08:36:36.078 INFO:teuthology.orchestra.run.smithi088.stderr:2: v1:172.21.15.88:6790/0 mon.c 2024-06-03T08:36:36.078 INFO:teuthology.orchestra.run.smithi088.stderr: 2024-06-03T08:36:36.078 INFO:teuthology.orchestra.run.smithi088.stderr:2024-06-03T08:36:36.075+0000 7ff21cff9640 10 monclient(hunting): _finish_auth 0 2024-06-03T08:36:36.078 INFO:teuthology.orchestra.run.smithi088.stderr:2024-06-03T08:36:36.076+0000 7ff21cff9640 1 -- 172.21.15.88:0/551408712 <== mon.2 v1:172.21.15.88:6790/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 764+0+0 (unknown 4050830884 0 0) 0x7ff20c004eb0 con 0x7ff22006ee00 2024-06-03T08:36:36.078 INFO:teuthology.orchestra.run.smithi088.stderr:2024-06-03T08:36:36.076+0000 7ff21cff9640 1 -- 172.21.15.88:0/551408712 --> v1:172.21.15.88:6790/0 -- auth(proto 2 165 bytes epoch 0) v1 -- 0x7ff220138820 con 0x7ff22006ee00 2024-06-03T08:36:36.078 INFO:teuthology.orchestra.run.smithi088.stderr:2024-06-03T08:36:36.076+0000 7ff21cff9640 1 -- 172.21.15.88:0/551408712 <== mon.2 v1:172.21.15.88:6790/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 2234374065 0 0) 0x7ff210001a60 con 0x7ff22006ee00 2024-06-03T08:36:36.078 INFO:teuthology.orchestra.run.smithi088.stderr:2024-06-03T08:36:36.076+0000 7ff21cff9640 10 monclient(hunting): handle_monmap mon_map magic: 0 v1 2024-06-03T08:36:36.078 INFO:teuthology.orchestra.run.smithi088.stderr:2024-06-03T08:36:36.076+0000 7ff21cff9640 10 monclient(hunting): got monmap 1 from mon.c (according to old e1) 2024-06-03T08:36:36.079 INFO:teuthology.orchestra.run.smithi088.stderr:2024-06-03T08:36:36.076+0000 7ff21cff9640 10 monclient(hunting): dump: 2024-06-03T08:36:36.079 INFO:teuthology.orchestra.run.smithi088.stderr:epoch 1 2024-06-03T08:36:36.079 INFO:teuthology.orchestra.run.smithi088.stderr:fsid ec3c3fdd-3ac6-4d7d-9412-683130ed8b04 2024-06-03T08:36:36.079 INFO:teuthology.orchestra.run.smithi088.stderr:last_changed 2024-06-03T08:34:31.885125+0000 2024-06-03T08:36:36.079 INFO:teuthology.orchestra.run.smithi088.stderr:created 2024-06-03T08:34:31.885125+0000 2024-06-03T08:36:36.079 INFO:teuthology.orchestra.run.smithi088.stderr:min_mon_release 18 (reef) 2024-06-03T08:36:36.079 INFO:teuthology.orchestra.run.smithi088.stderr:election_strategy: 3 2024-06-03T08:36:36.079 INFO:teuthology.orchestra.run.smithi088.stderr:0: v1:172.21.15.88:6789/0 mon.a 2024-06-03T08:36:36.079 INFO:teuthology.orchestra.run.smithi088.stderr:1: v1:172.21.15.192:6789/0 mon.b 2024-06-03T08:36:36.079 INFO:teuthology.orchestra.run.smithi088.stderr:2: v1:172.21.15.88:6790/0 mon.c 2024-06-03T08:36:36.079 INFO:teuthology.orchestra.run.smithi088.stderr: 2024-06-03T08:36:36.079 INFO:teuthology.orchestra.run.smithi088.stderr:2024-06-03T08:36:36.076+0000 7ff21cff9640 1 -- 172.21.15.88:0/551408712 <== mon.0 v1:172.21.15.88:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 764+0+0 (unknown 3030258420 0 0) 0x7ff220133ca0 con 0x7ff2200adb90 2024-06-03T08:36:36.079 INFO:teuthology.orchestra.run.smithi088.stderr:2024-06-03T08:36:36.076+0000 7ff21cff9640 1 -- 172.21.15.88:0/551408712 --> v1:172.21.15.88:6789/0 -- auth(proto 2 165 bytes epoch 0) v1 -- 0x7ff20c004eb0 con 0x7ff2200adb90 2024-06-03T08:36:36.079 INFO:teuthology.orchestra.run.smithi088.stderr:2024-06-03T08:36:36.076+0000 7ff21cff9640 1 -- 172.21.15.88:0/551408712 <== mon.0 v1:172.21.15.88:6789/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 2234374065 0 0) 0x7ff214001890 con 0x7ff2200adb90 2024-06-03T08:36:36.079 INFO:teuthology.orchestra.run.smithi088.stderr:2024-06-03T08:36:36.076+0000 7ff21cff9640 10 monclient(hunting): handle_monmap mon_map magic: 0 v1 2024-06-03T08:36:36.080 INFO:teuthology.orchestra.run.smithi088.stderr:2024-06-03T08:36:36.076+0000 7ff21cff9640 10 monclient(hunting): got monmap 1 from mon.a (according to old e1) 2024-06-03T08:36:36.080 INFO:teuthology.orchestra.run.smithi088.stderr:2024-06-03T08:36:36.076+0000 7ff21cff9640 10 monclient(hunting): dump: 2024-06-03T08:36:36.080 INFO:teuthology.orchestra.run.smithi088.stderr:epoch 1 2024-06-03T08:36:36.080 INFO:teuthology.orchestra.run.smithi088.stderr:fsid ec3c3fdd-3ac6-4d7d-9412-683130ed8b04 2024-06-03T08:36:36.080 INFO:teuthology.orchestra.run.smithi088.stderr:last_changed 2024-06-03T08:34:31.885125+0000 2024-06-03T08:36:36.080 INFO:teuthology.orchestra.run.smithi088.stderr:created 2024-06-03T08:34:31.885125+0000 2024-06-03T08:36:36.080 INFO:teuthology.orchestra.run.smithi088.stderr:min_mon_release 18 (reef) 2024-06-03T08:36:36.080 INFO:teuthology.orchestra.run.smithi088.stderr:election_strategy: 3 2024-06-03T08:36:36.080 INFO:teuthology.orchestra.run.smithi088.stderr:0: v1:172.21.15.88:6789/0 mon.a 2024-06-03T08:36:36.080 INFO:teuthology.orchestra.run.smithi088.stderr:1: v1:172.21.15.192:6789/0 mon.b 2024-06-03T08:36:36.080 INFO:teuthology.orchestra.run.smithi088.stderr:2: v1:172.21.15.88:6790/0 mon.c 2024-06-03T08:36:36.080 INFO:teuthology.orchestra.run.smithi088.stderr: 2024-06-03T08:36:36.082 INFO:teuthology.orchestra.run.smithi088.stderr:2024-06-03T08:36:36.079+0000 7ff21cff9640 1 -- 172.21.15.88:0/551408712 <== mon.2 v1:172.21.15.88:6790/0 4 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 580+0+0 (unknown 3962199831 0 0) 0x7ff220138820 con 0x7ff22006ee00 2024-06-03T08:36:36.082 INFO:teuthology.orchestra.run.smithi088.stderr:2024-06-03T08:36:36.079+0000 7ff21cff9640 1 -- 172.21.15.88:0/551408712 >> v1:172.21.15.192:6789/0 conn(0x7ff2201383e0 legacy=0x7ff220133590 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-06-03T08:36:36.082 INFO:teuthology.orchestra.run.smithi088.stderr:2024-06-03T08:36:36.079+0000 7ff21cff9640 1 -- 172.21.15.88:0/551408712 >> v1:172.21.15.88:6789/0 conn(0x7ff2200adb90 legacy=0x7ff220132e80 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-06-03T08:36:36.082 INFO:teuthology.orchestra.run.smithi088.stderr:2024-06-03T08:36:36.079+0000 7ff21cff9640 10 monclient: _finish_hunting 0 2024-06-03T08:36:36.082 INFO:teuthology.orchestra.run.smithi088.stderr:2024-06-03T08:36:36.079+0000 7ff21cff9640 1 monclient: found mon.c 2024-06-03T08:36:36.082 INFO:teuthology.orchestra.run.smithi088.stderr:2024-06-03T08:36:36.079+0000 7ff21cff9640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-06-03T08:36:36.082 INFO:teuthology.orchestra.run.smithi088.stderr:2024-06-03T08:36:36.079+0000 7ff21cff9640 10 monclient: _send_mon_message to mon.c at v1:172.21.15.88:6790/0 2024-06-03T08:36:36.082 INFO:teuthology.orchestra.run.smithi088.stderr:2024-06-03T08:36:36.079+0000 7ff21cff9640 1 -- 172.21.15.88:0/551408712 --> v1:172.21.15.88:6790/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7ff2201389b0 con 0x7ff22006ee00 2024-06-03T08:36:36.082 INFO:teuthology.orchestra.run.smithi088.stderr:2024-06-03T08:36:36.079+0000 7ff21cff9640 10 monclient: _finish_auth 0 2024-06-03T08:36:36.082 INFO:teuthology.orchestra.run.smithi088.stderr:2024-06-03T08:36:36.079+0000 7ff21cff9640 10 monclient: _check_auth_tickets 2024-06-03T08:36:36.082 INFO:teuthology.orchestra.run.smithi088.stderr:2024-06-03T08:36:36.079+0000 7ff21cff9640 20 monclient: _check_auth_rotating not needed by client.admin 2024-06-03T08:36:36.082 INFO:teuthology.orchestra.run.smithi088.stderr:2024-06-03T08:36:36.079+0000 7ff22643b640 5 monclient: authenticate success, global_id 4400 2024-06-03T08:36:36.082 INFO:teuthology.orchestra.run.smithi088.stderr:2024-06-03T08:36:36.079+0000 7ff22643b640 10 monclient: _renew_subs 2024-06-03T08:36:36.083 INFO:teuthology.orchestra.run.smithi088.stderr:2024-06-03T08:36:36.079+0000 7ff22643b640 10 monclient: _send_mon_message to mon.c at v1:172.21.15.88:6790/0 2024-06-03T08:36:36.083 INFO:teuthology.orchestra.run.smithi088.stderr:2024-06-03T08:36:36.079+0000 7ff22643b640 1 -- 172.21.15.88:0/551408712 --> v1:172.21.15.88:6790/0 -- mon_subscribe({mgrmap=0+}) v3 -- 0x7ff22013f050 con 0x7ff22006ee00 2024-06-03T08:36:36.083 INFO:teuthology.orchestra.run.smithi088.stderr:2024-06-03T08:36:36.079+0000 7ff22643b640 10 client.4400.objecter _maybe_request_map subscribing (onetime) to next osd map 2024-06-03T08:36:36.083 INFO:teuthology.orchestra.run.smithi088.stderr:2024-06-03T08:36:36.079+0000 7ff22643b640 10 monclient: _renew_subs 2024-06-03T08:36:36.083 INFO:teuthology.orchestra.run.smithi088.stderr:2024-06-03T08:36:36.079+0000 7ff22643b640 10 monclient: _send_mon_message to mon.c at v1:172.21.15.88:6790/0 2024-06-03T08:36:36.083 INFO:teuthology.orchestra.run.smithi088.stderr:2024-06-03T08:36:36.079+0000 7ff22643b640 1 -- 172.21.15.88:0/551408712 --> v1:172.21.15.88:6790/0 -- mon_subscribe({osdmap=0}) v3 -- 0x7ff22013b670 con 0x7ff22006ee00 2024-06-03T08:36:36.083 INFO:teuthology.orchestra.run.smithi088.stderr:2024-06-03T08:36:36.079+0000 7ff22643b640 1 librados: init done 2024-06-03T08:36:36.083 INFO:teuthology.orchestra.run.smithi088.stderr:2024-06-03T08:36:36.080+0000 7ff22643b640 10 start_mon_command cmd=[{"prefix": "get_command_descriptions"}] 2024-06-03T08:36:36.083 INFO:teuthology.orchestra.run.smithi088.stderr:2024-06-03T08:36:36.080+0000 7ff22643b640 10 monclient: _send_command 1 [{"prefix": "get_command_descriptions"}] 2024-06-03T08:36:36.083 INFO:teuthology.orchestra.run.smithi088.stderr:2024-06-03T08:36:36.080+0000 7ff22643b640 10 monclient: _send_mon_message to mon.c at v1:172.21.15.88:6790/0 2024-06-03T08:36:36.083 INFO:teuthology.orchestra.run.smithi088.stderr:2024-06-03T08:36:36.080+0000 7ff22643b640 1 -- 172.21.15.88:0/551408712 --> v1:172.21.15.88:6790/0 -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7ff22013f240 con 0x7ff22006ee00 2024-06-03T08:36:36.326 INFO:teuthology.orchestra.run.smithi088.stderr:2024-06-03T08:36:36.325+0000 7ff21cff9640 1 -- 172.21.15.88:0/551408712 <== mon.2 v1:172.21.15.88:6790/0 5 ==== config(0 keys) v1 ==== 4+0+0 (unknown 0 0 0) 0x7ff210004a90 con 0x7ff22006ee00 2024-06-03T08:36:36.326 INFO:teuthology.orchestra.run.smithi088.stderr:2024-06-03T08:36:36.325+0000 7ff21cff9640 10 monclient: handle_config config(0 keys) v1 2024-06-03T08:36:36.326 INFO:teuthology.orchestra.run.smithi088.stderr:2024-06-03T08:36:36.325+0000 7ff21cff9640 1 -- 172.21.15.88:0/551408712 <== mon.2 v1:172.21.15.88:6790/0 6 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 2234374065 0 0) 0x7ff210004da0 con 0x7ff22006ee00 2024-06-03T08:36:36.326 INFO:teuthology.orchestra.run.smithi088.stderr:2024-06-03T08:36:36.325+0000 7ff21cff9640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-06-03T08:36:36.326 INFO:teuthology.orchestra.run.smithi088.stderr:2024-06-03T08:36:36.325+0000 7ff21cff9640 10 monclient: got monmap 1 from mon.c (according to old e1) 2024-06-03T08:36:36.326 INFO:teuthology.orchestra.run.smithi088.stderr:2024-06-03T08:36:36.325+0000 7ff21cff9640 10 monclient: dump: 2024-06-03T08:36:36.326 INFO:teuthology.orchestra.run.smithi088.stderr:epoch 1 2024-06-03T08:36:36.326 INFO:teuthology.orchestra.run.smithi088.stderr:fsid ec3c3fdd-3ac6-4d7d-9412-683130ed8b04 2024-06-03T08:36:36.327 INFO:teuthology.orchestra.run.smithi088.stderr:last_changed 2024-06-03T08:34:31.885125+0000 2024-06-03T08:36:36.327 INFO:teuthology.orchestra.run.smithi088.stderr:created 2024-06-03T08:34:31.885125+0000 2024-06-03T08:36:36.327 INFO:teuthology.orchestra.run.smithi088.stderr:min_mon_release 18 (reef) 2024-06-03T08:36:36.327 INFO:teuthology.orchestra.run.smithi088.stderr:election_strategy: 3 2024-06-03T08:36:36.327 INFO:teuthology.orchestra.run.smithi088.stderr:0: v1:172.21.15.88:6789/0 mon.a 2024-06-03T08:36:36.327 INFO:teuthology.orchestra.run.smithi088.stderr:1: v1:172.21.15.192:6789/0 mon.b 2024-06-03T08:36:36.327 INFO:teuthology.orchestra.run.smithi088.stderr:2: v1:172.21.15.88:6790/0 mon.c 2024-06-03T08:36:36.327 INFO:teuthology.orchestra.run.smithi088.stderr: 2024-06-03T08:36:36.327 INFO:teuthology.orchestra.run.smithi088.stderr:2024-06-03T08:36:36.325+0000 7ff21e7fc640 4 set_mon_vals no callback set 2024-06-03T08:36:36.328 INFO:teuthology.orchestra.run.smithi088.stderr:2024-06-03T08:36:36.326+0000 7ff21cff9640 1 -- 172.21.15.88:0/551408712 <== mon.2 v1:172.21.15.88:6790/0 7 ==== mgrmap(e 5) v1 ==== 89110+0+0 (unknown 873533973 0 0) 0x7ff21001ab20 con 0x7ff22006ee00 2024-06-03T08:36:36.328 INFO:teuthology.orchestra.run.smithi088.stderr:2024-06-03T08:36:36.327+0000 7ff21f7fe640 10 monclient: get_auth_request con 0x7ff20c06d470 auth_method 0 2024-06-03T08:36:36.328 INFO:teuthology.orchestra.run.smithi088.stderr:2024-06-03T08:36:36.327+0000 7ff21cff9640 10 client.4400.objecter ms_handle_connect 0x7ff20c06d470 2024-06-03T08:36:36.331 INFO:teuthology.orchestra.run.smithi088.stderr:2024-06-03T08:36:36.331+0000 7ff21cff9640 1 -- 172.21.15.88:0/551408712 <== mon.2 v1:172.21.15.88:6790/0 8 ==== osd_map(26..26 src has 1..26) v4 ==== 4090+0+0 (unknown 3637450554 0 0) 0x7ff2100541c0 con 0x7ff22006ee00 2024-06-03T08:36:36.332 INFO:teuthology.orchestra.run.smithi088.stderr:2024-06-03T08:36:36.331+0000 7ff21cff9640 10 client.4400.objecter ms_dispatch 0x7ff220000b90 osd_map(26..26 src has 1..26) v4 2024-06-03T08:36:36.332 INFO:teuthology.orchestra.run.smithi088.stderr:2024-06-03T08:36:36.331+0000 7ff21cff9640 3 client.4400.objecter handle_osd_map got epochs [26,26] > 0 2024-06-03T08:36:36.332 INFO:teuthology.orchestra.run.smithi088.stderr:2024-06-03T08:36:36.331+0000 7ff21cff9640 3 client.4400.objecter handle_osd_map decoding full epoch 26 2024-06-03T08:36:36.332 INFO:teuthology.orchestra.run.smithi088.stderr:2024-06-03T08:36:36.331+0000 7ff21cff9640 20 client.4400.objecter dump_active .. 0 homeless 2024-06-03T08:36:36.332 INFO:teuthology.orchestra.run.smithi088.stderr:2024-06-03T08:36:36.331+0000 7ff21cff9640 1 -- 172.21.15.88:0/551408712 <== mon.2 v1:172.21.15.88:6790/0 9 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+182498 (unknown 1092875540 0 3016760700) 0x7ff21004fd70 con 0x7ff22006ee00 2024-06-03T08:36:36.332 INFO:teuthology.orchestra.run.smithi088.stderr:2024-06-03T08:36:36.331+0000 7ff21cff9640 10 monclient: handle_mon_command_ack 1 [{"prefix": "get_command_descriptions"}] 2024-06-03T08:36:36.332 INFO:teuthology.orchestra.run.smithi088.stderr:2024-06-03T08:36:36.331+0000 7ff21cff9640 10 monclient: _finish_command 1 = system:0 2024-06-03T08:36:36.510 INFO:teuthology.orchestra.run.smithi088.stderr:2024-06-03T08:36:36.509+0000 7ff22643b640 1 -- 172.21.15.88:0/551408712 --> v1:172.21.15.192:6800/34229 -- mgr_command(tid 0: {"prefix": "pg dump", "target": ["mon-mgr", ""], "format": "json"}) v1 -- 0x7ff2200adf70 con 0x7ff20c06d470 2024-06-03T08:36:36.512 INFO:teuthology.orchestra.run.smithi088.stderr:2024-06-03T08:36:36.512+0000 7ff21cff9640 1 -- 172.21.15.88:0/551408712 <== mgr.4110 v1:172.21.15.192:6800/34229 1 ==== mgr_command_reply(tid 0: 0 dumped all) v1 ==== 18+0+48018 (unknown 2965378022 0 3644972437) 0x7ff2140077c0 con 0x7ff20c06d470 2024-06-03T08:36:36.513 INFO:teuthology.orchestra.run.smithi088.stdout: 2024-06-03T08:36:36.513 INFO:teuthology.orchestra.run.smithi088.stderr:dumped all 2024-06-03T08:36:36.514 INFO:teuthology.orchestra.run.smithi088.stderr:2024-06-03T08:36:36.513+0000 7ff22643b640 10 librados: watch_flush enter 2024-06-03T08:36:36.514 INFO:teuthology.orchestra.run.smithi088.stderr:2024-06-03T08:36:36.514+0000 7ff22643b640 10 librados: watch_flush exit 2024-06-03T08:36:36.515 INFO:teuthology.orchestra.run.smithi088.stderr:2024-06-03T08:36:36.514+0000 7ff22643b640 20 client.4400.objecter shutdown clearing up homeless session... 2024-06-03T08:36:36.515 INFO:teuthology.orchestra.run.smithi088.stderr:2024-06-03T08:36:36.514+0000 7ff22643b640 10 client.4400.objecter successfully canceled tick 2024-06-03T08:36:36.515 INFO:teuthology.orchestra.run.smithi088.stderr:2024-06-03T08:36:36.514+0000 7ff22643b640 1 -- 172.21.15.88:0/551408712 >> v1:172.21.15.192:6800/34229 conn(0x7ff20c06d470 legacy=0x7ff20c06f930 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-06-03T08:36:36.515 INFO:teuthology.orchestra.run.smithi088.stderr:2024-06-03T08:36:36.514+0000 7ff22643b640 10 monclient: shutdown 2024-06-03T08:36:36.515 INFO:teuthology.orchestra.run.smithi088.stderr:2024-06-03T08:36:36.514+0000 7ff22643b640 20 monclient: shutdown discarding 0 pending message(s) 2024-06-03T08:36:36.515 INFO:teuthology.orchestra.run.smithi088.stderr:2024-06-03T08:36:36.514+0000 7ff22643b640 1 -- 172.21.15.88:0/551408712 >> v1:172.21.15.88:6790/0 conn(0x7ff22006ee00 legacy=0x7ff220138e00 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-06-03T08:36:36.515 INFO:teuthology.orchestra.run.smithi088.stderr:2024-06-03T08:36:36.514+0000 7ff22643b640 1 -- 172.21.15.88:0/551408712 shutdown_connections 2024-06-03T08:36:36.515 INFO:teuthology.orchestra.run.smithi088.stderr:2024-06-03T08:36:36.514+0000 7ff22643b640 1 -- 172.21.15.88:0/551408712 >> 172.21.15.88:0/551408712 conn(0x7ff2200b2f20 msgr2=0x7ff22006d370 unknown :-1 s=STATE_NONE l=0).mark_down 2024-06-03T08:36:36.515 INFO:teuthology.orchestra.run.smithi088.stderr:2024-06-03T08:36:36.514+0000 7ff22643b640 1 -- 172.21.15.88:0/551408712 shutdown_connections 2024-06-03T08:36:36.515 INFO:teuthology.orchestra.run.smithi088.stderr:2024-06-03T08:36:36.514+0000 7ff22643b640 1 -- 172.21.15.88:0/551408712 wait complete. 2024-06-03T08:36:36.515 INFO:teuthology.orchestra.run.smithi088.stderr:2024-06-03T08:36:36.514+0000 7ff22643b640 1 librados: shutdown 2024-06-03T08:36:36.524 INFO:teuthology.orchestra.run.smithi088.stdout:{"pg_ready":true,"pg_map":{"version":40,"stamp":"2024-06-03T08:36:34.591503+0000","last_osdmap_epoch":0,"last_pg_scan":0,"pg_stats_sum":{"stat_sum":{"num_bytes":459280,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":96,"num_read_kb":82,"num_write":113,"num_write_kb":1372,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2,"num_bytes_recovered":397840,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":76,"ondisk_log_size":76,"up":18,"acting":18,"num_store_stats":0},"osd_stats_sum":{"up_from":0,"seq":0,"num_pgs":18,"num_osds":8,"num_per_pool_osds":8,"num_per_pool_omap_osds":8,"kb":754974720,"kb_used":218692,"kb_used_data":2504,"kb_used_omap":12,"kb_used_meta":216051,"kb_avail":754756028,"statfs":{"total":773094113280,"available":772870172672,"internally_reserved":0,"allocated":2564096,"data_stored":1403524,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":12744,"internal_metadata":221236792},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":48,"apply_latency_ms":48,"commit_latency_ns":48000000,"apply_latency_ns":48000000},"alerts":[],"network_ping_times":[]},"pg_stats_delta":{"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2,"num_bytes_recovered":397840,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":0,"acting":0,"num_store_stats":0,"stamp_delta":"6.001199"},"pg_stats":[{"pgid":"2.7","version":"0'0","reported_seq":11,"reported_epoch":25,"state":"active+clean","last_fresh":"2024-06-03T08:36:31.935126+0000","last_change":"2024-06-03T08:36:30.385405+0000","last_active":"2024-06-03T08:36:31.935126+0000","last_peered":"2024-06-03T08:36:31.935126+0000","last_clean":"2024-06-03T08:36:31.935126+0000","last_became_active":"2024-06-03T08:36:30.380302+0000","last_became_peered":"2024-06-03T08:36:30.380302+0000","last_unstale":"2024-06-03T08:36:31.935126+0000","last_undegraded":"2024-06-03T08:36:31.935126+0000","last_fullsized":"2024-06-03T08:36:31.935126+0000","mapping_epoch":23,"log_start":"0'0","ondisk_log_start":"0'0","created":23,"last_epoch_clean":24,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-06-03T08:36:28.444764+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-06-03T08:36:28.444764+0000","last_clean_scrub_stamp":"2024-06-03T08:36:28.444764+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-06-04T18:46:33.475181+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0],"acting":[3,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"2.6","version":"0'0","reported_seq":11,"reported_epoch":25,"state":"active+clean","last_fresh":"2024-06-03T08:36:32.484953+0000","last_change":"2024-06-03T08:36:30.430760+0000","last_active":"2024-06-03T08:36:32.484953+0000","last_peered":"2024-06-03T08:36:32.484953+0000","last_clean":"2024-06-03T08:36:32.484953+0000","last_became_active":"2024-06-03T08:36:30.404665+0000","last_became_peered":"2024-06-03T08:36:30.404665+0000","last_unstale":"2024-06-03T08:36:32.484953+0000","last_undegraded":"2024-06-03T08:36:32.484953+0000","last_fullsized":"2024-06-03T08:36:32.484953+0000","mapping_epoch":23,"log_start":"0'0","ondisk_log_start":"0'0","created":23,"last_epoch_clean":24,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-06-03T08:36:28.444764+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-06-03T08:36:28.444764+0000","last_clean_scrub_stamp":"2024-06-03T08:36:28.444764+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-06-04T10:37:37.746074+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,0],"acting":[4,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"2.5","version":"0'0","reported_seq":11,"reported_epoch":25,"state":"active+clean","last_fresh":"2024-06-03T08:36:31.913806+0000","last_change":"2024-06-03T08:36:30.024235+0000","last_active":"2024-06-03T08:36:31.913806+0000","last_peered":"2024-06-03T08:36:31.913806+0000","last_clean":"2024-06-03T08:36:31.913806+0000","last_became_active":"2024-06-03T08:36:30.019040+0000","last_became_peered":"2024-06-03T08:36:30.019040+0000","last_unstale":"2024-06-03T08:36:31.913806+0000","last_undegraded":"2024-06-03T08:36:31.913806+0000","last_fullsized":"2024-06-03T08:36:31.913806+0000","mapping_epoch":23,"log_start":"0'0","ondisk_log_start":"0'0","created":23,"last_epoch_clean":24,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-06-03T08:36:28.444764+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-06-03T08:36:28.444764+0000","last_clean_scrub_stamp":"2024-06-03T08:36:28.444764+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-06-04T15:46:47.109502+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,4],"acting":[3,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"2.4","version":"0'0","reported_seq":11,"reported_epoch":26,"state":"active+clean","last_fresh":"2024-06-03T08:36:31.819576+0000","last_change":"2024-06-03T08:36:30.264987+0000","last_active":"2024-06-03T08:36:31.819576+0000","last_peered":"2024-06-03T08:36:31.819576+0000","last_clean":"2024-06-03T08:36:31.819576+0000","last_became_active":"2024-06-03T08:36:30.237862+0000","last_became_peered":"2024-06-03T08:36:30.237862+0000","last_unstale":"2024-06-03T08:36:31.819576+0000","last_undegraded":"2024-06-03T08:36:31.819576+0000","last_fullsized":"2024-06-03T08:36:31.819576+0000","mapping_epoch":23,"log_start":"0'0","ondisk_log_start":"0'0","created":23,"last_epoch_clean":24,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-06-03T08:36:28.444764+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-06-03T08:36:28.444764+0000","last_clean_scrub_stamp":"2024-06-03T08:36:28.444764+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-06-04T17:35:23.776572+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,6],"acting":[5,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"2.2","version":"0'0","reported_seq":11,"reported_epoch":25,"state":"active+clean","last_fresh":"2024-06-03T08:36:31.918592+0000","last_change":"2024-06-03T08:36:30.230431+0000","last_active":"2024-06-03T08:36:31.918592+0000","last_peered":"2024-06-03T08:36:31.918592+0000","last_clean":"2024-06-03T08:36:31.918592+0000","last_became_active":"2024-06-03T08:36:30.221718+0000","last_became_peered":"2024-06-03T08:36:30.221718+0000","last_unstale":"2024-06-03T08:36:31.918592+0000","last_undegraded":"2024-06-03T08:36:31.918592+0000","last_fullsized":"2024-06-03T08:36:31.918592+0000","mapping_epoch":23,"log_start":"0'0","ondisk_log_start":"0'0","created":23,"last_epoch_clean":24,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-06-03T08:36:28.444764+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-06-03T08:36:28.444764+0000","last_clean_scrub_stamp":"2024-06-03T08:36:28.444764+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-06-04T15:23:37.659061+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,6],"acting":[3,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"2.1","version":"0'0","reported_seq":11,"reported_epoch":26,"state":"active+clean","last_fresh":"2024-06-03T08:36:31.900749+0000","last_change":"2024-06-03T08:36:30.883360+0000","last_active":"2024-06-03T08:36:31.900749+0000","last_peered":"2024-06-03T08:36:31.900749+0000","last_clean":"2024-06-03T08:36:31.900749+0000","last_became_active":"2024-06-03T08:36:30.855073+0000","last_became_peered":"2024-06-03T08:36:30.855073+0000","last_unstale":"2024-06-03T08:36:31.900749+0000","last_undegraded":"2024-06-03T08:36:31.900749+0000","last_fullsized":"2024-06-03T08:36:31.900749+0000","mapping_epoch":23,"log_start":"0'0","ondisk_log_start":"0'0","created":23,"last_epoch_clean":25,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-06-03T08:36:28.444764+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-06-03T08:36:28.444764+0000","last_clean_scrub_stamp":"2024-06-03T08:36:28.444764+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-06-04T18:12:22.363473+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,5],"acting":[7,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"2.0","version":"0'0","reported_seq":11,"reported_epoch":25,"state":"active+clean","last_fresh":"2024-06-03T08:36:31.924510+0000","last_change":"2024-06-03T08:36:29.984096+0000","last_active":"2024-06-03T08:36:31.924510+0000","last_peered":"2024-06-03T08:36:31.924510+0000","last_clean":"2024-06-03T08:36:31.924510+0000","last_became_active":"2024-06-03T08:36:29.957875+0000","last_became_peered":"2024-06-03T08:36:29.957875+0000","last_unstale":"2024-06-03T08:36:31.924510+0000","last_undegraded":"2024-06-03T08:36:31.924510+0000","last_fullsized":"2024-06-03T08:36:31.924510+0000","mapping_epoch":23,"log_start":"0'0","ondisk_log_start":"0'0","created":23,"last_epoch_clean":24,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-06-03T08:36:28.444764+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-06-03T08:36:28.444764+0000","last_clean_scrub_stamp":"2024-06-03T08:36:28.444764+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-06-04T08:41:33.679479+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,5],"acting":[3,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"2.3","version":"0'0","reported_seq":11,"reported_epoch":25,"state":"active+clean","last_fresh":"2024-06-03T08:36:31.921442+0000","last_change":"2024-06-03T08:36:30.351670+0000","last_active":"2024-06-03T08:36:31.921442+0000","last_peered":"2024-06-03T08:36:31.921442+0000","last_clean":"2024-06-03T08:36:31.921442+0000","last_became_active":"2024-06-03T08:36:30.343560+0000","last_became_peered":"2024-06-03T08:36:30.343560+0000","last_unstale":"2024-06-03T08:36:31.921442+0000","last_undegraded":"2024-06-03T08:36:31.921442+0000","last_fullsized":"2024-06-03T08:36:31.921442+0000","mapping_epoch":23,"log_start":"0'0","ondisk_log_start":"0'0","created":23,"last_epoch_clean":24,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-06-03T08:36:28.444764+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-06-03T08:36:28.444764+0000","last_clean_scrub_stamp":"2024-06-03T08:36:28.444764+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-06-04T12:02:24.765135+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,2],"acting":[3,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.0","version":"19'76","reported_seq":131,"reported_epoch":25,"state":"active+clean","last_fresh":"2024-06-03T08:36:31.901743+0000","last_change":"2024-06-03T08:36:26.702702+0000","last_active":"2024-06-03T08:36:31.901743+0000","last_peered":"2024-06-03T08:36:31.901743+0000","last_clean":"2024-06-03T08:36:31.901743+0000","last_became_active":"2024-06-03T08:36:26.128378+0000","last_became_peered":"2024-06-03T08:36:26.128378+0000","last_unstale":"2024-06-03T08:36:31.901743+0000","last_undegraded":"2024-06-03T08:36:31.901743+0000","last_fullsized":"2024-06-03T08:36:31.901743+0000","mapping_epoch":20,"log_start":"0'0","ondisk_log_start":"0'0","created":16,"last_epoch_clean":22,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-06-03T08:36:15.542635+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-06-03T08:36:15.542635+0000","last_clean_scrub_stamp":"2024-06-03T08:36:15.542635+0000","objects_scrubbed":0,"log_size":76,"log_dups_size":0,"ondisk_log_size":76,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-06-04T12:41:53.731360+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":459280,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":96,"num_read_kb":82,"num_write":113,"num_write_kb":1372,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2,"num_bytes_recovered":397840,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,5],"acting":[1,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]}],"pool_stats":[{"poolid":2,"num_pg":8,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":16,"acting":16,"num_store_stats":7},{"poolid":1,"num_pg":1,"stat_sum":{"num_bytes":459280,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":96,"num_read_kb":82,"num_write":113,"num_write_kb":1372,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2,"num_bytes_recovered":397840,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":802816,"data_stored":795680,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":76,"ondisk_log_size":76,"up":2,"acting":2,"num_store_stats":2}],"osd_stats":[{"osd":6,"up_from":21,"seq":90194313226,"num_pgs":2,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":27232,"kb_used_data":208,"kb_used_omap":1,"kb_used_meta":27006,"kb_avail":94344608,"statfs":{"total":96636764160,"available":96608878592,"internally_reserved":0,"allocated":212992,"data_stored":71931,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1593,"internal_metadata":27654599},"hb_peers":[0,1,2,3,4,5,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":30,"apply_latency_ms":30,"commit_latency_ns":30000000,"apply_latency_ns":30000000},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":9.282}]},{"osd":1,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":7.718}]},{"osd":2,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":12.675000000000001}]},{"osd":3,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":6.6449999999999996}]},{"osd":4,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":15.222}]},{"osd":5,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":13.324}]},{"osd":7,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":12.085000000000001}]}]},{"osd":7,"up_from":20,"seq":85899345929,"num_pgs":1,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":27236,"kb_used_data":216,"kb_used_omap":1,"kb_used_meta":27006,"kb_avail":94344604,"statfs":{"total":96636764160,"available":96608874496,"internally_reserved":0,"allocated":221184,"data_stored":76559,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1589,"internal_metadata":27654603},"hb_peers":[0,1,2,3,4,5,6],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":11.688000000000001}]},{"osd":1,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":7.5970000000000004}]},{"osd":2,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":13.432}]},{"osd":3,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":6.6369999999999996}]},{"osd":4,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":12.353999999999999}]},{"osd":5,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":10.159000000000001}]},{"osd":6,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":14.718}]}]},{"osd":5,"up_from":20,"seq":85899345931,"num_pgs":4,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":27636,"kb_used_data":608,"kb_used_omap":1,"kb_used_meta":27006,"kb_avail":94344204,"statfs":{"total":96636764160,"available":96608464896,"internally_reserved":0,"allocated":622592,"data_stored":474399,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1594,"internal_metadata":27654598},"hb_peers":[0,1,2,3,4,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":14.609999999999999}]},{"osd":1,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":17.68}]},{"osd":2,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":6.5190000000000001}]},{"osd":3,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":16.898}]},{"osd":4,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":7.601}]},{"osd":6,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":13.211}]},{"osd":7,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":11.711}]}]},{"osd":4,"up_from":20,"seq":85899345929,"num_pgs":2,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":27240,"kb_used_data":216,"kb_used_omap":1,"kb_used_meta":27006,"kb_avail":94344600,"statfs":{"total":96636764160,"available":96608870400,"internally_reserved":0,"allocated":221184,"data_stored":76559,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1594,"internal_metadata":27654598},"hb_peers":[0,1,2,3,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":8,"apply_latency_ms":8,"commit_latency_ns":8000000,"apply_latency_ns":8000000},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":7.7750000000000004}]},{"osd":1,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":9.2070000000000007}]},{"osd":2,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":21.491}]},{"osd":3,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":9.9920000000000009}]},{"osd":5,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":22.495999999999999}]},{"osd":6,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":10.614000000000001}]},{"osd":7,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":12.737}]}]},{"osd":2,"up_from":20,"seq":85899345930,"num_pgs":1,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":27236,"kb_used_data":216,"kb_used_omap":1,"kb_used_meta":27006,"kb_avail":94344604,"statfs":{"total":96636764160,"available":96608874496,"internally_reserved":0,"allocated":221184,"data_stored":76559,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1594,"internal_metadata":27654598},"hb_peers":[0,1,3,4,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":6.2350000000000003}]},{"osd":1,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":7.7670000000000003}]},{"osd":3,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":13.222}]},{"osd":4,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":17.765999999999998}]},{"osd":5,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":10.678000000000001}]},{"osd":6,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":11.221}]},{"osd":7,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":15.510999999999999}]}]},{"osd":3,"up_from":14,"seq":60129542153,"num_pgs":5,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":27236,"kb_used_data":216,"kb_used_omap":1,"kb_used_meta":27006,"kb_avail":94344604,"statfs":{"total":96636764160,"available":96608874496,"internally_reserved":0,"allocated":221184,"data_stored":76559,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1592,"internal_metadata":27654600},"hb_peers":[0,1,2,4,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":5,"apply_latency_ms":5,"commit_latency_ns":5000000,"apply_latency_ns":5000000},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":8.1679999999999993}]},{"osd":1,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":10.41}]},{"osd":2,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":9.3439999999999994}]},{"osd":4,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":12.706}]},{"osd":5,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":17.991}]},{"osd":6,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":14.965}]},{"osd":7,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":11.981}]}]},{"osd":1,"up_from":14,"seq":60129542153,"num_pgs":1,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":27632,"kb_used_data":608,"kb_used_omap":1,"kb_used_meta":27006,"kb_avail":94344208,"statfs":{"total":96636764160,"available":96608468992,"internally_reserved":0,"allocated":622592,"data_stored":474399,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":27654597},"hb_peers":[0,2,3,4,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":5,"apply_latency_ms":5,"commit_latency_ns":5000000,"apply_latency_ns":5000000},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":18.745000000000001}]},{"osd":2,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":16.329000000000001}]},{"osd":3,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":8.673}]},{"osd":4,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":18.344000000000001}]},{"osd":5,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":22.225999999999999}]},{"osd":6,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":21.404}]},{"osd":7,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":9.5229999999999997}]}]},{"osd":0,"up_from":14,"seq":60129542151,"num_pgs":2,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":27244,"kb_used_data":216,"kb_used_omap":1,"kb_used_meta":27006,"kb_avail":94344596,"statfs":{"total":96636764160,"available":96608866304,"internally_reserved":0,"allocated":221184,"data_stored":76559,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1593,"internal_metadata":27654599},"hb_peers":[1,2,3,4,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[{"osd":1,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":8.3710000000000004}]},{"osd":2,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":7.4710000000000001}]},{"osd":3,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":43.481999999999999}]},{"osd":4,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":11.834}]},{"osd":5,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":7.0220000000000002}]},{"osd":6,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":9.5090000000000003}]},{"osd":7,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":16.553999999999998}]}]}],"pool_statfs":[{"poolid":1,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":401408,"data_stored":397840,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":5,"total":0,"available":0,"internally_reserved":0,"allocated":401408,"data_stored":397840,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":4,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":5,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":6,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0}]}} 2024-06-03T08:36:36.526 INFO:tasks.ceph.ceph_manager.ceph:clean! 2024-06-03T08:36:36.526 INFO:tasks.ceph:Waiting until ceph cluster ceph is healthy... 2024-06-03T08:36:36.526 INFO:tasks.ceph.ceph_manager.ceph:wait_until_healthy 2024-06-03T08:36:36.526 DEBUG:teuthology.orchestra.run.smithi088:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph health --format=json 2024-06-03T08:36:36.623 INFO:teuthology.orchestra.run.smithi088.stderr:2024-06-03T08:36:36.623+0000 7f3c61be9640 10 monclient: get_monmap_and_config 2024-06-03T08:36:36.624 INFO:teuthology.orchestra.run.smithi088.stderr:2024-06-03T08:36:36.623+0000 7f3c61be9640 10 monclient: build_initial_monmap 2024-06-03T08:36:36.624 INFO:teuthology.orchestra.run.smithi088.stderr:2024-06-03T08:36:36.623+0000 7f3c61be9640 10 monclient: monmap: 2024-06-03T08:36:36.624 INFO:teuthology.orchestra.run.smithi088.stderr:epoch 0 2024-06-03T08:36:36.624 INFO:teuthology.orchestra.run.smithi088.stderr:fsid ec3c3fdd-3ac6-4d7d-9412-683130ed8b04 2024-06-03T08:36:36.624 INFO:teuthology.orchestra.run.smithi088.stderr:last_changed 2024-06-03T08:36:36.624543+0000 2024-06-03T08:36:36.624 INFO:teuthology.orchestra.run.smithi088.stderr:created 2024-06-03T08:36:36.624543+0000 2024-06-03T08:36:36.624 INFO:teuthology.orchestra.run.smithi088.stderr:min_mon_release 0 (unknown) 2024-06-03T08:36:36.624 INFO:teuthology.orchestra.run.smithi088.stderr:election_strategy: 3 2024-06-03T08:36:36.624 INFO:teuthology.orchestra.run.smithi088.stderr:0: v1:172.21.15.88:6789/0 mon.noname-a 2024-06-03T08:36:36.624 INFO:teuthology.orchestra.run.smithi088.stderr:1: v1:172.21.15.192:6789/0 mon.noname-c 2024-06-03T08:36:36.625 INFO:teuthology.orchestra.run.smithi088.stderr:2: v1:172.21.15.88:6790/0 mon.noname-b 2024-06-03T08:36:36.625 INFO:teuthology.orchestra.run.smithi088.stderr: 2024-06-03T08:36:36.626 INFO:teuthology.orchestra.run.smithi088.stderr:2024-06-03T08:36:36.624+0000 7f3c61be9640 1 Processor -- start 2024-06-03T08:36:36.626 INFO:teuthology.orchestra.run.smithi088.stderr:2024-06-03T08:36:36.624+0000 7f3c61be9640 1 -- start start 2024-06-03T08:36:36.626 INFO:teuthology.orchestra.run.smithi088.stderr:2024-06-03T08:36:36.624+0000 7f3c61be9640 10 monclient: init 2024-06-03T08:36:36.626 INFO:teuthology.orchestra.run.smithi088.stderr:2024-06-03T08:36:36.624+0000 7f3c61be9640 10 monclient: _reopen_session rank -1 2024-06-03T08:36:36.626 INFO:teuthology.orchestra.run.smithi088.stderr:2024-06-03T08:36:36.624+0000 7f3c61be9640 10 monclient: _add_conns ranks=[1,0,2] 2024-06-03T08:36:36.626 INFO:teuthology.orchestra.run.smithi088.stderr:2024-06-03T08:36:36.624+0000 7f3c61be9640 10 monclient(hunting): picked mon.noname-c con 0x7f3c5c0adab0 addr v1:172.21.15.192:6789/0 2024-06-03T08:36:36.626 INFO:teuthology.orchestra.run.smithi088.stderr:2024-06-03T08:36:36.624+0000 7f3c61be9640 10 monclient(hunting): picked mon.noname-a con 0x7f3c5c06edd0 addr v1:172.21.15.88:6789/0 2024-06-03T08:36:36.626 INFO:teuthology.orchestra.run.smithi088.stderr:2024-06-03T08:36:36.624+0000 7f3c61be9640 10 monclient(hunting): picked mon.noname-b con 0x7f3c5c1416d0 addr v1:172.21.15.88:6790/0 2024-06-03T08:36:36.626 INFO:teuthology.orchestra.run.smithi088.stderr:2024-06-03T08:36:36.624+0000 7f3c61be9640 1 -- --> v1:172.21.15.88:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f3c5c070840 con 0x7f3c5c06edd0 2024-06-03T08:36:36.627 INFO:teuthology.orchestra.run.smithi088.stderr:2024-06-03T08:36:36.624+0000 7f3c61be9640 1 -- --> v1:172.21.15.192:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f3c5c070b20 con 0x7f3c5c0adab0 2024-06-03T08:36:36.627 INFO:teuthology.orchestra.run.smithi088.stderr:2024-06-03T08:36:36.624+0000 7f3c61be9640 1 -- --> v1:172.21.15.88:6790/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f3c5c071220 con 0x7f3c5c1416d0 2024-06-03T08:36:36.627 INFO:teuthology.orchestra.run.smithi088.stderr:2024-06-03T08:36:36.624+0000 7f3c61be9640 10 monclient(hunting): _renew_subs 2024-06-03T08:36:36.627 INFO:teuthology.orchestra.run.smithi088.stderr:2024-06-03T08:36:36.624+0000 7f3c61be9640 10 monclient(hunting): authenticate will time out at 1252.434847s 2024-06-03T08:36:36.627 INFO:teuthology.orchestra.run.smithi088.stderr:2024-06-03T08:36:36.625+0000 7f3c5b7fe640 1 --1- >> v1:172.21.15.192:6789/0 conn(0x7f3c5c0adab0 0x7f3c5c071770 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.192:6789/0 says I am v1:172.21.15.88:52762/0 (socket says 172.21.15.88:52762) 2024-06-03T08:36:36.627 INFO:teuthology.orchestra.run.smithi088.stderr:2024-06-03T08:36:36.625+0000 7f3c5b7fe640 1 -- 172.21.15.88:0/3589097677 learned_addr learned my addr 172.21.15.88:0/3589097677 (peer_addr_for_me v1:172.21.15.88:0/0) 2024-06-03T08:36:36.628 INFO:teuthology.orchestra.run.smithi088.stderr:2024-06-03T08:36:36.627+0000 7f3c5a7fc640 1 -- 172.21.15.88:0/3589097677 <== mon.2 v1:172.21.15.88:6790/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 3856625693 0 0) 0x7f3c5c071220 con 0x7f3c5c1416d0 2024-06-03T08:36:36.628 INFO:teuthology.orchestra.run.smithi088.stderr:2024-06-03T08:36:36.627+0000 7f3c5a7fc640 10 monclient(hunting): _negotiate 2024-06-03T08:36:36.628 INFO:teuthology.orchestra.run.smithi088.stderr:2024-06-03T08:36:36.627+0000 7f3c5a7fc640 10 monclient(hunting): _init_auth method 2 2024-06-03T08:36:36.628 INFO:teuthology.orchestra.run.smithi088.stderr:2024-06-03T08:36:36.627+0000 7f3c5a7fc640 10 monclient(hunting): _init_auth creating new auth 2024-06-03T08:36:36.628 INFO:teuthology.orchestra.run.smithi088.stderr:2024-06-03T08:36:36.627+0000 7f3c5a7fc640 10 monclient(hunting): my global_id is 4403 2024-06-03T08:36:36.628 INFO:teuthology.orchestra.run.smithi088.stderr:2024-06-03T08:36:36.627+0000 7f3c5a7fc640 1 -- 172.21.15.88:0/3589097677 --> v1:172.21.15.88:6790/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f3c48003520 con 0x7f3c5c1416d0 2024-06-03T08:36:36.628 INFO:teuthology.orchestra.run.smithi088.stderr:2024-06-03T08:36:36.627+0000 7f3c5a7fc640 1 -- 172.21.15.88:0/3589097677 <== mon.1 v1:172.21.15.192:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 3882650805 0 0) 0x7f3c5c070b20 con 0x7f3c5c0adab0 2024-06-03T08:36:36.629 INFO:teuthology.orchestra.run.smithi088.stderr:2024-06-03T08:36:36.627+0000 7f3c5a7fc640 10 monclient(hunting): _negotiate 2024-06-03T08:36:36.629 INFO:teuthology.orchestra.run.smithi088.stderr:2024-06-03T08:36:36.627+0000 7f3c5a7fc640 10 monclient(hunting): _init_auth method 2 2024-06-03T08:36:36.629 INFO:teuthology.orchestra.run.smithi088.stderr:2024-06-03T08:36:36.627+0000 7f3c5a7fc640 10 monclient(hunting): _init_auth creating new auth 2024-06-03T08:36:36.629 INFO:teuthology.orchestra.run.smithi088.stderr:2024-06-03T08:36:36.627+0000 7f3c5a7fc640 10 monclient(hunting): my global_id is 4399 2024-06-03T08:36:36.629 INFO:teuthology.orchestra.run.smithi088.stderr:2024-06-03T08:36:36.627+0000 7f3c5a7fc640 1 -- 172.21.15.88:0/3589097677 --> v1:172.21.15.192:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f3c5c071220 con 0x7f3c5c0adab0 2024-06-03T08:36:36.629 INFO:teuthology.orchestra.run.smithi088.stderr:2024-06-03T08:36:36.628+0000 7f3c5a7fc640 1 -- 172.21.15.88:0/3589097677 <== mon.0 v1:172.21.15.88:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 862348135 0 0) 0x7f3c5c070840 con 0x7f3c5c06edd0 2024-06-03T08:36:36.629 INFO:teuthology.orchestra.run.smithi088.stderr:2024-06-03T08:36:36.628+0000 7f3c5a7fc640 10 monclient(hunting): _negotiate 2024-06-03T08:36:36.629 INFO:teuthology.orchestra.run.smithi088.stderr:2024-06-03T08:36:36.628+0000 7f3c5a7fc640 10 monclient(hunting): _init_auth method 2 2024-06-03T08:36:36.629 INFO:teuthology.orchestra.run.smithi088.stderr:2024-06-03T08:36:36.628+0000 7f3c5a7fc640 10 monclient(hunting): _init_auth creating new auth 2024-06-03T08:36:36.629 INFO:teuthology.orchestra.run.smithi088.stderr:2024-06-03T08:36:36.628+0000 7f3c5a7fc640 10 monclient(hunting): my global_id is 4392 2024-06-03T08:36:36.629 INFO:teuthology.orchestra.run.smithi088.stderr:2024-06-03T08:36:36.628+0000 7f3c5a7fc640 1 -- 172.21.15.88:0/3589097677 --> v1:172.21.15.88:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f3c5c070b20 con 0x7f3c5c06edd0 2024-06-03T08:36:36.631 INFO:teuthology.orchestra.run.smithi088.stderr:2024-06-03T08:36:36.631+0000 7f3c5a7fc640 1 -- 172.21.15.88:0/3589097677 <== mon.1 v1:172.21.15.192:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 214+0+0 (unknown 1160427518 0 0) 0x7f3c5c071220 con 0x7f3c5c0adab0 2024-06-03T08:36:36.632 INFO:teuthology.orchestra.run.smithi088.stderr:2024-06-03T08:36:36.631+0000 7f3c5a7fc640 1 -- 172.21.15.88:0/3589097677 >> v1:172.21.15.88:6790/0 conn(0x7f3c5c1416d0 legacy=0x7f3c5c149a70 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-06-03T08:36:36.632 INFO:teuthology.orchestra.run.smithi088.stderr:2024-06-03T08:36:36.631+0000 7f3c5a7fc640 1 -- 172.21.15.88:0/3589097677 >> v1:172.21.15.88:6789/0 conn(0x7f3c5c06edd0 legacy=0x7f3c5c13b4c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-06-03T08:36:36.632 INFO:teuthology.orchestra.run.smithi088.stderr:2024-06-03T08:36:36.631+0000 7f3c5a7fc640 10 monclient: _finish_hunting 0 2024-06-03T08:36:36.632 INFO:teuthology.orchestra.run.smithi088.stderr:2024-06-03T08:36:36.631+0000 7f3c5a7fc640 1 monclient: found mon.noname-c 2024-06-03T08:36:36.632 INFO:teuthology.orchestra.run.smithi088.stderr:2024-06-03T08:36:36.631+0000 7f3c5a7fc640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-06-03T08:36:36.632 INFO:teuthology.orchestra.run.smithi088.stderr:2024-06-03T08:36:36.631+0000 7f3c5a7fc640 10 monclient: _send_mon_message to mon.noname-c at v1:172.21.15.192:6789/0 2024-06-03T08:36:36.632 INFO:teuthology.orchestra.run.smithi088.stderr:2024-06-03T08:36:36.631+0000 7f3c5a7fc640 1 -- 172.21.15.88:0/3589097677 --> v1:172.21.15.192:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f3c5c071e80 con 0x7f3c5c0adab0 2024-06-03T08:36:36.632 INFO:teuthology.orchestra.run.smithi088.stderr:2024-06-03T08:36:36.631+0000 7f3c5a7fc640 10 monclient: _finish_auth 0 2024-06-03T08:36:36.632 INFO:teuthology.orchestra.run.smithi088.stderr:2024-06-03T08:36:36.631+0000 7f3c5a7fc640 10 monclient: _check_auth_tickets 2024-06-03T08:36:36.632 INFO:teuthology.orchestra.run.smithi088.stderr:2024-06-03T08:36:36.631+0000 7f3c5a7fc640 20 monclient: _check_auth_rotating not needed by client.admin 2024-06-03T08:36:36.632 INFO:teuthology.orchestra.run.smithi088.stderr:2024-06-03T08:36:36.631+0000 7f3c5a7fc640 1 -- 172.21.15.88:0/3589097677 <== mon.1 v1:172.21.15.192:6789/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 2234374065 0 0) 0x7f3c4c0015f0 con 0x7f3c5c0adab0 2024-06-03T08:36:36.632 INFO:teuthology.orchestra.run.smithi088.stderr:2024-06-03T08:36:36.631+0000 7f3c5a7fc640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-06-03T08:36:36.633 INFO:teuthology.orchestra.run.smithi088.stderr:2024-06-03T08:36:36.631+0000 7f3c5a7fc640 10 monclient: got monmap 1 from mon.noname-c (according to old e1) 2024-06-03T08:36:36.633 INFO:teuthology.orchestra.run.smithi088.stderr:2024-06-03T08:36:36.631+0000 7f3c5a7fc640 10 monclient: dump: 2024-06-03T08:36:36.633 INFO:teuthology.orchestra.run.smithi088.stderr:epoch 1 2024-06-03T08:36:36.633 INFO:teuthology.orchestra.run.smithi088.stderr:fsid ec3c3fdd-3ac6-4d7d-9412-683130ed8b04 2024-06-03T08:36:36.633 INFO:teuthology.orchestra.run.smithi088.stderr:last_changed 2024-06-03T08:34:31.885125+0000 2024-06-03T08:36:36.633 INFO:teuthology.orchestra.run.smithi088.stderr:created 2024-06-03T08:34:31.885125+0000 2024-06-03T08:36:36.633 INFO:teuthology.orchestra.run.smithi088.stderr:min_mon_release 18 (reef) 2024-06-03T08:36:36.633 INFO:teuthology.orchestra.run.smithi088.stderr:election_strategy: 3 2024-06-03T08:36:36.633 INFO:teuthology.orchestra.run.smithi088.stderr:0: v1:172.21.15.88:6789/0 mon.a 2024-06-03T08:36:36.633 INFO:teuthology.orchestra.run.smithi088.stderr:1: v1:172.21.15.192:6789/0 mon.b 2024-06-03T08:36:36.633 INFO:teuthology.orchestra.run.smithi088.stderr:2: v1:172.21.15.88:6790/0 mon.c 2024-06-03T08:36:36.633 INFO:teuthology.orchestra.run.smithi088.stderr: 2024-06-03T08:36:36.633 INFO:teuthology.orchestra.run.smithi088.stderr:2024-06-03T08:36:36.631+0000 7f3c61be9640 5 monclient: authenticate success, global_id 4399 2024-06-03T08:36:36.634 INFO:teuthology.orchestra.run.smithi088.stderr:2024-06-03T08:36:36.631+0000 7f3c61be9640 20 monclient: get_monmap_and_config waiting for monmap|config 2024-06-03T08:36:36.636 INFO:teuthology.orchestra.run.smithi088.stderr:2024-06-03T08:36:36.633+0000 7f3c5a7fc640 1 -- 172.21.15.88:0/3589097677 <== mon.1 v1:172.21.15.192:6789/0 4 ==== config(0 keys) v1 ==== 4+0+0 (unknown 0 0 0) 0x7f3c4c001810 con 0x7f3c5c0adab0 2024-06-03T08:36:36.636 INFO:teuthology.orchestra.run.smithi088.stderr:2024-06-03T08:36:36.633+0000 7f3c5a7fc640 10 monclient: handle_config config(0 keys) v1 2024-06-03T08:36:36.637 INFO:teuthology.orchestra.run.smithi088.stderr:2024-06-03T08:36:36.633+0000 7f3c5a7fc640 1 -- 172.21.15.88:0/3589097677 <== mon.1 v1:172.21.15.192:6789/0 5 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 2234374065 0 0) 0x7f3c4c001be0 con 0x7f3c5c0adab0 2024-06-03T08:36:36.637 INFO:teuthology.orchestra.run.smithi088.stderr:2024-06-03T08:36:36.633+0000 7f3c61be9640 10 monclient: get_monmap_and_config success 2024-06-03T08:36:36.637 INFO:teuthology.orchestra.run.smithi088.stderr:2024-06-03T08:36:36.633+0000 7f3c61be9640 4 set_mon_vals no callback set 2024-06-03T08:36:36.637 INFO:teuthology.orchestra.run.smithi088.stderr:2024-06-03T08:36:36.633+0000 7f3c5a7fc640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-06-03T08:36:36.637 INFO:teuthology.orchestra.run.smithi088.stderr:2024-06-03T08:36:36.633+0000 7f3c5a7fc640 10 monclient: got monmap 1 from mon.b (according to old e1) 2024-06-03T08:36:36.637 INFO:teuthology.orchestra.run.smithi088.stderr:2024-06-03T08:36:36.633+0000 7f3c5a7fc640 10 monclient: dump: 2024-06-03T08:36:36.637 INFO:teuthology.orchestra.run.smithi088.stderr:epoch 1 2024-06-03T08:36:36.637 INFO:teuthology.orchestra.run.smithi088.stderr:fsid ec3c3fdd-3ac6-4d7d-9412-683130ed8b04 2024-06-03T08:36:36.637 INFO:teuthology.orchestra.run.smithi088.stderr:last_changed 2024-06-03T08:34:31.885125+0000 2024-06-03T08:36:36.637 INFO:teuthology.orchestra.run.smithi088.stderr:created 2024-06-03T08:34:31.885125+0000 2024-06-03T08:36:36.637 INFO:teuthology.orchestra.run.smithi088.stderr:min_mon_release 18 (reef) 2024-06-03T08:36:36.637 INFO:teuthology.orchestra.run.smithi088.stderr:election_strategy: 3 2024-06-03T08:36:36.638 INFO:teuthology.orchestra.run.smithi088.stderr:0: v1:172.21.15.88:6789/0 mon.a 2024-06-03T08:36:36.638 INFO:teuthology.orchestra.run.smithi088.stderr:1: v1:172.21.15.192:6789/0 mon.b 2024-06-03T08:36:36.638 INFO:teuthology.orchestra.run.smithi088.stderr:2: v1:172.21.15.88:6790/0 mon.c 2024-06-03T08:36:36.638 INFO:teuthology.orchestra.run.smithi088.stderr: 2024-06-03T08:36:36.638 INFO:teuthology.orchestra.run.smithi088.stderr:2024-06-03T08:36:36.633+0000 7f3c61be9640 10 monclient: shutdown 2024-06-03T08:36:36.638 INFO:teuthology.orchestra.run.smithi088.stderr:2024-06-03T08:36:36.633+0000 7f3c61be9640 20 monclient: shutdown discarding 0 pending message(s) 2024-06-03T08:36:36.638 INFO:teuthology.orchestra.run.smithi088.stderr:2024-06-03T08:36:36.633+0000 7f3c61be9640 1 -- 172.21.15.88:0/3589097677 >> v1:172.21.15.192:6789/0 conn(0x7f3c5c0adab0 legacy=0x7f3c5c071770 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-06-03T08:36:36.638 INFO:teuthology.orchestra.run.smithi088.stderr:2024-06-03T08:36:36.633+0000 7f3c61be9640 1 -- 172.21.15.88:0/3589097677 shutdown_connections 2024-06-03T08:36:36.638 INFO:teuthology.orchestra.run.smithi088.stderr:2024-06-03T08:36:36.633+0000 7f3c61be9640 1 -- 172.21.15.88:0/3589097677 >> 172.21.15.88:0/3589097677 conn(0x7f3c5c0b2e40 msgr2=0x7f3c5c0b3250 unknown :-1 s=STATE_NONE l=0).mark_down 2024-06-03T08:36:36.638 INFO:teuthology.orchestra.run.smithi088.stderr:2024-06-03T08:36:36.633+0000 7f3c61be9640 1 -- 172.21.15.88:0/3589097677 shutdown_connections 2024-06-03T08:36:36.638 INFO:teuthology.orchestra.run.smithi088.stderr:2024-06-03T08:36:36.633+0000 7f3c61be9640 1 -- 172.21.15.88:0/3589097677 wait complete. 2024-06-03T08:36:36.639 INFO:teuthology.orchestra.run.smithi088.stderr:2024-06-03T08:36:36.634+0000 7f3c61be9640 10 monclient: build_initial_monmap 2024-06-03T08:36:36.639 INFO:teuthology.orchestra.run.smithi088.stderr:2024-06-03T08:36:36.634+0000 7f3c61be9640 10 monclient: monmap: 2024-06-03T08:36:36.639 INFO:teuthology.orchestra.run.smithi088.stderr:epoch 0 2024-06-03T08:36:36.639 INFO:teuthology.orchestra.run.smithi088.stderr:fsid 00000000-0000-0000-0000-000000000000 2024-06-03T08:36:36.639 INFO:teuthology.orchestra.run.smithi088.stderr:last_changed 0.000000 2024-06-03T08:36:36.639 INFO:teuthology.orchestra.run.smithi088.stderr:created 0.000000 2024-06-03T08:36:36.639 INFO:teuthology.orchestra.run.smithi088.stderr:min_mon_release 0 (unknown) 2024-06-03T08:36:36.639 INFO:teuthology.orchestra.run.smithi088.stderr:election_strategy: 1 2024-06-03T08:36:36.639 INFO:teuthology.orchestra.run.smithi088.stderr:0: v1:172.21.15.88:6789/0 mon.noname-a 2024-06-03T08:36:36.639 INFO:teuthology.orchestra.run.smithi088.stderr:1: v1:172.21.15.192:6789/0 mon.noname-b 2024-06-03T08:36:36.639 INFO:teuthology.orchestra.run.smithi088.stderr:2: v1:172.21.15.88:6790/0 mon.noname-c 2024-06-03T08:36:36.639 INFO:teuthology.orchestra.run.smithi088.stderr: 2024-06-03T08:36:36.639 INFO:teuthology.orchestra.run.smithi088.stderr:2024-06-03T08:36:36.634+0000 7f3c61be9640 1 librados: starting msgr at 2024-06-03T08:36:36.640 INFO:teuthology.orchestra.run.smithi088.stderr:2024-06-03T08:36:36.634+0000 7f3c61be9640 1 librados: starting objecter 2024-06-03T08:36:36.640 INFO:teuthology.orchestra.run.smithi088.stderr:2024-06-03T08:36:36.634+0000 7f3c61be9640 1 Processor -- start 2024-06-03T08:36:36.640 INFO:teuthology.orchestra.run.smithi088.stderr:2024-06-03T08:36:36.635+0000 7f3c61be9640 1 -- start start 2024-06-03T08:36:36.640 INFO:teuthology.orchestra.run.smithi088.stderr:2024-06-03T08:36:36.635+0000 7f3c61be9640 1 librados: setting wanted keys 2024-06-03T08:36:36.640 INFO:teuthology.orchestra.run.smithi088.stderr:2024-06-03T08:36:36.635+0000 7f3c61be9640 1 librados: calling monclient init 2024-06-03T08:36:36.640 INFO:teuthology.orchestra.run.smithi088.stderr:2024-06-03T08:36:36.635+0000 7f3c61be9640 10 monclient: init 2024-06-03T08:36:36.640 INFO:teuthology.orchestra.run.smithi088.stderr:2024-06-03T08:36:36.635+0000 7f3c61be9640 10 monclient: _reopen_session rank -1 2024-06-03T08:36:36.640 INFO:teuthology.orchestra.run.smithi088.stderr:2024-06-03T08:36:36.635+0000 7f3c61be9640 10 monclient: _add_conns ranks=[2,1,0] 2024-06-03T08:36:36.640 INFO:teuthology.orchestra.run.smithi088.stderr:2024-06-03T08:36:36.635+0000 7f3c61be9640 10 monclient(hunting): picked mon.noname-c con 0x7f3c5c06edd0 addr v1:172.21.15.88:6790/0 2024-06-03T08:36:36.640 INFO:teuthology.orchestra.run.smithi088.stderr:2024-06-03T08:36:36.635+0000 7f3c61be9640 10 monclient(hunting): picked mon.noname-b con 0x7f3c5c0adab0 addr v1:172.21.15.192:6789/0 2024-06-03T08:36:36.640 INFO:teuthology.orchestra.run.smithi088.stderr:2024-06-03T08:36:36.635+0000 7f3c61be9640 10 monclient(hunting): picked mon.noname-a con 0x7f3c5c1416d0 addr v1:172.21.15.88:6789/0 2024-06-03T08:36:36.640 INFO:teuthology.orchestra.run.smithi088.stderr:2024-06-03T08:36:36.635+0000 7f3c61be9640 1 -- --> v1:172.21.15.88:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f3c5c135d60 con 0x7f3c5c1416d0 2024-06-03T08:36:36.640 INFO:teuthology.orchestra.run.smithi088.stderr:2024-06-03T08:36:36.635+0000 7f3c61be9640 1 -- --> v1:172.21.15.192:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f3c5c1cfbd0 con 0x7f3c5c0adab0 2024-06-03T08:36:36.640 INFO:teuthology.orchestra.run.smithi088.stderr:2024-06-03T08:36:36.635+0000 7f3c61be9640 1 -- --> v1:172.21.15.88:6790/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f3c5c13d500 con 0x7f3c5c06edd0 2024-06-03T08:36:36.640 INFO:teuthology.orchestra.run.smithi088.stderr:2024-06-03T08:36:36.635+0000 7f3c61be9640 10 monclient(hunting): _renew_subs 2024-06-03T08:36:36.641 INFO:teuthology.orchestra.run.smithi088.stderr:2024-06-03T08:36:36.635+0000 7f3c61be9640 10 monclient(hunting): authenticate will time out at 1252.445637s 2024-06-03T08:36:36.641 INFO:teuthology.orchestra.run.smithi088.stderr:2024-06-03T08:36:36.636+0000 7f3c5affd640 1 --1- >> v1:172.21.15.192:6789/0 conn(0x7f3c5c0adab0 0x7f3c5c1d2880 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=1).handle_server_banner_and_identify peer v1:172.21.15.192:6789/0 says I am v1:172.21.15.88:52764/0 (socket says 172.21.15.88:52764) 2024-06-03T08:36:36.641 INFO:teuthology.orchestra.run.smithi088.stderr:2024-06-03T08:36:36.636+0000 7f3c5affd640 1 -- 172.21.15.88:0/756374521 learned_addr learned my addr 172.21.15.88:0/756374521 (peer_addr_for_me v1:172.21.15.88:0/0) 2024-06-03T08:36:36.641 INFO:teuthology.orchestra.run.smithi088.stderr:2024-06-03T08:36:36.636+0000 7f3c53fff640 10 client.?.objecter ms_handle_connect 0x7f3c5c0adab0 2024-06-03T08:36:36.641 INFO:teuthology.orchestra.run.smithi088.stderr:2024-06-03T08:36:36.636+0000 7f3c53fff640 10 client.?.objecter resend_mon_ops 2024-06-03T08:36:36.641 INFO:teuthology.orchestra.run.smithi088.stderr:2024-06-03T08:36:36.636+0000 7f3c53fff640 10 client.?.objecter ms_handle_connect 0x7f3c5c1416d0 2024-06-03T08:36:36.641 INFO:teuthology.orchestra.run.smithi088.stderr:2024-06-03T08:36:36.636+0000 7f3c53fff640 10 client.?.objecter resend_mon_ops 2024-06-03T08:36:36.641 INFO:teuthology.orchestra.run.smithi088.stderr:2024-06-03T08:36:36.636+0000 7f3c53fff640 10 client.?.objecter ms_handle_connect 0x7f3c5c06edd0 2024-06-03T08:36:36.641 INFO:teuthology.orchestra.run.smithi088.stderr:2024-06-03T08:36:36.636+0000 7f3c53fff640 10 client.?.objecter resend_mon_ops 2024-06-03T08:36:36.642 INFO:teuthology.orchestra.run.smithi088.stderr:2024-06-03T08:36:36.637+0000 7f3c53fff640 1 -- 172.21.15.88:0/756374521 <== mon.1 v1:172.21.15.192:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 3094224160 0 0) 0x7f3c5c1cfbd0 con 0x7f3c5c0adab0 2024-06-03T08:36:36.642 INFO:teuthology.orchestra.run.smithi088.stderr:2024-06-03T08:36:36.637+0000 7f3c53fff640 10 monclient(hunting): _negotiate 2024-06-03T08:36:36.642 INFO:teuthology.orchestra.run.smithi088.stderr:2024-06-03T08:36:36.637+0000 7f3c53fff640 10 monclient(hunting): _init_auth method 2 2024-06-03T08:36:36.642 INFO:teuthology.orchestra.run.smithi088.stderr:2024-06-03T08:36:36.637+0000 7f3c53fff640 10 monclient(hunting): _init_auth creating new auth 2024-06-03T08:36:36.642 INFO:teuthology.orchestra.run.smithi088.stderr:2024-06-03T08:36:36.637+0000 7f3c53fff640 10 monclient(hunting): my global_id is 4402 2024-06-03T08:36:36.642 INFO:teuthology.orchestra.run.smithi088.stderr:2024-06-03T08:36:36.637+0000 7f3c53fff640 1 -- 172.21.15.88:0/756374521 --> v1:172.21.15.192:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f3c3c004eb0 con 0x7f3c5c0adab0 2024-06-03T08:36:36.643 INFO:teuthology.orchestra.run.smithi088.stderr:2024-06-03T08:36:36.638+0000 7f3c53fff640 1 -- 172.21.15.88:0/756374521 <== mon.2 v1:172.21.15.88:6790/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 3131971590 0 0) 0x7f3c5c13d500 con 0x7f3c5c06edd0 2024-06-03T08:36:36.643 INFO:teuthology.orchestra.run.smithi088.stderr:2024-06-03T08:36:36.638+0000 7f3c53fff640 10 monclient(hunting): _negotiate 2024-06-03T08:36:36.643 INFO:teuthology.orchestra.run.smithi088.stderr:2024-06-03T08:36:36.638+0000 7f3c53fff640 10 monclient(hunting): _init_auth method 2 2024-06-03T08:36:36.643 INFO:teuthology.orchestra.run.smithi088.stderr:2024-06-03T08:36:36.638+0000 7f3c53fff640 10 monclient(hunting): _init_auth creating new auth 2024-06-03T08:36:36.643 INFO:teuthology.orchestra.run.smithi088.stderr:2024-06-03T08:36:36.638+0000 7f3c53fff640 10 monclient(hunting): my global_id is 4406 2024-06-03T08:36:36.643 INFO:teuthology.orchestra.run.smithi088.stderr:2024-06-03T08:36:36.638+0000 7f3c53fff640 1 -- 172.21.15.88:0/756374521 --> v1:172.21.15.88:6790/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f3c5c1cfbd0 con 0x7f3c5c06edd0 2024-06-03T08:36:36.643 INFO:teuthology.orchestra.run.smithi088.stderr:2024-06-03T08:36:36.638+0000 7f3c53fff640 1 -- 172.21.15.88:0/756374521 <== mon.0 v1:172.21.15.88:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 852013390 0 0) 0x7f3c5c135d60 con 0x7f3c5c1416d0 2024-06-03T08:36:36.643 INFO:teuthology.orchestra.run.smithi088.stderr:2024-06-03T08:36:36.638+0000 7f3c53fff640 10 monclient(hunting): _negotiate 2024-06-03T08:36:36.643 INFO:teuthology.orchestra.run.smithi088.stderr:2024-06-03T08:36:36.638+0000 7f3c53fff640 10 monclient(hunting): _init_auth method 2 2024-06-03T08:36:36.643 INFO:teuthology.orchestra.run.smithi088.stderr:2024-06-03T08:36:36.638+0000 7f3c53fff640 10 monclient(hunting): _init_auth creating new auth 2024-06-03T08:36:36.643 INFO:teuthology.orchestra.run.smithi088.stderr:2024-06-03T08:36:36.638+0000 7f3c53fff640 10 monclient(hunting): my global_id is 4395 2024-06-03T08:36:36.643 INFO:teuthology.orchestra.run.smithi088.stderr:2024-06-03T08:36:36.638+0000 7f3c53fff640 1 -- 172.21.15.88:0/756374521 --> v1:172.21.15.88:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f3c5c13d500 con 0x7f3c5c1416d0 2024-06-03T08:36:36.645 INFO:teuthology.orchestra.run.smithi088.stderr:2024-06-03T08:36:36.642+0000 7f3c53fff640 1 -- 172.21.15.88:0/756374521 <== mon.1 v1:172.21.15.192:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 764+0+0 (unknown 3638743351 0 0) 0x7f3c3c004eb0 con 0x7f3c5c0adab0 2024-06-03T08:36:36.645 INFO:teuthology.orchestra.run.smithi088.stderr:2024-06-03T08:36:36.643+0000 7f3c53fff640 1 -- 172.21.15.88:0/756374521 --> v1:172.21.15.192:6789/0 -- auth(proto 2 165 bytes epoch 0) v1 -- 0x7f3c5c135d60 con 0x7f3c5c0adab0 2024-06-03T08:36:36.645 INFO:teuthology.orchestra.run.smithi088.stderr:2024-06-03T08:36:36.643+0000 7f3c53fff640 1 -- 172.21.15.88:0/756374521 <== mon.1 v1:172.21.15.192:6789/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 2234374065 0 0) 0x7f3c44001970 con 0x7f3c5c0adab0 2024-06-03T08:36:36.645 INFO:teuthology.orchestra.run.smithi088.stderr:2024-06-03T08:36:36.643+0000 7f3c53fff640 10 monclient(hunting): handle_monmap mon_map magic: 0 v1 2024-06-03T08:36:36.645 INFO:teuthology.orchestra.run.smithi088.stderr:2024-06-03T08:36:36.643+0000 7f3c53fff640 10 monclient(hunting): got monmap 1 from mon.noname-b (according to old e1) 2024-06-03T08:36:36.645 INFO:teuthology.orchestra.run.smithi088.stderr:2024-06-03T08:36:36.643+0000 7f3c53fff640 10 monclient(hunting): dump: 2024-06-03T08:36:36.645 INFO:teuthology.orchestra.run.smithi088.stderr:epoch 1 2024-06-03T08:36:36.645 INFO:teuthology.orchestra.run.smithi088.stderr:fsid ec3c3fdd-3ac6-4d7d-9412-683130ed8b04 2024-06-03T08:36:36.645 INFO:teuthology.orchestra.run.smithi088.stderr:last_changed 2024-06-03T08:34:31.885125+0000 2024-06-03T08:36:36.645 INFO:teuthology.orchestra.run.smithi088.stderr:created 2024-06-03T08:34:31.885125+0000 2024-06-03T08:36:36.645 INFO:teuthology.orchestra.run.smithi088.stderr:min_mon_release 18 (reef) 2024-06-03T08:36:36.646 INFO:teuthology.orchestra.run.smithi088.stderr:election_strategy: 3 2024-06-03T08:36:36.646 INFO:teuthology.orchestra.run.smithi088.stderr:0: v1:172.21.15.88:6789/0 mon.a 2024-06-03T08:36:36.646 INFO:teuthology.orchestra.run.smithi088.stderr:1: v1:172.21.15.192:6789/0 mon.b 2024-06-03T08:36:36.646 INFO:teuthology.orchestra.run.smithi088.stderr:2: v1:172.21.15.88:6790/0 mon.c 2024-06-03T08:36:36.646 INFO:teuthology.orchestra.run.smithi088.stderr: 2024-06-03T08:36:36.646 INFO:teuthology.orchestra.run.smithi088.stderr:2024-06-03T08:36:36.643+0000 7f3c53fff640 10 monclient(hunting): _finish_auth 0 2024-06-03T08:36:36.646 INFO:teuthology.orchestra.run.smithi088.stderr:2024-06-03T08:36:36.643+0000 7f3c53fff640 1 -- 172.21.15.88:0/756374521 <== mon.2 v1:172.21.15.88:6790/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 764+0+0 (unknown 2516711262 0 0) 0x7f3c5c1cfbd0 con 0x7f3c5c06edd0 2024-06-03T08:36:36.646 INFO:teuthology.orchestra.run.smithi088.stderr:2024-06-03T08:36:36.643+0000 7f3c53fff640 1 -- 172.21.15.88:0/756374521 --> v1:172.21.15.88:6790/0 -- auth(proto 2 165 bytes epoch 0) v1 -- 0x7f3c3c004eb0 con 0x7f3c5c06edd0 2024-06-03T08:36:36.646 INFO:teuthology.orchestra.run.smithi088.stderr:2024-06-03T08:36:36.643+0000 7f3c53fff640 1 -- 172.21.15.88:0/756374521 <== mon.2 v1:172.21.15.88:6790/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 2234374065 0 0) 0x7f3c4c001b20 con 0x7f3c5c06edd0 2024-06-03T08:36:36.646 INFO:teuthology.orchestra.run.smithi088.stderr:2024-06-03T08:36:36.643+0000 7f3c53fff640 10 monclient(hunting): handle_monmap mon_map magic: 0 v1 2024-06-03T08:36:36.646 INFO:teuthology.orchestra.run.smithi088.stderr:2024-06-03T08:36:36.643+0000 7f3c53fff640 10 monclient(hunting): got monmap 1 from mon.c (according to old e1) 2024-06-03T08:36:36.647 INFO:teuthology.orchestra.run.smithi088.stderr:2024-06-03T08:36:36.643+0000 7f3c53fff640 10 monclient(hunting): dump: 2024-06-03T08:36:36.647 INFO:teuthology.orchestra.run.smithi088.stderr:epoch 1 2024-06-03T08:36:36.647 INFO:teuthology.orchestra.run.smithi088.stderr:fsid ec3c3fdd-3ac6-4d7d-9412-683130ed8b04 2024-06-03T08:36:36.647 INFO:teuthology.orchestra.run.smithi088.stderr:last_changed 2024-06-03T08:34:31.885125+0000 2024-06-03T08:36:36.647 INFO:teuthology.orchestra.run.smithi088.stderr:created 2024-06-03T08:34:31.885125+0000 2024-06-03T08:36:36.647 INFO:teuthology.orchestra.run.smithi088.stderr:min_mon_release 18 (reef) 2024-06-03T08:36:36.647 INFO:teuthology.orchestra.run.smithi088.stderr:election_strategy: 3 2024-06-03T08:36:36.647 INFO:teuthology.orchestra.run.smithi088.stderr:0: v1:172.21.15.88:6789/0 mon.a 2024-06-03T08:36:36.647 INFO:teuthology.orchestra.run.smithi088.stderr:1: v1:172.21.15.192:6789/0 mon.b 2024-06-03T08:36:36.647 INFO:teuthology.orchestra.run.smithi088.stderr:2: v1:172.21.15.88:6790/0 mon.c 2024-06-03T08:36:36.647 INFO:teuthology.orchestra.run.smithi088.stderr: 2024-06-03T08:36:36.647 INFO:teuthology.orchestra.run.smithi088.stderr:2024-06-03T08:36:36.643+0000 7f3c53fff640 1 -- 172.21.15.88:0/756374521 <== mon.0 v1:172.21.15.88:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 764+0+0 (unknown 1956574042 0 0) 0x7f3c5c13d500 con 0x7f3c5c1416d0 2024-06-03T08:36:36.648 INFO:teuthology.orchestra.run.smithi088.stderr:2024-06-03T08:36:36.643+0000 7f3c53fff640 1 -- 172.21.15.88:0/756374521 --> v1:172.21.15.88:6789/0 -- auth(proto 2 165 bytes epoch 0) v1 -- 0x7f3c5c1cfbd0 con 0x7f3c5c1416d0 2024-06-03T08:36:36.648 INFO:teuthology.orchestra.run.smithi088.stderr:2024-06-03T08:36:36.643+0000 7f3c53fff640 1 -- 172.21.15.88:0/756374521 <== mon.0 v1:172.21.15.88:6789/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 2234374065 0 0) 0x7f3c54001b50 con 0x7f3c5c1416d0 2024-06-03T08:36:36.648 INFO:teuthology.orchestra.run.smithi088.stderr:2024-06-03T08:36:36.643+0000 7f3c53fff640 10 monclient(hunting): handle_monmap mon_map magic: 0 v1 2024-06-03T08:36:36.648 INFO:teuthology.orchestra.run.smithi088.stderr:2024-06-03T08:36:36.644+0000 7f3c53fff640 10 monclient(hunting): got monmap 1 from mon.a (according to old e1) 2024-06-03T08:36:36.648 INFO:teuthology.orchestra.run.smithi088.stderr:2024-06-03T08:36:36.644+0000 7f3c53fff640 10 monclient(hunting): dump: 2024-06-03T08:36:36.648 INFO:teuthology.orchestra.run.smithi088.stderr:epoch 1 2024-06-03T08:36:36.648 INFO:teuthology.orchestra.run.smithi088.stderr:fsid ec3c3fdd-3ac6-4d7d-9412-683130ed8b04 2024-06-03T08:36:36.648 INFO:teuthology.orchestra.run.smithi088.stderr:last_changed 2024-06-03T08:34:31.885125+0000 2024-06-03T08:36:36.648 INFO:teuthology.orchestra.run.smithi088.stderr:created 2024-06-03T08:34:31.885125+0000 2024-06-03T08:36:36.648 INFO:teuthology.orchestra.run.smithi088.stderr:min_mon_release 18 (reef) 2024-06-03T08:36:36.648 INFO:teuthology.orchestra.run.smithi088.stderr:election_strategy: 3 2024-06-03T08:36:36.648 INFO:teuthology.orchestra.run.smithi088.stderr:0: v1:172.21.15.88:6789/0 mon.a 2024-06-03T08:36:36.648 INFO:teuthology.orchestra.run.smithi088.stderr:1: v1:172.21.15.192:6789/0 mon.b 2024-06-03T08:36:36.649 INFO:teuthology.orchestra.run.smithi088.stderr:2: v1:172.21.15.88:6790/0 mon.c 2024-06-03T08:36:36.649 INFO:teuthology.orchestra.run.smithi088.stderr: 2024-06-03T08:36:36.650 INFO:teuthology.orchestra.run.smithi088.stderr:2024-06-03T08:36:36.646+0000 7f3c53fff640 1 -- 172.21.15.88:0/756374521 <== mon.1 v1:172.21.15.192:6789/0 4 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 580+0+0 (unknown 268409323 0 0) 0x7f3c5c135d60 con 0x7f3c5c0adab0 2024-06-03T08:36:36.650 INFO:teuthology.orchestra.run.smithi088.stderr:2024-06-03T08:36:36.646+0000 7f3c53fff640 1 -- 172.21.15.88:0/756374521 >> v1:172.21.15.88:6790/0 conn(0x7f3c5c06edd0 legacy=0x7f3c5c1d2170 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-06-03T08:36:36.650 INFO:teuthology.orchestra.run.smithi088.stderr:2024-06-03T08:36:36.646+0000 7f3c53fff640 1 -- 172.21.15.88:0/756374521 >> v1:172.21.15.88:6789/0 conn(0x7f3c5c1416d0 legacy=0x7f3c5c1dcbb0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-06-03T08:36:36.650 INFO:teuthology.orchestra.run.smithi088.stderr:2024-06-03T08:36:36.646+0000 7f3c53fff640 10 monclient: _finish_hunting 0 2024-06-03T08:36:36.650 INFO:teuthology.orchestra.run.smithi088.stderr:2024-06-03T08:36:36.646+0000 7f3c53fff640 1 monclient: found mon.b 2024-06-03T08:36:36.650 INFO:teuthology.orchestra.run.smithi088.stderr:2024-06-03T08:36:36.646+0000 7f3c53fff640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-06-03T08:36:36.650 INFO:teuthology.orchestra.run.smithi088.stderr:2024-06-03T08:36:36.646+0000 7f3c53fff640 10 monclient: _send_mon_message to mon.b at v1:172.21.15.192:6789/0 2024-06-03T08:36:36.650 INFO:teuthology.orchestra.run.smithi088.stderr:2024-06-03T08:36:36.646+0000 7f3c53fff640 1 -- 172.21.15.88:0/756374521 --> v1:172.21.15.192:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f3c5c13d690 con 0x7f3c5c0adab0 2024-06-03T08:36:36.651 INFO:teuthology.orchestra.run.smithi088.stderr:2024-06-03T08:36:36.646+0000 7f3c53fff640 10 monclient: _finish_auth 0 2024-06-03T08:36:36.651 INFO:teuthology.orchestra.run.smithi088.stderr:2024-06-03T08:36:36.646+0000 7f3c53fff640 10 monclient: _check_auth_tickets 2024-06-03T08:36:36.651 INFO:teuthology.orchestra.run.smithi088.stderr:2024-06-03T08:36:36.646+0000 7f3c53fff640 20 monclient: _check_auth_rotating not needed by client.admin 2024-06-03T08:36:36.651 INFO:teuthology.orchestra.run.smithi088.stderr:2024-06-03T08:36:36.646+0000 7f3c61be9640 5 monclient: authenticate success, global_id 4402 2024-06-03T08:36:36.651 INFO:teuthology.orchestra.run.smithi088.stderr:2024-06-03T08:36:36.646+0000 7f3c61be9640 10 monclient: _renew_subs 2024-06-03T08:36:36.651 INFO:teuthology.orchestra.run.smithi088.stderr:2024-06-03T08:36:36.646+0000 7f3c61be9640 10 monclient: _send_mon_message to mon.b at v1:172.21.15.192:6789/0 2024-06-03T08:36:36.651 INFO:teuthology.orchestra.run.smithi088.stderr:2024-06-03T08:36:36.646+0000 7f3c61be9640 1 -- 172.21.15.88:0/756374521 --> v1:172.21.15.192:6789/0 -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f3c5c13c930 con 0x7f3c5c0adab0 2024-06-03T08:36:36.651 INFO:teuthology.orchestra.run.smithi088.stderr:2024-06-03T08:36:36.646+0000 7f3c61be9640 10 client.4402.objecter _maybe_request_map subscribing (onetime) to next osd map 2024-06-03T08:36:36.651 INFO:teuthology.orchestra.run.smithi088.stderr:2024-06-03T08:36:36.646+0000 7f3c61be9640 10 monclient: _renew_subs 2024-06-03T08:36:36.651 INFO:teuthology.orchestra.run.smithi088.stderr:2024-06-03T08:36:36.646+0000 7f3c61be9640 10 monclient: _send_mon_message to mon.b at v1:172.21.15.192:6789/0 2024-06-03T08:36:36.651 INFO:teuthology.orchestra.run.smithi088.stderr:2024-06-03T08:36:36.646+0000 7f3c61be9640 1 -- 172.21.15.88:0/756374521 --> v1:172.21.15.192:6789/0 -- mon_subscribe({osdmap=0}) v3 -- 0x7f3c5c1de2d0 con 0x7f3c5c0adab0 2024-06-03T08:36:36.651 INFO:teuthology.orchestra.run.smithi088.stderr:2024-06-03T08:36:36.646+0000 7f3c61be9640 1 librados: init done 2024-06-03T08:36:36.651 INFO:teuthology.orchestra.run.smithi088.stderr:2024-06-03T08:36:36.647+0000 7f3c61be9640 10 start_mon_command cmd=[{"prefix": "get_command_descriptions"}] 2024-06-03T08:36:36.651 INFO:teuthology.orchestra.run.smithi088.stderr:2024-06-03T08:36:36.647+0000 7f3c61be9640 10 monclient: _send_command 1 [{"prefix": "get_command_descriptions"}] 2024-06-03T08:36:36.652 INFO:teuthology.orchestra.run.smithi088.stderr:2024-06-03T08:36:36.647+0000 7f3c61be9640 10 monclient: _send_mon_message to mon.b at v1:172.21.15.192:6789/0 2024-06-03T08:36:36.652 INFO:teuthology.orchestra.run.smithi088.stderr:2024-06-03T08:36:36.647+0000 7f3c61be9640 1 -- 172.21.15.88:0/756374521 --> v1:172.21.15.192:6789/0 -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f3c5c070050 con 0x7f3c5c0adab0 2024-06-03T08:36:36.661 INFO:teuthology.orchestra.run.smithi088.stderr:2024-06-03T08:36:36.661+0000 7f3c53fff640 1 -- 172.21.15.88:0/756374521 <== mon.1 v1:172.21.15.192:6789/0 5 ==== config(0 keys) v1 ==== 4+0+0 (unknown 0 0 0) 0x7f3c44003610 con 0x7f3c5c0adab0 2024-06-03T08:36:36.662 INFO:teuthology.orchestra.run.smithi088.stderr:2024-06-03T08:36:36.661+0000 7f3c53fff640 10 monclient: handle_config config(0 keys) v1 2024-06-03T08:36:36.662 INFO:teuthology.orchestra.run.smithi088.stderr:2024-06-03T08:36:36.661+0000 7f3c53fff640 1 -- 172.21.15.88:0/756374521 <== mon.1 v1:172.21.15.192:6789/0 6 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 2234374065 0 0) 0x7f3c440039a0 con 0x7f3c5c0adab0 2024-06-03T08:36:36.662 INFO:teuthology.orchestra.run.smithi088.stderr:2024-06-03T08:36:36.661+0000 7f3c53fff640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-06-03T08:36:36.662 INFO:teuthology.orchestra.run.smithi088.stderr:2024-06-03T08:36:36.661+0000 7f3c53fff640 10 monclient: got monmap 1 from mon.b (according to old e1) 2024-06-03T08:36:36.662 INFO:teuthology.orchestra.run.smithi088.stderr:2024-06-03T08:36:36.661+0000 7f3c53fff640 10 monclient: dump: 2024-06-03T08:36:36.662 INFO:teuthology.orchestra.run.smithi088.stderr:epoch 1 2024-06-03T08:36:36.662 INFO:teuthology.orchestra.run.smithi088.stderr:fsid ec3c3fdd-3ac6-4d7d-9412-683130ed8b04 2024-06-03T08:36:36.662 INFO:teuthology.orchestra.run.smithi088.stderr:last_changed 2024-06-03T08:34:31.885125+0000 2024-06-03T08:36:36.662 INFO:teuthology.orchestra.run.smithi088.stderr:created 2024-06-03T08:34:31.885125+0000 2024-06-03T08:36:36.662 INFO:teuthology.orchestra.run.smithi088.stderr:min_mon_release 18 (reef) 2024-06-03T08:36:36.662 INFO:teuthology.orchestra.run.smithi088.stderr:election_strategy: 3 2024-06-03T08:36:36.662 INFO:teuthology.orchestra.run.smithi088.stderr:0: v1:172.21.15.88:6789/0 mon.a 2024-06-03T08:36:36.663 INFO:teuthology.orchestra.run.smithi088.stderr:1: v1:172.21.15.192:6789/0 mon.b 2024-06-03T08:36:36.663 INFO:teuthology.orchestra.run.smithi088.stderr:2: v1:172.21.15.88:6790/0 mon.c 2024-06-03T08:36:36.663 INFO:teuthology.orchestra.run.smithi088.stderr: 2024-06-03T08:36:36.663 INFO:teuthology.orchestra.run.smithi088.stderr:2024-06-03T08:36:36.661+0000 7f3c59ffb640 4 set_mon_vals no callback set 2024-06-03T08:36:36.663 INFO:teuthology.orchestra.run.smithi088.stderr:2024-06-03T08:36:36.662+0000 7f3c53fff640 1 -- 172.21.15.88:0/756374521 <== mon.1 v1:172.21.15.192:6789/0 7 ==== mgrmap(e 5) v1 ==== 89110+0+0 (unknown 873533973 0 0) 0x7f3c44003b00 con 0x7f3c5c0adab0 2024-06-03T08:36:36.663 INFO:teuthology.orchestra.run.smithi088.stderr:2024-06-03T08:36:36.663+0000 7f3c5b7fe640 10 monclient: get_auth_request con 0x7f3c3c06d470 auth_method 0 2024-06-03T08:36:36.663 INFO:teuthology.orchestra.run.smithi088.stderr:2024-06-03T08:36:36.663+0000 7f3c53fff640 10 client.4402.objecter ms_handle_connect 0x7f3c3c06d470 2024-06-03T08:36:36.665 INFO:teuthology.orchestra.run.smithi088.stderr:2024-06-03T08:36:36.665+0000 7f3c53fff640 1 -- 172.21.15.88:0/756374521 <== mon.1 v1:172.21.15.192:6789/0 8 ==== osd_map(26..26 src has 1..26) v4 ==== 4090+0+0 (unknown 3637450554 0 0) 0x7f3c44054110 con 0x7f3c5c0adab0 2024-06-03T08:36:36.665 INFO:teuthology.orchestra.run.smithi088.stderr:2024-06-03T08:36:36.665+0000 7f3c53fff640 10 client.4402.objecter ms_dispatch 0x7f3c5c000b90 osd_map(26..26 src has 1..26) v4 2024-06-03T08:36:36.665 INFO:teuthology.orchestra.run.smithi088.stderr:2024-06-03T08:36:36.665+0000 7f3c53fff640 3 client.4402.objecter handle_osd_map got epochs [26,26] > 0 2024-06-03T08:36:36.665 INFO:teuthology.orchestra.run.smithi088.stderr:2024-06-03T08:36:36.665+0000 7f3c53fff640 3 client.4402.objecter handle_osd_map decoding full epoch 26 2024-06-03T08:36:36.665 INFO:teuthology.orchestra.run.smithi088.stderr:2024-06-03T08:36:36.665+0000 7f3c53fff640 20 client.4402.objecter dump_active .. 0 homeless 2024-06-03T08:36:36.896 INFO:teuthology.orchestra.run.smithi088.stderr:2024-06-03T08:36:36.896+0000 7f3c53fff640 1 -- 172.21.15.88:0/756374521 <== mon.1 v1:172.21.15.192:6789/0 9 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+182498 (unknown 1092875540 0 3016760700) 0x7f3c4404fcc0 con 0x7f3c5c0adab0 2024-06-03T08:36:36.896 INFO:teuthology.orchestra.run.smithi088.stderr:2024-06-03T08:36:36.896+0000 7f3c53fff640 10 monclient: handle_mon_command_ack 1 [{"prefix": "get_command_descriptions"}] 2024-06-03T08:36:36.896 INFO:teuthology.orchestra.run.smithi088.stderr:2024-06-03T08:36:36.896+0000 7f3c53fff640 10 monclient: _finish_command 1 = system:0 2024-06-03T08:36:37.108 INFO:teuthology.orchestra.run.smithi088.stderr:2024-06-03T08:36:37.108+0000 7f3c61be9640 10 start_mon_command cmd=[{"prefix": "health", "format": "json"}] 2024-06-03T08:36:37.108 INFO:teuthology.orchestra.run.smithi088.stderr:2024-06-03T08:36:37.108+0000 7f3c61be9640 10 monclient: _send_command 2 [{"prefix": "health", "format": "json"}] 2024-06-03T08:36:37.108 INFO:teuthology.orchestra.run.smithi088.stderr:2024-06-03T08:36:37.108+0000 7f3c61be9640 10 monclient: _send_mon_message to mon.b at v1:172.21.15.192:6789/0 2024-06-03T08:36:37.108 INFO:teuthology.orchestra.run.smithi088.stderr:2024-06-03T08:36:37.108+0000 7f3c61be9640 1 -- 172.21.15.88:0/756374521 --> v1:172.21.15.192:6789/0 -- mon_command({"prefix": "health", "format": "json"} v 0) v1 -- 0x7f3c5c04ea60 con 0x7f3c5c0adab0 2024-06-03T08:36:37.113 INFO:teuthology.orchestra.run.smithi088.stderr:2024-06-03T08:36:37.113+0000 7f3c53fff640 1 -- 172.21.15.88:0/756374521 <== mon.1 v1:172.21.15.192:6789/0 10 ==== mon_command_ack([{"prefix": "health", "format": "json"}]=0 v0) v1 ==== 72+0+46 (unknown 3730786198 0 4185958460) 0x7f3c44051bf0 con 0x7f3c5c0adab0 2024-06-03T08:36:37.114 INFO:teuthology.orchestra.run.smithi088.stderr:2024-06-03T08:36:37.113+0000 7f3c53fff640 10 monclient: handle_mon_command_ack 2 [{"prefix": "health", "format": "json"}] 2024-06-03T08:36:37.114 INFO:teuthology.orchestra.run.smithi088.stderr:2024-06-03T08:36:37.113+0000 7f3c53fff640 10 monclient: _finish_command 2 = system:0 2024-06-03T08:36:37.114 INFO:teuthology.orchestra.run.smithi088.stdout: 2024-06-03T08:36:37.114 INFO:teuthology.orchestra.run.smithi088.stdout:{"status":"HEALTH_OK","checks":{},"mutes":[]} 2024-06-03T08:36:37.116 INFO:teuthology.orchestra.run.smithi088.stderr:2024-06-03T08:36:37.115+0000 7f3c61be9640 10 librados: watch_flush enter 2024-06-03T08:36:37.116 INFO:teuthology.orchestra.run.smithi088.stderr:2024-06-03T08:36:37.115+0000 7f3c61be9640 10 librados: watch_flush exit 2024-06-03T08:36:37.116 INFO:teuthology.orchestra.run.smithi088.stderr:2024-06-03T08:36:37.115+0000 7f3c61be9640 20 client.4402.objecter shutdown clearing up homeless session... 2024-06-03T08:36:37.116 INFO:teuthology.orchestra.run.smithi088.stderr:2024-06-03T08:36:37.115+0000 7f3c61be9640 10 client.4402.objecter successfully canceled tick 2024-06-03T08:36:37.116 INFO:teuthology.orchestra.run.smithi088.stderr:2024-06-03T08:36:37.115+0000 7f3c61be9640 1 -- 172.21.15.88:0/756374521 >> v1:172.21.15.192:6800/34229 conn(0x7f3c3c06d470 legacy=0x7f3c3c06f930 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-06-03T08:36:37.116 INFO:teuthology.orchestra.run.smithi088.stderr:2024-06-03T08:36:37.115+0000 7f3c61be9640 10 monclient: shutdown 2024-06-03T08:36:37.117 INFO:teuthology.orchestra.run.smithi088.stderr:2024-06-03T08:36:37.115+0000 7f3c61be9640 20 monclient: shutdown discarding 0 pending message(s) 2024-06-03T08:36:37.117 INFO:teuthology.orchestra.run.smithi088.stderr:2024-06-03T08:36:37.115+0000 7f3c61be9640 1 -- 172.21.15.88:0/756374521 >> v1:172.21.15.192:6789/0 conn(0x7f3c5c0adab0 legacy=0x7f3c5c1d2880 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-06-03T08:36:37.117 INFO:teuthology.orchestra.run.smithi088.stderr:2024-06-03T08:36:37.115+0000 7f3c61be9640 1 -- 172.21.15.88:0/756374521 shutdown_connections 2024-06-03T08:36:37.117 INFO:teuthology.orchestra.run.smithi088.stderr:2024-06-03T08:36:37.115+0000 7f3c61be9640 1 -- 172.21.15.88:0/756374521 >> 172.21.15.88:0/756374521 conn(0x7f3c5c0b2e40 msgr2=0x7f3c5c0737f0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-06-03T08:36:37.117 INFO:teuthology.orchestra.run.smithi088.stderr:2024-06-03T08:36:37.115+0000 7f3c61be9640 1 -- 172.21.15.88:0/756374521 shutdown_connections 2024-06-03T08:36:37.117 INFO:teuthology.orchestra.run.smithi088.stderr:2024-06-03T08:36:37.115+0000 7f3c61be9640 1 -- 172.21.15.88:0/756374521 wait complete. 2024-06-03T08:36:37.117 INFO:teuthology.orchestra.run.smithi088.stderr:2024-06-03T08:36:37.115+0000 7f3c61be9640 1 librados: shutdown 2024-06-03T08:36:37.127 INFO:tasks.ceph.ceph_manager.ceph:wait_until_healthy done 2024-06-03T08:36:37.127 INFO:teuthology.run_tasks:Running task workunit... 2024-06-03T08:36:37.141 INFO:tasks.workunit:Pulling workunits from ref f1abf795f4fc15bf2f30ac90c811a53ab21d5d65 2024-06-03T08:36:37.141 INFO:tasks.workunit:Making a separate scratch dir for every client... 2024-06-03T08:36:37.141 DEBUG:teuthology.orchestra.run.smithi088:> stat -- /home/ubuntu/cephtest/mnt.0 2024-06-03T08:36:37.156 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-03T08:36:37.157 INFO:teuthology.orchestra.run.smithi088.stderr:stat: cannot statx '/home/ubuntu/cephtest/mnt.0': No such file or directory 2024-06-03T08:36:37.157 DEBUG:teuthology.orchestra.run.smithi088:> mkdir -- /home/ubuntu/cephtest/mnt.0 2024-06-03T08:36:37.212 INFO:tasks.workunit:Created dir /home/ubuntu/cephtest/mnt.0 2024-06-03T08:36:37.212 DEBUG:teuthology.orchestra.run.smithi088:> cd -- /home/ubuntu/cephtest/mnt.0 && mkdir -- client.0 2024-06-03T08:36:37.267 INFO:tasks.workunit:timeout=6h 2024-06-03T08:36:37.267 INFO:tasks.workunit:cleanup=True 2024-06-03T08:36:37.268 DEBUG:teuthology.orchestra.run.smithi088:> rm -rf /home/ubuntu/cephtest/clone.client.0 && git clone https://git.ceph.com/ceph.git /home/ubuntu/cephtest/clone.client.0 && cd /home/ubuntu/cephtest/clone.client.0 && git checkout f1abf795f4fc15bf2f30ac90c811a53ab21d5d65 2024-06-03T08:36:37.323 INFO:tasks.workunit.client.0.smithi088.stderr:Cloning into '/home/ubuntu/cephtest/clone.client.0'... 2024-06-03T08:36:43.794 INFO:tasks.ceph.mon.a.smithi088.stderr:==00:00:01:39.119 34743== Warning: unimplemented fcntl command: 1036 2024-06-03T08:36:43.841 INFO:tasks.ceph.mon.a.smithi088.stderr:==00:00:01:39.167 34743== Warning: unimplemented fcntl command: 1036 2024-06-03T08:36:43.843 INFO:tasks.ceph.mon.b.smithi192.stderr:==00:00:01:39.146 34228== Warning: unimplemented fcntl command: 1036 2024-06-03T08:36:43.845 INFO:tasks.ceph.mon.b.smithi192.stderr:==00:00:01:39.148 34228== Warning: unimplemented fcntl command: 1036 2024-06-03T08:36:43.880 INFO:tasks.ceph.mon.a.smithi088.stderr:==00:00:01:39.206 34743== Warning: unimplemented fcntl command: 1036 2024-06-03T08:36:43.884 INFO:tasks.ceph.mon.b.smithi192.stderr:==00:00:01:39.186 34228== Warning: unimplemented fcntl command: 1036 2024-06-03T08:36:43.887 INFO:tasks.ceph.mon.c.smithi088.stderr:==00:00:01:39.213 34749== Warning: unimplemented fcntl command: 1036 2024-06-03T08:36:43.891 INFO:tasks.ceph.mon.c.smithi088.stderr:==00:00:01:39.216 34749== Warning: unimplemented fcntl command: 1036 2024-06-03T08:36:43.931 INFO:tasks.ceph.mon.c.smithi088.stderr:==00:00:01:39.256 34749== Warning: unimplemented fcntl command: 1036 2024-06-03T08:37:03.918 INFO:tasks.ceph.mon.a.smithi088.stderr:==00:00:01:59.243 34743== Warning: unimplemented fcntl command: 1036 2024-06-03T08:37:03.922 INFO:tasks.ceph.mon.a.smithi088.stderr:==00:00:01:59.247 34743== Warning: unimplemented fcntl command: 1036 2024-06-03T08:37:03.936 INFO:tasks.ceph.mon.a.smithi088.stderr:==00:00:01:59.262 34743== Warning: unimplemented fcntl command: 1036 2024-06-03T08:37:03.970 INFO:tasks.ceph.mon.b.smithi192.stderr:==00:00:01:59.272 34228== Warning: unimplemented fcntl command: 1036 2024-06-03T08:37:03.970 INFO:tasks.ceph.mon.c.smithi088.stderr:==00:00:01:59.295 34749== Warning: unimplemented fcntl command: 1036 2024-06-03T08:37:03.971 INFO:tasks.ceph.mon.c.smithi088.stderr:==00:00:01:59.297 34749== Warning: unimplemented fcntl command: 1036 2024-06-03T08:37:03.971 INFO:tasks.ceph.mon.b.smithi192.stderr:==00:00:01:59.273 34228== Warning: unimplemented fcntl command: 1036 2024-06-03T08:37:03.985 INFO:tasks.ceph.mon.b.smithi192.stderr:==00:00:01:59.287 34228== Warning: unimplemented fcntl command: 1036 2024-06-03T08:37:03.992 INFO:tasks.ceph.mon.c.smithi088.stderr:==00:00:01:59.317 34749== Warning: unimplemented fcntl command: 1036 2024-06-03T08:37:23.934 INFO:tasks.ceph.mon.a.smithi088.stderr:==00:00:02:19.259 34743== Warning: unimplemented fcntl command: 1036 2024-06-03T08:37:23.983 INFO:tasks.ceph.mon.a.smithi088.stderr:==00:00:02:19.309 34743== Warning: unimplemented fcntl command: 1036 2024-06-03T08:37:23.996 INFO:tasks.ceph.mon.a.smithi088.stderr:==00:00:02:19.322 34743== Warning: unimplemented fcntl command: 1036 2024-06-03T08:37:24.032 INFO:tasks.ceph.mon.b.smithi192.stderr:==00:00:02:19.334 34228== Warning: unimplemented fcntl command: 1036 2024-06-03T08:37:24.033 INFO:tasks.ceph.mon.c.smithi088.stderr:==00:00:02:19.359 34749== Warning: unimplemented fcntl command: 1036 2024-06-03T08:37:24.034 INFO:tasks.ceph.mon.b.smithi192.stderr:==00:00:02:19.336 34228== Warning: unimplemented fcntl command: 1036 2024-06-03T08:37:24.034 INFO:tasks.ceph.mon.c.smithi088.stderr:==00:00:02:19.360 34749== Warning: unimplemented fcntl command: 1036 2024-06-03T08:37:24.049 INFO:tasks.ceph.mon.b.smithi192.stderr:==00:00:02:19.352 34228== Warning: unimplemented fcntl command: 1036 2024-06-03T08:37:24.050 INFO:tasks.ceph.mon.c.smithi088.stderr:==00:00:02:19.376 34749== Warning: unimplemented fcntl command: 1036 2024-06-03T08:37:43.984 INFO:tasks.ceph.mon.a.smithi088.stderr:==00:00:02:39.309 34743== Warning: unimplemented fcntl command: 1036 2024-06-03T08:37:44.064 INFO:tasks.ceph.mon.a.smithi088.stderr:==00:00:02:39.390 34743== Warning: unimplemented fcntl command: 1036 2024-06-03T08:37:44.083 INFO:tasks.ceph.mon.a.smithi088.stderr:==00:00:02:39.409 34743== Warning: unimplemented fcntl command: 1036 2024-06-03T08:37:44.113 INFO:tasks.ceph.mon.b.smithi192.stderr:==00:00:02:39.415 34228== Warning: unimplemented fcntl command: 1036 2024-06-03T08:37:44.114 INFO:tasks.ceph.mon.b.smithi192.stderr:==00:00:02:39.416 34228== Warning: unimplemented fcntl command: 1036 2024-06-03T08:37:44.128 INFO:tasks.ceph.mon.b.smithi192.stderr:==00:00:02:39.431 34228== Warning: unimplemented fcntl command: 1036 2024-06-03T08:37:44.145 INFO:tasks.ceph.mon.c.smithi088.stderr:==00:00:02:39.471 34749== Warning: unimplemented fcntl command: 1036 2024-06-03T08:37:44.151 INFO:tasks.ceph.mon.c.smithi088.stderr:==00:00:02:39.476 34749== Warning: unimplemented fcntl command: 1036 2024-06-03T08:37:44.174 INFO:tasks.ceph.mon.c.smithi088.stderr:==00:00:02:39.500 34749== Warning: unimplemented fcntl command: 1036 2024-06-03T08:37:52.433 INFO:tasks.workunit.client.0.smithi088.stderr:Note: switching to 'f1abf795f4fc15bf2f30ac90c811a53ab21d5d65'. 2024-06-03T08:37:52.433 INFO:tasks.workunit.client.0.smithi088.stderr: 2024-06-03T08:37:52.433 INFO:tasks.workunit.client.0.smithi088.stderr:You are in 'detached HEAD' state. You can look around, make experimental 2024-06-03T08:37:52.433 INFO:tasks.workunit.client.0.smithi088.stderr:changes and commit them, and you can discard any commits you make in this 2024-06-03T08:37:52.433 INFO:tasks.workunit.client.0.smithi088.stderr:state without impacting any branches by switching back to a branch. 2024-06-03T08:37:52.433 INFO:tasks.workunit.client.0.smithi088.stderr: 2024-06-03T08:37:52.433 INFO:tasks.workunit.client.0.smithi088.stderr:If you want to create a new branch to retain commits you create, you may 2024-06-03T08:37:52.433 INFO:tasks.workunit.client.0.smithi088.stderr:do so (now or later) by using -c with the switch command. Example: 2024-06-03T08:37:52.433 INFO:tasks.workunit.client.0.smithi088.stderr: 2024-06-03T08:37:52.434 INFO:tasks.workunit.client.0.smithi088.stderr: git switch -c 2024-06-03T08:37:52.434 INFO:tasks.workunit.client.0.smithi088.stderr: 2024-06-03T08:37:52.434 INFO:tasks.workunit.client.0.smithi088.stderr:Or undo this operation with: 2024-06-03T08:37:52.434 INFO:tasks.workunit.client.0.smithi088.stderr: 2024-06-03T08:37:52.434 INFO:tasks.workunit.client.0.smithi088.stderr: git switch - 2024-06-03T08:37:52.434 INFO:tasks.workunit.client.0.smithi088.stderr: 2024-06-03T08:37:52.434 INFO:tasks.workunit.client.0.smithi088.stderr:Turn off this advice by setting config variable advice.detachedHead to false 2024-06-03T08:37:52.434 INFO:tasks.workunit.client.0.smithi088.stderr: 2024-06-03T08:37:52.434 INFO:tasks.workunit.client.0.smithi088.stderr:HEAD is now at f1abf795f4f Merge pull request #57815 from zdover23/wip-doc-2024-06-01-backport-57804-to-reef 2024-06-03T08:37:52.440 DEBUG:teuthology.orchestra.run.smithi088:> cd -- /home/ubuntu/cephtest/clone.client.0/qa/workunits && if test -e Makefile ; then make ; fi && find -executable -type f -printf '%P\0' >/home/ubuntu/cephtest/workunits.list.client.0 2024-06-03T08:37:52.530 INFO:tasks.workunit.client.0.smithi088.stdout:for d in direct_io fs ; do ( cd $d ; make all ) ; done 2024-06-03T08:37:52.532 INFO:tasks.workunit.client.0.smithi088.stdout:make[1]: Entering directory '/home/ubuntu/cephtest/clone.client.0/qa/workunits/direct_io' 2024-06-03T08:37:52.532 INFO:tasks.workunit.client.0.smithi088.stdout:cc -Wall -Wextra -D_GNU_SOURCE direct_io_test.c -o direct_io_test 2024-06-03T08:37:53.099 INFO:tasks.workunit.client.0.smithi088.stdout:cc -Wall -Wextra -D_GNU_SOURCE test_sync_io.c -o test_sync_io 2024-06-03T08:37:53.154 INFO:tasks.workunit.client.0.smithi088.stdout:cc -Wall -Wextra -D_GNU_SOURCE test_short_dio_read.c -o test_short_dio_read 2024-06-03T08:37:53.193 INFO:tasks.workunit.client.0.smithi088.stdout:make[1]: Leaving directory '/home/ubuntu/cephtest/clone.client.0/qa/workunits/direct_io' 2024-06-03T08:37:53.195 INFO:tasks.workunit.client.0.smithi088.stdout:make[1]: Entering directory '/home/ubuntu/cephtest/clone.client.0/qa/workunits/fs' 2024-06-03T08:37:53.195 INFO:tasks.workunit.client.0.smithi088.stdout:cc -Wall -Wextra -D_GNU_SOURCE test_o_trunc.c -o test_o_trunc 2024-06-03T08:37:53.234 INFO:tasks.workunit.client.0.smithi088.stdout:make[1]: Leaving directory '/home/ubuntu/cephtest/clone.client.0/qa/workunits/fs' 2024-06-03T08:37:53.237 DEBUG:teuthology.orchestra.run.smithi088:> set -ex 2024-06-03T08:37:53.237 DEBUG:teuthology.orchestra.run.smithi088:> dd if=/home/ubuntu/cephtest/workunits.list.client.0 of=/dev/stdout 2024-06-03T08:37:53.253 INFO:tasks.workunit:Running workunits matching rados/test.sh on client.0... 2024-06-03T08:37:53.254 INFO:tasks.workunit:Running workunit rados/test.sh... 2024-06-03T08:37:53.254 DEBUG:teuthology.orchestra.run.smithi088:workunit test rados/test.sh> mkdir -p -- /home/ubuntu/cephtest/mnt.0/client.0/tmp && cd -- /home/ubuntu/cephtest/mnt.0/client.0/tmp && CEPH_CLI_TEST_DUP_COMMAND=1 CEPH_REF=f1abf795f4fc15bf2f30ac90c811a53ab21d5d65 TESTDIR="/home/ubuntu/cephtest" CEPH_ARGS="--cluster ceph" CEPH_ID="0" PATH=$PATH:/usr/sbin CEPH_BASE=/home/ubuntu/cephtest/clone.client.0 CEPH_ROOT=/home/ubuntu/cephtest/clone.client.0 CEPH_MNT=/home/ubuntu/cephtest/mnt.0 ALLOW_TIMEOUTS=1 adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 6h /home/ubuntu/cephtest/clone.client.0/qa/workunits/rados/test.sh 2024-06-03T08:37:53.314 INFO:tasks.workunit.client.0.smithi088.stderr:+ parallel=1 2024-06-03T08:37:53.314 INFO:tasks.workunit.client.0.smithi088.stderr:+ '[' '' = --serial ']' 2024-06-03T08:37:53.314 INFO:tasks.workunit.client.0.smithi088.stderr:+ color= 2024-06-03T08:37:53.314 INFO:tasks.workunit.client.0.smithi088.stderr:+ '[' -t 1 ']' 2024-06-03T08:37:53.314 INFO:tasks.workunit.client.0.smithi088.stderr:+ trap cleanup EXIT ERR HUP INT QUIT 2024-06-03T08:37:53.314 INFO:tasks.workunit.client.0.smithi088.stderr:+ declare -A pids 2024-06-03T08:37:53.314 INFO:tasks.workunit.client.0.smithi088.stderr:+ for f in api_aio api_aio_pp api_io api_io_pp api_asio api_list api_lock api_lock_pp api_misc api_misc_pp api_tier_pp api_pool api_snapshots api_snapshots_pp api_stat api_stat_pp api_watch_notify api_watch_notify_pp api_cmd api_cmd_pp api_service api_service_pp api_c_write_operations api_c_read_operations api_cls_remote_reads list_parallel open_pools_parallel delete_pools_parallel 2024-06-03T08:37:53.314 INFO:tasks.workunit.client.0.smithi088.stderr:+ '[' 1 -eq 1 ']' 2024-06-03T08:37:53.315 INFO:tasks.workunit.client.0.smithi088.stderr:++ printf %25s api_aio 2024-06-03T08:37:53.315 INFO:tasks.workunit.client.0.smithi088.stderr:+ r=' api_aio' 2024-06-03T08:37:53.315 INFO:tasks.workunit.client.0.smithi088.stderr:++ echo api_aio 2024-06-03T08:37:53.315 INFO:tasks.workunit.client.0.smithi088.stderr:++ awk '{print $1}' 2024-06-03T08:37:53.315 INFO:tasks.workunit.client.0.smithi088.stderr:+ ff=api_aio 2024-06-03T08:37:53.315 INFO:tasks.workunit.client.0.smithi088.stderr:+ pid=40205 2024-06-03T08:37:53.315 INFO:tasks.workunit.client.0.smithi088.stderr:+ echo 'test api_aio on pid 40205' 2024-06-03T08:37:53.316 INFO:tasks.workunit.client.0.smithi088.stdout:test api_aio on pid 40205 2024-06-03T08:37:53.319 INFO:tasks.workunit.client.0.smithi088.stderr:+ pids[$f]=40205 2024-06-03T08:37:53.319 INFO:tasks.workunit.client.0.smithi088.stderr:+ for f in api_aio api_aio_pp api_io api_io_pp api_asio api_list api_lock api_lock_pp api_misc api_misc_pp api_tier_pp api_pool api_snapshots api_snapshots_pp api_stat api_stat_pp api_watch_notify api_watch_notify_pp api_cmd api_cmd_pp api_service api_service_pp api_c_write_operations api_c_read_operations api_cls_remote_reads list_parallel open_pools_parallel delete_pools_parallel 2024-06-03T08:37:53.319 INFO:tasks.workunit.client.0.smithi088.stderr:+ '[' 1 -eq 1 ']' 2024-06-03T08:37:53.319 INFO:tasks.workunit.client.0.smithi088.stderr:+ bash -o pipefail -exc 'ceph_test_rados_api_aio 2>&1 | tee ceph_test_rados_api_aio.log | sed "s/^/ api_aio: /"' 2024-06-03T08:37:53.319 INFO:tasks.workunit.client.0.smithi088.stderr:++ printf %25s api_aio_pp 2024-06-03T08:37:53.319 INFO:tasks.workunit.client.0.smithi088.stderr:+ r=' api_aio_pp' 2024-06-03T08:37:53.319 INFO:tasks.workunit.client.0.smithi088.stderr:++ echo api_aio_pp 2024-06-03T08:37:53.319 INFO:tasks.workunit.client.0.smithi088.stderr:++ awk '{print $1}' 2024-06-03T08:37:53.319 INFO:tasks.workunit.client.0.smithi088.stderr:+ '[' -f /etc/bashrc ']' 2024-06-03T08:37:53.319 INFO:tasks.workunit.client.0.smithi088.stderr:+ . /etc/bashrc 2024-06-03T08:37:53.319 INFO:tasks.workunit.client.0.smithi088.stderr:++ '[' -z '' ']' 2024-06-03T08:37:53.319 INFO:tasks.workunit.client.0.smithi088.stderr:++ BASHRCSOURCED=Y 2024-06-03T08:37:53.320 INFO:tasks.workunit.client.0.smithi088.stderr:++ '[' '' ']' 2024-06-03T08:37:53.320 INFO:tasks.workunit.client.0.smithi088.stderr:++ shopt -q login_shell 2024-06-03T08:37:53.320 INFO:tasks.workunit.client.0.smithi088.stderr:+++ umask 2024-06-03T08:37:53.320 INFO:tasks.workunit.client.0.smithi088.stderr:++ '[' 0022 -eq 0 ']' 2024-06-03T08:37:53.320 INFO:tasks.workunit.client.0.smithi088.stderr:++ SHELL=/bin/bash 2024-06-03T08:37:53.320 INFO:tasks.workunit.client.0.smithi088.stderr:++ for i in /etc/profile.d/*.sh 2024-06-03T08:37:53.320 INFO:tasks.workunit.client.0.smithi088.stderr:++ '[' -r /etc/profile.d/bash_completion.sh ']' 2024-06-03T08:37:53.320 INFO:tasks.workunit.client.0.smithi088.stderr:++ '[' '' ']' 2024-06-03T08:37:53.320 INFO:tasks.workunit.client.0.smithi088.stderr:++ . /etc/profile.d/bash_completion.sh 2024-06-03T08:37:53.320 INFO:tasks.workunit.client.0.smithi088.stderr:+++ '[' 'x5.1.8(1)-release' '!=' x -a x '!=' x -a x = x ']' 2024-06-03T08:37:53.320 INFO:tasks.workunit.client.0.smithi088.stderr:++ for i in /etc/profile.d/*.sh 2024-06-03T08:37:53.320 INFO:tasks.workunit.client.0.smithi088.stderr:++ '[' -r /etc/profile.d/cobbler.sh ']' 2024-06-03T08:37:53.320 INFO:tasks.workunit.client.0.smithi088.stderr:++ '[' '' ']' 2024-06-03T08:37:53.321 INFO:tasks.workunit.client.0.smithi088.stderr:++ . /etc/profile.d/cobbler.sh 2024-06-03T08:37:53.321 INFO:tasks.workunit.client.0.smithi088.stderr:+++ export COBBLER_SERVER=172.21.0.11 2024-06-03T08:37:53.321 INFO:tasks.workunit.client.0.smithi088.stderr:+++ COBBLER_SERVER=172.21.0.11 2024-06-03T08:37:53.321 INFO:tasks.workunit.client.0.smithi088.stderr:++ for i in /etc/profile.d/*.sh 2024-06-03T08:37:53.321 INFO:tasks.workunit.client.0.smithi088.stderr:++ '[' -r /etc/profile.d/colorgrep.sh ']' 2024-06-03T08:37:53.321 INFO:tasks.workunit.client.0.smithi088.stderr:++ '[' '' ']' 2024-06-03T08:37:53.321 INFO:tasks.workunit.client.0.smithi088.stderr:++ . /etc/profile.d/colorgrep.sh 2024-06-03T08:37:53.321 INFO:tasks.workunit.client.0.smithi088.stderr:+++ /usr/libexec/grepconf.sh -c 2024-06-03T08:37:53.321 INFO:tasks.workunit.client.0.smithi088.stderr:+ ff=api_aio_pp 2024-06-03T08:37:53.321 INFO:tasks.workunit.client.0.smithi088.stderr:+ pid=40212 2024-06-03T08:37:53.321 INFO:tasks.workunit.client.0.smithi088.stderr:+ echo 'test api_aio_pp on pid 40212' 2024-06-03T08:37:53.322 INFO:tasks.workunit.client.0.smithi088.stdout:test api_aio_pp on pid 40212 2024-06-03T08:37:53.325 INFO:tasks.workunit.client.0.smithi088.stderr:+ pids[$f]=40212 2024-06-03T08:37:53.325 INFO:tasks.workunit.client.0.smithi088.stderr:+ for f in api_aio api_aio_pp api_io api_io_pp api_asio api_list api_lock api_lock_pp api_misc api_misc_pp api_tier_pp api_pool api_snapshots api_snapshots_pp api_stat api_stat_pp api_watch_notify api_watch_notify_pp api_cmd api_cmd_pp api_service api_service_pp api_c_write_operations api_c_read_operations api_cls_remote_reads list_parallel open_pools_parallel delete_pools_parallel 2024-06-03T08:37:53.325 INFO:tasks.workunit.client.0.smithi088.stderr:+ '[' 1 -eq 1 ']' 2024-06-03T08:37:53.325 INFO:tasks.workunit.client.0.smithi088.stderr:+ bash -o pipefail -exc 'ceph_test_rados_api_aio_pp 2>&1 | tee ceph_test_rados_api_aio_pp.log | sed "s/^/ api_aio_pp: /"' 2024-06-03T08:37:53.325 INFO:tasks.workunit.client.0.smithi088.stderr:++ printf %25s api_io 2024-06-03T08:37:53.326 INFO:tasks.workunit.client.0.smithi088.stderr:+ r=' api_io' 2024-06-03T08:37:53.326 INFO:tasks.workunit.client.0.smithi088.stderr:++ echo api_io 2024-06-03T08:37:53.326 INFO:tasks.workunit.client.0.smithi088.stderr:++ awk '{print $1}' 2024-06-03T08:37:53.326 INFO:tasks.workunit.client.0.smithi088.stderr:+ '[' -f /etc/bashrc ']' 2024-06-03T08:37:53.326 INFO:tasks.workunit.client.0.smithi088.stderr:+ . /etc/bashrc 2024-06-03T08:37:53.326 INFO:tasks.workunit.client.0.smithi088.stderr:++ '[' -z '' ']' 2024-06-03T08:37:53.326 INFO:tasks.workunit.client.0.smithi088.stderr:++ BASHRCSOURCED=Y 2024-06-03T08:37:53.326 INFO:tasks.workunit.client.0.smithi088.stderr:++ '[' '' ']' 2024-06-03T08:37:53.326 INFO:tasks.workunit.client.0.smithi088.stderr:++ shopt -q login_shell 2024-06-03T08:37:53.326 INFO:tasks.workunit.client.0.smithi088.stderr:+++ umask 2024-06-03T08:37:53.326 INFO:tasks.workunit.client.0.smithi088.stderr:++ '[' 0022 -eq 0 ']' 2024-06-03T08:37:53.326 INFO:tasks.workunit.client.0.smithi088.stderr:++ SHELL=/bin/bash 2024-06-03T08:37:53.327 INFO:tasks.workunit.client.0.smithi088.stderr:++ for i in /etc/profile.d/*.sh 2024-06-03T08:37:53.327 INFO:tasks.workunit.client.0.smithi088.stderr:++ '[' -r /etc/profile.d/bash_completion.sh ']' 2024-06-03T08:37:53.327 INFO:tasks.workunit.client.0.smithi088.stderr:++ '[' '' ']' 2024-06-03T08:37:53.327 INFO:tasks.workunit.client.0.smithi088.stderr:++ . /etc/profile.d/bash_completion.sh 2024-06-03T08:37:53.327 INFO:tasks.workunit.client.0.smithi088.stderr:+++ alias 'grep=grep --color=auto' 2024-06-03T08:37:53.327 INFO:tasks.workunit.client.0.smithi088.stderr:+++ alias 'egrep=egrep --color=auto' 2024-06-03T08:37:53.327 INFO:tasks.workunit.client.0.smithi088.stderr:+++ alias 'fgrep=fgrep --color=auto' 2024-06-03T08:37:53.327 INFO:tasks.workunit.client.0.smithi088.stderr:+++ '[' 'x5.1.8(1)-release' '!=' x -a x '!=' x -a x = x ']' 2024-06-03T08:37:53.327 INFO:tasks.workunit.client.0.smithi088.stderr:++ for i in /etc/profile.d/*.sh 2024-06-03T08:37:53.327 INFO:tasks.workunit.client.0.smithi088.stderr:++ for i in /etc/profile.d/*.sh 2024-06-03T08:37:53.327 INFO:tasks.workunit.client.0.smithi088.stderr:++ '[' -r /etc/profile.d/cobbler.sh ']' 2024-06-03T08:37:53.327 INFO:tasks.workunit.client.0.smithi088.stderr:++ '[' -r /etc/profile.d/colorls.sh ']' 2024-06-03T08:37:53.327 INFO:tasks.workunit.client.0.smithi088.stderr:++ '[' '' ']' 2024-06-03T08:37:53.328 INFO:tasks.workunit.client.0.smithi088.stderr:++ . /etc/profile.d/cobbler.sh 2024-06-03T08:37:53.328 INFO:tasks.workunit.client.0.smithi088.stderr:++ '[' '' ']' 2024-06-03T08:37:53.328 INFO:tasks.workunit.client.0.smithi088.stderr:++ . /etc/profile.d/colorls.sh 2024-06-03T08:37:53.328 INFO:tasks.workunit.client.0.smithi088.stderr:+++ export COBBLER_SERVER=172.21.0.11 2024-06-03T08:37:53.328 INFO:tasks.workunit.client.0.smithi088.stderr:+++ COBBLER_SERVER=172.21.0.11 2024-06-03T08:37:53.328 INFO:tasks.workunit.client.0.smithi088.stderr:++ for i in /etc/profile.d/*.sh 2024-06-03T08:37:53.328 INFO:tasks.workunit.client.0.smithi088.stderr:+++ '[' '!' -t 0 ']' 2024-06-03T08:37:53.328 INFO:tasks.workunit.client.0.smithi088.stderr:++ '[' -r /etc/profile.d/colorgrep.sh ']' 2024-06-03T08:37:53.328 INFO:tasks.workunit.client.0.smithi088.stderr:+++ return 2024-06-03T08:37:53.328 INFO:tasks.workunit.client.0.smithi088.stderr:++ '[' '' ']' 2024-06-03T08:37:53.328 INFO:tasks.workunit.client.0.smithi088.stderr:++ for i in /etc/profile.d/*.sh 2024-06-03T08:37:53.328 INFO:tasks.workunit.client.0.smithi088.stderr:++ . /etc/profile.d/colorgrep.sh 2024-06-03T08:37:53.328 INFO:tasks.workunit.client.0.smithi088.stderr:++ '[' -r /etc/profile.d/colorsysstat.sh ']' 2024-06-03T08:37:53.329 INFO:tasks.workunit.client.0.smithi088.stderr:++ '[' '' ']' 2024-06-03T08:37:53.329 INFO:tasks.workunit.client.0.smithi088.stderr:++ . /etc/profile.d/colorsysstat.sh 2024-06-03T08:37:53.329 INFO:tasks.workunit.client.0.smithi088.stderr:+++ /usr/libexec/grepconf.sh -c 2024-06-03T08:37:53.329 INFO:tasks.workunit.client.0.smithi088.stderr:+++ export S_COLORS=auto 2024-06-03T08:37:53.329 INFO:tasks.workunit.client.0.smithi088.stderr:+++ S_COLORS=auto 2024-06-03T08:37:53.329 INFO:tasks.workunit.client.0.smithi088.stderr:++ for i in /etc/profile.d/*.sh 2024-06-03T08:37:53.329 INFO:tasks.workunit.client.0.smithi088.stderr:++ '[' -r /etc/profile.d/colorxzgrep.sh ']' 2024-06-03T08:37:53.329 INFO:tasks.workunit.client.0.smithi088.stderr:++ '[' '' ']' 2024-06-03T08:37:53.329 INFO:tasks.workunit.client.0.smithi088.stderr:++ . /etc/profile.d/colorxzgrep.sh 2024-06-03T08:37:53.329 INFO:tasks.workunit.client.0.smithi088.stderr:+++ /usr/libexec/grepconf.sh -c 2024-06-03T08:37:53.329 INFO:tasks.workunit.client.0.smithi088.stderr:+ ff=api_io 2024-06-03T08:37:53.329 INFO:tasks.workunit.client.0.smithi088.stderr:+ pid=40221 2024-06-03T08:37:53.330 INFO:tasks.workunit.client.0.smithi088.stdout:test api_io on pid 40221 2024-06-03T08:37:53.333 INFO:tasks.workunit.client.0.smithi088.stderr:+ echo 'test api_io on pid 40221' 2024-06-03T08:37:53.334 INFO:tasks.workunit.client.0.smithi088.stderr:+ pids[$f]=40221 2024-06-03T08:37:53.334 INFO:tasks.workunit.client.0.smithi088.stderr:+ for f in api_aio api_aio_pp api_io api_io_pp api_asio api_list api_lock api_lock_pp api_misc api_misc_pp api_tier_pp api_pool api_snapshots api_snapshots_pp api_stat api_stat_pp api_watch_notify api_watch_notify_pp api_cmd api_cmd_pp api_service api_service_pp api_c_write_operations api_c_read_operations api_cls_remote_reads list_parallel open_pools_parallel delete_pools_parallel 2024-06-03T08:37:53.334 INFO:tasks.workunit.client.0.smithi088.stderr:+ bash -o pipefail -exc 'ceph_test_rados_api_io 2>&1 | tee ceph_test_rados_api_io.log | sed "s/^/ api_io: /"' 2024-06-03T08:37:53.334 INFO:tasks.workunit.client.0.smithi088.stderr:+ '[' 1 -eq 1 ']' 2024-06-03T08:37:53.334 INFO:tasks.workunit.client.0.smithi088.stderr:++ printf %25s api_io_pp 2024-06-03T08:37:53.334 INFO:tasks.workunit.client.0.smithi088.stderr:+ r=' api_io_pp' 2024-06-03T08:37:53.334 INFO:tasks.workunit.client.0.smithi088.stderr:++ echo api_io_pp 2024-06-03T08:37:53.334 INFO:tasks.workunit.client.0.smithi088.stderr:++ awk '{print $1}' 2024-06-03T08:37:53.334 INFO:tasks.workunit.client.0.smithi088.stderr:+ '[' -f /etc/bashrc ']' 2024-06-03T08:37:53.334 INFO:tasks.workunit.client.0.smithi088.stderr:+ . /etc/bashrc 2024-06-03T08:37:53.334 INFO:tasks.workunit.client.0.smithi088.stderr:++ '[' -z '' ']' 2024-06-03T08:37:53.335 INFO:tasks.workunit.client.0.smithi088.stderr:++ BASHRCSOURCED=Y 2024-06-03T08:37:53.335 INFO:tasks.workunit.client.0.smithi088.stderr:++ '[' '' ']' 2024-06-03T08:37:53.335 INFO:tasks.workunit.client.0.smithi088.stderr:++ shopt -q login_shell 2024-06-03T08:37:53.335 INFO:tasks.workunit.client.0.smithi088.stderr:+++ umask 2024-06-03T08:37:53.335 INFO:tasks.workunit.client.0.smithi088.stderr:+++ alias 'grep=grep --color=auto' 2024-06-03T08:37:53.335 INFO:tasks.workunit.client.0.smithi088.stderr:++ '[' 0022 -eq 0 ']' 2024-06-03T08:37:53.335 INFO:tasks.workunit.client.0.smithi088.stderr:++ SHELL=/bin/bash 2024-06-03T08:37:53.335 INFO:tasks.workunit.client.0.smithi088.stderr:+++ alias 'egrep=egrep --color=auto' 2024-06-03T08:37:53.335 INFO:tasks.workunit.client.0.smithi088.stderr:+++ alias 'fgrep=fgrep --color=auto' 2024-06-03T08:37:53.335 INFO:tasks.workunit.client.0.smithi088.stderr:++ for i in /etc/profile.d/*.sh 2024-06-03T08:37:53.335 INFO:tasks.workunit.client.0.smithi088.stderr:++ '[' -r /etc/profile.d/colorls.sh ']' 2024-06-03T08:37:53.335 INFO:tasks.workunit.client.0.smithi088.stderr:++ for i in /etc/profile.d/*.sh 2024-06-03T08:37:53.335 INFO:tasks.workunit.client.0.smithi088.stderr:++ '[' '' ']' 2024-06-03T08:37:53.336 INFO:tasks.workunit.client.0.smithi088.stderr:++ '[' -r /etc/profile.d/bash_completion.sh ']' 2024-06-03T08:37:53.336 INFO:tasks.workunit.client.0.smithi088.stderr:++ . /etc/profile.d/colorls.sh 2024-06-03T08:37:53.336 INFO:tasks.workunit.client.0.smithi088.stderr:++ '[' '' ']' 2024-06-03T08:37:53.336 INFO:tasks.workunit.client.0.smithi088.stderr:++ . /etc/profile.d/bash_completion.sh 2024-06-03T08:37:53.336 INFO:tasks.workunit.client.0.smithi088.stderr:+++ '[' '!' -t 0 ']' 2024-06-03T08:37:53.336 INFO:tasks.workunit.client.0.smithi088.stderr:+++ return 2024-06-03T08:37:53.336 INFO:tasks.workunit.client.0.smithi088.stderr:++ for i in /etc/profile.d/*.sh 2024-06-03T08:37:53.336 INFO:tasks.workunit.client.0.smithi088.stderr:++ '[' -r /etc/profile.d/colorsysstat.sh ']' 2024-06-03T08:37:53.336 INFO:tasks.workunit.client.0.smithi088.stderr:++ '[' '' ']' 2024-06-03T08:37:53.336 INFO:tasks.workunit.client.0.smithi088.stderr:+++ '[' 'x5.1.8(1)-release' '!=' x -a x '!=' x -a x = x ']' 2024-06-03T08:37:53.336 INFO:tasks.workunit.client.0.smithi088.stderr:++ . /etc/profile.d/colorsysstat.sh 2024-06-03T08:37:53.336 INFO:tasks.workunit.client.0.smithi088.stderr:++ for i in /etc/profile.d/*.sh 2024-06-03T08:37:53.336 INFO:tasks.workunit.client.0.smithi088.stderr:+++ alias 'xzgrep=xzgrep --color=auto' 2024-06-03T08:37:53.337 INFO:tasks.workunit.client.0.smithi088.stderr:++ '[' -r /etc/profile.d/cobbler.sh ']' 2024-06-03T08:37:53.337 INFO:tasks.workunit.client.0.smithi088.stderr:++ '[' '' ']' 2024-06-03T08:37:53.337 INFO:tasks.workunit.client.0.smithi088.stderr:+++ export S_COLORS=auto 2024-06-03T08:37:53.337 INFO:tasks.workunit.client.0.smithi088.stderr:++ . /etc/profile.d/cobbler.sh 2024-06-03T08:37:53.337 INFO:tasks.workunit.client.0.smithi088.stderr:+++ S_COLORS=auto 2024-06-03T08:37:53.337 INFO:tasks.workunit.client.0.smithi088.stderr:+++ alias 'xzegrep=xzegrep --color=auto' 2024-06-03T08:37:53.337 INFO:tasks.workunit.client.0.smithi088.stderr:++ for i in /etc/profile.d/*.sh 2024-06-03T08:37:53.337 INFO:tasks.workunit.client.0.smithi088.stderr:++ '[' -r /etc/profile.d/colorxzgrep.sh ']' 2024-06-03T08:37:53.337 INFO:tasks.workunit.client.0.smithi088.stderr:+++ export COBBLER_SERVER=172.21.0.11 2024-06-03T08:37:53.337 INFO:tasks.workunit.client.0.smithi088.stderr:++ '[' '' ']' 2024-06-03T08:37:53.337 INFO:tasks.workunit.client.0.smithi088.stderr:+++ COBBLER_SERVER=172.21.0.11 2024-06-03T08:37:53.337 INFO:tasks.workunit.client.0.smithi088.stderr:+++ alias 'xzfgrep=xzfgrep --color=auto' 2024-06-03T08:37:53.337 INFO:tasks.workunit.client.0.smithi088.stderr:++ . /etc/profile.d/colorxzgrep.sh 2024-06-03T08:37:53.338 INFO:tasks.workunit.client.0.smithi088.stderr:++ for i in /etc/profile.d/*.sh 2024-06-03T08:37:53.338 INFO:tasks.workunit.client.0.smithi088.stderr:++ '[' -r /etc/profile.d/colorgrep.sh ']' 2024-06-03T08:37:53.338 INFO:tasks.workunit.client.0.smithi088.stderr:++ '[' '' ']' 2024-06-03T08:37:53.338 INFO:tasks.workunit.client.0.smithi088.stderr:++ for i in /etc/profile.d/*.sh 2024-06-03T08:37:53.338 INFO:tasks.workunit.client.0.smithi088.stderr:+++ /usr/libexec/grepconf.sh -c 2024-06-03T08:37:53.338 INFO:tasks.workunit.client.0.smithi088.stderr:++ . /etc/profile.d/colorgrep.sh 2024-06-03T08:37:53.338 INFO:tasks.workunit.client.0.smithi088.stderr:++ '[' -r /etc/profile.d/colorzgrep.sh ']' 2024-06-03T08:37:53.338 INFO:tasks.workunit.client.0.smithi088.stderr:++ '[' '' ']' 2024-06-03T08:37:53.338 INFO:tasks.workunit.client.0.smithi088.stderr:++ . /etc/profile.d/colorzgrep.sh 2024-06-03T08:37:53.338 INFO:tasks.workunit.client.0.smithi088.stderr:+++ /usr/libexec/grepconf.sh -c 2024-06-03T08:37:53.338 INFO:tasks.workunit.client.0.smithi088.stderr:+++ '[' -f /usr/libexec/grepconf.sh ']' 2024-06-03T08:37:53.338 INFO:tasks.workunit.client.0.smithi088.stderr:+++ /usr/libexec/grepconf.sh -c 2024-06-03T08:37:53.338 INFO:tasks.workunit.client.0.smithi088.stderr:+ ff=api_io_pp 2024-06-03T08:37:53.339 INFO:tasks.workunit.client.0.smithi088.stderr:+ pid=40232 2024-06-03T08:37:53.339 INFO:tasks.workunit.client.0.smithi088.stdout:test api_io_pp on pid 40232 2024-06-03T08:37:53.344 INFO:tasks.workunit.client.0.smithi088.stderr:+ echo 'test api_io_pp on pid 40232' 2024-06-03T08:37:53.345 INFO:tasks.workunit.client.0.smithi088.stderr:+ pids[$f]=40232 2024-06-03T08:37:53.345 INFO:tasks.workunit.client.0.smithi088.stderr:+ bash -o pipefail -exc 'ceph_test_rados_api_io_pp 2>&1 | tee ceph_test_rados_api_io_pp.log | sed "s/^/ api_io_pp: /"' 2024-06-03T08:37:53.345 INFO:tasks.workunit.client.0.smithi088.stderr:+ for f in api_aio api_aio_pp api_io api_io_pp api_asio api_list api_lock api_lock_pp api_misc api_misc_pp api_tier_pp api_pool api_snapshots api_snapshots_pp api_stat api_stat_pp api_watch_notify api_watch_notify_pp api_cmd api_cmd_pp api_service api_service_pp api_c_write_operations api_c_read_operations api_cls_remote_reads list_parallel open_pools_parallel delete_pools_parallel 2024-06-03T08:37:53.345 INFO:tasks.workunit.client.0.smithi088.stderr:+ '[' 1 -eq 1 ']' 2024-06-03T08:37:53.345 INFO:tasks.workunit.client.0.smithi088.stderr:++ printf %25s api_asio 2024-06-03T08:37:53.345 INFO:tasks.workunit.client.0.smithi088.stderr:+ r=' api_asio' 2024-06-03T08:37:53.345 INFO:tasks.workunit.client.0.smithi088.stderr:++ echo api_asio 2024-06-03T08:37:53.345 INFO:tasks.workunit.client.0.smithi088.stderr:++ awk '{print $1}' 2024-06-03T08:37:53.345 INFO:tasks.workunit.client.0.smithi088.stderr:+ '[' -f /etc/bashrc ']' 2024-06-03T08:37:53.345 INFO:tasks.workunit.client.0.smithi088.stderr:+ . /etc/bashrc 2024-06-03T08:37:53.345 INFO:tasks.workunit.client.0.smithi088.stderr:++ '[' -z '' ']' 2024-06-03T08:37:53.345 INFO:tasks.workunit.client.0.smithi088.stderr:++ BASHRCSOURCED=Y 2024-06-03T08:37:53.345 INFO:tasks.workunit.client.0.smithi088.stderr:++ '[' '' ']' 2024-06-03T08:37:53.346 INFO:tasks.workunit.client.0.smithi088.stderr:++ shopt -q login_shell 2024-06-03T08:37:53.346 INFO:tasks.workunit.client.0.smithi088.stderr:+++ alias 'grep=grep --color=auto' 2024-06-03T08:37:53.346 INFO:tasks.workunit.client.0.smithi088.stderr:+++ alias 'egrep=egrep --color=auto' 2024-06-03T08:37:53.346 INFO:tasks.workunit.client.0.smithi088.stderr:+++ umask 2024-06-03T08:37:53.346 INFO:tasks.workunit.client.0.smithi088.stderr:+++ alias 'zgrep=zgrep --color=auto' 2024-06-03T08:37:53.346 INFO:tasks.workunit.client.0.smithi088.stderr:+++ alias 'fgrep=fgrep --color=auto' 2024-06-03T08:37:53.346 INFO:tasks.workunit.client.0.smithi088.stderr:+++ alias 'zfgrep=zfgrep --color=auto' 2024-06-03T08:37:53.346 INFO:tasks.workunit.client.0.smithi088.stderr:++ for i in /etc/profile.d/*.sh 2024-06-03T08:37:53.346 INFO:tasks.workunit.client.0.smithi088.stderr:+++ alias 'zegrep=zegrep --color=auto' 2024-06-03T08:37:53.346 INFO:tasks.workunit.client.0.smithi088.stderr:++ '[' -r /etc/profile.d/colorls.sh ']' 2024-06-03T08:37:53.346 INFO:tasks.workunit.client.0.smithi088.stderr:++ '[' '' ']' 2024-06-03T08:37:53.346 INFO:tasks.workunit.client.0.smithi088.stderr:++ . /etc/profile.d/colorls.sh 2024-06-03T08:37:53.347 INFO:tasks.workunit.client.0.smithi088.stderr:++ for i in /etc/profile.d/*.sh 2024-06-03T08:37:53.347 INFO:tasks.workunit.client.0.smithi088.stderr:++ '[' -r /etc/profile.d/debuginfod.sh ']' 2024-06-03T08:37:53.347 INFO:tasks.workunit.client.0.smithi088.stderr:++ '[' '' ']' 2024-06-03T08:37:53.347 INFO:tasks.workunit.client.0.smithi088.stderr:++ . /etc/profile.d/debuginfod.sh 2024-06-03T08:37:53.347 INFO:tasks.workunit.client.0.smithi088.stderr:+++ '[' '!' -t 0 ']' 2024-06-03T08:37:53.347 INFO:tasks.workunit.client.0.smithi088.stderr:+++ return 2024-06-03T08:37:53.347 INFO:tasks.workunit.client.0.smithi088.stderr:++ for i in /etc/profile.d/*.sh 2024-06-03T08:37:53.347 INFO:tasks.workunit.client.0.smithi088.stderr:++ '[' -r /etc/profile.d/colorsysstat.sh ']' 2024-06-03T08:37:53.347 INFO:tasks.workunit.client.0.smithi088.stderr:++ '[' '' ']' 2024-06-03T08:37:53.347 INFO:tasks.workunit.client.0.smithi088.stderr:+++ '[' -z 'https://debuginfod.centos.org/ ' ']' 2024-06-03T08:37:53.347 INFO:tasks.workunit.client.0.smithi088.stderr:++ . /etc/profile.d/colorsysstat.sh 2024-06-03T08:37:53.347 INFO:tasks.workunit.client.0.smithi088.stderr:++ '[' 0022 -eq 0 ']' 2024-06-03T08:37:53.348 INFO:tasks.workunit.client.0.smithi088.stderr:++ for i in /etc/profile.d/*.sh 2024-06-03T08:37:53.348 INFO:tasks.workunit.client.0.smithi088.stderr:++ '[' -r /etc/profile.d/gawk.sh ']' 2024-06-03T08:37:53.348 INFO:tasks.workunit.client.0.smithi088.stderr:++ '[' '' ']' 2024-06-03T08:37:53.348 INFO:tasks.workunit.client.0.smithi088.stderr:++ . /etc/profile.d/gawk.sh 2024-06-03T08:37:53.348 INFO:tasks.workunit.client.0.smithi088.stderr:++ SHELL=/bin/bash 2024-06-03T08:37:53.348 INFO:tasks.workunit.client.0.smithi088.stderr:+++ export S_COLORS=auto 2024-06-03T08:37:53.348 INFO:tasks.workunit.client.0.smithi088.stderr:+++ S_COLORS=auto 2024-06-03T08:37:53.348 INFO:tasks.workunit.client.0.smithi088.stderr:++ for i in /etc/profile.d/*.sh 2024-06-03T08:37:53.348 INFO:tasks.workunit.client.0.smithi088.stderr:++ '[' -r /etc/profile.d/colorxzgrep.sh ']' 2024-06-03T08:37:53.348 INFO:tasks.workunit.client.0.smithi088.stderr:++ '[' '' ']' 2024-06-03T08:37:53.348 INFO:tasks.workunit.client.0.smithi088.stderr:++ . /etc/profile.d/colorxzgrep.sh 2024-06-03T08:37:53.348 INFO:tasks.workunit.client.0.smithi088.stderr:+++ /usr/libexec/grepconf.sh -c 2024-06-03T08:37:53.348 INFO:tasks.workunit.client.0.smithi088.stderr:++ for i in /etc/profile.d/*.sh 2024-06-03T08:37:53.349 INFO:tasks.workunit.client.0.smithi088.stderr:++ for i in /etc/profile.d/*.sh 2024-06-03T08:37:53.349 INFO:tasks.workunit.client.0.smithi088.stderr:++ '[' -r /etc/profile.d/lang.sh ']' 2024-06-03T08:37:53.349 INFO:tasks.workunit.client.0.smithi088.stderr:++ '[' '' ']' 2024-06-03T08:37:53.349 INFO:tasks.workunit.client.0.smithi088.stderr:++ . /etc/profile.d/lang.sh 2024-06-03T08:37:53.349 INFO:tasks.workunit.client.0.smithi088.stderr:++ '[' -r /etc/profile.d/bash_completion.sh ']' 2024-06-03T08:37:53.349 INFO:tasks.workunit.client.0.smithi088.stderr:++ '[' '' ']' 2024-06-03T08:37:53.349 INFO:tasks.workunit.client.0.smithi088.stderr:+++ alias 'xzgrep=xzgrep --color=auto' 2024-06-03T08:37:53.349 INFO:tasks.workunit.client.0.smithi088.stderr:++ . /etc/profile.d/bash_completion.sh 2024-06-03T08:37:53.349 INFO:tasks.workunit.client.0.smithi088.stderr:+++ unset LANG_backup 2024-06-03T08:37:53.349 INFO:tasks.workunit.client.0.smithi088.stderr:+++ alias 'xzegrep=xzegrep --color=auto' 2024-06-03T08:37:53.349 INFO:tasks.workunit.client.0.smithi088.stderr:+++ alias 'xzfgrep=xzfgrep --color=auto' 2024-06-03T08:37:53.349 INFO:tasks.workunit.client.0.smithi088.stderr:++ for i in /etc/profile.d/*.sh 2024-06-03T08:37:53.349 INFO:tasks.workunit.client.0.smithi088.stderr:++ '[' -r /etc/profile.d/colorzgrep.sh ']' 2024-06-03T08:37:53.349 INFO:tasks.workunit.client.0.smithi088.stderr:++ '[' '' ']' 2024-06-03T08:37:53.350 INFO:tasks.workunit.client.0.smithi088.stderr:++ . /etc/profile.d/colorzgrep.sh 2024-06-03T08:37:53.350 INFO:tasks.workunit.client.0.smithi088.stderr:+++ '[' 'x5.1.8(1)-release' '!=' x -a x '!=' x -a x = x ']' 2024-06-03T08:37:53.350 INFO:tasks.workunit.client.0.smithi088.stderr:+++ '[' -f /usr/libexec/grepconf.sh ']' 2024-06-03T08:37:53.350 INFO:tasks.workunit.client.0.smithi088.stderr:+++ /usr/libexec/grepconf.sh -c 2024-06-03T08:37:53.350 INFO:tasks.workunit.client.0.smithi088.stderr:++ for i in /etc/profile.d/*.sh 2024-06-03T08:37:53.350 INFO:tasks.workunit.client.0.smithi088.stderr:++ '[' -r /etc/profile.d/cobbler.sh ']' 2024-06-03T08:37:53.350 INFO:tasks.workunit.client.0.smithi088.stderr:++ '[' '' ']' 2024-06-03T08:37:53.350 INFO:tasks.workunit.client.0.smithi088.stderr:++ . /etc/profile.d/cobbler.sh 2024-06-03T08:37:53.350 INFO:tasks.workunit.client.0.smithi088.stderr:+++ export COBBLER_SERVER=172.21.0.11 2024-06-03T08:37:53.350 INFO:tasks.workunit.client.0.smithi088.stderr:+++ COBBLER_SERVER=172.21.0.11 2024-06-03T08:37:53.350 INFO:tasks.workunit.client.0.smithi088.stderr:++ for i in /etc/profile.d/*.sh 2024-06-03T08:37:53.350 INFO:tasks.workunit.client.0.smithi088.stderr:++ '[' -r /etc/profile.d/colorgrep.sh ']' 2024-06-03T08:37:53.351 INFO:tasks.workunit.client.0.smithi088.stderr:++ '[' '' ']' 2024-06-03T08:37:53.351 INFO:tasks.workunit.client.0.smithi088.stderr:++ . /etc/profile.d/colorgrep.sh 2024-06-03T08:37:53.351 INFO:tasks.workunit.client.0.smithi088.stderr:+++ /usr/libexec/grepconf.sh -c 2024-06-03T08:37:53.351 INFO:tasks.workunit.client.0.smithi088.stderr:++++ /usr/bin/locale 2024-06-03T08:37:53.351 INFO:tasks.workunit.client.0.smithi088.stderr:+ ff=api_asio 2024-06-03T08:37:53.351 INFO:tasks.workunit.client.0.smithi088.stderr:+ pid=40246 2024-06-03T08:37:53.351 INFO:tasks.workunit.client.0.smithi088.stdout:test api_asio on pid 40246 2024-06-03T08:37:53.357 INFO:tasks.workunit.client.0.smithi088.stderr:+ echo 'test api_asio on pid 40246' 2024-06-03T08:37:53.357 INFO:tasks.workunit.client.0.smithi088.stderr:+ pids[$f]=40246 2024-06-03T08:37:53.357 INFO:tasks.workunit.client.0.smithi088.stderr:+ bash -o pipefail -exc 'ceph_test_rados_api_asio 2>&1 | tee ceph_test_rados_api_asio.log | sed "s/^/ api_asio: /"' 2024-06-03T08:37:53.357 INFO:tasks.workunit.client.0.smithi088.stderr:+ for f in api_aio api_aio_pp api_io api_io_pp api_asio api_list api_lock api_lock_pp api_misc api_misc_pp api_tier_pp api_pool api_snapshots api_snapshots_pp api_stat api_stat_pp api_watch_notify api_watch_notify_pp api_cmd api_cmd_pp api_service api_service_pp api_c_write_operations api_c_read_operations api_cls_remote_reads list_parallel open_pools_parallel delete_pools_parallel 2024-06-03T08:37:53.357 INFO:tasks.workunit.client.0.smithi088.stderr:+ '[' 1 -eq 1 ']' 2024-06-03T08:37:53.357 INFO:tasks.workunit.client.0.smithi088.stderr:++ printf %25s api_list 2024-06-03T08:37:53.357 INFO:tasks.workunit.client.0.smithi088.stderr:+++ '[' -n '' ']' 2024-06-03T08:37:53.357 INFO:tasks.workunit.client.0.smithi088.stderr:+++ LANG_backup=en_US.UTF-8 2024-06-03T08:37:53.357 INFO:tasks.workunit.client.0.smithi088.stderr:+++ for config in /etc/locale.conf "${HOME}/.i18n" 2024-06-03T08:37:53.358 INFO:tasks.workunit.client.0.smithi088.stderr:+ r=' api_list' 2024-06-03T08:37:53.358 INFO:tasks.workunit.client.0.smithi088.stderr:+++ '[' -f /etc/locale.conf ']' 2024-06-03T08:37:53.358 INFO:tasks.workunit.client.0.smithi088.stderr:+++ '[' -x /usr/bin/sed ']' 2024-06-03T08:37:53.358 INFO:tasks.workunit.client.0.smithi088.stderr:++++ /usr/bin/sed -r -e 's/^[[:blank:]]*([[:upper:]_]+)=([[:print:][:digit:]\._-]+|"[[:print:][:digit:]\._-]+")/export \1=\2/;t;d' /etc/locale.conf 2024-06-03T08:37:53.358 INFO:tasks.workunit.client.0.smithi088.stderr:++ echo api_list 2024-06-03T08:37:53.358 INFO:tasks.workunit.client.0.smithi088.stderr:++ awk '{print $1}' 2024-06-03T08:37:53.358 INFO:tasks.workunit.client.0.smithi088.stderr:+ '[' -f /etc/bashrc ']' 2024-06-03T08:37:53.358 INFO:tasks.workunit.client.0.smithi088.stderr:+ . /etc/bashrc 2024-06-03T08:37:53.358 INFO:tasks.workunit.client.0.smithi088.stderr:++ '[' -z '' ']' 2024-06-03T08:37:53.358 INFO:tasks.workunit.client.0.smithi088.stderr:++ BASHRCSOURCED=Y 2024-06-03T08:37:53.358 INFO:tasks.workunit.client.0.smithi088.stderr:++ '[' '' ']' 2024-06-03T08:37:53.358 INFO:tasks.workunit.client.0.smithi088.stderr:++ shopt -q login_shell 2024-06-03T08:37:53.359 INFO:tasks.workunit.client.0.smithi088.stderr:+++ alias 'grep=grep --color=auto' 2024-06-03T08:37:53.359 INFO:tasks.workunit.client.0.smithi088.stderr:+++ alias 'egrep=egrep --color=auto' 2024-06-03T08:37:53.359 INFO:tasks.workunit.client.0.smithi088.stderr:+++ alias 'fgrep=fgrep --color=auto' 2024-06-03T08:37:53.359 INFO:tasks.workunit.client.0.smithi088.stderr:++ for i in /etc/profile.d/*.sh 2024-06-03T08:37:53.359 INFO:tasks.workunit.client.0.smithi088.stderr:++ '[' -r /etc/profile.d/colorls.sh ']' 2024-06-03T08:37:53.359 INFO:tasks.workunit.client.0.smithi088.stderr:++ '[' '' ']' 2024-06-03T08:37:53.359 INFO:tasks.workunit.client.0.smithi088.stderr:++ . /etc/profile.d/colorls.sh 2024-06-03T08:37:53.359 INFO:tasks.workunit.client.0.smithi088.stderr:+++ '[' '!' -t 0 ']' 2024-06-03T08:37:53.359 INFO:tasks.workunit.client.0.smithi088.stderr:+++ umask 2024-06-03T08:37:53.359 INFO:tasks.workunit.client.0.smithi088.stderr:+++ return 2024-06-03T08:37:53.359 INFO:tasks.workunit.client.0.smithi088.stderr:++ for i in /etc/profile.d/*.sh 2024-06-03T08:37:53.359 INFO:tasks.workunit.client.0.smithi088.stderr:++ '[' -r /etc/profile.d/colorsysstat.sh ']' 2024-06-03T08:37:53.359 INFO:tasks.workunit.client.0.smithi088.stderr:++ '[' '' ']' 2024-06-03T08:37:53.360 INFO:tasks.workunit.client.0.smithi088.stderr:++ . /etc/profile.d/colorsysstat.sh 2024-06-03T08:37:53.360 INFO:tasks.workunit.client.0.smithi088.stderr:+++ eval export 'LANG="en_US.UTF-8"' 2024-06-03T08:37:53.360 INFO:tasks.workunit.client.0.smithi088.stderr:+++ alias 'xzgrep=xzgrep --color=auto' 2024-06-03T08:37:53.360 INFO:tasks.workunit.client.0.smithi088.stderr:++++ export LANG=en_US.UTF-8 2024-06-03T08:37:53.360 INFO:tasks.workunit.client.0.smithi088.stderr:++++ LANG=en_US.UTF-8 2024-06-03T08:37:53.360 INFO:tasks.workunit.client.0.smithi088.stderr:+++ export S_COLORS=auto 2024-06-03T08:37:53.360 INFO:tasks.workunit.client.0.smithi088.stderr:+++ S_COLORS=auto 2024-06-03T08:37:53.360 INFO:tasks.workunit.client.0.smithi088.stderr:+++ alias 'xzegrep=xzegrep --color=auto' 2024-06-03T08:37:53.360 INFO:tasks.workunit.client.0.smithi088.stderr:++ for i in /etc/profile.d/*.sh 2024-06-03T08:37:53.360 INFO:tasks.workunit.client.0.smithi088.stderr:+++ for config in /etc/locale.conf "${HOME}/.i18n" 2024-06-03T08:37:53.360 INFO:tasks.workunit.client.0.smithi088.stderr:++ '[' -r /etc/profile.d/colorxzgrep.sh ']' 2024-06-03T08:37:53.360 INFO:tasks.workunit.client.0.smithi088.stderr:+++ alias 'xzfgrep=xzfgrep --color=auto' 2024-06-03T08:37:53.360 INFO:tasks.workunit.client.0.smithi088.stderr:++ '[' '' ']' 2024-06-03T08:37:53.361 INFO:tasks.workunit.client.0.smithi088.stderr:+++ '[' -f /home/ubuntu/.i18n ']' 2024-06-03T08:37:53.361 INFO:tasks.workunit.client.0.smithi088.stderr:++ . /etc/profile.d/colorxzgrep.sh 2024-06-03T08:37:53.361 INFO:tasks.workunit.client.0.smithi088.stderr:++ for i in /etc/profile.d/*.sh 2024-06-03T08:37:53.361 INFO:tasks.workunit.client.0.smithi088.stderr:+++ alias 'zgrep=zgrep --color=auto' 2024-06-03T08:37:53.361 INFO:tasks.workunit.client.0.smithi088.stderr:+++ '[' -n en_US.UTF-8 ']' 2024-06-03T08:37:53.361 INFO:tasks.workunit.client.0.smithi088.stderr:++ '[' -r /etc/profile.d/colorzgrep.sh ']' 2024-06-03T08:37:53.361 INFO:tasks.workunit.client.0.smithi088.stderr:+++ /usr/libexec/grepconf.sh -c 2024-06-03T08:37:53.361 INFO:tasks.workunit.client.0.smithi088.stderr:++ '[' '' ']' 2024-06-03T08:37:53.361 INFO:tasks.workunit.client.0.smithi088.stderr:+++ LANG=en_US.UTF-8 2024-06-03T08:37:53.361 INFO:tasks.workunit.client.0.smithi088.stderr:++ '[' 0022 -eq 0 ']' 2024-06-03T08:37:53.361 INFO:tasks.workunit.client.0.smithi088.stderr:++ . /etc/profile.d/colorzgrep.sh 2024-06-03T08:37:53.361 INFO:tasks.workunit.client.0.smithi088.stderr:++ SHELL=/bin/bash 2024-06-03T08:37:53.361 INFO:tasks.workunit.client.0.smithi088.stderr:+++ alias 'zfgrep=zfgrep --color=auto' 2024-06-03T08:37:53.362 INFO:tasks.workunit.client.0.smithi088.stderr:+++ '[' -f /usr/libexec/grepconf.sh ']' 2024-06-03T08:37:53.362 INFO:tasks.workunit.client.0.smithi088.stderr:+++ /usr/libexec/grepconf.sh -c 2024-06-03T08:37:53.362 INFO:tasks.workunit.client.0.smithi088.stderr:+++ unset LANG_backup config 2024-06-03T08:37:53.362 INFO:tasks.workunit.client.0.smithi088.stderr:+++ alias 'zegrep=zegrep --color=auto' 2024-06-03T08:37:53.362 INFO:tasks.workunit.client.0.smithi088.stderr:++ for i in /etc/profile.d/*.sh 2024-06-03T08:37:53.362 INFO:tasks.workunit.client.0.smithi088.stderr:+++ '[' -n '' ']' 2024-06-03T08:37:53.362 INFO:tasks.workunit.client.0.smithi088.stderr:++ '[' -r /etc/profile.d/debuginfod.sh ']' 2024-06-03T08:37:53.362 INFO:tasks.workunit.client.0.smithi088.stderr:++ '[' '' ']' 2024-06-03T08:37:53.362 INFO:tasks.workunit.client.0.smithi088.stderr:++ for i in /etc/profile.d/*.sh 2024-06-03T08:37:53.362 INFO:tasks.workunit.client.0.smithi088.stderr:++ . /etc/profile.d/debuginfod.sh 2024-06-03T08:37:53.362 INFO:tasks.workunit.client.0.smithi088.stderr:++ '[' -r /etc/profile.d/bash_completion.sh ']' 2024-06-03T08:37:53.362 INFO:tasks.workunit.client.0.smithi088.stderr:++ '[' '' ']' 2024-06-03T08:37:53.362 INFO:tasks.workunit.client.0.smithi088.stderr:++ . /etc/profile.d/bash_completion.sh 2024-06-03T08:37:53.363 INFO:tasks.workunit.client.0.smithi088.stderr:+++ '[' -z 'https://debuginfod.centos.org/ ' ']' 2024-06-03T08:37:53.363 INFO:tasks.workunit.client.0.smithi088.stderr:+++ '[' -n en_US.UTF-8 ']' 2024-06-03T08:37:53.363 INFO:tasks.workunit.client.0.smithi088.stderr:++ for i in /etc/profile.d/*.sh 2024-06-03T08:37:53.363 INFO:tasks.workunit.client.0.smithi088.stderr:++ '[' -r /etc/profile.d/gawk.sh ']' 2024-06-03T08:37:53.363 INFO:tasks.workunit.client.0.smithi088.stderr:+++ '[' dumb = linux ']' 2024-06-03T08:37:53.363 INFO:tasks.workunit.client.0.smithi088.stderr:++ '[' '' ']' 2024-06-03T08:37:53.363 INFO:tasks.workunit.client.0.smithi088.stderr:++ . /etc/profile.d/gawk.sh 2024-06-03T08:37:53.363 INFO:tasks.workunit.client.0.smithi088.stderr:++ for i in /etc/profile.d/*.sh 2024-06-03T08:37:53.363 INFO:tasks.workunit.client.0.smithi088.stderr:++ '[' -r /etc/profile.d/less.sh ']' 2024-06-03T08:37:53.363 INFO:tasks.workunit.client.0.smithi088.stderr:+++ '[' 'x5.1.8(1)-release' '!=' x -a x '!=' x -a x = x ']' 2024-06-03T08:37:53.363 INFO:tasks.workunit.client.0.smithi088.stderr:++ '[' '' ']' 2024-06-03T08:37:53.363 INFO:tasks.workunit.client.0.smithi088.stderr:++ . /etc/profile.d/less.sh 2024-06-03T08:37:53.363 INFO:tasks.workunit.client.0.smithi088.stderr:++ for i in /etc/profile.d/*.sh 2024-06-03T08:37:53.363 INFO:tasks.workunit.client.0.smithi088.stderr:++ '[' -r /etc/profile.d/cobbler.sh ']' 2024-06-03T08:37:53.364 INFO:tasks.workunit.client.0.smithi088.stderr:++ '[' '' ']' 2024-06-03T08:37:53.364 INFO:tasks.workunit.client.0.smithi088.stderr:++ . /etc/profile.d/cobbler.sh 2024-06-03T08:37:53.364 INFO:tasks.workunit.client.0.smithi088.stderr:+++ '[' -z '||/usr/bin/lesspipe.sh %s' ']' 2024-06-03T08:37:53.364 INFO:tasks.workunit.client.0.smithi088.stderr:+++ export COBBLER_SERVER=172.21.0.11 2024-06-03T08:37:53.364 INFO:tasks.workunit.client.0.smithi088.stderr:+++ COBBLER_SERVER=172.21.0.11 2024-06-03T08:37:53.364 INFO:tasks.workunit.client.0.smithi088.stderr:++ for i in /etc/profile.d/*.sh 2024-06-03T08:37:53.364 INFO:tasks.workunit.client.0.smithi088.stderr:++ for i in /etc/profile.d/*.sh 2024-06-03T08:37:53.364 INFO:tasks.workunit.client.0.smithi088.stderr:++ for i in /etc/profile.d/*.sh 2024-06-03T08:37:53.364 INFO:tasks.workunit.client.0.smithi088.stderr:++ '[' -r /etc/profile.d/which2.sh ']' 2024-06-03T08:37:53.364 INFO:tasks.workunit.client.0.smithi088.stderr:++ '[' -r /etc/profile.d/colorgrep.sh ']' 2024-06-03T08:37:53.364 INFO:tasks.workunit.client.0.smithi088.stderr:++ '[' -r /etc/profile.d/lang.sh ']' 2024-06-03T08:37:53.364 INFO:tasks.workunit.client.0.smithi088.stderr:++ '[' '' ']' 2024-06-03T08:37:53.364 INFO:tasks.workunit.client.0.smithi088.stderr:++ '[' '' ']' 2024-06-03T08:37:53.364 INFO:tasks.workunit.client.0.smithi088.stderr:++ '[' '' ']' 2024-06-03T08:37:53.365 INFO:tasks.workunit.client.0.smithi088.stderr:++ . /etc/profile.d/which2.sh 2024-06-03T08:37:53.365 INFO:tasks.workunit.client.0.smithi088.stderr:++ . /etc/profile.d/colorgrep.sh 2024-06-03T08:37:53.365 INFO:tasks.workunit.client.0.smithi088.stderr:++ . /etc/profile.d/lang.sh 2024-06-03T08:37:53.365 INFO:tasks.workunit.client.0.smithi088.stderr:+++ /usr/libexec/grepconf.sh -c 2024-06-03T08:37:53.365 INFO:tasks.workunit.client.0.smithi088.stderr:+++ unset LANG_backup 2024-06-03T08:37:53.365 INFO:tasks.workunit.client.0.smithi088.stderr:+++ case "$(basename $(readlink /proc/$$/exe))" in 2024-06-03T08:37:53.365 INFO:tasks.workunit.client.0.smithi088.stderr:++++ /usr/bin/locale 2024-06-03T08:37:53.365 INFO:tasks.workunit.client.0.smithi088.stderr:+++++ readlink /proc/40205/exe 2024-06-03T08:37:53.365 INFO:tasks.workunit.client.0.smithi088.stderr:+ ff=api_list 2024-06-03T08:37:53.365 INFO:tasks.workunit.client.0.smithi088.stderr:+ pid=40263 2024-06-03T08:37:53.365 INFO:tasks.workunit.client.0.smithi088.stdout:test api_list on pid 40263 2024-06-03T08:37:53.373 INFO:tasks.workunit.client.0.smithi088.stderr:+ echo 'test api_list on pid 40263' 2024-06-03T08:37:53.373 INFO:tasks.workunit.client.0.smithi088.stderr:+ pids[$f]=40263 2024-06-03T08:37:53.373 INFO:tasks.workunit.client.0.smithi088.stderr:+ for f in api_aio api_aio_pp api_io api_io_pp api_asio api_list api_lock api_lock_pp api_misc api_misc_pp api_tier_pp api_pool api_snapshots api_snapshots_pp api_stat api_stat_pp api_watch_notify api_watch_notify_pp api_cmd api_cmd_pp api_service api_service_pp api_c_write_operations api_c_read_operations api_cls_remote_reads list_parallel open_pools_parallel delete_pools_parallel 2024-06-03T08:37:53.373 INFO:tasks.workunit.client.0.smithi088.stderr:+ '[' 1 -eq 1 ']' 2024-06-03T08:37:53.373 INFO:tasks.workunit.client.0.smithi088.stderr:+ bash -o pipefail -exc 'ceph_test_rados_api_list 2>&1 | tee ceph_test_rados_api_list.log | sed "s/^/ api_list: /"' 2024-06-03T08:37:53.373 INFO:tasks.workunit.client.0.smithi088.stderr:++ printf %25s api_lock 2024-06-03T08:37:53.373 INFO:tasks.workunit.client.0.smithi088.stderr:+ r=' api_lock' 2024-06-03T08:37:53.374 INFO:tasks.workunit.client.0.smithi088.stderr:++++ basename /usr/bin/bash 2024-06-03T08:37:53.374 INFO:tasks.workunit.client.0.smithi088.stderr:++ echo api_lock 2024-06-03T08:37:53.374 INFO:tasks.workunit.client.0.smithi088.stderr:+++ '[' -n '' ']' 2024-06-03T08:37:53.374 INFO:tasks.workunit.client.0.smithi088.stderr:+++ LANG_backup=en_US.UTF-8 2024-06-03T08:37:53.374 INFO:tasks.workunit.client.0.smithi088.stderr:+++ for config in /etc/locale.conf "${HOME}/.i18n" 2024-06-03T08:37:53.374 INFO:tasks.workunit.client.0.smithi088.stderr:++ awk '{print $1}' 2024-06-03T08:37:53.374 INFO:tasks.workunit.client.0.smithi088.stderr:+++ '[' -f /etc/locale.conf ']' 2024-06-03T08:37:53.374 INFO:tasks.workunit.client.0.smithi088.stderr:+++ '[' -x /usr/bin/sed ']' 2024-06-03T08:37:53.374 INFO:tasks.workunit.client.0.smithi088.stderr:+ '[' -f /etc/bashrc ']' 2024-06-03T08:37:53.374 INFO:tasks.workunit.client.0.smithi088.stderr:+ . /etc/bashrc 2024-06-03T08:37:53.374 INFO:tasks.workunit.client.0.smithi088.stderr:++++ /usr/bin/sed -r -e 's/^[[:blank:]]*([[:upper:]_]+)=([[:print:][:digit:]\._-]+|"[[:print:][:digit:]\._-]+")/export \1=\2/;t;d' /etc/locale.conf 2024-06-03T08:37:53.374 INFO:tasks.workunit.client.0.smithi088.stderr:++ '[' -z '' ']' 2024-06-03T08:37:53.374 INFO:tasks.workunit.client.0.smithi088.stderr:++ BASHRCSOURCED=Y 2024-06-03T08:37:53.375 INFO:tasks.workunit.client.0.smithi088.stderr:++ '[' '' ']' 2024-06-03T08:37:53.375 INFO:tasks.workunit.client.0.smithi088.stderr:++ shopt -q login_shell 2024-06-03T08:37:53.375 INFO:tasks.workunit.client.0.smithi088.stderr:+++ alias 'xzgrep=xzgrep --color=auto' 2024-06-03T08:37:53.375 INFO:tasks.workunit.client.0.smithi088.stderr:+++ alias 'xzegrep=xzegrep --color=auto' 2024-06-03T08:37:53.375 INFO:tasks.workunit.client.0.smithi088.stderr:+++ which_declare='declare -f' 2024-06-03T08:37:53.375 INFO:tasks.workunit.client.0.smithi088.stderr:+++ which_opt=-f 2024-06-03T08:37:53.375 INFO:tasks.workunit.client.0.smithi088.stderr:+++ alias 'xzfgrep=xzfgrep --color=auto' 2024-06-03T08:37:53.375 INFO:tasks.workunit.client.0.smithi088.stderr:+++ umask 2024-06-03T08:37:53.375 INFO:tasks.workunit.client.0.smithi088.stderr:++ for i in /etc/profile.d/*.sh 2024-06-03T08:37:53.375 INFO:tasks.workunit.client.0.smithi088.stderr:+++ export which_declare 2024-06-03T08:37:53.375 INFO:tasks.workunit.client.0.smithi088.stderr:++ '[' -r /etc/profile.d/colorzgrep.sh ']' 2024-06-03T08:37:53.375 INFO:tasks.workunit.client.0.smithi088.stderr:+++ export -f which 2024-06-03T08:37:53.375 INFO:tasks.workunit.client.0.smithi088.stderr:++ '[' '' ']' 2024-06-03T08:37:53.375 INFO:tasks.workunit.client.0.smithi088.stderr:++ . /etc/profile.d/colorzgrep.sh 2024-06-03T08:37:53.376 INFO:tasks.workunit.client.0.smithi088.stderr:++ unset i 2024-06-03T08:37:53.376 INFO:tasks.workunit.client.0.smithi088.stderr:+++ alias 'zgrep=zgrep --color=auto' 2024-06-03T08:37:53.376 INFO:tasks.workunit.client.0.smithi088.stderr:++ unset -f pathmunge 2024-06-03T08:37:53.376 INFO:tasks.workunit.client.0.smithi088.stderr:+++ '[' -f /usr/libexec/grepconf.sh ']' 2024-06-03T08:37:53.376 INFO:tasks.workunit.client.0.smithi088.stderr:+++ /usr/libexec/grepconf.sh -c 2024-06-03T08:37:53.376 INFO:tasks.workunit.client.0.smithi088.stderr:+++ alias 'zfgrep=zfgrep --color=auto' 2024-06-03T08:37:53.376 INFO:tasks.workunit.client.0.smithi088.stderr:+++ alias 'zegrep=zegrep --color=auto' 2024-06-03T08:37:53.376 INFO:tasks.workunit.client.0.smithi088.stderr:+ [[ /home/ubuntu/.local/bin:/home/ubuntu/bin:/usr/local/bin:/usr/bin:/usr/local/sbin:/usr/sbin:/usr/sbin =~ /home/ubuntu/\.local/bin:/home/ubuntu/bin: ]] 2024-06-03T08:37:53.376 INFO:tasks.workunit.client.0.smithi088.stderr:++ for i in /etc/profile.d/*.sh 2024-06-03T08:37:53.376 INFO:tasks.workunit.client.0.smithi088.stderr:++ '[' -r /etc/profile.d/debuginfod.sh ']' 2024-06-03T08:37:53.376 INFO:tasks.workunit.client.0.smithi088.stderr:++ '[' '' ']' 2024-06-03T08:37:53.376 INFO:tasks.workunit.client.0.smithi088.stderr:++ '[' 0022 -eq 0 ']' 2024-06-03T08:37:53.376 INFO:tasks.workunit.client.0.smithi088.stderr:++ . /etc/profile.d/debuginfod.sh 2024-06-03T08:37:53.377 INFO:tasks.workunit.client.0.smithi088.stderr:++ SHELL=/bin/bash 2024-06-03T08:37:53.377 INFO:tasks.workunit.client.0.smithi088.stderr:+++ '[' -z 'https://debuginfod.centos.org/ ' ']' 2024-06-03T08:37:53.377 INFO:tasks.workunit.client.0.smithi088.stderr:+ export PATH 2024-06-03T08:37:53.377 INFO:tasks.workunit.client.0.smithi088.stderr:++ for i in /etc/profile.d/*.sh 2024-06-03T08:37:53.377 INFO:tasks.workunit.client.0.smithi088.stderr:++ for i in /etc/profile.d/*.sh 2024-06-03T08:37:53.377 INFO:tasks.workunit.client.0.smithi088.stderr:++ '[' -r /etc/profile.d/gawk.sh ']' 2024-06-03T08:37:53.377 INFO:tasks.workunit.client.0.smithi088.stderr:++ '[' -r /etc/profile.d/bash_completion.sh ']' 2024-06-03T08:37:53.377 INFO:tasks.workunit.client.0.smithi088.stderr:+ '[' -d /home/ubuntu/.bashrc.d ']' 2024-06-03T08:37:53.377 INFO:tasks.workunit.client.0.smithi088.stderr:++ '[' '' ']' 2024-06-03T08:37:53.377 INFO:tasks.workunit.client.0.smithi088.stderr:++ '[' '' ']' 2024-06-03T08:37:53.377 INFO:tasks.workunit.client.0.smithi088.stderr:++ . /etc/profile.d/gawk.sh 2024-06-03T08:37:53.377 INFO:tasks.workunit.client.0.smithi088.stderr:++ . /etc/profile.d/bash_completion.sh 2024-06-03T08:37:53.377 INFO:tasks.workunit.client.0.smithi088.stderr:+ unset rc 2024-06-03T08:37:53.378 INFO:tasks.workunit.client.0.smithi088.stderr:+++ '[' 'x5.1.8(1)-release' '!=' x -a x '!=' x -a x = x ']' 2024-06-03T08:37:53.378 INFO:tasks.workunit.client.0.smithi088.stderr:++ for i in /etc/profile.d/*.sh 2024-06-03T08:37:53.378 INFO:tasks.workunit.client.0.smithi088.stderr:++ '[' -r /etc/profile.d/cobbler.sh ']' 2024-06-03T08:37:53.378 INFO:tasks.workunit.client.0.smithi088.stderr:++ '[' '' ']' 2024-06-03T08:37:53.378 INFO:tasks.workunit.client.0.smithi088.stderr:++ . /etc/profile.d/cobbler.sh 2024-06-03T08:37:53.378 INFO:tasks.workunit.client.0.smithi088.stderr:++ for i in /etc/profile.d/*.sh 2024-06-03T08:37:53.378 INFO:tasks.workunit.client.0.smithi088.stderr:+++ export COBBLER_SERVER=172.21.0.11 2024-06-03T08:37:53.378 INFO:tasks.workunit.client.0.smithi088.stderr:+++ COBBLER_SERVER=172.21.0.11 2024-06-03T08:37:53.378 INFO:tasks.workunit.client.0.smithi088.stderr:++ '[' -r /etc/profile.d/lang.sh ']' 2024-06-03T08:37:53.378 INFO:tasks.workunit.client.0.smithi088.stderr:++ for i in /etc/profile.d/*.sh 2024-06-03T08:37:53.378 INFO:tasks.workunit.client.0.smithi088.stderr:++ '[' '' ']' 2024-06-03T08:37:53.378 INFO:tasks.workunit.client.0.smithi088.stderr:++ '[' -r /etc/profile.d/colorgrep.sh ']' 2024-06-03T08:37:53.378 INFO:tasks.workunit.client.0.smithi088.stderr:++ '[' '' ']' 2024-06-03T08:37:53.379 INFO:tasks.workunit.client.0.smithi088.stderr:++ . /etc/profile.d/lang.sh 2024-06-03T08:37:53.379 INFO:tasks.workunit.client.0.smithi088.stderr:++ . /etc/profile.d/colorgrep.sh 2024-06-03T08:37:53.379 INFO:tasks.workunit.client.0.smithi088.stderr:+++ /usr/libexec/grepconf.sh -c 2024-06-03T08:37:53.379 INFO:tasks.workunit.client.0.smithi088.stderr:+++ unset LANG_backup 2024-06-03T08:37:53.379 INFO:tasks.workunit.client.0.smithi088.stderr:+ ceph_test_rados_api_aio 2024-06-03T08:37:53.379 INFO:tasks.workunit.client.0.smithi088.stderr:+++ alias 'grep=grep --color=auto' 2024-06-03T08:37:53.379 INFO:tasks.workunit.client.0.smithi088.stderr:+ tee ceph_test_rados_api_aio.log 2024-06-03T08:37:53.379 INFO:tasks.workunit.client.0.smithi088.stderr:++++ /usr/bin/locale 2024-06-03T08:37:53.379 INFO:tasks.workunit.client.0.smithi088.stderr:+++ alias 'egrep=egrep --color=auto' 2024-06-03T08:37:53.379 INFO:tasks.workunit.client.0.smithi088.stderr:+++ eval export 'LANG="en_US.UTF-8"' 2024-06-03T08:37:53.379 INFO:tasks.workunit.client.0.smithi088.stderr:+++ alias 'fgrep=fgrep --color=auto' 2024-06-03T08:37:53.379 INFO:tasks.workunit.client.0.smithi088.stderr:++++ export LANG=en_US.UTF-8 2024-06-03T08:37:53.380 INFO:tasks.workunit.client.0.smithi088.stderr:++++ LANG=en_US.UTF-8 2024-06-03T08:37:53.380 INFO:tasks.workunit.client.0.smithi088.stderr:++ for i in /etc/profile.d/*.sh 2024-06-03T08:37:53.380 INFO:tasks.workunit.client.0.smithi088.stderr:+++ for config in /etc/locale.conf "${HOME}/.i18n" 2024-06-03T08:37:53.380 INFO:tasks.workunit.client.0.smithi088.stderr:++ '[' -r /etc/profile.d/colorls.sh ']' 2024-06-03T08:37:53.380 INFO:tasks.workunit.client.0.smithi088.stderr:+++ '[' -f /home/ubuntu/.i18n ']' 2024-06-03T08:37:53.380 INFO:tasks.workunit.client.0.smithi088.stderr:++ '[' '' ']' 2024-06-03T08:37:53.380 INFO:tasks.workunit.client.0.smithi088.stderr:++ . /etc/profile.d/colorls.sh 2024-06-03T08:37:53.380 INFO:tasks.workunit.client.0.smithi088.stderr:+++ '[' -n en_US.UTF-8 ']' 2024-06-03T08:37:53.380 INFO:tasks.workunit.client.0.smithi088.stderr:+++ LANG=en_US.UTF-8 2024-06-03T08:37:53.380 INFO:tasks.workunit.client.0.smithi088.stderr:+++ unset LANG_backup config 2024-06-03T08:37:53.380 INFO:tasks.workunit.client.0.smithi088.stderr:+++ '[' '!' -t 0 ']' 2024-06-03T08:37:53.380 INFO:tasks.workunit.client.0.smithi088.stderr:+++ return 2024-06-03T08:37:53.380 INFO:tasks.workunit.client.0.smithi088.stderr:+++ '[' -n '' ']' 2024-06-03T08:37:53.380 INFO:tasks.workunit.client.0.smithi088.stderr:++ for i in /etc/profile.d/*.sh 2024-06-03T08:37:53.381 INFO:tasks.workunit.client.0.smithi088.stderr:++ '[' -r /etc/profile.d/colorsysstat.sh ']' 2024-06-03T08:37:53.381 INFO:tasks.workunit.client.0.smithi088.stderr:++ '[' '' ']' 2024-06-03T08:37:53.381 INFO:tasks.workunit.client.0.smithi088.stderr:++ . /etc/profile.d/colorsysstat.sh 2024-06-03T08:37:53.381 INFO:tasks.workunit.client.0.smithi088.stderr:+++ export S_COLORS=auto 2024-06-03T08:37:53.381 INFO:tasks.workunit.client.0.smithi088.stderr:+++ S_COLORS=auto 2024-06-03T08:37:53.381 INFO:tasks.workunit.client.0.smithi088.stderr:+++ '[' -n en_US.UTF-8 ']' 2024-06-03T08:37:53.381 INFO:tasks.workunit.client.0.smithi088.stderr:++ for i in /etc/profile.d/*.sh 2024-06-03T08:37:53.381 INFO:tasks.workunit.client.0.smithi088.stderr:+++ '[' dumb = linux ']' 2024-06-03T08:37:53.381 INFO:tasks.workunit.client.0.smithi088.stderr:++ '[' -r /etc/profile.d/colorxzgrep.sh ']' 2024-06-03T08:37:53.381 INFO:tasks.workunit.client.0.smithi088.stderr:++ '[' '' ']' 2024-06-03T08:37:53.381 INFO:tasks.workunit.client.0.smithi088.stderr:++ . /etc/profile.d/colorxzgrep.sh 2024-06-03T08:37:53.381 INFO:tasks.workunit.client.0.smithi088.stderr:++ for i in /etc/profile.d/*.sh 2024-06-03T08:37:53.381 INFO:tasks.workunit.client.0.smithi088.stderr:++ '[' -r /etc/profile.d/less.sh ']' 2024-06-03T08:37:53.382 INFO:tasks.workunit.client.0.smithi088.stderr:++ '[' '' ']' 2024-06-03T08:37:53.382 INFO:tasks.workunit.client.0.smithi088.stderr:++ . /etc/profile.d/less.sh 2024-06-03T08:37:53.382 INFO:tasks.workunit.client.0.smithi088.stderr:+++ /usr/libexec/grepconf.sh -c 2024-06-03T08:37:53.382 INFO:tasks.workunit.client.0.smithi088.stderr:+++ '[' -z '||/usr/bin/lesspipe.sh %s' ']' 2024-06-03T08:37:53.382 INFO:tasks.workunit.client.0.smithi088.stderr:++ for i in /etc/profile.d/*.sh 2024-06-03T08:37:53.382 INFO:tasks.workunit.client.0.smithi088.stderr:++ '[' -r /etc/profile.d/which2.sh ']' 2024-06-03T08:37:53.382 INFO:tasks.workunit.client.0.smithi088.stderr:++ '[' '' ']' 2024-06-03T08:37:53.382 INFO:tasks.workunit.client.0.smithi088.stderr:++ . /etc/profile.d/which2.sh 2024-06-03T08:37:53.382 INFO:tasks.workunit.client.0.smithi088.stderr:+++ case "$(basename $(readlink /proc/$$/exe))" in 2024-06-03T08:37:53.382 INFO:tasks.workunit.client.0.smithi088.stderr:+ sed 's/^/ api_aio: /' 2024-06-03T08:37:53.382 INFO:tasks.workunit.client.0.smithi088.stderr:+ ff=api_lock 2024-06-03T08:37:53.382 INFO:tasks.workunit.client.0.smithi088.stderr:+ pid=40284 2024-06-03T08:37:53.382 INFO:tasks.workunit.client.0.smithi088.stderr:+++ '[' -n '' ']' 2024-06-03T08:37:53.382 INFO:tasks.workunit.client.0.smithi088.stderr:+++ LANG_backup=en_US.UTF-8 2024-06-03T08:37:53.383 INFO:tasks.workunit.client.0.smithi088.stderr:+++++ readlink /proc/40212/exe 2024-06-03T08:37:53.383 INFO:tasks.workunit.client.0.smithi088.stderr:+ echo 'test api_lock on pid 40284' 2024-06-03T08:37:53.383 INFO:tasks.workunit.client.0.smithi088.stderr:+++ for config in /etc/locale.conf "${HOME}/.i18n" 2024-06-03T08:37:53.383 INFO:tasks.workunit.client.0.smithi088.stderr:+++ '[' -f /etc/locale.conf ']' 2024-06-03T08:37:53.383 INFO:tasks.workunit.client.0.smithi088.stderr:+++ '[' -x /usr/bin/sed ']' 2024-06-03T08:37:53.383 INFO:tasks.workunit.client.0.smithi088.stdout:test api_lock on pid 40284 2024-06-03T08:37:53.391 INFO:tasks.workunit.client.0.smithi088.stderr:+ pids[$f]=40284 2024-06-03T08:37:53.391 INFO:tasks.workunit.client.0.smithi088.stderr:+ for f in api_aio api_aio_pp api_io api_io_pp api_asio api_list api_lock api_lock_pp api_misc api_misc_pp api_tier_pp api_pool api_snapshots api_snapshots_pp api_stat api_stat_pp api_watch_notify api_watch_notify_pp api_cmd api_cmd_pp api_service api_service_pp api_c_write_operations api_c_read_operations api_cls_remote_reads list_parallel open_pools_parallel delete_pools_parallel 2024-06-03T08:37:53.391 INFO:tasks.workunit.client.0.smithi088.stderr:+ '[' 1 -eq 1 ']' 2024-06-03T08:37:53.391 INFO:tasks.workunit.client.0.smithi088.stderr:++++ /usr/bin/sed -r -e 's/^[[:blank:]]*([[:upper:]_]+)=([[:print:][:digit:]\._-]+|"[[:print:][:digit:]\._-]+")/export \1=\2/;t;d' /etc/locale.conf 2024-06-03T08:37:53.391 INFO:tasks.workunit.client.0.smithi088.stderr:+ bash -o pipefail -exc 'ceph_test_rados_api_lock 2>&1 | tee ceph_test_rados_api_lock.log | sed "s/^/ api_lock: /"' 2024-06-03T08:37:53.391 INFO:tasks.workunit.client.0.smithi088.stderr:++ printf %25s api_lock_pp 2024-06-03T08:37:53.391 INFO:tasks.workunit.client.0.smithi088.stderr:+ r=' api_lock_pp' 2024-06-03T08:37:53.391 INFO:tasks.workunit.client.0.smithi088.stderr:++++ basename /usr/bin/bash 2024-06-03T08:37:53.391 INFO:tasks.workunit.client.0.smithi088.stderr:+++ alias 'zgrep=zgrep --color=auto' 2024-06-03T08:37:53.391 INFO:tasks.workunit.client.0.smithi088.stderr:+++ alias 'zfgrep=zfgrep --color=auto' 2024-06-03T08:37:53.391 INFO:tasks.workunit.client.0.smithi088.stderr:+++ alias 'zegrep=zegrep --color=auto' 2024-06-03T08:37:53.391 INFO:tasks.workunit.client.0.smithi088.stderr:++ for i in /etc/profile.d/*.sh 2024-06-03T08:37:53.392 INFO:tasks.workunit.client.0.smithi088.stderr:++ '[' -r /etc/profile.d/debuginfod.sh ']' 2024-06-03T08:37:53.392 INFO:tasks.workunit.client.0.smithi088.stderr:++ '[' '' ']' 2024-06-03T08:37:53.392 INFO:tasks.workunit.client.0.smithi088.stderr:++ . /etc/profile.d/debuginfod.sh 2024-06-03T08:37:53.392 INFO:tasks.workunit.client.0.smithi088.stderr:+++ '[' -z 'https://debuginfod.centos.org/ ' ']' 2024-06-03T08:37:53.392 INFO:tasks.workunit.client.0.smithi088.stderr:++ for i in /etc/profile.d/*.sh 2024-06-03T08:37:53.392 INFO:tasks.workunit.client.0.smithi088.stderr:++ '[' -r /etc/profile.d/gawk.sh ']' 2024-06-03T08:37:53.392 INFO:tasks.workunit.client.0.smithi088.stderr:++ '[' '' ']' 2024-06-03T08:37:53.392 INFO:tasks.workunit.client.0.smithi088.stderr:++ . /etc/profile.d/gawk.sh 2024-06-03T08:37:53.392 INFO:tasks.workunit.client.0.smithi088.stderr:++ echo api_lock_pp 2024-06-03T08:37:53.392 INFO:tasks.workunit.client.0.smithi088.stderr:++ for i in /etc/profile.d/*.sh 2024-06-03T08:37:53.392 INFO:tasks.workunit.client.0.smithi088.stderr:++ '[' -r /etc/profile.d/lang.sh ']' 2024-06-03T08:37:53.392 INFO:tasks.workunit.client.0.smithi088.stderr:++ '[' '' ']' 2024-06-03T08:37:53.392 INFO:tasks.workunit.client.0.smithi088.stderr:++ . /etc/profile.d/lang.sh 2024-06-03T08:37:53.393 INFO:tasks.workunit.client.0.smithi088.stderr:++ awk '{print $1}' 2024-06-03T08:37:53.393 INFO:tasks.workunit.client.0.smithi088.stderr:+++ unset LANG_backup 2024-06-03T08:37:53.393 INFO:tasks.workunit.client.0.smithi088.stderr:+++ eval export 'LANG="en_US.UTF-8"' 2024-06-03T08:37:53.393 INFO:tasks.workunit.client.0.smithi088.stderr:+++ which_declare='declare -f' 2024-06-03T08:37:53.393 INFO:tasks.workunit.client.0.smithi088.stderr:++++ export LANG=en_US.UTF-8 2024-06-03T08:37:53.393 INFO:tasks.workunit.client.0.smithi088.stderr:++++ LANG=en_US.UTF-8 2024-06-03T08:37:53.393 INFO:tasks.workunit.client.0.smithi088.stderr:+++ for config in /etc/locale.conf "${HOME}/.i18n" 2024-06-03T08:37:53.393 INFO:tasks.workunit.client.0.smithi088.stderr:+ '[' -f /etc/bashrc ']' 2024-06-03T08:37:53.393 INFO:tasks.workunit.client.0.smithi088.stderr:+++ which_opt=-f 2024-06-03T08:37:53.393 INFO:tasks.workunit.client.0.smithi088.stderr:+ . /etc/bashrc 2024-06-03T08:37:53.393 INFO:tasks.workunit.client.0.smithi088.stderr:+++ '[' -f /home/ubuntu/.i18n ']' 2024-06-03T08:37:53.393 INFO:tasks.workunit.client.0.smithi088.stderr:+++ export which_declare 2024-06-03T08:37:53.393 INFO:tasks.workunit.client.0.smithi088.stderr:+++ export -f which 2024-06-03T08:37:53.394 INFO:tasks.workunit.client.0.smithi088.stderr:+++ alias 'grep=grep --color=auto' 2024-06-03T08:37:53.394 INFO:tasks.workunit.client.0.smithi088.stderr:+++ '[' -n en_US.UTF-8 ']' 2024-06-03T08:37:53.394 INFO:tasks.workunit.client.0.smithi088.stderr:+++ LANG=en_US.UTF-8 2024-06-03T08:37:53.394 INFO:tasks.workunit.client.0.smithi088.stderr:++++ /usr/bin/locale 2024-06-03T08:37:53.394 INFO:tasks.workunit.client.0.smithi088.stderr:++ unset i 2024-06-03T08:37:53.394 INFO:tasks.workunit.client.0.smithi088.stderr:++ unset -f pathmunge 2024-06-03T08:37:53.394 INFO:tasks.workunit.client.0.smithi088.stderr:+++ unset LANG_backup config 2024-06-03T08:37:53.394 INFO:tasks.workunit.client.0.smithi088.stderr:+++ alias 'egrep=egrep --color=auto' 2024-06-03T08:37:53.394 INFO:tasks.workunit.client.0.smithi088.stderr:+++ alias 'fgrep=fgrep --color=auto' 2024-06-03T08:37:53.394 INFO:tasks.workunit.client.0.smithi088.stderr:+++ '[' -n '' ']' 2024-06-03T08:37:53.394 INFO:tasks.workunit.client.0.smithi088.stderr:+ [[ /home/ubuntu/.local/bin:/home/ubuntu/bin:/usr/local/bin:/usr/bin:/usr/local/sbin:/usr/sbin:/usr/sbin =~ /home/ubuntu/\.local/bin:/home/ubuntu/bin: ]] 2024-06-03T08:37:53.394 INFO:tasks.workunit.client.0.smithi088.stderr:++ '[' -z '' ']' 2024-06-03T08:37:53.394 INFO:tasks.workunit.client.0.smithi088.stderr:++ for i in /etc/profile.d/*.sh 2024-06-03T08:37:53.395 INFO:tasks.workunit.client.0.smithi088.stderr:++ BASHRCSOURCED=Y 2024-06-03T08:37:53.395 INFO:tasks.workunit.client.0.smithi088.stderr:++ '[' -r /etc/profile.d/colorls.sh ']' 2024-06-03T08:37:53.395 INFO:tasks.workunit.client.0.smithi088.stderr:++ '[' '' ']' 2024-06-03T08:37:53.395 INFO:tasks.workunit.client.0.smithi088.stderr:++ shopt -q login_shell 2024-06-03T08:37:53.395 INFO:tasks.workunit.client.0.smithi088.stderr:++ '[' '' ']' 2024-06-03T08:37:53.395 INFO:tasks.workunit.client.0.smithi088.stderr:++ . /etc/profile.d/colorls.sh 2024-06-03T08:37:53.395 INFO:tasks.workunit.client.0.smithi088.stderr:+++ '[' -n en_US.UTF-8 ']' 2024-06-03T08:37:53.395 INFO:tasks.workunit.client.0.smithi088.stderr:+++ '[' dumb = linux ']' 2024-06-03T08:37:53.395 INFO:tasks.workunit.client.0.smithi088.stderr:++ for i in /etc/profile.d/*.sh 2024-06-03T08:37:53.395 INFO:tasks.workunit.client.0.smithi088.stderr:+ export PATH 2024-06-03T08:37:53.395 INFO:tasks.workunit.client.0.smithi088.stderr:++ '[' -r /etc/profile.d/less.sh ']' 2024-06-03T08:37:53.395 INFO:tasks.workunit.client.0.smithi088.stderr:+++ '[' '!' -t 0 ']' 2024-06-03T08:37:53.395 INFO:tasks.workunit.client.0.smithi088.stderr:++ '[' '' ']' 2024-06-03T08:37:53.396 INFO:tasks.workunit.client.0.smithi088.stderr:+++ return 2024-06-03T08:37:53.396 INFO:tasks.workunit.client.0.smithi088.stderr:++ . /etc/profile.d/less.sh 2024-06-03T08:37:53.396 INFO:tasks.workunit.client.0.smithi088.stderr:++ for i in /etc/profile.d/*.sh 2024-06-03T08:37:53.396 INFO:tasks.workunit.client.0.smithi088.stderr:+ '[' -d /home/ubuntu/.bashrc.d ']' 2024-06-03T08:37:53.396 INFO:tasks.workunit.client.0.smithi088.stderr:++ '[' -r /etc/profile.d/colorsysstat.sh ']' 2024-06-03T08:37:53.396 INFO:tasks.workunit.client.0.smithi088.stderr:+ unset rc 2024-06-03T08:37:53.396 INFO:tasks.workunit.client.0.smithi088.stderr:++ '[' '' ']' 2024-06-03T08:37:53.396 INFO:tasks.workunit.client.0.smithi088.stderr:++ . /etc/profile.d/colorsysstat.sh 2024-06-03T08:37:53.396 INFO:tasks.workunit.client.0.smithi088.stderr:+++ '[' -z '||/usr/bin/lesspipe.sh %s' ']' 2024-06-03T08:37:53.396 INFO:tasks.workunit.client.0.smithi088.stderr:++ for i in /etc/profile.d/*.sh 2024-06-03T08:37:53.396 INFO:tasks.workunit.client.0.smithi088.stderr:++ '[' -r /etc/profile.d/which2.sh ']' 2024-06-03T08:37:53.396 INFO:tasks.workunit.client.0.smithi088.stderr:+++ export S_COLORS=auto 2024-06-03T08:37:53.396 INFO:tasks.workunit.client.0.smithi088.stderr:++ '[' '' ']' 2024-06-03T08:37:53.397 INFO:tasks.workunit.client.0.smithi088.stderr:+++ S_COLORS=auto 2024-06-03T08:37:53.397 INFO:tasks.workunit.client.0.smithi088.stderr:++ . /etc/profile.d/which2.sh 2024-06-03T08:37:53.397 INFO:tasks.workunit.client.0.smithi088.stderr:++ for i in /etc/profile.d/*.sh 2024-06-03T08:37:53.397 INFO:tasks.workunit.client.0.smithi088.stderr:++ '[' -r /etc/profile.d/colorxzgrep.sh ']' 2024-06-03T08:37:53.397 INFO:tasks.workunit.client.0.smithi088.stderr:++ '[' '' ']' 2024-06-03T08:37:53.397 INFO:tasks.workunit.client.0.smithi088.stderr:++ . /etc/profile.d/colorxzgrep.sh 2024-06-03T08:37:53.397 INFO:tasks.workunit.client.0.smithi088.stderr:+++ case "$(basename $(readlink /proc/$$/exe))" in 2024-06-03T08:37:53.397 INFO:tasks.workunit.client.0.smithi088.stderr:+++ /usr/libexec/grepconf.sh -c 2024-06-03T08:37:53.397 INFO:tasks.workunit.client.0.smithi088.stderr:+++ umask 2024-06-03T08:37:53.397 INFO:tasks.workunit.client.0.smithi088.stderr:+ ceph_test_rados_api_aio_pp 2024-06-03T08:37:53.397 INFO:tasks.workunit.client.0.smithi088.stderr:++ '[' 0022 -eq 0 ']' 2024-06-03T08:37:53.397 INFO:tasks.workunit.client.0.smithi088.stderr:++ SHELL=/bin/bash 2024-06-03T08:37:53.397 INFO:tasks.workunit.client.0.smithi088.stderr:++ for i in /etc/profile.d/*.sh 2024-06-03T08:37:53.398 INFO:tasks.workunit.client.0.smithi088.stderr:+++ alias 'xzgrep=xzgrep --color=auto' 2024-06-03T08:37:53.398 INFO:tasks.workunit.client.0.smithi088.stderr:+++ alias 'xzegrep=xzegrep --color=auto' 2024-06-03T08:37:53.398 INFO:tasks.workunit.client.0.smithi088.stderr:+ sed 's/^/ api_aio_pp: /' 2024-06-03T08:37:53.398 INFO:tasks.workunit.client.0.smithi088.stderr:++ '[' -r /etc/profile.d/bash_completion.sh ']' 2024-06-03T08:37:53.398 INFO:tasks.workunit.client.0.smithi088.stderr:++ '[' '' ']' 2024-06-03T08:37:53.398 INFO:tasks.workunit.client.0.smithi088.stderr:++ . /etc/profile.d/bash_completion.sh 2024-06-03T08:37:53.398 INFO:tasks.workunit.client.0.smithi088.stderr:+++ alias 'xzfgrep=xzfgrep --color=auto' 2024-06-03T08:37:53.398 INFO:tasks.workunit.client.0.smithi088.stderr:+++ '[' 'x5.1.8(1)-release' '!=' x -a x '!=' x -a x = x ']' 2024-06-03T08:37:53.398 INFO:tasks.workunit.client.0.smithi088.stderr:++ for i in /etc/profile.d/*.sh 2024-06-03T08:37:53.398 INFO:tasks.workunit.client.0.smithi088.stderr:++ '[' -r /etc/profile.d/cobbler.sh ']' 2024-06-03T08:37:53.398 INFO:tasks.workunit.client.0.smithi088.stderr:++ '[' '' ']' 2024-06-03T08:37:53.398 INFO:tasks.workunit.client.0.smithi088.stderr:++ . /etc/profile.d/cobbler.sh 2024-06-03T08:37:53.398 INFO:tasks.workunit.client.0.smithi088.stderr:+++ export COBBLER_SERVER=172.21.0.11 2024-06-03T08:37:53.399 INFO:tasks.workunit.client.0.smithi088.stderr:++ for i in /etc/profile.d/*.sh 2024-06-03T08:37:53.399 INFO:tasks.workunit.client.0.smithi088.stderr:++ '[' -r /etc/profile.d/colorzgrep.sh ']' 2024-06-03T08:37:53.399 INFO:tasks.workunit.client.0.smithi088.stderr:++ '[' '' ']' 2024-06-03T08:37:53.399 INFO:tasks.workunit.client.0.smithi088.stderr:++ . /etc/profile.d/colorzgrep.sh 2024-06-03T08:37:53.399 INFO:tasks.workunit.client.0.smithi088.stderr:+++ COBBLER_SERVER=172.21.0.11 2024-06-03T08:37:53.399 INFO:tasks.workunit.client.0.smithi088.stderr:+++ '[' -f /usr/libexec/grepconf.sh ']' 2024-06-03T08:37:53.399 INFO:tasks.workunit.client.0.smithi088.stderr:+++ /usr/libexec/grepconf.sh -c 2024-06-03T08:37:53.399 INFO:tasks.workunit.client.0.smithi088.stderr:++ for i in /etc/profile.d/*.sh 2024-06-03T08:37:53.399 INFO:tasks.workunit.client.0.smithi088.stderr:++ '[' -r /etc/profile.d/colorgrep.sh ']' 2024-06-03T08:37:53.399 INFO:tasks.workunit.client.0.smithi088.stderr:++ '[' '' ']' 2024-06-03T08:37:53.399 INFO:tasks.workunit.client.0.smithi088.stderr:++ . /etc/profile.d/colorgrep.sh 2024-06-03T08:37:53.399 INFO:tasks.workunit.client.0.smithi088.stderr:+++ /usr/libexec/grepconf.sh -c 2024-06-03T08:37:53.399 INFO:tasks.workunit.client.0.smithi088.stderr:+++++ readlink /proc/40221/exe 2024-06-03T08:37:53.400 INFO:tasks.workunit.client.0.smithi088.stderr:+ tee ceph_test_rados_api_aio_pp.log 2024-06-03T08:37:53.400 INFO:tasks.workunit.client.0.smithi088.stderr:+ ff=api_lock_pp 2024-06-03T08:37:53.400 INFO:tasks.workunit.client.0.smithi088.stderr:+ pid=40304 2024-06-03T08:37:53.400 INFO:tasks.workunit.client.0.smithi088.stderr:+ echo 'test api_lock_pp on pid 40304' 2024-06-03T08:37:53.400 INFO:tasks.workunit.client.0.smithi088.stdout:test api_lock_pp on pid 40304 2024-06-03T08:37:53.409 INFO:tasks.workunit.client.0.smithi088.stderr:+ pids[$f]=40304 2024-06-03T08:37:53.409 INFO:tasks.workunit.client.0.smithi088.stderr:+ for f in api_aio api_aio_pp api_io api_io_pp api_asio api_list api_lock api_lock_pp api_misc api_misc_pp api_tier_pp api_pool api_snapshots api_snapshots_pp api_stat api_stat_pp api_watch_notify api_watch_notify_pp api_cmd api_cmd_pp api_service api_service_pp api_c_write_operations api_c_read_operations api_cls_remote_reads list_parallel open_pools_parallel delete_pools_parallel 2024-06-03T08:37:53.409 INFO:tasks.workunit.client.0.smithi088.stderr:++++ basename /usr/bin/bash 2024-06-03T08:37:53.409 INFO:tasks.workunit.client.0.smithi088.stderr:+ '[' 1 -eq 1 ']' 2024-06-03T08:37:53.409 INFO:tasks.workunit.client.0.smithi088.stderr:+++ '[' -n '' ']' 2024-06-03T08:37:53.409 INFO:tasks.workunit.client.0.smithi088.stderr:+++ LANG_backup=en_US.UTF-8 2024-06-03T08:37:53.409 INFO:tasks.workunit.client.0.smithi088.stderr:+++ for config in /etc/locale.conf "${HOME}/.i18n" 2024-06-03T08:37:53.409 INFO:tasks.workunit.client.0.smithi088.stderr:+++ '[' -f /etc/locale.conf ']' 2024-06-03T08:37:53.410 INFO:tasks.workunit.client.0.smithi088.stderr:+ bash -o pipefail -exc 'ceph_test_rados_api_lock_pp 2>&1 | tee ceph_test_rados_api_lock_pp.log | sed "s/^/ api_lock_pp: /"' 2024-06-03T08:37:53.410 INFO:tasks.workunit.client.0.smithi088.stderr:+++ '[' -x /usr/bin/sed ']' 2024-06-03T08:37:53.410 INFO:tasks.workunit.client.0.smithi088.stderr:++ printf %25s api_misc 2024-06-03T08:37:53.410 INFO:tasks.workunit.client.0.smithi088.stderr:+ r=' api_misc' 2024-06-03T08:37:53.410 INFO:tasks.workunit.client.0.smithi088.stderr:+++ which_declare='declare -f' 2024-06-03T08:37:53.410 INFO:tasks.workunit.client.0.smithi088.stderr:+++ which_opt=-f 2024-06-03T08:37:53.410 INFO:tasks.workunit.client.0.smithi088.stderr:+++ export which_declare 2024-06-03T08:37:53.410 INFO:tasks.workunit.client.0.smithi088.stderr:+++ export -f which 2024-06-03T08:37:53.410 INFO:tasks.workunit.client.0.smithi088.stderr:++ unset i 2024-06-03T08:37:53.410 INFO:tasks.workunit.client.0.smithi088.stderr:++++ /usr/bin/sed -r -e 's/^[[:blank:]]*([[:upper:]_]+)=([[:print:][:digit:]\._-]+|"[[:print:][:digit:]\._-]+")/export \1=\2/;t;d' /etc/locale.conf 2024-06-03T08:37:53.410 INFO:tasks.workunit.client.0.smithi088.stderr:+++ alias 'xzgrep=xzgrep --color=auto' 2024-06-03T08:37:53.410 INFO:tasks.workunit.client.0.smithi088.stderr:+++ alias 'xzegrep=xzegrep --color=auto' 2024-06-03T08:37:53.411 INFO:tasks.workunit.client.0.smithi088.stderr:++ unset -f pathmunge 2024-06-03T08:37:53.411 INFO:tasks.workunit.client.0.smithi088.stderr:++ echo api_misc 2024-06-03T08:37:53.411 INFO:tasks.workunit.client.0.smithi088.stderr:+ [[ /home/ubuntu/.local/bin:/home/ubuntu/bin:/usr/local/bin:/usr/bin:/usr/local/sbin:/usr/sbin:/usr/sbin =~ /home/ubuntu/\.local/bin:/home/ubuntu/bin: ]] 2024-06-03T08:37:53.411 INFO:tasks.workunit.client.0.smithi088.stderr:+++ alias 'xzfgrep=xzfgrep --color=auto' 2024-06-03T08:37:53.411 INFO:tasks.workunit.client.0.smithi088.stderr:+ export PATH 2024-06-03T08:37:53.411 INFO:tasks.workunit.client.0.smithi088.stderr:+ '[' -d /home/ubuntu/.bashrc.d ']' 2024-06-03T08:37:53.411 INFO:tasks.workunit.client.0.smithi088.stderr:++ for i in /etc/profile.d/*.sh 2024-06-03T08:37:53.411 INFO:tasks.workunit.client.0.smithi088.stderr:+ unset rc 2024-06-03T08:37:53.411 INFO:tasks.workunit.client.0.smithi088.stderr:+ '[' -f /etc/bashrc ']' 2024-06-03T08:37:53.411 INFO:tasks.workunit.client.0.smithi088.stderr:+ . /etc/bashrc 2024-06-03T08:37:53.411 INFO:tasks.workunit.client.0.smithi088.stderr:++ '[' -r /etc/profile.d/colorzgrep.sh ']' 2024-06-03T08:37:53.411 INFO:tasks.workunit.client.0.smithi088.stderr:++ '[' '' ']' 2024-06-03T08:37:53.411 INFO:tasks.workunit.client.0.smithi088.stderr:++ . /etc/profile.d/colorzgrep.sh 2024-06-03T08:37:53.412 INFO:tasks.workunit.client.0.smithi088.stderr:++ '[' -z '' ']' 2024-06-03T08:37:53.412 INFO:tasks.workunit.client.0.smithi088.stderr:++ BASHRCSOURCED=Y 2024-06-03T08:37:53.412 INFO:tasks.workunit.client.0.smithi088.stderr:++ '[' '' ']' 2024-06-03T08:37:53.412 INFO:tasks.workunit.client.0.smithi088.stderr:+ ceph_test_rados_api_io 2024-06-03T08:37:53.412 INFO:tasks.workunit.client.0.smithi088.stderr:++ shopt -q login_shell 2024-06-03T08:37:53.412 INFO:tasks.workunit.client.0.smithi088.stderr:+++ '[' -f /usr/libexec/grepconf.sh ']' 2024-06-03T08:37:53.412 INFO:tasks.workunit.client.0.smithi088.stderr:+++ /usr/libexec/grepconf.sh -c 2024-06-03T08:37:53.412 INFO:tasks.workunit.client.0.smithi088.stderr:++ awk '{print $1}' 2024-06-03T08:37:53.412 INFO:tasks.workunit.client.0.smithi088.stderr:+ tee ceph_test_rados_api_io.log 2024-06-03T08:37:53.412 INFO:tasks.workunit.client.0.smithi088.stderr:+++ umask 2024-06-03T08:37:53.412 INFO:tasks.workunit.client.0.smithi088.stderr:+++ alias 'zgrep=zgrep --color=auto' 2024-06-03T08:37:53.412 INFO:tasks.workunit.client.0.smithi088.stderr:+++ alias 'zfgrep=zfgrep --color=auto' 2024-06-03T08:37:53.412 INFO:tasks.workunit.client.0.smithi088.stderr:+++ alias 'zegrep=zegrep --color=auto' 2024-06-03T08:37:53.412 INFO:tasks.workunit.client.0.smithi088.stderr:++ for i in /etc/profile.d/*.sh 2024-06-03T08:37:53.413 INFO:tasks.workunit.client.0.smithi088.stderr:++ '[' -r /etc/profile.d/debuginfod.sh ']' 2024-06-03T08:37:53.413 INFO:tasks.workunit.client.0.smithi088.stderr:++ '[' '' ']' 2024-06-03T08:37:53.413 INFO:tasks.workunit.client.0.smithi088.stderr:++ . /etc/profile.d/debuginfod.sh 2024-06-03T08:37:53.413 INFO:tasks.workunit.client.0.smithi088.stderr:+++ '[' -z 'https://debuginfod.centos.org/ ' ']' 2024-06-03T08:37:53.413 INFO:tasks.workunit.client.0.smithi088.stderr:++ for i in /etc/profile.d/*.sh 2024-06-03T08:37:53.413 INFO:tasks.workunit.client.0.smithi088.stderr:++ '[' -r /etc/profile.d/gawk.sh ']' 2024-06-03T08:37:53.413 INFO:tasks.workunit.client.0.smithi088.stderr:++ '[' '' ']' 2024-06-03T08:37:53.413 INFO:tasks.workunit.client.0.smithi088.stderr:++ . /etc/profile.d/gawk.sh 2024-06-03T08:37:53.413 INFO:tasks.workunit.client.0.smithi088.stderr:++ for i in /etc/profile.d/*.sh 2024-06-03T08:37:53.413 INFO:tasks.workunit.client.0.smithi088.stderr:++ '[' 0022 -eq 0 ']' 2024-06-03T08:37:53.413 INFO:tasks.workunit.client.0.smithi088.stderr:++ SHELL=/bin/bash 2024-06-03T08:37:53.413 INFO:tasks.workunit.client.0.smithi088.stderr:++ '[' -r /etc/profile.d/lang.sh ']' 2024-06-03T08:37:53.414 INFO:tasks.workunit.client.0.smithi088.stderr:++ '[' '' ']' 2024-06-03T08:37:53.414 INFO:tasks.workunit.client.0.smithi088.stderr:++ for i in /etc/profile.d/*.sh 2024-06-03T08:37:53.414 INFO:tasks.workunit.client.0.smithi088.stderr:++ '[' -r /etc/profile.d/bash_completion.sh ']' 2024-06-03T08:37:53.414 INFO:tasks.workunit.client.0.smithi088.stderr:++ '[' '' ']' 2024-06-03T08:37:53.414 INFO:tasks.workunit.client.0.smithi088.stderr:++ . /etc/profile.d/bash_completion.sh 2024-06-03T08:37:53.414 INFO:tasks.workunit.client.0.smithi088.stderr:++ . /etc/profile.d/lang.sh 2024-06-03T08:37:53.414 INFO:tasks.workunit.client.0.smithi088.stderr:+++ eval export 'LANG="en_US.UTF-8"' 2024-06-03T08:37:53.414 INFO:tasks.workunit.client.0.smithi088.stderr:++++ export LANG=en_US.UTF-8 2024-06-03T08:37:53.414 INFO:tasks.workunit.client.0.smithi088.stderr:++++ LANG=en_US.UTF-8 2024-06-03T08:37:53.414 INFO:tasks.workunit.client.0.smithi088.stderr:+++ '[' 'x5.1.8(1)-release' '!=' x -a x '!=' x -a x = x ']' 2024-06-03T08:37:53.414 INFO:tasks.workunit.client.0.smithi088.stderr:++ for i in /etc/profile.d/*.sh 2024-06-03T08:37:53.414 INFO:tasks.workunit.client.0.smithi088.stderr:+++ for config in /etc/locale.conf "${HOME}/.i18n" 2024-06-03T08:37:53.414 INFO:tasks.workunit.client.0.smithi088.stderr:++ '[' -r /etc/profile.d/cobbler.sh ']' 2024-06-03T08:37:53.415 INFO:tasks.workunit.client.0.smithi088.stderr:+++ '[' -f /home/ubuntu/.i18n ']' 2024-06-03T08:37:53.415 INFO:tasks.workunit.client.0.smithi088.stderr:++ '[' '' ']' 2024-06-03T08:37:53.415 INFO:tasks.workunit.client.0.smithi088.stderr:++ . /etc/profile.d/cobbler.sh 2024-06-03T08:37:53.415 INFO:tasks.workunit.client.0.smithi088.stderr:+++ '[' -n en_US.UTF-8 ']' 2024-06-03T08:37:53.415 INFO:tasks.workunit.client.0.smithi088.stderr:+++ LANG=en_US.UTF-8 2024-06-03T08:37:53.415 INFO:tasks.workunit.client.0.smithi088.stderr:+++ export COBBLER_SERVER=172.21.0.11 2024-06-03T08:37:53.415 INFO:tasks.workunit.client.0.smithi088.stderr:+++ COBBLER_SERVER=172.21.0.11 2024-06-03T08:37:53.415 INFO:tasks.workunit.client.0.smithi088.stderr:+++ unset LANG_backup config 2024-06-03T08:37:53.415 INFO:tasks.workunit.client.0.smithi088.stderr:++ for i in /etc/profile.d/*.sh 2024-06-03T08:37:53.415 INFO:tasks.workunit.client.0.smithi088.stderr:++ '[' -r /etc/profile.d/colorgrep.sh ']' 2024-06-03T08:37:53.415 INFO:tasks.workunit.client.0.smithi088.stderr:++ '[' '' ']' 2024-06-03T08:37:53.415 INFO:tasks.workunit.client.0.smithi088.stderr:++ . /etc/profile.d/colorgrep.sh 2024-06-03T08:37:53.415 INFO:tasks.workunit.client.0.smithi088.stderr:+++ '[' -n '' ']' 2024-06-03T08:37:53.416 INFO:tasks.workunit.client.0.smithi088.stderr:+++ /usr/libexec/grepconf.sh -c 2024-06-03T08:37:53.416 INFO:tasks.workunit.client.0.smithi088.stderr:+++ '[' -n en_US.UTF-8 ']' 2024-06-03T08:37:53.416 INFO:tasks.workunit.client.0.smithi088.stderr:+++ unset LANG_backup 2024-06-03T08:37:53.416 INFO:tasks.workunit.client.0.smithi088.stderr:+++ '[' dumb = linux ']' 2024-06-03T08:37:53.416 INFO:tasks.workunit.client.0.smithi088.stderr:++ for i in /etc/profile.d/*.sh 2024-06-03T08:37:53.416 INFO:tasks.workunit.client.0.smithi088.stderr:++ '[' -r /etc/profile.d/less.sh ']' 2024-06-03T08:37:53.416 INFO:tasks.workunit.client.0.smithi088.stderr:+ sed 's/^/ api_io: /' 2024-06-03T08:37:53.416 INFO:tasks.workunit.client.0.smithi088.stderr:++ '[' '' ']' 2024-06-03T08:37:53.416 INFO:tasks.workunit.client.0.smithi088.stderr:++ . /etc/profile.d/less.sh 2024-06-03T08:37:53.416 INFO:tasks.workunit.client.0.smithi088.stderr:+++ '[' -z '||/usr/bin/lesspipe.sh %s' ']' 2024-06-03T08:37:53.416 INFO:tasks.workunit.client.0.smithi088.stderr:++ for i in /etc/profile.d/*.sh 2024-06-03T08:37:53.416 INFO:tasks.workunit.client.0.smithi088.stderr:++ '[' -r /etc/profile.d/which2.sh ']' 2024-06-03T08:37:53.416 INFO:tasks.workunit.client.0.smithi088.stderr:++ '[' '' ']' 2024-06-03T08:37:53.417 INFO:tasks.workunit.client.0.smithi088.stderr:+++ alias 'grep=grep --color=auto' 2024-06-03T08:37:53.417 INFO:tasks.workunit.client.0.smithi088.stderr:+++ alias 'egrep=egrep --color=auto' 2024-06-03T08:37:53.417 INFO:tasks.workunit.client.0.smithi088.stderr:++ . /etc/profile.d/which2.sh 2024-06-03T08:37:53.417 INFO:tasks.workunit.client.0.smithi088.stderr:+++ case "$(basename $(readlink /proc/$$/exe))" in 2024-06-03T08:37:53.417 INFO:tasks.workunit.client.0.smithi088.stderr:+++ alias 'fgrep=fgrep --color=auto' 2024-06-03T08:37:53.417 INFO:tasks.workunit.client.0.smithi088.stderr:++ for i in /etc/profile.d/*.sh 2024-06-03T08:37:53.417 INFO:tasks.workunit.client.0.smithi088.stderr:++ '[' -r /etc/profile.d/colorls.sh ']' 2024-06-03T08:37:53.417 INFO:tasks.workunit.client.0.smithi088.stderr:++ '[' '' ']' 2024-06-03T08:37:53.417 INFO:tasks.workunit.client.0.smithi088.stderr:++ . /etc/profile.d/colorls.sh 2024-06-03T08:37:53.417 INFO:tasks.workunit.client.0.smithi088.stderr:+++ '[' '!' -t 0 ']' 2024-06-03T08:37:53.417 INFO:tasks.workunit.client.0.smithi088.stderr:+++ return 2024-06-03T08:37:53.417 INFO:tasks.workunit.client.0.smithi088.stderr:++ for i in /etc/profile.d/*.sh 2024-06-03T08:37:53.417 INFO:tasks.workunit.client.0.smithi088.stderr:++++ /usr/bin/locale 2024-06-03T08:37:53.418 INFO:tasks.workunit.client.0.smithi088.stderr:++ '[' -r /etc/profile.d/colorsysstat.sh ']' 2024-06-03T08:37:53.418 INFO:tasks.workunit.client.0.smithi088.stderr:++ '[' '' ']' 2024-06-03T08:37:53.418 INFO:tasks.workunit.client.0.smithi088.stderr:++ . /etc/profile.d/colorsysstat.sh 2024-06-03T08:37:53.418 INFO:tasks.workunit.client.0.smithi088.stderr:+++ export S_COLORS=auto 2024-06-03T08:37:53.418 INFO:tasks.workunit.client.0.smithi088.stderr:+++ S_COLORS=auto 2024-06-03T08:37:53.418 INFO:tasks.workunit.client.0.smithi088.stderr:++ for i in /etc/profile.d/*.sh 2024-06-03T08:37:53.418 INFO:tasks.workunit.client.0.smithi088.stderr:++ '[' -r /etc/profile.d/colorxzgrep.sh ']' 2024-06-03T08:37:53.418 INFO:tasks.workunit.client.0.smithi088.stderr:++ '[' '' ']' 2024-06-03T08:37:53.418 INFO:tasks.workunit.client.0.smithi088.stderr:++ . /etc/profile.d/colorxzgrep.sh 2024-06-03T08:37:53.418 INFO:tasks.workunit.client.0.smithi088.stderr:+ ff=api_misc 2024-06-03T08:37:53.418 INFO:tasks.workunit.client.0.smithi088.stderr:+++++ readlink /proc/40232/exe 2024-06-03T08:37:53.418 INFO:tasks.workunit.client.0.smithi088.stderr:+++ /usr/libexec/grepconf.sh -c 2024-06-03T08:37:53.418 INFO:tasks.workunit.client.0.smithi088.stderr:+ pid=40322 2024-06-03T08:37:53.419 INFO:tasks.workunit.client.0.smithi088.stderr:+ echo 'test api_misc on pid 40322' 2024-06-03T08:37:53.419 INFO:tasks.workunit.client.0.smithi088.stdout:test api_misc on pid 40322 2024-06-03T08:37:53.429 INFO:tasks.workunit.client.0.smithi088.stderr:+ pids[$f]=40322 2024-06-03T08:37:53.429 INFO:tasks.workunit.client.0.smithi088.stderr:+ for f in api_aio api_aio_pp api_io api_io_pp api_asio api_list api_lock api_lock_pp api_misc api_misc_pp api_tier_pp api_pool api_snapshots api_snapshots_pp api_stat api_stat_pp api_watch_notify api_watch_notify_pp api_cmd api_cmd_pp api_service api_service_pp api_c_write_operations api_c_read_operations api_cls_remote_reads list_parallel open_pools_parallel delete_pools_parallel 2024-06-03T08:37:53.429 INFO:tasks.workunit.client.0.smithi088.stderr:+ '[' 1 -eq 1 ']' 2024-06-03T08:37:53.429 INFO:tasks.workunit.client.0.smithi088.stderr:++ printf %25s api_misc_pp 2024-06-03T08:37:53.429 INFO:tasks.workunit.client.0.smithi088.stderr:+ r=' api_misc_pp' 2024-06-03T08:37:53.429 INFO:tasks.workunit.client.0.smithi088.stderr:++++ basename /usr/bin/bash 2024-06-03T08:37:53.429 INFO:tasks.workunit.client.0.smithi088.stderr:+++ '[' -n '' ']' 2024-06-03T08:37:53.429 INFO:tasks.workunit.client.0.smithi088.stderr:+++ LANG_backup=en_US.UTF-8 2024-06-03T08:37:53.429 INFO:tasks.workunit.client.0.smithi088.stderr:+++ for config in /etc/locale.conf "${HOME}/.i18n" 2024-06-03T08:37:53.429 INFO:tasks.workunit.client.0.smithi088.stderr:+++ '[' -f /etc/locale.conf ']' 2024-06-03T08:37:53.430 INFO:tasks.workunit.client.0.smithi088.stderr:+++ '[' -x /usr/bin/sed ']' 2024-06-03T08:37:53.430 INFO:tasks.workunit.client.0.smithi088.stderr:++++ /usr/bin/sed -r -e 's/^[[:blank:]]*([[:upper:]_]+)=([[:print:][:digit:]\._-]+|"[[:print:][:digit:]\._-]+")/export \1=\2/;t;d' /etc/locale.conf 2024-06-03T08:37:53.430 INFO:tasks.workunit.client.0.smithi088.stderr:+++ which_declare='declare -f' 2024-06-03T08:37:53.430 INFO:tasks.workunit.client.0.smithi088.stderr:+++ which_opt=-f 2024-06-03T08:37:53.430 INFO:tasks.workunit.client.0.smithi088.stderr:+++ export which_declare 2024-06-03T08:37:53.430 INFO:tasks.workunit.client.0.smithi088.stderr:+++ export -f which 2024-06-03T08:37:53.430 INFO:tasks.workunit.client.0.smithi088.stderr:++ unset i 2024-06-03T08:37:53.430 INFO:tasks.workunit.client.0.smithi088.stderr:++ unset -f pathmunge 2024-06-03T08:37:53.430 INFO:tasks.workunit.client.0.smithi088.stderr:+ [[ /home/ubuntu/.local/bin:/home/ubuntu/bin:/usr/local/bin:/usr/bin:/usr/local/sbin:/usr/sbin:/usr/sbin =~ /home/ubuntu/\.local/bin:/home/ubuntu/bin: ]] 2024-06-03T08:37:53.430 INFO:tasks.workunit.client.0.smithi088.stderr:+ export PATH 2024-06-03T08:37:53.430 INFO:tasks.workunit.client.0.smithi088.stderr:+ '[' -d /home/ubuntu/.bashrc.d ']' 2024-06-03T08:37:53.430 INFO:tasks.workunit.client.0.smithi088.stderr:+ unset rc 2024-06-03T08:37:53.430 INFO:tasks.workunit.client.0.smithi088.stderr:+++ alias 'grep=grep --color=auto' 2024-06-03T08:37:53.431 INFO:tasks.workunit.client.0.smithi088.stderr:+++ alias 'egrep=egrep --color=auto' 2024-06-03T08:37:53.431 INFO:tasks.workunit.client.0.smithi088.stderr:+++ alias 'fgrep=fgrep --color=auto' 2024-06-03T08:37:53.431 INFO:tasks.workunit.client.0.smithi088.stderr:++ for i in /etc/profile.d/*.sh 2024-06-03T08:37:53.431 INFO:tasks.workunit.client.0.smithi088.stderr:+++ alias 'zgrep=zgrep --color=auto' 2024-06-03T08:37:53.431 INFO:tasks.workunit.client.0.smithi088.stderr:+ bash -o pipefail -exc 'ceph_test_rados_api_misc 2>&1 | tee ceph_test_rados_api_misc.log | sed "s/^/ api_misc: /"' 2024-06-03T08:37:53.431 INFO:tasks.workunit.client.0.smithi088.stderr:+++ alias 'zfgrep=zfgrep --color=auto' 2024-06-03T08:37:53.431 INFO:tasks.workunit.client.0.smithi088.stderr:+++ alias 'zegrep=zegrep --color=auto' 2024-06-03T08:37:53.431 INFO:tasks.workunit.client.0.smithi088.stderr:++ for i in /etc/profile.d/*.sh 2024-06-03T08:37:53.431 INFO:tasks.workunit.client.0.smithi088.stderr:++ '[' -r /etc/profile.d/debuginfod.sh ']' 2024-06-03T08:37:53.431 INFO:tasks.workunit.client.0.smithi088.stderr:++ '[' '' ']' 2024-06-03T08:37:53.431 INFO:tasks.workunit.client.0.smithi088.stderr:+++ eval export 'LANG="en_US.UTF-8"' 2024-06-03T08:37:53.431 INFO:tasks.workunit.client.0.smithi088.stderr:++ . /etc/profile.d/debuginfod.sh 2024-06-03T08:37:53.431 INFO:tasks.workunit.client.0.smithi088.stderr:++++ export LANG=en_US.UTF-8 2024-06-03T08:37:53.432 INFO:tasks.workunit.client.0.smithi088.stderr:++++ LANG=en_US.UTF-8 2024-06-03T08:37:53.432 INFO:tasks.workunit.client.0.smithi088.stderr:+++ for config in /etc/locale.conf "${HOME}/.i18n" 2024-06-03T08:37:53.432 INFO:tasks.workunit.client.0.smithi088.stderr:+++ '[' -z 'https://debuginfod.centos.org/ ' ']' 2024-06-03T08:37:53.432 INFO:tasks.workunit.client.0.smithi088.stderr:+ sed 's/^/ api_io_pp: /' 2024-06-03T08:37:53.432 INFO:tasks.workunit.client.0.smithi088.stderr:+++ '[' -f /home/ubuntu/.i18n ']' 2024-06-03T08:37:53.432 INFO:tasks.workunit.client.0.smithi088.stderr:++ for i in /etc/profile.d/*.sh 2024-06-03T08:37:53.432 INFO:tasks.workunit.client.0.smithi088.stderr:++ '[' -r /etc/profile.d/gawk.sh ']' 2024-06-03T08:37:53.432 INFO:tasks.workunit.client.0.smithi088.stderr:++ '[' '' ']' 2024-06-03T08:37:53.432 INFO:tasks.workunit.client.0.smithi088.stderr:+++ '[' -n en_US.UTF-8 ']' 2024-06-03T08:37:53.432 INFO:tasks.workunit.client.0.smithi088.stderr:++ . /etc/profile.d/gawk.sh 2024-06-03T08:37:53.432 INFO:tasks.workunit.client.0.smithi088.stderr:+++ LANG=en_US.UTF-8 2024-06-03T08:37:53.432 INFO:tasks.workunit.client.0.smithi088.stderr:+++ unset LANG_backup config 2024-06-03T08:37:53.432 INFO:tasks.workunit.client.0.smithi088.stderr:+++ '[' -n '' ']' 2024-06-03T08:37:53.432 INFO:tasks.workunit.client.0.smithi088.stderr:++ for i in /etc/profile.d/*.sh 2024-06-03T08:37:53.433 INFO:tasks.workunit.client.0.smithi088.stderr:++ '[' -r /etc/profile.d/lang.sh ']' 2024-06-03T08:37:53.433 INFO:tasks.workunit.client.0.smithi088.stderr:++ '[' '' ']' 2024-06-03T08:37:53.433 INFO:tasks.workunit.client.0.smithi088.stderr:+++ '[' -n en_US.UTF-8 ']' 2024-06-03T08:37:53.433 INFO:tasks.workunit.client.0.smithi088.stderr:++ . /etc/profile.d/lang.sh 2024-06-03T08:37:53.433 INFO:tasks.workunit.client.0.smithi088.stderr:+++ '[' dumb = linux ']' 2024-06-03T08:37:53.433 INFO:tasks.workunit.client.0.smithi088.stderr:++ for i in /etc/profile.d/*.sh 2024-06-03T08:37:53.433 INFO:tasks.workunit.client.0.smithi088.stderr:+++ unset LANG_backup 2024-06-03T08:37:53.433 INFO:tasks.workunit.client.0.smithi088.stderr:++ '[' -r /etc/profile.d/less.sh ']' 2024-06-03T08:37:53.433 INFO:tasks.workunit.client.0.smithi088.stderr:++ '[' '' ']' 2024-06-03T08:37:53.433 INFO:tasks.workunit.client.0.smithi088.stderr:++ . /etc/profile.d/less.sh 2024-06-03T08:37:53.433 INFO:tasks.workunit.client.0.smithi088.stderr:+++ '[' -z '||/usr/bin/lesspipe.sh %s' ']' 2024-06-03T08:37:53.433 INFO:tasks.workunit.client.0.smithi088.stderr:++ for i in /etc/profile.d/*.sh 2024-06-03T08:37:53.433 INFO:tasks.workunit.client.0.smithi088.stderr:++ '[' -r /etc/profile.d/which2.sh ']' 2024-06-03T08:37:53.434 INFO:tasks.workunit.client.0.smithi088.stderr:++ '[' '' ']' 2024-06-03T08:37:53.434 INFO:tasks.workunit.client.0.smithi088.stderr:++ . /etc/profile.d/which2.sh 2024-06-03T08:37:53.434 INFO:tasks.workunit.client.0.smithi088.stderr:+++ case "$(basename $(readlink /proc/$$/exe))" in 2024-06-03T08:37:53.434 INFO:tasks.workunit.client.0.smithi088.stderr:++ '[' -r /etc/profile.d/colorls.sh ']' 2024-06-03T08:37:53.434 INFO:tasks.workunit.client.0.smithi088.stderr:++ '[' '' ']' 2024-06-03T08:37:53.434 INFO:tasks.workunit.client.0.smithi088.stderr:+++ alias 'xzgrep=xzgrep --color=auto' 2024-06-03T08:37:53.434 INFO:tasks.workunit.client.0.smithi088.stderr:+++ alias 'xzegrep=xzegrep --color=auto' 2024-06-03T08:37:53.434 INFO:tasks.workunit.client.0.smithi088.stderr:++ . /etc/profile.d/colorls.sh 2024-06-03T08:37:53.434 INFO:tasks.workunit.client.0.smithi088.stderr:+++ '[' '!' -t 0 ']' 2024-06-03T08:37:53.434 INFO:tasks.workunit.client.0.smithi088.stderr:+++ alias 'xzfgrep=xzfgrep --color=auto' 2024-06-03T08:37:53.434 INFO:tasks.workunit.client.0.smithi088.stderr:++ for i in /etc/profile.d/*.sh 2024-06-03T08:37:53.434 INFO:tasks.workunit.client.0.smithi088.stderr:+++ return 2024-06-03T08:37:53.434 INFO:tasks.workunit.client.0.smithi088.stderr:++ for i in /etc/profile.d/*.sh 2024-06-03T08:37:53.435 INFO:tasks.workunit.client.0.smithi088.stderr:++ '[' -r /etc/profile.d/colorsysstat.sh ']' 2024-06-03T08:37:53.435 INFO:tasks.workunit.client.0.smithi088.stderr:++ '[' '' ']' 2024-06-03T08:37:53.435 INFO:tasks.workunit.client.0.smithi088.stderr:++ . /etc/profile.d/colorsysstat.sh 2024-06-03T08:37:53.435 INFO:tasks.workunit.client.0.smithi088.stderr:+++ export S_COLORS=auto 2024-06-03T08:37:53.435 INFO:tasks.workunit.client.0.smithi088.stderr:++ '[' -r /etc/profile.d/colorzgrep.sh ']' 2024-06-03T08:37:53.435 INFO:tasks.workunit.client.0.smithi088.stderr:++ '[' '' ']' 2024-06-03T08:37:53.435 INFO:tasks.workunit.client.0.smithi088.stderr:++ . /etc/profile.d/colorzgrep.sh 2024-06-03T08:37:53.435 INFO:tasks.workunit.client.0.smithi088.stderr:+++ '[' -f /usr/libexec/grepconf.sh ']' 2024-06-03T08:37:53.435 INFO:tasks.workunit.client.0.smithi088.stderr:+++ /usr/libexec/grepconf.sh -c 2024-06-03T08:37:53.435 INFO:tasks.workunit.client.0.smithi088.stderr:+++ S_COLORS=auto 2024-06-03T08:37:53.435 INFO:tasks.workunit.client.0.smithi088.stderr:++ for i in /etc/profile.d/*.sh 2024-06-03T08:37:53.435 INFO:tasks.workunit.client.0.smithi088.stderr:++++ /usr/bin/locale 2024-06-03T08:37:53.435 INFO:tasks.workunit.client.0.smithi088.stderr:++ '[' -r /etc/profile.d/colorxzgrep.sh ']' 2024-06-03T08:37:53.436 INFO:tasks.workunit.client.0.smithi088.stderr:++ '[' '' ']' 2024-06-03T08:37:53.436 INFO:tasks.workunit.client.0.smithi088.stderr:++ . /etc/profile.d/colorxzgrep.sh 2024-06-03T08:37:53.436 INFO:tasks.workunit.client.0.smithi088.stderr:+++ /usr/libexec/grepconf.sh -c 2024-06-03T08:37:53.436 INFO:tasks.workunit.client.0.smithi088.stderr:+ ceph_test_rados_api_io_pp 2024-06-03T08:37:53.436 INFO:tasks.workunit.client.0.smithi088.stderr:+ '[' -f /etc/bashrc ']' 2024-06-03T08:37:53.436 INFO:tasks.workunit.client.0.smithi088.stderr:+ . /etc/bashrc 2024-06-03T08:37:53.436 INFO:tasks.workunit.client.0.smithi088.stderr:++ '[' -z '' ']' 2024-06-03T08:37:53.436 INFO:tasks.workunit.client.0.smithi088.stderr:++ BASHRCSOURCED=Y 2024-06-03T08:37:53.436 INFO:tasks.workunit.client.0.smithi088.stderr:++ '[' '' ']' 2024-06-03T08:37:53.436 INFO:tasks.workunit.client.0.smithi088.stderr:++ shopt -q login_shell 2024-06-03T08:37:53.436 INFO:tasks.workunit.client.0.smithi088.stderr:+++ '[' -n '' ']' 2024-06-03T08:37:53.436 INFO:tasks.workunit.client.0.smithi088.stderr:+++ LANG_backup=en_US.UTF-8 2024-06-03T08:37:53.436 INFO:tasks.workunit.client.0.smithi088.stderr:+++ for config in /etc/locale.conf "${HOME}/.i18n" 2024-06-03T08:37:53.437 INFO:tasks.workunit.client.0.smithi088.stderr:+++ '[' -f /etc/locale.conf ']' 2024-06-03T08:37:53.437 INFO:tasks.workunit.client.0.smithi088.stderr:+++ '[' -x /usr/bin/sed ']' 2024-06-03T08:37:53.437 INFO:tasks.workunit.client.0.smithi088.stderr:+++++ readlink /proc/40246/exe 2024-06-03T08:37:53.437 INFO:tasks.workunit.client.0.smithi088.stderr:++ echo api_misc_pp 2024-06-03T08:37:53.437 INFO:tasks.workunit.client.0.smithi088.stderr:++ awk '{print $1}' 2024-06-03T08:37:53.437 INFO:tasks.workunit.client.0.smithi088.stderr:++++ /usr/bin/sed -r -e 's/^[[:blank:]]*([[:upper:]_]+)=([[:print:][:digit:]\._-]+|"[[:print:][:digit:]\._-]+")/export \1=\2/;t;d' /etc/locale.conf 2024-06-03T08:37:53.437 INFO:tasks.workunit.client.0.smithi088.stderr:++++ basename /usr/bin/bash 2024-06-03T08:37:53.437 INFO:tasks.workunit.client.0.smithi088.stderr:+++ umask 2024-06-03T08:37:53.437 INFO:tasks.workunit.client.0.smithi088.stderr:+++ alias 'xzgrep=xzgrep --color=auto' 2024-06-03T08:37:53.437 INFO:tasks.workunit.client.0.smithi088.stderr:+++ which_declare='declare -f' 2024-06-03T08:37:53.437 INFO:tasks.workunit.client.0.smithi088.stderr:+++ alias 'xzegrep=xzegrep --color=auto' 2024-06-03T08:37:53.437 INFO:tasks.workunit.client.0.smithi088.stderr:+++ alias 'xzfgrep=xzfgrep --color=auto' 2024-06-03T08:37:53.437 INFO:tasks.workunit.client.0.smithi088.stderr:++ for i in /etc/profile.d/*.sh 2024-06-03T08:37:53.437 INFO:tasks.workunit.client.0.smithi088.stderr:++ '[' -r /etc/profile.d/colorzgrep.sh ']' 2024-06-03T08:37:53.438 INFO:tasks.workunit.client.0.smithi088.stderr:++ '[' '' ']' 2024-06-03T08:37:53.438 INFO:tasks.workunit.client.0.smithi088.stderr:++ . /etc/profile.d/colorzgrep.sh 2024-06-03T08:37:53.438 INFO:tasks.workunit.client.0.smithi088.stderr:++ '[' 0022 -eq 0 ']' 2024-06-03T08:37:53.438 INFO:tasks.workunit.client.0.smithi088.stderr:++ SHELL=/bin/bash 2024-06-03T08:37:53.438 INFO:tasks.workunit.client.0.smithi088.stderr:+++ '[' -f /usr/libexec/grepconf.sh ']' 2024-06-03T08:37:53.438 INFO:tasks.workunit.client.0.smithi088.stderr:+++ /usr/libexec/grepconf.sh -c 2024-06-03T08:37:53.438 INFO:tasks.workunit.client.0.smithi088.stderr:+++ which_opt=-f 2024-06-03T08:37:53.438 INFO:tasks.workunit.client.0.smithi088.stderr:++ for i in /etc/profile.d/*.sh 2024-06-03T08:37:53.438 INFO:tasks.workunit.client.0.smithi088.stderr:+++ eval export 'LANG="en_US.UTF-8"' 2024-06-03T08:37:53.438 INFO:tasks.workunit.client.0.smithi088.stderr:++++ export LANG=en_US.UTF-8 2024-06-03T08:37:53.438 INFO:tasks.workunit.client.0.smithi088.stderr:++++ LANG=en_US.UTF-8 2024-06-03T08:37:53.438 INFO:tasks.workunit.client.0.smithi088.stderr:+++ export which_declare 2024-06-03T08:37:53.438 INFO:tasks.workunit.client.0.smithi088.stderr:+++ export -f which 2024-06-03T08:37:53.439 INFO:tasks.workunit.client.0.smithi088.stderr:+++ for config in /etc/locale.conf "${HOME}/.i18n" 2024-06-03T08:37:53.439 INFO:tasks.workunit.client.0.smithi088.stderr:++ unset i 2024-06-03T08:37:53.439 INFO:tasks.workunit.client.0.smithi088.stderr:++ '[' -r /etc/profile.d/bash_completion.sh ']' 2024-06-03T08:37:53.439 INFO:tasks.workunit.client.0.smithi088.stderr:++ unset -f pathmunge 2024-06-03T08:37:53.439 INFO:tasks.workunit.client.0.smithi088.stderr:++ '[' '' ']' 2024-06-03T08:37:53.439 INFO:tasks.workunit.client.0.smithi088.stderr:++ . /etc/profile.d/bash_completion.sh 2024-06-03T08:37:53.439 INFO:tasks.workunit.client.0.smithi088.stderr:+ [[ /home/ubuntu/.local/bin:/home/ubuntu/bin:/usr/local/bin:/usr/bin:/usr/local/sbin:/usr/sbin:/usr/sbin =~ /home/ubuntu/\.local/bin:/home/ubuntu/bin: ]] 2024-06-03T08:37:53.439 INFO:tasks.workunit.client.0.smithi088.stderr:+++ '[' -f /home/ubuntu/.i18n ']' 2024-06-03T08:37:53.439 INFO:tasks.workunit.client.0.smithi088.stderr:+++ '[' -n en_US.UTF-8 ']' 2024-06-03T08:37:53.439 INFO:tasks.workunit.client.0.smithi088.stderr:+++ LANG=en_US.UTF-8 2024-06-03T08:37:53.439 INFO:tasks.workunit.client.0.smithi088.stderr:+++ unset LANG_backup config 2024-06-03T08:37:53.439 INFO:tasks.workunit.client.0.smithi088.stderr:+++ '[' 'x5.1.8(1)-release' '!=' x -a x '!=' x -a x = x ']' 2024-06-03T08:37:53.439 INFO:tasks.workunit.client.0.smithi088.stderr:++ for i in /etc/profile.d/*.sh 2024-06-03T08:37:53.440 INFO:tasks.workunit.client.0.smithi088.stderr:++ '[' -r /etc/profile.d/cobbler.sh ']' 2024-06-03T08:37:53.440 INFO:tasks.workunit.client.0.smithi088.stderr:+++ '[' -n '' ']' 2024-06-03T08:37:53.440 INFO:tasks.workunit.client.0.smithi088.stderr:++ '[' '' ']' 2024-06-03T08:37:53.440 INFO:tasks.workunit.client.0.smithi088.stderr:++ . /etc/profile.d/cobbler.sh 2024-06-03T08:37:53.440 INFO:tasks.workunit.client.0.smithi088.stderr:+ ff=api_misc_pp 2024-06-03T08:37:53.440 INFO:tasks.workunit.client.0.smithi088.stderr:+++ export COBBLER_SERVER=172.21.0.11 2024-06-03T08:37:53.440 INFO:tasks.workunit.client.0.smithi088.stderr:+++ COBBLER_SERVER=172.21.0.11 2024-06-03T08:37:53.440 INFO:tasks.workunit.client.0.smithi088.stderr:++ for i in /etc/profile.d/*.sh 2024-06-03T08:37:53.440 INFO:tasks.workunit.client.0.smithi088.stderr:++ '[' -r /etc/profile.d/colorgrep.sh ']' 2024-06-03T08:37:53.440 INFO:tasks.workunit.client.0.smithi088.stderr:++ '[' '' ']' 2024-06-03T08:37:53.440 INFO:tasks.workunit.client.0.smithi088.stderr:+++ '[' -n en_US.UTF-8 ']' 2024-06-03T08:37:53.440 INFO:tasks.workunit.client.0.smithi088.stderr:++ . /etc/profile.d/colorgrep.sh 2024-06-03T08:37:53.440 INFO:tasks.workunit.client.0.smithi088.stderr:+++ '[' dumb = linux ']' 2024-06-03T08:37:53.440 INFO:tasks.workunit.client.0.smithi088.stderr:+++ /usr/libexec/grepconf.sh -c 2024-06-03T08:37:53.441 INFO:tasks.workunit.client.0.smithi088.stderr:++ for i in /etc/profile.d/*.sh 2024-06-03T08:37:53.441 INFO:tasks.workunit.client.0.smithi088.stderr:++ '[' -r /etc/profile.d/less.sh ']' 2024-06-03T08:37:53.441 INFO:tasks.workunit.client.0.smithi088.stderr:++ '[' '' ']' 2024-06-03T08:37:53.441 INFO:tasks.workunit.client.0.smithi088.stderr:++ . /etc/profile.d/less.sh 2024-06-03T08:37:53.441 INFO:tasks.workunit.client.0.smithi088.stderr:+ export PATH 2024-06-03T08:37:53.441 INFO:tasks.workunit.client.0.smithi088.stderr:+ '[' -d /home/ubuntu/.bashrc.d ']' 2024-06-03T08:37:53.441 INFO:tasks.workunit.client.0.smithi088.stderr:+++ '[' -z '||/usr/bin/lesspipe.sh %s' ']' 2024-06-03T08:37:53.441 INFO:tasks.workunit.client.0.smithi088.stderr:++ for i in /etc/profile.d/*.sh 2024-06-03T08:37:53.441 INFO:tasks.workunit.client.0.smithi088.stderr:++ '[' -r /etc/profile.d/which2.sh ']' 2024-06-03T08:37:53.441 INFO:tasks.workunit.client.0.smithi088.stderr:++ '[' '' ']' 2024-06-03T08:37:53.441 INFO:tasks.workunit.client.0.smithi088.stderr:++ . /etc/profile.d/which2.sh 2024-06-03T08:37:53.441 INFO:tasks.workunit.client.0.smithi088.stderr:+++ case "$(basename $(readlink /proc/$$/exe))" in 2024-06-03T08:37:53.441 INFO:tasks.workunit.client.0.smithi088.stderr:+ unset rc 2024-06-03T08:37:53.441 INFO:tasks.workunit.client.0.smithi088.stderr:+ pid=40347 2024-06-03T08:37:53.442 INFO:tasks.workunit.client.0.smithi088.stdout:test api_misc_pp on pid 40347 2024-06-03T08:37:53.447 INFO:tasks.workunit.client.0.smithi088.stderr:+ echo 'test api_misc_pp on pid 40347' 2024-06-03T08:37:53.447 INFO:tasks.workunit.client.0.smithi088.stderr:+ pids[$f]=40347 2024-06-03T08:37:53.447 INFO:tasks.workunit.client.0.smithi088.stderr:+ for f in api_aio api_aio_pp api_io api_io_pp api_asio api_list api_lock api_lock_pp api_misc api_misc_pp api_tier_pp api_pool api_snapshots api_snapshots_pp api_stat api_stat_pp api_watch_notify api_watch_notify_pp api_cmd api_cmd_pp api_service api_service_pp api_c_write_operations api_c_read_operations api_cls_remote_reads list_parallel open_pools_parallel delete_pools_parallel 2024-06-03T08:37:53.447 INFO:tasks.workunit.client.0.smithi088.stderr:+ '[' 1 -eq 1 ']' 2024-06-03T08:37:53.447 INFO:tasks.workunit.client.0.smithi088.stderr:+ bash -o pipefail -exc 'ceph_test_rados_api_misc_pp 2>&1 | tee ceph_test_rados_api_misc_pp.log | sed "s/^/ api_misc_pp: /"' 2024-06-03T08:37:53.447 INFO:tasks.workunit.client.0.smithi088.stderr:++ printf %25s api_tier_pp 2024-06-03T08:37:53.447 INFO:tasks.workunit.client.0.smithi088.stderr:+ sed 's/^/ api_asio: /' 2024-06-03T08:37:53.447 INFO:tasks.workunit.client.0.smithi088.stderr:+ tee ceph_test_rados_api_asio.log 2024-06-03T08:37:53.447 INFO:tasks.workunit.client.0.smithi088.stderr:+ '[' -f /etc/bashrc ']' 2024-06-03T08:37:53.447 INFO:tasks.workunit.client.0.smithi088.stderr:+ . /etc/bashrc 2024-06-03T08:37:53.448 INFO:tasks.workunit.client.0.smithi088.stderr:+ r=' api_tier_pp' 2024-06-03T08:37:53.448 INFO:tasks.workunit.client.0.smithi088.stderr:+++++ readlink /proc/40263/exe 2024-06-03T08:37:53.448 INFO:tasks.workunit.client.0.smithi088.stderr:++ '[' -z '' ']' 2024-06-03T08:37:53.448 INFO:tasks.workunit.client.0.smithi088.stderr:++ BASHRCSOURCED=Y 2024-06-03T08:37:53.448 INFO:tasks.workunit.client.0.smithi088.stderr:++ '[' '' ']' 2024-06-03T08:37:53.448 INFO:tasks.workunit.client.0.smithi088.stderr:++ shopt -q login_shell 2024-06-03T08:37:53.448 INFO:tasks.workunit.client.0.smithi088.stderr:++ echo api_tier_pp 2024-06-03T08:37:53.448 INFO:tasks.workunit.client.0.smithi088.stderr:+++ umask 2024-06-03T08:37:53.448 INFO:tasks.workunit.client.0.smithi088.stderr:++++ basename /usr/bin/bash 2024-06-03T08:37:53.448 INFO:tasks.workunit.client.0.smithi088.stderr:++ '[' 0022 -eq 0 ']' 2024-06-03T08:37:53.448 INFO:tasks.workunit.client.0.smithi088.stderr:++ SHELL=/bin/bash 2024-06-03T08:37:53.448 INFO:tasks.workunit.client.0.smithi088.stderr:++ for i in /etc/profile.d/*.sh 2024-06-03T08:37:53.448 INFO:tasks.workunit.client.0.smithi088.stderr:++ '[' -r /etc/profile.d/bash_completion.sh ']' 2024-06-03T08:37:53.448 INFO:tasks.workunit.client.0.smithi088.stderr:++ '[' '' ']' 2024-06-03T08:37:53.449 INFO:tasks.workunit.client.0.smithi088.stderr:++ . /etc/profile.d/bash_completion.sh 2024-06-03T08:37:53.449 INFO:tasks.workunit.client.0.smithi088.stderr:+++ alias 'grep=grep --color=auto' 2024-06-03T08:37:53.449 INFO:tasks.workunit.client.0.smithi088.stderr:+++ '[' 'x5.1.8(1)-release' '!=' x -a x '!=' x -a x = x ']' 2024-06-03T08:37:53.449 INFO:tasks.workunit.client.0.smithi088.stderr:++ for i in /etc/profile.d/*.sh 2024-06-03T08:37:53.449 INFO:tasks.workunit.client.0.smithi088.stderr:++ '[' -r /etc/profile.d/cobbler.sh ']' 2024-06-03T08:37:53.449 INFO:tasks.workunit.client.0.smithi088.stderr:++ '[' '' ']' 2024-06-03T08:37:53.449 INFO:tasks.workunit.client.0.smithi088.stderr:++ . /etc/profile.d/cobbler.sh 2024-06-03T08:37:53.449 INFO:tasks.workunit.client.0.smithi088.stderr:+++ export COBBLER_SERVER=172.21.0.11 2024-06-03T08:37:53.449 INFO:tasks.workunit.client.0.smithi088.stderr:+++ alias 'egrep=egrep --color=auto' 2024-06-03T08:37:53.449 INFO:tasks.workunit.client.0.smithi088.stderr:+++ alias 'fgrep=fgrep --color=auto' 2024-06-03T08:37:53.449 INFO:tasks.workunit.client.0.smithi088.stderr:+++ COBBLER_SERVER=172.21.0.11 2024-06-03T08:37:53.449 INFO:tasks.workunit.client.0.smithi088.stderr:++ for i in /etc/profile.d/*.sh 2024-06-03T08:37:53.449 INFO:tasks.workunit.client.0.smithi088.stderr:++ '[' -r /etc/profile.d/colorgrep.sh ']' 2024-06-03T08:37:53.450 INFO:tasks.workunit.client.0.smithi088.stderr:++ '[' '' ']' 2024-06-03T08:37:53.450 INFO:tasks.workunit.client.0.smithi088.stderr:++ . /etc/profile.d/colorgrep.sh 2024-06-03T08:37:53.450 INFO:tasks.workunit.client.0.smithi088.stderr:+++ /usr/libexec/grepconf.sh -c 2024-06-03T08:37:53.450 INFO:tasks.workunit.client.0.smithi088.stderr:++ for i in /etc/profile.d/*.sh 2024-06-03T08:37:53.450 INFO:tasks.workunit.client.0.smithi088.stderr:++ '[' -r /etc/profile.d/colorls.sh ']' 2024-06-03T08:37:53.450 INFO:tasks.workunit.client.0.smithi088.stderr:++ '[' '' ']' 2024-06-03T08:37:53.450 INFO:tasks.workunit.client.0.smithi088.stderr:+++ which_declare='declare -f' 2024-06-03T08:37:53.450 INFO:tasks.workunit.client.0.smithi088.stderr:+++ which_opt=-f 2024-06-03T08:37:53.450 INFO:tasks.workunit.client.0.smithi088.stderr:++ . /etc/profile.d/colorls.sh 2024-06-03T08:37:53.450 INFO:tasks.workunit.client.0.smithi088.stderr:+++ export which_declare 2024-06-03T08:37:53.450 INFO:tasks.workunit.client.0.smithi088.stderr:+++ export -f which 2024-06-03T08:37:53.450 INFO:tasks.workunit.client.0.smithi088.stderr:++ unset i 2024-06-03T08:37:53.450 INFO:tasks.workunit.client.0.smithi088.stderr:++ unset -f pathmunge 2024-06-03T08:37:53.451 INFO:tasks.workunit.client.0.smithi088.stderr:++ awk '{print $1}' 2024-06-03T08:37:53.451 INFO:tasks.workunit.client.0.smithi088.stderr:+ [[ /home/ubuntu/.local/bin:/home/ubuntu/bin:/usr/local/bin:/usr/bin:/usr/local/sbin:/usr/sbin:/usr/sbin =~ /home/ubuntu/\.local/bin:/home/ubuntu/bin: ]] 2024-06-03T08:37:53.451 INFO:tasks.workunit.client.0.smithi088.stderr:+++ '[' '!' -t 0 ']' 2024-06-03T08:37:53.451 INFO:tasks.workunit.client.0.smithi088.stderr:+++ return 2024-06-03T08:37:53.451 INFO:tasks.workunit.client.0.smithi088.stderr:++ for i in /etc/profile.d/*.sh 2024-06-03T08:37:53.451 INFO:tasks.workunit.client.0.smithi088.stderr:++ '[' -r /etc/profile.d/colorsysstat.sh ']' 2024-06-03T08:37:53.451 INFO:tasks.workunit.client.0.smithi088.stderr:++ '[' '' ']' 2024-06-03T08:37:53.451 INFO:tasks.workunit.client.0.smithi088.stderr:++ . /etc/profile.d/colorsysstat.sh 2024-06-03T08:37:53.451 INFO:tasks.workunit.client.0.smithi088.stderr:+ export PATH 2024-06-03T08:37:53.451 INFO:tasks.workunit.client.0.smithi088.stderr:+++ export S_COLORS=auto 2024-06-03T08:37:53.451 INFO:tasks.workunit.client.0.smithi088.stderr:+++ S_COLORS=auto 2024-06-03T08:37:53.451 INFO:tasks.workunit.client.0.smithi088.stderr:++ for i in /etc/profile.d/*.sh 2024-06-03T08:37:53.451 INFO:tasks.workunit.client.0.smithi088.stderr:++ '[' -r /etc/profile.d/colorxzgrep.sh ']' 2024-06-03T08:37:53.451 INFO:tasks.workunit.client.0.smithi088.stderr:++ '[' '' ']' 2024-06-03T08:37:53.452 INFO:tasks.workunit.client.0.smithi088.stderr:++ . /etc/profile.d/colorxzgrep.sh 2024-06-03T08:37:53.452 INFO:tasks.workunit.client.0.smithi088.stderr:+++ /usr/libexec/grepconf.sh -c 2024-06-03T08:37:53.452 INFO:tasks.workunit.client.0.smithi088.stderr:+ '[' -d /home/ubuntu/.bashrc.d ']' 2024-06-03T08:37:53.452 INFO:tasks.workunit.client.0.smithi088.stderr:+ unset rc 2024-06-03T08:37:53.452 INFO:tasks.workunit.client.0.smithi088.stderr:+ ceph_test_rados_api_list 2024-06-03T08:37:53.452 INFO:tasks.workunit.client.0.smithi088.stderr:+ sed 's/^/ api_list: /' 2024-06-03T08:37:53.452 INFO:tasks.workunit.client.0.smithi088.stderr:+ ff=api_tier_pp 2024-06-03T08:37:53.452 INFO:tasks.workunit.client.0.smithi088.stderr:+ pid=40367 2024-06-03T08:37:53.452 INFO:tasks.workunit.client.0.smithi088.stdout:test api_tier_pp on pid 40367 2024-06-03T08:37:53.457 INFO:tasks.workunit.client.0.smithi088.stderr:+ echo 'test api_tier_pp on pid 40367' 2024-06-03T08:37:53.457 INFO:tasks.workunit.client.0.smithi088.stderr:+ pids[$f]=40367 2024-06-03T08:37:53.457 INFO:tasks.workunit.client.0.smithi088.stderr:+ for f in api_aio api_aio_pp api_io api_io_pp api_asio api_list api_lock api_lock_pp api_misc api_misc_pp api_tier_pp api_pool api_snapshots api_snapshots_pp api_stat api_stat_pp api_watch_notify api_watch_notify_pp api_cmd api_cmd_pp api_service api_service_pp api_c_write_operations api_c_read_operations api_cls_remote_reads list_parallel open_pools_parallel delete_pools_parallel 2024-06-03T08:37:53.457 INFO:tasks.workunit.client.0.smithi088.stderr:+ '[' 1 -eq 1 ']' 2024-06-03T08:37:53.457 INFO:tasks.workunit.client.0.smithi088.stderr:+++ alias 'grep=grep --color=auto' 2024-06-03T08:37:53.458 INFO:tasks.workunit.client.0.smithi088.stderr:+ tee ceph_test_rados_api_list.log 2024-06-03T08:37:53.458 INFO:tasks.workunit.client.0.smithi088.stderr:+ bash -o pipefail -exc 'ceph_test_rados_api_tier_pp 2>&1 | tee ceph_test_rados_api_tier_pp.log | sed "s/^/ api_tier_pp: /"' 2024-06-03T08:37:53.458 INFO:tasks.workunit.client.0.smithi088.stderr:+++ alias 'egrep=egrep --color=auto' 2024-06-03T08:37:53.458 INFO:tasks.workunit.client.0.smithi088.stderr:+++ alias 'fgrep=fgrep --color=auto' 2024-06-03T08:37:53.458 INFO:tasks.workunit.client.0.smithi088.stderr:++ for i in /etc/profile.d/*.sh 2024-06-03T08:37:53.458 INFO:tasks.workunit.client.0.smithi088.stderr:++ '[' -r /etc/profile.d/colorls.sh ']' 2024-06-03T08:37:53.458 INFO:tasks.workunit.client.0.smithi088.stderr:++ '[' '' ']' 2024-06-03T08:37:53.458 INFO:tasks.workunit.client.0.smithi088.stderr:++ printf %25s api_pool 2024-06-03T08:37:53.458 INFO:tasks.workunit.client.0.smithi088.stderr:++ . /etc/profile.d/colorls.sh 2024-06-03T08:37:53.458 INFO:tasks.workunit.client.0.smithi088.stderr:+++ '[' '!' -t 0 ']' 2024-06-03T08:37:53.458 INFO:tasks.workunit.client.0.smithi088.stderr:+++ return 2024-06-03T08:37:53.458 INFO:tasks.workunit.client.0.smithi088.stderr:++ for i in /etc/profile.d/*.sh 2024-06-03T08:37:53.458 INFO:tasks.workunit.client.0.smithi088.stderr:++ '[' -r /etc/profile.d/colorsysstat.sh ']' 2024-06-03T08:37:53.459 INFO:tasks.workunit.client.0.smithi088.stderr:++ '[' '' ']' 2024-06-03T08:37:53.459 INFO:tasks.workunit.client.0.smithi088.stderr:+ r=' api_pool' 2024-06-03T08:37:53.459 INFO:tasks.workunit.client.0.smithi088.stderr:++ . /etc/profile.d/colorsysstat.sh 2024-06-03T08:37:53.459 INFO:tasks.workunit.client.0.smithi088.stderr:+++ export S_COLORS=auto 2024-06-03T08:37:53.459 INFO:tasks.workunit.client.0.smithi088.stderr:+++ S_COLORS=auto 2024-06-03T08:37:53.459 INFO:tasks.workunit.client.0.smithi088.stderr:++ for i in /etc/profile.d/*.sh 2024-06-03T08:37:53.459 INFO:tasks.workunit.client.0.smithi088.stderr:++ '[' -r /etc/profile.d/colorxzgrep.sh ']' 2024-06-03T08:37:53.459 INFO:tasks.workunit.client.0.smithi088.stderr:++ '[' '' ']' 2024-06-03T08:37:53.459 INFO:tasks.workunit.client.0.smithi088.stderr:++ . /etc/profile.d/colorxzgrep.sh 2024-06-03T08:37:53.459 INFO:tasks.workunit.client.0.smithi088.stderr:+++ /usr/libexec/grepconf.sh -c 2024-06-03T08:37:53.459 INFO:tasks.workunit.client.0.smithi088.stderr:++ awk '{print $1}' 2024-06-03T08:37:53.459 INFO:tasks.workunit.client.0.smithi088.stderr:+ '[' -f /etc/bashrc ']' 2024-06-03T08:37:53.459 INFO:tasks.workunit.client.0.smithi088.stderr:+ . /etc/bashrc 2024-06-03T08:37:53.460 INFO:tasks.workunit.client.0.smithi088.stderr:++ '[' -z '' ']' 2024-06-03T08:37:53.460 INFO:tasks.workunit.client.0.smithi088.stderr:++ BASHRCSOURCED=Y 2024-06-03T08:37:53.460 INFO:tasks.workunit.client.0.smithi088.stderr:++ '[' '' ']' 2024-06-03T08:37:53.460 INFO:tasks.workunit.client.0.smithi088.stderr:++ shopt -q login_shell 2024-06-03T08:37:53.460 INFO:tasks.workunit.client.0.smithi088.stderr:++ echo api_pool 2024-06-03T08:37:53.460 INFO:tasks.workunit.client.0.smithi088.stderr:+++ umask 2024-06-03T08:37:53.460 INFO:tasks.workunit.client.0.smithi088.stderr:++ '[' 0022 -eq 0 ']' 2024-06-03T08:37:53.460 INFO:tasks.workunit.client.0.smithi088.stderr:++ SHELL=/bin/bash 2024-06-03T08:37:53.460 INFO:tasks.workunit.client.0.smithi088.stderr:++ for i in /etc/profile.d/*.sh 2024-06-03T08:37:53.460 INFO:tasks.workunit.client.0.smithi088.stderr:++ '[' -r /etc/profile.d/bash_completion.sh ']' 2024-06-03T08:37:53.460 INFO:tasks.workunit.client.0.smithi088.stderr:++ '[' '' ']' 2024-06-03T08:37:53.460 INFO:tasks.workunit.client.0.smithi088.stderr:++ . /etc/profile.d/bash_completion.sh 2024-06-03T08:37:53.460 INFO:tasks.workunit.client.0.smithi088.stderr:+ ff=api_pool 2024-06-03T08:37:53.460 INFO:tasks.workunit.client.0.smithi088.stderr:+++ '[' 'x5.1.8(1)-release' '!=' x -a x '!=' x -a x = x ']' 2024-06-03T08:37:53.461 INFO:tasks.workunit.client.0.smithi088.stderr:++ for i in /etc/profile.d/*.sh 2024-06-03T08:37:53.461 INFO:tasks.workunit.client.0.smithi088.stderr:++ '[' -r /etc/profile.d/cobbler.sh ']' 2024-06-03T08:37:53.461 INFO:tasks.workunit.client.0.smithi088.stderr:+ pid=40375 2024-06-03T08:37:53.461 INFO:tasks.workunit.client.0.smithi088.stderr:++ '[' '' ']' 2024-06-03T08:37:53.461 INFO:tasks.workunit.client.0.smithi088.stderr:++ . /etc/profile.d/cobbler.sh 2024-06-03T08:37:53.461 INFO:tasks.workunit.client.0.smithi088.stderr:+++ export COBBLER_SERVER=172.21.0.11 2024-06-03T08:37:53.461 INFO:tasks.workunit.client.0.smithi088.stderr:+++ COBBLER_SERVER=172.21.0.11 2024-06-03T08:37:53.461 INFO:tasks.workunit.client.0.smithi088.stderr:++ for i in /etc/profile.d/*.sh 2024-06-03T08:37:53.461 INFO:tasks.workunit.client.0.smithi088.stderr:++ '[' -r /etc/profile.d/colorgrep.sh ']' 2024-06-03T08:37:53.461 INFO:tasks.workunit.client.0.smithi088.stderr:++ '[' '' ']' 2024-06-03T08:37:53.461 INFO:tasks.workunit.client.0.smithi088.stderr:++ . /etc/profile.d/colorgrep.sh 2024-06-03T08:37:53.461 INFO:tasks.workunit.client.0.smithi088.stderr:+++ /usr/libexec/grepconf.sh -c 2024-06-03T08:37:53.461 INFO:tasks.workunit.client.0.smithi088.stderr:+ echo 'test api_pool on pid 40375' 2024-06-03T08:37:53.462 INFO:tasks.workunit.client.0.smithi088.stdout:test api_pool on pid 40375 2024-06-03T08:37:53.482 INFO:tasks.workunit.client.0.smithi088.stderr:+ pids[$f]=40375 2024-06-03T08:37:53.482 INFO:tasks.workunit.client.0.smithi088.stderr:+ for f in api_aio api_aio_pp api_io api_io_pp api_asio api_list api_lock api_lock_pp api_misc api_misc_pp api_tier_pp api_pool api_snapshots api_snapshots_pp api_stat api_stat_pp api_watch_notify api_watch_notify_pp api_cmd api_cmd_pp api_service api_service_pp api_c_write_operations api_c_read_operations api_cls_remote_reads list_parallel open_pools_parallel delete_pools_parallel 2024-06-03T08:37:53.482 INFO:tasks.workunit.client.0.smithi088.stderr:+ '[' 1 -eq 1 ']' 2024-06-03T08:37:53.482 INFO:tasks.workunit.client.0.smithi088.stderr:+ bash -o pipefail -exc 'ceph_test_rados_api_pool 2>&1 | tee ceph_test_rados_api_pool.log | sed "s/^/ api_pool: /"' 2024-06-03T08:37:53.482 INFO:tasks.workunit.client.0.smithi088.stderr:+++ alias 'xzgrep=xzgrep --color=auto' 2024-06-03T08:37:53.482 INFO:tasks.workunit.client.0.smithi088.stderr:+++ alias 'xzegrep=xzegrep --color=auto' 2024-06-03T08:37:53.482 INFO:tasks.workunit.client.0.smithi088.stderr:+++ alias 'xzfgrep=xzfgrep --color=auto' 2024-06-03T08:37:53.482 INFO:tasks.workunit.client.0.smithi088.stderr:++ for i in /etc/profile.d/*.sh 2024-06-03T08:37:53.482 INFO:tasks.workunit.client.0.smithi088.stderr:++ '[' -r /etc/profile.d/colorzgrep.sh ']' 2024-06-03T08:37:53.483 INFO:tasks.workunit.client.0.smithi088.stderr:++ '[' '' ']' 2024-06-03T08:37:53.483 INFO:tasks.workunit.client.0.smithi088.stderr:++ . /etc/profile.d/colorzgrep.sh 2024-06-03T08:37:53.483 INFO:tasks.workunit.client.0.smithi088.stderr:+ tee ceph_test_rados_api_io_pp.log 2024-06-03T08:37:53.483 INFO:tasks.workunit.client.0.smithi088.stderr:+++ '[' -f /usr/libexec/grepconf.sh ']' 2024-06-03T08:37:53.483 INFO:tasks.workunit.client.0.smithi088.stderr:+++ /usr/libexec/grepconf.sh -c 2024-06-03T08:37:53.483 INFO:tasks.workunit.client.0.smithi088.stderr:+ '[' -f /etc/bashrc ']' 2024-06-03T08:37:53.483 INFO:tasks.workunit.client.0.smithi088.stderr:+ . /etc/bashrc 2024-06-03T08:37:53.483 INFO:tasks.workunit.client.0.smithi088.stderr:+++ alias 'zgrep=zgrep --color=auto' 2024-06-03T08:37:53.483 INFO:tasks.workunit.client.0.smithi088.stderr:+++ alias 'zfgrep=zfgrep --color=auto' 2024-06-03T08:37:53.483 INFO:tasks.workunit.client.0.smithi088.stderr:+++ alias 'zegrep=zegrep --color=auto' 2024-06-03T08:37:53.483 INFO:tasks.workunit.client.0.smithi088.stderr:++ for i in /etc/profile.d/*.sh 2024-06-03T08:37:53.483 INFO:tasks.workunit.client.0.smithi088.stderr:++ '[' -r /etc/profile.d/debuginfod.sh ']' 2024-06-03T08:37:53.483 INFO:tasks.workunit.client.0.smithi088.stderr:++ '[' '' ']' 2024-06-03T08:37:53.484 INFO:tasks.workunit.client.0.smithi088.stderr:++ . /etc/profile.d/debuginfod.sh 2024-06-03T08:37:53.484 INFO:tasks.workunit.client.0.smithi088.stderr:++ '[' -z '' ']' 2024-06-03T08:37:53.484 INFO:tasks.workunit.client.0.smithi088.stderr:++ BASHRCSOURCED=Y 2024-06-03T08:37:53.484 INFO:tasks.workunit.client.0.smithi088.stderr:++ '[' '' ']' 2024-06-03T08:37:53.484 INFO:tasks.workunit.client.0.smithi088.stderr:++ shopt -q login_shell 2024-06-03T08:37:53.484 INFO:tasks.workunit.client.0.smithi088.stderr:+++ '[' -z 'https://debuginfod.centos.org/ ' ']' 2024-06-03T08:37:53.484 INFO:tasks.workunit.client.0.smithi088.stderr:++ for i in /etc/profile.d/*.sh 2024-06-03T08:37:53.484 INFO:tasks.workunit.client.0.smithi088.stderr:++ '[' -r /etc/profile.d/gawk.sh ']' 2024-06-03T08:37:53.484 INFO:tasks.workunit.client.0.smithi088.stderr:++ '[' '' ']' 2024-06-03T08:37:53.484 INFO:tasks.workunit.client.0.smithi088.stderr:++ . /etc/profile.d/gawk.sh 2024-06-03T08:37:53.484 INFO:tasks.workunit.client.0.smithi088.stderr:+ ceph_test_rados_api_asio 2024-06-03T08:37:53.484 INFO:tasks.workunit.client.0.smithi088.stderr:++ for i in /etc/profile.d/*.sh 2024-06-03T08:37:53.484 INFO:tasks.workunit.client.0.smithi088.stderr:++ '[' -r /etc/profile.d/lang.sh ']' 2024-06-03T08:37:53.485 INFO:tasks.workunit.client.0.smithi088.stderr:++ '[' '' ']' 2024-06-03T08:37:53.485 INFO:tasks.workunit.client.0.smithi088.stderr:++ . /etc/profile.d/lang.sh 2024-06-03T08:37:53.485 INFO:tasks.workunit.client.0.smithi088.stderr:+++ unset LANG_backup 2024-06-03T08:37:53.485 INFO:tasks.workunit.client.0.smithi088.stderr:+++ umask 2024-06-03T08:37:53.485 INFO:tasks.workunit.client.0.smithi088.stderr:++ '[' 0022 -eq 0 ']' 2024-06-03T08:37:53.485 INFO:tasks.workunit.client.0.smithi088.stderr:++ SHELL=/bin/bash 2024-06-03T08:37:53.485 INFO:tasks.workunit.client.0.smithi088.stderr:++ for i in /etc/profile.d/*.sh 2024-06-03T08:37:53.485 INFO:tasks.workunit.client.0.smithi088.stderr:++ '[' -r /etc/profile.d/bash_completion.sh ']' 2024-06-03T08:37:53.485 INFO:tasks.workunit.client.0.smithi088.stderr:++ '[' '' ']' 2024-06-03T08:37:53.485 INFO:tasks.workunit.client.0.smithi088.stderr:++ . /etc/profile.d/bash_completion.sh 2024-06-03T08:37:53.485 INFO:tasks.workunit.client.0.smithi088.stderr:+++ '[' 'x5.1.8(1)-release' '!=' x -a x '!=' x -a x = x ']' 2024-06-03T08:37:53.485 INFO:tasks.workunit.client.0.smithi088.stderr:++ for i in /etc/profile.d/*.sh 2024-06-03T08:37:53.485 INFO:tasks.workunit.client.0.smithi088.stderr:++ '[' -r /etc/profile.d/cobbler.sh ']' 2024-06-03T08:37:53.486 INFO:tasks.workunit.client.0.smithi088.stderr:++ '[' '' ']' 2024-06-03T08:37:53.486 INFO:tasks.workunit.client.0.smithi088.stderr:++ . /etc/profile.d/cobbler.sh 2024-06-03T08:37:53.486 INFO:tasks.workunit.client.0.smithi088.stderr:+++ export COBBLER_SERVER=172.21.0.11 2024-06-03T08:37:53.486 INFO:tasks.workunit.client.0.smithi088.stderr:+++ COBBLER_SERVER=172.21.0.11 2024-06-03T08:37:53.486 INFO:tasks.workunit.client.0.smithi088.stderr:++ for i in /etc/profile.d/*.sh 2024-06-03T08:37:53.486 INFO:tasks.workunit.client.0.smithi088.stderr:+++ alias 'zgrep=zgrep --color=auto' 2024-06-03T08:37:53.486 INFO:tasks.workunit.client.0.smithi088.stderr:++ '[' -r /etc/profile.d/colorgrep.sh ']' 2024-06-03T08:37:53.486 INFO:tasks.workunit.client.0.smithi088.stderr:+++ alias 'zfgrep=zfgrep --color=auto' 2024-06-03T08:37:53.486 INFO:tasks.workunit.client.0.smithi088.stderr:+++ alias 'zegrep=zegrep --color=auto' 2024-06-03T08:37:53.486 INFO:tasks.workunit.client.0.smithi088.stderr:++ for i in /etc/profile.d/*.sh 2024-06-03T08:37:53.486 INFO:tasks.workunit.client.0.smithi088.stderr:++ '[' -r /etc/profile.d/debuginfod.sh ']' 2024-06-03T08:37:53.486 INFO:tasks.workunit.client.0.smithi088.stderr:++ '[' '' ']' 2024-06-03T08:37:53.486 INFO:tasks.workunit.client.0.smithi088.stderr:++ . /etc/profile.d/debuginfod.sh 2024-06-03T08:37:53.487 INFO:tasks.workunit.client.0.smithi088.stderr:++ '[' '' ']' 2024-06-03T08:37:53.487 INFO:tasks.workunit.client.0.smithi088.stderr:+++ '[' -z 'https://debuginfod.centos.org/ ' ']' 2024-06-03T08:37:53.487 INFO:tasks.workunit.client.0.smithi088.stderr:++ for i in /etc/profile.d/*.sh 2024-06-03T08:37:53.487 INFO:tasks.workunit.client.0.smithi088.stderr:++ '[' -r /etc/profile.d/gawk.sh ']' 2024-06-03T08:37:53.487 INFO:tasks.workunit.client.0.smithi088.stderr:++ '[' '' ']' 2024-06-03T08:37:53.487 INFO:tasks.workunit.client.0.smithi088.stderr:++ . /etc/profile.d/gawk.sh 2024-06-03T08:37:53.487 INFO:tasks.workunit.client.0.smithi088.stderr:++ . /etc/profile.d/colorgrep.sh 2024-06-03T08:37:53.487 INFO:tasks.workunit.client.0.smithi088.stderr:+++ /usr/libexec/grepconf.sh -c 2024-06-03T08:37:53.487 INFO:tasks.workunit.client.0.smithi088.stderr:++ for i in /etc/profile.d/*.sh 2024-06-03T08:37:53.487 INFO:tasks.workunit.client.0.smithi088.stderr:++ '[' -r /etc/profile.d/lang.sh ']' 2024-06-03T08:37:53.487 INFO:tasks.workunit.client.0.smithi088.stderr:++ '[' '' ']' 2024-06-03T08:37:53.487 INFO:tasks.workunit.client.0.smithi088.stderr:++ . /etc/profile.d/lang.sh 2024-06-03T08:37:53.487 INFO:tasks.workunit.client.0.smithi088.stderr:+++ unset LANG_backup 2024-06-03T08:37:53.487 INFO:tasks.workunit.client.0.smithi088.stderr:++++ /usr/bin/locale 2024-06-03T08:37:53.488 INFO:tasks.workunit.client.0.smithi088.stderr:+++ alias 'grep=grep --color=auto' 2024-06-03T08:37:53.488 INFO:tasks.workunit.client.0.smithi088.stderr:+++ alias 'egrep=egrep --color=auto' 2024-06-03T08:37:53.488 INFO:tasks.workunit.client.0.smithi088.stderr:+++ alias 'fgrep=fgrep --color=auto' 2024-06-03T08:37:53.488 INFO:tasks.workunit.client.0.smithi088.stderr:++ for i in /etc/profile.d/*.sh 2024-06-03T08:37:53.488 INFO:tasks.workunit.client.0.smithi088.stderr:++ '[' -r /etc/profile.d/colorls.sh ']' 2024-06-03T08:37:53.488 INFO:tasks.workunit.client.0.smithi088.stderr:++ '[' '' ']' 2024-06-03T08:37:53.488 INFO:tasks.workunit.client.0.smithi088.stderr:++ . /etc/profile.d/colorls.sh 2024-06-03T08:37:53.488 INFO:tasks.workunit.client.0.smithi088.stderr:+++ '[' '!' -t 0 ']' 2024-06-03T08:37:53.488 INFO:tasks.workunit.client.0.smithi088.stderr:+++ return 2024-06-03T08:37:53.488 INFO:tasks.workunit.client.0.smithi088.stderr:++ for i in /etc/profile.d/*.sh 2024-06-03T08:37:53.488 INFO:tasks.workunit.client.0.smithi088.stderr:++ '[' -r /etc/profile.d/colorsysstat.sh ']' 2024-06-03T08:37:53.488 INFO:tasks.workunit.client.0.smithi088.stderr:++ '[' '' ']' 2024-06-03T08:37:53.488 INFO:tasks.workunit.client.0.smithi088.stderr:++ . /etc/profile.d/colorsysstat.sh 2024-06-03T08:37:53.489 INFO:tasks.workunit.client.0.smithi088.stderr:+++ export S_COLORS=auto 2024-06-03T08:37:53.489 INFO:tasks.workunit.client.0.smithi088.stderr:+++ S_COLORS=auto 2024-06-03T08:37:53.489 INFO:tasks.workunit.client.0.smithi088.stderr:++ for i in /etc/profile.d/*.sh 2024-06-03T08:37:53.489 INFO:tasks.workunit.client.0.smithi088.stderr:++ '[' -r /etc/profile.d/colorxzgrep.sh ']' 2024-06-03T08:37:53.489 INFO:tasks.workunit.client.0.smithi088.stderr:++ '[' '' ']' 2024-06-03T08:37:53.489 INFO:tasks.workunit.client.0.smithi088.stderr:++ . /etc/profile.d/colorxzgrep.sh 2024-06-03T08:37:53.489 INFO:tasks.workunit.client.0.smithi088.stderr:+++ /usr/libexec/grepconf.sh -c 2024-06-03T08:37:53.489 INFO:tasks.workunit.client.0.smithi088.stderr:+++ '[' -n '' ']' 2024-06-03T08:37:53.489 INFO:tasks.workunit.client.0.smithi088.stderr:+++ LANG_backup=en_US.UTF-8 2024-06-03T08:37:53.489 INFO:tasks.workunit.client.0.smithi088.stderr:+++ for config in /etc/locale.conf "${HOME}/.i18n" 2024-06-03T08:37:53.489 INFO:tasks.workunit.client.0.smithi088.stderr:+++ '[' -f /etc/locale.conf ']' 2024-06-03T08:37:53.489 INFO:tasks.workunit.client.0.smithi088.stderr:+++ '[' -x /usr/bin/sed ']' 2024-06-03T08:37:53.489 INFO:tasks.workunit.client.0.smithi088.stderr:+++ alias 'grep=grep --color=auto' 2024-06-03T08:37:53.489 INFO:tasks.workunit.client.0.smithi088.stderr:+++ alias 'egrep=egrep --color=auto' 2024-06-03T08:37:53.490 INFO:tasks.workunit.client.0.smithi088.stderr:+++ alias 'fgrep=fgrep --color=auto' 2024-06-03T08:37:53.490 INFO:tasks.workunit.client.0.smithi088.stderr:++ for i in /etc/profile.d/*.sh 2024-06-03T08:37:53.490 INFO:tasks.workunit.client.0.smithi088.stderr:++ '[' -r /etc/profile.d/colorls.sh ']' 2024-06-03T08:37:53.490 INFO:tasks.workunit.client.0.smithi088.stderr:++ '[' '' ']' 2024-06-03T08:37:53.490 INFO:tasks.workunit.client.0.smithi088.stderr:+++ alias 'zgrep=zgrep --color=auto' 2024-06-03T08:37:53.490 INFO:tasks.workunit.client.0.smithi088.stderr:+++ alias 'zfgrep=zfgrep --color=auto' 2024-06-03T08:37:53.490 INFO:tasks.workunit.client.0.smithi088.stderr:++++ /usr/bin/sed -r -e 's/^[[:blank:]]*([[:upper:]_]+)=([[:print:][:digit:]\._-]+|"[[:print:][:digit:]\._-]+")/export \1=\2/;t;d' /etc/locale.conf 2024-06-03T08:37:53.490 INFO:tasks.workunit.client.0.smithi088.stderr:+++ alias 'zegrep=zegrep --color=auto' 2024-06-03T08:37:53.490 INFO:tasks.workunit.client.0.smithi088.stderr:++ for i in /etc/profile.d/*.sh 2024-06-03T08:37:53.490 INFO:tasks.workunit.client.0.smithi088.stderr:++ '[' -r /etc/profile.d/debuginfod.sh ']' 2024-06-03T08:37:53.490 INFO:tasks.workunit.client.0.smithi088.stderr:++ '[' '' ']' 2024-06-03T08:37:53.490 INFO:tasks.workunit.client.0.smithi088.stderr:++ . /etc/profile.d/debuginfod.sh 2024-06-03T08:37:53.490 INFO:tasks.workunit.client.0.smithi088.stderr:++ . /etc/profile.d/colorls.sh 2024-06-03T08:37:53.491 INFO:tasks.workunit.client.0.smithi088.stderr:+++ '[' -z 'https://debuginfod.centos.org/ ' ']' 2024-06-03T08:37:53.491 INFO:tasks.workunit.client.0.smithi088.stderr:++ for i in /etc/profile.d/*.sh 2024-06-03T08:37:53.491 INFO:tasks.workunit.client.0.smithi088.stderr:++ '[' -r /etc/profile.d/gawk.sh ']' 2024-06-03T08:37:53.491 INFO:tasks.workunit.client.0.smithi088.stderr:++ '[' '' ']' 2024-06-03T08:37:53.491 INFO:tasks.workunit.client.0.smithi088.stderr:++ . /etc/profile.d/gawk.sh 2024-06-03T08:37:53.491 INFO:tasks.workunit.client.0.smithi088.stderr:+++ '[' '!' -t 0 ']' 2024-06-03T08:37:53.491 INFO:tasks.workunit.client.0.smithi088.stderr:+++ return 2024-06-03T08:37:53.491 INFO:tasks.workunit.client.0.smithi088.stderr:++ for i in /etc/profile.d/*.sh 2024-06-03T08:37:53.491 INFO:tasks.workunit.client.0.smithi088.stderr:++ '[' -r /etc/profile.d/lang.sh ']' 2024-06-03T08:37:53.491 INFO:tasks.workunit.client.0.smithi088.stderr:++ '[' '' ']' 2024-06-03T08:37:53.491 INFO:tasks.workunit.client.0.smithi088.stderr:++ . /etc/profile.d/lang.sh 2024-06-03T08:37:53.491 INFO:tasks.workunit.client.0.smithi088.stderr:+++ unset LANG_backup 2024-06-03T08:37:53.491 INFO:tasks.workunit.client.0.smithi088.stderr:++ for i in /etc/profile.d/*.sh 2024-06-03T08:37:53.492 INFO:tasks.workunit.client.0.smithi088.stderr:++ '[' -r /etc/profile.d/colorsysstat.sh ']' 2024-06-03T08:37:53.492 INFO:tasks.workunit.client.0.smithi088.stderr:++ '[' '' ']' 2024-06-03T08:37:53.492 INFO:tasks.workunit.client.0.smithi088.stderr:++ . /etc/profile.d/colorsysstat.sh 2024-06-03T08:37:53.492 INFO:tasks.workunit.client.0.smithi088.stderr:+++ export S_COLORS=auto 2024-06-03T08:37:53.492 INFO:tasks.workunit.client.0.smithi088.stderr:+++ S_COLORS=auto 2024-06-03T08:37:53.492 INFO:tasks.workunit.client.0.smithi088.stderr:++ for i in /etc/profile.d/*.sh 2024-06-03T08:37:53.492 INFO:tasks.workunit.client.0.smithi088.stderr:++ '[' -r /etc/profile.d/colorxzgrep.sh ']' 2024-06-03T08:37:53.492 INFO:tasks.workunit.client.0.smithi088.stderr:++ '[' '' ']' 2024-06-03T08:37:53.492 INFO:tasks.workunit.client.0.smithi088.stderr:++ . /etc/profile.d/colorxzgrep.sh 2024-06-03T08:37:53.492 INFO:tasks.workunit.client.0.smithi088.stderr:+++ /usr/libexec/grepconf.sh -c 2024-06-03T08:37:53.492 INFO:tasks.workunit.client.0.smithi088.stderr:+++ alias 'xzgrep=xzgrep --color=auto' 2024-06-03T08:37:53.492 INFO:tasks.workunit.client.0.smithi088.stderr:+++ alias 'xzegrep=xzegrep --color=auto' 2024-06-03T08:37:53.492 INFO:tasks.workunit.client.0.smithi088.stderr:+++ alias 'xzfgrep=xzfgrep --color=auto' 2024-06-03T08:37:53.493 INFO:tasks.workunit.client.0.smithi088.stderr:++ for i in /etc/profile.d/*.sh 2024-06-03T08:37:53.493 INFO:tasks.workunit.client.0.smithi088.stderr:++ '[' -r /etc/profile.d/colorzgrep.sh ']' 2024-06-03T08:37:53.493 INFO:tasks.workunit.client.0.smithi088.stderr:++ '[' '' ']' 2024-06-03T08:37:53.493 INFO:tasks.workunit.client.0.smithi088.stderr:++ . /etc/profile.d/colorzgrep.sh 2024-06-03T08:37:53.493 INFO:tasks.workunit.client.0.smithi088.stderr:+++ '[' -f /usr/libexec/grepconf.sh ']' 2024-06-03T08:37:53.493 INFO:tasks.workunit.client.0.smithi088.stderr:+++ eval export 'LANG="en_US.UTF-8"' 2024-06-03T08:37:53.493 INFO:tasks.workunit.client.0.smithi088.stderr:++++ export LANG=en_US.UTF-8 2024-06-03T08:37:53.493 INFO:tasks.workunit.client.0.smithi088.stderr:++++ LANG=en_US.UTF-8 2024-06-03T08:37:53.493 INFO:tasks.workunit.client.0.smithi088.stderr:+++ for config in /etc/locale.conf "${HOME}/.i18n" 2024-06-03T08:37:53.493 INFO:tasks.workunit.client.0.smithi088.stderr:+++ '[' -f /home/ubuntu/.i18n ']' 2024-06-03T08:37:53.493 INFO:tasks.workunit.client.0.smithi088.stderr:+++ '[' -n en_US.UTF-8 ']' 2024-06-03T08:37:53.493 INFO:tasks.workunit.client.0.smithi088.stderr:+++ /usr/libexec/grepconf.sh -c 2024-06-03T08:37:53.493 INFO:tasks.workunit.client.0.smithi088.stderr:+++ LANG=en_US.UTF-8 2024-06-03T08:37:53.493 INFO:tasks.workunit.client.0.smithi088.stderr:+++ unset LANG_backup config 2024-06-03T08:37:53.494 INFO:tasks.workunit.client.0.smithi088.stderr:+++ '[' -n '' ']' 2024-06-03T08:37:53.494 INFO:tasks.workunit.client.0.smithi088.stderr:+++ '[' -n en_US.UTF-8 ']' 2024-06-03T08:37:53.494 INFO:tasks.workunit.client.0.smithi088.stderr:+++ '[' dumb = linux ']' 2024-06-03T08:37:53.494 INFO:tasks.workunit.client.0.smithi088.stderr:++ for i in /etc/profile.d/*.sh 2024-06-03T08:37:53.494 INFO:tasks.workunit.client.0.smithi088.stderr:++ '[' -r /etc/profile.d/less.sh ']' 2024-06-03T08:37:53.494 INFO:tasks.workunit.client.0.smithi088.stderr:++ '[' '' ']' 2024-06-03T08:37:53.494 INFO:tasks.workunit.client.0.smithi088.stderr:++ . /etc/profile.d/less.sh 2024-06-03T08:37:53.494 INFO:tasks.workunit.client.0.smithi088.stderr:+++ '[' -z '||/usr/bin/lesspipe.sh %s' ']' 2024-06-03T08:37:53.494 INFO:tasks.workunit.client.0.smithi088.stderr:++ for i in /etc/profile.d/*.sh 2024-06-03T08:37:53.494 INFO:tasks.workunit.client.0.smithi088.stderr:++ '[' -r /etc/profile.d/which2.sh ']' 2024-06-03T08:37:53.494 INFO:tasks.workunit.client.0.smithi088.stderr:++ '[' '' ']' 2024-06-03T08:37:53.494 INFO:tasks.workunit.client.0.smithi088.stderr:++ . /etc/profile.d/which2.sh 2024-06-03T08:37:53.494 INFO:tasks.workunit.client.0.smithi088.stderr:+++ case "$(basename $(readlink /proc/$$/exe))" in 2024-06-03T08:37:53.495 INFO:tasks.workunit.client.0.smithi088.stderr:++++ /usr/bin/locale 2024-06-03T08:37:53.495 INFO:tasks.workunit.client.0.smithi088.stderr:+++++ readlink /proc/40304/exe 2024-06-03T08:37:53.495 INFO:tasks.workunit.client.0.smithi088.stderr:+++ alias 'xzgrep=xzgrep --color=auto' 2024-06-03T08:37:53.495 INFO:tasks.workunit.client.0.smithi088.stderr:+++ '[' -n '' ']' 2024-06-03T08:37:53.495 INFO:tasks.workunit.client.0.smithi088.stderr:+++ LANG_backup=en_US.UTF-8 2024-06-03T08:37:53.495 INFO:tasks.workunit.client.0.smithi088.stderr:+++ alias 'xzegrep=xzegrep --color=auto' 2024-06-03T08:37:53.495 INFO:tasks.workunit.client.0.smithi088.stderr:+++ for config in /etc/locale.conf "${HOME}/.i18n" 2024-06-03T08:37:53.495 INFO:tasks.workunit.client.0.smithi088.stderr:+++ alias 'xzfgrep=xzfgrep --color=auto' 2024-06-03T08:37:53.495 INFO:tasks.workunit.client.0.smithi088.stderr:++ for i in /etc/profile.d/*.sh 2024-06-03T08:37:53.495 INFO:tasks.workunit.client.0.smithi088.stderr:++++ basename /usr/bin/bash 2024-06-03T08:37:53.495 INFO:tasks.workunit.client.0.smithi088.stderr:+++ '[' -f /etc/locale.conf ']' 2024-06-03T08:37:53.495 INFO:tasks.workunit.client.0.smithi088.stderr:+++ '[' -x /usr/bin/sed ']' 2024-06-03T08:37:53.495 INFO:tasks.workunit.client.0.smithi088.stderr:++++ /usr/bin/locale 2024-06-03T08:37:53.495 INFO:tasks.workunit.client.0.smithi088.stderr:++ '[' -r /etc/profile.d/colorzgrep.sh ']' 2024-06-03T08:37:53.496 INFO:tasks.workunit.client.0.smithi088.stderr:++ '[' '' ']' 2024-06-03T08:37:53.496 INFO:tasks.workunit.client.0.smithi088.stderr:++ . /etc/profile.d/colorzgrep.sh 2024-06-03T08:37:53.496 INFO:tasks.workunit.client.0.smithi088.stderr:+++ '[' -f /usr/libexec/grepconf.sh ']' 2024-06-03T08:37:53.496 INFO:tasks.workunit.client.0.smithi088.stderr:+++ /usr/libexec/grepconf.sh -c 2024-06-03T08:37:53.496 INFO:tasks.workunit.client.0.smithi088.stderr:++++ /usr/bin/sed -r -e 's/^[[:blank:]]*([[:upper:]_]+)=([[:print:][:digit:]\._-]+|"[[:print:][:digit:]\._-]+")/export \1=\2/;t;d' /etc/locale.conf 2024-06-03T08:37:53.496 INFO:tasks.workunit.client.0.smithi088.stderr:+++ alias 'zgrep=zgrep --color=auto' 2024-06-03T08:37:53.496 INFO:tasks.workunit.client.0.smithi088.stderr:+++ alias 'zfgrep=zfgrep --color=auto' 2024-06-03T08:37:53.496 INFO:tasks.workunit.client.0.smithi088.stderr:+++ alias 'zegrep=zegrep --color=auto' 2024-06-03T08:37:53.496 INFO:tasks.workunit.client.0.smithi088.stderr:++ for i in /etc/profile.d/*.sh 2024-06-03T08:37:53.496 INFO:tasks.workunit.client.0.smithi088.stderr:++ '[' -r /etc/profile.d/debuginfod.sh ']' 2024-06-03T08:37:53.496 INFO:tasks.workunit.client.0.smithi088.stderr:++ '[' '' ']' 2024-06-03T08:37:53.496 INFO:tasks.workunit.client.0.smithi088.stderr:++ . /etc/profile.d/debuginfod.sh 2024-06-03T08:37:53.496 INFO:tasks.workunit.client.0.smithi088.stderr:+++ which_declare='declare -f' 2024-06-03T08:37:53.497 INFO:tasks.workunit.client.0.smithi088.stderr:+++ which_opt=-f 2024-06-03T08:37:53.497 INFO:tasks.workunit.client.0.smithi088.stderr:+++ export which_declare 2024-06-03T08:37:53.497 INFO:tasks.workunit.client.0.smithi088.stderr:+++ '[' -z 'https://debuginfod.centos.org/ ' ']' 2024-06-03T08:37:53.497 INFO:tasks.workunit.client.0.smithi088.stderr:+++ export -f which 2024-06-03T08:37:53.497 INFO:tasks.workunit.client.0.smithi088.stderr:++ for i in /etc/profile.d/*.sh 2024-06-03T08:37:53.497 INFO:tasks.workunit.client.0.smithi088.stderr:++ '[' -r /etc/profile.d/gawk.sh ']' 2024-06-03T08:37:53.497 INFO:tasks.workunit.client.0.smithi088.stderr:++ '[' '' ']' 2024-06-03T08:37:53.497 INFO:tasks.workunit.client.0.smithi088.stderr:++ . /etc/profile.d/gawk.sh 2024-06-03T08:37:53.497 INFO:tasks.workunit.client.0.smithi088.stderr:++ unset i 2024-06-03T08:37:53.497 INFO:tasks.workunit.client.0.smithi088.stderr:++ unset -f pathmunge 2024-06-03T08:37:53.497 INFO:tasks.workunit.client.0.smithi088.stderr:+ [[ /home/ubuntu/.local/bin:/home/ubuntu/bin:/usr/local/bin:/usr/bin:/usr/local/sbin:/usr/sbin:/usr/sbin =~ /home/ubuntu/\.local/bin:/home/ubuntu/bin: ]] 2024-06-03T08:37:53.497 INFO:tasks.workunit.client.0.smithi088.stderr:++ for i in /etc/profile.d/*.sh 2024-06-03T08:37:53.497 INFO:tasks.workunit.client.0.smithi088.stderr:+ export PATH 2024-06-03T08:37:53.498 INFO:tasks.workunit.client.0.smithi088.stderr:++ '[' -r /etc/profile.d/lang.sh ']' 2024-06-03T08:37:53.498 INFO:tasks.workunit.client.0.smithi088.stderr:+ '[' -d /home/ubuntu/.bashrc.d ']' 2024-06-03T08:37:53.498 INFO:tasks.workunit.client.0.smithi088.stderr:+ unset rc 2024-06-03T08:37:53.498 INFO:tasks.workunit.client.0.smithi088.stderr:++ '[' '' ']' 2024-06-03T08:37:53.498 INFO:tasks.workunit.client.0.smithi088.stderr:++ . /etc/profile.d/lang.sh 2024-06-03T08:37:53.498 INFO:tasks.workunit.client.0.smithi088.stderr:+++ unset LANG_backup 2024-06-03T08:37:53.498 INFO:tasks.workunit.client.0.smithi088.stderr:+ tee ceph_test_rados_api_lock_pp.log 2024-06-03T08:37:53.498 INFO:tasks.workunit.client.0.smithi088.stderr:+++ '[' -n '' ']' 2024-06-03T08:37:53.498 INFO:tasks.workunit.client.0.smithi088.stderr:++ printf %25s api_snapshots 2024-06-03T08:37:53.498 INFO:tasks.workunit.client.0.smithi088.stderr:+++ LANG_backup=en_US.UTF-8 2024-06-03T08:37:53.498 INFO:tasks.workunit.client.0.smithi088.stderr:+++ for config in /etc/locale.conf "${HOME}/.i18n" 2024-06-03T08:37:53.498 INFO:tasks.workunit.client.0.smithi088.stderr:+++ '[' -f /etc/locale.conf ']' 2024-06-03T08:37:53.498 INFO:tasks.workunit.client.0.smithi088.stderr:+++ '[' -x /usr/bin/sed ']' 2024-06-03T08:37:53.499 INFO:tasks.workunit.client.0.smithi088.stderr:+ r=' api_snapshots' 2024-06-03T08:37:53.499 INFO:tasks.workunit.client.0.smithi088.stderr:+++ eval export 'LANG="en_US.UTF-8"' 2024-06-03T08:37:53.499 INFO:tasks.workunit.client.0.smithi088.stderr:++++ export LANG=en_US.UTF-8 2024-06-03T08:37:53.499 INFO:tasks.workunit.client.0.smithi088.stderr:++++ LANG=en_US.UTF-8 2024-06-03T08:37:53.499 INFO:tasks.workunit.client.0.smithi088.stderr:+++ for config in /etc/locale.conf "${HOME}/.i18n" 2024-06-03T08:37:53.499 INFO:tasks.workunit.client.0.smithi088.stderr:+ sed 's/^/ api_lock_pp: /' 2024-06-03T08:37:53.499 INFO:tasks.workunit.client.0.smithi088.stderr:+++ '[' -f /home/ubuntu/.i18n ']' 2024-06-03T08:37:53.499 INFO:tasks.workunit.client.0.smithi088.stderr:+++ '[' -n en_US.UTF-8 ']' 2024-06-03T08:37:53.499 INFO:tasks.workunit.client.0.smithi088.stderr:+++ LANG=en_US.UTF-8 2024-06-03T08:37:53.499 INFO:tasks.workunit.client.0.smithi088.stderr:+++ unset LANG_backup config 2024-06-03T08:37:53.499 INFO:tasks.workunit.client.0.smithi088.stderr:+++ '[' -n '' ']' 2024-06-03T08:37:53.499 INFO:tasks.workunit.client.0.smithi088.stderr:+++ '[' -n en_US.UTF-8 ']' 2024-06-03T08:37:53.499 INFO:tasks.workunit.client.0.smithi088.stderr:+++ '[' dumb = linux ']' 2024-06-03T08:37:53.500 INFO:tasks.workunit.client.0.smithi088.stderr:++ echo api_snapshots 2024-06-03T08:37:53.500 INFO:tasks.workunit.client.0.smithi088.stderr:++ for i in /etc/profile.d/*.sh 2024-06-03T08:37:53.500 INFO:tasks.workunit.client.0.smithi088.stderr:++ '[' -r /etc/profile.d/less.sh ']' 2024-06-03T08:37:53.500 INFO:tasks.workunit.client.0.smithi088.stderr:++ '[' '' ']' 2024-06-03T08:37:53.500 INFO:tasks.workunit.client.0.smithi088.stderr:++++ /usr/bin/sed -r -e 's/^[[:blank:]]*([[:upper:]_]+)=([[:print:][:digit:]\._-]+|"[[:print:][:digit:]\._-]+")/export \1=\2/;t;d' /etc/locale.conf 2024-06-03T08:37:53.500 INFO:tasks.workunit.client.0.smithi088.stderr:++ . /etc/profile.d/less.sh 2024-06-03T08:37:53.500 INFO:tasks.workunit.client.0.smithi088.stderr:++ awk '{print $1}' 2024-06-03T08:37:53.500 INFO:tasks.workunit.client.0.smithi088.stderr:+++ '[' -z '||/usr/bin/lesspipe.sh %s' ']' 2024-06-03T08:37:53.500 INFO:tasks.workunit.client.0.smithi088.stderr:++ for i in /etc/profile.d/*.sh 2024-06-03T08:37:53.500 INFO:tasks.workunit.client.0.smithi088.stderr:+ ceph_test_rados_api_lock_pp 2024-06-03T08:37:53.500 INFO:tasks.workunit.client.0.smithi088.stderr:++ '[' -r /etc/profile.d/which2.sh ']' 2024-06-03T08:37:53.500 INFO:tasks.workunit.client.0.smithi088.stderr:++ '[' '' ']' 2024-06-03T08:37:53.500 INFO:tasks.workunit.client.0.smithi088.stderr:++ . /etc/profile.d/which2.sh 2024-06-03T08:37:53.500 INFO:tasks.workunit.client.0.smithi088.stderr:+++ case "$(basename $(readlink /proc/$$/exe))" in 2024-06-03T08:37:53.501 INFO:tasks.workunit.client.0.smithi088.stderr:+++++ readlink /proc/40347/exe 2024-06-03T08:37:53.501 INFO:tasks.workunit.client.0.smithi088.stderr:++++ basename /usr/bin/bash 2024-06-03T08:37:53.501 INFO:tasks.workunit.client.0.smithi088.stderr:+++ eval export 'LANG="en_US.UTF-8"' 2024-06-03T08:37:53.501 INFO:tasks.workunit.client.0.smithi088.stderr:++++ export LANG=en_US.UTF-8 2024-06-03T08:37:53.501 INFO:tasks.workunit.client.0.smithi088.stderr:++++ LANG=en_US.UTF-8 2024-06-03T08:37:53.501 INFO:tasks.workunit.client.0.smithi088.stderr:+++ for config in /etc/locale.conf "${HOME}/.i18n" 2024-06-03T08:37:53.501 INFO:tasks.workunit.client.0.smithi088.stderr:+++ '[' -f /home/ubuntu/.i18n ']' 2024-06-03T08:37:53.501 INFO:tasks.workunit.client.0.smithi088.stderr:+++ '[' -n en_US.UTF-8 ']' 2024-06-03T08:37:53.501 INFO:tasks.workunit.client.0.smithi088.stderr:+++ LANG=en_US.UTF-8 2024-06-03T08:37:53.501 INFO:tasks.workunit.client.0.smithi088.stderr:+ ff=api_snapshots 2024-06-03T08:37:53.501 INFO:tasks.workunit.client.0.smithi088.stderr:+++ unset LANG_backup config 2024-06-03T08:37:53.501 INFO:tasks.workunit.client.0.smithi088.stderr:+++ '[' -n '' ']' 2024-06-03T08:37:53.501 INFO:tasks.workunit.client.0.smithi088.stderr:+ pid=40415 2024-06-03T08:37:53.502 INFO:tasks.workunit.client.0.smithi088.stdout:test api_snapshots on pid 40415 2024-06-03T08:37:53.512 INFO:tasks.workunit.client.0.smithi088.stderr:+ echo 'test api_snapshots on pid 40415' 2024-06-03T08:37:53.512 INFO:tasks.workunit.client.0.smithi088.stderr:+ pids[$f]=40415 2024-06-03T08:37:53.512 INFO:tasks.workunit.client.0.smithi088.stderr:+ for f in api_aio api_aio_pp api_io api_io_pp api_asio api_list api_lock api_lock_pp api_misc api_misc_pp api_tier_pp api_pool api_snapshots api_snapshots_pp api_stat api_stat_pp api_watch_notify api_watch_notify_pp api_cmd api_cmd_pp api_service api_service_pp api_c_write_operations api_c_read_operations api_cls_remote_reads list_parallel open_pools_parallel delete_pools_parallel 2024-06-03T08:37:53.512 INFO:tasks.workunit.client.0.smithi088.stderr:+ '[' 1 -eq 1 ']' 2024-06-03T08:37:53.512 INFO:tasks.workunit.client.0.smithi088.stderr:+++ which_declare='declare -f' 2024-06-03T08:37:53.512 INFO:tasks.workunit.client.0.smithi088.stderr:+++ which_opt=-f 2024-06-03T08:37:53.513 INFO:tasks.workunit.client.0.smithi088.stderr:+++ export which_declare 2024-06-03T08:37:53.513 INFO:tasks.workunit.client.0.smithi088.stderr:+++ export -f which 2024-06-03T08:37:53.513 INFO:tasks.workunit.client.0.smithi088.stderr:++ unset i 2024-06-03T08:37:53.513 INFO:tasks.workunit.client.0.smithi088.stderr:++ unset -f pathmunge 2024-06-03T08:37:53.513 INFO:tasks.workunit.client.0.smithi088.stderr:++ printf %25s api_snapshots_pp 2024-06-03T08:37:53.513 INFO:tasks.workunit.client.0.smithi088.stderr:+++ '[' -n en_US.UTF-8 ']' 2024-06-03T08:37:53.513 INFO:tasks.workunit.client.0.smithi088.stderr:+ [[ /home/ubuntu/.local/bin:/home/ubuntu/bin:/usr/local/bin:/usr/bin:/usr/local/sbin:/usr/sbin:/usr/sbin =~ /home/ubuntu/\.local/bin:/home/ubuntu/bin: ]] 2024-06-03T08:37:53.513 INFO:tasks.workunit.client.0.smithi088.stderr:+++ '[' dumb = linux ']' 2024-06-03T08:37:53.513 INFO:tasks.workunit.client.0.smithi088.stderr:++ for i in /etc/profile.d/*.sh 2024-06-03T08:37:53.513 INFO:tasks.workunit.client.0.smithi088.stderr:++ '[' -r /etc/profile.d/less.sh ']' 2024-06-03T08:37:53.513 INFO:tasks.workunit.client.0.smithi088.stderr:++ '[' '' ']' 2024-06-03T08:37:53.513 INFO:tasks.workunit.client.0.smithi088.stderr:++ . /etc/profile.d/less.sh 2024-06-03T08:37:53.513 INFO:tasks.workunit.client.0.smithi088.stderr:+++ '[' -z '||/usr/bin/lesspipe.sh %s' ']' 2024-06-03T08:37:53.514 INFO:tasks.workunit.client.0.smithi088.stderr:+ export PATH 2024-06-03T08:37:53.514 INFO:tasks.workunit.client.0.smithi088.stderr:+ '[' -d /home/ubuntu/.bashrc.d ']' 2024-06-03T08:37:53.514 INFO:tasks.workunit.client.0.smithi088.stderr:+ unset rc 2024-06-03T08:37:53.514 INFO:tasks.workunit.client.0.smithi088.stderr:++ for i in /etc/profile.d/*.sh 2024-06-03T08:37:53.514 INFO:tasks.workunit.client.0.smithi088.stderr:+ r=' api_snapshots_pp' 2024-06-03T08:37:53.514 INFO:tasks.workunit.client.0.smithi088.stderr:++ '[' -r /etc/profile.d/which2.sh ']' 2024-06-03T08:37:53.514 INFO:tasks.workunit.client.0.smithi088.stderr:++ '[' '' ']' 2024-06-03T08:37:53.514 INFO:tasks.workunit.client.0.smithi088.stderr:++ . /etc/profile.d/which2.sh 2024-06-03T08:37:53.514 INFO:tasks.workunit.client.0.smithi088.stderr:+++ case "$(basename $(readlink /proc/$$/exe))" in 2024-06-03T08:37:53.514 INFO:tasks.workunit.client.0.smithi088.stderr:+ tee ceph_test_rados_api_misc_pp.log 2024-06-03T08:37:53.514 INFO:tasks.workunit.client.0.smithi088.stderr:+ bash -o pipefail -exc 'ceph_test_rados_api_snapshots 2>&1 | tee ceph_test_rados_api_snapshots.log | sed "s/^/ api_snapshots: /"' 2024-06-03T08:37:53.514 INFO:tasks.workunit.client.0.smithi088.stderr:+ sed 's/^/ api_misc_pp: /' 2024-06-03T08:37:53.514 INFO:tasks.workunit.client.0.smithi088.stderr:+++++ readlink /proc/40284/exe 2024-06-03T08:37:53.515 INFO:tasks.workunit.client.0.smithi088.stderr:+ '[' -f /etc/bashrc ']' 2024-06-03T08:37:53.515 INFO:tasks.workunit.client.0.smithi088.stderr:+ . /etc/bashrc 2024-06-03T08:37:53.515 INFO:tasks.workunit.client.0.smithi088.stderr:++ '[' -z '' ']' 2024-06-03T08:37:53.515 INFO:tasks.workunit.client.0.smithi088.stderr:++ BASHRCSOURCED=Y 2024-06-03T08:37:53.515 INFO:tasks.workunit.client.0.smithi088.stderr:++ '[' '' ']' 2024-06-03T08:37:53.515 INFO:tasks.workunit.client.0.smithi088.stderr:++ shopt -q login_shell 2024-06-03T08:37:53.515 INFO:tasks.workunit.client.0.smithi088.stderr:++++ basename /usr/bin/bash 2024-06-03T08:37:53.515 INFO:tasks.workunit.client.0.smithi088.stderr:++ echo api_snapshots_pp 2024-06-03T08:37:53.515 INFO:tasks.workunit.client.0.smithi088.stderr:+++ umask 2024-06-03T08:37:53.515 INFO:tasks.workunit.client.0.smithi088.stderr:++ '[' 0022 -eq 0 ']' 2024-06-03T08:37:53.515 INFO:tasks.workunit.client.0.smithi088.stderr:++ SHELL=/bin/bash 2024-06-03T08:37:53.515 INFO:tasks.workunit.client.0.smithi088.stderr:++ for i in /etc/profile.d/*.sh 2024-06-03T08:37:53.515 INFO:tasks.workunit.client.0.smithi088.stderr:++ '[' -r /etc/profile.d/bash_completion.sh ']' 2024-06-03T08:37:53.516 INFO:tasks.workunit.client.0.smithi088.stderr:++ '[' '' ']' 2024-06-03T08:37:53.516 INFO:tasks.workunit.client.0.smithi088.stderr:++ . /etc/profile.d/bash_completion.sh 2024-06-03T08:37:53.516 INFO:tasks.workunit.client.0.smithi088.stderr:+++ which_declare='declare -f' 2024-06-03T08:37:53.516 INFO:tasks.workunit.client.0.smithi088.stderr:+++ '[' 'x5.1.8(1)-release' '!=' x -a x '!=' x -a x = x ']' 2024-06-03T08:37:53.516 INFO:tasks.workunit.client.0.smithi088.stderr:++ for i in /etc/profile.d/*.sh 2024-06-03T08:37:53.516 INFO:tasks.workunit.client.0.smithi088.stderr:++ '[' -r /etc/profile.d/cobbler.sh ']' 2024-06-03T08:37:53.516 INFO:tasks.workunit.client.0.smithi088.stderr:++ '[' '' ']' 2024-06-03T08:37:53.516 INFO:tasks.workunit.client.0.smithi088.stderr:++ . /etc/profile.d/cobbler.sh 2024-06-03T08:37:53.516 INFO:tasks.workunit.client.0.smithi088.stderr:+++ export COBBLER_SERVER=172.21.0.11 2024-06-03T08:37:53.516 INFO:tasks.workunit.client.0.smithi088.stderr:+++ which_opt=-f 2024-06-03T08:37:53.516 INFO:tasks.workunit.client.0.smithi088.stderr:+++ COBBLER_SERVER=172.21.0.11 2024-06-03T08:37:53.516 INFO:tasks.workunit.client.0.smithi088.stderr:+++ export which_declare 2024-06-03T08:37:53.516 INFO:tasks.workunit.client.0.smithi088.stderr:+++ export -f which 2024-06-03T08:37:53.517 INFO:tasks.workunit.client.0.smithi088.stderr:++ unset i 2024-06-03T08:37:53.517 INFO:tasks.workunit.client.0.smithi088.stderr:++ for i in /etc/profile.d/*.sh 2024-06-03T08:37:53.517 INFO:tasks.workunit.client.0.smithi088.stderr:++ '[' -r /etc/profile.d/colorgrep.sh ']' 2024-06-03T08:37:53.517 INFO:tasks.workunit.client.0.smithi088.stderr:++ '[' '' ']' 2024-06-03T08:37:53.517 INFO:tasks.workunit.client.0.smithi088.stderr:++ . /etc/profile.d/colorgrep.sh 2024-06-03T08:37:53.517 INFO:tasks.workunit.client.0.smithi088.stderr:+++ /usr/libexec/grepconf.sh -c 2024-06-03T08:37:53.517 INFO:tasks.workunit.client.0.smithi088.stderr:++ unset -f pathmunge 2024-06-03T08:37:53.517 INFO:tasks.workunit.client.0.smithi088.stderr:+ [[ /home/ubuntu/.local/bin:/home/ubuntu/bin:/usr/local/bin:/usr/bin:/usr/local/sbin:/usr/sbin:/usr/sbin =~ /home/ubuntu/\.local/bin:/home/ubuntu/bin: ]] 2024-06-03T08:37:53.517 INFO:tasks.workunit.client.0.smithi088.stderr:+ export PATH 2024-06-03T08:37:53.517 INFO:tasks.workunit.client.0.smithi088.stderr:+ '[' -d /home/ubuntu/.bashrc.d ']' 2024-06-03T08:37:53.517 INFO:tasks.workunit.client.0.smithi088.stderr:+ ceph_test_rados_api_misc_pp 2024-06-03T08:37:53.517 INFO:tasks.workunit.client.0.smithi088.stderr:+ unset rc 2024-06-03T08:37:53.517 INFO:tasks.workunit.client.0.smithi088.stderr:+++ alias 'xzgrep=xzgrep --color=auto' 2024-06-03T08:37:53.517 INFO:tasks.workunit.client.0.smithi088.stderr:+++ alias 'xzegrep=xzegrep --color=auto' 2024-06-03T08:37:53.518 INFO:tasks.workunit.client.0.smithi088.stderr:+ ceph_test_rados_api_lock 2024-06-03T08:37:53.518 INFO:tasks.workunit.client.0.smithi088.stderr:+++ alias 'xzfgrep=xzfgrep --color=auto' 2024-06-03T08:37:53.518 INFO:tasks.workunit.client.0.smithi088.stderr:++ for i in /etc/profile.d/*.sh 2024-06-03T08:37:53.518 INFO:tasks.workunit.client.0.smithi088.stderr:++ '[' -r /etc/profile.d/colorzgrep.sh ']' 2024-06-03T08:37:53.518 INFO:tasks.workunit.client.0.smithi088.stderr:++ '[' '' ']' 2024-06-03T08:37:53.518 INFO:tasks.workunit.client.0.smithi088.stderr:++ . /etc/profile.d/colorzgrep.sh 2024-06-03T08:37:53.518 INFO:tasks.workunit.client.0.smithi088.stderr:+++ '[' -f /usr/libexec/grepconf.sh ']' 2024-06-03T08:37:53.518 INFO:tasks.workunit.client.0.smithi088.stderr:+++ /usr/libexec/grepconf.sh -c 2024-06-03T08:37:53.518 INFO:tasks.workunit.client.0.smithi088.stderr:+++ alias 'zgrep=zgrep --color=auto' 2024-06-03T08:37:53.518 INFO:tasks.workunit.client.0.smithi088.stderr:+++ alias 'grep=grep --color=auto' 2024-06-03T08:37:53.518 INFO:tasks.workunit.client.0.smithi088.stderr:+ tee ceph_test_rados_api_lock.log 2024-06-03T08:37:53.518 INFO:tasks.workunit.client.0.smithi088.stderr:+++ alias 'egrep=egrep --color=auto' 2024-06-03T08:37:53.519 INFO:tasks.workunit.client.0.smithi088.stderr:+++ alias 'fgrep=fgrep --color=auto' 2024-06-03T08:37:53.519 INFO:tasks.workunit.client.0.smithi088.stderr:++ for i in /etc/profile.d/*.sh 2024-06-03T08:37:53.519 INFO:tasks.workunit.client.0.smithi088.stderr:++ '[' -r /etc/profile.d/colorls.sh ']' 2024-06-03T08:37:53.519 INFO:tasks.workunit.client.0.smithi088.stderr:++ '[' '' ']' 2024-06-03T08:37:53.519 INFO:tasks.workunit.client.0.smithi088.stderr:++ . /etc/profile.d/colorls.sh 2024-06-03T08:37:53.519 INFO:tasks.workunit.client.0.smithi088.stderr:+++ '[' '!' -t 0 ']' 2024-06-03T08:37:53.519 INFO:tasks.workunit.client.0.smithi088.stderr:+++ return 2024-06-03T08:37:53.519 INFO:tasks.workunit.client.0.smithi088.stderr:++ for i in /etc/profile.d/*.sh 2024-06-03T08:37:53.519 INFO:tasks.workunit.client.0.smithi088.stderr:++ '[' -r /etc/profile.d/colorsysstat.sh ']' 2024-06-03T08:37:53.519 INFO:tasks.workunit.client.0.smithi088.stderr:++ '[' '' ']' 2024-06-03T08:37:53.519 INFO:tasks.workunit.client.0.smithi088.stderr:++ . /etc/profile.d/colorsysstat.sh 2024-06-03T08:37:53.519 INFO:tasks.workunit.client.0.smithi088.stderr:+++ alias 'zfgrep=zfgrep --color=auto' 2024-06-03T08:37:53.519 INFO:tasks.workunit.client.0.smithi088.stderr:+++ export S_COLORS=auto 2024-06-03T08:37:53.519 INFO:tasks.workunit.client.0.smithi088.stderr:+++ S_COLORS=auto 2024-06-03T08:37:53.520 INFO:tasks.workunit.client.0.smithi088.stderr:++ for i in /etc/profile.d/*.sh 2024-06-03T08:37:53.520 INFO:tasks.workunit.client.0.smithi088.stderr:++ '[' -r /etc/profile.d/colorxzgrep.sh ']' 2024-06-03T08:37:53.520 INFO:tasks.workunit.client.0.smithi088.stderr:++ '[' '' ']' 2024-06-03T08:37:53.520 INFO:tasks.workunit.client.0.smithi088.stderr:++ . /etc/profile.d/colorxzgrep.sh 2024-06-03T08:37:53.520 INFO:tasks.workunit.client.0.smithi088.stderr:+++ /usr/libexec/grepconf.sh -c 2024-06-03T08:37:53.520 INFO:tasks.workunit.client.0.smithi088.stderr:+++ alias 'zegrep=zegrep --color=auto' 2024-06-03T08:37:53.520 INFO:tasks.workunit.client.0.smithi088.stderr:++ for i in /etc/profile.d/*.sh 2024-06-03T08:37:53.520 INFO:tasks.workunit.client.0.smithi088.stderr:++ '[' -r /etc/profile.d/debuginfod.sh ']' 2024-06-03T08:37:53.520 INFO:tasks.workunit.client.0.smithi088.stderr:++ '[' '' ']' 2024-06-03T08:37:53.520 INFO:tasks.workunit.client.0.smithi088.stderr:++ . /etc/profile.d/debuginfod.sh 2024-06-03T08:37:53.520 INFO:tasks.workunit.client.0.smithi088.stderr:+++ '[' -z 'https://debuginfod.centos.org/ ' ']' 2024-06-03T08:37:53.520 INFO:tasks.workunit.client.0.smithi088.stderr:++ for i in /etc/profile.d/*.sh 2024-06-03T08:37:53.521 INFO:tasks.workunit.client.0.smithi088.stderr:++ '[' -r /etc/profile.d/gawk.sh ']' 2024-06-03T08:37:53.521 INFO:tasks.workunit.client.0.smithi088.stderr:++ '[' '' ']' 2024-06-03T08:37:53.521 INFO:tasks.workunit.client.0.smithi088.stderr:++ . /etc/profile.d/gawk.sh 2024-06-03T08:37:53.521 INFO:tasks.workunit.client.0.smithi088.stderr:++++ /usr/bin/locale 2024-06-03T08:37:53.521 INFO:tasks.workunit.client.0.smithi088.stderr:++ for i in /etc/profile.d/*.sh 2024-06-03T08:37:53.521 INFO:tasks.workunit.client.0.smithi088.stderr:++ '[' -r /etc/profile.d/lang.sh ']' 2024-06-03T08:37:53.521 INFO:tasks.workunit.client.0.smithi088.stderr:++ '[' '' ']' 2024-06-03T08:37:53.521 INFO:tasks.workunit.client.0.smithi088.stderr:++ . /etc/profile.d/lang.sh 2024-06-03T08:37:53.521 INFO:tasks.workunit.client.0.smithi088.stderr:+++ unset LANG_backup 2024-06-03T08:37:53.521 INFO:tasks.workunit.client.0.smithi088.stderr:+++ alias 'xzgrep=xzgrep --color=auto' 2024-06-03T08:37:53.521 INFO:tasks.workunit.client.0.smithi088.stderr:+++ alias 'xzegrep=xzegrep --color=auto' 2024-06-03T08:37:53.521 INFO:tasks.workunit.client.0.smithi088.stderr:++++ /usr/bin/locale 2024-06-03T08:37:53.521 INFO:tasks.workunit.client.0.smithi088.stderr:+++ alias 'xzfgrep=xzfgrep --color=auto' 2024-06-03T08:37:53.521 INFO:tasks.workunit.client.0.smithi088.stderr:++ for i in /etc/profile.d/*.sh 2024-06-03T08:37:53.522 INFO:tasks.workunit.client.0.smithi088.stderr:++ '[' -r /etc/profile.d/colorzgrep.sh ']' 2024-06-03T08:37:53.522 INFO:tasks.workunit.client.0.smithi088.stderr:++ awk '{print $1}' 2024-06-03T08:37:53.522 INFO:tasks.workunit.client.0.smithi088.stderr:++ '[' '' ']' 2024-06-03T08:37:53.522 INFO:tasks.workunit.client.0.smithi088.stderr:++ . /etc/profile.d/colorzgrep.sh 2024-06-03T08:37:53.522 INFO:tasks.workunit.client.0.smithi088.stderr:+++ '[' -f /usr/libexec/grepconf.sh ']' 2024-06-03T08:37:53.522 INFO:tasks.workunit.client.0.smithi088.stderr:+++ /usr/libexec/grepconf.sh -c 2024-06-03T08:37:53.522 INFO:tasks.workunit.client.0.smithi088.stderr:+++ '[' -n '' ']' 2024-06-03T08:37:53.522 INFO:tasks.workunit.client.0.smithi088.stderr:+++ LANG_backup=en_US.UTF-8 2024-06-03T08:37:53.522 INFO:tasks.workunit.client.0.smithi088.stderr:+++ for config in /etc/locale.conf "${HOME}/.i18n" 2024-06-03T08:37:53.522 INFO:tasks.workunit.client.0.smithi088.stderr:+++ '[' -f /etc/locale.conf ']' 2024-06-03T08:37:53.522 INFO:tasks.workunit.client.0.smithi088.stderr:+++ '[' -x /usr/bin/sed ']' 2024-06-03T08:37:53.522 INFO:tasks.workunit.client.0.smithi088.stderr:+ ff=api_snapshots_pp 2024-06-03T08:37:53.522 INFO:tasks.workunit.client.0.smithi088.stderr:+ pid=40461 2024-06-03T08:37:53.523 INFO:tasks.workunit.client.0.smithi088.stderr:+ echo 'test api_snapshots_pp on pid 40461' 2024-06-03T08:37:53.523 INFO:tasks.workunit.client.0.smithi088.stdout:test api_snapshots_pp on pid 40461 2024-06-03T08:37:53.534 INFO:tasks.workunit.client.0.smithi088.stderr:++++ /usr/bin/sed -r -e 's/^[[:blank:]]*([[:upper:]_]+)=([[:print:][:digit:]\._-]+|"[[:print:][:digit:]\._-]+")/export \1=\2/;t;d' /etc/locale.conf 2024-06-03T08:37:53.534 INFO:tasks.workunit.client.0.smithi088.stderr:+ pids[$f]=40461 2024-06-03T08:37:53.534 INFO:tasks.workunit.client.0.smithi088.stderr:+ for f in api_aio api_aio_pp api_io api_io_pp api_asio api_list api_lock api_lock_pp api_misc api_misc_pp api_tier_pp api_pool api_snapshots api_snapshots_pp api_stat api_stat_pp api_watch_notify api_watch_notify_pp api_cmd api_cmd_pp api_service api_service_pp api_c_write_operations api_c_read_operations api_cls_remote_reads list_parallel open_pools_parallel delete_pools_parallel 2024-06-03T08:37:53.534 INFO:tasks.workunit.client.0.smithi088.stderr:+ '[' 1 -eq 1 ']' 2024-06-03T08:37:53.534 INFO:tasks.workunit.client.0.smithi088.stderr:+ sed 's/^/ api_lock: /' 2024-06-03T08:37:53.534 INFO:tasks.workunit.client.0.smithi088.stderr:+++ eval export 'LANG="en_US.UTF-8"' 2024-06-03T08:37:53.534 INFO:tasks.workunit.client.0.smithi088.stderr:++++ export LANG=en_US.UTF-8 2024-06-03T08:37:53.534 INFO:tasks.workunit.client.0.smithi088.stderr:++++ LANG=en_US.UTF-8 2024-06-03T08:37:53.535 INFO:tasks.workunit.client.0.smithi088.stderr:+++ for config in /etc/locale.conf "${HOME}/.i18n" 2024-06-03T08:37:53.535 INFO:tasks.workunit.client.0.smithi088.stderr:+++ '[' -f /home/ubuntu/.i18n ']' 2024-06-03T08:37:53.535 INFO:tasks.workunit.client.0.smithi088.stderr:+++ '[' -n en_US.UTF-8 ']' 2024-06-03T08:37:53.535 INFO:tasks.workunit.client.0.smithi088.stderr:+++ LANG=en_US.UTF-8 2024-06-03T08:37:53.535 INFO:tasks.workunit.client.0.smithi088.stderr:+++ unset LANG_backup config 2024-06-03T08:37:53.535 INFO:tasks.workunit.client.0.smithi088.stderr:+++ '[' -n '' ']' 2024-06-03T08:37:53.535 INFO:tasks.workunit.client.0.smithi088.stderr:+++ '[' -n en_US.UTF-8 ']' 2024-06-03T08:37:53.535 INFO:tasks.workunit.client.0.smithi088.stderr:+++ '[' dumb = linux ']' 2024-06-03T08:37:53.535 INFO:tasks.workunit.client.0.smithi088.stderr:++ for i in /etc/profile.d/*.sh 2024-06-03T08:37:53.535 INFO:tasks.workunit.client.0.smithi088.stderr:++ '[' -r /etc/profile.d/less.sh ']' 2024-06-03T08:37:53.535 INFO:tasks.workunit.client.0.smithi088.stderr:++ '[' '' ']' 2024-06-03T08:37:53.535 INFO:tasks.workunit.client.0.smithi088.stderr:++ . /etc/profile.d/less.sh 2024-06-03T08:37:53.535 INFO:tasks.workunit.client.0.smithi088.stderr:+++ '[' -z '||/usr/bin/lesspipe.sh %s' ']' 2024-06-03T08:37:53.535 INFO:tasks.workunit.client.0.smithi088.stderr:++ for i in /etc/profile.d/*.sh 2024-06-03T08:37:53.536 INFO:tasks.workunit.client.0.smithi088.stderr:++ '[' -r /etc/profile.d/which2.sh ']' 2024-06-03T08:37:53.536 INFO:tasks.workunit.client.0.smithi088.stderr:++ '[' '' ']' 2024-06-03T08:37:53.536 INFO:tasks.workunit.client.0.smithi088.stderr:++ . /etc/profile.d/which2.sh 2024-06-03T08:37:53.536 INFO:tasks.workunit.client.0.smithi088.stderr:+++ case "$(basename $(readlink /proc/$$/exe))" in 2024-06-03T08:37:53.536 INFO:tasks.workunit.client.0.smithi088.stderr:+++++ readlink /proc/40375/exe 2024-06-03T08:37:53.536 INFO:tasks.workunit.client.0.smithi088.stderr:++++ basename /usr/bin/bash 2024-06-03T08:37:53.536 INFO:tasks.workunit.client.0.smithi088.stderr:+++ which_declare='declare -f' 2024-06-03T08:37:53.536 INFO:tasks.workunit.client.0.smithi088.stderr:+++ which_opt=-f 2024-06-03T08:37:53.536 INFO:tasks.workunit.client.0.smithi088.stderr:+++ export which_declare 2024-06-03T08:37:53.536 INFO:tasks.workunit.client.0.smithi088.stderr:+++ export -f which 2024-06-03T08:37:53.536 INFO:tasks.workunit.client.0.smithi088.stderr:++ unset i 2024-06-03T08:37:53.536 INFO:tasks.workunit.client.0.smithi088.stderr:++ unset -f pathmunge 2024-06-03T08:37:53.536 INFO:tasks.workunit.client.0.smithi088.stderr:+ [[ /home/ubuntu/.local/bin:/home/ubuntu/bin:/usr/local/bin:/usr/bin:/usr/local/sbin:/usr/sbin:/usr/sbin =~ /home/ubuntu/\.local/bin:/home/ubuntu/bin: ]] 2024-06-03T08:37:53.537 INFO:tasks.workunit.client.0.smithi088.stderr:+ export PATH 2024-06-03T08:37:53.537 INFO:tasks.workunit.client.0.smithi088.stderr:+ '[' -d /home/ubuntu/.bashrc.d ']' 2024-06-03T08:37:53.537 INFO:tasks.workunit.client.0.smithi088.stderr:+ unset rc 2024-06-03T08:37:53.537 INFO:tasks.workunit.client.0.smithi088.stderr:+ sed 's/^/ api_pool: /' 2024-06-03T08:37:53.537 INFO:tasks.workunit.client.0.smithi088.stderr:+ tee ceph_test_rados_api_pool.log 2024-06-03T08:37:53.537 INFO:tasks.workunit.client.0.smithi088.stderr:+ ceph_test_rados_api_pool 2024-06-03T08:37:53.537 INFO:tasks.workunit.client.0.smithi088.stderr:++ printf %25s api_stat 2024-06-03T08:37:53.537 INFO:tasks.workunit.client.0.smithi088.stderr:+ r=' api_stat' 2024-06-03T08:37:53.537 INFO:tasks.workunit.client.0.smithi088.stderr:+ bash -o pipefail -exc 'ceph_test_rados_api_snapshots_pp 2>&1 | tee ceph_test_rados_api_snapshots_pp.log | sed "s/^/ api_snapshots_pp: /"' 2024-06-03T08:37:53.537 INFO:tasks.workunit.client.0.smithi088.stderr:+ '[' -f /etc/bashrc ']' 2024-06-03T08:37:53.537 INFO:tasks.workunit.client.0.smithi088.stderr:+ . /etc/bashrc 2024-06-03T08:37:53.537 INFO:tasks.workunit.client.0.smithi088.stderr:++ '[' -z '' ']' 2024-06-03T08:37:53.537 INFO:tasks.workunit.client.0.smithi088.stderr:++ BASHRCSOURCED=Y 2024-06-03T08:37:53.537 INFO:tasks.workunit.client.0.smithi088.stderr:++ '[' '' ']' 2024-06-03T08:37:53.538 INFO:tasks.workunit.client.0.smithi088.stderr:++ shopt -q login_shell 2024-06-03T08:37:53.538 INFO:tasks.workunit.client.0.smithi088.stderr:+++ alias 'zgrep=zgrep --color=auto' 2024-06-03T08:37:53.538 INFO:tasks.workunit.client.0.smithi088.stderr:+++ alias 'zfgrep=zfgrep --color=auto' 2024-06-03T08:37:53.538 INFO:tasks.workunit.client.0.smithi088.stderr:+++ alias 'zegrep=zegrep --color=auto' 2024-06-03T08:37:53.538 INFO:tasks.workunit.client.0.smithi088.stderr:++ for i in /etc/profile.d/*.sh 2024-06-03T08:37:53.538 INFO:tasks.workunit.client.0.smithi088.stderr:++ '[' -r /etc/profile.d/debuginfod.sh ']' 2024-06-03T08:37:53.538 INFO:tasks.workunit.client.0.smithi088.stderr:++ '[' '' ']' 2024-06-03T08:37:53.538 INFO:tasks.workunit.client.0.smithi088.stderr:++ . /etc/profile.d/debuginfod.sh 2024-06-03T08:37:53.538 INFO:tasks.workunit.client.0.smithi088.stderr:+++ '[' -z 'https://debuginfod.centos.org/ ' ']' 2024-06-03T08:37:53.538 INFO:tasks.workunit.client.0.smithi088.stderr:++ for i in /etc/profile.d/*.sh 2024-06-03T08:37:53.538 INFO:tasks.workunit.client.0.smithi088.stderr:++ '[' -r /etc/profile.d/gawk.sh ']' 2024-06-03T08:37:53.538 INFO:tasks.workunit.client.0.smithi088.stderr:++ '[' '' ']' 2024-06-03T08:37:53.538 INFO:tasks.workunit.client.0.smithi088.stderr:++ . /etc/profile.d/gawk.sh 2024-06-03T08:37:53.539 INFO:tasks.workunit.client.0.smithi088.stderr:++ for i in /etc/profile.d/*.sh 2024-06-03T08:37:53.539 INFO:tasks.workunit.client.0.smithi088.stderr:++ '[' -r /etc/profile.d/lang.sh ']' 2024-06-03T08:37:53.539 INFO:tasks.workunit.client.0.smithi088.stderr:++ '[' '' ']' 2024-06-03T08:37:53.539 INFO:tasks.workunit.client.0.smithi088.stderr:++ . /etc/profile.d/lang.sh 2024-06-03T08:37:53.539 INFO:tasks.workunit.client.0.smithi088.stderr:+++ unset LANG_backup 2024-06-03T08:37:53.539 INFO:tasks.workunit.client.0.smithi088.stderr:++++ /usr/bin/locale 2024-06-03T08:37:53.539 INFO:tasks.workunit.client.0.smithi088.stderr:+++ alias 'zgrep=zgrep --color=auto' 2024-06-03T08:37:53.539 INFO:tasks.workunit.client.0.smithi088.stderr:+++ alias 'zfgrep=zfgrep --color=auto' 2024-06-03T08:37:53.539 INFO:tasks.workunit.client.0.smithi088.stderr:+++ alias 'zegrep=zegrep --color=auto' 2024-06-03T08:37:53.539 INFO:tasks.workunit.client.0.smithi088.stderr:++ for i in /etc/profile.d/*.sh 2024-06-03T08:37:53.539 INFO:tasks.workunit.client.0.smithi088.stderr:++ '[' -r /etc/profile.d/debuginfod.sh ']' 2024-06-03T08:37:53.539 INFO:tasks.workunit.client.0.smithi088.stderr:++ '[' '' ']' 2024-06-03T08:37:53.539 INFO:tasks.workunit.client.0.smithi088.stderr:++ . /etc/profile.d/debuginfod.sh 2024-06-03T08:37:53.539 INFO:tasks.workunit.client.0.smithi088.stderr:+++ '[' -z 'https://debuginfod.centos.org/ ' ']' 2024-06-03T08:37:53.540 INFO:tasks.workunit.client.0.smithi088.stderr:++ for i in /etc/profile.d/*.sh 2024-06-03T08:37:53.540 INFO:tasks.workunit.client.0.smithi088.stderr:++ '[' -r /etc/profile.d/gawk.sh ']' 2024-06-03T08:37:53.540 INFO:tasks.workunit.client.0.smithi088.stderr:++ '[' '' ']' 2024-06-03T08:37:53.540 INFO:tasks.workunit.client.0.smithi088.stderr:++ . /etc/profile.d/gawk.sh 2024-06-03T08:37:53.540 INFO:tasks.workunit.client.0.smithi088.stderr:++ for i in /etc/profile.d/*.sh 2024-06-03T08:37:53.540 INFO:tasks.workunit.client.0.smithi088.stderr:++ '[' -r /etc/profile.d/lang.sh ']' 2024-06-03T08:37:53.540 INFO:tasks.workunit.client.0.smithi088.stderr:++ '[' '' ']' 2024-06-03T08:37:53.540 INFO:tasks.workunit.client.0.smithi088.stderr:++ . /etc/profile.d/lang.sh 2024-06-03T08:37:53.540 INFO:tasks.workunit.client.0.smithi088.stderr:+++ unset LANG_backup 2024-06-03T08:37:53.540 INFO:tasks.workunit.client.0.smithi088.stderr:+++ '[' -n '' ']' 2024-06-03T08:37:53.540 INFO:tasks.workunit.client.0.smithi088.stderr:+++ umask 2024-06-03T08:37:53.540 INFO:tasks.workunit.client.0.smithi088.stderr:+++ LANG_backup=en_US.UTF-8 2024-06-03T08:37:53.540 INFO:tasks.workunit.client.0.smithi088.stderr:+++ for config in /etc/locale.conf "${HOME}/.i18n" 2024-06-03T08:37:53.541 INFO:tasks.workunit.client.0.smithi088.stderr:+++ '[' -f /etc/locale.conf ']' 2024-06-03T08:37:53.541 INFO:tasks.workunit.client.0.smithi088.stderr:+++ '[' -x /usr/bin/sed ']' 2024-06-03T08:37:53.541 INFO:tasks.workunit.client.0.smithi088.stderr:++ '[' 0022 -eq 0 ']' 2024-06-03T08:37:53.541 INFO:tasks.workunit.client.0.smithi088.stderr:++++ /usr/bin/sed -r -e 's/^[[:blank:]]*([[:upper:]_]+)=([[:print:][:digit:]\._-]+|"[[:print:][:digit:]\._-]+")/export \1=\2/;t;d' /etc/locale.conf 2024-06-03T08:37:53.541 INFO:tasks.workunit.client.0.smithi088.stderr:++ awk '{print $1}' 2024-06-03T08:37:53.541 INFO:tasks.workunit.client.0.smithi088.stderr:++ echo api_stat 2024-06-03T08:37:53.541 INFO:tasks.workunit.client.0.smithi088.stderr:+++ eval export 'LANG="en_US.UTF-8"' 2024-06-03T08:37:53.541 INFO:tasks.workunit.client.0.smithi088.stderr:++++ export LANG=en_US.UTF-8 2024-06-03T08:37:53.541 INFO:tasks.workunit.client.0.smithi088.stderr:++++ /usr/bin/locale 2024-06-03T08:37:53.541 INFO:tasks.workunit.client.0.smithi088.stderr:++++ LANG=en_US.UTF-8 2024-06-03T08:37:53.541 INFO:tasks.workunit.client.0.smithi088.stderr:++ SHELL=/bin/bash 2024-06-03T08:37:53.541 INFO:tasks.workunit.client.0.smithi088.stderr:+++ for config in /etc/locale.conf "${HOME}/.i18n" 2024-06-03T08:37:53.541 INFO:tasks.workunit.client.0.smithi088.stderr:+++ '[' -f /home/ubuntu/.i18n ']' 2024-06-03T08:37:53.541 INFO:tasks.workunit.client.0.smithi088.stderr:++ for i in /etc/profile.d/*.sh 2024-06-03T08:37:53.542 INFO:tasks.workunit.client.0.smithi088.stderr:++ '[' -r /etc/profile.d/bash_completion.sh ']' 2024-06-03T08:37:53.542 INFO:tasks.workunit.client.0.smithi088.stderr:++ '[' '' ']' 2024-06-03T08:37:53.542 INFO:tasks.workunit.client.0.smithi088.stderr:++ . /etc/profile.d/bash_completion.sh 2024-06-03T08:37:53.542 INFO:tasks.workunit.client.0.smithi088.stderr:+++ '[' -n en_US.UTF-8 ']' 2024-06-03T08:37:53.542 INFO:tasks.workunit.client.0.smithi088.stderr:+++ LANG=en_US.UTF-8 2024-06-03T08:37:53.542 INFO:tasks.workunit.client.0.smithi088.stderr:+++ unset LANG_backup config 2024-06-03T08:37:53.542 INFO:tasks.workunit.client.0.smithi088.stderr:+++ '[' -n '' ']' 2024-06-03T08:37:53.542 INFO:tasks.workunit.client.0.smithi088.stderr:+++ '[' -n en_US.UTF-8 ']' 2024-06-03T08:37:53.542 INFO:tasks.workunit.client.0.smithi088.stderr:+++ '[' 'x5.1.8(1)-release' '!=' x -a x '!=' x -a x = x ']' 2024-06-03T08:37:53.542 INFO:tasks.workunit.client.0.smithi088.stderr:+++ '[' dumb = linux ']' 2024-06-03T08:37:53.542 INFO:tasks.workunit.client.0.smithi088.stderr:++ for i in /etc/profile.d/*.sh 2024-06-03T08:37:53.542 INFO:tasks.workunit.client.0.smithi088.stderr:++ '[' -r /etc/profile.d/less.sh ']' 2024-06-03T08:37:53.542 INFO:tasks.workunit.client.0.smithi088.stderr:++ '[' '' ']' 2024-06-03T08:37:53.542 INFO:tasks.workunit.client.0.smithi088.stderr:++ . /etc/profile.d/less.sh 2024-06-03T08:37:53.543 INFO:tasks.workunit.client.0.smithi088.stderr:++ for i in /etc/profile.d/*.sh 2024-06-03T08:37:53.543 INFO:tasks.workunit.client.0.smithi088.stderr:+++ '[' -z '||/usr/bin/lesspipe.sh %s' ']' 2024-06-03T08:37:53.543 INFO:tasks.workunit.client.0.smithi088.stderr:++ for i in /etc/profile.d/*.sh 2024-06-03T08:37:53.543 INFO:tasks.workunit.client.0.smithi088.stderr:++ '[' -r /etc/profile.d/which2.sh ']' 2024-06-03T08:37:53.543 INFO:tasks.workunit.client.0.smithi088.stderr:++ '[' '' ']' 2024-06-03T08:37:53.543 INFO:tasks.workunit.client.0.smithi088.stderr:++ . /etc/profile.d/which2.sh 2024-06-03T08:37:53.543 INFO:tasks.workunit.client.0.smithi088.stderr:++ '[' -r /etc/profile.d/cobbler.sh ']' 2024-06-03T08:37:53.543 INFO:tasks.workunit.client.0.smithi088.stderr:+++ case "$(basename $(readlink /proc/$$/exe))" in 2024-06-03T08:37:53.543 INFO:tasks.workunit.client.0.smithi088.stderr:++ '[' '' ']' 2024-06-03T08:37:53.543 INFO:tasks.workunit.client.0.smithi088.stderr:++ . /etc/profile.d/cobbler.sh 2024-06-03T08:37:53.543 INFO:tasks.workunit.client.0.smithi088.stderr:+++ export COBBLER_SERVER=172.21.0.11 2024-06-03T08:37:53.543 INFO:tasks.workunit.client.0.smithi088.stderr:+++ COBBLER_SERVER=172.21.0.11 2024-06-03T08:37:53.543 INFO:tasks.workunit.client.0.smithi088.stderr:++ for i in /etc/profile.d/*.sh 2024-06-03T08:37:53.544 INFO:tasks.workunit.client.0.smithi088.stderr:++ '[' -r /etc/profile.d/colorgrep.sh ']' 2024-06-03T08:37:53.544 INFO:tasks.workunit.client.0.smithi088.stderr:++ '[' '' ']' 2024-06-03T08:37:53.544 INFO:tasks.workunit.client.0.smithi088.stderr:++ . /etc/profile.d/colorgrep.sh 2024-06-03T08:37:53.544 INFO:tasks.workunit.client.0.smithi088.stderr:+++ /usr/libexec/grepconf.sh -c 2024-06-03T08:37:53.544 INFO:tasks.workunit.client.0.smithi088.stdout:test api_stat on pid 40512 2024-06-03T08:37:53.557 INFO:tasks.workunit.client.0.smithi088.stderr:+ ff=api_stat 2024-06-03T08:37:53.557 INFO:tasks.workunit.client.0.smithi088.stderr:+ pid=40512 2024-06-03T08:37:53.557 INFO:tasks.workunit.client.0.smithi088.stderr:+ echo 'test api_stat on pid 40512' 2024-06-03T08:37:53.557 INFO:tasks.workunit.client.0.smithi088.stderr:+ pids[$f]=40512 2024-06-03T08:37:53.558 INFO:tasks.workunit.client.0.smithi088.stderr:+ for f in api_aio api_aio_pp api_io api_io_pp api_asio api_list api_lock api_lock_pp api_misc api_misc_pp api_tier_pp api_pool api_snapshots api_snapshots_pp api_stat api_stat_pp api_watch_notify api_watch_notify_pp api_cmd api_cmd_pp api_service api_service_pp api_c_write_operations api_c_read_operations api_cls_remote_reads list_parallel open_pools_parallel delete_pools_parallel 2024-06-03T08:37:53.558 INFO:tasks.workunit.client.0.smithi088.stderr:+ '[' 1 -eq 1 ']' 2024-06-03T08:37:53.558 INFO:tasks.workunit.client.0.smithi088.stderr:+++ '[' -n '' ']' 2024-06-03T08:37:53.558 INFO:tasks.workunit.client.0.smithi088.stderr:+++ LANG_backup=en_US.UTF-8 2024-06-03T08:37:53.558 INFO:tasks.workunit.client.0.smithi088.stderr:+++ for config in /etc/locale.conf "${HOME}/.i18n" 2024-06-03T08:37:53.558 INFO:tasks.workunit.client.0.smithi088.stderr:+++ '[' -f /etc/locale.conf ']' 2024-06-03T08:37:53.558 INFO:tasks.workunit.client.0.smithi088.stderr:+++ '[' -x /usr/bin/sed ']' 2024-06-03T08:37:53.558 INFO:tasks.workunit.client.0.smithi088.stderr:+++ '[' -n '' ']' 2024-06-03T08:37:53.558 INFO:tasks.workunit.client.0.smithi088.stderr:+++ LANG_backup=en_US.UTF-8 2024-06-03T08:37:53.558 INFO:tasks.workunit.client.0.smithi088.stderr:+++ for config in /etc/locale.conf "${HOME}/.i18n" 2024-06-03T08:37:53.558 INFO:tasks.workunit.client.0.smithi088.stderr:+ bash -o pipefail -exc 'ceph_test_rados_api_stat 2>&1 | tee ceph_test_rados_api_stat.log | sed "s/^/ api_stat: /"' 2024-06-03T08:37:53.559 INFO:tasks.workunit.client.0.smithi088.stderr:+++ '[' -f /etc/locale.conf ']' 2024-06-03T08:37:53.559 INFO:tasks.workunit.client.0.smithi088.stderr:+++ '[' -x /usr/bin/sed ']' 2024-06-03T08:37:53.559 INFO:tasks.workunit.client.0.smithi088.stderr:+++++ readlink /proc/40322/exe 2024-06-03T08:37:53.559 INFO:tasks.workunit.client.0.smithi088.stderr:++++ basename /usr/bin/bash 2024-06-03T08:37:53.559 INFO:tasks.workunit.client.0.smithi088.stderr:++++ /usr/bin/sed -r -e 's/^[[:blank:]]*([[:upper:]_]+)=([[:print:][:digit:]\._-]+|"[[:print:][:digit:]\._-]+")/export \1=\2/;t;d' /etc/locale.conf 2024-06-03T08:37:53.559 INFO:tasks.workunit.client.0.smithi088.stderr:+ '[' -f /etc/bashrc ']' 2024-06-03T08:37:53.559 INFO:tasks.workunit.client.0.smithi088.stderr:+ . /etc/bashrc 2024-06-03T08:37:53.559 INFO:tasks.workunit.client.0.smithi088.stderr:++ printf %25s api_stat_pp 2024-06-03T08:37:53.559 INFO:tasks.workunit.client.0.smithi088.stderr:++ '[' -z '' ']' 2024-06-03T08:37:53.559 INFO:tasks.workunit.client.0.smithi088.stderr:++ BASHRCSOURCED=Y 2024-06-03T08:37:53.559 INFO:tasks.workunit.client.0.smithi088.stderr:++ '[' '' ']' 2024-06-03T08:37:53.559 INFO:tasks.workunit.client.0.smithi088.stderr:++ shopt -q login_shell 2024-06-03T08:37:53.559 INFO:tasks.workunit.client.0.smithi088.stderr:+++ which_declare='declare -f' 2024-06-03T08:37:53.560 INFO:tasks.workunit.client.0.smithi088.stderr:+++ which_opt=-f 2024-06-03T08:37:53.560 INFO:tasks.workunit.client.0.smithi088.stderr:+++ eval export 'LANG="en_US.UTF-8"' 2024-06-03T08:37:53.560 INFO:tasks.workunit.client.0.smithi088.stderr:++++ export LANG=en_US.UTF-8 2024-06-03T08:37:53.560 INFO:tasks.workunit.client.0.smithi088.stderr:++++ LANG=en_US.UTF-8 2024-06-03T08:37:53.560 INFO:tasks.workunit.client.0.smithi088.stderr:+++ for config in /etc/locale.conf "${HOME}/.i18n" 2024-06-03T08:37:53.560 INFO:tasks.workunit.client.0.smithi088.stderr:+++ export which_declare 2024-06-03T08:37:53.560 INFO:tasks.workunit.client.0.smithi088.stderr:+++ export -f which 2024-06-03T08:37:53.560 INFO:tasks.workunit.client.0.smithi088.stderr:+++ '[' -f /home/ubuntu/.i18n ']' 2024-06-03T08:37:53.560 INFO:tasks.workunit.client.0.smithi088.stderr:+++ '[' -n en_US.UTF-8 ']' 2024-06-03T08:37:53.560 INFO:tasks.workunit.client.0.smithi088.stderr:+++ LANG=en_US.UTF-8 2024-06-03T08:37:53.560 INFO:tasks.workunit.client.0.smithi088.stderr:+++ unset LANG_backup config 2024-06-03T08:37:53.560 INFO:tasks.workunit.client.0.smithi088.stderr:++ unset i 2024-06-03T08:37:53.560 INFO:tasks.workunit.client.0.smithi088.stderr:++ unset -f pathmunge 2024-06-03T08:37:53.561 INFO:tasks.workunit.client.0.smithi088.stderr:+ [[ /home/ubuntu/.local/bin:/home/ubuntu/bin:/usr/local/bin:/usr/bin:/usr/local/sbin:/usr/sbin:/usr/sbin =~ /home/ubuntu/\.local/bin:/home/ubuntu/bin: ]] 2024-06-03T08:37:53.561 INFO:tasks.workunit.client.0.smithi088.stderr:+++ '[' -n '' ']' 2024-06-03T08:37:53.561 INFO:tasks.workunit.client.0.smithi088.stderr:+++ '[' -n en_US.UTF-8 ']' 2024-06-03T08:37:53.561 INFO:tasks.workunit.client.0.smithi088.stderr:+ export PATH 2024-06-03T08:37:53.561 INFO:tasks.workunit.client.0.smithi088.stderr:+++ '[' dumb = linux ']' 2024-06-03T08:37:53.561 INFO:tasks.workunit.client.0.smithi088.stderr:++ for i in /etc/profile.d/*.sh 2024-06-03T08:37:53.561 INFO:tasks.workunit.client.0.smithi088.stderr:++ '[' -r /etc/profile.d/less.sh ']' 2024-06-03T08:37:53.561 INFO:tasks.workunit.client.0.smithi088.stderr:++ '[' '' ']' 2024-06-03T08:37:53.561 INFO:tasks.workunit.client.0.smithi088.stderr:++ . /etc/profile.d/less.sh 2024-06-03T08:37:53.561 INFO:tasks.workunit.client.0.smithi088.stderr:+++ '[' -z '||/usr/bin/lesspipe.sh %s' ']' 2024-06-03T08:37:53.561 INFO:tasks.workunit.client.0.smithi088.stderr:+ '[' -d /home/ubuntu/.bashrc.d ']' 2024-06-03T08:37:53.561 INFO:tasks.workunit.client.0.smithi088.stderr:+ unset rc 2024-06-03T08:37:53.561 INFO:tasks.workunit.client.0.smithi088.stderr:++ for i in /etc/profile.d/*.sh 2024-06-03T08:37:53.561 INFO:tasks.workunit.client.0.smithi088.stderr:++ '[' -r /etc/profile.d/which2.sh ']' 2024-06-03T08:37:53.562 INFO:tasks.workunit.client.0.smithi088.stderr:+ ceph_test_rados_api_misc 2024-06-03T08:37:53.562 INFO:tasks.workunit.client.0.smithi088.stderr:++++ /usr/bin/sed -r -e 's/^[[:blank:]]*([[:upper:]_]+)=([[:print:][:digit:]\._-]+|"[[:print:][:digit:]\._-]+")/export \1=\2/;t;d' /etc/locale.conf 2024-06-03T08:37:53.562 INFO:tasks.workunit.client.0.smithi088.stderr:++ '[' '' ']' 2024-06-03T08:37:53.562 INFO:tasks.workunit.client.0.smithi088.stderr:++ . /etc/profile.d/which2.sh 2024-06-03T08:37:53.562 INFO:tasks.workunit.client.0.smithi088.stderr:+++ case "$(basename $(readlink /proc/$$/exe))" in 2024-06-03T08:37:53.562 INFO:tasks.workunit.client.0.smithi088.stderr:+++ alias 'grep=grep --color=auto' 2024-06-03T08:37:53.562 INFO:tasks.workunit.client.0.smithi088.stderr:+++ alias 'egrep=egrep --color=auto' 2024-06-03T08:37:53.562 INFO:tasks.workunit.client.0.smithi088.stderr:+++ alias 'fgrep=fgrep --color=auto' 2024-06-03T08:37:53.562 INFO:tasks.workunit.client.0.smithi088.stderr:++ for i in /etc/profile.d/*.sh 2024-06-03T08:37:53.562 INFO:tasks.workunit.client.0.smithi088.stderr:++ '[' -r /etc/profile.d/colorls.sh ']' 2024-06-03T08:37:53.562 INFO:tasks.workunit.client.0.smithi088.stderr:++ '[' '' ']' 2024-06-03T08:37:53.562 INFO:tasks.workunit.client.0.smithi088.stderr:++ . /etc/profile.d/colorls.sh 2024-06-03T08:37:53.562 INFO:tasks.workunit.client.0.smithi088.stderr:+ r=' api_stat_pp' 2024-06-03T08:37:53.563 INFO:tasks.workunit.client.0.smithi088.stderr:+++ '[' '!' -t 0 ']' 2024-06-03T08:37:53.563 INFO:tasks.workunit.client.0.smithi088.stderr:+++ return 2024-06-03T08:37:53.563 INFO:tasks.workunit.client.0.smithi088.stderr:++ for i in /etc/profile.d/*.sh 2024-06-03T08:37:53.563 INFO:tasks.workunit.client.0.smithi088.stderr:++ '[' -r /etc/profile.d/colorsysstat.sh ']' 2024-06-03T08:37:53.563 INFO:tasks.workunit.client.0.smithi088.stderr:++ '[' '' ']' 2024-06-03T08:37:53.563 INFO:tasks.workunit.client.0.smithi088.stderr:++ . /etc/profile.d/colorsysstat.sh 2024-06-03T08:37:53.563 INFO:tasks.workunit.client.0.smithi088.stderr:+++ eval export 'LANG="en_US.UTF-8"' 2024-06-03T08:37:53.563 INFO:tasks.workunit.client.0.smithi088.stderr:+++ export S_COLORS=auto 2024-06-03T08:37:53.563 INFO:tasks.workunit.client.0.smithi088.stderr:+++ S_COLORS=auto 2024-06-03T08:37:53.563 INFO:tasks.workunit.client.0.smithi088.stderr:++ for i in /etc/profile.d/*.sh 2024-06-03T08:37:53.563 INFO:tasks.workunit.client.0.smithi088.stderr:++ '[' -r /etc/profile.d/colorxzgrep.sh ']' 2024-06-03T08:37:53.563 INFO:tasks.workunit.client.0.smithi088.stderr:++ '[' '' ']' 2024-06-03T08:37:53.563 INFO:tasks.workunit.client.0.smithi088.stderr:++ . /etc/profile.d/colorxzgrep.sh 2024-06-03T08:37:53.564 INFO:tasks.workunit.client.0.smithi088.stderr:+++ /usr/libexec/grepconf.sh -c 2024-06-03T08:37:53.564 INFO:tasks.workunit.client.0.smithi088.stderr:++++ export LANG=en_US.UTF-8 2024-06-03T08:37:53.564 INFO:tasks.workunit.client.0.smithi088.stderr:++++ LANG=en_US.UTF-8 2024-06-03T08:37:53.564 INFO:tasks.workunit.client.0.smithi088.stderr:+++ for config in /etc/locale.conf "${HOME}/.i18n" 2024-06-03T08:37:53.564 INFO:tasks.workunit.client.0.smithi088.stderr:+++ '[' -f /home/ubuntu/.i18n ']' 2024-06-03T08:37:53.564 INFO:tasks.workunit.client.0.smithi088.stderr:+++ '[' -n en_US.UTF-8 ']' 2024-06-03T08:37:53.564 INFO:tasks.workunit.client.0.smithi088.stderr:+++ LANG=en_US.UTF-8 2024-06-03T08:37:53.564 INFO:tasks.workunit.client.0.smithi088.stderr:+++ unset LANG_backup config 2024-06-03T08:37:53.564 INFO:tasks.workunit.client.0.smithi088.stderr:+++ '[' -n '' ']' 2024-06-03T08:37:53.564 INFO:tasks.workunit.client.0.smithi088.stderr:+++ '[' -n en_US.UTF-8 ']' 2024-06-03T08:37:53.564 INFO:tasks.workunit.client.0.smithi088.stderr:+++ '[' dumb = linux ']' 2024-06-03T08:37:53.564 INFO:tasks.workunit.client.0.smithi088.stderr:++ for i in /etc/profile.d/*.sh 2024-06-03T08:37:53.564 INFO:tasks.workunit.client.0.smithi088.stderr:++ '[' -r /etc/profile.d/less.sh ']' 2024-06-03T08:37:53.565 INFO:tasks.workunit.client.0.smithi088.stderr:++ '[' '' ']' 2024-06-03T08:37:53.565 INFO:tasks.workunit.client.0.smithi088.stderr:++ . /etc/profile.d/less.sh 2024-06-03T08:37:53.565 INFO:tasks.workunit.client.0.smithi088.stderr:+++ '[' -z '||/usr/bin/lesspipe.sh %s' ']' 2024-06-03T08:37:53.565 INFO:tasks.workunit.client.0.smithi088.stderr:++ for i in /etc/profile.d/*.sh 2024-06-03T08:37:53.565 INFO:tasks.workunit.client.0.smithi088.stderr:++ '[' -r /etc/profile.d/which2.sh ']' 2024-06-03T08:37:53.565 INFO:tasks.workunit.client.0.smithi088.stderr:++ '[' '' ']' 2024-06-03T08:37:53.565 INFO:tasks.workunit.client.0.smithi088.stderr:++ . /etc/profile.d/which2.sh 2024-06-03T08:37:53.565 INFO:tasks.workunit.client.0.smithi088.stderr:+++ case "$(basename $(readlink /proc/$$/exe))" in 2024-06-03T08:37:53.565 INFO:tasks.workunit.client.0.smithi088.stderr:+++++ readlink /proc/40415/exe 2024-06-03T08:37:53.565 INFO:tasks.workunit.client.0.smithi088.stderr:++++ basename /usr/bin/bash 2024-06-03T08:37:53.565 INFO:tasks.workunit.client.0.smithi088.stderr:+++ alias 'xzgrep=xzgrep --color=auto' 2024-06-03T08:37:53.565 INFO:tasks.workunit.client.0.smithi088.stderr:+++ alias 'xzegrep=xzegrep --color=auto' 2024-06-03T08:37:53.565 INFO:tasks.workunit.client.0.smithi088.stderr:+++ alias 'xzfgrep=xzfgrep --color=auto' 2024-06-03T08:37:53.565 INFO:tasks.workunit.client.0.smithi088.stderr:++ for i in /etc/profile.d/*.sh 2024-06-03T08:37:53.566 INFO:tasks.workunit.client.0.smithi088.stderr:++ '[' -r /etc/profile.d/colorzgrep.sh ']' 2024-06-03T08:37:53.566 INFO:tasks.workunit.client.0.smithi088.stderr:++ '[' '' ']' 2024-06-03T08:37:53.566 INFO:tasks.workunit.client.0.smithi088.stderr:++ . /etc/profile.d/colorzgrep.sh 2024-06-03T08:37:53.566 INFO:tasks.workunit.client.0.smithi088.stderr:+++ '[' -f /usr/libexec/grepconf.sh ']' 2024-06-03T08:37:53.566 INFO:tasks.workunit.client.0.smithi088.stderr:+++ /usr/libexec/grepconf.sh -c 2024-06-03T08:37:53.566 INFO:tasks.workunit.client.0.smithi088.stderr:+++ which_declare='declare -f' 2024-06-03T08:37:53.566 INFO:tasks.workunit.client.0.smithi088.stderr:+++ which_opt=-f 2024-06-03T08:37:53.566 INFO:tasks.workunit.client.0.smithi088.stderr:+++ export which_declare 2024-06-03T08:37:53.566 INFO:tasks.workunit.client.0.smithi088.stderr:+++ export -f which 2024-06-03T08:37:53.566 INFO:tasks.workunit.client.0.smithi088.stderr:++ unset i 2024-06-03T08:37:53.566 INFO:tasks.workunit.client.0.smithi088.stderr:++ unset -f pathmunge 2024-06-03T08:37:53.566 INFO:tasks.workunit.client.0.smithi088.stderr:++ awk '{print $1}' 2024-06-03T08:37:53.567 INFO:tasks.workunit.client.0.smithi088.stderr:+ [[ /home/ubuntu/.local/bin:/home/ubuntu/bin:/usr/local/bin:/usr/bin:/usr/local/sbin:/usr/sbin:/usr/sbin =~ /home/ubuntu/\.local/bin:/home/ubuntu/bin: ]] 2024-06-03T08:37:53.567 INFO:tasks.workunit.client.0.smithi088.stderr:+ export PATH 2024-06-03T08:37:53.567 INFO:tasks.workunit.client.0.smithi088.stderr:+++ umask 2024-06-03T08:37:53.567 INFO:tasks.workunit.client.0.smithi088.stderr:++ '[' 0022 -eq 0 ']' 2024-06-03T08:37:53.567 INFO:tasks.workunit.client.0.smithi088.stderr:+ '[' -d /home/ubuntu/.bashrc.d ']' 2024-06-03T08:37:53.567 INFO:tasks.workunit.client.0.smithi088.stderr:+ unset rc 2024-06-03T08:37:53.567 INFO:tasks.workunit.client.0.smithi088.stderr:++ SHELL=/bin/bash 2024-06-03T08:37:53.567 INFO:tasks.workunit.client.0.smithi088.stderr:+++++ readlink /proc/40367/exe 2024-06-03T08:37:53.567 INFO:tasks.workunit.client.0.smithi088.stderr:++ for i in /etc/profile.d/*.sh 2024-06-03T08:37:53.567 INFO:tasks.workunit.client.0.smithi088.stderr:++ '[' -r /etc/profile.d/bash_completion.sh ']' 2024-06-03T08:37:53.567 INFO:tasks.workunit.client.0.smithi088.stderr:++ '[' '' ']' 2024-06-03T08:37:53.567 INFO:tasks.workunit.client.0.smithi088.stderr:++ . /etc/profile.d/bash_completion.sh 2024-06-03T08:37:53.567 INFO:tasks.workunit.client.0.smithi088.stderr:+++ '[' 'x5.1.8(1)-release' '!=' x -a x '!=' x -a x = x ']' 2024-06-03T08:37:53.568 INFO:tasks.workunit.client.0.smithi088.stderr:++ for i in /etc/profile.d/*.sh 2024-06-03T08:37:53.568 INFO:tasks.workunit.client.0.smithi088.stderr:++ '[' -r /etc/profile.d/cobbler.sh ']' 2024-06-03T08:37:53.568 INFO:tasks.workunit.client.0.smithi088.stderr:++ '[' '' ']' 2024-06-03T08:37:53.568 INFO:tasks.workunit.client.0.smithi088.stderr:++ . /etc/profile.d/cobbler.sh 2024-06-03T08:37:53.568 INFO:tasks.workunit.client.0.smithi088.stderr:+++ export COBBLER_SERVER=172.21.0.11 2024-06-03T08:37:53.568 INFO:tasks.workunit.client.0.smithi088.stderr:+++ COBBLER_SERVER=172.21.0.11 2024-06-03T08:37:53.568 INFO:tasks.workunit.client.0.smithi088.stderr:++ for i in /etc/profile.d/*.sh 2024-06-03T08:37:53.568 INFO:tasks.workunit.client.0.smithi088.stderr:++ '[' -r /etc/profile.d/colorgrep.sh ']' 2024-06-03T08:37:53.568 INFO:tasks.workunit.client.0.smithi088.stderr:++ '[' '' ']' 2024-06-03T08:37:53.568 INFO:tasks.workunit.client.0.smithi088.stderr:++ . /etc/profile.d/colorgrep.sh 2024-06-03T08:37:53.568 INFO:tasks.workunit.client.0.smithi088.stderr:+++ /usr/libexec/grepconf.sh -c 2024-06-03T08:37:53.568 INFO:tasks.workunit.client.0.smithi088.stderr:++++ basename /usr/bin/bash 2024-06-03T08:37:53.568 INFO:tasks.workunit.client.0.smithi088.stderr:+ ceph_test_rados_api_snapshots 2024-06-03T08:37:53.569 INFO:tasks.workunit.client.0.smithi088.stderr:+++ which_declare='declare -f' 2024-06-03T08:37:53.569 INFO:tasks.workunit.client.0.smithi088.stderr:+++ which_opt=-f 2024-06-03T08:37:53.569 INFO:tasks.workunit.client.0.smithi088.stderr:+++ export which_declare 2024-06-03T08:37:53.569 INFO:tasks.workunit.client.0.smithi088.stderr:+++ export -f which 2024-06-03T08:37:53.569 INFO:tasks.workunit.client.0.smithi088.stderr:++ unset i 2024-06-03T08:37:53.569 INFO:tasks.workunit.client.0.smithi088.stderr:++ unset -f pathmunge 2024-06-03T08:37:53.569 INFO:tasks.workunit.client.0.smithi088.stderr:+ [[ /home/ubuntu/.local/bin:/home/ubuntu/bin:/usr/local/bin:/usr/bin:/usr/local/sbin:/usr/sbin:/usr/sbin =~ /home/ubuntu/\.local/bin:/home/ubuntu/bin: ]] 2024-06-03T08:37:53.569 INFO:tasks.workunit.client.0.smithi088.stderr:+++ alias 'zgrep=zgrep --color=auto' 2024-06-03T08:37:53.569 INFO:tasks.workunit.client.0.smithi088.stderr:+++ alias 'zfgrep=zfgrep --color=auto' 2024-06-03T08:37:53.569 INFO:tasks.workunit.client.0.smithi088.stderr:+++ alias 'zegrep=zegrep --color=auto' 2024-06-03T08:37:53.569 INFO:tasks.workunit.client.0.smithi088.stderr:++ for i in /etc/profile.d/*.sh 2024-06-03T08:37:53.569 INFO:tasks.workunit.client.0.smithi088.stderr:++ '[' -r /etc/profile.d/debuginfod.sh ']' 2024-06-03T08:37:53.569 INFO:tasks.workunit.client.0.smithi088.stderr:++ '[' '' ']' 2024-06-03T08:37:53.569 INFO:tasks.workunit.client.0.smithi088.stderr:++ . /etc/profile.d/debuginfod.sh 2024-06-03T08:37:53.570 INFO:tasks.workunit.client.0.smithi088.stderr:+ export PATH 2024-06-03T08:37:53.570 INFO:tasks.workunit.client.0.smithi088.stderr:+++ '[' -z 'https://debuginfod.centos.org/ ' ']' 2024-06-03T08:37:53.570 INFO:tasks.workunit.client.0.smithi088.stderr:++ for i in /etc/profile.d/*.sh 2024-06-03T08:37:53.570 INFO:tasks.workunit.client.0.smithi088.stderr:++ '[' -r /etc/profile.d/gawk.sh ']' 2024-06-03T08:37:53.570 INFO:tasks.workunit.client.0.smithi088.stderr:++ '[' '' ']' 2024-06-03T08:37:53.570 INFO:tasks.workunit.client.0.smithi088.stderr:++ . /etc/profile.d/gawk.sh 2024-06-03T08:37:53.570 INFO:tasks.workunit.client.0.smithi088.stderr:+ '[' -d /home/ubuntu/.bashrc.d ']' 2024-06-03T08:37:53.570 INFO:tasks.workunit.client.0.smithi088.stderr:+ unset rc 2024-06-03T08:37:53.570 INFO:tasks.workunit.client.0.smithi088.stderr:++ for i in /etc/profile.d/*.sh 2024-06-03T08:37:53.570 INFO:tasks.workunit.client.0.smithi088.stderr:++ '[' -r /etc/profile.d/lang.sh ']' 2024-06-03T08:37:53.570 INFO:tasks.workunit.client.0.smithi088.stderr:++ '[' '' ']' 2024-06-03T08:37:53.570 INFO:tasks.workunit.client.0.smithi088.stderr:++ . /etc/profile.d/lang.sh 2024-06-03T08:37:53.570 INFO:tasks.workunit.client.0.smithi088.stderr:+++ unset LANG_backup 2024-06-03T08:37:53.571 INFO:tasks.workunit.client.0.smithi088.stderr:+ tee ceph_test_rados_api_misc.log 2024-06-03T08:37:53.571 INFO:tasks.workunit.client.0.smithi088.stderr:+ ceph_test_rados_api_tier_pp 2024-06-03T08:37:53.571 INFO:tasks.workunit.client.0.smithi088.stderr:+ sed 's/^/ api_misc: /' 2024-06-03T08:37:53.571 INFO:tasks.workunit.client.0.smithi088.stderr:++ echo api_stat_pp 2024-06-03T08:37:53.571 INFO:tasks.workunit.client.0.smithi088.stderr:+ tee ceph_test_rados_api_snapshots.log 2024-06-03T08:37:53.571 INFO:tasks.workunit.client.0.smithi088.stderr:+ ff=api_stat_pp 2024-06-03T08:37:53.571 INFO:tasks.workunit.client.0.smithi088.stderr:+ pid=40564 2024-06-03T08:37:53.571 INFO:tasks.workunit.client.0.smithi088.stderr:+ echo 'test api_stat_pp on pid 40564' 2024-06-03T08:37:53.571 INFO:tasks.workunit.client.0.smithi088.stdout:test api_stat_pp on pid 40564 2024-06-03T08:37:53.576 INFO:tasks.workunit.client.0.smithi088.stderr:+ pids[$f]=40564 2024-06-03T08:37:53.576 INFO:tasks.workunit.client.0.smithi088.stderr:+ for f in api_aio api_aio_pp api_io api_io_pp api_asio api_list api_lock api_lock_pp api_misc api_misc_pp api_tier_pp api_pool api_snapshots api_snapshots_pp api_stat api_stat_pp api_watch_notify api_watch_notify_pp api_cmd api_cmd_pp api_service api_service_pp api_c_write_operations api_c_read_operations api_cls_remote_reads list_parallel open_pools_parallel delete_pools_parallel 2024-06-03T08:37:53.576 INFO:tasks.workunit.client.0.smithi088.stderr:+ '[' 1 -eq 1 ']' 2024-06-03T08:37:53.576 INFO:tasks.workunit.client.0.smithi088.stderr:+ bash -o pipefail -exc 'ceph_test_rados_api_stat_pp 2>&1 | tee ceph_test_rados_api_stat_pp.log | sed "s/^/ api_stat_pp: /"' 2024-06-03T08:37:53.576 INFO:tasks.workunit.client.0.smithi088.stderr:++++ /usr/bin/locale 2024-06-03T08:37:53.576 INFO:tasks.workunit.client.0.smithi088.stderr:+ '[' -f /etc/bashrc ']' 2024-06-03T08:37:53.576 INFO:tasks.workunit.client.0.smithi088.stderr:+ . /etc/bashrc 2024-06-03T08:37:53.576 INFO:tasks.workunit.client.0.smithi088.stderr:++ '[' -z '' ']' 2024-06-03T08:37:53.576 INFO:tasks.workunit.client.0.smithi088.stderr:++ BASHRCSOURCED=Y 2024-06-03T08:37:53.576 INFO:tasks.workunit.client.0.smithi088.stderr:++ '[' '' ']' 2024-06-03T08:37:53.576 INFO:tasks.workunit.client.0.smithi088.stderr:++ shopt -q login_shell 2024-06-03T08:37:53.576 INFO:tasks.workunit.client.0.smithi088.stderr:+ sed 's/^/ api_snapshots: /' 2024-06-03T08:37:53.577 INFO:tasks.workunit.client.0.smithi088.stderr:+++ '[' -n '' ']' 2024-06-03T08:37:53.577 INFO:tasks.workunit.client.0.smithi088.stderr:+++ umask 2024-06-03T08:37:53.577 INFO:tasks.workunit.client.0.smithi088.stderr:+++ LANG_backup=en_US.UTF-8 2024-06-03T08:37:53.577 INFO:tasks.workunit.client.0.smithi088.stderr:++ '[' 0022 -eq 0 ']' 2024-06-03T08:37:53.577 INFO:tasks.workunit.client.0.smithi088.stderr:++ SHELL=/bin/bash 2024-06-03T08:37:53.577 INFO:tasks.workunit.client.0.smithi088.stderr:+++ for config in /etc/locale.conf "${HOME}/.i18n" 2024-06-03T08:37:53.577 INFO:tasks.workunit.client.0.smithi088.stderr:+++ '[' -f /etc/locale.conf ']' 2024-06-03T08:37:53.577 INFO:tasks.workunit.client.0.smithi088.stderr:++ for i in /etc/profile.d/*.sh 2024-06-03T08:37:53.577 INFO:tasks.workunit.client.0.smithi088.stderr:+++ '[' -x /usr/bin/sed ']' 2024-06-03T08:37:53.577 INFO:tasks.workunit.client.0.smithi088.stderr:++ '[' -r /etc/profile.d/bash_completion.sh ']' 2024-06-03T08:37:53.577 INFO:tasks.workunit.client.0.smithi088.stderr:++ '[' '' ']' 2024-06-03T08:37:53.577 INFO:tasks.workunit.client.0.smithi088.stderr:++ . /etc/profile.d/bash_completion.sh 2024-06-03T08:37:53.577 INFO:tasks.workunit.client.0.smithi088.stderr:+++ '[' 'x5.1.8(1)-release' '!=' x -a x '!=' x -a x = x ']' 2024-06-03T08:37:53.577 INFO:tasks.workunit.client.0.smithi088.stderr:+ tee ceph_test_rados_api_tier_pp.log 2024-06-03T08:37:53.578 INFO:tasks.workunit.client.0.smithi088.stderr:++ for i in /etc/profile.d/*.sh 2024-06-03T08:37:53.578 INFO:tasks.workunit.client.0.smithi088.stderr:++ '[' -r /etc/profile.d/cobbler.sh ']' 2024-06-03T08:37:53.578 INFO:tasks.workunit.client.0.smithi088.stderr:++ '[' '' ']' 2024-06-03T08:37:53.578 INFO:tasks.workunit.client.0.smithi088.stderr:++ . /etc/profile.d/cobbler.sh 2024-06-03T08:37:53.578 INFO:tasks.workunit.client.0.smithi088.stderr:+++ export COBBLER_SERVER=172.21.0.11 2024-06-03T08:37:53.578 INFO:tasks.workunit.client.0.smithi088.stderr:+++ COBBLER_SERVER=172.21.0.11 2024-06-03T08:37:53.578 INFO:tasks.workunit.client.0.smithi088.stderr:++ for i in /etc/profile.d/*.sh 2024-06-03T08:37:53.578 INFO:tasks.workunit.client.0.smithi088.stderr:++ '[' -r /etc/profile.d/colorgrep.sh ']' 2024-06-03T08:37:53.578 INFO:tasks.workunit.client.0.smithi088.stderr:++ '[' '' ']' 2024-06-03T08:37:53.578 INFO:tasks.workunit.client.0.smithi088.stderr:++ . /etc/profile.d/colorgrep.sh 2024-06-03T08:37:53.578 INFO:tasks.workunit.client.0.smithi088.stderr:+++ /usr/libexec/grepconf.sh -c 2024-06-03T08:37:53.578 INFO:tasks.workunit.client.0.smithi088.stderr:+ sed 's/^/ api_tier_pp: /' 2024-06-03T08:37:53.578 INFO:tasks.workunit.client.0.smithi088.stderr:++ printf %25s api_watch_notify 2024-06-03T08:37:53.579 INFO:tasks.workunit.client.0.smithi088.stderr:+ r=' api_watch_notify' 2024-06-03T08:37:53.579 INFO:tasks.workunit.client.0.smithi088.stderr:++ awk '{print $1}' 2024-06-03T08:37:53.579 INFO:tasks.workunit.client.0.smithi088.stderr:++ echo api_watch_notify 2024-06-03T08:37:53.579 INFO:tasks.workunit.client.0.smithi088.stderr:+ ff=api_watch_notify 2024-06-03T08:37:53.579 INFO:tasks.workunit.client.0.smithi088.stderr:++++ /usr/bin/sed -r -e 's/^[[:blank:]]*([[:upper:]_]+)=([[:print:][:digit:]\._-]+|"[[:print:][:digit:]\._-]+")/export \1=\2/;t;d' /etc/locale.conf 2024-06-03T08:37:53.579 INFO:tasks.workunit.client.0.smithi088.stderr:+ pid=40590 2024-06-03T08:37:53.579 INFO:tasks.workunit.client.0.smithi088.stderr:+ echo 'test api_watch_notify on pid 40590' 2024-06-03T08:37:53.579 INFO:tasks.workunit.client.0.smithi088.stdout:test api_watch_notify on pid 40590 2024-06-03T08:37:53.585 INFO:tasks.workunit.client.0.smithi088.stderr:+ pids[$f]=40590 2024-06-03T08:37:53.585 INFO:tasks.workunit.client.0.smithi088.stderr:+ for f in api_aio api_aio_pp api_io api_io_pp api_asio api_list api_lock api_lock_pp api_misc api_misc_pp api_tier_pp api_pool api_snapshots api_snapshots_pp api_stat api_stat_pp api_watch_notify api_watch_notify_pp api_cmd api_cmd_pp api_service api_service_pp api_c_write_operations api_c_read_operations api_cls_remote_reads list_parallel open_pools_parallel delete_pools_parallel 2024-06-03T08:37:53.586 INFO:tasks.workunit.client.0.smithi088.stderr:+ '[' 1 -eq 1 ']' 2024-06-03T08:37:53.586 INFO:tasks.workunit.client.0.smithi088.stderr:+ bash -o pipefail -exc 'ceph_test_rados_api_watch_notify 2>&1 | tee ceph_test_rados_api_watch_notify.log | sed "s/^/ api_watch_notify: /"' 2024-06-03T08:37:53.586 INFO:tasks.workunit.client.0.smithi088.stderr:++ printf %25s api_watch_notify_pp 2024-06-03T08:37:53.586 INFO:tasks.workunit.client.0.smithi088.stderr:+ r=' api_watch_notify_pp' 2024-06-03T08:37:53.586 INFO:tasks.workunit.client.0.smithi088.stderr:+++ eval export 'LANG="en_US.UTF-8"' 2024-06-03T08:37:53.586 INFO:tasks.workunit.client.0.smithi088.stderr:++++ export LANG=en_US.UTF-8 2024-06-03T08:37:53.586 INFO:tasks.workunit.client.0.smithi088.stderr:++++ LANG=en_US.UTF-8 2024-06-03T08:37:53.586 INFO:tasks.workunit.client.0.smithi088.stderr:+++ for config in /etc/locale.conf "${HOME}/.i18n" 2024-06-03T08:37:53.586 INFO:tasks.workunit.client.0.smithi088.stderr:+++ '[' -f /home/ubuntu/.i18n ']' 2024-06-03T08:37:53.586 INFO:tasks.workunit.client.0.smithi088.stderr:+++ '[' -n en_US.UTF-8 ']' 2024-06-03T08:37:53.586 INFO:tasks.workunit.client.0.smithi088.stderr:+++ LANG=en_US.UTF-8 2024-06-03T08:37:53.586 INFO:tasks.workunit.client.0.smithi088.stderr:+++ unset LANG_backup config 2024-06-03T08:37:53.587 INFO:tasks.workunit.client.0.smithi088.stderr:+++ '[' -n '' ']' 2024-06-03T08:37:53.587 INFO:tasks.workunit.client.0.smithi088.stderr:++ awk '{print $1}' 2024-06-03T08:37:53.587 INFO:tasks.workunit.client.0.smithi088.stderr:+++ '[' -n en_US.UTF-8 ']' 2024-06-03T08:37:53.587 INFO:tasks.workunit.client.0.smithi088.stderr:+++ '[' dumb = linux ']' 2024-06-03T08:37:53.587 INFO:tasks.workunit.client.0.smithi088.stderr:++ for i in /etc/profile.d/*.sh 2024-06-03T08:37:53.587 INFO:tasks.workunit.client.0.smithi088.stderr:++ '[' -r /etc/profile.d/less.sh ']' 2024-06-03T08:37:53.587 INFO:tasks.workunit.client.0.smithi088.stderr:++ '[' '' ']' 2024-06-03T08:37:53.587 INFO:tasks.workunit.client.0.smithi088.stderr:++ . /etc/profile.d/less.sh 2024-06-03T08:37:53.587 INFO:tasks.workunit.client.0.smithi088.stderr:+++ '[' -z '||/usr/bin/lesspipe.sh %s' ']' 2024-06-03T08:37:53.587 INFO:tasks.workunit.client.0.smithi088.stderr:++ for i in /etc/profile.d/*.sh 2024-06-03T08:37:53.587 INFO:tasks.workunit.client.0.smithi088.stderr:++ '[' -r /etc/profile.d/which2.sh ']' 2024-06-03T08:37:53.587 INFO:tasks.workunit.client.0.smithi088.stderr:++ '[' '' ']' 2024-06-03T08:37:53.587 INFO:tasks.workunit.client.0.smithi088.stderr:++ . /etc/profile.d/which2.sh 2024-06-03T08:37:53.588 INFO:tasks.workunit.client.0.smithi088.stderr:++ echo api_watch_notify_pp 2024-06-03T08:37:53.588 INFO:tasks.workunit.client.0.smithi088.stderr:+++ case "$(basename $(readlink /proc/$$/exe))" in 2024-06-03T08:37:53.588 INFO:tasks.workunit.client.0.smithi088.stderr:+++ alias 'grep=grep --color=auto' 2024-06-03T08:37:53.588 INFO:tasks.workunit.client.0.smithi088.stderr:+++ alias 'egrep=egrep --color=auto' 2024-06-03T08:37:53.588 INFO:tasks.workunit.client.0.smithi088.stderr:+++ alias 'fgrep=fgrep --color=auto' 2024-06-03T08:37:53.588 INFO:tasks.workunit.client.0.smithi088.stderr:++ for i in /etc/profile.d/*.sh 2024-06-03T08:37:53.588 INFO:tasks.workunit.client.0.smithi088.stderr:++ '[' -r /etc/profile.d/colorls.sh ']' 2024-06-03T08:37:53.588 INFO:tasks.workunit.client.0.smithi088.stderr:++ '[' '' ']' 2024-06-03T08:37:53.588 INFO:tasks.workunit.client.0.smithi088.stderr:+ '[' -f /etc/bashrc ']' 2024-06-03T08:37:53.588 INFO:tasks.workunit.client.0.smithi088.stderr:+ . /etc/bashrc 2024-06-03T08:37:53.588 INFO:tasks.workunit.client.0.smithi088.stderr:++ . /etc/profile.d/colorls.sh 2024-06-03T08:37:53.588 INFO:tasks.workunit.client.0.smithi088.stderr:+++ '[' '!' -t 0 ']' 2024-06-03T08:37:53.588 INFO:tasks.workunit.client.0.smithi088.stderr:++ '[' -z '' ']' 2024-06-03T08:37:53.589 INFO:tasks.workunit.client.0.smithi088.stderr:++ BASHRCSOURCED=Y 2024-06-03T08:37:53.589 INFO:tasks.workunit.client.0.smithi088.stderr:++ '[' '' ']' 2024-06-03T08:37:53.589 INFO:tasks.workunit.client.0.smithi088.stderr:++ shopt -q login_shell 2024-06-03T08:37:53.589 INFO:tasks.workunit.client.0.smithi088.stderr:+++ return 2024-06-03T08:37:53.589 INFO:tasks.workunit.client.0.smithi088.stderr:++ for i in /etc/profile.d/*.sh 2024-06-03T08:37:53.589 INFO:tasks.workunit.client.0.smithi088.stderr:++ '[' -r /etc/profile.d/colorsysstat.sh ']' 2024-06-03T08:37:53.589 INFO:tasks.workunit.client.0.smithi088.stderr:++ '[' '' ']' 2024-06-03T08:37:53.589 INFO:tasks.workunit.client.0.smithi088.stderr:++ . /etc/profile.d/colorsysstat.sh 2024-06-03T08:37:53.589 INFO:tasks.workunit.client.0.smithi088.stderr:+++ export S_COLORS=auto 2024-06-03T08:37:53.589 INFO:tasks.workunit.client.0.smithi088.stderr:+++ S_COLORS=auto 2024-06-03T08:37:53.589 INFO:tasks.workunit.client.0.smithi088.stderr:++ for i in /etc/profile.d/*.sh 2024-06-03T08:37:53.589 INFO:tasks.workunit.client.0.smithi088.stderr:++ '[' -r /etc/profile.d/colorxzgrep.sh ']' 2024-06-03T08:37:53.589 INFO:tasks.workunit.client.0.smithi088.stderr:++ '[' '' ']' 2024-06-03T08:37:53.590 INFO:tasks.workunit.client.0.smithi088.stderr:++ . /etc/profile.d/colorxzgrep.sh 2024-06-03T08:37:53.590 INFO:tasks.workunit.client.0.smithi088.stderr:+++ /usr/libexec/grepconf.sh -c 2024-06-03T08:37:53.590 INFO:tasks.workunit.client.0.smithi088.stderr:+++++ readlink /proc/40461/exe 2024-06-03T08:37:53.590 INFO:tasks.workunit.client.0.smithi088.stderr:+++ umask 2024-06-03T08:37:53.590 INFO:tasks.workunit.client.0.smithi088.stderr:+++ alias 'grep=grep --color=auto' 2024-06-03T08:37:53.590 INFO:tasks.workunit.client.0.smithi088.stderr:++ '[' 0022 -eq 0 ']' 2024-06-03T08:37:53.590 INFO:tasks.workunit.client.0.smithi088.stderr:+++ alias 'egrep=egrep --color=auto' 2024-06-03T08:37:53.590 INFO:tasks.workunit.client.0.smithi088.stderr:+++ alias 'fgrep=fgrep --color=auto' 2024-06-03T08:37:53.590 INFO:tasks.workunit.client.0.smithi088.stderr:++ for i in /etc/profile.d/*.sh 2024-06-03T08:37:53.590 INFO:tasks.workunit.client.0.smithi088.stderr:++ SHELL=/bin/bash 2024-06-03T08:37:53.590 INFO:tasks.workunit.client.0.smithi088.stderr:++ '[' -r /etc/profile.d/colorls.sh ']' 2024-06-03T08:37:53.590 INFO:tasks.workunit.client.0.smithi088.stderr:++ '[' '' ']' 2024-06-03T08:37:53.590 INFO:tasks.workunit.client.0.smithi088.stderr:++ for i in /etc/profile.d/*.sh 2024-06-03T08:37:53.590 INFO:tasks.workunit.client.0.smithi088.stderr:++ '[' -r /etc/profile.d/bash_completion.sh ']' 2024-06-03T08:37:53.591 INFO:tasks.workunit.client.0.smithi088.stderr:++ '[' '' ']' 2024-06-03T08:37:53.591 INFO:tasks.workunit.client.0.smithi088.stderr:++ . /etc/profile.d/bash_completion.sh 2024-06-03T08:37:53.591 INFO:tasks.workunit.client.0.smithi088.stderr:++ . /etc/profile.d/colorls.sh 2024-06-03T08:37:53.591 INFO:tasks.workunit.client.0.smithi088.stderr:+++ '[' 'x5.1.8(1)-release' '!=' x -a x '!=' x -a x = x ']' 2024-06-03T08:37:53.591 INFO:tasks.workunit.client.0.smithi088.stderr:+++ '[' '!' -t 0 ']' 2024-06-03T08:37:53.591 INFO:tasks.workunit.client.0.smithi088.stderr:+++ return 2024-06-03T08:37:53.591 INFO:tasks.workunit.client.0.smithi088.stderr:++ for i in /etc/profile.d/*.sh 2024-06-03T08:37:53.591 INFO:tasks.workunit.client.0.smithi088.stderr:++ '[' -r /etc/profile.d/colorsysstat.sh ']' 2024-06-03T08:37:53.591 INFO:tasks.workunit.client.0.smithi088.stderr:++ '[' '' ']' 2024-06-03T08:37:53.591 INFO:tasks.workunit.client.0.smithi088.stderr:++ . /etc/profile.d/colorsysstat.sh 2024-06-03T08:37:53.591 INFO:tasks.workunit.client.0.smithi088.stderr:+++ export S_COLORS=auto 2024-06-03T08:37:53.591 INFO:tasks.workunit.client.0.smithi088.stderr:++ for i in /etc/profile.d/*.sh 2024-06-03T08:37:53.591 INFO:tasks.workunit.client.0.smithi088.stderr:++ '[' -r /etc/profile.d/cobbler.sh ']' 2024-06-03T08:37:53.592 INFO:tasks.workunit.client.0.smithi088.stderr:++ '[' '' ']' 2024-06-03T08:37:53.592 INFO:tasks.workunit.client.0.smithi088.stderr:++ . /etc/profile.d/cobbler.sh 2024-06-03T08:37:53.592 INFO:tasks.workunit.client.0.smithi088.stderr:+++ export COBBLER_SERVER=172.21.0.11 2024-06-03T08:37:53.592 INFO:tasks.workunit.client.0.smithi088.stderr:+++ S_COLORS=auto 2024-06-03T08:37:53.592 INFO:tasks.workunit.client.0.smithi088.stderr:++ for i in /etc/profile.d/*.sh 2024-06-03T08:37:53.592 INFO:tasks.workunit.client.0.smithi088.stderr:++ '[' -r /etc/profile.d/colorxzgrep.sh ']' 2024-06-03T08:37:53.592 INFO:tasks.workunit.client.0.smithi088.stderr:++ '[' '' ']' 2024-06-03T08:37:53.592 INFO:tasks.workunit.client.0.smithi088.stderr:++ . /etc/profile.d/colorxzgrep.sh 2024-06-03T08:37:53.592 INFO:tasks.workunit.client.0.smithi088.stderr:+++ /usr/libexec/grepconf.sh -c 2024-06-03T08:37:53.592 INFO:tasks.workunit.client.0.smithi088.stderr:+++ COBBLER_SERVER=172.21.0.11 2024-06-03T08:37:53.592 INFO:tasks.workunit.client.0.smithi088.stderr:++ for i in /etc/profile.d/*.sh 2024-06-03T08:37:53.592 INFO:tasks.workunit.client.0.smithi088.stderr:++ '[' -r /etc/profile.d/colorgrep.sh ']' 2024-06-03T08:37:53.592 INFO:tasks.workunit.client.0.smithi088.stderr:++ '[' '' ']' 2024-06-03T08:37:53.592 INFO:tasks.workunit.client.0.smithi088.stderr:++ . /etc/profile.d/colorgrep.sh 2024-06-03T08:37:53.593 INFO:tasks.workunit.client.0.smithi088.stderr:+++ /usr/libexec/grepconf.sh -c 2024-06-03T08:37:53.593 INFO:tasks.workunit.client.0.smithi088.stderr:+ ff=api_watch_notify_pp 2024-06-03T08:37:53.593 INFO:tasks.workunit.client.0.smithi088.stderr:+ pid=40610 2024-06-03T08:37:53.593 INFO:tasks.workunit.client.0.smithi088.stderr:+ echo 'test api_watch_notify_pp on pid 40610' 2024-06-03T08:37:53.593 INFO:tasks.workunit.client.0.smithi088.stdout:test api_watch_notify_pp on pid 40610 2024-06-03T08:37:53.602 INFO:tasks.workunit.client.0.smithi088.stderr:+ pids[$f]=40610 2024-06-03T08:37:53.602 INFO:tasks.workunit.client.0.smithi088.stderr:+ for f in api_aio api_aio_pp api_io api_io_pp api_asio api_list api_lock api_lock_pp api_misc api_misc_pp api_tier_pp api_pool api_snapshots api_snapshots_pp api_stat api_stat_pp api_watch_notify api_watch_notify_pp api_cmd api_cmd_pp api_service api_service_pp api_c_write_operations api_c_read_operations api_cls_remote_reads list_parallel open_pools_parallel delete_pools_parallel 2024-06-03T08:37:53.602 INFO:tasks.workunit.client.0.smithi088.stderr:+ '[' 1 -eq 1 ']' 2024-06-03T08:37:53.602 INFO:tasks.workunit.client.0.smithi088.stderr:++++ basename /usr/bin/bash 2024-06-03T08:37:53.602 INFO:tasks.workunit.client.0.smithi088.stderr:++ printf %25s api_cmd 2024-06-03T08:37:53.602 INFO:tasks.workunit.client.0.smithi088.stderr:+ r=' api_cmd' 2024-06-03T08:37:53.602 INFO:tasks.workunit.client.0.smithi088.stderr:+++ which_declare='declare -f' 2024-06-03T08:37:53.602 INFO:tasks.workunit.client.0.smithi088.stderr:+++ which_opt=-f 2024-06-03T08:37:53.602 INFO:tasks.workunit.client.0.smithi088.stderr:+++ export which_declare 2024-06-03T08:37:53.602 INFO:tasks.workunit.client.0.smithi088.stderr:++ echo api_cmd 2024-06-03T08:37:53.602 INFO:tasks.workunit.client.0.smithi088.stderr:+++ export -f which 2024-06-03T08:37:53.603 INFO:tasks.workunit.client.0.smithi088.stderr:++ unset i 2024-06-03T08:37:53.603 INFO:tasks.workunit.client.0.smithi088.stderr:++ unset -f pathmunge 2024-06-03T08:37:53.603 INFO:tasks.workunit.client.0.smithi088.stderr:+ [[ /home/ubuntu/.local/bin:/home/ubuntu/bin:/usr/local/bin:/usr/bin:/usr/local/sbin:/usr/sbin:/usr/sbin =~ /home/ubuntu/\.local/bin:/home/ubuntu/bin: ]] 2024-06-03T08:37:53.603 INFO:tasks.workunit.client.0.smithi088.stderr:+ bash -o pipefail -exc 'ceph_test_rados_api_watch_notify_pp 2>&1 | tee ceph_test_rados_api_watch_notify_pp.log | sed "s/^/ api_watch_notify_pp: /"' 2024-06-03T08:37:53.603 INFO:tasks.workunit.client.0.smithi088.stderr:+ export PATH 2024-06-03T08:37:53.603 INFO:tasks.workunit.client.0.smithi088.stderr:+ '[' -d /home/ubuntu/.bashrc.d ']' 2024-06-03T08:37:53.603 INFO:tasks.workunit.client.0.smithi088.stderr:+ unset rc 2024-06-03T08:37:53.603 INFO:tasks.workunit.client.0.smithi088.stderr:+++ alias 'xzgrep=xzgrep --color=auto' 2024-06-03T08:37:53.603 INFO:tasks.workunit.client.0.smithi088.stderr:+ ceph_test_rados_api_snapshots_pp 2024-06-03T08:37:53.603 INFO:tasks.workunit.client.0.smithi088.stderr:+++ alias 'xzegrep=xzegrep --color=auto' 2024-06-03T08:37:53.603 INFO:tasks.workunit.client.0.smithi088.stderr:+++ alias 'xzfgrep=xzfgrep --color=auto' 2024-06-03T08:37:53.603 INFO:tasks.workunit.client.0.smithi088.stderr:++ for i in /etc/profile.d/*.sh 2024-06-03T08:37:53.603 INFO:tasks.workunit.client.0.smithi088.stderr:++ '[' -r /etc/profile.d/colorzgrep.sh ']' 2024-06-03T08:37:53.603 INFO:tasks.workunit.client.0.smithi088.stderr:++ '[' '' ']' 2024-06-03T08:37:53.604 INFO:tasks.workunit.client.0.smithi088.stderr:++ . /etc/profile.d/colorzgrep.sh 2024-06-03T08:37:53.604 INFO:tasks.workunit.client.0.smithi088.stderr:+++ '[' -f /usr/libexec/grepconf.sh ']' 2024-06-03T08:37:53.604 INFO:tasks.workunit.client.0.smithi088.stderr:+++ /usr/libexec/grepconf.sh -c 2024-06-03T08:37:53.604 INFO:tasks.workunit.client.0.smithi088.stderr:+ tee ceph_test_rados_api_snapshots_pp.log 2024-06-03T08:37:53.604 INFO:tasks.workunit.client.0.smithi088.stderr:+ '[' -f /etc/bashrc ']' 2024-06-03T08:37:53.604 INFO:tasks.workunit.client.0.smithi088.stderr:+ . /etc/bashrc 2024-06-03T08:37:53.604 INFO:tasks.workunit.client.0.smithi088.stderr:++ '[' -z '' ']' 2024-06-03T08:37:53.604 INFO:tasks.workunit.client.0.smithi088.stderr:++ BASHRCSOURCED=Y 2024-06-03T08:37:53.604 INFO:tasks.workunit.client.0.smithi088.stderr:++ '[' '' ']' 2024-06-03T08:37:53.604 INFO:tasks.workunit.client.0.smithi088.stderr:++ shopt -q login_shell 2024-06-03T08:37:53.604 INFO:tasks.workunit.client.0.smithi088.stderr:+++ umask 2024-06-03T08:37:53.604 INFO:tasks.workunit.client.0.smithi088.stderr:++ '[' 0022 -eq 0 ']' 2024-06-03T08:37:53.604 INFO:tasks.workunit.client.0.smithi088.stderr:++ SHELL=/bin/bash 2024-06-03T08:37:53.604 INFO:tasks.workunit.client.0.smithi088.stderr:++ for i in /etc/profile.d/*.sh 2024-06-03T08:37:53.605 INFO:tasks.workunit.client.0.smithi088.stderr:+ sed 's/^/ api_snapshots_pp: /' 2024-06-03T08:37:53.605 INFO:tasks.workunit.client.0.smithi088.stderr:++ '[' -r /etc/profile.d/bash_completion.sh ']' 2024-06-03T08:37:53.605 INFO:tasks.workunit.client.0.smithi088.stderr:++ '[' '' ']' 2024-06-03T08:37:53.605 INFO:tasks.workunit.client.0.smithi088.stderr:++ . /etc/profile.d/bash_completion.sh 2024-06-03T08:37:53.605 INFO:tasks.workunit.client.0.smithi088.stderr:+++ '[' 'x5.1.8(1)-release' '!=' x -a x '!=' x -a x = x ']' 2024-06-03T08:37:53.605 INFO:tasks.workunit.client.0.smithi088.stderr:++ for i in /etc/profile.d/*.sh 2024-06-03T08:37:53.605 INFO:tasks.workunit.client.0.smithi088.stderr:++ '[' -r /etc/profile.d/cobbler.sh ']' 2024-06-03T08:37:53.605 INFO:tasks.workunit.client.0.smithi088.stderr:++ '[' '' ']' 2024-06-03T08:37:53.605 INFO:tasks.workunit.client.0.smithi088.stderr:++ . /etc/profile.d/cobbler.sh 2024-06-03T08:37:53.605 INFO:tasks.workunit.client.0.smithi088.stderr:+++ export COBBLER_SERVER=172.21.0.11 2024-06-03T08:37:53.605 INFO:tasks.workunit.client.0.smithi088.stderr:+++ COBBLER_SERVER=172.21.0.11 2024-06-03T08:37:53.605 INFO:tasks.workunit.client.0.smithi088.stderr:++ for i in /etc/profile.d/*.sh 2024-06-03T08:37:53.605 INFO:tasks.workunit.client.0.smithi088.stderr:++ '[' -r /etc/profile.d/colorgrep.sh ']' 2024-06-03T08:37:53.606 INFO:tasks.workunit.client.0.smithi088.stderr:++ '[' '' ']' 2024-06-03T08:37:53.606 INFO:tasks.workunit.client.0.smithi088.stderr:++ . /etc/profile.d/colorgrep.sh 2024-06-03T08:37:53.606 INFO:tasks.workunit.client.0.smithi088.stderr:+++ /usr/libexec/grepconf.sh -c 2024-06-03T08:37:53.606 INFO:tasks.workunit.client.0.smithi088.stderr:+++ alias 'zgrep=zgrep --color=auto' 2024-06-03T08:37:53.606 INFO:tasks.workunit.client.0.smithi088.stderr:+++ alias 'zfgrep=zfgrep --color=auto' 2024-06-03T08:37:53.606 INFO:tasks.workunit.client.0.smithi088.stderr:+++ alias 'zegrep=zegrep --color=auto' 2024-06-03T08:37:53.606 INFO:tasks.workunit.client.0.smithi088.stderr:++ for i in /etc/profile.d/*.sh 2024-06-03T08:37:53.606 INFO:tasks.workunit.client.0.smithi088.stderr:++ '[' -r /etc/profile.d/debuginfod.sh ']' 2024-06-03T08:37:53.606 INFO:tasks.workunit.client.0.smithi088.stderr:++ '[' '' ']' 2024-06-03T08:37:53.606 INFO:tasks.workunit.client.0.smithi088.stderr:++ . /etc/profile.d/debuginfod.sh 2024-06-03T08:37:53.606 INFO:tasks.workunit.client.0.smithi088.stderr:+++ '[' -z 'https://debuginfod.centos.org/ ' ']' 2024-06-03T08:37:53.606 INFO:tasks.workunit.client.0.smithi088.stderr:++ for i in /etc/profile.d/*.sh 2024-06-03T08:37:53.606 INFO:tasks.workunit.client.0.smithi088.stderr:++ '[' -r /etc/profile.d/gawk.sh ']' 2024-06-03T08:37:53.607 INFO:tasks.workunit.client.0.smithi088.stderr:++ '[' '' ']' 2024-06-03T08:37:53.607 INFO:tasks.workunit.client.0.smithi088.stderr:++ . /etc/profile.d/gawk.sh 2024-06-03T08:37:53.607 INFO:tasks.workunit.client.0.smithi088.stderr:++ for i in /etc/profile.d/*.sh 2024-06-03T08:37:53.607 INFO:tasks.workunit.client.0.smithi088.stderr:++ '[' -r /etc/profile.d/lang.sh ']' 2024-06-03T08:37:53.607 INFO:tasks.workunit.client.0.smithi088.stderr:++ '[' '' ']' 2024-06-03T08:37:53.607 INFO:tasks.workunit.client.0.smithi088.stderr:++ . /etc/profile.d/lang.sh 2024-06-03T08:37:53.607 INFO:tasks.workunit.client.0.smithi088.stderr:+++ unset LANG_backup 2024-06-03T08:37:53.607 INFO:tasks.workunit.client.0.smithi088.stderr:++ awk '{print $1}' 2024-06-03T08:37:53.607 INFO:tasks.workunit.client.0.smithi088.stderr:++++ /usr/bin/locale 2024-06-03T08:37:53.607 INFO:tasks.workunit.client.0.smithi088.stderr:+++ alias 'grep=grep --color=auto' 2024-06-03T08:37:53.607 INFO:tasks.workunit.client.0.smithi088.stderr:+++ alias 'egrep=egrep --color=auto' 2024-06-03T08:37:53.607 INFO:tasks.workunit.client.0.smithi088.stderr:+++ alias 'fgrep=fgrep --color=auto' 2024-06-03T08:37:53.607 INFO:tasks.workunit.client.0.smithi088.stderr:++ for i in /etc/profile.d/*.sh 2024-06-03T08:37:53.607 INFO:tasks.workunit.client.0.smithi088.stderr:++ '[' -r /etc/profile.d/colorls.sh ']' 2024-06-03T08:37:53.608 INFO:tasks.workunit.client.0.smithi088.stderr:++ '[' '' ']' 2024-06-03T08:37:53.608 INFO:tasks.workunit.client.0.smithi088.stderr:++ . /etc/profile.d/colorls.sh 2024-06-03T08:37:53.608 INFO:tasks.workunit.client.0.smithi088.stderr:+++ '[' '!' -t 0 ']' 2024-06-03T08:37:53.608 INFO:tasks.workunit.client.0.smithi088.stderr:+++ return 2024-06-03T08:37:53.608 INFO:tasks.workunit.client.0.smithi088.stderr:++ for i in /etc/profile.d/*.sh 2024-06-03T08:37:53.608 INFO:tasks.workunit.client.0.smithi088.stderr:++ '[' -r /etc/profile.d/colorsysstat.sh ']' 2024-06-03T08:37:53.608 INFO:tasks.workunit.client.0.smithi088.stderr:++ '[' '' ']' 2024-06-03T08:37:53.608 INFO:tasks.workunit.client.0.smithi088.stderr:++ . /etc/profile.d/colorsysstat.sh 2024-06-03T08:37:53.608 INFO:tasks.workunit.client.0.smithi088.stderr:+++ export S_COLORS=auto 2024-06-03T08:37:53.608 INFO:tasks.workunit.client.0.smithi088.stderr:+++ S_COLORS=auto 2024-06-03T08:37:53.608 INFO:tasks.workunit.client.0.smithi088.stderr:++ for i in /etc/profile.d/*.sh 2024-06-03T08:37:53.608 INFO:tasks.workunit.client.0.smithi088.stderr:++ '[' -r /etc/profile.d/colorxzgrep.sh ']' 2024-06-03T08:37:53.608 INFO:tasks.workunit.client.0.smithi088.stderr:++ '[' '' ']' 2024-06-03T08:37:53.608 INFO:tasks.workunit.client.0.smithi088.stderr:++ . /etc/profile.d/colorxzgrep.sh 2024-06-03T08:37:53.609 INFO:tasks.workunit.client.0.smithi088.stderr:+++ /usr/libexec/grepconf.sh -c 2024-06-03T08:37:53.609 INFO:tasks.workunit.client.0.smithi088.stderr:+++ '[' -n '' ']' 2024-06-03T08:37:53.609 INFO:tasks.workunit.client.0.smithi088.stderr:+++ LANG_backup=en_US.UTF-8 2024-06-03T08:37:53.609 INFO:tasks.workunit.client.0.smithi088.stderr:+++ for config in /etc/locale.conf "${HOME}/.i18n" 2024-06-03T08:37:53.609 INFO:tasks.workunit.client.0.smithi088.stderr:+++ '[' -f /etc/locale.conf ']' 2024-06-03T08:37:53.609 INFO:tasks.workunit.client.0.smithi088.stderr:+++ '[' -x /usr/bin/sed ']' 2024-06-03T08:37:53.609 INFO:tasks.workunit.client.0.smithi088.stderr:+ ff=api_cmd 2024-06-03T08:37:53.609 INFO:tasks.workunit.client.0.smithi088.stderr:+ pid=40634 2024-06-03T08:37:53.609 INFO:tasks.workunit.client.0.smithi088.stderr:+ echo 'test api_cmd on pid 40634' 2024-06-03T08:37:53.609 INFO:tasks.workunit.client.0.smithi088.stdout:test api_cmd on pid 40634 2024-06-03T08:37:53.613 INFO:tasks.workunit.client.0.smithi088.stderr:+ pids[$f]=40634 2024-06-03T08:37:53.613 INFO:tasks.workunit.client.0.smithi088.stderr:+ for f in api_aio api_aio_pp api_io api_io_pp api_asio api_list api_lock api_lock_pp api_misc api_misc_pp api_tier_pp api_pool api_snapshots api_snapshots_pp api_stat api_stat_pp api_watch_notify api_watch_notify_pp api_cmd api_cmd_pp api_service api_service_pp api_c_write_operations api_c_read_operations api_cls_remote_reads list_parallel open_pools_parallel delete_pools_parallel 2024-06-03T08:37:53.613 INFO:tasks.workunit.client.0.smithi088.stderr:+ '[' 1 -eq 1 ']' 2024-06-03T08:37:53.613 INFO:tasks.workunit.client.0.smithi088.stderr:+++ alias 'xzgrep=xzgrep --color=auto' 2024-06-03T08:37:53.613 INFO:tasks.workunit.client.0.smithi088.stderr:+ bash -o pipefail -exc 'ceph_test_rados_api_cmd 2>&1 | tee ceph_test_rados_api_cmd.log | sed "s/^/ api_cmd: /"' 2024-06-03T08:37:53.614 INFO:tasks.workunit.client.0.smithi088.stderr:+++ alias 'xzegrep=xzegrep --color=auto' 2024-06-03T08:37:53.614 INFO:tasks.workunit.client.0.smithi088.stderr:++++ /usr/bin/sed -r -e 's/^[[:blank:]]*([[:upper:]_]+)=([[:print:][:digit:]\._-]+|"[[:print:][:digit:]\._-]+")/export \1=\2/;t;d' /etc/locale.conf 2024-06-03T08:37:53.614 INFO:tasks.workunit.client.0.smithi088.stderr:+++ alias 'xzfgrep=xzfgrep --color=auto' 2024-06-03T08:37:53.614 INFO:tasks.workunit.client.0.smithi088.stderr:++ for i in /etc/profile.d/*.sh 2024-06-03T08:37:53.614 INFO:tasks.workunit.client.0.smithi088.stderr:++ '[' -r /etc/profile.d/colorzgrep.sh ']' 2024-06-03T08:37:53.614 INFO:tasks.workunit.client.0.smithi088.stderr:++ '[' '' ']' 2024-06-03T08:37:53.614 INFO:tasks.workunit.client.0.smithi088.stderr:++ . /etc/profile.d/colorzgrep.sh 2024-06-03T08:37:53.614 INFO:tasks.workunit.client.0.smithi088.stderr:+++ '[' -f /usr/libexec/grepconf.sh ']' 2024-06-03T08:37:53.614 INFO:tasks.workunit.client.0.smithi088.stderr:+++ /usr/libexec/grepconf.sh -c 2024-06-03T08:37:53.614 INFO:tasks.workunit.client.0.smithi088.stderr:++ printf %25s api_cmd_pp 2024-06-03T08:37:53.614 INFO:tasks.workunit.client.0.smithi088.stderr:+ r=' api_cmd_pp' 2024-06-03T08:37:53.614 INFO:tasks.workunit.client.0.smithi088.stderr:+++ eval export 'LANG="en_US.UTF-8"' 2024-06-03T08:37:53.615 INFO:tasks.workunit.client.0.smithi088.stderr:++ awk '{print $1}' 2024-06-03T08:37:53.615 INFO:tasks.workunit.client.0.smithi088.stderr:++++ export LANG=en_US.UTF-8 2024-06-03T08:37:53.615 INFO:tasks.workunit.client.0.smithi088.stderr:++++ LANG=en_US.UTF-8 2024-06-03T08:37:53.615 INFO:tasks.workunit.client.0.smithi088.stderr:+++ for config in /etc/locale.conf "${HOME}/.i18n" 2024-06-03T08:37:53.615 INFO:tasks.workunit.client.0.smithi088.stderr:+++ '[' -f /home/ubuntu/.i18n ']' 2024-06-03T08:37:53.615 INFO:tasks.workunit.client.0.smithi088.stderr:+++ '[' -n en_US.UTF-8 ']' 2024-06-03T08:37:53.615 INFO:tasks.workunit.client.0.smithi088.stderr:+++ LANG=en_US.UTF-8 2024-06-03T08:37:53.615 INFO:tasks.workunit.client.0.smithi088.stderr:+++ unset LANG_backup config 2024-06-03T08:37:53.615 INFO:tasks.workunit.client.0.smithi088.stderr:+++ '[' -n '' ']' 2024-06-03T08:37:53.615 INFO:tasks.workunit.client.0.smithi088.stderr:++ echo api_cmd_pp 2024-06-03T08:37:53.615 INFO:tasks.workunit.client.0.smithi088.stderr:+++ '[' -n en_US.UTF-8 ']' 2024-06-03T08:37:53.615 INFO:tasks.workunit.client.0.smithi088.stderr:+++ '[' dumb = linux ']' 2024-06-03T08:37:53.615 INFO:tasks.workunit.client.0.smithi088.stderr:++ for i in /etc/profile.d/*.sh 2024-06-03T08:37:53.615 INFO:tasks.workunit.client.0.smithi088.stderr:++ '[' -r /etc/profile.d/less.sh ']' 2024-06-03T08:37:53.616 INFO:tasks.workunit.client.0.smithi088.stderr:++ '[' '' ']' 2024-06-03T08:37:53.616 INFO:tasks.workunit.client.0.smithi088.stderr:++ . /etc/profile.d/less.sh 2024-06-03T08:37:53.616 INFO:tasks.workunit.client.0.smithi088.stderr:+++ '[' -z '||/usr/bin/lesspipe.sh %s' ']' 2024-06-03T08:37:53.616 INFO:tasks.workunit.client.0.smithi088.stderr:++ for i in /etc/profile.d/*.sh 2024-06-03T08:37:53.616 INFO:tasks.workunit.client.0.smithi088.stderr:++ '[' -r /etc/profile.d/which2.sh ']' 2024-06-03T08:37:53.616 INFO:tasks.workunit.client.0.smithi088.stderr:++ '[' '' ']' 2024-06-03T08:37:53.616 INFO:tasks.workunit.client.0.smithi088.stderr:++ . /etc/profile.d/which2.sh 2024-06-03T08:37:53.616 INFO:tasks.workunit.client.0.smithi088.stderr:+++ case "$(basename $(readlink /proc/$$/exe))" in 2024-06-03T08:37:53.616 INFO:tasks.workunit.client.0.smithi088.stderr:+ ff=api_cmd_pp 2024-06-03T08:37:53.616 INFO:tasks.workunit.client.0.smithi088.stderr:+ pid=40648 2024-06-03T08:37:53.616 INFO:tasks.workunit.client.0.smithi088.stdout:test api_cmd_pp on pid 40648 2024-06-03T08:37:53.624 INFO:tasks.workunit.client.0.smithi088.stderr:+ echo 'test api_cmd_pp on pid 40648' 2024-06-03T08:37:53.624 INFO:tasks.workunit.client.0.smithi088.stderr:+ pids[$f]=40648 2024-06-03T08:37:53.624 INFO:tasks.workunit.client.0.smithi088.stderr:+ for f in api_aio api_aio_pp api_io api_io_pp api_asio api_list api_lock api_lock_pp api_misc api_misc_pp api_tier_pp api_pool api_snapshots api_snapshots_pp api_stat api_stat_pp api_watch_notify api_watch_notify_pp api_cmd api_cmd_pp api_service api_service_pp api_c_write_operations api_c_read_operations api_cls_remote_reads list_parallel open_pools_parallel delete_pools_parallel 2024-06-03T08:37:53.624 INFO:tasks.workunit.client.0.smithi088.stderr:+ '[' 1 -eq 1 ']' 2024-06-03T08:37:53.624 INFO:tasks.workunit.client.0.smithi088.stderr:+ '[' -f /etc/bashrc ']' 2024-06-03T08:37:53.624 INFO:tasks.workunit.client.0.smithi088.stderr:+ . /etc/bashrc 2024-06-03T08:37:53.624 INFO:tasks.workunit.client.0.smithi088.stderr:++ '[' -z '' ']' 2024-06-03T08:37:53.624 INFO:tasks.workunit.client.0.smithi088.stderr:++ BASHRCSOURCED=Y 2024-06-03T08:37:53.625 INFO:tasks.workunit.client.0.smithi088.stderr:++ '[' '' ']' 2024-06-03T08:37:53.625 INFO:tasks.workunit.client.0.smithi088.stderr:++ shopt -q login_shell 2024-06-03T08:37:53.625 INFO:tasks.workunit.client.0.smithi088.stderr:++ printf %25s api_service 2024-06-03T08:37:53.625 INFO:tasks.workunit.client.0.smithi088.stderr:+ r=' api_service' 2024-06-03T08:37:53.625 INFO:tasks.workunit.client.0.smithi088.stderr:+++ alias 'grep=grep --color=auto' 2024-06-03T08:37:53.625 INFO:tasks.workunit.client.0.smithi088.stderr:+++ alias 'egrep=egrep --color=auto' 2024-06-03T08:37:53.625 INFO:tasks.workunit.client.0.smithi088.stderr:+++ alias 'fgrep=fgrep --color=auto' 2024-06-03T08:37:53.625 INFO:tasks.workunit.client.0.smithi088.stderr:++ for i in /etc/profile.d/*.sh 2024-06-03T08:37:53.625 INFO:tasks.workunit.client.0.smithi088.stderr:++ '[' -r /etc/profile.d/colorls.sh ']' 2024-06-03T08:37:53.625 INFO:tasks.workunit.client.0.smithi088.stderr:++ '[' '' ']' 2024-06-03T08:37:53.625 INFO:tasks.workunit.client.0.smithi088.stderr:++ . /etc/profile.d/colorls.sh 2024-06-03T08:37:53.625 INFO:tasks.workunit.client.0.smithi088.stderr:+++ '[' '!' -t 0 ']' 2024-06-03T08:37:53.625 INFO:tasks.workunit.client.0.smithi088.stderr:+++ return 2024-06-03T08:37:53.626 INFO:tasks.workunit.client.0.smithi088.stderr:++ for i in /etc/profile.d/*.sh 2024-06-03T08:37:53.626 INFO:tasks.workunit.client.0.smithi088.stderr:++ '[' -r /etc/profile.d/colorsysstat.sh ']' 2024-06-03T08:37:53.626 INFO:tasks.workunit.client.0.smithi088.stderr:++ '[' '' ']' 2024-06-03T08:37:53.626 INFO:tasks.workunit.client.0.smithi088.stderr:++ . /etc/profile.d/colorsysstat.sh 2024-06-03T08:37:53.626 INFO:tasks.workunit.client.0.smithi088.stderr:+++ export S_COLORS=auto 2024-06-03T08:37:53.626 INFO:tasks.workunit.client.0.smithi088.stderr:+++ S_COLORS=auto 2024-06-03T08:37:53.626 INFO:tasks.workunit.client.0.smithi088.stderr:++ for i in /etc/profile.d/*.sh 2024-06-03T08:37:53.626 INFO:tasks.workunit.client.0.smithi088.stderr:++ '[' -r /etc/profile.d/colorxzgrep.sh ']' 2024-06-03T08:37:53.626 INFO:tasks.workunit.client.0.smithi088.stderr:++ '[' '' ']' 2024-06-03T08:37:53.626 INFO:tasks.workunit.client.0.smithi088.stderr:++ . /etc/profile.d/colorxzgrep.sh 2024-06-03T08:37:53.626 INFO:tasks.workunit.client.0.smithi088.stderr:+++ /usr/libexec/grepconf.sh -c 2024-06-03T08:37:53.626 INFO:tasks.workunit.client.0.smithi088.stderr:+++ umask 2024-06-03T08:37:53.626 INFO:tasks.workunit.client.0.smithi088.stderr:+++++ readlink /proc/40512/exe 2024-06-03T08:37:53.626 INFO:tasks.workunit.client.0.smithi088.stderr:++ echo api_service 2024-06-03T08:37:53.627 INFO:tasks.workunit.client.0.smithi088.stderr:+++ alias 'xzgrep=xzgrep --color=auto' 2024-06-03T08:37:53.627 INFO:tasks.workunit.client.0.smithi088.stderr:+++ alias 'xzegrep=xzegrep --color=auto' 2024-06-03T08:37:53.627 INFO:tasks.workunit.client.0.smithi088.stderr:+++ alias 'xzfgrep=xzfgrep --color=auto' 2024-06-03T08:37:53.627 INFO:tasks.workunit.client.0.smithi088.stderr:++ for i in /etc/profile.d/*.sh 2024-06-03T08:37:53.627 INFO:tasks.workunit.client.0.smithi088.stderr:++ '[' -r /etc/profile.d/colorzgrep.sh ']' 2024-06-03T08:37:53.627 INFO:tasks.workunit.client.0.smithi088.stderr:++ '[' '' ']' 2024-06-03T08:37:53.627 INFO:tasks.workunit.client.0.smithi088.stderr:++ . /etc/profile.d/colorzgrep.sh 2024-06-03T08:37:53.627 INFO:tasks.workunit.client.0.smithi088.stderr:+++ '[' -f /usr/libexec/grepconf.sh ']' 2024-06-03T08:37:53.627 INFO:tasks.workunit.client.0.smithi088.stderr:+++ /usr/libexec/grepconf.sh -c 2024-06-03T08:37:53.627 INFO:tasks.workunit.client.0.smithi088.stderr:++ '[' 0022 -eq 0 ']' 2024-06-03T08:37:53.627 INFO:tasks.workunit.client.0.smithi088.stderr:++ SHELL=/bin/bash 2024-06-03T08:37:53.627 INFO:tasks.workunit.client.0.smithi088.stderr:+ bash -o pipefail -exc 'ceph_test_rados_api_cmd_pp 2>&1 | tee ceph_test_rados_api_cmd_pp.log | sed "s/^/ api_cmd_pp: /"' 2024-06-03T08:37:53.627 INFO:tasks.workunit.client.0.smithi088.stderr:++ for i in /etc/profile.d/*.sh 2024-06-03T08:37:53.627 INFO:tasks.workunit.client.0.smithi088.stderr:++ '[' -r /etc/profile.d/bash_completion.sh ']' 2024-06-03T08:37:53.628 INFO:tasks.workunit.client.0.smithi088.stderr:++ '[' '' ']' 2024-06-03T08:37:53.628 INFO:tasks.workunit.client.0.smithi088.stderr:++ . /etc/profile.d/bash_completion.sh 2024-06-03T08:37:53.628 INFO:tasks.workunit.client.0.smithi088.stderr:+++ '[' 'x5.1.8(1)-release' '!=' x -a x '!=' x -a x = x ']' 2024-06-03T08:37:53.628 INFO:tasks.workunit.client.0.smithi088.stderr:++ for i in /etc/profile.d/*.sh 2024-06-03T08:37:53.628 INFO:tasks.workunit.client.0.smithi088.stderr:++ '[' -r /etc/profile.d/cobbler.sh ']' 2024-06-03T08:37:53.628 INFO:tasks.workunit.client.0.smithi088.stderr:++ '[' '' ']' 2024-06-03T08:37:53.628 INFO:tasks.workunit.client.0.smithi088.stderr:++ . /etc/profile.d/cobbler.sh 2024-06-03T08:37:53.628 INFO:tasks.workunit.client.0.smithi088.stderr:+++ export COBBLER_SERVER=172.21.0.11 2024-06-03T08:37:53.628 INFO:tasks.workunit.client.0.smithi088.stderr:+++ COBBLER_SERVER=172.21.0.11 2024-06-03T08:37:53.628 INFO:tasks.workunit.client.0.smithi088.stderr:++ for i in /etc/profile.d/*.sh 2024-06-03T08:37:53.628 INFO:tasks.workunit.client.0.smithi088.stderr:++ '[' -r /etc/profile.d/colorgrep.sh ']' 2024-06-03T08:37:53.628 INFO:tasks.workunit.client.0.smithi088.stderr:++ '[' '' ']' 2024-06-03T08:37:53.628 INFO:tasks.workunit.client.0.smithi088.stderr:++ . /etc/profile.d/colorgrep.sh 2024-06-03T08:37:53.628 INFO:tasks.workunit.client.0.smithi088.stderr:+++ /usr/libexec/grepconf.sh -c 2024-06-03T08:37:53.629 INFO:tasks.workunit.client.0.smithi088.stderr:+ '[' -f /etc/bashrc ']' 2024-06-03T08:37:53.629 INFO:tasks.workunit.client.0.smithi088.stderr:+ . /etc/bashrc 2024-06-03T08:37:53.629 INFO:tasks.workunit.client.0.smithi088.stderr:++ '[' -z '' ']' 2024-06-03T08:37:53.629 INFO:tasks.workunit.client.0.smithi088.stderr:++ BASHRCSOURCED=Y 2024-06-03T08:37:53.629 INFO:tasks.workunit.client.0.smithi088.stderr:++ '[' '' ']' 2024-06-03T08:37:53.629 INFO:tasks.workunit.client.0.smithi088.stderr:++ shopt -q login_shell 2024-06-03T08:37:53.629 INFO:tasks.workunit.client.0.smithi088.stderr:++++ basename /usr/bin/bash 2024-06-03T08:37:53.629 INFO:tasks.workunit.client.0.smithi088.stderr:+++ alias 'zgrep=zgrep --color=auto' 2024-06-03T08:37:53.629 INFO:tasks.workunit.client.0.smithi088.stderr:+++ alias 'zfgrep=zfgrep --color=auto' 2024-06-03T08:37:53.629 INFO:tasks.workunit.client.0.smithi088.stderr:+++ alias 'zegrep=zegrep --color=auto' 2024-06-03T08:37:53.629 INFO:tasks.workunit.client.0.smithi088.stderr:++ for i in /etc/profile.d/*.sh 2024-06-03T08:37:53.629 INFO:tasks.workunit.client.0.smithi088.stderr:++ '[' -r /etc/profile.d/debuginfod.sh ']' 2024-06-03T08:37:53.629 INFO:tasks.workunit.client.0.smithi088.stderr:++ '[' '' ']' 2024-06-03T08:37:53.630 INFO:tasks.workunit.client.0.smithi088.stderr:++ . /etc/profile.d/debuginfod.sh 2024-06-03T08:37:53.630 INFO:tasks.workunit.client.0.smithi088.stderr:+++ '[' -z 'https://debuginfod.centos.org/ ' ']' 2024-06-03T08:37:53.630 INFO:tasks.workunit.client.0.smithi088.stderr:++ for i in /etc/profile.d/*.sh 2024-06-03T08:37:53.630 INFO:tasks.workunit.client.0.smithi088.stderr:+++ which_declare='declare -f' 2024-06-03T08:37:53.630 INFO:tasks.workunit.client.0.smithi088.stderr:++ '[' -r /etc/profile.d/gawk.sh ']' 2024-06-03T08:37:53.630 INFO:tasks.workunit.client.0.smithi088.stderr:++ '[' '' ']' 2024-06-03T08:37:53.630 INFO:tasks.workunit.client.0.smithi088.stderr:++ . /etc/profile.d/gawk.sh 2024-06-03T08:37:53.630 INFO:tasks.workunit.client.0.smithi088.stderr:+++ which_opt=-f 2024-06-03T08:37:53.630 INFO:tasks.workunit.client.0.smithi088.stderr:+++ export which_declare 2024-06-03T08:37:53.630 INFO:tasks.workunit.client.0.smithi088.stderr:++ for i in /etc/profile.d/*.sh 2024-06-03T08:37:53.630 INFO:tasks.workunit.client.0.smithi088.stderr:++ '[' -r /etc/profile.d/lang.sh ']' 2024-06-03T08:37:53.630 INFO:tasks.workunit.client.0.smithi088.stderr:++ '[' '' ']' 2024-06-03T08:37:53.631 INFO:tasks.workunit.client.0.smithi088.stderr:++ . /etc/profile.d/lang.sh 2024-06-03T08:37:53.631 INFO:tasks.workunit.client.0.smithi088.stderr:+++ unset LANG_backup 2024-06-03T08:37:53.631 INFO:tasks.workunit.client.0.smithi088.stderr:+++ export -f which 2024-06-03T08:37:53.631 INFO:tasks.workunit.client.0.smithi088.stderr:++ unset i 2024-06-03T08:37:53.631 INFO:tasks.workunit.client.0.smithi088.stderr:++ unset -f pathmunge 2024-06-03T08:37:53.631 INFO:tasks.workunit.client.0.smithi088.stderr:+ [[ /home/ubuntu/.local/bin:/home/ubuntu/bin:/usr/local/bin:/usr/bin:/usr/local/sbin:/usr/sbin:/usr/sbin =~ /home/ubuntu/\.local/bin:/home/ubuntu/bin: ]] 2024-06-03T08:37:53.631 INFO:tasks.workunit.client.0.smithi088.stderr:+ export PATH 2024-06-03T08:37:53.631 INFO:tasks.workunit.client.0.smithi088.stderr:+ '[' -d /home/ubuntu/.bashrc.d ']' 2024-06-03T08:37:53.631 INFO:tasks.workunit.client.0.smithi088.stderr:+ unset rc 2024-06-03T08:37:53.631 INFO:tasks.workunit.client.0.smithi088.stderr:+ ceph_test_rados_api_stat 2024-06-03T08:37:53.631 INFO:tasks.workunit.client.0.smithi088.stderr:+++ umask 2024-06-03T08:37:53.631 INFO:tasks.workunit.client.0.smithi088.stderr:+ tee ceph_test_rados_api_stat.log 2024-06-03T08:37:53.632 INFO:tasks.workunit.client.0.smithi088.stdout: api_asio: [==========] Running 12 tests from 1 test suite. 2024-06-03T08:37:53.632 INFO:tasks.workunit.client.0.smithi088.stdout: api_asio: [----------] Global test environment set-up. 2024-06-03T08:37:53.632 INFO:tasks.workunit.client.0.smithi088.stdout: api_asio: [----------] 12 tests from AsioRados 2024-06-03T08:37:53.632 INFO:tasks.workunit.client.0.smithi088.stdout: api_asio: 2024-06-03T08:37:53.451+0000 7f514d647780 10 monclient: get_monmap_and_config 2024-06-03T08:37:53.632 INFO:tasks.workunit.client.0.smithi088.stdout: api_asio: 2024-06-03T08:37:53.451+0000 7f514d647780 10 monclient: build_initial_monmap 2024-06-03T08:37:53.632 INFO:tasks.workunit.client.0.smithi088.stdout: api_asio: 2024-06-03T08:37:53.451+0000 7f514d647780 10 monclient: monmap: 2024-06-03T08:37:53.632 INFO:tasks.workunit.client.0.smithi088.stdout: api_asio: epoch 0 2024-06-03T08:37:53.632 INFO:tasks.workunit.client.0.smithi088.stdout: api_asio: fsid 00000000-0000-0000-0000-000000000000 2024-06-03T08:37:53.632 INFO:tasks.workunit.client.0.smithi088.stdout: api_asio: last_changed 0.000000 2024-06-03T08:37:53.632 INFO:tasks.workunit.client.0.smithi088.stdout: api_asio: created 0.000000 2024-06-03T08:37:53.632 INFO:tasks.workunit.client.0.smithi088.stdout: api_asio: min_mon_release 0 (unknown) 2024-06-03T08:37:53.632 INFO:tasks.workunit.client.0.smithi088.stdout: api_asio: election_strategy: 1 2024-06-03T08:37:53.633 INFO:tasks.workunit.client.0.smithi088.stdout: api_asio: 0: v1:172.21.15.88:6789/0 mon.noname-a 2024-06-03T08:37:53.633 INFO:tasks.workunit.client.0.smithi088.stdout: api_asio: 1: v1:172.21.15.192:6789/0 mon.noname-b 2024-06-03T08:37:53.633 INFO:tasks.workunit.client.0.smithi088.stdout: api_asio: 2: v1:172.21.15.88:6790/0 mon.noname-c 2024-06-03T08:37:53.633 INFO:tasks.workunit.client.0.smithi088.stdout: api_asio: 2024-06-03T08:37:53.633 INFO:tasks.workunit.client.0.smithi088.stdout: api_asio: 2024-06-03T08:37:53.452+0000 7f514d647780 1 Processor -- start 2024-06-03T08:37:53.633 INFO:tasks.workunit.client.0.smithi088.stdout: api_asio: 2024-06-03T08:37:53.452+0000 7f514d647780 1 -- start start 2024-06-03T08:37:53.633 INFO:tasks.workunit.client.0.smithi088.stdout: api_asio: 2024-06-03T08:37:53.452+0000 7f514d647780 10 monclient: init 2024-06-03T08:37:53.633 INFO:tasks.workunit.client.0.smithi088.stdout: api_asio: 2024-06-03T08:37:53.452+0000 7f514d647780 10 monclient: _reopen_session rank -1 2024-06-03T08:37:53.633 INFO:tasks.workunit.client.0.smithi088.stdout: api_asio: 2024-06-03T08:37:53.453+0000 7f514d647780 10 monclient: _add_conns ranks=[1,2,0] 2024-06-03T08:37:53.633 INFO:tasks.workunit.client.0.smithi088.stdout: api_asio: 2024-06-03T08:37:53.453+0000 7f514d647780 10 monclient(hunting): picked mon.noname-b con 0x56192203a160 addr v1:172.21.15.192:6789/0 2024-06-03T08:37:53.633 INFO:tasks.workunit.client.0.smithi088.stdout: api_asio: 2024-06-03T08:37:53.453+0000 7f514d647780 10 monclient(hunting): picked mon.noname-c con 0x56192203cd80 addr v1:172.21.15.88:6790/0 2024-06-03T08:37:53.633 INFO:tasks.workunit.client.0.smithi088.stdout: api_asio: 2024-06-03T08:37:53.453+0000 7f514d647780 10 monclient(hunting): picked mon.noname-a con 0x5619220ccfb0 addr v1:172.21.15.88:6789/0 2024-06-03T08:37:53.633 INFO:tasks.workunit.client.0.smithi088.stdout: api_asio: 2024-06-03T08:37:53.453+0000 7f514d647780 1 -- --> v1:172.21.15.88:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x561921f50ab0 con 0x5619220ccfb0 2024-06-03T08:37:53.634 INFO:tasks.workunit.client.0.smithi088.stdout: api_asio: 2024-06-03T08:37:53.453+0000 7f514d647780 1 -- --> v1:172.21.15.192:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x5619220b0a50 con 0x56192203a160 2024-06-03T08:37:53.634 INFO:tasks.workunit.client.0.smithi088.stdout: api_asio: 2024-06-03T08:37:53.453+0000 7f514d647780 1 -- --> v1:172.21.15.88:6790/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x56192203eff0 con 0x56192203cd80 2024-06-03T08:37:53.634 INFO:tasks.workunit.client.0.smithi088.stdout: api_asio: 2024-06-03T08:37:53.453+0000 7f514d647780 10 monclient(hunting): _renew_subs 2024-06-03T08:37:53.634 INFO:tasks.workunit.client.0.smithi088.stdout: api_asio: 2024-06-03T08:37:53.453+0000 7f514d647780 10 monclient(hunting): authenticate will time out at 1329.263013s 2024-06-03T08:37:53.634 INFO:tasks.workunit.client.0.smithi088.stdout: api_asio: 2024-06-03T08:37:53.457+0000 7f514a183640 1 --1- >> v1:172.21.15.88:6790/0 conn(0x56192203cd80 0x561922168780 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.88:6790/0 says I am v1:172.21.15.88:59370/0 (socket says 172.21.15.88:59370) 2024-06-03T08:37:53.634 INFO:tasks.workunit.client.0.smithi088.stdout: api_asio: 2024-06-03T08:37:53.457+0000 7f514a183640 1 -- 172.21.15.88:0/273312373 learned_addr learned my addr 172.21.15.88:0/273312373 (peer_addr_for_me v1:172.21.15.88:0/0) 2024-06-03T08:37:53.634 INFO:tasks.workunit.client.0.smithi088.stdout: api_asio: 2024-06-03T08:37:53.494+0000 7f5149181640 1 -- 172.21.15.88:0/273312373 <== mon.2 v1:172.21.15.88:6790/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 3948528074 0 0) 0x56192203eff0 con 0x56192203cd80 2024-06-03T08:37:53.634 INFO:tasks.workunit.client.0.smithi088.stdout: api_asio: 2024-06-03T08:37:53.494+0000 7f5149181640 10 monclient(hunting): _negotiate 2024-06-03T08:37:53.634 INFO:tasks.workunit.client.0.smithi088.stdout: api_asio: 2024-06-03T08:37:53.494+0000 7f5149181640 10 monclient(hunting): _init_auth method 2 2024-06-03T08:37:53.634 INFO:tasks.workunit.client.0.smithi088.stdout: api_asio: 2024-06-03T08:37:53.494+0000 7f5149181640 10 monclient(hunting): _init_auth creating new auth 2024-06-03T08:37:53.634 INFO:tasks.workunit.client.0.smithi088.stdout: api_asio: 2024-06-03T08:37:53.494+0000 7f5149181640 10 monclient(hunting): my global_id is 4436 2024-06-03T08:37:53.634 INFO:tasks.workunit.client.0.smithi088.stdout: api_asio: 2024-06-03T08:37:53.494+0000 7f5149181640 1 -- 172.21.15.88:0/273312373 --> v1:17test api_service on pid 40685 2024-06-03T08:37:53.645 INFO:tasks.workunit.client.0.smithi088.stderr:+++ alias 'xzgrep=xzgrep --color=auto' 2024-06-03T08:37:53.646 INFO:tasks.workunit.client.0.smithi088.stderr:+++ alias 'xzegrep=xzegrep --color=auto' 2024-06-03T08:37:53.646 INFO:tasks.workunit.client.0.smithi088.stderr:+++ alias 'xzfgrep=xzfgrep --color=auto' 2024-06-03T08:37:53.646 INFO:tasks.workunit.client.0.smithi088.stderr:++ for i in /etc/profile.d/*.sh 2024-06-03T08:37:53.646 INFO:tasks.workunit.client.0.smithi088.stderr:++ '[' -r /etc/profile.d/colorzgrep.sh ']' 2024-06-03T08:37:53.646 INFO:tasks.workunit.client.0.smithi088.stderr:++ '[' '' ']' 2024-06-03T08:37:53.646 INFO:tasks.workunit.client.0.smithi088.stderr:++ . /etc/profile.d/colorzgrep.sh 2024-06-03T08:37:53.646 INFO:tasks.workunit.client.0.smithi088.stderr:+++ '[' -f /usr/libexec/grepconf.sh ']' 2024-06-03T08:37:53.646 INFO:tasks.workunit.client.0.smithi088.stderr:+ sed 's/^/ api_stat: /' 2024-06-03T08:37:53.646 INFO:tasks.workunit.client.0.smithi088.stderr:+++ /usr/libexec/grepconf.sh -c 2024-06-03T08:37:53.646 INFO:tasks.workunit.client.0.smithi088.stderr:++ awk '{print $1}' 2024-06-03T08:37:53.646 INFO:tasks.workunit.client.0.smithi088.stderr:++ '[' 0022 -eq 0 ']' 2024-06-03T08:37:53.646 INFO:tasks.workunit.client.0.smithi088.stderr:++ SHELL=/bin/bash 2024-06-03T08:37:53.647 INFO:tasks.workunit.client.0.smithi088.stderr:++ for i in /etc/profile.d/*.sh 2024-06-03T08:37:53.647 INFO:tasks.workunit.client.0.smithi088.stderr:++ '[' -r /etc/profile.d/bash_completion.sh ']' 2024-06-03T08:37:53.647 INFO:tasks.workunit.client.0.smithi088.stderr:++ '[' '' ']' 2024-06-03T08:37:53.647 INFO:tasks.workunit.client.0.smithi088.stderr:++ . /etc/profile.d/bash_completion.sh 2024-06-03T08:37:53.647 INFO:tasks.workunit.client.0.smithi088.stderr:+++ '[' 'x5.1.8(1)-release' '!=' x -a x '!=' x -a x = x ']' 2024-06-03T08:37:53.647 INFO:tasks.workunit.client.0.smithi088.stderr:++ for i in /etc/profile.d/*.sh 2024-06-03T08:37:53.647 INFO:tasks.workunit.client.0.smithi088.stderr:++ '[' -r /etc/profile.d/cobbler.sh ']' 2024-06-03T08:37:53.647 INFO:tasks.workunit.client.0.smithi088.stderr:++ '[' '' ']' 2024-06-03T08:37:53.647 INFO:tasks.workunit.client.0.smithi088.stderr:++ . /etc/profile.d/cobbler.sh 2024-06-03T08:37:53.647 INFO:tasks.workunit.client.0.smithi088.stderr:+++ export COBBLER_SERVER=172.21.0.11 2024-06-03T08:37:53.647 INFO:tasks.workunit.client.0.smithi088.stderr:+++ COBBLER_SERVER=172.21.0.11 2024-06-03T08:37:53.647 INFO:tasks.workunit.client.0.smithi088.stderr:++ for i in /etc/profile.d/*.sh 2024-06-03T08:37:53.647 INFO:tasks.workunit.client.0.smithi088.stderr:++ '[' -r /etc/profile.d/colorgrep.sh ']' 2024-06-03T08:37:53.648 INFO:tasks.workunit.client.0.smithi088.stderr:++ '[' '' ']' 2024-06-03T08:37:53.648 INFO:tasks.workunit.client.0.smithi088.stderr:++ . /etc/profile.d/colorgrep.sh 2024-06-03T08:37:53.648 INFO:tasks.workunit.client.0.smithi088.stderr:+++ /usr/libexec/grepconf.sh -c 2024-06-03T08:37:53.648 INFO:tasks.workunit.client.0.smithi088.stderr:+++ alias 'zgrep=zgrep --color=auto' 2024-06-03T08:37:53.648 INFO:tasks.workunit.client.0.smithi088.stderr:+++ alias 'zfgrep=zfgrep --color=auto' 2024-06-03T08:37:53.648 INFO:tasks.workunit.client.0.smithi088.stderr:+++ alias 'zegrep=zegrep --color=auto' 2024-06-03T08:37:53.648 INFO:tasks.workunit.client.0.smithi088.stderr:++ for i in /etc/profile.d/*.sh 2024-06-03T08:37:53.648 INFO:tasks.workunit.client.0.smithi088.stderr:++ '[' -r /etc/profile.d/debuginfod.sh ']' 2024-06-03T08:37:53.648 INFO:tasks.workunit.client.0.smithi088.stderr:++ '[' '' ']' 2024-06-03T08:37:53.648 INFO:tasks.workunit.client.0.smithi088.stderr:++ . /etc/profile.d/debuginfod.sh 2024-06-03T08:37:53.648 INFO:tasks.workunit.client.0.smithi088.stderr:+++ '[' -z 'https://debuginfod.centos.org/ ' ']' 2024-06-03T08:37:53.648 INFO:tasks.workunit.client.0.smithi088.stderr:+ ff=api_service 2024-06-03T08:37:53.648 INFO:tasks.workunit.client.0.smithi088.stderr:++ for i in /etc/profile.d/*.sh 2024-06-03T08:37:53.649 INFO:tasks.workunit.client.0.smithi088.stderr:++ '[' -r /etc/profile.d/gawk.sh ']' 2024-06-03T08:37:53.649 INFO:tasks.workunit.client.0.smithi088.stderr:++ '[' '' ']' 2024-06-03T08:37:53.649 INFO:tasks.workunit.client.0.smithi088.stderr:++ . /etc/profile.d/gawk.sh 2024-06-03T08:37:53.649 INFO:tasks.workunit.client.0.smithi088.stderr:+ pid=40685 2024-06-03T08:37:53.649 INFO:tasks.workunit.client.0.smithi088.stderr:+ echo 'test api_service on pid 40685' 2024-06-03T08:37:53.649 INFO:tasks.workunit.client.0.smithi088.stderr:+ pids[$f]=40685 2024-06-03T08:37:53.649 INFO:tasks.workunit.client.0.smithi088.stderr:+ for f in api_aio api_aio_pp api_io api_io_pp api_asio api_list api_lock api_lock_pp api_misc api_misc_pp api_tier_pp api_pool api_snapshots api_snapshots_pp api_stat api_stat_pp api_watch_notify api_watch_notify_pp api_cmd api_cmd_pp api_service api_service_pp api_c_write_operations api_c_read_operations api_cls_remote_reads list_parallel open_pools_parallel delete_pools_parallel 2024-06-03T08:37:53.649 INFO:tasks.workunit.client.0.smithi088.stderr:+ '[' 1 -eq 1 ']' 2024-06-03T08:37:53.649 INFO:tasks.workunit.client.0.smithi088.stderr:++ for i in /etc/profile.d/*.sh 2024-06-03T08:37:53.649 INFO:tasks.workunit.client.0.smithi088.stderr:++ '[' -r /etc/profile.d/lang.sh ']' 2024-06-03T08:37:53.649 INFO:tasks.workunit.client.0.smithi088.stderr:++ '[' '' ']' 2024-06-03T08:37:53.649 INFO:tasks.workunit.client.0.smithi088.stderr:++ . /etc/profile.d/lang.sh 2024-06-03T08:37:53.649 INFO:tasks.workunit.client.0.smithi088.stderr:+++ unset LANG_backup 2024-06-03T08:37:53.650 INFO:tasks.workunit.client.0.smithi088.stderr:++ printf %25s api_service_pp 2024-06-03T08:37:53.650 INFO:tasks.workunit.client.0.smithi088.stderr:+ r=' api_service_pp' 2024-06-03T08:37:53.650 INFO:tasks.workunit.client.0.smithi088.stderr:+ bash -o pipefail -exc 'ceph_test_rados_api_service 2>&1 | tee ceph_test_rados_api_service.log | sed "s/^/ api_service: /"' 2024-06-03T08:37:53.650 INFO:tasks.workunit.client.0.smithi088.stderr:+++ alias 'grep=grep --color=auto' 2024-06-03T08:37:53.650 INFO:tasks.workunit.client.0.smithi088.stderr:+++ alias 'grep=grep --color=auto' 2024-06-03T08:37:53.650 INFO:tasks.workunit.client.0.smithi088.stderr:+++ alias 'egrep=egrep --color=auto' 2024-06-03T08:37:53.650 INFO:tasks.workunit.client.0.smithi088.stderr:+++ alias 'egrep=egrep --color=auto' 2024-06-03T08:37:53.650 INFO:tasks.workunit.client.0.smithi088.stderr:+++ alias 'fgrep=fgrep --color=auto' 2024-06-03T08:37:53.650 INFO:tasks.workunit.client.0.smithi088.stderr:++ for i in /etc/profile.d/*.sh 2024-06-03T08:37:53.650 INFO:tasks.workunit.client.0.smithi088.stderr:++ '[' -r /etc/profile.d/colorls.sh ']' 2024-06-03T08:37:53.650 INFO:tasks.workunit.client.0.smithi088.stderr:++ '[' '' ']' 2024-06-03T08:37:53.650 INFO:tasks.workunit.client.0.smithi088.stderr:++ . /etc/profile.d/colorls.sh 2024-06-03T08:37:53.650 INFO:tasks.workunit.client.0.smithi088.stderr:++++ /usr/bin/locale 2024-06-03T08:37:53.651 INFO:tasks.workunit.client.0.smithi088.stderr:+++ '[' '!' -t 0 ']' 2024-06-03T08:37:53.651 INFO:tasks.workunit.client.0.smithi088.stderr:+++ alias 'fgrep=fgrep --color=auto' 2024-06-03T08:37:53.651 INFO:tasks.workunit.client.0.smithi088.stderr:++ for i in /etc/profile.d/*.sh 2024-06-03T08:37:53.651 INFO:tasks.workunit.client.0.smithi088.stderr:+++ return 2024-06-03T08:37:53.651 INFO:tasks.workunit.client.0.smithi088.stderr:++ for i in /etc/profile.d/*.sh 2024-06-03T08:37:53.651 INFO:tasks.workunit.client.0.smithi088.stderr:++ '[' -r /etc/profile.d/colorsysstat.sh ']' 2024-06-03T08:37:53.651 INFO:tasks.workunit.client.0.smithi088.stderr:++ '[' '' ']' 2024-06-03T08:37:53.651 INFO:tasks.workunit.client.0.smithi088.stderr:++ . /etc/profile.d/colorsysstat.sh 2024-06-03T08:37:53.651 INFO:tasks.workunit.client.0.smithi088.stderr:+++ export S_COLORS=auto 2024-06-03T08:37:53.651 INFO:tasks.workunit.client.0.smithi088.stderr:++ '[' -r /etc/profile.d/colorls.sh ']' 2024-06-03T08:37:53.651 INFO:tasks.workunit.client.0.smithi088.stderr:++ '[' '' ']' 2024-06-03T08:37:53.651 INFO:tasks.workunit.client.0.smithi088.stderr:++ . /etc/profile.d/colorls.sh 2024-06-03T08:37:53.651 INFO:tasks.workunit.client.0.smithi088.stderr:+++ S_COLORS=auto 2024-06-03T08:37:53.651 INFO:tasks.workunit.client.0.smithi088.stderr:+++ '[' '!' -t 0 ']' 2024-06-03T08:37:53.652 INFO:tasks.workunit.client.0.smithi088.stderr:+++ return 2024-06-03T08:37:53.652 INFO:tasks.workunit.client.0.smithi088.stderr:++ for i in /etc/profile.d/*.sh 2024-06-03T08:37:53.652 INFO:tasks.workunit.client.0.smithi088.stderr:++ '[' -r /etc/profile.d/colorsysstat.sh ']' 2024-06-03T08:37:53.652 INFO:tasks.workunit.client.0.smithi088.stderr:++ '[' '' ']' 2024-06-03T08:37:53.652 INFO:tasks.workunit.client.0.smithi088.stderr:++ . /etc/profile.d/colorsysstat.sh 2024-06-03T08:37:53.652 INFO:tasks.workunit.client.0.smithi088.stderr:++ for i in /etc/profile.d/*.sh 2024-06-03T08:37:53.652 INFO:tasks.workunit.client.0.smithi088.stderr:+++ export S_COLORS=auto 2024-06-03T08:37:53.652 INFO:tasks.workunit.client.0.smithi088.stderr:+++ S_COLORS=auto 2024-06-03T08:37:53.652 INFO:tasks.workunit.client.0.smithi088.stderr:++ for i in /etc/profile.d/*.sh 2024-06-03T08:37:53.652 INFO:tasks.workunit.client.0.smithi088.stderr:++ '[' -r /etc/profile.d/colorxzgrep.sh ']' 2024-06-03T08:37:53.652 INFO:tasks.workunit.client.0.smithi088.stderr:++ '[' '' ']' 2024-06-03T08:37:53.652 INFO:tasks.workunit.client.0.smithi088.stderr:++ . /etc/profile.d/colorxzgrep.sh 2024-06-03T08:37:53.652 INFO:tasks.workunit.client.0.smithi088.stderr:+++ /usr/libexec/grepconf.sh -c 2024-06-03T08:37:53.653 INFO:tasks.workunit.client.0.smithi088.stderr:++ '[' -r /etc/profile.d/colorxzgrep.sh ']' 2024-06-03T08:37:53.653 INFO:tasks.workunit.client.0.smithi088.stderr:++ '[' '' ']' 2024-06-03T08:37:53.653 INFO:tasks.workunit.client.0.smithi088.stderr:++ . /etc/profile.d/colorxzgrep.sh 2024-06-03T08:37:53.653 INFO:tasks.workunit.client.0.smithi088.stderr:+++ /usr/libexec/grepconf.sh -c 2024-06-03T08:37:53.653 INFO:tasks.workunit.client.0.smithi088.stderr:+++ '[' -n '' ']' 2024-06-03T08:37:53.653 INFO:tasks.workunit.client.0.smithi088.stderr:+++ LANG_backup=en_US.UTF-8 2024-06-03T08:37:53.653 INFO:tasks.workunit.client.0.smithi088.stderr:+++ for config in /etc/locale.conf "${HOME}/.i18n" 2024-06-03T08:37:53.653 INFO:tasks.workunit.client.0.smithi088.stderr:+++ '[' -f /etc/locale.conf ']' 2024-06-03T08:37:53.653 INFO:tasks.workunit.client.0.smithi088.stderr:++++ /usr/bin/locale 2024-06-03T08:37:53.653 INFO:tasks.workunit.client.0.smithi088.stderr:+++ '[' -x /usr/bin/sed ']' 2024-06-03T08:37:53.653 INFO:tasks.workunit.client.0.smithi088.stderr:+ '[' -f /etc/bashrc ']' 2024-06-03T08:37:53.653 INFO:tasks.workunit.client.0.smithi088.stderr:+++ '[' -n '' ']' 2024-06-03T08:37:53.653 INFO:tasks.workunit.client.0.smithi088.stderr:+++ LANG_backup=en_US.UTF-8 2024-06-03T08:37:53.653 INFO:tasks.workunit.client.0.smithi088.stderr:++++ /usr/bin/sed -r -e 's/^[[:blank:]]*([[:upper:]_]+)=([[:print:][:digit:]\._-]+|"[[:print:][:digit:]\._-]+")/export \1=\2/;t;d' /etc/locale.conf 2024-06-03T08:37:53.654 INFO:tasks.workunit.client.0.smithi088.stderr:+++ for config in /etc/locale.conf "${HOME}/.i18n" 2024-06-03T08:37:53.654 INFO:tasks.workunit.client.0.smithi088.stderr:+++ '[' -f /etc/locale.conf ']' 2024-06-03T08:37:53.654 INFO:tasks.workunit.client.0.smithi088.stderr:+++ '[' -x /usr/bin/sed ']' 2024-06-03T08:37:53.654 INFO:tasks.workunit.client.0.smithi088.stderr:++++ /usr/bin/sed -r -e 's/^[[:blank:]]*([[:upper:]_]+)=([[:print:][:digit:]\._-]+|"[[:print:][:digit:]\._-]+")/export \1=\2/;t;d' /etc/locale.conf 2024-06-03T08:37:53.654 INFO:tasks.workunit.client.0.smithi088.stderr:++ awk '{print $1}' 2024-06-03T08:37:53.654 INFO:tasks.workunit.client.0.smithi088.stderr:+ . /etc/bashrc 2024-06-03T08:37:53.654 INFO:tasks.workunit.client.0.smithi088.stderr:++ '[' -z '' ']' 2024-06-03T08:37:53.654 INFO:tasks.workunit.client.0.smithi088.stderr:++ BASHRCSOURCED=Y 2024-06-03T08:37:53.654 INFO:tasks.workunit.client.0.smithi088.stderr:++ '[' '' ']' 2024-06-03T08:37:53.654 INFO:tasks.workunit.client.0.smithi088.stderr:++ shopt -q login_shell 2024-06-03T08:37:53.654 INFO:tasks.workunit.client.0.smithi088.stderr:+++ alias 'zgrep=zgrep --color=auto' 2024-06-03T08:37:53.654 INFO:tasks.workunit.client.0.smithi088.stderr:+++ alias 'zfgrep=zfgrep --color=auto' 2024-06-03T08:37:53.654 INFO:tasks.workunit.client.0.smithi088.stderr:+++ alias 'zegrep=zegrep --color=auto' 2024-06-03T08:37:53.655 INFO:tasks.workunit.client.0.smithi088.stderr:++ for i in /etc/profile.d/*.sh 2024-06-03T08:37:53.655 INFO:tasks.workunit.client.0.smithi088.stderr:++ '[' -r /etc/profile.d/debuginfod.sh ']' 2024-06-03T08:37:53.655 INFO:tasks.workunit.client.0.smithi088.stderr:++ '[' '' ']' 2024-06-03T08:37:53.655 INFO:tasks.workunit.client.0.smithi088.stderr:++ . /etc/profile.d/debuginfod.sh 2024-06-03T08:37:53.655 INFO:tasks.workunit.client.0.smithi088.stderr:+++ '[' -z 'https://debuginfod.centos.org/ ' ']' 2024-06-03T08:37:53.655 INFO:tasks.workunit.client.0.smithi088.stderr:+++ eval export 'LANG="en_US.UTF-8"' 2024-06-03T08:37:53.655 INFO:tasks.workunit.client.0.smithi088.stderr:++ for i in /etc/profile.d/*.sh 2024-06-03T08:37:53.655 INFO:tasks.workunit.client.0.smithi088.stderr:++ '[' -r /etc/profile.d/gawk.sh ']' 2024-06-03T08:37:53.655 INFO:tasks.workunit.client.0.smithi088.stderr:++++ export LANG=en_US.UTF-8 2024-06-03T08:37:53.655 INFO:tasks.workunit.client.0.smithi088.stderr:++ '[' '' ']' 2024-06-03T08:37:53.655 INFO:tasks.workunit.client.0.smithi088.stderr:++++ LANG=en_US.UTF-8 2024-06-03T08:37:53.655 INFO:tasks.workunit.client.0.smithi088.stderr:++ . /etc/profile.d/gawk.sh 2024-06-03T08:37:53.655 INFO:tasks.workunit.client.0.smithi088.stderr:+++ for config in /etc/locale.conf "${HOME}/.i18n" 2024-06-03T08:37:53.656 INFO:tasks.workunit.client.0.smithi088.stderr:+++ '[' -f /home/ubuntu/.i18n ']' 2024-06-03T08:37:53.656 INFO:tasks.workunit.client.0.smithi088.stderr:+++ '[' -n en_US.UTF-8 ']' 2024-06-03T08:37:53.656 INFO:tasks.workunit.client.0.smithi088.stderr:+++ LANG=en_US.UTF-8 2024-06-03T08:37:53.656 INFO:tasks.workunit.client.0.smithi088.stderr:+++ unset LANG_backup config 2024-06-03T08:37:53.656 INFO:tasks.workunit.client.0.smithi088.stderr:++ for i in /etc/profile.d/*.sh 2024-06-03T08:37:53.656 INFO:tasks.workunit.client.0.smithi088.stderr:+++ '[' -n '' ']' 2024-06-03T08:37:53.656 INFO:tasks.workunit.client.0.smithi088.stderr:++ '[' -r /etc/profile.d/lang.sh ']' 2024-06-03T08:37:53.656 INFO:tasks.workunit.client.0.smithi088.stderr:++ '[' '' ']' 2024-06-03T08:37:53.656 INFO:tasks.workunit.client.0.smithi088.stderr:++ . /etc/profile.d/lang.sh 2024-06-03T08:37:53.656 INFO:tasks.workunit.client.0.smithi088.stderr:+++ unset LANG_backup 2024-06-03T08:37:53.656 INFO:tasks.workunit.client.0.smithi088.stderr:+++ '[' -n en_US.UTF-8 ']' 2024-06-03T08:37:53.656 INFO:tasks.workunit.client.0.smithi088.stderr:+++ '[' dumb = linux ']' 2024-06-03T08:37:53.656 INFO:tasks.workunit.client.0.smithi088.stderr:+++ umask 2024-06-03T08:37:53.656 INFO:tasks.workunit.client.0.smithi088.stderr:++ for i in /etc/profile.d/*.sh 2024-06-03T08:37:53.657 INFO:tasks.workunit.client.0.smithi088.stderr:++ '[' -r /etc/profile.d/less.sh ']' 2024-06-03T08:37:53.657 INFO:tasks.workunit.client.0.smithi088.stderr:++ '[' '' ']' 2024-06-03T08:37:53.657 INFO:tasks.workunit.client.0.smithi088.stderr:++ . /etc/profile.d/less.sh 2024-06-03T08:37:53.657 INFO:tasks.workunit.client.0.smithi088.stderr:+++ '[' -z '||/usr/bin/lesspipe.sh %s' ']' 2024-06-03T08:37:53.657 INFO:tasks.workunit.client.0.smithi088.stderr:++ for i in /etc/profile.d/*.sh 2024-06-03T08:37:53.657 INFO:tasks.workunit.client.0.smithi088.stderr:++ '[' -r /etc/profile.d/which2.sh ']' 2024-06-03T08:37:53.657 INFO:tasks.workunit.client.0.smithi088.stderr:++ '[' '' ']' 2024-06-03T08:37:53.657 INFO:tasks.workunit.client.0.smithi088.stderr:++ . /etc/profile.d/which2.sh 2024-06-03T08:37:53.657 INFO:tasks.workunit.client.0.smithi088.stderr:+++ case "$(basename $(readlink /proc/$$/exe))" in 2024-06-03T08:37:53.657 INFO:tasks.workunit.client.0.smithi088.stderr:++ '[' 0022 -eq 0 ']' 2024-06-03T08:37:53.657 INFO:tasks.workunit.client.0.smithi088.stderr:++ SHELL=/bin/bash 2024-06-03T08:37:53.657 INFO:tasks.workunit.client.0.smithi088.stderr:++ for i in /etc/profile.d/*.sh 2024-06-03T08:37:53.657 INFO:tasks.workunit.client.0.smithi088.stderr:++ '[' -r /etc/profile.d/bash_completion.sh ']' 2024-06-03T08:37:53.658 INFO:tasks.workunit.client.0.smithi088.stderr:++ '[' '' ']' 2024-06-03T08:37:53.658 INFO:tasks.workunit.client.0.smithi088.stderr:++ . /etc/profile.d/bash_completion.sh 2024-06-03T08:37:53.658 INFO:tasks.workunit.client.0.smithi088.stderr:+++ '[' 'x5.1.8(1)-release' '!=' x -a x '!=' x -a x = x ']' 2024-06-03T08:37:53.658 INFO:tasks.workunit.client.0.smithi088.stderr:++ for i in /etc/profile.d/*.sh 2024-06-03T08:37:53.658 INFO:tasks.workunit.client.0.smithi088.stderr:++ '[' -r /etc/profile.d/cobbler.sh ']' 2024-06-03T08:37:53.658 INFO:tasks.workunit.client.0.smithi088.stderr:++ '[' '' ']' 2024-06-03T08:37:53.658 INFO:tasks.workunit.client.0.smithi088.stderr:++ . /etc/profile.d/cobbler.sh 2024-06-03T08:37:53.658 INFO:tasks.workunit.client.0.smithi088.stderr:+++ export COBBLER_SERVER=172.21.0.11 2024-06-03T08:37:53.658 INFO:tasks.workunit.client.0.smithi088.stderr:+++ COBBLER_SERVER=172.21.0.11 2024-06-03T08:37:53.658 INFO:tasks.workunit.client.0.smithi088.stderr:++ for i in /etc/profile.d/*.sh 2024-06-03T08:37:53.658 INFO:tasks.workunit.client.0.smithi088.stderr:++ '[' -r /etc/profile.d/colorgrep.sh ']' 2024-06-03T08:37:53.658 INFO:tasks.workunit.client.0.smithi088.stderr:++++ /usr/bin/locale 2024-06-03T08:37:53.658 INFO:tasks.workunit.client.0.smithi088.stderr:++ '[' '' ']' 2024-06-03T08:37:53.659 INFO:tasks.workunit.client.0.smithi088.stderr:++ . /etc/profile.d/colorgrep.sh 2024-06-03T08:37:53.659 INFO:tasks.workunit.client.0.smithi088.stderr:+++ /usr/libexec/grepconf.sh -c 2024-06-03T08:37:53.659 INFO:tasks.workunit.client.0.smithi088.stderr:++ echo api_service_pp 2024-06-03T08:37:53.659 INFO:tasks.workunit.client.0.smithi088.stderr:+++ '[' -n '' ']' 2024-06-03T08:37:53.659 INFO:tasks.workunit.client.0.smithi088.stderr:+++ LANG_backup=en_US.UTF-8 2024-06-03T08:37:53.659 INFO:tasks.workunit.client.0.smithi088.stderr:+++ for config in /etc/locale.conf "${HOME}/.i18n" 2024-06-03T08:37:53.659 INFO:tasks.workunit.client.0.smithi088.stderr:+++ '[' -f /etc/locale.conf ']' 2024-06-03T08:37:53.659 INFO:tasks.workunit.client.0.smithi088.stderr:+++ '[' -x /usr/bin/sed ']' 2024-06-03T08:37:53.659 INFO:tasks.workunit.client.0.smithi088.stderr:+ ff=api_service_pp 2024-06-03T08:37:53.659 INFO:tasks.workunit.client.0.smithi088.stderr:+++ eval export 'LANG="en_US.UTF-8"' 2024-06-03T08:37:53.659 INFO:tasks.workunit.client.0.smithi088.stderr:++++ export LANG=en_US.UTF-8 2024-06-03T08:37:53.659 INFO:tasks.workunit.client.0.smithi088.stderr:++++ LANG=en_US.UTF-8 2024-06-03T08:37:53.659 INFO:tasks.workunit.client.0.smithi088.stderr:+++ for config in /etc/locale.conf "${HOME}/.i18n" 2024-06-03T08:37:53.659 INFO:tasks.workunit.client.0.smithi088.stderr:+++ '[' -f /home/ubuntu/.i18n ']' 2024-06-03T08:37:53.659 INFO:tasks.workunit.client.0.smithi088.stderr:+++ '[' -n en_US.UTF-8 ']' 2024-06-03T08:37:53.660 INFO:tasks.workunit.client.0.smithi088.stderr:+++ LANG=en_US.UTF-8 2024-06-03T08:37:53.660 INFO:tasks.workunit.client.0.smithi088.stderr:+++ unset LANG_backup config 2024-06-03T08:37:53.660 INFO:tasks.workunit.client.0.smithi088.stderr:+++ '[' -n '' ']' 2024-06-03T08:37:53.660 INFO:tasks.workunit.client.0.smithi088.stderr:+++ '[' -n en_US.UTF-8 ']' 2024-06-03T08:37:53.660 INFO:tasks.workunit.client.0.smithi088.stderr:+++ '[' dumb = linux ']' 2024-06-03T08:37:53.660 INFO:tasks.workunit.client.0.smithi088.stderr:++ for i in /etc/profile.d/*.sh 2024-06-03T08:37:53.660 INFO:tasks.workunit.client.0.smithi088.stderr:++ '[' -r /etc/profile.d/less.sh ']' 2024-06-03T08:37:53.660 INFO:tasks.workunit.client.0.smithi088.stderr:++ '[' '' ']' 2024-06-03T08:37:53.660 INFO:tasks.workunit.client.0.smithi088.stderr:++ . /etc/profile.d/less.sh 2024-06-03T08:37:53.660 INFO:tasks.workunit.client.0.smithi088.stderr:+++ '[' -z '||/usr/bin/lesspipe.sh %s' ']' 2024-06-03T08:37:53.660 INFO:tasks.workunit.client.0.smithi088.stderr:++ for i in /etc/profile.d/*.sh 2024-06-03T08:37:53.660 INFO:tasks.workunit.client.0.smithi088.stderr:++ '[' -r /etc/profile.d/which2.sh ']' 2024-06-03T08:37:53.660 INFO:tasks.workunit.client.0.smithi088.stderr:++ '[' '' ']' 2024-06-03T08:37:53.661 INFO:tasks.workunit.client.0.smithi088.stderr:++ . /etc/profile.d/which2.sh 2024-06-03T08:37:53.661 INFO:tasks.workunit.client.0.smithi088.stderr:+++ case "$(basename $(readlink /proc/$$/exe))" in 2024-06-03T08:37:53.661 INFO:tasks.workunit.client.0.smithi088.stderr:+ pid=40723 2024-06-03T08:37:53.661 INFO:tasks.workunit.client.0.smithi088.stderr:+ echo 'test api_service_pp on pid 40723' 2024-06-03T08:37:53.661 INFO:tasks.workunit.client.0.smithi088.stdout:test api_service_pp on pid 40723 2024-06-03T08:37:53.668 INFO:tasks.workunit.client.0.smithi088.stderr:+ pids[$f]=40723 2024-06-03T08:37:53.668 INFO:tasks.workunit.client.0.smithi088.stderr:+ for f in api_aio api_aio_pp api_io api_io_pp api_asio api_list api_lock api_lock_pp api_misc api_misc_pp api_tier_pp api_pool api_snapshots api_snapshots_pp api_stat api_stat_pp api_watch_notify api_watch_notify_pp api_cmd api_cmd_pp api_service api_service_pp api_c_write_operations api_c_read_operations api_cls_remote_reads list_parallel open_pools_parallel delete_pools_parallel 2024-06-03T08:37:53.669 INFO:tasks.workunit.client.0.smithi088.stderr:+ '[' 1 -eq 1 ']' 2024-06-03T08:37:53.669 INFO:tasks.workunit.client.0.smithi088.stderr:+ bash -o pipefail -exc 'ceph_test_rados_api_service_pp 2>&1 | tee ceph_test_rados_api_service_pp.log | sed "s/^/ api_service_pp: /"' 2024-06-03T08:37:53.669 INFO:tasks.workunit.client.0.smithi088.stderr:++++ /usr/bin/sed -r -e 's/^[[:blank:]]*([[:upper:]_]+)=([[:print:][:digit:]\._-]+|"[[:print:][:digit:]\._-]+")/export \1=\2/;t;d' /etc/locale.conf 2024-06-03T08:37:53.669 INFO:tasks.workunit.client.0.smithi088.stderr:+++ alias 'xzgrep=xzgrep --color=auto' 2024-06-03T08:37:53.669 INFO:tasks.workunit.client.0.smithi088.stderr:+++ alias 'xzegrep=xzegrep --color=auto' 2024-06-03T08:37:53.669 INFO:tasks.workunit.client.0.smithi088.stderr:+++ alias 'xzfgrep=xzfgrep --color=auto' 2024-06-03T08:37:53.669 INFO:tasks.workunit.client.0.smithi088.stderr:++ for i in /etc/profile.d/*.sh 2024-06-03T08:37:53.669 INFO:tasks.workunit.client.0.smithi088.stderr:++ '[' -r /etc/profile.d/colorzgrep.sh ']' 2024-06-03T08:37:53.669 INFO:tasks.workunit.client.0.smithi088.stderr:++ '[' '' ']' 2024-06-03T08:37:53.669 INFO:tasks.workunit.client.0.smithi088.stderr:++ . /etc/profile.d/colorzgrep.sh 2024-06-03T08:37:53.669 INFO:tasks.workunit.client.0.smithi088.stderr:+++ '[' -f /usr/libexec/grepconf.sh ']' 2024-06-03T08:37:53.669 INFO:tasks.workunit.client.0.smithi088.stderr:+++ /usr/libexec/grepconf.sh -c 2024-06-03T08:37:53.669 INFO:tasks.workunit.client.0.smithi088.stderr:++ printf %25s api_c_write_operations 2024-06-03T08:37:53.670 INFO:tasks.workunit.client.0.smithi088.stderr:+ r=' api_c_write_operations' 2024-06-03T08:37:53.670 INFO:tasks.workunit.client.0.smithi088.stderr:+++++ readlink /proc/40590/exe 2024-06-03T08:37:53.670 INFO:tasks.workunit.client.0.smithi088.stderr:+ '[' -f /etc/bashrc ']' 2024-06-03T08:37:53.670 INFO:tasks.workunit.client.0.smithi088.stderr:+ . /etc/bashrc 2024-06-03T08:37:53.670 INFO:tasks.workunit.client.0.smithi088.stderr:++ awk '{print $1}' 2024-06-03T08:37:53.670 INFO:tasks.workunit.client.0.smithi088.stderr:++ '[' -z '' ']' 2024-06-03T08:37:53.670 INFO:tasks.workunit.client.0.smithi088.stderr:+++ alias 'xzgrep=xzgrep --color=auto' 2024-06-03T08:37:53.670 INFO:tasks.workunit.client.0.smithi088.stderr:++ BASHRCSOURCED=Y 2024-06-03T08:37:53.670 INFO:tasks.workunit.client.0.smithi088.stderr:++ '[' '' ']' 2024-06-03T08:37:53.670 INFO:tasks.workunit.client.0.smithi088.stderr:++ shopt -q login_shell 2024-06-03T08:37:53.670 INFO:tasks.workunit.client.0.smithi088.stderr:++++ basename /usr/bin/bash 2024-06-03T08:37:53.670 INFO:tasks.workunit.client.0.smithi088.stderr:+++ umask 2024-06-03T08:37:53.670 INFO:tasks.workunit.client.0.smithi088.stderr:+++++ readlink /proc/40564/exe 2024-06-03T08:37:53.671 INFO:tasks.workunit.client.0.smithi088.stderr:++ '[' 0022 -eq 0 ']' 2024-06-03T08:37:53.671 INFO:tasks.workunit.client.0.smithi088.stderr:++ SHELL=/bin/bash 2024-06-03T08:37:53.671 INFO:tasks.workunit.client.0.smithi088.stderr:++ for i in /etc/profile.d/*.sh 2024-06-03T08:37:53.671 INFO:tasks.workunit.client.0.smithi088.stderr:++ '[' -r /etc/profile.d/bash_completion.sh ']' 2024-06-03T08:37:53.671 INFO:tasks.workunit.client.0.smithi088.stderr:++ '[' '' ']' 2024-06-03T08:37:53.671 INFO:tasks.workunit.client.0.smithi088.stderr:++ . /etc/profile.d/bash_completion.sh 2024-06-03T08:37:53.671 INFO:tasks.workunit.client.0.smithi088.stderr:+++ alias 'xzegrep=xzegrep --color=auto' 2024-06-03T08:37:53.671 INFO:tasks.workunit.client.0.smithi088.stderr:+++ alias 'xzfgrep=xzfgrep --color=auto' 2024-06-03T08:37:53.671 INFO:tasks.workunit.client.0.smithi088.stderr:++ for i in /etc/profile.d/*.sh 2024-06-03T08:37:53.671 INFO:tasks.workunit.client.0.smithi088.stderr:+++ '[' 'x5.1.8(1)-release' '!=' x -a x '!=' x -a x = x ']' 2024-06-03T08:37:53.671 INFO:tasks.workunit.client.0.smithi088.stderr:++ '[' -r /etc/profile.d/colorzgrep.sh ']' 2024-06-03T08:37:53.671 INFO:tasks.workunit.client.0.smithi088.stderr:++ '[' '' ']' 2024-06-03T08:37:53.671 INFO:tasks.workunit.client.0.smithi088.stderr:++ for i in /etc/profile.d/*.sh 2024-06-03T08:37:53.672 INFO:tasks.workunit.client.0.smithi088.stderr:++ . /etc/profile.d/colorzgrep.sh 2024-06-03T08:37:53.672 INFO:tasks.workunit.client.0.smithi088.stderr:++ '[' -r /etc/profile.d/cobbler.sh ']' 2024-06-03T08:37:53.672 INFO:tasks.workunit.client.0.smithi088.stderr:++ '[' '' ']' 2024-06-03T08:37:53.672 INFO:tasks.workunit.client.0.smithi088.stderr:++ . /etc/profile.d/cobbler.sh 2024-06-03T08:37:53.672 INFO:tasks.workunit.client.0.smithi088.stderr:+++ export COBBLER_SERVER=172.21.0.11 2024-06-03T08:37:53.672 INFO:tasks.workunit.client.0.smithi088.stderr:+++ COBBLER_SERVER=172.21.0.11 2024-06-03T08:37:53.672 INFO:tasks.workunit.client.0.smithi088.stderr:++ for i in /etc/profile.d/*.sh 2024-06-03T08:37:53.672 INFO:tasks.workunit.client.0.smithi088.stderr:++ '[' -r /etc/profile.d/colorgrep.sh ']' 2024-06-03T08:37:53.672 INFO:tasks.workunit.client.0.smithi088.stderr:++ '[' '' ']' 2024-06-03T08:37:53.672 INFO:tasks.workunit.client.0.smithi088.stderr:++ . /etc/profile.d/colorgrep.sh 2024-06-03T08:37:53.672 INFO:tasks.workunit.client.0.smithi088.stderr:+++ /usr/libexec/grepconf.sh -c 2024-06-03T08:37:53.672 INFO:tasks.workunit.client.0.smithi088.stderr:+++ '[' -f /usr/libexec/grepconf.sh ']' 2024-06-03T08:37:53.672 INFO:tasks.workunit.client.0.smithi088.stderr:+++ /usr/libexec/grepconf.sh -c 2024-06-03T08:37:53.672 INFO:tasks.workunit.client.0.smithi088.stderr:++ echo api_c_write_operations 2024-06-03T08:37:53.673 INFO:tasks.workunit.client.0.smithi088.stderr:+++ which_declare='declare -f' 2024-06-03T08:37:53.673 INFO:tasks.workunit.client.0.smithi088.stderr:+++ which_opt=-f 2024-06-03T08:37:53.673 INFO:tasks.workunit.client.0.smithi088.stderr:+++ export which_declare 2024-06-03T08:37:53.673 INFO:tasks.workunit.client.0.smithi088.stderr:+++ export -f which 2024-06-03T08:37:53.673 INFO:tasks.workunit.client.0.smithi088.stderr:++ unset i 2024-06-03T08:37:53.673 INFO:tasks.workunit.client.0.smithi088.stderr:++ unset -f pathmunge 2024-06-03T08:37:53.673 INFO:tasks.workunit.client.0.smithi088.stderr:++++ basename /usr/bin/bash 2024-06-03T08:37:53.673 INFO:tasks.workunit.client.0.smithi088.stderr:+ [[ /home/ubuntu/.local/bin:/home/ubuntu/bin:/usr/local/bin:/usr/bin:/usr/local/sbin:/usr/sbin:/usr/sbin =~ /home/ubuntu/\.local/bin:/home/ubuntu/bin: ]] 2024-06-03T08:37:53.673 INFO:tasks.workunit.client.0.smithi088.stderr:+ export PATH 2024-06-03T08:37:53.673 INFO:tasks.workunit.client.0.smithi088.stderr:+ '[' -d /home/ubuntu/.bashrc.d ']' 2024-06-03T08:37:53.673 INFO:tasks.workunit.client.0.smithi088.stderr:+ unset rc 2024-06-03T08:37:53.673 INFO:tasks.workunit.client.0.smithi088.stderr:+++ eval export 'LANG="en_US.UTF-8"' 2024-06-03T08:37:53.673 INFO:tasks.workunit.client.0.smithi088.stderr:++++ export LANG=en_US.UTF-8 2024-06-03T08:37:53.674 INFO:tasks.workunit.client.0.smithi088.stderr:++++ LANG=en_US.UTF-8 2024-06-03T08:37:53.674 INFO:tasks.workunit.client.0.smithi088.stderr:+++ for config in /etc/locale.conf "${HOME}/.i18n" 2024-06-03T08:37:53.674 INFO:tasks.workunit.client.0.smithi088.stderr:+++ '[' -f /home/ubuntu/.i18n ']' 2024-06-03T08:37:53.674 INFO:tasks.workunit.client.0.smithi088.stderr:+++ '[' -n en_US.UTF-8 ']' 2024-06-03T08:37:53.674 INFO:tasks.workunit.client.0.smithi088.stderr:+++ LANG=en_US.UTF-8 2024-06-03T08:37:53.674 INFO:tasks.workunit.client.0.smithi088.stderr:+++ unset LANG_backup config 2024-06-03T08:37:53.674 INFO:tasks.workunit.client.0.smithi088.stderr:+++ '[' -n '' ']' 2024-06-03T08:37:53.674 INFO:tasks.workunit.client.0.smithi088.stderr:+++ '[' -n en_US.UTF-8 ']' 2024-06-03T08:37:53.674 INFO:tasks.workunit.client.0.smithi088.stderr:+++ '[' dumb = linux ']' 2024-06-03T08:37:53.674 INFO:tasks.workunit.client.0.smithi088.stderr:++ for i in /etc/profile.d/*.sh 2024-06-03T08:37:53.674 INFO:tasks.workunit.client.0.smithi088.stderr:++ '[' -r /etc/profile.d/less.sh ']' 2024-06-03T08:37:53.674 INFO:tasks.workunit.client.0.smithi088.stderr:++ '[' '' ']' 2024-06-03T08:37:53.674 INFO:tasks.workunit.client.0.smithi088.stderr:++ . /etc/profile.d/less.sh 2024-06-03T08:37:53.675 INFO:tasks.workunit.client.0.smithi088.stderr:+++ '[' -z '||/usr/bin/lesspipe.sh %s' ']' 2024-06-03T08:37:53.675 INFO:tasks.workunit.client.0.smithi088.stderr:++ for i in /etc/profile.d/*.sh 2024-06-03T08:37:53.675 INFO:tasks.workunit.client.0.smithi088.stderr:++ '[' -r /etc/profile.d/which2.sh ']' 2024-06-03T08:37:53.675 INFO:tasks.workunit.client.0.smithi088.stderr:++ '[' '' ']' 2024-06-03T08:37:53.675 INFO:tasks.workunit.client.0.smithi088.stderr:++ . /etc/profile.d/which2.sh 2024-06-03T08:37:53.675 INFO:tasks.workunit.client.0.smithi088.stderr:+++ case "$(basename $(readlink /proc/$$/exe))" in 2024-06-03T08:37:53.675 INFO:tasks.workunit.client.0.smithi088.stderr:+++ which_declare='declare -f' 2024-06-03T08:37:53.675 INFO:tasks.workunit.client.0.smithi088.stderr:+++ which_opt=-f 2024-06-03T08:37:53.675 INFO:tasks.workunit.client.0.smithi088.stderr:+++ export which_declare 2024-06-03T08:37:53.675 INFO:tasks.workunit.client.0.smithi088.stderr:+++ export -f which 2024-06-03T08:37:53.675 INFO:tasks.workunit.client.0.smithi088.stderr:+ sed 's/^/ api_watch_notify: /' 2024-06-03T08:37:53.675 INFO:tasks.workunit.client.0.smithi088.stderr:++ unset i 2024-06-03T08:37:53.675 INFO:tasks.workunit.client.0.smithi088.stderr:++ unset -f pathmunge 2024-06-03T08:37:53.675 INFO:tasks.workunit.client.0.smithi088.stderr:+ [[ /home/ubuntu/.local/bin:/home/ubuntu/bin:/usr/local/bin:/usr/bin:/usr/local/sbin:/usr/sbin:/usr/sbin =~ /home/ubuntu/\.local/bin:/home/ubuntu/bin: ]] 2024-06-03T08:37:53.676 INFO:tasks.workunit.client.0.smithi088.stderr:+ export PATH 2024-06-03T08:37:53.676 INFO:tasks.workunit.client.0.smithi088.stderr:+++ alias 'grep=grep --color=auto' 2024-06-03T08:37:53.676 INFO:tasks.workunit.client.0.smithi088.stderr:+++ alias 'egrep=egrep --color=auto' 2024-06-03T08:37:53.676 INFO:tasks.workunit.client.0.smithi088.stderr:+ ceph_test_rados_api_watch_notify 2024-06-03T08:37:53.676 INFO:tasks.workunit.client.0.smithi088.stderr:+++ alias 'fgrep=fgrep --color=auto' 2024-06-03T08:37:53.676 INFO:tasks.workunit.client.0.smithi088.stderr:++ for i in /etc/profile.d/*.sh 2024-06-03T08:37:53.676 INFO:tasks.workunit.client.0.smithi088.stderr:++ '[' -r /etc/profile.d/colorls.sh ']' 2024-06-03T08:37:53.676 INFO:tasks.workunit.client.0.smithi088.stderr:+ '[' -d /home/ubuntu/.bashrc.d ']' 2024-06-03T08:37:53.676 INFO:tasks.workunit.client.0.smithi088.stderr:++ '[' '' ']' 2024-06-03T08:37:53.676 INFO:tasks.workunit.client.0.smithi088.stderr:++ . /etc/profile.d/colorls.sh 2024-06-03T08:37:53.676 INFO:tasks.workunit.client.0.smithi088.stderr:+ unset rc 2024-06-03T08:37:53.676 INFO:tasks.workunit.client.0.smithi088.stderr:+++ '[' '!' -t 0 ']' 2024-06-03T08:37:53.676 INFO:tasks.workunit.client.0.smithi088.stderr:+++ return 2024-06-03T08:37:53.677 INFO:tasks.workunit.client.0.smithi088.stderr:+ tee ceph_test_rados_api_watch_notify.log 2024-06-03T08:37:53.677 INFO:tasks.workunit.client.0.smithi088.stderr:++ for i in /etc/profile.d/*.sh 2024-06-03T08:37:53.677 INFO:tasks.workunit.client.0.smithi088.stderr:++ '[' -r /etc/profile.d/colorsysstat.sh ']' 2024-06-03T08:37:53.677 INFO:tasks.workunit.client.0.smithi088.stderr:++ '[' '' ']' 2024-06-03T08:37:53.677 INFO:tasks.workunit.client.0.smithi088.stderr:++ . /etc/profile.d/colorsysstat.sh 2024-06-03T08:37:53.677 INFO:tasks.workunit.client.0.smithi088.stderr:+++ export S_COLORS=auto 2024-06-03T08:37:53.677 INFO:tasks.workunit.client.0.smithi088.stderr:+++ S_COLORS=auto 2024-06-03T08:37:53.677 INFO:tasks.workunit.client.0.smithi088.stderr:++ for i in /etc/profile.d/*.sh 2024-06-03T08:37:53.677 INFO:tasks.workunit.client.0.smithi088.stderr:++ '[' -r /etc/profile.d/colorxzgrep.sh ']' 2024-06-03T08:37:53.677 INFO:tasks.workunit.client.0.smithi088.stderr:++ '[' '' ']' 2024-06-03T08:37:53.677 INFO:tasks.workunit.client.0.smithi088.stderr:++ . /etc/profile.d/colorxzgrep.sh 2024-06-03T08:37:53.677 INFO:tasks.workunit.client.0.smithi088.stderr:+++ /usr/libexec/grepconf.sh -c 2024-06-03T08:37:53.677 INFO:tasks.workunit.client.0.smithi088.stderr:+ ceph_test_rados_api_stat_pp 2024-06-03T08:37:53.678 INFO:tasks.workunit.client.0.smithi088.stderr:+ tee ceph_test_rados_api_stat_pp.log 2024-06-03T08:37:53.678 INFO:tasks.workunit.client.0.smithi088.stderr:+ sed 's/^/ api_stat_pp: /' 2024-06-03T08:37:53.678 INFO:tasks.workunit.client.0.smithi088.stderr:+++ alias 'zgrep=zgrep --color=auto' 2024-06-03T08:37:53.678 INFO:tasks.workunit.client.0.smithi088.stderr:+++ alias 'zfgrep=zfgrep --color=auto' 2024-06-03T08:37:53.678 INFO:tasks.workunit.client.0.smithi088.stderr:+++ alias 'zegrep=zegrep --color=auto' 2024-06-03T08:37:53.678 INFO:tasks.workunit.client.0.smithi088.stderr:++ for i in /etc/profile.d/*.sh 2024-06-03T08:37:53.678 INFO:tasks.workunit.client.0.smithi088.stderr:++ '[' -r /etc/profile.d/debuginfod.sh ']' 2024-06-03T08:37:53.678 INFO:tasks.workunit.client.0.smithi088.stderr:++ '[' '' ']' 2024-06-03T08:37:53.678 INFO:tasks.workunit.client.0.smithi088.stderr:++ . /etc/profile.d/debuginfod.sh 2024-06-03T08:37:53.678 INFO:tasks.workunit.client.0.smithi088.stderr:+++ '[' -z 'https://debuginfod.centos.org/ ' ']' 2024-06-03T08:37:53.678 INFO:tasks.workunit.client.0.smithi088.stderr:++ for i in /etc/profile.d/*.sh 2024-06-03T08:37:53.678 INFO:tasks.workunit.client.0.smithi088.stderr:++ '[' -r /etc/profile.d/gawk.sh ']' 2024-06-03T08:37:53.678 INFO:tasks.workunit.client.0.smithi088.stderr:++ '[' '' ']' 2024-06-03T08:37:53.679 INFO:tasks.workunit.client.0.smithi088.stderr:++ . /etc/profile.d/gawk.sh 2024-06-03T08:37:53.679 INFO:tasks.workunit.client.0.smithi088.stderr:+++ alias 'zgrep=zgrep --color=auto' 2024-06-03T08:37:53.679 INFO:tasks.workunit.client.0.smithi088.stderr:++ for i in /etc/profile.d/*.sh 2024-06-03T08:37:53.679 INFO:tasks.workunit.client.0.smithi088.stderr:+++ alias 'zfgrep=zfgrep --color=auto' 2024-06-03T08:37:53.679 INFO:tasks.workunit.client.0.smithi088.stderr:++ '[' -r /etc/profile.d/lang.sh ']' 2024-06-03T08:37:53.679 INFO:tasks.workunit.client.0.smithi088.stderr:++ '[' '' ']' 2024-06-03T08:37:53.679 INFO:tasks.workunit.client.0.smithi088.stderr:+++ alias 'zegrep=zegrep --color=auto' 2024-06-03T08:37:53.679 INFO:tasks.workunit.client.0.smithi088.stderr:++ . /etc/profile.d/lang.sh 2024-06-03T08:37:53.679 INFO:tasks.workunit.client.0.smithi088.stderr:++ for i in /etc/profile.d/*.sh 2024-06-03T08:37:53.679 INFO:tasks.workunit.client.0.smithi088.stderr:+++ unset LANG_backup 2024-06-03T08:37:53.679 INFO:tasks.workunit.client.0.smithi088.stderr:++ '[' -r /etc/profile.d/debuginfod.sh ']' 2024-06-03T08:37:53.679 INFO:tasks.workunit.client.0.smithi088.stderr:++ '[' '' ']' 2024-06-03T08:37:53.679 INFO:tasks.workunit.client.0.smithi088.stderr:++ . /etc/profile.d/debuginfod.sh 2024-06-03T08:37:53.679 INFO:tasks.workunit.client.0.smithi088.stderr:+++ '[' -z 'https://debuginfod.centos.org/ ' ']' 2024-06-03T08:37:53.680 INFO:tasks.workunit.client.0.smithi088.stderr:++ for i in /etc/profile.d/*.sh 2024-06-03T08:37:53.680 INFO:tasks.workunit.client.0.smithi088.stderr:++ '[' -r /etc/profile.d/gawk.sh ']' 2024-06-03T08:37:53.680 INFO:tasks.workunit.client.0.smithi088.stderr:++ '[' '' ']' 2024-06-03T08:37:53.680 INFO:tasks.workunit.client.0.smithi088.stderr:++ . /etc/profile.d/gawk.sh 2024-06-03T08:37:53.680 INFO:tasks.workunit.client.0.smithi088.stderr:++ for i in /etc/profile.d/*.sh 2024-06-03T08:37:53.680 INFO:tasks.workunit.client.0.smithi088.stderr:++ '[' -r /etc/profile.d/lang.sh ']' 2024-06-03T08:37:53.680 INFO:tasks.workunit.client.0.smithi088.stderr:++ '[' '' ']' 2024-06-03T08:37:53.680 INFO:tasks.workunit.client.0.smithi088.stderr:++ . /etc/profile.d/lang.sh 2024-06-03T08:37:53.680 INFO:tasks.workunit.client.0.smithi088.stderr:+++ unset LANG_backup 2024-06-03T08:37:53.680 INFO:tasks.workunit.client.0.smithi088.stderr:+++++ readlink /proc/40610/exe 2024-06-03T08:37:53.680 INFO:tasks.workunit.client.0.smithi088.stderr:+++ alias 'grep=grep --color=auto' 2024-06-03T08:37:53.680 INFO:tasks.workunit.client.0.smithi088.stderr:+++ alias 'egrep=egrep --color=auto' 2024-06-03T08:37:53.680 INFO:tasks.workunit.client.0.smithi088.stderr:+++ alias 'fgrep=fgrep --color=auto' 2024-06-03T08:37:53.681 INFO:tasks.workunit.client.0.smithi088.stderr:++ for i in /etc/profile.d/*.sh 2024-06-03T08:37:53.681 INFO:tasks.workunit.client.0.smithi088.stderr:+ ff=api_c_write_operations 2024-06-03T08:37:53.681 INFO:tasks.workunit.client.0.smithi088.stderr:++ '[' -r /etc/profile.d/colorls.sh ']' 2024-06-03T08:37:53.681 INFO:tasks.workunit.client.0.smithi088.stderr:++ '[' '' ']' 2024-06-03T08:37:53.681 INFO:tasks.workunit.client.0.smithi088.stderr:++ . /etc/profile.d/colorls.sh 2024-06-03T08:37:53.681 INFO:tasks.workunit.client.0.smithi088.stderr:+++ '[' '!' -t 0 ']' 2024-06-03T08:37:53.681 INFO:tasks.workunit.client.0.smithi088.stderr:+++ return 2024-06-03T08:37:53.681 INFO:tasks.workunit.client.0.smithi088.stderr:++++ /usr/bin/locale 2024-06-03T08:37:53.681 INFO:tasks.workunit.client.0.smithi088.stderr:++ for i in /etc/profile.d/*.sh 2024-06-03T08:37:53.681 INFO:tasks.workunit.client.0.smithi088.stderr:++ '[' -r /etc/profile.d/colorsysstat.sh ']' 2024-06-03T08:37:53.681 INFO:tasks.workunit.client.0.smithi088.stderr:++ '[' '' ']' 2024-06-03T08:37:53.681 INFO:tasks.workunit.client.0.smithi088.stderr:++ . /etc/profile.d/colorsysstat.sh 2024-06-03T08:37:53.681 INFO:tasks.workunit.client.0.smithi088.stderr:+++ export S_COLORS=auto 2024-06-03T08:37:53.681 INFO:tasks.workunit.client.0.smithi088.stderr:+ pid=40764 2024-06-03T08:37:53.682 INFO:tasks.workunit.client.0.smithi088.stdout:test api_c_write_operations on pid 40764 2024-06-03T08:37:53.695 INFO:tasks.workunit.client.0.smithi088.stderr:+ echo 'test api_c_write_operations on pid 40764' 2024-06-03T08:37:53.695 INFO:tasks.workunit.client.0.smithi088.stderr:+ pids[$f]=40764 2024-06-03T08:37:53.695 INFO:tasks.workunit.client.0.smithi088.stderr:+ for f in api_aio api_aio_pp api_io api_io_pp api_asio api_list api_lock api_lock_pp api_misc api_misc_pp api_tier_pp api_pool api_snapshots api_snapshots_pp api_stat api_stat_pp api_watch_notify api_watch_notify_pp api_cmd api_cmd_pp api_service api_service_pp api_c_write_operations api_c_read_operations api_cls_remote_reads list_parallel open_pools_parallel delete_pools_parallel 2024-06-03T08:37:53.695 INFO:tasks.workunit.client.0.smithi088.stderr:+ '[' 1 -eq 1 ']' 2024-06-03T08:37:53.695 INFO:tasks.workunit.client.0.smithi088.stderr:+++ S_COLORS=auto 2024-06-03T08:37:53.695 INFO:tasks.workunit.client.0.smithi088.stderr:++ for i in /etc/profile.d/*.sh 2024-06-03T08:37:53.696 INFO:tasks.workunit.client.0.smithi088.stderr:++ '[' -r /etc/profile.d/colorxzgrep.sh ']' 2024-06-03T08:37:53.696 INFO:tasks.workunit.client.0.smithi088.stderr:++ '[' '' ']' 2024-06-03T08:37:53.696 INFO:tasks.workunit.client.0.smithi088.stderr:+ bash -o pipefail -exc 'ceph_test_rados_api_c_write_operations 2>&1 | tee ceph_test_rados_api_c_write_operations.log | sed "s/^/ api_c_write_operations: /"' 2024-06-03T08:37:53.696 INFO:tasks.workunit.client.0.smithi088.stderr:++ . /etc/profile.d/colorxzgrep.sh 2024-06-03T08:37:53.696 INFO:tasks.workunit.client.0.smithi088.stderr:+++ /usr/libexec/grepconf.sh -c 2024-06-03T08:37:53.696 INFO:tasks.workunit.client.0.smithi088.stderr:+++ alias 'xzgrep=xzgrep --color=auto' 2024-06-03T08:37:53.696 INFO:tasks.workunit.client.0.smithi088.stderr:+++ alias 'xzegrep=xzegrep --color=auto' 2024-06-03T08:37:53.696 INFO:tasks.workunit.client.0.smithi088.stderr:+++ alias 'xzfgrep=xzfgrep --color=auto' 2024-06-03T08:37:53.696 INFO:tasks.workunit.client.0.smithi088.stderr:++ for i in /etc/profile.d/*.sh 2024-06-03T08:37:53.696 INFO:tasks.workunit.client.0.smithi088.stderr:++ '[' -r /etc/profile.d/colorzgrep.sh ']' 2024-06-03T08:37:53.696 INFO:tasks.workunit.client.0.smithi088.stderr:++ '[' '' ']' 2024-06-03T08:37:53.696 INFO:tasks.workunit.client.0.smithi088.stderr:++ . /etc/profile.d/colorzgrep.sh 2024-06-03T08:37:53.697 INFO:tasks.workunit.client.0.smithi088.stderr:+++ '[' -f /usr/libexec/grepconf.sh ']' 2024-06-03T08:37:53.697 INFO:tasks.workunit.client.0.smithi088.stderr:+++ /usr/libexec/grepconf.sh -c 2024-06-03T08:37:53.697 INFO:tasks.workunit.client.0.smithi088.stderr:++++ basename /usr/bin/bash 2024-06-03T08:37:53.697 INFO:tasks.workunit.client.0.smithi088.stderr:+++ which_declare='declare -f' 2024-06-03T08:37:53.697 INFO:tasks.workunit.client.0.smithi088.stderr:+++ which_opt=-f 2024-06-03T08:37:53.697 INFO:tasks.workunit.client.0.smithi088.stderr:+++ export which_declare 2024-06-03T08:37:53.697 INFO:tasks.workunit.client.0.smithi088.stderr:+++ export -f which 2024-06-03T08:37:53.697 INFO:tasks.workunit.client.0.smithi088.stderr:++ unset i 2024-06-03T08:37:53.697 INFO:tasks.workunit.client.0.smithi088.stderr:++ unset -f pathmunge 2024-06-03T08:37:53.697 INFO:tasks.workunit.client.0.smithi088.stderr:+ '[' -f /etc/bashrc ']' 2024-06-03T08:37:53.697 INFO:tasks.workunit.client.0.smithi088.stderr:+ . /etc/bashrc 2024-06-03T08:37:53.697 INFO:tasks.workunit.client.0.smithi088.stderr:++++ /usr/bin/locale 2024-06-03T08:37:53.697 INFO:tasks.workunit.client.0.smithi088.stderr:+ [[ /home/ubuntu/.local/bin:/home/ubuntu/bin:/usr/local/bin:/usr/bin:/usr/local/sbin:/usr/sbin:/usr/sbin =~ /home/ubuntu/\.local/bin:/home/ubuntu/bin: ]] 2024-06-03T08:37:53.697 INFO:tasks.workunit.client.0.smithi088.stderr:++ '[' -z '' ']' 2024-06-03T08:37:53.698 INFO:tasks.workunit.client.0.smithi088.stderr:++ BASHRCSOURCED=Y 2024-06-03T08:37:53.698 INFO:tasks.workunit.client.0.smithi088.stderr:++ '[' '' ']' 2024-06-03T08:37:53.698 INFO:tasks.workunit.client.0.smithi088.stderr:++ shopt -q login_shell 2024-06-03T08:37:53.698 INFO:tasks.workunit.client.0.smithi088.stderr:+ export PATH 2024-06-03T08:37:53.698 INFO:tasks.workunit.client.0.smithi088.stderr:+ '[' -d /home/ubuntu/.bashrc.d ']' 2024-06-03T08:37:53.698 INFO:tasks.workunit.client.0.smithi088.stderr:+ unset rc 2024-06-03T08:37:53.698 INFO:tasks.workunit.client.0.smithi088.stderr:++ printf %25s api_c_read_operations 2024-06-03T08:37:53.698 INFO:tasks.workunit.client.0.smithi088.stderr:+ ceph_test_rados_api_watch_notify_pp 2024-06-03T08:37:53.698 INFO:tasks.workunit.client.0.smithi088.stderr:+++ '[' -n '' ']' 2024-06-03T08:37:53.698 INFO:tasks.workunit.client.0.smithi088.stderr:+++ LANG_backup=en_US.UTF-8 2024-06-03T08:37:53.698 INFO:tasks.workunit.client.0.smithi088.stderr:+ r=' api_c_read_operations' 2024-06-03T08:37:53.698 INFO:tasks.workunit.client.0.smithi088.stderr:+++ for config in /etc/locale.conf "${HOME}/.i18n" 2024-06-03T08:37:53.698 INFO:tasks.workunit.client.0.smithi088.stderr:+++ '[' -f /etc/locale.conf ']' 2024-06-03T08:37:53.699 INFO:tasks.workunit.client.0.smithi088.stderr:+++ '[' -x /usr/bin/sed ']' 2024-06-03T08:37:53.699 INFO:tasks.workunit.client.0.smithi088.stderr:+++ alias 'xzgrep=xzgrep --color=auto' 2024-06-03T08:37:53.699 INFO:tasks.workunit.client.0.smithi088.stderr:+++ alias 'xzegrep=xzegrep --color=auto' 2024-06-03T08:37:53.699 INFO:tasks.workunit.client.0.smithi088.stderr:+++ alias 'xzfgrep=xzfgrep --color=auto' 2024-06-03T08:37:53.699 INFO:tasks.workunit.client.0.smithi088.stderr:++ for i in /etc/profile.d/*.sh 2024-06-03T08:37:53.699 INFO:tasks.workunit.client.0.smithi088.stderr:++ '[' -r /etc/profile.d/colorzgrep.sh ']' 2024-06-03T08:37:53.699 INFO:tasks.workunit.client.0.smithi088.stderr:++ '[' '' ']' 2024-06-03T08:37:53.699 INFO:tasks.workunit.client.0.smithi088.stderr:++ . /etc/profile.d/colorzgrep.sh 2024-06-03T08:37:53.699 INFO:tasks.workunit.client.0.smithi088.stderr:++++ /usr/bin/sed -r -e 's/^[[:blank:]]*([[:upper:]_]+)=([[:print:][:digit:]\._-]+|"[[:print:][:digit:]\._-]+")/export \1=\2/;t;d' /etc/locale.conf 2024-06-03T08:37:53.699 INFO:tasks.workunit.client.0.smithi088.stderr:+++ '[' -f /usr/libexec/grepconf.sh ']' 2024-06-03T08:37:53.699 INFO:tasks.workunit.client.0.smithi088.stderr:+++ /usr/libexec/grepconf.sh -c 2024-06-03T08:37:53.699 INFO:tasks.workunit.client.0.smithi088.stderr:+++ umask 2024-06-03T08:37:53.699 INFO:tasks.workunit.client.0.smithi088.stderr:+ tee ceph_test_rados_api_watch_notify_pp.log 2024-06-03T08:37:53.700 INFO:tasks.workunit.client.0.smithi088.stderr:++ '[' 0022 -eq 0 ']' 2024-06-03T08:37:53.700 INFO:tasks.workunit.client.0.smithi088.stderr:++ SHELL=/bin/bash 2024-06-03T08:37:53.700 INFO:tasks.workunit.client.0.smithi088.stderr:++ for i in /etc/profile.d/*.sh 2024-06-03T08:37:53.700 INFO:tasks.workunit.client.0.smithi088.stderr:++ '[' -r /etc/profile.d/bash_completion.sh ']' 2024-06-03T08:37:53.700 INFO:tasks.workunit.client.0.smithi088.stderr:++ '[' '' ']' 2024-06-03T08:37:53.700 INFO:tasks.workunit.client.0.smithi088.stderr:++ . /etc/profile.d/bash_completion.sh 2024-06-03T08:37:53.700 INFO:tasks.workunit.client.0.smithi088.stderr:+++ '[' 'x5.1.8(1)-release' '!=' x -a x '!=' x -a x = x ']' 2024-06-03T08:37:53.700 INFO:tasks.workunit.client.0.smithi088.stderr:++ for i in /etc/profile.d/*.sh 2024-06-03T08:37:53.700 INFO:tasks.workunit.client.0.smithi088.stderr:++ '[' -r /etc/profile.d/cobbler.sh ']' 2024-06-03T08:37:53.700 INFO:tasks.workunit.client.0.smithi088.stderr:++ '[' '' ']' 2024-06-03T08:37:53.700 INFO:tasks.workunit.client.0.smithi088.stderr:+ sed 's/^/ api_watch_notify_pp: /' 2024-06-03T08:37:53.700 INFO:tasks.workunit.client.0.smithi088.stderr:++ . /etc/profile.d/cobbler.sh 2024-06-03T08:37:53.700 INFO:tasks.workunit.client.0.smithi088.stderr:+++ export COBBLER_SERVER=172.21.0.11 2024-06-03T08:37:53.700 INFO:tasks.workunit.client.0.smithi088.stderr:+++ COBBLER_SERVER=172.21.0.11 2024-06-03T08:37:53.701 INFO:tasks.workunit.client.0.smithi088.stderr:++ for i in /etc/profile.d/*.sh 2024-06-03T08:37:53.701 INFO:tasks.workunit.client.0.smithi088.stderr:++ '[' -r /etc/profile.d/colorgrep.sh ']' 2024-06-03T08:37:53.701 INFO:tasks.workunit.client.0.smithi088.stderr:++ '[' '' ']' 2024-06-03T08:37:53.701 INFO:tasks.workunit.client.0.smithi088.stderr:++ . /etc/profile.d/colorgrep.sh 2024-06-03T08:37:53.701 INFO:tasks.workunit.client.0.smithi088.stderr:+++ /usr/libexec/grepconf.sh -c 2024-06-03T08:37:53.701 INFO:tasks.workunit.client.0.smithi088.stderr:++ echo api_c_read_operations 2024-06-03T08:37:53.701 INFO:tasks.workunit.client.0.smithi088.stderr:++ awk '{print $1}' 2024-06-03T08:37:53.701 INFO:tasks.workunit.client.0.smithi088.stderr:+++ eval export 'LANG="en_US.UTF-8"' 2024-06-03T08:37:53.701 INFO:tasks.workunit.client.0.smithi088.stderr:+++ '[' -n '' ']' 2024-06-03T08:37:53.701 INFO:tasks.workunit.client.0.smithi088.stderr:++++ export LANG=en_US.UTF-8 2024-06-03T08:37:53.701 INFO:tasks.workunit.client.0.smithi088.stderr:++++ LANG=en_US.UTF-8 2024-06-03T08:37:53.701 INFO:tasks.workunit.client.0.smithi088.stderr:+++ LANG_backup=en_US.UTF-8 2024-06-03T08:37:53.701 INFO:tasks.workunit.client.0.smithi088.stderr:+++ for config in /etc/locale.conf "${HOME}/.i18n" 2024-06-03T08:37:53.701 INFO:tasks.workunit.client.0.smithi088.stderr:+++ '[' -f /home/ubuntu/.i18n ']' 2024-06-03T08:37:53.702 INFO:tasks.workunit.client.0.smithi088.stderr:+++ for config in /etc/locale.conf "${HOME}/.i18n" 2024-06-03T08:37:53.702 INFO:tasks.workunit.client.0.smithi088.stderr:+++ '[' -f /etc/locale.conf ']' 2024-06-03T08:37:53.702 INFO:tasks.workunit.client.0.smithi088.stderr:+++ '[' -n en_US.UTF-8 ']' 2024-06-03T08:37:53.702 INFO:tasks.workunit.client.0.smithi088.stderr:+++ '[' -x /usr/bin/sed ']' 2024-06-03T08:37:53.702 INFO:tasks.workunit.client.0.smithi088.stderr:+++ LANG=en_US.UTF-8 2024-06-03T08:37:53.702 INFO:tasks.workunit.client.0.smithi088.stderr:+++ unset LANG_backup config 2024-06-03T08:37:53.702 INFO:tasks.workunit.client.0.smithi088.stderr:+++ '[' -n '' ']' 2024-06-03T08:37:53.702 INFO:tasks.workunit.client.0.smithi088.stderr:+++ '[' -n en_US.UTF-8 ']' 2024-06-03T08:37:53.702 INFO:tasks.workunit.client.0.smithi088.stderr:+++ '[' dumb = linux ']' 2024-06-03T08:37:53.702 INFO:tasks.workunit.client.0.smithi088.stderr:++ for i in /etc/profile.d/*.sh 2024-06-03T08:37:53.702 INFO:tasks.workunit.client.0.smithi088.stderr:++ '[' -r /etc/profile.d/less.sh ']' 2024-06-03T08:37:53.702 INFO:tasks.workunit.client.0.smithi088.stderr:++ '[' '' ']' 2024-06-03T08:37:53.702 INFO:tasks.workunit.client.0.smithi088.stderr:++ . /etc/profile.d/less.sh 2024-06-03T08:37:53.703 INFO:tasks.workunit.client.0.smithi088.stderr:+++ '[' -z '||/usr/bin/lesspipe.sh %s' ']' 2024-06-03T08:37:53.703 INFO:tasks.workunit.client.0.smithi088.stderr:++ for i in /etc/profile.d/*.sh 2024-06-03T08:37:53.703 INFO:tasks.workunit.client.0.smithi088.stderr:++ '[' -r /etc/profile.d/which2.sh ']' 2024-06-03T08:37:53.703 INFO:tasks.workunit.client.0.smithi088.stderr:++ '[' '' ']' 2024-06-03T08:37:53.703 INFO:tasks.workunit.client.0.smithi088.stderr:++ . /etc/profile.d/which2.sh 2024-06-03T08:37:53.703 INFO:tasks.workunit.client.0.smithi088.stderr:+++ case "$(basename $(readlink /proc/$$/exe))" in 2024-06-03T08:37:53.703 INFO:tasks.workunit.client.0.smithi088.stderr:+++ alias 'zgrep=zgrep --color=auto' 2024-06-03T08:37:53.703 INFO:tasks.workunit.client.0.smithi088.stderr:+++ alias 'zfgrep=zfgrep --color=auto' 2024-06-03T08:37:53.703 INFO:tasks.workunit.client.0.smithi088.stderr:+++ alias 'zegrep=zegrep --color=auto' 2024-06-03T08:37:53.703 INFO:tasks.workunit.client.0.smithi088.stderr:++ for i in /etc/profile.d/*.sh 2024-06-03T08:37:53.703 INFO:tasks.workunit.client.0.smithi088.stderr:++++ /usr/bin/sed -r -e 's/^[[:blank:]]*([[:upper:]_]+)=([[:print:][:digit:]\._-]+|"[[:print:][:digit:]\._-]+")/export \1=\2/;t;d' /etc/locale.conf 2024-06-03T08:37:53.703 INFO:tasks.workunit.client.0.smithi088.stderr:++ '[' -r /etc/profile.d/debuginfod.sh ']' 2024-06-03T08:37:53.703 INFO:tasks.workunit.client.0.smithi088.stderr:++ '[' '' ']' 2024-06-03T08:37:53.704 INFO:tasks.workunit.client.0.smithi088.stderr:++ . /etc/profile.d/debuginfod.sh 2024-06-03T08:37:53.704 INFO:tasks.workunit.client.0.smithi088.stderr:+++ '[' -z 'https://debuginfod.centos.org/ ' ']' 2024-06-03T08:37:53.704 INFO:tasks.workunit.client.0.smithi088.stderr:++ for i in /etc/profile.d/*.sh 2024-06-03T08:37:53.704 INFO:tasks.workunit.client.0.smithi088.stderr:++ '[' -r /etc/profile.d/gawk.sh ']' 2024-06-03T08:37:53.704 INFO:tasks.workunit.client.0.smithi088.stderr:++ '[' '' ']' 2024-06-03T08:37:53.704 INFO:tasks.workunit.client.0.smithi088.stderr:++ . /etc/profile.d/gawk.sh 2024-06-03T08:37:53.704 INFO:tasks.workunit.client.0.smithi088.stderr:++ for i in /etc/profile.d/*.sh 2024-06-03T08:37:53.704 INFO:tasks.workunit.client.0.smithi088.stderr:++ '[' -r /etc/profile.d/lang.sh ']' 2024-06-03T08:37:53.704 INFO:tasks.workunit.client.0.smithi088.stderr:++ '[' '' ']' 2024-06-03T08:37:53.704 INFO:tasks.workunit.client.0.smithi088.stderr:++ . /etc/profile.d/lang.sh 2024-06-03T08:37:53.704 INFO:tasks.workunit.client.0.smithi088.stderr:+++ unset LANG_backup 2024-06-03T08:37:53.704 INFO:tasks.workunit.client.0.smithi088.stderr:++++ /usr/bin/locale 2024-06-03T08:37:53.704 INFO:tasks.workunit.client.0.smithi088.stderr:+++ eval export 'LANG="en_US.UTF-8"' 2024-06-03T08:37:53.704 INFO:tasks.workunit.client.0.smithi088.stderr:++++ export LANG=en_US.UTF-8 2024-06-03T08:37:53.705 INFO:tasks.workunit.client.0.smithi088.stderr:++++ LANG=en_US.UTF-8 2024-06-03T08:37:53.705 INFO:tasks.workunit.client.0.smithi088.stderr:+++ for config in /etc/locale.conf "${HOME}/.i18n" 2024-06-03T08:37:53.705 INFO:tasks.workunit.client.0.smithi088.stderr:+++ '[' -f /home/ubuntu/.i18n ']' 2024-06-03T08:37:53.705 INFO:tasks.workunit.client.0.smithi088.stderr:+++ '[' -n en_US.UTF-8 ']' 2024-06-03T08:37:53.705 INFO:tasks.workunit.client.0.smithi088.stderr:+++ LANG=en_US.UTF-8 2024-06-03T08:37:53.705 INFO:tasks.workunit.client.0.smithi088.stderr:+++ unset LANG_backup config 2024-06-03T08:37:53.705 INFO:tasks.workunit.client.0.smithi088.stderr:+++ '[' -n '' ']' 2024-06-03T08:37:53.705 INFO:tasks.workunit.client.0.smithi088.stderr:+++ '[' -n '' ']' 2024-06-03T08:37:53.705 INFO:tasks.workunit.client.0.smithi088.stderr:+++ LANG_backup=en_US.UTF-8 2024-06-03T08:37:53.705 INFO:tasks.workunit.client.0.smithi088.stderr:+++ '[' -n en_US.UTF-8 ']' 2024-06-03T08:37:53.705 INFO:tasks.workunit.client.0.smithi088.stderr:+++ '[' dumb = linux ']' 2024-06-03T08:37:53.705 INFO:tasks.workunit.client.0.smithi088.stderr:++ for i in /etc/profile.d/*.sh 2024-06-03T08:37:53.705 INFO:tasks.workunit.client.0.smithi088.stderr:+++ for config in /etc/locale.conf "${HOME}/.i18n" 2024-06-03T08:37:53.705 INFO:tasks.workunit.client.0.smithi088.stderr:++ '[' -r /etc/profile.d/less.sh ']' 2024-06-03T08:37:53.706 INFO:tasks.workunit.client.0.smithi088.stderr:++ '[' '' ']' 2024-06-03T08:37:53.706 INFO:tasks.workunit.client.0.smithi088.stderr:++ . /etc/profile.d/less.sh 2024-06-03T08:37:53.706 INFO:tasks.workunit.client.0.smithi088.stderr:+++ '[' -f /etc/locale.conf ']' 2024-06-03T08:37:53.706 INFO:tasks.workunit.client.0.smithi088.stderr:+++ '[' -z '||/usr/bin/lesspipe.sh %s' ']' 2024-06-03T08:37:53.706 INFO:tasks.workunit.client.0.smithi088.stderr:++ for i in /etc/profile.d/*.sh 2024-06-03T08:37:53.706 INFO:tasks.workunit.client.0.smithi088.stderr:++ '[' -r /etc/profile.d/which2.sh ']' 2024-06-03T08:37:53.706 INFO:tasks.workunit.client.0.smithi088.stderr:++ '[' '' ']' 2024-06-03T08:37:53.706 INFO:tasks.workunit.client.0.smithi088.stderr:++ . /etc/profile.d/which2.sh 2024-06-03T08:37:53.706 INFO:tasks.workunit.client.0.smithi088.stderr:+++ '[' -x /usr/bin/sed ']' 2024-06-03T08:37:53.706 INFO:tasks.workunit.client.0.smithi088.stderr:+++ case "$(basename $(readlink /proc/$$/exe))" in 2024-06-03T08:37:53.706 INFO:tasks.workunit.client.0.smithi088.stderr:++++ /usr/bin/sed -r -e 's/^[[:blank:]]*([[:upper:]_]+)=([[:print:][:digit:]\._-]+|"[[:print:][:digit:]\._-]+")/export \1=\2/;t;d' /etc/locale.conf 2024-06-03T08:37:53.706 INFO:tasks.workunit.client.0.smithi088.stderr:+++++ readlink /proc/40634/exe 2024-06-03T08:37:53.707 INFO:tasks.workunit.client.0.smithi088.stderr:++++ basename /usr/bin/bash 2024-06-03T08:37:53.707 INFO:tasks.workunit.client.0.smithi088.stderr:+++ eval export 'LANG="en_US.UTF-8"' 2024-06-03T08:37:53.707 INFO:tasks.workunit.client.0.smithi088.stderr:++++ export LANG=en_US.UTF-8 2024-06-03T08:37:53.707 INFO:tasks.workunit.client.0.smithi088.stderr:++++ LANG=en_US.UTF-8 2024-06-03T08:37:53.707 INFO:tasks.workunit.client.0.smithi088.stderr:+++ for config in /etc/locale.conf "${HOME}/.i18n" 2024-06-03T08:37:53.707 INFO:tasks.workunit.client.0.smithi088.stderr:+++ '[' -f /home/ubuntu/.i18n ']' 2024-06-03T08:37:53.707 INFO:tasks.workunit.client.0.smithi088.stderr:+++ '[' -n en_US.UTF-8 ']' 2024-06-03T08:37:53.707 INFO:tasks.workunit.client.0.smithi088.stderr:+++ LANG=en_US.UTF-8 2024-06-03T08:37:53.707 INFO:tasks.workunit.client.0.smithi088.stderr:+++ unset LANG_backup config 2024-06-03T08:37:53.707 INFO:tasks.workunit.client.0.smithi088.stderr:+++ '[' -n '' ']' 2024-06-03T08:37:53.707 INFO:tasks.workunit.client.0.smithi088.stderr:+++ '[' -n en_US.UTF-8 ']' 2024-06-03T08:37:53.707 INFO:tasks.workunit.client.0.smithi088.stderr:+++ '[' dumb = linux ']' 2024-06-03T08:37:53.707 INFO:tasks.workunit.client.0.smithi088.stderr:++ for i in /etc/profile.d/*.sh 2024-06-03T08:37:53.707 INFO:tasks.workunit.client.0.smithi088.stderr:++ '[' -r /etc/profile.d/less.sh ']' 2024-06-03T08:37:53.708 INFO:tasks.workunit.client.0.smithi088.stderr:++ '[' '' ']' 2024-06-03T08:37:53.708 INFO:tasks.workunit.client.0.smithi088.stderr:++ . /etc/profile.d/less.sh 2024-06-03T08:37:53.708 INFO:tasks.workunit.client.0.smithi088.stderr:+++ '[' -z '||/usr/bin/lesspipe.sh %s' ']' 2024-06-03T08:37:53.708 INFO:tasks.workunit.client.0.smithi088.stderr:++ for i in /etc/profile.d/*.sh 2024-06-03T08:37:53.708 INFO:tasks.workunit.client.0.smithi088.stderr:++ '[' -r /etc/profile.d/which2.sh ']' 2024-06-03T08:37:53.708 INFO:tasks.workunit.client.0.smithi088.stderr:++ '[' '' ']' 2024-06-03T08:37:53.708 INFO:tasks.workunit.client.0.smithi088.stderr:++ . /etc/profile.d/which2.sh 2024-06-03T08:37:53.708 INFO:tasks.workunit.client.0.smithi088.stdout:2.21.15.88:6790/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f5138001e10 con 0x56192203cd80 2024-06-03T08:37:53.708 INFO:tasks.workunit.client.0.smithi088.stdout: api_asio: 2024-06-03T08:37:53.517+0000 7f5149181640 1 -- 172.21.15.88:0/273312373 <== mon.1 v1:172.21.15.192:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 1012820588 0 0) 0x561921f51050 con 0x56192203a160 2024-06-03T08:37:53.708 INFO:tasks.workunit.client.0.smithi088.stdout: api_asio: 2024-06-03T08:37:53.517+0000 7f5149181640 10 monclient(hunting): _negotiate 2024-06-03T08:37:53.709 INFO:tasks.workunit.client.0.smithi088.stdout: api_asio: 2024-06-03T08:37:53.517+0000 7f5149181640 10 monclient(hunting): _init_auth method 2 2024-06-03T08:37:53.709 INFO:tasks.workunit.client.0.smithi088.stdout: api_asio: 2024-06-03T08:37:53.517+0000 7f5149181640 10 monclient(hunting): _init_auth creating new auth 2024-06-03T08:37:53.709 INFO:tasks.workunit.client.0.smithi088.stdout: api_asio: 2024-06-03T08:37:53.517+0000 7f5149181640 10 monclient(hunting): my global_id is 4438 2024-06-03T08:37:53.709 INFO:tasks.workunit.client.0.smithi088.stdout: api_asio: 2024-06-03T08:37:53.517+0000 7f5149181640 1 -- 172.21.15.88:0/273312373 --> v1:172.21.15.192:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x56192203eff0 con 0x56192203a160 2024-06-03T08:37:53.709 INFO:tasks.workunit.client.0.smithi088.stdout: api_asio: 2024-06-03T08:37:53.539+0000 7f5149181640 1 -- 172.21.15.88:0/273312373 <== mon.2 v1:172.21.15.88:6790/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 214+0+0 (unknown 3726647304 0 0) 0x7f5138001e10 con 0x56192203cd80 2024-06-03T08:37:53.709 INFO:tasks.workunit.client.0.smithi088.stdout: api_asio: 2024-06-03T08:37:53.539+0000 7f5149181640 1 -- 172.21.15.88:0/273312373 >> v1:172.21.15.192:6789/0 conn(0x56192203a160 legacy=0x5619220afb50 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-06-03T08:37:53.709 INFO:tasks.workunit.client.0.smithi088.stdout: api_asio: 2024-06-03T08:37:53.539+0000 7f5149181640 1 -- 172.21.15.88:0/273312373 >> v1:172.21.15.88:6789/0 conn(0x5619220ccfb0 legacy=0x56192216beb0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-06-03T08:37:53.709 INFO:tasks.workunit.client.0.smithi088.stdout: api_asio: 2024-06-03T08:37:53.539+0000 7f5149181640 10 monclient: _finish_hunting 0 2024-06-03T08:37:53.709 INFO:tasks.workunit.client.0.smithi088.stdout: api_asio: 2024-06-03T08:37:53.539+0000 7f5149181640 1 monclient: found mon.noname-c 2024-06-03T08:37:53.709 INFO:tasks.workunit.client.0.smithi088.stdout: api_asio: 2024-06-03T08:37:53.539+0000 7f5149181640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-06-03T08:37:53.709 INFO:tasks.workunit.client.0.smithi088.stdout: api_asio: 2024-06-03T08:37:53.539+0000 7f5149181640 10 monclient: _send_mon_message to mon.noname-c at v1:172.21.15.88:6790/0 2024-06-03T08:37:53.709 INFO:tasks.workunit.client.0.smithi088.stdout: api_asio: 2024-06-03T08:37:53.539+0000 7f5149181640 1 -- 172.21.15.88:0/273312373 --> v1:172.21.15.88:6790/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x561922039a80 con 0x56192203cd80 2024-06-03T08:37:53.709 INFO:tasks.workunit.client.0.smithi088.stdout: api_asio: 2024-06-03T08:37:53.539+0000 7f5149181640 10 monclient: _finish_auth 0 2024-06-03T08:37:53.710 INFO:tasks.workunit.client.0.smithi088.stdout: api_asio: 2024-06-03T08:37:53.539+0000 7f5149181640 10 monclient: _check_auth_tickets 2024-06-03T08:37:53.710 INFO:tasks.workunit.client.0.smithi088.stdout: api_asio: 2024-06-03T08:37:53.539+0000 7f5149181640 20 monclient: _check_auth_rotating not needed by client.admin 2024-06-03T08:37:53.710 INFO:tasks.workunit.client.0.smithi088.stdout: api_asio: 2024-06-03T08:37:53.539+0000 7f5149181640 1 -- 172.21.15.88:0/273312373 <== mon.2 v1:172.21.15.88:6790/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 2234374065 0 0) 0x7f513c0016d0 con 0x56192203cd80 2024-06-03T08:37:53.710 INFO:tasks.workunit.client.0.smithi088.stdout: api_asio: 2024-06-03T08:37:53.539+0000 7f514d647780 5 monclient: authenticate success, global_id 4436 2024-06-03T08:37:53.710 INFO:tasks.workunit.client.0.smithi088.stdout: api_asio: 2024-06-03T08:37:53.539+0000 7f514d647780 20 monclient: get_monmap_and_config waiting for monmap|config 2024-06-03T08:37:53.710 INFO:tasks.workunit.client.0.smithi088.stdout: api_asio: 2024-06-03T08:37:53.539+0000 7f5149181640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-06-03T08:37:53.710 INFO:tasks.workunit.client.0.smithi088.stdout: api_asio: 2024-06-03T08:37:53.539+0000 7f5149181640 10 monclient: got monmap 1 from mon.noname-c (according to old e1) 2024-06-03T08:37:53.710 INFO:tasks.workunit.client.0.smithi088.stdout: api_asio: 2024-06-03T08:37:53.539+0000 7f5149181640 10 monclient: dump: 2024-06-03T08:37:53.710 INFO:tasks.workunit.client.0.smithi088.stdout: api_asio: epoch 1 2024-06-03T08:37:53.710 INFO:tasks.workunit.client.0.smithi088.stdout: api_asio: fsid ec3c3fdd-3ac6-4d7d-9412-683130ed8b04 2024-06-03T08:37:53.710 INFO:tasks.workunit.client.0.smithi088.stdout: api_asio: last_changed 2024-06-03T08:34:31.885125+0000 2024-06-03T08:37:53.710 INFO:tasks.workunit.client.0.smithi088.stdout: api_asio: created 2024-06-03T08:34:31.885125+0000 2024-06-03T08:37:53.710 INFO:tasks.workunit.client.0.smithi088.stdout: api_asio: min_mon_release 18 (reef) 2024-06-03T08:37:53.711 INFO:tasks.workunit.client.0.smithi088.stdout: api_asio: election_strategy: 3 2024-06-03T08:37:53.711 INFO:tasks.workunit.client.0.smithi088.stdout: api_asio: 0: v1:172.21.15.88:6789/0 mon.a 2024-06-03T08:37:53.711 INFO:tasks.workunit.client.0.smithi088.stdout: api_asio: 1: v1:172.21.15.192:6789/0 mon.b 2024-06-03T08:37:53.711 INFO:tasks.workunit.client.0.smithi088.stdout: api_asio: 2: v1:172.21.15.88:6790/0 mon.c 2024-06-03T08:37:53.711 INFO:tasks.workunit.client.0.smithi088.stdout: api_asio: 2024-06-03T08:37:53.719 INFO:tasks.workunit.client.0.smithi088.stderr:+++ case "$(basename $(readlink /proc/$$/exe))" in 2024-06-03T08:37:53.719 INFO:tasks.workunit.client.0.smithi088.stderr:+++++ readlink /proc/40723/exe 2024-06-03T08:37:53.719 INFO:tasks.workunit.client.0.smithi088.stderr:+++ which_declare='declare -f' 2024-06-03T08:37:53.719 INFO:tasks.workunit.client.0.smithi088.stderr:+++ which_opt=-f 2024-06-03T08:37:53.719 INFO:tasks.workunit.client.0.smithi088.stderr:+++ export which_declare 2024-06-03T08:37:53.719 INFO:tasks.workunit.client.0.smithi088.stderr:+++ export -f which 2024-06-03T08:37:53.719 INFO:tasks.workunit.client.0.smithi088.stderr:++ unset i 2024-06-03T08:37:53.719 INFO:tasks.workunit.client.0.smithi088.stderr:++ unset -f pathmunge 2024-06-03T08:37:53.719 INFO:tasks.workunit.client.0.smithi088.stderr:+ [[ /home/ubuntu/.local/bin:/home/ubuntu/bin:/usr/local/bin:/usr/bin:/usr/local/sbin:/usr/sbin:/usr/sbin =~ /home/ubuntu/\.local/bin:/home/ubuntu/bin: ]] 2024-06-03T08:37:53.719 INFO:tasks.workunit.client.0.smithi088.stderr:+ export PATH 2024-06-03T08:37:53.719 INFO:tasks.workunit.client.0.smithi088.stderr:+ '[' -d /home/ubuntu/.bashrc.d ']' 2024-06-03T08:37:53.720 INFO:tasks.workunit.client.0.smithi088.stderr:+ unset rc 2024-06-03T08:37:53.720 INFO:tasks.workunit.client.0.smithi088.stderr:++++ basename /usr/bin/bash 2024-06-03T08:37:53.720 INFO:tasks.workunit.client.0.smithi088.stderr:+ tee ceph_test_rados_api_cmd.log 2024-06-03T08:37:53.720 INFO:tasks.workunit.client.0.smithi088.stderr:+++ which_declare='declare -f' 2024-06-03T08:37:53.720 INFO:tasks.workunit.client.0.smithi088.stderr:+++ which_opt=-f 2024-06-03T08:37:53.720 INFO:tasks.workunit.client.0.smithi088.stderr:+++ export which_declare 2024-06-03T08:37:53.720 INFO:tasks.workunit.client.0.smithi088.stderr:+++ export -f which 2024-06-03T08:37:53.720 INFO:tasks.workunit.client.0.smithi088.stderr:++ unset i 2024-06-03T08:37:53.720 INFO:tasks.workunit.client.0.smithi088.stderr:++ unset -f pathmunge 2024-06-03T08:37:53.720 INFO:tasks.workunit.client.0.smithi088.stderr:+ [[ /home/ubuntu/.local/bin:/home/ubuntu/bin:/usr/local/bin:/usr/bin:/usr/local/sbin:/usr/sbin:/usr/sbin =~ /home/ubuntu/\.local/bin:/home/ubuntu/bin: ]] 2024-06-03T08:37:53.720 INFO:tasks.workunit.client.0.smithi088.stderr:+ export PATH 2024-06-03T08:37:53.720 INFO:tasks.workunit.client.0.smithi088.stderr:+ '[' -d /home/ubuntu/.bashrc.d ']' 2024-06-03T08:37:53.721 INFO:tasks.workunit.client.0.smithi088.stderr:+ unset rc 2024-06-03T08:37:53.721 INFO:tasks.workunit.client.0.smithi088.stderr:+++++ readlink /proc/40648/exe 2024-06-03T08:37:53.721 INFO:tasks.workunit.client.0.smithi088.stderr:+ sed 's/^/ api_service_pp: /' 2024-06-03T08:37:53.721 INFO:tasks.workunit.client.0.smithi088.stderr:+ tee ceph_test_rados_api_service_pp.log 2024-06-03T08:37:53.721 INFO:tasks.workunit.client.0.smithi088.stderr:++++ basename /usr/bin/bash 2024-06-03T08:37:53.721 INFO:tasks.workunit.client.0.smithi088.stderr:+ sed 's/^/ api_cmd: /' 2024-06-03T08:37:53.721 INFO:tasks.workunit.client.0.smithi088.stderr:+++ alias 'grep=grep --color=auto' 2024-06-03T08:37:53.721 INFO:tasks.workunit.client.0.smithi088.stderr:+++ which_declare='declare -f' 2024-06-03T08:37:53.721 INFO:tasks.workunit.client.0.smithi088.stderr:+++ which_opt=-f 2024-06-03T08:37:53.721 INFO:tasks.workunit.client.0.smithi088.stderr:+++ export which_declare 2024-06-03T08:37:53.721 INFO:tasks.workunit.client.0.smithi088.stderr:+++ export -f which 2024-06-03T08:37:53.721 INFO:tasks.workunit.client.0.smithi088.stderr:+++ alias 'egrep=egrep --color=auto' 2024-06-03T08:37:53.721 INFO:tasks.workunit.client.0.smithi088.stderr:++ unset i 2024-06-03T08:37:53.721 INFO:tasks.workunit.client.0.smithi088.stderr:++ unset -f pathmunge 2024-06-03T08:37:53.722 INFO:tasks.workunit.client.0.smithi088.stderr:+ [[ /home/ubuntu/.local/bin:/home/ubuntu/bin:/usr/local/bin:/usr/bin:/usr/local/sbin:/usr/sbin:/usr/sbin =~ /home/ubuntu/\.local/bin:/home/ubuntu/bin: ]] 2024-06-03T08:37:53.722 INFO:tasks.workunit.client.0.smithi088.stderr:+++ alias 'fgrep=fgrep --color=auto' 2024-06-03T08:37:53.722 INFO:tasks.workunit.client.0.smithi088.stderr:++ for i in /etc/profile.d/*.sh 2024-06-03T08:37:53.722 INFO:tasks.workunit.client.0.smithi088.stderr:+ export PATH 2024-06-03T08:37:53.722 INFO:tasks.workunit.client.0.smithi088.stderr:+ '[' -d /home/ubuntu/.bashrc.d ']' 2024-06-03T08:37:53.722 INFO:tasks.workunit.client.0.smithi088.stderr:+ unset rc 2024-06-03T08:37:53.722 INFO:tasks.workunit.client.0.smithi088.stderr:++ '[' -r /etc/profile.d/colorls.sh ']' 2024-06-03T08:37:53.722 INFO:tasks.workunit.client.0.smithi088.stderr:++ '[' '' ']' 2024-06-03T08:37:53.722 INFO:tasks.workunit.client.0.smithi088.stderr:++ . /etc/profile.d/colorls.sh 2024-06-03T08:37:53.722 INFO:tasks.workunit.client.0.smithi088.stderr:+++ '[' '!' -t 0 ']' 2024-06-03T08:37:53.722 INFO:tasks.workunit.client.0.smithi088.stderr:+++ return 2024-06-03T08:37:53.722 INFO:tasks.workunit.client.0.smithi088.stderr:++ for i in /etc/profile.d/*.sh 2024-06-03T08:37:53.722 INFO:tasks.workunit.client.0.smithi088.stderr:++ '[' -r /etc/profile.d/colorsysstat.sh ']' 2024-06-03T08:37:53.723 INFO:tasks.workunit.client.0.smithi088.stderr:++ '[' '' ']' 2024-06-03T08:37:53.723 INFO:tasks.workunit.client.0.smithi088.stderr:++ . /etc/profile.d/colorsysstat.sh 2024-06-03T08:37:53.723 INFO:tasks.workunit.client.0.smithi088.stderr:+ ceph_test_rados_api_cmd_pp 2024-06-03T08:37:53.723 INFO:tasks.workunit.client.0.smithi088.stderr:+++ export S_COLORS=auto 2024-06-03T08:37:53.723 INFO:tasks.workunit.client.0.smithi088.stderr:+++ S_COLORS=auto 2024-06-03T08:37:53.723 INFO:tasks.workunit.client.0.smithi088.stderr:++ for i in /etc/profile.d/*.sh 2024-06-03T08:37:53.723 INFO:tasks.workunit.client.0.smithi088.stderr:++ '[' -r /etc/profile.d/colorxzgrep.sh ']' 2024-06-03T08:37:53.723 INFO:tasks.workunit.client.0.smithi088.stderr:++ '[' '' ']' 2024-06-03T08:37:53.723 INFO:tasks.workunit.client.0.smithi088.stderr:++ . /etc/profile.d/colorxzgrep.sh 2024-06-03T08:37:53.723 INFO:tasks.workunit.client.0.smithi088.stderr:+++ /usr/libexec/grepconf.sh -c 2024-06-03T08:37:53.723 INFO:tasks.workunit.client.0.smithi088.stderr:+ ceph_test_rados_api_cmd 2024-06-03T08:37:53.723 INFO:tasks.workunit.client.0.smithi088.stderr:+++ alias 'zgrep=zgrep --color=auto' 2024-06-03T08:37:53.723 INFO:tasks.workunit.client.0.smithi088.stderr:+++ alias 'zfgrep=zfgrep --color=auto' 2024-06-03T08:37:53.724 INFO:tasks.workunit.client.0.smithi088.stderr:+++ alias 'zegrep=zegrep --color=auto' 2024-06-03T08:37:53.724 INFO:tasks.workunit.client.0.smithi088.stderr:++ for i in /etc/profile.d/*.sh 2024-06-03T08:37:53.724 INFO:tasks.workunit.client.0.smithi088.stderr:++ '[' -r /etc/profile.d/debuginfod.sh ']' 2024-06-03T08:37:53.724 INFO:tasks.workunit.client.0.smithi088.stderr:++ '[' '' ']' 2024-06-03T08:37:53.724 INFO:tasks.workunit.client.0.smithi088.stderr:++ . /etc/profile.d/debuginfod.sh 2024-06-03T08:37:53.724 INFO:tasks.workunit.client.0.smithi088.stderr:+++ '[' -z 'https://debuginfod.centos.org/ ' ']' 2024-06-03T08:37:53.724 INFO:tasks.workunit.client.0.smithi088.stderr:++ for i in /etc/profile.d/*.sh 2024-06-03T08:37:53.724 INFO:tasks.workunit.client.0.smithi088.stderr:++ '[' -r /etc/profile.d/gawk.sh ']' 2024-06-03T08:37:53.724 INFO:tasks.workunit.client.0.smithi088.stderr:++ '[' '' ']' 2024-06-03T08:37:53.724 INFO:tasks.workunit.client.0.smithi088.stderr:++ . /etc/profile.d/gawk.sh 2024-06-03T08:37:53.724 INFO:tasks.workunit.client.0.smithi088.stderr:++ for i in /etc/profile.d/*.sh 2024-06-03T08:37:53.724 INFO:tasks.workunit.client.0.smithi088.stderr:++ '[' -r /etc/profile.d/lang.sh ']' 2024-06-03T08:37:53.724 INFO:tasks.workunit.client.0.smithi088.stderr:++ '[' '' ']' 2024-06-03T08:37:53.724 INFO:tasks.workunit.client.0.smithi088.stderr:++ . /etc/profile.d/lang.sh 2024-06-03T08:37:53.725 INFO:tasks.workunit.client.0.smithi088.stderr:+++ unset LANG_backup 2024-06-03T08:37:53.725 INFO:tasks.workunit.client.0.smithi088.stderr:+++ alias 'xzgrep=xzgrep --color=auto' 2024-06-03T08:37:53.725 INFO:tasks.workunit.client.0.smithi088.stderr:+ tee ceph_test_rados_api_cmd_pp.log 2024-06-03T08:37:53.725 INFO:tasks.workunit.client.0.smithi088.stderr:+++ alias 'xzegrep=xzegrep --color=auto' 2024-06-03T08:37:53.725 INFO:tasks.workunit.client.0.smithi088.stderr:+++ alias 'xzfgrep=xzfgrep --color=auto' 2024-06-03T08:37:53.725 INFO:tasks.workunit.client.0.smithi088.stderr:++ for i in /etc/profile.d/*.sh 2024-06-03T08:37:53.725 INFO:tasks.workunit.client.0.smithi088.stderr:++ '[' -r /etc/profile.d/colorzgrep.sh ']' 2024-06-03T08:37:53.725 INFO:tasks.workunit.client.0.smithi088.stderr:++ '[' '' ']' 2024-06-03T08:37:53.725 INFO:tasks.workunit.client.0.smithi088.stderr:++ . /etc/profile.d/colorzgrep.sh 2024-06-03T08:37:53.725 INFO:tasks.workunit.client.0.smithi088.stderr:+++ '[' -f /usr/libexec/grepconf.sh ']' 2024-06-03T08:37:53.725 INFO:tasks.workunit.client.0.smithi088.stderr:+++ /usr/libexec/grepconf.sh -c 2024-06-03T08:37:53.725 INFO:tasks.workunit.client.0.smithi088.stderr:+ ff=api_c_read_operations 2024-06-03T08:37:53.725 INFO:tasks.workunit.client.0.smithi088.stderr:+ ceph_test_rados_api_service_pp 2024-06-03T08:37:53.726 INFO:tasks.workunit.client.0.smithi088.stderr:+ pid=40836 2024-06-03T08:37:53.726 INFO:tasks.workunit.client.0.smithi088.stderr:+ echo 'test api_c_read_operations on pid 40836' 2024-06-03T08:37:53.726 INFO:tasks.workunit.client.0.smithi088.stdout: test api_c_read_operations on pid 40836 2024-06-03T08:37:53.735 INFO:tasks.workunit.client.0.smithi088.stderr:+ pids[$f]=40836 2024-06-03T08:37:53.736 INFO:tasks.workunit.client.0.smithi088.stderr:+ sed 's/^/ api_cmd_pp: /' 2024-06-03T08:37:53.736 INFO:tasks.workunit.client.0.smithi088.stderr:+ for f in api_aio api_aio_pp api_io api_io_pp api_asio api_list api_lock api_lock_pp api_misc api_misc_pp api_tier_pp api_pool api_snapshots api_snapshots_pp api_stat api_stat_pp api_watch_notify api_watch_notify_pp api_cmd api_cmd_pp api_service api_service_pp api_c_write_operations api_c_read_operations api_cls_remote_reads list_parallel open_pools_parallel delete_pools_parallel 2024-06-03T08:37:53.736 INFO:tasks.workunit.client.0.smithi088.stderr:+ '[' 1 -eq 1 ']' 2024-06-03T08:37:53.736 INFO:tasks.workunit.client.0.smithi088.stderr:++++ /usr/bin/locale 2024-06-03T08:37:53.736 INFO:tasks.workunit.client.0.smithi088.stderr:++ printf %25s api_cls_remote_reads 2024-06-03T08:37:53.736 INFO:tasks.workunit.client.0.smithi088.stderr:+ r=' api_cls_remote_reads' 2024-06-03T08:37:53.736 INFO:tasks.workunit.client.0.smithi088.stderr:+++ '[' -n '' ']' 2024-06-03T08:37:53.736 INFO:tasks.workunit.client.0.smithi088.stderr:+++ LANG_backup=en_US.UTF-8 2024-06-03T08:37:53.736 INFO:tasks.workunit.client.0.smithi088.stderr:+++ for config in /etc/locale.conf "${HOME}/.i18n" 2024-06-03T08:37:53.736 INFO:tasks.workunit.client.0.smithi088.stderr:+++ '[' -f /etc/locale.conf ']' 2024-06-03T08:37:53.736 INFO:tasks.workunit.client.0.smithi088.stderr:+++ '[' -x /usr/bin/sed ']' 2024-06-03T08:37:53.736 INFO:tasks.workunit.client.0.smithi088.stderr:+ bash -o pipefail -exc 'ceph_test_rados_api_c_read_operations 2>&1 | tee ceph_test_rados_api_c_read_operations.log | sed "s/^/ api_c_read_operations: /"' 2024-06-03T08:37:53.736 INFO:tasks.workunit.client.0.smithi088.stderr:+ '[' -f /etc/bashrc ']' 2024-06-03T08:37:53.737 INFO:tasks.workunit.client.0.smithi088.stderr:++++ /usr/bin/sed -r -e 's/^[[:blank:]]*([[:upper:]_]+)=([[:print:][:digit:]\._-]+|"[[:print:][:digit:]\._-]+")/export \1=\2/;t;d' /etc/locale.conf 2024-06-03T08:37:53.737 INFO:tasks.workunit.client.0.smithi088.stderr:+ . /etc/bashrc 2024-06-03T08:37:53.737 INFO:tasks.workunit.client.0.smithi088.stderr:++ '[' -z '' ']' 2024-06-03T08:37:53.737 INFO:tasks.workunit.client.0.smithi088.stderr:++ BASHRCSOURCED=Y 2024-06-03T08:37:53.737 INFO:tasks.workunit.client.0.smithi088.stderr:++ '[' '' ']' 2024-06-03T08:37:53.737 INFO:tasks.workunit.client.0.smithi088.stderr:++ shopt -q login_shell 2024-06-03T08:37:53.737 INFO:tasks.workunit.client.0.smithi088.stderr:+++ eval export 'LANG="en_US.UTF-8"' 2024-06-03T08:37:53.737 INFO:tasks.workunit.client.0.smithi088.stderr:++++ export LANG=en_US.UTF-8 2024-06-03T08:37:53.737 INFO:tasks.workunit.client.0.smithi088.stderr:++++ LANG=en_US.UTF-8 2024-06-03T08:37:53.737 INFO:tasks.workunit.client.0.smithi088.stderr:+++ for config in /etc/locale.conf "${HOME}/.i18n" 2024-06-03T08:37:53.737 INFO:tasks.workunit.client.0.smithi088.stderr:+++ '[' -f /home/ubuntu/.i18n ']' 2024-06-03T08:37:53.737 INFO:tasks.workunit.client.0.smithi088.stderr:+++ '[' -n en_US.UTF-8 ']' 2024-06-03T08:37:53.737 INFO:tasks.workunit.client.0.smithi088.stderr:+++ LANG=en_US.UTF-8 2024-06-03T08:37:53.738 INFO:tasks.workunit.client.0.smithi088.stderr:+++ unset LANG_backup config 2024-06-03T08:37:53.738 INFO:tasks.workunit.client.0.smithi088.stderr:+++ '[' -n '' ']' 2024-06-03T08:37:53.738 INFO:tasks.workunit.client.0.smithi088.stderr:+++ '[' -n en_US.UTF-8 ']' 2024-06-03T08:37:53.738 INFO:tasks.workunit.client.0.smithi088.stderr:+++ alias 'zgrep=zgrep --color=auto' 2024-06-03T08:37:53.738 INFO:tasks.workunit.client.0.smithi088.stderr:+++ '[' dumb = linux ']' 2024-06-03T08:37:53.738 INFO:tasks.workunit.client.0.smithi088.stderr:+++ alias 'zfgrep=zfgrep --color=auto' 2024-06-03T08:37:53.738 INFO:tasks.workunit.client.0.smithi088.stderr:++ for i in /etc/profile.d/*.sh 2024-06-03T08:37:53.738 INFO:tasks.workunit.client.0.smithi088.stderr:++ '[' -r /etc/profile.d/less.sh ']' 2024-06-03T08:37:53.738 INFO:tasks.workunit.client.0.smithi088.stderr:++ '[' '' ']' 2024-06-03T08:37:53.738 INFO:tasks.workunit.client.0.smithi088.stderr:++ . /etc/profile.d/less.sh 2024-06-03T08:37:53.738 INFO:tasks.workunit.client.0.smithi088.stderr:+++ '[' -z '||/usr/bin/lesspipe.sh %s' ']' 2024-06-03T08:37:53.738 INFO:tasks.workunit.client.0.smithi088.stderr:+++ alias 'zegrep=zegrep --color=auto' 2024-06-03T08:37:53.738 INFO:tasks.workunit.client.0.smithi088.stderr:++ for i in /etc/profile.d/*.sh 2024-06-03T08:37:53.739 INFO:tasks.workunit.client.0.smithi088.stderr:++ '[' -r /etc/profile.d/debuginfod.sh ']' 2024-06-03T08:37:53.739 INFO:tasks.workunit.client.0.smithi088.stderr:++ '[' '' ']' 2024-06-03T08:37:53.739 INFO:tasks.workunit.client.0.smithi088.stderr:++ . /etc/profile.d/debuginfod.sh 2024-06-03T08:37:53.739 INFO:tasks.workunit.client.0.smithi088.stderr:++ for i in /etc/profile.d/*.sh 2024-06-03T08:37:53.739 INFO:tasks.workunit.client.0.smithi088.stderr:+++ '[' -z 'https://debuginfod.centos.org/ ' ']' 2024-06-03T08:37:53.739 INFO:tasks.workunit.client.0.smithi088.stderr:++ for i in /etc/profile.d/*.sh 2024-06-03T08:37:53.739 INFO:tasks.workunit.client.0.smithi088.stderr:++ '[' -r /etc/profile.d/gawk.sh ']' 2024-06-03T08:37:53.739 INFO:tasks.workunit.client.0.smithi088.stderr:++ '[' '' ']' 2024-06-03T08:37:53.739 INFO:tasks.workunit.client.0.smithi088.stderr:++ . /etc/profile.d/gawk.sh 2024-06-03T08:37:53.739 INFO:tasks.workunit.client.0.smithi088.stderr:++ '[' -r /etc/profile.d/which2.sh ']' 2024-06-03T08:37:53.739 INFO:tasks.workunit.client.0.smithi088.stderr:++ '[' '' ']' 2024-06-03T08:37:53.739 INFO:tasks.workunit.client.0.smithi088.stderr:++ for i in /etc/profile.d/*.sh 2024-06-03T08:37:53.739 INFO:tasks.workunit.client.0.smithi088.stderr:++ '[' -r /etc/profile.d/lang.sh ']' 2024-06-03T08:37:53.740 INFO:tasks.workunit.client.0.smithi088.stderr:++ '[' '' ']' 2024-06-03T08:37:53.740 INFO:tasks.workunit.client.0.smithi088.stderr:++ . /etc/profile.d/lang.sh 2024-06-03T08:37:53.740 INFO:tasks.workunit.client.0.smithi088.stderr:+++ unset LANG_backup 2024-06-03T08:37:53.740 INFO:tasks.workunit.client.0.smithi088.stderr:++ . /etc/profile.d/which2.sh 2024-06-03T08:37:53.740 INFO:tasks.workunit.client.0.smithi088.stderr:+++ case "$(basename $(readlink /proc/$$/exe))" in 2024-06-03T08:37:53.740 INFO:tasks.workunit.client.0.smithi088.stderr:++++ /usr/bin/locale 2024-06-03T08:37:53.740 INFO:tasks.workunit.client.0.smithi088.stderr:+++++ readlink /proc/40685/exe 2024-06-03T08:37:53.740 INFO:tasks.workunit.client.0.smithi088.stderr:++++ basename /usr/bin/bash 2024-06-03T08:37:53.740 INFO:tasks.workunit.client.0.smithi088.stderr:+++ '[' -n '' ']' 2024-06-03T08:37:53.740 INFO:tasks.workunit.client.0.smithi088.stderr:+++ LANG_backup=en_US.UTF-8 2024-06-03T08:37:53.740 INFO:tasks.workunit.client.0.smithi088.stderr:+++ for config in /etc/locale.conf "${HOME}/.i18n" 2024-06-03T08:37:53.740 INFO:tasks.workunit.client.0.smithi088.stderr:+++ '[' -f /etc/locale.conf ']' 2024-06-03T08:37:53.740 INFO:tasks.workunit.client.0.smithi088.stderr:+++ '[' -x /usr/bin/sed ']' 2024-06-03T08:37:53.741 INFO:tasks.workunit.client.0.smithi088.stderr:+++ umask 2024-06-03T08:37:53.741 INFO:tasks.workunit.client.0.smithi088.stderr:++ '[' 0022 -eq 0 ']' 2024-06-03T08:37:53.741 INFO:tasks.workunit.client.0.smithi088.stderr:++ SHELL=/bin/bash 2024-06-03T08:37:53.741 INFO:tasks.workunit.client.0.smithi088.stderr:++ for i in /etc/profile.d/*.sh 2024-06-03T08:37:53.741 INFO:tasks.workunit.client.0.smithi088.stderr:++ '[' -r /etc/profile.d/bash_completion.sh ']' 2024-06-03T08:37:53.741 INFO:tasks.workunit.client.0.smithi088.stderr:++ '[' '' ']' 2024-06-03T08:37:53.741 INFO:tasks.workunit.client.0.smithi088.stderr:++ . /etc/profile.d/bash_completion.sh 2024-06-03T08:37:53.741 INFO:tasks.workunit.client.0.smithi088.stderr:+++ '[' 'x5.1.8(1)-release' '!=' x -a x '!=' x -a x = x ']' 2024-06-03T08:37:53.741 INFO:tasks.workunit.client.0.smithi088.stderr:++ for i in /etc/profile.d/*.sh 2024-06-03T08:37:53.741 INFO:tasks.workunit.client.0.smithi088.stderr:++ '[' -r /etc/profile.d/cobbler.sh ']' 2024-06-03T08:37:53.741 INFO:tasks.workunit.client.0.smithi088.stderr:++ '[' '' ']' 2024-06-03T08:37:53.741 INFO:tasks.workunit.client.0.smithi088.stderr:++ . /etc/profile.d/cobbler.sh 2024-06-03T08:37:53.741 INFO:tasks.workunit.client.0.smithi088.stderr:+++ export COBBLER_SERVER=172.21.0.11 2024-06-03T08:37:53.741 INFO:tasks.workunit.client.0.smithi088.stderr:+++ COBBLER_SERVER=172.21.0.11 2024-06-03T08:37:53.742 INFO:tasks.workunit.client.0.smithi088.stderr:++ for i in /etc/profile.d/*.sh 2024-06-03T08:37:53.742 INFO:tasks.workunit.client.0.smithi088.stderr:++ '[' -r /etc/profile.d/colorgrep.sh ']' 2024-06-03T08:37:53.742 INFO:tasks.workunit.client.0.smithi088.stderr:++ '[' '' ']' 2024-06-03T08:37:53.742 INFO:tasks.workunit.client.0.smithi088.stderr:++ . /etc/profile.d/colorgrep.sh 2024-06-03T08:37:53.742 INFO:tasks.workunit.client.0.smithi088.stderr:+++ /usr/libexec/grepconf.sh -c 2024-06-03T08:37:53.742 INFO:tasks.workunit.client.0.smithi088.stderr:+++ which_declare='declare -f' 2024-06-03T08:37:53.742 INFO:tasks.workunit.client.0.smithi088.stderr:+++ which_opt=-f 2024-06-03T08:37:53.742 INFO:tasks.workunit.client.0.smithi088.stderr:+++ export which_declare 2024-06-03T08:37:53.742 INFO:tasks.workunit.client.0.smithi088.stderr:+++ export -f which 2024-06-03T08:37:53.742 INFO:tasks.workunit.client.0.smithi088.stderr:++ unset i 2024-06-03T08:37:53.742 INFO:tasks.workunit.client.0.smithi088.stderr:++ unset -f pathmunge 2024-06-03T08:37:53.742 INFO:tasks.workunit.client.0.smithi088.stderr:+ [[ /home/ubuntu/.local/bin:/home/ubuntu/bin:/usr/local/bin:/usr/bin:/usr/local/sbin:/usr/sbin:/usr/sbin =~ /home/ubuntu/\.local/bin:/home/ubuntu/bin: ]] 2024-06-03T08:37:53.743 INFO:tasks.workunit.client.0.smithi088.stderr:+ export PATH 2024-06-03T08:37:53.743 INFO:tasks.workunit.client.0.smithi088.stderr:+ '[' -d /home/ubuntu/.bashrc.d ']' 2024-06-03T08:37:53.743 INFO:tasks.workunit.client.0.smithi088.stderr:+ unset rc 2024-06-03T08:37:53.743 INFO:tasks.workunit.client.0.smithi088.stderr:++++ /usr/bin/sed -r -e 's/^[[:blank:]]*([[:upper:]_]+)=([[:print:][:digit:]\._-]+|"[[:print:][:digit:]\._-]+")/export \1=\2/;t;d' /etc/locale.conf 2024-06-03T08:37:53.743 INFO:tasks.workunit.client.0.smithi088.stderr:++ echo api_cls_remote_reads 2024-06-03T08:37:53.743 INFO:tasks.workunit.client.0.smithi088.stderr:++ awk '{print $1}' 2024-06-03T08:37:53.743 INFO:tasks.workunit.client.0.smithi088.stderr:+++ eval export 'LANG="en_US.UTF-8"' 2024-06-03T08:37:53.743 INFO:tasks.workunit.client.0.smithi088.stderr:++++ export LANG=en_US.UTF-8 2024-06-03T08:37:53.743 INFO:tasks.workunit.client.0.smithi088.stderr:++++ LANG=en_US.UTF-8 2024-06-03T08:37:53.743 INFO:tasks.workunit.client.0.smithi088.stderr:+++ for config in /etc/locale.conf "${HOME}/.i18n" 2024-06-03T08:37:53.743 INFO:tasks.workunit.client.0.smithi088.stderr:+++ '[' -f /home/ubuntu/.i18n ']' 2024-06-03T08:37:53.743 INFO:tasks.workunit.client.0.smithi088.stderr:+++ '[' -n en_US.UTF-8 ']' 2024-06-03T08:37:53.743 INFO:tasks.workunit.client.0.smithi088.stderr:+++ LANG=en_US.UTF-8 2024-06-03T08:37:53.743 INFO:tasks.workunit.client.0.smithi088.stderr:+++ unset LANG_backup config 2024-06-03T08:37:53.743 INFO:tasks.workunit.client.0.smithi088.stderr:+++ '[' -n '' ']' 2024-06-03T08:37:53.744 INFO:tasks.workunit.client.0.smithi088.stderr:+++ '[' -n en_US.UTF-8 ']' 2024-06-03T08:37:53.744 INFO:tasks.workunit.client.0.smithi088.stderr:+++ '[' dumb = linux ']' 2024-06-03T08:37:53.744 INFO:tasks.workunit.client.0.smithi088.stderr:++ for i in /etc/profile.d/*.sh 2024-06-03T08:37:53.744 INFO:tasks.workunit.client.0.smithi088.stderr:++ '[' -r /etc/profile.d/less.sh ']' 2024-06-03T08:37:53.744 INFO:tasks.workunit.client.0.smithi088.stderr:++ '[' '' ']' 2024-06-03T08:37:53.744 INFO:tasks.workunit.client.0.smithi088.stderr:++ . /etc/profile.d/less.sh 2024-06-03T08:37:53.744 INFO:tasks.workunit.client.0.smithi088.stderr:+++ '[' -z '||/usr/bin/lesspipe.sh %s' ']' 2024-06-03T08:37:53.744 INFO:tasks.workunit.client.0.smithi088.stderr:++ for i in /etc/profile.d/*.sh 2024-06-03T08:37:53.744 INFO:tasks.workunit.client.0.smithi088.stderr:++ '[' -r /etc/profile.d/which2.sh ']' 2024-06-03T08:37:53.744 INFO:tasks.workunit.client.0.smithi088.stderr:++ '[' '' ']' 2024-06-03T08:37:53.744 INFO:tasks.workunit.client.0.smithi088.stderr:++ . /etc/profile.d/which2.sh 2024-06-03T08:37:53.744 INFO:tasks.workunit.client.0.smithi088.stderr:+++ case "$(basename $(readlink /proc/$$/exe))" in 2024-06-03T08:37:53.744 INFO:tasks.workunit.client.0.smithi088.stderr:+ ceph_test_rados_api_service 2024-06-03T08:37:53.744 INFO:tasks.workunit.client.0.smithi088.stderr:+ sed 's/^/ api_service: /' 2024-06-03T08:37:53.745 INFO:tasks.workunit.client.0.smithi088.stderr:+ ff=api_cls_remote_reads 2024-06-03T08:37:53.745 INFO:tasks.workunit.client.0.smithi088.stderr:+ pid=40886 2024-06-03T08:37:53.745 INFO:tasks.workunit.client.0.smithi088.stderr:+ echo 'test api_cls_remote_reads on pid 40886' 2024-06-03T08:37:53.745 INFO:tasks.workunit.client.0.smithi088.stdout:test api_cls_remote_reads on pid 40886 2024-06-03T08:37:53.748 INFO:tasks.workunit.client.0.smithi088.stderr:+ pids[$f]=40886 2024-06-03T08:37:53.748 INFO:tasks.workunit.client.0.smithi088.stderr:+ for f in api_aio api_aio_pp api_io api_io_pp api_asio api_list api_lock api_lock_pp api_misc api_misc_pp api_tier_pp api_pool api_snapshots api_snapshots_pp api_stat api_stat_pp api_watch_notify api_watch_notify_pp api_cmd api_cmd_pp api_service api_service_pp api_c_write_operations api_c_read_operations api_cls_remote_reads list_parallel open_pools_parallel delete_pools_parallel 2024-06-03T08:37:53.748 INFO:tasks.workunit.client.0.smithi088.stderr:+ '[' 1 -eq 1 ']' 2024-06-03T08:37:53.748 INFO:tasks.workunit.client.0.smithi088.stderr:+++ alias 'grep=grep --color=auto' 2024-06-03T08:37:53.748 INFO:tasks.workunit.client.0.smithi088.stderr:+++++ readlink /proc/40764/exe 2024-06-03T08:37:53.748 INFO:tasks.workunit.client.0.smithi088.stderr:+++ alias 'egrep=egrep --color=auto' 2024-06-03T08:37:53.748 INFO:tasks.workunit.client.0.smithi088.stderr:+++ alias 'fgrep=fgrep --color=auto' 2024-06-03T08:37:53.749 INFO:tasks.workunit.client.0.smithi088.stderr:++ for i in /etc/profile.d/*.sh 2024-06-03T08:37:53.749 INFO:tasks.workunit.client.0.smithi088.stderr:++ '[' -r /etc/profile.d/colorls.sh ']' 2024-06-03T08:37:53.749 INFO:tasks.workunit.client.0.smithi088.stderr:++ '[' '' ']' 2024-06-03T08:37:53.749 INFO:tasks.workunit.client.0.smithi088.stderr:++ . /etc/profile.d/colorls.sh 2024-06-03T08:37:53.749 INFO:tasks.workunit.client.0.smithi088.stderr:+++ '[' '!' -t 0 ']' 2024-06-03T08:37:53.749 INFO:tasks.workunit.client.0.smithi088.stderr:+++ return 2024-06-03T08:37:53.749 INFO:tasks.workunit.client.0.smithi088.stderr:++ for i in /etc/profile.d/*.sh 2024-06-03T08:37:53.749 INFO:tasks.workunit.client.0.smithi088.stderr:++ '[' -r /etc/profile.d/colorsysstat.sh ']' 2024-06-03T08:37:53.749 INFO:tasks.workunit.client.0.smithi088.stderr:++ '[' '' ']' 2024-06-03T08:37:53.749 INFO:tasks.workunit.client.0.smithi088.stderr:++ . /etc/profile.d/colorsysstat.sh 2024-06-03T08:37:53.749 INFO:tasks.workunit.client.0.smithi088.stderr:+ tee ceph_test_rados_api_service.log 2024-06-03T08:37:53.749 INFO:tasks.workunit.client.0.smithi088.stderr:+++ export S_COLORS=auto 2024-06-03T08:37:53.749 INFO:tasks.workunit.client.0.smithi088.stderr:+++ S_COLORS=auto 2024-06-03T08:37:53.750 INFO:tasks.workunit.client.0.smithi088.stderr:++ for i in /etc/profile.d/*.sh 2024-06-03T08:37:53.750 INFO:tasks.workunit.client.0.smithi088.stderr:++ '[' -r /etc/profile.d/colorxzgrep.sh ']' 2024-06-03T08:37:53.750 INFO:tasks.workunit.client.0.smithi088.stderr:++ '[' '' ']' 2024-06-03T08:37:53.750 INFO:tasks.workunit.client.0.smithi088.stderr:++ . /etc/profile.d/colorxzgrep.sh 2024-06-03T08:37:53.750 INFO:tasks.workunit.client.0.smithi088.stderr:+++ /usr/libexec/grepconf.sh -c 2024-06-03T08:37:53.750 INFO:tasks.workunit.client.0.smithi088.stderr:+ bash -o pipefail -exc 'ceph_test_rados_api_cls_remote_reads 2>&1 | tee ceph_test_rados_api_cls_remote_reads.log | sed "s/^/ api_cls_remote_reads: /"' 2024-06-03T08:37:53.750 INFO:tasks.workunit.client.0.smithi088.stderr:++ printf %25s list_parallel 2024-06-03T08:37:53.750 INFO:tasks.workunit.client.0.smithi088.stderr:+ r=' list_parallel' 2024-06-03T08:37:53.750 INFO:tasks.workunit.client.0.smithi088.stderr:++ awk '{print $1}' 2024-06-03T08:37:53.750 INFO:tasks.workunit.client.0.smithi088.stderr:+ '[' -f /etc/bashrc ']' 2024-06-03T08:37:53.750 INFO:tasks.workunit.client.0.smithi088.stderr:+ . /etc/bashrc 2024-06-03T08:37:53.750 INFO:tasks.workunit.client.0.smithi088.stderr:++ '[' -z '' ']' 2024-06-03T08:37:53.750 INFO:tasks.workunit.client.0.smithi088.stderr:++ BASHRCSOURCED=Y 2024-06-03T08:37:53.751 INFO:tasks.workunit.client.0.smithi088.stderr:++ '[' '' ']' 2024-06-03T08:37:53.751 INFO:tasks.workunit.client.0.smithi088.stderr:++ shopt -q login_shell 2024-06-03T08:37:53.751 INFO:tasks.workunit.client.0.smithi088.stderr:++ echo list_parallel 2024-06-03T08:37:53.751 INFO:tasks.workunit.client.0.smithi088.stderr:+ ff=list_parallel 2024-06-03T08:37:53.751 INFO:tasks.workunit.client.0.smithi088.stderr:+ pid=40899 2024-06-03T08:37:53.751 INFO:tasks.workunit.client.0.smithi088.stderr:+ echo 'test list_parallel on pid 40899' 2024-06-03T08:37:53.751 INFO:tasks.workunit.client.0.smithi088.stdout:test list_parallel on pid 40899 2024-06-03T08:37:53.754 INFO:tasks.workunit.client.0.smithi088.stderr:+ pids[$f]=40899 2024-06-03T08:37:53.754 INFO:tasks.workunit.client.0.smithi088.stderr:+ for f in api_aio api_aio_pp api_io api_io_pp api_asio api_list api_lock api_lock_pp api_misc api_misc_pp api_tier_pp api_pool api_snapshots api_snapshots_pp api_stat api_stat_pp api_watch_notify api_watch_notify_pp api_cmd api_cmd_pp api_service api_service_pp api_c_write_operations api_c_read_operations api_cls_remote_reads list_parallel open_pools_parallel delete_pools_parallel 2024-06-03T08:37:53.754 INFO:tasks.workunit.client.0.smithi088.stderr:+ '[' 1 -eq 1 ']' 2024-06-03T08:37:53.754 INFO:tasks.workunit.client.0.smithi088.stderr:+++ umask 2024-06-03T08:37:53.754 INFO:tasks.workunit.client.0.smithi088.stderr:++ '[' 0022 -eq 0 ']' 2024-06-03T08:37:53.754 INFO:tasks.workunit.client.0.smithi088.stderr:++ SHELL=/bin/bash 2024-06-03T08:37:53.754 INFO:tasks.workunit.client.0.smithi088.stderr:++ for i in /etc/profile.d/*.sh 2024-06-03T08:37:53.754 INFO:tasks.workunit.client.0.smithi088.stderr:++ '[' -r /etc/profile.d/bash_completion.sh ']' 2024-06-03T08:37:53.754 INFO:tasks.workunit.client.0.smithi088.stderr:++ '[' '' ']' 2024-06-03T08:37:53.754 INFO:tasks.workunit.client.0.smithi088.stderr:++ . /etc/profile.d/bash_completion.sh 2024-06-03T08:37:53.754 INFO:tasks.workunit.client.0.smithi088.stderr:+++ '[' 'x5.1.8(1)-release' '!=' x -a x '!=' x -a x = x ']' 2024-06-03T08:37:53.755 INFO:tasks.workunit.client.0.smithi088.stderr:++ for i in /etc/profile.d/*.sh 2024-06-03T08:37:53.755 INFO:tasks.workunit.client.0.smithi088.stderr:++ '[' -r /etc/profile.d/cobbler.sh ']' 2024-06-03T08:37:53.755 INFO:tasks.workunit.client.0.smithi088.stderr:++ '[' '' ']' 2024-06-03T08:37:53.755 INFO:tasks.workunit.client.0.smithi088.stderr:++ . /etc/profile.d/cobbler.sh 2024-06-03T08:37:53.755 INFO:tasks.workunit.client.0.smithi088.stderr:+++ export COBBLER_SERVER=172.21.0.11 2024-06-03T08:37:53.755 INFO:tasks.workunit.client.0.smithi088.stderr:+++ COBBLER_SERVER=172.21.0.11 2024-06-03T08:37:53.755 INFO:tasks.workunit.client.0.smithi088.stderr:++ for i in /etc/profile.d/*.sh 2024-06-03T08:37:53.755 INFO:tasks.workunit.client.0.smithi088.stderr:++ '[' -r /etc/profile.d/colorgrep.sh ']' 2024-06-03T08:37:53.755 INFO:tasks.workunit.client.0.smithi088.stderr:++ '[' '' ']' 2024-06-03T08:37:53.755 INFO:tasks.workunit.client.0.smithi088.stderr:++ . /etc/profile.d/colorgrep.sh 2024-06-03T08:37:53.755 INFO:tasks.workunit.client.0.smithi088.stderr:+++ /usr/libexec/grepconf.sh -c 2024-06-03T08:37:53.755 INFO:tasks.workunit.client.0.smithi088.stderr:++++ basename /usr/bin/bash 2024-06-03T08:37:53.755 INFO:tasks.workunit.client.0.smithi088.stderr:++ printf %25s open_pools_parallel 2024-06-03T08:37:53.755 INFO:tasks.workunit.client.0.smithi088.stderr:+ r=' open_pools_parallel' 2024-06-03T08:37:53.756 INFO:tasks.workunit.client.0.smithi088.stdout: api_asio: 2024-06-03T08:37:53.539+0000 7f514d647780 20 monclient: get_monmap_and_config waiting for monmap|config 2024-06-03T08:37:53.756 INFO:tasks.workunit.client.0.smithi088.stdout: api_asio: 2024-06-03T08:37:53.589+0000 7f5149181640 1 -- 172.21.15.88:0/273312373 <== mon.2 v1:172.21.15.88:6790/0 4 ==== config(0 keys) v1 ==== 4+0+0 (unknown 0 0 0) 0x7f513c0018f0 con 0x56192203cd80 2024-06-03T08:37:53.756 INFO:tasks.workunit.client.0.smithi088.stdout: api_asio: 2024-06-03T08:37:53.589+0000 7f5149181640 10 monclient: handle_config config(0 keys) v1 2024-06-03T08:37:53.756 INFO:tasks.workunit.client.0.smithi088.stdout: api_asio: 2024-06-03T08:37:53.589+0000 7f5149181640 1 -- 172.21.15.88:0/273312373 <== mon.2 v1:172.21.15.88:6790/0 5 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 2234374065 0 0) 0x7f513c001d00 con 0x56192203cd80 2024-06-03T08:37:53.756 INFO:tasks.workunit.client.0.smithi088.stdout: api_asio: 2024-06-03T08:37:53.589+0000 7f5149181640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-06-03T08:37:53.756 INFO:tasks.workunit.client.0.smithi088.stdout: api_asio: 2024-06-03T08:37:53.589+0000 7f5149181640 10 monclient: got monmap 1 from mon.c (according to old e1) 2024-06-03T08:37:53.756 INFO:tasks.workunit.client.0.smithi088.stdout: api_asio: 2024-06-03T08:37:53.589+0000 7f5149181640 10 monclient: dump: 2024-06-03T08:37:53.756 INFO:tasks.workunit.client.0.smithi088.stdout: api_asio: epoch 1 2024-06-03T08:37:53.756 INFO:tasks.workunit.client.0.smithi088.stdout: api_asio: fsid ec3c3fdd-3ac6-4d7d-9412-683130ed8b04 2024-06-03T08:37:53.756 INFO:tasks.workunit.client.0.smithi088.stdout: api_asio: last_changed 2024-06-03T08:34:31.885125+0000 2024-06-03T08:37:53.757 INFO:tasks.workunit.client.0.smithi088.stdout: api_asio: created 2024-06-03T08:34:31.885125+0000 2024-06-03T08:37:53.757 INFO:tasks.workunit.client.0.smithi088.stdout: api_asio: min_mon_release 18 (reef) 2024-06-03T08:37:53.757 INFO:tasks.workunit.client.0.smithi088.stdout: api_asio: election_strategy: 3 2024-06-03T08:37:53.757 INFO:tasks.workunit.client.0.smithi088.stdout: api_asio: 0: v1:172.21.15.88:6789/0 mon.a 2024-06-03T08:37:53.757 INFO:tasks.workunit.client.0.smithi088.stdout: api_asio: 1: v1:172.21.15.192:6789/0 mon.b 2024-06-03T08:37:53.757 INFO:tasks.workunit.client.0.smithi088.stdout: api_asio: 2: v1:172.21.15.88:6790/0 mon.c 2024-06-03T08:37:53.757 INFO:tasks.workunit.client.0.smithi088.stdout: api_asio: 2024-06-03T08:37:53.757 INFO:tasks.workunit.client.0.smithi088.stdout: api_asio: 2024-06-03T08:37:53.589+0000 7f514d647780 10 monclient: get_monmap_and_config success 2024-06-03T08:37:53.757 INFO:tasks.workunit.client.0.smithi088.stdout: api_asio: 2024-06-03T08:37:53.589+0000 7f514d647780 4 set_mon_vals no callback set 2024-06-03T08:37:53.757 INFO:tasks.workunit.client.0.smithi088.stdout: api_asio: 2024-06-03T08:37:53.590+0000 7f514d647780 10 monclient: shutdown 2024-06-03T08:37:53.757 INFO:tasks.workunit.client.0.smithi088.stdout: api_asio: 2024-06-03T08:37:53.590+0000 7f514d647780 20 monclient: shutdown discarding 0 pending message(s) 2024-06-03T08:37:53.757 INFO:tasks.workunit.client.0.smithi088.stdout: api_asio: 2024-06-03T08:37:53.590+0000 7f514d647780 1 -- 172.21.15.88:0/273312373 >> v1:172.21.15.88:6790/0 conn(0x56192203cd80 legacy=0x561922168780 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-06-03T08:37:53.757 INFO:tasks.workunit.client.0.smithi088.stdout: api_asio: 2024-06-03T08:37:53.590+0000 7f514d647780 1 -- 172.21.15.88:0/273312373 shutdown_connections 2024-06-03T08:37:53.758 INFO:tasks.workunit.client.0.smithi088.stdout: api_asio: 2024-06-03T08:37:53.590+0000 7f514d647780 1 -- 172.21.15.88:0/273312373 >> 172.21.15.88:0/273312373 conn(0x56192203c260 msgr2=0x5619220c61c0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-06-03T08:37:53.758 INFO:tasks.workunit.client.0.smithi088.stdout: api_asio: 2024-06-03T08:37:53.590+0000 7f514d647780 1 -- 172.21.15.88:0/273312373 shutdown_connections 2024-06-03T08:37:53.758 INFO:tasks.workunit.client.0.smithi088.stdout: api_asio: 2024-06-03T08:37:53.590+0000 7f514d647780 1 -- 172.21.15.88:0/273312373 wait complete. 2024-06-03T08:37:53.758 INFO:tasks.workunit.client.0.smithi088.stdout: api_asio: 2024-06-03T08:37:53.590+0000 7f514d647780 10 monclient: build_initial_monmap 2024-06-03T08:37:53.758 INFO:tasks.workunit.client.0.smithi088.stdout: api_asio: 2024-06-03T08:37:53.590+0000 7f514d647780 10 monclient: monmap: 2024-06-03T08:37:53.758 INFO:tasks.workunit.client.0.smithi088.stdout: api_asio: epoch 0 2024-06-03T08:37:53.758 INFO:tasks.workunit.client.0.smithi088.stdout: api_asio: fsid 00000000-0000-0000-0000-000000000000 2024-06-03T08:37:53.758 INFO:tasks.workunit.client.0.smithi088.stdout: api_asio: last_changed 0.000000 2024-06-03T08:37:53.758 INFO:tasks.workunit.client.0.smithi088.stdout: api_asio: created 0.000000 2024-06-03T08:37:53.758 INFO:tasks.workunit.client.0.smithi088.stdout: api_asio: min_mon_release 0 (unknown) 2024-06-03T08:37:53.758 INFO:tasks.workunit.client.0.smithi088.stdout: api_asio: election_strategy: 1 2024-06-03T08:37:53.758 INFO:tasks.workunit.client.0.smithi088.stdout: api_asio: 0: v1:172.21.15.88:6789/0 mon.noname-a 2024-06-03T08:37:53.759 INFO:tasks.workunit.client.0.smithi088.stdout: api_asio: 1: v1:172.21.15.192:6789/0 mon.noname-b 2024-06-03T08:37:53.759 INFO:tasks.workunit.client.0.smithi088.stdout: api_asio: 2: v1:172.21.15.88:6790/0 mon.noname-c 2024-06-03T08:37:53.759 INFO:tasks.workunit.client.0.smithi088.stdout: api_asio: 2024-06-03T08:37:53.759 INFO:tasks.workunit.client.0.smithi088.stdout: api_asio: 2024-06-03T08:37:53.591+0000 7f514d647780 1 librados: starting msgr at 2024-06-03T08:37:53.759 INFO:tasks.workunit.client.0.smithi088.stdout: api_asio: 2024-06-03T08:37:53.591+0000 7f514d647780 1 librados: starting objecter 2024-06-03T08:37:53.759 INFO:tasks.workunit.client.0.smithi088.stdout: api_asio: 2024-06-03T08:37:53.591+0000 7f514d647780 1 Processor -- start 2024-06-03T08:37:53.759 INFO:tasks.workunit.client.0.smithi088.stdout: api_asio: 2024-06-03T08:37:53.591+0000 7f514d647780 1 -- start start 2024-06-03T08:37:53.759 INFO:tasks.workunit.client.0.smithi088.stdout: api_asio: 2024-06-03T08:37:53.591+0000 7f514d647780 1 librados: setting wanted keys 2024-06-03T08:37:53.762 INFO:tasks.workunit.client.0.smithi088.stderr:+ bash -o pipefail -exc 'ceph_test_rados_list_parallel 2>&1 | tee ceph_test_rados_list_parallel.log | sed "s/^/ list_parallel: /"' 2024-06-03T08:37:53.762 INFO:tasks.workunit.client.0.smithi088.stderr:+ '[' -f /etc/bashrc ']' 2024-06-03T08:37:53.763 INFO:tasks.workunit.client.0.smithi088.stderr:+ . /etc/bashrc 2024-06-03T08:37:53.763 INFO:tasks.workunit.client.0.smithi088.stderr:+++ which_declare='declare -f' 2024-06-03T08:37:53.763 INFO:tasks.workunit.client.0.smithi088.stderr:+++ which_opt=-f 2024-06-03T08:37:53.763 INFO:tasks.workunit.client.0.smithi088.stderr:+++ export which_declare 2024-06-03T08:37:53.763 INFO:tasks.workunit.client.0.smithi088.stderr:++ echo open_pools_parallel 2024-06-03T08:37:53.763 INFO:tasks.workunit.client.0.smithi088.stderr:++ '[' -z '' ']' 2024-06-03T08:37:53.763 INFO:tasks.workunit.client.0.smithi088.stderr:++ BASHRCSOURCED=Y 2024-06-03T08:37:53.763 INFO:tasks.workunit.client.0.smithi088.stderr:++ '[' '' ']' 2024-06-03T08:37:53.763 INFO:tasks.workunit.client.0.smithi088.stderr:++ shopt -q login_shell 2024-06-03T08:37:53.763 INFO:tasks.workunit.client.0.smithi088.stderr:+++ export -f which 2024-06-03T08:37:53.763 INFO:tasks.workunit.client.0.smithi088.stderr:++ unset i 2024-06-03T08:37:53.763 INFO:tasks.workunit.client.0.smithi088.stderr:++ unset -f pathmunge 2024-06-03T08:37:53.764 INFO:tasks.workunit.client.0.smithi088.stderr:+ [[ /home/ubuntu/.local/bin:/home/ubuntu/bin:/usr/local/bin:/usr/bin:/usr/local/sbin:/usr/sbin:/usr/sbin =~ /home/ubuntu/\.local/bin:/home/ubuntu/bin: ]] 2024-06-03T08:37:53.764 INFO:tasks.workunit.client.0.smithi088.stderr:++ awk '{print $1}' 2024-06-03T08:37:53.764 INFO:tasks.workunit.client.0.smithi088.stderr:+++ umask 2024-06-03T08:37:53.764 INFO:tasks.workunit.client.0.smithi088.stderr:+ export PATH 2024-06-03T08:37:53.764 INFO:tasks.workunit.client.0.smithi088.stderr:+ '[' -d /home/ubuntu/.bashrc.d ']' 2024-06-03T08:37:53.764 INFO:tasks.workunit.client.0.smithi088.stderr:+ unset rc 2024-06-03T08:37:53.764 INFO:tasks.workunit.client.0.smithi088.stderr:+ tee ceph_test_rados_api_c_write_operations.log 2024-06-03T08:37:53.764 INFO:tasks.workunit.client.0.smithi088.stderr:++ '[' 0022 -eq 0 ']' 2024-06-03T08:37:53.764 INFO:tasks.workunit.client.0.smithi088.stderr:++ SHELL=/bin/bash 2024-06-03T08:37:53.764 INFO:tasks.workunit.client.0.smithi088.stderr:++ for i in /etc/profile.d/*.sh 2024-06-03T08:37:53.764 INFO:tasks.workunit.client.0.smithi088.stderr:++ '[' -r /etc/profile.d/bash_completion.sh ']' 2024-06-03T08:37:53.764 INFO:tasks.workunit.client.0.smithi088.stderr:++ '[' '' ']' 2024-06-03T08:37:53.764 INFO:tasks.workunit.client.0.smithi088.stderr:++ . /etc/profile.d/bash_completion.sh 2024-06-03T08:37:53.765 INFO:tasks.workunit.client.0.smithi088.stderr:+++ '[' 'x5.1.8(1)-release' '!=' x -a x '!=' x -a x = x ']' 2024-06-03T08:37:53.765 INFO:tasks.workunit.client.0.smithi088.stderr:++ for i in /etc/profile.d/*.sh 2024-06-03T08:37:53.765 INFO:tasks.workunit.client.0.smithi088.stderr:++ '[' -r /etc/profile.d/cobbler.sh ']' 2024-06-03T08:37:53.765 INFO:tasks.workunit.client.0.smithi088.stderr:++ '[' '' ']' 2024-06-03T08:37:53.765 INFO:tasks.workunit.client.0.smithi088.stderr:++ . /etc/profile.d/cobbler.sh 2024-06-03T08:37:53.765 INFO:tasks.workunit.client.0.smithi088.stderr:+++ export COBBLER_SERVER=172.21.0.11 2024-06-03T08:37:53.765 INFO:tasks.workunit.client.0.smithi088.stderr:+++ COBBLER_SERVER=172.21.0.11 2024-06-03T08:37:53.765 INFO:tasks.workunit.client.0.smithi088.stderr:++ for i in /etc/profile.d/*.sh 2024-06-03T08:37:53.765 INFO:tasks.workunit.client.0.smithi088.stderr:++ '[' -r /etc/profile.d/colorgrep.sh ']' 2024-06-03T08:37:53.765 INFO:tasks.workunit.client.0.smithi088.stderr:++ '[' '' ']' 2024-06-03T08:37:53.765 INFO:tasks.workunit.client.0.smithi088.stderr:++ . /etc/profile.d/colorgrep.sh 2024-06-03T08:37:53.765 INFO:tasks.workunit.client.0.smithi088.stderr:+++ /usr/libexec/grepconf.sh -c 2024-06-03T08:37:53.765 INFO:tasks.workunit.client.0.smithi088.stderr:+ sed 's/^/ api_c_write_operations: /' 2024-06-03T08:37:53.766 INFO:tasks.workunit.client.0.smithi088.stderr:+ ff=open_pools_parallel 2024-06-03T08:37:53.766 INFO:tasks.workunit.client.0.smithi088.stderr:+ pid=40936 2024-06-03T08:37:53.766 INFO:tasks.workunit.client.0.smithi088.stderr:+ echo 'test open_pools_parallel on pid 40936' 2024-06-03T08:37:53.766 INFO:tasks.workunit.client.0.smithi088.stdout: api_asio: 2024-06-03T08:37:53.591+0000 7f514d647780 1 librados: calling monclienttest open_pools_parallel on pid 40936 2024-06-03T08:37:53.771 INFO:tasks.workunit.client.0.smithi088.stderr:+ pids[$f]=40936 2024-06-03T08:37:53.771 INFO:tasks.workunit.client.0.smithi088.stderr:+ for f in api_aio api_aio_pp api_io api_io_pp api_asio api_list api_lock api_lock_pp api_misc api_misc_pp api_tier_pp api_pool api_snapshots api_snapshots_pp api_stat api_stat_pp api_watch_notify api_watch_notify_pp api_cmd api_cmd_pp api_service api_service_pp api_c_write_operations api_c_read_operations api_cls_remote_reads list_parallel open_pools_parallel delete_pools_parallel 2024-06-03T08:37:53.771 INFO:tasks.workunit.client.0.smithi088.stderr:+ '[' 1 -eq 1 ']' 2024-06-03T08:37:53.772 INFO:tasks.workunit.client.0.smithi088.stderr:+ ceph_test_rados_api_c_write_operations 2024-06-03T08:37:53.772 INFO:tasks.workunit.client.0.smithi088.stderr:++ printf %25s delete_pools_parallel 2024-06-03T08:37:53.772 INFO:tasks.workunit.client.0.smithi088.stderr:+ r=' delete_pools_parallel' 2024-06-03T08:37:53.772 INFO:tasks.workunit.client.0.smithi088.stderr:+ bash -o pipefail -exc 'ceph_test_rados_open_pools_parallel 2>&1 | tee ceph_test_rados_open_pools_parallel.log | sed "s/^/ open_pools_parallel: /"' 2024-06-03T08:37:53.772 INFO:tasks.workunit.client.0.smithi088.stderr:++ echo delete_pools_parallel 2024-06-03T08:37:53.772 INFO:tasks.workunit.client.0.smithi088.stderr:++ awk '{print $1}' 2024-06-03T08:37:53.772 INFO:tasks.workunit.client.0.smithi088.stderr:+++ alias 'grep=grep --color=auto' 2024-06-03T08:37:53.772 INFO:tasks.workunit.client.0.smithi088.stderr:+++ alias 'egrep=egrep --color=auto' 2024-06-03T08:37:53.772 INFO:tasks.workunit.client.0.smithi088.stderr:+++ alias 'fgrep=fgrep --color=auto' 2024-06-03T08:37:53.772 INFO:tasks.workunit.client.0.smithi088.stderr:++ for i in /etc/profile.d/*.sh 2024-06-03T08:37:53.772 INFO:tasks.workunit.client.0.smithi088.stderr:++ '[' -r /etc/profile.d/colorls.sh ']' 2024-06-03T08:37:53.772 INFO:tasks.workunit.client.0.smithi088.stderr:++ '[' '' ']' 2024-06-03T08:37:53.772 INFO:tasks.workunit.client.0.smithi088.stderr:++ . /etc/profile.d/colorls.sh 2024-06-03T08:37:53.772 INFO:tasks.workunit.client.0.smithi088.stderr:+ '[' -f /etc/bashrc ']' 2024-06-03T08:37:53.773 INFO:tasks.workunit.client.0.smithi088.stderr:+ . /etc/bashrc 2024-06-03T08:37:53.773 INFO:tasks.workunit.client.0.smithi088.stderr:+++ '[' '!' -t 0 ']' 2024-06-03T08:37:53.773 INFO:tasks.workunit.client.0.smithi088.stderr:+++ return 2024-06-03T08:37:53.773 INFO:tasks.workunit.client.0.smithi088.stderr:++ for i in /etc/profile.d/*.sh 2024-06-03T08:37:53.773 INFO:tasks.workunit.client.0.smithi088.stderr:++ '[' -r /etc/profile.d/colorsysstat.sh ']' 2024-06-03T08:37:53.773 INFO:tasks.workunit.client.0.smithi088.stderr:++ '[' '' ']' 2024-06-03T08:37:53.773 INFO:tasks.workunit.client.0.smithi088.stderr:++ . /etc/profile.d/colorsysstat.sh 2024-06-03T08:37:53.773 INFO:tasks.workunit.client.0.smithi088.stderr:+++ export S_COLORS=auto 2024-06-03T08:37:53.773 INFO:tasks.workunit.client.0.smithi088.stderr:+++ S_COLORS=auto 2024-06-03T08:37:53.773 INFO:tasks.workunit.client.0.smithi088.stderr:++ for i in /etc/profile.d/*.sh 2024-06-03T08:37:53.773 INFO:tasks.workunit.client.0.smithi088.stderr:++ '[' -r /etc/profile.d/colorxzgrep.sh ']' 2024-06-03T08:37:53.773 INFO:tasks.workunit.client.0.smithi088.stderr:++ '[' '' ']' 2024-06-03T08:37:53.773 INFO:tasks.workunit.client.0.smithi088.stderr:++ . /etc/profile.d/colorxzgrep.sh 2024-06-03T08:37:53.773 INFO:tasks.workunit.client.0.smithi088.stderr:+++ alias 'xzgrep=xzgrep --color=auto' 2024-06-03T08:37:53.774 INFO:tasks.workunit.client.0.smithi088.stderr:+++ /usr/libexec/grepconf.sh -c 2024-06-03T08:37:53.774 INFO:tasks.workunit.client.0.smithi088.stderr:+++ alias 'xzegrep=xzegrep --color=auto' 2024-06-03T08:37:53.774 INFO:tasks.workunit.client.0.smithi088.stderr:+++ alias 'xzfgrep=xzfgrep --color=auto' 2024-06-03T08:37:53.774 INFO:tasks.workunit.client.0.smithi088.stderr:++ for i in /etc/profile.d/*.sh 2024-06-03T08:37:53.774 INFO:tasks.workunit.client.0.smithi088.stderr:++ '[' -r /etc/profile.d/colorzgrep.sh ']' 2024-06-03T08:37:53.774 INFO:tasks.workunit.client.0.smithi088.stderr:++ '[' '' ']' 2024-06-03T08:37:53.774 INFO:tasks.workunit.client.0.smithi088.stderr:++ . /etc/profile.d/colorzgrep.sh 2024-06-03T08:37:53.774 INFO:tasks.workunit.client.0.smithi088.stderr:+++ '[' -f /usr/libexec/grepconf.sh ']' 2024-06-03T08:37:53.774 INFO:tasks.workunit.client.0.smithi088.stderr:+++ /usr/libexec/grepconf.sh -c 2024-06-03T08:37:53.774 INFO:tasks.workunit.client.0.smithi088.stderr:++ '[' -z '' ']' 2024-06-03T08:37:53.774 INFO:tasks.workunit.client.0.smithi088.stderr:++ BASHRCSOURCED=Y 2024-06-03T08:37:53.774 INFO:tasks.workunit.client.0.smithi088.stderr:++ '[' '' ']' 2024-06-03T08:37:53.774 INFO:tasks.workunit.client.0.smithi088.stderr:++ shopt -q login_shell 2024-06-03T08:37:53.775 INFO:tasks.workunit.client.0.smithi088.stderr:+++ umask 2024-06-03T08:37:53.775 INFO:tasks.workunit.client.0.smithi088.stderr:++ '[' 0022 -eq 0 ']' 2024-06-03T08:37:53.775 INFO:tasks.workunit.client.0.smithi088.stderr:++ SHELL=/bin/bash 2024-06-03T08:37:53.775 INFO:tasks.workunit.client.0.smithi088.stderr:++ for i in /etc/profile.d/*.sh 2024-06-03T08:37:53.775 INFO:tasks.workunit.client.0.smithi088.stderr:++ '[' -r /etc/profile.d/bash_completion.sh ']' 2024-06-03T08:37:53.775 INFO:tasks.workunit.client.0.smithi088.stderr:++ '[' '' ']' 2024-06-03T08:37:53.775 INFO:tasks.workunit.client.0.smithi088.stderr:++ . /etc/profile.d/bash_completion.sh 2024-06-03T08:37:53.775 INFO:tasks.workunit.client.0.smithi088.stderr:+++ '[' 'x5.1.8(1)-release' '!=' x -a x '!=' x -a x = x ']' 2024-06-03T08:37:53.775 INFO:tasks.workunit.client.0.smithi088.stderr:++ for i in /etc/profile.d/*.sh 2024-06-03T08:37:53.775 INFO:tasks.workunit.client.0.smithi088.stderr:++ '[' -r /etc/profile.d/cobbler.sh ']' 2024-06-03T08:37:53.775 INFO:tasks.workunit.client.0.smithi088.stderr:++ '[' '' ']' 2024-06-03T08:37:53.775 INFO:tasks.workunit.client.0.smithi088.stderr:++ . /etc/profile.d/cobbler.sh 2024-06-03T08:37:53.775 INFO:tasks.workunit.client.0.smithi088.stderr:+++ export COBBLER_SERVER=172.21.0.11 2024-06-03T08:37:53.776 INFO:tasks.workunit.client.0.smithi088.stderr:+++ COBBLER_SERVER=172.21.0.11 2024-06-03T08:37:53.776 INFO:tasks.workunit.client.0.smithi088.stderr:++ for i in /etc/profile.d/*.sh 2024-06-03T08:37:53.776 INFO:tasks.workunit.client.0.smithi088.stderr:++ '[' -r /etc/profile.d/colorgrep.sh ']' 2024-06-03T08:37:53.776 INFO:tasks.workunit.client.0.smithi088.stderr:++ '[' '' ']' 2024-06-03T08:37:53.776 INFO:tasks.workunit.client.0.smithi088.stderr:++ . /etc/profile.d/colorgrep.sh 2024-06-03T08:37:53.776 INFO:tasks.workunit.client.0.smithi088.stderr:+++ /usr/libexec/grepconf.sh -c 2024-06-03T08:37:53.776 INFO:tasks.workunit.client.0.smithi088.stderr:+ ff=delete_pools_parallel 2024-06-03T08:37:53.776 INFO:tasks.workunit.client.0.smithi088.stderr:+ pid=40950 2024-06-03T08:37:53.776 INFO:tasks.workunit.client.0.smithi088.stderr:+ echo 'test delete_pools_parallel on pid 40950' 2024-06-03T08:37:53.776 INFO:tasks.workunit.client.0.smithi088.stdout:test delete_pools_parallel on pid 40950 2024-06-03T08:37:53.808 INFO:tasks.workunit.client.0.smithi088.stderr:+ pids[$f]=40950 2024-06-03T08:37:53.808 INFO:tasks.workunit.client.0.smithi088.stderr:+ bash -o pipefail -exc 'ceph_test_rados_delete_pools_parallel 2>&1 | tee ceph_test_rados_delete_pools_parallel.log | sed "s/^/ delete_pools_parallel: /"' 2024-06-03T08:37:53.808 INFO:tasks.workunit.client.0.smithi088.stderr:+ ret=0 2024-06-03T08:37:53.808 INFO:tasks.workunit.client.0.smithi088.stderr:+ '[' 1 -eq 1 ']' 2024-06-03T08:37:53.808 INFO:tasks.workunit.client.0.smithi088.stderr:+ for t in "${!pids[@]}" 2024-06-03T08:37:53.808 INFO:tasks.workunit.client.0.smithi088.stderr:+ pid=40367 2024-06-03T08:37:53.809 INFO:tasks.workunit.client.0.smithi088.stderr:+ wait 40367 2024-06-03T08:37:53.809 INFO:tasks.workunit.client.0.smithi088.stderr:+++ alias 'zgrep=zgrep --color=auto' 2024-06-03T08:37:53.809 INFO:tasks.workunit.client.0.smithi088.stderr:+++ alias 'zfgrep=zfgrep --color=auto' 2024-06-03T08:37:53.809 INFO:tasks.workunit.client.0.smithi088.stderr:+++ alias 'zegrep=zegrep --color=auto' 2024-06-03T08:37:53.809 INFO:tasks.workunit.client.0.smithi088.stderr:++ for i in /etc/profile.d/*.sh 2024-06-03T08:37:53.809 INFO:tasks.workunit.client.0.smithi088.stderr:++ '[' -r /etc/profile.d/debuginfod.sh ']' 2024-06-03T08:37:53.809 INFO:tasks.workunit.client.0.smithi088.stderr:++ '[' '' ']' 2024-06-03T08:37:53.809 INFO:tasks.workunit.client.0.smithi088.stderr:++ . /etc/profile.d/debuginfod.sh 2024-06-03T08:37:53.809 INFO:tasks.workunit.client.0.smithi088.stderr:+++ '[' -z 'https://debuginfod.centos.org/ ' ']' 2024-06-03T08:37:53.809 INFO:tasks.workunit.client.0.smithi088.stderr:++ for i in /etc/profile.d/*.sh 2024-06-03T08:37:53.809 INFO:tasks.workunit.client.0.smithi088.stderr:++ '[' -r /etc/profile.d/gawk.sh ']' 2024-06-03T08:37:53.809 INFO:tasks.workunit.client.0.smithi088.stderr:++ '[' '' ']' 2024-06-03T08:37:53.809 INFO:tasks.workunit.client.0.smithi088.stderr:++ . /etc/profile.d/gawk.sh 2024-06-03T08:37:53.810 INFO:tasks.workunit.client.0.smithi088.stderr:+ '[' -f /etc/bashrc ']' 2024-06-03T08:37:53.810 INFO:tasks.workunit.client.0.smithi088.stderr:+ . /etc/bashrc 2024-06-03T08:37:53.810 INFO:tasks.workunit.client.0.smithi088.stderr:++ for i in /etc/profile.d/*.sh 2024-06-03T08:37:53.810 INFO:tasks.workunit.client.0.smithi088.stderr:++ '[' -r /etc/profile.d/lang.sh ']' 2024-06-03T08:37:53.810 INFO:tasks.workunit.client.0.smithi088.stderr:++ '[' '' ']' 2024-06-03T08:37:53.810 INFO:tasks.workunit.client.0.smithi088.stderr:++ . /etc/profile.d/lang.sh 2024-06-03T08:37:53.810 INFO:tasks.workunit.client.0.smithi088.stderr:+++ unset LANG_backup 2024-06-03T08:37:53.810 INFO:tasks.workunit.client.0.smithi088.stderr:++ '[' -z '' ']' 2024-06-03T08:37:53.810 INFO:tasks.workunit.client.0.smithi088.stderr:++ BASHRCSOURCED=Y 2024-06-03T08:37:53.810 INFO:tasks.workunit.client.0.smithi088.stderr:+++ alias 'grep=grep --color=auto' 2024-06-03T08:37:53.810 INFO:tasks.workunit.client.0.smithi088.stderr:++++ /usr/bin/locale 2024-06-03T08:37:53.810 INFO:tasks.workunit.client.0.smithi088.stderr:+++ alias 'egrep=egrep --color=auto' 2024-06-03T08:37:53.810 INFO:tasks.workunit.client.0.smithi088.stderr:+++ alias 'fgrep=fgrep --color=auto' 2024-06-03T08:37:53.811 INFO:tasks.workunit.client.0.smithi088.stderr:++ for i in /etc/profile.d/*.sh 2024-06-03T08:37:53.811 INFO:tasks.workunit.client.0.smithi088.stderr:++ '[' -r /etc/profile.d/colorls.sh ']' 2024-06-03T08:37:53.811 INFO:tasks.workunit.client.0.smithi088.stderr:++ '[' '' ']' 2024-06-03T08:37:53.811 INFO:tasks.workunit.client.0.smithi088.stderr:++ . /etc/profile.d/colorls.sh 2024-06-03T08:37:53.811 INFO:tasks.workunit.client.0.smithi088.stderr:+++ '[' '!' -t 0 ']' 2024-06-03T08:37:53.811 INFO:tasks.workunit.client.0.smithi088.stderr:++ '[' '' ']' 2024-06-03T08:37:53.811 INFO:tasks.workunit.client.0.smithi088.stderr:++ shopt -q login_shell 2024-06-03T08:37:53.811 INFO:tasks.workunit.client.0.smithi088.stderr:+++ return 2024-06-03T08:37:53.811 INFO:tasks.workunit.client.0.smithi088.stderr:++ for i in /etc/profile.d/*.sh 2024-06-03T08:37:53.811 INFO:tasks.workunit.client.0.smithi088.stderr:++ '[' -r /etc/profile.d/colorsysstat.sh ']' 2024-06-03T08:37:53.811 INFO:tasks.workunit.client.0.smithi088.stderr:++ '[' '' ']' 2024-06-03T08:37:53.811 INFO:tasks.workunit.client.0.smithi088.stderr:++ . /etc/profile.d/colorsysstat.sh 2024-06-03T08:37:53.811 INFO:tasks.workunit.client.0.smithi088.stderr:+++ export S_COLORS=auto 2024-06-03T08:37:53.812 INFO:tasks.workunit.client.0.smithi088.stderr:+++ S_COLORS=auto 2024-06-03T08:37:53.812 INFO:tasks.workunit.client.0.smithi088.stderr:++ for i in /etc/profile.d/*.sh 2024-06-03T08:37:53.812 INFO:tasks.workunit.client.0.smithi088.stderr:++ '[' -r /etc/profile.d/colorxzgrep.sh ']' 2024-06-03T08:37:53.812 INFO:tasks.workunit.client.0.smithi088.stderr:++ '[' '' ']' 2024-06-03T08:37:53.812 INFO:tasks.workunit.client.0.smithi088.stderr:++ . /etc/profile.d/colorxzgrep.sh 2024-06-03T08:37:53.812 INFO:tasks.workunit.client.0.smithi088.stderr:+++ umask 2024-06-03T08:37:53.812 INFO:tasks.workunit.client.0.smithi088.stderr:+++ /usr/libexec/grepconf.sh -c 2024-06-03T08:37:53.812 INFO:tasks.workunit.client.0.smithi088.stderr:++ '[' 0022 -eq 0 ']' 2024-06-03T08:37:53.812 INFO:tasks.workunit.client.0.smithi088.stderr:+++ '[' -n '' ']' 2024-06-03T08:37:53.812 INFO:tasks.workunit.client.0.smithi088.stderr:+++ LANG_backup=en_US.UTF-8 2024-06-03T08:37:53.812 INFO:tasks.workunit.client.0.smithi088.stderr:++ SHELL=/bin/bash 2024-06-03T08:37:53.812 INFO:tasks.workunit.client.0.smithi088.stderr:+++ for config in /etc/locale.conf "${HOME}/.i18n" 2024-06-03T08:37:53.812 INFO:tasks.workunit.client.0.smithi088.stderr:+++ '[' -f /etc/locale.conf ']' 2024-06-03T08:37:53.812 INFO:tasks.workunit.client.0.smithi088.stderr:+++ '[' -x /usr/bin/sed ']' 2024-06-03T08:37:53.813 INFO:tasks.workunit.client.0.smithi088.stderr:++ for i in /etc/profile.d/*.sh 2024-06-03T08:37:53.813 INFO:tasks.workunit.client.0.smithi088.stderr:++ '[' -r /etc/profile.d/bash_completion.sh ']' 2024-06-03T08:37:53.813 INFO:tasks.workunit.client.0.smithi088.stderr:+++ alias 'xzgrep=xzgrep --color=auto' 2024-06-03T08:37:53.813 INFO:tasks.workunit.client.0.smithi088.stderr:++ '[' '' ']' 2024-06-03T08:37:53.813 INFO:tasks.workunit.client.0.smithi088.stderr:++ . /etc/profile.d/bash_completion.sh 2024-06-03T08:37:53.813 INFO:tasks.workunit.client.0.smithi088.stderr:+++ alias 'xzegrep=xzegrep --color=auto' 2024-06-03T08:37:53.813 INFO:tasks.workunit.client.0.smithi088.stderr:+++ '[' 'x5.1.8(1)-release' '!=' x -a x '!=' x -a x = x ']' 2024-06-03T08:37:53.813 INFO:tasks.workunit.client.0.smithi088.stderr:++ for i in /etc/profile.d/*.sh 2024-06-03T08:37:53.813 INFO:tasks.workunit.client.0.smithi088.stderr:++ '[' -r /etc/profile.d/cobbler.sh ']' 2024-06-03T08:37:53.813 INFO:tasks.workunit.client.0.smithi088.stderr:+++ alias 'xzfgrep=xzfgrep --color=auto' 2024-06-03T08:37:53.813 INFO:tasks.workunit.client.0.smithi088.stderr:++ '[' '' ']' 2024-06-03T08:37:53.813 INFO:tasks.workunit.client.0.smithi088.stderr:++ . /etc/profile.d/cobbler.sh 2024-06-03T08:37:53.813 INFO:tasks.workunit.client.0.smithi088.stderr:++ for i in /etc/profile.d/*.sh 2024-06-03T08:37:53.813 INFO:tasks.workunit.client.0.smithi088.stderr:+++ export COBBLER_SERVER=172.21.0.11 2024-06-03T08:37:53.814 INFO:tasks.workunit.client.0.smithi088.stderr:+++ COBBLER_SERVER=172.21.0.11 2024-06-03T08:37:53.814 INFO:tasks.workunit.client.0.smithi088.stderr:++ '[' -r /etc/profile.d/colorzgrep.sh ']' 2024-06-03T08:37:53.814 INFO:tasks.workunit.client.0.smithi088.stderr:++ for i in /etc/profile.d/*.sh 2024-06-03T08:37:53.814 INFO:tasks.workunit.client.0.smithi088.stderr:++ '[' '' ']' 2024-06-03T08:37:53.814 INFO:tasks.workunit.client.0.smithi088.stderr:++ '[' -r /etc/profile.d/colorgrep.sh ']' 2024-06-03T08:37:53.814 INFO:tasks.workunit.client.0.smithi088.stderr:++ . /etc/profile.d/colorzgrep.sh 2024-06-03T08:37:53.814 INFO:tasks.workunit.client.0.smithi088.stderr:++ '[' '' ']' 2024-06-03T08:37:53.814 INFO:tasks.workunit.client.0.smithi088.stderr:++ . /etc/profile.d/colorgrep.sh 2024-06-03T08:37:53.814 INFO:tasks.workunit.client.0.smithi088.stderr:+++ '[' -f /usr/libexec/grepconf.sh ']' 2024-06-03T08:37:53.814 INFO:tasks.workunit.client.0.smithi088.stderr:+++ /usr/libexec/grepconf.sh -c 2024-06-03T08:37:53.814 INFO:tasks.workunit.client.0.smithi088.stderr:+++ /usr/libexec/grepconf.sh -c 2024-06-03T08:37:53.814 INFO:tasks.workunit.client.0.smithi088.stderr:++++ /usr/bin/sed -r -e 's/^[[:blank:]]*([[:upper:]_]+)=([[:print:][:digit:]\._-]+|"[[:print:][:digit:]\._-]+")/export \1=\2/;t;d' /etc/locale.conf 2024-06-03T08:37:53.814 INFO:tasks.workunit.client.0.smithi088.stderr:+++ eval export 'LANG="en_US.UTF-8"' 2024-06-03T08:37:53.815 INFO:tasks.workunit.client.0.smithi088.stderr:++++ export LANG=en_US.UTF-8 2024-06-03T08:37:53.815 INFO:tasks.workunit.client.0.smithi088.stderr:++++ LANG=en_US.UTF-8 2024-06-03T08:37:53.815 INFO:tasks.workunit.client.0.smithi088.stderr:+++ for config in /etc/locale.conf "${HOME}/.i18n" 2024-06-03T08:37:53.815 INFO:tasks.workunit.client.0.smithi088.stderr:+++ '[' -f /home/ubuntu/.i18n ']' 2024-06-03T08:37:53.815 INFO:tasks.workunit.client.0.smithi088.stderr:+++ '[' -n en_US.UTF-8 ']' 2024-06-03T08:37:53.815 INFO:tasks.workunit.client.0.smithi088.stderr:+++ LANG=en_US.UTF-8 2024-06-03T08:37:53.815 INFO:tasks.workunit.client.0.smithi088.stderr:+++ unset LANG_backup config 2024-06-03T08:37:53.815 INFO:tasks.workunit.client.0.smithi088.stderr:+++ '[' -n '' ']' 2024-06-03T08:37:53.815 INFO:tasks.workunit.client.0.smithi088.stderr:+++ '[' -n en_US.UTF-8 ']' 2024-06-03T08:37:53.815 INFO:tasks.workunit.client.0.smithi088.stderr:+++ '[' dumb = linux ']' 2024-06-03T08:37:53.815 INFO:tasks.workunit.client.0.smithi088.stderr:++ for i in /etc/profile.d/*.sh 2024-06-03T08:37:53.815 INFO:tasks.workunit.client.0.smithi088.stderr:++ '[' -r /etc/profile.d/less.sh ']' 2024-06-03T08:37:53.815 INFO:tasks.workunit.client.0.smithi088.stderr:++ '[' '' ']' 2024-06-03T08:37:53.816 INFO:tasks.workunit.client.0.smithi088.stderr:++ . /etc/profile.d/less.sh 2024-06-03T08:37:53.816 INFO:tasks.workunit.client.0.smithi088.stderr:+++ '[' -z '||/usr/bin/lesspipe.sh %s' ']' 2024-06-03T08:37:53.816 INFO:tasks.workunit.client.0.smithi088.stderr:++ for i in /etc/profile.d/*.sh 2024-06-03T08:37:53.816 INFO:tasks.workunit.client.0.smithi088.stderr:++ '[' -r /etc/profile.d/which2.sh ']' 2024-06-03T08:37:53.816 INFO:tasks.workunit.client.0.smithi088.stderr:++ '[' '' ']' 2024-06-03T08:37:53.816 INFO:tasks.workunit.client.0.smithi088.stderr:++ . /etc/profile.d/which2.sh 2024-06-03T08:37:53.816 INFO:tasks.workunit.client.0.smithi088.stderr:+++ case "$(basename $(readlink /proc/$$/exe))" in 2024-06-03T08:37:53.816 INFO:tasks.workunit.client.0.smithi088.stderr:+++ alias 'grep=grep --color=auto' 2024-06-03T08:37:53.816 INFO:tasks.workunit.client.0.smithi088.stderr:+++ alias 'egrep=egrep --color=auto' 2024-06-03T08:37:53.816 INFO:tasks.workunit.client.0.smithi088.stderr:+++ alias 'fgrep=fgrep --color=auto' 2024-06-03T08:37:53.816 INFO:tasks.workunit.client.0.smithi088.stderr:++ for i in /etc/profile.d/*.sh 2024-06-03T08:37:53.816 INFO:tasks.workunit.client.0.smithi088.stderr:++ '[' -r /etc/profile.d/colorls.sh ']' 2024-06-03T08:37:53.816 INFO:tasks.workunit.client.0.smithi088.stderr:++ '[' '' ']' 2024-06-03T08:37:53.817 INFO:tasks.workunit.client.0.smithi088.stderr:++ . /etc/profile.d/colorls.sh 2024-06-03T08:37:53.817 INFO:tasks.workunit.client.0.smithi088.stderr:+++ '[' '!' -t 0 ']' 2024-06-03T08:37:53.817 INFO:tasks.workunit.client.0.smithi088.stderr:+++ return 2024-06-03T08:37:53.817 INFO:tasks.workunit.client.0.smithi088.stderr:++ for i in /etc/profile.d/*.sh 2024-06-03T08:37:53.817 INFO:tasks.workunit.client.0.smithi088.stderr:++ '[' -r /etc/profile.d/colorsysstat.sh ']' 2024-06-03T08:37:53.817 INFO:tasks.workunit.client.0.smithi088.stderr:++ '[' '' ']' 2024-06-03T08:37:53.817 INFO:tasks.workunit.client.0.smithi088.stderr:++ . /etc/profile.d/colorsysstat.sh 2024-06-03T08:37:53.817 INFO:tasks.workunit.client.0.smithi088.stderr:+++ export S_COLORS=auto 2024-06-03T08:37:53.817 INFO:tasks.workunit.client.0.smithi088.stderr:+++ S_COLORS=auto 2024-06-03T08:37:53.817 INFO:tasks.workunit.client.0.smithi088.stderr:++ for i in /etc/profile.d/*.sh 2024-06-03T08:37:53.817 INFO:tasks.workunit.client.0.smithi088.stderr:++ '[' -r /etc/profile.d/colorxzgrep.sh ']' 2024-06-03T08:37:53.817 INFO:tasks.workunit.client.0.smithi088.stderr:++ '[' '' ']' 2024-06-03T08:37:53.817 INFO:tasks.workunit.client.0.smithi088.stderr:++ . /etc/profile.d/colorxzgrep.sh 2024-06-03T08:37:53.817 INFO:tasks.workunit.client.0.smithi088.stderr:+++ /usr/libexec/grepconf.sh -c 2024-06-03T08:37:53.818 INFO:tasks.workunit.client.0.smithi088.stderr:+++ alias 'xzgrep=xzgrep --color=auto' 2024-06-03T08:37:53.818 INFO:tasks.workunit.client.0.smithi088.stderr:+++ alias 'xzegrep=xzegrep --color=auto' 2024-06-03T08:37:53.818 INFO:tasks.workunit.client.0.smithi088.stderr:+++ alias 'xzfgrep=xzfgrep --color=auto' 2024-06-03T08:37:53.818 INFO:tasks.workunit.client.0.smithi088.stderr:++ for i in /etc/profile.d/*.sh 2024-06-03T08:37:53.818 INFO:tasks.workunit.client.0.smithi088.stderr:++ '[' -r /etc/profile.d/colorzgrep.sh ']' 2024-06-03T08:37:53.818 INFO:tasks.workunit.client.0.smithi088.stderr:++ '[' '' ']' 2024-06-03T08:37:53.818 INFO:tasks.workunit.client.0.smithi088.stderr:++ . /etc/profile.d/colorzgrep.sh 2024-06-03T08:37:53.818 INFO:tasks.workunit.client.0.smithi088.stderr:+++ '[' -f /usr/libexec/grepconf.sh ']' 2024-06-03T08:37:53.818 INFO:tasks.workunit.client.0.smithi088.stderr:+++ /usr/libexec/grepconf.sh -c 2024-06-03T08:37:53.818 INFO:tasks.workunit.client.0.smithi088.stderr:+++++ readlink /proc/40836/exe 2024-06-03T08:37:53.818 INFO:tasks.workunit.client.0.smithi088.stderr:++++ basename /usr/bin/bash 2024-06-03T08:37:53.818 INFO:tasks.workunit.client.0.smithi088.stderr:+++ alias 'zgrep=zgrep --color=auto' 2024-06-03T08:37:53.818 INFO:tasks.workunit.client.0.smithi088.stderr:+++ alias 'zfgrep=zfgrep --color=auto' 2024-06-03T08:37:53.818 INFO:tasks.workunit.client.0.smithi088.stderr:+++ alias 'zegrep=zegrep --color=auto' 2024-06-03T08:37:53.819 INFO:tasks.workunit.client.0.smithi088.stderr:++ for i in /etc/profile.d/*.sh 2024-06-03T08:37:53.819 INFO:tasks.workunit.client.0.smithi088.stderr:++ '[' -r /etc/profile.d/debuginfod.sh ']' 2024-06-03T08:37:53.819 INFO:tasks.workunit.client.0.smithi088.stderr:++ '[' '' ']' 2024-06-03T08:37:53.819 INFO:tasks.workunit.client.0.smithi088.stderr:++ . /etc/profile.d/debuginfod.sh 2024-06-03T08:37:53.819 INFO:tasks.workunit.client.0.smithi088.stderr:+++ '[' -z 'https://debuginfod.centos.org/ ' ']' 2024-06-03T08:37:53.819 INFO:tasks.workunit.client.0.smithi088.stderr:++ for i in /etc/profile.d/*.sh 2024-06-03T08:37:53.819 INFO:tasks.workunit.client.0.smithi088.stderr:++ '[' -r /etc/profile.d/gawk.sh ']' 2024-06-03T08:37:53.819 INFO:tasks.workunit.client.0.smithi088.stderr:++ '[' '' ']' 2024-06-03T08:37:53.819 INFO:tasks.workunit.client.0.smithi088.stderr:++ . /etc/profile.d/gawk.sh 2024-06-03T08:37:53.819 INFO:tasks.workunit.client.0.smithi088.stderr:++ for i in /etc/profile.d/*.sh 2024-06-03T08:37:53.819 INFO:tasks.workunit.client.0.smithi088.stderr:++ '[' -r /etc/profile.d/lang.sh ']' 2024-06-03T08:37:53.819 INFO:tasks.workunit.client.0.smithi088.stderr:++ '[' '' ']' 2024-06-03T08:37:53.819 INFO:tasks.workunit.client.0.smithi088.stderr:++ . /etc/profile.d/lang.sh 2024-06-03T08:37:53.820 INFO:tasks.workunit.client.0.smithi088.stderr:+++ unset LANG_backup 2024-06-03T08:37:53.820 INFO:tasks.workunit.client.0.smithi088.stderr:+++ which_declare='declare -f' 2024-06-03T08:37:53.820 INFO:tasks.workunit.client.0.smithi088.stderr:+++ which_opt=-f 2024-06-03T08:37:53.820 INFO:tasks.workunit.client.0.smithi088.stderr:+++ export which_declare 2024-06-03T08:37:53.820 INFO:tasks.workunit.client.0.smithi088.stderr:+++ export -f which 2024-06-03T08:37:53.820 INFO:tasks.workunit.client.0.smithi088.stderr:++ unset i 2024-06-03T08:37:53.820 INFO:tasks.workunit.client.0.smithi088.stderr:++ unset -f pathmunge 2024-06-03T08:37:53.820 INFO:tasks.workunit.client.0.smithi088.stderr:+ [[ /home/ubuntu/.local/bin:/home/ubuntu/bin:/usr/local/bin:/usr/bin:/usr/local/sbin:/usr/sbin:/usr/sbin =~ /home/ubuntu/\.local/bin:/home/ubuntu/bin: ]] 2024-06-03T08:37:53.820 INFO:tasks.workunit.client.0.smithi088.stderr:+ export PATH 2024-06-03T08:37:53.820 INFO:tasks.workunit.client.0.smithi088.stderr:+ '[' -d /home/ubuntu/.bashrc.d ']' 2024-06-03T08:37:53.820 INFO:tasks.workunit.client.0.smithi088.stderr:+ unset rc 2024-06-03T08:37:53.820 INFO:tasks.workunit.client.0.smithi088.stderr:+++ alias 'xzgrep=xzgrep --color=auto' 2024-06-03T08:37:53.820 INFO:tasks.workunit.client.0.smithi088.stderr:+++ alias 'xzegrep=xzegrep --color=auto' 2024-06-03T08:37:53.820 INFO:tasks.workunit.client.0.smithi088.stderr:+++ alias 'grep=grep --color=auto' 2024-06-03T08:37:53.821 INFO:tasks.workunit.client.0.smithi088.stderr:+++ alias 'xzfgrep=xzfgrep --color=auto' 2024-06-03T08:37:53.821 INFO:tasks.workunit.client.0.smithi088.stderr:++ for i in /etc/profile.d/*.sh 2024-06-03T08:37:53.821 INFO:tasks.workunit.client.0.smithi088.stderr:++ '[' -r /etc/profile.d/colorzgrep.sh ']' 2024-06-03T08:37:53.821 INFO:tasks.workunit.client.0.smithi088.stderr:++ '[' '' ']' 2024-06-03T08:37:53.821 INFO:tasks.workunit.client.0.smithi088.stderr:++ . /etc/profile.d/colorzgrep.sh 2024-06-03T08:37:53.821 INFO:tasks.workunit.client.0.smithi088.stderr:+++ alias 'egrep=egrep --color=auto' 2024-06-03T08:37:53.821 INFO:tasks.workunit.client.0.smithi088.stderr:+++ '[' -f /usr/libexec/grepconf.sh ']' 2024-06-03T08:37:53.821 INFO:tasks.workunit.client.0.smithi088.stderr:+++ /usr/libexec/grepconf.sh -c 2024-06-03T08:37:53.821 INFO:tasks.workunit.client.0.smithi088.stderr:+++ alias 'fgrep=fgrep --color=auto' 2024-06-03T08:37:53.821 INFO:tasks.workunit.client.0.smithi088.stderr:+ ceph_test_rados_api_c_read_operations 2024-06-03T08:37:53.821 INFO:tasks.workunit.client.0.smithi088.stderr:++ for i in /etc/profile.d/*.sh 2024-06-03T08:37:53.821 INFO:tasks.workunit.client.0.smithi088.stderr:++ '[' -r /etc/profile.d/colorls.sh ']' 2024-06-03T08:37:53.821 INFO:tasks.workunit.client.0.smithi088.stderr:++ '[' '' ']' 2024-06-03T08:37:53.822 INFO:tasks.workunit.client.0.smithi088.stderr:++ . /etc/profile.d/colorls.sh 2024-06-03T08:37:53.822 INFO:tasks.workunit.client.0.smithi088.stderr:+++ '[' '!' -t 0 ']' 2024-06-03T08:37:53.822 INFO:tasks.workunit.client.0.smithi088.stderr:+++ return 2024-06-03T08:37:53.822 INFO:tasks.workunit.client.0.smithi088.stderr:++ for i in /etc/profile.d/*.sh 2024-06-03T08:37:53.822 INFO:tasks.workunit.client.0.smithi088.stderr:++ '[' -r /etc/profile.d/colorsysstat.sh ']' 2024-06-03T08:37:53.822 INFO:tasks.workunit.client.0.smithi088.stderr:++ '[' '' ']' 2024-06-03T08:37:53.822 INFO:tasks.workunit.client.0.smithi088.stderr:++ . /etc/profile.d/colorsysstat.sh 2024-06-03T08:37:53.822 INFO:tasks.workunit.client.0.smithi088.stderr:+++ export S_COLORS=auto 2024-06-03T08:37:53.822 INFO:tasks.workunit.client.0.smithi088.stderr:+++ S_COLORS=auto 2024-06-03T08:37:53.822 INFO:tasks.workunit.client.0.smithi088.stderr:++ for i in /etc/profile.d/*.sh 2024-06-03T08:37:53.822 INFO:tasks.workunit.client.0.smithi088.stderr:++ '[' -r /etc/profile.d/colorxzgrep.sh ']' 2024-06-03T08:37:53.822 INFO:tasks.workunit.client.0.smithi088.stderr:++ '[' '' ']' 2024-06-03T08:37:53.822 INFO:tasks.workunit.client.0.smithi088.stderr:++ . /etc/profile.d/colorxzgrep.sh 2024-06-03T08:37:53.822 INFO:tasks.workunit.client.0.smithi088.stderr:+++ /usr/libexec/grepconf.sh -c 2024-06-03T08:37:53.823 INFO:tasks.workunit.client.0.smithi088.stderr:+ tee ceph_test_rados_api_c_read_operations.log 2024-06-03T08:37:53.823 INFO:tasks.workunit.client.0.smithi088.stderr:+ sed 's/^/ api_c_read_operations: /' 2024-06-03T08:37:53.823 INFO:tasks.workunit.client.0.smithi088.stderr:+++ alias 'zgrep=zgrep --color=auto' 2024-06-03T08:37:53.823 INFO:tasks.workunit.client.0.smithi088.stderr:+++ alias 'zfgrep=zfgrep --color=auto' 2024-06-03T08:37:53.823 INFO:tasks.workunit.client.0.smithi088.stderr:+++ alias 'zegrep=zegrep --color=auto' 2024-06-03T08:37:53.823 INFO:tasks.workunit.client.0.smithi088.stderr:++ for i in /etc/profile.d/*.sh 2024-06-03T08:37:53.823 INFO:tasks.workunit.client.0.smithi088.stderr:++ '[' -r /etc/profile.d/debuginfod.sh ']' 2024-06-03T08:37:53.823 INFO:tasks.workunit.client.0.smithi088.stderr:++ '[' '' ']' 2024-06-03T08:37:53.823 INFO:tasks.workunit.client.0.smithi088.stderr:++ . /etc/profile.d/debuginfod.sh 2024-06-03T08:37:53.823 INFO:tasks.workunit.client.0.smithi088.stderr:+++ '[' -z 'https://debuginfod.centos.org/ ' ']' 2024-06-03T08:37:53.823 INFO:tasks.workunit.client.0.smithi088.stderr:++ for i in /etc/profile.d/*.sh 2024-06-03T08:37:53.823 INFO:tasks.workunit.client.0.smithi088.stderr:++ '[' -r /etc/profile.d/gawk.sh ']' 2024-06-03T08:37:53.823 INFO:tasks.workunit.client.0.smithi088.stderr:++ '[' '' ']' 2024-06-03T08:37:53.824 INFO:tasks.workunit.client.0.smithi088.stderr:++ . /etc/profile.d/gawk.sh 2024-06-03T08:37:53.824 INFO:tasks.workunit.client.0.smithi088.stderr:++ for i in /etc/profile.d/*.sh 2024-06-03T08:37:53.824 INFO:tasks.workunit.client.0.smithi088.stderr:++ '[' -r /etc/profile.d/lang.sh ']' 2024-06-03T08:37:53.824 INFO:tasks.workunit.client.0.smithi088.stderr:++ '[' '' ']' 2024-06-03T08:37:53.824 INFO:tasks.workunit.client.0.smithi088.stderr:++ . /etc/profile.d/lang.sh 2024-06-03T08:37:53.824 INFO:tasks.workunit.client.0.smithi088.stderr:++++ /usr/bin/locale 2024-06-03T08:37:53.824 INFO:tasks.workunit.client.0.smithi088.stderr:+++ unset LANG_backup 2024-06-03T08:37:53.824 INFO:tasks.workunit.client.0.smithi088.stderr:++++ /usr/bin/locale 2024-06-03T08:37:53.824 INFO:tasks.workunit.client.0.smithi088.stderr:+++ alias 'zgrep=zgrep --color=auto' 2024-06-03T08:37:53.824 INFO:tasks.workunit.client.0.smithi088.stderr:+++ alias 'zfgrep=zfgrep --color=auto' 2024-06-03T08:37:53.824 INFO:tasks.workunit.client.0.smithi088.stderr:+++ alias 'zegrep=zegrep --color=auto' 2024-06-03T08:37:53.824 INFO:tasks.workunit.client.0.smithi088.stderr:++ for i in /etc/profile.d/*.sh 2024-06-03T08:37:53.824 INFO:tasks.workunit.client.0.smithi088.stderr:++ '[' -r /etc/profile.d/debuginfod.sh ']' 2024-06-03T08:37:53.824 INFO:tasks.workunit.client.0.smithi088.stderr:++ '[' '' ']' 2024-06-03T08:37:53.825 INFO:tasks.workunit.client.0.smithi088.stderr:++ . /etc/profile.d/debuginfod.sh 2024-06-03T08:37:53.825 INFO:tasks.workunit.client.0.smithi088.stderr:+++ '[' -z 'https://debuginfod.centos.org/ ' ']' 2024-06-03T08:37:53.825 INFO:tasks.workunit.client.0.smithi088.stderr:++ for i in /etc/profile.d/*.sh 2024-06-03T08:37:53.825 INFO:tasks.workunit.client.0.smithi088.stderr:++ '[' -r /etc/profile.d/gawk.sh ']' 2024-06-03T08:37:53.825 INFO:tasks.workunit.client.0.smithi088.stderr:++ '[' '' ']' 2024-06-03T08:37:53.825 INFO:tasks.workunit.client.0.smithi088.stderr:++ . /etc/profile.d/gawk.sh 2024-06-03T08:37:53.825 INFO:tasks.workunit.client.0.smithi088.stderr:+++ '[' -n '' ']' 2024-06-03T08:37:53.825 INFO:tasks.workunit.client.0.smithi088.stderr:+++ LANG_backup=en_US.UTF-8 2024-06-03T08:37:53.825 INFO:tasks.workunit.client.0.smithi088.stderr:+++ for config in /etc/locale.conf "${HOME}/.i18n" 2024-06-03T08:37:53.825 INFO:tasks.workunit.client.0.smithi088.stderr:+++ '[' -f /etc/locale.conf ']' 2024-06-03T08:37:53.825 INFO:tasks.workunit.client.0.smithi088.stderr:+++ '[' -x /usr/bin/sed ']' 2024-06-03T08:37:53.825 INFO:tasks.workunit.client.0.smithi088.stderr:++ for i in /etc/profile.d/*.sh 2024-06-03T08:37:53.825 INFO:tasks.workunit.client.0.smithi088.stderr:++ '[' -r /etc/profile.d/lang.sh ']' 2024-06-03T08:37:53.826 INFO:tasks.workunit.client.0.smithi088.stderr:++ '[' '' ']' 2024-06-03T08:37:53.826 INFO:tasks.workunit.client.0.smithi088.stderr:++ . /etc/profile.d/lang.sh 2024-06-03T08:37:53.826 INFO:tasks.workunit.client.0.smithi088.stderr:+++ unset LANG_backup 2024-06-03T08:37:53.826 INFO:tasks.workunit.client.0.smithi088.stderr:++++ /usr/bin/sed -r -e 's/^[[:blank:]]*([[:upper:]_]+)=([[:print:][:digit:]\._-]+|"[[:print:][:digit:]\._-]+")/export \1=\2/;t;d' /etc/locale.conf 2024-06-03T08:37:53.826 INFO:tasks.workunit.client.0.smithi088.stderr:++++ /usr/bin/locale 2024-06-03T08:37:53.826 INFO:tasks.workunit.client.0.smithi088.stderr:+++ '[' -n '' ']' 2024-06-03T08:37:53.826 INFO:tasks.workunit.client.0.smithi088.stderr:+++ LANG_backup=en_US.UTF-8 2024-06-03T08:37:53.826 INFO:tasks.workunit.client.0.smithi088.stderr:+++ for config in /etc/locale.conf "${HOME}/.i18n" 2024-06-03T08:37:53.826 INFO:tasks.workunit.client.0.smithi088.stderr:+++ '[' -f /etc/locale.conf ']' 2024-06-03T08:37:53.826 INFO:tasks.workunit.client.0.smithi088.stderr:+++ '[' -x /usr/bin/sed ']' 2024-06-03T08:37:53.826 INFO:tasks.workunit.client.0.smithi088.stderr:++++ /usr/bin/sed -r -e 's/^[[:blank:]]*([[:upper:]_]+)=([[:print:][:digit:]\._-]+|"[[:print:][:digit:]\._-]+")/export \1=\2/;t;d' /etc/locale.conf 2024-06-03T08:37:53.826 INFO:tasks.workunit.client.0.smithi088.stderr:+++ eval export 'LANG="en_US.UTF-8"' 2024-06-03T08:37:53.826 INFO:tasks.workunit.client.0.smithi088.stderr:+++ '[' -n '' ']' 2024-06-03T08:37:53.827 INFO:tasks.workunit.client.0.smithi088.stderr:+++ LANG_backup=en_US.UTF-8 2024-06-03T08:37:53.827 INFO:tasks.workunit.client.0.smithi088.stderr:+++ for config in /etc/locale.conf "${HOME}/.i18n" 2024-06-03T08:37:53.827 INFO:tasks.workunit.client.0.smithi088.stderr:++++ export LANG=en_US.UTF-8 2024-06-03T08:37:53.827 INFO:tasks.workunit.client.0.smithi088.stderr:++++ LANG=en_US.UTF-8 2024-06-03T08:37:53.827 INFO:tasks.workunit.client.0.smithi088.stderr:+++ for config in /etc/locale.conf "${HOME}/.i18n" 2024-06-03T08:37:53.827 INFO:tasks.workunit.client.0.smithi088.stderr:+++ '[' -f /etc/locale.conf ']' 2024-06-03T08:37:53.827 INFO:tasks.workunit.client.0.smithi088.stderr:+++ '[' -x /usr/bin/sed ']' 2024-06-03T08:37:53.827 INFO:tasks.workunit.client.0.smithi088.stderr:+++ '[' -f /home/ubuntu/.i18n ']' 2024-06-03T08:37:53.827 INFO:tasks.workunit.client.0.smithi088.stderr:+++ '[' -n en_US.UTF-8 ']' 2024-06-03T08:37:53.827 INFO:tasks.workunit.client.0.smithi088.stderr:+++ LANG=en_US.UTF-8 2024-06-03T08:37:53.827 INFO:tasks.workunit.client.0.smithi088.stderr:+++ unset LANG_backup config 2024-06-03T08:37:53.827 INFO:tasks.workunit.client.0.smithi088.stderr:++++ /usr/bin/sed -r -e 's/^[[:blank:]]*([[:upper:]_]+)=([[:print:][:digit:]\._-]+|"[[:print:][:digit:]\._-]+")/export \1=\2/;t;d' /etc/locale.conf 2024-06-03T08:37:53.827 INFO:tasks.workunit.client.0.smithi088.stderr:+++ '[' -n '' ']' 2024-06-03T08:37:53.828 INFO:tasks.workunit.client.0.smithi088.stderr:+++ '[' -n en_US.UTF-8 ']' 2024-06-03T08:37:53.828 INFO:tasks.workunit.client.0.smithi088.stderr:+++ '[' dumb = linux ']' 2024-06-03T08:37:53.828 INFO:tasks.workunit.client.0.smithi088.stderr:++ for i in /etc/profile.d/*.sh 2024-06-03T08:37:53.828 INFO:tasks.workunit.client.0.smithi088.stderr:++ '[' -r /etc/profile.d/less.sh ']' 2024-06-03T08:37:53.828 INFO:tasks.workunit.client.0.smithi088.stderr:++ '[' '' ']' 2024-06-03T08:37:53.828 INFO:tasks.workunit.client.0.smithi088.stderr:++ . /etc/profile.d/less.sh 2024-06-03T08:37:53.828 INFO:tasks.workunit.client.0.smithi088.stderr:+++ '[' -z '||/usr/bin/lesspipe.sh %s' ']' 2024-06-03T08:37:53.828 INFO:tasks.workunit.client.0.smithi088.stderr:+++ eval export 'LANG="en_US.UTF-8"' 2024-06-03T08:37:53.828 INFO:tasks.workunit.client.0.smithi088.stderr:++ for i in /etc/profile.d/*.sh 2024-06-03T08:37:53.828 INFO:tasks.workunit.client.0.smithi088.stderr:++ '[' -r /etc/profile.d/which2.sh ']' 2024-06-03T08:37:53.828 INFO:tasks.workunit.client.0.smithi088.stderr:++ '[' '' ']' 2024-06-03T08:37:53.828 INFO:tasks.workunit.client.0.smithi088.stderr:++ . /etc/profile.d/which2.sh 2024-06-03T08:37:53.828 INFO:tasks.workunit.client.0.smithi088.stderr:+++ case "$(basename $(readlink /proc/$$/exe))" in 2024-06-03T08:37:53.829 INFO:tasks.workunit.client.0.smithi088.stderr:++++ export LANG=en_US.UTF-8 2024-06-03T08:37:53.829 INFO:tasks.workunit.client.0.smithi088.stderr:++++ LANG=en_US.UTF-8 2024-06-03T08:37:53.829 INFO:tasks.workunit.client.0.smithi088.stderr:+++ for config in /etc/locale.conf "${HOME}/.i18n" 2024-06-03T08:37:53.829 INFO:tasks.workunit.client.0.smithi088.stderr:+++ '[' -f /home/ubuntu/.i18n ']' 2024-06-03T08:37:53.829 INFO:tasks.workunit.client.0.smithi088.stderr:+++ '[' -n en_US.UTF-8 ']' 2024-06-03T08:37:53.829 INFO:tasks.workunit.client.0.smithi088.stderr:+++ LANG=en_US.UTF-8 2024-06-03T08:37:53.829 INFO:tasks.workunit.client.0.smithi088.stderr:+++ unset LANG_backup config 2024-06-03T08:37:53.829 INFO:tasks.workunit.client.0.smithi088.stderr:+++ '[' -n '' ']' 2024-06-03T08:37:53.829 INFO:tasks.workunit.client.0.smithi088.stderr:+++++ readlink /proc/40899/exe 2024-06-03T08:37:53.829 INFO:tasks.workunit.client.0.smithi088.stderr:+++ '[' -n en_US.UTF-8 ']' 2024-06-03T08:37:53.829 INFO:tasks.workunit.client.0.smithi088.stderr:+++ '[' dumb = linux ']' 2024-06-03T08:37:53.829 INFO:tasks.workunit.client.0.smithi088.stderr:++ for i in /etc/profile.d/*.sh 2024-06-03T08:37:53.829 INFO:tasks.workunit.client.0.smithi088.stderr:++ '[' -r /etc/profile.d/less.sh ']' 2024-06-03T08:37:53.829 INFO:tasks.workunit.client.0.smithi088.stderr:++ '[' '' ']' 2024-06-03T08:37:53.830 INFO:tasks.workunit.client.0.smithi088.stderr:++ . /etc/profile.d/less.sh 2024-06-03T08:37:53.830 INFO:tasks.workunit.client.0.smithi088.stderr:+++ '[' -z '||/usr/bin/lesspipe.sh %s' ']' 2024-06-03T08:37:53.830 INFO:tasks.workunit.client.0.smithi088.stderr:++ for i in /etc/profile.d/*.sh 2024-06-03T08:37:53.830 INFO:tasks.workunit.client.0.smithi088.stderr:+++ eval export 'LANG="en_US.UTF-8"' 2024-06-03T08:37:53.830 INFO:tasks.workunit.client.0.smithi088.stderr:++++ export LANG=en_US.UTF-8 2024-06-03T08:37:53.830 INFO:tasks.workunit.client.0.smithi088.stderr:++++ LANG=en_US.UTF-8 2024-06-03T08:37:53.830 INFO:tasks.workunit.client.0.smithi088.stderr:++ '[' -r /etc/profile.d/which2.sh ']' 2024-06-03T08:37:53.830 INFO:tasks.workunit.client.0.smithi088.stderr:+++ for config in /etc/locale.conf "${HOME}/.i18n" 2024-06-03T08:37:53.830 INFO:tasks.workunit.client.0.smithi088.stderr:+++ '[' -f /home/ubuntu/.i18n ']' 2024-06-03T08:37:53.830 INFO:tasks.workunit.client.0.smithi088.stderr:+++ '[' -n en_US.UTF-8 ']' 2024-06-03T08:37:53.830 INFO:tasks.workunit.client.0.smithi088.stderr:++ '[' '' ']' 2024-06-03T08:37:53.830 INFO:tasks.workunit.client.0.smithi088.stderr:++ . /etc/profile.d/which2.sh 2024-06-03T08:37:53.831 INFO:tasks.workunit.client.0.smithi088.stderr:+++ LANG=en_US.UTF-8 2024-06-03T08:37:53.831 INFO:tasks.workunit.client.0.smithi088.stderr:+++ case "$(basename $(readlink /proc/$$/exe))" in 2024-06-03T08:37:53.831 INFO:tasks.workunit.client.0.smithi088.stderr:+++ unset LANG_backup config 2024-06-03T08:37:53.831 INFO:tasks.workunit.client.0.smithi088.stderr:+++ '[' -n '' ']' 2024-06-03T08:37:53.831 INFO:tasks.workunit.client.0.smithi088.stderr:+++ '[' -n en_US.UTF-8 ']' 2024-06-03T08:37:53.831 INFO:tasks.workunit.client.0.smithi088.stderr:+++ '[' dumb = linux ']' 2024-06-03T08:37:53.831 INFO:tasks.workunit.client.0.smithi088.stderr:++ for i in /etc/profile.d/*.sh 2024-06-03T08:37:53.831 INFO:tasks.workunit.client.0.smithi088.stderr:++ '[' -r /etc/profile.d/less.sh ']' 2024-06-03T08:37:53.831 INFO:tasks.workunit.client.0.smithi088.stderr:+++++ readlink /proc/40886/exe 2024-06-03T08:37:53.831 INFO:tasks.workunit.client.0.smithi088.stderr:++ '[' '' ']' 2024-06-03T08:37:53.831 INFO:tasks.workunit.client.0.smithi088.stderr:++ . /etc/profile.d/less.sh 2024-06-03T08:37:53.831 INFO:tasks.workunit.client.0.smithi088.stderr:+++ '[' -z '||/usr/bin/lesspipe.sh %s' ']' 2024-06-03T08:37:53.831 INFO:tasks.workunit.client.0.smithi088.stderr:++ for i in /etc/profile.d/*.sh 2024-06-03T08:37:53.831 INFO:tasks.workunit.client.0.smithi088.stderr:++ '[' -r /etc/profile.d/which2.sh ']' 2024-06-03T08:37:53.832 INFO:tasks.workunit.client.0.smithi088.stderr:++ '[' '' ']' 2024-06-03T08:37:53.832 INFO:tasks.workunit.client.0.smithi088.stderr:++++ basename /usr/bin/bash 2024-06-03T08:37:53.832 INFO:tasks.workunit.client.0.smithi088.stderr:++ . /etc/profile.d/which2.sh 2024-06-03T08:37:53.832 INFO:tasks.workunit.client.0.smithi088.stderr:+++ case "$(basename $(readlink /proc/$$/exe))" in 2024-06-03T08:37:53.832 INFO:tasks.workunit.client.0.smithi088.stderr:+++++ readlink /proc/40950/exe 2024-06-03T08:37:53.832 INFO:tasks.workunit.client.0.smithi088.stderr:++++ basename /usr/bin/bash 2024-06-03T08:37:53.832 INFO:tasks.workunit.client.0.smithi088.stderr:+++ which_declare='declare -f' 2024-06-03T08:37:53.832 INFO:tasks.workunit.client.0.smithi088.stderr:+++ which_opt=-f 2024-06-03T08:37:53.832 INFO:tasks.workunit.client.0.smithi088.stderr:+++ export which_declare 2024-06-03T08:37:53.832 INFO:tasks.workunit.client.0.smithi088.stderr:+++ export -f which 2024-06-03T08:37:53.832 INFO:tasks.workunit.client.0.smithi088.stderr:++ unset i 2024-06-03T08:37:53.832 INFO:tasks.workunit.client.0.smithi088.stderr:++ unset -f pathmunge 2024-06-03T08:37:53.832 INFO:tasks.workunit.client.0.smithi088.stderr:+ [[ /home/ubuntu/.local/bin:/home/ubuntu/bin:/usr/local/bin:/usr/bin:/usr/local/sbin:/usr/sbin:/usr/sbin =~ /home/ubuntu/\.local/bin:/home/ubuntu/bin: ]] 2024-06-03T08:37:53.833 INFO:tasks.workunit.client.0.smithi088.stderr:+ export PATH 2024-06-03T08:37:53.833 INFO:tasks.workunit.client.0.smithi088.stderr:+++ which_declare='declare -f' 2024-06-03T08:37:53.833 INFO:tasks.workunit.client.0.smithi088.stderr:+ '[' -d /home/ubuntu/.bashrc.d ']' 2024-06-03T08:37:53.833 INFO:tasks.workunit.client.0.smithi088.stderr:+ unset rc 2024-06-03T08:37:53.833 INFO:tasks.workunit.client.0.smithi088.stderr:+++ which_opt=-f 2024-06-03T08:37:53.833 INFO:tasks.workunit.client.0.smithi088.stderr:+++ export which_declare 2024-06-03T08:37:53.833 INFO:tasks.workunit.client.0.smithi088.stderr:+++ export -f which 2024-06-03T08:37:53.833 INFO:tasks.workunit.client.0.smithi088.stderr:++ unset i 2024-06-03T08:37:53.833 INFO:tasks.workunit.client.0.smithi088.stderr:++++ basename /usr/bin/bash 2024-06-03T08:37:53.833 INFO:tasks.workunit.client.0.smithi088.stderr:++ unset -f pathmunge 2024-06-03T08:37:53.833 INFO:tasks.workunit.client.0.smithi088.stderr:+ [[ /home/ubuntu/.local/bin:/home/ubuntu/bin:/usr/local/bin:/usr/bin:/usr/local/sbin:/usr/sbin:/usr/sbin =~ /home/ubuntu/\.local/bin:/home/ubuntu/bin: ]] 2024-06-03T08:37:53.833 INFO:tasks.workunit.client.0.smithi088.stderr:+ ceph_test_rados_list_parallel 2024-06-03T08:37:53.833 INFO:tasks.workunit.client.0.smithi088.stderr:+ tee ceph_test_rados_list_parallel.log 2024-06-03T08:37:53.833 INFO:tasks.workunit.client.0.smithi088.stderr:+++ which_declare='declare -f' 2024-06-03T08:37:53.834 INFO:tasks.workunit.client.0.smithi088.stderr:+++ which_opt=-f 2024-06-03T08:37:53.834 INFO:tasks.workunit.client.0.smithi088.stderr:+++ export which_declare 2024-06-03T08:37:53.834 INFO:tasks.workunit.client.0.smithi088.stderr:+++ export -f which 2024-06-03T08:37:53.834 INFO:tasks.workunit.client.0.smithi088.stderr:++ unset i 2024-06-03T08:37:53.834 INFO:tasks.workunit.client.0.smithi088.stderr:++ unset -f pathmunge 2024-06-03T08:37:53.834 INFO:tasks.workunit.client.0.smithi088.stderr:+ [[ /home/ubuntu/.local/bin:/home/ubuntu/bin:/usr/local/bin:/usr/bin:/usr/local/sbin:/usr/sbin:/usr/sbin =~ /home/ubuntu/\.local/bin:/home/ubuntu/bin: ]] 2024-06-03T08:37:53.834 INFO:tasks.workunit.client.0.smithi088.stderr:+ export PATH 2024-06-03T08:37:53.834 INFO:tasks.workunit.client.0.smithi088.stderr:+ export PATH 2024-06-03T08:37:53.834 INFO:tasks.workunit.client.0.smithi088.stderr:+ '[' -d /home/ubuntu/.bashrc.d ']' 2024-06-03T08:37:53.834 INFO:tasks.workunit.client.0.smithi088.stderr:+ '[' -d /home/ubuntu/.bashrc.d ']' 2024-06-03T08:37:53.834 INFO:tasks.workunit.client.0.smithi088.stderr:+ unset rc 2024-06-03T08:37:53.834 INFO:tasks.workunit.client.0.smithi088.stderr:+ unset rc 2024-06-03T08:37:53.834 INFO:tasks.workunit.client.0.smithi088.stderr:+ ceph_test_rados_delete_pools_parallel 2024-06-03T08:37:53.835 INFO:tasks.workunit.client.0.smithi088.stderr:+ sed 's/^/ api_cls_remote_reads: /' 2024-06-03T08:37:53.835 INFO:tasks.workunit.client.0.smithi088.stderr:+ tee ceph_test_rados_api_cls_remote_reads.log 2024-06-03T08:37:53.835 INFO:tasks.workunit.client.0.smithi088.stderr:+ ceph_test_rados_api_cls_remote_reads 2024-06-03T08:37:53.835 INFO:tasks.workunit.client.0.smithi088.stderr:+ sed 's/^/ delete_pools_parallel: /' 2024-06-03T08:37:53.835 INFO:tasks.workunit.client.0.smithi088.stderr:+ tee ceph_test_rados_delete_pools_parallel.log 2024-06-03T08:37:53.835 INFO:tasks.workunit.client.0.smithi088.stderr:+++ alias 'xzgrep=xzgrep --color=auto' 2024-06-03T08:37:53.835 INFO:tasks.workunit.client.0.smithi088.stderr:+++ alias 'xzegrep=xzegrep --color=auto' 2024-06-03T08:37:53.835 INFO:tasks.workunit.client.0.smithi088.stderr:+++ alias 'xzfgrep=xzfgrep --color=auto' 2024-06-03T08:37:53.835 INFO:tasks.workunit.client.0.smithi088.stderr:++ for i in /etc/profile.d/*.sh 2024-06-03T08:37:53.835 INFO:tasks.workunit.client.0.smithi088.stderr:++ '[' -r /etc/profile.d/colorzgrep.sh ']' 2024-06-03T08:37:53.835 INFO:tasks.workunit.client.0.smithi088.stderr:++ '[' '' ']' 2024-06-03T08:37:53.835 INFO:tasks.workunit.client.0.smithi088.stderr:++ . /etc/profile.d/colorzgrep.sh 2024-06-03T08:37:53.835 INFO:tasks.workunit.client.0.smithi088.stderr:+++ '[' -f /usr/libexec/grepconf.sh ']' 2024-06-03T08:37:53.836 INFO:tasks.workunit.client.0.smithi088.stderr:+++ /usr/libexec/grepconf.sh -c 2024-06-03T08:37:53.836 INFO:tasks.workunit.client.0.smithi088.stderr:+ sed 's/^/ list_parallel: /' 2024-06-03T08:37:53.836 INFO:tasks.workunit.client.0.smithi088.stderr:+++ alias 'zgrep=zgrep --color=auto' 2024-06-03T08:37:53.836 INFO:tasks.workunit.client.0.smithi088.stderr:+++ alias 'zfgrep=zfgrep --color=auto' 2024-06-03T08:37:53.836 INFO:tasks.workunit.client.0.smithi088.stderr:+++ alias 'zegrep=zegrep --color=auto' 2024-06-03T08:37:53.836 INFO:tasks.workunit.client.0.smithi088.stderr:++ for i in /etc/profile.d/*.sh 2024-06-03T08:37:53.836 INFO:tasks.workunit.client.0.smithi088.stderr:++ '[' -r /etc/profile.d/debuginfod.sh ']' 2024-06-03T08:37:53.836 INFO:tasks.workunit.client.0.smithi088.stderr:++ '[' '' ']' 2024-06-03T08:37:53.836 INFO:tasks.workunit.client.0.smithi088.stderr:++ . /etc/profile.d/debuginfod.sh 2024-06-03T08:37:53.836 INFO:tasks.workunit.client.0.smithi088.stderr:+++ '[' -z 'https://debuginfod.centos.org/ ' ']' 2024-06-03T08:37:53.836 INFO:tasks.workunit.client.0.smithi088.stderr:++ for i in /etc/profile.d/*.sh 2024-06-03T08:37:53.836 INFO:tasks.workunit.client.0.smithi088.stderr:++ '[' -r /etc/profile.d/gawk.sh ']' 2024-06-03T08:37:53.836 INFO:tasks.workunit.client.0.smithi088.stderr:++ '[' '' ']' 2024-06-03T08:37:53.836 INFO:tasks.workunit.client.0.smithi088.stderr:++ . /etc/profile.d/gawk.sh 2024-06-03T08:37:53.837 INFO:tasks.workunit.client.0.smithi088.stderr:++ for i in /etc/profile.d/*.sh 2024-06-03T08:37:53.837 INFO:tasks.workunit.client.0.smithi088.stderr:++ '[' -r /etc/profile.d/lang.sh ']' 2024-06-03T08:37:53.837 INFO:tasks.workunit.client.0.smithi088.stderr:++ '[' '' ']' 2024-06-03T08:37:53.837 INFO:tasks.workunit.client.0.smithi088.stderr:++ . /etc/profile.d/lang.sh 2024-06-03T08:37:53.837 INFO:tasks.workunit.client.0.smithi088.stderr:+++ unset LANG_backup 2024-06-03T08:37:53.837 INFO:tasks.workunit.client.0.smithi088.stderr:++++ /usr/bin/locale 2024-06-03T08:37:53.837 INFO:tasks.workunit.client.0.smithi088.stderr:+++ '[' -n '' ']' 2024-06-03T08:37:53.837 INFO:tasks.workunit.client.0.smithi088.stderr:+++ LANG_backup=en_US.UTF-8 2024-06-03T08:37:53.837 INFO:tasks.workunit.client.0.smithi088.stderr:+++ for config in /etc/locale.conf "${HOME}/.i18n" 2024-06-03T08:37:53.837 INFO:tasks.workunit.client.0.smithi088.stderr:+++ '[' -f /etc/locale.conf ']' 2024-06-03T08:37:53.837 INFO:tasks.workunit.client.0.smithi088.stderr:+++ '[' -x /usr/bin/sed ']' 2024-06-03T08:37:53.837 INFO:tasks.workunit.client.0.smithi088.stderr:++++ /usr/bin/sed -r -e 's/^[[:blank:]]*([[:upper:]_]+)=([[:print:][:digit:]\._-]+|"[[:print:][:digit:]\._-]+")/export \1=\2/;t;d' /etc/locale.conf 2024-06-03T08:37:53.837 INFO:tasks.workunit.client.0.smithi088.stderr:+++ eval export 'LANG="en_US.UTF-8"' 2024-06-03T08:37:53.838 INFO:tasks.workunit.client.0.smithi088.stderr:++++ export LANG=en_US.UTF-8 2024-06-03T08:37:53.838 INFO:tasks.workunit.client.0.smithi088.stderr:++++ LANG=en_US.UTF-8 2024-06-03T08:37:53.838 INFO:tasks.workunit.client.0.smithi088.stderr:+++ for config in /etc/locale.conf "${HOME}/.i18n" 2024-06-03T08:37:53.838 INFO:tasks.workunit.client.0.smithi088.stderr:+++ '[' -f /home/ubuntu/.i18n ']' 2024-06-03T08:37:53.838 INFO:tasks.workunit.client.0.smithi088.stderr:+++ '[' -n en_US.UTF-8 ']' 2024-06-03T08:37:53.838 INFO:tasks.workunit.client.0.smithi088.stderr:+++ LANG=en_US.UTF-8 2024-06-03T08:37:53.838 INFO:tasks.workunit.client.0.smithi088.stderr:+++ unset LANG_backup config 2024-06-03T08:37:53.838 INFO:tasks.workunit.client.0.smithi088.stderr:+++ '[' -n '' ']' 2024-06-03T08:37:53.838 INFO:tasks.workunit.client.0.smithi088.stderr:+++ '[' -n en_US.UTF-8 ']' 2024-06-03T08:37:53.838 INFO:tasks.workunit.client.0.smithi088.stderr:+++ '[' dumb = linux ']' 2024-06-03T08:37:53.838 INFO:tasks.workunit.client.0.smithi088.stderr:++ for i in /etc/profile.d/*.sh 2024-06-03T08:37:53.838 INFO:tasks.workunit.client.0.smithi088.stderr:++ '[' -r /etc/profile.d/less.sh ']' 2024-06-03T08:37:53.838 INFO:tasks.workunit.client.0.smithi088.stderr:++ '[' '' ']' 2024-06-03T08:37:53.839 INFO:tasks.workunit.client.0.smithi088.stderr:++ . /etc/profile.d/less.sh 2024-06-03T08:37:53.839 INFO:tasks.workunit.client.0.smithi088.stderr:+++ '[' -z '||/usr/bin/lesspipe.sh %s' ']' 2024-06-03T08:37:53.839 INFO:tasks.workunit.client.0.smithi088.stderr:++ for i in /etc/profile.d/*.sh 2024-06-03T08:37:53.839 INFO:tasks.workunit.client.0.smithi088.stderr:++ '[' -r /etc/profile.d/which2.sh ']' 2024-06-03T08:37:53.839 INFO:tasks.workunit.client.0.smithi088.stderr:++ '[' '' ']' 2024-06-03T08:37:53.839 INFO:tasks.workunit.client.0.smithi088.stderr:++ . /etc/profile.d/which2.sh 2024-06-03T08:37:53.839 INFO:tasks.workunit.client.0.smithi088.stderr:+++ case "$(basename $(readlink /proc/$$/exe))" in 2024-06-03T08:37:53.839 INFO:tasks.workunit.client.0.smithi088.stderr:+++++ readlink /proc/40936/exe 2024-06-03T08:37:53.839 INFO:tasks.workunit.client.0.smithi088.stderr:++++ basename /usr/bin/bash 2024-06-03T08:37:53.839 INFO:tasks.workunit.client.0.smithi088.stderr:+++ which_declare='declare -f' 2024-06-03T08:37:53.839 INFO:tasks.workunit.client.0.smithi088.stderr:+++ which_opt=-f 2024-06-03T08:37:53.839 INFO:tasks.workunit.client.0.smithi088.stderr:+++ export which_declare 2024-06-03T08:37:53.839 INFO:tasks.workunit.client.0.smithi088.stderr:+++ export -f which 2024-06-03T08:37:53.839 INFO:tasks.workunit.client.0.smithi088.stderr:++ unset i 2024-06-03T08:37:53.840 INFO:tasks.workunit.client.0.smithi088.stderr:++ unset -f pathmunge 2024-06-03T08:37:53.840 INFO:tasks.workunit.client.0.smithi088.stderr:+ [[ /home/ubuntu/.local/bin:/home/ubuntu/bin:/usr/local/bin:/usr/bin:/usr/local/sbin:/usr/sbin:/usr/sbin =~ /home/ubuntu/\.local/bin:/home/ubuntu/bin: ]] 2024-06-03T08:37:53.840 INFO:tasks.workunit.client.0.smithi088.stderr:+ export PATH 2024-06-03T08:37:53.840 INFO:tasks.workunit.client.0.smithi088.stderr:+ '[' -d /home/ubuntu/.bashrc.d ']' 2024-06-03T08:37:53.840 INFO:tasks.workunit.client.0.smithi088.stderr:+ unset rc 2024-06-03T08:37:53.840 INFO:tasks.workunit.client.0.smithi088.stderr:+ ceph_test_rados_open_pools_parallel 2024-06-03T08:37:53.840 INFO:tasks.workunit.client.0.smithi088.stderr:+ tee ceph_test_rados_open_pools_parallel.log 2024-06-03T08:37:53.840 INFO:tasks.workunit.client.0.smithi088.stderr:+ sed 's/^/ open_pools_parallel: /' 2024-06-03T08:37:53.840 INFO:tasks.workunit.client.0.smithi088.stdout: init 2024-06-03T08:37:53.841 INFO:tasks.workunit.client.0.smithi088.stdout: api_asio: 2024-06-03T08:37:53.591+0000 7f514d647780 10 monclient: init 2024-06-03T08:37:53.841 INFO:tasks.workunit.client.0.smithi088.stdout: api_asio: 2024-06-03T08:37:53.592+0000 7f514d647780 10 monclient: _reopen_session rank -1 2024-06-03T08:37:53.841 INFO:tasks.workunit.client.0.smithi088.stdout: api_asio: 2024-06-03T08:37:53.592+0000 7f514d647780 10 monclient: _add_conns ranks=[0,1,2] 2024-06-03T08:37:53.841 INFO:tasks.workunit.client.0.smithi088.stdout: api_asio: 2024-06-03T08:37:53.592+0000 7f514d647780 10 monclient(hunting): picked mon.noname-a con 0x56192203a160 addr v1:172.21.15.88:6789/0 2024-06-03T08:37:53.841 INFO:tasks.workunit.client.0.smithi088.stdout: api_asio: 2024-06-03T08:37:53.592+0000 7f514d647780 10 monclient(hunting): picked mon.noname-b con 0x56192203cd80 addr v1:172.21.15.192:6789/0 2024-06-03T08:37:53.841 INFO:tasks.workunit.client.0.smithi088.stdout: api_asio: 2024-06-03T08:37:53.592+0000 7f514d647780 10 monclient(hunting): picked mon.noname-c con 0x5619220ccfb0 addr v1:172.21.15.88:6790/0 2024-06-03T08:37:53.841 INFO:tasks.workunit.client.0.smithi088.stdout: api_asio: 2024-06-03T08:37:53.592+0000 7f514d647780 1 -- --> v1:172.21.15.88:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x5619220c68b0 con 0x56192203a160 2024-06-03T08:37:53.841 INFO:tasks.workunit.client.0.smithi088.stdout: api_asio: 2024-06-03T08:37:53.592+0000 7f514d647780 1 -- --> v1:172.21.15.192:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x5619220c56a0 con 0x56192203cd80 2024-06-03T08:37:53.841 INFO:tasks.workunit.client.0.smithi088.stdout: api_asio: 2024-06-03T08:37:53.592+0000 7f514d647780 1 -- --> v1:172.21.15.88:6790/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x5619220393d0 con 0x5619220ccfb0 2024-06-03T08:37:53.841 INFO:tasks.workunit.client.0.smithi088.stdout: api_asio: 2024-06-03T08:37:53.592+0000 7f514d647780 10 monclient(hunting): _renew_subs 2024-06-03T08:37:53.841 INFO:tasks.workunit.client.0.smithi088.stdout: api_asio: 2024-06-03T08:37:53.592+0000 7f514d647780 10 monclient(hunting): authenticate will time out at 1329.402317s 2024-06-03T08:37:53.841 INFO:tasks.workunit.client.0.smithi088.stdout: api_asio: 2024-06-03T08:37:53.601+0000 7f514b185640 1 --1- >> v1:172.21.15.88:6790/0 conn(0x5619220ccfb0 0x5619220dea00 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=1).handle_server_banner_and_identify peer v1:172.21.15.88:6790/0 says I am v1:172.21.15.88:59564/0 (socket says 172.21.15.88:59564) 2024-06-03T08:37:53.841 INFO:tasks.workunit.client.0.smithi088.stdout: api_asio: 2024-06-03T08:37:53.601+0000 7f514b185640 1 -- 172.21.15.88:0/320712639 learned_addr learned my addr 172.21.15.88:0/320712639 (peer_addr_for_me v1:172.21.15.88:0/0) 2024-06-03T08:37:53.842 INFO:tasks.workunit.client.0.smithi088.stdout: api_asio: 2024-06-03T08:37:53.610+0000 7f5133fff640 10 client.?.objecter ms_handle_connect 0x5619220ccfb0 2024-06-03T08:37:53.842 INFO:tasks.workunit.client.0.smithi088.stdout: api_asio: 2024-06-03T08:37:53.610+0000 7f5133fff640 10 client.?.objecter resend_mon_ops 2024-06-03T08:37:53.842 INFO:tasks.workunit.client.0.smithi088.stdout: api_asio: 2024-06-03T08:37:53.653+0000 7f5133fff640 10 client.?.objecter ms_handle_connect 0x56192203cd80 2024-06-03T08:37:53.842 INFO:tasks.workunit.client.0.smithi088.stdout: api_asio: 2024-06-03T08:37:53.653+0000 7f5133fff640 10 client.?.objecter resend_mon_ops 2024-06-03T08:37:53.842 INFO:tasks.workunit.client.0.smithi088.stdout: api_asio: 2024-06-03T08:37:53.654+0000 7f5133fff640 1 -- 172.21.15.88:0/320712639 <== mon.2 v1:172.21.15.88:6790/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 2226495019 0 0) 0x5619220393d0 con 0x5619220ccfb0 2024-06-03T08:37:53.842 INFO:tasks.workunit.client.0.smithi088.stdout: api_asio: 2024-06-03T08:37:53.654+0000 7f5133fff640 10 monclient(hunting): _negotiate 2024-06-03T08:37:53.842 INFO:tasks.workunit.client.0.smithi088.stdout: api_asio: 2024-06-03T08:37:53.654+0000 7f5133fff640 10 monclient(hunting): _init_auth method 2 2024-06-03T08:37:53.842 INFO:tasks.workunit.client.0.smithi088.stdout: api_asio: 2024-06-03T08:37:53.654+0000 7f5133fff640 10 monclient(hunting): _init_auth creating new auth 2024-06-03T08:37:53.842 INFO:tasks.workunit.client.0.smithi088.stdout: api_asio: 2024-06-03T08:37:53.654+0000 7f5133fff640 10 monclient(hunting): my global_id is 4490 2024-06-03T08:37:53.842 INFO:tasks.workunit.client.0.smithi088.stdout: api_asio: 2024-06-03T08:37:53.654+0000 7f5133fff640 1 -- 172.21.15.88:0/320712639 --> v1:172.21.15.88:6790/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f5138001d10 con 0x5619220ccfb0 2024-06-03T08:37:53.842 INFO:tasks.workunit.client.0.smithi088.stdout: api_asio: 2024-06-03T08:37:53.692+0000 7f5133fff640 1 -- 172.21.15.88:0/320712639 <== mon.1 v1:172.21.15.192:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 436991459 0 0) 0x5619220c56a0 con 0x56192203cd80 2024-06-03T08:37:53.842 INFO:tasks.workunit.client.0.smithi088.stdout: api_asio: 2024-06-03T08:37:53.692+0000 7f5133fff640 10 monclient(hunting): _negotiate 2024-06-03T08:37:53.843 INFO:tasks.workunit.client.0.smithi088.stdout: api_asio: 2024-06-03T08:37:53.692+0000 7f5133fff640 10 monclient(hunting): _init_auth method 2 2024-06-03T08:37:53.843 INFO:tasks.workunit.client.0.smithi088.stdout: api_asio: 2024-06-03T08:37:53.692+0000 7f5133fff640 10 monclient(hunting): _init_auth creating new auth 2024-06-03T08:37:53.843 INFO:tasks.workunit.client.0.smithi088.stdout: api_asio: 2024-06-03T08:37:53.692+0000 7f5133fff640 10 monclient(hunting): my global_id is 4498 2024-06-03T08:37:53.843 INFO:tasks.workunit.client.0.smithi088.stdout: api_asio: 2024-06-03T08:37:53.692+0000 7f5133fff640 1 -- 172.21.15.88:0/320712639 --> v1:172.21.15.192:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x5619220393d0 con 0x56192203cd80 2024-06-03T08:37:53.843 INFO:tasks.workunit.client.0.smithi088.stdout: api_asio: 2024-06-03T08:37:53.712+0000 7f5133fff640 10 client.?.objecter ms_handle_connect 0x56192203a160 2024-06-03T08:37:53.843 INFO:tasks.workunit.client.0.smithi088.stdout: api_asio: 2024-06-03T08:37:53.713+0000 7f5133fff640 10 client.?.objecter resend_mon_ops 2024-06-03T08:37:53.843 INFO:tasks.workunit.client.0.smithi088.stdout: api_asio: 2024-06-03T08:37:53.717+0000 7f5133fff640 1 -- 172.21.15.88:0/320712639 <== mon.2 v1:172.21.15.88:6790/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 764+0+0 (unknown 207387111 0 0) 0x7f5138001d10 con 0x5619220ccfb0 2024-06-03T08:37:53.843 INFO:tasks.workunit.client.0.smithi088.stdout: api_asio: 2024-06-03T08:37:53.717+0000 7f5133fff640 1 -- 172.21.15.88:0/320712639 --> v1:172.21.15.88:6790/0 -- auth(proto 2 165 bytes epoch 0) v1 -- 0x5619220c56a0 con 0x5619220ccfb0 2024-06-03T08:37:53.843 INFO:tasks.workunit.client.0.smithi088.stdout: api_asio: 2024-06-03T08:37:53.717+0000 7f5133fff640 1 -- 172.21.15.88:0/320712639 <== mon.2 v1:172.21.15.88:6790/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 2234374065 0 0) 0x7f5144002a90 con 0x5619220ccfb0 2024-06-03T08:37:53.843 INFO:tasks.workunit.client.0.smithi088.stdout: api_asio: 2024-06-03T08:37:53.717+0000 7f5133fff640 10 monclient(hunting): handle_monmap mon_map magic: 0 v1 2024-06-03T08:37:53.843 INFO:tasks.workunit.client.0.smithi088.stdout: api_asio: 2024-06-03T08:37:53.717+0000 7f5133fff640 10 monclient(hunting): got monmap 1 from mon.noname-c (according to old e1) 2024-06-03T08:37:53.843 INFO:tasks.workunit.client.0.smithi088.stdout: api_asio: 2024-06-03T08:37:53.717+0000 7f5133fff640 10 monclient(hunting): dump: 2024-06-03T08:37:53.843 INFO:tasks.workunit.client.0.smithi088.stdout: api_asio: epoch 1 2024-06-03T08:37:53.843 INFO:tasks.workunit.client.0.smithi088.stdout: api_asio: fsid ec3c3fdd-3ac6-4d7d-9412-683130ed8b04 2024-06-03T08:37:53.844 INFO:tasks.workunit.client.0.smithi088.stdout: api_asio: last_changed 2024-06-03T08:34:31.885125+0000 2024-06-03T08:37:53.844 INFO:tasks.workunit.client.0.smithi088.stdout: api_asio: created 2024-06-03T08:34:31.885125+0000 2024-06-03T08:37:53.844 INFO:tasks.workunit.client.0.smithi088.stdout: api_asio: min_mon_release 18 (reef) 2024-06-03T08:37:53.844 INFO:tasks.workunit.client.0.smithi088.stdout: api_asio: election_strategy: 3 2024-06-03T08:37:53.844 INFO:tasks.workunit.client.0.smithi088.stdout: api_asio: 0: v1:172.21.15.88:6789/0 mon.a 2024-06-03T08:37:53.844 INFO:tasks.workunit.client.0.smithi088.stdout: api_asio: 1: v1:172.21.15.192:6789/0 mon.b 2024-06-03T08:37:53.844 INFO:tasks.workunit.client.0.smithi088.stdout: api_asio: 2: v1:172.21.15.88:6790/0 mon.c 2024-06-03T08:37:53.844 INFO:tasks.workunit.client.0.smithi088.stdout: api_asio: 2024-06-03T08:37:53.844 INFO:tasks.workunit.client.0.smithi088.stdout: api_asio: 2024-06-03T08:37:53.717+0000 7f5133fff640 10 monclient(hunting): _finish_auth 0 2024-06-03T08:37:53.844 INFO:tasks.workunit.client.0.smithi088.stdout: api_asio: 2024-06-03T08:37:53.751+0000 7f5133fff640 1 -- 172.21.15.88:0/320712639 <== mon.1 v1:172.21.15.192:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 764+0+0 (unknown 3242590620 0 0) 0x5619220393d0 con 0x56192203cd80 2024-06-03T08:37:53.844 INFO:tasks.workunit.client.0.smithi088.stdout: api_asio: 2024-06-03T08:37:53.751+0000 7f5133fff640 1 -- 172.21.15.88:0/320712639 --> v1:172.21.15.192:6789/0 -- auth(proto 2 165 bytes epoch 0) v1 -- 0x7f5138001d10 con 0x56192203cd80 2024-06-03T08:37:53.844 INFO:tasks.workunit.client.0.smithi088.stdout: api_asio: 2024-06-03T08:37:53.751+0000 7f5133fff640 1 -- 172.21.15.88:0/320712639 <== mon.1 v1:172.21.15.192:6789/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 2234374065 0 0) 0x7f513c001450 con 0x56192203cd80 2024-06-03T08:37:53.844 INFO:tasks.workunit.client.0.smithi088.stdout: api_asio: 2024-06-03T08:37:53.751+0000 7f5133fff640 10 monclient(hunting): handle_monmap mon_map magic: 0 v1 2024-06-03T08:37:53.845 INFO:tasks.workunit.client.0.smithi088.stdout: api_asio: 2024-06-03T08:37:53.751+0000 7f5133fff640 10 monclient(hunting): got monmap 1 from mon.b (according to old e1) 2024-06-03T08:37:53.845 INFO:tasks.workunit.client.0.smithi088.stdout: api_asio: 2024-06-03T08:37:53.751+0000 7f5133fff640 10 monclient(hunting): dump: 2024-06-03T08:37:53.845 INFO:tasks.workunit.client.0.smithi088.stdout: api_asio: epoch 1 2024-06-03T08:37:53.845 INFO:tasks.workunit.client.0.smithi088.stdout: api_asio: fsid ec3c3fdd-3ac6-4d7d-9412-683130ed8b04 2024-06-03T08:37:53.845 INFO:tasks.workunit.client.0.smithi088.stdout: api_asio: last_changed 2024-06-03T08:34:31.885125+0000 2024-06-03T08:37:53.845 INFO:tasks.workunit.client.0.smithi088.stdout: api_asio: created 2024-06-03T08:34:31.885125+0000 2024-06-03T08:37:53.845 INFO:tasks.workunit.client.0.smithi088.stdout: api_asio: min_mon_release 18 (reef) 2024-06-03T08:37:53.845 INFO:tasks.workunit.client.0.smithi088.stdout: api_asio: election_strategy: 3 2024-06-03T08:37:53.845 INFO:tasks.workunit.client.0.smithi088.stdout: api_asio: 0: v1:172.21.15.88:6789/0 mon.a 2024-06-03T08:37:53.845 INFO:tasks.workunit.client.0.smithi088.stdout: api_asio: 1: v1:172.21.15.192:6789/0 mon.b 2024-06-03T08:37:53.845 INFO:tasks.workunit.client.0.smithi088.stdout: api_asio: 2: v1:172.21.15.88:6790/0 mon.c 2024-06-03T08:37:53.845 INFO:tasks.workunit.client.0.smithi088.stdout: api_asio: 2024-06-03T08:37:53.845 INFO:tasks.workunit.client.0.smithi088.stdout: api_asio: 2024-06-03T08:37:53.763+0000 7f5133fff640 1 -- 172.21.15.88:0/320712639 <== mon.0 v1:172.21.15.88:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 1882908852 0 0) 0x5619220c68b0 con 0x56192203a160 2024-06-03T08:37:53.854 INFO:tasks.workunit.client.0.smithi088.stdout: api_asio: 2024-06-03T08:37:53.763+0000 7f5133fff640 10 monclient(hunting): _negotiate 2024-06-03T08:37:53.854 INFO:tasks.workunit.client.0.smithi088.stdout: api_asio: 2024-06-03T08:37:53.763+0000 7f5133fff640 10 monclient(hunting): _init_auth method 2 2024-06-03T08:37:53.854 INFO:tasks.workunit.client.0.smithi088.stdout: api_asio: 2024-06-03T08:37:53.763+0000 7f5133fff640 10 monclient(hunting): _init_auth creating new auth 2024-06-03T08:37:53.854 INFO:tasks.workunit.client.0.smithi088.stdout: api_asio: 2024-06-03T08:37:53.763+0000 7f5133fff640 10 monclient(hunting): my global_id is 4473 2024-06-03T08:37:53.854 INFO:tasks.workunit.client.0.smithi088.stdout: api_asio: 2024-06-03T08:37:53.763+0000 7f5133fff640 1 -- 172.21.15.88:0/320712639 --> v1:172.21.15.88:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x5619220393d0 con 0x56192203a160 2024-06-03T08:37:53.854 INFO:tasks.workunit.client.0.smithi088.stdout: api_asio: 2024-06-03T08:37:53.766+0000 7f5133fff640 1 -- 172.21.15.88:0/320712639 <== mon.2 v1:172.21.15.88:6790/0 4 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 580+0+0 (unknown 3128416108 0 0) 0x5619220c56a0 con 0x5619220ccfb0 2024-06-03T08:37:53.854 INFO:tasks.workunit.client.0.smithi088.stdout: api_asio: 2024-06-03T08:37:53.766+0000 7f5133fff640 1 -- 172.21.15.88:0/320712639 >> v1:172.21.15.192:6789/0 conn(0x56192203cd80 legacy=0x5619220c4f90 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-06-03T08:37:53.854 INFO:tasks.workunit.client.0.smithi088.stdout: api_asio: 2024-06-03T08:37:53.766+0000 7f5133fff640 1 -- 172.21.15.88:0/320712639 >> v1:172.21.15.88:6789/0 conn(0x56192203a160 legacy=0x5619220b32a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-06-03T08:37:53.854 INFO:tasks.workunit.client.0.smithi088.stdout: api_asio: 2024-06-03T08:37:53.766+0000 7f5133fff640 10 monclient: _finish_hunting 0 2024-06-03T08:37:53.854 INFO:tasks.workunit.client.0.smithi088.stdout: api_asio: 2024-06-03T08:37:53.766+0000 7f5133fff640 1 monclient: found mon.c 2024-06-03T08:37:53.854 INFO:tasks.workunit.client.0.smithi088.stdout: api_asio: 2024-06-03T08:37:53.766+0000 7f5133fff640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-06-03T08:37:53.854 INFO:tasks.workunit.client.0.smithi088.stdout: api_asio: 2024-06-03T08:37:53.766+0000 7f5133fff640 10 monclient: _send_mon_message to mon.c at v1:172.21.15.88:6790/0 2024-06-03T08:37:53.855 INFO:tasks.workunit.client.0.smithi088.stdout: api_asio: 2024-06-03T08:37:53.766+0000 7f5133fff640 1 -- 172.21.15.88:0/320712639 --> v1:172.21.15.88:6790/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x5619220c6d40 con 0x5619220ccfb0 2024-06-03T08:37:53.855 INFO:tasks.workunit.client.0.smithi088.stdout: api_asio: 2024-06-03T08:37:53.766+0000 7f5133fff640 10 monclient: _finish_auth 0 2024-06-03T08:37:53.855 INFO:tasks.workunit.client.0.smithi088.stdout: api_asio: 2024-06-03T08:37:53.766+0000 7f5133fff640 10 monclient: _check_auth_tickets 2024-06-03T08:37:53.855 INFO:tasks.workunit.client.0.smithi088.stdout: api_asio: 2024-06-03T08:37:53.766+0000 7f5133fff640 20 monclient: _check_auth_rotating not needed by client.admin 2024-06-03T08:37:53.855 INFO:tasks.workunit.client.0.smithi088.stdout: api_asio: 2024-06-03T08:37:53.766+0000 7f514d647780 5 monclient: authenticate success, global_id 4490 2024-06-03T08:37:53.855 INFO:tasks.workunit.client.0.smithi088.stdout: api_asio: 2024-06-03T08:37:53.766+0000 7f514d647780 10 monclient: _renew_subs 2024-06-03T08:37:53.855 INFO:tasks.workunit.client.0.smithi088.stdout: api_asio: 2024-06-03T08:37:53.766+0000 7f514d647780 10 monclient: _send_mon_message to mon.c at v1:172.21.15.88:6790/0 2024-06-03T08:37:53.855 INFO:tasks.workunit.client.0.smithi088.stdout: api_asio: 2024-06-03T08:37:53.766+0000 7f514d647780 1 -- 172.21.15.88:0/320712639 --> v1:172.21.15.88:6790/0 -- mon_subscribe({mgrmap=0+}) v3 -- 0x5619220b39b0 con 0x5619220ccfb0 2024-06-03T08:37:53.855 INFO:tasks.workunit.client.0.smithi088.stdout: api_asio: 2024-06-03T08:37:53.766+0000 7f514d647780 10 client.4490.objecter _maybe_request_map subscribing (onetime) to next osd map 2024-06-03T08:37:53.855 INFO:tasks.workunit.client.0.smithi088.stdout: api_asio: 2024-06-03T08:37:53.766+0000 7f514d647780 10 monclient: _renew_subs 2024-06-03T08:37:53.855 INFO:tasks.workunit.client.0.smithi088.stdout: api_asio: 2024-06-03T08:37:53.766+0000 7f514d647780 10 monclient: _send_mon_message to mon.c at v1:172.21.15.88:6790/0 2024-06-03T08:37:53.855 INFO:tasks.workunit.client.0.smithi088.stdout: api_asio: 2024-06-03T08:37:53.766+0000 7f514d647780 1 -- 172.21.15.88:0/320712639 --> v1:172.21.15.88:6790/0 -- mon_subscribe({osdmap=0}) v3 -- 0x5619220c5f50 con 0x5619220ccfb0 2024-06-03T08:37:53.856 INFO:tasks.workunit.client.0.smithi088.stdout: api_asio: 2024-06-03T08:37:53.766+0000 7f514d647780 1 librados: init done 2024-06-03T08:37:53.856 INFO:tasks.workunit.client.0.smithi088.stdout: api_asio: 2024-06-03T08:37:53.766+0000 7f514d647780 10 librados: wait_for_osdmap waiting 2024-06-03T08:37:53.856 INFO:tasks.workunit.client.0.smithi088.stdout: api_asio: 2024-06-03T08:37:53.853+0000 7f5133fff640 1 -- 172.21.15.88:0/320712639 <== mon.2 v1:172.21.15.88:6790/0 5 ==== config(0 keys) v1 ==== 4+0+0 (unknown 0 0 0) 0x7f5144003040 con 0x5619220ccfb0 2024-06-03T08:37:53.856 INFO:tasks.workunit.client.0.smithi088.stdout: api_asio: 2024-06-03T08:37:53.853+0000 7f5133fff640 10 monclient: handle_config config(0 keys) v1 2024-06-03T08:37:56.104 INFO:tasks.workunit.client.0.smithi088.stdout: api_asio: 2024-06-03T08:37:53.853+0000 7f5133fff640 1 -- 172.21.15.88:0/320712639 <== mon.2 v1:172.21.15.88:6790/0 6 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 2234374065 0 0) 0x7f5144001450 con 0x5619220ccfb0 2024-06-03T08:37:56.105 INFO:tasks.workunit.client.0.smithi088.stdout: api_asio: 2024-06-03T08:37:53.853+0000 7f5133fff640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-06-03T08:37:56.105 INFO:tasks.workunit.client.0.smithi088.stdout: api_asio: 2024-06-03T08:37:53.853+0000 7f5133fff640 10 monclient: got monmap 1 from mon.c (according to old e1) 2024-06-03T08:37:56.105 INFO:tasks.workunit.client.0.smithi088.stdout: api_asio: 2024-06-03T08:37:53.853+0000 7f5133fff640 10 monclient: dump: 2024-06-03T08:37:56.105 INFO:tasks.workunit.client.0.smithi088.stdout: api_asio: epoch 1 2024-06-03T08:37:56.105 INFO:tasks.workunit.client.0.smithi088.stdout: api_asio: fsid ec3c3fdd-3ac6-4d7d-9412-683130ed8b04 2024-06-03T08:37:56.105 INFO:tasks.workunit.client.0.smithi088.stdout: api_asio: last_changed 2024-06-03T08:34:31.885125+0000 2024-06-03T08:37:56.105 INFO:tasks.workunit.client.0.smithi088.stdout: api_asio: created 2024-06-03T08:34:31.885125+0000 2024-06-03T08:37:56.105 INFO:tasks.workunit.client.0.smithi088.stdout: api_asio: min_mon_release 18 (reef) 2024-06-03T08:37:56.105 INFO:tasks.workunit.client.0.smithi088.stdout: api_asio: election_strategy: 3 2024-06-03T08:37:56.105 INFO:tasks.workunit.client.0.smithi088.stdout: api_asio: 0: v1:172.21.15.88:6789/0 mon.a 2024-06-03T08:37:56.105 INFO:tasks.workunit.client.0.smithi088.stdout: api_asio: 1: v1:172.21.15.192:6789/0 mon.b 2024-06-03T08:37:56.105 INFO:tasks.workunit.client.0.smithi088.stdout: api_asio: 2: v1:172.21.15.88:6790/0 mon.c 2024-06-03T08:37:56.106 INFO:tasks.workunit.client.0.smithi088.stdout: api_asio: 2024-06-03T08:37:56.106 INFO:tasks.workunit.client.0.smithi088.stdout: api_asio: 2024-06-03T08:37:53.853+0000 7f5149181640 4 set_mon_vals no callback set 2024-06-03T08:37:56.106 INFO:tasks.workunit.client.0.smithi088.stdout: api_asio: 2024-06-03T08:37:53.854+0000 7f5133fff640 1 -- 172.21.15.88:0/320712639 <== mon.2 v1:172.21.15.88:6790/0 7 ==== mgrmap(e 5) v1 ==== 89110+0+0 (unknown 873533973 0 0) 0x7f51440015b0 con 0x5619220ccfb0 2024-06-03T08:37:56.106 INFO:tasks.workunit.client.0.smithi088.stdout: api_asio: 2024-06-03T08:37:53.855+0000 7f514a984640 10 monclient: get_auth_request con 0x7f513806b750 auth_method 0 2024-06-03T08:37:56.106 INFO:tasks.workunit.client.0.smithi088.stdout: api_asio: 2024-06-03T08:37:53.855+0000 7f5133fff640 10 client.4490.objecter ms_handle_connect 0x7f513806b750 2024-06-03T08:37:56.106 INFO:tasks.workunit.client.0.smithi088.stdout: api_asio: 2024-06-03T08:37:53.857+0000 7f5133fff640 1 -- 172.21.15.88:0/320712639 <== mon.2 v1:172.21.15.88:6790/0 8 ==== osd_map(26..26 src has 1..26) v4 ==== 4090+0+0 (unknown 3637450554 0 0) 0x7f5144052be0 con 0x5619220ccfb0 2024-06-03T08:37:56.106 INFO:tasks.workunit.client.0.smithi088.stdout: api_asio: 2024-06-03T08:37:53.857+0000 7f5133fff640 10 client.4490.objecter ms_dispatch 0x561921fa3060 osd_map(26..26 src has 1..26) v4 2024-06-03T08:37:56.106 INFO:tasks.workunit.client.0.smithi088.stdout: api_asio: 2024-06-03T08:37:53.857+0000 7f5133fff640 3 client.4490.objecter handle_osd_map got epochs [26,26] > 0 2024-06-03T08:37:56.106 INFO:tasks.workunit.client.0.smithi088.stdout: api_asio: 2024-06-03T08:37:53.857+0000 7f5133fff640 3 client.4490.objecter handle_osd_map decoding full epoch 26 2024-06-03T08:37:56.106 INFO:tasks.workunit.client.0.smithi088.stdout: api_asio: 2024-06-03T08:37:53.857+0000 7f5133fff640 20 client.4490.objecter dump_active .. 0 homeless 2024-06-03T08:37:56.106 INFO:tasks.workunit.client.0.smithi088.stdout: api_asio: 2024-06-03T08:37:53.857+0000 7f514d647780 10 librados: wait_for_osdmap done waiting 2024-06-03T08:37:56.106 INFO:tasks.workunit.client.0.smithi088.stdout: api_asio: 2024-06-03T08:37:53.857+0000 7f514d647780 10 monclient: _send_mon_message to mon.c at v1:172.21.15.88:6790/0 2024-06-03T08:37:56.107 INFO:tasks.workunit.client.0.smithi088.stdout: api_asio: 2024-06-03T08:37:53.857+0000 7f514d647780 1 -- 172.21.15.88:0/320712639 --> v1:172.21.15.88:6790/0 -- mon_get_version(what=osdmap handle=1) v1 -- 0x561921fabf80 con 0x5619220ccfb0 2024-06-03T08:37:56.107 INFO:tasks.workunit.client.0.smithi088.stdout: api_asio: 2024-06-03T08:37:53.923+0000 7f5133fff640 1 -- 172.21.15.88:0/320712639 <== mon.2 v1:172.21.15.88:6790/0 9 ==== mon_get_version_reply(handle=1 version=26) v2 ==== 24+0+0 (unknown 2370098492 0 0) 0x7f514404e790 con 0x5619220ccfb0 2024-06-03T08:37:56.107 INFO:tasks.workunit.client.0.smithi088.stdout: api_asio: 2024-06-03T08:37:53.923+0000 7f5133fff640 10 monclient: handle_get_version_reply finishing 1 version 26 2024-06-03T08:37:56.107 INFO:tasks.workunit.client.0.smithi088.stdout: api_asio: 2024-06-03T08:37:53.923+0000 7f5148980640 10 client.4490.objecter _get_latest_version latest 26, have it 2024-06-03T08:37:56.107 INFO:tasks.workunit.client.0.smithi088.stdout: api_asio: 2024-06-03T08:37:53.923+0000 7f514d647780 10 client.4490.objecter create_pool name=ceph_test_rados_api_asio 2024-06-03T08:37:56.107 INFO:tasks.workunit.client.0.smithi088.stdout: api_asio: 2024-06-03T08:37:53.923+0000 7f514d647780 10 client.4490.objecter pool_op_submit 1 2024-06-03T08:37:56.107 INFO:tasks.workunit.client.0.smithi088.stdout: api_asio: 2024-06-03T08:37:53.923+0000 7f514d647780 10 monclient: _send_mon_message to mon.c at v1:172.21.15.88:6790/0 2024-06-03T08:37:56.107 INFO:tasks.workunit.client.0.smithi088.stdout: api_asio: 2024-06-03T08:37:53.923+0000 7f514d647780 1 -- 172.21.15.88:0/320712639 --> v1:172.21.15.88:6790/0 -- pool_op(create pool 0 tid 1 name ceph_test_rados_api_asio v0) v4 -- 0x561921fbcf00 con 0x5619220ccfb0 2024-06-03T08:37:56.127 INFO:tasks.workunit.client.0.smithi088.stdout: api_asio: 2024-06-03T08:37:56.104+0000 7f5133fff640 1 -- 172.21.15.88:0/320712639 <== mon.2 v1:172.21.15.88:6790/0 10 ==== pool_op_reply(tid 1 (0) Success v28) v1 ==== 43+0+0 (unknown 2880682622 0 0) 0x7f514404e5c0 con 0x5619220ccfb0 2024-06-03T08:37:56.127 INFO:tasks.workunit.client.0.smithi088.stdout: api_asio: 2024-06-03T08:37:56.104+0000 7f5133fff640 10 client.4490.objecter ms_dispatch 0x561921fa3060 pool_op_reply(tid 1 (0) Success v28) v1 2024-06-03T08:37:56.127 INFO:tasks.workunit.client.0.smithi088.stdout: api_asio: 2024-06-03T08:37:56.104+0000 7f5133fff640 10 client.4490.objecter handle_pool_op_reply pool_op_reply(tid 1 (0) Success v28) v1 2024-06-03T08:37:56.127 INFO:tasks.workunit.client.0.smithi088.stdout: api_asio: 2024-06-03T08:37:56.104+0000 7f5133fff640 10 client.4490.objecter have request 1 at 0x5619220b3ba0 Op: create 2024-06-03T08:37:56.127 INFO:tasks.workunit.client.0.smithi088.stdout: api_asio: 2024-06-03T08:37:56.104+0000 7f5133fff640 20 client.4490.objecter waiting for client to reach epoch 28 before calling back 2024-06-03T08:37:56.127 INFO:tasks.workunit.client.0.smithi088.stdout: api_asio: 2024-06-03T08:37:56.104+0000 7f5133fff640 10 client.4490.objecter _maybe_request_map subscribing (onetime) to next osd map 2024-06-03T08:37:56.127 INFO:tasks.workunit.client.0.smithi088.stdout: api_asio: 2024-06-03T08:37:56.104+0000 7f5133fff640 10 monclient: _renew_subs 2024-06-03T08:37:56.127 INFO:tasks.workunit.client.0.smithi088.stdout: api_asio: 2024-06-03T08:37:56.104+0000 7f5133fff640 10 monclient: _send_mon_message to mon.c at v1:172.21.15.88:6790/0 2024-06-03T08:37:56.128 INFO:tasks.workunit.client.0.smithi088.stdout: api_asio: 2024-06-03T08:37:56.104+0000 7f5133fff640 1 -- 172.21.15.88:0/320712639 --> v1:172.21.15.88:6790/0 -- mon_subscribe({osdmap=27}) v3 -- 0x5619220c56a0 con 0x5619220ccfb0 2024-06-03T08:37:56.128 INFO:tasks.workunit.client.0.smithi088.stdout: api_asio: 2024-06-03T08:37:56.104+0000 7f5133fff640 10 client.4490.objecter done 2024-06-03T08:37:56.128 INFO:tasks.workunit.client.0.smithi088.stdout: api_asio: 2024-06-03T08:37:56.126+0000 7f5133fff640 1 -- 172.21.15.88:0/320712639 <== mon.2 v1:172.21.15.88:6790/0 11 ==== osd_map(27..28 src has 1..28) v4 ==== 10792+0+0 (unknown 551191610 0 0) 0x7f5144056310 con 0x5619220ccfb0 2024-06-03T08:37:56.128 INFO:tasks.workunit.client.0.smithi088.stdout: api_asio: 2024-06-03T08:37:56.126+0000 7f5133fff640 10 client.4490.objecter ms_dispatch 0x561921fa3060 osd_map(27..28 src has 1..28) v4 2024-06-03T08:37:56.128 INFO:tasks.workunit.client.0.smithi088.stdout: api_asio: 2024-06-03T08:37:56.126+0000 7f5133fff640 3 client.4490.objecter handle_osd_map got epochs [27,28] > 26 2024-06-03T08:37:56.128 INFO:tasks.workunit.client.0.smithi088.stdout: api_asio: 2024-06-03T08:37:56.126+0000 7f5133fff640 3 client.4490.objecter handle_osd_map decoding incremental epoch 27 2024-06-03T08:37:56.128 INFO:tasks.workunit.client.0.smithi088.stdout: api_asio: 2024-06-03T08:37:56.126+0000 7f5133fff640 3 client.4490.objecter handle_osd_map decoding incremental epoch 28 2024-06-03T08:37:56.128 INFO:tasks.workunit.client.0.smithi088.stdout: api_asio: 2024-06-03T08:37:56.127+0000 7f5133fff640 20 client.4490.objecter dump_active .. 0 homeless 2024-06-03T08:37:56.128 INFO:tasks.workunit.client.0.smithi088.stdout: api_asio: 2024-06-03T08:37:56.127+0000 7f514d647780 10 librados: set snap read head -> 1 2024-06-03T08:37:56.128 INFO:tasks.workunit.client.0.smithi088.stdout: api_asio: 2024-06-03T08:37:56.127+0000 7f514d647780 10 librados: writefull oid=exist nspace= 2024-06-03T08:37:56.128 INFO:tasks.workunit.client.0.smithi088.stdout: api_asio: 2024-06-03T08:37:56.127+0000 7f514d647780 10 client.4490.objecter _op_submit op 0x5619220b1610 2024-06-03T08:37:56.128 INFO:tasks.workunit.client.0.smithi088.stdout: api_asio: 2024-06-03T08:37:56.127+0000 7f514d647780 20 client.4490.objecter _calc_target epoch 28 base exist @17 precalc_pgid 0 pgid 0.0 is_write 2024-06-03T08:37:56.128 INFO:tasks.workunit.client.0.smithi088.stdout: api_asio: 2024-06-03T08:37:56.127+0000 7f514d647780 20 client.4490.objecter _calc_target target exist @17 -> pgid 17.d3a59201 2024-06-03T08:37:56.128 INFO:tasks.workunit.client.0.smithi088.stdout: api_asio: 2024-06-03T08:37:56.127+0000 7f514d647780 10 client.4490.objecter _calc_target raw pgid 17.d3a59201 -> actual 17.1 acting [5,6] primary 5 2024-06-03T08:37:56.129 INFO:tasks.workunit.client.0.smithi088.stdout: api_asio: 2024-06-03T08:37:56.127+0000 7f514d647780 20 client.4490.objecter _get_session s=0x5619220df110 osd=5 3 2024-06-03T08:37:56.129 INFO:tasks.workunit.client.0.smithi088.stdout: api_asio: 2024-06-03T08:37:56.127+0000 7f514d647780 10 client.4490.objecter _op_submit oid exist '@17' '@17' [writefull 0~5 in=5b] tid 2 osd.5 2024-06-03T08:37:56.129 INFO:tasks.workunit.client.0.smithi088.stdout: api_asio: 2024-06-03T08:37:56.127+0000 7f514d647780 20 client.4490.objecter get_session s=0x5619220df110 osd=5 3 2024-06-03T08:37:56.129 INFO:tasks.workunit.client.0.smithi088.stdout: api_asio: 2024-06-03T08:37:56.127+0000 7f514d647780 15 client.4490.objecter _session_op_assign 5 2 2024-06-03T08:37:56.129 INFO:tasks.workunit.client.0.smithi088.stdout: api_asio: 2024-06-03T08:37:56.127+0000 7f514d647780 15 client.4490.objecter _send_op 2 to 17.1 on osd.5 2024-06-03T08:37:56.129 INFO:tasks.workunit.client.0.smithi088.stdout: api_asio: 2024-06-03T08:37:56.127+0000 7f514d647780 1 -- 172.21.15.88:0/320712639 --> v1:172.21.15.192:6805/34786 -- osd_op(unknown.0.0:2 17.1 17:8049a5cb:::exist:head [writefull 0~5 in=5b] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e28) v8 -- 0x561922179700 con 0x5619220df280 2024-06-03T08:38:05.320 INFO:tasks.ceph.mon.a.smithi088.stderr:==00:00:03:00.645 34743== Warning: unimplemented fcntl command: 1036 2024-06-03T08:38:05.327 INFO:tasks.ceph.mon.a.smithi088.stderr:==00:00:03:00.652 34743== Warning: unimplemented fcntl command: 1036 2024-06-03T08:38:05.414 INFO:tasks.ceph.mon.c.smithi088.stderr:==00:00:03:00.740 34749== Warning: unimplemented fcntl command: 1036 2024-06-03T08:38:05.527 INFO:tasks.ceph.mon.c.smithi088.stderr:==00:00:03:00.852 34749== Warning: unimplemented fcntl command: 1036 2024-06-03T08:38:05.628 INFO:tasks.ceph.mon.a.smithi088.stderr:==00:00:03:00.953 34743== Warning: unimplemented fcntl command: 1036 2024-06-03T08:38:05.631 INFO:tasks.ceph.mon.b.smithi192.stderr:==00:00:03:00.933 34228== Warning: unimplemented fcntl command: 1036 2024-06-03T08:38:05.649 INFO:tasks.ceph.mon.b.smithi192.stderr:==00:00:03:00.952 34228== Warning: unimplemented fcntl command: 1036 2024-06-03T08:38:05.705 INFO:tasks.workunit.client.0.smithi088.stdout: api_asio: 2024-06-03T08:37:56 api_service: [==========] Running 4 tests from 1 test suite. 2024-06-03T08:38:05.705 INFO:tasks.workunit.client.0.smithi088.stdout: api_service: [----------] Global test environment set-up. 2024-06-03T08:38:05.705 INFO:tasks.workunit.client.0.smithi088.stdout: api_service: [----------] 4 tests from LibRadosService 2024-06-03T08:38:05.705 INFO:tasks.workunit.client.0.smithi088.stdout: api_service: [ RUN ] LibRadosService.RegisterEarly 2024-06-03T08:38:05.705 INFO:tasks.workunit.client.0.smithi088.stdout: api_service: [ OK ] LibRadosService.RegisterEarly (5813 ms) 2024-06-03T08:38:05.705 INFO:tasks.workunit.client.0.smithi088.stdout: api_service: [ RUN ] LibRadosService.RegisterLate 2024-06-03T08:38:05.706 INFO:tasks.workunit.client.0.smithi088.stdout: api_service: [ OK ] LibRadosService.RegisterLate (314 ms) 2024-06-03T08:38:05.706 INFO:tasks.workunit.client.0.smithi088.stdout: api_service: [ RUN ] LibRadosService.StatusFormat 2024-06-03T08:38:05.706 INFO:tasks.workunit.client.0.smithi088.stdout: api_service: cluster: 2024-06-03T08:38:05.706 INFO:tasks.workunit.client.0.smithi088.stdout: api_service: id: ec3c3fdd-3ac6-4d7d-9412-683130ed8b04 2024-06-03T08:38:05.706 INFO:tasks.workunit.client.0.smithi088.stdout: api_service: health: HEALTH_WARN 2024-06-03T08:38:05.706 INFO:tasks.workunit.client.0.smithi088.stdout: api_service: 9 pool(s) do not have an application enabled 2024-06-03T08:38:05.706 INFO:tasks.workunit.client.0.smithi088.stdout: api_service: 2024-06-03T08:38:05.706 INFO:tasks.workunit.client.0.smithi088.stdout: api_service: services: 2024-06-03T08:38:05.706 INFO:tasks.workunit.client.0.smithi088.stdout: api_service: mon: 3 daemons, quorum a,b,c (age 2m) 2024-06-03T08:38:05.706 INFO:tasks.workunit.client.0.smithi088.stdout: api_service: mgr: x(active, since 2m), standbys: y 2024-06-03T08:38:05.706 INFO:tasks.workunit.client.0.smithi088.stdout: api_service: osd: 8 osds: 8 up (since 97s), 8 in (since 2m) 2024-06-03T08:38:05.706 INFO:tasks.workunit.client.0.smithi088.stdout: api_service: laundry: 2 daemons active (1 hosts) 2024-06-03T08:38:05.707 INFO:tasks.workunit.client.0.smithi088.stdout: api_service: 2024-06-03T08:38:05.707 INFO:tasks.workunit.client.0.smithi088.stdout: api_service: data: 2024-06-03T08:38:05.707 INFO:tasks.workunit.client.0.smithi088.stdout: api_service: pools: 28 pools, 817 pgs 2024-06-03T08:38:05.707 INFO:tasks.workunit.client.0.smithi088.stdout: api_service: objects: 2 objects, 449 KiB 2024-06-03T08:38:05.707 INFO:tasks.workunit.client.0.smithi088.stdout: api_service: usage: 214 MiB used, 720 GiB / 720 GiB avail 2024-06-03T08:38:05.707 INFO:tasks.workunit.client.0.smithi088.stdout: api_service: pgs: 98.898% pgs unknown 2024-06-03T08:38:05.707 INFO:tasks.workunit.client.0.smithi088.stdout: api_service: 808 unknown 2024-06-03T08:38:05.707 INFO:tasks.workunit.client.0.smithi088.stdout: api_service: 9 active+clean 2024-06-03T08:38:05.707 INFO:tasks.workunit.client.0.smithi088.stdout: api_service: 2024-06-03T08:38:05.707 INFO:tasks.workunit.client.0.smithi088.stdout: api_service: 2024-06-03T08:38:05.707 INFO:tasks.workunit.client.0.smithi088.stdout: api_service: cluster: 2024-06-03T08:38:05.707 INFO:tasks.workunit.client.0.smithi088.stdout: api_service: id: ec3c3fdd-3ac6-4d7d-9412-683130ed8b04 2024-06-03T08:38:05.707 INFO:tasks.workunit.client.0.smithi088.stdout: api_service: health: HEALTH_WARN 2024-06-03T08:38:05.707 INFO:tasks.workunit.client.0.smithi088.stdout: api_service: 8 pool(s) do not have an application enabled 2024-06-03T08:38:05.708 INFO:tasks.workunit.client.0.smithi088.stdout: api_service: 2024-06-03T08:38:05.708 INFO:tasks.workunit.client.0.smithi088.stdout: api_service: services: 2024-06-03T08:38:05.708 INFO:tasks.workunit.client.0.smithi088.stdout: api_service: mon: 3 daemons, quorum a,b,c (age 2m) 2024-06-03T08:38:05.708 INFO:tasks.workunit.client.0.smithi088.stdout: api_service: mgr: x(active, since 2m), standbys: y 2024-06-03T08:38:05.708 INFO:tasks.workunit.client.0.smithi088.stdout: api_service: osd: 8 osds: 8 up (since 99s), 8 in (since 2m) 2024-06-03T08:38:05.708 INFO:tasks.workunit.client.0.smithi088.stdout: api_service: foo: 11 portals active (1 hosts, 3 zones) 2024-06-03T08:38:05.708 INFO:tasks.workunit.client.0.smithi088.stdout: api_service: 2024-06-03T08:38:05.708 INFO:tasks.workunit.client.0.smithi088.stdout: api_service: data: 2024-06-03T08:38:05.708 INFO:tasks.workunit.client.0.smithi088.stdout: api_service: pools: 29 pools, 849 pgs 2024-06-03T08:38:05.708 INFO:tasks.workunit.client.0.smithi088.stdout: api_service: objects: 3 objects, 449 KiB 2024-06-03T08:38:05.708 INFO:tasks.workunit.client.0.smithi088.stdout: api_service: usage: 214 MiB used, 720 GiB / 720 GiB avail 2024-06-03T08:38:05.708 INFO:tasks.workunit.client.0.smithi088.stdout: api_service: pgs: 48.645% pgs unknown 2024-06-03T08:38:05.708 INFO:tasks.workunit.client.0.smithi088.stdout: api_service: 47.350% pgs not active 2024-06-03T08:38:05.708 INFO:tasks.workunit.client.0.smithi088.stdout: api_service: 413 unknown 2024-06-03T08:38:05.709 INFO:tasks.workunit.client.0.smithi088.stdout: api_service: 368 creating+peering 2024-06-03T08:38:05.709 INFO:tasks.workunit.client.0.smithi088.stdout: api_service: 34 active+clean 2024-06-03T08:38:05.709 INFO:tasks.workunit.client.0.smithi088.stdout: api_service: 34 creating+activating 2024-06-03T08:38:05.709 INFO:tasks.workunit.client.0.smithi088.stdout: api_service: 2024-06-03T08:38:05.709 INFO:tasks.workunit.client.0.smithi088.stdout: api_service: io: 2024-06-03T08:38:05.709 INFO:tasks.workunit.client.0.smithi088.stdout: api_service: client: 215 B/s wr, 0 op/s rd, 0 op/s wr 2024-06-03T08:38:05.709 INFO:tasks.workunit.client.0.smithi088.stdout: api_service: 2024-06-03T08:38:05.709 INFO:tasks.workunit.client.0.smithi088.stdout: api_service: 2024-06-03T08:38:05.709 INFO:tasks.workunit.client.0.smithi088.stdout: api_service: cluster: 2024-06-03T08:38:05.709 INFO:tasks.workunit.client.0.smithi088.stdout: api_service: id: ec3c3fdd-3ac6-4d7d-9412-683130ed8b04 2024-06-03T08:38:05.709 INFO:tasks.workunit.client.0.smithi088.stdout: api_service: health: HEALTH_WARN 2024-06-03T08:38:05.709 INFO:tasks.workunit.client.0.smithi088.stdout: api_service: 7 pool(s) do not have an application enabled 2024-06-03T08:38:05.709 INFO:tasks.workunit.client.0.smithi088.stdout: api_service: 2024-06-03T08:38:05.709 INFO:tasks.workunit.client.0.smithi088.stdout: api_service: services: 2024-06-03T08:38:05.710 INFO:tasks.workunit.client.0.smithi088.stdout: api_service: mon: 3 daemons, quorum a,b,c (age 2m) 2024-06-03T08:38:05.710 INFO:tasks.workunit.client.0.smithi088.stdout: api_service: mgr: x(active, since 2m), standbys: y 2024-06-03T08:38:05.710 INFO:tasks.workunit.client.0.smithi088.stdout: api_service: osd: 8 osds: 8 up (since 101s), 8 in (since 2m) 2024-06-03T08:38:05.710 INFO:tasks.workunit.client.0.smithi088.stdout: api_service: foo: 16 portals active (1 hosts, 3 zones) 2024-06-03T08:38:05.710 INFO:tasks.workunit.client.0.smithi088.stdout: api_service: laundry: 1 daemon active (1 hosts) 2024-06-03T08:38:05.710 INFO:tasks.workunit.client.0.smithi088.stdout: api_service: 2024-06-03T08:38:05.710 INFO:tasks.workunit.client.0.smithi088.stdout: api_service: data: 2024-06-03T08:38:05.710 INFO:tasks.workunit.client.0.smithi088.stdout: api_service: pools: 29 pools, 849 pgs 2024-06-03T08:38:05.710 INFO:tasks.workunit.client.0.smithi088.stdout: api_service: objects: 3 objects, 449 KiB 2024-06-03T08:38:05.710 INFO:tasks.workunit.client.0.smithi088.stdout: api_service: usage: 214 MiB used, 720 GiB / 720 GiB avail 2024-06-03T08:38:05.722 INFO:tasks.ceph.mon.c.smithi088.stderr:==00:00:03:01.048 34749== Warning: unimplemented fcntl command: 1036 2024-06-03T08:38:05.799 INFO:tasks.ceph.mon.b.smithi192.stderr:==00:00:03:01.101 34228== Warning: unimplemented fcntl command: 1036 2024-06-03T08:38:06.594 INFO:tasks.workunit.client.0.smithi088.stdout: api_service: pgs: 19.081% pgs un.127+0000 7f514d647780 20 client.4490.objecter put_session s=0x5619220df110 osd=5 4 2024-06-03T08:38:06.594 INFO:tasks.workunit.client.0.smithi088.stdout: api_asio: 2024-06-03T08:37:56.127+0000 7f514d647780 5 client.4490.objecter 1 in flight 2024-06-03T08:38:06.594 INFO:tasks.workunit.client.0.smithi088.stdout: api_asio: 2024-06-03T08:37:56.138+0000 7f514a183640 10 monclient: get_auth_request con 0x5619220df280 auth_method 0 2024-06-03T08:38:06.594 INFO:tasks.workunit.client.0.smithi088.stdout: api_asio: 2024-06-03T08:37:56.143+0000 7f5133fff640 10 client.4490.objecter ms_handle_connect 0x5619220df280 2024-06-03T08:38:06.594 INFO:tasks.workunit.client.0.smithi088.stdout: api_asio: 2024-06-03T08:37:56.592+0000 7f5132ffd640 10 monclient: tick 2024-06-03T08:38:06.594 INFO:tasks.workunit.client.0.smithi088.stdout: api_asio: 2024-06-03T08:37:56.592+0000 7f5132ffd640 10 monclient: _check_auth_tickets 2024-06-03T08:38:06.594 INFO:tasks.workunit.client.0.smithi088.stdout: api_asio: 2024-06-03T08:37:56.592+0000 7f5132ffd640 20 monclient: _check_auth_rotating not needed by client.admin 2024-06-03T08:38:06.594 INFO:tasks.workunit.client.0.smithi088.stdout: api_asio: 2024-06-03T08:37:56.592+0000 7f5132ffd640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-06-03T08:38:06.594 INFO:tasks.workunit.client.0.smithi088.stdout: api_asio: 2024-06-03T08:37:57.592+0000 7f5132ffd640 10 monclient: tick 2024-06-03T08:38:06.595 INFO:tasks.workunit.client.0.smithi088.stdout: api_asio: 2024-06-03T08:37:57.592+0000 7f5132ffd640 10 monclient: _check_auth_tickets 2024-06-03T08:38:06.595 INFO:tasks.workunit.client.0.smithi088.stdout: api_asio: 2024-06-03T08:37:57.592+0000 7f5132ffd640 20 monclient: _check_auth_rotating not needed by client.admin 2024-06-03T08:38:06.595 INFO:tasks.workunit.client.0.smithi088.stdout: api_asio: 2024-06-03T08:37:58.592+0000 7f5132ffd640 10 monclient: tick 2024-06-03T08:38:06.595 INFO:tasks.workunit.client.0.smithi088.stdout: api_asio: 2024-06-03T08:37:58.592+0000 7f5132ffd640 10 monclient: _check_auth_tickets 2024-06-03T08:38:06.595 INFO:tasks.workunit.client.0.smithi088.stdout: api_asio: 2024-06-03T08:37:58.592+0000 7f5132ffd640 20 monclient: _check_auth_rotating not needed by client.admin 2024-06-03T08:38:06.595 INFO:tasks.workunit.client.0.smithi088.stdout: api_asio: 2024-06-03T08:37:58.767+0000 7f514cbe8640 10 client.4490.objecter tick 2024-06-03T08:38:06.595 INFO:tasks.workunit.client.0.smithi088.stdout: api_asio: 2024-06-03T08:37:59.592+0000 7f5132ffd640 10 monclient: tick 2024-06-03T08:38:06.595 INFO:tasks.workunit.client.0.smithi088.stdout: api_asio: 2024-06-03T08:37:59.592+0000 7f5132ffd640 10 monclient: _check_auth_tickets 2024-06-03T08:38:06.595 INFO:tasks.workunit.client.0.smithi088.stdout: api_asio: 2024-06-03T08:37:59.592+0000 7f5132ffd640 20 monclient: _check_auth_rotating not needed by client.admin 2024-06-03T08:38:06.595 INFO:tasks.workunit.client.0.smithi088.stdout: api_asio: 2024-06-03T08:38:00.593+0000 7f5132ffd640 10 monclient: tick 2024-06-03T08:38:06.595 INFO:tasks.workunit.client.0.smithi088.stdout: api_asio: 2024-06-03T08:38:00.593+0000 7f5132ffd640 10 monclient: _check_auth_tickets 2024-06-03T08:38:06.595 INFO:tasks.workunit.client.0.smithi088.stdout: api_asio: 2024-06-03T08:38:00.593+0000 7f5132ffd640 20 monclient: _check_auth_rotating not needed by client.admin 2024-06-03T08:38:06.595 INFO:tasks.workunit.client.0.smithi088.stdout: api_asio: 2024-06-03T08:38:01.593+0000 7f5132ffd640 10 monclient: tick 2024-06-03T08:38:06.596 INFO:tasks.workunit.client.0.smithi088.stdout: api_asio: 2024-06-03T08:38:01.593+0000 7f5132ffd640 10 monclient: _check_auth_tickets 2024-06-03T08:38:06.596 INFO:tasks.workunit.client.0.smithi088.stdout: api_asio: 2024-06-03T08:38:01.593+0000 7f5132ffd640 20 monclient: _check_auth_rotating not needed by client.admin 2024-06-03T08:38:06.596 INFO:tasks.workunit.client.0.smithi088.stdout: api_asio: 2024-06-03T08:38:02.593+0000 7f5132ffd640 10 monclient: tick 2024-06-03T08:38:06.596 INFO:tasks.workunit.client.0.smithi088.stdout: api_asio: 2024-06-03T08:38:02.593+0000 7f5132ffd640 10 monclient: _check_auth_tickets 2024-06-03T08:38:06.596 INFO:tasks.workunit.client.0.smithi088.stdout: api_asio: 2024-06-03T08:38:02.593+0000 7f5132ffd640 20 monclient: _check_auth_rotating not needed by client.admin 2024-06-03T08:38:06.596 INFO:tasks.workunit.client.0.smithi088.stdout: api_asio: 2024-06-03T08:38:03.593+0000 7f5132ffd640 10 monclient: tick 2024-06-03T08:38:06.596 INFO:tasks.workunit.client.0.smithi088.stdout: api_asio: 2024-06-03T08:38:03.593+0000 7f5132ffd640 10 monclient: _check_auth_tickets 2024-06-03T08:38:06.596 INFO:tasks.workunit.client.0.smithi088.stdout: api_asio: 2024-06-03T08:38:03.593+0000 7f5132ffd640 20 monclient: _check_auth_rotating not needed by client.admin 2024-06-03T08:38:06.596 INFO:tasks.workunit.client.0.smithi088.stdout: api_asio: 2024-06-03T08:38:03.768+0000 7f514cbe8640 10 client.4490.objecter tick 2024-06-03T08:38:06.596 INFO:tasks.workunit.client.0.smithi088.stdout: api_asio: 2024-06-03T08:38:04.593+0000 7f5132ffd640 10 monclient: tick 2024-06-03T08:38:06.596 INFO:tasks.workunit.client.0.smithi088.stdout: api_asio: 2024-06-03T08:38:04.593+0000 7f5132ffd640 10 monclient: _check_auth_tickets 2024-06-03T08:38:06.596 INFO:tasks.workunit.client.0.smithi088.stdout: api_asio: 2024-06-03T08:38:04.593+0000 7f5132ffd640 20 monclient: _check_auth_rotating not needed by client.admin 2024-06-03T08:38:06.596 INFO:tasks.workunit.client.0.smithi088.stdout: api_asio: 2024-06-03T08:38:05.593+0000 7f5132ffd640 10 monclient: tick 2024-06-03T08:38:06.596 INFO:tasks.workunit.client.0.smithi088.stdout: api_asio: 2024-06-03T08:38:05.593+0000 7f5132ffd640 10 monclient: _check_auth_tickets 2024-06-03T08:38:06.597 INFO:tasks.workunit.client.0.smithi088.stdout: api_asio: 2024-06-03T08:38:05.593+0000 7f5132ffd640 20 monclient: _check_auth_rotating not needed by client.admin 2024-06-03T08:38:06.597 INFO:tasks.workunit.client.0.smithi088.stdout: api_asio: 2024-06-03T08:38:06.593+0000 7f5132ffd640 10 monclient: tick 2024-06-03T08:38:08.844 INFO:tasks.workunit.client.0.smithi088.stdout: api_asio: 2024-06-03T08:38:06.593+0000 7f5132ffd640 10 monclient: _check_auth_tickets 2024-06-03T08:38:08.844 INFO:tasks.workunit.client.0.smithi088.stdout: api_asio: 2024-06-03T08:38:06.593+0000 7f5132ffd640 20 monclient: _check_auth_rotating not needed by client.admin 2024-06-03T08:38:08.844 INFO:tasks.workunit.client.0.smithi088.stdout: api_asio: 2024-06-03T08:38:06.593+0000 7f5132ffd640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-06-03T08:38:08.845 INFO:tasks.workunit.client.0.smithi088.stdout: api_asio: 2024-06-03T08:38:07.593+0000 7f5132ffd640 10 monclient: tick 2024-06-03T08:38:08.845 INFO:tasks.workunit.client.0.smithi088.stdout: api_asio: 2024-06-03T08:38:07.593+0000 7f5132ffd640 10 monclient: _check_auth_tickets 2024-06-03T08:38:08.845 INFO:tasks.workunit.client.0.smithi088.stdout: api_asio: 2024-06-03T08:38:07.593+0000 7f5132ffd640 20 monclient: _check_auth_rotating not needed by client.admin 2024-06-03T08:38:08.845 INFO:tasks.workunit.client.0.smithi088.stdout: api_asio: 2024-06-03T08:38:08.593+0000 7f5132ffd640 10 monclient: tick 2024-06-03T08:38:08.845 INFO:tasks.workunit.client.0.smithi088.stdout: api_asio: 2024-06-03T08:38:08.593+0000 7f5132ffd640 10 monclient: _check_auth_tickets 2024-06-03T08:38:08.845 INFO:tasks.workunit.client.0.smithi088.stdout: api_asio: 2024-06-03T08:38:08.593+0000 7f5132ffd640 20 monclient: _check_auth_rotating not needed by client.admin 2024-06-03T08:38:08.845 INFO:tasks.workunit.client.0.smithi088.stdout: api_asio: 2024-06-03T08:38:08.769+0000 7f514cbe8640 10 client.4490.objecter tick 2024-06-03T08:38:08.845 INFO:tasks.workunit.client.0.smithi088.stdout: api_asio: 2024-06-03T08:38:08.769+0000 7f514cbe8640 2 client.4490.objecter tid 2 on osd.5 is laggy 2024-06-03T08:38:08.845 INFO:tasks.workunit.client.0.smithi088.stdout: api_asio: 2024-06-03T08:38:08.769+0000 7f514cbe8640 10 client.4490.objecter _maybe_request_map subscribing (onetime) to next osd map 2024-06-03T08:38:08.845 INFO:tasks.workunit.client.0.smithi088.stdout: api_asio: 2024-06-03T08:38:08.769+0000 7f514cbe8640 10 monclient: _renew_subs 2024-06-03T08:38:08.845 INFO:tasks.workunit.client.0.smithi088.stdout: api_asio: 2024-06-03T08:38:08.769+0000 7f514cbe8640 10 monclient: _send_mon_message to mon.c at v1:172.21.15.88:6790/0 2024-06-03T08:38:08.845 INFO:tasks.workunit.client.0.smithi088.stdout: api_asio: 2024-06-03T08:38:08.769+0000 7f514cbe8640 1 -- 172.21.15.88:0/320712639 --> v1:172.21.15.88:6790/0 -- mon_subscribe({osdmap=29}) v3 -- 0x7f511c002f90 con 0x5619220ccfb0 2024-06-03T08:38:08.845 INFO:tasks.workunit.client.0.smithi088.stdout: api_asio: 2024-06-03T08:38:08.769+0000 7f514cbe8640 1 -- 172.21.15.88:0/320712639 --> v1:172.21.15.192:6805/34786 -- ping magic: 0 v1 -- 0x7f511c003330 con 0x5619220df280 2024-06-03T08:38:08.845 INFO:tasks.workunit.client.0.smithi088.stdout: api_asio: 2024-06-03T08:38:08.843+0000 7f5133fff640 1 -- 172.21.15.88:0/320712639 <== mon.2 v1:172.21.15.88:6790/0 12 ==== osd_map(29..38 src has 1..38) v4 ==== 15378+0+0 (unknown 2325344971 0 0) 0x7f514405a960 con 0x5619220ccfb0 2024-06-03T08:38:08.846 INFO:tasks.workunit.client.0.smithi088.stdout: api_asio: 2024-06-03T08:38:08.843+0000 7f5133fff640 10 client.4490.objecter ms_dispatch 0x561921fa3060 osd_map(29..38 src has 1..38) v4 2024-06-03T08:38:08.846 INFO:tasks.workunit.client.0.smithi088.stdout: api_asio: 2024-06-03T08:38:08.843+0000 7f5133fff640 3 client.4490.objecter handle_osd_map got epochs [29,38] > 28 2024-06-03T08:38:08.846 INFO:tasks.workunit.client.0.smithi088.stdout: api_asio: 2024-06-03T08:38:08.843+0000 7f5133fff640 3 client.4490.objecter handle_osd_map decoding incremental epoch 29 2024-06-03T08:38:08.846 INFO:tasks.workunit.client.0.smithi088.stdout: api_asio: 2024-06-03T08:38:08.843+0000 7f5133fff640 10 client.4490.objecter checking op 2 2024-06-03T08:38:08.846 INFO:tasks.workunit.client.0.smithi088.stdout: api_asio: 2024-06-03T08:38:08.843+0000 7f5133fff640 20 client.4490.objecter _calc_target epoch 29 base exist @17 precalc_pgid 0 pgid 0.0 is_write 2024-06-03T08:38:08.846 INFO:tasks.workunit.client.0.smithi088.stdout: api_asio: 2024-06-03T08:38:08.843+0000 7f5133fff640 20 client.4490.objecter _calc_target target exist @17 -> pgid 17.d3a59201 2024-06-03T08:38:08.846 INFO:tasks.workunit.client.0.smithi088.stdout: api_asio: 2024-06-03T08:38:08.843+0000 7f5133fff640 3 client.4490.objecter handle_osd_map decoding incremental epoch 30 2024-06-03T08:38:08.846 INFO:tasks.workunit.client.0.smithi088.stdout: api_asio: 2024-06-03T08:38:08.843+0000 7f5133fff640 10 client.4490.objecter checking op 2 2024-06-03T08:38:08.846 INFO:tasks.workunit.client.0.smithi088.stdout: api_asio: 2024-06-03T08:38:08.843+0000 7f5133fff640 20 client.4490.objecter _calc_target epoch 30 base exist @17 precalc_pgid 0 pgid 0.0 is_write 2024-06-03T08:38:08.846 INFO:tasks.workunit.client.0.smithi088.stdout: api_asio: 2024-06-03T08:38:08.843+0000 7f5133fff640 20 client.4490.objecter _calc_target target exist @17 -> pgid 17.d3a59201 2024-06-03T08:38:08.846 INFO:tasks.workunit.client.0.smithi088.stdout: api_asio: 2024-06-03T08:38:08.843+0000 7f5133fff640 3 client.4490.objecter handle_osd_map decoding incremental epoch 31 2024-06-03T08:38:08.847 INFO:tasks.workunit.client.0.smithi088.stdout: api_asio: 2024-06-03T08:38:08.843+0000 7f5133fff640 10 client.4490.objecter checking op 2 2024-06-03T08:38:08.847 INFO:tasks.workunit.client.0.smithi088.stdout: api_asio: 2024-06-03T08:38:08.843+0000 7f5133fff640 20 client.4490.objecter _calc_target epoch 31 base exist @17 precalc_pgid 0 pgid 0.0 is_write 2024-06-03T08:38:08.847 INFO:tasks.workunit.client.0.smithi088.stdout: api_asio: 2024-06-03T08:38:08.843+0000 7f5133fff640 20 client.4490.objecter _calc_target target exist @17 -> pgid 17.d3a59201 2024-06-03T08:38:08.847 INFO:tasks.workunit.client.0.smithi088.stdout: api_asio: 2024-06-03T08:38:08.843+0000 7f5133fff640 3 client.4490.objecter handle_osd_map decoding incremental epoch 32 2024-06-03T08:38:08.847 INFO:tasks.workunit.client.0.smithi088.stdout: api_asio: 2024-06-03T08:38:08.843+0000 7f5133fff640 10 client.4490.objecter checking op 2 2024-06-03T08:38:08.847 INFO:tasks.workunit.client.0.smithi088.stdout: api_asio: 2024-06-03T08:38:08.843+0000 7f5133fff640 20 client.4490.objecter _calc_target epoch 32 base exist @17 precalc_pgid 0 pgid 0.0 is_write 2024-06-03T08:38:08.847 INFO:tasks.workunit.client.0.smithi088.stdout: api_asio: 2024-06-03T08:38:08.843+0000 7f5133fff640 20 client.4490.objecter _calc_target target exist @17 -> pgid 17.d3a59201 2024-06-03T08:38:08.847 INFO:tasks.workunit.client.0.smithi088.stdout: api_asio: 2024-06-03T08:38:08.843+0000 7f5133fff640 3 client.4490.objecter handle_osd_map decoding incremental epoch 33 2024-06-03T08:38:08.847 INFO:tasks.workunit.client.0.smithi088.stdout: api_asio: 2024-06-03T08:38:08.843+0000 7f5133fff640 10 client.4490.objecter checking op 2 2024-06-03T08:38:08.847 INFO:tasks.workunit.client.0.smithi088.stdout: api_asio: 2024-06-03T08:38:08.843+0000 7f5133fff640 20 client.4490.objecter _calc_target epoch 33 base exist @17 precalc_pgid 0 pgid 0.0 is_write 2024-06-03T08:38:08.847 INFO:tasks.workunit.client.0.smithi088.stdout: api_asio: 2024-06-03T08:38:08.843+0000 7f5133fff640 20 client.4490.objecter _calc_target target exist @17 -> pgid 17.d3a59201 2024-06-03T08:38:08.847 INFO:tasks.workunit.client.0.smithi088.stdout: api_asio: 2024-06-03T08:38:08.843+0000 7f5133fff640 3 client.4490.objecter handle_osd_map decoding incremental epoch 34 2024-06-03T08:38:08.847 INFO:tasks.workunit.client.0.smithi088.stdout: api_asio: 2024-06-03T08:38:08.843+0000 7f5133fff640 10 client.4490.objecter checking op 2 2024-06-03T08:38:08.848 INFO:tasks.workunit.client.0.smithi088.stdout: api_asio: 2024-06-03T08:38:08.843+0000 7f5133fff640 20 client.4490.objecter _calc_target epoch 34 base exist @17 precalc_pgid 0 pgid 0.0 is_write 2024-06-03T08:38:08.848 INFO:tasks.workunit.client.0.smithi088.stdout: api_asio: 2024-06-03T08:38:08.843+0000 7f5133fff640 20 client.4490.objecter _calc_target target exist @17 -> pgid 17.d3a59201 2024-06-03T08:38:08.848 INFO:tasks.workunit.client.0.smithi088.stdout: api_asio: 2024-06-03T08:38:08.843+0000 7f5133fff640 3 client.4490.objecter handle_osd_map decoding incremental epoch 35 2024-06-03T08:38:08.848 INFO:tasks.workunit.client.0.smithi088.stdout: api_asio: 2024-06-03T08:38:08.843+0000 7f5133fff640 10 client.4490.objecter checking op 2 2024-06-03T08:38:08.848 INFO:tasks.workunit.client.0.smithi088.stdout: api_asio: 2024-06-03T08:38:08.843+0000 7f5133fff640 20 client.4490.objecter _calc_target epoch 35 base exist @17 precalc_pgid 0 pgid 0.0 is_write 2024-06-03T08:38:08.848 INFO:tasks.workunit.client.0.smithi088.stdout: api_asio: 2024-06-03T08:38:08.843+0000 7f5133fff640 20 client.4490.objecter _calc_target target exist @17 -> pgid 17.d3a59201 2024-06-03T08:38:08.848 INFO:tasks.workunit.client.0.smithi088.stdout: api_asio: 2024-06-03T08:38:08.843+0000 7f5133fff640 3 client.4490.objecter handle_osd_map decoding incremental epoch 36 2024-06-03T08:38:08.848 INFO:tasks.workunit.client.0.smithi088.stdout: api_asio: 2024-06-03T08:38:08.843+0000 7f5133fff640 10 client.4490.objecter checking op 2 2024-06-03T08:38:08.848 INFO:tasks.workunit.client.0.smithi088.stdout: api_asio: 2024-06-03T08:38:08.843+0000 7f5133fff640 20 client.4490.objecter _calc_target epoch 36 base exist @17 precalc_pgid 0 pgid 0.0 is_write 2024-06-03T08:38:08.848 INFO:tasks.workunit.client.0.smithi088.stdout: api_asio: 2024-06-03T08:38:08.843+0000 7f5133fff640 20 client.4490.objecter _calc_target target exist @17 -> pgid 17.d3a59201 2024-06-03T08:38:08.848 INFO:tasks.workunit.client.0.smithi088.stdout: api_asio: 2024-06-03T08:38:08.843+0000 7f5133fff640 3 client.4490.objecter handle_osd_map decoding incremental epoch 37 2024-06-03T08:38:08.848 INFO:tasks.workunit.client.0.smithi088.stdout: api_asio: 2024-06-03T08:38:08.844+0000 7f5133fff640 10 client.4490.objecter checking op 2 2024-06-03T08:38:08.848 INFO:tasks.workunit.client.0.smithi088.stdout: api_asio: 2024-06-03T08:38:08.844+0000 7f5133fff640 20 client.4490.objecter _calc_target epoch 37 base exist @17 precalc_pgid 0 pgid 0.0 is_write 2024-06-03T08:38:08.848 INFO:tasks.workunit.client.0.smithi088.stdout: api_asio: 2024-06-03T08:38:08.844+0000 7f5133fff640 20 client.4490.objecter _calc_target target exist @17 -> pgid 17.d3a59201 2024-06-03T08:38:08.849 INFO:tasks.workunit.client.0.smithi088.stdout: api_asio: 2024-06-03T08:38:08.844+0000 7f5133fff640 3 client.4490.objecter handle_osd_map decoding incremental epoch 38 2024-06-03T08:38:08.849 INFO:tasks.workunit.client.0.smithi088.stdout: api_asio: 2024-06-03T08:38:08.844+0000 7f5133fff640 10 client.4490.objecter checking op 2 2024-06-03T08:38:08.849 INFO:tasks.workunit.client.0.smithi088.stdout: api_asio: 2024-06-03T08:38:08.844+0000 7f5133fff640 20 client.4490.objecter _calc_target epoch 38 base exist @17 precalc_pgid 0 pgid 0.0 is_write 2024-06-03T08:38:08.849 INFO:tasks.workunit.client.0.smithi088.stdout: api_asio: 2024-06-03T08:38:08.844+0000 7f5133fff640 20 client.4490.objecter _calc_target target exist @17 -> pgid 17.d3a59201 2024-06-03T08:38:08.849 INFO:tasks.workunit.client.0.smithi088.stdout: api_asio: 2024-06-03T08:38:08.844+0000 7f5133fff640 20 client.4490.objecter dump_active .. 0 homeless 2024-06-03T08:38:13.772 INFO:tasks.workunit.client.0.smithi088.stdout: api_asio: 2024-06-03T08:38:08.844+0000 7f5133fff640 20 client.4490.objecter 2 17.d3a59201 osd.5 exist [writefull 0~5 in=5b] 2024-06-03T08:38:13.773 INFO:tasks.workunit.client.0.smithi088.stdout: api_asio: 2024-06-03T08:38:09.594+0000 7f5132ffd640 10 monclient: tick 2024-06-03T08:38:13.773 INFO:tasks.workunit.client.0.smithi088.stdout: api_asio: 2024-06-03T08:38:09.594+0000 7f5132ffd640 10 monclient: _check_auth_tickets 2024-06-03T08:38:13.773 INFO:tasks.workunit.client.0.smithi088.stdout: api_asio: 2024-06-03T08:38:09.594+0000 7f5132ffd640 20 monclient: _check_auth_rotating not needed by client.admin 2024-06-03T08:38:13.773 INFO:tasks.workunit.client.0.smithi088.stdout: api_asio: 2024-06-03T08:38:10.593+0000 7f5132ffd640 10 monclient: tick 2024-06-03T08:38:13.773 INFO:tasks.workunit.client.0.smithi088.stdout: api_asio: 2024-06-03T08:38:10.594+0000 7f5132ffd640 10 monclient: _check_auth_tickets 2024-06-03T08:38:13.773 INFO:tasks.workunit.client.0.smithi088.stdout: api_asio: 2024-06-03T08:38:10.594+0000 7f5132ffd640 20 monclient: _check_auth_rotating not needed by client.admin 2024-06-03T08:38:13.773 INFO:tasks.workunit.client.0.smithi088.stdout: api_asio: 2024-06-03T08:38:11.594+0000 7f5132ffd640 10 monclient: tick 2024-06-03T08:38:13.773 INFO:tasks.workunit.client.0.smithi088.stdout: api_asio: 2024-06-03T08:38:11.594+0000 7f5132ffd640 10 monclient: _check_auth_tickets 2024-06-03T08:38:13.773 INFO:tasks.workunit.client.0.smithi088.stdout: api_asio: 2024-06-03T08:38:11.594+0000 7f5132ffd640 20 monclient: _check_auth_rotating not needed by client.admin 2024-06-03T08:38:13.773 INFO:tasks.workunit.client.0.smithi088.stdout: api_asio: 2024-06-03T08:38:12.594+0000 7f5132ffd640 10 monclient: tick 2024-06-03T08:38:13.773 INFO:tasks.workunit.client.0.smithi088.stdout: api_asio: 2024-06-03T08:38:12.594+0000 7f5132ffd640 10 monclient: _check_auth_tickets 2024-06-03T08:38:13.773 INFO:tasks.workunit.client.0.smithi088.stdout: api_asio: 2024-06-03T08:38:12.594+0000 7f5132ffd640 20 monclient: _check_auth_rotating not needed by client.admin 2024-06-03T08:38:13.774 INFO:tasks.workunit.client.0.smithi088.stdout: api_asio: 2024-06-03T08:38:13.594+0000 7f5132ffd640 10 monclient: tick 2024-06-03T08:38:13.774 INFO:tasks.workunit.client.0.smithi088.stdout: api_asio: 2024-06-03T08:38:13.594+0000 7f5132ffd640 10 monclient: _check_auth_tickets 2024-06-03T08:38:13.774 INFO:tasks.workunit.client.0.smithi088.stdout: api_asio: 2024-06-03T08:38:13.594+0000 7f5132ffd640 20 monclient: _check_auth_rotating not needed by client.admin 2024-06-03T08:38:13.774 INFO:tasks.workunit.client.0.smithi088.stdout: api_asio: 2024-06-03T08:38:13.769+0000 7f514cbe8640 10 client.4490.objecter tick 2024-06-03T08:38:13.774 INFO:tasks.workunit.client.0.smithi088.stdout: api_asio: 2024-06-03T08:38:13.769+0000 7f514cbe8640 2 client.4490.objecter tid 2 on osd.5 is laggy 2024-06-03T08:38:13.774 INFO:tasks.workunit.client.0.smithi088.stdout: api_asio: 2024-06-03T08:38:13.769+0000 7f514cbe8640 10 client.4490.objecter _maybe_request_map subscribing (onetime) to next osd map 2024-06-03T08:38:13.774 INFO:tasks.workunit.client.0.smithi088.stdout: api_asio: 2024-06-03T08:38:13.769+0000 7f514cbe8640 10 monclient: _renew_subs 2024-06-03T08:38:13.774 INFO:tasks.workunit.client.0.smithi088.stdout: api_asio: 2024-06-03T08:38:13.770+0000 7f514cbe8640 10 monclient: _send_mon_message to mon.c at v1:172.21.15.88:6790/0 2024-06-03T08:38:13.774 INFO:tasks.workunit.client.0.smithi088.stdout: api_asio: 2024-06-03T08:38:13.770+0000 7f514cbe8640 1 -- 172.21.15.88:0/320712639 --> v1:172.21.15.88:6790/0 -- mon_subscribe({osdmap=39}) v3 -- 0x7f511c003500 con 0x5619220ccfb0 2024-06-03T08:38:13.774 INFO:tasks.workunit.client.0.smithi088.stdout: api_asio: 2024-06-03T08:38:13.770+0000 7f514cbe8640 1 -- 172.21.15.88:0/320712639 --> v1:172.21.15.192:6805/34786 -- ping magic: 0 v1 -- 0x7f511c003750 con 0x5619220df280 2024-06-03T08:38:13.774 INFO:tasks.workunit.client.0.smithi088.stdout: api_asio: 2024-06-03T08:38:13.772+0000 7f5133fff640 1 -- 172.21.15.88:0/320712639 <== mon.2 v1:172.21.15.88:6790/0 13 ==== osd_map(39..42 src has 1..42) v4 ==== 2522+0+0 (unknown 1230483908 0 0) 0x7f5144081e60 con 0x5619220ccfb0 2024-06-03T08:38:13.774 INFO:tasks.workunit.client.0.smithi088.stdout: api_asio: 2024-06-03T08:38:13.772+0000 7f5133fff640 10 client.4490.objecter ms_dispatch 0x561921fa3060 osd_map(39..42 src has 1..42) v4 2024-06-03T08:38:13.774 INFO:tasks.workunit.client.0.smithi088.stdout: api_asio: 2024-06-03T08:38:13.772+0000 7f5133fff640 3 client.4490.objecter handle_osd_map got epochs [39,42] > 38 2024-06-03T08:38:13.775 INFO:tasks.workunit.client.0.smithi088.stdout: api_asio: 2024-06-03T08:38:13.772+0000 7f5133fff640 3 client.4490.objecter handle_osd_map decoding incremental epoch 39 2024-06-03T08:38:13.775 INFO:tasks.workunit.client.0.smithi088.stdout: api_asio: 2024-06-03T08:38:13.772+0000 7f5133fff640 10 client.4490.objecter checking op 2 2024-06-03T08:38:13.775 INFO:tasks.workunit.client.0.smithi088.stdout: api_asio: 2024-06-03T08:38:13.772+0000 7f5133fff640 20 client.4490.objecter _calc_target epoch 39 base exist @17 precalc_pgid 0 pgid 0.0 is_write 2024-06-03T08:38:13.775 INFO:tasks.workunit.client.0.smithi088.stdout: api_asio: 2024-06-03T08:38:13.772+0000 7f5133fff640 20 client.4490.objecter _calc_target target exist @17 -> pgid 17.d3a59201 2024-06-03T08:38:13.775 INFO:tasks.workunit.client.0.smithi088.stdout: api_asio: 2024-06-03T08:38:13.772+0000 7f5133fff640 3 client.4490.objecter handle_osd_map decoding incremental epoch 40 2024-06-03T08:38:13.775 INFO:tasks.workunit.client.0.smithi088.stdout: api_asio: 2024-06-03T08:38:13.772+0000 7f5133fff640 10 client.4490.objecter checking op 2 2024-06-03T08:38:17.409 INFO:tasks.workunit.client.0.smithi088.stdout: api_asio: 2024-06-03T08:38:13.772+0000 7f5133fff640 20 client.4490.objecter _calc_target epoch 40 base exist @17 precalc_pgid 0 pgid 0.0 is_write 2024-06-03T08:38:17.409 INFO:tasks.workunit.client.0.smithi088.stdout: api_asio: 2024-06-03T08:38:13.772+0000 7f5133fff640 20 client.4490.objecter _calc_target target exist @17 -> pgid 17.d3a59201 2024-06-03T08:38:17.409 INFO:tasks.workunit.client.0.smithi088.stdout: api_asio: 2024-06-03T08:38:13.772+0000 7f5133fff640 3 client.4490.objecter handle_osd_map decoding incremental epoch 41 2024-06-03T08:38:17.409 INFO:tasks.workunit.client.0.smithi088.stdout: api_asio: 2024-06-03T08:38:13.772+0000 7f5133fff640 10 client.4490.objecter checking op 2 2024-06-03T08:38:17.409 INFO:tasks.workunit.client.0.smithi088.stdout: api_asio: 2024-06-03T08:38:13.772+0000 7f5133fff640 20 client.4490.objecter _calc_target epoch 41 base exist @17 precalc_pgid 0 pgid 0.0 is_write 2024-06-03T08:38:17.409 INFO:tasks.workunit.client.0.smithi088.stdout: api_asio: 2024-06-03T08:38:13.772+0000 7f5133fff640 20 client.4490.objecter _calc_target target exist @17 -> pgid 17.d3a59201 2024-06-03T08:38:17.410 INFO:tasks.workunit.client.0.smithi088.stdout: api_asio: 2024-06-03T08:38:13.772+0000 7f5133fff640 3 client.4490.objecter handle_osd_map decoding incremental epoch 42 2024-06-03T08:38:17.410 INFO:tasks.workunit.client.0.smithi088.stdout: api_asio: 2024-06-03T08:38:13.772+0000 7f5133fff640 10 client.4490.objecter checking op 2 2024-06-03T08:38:17.410 INFO:tasks.workunit.client.0.smithi088.stdout: api_asio: 2024-06-03T08:38:13.772+0000 7f5133fff640 20 client.4490.objecter _calc_target epoch 42 base exist @17 precalc_pgid 0 pgid 0.0 is_write 2024-06-03T08:38:17.410 INFO:tasks.workunit.client.0.smithi088.stdout: api_asio: 2024-06-03T08:38:13.772+0000 7f5133fff640 20 client.4490.objecter _calc_target target exist @17 -> pgid 17.d3a59201 2024-06-03T08:38:17.410 INFO:tasks.workunit.client.0.smithi088.stdout: api_asio: 2024-06-03T08:38:13.772+0000 7f5133fff640 20 client.4490.objecter dump_active .. 0 homeless 2024-06-03T08:38:17.410 INFO:tasks.workunit.client.0.smithi088.stdout: api_asio: 2024-06-03T08:38:13.772+0000 7f5133fff640 20 client.4490.objecter 2 17.d3a59201 osd.5 exist [writefull 0~5 in=5b] 2024-06-03T08:38:17.410 INFO:tasks.workunit.client.0.smithi088.stdout: api_asio: 2024-06-03T08:38:14.594+0000 7f5132ffd640 10 monclient: tick 2024-06-03T08:38:17.410 INFO:tasks.workunit.client.0.smithi088.stdout: api_asio: 2024-06-03T08:38:14.594+0000 7f5132ffd640 10 monclient: _check_auth_tickets 2024-06-03T08:38:17.410 INFO:tasks.workunit.client.0.smithi088.stdout: api_asio: 2024-06-03T08:38:14.594+0000 7f5132ffd640 20 monclient: _check_auth_rotating not needed by client.admin 2024-06-03T08:38:17.410 INFO:tasks.workunit.client.0.smithi088.stdout: api_asio: 2024-06-03T08:38:15.594+0000 7f5132ffd640 10 monclient: tick 2024-06-03T08:38:17.410 INFO:tasks.workunit.client.0.smithi088.stdout: api_asio: 2024-06-03T08:38:15.594+0000 7f5132ffd640 10 monclient: _check_auth_tickets 2024-06-03T08:38:17.410 INFO:tasks.workunit.client.0.smithi088.stdout: api_asio: 2024-06-03T08:38:15.594+0000 7f5132ffd640 20 monclient: _check_auth_rotating not needed by client.admin 2024-06-03T08:38:17.410 INFO:tasks.workunit.client.0.smithi088.stdout: api_asio: 2024-06-03T08:38:16.594+0000 7f5132ffd640 10 monclient: tick 2024-06-03T08:38:17.411 INFO:tasks.workunit.client.0.smithi088.stdout: api_asio: 2024-06-03T08:38:16.594+0000 7f5132ffd640 10 monclient: _check_auth_tickets 2024-06-03T08:38:17.411 INFO:tasks.workunit.client.0.smithi088.stdout: api_asio: 2024-06-03T08:38:16.594+0000 7f5132ffd640 20 monclient: _check_auth_rotating not needed by client.admin 2024-06-03T08:38:17.411 INFO:tasks.workunit.client.0.smithi088.stdout: api_asio: 2024-06-03T08:38:16.594+0000 7f5132ffd640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-06-03T08:38:17.411 INFO:tasks.workunit.client.0.smithi088.stdout: api_asio: 2024-06-03T08:38:17.407+0000 7f5133fff640 1 -- 172.21.15.88:0/320712639 <== osd.5 v1:172.21.15.192:6805/34786 1 ==== osd_map(29..44 src has 1..44) v4 ==== 19869+0+0 (unknown 2762274391 0 0) 0x7f513c0040e0 con 0x5619220df280 2024-06-03T08:38:17.411 INFO:tasks.workunit.client.0.smithi088.stdout: api_asio: 2024-06-03T08:38:17.407+0000 7f5133fff640 10 client.4490.objecter ms_dispatch 0x561921fa3060 osd_map(29..44 src has 1..44) v4 2024-06-03T08:38:17.411 INFO:tasks.workunit.client.0.smithi088.stdout: api_asio: 2024-06-03T08:38:17.407+0000 7f5133fff640 3 client.4490.objecter handle_osd_map got epochs [29,44] > 42 2024-06-03T08:38:17.411 INFO:tasks.workunit.client.0.smithi088.stdout: api_asio: 2024-06-03T08:38:17.407+0000 7f5133fff640 3 client.4490.objecter handle_osd_map decoding incremental epoch 43 2024-06-03T08:38:17.411 INFO:tasks.workunit.client.0.smithi088.stdout: api_asio: 2024-06-03T08:38:17.407+0000 7f5133fff640 10 client.4490.objecter checking op 2 2024-06-03T08:38:17.411 INFO:tasks.workunit.client.0.smithi088.stdout: api_asio: 2024-06-03T08:38:17.407+0000 7f5133fff640 20 client.4490.objecter _calc_target epoch 43 base exist @17 precalc_pgid 0 pgid 0.0 is_write 2024-06-03T08:38:17.411 INFO:tasks.workunit.client.0.smithi088.stdout: api_asio: 2024-06-03T08:38:17.407+0000 7f5133fff640 20 client.4490.objecter _calc_target target exist @17 -> pgid 17.d3a59201 2024-06-03T08:38:17.411 INFO:tasks.workunit.client.0.smithi088.stdout: api_asio: 2024-06-03T08:38:17.407+0000 7f5133fff640 3 client.4490.objecter handle_osd_map decoding incremental epoch 44 2024-06-03T08:38:17.411 INFO:tasks.workunit.client.0.smithi088.stdout: api_asio: 2024-06-03T08:38:17.408+0000 7f5133fff640 10 client.4490.objecter checking op 2 2024-06-03T08:38:19.595 INFO:tasks.workunit.client.0.smithi088.stdout: api_asio: 2024-06-03T08:38:17.408+0000 7f5133fff640 20 client.4490.objecter _calc_target epoch 44 base exist @17 precalc_pgid 0 pgid 0.0 is_write 2024-06-03T08:38:19.595 INFO:tasks.workunit.client.0.smithi088.stdout: api_asio: 2024-06-03T08:38:17.408+0000 7f5133fff640 20 client.4490.objecter _calc_target target exist @17 -> pgid 17.d3a59201 2024-06-03T08:38:19.595 INFO:tasks.workunit.client.0.smithi088.stdout: api_asio: 2024-06-03T08:38:17.408+0000 7f5133fff640 20 client.4490.objecter dump_active .. 0 homeless 2024-06-03T08:38:19.595 INFO:tasks.workunit.client.0.smithi088.stdout: api_asio: 2024-06-03T08:38:17.408+0000 7f5133fff640 20 client.4490.objecter 2 17.d3a59201 osd.5 exist [writefull 0~5 in=5b] 2024-06-03T08:38:19.595 INFO:tasks.workunit.client.0.smithi088.stdout: api_asio: 2024-06-03T08:38:17.594+0000 7f5132ffd640 10 monclient: tick 2024-06-03T08:38:19.595 INFO:tasks.workunit.client.0.smithi088.stdout: api_asio: 2024-06-03T08:38:17.594+0000 7f5132ffd640 10 monclient: _check_auth_tickets 2024-06-03T08:38:19.595 INFO:tasks.workunit.client.0.smithi088.stdout: api_asio: 2024-06-03T08:38:17.594+0000 7f5132ffd640 20 monclient: _check_auth_rotating not needed by client.admin 2024-06-03T08:38:19.596 INFO:tasks.workunit.client.0.smithi088.stdout: api_asio: 2024-06-03T08:38:18.594+0000 7f5132ffd640 10 monclient: tick 2024-06-03T08:38:19.596 INFO:tasks.workunit.client.0.smithi088.stdout: api_asio: 2024-06-03T08:38:18.594+0000 7f5132ffd640 10 monclient: _check_auth_tickets 2024-06-03T08:38:19.596 INFO:tasks.workunit.client.0.smithi088.stdout: api_asio: 2024-06-03T08:38:18.594+0000 7f5132ffd640 20 monclient: _check_auth_rotating not needed by client.admin 2024-06-03T08:38:19.596 INFO:tasks.workunit.client.0.smithi088.stdout: api_asio: 2024-06-03T08:38:18.771+0000 7f514cbe8640 10 client.4490.objecter tick 2024-06-03T08:38:19.596 INFO:tasks.workunit.client.0.smithi088.stdout: api_asio: 2024-06-03T08:38:18.771+0000 7f514cbe8640 2 client.4490.objecter tid 2 on osd.5 is laggy 2024-06-03T08:38:19.596 INFO:tasks.workunit.client.0.smithi088.stdout: api_asio: 2024-06-03T08:38:18.771+0000 7f514cbe8640 10 client.4490.objecter _maybe_request_map subscribing (onetime) to next osd map 2024-06-03T08:38:19.596 INFO:tasks.workunit.client.0.smithi088.stdout: api_asio: 2024-06-03T08:38:18.771+0000 7f514cbe8640 10 monclient: _renew_subs 2024-06-03T08:38:19.596 INFO:tasks.workunit.client.0.smithi088.stdout: api_asio: 2024-06-03T08:38:18.771+0000 7f514cbe8640 10 monclient: _send_mon_message to mon.c at v1:172.21.15.88:6790/0 2024-06-03T08:38:19.596 INFO:tasks.workunit.client.0.smithi088.stdout: api_asio: 2024-06-03T08:38:18.771+0000 7f514cbe8640 1 -- 172.21.15.88:0/320712639 --> v1:172.21.15.88:6790/0 -- mon_subscribe({osdmap=45}) v3 -- 0x7f511c003500 con 0x5619220ccfb0 2024-06-03T08:38:19.596 INFO:tasks.workunit.client.0.smithi088.stdout: api_asio: 2024-06-03T08:38:18.771+0000 7f514cbe8640 1 -- 172.21.15.88:0/320712639 --> v1:172.21.15.192:6805/34786 -- ping magic: 0 v1 -- 0x7f511c0039e0 con 0x5619220df280 2024-06-03T08:38:19.596 INFO:tasks.workunit.client.0.smithi088.stdout: api_asio: 2024-06-03T08:38:18.772+0000 7f5133fff640 1 -- 172.21.15.88:0/320712639 <== mon.2 v1:172.21.15.88:6790/0 14 ==== osd_map(45..46 src has 1..46) v4 ==== 1671+0+0 (unknown 4263727864 0 0) 0x7f5144051b30 con 0x5619220ccfb0 2024-06-03T08:38:19.597 INFO:tasks.workunit.client.0.smithi088.stdout: api_asio: 2024-06-03T08:38:18.772+0000 7f5133fff640 10 client.4490.objecter ms_dispatch 0x561921fa3060 osd_map(45..46 src has 1..46) v4 2024-06-03T08:38:19.597 INFO:tasks.workunit.client.0.smithi088.stdout: api_asio: 2024-06-03T08:38:18.772+0000 7f5133fff640 3 client.4490.objecter handle_osd_map got epochs [45,46] > 44 2024-06-03T08:38:19.597 INFO:tasks.workunit.client.0.smithi088.stdout: api_asio: 2024-06-03T08:38:18.772+0000 7f5133fff640 3 client.4490.objecter handle_osd_map decoding incremental epoch 45 2024-06-03T08:38:19.597 INFO:tasks.workunit.client.0.smithi088.stdout: api_asio: 2024-06-03T08:38:18.772+0000 7f5133fff640 10 client.4490.objecter checking op 2 2024-06-03T08:38:19.597 INFO:tasks.workunit.client.0.smithi088.stdout: api_asio: 2024-06-03T08:38:18.772+0000 7f5133fff640 20 client.4490.objecter _calc_target epoch 45 base exist @17 precalc_pgid 0 pgid 0.0 is_write 2024-06-03T08:38:19.597 INFO:tasks.workunit.client.0.smithi088.stdout: api_asio: 2024-06-03T08:38:18.772+0000 7f5133fff640 20 client.4490.objecter _calc_target target exist @17 -> pgid 17.d3a59201 2024-06-03T08:38:19.597 INFO:tasks.workunit.client.0.smithi088.stdout: api_asio: 2024-06-03T08:38:18.772+0000 7f5133fff640 3 client.4490.objecter handle_osd_map decoding incremental epoch 46 2024-06-03T08:38:19.597 INFO:tasks.workunit.client.0.smithi088.stdout: api_asio: 2024-06-03T08:38:18.773+0000 7f5133fff640 10 client.4490.objecter checking op 2 2024-06-03T08:38:19.597 INFO:tasks.workunit.client.0.smithi088.stdout: api_asio: 2024-06-03T08:38:18.773+0000 7f5133fff640 20 client.4490.objecter _calc_target epoch 46 base exist @17 precalc_pgid 0 pgid 0.0 is_write 2024-06-03T08:38:19.597 INFO:tasks.workunit.client.0.smithi088.stdout: api_asio: 2024-06-03T08:38:18.773+0000 7f5133fff640 20 client.4490.objecter _calc_target target exist @17 -> pgid 17.d3a59201 2024-06-03T08:38:19.597 INFO:tasks.workunit.client.0.smithi088.stdout: api_asio: 2024-06-03T08:38:18.773+0000 7f5133fff640 20 client.4490.objecter dump_active .. 0 homeless 2024-06-03T08:38:19.598 INFO:tasks.workunit.client.0.smithi088.stdout: api_asio: 2024-06-03T08:38:18.773+0000 7f5133fff640 20 client.4490.objecter 2 17.d3a59201 osd.5 exist [writefull 0~5 in=5b] 2024-06-03T08:38:23.774 INFO:tasks.workunit.client.0.smithi088.stdout: api_asio: 2024-06-03T08:38:19.594+0000 7f5132ffd640 10 monclient: tick 2024-06-03T08:38:23.774 INFO:tasks.workunit.client.0.smithi088.stdout: api_asio: 2024-06-03T08:38:19.594+0000 7f5132ffd640 10 monclient: _check_auth_tickets 2024-06-03T08:38:23.774 INFO:tasks.workunit.client.0.smithi088.stdout: api_asio: 2024-06-03T08:38:19.594+0000 7f5132ffd640 20 monclient: _check_auth_rotating not needed by client.admin 2024-06-03T08:38:23.774 INFO:tasks.workunit.client.0.smithi088.stdout: api_asio: 2024-06-03T08:38:20.594+0000 7f5132ffd640 10 monclient: tick 2024-06-03T08:38:23.775 INFO:tasks.workunit.client.0.smithi088.stdout: api_asio: 2024-06-03T08:38:20.594+0000 7f5132ffd640 10 monclient: _check_auth_tickets 2024-06-03T08:38:23.775 INFO:tasks.workunit.client.0.smithi088.stdout: api_asio: 2024-06-03T08:38:20.594+0000 7f5132ffd640 20 monclient: _check_auth_rotating not needed by client.admin 2024-06-03T08:38:23.775 INFO:tasks.workunit.client.0.smithi088.stdout: api_asio: 2024-06-03T08:38:21.594+0000 7f5132ffd640 10 monclient: tick 2024-06-03T08:38:23.775 INFO:tasks.workunit.client.0.smithi088.stdout: api_asio: 2024-06-03T08:38:21.594+0000 7f5132ffd640 10 monclient: _check_auth_tickets 2024-06-03T08:38:23.775 INFO:tasks.workunit.client.0.smithi088.stdout: api_asio: 2024-06-03T08:38:21.594+0000 7f5132ffd640 20 monclient: _check_auth_rotating not needed by client.admin 2024-06-03T08:38:23.775 INFO:tasks.workunit.client.0.smithi088.stdout: api_asio: 2024-06-03T08:38:22.594+0000 7f5132ffd640 10 monclient: tick 2024-06-03T08:38:23.775 INFO:tasks.workunit.client.0.smithi088.stdout: api_asio: 2024-06-03T08:38:22.594+0000 7f5132ffd640 10 monclient: _check_auth_tickets 2024-06-03T08:38:23.775 INFO:tasks.workunit.client.0.smithi088.stdout: api_asio: 2024-06-03T08:38:22.594+0000 7f5132ffd640 20 monclient: _check_auth_rotating not needed by client.admin 2024-06-03T08:38:23.775 INFO:tasks.workunit.client.0.smithi088.stdout: api_asio: 2024-06-03T08:38:23.595+0000 7f5132ffd640 10 monclient: tick 2024-06-03T08:38:23.775 INFO:tasks.workunit.client.0.smithi088.stdout: api_asio: 2024-06-03T08:38:23.595+0000 7f5132ffd640 10 monclient: _check_auth_tickets 2024-06-03T08:38:23.775 INFO:tasks.workunit.client.0.smithi088.stdout: api_asio: 2024-06-03T08:38:23.595+0000 7f5132ffd640 20 monclient: _check_auth_rotating not needed by client.admin 2024-06-03T08:38:23.775 INFO:tasks.workunit.client.0.smithi088.stdout: api_asio: 2024-06-03T08:38:23.772+0000 7f514cbe8640 10 client.4490.objecter tick 2024-06-03T08:38:23.775 INFO:tasks.workunit.client.0.smithi088.stdout: api_asio: 2024-06-03T08:38:23.772+0000 7f514cbe8640 2 client.4490.objecter tid 2 on osd.5 is laggy 2024-06-03T08:38:23.776 INFO:tasks.workunit.client.0.smithi088.stdout: api_asio: 2024-06-03T08:38:23.772+0000 7f514cbe8640 10 client.4490.objecter _maybe_request_map subscribing (onetime) to next osd map 2024-06-03T08:38:23.776 INFO:tasks.workunit.client.0.smithi088.stdout: api_asio: 2024-06-03T08:38:23.772+0000 7f514cbe8640 10 monclient: _renew_subs 2024-06-03T08:38:23.776 INFO:tasks.workunit.client.0.smithi088.stdout: api_asio: 2024-06-03T08:38:23.772+0000 7f514cbe8640 10 monclient: _send_mon_message to mon.c at v1:172.21.15.88:6790/0 2024-06-03T08:38:23.776 INFO:tasks.workunit.client.0.smithi088.stdout: api_asio: 2024-06-03T08:38:23.772+0000 7f514cbe8640 1 -- 172.21.15.88:0/320712639 --> v1:172.21.15.88:6790/0 -- mon_subscribe({osdmap=47}) v3 -- 0x7f511c003500 con 0x5619220ccfb0 2024-06-03T08:38:23.776 INFO:tasks.workunit.client.0.smithi088.stdout: api_asio: 2024-06-03T08:38:23.772+0000 7f514cbe8640 1 -- 172.21.15.88:0/320712639 --> v1:172.21.15.192:6805/34786 -- ping magic: 0 v1 -- 0x7f511c003be0 con 0x5619220df280 2024-06-03T08:38:23.776 INFO:tasks.workunit.client.0.smithi088.stdout: api_asio: 2024-06-03T08:38:23.773+0000 7f5133fff640 1 -- 172.21.15.88:0/320712639 <== mon.2 v1:172.21.15.88:6790/0 15 ==== osd_map(47..51 src has 1..51) v4 ==== 3899+0+0 (unknown 1231779104 0 0) 0x7f5144052740 con 0x5619220ccfb0 2024-06-03T08:38:23.776 INFO:tasks.workunit.client.0.smithi088.stdout: api_asio: 2024-06-03T08:38:23.773+0000 7f5133fff640 10 client.4490.objecter ms_dispatch 0x561921fa3060 osd_map(47..51 src has 1..51) v4 2024-06-03T08:38:23.776 INFO:tasks.workunit.client.0.smithi088.stdout: api_asio: 2024-06-03T08:38:23.773+0000 7f5133fff640 3 client.4490.objecter handle_osd_map got epochs [47,51] > 46 2024-06-03T08:38:23.776 INFO:tasks.workunit.client.0.smithi088.stdout: api_asio: 2024-06-03T08:38:23.773+0000 7f5133fff640 3 client.4490.objecter handle_osd_map decoding incremental epoch 47 2024-06-03T08:38:23.776 INFO:tasks.workunit.client.0.smithi088.stdout: api_asio: 2024-06-03T08:38:23.773+0000 7f5133fff640 10 client.4490.objecter checking op 2 2024-06-03T08:38:23.776 INFO:tasks.workunit.client.0.smithi088.stdout: api_asio: 2024-06-03T08:38:23.773+0000 7f5133fff640 20 client.4490.objecter _calc_target epoch 47 base exist @17 precalc_pgid 0 pgid 0.0 is_write 2024-06-03T08:38:23.776 INFO:tasks.workunit.client.0.smithi088.stdout: api_asio: 2024-06-03T08:38:23.773+0000 7f5133fff640 20 client.4490.objecter _calc_target target exist @17 -> pgid 17.d3a59201 2024-06-03T08:38:23.776 INFO:tasks.workunit.client.0.smithi088.stdout: api_asio: 2024-06-03T08:38:23.773+0000 7f5133fff640 3 client.4490.objecter handle_osd_map decoding incremental epoch 48 2024-06-03T08:38:23.776 INFO:tasks.workunit.client.0.smithi088.stdout: api_asio: 2024-06-03T08:38:23.773+0000 7f5133fff640 10 client.4490.objecter checking op 2 2024-06-03T08:38:23.777 INFO:tasks.workunit.client.0.smithi088.stdout: api_asio: 2024-06-03T08:38:23.773+0000 7f5133fff640 20 client.4490.objecter _calc_target epoch 48 base exist @17 precalc_pgid 0 pgid 0.0 is_write 2024-06-03T08:38:24.777 INFO:tasks.workunit.client.0.smithi088.stdout: api_asio: 2024-06-03T08:38:23.773+0000 7f5133fff640 20 client.4490.objecter _calc_target target exist @17 -> pgid 17.d3a59201 2024-06-03T08:38:24.778 INFO:tasks.workunit.client.0.smithi088.stdout: api_asio: 2024-06-03T08:38:23.773+0000 7f5133fff640 3 client.4490.objecter handle_osd_map decoding incremental epoch 49 2024-06-03T08:38:24.778 INFO:tasks.workunit.client.0.smithi088.stdout: api_asio: 2024-06-03T08:38:23.773+0000 7f5133fff640 10 client.4490.objecter checking op 2 2024-06-03T08:38:24.778 INFO:tasks.workunit.client.0.smithi088.stdout: api_asio: 2024-06-03T08:38:23.773+0000 7f5133fff640 20 client.4490.objecter _calc_target epoch 49 base exist @17 precalc_pgid 0 pgid 0.0 is_write 2024-06-03T08:38:24.778 INFO:tasks.workunit.client.0.smithi088.stdout: api_asio: 2024-06-03T08:38:23.773+0000 7f5133fff640 20 client.4490.objecter _calc_target target exist @17 -> pgid 17.d3a59201 2024-06-03T08:38:24.778 INFO:tasks.workunit.client.0.smithi088.stdout: api_asio: 2024-06-03T08:38:23.773+0000 7f5133fff640 3 client.4490.objecter handle_osd_map decoding incremental epoch 50 2024-06-03T08:38:24.778 INFO:tasks.workunit.client.0.smithi088.stdout: api_asio: 2024-06-03T08:38:23.773+0000 7f5133fff640 10 client.4490.objecter checking op 2 2024-06-03T08:38:24.778 INFO:tasks.workunit.client.0.smithi088.stdout: api_asio: 2024-06-03T08:38:23.773+0000 7f5133fff640 20 client.4490.objecter _calc_target epoch 50 base exist @17 precalc_pgid 0 pgid 0.0 is_write 2024-06-03T08:38:24.778 INFO:tasks.workunit.client.0.smithi088.stdout: api_asio: 2024-06-03T08:38:23.773+0000 7f5133fff640 20 client.4490.objecter _calc_target target exist @17 -> pgid 17.d3a59201 2024-06-03T08:38:24.778 INFO:tasks.workunit.client.0.smithi088.stdout: api_asio: 2024-06-03T08:38:23.773+0000 7f5133fff640 3 client.4490.objecter handle_osd_map decoding incremental epoch 51 2024-06-03T08:38:24.778 INFO:tasks.workunit.client.0.smithi088.stdout: api_asio: 2024-06-03T08:38:23.774+0000 7f5133fff640 10 client.4490.objecter checking op 2 2024-06-03T08:38:24.778 INFO:tasks.workunit.client.0.smithi088.stdout: api_asio: 2024-06-03T08:38:23.774+0000 7f5133fff640 20 client.4490.objecter _calc_target epoch 51 base exist @17 precalc_pgid 0 pgid 0.0 is_write 2024-06-03T08:38:24.779 INFO:tasks.workunit.client.0.smithi088.stdout: api_asio: 2024-06-03T08:38:23.774+0000 7f5133fff640 20 client.4490.objecter _calc_target target exist @17 -> pgid 17.d3a59201 2024-06-03T08:38:24.779 INFO:tasks.workunit.client.0.smithi088.stdout: api_asio: 2024-06-03T08:38:23.774+0000 7f5133fff640 10 client.4490.objecter _maybe_request_map subscribing (onetime) to next osd map 2024-06-03T08:38:24.779 INFO:tasks.workunit.client.0.smithi088.stdout: api_asio: 2024-06-03T08:38:23.774+0000 7f5133fff640 10 monclient: _renew_subs 2024-06-03T08:38:24.779 INFO:tasks.workunit.client.0.smithi088.stdout: api_asio: 2024-06-03T08:38:23.774+0000 7f5133fff640 10 monclient: _send_mon_message to mon.c at v1:172.21.15.88:6790/0 2024-06-03T08:38:24.779 INFO:tasks.workunit.client.0.smithi088.stdout: api_asio: 2024-06-03T08:38:23.774+0000 7f5133fff640 1 -- 172.21.15.88:0/320712639 --> v1:172.21.15.88:6790/0 -- mon_subscribe({osdmap=52}) v3 -- 0x5619220c68b0 con 0x5619220ccfb0 2024-06-03T08:38:24.779 INFO:tasks.workunit.client.0.smithi088.stdout: api_asio: 2024-06-03T08:38:23.774+0000 7f5133fff640 20 client.4490.objecter dump_active .. 0 homeless 2024-06-03T08:38:24.779 INFO:tasks.workunit.client.0.smithi088.stdout: api_asio: 2024-06-03T08:38:23.774+0000 7f5133fff640 20 client.4490.objecter 2 17.d3a59201 osd.5 exist [writefull 0~5 in=5b] 2024-06-03T08:38:24.779 INFO:tasks.workunit.client.0.smithi088.stdout: api_asio: 2024-06-03T08:38:24.595+0000 7f5132ffd640 10 monclient: tick 2024-06-03T08:38:24.779 INFO:tasks.workunit.client.0.smithi088.stdout: api_asio: 2024-06-03T08:38:24.595+0000 7f5132ffd640 10 monclient: _check_auth_tickets 2024-06-03T08:38:24.779 INFO:tasks.workunit.client.0.smithi088.stdout: api_asio: 2024-06-03T08:38:24.595+0000 7f5132ffd640 20 monclient: _check_auth_rotating not needed by client.admin 2024-06-03T08:38:24.779 INFO:tasks.workunit.client.0.smithi088.stdout: api_asio: 2024-06-03T08:38:24.777+0000 7f5133fff640 1 -- 172.21.15.88:0/320712639 <== mon.2 v1:172.21.15.88:6790/0 16 ==== osd_map(52..52 src has 1..52) v4 ==== 764+0+0 (unknown 2533411274 0 0) 0x7f5144057230 con 0x5619220ccfb0 2024-06-03T08:38:24.779 INFO:tasks.workunit.client.0.smithi088.stdout: api_asio: 2024-06-03T08:38:24.777+0000 7f5133fff640 10 client.4490.objecter ms_dispatch 0x561921fa3060 osd_map(52..52 src has 1..52) v4 2024-06-03T08:38:24.780 INFO:tasks.workunit.client.0.smithi088.stdout: api_asio: 2024-06-03T08:38:24.777+0000 7f5133fff640 3 client.4490.objecter handle_osd_map got epochs [52,52] > 51 2024-06-03T08:38:24.780 INFO:tasks.workunit.client.0.smithi088.stdout: api_asio: 2024-06-03T08:38:24.777+0000 7f5133fff640 3 client.4490.objecter handle_osd_map decoding incremental epoch 52 2024-06-03T08:38:24.780 INFO:tasks.workunit.client.0.smithi088.stdout: api_asio: 2024-06-03T08:38:24.777+0000 7f5133fff640 10 client.4490.objecter checking op 2 2024-06-03T08:38:24.780 INFO:tasks.workunit.client.0.smithi088.stdout: api_asio: 2024-06-03T08:38:24.777+0000 7f5133fff640 20 client.4490.objecter _calc_target epoch 52 base exist @17 precalc_pgid 0 pgid 0.0 is_write 2024-06-03T08:38:24.780 INFO:tasks.workunit.client.0.smithi088.stdout: api_asio: 2024-06-03T08:38:24.777+0000 7f5133fff640 20 client.4490.objecter _calc_target target exist @17 -> pgid 17.d3a59201 2024-06-03T08:38:24.780 INFO:tasks.workunit.client.0.smithi088.stdout: api_asio: 2024-06-03T08:38:24.777+0000 7f5133fff640 10 client.4490.objecter _maybe_request_map subscribing (onetime) to next osd map 2024-06-03T08:38:25.144 INFO:tasks.ceph.mon.a.smithi088.stderr:==00:00:03:20.470 34743== Warning: unimplemented fcntl command: 1036 2024-06-03T08:38:25.151 INFO:tasks.ceph.mon.a.smithi088.stderr:==00:00:03:20.477 34743== Warning: unimplemented fcntl command: 1036 2024-06-03T08:38:25.262 INFO:tasks.ceph.mon.b.smithi192.stderr:==00:00:03:20.564 34228== Warning: unimplemented fcntl command: 1036 2024-06-03T08:38:25.500 INFO:tasks.ceph.mon.b.smithi192.stderr:==00:00:03:20.803 34228== Warning: unimplemented fcntl command: 1036 2024-06-03T08:38:25.652 INFO:tasks.ceph.mon.b.smithi192.stderr:==00:00:03:20.954 34228== Warning: unimplemented fcntl command: 1036 2024-06-03T08:38:25.865 INFO:tasks.ceph.mon.a.smithi088.stderr:==00:00:03:21.190 34743== Warning: unimplemented fcntl command: 1036 2024-06-03T08:38:26.010 INFO:tasks.workunit.client.0.smithi088.stdout:known 2024-06-03T08:38:26.010 INFO:tasks.workunit.client.0.smithi088.stdout: api_service: 76.090% pgs not active 2024-06-03T08:38:26.010 INFO:tasks.workunit.client.0.smithi088.stdout: api_service: 595 creating+peering 2024-06-03T08:38:26.010 INFO:tasks.workunit.client.0.smithi088.stdout: api_service: 162 unknown 2024-06-03T08:38:26.010 INFO:tasks.workunit.client.0.smithi088.stdout: api_service: 51 creating+activating 2024-06-03T08:38:26.010 INFO:tasks.workunit.client.0.smithi088.stdout: api_service: 41 active+clean 2024-06-03T08:38:26.011 INFO:tasks.workunit.client.0.smithi088.stdout: api_service: 2024-06-03T08:38:26.011 INFO:tasks.workunit.client.0.smithi088.stdout: api_service: io: 2024-06-03T08:38:26.011 INFO:tasks.workunit.client.0.smithi088.stdout: api_service: client: 239 B/s wr, 0 op/s rd, 0 op/s wr 2024-06-03T08:38:26.011 INFO:tasks.workunit.client.0.smithi088.stdout: api_service: 2024-06-03T08:38:26.011 INFO:tasks.workunit.client.0.smithi088.stdout: api_service: 2024-06-03T08:38:26.011 INFO:tasks.workunit.client.0.smithi088.stdout: api_service: [ OK ] LibRadosService.StatusFormat (5987 ms) 2024-06-03T08:38:26.011 INFO:tasks.workunit.client.0.smithi088.stdout: api_service: [ RUN ] LibRadosService.Status 2024-06-03T08:38:26.011 INFO:tasks.workunit.client.0.smithi088.stdout: api_service: [ OK ] LibRadosService.Status (20270 ms) 2024-06-03T08:38:26.011 INFO:tasks.workunit.client.0.smithi088.stdout: api_service: [----------] 4 tests from LibRadosService (32384 ms total) 2024-06-03T08:38:26.011 INFO:tasks.workunit.client.0.smithi088.stdout: api_service: 2024-06-03T08:38:26.011 INFO:tasks.workunit.client.0.smithi088.stdout: api_service: [----------] Global test environment tear-down 2024-06-03T08:38:26.011 INFO:tasks.workunit.client.0.smithi088.stdout: api_service: [==========] 4 tests from 1 test suite ran. (32384 ms total) 2024-06-03T08:38:26.011 INFO:tasks.workunit.client.0.smithi088.stdout: api_service: [ PASSED ] 4 tests. 2024-06-03T08:38:26.176 INFO:tasks.ceph.mon.c.smithi088.stderr:==00:00:03:21.502 34749== Warning: unimplemented fcntl command: 1036 2024-06-03T08:38:26.196 INFO:tasks.ceph.mon.c.smithi088.stderr:==00:00:03:21.522 34749== Warning: unimplemented fcntl command: 1036 2024-06-03T08:38:26.234 INFO:tasks.ceph.osd.7.smithi192.stderr:2024-06-03T08:38:26.223+0000 12a6c640 -1 osd.7 52 get_health_metrics reporting 1 slow ops, oldest is osd_op(client.4536.0:2 21.1a 21.db6ee63a (undecoded) ondisk+write+known_if_redirected+supports_pool_eio e28) 2024-06-03T08:38:26.312 INFO:tasks.workunit.client.0.smithi088.stdout: api_service_pp: [==========] Running 4 tests from 1 test suite. 2024-06-03T08:38:26.312 INFO:tasks.workunit.client.0.smithi088.stdout: api_service_pp: [----------] Global test environment set-up. 2024-06-03T08:38:26.312 INFO:tasks.workunit.client.0.smithi088.stdout: api_service_pp: [----------] 4 tests from LibRadosServicePP 2024-06-03T08:38:26.312 INFO:tasks.workunit.client.0.smithi088.stdout: api_service_pp: [ RUN ] LibRadosServicePP.RegisterEarly 2024-06-03T08:38:26.312 INFO:tasks.workunit.client.0.smithi088.stdout: api_service_pp: [ OK ] LibRadosServicePP.RegisterEarly (6221 ms) 2024-06-03T08:38:26.312 INFO:tasks.workunit.client.0.smithi088.stdout: api_service_pp: [ RUN ] LibRadosServicePP.RegisterLate 2024-06-03T08:38:26.312 INFO:tasks.workunit.client.0.smithi088.stdout: api_service_pp: [ OK ] LibRadosServicePP.RegisterLate (532 ms) 2024-06-03T08:38:26.312 INFO:tasks.workunit.client.0.smithi088.stdout: api_service_pp: [ RUN ] LibRadosServicePP.Status 2024-06-03T08:38:26.312 INFO:tasks.workunit.client.0.smithi088.stdout: api_service_pp: [ OK ] LibRadosServicePP.Status (20335 ms) 2024-06-03T08:38:26.312 INFO:tasks.workunit.client.0.smithi088.stdout: api_service_pp: [ RUN ] LibRadosServicePP.Close 2024-06-03T08:38:26.313 INFO:tasks.workunit.client.0.smithi088.stdout: api_service_pp: attempt 0 of 20 2024-06-03T08:38:26.313 INFO:tasks.workunit.client.0.smithi088.stdout: api_service_pp: failed to deregister: 2024-06-03T08:38:26.313 INFO:tasks.workunit.client.0.smithi088.stdout: api_service_pp: { 2024-06-03T08:38:26.313 INFO:tasks.workunit.client.0.smithi088.stdout: api_service_pp: "epoch": 12, 2024-06-03T08:38:26.313 INFO:tasks.workunit.client.0.smithi088.stdout: api_service_pp: "modified": "2024-06-03T08:38:22.624631+0000", 2024-06-03T08:38:26.313 INFO:tasks.workunit.client.0.smithi088.stdout: api_service_pp: "services": { 2024-06-03T08:38:26.313 INFO:tasks.workunit.client.0.smithi088.stdout: api_service_pp: "laundry": { 2024-06-03T08:38:26.313 INFO:tasks.workunit.client.0.smithi088.stdout: api_service_pp: "daemons": { 2024-06-03T08:38:26.313 INFO:tasks.workunit.client.0.smithi088.stdout: api_service_pp: "summary": "", 2024-06-03T08:38:26.313 INFO:tasks.workunit.client.0.smithi088.stdout: api_service_pp: "close-test-pid40818": { 2024-06-03T08:38:26.313 INFO:tasks.workunit.client.0.smithi088.stdout: api_service_pp: "start_epoch": 12, 2024-06-03T08:38:26.313 INFO:tasks.workunit.client.0.smithi088.stdout: api_service_pp: "start_stamp": "2024-06-03T08:38:20.752766+0000", 2024-06-03T08:38:26.313 INFO:tasks.workunit.client.0.smithi088.stdout: api_service_pp: "gid": 4756, 2024-06-03T08:38:26.314 INFO:tasks.workunit.client.0.smithi088.stdout: api_service_pp: "addr": "172.21.15.88:0/2303049498", 2024-06-03T08:38:26.314 INFO:tasks.workunit.client.0.smithi088.stdout: api_service_pp: "metadata": { 2024-06-03T08:38:26.314 INFO:tasks.workunit.client.0.smithi088.stdout: api_service_pp: "arch": "x86_64", 2024-06-03T08:38:26.314 INFO:tasks.workunit.client.0.smithi088.stdout: api_service_pp: "ceph_release": "reef", 2024-06-03T08:38:26.314 INFO:tasks.workunit.client.0.smithi088.stdout: api_service_pp: "ceph_version": "ceph version 18.2.2-1389-gf1abf795 (f1abf795f4fc15bf2f30ac90c811a53ab21d5d65) reef (stable)", 2024-06-03T08:38:26.314 INFO:tasks.workunit.client.0.smithi088.stdout: api_service_pp: "ceph_version_short": "18.2.2-1389-gf1abf795", 2024-06-03T08:38:26.314 INFO:tasks.workunit.client.0.smithi088.stdout: api_service_pp: "cpu": "Intel(R) Xeon(R) CPU E5-1620 v4 @ 3.50GHz", 2024-06-03T08:38:26.314 INFO:tasks.workunit.client.0.smithi088.stdout: api_service_pp: "distro": "centos", 2024-06-03T08:38:26.314 INFO:tasks.workunit.client.0.smithi088.stdout: api_service_pp: "distro_description": "CentOS Stream 9", 2024-06-03T08:38:26.314 INFO:tasks.workunit.client.0.smithi088.stdout: api_service_pp: "distro_version": "9", 2024-06-03T08:38:26.314 INFO:tasks.workunit.client.0.smithi088.stdout: api_service_pp: "foo": "bar", 2024-06-03T08:38:26.314 INFO:tasks.workunit.client.0.smithi088.stdout: api_service_pp: "hostname": "smithi088", 2024-06-03T08:38:26.314 INFO:tasks.workunit.client.0.smithi088.stdout: api_service_pp: "kernel_description": "#1 SMP PREEMPT_DYNAMIC Sat May 18 20:39:48 UTC 2024", 2024-06-03T08:38:26.315 INFO:tasks.workunit.client.0.smithi088.stdout: api_service_pp: "kernel_version": "5.14.0-452.el9.x86_64", 2024-06-03T08:38:26.315 INFO:tasks.workunit.client.0.smithi088.stdout: api_service_pp: "mem_swap_kb": "0", 2024-06-03T08:38:26.315 INFO:tasks.workunit.client.0.smithi088.stdout: api_service_pp: "mem_total_kb": "32488156", 2024-06-03T08:38:26.315 INFO:tasks.workunit.client.0.smithi088.stdout: api_service_pp: "os": "Linux", 2024-06-03T08:38:26.315 INFO:tasks.workunit.client.0.smithi088.stdout: api_service_pp: "this": "that" 2024-06-03T08:38:26.315 INFO:tasks.workunit.client.0.smithi088.stdout: api_service_pp: }, 2024-06-03T08:38:26.315 INFO:tasks.workunit.client.0.smithi088.stdout: api_service_pp: "task_status": {} 2024-06-03T08:38:26.315 INFO:tasks.workunit.client.0.smithi088.stdout: api_service_pp: }, 2024-06-03T08:38:26.315 INFO:tasks.workunit.client.0.smithi088.stdout: api_service_pp: "pid40871": { 2024-06-03T08:38:26.315 INFO:tasks.workunit.client.0.smithi088.stdout: api_service_pp: "start_epoch": 11, 2024-06-03T08:38:26.315 INFO:tasks.workunit.client.0.smithi088.stdout: api_service_pp: "start_stamp": "2024-06-03T08:38:06.039408+0000", 2024-06-03T08:38:26.315 INFO:tasks.workunit.client.0.smithi088.stdout: api_service_pp: "gid": 4719, 2024-06-03T08:38:26.315 INFO:tasks.workunit.client.0.smithi088.stdout: api_service_pp: "addr": "172.21.15.88:0/3197391072", 2024-06-03T08:38:26.316 INFO:tasks.workunit.client.0.smithi088.stdout: api_service_pp: "metadata": { 2024-06-03T08:38:26.316 INFO:tasks.workunit.client.0.smithi088.stdout: api_service_pp: "arch": "x86_64", 2024-06-03T08:38:26.316 INFO:tasks.workunit.client.0.smithi088.stdout: api_service_pp: "ceph_release": "reef", 2024-06-03T08:38:26.316 INFO:tasks.workunit.client.0.smithi088.stdout: api_service_pp: "ceph_version": "ceph version 18.2.2-1389-gf1abf795 (f1abf795f4fc15bf2f30ac90c811a53ab21d5d65) reef (stable)", 2024-06-03T08:38:26.316 INFO:tasks.workunit.client.0.smithi088.stdout: api_service_pp: "ceph_version_short": "18.2.2-1389-gf1abf795", 2024-06-03T08:38:26.316 INFO:tasks.workunit.client.0.smithi088.stdout: api_service_pp: "cpu": "Intel(R) Xeon(R) CPU E5-1620 v4 @ 3.50GHz", 2024-06-03T08:38:26.338 INFO:tasks.ceph.mon.c.smithi088.stderr:==00:00:03:21.663 34749== Warning: unimplemented fcntl command: 1036 2024-06-03T08:38:26.409 INFO:tasks.workunit.client.0.smithi088.stdout: api_misc: [==========] Running 12 tests from 4 test suites. 2024-06-03T08:38:26.409 INFO:tasks.workunit.client.0.smithi088.stdout: api_misc: [----------] Global test environment set-up. 2024-06-03T08:38:26.409 INFO:tasks.workunit.client.0.smithi088.stdout: api_misc: [----------] 1 test from LibRadosMiscVersion 2024-06-03T08:38:26.409 INFO:tasks.workunit.client.0.smithi088.stdout: api_misc: [ RUN ] LibRadosMiscVersion.Version 2024-06-03T08:38:26.410 INFO:tasks.workunit.client.0.smithi088.stdout: api_misc: [ OK ] LibRadosMiscVersion.Version (0 ms) 2024-06-03T08:38:26.410 INFO:tasks.workunit.client.0.smithi088.stdout: api_misc: [----------] 1 test from LibRadosMiscVersion (0 ms total) 2024-06-03T08:38:26.410 INFO:tasks.workunit.client.0.smithi088.stdout: api_misc: 2024-06-03T08:38:26.410 INFO:tasks.workunit.client.0.smithi088.stdout: api_misc: [----------] 2 tests from LibRadosMiscConnectFailure 2024-06-03T08:38:26.410 INFO:tasks.workunit.client.0.smithi088.stdout: api_misc: [ RUN ] LibRadosMiscConnectFailure.ConnectFailure 2024-06-03T08:38:26.410 INFO:tasks.workunit.client.0.smithi088.stdout: api_misc: unable to get monitor info from DNS SRV with service name: ceph-mon 2024-06-03T08:38:26.410 INFO:tasks.workunit.client.0.smithi088.stdout: api_misc: 2024-06-03T08:37:53.836+0000 7f1b4a571780 -1 failed for service _ceph-mon._tcp 2024-06-03T08:38:26.410 INFO:tasks.workunit.client.0.smithi088.stdout: api_misc: 2024-06-03T08:37:53.836+0000 7f1b4a571780 -1 monclient: get_monmap_and_config cannot identify monitors to contact 2024-06-03T08:38:26.410 INFO:tasks.workunit.client.0.smithi088.stdout: api_misc: [ OK ] LibRadosMiscConnectFailure.ConnectFailure (372 ms) 2024-06-03T08:38:26.410 INFO:tasks.workunit.client.0.smithi088.stdout: api_misc: [ RUN ] LibRadosMiscConnectFailure.ConnectTimeout 2024-06-03T08:38:26.410 INFO:tasks.workunit.client.0.smithi088.stdout: api_misc: [ OK ] LibRadosMiscConnectFailure.ConnectTimeout (2006 ms) 2024-06-03T08:38:26.410 INFO:tasks.workunit.client.0.smithi088.stdout: api_misc: [----------] 2 tests from LibRadosMiscConnectFailure (2379 ms total) 2024-06-03T08:38:26.411 INFO:tasks.workunit.client.0.smithi088.stdout: api_misc: 2024-06-03T08:38:26.411 INFO:tasks.workunit.client.0.smithi088.stdout: api_misc: [----------] 1 test from LibRadosMiscPool 2024-06-03T08:38:26.411 INFO:tasks.workunit.client.0.smithi088.stdout: api_misc: [ RUN ] LibRadosMiscPool.PoolCreationRace 2024-06-03T08:38:26.411 INFO:tasks.workunit.client.0.smithi088.stdout: api_misc: started 0x558841982010 2024-06-03T08:38:26.411 INFO:tasks.workunit.client.0.smithi088.stdout: api_misc: started 0x558841ba65c0 2024-06-03T08:38:26.411 INFO:tasks.workunit.client.0.smithi088.stdout: api_misc: started 0x558841b9b7e0 2024-06-03T08:38:26.411 INFO:tasks.workunit.client.0.smithi088.stdout: api_misc: started 0x558841d13070 2024-06-03T08:38:26.411 INFO:tasks.workunit.client.0.smithi088.stdout: api_misc: started 0x558841d13af0 2024-06-03T08:38:26.411 INFO:tasks.workunit.client.0.smithi088.stdout: api_misc: started 0x558841d14530 2024-06-03T08:38:26.411 INFO:tasks.workunit.client.0.smithi088.stdout: api_misc: started 0x558841d14fb0 2024-06-03T08:38:26.411 INFO:tasks.workunit.client.0.smithi088.stdout: api_misc: started 0x558841d15a90 2024-06-03T08:38:26.411 INFO:tasks.workunit.client.0.smithi088.stdout: api_misc: started 0x558841d16570 2024-06-03T08:38:26.411 INFO:tasks.workunit.client.0.smithi088.stdout: api_misc: started 0x558841d17020 2024-06-03T08:38:26.411 INFO:tasks.workunit.client.0.smithi088.stdout: api_misc: started 0x558841d17b20 2024-06-03T08:38:26.412 INFO:tasks.workunit.client.0.smithi088.stdout: api_misc: started 0x558841bdb8b0 2024-06-03T08:38:26.412 INFO:tasks.workunit.client.0.smithi088.stdout: api_misc: started 0x558841bdbf10 2024-06-03T08:38:26.412 INFO:tasks.workunit.client.0.smithi088.stdout: api_misc: started 0x558841bdc570 2024-06-03T08:38:26.412 INFO:tasks.workunit.client.0.smithi088.stdout: api_misc: started 0x558841d0f9d0 2024-06-03T08:38:26.412 INFO:tasks.workunit.client.0.smithi088.stdout: api_misc: started 0x558841d10550 2024-06-03T08:38:26.412 INFO:tasks.workunit.client.0.smithi088.stdout: api_misc: started 0x558841d110f0 2024-06-03T08:38:26.412 INFO:tasks.workunit.client.0.smithi088.stdout: api_misc: started 0x558841ad4a70 2024-06-03T08:38:26.412 INFO:tasks.workunit.client.0.smithi088.stdout: api_misc: started 0x558841ad5550 2024-06-03T08:38:26.412 INFO:tasks.workunit.client.0.smithi088.stdout: api_misc: started 0x558841ad5bf0 2024-06-03T08:38:26.412 INFO:tasks.workunit.client.0.smithi088.stdout: api_misc: started 0x558841ad5140 2024-06-03T08:38:26.412 INFO:tasks.workunit.client.0.smithi088.stdout: api_misc: started 0x558841ad5360 2024-06-03T08:38:26.412 INFO:tasks.workunit.client.0.smithi088.stdout: api_misc: started 0x558841ad4750 2024-06-03T08:38:26.412 INFO:tasks.workunit.client.0.smithi088.stdout: api_misc: started 0x558842662fb0 2024-06-03T08:38:26.412 INFO:tasks.workunit.client.0.smithi088.stdout: api_misc: started 0x558841d10da0 2024-06-03T08:38:26.413 INFO:tasks.workunit.client.0.smithi088.stdout: api_misc: started 0x558841d101a0 2024-06-03T08:38:26.413 INFO:tasks.workunit.client.0.smithi088.stdout: api_misc: started 26 aios 2024-06-03T08:38:26.413 INFO:tasks.workunit.client.0.smithi088.stdout: api_misc: waiting 0x558841982010 2024-06-03T08:38:26.413 INFO:tasks.workunit.client.0.smithi088.stdout: api_misc: waiting 0x558841ba65c0 2024-06-03T08:38:26.413 INFO:tasks.workunit.client.0.smithi088.stdout: api_misc: waiting 0x558841b9b7e0 2024-06-03T08:38:26.413 INFO:tasks.workunit.client.0.smithi088.stdout: api_misc: waiting 0x558841d13070 2024-06-03T08:38:26.413 INFO:tasks.workunit.client.0.smithi088.stdout: api_misc: waiting 0x558841d13af0 2024-06-03T08:38:26.413 INFO:tasks.workunit.client.0.smithi088.stdout: api_misc: waiting 0x558841d14530 2024-06-03T08:38:26.413 INFO:tasks.workunit.client.0.smithi088.stdout: api_misc: waiting 0x558841d14fb0 2024-06-03T08:38:26.413 INFO:tasks.workunit.client.0.smithi088.stdout: api_misc: waiting 0x558841d15a90 2024-06-03T08:38:26.413 INFO:tasks.workunit.client.0.smithi088.stdout: api_misc: waiting 0x558841d16570 2024-06-03T08:38:26.413 INFO:tasks.workunit.client.0.smithi088.stdout: api_misc: waiting 0x558841d17020 2024-06-03T08:38:26.413 INFO:tasks.workunit.client.0.smithi088.stdout: api_misc: waiting 0x558841d17b20 2024-06-03T08:38:26.414 INFO:tasks.workunit.client.0.smithi088.stdout: api_misc: waiting 0x558841bdb8b0 2024-06-03T08:38:26.414 INFO:tasks.workunit.client.0.smithi088.stdout: api_misc: waiting 0x558841bdbf10 2024-06-03T08:38:26.414 INFO:tasks.workunit.client.0.smithi088.stdout: api_misc: waiting 0x558841bdc570 2024-06-03T08:38:26.414 INFO:tasks.workunit.client.0.smithi088.stdout: api_misc: waiting 0x558841d0f9d0 2024-06-03T08:38:26.414 INFO:tasks.workunit.client.0.smithi088.stdout: api_misc: waiting 0x558841d10550 2024-06-03T08:38:26.414 INFO:tasks.workunit.client.0.smithi088.stdout: api_misc: waiting 0x558841d110f0 2024-06-03T08:38:26.414 INFO:tasks.workunit.client.0.smithi088.stdout: api_misc: waiting 0x558841ad4a70 2024-06-03T08:38:26.414 INFO:tasks.workunit.client.0.smithi088.stdout: api_misc: waiting 0x558841ad5550 2024-06-03T08:38:26.414 INFO:tasks.workunit.client.0.smithi088.stdout: api_misc: waiting 0x558841ad5bf0 2024-06-03T08:38:26.414 INFO:tasks.workunit.client.0.smithi088.stdout: api_misc: waiting 0x558841ad5140 2024-06-03T08:38:26.414 INFO:tasks.workunit.client.0.smithi088.stdout: api_misc: waiting 0x558841ad5360 2024-06-03T08:38:26.414 INFO:tasks.workunit.client.0.smithi088.stdout: api_misc: waiting 0x558841ad4750 2024-06-03T08:38:26.414 INFO:tasks.workunit.client.0.smithi088.stdout: api_misc: waiting 0x558842662fb0 2024-06-03T08:38:26.446 INFO:tasks.ceph.osd.5.smithi192.stderr:2024-06-03T08:38:26.441+0000 12a6c640 -1 osd.5 52 get_health_metrics reporting 1 slow ops, oldest is osd_op(client.4490.0:2 17.1 17:8049a5cb:::exist:head [writefull 0~5 in=5b] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e28) 2024-06-03T08:38:26.936 INFO:tasks.workunit.client.0.smithi088.stdout: api_misc: waiting 0x55 api_asio: 2024-06-03T08:38:24.777+0000 7f5133fff640 10 monclient: _renew_subs 2024-06-03T08:38:26.936 INFO:tasks.workunit.client.0.smithi088.stdout: api_asio: 2024-06-03T08:38:24.777+0000 7f5133fff640 10 monclient: _send_mon_message to mon.c at v1:172.21.15.88:6790/0 2024-06-03T08:38:26.936 INFO:tasks.workunit.client.0.smithi088.stdout: api_asio: 2024-06-03T08:38:24.777+0000 7f5133fff640 1 -- 172.21.15.88:0/320712639 --> v1:172.21.15.88:6790/0 -- mon_subscribe({osdmap=53}) v3 -- 0x7f5138079640 con 0x5619220ccfb0 2024-06-03T08:38:26.936 INFO:tasks.workunit.client.0.smithi088.stdout: api_asio: 2024-06-03T08:38:24.777+0000 7f5133fff640 20 client.4490.objecter dump_active .. 0 homeless 2024-06-03T08:38:26.936 INFO:tasks.workunit.client.0.smithi088.stdout: api_asio: 2024-06-03T08:38:24.777+0000 7f5133fff640 20 client.4490.objecter 2 17.d3a59201 osd.5 exist [writefull 0~5 in=5b] 2024-06-03T08:38:26.937 INFO:tasks.workunit.client.0.smithi088.stdout: api_asio: 2024-06-03T08:38:25.595+0000 7f5132ffd640 10 monclient: tick 2024-06-03T08:38:26.937 INFO:tasks.workunit.client.0.smithi088.stdout: api_asio: 2024-06-03T08:38:25.595+0000 7f5132ffd640 10 monclient: _check_auth_tickets 2024-06-03T08:38:26.937 INFO:tasks.workunit.client.0.smithi088.stdout: api_asio: 2024-06-03T08:38:25.595+0000 7f5132ffd640 20 monclient: _check_auth_rotating not needed by client.admin 2024-06-03T08:38:26.937 INFO:tasks.workunit.client.0.smithi088.stdout: api_asio: 2024-06-03T08:38:26.595+0000 7f5132ffd640 10 monclient: tick 2024-06-03T08:38:26.937 INFO:tasks.workunit.client.0.smithi088.stdout: api_asio: 2024-06-03T08:38:26.595+0000 7f5132ffd640 10 monclient: _check_auth_tickets 2024-06-03T08:38:26.937 INFO:tasks.workunit.client.0.smithi088.stdout: api_asio: 2024-06-03T08:38:26.595+0000 7f5132ffd640 20 monclient: _check_auth_rotating not needed by client.admin 2024-06-03T08:38:26.937 INFO:tasks.workunit.client.0.smithi088.stdout: api_asio: 2024-06-03T08:38:26.595+0000 7f5132ffd640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-06-03T08:38:26.937 INFO:tasks.workunit.client.0.smithi088.stdout: api_asio: 2024-06-03T08:38:26.904+0000 7f5133fff640 1 -- 172.21.15.88:0/320712639 <== mon.2 v1:172.21.15.88:6790/0 17 ==== osd_map(53..53 src has 1..53) v4 ==== 1594+0+0 (unknown 3355576616 0 0) 0x7f51440501e0 con 0x5619220ccfb0 2024-06-03T08:38:26.937 INFO:tasks.workunit.client.0.smithi088.stdout: api_asio: 2024-06-03T08:38:26.904+0000 7f5133fff640 10 client.4490.objecter ms_dispatch 0x561921fa3060 osd_map(53..53 src has 1..53) v4 2024-06-03T08:38:26.937 INFO:tasks.workunit.client.0.smithi088.stdout: api_asio: 2024-06-03T08:38:26.904+0000 7f5133fff640 3 client.4490.objecter handle_osd_map got epochs [53,53] > 52 2024-06-03T08:38:26.937 INFO:tasks.workunit.client.0.smithi088.stdout: api_asio: 2024-06-03T08:38:26.904+0000 7f5133fff640 3 client.4490.objecter handle_osd_map decoding incremental epoch 53 2024-06-03T08:38:26.937 INFO:tasks.workunit.client.0.smithi088.stdout: api_asio: 2024-06-03T08:38:26.904+0000 7f5133fff640 10 client.4490.objecter checking op 2 2024-06-03T08:38:26.938 INFO:tasks.workunit.client.0.smithi088.stdout: api_asio: 2024-06-03T08:38:26.904+0000 7f5133fff640 20 client.4490.objecter _calc_target epoch 53 base exist @17 precalc_pgid 0 pgid 0.0 is_write 2024-06-03T08:38:26.938 INFO:tasks.workunit.client.0.smithi088.stdout: api_asio: 2024-06-03T08:38:26.904+0000 7f5133fff640 20 client.4490.objecter _calc_target target exist @17 -> pgid 17.d3a59201 2024-06-03T08:38:26.938 INFO:tasks.workunit.client.0.smithi088.stdout: api_asio: 2024-06-03T08:38:26.904+0000 7f5133fff640 10 client.4490.objecter _maybe_request_map subscribing (onetime) to next osd map 2024-06-03T08:38:26.938 INFO:tasks.workunit.client.0.smithi088.stdout: api_asio: 2024-06-03T08:38:26.904+0000 7f5133fff640 10 monclient: _renew_subs 2024-06-03T08:38:26.938 INFO:tasks.workunit.client.0.smithi088.stdout: api_asio: 2024-06-03T08:38:26.904+0000 7f5133fff640 10 monclient: _send_mon_message to mon.c at v1:172.21.15.88:6790/0 2024-06-03T08:38:26.938 INFO:tasks.workunit.client.0.smithi088.stdout: api_asio: 2024-06-03T08:38:26.904+0000 7f5133fff640 1 -- 172.21.15.88:0/320712639 --> v1:172.21.15.88:6790/0 -- mon_subscribe({osdmap=54}) v3 -- 0x7f513808be30 con 0x5619220ccfb0 2024-06-03T08:38:26.938 INFO:tasks.workunit.client.0.smithi088.stdout: api_asio: 2024-06-03T08:38:26.904+0000 7f5133fff640 20 client.4490.objecter dump_active .. 0 homeless 2024-06-03T08:38:26.938 INFO:tasks.workunit.client.0.smithi088.stdout: api_asio: 2024-06-03T08:38:26.904+0000 7f5133fff640 20 client.4490.objecter 2 17.d3a59201 osd.5 exist [writefull 0~5 in=5b] 2024-06-03T08:38:26.938 INFO:tasks.workunit.client.0.smithi088.stdout: api_asio: 2024-06-03T08:38:26.935+0000 7f514a183640 1 -- 172.21.15.88:0/320712639 <== osd.5 v1:172.21.15.192:6805/34786 2 ==== osd_op_reply(2 exist [writefull 0~5] v44'1 uv1 ondisk = 0) v8 ==== 149+0+0 (unknown 1493069790 0 0) 0x7f513c0043a0 con 0x5619220df280 2024-06-03T08:38:26.938 INFO:tasks.workunit.client.0.smithi088.stdout: api_asio: 2024-06-03T08:38:26.935+0000 7f514a183640 10 client.4490.objecter ms_dispatch 0x561921fa3060 osd_op_reply(2 exist [writefull 0~5] v44'1 uv1 ondisk = 0) v8 2024-06-03T08:38:26.938 INFO:tasks.workunit.client.0.smithi088.stdout: api_asio: 2024-06-03T08:38:26.935+0000 7f514a183640 10 client.4490.objecter in handle_osd_op_reply 2024-06-03T08:38:26.938 INFO:tasks.workunit.client.0.smithi088.stdout: api_asio: 2024-06-03T08:38:26.935+0000 7f514a183640 7 client.4490.objecter handle_osd_op_reply 2 ondisk uv 1 in 17.1 attempt 0 2024-06-03T08:38:26.939 INFO:tasks.workunit.client.0.smithi088.stdout: api_asio: 2024-06-03T08:38:26.935+0000 7f514a183640 10 client.4490.objecter op 0 rval 0 len 0 2024-06-03T08:38:26.939 INFO:tasks.workunit.client.0.smithi088.stdout: api_asio: 2024-06-03T08:38:26.935+0000 7f514a183640 15 client.4490.objecter handle_osd_op_reply completed tid 2 2024-06-03T08:38:26.939 INFO:tasks.workunit.client.0.smithi088.stdout: api_asio: 2024-06-03T08:38:26.935+0000 7f514a183640 15 client.4490.objecter _finish_op 2 2024-06-03T08:38:26.939 INFO:tasks.workunit.client.0.smithi088.stdout: api_asio: 2024-06-03T08:38:26.935+0000 7f514a183640 20 client.4490.objecter put_session s=0x5619220df110 osd=5 4 2024-06-03T08:38:26.939 INFO:tasks.workunit.client.0.smithi088.stdout: api_asio: 2024-06-03T08:38:26.935+0000 7f514a183640 15 client.4490.objecter _session_op_remove 5 2 2024-06-03T08:38:26.939 INFO:tasks.workunit.client.0.smithi088.stdout: api_asio: 2024-06-03T08:38:26.935+0000 7f514a183640 5 client.4490.objecter 0 in flight 2024-06-03T08:38:26.939 INFO:tasks.workunit.client.0.smithi088.stdout: api_asio: [ RUN ] AsioRados.AsyncReadCallback 2024-06-03T08:38:26.939 INFO:tasks.workunit.client.0.smithi088.stdout: api_asio: 2024-06-03T08:38:26.935+0000 7f514d647780 10 librados: Objecter returned from writefull r=0 2024-06-03T08:38:26.939 INFO:tasks.workunit.client.0.smithi088.stdout: api_asio: 2024-06-03T08:38:26.935+0000 7f514d647780 10 client.4490.objecter _op_submit op 0x5619220b1610 2024-06-03T08:38:26.939 INFO:tasks.workunit.client.0.smithi088.stdout: api_asio: 2024-06-03T08:38:26.935+0000 7f514d647780 20 client.4490.objecter _calc_target epoch 53 base exist @17 precalc_pgid 0 pgid 0.0 is_read 2024-06-03T08:38:26.939 INFO:tasks.workunit.client.0.smithi088.stdout: api_asio: 2024-06-03T08:38:26.935+0000 7f514d647780 20 client.4490.objecter _calc_target target exist @17 -> pgid 17.d3a59201 2024-06-03T08:38:26.940 INFO:tasks.workunit.client.0.smithi088.stdout: api_asio: 2024-06-03T08:38:26.935+0000 7f514d647780 10 client.4490.objecter _calc_target raw pgid 17.d3a59201 -> actual 17.1 acting [5,6] primary 5 2024-06-03T08:38:26.940 INFO:tasks.workunit.client.0.smithi088.stdout: api_asio: 2024-06-03T08:38:26.935+0000 7f514d647780 20 client.4490.objecter _get_session s=0x5619220df110 osd=5 3 2024-06-03T08:38:26.940 INFO:tasks.workunit.client.0.smithi088.stdout: api_asio: 2024-06-03T08:38:26.935+0000 7f514d647780 10 client.4490.objecter _op_submit oid exist '@17' '@17' [read 0~256] tid 3 osd.5 2024-06-03T08:38:26.940 INFO:tasks.workunit.client.0.smithi088.stdout: api_asio: 2024-06-03T08:38:26.935+0000 7f514d647780 20 client.4490.objecter get_session s=0x5619220df110 osd=5 3 2024-06-03T08:38:26.940 INFO:tasks.workunit.client.0.smithi088.stdout: api_asio: 2024-06-03T08:38:26.935+0000 7f514d647780 15 client.4490.objecter _session_op_assign 5 3 2024-06-03T08:38:26.940 INFO:tasks.workunit.client.0.smithi088.stdout: api_asio: 2024-06-03T08:38:26.935+0000 7f514d647780 15 client.4490.objecter _send_op 3 to 17.1 on osd.5 2024-06-03T08:38:26.940 INFO:tasks.workunit.client.0.smithi088.stdout: api_asio: 2024-06-03T08:38:26.935+0000 7f514d647780 1 -- 172.21.15.88:0/320712639 --> v1:172.21.15.192:6805/34786 -- osd_op(unknown.0.0:3 17.1 17:8049a5cb:::exist:head [read 0~256] snapc 0=[] ondisk+read+known_if_redirected+supports_pool_eio e53) v8 -- 0x561922174ab0 con 0x5619220df280 2024-06-03T08:38:26.940 INFO:tasks.workunit.client.0.smithi088.stdout: api_asio: 2024-06-03T08:38:26.935+0000 7f514d647780 20 client.4490.objecter put_session s=0x5619220df110 osd=5 4 2024-06-03T08:38:26.940 INFO:tasks.workunit.client.0.smithi088.stdout: api_asio: 2024-06-03T08:38:26.935+0000 7f514d647780 5 client.4490.objecter 1 in flight 2024-06-03T08:38:26.940 INFO:tasks.workunit.client.0.smithi088.stdout: api_asio: 2024-06-03T08:38:26.935+0000 7f514d647780 10 client.4490.objecter _op_submit op 0x561922174e60 2024-06-03T08:38:26.940 INFO:tasks.workunit.client.0.smithi088.stdout: api_asio: 2024-06-03T08:38:26.935+0000 7f514d647780 20 client.4490.objecter _calc_target epoch 53 base noexist @17 precalc_pgid 0 pgid 0.0 is_read 2024-06-03T08:38:26.940 INFO:tasks.workunit.client.0.smithi088.stdout: api_asio: 2024-06-03T08:38:26.935+0000 7f514d647780 20 client.4490.objecter _calc_target target noexist @17 -> pgid 17.a7bbaa1 2024-06-03T08:38:26.940 INFO:tasks.workunit.client.0.smithi088.stdout: api_asio: 2024-06-03T08:38:26.935+0000 7f514d647780 10 client.4490.objecter _calc_target raw pgid 17.a7bbaa1 -> actual 17.1 acting [5,6] primary 5 2024-06-03T08:38:26.941 INFO:tasks.workunit.client.0.smithi088.stdout: api_asio: 2024-06-03T08:38:26.935+0000 7f514d647780 20 client.4490.objecter _get_session s=0x5619220df110 osd=5 4 2024-06-03T08:38:26.941 INFO:tasks.workunit.client.0.smithi088.stdout: api_asio: 2024-06-03T08:38:26.935+0000 7f514d647780 10 client.4490.objecter _op_submit oid noexist '@17' '@17' [read 0~256] tid 4 osd.5 2024-06-03T08:38:26.941 INFO:tasks.workunit.client.0.smithi088.stdout: api_asio: 2024-06-03T08:38:26.935+0000 7f514d647780 20 client.4490.objecter get_session s=0x5619220df110 osd=5 4 2024-06-03T08:38:26.941 INFO:tasks.workunit.client.0.smithi088.stdout: api_asio: 2024-06-03T08:38:26.935+0000 7f514d647780 15 client.4490.objecter _session_op_assign 5 4 2024-06-03T08:38:26.941 INFO:tasks.workunit.client.0.smithi088.stdout: api_asio: 2024-06-03T08:38:26.935+0000 7f514d647780 15 client.4490.objecter _send_op 4 to 17.1 on osd.5 2024-06-03T08:38:27.567 INFO:tasks.ceph.osd.7.smithi192.stderr:2024-06-03T08:38:27.566+0000 12a6c640 -1 osd.7 53 get_health_metrics reporting 1 slow ops, oldest is osd_op(client.4536.0:2 21.1a 21.db6ee63a (undecoded) ondisk+write+known_if_redirected+supports_pool_eio e28) 2024-06-03T08:38:27.626 INFO:tasks.workunit.client.0.smithi088.stdout: api_asio: 2024-06-03T08:38:26.935+0000 7f514d647780 1 -- 172.21.15.88:0/320712639 --> v1:172.21.15.192:6805/34786 -- osd_op(unknown.0.0:4 17.1 17:855dde50:::noexist:head [read 0~256] snapc 0=[] ondisk+read+known_if_redirected+supports_pool_eio e53) v8 -- 0x561922175350 con 0x5619220df280 2024-06-03T08:38:27.626 INFO:tasks.workunit.client.0.smithi088.stdout: api_asio: 2024-06-03T08:38:26.935+0000 7f514d647780 20 client.4490.objecter put_session s=0x5619220df110 osd=5 5 2024-06-03T08:38:27.626 INFO:tasks.workunit.client.0.smithi088.stdout: api_asio: 2024-06-03T08:38:26.935+0000 7f514d647780 5 client.4490.objecter 2 in flight 2024-06-03T08:38:27.626 INFO:tasks.workunit.client.0.smithi088.stdout: api_asio: 2024-06-03T08:38:27.352+0000 7f514a183640 1 -- 172.21.15.88:0/320712639 <== osd.5 v1:172.21.15.192:6805/34786 3 ==== osd_op_reply(3 exist [read 0~5 out=5b] v0'0 uv1 ondisk = 0) v8 ==== 149+0+5 (unknown 704808412 0 3741568377) 0x7f513c0043a0 con 0x5619220df280 2024-06-03T08:38:27.626 INFO:tasks.workunit.client.0.smithi088.stdout: api_asio: 2024-06-03T08:38:27.352+0000 7f514a183640 10 client.4490.objecter ms_dispatch 0x561921fa3060 osd_op_reply(3 exist [read 0~5 out=5b] v0'0 uv1 ondisk = 0) v8 2024-06-03T08:38:27.627 INFO:tasks.workunit.client.0.smithi088.stdout: api_asio: 2024-06-03T08:38:27.352+0000 7f514a183640 10 client.4490.objecter in handle_osd_op_reply 2024-06-03T08:38:27.627 INFO:tasks.workunit.client.0.smithi088.stdout: api_asio: 2024-06-03T08:38:27.352+0000 7f514a183640 7 client.4490.objecter handle_osd_op_reply 3 ondisk uv 1 in 17.1 attempt 0 2024-06-03T08:38:27.627 INFO:tasks.workunit.client.0.smithi088.stdout: api_asio: 2024-06-03T08:38:27.352+0000 7f514a183640 10 client.4490.objecter op 0 rval 0 len 5 2024-06-03T08:38:27.627 INFO:tasks.workunit.client.0.smithi088.stdout: api_asio: 2024-06-03T08:38:27.352+0000 7f514a183640 15 client.4490.objecter handle_osd_op_reply completed tid 3 2024-06-03T08:38:27.627 INFO:tasks.workunit.client.0.smithi088.stdout: api_asio: 2024-06-03T08:38:27.352+0000 7f514a183640 15 client.4490.objecter _finish_op 3 2024-06-03T08:38:27.627 INFO:tasks.workunit.client.0.smithi088.stdout: api_asio: 2024-06-03T08:38:27.352+0000 7f514a183640 20 client.4490.objecter put_session s=0x5619220df110 osd=5 5 2024-06-03T08:38:27.627 INFO:tasks.workunit.client.0.smithi088.stdout: api_asio: 2024-06-03T08:38:27.352+0000 7f514a183640 15 client.4490.objecter _session_op_remove 5 3 2024-06-03T08:38:27.627 INFO:tasks.workunit.client.0.smithi088.stdout: api_asio: 2024-06-03T08:38:27.352+0000 7f514a183640 5 client.4490.objecter 1 in flight 2024-06-03T08:38:27.627 INFO:tasks.workunit.client.0.smithi088.stdout: api_asio: 2024-06-03T08:38:27.595+0000 7f5132ffd640 10 monclient: tick 2024-06-03T08:38:27.627 INFO:tasks.workunit.client.0.smithi088.stdout: api_asio: 2024-06-03T08:38:27.595+0000 7f5132ffd640 10 monclient: _check_auth_tickets 2024-06-03T08:38:27.627 INFO:tasks.workunit.client.0.smithi088.stdout: api_asio: 2024-06-03T08:38:27.595+0000 7f5132ffd640 20 monclient: _check_auth_rotating not needed by client.admin 2024-06-03T08:38:27.627 INFO:tasks.workunit.client.0.smithi088.stdout: api_asio: 2024-06-03T08:38:27.626+0000 7f514a183640 1 -- 172.21.15.88:0/320712639 <== osd.5 v1:172.21.15.192:6805/34786 4 ==== osd_op_reply(4 noexist [read 0~256] v0'0 uv0 ondisk = -2 ((2) No such file or directory)) v8 ==== 151+0+0 (unknown 3185144069 0 0) 0x7f513c0043a0 con 0x5619220df280 2024-06-03T08:38:27.627 INFO:tasks.workunit.client.0.smithi088.stdout: api_asio: 2024-06-03T08:38:27.626+0000 7f514a183640 10 client.4490.objecter ms_dispatch 0x561921fa3060 osd_op_reply(4 noexist [read 0~256] v0'0 uv0 ondisk = -2 ((2) No such file or directory)) v8 2024-06-03T08:38:27.628 INFO:tasks.workunit.client.0.smithi088.stdout: api_asio: 2024-06-03T08:38:27.626+0000 7f514a183640 10 client.4490.objecter in handle_osd_op_reply 2024-06-03T08:38:27.628 INFO:tasks.workunit.client.0.smithi088.stdout: api_asio: 2024-06-03T08:38:27.626+0000 7f514a183640 7 client.4490.objecter handle_osd_op_reply 4 ondisk uv 0 in 17.1 attempt 0 2024-06-03T08:38:27.628 INFO:tasks.workunit.client.0.smithi088.stdout: api_asio: 2024-06-03T08:38:27.626+0000 7f514a183640 10 client.4490.objecter handle_osd_op_reply copying resulting 0 into existing ceph::buffer of length 0 2024-06-03T08:38:27.628 INFO:tasks.workunit.client.0.smithi088.stdout: api_asio: 2024-06-03T08:38:27.626+0000 7f514a183640 10 client.4490.objecter op 0 rval 0 len 0 2024-06-03T08:38:27.628 INFO:tasks.workunit.client.0.smithi088.stdout: api_asio: 2024-06-03T08:38:27.626+0000 7f514a183640 15 client.4490.objecter handle_osd_op_reply completed tid 4 2024-06-03T08:38:27.628 INFO:tasks.workunit.client.0.smithi088.stdout: api_asio: 2024-06-03T08:38:27.626+0000 7f514a183640 15 client.4490.objecter _finish_op 4 2024-06-03T08:38:27.628 INFO:tasks.workunit.client.0.smithi088.stdout: api_asio: 2024-06-03T08:38:27.626+0000 7f514a183640 20 client.4490.objecter put_session s=0x5619220df110 osd=5 4 2024-06-03T08:38:27.628 INFO:tasks.workunit.client.0.smithi088.stdout: api_asio: 2024-06-03T08:38:27.626+0000 7f514a183640 15 client.4490.objecter _session_op_remove 5 4 2024-06-03T08:38:27.628 INFO:tasks.workunit.client.0.smithi088.stdout: api_asio: 2024-06-03T08:38:27.626+0000 7f514a183640 5 client.4490.objecter 0 in flight 2024-06-03T08:38:27.628 INFO:tasks.workunit.client.0.smithi088.stdout: api_asio: [ OK ] AsioRados.AsyncReadCallback (691 ms) 2024-06-03T08:38:27.628 INFO:tasks.workunit.client.0.smithi088.stdout: api_asio: [ RUN ] AsioRados.AsyncReadFuture 2024-06-03T08:38:27.628 INFO:tasks.workunit.client.0.smithi088.stdout: api_asio: 2024-06-03T08:38:27.626+0000 7f514d647780 10 client.4490.objecter _op_submit op 0x561922174e60 2024-06-03T08:38:27.767 INFO:tasks.workunit.client.0.smithi088.stdout: api_asio: 2024-06-03T08:38:27.626+0000 7f514d647780 20 client.4490.objecter _calc_target epoch 53 base exist @17 precalc_pgid 0 pgid 0.0 is_read 2024-06-03T08:38:27.768 INFO:tasks.workunit.client.0.smithi088.stdout: api_asio: 2024-06-03T08:38:27.626+0000 7f514d647780 20 client.4490.objecter _calc_target target exist @17 -> pgid 17.d3a59201 2024-06-03T08:38:27.768 INFO:tasks.workunit.client.0.smithi088.stdout: api_asio: 2024-06-03T08:38:27.626+0000 7f514d647780 10 client.4490.objecter _calc_target raw pgid 17.d3a59201 -> actual 17.1 acting [5,6] primary 5 2024-06-03T08:38:27.768 INFO:tasks.workunit.client.0.smithi088.stdout: api_asio: 2024-06-03T08:38:27.626+0000 7f514d647780 20 client.4490.objecter _get_session s=0x5619220df110 osd=5 3 2024-06-03T08:38:27.768 INFO:tasks.workunit.client.0.smithi088.stdout: api_asio: 2024-06-03T08:38:27.626+0000 7f514d647780 10 client.4490.objecter _op_submit oid exist '@17' '@17' [read 0~256] tid 5 osd.5 2024-06-03T08:38:27.768 INFO:tasks.workunit.client.0.smithi088.stdout: api_asio: 2024-06-03T08:38:27.626+0000 7f514d647780 20 client.4490.objecter get_session s=0x5619220df110 osd=5 3 2024-06-03T08:38:27.768 INFO:tasks.workunit.client.0.smithi088.stdout: api_asio: 2024-06-03T08:38:27.626+0000 7f514d647780 15 client.4490.objecter _session_op_assign 5 5 2024-06-03T08:38:27.768 INFO:tasks.workunit.client.0.smithi088.stdout: api_asio: 2024-06-03T08:38:27.626+0000 7f514d647780 15 client.4490.objecter _send_op 5 to 17.1 on osd.5 2024-06-03T08:38:27.768 INFO:tasks.workunit.client.0.smithi088.stdout: api_asio: 2024-06-03T08:38:27.626+0000 7f514d647780 1 -- 172.21.15.88:0/320712639 --> v1:172.21.15.192:6805/34786 -- osd_op(unknown.0.0:5 17.1 17:8049a5cb:::exist:head [read 0~256] snapc 0=[] ondisk+read+known_if_redirected+supports_pool_eio e53) v8 -- 0x5619220b1610 con 0x5619220df280 2024-06-03T08:38:27.768 INFO:tasks.workunit.client.0.smithi088.stdout: api_asio: 2024-06-03T08:38:27.626+0000 7f514d647780 20 client.4490.objecter put_session s=0x5619220df110 osd=5 4 2024-06-03T08:38:27.768 INFO:tasks.workunit.client.0.smithi088.stdout: api_asio: 2024-06-03T08:38:27.626+0000 7f514d647780 5 client.4490.objecter 1 in flight 2024-06-03T08:38:27.768 INFO:tasks.workunit.client.0.smithi088.stdout: api_asio: 2024-06-03T08:38:27.626+0000 7f514d647780 10 client.4490.objecter _op_submit op 0x561922179ab0 2024-06-03T08:38:27.769 INFO:tasks.workunit.client.0.smithi088.stdout: api_asio: 2024-06-03T08:38:27.626+0000 7f514d647780 20 client.4490.objecter _calc_target epoch 53 base noexist @17 precalc_pgid 0 pgid 0.0 is_read 2024-06-03T08:38:27.769 INFO:tasks.workunit.client.0.smithi088.stdout: api_asio: 2024-06-03T08:38:27.626+0000 7f514d647780 20 client.4490.objecter _calc_target target noexist @17 -> pgid 17.a7bbaa1 2024-06-03T08:38:27.769 INFO:tasks.workunit.client.0.smithi088.stdout: api_asio: 2024-06-03T08:38:27.626+0000 7f514d647780 10 client.4490.objecter _calc_target raw pgid 17.a7bbaa1 -> actual 17.1 acting [5,6] primary 5 2024-06-03T08:38:27.769 INFO:tasks.workunit.client.0.smithi088.stdout: api_asio: 2024-06-03T08:38:27.626+0000 7f514d647780 20 client.4490.objecter _get_session s=0x5619220df110 osd=5 4 2024-06-03T08:38:27.769 INFO:tasks.workunit.client.0.smithi088.stdout: api_asio: 2024-06-03T08:38:27.626+0000 7f514d647780 10 client.4490.objecter _op_submit oid noexist '@17' '@17' [read 0~256] tid 6 osd.5 2024-06-03T08:38:27.769 INFO:tasks.workunit.client.0.smithi088.stdout: api_asio: 2024-06-03T08:38:27.626+0000 7f514d647780 20 client.4490.objecter get_session s=0x5619220df110 osd=5 4 2024-06-03T08:38:27.769 INFO:tasks.workunit.client.0.smithi088.stdout: api_asio: 2024-06-03T08:38:27.626+0000 7f514d647780 15 client.4490.objecter _session_op_assign 5 6 2024-06-03T08:38:27.769 INFO:tasks.workunit.client.0.smithi088.stdout: api_asio: 2024-06-03T08:38:27.626+0000 7f514d647780 15 client.4490.objecter _send_op 6 to 17.1 on osd.5 2024-06-03T08:38:27.769 INFO:tasks.workunit.client.0.smithi088.stdout: api_asio: 2024-06-03T08:38:27.626+0000 7f514d647780 1 -- 172.21.15.88:0/320712639 --> v1:172.21.15.192:6805/34786 -- osd_op(unknown.0.0:6 17.1 17:855dde50:::noexist:head [read 0~256] snapc 0=[] ondisk+read+known_if_redirected+supports_pool_eio e53) v8 -- 0x561922179fa0 con 0x5619220df280 2024-06-03T08:38:27.769 INFO:tasks.workunit.client.0.smithi088.stdout: api_asio: 2024-06-03T08:38:27.626+0000 7f514d647780 20 client.4490.objecter put_session s=0x5619220df110 osd=5 5 2024-06-03T08:38:27.769 INFO:tasks.workunit.client.0.smithi088.stdout: api_asio: 2024-06-03T08:38:27.626+0000 7f514d647780 5 client.4490.objecter 2 in flight 2024-06-03T08:38:27.769 INFO:tasks.workunit.client.0.smithi088.stdout: api_asio: 2024-06-03T08:38:27.767+0000 7f514a183640 1 -- 172.21.15.88:0/320712639 <== osd.5 v1:172.21.15.192:6805/34786 5 ==== osd_op_reply(5 exist [read 0~5 out=5b] v0'0 uv1 ondisk = 0) v8 ==== 149+0+5 (unknown 704808412 0 3741568377) 0x7f513c0043a0 con 0x5619220df280 2024-06-03T08:38:27.769 INFO:tasks.workunit.client.0.smithi088.stdout: api_asio: 2024-06-03T08:38:27.767+0000 7f514a183640 10 client.4490.objecter ms_dispatch 0x561921fa3060 osd_op_reply(5 exist [read 0~5 out=5b] v0'0 uv1 ondisk = 0) v8 2024-06-03T08:38:27.770 INFO:tasks.workunit.client.0.smithi088.stdout: api_asio: 2024-06-03T08:38:27.767+0000 7f514a183640 10 client.4490.objecter in handle_osd_op_reply 2024-06-03T08:38:27.770 INFO:tasks.workunit.client.0.smithi088.stdout: api_asio: 2024-06-03T08:38:27.767+0000 7f514a183640 7 client.4490.objecter handle_osd_op_reply 5 ondisk uv 1 in 17.1 attempt 0 2024-06-03T08:38:27.770 INFO:tasks.workunit.client.0.smithi088.stdout: api_asio: 2024-06-03T08:38:27.767+0000 7f514a183640 10 client.4490.objecter op 0 rval 0 len 5 2024-06-03T08:38:27.770 INFO:tasks.workunit.client.0.smithi088.stdout: api_asio: 2024-06-03T08:38:27.767+0000 7f514a183640 15 client.4490.objecter handle_osd_op_reply completed tid 5 2024-06-03T08:38:27.770 INFO:tasks.workunit.client.0.smithi088.stdout: api_asio: 2024-06-03T08:38:27.767+0000 7f514a183640 15 client.4490.objecter _finish_op 5 2024-06-03T08:38:27.770 INFO:tasks.workunit.client.0.smithi088.stdout: api_asio: 2024-06-03T08:38:27.767+0000 7f514a183640 20 client.4490.objecter put_session s=0x5619220df110 osd=5 5 2024-06-03T08:38:27.770 INFO:tasks.workunit.client.0.smithi088.stdout: api_asio: 2024-06-03T08:38:27.767+0000 7f514a183640 15 client.4490.objecter _session_op_remove 5 5 2024-06-03T08:38:27.770 INFO:tasks.workunit.client.0.smithi088.stdout: api_asio: 2024-06-03T08:38:27.767+0000 7f514a183640 5 client.4490.objecter 1 in flight 2024-06-03T08:38:27.770 INFO:tasks.workunit.client.0.smithi088.stdout: api_asio: 2024-06-03T08:38:27.767+0000 7f514a183640 1 -- 172.21.15.88:0/320712639 <== osd.5 v1:172.21.15.192:6805/34786 6 ==== osd_op_reply(6 noexist [read 0~256] v0'0 uv0 ondisk = -2 ((2) No such file or directory)) v8 ==== 151+0+0 (unknown 3185144069 0 0) 0x7f513c0043a0 con 0x5619220df280 2024-06-03T08:38:27.770 INFO:tasks.workunit.client.0.smithi088.stdout: api_asio: 2024-06-03T08:38:27.767+0000 7f514a183640 10 client.4490.objecter ms_dispatch 0x561921fa3060 osd_op_reply(6 noexist [read 0~256] v0'0 uv0 ondisk = -2 ((2) No such file or directory)) v8 2024-06-03T08:38:27.770 INFO:tasks.workunit.client.0.smithi088.stdout: api_asio: 2024-06-03T08:38:27.767+0000 7f514a183640 10 client.4490.objecter in handle_osd_op_reply 2024-06-03T08:38:27.770 INFO:tasks.workunit.client.0.smithi088.stdout: api_asio: 2024-06-03T08:38:27.767+0000 7f514a183640 7 client.4490.objecter handle_osd_op_reply 6 ondisk uv 0 in 17.1 attempt 0 2024-06-03T08:38:27.770 INFO:tasks.workunit.client.0.smithi088.stdout: api_asio: 2024-06-03T08:38:27.767+0000 7f514a183640 10 client.4490.objecter handle_osd_op_reply copying resulting 0 into existing ceph::buffer of length 0 2024-06-03T08:38:27.771 INFO:tasks.workunit.client.0.smithi088.stdout: api_asio: 2024-06-03T08:38:27.767+0000 7f514a183640 10 client.4490.objecter op 0 rval 0 len 0 2024-06-03T08:38:27.771 INFO:tasks.workunit.client.0.smithi088.stdout: api_asio: 2024-06-03T08:38:27.767+0000 7f514a183640 15 client.4490.objecter handle_osd_op_reply completed tid 6 2024-06-03T08:38:27.771 INFO:tasks.workunit.client.0.smithi088.stdout: api_asio: 2024-06-03T08:38:27.767+0000 7f514a183640 15 client.4490.objecter _finish_op 6 2024-06-03T08:38:27.771 INFO:tasks.workunit.client.0.smithi088.stdout: api_asio: 2024-06-03T08:38:27.767+0000 7f514a183640 20 client.4490.objecter put_session s=0x5619220df110 osd=5 4 2024-06-03T08:38:27.771 INFO:tasks.workunit.client.0.smithi088.stdout: api_asio: 2024-06-03T08:38:27.767+0000 7f514a183640 15 client.4490.objecter _session_op_remove 5 6 2024-06-03T08:38:27.771 INFO:tasks.workunit.client.0.smithi088.stdout: api_asio: 2024-06-03T08:38:27.767+0000 7f514a183640 5 client.4490.objecter 0 in flight 2024-06-03T08:38:27.771 INFO:tasks.workunit.client.0.smithi088.stdout: api_asio: [ OK ] AsioRados.AsyncReadFuture (141 ms) 2024-06-03T08:38:27.771 INFO:tasks.workunit.client.0.smithi088.stdout: api_asio: [ RUN ] AsioRados.AsyncReadYield 2024-06-03T08:38:27.771 INFO:tasks.workunit.client.0.smithi088.stdout: api_asio: 2024-06-03T08:38:27.767+0000 7f514d647780 10 client.4490.objecter _op_submit op 0x561922179ab0 2024-06-03T08:38:27.771 INFO:tasks.workunit.client.0.smithi088.stdout: api_asio: 2024-06-03T08:38:27.767+0000 7f514d647780 20 client.4490.objecter _calc_target epoch 53 base exist @17 precalc_pgid 0 pgid 0.0 is_read 2024-06-03T08:38:27.771 INFO:tasks.workunit.client.0.smithi088.stdout: api_asio: 2024-06-03T08:38:27.767+0000 7f514d647780 20 client.4490.objecter _calc_target target exist @17 -> pgid 17.d3a59201 2024-06-03T08:38:27.771 INFO:tasks.workunit.client.0.smithi088.stdout: api_asio: 2024-06-03T08:38:27.767+0000 7f514d647780 10 client.4490.objecter _calc_target raw pgid 17.d3a59201 -> actual 17.1 acting [5,6] primary 5 2024-06-03T08:38:27.771 INFO:tasks.workunit.client.0.smithi088.stdout: api_asio: 2024-06-03T08:38:27.767+0000 7f514d647780 20 client.4490.objecter _get_session s=0x5619220df110 osd=5 3 2024-06-03T08:38:27.772 INFO:tasks.workunit.client.0.smithi088.stdout: api_asio: 2024-06-03T08:38:27.767+0000 7f514d647780 10 client.4490.objecter _op_submit oid exist '@17' '@17' [read 0~256] tid 7 osd.5 2024-06-03T08:38:27.772 INFO:tasks.workunit.client.0.smithi088.stdout: api_asio: 2024-06-03T08:38:27.767+0000 7f514d647780 20 client.4490.objecter get_session s=0x5619220df110 osd=5 3 2024-06-03T08:38:27.772 INFO:tasks.workunit.client.0.smithi088.stdout: api_asio: 2024-06-03T08:38:27.767+0000 7f514d647780 15 client.4490.objecter _session_op_assign 5 7 2024-06-03T08:38:27.772 INFO:tasks.workunit.client.0.smithi088.stdout: api_asio: 2024-06-03T08:38:27.767+0000 7f514d647780 15 client.4490.objecter _send_op 7 to 17.1 on osd.5 2024-06-03T08:38:27.772 INFO:tasks.workunit.client.0.smithi088.stdout: api_asio: 2024-06-03T08:38:27.767+0000 7f514d647780 1 -- 172.21.15.88:0/320712639 --> v1:172.21.15.192:6805/34786 -- osd_op(unknown.0.0:7 17.1 17:8049a5cb:::exist:head [read 0~256] snapc 0=[] ondisk+read+known_if_redirected+supports_pool_eio e53) v8 -- 0x561922174e60 con 0x5619220df280 2024-06-03T08:38:27.808 INFO:tasks.workunit.client.0.smithi088.stdout: api_asio: 2024-06-03T08:38:27.767+0000 7f514d647780 20 client.4490.objecter put_session s=0x5619220df110 osd=5 4 2024-06-03T08:38:27.809 INFO:tasks.workunit.client.0.smithi088.stdout: api_asio: 2024-06-03T08:38:27.767+0000 7f514d647780 5 client.4490.objecter 1 in flight 2024-06-03T08:38:27.809 INFO:tasks.workunit.client.0.smithi088.stdout: api_asio: 2024-06-03T08:38:27.767+0000 7f514d647780 10 client.4490.objecter _op_submit op 0x5619221ba9e0 2024-06-03T08:38:27.809 INFO:tasks.workunit.client.0.smithi088.stdout: api_asio: 2024-06-03T08:38:27.767+0000 7f514d647780 20 client.4490.objecter _calc_target epoch 53 base noexist @17 precalc_pgid 0 pgid 0.0 is_read 2024-06-03T08:38:27.809 INFO:tasks.workunit.client.0.smithi088.stdout: api_asio: 2024-06-03T08:38:27.767+0000 7f514d647780 20 client.4490.objecter _calc_target target noexist @17 -> pgid 17.a7bbaa1 2024-06-03T08:38:27.809 INFO:tasks.workunit.client.0.smithi088.stdout: api_asio: 2024-06-03T08:38:27.767+0000 7f514d647780 10 client.4490.objecter _calc_target raw pgid 17.a7bbaa1 -> actual 17.1 acting [5,6] primary 5 2024-06-03T08:38:27.809 INFO:tasks.workunit.client.0.smithi088.stdout: api_asio: 2024-06-03T08:38:27.767+0000 7f514d647780 20 client.4490.objecter _get_session s=0x5619220df110 osd=5 4 2024-06-03T08:38:27.809 INFO:tasks.workunit.client.0.smithi088.stdout: api_asio: 2024-06-03T08:38:27.767+0000 7f514d647780 10 client.4490.objecter _op_submit oid noexist '@17' '@17' [read 0~256] tid 8 osd.5 2024-06-03T08:38:27.809 INFO:tasks.workunit.client.0.smithi088.stdout: api_asio: 2024-06-03T08:38:27.767+0000 7f514d647780 20 client.4490.objecter get_session s=0x5619220df110 osd=5 4 2024-06-03T08:38:27.809 INFO:tasks.workunit.client.0.smithi088.stdout: api_asio: 2024-06-03T08:38:27.767+0000 7f514d647780 15 client.4490.objecter _session_op_assign 5 8 2024-06-03T08:38:27.809 INFO:tasks.workunit.client.0.smithi088.stdout: api_asio: 2024-06-03T08:38:27.767+0000 7f514d647780 15 client.4490.objecter _send_op 8 to 17.1 on osd.5 2024-06-03T08:38:27.809 INFO:tasks.workunit.client.0.smithi088.stdout: api_asio: 2024-06-03T08:38:27.767+0000 7f514d647780 1 -- 172.21.15.88:0/320712639 --> v1:172.21.15.192:6805/34786 -- osd_op(unknown.0.0:8 17.1 17:855dde50:::noexist:head [read 0~256] snapc 0=[] ondisk+read+known_if_redirected+supports_pool_eio e53) v8 -- 0x5619221baed0 con 0x5619220df280 2024-06-03T08:38:27.809 INFO:tasks.workunit.client.0.smithi088.stdout: api_asio: 2024-06-03T08:38:27.767+0000 7f514d647780 20 client.4490.objecter put_session s=0x5619220df110 osd=5 5 2024-06-03T08:38:27.809 INFO:tasks.workunit.client.0.smithi088.stdout: api_asio: 2024-06-03T08:38:27.767+0000 7f514d647780 5 client.4490.objecter 2 in flight 2024-06-03T08:38:27.810 INFO:tasks.workunit.client.0.smithi088.stdout: api_asio: 2024-06-03T08:38:27.808+0000 7f514a183640 1 -- 172.21.15.88:0/320712639 <== osd.5 v1:172.21.15.192:6805/34786 7 ==== osd_op_reply(7 exist [read 0~5 out=5b] v0'0 uv1 ondisk = 0) v8 ==== 149+0+5 (unknown 704808412 0 3741568377) 0x7f513c0043a0 con 0x5619220df280 2024-06-03T08:38:27.810 INFO:tasks.workunit.client.0.smithi088.stdout: api_asio: 2024-06-03T08:38:27.808+0000 7f514a183640 10 client.4490.objecter ms_dispatch 0x561921fa3060 osd_op_reply(7 exist [read 0~5 out=5b] v0'0 uv1 ondisk = 0) v8 2024-06-03T08:38:27.810 INFO:tasks.workunit.client.0.smithi088.stdout: api_asio: 2024-06-03T08:38:27.808+0000 7f514a183640 10 client.4490.objecter in handle_osd_op_reply 2024-06-03T08:38:27.810 INFO:tasks.workunit.client.0.smithi088.stdout: api_asio: 2024-06-03T08:38:27.808+0000 7f514a183640 7 client.4490.objecter handle_osd_op_reply 7 ondisk uv 1 in 17.1 attempt 0 2024-06-03T08:38:27.810 INFO:tasks.workunit.client.0.smithi088.stdout: api_asio: 2024-06-03T08:38:27.808+0000 7f514a183640 10 client.4490.objecter op 0 rval 0 len 5 2024-06-03T08:38:27.810 INFO:tasks.workunit.client.0.smithi088.stdout: api_asio: 2024-06-03T08:38:27.808+0000 7f514a183640 15 client.4490.objecter handle_osd_op_reply completed tid 7 2024-06-03T08:38:27.810 INFO:tasks.workunit.client.0.smithi088.stdout: api_asio: 2024-06-03T08:38:27.808+0000 7f514a183640 15 client.4490.objecter _finish_op 7 2024-06-03T08:38:27.810 INFO:tasks.workunit.client.0.smithi088.stdout: api_asio: 2024-06-03T08:38:27.808+0000 7f514a183640 20 client.4490.objecter put_session s=0x5619220df110 osd=5 5 2024-06-03T08:38:27.810 INFO:tasks.workunit.client.0.smithi088.stdout: api_asio: 2024-06-03T08:38:27.808+0000 7f514a183640 15 client.4490.objecter _session_op_remove 5 7 2024-06-03T08:38:27.810 INFO:tasks.workunit.client.0.smithi088.stdout: api_asio: 2024-06-03T08:38:27.808+0000 7f514a183640 5 client.4490.objecter 1 in flight 2024-06-03T08:38:27.810 INFO:tasks.workunit.client.0.smithi088.stdout: api_asio: 2024-06-03T08:38:27.808+0000 7f514a183640 1 -- 172.21.15.88:0/320712639 <== osd.5 v1:172.21.15.192:6805/34786 8 ==== osd_op_reply(8 noexist [read 0~256] v0'0 uv0 ondisk = -2 ((2) No such file or directory)) v8 ==== 151+0+0 (unknown 3185144069 0 0) 0x7f513c0043a0 con 0x5619220df280 2024-06-03T08:38:27.810 INFO:tasks.workunit.client.0.smithi088.stdout: api_asio: 2024-06-03T08:38:27.808+0000 7f514a183640 10 client.4490.objecter ms_dispatch 0x561921fa3060 osd_op_reply(8 noexist [read 0~256] v0'0 uv0 ondisk = -2 ((2) No such file or directory)) v8 2024-06-03T08:38:27.810 INFO:tasks.workunit.client.0.smithi088.stdout: api_asio: 2024-06-03T08:38:27.808+0000 7f514a183640 10 client.4490.objecter in handle_osd_op_reply 2024-06-03T08:38:27.924 INFO:tasks.workunit.client.0.smithi088.stdout: api_asio: 2024-06-03T08:38:27.808+0000 7f514a183640 7 client.4490.objecter handle_osd_op_reply 8 ondisk uv 0 in 17.1 attempt 0 2024-06-03T08:38:27.924 INFO:tasks.workunit.client.0.smithi088.stdout: api_asio: 2024-06-03T08:38:27.808+0000 7f514a183640 10 client.4490.objecter handle_osd_op_reply copying resulting 0 into existing ceph::buffer of length 0 2024-06-03T08:38:27.924 INFO:tasks.workunit.client.0.smithi088.stdout: api_asio: 2024-06-03T08:38:27.808+0000 7f514a183640 10 client.4490.objecter op 0 rval 0 len 0 2024-06-03T08:38:27.924 INFO:tasks.workunit.client.0.smithi088.stdout: api_asio: 2024-06-03T08:38:27.808+0000 7f514a183640 15 client.4490.objecter handle_osd_op_reply completed tid 8 2024-06-03T08:38:27.924 INFO:tasks.workunit.client.0.smithi088.stdout: api_asio: 2024-06-03T08:38:27.808+0000 7f514a183640 15 client.4490.objecter _finish_op 8 2024-06-03T08:38:27.924 INFO:tasks.workunit.client.0.smithi088.stdout: api_asio: 2024-06-03T08:38:27.808+0000 7f514a183640 20 client.4490.objecter put_session s=0x5619220df110 osd=5 4 2024-06-03T08:38:27.924 INFO:tasks.workunit.client.0.smithi088.stdout: api_asio: 2024-06-03T08:38:27.808+0000 7f514a183640 15 client.4490.objecter _session_op_remove 5 8 2024-06-03T08:38:27.925 INFO:tasks.workunit.client.0.smithi088.stdout: api_asio: 2024-06-03T08:38:27.808+0000 7f514a183640 5 client.4490.objecter 0 in flight 2024-06-03T08:38:27.925 INFO:tasks.workunit.client.0.smithi088.stdout: api_asio: [ OK ] AsioRados.AsyncReadYield (41 ms) 2024-06-03T08:38:27.925 INFO:tasks.workunit.client.0.smithi088.stdout: api_asio: [ RUN ] AsioRados.AsyncWriteCallback 2024-06-03T08:38:27.925 INFO:tasks.workunit.client.0.smithi088.stdout: api_asio: 2024-06-03T08:38:27.808+0000 7f514d647780 20 librados: aio_write exist 0~5 snapc=0=[] snap_seq=head 2024-06-03T08:38:27.925 INFO:tasks.workunit.client.0.smithi088.stdout: api_asio: 2024-06-03T08:38:27.808+0000 7f514d647780 20 librados: queue_aio_write 0x561921f4e5e0 completion 0x561922179ab0 write_seq 1 2024-06-03T08:38:27.925 INFO:tasks.workunit.client.0.smithi088.stdout: api_asio: 2024-06-03T08:38:27.808+0000 7f514d647780 10 client.4490.objecter _op_submit op 0x56192217b360 2024-06-03T08:38:27.925 INFO:tasks.workunit.client.0.smithi088.stdout: api_asio: 2024-06-03T08:38:27.808+0000 7f514d647780 20 client.4490.objecter _calc_target epoch 53 base exist @17 precalc_pgid 0 pgid 0.0 is_write 2024-06-03T08:38:27.925 INFO:tasks.workunit.client.0.smithi088.stdout: api_asio: 2024-06-03T08:38:27.808+0000 7f514d647780 20 client.4490.objecter _calc_target target exist @17 -> pgid 17.d3a59201 2024-06-03T08:38:27.925 INFO:tasks.workunit.client.0.smithi088.stdout: api_asio: 2024-06-03T08:38:27.808+0000 7f514d647780 10 client.4490.objecter _calc_target raw pgid 17.d3a59201 -> actual 17.1 acting [5,6] primary 5 2024-06-03T08:38:27.925 INFO:tasks.workunit.client.0.smithi088.stdout: api_asio: 2024-06-03T08:38:27.808+0000 7f514d647780 20 client.4490.objecter _get_session s=0x5619220df110 osd=5 3 2024-06-03T08:38:27.925 INFO:tasks.workunit.client.0.smithi088.stdout: api_asio: 2024-06-03T08:38:27.808+0000 7f514d647780 10 client.4490.objecter _op_submit oid exist '@17' '@17' [write 0~5 in=5b] tid 9 osd.5 2024-06-03T08:38:27.925 INFO:tasks.workunit.client.0.smithi088.stdout: api_asio: 2024-06-03T08:38:27.808+0000 7f514d647780 20 client.4490.objecter get_session s=0x5619220df110 osd=5 3 2024-06-03T08:38:27.926 INFO:tasks.workunit.client.0.smithi088.stdout: api_asio: 2024-06-03T08:38:27.808+0000 7f514d647780 15 client.4490.objecter _session_op_assign 5 9 2024-06-03T08:38:27.926 INFO:tasks.workunit.client.0.smithi088.stdout: api_asio: 2024-06-03T08:38:27.808+0000 7f514d647780 15 client.4490.objecter _send_op 9 to 17.1 on osd.5 2024-06-03T08:38:27.926 INFO:tasks.workunit.client.0.smithi088.stdout: api_asio: 2024-06-03T08:38:27.808+0000 7f514d647780 1 -- 172.21.15.88:0/320712639 --> v1:172.21.15.192:6805/34786 -- osd_op(unknown.0.0:9 17.1 17:8049a5cb:::exist:head [write 0~5 in=5b] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e53) v8 -- 0x561922179bc0 con 0x5619220df280 2024-06-03T08:38:27.926 INFO:tasks.workunit.client.0.smithi088.stdout: api_asio: 2024-06-03T08:38:27.808+0000 7f514d647780 20 client.4490.objecter put_session s=0x5619220df110 osd=5 4 2024-06-03T08:38:27.926 INFO:tasks.workunit.client.0.smithi088.stdout: api_asio: 2024-06-03T08:38:27.808+0000 7f514d647780 5 client.4490.objecter 1 in flight 2024-06-03T08:38:27.926 INFO:tasks.workunit.client.0.smithi088.stdout: api_asio: 2024-06-03T08:38:27.808+0000 7f514d647780 20 librados: aio_write exist 0~5 snapc=0=[] snap_seq=1 2024-06-03T08:38:27.926 INFO:tasks.workunit.client.0.smithi088.stdout: api_asio: 2024-06-03T08:38:27.923+0000 7f514a183640 1 -- 172.21.15.88:0/320712639 <== osd.5 v1:172.21.15.192:6805/34786 9 ==== osd_op_reply(9 exist [write 0~5] v53'2 uv2 ondisk = 0) v8 ==== 149+0+0 (unknown 3434986914 0 0) 0x7f513c0043a0 con 0x5619220df280 2024-06-03T08:38:27.926 INFO:tasks.workunit.client.0.smithi088.stdout: api_asio: 2024-06-03T08:38:27.923+0000 7f514a183640 10 client.4490.objecter ms_dispatch 0x561921fa3060 osd_op_reply(9 exist [write 0~5] v53'2 uv2 ondisk = 0) v8 2024-06-03T08:38:27.926 INFO:tasks.workunit.client.0.smithi088.stdout: api_asio: 2024-06-03T08:38:27.923+0000 7f514a183640 10 client.4490.objecter in handle_osd_op_reply 2024-06-03T08:38:27.926 INFO:tasks.workunit.client.0.smithi088.stdout: api_asio: 2024-06-03T08:38:27.923+0000 7f514a183640 7 client.4490.objecter handle_osd_op_reply 9 ondisk uv 2 in 17.1 attempt 0 2024-06-03T08:38:27.943 INFO:tasks.workunit.client.0.smithi088.stdout: api_asio: 2024-06-03T08:38:27.923+0000 7f514a183640 10 client.4490.objecter op 0 rval 0 len 0 2024-06-03T08:38:27.943 INFO:tasks.workunit.client.0.smithi088.stdout: api_asio: 2024-06-03T08:38:27.923+0000 7f514a183640 15 client.4490.objecter handle_osd_op_reply completed tid 9 2024-06-03T08:38:27.943 INFO:tasks.workunit.client.0.smithi088.stdout: api_asio: 2024-06-03T08:38:27.923+0000 7f514a183640 15 client.4490.objecter _finish_op 9 2024-06-03T08:38:27.943 INFO:tasks.workunit.client.0.smithi088.stdout: api_asio: 2024-06-03T08:38:27.923+0000 7f514a183640 20 client.4490.objecter put_session s=0x5619220df110 osd=5 4 2024-06-03T08:38:27.943 INFO:tasks.workunit.client.0.smithi088.stdout: api_asio: 2024-06-03T08:38:27.923+0000 7f514a183640 15 client.4490.objecter _session_op_remove 5 9 2024-06-03T08:38:27.943 INFO:tasks.workunit.client.0.smithi088.stdout: api_asio: 2024-06-03T08:38:27.923+0000 7f514a183640 5 client.4490.objecter 0 in flight 2024-06-03T08:38:27.943 INFO:tasks.workunit.client.0.smithi088.stdout: api_asio: 2024-06-03T08:38:27.923+0000 7f514a183640 20 librados: complete_aio_write 0x561922179ab0 2024-06-03T08:38:27.943 INFO:tasks.workunit.client.0.smithi088.stdout: api_asio: [ OK ] AsioRados.AsyncWriteCallback (116 ms) 2024-06-03T08:38:27.943 INFO:tasks.workunit.client.0.smithi088.stdout: api_asio: [ RUN ] AsioRados.AsyncWriteFuture 2024-06-03T08:38:27.943 INFO:tasks.workunit.client.0.smithi088.stdout: api_asio: 2024-06-03T08:38:27.924+0000 7f514d647780 20 librados: aio_write exist 0~5 snapc=0=[] snap_seq=head 2024-06-03T08:38:27.943 INFO:tasks.workunit.client.0.smithi088.stdout: api_asio: 2024-06-03T08:38:27.924+0000 7f514d647780 20 librados: queue_aio_write 0x561921f4e5e0 completion 0x56192217b850 write_seq 2 2024-06-03T08:38:27.943 INFO:tasks.workunit.client.0.smithi088.stdout: api_asio: 2024-06-03T08:38:27.924+0000 7f514d647780 10 client.4490.objecter _op_submit op 0x56192217b360 2024-06-03T08:38:27.944 INFO:tasks.workunit.client.0.smithi088.stdout: api_asio: 2024-06-03T08:38:27.924+0000 7f514d647780 20 client.4490.objecter _calc_target epoch 53 base exist @17 precalc_pgid 0 pgid 0.0 is_write 2024-06-03T08:38:27.944 INFO:tasks.workunit.client.0.smithi088.stdout: api_asio: 2024-06-03T08:38:27.924+0000 7f514d647780 20 client.4490.objecter _calc_target target exist @17 -> pgid 17.d3a59201 2024-06-03T08:38:27.944 INFO:tasks.workunit.client.0.smithi088.stdout: api_asio: 2024-06-03T08:38:27.924+0000 7f514d647780 10 client.4490.objecter _calc_target raw pgid 17.d3a59201 -> actual 17.1 acting [5,6] primary 5 2024-06-03T08:38:27.944 INFO:tasks.workunit.client.0.smithi088.stdout: api_asio: 2024-06-03T08:38:27.924+0000 7f514d647780 20 client.4490.objecter _get_session s=0x5619220df110 osd=5 3 2024-06-03T08:38:27.944 INFO:tasks.workunit.client.0.smithi088.stdout: api_asio: 2024-06-03T08:38:27.924+0000 7f514d647780 10 client.4490.objecter _op_submit oid exist '@17' '@17' [write 0~5 in=5b] tid 10 osd.5 2024-06-03T08:38:27.944 INFO:tasks.workunit.client.0.smithi088.stdout: api_asio: 2024-06-03T08:38:27.924+0000 7f514d647780 20 client.4490.objecter get_session s=0x5619220df110 osd=5 3 2024-06-03T08:38:27.944 INFO:tasks.workunit.client.0.smithi088.stdout: api_asio: 2024-06-03T08:38:27.924+0000 7f514d647780 15 client.4490.objecter _session_op_assign 5 10 2024-06-03T08:38:27.944 INFO:tasks.workunit.client.0.smithi088.stdout: api_asio: 2024-06-03T08:38:27.924+0000 7f514d647780 15 client.4490.objecter _send_op 10 to 17.1 on osd.5 2024-06-03T08:38:27.944 INFO:tasks.workunit.client.0.smithi088.stdout: api_asio: 2024-06-03T08:38:27.924+0000 7f514d647780 1 -- 172.21.15.88:0/320712639 --> v1:172.21.15.192:6805/34786 -- osd_op(unknown.0.0:10 17.1 17:8049a5cb:::exist:head [write 0~5 in=5b] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e53) v8 -- 0x561922179bc0 con 0x5619220df280 2024-06-03T08:38:27.944 INFO:tasks.workunit.client.0.smithi088.stdout: api_asio: 2024-06-03T08:38:27.924+0000 7f514d647780 20 client.4490.objecter put_session s=0x5619220df110 osd=5 4 2024-06-03T08:38:27.944 INFO:tasks.workunit.client.0.smithi088.stdout: api_asio: 2024-06-03T08:38:27.924+0000 7f514d647780 5 client.4490.objecter 1 in flight 2024-06-03T08:38:27.944 INFO:tasks.workunit.client.0.smithi088.stdout: api_asio: 2024-06-03T08:38:27.924+0000 7f514d647780 20 librados: aio_write exist 0~5 snapc=0=[] snap_seq=1 2024-06-03T08:38:27.944 INFO:tasks.workunit.client.0.smithi088.stdout: api_asio: 2024-06-03T08:38:27.942+0000 7f5133fff640 1 -- 172.21.15.88:0/320712639 <== mon.2 v1:172.21.15.88:6790/0 18 ==== osd_map(54..54 src has 1..54) v4 ==== 2152+0+0 (unknown 1612027039 0 0) 0x7f51440503f0 con 0x5619220ccfb0 2024-06-03T08:38:27.945 INFO:tasks.workunit.client.0.smithi088.stdout: api_asio: 2024-06-03T08:38:27.942+0000 7f5133fff640 10 client.4490.objecter ms_dispatch 0x561921fa3060 osd_map(54..54 src has 1..54) v4 2024-06-03T08:38:27.945 INFO:tasks.workunit.client.0.smithi088.stdout: api_asio: 2024-06-03T08:38:27.942+0000 7f5133fff640 3 client.4490.objecter handle_osd_map got epochs [54,54] > 53 2024-06-03T08:38:27.945 INFO:tasks.workunit.client.0.smithi088.stdout: api_asio: 2024-06-03T08:38:27.942+0000 7f5133fff640 3 client.4490.objecter handle_osd_map decoding incremental epoch 54 2024-06-03T08:38:27.945 INFO:tasks.workunit.client.0.smithi088.stdout: api_asio: 2024-06-03T08:38:27.942+0000 7f5133fff640 10 client.4490.objecter checking op 10 2024-06-03T08:38:27.945 INFO:tasks.workunit.client.0.smithi088.stdout: api_asio: 2024-06-03T08:38:27.942+0000 7f5133fff640 20 client.4490.objecter _calc_target epoch 54 base exist @17 precalc_pgid 0 pgid 0.0 is_write 2024-06-03T08:38:28.057 INFO:tasks.workunit.client.0.smithi088.stdout: api_asio: 2024-06-03T08:38:27.942+0000 7f5133fff640 20 client.4490.objecter _calc_target target exist @17 -> pgid 17.d3a59201 2024-06-03T08:38:28.057 INFO:tasks.workunit.client.0.smithi088.stdout: api_asio: 2024-06-03T08:38:27.942+0000 7f5133fff640 10 client.4490.objecter _maybe_request_map subscribing (onetime) to next osd map 2024-06-03T08:38:28.057 INFO:tasks.workunit.client.0.smithi088.stdout: api_asio: 2024-06-03T08:38:27.942+0000 7f5133fff640 10 monclient: _renew_subs 2024-06-03T08:38:28.058 INFO:tasks.workunit.client.0.smithi088.stdout: api_asio: 2024-06-03T08:38:27.942+0000 7f5133fff640 10 monclient: _send_mon_message to mon.c at v1:172.21.15.88:6790/0 2024-06-03T08:38:28.058 INFO:tasks.workunit.client.0.smithi088.stdout: api_asio: 2024-06-03T08:38:27.942+0000 7f5133fff640 1 -- 172.21.15.88:0/320712639 --> v1:172.21.15.88:6790/0 -- mon_subscribe({osdmap=55}) v3 -- 0x7f513807f3a0 con 0x5619220ccfb0 2024-06-03T08:38:28.058 INFO:tasks.workunit.client.0.smithi088.stdout: api_asio: 2024-06-03T08:38:27.942+0000 7f5133fff640 20 client.4490.objecter dump_active .. 0 homeless 2024-06-03T08:38:28.058 INFO:tasks.workunit.client.0.smithi088.stdout: api_asio: 2024-06-03T08:38:27.942+0000 7f5133fff640 20 client.4490.objecter 10 17.d3a59201 osd.5 exist [write 0~5 in=5b] 2024-06-03T08:38:28.058 INFO:tasks.workunit.client.0.smithi088.stdout: api_asio: 2024-06-03T08:38:28.057+0000 7f514a183640 1 -- 172.21.15.88:0/320712639 <== osd.5 v1:172.21.15.192:6805/34786 10 ==== osd_op_reply(10 exist [write 0~5] v53'3 uv3 ondisk = 0) v8 ==== 149+0+0 (unknown 3091007094 0 0) 0x7f513c0043a0 con 0x5619220df280 2024-06-03T08:38:28.058 INFO:tasks.workunit.client.0.smithi088.stdout: api_asio: 2024-06-03T08:38:28.057+0000 7f514a183640 10 client.4490.objecter ms_dispatch 0x561921fa3060 osd_op_reply(10 exist [write 0~5] v53'3 uv3 ondisk = 0) v8 2024-06-03T08:38:28.058 INFO:tasks.workunit.client.0.smithi088.stdout: api_asio: 2024-06-03T08:38:28.057+0000 7f514a183640 10 client.4490.objecter in handle_osd_op_reply 2024-06-03T08:38:28.058 INFO:tasks.workunit.client.0.smithi088.stdout: api_asio: 2024-06-03T08:38:28.057+0000 7f514a183640 7 client.4490.objecter handle_osd_op_reply 10 ondisk uv 3 in 17.1 attempt 0 2024-06-03T08:38:28.058 INFO:tasks.workunit.client.0.smithi088.stdout: api_asio: 2024-06-03T08:38:28.057+0000 7f514a183640 10 client.4490.objecter op 0 rval 0 len 0 2024-06-03T08:38:28.058 INFO:tasks.workunit.client.0.smithi088.stdout: api_asio: 2024-06-03T08:38:28.057+0000 7f514a183640 15 client.4490.objecter handle_osd_op_reply completed tid 10 2024-06-03T08:38:28.058 INFO:tasks.workunit.client.0.smithi088.stdout: api_asio: 2024-06-03T08:38:28.057+0000 7f514a183640 15 client.4490.objecter _finish_op 10 2024-06-03T08:38:28.058 INFO:tasks.workunit.client.0.smithi088.stdout: api_asio: 2024-06-03T08:38:28.057+0000 7f514a183640 20 client.4490.objecter put_session s=0x5619220df110 osd=5 4 2024-06-03T08:38:28.058 INFO:tasks.workunit.client.0.smithi088.stdout: api_asio: 2024-06-03T08:38:28.057+0000 7f514a183640 15 client.4490.objecter _session_op_remove 5 10 2024-06-03T08:38:28.059 INFO:tasks.workunit.client.0.smithi088.stdout: api_asio: 2024-06-03T08:38:28.057+0000 7f514a183640 5 client.4490.objecter 0 in flight 2024-06-03T08:38:28.059 INFO:tasks.workunit.client.0.smithi088.stdout: api_asio: 2024-06-03T08:38:28.057+0000 7f514a183640 20 librados: complete_aio_write 0x56192217b850 2024-06-03T08:38:28.059 INFO:tasks.workunit.client.0.smithi088.stdout: api_asio: [ OK ] AsioRados.AsyncWriteFuture (133 ms) 2024-06-03T08:38:28.059 INFO:tasks.workunit.client.0.smithi088.stdout: api_asio: [ RUN ] AsioRados.AsyncWriteYield 2024-06-03T08:38:28.059 INFO:tasks.workunit.client.0.smithi088.stdout: api_asio: 2024-06-03T08:38:28.057+0000 7f514d647780 20 librados: aio_write exist 0~5 snapc=0=[] snap_seq=head 2024-06-03T08:38:28.059 INFO:tasks.workunit.client.0.smithi088.stdout: api_asio: 2024-06-03T08:38:28.057+0000 7f514d647780 20 librados: queue_aio_write 0x561921f4e5e0 completion 0x561922179de0 write_seq 3 2024-06-03T08:38:28.059 INFO:tasks.workunit.client.0.smithi088.stdout: api_asio: 2024-06-03T08:38:28.057+0000 7f514d647780 10 client.4490.objecter _op_submit op 0x56192217b360 2024-06-03T08:38:28.059 INFO:tasks.workunit.client.0.smithi088.stdout: api_asio: 2024-06-03T08:38:28.057+0000 7f514d647780 20 client.4490.objecter _calc_target epoch 54 base exist @17 precalc_pgid 0 pgid 0.0 is_write 2024-06-03T08:38:28.059 INFO:tasks.workunit.client.0.smithi088.stdout: api_asio: 2024-06-03T08:38:28.057+0000 7f514d647780 20 client.4490.objecter _calc_target target exist @17 -> pgid 17.d3a59201 2024-06-03T08:38:28.059 INFO:tasks.workunit.client.0.smithi088.stdout: api_asio: 2024-06-03T08:38:28.057+0000 7f514d647780 10 client.4490.objecter _calc_target raw pgid 17.d3a59201 -> actual 17.1 acting [5,6] primary 5 2024-06-03T08:38:28.059 INFO:tasks.workunit.client.0.smithi088.stdout: api_asio: 2024-06-03T08:38:28.057+0000 7f514d647780 20 client.4490.objecter _get_session s=0x5619220df110 osd=5 3 2024-06-03T08:38:28.059 INFO:tasks.workunit.client.0.smithi088.stdout: api_asio: 2024-06-03T08:38:28.057+0000 7f514d647780 10 client.4490.objecter _op_submit oid exist '@17' '@17' [write 0~5 in=5b] tid 11 osd.5 2024-06-03T08:38:28.059 INFO:tasks.workunit.client.0.smithi088.stdout: api_asio: 2024-06-03T08:38:28.057+0000 7f514d647780 20 client.4490.objecter get_session s=0x5619220df110 osd=5 3 2024-06-03T08:38:28.059 INFO:tasks.workunit.client.0.smithi088.stdout: api_asio: 2024-06-03T08:38:28.057+0000 7f514d647780 15 client.4490.objecter _session_op_assign 5 11 2024-06-03T08:38:28.710 INFO:tasks.ceph.osd.7.smithi192.stderr:2024-06-03T08:38:28.549+0000 12a6c640 -1 osd.7 54 get_health_metrics reporting 1 slow ops, oldest is osd_op(client.4536.0:2 21.1a 21.db6ee63a (undecoded) ondisk+write+known_if_redirected+supports_pool_eio e28) 2024-06-03T08:38:29.053 INFO:tasks.ceph.osd.6.smithi192.stderr:2024-06-03T08:38:29.029+0000 12a6c640 -1 osd.6 54 get_health_metrics reporting 1 slow ops, oldest is osd_op(client.4524.0:2 16.1a 16:5c6776db:::0.obj:head [write 0~248 in=248b] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e28) 2024-06-03T08:38:29.513 INFO:tasks.workunit.client.0.smithi088.stdout: api_asio: 2024-06-03T08:38:28.057+0000 7f514d647780 15 client.4490.objecter _send_op 11 to 17.1 on osd.5 2024-06-03T08:38:29.513 INFO:tasks.workunit.client.0.smithi088.stdout: api_asio: 2024-06-03T08:38:28.057+0000 7f514d647780 1 -- 172.21.15.88:0/320712639 --> v1:172.21.15.192:6805/34786 -- osd_op(unknown.0.0:11 17.1 17:8049a5cb:::exist:head [write 0~5 in=5b] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e54) v8 -- 0x56192219e100 con 0x5619220df280 2024-06-03T08:38:29.513 INFO:tasks.workunit.client.0.smithi088.stdout: api_asio: 2024-06-03T08:38:28.057+0000 7f514d647780 20 client.4490.objecter put_session s=0x5619220df110 osd=5 4 2024-06-03T08:38:29.513 INFO:tasks.workunit.client.0.smithi088.stdout: api_asio: 2024-06-03T08:38:28.057+0000 7f514d647780 5 client.4490.objecter 1 in flight 2024-06-03T08:38:29.513 INFO:tasks.workunit.client.0.smithi088.stdout: api_asio: 2024-06-03T08:38:28.057+0000 7f514d647780 20 librados: aio_write exist 0~5 snapc=0=[] snap_seq=1 2024-06-03T08:38:29.513 INFO:tasks.workunit.client.0.smithi088.stdout: api_asio: 2024-06-03T08:38:28.595+0000 7f5132ffd640 10 monclient: tick 2024-06-03T08:38:29.513 INFO:tasks.workunit.client.0.smithi088.stdout: api_asio: 2024-06-03T08:38:28.595+0000 7f5132ffd640 10 monclient: _check_auth_tickets 2024-06-03T08:38:29.513 INFO:tasks.workunit.client.0.smithi088.stdout: api_asio: 2024-06-03T08:38:28.595+0000 7f5132ffd640 20 monclient: _check_auth_rotating not needed by client.admin 2024-06-03T08:38:29.513 INFO:tasks.workunit.client.0.smithi088.stdout: api_asio: 2024-06-03T08:38:28.773+0000 7f514cbe8640 10 client.4490.objecter tick 2024-06-03T08:38:29.513 INFO:tasks.workunit.client.0.smithi088.stdout: api_asio: 2024-06-03T08:38:29.200+0000 7f5133fff640 1 -- 172.21.15.88:0/320712639 <== mon.2 v1:172.21.15.88:6790/0 19 ==== osd_map(55..55 src has 1..55) v4 ==== 1055+0+0 (unknown 3112924137 0 0) 0x7f51440503f0 con 0x5619220ccfb0 2024-06-03T08:38:29.514 INFO:tasks.workunit.client.0.smithi088.stdout: api_asio: 2024-06-03T08:38:29.200+0000 7f5133fff640 10 client.4490.objecter ms_dispatch 0x561921fa3060 osd_map(55..55 src has 1..55) v4 2024-06-03T08:38:29.514 INFO:tasks.workunit.client.0.smithi088.stdout: api_asio: 2024-06-03T08:38:29.200+0000 7f5133fff640 3 client.4490.objecter handle_osd_map got epochs [55,55] > 54 2024-06-03T08:38:29.514 INFO:tasks.workunit.client.0.smithi088.stdout: api_asio: 2024-06-03T08:38:29.200+0000 7f5133fff640 3 client.4490.objecter handle_osd_map decoding incremental epoch 55 2024-06-03T08:38:29.514 INFO:tasks.workunit.client.0.smithi088.stdout: api_asio: 2024-06-03T08:38:29.200+0000 7f5133fff640 10 client.4490.objecter checking op 11 2024-06-03T08:38:29.514 INFO:tasks.workunit.client.0.smithi088.stdout: api_asio: 2024-06-03T08:38:29.200+0000 7f5133fff640 20 client.4490.objecter _calc_target epoch 55 base exist @17 precalc_pgid 0 pgid 0.0 is_write 2024-06-03T08:38:29.514 INFO:tasks.workunit.client.0.smithi088.stdout: api_asio: 2024-06-03T08:38:29.200+0000 7f5133fff640 20 client.4490.objecter _calc_target target exist @17 -> pgid 17.d3a59201 2024-06-03T08:38:29.514 INFO:tasks.workunit.client.0.smithi088.stdout: api_asio: 2024-06-03T08:38:29.200+0000 7f5133fff640 10 client.4490.objecter _maybe_request_map subscribing (onetime) to next osd map 2024-06-03T08:38:29.514 INFO:tasks.workunit.client.0.smithi088.stdout: api_asio: 2024-06-03T08:38:29.200+0000 7f5133fff640 10 monclient: _renew_subs 2024-06-03T08:38:29.514 INFO:tasks.workunit.client.0.smithi088.stdout: api_asio: 2024-06-03T08:38:29.200+0000 7f5133fff640 10 monclient: _send_mon_message to mon.c at v1:172.21.15.88:6790/0 2024-06-03T08:38:29.514 INFO:tasks.workunit.client.0.smithi088.stdout: api_asio: 2024-06-03T08:38:29.200+0000 7f5133fff640 1 -- 172.21.15.88:0/320712639 --> v1:172.21.15.88:6790/0 -- mon_subscribe({osdmap=56}) v3 -- 0x7f513807f3a0 con 0x5619220ccfb0 2024-06-03T08:38:29.514 INFO:tasks.workunit.client.0.smithi088.stdout: api_asio: 2024-06-03T08:38:29.200+0000 7f5133fff640 20 client.4490.objecter dump_active .. 0 homeless 2024-06-03T08:38:29.514 INFO:tasks.workunit.client.0.smithi088.stdout: api_asio: 2024-06-03T08:38:29.200+0000 7f5133fff640 20 client.4490.objecter 11 17.d3a59201 osd.5 exist [write 0~5 in=5b] 2024-06-03T08:38:29.514 INFO:tasks.workunit.client.0.smithi088.stdout: api_asio: 2024-06-03T08:38:29.512+0000 7f514a183640 1 -- 172.21.15.88:0/320712639 <== osd.5 v1:172.21.15.192:6805/34786 11 ==== osd_op_reply(11 exist [write 0~5] v54'4 uv4 ondisk = 0) v8 ==== 149+0+0 (unknown 3104122088 0 0) 0x7f513c0043a0 con 0x5619220df280 2024-06-03T08:38:29.515 INFO:tasks.workunit.client.0.smithi088.stdout: api_asio: 2024-06-03T08:38:29.512+0000 7f514a183640 10 client.4490.objecter ms_dispatch 0x561921fa3060 osd_op_reply(11 exist [write 0~5] v54'4 uv4 ondisk = 0) v8 2024-06-03T08:38:29.515 INFO:tasks.workunit.client.0.smithi088.stdout: api_asio: 2024-06-03T08:38:29.512+0000 7f514a183640 10 client.4490.objecter in handle_osd_op_reply 2024-06-03T08:38:29.515 INFO:tasks.workunit.client.0.smithi088.stdout: api_asio: 2024-06-03T08:38:29.512+0000 7f514a183640 7 client.4490.objecter handle_osd_op_reply 11 ondisk uv 4 in 17.1 attempt 0 2024-06-03T08:38:29.515 INFO:tasks.workunit.client.0.smithi088.stdout: api_asio: 2024-06-03T08:38:29.512+0000 7f514a183640 10 client.4490.objecter op 0 rval 0 len 0 2024-06-03T08:38:29.515 INFO:tasks.workunit.client.0.smithi088.stdout: api_asio: 2024-06-03T08:38:29.512+0000 7f514a183640 15 client.4490.objecter handle_osd_op_reply completed tid 11 2024-06-03T08:38:29.515 INFO:tasks.workunit.client.0.smithi088.stdout: api_asio: 2024-06-03T08:38:29.512+0000 7f514a183640 15 client.4490.objecter _finish_op 11 2024-06-03T08:38:29.515 INFO:tasks.workunit.client.0.smithi088.stdout: api_asio: 2024-06-03T08:38:29.512+0000 7f514a183640 20 client.4490.objecter put_session s=0x5619220df110 osd=5 4 2024-06-03T08:38:29.515 INFO:tasks.workunit.client.0.smithi088.stdout: api_asio: 2024-06-03T08:38:29.512+0000 7f514a183640 15 client.4490.objecter _session_op_remove 5 11 2024-06-03T08:38:29.515 INFO:tasks.workunit.client.0.smithi088.stdout: api_asio: 2024-06-03T08:38:29.512+0000 7f514a183640 5 client.4490.objecter 0 in flight 2024-06-03T08:38:29.516 INFO:tasks.workunit.client.0.smithi088.stdout: api_asio: 2024-06-03T08:38:29.512+0000 7f514a183640 20 librados: complete_aio_write 0x561922179de0 2024-06-03T08:38:29.516 INFO:tasks.workunit.client.0.smithi088.stdout: api_asio: [ OK ] AsioRados.AsyncWriteYield (1456 ms) 2024-06-03T08:38:29.516 INFO:tasks.workunit.client.0.smithi088.stdout: api_asio: [ RUN ] AsioRados.AsyncReadOperationCallback 2024-06-03T08:38:29.516 INFO:tasks.workunit.client.0.smithi088.stdout: api_asio: 2024-06-03T08:38:29.513+0000 7f514d647780 10 client.4490.objecter _op_submit op 0x56192217a5a0 2024-06-03T08:38:29.516 INFO:tasks.workunit.client.0.smithi088.stdout: api_asio: 2024-06-03T08:38:29.513+0000 7f514d647780 20 client.4490.objecter _calc_target epoch 55 base exist @17 precalc_pgid 0 pgid 0.0 is_read 2024-06-03T08:38:29.516 INFO:tasks.workunit.client.0.smithi088.stdout: api_asio: 2024-06-03T08:38:29.513+0000 7f514d647780 20 client.4490.objecter _calc_target target exist @17 -> pgid 17.d3a59201 2024-06-03T08:38:29.516 INFO:tasks.workunit.client.0.smithi088.stdout: api_asio: 2024-06-03T08:38:29.513+0000 7f514d647780 10 client.4490.objecter _calc_target raw pgid 17.d3a59201 -> actual 17.1 acting [5,6] primary 5 2024-06-03T08:38:29.516 INFO:tasks.workunit.client.0.smithi088.stdout: api_asio: 2024-06-03T08:38:29.513+0000 7f514d647780 20 client.4490.objecter _get_session s=0x5619220df110 osd=5 3 2024-06-03T08:38:29.516 INFO:tasks.workunit.client.0.smithi088.stdout: api_asio: 2024-06-03T08:38:29.513+0000 7f514d647780 10 client.4490.objecter _op_submit oid exist '@17' '@17' [read 0~0] tid 12 osd.5 2024-06-03T08:38:29.516 INFO:tasks.workunit.client.0.smithi088.stdout: api_asio: 2024-06-03T08:38:29.513+0000 7f514d647780 20 client.4490.objecter get_session s=0x5619220df110 osd=5 3 2024-06-03T08:38:29.516 INFO:tasks.workunit.client.0.smithi088.stdout: api_asio: 2024-06-03T08:38:29.513+0000 7f514d647780 15 client.4490.objecter _session_op_assign 5 12 2024-06-03T08:38:29.516 INFO:tasks.workunit.client.0.smithi088.stdout: api_asio: 2024-06-03T08:38:29.513+0000 7f514d647780 15 client.4490.objecter _send_op 12 to 17.1 on osd.5 2024-06-03T08:38:29.516 INFO:tasks.workunit.client.0.smithi088.stdout: api_asio: 2024-06-03T08:38:29.513+0000 7f514d647780 1 -- 172.21.15.88:0/320712639 --> v1:172.21.15.192:6805/34786 -- osd_op(unknown.0.0:12 17.1 17:8049a5cb:::exist:head [read 0~0] snapc 0=[] ondisk+read+known_if_redirected+supports_pool_eio e55) v8 -- 0x56192217aa90 con 0x5619220df280 2024-06-03T08:38:29.516 INFO:tasks.workunit.client.0.smithi088.stdout: api_asio: 2024-06-03T08:38:29.513+0000 7f514d647780 20 client.4490.objecter put_session s=0x5619220df110 osd=5 4 2024-06-03T08:38:29.517 INFO:tasks.workunit.client.0.smithi088.stdout: api_asio: 2024-06-03T08:38:29.513+0000 7f514d647780 5 client.4490.objecter 1 in flight 2024-06-03T08:38:29.517 INFO:tasks.workunit.client.0.smithi088.stdout: api_asio: 2024-06-03T08:38:29.513+0000 7f514d647780 10 client.4490.objecter _op_submit op 0x56192217af50 2024-06-03T08:38:29.517 INFO:tasks.workunit.client.0.smithi088.stdout: api_asio: 2024-06-03T08:38:29.513+0000 7f514d647780 20 client.4490.objecter _calc_target epoch 55 base noexist @17 precalc_pgid 0 pgid 0.0 is_read 2024-06-03T08:38:29.517 INFO:tasks.workunit.client.0.smithi088.stdout: api_asio: 2024-06-03T08:38:29.513+0000 7f514d647780 20 client.4490.objecter _calc_target target noexist @17 -> pgid 17.a7bbaa1 2024-06-03T08:38:29.517 INFO:tasks.workunit.client.0.smithi088.stdout: api_asio: 2024-06-03T08:38:29.513+0000 7f514d647780 10 client.4490.objecter _calc_target raw pgid 17.a7bbaa1 -> actual 17.1 acting [5,6] primary 5 2024-06-03T08:38:29.517 INFO:tasks.workunit.client.0.smithi088.stdout: api_asio: 2024-06-03T08:38:29.513+0000 7f514d647780 20 client.4490.objecter _get_session s=0x5619220df110 osd=5 4 2024-06-03T08:38:29.517 INFO:tasks.workunit.client.0.smithi088.stdout: api_asio: 2024-06-03T08:38:29.513+0000 7f514d647780 10 client.4490.objecter _op_submit oid noexist '@17' '@17' [read 0~0] tid 13 osd.5 2024-06-03T08:38:29.517 INFO:tasks.workunit.client.0.smithi088.stdout: api_asio: 2024-06-03T08:38:29.513+0000 7f514d647780 20 client.4490.objecter get_session s=0x5619220df110 osd=5 4 2024-06-03T08:38:29.517 INFO:tasks.workunit.client.0.smithi088.stdout: api_asio: 2024-06-03T08:38:29.513+0000 7f514d647780 15 client.4490.objecter _session_op_assign 5 13 2024-06-03T08:38:29.517 INFO:tasks.workunit.client.0.smithi088.stdout: api_asio: 2024-06-03T08:38:29.513+0000 7f514d647780 15 client.4490.objecter _send_op 13 to 17.1 on osd.5 2024-06-03T08:38:29.517 INFO:tasks.workunit.client.0.smithi088.stdout: api_asio: 2024-06-03T08:38:29.513+0000 7f514d647780 1 -- 172.21.15.88:0/320712639 --> v1:172.21.15.192:6805/34786 -- osd_op(unknown.0.0:13 17.1 17:855dde50:::noexist:head [read 0~0] snapc 0=[] ondisk+read+known_if_redirected+supports_pool_eio e55) v8 -- 0x56192217b440 con 0x5619220df280 2024-06-03T08:38:29.517 INFO:tasks.workunit.client.0.smithi088.stdout: api_asio: 2024-06-03T08:38:29.513+0000 7f514d647780 20 client.4490.objecter put_session s=0x5619220df110 osd=5 5 2024-06-03T08:38:29.978 INFO:tasks.ceph.osd.7.smithi192.stderr:2024-06-03T08:38:29.973+0000 12a6c640 -1 osd.7 55 get_health_metrics reporting 2 slow ops, oldest is osd_op(client.4536.0:2 21.1a 21.db6ee63a (undecoded) ondisk+write+known_if_redirected+supports_pool_eio e28) 2024-06-03T08:38:30.159 INFO:tasks.workunit.client.0.smithi088.stdout: api_asio: 2024-06-03T08:38:29.513+0000 7f514d647780 5 client.4490.objecter 2 in flight 2024-06-03T08:38:30.160 INFO:tasks.workunit.client.0.smithi088.stdout: api_asio: 2024-06-03T08:38:29.595+0000 7f5132ffd640 10 monclient: tick 2024-06-03T08:38:30.160 INFO:tasks.workunit.client.0.smithi088.stdout: api_asio: 2024-06-03T08:38:29.595+0000 7f5132ffd640 10 monclient: _check_auth_tickets 2024-06-03T08:38:30.160 INFO:tasks.workunit.client.0.smithi088.stdout: api_asio: 2024-06-03T08:38:29.595+0000 7f5132ffd640 20 monclient: _check_auth_rotating not needed by client.admin 2024-06-03T08:38:30.160 INFO:tasks.workunit.client.0.smithi088.stdout: api_asio: 2024-06-03T08:38:30.047+0000 7f5133fff640 1 -- 172.21.15.88:0/320712639 <== mon.2 v1:172.21.15.88:6790/0 20 ==== osd_map(56..56 src has 1..56) v4 ==== 1065+0+0 (unknown 1644892552 0 0) 0x7f51440503f0 con 0x5619220ccfb0 2024-06-03T08:38:30.160 INFO:tasks.workunit.client.0.smithi088.stdout: api_asio: 2024-06-03T08:38:30.047+0000 7f5133fff640 10 client.4490.objecter ms_dispatch 0x561921fa3060 osd_map(56..56 src has 1..56) v4 2024-06-03T08:38:30.160 INFO:tasks.workunit.client.0.smithi088.stdout: api_asio: 2024-06-03T08:38:30.047+0000 7f5133fff640 3 client.4490.objecter handle_osd_map got epochs [56,56] > 55 2024-06-03T08:38:30.160 INFO:tasks.workunit.client.0.smithi088.stdout: api_asio: 2024-06-03T08:38:30.047+0000 7f5133fff640 3 client.4490.objecter handle_osd_map decoding incremental epoch 56 2024-06-03T08:38:30.160 INFO:tasks.workunit.client.0.smithi088.stdout: api_asio: 2024-06-03T08:38:30.047+0000 7f5133fff640 10 client.4490.objecter checking op 12 2024-06-03T08:38:30.160 INFO:tasks.workunit.client.0.smithi088.stdout: api_asio: 2024-06-03T08:38:30.047+0000 7f5133fff640 20 client.4490.objecter _calc_target epoch 56 base exist @17 precalc_pgid 0 pgid 0.0 is_read 2024-06-03T08:38:30.160 INFO:tasks.workunit.client.0.smithi088.stdout: api_asio: 2024-06-03T08:38:30.047+0000 7f5133fff640 20 client.4490.objecter _calc_target target exist @17 -> pgid 17.d3a59201 2024-06-03T08:38:30.160 INFO:tasks.workunit.client.0.smithi088.stdout: api_asio: 2024-06-03T08:38:30.047+0000 7f5133fff640 10 client.4490.objecter checking op 13 2024-06-03T08:38:30.160 INFO:tasks.workunit.client.0.smithi088.stdout: api_asio: 2024-06-03T08:38:30.047+0000 7f5133fff640 20 client.4490.objecter _calc_target epoch 56 base noexist @17 precalc_pgid 0 pgid 0.0 is_read 2024-06-03T08:38:30.161 INFO:tasks.workunit.client.0.smithi088.stdout: api_asio: 2024-06-03T08:38:30.047+0000 7f5133fff640 20 client.4490.objecter _calc_target target noexist @17 -> pgid 17.a7bbaa1 2024-06-03T08:38:30.161 INFO:tasks.workunit.client.0.smithi088.stdout: api_asio: 2024-06-03T08:38:30.047+0000 7f5133fff640 10 client.4490.objecter _maybe_request_map subscribing (onetime) to next osd map 2024-06-03T08:38:30.161 INFO:tasks.workunit.client.0.smithi088.stdout: api_asio: 2024-06-03T08:38:30.047+0000 7f5133fff640 10 monclient: _renew_subs 2024-06-03T08:38:30.161 INFO:tasks.workunit.client.0.smithi088.stdout: api_asio: 2024-06-03T08:38:30.047+0000 7f5133fff640 10 monclient: _send_mon_message to mon.c at v1:172.21.15.88:6790/0 2024-06-03T08:38:30.161 INFO:tasks.workunit.client.0.smithi088.stdout: api_asio: 2024-06-03T08:38:30.047+0000 7f5133fff640 1 -- 172.21.15.88:0/320712639 --> v1:172.21.15.88:6790/0 -- mon_subscribe({osdmap=57}) v3 -- 0x7f513807f3a0 con 0x5619220ccfb0 2024-06-03T08:38:30.161 INFO:tasks.workunit.client.0.smithi088.stdout: api_asio: 2024-06-03T08:38:30.047+0000 7f5133fff640 20 client.4490.objecter dump_active .. 0 homeless 2024-06-03T08:38:30.161 INFO:tasks.workunit.client.0.smithi088.stdout: api_asio: 2024-06-03T08:38:30.047+0000 7f5133fff640 20 client.4490.objecter 12 17.d3a59201 osd.5 exist [read 0~0] 2024-06-03T08:38:30.161 INFO:tasks.workunit.client.0.smithi088.stdout: api_asio: 2024-06-03T08:38:30.047+0000 7f5133fff640 20 client.4490.objecter 13 17.a7bbaa1 osd.5 noexist [read 0~0] 2024-06-03T08:38:30.161 INFO:tasks.workunit.client.0.smithi088.stdout: api_asio: 2024-06-03T08:38:30.159+0000 7f514a183640 1 -- 172.21.15.88:0/320712639 <== osd.5 v1:172.21.15.192:6805/34786 12 ==== osd_op_reply(12 exist [read 0~5 out=5b] v0'0 uv4 ondisk = 0) v8 ==== 149+0+5 (unknown 1370764936 0 3741568377) 0x7f513c0043a0 con 0x5619220df280 2024-06-03T08:38:30.161 INFO:tasks.workunit.client.0.smithi088.stdout: api_asio: 2024-06-03T08:38:30.159+0000 7f514a183640 10 client.4490.objecter ms_dispatch 0x561921fa3060 osd_op_reply(12 exist [read 0~5 out=5b] v0'0 uv4 ondisk = 0) v8 2024-06-03T08:38:30.161 INFO:tasks.workunit.client.0.smithi088.stdout: api_asio: 2024-06-03T08:38:30.159+0000 7f514a183640 10 client.4490.objecter in handle_osd_op_reply 2024-06-03T08:38:30.161 INFO:tasks.workunit.client.0.smithi088.stdout: api_asio: 2024-06-03T08:38:30.159+0000 7f514a183640 7 client.4490.objecter handle_osd_op_reply 12 ondisk uv 4 in 17.1 attempt 0 2024-06-03T08:38:30.161 INFO:tasks.workunit.client.0.smithi088.stdout: api_asio: 2024-06-03T08:38:30.159+0000 7f514a183640 10 client.4490.objecter op 0 rval 0 len 5 2024-06-03T08:38:30.161 INFO:tasks.workunit.client.0.smithi088.stdout: api_asio: 2024-06-03T08:38:30.159+0000 7f514a183640 15 client.4490.objecter handle_osd_op_reply completed tid 12 2024-06-03T08:38:30.162 INFO:tasks.workunit.client.0.smithi088.stdout: api_asio: 2024-06-03T08:38:30.159+0000 7f514a183640 15 client.4490.objecter _finish_op 12 2024-06-03T08:38:30.162 INFO:tasks.workunit.client.0.smithi088.stdout: api_asio: 2024-06-03T08:38:30.159+0000 7f514a183640 20 client.4490.objecter put_session s=0x5619220df110 osd=5 5 2024-06-03T08:38:30.162 INFO:tasks.workunit.client.0.smithi088.stdout: api_asio: 2024-06-03T08:38:30.159+0000 7f514a183640 15 client.4490.objecter _session_op_remove 5 12 2024-06-03T08:38:30.162 INFO:tasks.workunit.client.0.smithi088.stdout: api_asio: 2024-06-03T08:38:30.159+0000 7f514a183640 5 client.4490.objecter 1 in flight 2024-06-03T08:38:30.162 INFO:tasks.workunit.client.0.smithi088.stdout: api_asio: 2024-06-03T08:38:30.159+0000 7f514a183640 1 -- 172.21.15.88:0/320712639 <== osd.5 v1:172.21.15.192:6805/34786 13 ==== osd_op_reply(13 noexist [read 0~0] v0'0 uv0 ondisk = -2 ((2) No such file or directory)) v8 ==== 151+0+0 (unknown 3652525720 0 0) 0x7f513c0043a0 con 0x5619220df280 2024-06-03T08:38:30.162 INFO:tasks.workunit.client.0.smithi088.stdout: api_asio: 2024-06-03T08:38:30.159+0000 7f514a183640 10 client.4490.objecter ms_dispatch 0x561921fa3060 osd_op_reply(13 noexist [read 0~0] v0'0 uv0 ondisk = -2 ((2) No such file or directory)) v8 2024-06-03T08:38:30.162 INFO:tasks.workunit.client.0.smithi088.stdout: api_asio: 2024-06-03T08:38:30.159+0000 7f514a183640 10 client.4490.objecter in handle_osd_op_reply 2024-06-03T08:38:30.162 INFO:tasks.workunit.client.0.smithi088.stdout: api_asio: 2024-06-03T08:38:30.159+0000 7f514a183640 7 client.4490.objecter handle_osd_op_reply 13 ondisk uv 0 in 17.1 attempt 0 2024-06-03T08:38:30.162 INFO:tasks.workunit.client.0.smithi088.stdout: api_asio: 2024-06-03T08:38:30.159+0000 7f514a183640 10 client.4490.objecter handle_osd_op_reply copying resulting 0 into existing ceph::buffer of length 0 2024-06-03T08:38:30.162 INFO:tasks.workunit.client.0.smithi088.stdout: api_asio: 2024-06-03T08:38:30.159+0000 7f514a183640 10 client.4490.objecter op 0 rval 0 len 0 2024-06-03T08:38:30.162 INFO:tasks.workunit.client.0.smithi088.stdout: api_asio: 2024-06-03T08:38:30.159+0000 7f514a183640 15 client.4490.objecter handle_osd_op_reply completed tid 13 2024-06-03T08:38:30.162 INFO:tasks.workunit.client.0.smithi088.stdout: api_asio: 2024-06-03T08:38:30.159+0000 7f514a183640 15 client.4490.objecter _finish_op 13 2024-06-03T08:38:30.162 INFO:tasks.workunit.client.0.smithi088.stdout: api_asio: 2024-06-03T08:38:30.159+0000 7f514a183640 20 client.4490.objecter put_session s=0x5619220df110 osd=5 4 2024-06-03T08:38:30.163 INFO:tasks.workunit.client.0.smithi088.stdout: api_asio: 2024-06-03T08:38:30.159+0000 7f514a183640 15 client.4490.objecter _session_op_remove 5 13 2024-06-03T08:38:30.163 INFO:tasks.workunit.client.0.smithi088.stdout: api_asio: 2024-06-03T08:38:30.159+0000 7f514a183640 5 client.4490.objecter 0 in flight 2024-06-03T08:38:30.163 INFO:tasks.workunit.client.0.smithi088.stdout: api_asio: [ OK ] AsioRados.AsyncReadOperationCallback (646 ms) 2024-06-03T08:38:30.163 INFO:tasks.workunit.client.0.smithi088.stdout: api_asio: [ RUN ] AsioRados.AsyncReadOperationFuture 2024-06-03T08:38:30.163 INFO:tasks.workunit.client.0.smithi088.stdout: api_asio: 2024-06-03T08:38:30.159+0000 7f514d647780 10 client.4490.objecter _op_submit op 0x56192217a6b0 2024-06-03T08:38:30.163 INFO:tasks.workunit.client.0.smithi088.stdout: api_asio: 2024-06-03T08:38:30.159+0000 7f514d647780 20 client.4490.objecter _calc_target epoch 56 base exist @17 precalc_pgid 0 pgid 0.0 is_read 2024-06-03T08:38:30.163 INFO:tasks.workunit.client.0.smithi088.stdout: api_asio: 2024-06-03T08:38:30.159+0000 7f514d647780 20 client.4490.objecter _calc_target target exist @17 -> pgid 17.d3a59201 2024-06-03T08:38:30.163 INFO:tasks.workunit.client.0.smithi088.stdout: api_asio: 2024-06-03T08:38:30.159+0000 7f514d647780 10 client.4490.objecter _calc_target raw pgid 17.d3a59201 -> actual 17.1 acting [5,6] primary 5 2024-06-03T08:38:30.163 INFO:tasks.workunit.client.0.smithi088.stdout: api_asio: 2024-06-03T08:38:30.159+0000 7f514d647780 20 client.4490.objecter _get_session s=0x5619220df110 osd=5 3 2024-06-03T08:38:30.163 INFO:tasks.workunit.client.0.smithi088.stdout: api_asio: 2024-06-03T08:38:30.159+0000 7f514d647780 10 client.4490.objecter _op_submit oid exist '@17' '@17' [read 0~0] tid 14 osd.5 2024-06-03T08:38:30.163 INFO:tasks.workunit.client.0.smithi088.stdout: api_asio: 2024-06-03T08:38:30.159+0000 7f514d647780 20 client.4490.objecter get_session s=0x5619220df110 osd=5 3 2024-06-03T08:38:30.163 INFO:tasks.workunit.client.0.smithi088.stdout: api_asio: 2024-06-03T08:38:30.159+0000 7f514d647780 15 client.4490.objecter _session_op_assign 5 14 2024-06-03T08:38:30.163 INFO:tasks.workunit.client.0.smithi088.stdout: api_asio: 2024-06-03T08:38:30.159+0000 7f514d647780 15 client.4490.objecter _send_op 14 to 17.1 on osd.5 2024-06-03T08:38:30.163 INFO:tasks.workunit.client.0.smithi088.stdout: api_asio: 2024-06-03T08:38:30.159+0000 7f514d647780 1 -- 172.21.15.88:0/320712639 --> v1:172.21.15.192:6805/34786 -- osd_op(unknown.0.0:14 17.1 17:8049a5cb:::exist:head [read 0~0] snapc 0=[] ondisk+read+known_if_redirected+supports_pool_eio e56) v8 -- 0x56192217af50 con 0x5619220df280 2024-06-03T08:38:30.164 INFO:tasks.workunit.client.0.smithi088.stdout: api_asio: 2024-06-03T08:38:30.159+0000 7f514d647780 20 client.4490.objecter put_session s=0x5619220df110 osd=5 4 2024-06-03T08:38:30.164 INFO:tasks.workunit.client.0.smithi088.stdout: api_asio: 2024-06-03T08:38:30.159+0000 7f514d647780 5 client.4490.objecter 1 in flight 2024-06-03T08:38:30.164 INFO:tasks.workunit.client.0.smithi088.stdout: api_asio: 2024-06-03T08:38:30.159+0000 7f514d647780 10 client.4490.objecter _op_submit op 0x56192219e6b0 2024-06-03T08:38:30.254 INFO:tasks.ceph.osd.6.smithi192.stderr:2024-06-03T08:38:30.252+0000 12a6c640 -1 osd.6 55 get_health_metrics reporting 1 slow ops, oldest is osd_op(client.4524.0:2 16.1a 16:5c6776db:::0.obj:head [write 0~248 in=248b] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e28) 2024-06-03T08:38:31.033 INFO:tasks.ceph.osd.7.smithi192.stderr:2024-06-03T08:38:31.007+0000 12a6c640 -1 osd.7 56 get_health_metrics reporting 2 slow ops, oldest is osd_op(client.4536.0:2 21.1a 21.db6ee63a (undecoded) ondisk+write+known_if_redirected+supports_pool_eio e28) 2024-06-03T08:38:31.339 INFO:tasks.ceph.osd.6.smithi192.stderr:2024-06-03T08:38:31.335+0000 12a6c640 -1 osd.6 56 get_health_metrics reporting 1 slow ops, oldest is osd_op(client.4524.0:2 16.1a 16:5c6776db:::0.obj:head [write 0~248 in=248b] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e28) 2024-06-03T08:38:32.303 INFO:tasks.ceph.osd.7.smithi192.stderr:2024-06-03T08:38:32.279+0000 12a6c640 -1 osd.7 57 get_health_metrics reporting 2 slow ops, oldest is osd_op(client.4536.0:2 21.1a 21.db6ee63a (undecoded) ondisk+write+known_if_redirected+supports_pool_eio e28) 2024-06-03T08:38:32.466 INFO:tasks.ceph.osd.6.smithi192.stderr:2024-06-03T08:38:32.456+0000 12a6c640 -1 osd.6 57 get_health_metrics reporting 1 slow ops, oldest is osd_op(client.4524.0:2 16.1a 16:5c6776db:::0.obj:head [write 0~248 in=248b] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e28) 2024-06-03T08:38:32.596 INFO:tasks.workunit.client.0.smithi088.stdout: api_asio: 2024-06-03T08:38:30.159+0000 7f514d647780 20 client.4490.objecter _calc_target epoch 56 base noexist @17 precalc_pgid 0 pgid 0.0 is_read 2024-06-03T08:38:32.596 INFO:tasks.workunit.client.0.smithi088.stdout: api_asio: 2024-06-03T08:38:30.159+0000 7f514d647780 20 client.4490.objecter _calc_target target noexist @17 -> pgid 17.a7bbaa1 2024-06-03T08:38:32.596 INFO:tasks.workunit.client.0.smithi088.stdout: api_asio: 2024-06-03T08:38:30.159+0000 7f514d647780 10 client.4490.objecter _calc_target raw pgid 17.a7bbaa1 -> actual 17.1 acting [5,6] primary 5 2024-06-03T08:38:32.596 INFO:tasks.workunit.client.0.smithi088.stdout: api_asio: 2024-06-03T08:38:30.159+0000 7f514d647780 20 client.4490.objecter _get_session s=0x5619220df110 osd=5 4 2024-06-03T08:38:32.596 INFO:tasks.workunit.client.0.smithi088.stdout: api_asio: 2024-06-03T08:38:30.159+0000 7f514d647780 10 client.4490.objecter _op_submit oid noexist '@17' '@17' [read 0~0] tid 15 osd.5 2024-06-03T08:38:32.596 INFO:tasks.workunit.client.0.smithi088.stdout: api_asio: 2024-06-03T08:38:30.159+0000 7f514d647780 20 client.4490.objecter get_session s=0x5619220df110 osd=5 4 2024-06-03T08:38:32.596 INFO:tasks.workunit.client.0.smithi088.stdout: api_asio: 2024-06-03T08:38:30.159+0000 7f514d647780 15 client.4490.objecter _session_op_assign 5 15 2024-06-03T08:38:32.596 INFO:tasks.workunit.client.0.smithi088.stdout: api_asio: 2024-06-03T08:38:30.159+0000 7f514d647780 15 client.4490.objecter _send_op 15 to 17.1 on osd.5 2024-06-03T08:38:32.596 INFO:tasks.workunit.client.0.smithi088.stdout: api_asio: 2024-06-03T08:38:30.159+0000 7f514d647780 1 -- 172.21.15.88:0/320712639 --> v1:172.21.15.192:6805/34786 -- osd_op(unknown.0.0:15 17.1 17:855dde50:::noexist:head [read 0~0] snapc 0=[] ondisk+read+known_if_redirected+supports_pool_eio e56) v8 -- 0x56192217b490 con 0x5619220df280 2024-06-03T08:38:32.597 INFO:tasks.workunit.client.0.smithi088.stdout: api_asio: 2024-06-03T08:38:30.159+0000 7f514d647780 20 client.4490.objecter put_session s=0x5619220df110 osd=5 5 2024-06-03T08:38:32.597 INFO:tasks.workunit.client.0.smithi088.stdout: api_asio: 2024-06-03T08:38:30.159+0000 7f514d647780 5 client.4490.objecter 2 in flight 2024-06-03T08:38:32.597 INFO:tasks.workunit.client.0.smithi088.stdout: api_asio: 2024-06-03T08:38:30.595+0000 7f5132ffd640 10 monclient: tick 2024-06-03T08:38:32.597 INFO:tasks.workunit.client.0.smithi088.stdout: api_asio: 2024-06-03T08:38:30.595+0000 7f5132ffd640 10 monclient: _check_auth_tickets 2024-06-03T08:38:32.597 INFO:tasks.workunit.client.0.smithi088.stdout: api_asio: 2024-06-03T08:38:30.595+0000 7f5132ffd640 20 monclient: _check_auth_rotating not needed by client.admin 2024-06-03T08:38:32.597 INFO:tasks.workunit.client.0.smithi088.stdout: api_asio: 2024-06-03T08:38:31.135+0000 7f5133fff640 1 -- 172.21.15.88:0/320712639 <== mon.2 v1:172.21.15.88:6790/0 21 ==== osd_map(57..57 src has 1..57) v4 ==== 1426+0+0 (unknown 1901431947 0 0) 0x7f51440503f0 con 0x5619220ccfb0 2024-06-03T08:38:32.597 INFO:tasks.workunit.client.0.smithi088.stdout: api_asio: 2024-06-03T08:38:31.135+0000 7f5133fff640 10 client.4490.objecter ms_dispatch 0x561921fa3060 osd_map(57..57 src has 1..57) v4 2024-06-03T08:38:32.597 INFO:tasks.workunit.client.0.smithi088.stdout: api_asio: 2024-06-03T08:38:31.135+0000 7f5133fff640 3 client.4490.objecter handle_osd_map got epochs [57,57] > 56 2024-06-03T08:38:32.597 INFO:tasks.workunit.client.0.smithi088.stdout: api_asio: 2024-06-03T08:38:31.135+0000 7f5133fff640 3 client.4490.objecter handle_osd_map decoding incremental epoch 57 2024-06-03T08:38:32.597 INFO:tasks.workunit.client.0.smithi088.stdout: api_asio: 2024-06-03T08:38:31.135+0000 7f5133fff640 10 client.4490.objecter checking op 14 2024-06-03T08:38:32.597 INFO:tasks.workunit.client.0.smithi088.stdout: api_asio: 2024-06-03T08:38:31.135+0000 7f5133fff640 20 client.4490.objecter _calc_target epoch 57 base exist @17 precalc_pgid 0 pgid 0.0 is_read 2024-06-03T08:38:32.597 INFO:tasks.workunit.client.0.smithi088.stdout: api_asio: 2024-06-03T08:38:31.135+0000 7f5133fff640 20 client.4490.objecter _calc_target target exist @17 -> pgid 17.d3a59201 2024-06-03T08:38:32.597 INFO:tasks.workunit.client.0.smithi088.stdout: api_asio: 2024-06-03T08:38:31.135+0000 7f5133fff640 10 client.4490.objecter checking op 15 2024-06-03T08:38:32.598 INFO:tasks.workunit.client.0.smithi088.stdout: api_asio: 2024-06-03T08:38:31.135+0000 7f5133fff640 20 client.4490.objecter _calc_target epoch 57 base noexist @17 precalc_pgid 0 pgid 0.0 is_read 2024-06-03T08:38:32.598 INFO:tasks.workunit.client.0.smithi088.stdout: api_asio: 2024-06-03T08:38:31.135+0000 7f5133fff640 20 client.4490.objecter _calc_target target noexist @17 -> pgid 17.a7bbaa1 2024-06-03T08:38:32.598 INFO:tasks.workunit.client.0.smithi088.stdout: api_asio: 2024-06-03T08:38:31.135+0000 7f5133fff640 20 client.4490.objecter dump_active .. 0 homeless 2024-06-03T08:38:32.598 INFO:tasks.workunit.client.0.smithi088.stdout: api_asio: 2024-06-03T08:38:31.135+0000 7f5133fff640 20 client.4490.objecter 14 17.d3a59201 osd.5 exist [read 0~0] 2024-06-03T08:38:32.598 INFO:tasks.workunit.client.0.smithi088.stdout: api_asio: 2024-06-03T08:38:31.135+0000 7f5133fff640 20 client.4490.objecter 15 17.a7bbaa1 osd.5 noexist [read 0~0] 2024-06-03T08:38:32.598 INFO:tasks.workunit.client.0.smithi088.stdout: api_asio: 2024-06-03T08:38:31.595+0000 7f5132ffd640 10 monclient: tick 2024-06-03T08:38:32.598 INFO:tasks.workunit.client.0.smithi088.stdout: api_asio: 2024-06-03T08:38:31.595+0000 7f5132ffd640 10 monclient: _check_auth_tickets 2024-06-03T08:38:32.598 INFO:tasks.workunit.client.0.smithi088.stdout: api_asio: 2024-06-03T08:38:31.595+0000 7f5132ffd640 20 monclient: _check_auth_rotating not needed by client.admin 2024-06-03T08:38:32.667 INFO:tasks.workunit.client.0.smithi088.stdout: api_asio: 2024-06-03T08:38:32.595+0000 7f5132ffd640 10 monclient: tick 2024-06-03T08:38:32.667 INFO:tasks.workunit.client.0.smithi088.stdout: api_asio: 2024-06-03T08:38:32.595+0000 7f5132ffd640 10 monclient: _check_auth_tickets 2024-06-03T08:38:32.667 INFO:tasks.workunit.client.0.smithi088.stdout: api_asio: 2024-06-03T08:38:32.595+0000 7f5132ffd640 20 monclient: _check_auth_rotating not needed by client.admin 2024-06-03T08:38:32.667 INFO:tasks.workunit.client.0.smithi088.stdout: api_asio: 2024-06-03T08:38:32.666+0000 7f514a183640 1 -- 172.21.15.88:0/320712639 <== osd.5 v1:172.21.15.192:6805/34786 14 ==== osd_op_reply(14 exist [read 0~5 out=5b] v0'0 uv4 ondisk = 0) v8 ==== 149+0+5 (unknown 3926757721 0 3741568377) 0x7f513c0043a0 con 0x5619220df280 2024-06-03T08:38:32.667 INFO:tasks.workunit.client.0.smithi088.stdout: api_asio: 2024-06-03T08:38:32.666+0000 7f514a183640 10 client.4490.objecter ms_dispatch 0x561921fa3060 osd_op_reply(14 exist [read 0~5 out=5b] v0'0 uv4 ondisk = 0) v8 2024-06-03T08:38:32.667 INFO:tasks.workunit.client.0.smithi088.stdout: api_asio: 2024-06-03T08:38:32.666+0000 7f514a183640 10 client.4490.objecter in handle_osd_op_reply 2024-06-03T08:38:32.667 INFO:tasks.workunit.client.0.smithi088.stdout: api_asio: 2024-06-03T08:38:32.666+0000 7f514a183640 7 client.4490.objecter handle_osd_op_reply 14 ondisk uv 4 in 17.1 attempt 0 2024-06-03T08:38:32.667 INFO:tasks.workunit.client.0.smithi088.stdout: api_asio: 2024-06-03T08:38:32.666+0000 7f514a183640 10 client.4490.objecter op 0 rval 0 len 5 2024-06-03T08:38:32.668 INFO:tasks.workunit.client.0.smithi088.stdout: api_asio: 2024-06-03T08:38:32.666+0000 7f514a183640 15 client.4490.objecter handle_osd_op_reply completed tid 14 2024-06-03T08:38:32.668 INFO:tasks.workunit.client.0.smithi088.stdout: api_asio: 2024-06-03T08:38:32.666+0000 7f514a183640 15 client.4490.objecter _finish_op 14 2024-06-03T08:38:32.668 INFO:tasks.workunit.client.0.smithi088.stdout: api_asio: 2024-06-03T08:38:32.666+0000 7f514a183640 20 client.4490.objecter put_session s=0x5619220df110 osd=5 5 2024-06-03T08:38:32.668 INFO:tasks.workunit.client.0.smithi088.stdout: api_asio: 2024-06-03T08:38:32.666+0000 7f514a183640 15 client.4490.objecter _session_op_remove 5 14 2024-06-03T08:38:32.668 INFO:tasks.workunit.client.0.smithi088.stdout: api_asio: 2024-06-03T08:38:32.666+0000 7f514a183640 5 client.4490.objecter 1 in flight 2024-06-03T08:38:32.668 INFO:tasks.workunit.client.0.smithi088.stdout: api_asio: 2024-06-03T08:38:32.666+0000 7f514a183640 1 -- 172.21.15.88:0/320712639 <== osd.5 v1:172.21.15.192:6805/34786 15 ==== osd_op_reply(15 noexist [read 0~0] v0'0 uv0 ondisk = -2 ((2) No such file or directory)) v8 ==== 151+0+0 (unknown 1645001033 0 0) 0x7f513c0043a0 con 0x5619220df280 2024-06-03T08:38:32.668 INFO:tasks.workunit.client.0.smithi088.stdout: api_asio: 2024-06-03T08:38:32.666+0000 7f514a183640 10 client.4490.objecter ms_dispatch 0x561921fa3060 osd_op_reply(15 noexist [read 0~0] v0'0 uv0 ondisk = -2 ((2) No such file or directory)) v8 2024-06-03T08:38:32.668 INFO:tasks.workunit.client.0.smithi088.stdout: api_asio: 2024-06-03T08:38:32.666+0000 7f514a183640 10 client.4490.objecter in handle_osd_op_reply 2024-06-03T08:38:32.668 INFO:tasks.workunit.client.0.smithi088.stdout: api_asio: 2024-06-03T08:38:32.666+0000 7f514a183640 7 client.4490.objecter handle_osd_op_reply 15 ondisk uv 0 in 17.1 attempt 0 2024-06-03T08:38:32.668 INFO:tasks.workunit.client.0.smithi088.stdout: api_asio: 2024-06-03T08:38:32.666+0000 7f514a183640 10 client.4490.objecter handle_osd_op_reply copying resulting 0 into existing ceph::buffer of length 0 2024-06-03T08:38:32.668 INFO:tasks.workunit.client.0.smithi088.stdout: api_asio: 2024-06-03T08:38:32.666+0000 7f514a183640 10 client.4490.objecter op 0 rval 0 len 0 2024-06-03T08:38:32.668 INFO:tasks.workunit.client.0.smithi088.stdout: api_asio: 2024-06-03T08:38:32.666+0000 7f514a183640 15 client.4490.objecter handle_osd_op_reply completed tid 15 2024-06-03T08:38:32.669 INFO:tasks.workunit.client.0.smithi088.stdout: api_asio: 2024-06-03T08:38:32.666+0000 7f514a183640 15 client.4490.objecter _finish_op 15 2024-06-03T08:38:32.669 INFO:tasks.workunit.client.0.smithi088.stdout: api_asio: 2024-06-03T08:38:32.666+0000 7f514a183640 20 client.4490.objecter put_session s=0x5619220df110 osd=5 4 2024-06-03T08:38:32.669 INFO:tasks.workunit.client.0.smithi088.stdout: api_asio: 2024-06-03T08:38:32.666+0000 7f514a183640 15 client.4490.objecter _session_op_remove 5 15 2024-06-03T08:38:32.669 INFO:tasks.workunit.client.0.smithi088.stdout: api_asio: 2024-06-03T08:38:32.666+0000 7f514a183640 5 client.4490.objecter 0 in flight 2024-06-03T08:38:32.669 INFO:tasks.workunit.client.0.smithi088.stdout: api_asio: [ OK ] AsioRados.AsyncReadOperationFuture (2507 ms) 2024-06-03T08:38:32.669 INFO:tasks.workunit.client.0.smithi088.stdout: api_asio: [ RUN ] AsioRados.AsyncReadOperationYield 2024-06-03T08:38:32.669 INFO:tasks.workunit.client.0.smithi088.stdout: api_asio: 2024-06-03T08:38:32.666+0000 7f514d647780 10 client.4490.objecter _op_submit op 0x56192219da90 2024-06-03T08:38:32.669 INFO:tasks.workunit.client.0.smithi088.stdout: api_asio: 2024-06-03T08:38:32.666+0000 7f514d647780 20 client.4490.objecter _calc_target epoch 57 base exist @17 precalc_pgid 0 pgid 0.0 is_read 2024-06-03T08:38:32.669 INFO:tasks.workunit.client.0.smithi088.stdout: api_asio: 2024-06-03T08:38:32.666+0000 7f514d647780 20 client.4490.objecter _calc_target target exist @17 -> pgid 17.d3a59201 2024-06-03T08:38:33.642 INFO:tasks.ceph.osd.2.smithi088.stderr:2024-06-03T08:38:33.597+0000 12a6c640 -1 osd.2 57 get_health_metrics reporting 1 slow ops, oldest is osd_op(client.4590.0:2 30.6 30.7fc1f406 (undecoded) ondisk+write+known_if_redirected+supports_pool_eio e31) 2024-06-03T08:38:33.698 INFO:tasks.ceph.osd.7.smithi192.stderr:2024-06-03T08:38:33.690+0000 12a6c640 -1 osd.7 58 get_health_metrics reporting 2 slow ops, oldest is osd_op(client.4536.0:2 21.1a 21:5c6776db:::0.obj:head [write 0~152 in=152b] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e28) 2024-06-03T08:38:34.522 INFO:tasks.workunit.client.0.smithi088.stdout: api_asio: 2024-06-03T08:38:32.666+0000 7f514d647780 10 client.4490.objecter _calc_target raw pgid 17.d3a59201 -> actual 17.1 acting [5,6] primary 5 2024-06-03T08:38:34.522 INFO:tasks.workunit.client.0.smithi088.stdout: api_asio: 2024-06-03T08:38:32.666+0000 7f514d647780 20 client.4490.objecter _get_session s=0x5619220df110 osd=5 3 2024-06-03T08:38:34.522 INFO:tasks.workunit.client.0.smithi088.stdout: api_asio: 2024-06-03T08:38:32.666+0000 7f514d647780 10 client.4490.objecter _op_submit oid exist '@17' '@17' [read 0~0] tid 16 osd.5 2024-06-03T08:38:34.522 INFO:tasks.workunit.client.0.smithi088.stdout: api_asio: 2024-06-03T08:38:32.666+0000 7f514d647780 20 client.4490.objecter get_session s=0x5619220df110 osd=5 3 2024-06-03T08:38:34.522 INFO:tasks.workunit.client.0.smithi088.stdout: api_asio: 2024-06-03T08:38:32.666+0000 7f514d647780 15 client.4490.objecter _session_op_assign 5 16 2024-06-03T08:38:34.522 INFO:tasks.workunit.client.0.smithi088.stdout: api_asio: 2024-06-03T08:38:32.666+0000 7f514d647780 15 client.4490.objecter _send_op 16 to 17.1 on osd.5 2024-06-03T08:38:34.522 INFO:tasks.workunit.client.0.smithi088.stdout: api_asio: 2024-06-03T08:38:32.666+0000 7f514d647780 1 -- 172.21.15.88:0/320712639 --> v1:172.21.15.192:6805/34786 -- osd_op(unknown.0.0:16 17.1 17:8049a5cb:::exist:head [read 0~0] snapc 0=[] ondisk+read+known_if_redirected+supports_pool_eio e57) v8 -- 0x56192217af50 con 0x5619220df280 2024-06-03T08:38:34.522 INFO:tasks.workunit.client.0.smithi088.stdout: api_asio: 2024-06-03T08:38:32.666+0000 7f514d647780 20 client.4490.objecter put_session s=0x5619220df110 osd=5 4 2024-06-03T08:38:34.523 INFO:tasks.workunit.client.0.smithi088.stdout: api_asio: 2024-06-03T08:38:32.666+0000 7f514d647780 5 client.4490.objecter 1 in flight 2024-06-03T08:38:34.523 INFO:tasks.workunit.client.0.smithi088.stdout: api_asio: 2024-06-03T08:38:32.666+0000 7f514d647780 10 client.4490.objecter _op_submit op 0x56192219e6b0 2024-06-03T08:38:34.523 INFO:tasks.workunit.client.0.smithi088.stdout: api_asio: 2024-06-03T08:38:32.666+0000 7f514d647780 20 client.4490.objecter _calc_target epoch 57 base noexist @17 precalc_pgid 0 pgid 0.0 is_read 2024-06-03T08:38:34.523 INFO:tasks.workunit.client.0.smithi088.stdout: api_asio: 2024-06-03T08:38:32.666+0000 7f514d647780 20 client.4490.objecter _calc_target target noexist @17 -> pgid 17.a7bbaa1 2024-06-03T08:38:34.523 INFO:tasks.workunit.client.0.smithi088.stdout: api_asio: 2024-06-03T08:38:32.666+0000 7f514d647780 10 client.4490.objecter _calc_target raw pgid 17.a7bbaa1 -> actual 17.1 acting [5,6] primary 5 2024-06-03T08:38:34.523 INFO:tasks.workunit.client.0.smithi088.stdout: api_asio: 2024-06-03T08:38:32.666+0000 7f514d647780 20 client.4490.objecter _get_session s=0x5619220df110 osd=5 4 2024-06-03T08:38:34.523 INFO:tasks.workunit.client.0.smithi088.stdout: api_asio: 2024-06-03T08:38:32.666+0000 7f514d647780 10 client.4490.objecter _op_submit oid noexist '@17' '@17' [read 0~0] tid 17 osd.5 2024-06-03T08:38:34.523 INFO:tasks.workunit.client.0.smithi088.stdout: api_asio: 2024-06-03T08:38:32.666+0000 7f514d647780 20 client.4490.objecter get_session s=0x5619220df110 osd=5 4 2024-06-03T08:38:34.523 INFO:tasks.workunit.client.0.smithi088.stdout: api_asio: 2024-06-03T08:38:32.666+0000 7f514d647780 15 client.4490.objecter _session_op_assign 5 17 2024-06-03T08:38:34.523 INFO:tasks.workunit.client.0.smithi088.stdout: api_asio: 2024-06-03T08:38:32.666+0000 7f514d647780 15 client.4490.objecter _send_op 17 to 17.1 on osd.5 2024-06-03T08:38:34.523 INFO:tasks.workunit.client.0.smithi088.stdout: api_asio: 2024-06-03T08:38:32.666+0000 7f514d647780 1 -- 172.21.15.88:0/320712639 --> v1:172.21.15.192:6805/34786 -- osd_op(unknown.0.0:17 17.1 17:855dde50:::noexist:head [read 0~0] snapc 0=[] ondisk+read+known_if_redirected+supports_pool_eio e57) v8 -- 0x56192219eba0 con 0x5619220df280 2024-06-03T08:38:34.523 INFO:tasks.workunit.client.0.smithi088.stdout: api_asio: 2024-06-03T08:38:32.666+0000 7f514d647780 20 client.4490.objecter put_session s=0x5619220df110 osd=5 5 2024-06-03T08:38:34.524 INFO:tasks.workunit.client.0.smithi088.stdout: api_asio: 2024-06-03T08:38:32.666+0000 7f514d647780 5 client.4490.objecter 2 in flight 2024-06-03T08:38:34.524 INFO:tasks.workunit.client.0.smithi088.stdout: api_asio: 2024-06-03T08:38:33.596+0000 7f5132ffd640 10 monclient: tick 2024-06-03T08:38:34.524 INFO:tasks.workunit.client.0.smithi088.stdout: api_asio: 2024-06-03T08:38:33.596+0000 7f5132ffd640 10 monclient: _check_auth_tickets 2024-06-03T08:38:34.524 INFO:tasks.workunit.client.0.smithi088.stdout: api_asio: 2024-06-03T08:38:33.596+0000 7f5132ffd640 20 monclient: _check_auth_rotating not needed by client.admin 2024-06-03T08:38:34.524 INFO:tasks.workunit.client.0.smithi088.stdout: api_asio: 2024-06-03T08:38:33.774+0000 7f514cbe8640 10 client.4490.objecter tick 2024-06-03T08:38:34.524 INFO:tasks.workunit.client.0.smithi088.stdout: api_asio: 2024-06-03T08:38:34.521+0000 7f514a183640 1 -- 172.21.15.88:0/320712639 <== osd.5 v1:172.21.15.192:6805/34786 16 ==== osd_op_reply(16 exist [read 0~5 out=5b] v0'0 uv4 ondisk = 0) v8 ==== 149+0+5 (unknown 1812562675 0 3741568377) 0x7f513c0043a0 con 0x5619220df280 2024-06-03T08:38:34.524 INFO:tasks.workunit.client.0.smithi088.stdout: api_asio: 2024-06-03T08:38:34.521+0000 7f514a183640 10 client.4490.objecter ms_dispatch 0x561921fa3060 osd_op_reply(16 exist [read 0~5 out=5b] v0'0 uv4 ondisk = 0) v8 2024-06-03T08:38:34.524 INFO:tasks.workunit.client.0.smithi088.stdout: api_asio: 2024-06-03T08:38:34.521+0000 7f514a183640 10 client.4490.objecter in handle_osd_op_reply 2024-06-03T08:38:34.524 INFO:tasks.workunit.client.0.smithi088.stdout: api_asio: 2024-06-03T08:38:34.521+0000 7f514a183640 7 client.4490.objecter handle_osd_op_reply 16 ondisk uv 4 in 17.1 attempt 0 2024-06-03T08:38:34.524 INFO:tasks.workunit.client.0.smithi088.stdout: api_asio: 2024-06-03T08:38:34.521+0000 7f514a183640 10 client.4490.objecter op 0 rval 0 len 5 2024-06-03T08:38:34.524 INFO:tasks.workunit.client.0.smithi088.stdout: api_asio: 2024-06-03T08:38:34.521+0000 7f514a183640 15 client.4490.objecter handle_osd_op_reply completed tid 16 2024-06-03T08:38:34.524 INFO:tasks.workunit.client.0.smithi088.stdout: api_asio: 2024-06-03T08:38:34.521+0000 7f514a183640 15 client.4490.objecter _finish_op 16 2024-06-03T08:38:34.524 INFO:tasks.workunit.client.0.smithi088.stdout: api_asio: 2024-06-03T08:38:34.521+0000 7f514a183640 20 client.4490.objecter put_session s=0x5619220df110 osd=5 5 2024-06-03T08:38:34.525 INFO:tasks.workunit.client.0.smithi088.stdout: api_asio: 2024-06-03T08:38:34.521+0000 7f514a183640 15 client.4490.objecter _session_op_remove 5 16 2024-06-03T08:38:34.525 INFO:tasks.workunit.client.0.smithi088.stdout: api_asio: 2024-06-03T08:38:34.521+0000 7f514a183640 5 client.4490.objecter 1 in flight 2024-06-03T08:38:34.525 INFO:tasks.workunit.client.0.smithi088.stdout: api_asio: [ OK ] AsioRados.AsyncReadOperationYield (1855 ms) 2024-06-03T08:38:34.525 INFO:tasks.workunit.client.0.smithi088.stdout: api_asio: [ RUN ] AsioRados.AsyncWriteOperationCallback 2024-06-03T08:38:34.525 INFO:tasks.workunit.client.0.smithi088.stdout: api_asio: 2024-06-03T08:38:34.521+0000 7f514a183640 1 -- 172.21.15.88:0/320712639 <== osd.5 v1:172.21.15.192:6805/34786 17 ==== osd_op_reply(17 noexist [read 0~0] v0'0 uv0 ondisk = -2 ((2) No such file or directory)) v8 ==== 151+0+0 (unknown 1810426092 0 0) 0x7f513c0043a0 con 0x5619220df280 2024-06-03T08:38:34.525 INFO:tasks.workunit.client.0.smithi088.stdout: api_asio: 2024-06-03T08:38:34.521+0000 7f514a183640 10 client.4490.objecter ms_dispatch 0x561921fa3060 osd_op_reply(17 noexist [read 0~0] v0'0 uv0 ondisk = -2 ((2) No such file or directory)) v8 2024-06-03T08:38:34.525 INFO:tasks.workunit.client.0.smithi088.stdout: api_asio: 2024-06-03T08:38:34.521+0000 7f514a183640 10 client.4490.objecter in handle_osd_op_reply 2024-06-03T08:38:34.525 INFO:tasks.workunit.client.0.smithi088.stdout: api_asio: 2024-06-03T08:38:34.521+0000 7f514a183640 7 client.4490.objecter handle_osd_op_reply 17 ondisk uv 0 in 17.1 attempt 0 2024-06-03T08:38:34.525 INFO:tasks.workunit.client.0.smithi088.stdout: api_asio: 2024-06-03T08:38:34.521+0000 7f514a183640 10 client.4490.objecter handle_osd_op_reply copying resulting 0 into existing ceph::buffer of length 0 2024-06-03T08:38:34.525 INFO:tasks.workunit.client.0.smithi088.stdout: api_asio: 2024-06-03T08:38:34.521+0000 7f514a183640 10 client.4490.objecter op 0 rval 0 len 0 2024-06-03T08:38:34.525 INFO:tasks.workunit.client.0.smithi088.stdout: api_asio: 2024-06-03T08:38:34.521+0000 7f514a183640 15 client.4490.objecter handle_osd_op_reply completed tid 17 2024-06-03T08:38:34.525 INFO:tasks.workunit.client.0.smithi088.stdout: api_asio: 2024-06-03T08:38:34.521+0000 7f514a183640 15 client.4490.objecter _finish_op 17 2024-06-03T08:38:34.525 INFO:tasks.workunit.client.0.smithi088.stdout: api_asio: 2024-06-03T08:38:34.521+0000 7f514a183640 20 client.4490.objecter put_session s=0x5619220df110 osd=5 4 2024-06-03T08:38:34.525 INFO:tasks.workunit.client.0.smithi088.stdout: api_asio: 2024-06-03T08:38:34.521+0000 7f514a183640 15 client.4490.objecter _session_op_remove 5 17 2024-06-03T08:38:34.526 INFO:tasks.workunit.client.0.smithi088.stdout: api_asio: 2024-06-03T08:38:34.521+0000 7f514a183640 5 client.4490.objecter 0 in flight 2024-06-03T08:38:34.526 INFO:tasks.workunit.client.0.smithi088.stdout: api_asio: 2024-06-03T08:38:34.521+0000 7f514d647780 20 librados: queue_aio_write 0x561921f4e5e0 completion 0x56192219e2e0 write_seq 4 2024-06-03T08:38:34.526 INFO:tasks.workunit.client.0.smithi088.stdout: api_asio: 2024-06-03T08:38:34.521+0000 7f514d647780 10 client.4490.objecter _op_submit op 0x56192217a6b0 2024-06-03T08:38:34.526 INFO:tasks.workunit.client.0.smithi088.stdout: api_asio: 2024-06-03T08:38:34.521+0000 7f514d647780 20 client.4490.objecter _calc_target epoch 57 base exist @17 precalc_pgid 0 pgid 0.0 is_write 2024-06-03T08:38:34.526 INFO:tasks.workunit.client.0.smithi088.stdout: api_asio: 2024-06-03T08:38:34.521+0000 7f514d647780 20 client.4490.objecter _calc_target target exist @17 -> pgid 17.d3a59201 2024-06-03T08:38:34.526 INFO:tasks.workunit.client.0.smithi088.stdout: api_asio: 2024-06-03T08:38:34.521+0000 7f514d647780 10 client.4490.objecter _calc_target raw pgid 17.d3a59201 -> actual 17.1 acting [5,6] primary 5 2024-06-03T08:38:34.526 INFO:tasks.workunit.client.0.smithi088.stdout: api_asio: 2024-06-03T08:38:34.521+0000 7f514d647780 20 client.4490.objecter _get_session s=0x5619220df110 osd=5 3 2024-06-03T08:38:34.526 INFO:tasks.workunit.client.0.smithi088.stdout: api_asio: 2024-06-03T08:38:34.521+0000 7f514d647780 10 client.4490.objecter _op_submit oid exist '@17' '@17' [writefull 0~5 in=5b] tid 18 osd.5 2024-06-03T08:38:34.526 INFO:tasks.workunit.client.0.smithi088.stdout: api_asio: 2024-06-03T08:38:34.521+0000 7f514d647780 20 client.4490.objecter get_session s=0x5619220df110 osd=5 3 2024-06-03T08:38:34.526 INFO:tasks.workunit.client.0.smithi088.stdout: api_asio: 2024-06-03T08:38:34.521+0000 7f514d647780 15 client.4490.objecter _session_op_assign 5 18 2024-06-03T08:38:34.526 INFO:tasks.workunit.client.0.smithi088.stdout: api_asio: 2024-06-03T08:38:34.521+0000 7f514d647780 15 client.4490.objecter _send_op 18 to 17.1 on osd.5 2024-06-03T08:38:34.694 INFO:tasks.ceph.osd.2.smithi088.stderr:2024-06-03T08:38:34.665+0000 12a6c640 -1 osd.2 58 get_health_metrics reporting 1 slow ops, oldest is osd_op(client.4590.0:2 30.6 30.7fc1f406 (undecoded) ondisk+write+known_if_redirected+supports_pool_eio e31) 2024-06-03T08:38:35.152 INFO:tasks.workunit.client.0.smithi088.stdout: api_asio: 2024-06-03T08:38:34.521+0000 7f514d647780 1 -- 172.21.15.88:0/320712639 --> v1 api_cls_remote_reads: [==========] Running 1 test from 1 test suite. 2024-06-03T08:38:35.152 INFO:tasks.workunit.client.0.smithi088.stdout: api_cls_remote_reads: [----------] Global test environment set-up. 2024-06-03T08:38:35.152 INFO:tasks.workunit.client.0.smithi088.stdout: api_cls_remote_reads: [----------] 1 test from ClsTestRemoteReads 2024-06-03T08:38:35.152 INFO:tasks.workunit.client.0.smithi088.stdout: api_cls_remote_reads: [ RUN ] ClsTestRemoteReads.TestGather 2024-06-03T08:38:35.152 INFO:tasks.workunit.client.0.smithi088.stdout: api_cls_remote_reads: [ OK ] ClsTestRemoteReads.TestGather (41475 ms) 2024-06-03T08:38:35.152 INFO:tasks.workunit.client.0.smithi088.stdout: api_cls_remote_reads: [----------] 1 test from ClsTestRemoteReads (41475 ms total) 2024-06-03T08:38:35.152 INFO:tasks.workunit.client.0.smithi088.stdout: api_cls_remote_reads: 2024-06-03T08:38:35.152 INFO:tasks.workunit.client.0.smithi088.stdout: api_cls_remote_reads: [----------] Global test environment tear-down 2024-06-03T08:38:35.153 INFO:tasks.workunit.client.0.smithi088.stdout: api_cls_remote_reads: [==========] 1 test from 1 test suite ran. (41475 ms total) 2024-06-03T08:38:35.153 INFO:tasks.workunit.client.0.smithi088.stdout: api_cls_remote_reads: [ PASSED ] 1 test. 2024-06-03T08:38:35.815 INFO:tasks.workunit.client.0.smithi088.stdout::172.21.15.192:6805/34786 -- osd_op(unknown.0.0:18 17.1 17:8049a5cb:::exist:head [writefull 0~5 in=5b] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e57) v8 -- 0x56192217af50 con 0x5619220df280 2024-06-03T08:38:35.815 INFO:tasks.workunit.client.0.smithi088.stdout: api_asio: 2024-06-03T08:38:34.521+0000 7f514d647780 20 client.4490.objecter put_session s=0x5619220df110 osd=5 4 2024-06-03T08:38:35.815 INFO:tasks.workunit.client.0.smithi088.stdout: api_asio: 2024-06-03T08:38:34.521+0000 7f514d647780 5 client.4490.objecter 1 in flight 2024-06-03T08:38:35.815 INFO:tasks.workunit.client.0.smithi088.stdout: api_asio: 2024-06-03T08:38:34.596+0000 7f5132ffd640 10 monclient: tick 2024-06-03T08:38:35.815 INFO:tasks.workunit.client.0.smithi088.stdout: api_asio: 2024-06-03T08:38:34.596+0000 7f5132ffd640 10 monclient: _check_auth_tickets 2024-06-03T08:38:35.815 INFO:tasks.workunit.client.0.smithi088.stdout: api_asio: 2024-06-03T08:38:34.596+0000 7f5132ffd640 20 monclient: _check_auth_rotating not needed by client.admin 2024-06-03T08:38:35.815 INFO:tasks.workunit.client.0.smithi088.stdout: api_asio: 2024-06-03T08:38:35.091+0000 7f5133fff640 1 -- 172.21.15.88:0/320712639 <== osd.5 v1:172.21.15.192:6805/34786 18 ==== osd_map(58..58 src has 1..59) v4 ==== 1038+0+0 (unknown 186967071 0 0) 0x7f513c004c90 con 0x5619220df280 2024-06-03T08:38:35.815 INFO:tasks.workunit.client.0.smithi088.stdout: api_asio: 2024-06-03T08:38:35.091+0000 7f5133fff640 10 client.4490.objecter ms_dispatch 0x561921fa3060 osd_map(58..58 src has 1..59) v4 2024-06-03T08:38:35.815 INFO:tasks.workunit.client.0.smithi088.stdout: api_asio: 2024-06-03T08:38:35.091+0000 7f5133fff640 3 client.4490.objecter handle_osd_map got epochs [58,58] > 57 2024-06-03T08:38:35.816 INFO:tasks.workunit.client.0.smithi088.stdout: api_asio: 2024-06-03T08:38:35.091+0000 7f5133fff640 3 client.4490.objecter handle_osd_map decoding incremental epoch 58 2024-06-03T08:38:35.816 INFO:tasks.workunit.client.0.smithi088.stdout: api_asio: 2024-06-03T08:38:35.091+0000 7f5133fff640 10 client.4490.objecter checking op 18 2024-06-03T08:38:35.816 INFO:tasks.workunit.client.0.smithi088.stdout: api_asio: 2024-06-03T08:38:35.091+0000 7f5133fff640 20 client.4490.objecter _calc_target epoch 58 base exist @17 precalc_pgid 0 pgid 0.0 is_write 2024-06-03T08:38:35.816 INFO:tasks.workunit.client.0.smithi088.stdout: api_asio: 2024-06-03T08:38:35.091+0000 7f5133fff640 20 client.4490.objecter _calc_target target exist @17 -> pgid 17.d3a59201 2024-06-03T08:38:35.816 INFO:tasks.workunit.client.0.smithi088.stdout: api_asio: 2024-06-03T08:38:35.091+0000 7f5133fff640 20 client.4490.objecter dump_active .. 0 homeless 2024-06-03T08:38:35.816 INFO:tasks.workunit.client.0.smithi088.stdout: api_asio: 2024-06-03T08:38:35.091+0000 7f5133fff640 20 client.4490.objecter 18 17.d3a59201 osd.5 exist [writefull 0~5 in=5b] 2024-06-03T08:38:35.816 INFO:tasks.workunit.client.0.smithi088.stdout: api_asio: 2024-06-03T08:38:35.596+0000 7f5132ffd640 10 monclient: tick 2024-06-03T08:38:35.816 INFO:tasks.workunit.client.0.smithi088.stdout: api_asio: 2024-06-03T08:38:35.596+0000 7f5132ffd640 10 monclient: _check_auth_tickets 2024-06-03T08:38:35.816 INFO:tasks.workunit.client.0.smithi088.stdout: api_asio: 2024-06-03T08:38:35.596+0000 7f5132ffd640 20 monclient: _check_auth_rotating not needed by client.admin 2024-06-03T08:38:35.816 INFO:tasks.workunit.client.0.smithi088.stdout: api_asio: 2024-06-03T08:38:35.814+0000 7f514a183640 1 -- 172.21.15.88:0/320712639 <== osd.5 v1:172.21.15.192:6805/34786 19 ==== osd_op_reply(18 exist [writefull 0~5] v58'5 uv5 ondisk = 0) v8 ==== 149+0+0 (unknown 2013495893 0 0) 0x7f513c0043a0 con 0x5619220df280 2024-06-03T08:38:35.816 INFO:tasks.workunit.client.0.smithi088.stdout: api_asio: 2024-06-03T08:38:35.814+0000 7f514a183640 10 client.4490.objecter ms_dispatch 0x561921fa3060 osd_op_reply(18 exist [writefull 0~5] v58'5 uv5 ondisk = 0) v8 2024-06-03T08:38:35.816 INFO:tasks.workunit.client.0.smithi088.stdout: api_asio: 2024-06-03T08:38:35.814+0000 7f514a183640 10 client.4490.objecter in handle_osd_op_reply 2024-06-03T08:38:35.817 INFO:tasks.workunit.client.0.smithi088.stdout: api_asio: 2024-06-03T08:38:35.814+0000 7f514a183640 7 client.4490.objecter handle_osd_op_reply 18 ondisk uv 5 in 17.1 attempt 0 2024-06-03T08:38:35.817 INFO:tasks.workunit.client.0.smithi088.stdout: api_asio: 2024-06-03T08:38:35.814+0000 7f514a183640 10 client.4490.objecter op 0 rval 0 len 0 2024-06-03T08:38:35.817 INFO:tasks.workunit.client.0.smithi088.stdout: api_asio: 2024-06-03T08:38:35.814+0000 7f514a183640 15 client.4490.objecter handle_osd_op_reply completed tid 18 2024-06-03T08:38:35.817 INFO:tasks.workunit.client.0.smithi088.stdout: api_asio: 2024-06-03T08:38:35.814+0000 7f514a183640 15 client.4490.objecter _finish_op 18 2024-06-03T08:38:35.817 INFO:tasks.workunit.client.0.smithi088.stdout: api_asio: 2024-06-03T08:38:35.814+0000 7f514a183640 20 client.4490.objecter put_session s=0x5619220df110 osd=5 4 2024-06-03T08:38:35.817 INFO:tasks.workunit.client.0.smithi088.stdout: api_asio: 2024-06-03T08:38:35.814+0000 7f514a183640 15 client.4490.objecter _session_op_remove 5 18 2024-06-03T08:38:35.817 INFO:tasks.workunit.client.0.smithi088.stdout: api_asio: 2024-06-03T08:38:35.814+0000 7f514a183640 5 client.4490.objecter 0 in flight 2024-06-03T08:38:35.817 INFO:tasks.workunit.client.0.smithi088.stdout: api_asio: 2024-06-03T08:38:35.814+0000 7f514a183640 20 librados: complete_aio_write 0x56192219e2e0 2024-06-03T08:38:35.817 INFO:tasks.workunit.client.0.smithi088.stdout: api_asio: [ OK ] AsioRados.AsyncWriteOperationCallback (1293 ms) 2024-06-03T08:38:36.452 INFO:tasks.ceph.osd.2.smithi088.stderr:2024-06-03T08:38:36.388+0000 12a6c640 -1 osd.2 60 get_health_metrics reporting 1 slow ops, oldest is osd_op(client.4590.0:2 30.6 30.7fc1f406 (undecoded) ondisk+write+known_if_redirected+supports_pool_eio e31) 2024-06-03T08:38:36.949 INFO:tasks.workunit.client.0.smithi088.stdout: api_asio: [ RUN ] AsioRados.AsyncWriteOperationFuture 2024-06-03T08:38:36.949 INFO:tasks.workunit.client.0.smithi088.stdout: api_asio: 2024-06-03T08:38:35.814+0000 7f514d647780 20 librados: queue_aio_write 0x561921f4e5e0 completion 0x56192219e3f0 write_seq 5 2024-06-03T08:38:36.949 INFO:tasks.workunit.client.0.smithi088.stdout: api_asio: 2024-06-03T08:38:35.814+0000 7f514d647780 10 client.4490.objecter _op_submit op 0x56192217a6b0 2024-06-03T08:38:36.949 INFO:tasks.workunit.client.0.smithi088.stdout: api_asio: 2024-06-03T08:38:35.814+0000 7f514d647780 20 client.4490.objecter _calc_target epoch 58 base exist @17 precalc_pgid 0 pgid 0.0 is_write 2024-06-03T08:38:36.949 INFO:tasks.workunit.client.0.smithi088.stdout: api_asio: 2024-06-03T08:38:35.814+0000 7f514d647780 20 client.4490.objecter _calc_target target exist @17 -> pgid 17.d3a59201 2024-06-03T08:38:36.950 INFO:tasks.workunit.client.0.smithi088.stdout: api_asio: 2024-06-03T08:38:35.814+0000 7f514d647780 10 client.4490.objecter _calc_target raw pgid 17.d3a59201 -> actual 17.1 acting [5,6] primary 5 2024-06-03T08:38:36.950 INFO:tasks.workunit.client.0.smithi088.stdout: api_asio: 2024-06-03T08:38:35.814+0000 7f514d647780 20 client.4490.objecter _get_session s=0x5619220df110 osd=5 3 2024-06-03T08:38:36.950 INFO:tasks.workunit.client.0.smithi088.stdout: api_asio: 2024-06-03T08:38:35.814+0000 7f514d647780 10 client.4490.objecter _op_submit oid exist '@17' '@17' [writefull 0~5 in=5b] tid 19 osd.5 2024-06-03T08:38:36.950 INFO:tasks.workunit.client.0.smithi088.stdout: api_asio: 2024-06-03T08:38:35.814+0000 7f514d647780 20 client.4490.objecter get_session s=0x5619220df110 osd=5 3 2024-06-03T08:38:36.950 INFO:tasks.workunit.client.0.smithi088.stdout: api_asio: 2024-06-03T08:38:35.814+0000 7f514d647780 15 client.4490.objecter _session_op_assign 5 19 2024-06-03T08:38:36.950 INFO:tasks.workunit.client.0.smithi088.stdout: api_asio: 2024-06-03T08:38:35.814+0000 7f514d647780 15 client.4490.objecter _send_op 19 to 17.1 on osd.5 2024-06-03T08:38:36.950 INFO:tasks.workunit.client.0.smithi088.stdout: api_asio: 2024-06-03T08:38:35.814+0000 7f514d647780 1 -- 172.21.15.88:0/320712639 --> v1:172.21.15.192:6805/34786 -- osd_op(unknown.0.0:19 17.1 17:8049a5cb:::exist:head [writefull 0~5 in=5b] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e58) v8 -- 0x56192217af50 con 0x5619220df280 2024-06-03T08:38:36.950 INFO:tasks.workunit.client.0.smithi088.stdout: api_asio: 2024-06-03T08:38:35.814+0000 7f514d647780 20 client.4490.objecter put_session s=0x5619220df110 osd=5 4 2024-06-03T08:38:36.950 INFO:tasks.workunit.client.0.smithi088.stdout: api_asio: 2024-06-03T08:38:35.814+0000 7f514d647780 5 client.4490.objecter 1 in flight 2024-06-03T08:38:36.950 INFO:tasks.workunit.client.0.smithi088.stdout: api_asio: 2024-06-03T08:38:36.596+0000 7f5132ffd640 10 monclient: tick 2024-06-03T08:38:36.950 INFO:tasks.workunit.client.0.smithi088.stdout: api_asio: 2024-06-03T08:38:36.596+0000 7f5132ffd640 10 monclient: _check_auth_tickets 2024-06-03T08:38:36.950 INFO:tasks.workunit.client.0.smithi088.stdout: api_asio: 2024-06-03T08:38:36.596+0000 7f5132ffd640 20 monclient: _check_auth_rotating not needed by client.admin 2024-06-03T08:38:36.950 INFO:tasks.workunit.client.0.smithi088.stdout: api_asio: 2024-06-03T08:38:36.596+0000 7f5132ffd640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-06-03T08:38:36.951 INFO:tasks.workunit.client.0.smithi088.stdout: api_asio: 2024-06-03T08:38:36.949+0000 7f5133fff640 1 -- 172.21.15.88:0/320712639 <== osd.5 v1:172.21.15.192:6805/34786 20 ==== osd_map(59..60 src has 1..61) v4 ==== 3366+0+0 (unknown 2458551693 0 0) 0x7f513c004820 con 0x5619220df280 2024-06-03T08:38:36.951 INFO:tasks.workunit.client.0.smithi088.stdout: api_asio: 2024-06-03T08:38:36.949+0000 7f5133fff640 10 client.4490.objecter ms_dispatch 0x561921fa3060 osd_map(59..60 src has 1..61) v4 2024-06-03T08:38:36.951 INFO:tasks.workunit.client.0.smithi088.stdout: api_asio: 2024-06-03T08:38:36.949+0000 7f5133fff640 3 client.4490.objecter handle_osd_map got epochs [59,60] > 58 2024-06-03T08:38:36.951 INFO:tasks.workunit.client.0.smithi088.stdout: api_asio: 2024-06-03T08:38:36.949+0000 7f5133fff640 3 client.4490.objecter handle_osd_map decoding incremental epoch 59 2024-06-03T08:38:36.951 INFO:tasks.workunit.client.0.smithi088.stdout: api_asio: 2024-06-03T08:38:36.949+0000 7f5133fff640 10 client.4490.objecter checking op 19 2024-06-03T08:38:36.951 INFO:tasks.workunit.client.0.smithi088.stdout: api_asio: 2024-06-03T08:38:36.949+0000 7f5133fff640 20 client.4490.objecter _calc_target epoch 59 base exist @17 precalc_pgid 0 pgid 0.0 is_write 2024-06-03T08:38:36.951 INFO:tasks.workunit.client.0.smithi088.stdout: api_asio: 2024-06-03T08:38:36.949+0000 7f5133fff640 20 client.4490.objecter _calc_target target exist @17 -> pgid 17.d3a59201 2024-06-03T08:38:36.951 INFO:tasks.workunit.client.0.smithi088.stdout: api_asio: 2024-06-03T08:38:36.949+0000 7f5133fff640 3 client.4490.objecter handle_osd_map decoding incremental epoch 60 2024-06-03T08:38:36.951 INFO:tasks.workunit.client.0.smithi088.stdout: api_asio: 2024-06-03T08:38:36.949+0000 7f5133fff640 10 client.4490.objecter checking op 19 2024-06-03T08:38:36.951 INFO:tasks.workunit.client.0.smithi088.stdout: api_asio: 2024-06-03T08:38:36.949+0000 7f5133fff640 20 client.4490.objecter _calc_target epoch 60 base exist @17 precalc_pgid 0 pgid 0.0 is_write 2024-06-03T08:38:36.952 INFO:tasks.workunit.client.0.smithi088.stdout: api_asio: 2024-06-03T08:38:36.949+0000 7f5133fff640 20 client.4490.objecter _calc_target target exist @17 -> pgid 17.d3a592 api_service_pp: "distro": "centos", 2024-06-03T08:38:36.952 INFO:tasks.workunit.client.0.smithi088.stdout: api_service_pp: "distro_description": "CentOS Stream 9", 2024-06-03T08:38:36.952 INFO:tasks.workunit.client.0.smithi088.stdout: api_service_pp: "distro_version": "9", 2024-06-03T08:38:36.952 INFO:tasks.workunit.client.0.smithi088.stdout: api_service_pp: "foo": "bar", 2024-06-03T08:38:36.952 INFO:tasks.workunit.client.0.smithi088.stdout: api_service_pp: "hostname": "smithi088", 2024-06-03T08:38:36.952 INFO:tasks.workunit.client.0.smithi088.stdout: api_service_pp: "kernel_description": "#1 SMP PREEMPT_DYNAMIC Sat May 18 20:39:48 UTC 2024", 2024-06-03T08:38:36.952 INFO:tasks.workunit.client.0.smithi088.stdout: api_service_pp: "kernel_version": "5.14.0-452.el9.x86_64", 2024-06-03T08:38:36.952 INFO:tasks.workunit.client.0.smithi088.stdout: api_service_pp: "mem_swap_kb": "0", 2024-06-03T08:38:36.952 INFO:tasks.workunit.client.0.smithi088.stdout: api_service_pp: "mem_total_kb": "32488156", 2024-06-03T08:38:36.952 INFO:tasks.workunit.client.0.smithi088.stdout: api_service_pp: "os": "Linux", 2024-06-03T08:38:36.952 INFO:tasks.workunit.client.0.smithi088.stdout: api_service_pp: "this": "that" 2024-06-03T08:38:36.952 INFO:tasks.workunit.client.0.smithi088.stdout: api_service_pp: }, 2024-06-03T08:38:36.952 INFO:tasks.workunit.client.0.smithi088.stdout: api_service_pp: "task_status": {} 2024-06-03T08:38:36.953 INFO:tasks.workunit.client.0.smithi088.stdout: api_service_pp: } 2024-06-03T08:38:36.953 INFO:tasks.workunit.client.0.smithi088.stdout: api_service_pp: } 2024-06-03T08:38:36.953 INFO:tasks.workunit.client.0.smithi088.stdout: api_service_pp: } 2024-06-03T08:38:36.953 INFO:tasks.workunit.client.0.smithi088.stdout: api_service_pp: } 2024-06-03T08:38:36.953 INFO:tasks.workunit.client.0.smithi088.stdout: api_service_pp: } 2024-06-03T08:38:36.953 INFO:tasks.workunit.client.0.smithi088.stdout: api_service_pp: 2024-06-03T08:38:36.953 INFO:tasks.workunit.client.0.smithi088.stdout: api_service_pp: attempt 1 of 20 2024-06-03T08:38:36.953 INFO:tasks.workunit.client.0.smithi088.stdout: api_service_pp: failed to deregister: 2024-06-03T08:38:36.953 INFO:tasks.workunit.client.0.smithi088.stdout: api_service_pp: { 2024-06-03T08:38:36.953 INFO:tasks.workunit.client.0.smithi088.stdout: api_service_pp: "epoch": 14, 2024-06-03T08:38:36.953 INFO:tasks.workunit.client.0.smithi088.stdout: api_service_pp: "modified": "2024-06-03T08:38:26.627700+0000", 2024-06-03T08:38:36.953 INFO:tasks.workunit.client.0.smithi088.stdout: api_service_pp: "services": { 2024-06-03T08:38:36.953 INFO:tasks.workunit.client.0.smithi088.stdout: api_service_pp: "laundry": { 2024-06-03T08:38:36.953 INFO:tasks.workunit.client.0.smithi088.stdout: api_service_pp: "daemons": { 2024-06-03T08:38:36.954 INFO:tasks.workunit.client.0.smithi088.stdout: api_service_pp: "summary": "", 2024-06-03T08:38:36.954 INFO:tasks.workunit.client.0.smithi088.stdout: api_service_pp: "close-test-pid40818": { 2024-06-03T08:38:36.954 INFO:tasks.workunit.client.0.smithi088.stdout: api_service_pp: "start_epoch": 14, 2024-06-03T08:38:36.954 INFO:tasks.workunit.client.0.smithi088.stdout: api_service_pp: "start_stamp": "2024-06-03T08:38:26.568913+0000", 2024-06-03T08:38:36.954 INFO:tasks.workunit.client.0.smithi088.stdout: api_service_pp: "gid": 4758, 2024-06-03T08:38:36.954 INFO:tasks.workunit.client.0.smithi088.stdout: api_service_pp: "addr": "172.21.15.88:0/890727653", 2024-06-03T08:38:36.954 INFO:tasks.workunit.client.0.smithi088.stdout: api_service_pp: "metadata": { 2024-06-03T08:38:36.954 INFO:tasks.workunit.client.0.smithi088.stdout: api_service_pp: "arch": "x86_64", 2024-06-03T08:38:36.954 INFO:tasks.workunit.client.0.smithi088.stdout: api_service_pp: "ceph_release": "reef", 2024-06-03T08:38:36.954 INFO:tasks.workunit.client.0.smithi088.stdout: api_service_pp: "ceph_version": "ceph version 18.2.2-1389-gf1abf795 (f1abf795f4fc15bf2f30ac90c811a53ab21d5d65) reef (stable)", 2024-06-03T08:38:36.954 INFO:tasks.workunit.client.0.smithi088.stdout: api_service_pp: "ceph_version_short": "18.2.2-1389-gf1abf795", 2024-06-03T08:38:36.954 INFO:tasks.workunit.client.0.smithi088.stdout: api_service_pp: "cpu": "Intel(R) Xeon(R) CPU E5-1620 v4 @ 3.50GHz", 2024-06-03T08:38:36.955 INFO:tasks.workunit.client.0.smithi088.stdout: api_service_pp: "distro": "centos", 2024-06-03T08:38:36.955 INFO:tasks.workunit.client.0.smithi088.stdout: api_service_pp: "distro_description": "CentOS Stream 9", 2024-06-03T08:38:36.955 INFO:tasks.workunit.client.0.smithi088.stdout: api_service_pp: "distro_version": "9", 2024-06-03T08:38:36.955 INFO:tasks.workunit.client.0.smithi088.stdout: api_service_pp: "foo": "bar", 2024-06-03T08:38:36.955 INFO:tasks.workunit.client.0.smithi088.stdout: api_service_pp: "hostname": "smithi088", 2024-06-03T08:38:36.955 INFO:tasks.workunit.client.0.smithi088.stdout: api_service_pp: "kernel_description": "#1 SMP PREEMPT_DYNAMIC Sat May 18 20:39:48 UTC 2024", 2024-06-03T08:38:36.955 INFO:tasks.workunit.client.0.smithi088.stdout: api_service_pp: "kernel_version": "5.14.0-452.el9.x86_64", 2024-06-03T08:38:36.955 INFO:tasks.workunit.client.0.smithi088.stdout: api_service_pp: "mem_swap_kb": "0", 2024-06-03T08:38:36.955 INFO:tasks.workunit.client.0.smithi088.stdout: api_service_pp: "mem_total_kb": "32488156", 2024-06-03T08:38:36.955 INFO:tasks.workunit.client.0.smithi088.stdout: api_service_pp: "os": "Linux", 2024-06-03T08:38:36.955 INFO:tasks.workunit.client.0.smithi088.stdout: api_service_pp: "this": "that" 2024-06-03T08:38:36.955 INFO:tasks.workunit.client.0.smithi088.stdout: api_service_pp: }, 2024-06-03T08:38:36.955 INFO:tasks.workunit.client.0.smithi088.stdout: api_service_pp: "task_status": {} 2024-06-03T08:38:36.955 INFO:tasks.workunit.client.0.smithi088.stdout: api_service_pp: } 2024-06-03T08:38:36.956 INFO:tasks.workunit.client.0.smithi088.stdout: api_service_pp: } 2024-06-03T08:38:36.956 INFO:tasks.workunit.client.0.smithi088.stdout: api_service_pp: } 2024-06-03T08:38:36.956 INFO:tasks.workunit.client.0.smithi088.stdout: api_service_pp: } 2024-06-03T08:38:36.956 INFO:tasks.workunit.client.0.smithi088.stdout: api_service_pp: } 2024-06-03T08:38:36.956 INFO:tasks.workunit.client.0.smithi088.stdout: api_service_pp: 2024-06-03T08:38:36.956 INFO:tasks.workunit.client.0.smithi088.stdout: api_service_pp: attempt 2 of 20 2024-06-03T08:38:36.956 INFO:tasks.workunit.client.0.smithi088.stdout: api_service_pp: failed to deregister: 2024-06-03T08:38:36.956 INFO:tasks.workunit.client.0.smithi088.stdout: api_service_pp: { 2024-06-03T08:38:36.956 INFO:tasks.workunit.client.0.smithi088.stdout: api_service_pp: "epoch": 16, 2024-06-03T08:38:36.956 INFO:tasks.workunit.client.0.smithi088.stdout: api_service_pp: "modified": "2024-06-03T08:38:32.630295+0000", 2024-06-03T08:38:36.956 INFO:tasks.workunit.client.0.smithi088.stdout: api_service_pp: "services": { 2024-06-03T08:38:39.313 INFO:tasks.workunit.client.0.smithi088.stdout: 01 2024-06-03T08:38:39.313 INFO:tasks.workunit.client.0.smithi088.stdout: api_asio: 2024-06-03T08:38:36.949+0000 7f5133fff640 20 client.4490.objecter dump_active .. 0 homeless 2024-06-03T08:38:39.313 INFO:tasks.workunit.client.0.smithi088.stdout: api_asio: 2024-06-03T08:38:36.949+0000 7f5133fff640 20 client.4490.objecter 19 17.d3a59201 osd.5 exist [writefull 0~5 in=5b] 2024-06-03T08:38:39.313 INFO:tasks.workunit.client.0.smithi088.stdout: api_asio: 2024-06-03T08:38:37.596+0000 7f5132ffd640 10 monclient: tick 2024-06-03T08:38:39.313 INFO:tasks.workunit.client.0.smithi088.stdout: api_asio: 2024-06-03T08:38:37.596+0000 7f5132ffd640 10 monclient: _check_auth_tickets 2024-06-03T08:38:39.313 INFO:tasks.workunit.client.0.smithi088.stdout: api_asio: 2024-06-03T08:38:37.596+0000 7f5132ffd640 20 monclient: _check_auth_rotating not needed by client.admin 2024-06-03T08:38:39.313 INFO:tasks.workunit.client.0.smithi088.stdout: api_asio: 2024-06-03T08:38:38.596+0000 7f5132ffd640 10 monclient: tick 2024-06-03T08:38:39.313 INFO:tasks.workunit.client.0.smithi088.stdout: api_asio: 2024-06-03T08:38:38.596+0000 7f5132ffd640 10 monclient: _check_auth_tickets 2024-06-03T08:38:39.313 INFO:tasks.workunit.client.0.smithi088.stdout: api_asio: 2024-06-03T08:38:38.596+0000 7f5132ffd640 20 monclient: _check_auth_rotating not needed by client.admin 2024-06-03T08:38:39.314 INFO:tasks.workunit.client.0.smithi088.stdout: api_asio: 2024-06-03T08:38:38.775+0000 7f514cbe8640 10 client.4490.objecter tick 2024-06-03T08:38:39.314 INFO:tasks.workunit.client.0.smithi088.stdout: api_asio: 2024-06-03T08:38:39.312+0000 7f514a183640 1 -- 172.21.15.88:0/320712639 <== osd.5 v1:172.21.15.192:6805/34786 21 ==== osd_op_reply(19 exist [writefull 0~5] v60'6 uv6 ondisk = 0) v8 ==== 149+0+0 (unknown 1932304559 0 0) 0x7f513c0043a0 con 0x5619220df280 2024-06-03T08:38:39.314 INFO:tasks.workunit.client.0.smithi088.stdout: api_asio: 2024-06-03T08:38:39.312+0000 7f514a183640 10 client.4490.objecter ms_dispatch 0x561921fa3060 osd_op_reply(19 exist [writefull 0~5] v60'6 uv6 ondisk = 0) v8 2024-06-03T08:38:39.314 INFO:tasks.workunit.client.0.smithi088.stdout: api_asio: 2024-06-03T08:38:39.312+0000 7f514a183640 10 client.4490.objecter in handle_osd_op_reply 2024-06-03T08:38:39.314 INFO:tasks.workunit.client.0.smithi088.stdout: api_asio: 2024-06-03T08:38:39.312+0000 7f514a183640 7 client.4490.objecter handle_osd_op_reply 19 ondisk uv 6 in 17.1 attempt 0 2024-06-03T08:38:39.314 INFO:tasks.workunit.client.0.smithi088.stdout: api_asio: 2024-06-03T08:38:39.312+0000 7f514a183640 10 client.4490.objecter op 0 rval 0 len 0 2024-06-03T08:38:39.314 INFO:tasks.workunit.client.0.smithi088.stdout: api_asio: 2024-06-03T08:38:39.312+0000 7f514a183640 15 client.4490.objecter handle_osd_op_reply completed tid 19 2024-06-03T08:38:39.314 INFO:tasks.workunit.client.0.smithi088.stdout: api_asio: 2024-06-03T08:38:39.312+0000 7f514a183640 15 client.4490.objecter _finish_op 19 2024-06-03T08:38:39.314 INFO:tasks.workunit.client.0.smithi088.stdout: api_asio: 2024-06-03T08:38:39.312+0000 7f514a183640 20 client.4490.objecter put_session s=0x5619220df110 osd=5 4 2024-06-03T08:38:39.314 INFO:tasks.workunit.client.0.smithi088.stdout: api_asio: 2024-06-03T08:38:39.312+0000 7f514a183640 15 client.4490.objecter _session_op_remove 5 19 2024-06-03T08:38:39.314 INFO:tasks.workunit.client.0.smithi088.stdout: api_asio: 2024-06-03T08:38:39.312+0000 7f514a183640 5 client.4490.objecter 0 in flight 2024-06-03T08:38:39.314 INFO:tasks.workunit.client.0.smithi088.stdout: api_asio: 2024-06-03T08:38:39.312+0000 7f514a183640 20 librados: complete_aio_write 0x56192219e3f0 2024-06-03T08:38:39.314 INFO:tasks.workunit.client.0.smithi088.stdout: api_asio: [ OK ] AsioRados.AsyncWriteOperationFuture (3498 ms) 2024-06-03T08:38:39.315 INFO:tasks.workunit.client.0.smithi088.stdout: api_asio: [ RUN ] AsioRados.AsyncWriteOperationYield 2024-06-03T08:38:39.315 INFO:tasks.workunit.client.0.smithi088.stdout: api_asio: 2024-06-03T08:38:39.312+0000 7f514d647780 20 librados: queue_aio_write 0x561921f4e5e0 completion 0x56192219e2e0 write_seq 6 2024-06-03T08:38:39.315 INFO:tasks.workunit.client.0.smithi088.stdout: api_asio: 2024-06-03T08:38:39.312+0000 7f514d647780 10 client.4490.objecter _op_submit op 0x56192219da90 2024-06-03T08:38:39.315 INFO:tasks.workunit.client.0.smithi088.stdout: api_asio: 2024-06-03T08:38:39.312+0000 7f514d647780 20 client.4490.objecter _calc_target epoch 60 base exist @17 precalc_pgid 0 pgid 0.0 is_write 2024-06-03T08:38:39.315 INFO:tasks.workunit.client.0.smithi088.stdout: api_asio: 2024-06-03T08:38:39.312+0000 7f514d647780 20 client.4490.objecter _calc_target target exist @17 -> pgid 17.d3a59201 2024-06-03T08:38:39.315 INFO:tasks.workunit.client.0.smithi088.stdout: api_asio: 2024-06-03T08:38:39.312+0000 7f514d647780 10 client.4490.objecter _calc_target raw pgid 17.d3a59201 -> actual 17.1 acting [5,6] primary 5 2024-06-03T08:38:39.315 INFO:tasks.workunit.client.0.smithi088.stdout: api_asio: 2024-06-03T08:38:39.312+0000 7f514d647780 20 client.4490.objecter _get_session s=0x5619220df110 osd=5 3 2024-06-03T08:38:39.315 INFO:tasks.workunit.client.0.smithi088.stdout: api_asio: 2024-06-03T08:38:39.312+0000 7f514d647780 10 client.4490.objecter _op_submit oid exist '@17' '@17' [writefull 0~5 in=5b] tid 20 osd.5 2024-06-03T08:38:39.315 INFO:tasks.workunit.client.0.smithi088.stdout: api_asio: 2024-06-03T08:38:39.312+0000 7f514d647780 20 client.4490.objecter get_session s=0x5619220df110 osd=5 3 2024-06-03T08:38:39.315 INFO:tasks.workunit.client.0.smithi088.stdout: api_asio: 2024-06-03T08:38:39.312+0000 7f514d647780 15 client.4490.objecter _session_op_assign 5 20 2024-06-03T08:38:40.287 INFO:tasks.ceph.osd.7.smithi192.stderr:2024-06-03T08:38:40.270+0000 12a6c640 -1 osd.7 65 get_health_metrics reporting 2 slow ops, oldest is osd_op(client.4536.0:2 21.1a 21:5c6776db:::0.obj:head [write 0~152 in=152b] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e28) 2024-06-03T08:38:40.626 INFO:tasks.workunit.client.0.smithi088.stdout: api_asio: 2024-06-03T08:38:39.312+0000 7f514d647780 15 client.4490.objecter _send_op 20 to 17.1 on osd.5 2024-06-03T08:38:40.626 INFO:tasks.workunit.client.0.smithi088.stdout: api_asio: 2024-06-03T08:38:39.312+0000 7f514d647780 1 -- 172.21.15.88:0/320712639 --> v1:172.21.15.192:6805/34786 -- osd_op(unknown.0.0:20 17.1 17:8049a5cb:::exist:head [writefull 0~5 in=5b] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e60) v8 -- 0x56192217af50 con 0x5619220df280 2024-06-03T08:38:40.626 INFO:tasks.workunit.client.0.smithi088.stdout: api_asio: 2024-06-03T08:38:39.312+0000 7f514d647780 20 client.4490.objecter put_session s=0x5619220df110 osd=5 4 2024-06-03T08:38:40.626 INFO:tasks.workunit.client.0.smithi088.stdout: api_asio: 2024-06-03T08:38:39.312+0000 7f514d647780 5 client.4490.objecter 1 in flight 2024-06-03T08:38:40.627 INFO:tasks.workunit.client.0.smithi088.stdout: api_asio: 2024-06-03T08:38:39.596+0000 7f5132ffd640 10 monclient: tick 2024-06-03T08:38:40.627 INFO:tasks.workunit.client.0.smithi088.stdout: api_asio: 2024-06-03T08:38:39.596+0000 7f5132ffd640 10 monclient: _check_auth_tickets 2024-06-03T08:38:40.627 INFO:tasks.workunit.client.0.smithi088.stdout: api_asio: 2024-06-03T08:38:39.596+0000 7f5132ffd640 20 monclient: _check_auth_rotating not needed by client.admin 2024-06-03T08:38:40.627 INFO:tasks.workunit.client.0.smithi088.stdout: api_asio: 2024-06-03T08:38:40.596+0000 7f5132ffd640 10 monclient: tick 2024-06-03T08:38:40.627 INFO:tasks.workunit.client.0.smithi088.stdout: api_asio: 2024-06-03T08:38:40.596+0000 7f5132ffd640 10 monclient: _check_auth_tickets 2024-06-03T08:38:40.627 INFO:tasks.workunit.client.0.smithi088.stdout: api_asio: 2024-06-03T08:38:40.596+0000 7f5132ffd640 20 monclient: _check_auth_rotating not needed by client.admin 2024-06-03T08:38:40.627 INFO:tasks.workunit.client.0.smithi088.stdout: api_asio: 2024-06-03T08:38:40.623+0000 7f5133fff640 1 -- 172.21.15.88:0/320712639 <== osd.5 v1:172.21.15.192:6805/34786 22 ==== osd_map(61..65 src has 1..65) v4 ==== 4627+0+0 (unknown 3150600910 0 0) 0x7f513c004820 con 0x5619220df280 2024-06-03T08:38:40.627 INFO:tasks.workunit.client.0.smithi088.stdout: api_asio: 2024-06-03T08:38:40.623+0000 7f5133fff640 10 client.4490.objecter ms_dispatch 0x561921fa3060 osd_map(61..65 src has 1..65) v4 2024-06-03T08:38:40.627 INFO:tasks.workunit.client.0.smithi088.stdout: api_asio: 2024-06-03T08:38:40.623+0000 7f5133fff640 3 client.4490.objecter handle_osd_map got epochs [61,65] > 60 2024-06-03T08:38:40.627 INFO:tasks.workunit.client.0.smithi088.stdout: api_asio: 2024-06-03T08:38:40.623+0000 7f5133fff640 3 client.4490.objecter handle_osd_map decoding incremental epoch 61 2024-06-03T08:38:40.627 INFO:tasks.workunit.client.0.smithi088.stdout: api_asio: 2024-06-03T08:38:40.623+0000 7f5133fff640 10 client.4490.objecter checking op 20 2024-06-03T08:38:40.627 INFO:tasks.workunit.client.0.smithi088.stdout: api_asio: 2024-06-03T08:38:40.623+0000 7f5133fff640 20 client.4490.objecter _calc_target epoch 61 base exist @17 precalc_pgid 0 pgid 0.0 is_write 2024-06-03T08:38:40.628 INFO:tasks.workunit.client.0.smithi088.stdout: api_asio: 2024-06-03T08:38:40.623+0000 7f5133fff640 20 client.4490.objecter _calc_target target exist @17 -> pgid 17.d3a59201 2024-06-03T08:38:40.628 INFO:tasks.workunit.client.0.smithi088.stdout: api_asio: 2024-06-03T08:38:40.623+0000 7f5133fff640 3 client.4490.objecter handle_osd_map decoding incremental epoch 62 2024-06-03T08:38:40.628 INFO:tasks.workunit.client.0.smithi088.stdout: api_asio: 2024-06-03T08:38:40.623+0000 7f5133fff640 10 client.4490.objecter checking op 20 2024-06-03T08:38:40.628 INFO:tasks.workunit.client.0.smithi088.stdout: api_asio: 2024-06-03T08:38:40.623+0000 7f5133fff640 20 client.4490.objecter _calc_target epoch 62 base exist @17 precalc_pgid 0 pgid 0.0 is_write 2024-06-03T08:38:40.628 INFO:tasks.workunit.client.0.smithi088.stdout: api_asio: 2024-06-03T08:38:40.623+0000 7f5133fff640 20 client.4490.objecter _calc_target target exist @17 -> pgid 17.d3a59201 2024-06-03T08:38:40.628 INFO:tasks.workunit.client.0.smithi088.stdout: api_asio: 2024-06-03T08:38:40.623+0000 7f5133fff640 3 client.4490.objecter handle_osd_map decoding incremental epoch 63 2024-06-03T08:38:40.628 INFO:tasks.workunit.client.0.smithi088.stdout: api_asio: 2024-06-03T08:38:40.623+0000 7f5133fff640 10 client.4490.objecter checking op 20 2024-06-03T08:38:40.628 INFO:tasks.workunit.client.0.smithi088.stdout: api_asio: 2024-06-03T08:38:40.623+0000 7f5133fff640 20 client.4490.objecter _calc_target epoch 63 base exist @17 precalc_pgid 0 pgid 0.0 is_write 2024-06-03T08:38:40.628 INFO:tasks.workunit.client.0.smithi088.stdout: api_asio: 2024-06-03T08:38:40.623+0000 7f5133fff640 20 client.4490.objecter _calc_target target exist @17 -> pgid 17.d3a59201 2024-06-03T08:38:40.628 INFO:tasks.workunit.client.0.smithi088.stdout: api_asio: 2024-06-03T08:38:40.623+0000 7f5133fff640 3 client.4490.objecter handle_osd_map decoding incremental epoch 64 2024-06-03T08:38:40.628 INFO:tasks.workunit.client.0.smithi088.stdout: api_asio: 2024-06-03T08:38:40.623+0000 7f5133fff640 10 client.4490.objecter checking op 20 2024-06-03T08:38:40.628 INFO:tasks.workunit.client.0.smithi088.stdout: api_asio: 2024-06-03T08:38:40.623+0000 7f5133fff640 20 client.4490.objecter _calc_target epoch 64 base exist @17 precalc_pgid 0 pgid 0.0 is_write 2024-06-03T08:38:40.628 INFO:tasks.workunit.client.0.smithi088.stdout: api_asio: 2024-06-03T08:38:40.623+0000 7f5133fff640 20 client.4490.objecter _calc_target target exist @17 -> pgid 17.d3a59201 2024-06-03T08:38:41.374 INFO:tasks.workunit.client.0.smithi088.stdout: api_asio: 2024-06-03T08:38:40.623+0000 7f5133fff640 3 client.4490.objecter handle_osd_map decoding incremental epo api_pool: Running main() from gmock_main.cc 2024-06-03T08:38:41.374 INFO:tasks.workunit.client.0.smithi088.stdout: api_pool: [==========] Running 9 tests from 1 test suite. 2024-06-03T08:38:41.375 INFO:tasks.workunit.client.0.smithi088.stdout: api_pool: [----------] Global test environment set-up. 2024-06-03T08:38:41.375 INFO:tasks.workunit.client.0.smithi088.stdout: api_pool: [----------] 9 tests from LibRadosPools 2024-06-03T08:38:41.375 INFO:tasks.workunit.client.0.smithi088.stdout: api_pool: [ RUN ] LibRadosPools.PoolList 2024-06-03T08:38:41.375 INFO:tasks.workunit.client.0.smithi088.stdout: api_pool: [ OK ] LibRadosPools.PoolList (5104 ms) 2024-06-03T08:38:41.375 INFO:tasks.workunit.client.0.smithi088.stdout: api_pool: [ RUN ] LibRadosPools.PoolLookup 2024-06-03T08:38:41.375 INFO:tasks.workunit.client.0.smithi088.stdout: api_pool: [ OK ] LibRadosPools.PoolLookup (4042 ms) 2024-06-03T08:38:41.375 INFO:tasks.workunit.client.0.smithi088.stdout: api_pool: [ RUN ] LibRadosPools.PoolLookup2 2024-06-03T08:38:41.375 INFO:tasks.workunit.client.0.smithi088.stdout: api_pool: [ OK ] LibRadosPools.PoolLookup2 (3916 ms) 2024-06-03T08:38:41.375 INFO:tasks.workunit.client.0.smithi088.stdout: api_pool: [ RUN ] LibRadosPools.PoolLookupOtherInstance 2024-06-03T08:38:41.375 INFO:tasks.workunit.client.0.smithi088.stdout: api_pool: [ OK ] LibRadosPools.PoolLookupOtherInstance (3592 ms) 2024-06-03T08:38:41.375 INFO:tasks.workunit.client.0.smithi088.stdout: api_pool: [ RUN ] LibRadosPools.PoolReverseLookupOtherInstance 2024-06-03T08:38:41.375 INFO:tasks.workunit.client.0.smithi088.stdout: api_pool: [ OK ] LibRadosPools.PoolReverseLookupOtherInstance (3605 ms) 2024-06-03T08:38:41.375 INFO:tasks.workunit.client.0.smithi088.stdout: api_pool: [ RUN ] LibRadosPools.PoolDelete 2024-06-03T08:38:41.375 INFO:tasks.workunit.client.0.smithi088.stdout: api_pool: [ OK ] LibRadosPools.PoolDelete (5969 ms) 2024-06-03T08:38:41.376 INFO:tasks.workunit.client.0.smithi088.stdout: api_pool: [ RUN ] LibRadosPools.PoolCreateDelete 2024-06-03T08:38:41.376 INFO:tasks.workunit.client.0.smithi088.stdout: api_pool: [ OK ] LibRadosPools.PoolCreateDelete (8462 ms) 2024-06-03T08:38:41.376 INFO:tasks.workunit.client.0.smithi088.stdout: api_pool: [ RUN ] LibRadosPools.PoolCreateWithCrushRule 2024-06-03T08:38:41.376 INFO:tasks.workunit.client.0.smithi088.stdout: api_pool: [ OK ] LibRadosPools.PoolCreateWithCrushRule (5360 ms) 2024-06-03T08:38:41.376 INFO:tasks.workunit.client.0.smithi088.stdout: api_pool: [ RUN ] LibRadosPools.PoolGetBaseTier 2024-06-03T08:38:41.376 INFO:tasks.workunit.client.0.smithi088.stdout: api_pool: [ OK ] LibRadosPools.PoolGetBaseTier (7893 ms) 2024-06-03T08:38:41.376 INFO:tasks.workunit.client.0.smithi088.stdout: api_pool: [----------] 9 tests from LibRadosPools (47944 ms total) 2024-06-03T08:38:41.376 INFO:tasks.workunit.client.0.smithi088.stdout: api_pool: 2024-06-03T08:38:41.376 INFO:tasks.workunit.client.0.smithi088.stdout: api_pool: [----------] Global test environment tear-down 2024-06-03T08:38:41.376 INFO:tasks.workunit.client.0.smithi088.stdout: api_pool: [==========] 9 tests from 1 test suite ran. (47944 ms total) 2024-06-03T08:38:41.376 INFO:tasks.workunit.client.0.smithi088.stdout: api_pool: [ PASSED ] 9 tests. 2024-06-03T08:38:42.084 INFO:tasks.workunit.client.0.smithi088.stdout: api_service_pp: "laundry": { 2024-06-03T08:38:42.084 INFO:tasks.workunit.client.0.smithi088.stdout: api_service_pp: "daemons": { 2024-06-03T08:38:42.084 INFO:tasks.workunit.client.0.smithi088.stdout: api_service_pp: "summary": "", 2024-06-03T08:38:42.084 INFO:tasks.workunit.client.0.smithi088.stdout: api_service_pp: "close-test-pid40818": { 2024-06-03T08:38:42.084 INFO:tasks.workunit.client.0.smithi088.stdout: api_service_pp: "start_epoch": 16, 2024-06-03T08:38:42.084 INFO:tasks.workunit.client.0.smithi088.stdout: api_service_pp: "start_stamp": "2024-06-03T08:38:31.851443+0000", 2024-06-03T08:38:42.084 INFO:tasks.workunit.client.0.smithi088.stdout: api_service_pp: "gid": 4793, 2024-06-03T08:38:42.085 INFO:tasks.workunit.client.0.smithi088.stdout: api_service_pp: "addr": "172.21.15.88:0/3918888022", 2024-06-03T08:38:42.085 INFO:tasks.workunit.client.0.smithi088.stdout: api_service_pp: "metadata": { 2024-06-03T08:38:42.085 INFO:tasks.workunit.client.0.smithi088.stdout: api_service_pp: "arch": "x86_64", 2024-06-03T08:38:42.085 INFO:tasks.workunit.client.0.smithi088.stdout: api_service_pp: "ceph_release": "reef", 2024-06-03T08:38:42.085 INFO:tasks.workunit.client.0.smithi088.stdout: api_service_pp: "ceph_version": "ceph version 18.2.2-1389-gf1abf795 (f1abf795f4fc15bf2f30ac90c811a53ab21d5d65) reef (stable)", 2024-06-03T08:38:42.085 INFO:tasks.workunit.client.0.smithi088.stdout: api_service_pp: "ceph_version_short": "18.2.2-1389-gf1abf795", 2024-06-03T08:38:42.085 INFO:tasks.workunit.client.0.smithi088.stdout: api_service_pp: "cpu": "Intel(R) Xeon(R) CPU E5-1620 v4 @ 3.50GHz", 2024-06-03T08:38:42.085 INFO:tasks.workunit.client.0.smithi088.stdout: api_service_pp: "distro": "centos", 2024-06-03T08:38:42.085 INFO:tasks.workunit.client.0.smithi088.stdout: api_service_pp: "distro_description": "CentOS Stream 9", 2024-06-03T08:38:42.085 INFO:tasks.workunit.client.0.smithi088.stdout: api_service_pp: "distro_version": "9", 2024-06-03T08:38:42.085 INFO:tasks.workunit.client.0.smithi088.stdout: api_service_pp: "foo": "bar", 2024-06-03T08:38:42.085 INFO:tasks.workunit.client.0.smithi088.stdout: api_service_pp: "hostname": "smithi088", 2024-06-03T08:38:42.085 INFO:tasks.workunit.client.0.smithi088.stdout: api_service_pp: "kernel_description": "#1 SMP PREEMPT_DYNAMIC Sat May 18 20:39:48 UTC 2024", 2024-06-03T08:38:42.085 INFO:tasks.workunit.client.0.smithi088.stdout: api_service_pp: "kernel_version": "5.14.0-452.el9.x86_64", 2024-06-03T08:38:42.086 INFO:tasks.workunit.client.0.smithi088.stdout: api_service_pp: "mem_swap_kb": "0", 2024-06-03T08:38:42.086 INFO:tasks.workunit.client.0.smithi088.stdout: api_service_pp: "mem_total_kb": "32488156", 2024-06-03T08:38:42.086 INFO:tasks.workunit.client.0.smithi088.stdout: api_service_pp: "os": "Linux", 2024-06-03T08:38:42.086 INFO:tasks.workunit.client.0.smithi088.stdout: api_service_pp: "this": "that" 2024-06-03T08:38:42.086 INFO:tasks.workunit.client.0.smithi088.stdout: api_service_pp: }, 2024-06-03T08:38:42.086 INFO:tasks.workunit.client.0.smithi088.stdout: api_service_pp: "task_status": {} 2024-06-03T08:38:42.086 INFO:tasks.workunit.client.0.smithi088.stdout: api_service_pp: } 2024-06-03T08:38:42.086 INFO:tasks.workunit.client.0.smithi088.stdout: api_service_pp: } 2024-06-03T08:38:42.086 INFO:tasks.workunit.client.0.smithi088.stdout: api_service_pp: } 2024-06-03T08:38:42.086 INFO:tasks.workunit.client.0.smithi088.stdout: api_service_pp: } 2024-06-03T08:38:42.086 INFO:tasks.workunit.client.0.smithi088.stdout: api_service_pp: } 2024-06-03T08:38:42.086 INFO:tasks.workunit.client.0.smithi088.stdout: api_service_pp: 2024-06-03T08:38:42.086 INFO:tasks.workunit.client.0.smithi088.stdout: api_service_pp: attempt 3 of 20 2024-06-03T08:38:42.086 INFO:tasks.workunit.client.0.smithi088.stdout: api_service_pp: [ OK ] LibRadosServicePP.Close (21399 ms) 2024-06-03T08:38:42.086 INFO:tasks.workunit.client.0.smithi088.stdout: api_service_pp: [----------] 4 tests from LibRadosServicePP (48487 ms total) 2024-06-03T08:38:42.087 INFO:tasks.workunit.client.0.smithi088.stdout: api_service_pp: 2024-06-03T08:38:42.087 INFO:tasks.workunit.client.0.smithi088.stdout: api_service_pp: [----------] Global test environment tear-down 2024-06-03T08:38:42.087 INFO:tasks.workunit.client.0.smithi088.stdout: api_service_pp: [==========] 4 tests from 1 test suite ran. (48487 ms total) 2024-06-03T08:38:42.087 INFO:tasks.workunit.client.0.smithi088.stdout: api_service_pp: [ PASSED ] 4 tests. 2024-06-03T08:38:44.016 INFO:tasks.workunit.client.0.smithi088.stdout:ch 65 2024-06-03T08:38:44.016 INFO:tasks.workunit.client.0.smithi088.stdout: api_asio: 2024-06-03T08:38:40.623+0000 7f5133fff640 10 client.4490.objecter checking op 20 2024-06-03T08:38:44.017 INFO:tasks.workunit.client.0.smithi088.stdout: api_asio: 2024-06-03T08:38:40.623+0000 7f5133fff640 20 client.4490.objecter _calc_target epoch 65 base exist @17 precalc_pgid 0 pgid 0.0 is_write 2024-06-03T08:38:44.017 INFO:tasks.workunit.client.0.smithi088.stdout: api_asio: 2024-06-03T08:38:40.623+0000 7f5133fff640 20 client.4490.objecter _calc_target target exist @17 -> pgid 17.d3a59201 2024-06-03T08:38:44.017 INFO:tasks.workunit.client.0.smithi088.stdout: api_asio: 2024-06-03T08:38:40.623+0000 7f5133fff640 20 client.4490.objecter dump_active .. 0 homeless 2024-06-03T08:38:44.017 INFO:tasks.workunit.client.0.smithi088.stdout: api_asio: 2024-06-03T08:38:40.623+0000 7f5133fff640 20 client.4490.objecter 20 17.d3a59201 osd.5 exist [writefull 0~5 in=5b] 2024-06-03T08:38:44.017 INFO:tasks.workunit.client.0.smithi088.stdout: api_asio: 2024-06-03T08:38:41.596+0000 7f5132ffd640 10 monclient: tick 2024-06-03T08:38:44.017 INFO:tasks.workunit.client.0.smithi088.stdout: api_asio: 2024-06-03T08:38:41.596+0000 7f5132ffd640 10 monclient: _check_auth_tickets 2024-06-03T08:38:44.017 INFO:tasks.workunit.client.0.smithi088.stdout: api_asio: 2024-06-03T08:38:41.596+0000 7f5132ffd640 20 monclient: _check_auth_rotating not needed by client.admin 2024-06-03T08:38:44.017 INFO:tasks.workunit.client.0.smithi088.stdout: api_asio: 2024-06-03T08:38:42.596+0000 7f5132ffd640 10 monclient: tick 2024-06-03T08:38:44.017 INFO:tasks.workunit.client.0.smithi088.stdout: api_asio: 2024-06-03T08:38:42.596+0000 7f5132ffd640 10 monclient: _check_auth_tickets 2024-06-03T08:38:44.017 INFO:tasks.workunit.client.0.smithi088.stdout: api_asio: 2024-06-03T08:38:42.596+0000 7f5132ffd640 20 monclient: _check_auth_rotating not needed by client.admin 2024-06-03T08:38:44.017 INFO:tasks.workunit.client.0.smithi088.stdout: api_asio: 2024-06-03T08:38:43.596+0000 7f5132ffd640 10 monclient: tick 2024-06-03T08:38:44.017 INFO:tasks.workunit.client.0.smithi088.stdout: api_asio: 2024-06-03T08:38:43.596+0000 7f5132ffd640 10 monclient: _check_auth_tickets 2024-06-03T08:38:44.018 INFO:tasks.workunit.client.0.smithi088.stdout: api_asio: 2024-06-03T08:38:43.596+0000 7f5132ffd640 20 monclient: _check_auth_rotating not needed by client.admin 2024-06-03T08:38:44.018 INFO:tasks.workunit.client.0.smithi088.stdout: api_asio: 2024-06-03T08:38:43.776+0000 7f514cbe8640 10 client.4490.objecter tick 2024-06-03T08:38:44.018 INFO:tasks.workunit.client.0.smithi088.stdout: api_asio: 2024-06-03T08:38:44.016+0000 7f514a183640 1 -- 172.21.15.88:0/320712639 <== osd.5 v1:172.21.15.192:6805/34786 23 ==== osd_op_reply(20 exist [writefull 0~5] v65'7 uv7 ondisk = 0) v8 ==== 149+0+0 (unknown 3411541085 0 0) 0x7f513c0043a0 con 0x5619220df280 2024-06-03T08:38:44.018 INFO:tasks.workunit.client.0.smithi088.stdout: api_asio: 2024-06-03T08:38:44.016+0000 7f514a183640 10 client.4490.objecter ms_dispatch 0x561921fa3060 osd_op_reply(20 exist [writefull 0~5] v65'7 uv7 ondisk = 0) v8 2024-06-03T08:38:44.018 INFO:tasks.workunit.client.0.smithi088.stdout: api_asio: 2024-06-03T08:38:44.016+0000 7f514a183640 10 client.4490.objecter in handle_osd_op_reply 2024-06-03T08:38:44.018 INFO:tasks.workunit.client.0.smithi088.stdout: api_asio: 2024-06-03T08:38:44.016+0000 7f514a183640 7 client.4490.objecter handle_osd_op_reply 20 ondisk uv 7 in 17.1 attempt 0 2024-06-03T08:38:44.018 INFO:tasks.workunit.client.0.smithi088.stdout: api_asio: 2024-06-03T08:38:44.016+0000 7f514a183640 10 client.4490.objecter op 0 rval 0 len 0 2024-06-03T08:38:44.018 INFO:tasks.workunit.client.0.smithi088.stdout: api_asio: 2024-06-03T08:38:44.016+0000 7f514a183640 15 client.4490.objecter handle_osd_op_reply completed tid 20 2024-06-03T08:38:44.018 INFO:tasks.workunit.client.0.smithi088.stdout: api_asio: 2024-06-03T08:38:44.016+0000 7f514a183640 15 client.4490.objecter _finish_op 20 2024-06-03T08:38:44.018 INFO:tasks.workunit.client.0.smithi088.stdout: api_asio: 2024-06-03T08:38:44.016+0000 7f514a183640 20 client.4490.objecter put_session s=0x5619220df110 osd=5 4 2024-06-03T08:38:44.018 INFO:tasks.workunit.client.0.smithi088.stdout: api_asio: 2024-06-03T08:38:44.016+0000 7f514a183640 15 client.4490.objecter _session_op_remove 5 20 2024-06-03T08:38:44.018 INFO:tasks.workunit.client.0.smithi088.stdout: api_asio: 2024-06-03T08:38:44.016+0000 7f514a183640 5 client.4490.objecter 0 in flight 2024-06-03T08:38:44.018 INFO:tasks.workunit.client.0.smithi088.stdout: api_asio: 2024-06-03T08:38:44.016+0000 7f514a183640 20 librados: complete_aio_write 0x56192219e2e0 2024-06-03T08:38:44.019 INFO:tasks.workunit.client.0.smithi088.stdout: api_asio: [ OK ] AsioRados.AsyncWriteOperationYield (4704 ms) 2024-06-03T08:38:44.019 INFO:tasks.workunit.client.0.smithi088.stdout: api_asio: 2024-06-03T08:38:44.016+0000 7f514d647780 10 librados: watch_flush enter 2024-06-03T08:38:44.019 INFO:tasks.workunit.client.0.smithi088.stdout: api_asio: 2024-06-03T08:38:44.016+0000 7f514d647780 10 librados: watch_flush exit 2024-06-03T08:38:44.019 INFO:tasks.workunit.client.0.smithi088.stdout: api_asio: 2024-06-03T08:38:44.016+0000 7f514d647780 10 client.4490.objecter close_session for osd.5 2024-06-03T08:38:44.019 INFO:tasks.workunit.client.0.smithi088.stdout: api_asio: 2024-06-03T08:38:44.016+0000 7f514d647780 1 -- 172.21.15.88:0/320712639 >> v1:172.21.15.192:6805/34786 conn(0x5619220df280 legacy=0x561922177fe0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-06-03T08:38:44.019 INFO:tasks.workunit.client.0.smithi088.stdout: api_asio: 2024-06-03T08:38:44.016+0000 7f514d647780 20 client.4490.objecter put_session s=0x5619220df110 osd=5 1 2024-06-03T08:38:44.020 INFO:tasks.workunit.client.0.smithi088.stdout: api_asio: 2024-06-03T08:38:44.016+0000 7f514d647780 20 client.4490.objecter shutdown clearing up homeless session... 2024-06-03T08:38:44.020 INFO:tasks.workunit.client.0.smithi088.stdout: api_asio: 2024-06-03T08:38:44.016+0000 7f514d647780 10 client.4490.objecter successfully canceled tick 2024-06-03T08:38:44.020 INFO:tasks.workunit.client.0.smithi088.stdout: api_asio: 2024-06-03T08:38:44.016+0000 7f514d647780 1 -- 172.21.15.88:0/320712639 >> v1:172.21.15.192:6800/34229 conn(0x7f513806b750 legacy=0x7f513806dc10 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-06-03T08:38:44.020 INFO:tasks.workunit.client.0.smithi088.stdout: api_asio: 2024-06-03T08:38:44.016+0000 7f514d647780 10 monclient: shutdown 2024-06-03T08:38:44.020 INFO:tasks.workunit.client.0.smithi088.stdout: api_asio: 2024-06-03T08:38:44.016+0000 7f514d647780 20 monclient: shutdown discarding 0 pending message(s) 2024-06-03T08:38:44.020 INFO:tasks.workunit.client.0.smithi088.stdout: api_asio: 2024-06-03T08:38:44.016+0000 7f514d647780 1 -- 172.21.15.88:0/320712639 >> v1:172.21.15.88:6790/0 conn(0x5619220ccfb0 legacy=0x5619220dea00 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-06-03T08:38:44.020 INFO:tasks.workunit.client.0.smithi088.stdout: api_asio: 2024-06-03T08:38:44.016+0000 7f514b185640 1 -- 172.21.15.88:0/320712639 reap_dead start 2024-06-03T08:38:44.020 INFO:tasks.workunit.client.0.smithi088.stdout: api_asio: 2024-06-03T08:38:44.016+0000 7f514d647780 1 -- 172.21.15.88:0/320712639 shutdown_connections 2024-06-03T08:38:44.020 INFO:tasks.workunit.client.0.smithi088.stdout: api_asio: 2024-06-03T08:38:44.016+0000 7f514d647780 1 -- 172.21.15.88:0/320712639 >> 172.21.15.88:0/320712639 conn(0x56192203c260 msgr2=0x561922038580 unknown :-1 s=STATE_NONE l=0).mark_down 2024-06-03T08:38:44.020 INFO:tasks.workunit.client.0.smithi088.stdout: api_asio: 2024-06-03T08:38:44.016+0000 7f514d647780 1 -- 172.21.15.88:0/320712639 shutdown_connections 2024-06-03T08:38:44.021 INFO:tasks.workunit.client.0.smithi088.stdout: api_asio: 2024-06-03T08:38:44.016+0000 7f514d647780 1 -- 172.21.15.88:0/320712639 wait complete. 2024-06-03T08:38:44.021 INFO:tasks.workunit.client.0.smithi088.stdout: api_asio: 2024-06-03T08:38:44.016+0000 7f514d647780 1 librados: shutdown 2024-06-03T08:38:44.021 INFO:tasks.workunit.client.0.smithi088.stdout: api_asio: [----------] 12 tests from AsioRados (17081 ms total) 2024-06-03T08:38:44.021 INFO:tasks.workunit.client.0.smithi088.stdout: api_asio: 2024-06-03T08:38:44.021 INFO:tasks.workunit.client.0.smithi088.stdout: api_asio: [----------] Global test environment tear-down 2024-06-03T08:38:44.021 INFO:tasks.workunit.client.0.smithi088.stdout: api_asio: [==========] 12 tests from 1 test suite ran. (50566 ms total) 2024-06-03T08:38:44.021 INFO:tasks.workunit.client.0.smithi088.stdout: api_asio: [ PASSED ] 12 tests. 2024-06-03T08:38:45.360 INFO:tasks.ceph.mon.a.smithi088.stderr:==00:00:03:40.685 34743== Warning: unimplemented fcntl command: 1036 2024-06-03T08:38:45.364 INFO:tasks.ceph.mon.a.smithi088.stderr:==00:00:03:40.689 34743== Warning: unimplemented fcntl command: 1036 2024-06-03T08:38:45.401 INFO:tasks.ceph.mon.c.smithi088.stderr:==00:00:03:40.726 34749== Warning: unimplemented fcntl command: 1036 2024-06-03T08:38:45.434 INFO:tasks.ceph.mon.b.smithi192.stderr:==00:00:03:40.736 34228== Warning: unimplemented fcntl command: 1036 2024-06-03T08:38:45.437 INFO:tasks.ceph.mon.b.smithi192.stderr:==00:00:03:40.739 34228== Warning: unimplemented fcntl command: 1036 2024-06-03T08:38:45.447 INFO:tasks.ceph.mon.c.smithi088.stderr:==00:00:03:40.773 34749== Warning: unimplemented fcntl command: 1036 2024-06-03T08:38:45.518 INFO:tasks.ceph.mon.b.smithi192.stderr:==00:00:03:40.821 34228== Warning: unimplemented fcntl command: 1036 2024-06-03T08:38:45.558 INFO:tasks.ceph.mon.c.smithi088.stderr:==00:00:03:40.884 34749== Warning: unimplemented fcntl command: 1036 2024-06-03T08:38:45.573 INFO:tasks.ceph.mon.a.smithi088.stderr:==00:00:03:40.898 34743== Warning: unimplemented fcntl command: 1036 2024-06-03T08:39:03.787 INFO:tasks.workunit.client.0.smithi088.stdout: api_cmd_pp: Running main() from gmock_main.cc 2024-06-03T08:39:03.788 INFO:tasks.workunit.client.0.smithi088.stdout: api_cmd_pp: [==========] Running 3 tests from 1 test suite. 2024-06-03T08:39:03.788 INFO:tasks.workunit.client.0.smithi088.stdout: api_cmd_pp: [----------] Global test environment set-up. 2024-06-03T08:39:03.788 INFO:tasks.workunit.client.0.smithi088.stdout: api_cmd_pp: [----------] 3 tests from LibRadosCmd 2024-06-03T08:39:03.788 INFO:tasks.workunit.client.0.smithi088.stdout: api_cmd_pp: [ RUN ] LibRadosCmd.MonDescribePP 2024-06-03T08:39:03.788 INFO:tasks.workunit.client.0.smithi088.stdout: api_cmd_pp: [ OK ] LibRadosCmd.MonDescribePP (692 ms) 2024-06-03T08:39:03.788 INFO:tasks.workunit.client.0.smithi088.stdout: api_cmd_pp: [ RUN ] LibRadosCmd.OSDCmdPP 2024-06-03T08:39:03.788 INFO:tasks.workunit.client.0.smithi088.stdout: api_cmd_pp: [ OK ] LibRadosCmd.OSDCmdPP (1298 ms) 2024-06-03T08:39:03.788 INFO:tasks.workunit.client.0.smithi088.stdout: api_cmd_pp: [ RUN ] LibRadosCmd.PGCmdPP 2024-06-03T08:39:03.788 INFO:tasks.workunit.client.0.smithi088.stdout: api_cmd_pp: [ OK ] LibRadosCmd.PGCmdPP (68227 ms) 2024-06-03T08:39:03.788 INFO:tasks.workunit.client.0.smithi088.stdout: api_cmd_pp: [----------] 3 tests from LibRadosCmd (70217 ms total) 2024-06-03T08:39:03.789 INFO:tasks.workunit.client.0.smithi088.stdout: api_cmd_pp: 2024-06-03T08:39:03.789 INFO:tasks.workunit.client.0.smithi088.stdout: api_cmd_pp: [----------] Global test environment tear-down 2024-06-03T08:39:03.789 INFO:tasks.workunit.client.0.smithi088.stdout: api_cmd_pp: [==========] 3 tests from 1 test suite ran. (70217 ms total) 2024-06-03T08:39:03.789 INFO:tasks.workunit.client.0.smithi088.stdout: api_cmd_pp: [ PASSED ] 3 tests. 2024-06-03T08:39:11.382 INFO:tasks.workunit.client.0.smithi088.stdout: api_c_read_operations: Running main() from gmock_main.cc 2024-06-03T08:39:11.382 INFO:tasks.workunit.client.0.smithi088.stdout: api_c_read_operations: [==========] Running 17 tests from 1 test suite. 2024-06-03T08:39:11.382 INFO:tasks.workunit.client.0.smithi088.stdout: api_c_read_operations: [----------] Global test environment set-up. 2024-06-03T08:39:11.382 INFO:tasks.workunit.client.0.smithi088.stdout: api_c_read_operations: [----------] 17 tests from CReadOpsTest 2024-06-03T08:39:11.382 INFO:tasks.workunit.client.0.smithi088.stdout: api_c_read_operations: [ RUN ] CReadOpsTest.NewDelete 2024-06-03T08:39:11.383 INFO:tasks.workunit.client.0.smithi088.stdout: api_c_read_operations: [ OK ] CReadOpsTest.NewDelete (0 ms) 2024-06-03T08:39:11.383 INFO:tasks.workunit.client.0.smithi088.stdout: api_c_read_operations: [ RUN ] CReadOpsTest.SetOpFlags 2024-06-03T08:39:11.383 INFO:tasks.workunit.client.0.smithi088.stdout: api_c_read_operations: [ OK ] CReadOpsTest.SetOpFlags (29051 ms) 2024-06-03T08:39:11.383 INFO:tasks.workunit.client.0.smithi088.stdout: api_c_read_operations: [ RUN ] CReadOpsTest.AssertExists 2024-06-03T08:39:11.383 INFO:tasks.workunit.client.0.smithi088.stdout: api_c_read_operations: [ OK ] CReadOpsTest.AssertExists (11854 ms) 2024-06-03T08:39:11.383 INFO:tasks.workunit.client.0.smithi088.stdout: api_c_read_operations: [ RUN ] CReadOpsTest.AssertVersion 2024-06-03T08:39:11.383 INFO:tasks.workunit.client.0.smithi088.stdout: api_c_read_operations: [ OK ] CReadOpsTest.AssertVersion (3324 ms) 2024-06-03T08:39:11.383 INFO:tasks.workunit.client.0.smithi088.stdout: api_c_read_operations: [ RUN ] CReadOpsTest.CmpXattr 2024-06-03T08:39:11.383 INFO:tasks.workunit.client.0.smithi088.stdout: api_c_read_operations: [ OK ] CReadOpsTest.CmpXattr (12766 ms) 2024-06-03T08:39:11.383 INFO:tasks.workunit.client.0.smithi088.stdout: api_c_read_operations: [ RUN ] CReadOpsTest.Read 2024-06-03T08:39:11.383 INFO:tasks.workunit.client.0.smithi088.stdout: api_c_read_operations: [ OK ] CReadOpsTest.Read (3100 ms) 2024-06-03T08:39:11.383 INFO:tasks.workunit.client.0.smithi088.stdout: api_c_read_operations: [ RUN ] CReadOpsTest.Checksum 2024-06-03T08:39:11.383 INFO:tasks.workunit.client.0.smithi088.stdout: api_c_read_operations: [ OK ] CReadOpsTest.Checksum (352 ms) 2024-06-03T08:39:11.384 INFO:tasks.workunit.client.0.smithi088.stdout: api_c_read_operations: [ RUN ] CReadOpsTest.RWOrderedRead 2024-06-03T08:39:11.384 INFO:tasks.workunit.client.0.smithi088.stdout: api_c_read_operations: [ OK ] CReadOpsTest.RWOrderedRead (180 ms) 2024-06-03T08:39:11.384 INFO:tasks.workunit.client.0.smithi088.stdout: api_c_read_operations: [ RUN ] CReadOpsTest.ShortRead 2024-06-03T08:39:11.384 INFO:tasks.workunit.client.0.smithi088.stdout: api_c_read_operations: [ OK ] CReadOpsTest.ShortRead (222 ms) 2024-06-03T08:39:11.384 INFO:tasks.workunit.client.0.smithi088.stdout: api_c_read_operations: [ RUN ] CReadOpsTest.Exec 2024-06-03T08:39:11.384 INFO:tasks.workunit.client.0.smithi088.stdout: api_c_read_operations: [ OK ] CReadOpsTest.Exec (105 ms) 2024-06-03T08:39:11.384 INFO:tasks.workunit.client.0.smithi088.stdout: api_c_read_operations: [ RUN ] CReadOpsTest.ExecUserBuf 2024-06-03T08:39:11.384 INFO:tasks.workunit.client.0.smithi088.stdout: api_c_read_operations: [ OK ] CReadOpsTest.ExecUserBuf (87 ms) 2024-06-03T08:39:11.384 INFO:tasks.workunit.client.0.smithi088.stdout: api_c_read_operations: [ RUN ] CReadOpsTest.Stat 2024-06-03T08:39:11.384 INFO:tasks.workunit.client.0.smithi088.stdout: api_c_read_operations: [ OK ] CReadOpsTest.Stat (133 ms) 2024-06-03T08:39:11.384 INFO:tasks.workunit.client.0.smithi088.stdout: api_c_read_operations: [ RUN ] CReadOpsTest.Stat2 2024-06-03T08:39:11.385 INFO:tasks.workunit.client.0.smithi088.stdout: api_c_read_operations: [ OK ] CReadOpsTest.Stat2 (131 ms) 2024-06-03T08:39:11.385 INFO:tasks.workunit.client.0.smithi088.stdout: api_c_read_operations: [ RUN ] CReadOpsTest.Omap 2024-06-03T08:39:11.385 INFO:tasks.workunit.client.0.smithi088.stdout: api_c_read_operations: [ OK ] CReadOpsTest.Omap (371 ms) 2024-06-03T08:39:11.385 INFO:tasks.workunit.client.0.smithi088.stdout: api_c_read_operations: [ RUN ] CReadOpsTest.OmapNuls 2024-06-03T08:39:11.385 INFO:tasks.workunit.client.0.smithi088.stdout: api_c_read_operations: [ OK ] CReadOpsTest.OmapNuls (4680 ms) 2024-06-03T08:39:11.385 INFO:tasks.workunit.client.0.smithi088.stdout: api_c_read_operations: [ RUN ] CReadOpsTest.GetXattrs 2024-06-03T08:39:11.385 INFO:tasks.workunit.client.0.smithi088.stdout: api_c_read_operations: [ OK ] CReadOpsTest.GetXattrs (597 ms) 2024-06-03T08:39:11.385 INFO:tasks.workunit.client.0.smithi088.stdout: api_c_read_operations: [ RUN ] CReadOpsTest.CmpExt 2024-06-03T08:39:11.385 INFO:tasks.workunit.client.0.smithi088.stdout: api_c_read_operations: [ OK ] CReadOpsTest.CmpExt (711 ms) 2024-06-03T08:39:11.385 INFO:tasks.workunit.client.0.smithi088.stdout: api_c_read_operations: [----------] 17 tests from CReadOpsTest (67665 ms total) 2024-06-03T08:39:11.385 INFO:tasks.workunit.client.0.smithi088.stdout: api_c_read_operations: 2024-06-03T08:39:11.385 INFO:tasks.workunit.client.0.smithi088.stdout: api_c_read_operations: [----------] Global test environment tear-down 2024-06-03T08:39:11.385 INFO:tasks.workunit.client.0.smithi088.stdout: api_c_read_operations: [==========] 17 tests from 1 test suite ran. (77741 ms total) 2024-06-03T08:39:11.385 INFO:tasks.workunit.client.0.smithi088.stdout: api_c_read_operations: [ PASSED ] 17 tests. 2024-06-03T08:39:11.958 INFO:tasks.workunit.client.0.smithi088.stdout: api_cmd: Running main() from gmock_main.cc 2024-06-03T08:39:11.958 INFO:tasks.workunit.client.0.smithi088.stdout: api_cmd: [==========] Running 4 tests from 1 test suite. 2024-06-03T08:39:11.958 INFO:tasks.workunit.client.0.smithi088.stdout: api_cmd: [----------] Global test environment set-up. 2024-06-03T08:39:11.958 INFO:tasks.workunit.client.0.smithi088.stdout: api_cmd: [----------] 4 tests from LibRadosCmd 2024-06-03T08:39:11.958 INFO:tasks.workunit.client.0.smithi088.stdout: api_cmd: [ RUN ] LibRadosCmd.MonDescribe 2024-06-03T08:39:11.958 INFO:tasks.workunit.client.0.smithi088.stdout: api_cmd: [ OK ] LibRadosCmd.MonDescribe (667 ms) 2024-06-03T08:39:11.958 INFO:tasks.workunit.client.0.smithi088.stdout: api_cmd: [ RUN ] LibRadosCmd.OSDCmd 2024-06-03T08:39:11.958 INFO:tasks.workunit.client.0.smithi088.stdout: api_cmd: [ OK ] LibRadosCmd.OSDCmd (736 ms) 2024-06-03T08:39:11.958 INFO:tasks.workunit.client.0.smithi088.stdout: api_cmd: [ RUN ] LibRadosCmd.PGCmd 2024-06-03T08:39:11.958 INFO:tasks.workunit.client.0.smithi088.stdout: api_cmd: [ OK ] LibRadosCmd.PGCmd (68821 ms) 2024-06-03T08:39:11.958 INFO:tasks.workunit.client.0.smithi088.stdout: api_cmd: [ RUN ] LibRadosCmd.WatchLog 2024-06-03T08:39:11.959 INFO:tasks.workunit.client.0.smithi088.stdout: api_cmd: got: 2024-06-03T08:39:01.418797+0000 mon.a [INF] from='client.? v1:172.21.15.88:0/762073767' entity='client.admin' cmd='[{"prefix": "osd tier set-overlay", "pool": "LibRadosWatchNotifyPPTests/LibRadosWatchNotifyPP_smithi088-40779-4", "overlaypool": "test-rados-api-smithi088-40779-6"}]': finished 2024-06-03T08:39:11.959 INFO:tasks.workunit.client.0.smithi088.stdout: api_cmd: got: 2024-06-03T08:39:01.573686+0000 mon.a [INF] from='client.4831 ' entity='client.admin' cmd=[{"prefix": "osd pool application enable","pool": "test-rados-api-smithi088-40561-4-cache","app": "rados","yes_i_really_mean_it": true}]: dispatch 2024-06-03T08:39:11.959 INFO:tasks.workunit.client.0.smithi088.stdout: api_cmd: got: 2024-06-03T08:39:02.295397+0000 mon.a [INF] from='client.? v1:172.21.15.88:0/762073767' entity='client.admin' cmd=[{"prefix": "osd tier cache-mode", "pool": "test-rados-api-smithi088-40779-6", "mode": "writeback"}]: dispatch 2024-06-03T08:39:11.959 INFO:tasks.workunit.client.0.smithi088.stdout: api_cmd: got: 2024-06-03T08:39:04.997324+0000 mon.a [INF] from='client.? v1:172.21.15.88:0/3166391489' entity='client.admin' cmd=[{"prefix": "osd erasure-code-profile rm", "name": "testprofile-LibRadosStatEC_smithi088-40680-7"}]: dispatch 2024-06-03T08:39:11.959 INFO:tasks.workunit.client.0.smithi088.stdout: api_cmd: got: 2024-06-03T08:39:05.087047+0000 mon.a [INF] from='client.4831 ' entity='client.admin' cmd='[{"prefix": "osd pool application enable","pool": "test-rados-api-smithi088-40561-4-cache","app": "rados","yes_i_really_mean_it": true}]': finished 2024-06-03T08:39:11.959 INFO:tasks.workunit.client.0.smithi088.stdout: api_cmd: got: 2024-06-03T08:39:05.406877+0000 mon.a [WRN] Health check update: 7 pool(s) do not have an application enabled (POOL_APP_NOT_ENABLED) 2024-06-03T08:39:11.960 INFO:tasks.workunit.client.0.smithi088.stdout: api_cmd: got: 2024-06-03T08:39:05.439119+0000 mon.a [WRN] Health check failed: 1 cache pools are missing hit_sets (CACHE_POOL_NO_HIT_SET) 2024-06-03T08:39:11.960 INFO:tasks.workunit.client.0.smithi088.stdout: api_cmd: got: 2024-06-03T08:39:05.903414+0000 mon.a [INF] from='client.? v1:172.21.15.88:0/3166391489' entity='client.admin' cmd=[{"prefix": "osd crush rule rm", "name":"LibRadosStatEC_smithi088-40680-7"}]: dispatch 2024-06-03T08:39:11.960 INFO:tasks.workunit.client.0.smithi088.stdout: api_cmd: got: 2024-06-03T08:39:07.334324+0000 mon.b [INF] from='client.? v1:172.21.15.88:0/1870456018' entity='client.admin' cmd=[{"prefix": "osd tier add", "pool": "test-rados-api-smithi088-40561-4", "tierpool": "test-rados-api-smithi088-40561-4-cache"}]: dispatch 2024-06-03T08:39:11.960 INFO:tasks.workunit.client.0.smithi088.stdout: api_cmd: got: 2024-06-03T08:39:07.476791+0000 mon.a [INF] from='client.? v1:172.21.15.88:0/3885524826' entity='client.admin' cmd=[{"prefix": "osd erasure-code-profile rm", "name": "testprofile-LibRadosLockECPP_smithi088-40400-10"}]: dispatch 2024-06-03T08:39:11.960 INFO:tasks.workunit.client.0.smithi088.stdout: api_cmd: got: 2024-06-03T08:39:07.616202+0000 mon.a [INF] from='client.? v1:172.21.15.88:0/762073767' entity='client.admin' cmd='[{"prefix": "osd tier cache-mode", "pool": "test-rados-api-smithi088-40779-6", "mode": "writeback"}]': finished 2024-06-03T08:39:11.960 INFO:tasks.workunit.client.0.smithi088.stdout: api_cmd: got: 2024-06-03T08:39:07.794678+0000 mon.a [INF] from='client.? v1:172.21.15.88:0/3885524826' entity='client.admin' cmd=[{"prefix": "osd crush rule rm", "name":"LibRadosLockECPP_smithi088-40400-10"}]: dispatch 2024-06-03T08:39:11.960 INFO:tasks.workunit.client.0.smithi088.stdout: api_cmd: got: 2024-06-03T08:39:08.235475+0000 client.admin [INF] onexx 2024-06-03T08:39:11.961 INFO:tasks.workunit.client.0.smithi088.stdout: api_cmd: got: 2024-06-03T08:39:08.260265+0000 mon.a [INF] from='client.? v1:172.21.15.88:0/917160843' entity='client.admin' cmd=[{"prefix":"log", "logtext":["onexx"]}]: dispatch 2024-06-03T08:39:15.525 INFO:tasks.workunit.client.0.smithi088.stdout: api_cmd: got: 2024-06-03T08:39:08.421857+0000 mon.a [INF] from='client.? v1:172.21.15.88:0/3166 api_watch_notify_pp: Running main() from gmock_main.cc 2024-06-03T08:39:15.526 INFO:tasks.workunit.client.0.smithi088.stdout: api_watch_notify_pp: [==========] Running 16 tests from 2 test suites. 2024-06-03T08:39:15.526 INFO:tasks.workunit.client.0.smithi088.stdout: api_watch_notify_pp: [----------] Global test environment set-up. 2024-06-03T08:39:15.526 INFO:tasks.workunit.client.0.smithi088.stdout: api_watch_notify_pp: [----------] 2 tests from LibRadosWatchNotifyECPP 2024-06-03T08:39:15.526 INFO:tasks.workunit.client.0.smithi088.stdout: api_watch_notify_pp: [ RUN ] LibRadosWatchNotifyECPP.WatchNotify 2024-06-03T08:39:15.526 INFO:tasks.workunit.client.0.smithi088.stdout: api_watch_notify_pp: notify 2024-06-03T08:39:15.526 INFO:tasks.workunit.client.0.smithi088.stdout: api_watch_notify_pp: [ OK ] LibRadosWatchNotifyECPP.WatchNotify (14676 ms) 2024-06-03T08:39:15.526 INFO:tasks.workunit.client.0.smithi088.stdout: api_watch_notify_pp: [ RUN ] LibRadosWatchNotifyECPP.WatchNotifyTimeout 2024-06-03T08:39:15.526 INFO:tasks.workunit.client.0.smithi088.stdout: api_watch_notify_pp: [ OK ] LibRadosWatchNotifyECPP.WatchNotifyTimeout (8726 ms) 2024-06-03T08:39:15.526 INFO:tasks.workunit.client.0.smithi088.stdout: api_watch_notify_pp: [----------] 2 tests from LibRadosWatchNotifyECPP (23402 ms total) 2024-06-03T08:39:15.526 INFO:tasks.workunit.client.0.smithi088.stdout: api_watch_notify_pp: 2024-06-03T08:39:15.526 INFO:tasks.workunit.client.0.smithi088.stdout: api_watch_notify_pp: [----------] 14 tests from LibRadosWatchNotifyPPTests/LibRadosWatchNotifyPP 2024-06-03T08:39:15.526 INFO:tasks.workunit.client.0.smithi088.stdout: api_watch_notify_pp: [ RUN ] LibRadosWatchNotifyPPTests/LibRadosWatchNotifyPP.WatchNotify/0 2024-06-03T08:39:15.527 INFO:tasks.workunit.client.0.smithi088.stdout: api_watch_notify_pp: notify 2024-06-03T08:39:15.527 INFO:tasks.workunit.client.0.smithi088.stdout: api_watch_notify_pp: [ OK ] LibRadosWatchNotifyPPTests/LibRadosWatchNotifyPP.WatchNotify/0 (22184 ms) 2024-06-03T08:39:15.527 INFO:tasks.workunit.client.0.smithi088.stdout: api_watch_notify_pp: [ RUN ] LibRadosWatchNotifyPPTests/LibRadosWatchNotifyPP.WatchNotify/1 2024-06-03T08:39:15.527 INFO:tasks.workunit.client.0.smithi088.stdout: api_watch_notify_pp: notify 2024-06-03T08:39:15.527 INFO:tasks.workunit.client.0.smithi088.stdout: api_watch_notify_pp: notify 2024-06-03T08:39:15.527 INFO:tasks.workunit.client.0.smithi088.stdout: api_watch_notify_pp: [ OK ] LibRadosWatchNotifyPPTests/LibRadosWatchNotifyPP.WatchNotify/1 (18918 ms) 2024-06-03T08:39:15.527 INFO:tasks.workunit.client.0.smithi088.stdout: api_watch_notify_pp: [ RUN ] LibRadosWatchNotifyPPTests/LibRadosWatchNotifyPP.WatchNotifyTimeout/0 2024-06-03T08:39:15.527 INFO:tasks.workunit.client.0.smithi088.stdout: api_watch_notify_pp: [ OK ] LibRadosWatchNotifyPPTests/LibRadosWatchNotifyPP.WatchNotifyTimeout/0 (407 ms) 2024-06-03T08:39:15.527 INFO:tasks.workunit.client.0.smithi088.stdout: api_watch_notify_pp: [ RUN ] LibRadosWatchNotifyPPTests/LibRadosWatchNotifyPP.WatchNotifyTimeout/1 2024-06-03T08:39:15.527 INFO:tasks.workunit.client.0.smithi088.stdout: api_watch_notify_pp: [ OK ] LibRadosWatchNotifyPPTests/LibRadosWatchNotifyPP.WatchNotifyTimeout/1 (1902 ms) 2024-06-03T08:39:15.527 INFO:tasks.workunit.client.0.smithi088.stdout: api_watch_notify_pp: [ RUN ] LibRadosWatchNotifyPPTests/LibRadosWatchNotifyPP.WatchNotify2/0 2024-06-03T08:39:15.527 INFO:tasks.workunit.client.0.smithi088.stdout: api_watch_notify_pp: handle_notify cookie 94492607138880 notify_id 356482285571 notifier_gid 4761 2024-06-03T08:39:15.527 INFO:tasks.workunit.client.0.smithi088.stdout: api_watch_notify_pp: handle_notify cookie 94492607138880 notify_id 356482285572 notifier_gid 4761 2024-06-03T08:39:15.528 INFO:tasks.workunit.client.0.smithi088.stdout: api_watch_notify_pp: [ OK ] LibRadosWatchNotifyPPTests/LibRadosWatchNotifyPP.WatchNotify2/0 (244 ms) 2024-06-03T08:39:15.528 INFO:tasks.workunit.client.0.smithi088.stdout: api_watch_notify_pp: [ RUN ] LibRadosWatchNotifyPPTests/LibRadosWatchNotifyPP.WatchNotify2/1 2024-06-03T08:39:15.528 INFO:tasks.workunit.client.0.smithi088.stdout: api_watch_notify_pp: handle_notify cookie 94492607142176 notify_id 356482285568 notifier_gid 4761 2024-06-03T08:39:15.528 INFO:tasks.workunit.client.0.smithi088.stdout: api_watch_notify_pp: [ OK ] LibRadosWatchNotifyPPTests/LibRadosWatchNotifyPP.WatchNotify2/1 (251 ms) 2024-06-03T08:39:15.528 INFO:tasks.workunit.client.0.smithi088.stdout: api_watch_notify_pp: [ RUN ] LibRadosWatchNotifyPPTests/LibRadosWatchNotifyPP.AioWatchNotify2/0 2024-06-03T08:39:15.528 INFO:tasks.workunit.client.0.smithi088.stdout: api_watch_notify_pp: handle_notify cookie 94492607142176 notify_id 360777252865 notifier_gid 4761 2024-06-03T08:39:15.528 INFO:tasks.workunit.client.0.smithi088.stdout: api_watch_notify_pp: [ OK ] LibRadosWatchNotifyPPTests/LibRadosWatchNotifyPP.AioWatchNotify2/0 (1383 ms) 2024-06-03T08:39:15.528 INFO:tasks.workunit.client.0.smithi088.stdout: api_watch_notify_pp: [ RUN ] LibRadosWatchNotifyPPTests/LibRadosWatchNotifyPP.AioWatchNotify2/1 2024-06-03T08:39:15.528 INFO:tasks.workunit.client.0.smithi088.stdout: api_watch_notify_pp: handle_notify cookie 94492607142096 notify_id 360777252866 notifier_gid 4761 2024-06-03T08:39:15.528 INFO:tasks.workunit.client.0.smithi088.stdout: api_watch_notify_pp: [ OK ] LibRadosWatchNotifyPPTests/LibRadosWatchNotifyPP.AioWatchNotify2/1 (249 ms) 2024-06-03T08:39:15.528 INFO:tasks.workunit.client.0.smithi088.stdout: api_watch_notify_pp: [ RUN ] LibRadosWatchNotifyPPTests/LibRadosWatchNotifyPP.AioNotify/0 2024-06-03T08:39:15.528 INFO:tasks.workunit.client.0.smithi088.stdout: api_watch_notify_pp: handle_notify cookie 94492607142096 notify_id 360777252867 notifier_gid 4761 2024-06-03T08:39:15.528 INFO:tasks.workunit.client.0.smithi088.stdout: api_watch_notify_pp: [ OK ] LibRadosWatchNotifyPPTests/LibRadosWatchNotifyPP.AioNotify/0 (251 ms) 2024-06-03T08:39:15.529 INFO:tasks.workunit.client.0.smithi088.stdout: api_watch_notify_pp: [ RUN ] LibRadosWatchNotifyPPTests/LibRadosWatchNotifyPP.AioNotify/1 2024-06-03T08:39:15.529 INFO:tasks.workunit.client.0.smithi088.stdout: api_watch_notify_pp: handle_notify cookie 94492607142096 notify_id 360777252868 notifier_gid 4761 2024-06-03T08:39:15.529 INFO:tasks.workunit.client.0.smithi088.stdout: api_watch_notify_pp: [ OK ] LibRadosWatchNotifyPPTests/LibRadosWatchNotifyPP.AioNotify/1 (259 ms) 2024-06-03T08:39:15.529 INFO:tasks.workunit.client.0.smithi088.stdout: api_watch_notify_pp: [ RUN ] LibRadosWatchNotifyPPTests/LibRadosWatchNotifyPP.WatchNotify2Timeout/0 2024-06-03T08:39:15.529 INFO:tasks.workunit.client.0.smithi088.stdout: api_watch_notify_pp: trying... 2024-06-03T08:39:16.479 INFO:tasks.workunit.client.0.smithi088.stdout: api_watch_notify_pp: handle_no391489' entity='client.admin' cmd=[{"prefix": "osd erasure-code-profile set", "name": "testprofile-LibRadosStatEC_smithi088-40680-7", "profile": [ "k=2", "m=1", "crush-failure-domain=osd"]}]: dispatch 2024-06-03T08:39:16.480 INFO:tasks.workunit.client.0.smithi088.stdout: api_cmd: got: 2024-06-03T08:39:08.448387+0000 mon.a [INF] from='client.? v1:172.21.15.88:0/3885524826' entity='client.admin' cmd=[{"prefix": "osd erasure-code-profile set", "name": "testprofile-LibRadosLockECPP_smithi088-40400-10", "profile": [ "k=2", "m=1", "crush-failure-domain=osd"]}]: dispatch 2024-06-03T08:39:16.480 INFO:tasks.workunit.client.0.smithi088.stdout: api_cmd: got: 2024-06-03T08:39:08.810481+0000 mon.a [INF] from='client.4831 ' entity='client.admin' cmd=[{"prefix": "osd tier add", "pool": "test-rados-api-smithi088-40561-4", "tierpool": "test-rados-api-smithi088-40561-4-cache"}]: dispatch 2024-06-03T08:39:16.480 INFO:tasks.workunit.client.0.smithi088.stdout: api_cmd: got: 2024-06-03T08:39:09.371109+0000 mon.a [INF] from='client.? v1:172.21.15.88:0/3241258867' entity='client.admin' cmd=[{"prefix": "osd pool application enable","pool": "WaitForSafe_smithi088-40274-3","app": "rados","yes_i_really_mean_it": true}]: dispatch 2024-06-03T08:39:16.480 INFO:tasks.workunit.client.0.smithi088.stdout: api_cmd: got: 2024-06-03T08:39:10.204226+0000 mon.a [INF] from='client.? v1:172.21.15.88:0/3166391489' entity='client.admin' cmd='[{"prefix": "osd erasure-code-profile set", "name": "testprofile-LibRadosStatEC_smithi088-40680-7", "profile": [ "k=2", "m=1", "crush-failure-domain=osd"]}]': finished 2024-06-03T08:39:16.480 INFO:tasks.workunit.client.0.smithi088.stdout: api_cmd: got: 2024-06-03T08:39:10.209580+0000 mon.a [INF] from='client.? v1:172.21.15.88:0/3885524826' entity='client.admin' cmd='[{"prefix": "osd erasure-code-profile set", "name": "testprofile-LibRadosLockECPP_smithi088-40400-10", "profile": [ "k=2", "m=1", "crush-failure-domain=osd"]}]': finished 2024-06-03T08:39:16.480 INFO:tasks.workunit.client.0.smithi088.stdout: api_cmd: got: 2024-06-03T08:39:10.472437+0000 mon.a [WRN] Health check update: 6 pool(s) do not have an application enabled (POOL_APP_NOT_ENABLED) 2024-06-03T08:39:16.480 INFO:tasks.workunit.client.0.smithi088.stdout: api_cmd: got: 2024-06-03T08:39:10.782187+0000 mon.a [INF] from='client.? v1:172.21.15.88:0/3166391489' entity='client.admin' cmd=[{"prefix": "osd pool create", "pool": "LibRadosStatEC_smithi088-40680-7", "pool_type":"erasure", "pg_num":8, "pgp_num":8, "erasure_code_profile":"testprofile-LibRadosStatEC_smithi088-40680-7"}]: dispatch 2024-06-03T08:39:16.480 INFO:tasks.workunit.client.0.smithi088.stdout: api_cmd: got: 2024-06-03T08:39:10.827564+0000 mon.a [INF] from='client.? v1:172.21.15.88:0/1362627465' entity='client.admin' cmd=[{"prefix": "osd erasure-code-profile rm", "name": "testprofile-LibRadosIoEC_smithi088-40312-16"}]: dispatch 2024-06-03T08:39:16.480 INFO:tasks.workunit.client.0.smithi088.stdout: api_cmd: got: 2024-06-03T08:39:10.939041+0000 mon.a [INF] from='client.? v1:172.21.15.88:0/3885524826' entity='client.admin' cmd=[{"prefix": "osd pool create", "pool": "LibRadosLockECPP_smithi088-40400-10", "pool_type":"erasure", "pg_num":8, "pgp_num":8, "erasure_code_profile":"testprofile-LibRadosLockECPP_smithi088-40400-10"}]: dispatch 2024-06-03T08:39:16.480 INFO:tasks.workunit.client.0.smithi088.stdout: api_cmd: got: 2024-06-03T08:39:11.054013+0000 mon.a [INF] from='client.? v1:172.21.15.88:0/770796327' entity='client.admin' cmd=[{"prefix": "osd pool application enable","pool": "test-rados-api-smithi088-40927-4","app": "rados","yes_i_really_mean_it": true}]: dispatch 2024-06-03T08:39:16.480 INFO:tasks.workunit.client.0.smithi088.stdout: api_cmd: got: 2024-06-03T08:39:11.135223+0000 mon.b [INF] from='client.? v1:172.21.15.88:0/3458367578' entity='client.admin' cmd=[{"prefix": "osd erasure-code-profile rm", "name": "testprofile-LibRadosStatECPP_smithi088-40747-7"}]: dispatch 2024-06-03T08:39:16.480 INFO:tasks.workunit.client.0.smithi088.stdout: api_cmd: got: 2024-06-03T08:39:11.558779+0000 mon.a [INF] from='client.? v1:172.21.15.88:0/917160843' entity='client.admin' cmd='[{"prefix":"log", "logtext":["onexx"]}]': finished 2024-06-03T08:39:16.481 INFO:tasks.workunit.client.0.smithi088.stdout: api_cmd: got: 2024-06-03T08:39:11.677389+0000 mon.a [INF] from='client.? v1:172.21.15.88:0/3913941897' entity='client.admin' cmd=[{"prefix": "osd erasure-code-profile rm", "name": "testprofile-LibRadosIoECPP_smithi088-40330-22"}]: dispatch 2024-06-03T08:39:16.481 INFO:tasks.workunit.client.0.smithi088.stdout: api_cmd: got: 2024-06-03T08:39:11.698914+0000 mon.a [INF] from='client.? v1:172.21.15.88:0/2135829355' entity='client.admin' cmd=[{"prefix": "osd erasure-code-profile rm", "name": "testprofile-LibRadosLockEC_smithi088-40432-10"}]: dispatch 2024-06-03T08:39:17.588 INFO:tasks.ceph.mon.a.smithi088.stderr:==00:00:04:12.914 34743== Warning: unimplemented fcntl command: 1036 2024-06-03T08:39:17.620 INFO:tasks.ceph.mon.a.smithi088.stderr:==00:00:04:12.945 34743== Warning: unimplemented fcntl command: 1036 2024-06-03T08:39:17.776 INFO:tasks.ceph.mon.a.smithi088.stderr:==00:00:04:13.101 34743== Warning: unimplemented fcntl command: 1036 2024-06-03T08:39:17.975 INFO:tasks.workunit.client.0.smithi088.stdout: api_list: [==========] Running 11 tests from 3 test suites. 2024-06-03T08:39:17.975 INFO:tasks.workunit.client.0.smithi088.stdout: api_list: [----------] Global test environment set-up. 2024-06-03T08:39:17.975 INFO:tasks.workunit.client.0.smithi088.stdout: api_list: [----------] 7 tests from LibRadosList 2024-06-03T08:39:17.975 INFO:tasks.workunit.client.0.smithi088.stdout: api_list: [ RUN ] LibRadosList.ListObjects 2024-06-03T08:39:17.975 INFO:tasks.workunit.client.0.smithi088.stdout: api_list: [ OK ] LibRadosList.ListObjects (60841 ms) 2024-06-03T08:39:17.975 INFO:tasks.workunit.client.0.smithi088.stdout: api_list: [ RUN ] LibRadosList.ListObjectsZeroInName 2024-06-03T08:39:17.975 INFO:tasks.workunit.client.0.smithi088.stdout: api_list: [ OK ] LibRadosList.ListObjectsZeroInName (795 ms) 2024-06-03T08:39:17.975 INFO:tasks.workunit.client.0.smithi088.stdout: api_list: [ RUN ] LibRadosList.ListObjectsNS 2024-06-03T08:39:17.975 INFO:tasks.workunit.client.0.smithi088.stdout: api_list: myset foo1,foo2,foo3 2024-06-03T08:39:17.975 INFO:tasks.workunit.client.0.smithi088.stdout: api_list: foo1 2024-06-03T08:39:17.976 INFO:tasks.workunit.client.0.smithi088.stdout: api_list: foo2 2024-06-03T08:39:17.976 INFO:tasks.workunit.client.0.smithi088.stdout: api_list: foo3 2024-06-03T08:39:17.976 INFO:tasks.workunit.client.0.smithi088.stdout: api_list: myset foo1,foo4,foo5 2024-06-03T08:39:17.976 INFO:tasks.workunit.client.0.smithi088.stdout: api_list: foo4 2024-06-03T08:39:17.976 INFO:tasks.workunit.client.0.smithi088.stdout: api_list: foo5 2024-06-03T08:39:17.976 INFO:tasks.workunit.client.0.smithi088.stdout: api_list: foo1 2024-06-03T08:39:17.976 INFO:tasks.workunit.client.0.smithi088.stdout: api_list: myset foo6,foo7 2024-06-03T08:39:17.976 INFO:tasks.workunit.client.0.smithi088.stdout: api_list: foo7 2024-06-03T08:39:17.976 INFO:tasks.workunit.client.0.smithi088.stdout: api_list: foo6 2024-06-03T08:39:17.976 INFO:tasks.workunit.client.0.smithi088.stdout: api_list: myset :foo1,:foo2,:foo3,ns1:foo1,ns1:foo4,ns1:foo5,ns2:foo6,ns2:foo7 2024-06-03T08:39:17.976 INFO:tasks.workunit.client.0.smithi088.stdout: api_list: ns1:foo4 2024-06-03T08:39:17.976 INFO:tasks.workunit.client.0.smithi088.stdout: api_list: ns1:foo5 2024-06-03T08:39:17.976 INFO:tasks.workunit.client.0.smithi088.stdout: api_list: ns2:foo7 2024-06-03T08:39:17.977 INFO:tasks.workunit.client.0.smithi088.stdout: api_list: ns2:foo6 2024-06-03T08:39:17.977 INFO:tasks.workunit.client.0.smithi088.stdout: api_list: ns1:foo1 2024-06-03T08:39:17.977 INFO:tasks.workunit.client.0.smithi088.stdout: api_list: :foo1 2024-06-03T08:39:17.977 INFO:tasks.workunit.client.0.smithi088.stdout: api_list: :foo2 2024-06-03T08:39:17.977 INFO:tasks.workunit.client.0.smithi088.stdout: api_list: :foo3 2024-06-03T08:39:17.977 INFO:tasks.workunit.client.0.smithi088.stdout: api_list: [ OK ] LibRadosList.ListObjectsNS (9280 ms) 2024-06-03T08:39:17.977 INFO:tasks.workunit.client.0.smithi088.stdout: api_list: [ RUN ] LibRadosList.ListObjectsStart 2024-06-03T08:39:17.977 INFO:tasks.workunit.client.0.smithi088.stdout: api_list: 1 0 2024-06-03T08:39:17.977 INFO:tasks.workunit.client.0.smithi088.stdout: api_list: 10 0 2024-06-03T08:39:17.977 INFO:tasks.workunit.client.0.smithi088.stdout: api_list: 13 0 2024-06-03T08:39:17.977 INFO:tasks.workunit.client.0.smithi088.stdout: api_list: 7 0 2024-06-03T08:39:17.977 INFO:tasks.workunit.client.0.smithi088.stdout: api_list: 14 0 2024-06-03T08:39:17.977 INFO:tasks.workunit.client.0.smithi088.stdout: api_list: 0 0 2024-06-03T08:39:17.978 INFO:tasks.workunit.client.0.smithi088.stdout: api_list: 15 0 2024-06-03T08:39:17.978 INFO:tasks.workunit.client.0.smithi088.stdout: api_list: 11 0 2024-06-03T08:39:17.978 INFO:tasks.workunit.client.0.smithi088.stdout: api_list: 5 0 2024-06-03T08:39:17.978 INFO:tasks.workunit.client.0.smithi088.stdout: api_list: 8 0 2024-06-03T08:39:17.978 INFO:tasks.workunit.client.0.smithi088.stdout: api_list: 6 0 2024-06-03T08:39:17.978 INFO:tasks.workunit.client.0.smithi088.stdout: api_list: 3 0 2024-06-03T08:39:17.978 INFO:tasks.workunit.client.0.smithi088.stdout: api_list: 4 0 2024-06-03T08:39:17.978 INFO:tasks.workunit.client.0.smithi088.stdout: api_list: 12 0 2024-06-03T08:39:17.978 INFO:tasks.workunit.client.0.smithi088.stdout: api_list: 9 0 2024-06-03T08:39:17.978 INFO:tasks.workunit.client.0.smithi088.stdout: api_list: 2 0 2024-06-03T08:39:17.978 INFO:tasks.workunit.client.0.smithi088.stdout: api_list: have 1 expect one of 0,1,10,11,12,13,14,15,2,3,4,5,6,7,8,9 2024-06-03T08:39:17.978 INFO:tasks.workunit.client.0.smithi088.stdout: api_list: [ OK ] LibRadosList.ListObjectsStart (7277 ms) 2024-06-03T08:39:17.978 INFO:tasks.workunit.client.0.smithi088.stdout: api_list: [ RUN ] LibRadosList.ListObjectsCursor 2024-06-03T08:39:17.979 INFO:tasks.workunit.client.0.smithi088.stdout: api_list: x cursor=MIN 2024-06-03T08:39:17.979 INFO:tasks.workunit.client.0.smithi088.stdout: api_list: > oid=1 cursor=6:02547ec2:::1:head 2024-06-03T08:39:17.979 INFO:tasks.workunit.client.0.smithi088.stdout: api_list: > oid=10 cursor=6:52ea6a34:::10:head 2024-06-03T08:39:17.979 INFO:tasks.workunit.client.0.smithi088.stdout: api_list: > oid=13 cursor=6:566253c9:::13:head 2024-06-03T08:39:17.979 INFO:tasks.workunit.client.0.smithi088.stdout: api_list: > oid=7 cursor=6:5c6b0b28:::7:head 2024-06-03T08:39:17.979 INFO:tasks.workunit.client.0.smithi088.stdout: api_list: > oid=14 cursor=6:62a1935d:::14:head 2024-06-03T08:39:17.979 INFO:tasks.workunit.client.0.smithi088.stdout: api_list: > oid=0 cursor=6:6cac518f:::0:head 2024-06-03T08:39:17.979 INFO:tasks.workunit.client.0.smithi088.stdout: api_list: > oid=15 cursor=6:863748b0:::15:head 2024-06-03T08:39:17.979 INFO:tasks.workunit.client.0.smithi088.stdout: api_list: > oid=11 cursor=6:89d3ae78:::11:head 2024-06-03T08:39:17.979 INFO:tasks.workunit.client.0.smithi088.stdout: api_list: > oid=5 cursor=6:b29083e3:::5:head 2024-06-03T08:39:17.979 INFO:tasks.workunit.client.0.smithi088.stdout: api_list: > oid=8 cursor=6:bd63b0f1:::8:head 2024-06-03T08:39:17.979 INFO:tasks.workunit.client.0.smithi088.stdout: api_list: > oid=6 cursor=6:c4fdafeb:::6:head 2024-06-03T08:39:17.979 INFO:tasks.workunit.client.0.smithi088.stdout: api_list: > oid=3 cursor=6:cfc208b3:::3:head 2024-06-03T08:39:17.979 INFO:tasks.workunit.client.0.smithi088.stdout: api_list: > oid=4 cursor=6:d83876eb:::4:head 2024-06-03T08:39:17.980 INFO:tasks.workunit.client.0.smithi088.stdout: api_list: > oid=12 cursor=6:de5d7c5f:::12:head 2024-06-03T08:39:17.980 INFO:tasks.workunit.client.0.smithi088.stdout: api_list: > oid=9 cursor=6:e960b815:::9:head 2024-06-03T08:39:17.980 INFO:tasks.workunit.client.0.smithi088.stdout: api_list: > oid=2 cursor=6:f905c69b:::2:head 2024-06-03T08:39:17.980 INFO:tasks.workunit.client.0.smithi088.stdout: api_list: FIRST> seek to MIN oid=1 2024-06-03T08:39:17.980 INFO:tasks.workunit.client.0.smithi088.stdout: api_list: : oid=1 cursor=6:02547ec2:::1:head 2024-06-03T08:39:17.980 INFO:tasks.workunit.client.0.smithi088.stdout: api_list: : seek to 6:02547ec2:::1:head 2024-06-03T08:39:17.980 INFO:tasks.workunit.client.0.smithi088.stdout: api_list: > 6:02547ec2:::1:head -> 1 2024-06-03T08:39:17.980 INFO:tasks.workunit.client.0.smithi088.stdout: api_list: : oid=10 cursor=6:52ea6a34:::10:head 2024-06-03T08:39:17.980 INFO:tasks.workunit.client.0.smithi088.stdout: api_list: : seek to 6:52ea6a34:::10:head 2024-06-03T08:39:17.980 INFO:tasks.workunit.client.0.smithi088.stdout: api_list: > 6:52ea6a34:::10:head -> 10 2024-06-03T08:39:17.980 INFO:tasks.workunit.client.0.smithi088.stdout: api_list: : oid=13 cursor=6:566253c9:::13:head 2024-06-03T08:39:17.980 INFO:tasks.workunit.client.0.smithi088.stdout: api_list: : seek to 6:566253c9:::13:head 2024-06-03T08:39:17.980 INFO:tasks.workunit.client.0.smithi088.stdout: api_list: > 6:566253c9:::13:head -> 13 2024-06-03T08:39:17.981 INFO:tasks.workunit.client.0.smithi088.stdout: api_list: : oid=7 cursor=6:5c6b0b28:::7:head 2024-06-03T08:39:17.981 INFO:tasks.workunit.client.0.smithi088.stdout: api_list: : seek to 6:5c6b0b28:::7:head 2024-06-03T08:39:17.981 INFO:tasks.workunit.client.0.smithi088.stdout: api_list: > 6:5c6b0b28:::7:head -> 7 2024-06-03T08:39:18.113 INFO:tasks.ceph.mon.b.smithi192.stderr:==00:00:04:13.415 34228== Warning: unimplemented fcntl command: 1036 2024-06-03T08:39:18.122 INFO:tasks.ceph.mon.c.smithi088.stderr:==00:00:04:13.447 34749== Warning: unimplemented fcntl command: 1036 2024-06-03T08:39:18.152 INFO:tasks.ceph.mon.b.smithi192.stderr:==00:00:04:13.454 34228== Warning: unimplemented fcntl command: 1036 2024-06-03T08:39:18.313 INFO:tasks.ceph.mon.c.smithi088.stderr:==00:00:04:13.639 34749== Warning: unimplemented fcntl command: 1036 2024-06-03T08:39:18.442 INFO:tasks.ceph.mon.b.smithi192.stderr:==00:00:04:13.744 34228== Warning: unimplemented fcntl command: 1036 2024-06-03T08:39:18.453 INFO:tasks.ceph.mon.c.smithi088.stderr:==00:00:04:13.778 34749== Warning: unimplemented fcntl command: 1036 2024-06-03T08:39:19.547 INFO:tasks.workunit.client.0.smithi088.stdout: api_list: : oid=14 cursor=6:62a1935d:::14:h api_cmd: got: 2024-06-03T08:39:12.251057+0000 client.admin [INF] twoxx 2024-06-03T08:39:19.547 INFO:tasks.workunit.client.0.smithi088.stdout: api_cmd: got: 2024-06-03T08:39:12.621069+0000 mon.a [INF] from='client.? v1:172.21.15.88:0/917160843' entity='client.admin' cmd=[{"prefix":"log", "logtext":["twoxx"]}]: dispatch 2024-06-03T08:39:19.547 INFO:tasks.workunit.client.0.smithi088.stdout: api_cmd: got: 2024-06-03T08:39:12.664953+0000 mon.a [INF] from='client.4819 ' entity='client.admin' cmd=[{"prefix": "osd erasure-code-profile rm", "name": "testprofile-LibRadosStatECPP_smithi088-40747-7"}]: dispatch 2024-06-03T08:39:19.547 INFO:tasks.workunit.client.0.smithi088.stdout: api_cmd: got: 2024-06-03T08:39:12.765671+0000 mon.a [INF] from='client.4831 ' entity='client.admin' cmd='[{"prefix": "osd tier add", "pool": "test-rados-api-smithi088-40561-4", "tierpool": "test-rados-api-smithi088-40561-4-cache"}]': finished 2024-06-03T08:39:19.548 INFO:tasks.workunit.client.0.smithi088.stdout: api_cmd: got: 2024-06-03T08:39:12.768122+0000 mon.a [INF] from='client.? v1:172.21.15.88:0/3241258867' entity='client.admin' cmd='[{"prefix": "osd pool application enable","pool": "WaitForSafe_smithi088-40274-3","app": "rados","yes_i_really_mean_it": true}]': finished 2024-06-03T08:39:19.548 INFO:tasks.workunit.client.0.smithi088.stdout: api_cmd: got: 2024-06-03T08:39:13.358457+0000 mon.b [INF] from='client.? v1:172.21.15.88:0/1870456018' entity='client.admin' cmd=[{"prefix": "osd tier set-overlay", "pool": "test-rados-api-smithi088-40561-4", "overlaypool": "test-rados-api-smithi088-40561-4-cache"}]: dispatch 2024-06-03T08:39:19.548 INFO:tasks.workunit.client.0.smithi088.stdout: api_cmd: got: 2024-06-03T08:39:13.417739+0000 mon.a [INF] from='client.? v1:172.21.15.88:0/3913941897' entity='client.admin' cmd=[{"prefix": "osd crush rule rm", "name":"LibRadosIoECPP_smithi088-40330-22"}]: dispatch 2024-06-03T08:39:19.548 INFO:tasks.workunit.client.0.smithi088.stdout: api_cmd: got: 2024-06-03T08:39:13.440270+0000 mon.a [INF] from='client.? v1:172.21.15.88:0/2135829355' entity='client.admin' cmd=[{"prefix": "osd crush rule rm", "name":"LibRadosLockEC_smithi088-40432-10"}]: dispatch 2024-06-03T08:39:19.548 INFO:tasks.workunit.client.0.smithi088.stdout: api_cmd: got: 2024-06-03T08:39:13.678134+0000 mon.a [INF] from='client.? v1:172.21.15.88:0/917160843' entity='client.admin' cmd='[{"prefix":"log", "logtext":["twoxx"]}]': finished 2024-06-03T08:39:19.548 INFO:tasks.workunit.client.0.smithi088.stdout: api_cmd: got: 2024-06-03T08:39:13.753625+0000 mon.a [INF] from='client.? v1:172.21.15.88:0/1362627465' entity='client.admin' cmd=[{"prefix": "osd crush rule rm", "name":"LibRadosIoEC_smithi088-40312-16"}]: dispatch 2024-06-03T08:39:19.548 INFO:tasks.workunit.client.0.smithi088.stdout: api_cmd: got: 2024-06-03T08:39:14.195630+0000 mon.a [INF] from='client.? v1:172.21.15.88:0/3913941897' entity='client.admin' cmd=[{"prefix": "osd erasure-code-profile set", "name": "testprofile-LibRadosIoECPP_smithi088-40330-22", "profile": [ "k=2", "m=1", "crush-failure-domain=osd"]}]: dispatch 2024-06-03T08:39:19.548 INFO:tasks.workunit.client.0.smithi088.stdout: api_cmd: got: 2024-06-03T08:39:14.222354+0000 mon.a [INF] from='client.? v1:172.21.15.88:0/2135829355' entity='client.admin' cmd=[{"prefix": "osd erasure-code-profile set", "name": "testprofile-LibRadosLockEC_smithi088-40432-10", "profile": [ "k=2", "m=1", "crush-failure-domain=osd"]}]: dispatch 2024-06-03T08:39:19.548 INFO:tasks.workunit.client.0.smithi088.stdout: api_cmd: got: 2024-06-03T08:39:14.396759+0000 mon.a [INF] from='client.4831 ' entity='client.admin' cmd=[{"prefix": "osd tier set-overlay", "pool": "test-rados-api-smithi088-40561-4", "overlaypool": "test-rados-api-smithi088-40561-4-cache"}]: dispatch 2024-06-03T08:39:19.548 INFO:tasks.workunit.client.0.smithi088.stdout: api_cmd: got: 2024-06-03T08:39:15.529555+0000 mon.a [WRN] Health check update: 4 pool(s) do not have an application enabled (POOL_APP_NOT_ENABLED) 2024-06-03T08:39:19.548 INFO:tasks.workunit.client.0.smithi088.stdout: api_cmd: got: 2024-06-03T08:39:16.051469+0000 client.admin [INF] threexx 2024-06-03T08:39:19.548 INFO:tasks.workunit.client.0.smithi088.stdout: api_cmd: got: 2024-06-03T08:39:16.236896+0000 mon.a [INF] from='client.? v1:172.21.15.88:0/770796327' entity='client.admin' cmd='[{"prefix": "osd pool application enable","pool": "test-rados-api-smithi088-40927-4","app": "rados","yes_i_really_mean_it": true}]': finished 2024-06-03T08:39:19.549 INFO:tasks.workunit.client.0.smithi088.stdout: api_cmd: got: 2024-06-03T08:39:16.239764+0000 mon.a [INF] from='client.4819 ' entity='client.admin' cmd='[{"prefix": "osd erasure-code-profile rm", "name": "testprofile-LibRadosStatECPP_smithi088-40747-7"}]': finished 2024-06-03T08:39:20.816 INFO:tasks.workunit.client.0.smithi088.stdout: api_cmd: got: 2024-06-03T08:39:16.402029+0000 mon.a [INF] from='client.? v1:172.21.15.88:0/917160843' entity='client.admin' cmd=[{"prefix":"loead 2024-06-03T08:39:20.816 INFO:tasks.workunit.client.0.smithi088.stdout: api_list: : seek to 6:62a1935d:::14:head 2024-06-03T08:39:20.816 INFO:tasks.workunit.client.0.smithi088.stdout: api_list: > 6:62a1935d:::14:head -> 14 2024-06-03T08:39:20.816 INFO:tasks.workunit.client.0.smithi088.stdout: api_list: : oid=0 cursor=6:6cac518f:::0:head 2024-06-03T08:39:20.816 INFO:tasks.workunit.client.0.smithi088.stdout: api_list: : seek to 6:6cac518f:::0:head 2024-06-03T08:39:20.816 INFO:tasks.workunit.client.0.smithi088.stdout: api_list: > 6:6cac518f:::0:head -> 0 2024-06-03T08:39:20.816 INFO:tasks.workunit.client.0.smithi088.stdout: api_list: : oid=15 cursor=6:863748b0:::15:head 2024-06-03T08:39:20.816 INFO:tasks.workunit.client.0.smithi088.stdout: api_list: : seek to 6:863748b0:::15:head 2024-06-03T08:39:20.817 INFO:tasks.workunit.client.0.smithi088.stdout: api_list: > 6:863748b0:::15:head -> 15 2024-06-03T08:39:20.817 INFO:tasks.workunit.client.0.smithi088.stdout: api_list: : oid=11 cursor=6:89d3ae78:::11:head 2024-06-03T08:39:20.817 INFO:tasks.workunit.client.0.smithi088.stdout: api_list: : seek to 6:89d3ae78:::11:head 2024-06-03T08:39:20.817 INFO:tasks.workunit.client.0.smithi088.stdout: api_list: > 6:89d3ae78:::11:head -> 11 2024-06-03T08:39:20.817 INFO:tasks.workunit.client.0.smithi088.stdout: api_list: : oid=5 cursor=6:b29083e3:::5:head 2024-06-03T08:39:20.817 INFO:tasks.workunit.client.0.smithi088.stdout: api_list: : seek to 6:b29083e3:::5:head 2024-06-03T08:39:20.817 INFO:tasks.workunit.client.0.smithi088.stdout: api_list: > 6:b29083e3:::5:head -> 5 2024-06-03T08:39:20.817 INFO:tasks.workunit.client.0.smithi088.stdout: api_list: : oid=8 cursor=6:bd63b0f1:::8:head 2024-06-03T08:39:20.817 INFO:tasks.workunit.client.0.smithi088.stdout: api_list: : seek to 6:bd63b0f1:::8:head 2024-06-03T08:39:20.817 INFO:tasks.workunit.client.0.smithi088.stdout: api_list: > 6:bd63b0f1:::8:head -> 8 2024-06-03T08:39:20.817 INFO:tasks.workunit.client.0.smithi088.stdout: api_list: : oid=6 cursor=6:c4fdafeb:::6:head 2024-06-03T08:39:20.817 INFO:tasks.workunit.client.0.smithi088.stdout: api_list: : seek to 6:c4fdafeb:::6:head 2024-06-03T08:39:20.817 INFO:tasks.workunit.client.0.smithi088.stdout: api_list: > 6:c4fdafeb:::6:head -> 6 2024-06-03T08:39:20.817 INFO:tasks.workunit.client.0.smithi088.stdout: api_list: : oid=3 cursor=6:cfc208b3:::3:head 2024-06-03T08:39:20.818 INFO:tasks.workunit.client.0.smithi088.stdout: api_list: : seek to 6:cfc208b3:::3:head 2024-06-03T08:39:20.818 INFO:tasks.workunit.client.0.smithi088.stdout: api_list: > 6:cfc208b3:::3:head -> 3 2024-06-03T08:39:20.818 INFO:tasks.workunit.client.0.smithi088.stdout: api_list: : oid=4 cursor=6:d83876eb:::4:head 2024-06-03T08:39:20.818 INFO:tasks.workunit.client.0.smithi088.stdout: api_list: : seek to 6:d83876eb:::4:head 2024-06-03T08:39:20.818 INFO:tasks.workunit.client.0.smithi088.stdout: api_list: > 6:d83876eb:::4:head -> 4 2024-06-03T08:39:20.818 INFO:tasks.workunit.client.0.smithi088.stdout: api_list: : oid=12 cursor=6:de5d7c5f:::12:head 2024-06-03T08:39:20.818 INFO:tasks.workunit.client.0.smithi088.stdout: api_list: : seek to 6:de5d7c5f:::12:head 2024-06-03T08:39:20.818 INFO:tasks.workunit.client.0.smithi088.stdout: api_list: > 6:de5d7c5f:::12:head -> 12 2024-06-03T08:39:20.818 INFO:tasks.workunit.client.0.smithi088.stdout: api_list: : oid=9 cursor=6:e960b815:::9:head 2024-06-03T08:39:20.818 INFO:tasks.workunit.client.0.smithi088.stdout: api_list: : seek to 6:e960b815:::9:head 2024-06-03T08:39:20.818 INFO:tasks.workunit.client.0.smithi088.stdout: api_list: > 6:e960b815:::9:head -> 9 2024-06-03T08:39:20.818 INFO:tasks.workunit.client.0.smithi088.stdout: api_list: : oid=2 cursor=6:f905c69b:::2:head 2024-06-03T08:39:20.818 INFO:tasks.workunit.client.0.smithi088.stdout: api_list: : seek to 6:f905c69b:::2:head 2024-06-03T08:39:20.819 INFO:tasks.workunit.client.0.smithi088.stdout: api_list: > 6:f905c69b:::2:head -> 2 2024-06-03T08:39:20.819 INFO:tasks.workunit.client.0.smithi088.stdout: api_list: : seek to 6:52ea6a34:::10:head 2024-06-03T08:39:20.819 INFO:tasks.workunit.client.0.smithi088.stdout: api_list: : cursor()=6:52ea6a34:::10:head expected=6:52ea6a34:::10:head 2024-06-03T08:39:20.819 INFO:tasks.workunit.client.0.smithi088.stdout: api_list: > 6:52ea6a34:::10:head -> 10 2024-06-03T08:39:20.819 INFO:tasks.workunit.client.0.smithi088.stdout: api_list: : entry=10 expected=10 2024-06-03T08:39:20.819 INFO:tasks.workunit.client.0.smithi088.stdout: api_list: : seek to 6:62a1935d:::14:head 2024-06-03T08:39:20.819 INFO:tasks.workunit.client.0.smithi088.stdout: api_list: : cursor()=6:62a1935d:::14:head expected=6:62a1935d:::14:head 2024-06-03T08:39:20.819 INFO:tasks.workunit.client.0.smithi088.stdout: api_list: > 6:62a1935d:::14:head -> 14 2024-06-03T08:39:20.819 INFO:tasks.workunit.client.0.smithi088.stdout: api_list: : entry=14 expected=14 2024-06-03T08:39:20.819 INFO:tasks.workunit.client.0.smithi088.stdout: api_list: : seek to 6:f905c69b:::2:head 2024-06-03T08:39:20.819 INFO:tasks.workunit.client.0.smithi088.stdout: api_list: : cursor()=6:f905c69b:::2:head expected=6:f905c69b:::2:head 2024-06-03T08:39:20.819 INFO:tasks.workunit.client.0.smithi088.stdout: api_list: > 6:f905c69b:::2:head -> 2 2024-06-03T08:39:20.819 INFO:tasks.workunit.client.0.smithi088.stdout: api_list: : entry=2 expected=2 2024-06-03T08:39:20.819 INFO:tasks.workunit.client.0.smithi088.stdout: api_list: : seek to 6:e960b815:::9:head 2024-06-03T08:39:20.820 INFO:tasks.workunit.client.0.smithi088.stdout: api_list: : cursor()=6:e960b815:::9:head expected=6:e960b815:::9:head 2024-06-03T08:39:20.820 INFO:tasks.workunit.client.0.smithi088.stdout: api_list: > 6:e960b815:::9:head -> 9 2024-06-03T08:39:20.820 INFO:tasks.workunit.client.0.smithi088.stdout: api_list: : entry=9 expected=9 2024-06-03T08:39:20.820 INFO:tasks.workunit.client.0.smithi088.stdout: api_list: : seek to 6:de5d7c5f:::12:head 2024-06-03T08:39:20.820 INFO:tasks.workunit.client.0.smithi088.stdout: api_list: : cursor()=6:de5d7c5f:::12:head expected=6:de5d7c5f:::12:head 2024-06-03T08:39:20.820 INFO:tasks.workunit.client.0.smithi088.stdout: api_list: > 6:de5d7c5f:::12:head -> 12 2024-06-03T08:39:20.820 INFO:tasks.workunit.client.0.smithi088.stdout: api_list: : entry=12 expected=12 2024-06-03T08:39:20.820 INFO:tasks.workunit.client.0.smithi088.stdout: api_list: : seek to 6:d83876eb:::4:head 2024-06-03T08:39:20.820 INFO:tasks.workunit.client.0.smithi088.stdout: api_list: : cursor()=6:d83876eb:::4:head expected=6:d83876eb:::4:head 2024-06-03T08:39:20.820 INFO:tasks.workunit.client.0.smithi088.stdout: api_list: > 6:d83876eb:::4:head -> 4 2024-06-03T08:39:20.820 INFO:tasks.workunit.client.0.smithi088.stdout: api_list: : entry=4 expected=4 2024-06-03T08:39:20.820 INFO:tasks.workunit.client.0.smithi088.stdout: api_list: : seek to 6:cfc208b3:::3:head 2024-06-03T08:39:20.820 INFO:tasks.workunit.client.0.smithi088.stdout: api_list: : cursor()=6:cfc208b3:::3:head expected=6:cfc208b3:::3:head 2024-06-03T08:39:20.821 INFO:tasks.workunit.client.0.smithi088.stdout: api_list: > 6:cfc208b3:::3:head -> 3 2024-06-03T08:39:20.821 INFO:tasks.workunit.client.0.smithi088.stdout: api_list: : entry=3 expected=3 2024-06-03T08:39:20.821 INFO:tasks.workunit.client.0.smithi088.stdout: api_list: : seek to 6:c4fdafeb:::6:head 2024-06-03T08:39:20.821 INFO:tasks.workunit.client.0.smithi088.stdout: api_list: : cursor()=6:c4fdafeb:::6:head expected=6:c4fdafeb:::6:head 2024-06-03T08:39:20.821 INFO:tasks.workunit.client.0.smithi088.stdout: api_list: > 6:c4fdafeb:::6:head -> 6 2024-06-03T08:39:20.821 INFO:tasks.workunit.client.0.smithi088.stdout: api_list: : entry=6 expected=6 2024-06-03T08:39:20.821 INFO:tasks.workunit.client.0.smithi088.stdout: api_list: : seek to 6:bd63b0f1:::8:head 2024-06-03T08:39:23.901 INFO:tasks.workunit.client.0.smithi088.stdout: api_stat_pp: Running main() from gmock_main.cc 2024-06-03T08:39:23.901 INFO:tasks.workunit.client.0.smithi088.stdout: api_stat_pp: [==========] Running 9 tests from 2 test suites. 2024-06-03T08:39:23.901 INFO:tasks.workunit.client.0.smithi088.stdout: api_stat_pp: [----------] Global test environment set-up. 2024-06-03T08:39:23.902 INFO:tasks.workunit.client.0.smithi088.stdout: api_stat_pp: [----------] 5 tests from LibRadosStatPP 2024-06-03T08:39:23.902 INFO:tasks.workunit.client.0.smithi088.stdout: api_stat_pp: seed 40747 2024-06-03T08:39:23.902 INFO:tasks.workunit.client.0.smithi088.stdout: api_stat_pp: [ RUN ] LibRadosStatPP.StatPP 2024-06-03T08:39:23.902 INFO:tasks.workunit.client.0.smithi088.stdout: api_stat_pp: [ OK ] LibRadosStatPP.StatPP (20760 ms) 2024-06-03T08:39:23.902 INFO:tasks.workunit.client.0.smithi088.stdout: api_stat_pp: [ RUN ] LibRadosStatPP.Stat2Mtime2PP 2024-06-03T08:39:23.902 INFO:tasks.workunit.client.0.smithi088.stdout: api_stat_pp: [ OK ] LibRadosStatPP.Stat2Mtime2PP (8233 ms) 2024-06-03T08:39:23.902 INFO:tasks.workunit.client.0.smithi088.stdout: api_stat_pp: [ RUN ] LibRadosStatPP.ClusterStatPP 2024-06-03T08:39:23.902 INFO:tasks.workunit.client.0.smithi088.stdout: api_stat_pp: [ OK ] LibRadosStatPP.ClusterStatPP (172 ms) 2024-06-03T08:39:23.902 INFO:tasks.workunit.client.0.smithi088.stdout: api_stat_pp: [ RUN ] LibRadosStatPP.PoolStatPP 2024-06-03T08:39:23.902 INFO:tasks.workunit.client.0.smithi088.stdout: api_stat_pp: [ OK ] LibRadosStatPP.PoolStatPP (11027 ms) 2024-06-03T08:39:23.902 INFO:tasks.workunit.client.0.smithi088.stdout: api_stat_pp: [ RUN ] LibRadosStatPP.StatPPNS 2024-06-03T08:39:23.902 INFO:tasks.workunit.client.0.smithi088.stdout: api_stat_pp: [ OK ] LibRadosStatPP.StatPPNS (7276 ms) 2024-06-03T08:39:23.903 INFO:tasks.workunit.client.0.smithi088.stdout: api_stat_pp: [----------] 5 tests from LibRadosStatPP (47468 ms total) 2024-06-03T08:39:23.903 INFO:tasks.workunit.client.0.smithi088.stdout: api_stat_pp: 2024-06-03T08:39:23.903 INFO:tasks.workunit.client.0.smithi088.stdout: api_stat_pp: [----------] 4 tests from LibRadosStatECPP 2024-06-03T08:39:23.903 INFO:tasks.workunit.client.0.smithi088.stdout: api_stat_pp: [ RUN ] LibRadosStatECPP.StatPP 2024-06-03T08:39:23.903 INFO:tasks.workunit.client.0.smithi088.stdout: api_stat_pp: [ OK ] LibRadosStatECPP.StatPP (12219 ms) 2024-06-03T08:39:23.903 INFO:tasks.workunit.client.0.smithi088.stdout: api_stat_pp: [ RUN ] LibRadosStatECPP.ClusterStatPP 2024-06-03T08:39:23.903 INFO:tasks.workunit.client.0.smithi088.stdout: api_stat_pp: [ OK ] LibRadosStatECPP.ClusterStatPP (655 ms) 2024-06-03T08:39:23.903 INFO:tasks.workunit.client.0.smithi088.stdout: api_stat_pp: [ RUN ] LibRadosStatECPP.PoolStatPP 2024-06-03T08:39:23.903 INFO:tasks.workunit.client.0.smithi088.stdout: api_stat_pp: [ OK ] LibRadosStatECPP.PoolStatPP (1830 ms) 2024-06-03T08:39:23.903 INFO:tasks.workunit.client.0.smithi088.stdout: api_stat_pp: [ RUN ] LibRadosStatECPP.StatPPNS 2024-06-03T08:39:23.903 INFO:tasks.workunit.client.0.smithi088.stdout: api_stat_pp: [ OK ] LibRadosStatECPP.StatPPNS (1299 ms) 2024-06-03T08:39:23.903 INFO:tasks.workunit.client.0.smithi088.stdout: api_stat_pp: [----------] 4 tests from LibRadosStatECPP (16003 ms total) 2024-06-03T08:39:23.903 INFO:tasks.workunit.client.0.smithi088.stdout: api_stat_pp: 2024-06-03T08:39:23.903 INFO:tasks.workunit.client.0.smithi088.stdout: api_stat_pp: [----------] Global test environment tear-down 2024-06-03T08:39:23.904 INFO:tasks.workunit.client.0.smithi088.stdout: api_stat_pp: [==========] 9 tests from 2 test suites ran. (90364 ms total) 2024-06-03T08:39:23.904 INFO:tasks.workunit.client.0.smithi088.stdout: api_stat_pp: [ PASSED ] 9 tests. 2024-06-03T08:39:24.421 INFO:tasks.workunit.client.0.smithi088.stdout:g", "logtext":["threexx"]}]: dispatch 2024-06-03T08:39:24.421 INFO:tasks.workunit.client.0.smithi088.stdout: api_cmd: got: 2024-06-03T08:39:16.994040+0000 mon.a [INF] from='client.? v1:172.21.15.88:0/1362627465' entity='client.admin' cmd=[{"prefix": "osd erasure-code-profile set", "name": "testprofile-LibRadosIoEC_smithi088-40312-16", "profile": [ "k=2", "m=1", "crush-failure-domain=osd"]}]: dispatch 2024-06-03T08:39:24.421 INFO:tasks.workunit.client.0.smithi088.stdout: api_cmd: [ OK ] LibRadosCmd.WatchLog (20630 ms) 2024-06-03T08:39:24.421 INFO:tasks.workunit.client.0.smithi088.stdout: api_cmd: [----------] 4 tests from LibRadosCmd (90854 ms total) 2024-06-03T08:39:24.421 INFO:tasks.workunit.client.0.smithi088.stdout: api_cmd: 2024-06-03T08:39:24.421 INFO:tasks.workunit.client.0.smithi088.stdout: api_cmd: [----------] Global test environment tear-down 2024-06-03T08:39:24.421 INFO:tasks.workunit.client.0.smithi088.stdout: api_cmd: [==========] 4 tests from 1 test suite ran. (90854 ms total) 2024-06-03T08:39:24.421 INFO:tasks.workunit.client.0.smithi088.stdout: api_cmd: [ PASSED ] 4 tests. 2024-06-03T08:39:35.663 INFO:tasks.workunit.client.0.smithi088.stdout: open_pools_parallel: process_1_[41094]: starting. 2024-06-03T08:39:35.663 INFO:tasks.workunit.client.0.smithi088.stdout: open_pools_parallel: process_1_[41094]: creating pool ceph_test_rados_open_pools_parallel.smithi088-41060 2024-06-03T08:39:35.663 INFO:tasks.workunit.client.0.smithi088.stdout: open_pools_parallel: process_1_[41094]: created object 0... 2024-06-03T08:39:35.663 INFO:tasks.workunit.client.0.smithi088.stdout: open_pools_parallel: process_1_[41094]: created object 25... 2024-06-03T08:39:35.664 INFO:tasks.workunit.client.0.smithi088.stdout: open_pools_parallel: process_1_[41094]: created object 49... 2024-06-03T08:39:35.664 INFO:tasks.workunit.client.0.smithi088.stdout: open_pools_parallel: process_1_[41094]: finishing. 2024-06-03T08:39:35.664 INFO:tasks.workunit.client.0.smithi088.stdout: open_pools_parallel: process_1_[41094]: shutting down. 2024-06-03T08:39:35.664 INFO:tasks.workunit.client.0.smithi088.stdout: open_pools_parallel: process_2_[41095]: starting. 2024-06-03T08:39:35.664 INFO:tasks.workunit.client.0.smithi088.stdout: open_pools_parallel: process_2_[41095]: rados_pool_create. 2024-06-03T08:39:35.664 INFO:tasks.workunit.client.0.smithi088.stdout: open_pools_parallel: process_2_[41095]: rados_ioctx_create. 2024-06-03T08:39:35.664 INFO:tasks.workunit.client.0.smithi088.stdout: open_pools_parallel: process_2_[41095]: shutting down. 2024-06-03T08:39:35.664 INFO:tasks.workunit.client.0.smithi088.stdout: open_pools_parallel: ******************************* 2024-06-03T08:39:35.664 INFO:tasks.workunit.client.0.smithi088.stdout: open_pools_parallel: process_3_[51261]: starting. 2024-06-03T08:39:35.664 INFO:tasks.workunit.client.0.smithi088.stdout: open_pools_parallel: process_3_[51261]: creating pool ceph_test_rados_open_pools_parallel.smithi088-41060 2024-06-03T08:39:35.664 INFO:tasks.workunit.client.0.smithi088.stdout: open_pools_parallel: process_3_[51261]: created object 0... 2024-06-03T08:39:35.664 INFO:tasks.workunit.client.0.smithi088.stdout: open_pools_parallel: process_3_[51261]: created object 25... 2024-06-03T08:39:35.664 INFO:tasks.workunit.client.0.smithi088.stdout: open_pools_parallel: process_3_[51261]: created object 49... 2024-06-03T08:39:35.665 INFO:tasks.workunit.client.0.smithi088.stdout: open_pools_parallel: process_3_[51261]: finishing. 2024-06-03T08:39:35.665 INFO:tasks.workunit.client.0.smithi088.stdout: open_pools_parallel: process_3_[51261]: shutting down. 2024-06-03T08:39:35.665 INFO:tasks.workunit.client.0.smithi088.stdout: open_pools_parallel: ******************************* 2024-06-03T08:39:35.665 INFO:tasks.workunit.client.0.smithi088.stdout: open_pools_parallel: process_4_[51262]: starting. 2024-06-03T08:39:35.665 INFO:tasks.workunit.client.0.smithi088.stdout: open_pools_parallel: process_4_[51262]: rados_pool_create. 2024-06-03T08:39:35.665 INFO:tasks.workunit.client.0.smithi088.stdout: open_pools_parallel: process_4_[51262]: rados_ioctx_create. 2024-06-03T08:39:35.665 INFO:tasks.workunit.client.0.smithi088.stdout: open_pools_parallel: process_4_[51262]: shutting down. 2024-06-03T08:39:35.665 INFO:tasks.workunit.client.0.smithi088.stdout: open_pools_parallel: ******************************* 2024-06-03T08:39:35.665 INFO:tasks.workunit.client.0.smithi088.stdout: open_pools_parallel: ******************************* 2024-06-03T08:39:35.665 INFO:tasks.workunit.client.0.smithi088.stdout: open_pools_parallel: ******* SUCCESS ********** 2024-06-03T08:39:36.017 INFO:tasks.workunit.client.0.smithi088.stdout: delete_pools_parallel: process_1_[41023]: starting. 2024-06-03T08:39:36.018 INFO:tasks.workunit.client.0.smithi088.stdout: delete_pools_parallel: process_1_[41023]: creating pool ceph_test_rados_delete_pools_parallel.smithi088-41002 2024-06-03T08:39:36.018 INFO:tasks.workunit.client.0.smithi088.stdout: delete_pools_parallel: process_1_[41023]: created object 0... 2024-06-03T08:39:36.018 INFO:tasks.workunit.client.0.smithi088.stdout: delete_pools_parallel: process_1_[41023]: created object 25... 2024-06-03T08:39:36.018 INFO:tasks.workunit.client.0.smithi088.stdout: delete_pools_parallel: process_1_[41023]: created object 49... 2024-06-03T08:39:36.018 INFO:tasks.workunit.client.0.smithi088.stdout: delete_pools_parallel: process_1_[41023]: finishing. 2024-06-03T08:39:36.018 INFO:tasks.workunit.client.0.smithi088.stdout: delete_pools_parallel: process_1_[41023]: shutting down. 2024-06-03T08:39:36.018 INFO:tasks.workunit.client.0.smithi088.stdout: delete_pools_parallel: process_2_[41024]: starting. 2024-06-03T08:39:36.018 INFO:tasks.workunit.client.0.smithi088.stdout: delete_pools_parallel: process_2_[41024]: deleting pool ceph_test_rados_delete_pools_parallel.smithi088-41002 2024-06-03T08:39:36.018 INFO:tasks.workunit.client.0.smithi088.stdout: delete_pools_parallel: process_2_[41024]: shutting down. 2024-06-03T08:39:36.018 INFO:tasks.workunit.client.0.smithi088.stdout: delete_pools_parallel: ******************************* 2024-06-03T08:39:36.018 INFO:tasks.workunit.client.0.smithi088.stdout: delete_pools_parallel: process_3_[50411]: starting. 2024-06-03T08:39:36.018 INFO:tasks.workunit.client.0.smithi088.stdout: delete_pools_parallel: process_3_[50411]: creating pool ceph_test_rados_delete_pools_parallel.smithi088-41002 2024-06-03T08:39:36.019 INFO:tasks.workunit.client.0.smithi088.stdout: delete_pools_parallel: process_3_[50411]: created object 0... 2024-06-03T08:39:36.019 INFO:tasks.workunit.client.0.smithi088.stdout: delete_pools_parallel: process_3_[50411]: created object 25... 2024-06-03T08:39:36.019 INFO:tasks.workunit.client.0.smithi088.stdout: delete_pools_parallel: process_3_[50411]: created object 49... 2024-06-03T08:39:36.019 INFO:tasks.workunit.client.0.smithi088.stdout: delete_pools_parallel: process_3_[50411]: finishing. 2024-06-03T08:39:36.019 INFO:tasks.workunit.client.0.smithi088.stdout: delete_pools_parallel: process_3_[50411]: shutting down. 2024-06-03T08:39:36.019 INFO:tasks.workunit.client.0.smithi088.stdout: delete_pools_parallel: ******************************* 2024-06-03T08:39:36.019 INFO:tasks.workunit.client.0.smithi088.stdout: delete_pools_parallel: process_5_[50413]: starting. 2024-06-03T08:39:36.019 INFO:tasks.workunit.client.0.smithi088.stdout: delete_pools_parallel: process_5_[50413]: listing objects. 2024-06-03T08:39:36.019 INFO:tasks.workunit.client.0.smithi088.stdout: delete_pools_parallel: process_5_[50413]: listed object 0... 2024-06-03T08:39:36.019 INFO:tasks.workunit.client.0.smithi088.stdout: delete_pools_parallel: process_5_[50413]: listed object 25... 2024-06-03T08:39:36.019 INFO:tasks.workunit.client.0.smithi088.stdout: delete_pools_parallel: process_5_[50413]: saw 50 objects 2024-06-03T08:39:36.019 INFO:tasks.workunit.client.0.smithi088.stdout: delete_pools_parallel: process_5_[50413]: shutting down. 2024-06-03T08:39:36.019 INFO:tasks.workunit.client.0.smithi088.stdout: delete_pools_parallel: ******************************* 2024-06-03T08:39:36.020 INFO:tasks.workunit.client.0.smithi088.stdout: delete_pools_parallel: process_4_[50412]: starting. 2024-06-03T08:39:36.020 INFO:tasks.workunit.client.0.smithi088.stdout: delete_pools_parallel: process_4_[50412]: deleting pool ceph_test_rados_delete_pools_parallel.smithi088-41002 2024-06-03T08:39:36.020 INFO:tasks.workunit.client.0.smithi088.stdout: delete_pools_parallel: process_4_[50412]: shutting down. 2024-06-03T08:39:36.020 INFO:tasks.workunit.client.0.smithi088.stdout: delete_pools_parallel: ******************************* 2024-06-03T08:39:36.020 INFO:tasks.workunit.client.0.smithi088.stdout: delete_pools_parallel: ******************************* 2024-06-03T08:39:36.020 INFO:tasks.workunit.client.0.smithi088.stdout: delete_pools_parallel: ******* SUCCESS ********** 2024-06-03T08:39:38.321 INFO:tasks.workunit.client.0.smithi088.stdout: api_io_pp: Running main() from gmock_main.cc 2024-06-03T08:39:38.322 INFO:tasks.workunit.client.0.smithi088.stdout: api_io_pp: [==========] Running 37 tests from 2 test suites. 2024-06-03T08:39:38.322 INFO:tasks.workunit.client.0.smithi088.stdout: api_io_pp: [----------] Global test environment set-up. 2024-06-03T08:39:38.322 INFO:tasks.workunit.client.0.smithi088.stdout: api_io_pp: [----------] 20 tests from LibRadosIoPP 2024-06-03T08:39:38.322 INFO:tasks.workunit.client.0.smithi088.stdout: api_io_pp: seed 40330 2024-06-03T08:39:38.322 INFO:tasks.workunit.client.0.smithi088.stdout: api_io_pp: [ RUN ] LibRadosIoPP.TooBigPP 2024-06-03T08:39:38.322 INFO:tasks.workunit.client.0.smithi088.stdout: api_io_pp: [ OK ] LibRadosIoPP.TooBigPP (0 ms) 2024-06-03T08:39:38.322 INFO:tasks.workunit.client.0.smithi088.stdout: api_io_pp: [ RUN ] LibRadosIoPP.SimpleWritePP 2024-06-03T08:39:38.322 INFO:tasks.workunit.client.0.smithi088.stdout: api_io_pp: [ OK ] LibRadosIoPP.SimpleWritePP (29892 ms) 2024-06-03T08:39:38.322 INFO:tasks.workunit.client.0.smithi088.stdout: api_io_pp: [ RUN ] LibRadosIoPP.ReadOpPP 2024-06-03T08:39:38.322 INFO:tasks.workunit.client.0.smithi088.stdout: api_io_pp: [ OK ] LibRadosIoPP.ReadOpPP (14523 ms) 2024-06-03T08:39:38.322 INFO:tasks.workunit.client.0.smithi088.stdout: api_io_pp: [ RUN ] LibRadosIoPP.SparseReadOpPP 2024-06-03T08:39:38.323 INFO:tasks.workunit.client.0.smithi088.stdout: api_io_pp: [ OK ] LibRadosIoPP.SparseReadOpPP (2882 ms) 2024-06-03T08:39:38.323 INFO:tasks.workunit.client.0.smithi088.stdout: api_io_pp: [ RUN ] LibRadosIoPP.RoundTripPP 2024-06-03T08:39:38.323 INFO:tasks.workunit.client.0.smithi088.stdout: api_io_pp: [ OK ] LibRadosIoPP.RoundTripPP (1721 ms) 2024-06-03T08:39:38.323 INFO:tasks.workunit.client.0.smithi088.stdout: api_io_pp: [ RUN ] LibRadosIoPP.RoundTripPP2 2024-06-03T08:39:38.323 INFO:tasks.workunit.client.0.smithi088.stdout: api_io_pp: [ OK ] LibRadosIoPP.RoundTripPP2 (3470 ms) 2024-06-03T08:39:38.323 INFO:tasks.workunit.client.0.smithi088.stdout: api_io_pp: [ RUN ] LibRadosIoPP.Checksum 2024-06-03T08:39:38.323 INFO:tasks.workunit.client.0.smithi088.stdout: api_io_pp: [ OK ] LibRadosIoPP.Checksum (1458 ms) 2024-06-03T08:39:38.323 INFO:tasks.workunit.client.0.smithi088.stdout: api_io_pp: [ RUN ] LibRadosIoPP.ReadIntoBufferlist 2024-06-03T08:39:38.323 INFO:tasks.workunit.client.0.smithi088.stdout: api_io_pp: [ OK ] LibRadosIoPP.ReadIntoBufferlist (1170 ms) 2024-06-03T08:39:38.323 INFO:tasks.workunit.client.0.smithi088.stdout: api_io_pp: [ RUN ] LibRadosIoPP.OverlappingWriteRoundTripPP 2024-06-03T08:39:38.323 INFO:tasks.workunit.client.0.smithi088.stdout: api_io_pp: [ OK ] LibRadosIoPP.OverlappingWriteRoundTripPP (945 ms) 2024-06-03T08:39:38.323 INFO:tasks.workunit.client.0.smithi088.stdout: api_io_pp: [ RUN ] LibRadosIoPP.WriteFullRoundTripPP 2024-06-03T08:39:38.323 INFO:tasks.workunit.client.0.smithi088.stdout: api_io_pp: [ OK ] LibRadosIoPP.WriteFullRoundTripPP (123 ms) 2024-06-03T08:39:38.323 INFO:tasks.workunit.client.0.smithi088.stdout: api_io_pp: [ RUN ] LibRadosIoPP.WriteFullRoundTripPP2 2024-06-03T08:39:38.323 INFO:tasks.workunit.client.0.smithi088.stdout: api_io_pp: [ OK ] LibRadosIoPP.WriteFullRoundTripPP2 (33 ms) 2024-06-03T08:39:38.324 INFO:tasks.workunit.client.0.smithi088.stdout: api_io_pp: [ RUN ] LibRadosIoPP.AppendRoundTripPP 2024-06-03T08:39:38.324 INFO:tasks.workunit.client.0.smithi088.stdout: api_io_pp: [ OK ] LibRadosIoPP.AppendRoundTripPP (690 ms) 2024-06-03T08:39:38.324 INFO:tasks.workunit.client.0.smithi088.stdout: api_io_pp: [ RUN ] LibRadosIoPP.TruncTestPP 2024-06-03T08:39:38.324 INFO:tasks.workunit.client.0.smithi088.stdout: api_io_pp: [ OK ] LibRadosIoPP.TruncTestPP (1417 ms) 2024-06-03T08:39:38.324 INFO:tasks.workunit.client.0.smithi088.stdout: api_io_pp: [ RUN ] LibRadosIoPP.RemoveTestPP 2024-06-03T08:39:38.324 INFO:tasks.workunit.client.0.smithi088.stdout: api_io_pp: [ OK ] LibRadosIoPP.RemoveTestPP (1268 ms) 2024-06-03T08:39:38.324 INFO:tasks.workunit.client.0.smithi088.stdout: api_io_pp: [ RUN ] LibRadosIoPP.XattrsRoundTripPP 2024-06-03T08:39:38.324 INFO:tasks.workunit.client.0.smithi088.stdout: api_io_pp: [ OK ] LibRadosIoPP.XattrsRoundTripPP (675 ms) 2024-06-03T08:39:38.324 INFO:tasks.workunit.client.0.smithi088.stdout: api_io_pp: [ RUN ] LibRadosIoPP.RmXattrPP 2024-06-03T08:39:38.324 INFO:tasks.workunit.client.0.smithi088.stdout: api_io_pp: [ OK ] LibRadosIoPP.RmXattrPP (446 ms) 2024-06-03T08:39:38.324 INFO:tasks.workunit.client.0.smithi088.stdout: api_io_pp: [ RUN ] LibRadosIoPP.XattrListPP 2024-06-03T08:39:38.324 INFO:tasks.workunit.client.0.smithi088.stdout: api_io_pp: [ OK ] LibRadosIoPP.XattrListPP (187 ms) 2024-06-03T08:39:38.324 INFO:tasks.workunit.client.0.smithi088.stdout: api_io_pp: [ RUN ] LibRadosIoPP.CmpExtPP 2024-06-03T08:39:38.324 INFO:tasks.workunit.client.0.smithi088.stdout: api_io_pp: [ OK ] LibRadosIoPP.CmpExtPP (122 ms) 2024-06-03T08:39:38.324 INFO:tasks.workunit.client.0.smithi088.stdout: api_io_pp: [ RUN ] LibRadosIoPP.CmpExtDNEPP 2024-06-03T08:39:38.325 INFO:tasks.workunit.client.0.smithi088.stdout: api_io_pp: [ OK ] LibRadosIoPP.CmpExtDNEPP (42 ms) 2024-06-03T08:39:38.325 INFO:tasks.workunit.client.0.smithi088.stdout: api_io_pp: [ RUN ] LibRadosIoPP.CmpExtMismatchPP 2024-06-03T08:39:38.325 INFO:tasks.workunit.client.0.smithi088.stdout: api_io_pp: [ OK ] LibRadosIoPP.CmpExtMismatchPP (131 ms) 2024-06-03T08:39:38.325 INFO:tasks.workunit.client.0.smithi088.stdout: api_io_pp: [----------] 20 tests from LibRadosIoPP (61195 ms total) 2024-06-03T08:39:38.325 INFO:tasks.workunit.client.0.smithi088.stdout: api_io_pp: 2024-06-03T08:39:38.325 INFO:tasks.workunit.client.0.smithi088.stdout: api_io_pp: [----------] 17 tests from LibRadosIoECPP 2024-06-03T08:39:38.325 INFO:tasks.workunit.client.0.smithi088.stdout: api_io_pp: [ RUN ] LibRadosIoECPP.SimpleWritePP 2024-06-03T08:39:38.325 INFO:tasks.workunit.client.0.smithi088.stdout: api_io_pp: [ OK ] LibRadosIoECPP.SimpleWritePP (6700 ms) 2024-06-03T08:39:38.325 INFO:tasks.workunit.client.0.smithi088.stdout: api_io_pp: [ RUN ] LibRadosIoECPP.ReadOpPP 2024-06-03T08:39:38.325 INFO:tasks.workunit.client.0.smithi088.stdout: api_io_pp: [ OK ] LibRadosIoECPP.ReadOpPP (2765 ms) 2024-06-03T08:39:38.325 INFO:tasks.workunit.client.0.smithi088.stdout: api_io_pp: [ RUN ] LibRadosIoECPP.SparseReadOpPP 2024-06-03T08:39:38.325 INFO:tasks.workunit.client.0.smithi088.stdout: api_io_pp: [ OK ] LibRadosIoECPP.SparseReadOpPP (386 ms) 2024-06-03T08:39:38.325 INFO:tasks.workunit.client.0.smithi088.stdout: api_io_pp: [ RUN ] LibRadosIoECPP.RoundTripPP 2024-06-03T08:39:38.325 INFO:tasks.workunit.client.0.smithi088.stdout: api_io_pp: [ OK ] LibRadosIoECPP.RoundTripPP (111 ms) 2024-06-03T08:39:38.326 INFO:tasks.workunit.client.0.smithi088.stdout: api_io_pp: [ RUN ] LibRadosIoECPP.RoundTripPP2 2024-06-03T08:39:42.594 INFO:tasks.workunit.client.0.smithi088.stdout: api_io_pp: [ OK ] LibRadosIoECPP.RoundTri api_stat: Running main() from gmock_main.cc 2024-06-03T08:39:42.595 INFO:tasks.workunit.client.0.smithi088.stdout: api_stat: [==========] Running 9 tests from 2 test suites. 2024-06-03T08:39:42.595 INFO:tasks.workunit.client.0.smithi088.stdout: api_stat: [----------] Global test environment set-up. 2024-06-03T08:39:42.595 INFO:tasks.workunit.client.0.smithi088.stdout: api_stat: [----------] 5 tests from LibRadosStat 2024-06-03T08:39:42.595 INFO:tasks.workunit.client.0.smithi088.stdout: api_stat: [ RUN ] LibRadosStat.Stat 2024-06-03T08:39:42.595 INFO:tasks.workunit.client.0.smithi088.stdout: api_stat: [ OK ] LibRadosStat.Stat (27403 ms) 2024-06-03T08:39:42.595 INFO:tasks.workunit.client.0.smithi088.stdout: api_stat: [ RUN ] LibRadosStat.Stat2 2024-06-03T08:39:42.595 INFO:tasks.workunit.client.0.smithi088.stdout: api_stat: [ OK ] LibRadosStat.Stat2 (8953 ms) 2024-06-03T08:39:42.595 INFO:tasks.workunit.client.0.smithi088.stdout: api_stat: [ RUN ] LibRadosStat.StatNS 2024-06-03T08:39:42.595 INFO:tasks.workunit.client.0.smithi088.stdout: api_stat: [ OK ] LibRadosStat.StatNS (11909 ms) 2024-06-03T08:39:42.595 INFO:tasks.workunit.client.0.smithi088.stdout: api_stat: [ RUN ] LibRadosStat.ClusterStat 2024-06-03T08:39:42.596 INFO:tasks.workunit.client.0.smithi088.stdout: api_stat: [ OK ] LibRadosStat.ClusterStat (151 ms) 2024-06-03T08:39:42.596 INFO:tasks.workunit.client.0.smithi088.stdout: api_stat: [ RUN ] LibRadosStat.PoolStat 2024-06-03T08:39:42.596 INFO:tasks.workunit.client.0.smithi088.stdout: api_stat: [ OK ] LibRadosStat.PoolStat (8252 ms) 2024-06-03T08:39:42.596 INFO:tasks.workunit.client.0.smithi088.stdout: api_stat: [----------] 5 tests from LibRadosStat (56668 ms total) 2024-06-03T08:39:42.596 INFO:tasks.workunit.client.0.smithi088.stdout: api_stat: 2024-06-03T08:39:42.596 INFO:tasks.workunit.client.0.smithi088.stdout: api_stat: [----------] 4 tests from LibRadosStatEC 2024-06-03T08:39:42.596 INFO:tasks.workunit.client.0.smithi088.stdout: api_stat: [ RUN ] LibRadosStatEC.Stat 2024-06-03T08:39:42.596 INFO:tasks.workunit.client.0.smithi088.stdout: api_stat: [ OK ] LibRadosStatEC.Stat (7042 ms) 2024-06-03T08:39:42.596 INFO:tasks.workunit.client.0.smithi088.stdout: api_stat: [ RUN ] LibRadosStatEC.StatNS 2024-06-03T08:39:42.596 INFO:tasks.workunit.client.0.smithi088.stdout: api_stat: [ OK ] LibRadosStatEC.StatNS (484 ms) 2024-06-03T08:39:42.596 INFO:tasks.workunit.client.0.smithi088.stdout: api_stat: [ RUN ] LibRadosStatEC.ClusterStat 2024-06-03T08:39:42.596 INFO:tasks.workunit.client.0.smithi088.stdout: api_stat: [ OK ] LibRadosStatEC.ClusterStat (444 ms) 2024-06-03T08:39:42.596 INFO:tasks.workunit.client.0.smithi088.stdout: api_stat: [ RUN ] LibRadosStatEC.PoolStat 2024-06-03T08:39:42.597 INFO:tasks.workunit.client.0.smithi088.stdout: api_stat: [ OK ] LibRadosStatEC.PoolStat (682 ms) 2024-06-03T08:39:42.597 INFO:tasks.workunit.client.0.smithi088.stdout: api_stat: [----------] 4 tests from LibRadosStatEC (8652 ms total) 2024-06-03T08:39:42.597 INFO:tasks.workunit.client.0.smithi088.stdout: api_stat: 2024-06-03T08:39:42.597 INFO:tasks.workunit.client.0.smithi088.stdout: api_stat: [----------] Global test environment tear-down 2024-06-03T08:39:42.597 INFO:tasks.workunit.client.0.smithi088.stdout: api_stat: [==========] 9 tests from 2 test suites ran. (109076 ms total) 2024-06-03T08:39:42.597 INFO:tasks.workunit.client.0.smithi088.stdout: api_stat: [ PASSED ] 9 tests. 2024-06-03T08:39:44.095 INFO:tasks.workunit.client.0.smithi088.stdout:tify cookie 94492607142096 notify_id 360777252869 notifier_gid 4761 2024-06-03T08:39:44.095 INFO:tasks.workunit.client.0.smithi088.stdout: api_watch_notify_pp: timed out 2024-06-03T08:39:44.096 INFO:tasks.workunit.client.0.smithi088.stdout: api_watch_notify_pp: flushing 2024-06-03T08:39:44.096 INFO:tasks.workunit.client.0.smithi088.stdout: api_watch_notify_pp: flushed 2024-06-03T08:39:44.096 INFO:tasks.workunit.client.0.smithi088.stdout: api_watch_notify_pp: [ OK ] LibRadosWatchNotifyPPTests/LibRadosWatchNotifyPP.WatchNotify2Timeout/0 (3204 ms) 2024-06-03T08:39:44.096 INFO:tasks.workunit.client.0.smithi088.stdout: api_watch_notify_pp: [ RUN ] LibRadosWatchNotifyPPTests/LibRadosWatchNotifyPP.WatchNotify2Timeout/1 2024-06-03T08:39:44.096 INFO:tasks.workunit.client.0.smithi088.stdout: api_watch_notify_pp: trying... 2024-06-03T08:39:44.096 INFO:tasks.workunit.client.0.smithi088.stdout: api_watch_notify_pp: handle_notify cookie 94492607144304 notify_id 365072220166 notifier_gid 4761 2024-06-03T08:39:44.096 INFO:tasks.workunit.client.0.smithi088.stdout: api_watch_notify_pp: timed out 2024-06-03T08:39:44.096 INFO:tasks.workunit.client.0.smithi088.stdout: api_watch_notify_pp: flushing 2024-06-03T08:39:44.096 INFO:tasks.workunit.client.0.smithi088.stdout: api_watch_notify_pp: flushed 2024-06-03T08:39:44.096 INFO:tasks.workunit.client.0.smithi088.stdout: api_watch_notify_pp: [ OK ] LibRadosWatchNotifyPPTests/LibRadosWatchNotifyPP.WatchNotify2Timeout/1 (3271 ms) 2024-06-03T08:39:44.096 INFO:tasks.workunit.client.0.smithi088.stdout: api_watch_notify_pp: [ RUN ] LibRadosWatchNotifyPPTests/LibRadosWatchNotifyPP.WatchNotify3/0 2024-06-03T08:39:44.096 INFO:tasks.workunit.client.0.smithi088.stdout: api_watch_notify_pp: List watches 2024-06-03T08:39:44.096 INFO:tasks.workunit.client.0.smithi088.stdout: api_watch_notify_pp: notify2 2024-06-03T08:39:44.097 INFO:tasks.workunit.client.0.smithi088.stdout: api_watch_notify_pp: handle_notify cookie 94492607144304 notify_id 369367187463 notifier_gid 4761 2024-06-03T08:39:44.097 INFO:tasks.workunit.client.0.smithi088.stdout: api_watch_notify_pp: notify2 done 2024-06-03T08:39:44.097 INFO:tasks.workunit.client.0.smithi088.stdout: api_watch_notify_pp: watch_check 2024-06-03T08:39:44.097 INFO:tasks.workunit.client.0.smithi088.stdout: api_watch_notify_pp: unwatch2 2024-06-03T08:39:44.097 INFO:tasks.workunit.client.0.smithi088.stdout: api_watch_notify_pp: flushing 2024-06-03T08:39:44.097 INFO:tasks.workunit.client.0.smithi088.stdout: api_watch_notify_pp: done 2024-06-03T08:39:44.097 INFO:tasks.workunit.client.0.smithi088.stdout: api_watch_notify_pp: [ OK ] LibRadosWatchNotifyPPTests/LibRadosWatchNotifyPP.WatchNotify3/0 (4635 ms) 2024-06-03T08:39:44.097 INFO:tasks.workunit.client.0.smithi088.stdout: api_watch_notify_pp: [ RUN ] LibRadosWatchNotifyPPTests/LibRadosWatchNotifyPP.WatchNotify3/1 2024-06-03T08:39:44.097 INFO:tasks.workunit.client.0.smithi088.stdout: api_watch_notify_pp: List watches 2024-06-03T08:39:44.097 INFO:tasks.workunit.client.0.smithi088.stdout: api_watch_notify_pp: notify2 2024-06-03T08:39:44.097 INFO:tasks.workunit.client.0.smithi088.stdout: api_watch_notify_pp: handle_notify cookie 94492607144304 notify_id 382252089352 notifier_gid 4761 2024-06-03T08:39:44.097 INFO:tasks.workunit.client.0.smithi088.stdout: api_watch_notify_pp: notify2 done 2024-06-03T08:39:44.097 INFO:tasks.workunit.client.0.smithi088.stdout: api_watch_notify_pp: watch_check 2024-06-03T08:39:44.098 INFO:tasks.workunit.client.0.smithi088.stdout: api_watch_notify_pp: unwatch2 2024-06-03T08:39:44.098 INFO:tasks.workunit.client.0.smithi088.stdout: api_watch_notify_pp: flushing 2024-06-03T08:39:44.098 INFO:tasks.workunit.client.0.smithi088.stdout: api_watch_notify_pp: done 2024-06-03T08:39:44.098 INFO:tasks.workunit.client.0.smithi088.stdout: api_watch_notify_pp: [ OK ] LibRadosWatchNotifyPPTests/LibRadosWatchNotifyPP.WatchNotify3/1 (4189 ms) 2024-06-03T08:39:44.098 INFO:tasks.workunit.client.0.smithi088.stdout: api_watch_notify_pp: [----------] 14 tests from LibRadosWatchNotifyPPTests/LibRadosWatchNotifyPP (61347 ms total) 2024-06-03T08:39:44.098 INFO:tasks.workunit.client.0.smithi088.stdout: api_watch_notify_pp: 2024-06-03T08:39:44.098 INFO:tasks.workunit.client.0.smithi088.stdout: api_watch_notify_pp: [----------] Global test environment tear-down 2024-06-03T08:39:44.098 INFO:tasks.workunit.client.0.smithi088.stdout: api_watch_notify_pp: [==========] 16 tests from 2 test suites ran. (110538 ms total) 2024-06-03T08:39:44.098 INFO:tasks.workunit.client.0.smithi088.stdout: api_watch_notify_pp: [ PASSED ] 16 tests. 2024-06-03T08:39:44.654 INFO:tasks.workunit.client.0.smithi088.stdout: api_lock_pp: Running main() from gmock_main.cc 2024-06-03T08:39:44.654 INFO:tasks.workunit.client.0.smithi088.stdout: api_lock_pp: [==========] Running 16 tests from 2 test suites. 2024-06-03T08:39:44.654 INFO:tasks.workunit.client.0.smithi088.stdout: api_lock_pp: [----------] Global test environment set-up. 2024-06-03T08:39:44.654 INFO:tasks.workunit.client.0.smithi088.stdout: api_lock_pp: [----------] 8 tests from LibRadosLockPP 2024-06-03T08:39:44.654 INFO:tasks.workunit.client.0.smithi088.stdout: api_lock_pp: seed 40400 2024-06-03T08:39:44.654 INFO:tasks.workunit.client.0.smithi088.stdout: api_lock_pp: [ RUN ] LibRadosLockPP.LockExclusivePP 2024-06-03T08:39:44.655 INFO:tasks.workunit.client.0.smithi088.stdout: api_lock_pp: [ OK ] LibRadosLockPP.LockExclusivePP (29082 ms) 2024-06-03T08:39:44.655 INFO:tasks.workunit.client.0.smithi088.stdout: api_lock_pp: [ RUN ] LibRadosLockPP.LockSharedPP 2024-06-03T08:39:44.655 INFO:tasks.workunit.client.0.smithi088.stdout: api_lock_pp: [ OK ] LibRadosLockPP.LockSharedPP (877 ms) 2024-06-03T08:39:44.655 INFO:tasks.workunit.client.0.smithi088.stdout: api_lock_pp: [ RUN ] LibRadosLockPP.LockExclusiveDurPP 2024-06-03T08:39:44.655 INFO:tasks.workunit.client.0.smithi088.stdout: api_lock_pp: [ OK ] LibRadosLockPP.LockExclusiveDurPP (16428 ms) 2024-06-03T08:39:44.655 INFO:tasks.workunit.client.0.smithi088.stdout: api_lock_pp: [ RUN ] LibRadosLockPP.LockSharedDurPP 2024-06-03T08:39:44.655 INFO:tasks.workunit.client.0.smithi088.stdout: api_lock_pp: [ OK ] LibRadosLockPP.LockSharedDurPP (5161 ms) 2024-06-03T08:39:44.655 INFO:tasks.workunit.client.0.smithi088.stdout: api_lock_pp: [ RUN ] LibRadosLockPP.LockMayRenewPP 2024-06-03T08:39:44.655 INFO:tasks.workunit.client.0.smithi088.stdout: api_lock_pp: [ OK ] LibRadosLockPP.LockMayRenewPP (4529 ms) 2024-06-03T08:39:44.655 INFO:tasks.workunit.client.0.smithi088.stdout: api_lock_pp: [ RUN ] LibRadosLockPP.UnlockPP 2024-06-03T08:39:44.655 INFO:tasks.workunit.client.0.smithi088.stdout: api_lock_pp: [ OK ] LibRadosLockPP.UnlockPP (840 ms) 2024-06-03T08:39:44.655 INFO:tasks.workunit.client.0.smithi088.stdout: api_lock_pp: [ RUN ] LibRadosLockPP.ListLockersPP 2024-06-03T08:39:44.655 INFO:tasks.workunit.client.0.smithi088.stdout: api_lock_pp: [ OK ] LibRadosLockPP.ListLockersPP (207 ms) 2024-06-03T08:39:44.656 INFO:tasks.workunit.client.0.smithi088.stdout: api_lock_pp: [ RUN ] LibRadosLockPP.BreakLockPP 2024-06-03T08:39:44.656 INFO:tasks.workunit.client.0.smithi088.stdout: api_lock_pp: [ OK ] LibRadosLockPP.BreakLockPP (1395 ms) 2024-06-03T08:39:44.656 INFO:tasks.workunit.client.0.smithi088.stdout: api_lock_pp: [----------] 8 tests from LibRadosLockPP (58519 ms total) 2024-06-03T08:39:44.656 INFO:tasks.workunit.client.0.smithi088.stdout: api_lock_pp: 2024-06-03T08:39:44.656 INFO:tasks.workunit.client.0.smithi088.stdout: api_lock_pp: [----------] 8 tests from LibRadosLockECPP 2024-06-03T08:39:44.656 INFO:tasks.workunit.client.0.smithi088.stdout: api_lock_pp: [ RUN ] LibRadosLockECPP.LockExclusivePP 2024-06-03T08:39:44.656 INFO:tasks.workunit.client.0.smithi088.stdout: api_lock_pp: [ OK ] LibRadosLockECPP.LockExclusivePP (6999 ms) 2024-06-03T08:39:44.656 INFO:tasks.workunit.client.0.smithi088.stdout: api_lock_pp: [ RUN ] LibRadosLockECPP.LockSharedPP 2024-06-03T08:39:44.656 INFO:tasks.workunit.client.0.smithi088.stdout: api_lock_pp: [ OK ] LibRadosLockECPP.LockSharedPP (106 ms) 2024-06-03T08:39:44.656 INFO:tasks.workunit.client.0.smithi088.stdout: api_lock_pp: [ RUN ] LibRadosLockECPP.LockExclusiveDurPP 2024-06-03T08:39:44.656 INFO:tasks.workunit.client.0.smithi088.stdout: api_lock_pp: [ OK ] LibRadosLockECPP.LockExclusiveDurPP (1188 ms) 2024-06-03T08:39:44.656 INFO:tasks.workunit.client.0.smithi088.stdout: api_lock_pp: [ RUN ] LibRadosLockECPP.LockSharedDurPP 2024-06-03T08:39:44.656 INFO:tasks.workunit.client.0.smithi088.stdout: api_lock_pp: [ OK ] LibRadosLockECPP.LockSharedDurPP (1119 ms) 2024-06-03T08:39:44.656 INFO:tasks.workunit.client.0.smithi088.stdout: api_lock_pp: [ RUN ] LibRadosLockECPP.LockMayRenewPP 2024-06-03T08:39:44.657 INFO:tasks.workunit.client.0.smithi088.stdout: api_lock_pp: [ OK ] LibRadosLockECPP.LockMayRenewPP (199 ms) 2024-06-03T08:39:44.657 INFO:tasks.workunit.client.0.smithi088.stdout: api_lock_pp: [ RUN ] LibRadosLockECPP.UnlockPP 2024-06-03T08:39:44.657 INFO:tasks.workunit.client.0.smithi088.stdout: api_lock_pp: [ OK ] LibRadosLockECPP.UnlockPP (1194 ms) 2024-06-03T08:39:44.657 INFO:tasks.workunit.client.0.smithi088.stdout: api_lock_pp: [ RUN ] LibRadosLockECPP.ListLockersPP 2024-06-03T08:39:44.657 INFO:tasks.workunit.client.0.smithi088.stdout: api_lock_pp: [ OK ] LibRadosLockECPP.ListLockersPP (313 ms) 2024-06-03T08:39:44.657 INFO:tasks.workunit.client.0.smithi088.stdout: api_lock_pp: [ RUN ] LibRadosLockECPP.BreakLockPP 2024-06-03T08:39:44.657 INFO:tasks.workunit.client.0.smithi088.stdout: api_lock_pp: [ OK ] LibRadosLockECPP.BreakLockPP (137 ms) 2024-06-03T08:39:44.657 INFO:tasks.workunit.client.0.smithi088.stdout: api_lock_pp: [----------] 8 tests from LibRadosLockECPP (11255 ms total) 2024-06-03T08:39:44.657 INFO:tasks.workunit.client.0.smithi088.stdout: api_lock_pp: 2024-06-03T08:39:44.657 INFO:tasks.workunit.client.0.smithi088.stdout: api_lock_pp: [----------] Global test environment tear-down 2024-06-03T08:39:44.657 INFO:tasks.workunit.client.0.smithi088.stdout: api_lock_pp: [==========] 16 tests from 2 test suites ran. (111212 ms total) 2024-06-03T08:39:44.657 INFO:tasks.workunit.client.0.smithi088.stdout: api_lock_pp: [ PASSED ] 16 tests. 2024-06-03T08:39:47.147 INFO:tasks.workunit.client.0.smithi088.stdout: list_parallel: process_1_[41018]: starting. 2024-06-03T08:39:47.147 INFO:tasks.workunit.client.0.smithi088.stdout: list_parallel: process_1_[41018]: creating pool ceph_test_rados_list_parallel.smithi088-40998 2024-06-03T08:39:47.147 INFO:tasks.workunit.client.0.smithi088.stdout: list_parallel: process_1_[41018]: created object 0... 2024-06-03T08:39:47.147 INFO:tasks.workunit.client.0.smithi088.stdout: list_parallel: process_1_[41018]: created object 25... 2024-06-03T08:39:47.147 INFO:tasks.workunit.client.0.smithi088.stdout: list_parallel: process_1_[41018]: created object 49... 2024-06-03T08:39:47.147 INFO:tasks.workunit.client.0.smithi088.stdout: list_parallel: process_1_[41018]: finishing. 2024-06-03T08:39:47.148 INFO:tasks.workunit.client.0.smithi088.stdout: list_parallel: process_1_[41018]: shutting down. 2024-06-03T08:39:47.148 INFO:tasks.workunit.client.0.smithi088.stdout: list_parallel: process_2_[41020]: starting. 2024-06-03T08:39:47.148 INFO:tasks.workunit.client.0.smithi088.stdout: list_parallel: process_2_[41020]: listing objects. 2024-06-03T08:39:47.148 INFO:tasks.workunit.client.0.smithi088.stdout: list_parallel: process_2_[41020]: listed object 0... 2024-06-03T08:39:47.148 INFO:tasks.workunit.client.0.smithi088.stdout: list_parallel: process_2_[41020]: listed object 25... 2024-06-03T08:39:47.148 INFO:tasks.workunit.client.0.smithi088.stdout: list_parallel: process_2_[41020]: saw 50 objects 2024-06-03T08:39:47.148 INFO:tasks.workunit.client.0.smithi088.stdout: list_parallel: process_2_[41020]: shutting down. 2024-06-03T08:39:47.148 INFO:tasks.workunit.client.0.smithi088.stdout: list_parallel: ******************************* 2024-06-03T08:39:47.148 INFO:tasks.workunit.client.0.smithi088.stdout: list_parallel: process_3_[51000]: starting. 2024-06-03T08:39:47.148 INFO:tasks.workunit.client.0.smithi088.stdout: list_parallel: process_3_[51000]: creating pool ceph_test_rados_list_parallel.smithi088-40998 2024-06-03T08:39:47.148 INFO:tasks.workunit.client.0.smithi088.stdout: list_parallel: process_3_[51000]: created object 0... 2024-06-03T08:39:47.148 INFO:tasks.workunit.client.0.smithi088.stdout: list_parallel: process_3_[51000]: created object 25... 2024-06-03T08:39:47.148 INFO:tasks.workunit.client.0.smithi088.stdout: list_parallel: process_3_[51000]: created object 49... 2024-06-03T08:39:47.149 INFO:tasks.workunit.client.0.smithi088.stdout: list_parallel: process_3_[51000]: finishing. 2024-06-03T08:39:47.149 INFO:tasks.workunit.client.0.smithi088.stdout: list_parallel: process_3_[51000]: shutting down. 2024-06-03T08:39:47.149 INFO:tasks.workunit.client.0.smithi088.stdout: list_parallel: ******************************* 2024-06-03T08:39:47.149 INFO:tasks.workunit.client.0.smithi088.stdout: list_parallel: process_4_[51001]: starting. 2024-06-03T08:39:47.149 INFO:tasks.workunit.client.0.smithi088.stdout: list_parallel: process_4_[51001]: listing objects. 2024-06-03T08:39:47.149 INFO:tasks.workunit.client.0.smithi088.stdout: list_parallel: process_4_[51001]: listed object 0... 2024-06-03T08:39:47.149 INFO:tasks.workunit.client.0.smithi088.stdout: list_parallel: process_4_[51001]: listed object 25... 2024-06-03T08:39:47.149 INFO:tasks.workunit.client.0.smithi088.stdout: list_parallel: process_4_[51001]: saw 45 objects 2024-06-03T08:39:47.149 INFO:tasks.workunit.client.0.smithi088.stdout: list_parallel: process_4_[51001]: shutting down. 2024-06-03T08:39:47.149 INFO:tasks.workunit.client.0.smithi088.stdout: list_parallel: ******************************* 2024-06-03T08:39:47.149 INFO:tasks.workunit.client.0.smithi088.stdout: list_parallel: process_5_[51002]: starting. 2024-06-03T08:39:47.149 INFO:tasks.workunit.client.0.smithi088.stdout: list_parallel: process_5_[51002]: removed 25 objects... 2024-06-03T08:39:47.149 INFO:tasks.workunit.client.0.smithi088.stdout: list_parallel: process_5_[51002]: removed half of the objects 2024-06-03T08:39:47.150 INFO:tasks.workunit.client.0.smithi088.stdout: list_parallel: process_5_[51002]: removed 50 objects... 2024-06-03T08:39:47.150 INFO:tasks.workunit.client.0.smithi088.stdout: list_parallel: process_5_[51002]: removed 50 objects 2024-06-03T08:39:47.150 INFO:tasks.workunit.client.0.smithi088.stdout: list_parallel: process_5_[51002]: shutting down. 2024-06-03T08:39:47.150 INFO:tasks.workunit.client.0.smithi088.stdout: list_parallel: ******************************* 2024-06-03T08:39:47.150 INFO:tasks.workunit.client.0.smithi088.stdout: list_parallel: ******************************* 2024-06-03T08:39:47.150 INFO:tasks.workunit.client.0.smithi088.stdout: list_parallel: process_6_[55054]: starting. 2024-06-03T08:39:47.150 INFO:tasks.workunit.client.0.smithi088.stdout: list_parallel: process_6_[55054]: creating pool ceph_test_rados_list_parallel.smithi088-40998 2024-06-03T08:39:47.150 INFO:tasks.workunit.client.0.smithi088.stdout: list_parallel: process_6_[55054]: created object 0... 2024-06-03T08:39:47.150 INFO:tasks.workunit.client.0.smithi088.stdout: list_parallel: process_6_[55054]: created object 25... 2024-06-03T08:39:47.150 INFO:tasks.workunit.client.0.smithi088.stdout: list_parallel: process_6_[55054]: created object 49... 2024-06-03T08:39:47.150 INFO:tasks.workunit.client.0.smithi088.stdout: list_parallel: process_6_[55054]: finishing. 2024-06-03T08:39:47.150 INFO:tasks.workunit.client.0.smithi088.stdout: list_parallel: process_6_[55054]: shutting down. 2024-06-03T08:39:47.150 INFO:tasks.workunit.client.0.smithi088.stdout: list_parallel: ******************************* 2024-06-03T08:39:47.151 INFO:tasks.workunit.client.0.smithi088.stdout: list_parallel: ******************************* 2024-06-03T08:39:47.151 INFO:tasks.workunit.client.0.smithi088.stdout: list_parallel: process_7_[55055]: starting. 2024-06-03T08:39:47.151 INFO:tasks.workunit.client.0.smithi088.stdout: list_parallel: process_7_[55055]: listing objects. 2024-06-03T08:39:47.151 INFO:tasks.workunit.client.0.smithi088.stdout: list_parallel: process_7_[55055]: listed object 0... 2024-06-03T08:39:47.151 INFO:tasks.workunit.client.0.smithi088.stdout: list_parallel: process_7_[55055]: listed object 25... 2024-06-03T08:39:47.151 INFO:tasks.workunit.client.0.smithi088.stdout: list_parallel: process_7_[55055]: listed object 50... 2024-06-03T08:39:47.151 INFO:tasks.workunit.client.0.smithi088.stdout: list_parallel: process_7_[55055]: saw 53 objects 2024-06-03T08:39:47.151 INFO:tasks.workunit.client.0.smithi088.stdout: list_parallel: process_7_[55055]: shutting down. 2024-06-03T08:39:47.151 INFO:tasks.workunit.client.0.smithi088.stdout: list_parallel: ******************************* 2024-06-03T08:39:47.151 INFO:tasks.workunit.client.0.smithi088.stdout: list_parallel: ******************************* 2024-06-03T08:39:47.151 INFO:tasks.workunit.client.0.smithi088.stdout: list_parallel: process_8_[55056]: starting. 2024-06-03T08:39:47.151 INFO:tasks.workunit.client.0.smithi088.stdout: list_parallel: process_8_[55056]: added 25 objects... 2024-06-03T08:39:47.151 INFO:tasks.workunit.client.0.smithi088.stdout: list_parallel: process_8_[55056]: added half of the objects 2024-06-03T08:39:47.152 INFO:tasks.workunit.client.0.smithi088.stdout: list_parallel: process_8_[55056]: added 50 objects... 2024-06-03T08:39:47.152 INFO:tasks.workunit.client.0.smithi088.stdout: list_parallel: process_8_[55056]: added 50 objects 2024-06-03T08:39:47.152 INFO:tasks.workunit.client.0.smithi088.stdout: list_parallel: process_8_[55056]: shutting down. 2024-06-03T08:39:47.152 INFO:tasks.workunit.client.0.smithi088.stdout: list_parallel: ******************************* 2024-06-03T08:39:47.152 INFO:tasks.workunit.client.0.smithi088.stdout: list_parallel: ******************************* 2024-06-03T08:39:53.857 INFO:tasks.workunit.client.0.smithi088.stdout: list_parallel: ********************** api_watch_notify: Running main() from gmock_main.cc 2024-06-03T08:39:53.858 INFO:tasks.workunit.client.0.smithi088.stdout: api_watch_notify: [==========] Running 11 tests from 2 test suites. 2024-06-03T08:39:53.858 INFO:tasks.workunit.client.0.smithi088.stdout: api_watch_notify: [----------] Global test environment set-up. 2024-06-03T08:39:53.858 INFO:tasks.workunit.client.0.smithi088.stdout: api_watch_notify: [----------] 10 tests from LibRadosWatchNotify 2024-06-03T08:39:53.858 INFO:tasks.workunit.client.0.smithi088.stdout: api_watch_notify: [ RUN ] LibRadosWatchNotify.WatchNotify 2024-06-03T08:39:53.858 INFO:tasks.workunit.client.0.smithi088.stdout: api_watch_notify: watch_notify_test_cb 2024-06-03T08:39:53.858 INFO:tasks.workunit.client.0.smithi088.stdout: api_watch_notify: [ OK ] LibRadosWatchNotify.WatchNotify (27414 ms) 2024-06-03T08:39:53.858 INFO:tasks.workunit.client.0.smithi088.stdout: api_watch_notify: [ RUN ] LibRadosWatchNotify.Watch2Delete 2024-06-03T08:39:53.858 INFO:tasks.workunit.client.0.smithi088.stdout: api_watch_notify: waiting up to 300 for disconnect notification ... 2024-06-03T08:39:53.858 INFO:tasks.workunit.client.0.smithi088.stdout: api_watch_notify: watch_notify2_test_errcb cookie 94652209202064 err -107 2024-06-03T08:39:53.859 INFO:tasks.workunit.client.0.smithi088.stdout: api_watch_notify: [ OK ] LibRadosWatchNotify.Watch2Delete (8595 ms) 2024-06-03T08:39:53.859 INFO:tasks.workunit.client.0.smithi088.stdout: api_watch_notify: [ RUN ] LibRadosWatchNotify.AioWatchDelete 2024-06-03T08:39:53.859 INFO:tasks.workunit.client.0.smithi088.stdout: api_watch_notify: waiting up to 300 for disconnect notification ... 2024-06-03T08:39:53.859 INFO:tasks.workunit.client.0.smithi088.stdout: api_watch_notify: watch_notify2_test_errcb cookie 94652209217584 err -107 2024-06-03T08:39:53.859 INFO:tasks.workunit.client.0.smithi088.stdout: api_watch_notify: [ OK ] LibRadosWatchNotify.AioWatchDelete (6857 ms) 2024-06-03T08:39:53.859 INFO:tasks.workunit.client.0.smithi088.stdout: api_watch_notify: [ RUN ] LibRadosWatchNotify.WatchNotify2 2024-06-03T08:39:53.859 INFO:tasks.workunit.client.0.smithi088.stdout: api_watch_notify: watch_notify2_test_cb from 4521 notify_id 287762808832 cookie 94652209242256 2024-06-03T08:39:53.859 INFO:tasks.workunit.client.0.smithi088.stdout: api_watch_notify: [ OK ] LibRadosWatchNotify.WatchNotify2 (9682 ms) 2024-06-03T08:39:53.859 INFO:tasks.workunit.client.0.smithi088.stdout: api_watch_notify: [ RUN ] LibRadosWatchNotify.AioWatchNotify2 2024-06-03T08:39:53.859 INFO:tasks.workunit.client.0.smithi088.stdout: api_watch_notify: watch_notify2_test_cb from 4521 notify_id 322122547200 cookie 94652209242208 2024-06-03T08:39:53.859 INFO:tasks.workunit.client.0.smithi088.stdout: api_watch_notify: [ OK ] LibRadosWatchNotify.AioWatchNotify2 (3602 ms) 2024-06-03T08:39:53.859 INFO:tasks.workunit.client.0.smithi088.stdout: api_watch_notify: [ RUN ] LibRadosWatchNotify.AioNotify 2024-06-03T08:39:53.859 INFO:tasks.workunit.client.0.smithi088.stdout: api_watch_notify: watch_notify2_test_cb from 4521 notify_id 330712481793 cookie 94652209262816 2024-06-03T08:39:53.860 INFO:tasks.workunit.client.0.smithi088.stdout: api_watch_notify: [ OK ] LibRadosWatchNotify.AioNotify (2470 ms) 2024-06-03T08:39:53.860 INFO:tasks.workunit.client.0.smithi088.stdout: api_watch_notify: [ RUN ] LibRadosWatchNotify.WatchNotify2Multi 2024-06-03T08:39:53.860 INFO:tasks.workunit.client.0.smithi088.stdout: api_watch_notify: watch_notify2_test_cb from 4521 notify_id 335007449090 cookie 94652209162352 2024-06-03T08:39:53.860 INFO:tasks.workunit.client.0.smithi088.stdout: api_watch_notify: watch_notify2_test_cb from 4521 notify_id 335007449090 cookie 94652209262816 2024-06-03T08:39:53.860 INFO:tasks.workunit.client.0.smithi088.stdout: api_watch_notify: [ OK ] LibRadosWatchNotify.WatchNotify2Multi (1879 ms) 2024-06-03T08:39:53.860 INFO:tasks.workunit.client.0.smithi088.stdout: api_watch_notify: [ RUN ] LibRadosWatchNotify.WatchNotify2Timeout 2024-06-03T08:39:53.860 INFO:tasks.workunit.client.0.smithi088.stdout: api_watch_notify: watch_notify2_test_cb from 4521 notify_id 335007449088 cookie 94652209262816 2024-06-03T08:39:53.860 INFO:tasks.workunit.client.0.smithi088.stdout: api_watch_notify: watch_notify2_test_cb from 4521 notify_id 339302416385 cookie 94652209262816 2024-06-03T08:39:53.860 INFO:tasks.workunit.client.0.smithi088.stdout: api_watch_notify: [ OK ] LibRadosWatchNotify.WatchNotify2Timeout (3325 ms) 2024-06-03T08:39:53.860 INFO:tasks.workunit.client.0.smithi088.stdout: api_watch_notify: [ RUN ] LibRadosWatchNotify.Watch3Timeout 2024-06-03T08:39:53.860 INFO:tasks.workunit.client.0.smithi088.stdout: api_watch_notify: waiting up to 1024 for osd to time us out ... 2024-06-03T08:39:53.860 INFO:tasks.workunit.client.0.smithi088.stdout: api_watch_notify: watch_notify2_test_errcb cookie 94652209281040 err -107 2024-06-03T08:39:53.861 INFO:tasks.workunit.client.0.smithi088.stdout: api_watch_notify: watch_notify2_test_cb from 4521 notify_id 347892350977 cookie 94652209281040 2024-06-03T08:39:53.861 INFO:tasks.workunit.client.0.smithi088.stdout: api_watch_notify: [ OK ] LibRadosWatchNotify.Watch3Timeout (6100 ms) 2024-06-03T08:39:53.861 INFO:tasks.workunit.client.0.smithi088.stdout: api_watch_notify: [ RUN ] LibRadosWatchNotify.AioWatchDelete2 2024-06-03T08:39:53.861 INFO:tasks.workunit.client.0.smithi088.stdout: api_watch_notify: waiting up to 30 for disconnect notification ... 2024-06-03T08:39:53.861 INFO:tasks.workunit.client.0.smithi088.stdout: api_watch_notify: watch_notify2_test_errcb cookie 94652209281040 err -107 2024-06-03T08:39:53.861 INFO:tasks.workunit.client.0.smithi088.stdout: api_watch_notify: [ OK ] LibRadosWatchNotify.AioWatchDelete2 (2196 ms) 2024-06-03T08:39:53.861 INFO:tasks.workunit.client.0.smithi088.stdout: api_watch_notify: [----------] 10 tests from LibRadosWatchNotify (72120 ms total) 2024-06-03T08:39:53.861 INFO:tasks.workunit.client.0.smithi088.stdout: api_watch_notify: 2024-06-03T08:39:53.861 INFO:tasks.workunit.client.0.smithi088.stdout: api_watch_notify: [----------] 1 test from LibRadosWatchNotifyEC 2024-06-03T08:39:53.861 INFO:tasks.workunit.client.0.smithi088.stdout: api_watch_notify: [ RUN ] LibRadosWatchNotifyEC.WatchNotify 2024-06-03T08:39:53.862 INFO:tasks.workunit.client.0.smithi088.stdout: api_watch_notify: watch_notify_test_cb 2024-06-03T08:39:53.862 INFO:tasks.workunit.client.0.smithi088.stdout: api_watch_notify: [ OK ] LibRadosWatchNotifyEC.WatchNotify (5887 ms) 2024-06-03T08:39:53.862 INFO:tasks.workunit.client.0.smithi088.stdout: api_watch_notify: [----------] 1 test from LibRadosWatchNotifyEC (5887 ms total) 2024-06-03T08:39:53.862 INFO:tasks.workunit.client.0.smithi088.stdout: api_watch_notify: 2024-06-03T08:39:53.862 INFO:tasks.workunit.client.0.smithi088.stdout: api_watch_notify: [----------] Global test environment tear-down 2024-06-03T08:39:53.862 INFO:tasks.workunit.client.0.smithi088.stdout: api_watch_notify: [==========] 11 tests from 2 test suites ran. (120313 ms total) 2024-06-03T08:39:53.863 INFO:tasks.workunit.client.0.smithi088.stdout: api_watch_notify: [ PASSED ] 11 tests. 2024-06-03T08:39:54.412 INFO:tasks.workunit.client.0.smithi088.stdout: api_io: Running main() from gmock_main.cc 2024-06-03T08:39:54.412 INFO:tasks.workunit.client.0.smithi088.stdout: api_io: [==========] Running 24 tests from 2 test suites. 2024-06-03T08:39:54.413 INFO:tasks.workunit.client.0.smithi088.stdout: api_io: [----------] Global test environment set-up. 2024-06-03T08:39:54.413 INFO:tasks.workunit.client.0.smithi088.stdout: api_io: [----------] 14 tests from LibRadosIo 2024-06-03T08:39:54.413 INFO:tasks.workunit.client.0.smithi088.stdout: api_io: [ RUN ] LibRadosIo.SimpleWrite 2024-06-03T08:39:54.413 INFO:tasks.workunit.client.0.smithi088.stdout: api_io: [ OK ] LibRadosIo.SimpleWrite (28482 ms) 2024-06-03T08:39:54.413 INFO:tasks.workunit.client.0.smithi088.stdout: api_io: [ RUN ] LibRadosIo.TooBig 2024-06-03T08:39:54.413 INFO:tasks.workunit.client.0.smithi088.stdout: api_io: [ OK ] LibRadosIo.TooBig (0 ms) 2024-06-03T08:39:54.413 INFO:tasks.workunit.client.0.smithi088.stdout: api_io: [ RUN ] LibRadosIo.ReadTimeout 2024-06-03T08:39:54.413 INFO:tasks.workunit.client.0.smithi088.stdout: api_io: no timeout :/ 2024-06-03T08:39:54.413 INFO:tasks.workunit.client.0.smithi088.stdout: api_io: no timeout :/ 2024-06-03T08:39:54.413 INFO:tasks.workunit.client.0.smithi088.stdout: api_io: no timeout :/ 2024-06-03T08:39:54.413 INFO:tasks.workunit.client.0.smithi088.stdout: api_io: no timeout :/ 2024-06-03T08:39:54.413 INFO:tasks.workunit.client.0.smithi088.stdout: api_io: no timeout :/ 2024-06-03T08:39:54.413 INFO:tasks.workunit.client.0.smithi088.stdout: api_io: [ OK ] LibRadosIo.ReadTimeout (1112 ms) 2024-06-03T08:39:54.414 INFO:tasks.workunit.client.0.smithi088.stdout: api_io: [ RUN ] LibRadosIo.RoundTrip 2024-06-03T08:39:54.414 INFO:tasks.workunit.client.0.smithi088.stdout: api_io: [ OK ] LibRadosIo.RoundTrip (15181 ms) 2024-06-03T08:39:54.414 INFO:tasks.workunit.client.0.smithi088.stdout: api_io: [ RUN ] LibRadosIo.Checksum 2024-06-03T08:39:54.414 INFO:tasks.workunit.client.0.smithi088.stdout: api_io: [ OK ] LibRadosIo.Checksum (6671 ms) 2024-06-03T08:39:54.414 INFO:tasks.workunit.client.0.smithi088.stdout: api_io: [ RUN ] LibRadosIo.OverlappingWriteRoundTrip 2024-06-03T08:39:54.414 INFO:tasks.workunit.client.0.smithi088.stdout: api_io: [ OK ] LibRadosIo.OverlappingWriteRoundTrip (3124 ms) 2024-06-03T08:39:54.414 INFO:tasks.workunit.client.0.smithi088.stdout: api_io: [ RUN ] LibRadosIo.WriteFullRoundTrip 2024-06-03T08:39:54.414 INFO:tasks.workunit.client.0.smithi088.stdout: api_io: [ OK ] LibRadosIo.WriteFullRoundTrip (2784 ms) 2024-06-03T08:39:54.414 INFO:tasks.workunit.client.0.smithi088.stdout: api_io: [ RUN ] LibRadosIo.AppendRoundTrip 2024-06-03T08:39:54.414 INFO:tasks.workunit.client.0.smithi088.stdout: api_io: [ OK ] LibRadosIo.AppendRoundTrip (92 ms) 2024-06-03T08:39:54.414 INFO:tasks.workunit.client.0.smithi088.stdout: api_io: [ RUN ] LibRadosIo.ZeroLenZero 2024-06-03T08:39:54.414 INFO:tasks.workunit.client.0.smithi088.stdout: api_io: [ OK ] LibRadosIo.ZeroLenZero (72 ms) 2024-06-03T08:39:54.414 INFO:tasks.workunit.client.0.smithi088.stdout: api_io: [ RUN ] LibRadosIo.TruncTest 2024-06-03T08:39:54.415 INFO:tasks.workunit.client.0.smithi088.stdout: api_io: [ OK ] LibRadosIo.TruncTest (108 ms) 2024-06-03T08:39:54.415 INFO:tasks.workunit.client.0.smithi088.stdout: api_io: [ RUN ] LibRadosIo.RemoveTest 2024-06-03T08:39:54.415 INFO:tasks.workunit.client.0.smithi088.stdout: api_io: [ OK ] LibRadosIo.RemoveTest (272 ms) 2024-06-03T08:39:54.415 INFO:tasks.workunit.client.0.smithi088.stdout: api_io: [ RUN ] LibRadosIo.XattrsRoundTrip 2024-06-03T08:39:54.415 INFO:tasks.workunit.client.0.smithi088.stdout: api_io: [ OK ] LibRadosIo.XattrsRoundTrip (200 ms) 2024-06-03T08:39:54.415 INFO:tasks.workunit.client.0.smithi088.stdout: api_io: [ RUN ] LibRadosIo.RmXattr 2024-06-03T08:39:54.415 INFO:tasks.workunit.client.0.smithi088.stdout: api_io: [ OK ] LibRadosIo.RmXattr (335 ms) 2024-06-03T08:39:54.415 INFO:tasks.workunit.client.0.smithi088.stdout: api_io: [ RUN ] LibRadosIo.XattrIter 2024-06-03T08:39:54.415 INFO:tasks.workunit.client.0.smithi088.stdout: api_io: [ OK ] LibRadosIo.XattrIter (2831 ms) 2024-06-03T08:39:54.415 INFO:tasks.workunit.client.0.smithi088.stdout: api_io: [----------] 14 tests from LibRadosIo (61266 ms total) 2024-06-03T08:39:54.415 INFO:tasks.workunit.client.0.smithi088.stdout: api_io: 2024-06-03T08:39:54.415 INFO:tasks.workunit.client.0.smithi088.stdout: api_io: [----------] 10 tests from LibRadosIoEC 2024-06-03T08:39:54.415 INFO:tasks.workunit.client.0.smithi088.stdout: api_io: [ RUN ] LibRadosIoEC.SimpleWrite 2024-06-03T08:39:54.416 INFO:tasks.workunit.client.0.smithi088.stdout: api_io: [ OK ] LibRadosIoEC.SimpleWrite (8964 ms) 2024-06-03T08:39:54.416 INFO:tasks.workunit.client.0.smithi088.stdout: api_io: [ RUN ] LibRadosIoEC.RoundTrip 2024-06-03T08:39:54.416 INFO:tasks.workunit.client.0.smithi088.stdout: api_io: [ OK ] LibRadosIoEC.RoundTrip (2156 ms) 2024-06-03T08:39:54.416 INFO:tasks.workunit.client.0.smithi088.stdout: api_io: [ RUN ] LibRadosIoEC.OverlappingWriteRoundTrip 2024-06-03T08:39:54.416 INFO:tasks.workunit.client.0.smithi088.stdout: api_io: [ OK ] LibRadosIoEC.OverlappingWriteRoundTrip (572 ms) 2024-06-03T08:39:54.416 INFO:tasks.workunit.client.0.smithi088.stdout: api_io: [ RUN ] LibRadosIoEC.WriteFullRoundTrip 2024-06-03T08:39:54.416 INFO:tasks.workunit.client.0.smithi088.stdout: api_io: [ OK ] LibRadosIoEC.WriteFullRoundTrip (328 ms) 2024-06-03T08:39:54.416 INFO:tasks.workunit.client.0.smithi088.stdout: api_io: [ RUN ] LibRadosIoEC.AppendRoundTrip 2024-06-03T08:39:54.416 INFO:tasks.workunit.client.0.smithi088.stdout: api_io: [ OK ] LibRadosIoEC.AppendRoundTrip (277 ms) 2024-06-03T08:39:54.416 INFO:tasks.workunit.client.0.smithi088.stdout: api_io: [ RUN ] LibRadosIoEC.TruncTest 2024-06-03T08:39:54.416 INFO:tasks.workunit.client.0.smithi088.stdout: api_io: [ OK ] LibRadosIoEC.TruncTest (110 ms) 2024-06-03T08:39:54.416 INFO:tasks.workunit.client.0.smithi088.stdout: api_io: [ RUN ] LibRadosIoEC.RemoveTest 2024-06-03T08:39:54.416 INFO:tasks.workunit.client.0.smithi088.stdout: api_io: [ OK ] LibRadosIoEC.RemoveTest (287 ms) 2024-06-03T08:39:54.416 INFO:tasks.workunit.client.0.smithi088.stdout: api_io: [ RUN ] LibRadosIoEC.XattrsRoundTrip 2024-06-03T08:39:54.417 INFO:tasks.workunit.client.0.smithi088.stdout: api_io: [ OK ] LibRadosIoEC.XattrsRoundTrip (939 ms) 2024-06-03T08:39:54.417 INFO:tasks.workunit.client.0.smithi088.stdout: api_io: [ RUN ] LibRadosIoEC.RmXattr 2024-06-03T08:39:54.417 INFO:tasks.workunit.client.0.smithi088.stdout: api_io: [ OK ] LibRadosIoEC.RmXattr (594 ms) 2024-06-03T08:39:54.417 INFO:tasks.workunit.client.0.smithi088.stdout: api_io: [ RUN ] LibRadosIoEC.XattrIter 2024-06-03T08:39:54.417 INFO:tasks.workunit.client.0.smithi088.stdout: api_io: [ OK ] LibRadosIoEC.XattrIter (215 ms) 2024-06-03T08:39:54.417 INFO:tasks.workunit.client.0.smithi088.stdout: api_io: [----------] 10 tests from LibRadosIoEC (14442 ms total) 2024-06-03T08:39:54.417 INFO:tasks.workunit.client.0.smithi088.stdout: api_io: 2024-06-03T08:39:54.417 INFO:tasks.workunit.client.0.smithi088.stdout: api_io: [----------] Global test environment tear-down 2024-06-03T08:39:54.417 INFO:tasks.workunit.client.0.smithi088.stdout: api_io: [==========] 24 tests from 2 test suites ran. (121056 ms total) 2024-06-03T08:39:54.418 INFO:tasks.workunit.client.0.smithi088.stdout: api_io: [ PASSED ] 24 tests. 2024-06-03T08:39:57.248 INFO:tasks.workunit.client.0.smithi088.stdout:pPP2 (117 ms) 2024-06-03T08:39:57.248 INFO:tasks.workunit.client.0.smithi088.stdout: api_io_pp: [ RUN ] LibRadosIoECPP.OverlappingWriteRoundTripPP 2024-06-03T08:39:57.248 INFO:tasks.workunit.client.0.smithi088.stdout: api_io_pp: [ OK ] LibRadosIoECPP.OverlappingWriteRoundTripPP (355 ms) 2024-06-03T08:39:57.248 INFO:tasks.workunit.client.0.smithi088.stdout: api_io_pp: [ RUN ] LibRadosIoECPP.WriteFullRoundTripPP 2024-06-03T08:39:57.248 INFO:tasks.workunit.client.0.smithi088.stdout: api_io_pp: [ OK ] LibRadosIoECPP.WriteFullRoundTripPP (736 ms) 2024-06-03T08:39:57.248 INFO:tasks.workunit.client.0.smithi088.stdout: api_io_pp: [ RUN ] LibRadosIoECPP.WriteFullRoundTripPP2 2024-06-03T08:39:57.248 INFO:tasks.workunit.client.0.smithi088.stdout: api_io_pp: [ OK ] LibRadosIoECPP.WriteFullRoundTripPP2 (443 ms) 2024-06-03T08:39:57.248 INFO:tasks.workunit.client.0.smithi088.stdout: api_io_pp: [ RUN ] LibRadosIoECPP.AppendRoundTripPP 2024-06-03T08:39:57.248 INFO:tasks.workunit.client.0.smithi088.stdout: api_io_pp: [ OK ] LibRadosIoECPP.AppendRoundTripPP (344 ms) 2024-06-03T08:39:57.249 INFO:tasks.workunit.client.0.smithi088.stdout: api_io_pp: [ RUN ] LibRadosIoECPP.TruncTestPP 2024-06-03T08:39:57.249 INFO:tasks.workunit.client.0.smithi088.stdout: api_io_pp: [ OK ] LibRadosIoECPP.TruncTestPP (164 ms) 2024-06-03T08:39:57.249 INFO:tasks.workunit.client.0.smithi088.stdout: api_io_pp: [ RUN ] LibRadosIoECPP.RemoveTestPP 2024-06-03T08:39:57.249 INFO:tasks.workunit.client.0.smithi088.stdout: api_io_pp: [ OK ] LibRadosIoECPP.RemoveTestPP (147 ms) 2024-06-03T08:39:57.249 INFO:tasks.workunit.client.0.smithi088.stdout: api_io_pp: [ RUN ] LibRadosIoECPP.XattrsRoundTripPP 2024-06-03T08:39:57.249 INFO:tasks.workunit.client.0.smithi088.stdout: api_io_pp: [ OK ] LibRadosIoECPP.XattrsRoundTripPP (151 ms) 2024-06-03T08:39:57.249 INFO:tasks.workunit.client.0.smithi088.stdout: api_io_pp: [ RUN ] LibRadosIoECPP.RmXattrPP 2024-06-03T08:39:57.249 INFO:tasks.workunit.client.0.smithi088.stdout: api_io_pp: [ OK ] LibRadosIoECPP.RmXattrPP (1474 ms) 2024-06-03T08:39:57.249 INFO:tasks.workunit.client.0.smithi088.stdout: api_io_pp: [ RUN ] LibRadosIoECPP.XattrListPP 2024-06-03T08:39:57.249 INFO:tasks.workunit.client.0.smithi088.stdout: api_io_pp: [ OK ] LibRadosIoECPP.XattrListPP (696 ms) 2024-06-03T08:39:57.249 INFO:tasks.workunit.client.0.smithi088.stdout: api_io_pp: [ RUN ] LibRadosIoECPP.CmpExtPP 2024-06-03T08:39:57.250 INFO:tasks.workunit.client.0.smithi088.stdout: api_io_pp: [ OK ] LibRadosIoECPP.CmpExtPP (187 ms) 2024-06-03T08:39:57.250 INFO:tasks.workunit.client.0.smithi088.stdout: api_io_pp: [ RUN ] LibRadosIoECPP.CmpExtDNEPP 2024-06-03T08:39:57.250 INFO:tasks.workunit.client.0.smithi088.stdout: api_io_pp: [ OK ] LibRadosIoECPP.CmpExtDNEPP (102 ms) 2024-06-03T08:39:57.250 INFO:tasks.workunit.client.0.smithi088.stdout: api_io_pp: [ RUN ] LibRadosIoECPP.CmpExtMismatchPP 2024-06-03T08:39:57.250 INFO:tasks.workunit.client.0.smithi088.stdout: api_io_pp: [ OK ] LibRadosIoECPP.CmpExtMismatchPP (168 ms) 2024-06-03T08:39:57.250 INFO:tasks.workunit.client.0.smithi088.stdout: api_io_pp: [----------] 17 tests from LibRadosIoECPP (15046 ms total) 2024-06-03T08:39:57.250 INFO:tasks.workunit.client.0.smithi088.stdout: api_io_pp: 2024-06-03T08:39:57.250 INFO:tasks.workunit.client.0.smithi088.stdout: api_io_pp: [----------] Global test environment tear-down 2024-06-03T08:39:57.250 INFO:tasks.workunit.client.0.smithi088.stdout: api_io_pp: [==========] 37 tests from 2 test suites ran. (123882 ms total) 2024-06-03T08:39:57.250 INFO:tasks.workunit.client.0.smithi088.stdout: api_io_pp: [ PASSED ] 37 tests. 2024-06-03T08:40:00.787 INFO:tasks.workunit.client.0.smithi088.stdout:********* 2024-06-03T08:40:00.788 INFO:tasks.workunit.client.0.smithi088.stdout: list_parallel: process_9_[61708]: starting. 2024-06-03T08:40:00.788 INFO:tasks.workunit.client.0.smithi088.stdout: list_parallel: process_9_[61708]: creating pool ceph_test_rados_list_parallel.smithi088-40998 2024-06-03T08:40:00.788 INFO:tasks.workunit.client.0.smithi088.stdout: list_parallel: process_9_[61708]: created object 0... 2024-06-03T08:40:00.788 INFO:tasks.workunit.client.0.smithi088.stdout: list_parallel: process_9_[61708]: created object 25... 2024-06-03T08:40:00.788 INFO:tasks.workunit.client.0.smithi088.stdout: list_parallel: process_9_[61708]: created object 49... 2024-06-03T08:40:00.788 INFO:tasks.workunit.client.0.smithi088.stdout: list_parallel: process_9_[61708]: finishing. 2024-06-03T08:40:00.788 INFO:tasks.workunit.client.0.smithi088.stdout: list_parallel: process_9_[61708]: shutting down. 2024-06-03T08:40:00.788 INFO:tasks.workunit.client.0.smithi088.stdout: list_parallel: ******************************* 2024-06-03T08:40:00.788 INFO:tasks.workunit.client.0.smithi088.stdout: list_parallel: ******************************* 2024-06-03T08:40:00.788 INFO:tasks.workunit.client.0.smithi088.stdout: list_parallel: ******************************* 2024-06-03T08:40:00.788 INFO:tasks.workunit.client.0.smithi088.stdout: list_parallel: process_10_[61709]: starting. 2024-06-03T08:40:00.788 INFO:tasks.workunit.client.0.smithi088.stdout: list_parallel: process_10_[61709]: listing objects. 2024-06-03T08:40:00.789 INFO:tasks.workunit.client.0.smithi088.stdout: list_parallel: process_10_[61709]: listed object 0... 2024-06-03T08:40:00.789 INFO:tasks.workunit.client.0.smithi088.stdout: list_parallel: process_10_[61709]: listed object 25... 2024-06-03T08:40:00.789 INFO:tasks.workunit.client.0.smithi088.stdout: list_parallel: process_10_[61709]: listed object 50... 2024-06-03T08:40:00.789 INFO:tasks.workunit.client.0.smithi088.stdout: list_parallel: process_10_[61709]: listed object 75... 2024-06-03T08:40:00.789 INFO:tasks.workunit.client.0.smithi088.stdout: list_parallel: process_10_[61709]: saw 100 objects 2024-06-03T08:40:00.789 INFO:tasks.workunit.client.0.smithi088.stdout: list_parallel: process_10_[61709]: shutting down. 2024-06-03T08:40:00.789 INFO:tasks.workunit.client.0.smithi088.stdout: list_parallel: ******************************* 2024-06-03T08:40:00.789 INFO:tasks.workunit.client.0.smithi088.stdout: list_parallel: ******************************* 2024-06-03T08:40:00.789 INFO:tasks.workunit.client.0.smithi088.stdout: list_parallel: ******************************* 2024-06-03T08:40:00.789 INFO:tasks.workunit.client.0.smithi088.stdout: list_parallel: process_13_[61712]: starting. 2024-06-03T08:40:00.789 INFO:tasks.workunit.client.0.smithi088.stdout: list_parallel: process_13_[61712]: removed 25 objects... 2024-06-03T08:40:00.789 INFO:tasks.workunit.client.0.smithi088.stdout: list_parallel: process_13_[61712]: removed half of the objects 2024-06-03T08:40:00.789 INFO:tasks.workunit.client.0.smithi088.stdout: list_parallel: process_13_[61712]: removed 50 objects... 2024-06-03T08:40:00.790 INFO:tasks.workunit.client.0.smithi088.stdout: list_parallel: process_13_[61712]: removed 50 objects 2024-06-03T08:40:00.790 INFO:tasks.workunit.client.0.smithi088.stdout: list_parallel: process_13_[61712]: shutting down. 2024-06-03T08:40:00.790 INFO:tasks.workunit.client.0.smithi088.stdout: list_parallel: ******************************* 2024-06-03T08:40:00.790 INFO:tasks.workunit.client.0.smithi088.stdout: list_parallel: ******************************* 2024-06-03T08:40:00.790 INFO:tasks.workunit.client.0.smithi088.stdout: list_parallel: ******************************* 2024-06-03T08:40:00.790 INFO:tasks.workunit.client.0.smithi088.stdout: list_parallel: process_12_[61711]: starting. 2024-06-03T08:40:00.790 INFO:tasks.workunit.client.0.smithi088.stdout: list_parallel: process_12_[61711]: added 25 objects... 2024-06-03T08:40:00.790 INFO:tasks.workunit.client.0.smithi088.stdout: list_parallel: process_12_[61711]: added half of the objects 2024-06-03T08:40:00.790 INFO:tasks.workunit.client.0.smithi088.stdout: list_parallel: process_12_[61711]: added 50 objects... 2024-06-03T08:40:00.790 INFO:tasks.workunit.client.0.smithi088.stdout: list_parallel: process_12_[61711]: added 50 objects 2024-06-03T08:40:00.790 INFO:tasks.workunit.client.0.smithi088.stdout: list_parallel: process_12_[61711]: shutting down. 2024-06-03T08:40:00.790 INFO:tasks.workunit.client.0.smithi088.stdout: list_parallel: ******************************* 2024-06-03T08:40:00.790 INFO:tasks.workunit.client.0.smithi088.stdout: list_parallel: ******************************* 2024-06-03T08:40:00.791 INFO:tasks.workunit.client.0.smithi088.stdout: list_parallel: ******************************* 2024-06-03T08:40:00.791 INFO:tasks.workunit.client.0.smithi088.stdout: list_parallel: process_11_[61710]: starting. 2024-06-03T08:40:00.791 INFO:tasks.workunit.client.0.smithi088.stdout: list_parallel: process_11_[61710]: added 25 objects... 2024-06-03T08:40:00.791 INFO:tasks.workunit.client.0.smithi088.stdout: list_parallel: process_11_[61710]: added half of the objects 2024-06-03T08:40:00.791 INFO:tasks.workunit.client.0.smithi088.stdout: list_parallel: process_11_[61710]: added 50 objects... 2024-06-03T08:40:00.791 INFO:tasks.workunit.client.0.smithi088.stdout: list_parallel: process_11_[61710]: added 50 objects 2024-06-03T08:40:00.791 INFO:tasks.workunit.client.0.smithi088.stdout: list_parallel: process_11_[61710]: shutting down. 2024-06-03T08:40:00.791 INFO:tasks.workunit.client.0.smithi088.stdout: list_parallel: ******************************* 2024-06-03T08:40:00.791 INFO:tasks.workunit.client.0.smithi088.stdout: list_parallel: ******************************* 2024-06-03T08:40:00.791 INFO:tasks.workunit.client.0.smithi088.stdout: list_parallel: ******************************* 2024-06-03T08:40:00.791 INFO:tasks.workunit.client.0.smithi088.stdout: list_parallel: ******************************* 2024-06-03T08:40:00.791 INFO:tasks.workunit.client.0.smithi088.stdout: list_parallel: process_14_[67725]: starting. 2024-06-03T08:40:00.791 INFO:tasks.workunit.client.0.smithi088.stdout: list_parallel: process_14_[67725]: creating pool ceph_test_rados_list_parallel.smithi088-40998 2024-06-03T08:40:00.791 INFO:tasks.workunit.client.0.smithi088.stdout: list_parallel: process_14_[67725]: created object 0... 2024-06-03T08:40:00.792 INFO:tasks.workunit.client.0.smithi088.stdout: list_parallel: process_14_[67725]: created object 25... 2024-06-03T08:40:00.792 INFO:tasks.workunit.client.0.smithi088.stdout: list_parallel: process_14_[67725]: created object 49... 2024-06-03T08:40:00.792 INFO:tasks.workunit.client.0.smithi088.stdout: list_parallel: process_14_[67725]: finishing. 2024-06-03T08:40:00.792 INFO:tasks.workunit.client.0.smithi088.stdout: list_parallel: process_14_[67725]: shutting down. 2024-06-03T08:40:00.792 INFO:tasks.workunit.client.0.smithi088.stdout: list_parallel: ******************************* 2024-06-03T08:40:00.792 INFO:tasks.workunit.client.0.smithi088.stdout: list_parallel: ******************************* 2024-06-03T08:40:00.792 INFO:tasks.workunit.client.0.smithi088.stdout: list_parallel: ******************************* 2024-06-03T08:40:00.792 INFO:tasks.workunit.client.0.smithi088.stdout: list_parallel: ******************************* 2024-06-03T08:40:00.792 INFO:tasks.workunit.client.0.smithi088.stdout: list_parallel: process_15_[67726]: starting. 2024-06-03T08:40:00.792 INFO:tasks.workunit.client.0.smithi088.stdout: list_parallel: process_15_[67726]: listing objects. 2024-06-03T08:40:00.792 INFO:tasks.workunit.client.0.smithi088.stdout: list_parallel: process_15_[67726]: listed object 0... 2024-06-03T08:40:02.224 INFO:tasks.workunit.client.0.smithi088.stdout: list_parallel: process_15_[67726]: listed object api_lock: Running main() from gmock_main.cc 2024-06-03T08:40:02.225 INFO:tasks.workunit.client.0.smithi088.stdout: api_lock: [==========] Running 16 tests from 2 test suites. 2024-06-03T08:40:02.225 INFO:tasks.workunit.client.0.smithi088.stdout: api_lock: [----------] Global test environment set-up. 2024-06-03T08:40:02.225 INFO:tasks.workunit.client.0.smithi088.stdout: api_lock: [----------] 8 tests from LibRadosLock 2024-06-03T08:40:02.225 INFO:tasks.workunit.client.0.smithi088.stdout: api_lock: [ RUN ] LibRadosLock.LockExclusive 2024-06-03T08:40:02.225 INFO:tasks.workunit.client.0.smithi088.stdout: api_lock: [ OK ] LibRadosLock.LockExclusive (16257 ms) 2024-06-03T08:40:02.225 INFO:tasks.workunit.client.0.smithi088.stdout: api_lock: [ RUN ] LibRadosLock.LockShared 2024-06-03T08:40:02.225 INFO:tasks.workunit.client.0.smithi088.stdout: api_lock: [ OK ] LibRadosLock.LockShared (4835 ms) 2024-06-03T08:40:02.225 INFO:tasks.workunit.client.0.smithi088.stdout: api_lock: [ RUN ] LibRadosLock.LockExclusiveDur 2024-06-03T08:40:02.225 INFO:tasks.workunit.client.0.smithi088.stdout: api_lock: [ OK ] LibRadosLock.LockExclusiveDur (34145 ms) 2024-06-03T08:40:02.225 INFO:tasks.workunit.client.0.smithi088.stdout: api_lock: [ RUN ] LibRadosLock.LockSharedDur 2024-06-03T08:40:02.225 INFO:tasks.workunit.client.0.smithi088.stdout: api_lock: [ OK ] LibRadosLock.LockSharedDur (1481 ms) 2024-06-03T08:40:02.226 INFO:tasks.workunit.client.0.smithi088.stdout: api_lock: [ RUN ] LibRadosLock.LockMayRenew 2024-06-03T08:40:02.226 INFO:tasks.workunit.client.0.smithi088.stdout: api_lock: [ OK ] LibRadosLock.LockMayRenew (188 ms) 2024-06-03T08:40:02.226 INFO:tasks.workunit.client.0.smithi088.stdout: api_lock: [ RUN ] LibRadosLock.Unlock 2024-06-03T08:40:02.226 INFO:tasks.workunit.client.0.smithi088.stdout: api_lock: [ OK ] LibRadosLock.Unlock (94 ms) 2024-06-03T08:40:02.226 INFO:tasks.workunit.client.0.smithi088.stdout: api_lock: [ RUN ] LibRadosLock.ListLockers 2024-06-03T08:40:02.226 INFO:tasks.workunit.client.0.smithi088.stdout: api_lock: [ OK ] LibRadosLock.ListLockers (3179 ms) 2024-06-03T08:40:02.226 INFO:tasks.workunit.client.0.smithi088.stdout: api_lock: [ RUN ] LibRadosLock.BreakLock 2024-06-03T08:40:02.226 INFO:tasks.workunit.client.0.smithi088.stdout: api_lock: [ OK ] LibRadosLock.BreakLock (130 ms) 2024-06-03T08:40:02.226 INFO:tasks.workunit.client.0.smithi088.stdout: api_lock: [----------] 8 tests from LibRadosLock (60309 ms total) 2024-06-03T08:40:02.226 INFO:tasks.workunit.client.0.smithi088.stdout: api_lock: 2024-06-03T08:40:02.226 INFO:tasks.workunit.client.0.smithi088.stdout: api_lock: [----------] 8 tests from LibRadosLockEC 2024-06-03T08:40:02.226 INFO:tasks.workunit.client.0.smithi088.stdout: api_lock: [ RUN ] LibRadosLockEC.LockExclusive 2024-06-03T08:40:02.226 INFO:tasks.workunit.client.0.smithi088.stdout: api_lock: [ OK ] LibRadosLockEC.LockExclusive (3313 ms) 2024-06-03T08:40:02.226 INFO:tasks.workunit.client.0.smithi088.stdout: api_lock: [ RUN ] LibRadosLockEC.LockShared 2024-06-03T08:40:02.227 INFO:tasks.workunit.client.0.smithi088.stdout: api_lock: [ OK ] LibRadosLockEC.LockShared (110 ms) 2024-06-03T08:40:02.227 INFO:tasks.workunit.client.0.smithi088.stdout: api_lock: [ RUN ] LibRadosLockEC.LockExclusiveDur 2024-06-03T08:40:02.227 INFO:tasks.workunit.client.0.smithi088.stdout: api_lock: [ OK ] LibRadosLockEC.LockExclusiveDur (1937 ms) 2024-06-03T08:40:02.227 INFO:tasks.workunit.client.0.smithi088.stdout: api_lock: [ RUN ] LibRadosLockEC.LockSharedDur 2024-06-03T08:40:02.227 INFO:tasks.workunit.client.0.smithi088.stdout: api_lock: [ OK ] LibRadosLockEC.LockSharedDur (1069 ms) 2024-06-03T08:40:02.227 INFO:tasks.workunit.client.0.smithi088.stdout: api_lock: [ RUN ] LibRadosLockEC.LockMayRenew 2024-06-03T08:40:02.227 INFO:tasks.workunit.client.0.smithi088.stdout: api_lock: [ OK ] LibRadosLockEC.LockMayRenew (163 ms) 2024-06-03T08:40:02.227 INFO:tasks.workunit.client.0.smithi088.stdout: api_lock: [ RUN ] LibRadosLockEC.Unlock 2024-06-03T08:40:02.227 INFO:tasks.workunit.client.0.smithi088.stdout: api_lock: [ OK ] LibRadosLockEC.Unlock (673 ms) 2024-06-03T08:40:02.227 INFO:tasks.workunit.client.0.smithi088.stdout: api_lock: [ RUN ] LibRadosLockEC.ListLockers 2024-06-03T08:40:02.227 INFO:tasks.workunit.client.0.smithi088.stdout: api_lock: [ OK ] LibRadosLockEC.ListLockers (887 ms) 2024-06-03T08:40:02.227 INFO:tasks.workunit.client.0.smithi088.stdout: api_lock: [ RUN ] LibRadosLockEC.BreakLock 2024-06-03T08:40:02.227 INFO:tasks.workunit.client.0.smithi088.stdout: api_lock: [ OK ] LibRadosLockEC.BreakLock (318 ms) 2024-06-03T08:40:02.228 INFO:tasks.workunit.client.0.smithi088.stdout: api_lock: [----------] 8 tests from LibRadosLockEC (8470 ms total) 2024-06-03T08:40:02.228 INFO:tasks.workunit.client.0.smithi088.stdout: api_lock: 2024-06-03T08:40:02.228 INFO:tasks.workunit.client.0.smithi088.stdout: api_lock: [----------] Global test environment tear-down 2024-06-03T08:40:02.228 INFO:tasks.workunit.client.0.smithi088.stdout: api_lock: [==========] 16 tests from 2 test suites ran. (128817 ms total) 2024-06-03T08:40:02.228 INFO:tasks.workunit.client.0.smithi088.stdout: api_lock: [ PASSED ] 16 tests. 2024-06-03T08:40:11.111 INFO:tasks.workunit.client.0.smithi088.stdout:25... 2024-06-03T08:40:11.111 INFO:tasks.workunit.client.0.smithi088.stdout: list_parallel: process_15_[67726]: listed object 50... 2024-06-03T08:40:11.111 INFO:tasks.workunit.client.0.smithi088.stdout: list_parallel: process_15_[67726]: listed object 75... 2024-06-03T08:40:11.111 INFO:tasks.workunit.client.0.smithi088.stdout: list_parallel: process_15_[67726]: listed object 100... 2024-06-03T08:40:11.111 INFO:tasks.workunit.client.0.smithi088.stdout: list_parallel: process_15_[67726]: listed object 125... 2024-06-03T08:40:11.111 INFO:tasks.workunit.client.0.smithi088.stdout: list_parallel: process_15_[67726]: saw 150 objects 2024-06-03T08:40:11.111 INFO:tasks.workunit.client.0.smithi088.stdout: list_parallel: process_15_[67726]: shutting down. 2024-06-03T08:40:11.112 INFO:tasks.workunit.client.0.smithi088.stdout: list_parallel: ******************************* 2024-06-03T08:40:11.112 INFO:tasks.workunit.client.0.smithi088.stdout: list_parallel: ******************************* 2024-06-03T08:40:11.112 INFO:tasks.workunit.client.0.smithi088.stdout: list_parallel: ******************************* 2024-06-03T08:40:11.112 INFO:tasks.workunit.client.0.smithi088.stdout: list_parallel: ******************************* 2024-06-03T08:40:11.112 INFO:tasks.workunit.client.0.smithi088.stdout: list_parallel: process_16_[67727]: starting. 2024-06-03T08:40:11.112 INFO:tasks.workunit.client.0.smithi088.stdout: list_parallel: process_16_[67727]: added 25 objects... 2024-06-03T08:40:11.112 INFO:tasks.workunit.client.0.smithi088.stdout: list_parallel: process_16_[67727]: added half of the objects 2024-06-03T08:40:11.112 INFO:tasks.workunit.client.0.smithi088.stdout: list_parallel: process_16_[67727]: added 50 objects... 2024-06-03T08:40:11.112 INFO:tasks.workunit.client.0.smithi088.stdout: list_parallel: process_16_[67727]: added 50 objects 2024-06-03T08:40:11.112 INFO:tasks.workunit.client.0.smithi088.stdout: list_parallel: process_16_[67727]: shutting down. 2024-06-03T08:40:11.112 INFO:tasks.workunit.client.0.smithi088.stdout: list_parallel: ******************************* 2024-06-03T08:40:11.113 INFO:tasks.workunit.client.0.smithi088.stdout: list_parallel: ******************************* 2024-06-03T08:40:11.113 INFO:tasks.workunit.client.0.smithi088.stdout: list_parallel: ******************************* 2024-06-03T08:40:11.113 INFO:tasks.workunit.client.0.smithi088.stdout: list_parallel: ******************************* 2024-06-03T08:40:11.113 INFO:tasks.workunit.client.0.smithi088.stdout: list_parallel: ******************************* 2024-06-03T08:40:11.113 INFO:tasks.workunit.client.0.smithi088.stdout: list_parallel: ******* SUCCESS ********** 2024-06-03T08:40:25.644 INFO:tasks.workunit.client.0.smithi088.stdout: api_c_write_operations: Running main() from gmock_main.cc 2024-06-03T08:40:25.645 INFO:tasks.workunit.client.0.smithi088.stdout: api_c_write_operations: [==========] Running 8 tests from 2 test suites. 2024-06-03T08:40:25.645 INFO:tasks.workunit.client.0.smithi088.stdout: api_c_write_operations: [----------] Global test environment set-up. 2024-06-03T08:40:25.645 INFO:tasks.workunit.client.0.smithi088.stdout: api_c_write_operations: [----------] 1 test from LibradosCWriteOps 2024-06-03T08:40:25.645 INFO:tasks.workunit.client.0.smithi088.stdout: api_c_write_operations: [ RUN ] LibradosCWriteOps.NewDelete 2024-06-03T08:40:25.645 INFO:tasks.workunit.client.0.smithi088.stdout: api_c_write_operations: [ OK ] LibradosCWriteOps.NewDelete (0 ms) 2024-06-03T08:40:25.645 INFO:tasks.workunit.client.0.smithi088.stdout: api_c_write_operations: [----------] 1 test from LibradosCWriteOps (0 ms total) 2024-06-03T08:40:25.645 INFO:tasks.workunit.client.0.smithi088.stdout: api_c_write_operations: 2024-06-03T08:40:25.645 INFO:tasks.workunit.client.0.smithi088.stdout: api_c_write_operations: [----------] 7 tests from LibRadosCWriteOps 2024-06-03T08:40:25.646 INFO:tasks.workunit.client.0.smithi088.stdout: api_c_write_operations: [ RUN ] LibRadosCWriteOps.assertExists 2024-06-03T08:40:25.646 INFO:tasks.workunit.client.0.smithi088.stdout: api_c_write_operations: [ OK ] LibRadosCWriteOps.assertExists (15234 ms) 2024-06-03T08:40:25.646 INFO:tasks.workunit.client.0.smithi088.stdout: api_c_write_operations: [ RUN ] LibRadosCWriteOps.WriteOpAssertVersion 2024-06-03T08:40:25.646 INFO:tasks.workunit.client.0.smithi088.stdout: api_c_write_operations: [ OK ] LibRadosCWriteOps.WriteOpAssertVersion (36302 ms) 2024-06-03T08:40:25.646 INFO:tasks.workunit.client.0.smithi088.stdout: api_c_write_operations: [ RUN ] LibRadosCWriteOps.Xattrs 2024-06-03T08:40:25.646 INFO:tasks.workunit.client.0.smithi088.stdout: api_c_write_operations: [ OK ] LibRadosCWriteOps.Xattrs (14062 ms) 2024-06-03T08:40:25.646 INFO:tasks.workunit.client.0.smithi088.stdout: api_c_write_operations: [ RUN ] LibRadosCWriteOps.Write 2024-06-03T08:40:25.646 INFO:tasks.workunit.client.0.smithi088.stdout: api_c_write_operations: [ OK ] LibRadosCWriteOps.Write (25590 ms) 2024-06-03T08:40:25.646 INFO:tasks.workunit.client.0.smithi088.stdout: api_c_write_operations: [ RUN ] LibRadosCWriteOps.Exec 2024-06-03T08:40:25.646 INFO:tasks.workunit.client.0.smithi088.stdout: api_c_write_operations: [ OK ] LibRadosCWriteOps.Exec (25850 ms) 2024-06-03T08:40:25.646 INFO:tasks.workunit.client.0.smithi088.stdout: api_c_write_operations: [ RUN ] LibRadosCWriteOps.WriteSame 2024-06-03T08:40:25.646 INFO:tasks.workunit.client.0.smithi088.stdout: api_c_write_operations: [ OK ] LibRadosCWriteOps.WriteSame (15852 ms) 2024-06-03T08:40:25.647 INFO:tasks.workunit.client.0.smithi088.stdout: api_c_write_operations: [ RUN ] LibRadosCWriteOps.CmpExt 2024-06-03T08:40:25.647 INFO:tasks.workunit.client.0.smithi088.stdout: api_c_write_operations: [ OK ] LibRadosCWriteOps.CmpExt (19127 ms) 2024-06-03T08:40:25.647 INFO:tasks.workunit.client.0.smithi088.stdout: api_c_write_operations: [----------] 7 tests from LibRadosCWriteOps (152017 ms total) 2024-06-03T08:40:25.647 INFO:tasks.workunit.client.0.smithi088.stdout: api_c_write_operations: 2024-06-03T08:40:25.647 INFO:tasks.workunit.client.0.smithi088.stdout: api_c_write_operations: [----------] Global test environment tear-down 2024-06-03T08:40:25.647 INFO:tasks.workunit.client.0.smithi088.stdout: api_c_write_operations: [==========] 8 tests from 2 test suites ran. (152018 ms total) 2024-06-03T08:40:25.647 INFO:tasks.workunit.client.0.smithi088.stdout: api_c_write_operations: [ PASSED ] 8 tests. 2024-06-03T08:40:58.343 INFO:tasks.ceph.mon.a.smithi088.stderr:==00:00:05:53.668 34743== Warning: unimplemented fcntl command: 1036 2024-06-03T08:40:58.359 INFO:tasks.ceph.mon.a.smithi088.stderr:==00:00:05:53.684 34743== Warning: unimplemented fcntl command: 1036 2024-06-03T08:40:58.667 INFO:tasks.ceph.mon.a.smithi088.stderr:==00:00:05:53.992 34743== Warning: unimplemented fcntl command: 1036 2024-06-03T08:40:59.004 INFO:tasks.ceph.mon.b.smithi192.stderr:==00:00:05:54.306 34228== Warning: unimplemented fcntl command: 1036 2024-06-03T08:40:59.007 INFO:tasks.ceph.mon.b.smithi192.stderr:==00:00:05:54.310 34228== Warning: unimplemented fcntl command: 1036 2024-06-03T08:40:59.089 INFO:tasks.ceph.mon.c.smithi088.stderr:==00:00:05:54.414 34749== Warning: unimplemented fcntl command: 1036 2024-06-03T08:40:59.094 INFO:tasks.ceph.mon.c.smithi088.stderr:==00:00:05:54.420 34749== Warning: unimplemented fcntl command: 1036 2024-06-03T08:40:59.328 INFO:tasks.ceph.mon.b.smithi192.stderr:==00:00:05:54.630 34228== Warning: unimplemented fcntl command: 1036 2024-06-03T08:40:59.423 INFO:tasks.ceph.mon.c.smithi088.stderr:==00:00:05:54.748 34749== Warning: unimplemented fcntl command: 1036 2024-06-03T08:41:11.769 INFO:tasks.workunit.client.0.smithi088.stdout:8841d10da0 2024-06-03T08:41:11.769 INFO:tasks.workunit.client.0.smithi088.stdout: api_misc: waiting 0x558841d101a0 2024-06-03T08:41:11.769 INFO:tasks.workunit.client.0.smithi088.stdout: api_misc: done. 2024-06-03T08:41:11.769 INFO:tasks.workunit.client.0.smithi088.stdout: api_misc: [ OK ] LibRadosMiscPool.PoolCreationRace (33344 ms) 2024-06-03T08:41:11.769 INFO:tasks.workunit.client.0.smithi088.stdout: api_misc: [----------] 1 test from LibRadosMiscPool (33344 ms total) 2024-06-03T08:41:11.769 INFO:tasks.workunit.client.0.smithi088.stdout: api_misc: 2024-06-03T08:41:11.769 INFO:tasks.workunit.client.0.smithi088.stdout: api_misc: [----------] 8 tests from LibRadosMisc 2024-06-03T08:41:11.770 INFO:tasks.workunit.client.0.smithi088.stdout: api_misc: [ RUN ] LibRadosMisc.ClusterFSID 2024-06-03T08:41:11.770 INFO:tasks.workunit.client.0.smithi088.stdout: api_misc: [ OK ] LibRadosMisc.ClusterFSID (1 ms) 2024-06-03T08:41:11.770 INFO:tasks.workunit.client.0.smithi088.stdout: api_misc: [ RUN ] LibRadosMisc.Exec 2024-06-03T08:41:11.770 INFO:tasks.workunit.client.0.smithi088.stdout: api_misc: [ OK ] LibRadosMisc.Exec (5124 ms) 2024-06-03T08:41:11.770 INFO:tasks.workunit.client.0.smithi088.stdout: api_misc: [ RUN ] LibRadosMisc.WriteSame 2024-06-03T08:41:11.770 INFO:tasks.workunit.client.0.smithi088.stdout: api_misc: [ OK ] LibRadosMisc.WriteSame (6235 ms) 2024-06-03T08:41:11.770 INFO:tasks.workunit.client.0.smithi088.stdout: api_misc: [ RUN ] LibRadosMisc.CmpExt 2024-06-03T08:41:11.770 INFO:tasks.workunit.client.0.smithi088.stdout: api_misc: [ OK ] LibRadosMisc.CmpExt (2020 ms) 2024-06-03T08:41:11.770 INFO:tasks.workunit.client.0.smithi088.stdout: api_misc: [ RUN ] LibRadosMisc.Applications 2024-06-03T08:41:11.770 INFO:tasks.workunit.client.0.smithi088.stdout: api_misc: [ OK ] LibRadosMisc.Applications (4825 ms) 2024-06-03T08:41:11.770 INFO:tasks.workunit.client.0.smithi088.stdout: api_misc: [ RUN ] LibRadosMisc.MinCompatOSD 2024-06-03T08:41:11.770 INFO:tasks.workunit.client.0.smithi088.stdout: api_misc: [ OK ] LibRadosMisc.MinCompatOSD (0 ms) 2024-06-03T08:41:11.771 INFO:tasks.workunit.client.0.smithi088.stdout: api_misc: [ RUN ] LibRadosMisc.MinCompatClient 2024-06-03T08:41:11.771 INFO:tasks.workunit.client.0.smithi088.stdout: api_misc: [ OK ] LibRadosMisc.MinCompatClient (0 ms) 2024-06-03T08:41:11.771 INFO:tasks.workunit.client.0.smithi088.stdout: api_misc: [ RUN ] LibRadosMisc.ShutdownRace 2024-06-03T08:41:11.771 INFO:tasks.workunit.client.0.smithi088.stdout: api_misc: [ OK ] LibRadosMisc.ShutdownRace (141554 ms) 2024-06-03T08:41:11.771 INFO:tasks.workunit.client.0.smithi088.stdout: api_misc: [----------] 8 tests from LibRadosMisc (159759 ms total) 2024-06-03T08:41:11.771 INFO:tasks.workunit.client.0.smithi088.stdout: api_misc: 2024-06-03T08:41:11.771 INFO:tasks.workunit.client.0.smithi088.stdout: api_misc: [----------] Global test environment tear-down 2024-06-03T08:41:11.771 INFO:tasks.workunit.client.0.smithi088.stdout: api_misc: [==========] 12 tests from 4 test suites ran. (198297 ms total) 2024-06-03T08:41:11.771 INFO:tasks.workunit.client.0.smithi088.stdout: api_misc: [ PASSED ] 12 tests. 2024-06-03T08:41:22.008 INFO:tasks.ceph.mon.a.smithi088.stderr:==00:00:06:17.333 34743== Warning: unimplemented fcntl command: 1036 2024-06-03T08:41:22.009 INFO:tasks.ceph.mon.a.smithi088.stderr:==00:00:06:17.335 34743== Warning: unimplemented fcntl command: 1036 2024-06-03T08:41:22.049 INFO:tasks.ceph.mon.a.smithi088.stderr:==00:00:06:17.375 34743== Warning: unimplemented fcntl command: 1036 2024-06-03T08:41:22.055 INFO:tasks.ceph.mon.b.smithi192.stderr:==00:00:06:17.357 34228== Warning: unimplemented fcntl command: 1036 2024-06-03T08:41:22.057 INFO:tasks.ceph.mon.b.smithi192.stderr:==00:00:06:17.359 34228== Warning: unimplemented fcntl command: 1036 2024-06-03T08:41:22.059 INFO:tasks.ceph.mon.c.smithi088.stderr:==00:00:06:17.385 34749== Warning: unimplemented fcntl command: 1036 2024-06-03T08:41:22.061 INFO:tasks.ceph.mon.c.smithi088.stderr:==00:00:06:17.386 34749== Warning: unimplemented fcntl command: 1036 2024-06-03T08:41:22.102 INFO:tasks.ceph.mon.c.smithi088.stderr:==00:00:06:17.428 34749== Warning: unimplemented fcntl command: 1036 2024-06-03T08:41:22.122 INFO:tasks.ceph.mon.b.smithi192.stderr:==00:00:06:17.424 34228== Warning: unimplemented fcntl command: 1036 2024-06-03T08:41:22.562 INFO:tasks.ceph.osd.2.smithi088.stderr:2024-06-03T08:41:22.561+0000 12a6c640 -1 osd.2 143 get_health_metrics reporting 1 slow ops, oldest is osd_op(client.23997.0:2 83.6 83.7fc1f406 (undecoded) ondisk+write+known_if_redirected+supports_pool_eio e123) 2024-06-03T08:41:23.593 INFO:tasks.ceph.osd.2.smithi088.stderr:2024-06-03T08:41:23.592+0000 12a6c640 -1 osd.2 143 get_health_metrics reporting 1 slow ops, oldest is osd_op(client.23997.0:2 83.6 83.7fc1f406 (undecoded) ondisk+write+known_if_redirected+supports_pool_eio e123) 2024-06-03T08:41:24.595 INFO:tasks.ceph.osd.2.smithi088.stderr:2024-06-03T08:41:24.594+0000 12a6c640 -1 osd.2 144 get_health_metrics reporting 1 slow ops, oldest is osd_op(client.23997.0:2 83.6 83.7fc1f406 (undecoded) ondisk+write+known_if_redirected+supports_pool_eio e123) 2024-06-03T08:41:25.592 INFO:tasks.ceph.osd.2.smithi088.stderr:2024-06-03T08:41:25.591+0000 12a6c640 -1 osd.2 145 get_health_metrics reporting 1 slow ops, oldest is osd_op(client.23997.0:2 83.6 83.7fc1f406 (undecoded) ondisk+write+known_if_redirected+supports_pool_eio e123) 2024-06-03T08:41:26.586 INFO:tasks.ceph.osd.2.smithi088.stderr:2024-06-03T08:41:26.585+0000 12a6c640 -1 osd.2 146 get_health_metrics reporting 1 slow ops, oldest is osd_op(client.23997.0:2 83.6 83.7fc1f406 (undecoded) ondisk+write+known_if_redirected+supports_pool_eio e123) 2024-06-03T08:41:37.224 INFO:tasks.workunit.client.0.smithi088.stdout: api_snapshots: Running main() from gmock_main.cc 2024-06-03T08:41:37.224 INFO:tasks.workunit.client.0.smithi088.stdout: api_snapshots: [==========] Running 13 tests from 4 test suites. 2024-06-03T08:41:37.224 INFO:tasks.workunit.client.0.smithi088.stdout: api_snapshots: [----------] Global test environment set-up. 2024-06-03T08:41:37.224 INFO:tasks.workunit.client.0.smithi088.stdout: api_snapshots: [----------] 4 tests from LibRadosSnapshots 2024-06-03T08:41:37.224 INFO:tasks.workunit.client.0.smithi088.stdout: api_snapshots: [ RUN ] LibRadosSnapshots.SnapList 2024-06-03T08:41:37.224 INFO:tasks.workunit.client.0.smithi088.stdout: api_snapshots: [ OK ] LibRadosSnapshots.SnapList (19621 ms) 2024-06-03T08:41:37.224 INFO:tasks.workunit.client.0.smithi088.stdout: api_snapshots: [ RUN ] LibRadosSnapshots.SnapRemove 2024-06-03T08:41:37.224 INFO:tasks.workunit.client.0.smithi088.stdout: api_snapshots: [ OK ] LibRadosSnapshots.SnapRemove (10896 ms) 2024-06-03T08:41:37.224 INFO:tasks.workunit.client.0.smithi088.stdout: api_snapshots: [ RUN ] LibRadosSnapshots.Rollback 2024-06-03T08:41:37.225 INFO:tasks.workunit.client.0.smithi088.stdout: api_snapshots: [ OK ] LibRadosSnapshots.Rollback (8058 ms) 2024-06-03T08:41:37.225 INFO:tasks.workunit.client.0.smithi088.stdout: api_snapshots: [ RUN ] LibRadosSnapshots.SnapGetName 2024-06-03T08:41:37.225 INFO:tasks.workunit.client.0.smithi088.stdout: api_snapshots: [ OK ] LibRadosSnapshots.SnapGetName (8297 ms) 2024-06-03T08:41:37.225 INFO:tasks.workunit.client.0.smithi088.stdout: api_snapshots: [----------] 4 tests from LibRadosSnapshots (46872 ms total) 2024-06-03T08:41:37.225 INFO:tasks.workunit.client.0.smithi088.stdout: api_snapshots: 2024-06-03T08:41:37.225 INFO:tasks.workunit.client.0.smithi088.stdout: api_snapshots: [----------] 3 tests from LibRadosSnapshotsSelfManaged 2024-06-03T08:41:37.225 INFO:tasks.workunit.client.0.smithi088.stdout: api_snapshots: [ RUN ] LibRadosSnapshotsSelfManaged.Snap 2024-06-03T08:41:37.225 INFO:tasks.workunit.client.0.smithi088.stdout: api_snapshots: [ OK ] LibRadosSnapshotsSelfManaged.Snap (21743 ms) 2024-06-03T08:41:37.225 INFO:tasks.workunit.client.0.smithi088.stdout: api_snapshots: [ RUN ] LibRadosSnapshotsSelfManaged.Rollback 2024-06-03T08:41:37.225 INFO:tasks.workunit.client.0.smithi088.stdout: api_snapshots: [ OK ] LibRadosSnapshotsSelfManaged.Rollback (22620 ms) 2024-06-03T08:41:37.225 INFO:tasks.workunit.client.0.smithi088.stdout: api_snapshots: [ RUN ] LibRadosSnapshotsSelfManaged.FutureSnapRollback 2024-06-03T08:41:37.225 INFO:tasks.workunit.client.0.smithi088.stdout: api_snapshots: [ OK ] LibRadosSnapshotsSelfManaged.FutureSnapRollback (27437 ms) 2024-06-03T08:41:37.226 INFO:tasks.workunit.client.0.smithi088.stdout: api_snapshots: [----------] 3 tests from LibRadosSnapshotsSelfManaged (71800 ms total) 2024-06-03T08:41:37.226 INFO:tasks.workunit.client.0.smithi088.stdout: api_snapshots: 2024-06-03T08:41:37.226 INFO:tasks.workunit.client.0.smithi088.stdout: api_snapshots: [----------] 4 tests from LibRadosSnapshotsEC 2024-06-03T08:41:37.226 INFO:tasks.workunit.client.0.smithi088.stdout: api_snapshots: [ RUN ] LibRadosSnapshotsEC.SnapList 2024-06-03T08:41:37.226 INFO:tasks.workunit.client.0.smithi088.stdout: api_snapshots: [ OK ] LibRadosSnapshotsEC.SnapList (11263 ms) 2024-06-03T08:41:37.226 INFO:tasks.workunit.client.0.smithi088.stdout: api_snapshots: [ RUN ] LibRadosSnapshotsEC.SnapRemove 2024-06-03T08:41:37.226 INFO:tasks.workunit.client.0.smithi088.stdout: api_snapshots: [ OK ] LibRadosSnapshotsEC.SnapRemove (8961 ms) 2024-06-03T08:41:37.226 INFO:tasks.workunit.client.0.smithi088.stdout: api_snapshots: [ RUN ] LibRadosSnapshotsEC.Rollback 2024-06-03T08:41:37.226 INFO:tasks.workunit.client.0.smithi088.stdout: api_snapshots: [ OK ] LibRadosSnapshotsEC.Rollback (9797 ms) 2024-06-03T08:41:37.226 INFO:tasks.workunit.client.0.smithi088.stdout: api_snapshots: [ RUN ] LibRadosSnapshotsEC.SnapGetName 2024-06-03T08:41:37.226 INFO:tasks.workunit.client.0.smithi088.stdout: api_snapshots: [ OK ] LibRadosSnapshotsEC.SnapGetName (8847 ms) 2024-06-03T08:41:37.226 INFO:tasks.workunit.client.0.smithi088.stdout: api_snapshots: [----------] 4 tests from LibRadosSnapshotsEC (38868 ms total) 2024-06-03T08:41:37.227 INFO:tasks.workunit.client.0.smithi088.stdout: api_snapshots: 2024-06-03T08:41:37.227 INFO:tasks.workunit.client.0.smithi088.stdout: api_snapshots: [----------] 2 tests from LibRadosSnapshotsSelfManagedEC 2024-06-03T08:41:37.227 INFO:tasks.workunit.client.0.smithi088.stdout: api_snapshots: [ RUN ] LibRadosSnapshotsSelfManagedEC.Snap 2024-06-03T08:41:37.227 INFO:tasks.workunit.client.0.smithi088.stdout: api_snapshots: [ OK ] LibRadosSnapshotsSelfManagedEC.Snap (4876 ms) 2024-06-03T08:41:37.227 INFO:tasks.workunit.client.0.smithi088.stdout: api_snapshots: [ RUN ] LibRadosSnapshotsSelfManagedEC.Rollback 2024-06-03T08:41:37.227 INFO:tasks.workunit.client.0.smithi088.stdout: api_snapshots: [ OK ] LibRadosSnapshotsSelfManagedEC.Rollback (17179 ms) 2024-06-03T08:41:37.227 INFO:tasks.workunit.client.0.smithi088.stdout: api_snapshots: [----------] 2 tests from LibRadosSnapshotsSelfManagedEC (22055 ms total) 2024-06-03T08:41:37.227 INFO:tasks.workunit.client.0.smithi088.stdout: api_snapshots: 2024-06-03T08:41:37.227 INFO:tasks.workunit.client.0.smithi088.stdout: api_snapshots: [----------] Global test environment tear-down 2024-06-03T08:41:37.227 INFO:tasks.workunit.client.0.smithi088.stdout: api_snapshots: [==========] 13 tests from 4 test suites ran. (223765 ms total) 2024-06-03T08:41:37.227 INFO:tasks.workunit.client.0.smithi088.stdout: api_snapshots: [ PASSED ] 13 tests. 2024-06-03T08:41:42.142 INFO:tasks.ceph.mon.a.smithi088.stderr:==00:00:06:37.467 34743== Warning: unimplemented fcntl command: 1036 2024-06-03T08:41:42.146 INFO:tasks.ceph.mon.a.smithi088.stderr:==00:00:06:37.471 34743== Warning: unimplemented fcntl command: 1036 2024-06-03T08:41:42.194 INFO:tasks.ceph.mon.a.smithi088.stderr:==00:00:06:37.519 34743== Warning: unimplemented fcntl command: 1036 2024-06-03T08:41:42.196 INFO:tasks.ceph.mon.c.smithi088.stderr:==00:00:06:37.522 34749== Warning: unimplemented fcntl command: 1036 2024-06-03T08:41:42.198 INFO:tasks.ceph.mon.c.smithi088.stderr:==00:00:06:37.524 34749== Warning: unimplemented fcntl command: 1036 2024-06-03T08:41:42.216 INFO:tasks.ceph.mon.b.smithi192.stderr:==00:00:06:37.518 34228== Warning: unimplemented fcntl command: 1036 2024-06-03T08:41:42.219 INFO:tasks.ceph.mon.b.smithi192.stderr:==00:00:06:37.521 34228== Warning: unimplemented fcntl command: 1036 2024-06-03T08:41:42.246 INFO:tasks.ceph.mon.c.smithi088.stderr:==00:00:06:37.572 34749== Warning: unimplemented fcntl command: 1036 2024-06-03T08:41:42.267 INFO:tasks.ceph.mon.b.smithi192.stderr:==00:00:06:37.569 34228== Warning: unimplemented fcntl command: 1036 2024-06-03T08:41:42.738 INFO:tasks.workunit.client.0.smithi088.stdout: api_snapshots_pp: Running main() from gmock_main.cc 2024-06-03T08:41:42.739 INFO:tasks.workunit.client.0.smithi088.stdout: api_snapshots_pp: [==========] Running 21 tests from 5 test suites. 2024-06-03T08:41:42.739 INFO:tasks.workunit.client.0.smithi088.stdout: api_snapshots_pp: [----------] Global test environment set-up. 2024-06-03T08:41:42.739 INFO:tasks.workunit.client.0.smithi088.stdout: api_snapshots_pp: [----------] 5 tests from LibRadosSnapshotsPP 2024-06-03T08:41:42.739 INFO:tasks.workunit.client.0.smithi088.stdout: api_snapshots_pp: seed 40616 2024-06-03T08:41:42.739 INFO:tasks.workunit.client.0.smithi088.stdout: api_snapshots_pp: [ RUN ] LibRadosSnapshotsPP.SnapListPP 2024-06-03T08:41:42.739 INFO:tasks.workunit.client.0.smithi088.stdout: api_snapshots_pp: [ OK ] LibRadosSnapshotsPP.SnapListPP (19614 ms) 2024-06-03T08:41:42.739 INFO:tasks.workunit.client.0.smithi088.stdout: api_snapshots_pp: [ RUN ] LibRadosSnapshotsPP.SnapRemovePP 2024-06-03T08:41:42.739 INFO:tasks.workunit.client.0.smithi088.stdout: api_snapshots_pp: [ OK ] LibRadosSnapshotsPP.SnapRemovePP (6458 ms) 2024-06-03T08:41:42.739 INFO:tasks.workunit.client.0.smithi088.stdout: api_snapshots_pp: [ RUN ] LibRadosSnapshotsPP.RollbackPP 2024-06-03T08:41:42.739 INFO:tasks.workunit.client.0.smithi088.stdout: api_snapshots_pp: [ OK ] LibRadosSnapshotsPP.RollbackPP (10964 ms) 2024-06-03T08:41:42.739 INFO:tasks.workunit.client.0.smithi088.stdout: api_snapshots_pp: [ RUN ] LibRadosSnapshotsPP.SnapGetNamePP 2024-06-03T08:41:42.740 INFO:tasks.workunit.client.0.smithi088.stdout: api_snapshots_pp: [ OK ] LibRadosSnapshotsPP.SnapGetNamePP (5676 ms) 2024-06-03T08:41:42.740 INFO:tasks.workunit.client.0.smithi088.stdout: api_snapshots_pp: [ RUN ] LibRadosSnapshotsPP.SnapCreateRemovePP 2024-06-03T08:41:42.740 INFO:tasks.workunit.client.0.smithi088.stdout: api_snapshots_pp: [ OK ] LibRadosSnapshotsPP.SnapCreateRemovePP (8147 ms) 2024-06-03T08:41:42.740 INFO:tasks.workunit.client.0.smithi088.stdout: api_snapshots_pp: [----------] 5 tests from LibRadosSnapshotsPP (50859 ms total) 2024-06-03T08:41:42.740 INFO:tasks.workunit.client.0.smithi088.stdout: api_snapshots_pp: 2024-06-03T08:41:42.740 INFO:tasks.workunit.client.0.smithi088.stdout: api_snapshots_pp: [----------] 7 tests from LibRadosSnapshotsSelfManagedPP 2024-06-03T08:41:42.740 INFO:tasks.workunit.client.0.smithi088.stdout: api_snapshots_pp: [ RUN ] LibRadosSnapshotsSelfManagedPP.SnapPP 2024-06-03T08:41:42.740 INFO:tasks.workunit.client.0.smithi088.stdout: api_snapshots_pp: [ OK ] LibRadosSnapshotsSelfManagedPP.SnapPP (23625 ms) 2024-06-03T08:41:42.740 INFO:tasks.workunit.client.0.smithi088.stdout: api_snapshots_pp: [ RUN ] LibRadosSnapshotsSelfManagedPP.RollbackPP 2024-06-03T08:41:42.740 INFO:tasks.workunit.client.0.smithi088.stdout: api_snapshots_pp: [ OK ] LibRadosSnapshotsSelfManagedPP.RollbackPP (21386 ms) 2024-06-03T08:41:42.740 INFO:tasks.workunit.client.0.smithi088.stdout: api_snapshots_pp: [ RUN ] LibRadosSnapshotsSelfManagedPP.SnapOverlapPP 2024-06-03T08:41:42.740 INFO:tasks.workunit.client.0.smithi088.stdout: api_snapshots_pp: [ OK ] LibRadosSnapshotsSelfManagedPP.SnapOverlapPP (31634 ms) 2024-06-03T08:41:42.741 INFO:tasks.workunit.client.0.smithi088.stdout: api_snapshots_pp: [ RUN ] LibRadosSnapshotsSelfManagedPP.Bug11677 2024-06-03T08:41:42.741 INFO:tasks.workunit.client.0.smithi088.stdout: api_snapshots_pp: [ OK ] LibRadosSnapshotsSelfManagedPP.Bug11677 (19406 ms) 2024-06-03T08:41:42.741 INFO:tasks.workunit.client.0.smithi088.stdout: api_snapshots_pp: [ RUN ] LibRadosSnapshotsSelfManagedPP.OrderSnap 2024-06-03T08:41:42.741 INFO:tasks.workunit.client.0.smithi088.stdout: api_snapshots_pp: [ OK ] LibRadosSnapshotsSelfManagedPP.OrderSnap (9298 ms) 2024-06-03T08:41:42.741 INFO:tasks.workunit.client.0.smithi088.stdout: api_snapshots_pp: [ RUN ] LibRadosSnapshotsSelfManagedPP.WriteRollback 2024-06-03T08:41:42.741 INFO:tasks.workunit.client.0.smithi088.stdout: api_snapshots_pp: /home/jenkins-build/build/workspace/ceph-dev-build/ARCH/x86_64/AVAILABLE_ARCH/x86_64/AVAILABLE_DIST/centos9/DIST/centos9/MACHINE_SIZE/gigantic/release/18.2.2-1389-gf1abf795/rpm/el9/BUILD/ceph-18.2.2-1389-gf1abf795/src/test/librados/snapshots_cxx.cc:463: Skipped 2024-06-03T08:41:42.741 INFO:tasks.workunit.client.0.smithi088.stdout: api_snapshots_pp: 2024-06-03T08:41:42.741 INFO:tasks.workunit.client.0.smithi088.stdout: api_snapshots_pp: [ SKIPPED ] LibRadosSnapshotsSelfManagedPP.WriteRollback (0 ms) 2024-06-03T08:41:42.741 INFO:tasks.workunit.client.0.smithi088.stdout: api_snapshots_pp: [ RUN ] LibRadosSnapshotsSelfManagedPP.ReusePurgedSnap 2024-06-03T08:41:42.741 INFO:tasks.workunit.client.0.smithi088.stdout: api_snapshots_pp: deleting snap 23 in pool LibRadosSnapshotsSelfManagedPP_smithi088-40616-7 2024-06-03T08:41:42.741 INFO:tasks.workunit.client.0.smithi088.stdout: api_snapshots_pp: waiting for snaps to purge 2024-06-03T08:41:42.741 INFO:tasks.workunit.client.0.smithi088.stdout: api_snapshots_pp: [ OK ] LibRadosSnapshotsSelfManagedPP.ReusePurgedSnap (46756 ms) 2024-06-03T08:41:42.741 INFO:tasks.workunit.client.0.smithi088.stdout: api_snapshots_pp: [----------] 7 tests from LibRadosSnapshotsSelfManagedPP (152105 ms total) 2024-06-03T08:41:42.742 INFO:tasks.workunit.client.0.smithi088.stdout: api_snapshots_pp: 2024-06-03T08:41:42.742 INFO:tasks.workunit.client.0.smithi088.stdout: api_snapshots_pp: [----------] 2 tests from LibRadosPoolIsInSelfmanagedSnapsMode 2024-06-03T08:41:42.742 INFO:tasks.workunit.client.0.smithi088.stdout: api_snapshots_pp: [ RUN ] LibRadosPoolIsInSelfmanagedSnapsMode.NotConnected 2024-06-03T08:41:42.742 INFO:tasks.workunit.client.0.smithi088.stdout: api_snapshots_pp: [ OK ] LibRadosPoolIsInSelfmanagedSnapsMode.NotConnected (8 ms) 2024-06-03T08:41:42.742 INFO:tasks.workunit.client.0.smithi088.stdout: api_snapshots_pp: [ RUN ] LibRadosPoolIsInSelfmanagedSnapsMode.FreshInstance 2024-06-03T08:41:42.742 INFO:tasks.workunit.client.0.smithi088.stdout: api_snapshots_pp: [ OK ] LibRadosPoolIsInSelfmanagedSnapsMode.FreshInstance (5706 ms) 2024-06-03T08:41:42.742 INFO:tasks.workunit.client.0.smithi088.stdout: api_snapshots_pp: [----------] 2 tests from LibRadosPoolIsInSelfmanagedSnapsMode (5714 ms total) 2024-06-03T08:41:42.742 INFO:tasks.workunit.client.0.smithi088.stdout: api_snapshots_pp: 2024-06-03T08:41:42.742 INFO:tasks.workunit.client.0.smithi088.stdout: api_snapshots_pp: [----------] 4 tests from LibRadosSnapshotsECPP 2024-06-03T08:41:42.742 INFO:tasks.workunit.client.0.smithi088.stdout: api_snapshots_pp: [ RUN ] LibRadosSnapshotsECPP.SnapListPP 2024-06-03T08:41:42.742 INFO:tasks.workunit.client.0.smithi088.stdout: api_snapshots_pp: [ OK ] LibRadosSnapshotsECPP.SnapListPP (4755 ms) 2024-06-03T08:41:42.742 INFO:tasks.workunit.client.0.smithi088.stdout: api_snapshots_pp: [ RUN ] LibRadosSnapshotsECPP.SnapRemovePP 2024-06-03T08:42:02.187 INFO:tasks.ceph.mon.a.smithi088.stderr:==00:00:06:57.512 34743== Warning: unimplemented fcntl command: 1036 2024-06-03T08:42:02.238 INFO:tasks.ceph.mon.a.smithi088.stderr:==00:00:06:57.563 34743== Warning: unimplemented fcntl command: 1036 2024-06-03T08:42:02.283 INFO:tasks.ceph.mon.a.smithi088.stderr:==00:00:06:57.609 34743== Warning: unimplemented fcntl command: 1036 2024-06-03T08:42:02.284 INFO:tasks.ceph.mon.b.smithi192.stderr:==00:00:06:57.586 34228== Warning: unimplemented fcntl command: 1036 2024-06-03T08:42:02.285 INFO:tasks.ceph.mon.b.smithi192.stderr:==00:00:06:57.588 34228== Warning: unimplemented fcntl command: 1036 2024-06-03T08:42:02.287 INFO:tasks.ceph.mon.c.smithi088.stderr:==00:00:06:57.612 34749== Warning: unimplemented fcntl command: 1036 2024-06-03T08:42:02.290 INFO:tasks.ceph.mon.c.smithi088.stderr:==00:00:06:57.616 34749== Warning: unimplemented fcntl command: 1036 2024-06-03T08:42:02.317 INFO:tasks.ceph.mon.b.smithi192.stderr:==00:00:06:57.619 34228== Warning: unimplemented fcntl command: 1036 2024-06-03T08:42:02.321 INFO:tasks.ceph.mon.c.smithi088.stderr:==00:00:06:57.647 34749== Warning: unimplemented fcntl command: 1036 2024-06-03T08:42:09.210 INFO:tasks.workunit.client.0.smithi088.stdout: api_snapshots_pp: [ OK ] LibRad api_misc_pp: [==========] Running 31 tests from 7 test suites. 2024-06-03T08:42:09.210 INFO:tasks.workunit.client.0.smithi088.stdout: api_misc_pp: [----------] Global test environment set-up. 2024-06-03T08:42:09.210 INFO:tasks.workunit.client.0.smithi088.stdout: api_misc_pp: [----------] 1 test from LibRadosMiscVersion 2024-06-03T08:42:09.210 INFO:tasks.workunit.client.0.smithi088.stdout: api_misc_pp: [ RUN ] LibRadosMiscVersion.VersionPP 2024-06-03T08:42:09.210 INFO:tasks.workunit.client.0.smithi088.stdout: api_misc_pp: [ OK ] LibRadosMiscVersion.VersionPP (0 ms) 2024-06-03T08:42:09.210 INFO:tasks.workunit.client.0.smithi088.stdout: api_misc_pp: [----------] 1 test from LibRadosMiscVersion (0 ms total) 2024-06-03T08:42:09.210 INFO:tasks.workunit.client.0.smithi088.stdout: api_misc_pp: 2024-06-03T08:42:09.210 INFO:tasks.workunit.client.0.smithi088.stdout: api_misc_pp: [----------] 22 tests from LibRadosMiscPP 2024-06-03T08:42:09.210 INFO:tasks.workunit.client.0.smithi088.stdout: api_misc_pp: seed 40418 2024-06-03T08:42:09.211 INFO:tasks.workunit.client.0.smithi088.stdout: api_misc_pp: [ RUN ] LibRadosMiscPP.WaitOSDMapPP 2024-06-03T08:42:09.211 INFO:tasks.workunit.client.0.smithi088.stdout: api_misc_pp: [ OK ] LibRadosMiscPP.WaitOSDMapPP (15 ms) 2024-06-03T08:42:09.211 INFO:tasks.workunit.client.0.smithi088.stdout: api_misc_pp: [ RUN ] LibRadosMiscPP.LongNamePP 2024-06-03T08:42:09.211 INFO:tasks.workunit.client.0.smithi088.stdout: api_misc_pp: [ OK ] LibRadosMiscPP.LongNamePP (32823 ms) 2024-06-03T08:42:09.211 INFO:tasks.workunit.client.0.smithi088.stdout: api_misc_pp: [ RUN ] LibRadosMiscPP.LongLocatorPP 2024-06-03T08:42:09.211 INFO:tasks.workunit.client.0.smithi088.stdout: api_misc_pp: [ OK ] LibRadosMiscPP.LongLocatorPP (8086 ms) 2024-06-03T08:42:09.211 INFO:tasks.workunit.client.0.smithi088.stdout: api_misc_pp: [ RUN ] LibRadosMiscPP.LongNSpacePP 2024-06-03T08:42:09.211 INFO:tasks.workunit.client.0.smithi088.stdout: api_misc_pp: [ OK ] LibRadosMiscPP.LongNSpacePP (8200 ms) 2024-06-03T08:42:09.211 INFO:tasks.workunit.client.0.smithi088.stdout: api_misc_pp: [ RUN ] LibRadosMiscPP.LongAttrNamePP 2024-06-03T08:42:09.211 INFO:tasks.workunit.client.0.smithi088.stdout: api_misc_pp: [ OK ] LibRadosMiscPP.LongAttrNamePP (6182 ms) 2024-06-03T08:42:09.211 INFO:tasks.workunit.client.0.smithi088.stdout: api_misc_pp: [ RUN ] LibRadosMiscPP.ExecPP 2024-06-03T08:42:09.211 INFO:tasks.workunit.client.0.smithi088.stdout: api_misc_pp: [ OK ] LibRadosMiscPP.ExecPP (348 ms) 2024-06-03T08:42:09.211 INFO:tasks.workunit.client.0.smithi088.stdout: api_misc_pp: [ RUN ] LibRadosMiscPP.BadFlagsPP 2024-06-03T08:42:09.211 INFO:tasks.workunit.client.0.smithi088.stdout: api_misc_pp: [ OK ] LibRadosMiscPP.BadFlagsPP (440 ms) 2024-06-03T08:42:09.211 INFO:tasks.workunit.client.0.smithi088.stdout: api_misc_pp: [ RUN ] LibRadosMiscPP.Operate1PP 2024-06-03T08:42:09.212 INFO:tasks.workunit.client.0.smithi088.stdout: api_misc_pp: [ OK ] LibRadosMiscPP.Operate1PP (944 ms) 2024-06-03T08:42:09.212 INFO:tasks.workunit.client.0.smithi088.stdout: api_misc_pp: [ RUN ] LibRadosMiscPP.Operate2PP 2024-06-03T08:42:09.212 INFO:tasks.workunit.client.0.smithi088.stdout: api_misc_pp: [ OK ] LibRadosMiscPP.Operate2PP (46 ms) 2024-06-03T08:42:09.212 INFO:tasks.workunit.client.0.smithi088.stdout: api_misc_pp: [ RUN ] LibRadosMiscPP.BigObjectPP 2024-06-03T08:42:09.212 INFO:tasks.workunit.client.0.smithi088.stdout: api_misc_pp: [ OK ] LibRadosMiscPP.BigObjectPP (253 ms) 2024-06-03T08:42:09.212 INFO:tasks.workunit.client.0.smithi088.stdout: api_misc_pp: [ RUN ] LibRadosMiscPP.AioOperatePP 2024-06-03T08:42:09.212 INFO:tasks.workunit.client.0.smithi088.stdout: api_misc_pp: [ OK ] LibRadosMiscPP.AioOperatePP (55 ms) 2024-06-03T08:42:09.212 INFO:tasks.workunit.client.0.smithi088.stdout: api_misc_pp: [ RUN ] LibRadosMiscPP.AssertExistsPP 2024-06-03T08:42:09.212 INFO:tasks.workunit.client.0.smithi088.stdout: api_misc_pp: [ OK ] LibRadosMiscPP.AssertExistsPP (1567 ms) 2024-06-03T08:42:09.212 INFO:tasks.workunit.client.0.smithi088.stdout: api_misc_pp: [ RUN ] LibRadosMiscPP.AssertVersionPP 2024-06-03T08:42:09.212 INFO:tasks.workunit.client.0.smithi088.stdout: api_misc_pp: [ OK ] LibRadosMiscPP.AssertVersionPP (783 ms) 2024-06-03T08:42:09.212 INFO:tasks.workunit.client.0.smithi088.stdout: api_misc_pp: [ RUN ] LibRadosMiscPP.BigAttrPP 2024-06-03T08:42:09.212 INFO:tasks.workunit.client.0.smithi088.stdout: api_misc_pp: osd_max_attr_size = 0 2024-06-03T08:42:09.212 INFO:tasks.workunit.client.0.smithi088.stdout: api_misc_pp: osd_max_attr_size == 0; skipping test 2024-06-03T08:42:09.212 INFO:tasks.workunit.client.0.smithi088.stdout: api_misc_pp: [ OK ] LibRadosMiscPP.BigAttrPP (85056 ms) 2024-06-03T08:42:09.213 INFO:tasks.workunit.client.0.smithi088.stdout: api_misc_pp: [ RUN ] LibRadosMiscPP.CopyPP 2024-06-03T08:42:09.213 INFO:tasks.workunit.client.0.smithi088.stdout: api_misc_pp: [ OK ] LibRadosMiscPP.CopyPP (9801 ms) 2024-06-03T08:42:09.213 INFO:tasks.workunit.client.0.smithi088.stdout: api_misc_pp: [ RUN ] LibRadosMiscPP.CopyScrubPP 2024-06-03T08:42:09.213 INFO:tasks.workunit.client.0.smithi088.stdout: api_misc_pp: waiting for initial deep scrubs... 2024-06-03T08:42:09.213 INFO:tasks.workunit.client.0.smithi088.stdout: api_misc_pp: done waiting, doing copies 2024-06-03T08:42:09.213 INFO:tasks.workunit.client.0.smithi088.stdout: api_misc_pp: waiting for final deep scrubs... 2024-06-03T08:42:09.213 INFO:tasks.workunit.client.0.smithi088.stdout: api_misc_pp: done waiting 2024-06-03T08:42:09.213 INFO:tasks.workunit.client.0.smithi088.stdout: api_misc_pp: [ OK ] LibRadosMiscPP.CopyScrubPP (91233 ms) 2024-06-03T08:42:09.213 INFO:tasks.workunit.client.0.smithi088.stdout: api_misc_pp: [ RUN ] LibRadosMiscPP.WriteSamePP 2024-06-03T08:42:09.213 INFO:tasks.workunit.client.0.smithi088.stdout: api_misc_pp: [ OK ] LibRadosMiscPP.WriteSamePP (94 ms) 2024-06-03T08:42:09.213 INFO:tasks.workunit.client.0.smithi088.stdout: api_misc_pp: [ RUN ] LibRadosMiscPP.CmpExtPP 2024-06-03T08:42:09.213 INFO:tasks.workunit.client.0.smithi088.stdout: api_misc_pp: [ OK ] LibRadosMiscPP.CmpExtPP (39 ms) 2024-06-03T08:42:09.213 INFO:tasks.workunit.client.0.smithi088.stdout: api_misc_pp: [ RUN ] LibRadosMiscPP.Applications 2024-06-03T08:42:09.213 INFO:tasks.workunit.client.0.smithi088.stdout: api_misc_pp: [ OK ] LibRadosMiscPP.Applications (4738 ms) 2024-06-03T08:42:09.214 INFO:tasks.workunit.client.0.smithi088.stdout: api_misc_pp: [ RUN ] LibRadosMiscPP.MinCompatOSD 2024-06-03T08:42:09.214 INFO:tasks.workunit.client.0.smithi088.stdout: api_misc_pp: [ OK ] LibRadosMiscPP.MinCompatOSD (0 ms) 2024-06-03T08:42:09.214 INFO:tasks.workunit.client.0.smithi088.stdout: api_misc_pp: [ RUN ] LibRadosMiscPP.MinCompatClient 2024-06-03T08:42:09.214 INFO:tasks.workunit.client.0.smithi088.stdout: api_misc_pp: [ OK ] LibRadosMiscPP.MinCompatClient (0 ms) 2024-06-03T08:42:09.214 INFO:tasks.workunit.client.0.smithi088.stdout: api_misc_pp: [ RUN ] LibRadosMiscPP.Conf 2024-06-03T08:42:09.214 INFO:tasks.workunit.client.0.smithi088.stdout: api_misc_pp: [ OK ] LibRadosMiscPP.Conf (0 ms) 2024-06-03T08:42:22.271 INFO:tasks.ceph.mon.a.smithi088.stderr:==00:00:07:17.596 34743== Warning: unimplemented fcntl command: 1036 2024-06-03T08:42:22.321 INFO:tasks.ceph.mon.b.smithi192.stderr:==00:00:07:17.623 34228== Warning: unimplemented fcntl command: 1036 2024-06-03T08:42:22.325 INFO:tasks.ceph.mon.b.smithi192.stderr:==00:00:07:17.627 34228== Warning: unimplemented fcntl command: 1036 2024-06-03T08:42:22.326 INFO:tasks.ceph.mon.a.smithi088.stderr:==00:00:07:17.651 34743== Warning: unimplemented fcntl command: 1036 2024-06-03T08:42:22.377 INFO:tasks.ceph.mon.b.smithi192.stderr:==00:00:07:17.679 34228== Warning: unimplemented fcntl command: 1036 2024-06-03T08:42:22.377 INFO:tasks.ceph.mon.c.smithi088.stderr:==00:00:07:17.702 34749== Warning: unimplemented fcntl command: 1036 2024-06-03T08:42:22.378 INFO:tasks.ceph.mon.c.smithi088.stderr:==00:00:07:17.704 34749== Warning: unimplemented fcntl command: 1036 2024-06-03T08:42:22.398 INFO:tasks.ceph.mon.a.smithi088.stderr:==00:00:07:17.724 34743== Warning: unimplemented fcntl command: 1036 2024-06-03T08:42:22.439 INFO:tasks.ceph.mon.c.smithi088.stderr:==00:00:07:17.764 34749== Warning: unimplemented fcntl command: 1036 2024-06-03T08:42:42.453 INFO:tasks.ceph.mon.a.smithi088.stderr:==00:00:07:37.779 34743== Warning: unimplemented fcntl command: 1036 2024-06-03T08:42:42.520 INFO:tasks.ceph.mon.a.smithi088.stderr:==00:00:07:37.845 34743== Warning: unimplemented fcntl command: 1036 2024-06-03T08:42:42.554 INFO:tasks.ceph.mon.b.smithi192.stderr:==00:00:07:37.856 34228== Warning: unimplemented fcntl command: 1036 2024-06-03T08:42:42.556 INFO:tasks.ceph.mon.b.smithi192.stderr:==00:00:07:37.858 34228== Warning: unimplemented fcntl command: 1036 2024-06-03T08:42:42.567 INFO:tasks.ceph.mon.a.smithi088.stderr:==00:00:07:37.893 34743== Warning: unimplemented fcntl command: 1036 2024-06-03T08:42:42.587 INFO:tasks.ceph.mon.c.smithi088.stderr:==00:00:07:37.912 34749== Warning: unimplemented fcntl command: 1036 2024-06-03T08:42:42.588 INFO:tasks.ceph.mon.c.smithi088.stderr:==00:00:07:37.914 34749== Warning: unimplemented fcntl command: 1036 2024-06-03T08:42:42.602 INFO:tasks.ceph.mon.b.smithi192.stderr:==00:00:07:37.904 34228== Warning: unimplemented fcntl command: 1036 2024-06-03T08:42:42.644 INFO:tasks.ceph.mon.c.smithi088.stderr:==00:00:07:37.970 34749== Warning: unimplemented fcntl command: 1036 2024-06-03T08:42:48.983 INFO:tasks.ceph.osd.4.smithi192.stderr:2024-06-03T08:42:48.977+0000 12a6c640 -1 osd.4 206 get_health_metrics reporting 1 slow ops, oldest is osd_op(client.30120.0:6 107.0s0 107.201c25d0 (undecoded) ondisk+write+known_if_redirected+supports_pool_eio e184) 2024-06-03T08:42:50.007 INFO:tasks.ceph.osd.4.smithi192.stderr:2024-06-03T08:42:50.007+0000 12a6c640 -1 osd.4 206 get_health_metrics reporting 1 slow ops, oldest is osd_op(client.30120.0:6 107.0s0 107.201c25d0 (undecoded) ondisk+write+known_if_redirected+supports_pool_eio e184) 2024-06-03T08:42:51.037 INFO:tasks.ceph.osd.4.smithi192.stderr:2024-06-03T08:42:51.036+0000 12a6c640 -1 osd.4 206 get_health_metrics reporting 1 slow ops, oldest is osd_op(client.30120.0:6 107.0s0 107.201c25d0 (undecoded) ondisk+write+known_if_redirected+supports_pool_eio e184) 2024-06-03T08:42:51.988 INFO:tasks.ceph.osd.4.smithi192.stderr:2024-06-03T08:42:51.988+0000 12a6c640 -1 osd.4 206 get_health_metrics reporting 1 slow ops, oldest is osd_op(client.30120.0:6 107.0s0 107.201c25d0 (undecoded) ondisk+write+known_if_redirected+supports_pool_eio e184) 2024-06-03T08:42:53.015 INFO:tasks.ceph.osd.4.smithi192.stderr:2024-06-03T08:42:53.014+0000 12a6c640 -1 osd.4 206 get_health_metrics reporting 1 slow ops, oldest is osd_op(client.30120.0:6 107.0s0 107.201c25d0 (undecoded) ondisk+write+known_if_redirected+supports_pool_eio e184) 2024-06-03T08:42:54.047 INFO:tasks.ceph.osd.4.smithi192.stderr:2024-06-03T08:42:54.045+0000 12a6c640 -1 osd.4 206 get_health_metrics reporting 1 slow ops, oldest is osd_op(client.30120.0:6 107.0s0 107.201c25d0 (undecoded) ondisk+write+known_if_redirected+supports_pool_eio e184) 2024-06-03T08:42:55.054 INFO:tasks.ceph.osd.4.smithi192.stderr:2024-06-03T08:42:55.051+0000 12a6c640 -1 osd.4 206 get_health_metrics reporting 1 slow ops, oldest is osd_op(client.30120.0:6 107.0s0 107.201c25d0 (undecoded) ondisk+write+known_if_redirected+supports_pool_eio e184) 2024-06-03T08:42:56.031 INFO:tasks.ceph.osd.4.smithi192.stderr:2024-06-03T08:42:56.029+0000 12a6c640 -1 osd.4 206 get_health_metrics reporting 2 slow ops, oldest is osd_op(client.30120.0:6 107.0s0 107.201c25d0 (undecoded) ondisk+write+known_if_redirected+supports_pool_eio e184) 2024-06-03T08:42:57.078 INFO:tasks.ceph.osd.4.smithi192.stderr:2024-06-03T08:42:57.077+0000 12a6c640 -1 osd.4 206 get_health_metrics reporting 2 slow ops, oldest is osd_op(client.30120.0:6 107.0s0 107.201c25d0 (undecoded) ondisk+write+known_if_redirected+supports_pool_eio e184) 2024-06-03T08:42:58.115 INFO:tasks.ceph.osd.4.smithi192.stderr:2024-06-03T08:42:58.113+0000 12a6c640 -1 osd.4 206 get_health_metrics reporting 2 slow ops, oldest is osd_op(client.30120.0:6 107.0s0 107.201c25d0 (undecoded) ondisk+write+known_if_redirected+supports_pool_eio e184) 2024-06-03T08:42:59.139 INFO:tasks.ceph.osd.4.smithi192.stderr:2024-06-03T08:42:59.137+0000 12a6c640 -1 osd.4 206 get_health_metrics reporting 2 slow ops, oldest is osd_op(client.30120.0:6 107.0s0 107.201c25d0 (undecoded) ondisk+write+known_if_redirected+supports_pool_eio e184) 2024-06-03T08:43:00.135 INFO:tasks.ceph.osd.4.smithi192.stderr:2024-06-03T08:43:00.133+0000 12a6c640 -1 osd.4 206 get_health_metrics reporting 2 slow ops, oldest is osd_op(client.30120.0:6 107.0s0 107.201c25d0 (undecoded) ondisk+write+known_if_redirected+supports_pool_eio e184) 2024-06-03T08:43:01.104 INFO:tasks.ceph.osd.4.smithi192.stderr:2024-06-03T08:43:01.102+0000 12a6c640 -1 osd.4 206 get_health_metrics reporting 2 slow ops, oldest is osd_op(client.30120.0:6 107.0s0 107.201c25d0 (undecoded) ondisk+write+known_if_redirected+supports_pool_eio e184) 2024-06-03T08:43:01.881 INFO:tasks.ceph.osd.1.smithi088.stderr:2024-06-03T08:43:01.874+0000 ea64640 -1 osd.1 206 heartbeat_check: no reply from 172.21.15.192:6815 osd.6 since back 2024-06-03T08:41:41.838399+0000 front 2024-06-03T08:41:40.773442+0000 (oldest deadline 2024-06-03T08:43:01.678373+0000) 2024-06-03T08:43:02.063 INFO:tasks.ceph.osd.4.smithi192.stderr:2024-06-03T08:43:02.062+0000 12a6c640 -1 osd.4 206 get_health_metrics reporting 2 slow ops, oldest is osd_op(client.30120.0:6 107.0s0 107.201c25d0 (undecoded) ondisk+write+known_if_redirected+supports_pool_eio e184) 2024-06-03T08:43:02.205 INFO:tasks.ceph.osd.2.smithi088.stderr:2024-06-03T08:43:02.200+0000 12a6c640 -1 osd.2 206 heartbeat_check: no reply from 172.21.15.192:6815 osd.6 since back 2024-06-03T08:41:38.263775+0000 front 2024-06-03T08:41:37.326382+0000 (oldest deadline 2024-06-03T08:43:01.964471+0000) 2024-06-03T08:43:02.269 INFO:tasks.ceph.osd.5.smithi192.stderr:2024-06-03T08:43:02.265+0000 12a6c640 -1 osd.5 206 heartbeat_check: no reply from 172.21.15.192:6815 osd.6 since back 2024-06-03T08:41:39.827220+0000 front 2024-06-03T08:41:39.838334+0000 (oldest deadline 2024-06-03T08:43:02.155287+0000) 2024-06-03T08:43:02.307 INFO:tasks.ceph.osd.0.smithi088.stderr:2024-06-03T08:43:02.302+0000 ea64640 -1 osd.0 206 heartbeat_check: no reply from 172.21.15.192:6815 osd.6 since back 2024-06-03T08:41:36.496442+0000 front 2024-06-03T08:41:36.490556+0000 (oldest deadline 2024-06-03T08:43:01.891107+0000) 2024-06-03T08:43:02.385 INFO:tasks.ceph.mon.a.smithi088.stderr:==00:00:07:57.711 34743== Warning: unimplemented fcntl command: 1036 2024-06-03T08:43:02.436 INFO:tasks.ceph.mon.a.smithi088.stderr:==00:00:07:57.762 34743== Warning: unimplemented fcntl command: 1036 2024-06-03T08:43:02.459 INFO:tasks.ceph.mon.a.smithi088.stderr:==00:00:07:57.785 34743== Warning: unimplemented fcntl command: 1036 2024-06-03T08:43:02.485 INFO:tasks.ceph.mon.b.smithi192.stderr:==00:00:07:57.787 34228== Warning: unimplemented fcntl command: 1036 2024-06-03T08:43:02.485 INFO:tasks.ceph.mon.c.smithi088.stderr:==00:00:07:57.811 34749== Warning: unimplemented fcntl command: 1036 2024-06-03T08:43:02.486 INFO:tasks.ceph.mon.b.smithi192.stderr:==00:00:07:57.789 34228== Warning: unimplemented fcntl command: 1036 2024-06-03T08:43:02.486 INFO:tasks.ceph.mon.c.smithi088.stderr:==00:00:07:57.812 34749== Warning: unimplemented fcntl command: 1036 2024-06-03T08:43:02.509 INFO:tasks.ceph.mon.c.smithi088.stderr:==00:00:07:57.834 34749== Warning: unimplemented fcntl command: 1036 2024-06-03T08:43:02.509 INFO:tasks.ceph.mon.b.smithi192.stderr:==00:00:07:57.811 34228== Warning: unimplemented fcntl command: 1036 2024-06-03T08:43:02.873 INFO:tasks.ceph.osd.1.smithi088.stderr:2024-06-03T08:43:02.872+0000 ea64640 -1 osd.1 206 heartbeat_check: no reply from 172.21.15.192:6815 osd.6 since back 2024-06-03T08:41:41.838399+0000 front 2024-06-03T08:41:40.773442+0000 (oldest deadline 2024-06-03T08:43:01.678373+0000) 2024-06-03T08:43:03.094 INFO:tasks.ceph.osd.4.smithi192.stderr:2024-06-03T08:43:03.094+0000 12a6c640 -1 osd.4 206 get_health_metrics reporting 2 slow ops, oldest is osd_op(client.30120.0:6 107.0s0 107.201c25d0 (undecoded) ondisk+write+known_if_redirected+supports_pool_eio e184) 2024-06-03T08:43:03.204 INFO:tasks.ceph.osd.2.smithi088.stderr:2024-06-03T08:43:03.202+0000 12a6c640 -1 osd.2 206 heartbeat_check: no reply from 172.21.15.192:6815 osd.6 since back 2024-06-03T08:41:38.263775+0000 front 2024-06-03T08:41:37.326382+0000 (oldest deadline 2024-06-03T08:43:01.964471+0000) 2024-06-03T08:43:03.316 INFO:tasks.ceph.osd.5.smithi192.stderr:2024-06-03T08:43:03.315+0000 12a6c640 -1 osd.5 206 heartbeat_check: no reply from 172.21.15.192:6815 osd.6 since back 2024-06-03T08:41:39.827220+0000 front 2024-06-03T08:41:39.838334+0000 (oldest deadline 2024-06-03T08:43:02.155287+0000) 2024-06-03T08:43:03.332 INFO:tasks.ceph.osd.0.smithi088.stderr:2024-06-03T08:43:03.331+0000 ea64640 -1 osd.0 206 heartbeat_check: no reply from 172.21.15.192:6815 osd.6 since back 2024-06-03T08:41:36.496442+0000 front 2024-06-03T08:41:36.490556+0000 (oldest deadline 2024-06-03T08:43:01.891107+0000) 2024-06-03T08:43:03.502 INFO:tasks.ceph.osd.3.smithi088.stderr:2024-06-03T08:43:03.495+0000 ea64640 -1 osd.3 206 heartbeat_check: no reply from 172.21.15.192:6815 osd.6 since back 2024-06-03T08:41:40.370510+0000 front 2024-06-03T08:41:40.372464+0000 (oldest deadline 2024-06-03T08:43:03.306973+0000) 2024-06-03T08:43:03.718 INFO:tasks.ceph.osd.7.smithi192.stderr:2024-06-03T08:43:03.715+0000 12a6c640 -1 osd.7 206 heartbeat_check: no reply from 172.21.15.192:6815 osd.6 since back 2024-06-03T08:41:39.917746+0000 front 2024-06-03T08:41:39.915022+0000 (oldest deadline 2024-06-03T08:43:03.407561+0000) 2024-06-03T08:43:03.854 INFO:tasks.ceph.osd.1.smithi088.stderr:2024-06-03T08:43:03.853+0000 ea64640 -1 osd.1 206 heartbeat_check: no reply from 172.21.15.192:6815 osd.6 since back 2024-06-03T08:41:41.838399+0000 front 2024-06-03T08:41:40.773442+0000 (oldest deadline 2024-06-03T08:43:01.678373+0000) 2024-06-03T08:43:04.123 INFO:tasks.ceph.osd.4.smithi192.stderr:2024-06-03T08:43:04.122+0000 12a6c640 -1 osd.4 206 get_health_metrics reporting 2 slow ops, oldest is osd_op(client.30120.0:6 107.0s0 107.201c25d0 (undecoded) ondisk+write+known_if_redirected+supports_pool_eio e184) 2024-06-03T08:43:04.237 INFO:tasks.ceph.osd.2.smithi088.stderr:2024-06-03T08:43:04.236+0000 12a6c640 -1 osd.2 206 heartbeat_check: no reply from 172.21.15.192:6815 osd.6 since back 2024-06-03T08:41:38.263775+0000 front 2024-06-03T08:41:37.326382+0000 (oldest deadline 2024-06-03T08:43:01.964471+0000) 2024-06-03T08:43:04.296 INFO:tasks.ceph.osd.0.smithi088.stderr:2024-06-03T08:43:04.292+0000 ea64640 -1 osd.0 206 heartbeat_check: no reply from 172.21.15.192:6815 osd.6 since back 2024-06-03T08:41:36.496442+0000 front 2024-06-03T08:41:36.490556+0000 (oldest deadline 2024-06-03T08:43:01.891107+0000) 2024-06-03T08:43:04.339 INFO:tasks.ceph.osd.5.smithi192.stderr:2024-06-03T08:43:04.338+0000 12a6c640 -1 osd.5 206 heartbeat_check: no reply from 172.21.15.192:6815 osd.6 since back 2024-06-03T08:41:39.827220+0000 front 2024-06-03T08:41:39.838334+0000 (oldest deadline 2024-06-03T08:43:02.155287+0000) 2024-06-03T08:43:04.555 INFO:tasks.ceph.osd.3.smithi088.stderr:2024-06-03T08:43:04.552+0000 ea64640 -1 osd.3 206 heartbeat_check: no reply from 172.21.15.192:6815 osd.6 since back 2024-06-03T08:41:40.370510+0000 front 2024-06-03T08:41:40.372464+0000 (oldest deadline 2024-06-03T08:43:03.306973+0000) 2024-06-03T08:43:04.681 INFO:tasks.ceph.osd.7.smithi192.stderr:2024-06-03T08:43:04.680+0000 12a6c640 -1 osd.7 206 heartbeat_check: no reply from 172.21.15.192:6815 osd.6 since back 2024-06-03T08:41:39.917746+0000 front 2024-06-03T08:41:39.915022+0000 (oldest deadline 2024-06-03T08:43:03.407561+0000) 2024-06-03T08:43:04.827 INFO:tasks.ceph.osd.1.smithi088.stderr:2024-06-03T08:43:04.827+0000 ea64640 -1 osd.1 206 heartbeat_check: no reply from 172.21.15.192:6815 osd.6 since back 2024-06-03T08:41:41.838399+0000 front 2024-06-03T08:41:40.773442+0000 (oldest deadline 2024-06-03T08:43:01.678373+0000) 2024-06-03T08:43:05.408 INFO:tasks.ceph.osd.4.smithi192.stderr:2024-06-03T08:43:05.363+0000 12a6c640 -1 osd.4 207 get_health_metrics reporting 2 slow ops, oldest is osd_op(client.30120.0:6 107.0s0 107.201c25d0 (undecoded) ondisk+write+known_if_redirected+supports_pool_eio e184) 2024-06-03T08:43:05.600 INFO:tasks.ceph.osd.3.smithi088.stderr:2024-06-03T08:43:05.593+0000 ea64640 -1 osd.3 207 get_health_metrics reporting 1 slow ops, oldest is osd_op(client.4181.0:13 2.2 2:4e99cc3e:::rbd_mirror_snapshot_schedule:head [omap-get-vals in=16b] snapc 0=[] ondisk+read+known_if_redirected+supports_pool_eio e155) 2024-06-03T08:43:06.475 INFO:tasks.ceph.osd.4.smithi192.stderr:2024-06-03T08:43:06.469+0000 12a6c640 -1 osd.4 208 get_health_metrics reporting 2 slow ops, oldest is osd_op(client.30120.0:6 107.0s0 107.201c25d0 (undecoded) ondisk+write+known_if_redirected+supports_pool_eio e184) 2024-06-03T08:43:06.581 INFO:tasks.ceph.osd.3.smithi088.stderr:2024-06-03T08:43:06.580+0000 ea64640 -1 osd.3 208 get_health_metrics reporting 1 slow ops, oldest is osd_op(client.4181.0:13 2.2 2:4e99cc3e:::rbd_mirror_snapshot_schedule:head [omap-get-vals in=16b] snapc 0=[] ondisk+read+known_if_redirected+supports_pool_eio e155) 2024-06-03T08:43:17.995 INFO:tasks.workunit.client.0.smithi088.stdout: api_misc_pp: [----------] 2 api_aio_pp: Running main() from gmock_main.cc 2024-06-03T08:43:17.995 INFO:tasks.workunit.client.0.smithi088.stdout: api_aio_pp: [==========] Running 57 tests from 4 test suites. 2024-06-03T08:43:17.995 INFO:tasks.workunit.client.0.smithi088.stdout: api_aio_pp: [----------] Global test environment set-up. 2024-06-03T08:43:17.995 INFO:tasks.workunit.client.0.smithi088.stdout: api_aio_pp: [----------] 32 tests from LibRadosAio 2024-06-03T08:43:17.995 INFO:tasks.workunit.client.0.smithi088.stdout: api_aio_pp: [ RUN ] LibRadosAio.TooBigPP 2024-06-03T08:43:17.995 INFO:tasks.workunit.client.0.smithi088.stdout: api_aio_pp: [ OK ] LibRadosAio.TooBigPP (4071 ms) 2024-06-03T08:43:17.996 INFO:tasks.workunit.client.0.smithi088.stdout: api_aio_pp: [ RUN ] LibRadosAio.PoolQuotaPP 2024-06-03T08:43:17.996 INFO:tasks.workunit.client.0.smithi088.stdout: api_aio_pp: [ OK ] LibRadosAio.PoolQuotaPP (33848 ms) 2024-06-03T08:43:17.996 INFO:tasks.workunit.client.0.smithi088.stdout: api_aio_pp: [ RUN ] LibRadosAio.SimpleWritePP 2024-06-03T08:43:17.996 INFO:tasks.workunit.client.0.smithi088.stdout: api_aio_pp: [ OK ] LibRadosAio.SimpleWritePP (18980 ms) 2024-06-03T08:43:17.996 INFO:tasks.workunit.client.0.smithi088.stdout: api_aio_pp: [ RUN ] LibRadosAio.WaitForSafePP 2024-06-03T08:43:17.996 INFO:tasks.workunit.client.0.smithi088.stdout: api_aio_pp: [ OK ] LibRadosAio.WaitForSafePP (18241 ms) 2024-06-03T08:43:17.996 INFO:tasks.workunit.client.0.smithi088.stdout: api_aio_pp: [ RUN ] LibRadosAio.RoundTripPP 2024-06-03T08:43:17.996 INFO:tasks.workunit.client.0.smithi088.stdout: api_aio_pp: [ OK ] LibRadosAio.RoundTripPP (24873 ms) 2024-06-03T08:43:17.996 INFO:tasks.workunit.client.0.smithi088.stdout: api_aio_pp: [ RUN ] LibRadosAio.RoundTripPP2 2024-06-03T08:43:17.996 INFO:tasks.workunit.client.0.smithi088.stdout: api_aio_pp: [ OK ] LibRadosAio.RoundTripPP2 (23148 ms) 2024-06-03T08:43:17.996 INFO:tasks.workunit.client.0.smithi088.stdout: api_aio_pp: [ RUN ] LibRadosAio.RoundTripPP3 2024-06-03T08:43:17.996 INFO:tasks.workunit.client.0.smithi088.stdout: api_aio_pp: [ OK ] LibRadosAio.RoundTripPP3 (18543 ms) 2024-06-03T08:43:17.996 INFO:tasks.workunit.client.0.smithi088.stdout: api_aio_pp: [ RUN ] LibRadosAio.RoundTripSparseReadPP 2024-06-03T08:43:17.996 INFO:tasks.workunit.client.0.smithi088.stdout: api_aio_pp: [ OK ] LibRadosAio.RoundTripSparseReadPP (14850 ms) 2024-06-03T08:43:17.997 INFO:tasks.workunit.client.0.smithi088.stdout: api_aio_pp: [ RUN ] LibRadosAio.IsCompletePP 2024-06-03T08:43:17.997 INFO:tasks.workunit.client.0.smithi088.stdout: api_aio_pp: [ OK ] LibRadosAio.IsCompletePP (19342 ms) 2024-06-03T08:43:17.997 INFO:tasks.workunit.client.0.smithi088.stdout: api_aio_pp: [ RUN ] LibRadosAio.IsSafePP 2024-06-03T08:43:17.997 INFO:tasks.workunit.client.0.smithi088.stdout: api_aio_pp: [ OK ] LibRadosAio.IsSafePP (17275 ms) 2024-06-03T08:43:17.997 INFO:tasks.workunit.client.0.smithi088.stdout: api_aio_pp: [ RUN ] LibRadosAio.ReturnValuePP 2024-06-03T08:43:17.997 INFO:tasks.workunit.client.0.smithi088.stdout: api_aio_pp: [ OK ] LibRadosAio.ReturnValuePP (6632 ms) 2024-06-03T08:43:17.997 INFO:tasks.workunit.client.0.smithi088.stdout: api_aio_pp: [ RUN ] LibRadosAio.FlushPP 2024-06-03T08:43:17.997 INFO:tasks.workunit.client.0.smithi088.stdout: api_aio_pp: [ OK ] LibRadosAio.FlushPP (3337 ms) 2024-06-03T08:43:17.997 INFO:tasks.workunit.client.0.smithi088.stdout: api_aio_pp: [ RUN ] LibRadosAio.FlushAsyncPP 2024-06-03T08:43:17.997 INFO:tasks.workunit.client.0.smithi088.stdout: api_aio_pp: [ OK ] LibRadosAio.FlushAsyncPP (3533 ms) 2024-06-03T08:43:17.997 INFO:tasks.workunit.client.0.smithi088.stdout: api_aio_pp: [ RUN ] LibRadosAio.RoundTripWriteFullPP 2024-06-03T08:43:17.997 INFO:tasks.workunit.client.0.smithi088.stdout: api_aio_pp: [ OK ] LibRadosAio.RoundTripWriteFullPP (3623 ms) 2024-06-03T08:43:17.997 INFO:tasks.workunit.client.0.smithi088.stdout: api_aio_pp: [ RUN ] LibRadosAio.RoundTripWriteFullPP2 2024-06-03T08:43:17.997 INFO:tasks.workunit.client.0.smithi088.stdout: api_aio_pp: [ OK ] LibRadosAio.RoundTripWriteFullPP2 (3221 ms) 2024-06-03T08:43:17.998 INFO:tasks.workunit.client.0.smithi088.stdout: api_aio_pp: [ RUN ] LibRadosAio.RoundTripWriteSamePP 2024-06-03T08:43:17.998 INFO:tasks.workunit.client.0.smithi088.stdout: api_aio_pp: [ OK ] LibRadosAio.RoundTripWriteSamePP (4527 ms) 2024-06-03T08:43:17.998 INFO:tasks.workunit.client.0.smithi088.stdout: api_aio_pp: [ RUN ] LibRadosAio.RoundTripWriteSamePP2 2024-06-03T08:43:17.998 INFO:tasks.workunit.client.0.smithi088.stdout: api_aio_pp: [ OK ] LibRadosAio.RoundTripWriteSamePP2 (3236 ms) 2024-06-03T08:43:17.998 INFO:tasks.workunit.client.0.smithi088.stdout: api_aio_pp: [ RUN ] LibRadosAio.SimpleStatPPNS 2024-06-03T08:43:17.998 INFO:tasks.workunit.client.0.smithi088.stdout: api_aio_pp: [ OK ] LibRadosAio.SimpleStatPPNS (3585 ms) 2024-06-03T08:43:17.998 INFO:tasks.workunit.client.0.smithi088.stdout: api_aio_pp: [ RUN ] LibRadosAio.SimpleStatPP 2024-06-03T08:43:17.998 INFO:tasks.workunit.client.0.smithi088.stdout: api_aio_pp: [ OK ] LibRadosAio.SimpleStatPP (3293 ms) 2024-06-03T08:43:17.998 INFO:tasks.workunit.client.0.smithi088.stdout: api_aio_pp: [ RUN ] LibRadosAio.OperateMtime 2024-06-03T08:43:17.998 INFO:tasks.workunit.client.0.smithi088.stdout: api_aio_pp: [ OK ] LibRadosAio.OperateMtime (4451 ms) 2024-06-03T08:43:17.998 INFO:tasks.workunit.client.0.smithi088.stdout: api_aio_pp: [ RUN ] LibRadosAio.OperateMtime2 2024-06-03T08:43:17.998 INFO:tasks.workunit.client.0.smithi088.stdout: api_aio_pp: [ OK ] LibRadosAio.OperateMtime2 (30791 ms) 2024-06-03T08:43:17.998 INFO:tasks.workunit.client.0.smithi088.stdout: api_aio_pp: [ RUN ] LibRadosAio.StatRemovePP 2024-06-03T08:43:17.999 INFO:tasks.workunit.client.0.smithi088.stdout: api_aio_pp: [ OK ] LibRadosAio.StatRemovePP (2770 ms) 2024-06-03T08:43:17.999 INFO:tasks.workunit.client.0.smithi088.stdout: api_aio_pp: [ RUN ] LibRadosAio.ExecuteClassPP 2024-06-03T08:43:17.999 INFO:tasks.workunit.client.0.smithi088.stdout: api_aio_pp: [ OK ] LibRadosAio.ExecuteClassPP (3549 ms) 2024-06-03T08:43:17.999 INFO:tasks.workunit.client.0.smithi088.stdout: api_aio_pp: [ RUN ] LibRadosAio.OmapPP 2024-06-03T08:43:17.999 INFO:tasks.workunit.client.0.smithi088.stdout: api_aio_pp: [ OK ] LibRadosAio.OmapPP (44427 ms) 2024-06-03T08:43:17.999 INFO:tasks.workunit.client.0.smithi088.stdout: api_aio_pp: [ RUN ] LibRadosAio.MultiWritePP 2024-06-03T08:43:17.999 INFO:tasks.workunit.client.0.smithi088.stdout: api_aio_pp: [ OK ] LibRadosAio.MultiWritePP (3282 ms) 2024-06-03T08:43:17.999 INFO:tasks.workunit.client.0.smithi088.stdout: api_aio_pp: [ RUN ] LibRadosAio.AioUnlockPP 2024-06-03T08:43:17.999 INFO:tasks.workunit.client.0.smithi088.stdout: api_aio_pp: [ OK ] LibRadosAio.AioUnlockPP (2946 ms) 2024-06-03T08:43:17.999 INFO:tasks.workunit.client.0.smithi088.stdout: api_aio_pp: [ RUN ] LibRadosAio.RoundTripAppendPP 2024-06-03T08:43:22.807 INFO:tasks.ceph.mon.a.smithi088.stderr:==00:00:08:18.133 34743== Warning: unimplemented fcntl command: 1036 2024-06-03T08:43:22.867 INFO:tasks.ceph.mon.a.smithi088.stderr:==00:00:08:18.192 34743== Warning: unimplemented fcntl command: 1036 2024-06-03T08:43:22.881 INFO:tasks.ceph.mon.c.smithi088.stderr:==00:00:08:18.207 34749== Warning: unimplemented fcntl command: 1036 2024-06-03T08:43:22.895 INFO:tasks.ceph.mon.c.smithi088.stderr:==00:00:08:18.221 34749== Warning: unimplemented fcntl command: 1036 2024-06-03T08:43:22.960 INFO:tasks.ceph.mon.c.smithi088.stderr:==00:00:08:18.285 34749== Warning: unimplemented fcntl command: 1036 2024-06-03T08:43:22.961 INFO:tasks.ceph.mon.a.smithi088.stderr:==00:00:08:18.287 34743== Warning: unimplemented fcntl command: 1036 2024-06-03T08:43:22.964 INFO:tasks.ceph.mon.b.smithi192.stderr:==00:00:08:18.267 34228== Warning: unimplemented fcntl command: 1036 2024-06-03T08:43:22.975 INFO:tasks.ceph.mon.b.smithi192.stderr:==00:00:08:18.277 34228== Warning: unimplemented fcntl command: 1036 2024-06-03T08:43:23.044 INFO:tasks.ceph.mon.b.smithi192.stderr:==00:00:08:18.346 34228== Warning: unimplemented fcntl command: 1036 2024-06-03T08:43:26.858 INFO:tasks.workunit.client.0.smithi088.stdout: api_aio_pp: [ OK ] L api_aio: Running main() from gmock_main.cc 2024-06-03T08:43:26.858 INFO:tasks.workunit.client.0.smithi088.stdout: api_aio: [==========] Running 42 tests from 2 test suites. 2024-06-03T08:43:26.858 INFO:tasks.workunit.client.0.smithi088.stdout: api_aio: [----------] Global test environment set-up. 2024-06-03T08:43:26.858 INFO:tasks.workunit.client.0.smithi088.stdout: api_aio: [----------] 26 tests from LibRadosAio 2024-06-03T08:43:26.858 INFO:tasks.workunit.client.0.smithi088.stdout: api_aio: [ RUN ] LibRadosAio.TooBig 2024-06-03T08:43:26.858 INFO:tasks.workunit.client.0.smithi088.stdout: api_aio: [ OK ] LibRadosAio.TooBig (4135 ms) 2024-06-03T08:43:26.858 INFO:tasks.workunit.client.0.smithi088.stdout: api_aio: [ RUN ] LibRadosAio.SimpleWrite 2024-06-03T08:43:26.858 INFO:tasks.workunit.client.0.smithi088.stdout: api_aio: [ OK ] LibRadosAio.SimpleWrite (58656 ms) 2024-06-03T08:43:26.858 INFO:tasks.workunit.client.0.smithi088.stdout: api_aio: [ RUN ] LibRadosAio.WaitForSafe 2024-06-03T08:43:26.858 INFO:tasks.workunit.client.0.smithi088.stdout: api_aio: [ OK ] LibRadosAio.WaitForSafe (23739 ms) 2024-06-03T08:43:26.859 INFO:tasks.workunit.client.0.smithi088.stdout: api_aio: [ RUN ] LibRadosAio.RoundTrip 2024-06-03T08:43:26.859 INFO:tasks.workunit.client.0.smithi088.stdout: api_aio: [ OK ] LibRadosAio.RoundTrip (21798 ms) 2024-06-03T08:43:26.859 INFO:tasks.workunit.client.0.smithi088.stdout: api_aio: [ RUN ] LibRadosAio.RoundTrip2 2024-06-03T08:43:26.859 INFO:tasks.workunit.client.0.smithi088.stdout: api_aio: [ OK ] LibRadosAio.RoundTrip2 (21445 ms) 2024-06-03T08:43:26.859 INFO:tasks.workunit.client.0.smithi088.stdout: api_aio: [ RUN ] LibRadosAio.RoundTrip3 2024-06-03T08:43:26.859 INFO:tasks.workunit.client.0.smithi088.stdout: api_aio: [ OK ] LibRadosAio.RoundTrip3 (16815 ms) 2024-06-03T08:43:26.859 INFO:tasks.workunit.client.0.smithi088.stdout: api_aio: [ RUN ] LibRadosAio.RoundTripAppend 2024-06-03T08:43:26.859 INFO:tasks.workunit.client.0.smithi088.stdout: api_aio: [ OK ] LibRadosAio.RoundTripAppend (16387 ms) 2024-06-03T08:43:26.859 INFO:tasks.workunit.client.0.smithi088.stdout: api_aio: [ RUN ] LibRadosAio.RemoveTest 2024-06-03T08:43:26.859 INFO:tasks.workunit.client.0.smithi088.stdout: api_aio: [ OK ] LibRadosAio.RemoveTest (52891 ms) 2024-06-03T08:43:26.859 INFO:tasks.workunit.client.0.smithi088.stdout: api_aio: [ RUN ] LibRadosAio.XattrsRoundTrip 2024-06-03T08:43:26.859 INFO:tasks.workunit.client.0.smithi088.stdout: api_aio: [ OK ] LibRadosAio.XattrsRoundTrip (4272 ms) 2024-06-03T08:43:26.859 INFO:tasks.workunit.client.0.smithi088.stdout: api_aio: [ RUN ] LibRadosAio.RmXattr 2024-06-03T08:43:26.859 INFO:tasks.workunit.client.0.smithi088.stdout: api_aio: [ OK ] LibRadosAio.RmXattr (3108 ms) 2024-06-03T08:43:26.860 INFO:tasks.workunit.client.0.smithi088.stdout: api_aio: [ RUN ] LibRadosAio.XattrIter 2024-06-03T08:43:26.860 INFO:tasks.workunit.client.0.smithi088.stdout: api_aio: [ OK ] LibRadosAio.XattrIter (3802 ms) 2024-06-03T08:43:26.860 INFO:tasks.workunit.client.0.smithi088.stdout: api_aio: [ RUN ] LibRadosAio.IsComplete 2024-06-03T08:43:26.860 INFO:tasks.workunit.client.0.smithi088.stdout: api_aio: [ OK ] LibRadosAio.IsComplete (3442 ms) 2024-06-03T08:43:26.860 INFO:tasks.workunit.client.0.smithi088.stdout: api_aio: [ RUN ] LibRadosAio.IsSafe 2024-06-03T08:43:26.860 INFO:tasks.workunit.client.0.smithi088.stdout: api_aio: [ OK ] LibRadosAio.IsSafe (3121 ms) 2024-06-03T08:43:26.860 INFO:tasks.workunit.client.0.smithi088.stdout: api_aio: [ RUN ] LibRadosAio.ReturnValue 2024-06-03T08:43:26.860 INFO:tasks.workunit.client.0.smithi088.stdout: api_aio: [ OK ] LibRadosAio.ReturnValue (29811 ms) 2024-06-03T08:43:26.860 INFO:tasks.workunit.client.0.smithi088.stdout: api_aio: [ RUN ] LibRadosAio.Flush 2024-06-03T08:43:26.860 INFO:tasks.workunit.client.0.smithi088.stdout: api_aio: [ OK ] LibRadosAio.Flush (2689 ms) 2024-06-03T08:43:26.860 INFO:tasks.workunit.client.0.smithi088.stdout: api_aio: [ RUN ] LibRadosAio.FlushAsync 2024-06-03T08:43:26.860 INFO:tasks.workunit.client.0.smithi088.stdout: api_aio: [ OK ] LibRadosAio.FlushAsync (3641 ms) 2024-06-03T08:43:26.860 INFO:tasks.workunit.client.0.smithi088.stdout: api_aio: [ RUN ] LibRadosAio.RoundTripWriteFull 2024-06-03T08:43:26.860 INFO:tasks.workunit.client.0.smithi088.stdout: api_aio: [ OK ] LibRadosAio.RoundTripWriteFull (3426 ms) 2024-06-03T08:43:26.860 INFO:tasks.workunit.client.0.smithi088.stdout: api_aio: [ RUN ] LibRadosAio.RoundTripWriteSame 2024-06-03T08:43:26.861 INFO:tasks.workunit.client.0.smithi088.stdout: api_aio: [ OK ] LibRadosAio.RoundTripWriteSame (3405 ms) 2024-06-03T08:43:26.861 INFO:tasks.workunit.client.0.smithi088.stdout: api_aio: [ RUN ] LibRadosAio.SimpleStat 2024-06-03T08:43:26.861 INFO:tasks.workunit.client.0.smithi088.stdout: api_aio: [ OK ] LibRadosAio.SimpleStat (3584 ms) 2024-06-03T08:43:26.861 INFO:tasks.workunit.client.0.smithi088.stdout: api_aio: [ RUN ] LibRadosAio.OperateMtime 2024-06-03T08:43:26.861 INFO:tasks.workunit.client.0.smithi088.stdout: api_aio: [ OK ] LibRadosAio.OperateMtime (3254 ms) 2024-06-03T08:43:26.861 INFO:tasks.workunit.client.0.smithi088.stdout: api_aio: [ RUN ] LibRadosAio.Operate2Mtime 2024-06-03T08:43:26.861 INFO:tasks.workunit.client.0.smithi088.stdout: api_aio: [ OK ] LibRadosAio.Operate2Mtime (30740 ms) 2024-06-03T08:43:26.861 INFO:tasks.workunit.client.0.smithi088.stdout: api_aio: [ RUN ] LibRadosAio.SimpleStatNS 2024-06-03T08:43:26.861 INFO:tasks.workunit.client.0.smithi088.stdout: api_aio: [ OK ] LibRadosAio.SimpleStatNS (4364 ms) 2024-06-03T08:43:26.861 INFO:tasks.workunit.client.0.smithi088.stdout: api_aio: [ RUN ] LibRadosAio.StatRemove 2024-06-03T08:43:26.861 INFO:tasks.workunit.client.0.smithi088.stdout: api_aio: [ OK ] LibRadosAio.StatRemove (3941 ms) 2024-06-03T08:43:26.861 INFO:tasks.workunit.client.0.smithi088.stdout: api_aio: [ RUN ] LibRadosAio.ExecuteClass 2024-06-03T08:43:26.861 INFO:tasks.workunit.client.0.smithi088.stdout: api_aio: [ OK ] LibRadosAio.ExecuteClass (4247 ms) 2024-06-03T08:43:26.861 INFO:tasks.workunit.client.0.smithi088.stdout: api_aio: [ RUN ] LibRadosAio.MultiWrite 2024-06-03T08:43:26.862 INFO:tasks.workunit.client.0.smithi088.stdout: api_aio: [ OK ] LibRadosAio.MultiWrite (3694 ms) 2024-06-03T08:43:26.862 INFO:tasks.workunit.client.0.smithi088.stdout: api_aio: [ RUN ] LibRadosAio.AioUnlock 2024-06-03T08:43:26.862 INFO:tasks.workunit.client.0.smithi088.stdout: api_aio: [ OK ] LibRadosAio.AioUnlock (3098 ms) 2024-06-03T08:43:26.862 INFO:tasks.workunit.client.0.smithi088.stdout: api_aio: [----------] 26 tests from LibRadosAio (333507 ms total) 2024-06-03T08:43:26.862 INFO:tasks.workunit.client.0.smithi088.stdout: api_aio: 2024-06-03T08:43:26.862 INFO:tasks.workunit.client.0.smithi088.stdout: api_aio: [----------] 16 tests from LibRadosAioEC 2024-06-03T08:43:31.060 INFO:tasks.workunit.client.0.smithi088.stdout: api_aio: [ RUN ] LibRadosAioEC.Sim2 tests from LibRadosMiscPP (250704 ms total) 2024-06-03T08:43:31.060 INFO:tasks.workunit.client.0.smithi088.stdout: api_misc_pp: 2024-06-03T08:43:31.061 INFO:tasks.workunit.client.0.smithi088.stdout: api_misc_pp: [----------] 1 test from LibRadosTwoPoolsECPP 2024-06-03T08:43:31.061 INFO:tasks.workunit.client.0.smithi088.stdout: api_misc_pp: [ RUN ] LibRadosTwoPoolsECPP.CopyFrom 2024-06-03T08:43:31.061 INFO:tasks.workunit.client.0.smithi088.stdout: api_misc_pp: [ OK ] LibRadosTwoPoolsECPP.CopyFrom (51531 ms) 2024-06-03T08:43:31.061 INFO:tasks.workunit.client.0.smithi088.stdout: api_misc_pp: [----------] 1 test from LibRadosTwoPoolsECPP (51531 ms total) 2024-06-03T08:43:31.061 INFO:tasks.workunit.client.0.smithi088.stdout: api_misc_pp: 2024-06-03T08:43:31.061 INFO:tasks.workunit.client.0.smithi088.stdout: api_misc_pp: [----------] 2 tests from LibRadosChecksum/0, where TypeParam = LibRadosChecksumParams<(rados_checksum_type_t)0, Checksummer::xxhash32, ceph_le > 2024-06-03T08:43:31.061 INFO:tasks.workunit.client.0.smithi088.stdout: api_misc_pp: [ RUN ] LibRadosChecksum/0.Subset 2024-06-03T08:43:31.061 INFO:tasks.workunit.client.0.smithi088.stdout: api_misc_pp: [ OK ] LibRadosChecksum/0.Subset (727 ms) 2024-06-03T08:43:31.061 INFO:tasks.workunit.client.0.smithi088.stdout: api_misc_pp: [ RUN ] LibRadosChecksum/0.Chunked 2024-06-03T08:43:31.062 INFO:tasks.workunit.client.0.smithi088.stdout: api_misc_pp: [ OK ] LibRadosChecksum/0.Chunked (213 ms) 2024-06-03T08:43:31.062 INFO:tasks.workunit.client.0.smithi088.stdout: api_misc_pp: [----------] 2 tests from LibRadosChecksum/0 (940 ms total) 2024-06-03T08:43:31.062 INFO:tasks.workunit.client.0.smithi088.stdout: api_misc_pp: 2024-06-03T08:43:31.062 INFO:tasks.workunit.client.0.smithi088.stdout: api_misc_pp: [----------] 2 tests from LibRadosChecksum/1, where TypeParam = LibRadosChecksumParams<(rados_checksum_type_t)1, Checksummer::xxhash64, ceph_le > 2024-06-03T08:43:31.062 INFO:tasks.workunit.client.0.smithi088.stdout: api_misc_pp: [ RUN ] LibRadosChecksum/1.Subset 2024-06-03T08:43:31.062 INFO:tasks.workunit.client.0.smithi088.stdout: api_misc_pp: [ OK ] LibRadosChecksum/1.Subset (1150 ms) 2024-06-03T08:43:31.062 INFO:tasks.workunit.client.0.smithi088.stdout: api_misc_pp: [ RUN ] LibRadosChecksum/1.Chunked 2024-06-03T08:43:31.062 INFO:tasks.workunit.client.0.smithi088.stdout: api_misc_pp: [ OK ] LibRadosChecksum/1.Chunked (595 ms) 2024-06-03T08:43:31.062 INFO:tasks.workunit.client.0.smithi088.stdout: api_misc_pp: [----------] 2 tests from LibRadosChecksum/1 (1745 ms total) 2024-06-03T08:43:31.062 INFO:tasks.workunit.client.0.smithi088.stdout: api_misc_pp: 2024-06-03T08:43:31.062 INFO:tasks.workunit.client.0.smithi088.stdout: api_misc_pp: [----------] 2 tests from LibRadosChecksum/2, where TypeParam = LibRadosChecksumParams<(rados_checksum_type_t)2, Checksummer::crc32c, ceph_le > 2024-06-03T08:43:31.062 INFO:tasks.workunit.client.0.smithi088.stdout: api_misc_pp: [ RUN ] LibRadosChecksum/2.Subset 2024-06-03T08:43:31.063 INFO:tasks.workunit.client.0.smithi088.stdout: api_misc_pp: [ OK ] LibRadosChecksum/2.Subset (584 ms) 2024-06-03T08:43:31.063 INFO:tasks.workunit.client.0.smithi088.stdout: api_misc_pp: [ RUN ] LibRadosChecksum/2.Chunked 2024-06-03T08:43:31.063 INFO:tasks.workunit.client.0.smithi088.stdout: api_misc_pp: [ OK ] LibRadosChecksum/2.Chunked (752 ms) 2024-06-03T08:43:31.063 INFO:tasks.workunit.client.0.smithi088.stdout: api_misc_pp: [----------] 2 tests from LibRadosChecksum/2 (1336 ms total) 2024-06-03T08:43:31.063 INFO:tasks.workunit.client.0.smithi088.stdout: api_misc_pp: 2024-06-03T08:43:31.063 INFO:tasks.workunit.client.0.smithi088.stdout: api_misc_pp: [----------] 1 test from LibRadosMiscECPP 2024-06-03T08:43:31.063 INFO:tasks.workunit.client.0.smithi088.stdout: api_misc_pp: [ RUN ] LibRadosMiscECPP.CompareExtentRange 2024-06-03T08:43:31.063 INFO:tasks.workunit.client.0.smithi088.stdout: api_misc_pp: [ OK ] LibRadosMiscECPP.CompareExtentRange (1737 ms) 2024-06-03T08:43:31.063 INFO:tasks.workunit.client.0.smithi088.stdout: api_misc_pp: [----------] 1 test from LibRadosMiscECPP (1737 ms total) 2024-06-03T08:43:31.063 INFO:tasks.workunit.client.0.smithi088.stdout: api_misc_pp: 2024-06-03T08:43:31.063 INFO:tasks.workunit.client.0.smithi088.stdout: api_misc_pp: [----------] Global test environment tear-down 2024-06-03T08:43:31.063 INFO:tasks.workunit.client.0.smithi088.stdout: api_misc_pp: [==========] 31 tests from 7 test suites ran. (337626 ms total) 2024-06-03T08:43:31.064 INFO:tasks.workunit.client.0.smithi088.stdout: api_misc_pp: [ PASSED ] 31 tests. 2024-06-03T08:43:35.920 INFO:tasks.workunit.client.0.smithi088.stdout:osSnapshotsECPP.SnapRemovePP (2329 ms) 2024-06-03T08:43:35.920 INFO:tasks.workunit.client.0.smithi088.stdout: api_snapshots_pp: [ RUN ] LibRadosSnapshotsECPP.RollbackPP 2024-06-03T08:43:35.920 INFO:tasks.workunit.client.0.smithi088.stdout: api_snapshots_pp: [ OK ] LibRadosSnapshotsECPP.RollbackPP (86979 ms) 2024-06-03T08:43:35.920 INFO:tasks.workunit.client.0.smithi088.stdout: api_snapshots_pp: [ RUN ] LibRadosSnapshotsECPP.SnapGetNamePP 2024-06-03T08:43:35.920 INFO:tasks.workunit.client.0.smithi088.stdout: api_snapshots_pp: [ OK ] LibRadosSnapshotsECPP.SnapGetNamePP (2157 ms) 2024-06-03T08:43:35.920 INFO:tasks.workunit.client.0.smithi088.stdout: api_snapshots_pp: [----------] 4 tests from LibRadosSnapshotsECPP (96220 ms total) 2024-06-03T08:43:35.920 INFO:tasks.workunit.client.0.smithi088.stdout: api_snapshots_pp: 2024-06-03T08:43:35.920 INFO:tasks.workunit.client.0.smithi088.stdout: api_snapshots_pp: [----------] 3 tests from LibRadosSnapshotsSelfManagedECPP 2024-06-03T08:43:35.920 INFO:tasks.workunit.client.0.smithi088.stdout: api_snapshots_pp: [ RUN ] LibRadosSnapshotsSelfManagedECPP.SnapPP 2024-06-03T08:43:35.920 INFO:tasks.workunit.client.0.smithi088.stdout: api_snapshots_pp: [ OK ] LibRadosSnapshotsSelfManagedECPP.SnapPP (5521 ms) 2024-06-03T08:43:35.920 INFO:tasks.workunit.client.0.smithi088.stdout: api_snapshots_pp: [ RUN ] LibRadosSnapshotsSelfManagedECPP.RollbackPP 2024-06-03T08:43:35.921 INFO:tasks.workunit.client.0.smithi088.stdout: api_snapshots_pp: [ OK ] LibRadosSnapshotsSelfManagedECPP.RollbackPP (5452 ms) 2024-06-03T08:43:35.921 INFO:tasks.workunit.client.0.smithi088.stdout: api_snapshots_pp: [ RUN ] LibRadosSnapshotsSelfManagedECPP.Bug11677 2024-06-03T08:43:35.921 INFO:tasks.workunit.client.0.smithi088.stdout: api_snapshots_pp: [ OK ] LibRadosSnapshotsSelfManagedECPP.Bug11677 (3664 ms) 2024-06-03T08:43:35.921 INFO:tasks.workunit.client.0.smithi088.stdout: api_snapshots_pp: [----------] 3 tests from LibRadosSnapshotsSelfManagedECPP (14637 ms total) 2024-06-03T08:43:35.921 INFO:tasks.workunit.client.0.smithi088.stdout: api_snapshots_pp: 2024-06-03T08:43:35.921 INFO:tasks.workunit.client.0.smithi088.stdout: api_snapshots_pp: [----------] Global test environment tear-down 2024-06-03T08:43:35.921 INFO:tasks.workunit.client.0.smithi088.stdout: api_snapshots_pp: [==========] 21 tests from 5 test suites ran. (342431 ms total) 2024-06-03T08:43:35.921 INFO:tasks.workunit.client.0.smithi088.stdout: api_snapshots_pp: [ PASSED ] 20 tests. 2024-06-03T08:43:35.921 INFO:tasks.workunit.client.0.smithi088.stdout: api_snapshots_pp: [ SKIPPED ] 1 test, listed below: 2024-06-03T08:43:35.921 INFO:tasks.workunit.client.0.smithi088.stdout: api_snapshots_pp: [ SKIPPED ] LibRadosSnapshotsSelfManagedPP.WriteRollback 2024-06-03T08:43:43.162 INFO:tasks.ceph.mon.a.smithi088.stderr:==00:00:08:38.487 34743== Warning: unimplemented fcntl command: 1036 2024-06-03T08:43:43.166 INFO:tasks.ceph.mon.a.smithi088.stderr:==00:00:08:38.491 34743== Warning: unimplemented fcntl command: 1036 2024-06-03T08:43:43.220 INFO:tasks.ceph.mon.a.smithi088.stderr:==00:00:08:38.545 34743== Warning: unimplemented fcntl command: 1036 2024-06-03T08:43:43.222 INFO:tasks.ceph.mon.b.smithi192.stderr:==00:00:08:38.524 34228== Warning: unimplemented fcntl command: 1036 2024-06-03T08:43:43.225 INFO:tasks.ceph.mon.c.smithi088.stderr:==00:00:08:38.551 34749== Warning: unimplemented fcntl command: 1036 2024-06-03T08:43:43.226 INFO:tasks.ceph.mon.b.smithi192.stderr:==00:00:08:38.528 34228== Warning: unimplemented fcntl command: 1036 2024-06-03T08:43:43.230 INFO:tasks.ceph.mon.c.smithi088.stderr:==00:00:08:38.556 34749== Warning: unimplemented fcntl command: 1036 2024-06-03T08:43:43.281 INFO:tasks.ceph.mon.b.smithi192.stderr:==00:00:08:38.583 34228== Warning: unimplemented fcntl command: 1036 2024-06-03T08:43:43.292 INFO:tasks.ceph.mon.c.smithi088.stderr:==00:00:08:38.617 34749== Warning: unimplemented fcntl command: 1036 2024-06-03T08:43:43.531 INFO:tasks.ceph.osd.6.smithi192.stderr:2024-06-03T08:43:43.516+0000 12a6c640 -1 osd.6 162 get_health_metrics reporting 1 slow ops, oldest is replica_scrub(pg: 11.6,from:0'0,to:120'9,epoch:161/28,start:11:60000000::::0,end:MAX,chunky:1,deep:1,version:9,allow_preemption:1,priority=120) 2024-06-03T08:43:49.455 INFO:tasks.ceph.mon.a.smithi088.stderr:==00:00:08:44.781 34743== Warning: unimplemented fcntl command: 1036 2024-06-03T08:43:49.461 INFO:tasks.ceph.mon.a.smithi088.stderr:==00:00:08:44.786 34743== Warning: unimplemented fcntl command: 1036 2024-06-03T08:43:49.497 INFO:tasks.ceph.mon.a.smithi088.stderr:==00:00:08:44.823 34743== Warning: unimplemented fcntl command: 1036 2024-06-03T08:43:49.537 INFO:tasks.ceph.mon.b.smithi192.stderr:==00:00:08:44.839 34228== Warning: unimplemented fcntl command: 1036 2024-06-03T08:43:49.547 INFO:tasks.ceph.mon.b.smithi192.stderr:==00:00:08:44.849 34228== Warning: unimplemented fcntl command: 1036 2024-06-03T08:43:49.555 INFO:tasks.ceph.mon.c.smithi088.stderr:==00:00:08:44.881 34749== Warning: unimplemented fcntl command: 1036 2024-06-03T08:43:49.557 INFO:tasks.ceph.mon.c.smithi088.stderr:==00:00:08:44.883 34749== Warning: unimplemented fcntl command: 1036 2024-06-03T08:43:49.586 INFO:tasks.ceph.mon.b.smithi192.stderr:==00:00:08:44.888 34228== Warning: unimplemented fcntl command: 1036 2024-06-03T08:43:49.594 INFO:tasks.ceph.mon.c.smithi088.stderr:==00:00:08:44.919 34749== Warning: unimplemented fcntl command: 1036 2024-06-03T08:43:51.689 INFO:tasks.ceph.osd.6.smithi192.stderr:2024-06-03T08:43:51.687+0000 1426f640 -1 osd.6 249 set_numa_affinity unable to identify public interface '' numa node: (2) No such file or directory 2024-06-03T08:43:56.165 INFO:tasks.workunit.client.0.smithi088.stdout: api_tier_pp: [==========] Running 77 tests from 4 test suites. 2024-06-03T08:43:56.165 INFO:tasks.workunit.client.0.smithi088.stdout: api_tier_pp: [----------] Global test environment set-up. 2024-06-03T08:43:56.165 INFO:tasks.workunit.client.0.smithi088.stdout: api_tier_pp: [----------] 3 tests from LibRadosTierPP 2024-06-03T08:43:56.165 INFO:tasks.workunit.client.0.smithi088.stdout: api_tier_pp: seed 40561 2024-06-03T08:43:56.165 INFO:tasks.workunit.client.0.smithi088.stdout: api_tier_pp: [ RUN ] LibRadosTierPP.Dirty 2024-06-03T08:43:56.165 INFO:tasks.workunit.client.0.smithi088.stdout: api_tier_pp: [ OK ] LibRadosTierPP.Dirty (52115 ms) 2024-06-03T08:43:56.165 INFO:tasks.workunit.client.0.smithi088.stdout: api_tier_pp: [ RUN ] LibRadosTierPP.FlushWriteRaces 2024-06-03T08:43:56.165 INFO:tasks.workunit.client.0.smithi088.stdout: api_tier_pp: [ OK ] LibRadosTierPP.FlushWriteRaces (60879 ms) 2024-06-03T08:43:56.165 INFO:tasks.workunit.client.0.smithi088.stdout: api_tier_pp: [ RUN ] LibRadosTierPP.HitSetNone 2024-06-03T08:43:56.165 INFO:tasks.workunit.client.0.smithi088.stdout: api_tier_pp: [ OK ] LibRadosTierPP.HitSetNone (375 ms) 2024-06-03T08:43:56.166 INFO:tasks.workunit.client.0.smithi088.stdout: api_tier_pp: [----------] 3 tests from LibRadosTierPP (113369 ms total) 2024-06-03T08:43:56.166 INFO:tasks.workunit.client.0.smithi088.stdout: api_tier_pp: 2024-06-03T08:43:56.166 INFO:tasks.workunit.client.0.smithi088.stdout: api_tier_pp: [----------] 48 tests from LibRadosTwoPoolsPP 2024-06-03T08:43:56.166 INFO:tasks.workunit.client.0.smithi088.stdout: api_tier_pp: [ RUN ] LibRadosTwoPoolsPP.Overlay 2024-06-03T08:43:56.166 INFO:tasks.workunit.client.0.smithi088.stdout: api_tier_pp: [ OK ] LibRadosTwoPoolsPP.Overlay (30457 ms) 2024-06-03T08:43:56.166 INFO:tasks.workunit.client.0.smithi088.stdout: api_tier_pp: [ RUN ] LibRadosTwoPoolsPP.Promote 2024-06-03T08:43:56.166 INFO:tasks.workunit.client.0.smithi088.stdout: api_tier_pp: [ OK ] LibRadosTwoPoolsPP.Promote (54790 ms) 2024-06-03T08:43:56.166 INFO:tasks.workunit.client.0.smithi088.stdout: api_tier_pp: [ RUN ] LibRadosTwoPoolsPP.PromoteSnap 2024-06-03T08:43:56.166 INFO:tasks.workunit.client.0.smithi088.stdout: api_tier_pp: got ENOENT removing baz in ns test-rados-api-smithi088-40561-12 2024-06-03T08:43:56.166 INFO:tasks.workunit.client.0.smithi088.stdout: api_tier_pp: got ENOENT on 1 objects, waiting a bit for snap trimming before retrying 19 more times... 2024-06-03T08:43:56.166 INFO:tasks.workunit.client.0.smithi088.stdout: api_tier_pp: got ENOENT removing baz in ns test-rados-api-smithi088-40561-12 2024-06-03T08:43:56.166 INFO:tasks.workunit.client.0.smithi088.stdout: api_tier_pp: got ENOENT on 1 objects, waiting a bit for snap trimming before retrying 18 more times... 2024-06-03T08:43:56.166 INFO:tasks.workunit.client.0.smithi088.stdout: api_tier_pp: got ENOENT removing baz in ns test-rados-api-smithi088-40561-12 2024-06-03T08:43:56.167 INFO:tasks.workunit.client.0.smithi088.stdout: api_tier_pp: got ENOENT on 1 objects, waiting a bit for snap trimming before retrying 17 more times... 2024-06-03T08:43:56.167 INFO:tasks.workunit.client.0.smithi088.stdout: api_tier_pp: got ENOENT removing baz in ns test-rados-api-smithi088-40561-12 2024-06-03T08:43:56.167 INFO:tasks.workunit.client.0.smithi088.stdout: api_tier_pp: got ENOENT on 1 objects, waiting a bit for snap trimming before retrying 16 more times... 2024-06-03T08:43:56.167 INFO:tasks.workunit.client.0.smithi088.stdout: api_tier_pp: got ENOENT removing baz in ns test-rados-api-smithi088-40561-12 2024-06-03T08:43:56.167 INFO:tasks.workunit.client.0.smithi088.stdout: api_tier_pp: got ENOENT on 1 objects, waiting a bit for snap trimming before retrying 15 more times... 2024-06-03T08:43:56.167 INFO:tasks.workunit.client.0.smithi088.stdout: api_tier_pp: got ENOENT removing baz in ns test-rados-api-smithi088-40561-12 2024-06-03T08:43:56.167 INFO:tasks.workunit.client.0.smithi088.stdout: api_tier_pp: got ENOENT on 1 objects, waiting a bit for snap trimming before retrying 14 more times... 2024-06-03T08:43:56.167 INFO:tasks.workunit.client.0.smithi088.stdout: api_tier_pp: got ENOENT removing baz in ns test-rados-api-smithi088-40561-12 2024-06-03T08:43:56.167 INFO:tasks.workunit.client.0.smithi088.stdout: api_tier_pp: got ENOENT on 1 objects, waiting a bit for snap trimming before retrying 13 more times... 2024-06-03T08:43:56.167 INFO:tasks.workunit.client.0.smithi088.stdout: api_tier_pp: got ENOENT removing baz in ns test-rados-api-smithi088-40561-12 2024-06-03T08:43:56.167 INFO:tasks.workunit.client.0.smithi088.stdout: api_tier_pp: got ENOENT on 1 objects, waiting a bit for snap trimming before retrying 12 more times... 2024-06-03T08:43:56.167 INFO:tasks.workunit.client.0.smithi088.stdout: api_tier_pp: got ENOENT removing baz in ns test-rados-api-smithi088-40561-12 2024-06-03T08:43:56.167 INFO:tasks.workunit.client.0.smithi088.stdout: api_tier_pp: got ENOENT on 1 objects, waiting a bit for snap trimming before retrying 11 more times... 2024-06-03T08:43:56.169 INFO:tasks.workunit.client.0.smithi088.stdout: api_tier_pp: got ENOENT removing baz in ns test-rados-api-smithi088-40561-12 2024-06-03T08:43:56.169 INFO:tasks.workunit.client.0.smithi088.stdout: api_tier_pp: got ENOENT on 1 objects, waiting a bit for snap trimming before retrying 10 more times... 2024-06-03T08:43:56.169 INFO:tasks.workunit.client.0.smithi088.stdout: api_tier_pp: got ENOENT removing baz in ns test-rados-api-smithi088-40561-12 2024-06-03T08:43:56.169 INFO:tasks.workunit.client.0.smithi088.stdout: api_tier_pp: got ENOENT on 1 objects, waiting a bit for snap trimming before retrying 9 more times... 2024-06-03T08:43:56.169 INFO:tasks.workunit.client.0.smithi088.stdout: api_tier_pp: got ENOENT removing baz in ns test-rados-api-smithi088-40561-12 2024-06-03T08:43:56.169 INFO:tasks.workunit.client.0.smithi088.stdout: api_tier_pp: got ENOENT on 1 objects, waiting a bit for snap trimming before retrying 8 more times... 2024-06-03T08:43:56.169 INFO:tasks.workunit.client.0.smithi088.stdout: api_tier_pp: got ENOENT removing baz in ns test-rados-api-smithi088-40561-12 2024-06-03T08:43:56.169 INFO:tasks.workunit.client.0.smithi088.stdout: api_tier_pp: got ENOENT on 1 objects, waiting a bit for snap trimming before retrying 7 more times... 2024-06-03T08:43:56.169 INFO:tasks.workunit.client.0.smithi088.stdout: api_tier_pp: got ENOENT removing baz in ns test-rados-api-smithi088-40561-12 2024-06-03T08:44:02.998 INFO:tasks.ceph.mon.a.smithi088.stderr:==00:00:08:58.323 34743== Warning: unimplemented fcntl command: 1036 2024-06-03T08:44:03.002 INFO:tasks.ceph.mon.a.smithi088.stderr:==00:00:08:58.327 34743== Warning: unimplemented fcntl command: 1036 2024-06-03T08:44:03.045 INFO:tasks.ceph.mon.a.smithi088.stderr:==00:00:08:58.370 34743== Warning: unimplemented fcntl command: 1036 2024-06-03T08:44:03.065 INFO:tasks.ceph.mon.b.smithi192.stderr:==00:00:08:58.368 34228== Warning: unimplemented fcntl command: 1036 2024-06-03T08:44:03.073 INFO:tasks.ceph.mon.c.smithi088.stderr:==00:00:08:58.398 34749== Warning: unimplemented fcntl command: 1036 2024-06-03T08:44:03.074 INFO:tasks.ceph.mon.b.smithi192.stderr:==00:00:08:58.377 34228== Warning: unimplemented fcntl command: 1036 2024-06-03T08:44:03.078 INFO:tasks.ceph.mon.c.smithi088.stderr:==00:00:08:58.404 34749== Warning: unimplemented fcntl command: 1036 2024-06-03T08:44:03.114 INFO:tasks.ceph.mon.b.smithi192.stderr:==00:00:08:58.416 34228== Warning: unimplemented fcntl command: 1036 2024-06-03T08:44:03.127 INFO:tasks.ceph.mon.c.smithi088.stderr:==00:00:08:58.452 34749== Warning: unimplemented fcntl command: 1036 2024-06-03T08:44:23.030 INFO:tasks.ceph.mon.a.smithi088.stderr:==00:00:09:18.355 34743== Warning: unimplemented fcntl command: 1036 2024-06-03T08:44:23.037 INFO:tasks.ceph.mon.a.smithi088.stderr:==00:00:09:18.362 34743== Warning: unimplemented fcntl command: 1036 2024-06-03T08:44:23.091 INFO:tasks.ceph.mon.c.smithi088.stderr:==00:00:09:18.416 34749== Warning: unimplemented fcntl command: 1036 2024-06-03T08:44:23.092 INFO:tasks.ceph.mon.a.smithi088.stderr:==00:00:09:18.418 34743== Warning: unimplemented fcntl command: 1036 2024-06-03T08:44:23.093 INFO:tasks.ceph.mon.b.smithi192.stderr:==00:00:09:18.396 34228== Warning: unimplemented fcntl command: 1036 2024-06-03T08:44:23.098 INFO:tasks.ceph.mon.b.smithi192.stderr:==00:00:09:18.400 34228== Warning: unimplemented fcntl command: 1036 2024-06-03T08:44:23.099 INFO:tasks.ceph.mon.c.smithi088.stderr:==00:00:09:18.425 34749== Warning: unimplemented fcntl command: 1036 2024-06-03T08:44:23.146 INFO:tasks.ceph.mon.b.smithi192.stderr:==00:00:09:18.448 34228== Warning: unimplemented fcntl command: 1036 2024-06-03T08:44:23.155 INFO:tasks.ceph.mon.c.smithi088.stderr:==00:00:09:18.481 34749== Warning: unimplemented fcntl command: 1036 2024-06-03T08:44:42.081 INFO:tasks.workunit.client.0.smithi088.stdout: api_tier_pp: got ENOENT on 1 obj api_list: : cursor()=6:bd63b0f1:::8:head expected=6:bd63b0f1:::8:head 2024-06-03T08:44:42.081 INFO:tasks.workunit.client.0.smithi088.stdout: api_list: > 6:bd63b0f1:::8:head -> 8 2024-06-03T08:44:42.081 INFO:tasks.workunit.client.0.smithi088.stdout: api_list: : entry=8 expected=8 2024-06-03T08:44:42.081 INFO:tasks.workunit.client.0.smithi088.stdout: api_list: : seek to 6:b29083e3:::5:head 2024-06-03T08:44:42.081 INFO:tasks.workunit.client.0.smithi088.stdout: api_list: : cursor()=6:b29083e3:::5:head expected=6:b29083e3:::5:head 2024-06-03T08:44:42.081 INFO:tasks.workunit.client.0.smithi088.stdout: api_list: > 6:b29083e3:::5:head -> 5 2024-06-03T08:44:42.081 INFO:tasks.workunit.client.0.smithi088.stdout: api_list: : entry=5 expected=5 2024-06-03T08:44:42.082 INFO:tasks.workunit.client.0.smithi088.stdout: api_list: : seek to 6:89d3ae78:::11:head 2024-06-03T08:44:42.082 INFO:tasks.workunit.client.0.smithi088.stdout: api_list: : cursor()=6:89d3ae78:::11:head expected=6:89d3ae78:::11:head 2024-06-03T08:44:42.082 INFO:tasks.workunit.client.0.smithi088.stdout: api_list: > 6:89d3ae78:::11:head -> 11 2024-06-03T08:44:42.082 INFO:tasks.workunit.client.0.smithi088.stdout: api_list: : entry=11 expected=11 2024-06-03T08:44:42.082 INFO:tasks.workunit.client.0.smithi088.stdout: api_list: : seek to 6:863748b0:::15:head 2024-06-03T08:44:42.082 INFO:tasks.workunit.client.0.smithi088.stdout: api_list: : cursor()=6:863748b0:::15:head expected=6:863748b0:::15:head 2024-06-03T08:44:42.082 INFO:tasks.workunit.client.0.smithi088.stdout: api_list: > 6:863748b0:::15:head -> 15 2024-06-03T08:44:42.082 INFO:tasks.workunit.client.0.smithi088.stdout: api_list: : entry=15 expected=15 2024-06-03T08:44:42.082 INFO:tasks.workunit.client.0.smithi088.stdout: api_list: : seek to 6:6cac518f:::0:head 2024-06-03T08:44:42.082 INFO:tasks.workunit.client.0.smithi088.stdout: api_list: : cursor()=6:6cac518f:::0:head expected=6:6cac518f:::0:head 2024-06-03T08:44:42.082 INFO:tasks.workunit.client.0.smithi088.stdout: api_list: > 6:6cac518f:::0:head -> 0 2024-06-03T08:44:42.082 INFO:tasks.workunit.client.0.smithi088.stdout: api_list: : entry=0 expected=0 2024-06-03T08:44:42.083 INFO:tasks.workunit.client.0.smithi088.stdout: api_list: : seek to 6:566253c9:::13:head 2024-06-03T08:44:42.083 INFO:tasks.workunit.client.0.smithi088.stdout: api_list: : cursor()=6:566253c9:::13:head expected=6:566253c9:::13:head 2024-06-03T08:44:42.083 INFO:tasks.workunit.client.0.smithi088.stdout: api_list: > 6:566253c9:::13:head -> 13 2024-06-03T08:44:42.083 INFO:tasks.workunit.client.0.smithi088.stdout: api_list: : entry=13 expected=13 2024-06-03T08:44:42.083 INFO:tasks.workunit.client.0.smithi088.stdout: api_list: : seek to 6:02547ec2:::1:head 2024-06-03T08:44:42.083 INFO:tasks.workunit.client.0.smithi088.stdout: api_list: : cursor()=6:02547ec2:::1:head expected=6:02547ec2:::1:head 2024-06-03T08:44:42.083 INFO:tasks.workunit.client.0.smithi088.stdout: api_list: > 6:02547ec2:::1:head -> 1 2024-06-03T08:44:42.083 INFO:tasks.workunit.client.0.smithi088.stdout: api_list: : entry=1 expected=1 2024-06-03T08:44:42.083 INFO:tasks.workunit.client.0.smithi088.stdout: api_list: : seek to 6:5c6b0b28:::7:head 2024-06-03T08:44:42.083 INFO:tasks.workunit.client.0.smithi088.stdout: api_list: : cursor()=6:5c6b0b28:::7:head expected=6:5c6b0b28:::7:head 2024-06-03T08:44:42.083 INFO:tasks.workunit.client.0.smithi088.stdout: api_list: > 6:5c6b0b28:::7:head -> 7 2024-06-03T08:44:42.083 INFO:tasks.workunit.client.0.smithi088.stdout: api_list: : entry=7 expected=7 2024-06-03T08:44:42.083 INFO:tasks.workunit.client.0.smithi088.stdout: api_list: [ OK ] LibRadosList.ListObjectsCursor (9217 ms) 2024-06-03T08:44:42.083 INFO:tasks.workunit.client.0.smithi088.stdout: api_list: [ RUN ] LibRadosList.EnumerateObjects 2024-06-03T08:44:42.084 INFO:tasks.workunit.client.0.smithi088.stdout: api_list: [ OK ] LibRadosList.EnumerateObjects (296123 ms) 2024-06-03T08:44:42.084 INFO:tasks.workunit.client.0.smithi088.stdout: api_list: [ RUN ] LibRadosList.EnumerateObjectsSplit 2024-06-03T08:44:42.084 INFO:tasks.workunit.client.0.smithi088.stdout: api_list: split 0/5 -> MIN 6:33333333::::head 2024-06-03T08:44:42.084 INFO:tasks.workunit.client.0.smithi088.stdout: api_list: split 1/5 -> 6:33333333::::head 6:66666666::::head 2024-06-03T08:44:42.084 INFO:tasks.workunit.client.0.smithi088.stdout: api_list: split 2/5 -> 6:66666666::::head 6:99999999::::head 2024-06-03T08:44:42.084 INFO:tasks.workunit.client.0.smithi088.stdout: api_list: split 3/5 -> 6:99999999::::head 6:cccccccc::::head 2024-06-03T08:44:42.084 INFO:tasks.workunit.client.0.smithi088.stdout: api_list: split 4/5 -> 6:cccccccc::::head MAX 2024-06-03T08:44:42.084 INFO:tasks.workunit.client.0.smithi088.stdout: api_list: [ OK ] LibRadosList.EnumerateObjectsSplit (14640 ms) 2024-06-03T08:44:42.084 INFO:tasks.workunit.client.0.smithi088.stdout: api_list: [----------] 7 tests from LibRadosList (398173 ms total) 2024-06-03T08:44:42.084 INFO:tasks.workunit.client.0.smithi088.stdout: api_list: 2024-06-03T08:44:42.084 INFO:tasks.workunit.client.0.smithi088.stdout: api_list: [----------] 3 tests from LibRadosListEC 2024-06-03T08:44:42.084 INFO:tasks.workunit.client.0.smithi088.stdout: api_list: [ RUN ] LibRadosListEC.ListObjects 2024-06-03T08:44:42.084 INFO:tasks.workunit.client.0.smithi088.stdout: api_list: [ OK ] LibRadosListEC.ListObjects (1607 ms) 2024-06-03T08:44:42.085 INFO:tasks.workunit.client.0.smithi088.stdout: api_list: [ RUN ] LibRadosListEC.ListObjectsNS 2024-06-03T08:44:42.085 INFO:tasks.workunit.client.0.smithi088.stdout: api_list: myset foo1,foo2,foo3 2024-06-03T08:44:42.085 INFO:tasks.workunit.client.0.smithi088.stdout: api_list: foo1 2024-06-03T08:44:42.085 INFO:tasks.workunit.client.0.smithi088.stdout: api_list: foo2 2024-06-03T08:44:42.085 INFO:tasks.workunit.client.0.smithi088.stdout: api_list: foo3 2024-06-03T08:44:42.085 INFO:tasks.workunit.client.0.smithi088.stdout: api_list: myset foo1,foo4,foo5 2024-06-03T08:44:42.085 INFO:tasks.workunit.client.0.smithi088.stdout: api_list: foo4 2024-06-03T08:44:42.085 INFO:tasks.workunit.client.0.smithi088.stdout: api_list: foo5 2024-06-03T08:44:42.085 INFO:tasks.workunit.client.0.smithi088.stdout: api_list: foo1 2024-06-03T08:44:42.085 INFO:tasks.workunit.client.0.smithi088.stdout: api_list: myset foo6,foo7 2024-06-03T08:44:42.085 INFO:tasks.workunit.client.0.smithi088.stdout: api_list: foo7 2024-06-03T08:44:42.085 INFO:tasks.workunit.client.0.smithi088.stdout: api_list: foo6 2024-06-03T08:44:42.085 INFO:tasks.workunit.client.0.smithi088.stdout: api_list: myset :foo1,:foo2,:foo3,ns1:foo1,ns1:foo4,ns1:foo5,ns2:foo6,ns2:foo7 2024-06-03T08:44:42.086 INFO:tasks.workunit.client.0.smithi088.stdout: api_list: ns1:foo4 2024-06-03T08:44:42.086 INFO:tasks.workunit.client.0.smithi088.stdout: api_list: ns1:foo5 2024-06-03T08:44:42.086 INFO:tasks.workunit.client.0.smithi088.stdout: api_list: ns2:foo7 2024-06-03T08:44:42.086 INFO:tasks.workunit.client.0.smithi088.stdout: api_list: ns2:foo6 2024-06-03T08:44:42.086 INFO:tasks.workunit.client.0.smithi088.stdout: api_list: ns1:foo1 2024-06-03T08:44:42.086 INFO:tasks.workunit.client.0.smithi088.stdout: api_list: :foo1 2024-06-03T08:44:42.086 INFO:tasks.workunit.client.0.smithi088.stdout: api_list: :foo2 2024-06-03T08:44:42.086 INFO:tasks.workunit.client.0.smithi088.stdout: api_list: :foo3 2024-06-03T08:44:42.086 INFO:tasks.workunit.client.0.smithi088.stdout: api_list: [ OK ] LibRadosListEC.ListObjectsNS (1235 ms) 2024-06-03T08:44:42.086 INFO:tasks.workunit.client.0.smithi088.stdout: api_list: [ RUN ] LibRadosListEC.ListObjectsStart 2024-06-03T08:44:42.086 INFO:tasks.workunit.client.0.smithi088.stdout: api_list: 1 0 2024-06-03T08:44:42.086 INFO:tasks.workunit.client.0.smithi088.stdout: api_list: 10 0 2024-06-03T08:44:42.086 INFO:tasks.workunit.client.0.smithi088.stdout: api_list: 13 0 2024-06-03T08:44:43.160 INFO:tasks.ceph.mon.a.smithi088.stderr:==00:00:09:38.486 34743== Warning: unimplemented fcntl command: 1036 2024-06-03T08:44:43.165 INFO:tasks.ceph.mon.a.smithi088.stderr:==00:00:09:38.491 34743== Warning: unimplemented fcntl command: 1036 2024-06-03T08:44:43.212 INFO:tasks.ceph.mon.a.smithi088.stderr:==00:00:09:38.538 34743== Warning: unimplemented fcntl command: 1036 2024-06-03T08:44:43.216 INFO:tasks.ceph.mon.b.smithi192.stderr:==00:00:09:38.519 34228== Warning: unimplemented fcntl command: 1036 2024-06-03T08:44:43.221 INFO:tasks.ceph.mon.b.smithi192.stderr:==00:00:09:38.523 34228== Warning: unimplemented fcntl command: 1036 2024-06-03T08:44:43.266 INFO:tasks.ceph.mon.b.smithi192.stderr:==00:00:09:38.568 34228== Warning: unimplemented fcntl command: 1036 2024-06-03T08:44:43.274 INFO:tasks.ceph.mon.c.smithi088.stderr:==00:00:09:38.600 34749== Warning: unimplemented fcntl command: 1036 2024-06-03T08:44:43.277 INFO:tasks.ceph.mon.c.smithi088.stderr:==00:00:09:38.602 34749== Warning: unimplemented fcntl command: 1036 2024-06-03T08:44:43.335 INFO:tasks.ceph.mon.c.smithi088.stderr:==00:00:09:38.661 34749== Warning: unimplemented fcntl command: 1036 2024-06-03T08:44:49.556 INFO:tasks.workunit.client.0.smithi088.stdout: api_list: 7 0 2024-06-03T08:44:49.556 INFO:tasks.workunit.client.0.smithi088.stdout: api_list: 14 0 2024-06-03T08:44:49.556 INFO:tasks.workunit.client.0.smithi088.stdout: api_list: 0 0 2024-06-03T08:44:49.556 INFO:tasks.workunit.client.0.smithi088.stdout: api_list: 15 0 2024-06-03T08:44:49.556 INFO:tasks.workunit.client.0.smithi088.stdout: api_list: 11 0 2024-06-03T08:44:49.556 INFO:tasks.workunit.client.0.smithi088.stdout: api_list: 5 0 2024-06-03T08:44:49.556 INFO:tasks.workunit.client.0.smithi088.stdout: api_list: 8 0 2024-06-03T08:44:49.556 INFO:tasks.workunit.client.0.smithi088.stdout: api_list: 6 0 2024-06-03T08:44:49.557 INFO:tasks.workunit.client.0.smithi088.stdout: api_list: 3 0 2024-06-03T08:44:49.557 INFO:tasks.workunit.client.0.smithi088.stdout: api_list: 4 0 2024-06-03T08:44:49.557 INFO:tasks.workunit.client.0.smithi088.stdout: api_list: 12 0 2024-06-03T08:44:49.557 INFO:tasks.workunit.client.0.smithi088.stdout: api_list: 9 0 2024-06-03T08:44:49.557 INFO:tasks.workunit.client.0.smithi088.stdout: api_list: 2 0 2024-06-03T08:44:49.557 INFO:tasks.workunit.client.0.smithi088.stdout: api_list: have 1 expect one of 0,1,10,11,12,13,14,15,2,3,4,5,6,7,8,9 2024-06-03T08:44:49.557 INFO:tasks.workunit.client.0.smithi088.stdout: api_list: [ OK ] LibRadosListEC.ListObjectsStart (2479 ms) 2024-06-03T08:44:49.557 INFO:tasks.workunit.client.0.smithi088.stdout: api_list: [----------] 3 tests from LibRadosListEC (5321 ms total) 2024-06-03T08:44:49.557 INFO:tasks.workunit.client.0.smithi088.stdout: api_list: 2024-06-03T08:44:49.557 INFO:tasks.workunit.client.0.smithi088.stdout: api_list: [----------] 1 test from LibRadosListNP 2024-06-03T08:44:49.557 INFO:tasks.workunit.client.0.smithi088.stdout: api_list: [ RUN ] LibRadosListNP.ListObjectsError 2024-06-03T08:44:49.557 INFO:tasks.workunit.client.0.smithi088.stdout: api_list: [ OK ] LibRadosListNP.ListObjectsError (2942 ms) 2024-06-03T08:44:49.558 INFO:tasks.workunit.client.0.smithi088.stdout: api_list: [----------] 1 test from LibRadosListNP (2942 ms total) 2024-06-03T08:44:49.558 INFO:tasks.workunit.client.0.smithi088.stdout: api_list: 2024-06-03T08:44:49.558 INFO:tasks.workunit.client.0.smithi088.stdout: api_list: [----------] Global test environment tear-down 2024-06-03T08:44:49.558 INFO:tasks.workunit.client.0.smithi088.stdout: api_list: [==========] 11 tests from 3 test suites ran. (416161 ms total) 2024-06-03T08:44:49.558 INFO:tasks.workunit.client.0.smithi088.stdout: api_list: [ PASSED ] 11 tests. 2024-06-03T08:45:03.172 INFO:tasks.ceph.mon.a.smithi088.stderr:==00:00:09:58.498 34743== Warning: unimplemented fcntl command: 1036 2024-06-03T08:45:03.175 INFO:tasks.ceph.mon.a.smithi088.stderr:==00:00:09:58.501 34743== Warning: unimplemented fcntl command: 1036 2024-06-03T08:45:03.234 INFO:tasks.ceph.mon.a.smithi088.stderr:==00:00:09:58.559 34743== Warning: unimplemented fcntl command: 1036 2024-06-03T08:45:03.238 INFO:tasks.ceph.mon.b.smithi192.stderr:==00:00:09:58.540 34228== Warning: unimplemented fcntl command: 1036 2024-06-03T08:45:03.240 INFO:tasks.ceph.mon.b.smithi192.stderr:==00:00:09:58.542 34228== Warning: unimplemented fcntl command: 1036 2024-06-03T08:45:03.242 INFO:tasks.ceph.mon.c.smithi088.stderr:==00:00:09:58.568 34749== Warning: unimplemented fcntl command: 1036 2024-06-03T08:45:03.244 INFO:tasks.ceph.mon.c.smithi088.stderr:==00:00:09:58.569 34749== Warning: unimplemented fcntl command: 1036 2024-06-03T08:45:03.278 INFO:tasks.ceph.mon.b.smithi192.stderr:==00:00:09:58.580 34228== Warning: unimplemented fcntl command: 1036 2024-06-03T08:45:03.292 INFO:tasks.ceph.mon.c.smithi088.stderr:==00:00:09:58.617 34749== Warning: unimplemented fcntl command: 1036 2024-06-03T08:45:23.142 INFO:tasks.ceph.mon.a.smithi088.stderr:==00:00:10:18.468 34743== Warning: unimplemented fcntl command: 1036 2024-06-03T08:45:23.144 INFO:tasks.ceph.mon.a.smithi088.stderr:==00:00:10:18.469 34743== Warning: unimplemented fcntl command: 1036 2024-06-03T08:45:23.176 INFO:tasks.ceph.mon.a.smithi088.stderr:==00:00:10:18.502 34743== Warning: unimplemented fcntl command: 1036 2024-06-03T08:45:23.189 INFO:tasks.ceph.mon.b.smithi192.stderr:==00:00:10:18.492 34228== Warning: unimplemented fcntl command: 1036 2024-06-03T08:45:23.191 INFO:tasks.ceph.mon.b.smithi192.stderr:==00:00:10:18.493 34228== Warning: unimplemented fcntl command: 1036 2024-06-03T08:45:23.192 INFO:tasks.ceph.mon.c.smithi088.stderr:==00:00:10:18.518 34749== Warning: unimplemented fcntl command: 1036 2024-06-03T08:45:23.194 INFO:tasks.ceph.mon.c.smithi088.stderr:==00:00:10:18.520 34749== Warning: unimplemented fcntl command: 1036 2024-06-03T08:45:23.212 INFO:tasks.ceph.mon.b.smithi192.stderr:==00:00:10:18.514 34228== Warning: unimplemented fcntl command: 1036 2024-06-03T08:45:23.215 INFO:tasks.ceph.mon.c.smithi088.stderr:==00:00:10:18.541 34749== Warning: unimplemented fcntl command: 1036 2024-06-03T08:45:23.389 INFO:tasks.ceph.osd.2.smithi088.stderr:2024-06-03T08:45:23.388+0000 12a6c640 -1 osd.2 328 get_health_metrics reporting 1 slow ops, oldest is osd_op(client.16647.0:1480 148.1b 148:d8b4f36d:test-rados-api-smithi088-40561-14::foo74:5 [read 0~1] snapc 0=[] ondisk+read+known_if_redirected+supports_pool_eio e312) 2024-06-03T08:45:24.430 INFO:tasks.ceph.osd.2.smithi088.stderr:2024-06-03T08:45:24.429+0000 12a6c640 -1 osd.2 328 get_health_metrics reporting 1 slow ops, oldest is osd_op(client.16647.0:1480 148.1b 148:d8b4f36d:test-rados-api-smithi088-40561-14::foo74:5 [read 0~1] snapc 0=[] ondisk+read+known_if_redirected+supports_pool_eio e312) 2024-06-03T08:45:25.465 INFO:tasks.ceph.osd.2.smithi088.stderr:2024-06-03T08:45:25.464+0000 12a6c640 -1 osd.2 328 get_health_metrics reporting 1 slow ops, oldest is osd_op(client.16647.0:1480 148.1b 148:d8b4f36d:test-rados-api-smithi088-40561-14::foo74:5 [read 0~1] snapc 0=[] ondisk+read+known_if_redirected+supports_pool_eio e312) 2024-06-03T08:45:26.488 INFO:tasks.ceph.osd.2.smithi088.stderr:2024-06-03T08:45:26.487+0000 12a6c640 -1 osd.2 328 get_health_metrics reporting 1 slow ops, oldest is osd_op(client.16647.0:1480 148.1b 148:d8b4f36d:test-rados-api-smithi088-40561-14::foo74:5 [read 0~1] snapc 0=[] ondisk+read+known_if_redirected+supports_pool_eio e312) 2024-06-03T08:45:27.442 INFO:tasks.ceph.osd.2.smithi088.stderr:2024-06-03T08:45:27.442+0000 12a6c640 -1 osd.2 328 get_health_metrics reporting 1 slow ops, oldest is osd_op(client.16647.0:1480 148.1b 148:d8b4f36d:test-rados-api-smithi088-40561-14::foo74:5 [read 0~1] snapc 0=[] ondisk+read+known_if_redirected+supports_pool_eio e312) 2024-06-03T08:45:28.473 INFO:tasks.ceph.osd.2.smithi088.stderr:2024-06-03T08:45:28.472+0000 12a6c640 -1 osd.2 328 get_health_metrics reporting 1 slow ops, oldest is osd_op(client.16647.0:1480 148.1b 148:d8b4f36d:test-rados-api-smithi088-40561-14::foo74:5 [read 0~1] snapc 0=[] ondisk+read+known_if_redirected+supports_pool_eio e312) 2024-06-03T08:45:29.433 INFO:tasks.ceph.osd.2.smithi088.stderr:2024-06-03T08:45:29.432+0000 12a6c640 -1 osd.2 328 get_health_metrics reporting 1 slow ops, oldest is osd_op(client.16647.0:1480 148.1b 148:d8b4f36d:test-rados-api-smithi088-40561-14::foo74:5 [read 0~1] snapc 0=[] ondisk+read+known_if_redirected+supports_pool_eio e312) 2024-06-03T08:45:48.213 INFO:tasks.ceph.mon.a.smithi088.stderr:==00:00:10:43.538 34743== Warning: unimplemented fcntl command: 1036 2024-06-03T08:45:48.217 INFO:tasks.ceph.mon.a.smithi088.stderr:==00:00:10:43.542 34743== Warning: unimplemented fcntl command: 1036 2024-06-03T08:45:48.259 INFO:tasks.ceph.mon.a.smithi088.stderr:==00:00:10:43.585 34743== Warning: unimplemented fcntl command: 1036 2024-06-03T08:45:48.265 INFO:tasks.ceph.mon.b.smithi192.stderr:==00:00:10:43.567 34228== Warning: unimplemented fcntl command: 1036 2024-06-03T08:45:48.266 INFO:tasks.ceph.mon.b.smithi192.stderr:==00:00:10:43.568 34228== Warning: unimplemented fcntl command: 1036 2024-06-03T08:45:48.281 INFO:tasks.ceph.mon.c.smithi088.stderr:==00:00:10:43.606 34749== Warning: unimplemented fcntl command: 1036 2024-06-03T08:45:48.282 INFO:tasks.ceph.mon.c.smithi088.stderr:==00:00:10:43.608 34749== Warning: unimplemented fcntl command: 1036 2024-06-03T08:45:48.293 INFO:tasks.ceph.mon.b.smithi192.stderr:==00:00:10:43.595 34228== Warning: unimplemented fcntl command: 1036 2024-06-03T08:45:48.309 INFO:tasks.ceph.mon.c.smithi088.stderr:==00:00:10:43.635 34749== Warning: unimplemented fcntl command: 1036 2024-06-03T08:45:57.432 INFO:tasks.ceph.mon.a.smithi088.stderr:==00:00:10:52.757 34743== Warning: unimplemented fcntl command: 1036 2024-06-03T08:45:57.435 INFO:tasks.ceph.mon.a.smithi088.stderr:==00:00:10:52.761 34743== Warning: unimplemented fcntl command: 1036 2024-06-03T08:45:57.451 INFO:tasks.ceph.mon.a.smithi088.stderr:==00:00:10:52.777 34743== Warning: unimplemented fcntl command: 1036 2024-06-03T08:45:57.487 INFO:tasks.ceph.mon.b.smithi192.stderr:==00:00:10:52.789 34228== Warning: unimplemented fcntl command: 1036 2024-06-03T08:45:57.488 INFO:tasks.ceph.mon.b.smithi192.stderr:==00:00:10:52.791 34228== Warning: unimplemented fcntl command: 1036 2024-06-03T08:45:57.489 INFO:tasks.ceph.mon.c.smithi088.stderr:==00:00:10:52.815 34749== Warning: unimplemented fcntl command: 1036 2024-06-03T08:45:57.491 INFO:tasks.ceph.mon.c.smithi088.stderr:==00:00:10:52.817 34749== Warning: unimplemented fcntl command: 1036 2024-06-03T08:45:57.505 INFO:tasks.ceph.mon.b.smithi192.stderr:==00:00:10:52.807 34228== Warning: unimplemented fcntl command: 1036 2024-06-03T08:45:57.508 INFO:tasks.ceph.mon.c.smithi088.stderr:==00:00:10:52.833 34749== Warning: unimplemented fcntl command: 1036 2024-06-03T08:46:08.257 INFO:tasks.ceph.mon.a.smithi088.stderr:==00:00:11:03.582 34743== Warning: unimplemented fcntl command: 1036 2024-06-03T08:46:08.258 INFO:tasks.ceph.mon.a.smithi088.stderr:==00:00:11:03.584 34743== Warning: unimplemented fcntl command: 1036 2024-06-03T08:46:08.272 INFO:tasks.ceph.mon.a.smithi088.stderr:==00:00:11:03.598 34743== Warning: unimplemented fcntl command: 1036 2024-06-03T08:46:08.303 INFO:tasks.ceph.mon.b.smithi192.stderr:==00:00:11:03.605 34228== Warning: unimplemented fcntl command: 1036 2024-06-03T08:46:08.304 INFO:tasks.ceph.mon.b.smithi192.stderr:==00:00:11:03.606 34228== Warning: unimplemented fcntl command: 1036 2024-06-03T08:46:08.307 INFO:tasks.ceph.mon.c.smithi088.stderr:==00:00:11:03.632 34749== Warning: unimplemented fcntl command: 1036 2024-06-03T08:46:08.309 INFO:tasks.ceph.mon.c.smithi088.stderr:==00:00:11:03.635 34749== Warning: unimplemented fcntl command: 1036 2024-06-03T08:46:08.316 INFO:tasks.ceph.mon.b.smithi192.stderr:==00:00:11:03.618 34228== Warning: unimplemented fcntl command: 1036 2024-06-03T08:46:08.321 INFO:tasks.ceph.mon.c.smithi088.stderr:==00:00:11:03.647 34749== Warning: unimplemented fcntl command: 1036 2024-06-03T08:46:08.725 INFO:tasks.ceph.osd.2.smithi088.stderr:2024-06-03T08:46:08.724+0000 12a6c640 -1 osd.2 338 get_health_metrics reporting 1 slow ops, oldest is osd_op(client.30025.0:1 172.6s0 172.43d1134e (undecoded) ondisk+write+known_if_redirected+supports_pool_eio e335) 2024-06-03T08:46:09.700 INFO:tasks.ceph.osd.2.smithi088.stderr:2024-06-03T08:46:09.699+0000 12a6c640 -1 osd.2 338 get_health_metrics reporting 1 slow ops, oldest is osd_op(client.30025.0:1 172.6s0 172.43d1134e (undecoded) ondisk+write+known_if_redirected+supports_pool_eio e335) 2024-06-03T08:46:10.651 INFO:tasks.ceph.osd.2.smithi088.stderr:2024-06-03T08:46:10.650+0000 12a6c640 -1 osd.2 338 get_health_metrics reporting 1 slow ops, oldest is osd_op(client.30025.0:1 172.6s0 172.43d1134e (undecoded) ondisk+write+known_if_redirected+supports_pool_eio e335) 2024-06-03T08:46:11.616 INFO:tasks.ceph.osd.2.smithi088.stderr:2024-06-03T08:46:11.616+0000 12a6c640 -1 osd.2 338 get_health_metrics reporting 1 slow ops, oldest is osd_op(client.30025.0:1 172.6s0 172.43d1134e (undecoded) ondisk+write+known_if_redirected+supports_pool_eio e335) 2024-06-03T08:46:12.602 INFO:tasks.ceph.osd.2.smithi088.stderr:2024-06-03T08:46:12.601+0000 12a6c640 -1 osd.2 338 get_health_metrics reporting 1 slow ops, oldest is osd_op(client.30025.0:1 172.6s0 172.43d1134e (undecoded) ondisk+write+known_if_redirected+supports_pool_eio e335) 2024-06-03T08:46:13.618 INFO:tasks.ceph.osd.2.smithi088.stderr:2024-06-03T08:46:13.618+0000 12a6c640 -1 osd.2 338 get_health_metrics reporting 1 slow ops, oldest is osd_op(client.30025.0:1 172.6s0 172.43d1134e (undecoded) ondisk+write+known_if_redirected+supports_pool_eio e335) 2024-06-03T08:46:14.583 INFO:tasks.ceph.osd.2.smithi088.stderr:2024-06-03T08:46:14.583+0000 12a6c640 -1 osd.2 338 get_health_metrics reporting 1 slow ops, oldest is osd_op(client.30025.0:1 172.6s0 172.43d1134e (undecoded) ondisk+write+known_if_redirected+supports_pool_eio e335) 2024-06-03T08:46:15.545 INFO:tasks.ceph.osd.2.smithi088.stderr:2024-06-03T08:46:15.544+0000 12a6c640 -1 osd.2 338 get_health_metrics reporting 1 slow ops, oldest is osd_op(client.30025.0:1 172.6s0 172.43d1134e (undecoded) ondisk+write+known_if_redirected+supports_pool_eio e335) 2024-06-03T08:46:16.535 INFO:tasks.ceph.osd.2.smithi088.stderr:2024-06-03T08:46:16.534+0000 12a6c640 -1 osd.2 338 get_health_metrics reporting 1 slow ops, oldest is osd_op(client.30025.0:1 172.6s0 172.43d1134e (undecoded) ondisk+write+known_if_redirected+supports_pool_eio e335) 2024-06-03T08:46:17.558 INFO:tasks.ceph.osd.2.smithi088.stderr:2024-06-03T08:46:17.557+0000 12a6c640 -1 osd.2 338 get_health_metrics reporting 1 slow ops, oldest is osd_op(client.30025.0:1 172.6s0 172.43d1134e (undecoded) ondisk+write+known_if_redirected+supports_pool_eio e335) 2024-06-03T08:46:18.346 INFO:tasks.ceph.osd.5.smithi192.stderr:2024-06-03T08:46:18.343+0000 12a6c640 -1 osd.5 322 heartbeat_check: no reply from 172.21.15.88:6814 osd.3 since back 2024-06-03T08:44:49.833435+0000 front 2024-06-03T08:44:49.774528+0000 (oldest deadline 2024-06-03T08:46:14.470469+0000) 2024-06-03T08:46:18.409 INFO:tasks.ceph.osd.5.smithi192.stderr:2024-06-03T08:46:18.402+0000 12a6c640 -1 osd.5 322 get_health_metrics reporting 1 slow ops, oldest is osd_op(client.30435.0:1 170.6s0 170.7fc1f406 (undecoded) ondisk+write+known_if_redirected+supports_pool_eio e322) 2024-06-03T08:46:18.575 INFO:tasks.ceph.osd.2.smithi088.stderr:2024-06-03T08:46:18.574+0000 12a6c640 -1 osd.2 338 get_health_metrics reporting 1 slow ops, oldest is osd_op(client.30025.0:1 172.6s0 172.43d1134e (undecoded) ondisk+write+known_if_redirected+supports_pool_eio e335) 2024-06-03T08:46:19.381 INFO:tasks.ceph.osd.5.smithi192.stderr:2024-06-03T08:46:19.378+0000 12a6c640 -1 osd.5 336 get_health_metrics reporting 1 slow ops, oldest is osd_op(client.30435.0:1 170.6s0 170.7fc1f406 (undecoded) ondisk+write+known_if_redirected+supports_pool_eio e322) 2024-06-03T08:46:19.550 INFO:tasks.ceph.osd.2.smithi088.stderr:2024-06-03T08:46:19.549+0000 12a6c640 -1 osd.2 338 get_health_metrics reporting 1 slow ops, oldest is osd_op(client.30025.0:1 172.6s0 172.43d1134e (undecoded) ondisk+write+known_if_redirected+supports_pool_eio e335) 2024-06-03T08:46:28.334 INFO:tasks.ceph.mon.a.smithi088.stderr:==00:00:11:23.659 34743== Warning: unimplemented fcntl command: 1036 2024-06-03T08:46:28.336 INFO:tasks.ceph.mon.a.smithi088.stderr:==00:00:11:23.661 34743== Warning: unimplemented fcntl command: 1036 2024-06-03T08:46:28.364 INFO:tasks.ceph.mon.a.smithi088.stderr:==00:00:11:23.689 34743== Warning: unimplemented fcntl command: 1036 2024-06-03T08:46:28.389 INFO:tasks.ceph.mon.b.smithi192.stderr:==00:00:11:23.691 34228== Warning: unimplemented fcntl command: 1036 2024-06-03T08:46:28.391 INFO:tasks.ceph.mon.b.smithi192.stderr:==00:00:11:23.693 34228== Warning: unimplemented fcntl command: 1036 2024-06-03T08:46:28.400 INFO:tasks.ceph.mon.c.smithi088.stderr:==00:00:11:23.725 34749== Warning: unimplemented fcntl command: 1036 2024-06-03T08:46:28.401 INFO:tasks.ceph.mon.c.smithi088.stderr:==00:00:11:23.727 34749== Warning: unimplemented fcntl command: 1036 2024-06-03T08:46:28.494 INFO:tasks.ceph.mon.b.smithi192.stderr:==00:00:11:23.795 34228== Warning: unimplemented fcntl command: 1036 2024-06-03T08:46:28.519 INFO:tasks.ceph.mon.c.smithi088.stderr:==00:00:11:23.844 34749== Warning: unimplemented fcntl command: 1036 2024-06-03T08:46:44.422 INFO:tasks.workunit.client.0.smithi088.stdout:pleWrite 2024-06-03T08:46:44.423 INFO:tasks.workunit.client.0.smithi088.stdout: api_aio: [ OK ] LibRadosAioEC.SimpleWrite (6981 ms) 2024-06-03T08:46:44.423 INFO:tasks.workunit.client.0.smithi088.stdout: api_aio: [ RUN ] LibRadosAioEC.WaitForComplete 2024-06-03T08:46:44.423 INFO:tasks.workunit.client.0.smithi088.stdout: api_aio: [ OK ] LibRadosAioEC.WaitForComplete (7211 ms) 2024-06-03T08:46:44.423 INFO:tasks.workunit.client.0.smithi088.stdout: api_aio: [ RUN ] LibRadosAioEC.RoundTrip 2024-06-03T08:46:44.423 INFO:tasks.workunit.client.0.smithi088.stdout: api_aio: [ OK ] LibRadosAioEC.RoundTrip (11810 ms) 2024-06-03T08:46:44.423 INFO:tasks.workunit.client.0.smithi088.stdout: api_aio: [ RUN ] LibRadosAioEC.RoundTrip2 2024-06-03T08:46:44.423 INFO:tasks.workunit.client.0.smithi088.stdout: api_aio: [ OK ] LibRadosAioEC.RoundTrip2 (7434 ms) 2024-06-03T08:46:44.423 INFO:tasks.workunit.client.0.smithi088.stdout: api_aio: [ RUN ] LibRadosAioEC.RoundTripAppend 2024-06-03T08:46:44.423 INFO:tasks.workunit.client.0.smithi088.stdout: api_aio: [ OK ] LibRadosAioEC.RoundTripAppend (7931 ms) 2024-06-03T08:46:44.423 INFO:tasks.workunit.client.0.smithi088.stdout: api_aio: [ RUN ] LibRadosAioEC.IsComplete 2024-06-03T08:46:44.423 INFO:tasks.workunit.client.0.smithi088.stdout: api_aio: [ OK ] LibRadosAioEC.IsComplete (7757 ms) 2024-06-03T08:46:44.423 INFO:tasks.workunit.client.0.smithi088.stdout: api_aio: [ RUN ] LibRadosAioEC.IsSafe 2024-06-03T08:46:44.424 INFO:tasks.workunit.client.0.smithi088.stdout: api_aio: [ OK ] LibRadosAioEC.IsSafe (6999 ms) 2024-06-03T08:46:44.424 INFO:tasks.workunit.client.0.smithi088.stdout: api_aio: [ RUN ] LibRadosAioEC.ReturnValue 2024-06-03T08:46:44.424 INFO:tasks.workunit.client.0.smithi088.stdout: api_aio: [ OK ] LibRadosAioEC.ReturnValue (7156 ms) 2024-06-03T08:46:44.424 INFO:tasks.workunit.client.0.smithi088.stdout: api_aio: [ RUN ] LibRadosAioEC.Flush 2024-06-03T08:46:44.424 INFO:tasks.workunit.client.0.smithi088.stdout: api_aio: [ OK ] LibRadosAioEC.Flush (6956 ms) 2024-06-03T08:46:44.424 INFO:tasks.workunit.client.0.smithi088.stdout: api_aio: [ RUN ] LibRadosAioEC.FlushAsync 2024-06-03T08:46:44.424 INFO:tasks.workunit.client.0.smithi088.stdout: api_aio: [ OK ] LibRadosAioEC.FlushAsync (7176 ms) 2024-06-03T08:46:44.424 INFO:tasks.workunit.client.0.smithi088.stdout: api_aio: [ RUN ] LibRadosAioEC.RoundTripWriteFull 2024-06-03T08:46:44.424 INFO:tasks.workunit.client.0.smithi088.stdout: api_aio: [ OK ] LibRadosAioEC.RoundTripWriteFull (7440 ms) 2024-06-03T08:46:44.424 INFO:tasks.workunit.client.0.smithi088.stdout: api_aio: [ RUN ] LibRadosAioEC.SimpleStat 2024-06-03T08:46:44.424 INFO:tasks.workunit.client.0.smithi088.stdout: api_aio: [ OK ] LibRadosAioEC.SimpleStat (6860 ms) 2024-06-03T08:46:44.424 INFO:tasks.workunit.client.0.smithi088.stdout: api_aio: [ RUN ] LibRadosAioEC.SimpleStatNS 2024-06-03T08:46:44.424 INFO:tasks.workunit.client.0.smithi088.stdout: api_aio: [ OK ] LibRadosAioEC.SimpleStatNS (84368 ms) 2024-06-03T08:46:44.425 INFO:tasks.workunit.client.0.smithi088.stdout: api_aio: [ RUN ] LibRadosAioEC.StatRemove 2024-06-03T08:46:44.425 INFO:tasks.workunit.client.0.smithi088.stdout: api_aio: [ OK ] LibRadosAioEC.StatRemove (7591 ms) 2024-06-03T08:46:44.425 INFO:tasks.workunit.client.0.smithi088.stdout: api_aio: [ RUN ] LibRadosAioEC.ExecuteClass 2024-06-03T08:46:44.425 INFO:tasks.workunit.client.0.smithi088.stdout: api_aio: [ OK ] LibRadosAioEC.ExecuteClass (7001 ms) 2024-06-03T08:46:44.425 INFO:tasks.workunit.client.0.smithi088.stdout: api_aio: [ RUN ] LibRadosAioEC.MultiWrite 2024-06-03T08:46:44.425 INFO:tasks.workunit.client.0.smithi088.stdout: api_aio: [ OK ] LibRadosAioEC.MultiWrite (6888 ms) 2024-06-03T08:46:44.425 INFO:tasks.workunit.client.0.smithi088.stdout: api_aio: [----------] 16 tests from LibRadosAioEC (197559 ms total) 2024-06-03T08:46:44.425 INFO:tasks.workunit.client.0.smithi088.stdout: api_aio: 2024-06-03T08:46:44.425 INFO:tasks.workunit.client.0.smithi088.stdout: api_aio: [----------] Global test environment tear-down 2024-06-03T08:46:44.425 INFO:tasks.workunit.client.0.smithi088.stdout: api_aio: [==========] 42 tests from 2 test suites ran. (531066 ms total) 2024-06-03T08:46:44.425 INFO:tasks.workunit.client.0.smithi088.stdout: api_aio: [ PASSED ] 42 tests. 2024-06-03T08:46:48.512 INFO:tasks.ceph.mon.a.smithi088.stderr:==00:00:11:43.838 34743== Warning: unimplemented fcntl command: 1036 2024-06-03T08:46:48.516 INFO:tasks.ceph.mon.a.smithi088.stderr:==00:00:11:43.841 34743== Warning: unimplemented fcntl command: 1036 2024-06-03T08:46:48.549 INFO:tasks.ceph.mon.a.smithi088.stderr:==00:00:11:43.875 34743== Warning: unimplemented fcntl command: 1036 2024-06-03T08:46:48.567 INFO:tasks.ceph.mon.c.smithi088.stderr:==00:00:11:43.892 34749== Warning: unimplemented fcntl command: 1036 2024-06-03T08:46:48.568 INFO:tasks.ceph.mon.c.smithi088.stderr:==00:00:11:43.894 34749== Warning: unimplemented fcntl command: 1036 2024-06-03T08:46:48.577 INFO:tasks.ceph.mon.b.smithi192.stderr:==00:00:11:43.879 34228== Warning: unimplemented fcntl command: 1036 2024-06-03T08:46:48.592 INFO:tasks.ceph.mon.b.smithi192.stderr:==00:00:11:43.894 34228== Warning: unimplemented fcntl command: 1036 2024-06-03T08:46:48.606 INFO:tasks.ceph.mon.c.smithi088.stderr:==00:00:11:43.931 34749== Warning: unimplemented fcntl command: 1036 2024-06-03T08:46:48.631 INFO:tasks.ceph.mon.b.smithi192.stderr:==00:00:11:43.933 34228== Warning: unimplemented fcntl command: 1036 2024-06-03T08:47:04.349 INFO:tasks.ceph.osd.4.smithi192.stderr:2024-06-03T08:47:04.348+0000 12a6c640 -1 osd.4 368 get_health_metrics reporting 1 slow ops, oldest is osd_op(client.30040.0:1 176.5s0 176.ac5de9d5 (undecoded) ondisk+read+known_if_redirected+supports_pool_eio e352) 2024-06-03T08:47:05.377 INFO:tasks.ceph.osd.4.smithi192.stderr:2024-06-03T08:47:05.377+0000 12a6c640 -1 osd.4 368 get_health_metrics reporting 1 slow ops, oldest is osd_op(client.30040.0:1 176.5s0 176.ac5de9d5 (undecoded) ondisk+read+known_if_redirected+supports_pool_eio e352) 2024-06-03T08:47:06.411 INFO:tasks.ceph.osd.4.smithi192.stderr:2024-06-03T08:47:06.410+0000 12a6c640 -1 osd.4 368 get_health_metrics reporting 1 slow ops, oldest is osd_op(client.30040.0:1 176.5s0 176.ac5de9d5 (undecoded) ondisk+read+known_if_redirected+supports_pool_eio e352) 2024-06-03T08:47:07.444 INFO:tasks.ceph.osd.4.smithi192.stderr:2024-06-03T08:47:07.443+0000 12a6c640 -1 osd.4 368 get_health_metrics reporting 1 slow ops, oldest is osd_op(client.30040.0:1 176.5s0 176.ac5de9d5 (undecoded) ondisk+read+known_if_redirected+supports_pool_eio e352) 2024-06-03T08:47:08.433 INFO:tasks.ceph.mon.a.smithi088.stderr:==00:00:12:03.757 34743== Warning: unimplemented fcntl command: 1036 2024-06-03T08:47:08.435 INFO:tasks.ceph.mon.a.smithi088.stderr:==00:00:12:03.760 34743== Warning: unimplemented fcntl command: 1036 2024-06-03T08:47:08.465 INFO:tasks.ceph.mon.a.smithi088.stderr:==00:00:12:03.790 34743== Warning: unimplemented fcntl command: 1036 2024-06-03T08:47:08.482 INFO:tasks.ceph.mon.b.smithi192.stderr:==00:00:12:03.784 34228== Warning: unimplemented fcntl command: 1036 2024-06-03T08:47:08.483 INFO:tasks.ceph.mon.b.smithi192.stderr:==00:00:12:03.785 34228== Warning: unimplemented fcntl command: 1036 2024-06-03T08:47:08.488 INFO:tasks.ceph.osd.4.smithi192.stderr:2024-06-03T08:47:08.487+0000 12a6c640 -1 osd.4 368 get_health_metrics reporting 1 slow ops, oldest is osd_op(client.30040.0:1 176.5s0 176.ac5de9d5 (undecoded) ondisk+read+known_if_redirected+supports_pool_eio e352) 2024-06-03T08:47:08.492 INFO:tasks.ceph.mon.c.smithi088.stderr:==00:00:12:03.818 34749== Warning: unimplemented fcntl command: 1036 2024-06-03T08:47:08.496 INFO:tasks.ceph.mon.c.smithi088.stderr:==00:00:12:03.821 34749== Warning: unimplemented fcntl command: 1036 2024-06-03T08:47:08.503 INFO:tasks.ceph.mon.b.smithi192.stderr:==00:00:12:03.805 34228== Warning: unimplemented fcntl command: 1036 2024-06-03T08:47:08.515 INFO:tasks.ceph.mon.c.smithi088.stderr:==00:00:12:03.841 34749== Warning: unimplemented fcntl command: 1036 2024-06-03T08:47:09.443 INFO:tasks.ceph.osd.4.smithi192.stderr:2024-06-03T08:47:09.443+0000 12a6c640 -1 osd.4 368 get_health_metrics reporting 1 slow ops, oldest is osd_op(client.30040.0:1 176.5s0 176.ac5de9d5 (undecoded) ondisk+read+known_if_redirected+supports_pool_eio e352) 2024-06-03T08:47:10.413 INFO:tasks.ceph.osd.4.smithi192.stderr:2024-06-03T08:47:10.413+0000 12a6c640 -1 osd.4 368 get_health_metrics reporting 1 slow ops, oldest is osd_op(client.30040.0:1 176.5s0 176.ac5de9d5 (undecoded) ondisk+read+known_if_redirected+supports_pool_eio e352) 2024-06-03T08:47:11.424 INFO:tasks.ceph.osd.4.smithi192.stderr:2024-06-03T08:47:11.423+0000 12a6c640 -1 osd.4 368 get_health_metrics reporting 1 slow ops, oldest is osd_op(client.30040.0:1 176.5s0 176.ac5de9d5 (undecoded) ondisk+read+known_if_redirected+supports_pool_eio e352) 2024-06-03T08:47:12.471 INFO:tasks.ceph.osd.4.smithi192.stderr:2024-06-03T08:47:12.471+0000 12a6c640 -1 osd.4 368 get_health_metrics reporting 1 slow ops, oldest is osd_op(client.30040.0:1 176.5s0 176.ac5de9d5 (undecoded) ondisk+read+known_if_redirected+supports_pool_eio e352) 2024-06-03T08:47:13.436 INFO:tasks.ceph.osd.4.smithi192.stderr:2024-06-03T08:47:13.435+0000 12a6c640 -1 osd.4 368 get_health_metrics reporting 1 slow ops, oldest is osd_op(client.30040.0:1 176.5s0 176.ac5de9d5 (undecoded) ondisk+read+known_if_redirected+supports_pool_eio e352) 2024-06-03T08:47:14.423 INFO:tasks.ceph.osd.4.smithi192.stderr:2024-06-03T08:47:14.423+0000 12a6c640 -1 osd.4 368 get_health_metrics reporting 1 slow ops, oldest is osd_op(client.30040.0:1 176.5s0 176.ac5de9d5 (undecoded) ondisk+read+known_if_redirected+supports_pool_eio e352) 2024-06-03T08:47:15.464 INFO:tasks.ceph.osd.4.smithi192.stderr:2024-06-03T08:47:15.463+0000 12a6c640 -1 osd.4 368 get_health_metrics reporting 1 slow ops, oldest is osd_op(client.30040.0:1 176.5s0 176.ac5de9d5 (undecoded) ondisk+read+known_if_redirected+supports_pool_eio e352) 2024-06-03T08:47:16.490 INFO:tasks.ceph.osd.4.smithi192.stderr:2024-06-03T08:47:16.489+0000 12a6c640 -1 osd.4 368 get_health_metrics reporting 1 slow ops, oldest is osd_op(client.30040.0:1 176.5s0 176.ac5de9d5 (undecoded) ondisk+read+known_if_redirected+supports_pool_eio e352) 2024-06-03T08:47:17.455 INFO:tasks.ceph.osd.4.smithi192.stderr:2024-06-03T08:47:17.454+0000 12a6c640 -1 osd.4 368 get_health_metrics reporting 1 slow ops, oldest is osd_op(client.30040.0:1 176.5s0 176.ac5de9d5 (undecoded) ondisk+read+known_if_redirected+supports_pool_eio e352) 2024-06-03T08:47:18.450 INFO:tasks.ceph.osd.4.smithi192.stderr:2024-06-03T08:47:18.449+0000 12a6c640 -1 osd.4 368 get_health_metrics reporting 1 slow ops, oldest is osd_op(client.30040.0:1 176.5s0 176.ac5de9d5 (undecoded) ondisk+read+known_if_redirected+supports_pool_eio e352) 2024-06-03T08:47:19.447 INFO:tasks.ceph.osd.4.smithi192.stderr:2024-06-03T08:47:19.445+0000 12a6c640 -1 osd.4 368 get_health_metrics reporting 1 slow ops, oldest is osd_op(client.30040.0:1 176.5s0 176.ac5de9d5 (undecoded) ondisk+read+known_if_redirected+supports_pool_eio e352) 2024-06-03T08:47:20.426 INFO:tasks.ceph.osd.4.smithi192.stderr:2024-06-03T08:47:20.425+0000 12a6c640 -1 osd.4 368 get_health_metrics reporting 1 slow ops, oldest is osd_op(client.30040.0:1 176.5s0 176.ac5de9d5 (undecoded) ondisk+read+known_if_redirected+supports_pool_eio e352) 2024-06-03T08:47:21.432 INFO:tasks.ceph.osd.4.smithi192.stderr:2024-06-03T08:47:21.430+0000 12a6c640 -1 osd.4 368 get_health_metrics reporting 1 slow ops, oldest is osd_op(client.30040.0:1 176.5s0 176.ac5de9d5 (undecoded) ondisk+read+known_if_redirected+supports_pool_eio e352) 2024-06-03T08:47:22.453 INFO:tasks.ceph.osd.4.smithi192.stderr:2024-06-03T08:47:22.452+0000 12a6c640 -1 osd.4 368 get_health_metrics reporting 1 slow ops, oldest is osd_op(client.30040.0:1 176.5s0 176.ac5de9d5 (undecoded) ondisk+read+known_if_redirected+supports_pool_eio e352) 2024-06-03T08:47:23.491 INFO:tasks.ceph.osd.4.smithi192.stderr:2024-06-03T08:47:23.490+0000 12a6c640 -1 osd.4 368 get_health_metrics reporting 1 slow ops, oldest is osd_op(client.30040.0:1 176.5s0 176.ac5de9d5 (undecoded) ondisk+read+known_if_redirected+supports_pool_eio e352) 2024-06-03T08:47:24.477 INFO:tasks.ceph.osd.4.smithi192.stderr:2024-06-03T08:47:24.475+0000 12a6c640 -1 osd.4 368 get_health_metrics reporting 1 slow ops, oldest is osd_op(client.30040.0:1 176.5s0 176.ac5de9d5 (undecoded) ondisk+read+known_if_redirected+supports_pool_eio e352) 2024-06-03T08:47:25.439 INFO:tasks.ceph.osd.4.smithi192.stderr:2024-06-03T08:47:25.437+0000 12a6c640 -1 osd.4 368 get_health_metrics reporting 1 slow ops, oldest is osd_op(client.30040.0:1 176.5s0 176.ac5de9d5 (undecoded) ondisk+read+known_if_redirected+supports_pool_eio e352) 2024-06-03T08:47:26.472 INFO:tasks.ceph.osd.4.smithi192.stderr:2024-06-03T08:47:26.472+0000 12a6c640 -1 osd.4 368 get_health_metrics reporting 1 slow ops, oldest is osd_op(client.30040.0:1 176.5s0 176.ac5de9d5 (undecoded) ondisk+read+known_if_redirected+supports_pool_eio e352) 2024-06-03T08:47:27.454 INFO:tasks.ceph.osd.4.smithi192.stderr:2024-06-03T08:47:27.453+0000 12a6c640 -1 osd.4 368 get_health_metrics reporting 1 slow ops, oldest is osd_op(client.30040.0:1 176.5s0 176.ac5de9d5 (undecoded) ondisk+read+known_if_redirected+supports_pool_eio e352) 2024-06-03T08:47:28.419 INFO:tasks.ceph.osd.4.smithi192.stderr:2024-06-03T08:47:28.418+0000 12a6c640 -1 osd.4 368 get_health_metrics reporting 1 slow ops, oldest is osd_op(client.30040.0:1 176.5s0 176.ac5de9d5 (undecoded) ondisk+read+known_if_redirected+supports_pool_eio e352) 2024-06-03T08:47:28.548 INFO:tasks.ceph.mon.a.smithi088.stderr:==00:00:12:23.874 34743== Warning: unimplemented fcntl command: 1036 2024-06-03T08:47:28.607 INFO:tasks.ceph.mon.a.smithi088.stderr:==00:00:12:23.932 34743== Warning: unimplemented fcntl command: 1036 2024-06-03T08:47:28.627 INFO:tasks.ceph.mon.a.smithi088.stderr:==00:00:12:23.953 34743== Warning: unimplemented fcntl command: 1036 2024-06-03T08:47:28.646 INFO:tasks.ceph.mon.b.smithi192.stderr:==00:00:12:23.949 34228== Warning: unimplemented fcntl command: 1036 2024-06-03T08:47:28.650 INFO:tasks.ceph.mon.b.smithi192.stderr:==00:00:12:23.953 34228== Warning: unimplemented fcntl command: 1036 2024-06-03T08:47:28.658 INFO:tasks.ceph.mon.c.smithi088.stderr:==00:00:12:23.983 34749== Warning: unimplemented fcntl command: 1036 2024-06-03T08:47:28.659 INFO:tasks.ceph.mon.c.smithi088.stderr:==00:00:12:23.985 34749== Warning: unimplemented fcntl command: 1036 2024-06-03T08:47:28.668 INFO:tasks.ceph.mon.b.smithi192.stderr:==00:00:12:23.970 34228== Warning: unimplemented fcntl command: 1036 2024-06-03T08:47:28.683 INFO:tasks.ceph.mon.c.smithi088.stderr:==00:00:12:24.009 34749== Warning: unimplemented fcntl command: 1036 2024-06-03T08:47:29.394 INFO:tasks.ceph.osd.4.smithi192.stderr:2024-06-03T08:47:29.393+0000 12a6c640 -1 osd.4 368 get_health_metrics reporting 1 slow ops, oldest is osd_op(client.30040.0:1 176.5s0 176.ac5de9d5 (undecoded) ondisk+read+known_if_redirected+supports_pool_eio e352) 2024-06-03T08:47:30.443 INFO:tasks.ceph.osd.4.smithi192.stderr:2024-06-03T08:47:30.442+0000 12a6c640 -1 osd.4 368 get_health_metrics reporting 1 slow ops, oldest is osd_op(client.30040.0:1 176.5s0 176.ac5de9d5 (undecoded) ondisk+read+known_if_redirected+supports_pool_eio e352) 2024-06-03T08:47:31.441 INFO:tasks.ceph.osd.4.smithi192.stderr:2024-06-03T08:47:31.441+0000 12a6c640 -1 osd.4 368 get_health_metrics reporting 1 slow ops, oldest is osd_op(client.30040.0:1 176.5s0 176.ac5de9d5 (undecoded) ondisk+read+known_if_redirected+supports_pool_eio e352) 2024-06-03T08:47:32.401 INFO:tasks.ceph.osd.4.smithi192.stderr:2024-06-03T08:47:32.400+0000 12a6c640 -1 osd.4 368 get_health_metrics reporting 1 slow ops, oldest is osd_op(client.30040.0:1 176.5s0 176.ac5de9d5 (undecoded) ondisk+read+known_if_redirected+supports_pool_eio e352) 2024-06-03T08:47:43.732 INFO:tasks.ceph.mon.a.smithi088.stderr:==00:00:12:39.057 34743== Warning: unimplemented fcntl command: 1036 2024-06-03T08:47:43.737 INFO:tasks.ceph.mon.a.smithi088.stderr:==00:00:12:39.062 34743== Warning: unimplemented fcntl command: 1036 2024-06-03T08:47:43.782 INFO:tasks.ceph.mon.a.smithi088.stderr:==00:00:12:39.108 34743== Warning: unimplemented fcntl command: 1036 2024-06-03T08:47:43.796 INFO:tasks.ceph.mon.c.smithi088.stderr:==00:00:12:39.121 34749== Warning: unimplemented fcntl command: 1036 2024-06-03T08:47:43.796 INFO:tasks.ceph.mon.b.smithi192.stderr:==00:00:12:39.099 34228== Warning: unimplemented fcntl command: 1036 2024-06-03T08:47:43.798 INFO:tasks.ceph.mon.b.smithi192.stderr:==00:00:12:39.100 34228== Warning: unimplemented fcntl command: 1036 2024-06-03T08:47:43.809 INFO:tasks.ceph.mon.c.smithi088.stderr:==00:00:12:39.135 34749== Warning: unimplemented fcntl command: 1036 2024-06-03T08:47:43.834 INFO:tasks.ceph.mon.b.smithi192.stderr:==00:00:12:39.136 34228== Warning: unimplemented fcntl command: 1036 2024-06-03T08:47:43.856 INFO:tasks.ceph.mon.c.smithi088.stderr:==00:00:12:39.182 34749== Warning: unimplemented fcntl command: 1036 2024-06-03T08:47:48.570 INFO:tasks.ceph.mon.a.smithi088.stderr:==00:00:12:43.895 34743== Warning: unimplemented fcntl command: 1036 2024-06-03T08:47:48.582 INFO:tasks.ceph.mon.a.smithi088.stderr:==00:00:12:43.908 34743== Warning: unimplemented fcntl command: 1036 2024-06-03T08:47:48.598 INFO:tasks.ceph.mon.a.smithi088.stderr:==00:00:12:43.924 34743== Warning: unimplemented fcntl command: 1036 2024-06-03T08:47:48.618 INFO:tasks.ceph.mon.b.smithi192.stderr:==00:00:12:43.920 34228== Warning: unimplemented fcntl command: 1036 2024-06-03T08:47:48.619 INFO:tasks.ceph.mon.b.smithi192.stderr:==00:00:12:43.921 34228== Warning: unimplemented fcntl command: 1036 2024-06-03T08:47:48.630 INFO:tasks.ceph.mon.c.smithi088.stderr:==00:00:12:43.955 34749== Warning: unimplemented fcntl command: 1036 2024-06-03T08:47:48.631 INFO:tasks.ceph.mon.c.smithi088.stderr:==00:00:12:43.957 34749== Warning: unimplemented fcntl command: 1036 2024-06-03T08:47:48.632 INFO:tasks.ceph.mon.b.smithi192.stderr:==00:00:12:43.935 34228== Warning: unimplemented fcntl command: 1036 2024-06-03T08:47:48.646 INFO:tasks.ceph.mon.c.smithi088.stderr:==00:00:12:43.972 34749== Warning: unimplemented fcntl command: 1036 2024-06-03T08:47:56.826 INFO:tasks.workunit.client.0.smithi088.stdout:ibRadosAio.RoundTripAppendPP (4260 ms) 2024-06-03T08:47:56.826 INFO:tasks.workunit.client.0.smithi088.stdout: api_aio_pp: [ RUN ] LibRadosAio.RacingRemovePP 2024-06-03T08:47:56.826 INFO:tasks.workunit.client.0.smithi088.stdout: api_aio_pp: [ OK ] LibRadosAio.RacingRemovePP (3143 ms) 2024-06-03T08:47:56.826 INFO:tasks.workunit.client.0.smithi088.stdout: api_aio_pp: [ RUN ] LibRadosAio.RoundTripCmpExtPP 2024-06-03T08:47:56.826 INFO:tasks.workunit.client.0.smithi088.stdout: api_aio_pp: [ OK ] LibRadosAio.RoundTripCmpExtPP (4692 ms) 2024-06-03T08:47:56.827 INFO:tasks.workunit.client.0.smithi088.stdout: api_aio_pp: [ RUN ] LibRadosAio.RoundTripCmpExtPP2 2024-06-03T08:47:56.827 INFO:tasks.workunit.client.0.smithi088.stdout: api_aio_pp: [ OK ] LibRadosAio.RoundTripCmpExtPP2 (3166 ms) 2024-06-03T08:47:56.827 INFO:tasks.workunit.client.0.smithi088.stdout: api_aio_pp: [ RUN ] LibRadosAio.PoolEIOFlag 2024-06-03T08:47:56.827 INFO:tasks.workunit.client.0.smithi088.stdout: api_aio_pp: setting pool EIO 2024-06-03T08:47:56.827 INFO:tasks.workunit.client.0.smithi088.stdout: api_aio_pp: max_success 43, min_failed 44 2024-06-03T08:47:56.827 INFO:tasks.workunit.client.0.smithi088.stdout: api_aio_pp: [ OK ] LibRadosAio.PoolEIOFlag (6926 ms) 2024-06-03T08:47:56.827 INFO:tasks.workunit.client.0.smithi088.stdout: api_aio_pp: [ RUN ] LibRadosAio.MultiReads 2024-06-03T08:47:56.827 INFO:tasks.workunit.client.0.smithi088.stdout: api_aio_pp: [ OK ] LibRadosAio.MultiReads (4059 ms) 2024-06-03T08:47:56.827 INFO:tasks.workunit.client.0.smithi088.stdout: api_aio_pp: [----------] 32 tests from LibRadosAio (346628 ms total) 2024-06-03T08:47:56.827 INFO:tasks.workunit.client.0.smithi088.stdout: api_aio_pp: 2024-06-03T08:47:56.827 INFO:tasks.workunit.client.0.smithi088.stdout: api_aio_pp: [----------] 4 tests from LibRadosAioPP 2024-06-03T08:47:56.828 INFO:tasks.workunit.client.0.smithi088.stdout: api_aio_pp: [ RUN ] LibRadosAioPP.ReadIntoBufferlist 2024-06-03T08:47:56.828 INFO:tasks.workunit.client.0.smithi088.stdout: api_aio_pp: [ OK ] LibRadosAioPP.ReadIntoBufferlist (2960 ms) 2024-06-03T08:47:56.828 INFO:tasks.workunit.client.0.smithi088.stdout: api_aio_pp: [ RUN ] LibRadosAioPP.XattrsRoundTripPP 2024-06-03T08:47:56.828 INFO:tasks.workunit.client.0.smithi088.stdout: api_aio_pp: [ OK ] LibRadosAioPP.XattrsRoundTripPP (9432 ms) 2024-06-03T08:47:56.828 INFO:tasks.workunit.client.0.smithi088.stdout: api_aio_pp: [ RUN ] LibRadosAioPP.RmXattrPP 2024-06-03T08:47:56.828 INFO:tasks.workunit.client.0.smithi088.stdout: api_aio_pp: [ OK ] LibRadosAioPP.RmXattrPP (15859 ms) 2024-06-03T08:47:56.828 INFO:tasks.workunit.client.0.smithi088.stdout: api_aio_pp: [ RUN ] LibRadosAioPP.RemoveTestPP 2024-06-03T08:47:56.828 INFO:tasks.workunit.client.0.smithi088.stdout: api_aio_pp: [ OK ] LibRadosAioPP.RemoveTestPP (3358 ms) 2024-06-03T08:47:56.828 INFO:tasks.workunit.client.0.smithi088.stdout: api_aio_pp: [----------] 4 tests from LibRadosAioPP (31609 ms total) 2024-06-03T08:47:56.828 INFO:tasks.workunit.client.0.smithi088.stdout: api_aio_pp: 2024-06-03T08:47:56.828 INFO:tasks.workunit.client.0.smithi088.stdout: api_aio_pp: [----------] 1 test from LibRadosIoPP 2024-06-03T08:47:56.828 INFO:tasks.workunit.client.0.smithi088.stdout: api_aio_pp: [ RUN ] LibRadosIoPP.XattrListPP 2024-06-03T08:47:56.828 INFO:tasks.workunit.client.0.smithi088.stdout: api_aio_pp: [ OK ] LibRadosIoPP.XattrListPP (3380 ms) 2024-06-03T08:47:56.829 INFO:tasks.workunit.client.0.smithi088.stdout: api_aio_pp: [----------] 1 test from LibRadosIoPP (3380 ms total) 2024-06-03T08:47:56.829 INFO:tasks.workunit.client.0.smithi088.stdout: api_aio_pp: 2024-06-03T08:47:56.829 INFO:tasks.workunit.client.0.smithi088.stdout: api_aio_pp: [----------] 20 tests from LibRadosAioEC 2024-06-03T08:47:56.829 INFO:tasks.workunit.client.0.smithi088.stdout: api_aio_pp: [ RUN ] LibRadosAioEC.SimpleWritePP 2024-06-03T08:47:56.829 INFO:tasks.workunit.client.0.smithi088.stdout: api_aio_pp: [ OK ] LibRadosAioEC.SimpleWritePP (14159 ms) 2024-06-03T08:47:56.829 INFO:tasks.workunit.client.0.smithi088.stdout: api_aio_pp: [ RUN ] LibRadosAioEC.WaitForSafePP 2024-06-03T08:47:56.829 INFO:tasks.workunit.client.0.smithi088.stdout: api_aio_pp: [ OK ] LibRadosAioEC.WaitForSafePP (6972 ms) 2024-06-03T08:47:56.829 INFO:tasks.workunit.client.0.smithi088.stdout: api_aio_pp: [ RUN ] LibRadosAioEC.RoundTripPP 2024-06-03T08:47:56.829 INFO:tasks.workunit.client.0.smithi088.stdout: api_aio_pp: [ OK ] LibRadosAioEC.RoundTripPP (7157 ms) 2024-06-03T08:47:56.829 INFO:tasks.workunit.client.0.smithi088.stdout: api_aio_pp: [ RUN ] LibRadosAioEC.RoundTripPP2 2024-06-03T08:47:56.829 INFO:tasks.workunit.client.0.smithi088.stdout: api_aio_pp: [ OK ] LibRadosAioEC.RoundTripPP2 (7304 ms) 2024-06-03T08:47:56.829 INFO:tasks.workunit.client.0.smithi088.stdout: api_aio_pp: [ RUN ] LibRadosAioEC.RoundTripPP3 2024-06-03T08:47:56.829 INFO:tasks.workunit.client.0.smithi088.stdout: api_aio_pp: [ OK ] LibRadosAioEC.RoundTripPP3 (3544 ms) 2024-06-03T08:47:56.829 INFO:tasks.workunit.client.0.smithi088.stdout: api_aio_pp: [ RUN ] LibRadosAioEC.RoundTripSparseReadPP 2024-06-03T08:47:56.830 INFO:tasks.workunit.client.0.smithi088.stdout: api_aio_pp: [ OK ] LibRadosAioEC.RoundTripSparseReadPP (7615 ms) 2024-06-03T08:47:56.830 INFO:tasks.workunit.client.0.smithi088.stdout: api_aio_pp: [ RUN ] LibRadosAioEC.RoundTripAppendPP 2024-06-03T08:47:56.830 INFO:tasks.workunit.client.0.smithi088.stdout: api_aio_pp: [ OK ] LibRadosAioEC.RoundTripAppendPP (32967 ms) 2024-06-03T08:47:56.830 INFO:tasks.workunit.client.0.smithi088.stdout: api_aio_pp: [ RUN ] LibRadosAioEC.IsCompletePP 2024-06-03T08:47:56.830 INFO:tasks.workunit.client.0.smithi088.stdout: api_aio_pp: [ OK ] LibRadosAioEC.IsCompletePP (48229 ms) 2024-06-03T08:47:56.830 INFO:tasks.workunit.client.0.smithi088.stdout: api_aio_pp: [ RUN ] LibRadosAioEC.IsSafePP 2024-06-03T08:47:56.830 INFO:tasks.workunit.client.0.smithi088.stdout: api_aio_pp: [ OK ] LibRadosAioEC.IsSafePP (7617 ms) 2024-06-03T08:47:56.830 INFO:tasks.workunit.client.0.smithi088.stdout: api_aio_pp: [ RUN ] LibRadosAioEC.ReturnValuePP 2024-06-03T08:47:56.830 INFO:tasks.workunit.client.0.smithi088.stdout: api_aio_pp: [ OK ] LibRadosAioEC.ReturnValuePP (64029 ms) 2024-06-03T08:47:56.830 INFO:tasks.workunit.client.0.smithi088.stdout: api_aio_pp: [ RUN ] LibRadosAioEC.FlushPP 2024-06-03T08:47:56.830 INFO:tasks.workunit.client.0.smithi088.stdout: api_aio_pp: [ OK ] LibRadosAioEC.FlushPP (7553 ms) 2024-06-03T08:47:56.830 INFO:tasks.workunit.client.0.smithi088.stdout: api_aio_pp: [ RUN ] LibRadosAioEC.FlushAsyncPP 2024-06-03T08:47:56.830 INFO:tasks.workunit.client.0.smithi088.stdout: api_aio_pp: [ OK ] LibRadosAioEC.FlushAsyncPP (6812 ms) 2024-06-03T08:47:56.831 INFO:tasks.workunit.client.0.smithi088.stdout: api_aio_pp: [ RUN ] LibRadosAioEC.RoundTripWriteFullPP 2024-06-03T08:47:56.831 INFO:tasks.workunit.client.0.smithi088.stdout: api_aio_pp: [ OK ] LibRadosAioEC.RoundTripWriteFullPP (7898 ms) 2024-06-03T08:47:58.767 INFO:tasks.ceph.mon.a.smithi088.stderr:==00:00:12:54.093 34743== Warning: unimplemented fcntl command: 1036 2024-06-03T08:47:58.772 INFO:tasks.ceph.mon.a.smithi088.stderr:==00:00:12:54.098 34743== Warning: unimplemented fcntl command: 1036 2024-06-03T08:47:58.811 INFO:tasks.ceph.mon.a.smithi088.stderr:==00:00:12:54.137 34743== Warning: unimplemented fcntl command: 1036 2024-06-03T08:47:58.838 INFO:tasks.ceph.mon.b.smithi192.stderr:==00:00:12:54.140 34228== Warning: unimplemented fcntl command: 1036 2024-06-03T08:47:58.839 INFO:tasks.ceph.mon.c.smithi088.stderr:==00:00:12:54.165 34749== Warning: unimplemented fcntl command: 1036 2024-06-03T08:47:58.843 INFO:tasks.ceph.mon.c.smithi088.stderr:==00:00:12:54.169 34749== Warning: unimplemented fcntl command: 1036 2024-06-03T08:47:58.862 INFO:tasks.ceph.mon.b.smithi192.stderr:==00:00:12:54.164 34228== Warning: unimplemented fcntl command: 1036 2024-06-03T08:47:58.878 INFO:tasks.ceph.mon.c.smithi088.stderr:==00:00:12:54.204 34749== Warning: unimplemented fcntl command: 1036 2024-06-03T08:47:58.904 INFO:tasks.ceph.mon.b.smithi192.stderr:==00:00:12:54.206 34228== Warning: unimplemented fcntl command: 1036 2024-06-03T08:48:08.664 INFO:tasks.ceph.mon.a.smithi088.stderr:==00:00:13:03.989 34743== Warning: unimplemented fcntl command: 1036 2024-06-03T08:48:08.668 INFO:tasks.ceph.mon.a.smithi088.stderr:==00:00:13:03.994 34743== Warning: unimplemented fcntl command: 1036 2024-06-03T08:48:08.696 INFO:tasks.ceph.mon.a.smithi088.stderr:==00:00:13:04.021 34743== Warning: unimplemented fcntl command: 1036 2024-06-03T08:48:08.718 INFO:tasks.ceph.mon.b.smithi192.stderr:==00:00:13:04.021 34228== Warning: unimplemented fcntl command: 1036 2024-06-03T08:48:08.721 INFO:tasks.ceph.mon.b.smithi192.stderr:==00:00:13:04.024 34228== Warning: unimplemented fcntl command: 1036 2024-06-03T08:48:08.731 INFO:tasks.ceph.mon.c.smithi088.stderr:==00:00:13:04.057 34749== Warning: unimplemented fcntl command: 1036 2024-06-03T08:48:08.736 INFO:tasks.ceph.mon.c.smithi088.stderr:==00:00:13:04.062 34749== Warning: unimplemented fcntl command: 1036 2024-06-03T08:48:08.747 INFO:tasks.ceph.mon.b.smithi192.stderr:==00:00:13:04.050 34228== Warning: unimplemented fcntl command: 1036 2024-06-03T08:48:08.761 INFO:tasks.ceph.mon.c.smithi088.stderr:==00:00:13:04.086 34749== Warning: unimplemented fcntl command: 1036 2024-06-03T08:48:13.779 INFO:tasks.ceph.mon.a.smithi088.stderr:==00:00:13:09.105 34743== Warning: unimplemented fcntl command: 1036 2024-06-03T08:48:13.828 INFO:tasks.ceph.mon.a.smithi088.stderr:==00:00:13:09.154 34743== Warning: unimplemented fcntl command: 1036 2024-06-03T08:48:13.852 INFO:tasks.ceph.mon.a.smithi088.stderr:==00:00:13:09.177 34743== Warning: unimplemented fcntl command: 1036 2024-06-03T08:48:13.876 INFO:tasks.ceph.mon.b.smithi192.stderr:==00:00:13:09.178 34228== Warning: unimplemented fcntl command: 1036 2024-06-03T08:48:13.880 INFO:tasks.ceph.mon.b.smithi192.stderr:==00:00:13:09.182 34228== Warning: unimplemented fcntl command: 1036 2024-06-03T08:48:13.894 INFO:tasks.ceph.mon.c.smithi088.stderr:==00:00:13:09.220 34749== Warning: unimplemented fcntl command: 1036 2024-06-03T08:48:13.895 INFO:tasks.ceph.mon.c.smithi088.stderr:==00:00:13:09.221 34749== Warning: unimplemented fcntl command: 1036 2024-06-03T08:48:13.898 INFO:tasks.ceph.mon.b.smithi192.stderr:==00:00:13:09.201 34228== Warning: unimplemented fcntl command: 1036 2024-06-03T08:48:13.913 INFO:tasks.ceph.mon.c.smithi088.stderr:==00:00:13:09.239 34749== Warning: unimplemented fcntl command: 1036 2024-06-03T08:48:15.025 INFO:tasks.ceph.mon.a.smithi088.stderr:==00:00:13:10.351 34743== Warning: unimplemented fcntl command: 1036 2024-06-03T08:48:15.027 INFO:tasks.ceph.mon.a.smithi088.stderr:==00:00:13:10.353 34743== Warning: unimplemented fcntl command: 1036 2024-06-03T08:48:15.042 INFO:tasks.ceph.mon.a.smithi088.stderr:==00:00:13:10.368 34743== Warning: unimplemented fcntl command: 1036 2024-06-03T08:48:15.078 INFO:tasks.ceph.mon.b.smithi192.stderr:==00:00:13:10.381 34228== Warning: unimplemented fcntl command: 1036 2024-06-03T08:48:15.081 INFO:tasks.ceph.mon.b.smithi192.stderr:==00:00:13:10.384 34228== Warning: unimplemented fcntl command: 1036 2024-06-03T08:48:15.090 INFO:tasks.ceph.mon.c.smithi088.stderr:==00:00:13:10.416 34749== Warning: unimplemented fcntl command: 1036 2024-06-03T08:48:15.091 INFO:tasks.ceph.mon.c.smithi088.stderr:==00:00:13:10.417 34749== Warning: unimplemented fcntl command: 1036 2024-06-03T08:48:15.095 INFO:tasks.ceph.mon.b.smithi192.stderr:==00:00:13:10.398 34228== Warning: unimplemented fcntl command: 1036 2024-06-03T08:48:15.107 INFO:tasks.ceph.mon.c.smithi088.stderr:==00:00:13:10.433 34749== Warning: unimplemented fcntl command: 1036 2024-06-03T08:48:28.797 INFO:tasks.ceph.mon.a.smithi088.stderr:==00:00:13:24.123 34743== Warning: unimplemented fcntl command: 1036 2024-06-03T08:48:28.800 INFO:tasks.ceph.mon.a.smithi088.stderr:==00:00:13:24.125 34743== Warning: unimplemented fcntl command: 1036 2024-06-03T08:48:28.847 INFO:tasks.ceph.mon.c.smithi088.stderr:==00:00:13:24.173 34749== Warning: unimplemented fcntl command: 1036 2024-06-03T08:48:28.850 INFO:tasks.ceph.mon.a.smithi088.stderr:==00:00:13:24.175 34743== Warning: unimplemented fcntl command: 1036 2024-06-03T08:48:28.851 INFO:tasks.ceph.mon.c.smithi088.stderr:==00:00:13:24.177 34749== Warning: unimplemented fcntl command: 1036 2024-06-03T08:48:28.862 INFO:tasks.ceph.mon.b.smithi192.stderr:==00:00:13:24.165 34228== Warning: unimplemented fcntl command: 1036 2024-06-03T08:48:28.863 INFO:tasks.ceph.mon.b.smithi192.stderr:==00:00:13:24.166 34228== Warning: unimplemented fcntl command: 1036 2024-06-03T08:48:28.880 INFO:tasks.ceph.mon.c.smithi088.stderr:==00:00:13:24.206 34749== Warning: unimplemented fcntl command: 1036 2024-06-03T08:48:28.892 INFO:tasks.ceph.mon.b.smithi192.stderr:==00:00:13:24.194 34228== Warning: unimplemented fcntl command: 1036 2024-06-03T08:48:29.540 INFO:tasks.ceph.mon.c.smithi088.stderr:==00:00:13:24.866 34749== Warning: unimplemented fcntl command: 1036 2024-06-03T08:48:29.587 INFO:tasks.ceph.mon.c.smithi088.stderr:==00:00:13:24.913 34749== Warning: unimplemented fcntl command: 1036 2024-06-03T08:48:29.592 INFO:tasks.ceph.mon.b.smithi192.stderr:==00:00:13:24.894 34228== Warning: unimplemented fcntl command: 1036 2024-06-03T08:48:29.593 INFO:tasks.ceph.mon.b.smithi192.stderr:==00:00:13:24.895 34228== Warning: unimplemented fcntl command: 1036 2024-06-03T08:48:29.603 INFO:tasks.ceph.mon.b.smithi192.stderr:==00:00:13:24.906 34228== Warning: unimplemented fcntl command: 1036 2024-06-03T08:48:29.608 INFO:tasks.ceph.mon.c.smithi088.stderr:==00:00:13:24.934 34749== Warning: unimplemented fcntl command: 1036 2024-06-03T08:48:29.645 INFO:tasks.ceph.mon.a.smithi088.stderr:==00:00:13:24.971 34743== Warning: unimplemented fcntl command: 1036 2024-06-03T08:48:29.648 INFO:tasks.ceph.mon.a.smithi088.stderr:==00:00:13:24.974 34743== Warning: unimplemented fcntl command: 1036 2024-06-03T08:48:29.659 INFO:tasks.ceph.mon.a.smithi088.stderr:==00:00:13:24.985 34743== Warning: unimplemented fcntl command: 1036 2024-06-03T08:48:43.195 INFO:tasks.workunit.client.0.smithi088.stdout: api_aio_pp: [ RUN ] LibRadosAioEC.RoundTripWriteFullPP2 2024-06-03T08:48:43.195 INFO:tasks.workunit.client.0.smithi088.stdout: api_aio_pp: [ OK ] LibRadosAioEC.RoundTripWriteFullPP2 (2899 ms) 2024-06-03T08:48:43.196 INFO:tasks.workunit.client.0.smithi088.stdout: api_aio_pp: [ RUN ] LibRadosAioEC.SimpleStatPP 2024-06-03T08:48:43.196 INFO:tasks.workunit.client.0.smithi088.stdout: api_aio_pp: [ OK ] LibRadosAioEC.SimpleStatPP (7222 ms) 2024-06-03T08:48:43.196 INFO:tasks.workunit.client.0.smithi088.stdout: api_aio_pp: [ RUN ] LibRadosAioEC.SimpleStatPPNS 2024-06-03T08:48:43.196 INFO:tasks.workunit.client.0.smithi088.stdout: api_aio_pp: [ OK ] LibRadosAioEC.SimpleStatPPNS (6949 ms) 2024-06-03T08:48:43.196 INFO:tasks.workunit.client.0.smithi088.stdout: api_aio_pp: [ RUN ] LibRadosAioEC.StatRemovePP 2024-06-03T08:48:43.196 INFO:tasks.workunit.client.0.smithi088.stdout: api_aio_pp: [ OK ] LibRadosAioEC.StatRemovePP (7555 ms) 2024-06-03T08:48:43.196 INFO:tasks.workunit.client.0.smithi088.stdout: api_aio_pp: [ RUN ] LibRadosAioEC.ExecuteClassPP 2024-06-03T08:48:43.196 INFO:tasks.workunit.client.0.smithi088.stdout: api_aio_pp: [ OK ] LibRadosAioEC.ExecuteClassPP (6943 ms) 2024-06-03T08:48:43.196 INFO:tasks.workunit.client.0.smithi088.stdout: api_aio_pp: [ RUN ] LibRadosAioEC.OmapPP 2024-06-03T08:48:43.196 INFO:tasks.workunit.client.0.smithi088.stdout: api_aio_pp: [ OK ] LibRadosAioEC.OmapPP (7658 ms) 2024-06-03T08:48:43.196 INFO:tasks.workunit.client.0.smithi088.stdout: api_aio_pp: [ RUN ] LibRadosAioEC.MultiWritePP 2024-06-03T08:48:43.196 INFO:tasks.workunit.client.0.smithi088.stdout: api_aio_pp: [ OK ] LibRadosAioEC.MultiWritePP (7142 ms) 2024-06-03T08:48:43.196 INFO:tasks.workunit.client.0.smithi088.stdout: api_aio_pp: [----------] 20 tests from LibRadosAioEC (268224 ms total) 2024-06-03T08:48:43.197 INFO:tasks.workunit.client.0.smithi088.stdout: api_aio_pp: 2024-06-03T08:48:43.197 INFO:tasks.workunit.client.0.smithi088.stdout: api_aio_pp: [----------] Global test environment tear-down 2024-06-03T08:48:43.197 INFO:tasks.workunit.client.0.smithi088.stdout: api_aio_pp: [==========] 57 tests from 4 test suites ran. (649841 ms total) 2024-06-03T08:48:43.197 INFO:tasks.workunit.client.0.smithi088.stdout: api_aio_pp: [ PASSED ] 57 tests. 2024-06-03T08:48:43.849 INFO:tasks.ceph.mon.a.smithi088.stderr:==00:00:13:39.175 34743== Warning: unimplemented fcntl command: 1036 2024-06-03T08:48:43.851 INFO:tasks.ceph.mon.a.smithi088.stderr:==00:00:13:39.176 34743== Warning: unimplemented fcntl command: 1036 2024-06-03T08:48:43.880 INFO:tasks.ceph.mon.a.smithi088.stderr:==00:00:13:39.206 34743== Warning: unimplemented fcntl command: 1036 2024-06-03T08:48:43.901 INFO:tasks.ceph.mon.b.smithi192.stderr:==00:00:13:39.203 34228== Warning: unimplemented fcntl command: 1036 2024-06-03T08:48:43.902 INFO:tasks.ceph.mon.b.smithi192.stderr:==00:00:13:39.204 34228== Warning: unimplemented fcntl command: 1036 2024-06-03T08:48:43.924 INFO:tasks.ceph.mon.c.smithi088.stderr:==00:00:13:39.249 34749== Warning: unimplemented fcntl command: 1036 2024-06-03T08:48:43.925 INFO:tasks.ceph.mon.c.smithi088.stderr:==00:00:13:39.251 34749== Warning: unimplemented fcntl command: 1036 2024-06-03T08:48:43.944 INFO:tasks.ceph.mon.b.smithi192.stderr:==00:00:13:39.246 34228== Warning: unimplemented fcntl command: 1036 2024-06-03T08:48:43.953 INFO:tasks.ceph.mon.c.smithi088.stderr:==00:00:13:39.278 34749== Warning: unimplemented fcntl command: 1036 2024-06-03T08:48:48.863 INFO:tasks.ceph.mon.a.smithi088.stderr:==00:00:13:44.188 34743== Warning: unimplemented fcntl command: 1036 2024-06-03T08:48:48.866 INFO:tasks.ceph.mon.a.smithi088.stderr:==00:00:13:44.192 34743== Warning: unimplemented fcntl command: 1036 2024-06-03T08:48:48.896 INFO:tasks.ceph.mon.a.smithi088.stderr:==00:00:13:44.222 34743== Warning: unimplemented fcntl command: 1036 2024-06-03T08:48:48.911 INFO:tasks.ceph.mon.b.smithi192.stderr:==00:00:13:44.214 34228== Warning: unimplemented fcntl command: 1036 2024-06-03T08:48:48.914 INFO:tasks.ceph.mon.b.smithi192.stderr:==00:00:13:44.216 34228== Warning: unimplemented fcntl command: 1036 2024-06-03T08:48:48.914 INFO:tasks.ceph.mon.c.smithi088.stderr:==00:00:13:44.239 34749== Warning: unimplemented fcntl command: 1036 2024-06-03T08:48:48.915 INFO:tasks.ceph.mon.c.smithi088.stderr:==00:00:13:44.241 34749== Warning: unimplemented fcntl command: 1036 2024-06-03T08:48:48.925 INFO:tasks.ceph.mon.c.smithi088.stderr:==00:00:13:44.251 34749== Warning: unimplemented fcntl command: 1036 2024-06-03T08:48:48.927 INFO:tasks.ceph.mon.b.smithi192.stderr:==00:00:13:44.229 34228== Warning: unimplemented fcntl command: 1036 2024-06-03T08:49:03.896 INFO:tasks.ceph.mon.a.smithi088.stderr:==00:00:13:59.221 34743== Warning: unimplemented fcntl command: 1036 2024-06-03T08:49:03.897 INFO:tasks.ceph.mon.a.smithi088.stderr:==00:00:13:59.223 34743== Warning: unimplemented fcntl command: 1036 2024-06-03T08:49:03.912 INFO:tasks.ceph.mon.a.smithi088.stderr:==00:00:13:59.237 34743== Warning: unimplemented fcntl command: 1036 2024-06-03T08:49:03.946 INFO:tasks.ceph.mon.c.smithi088.stderr:==00:00:13:59.272 34749== Warning: unimplemented fcntl command: 1036 2024-06-03T08:49:03.947 INFO:tasks.ceph.mon.b.smithi192.stderr:==00:00:13:59.249 34228== Warning: unimplemented fcntl command: 1036 2024-06-03T08:49:03.948 INFO:tasks.ceph.mon.b.smithi192.stderr:==00:00:13:59.250 34228== Warning: unimplemented fcntl command: 1036 2024-06-03T08:49:03.949 INFO:tasks.ceph.mon.c.smithi088.stderr:==00:00:13:59.274 34749== Warning: unimplemented fcntl command: 1036 2024-06-03T08:49:03.963 INFO:tasks.ceph.mon.b.smithi192.stderr:==00:00:13:59.265 34228== Warning: unimplemented fcntl command: 1036 2024-06-03T08:49:03.971 INFO:tasks.ceph.mon.c.smithi088.stderr:==00:00:13:59.296 34749== Warning: unimplemented fcntl command: 1036 2024-06-03T08:49:08.934 INFO:tasks.ceph.mon.a.smithi088.stderr:==00:00:14:04.259 34743== Warning: unimplemented fcntl command: 1036 2024-06-03T08:49:08.937 INFO:tasks.ceph.mon.a.smithi088.stderr:==00:00:14:04.263 34743== Warning: unimplemented fcntl command: 1036 2024-06-03T08:49:08.950 INFO:tasks.ceph.mon.a.smithi088.stderr:==00:00:14:04.276 34743== Warning: unimplemented fcntl command: 1036 2024-06-03T08:49:08.985 INFO:tasks.ceph.mon.b.smithi192.stderr:==00:00:14:04.288 34228== Warning: unimplemented fcntl command: 1036 2024-06-03T08:49:08.987 INFO:tasks.ceph.mon.c.smithi088.stderr:==00:00:14:04.312 34749== Warning: unimplemented fcntl command: 1036 2024-06-03T08:49:08.989 INFO:tasks.ceph.mon.b.smithi192.stderr:==00:00:14:04.291 34228== Warning: unimplemented fcntl command: 1036 2024-06-03T08:49:08.990 INFO:tasks.ceph.mon.c.smithi088.stderr:==00:00:14:04.316 34749== Warning: unimplemented fcntl command: 1036 2024-06-03T08:49:08.999 INFO:tasks.ceph.mon.b.smithi192.stderr:==00:00:14:04.302 34228== Warning: unimplemented fcntl command: 1036 2024-06-03T08:49:09.001 INFO:tasks.ceph.mon.c.smithi088.stderr:==00:00:14:04.326 34749== Warning: unimplemented fcntl command: 1036 2024-06-03T08:49:23.969 INFO:tasks.ceph.mon.a.smithi088.stderr:==00:00:14:19.294 34743== Warning: unimplemented fcntl command: 1036 2024-06-03T08:49:23.970 INFO:tasks.ceph.mon.a.smithi088.stderr:==00:00:14:19.296 34743== Warning: unimplemented fcntl command: 1036 2024-06-03T08:49:23.990 INFO:tasks.ceph.mon.a.smithi088.stderr:==00:00:14:19.316 34743== Warning: unimplemented fcntl command: 1036 2024-06-03T08:49:24.018 INFO:tasks.ceph.mon.b.smithi192.stderr:==00:00:14:19.321 34228== Warning: unimplemented fcntl command: 1036 2024-06-03T08:49:24.020 INFO:tasks.ceph.mon.b.smithi192.stderr:==00:00:14:19.322 34228== Warning: unimplemented fcntl command: 1036 2024-06-03T08:49:24.021 INFO:tasks.ceph.mon.c.smithi088.stderr:==00:00:14:19.347 34749== Warning: unimplemented fcntl command: 1036 2024-06-03T08:49:24.024 INFO:tasks.ceph.mon.c.smithi088.stderr:==00:00:14:19.350 34749== Warning: unimplemented fcntl command: 1036 2024-06-03T08:49:24.035 INFO:tasks.ceph.mon.b.smithi192.stderr:==00:00:14:19.337 34228== Warning: unimplemented fcntl command: 1036 2024-06-03T08:49:24.041 INFO:tasks.ceph.mon.c.smithi088.stderr:==00:00:14:19.367 34749== Warning: unimplemented fcntl command: 1036 2024-06-03T08:49:28.930 INFO:tasks.ceph.mon.a.smithi088.stderr:==00:00:14:24.256 34743== Warning: unimplemented fcntl command: 1036 2024-06-03T08:49:28.977 INFO:tasks.ceph.mon.a.smithi088.stderr:==00:00:14:24.303 34743== Warning: unimplemented fcntl command: 1036 2024-06-03T08:49:28.987 INFO:tasks.ceph.mon.a.smithi088.stderr:==00:00:14:24.313 34743== Warning: unimplemented fcntl command: 1036 2024-06-03T08:49:29.023 INFO:tasks.ceph.mon.b.smithi192.stderr:==00:00:14:24.325 34228== Warning: unimplemented fcntl command: 1036 2024-06-03T08:49:29.024 INFO:tasks.ceph.mon.b.smithi192.stderr:==00:00:14:24.327 34228== Warning: unimplemented fcntl command: 1036 2024-06-03T08:49:29.027 INFO:tasks.ceph.mon.c.smithi088.stderr:==00:00:14:24.352 34749== Warning: unimplemented fcntl command: 1036 2024-06-03T08:49:29.028 INFO:tasks.ceph.mon.c.smithi088.stderr:==00:00:14:24.354 34749== Warning: unimplemented fcntl command: 1036 2024-06-03T08:49:29.033 INFO:tasks.ceph.mon.b.smithi192.stderr:==00:00:14:24.336 34228== Warning: unimplemented fcntl command: 1036 2024-06-03T08:49:29.038 INFO:tasks.ceph.mon.c.smithi088.stderr:==00:00:14:24.364 34749== Warning: unimplemented fcntl command: 1036 2024-06-03T08:49:39.035 INFO:tasks.ceph.mon.a.smithi088.stderr:==00:00:14:34.360 34743== Warning: unimplemented fcntl command: 1036 2024-06-03T08:49:39.038 INFO:tasks.ceph.mon.a.smithi088.stderr:==00:00:14:34.363 34743== Warning: unimplemented fcntl command: 1036 2024-06-03T08:49:39.060 INFO:tasks.ceph.mon.a.smithi088.stderr:==00:00:14:34.386 34743== Warning: unimplemented fcntl command: 1036 2024-06-03T08:49:39.087 INFO:tasks.ceph.mon.c.smithi088.stderr:==00:00:14:34.413 34749== Warning: unimplemented fcntl command: 1036 2024-06-03T08:49:39.089 INFO:tasks.ceph.mon.c.smithi088.stderr:==00:00:14:34.414 34749== Warning: unimplemented fcntl command: 1036 2024-06-03T08:49:39.090 INFO:tasks.ceph.mon.b.smithi192.stderr:==00:00:14:34.393 34228== Warning: unimplemented fcntl command: 1036 2024-06-03T08:49:39.092 INFO:tasks.ceph.mon.b.smithi192.stderr:==00:00:14:34.394 34228== Warning: unimplemented fcntl command: 1036 2024-06-03T08:49:39.111 INFO:tasks.ceph.mon.c.smithi088.stderr:==00:00:14:34.437 34749== Warning: unimplemented fcntl command: 1036 2024-06-03T08:49:39.114 INFO:tasks.ceph.mon.b.smithi192.stderr:==00:00:14:34.416 34228== Warning: unimplemented fcntl command: 1036 2024-06-03T08:49:49.063 INFO:tasks.ceph.mon.a.smithi088.stderr:==00:00:14:44.388 34743== Warning: unimplemented fcntl command: 1036 2024-06-03T08:49:49.066 INFO:tasks.ceph.mon.a.smithi088.stderr:==00:00:14:44.392 34743== Warning: unimplemented fcntl command: 1036 2024-06-03T08:49:49.089 INFO:tasks.ceph.mon.a.smithi088.stderr:==00:00:14:44.415 34743== Warning: unimplemented fcntl command: 1036 2024-06-03T08:49:49.113 INFO:tasks.ceph.mon.b.smithi192.stderr:==00:00:14:44.415 34228== Warning: unimplemented fcntl command: 1036 2024-06-03T08:49:49.114 INFO:tasks.ceph.mon.b.smithi192.stderr:==00:00:14:44.416 34228== Warning: unimplemented fcntl command: 1036 2024-06-03T08:49:49.116 INFO:tasks.ceph.mon.c.smithi088.stderr:==00:00:14:44.442 34749== Warning: unimplemented fcntl command: 1036 2024-06-03T08:49:49.117 INFO:tasks.ceph.mon.c.smithi088.stderr:==00:00:14:44.443 34749== Warning: unimplemented fcntl command: 1036 2024-06-03T08:49:49.135 INFO:tasks.ceph.mon.b.smithi192.stderr:==00:00:14:44.438 34228== Warning: unimplemented fcntl command: 1036 2024-06-03T08:49:49.141 INFO:tasks.ceph.mon.c.smithi088.stderr:==00:00:14:44.467 34749== Warning: unimplemented fcntl command: 1036 2024-06-03T08:50:09.150 INFO:tasks.ceph.mon.a.smithi088.stderr:==00:00:15:04.475 34743== Warning: unimplemented fcntl command: 1036 2024-06-03T08:50:09.154 INFO:tasks.ceph.mon.a.smithi088.stderr:==00:00:15:04.480 34743== Warning: unimplemented fcntl command: 1036 2024-06-03T08:50:09.196 INFO:tasks.ceph.mon.a.smithi088.stderr:==00:00:15:04.522 34743== Warning: unimplemented fcntl command: 1036 2024-06-03T08:50:09.210 INFO:tasks.ceph.mon.b.smithi192.stderr:==00:00:15:04.512 34228== Warning: unimplemented fcntl command: 1036 2024-06-03T08:50:09.212 INFO:tasks.ceph.mon.b.smithi192.stderr:==00:00:15:04.514 34228== Warning: unimplemented fcntl command: 1036 2024-06-03T08:50:09.216 INFO:tasks.ceph.mon.c.smithi088.stderr:==00:00:15:04.542 34749== Warning: unimplemented fcntl command: 1036 2024-06-03T08:50:09.220 INFO:tasks.ceph.mon.c.smithi088.stderr:==00:00:15:04.546 34749== Warning: unimplemented fcntl command: 1036 2024-06-03T08:50:09.238 INFO:tasks.ceph.mon.b.smithi192.stderr:==00:00:15:04.541 34228== Warning: unimplemented fcntl command: 1036 2024-06-03T08:50:09.249 INFO:tasks.ceph.mon.c.smithi088.stderr:==00:00:15:04.575 34749== Warning: unimplemented fcntl command: 1036 2024-06-03T08:50:09.859 INFO:tasks.ceph.mon.b.smithi192.stderr:==00:00:15:05.161 34228== Warning: unimplemented fcntl command: 1036 2024-06-03T08:50:09.860 INFO:tasks.ceph.mon.b.smithi192.stderr:==00:00:15:05.162 34228== Warning: unimplemented fcntl command: 1036 2024-06-03T08:50:09.865 INFO:tasks.ceph.mon.a.smithi088.stderr:==00:00:15:05.191 34743== Warning: unimplemented fcntl command: 1036 2024-06-03T08:50:09.866 INFO:tasks.ceph.mon.a.smithi088.stderr:==00:00:15:05.192 34743== Warning: unimplemented fcntl command: 1036 2024-06-03T08:50:09.868 INFO:tasks.ceph.mon.b.smithi192.stderr:==00:00:15:05.171 34228== Warning: unimplemented fcntl command: 1036 2024-06-03T08:50:09.881 INFO:tasks.ceph.mon.a.smithi088.stderr:==00:00:15:05.207 34743== Warning: unimplemented fcntl command: 1036 2024-06-03T08:50:09.886 INFO:tasks.ceph.mon.c.smithi088.stderr:==00:00:15:05.212 34749== Warning: unimplemented fcntl command: 1036 2024-06-03T08:50:09.889 INFO:tasks.ceph.mon.c.smithi088.stderr:==00:00:15:05.213 34749== Warning: unimplemented fcntl command: 1036 2024-06-03T08:50:09.895 INFO:tasks.ceph.mon.c.smithi088.stderr:==00:00:15:05.221 34749== Warning: unimplemented fcntl command: 1036 2024-06-03T08:50:29.282 INFO:tasks.ceph.mon.a.smithi088.stderr:==00:00:15:24.608 34743== Warning: unimplemented fcntl command: 1036 2024-06-03T08:50:29.285 INFO:tasks.ceph.mon.a.smithi088.stderr:==00:00:15:24.611 34743== Warning: unimplemented fcntl command: 1036 2024-06-03T08:50:29.293 INFO:tasks.ceph.mon.b.smithi192.stderr:==00:00:15:24.595 34228== Warning: unimplemented fcntl command: 1036 2024-06-03T08:50:29.299 INFO:tasks.ceph.mon.c.smithi088.stderr:==00:00:15:24.625 34749== Warning: unimplemented fcntl command: 1036 2024-06-03T08:50:29.348 INFO:tasks.ceph.mon.b.smithi192.stderr:==00:00:15:24.651 34228== Warning: unimplemented fcntl command: 1036 2024-06-03T08:50:29.371 INFO:tasks.ceph.mon.c.smithi088.stderr:==00:00:15:24.697 34749== Warning: unimplemented fcntl command: 1036 2024-06-03T08:50:29.376 INFO:tasks.ceph.mon.b.smithi192.stderr:==00:00:15:24.679 34228== Warning: unimplemented fcntl command: 1036 2024-06-03T08:50:29.402 INFO:tasks.ceph.mon.c.smithi088.stderr:==00:00:15:24.728 34749== Warning: unimplemented fcntl command: 1036 2024-06-03T08:50:29.422 INFO:tasks.ceph.mon.a.smithi088.stderr:==00:00:15:24.748 34743== Warning: unimplemented fcntl command: 1036 2024-06-03T08:50:34.214 INFO:tasks.ceph.mon.a.smithi088.stderr:==00:00:15:29.539 34743== Warning: unimplemented fcntl command: 1036 2024-06-03T08:50:34.219 INFO:tasks.ceph.mon.a.smithi088.stderr:==00:00:15:29.545 34743== Warning: unimplemented fcntl command: 1036 2024-06-03T08:50:34.233 INFO:tasks.ceph.mon.a.smithi088.stderr:==00:00:15:29.559 34743== Warning: unimplemented fcntl command: 1036 2024-06-03T08:50:34.264 INFO:tasks.ceph.mon.b.smithi192.stderr:==00:00:15:29.566 34228== Warning: unimplemented fcntl command: 1036 2024-06-03T08:50:34.265 INFO:tasks.ceph.mon.b.smithi192.stderr:==00:00:15:29.568 34228== Warning: unimplemented fcntl command: 1036 2024-06-03T08:50:34.268 INFO:tasks.ceph.mon.c.smithi088.stderr:==00:00:15:29.593 34749== Warning: unimplemented fcntl command: 1036 2024-06-03T08:50:34.269 INFO:tasks.ceph.mon.c.smithi088.stderr:==00:00:15:29.595 34749== Warning: unimplemented fcntl command: 1036 2024-06-03T08:50:34.277 INFO:tasks.ceph.mon.b.smithi192.stderr:==00:00:15:29.580 34228== Warning: unimplemented fcntl command: 1036 2024-06-03T08:50:34.282 INFO:tasks.ceph.mon.c.smithi088.stderr:==00:00:15:29.608 34749== Warning: unimplemented fcntl command: 1036 2024-06-03T08:50:49.320 INFO:tasks.ceph.mon.a.smithi088.stderr:==00:00:15:44.646 34743== Warning: unimplemented fcntl command: 1036 2024-06-03T08:50:49.379 INFO:tasks.ceph.mon.a.smithi088.stderr:==00:00:15:44.704 34743== Warning: unimplemented fcntl command: 1036 2024-06-03T08:50:49.431 INFO:tasks.ceph.mon.b.smithi192.stderr:==00:00:15:44.734 34228== Warning: unimplemented fcntl command: 1036 2024-06-03T08:50:49.434 INFO:tasks.ceph.mon.b.smithi192.stderr:==00:00:15:44.737 34228== Warning: unimplemented fcntl command: 1036 2024-06-03T08:50:49.438 INFO:tasks.ceph.mon.c.smithi088.stderr:==00:00:15:44.764 34749== Warning: unimplemented fcntl command: 1036 2024-06-03T08:50:49.444 INFO:tasks.ceph.mon.c.smithi088.stderr:==00:00:15:44.770 34749== Warning: unimplemented fcntl command: 1036 2024-06-03T08:50:49.447 INFO:tasks.ceph.mon.a.smithi088.stderr:==00:00:15:44.773 34743== Warning: unimplemented fcntl command: 1036 2024-06-03T08:50:49.473 INFO:tasks.ceph.mon.b.smithi192.stderr:==00:00:15:44.775 34228== Warning: unimplemented fcntl command: 1036 2024-06-03T08:50:49.488 INFO:tasks.ceph.mon.c.smithi088.stderr:==00:00:15:44.814 34749== Warning: unimplemented fcntl command: 1036 2024-06-03T08:50:49.944 INFO:tasks.ceph.mon.b.smithi192.stderr:==00:00:15:45.246 34228== Warning: unimplemented fcntl command: 1036 2024-06-03T08:50:49.945 INFO:tasks.ceph.mon.b.smithi192.stderr:==00:00:15:45.247 34228== Warning: unimplemented fcntl command: 1036 2024-06-03T08:50:49.952 INFO:tasks.ceph.mon.b.smithi192.stderr:==00:00:15:45.255 34228== Warning: unimplemented fcntl command: 1036 2024-06-03T08:50:49.981 INFO:tasks.ceph.mon.a.smithi088.stderr:==00:00:15:45.306 34743== Warning: unimplemented fcntl command: 1036 2024-06-03T08:50:49.982 INFO:tasks.ceph.mon.a.smithi088.stderr:==00:00:15:45.308 34743== Warning: unimplemented fcntl command: 1036 2024-06-03T08:50:49.990 INFO:tasks.ceph.mon.a.smithi088.stderr:==00:00:15:45.315 34743== Warning: unimplemented fcntl command: 1036 2024-06-03T08:50:50.013 INFO:tasks.ceph.mon.c.smithi088.stderr:==00:00:15:45.339 34749== Warning: unimplemented fcntl command: 1036 2024-06-03T08:50:50.014 INFO:tasks.ceph.mon.c.smithi088.stderr:==00:00:15:45.340 34749== Warning: unimplemented fcntl command: 1036 2024-06-03T08:50:50.022 INFO:tasks.ceph.mon.c.smithi088.stderr:==00:00:15:45.348 34749== Warning: unimplemented fcntl command: 1036 2024-06-03T08:51:04.408 INFO:tasks.ceph.mon.a.smithi088.stderr:==00:00:15:59.734 34743== Warning: unimplemented fcntl command: 1036 2024-06-03T08:51:04.412 INFO:tasks.ceph.mon.a.smithi088.stderr:==00:00:15:59.738 34743== Warning: unimplemented fcntl command: 1036 2024-06-03T08:51:04.439 INFO:tasks.ceph.mon.a.smithi088.stderr:==00:00:15:59.764 34743== Warning: unimplemented fcntl command: 1036 2024-06-03T08:51:04.457 INFO:tasks.ceph.mon.b.smithi192.stderr:==00:00:15:59.760 34228== Warning: unimplemented fcntl command: 1036 2024-06-03T08:51:04.458 INFO:tasks.ceph.mon.b.smithi192.stderr:==00:00:15:59.761 34228== Warning: unimplemented fcntl command: 1036 2024-06-03T08:51:04.460 INFO:tasks.ceph.mon.c.smithi088.stderr:==00:00:15:59.786 34749== Warning: unimplemented fcntl command: 1036 2024-06-03T08:51:04.461 INFO:tasks.ceph.mon.c.smithi088.stderr:==00:00:15:59.787 34749== Warning: unimplemented fcntl command: 1036 2024-06-03T08:51:04.486 INFO:tasks.ceph.mon.b.smithi192.stderr:==00:00:15:59.788 34228== Warning: unimplemented fcntl command: 1036 2024-06-03T08:51:04.488 INFO:tasks.ceph.mon.c.smithi088.stderr:==00:00:15:59.814 34749== Warning: unimplemented fcntl command: 1036 2024-06-03T08:51:09.484 INFO:tasks.ceph.mon.a.smithi088.stderr:==00:00:16:04.810 34743== Warning: unimplemented fcntl command: 1036 2024-06-03T08:51:09.549 INFO:tasks.ceph.mon.a.smithi088.stderr:==00:00:16:04.874 34743== Warning: unimplemented fcntl command: 1036 2024-06-03T08:51:09.570 INFO:tasks.ceph.mon.a.smithi088.stderr:==00:00:16:04.896 34743== Warning: unimplemented fcntl command: 1036 2024-06-03T08:51:09.596 INFO:tasks.ceph.mon.b.smithi192.stderr:==00:00:16:04.898 34228== Warning: unimplemented fcntl command: 1036 2024-06-03T08:51:09.597 INFO:tasks.ceph.mon.b.smithi192.stderr:==00:00:16:04.900 34228== Warning: unimplemented fcntl command: 1036 2024-06-03T08:51:09.608 INFO:tasks.ceph.mon.c.smithi088.stderr:==00:00:16:04.932 34749== Warning: unimplemented fcntl command: 1036 2024-06-03T08:51:09.608 INFO:tasks.ceph.mon.c.smithi088.stderr:==00:00:16:04.933 34749== Warning: unimplemented fcntl command: 1036 2024-06-03T08:51:09.612 INFO:tasks.ceph.mon.b.smithi192.stderr:==00:00:16:04.915 34228== Warning: unimplemented fcntl command: 1036 2024-06-03T08:51:09.623 INFO:tasks.ceph.mon.c.smithi088.stderr:==00:00:16:04.949 34749== Warning: unimplemented fcntl command: 1036 2024-06-03T08:51:19.463 INFO:tasks.ceph.mon.a.smithi088.stderr:==00:00:16:14.788 34743== Warning: unimplemented fcntl command: 1036 2024-06-03T08:51:19.464 INFO:tasks.ceph.mon.a.smithi088.stderr:==00:00:16:14.790 34743== Warning: unimplemented fcntl command: 1036 2024-06-03T08:51:19.499 INFO:tasks.ceph.mon.a.smithi088.stderr:==00:00:16:14.825 34743== Warning: unimplemented fcntl command: 1036 2024-06-03T08:51:19.516 INFO:tasks.ceph.mon.b.smithi192.stderr:==00:00:16:14.818 34228== Warning: unimplemented fcntl command: 1036 2024-06-03T08:51:19.517 INFO:tasks.ceph.mon.b.smithi192.stderr:==00:00:16:14.820 34228== Warning: unimplemented fcntl command: 1036 2024-06-03T08:51:19.522 INFO:tasks.ceph.mon.c.smithi088.stderr:==00:00:16:14.848 34749== Warning: unimplemented fcntl command: 1036 2024-06-03T08:51:19.527 INFO:tasks.ceph.mon.c.smithi088.stderr:==00:00:16:14.853 34749== Warning: unimplemented fcntl command: 1036 2024-06-03T08:51:19.535 INFO:tasks.ceph.mon.b.smithi192.stderr:==00:00:16:14.837 34228== Warning: unimplemented fcntl command: 1036 2024-06-03T08:51:19.547 INFO:tasks.ceph.mon.c.smithi088.stderr:==00:00:16:14.873 34749== Warning: unimplemented fcntl command: 1036 2024-06-03T08:51:34.511 INFO:tasks.ceph.mon.a.smithi088.stderr:==00:00:16:29.837 34743== Warning: unimplemented fcntl command: 1036 2024-06-03T08:51:34.514 INFO:tasks.ceph.mon.a.smithi088.stderr:==00:00:16:29.840 34743== Warning: unimplemented fcntl command: 1036 2024-06-03T08:51:34.541 INFO:tasks.ceph.mon.a.smithi088.stderr:==00:00:16:29.867 34743== Warning: unimplemented fcntl command: 1036 2024-06-03T08:51:34.561 INFO:tasks.ceph.mon.c.smithi088.stderr:==00:00:16:29.887 34749== Warning: unimplemented fcntl command: 1036 2024-06-03T08:51:34.563 INFO:tasks.ceph.mon.b.smithi192.stderr:==00:00:16:29.866 34228== Warning: unimplemented fcntl command: 1036 2024-06-03T08:51:34.564 INFO:tasks.ceph.mon.b.smithi192.stderr:==00:00:16:29.867 34228== Warning: unimplemented fcntl command: 1036 2024-06-03T08:51:34.567 INFO:tasks.ceph.mon.c.smithi088.stderr:==00:00:16:29.893 34749== Warning: unimplemented fcntl command: 1036 2024-06-03T08:51:34.590 INFO:tasks.ceph.mon.b.smithi192.stderr:==00:00:16:29.893 34228== Warning: unimplemented fcntl command: 1036 2024-06-03T08:51:34.606 INFO:tasks.ceph.mon.c.smithi088.stderr:==00:00:16:29.932 34749== Warning: unimplemented fcntl command: 1036 2024-06-03T08:51:35.400 INFO:tasks.ceph.mon.c.smithi088.stderr:==00:00:16:30.726 34749== Warning: unimplemented fcntl command: 1036 2024-06-03T08:51:35.402 INFO:tasks.ceph.mon.c.smithi088.stderr:==00:00:16:30.728 34749== Warning: unimplemented fcntl command: 1036 2024-06-03T08:51:35.413 INFO:tasks.ceph.mon.c.smithi088.stderr:==00:00:16:30.739 34749== Warning: unimplemented fcntl command: 1036 2024-06-03T08:51:35.443 INFO:tasks.ceph.mon.b.smithi192.stderr:==00:00:16:30.746 34228== Warning: unimplemented fcntl command: 1036 2024-06-03T08:51:35.445 INFO:tasks.ceph.mon.b.smithi192.stderr:==00:00:16:30.748 34228== Warning: unimplemented fcntl command: 1036 2024-06-03T08:51:35.459 INFO:tasks.ceph.mon.b.smithi192.stderr:==00:00:16:30.762 34228== Warning: unimplemented fcntl command: 1036 2024-06-03T08:51:35.540 INFO:tasks.ceph.mon.a.smithi088.stderr:==00:00:16:30.866 34743== Warning: unimplemented fcntl command: 1036 2024-06-03T08:51:35.541 INFO:tasks.ceph.mon.a.smithi088.stderr:==00:00:16:30.867 34743== Warning: unimplemented fcntl command: 1036 2024-06-03T08:51:35.553 INFO:tasks.ceph.mon.a.smithi088.stderr:==00:00:16:30.879 34743== Warning: unimplemented fcntl command: 1036 2024-06-03T08:51:45.088 INFO:tasks.workunit.client.0.smithi088.stdout:ects, waiting a bit for snap trimming before retrying 6 more times... 2024-06-03T08:51:45.088 INFO:tasks.workunit.client.0.smithi088.stdout: api_tier_pp: [ OK ] LibRadosTwoPoolsPP.PromoteSnap (141527 ms) 2024-06-03T08:51:45.088 INFO:tasks.workunit.client.0.smithi088.stdout: api_tier_pp: [ RUN ] LibRadosTwoPoolsPP.PromoteSnapScrub 2024-06-03T08:51:45.088 INFO:tasks.workunit.client.0.smithi088.stdout: api_tier_pp: my_snaps [4] 2024-06-03T08:51:45.088 INFO:tasks.workunit.client.0.smithi088.stdout: api_tier_pp: my_snaps [5,4] 2024-06-03T08:51:45.088 INFO:tasks.workunit.client.0.smithi088.stdout: api_tier_pp: my_snaps [6,5,4] 2024-06-03T08:51:45.089 INFO:tasks.workunit.client.0.smithi088.stdout: api_tier_pp: my_snaps [7,6,5,4] 2024-06-03T08:51:45.089 INFO:tasks.workunit.client.0.smithi088.stdout: api_tier_pp: promoting some heads 2024-06-03T08:51:45.089 INFO:tasks.workunit.client.0.smithi088.stdout: api_tier_pp: promoting from clones for snap 7 2024-06-03T08:51:45.089 INFO:tasks.workunit.client.0.smithi088.stdout: api_tier_pp: promoting from clones for snap 6 2024-06-03T08:51:45.089 INFO:tasks.workunit.client.0.smithi088.stdout: api_tier_pp: promoting from clones for snap 5 2024-06-03T08:51:45.089 INFO:tasks.workunit.client.0.smithi088.stdout: api_tier_pp: promoting from clones for snap 4 2024-06-03T08:51:45.089 INFO:tasks.workunit.client.0.smithi088.stdout: api_tier_pp: waiting for scrubs... 2024-06-03T08:51:45.089 INFO:tasks.workunit.client.0.smithi088.stdout: api_tier_pp: done waiting 2024-06-03T08:51:45.089 INFO:tasks.workunit.client.0.smithi088.stdout: api_tier_pp: [ OK ] LibRadosTwoPoolsPP.PromoteSnapScrub (223987 ms) 2024-06-03T08:51:45.089 INFO:tasks.workunit.client.0.smithi088.stdout: api_tier_pp: [ RUN ] LibRadosTwoPoolsPP.PromoteSnapTrimRace 2024-06-03T08:51:45.089 INFO:tasks.workunit.client.0.smithi088.stdout: api_tier_pp: [ OK ] LibRadosTwoPoolsPP.PromoteSnapTrimRace (10214 ms) 2024-06-03T08:51:45.089 INFO:tasks.workunit.client.0.smithi088.stdout: api_tier_pp: [ RUN ] LibRadosTwoPoolsPP.Whiteout 2024-06-03T08:51:45.089 INFO:tasks.workunit.client.0.smithi088.stdout: api_tier_pp: [ OK ] LibRadosTwoPoolsPP.Whiteout (8558 ms) 2024-06-03T08:51:45.090 INFO:tasks.workunit.client.0.smithi088.stdout: api_tier_pp: [ RUN ] LibRadosTwoPoolsPP.WhiteoutDeleteCreate 2024-06-03T08:51:45.090 INFO:tasks.workunit.client.0.smithi088.stdout: api_tier_pp: [ OK ] LibRadosTwoPoolsPP.WhiteoutDeleteCreate (8419 ms) 2024-06-03T08:51:45.090 INFO:tasks.workunit.client.0.smithi088.stdout: api_tier_pp: [ RUN ] LibRadosTwoPoolsPP.Evict 2024-06-03T08:51:45.090 INFO:tasks.workunit.client.0.smithi088.stdout: api_tier_pp: [ OK ] LibRadosTwoPoolsPP.Evict (10010 ms) 2024-06-03T08:51:45.090 INFO:tasks.workunit.client.0.smithi088.stdout: api_tier_pp: [ RUN ] LibRadosTwoPoolsPP.EvictSnap 2024-06-03T08:51:45.090 INFO:tasks.workunit.client.0.smithi088.stdout: api_tier_pp: [ OK ] LibRadosTwoPoolsPP.EvictSnap (72307 ms) 2024-06-03T08:51:45.090 INFO:tasks.workunit.client.0.smithi088.stdout: api_tier_pp: [ RUN ] LibRadosTwoPoolsPP.EvictSnap2 2024-06-03T08:51:45.090 INFO:tasks.workunit.client.0.smithi088.stdout: api_tier_pp: [ OK ] LibRadosTwoPoolsPP.EvictSnap2 (8916 ms) 2024-06-03T08:51:45.090 INFO:tasks.workunit.client.0.smithi088.stdout: api_tier_pp: [ RUN ] LibRadosTwoPoolsPP.ListSnap 2024-06-03T08:51:45.090 INFO:tasks.workunit.client.0.smithi088.stdout: api_tier_pp: [ OK ] LibRadosTwoPoolsPP.ListSnap (10358 ms) 2024-06-03T08:51:45.090 INFO:tasks.workunit.client.0.smithi088.stdout: api_tier_pp: [ RUN ] LibRadosTwoPoolsPP.EvictSnapRollbackReadRace 2024-06-03T08:51:45.090 INFO:tasks.workunit.client.0.smithi088.stdout: api_tier_pp: [ OK ] LibRadosTwoPoolsPP.EvictSnapRollbackReadRace (37084 ms) 2024-06-03T08:51:45.091 INFO:tasks.workunit.client.0.smithi088.stdout: api_tier_pp: [ RUN ] LibRadosTwoPoolsPP.TryFlush 2024-06-03T08:51:45.091 INFO:tasks.workunit.client.0.smithi088.stdout: api_tier_pp: [ OK ] LibRadosTwoPoolsPP.TryFlush (9016 ms) 2024-06-03T08:51:45.091 INFO:tasks.workunit.client.0.smithi088.stdout: api_tier_pp: [ RUN ] LibRadosTwoPoolsPP.Flush 2024-06-03T08:51:45.091 INFO:tasks.workunit.client.0.smithi088.stdout: api_tier_pp: [ OK ] LibRadosTwoPoolsPP.Flush (11019 ms) 2024-06-03T08:51:45.091 INFO:tasks.workunit.client.0.smithi088.stdout: api_tier_pp: [ RUN ] LibRadosTwoPoolsPP.FlushSnap 2024-06-03T08:51:45.091 INFO:tasks.workunit.client.0.smithi088.stdout: api_tier_pp: [ OK ] LibRadosTwoPoolsPP.FlushSnap (14538 ms) 2024-06-03T08:51:45.091 INFO:tasks.workunit.client.0.smithi088.stdout: api_tier_pp: [ RUN ] LibRadosTwoPoolsPP.FlushTryFlushRaces 2024-06-03T08:51:45.091 INFO:tasks.workunit.client.0.smithi088.stdout: api_tier_pp: [ OK ] LibRadosTwoPoolsPP.FlushTryFlushRaces (8577 ms) 2024-06-03T08:51:45.091 INFO:tasks.workunit.client.0.smithi088.stdout: api_tier_pp: [ RUN ] LibRadosTwoPoolsPP.TryFlushReadRace 2024-06-03T08:51:45.091 INFO:tasks.workunit.client.0.smithi088.stdout: api_tier_pp: [ OK ] LibRadosTwoPoolsPP.TryFlushReadRace (11831 ms) 2024-06-03T08:51:45.091 INFO:tasks.workunit.client.0.smithi088.stdout: api_tier_pp: [ RUN ] LibRadosTwoPoolsPP.HitSetRead 2024-06-03T08:51:45.091 INFO:tasks.workunit.client.0.smithi088.stdout: api_tier_pp: hmm, no HitSet yet 2024-06-03T08:51:45.091 INFO:tasks.workunit.client.0.smithi088.stdout: api_tier_pp: ok, hit_set contains 201:602f83fe:::foo:head 2024-06-03T08:51:45.091 INFO:tasks.workunit.client.0.smithi088.stdout: api_tier_pp: [ OK ] LibRadosTwoPoolsPP.HitSetRead (9821 ms) 2024-06-03T08:51:45.092 INFO:tasks.workunit.client.0.smithi088.stdout: api_tier_pp: [ RUN ] LibRadosTwoPoolsPP.HitSetWrite 2024-06-03T08:51:45.092 INFO:tasks.workunit.client.0.smithi088.stdout: api_tier_pp: pg_num = 32 2024-06-03T08:51:45.092 INFO:tasks.workunit.client.0.smithi088.stdout: api_tier_pp: pg 0 ls 1717404700,0 2024-06-03T08:51:45.092 INFO:tasks.workunit.client.0.smithi088.stdout: api_tier_pp: pg 1 ls 1717404700,0 2024-06-03T08:51:45.092 INFO:tasks.workunit.client.0.smithi088.stdout: api_tier_pp: pg 2 ls 1717404700,0 2024-06-03T08:51:45.092 INFO:tasks.workunit.client.0.smithi088.stdout: api_tier_pp: pg 3 ls 1717404700,0 2024-06-03T08:51:45.092 INFO:tasks.workunit.client.0.smithi088.stdout: api_tier_pp: pg 4 ls 1717404700,0 2024-06-03T08:51:45.092 INFO:tasks.workunit.client.0.smithi088.stdout: api_tier_pp: pg 5 ls 1717404700,0 2024-06-03T08:51:45.092 INFO:tasks.workunit.client.0.smithi088.stdout: api_tier_pp: pg 6 ls 1717404700,0 2024-06-03T08:51:45.092 INFO:tasks.workunit.client.0.smithi088.stdout: api_tier_pp: pg 7 ls 1717404700,0 2024-06-03T08:51:45.092 INFO:tasks.workunit.client.0.smithi088.stdout: api_tier_pp: pg 8 ls 1717404700,0 2024-06-03T08:51:45.092 INFO:tasks.workunit.client.0.smithi088.stdout: api_tier_pp: pg 9 ls 1717404700,0 2024-06-03T08:51:45.092 INFO:tasks.workunit.client.0.smithi088.stdout: api_tier_pp: pg 10 ls 1717404700,0 2024-06-03T08:51:45.093 INFO:tasks.workunit.client.0.smithi088.stdout: api_tier_pp: pg 11 ls 1717404700,0 2024-06-03T08:51:45.093 INFO:tasks.workunit.client.0.smithi088.stdout: api_tier_pp: pg 12 ls 1717404700,0 2024-06-03T08:51:45.093 INFO:tasks.workunit.client.0.smithi088.stdout: api_tier_pp: pg 13 ls 1717404700,0 2024-06-03T08:51:45.093 INFO:tasks.workunit.client.0.smithi088.stdout: api_tier_pp: pg 14 ls 1717404700,0 2024-06-03T08:51:45.093 INFO:tasks.workunit.client.0.smithi088.stdout: api_tier_pp: pg 15 ls 1717404700,0 2024-06-03T08:51:45.093 INFO:tasks.workunit.client.0.smithi088.stdout: api_tier_pp: pg 16 ls 1717404700,0 2024-06-03T08:51:45.283 INFO:tasks.workunit.client.0.smithi088.stdout: api_tier_pp: pg 17 ls 1717404700,0 2024-06-03T08:51:45.283 INFO:tasks.workunit.client.0.smithi088.stdout: api_tier_pp: pg 18 ls 1717404700,0 2024-06-03T08:51:45.283 INFO:tasks.workunit.client.0.smithi088.stdout: api_tier_pp: pg 19 ls 1717404700,0 2024-06-03T08:51:45.284 INFO:tasks.workunit.client.0.smithi088.stdout: api_tier_pp: pg 20 ls 1717404700,0 2024-06-03T08:51:45.284 INFO:tasks.workunit.client.0.smithi088.stdout: api_tier_pp: pg 21 ls 1717404700,0 2024-06-03T08:51:45.284 INFO:tasks.workunit.client.0.smithi088.stdout: api_tier_pp: pg 22 ls 1717404700,0 2024-06-03T08:51:45.284 INFO:tasks.workunit.client.0.smithi088.stdout: api_tier_pp: pg 23 ls 1717404700,0 2024-06-03T08:51:45.284 INFO:tasks.workunit.client.0.smithi088.stdout: api_tier_pp: pg 24 ls 1717404700,0 2024-06-03T08:51:45.284 INFO:tasks.workunit.client.0.smithi088.stdout: api_tier_pp: pg 25 ls 1717404700,0 2024-06-03T08:51:45.284 INFO:tasks.workunit.client.0.smithi088.stdout: api_tier_pp: pg 26 ls 1717404700,0 2024-06-03T08:51:45.284 INFO:tasks.workunit.client.0.smithi088.stdout: api_tier_pp: pg 27 ls 1717404700,0 2024-06-03T08:51:45.284 INFO:tasks.workunit.client.0.smithi088.stdout: api_tier_pp: pg 28 ls 1717404700,0 2024-06-03T08:51:45.284 INFO:tasks.workunit.client.0.smithi088.stdout: api_tier_pp: pg 29 ls 1717404700,0 2024-06-03T08:51:45.284 INFO:tasks.workunit.client.0.smithi088.stdout: api_tier_pp: pg 30 ls 1717404700,0 2024-06-03T08:51:45.284 INFO:tasks.workunit.client.0.smithi088.stdout: api_tier_pp: pg 31 ls 1717404700,0 2024-06-03T08:51:45.285 INFO:tasks.workunit.client.0.smithi088.stdout: api_tier_pp: pg_num = 32 2024-06-03T08:51:45.285 INFO:tasks.workunit.client.0.smithi088.stdout: api_tier_pp: checking for 202:6cac518f:::0:head 2024-06-03T08:51:45.285 INFO:tasks.workunit.client.0.smithi088.stdout: api_tier_pp: checking for 202:02547ec2:::1:head 2024-06-03T08:51:45.285 INFO:tasks.workunit.client.0.smithi088.stdout: api_tier_pp: checking for 202:f905c69b:::2:head 2024-06-03T08:51:45.285 INFO:tasks.workunit.client.0.smithi088.stdout: api_tier_pp: checking for 202:cfc208b3:::3:head 2024-06-03T08:51:45.285 INFO:tasks.workunit.client.0.smithi088.stdout: api_tier_pp: checking for 202:d83876eb:::4:head 2024-06-03T08:51:45.285 INFO:tasks.workunit.client.0.smithi088.stdout: api_tier_pp: checking for 202:b29083e3:::5:head 2024-06-03T08:51:45.285 INFO:tasks.workunit.client.0.smithi088.stdout: api_tier_pp: checking for 202:c4fdafeb:::6:head 2024-06-03T08:51:45.285 INFO:tasks.workunit.client.0.smithi088.stdout: api_tier_pp: checking for 202:5c6b0b28:::7:head 2024-06-03T08:51:45.285 INFO:tasks.workunit.client.0.smithi088.stdout: api_tier_pp: checking for 202:bd63b0f1:::8:head 2024-06-03T08:51:45.285 INFO:tasks.workunit.client.0.smithi088.stdout: api_tier_pp: checking for 202:e960b815:::9:head 2024-06-03T08:51:45.285 INFO:tasks.workunit.client.0.smithi088.stdout: api_tier_pp: checking for 202:52ea6a34:::10:head 2024-06-03T08:51:45.285 INFO:tasks.workunit.client.0.smithi088.stdout: api_tier_pp: checking for 202:89d3ae78:::11:head 2024-06-03T08:51:45.286 INFO:tasks.workunit.client.0.smithi088.stdout: api_tier_pp: checking for 202:de5d7c5f:::12:head 2024-06-03T08:51:45.286 INFO:tasks.workunit.client.0.smithi088.stdout: api_tier_pp: checking for 202:566253c9:::13:head 2024-06-03T08:51:45.286 INFO:tasks.workunit.client.0.smithi088.stdout: api_tier_pp: checking for 202:62a1935d:::14:head 2024-06-03T08:51:45.286 INFO:tasks.workunit.client.0.smithi088.stdout: api_tier_pp: checking for 202:863748b0:::15:head 2024-06-03T08:51:45.286 INFO:tasks.workunit.client.0.smithi088.stdout: api_tier_pp: checking for 202:3958e169:::16:head 2024-06-03T08:51:45.286 INFO:tasks.workunit.client.0.smithi088.stdout: api_tier_pp: checking for 202:4d4dabf9:::17:head 2024-06-03T08:51:45.286 INFO:tasks.workunit.client.0.smithi088.stdout: api_tier_pp: checking for 202:8391935d:::18:head 2024-06-03T08:51:45.286 INFO:tasks.workunit.client.0.smithi088.stdout: api_tier_pp: checking for 202:28883081:::19:head 2024-06-03T08:51:45.286 INFO:tasks.workunit.client.0.smithi088.stdout: api_tier_pp: checking for 202:69259c59:::20:head 2024-06-03T08:51:45.286 INFO:tasks.workunit.client.0.smithi088.stdout: api_tier_pp: checking for 202:4bdb80b7:::21:head 2024-06-03T08:51:45.286 INFO:tasks.workunit.client.0.smithi088.stdout: api_tier_pp: checking for 202:a11c5d71:::22:head 2024-06-03T08:51:45.286 INFO:tasks.workunit.client.0.smithi088.stdout: api_tier_pp: checking for 202:271af37b:::23:head 2024-06-03T08:51:45.287 INFO:tasks.workunit.client.0.smithi088.stdout: api_tier_pp: checking for 202:95b121be:::24:head 2024-06-03T08:51:45.287 INFO:tasks.workunit.client.0.smithi088.stdout: api_tier_pp: checking for 202:58d1031b:::25:head 2024-06-03T08:51:45.287 INFO:tasks.workunit.client.0.smithi088.stdout: api_tier_pp: checking for 202:0a050783:::26:head 2024-06-03T08:51:45.287 INFO:tasks.workunit.client.0.smithi088.stdout: api_tier_pp: checking for 202:c709704c:::27:head 2024-06-03T08:51:45.287 INFO:tasks.workunit.client.0.smithi088.stdout: api_tier_pp: checking for 202:cbe56eaf:::28:head 2024-06-03T08:51:45.287 INFO:tasks.workunit.client.0.smithi088.stdout: api_tier_pp: checking for 202:86b4b162:::29:head 2024-06-03T08:51:45.287 INFO:tasks.workunit.client.0.smithi088.stdout: api_tier_pp: checking for 202:70d89383:::30:head 2024-06-03T08:51:45.287 INFO:tasks.workunit.client.0.smithi088.stdout: api_tier_pp: checking for 202:dd450c7c:::31:head 2024-06-03T08:51:45.287 INFO:tasks.workunit.client.0.smithi088.stdout: api_tier_pp: checking for 202:6d5729b1:::32:head 2024-06-03T08:51:45.287 INFO:tasks.workunit.client.0.smithi088.stdout: api_tier_pp: checking for 202:c388f3fb:::33:head 2024-06-03T08:51:45.287 INFO:tasks.workunit.client.0.smithi088.stdout: api_tier_pp: checking for 202:56cfea31:::34:head 2024-06-03T08:51:45.287 INFO:tasks.workunit.client.0.smithi088.stdout: api_tier_pp: checking for 202:9dbc1bf7:::35:head 2024-06-03T08:51:45.288 INFO:tasks.workunit.client.0.smithi088.stdout: api_tier_pp: checking for 202:40b74ccd:::36:head 2024-06-03T08:51:45.288 INFO:tasks.workunit.client.0.smithi088.stdout: api_tier_pp: checking for 202:4d5aaf42:::37:head 2024-06-03T08:51:45.288 INFO:tasks.workunit.client.0.smithi088.stdout: api_tier_pp: checking for 202:920f362c:::38:head 2024-06-03T08:51:45.288 INFO:tasks.workunit.client.0.smithi088.stdout: api_tier_pp: checking for 202:6cc53222:::39:head 2024-06-03T08:51:45.288 INFO:tasks.workunit.client.0.smithi088.stdout: api_tier_pp: checking for 202:9cad833f:::40:head 2024-06-03T08:51:45.288 INFO:tasks.workunit.client.0.smithi088.stdout: api_tier_pp: checking for 202:1ea84d41:::41:head 2024-06-03T08:51:45.288 INFO:tasks.workunit.client.0.smithi088.stdout: api_tier_pp: checking for 202:c4480ef6:::42:head 2024-06-03T08:51:45.288 INFO:tasks.workunit.client.0.smithi088.stdout: api_tier_pp: checking for 202:a694361e:::43:head 2024-06-03T08:51:45.288 INFO:tasks.workunit.client.0.smithi088.stdout: api_tier_pp: checking for 202:d1bd33e9:::44:head 2024-06-03T08:51:45.288 INFO:tasks.workunit.client.0.smithi088.stdout: api_tier_pp: checking for 202:ddc2cd5d:::45:head 2024-06-03T08:51:45.288 INFO:tasks.workunit.client.0.smithi088.stdout: api_tier_pp: checking for 202:2b782207:::46:head 2024-06-03T08:51:45.288 INFO:tasks.workunit.client.0.smithi088.stdout: api_tier_pp: checking for 202:7b187fca:::47:head 2024-06-03T08:51:45.288 INFO:tasks.workunit.client.0.smithi088.stdout: api_tier_pp: checking for 202:90ecdf6f:::48:head 2024-06-03T08:51:45.289 INFO:tasks.workunit.client.0.smithi088.stdout: api_tier_pp: checking for 202:a5ed95fe:::49:head 2024-06-03T08:51:45.289 INFO:tasks.workunit.client.0.smithi088.stdout: api_tier_pp: checking for 202:ea0eaa55:::50:head 2024-06-03T08:51:45.289 INFO:tasks.workunit.client.0.smithi088.stdout: api_tier_pp: checking for 202:f33ef17b:::51:head 2024-06-03T08:51:45.289 INFO:tasks.workunit.client.0.smithi088.stdout: api_tier_pp: checking for 202:a0d1b2f6:::52:head 2024-06-03T08:51:45.289 INFO:tasks.workunit.client.0.smithi088.stdout: api_tier_pp: checking for 202:60c5229e:::53:head 2024-06-03T08:51:45.289 INFO:tasks.workunit.client.0.smithi088.stdout: api_tier_pp: checking for 202:edcbc575:::54:head 2024-06-03T08:51:45.289 INFO:tasks.workunit.client.0.smithi088.stdout: api_tier_pp: checking for 202:102cf253:::55:head 2024-06-03T08:51:45.289 INFO:tasks.workunit.client.0.smithi088.stdout: api_tier_pp: checking for 202:efb7fb0b:::56:head 2024-06-03T08:51:45.289 INFO:tasks.workunit.client.0.smithi088.stdout: api_tier_pp: checking for 202:50d0a326:::57:head 2024-06-03T08:51:45.289 INFO:tasks.workunit.client.0.smithi088.stdout: api_tier_pp: checking for 202:d4dc5daf:::58:head 2024-06-03T08:51:45.289 INFO:tasks.workunit.client.0.smithi088.stdout: api_tier_pp: checking for 202:3a130462:::59:head 2024-06-03T08:51:45.289 INFO:tasks.workunit.client.0.smithi088.stdout: api_tier_pp: checking for 202:ec87ed71:::60:head 2024-06-03T08:51:45.289 INFO:tasks.workunit.client.0.smithi088.stdout: api_tier_pp: checking for 202:d5bc9454:::61:head 2024-06-03T08:51:45.289 INFO:tasks.workunit.client.0.smithi088.stdout: api_tier_pp: checking for 202:3ddfe313:::62:head 2024-06-03T08:51:45.290 INFO:tasks.workunit.client.0.smithi088.stdout: api_tier_pp: checking for 202:7c2816b9:::63:head 2024-06-03T08:51:45.290 INFO:tasks.workunit.client.0.smithi088.stdout: api_tier_pp: checking for 202:47e00e4d:::64:head 2024-06-03T08:51:45.290 INFO:tasks.workunit.client.0.smithi088.stdout: api_tier_pp: checking for 202:c6410c18:::65:head 2024-06-03T08:51:45.290 INFO:tasks.workunit.client.0.smithi088.stdout: api_tier_pp: checking for 202:b48ed237:::66:head 2024-06-03T08:51:45.290 INFO:tasks.workunit.client.0.smithi088.stdout: api_tier_pp: checking for 202:cd63ad31:::67:head 2024-06-03T08:51:45.290 INFO:tasks.workunit.client.0.smithi088.stdout: api_tier_pp: checking for 202:b179e92b:::68:head 2024-06-03T08:51:45.290 INFO:tasks.workunit.client.0.smithi088.stdout: api_tier_pp: checking for 202:0d9f741a:::69:head 2024-06-03T08:51:45.290 INFO:tasks.workunit.client.0.smithi088.stdout: api_tier_pp: checking for 202:6d3352ae:::70:head 2024-06-03T08:51:45.290 INFO:tasks.workunit.client.0.smithi088.stdout: api_tier_pp: checking for 202:c6d5c19e:::71:head 2024-06-03T08:51:45.290 INFO:tasks.workunit.client.0.smithi088.stdout: api_tier_pp: checking for 202:bc4729c3:::72:head 2024-06-03T08:51:45.290 INFO:tasks.workunit.client.0.smithi088.stdout: api_tier_pp: checking for 202:77e930b9:::73:head 2024-06-03T08:51:45.290 INFO:tasks.workunit.client.0.smithi088.stdout: api_tier_pp: checking for 202:0abeecfd:::74:head 2024-06-03T08:51:45.290 INFO:tasks.workunit.client.0.smithi088.stdout: api_tier_pp: checking for 202:b7c37e15:::75:head 2024-06-03T08:51:45.291 INFO:tasks.workunit.client.0.smithi088.stdout: api_tier_pp: checking for 202:b6378398:::76:head 2024-06-03T08:51:45.291 INFO:tasks.workunit.client.0.smithi088.stdout: api_tier_pp: checking for 202:02bd68de:::77:head 2024-06-03T08:51:45.291 INFO:tasks.workunit.client.0.smithi088.stdout: api_tier_pp: checking for 202:cc795d2d:::78:head 2024-06-03T08:51:45.291 INFO:tasks.workunit.client.0.smithi088.stdout: api_tier_pp: checking for 202:630d4fea:::79:head 2024-06-03T08:51:45.291 INFO:tasks.workunit.client.0.smithi088.stdout: api_tier_pp: checking for 202:e0d29ef5:::80:head 2024-06-03T08:51:45.291 INFO:tasks.workunit.client.0.smithi088.stdout: api_tier_pp: checking for 202:fd6f13d2:::81:head 2024-06-03T08:51:45.291 INFO:tasks.workunit.client.0.smithi088.stdout: api_tier_pp: checking for 202:606461d5:::82:head 2024-06-03T08:51:45.291 INFO:tasks.workunit.client.0.smithi088.stdout: api_tier_pp: checking for 202:eadbdc43:::83:head 2024-06-03T08:51:45.291 INFO:tasks.workunit.client.0.smithi088.stdout: api_tier_pp: checking for 202:8761d0bb:::84:head 2024-06-03T08:51:45.291 INFO:tasks.workunit.client.0.smithi088.stdout: api_tier_pp: checking for 202:9ef0186f:::85:head 2024-06-03T08:51:45.291 INFO:tasks.workunit.client.0.smithi088.stdout: api_tier_pp: checking for 202:e0d41294:::86:head 2024-06-03T08:51:45.291 INFO:tasks.workunit.client.0.smithi088.stdout: api_tier_pp: checking for 202:961de695:::87:head 2024-06-03T08:51:45.291 INFO:tasks.workunit.client.0.smithi088.stdout: api_tier_pp: checking for 202:1423148f:::88:head 2024-06-03T08:51:45.291 INFO:tasks.workunit.client.0.smithi088.stdout: api_tier_pp: checking for 202:633a8fa2:::89:head 2024-06-03T08:51:45.291 INFO:tasks.workunit.client.0.smithi088.stdout: api_tier_pp: checking for 202:a8653809:::90:head 2024-06-03T08:51:45.292 INFO:tasks.workunit.client.0.smithi088.stdout: api_tier_pp: checking for 202:3dac8b33:::91:head 2024-06-03T08:51:45.292 INFO:tasks.workunit.client.0.smithi088.stdout: api_tier_pp: checking for 202:35aad435:::92:head 2024-06-03T08:51:45.292 INFO:tasks.workunit.client.0.smithi088.stdout: api_tier_pp: checking for 202:f6dcc343:::93:head 2024-06-03T08:51:45.292 INFO:tasks.workunit.client.0.smithi088.stdout: api_tier_pp: checking for 202:dbbdad87:::94:head 2024-06-03T08:51:45.292 INFO:tasks.workunit.client.0.smithi088.stdout: api_tier_pp: checking for 202:1cb48ce0:::95:head 2024-06-03T08:51:45.292 INFO:tasks.workunit.client.0.smithi088.stdout: api_tier_pp: checking for 202:03cd461c:::96:head 2024-06-03T08:51:45.292 INFO:tasks.workunit.client.0.smithi088.stdout: api_tier_pp: checking for 202:17a4ea99:::97:head 2024-06-03T08:51:45.292 INFO:tasks.workunit.client.0.smithi088.stdout: api_tier_pp: checking for 202:9993c9a7:::98:head 2024-06-03T08:51:45.292 INFO:tasks.workunit.client.0.smithi088.stdout: api_tier_pp: checking for 202:6394211c:::99:head 2024-06-03T08:51:45.292 INFO:tasks.workunit.client.0.smithi088.stdout: api_tier_pp: checking for 202:94c7ae57:::100:head 2024-06-03T08:51:45.292 INFO:tasks.workunit.client.0.smithi088.stdout: api_tier_pp: checking for 202:6fdee5bb:::101:head 2024-06-03T08:51:45.292 INFO:tasks.workunit.client.0.smithi088.stdout: api_tier_pp: checking for 202:9a477fd1:::102:head 2024-06-03T08:51:45.292 INFO:tasks.workunit.client.0.smithi088.stdout: api_tier_pp: checking for 202:eb850916:::103:head 2024-06-03T08:51:45.292 INFO:tasks.workunit.client.0.smithi088.stdout: api_tier_pp: checking for 202:affc56b9:::104:head 2024-06-03T08:51:45.292 INFO:tasks.workunit.client.0.smithi088.stdout: api_tier_pp: checking for 202:b42dc814:::105:head 2024-06-03T08:51:45.293 INFO:tasks.workunit.client.0.smithi088.stdout: api_tier_pp: checking for 202:f319f8f0:::106:head 2024-06-03T08:51:45.293 INFO:tasks.workunit.client.0.smithi088.stdout: api_tier_pp: checking for 202:9a40b9de:::107:head 2024-06-03T08:51:45.293 INFO:tasks.workunit.client.0.smithi088.stdout: api_tier_pp: checking for 202:8b524f28:::108:head 2024-06-03T08:51:45.293 INFO:tasks.workunit.client.0.smithi088.stdout: api_tier_pp: checking for 202:e3de589f:::109:head 2024-06-03T08:51:45.293 INFO:tasks.workunit.client.0.smithi088.stdout: api_tier_pp: checking for 202:90f90a5b:::110:head 2024-06-03T08:51:45.293 INFO:tasks.workunit.client.0.smithi088.stdout: api_tier_pp: checking for 202:a7b4f1d7:::111:head 2024-06-03T08:51:45.293 INFO:tasks.workunit.client.0.smithi088.stdout: api_tier_pp: checking for 202:af51766e:::112:head 2024-06-03T08:51:45.293 INFO:tasks.workunit.client.0.smithi088.stdout: api_tier_pp: checking for 202:b6f90bd1:::113:head 2024-06-03T08:51:45.293 INFO:tasks.workunit.client.0.smithi088.stdout: api_tier_pp: checking for 202:e0261208:::114:head 2024-06-03T08:51:45.293 INFO:tasks.workunit.client.0.smithi088.stdout: api_tier_pp: checking for 202:c9569ef7:::115:head 2024-06-03T08:51:45.293 INFO:tasks.workunit.client.0.smithi088.stdout: api_tier_pp: checking for 202:61bebe50:::116:head 2024-06-03T08:51:45.293 INFO:tasks.workunit.client.0.smithi088.stdout: api_tier_pp: checking for 202:fe93412b:::117:head 2024-06-03T08:51:45.293 INFO:tasks.workunit.client.0.smithi088.stdout: api_tier_pp: checking for 202:d3d38bee:::118:head 2024-06-03T08:51:45.294 INFO:tasks.workunit.client.0.smithi088.stdout: api_tier_pp: checking for 202:3100ba0c:::119:head 2024-06-03T08:51:45.294 INFO:tasks.workunit.client.0.smithi088.stdout: api_tier_pp: checking for 202:d0560ada:::120:head 2024-06-03T08:51:45.294 INFO:tasks.workunit.client.0.smithi088.stdout: api_tier_pp: checking for 202:f0ea8b35:::121:head 2024-06-03T08:51:45.294 INFO:tasks.workunit.client.0.smithi088.stdout: api_tier_pp: checking for 202:766f231a:::122:head 2024-06-03T08:51:45.294 INFO:tasks.workunit.client.0.smithi088.stdout: api_tier_pp: checking for 202:a07a2582:::123:head 2024-06-03T08:51:45.294 INFO:tasks.workunit.client.0.smithi088.stdout: api_tier_pp: checking for 202:bd7c6b3a:::124:head 2024-06-03T08:51:45.294 INFO:tasks.workunit.client.0.smithi088.stdout: api_tier_pp: checking for 202:fb2ddaff:::125:head 2024-06-03T08:51:45.294 INFO:tasks.workunit.client.0.smithi088.stdout: api_tier_pp: checking for 202:4408e1fe:::126:head 2024-06-03T08:51:45.294 INFO:tasks.workunit.client.0.smithi088.stdout: api_tier_pp: checking for 202:ee1df7a7:::127:head 2024-06-03T08:51:45.294 INFO:tasks.workunit.client.0.smithi088.stdout: api_tier_pp: checking for 202:c3002909:::128:head 2024-06-03T08:51:45.294 INFO:tasks.workunit.client.0.smithi088.stdout: api_tier_pp: checking for 202:4f48ffa9:::129:head 2024-06-03T08:51:45.294 INFO:tasks.workunit.client.0.smithi088.stdout: api_tier_pp: checking for 202:edf38733:::130:head 2024-06-03T08:51:45.294 INFO:tasks.workunit.client.0.smithi088.stdout: api_tier_pp: checking for 202:c08425c0:::131:head 2024-06-03T08:51:45.295 INFO:tasks.workunit.client.0.smithi088.stdout: api_tier_pp: checking for 202:5f902d98:::132:head 2024-06-03T08:51:45.295 INFO:tasks.workunit.client.0.smithi088.stdout: api_tier_pp: checking for 202:41ea2c93:::133:head 2024-06-03T08:51:45.295 INFO:tasks.workunit.client.0.smithi088.stdout: api_tier_pp: checking for 202:813cee13:::134:head 2024-06-03T08:51:45.295 INFO:tasks.workunit.client.0.smithi088.stdout: api_tier_pp: checking for 202:0131818d:::135:head 2024-06-03T08:51:45.295 INFO:tasks.workunit.client.0.smithi088.stdout: api_tier_pp: checking for 202:26ba5a85:::136:head 2024-06-03T08:51:45.295 INFO:tasks.workunit.client.0.smithi088.stdout: api_tier_pp: checking for 202:381b8a5a:::137:head 2024-06-03T08:51:45.295 INFO:tasks.workunit.client.0.smithi088.stdout: api_tier_pp: checking for 202:28797e47:::138:head 2024-06-03T08:51:45.295 INFO:tasks.workunit.client.0.smithi088.stdout: api_tier_pp: checking for 202:bfca7f22:::139:head 2024-06-03T08:51:45.295 INFO:tasks.workunit.client.0.smithi088.stdout: api_tier_pp: checking for 202:36807075:::140:head 2024-06-03T08:51:45.295 INFO:tasks.workunit.client.0.smithi088.stdout: api_tier_pp: checking for 202:80b03975:::141:head 2024-06-03T08:51:45.295 INFO:tasks.workunit.client.0.smithi088.stdout: api_tier_pp: checking for 202:5c15709b:::142:head 2024-06-03T08:51:45.295 INFO:tasks.workunit.client.0.smithi088.stdout: api_tier_pp: checking for 202:f39ea15e:::143:head 2024-06-03T08:51:45.295 INFO:tasks.workunit.client.0.smithi088.stdout: api_tier_pp: checking for 202:ea992956:::144:head 2024-06-03T08:51:45.296 INFO:tasks.workunit.client.0.smithi088.stdout: api_tier_pp: checking for 202:48887b1c:::145:head 2024-06-03T08:51:45.296 INFO:tasks.workunit.client.0.smithi088.stdout: api_tier_pp: checking for 202:9f24a9dd:::146:head 2024-06-03T08:51:45.296 INFO:tasks.workunit.client.0.smithi088.stdout: api_tier_pp: checking for 202:987f100b:::147:head 2024-06-03T08:51:45.296 INFO:tasks.workunit.client.0.smithi088.stdout: api_tier_pp: checking for 202:d2dd3581:::148:head 2024-06-03T08:51:45.296 INFO:tasks.workunit.client.0.smithi088.stdout: api_tier_pp: checking for 202:7fed1808:::149:head 2024-06-03T08:51:45.296 INFO:tasks.workunit.client.0.smithi088.stdout: api_tier_pp: checking for 202:c80b70e9:::150:head 2024-06-03T08:51:45.296 INFO:tasks.workunit.client.0.smithi088.stdout: api_tier_pp: checking for 202:85ed90f9:::151:head 2024-06-03T08:51:45.296 INFO:tasks.workunit.client.0.smithi088.stdout: api_tier_pp: checking for 202:36428b24:::152:head 2024-06-03T08:51:45.296 INFO:tasks.workunit.client.0.smithi088.stdout: api_tier_pp: checking for 202:d044c34a:::153:head 2024-06-03T08:51:45.296 INFO:tasks.workunit.client.0.smithi088.stdout: api_tier_pp: checking for 202:7c18bf58:::154:head 2024-06-03T08:51:45.296 INFO:tasks.workunit.client.0.smithi088.stdout: api_tier_pp: checking for 202:d1c21232:::155:head 2024-06-03T08:51:45.296 INFO:tasks.workunit.client.0.smithi088.stdout: api_tier_pp: checking for 202:a7a3c575:::156:head 2024-06-03T08:51:45.296 INFO:tasks.workunit.client.0.smithi088.stdout: api_tier_pp: checking for 202:87da0633:::157:head 2024-06-03T08:51:45.297 INFO:tasks.workunit.client.0.smithi088.stdout: api_tier_pp: checking for 202:d5ac3822:::158:head 2024-06-03T08:51:45.297 INFO:tasks.workunit.client.0.smithi088.stdout: api_tier_pp: checking for 202:3f20522d:::159:head 2024-06-03T08:51:45.297 INFO:tasks.workunit.client.0.smithi088.stdout: api_tier_pp: checking for 202:6ca26563:::160:head 2024-06-03T08:51:45.297 INFO:tasks.workunit.client.0.smithi088.stdout: api_tier_pp: checking for 202:532ce135:::161:head 2024-06-03T08:51:45.297 INFO:tasks.workunit.client.0.smithi088.stdout: api_tier_pp: checking for 202:c78863e6:::162:head 2024-06-03T08:51:45.297 INFO:tasks.workunit.client.0.smithi088.stdout: api_tier_pp: checking for 202:7462ddf6:::163:head 2024-06-03T08:51:45.297 INFO:tasks.workunit.client.0.smithi088.stdout: api_tier_pp: checking for 202:5d165639:::164:head 2024-06-03T08:51:45.297 INFO:tasks.workunit.client.0.smithi088.stdout: api_tier_pp: checking for 202:f43765fc:::165:head 2024-06-03T08:51:45.297 INFO:tasks.workunit.client.0.smithi088.stdout: api_tier_pp: checking for 202:b4c720e9:::166:head 2024-06-03T08:51:45.297 INFO:tasks.workunit.client.0.smithi088.stdout: api_tier_pp: checking for 202:e694b040:::167:head 2024-06-03T08:51:45.297 INFO:tasks.workunit.client.0.smithi088.stdout: api_tier_pp: checking for 202:afa38db2:::168:head 2024-06-03T08:51:45.297 INFO:tasks.workunit.client.0.smithi088.stdout: api_tier_pp: checking for 202:77ba9f53:::169:head 2024-06-03T08:51:45.298 INFO:tasks.workunit.client.0.smithi088.stdout: api_tier_pp: checking for 202:87495034:::170:head 2024-06-03T08:51:45.298 INFO:tasks.workunit.client.0.smithi088.stdout: api_tier_pp: checking for 202:7c96bf0e:::171:head 2024-06-03T08:51:45.298 INFO:tasks.workunit.client.0.smithi088.stdout: api_tier_pp: checking for 202:dbe346cc:::172:head 2024-06-03T08:51:45.298 INFO:tasks.workunit.client.0.smithi088.stdout: api_tier_pp: checking for 202:e943ec24:::173:head 2024-06-03T08:51:45.298 INFO:tasks.workunit.client.0.smithi088.stdout: api_tier_pp: checking for 202:f97a9c0c:::174:head 2024-06-03T08:51:45.298 INFO:tasks.workunit.client.0.smithi088.stdout: api_tier_pp: checking for 202:6f26e74d:::175:head 2024-06-03T08:51:45.298 INFO:tasks.workunit.client.0.smithi088.stdout: api_tier_pp: checking for 202:4f95e106:::176:head 2024-06-03T08:51:45.298 INFO:tasks.workunit.client.0.smithi088.stdout: api_tier_pp: checking for 202:0e6f2f8f:::177:head 2024-06-03T08:51:45.298 INFO:tasks.workunit.client.0.smithi088.stdout: api_tier_pp: checking for 202:05db05f1:::178:head 2024-06-03T08:51:45.298 INFO:tasks.workunit.client.0.smithi088.stdout: api_tier_pp: checking for 202:38a78d66:::179:head 2024-06-03T08:51:45.298 INFO:tasks.workunit.client.0.smithi088.stdout: api_tier_pp: checking for 202:d095610b:::180:head 2024-06-03T08:51:49.644 INFO:tasks.ceph.mon.a.smithi088.stderr:==00:00:16:44.969 34743== Warning: unimplemented fcntl command: 1036 2024-06-03T08:51:49.693 INFO:tasks.ceph.mon.a.smithi088.stderr:==00:00:16:45.018 34743== Warning: unimplemented fcntl command: 1036 2024-06-03T08:51:49.717 INFO:tasks.ceph.mon.a.smithi088.stderr:==00:00:16:45.043 34743== Warning: unimplemented fcntl command: 1036 2024-06-03T08:51:49.744 INFO:tasks.ceph.mon.b.smithi192.stderr:==00:00:16:45.046 34228== Warning: unimplemented fcntl command: 1036 2024-06-03T08:51:49.745 INFO:tasks.ceph.mon.b.smithi192.stderr:==00:00:16:45.047 34228== Warning: unimplemented fcntl command: 1036 2024-06-03T08:51:49.758 INFO:tasks.ceph.mon.c.smithi088.stderr:==00:00:16:45.084 34749== Warning: unimplemented fcntl command: 1036 2024-06-03T08:51:49.760 INFO:tasks.ceph.mon.c.smithi088.stderr:==00:00:16:45.086 34749== Warning: unimplemented fcntl command: 1036 2024-06-03T08:51:49.767 INFO:tasks.ceph.mon.b.smithi192.stderr:==00:00:16:45.070 34228== Warning: unimplemented fcntl command: 1036 2024-06-03T08:51:49.782 INFO:tasks.ceph.mon.c.smithi088.stderr:==00:00:16:45.108 34749== Warning: unimplemented fcntl command: 1036 2024-06-03T08:51:54.629 INFO:tasks.ceph.mon.a.smithi088.stderr:==00:00:16:49.955 34743== Warning: unimplemented fcntl command: 1036 2024-06-03T08:51:54.693 INFO:tasks.ceph.mon.b.smithi192.stderr:==00:00:16:49.995 34228== Warning: unimplemented fcntl command: 1036 2024-06-03T08:51:54.694 INFO:tasks.ceph.mon.b.smithi192.stderr:==00:00:16:49.997 34228== Warning: unimplemented fcntl command: 1036 2024-06-03T08:51:54.696 INFO:tasks.ceph.mon.a.smithi088.stderr:==00:00:16:50.022 34743== Warning: unimplemented fcntl command: 1036 2024-06-03T08:51:54.711 INFO:tasks.ceph.mon.b.smithi192.stderr:==00:00:16:50.013 34228== Warning: unimplemented fcntl command: 1036 2024-06-03T08:51:54.723 INFO:tasks.ceph.mon.a.smithi088.stderr:==00:00:16:50.049 34743== Warning: unimplemented fcntl command: 1036 2024-06-03T08:51:54.767 INFO:tasks.ceph.mon.c.smithi088.stderr:==00:00:16:50.093 34749== Warning: unimplemented fcntl command: 1036 2024-06-03T08:51:54.768 INFO:tasks.ceph.mon.c.smithi088.stderr:==00:00:16:50.094 34749== Warning: unimplemented fcntl command: 1036 2024-06-03T08:51:54.780 INFO:tasks.ceph.mon.c.smithi088.stderr:==00:00:16:50.106 34749== Warning: unimplemented fcntl command: 1036 2024-06-03T08:52:04.556 INFO:tasks.ceph.mon.a.smithi088.stderr:==00:00:16:59.881 34743== Warning: unimplemented fcntl command: 1036 2024-06-03T08:52:04.604 INFO:tasks.ceph.mon.b.smithi192.stderr:==00:00:16:59.906 34228== Warning: unimplemented fcntl command: 1036 2024-06-03T08:52:04.605 INFO:tasks.ceph.mon.b.smithi192.stderr:==00:00:16:59.908 34228== Warning: unimplemented fcntl command: 1036 2024-06-03T08:52:04.607 INFO:tasks.ceph.mon.a.smithi088.stderr:==00:00:16:59.933 34743== Warning: unimplemented fcntl command: 1036 2024-06-03T08:52:04.613 INFO:tasks.ceph.mon.c.smithi088.stderr:==00:00:16:59.939 34749== Warning: unimplemented fcntl command: 1036 2024-06-03T08:52:04.617 INFO:tasks.ceph.mon.c.smithi088.stderr:==00:00:16:59.943 34749== Warning: unimplemented fcntl command: 1036 2024-06-03T08:52:04.625 INFO:tasks.ceph.mon.b.smithi192.stderr:==00:00:16:59.928 34228== Warning: unimplemented fcntl command: 1036 2024-06-03T08:52:04.631 INFO:tasks.ceph.mon.a.smithi088.stderr:==00:00:16:59.957 34743== Warning: unimplemented fcntl command: 1036 2024-06-03T08:52:04.636 INFO:tasks.ceph.mon.c.smithi088.stderr:==00:00:16:59.962 34749== Warning: unimplemented fcntl command: 1036 2024-06-03T08:52:14.760 INFO:tasks.ceph.mon.a.smithi088.stderr:==00:00:17:10.085 34743== Warning: unimplemented fcntl command: 1036 2024-06-03T08:52:14.763 INFO:tasks.ceph.mon.a.smithi088.stderr:==00:00:17:10.089 34743== Warning: unimplemented fcntl command: 1036 2024-06-03T08:52:14.778 INFO:tasks.ceph.mon.a.smithi088.stderr:==00:00:17:10.104 34743== Warning: unimplemented fcntl command: 1036 2024-06-03T08:52:14.812 INFO:tasks.ceph.mon.c.smithi088.stderr:==00:00:17:10.138 34749== Warning: unimplemented fcntl command: 1036 2024-06-03T08:52:14.816 INFO:tasks.ceph.mon.c.smithi088.stderr:==00:00:17:10.142 34749== Warning: unimplemented fcntl command: 1036 2024-06-03T08:52:14.816 INFO:tasks.ceph.mon.b.smithi192.stderr:==00:00:17:10.119 34228== Warning: unimplemented fcntl command: 1036 2024-06-03T08:52:14.817 INFO:tasks.ceph.mon.b.smithi192.stderr:==00:00:17:10.120 34228== Warning: unimplemented fcntl command: 1036 2024-06-03T08:52:14.830 INFO:tasks.ceph.mon.c.smithi088.stderr:==00:00:17:10.156 34749== Warning: unimplemented fcntl command: 1036 2024-06-03T08:52:14.833 INFO:tasks.ceph.mon.b.smithi192.stderr:==00:00:17:10.135 34228== Warning: unimplemented fcntl command: 1036 2024-06-03T08:52:19.714 INFO:tasks.ceph.mon.a.smithi088.stderr:==00:00:17:15.040 34743== Warning: unimplemented fcntl command: 1036 2024-06-03T08:52:19.716 INFO:tasks.ceph.mon.a.smithi088.stderr:==00:00:17:15.042 34743== Warning: unimplemented fcntl command: 1036 2024-06-03T08:52:19.730 INFO:tasks.ceph.mon.a.smithi088.stderr:==00:00:17:15.056 34743== Warning: unimplemented fcntl command: 1036 2024-06-03T08:52:19.762 INFO:tasks.ceph.mon.b.smithi192.stderr:==00:00:17:15.065 34228== Warning: unimplemented fcntl command: 1036 2024-06-03T08:52:19.763 INFO:tasks.ceph.mon.b.smithi192.stderr:==00:00:17:15.066 34228== Warning: unimplemented fcntl command: 1036 2024-06-03T08:52:19.769 INFO:tasks.ceph.mon.c.smithi088.stderr:==00:00:17:15.095 34749== Warning: unimplemented fcntl command: 1036 2024-06-03T08:52:19.770 INFO:tasks.ceph.mon.c.smithi088.stderr:==00:00:17:15.096 34749== Warning: unimplemented fcntl command: 1036 2024-06-03T08:52:19.776 INFO:tasks.ceph.mon.b.smithi192.stderr:==00:00:17:15.079 34228== Warning: unimplemented fcntl command: 1036 2024-06-03T08:52:19.784 INFO:tasks.ceph.mon.c.smithi088.stderr:==00:00:17:15.110 34749== Warning: unimplemented fcntl command: 1036 2024-06-03T08:52:34.871 INFO:tasks.ceph.mon.a.smithi088.stderr:==00:00:17:30.196 34743== Warning: unimplemented fcntl command: 1036 2024-06-03T08:52:34.873 INFO:tasks.ceph.mon.a.smithi088.stderr:==00:00:17:30.199 34743== Warning: unimplemented fcntl command: 1036 2024-06-03T08:52:34.931 INFO:tasks.ceph.mon.b.smithi192.stderr:==00:00:17:30.234 34228== Warning: unimplemented fcntl command: 1036 2024-06-03T08:52:34.945 INFO:tasks.ceph.mon.b.smithi192.stderr:==00:00:17:30.248 34228== Warning: unimplemented fcntl command: 1036 2024-06-03T08:52:34.946 INFO:tasks.ceph.mon.c.smithi088.stderr:==00:00:17:30.272 34749== Warning: unimplemented fcntl command: 1036 2024-06-03T08:52:34.948 INFO:tasks.ceph.mon.c.smithi088.stderr:==00:00:17:30.274 34749== Warning: unimplemented fcntl command: 1036 2024-06-03T08:52:34.973 INFO:tasks.ceph.mon.b.smithi192.stderr:==00:00:17:30.275 34228== Warning: unimplemented fcntl command: 1036 2024-06-03T08:52:34.980 INFO:tasks.ceph.mon.c.smithi088.stderr:==00:00:17:30.306 34749== Warning: unimplemented fcntl command: 1036 2024-06-03T08:52:34.990 INFO:tasks.ceph.mon.a.smithi088.stderr:==00:00:17:30.316 34743== Warning: unimplemented fcntl command: 1036 2024-06-03T08:52:39.781 INFO:tasks.ceph.mon.a.smithi088.stderr:==00:00:17:35.106 34743== Warning: unimplemented fcntl command: 1036 2024-06-03T08:52:39.783 INFO:tasks.ceph.mon.a.smithi088.stderr:==00:00:17:35.109 34743== Warning: unimplemented fcntl command: 1036 2024-06-03T08:52:39.797 INFO:tasks.ceph.mon.a.smithi088.stderr:==00:00:17:35.123 34743== Warning: unimplemented fcntl command: 1036 2024-06-03T08:52:39.834 INFO:tasks.ceph.mon.c.smithi088.stderr:==00:00:17:35.160 34749== Warning: unimplemented fcntl command: 1036 2024-06-03T08:52:39.834 INFO:tasks.ceph.mon.b.smithi192.stderr:==00:00:17:35.137 34228== Warning: unimplemented fcntl command: 1036 2024-06-03T08:52:39.835 INFO:tasks.ceph.mon.c.smithi088.stderr:==00:00:17:35.161 34749== Warning: unimplemented fcntl command: 1036 2024-06-03T08:52:39.837 INFO:tasks.ceph.mon.b.smithi192.stderr:==00:00:17:35.140 34228== Warning: unimplemented fcntl command: 1036 2024-06-03T08:52:39.848 INFO:tasks.ceph.mon.c.smithi088.stderr:==00:00:17:35.174 34749== Warning: unimplemented fcntl command: 1036 2024-06-03T08:52:39.849 INFO:tasks.ceph.mon.b.smithi192.stderr:==00:00:17:35.152 34228== Warning: unimplemented fcntl command: 1036 2024-06-03T08:52:49.975 INFO:tasks.ceph.mon.a.smithi088.stderr:==00:00:17:45.300 34743== Warning: unimplemented fcntl command: 1036 2024-06-03T08:52:49.977 INFO:tasks.ceph.mon.a.smithi088.stderr:==00:00:17:45.302 34743== Warning: unimplemented fcntl command: 1036 2024-06-03T08:52:49.980 INFO:tasks.ceph.mon.b.smithi192.stderr:==00:00:17:45.282 34228== Warning: unimplemented fcntl command: 1036 2024-06-03T08:52:49.982 INFO:tasks.ceph.mon.b.smithi192.stderr:==00:00:17:45.284 34228== Warning: unimplemented fcntl command: 1036 2024-06-03T08:52:49.998 INFO:tasks.ceph.mon.a.smithi088.stderr:==00:00:17:45.324 34743== Warning: unimplemented fcntl command: 1036 2024-06-03T08:52:50.002 INFO:tasks.ceph.mon.b.smithi192.stderr:==00:00:17:45.305 34228== Warning: unimplemented fcntl command: 1036 2024-06-03T08:52:50.038 INFO:tasks.ceph.mon.c.smithi088.stderr:==00:00:17:45.364 34749== Warning: unimplemented fcntl command: 1036 2024-06-03T08:52:50.039 INFO:tasks.ceph.mon.c.smithi088.stderr:==00:00:17:45.365 34749== Warning: unimplemented fcntl command: 1036 2024-06-03T08:52:50.052 INFO:tasks.ceph.mon.c.smithi088.stderr:==00:00:17:45.378 34749== Warning: unimplemented fcntl command: 1036 2024-06-03T08:52:58.095 INFO:tasks.ceph.mon.c.smithi088.stderr:==00:00:17:53.421 34749== Warning: unimplemented fcntl command: 1036 2024-06-03T08:52:58.100 INFO:tasks.ceph.mon.c.smithi088.stderr:==00:00:17:53.426 34749== Warning: unimplemented fcntl command: 1036 2024-06-03T08:52:58.143 INFO:tasks.ceph.mon.b.smithi192.stderr:==00:00:17:53.446 34228== Warning: unimplemented fcntl command: 1036 2024-06-03T08:52:58.146 INFO:tasks.ceph.mon.a.smithi088.stderr:==00:00:17:53.472 34743== Warning: unimplemented fcntl command: 1036 2024-06-03T08:52:58.150 INFO:tasks.ceph.mon.b.smithi192.stderr:==00:00:17:53.452 34228== Warning: unimplemented fcntl command: 1036 2024-06-03T08:52:58.155 INFO:tasks.ceph.mon.a.smithi088.stderr:==00:00:17:53.481 34743== Warning: unimplemented fcntl command: 1036 2024-06-03T08:52:58.186 INFO:tasks.ceph.mon.a.smithi088.stderr:==00:00:17:53.512 34743== Warning: unimplemented fcntl command: 1036 2024-06-03T08:52:58.189 INFO:tasks.ceph.mon.b.smithi192.stderr:==00:00:17:53.492 34228== Warning: unimplemented fcntl command: 1036 2024-06-03T08:52:58.196 INFO:tasks.ceph.mon.c.smithi088.stderr:==00:00:17:53.522 34749== Warning: unimplemented fcntl command: 1036 2024-06-03T08:52:59.802 INFO:tasks.ceph.mon.a.smithi088.stderr:==00:00:17:55.127 34743== Warning: unimplemented fcntl command: 1036 2024-06-03T08:52:59.848 INFO:tasks.ceph.mon.a.smithi088.stderr:==00:00:17:55.174 34743== Warning: unimplemented fcntl command: 1036 2024-06-03T08:52:59.858 INFO:tasks.ceph.mon.a.smithi088.stderr:==00:00:17:55.184 34743== Warning: unimplemented fcntl command: 1036 2024-06-03T08:52:59.903 INFO:tasks.ceph.mon.b.smithi192.stderr:==00:00:17:55.206 34228== Warning: unimplemented fcntl command: 1036 2024-06-03T08:52:59.904 INFO:tasks.ceph.mon.b.smithi192.stderr:==00:00:17:55.207 34228== Warning: unimplemented fcntl command: 1036 2024-06-03T08:52:59.912 INFO:tasks.ceph.mon.c.smithi088.stderr:==00:00:17:55.238 34749== Warning: unimplemented fcntl command: 1036 2024-06-03T08:52:59.914 INFO:tasks.ceph.mon.c.smithi088.stderr:==00:00:17:55.240 34749== Warning: unimplemented fcntl command: 1036 2024-06-03T08:52:59.914 INFO:tasks.ceph.mon.b.smithi192.stderr:==00:00:17:55.217 34228== Warning: unimplemented fcntl command: 1036 2024-06-03T08:52:59.925 INFO:tasks.ceph.mon.c.smithi088.stderr:==00:00:17:55.251 34749== Warning: unimplemented fcntl command: 1036 2024-06-03T08:53:04.945 INFO:tasks.ceph.mon.a.smithi088.stderr:==00:00:18:00.270 34743== Warning: unimplemented fcntl command: 1036 2024-06-03T08:53:04.994 INFO:tasks.ceph.mon.a.smithi088.stderr:==00:00:18:00.320 34743== Warning: unimplemented fcntl command: 1036 2024-06-03T08:53:05.013 INFO:tasks.ceph.mon.a.smithi088.stderr:==00:00:18:00.339 34743== Warning: unimplemented fcntl command: 1036 2024-06-03T08:53:05.043 INFO:tasks.ceph.mon.b.smithi192.stderr:==00:00:18:00.346 34228== Warning: unimplemented fcntl command: 1036 2024-06-03T08:53:05.045 INFO:tasks.ceph.mon.b.smithi192.stderr:==00:00:18:00.348 34228== Warning: unimplemented fcntl command: 1036 2024-06-03T08:53:05.048 INFO:tasks.ceph.mon.c.smithi088.stderr:==00:00:18:00.374 34749== Warning: unimplemented fcntl command: 1036 2024-06-03T08:53:05.050 INFO:tasks.ceph.mon.c.smithi088.stderr:==00:00:18:00.376 34749== Warning: unimplemented fcntl command: 1036 2024-06-03T08:53:05.058 INFO:tasks.ceph.mon.b.smithi192.stderr:==00:00:18:00.360 34228== Warning: unimplemented fcntl command: 1036 2024-06-03T08:53:05.063 INFO:tasks.ceph.mon.c.smithi088.stderr:==00:00:18:00.389 34749== Warning: unimplemented fcntl command: 1036 2024-06-03T08:53:19.992 INFO:tasks.ceph.mon.a.smithi088.stderr:==00:00:18:15.317 34743== Warning: unimplemented fcntl command: 1036 2024-06-03T08:53:20.000 INFO:tasks.ceph.mon.a.smithi088.stderr:==00:00:18:15.325 34743== Warning: unimplemented fcntl command: 1036 2024-06-03T08:53:20.000 INFO:tasks.ceph.mon.b.smithi192.stderr:==00:00:18:15.303 34228== Warning: unimplemented fcntl command: 1036 2024-06-03T08:53:20.002 INFO:tasks.ceph.mon.b.smithi192.stderr:==00:00:18:15.304 34228== Warning: unimplemented fcntl command: 1036 2024-06-03T08:53:20.005 INFO:tasks.ceph.mon.c.smithi088.stderr:==00:00:18:15.331 34749== Warning: unimplemented fcntl command: 1036 2024-06-03T08:53:20.008 INFO:tasks.ceph.mon.c.smithi088.stderr:==00:00:18:15.334 34749== Warning: unimplemented fcntl command: 1036 2024-06-03T08:53:20.117 INFO:tasks.ceph.mon.c.smithi088.stderr:==00:00:18:15.442 34749== Warning: unimplemented fcntl command: 1036 2024-06-03T08:53:20.118 INFO:tasks.ceph.mon.b.smithi192.stderr:==00:00:18:15.421 34228== Warning: unimplemented fcntl command: 1036 2024-06-03T08:53:20.131 INFO:tasks.ceph.mon.a.smithi088.stderr:==00:00:18:15.457 34743== Warning: unimplemented fcntl command: 1036 2024-06-03T08:53:24.937 INFO:tasks.ceph.mon.a.smithi088.stderr:==00:00:18:20.263 34743== Warning: unimplemented fcntl command: 1036 2024-06-03T08:53:24.940 INFO:tasks.ceph.mon.a.smithi088.stderr:==00:00:18:20.266 34743== Warning: unimplemented fcntl command: 1036 2024-06-03T08:53:24.960 INFO:tasks.ceph.mon.a.smithi088.stderr:==00:00:18:20.286 34743== Warning: unimplemented fcntl command: 1036 2024-06-03T08:53:24.988 INFO:tasks.ceph.mon.b.smithi192.stderr:==00:00:18:20.290 34228== Warning: unimplemented fcntl command: 1036 2024-06-03T08:53:24.991 INFO:tasks.ceph.mon.b.smithi192.stderr:==00:00:18:20.294 34228== Warning: unimplemented fcntl command: 1036 2024-06-03T08:53:24.993 INFO:tasks.ceph.mon.c.smithi088.stderr:==00:00:18:20.319 34749== Warning: unimplemented fcntl command: 1036 2024-06-03T08:53:24.996 INFO:tasks.ceph.mon.c.smithi088.stderr:==00:00:18:20.322 34749== Warning: unimplemented fcntl command: 1036 2024-06-03T08:53:25.005 INFO:tasks.ceph.mon.b.smithi192.stderr:==00:00:18:20.307 34228== Warning: unimplemented fcntl command: 1036 2024-06-03T08:53:25.015 INFO:tasks.ceph.mon.c.smithi088.stderr:==00:00:18:20.341 34749== Warning: unimplemented fcntl command: 1036 2024-06-03T08:53:35.059 INFO:tasks.ceph.mon.a.smithi088.stderr:==00:00:18:30.384 34743== Warning: unimplemented fcntl command: 1036 2024-06-03T08:53:35.107 INFO:tasks.ceph.mon.a.smithi088.stderr:==00:00:18:30.433 34743== Warning: unimplemented fcntl command: 1036 2024-06-03T08:53:35.131 INFO:tasks.ceph.mon.a.smithi088.stderr:==00:00:18:30.456 34743== Warning: unimplemented fcntl command: 1036 2024-06-03T08:53:35.157 INFO:tasks.ceph.mon.b.smithi192.stderr:==00:00:18:30.460 34228== Warning: unimplemented fcntl command: 1036 2024-06-03T08:53:35.158 INFO:tasks.ceph.mon.c.smithi088.stderr:==00:00:18:30.484 34749== Warning: unimplemented fcntl command: 1036 2024-06-03T08:53:35.160 INFO:tasks.ceph.mon.c.smithi088.stderr:==00:00:18:30.486 34749== Warning: unimplemented fcntl command: 1036 2024-06-03T08:53:35.161 INFO:tasks.ceph.mon.b.smithi192.stderr:==00:00:18:30.463 34228== Warning: unimplemented fcntl command: 1036 2024-06-03T08:53:35.183 INFO:tasks.ceph.mon.c.smithi088.stderr:==00:00:18:30.509 34749== Warning: unimplemented fcntl command: 1036 2024-06-03T08:53:35.186 INFO:tasks.ceph.mon.b.smithi192.stderr:==00:00:18:30.488 34228== Warning: unimplemented fcntl command: 1036 2024-06-03T08:53:36.806 INFO:tasks.workunit.client.0.smithi088.stdout: api_tier_pp: checking for 202:a1a9d709:::181:head 2024-06-03T08:53:36.806 INFO:tasks.workunit.client.0.smithi088.stdout: api_tier_pp: checking for 202:1e5d39db:::182:head 2024-06-03T08:53:36.806 INFO:tasks.workunit.client.0.smithi088.stdout: api_tier_pp: checking for 202:f7df4fb9:::183:head 2024-06-03T08:53:36.807 INFO:tasks.workunit.client.0.smithi088.stdout: api_tier_pp: checking for 202:03a7f161:::184:head 2024-06-03T08:53:36.807 INFO:tasks.workunit.client.0.smithi088.stdout: api_tier_pp: checking for 202:ba70721e:::185:head 2024-06-03T08:53:36.807 INFO:tasks.workunit.client.0.smithi088.stdout: api_tier_pp: checking for 202:28e5662d:::186:head 2024-06-03T08:53:36.807 INFO:tasks.workunit.client.0.smithi088.stdout: api_tier_pp: checking for 202:973d52de:::187:head 2024-06-03T08:53:36.807 INFO:tasks.workunit.client.0.smithi088.stdout: api_tier_pp: checking for 202:4303eb1c:::188:head 2024-06-03T08:53:36.807 INFO:tasks.workunit.client.0.smithi088.stdout: api_tier_pp: checking for 202:b990b48e:::189:head 2024-06-03T08:53:36.807 INFO:tasks.workunit.client.0.smithi088.stdout: api_tier_pp: checking for 202:29b8165b:::190:head 2024-06-03T08:53:36.807 INFO:tasks.workunit.client.0.smithi088.stdout: api_tier_pp: checking for 202:3547f197:::191:head 2024-06-03T08:53:36.807 INFO:tasks.workunit.client.0.smithi088.stdout: api_tier_pp: checking for 202:7e260936:::192:head 2024-06-03T08:53:36.807 INFO:tasks.workunit.client.0.smithi088.stdout: api_tier_pp: checking for 202:1abec7b1:::193:head 2024-06-03T08:53:36.807 INFO:tasks.workunit.client.0.smithi088.stdout: api_tier_pp: checking for 202:10fdda93:::194:head 2024-06-03T08:53:36.808 INFO:tasks.workunit.client.0.smithi088.stdout: api_tier_pp: checking for 202:15817eea:::195:head 2024-06-03T08:53:36.808 INFO:tasks.workunit.client.0.smithi088.stdout: api_tier_pp: checking for 202:770bab57:::196:head 2024-06-03T08:53:36.808 INFO:tasks.workunit.client.0.smithi088.stdout: api_tier_pp: checking for 202:ed9e13e7:::197:head 2024-06-03T08:53:36.808 INFO:tasks.workunit.client.0.smithi088.stdout: api_tier_pp: checking for 202:71471a8f:::198:head 2024-06-03T08:53:36.808 INFO:tasks.workunit.client.0.smithi088.stdout: api_tier_pp: checking for 202:10fb1d02:::199:head 2024-06-03T08:53:36.808 INFO:tasks.workunit.client.0.smithi088.stdout: api_tier_pp: [ OK ] LibRadosTwoPoolsPP.HitSetWrite (20657 ms) 2024-06-03T08:53:36.808 INFO:tasks.workunit.client.0.smithi088.stdout: api_tier_pp: [ RUN ] LibRadosTwoPoolsPP.HitSetTrim 2024-06-03T08:53:36.808 INFO:tasks.workunit.client.0.smithi088.stdout: api_tier_pp: got ls 1717404721,0 2024-06-03T08:53:36.808 INFO:tasks.workunit.client.0.smithi088.stdout: api_tier_pp: first is 1717404721 2024-06-03T08:53:36.808 INFO:tasks.workunit.client.0.smithi088.stdout: api_tier_pp: got ls 1717404721,0 2024-06-03T08:53:36.808 INFO:tasks.workunit.client.0.smithi088.stdout: api_tier_pp: got ls 1717404721,0 2024-06-03T08:53:36.808 INFO:tasks.workunit.client.0.smithi088.stdout: api_tier_pp: got ls 1717404721,1717404724,1717404725,0 2024-06-03T08:53:36.809 INFO:tasks.workunit.client.0.smithi088.stdout: api_tier_pp: got ls 1717404721,1717404724,1717404725,0 2024-06-03T08:53:36.809 INFO:tasks.workunit.client.0.smithi088.stdout: api_tier_pp: got ls 1717404721,1717404724,1717404725,0 2024-06-03T08:53:36.809 INFO:tasks.workunit.client.0.smithi088.stdout: api_tier_pp: got ls 1717404721,1717404724,1717404725,1717404727,1717404728,0 2024-06-03T08:53:36.809 INFO:tasks.workunit.client.0.smithi088.stdout: api_tier_pp: got ls 1717404721,1717404724,1717404725,1717404727,1717404728,0 2024-06-03T08:53:36.809 INFO:tasks.workunit.client.0.smithi088.stdout: api_tier_pp: got ls 1717404721,1717404724,1717404725,1717404727,1717404728,0 2024-06-03T08:53:36.809 INFO:tasks.workunit.client.0.smithi088.stdout: api_tier_pp: got ls 1717404721,1717404724,1717404725,1717404727,1717404728,1717404730,1717404731,0 2024-06-03T08:53:36.809 INFO:tasks.workunit.client.0.smithi088.stdout: api_tier_pp: got ls 1717404721,1717404724,1717404725,1717404727,1717404728,1717404730,1717404731,0 2024-06-03T08:53:36.809 INFO:tasks.workunit.client.0.smithi088.stdout: api_tier_pp: got ls 1717404721,1717404724,1717404725,1717404727,1717404728,1717404730,1717404731,0 2024-06-03T08:53:36.809 INFO:tasks.workunit.client.0.smithi088.stdout: api_tier_pp: got ls 1717404725,1717404727,1717404728,1717404730,1717404731,1717404733,1717404734,0 2024-06-03T08:53:36.809 INFO:tasks.workunit.client.0.smithi088.stdout: api_tier_pp: first now 1717404725, trimmed 2024-06-03T08:53:36.809 INFO:tasks.workunit.client.0.smithi088.stdout: api_tier_pp: [ OK ] LibRadosTwoPoolsPP.HitSetTrim (22932 ms) 2024-06-03T08:53:36.809 INFO:tasks.workunit.client.0.smithi088.stdout: api_tier_pp: [ RUN ] LibRadosTwoPoolsPP.PromoteOn2ndRead 2024-06-03T08:53:36.810 INFO:tasks.workunit.client.0.smithi088.stdout: api_tier_pp: foo0 2024-06-03T08:53:36.810 INFO:tasks.workunit.client.0.smithi088.stdout: api_tier_pp: verifying foo0 is eventually promoted 2024-06-03T08:53:36.810 INFO:tasks.workunit.client.0.smithi088.stdout: api_tier_pp: [ OK ] LibRadosTwoPoolsPP.PromoteOn2ndRead (15172 ms) 2024-06-03T08:53:36.810 INFO:tasks.workunit.client.0.smithi088.stdout: api_tier_pp: [ RUN ] LibRadosTwoPoolsPP.ProxyRead 2024-06-03T08:53:36.810 INFO:tasks.workunit.client.0.smithi088.stdout: api_tier_pp: [ OK ] LibRadosTwoPoolsPP.ProxyRead (20088 ms) 2024-06-03T08:53:36.810 INFO:tasks.workunit.client.0.smithi088.stdout: api_tier_pp: [ RUN ] LibRadosTwoPoolsPP.CachePin 2024-06-03T08:53:36.810 INFO:tasks.workunit.client.0.smithi088.stdout: api_tier_pp: [ OK ] LibRadosTwoPoolsPP.CachePin (23863 ms) 2024-06-03T08:53:36.810 INFO:tasks.workunit.client.0.smithi088.stdout: api_tier_pp: [ RUN ] LibRadosTwoPoolsPP.SetRedirectRead 2024-06-03T08:53:36.810 INFO:tasks.workunit.client.0.smithi088.stdout: api_tier_pp: [ OK ] LibRadosTwoPoolsPP.SetRedirectRead (4287 ms) 2024-06-03T08:53:36.810 INFO:tasks.workunit.client.0.smithi088.stdout: api_tier_pp: [ RUN ] LibRadosTwoPoolsPP.ManifestPromoteRead 2024-06-03T08:53:36.810 INFO:tasks.workunit.client.0.smithi088.stdout: api_tier_pp: require_osd_release = reef 2024-06-03T08:53:36.810 INFO:tasks.workunit.client.0.smithi088.stdout: api_tier_pp: [ OK ] LibRadosTwoPoolsPP.ManifestPromoteRead (3587 ms) 2024-06-03T08:53:36.811 INFO:tasks.workunit.client.0.smithi088.stdout: api_tier_pp: [ RUN ] LibRadosTwoPoolsPP.ManifestRefRead 2024-06-03T08:53:36.811 INFO:tasks.workunit.client.0.smithi088.stdout: api_tier_pp: [ OK ] LibRadosTwoPoolsPP.ManifestRefRead (4554 ms) 2024-06-03T08:53:36.811 INFO:tasks.workunit.client.0.smithi088.stdout: api_tier_pp: [ RUN ] LibRadosTwoPoolsPP.ManifestUnset 2024-06-03T08:53:36.811 INFO:tasks.workunit.client.0.smithi088.stdout: api_tier_pp: require_osd_release = reef 2024-06-03T08:53:36.811 INFO:tasks.workunit.client.0.smithi088.stdout: api_tier_pp: [ OK ] LibRadosTwoPoolsPP.ManifestUnset (4412 ms) 2024-06-03T08:53:36.811 INFO:tasks.workunit.client.0.smithi088.stdout: api_tier_pp: [ RUN ] LibRadosTwoPoolsPP.ManifestDedupRefRead 2024-06-03T08:53:36.811 INFO:tasks.workunit.client.0.smithi088.stdout: api_tier_pp: require_osd_release = reef 2024-06-03T08:53:45.046 INFO:tasks.ceph.mon.a.smithi088.stderr:==00:00:18:40.371 34743== Warning: unimplemented fcntl command: 1036 2024-06-03T08:53:45.049 INFO:tasks.ceph.mon.a.smithi088.stderr:==00:00:18:40.375 34743== Warning: unimplemented fcntl command: 1036 2024-06-03T08:53:45.068 INFO:tasks.ceph.mon.a.smithi088.stderr:==00:00:18:40.394 34743== Warning: unimplemented fcntl command: 1036 2024-06-03T08:53:45.097 INFO:tasks.ceph.mon.b.smithi192.stderr:==00:00:18:40.399 34228== Warning: unimplemented fcntl command: 1036 2024-06-03T08:53:45.099 INFO:tasks.ceph.mon.c.smithi088.stderr:==00:00:18:40.425 34749== Warning: unimplemented fcntl command: 1036 2024-06-03T08:53:45.100 INFO:tasks.ceph.mon.b.smithi192.stderr:==00:00:18:40.402 34228== Warning: unimplemented fcntl command: 1036 2024-06-03T08:53:45.100 INFO:tasks.ceph.mon.c.smithi088.stderr:==00:00:18:40.426 34749== Warning: unimplemented fcntl command: 1036 2024-06-03T08:53:45.119 INFO:tasks.ceph.mon.b.smithi192.stderr:==00:00:18:40.421 34228== Warning: unimplemented fcntl command: 1036 2024-06-03T08:53:45.121 INFO:tasks.ceph.mon.c.smithi088.stderr:==00:00:18:40.447 34749== Warning: unimplemented fcntl command: 1036 2024-06-03T08:53:50.022 INFO:tasks.ceph.mon.a.smithi088.stderr:==00:00:18:45.347 34743== Warning: unimplemented fcntl command: 1036 2024-06-03T08:53:50.070 INFO:tasks.ceph.mon.a.smithi088.stderr:==00:00:18:45.396 34743== Warning: unimplemented fcntl command: 1036 2024-06-03T08:53:50.072 INFO:tasks.ceph.mon.b.smithi192.stderr:==00:00:18:45.375 34228== Warning: unimplemented fcntl command: 1036 2024-06-03T08:53:50.075 INFO:tasks.ceph.mon.b.smithi192.stderr:==00:00:18:45.378 34228== Warning: unimplemented fcntl command: 1036 2024-06-03T08:53:50.084 INFO:tasks.ceph.mon.a.smithi088.stderr:==00:00:18:45.410 34743== Warning: unimplemented fcntl command: 1036 2024-06-03T08:53:50.088 INFO:tasks.ceph.mon.b.smithi192.stderr:==00:00:18:45.391 34228== Warning: unimplemented fcntl command: 1036 2024-06-03T08:53:50.120 INFO:tasks.ceph.mon.c.smithi088.stderr:==00:00:18:45.446 34749== Warning: unimplemented fcntl command: 1036 2024-06-03T08:53:50.122 INFO:tasks.ceph.mon.c.smithi088.stderr:==00:00:18:45.448 34749== Warning: unimplemented fcntl command: 1036 2024-06-03T08:53:50.134 INFO:tasks.ceph.mon.c.smithi088.stderr:==00:00:18:45.459 34749== Warning: unimplemented fcntl command: 1036 2024-06-03T08:54:05.248 INFO:tasks.ceph.mon.a.smithi088.stderr:==00:00:19:00.573 34743== Warning: unimplemented fcntl command: 1036 2024-06-03T08:54:05.250 INFO:tasks.ceph.mon.a.smithi088.stderr:==00:00:19:00.576 34743== Warning: unimplemented fcntl command: 1036 2024-06-03T08:54:05.269 INFO:tasks.ceph.mon.a.smithi088.stderr:==00:00:19:00.595 34743== Warning: unimplemented fcntl command: 1036 2024-06-03T08:54:05.297 INFO:tasks.ceph.mon.b.smithi192.stderr:==00:00:19:00.600 34228== Warning: unimplemented fcntl command: 1036 2024-06-03T08:54:05.299 INFO:tasks.ceph.mon.b.smithi192.stderr:==00:00:19:00.602 34228== Warning: unimplemented fcntl command: 1036 2024-06-03T08:54:05.319 INFO:tasks.ceph.mon.b.smithi192.stderr:==00:00:19:00.621 34228== Warning: unimplemented fcntl command: 1036 2024-06-03T08:54:05.319 INFO:tasks.ceph.mon.c.smithi088.stderr:==00:00:19:00.645 34749== Warning: unimplemented fcntl command: 1036 2024-06-03T08:54:05.323 INFO:tasks.ceph.mon.c.smithi088.stderr:==00:00:19:00.649 34749== Warning: unimplemented fcntl command: 1036 2024-06-03T08:54:05.344 INFO:tasks.ceph.mon.c.smithi088.stderr:==00:00:19:00.670 34749== Warning: unimplemented fcntl command: 1036 2024-06-03T08:54:10.134 INFO:tasks.ceph.mon.a.smithi088.stderr:==00:00:19:05.460 34743== Warning: unimplemented fcntl command: 1036 2024-06-03T08:54:10.137 INFO:tasks.ceph.mon.a.smithi088.stderr:==00:00:19:05.463 34743== Warning: unimplemented fcntl command: 1036 2024-06-03T08:54:10.148 INFO:tasks.ceph.mon.a.smithi088.stderr:==00:00:19:05.474 34743== Warning: unimplemented fcntl command: 1036 2024-06-03T08:54:10.185 INFO:tasks.ceph.mon.c.smithi088.stderr:==00:00:19:05.511 34749== Warning: unimplemented fcntl command: 1036 2024-06-03T08:54:10.187 INFO:tasks.ceph.mon.b.smithi192.stderr:==00:00:19:05.489 34228== Warning: unimplemented fcntl command: 1036 2024-06-03T08:54:10.188 INFO:tasks.ceph.mon.b.smithi192.stderr:==00:00:19:05.491 34228== Warning: unimplemented fcntl command: 1036 2024-06-03T08:54:10.190 INFO:tasks.ceph.mon.c.smithi088.stderr:==00:00:19:05.516 34749== Warning: unimplemented fcntl command: 1036 2024-06-03T08:54:10.199 INFO:tasks.ceph.mon.b.smithi192.stderr:==00:00:19:05.502 34228== Warning: unimplemented fcntl command: 1036 2024-06-03T08:54:10.201 INFO:tasks.ceph.mon.c.smithi088.stderr:==00:00:19:05.527 34749== Warning: unimplemented fcntl command: 1036 2024-06-03T08:54:25.200 INFO:tasks.ceph.mon.a.smithi088.stderr:==00:00:19:20.525 34743== Warning: unimplemented fcntl command: 1036 2024-06-03T08:54:25.202 INFO:tasks.ceph.mon.a.smithi088.stderr:==00:00:19:20.528 34743== Warning: unimplemented fcntl command: 1036 2024-06-03T08:54:25.227 INFO:tasks.ceph.mon.a.smithi088.stderr:==00:00:19:20.553 34743== Warning: unimplemented fcntl command: 1036 2024-06-03T08:54:25.252 INFO:tasks.ceph.mon.b.smithi192.stderr:==00:00:19:20.554 34228== Warning: unimplemented fcntl command: 1036 2024-06-03T08:54:25.253 INFO:tasks.ceph.mon.b.smithi192.stderr:==00:00:19:20.556 34228== Warning: unimplemented fcntl command: 1036 2024-06-03T08:54:25.258 INFO:tasks.ceph.mon.c.smithi088.stderr:==00:00:19:20.581 34749== Warning: unimplemented fcntl command: 1036 2024-06-03T08:54:25.258 INFO:tasks.ceph.mon.c.smithi088.stderr:==00:00:19:20.582 34749== Warning: unimplemented fcntl command: 1036 2024-06-03T08:54:25.274 INFO:tasks.ceph.mon.b.smithi192.stderr:==00:00:19:20.577 34228== Warning: unimplemented fcntl command: 1036 2024-06-03T08:54:25.279 INFO:tasks.ceph.mon.c.smithi088.stderr:==00:00:19:20.605 34749== Warning: unimplemented fcntl command: 1036 2024-06-03T08:54:25.880 INFO:tasks.ceph.mon.a.smithi088.stderr:==00:00:19:21.206 34743== Warning: unimplemented fcntl command: 1036 2024-06-03T08:54:25.881 INFO:tasks.ceph.mon.a.smithi088.stderr:==00:00:19:21.207 34743== Warning: unimplemented fcntl command: 1036 2024-06-03T08:54:25.889 INFO:tasks.ceph.mon.b.smithi192.stderr:==00:00:19:21.192 34228== Warning: unimplemented fcntl command: 1036 2024-06-03T08:54:25.891 INFO:tasks.ceph.mon.b.smithi192.stderr:==00:00:19:21.193 34228== Warning: unimplemented fcntl command: 1036 2024-06-03T08:54:25.895 INFO:tasks.ceph.mon.a.smithi088.stderr:==00:00:19:21.221 34743== Warning: unimplemented fcntl command: 1036 2024-06-03T08:54:25.903 INFO:tasks.ceph.mon.b.smithi192.stderr:==00:00:19:21.205 34228== Warning: unimplemented fcntl command: 1036 2024-06-03T08:54:25.929 INFO:tasks.ceph.mon.c.smithi088.stderr:==00:00:19:21.254 34749== Warning: unimplemented fcntl command: 1036 2024-06-03T08:54:25.930 INFO:tasks.ceph.mon.c.smithi088.stderr:==00:00:19:21.256 34749== Warning: unimplemented fcntl command: 1036 2024-06-03T08:54:25.940 INFO:tasks.ceph.mon.c.smithi088.stderr:==00:00:19:21.266 34749== Warning: unimplemented fcntl command: 1036 2024-06-03T08:54:45.384 INFO:tasks.ceph.mon.a.smithi088.stderr:==00:00:19:40.710 34743== Warning: unimplemented fcntl command: 1036 2024-06-03T08:54:45.386 INFO:tasks.ceph.mon.a.smithi088.stderr:==00:00:19:40.712 34743== Warning: unimplemented fcntl command: 1036 2024-06-03T08:54:45.426 INFO:tasks.ceph.mon.a.smithi088.stderr:==00:00:19:40.752 34743== Warning: unimplemented fcntl command: 1036 2024-06-03T08:54:45.433 INFO:tasks.ceph.mon.b.smithi192.stderr:==00:00:19:40.736 34228== Warning: unimplemented fcntl command: 1036 2024-06-03T08:54:45.435 INFO:tasks.ceph.mon.b.smithi192.stderr:==00:00:19:40.737 34228== Warning: unimplemented fcntl command: 1036 2024-06-03T08:54:45.441 INFO:tasks.ceph.mon.c.smithi088.stderr:==00:00:19:40.767 34749== Warning: unimplemented fcntl command: 1036 2024-06-03T08:54:45.443 INFO:tasks.ceph.mon.c.smithi088.stderr:==00:00:19:40.769 34749== Warning: unimplemented fcntl command: 1036 2024-06-03T08:54:45.461 INFO:tasks.ceph.mon.b.smithi192.stderr:==00:00:19:40.763 34228== Warning: unimplemented fcntl command: 1036 2024-06-03T08:54:45.469 INFO:tasks.ceph.mon.c.smithi088.stderr:==00:00:19:40.795 34749== Warning: unimplemented fcntl command: 1036 2024-06-03T08:54:50.278 INFO:tasks.ceph.mon.a.smithi088.stderr:==00:00:19:45.604 34743== Warning: unimplemented fcntl command: 1036 2024-06-03T08:54:50.282 INFO:tasks.ceph.mon.a.smithi088.stderr:==00:00:19:45.608 34743== Warning: unimplemented fcntl command: 1036 2024-06-03T08:54:50.295 INFO:tasks.ceph.mon.a.smithi088.stderr:==00:00:19:45.621 34743== Warning: unimplemented fcntl command: 1036 2024-06-03T08:54:50.328 INFO:tasks.ceph.mon.b.smithi192.stderr:==00:00:19:45.630 34228== Warning: unimplemented fcntl command: 1036 2024-06-03T08:54:50.329 INFO:tasks.ceph.mon.b.smithi192.stderr:==00:00:19:45.631 34228== Warning: unimplemented fcntl command: 1036 2024-06-03T08:54:50.341 INFO:tasks.ceph.mon.b.smithi192.stderr:==00:00:19:45.644 34228== Warning: unimplemented fcntl command: 1036 2024-06-03T08:54:50.348 INFO:tasks.ceph.mon.c.smithi088.stderr:==00:00:19:45.674 34749== Warning: unimplemented fcntl command: 1036 2024-06-03T08:54:50.349 INFO:tasks.ceph.mon.c.smithi088.stderr:==00:00:19:45.675 34749== Warning: unimplemented fcntl command: 1036 2024-06-03T08:54:50.362 INFO:tasks.ceph.mon.c.smithi088.stderr:==00:00:19:45.688 34749== Warning: unimplemented fcntl command: 1036 2024-06-03T08:55:00.303 INFO:tasks.ceph.mon.a.smithi088.stderr:==00:00:19:55.629 34743== Warning: unimplemented fcntl command: 1036 2024-06-03T08:55:00.305 INFO:tasks.ceph.mon.a.smithi088.stderr:==00:00:19:55.631 34743== Warning: unimplemented fcntl command: 1036 2024-06-03T08:55:00.322 INFO:tasks.ceph.mon.a.smithi088.stderr:==00:00:19:55.648 34743== Warning: unimplemented fcntl command: 1036 2024-06-03T08:55:00.353 INFO:tasks.ceph.mon.c.smithi088.stderr:==00:00:19:55.678 34749== Warning: unimplemented fcntl command: 1036 2024-06-03T08:55:00.355 INFO:tasks.ceph.mon.b.smithi192.stderr:==00:00:19:55.658 34228== Warning: unimplemented fcntl command: 1036 2024-06-03T08:55:00.355 INFO:tasks.ceph.mon.c.smithi088.stderr:==00:00:19:55.681 34749== Warning: unimplemented fcntl command: 1036 2024-06-03T08:55:00.356 INFO:tasks.ceph.mon.b.smithi192.stderr:==00:00:19:55.659 34228== Warning: unimplemented fcntl command: 1036 2024-06-03T08:55:00.369 INFO:tasks.ceph.mon.c.smithi088.stderr:==00:00:19:55.695 34749== Warning: unimplemented fcntl command: 1036 2024-06-03T08:55:00.370 INFO:tasks.ceph.mon.b.smithi192.stderr:==00:00:19:55.672 34228== Warning: unimplemented fcntl command: 1036 2024-06-03T08:55:10.326 INFO:tasks.ceph.mon.a.smithi088.stderr:==00:00:20:05.652 34743== Warning: unimplemented fcntl command: 1036 2024-06-03T08:55:10.328 INFO:tasks.ceph.mon.a.smithi088.stderr:==00:00:20:05.654 34743== Warning: unimplemented fcntl command: 1036 2024-06-03T08:55:10.339 INFO:tasks.ceph.mon.a.smithi088.stderr:==00:00:20:05.665 34743== Warning: unimplemented fcntl command: 1036 2024-06-03T08:55:10.378 INFO:tasks.ceph.mon.b.smithi192.stderr:==00:00:20:05.680 34228== Warning: unimplemented fcntl command: 1036 2024-06-03T08:55:10.380 INFO:tasks.ceph.mon.c.smithi088.stderr:==00:00:20:05.706 34749== Warning: unimplemented fcntl command: 1036 2024-06-03T08:55:10.380 INFO:tasks.ceph.mon.b.smithi192.stderr:==00:00:20:05.683 34228== Warning: unimplemented fcntl command: 1036 2024-06-03T08:55:10.381 INFO:tasks.ceph.mon.c.smithi088.stderr:==00:00:20:05.707 34749== Warning: unimplemented fcntl command: 1036 2024-06-03T08:55:10.393 INFO:tasks.ceph.mon.b.smithi192.stderr:==00:00:20:05.696 34228== Warning: unimplemented fcntl command: 1036 2024-06-03T08:55:10.394 INFO:tasks.ceph.mon.c.smithi088.stderr:==00:00:20:05.720 34749== Warning: unimplemented fcntl command: 1036 2024-06-03T08:55:20.449 INFO:tasks.ceph.mon.a.smithi088.stderr:==00:00:20:15.775 34743== Warning: unimplemented fcntl command: 1036 2024-06-03T08:55:20.451 INFO:tasks.ceph.mon.a.smithi088.stderr:==00:00:20:15.777 34743== Warning: unimplemented fcntl command: 1036 2024-06-03T08:55:20.456 INFO:tasks.ceph.mon.b.smithi192.stderr:==00:00:20:15.758 34228== Warning: unimplemented fcntl command: 1036 2024-06-03T08:55:20.457 INFO:tasks.ceph.mon.c.smithi088.stderr:==00:00:20:15.783 34749== Warning: unimplemented fcntl command: 1036 2024-06-03T08:55:20.503 INFO:tasks.ceph.mon.b.smithi192.stderr:==00:00:20:15.806 34228== Warning: unimplemented fcntl command: 1036 2024-06-03T08:55:20.519 INFO:tasks.ceph.mon.c.smithi088.stderr:==00:00:20:15.845 34749== Warning: unimplemented fcntl command: 1036 2024-06-03T08:55:20.524 INFO:tasks.ceph.mon.b.smithi192.stderr:==00:00:20:15.826 34228== Warning: unimplemented fcntl command: 1036 2024-06-03T08:55:20.535 INFO:tasks.ceph.mon.a.smithi088.stderr:==00:00:20:15.861 34743== Warning: unimplemented fcntl command: 1036 2024-06-03T08:55:20.542 INFO:tasks.ceph.mon.c.smithi088.stderr:==00:00:20:15.868 34749== Warning: unimplemented fcntl command: 1036 2024-06-03T08:55:30.519 INFO:tasks.ceph.mon.a.smithi088.stderr:==00:00:20:25.844 34743== Warning: unimplemented fcntl command: 1036 2024-06-03T08:55:30.526 INFO:tasks.ceph.mon.a.smithi088.stderr:==00:00:20:25.852 34743== Warning: unimplemented fcntl command: 1036 2024-06-03T08:55:30.534 INFO:tasks.ceph.mon.b.smithi192.stderr:==00:00:20:25.837 34228== Warning: unimplemented fcntl command: 1036 2024-06-03T08:55:30.538 INFO:tasks.ceph.mon.b.smithi192.stderr:==00:00:20:25.841 34228== Warning: unimplemented fcntl command: 1036 2024-06-03T08:55:30.552 INFO:tasks.ceph.mon.a.smithi088.stderr:==00:00:20:25.878 34743== Warning: unimplemented fcntl command: 1036 2024-06-03T08:55:30.561 INFO:tasks.ceph.mon.b.smithi192.stderr:==00:00:20:25.864 34228== Warning: unimplemented fcntl command: 1036 2024-06-03T08:55:30.581 INFO:tasks.ceph.mon.c.smithi088.stderr:==00:00:20:25.906 34749== Warning: unimplemented fcntl command: 1036 2024-06-03T08:55:30.582 INFO:tasks.ceph.mon.c.smithi088.stderr:==00:00:20:25.908 34749== Warning: unimplemented fcntl command: 1036 2024-06-03T08:55:30.605 INFO:tasks.ceph.mon.c.smithi088.stderr:==00:00:20:25.931 34749== Warning: unimplemented fcntl command: 1036 2024-06-03T08:55:31.424 INFO:tasks.ceph.mon.b.smithi192.stderr:==00:00:20:26.727 34228== Warning: unimplemented fcntl command: 1036 2024-06-03T08:55:31.426 INFO:tasks.ceph.mon.b.smithi192.stderr:==00:00:20:26.729 34228== Warning: unimplemented fcntl command: 1036 2024-06-03T08:55:31.441 INFO:tasks.ceph.mon.b.smithi192.stderr:==00:00:20:26.744 34228== Warning: unimplemented fcntl command: 1036 2024-06-03T08:55:31.465 INFO:tasks.ceph.mon.c.smithi088.stderr:==00:00:20:26.790 34749== Warning: unimplemented fcntl command: 1036 2024-06-03T08:55:31.466 INFO:tasks.ceph.mon.c.smithi088.stderr:==00:00:20:26.792 34749== Warning: unimplemented fcntl command: 1036 2024-06-03T08:55:31.492 INFO:tasks.ceph.mon.c.smithi088.stderr:==00:00:20:26.818 34749== Warning: unimplemented fcntl command: 1036 2024-06-03T08:55:31.805 INFO:tasks.ceph.mon.a.smithi088.stderr:==00:00:20:27.131 34743== Warning: unimplemented fcntl command: 1036 2024-06-03T08:55:31.807 INFO:tasks.ceph.mon.a.smithi088.stderr:==00:00:20:27.133 34743== Warning: unimplemented fcntl command: 1036 2024-06-03T08:55:31.829 INFO:tasks.ceph.mon.a.smithi088.stderr:==00:00:20:27.155 34743== Warning: unimplemented fcntl command: 1036 2024-06-03T08:55:34.779 INFO:tasks.ceph.osd.5.smithi192.stderr:2024-06-03T08:55:34.762+0000 25e9c640 -1 snap_mapper.add_oid found existing snaps mapped on 74:0f7ab152:test-rados-api-smithi088-40561-76::foo:2a, removing 2024-06-03T08:55:34.786 INFO:tasks.ceph.osd.6.smithi192.stderr:2024-06-03T08:55:34.777+0000 28aa2640 -1 snap_mapper.add_oid found existing snaps mapped on 74:0f7ab152:test-rados-api-smithi088-40561-76::foo:2a, removing 2024-06-03T08:55:35.537 INFO:tasks.ceph.mon.a.smithi088.stderr:==00:00:20:30.862 34743== Warning: unimplemented fcntl command: 1036 2024-06-03T08:55:35.539 INFO:tasks.ceph.mon.a.smithi088.stderr:==00:00:20:30.865 34743== Warning: unimplemented fcntl command: 1036 2024-06-03T08:55:35.558 INFO:tasks.ceph.mon.a.smithi088.stderr:==00:00:20:30.884 34743== Warning: unimplemented fcntl command: 1036 2024-06-03T08:55:35.595 INFO:tasks.ceph.mon.b.smithi192.stderr:==00:00:20:30.897 34228== Warning: unimplemented fcntl command: 1036 2024-06-03T08:55:35.598 INFO:tasks.ceph.mon.c.smithi088.stderr:==00:00:20:30.924 34749== Warning: unimplemented fcntl command: 1036 2024-06-03T08:55:35.598 INFO:tasks.ceph.mon.b.smithi192.stderr:==00:00:20:30.901 34228== Warning: unimplemented fcntl command: 1036 2024-06-03T08:55:35.599 INFO:tasks.ceph.mon.c.smithi088.stderr:==00:00:20:30.925 34749== Warning: unimplemented fcntl command: 1036 2024-06-03T08:55:35.612 INFO:tasks.ceph.mon.b.smithi192.stderr:==00:00:20:30.915 34228== Warning: unimplemented fcntl command: 1036 2024-06-03T08:55:35.616 INFO:tasks.ceph.mon.c.smithi088.stderr:==00:00:20:30.942 34749== Warning: unimplemented fcntl command: 1036 2024-06-03T08:55:40.813 INFO:tasks.ceph.osd.0.smithi088.stderr:2024-06-03T08:55:40.806+0000 22295640 -1 snap_mapper.add_oid found existing snaps mapped on 74:8a027439:test-rados-api-smithi088-40561-78::foo:2b, removing 2024-06-03T08:55:40.822 INFO:tasks.ceph.osd.4.smithi192.stderr:2024-06-03T08:55:40.815+0000 2d2ab640 -1 snap_mapper.add_oid found existing snaps mapped on 74:8a027439:test-rados-api-smithi088-40561-78::foo:2b, removing 2024-06-03T08:55:50.592 INFO:tasks.ceph.mon.a.smithi088.stderr:==00:00:20:45.918 34743== Warning: unimplemented fcntl command: 1036 2024-06-03T08:55:50.595 INFO:tasks.ceph.mon.a.smithi088.stderr:==00:00:20:45.921 34743== Warning: unimplemented fcntl command: 1036 2024-06-03T08:55:50.596 INFO:tasks.ceph.mon.b.smithi192.stderr:==00:00:20:45.899 34228== Warning: unimplemented fcntl command: 1036 2024-06-03T08:55:50.599 INFO:tasks.ceph.mon.b.smithi192.stderr:==00:00:20:45.902 34228== Warning: unimplemented fcntl command: 1036 2024-06-03T08:55:50.621 INFO:tasks.ceph.mon.a.smithi088.stderr:==00:00:20:45.947 34743== Warning: unimplemented fcntl command: 1036 2024-06-03T08:55:50.624 INFO:tasks.ceph.mon.b.smithi192.stderr:==00:00:20:45.926 34228== Warning: unimplemented fcntl command: 1036 2024-06-03T08:55:50.646 INFO:tasks.ceph.mon.c.smithi088.stderr:==00:00:20:45.972 34749== Warning: unimplemented fcntl command: 1036 2024-06-03T08:55:50.647 INFO:tasks.ceph.mon.c.smithi088.stderr:==00:00:20:45.973 34749== Warning: unimplemented fcntl command: 1036 2024-06-03T08:55:50.671 INFO:tasks.ceph.mon.c.smithi088.stderr:==00:00:20:45.997 34749== Warning: unimplemented fcntl command: 1036 2024-06-03T08:56:07.469 INFO:tasks.ceph.osd.6.smithi192.stderr:2024-06-03T08:56:07.453+0000 2629d640 -1 snap_mapper.add_oid found existing snaps mapped on 220:a76eb96c:test-rados-api-smithi088-40561-80::foo:2, removing 2024-06-03T08:56:07.474 INFO:tasks.ceph.osd.7.smithi192.stderr:2024-06-03T08:56:07.467+0000 2f2af640 -1 snap_mapper.add_oid found existing snaps mapped on 220:a76eb96c:test-rados-api-smithi088-40561-80::foo:2, removing 2024-06-03T08:56:10.820 INFO:tasks.ceph.mon.a.smithi088.stderr:==00:00:21:06.145 34743== Warning: unimplemented fcntl command: 1036 2024-06-03T08:56:10.827 INFO:tasks.ceph.mon.a.smithi088.stderr:==00:00:21:06.153 34743== Warning: unimplemented fcntl command: 1036 2024-06-03T08:56:10.865 INFO:tasks.ceph.mon.a.smithi088.stderr:==00:00:21:06.191 34743== Warning: unimplemented fcntl command: 1036 2024-06-03T08:56:10.884 INFO:tasks.ceph.mon.b.smithi192.stderr:==00:00:21:06.187 34228== Warning: unimplemented fcntl command: 1036 2024-06-03T08:56:10.888 INFO:tasks.ceph.mon.c.smithi088.stderr:==00:00:21:06.214 34749== Warning: unimplemented fcntl command: 1036 2024-06-03T08:56:10.890 INFO:tasks.ceph.mon.c.smithi088.stderr:==00:00:21:06.216 34749== Warning: unimplemented fcntl command: 1036 2024-06-03T08:56:10.892 INFO:tasks.ceph.mon.b.smithi192.stderr:==00:00:21:06.195 34228== Warning: unimplemented fcntl command: 1036 2024-06-03T08:56:10.913 INFO:tasks.ceph.mon.b.smithi192.stderr:==00:00:21:06.216 34228== Warning: unimplemented fcntl command: 1036 2024-06-03T08:56:10.920 INFO:tasks.ceph.mon.c.smithi088.stderr:==00:00:21:06.246 34749== Warning: unimplemented fcntl command: 1036 2024-06-03T08:56:25.690 INFO:tasks.ceph.mon.a.smithi088.stderr:==00:00:21:21.016 34743== Warning: unimplemented fcntl command: 1036 2024-06-03T08:56:25.700 INFO:tasks.ceph.mon.a.smithi088.stderr:==00:00:21:21.026 34743== Warning: unimplemented fcntl command: 1036 2024-06-03T08:56:25.730 INFO:tasks.ceph.mon.a.smithi088.stderr:==00:00:21:21.056 34743== Warning: unimplemented fcntl command: 1036 2024-06-03T08:56:25.739 INFO:tasks.ceph.mon.b.smithi192.stderr:==00:00:21:21.042 34228== Warning: unimplemented fcntl command: 1036 2024-06-03T08:56:25.741 INFO:tasks.ceph.mon.c.smithi088.stderr:==00:00:21:21.067 34749== Warning: unimplemented fcntl command: 1036 2024-06-03T08:56:25.745 INFO:tasks.ceph.mon.b.smithi192.stderr:==00:00:21:21.048 34228== Warning: unimplemented fcntl command: 1036 2024-06-03T08:56:25.747 INFO:tasks.ceph.mon.c.smithi088.stderr:==00:00:21:21.073 34749== Warning: unimplemented fcntl command: 1036 2024-06-03T08:56:25.771 INFO:tasks.ceph.mon.b.smithi192.stderr:==00:00:21:21.074 34228== Warning: unimplemented fcntl command: 1036 2024-06-03T08:56:25.774 INFO:tasks.ceph.mon.c.smithi088.stderr:==00:00:21:21.100 34749== Warning: unimplemented fcntl command: 1036 2024-06-03T08:56:30.651 INFO:tasks.ceph.mon.a.smithi088.stderr:==00:00:21:25.976 34743== Warning: unimplemented fcntl command: 1036 2024-06-03T08:56:30.700 INFO:tasks.ceph.mon.b.smithi192.stderr:==00:00:21:26.002 34228== Warning: unimplemented fcntl command: 1036 2024-06-03T08:56:30.700 INFO:tasks.ceph.mon.a.smithi088.stderr:==00:00:21:26.026 34743== Warning: unimplemented fcntl command: 1036 2024-06-03T08:56:30.714 INFO:tasks.ceph.mon.a.smithi088.stderr:==00:00:21:26.040 34743== Warning: unimplemented fcntl command: 1036 2024-06-03T08:56:30.716 INFO:tasks.ceph.mon.b.smithi192.stderr:==00:00:21:26.019 34228== Warning: unimplemented fcntl command: 1036 2024-06-03T08:56:30.728 INFO:tasks.ceph.mon.b.smithi192.stderr:==00:00:21:26.031 34228== Warning: unimplemented fcntl command: 1036 2024-06-03T08:56:30.750 INFO:tasks.ceph.mon.c.smithi088.stderr:==00:00:21:26.076 34749== Warning: unimplemented fcntl command: 1036 2024-06-03T08:56:30.752 INFO:tasks.ceph.mon.c.smithi088.stderr:==00:00:21:26.077 34749== Warning: unimplemented fcntl command: 1036 2024-06-03T08:56:30.770 INFO:tasks.ceph.mon.c.smithi088.stderr:==00:00:21:26.096 34749== Warning: unimplemented fcntl command: 1036 2024-06-03T08:56:40.772 INFO:tasks.ceph.mon.a.smithi088.stderr:==00:00:21:36.097 34743== Warning: unimplemented fcntl command: 1036 2024-06-03T08:56:40.820 INFO:tasks.ceph.mon.b.smithi192.stderr:==00:00:21:36.122 34228== Warning: unimplemented fcntl command: 1036 2024-06-03T08:56:40.821 INFO:tasks.ceph.mon.b.smithi192.stderr:==00:00:21:36.123 34228== Warning: unimplemented fcntl command: 1036 2024-06-03T08:56:40.828 INFO:tasks.ceph.mon.a.smithi088.stderr:==00:00:21:36.154 34743== Warning: unimplemented fcntl command: 1036 2024-06-03T08:56:40.840 INFO:tasks.ceph.mon.b.smithi192.stderr:==00:00:21:36.143 34228== Warning: unimplemented fcntl command: 1036 2024-06-03T08:56:40.866 INFO:tasks.ceph.mon.a.smithi088.stderr:==00:00:21:36.192 34743== Warning: unimplemented fcntl command: 1036 2024-06-03T08:56:40.875 INFO:tasks.ceph.mon.c.smithi088.stderr:==00:00:21:36.201 34749== Warning: unimplemented fcntl command: 1036 2024-06-03T08:56:40.878 INFO:tasks.ceph.mon.c.smithi088.stderr:==00:00:21:36.204 34749== Warning: unimplemented fcntl command: 1036 2024-06-03T08:56:40.900 INFO:tasks.ceph.mon.c.smithi088.stderr:==00:00:21:36.226 34749== Warning: unimplemented fcntl command: 1036 2024-06-03T08:56:50.943 INFO:tasks.ceph.mon.a.smithi088.stderr:==00:00:21:46.269 34743== Warning: unimplemented fcntl command: 1036 2024-06-03T08:56:50.946 INFO:tasks.ceph.mon.a.smithi088.stderr:==00:00:21:46.272 34743== Warning: unimplemented fcntl command: 1036 2024-06-03T08:56:50.978 INFO:tasks.ceph.mon.a.smithi088.stderr:==00:00:21:46.304 34743== Warning: unimplemented fcntl command: 1036 2024-06-03T08:56:50.992 INFO:tasks.ceph.mon.b.smithi192.stderr:==00:00:21:46.295 34228== Warning: unimplemented fcntl command: 1036 2024-06-03T08:56:50.995 INFO:tasks.ceph.mon.b.smithi192.stderr:==00:00:21:46.298 34228== Warning: unimplemented fcntl command: 1036 2024-06-03T08:56:50.996 INFO:tasks.ceph.mon.c.smithi088.stderr:==00:00:21:46.322 34749== Warning: unimplemented fcntl command: 1036 2024-06-03T08:56:50.999 INFO:tasks.ceph.mon.c.smithi088.stderr:==00:00:21:46.325 34749== Warning: unimplemented fcntl command: 1036 2024-06-03T08:56:51.013 INFO:tasks.ceph.mon.b.smithi192.stderr:==00:00:21:46.315 34228== Warning: unimplemented fcntl command: 1036 2024-06-03T08:56:51.025 INFO:tasks.ceph.mon.c.smithi088.stderr:==00:00:21:46.351 34749== Warning: unimplemented fcntl command: 1036 2024-06-03T08:56:55.907 INFO:tasks.ceph.mon.a.smithi088.stderr:==00:00:21:51.233 34743== Warning: unimplemented fcntl command: 1036 2024-06-03T08:56:55.909 INFO:tasks.ceph.mon.a.smithi088.stderr:==00:00:21:51.235 34743== Warning: unimplemented fcntl command: 1036 2024-06-03T08:56:55.928 INFO:tasks.ceph.mon.a.smithi088.stderr:==00:00:21:51.254 34743== Warning: unimplemented fcntl command: 1036 2024-06-03T08:56:55.955 INFO:tasks.ceph.mon.b.smithi192.stderr:==00:00:21:51.258 34228== Warning: unimplemented fcntl command: 1036 2024-06-03T08:56:55.957 INFO:tasks.ceph.mon.b.smithi192.stderr:==00:00:21:51.259 34228== Warning: unimplemented fcntl command: 1036 2024-06-03T08:56:55.969 INFO:tasks.ceph.mon.b.smithi192.stderr:==00:00:21:51.272 34228== Warning: unimplemented fcntl command: 1036 2024-06-03T08:56:55.973 INFO:tasks.ceph.mon.c.smithi088.stderr:==00:00:21:51.299 34749== Warning: unimplemented fcntl command: 1036 2024-06-03T08:56:55.975 INFO:tasks.ceph.mon.c.smithi088.stderr:==00:00:21:51.301 34749== Warning: unimplemented fcntl command: 1036 2024-06-03T08:56:55.993 INFO:tasks.ceph.mon.c.smithi088.stderr:==00:00:21:51.319 34749== Warning: unimplemented fcntl command: 1036 2024-06-03T08:57:10.798 INFO:tasks.ceph.mon.a.smithi088.stderr:==00:00:22:06.124 34743== Warning: unimplemented fcntl command: 1036 2024-06-03T08:57:10.845 INFO:tasks.ceph.mon.b.smithi192.stderr:==00:00:22:06.148 34228== Warning: unimplemented fcntl command: 1036 2024-06-03T08:57:10.846 INFO:tasks.ceph.mon.b.smithi192.stderr:==00:00:22:06.149 34228== Warning: unimplemented fcntl command: 1036 2024-06-03T08:57:10.849 INFO:tasks.ceph.mon.c.smithi088.stderr:==00:00:22:06.175 34749== Warning: unimplemented fcntl command: 1036 2024-06-03T08:57:10.850 INFO:tasks.ceph.mon.c.smithi088.stderr:==00:00:22:06.176 34749== Warning: unimplemented fcntl command: 1036 2024-06-03T08:57:10.864 INFO:tasks.ceph.mon.a.smithi088.stderr:==00:00:22:06.190 34743== Warning: unimplemented fcntl command: 1036 2024-06-03T08:57:10.877 INFO:tasks.ceph.mon.b.smithi192.stderr:==00:00:22:06.180 34228== Warning: unimplemented fcntl command: 1036 2024-06-03T08:57:10.879 INFO:tasks.ceph.mon.c.smithi088.stderr:==00:00:22:06.205 34749== Warning: unimplemented fcntl command: 1036 2024-06-03T08:57:10.902 INFO:tasks.ceph.mon.a.smithi088.stderr:==00:00:22:06.228 34743== Warning: unimplemented fcntl command: 1036 2024-06-03T08:57:11.565 INFO:tasks.ceph.mon.b.smithi192.stderr:==00:00:22:06.867 34228== Warning: unimplemented fcntl command: 1036 2024-06-03T08:57:11.566 INFO:tasks.ceph.mon.b.smithi192.stderr:==00:00:22:06.868 34228== Warning: unimplemented fcntl command: 1036 2024-06-03T08:57:11.576 INFO:tasks.ceph.mon.b.smithi192.stderr:==00:00:22:06.879 34228== Warning: unimplemented fcntl command: 1036 2024-06-03T08:57:11.625 INFO:tasks.ceph.mon.c.smithi088.stderr:==00:00:22:06.951 34749== Warning: unimplemented fcntl command: 1036 2024-06-03T08:57:11.626 INFO:tasks.ceph.mon.c.smithi088.stderr:==00:00:22:06.952 34749== Warning: unimplemented fcntl command: 1036 2024-06-03T08:57:11.634 INFO:tasks.ceph.mon.c.smithi088.stderr:==00:00:22:06.960 34749== Warning: unimplemented fcntl command: 1036 2024-06-03T08:57:11.713 INFO:tasks.ceph.mon.a.smithi088.stderr:==00:00:22:07.039 34743== Warning: unimplemented fcntl command: 1036 2024-06-03T08:57:11.714 INFO:tasks.ceph.mon.a.smithi088.stderr:==00:00:22:07.040 34743== Warning: unimplemented fcntl command: 1036 2024-06-03T08:57:11.725 INFO:tasks.ceph.mon.a.smithi088.stderr:==00:00:22:07.051 34743== Warning: unimplemented fcntl command: 1036 2024-06-03T08:57:25.887 INFO:tasks.ceph.mon.a.smithi088.stderr:==00:00:22:21.212 34743== Warning: unimplemented fcntl command: 1036 2024-06-03T08:57:25.890 INFO:tasks.ceph.mon.a.smithi088.stderr:==00:00:22:21.217 34743== Warning: unimplemented fcntl command: 1036 2024-06-03T08:57:25.916 INFO:tasks.ceph.mon.a.smithi088.stderr:==00:00:22:21.242 34743== Warning: unimplemented fcntl command: 1036 2024-06-03T08:57:25.937 INFO:tasks.ceph.mon.b.smithi192.stderr:==00:00:22:21.240 34228== Warning: unimplemented fcntl command: 1036 2024-06-03T08:57:25.939 INFO:tasks.ceph.mon.b.smithi192.stderr:==00:00:22:21.242 34228== Warning: unimplemented fcntl command: 1036 2024-06-03T08:57:25.942 INFO:tasks.ceph.mon.c.smithi088.stderr:==00:00:22:21.268 34749== Warning: unimplemented fcntl command: 1036 2024-06-03T08:57:25.943 INFO:tasks.ceph.mon.c.smithi088.stderr:==00:00:22:21.269 34749== Warning: unimplemented fcntl command: 1036 2024-06-03T08:57:25.964 INFO:tasks.ceph.mon.b.smithi192.stderr:==00:00:22:21.267 34228== Warning: unimplemented fcntl command: 1036 2024-06-03T08:57:25.969 INFO:tasks.ceph.mon.c.smithi088.stderr:==00:00:22:21.295 34749== Warning: unimplemented fcntl command: 1036 2024-06-03T08:57:31.002 INFO:tasks.ceph.mon.b.smithi192.stderr:==00:00:22:26.305 34228== Warning: unimplemented fcntl command: 1036 2024-06-03T08:57:31.006 INFO:tasks.ceph.mon.b.smithi192.stderr:==00:00:22:26.309 34228== Warning: unimplemented fcntl command: 1036 2024-06-03T08:57:31.011 INFO:tasks.ceph.mon.a.smithi088.stderr:==00:00:22:26.334 34743== Warning: unimplemented fcntl command: 1036 2024-06-03T08:57:31.020 INFO:tasks.ceph.mon.b.smithi192.stderr:==00:00:22:26.323 34228== Warning: unimplemented fcntl command: 1036 2024-06-03T08:57:31.046 INFO:tasks.ceph.mon.a.smithi088.stderr:==00:00:22:26.372 34743== Warning: unimplemented fcntl command: 1036 2024-06-03T08:57:31.058 INFO:tasks.ceph.mon.a.smithi088.stderr:==00:00:22:26.384 34743== Warning: unimplemented fcntl command: 1036 2024-06-03T08:57:31.061 INFO:tasks.ceph.mon.c.smithi088.stderr:==00:00:22:26.387 34749== Warning: unimplemented fcntl command: 1036 2024-06-03T08:57:31.066 INFO:tasks.ceph.mon.c.smithi088.stderr:==00:00:22:26.392 34749== Warning: unimplemented fcntl command: 1036 2024-06-03T08:57:31.078 INFO:tasks.ceph.mon.c.smithi088.stderr:==00:00:22:26.404 34749== Warning: unimplemented fcntl command: 1036 2024-06-03T08:57:46.046 INFO:tasks.ceph.mon.a.smithi088.stderr:==00:00:22:41.371 34743== Warning: unimplemented fcntl command: 1036 2024-06-03T08:57:46.049 INFO:tasks.ceph.mon.a.smithi088.stderr:==00:00:22:41.375 34743== Warning: unimplemented fcntl command: 1036 2024-06-03T08:57:46.076 INFO:tasks.ceph.mon.a.smithi088.stderr:==00:00:22:41.401 34743== Warning: unimplemented fcntl command: 1036 2024-06-03T08:57:46.103 INFO:tasks.ceph.mon.b.smithi192.stderr:==00:00:22:41.406 34228== Warning: unimplemented fcntl command: 1036 2024-06-03T08:57:46.105 INFO:tasks.ceph.mon.c.smithi088.stderr:==00:00:22:41.431 34749== Warning: unimplemented fcntl command: 1036 2024-06-03T08:57:46.105 INFO:tasks.ceph.mon.b.smithi192.stderr:==00:00:22:41.408 34228== Warning: unimplemented fcntl command: 1036 2024-06-03T08:57:46.110 INFO:tasks.ceph.mon.c.smithi088.stderr:==00:00:22:41.436 34749== Warning: unimplemented fcntl command: 1036 2024-06-03T08:57:46.122 INFO:tasks.ceph.mon.b.smithi192.stderr:==00:00:22:41.424 34228== Warning: unimplemented fcntl command: 1036 2024-06-03T08:57:46.135 INFO:tasks.ceph.mon.c.smithi088.stderr:==00:00:22:41.461 34749== Warning: unimplemented fcntl command: 1036 2024-06-03T08:57:51.104 INFO:tasks.ceph.mon.a.smithi088.stderr:==00:00:22:46.430 34743== Warning: unimplemented fcntl command: 1036 2024-06-03T08:57:51.166 INFO:tasks.ceph.mon.c.smithi088.stderr:==00:00:22:46.492 34749== Warning: unimplemented fcntl command: 1036 2024-06-03T08:57:51.169 INFO:tasks.ceph.mon.c.smithi088.stderr:==00:00:22:46.493 34749== Warning: unimplemented fcntl command: 1036 2024-06-03T08:57:51.170 INFO:tasks.ceph.mon.a.smithi088.stderr:==00:00:22:46.494 34743== Warning: unimplemented fcntl command: 1036 2024-06-03T08:57:51.176 INFO:tasks.ceph.mon.b.smithi192.stderr:==00:00:22:46.479 34228== Warning: unimplemented fcntl command: 1036 2024-06-03T08:57:51.178 INFO:tasks.ceph.mon.b.smithi192.stderr:==00:00:22:46.480 34228== Warning: unimplemented fcntl command: 1036 2024-06-03T08:57:51.180 INFO:tasks.ceph.mon.c.smithi088.stderr:==00:00:22:46.506 34749== Warning: unimplemented fcntl command: 1036 2024-06-03T08:57:51.195 INFO:tasks.ceph.mon.a.smithi088.stderr:==00:00:22:46.521 34743== Warning: unimplemented fcntl command: 1036 2024-06-03T08:57:51.197 INFO:tasks.ceph.mon.b.smithi192.stderr:==00:00:22:46.500 34228== Warning: unimplemented fcntl command: 1036 2024-06-03T08:57:52.575 INFO:tasks.ceph.mon.a.smithi088.stderr:==00:00:22:47.901 34743== Warning: unimplemented fcntl command: 1036 2024-06-03T08:57:52.581 INFO:tasks.ceph.mon.a.smithi088.stderr:==00:00:22:47.907 34743== Warning: unimplemented fcntl command: 1036 2024-06-03T08:57:52.596 INFO:tasks.ceph.mon.a.smithi088.stderr:==00:00:22:47.922 34743== Warning: unimplemented fcntl command: 1036 2024-06-03T08:57:52.637 INFO:tasks.ceph.mon.b.smithi192.stderr:==00:00:22:47.939 34228== Warning: unimplemented fcntl command: 1036 2024-06-03T08:57:52.640 INFO:tasks.ceph.mon.b.smithi192.stderr:==00:00:22:47.943 34228== Warning: unimplemented fcntl command: 1036 2024-06-03T08:57:52.657 INFO:tasks.ceph.mon.c.smithi088.stderr:==00:00:22:47.982 34749== Warning: unimplemented fcntl command: 1036 2024-06-03T08:57:52.658 INFO:tasks.ceph.mon.b.smithi192.stderr:==00:00:22:47.960 34228== Warning: unimplemented fcntl command: 1036 2024-06-03T08:57:52.661 INFO:tasks.ceph.mon.c.smithi088.stderr:==00:00:22:47.987 34749== Warning: unimplemented fcntl command: 1036 2024-06-03T08:57:52.675 INFO:tasks.ceph.mon.c.smithi088.stderr:==00:00:22:48.001 34749== Warning: unimplemented fcntl command: 1036 2024-06-03T08:58:01.017 INFO:tasks.ceph.mon.a.smithi088.stderr:==00:00:22:56.343 34743== Warning: unimplemented fcntl command: 1036 2024-06-03T08:58:01.031 INFO:tasks.ceph.mon.a.smithi088.stderr:==00:00:22:56.357 34743== Warning: unimplemented fcntl command: 1036 2024-06-03T08:58:01.042 INFO:tasks.ceph.mon.a.smithi088.stderr:==00:00:22:56.368 34743== Warning: unimplemented fcntl command: 1036 2024-06-03T08:58:01.081 INFO:tasks.ceph.mon.c.smithi088.stderr:==00:00:22:56.407 34749== Warning: unimplemented fcntl command: 1036 2024-06-03T08:58:01.082 INFO:tasks.ceph.mon.c.smithi088.stderr:==00:00:22:56.408 34749== Warning: unimplemented fcntl command: 1036 2024-06-03T08:58:01.084 INFO:tasks.ceph.mon.b.smithi192.stderr:==00:00:22:56.387 34228== Warning: unimplemented fcntl command: 1036 2024-06-03T08:58:01.086 INFO:tasks.ceph.mon.b.smithi192.stderr:==00:00:22:56.388 34228== Warning: unimplemented fcntl command: 1036 2024-06-03T08:58:01.094 INFO:tasks.ceph.mon.c.smithi088.stderr:==00:00:22:56.420 34749== Warning: unimplemented fcntl command: 1036 2024-06-03T08:58:01.098 INFO:tasks.ceph.mon.b.smithi192.stderr:==00:00:22:56.401 34228== Warning: unimplemented fcntl command: 1036 2024-06-03T08:58:02.779 INFO:tasks.ceph.osd.1.smithi088.stderr:2024-06-03T08:58:02.764+0000 21e94640 -1 snap_mapper.add_oid found existing snaps mapped on 74:86ddbdd4:test-rados-api-smithi088-40561-97::foo:33, removing 2024-06-03T08:58:02.801 INFO:tasks.ceph.osd.0.smithi088.stderr:2024-06-03T08:58:02.796+0000 2529b640 -1 snap_mapper.add_oid found existing snaps mapped on 74:86ddbdd4:test-rados-api-smithi088-40561-97::foo:33, removing 2024-06-03T08:58:11.182 INFO:tasks.ceph.mon.a.smithi088.stderr:==00:00:23:06.507 34743== Warning: unimplemented fcntl command: 1036 2024-06-03T08:58:11.238 INFO:tasks.ceph.mon.a.smithi088.stderr:==00:00:23:06.564 34743== Warning: unimplemented fcntl command: 1036 2024-06-03T08:58:11.259 INFO:tasks.ceph.mon.a.smithi088.stderr:==00:00:23:06.585 34743== Warning: unimplemented fcntl command: 1036 2024-06-03T08:58:11.286 INFO:tasks.ceph.mon.b.smithi192.stderr:==00:00:23:06.585 34228== Warning: unimplemented fcntl command: 1036 2024-06-03T08:58:11.286 INFO:tasks.ceph.mon.b.smithi192.stderr:==00:00:23:06.587 34228== Warning: unimplemented fcntl command: 1036 2024-06-03T08:58:11.304 INFO:tasks.ceph.mon.b.smithi192.stderr:==00:00:23:06.606 34228== Warning: unimplemented fcntl command: 1036 2024-06-03T08:58:11.311 INFO:tasks.ceph.mon.c.smithi088.stderr:==00:00:23:06.636 34749== Warning: unimplemented fcntl command: 1036 2024-06-03T08:58:11.312 INFO:tasks.ceph.mon.c.smithi088.stderr:==00:00:23:06.638 34749== Warning: unimplemented fcntl command: 1036 2024-06-03T08:58:11.332 INFO:tasks.ceph.mon.c.smithi088.stderr:==00:00:23:06.658 34749== Warning: unimplemented fcntl command: 1036 2024-06-03T08:58:16.022 INFO:tasks.ceph.mon.a.smithi088.stderr:==00:00:23:11.348 34743== Warning: unimplemented fcntl command: 1036 2024-06-03T08:58:16.070 INFO:tasks.ceph.mon.b.smithi192.stderr:==00:00:23:11.373 34228== Warning: unimplemented fcntl command: 1036 2024-06-03T08:58:16.072 INFO:tasks.ceph.mon.a.smithi088.stderr:==00:00:23:11.398 34743== Warning: unimplemented fcntl command: 1036 2024-06-03T08:58:16.075 INFO:tasks.ceph.mon.b.smithi192.stderr:==00:00:23:11.378 34228== Warning: unimplemented fcntl command: 1036 2024-06-03T08:58:16.088 INFO:tasks.ceph.mon.a.smithi088.stderr:==00:00:23:11.414 34743== Warning: unimplemented fcntl command: 1036 2024-06-03T08:58:16.090 INFO:tasks.ceph.mon.b.smithi192.stderr:==00:00:23:11.393 34228== Warning: unimplemented fcntl command: 1036 2024-06-03T08:58:16.121 INFO:tasks.ceph.mon.c.smithi088.stderr:==00:00:23:11.447 34749== Warning: unimplemented fcntl command: 1036 2024-06-03T08:58:16.122 INFO:tasks.ceph.mon.c.smithi088.stderr:==00:00:23:11.448 34749== Warning: unimplemented fcntl command: 1036 2024-06-03T08:58:16.137 INFO:tasks.ceph.mon.c.smithi088.stderr:==00:00:23:11.463 34749== Warning: unimplemented fcntl command: 1036 2024-06-03T08:58:17.098 INFO:tasks.workunit.client.0.smithi088.stdout: api_tier_pp: [ OK ] LibRadosTwoPoolsPP.ManifestDedupRefRead (4476 ms) 2024-06-03T08:58:17.098 INFO:tasks.workunit.client.0.smithi088.stdout: api_tier_pp: [ RUN ] LibRadosTwoPoolsPP.ManifestSnapRefcount 2024-06-03T08:58:17.098 INFO:tasks.workunit.client.0.smithi088.stdout: api_tier_pp: require_osd_release = reef 2024-06-03T08:58:17.098 INFO:tasks.workunit.client.0.smithi088.stdout: api_tier_pp: [ OK ] LibRadosTwoPoolsPP.ManifestSnapRefcount (40513 ms) 2024-06-03T08:58:17.098 INFO:tasks.workunit.client.0.smithi088.stdout: api_tier_pp: [ RUN ] LibRadosTwoPoolsPP.ManifestSnapRefcount2 2024-06-03T08:58:17.099 INFO:tasks.workunit.client.0.smithi088.stdout: api_tier_pp: require_osd_release = reef 2024-06-03T08:58:17.099 INFO:tasks.workunit.client.0.smithi088.stdout: api_tier_pp: [ OK ] LibRadosTwoPoolsPP.ManifestSnapRefcount2 (17924 ms) 2024-06-03T08:58:17.099 INFO:tasks.workunit.client.0.smithi088.stdout: api_tier_pp: [ RUN ] LibRadosTwoPoolsPP.ManifestTestSnapCreate 2024-06-03T08:58:17.099 INFO:tasks.workunit.client.0.smithi088.stdout: api_tier_pp: require_osd_release = reef 2024-06-03T08:58:17.099 INFO:tasks.workunit.client.0.smithi088.stdout: api_tier_pp: [ OK ] LibRadosTwoPoolsPP.ManifestTestSnapCreate (5516 ms) 2024-06-03T08:58:17.099 INFO:tasks.workunit.client.0.smithi088.stdout: api_tier_pp: [ RUN ] LibRadosTwoPoolsPP.ManifestRedirectAfterPromote 2024-06-03T08:58:17.099 INFO:tasks.workunit.client.0.smithi088.stdout: api_tier_pp: require_osd_release = reef 2024-06-03T08:58:17.099 INFO:tasks.workunit.client.0.smithi088.stdout: api_tier_pp: [ OK ] LibRadosTwoPoolsPP.ManifestRedirectAfterPromote (6672 ms) 2024-06-03T08:58:17.099 INFO:tasks.workunit.client.0.smithi088.stdout: api_tier_pp: [ RUN ] LibRadosTwoPoolsPP.ManifestCheckRefcountWhenModification 2024-06-03T08:58:17.099 INFO:tasks.workunit.client.0.smithi088.stdout: api_tier_pp: require_osd_release = reef 2024-06-03T08:58:17.099 INFO:tasks.workunit.client.0.smithi088.stdout: api_tier_pp: [ OK ] LibRadosTwoPoolsPP.ManifestCheckRefcountWhenModification (26139 ms) 2024-06-03T08:58:17.099 INFO:tasks.workunit.client.0.smithi088.stdout: api_tier_pp: [ RUN ] LibRadosTwoPoolsPP.ManifestSnapIncCount 2024-06-03T08:58:17.100 INFO:tasks.workunit.client.0.smithi088.stdout: api_tier_pp: require_osd_release = reef 2024-06-03T08:58:17.100 INFO:tasks.workunit.client.0.smithi088.stdout: api_tier_pp: [ OK ] LibRadosTwoPoolsPP.ManifestSnapIncCount (16609 ms) 2024-06-03T08:58:17.100 INFO:tasks.workunit.client.0.smithi088.stdout: api_tier_pp: [ RUN ] LibRadosTwoPoolsPP.ManifestEvict 2024-06-03T08:58:17.100 INFO:tasks.workunit.client.0.smithi088.stdout: api_tier_pp: require_osd_release = reef 2024-06-03T08:58:17.100 INFO:tasks.workunit.client.0.smithi088.stdout: api_tier_pp: [ OK ] LibRadosTwoPoolsPP.ManifestEvict (6781 ms) 2024-06-03T08:58:17.100 INFO:tasks.workunit.client.0.smithi088.stdout: api_tier_pp: [ RUN ] LibRadosTwoPoolsPP.ManifestEvictPromote 2024-06-03T08:58:17.100 INFO:tasks.workunit.client.0.smithi088.stdout: api_tier_pp: require_osd_release = reef 2024-06-03T08:58:17.100 INFO:tasks.workunit.client.0.smithi088.stdout: api_tier_pp: [ OK ] LibRadosTwoPoolsPP.ManifestEvictPromote (5554 ms) 2024-06-03T08:58:17.100 INFO:tasks.workunit.client.0.smithi088.stdout: api_tier_pp: [ RUN ] LibRadosTwoPoolsPP.ManifestSnapSizeMismatch 2024-06-03T08:58:17.100 INFO:tasks.workunit.client.0.smithi088.stdout: api_tier_pp: require_osd_release = reef 2024-06-03T08:58:17.100 INFO:tasks.workunit.client.0.smithi088.stdout: api_tier_pp: waiting for scrubs... 2024-06-03T08:58:17.100 INFO:tasks.workunit.client.0.smithi088.stdout: api_tier_pp: done waiting 2024-06-03T08:58:17.100 INFO:tasks.workunit.client.0.smithi088.stdout: api_tier_pp: [ OK ] LibRadosTwoPoolsPP.ManifestSnapSizeMismatch (25770 ms) 2024-06-03T08:58:17.101 INFO:tasks.workunit.client.0.smithi088.stdout: api_tier_pp: [ RUN ] LibRadosTwoPoolsPP.DedupFlushRead 2024-06-03T08:58:17.101 INFO:tasks.workunit.client.0.smithi088.stdout: api_tier_pp: require_osd_release = reef 2024-06-03T08:58:17.101 INFO:tasks.workunit.client.0.smithi088.stdout: api_tier_pp: [ OK ] LibRadosTwoPoolsPP.DedupFlushRead (12393 ms) 2024-06-03T08:58:17.101 INFO:tasks.workunit.client.0.smithi088.stdout: api_tier_pp: [ RUN ] LibRadosTwoPoolsPP.ManifestFlushSnap 2024-06-03T08:58:17.101 INFO:tasks.workunit.client.0.smithi088.stdout: api_tier_pp: require_osd_release = reef 2024-06-03T08:58:17.101 INFO:tasks.workunit.client.0.smithi088.stdout: api_tier_pp: [ OK ] LibRadosTwoPoolsPP.ManifestFlushSnap (25526 ms) 2024-06-03T08:58:17.101 INFO:tasks.workunit.client.0.smithi088.stdout: api_tier_pp: [ RUN ] LibRadosTwoPoolsPP.ManifestFlushDupCount 2024-06-03T08:58:17.101 INFO:tasks.workunit.client.0.smithi088.stdout: api_tier_pp: require_osd_release = reef 2024-06-03T08:58:17.101 INFO:tasks.workunit.client.0.smithi088.stdout: api_tier_pp: [ OK ] LibRadosTwoPoolsPP.ManifestFlushDupCount (9592 ms) 2024-06-03T08:58:17.101 INFO:tasks.workunit.client.0.smithi088.stdout: api_tier_pp: [ RUN ] LibRadosTwoPoolsPP.TierFlushDuringFlush 2024-06-03T08:58:17.101 INFO:tasks.workunit.client.0.smithi088.stdout: api_tier_pp: require_osd_release = reef 2024-06-03T08:58:17.101 INFO:tasks.workunit.client.0.smithi088.stdout: api_tier_pp: [ OK ] LibRadosTwoPoolsPP.TierFlushDuringFlush (11634 ms) 2024-06-03T08:58:17.101 INFO:tasks.workunit.client.0.smithi088.stdout: api_tier_pp: [ RUN ] LibRadosTwoPoolsPP.ManifestSnapHasChunk 2024-06-03T08:58:17.101 INFO:tasks.workunit.client.0.smithi088.stdout: api_tier_pp: require_osd_release = reef 2024-06-03T08:58:17.102 INFO:tasks.workunit.client.0.smithi088.stdout: api_tier_pp: [ OK ] LibRadosTwoPoolsPP.ManifestSnapHasChunk (8495 ms) 2024-06-03T08:58:17.102 INFO:tasks.workunit.client.0.smithi088.stdout: api_tier_pp: [ RUN ] LibRadosTwoPoolsPP.ManifestRollback 2024-06-03T08:58:17.102 INFO:tasks.workunit.client.0.smithi088.stdout: api_tier_pp: require_osd_release = reef 2024-06-03T08:58:17.102 INFO:tasks.workunit.client.0.smithi088.stdout: api_tier_pp: [ OK ] LibRadosTwoPoolsPP.ManifestRollback (6430 ms) 2024-06-03T08:58:17.102 INFO:tasks.workunit.client.0.smithi088.stdout: api_tier_pp: [ RUN ] LibRadosTwoPoolsPP.ManifestRollbackRefcount 2024-06-03T08:58:17.102 INFO:tasks.workunit.client.0.smithi088.stdout: api_tier_pp: require_osd_release = reef 2024-06-03T08:58:17.102 INFO:tasks.workunit.client.0.smithi088.stdout: api_tier_pp: [ OK ] LibRadosTwoPoolsPP.ManifestRollbackRefcount (26972 ms) 2024-06-03T08:58:17.102 INFO:tasks.workunit.client.0.smithi088.stdout: api_tier_pp: [ RUN ] LibRadosTwoPoolsPP.ManifestEvictRollback 2024-06-03T08:58:17.102 INFO:tasks.workunit.client.0.smithi088.stdout: api_tier_pp: require_osd_release = reef 2024-06-03T08:58:17.102 INFO:tasks.workunit.client.0.smithi088.stdout: api_tier_pp: [ OK ] LibRadosTwoPoolsPP.ManifestEvictRollback (14896 ms) 2024-06-03T08:58:17.102 INFO:tasks.workunit.client.0.smithi088.stdout: api_tier_pp: [ RUN ] LibRadosTwoPoolsPP.PropagateBaseTierError 2024-06-03T08:58:31.234 INFO:tasks.ceph.mon.a.smithi088.stderr:==00:00:23:26.559 34743== Warning: unimplemented fcntl command: 1036 2024-06-03T08:58:31.238 INFO:tasks.ceph.mon.a.smithi088.stderr:==00:00:23:26.563 34743== Warning: unimplemented fcntl command: 1036 2024-06-03T08:58:31.269 INFO:tasks.ceph.mon.a.smithi088.stderr:==00:00:23:26.595 34743== Warning: unimplemented fcntl command: 1036 2024-06-03T08:58:31.293 INFO:tasks.ceph.mon.c.smithi088.stderr:==00:00:23:26.619 34749== Warning: unimplemented fcntl command: 1036 2024-06-03T08:58:31.294 INFO:tasks.ceph.mon.c.smithi088.stderr:==00:00:23:26.620 34749== Warning: unimplemented fcntl command: 1036 2024-06-03T08:58:31.297 INFO:tasks.ceph.mon.b.smithi192.stderr:==00:00:23:26.599 34228== Warning: unimplemented fcntl command: 1036 2024-06-03T08:58:31.300 INFO:tasks.ceph.mon.b.smithi192.stderr:==00:00:23:26.603 34228== Warning: unimplemented fcntl command: 1036 2024-06-03T08:58:31.327 INFO:tasks.ceph.mon.c.smithi088.stderr:==00:00:23:26.653 34749== Warning: unimplemented fcntl command: 1036 2024-06-03T08:58:31.329 INFO:tasks.ceph.mon.b.smithi192.stderr:==00:00:23:26.632 34228== Warning: unimplemented fcntl command: 1036 2024-06-03T08:58:36.147 INFO:tasks.ceph.mon.a.smithi088.stderr:==00:00:23:31.473 34743== Warning: unimplemented fcntl command: 1036 2024-06-03T08:58:36.151 INFO:tasks.ceph.mon.a.smithi088.stderr:==00:00:23:31.477 34743== Warning: unimplemented fcntl command: 1036 2024-06-03T08:58:36.165 INFO:tasks.ceph.mon.a.smithi088.stderr:==00:00:23:31.491 34743== Warning: unimplemented fcntl command: 1036 2024-06-03T08:58:36.200 INFO:tasks.ceph.mon.b.smithi192.stderr:==00:00:23:31.503 34228== Warning: unimplemented fcntl command: 1036 2024-06-03T08:58:36.202 INFO:tasks.ceph.mon.b.smithi192.stderr:==00:00:23:31.505 34228== Warning: unimplemented fcntl command: 1036 2024-06-03T08:58:36.205 INFO:tasks.ceph.mon.c.smithi088.stderr:==00:00:23:31.531 34749== Warning: unimplemented fcntl command: 1036 2024-06-03T08:58:36.207 INFO:tasks.ceph.mon.c.smithi088.stderr:==00:00:23:31.533 34749== Warning: unimplemented fcntl command: 1036 2024-06-03T08:58:36.218 INFO:tasks.ceph.mon.b.smithi192.stderr:==00:00:23:31.521 34228== Warning: unimplemented fcntl command: 1036 2024-06-03T08:58:36.223 INFO:tasks.ceph.mon.c.smithi088.stderr:==00:00:23:31.549 34749== Warning: unimplemented fcntl command: 1036 2024-06-03T08:58:41.429 INFO:tasks.ceph.mon.a.smithi088.stderr:==00:00:23:36.755 34743== Warning: unimplemented fcntl command: 1036 2024-06-03T08:58:41.433 INFO:tasks.ceph.mon.a.smithi088.stderr:==00:00:23:36.759 34743== Warning: unimplemented fcntl command: 1036 2024-06-03T08:58:41.459 INFO:tasks.ceph.mon.a.smithi088.stderr:==00:00:23:36.785 34743== Warning: unimplemented fcntl command: 1036 2024-06-03T08:58:41.498 INFO:tasks.ceph.mon.b.smithi192.stderr:==00:00:23:36.801 34228== Warning: unimplemented fcntl command: 1036 2024-06-03T08:58:41.500 INFO:tasks.ceph.mon.b.smithi192.stderr:==00:00:23:36.803 34228== Warning: unimplemented fcntl command: 1036 2024-06-03T08:58:41.503 INFO:tasks.ceph.mon.c.smithi088.stderr:==00:00:23:36.829 34749== Warning: unimplemented fcntl command: 1036 2024-06-03T08:58:41.504 INFO:tasks.ceph.mon.c.smithi088.stderr:==00:00:23:36.830 34749== Warning: unimplemented fcntl command: 1036 2024-06-03T08:58:41.526 INFO:tasks.ceph.mon.b.smithi192.stderr:==00:00:23:36.829 34228== Warning: unimplemented fcntl command: 1036 2024-06-03T08:58:41.531 INFO:tasks.ceph.mon.c.smithi088.stderr:==00:00:23:36.857 34749== Warning: unimplemented fcntl command: 1036 2024-06-03T08:58:51.449 INFO:tasks.ceph.mon.a.smithi088.stderr:==00:00:23:46.774 34743== Warning: unimplemented fcntl command: 1036 2024-06-03T08:58:51.452 INFO:tasks.ceph.mon.a.smithi088.stderr:==00:00:23:46.778 34743== Warning: unimplemented fcntl command: 1036 2024-06-03T08:58:51.478 INFO:tasks.ceph.mon.a.smithi088.stderr:==00:00:23:46.804 34743== Warning: unimplemented fcntl command: 1036 2024-06-03T08:58:51.501 INFO:tasks.ceph.mon.b.smithi192.stderr:==00:00:23:46.804 34228== Warning: unimplemented fcntl command: 1036 2024-06-03T08:58:51.503 INFO:tasks.ceph.mon.b.smithi192.stderr:==00:00:23:46.806 34228== Warning: unimplemented fcntl command: 1036 2024-06-03T08:58:51.509 INFO:tasks.ceph.mon.c.smithi088.stderr:==00:00:23:46.835 34749== Warning: unimplemented fcntl command: 1036 2024-06-03T08:58:51.512 INFO:tasks.ceph.mon.c.smithi088.stderr:==00:00:23:46.839 34749== Warning: unimplemented fcntl command: 1036 2024-06-03T08:58:51.528 INFO:tasks.ceph.mon.b.smithi192.stderr:==00:00:23:46.831 34228== Warning: unimplemented fcntl command: 1036 2024-06-03T08:58:51.537 INFO:tasks.ceph.mon.c.smithi088.stderr:==00:00:23:46.863 34749== Warning: unimplemented fcntl command: 1036 2024-06-03T08:58:52.166 INFO:tasks.ceph.mon.b.smithi192.stderr:==00:00:23:47.468 34228== Warning: unimplemented fcntl command: 1036 2024-06-03T08:58:52.167 INFO:tasks.ceph.mon.b.smithi192.stderr:==00:00:23:47.470 34228== Warning: unimplemented fcntl command: 1036 2024-06-03T08:58:52.226 INFO:tasks.ceph.mon.a.smithi088.stderr:==00:00:23:47.552 34743== Warning: unimplemented fcntl command: 1036 2024-06-03T08:58:52.228 INFO:tasks.ceph.mon.a.smithi088.stderr:==00:00:23:47.554 34743== Warning: unimplemented fcntl command: 1036 2024-06-03T08:58:52.246 INFO:tasks.ceph.mon.b.smithi192.stderr:==00:00:23:47.548 34228== Warning: unimplemented fcntl command: 1036 2024-06-03T08:58:52.246 INFO:tasks.ceph.mon.a.smithi088.stderr:==00:00:23:47.572 34743== Warning: unimplemented fcntl command: 1036 2024-06-03T08:58:52.247 INFO:tasks.ceph.mon.c.smithi088.stderr:==00:00:23:47.572 34749== Warning: unimplemented fcntl command: 1036 2024-06-03T08:58:52.247 INFO:tasks.ceph.mon.c.smithi088.stderr:==00:00:23:47.573 34749== Warning: unimplemented fcntl command: 1036 2024-06-03T08:58:52.258 INFO:tasks.ceph.mon.c.smithi088.stderr:==00:00:23:47.584 34749== Warning: unimplemented fcntl command: 1036 2024-06-03T08:59:06.495 INFO:tasks.ceph.mon.a.smithi088.stderr:==00:00:24:01.820 34743== Warning: unimplemented fcntl command: 1036 2024-06-03T08:59:06.498 INFO:tasks.ceph.mon.a.smithi088.stderr:==00:00:24:01.824 34743== Warning: unimplemented fcntl command: 1036 2024-06-03T08:59:06.529 INFO:tasks.ceph.mon.a.smithi088.stderr:==00:00:24:01.855 34743== Warning: unimplemented fcntl command: 1036 2024-06-03T08:59:06.547 INFO:tasks.ceph.mon.b.smithi192.stderr:==00:00:24:01.850 34228== Warning: unimplemented fcntl command: 1036 2024-06-03T08:59:06.548 INFO:tasks.ceph.mon.c.smithi088.stderr:==00:00:24:01.874 34749== Warning: unimplemented fcntl command: 1036 2024-06-03T08:59:06.548 INFO:tasks.ceph.mon.b.smithi192.stderr:==00:00:24:01.851 34228== Warning: unimplemented fcntl command: 1036 2024-06-03T08:59:06.550 INFO:tasks.ceph.mon.c.smithi088.stderr:==00:00:24:01.876 34749== Warning: unimplemented fcntl command: 1036 2024-06-03T08:59:06.577 INFO:tasks.ceph.mon.b.smithi192.stderr:==00:00:24:01.879 34228== Warning: unimplemented fcntl command: 1036 2024-06-03T08:59:06.577 INFO:tasks.ceph.mon.c.smithi088.stderr:==00:00:24:01.903 34749== Warning: unimplemented fcntl command: 1036 2024-06-03T08:59:11.659 INFO:tasks.ceph.mon.a.smithi088.stderr:==00:00:24:06.985 34743== Warning: unimplemented fcntl command: 1036 2024-06-03T08:59:11.662 INFO:tasks.ceph.mon.a.smithi088.stderr:==00:00:24:06.989 34743== Warning: unimplemented fcntl command: 1036 2024-06-03T08:59:11.679 INFO:tasks.ceph.mon.a.smithi088.stderr:==00:00:24:07.005 34743== Warning: unimplemented fcntl command: 1036 2024-06-03T08:59:11.711 INFO:tasks.ceph.mon.c.smithi088.stderr:==00:00:24:07.037 34749== Warning: unimplemented fcntl command: 1036 2024-06-03T08:59:11.715 INFO:tasks.ceph.mon.c.smithi088.stderr:==00:00:24:07.041 34749== Warning: unimplemented fcntl command: 1036 2024-06-03T08:59:11.717 INFO:tasks.ceph.mon.b.smithi192.stderr:==00:00:24:07.020 34228== Warning: unimplemented fcntl command: 1036 2024-06-03T08:59:11.721 INFO:tasks.ceph.mon.b.smithi192.stderr:==00:00:24:07.024 34228== Warning: unimplemented fcntl command: 1036 2024-06-03T08:59:11.739 INFO:tasks.ceph.mon.c.smithi088.stderr:==00:00:24:07.065 34749== Warning: unimplemented fcntl command: 1036 2024-06-03T08:59:11.745 INFO:tasks.ceph.mon.b.smithi192.stderr:==00:00:24:07.048 34228== Warning: unimplemented fcntl command: 1036 2024-06-03T08:59:16.585 INFO:tasks.ceph.mon.a.smithi088.stderr:==00:00:24:11.911 34743== Warning: unimplemented fcntl command: 1036 2024-06-03T08:59:16.632 INFO:tasks.ceph.mon.a.smithi088.stderr:==00:00:24:11.958 34743== Warning: unimplemented fcntl command: 1036 2024-06-03T08:59:16.655 INFO:tasks.ceph.mon.a.smithi088.stderr:==00:00:24:11.981 34743== Warning: unimplemented fcntl command: 1036 2024-06-03T08:59:16.684 INFO:tasks.ceph.mon.b.smithi192.stderr:==00:00:24:11.987 34228== Warning: unimplemented fcntl command: 1036 2024-06-03T08:59:16.685 INFO:tasks.ceph.mon.b.smithi192.stderr:==00:00:24:11.988 34228== Warning: unimplemented fcntl command: 1036 2024-06-03T08:59:16.690 INFO:tasks.ceph.mon.c.smithi088.stderr:==00:00:24:12.016 34749== Warning: unimplemented fcntl command: 1036 2024-06-03T08:59:16.693 INFO:tasks.ceph.mon.c.smithi088.stderr:==00:00:24:12.019 34749== Warning: unimplemented fcntl command: 1036 2024-06-03T08:59:16.700 INFO:tasks.ceph.mon.b.smithi192.stderr:==00:00:24:12.003 34228== Warning: unimplemented fcntl command: 1036 2024-06-03T08:59:16.709 INFO:tasks.ceph.mon.c.smithi088.stderr:==00:00:24:12.035 34749== Warning: unimplemented fcntl command: 1036 2024-06-03T08:59:21.821 INFO:tasks.ceph.mon.a.smithi088.stderr:==00:00:24:17.147 34743== Warning: unimplemented fcntl command: 1036 2024-06-03T08:59:21.880 INFO:tasks.ceph.mon.b.smithi192.stderr:==00:00:24:17.182 34228== Warning: unimplemented fcntl command: 1036 2024-06-03T08:59:21.886 INFO:tasks.ceph.mon.b.smithi192.stderr:==00:00:24:17.187 34228== Warning: unimplemented fcntl command: 1036 2024-06-03T08:59:21.887 INFO:tasks.ceph.mon.a.smithi088.stderr:==00:00:24:17.213 34743== Warning: unimplemented fcntl command: 1036 2024-06-03T08:59:21.907 INFO:tasks.ceph.mon.b.smithi192.stderr:==00:00:24:17.210 34228== Warning: unimplemented fcntl command: 1036 2024-06-03T08:59:21.920 INFO:tasks.ceph.mon.a.smithi088.stderr:==00:00:24:17.246 34743== Warning: unimplemented fcntl command: 1036 2024-06-03T08:59:21.948 INFO:tasks.ceph.mon.c.smithi088.stderr:==00:00:24:17.273 34749== Warning: unimplemented fcntl command: 1036 2024-06-03T08:59:21.957 INFO:tasks.ceph.mon.c.smithi088.stderr:==00:00:24:17.283 34749== Warning: unimplemented fcntl command: 1036 2024-06-03T08:59:21.979 INFO:tasks.ceph.mon.c.smithi088.stderr:==00:00:24:17.305 34749== Warning: unimplemented fcntl command: 1036 2024-06-03T08:59:31.839 INFO:tasks.ceph.mon.a.smithi088.stderr:==00:00:24:27.165 34743== Warning: unimplemented fcntl command: 1036 2024-06-03T08:59:31.844 INFO:tasks.ceph.mon.a.smithi088.stderr:==00:00:24:27.170 34743== Warning: unimplemented fcntl command: 1036 2024-06-03T08:59:31.866 INFO:tasks.ceph.mon.a.smithi088.stderr:==00:00:24:27.192 34743== Warning: unimplemented fcntl command: 1036 2024-06-03T08:59:31.890 INFO:tasks.ceph.mon.b.smithi192.stderr:==00:00:24:27.192 34228== Warning: unimplemented fcntl command: 1036 2024-06-03T08:59:31.891 INFO:tasks.ceph.mon.b.smithi192.stderr:==00:00:24:27.194 34228== Warning: unimplemented fcntl command: 1036 2024-06-03T08:59:31.892 INFO:tasks.ceph.mon.c.smithi088.stderr:==00:00:24:27.218 34749== Warning: unimplemented fcntl command: 1036 2024-06-03T08:59:31.894 INFO:tasks.ceph.mon.c.smithi088.stderr:==00:00:24:27.220 34749== Warning: unimplemented fcntl command: 1036 2024-06-03T08:59:31.914 INFO:tasks.ceph.mon.b.smithi192.stderr:==00:00:24:27.217 34228== Warning: unimplemented fcntl command: 1036 2024-06-03T08:59:31.917 INFO:tasks.ceph.mon.c.smithi088.stderr:==00:00:24:27.243 34749== Warning: unimplemented fcntl command: 1036 2024-06-03T08:59:36.720 INFO:tasks.ceph.mon.a.smithi088.stderr:==00:00:24:32.045 34743== Warning: unimplemented fcntl command: 1036 2024-06-03T08:59:36.770 INFO:tasks.ceph.mon.a.smithi088.stderr:==00:00:24:32.095 34743== Warning: unimplemented fcntl command: 1036 2024-06-03T08:59:36.775 INFO:tasks.ceph.mon.b.smithi192.stderr:==00:00:24:32.077 34228== Warning: unimplemented fcntl command: 1036 2024-06-03T08:59:36.776 INFO:tasks.ceph.mon.b.smithi192.stderr:==00:00:24:32.079 34228== Warning: unimplemented fcntl command: 1036 2024-06-03T08:59:36.784 INFO:tasks.ceph.mon.a.smithi088.stderr:==00:00:24:32.110 34743== Warning: unimplemented fcntl command: 1036 2024-06-03T08:59:36.789 INFO:tasks.ceph.mon.b.smithi192.stderr:==00:00:24:32.092 34228== Warning: unimplemented fcntl command: 1036 2024-06-03T08:59:36.820 INFO:tasks.ceph.mon.c.smithi088.stderr:==00:00:24:32.146 34749== Warning: unimplemented fcntl command: 1036 2024-06-03T08:59:36.823 INFO:tasks.ceph.mon.c.smithi088.stderr:==00:00:24:32.150 34749== Warning: unimplemented fcntl command: 1036 2024-06-03T08:59:36.836 INFO:tasks.ceph.mon.c.smithi088.stderr:==00:00:24:32.162 34749== Warning: unimplemented fcntl command: 1036 2024-06-03T08:59:51.806 INFO:tasks.ceph.mon.a.smithi088.stderr:==00:00:24:47.132 34743== Warning: unimplemented fcntl command: 1036 2024-06-03T08:59:51.810 INFO:tasks.ceph.mon.a.smithi088.stderr:==00:00:24:47.136 34743== Warning: unimplemented fcntl command: 1036 2024-06-03T08:59:51.848 INFO:tasks.ceph.mon.a.smithi088.stderr:==00:00:24:47.174 34743== Warning: unimplemented fcntl command: 1036 2024-06-03T08:59:51.859 INFO:tasks.ceph.mon.b.smithi192.stderr:==00:00:24:47.162 34228== Warning: unimplemented fcntl command: 1036 2024-06-03T08:59:51.862 INFO:tasks.ceph.mon.b.smithi192.stderr:==00:00:24:47.165 34228== Warning: unimplemented fcntl command: 1036 2024-06-03T08:59:51.872 INFO:tasks.ceph.mon.c.smithi088.stderr:==00:00:24:47.198 34749== Warning: unimplemented fcntl command: 1036 2024-06-03T08:59:51.875 INFO:tasks.ceph.mon.c.smithi088.stderr:==00:00:24:47.201 34749== Warning: unimplemented fcntl command: 1036 2024-06-03T08:59:51.886 INFO:tasks.ceph.mon.b.smithi192.stderr:==00:00:24:47.189 34228== Warning: unimplemented fcntl command: 1036 2024-06-03T08:59:51.899 INFO:tasks.ceph.mon.c.smithi088.stderr:==00:00:24:47.225 34749== Warning: unimplemented fcntl command: 1036 2024-06-03T08:59:52.464 INFO:tasks.ceph.mon.a.smithi088.stderr:==00:00:24:47.790 34743== Warning: unimplemented fcntl command: 1036 2024-06-03T08:59:52.465 INFO:tasks.ceph.mon.a.smithi088.stderr:==00:00:24:47.791 34743== Warning: unimplemented fcntl command: 1036 2024-06-03T08:59:52.469 INFO:tasks.ceph.mon.b.smithi192.stderr:==00:00:24:47.772 34228== Warning: unimplemented fcntl command: 1036 2024-06-03T08:59:52.471 INFO:tasks.ceph.mon.b.smithi192.stderr:==00:00:24:47.773 34228== Warning: unimplemented fcntl command: 1036 2024-06-03T08:59:52.474 INFO:tasks.ceph.mon.a.smithi088.stderr:==00:00:24:47.800 34743== Warning: unimplemented fcntl command: 1036 2024-06-03T08:59:52.481 INFO:tasks.ceph.mon.b.smithi192.stderr:==00:00:24:47.783 34228== Warning: unimplemented fcntl command: 1036 2024-06-03T08:59:52.504 INFO:tasks.ceph.mon.c.smithi088.stderr:==00:00:24:47.830 34749== Warning: unimplemented fcntl command: 1036 2024-06-03T08:59:52.505 INFO:tasks.ceph.mon.c.smithi088.stderr:==00:00:24:47.831 34749== Warning: unimplemented fcntl command: 1036 2024-06-03T08:59:52.514 INFO:tasks.ceph.mon.c.smithi088.stderr:==00:00:24:47.840 34749== Warning: unimplemented fcntl command: 1036 2024-06-03T09:00:01.431 INFO:tasks.ceph.mon.a.smithi088.stderr:==00:00:24:56.757 34743== Warning: unimplemented fcntl command: 1036 2024-06-03T09:00:01.433 INFO:tasks.ceph.mon.a.smithi088.stderr:==00:00:24:56.760 34743== Warning: unimplemented fcntl command: 1036 2024-06-03T09:00:01.446 INFO:tasks.ceph.mon.b.smithi192.stderr:==00:00:24:56.748 34228== Warning: unimplemented fcntl command: 1036 2024-06-03T09:00:01.446 INFO:tasks.ceph.mon.c.smithi088.stderr:==00:00:24:56.773 34749== Warning: unimplemented fcntl command: 1036 2024-06-03T09:00:01.453 INFO:tasks.ceph.mon.c.smithi088.stderr:==00:00:24:56.779 34749== Warning: unimplemented fcntl command: 1036 2024-06-03T09:00:01.489 INFO:tasks.ceph.mon.b.smithi192.stderr:==00:00:24:56.791 34228== Warning: unimplemented fcntl command: 1036 2024-06-03T09:00:01.505 INFO:tasks.ceph.mon.b.smithi192.stderr:==00:00:24:56.807 34228== Warning: unimplemented fcntl command: 1036 2024-06-03T09:00:01.529 INFO:tasks.ceph.mon.a.smithi088.stderr:==00:00:24:56.855 34743== Warning: unimplemented fcntl command: 1036 2024-06-03T09:00:01.538 INFO:tasks.ceph.mon.c.smithi088.stderr:==00:00:24:56.865 34749== Warning: unimplemented fcntl command: 1036 2024-06-03T09:00:12.005 INFO:tasks.ceph.mon.a.smithi088.stderr:==00:00:25:07.330 34743== Warning: unimplemented fcntl command: 1036 2024-06-03T09:00:12.063 INFO:tasks.ceph.mon.b.smithi192.stderr:==00:00:25:07.366 34228== Warning: unimplemented fcntl command: 1036 2024-06-03T09:00:12.065 INFO:tasks.ceph.mon.b.smithi192.stderr:==00:00:25:07.367 34228== Warning: unimplemented fcntl command: 1036 2024-06-03T09:00:12.067 INFO:tasks.ceph.mon.a.smithi088.stderr:==00:00:25:07.393 34743== Warning: unimplemented fcntl command: 1036 2024-06-03T09:00:12.089 INFO:tasks.ceph.mon.b.smithi192.stderr:==00:00:25:07.392 34228== Warning: unimplemented fcntl command: 1036 2024-06-03T09:00:12.090 INFO:tasks.ceph.mon.a.smithi088.stderr:==00:00:25:07.416 34743== Warning: unimplemented fcntl command: 1036 2024-06-03T09:00:12.133 INFO:tasks.ceph.mon.c.smithi088.stderr:==00:00:25:07.459 34749== Warning: unimplemented fcntl command: 1036 2024-06-03T09:00:12.135 INFO:tasks.ceph.mon.c.smithi088.stderr:==00:00:25:07.461 34749== Warning: unimplemented fcntl command: 1036 2024-06-03T09:00:12.161 INFO:tasks.ceph.mon.c.smithi088.stderr:==00:00:25:07.487 34749== Warning: unimplemented fcntl command: 1036 2024-06-03T09:00:27.181 INFO:tasks.ceph.mon.a.smithi088.stderr:==00:00:25:22.506 34743== Warning: unimplemented fcntl command: 1036 2024-06-03T09:00:27.186 INFO:tasks.ceph.mon.a.smithi088.stderr:==00:00:25:22.512 34743== Warning: unimplemented fcntl command: 1036 2024-06-03T09:00:27.215 INFO:tasks.ceph.mon.a.smithi088.stderr:==00:00:25:22.541 34743== Warning: unimplemented fcntl command: 1036 2024-06-03T09:00:27.233 INFO:tasks.ceph.mon.b.smithi192.stderr:==00:00:25:22.536 34228== Warning: unimplemented fcntl command: 1036 2024-06-03T09:00:27.234 INFO:tasks.ceph.mon.b.smithi192.stderr:==00:00:25:22.537 34228== Warning: unimplemented fcntl command: 1036 2024-06-03T09:00:27.237 INFO:tasks.ceph.mon.c.smithi088.stderr:==00:00:25:22.563 34749== Warning: unimplemented fcntl command: 1036 2024-06-03T09:00:27.240 INFO:tasks.ceph.mon.c.smithi088.stderr:==00:00:25:22.567 34749== Warning: unimplemented fcntl command: 1036 2024-06-03T09:00:27.263 INFO:tasks.ceph.mon.b.smithi192.stderr:==00:00:25:22.565 34228== Warning: unimplemented fcntl command: 1036 2024-06-03T09:00:27.271 INFO:tasks.ceph.mon.c.smithi088.stderr:==00:00:25:22.597 34749== Warning: unimplemented fcntl command: 1036 2024-06-03T09:00:31.988 INFO:tasks.ceph.mon.a.smithi088.stderr:==00:00:25:27.314 34743== Warning: unimplemented fcntl command: 1036 2024-06-03T09:00:31.992 INFO:tasks.ceph.mon.a.smithi088.stderr:==00:00:25:27.318 34743== Warning: unimplemented fcntl command: 1036 2024-06-03T09:00:32.007 INFO:tasks.ceph.mon.a.smithi088.stderr:==00:00:25:27.333 34743== Warning: unimplemented fcntl command: 1036 2024-06-03T09:00:32.038 INFO:tasks.ceph.mon.b.smithi192.stderr:==00:00:25:27.341 34228== Warning: unimplemented fcntl command: 1036 2024-06-03T09:00:32.039 INFO:tasks.ceph.mon.b.smithi192.stderr:==00:00:25:27.342 34228== Warning: unimplemented fcntl command: 1036 2024-06-03T09:00:32.050 INFO:tasks.ceph.mon.c.smithi088.stderr:==00:00:25:27.376 34749== Warning: unimplemented fcntl command: 1036 2024-06-03T09:00:32.051 INFO:tasks.ceph.mon.c.smithi088.stderr:==00:00:25:27.377 34749== Warning: unimplemented fcntl command: 1036 2024-06-03T09:00:32.054 INFO:tasks.ceph.mon.b.smithi192.stderr:==00:00:25:27.356 34228== Warning: unimplemented fcntl command: 1036 2024-06-03T09:00:32.065 INFO:tasks.ceph.mon.c.smithi088.stderr:==00:00:25:27.391 34749== Warning: unimplemented fcntl command: 1036 2024-06-03T09:00:41.968 INFO:tasks.ceph.mon.a.smithi088.stderr:==00:00:25:37.294 34743== Warning: unimplemented fcntl command: 1036 2024-06-03T09:00:42.018 INFO:tasks.ceph.mon.b.smithi192.stderr:==00:00:25:37.320 34228== Warning: unimplemented fcntl command: 1036 2024-06-03T09:00:42.019 INFO:tasks.ceph.mon.a.smithi088.stderr:==00:00:25:37.345 34743== Warning: unimplemented fcntl command: 1036 2024-06-03T09:00:42.021 INFO:tasks.ceph.mon.b.smithi192.stderr:==00:00:25:37.324 34228== Warning: unimplemented fcntl command: 1036 2024-06-03T09:00:42.039 INFO:tasks.ceph.mon.a.smithi088.stderr:==00:00:25:37.365 34743== Warning: unimplemented fcntl command: 1036 2024-06-03T09:00:42.041 INFO:tasks.ceph.mon.b.smithi192.stderr:==00:00:25:37.344 34228== Warning: unimplemented fcntl command: 1036 2024-06-03T09:00:42.064 INFO:tasks.ceph.mon.c.smithi088.stderr:==00:00:25:37.390 34749== Warning: unimplemented fcntl command: 1036 2024-06-03T09:00:42.067 INFO:tasks.ceph.mon.c.smithi088.stderr:==00:00:25:37.393 34749== Warning: unimplemented fcntl command: 1036 2024-06-03T09:00:42.088 INFO:tasks.ceph.mon.c.smithi088.stderr:==00:00:25:37.414 34749== Warning: unimplemented fcntl command: 1036 2024-06-03T09:00:52.244 INFO:tasks.ceph.mon.a.smithi088.stderr:==00:00:25:47.570 34743== Warning: unimplemented fcntl command: 1036 2024-06-03T09:00:52.248 INFO:tasks.ceph.mon.a.smithi088.stderr:==00:00:25:47.575 34743== Warning: unimplemented fcntl command: 1036 2024-06-03T09:00:52.298 INFO:tasks.ceph.mon.b.smithi192.stderr:==00:00:25:47.601 34228== Warning: unimplemented fcntl command: 1036 2024-06-03T09:00:52.299 INFO:tasks.ceph.mon.c.smithi088.stderr:==00:00:25:47.625 34749== Warning: unimplemented fcntl command: 1036 2024-06-03T09:00:52.301 INFO:tasks.ceph.mon.b.smithi192.stderr:==00:00:25:47.604 34228== Warning: unimplemented fcntl command: 1036 2024-06-03T09:00:52.303 INFO:tasks.ceph.mon.c.smithi088.stderr:==00:00:25:47.629 34749== Warning: unimplemented fcntl command: 1036 2024-06-03T09:00:52.319 INFO:tasks.ceph.mon.a.smithi088.stderr:==00:00:25:47.645 34743== Warning: unimplemented fcntl command: 1036 2024-06-03T09:00:52.351 INFO:tasks.ceph.mon.b.smithi192.stderr:==00:00:25:47.654 34228== Warning: unimplemented fcntl command: 1036 2024-06-03T09:00:52.352 INFO:tasks.ceph.mon.c.smithi088.stderr:==00:00:25:47.678 34749== Warning: unimplemented fcntl command: 1036 2024-06-03T09:00:57.250 INFO:tasks.ceph.mon.a.smithi088.stderr:==00:00:25:52.576 34743== Warning: unimplemented fcntl command: 1036 2024-06-03T09:00:57.254 INFO:tasks.ceph.mon.a.smithi088.stderr:==00:00:25:52.580 34743== Warning: unimplemented fcntl command: 1036 2024-06-03T09:00:57.268 INFO:tasks.ceph.mon.a.smithi088.stderr:==00:00:25:52.594 34743== Warning: unimplemented fcntl command: 1036 2024-06-03T09:00:57.301 INFO:tasks.ceph.mon.b.smithi192.stderr:==00:00:25:52.603 34228== Warning: unimplemented fcntl command: 1036 2024-06-03T09:00:57.301 INFO:tasks.ceph.mon.c.smithi088.stderr:==00:00:25:52.628 34749== Warning: unimplemented fcntl command: 1036 2024-06-03T09:00:57.304 INFO:tasks.ceph.mon.b.smithi192.stderr:==00:00:25:52.607 34228== Warning: unimplemented fcntl command: 1036 2024-06-03T09:00:57.305 INFO:tasks.ceph.mon.c.smithi088.stderr:==00:00:25:52.631 34749== Warning: unimplemented fcntl command: 1036 2024-06-03T09:00:57.317 INFO:tasks.ceph.mon.b.smithi192.stderr:==00:00:25:52.620 34228== Warning: unimplemented fcntl command: 1036 2024-06-03T09:00:57.318 INFO:tasks.ceph.mon.c.smithi088.stderr:==00:00:25:52.644 34749== Warning: unimplemented fcntl command: 1036 2024-06-03T09:01:12.167 INFO:tasks.ceph.mon.a.smithi088.stderr:==00:00:26:07.492 34743== Warning: unimplemented fcntl command: 1036 2024-06-03T09:01:12.171 INFO:tasks.ceph.mon.a.smithi088.stderr:==00:00:26:07.497 34743== Warning: unimplemented fcntl command: 1036 2024-06-03T09:01:12.187 INFO:tasks.ceph.mon.a.smithi088.stderr:==00:00:26:07.513 34743== Warning: unimplemented fcntl command: 1036 2024-06-03T09:01:12.220 INFO:tasks.ceph.mon.c.smithi088.stderr:==00:00:26:07.546 34749== Warning: unimplemented fcntl command: 1036 2024-06-03T09:01:12.222 INFO:tasks.ceph.mon.b.smithi192.stderr:==00:00:26:07.525 34228== Warning: unimplemented fcntl command: 1036 2024-06-03T09:01:12.223 INFO:tasks.ceph.mon.c.smithi088.stderr:==00:00:26:07.549 34749== Warning: unimplemented fcntl command: 1036 2024-06-03T09:01:12.223 INFO:tasks.ceph.mon.b.smithi192.stderr:==00:00:26:07.526 34228== Warning: unimplemented fcntl command: 1036 2024-06-03T09:01:12.238 INFO:tasks.ceph.mon.b.smithi192.stderr:==00:00:26:07.540 34228== Warning: unimplemented fcntl command: 1036 2024-06-03T09:01:12.239 INFO:tasks.ceph.mon.c.smithi088.stderr:==00:00:26:07.565 34749== Warning: unimplemented fcntl command: 1036 2024-06-03T09:01:17.182 INFO:tasks.ceph.mon.a.smithi088.stderr:==00:00:26:12.508 34743== Warning: unimplemented fcntl command: 1036 2024-06-03T09:01:17.184 INFO:tasks.ceph.mon.a.smithi088.stderr:==00:00:26:12.510 34743== Warning: unimplemented fcntl command: 1036 2024-06-03T09:01:17.197 INFO:tasks.ceph.mon.a.smithi088.stderr:==00:00:26:12.523 34743== Warning: unimplemented fcntl command: 1036 2024-06-03T09:01:17.232 INFO:tasks.ceph.mon.c.smithi088.stderr:==00:00:26:12.558 34749== Warning: unimplemented fcntl command: 1036 2024-06-03T09:01:17.233 INFO:tasks.ceph.mon.b.smithi192.stderr:==00:00:26:12.536 34228== Warning: unimplemented fcntl command: 1036 2024-06-03T09:01:17.234 INFO:tasks.ceph.mon.c.smithi088.stderr:==00:00:26:12.560 34749== Warning: unimplemented fcntl command: 1036 2024-06-03T09:01:17.235 INFO:tasks.ceph.mon.b.smithi192.stderr:==00:00:26:12.538 34228== Warning: unimplemented fcntl command: 1036 2024-06-03T09:01:17.243 INFO:tasks.ceph.mon.c.smithi088.stderr:==00:00:26:12.569 34749== Warning: unimplemented fcntl command: 1036 2024-06-03T09:01:17.244 INFO:tasks.ceph.mon.b.smithi192.stderr:==00:00:26:12.547 34228== Warning: unimplemented fcntl command: 1036 2024-06-03T09:01:32.363 INFO:tasks.ceph.mon.a.smithi088.stderr:==00:00:26:27.689 34743== Warning: unimplemented fcntl command: 1036 2024-06-03T09:01:32.366 INFO:tasks.ceph.mon.a.smithi088.stderr:==00:00:26:27.692 34743== Warning: unimplemented fcntl command: 1036 2024-06-03T09:01:32.391 INFO:tasks.ceph.mon.a.smithi088.stderr:==00:00:26:27.717 34743== Warning: unimplemented fcntl command: 1036 2024-06-03T09:01:32.413 INFO:tasks.ceph.mon.b.smithi192.stderr:==00:00:26:27.716 34228== Warning: unimplemented fcntl command: 1036 2024-06-03T09:01:32.414 INFO:tasks.ceph.mon.b.smithi192.stderr:==00:00:26:27.717 34228== Warning: unimplemented fcntl command: 1036 2024-06-03T09:01:32.424 INFO:tasks.ceph.mon.c.smithi088.stderr:==00:00:26:27.750 34749== Warning: unimplemented fcntl command: 1036 2024-06-03T09:01:32.425 INFO:tasks.ceph.mon.c.smithi088.stderr:==00:00:26:27.752 34749== Warning: unimplemented fcntl command: 1036 2024-06-03T09:01:32.438 INFO:tasks.ceph.mon.b.smithi192.stderr:==00:00:26:27.741 34228== Warning: unimplemented fcntl command: 1036 2024-06-03T09:01:32.455 INFO:tasks.ceph.mon.c.smithi088.stderr:==00:00:26:27.781 34749== Warning: unimplemented fcntl command: 1036 2024-06-03T09:01:47.289 INFO:tasks.ceph.mon.a.smithi088.stderr:==00:00:26:42.615 34743== Warning: unimplemented fcntl command: 1036 2024-06-03T09:01:47.291 INFO:tasks.ceph.mon.a.smithi088.stderr:==00:00:26:42.617 34743== Warning: unimplemented fcntl command: 1036 2024-06-03T09:01:47.331 INFO:tasks.ceph.mon.a.smithi088.stderr:==00:00:26:42.657 34743== Warning: unimplemented fcntl command: 1036 2024-06-03T09:01:47.342 INFO:tasks.ceph.mon.b.smithi192.stderr:==00:00:26:42.644 34228== Warning: unimplemented fcntl command: 1036 2024-06-03T09:01:47.343 INFO:tasks.ceph.mon.b.smithi192.stderr:==00:00:26:42.646 34228== Warning: unimplemented fcntl command: 1036 2024-06-03T09:01:47.356 INFO:tasks.ceph.mon.c.smithi088.stderr:==00:00:26:42.682 34749== Warning: unimplemented fcntl command: 1036 2024-06-03T09:01:47.358 INFO:tasks.ceph.mon.c.smithi088.stderr:==00:00:26:42.684 34749== Warning: unimplemented fcntl command: 1036 2024-06-03T09:01:47.368 INFO:tasks.ceph.mon.b.smithi192.stderr:==00:00:26:42.671 34228== Warning: unimplemented fcntl command: 1036 2024-06-03T09:01:47.394 INFO:tasks.ceph.mon.c.smithi088.stderr:==00:00:26:42.720 34749== Warning: unimplemented fcntl command: 1036 2024-06-03T09:01:52.299 INFO:tasks.ceph.mon.a.smithi088.stderr:==00:00:26:47.625 34743== Warning: unimplemented fcntl command: 1036 2024-06-03T09:01:52.303 INFO:tasks.ceph.mon.a.smithi088.stderr:==00:00:26:47.629 34743== Warning: unimplemented fcntl command: 1036 2024-06-03T09:01:52.317 INFO:tasks.ceph.mon.a.smithi088.stderr:==00:00:26:47.643 34743== Warning: unimplemented fcntl command: 1036 2024-06-03T09:01:52.349 INFO:tasks.ceph.mon.b.smithi192.stderr:==00:00:26:47.651 34228== Warning: unimplemented fcntl command: 1036 2024-06-03T09:01:52.350 INFO:tasks.ceph.mon.b.smithi192.stderr:==00:00:26:47.653 34228== Warning: unimplemented fcntl command: 1036 2024-06-03T09:01:52.353 INFO:tasks.ceph.mon.c.smithi088.stderr:==00:00:26:47.679 34749== Warning: unimplemented fcntl command: 1036 2024-06-03T09:01:52.354 INFO:tasks.ceph.mon.c.smithi088.stderr:==00:00:26:47.680 34749== Warning: unimplemented fcntl command: 1036 2024-06-03T09:01:52.363 INFO:tasks.ceph.mon.b.smithi192.stderr:==00:00:26:47.666 34228== Warning: unimplemented fcntl command: 1036 2024-06-03T09:01:52.374 INFO:tasks.ceph.mon.c.smithi088.stderr:==00:00:26:47.700 34749== Warning: unimplemented fcntl command: 1036 2024-06-03T09:02:02.281 INFO:tasks.ceph.mon.a.smithi088.stderr:==00:00:26:57.607 34743== Warning: unimplemented fcntl command: 1036 2024-06-03T09:02:02.330 INFO:tasks.ceph.mon.b.smithi192.stderr:==00:00:26:57.633 34228== Warning: unimplemented fcntl command: 1036 2024-06-03T09:02:02.333 INFO:tasks.ceph.mon.b.smithi192.stderr:==00:00:26:57.636 34228== Warning: unimplemented fcntl command: 1036 2024-06-03T09:02:02.335 INFO:tasks.ceph.mon.a.smithi088.stderr:==00:00:26:57.662 34743== Warning: unimplemented fcntl command: 1036 2024-06-03T09:02:02.355 INFO:tasks.ceph.mon.b.smithi192.stderr:==00:00:26:57.658 34228== Warning: unimplemented fcntl command: 1036 2024-06-03T09:02:02.364 INFO:tasks.ceph.mon.a.smithi088.stderr:==00:00:26:57.690 34743== Warning: unimplemented fcntl command: 1036 2024-06-03T09:02:02.395 INFO:tasks.ceph.mon.c.smithi088.stderr:==00:00:26:57.721 34749== Warning: unimplemented fcntl command: 1036 2024-06-03T09:02:02.399 INFO:tasks.ceph.mon.c.smithi088.stderr:==00:00:26:57.725 34749== Warning: unimplemented fcntl command: 1036 2024-06-03T09:02:02.421 INFO:tasks.ceph.mon.c.smithi088.stderr:==00:00:26:57.747 34749== Warning: unimplemented fcntl command: 1036 2024-06-03T09:02:12.560 INFO:tasks.ceph.mon.b.smithi192.stderr:==00:00:27:07.863 34228== Warning: unimplemented fcntl command: 1036 2024-06-03T09:02:12.561 INFO:tasks.ceph.mon.a.smithi088.stderr:==00:00:27:07.887 34743== Warning: unimplemented fcntl command: 1036 2024-06-03T09:02:12.562 INFO:tasks.ceph.mon.b.smithi192.stderr:==00:00:27:07.864 34228== Warning: unimplemented fcntl command: 1036 2024-06-03T09:02:12.569 INFO:tasks.ceph.mon.a.smithi088.stderr:==00:00:27:07.895 34743== Warning: unimplemented fcntl command: 1036 2024-06-03T09:02:12.597 INFO:tasks.ceph.mon.b.smithi192.stderr:==00:00:27:07.899 34228== Warning: unimplemented fcntl command: 1036 2024-06-03T09:02:12.624 INFO:tasks.ceph.mon.a.smithi088.stderr:==00:00:27:07.950 34743== Warning: unimplemented fcntl command: 1036 2024-06-03T09:02:12.639 INFO:tasks.ceph.mon.c.smithi088.stderr:==00:00:27:07.965 34749== Warning: unimplemented fcntl command: 1036 2024-06-03T09:02:12.640 INFO:tasks.ceph.mon.c.smithi088.stderr:==00:00:27:07.967 34749== Warning: unimplemented fcntl command: 1036 2024-06-03T09:02:12.671 INFO:tasks.ceph.mon.c.smithi088.stderr:==00:00:27:07.997 34749== Warning: unimplemented fcntl command: 1036 2024-06-03T09:02:13.419 INFO:tasks.ceph.mon.b.smithi192.stderr:==00:00:27:08.722 34228== Warning: unimplemented fcntl command: 1036 2024-06-03T09:02:13.422 INFO:tasks.ceph.mon.c.smithi088.stderr:==00:00:27:08.748 34749== Warning: unimplemented fcntl command: 1036 2024-06-03T09:02:13.423 INFO:tasks.ceph.mon.b.smithi192.stderr:==00:00:27:08.726 34228== Warning: unimplemented fcntl command: 1036 2024-06-03T09:02:13.423 INFO:tasks.ceph.mon.c.smithi088.stderr:==00:00:27:08.749 34749== Warning: unimplemented fcntl command: 1036 2024-06-03T09:02:13.450 INFO:tasks.ceph.mon.c.smithi088.stderr:==00:00:27:08.776 34749== Warning: unimplemented fcntl command: 1036 2024-06-03T09:02:13.505 INFO:tasks.ceph.mon.b.smithi192.stderr:==00:00:27:08.807 34228== Warning: unimplemented fcntl command: 1036 2024-06-03T09:02:13.609 INFO:tasks.ceph.mon.a.smithi088.stderr:==00:00:27:08.935 34743== Warning: unimplemented fcntl command: 1036 2024-06-03T09:02:13.610 INFO:tasks.ceph.mon.a.smithi088.stderr:==00:00:27:08.936 34743== Warning: unimplemented fcntl command: 1036 2024-06-03T09:02:13.626 INFO:tasks.ceph.mon.a.smithi088.stderr:==00:00:27:08.952 34743== Warning: unimplemented fcntl command: 1036 2024-06-03T09:02:17.516 INFO:tasks.ceph.mon.a.smithi088.stderr:==00:00:27:12.841 34743== Warning: unimplemented fcntl command: 1036 2024-06-03T09:02:17.519 INFO:tasks.ceph.mon.a.smithi088.stderr:==00:00:27:12.845 34743== Warning: unimplemented fcntl command: 1036 2024-06-03T09:02:17.532 INFO:tasks.ceph.mon.a.smithi088.stderr:==00:00:27:12.858 34743== Warning: unimplemented fcntl command: 1036 2024-06-03T09:02:17.568 INFO:tasks.ceph.mon.b.smithi192.stderr:==00:00:27:12.871 34228== Warning: unimplemented fcntl command: 1036 2024-06-03T09:02:17.569 INFO:tasks.ceph.mon.c.smithi088.stderr:==00:00:27:12.894 34749== Warning: unimplemented fcntl command: 1036 2024-06-03T09:02:17.572 INFO:tasks.ceph.mon.b.smithi192.stderr:==00:00:27:12.875 34228== Warning: unimplemented fcntl command: 1036 2024-06-03T09:02:17.573 INFO:tasks.ceph.mon.c.smithi088.stderr:==00:00:27:12.899 34749== Warning: unimplemented fcntl command: 1036 2024-06-03T09:02:17.587 INFO:tasks.ceph.mon.b.smithi192.stderr:==00:00:27:12.890 34228== Warning: unimplemented fcntl command: 1036 2024-06-03T09:02:17.587 INFO:tasks.ceph.mon.c.smithi088.stderr:==00:00:27:12.914 34749== Warning: unimplemented fcntl command: 1036 2024-06-03T09:02:30.701 INFO:tasks.workunit.client.0.smithi088.stdout: api_tier_pp: [ OK ] LibRadosTwoPoolsPP.PropagateBaseTierError (12876 ms) 2024-06-03T09:02:30.701 INFO:tasks.workunit.client.0.smithi088.stdout: api_tier_pp: [ RUN ] LibRadosTwoPoolsPP.HelloWriteReturn 2024-06-03T09:02:30.701 INFO:tasks.workunit.client.0.smithi088.stdout: api_tier_pp: 00000000 79 6f 75 20 6d 69 67 68 74 20 73 65 65 20 74 68 |you might see th| 2024-06-03T09:02:30.701 INFO:tasks.workunit.client.0.smithi088.stdout: api_tier_pp: 00000010 69 73 |is| 2024-06-03T09:02:30.701 INFO:tasks.workunit.client.0.smithi088.stdout: api_tier_pp: 00000012 2024-06-03T09:02:30.701 INFO:tasks.workunit.client.0.smithi088.stdout: api_tier_pp: [ OK ] LibRadosTwoPoolsPP.HelloWriteReturn (12890 ms) 2024-06-03T09:02:30.701 INFO:tasks.workunit.client.0.smithi088.stdout: api_tier_pp: [ RUN ] LibRadosTwoPoolsPP.TierFlushDuringUnsetDedupTier 2024-06-03T09:02:30.701 INFO:tasks.workunit.client.0.smithi088.stdout: api_tier_pp: require_osd_release = reef 2024-06-03T09:02:30.702 INFO:tasks.workunit.client.0.smithi088.stdout: api_tier_pp: [ OK ] LibRadosTwoPoolsPP.TierFlushDuringUnsetDedupTier (7424 ms) 2024-06-03T09:02:30.702 INFO:tasks.workunit.client.0.smithi088.stdout: api_tier_pp: [----------] 48 tests from LibRadosTwoPoolsPP (1106063 ms total) 2024-06-03T09:02:30.702 INFO:tasks.workunit.client.0.smithi088.stdout: api_tier_pp: 2024-06-03T09:02:30.702 INFO:tasks.workunit.client.0.smithi088.stdout: api_tier_pp: [----------] 4 tests from LibRadosTierECPP 2024-06-03T09:02:30.702 INFO:tasks.workunit.client.0.smithi088.stdout: api_tier_pp: [ RUN ] LibRadosTierECPP.Dirty 2024-06-03T09:02:30.702 INFO:tasks.workunit.client.0.smithi088.stdout: api_tier_pp: [ OK ] LibRadosTierECPP.Dirty (1556 ms) 2024-06-03T09:02:30.702 INFO:tasks.workunit.client.0.smithi088.stdout: api_tier_pp: [ RUN ] LibRadosTierECPP.FlushWriteRaces 2024-06-03T09:02:30.702 INFO:tasks.workunit.client.0.smithi088.stdout: api_tier_pp: [ OK ] LibRadosTierECPP.FlushWriteRaces (12115 ms) 2024-06-03T09:02:30.702 INFO:tasks.workunit.client.0.smithi088.stdout: api_tier_pp: [ RUN ] LibRadosTierECPP.CallForcesPromote 2024-06-03T09:02:30.702 INFO:tasks.workunit.client.0.smithi088.stdout: api_tier_pp: [ OK ] LibRadosTierECPP.CallForcesPromote (18232 ms) 2024-06-03T09:02:30.702 INFO:tasks.workunit.client.0.smithi088.stdout: api_tier_pp: [ RUN ] LibRadosTierECPP.HitSetNone 2024-06-03T09:02:30.703 INFO:tasks.workunit.client.0.smithi088.stdout: api_tier_pp: [ OK ] LibRadosTierECPP.HitSetNone (18 ms) 2024-06-03T09:02:30.703 INFO:tasks.workunit.client.0.smithi088.stdout: api_tier_pp: [----------] 4 tests from LibRadosTierECPP (31921 ms total) 2024-06-03T09:02:30.703 INFO:tasks.workunit.client.0.smithi088.stdout: api_tier_pp: 2024-06-03T09:02:30.703 INFO:tasks.workunit.client.0.smithi088.stdout: api_tier_pp: [----------] 22 tests from LibRadosTwoPoolsECPP 2024-06-03T09:02:30.703 INFO:tasks.workunit.client.0.smithi088.stdout: api_tier_pp: [ RUN ] LibRadosTwoPoolsECPP.Overlay 2024-06-03T09:02:30.703 INFO:tasks.workunit.client.0.smithi088.stdout: api_tier_pp: [ OK ] LibRadosTwoPoolsECPP.Overlay (7848 ms) 2024-06-03T09:02:30.703 INFO:tasks.workunit.client.0.smithi088.stdout: api_tier_pp: [ RUN ] LibRadosTwoPoolsECPP.Promote 2024-06-03T09:02:30.703 INFO:tasks.workunit.client.0.smithi088.stdout: api_tier_pp: [ OK ] LibRadosTwoPoolsECPP.Promote (9540 ms) 2024-06-03T09:02:30.703 INFO:tasks.workunit.client.0.smithi088.stdout: api_tier_pp: [ RUN ] LibRadosTwoPoolsECPP.PromoteSnap 2024-06-03T09:02:30.703 INFO:tasks.workunit.client.0.smithi088.stdout: api_tier_pp: waiting for scrub... 2024-06-03T09:02:30.703 INFO:tasks.workunit.client.0.smithi088.stdout: api_tier_pp: done waiting 2024-06-03T09:02:30.703 INFO:tasks.workunit.client.0.smithi088.stdout: api_tier_pp: [ OK ] LibRadosTwoPoolsECPP.PromoteSnap (26673 ms) 2024-06-03T09:02:30.703 INFO:tasks.workunit.client.0.smithi088.stdout: api_tier_pp: [ RUN ] LibRadosTwoPoolsECPP.PromoteSnapTrimRace 2024-06-03T09:02:30.704 INFO:tasks.workunit.client.0.smithi088.stdout: api_tier_pp: [ OK ] LibRadosTwoPoolsECPP.PromoteSnapTrimRace (11190 ms) 2024-06-03T09:02:30.704 INFO:tasks.workunit.client.0.smithi088.stdout: api_tier_pp: [ RUN ] LibRadosTwoPoolsECPP.Whiteout 2024-06-03T09:02:30.704 INFO:tasks.workunit.client.0.smithi088.stdout: api_tier_pp: [ OK ] LibRadosTwoPoolsECPP.Whiteout (10029 ms) 2024-06-03T09:02:30.704 INFO:tasks.workunit.client.0.smithi088.stdout: api_tier_pp: [ RUN ] LibRadosTwoPoolsECPP.Evict 2024-06-03T09:02:30.704 INFO:tasks.workunit.client.0.smithi088.stdout: api_tier_pp: [ OK ] LibRadosTwoPoolsECPP.Evict (10017 ms) 2024-06-03T09:02:30.704 INFO:tasks.workunit.client.0.smithi088.stdout: api_tier_pp: [ RUN ] LibRadosTwoPoolsECPP.EvictSnap 2024-06-03T09:02:30.704 INFO:tasks.workunit.client.0.smithi088.stdout: api_tier_pp: [ OK ] LibRadosTwoPoolsECPP.EvictSnap (11226 ms) 2024-06-03T09:02:30.704 INFO:tasks.workunit.client.0.smithi088.stdout: api_tier_pp: [ RUN ] LibRadosTwoPoolsECPP.TryFlush 2024-06-03T09:02:30.704 INFO:tasks.workunit.client.0.smithi088.stdout: api_tier_pp: [ OK ] LibRadosTwoPoolsECPP.TryFlush (36977 ms) 2024-06-03T09:02:30.704 INFO:tasks.workunit.client.0.smithi088.stdout: api_tier_pp: [ RUN ] LibRadosTwoPoolsECPP.FailedFlush 2024-06-03T09:02:30.704 INFO:tasks.workunit.client.0.smithi088.stdout: api_tier_pp: [ OK ] LibRadosTwoPoolsECPP.FailedFlush (14314 ms) 2024-06-03T09:02:30.704 INFO:tasks.workunit.client.0.smithi088.stdout: api_tier_pp: [ RUN ] LibRadosTwoPoolsECPP.Flush 2024-06-03T09:02:30.704 INFO:tasks.workunit.client.0.smithi088.stdout: api_tier_pp: [ OK ] LibRadosTwoPoolsECPP.Flush (10026 ms) 2024-06-03T09:02:30.705 INFO:tasks.workunit.client.0.smithi088.stdout: api_tier_pp: [ RUN ] LibRadosTwoPoolsECPP.FlushSnap 2024-06-03T09:02:30.705 INFO:tasks.workunit.client.0.smithi088.stdout: api_tier_pp: [ OK ] LibRadosTwoPoolsECPP.FlushSnap (15395 ms) 2024-06-03T09:02:30.705 INFO:tasks.workunit.client.0.smithi088.stdout: api_tier_pp: [ RUN ] LibRadosTwoPoolsECPP.FlushTryFlushRaces 2024-06-03T09:02:30.705 INFO:tasks.workunit.client.0.smithi088.stdout: api_tier_pp: [ OK ] LibRadosTwoPoolsECPP.FlushTryFlushRaces (7967 ms) 2024-06-03T09:02:30.705 INFO:tasks.workunit.client.0.smithi088.stdout: api_tier_pp: [ RUN ] LibRadosTwoPoolsECPP.TryFlushReadRace 2024-06-03T09:02:30.705 INFO:tasks.workunit.client.0.smithi088.stdout: api_tier_pp: [ OK ] LibRadosTwoPoolsECPP.TryFlushReadRace (9584 ms) 2024-06-03T09:02:30.705 INFO:tasks.workunit.client.0.smithi088.stdout: api_tier_pp: [ RUN ] LibRadosTwoPoolsECPP.HitSetRead 2024-06-03T09:02:30.705 INFO:tasks.workunit.client.0.smithi088.stdout: api_tier_pp: hmm, no HitSet yet 2024-06-03T09:02:30.705 INFO:tasks.workunit.client.0.smithi088.stdout: api_tier_pp: ok, hit_set contains 252:602f83fe:::foo:head 2024-06-03T09:02:30.705 INFO:tasks.workunit.client.0.smithi088.stdout: api_tier_pp: [ OK ] LibRadosTwoPoolsECPP.HitSetRead (9438 ms) 2024-06-03T09:02:32.704 INFO:tasks.ceph.mon.a.smithi088.stderr:==00:00:27:28.029 34743== Warning: unimplemented fcntl command: 1036 2024-06-03T09:02:32.708 INFO:tasks.ceph.mon.a.smithi088.stderr:==00:00:27:28.034 34743== Warning: unimplemented fcntl command: 1036 2024-06-03T09:02:32.742 INFO:tasks.ceph.mon.a.smithi088.stderr:==00:00:27:28.068 34743== Warning: unimplemented fcntl command: 1036 2024-06-03T09:02:32.772 INFO:tasks.ceph.mon.b.smithi192.stderr:==00:00:27:28.075 34228== Warning: unimplemented fcntl command: 1036 2024-06-03T09:02:32.774 INFO:tasks.ceph.mon.b.smithi192.stderr:==00:00:27:28.077 34228== Warning: unimplemented fcntl command: 1036 2024-06-03T09:02:32.778 INFO:tasks.ceph.mon.c.smithi088.stderr:==00:00:27:28.104 34749== Warning: unimplemented fcntl command: 1036 2024-06-03T09:02:32.783 INFO:tasks.ceph.mon.c.smithi088.stderr:==00:00:27:28.109 34749== Warning: unimplemented fcntl command: 1036 2024-06-03T09:02:32.809 INFO:tasks.ceph.mon.b.smithi192.stderr:==00:00:27:28.112 34228== Warning: unimplemented fcntl command: 1036 2024-06-03T09:02:32.850 INFO:tasks.ceph.mon.c.smithi088.stderr:==00:00:27:28.176 34749== Warning: unimplemented fcntl command: 1036 2024-06-03T09:02:47.526 INFO:tasks.ceph.mon.a.smithi088.stderr:==00:00:27:42.852 34743== Warning: unimplemented fcntl command: 1036 2024-06-03T09:02:47.577 INFO:tasks.ceph.mon.b.smithi192.stderr:==00:00:27:42.880 34228== Warning: unimplemented fcntl command: 1036 2024-06-03T09:02:47.578 INFO:tasks.ceph.mon.a.smithi088.stderr:==00:00:27:42.904 34743== Warning: unimplemented fcntl command: 1036 2024-06-03T09:02:47.580 INFO:tasks.ceph.mon.b.smithi192.stderr:==00:00:27:42.883 34228== Warning: unimplemented fcntl command: 1036 2024-06-03T09:02:47.599 INFO:tasks.ceph.mon.b.smithi192.stderr:==00:00:27:42.901 34228== Warning: unimplemented fcntl command: 1036 2024-06-03T09:02:47.599 INFO:tasks.ceph.mon.a.smithi088.stderr:==00:00:27:42.925 34743== Warning: unimplemented fcntl command: 1036 2024-06-03T09:02:47.624 INFO:tasks.ceph.mon.c.smithi088.stderr:==00:00:27:42.950 34749== Warning: unimplemented fcntl command: 1036 2024-06-03T09:02:47.629 INFO:tasks.ceph.mon.c.smithi088.stderr:==00:00:27:42.955 34749== Warning: unimplemented fcntl command: 1036 2024-06-03T09:02:47.657 INFO:tasks.ceph.mon.c.smithi088.stderr:==00:00:27:42.983 34749== Warning: unimplemented fcntl command: 1036 2024-06-03T09:02:52.665 INFO:tasks.ceph.mon.b.smithi192.stderr:==00:00:27:47.967 34228== Warning: unimplemented fcntl command: 1036 2024-06-03T09:02:52.665 INFO:tasks.ceph.mon.b.smithi192.stderr:==00:00:27:47.968 34228== Warning: unimplemented fcntl command: 1036 2024-06-03T09:02:52.671 INFO:tasks.ceph.mon.a.smithi088.stderr:==00:00:27:47.997 34743== Warning: unimplemented fcntl command: 1036 2024-06-03T09:02:52.675 INFO:tasks.ceph.mon.a.smithi088.stderr:==00:00:27:48.001 34743== Warning: unimplemented fcntl command: 1036 2024-06-03T09:02:52.678 INFO:tasks.ceph.mon.b.smithi192.stderr:==00:00:27:47.980 34228== Warning: unimplemented fcntl command: 1036 2024-06-03T09:02:52.693 INFO:tasks.ceph.mon.a.smithi088.stderr:==00:00:27:48.019 34743== Warning: unimplemented fcntl command: 1036 2024-06-03T09:02:52.731 INFO:tasks.ceph.mon.c.smithi088.stderr:==00:00:27:48.057 34749== Warning: unimplemented fcntl command: 1036 2024-06-03T09:02:52.736 INFO:tasks.ceph.mon.c.smithi088.stderr:==00:00:27:48.062 34749== Warning: unimplemented fcntl command: 1036 2024-06-03T09:02:52.753 INFO:tasks.ceph.mon.c.smithi088.stderr:==00:00:27:48.079 34749== Warning: unimplemented fcntl command: 1036 2024-06-03T09:03:02.691 INFO:tasks.ceph.mon.a.smithi088.stderr:==00:00:27:58.017 34743== Warning: unimplemented fcntl command: 1036 2024-06-03T09:03:02.742 INFO:tasks.ceph.mon.a.smithi088.stderr:==00:00:27:58.068 34743== Warning: unimplemented fcntl command: 1036 2024-06-03T09:03:02.779 INFO:tasks.ceph.mon.a.smithi088.stderr:==00:00:27:58.105 34743== Warning: unimplemented fcntl command: 1036 2024-06-03T09:03:02.790 INFO:tasks.ceph.mon.b.smithi192.stderr:==00:00:27:58.093 34228== Warning: unimplemented fcntl command: 1036 2024-06-03T09:03:02.792 INFO:tasks.ceph.mon.b.smithi192.stderr:==00:00:27:58.095 34228== Warning: unimplemented fcntl command: 1036 2024-06-03T09:03:02.817 INFO:tasks.ceph.mon.b.smithi192.stderr:==00:00:27:58.120 34228== Warning: unimplemented fcntl command: 1036 2024-06-03T09:03:02.826 INFO:tasks.ceph.mon.c.smithi088.stderr:==00:00:27:58.152 34749== Warning: unimplemented fcntl command: 1036 2024-06-03T09:03:02.831 INFO:tasks.ceph.mon.c.smithi088.stderr:==00:00:27:58.157 34749== Warning: unimplemented fcntl command: 1036 2024-06-03T09:03:02.865 INFO:tasks.ceph.mon.c.smithi088.stderr:==00:00:27:58.191 34749== Warning: unimplemented fcntl command: 1036 2024-06-03T09:03:12.642 INFO:tasks.ceph.mon.a.smithi088.stderr:==00:00:28:07.967 34743== Warning: unimplemented fcntl command: 1036 2024-06-03T09:03:12.649 INFO:tasks.ceph.mon.a.smithi088.stderr:==00:00:28:07.975 34743== Warning: unimplemented fcntl command: 1036 2024-06-03T09:03:12.674 INFO:tasks.ceph.mon.a.smithi088.stderr:==00:00:28:08.000 34743== Warning: unimplemented fcntl command: 1036 2024-06-03T09:03:12.692 INFO:tasks.ceph.mon.b.smithi192.stderr:==00:00:28:07.995 34228== Warning: unimplemented fcntl command: 1036 2024-06-03T09:03:12.694 INFO:tasks.ceph.mon.b.smithi192.stderr:==00:00:28:07.996 34228== Warning: unimplemented fcntl command: 1036 2024-06-03T09:03:12.697 INFO:tasks.ceph.mon.c.smithi088.stderr:==00:00:28:08.023 34749== Warning: unimplemented fcntl command: 1036 2024-06-03T09:03:12.703 INFO:tasks.ceph.mon.c.smithi088.stderr:==00:00:28:08.029 34749== Warning: unimplemented fcntl command: 1036 2024-06-03T09:03:12.718 INFO:tasks.ceph.mon.b.smithi192.stderr:==00:00:28:08.020 34228== Warning: unimplemented fcntl command: 1036 2024-06-03T09:03:12.729 INFO:tasks.ceph.mon.c.smithi088.stderr:==00:00:28:08.055 34749== Warning: unimplemented fcntl command: 1036 2024-06-03T09:03:17.657 INFO:tasks.ceph.mon.a.smithi088.stderr:==00:00:28:12.983 34743== Warning: unimplemented fcntl command: 1036 2024-06-03T09:03:17.659 INFO:tasks.ceph.mon.a.smithi088.stderr:==00:00:28:12.985 34743== Warning: unimplemented fcntl command: 1036 2024-06-03T09:03:17.679 INFO:tasks.ceph.mon.a.smithi088.stderr:==00:00:28:13.005 34743== Warning: unimplemented fcntl command: 1036 2024-06-03T09:03:17.704 INFO:tasks.ceph.mon.b.smithi192.stderr:==00:00:28:13.007 34228== Warning: unimplemented fcntl command: 1036 2024-06-03T09:03:17.705 INFO:tasks.ceph.mon.b.smithi192.stderr:==00:00:28:13.008 34228== Warning: unimplemented fcntl command: 1036 2024-06-03T09:03:17.714 INFO:tasks.ceph.mon.c.smithi088.stderr:==00:00:28:13.040 34749== Warning: unimplemented fcntl command: 1036 2024-06-03T09:03:17.716 INFO:tasks.ceph.mon.c.smithi088.stderr:==00:00:28:13.042 34749== Warning: unimplemented fcntl command: 1036 2024-06-03T09:03:17.718 INFO:tasks.ceph.mon.b.smithi192.stderr:==00:00:28:13.021 34228== Warning: unimplemented fcntl command: 1036 2024-06-03T09:03:17.729 INFO:tasks.ceph.mon.c.smithi088.stderr:==00:00:28:13.055 34749== Warning: unimplemented fcntl command: 1036 2024-06-03T09:03:32.715 INFO:tasks.ceph.mon.a.smithi088.stderr:==00:00:28:28.041 34743== Warning: unimplemented fcntl command: 1036 2024-06-03T09:03:32.719 INFO:tasks.ceph.mon.a.smithi088.stderr:==00:00:28:28.045 34743== Warning: unimplemented fcntl command: 1036 2024-06-03T09:03:32.742 INFO:tasks.ceph.mon.a.smithi088.stderr:==00:00:28:28.068 34743== Warning: unimplemented fcntl command: 1036 2024-06-03T09:03:32.768 INFO:tasks.ceph.mon.b.smithi192.stderr:==00:00:28:28.070 34228== Warning: unimplemented fcntl command: 1036 2024-06-03T09:03:32.770 INFO:tasks.ceph.mon.b.smithi192.stderr:==00:00:28:28.073 34228== Warning: unimplemented fcntl command: 1036 2024-06-03T09:03:32.786 INFO:tasks.ceph.mon.c.smithi088.stderr:==00:00:28:28.112 34749== Warning: unimplemented fcntl command: 1036 2024-06-03T09:03:32.787 INFO:tasks.ceph.mon.c.smithi088.stderr:==00:00:28:28.113 34749== Warning: unimplemented fcntl command: 1036 2024-06-03T09:03:32.788 INFO:tasks.ceph.mon.b.smithi192.stderr:==00:00:28:28.091 34228== Warning: unimplemented fcntl command: 1036 2024-06-03T09:03:32.806 INFO:tasks.ceph.mon.c.smithi088.stderr:==00:00:28:28.132 34749== Warning: unimplemented fcntl command: 1036 2024-06-03T09:03:33.448 INFO:tasks.ceph.mon.b.smithi192.stderr:==00:00:28:28.751 34228== Warning: unimplemented fcntl command: 1036 2024-06-03T09:03:33.449 INFO:tasks.ceph.mon.b.smithi192.stderr:==00:00:28:28.752 34228== Warning: unimplemented fcntl command: 1036 2024-06-03T09:03:33.458 INFO:tasks.ceph.mon.b.smithi192.stderr:==00:00:28:28.761 34228== Warning: unimplemented fcntl command: 1036 2024-06-03T09:03:33.472 INFO:tasks.ceph.mon.c.smithi088.stderr:==00:00:28:28.798 34749== Warning: unimplemented fcntl command: 1036 2024-06-03T09:03:33.473 INFO:tasks.ceph.mon.c.smithi088.stderr:==00:00:28:28.799 34749== Warning: unimplemented fcntl command: 1036 2024-06-03T09:03:33.482 INFO:tasks.ceph.mon.c.smithi088.stderr:==00:00:28:28.808 34749== Warning: unimplemented fcntl command: 1036 2024-06-03T09:03:33.514 INFO:tasks.ceph.mon.a.smithi088.stderr:==00:00:28:28.840 34743== Warning: unimplemented fcntl command: 1036 2024-06-03T09:03:33.515 INFO:tasks.ceph.mon.a.smithi088.stderr:==00:00:28:28.841 34743== Warning: unimplemented fcntl command: 1036 2024-06-03T09:03:33.529 INFO:tasks.ceph.mon.a.smithi088.stderr:==00:00:28:28.855 34743== Warning: unimplemented fcntl command: 1036 2024-06-03T09:03:47.766 INFO:tasks.ceph.mon.a.smithi088.stderr:==00:00:28:43.091 34743== Warning: unimplemented fcntl command: 1036 2024-06-03T09:03:47.769 INFO:tasks.ceph.mon.a.smithi088.stderr:==00:00:28:43.096 34743== Warning: unimplemented fcntl command: 1036 2024-06-03T09:03:47.791 INFO:tasks.ceph.mon.a.smithi088.stderr:==00:00:28:43.117 34743== Warning: unimplemented fcntl command: 1036 2024-06-03T09:03:47.818 INFO:tasks.ceph.mon.b.smithi192.stderr:==00:00:28:43.121 34228== Warning: unimplemented fcntl command: 1036 2024-06-03T09:03:47.819 INFO:tasks.ceph.mon.b.smithi192.stderr:==00:00:28:43.122 34228== Warning: unimplemented fcntl command: 1036 2024-06-03T09:03:47.820 INFO:tasks.ceph.mon.c.smithi088.stderr:==00:00:28:43.146 34749== Warning: unimplemented fcntl command: 1036 2024-06-03T09:03:47.821 INFO:tasks.ceph.mon.c.smithi088.stderr:==00:00:28:43.147 34749== Warning: unimplemented fcntl command: 1036 2024-06-03T09:03:47.837 INFO:tasks.ceph.mon.b.smithi192.stderr:==00:00:28:43.140 34228== Warning: unimplemented fcntl command: 1036 2024-06-03T09:03:47.849 INFO:tasks.ceph.mon.c.smithi088.stderr:==00:00:28:43.175 34749== Warning: unimplemented fcntl command: 1036 2024-06-03T09:03:52.780 INFO:tasks.ceph.mon.a.smithi088.stderr:==00:00:28:48.106 34743== Warning: unimplemented fcntl command: 1036 2024-06-03T09:03:52.782 INFO:tasks.ceph.mon.a.smithi088.stderr:==00:00:28:48.109 34743== Warning: unimplemented fcntl command: 1036 2024-06-03T09:03:52.796 INFO:tasks.ceph.mon.a.smithi088.stderr:==00:00:28:48.122 34743== Warning: unimplemented fcntl command: 1036 2024-06-03T09:03:52.832 INFO:tasks.ceph.mon.c.smithi088.stderr:==00:00:28:48.158 34749== Warning: unimplemented fcntl command: 1036 2024-06-03T09:03:52.833 INFO:tasks.ceph.mon.b.smithi192.stderr:==00:00:28:48.135 34228== Warning: unimplemented fcntl command: 1036 2024-06-03T09:03:52.833 INFO:tasks.ceph.mon.c.smithi088.stderr:==00:00:28:48.160 34749== Warning: unimplemented fcntl command: 1036 2024-06-03T09:03:52.846 INFO:tasks.ceph.mon.c.smithi088.stderr:==00:00:28:48.172 34749== Warning: unimplemented fcntl command: 1036 2024-06-03T09:03:52.847 INFO:tasks.ceph.mon.b.smithi192.stderr:==00:00:28:48.150 34228== Warning: unimplemented fcntl command: 1036 2024-06-03T09:03:52.857 INFO:tasks.ceph.mon.b.smithi192.stderr:==00:00:28:48.160 34228== Warning: unimplemented fcntl command: 1036 2024-06-03T09:04:08.013 INFO:tasks.ceph.mon.a.smithi088.stderr:==00:00:29:03.338 34743== Warning: unimplemented fcntl command: 1036 2024-06-03T09:04:08.017 INFO:tasks.ceph.mon.a.smithi088.stderr:==00:00:29:03.343 34743== Warning: unimplemented fcntl command: 1036 2024-06-03T09:04:08.046 INFO:tasks.ceph.mon.a.smithi088.stderr:==00:00:29:03.372 34743== Warning: unimplemented fcntl command: 1036 2024-06-03T09:04:08.082 INFO:tasks.ceph.mon.c.smithi088.stderr:==00:00:29:03.408 34749== Warning: unimplemented fcntl command: 1036 2024-06-03T09:04:08.085 INFO:tasks.ceph.mon.b.smithi192.stderr:==00:00:29:03.388 34228== Warning: unimplemented fcntl command: 1036 2024-06-03T09:04:08.087 INFO:tasks.ceph.mon.b.smithi192.stderr:==00:00:29:03.390 34228== Warning: unimplemented fcntl command: 1036 2024-06-03T09:04:08.087 INFO:tasks.ceph.mon.c.smithi088.stderr:==00:00:29:03.413 34749== Warning: unimplemented fcntl command: 1036 2024-06-03T09:04:08.107 INFO:tasks.ceph.mon.b.smithi192.stderr:==00:00:29:03.410 34228== Warning: unimplemented fcntl command: 1036 2024-06-03T09:04:08.110 INFO:tasks.ceph.mon.c.smithi088.stderr:==00:00:29:03.436 34749== Warning: unimplemented fcntl command: 1036 2024-06-03T09:04:13.041 INFO:tasks.ceph.mon.a.smithi088.stderr:==00:00:29:08.367 34743== Warning: unimplemented fcntl command: 1036 2024-06-03T09:04:13.043 INFO:tasks.ceph.mon.a.smithi088.stderr:==00:00:29:08.369 34743== Warning: unimplemented fcntl command: 1036 2024-06-03T09:04:13.056 INFO:tasks.ceph.mon.a.smithi088.stderr:==00:00:29:08.382 34743== Warning: unimplemented fcntl command: 1036 2024-06-03T09:04:13.092 INFO:tasks.ceph.mon.b.smithi192.stderr:==00:00:29:08.395 34228== Warning: unimplemented fcntl command: 1036 2024-06-03T09:04:13.095 INFO:tasks.ceph.mon.c.smithi088.stderr:==00:00:29:08.421 34749== Warning: unimplemented fcntl command: 1036 2024-06-03T09:04:13.095 INFO:tasks.ceph.mon.b.smithi192.stderr:==00:00:29:08.398 34228== Warning: unimplemented fcntl command: 1036 2024-06-03T09:04:13.096 INFO:tasks.ceph.mon.c.smithi088.stderr:==00:00:29:08.422 34749== Warning: unimplemented fcntl command: 1036 2024-06-03T09:04:13.109 INFO:tasks.ceph.mon.b.smithi192.stderr:==00:00:29:08.411 34228== Warning: unimplemented fcntl command: 1036 2024-06-03T09:04:13.109 INFO:tasks.ceph.mon.c.smithi088.stderr:==00:00:29:08.436 34749== Warning: unimplemented fcntl command: 1036 2024-06-03T09:04:22.902 INFO:tasks.ceph.mon.a.smithi088.stderr:==00:00:29:18.227 34743== Warning: unimplemented fcntl command: 1036 2024-06-03T09:04:22.958 INFO:tasks.ceph.mon.a.smithi088.stderr:==00:00:29:18.283 34743== Warning: unimplemented fcntl command: 1036 2024-06-03T09:04:22.975 INFO:tasks.ceph.mon.a.smithi088.stderr:==00:00:29:18.301 34743== Warning: unimplemented fcntl command: 1036 2024-06-03T09:04:23.004 INFO:tasks.ceph.mon.b.smithi192.stderr:==00:00:29:18.307 34228== Warning: unimplemented fcntl command: 1036 2024-06-03T09:04:23.005 INFO:tasks.ceph.mon.b.smithi192.stderr:==00:00:29:18.308 34228== Warning: unimplemented fcntl command: 1036 2024-06-03T09:04:23.010 INFO:tasks.ceph.mon.c.smithi088.stderr:==00:00:29:18.336 34749== Warning: unimplemented fcntl command: 1036 2024-06-03T09:04:23.011 INFO:tasks.ceph.mon.c.smithi088.stderr:==00:00:29:18.337 34749== Warning: unimplemented fcntl command: 1036 2024-06-03T09:04:23.018 INFO:tasks.ceph.mon.b.smithi192.stderr:==00:00:29:18.321 34228== Warning: unimplemented fcntl command: 1036 2024-06-03T09:04:23.030 INFO:tasks.ceph.mon.c.smithi088.stderr:==00:00:29:18.356 34749== Warning: unimplemented fcntl command: 1036 2024-06-03T09:04:32.938 INFO:tasks.ceph.mon.a.smithi088.stderr:==00:00:29:28.264 34743== Warning: unimplemented fcntl command: 1036 2024-06-03T09:04:32.990 INFO:tasks.ceph.mon.a.smithi088.stderr:==00:00:29:28.315 34743== Warning: unimplemented fcntl command: 1036 2024-06-03T09:04:32.994 INFO:tasks.ceph.mon.b.smithi192.stderr:==00:00:29:28.296 34228== Warning: unimplemented fcntl command: 1036 2024-06-03T09:04:32.997 INFO:tasks.ceph.mon.b.smithi192.stderr:==00:00:29:28.300 34228== Warning: unimplemented fcntl command: 1036 2024-06-03T09:04:33.016 INFO:tasks.ceph.mon.a.smithi088.stderr:==00:00:29:28.342 34743== Warning: unimplemented fcntl command: 1036 2024-06-03T09:04:33.019 INFO:tasks.ceph.mon.b.smithi192.stderr:==00:00:29:28.322 34228== Warning: unimplemented fcntl command: 1036 2024-06-03T09:04:33.044 INFO:tasks.ceph.mon.c.smithi088.stderr:==00:00:29:28.370 34749== Warning: unimplemented fcntl command: 1036 2024-06-03T09:04:33.045 INFO:tasks.ceph.mon.c.smithi088.stderr:==00:00:29:28.371 34749== Warning: unimplemented fcntl command: 1036 2024-06-03T09:04:33.068 INFO:tasks.ceph.mon.c.smithi088.stderr:==00:00:29:28.393 34749== Warning: unimplemented fcntl command: 1036 2024-06-03T09:04:37.272 INFO:tasks.ceph.mon.a.smithi088.stderr:==00:00:29:32.598 34743== Warning: unimplemented fcntl command: 1036 2024-06-03T09:04:37.277 INFO:tasks.ceph.mon.a.smithi088.stderr:==00:00:29:32.603 34743== Warning: unimplemented fcntl command: 1036 2024-06-03T09:04:37.289 INFO:tasks.ceph.mon.c.smithi088.stderr:==00:00:29:32.615 34749== Warning: unimplemented fcntl command: 1036 2024-06-03T09:04:37.291 INFO:tasks.ceph.mon.c.smithi088.stderr:==00:00:29:32.617 34749== Warning: unimplemented fcntl command: 1036 2024-06-03T09:04:37.346 INFO:tasks.ceph.mon.b.smithi192.stderr:==00:00:29:32.648 34228== Warning: unimplemented fcntl command: 1036 2024-06-03T09:04:37.349 INFO:tasks.ceph.mon.b.smithi192.stderr:==00:00:29:32.652 34228== Warning: unimplemented fcntl command: 1036 2024-06-03T09:04:37.363 INFO:tasks.ceph.mon.c.smithi088.stderr:==00:00:29:32.689 34749== Warning: unimplemented fcntl command: 1036 2024-06-03T09:04:37.374 INFO:tasks.ceph.mon.a.smithi088.stderr:==00:00:29:32.700 34743== Warning: unimplemented fcntl command: 1036 2024-06-03T09:04:37.378 INFO:tasks.ceph.mon.b.smithi192.stderr:==00:00:29:32.680 34228== Warning: unimplemented fcntl command: 1036 2024-06-03T09:04:38.130 INFO:tasks.ceph.mon.c.smithi088.stderr:==00:00:29:33.456 34749== Warning: unimplemented fcntl command: 1036 2024-06-03T09:04:38.131 INFO:tasks.ceph.mon.c.smithi088.stderr:==00:00:29:33.457 34749== Warning: unimplemented fcntl command: 1036 2024-06-03T09:04:38.140 INFO:tasks.ceph.mon.c.smithi088.stderr:==00:00:29:33.466 34749== Warning: unimplemented fcntl command: 1036 2024-06-03T09:04:38.164 INFO:tasks.ceph.mon.b.smithi192.stderr:==00:00:29:33.467 34228== Warning: unimplemented fcntl command: 1036 2024-06-03T09:04:38.165 INFO:tasks.ceph.mon.b.smithi192.stderr:==00:00:29:33.468 34228== Warning: unimplemented fcntl command: 1036 2024-06-03T09:04:38.173 INFO:tasks.ceph.mon.b.smithi192.stderr:==00:00:29:33.476 34228== Warning: unimplemented fcntl command: 1036 2024-06-03T09:04:38.249 INFO:tasks.ceph.mon.a.smithi088.stderr:==00:00:29:33.574 34743== Warning: unimplemented fcntl command: 1036 2024-06-03T09:04:38.250 INFO:tasks.ceph.mon.a.smithi088.stderr:==00:00:29:33.576 34743== Warning: unimplemented fcntl command: 1036 2024-06-03T09:04:38.259 INFO:tasks.ceph.mon.a.smithi088.stderr:==00:00:29:33.585 34743== Warning: unimplemented fcntl command: 1036 2024-06-03T09:04:53.009 INFO:tasks.ceph.mon.a.smithi088.stderr:==00:00:29:48.335 34743== Warning: unimplemented fcntl command: 1036 2024-06-03T09:04:53.059 INFO:tasks.ceph.mon.b.smithi192.stderr:==00:00:29:48.361 34228== Warning: unimplemented fcntl command: 1036 2024-06-03T09:04:53.062 INFO:tasks.ceph.mon.a.smithi088.stderr:==00:00:29:48.388 34743== Warning: unimplemented fcntl command: 1036 2024-06-03T09:04:53.063 INFO:tasks.ceph.mon.b.smithi192.stderr:==00:00:29:48.365 34228== Warning: unimplemented fcntl command: 1036 2024-06-03T09:04:53.081 INFO:tasks.ceph.mon.b.smithi192.stderr:==00:00:29:48.384 34228== Warning: unimplemented fcntl command: 1036 2024-06-03T09:04:53.082 INFO:tasks.ceph.mon.a.smithi088.stderr:==00:00:29:48.408 34743== Warning: unimplemented fcntl command: 1036 2024-06-03T09:04:53.121 INFO:tasks.ceph.mon.c.smithi088.stderr:==00:00:29:48.447 34749== Warning: unimplemented fcntl command: 1036 2024-06-03T09:04:53.122 INFO:tasks.ceph.mon.c.smithi088.stderr:==00:00:29:48.449 34749== Warning: unimplemented fcntl command: 1036 2024-06-03T09:04:53.141 INFO:tasks.ceph.mon.c.smithi088.stderr:==00:00:29:48.467 34749== Warning: unimplemented fcntl command: 1036 2024-06-03T09:04:58.024 INFO:tasks.ceph.mon.a.smithi088.stderr:==00:00:29:53.350 34743== Warning: unimplemented fcntl command: 1036 2024-06-03T09:04:58.071 INFO:tasks.ceph.mon.a.smithi088.stderr:==00:00:29:53.396 34743== Warning: unimplemented fcntl command: 1036 2024-06-03T09:04:58.081 INFO:tasks.ceph.mon.a.smithi088.stderr:==00:00:29:53.407 34743== Warning: unimplemented fcntl command: 1036 2024-06-03T09:04:58.117 INFO:tasks.ceph.mon.b.smithi192.stderr:==00:00:29:53.420 34228== Warning: unimplemented fcntl command: 1036 2024-06-03T09:04:58.120 INFO:tasks.ceph.mon.b.smithi192.stderr:==00:00:29:53.422 34228== Warning: unimplemented fcntl command: 1036 2024-06-03T09:04:58.125 INFO:tasks.ceph.mon.c.smithi088.stderr:==00:00:29:53.451 34749== Warning: unimplemented fcntl command: 1036 2024-06-03T09:04:58.126 INFO:tasks.ceph.mon.c.smithi088.stderr:==00:00:29:53.452 34749== Warning: unimplemented fcntl command: 1036 2024-06-03T09:04:58.129 INFO:tasks.ceph.mon.b.smithi192.stderr:==00:00:29:53.432 34228== Warning: unimplemented fcntl command: 1036 2024-06-03T09:04:58.136 INFO:tasks.ceph.mon.c.smithi088.stderr:==00:00:29:53.462 34749== Warning: unimplemented fcntl command: 1036 2024-06-03T09:05:08.246 INFO:tasks.ceph.osd.3.smithi088.stderr:2024-06-03T09:05:08.245+0000 ea64640 -1 osd.3 998 get_health_metrics reporting 1 slow ops, oldest is osd_op(client.30047.0:3124 238.1s0 238:8fc3729e:test-rados-api-smithi088-40561-153::bar:head [writefull 0~10 in=10b] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e996) 2024-06-03T09:05:09.269 INFO:tasks.ceph.osd.3.smithi088.stderr:2024-06-03T09:05:09.269+0000 ea64640 -1 osd.3 998 get_health_metrics reporting 1 slow ops, oldest is osd_op(client.30047.0:3124 238.1s0 238:8fc3729e:test-rados-api-smithi088-40561-153::bar:head [writefull 0~10 in=10b] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e996) 2024-06-03T09:05:10.304 INFO:tasks.ceph.osd.3.smithi088.stderr:2024-06-03T09:05:10.303+0000 ea64640 -1 osd.3 998 get_health_metrics reporting 1 slow ops, oldest is osd_op(client.30047.0:3124 238.1s0 238:8fc3729e:test-rados-api-smithi088-40561-153::bar:head [writefull 0~10 in=10b] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e996) 2024-06-03T09:05:11.275 INFO:tasks.ceph.osd.3.smithi088.stderr:2024-06-03T09:05:11.274+0000 ea64640 -1 osd.3 998 get_health_metrics reporting 1 slow ops, oldest is osd_op(client.30047.0:3124 238.1s0 238:8fc3729e:test-rados-api-smithi088-40561-153::bar:head [writefull 0~10 in=10b] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e996) 2024-06-03T09:05:12.256 INFO:tasks.ceph.osd.3.smithi088.stderr:2024-06-03T09:05:12.255+0000 ea64640 -1 osd.3 998 get_health_metrics reporting 1 slow ops, oldest is osd_op(client.30047.0:3124 238.1s0 238:8fc3729e:test-rados-api-smithi088-40561-153::bar:head [writefull 0~10 in=10b] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e996) 2024-06-03T09:05:13.118 INFO:tasks.ceph.mon.a.smithi088.stderr:==00:00:30:08.444 34743== Warning: unimplemented fcntl command: 1036 2024-06-03T09:05:13.120 INFO:tasks.ceph.mon.a.smithi088.stderr:==00:00:30:08.446 34743== Warning: unimplemented fcntl command: 1036 2024-06-03T09:05:13.136 INFO:tasks.ceph.mon.a.smithi088.stderr:==00:00:30:08.462 34743== Warning: unimplemented fcntl command: 1036 2024-06-03T09:05:13.161 INFO:tasks.ceph.mon.b.smithi192.stderr:==00:00:30:08.464 34228== Warning: unimplemented fcntl command: 1036 2024-06-03T09:05:13.164 INFO:tasks.ceph.mon.b.smithi192.stderr:==00:00:30:08.467 34228== Warning: unimplemented fcntl command: 1036 2024-06-03T09:05:13.170 INFO:tasks.ceph.mon.c.smithi088.stderr:==00:00:30:08.496 34749== Warning: unimplemented fcntl command: 1036 2024-06-03T09:05:13.174 INFO:tasks.ceph.mon.c.smithi088.stderr:==00:00:30:08.500 34749== Warning: unimplemented fcntl command: 1036 2024-06-03T09:05:13.179 INFO:tasks.ceph.mon.b.smithi192.stderr:==00:00:30:08.481 34228== Warning: unimplemented fcntl command: 1036 2024-06-03T09:05:13.190 INFO:tasks.ceph.mon.c.smithi088.stderr:==00:00:30:08.516 34749== Warning: unimplemented fcntl command: 1036 2024-06-03T09:05:13.296 INFO:tasks.ceph.osd.3.smithi088.stderr:2024-06-03T09:05:13.294+0000 ea64640 -1 osd.3 998 get_health_metrics reporting 1 slow ops, oldest is osd_op(client.30047.0:3124 238.1s0 238:8fc3729e:test-rados-api-smithi088-40561-153::bar:head [writefull 0~10 in=10b] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e996) 2024-06-03T09:05:13.781 INFO:tasks.ceph.mon.b.smithi192.stderr:==00:00:30:09.084 34228== Warning: unimplemented fcntl command: 1036 2024-06-03T09:05:13.784 INFO:tasks.ceph.mon.b.smithi192.stderr:==00:00:30:09.086 34228== Warning: unimplemented fcntl command: 1036 2024-06-03T09:05:13.788 INFO:tasks.ceph.mon.a.smithi088.stderr:==00:00:30:09.114 34743== Warning: unimplemented fcntl command: 1036 2024-06-03T09:05:13.789 INFO:tasks.ceph.mon.a.smithi088.stderr:==00:00:30:09.115 34743== Warning: unimplemented fcntl command: 1036 2024-06-03T09:05:13.792 INFO:tasks.ceph.mon.b.smithi192.stderr:==00:00:30:09.095 34228== Warning: unimplemented fcntl command: 1036 2024-06-03T09:05:13.793 INFO:tasks.ceph.mon.c.smithi088.stderr:==00:00:30:09.119 34749== Warning: unimplemented fcntl command: 1036 2024-06-03T09:05:13.794 INFO:tasks.ceph.mon.c.smithi088.stderr:==00:00:30:09.120 34749== Warning: unimplemented fcntl command: 1036 2024-06-03T09:05:13.797 INFO:tasks.ceph.mon.a.smithi088.stderr:==00:00:30:09.123 34743== Warning: unimplemented fcntl command: 1036 2024-06-03T09:05:13.802 INFO:tasks.ceph.mon.c.smithi088.stderr:==00:00:30:09.128 34749== Warning: unimplemented fcntl command: 1036 2024-06-03T09:05:14.269 INFO:tasks.ceph.osd.3.smithi088.stderr:2024-06-03T09:05:14.268+0000 ea64640 -1 osd.3 998 get_health_metrics reporting 1 slow ops, oldest is osd_op(client.30047.0:3124 238.1s0 238:8fc3729e:test-rados-api-smithi088-40561-153::bar:head [writefull 0~10 in=10b] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e996) 2024-06-03T09:05:15.244 INFO:tasks.ceph.osd.3.smithi088.stderr:2024-06-03T09:05:15.242+0000 ea64640 -1 osd.3 998 get_health_metrics reporting 1 slow ops, oldest is osd_op(client.30047.0:3124 238.1s0 238:8fc3729e:test-rados-api-smithi088-40561-153::bar:head [writefull 0~10 in=10b] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e996) 2024-06-03T09:05:16.224 INFO:tasks.ceph.osd.3.smithi088.stderr:2024-06-03T09:05:16.224+0000 ea64640 -1 osd.3 998 get_health_metrics reporting 1 slow ops, oldest is osd_op(client.30047.0:3124 238.1s0 238:8fc3729e:test-rados-api-smithi088-40561-153::bar:head [writefull 0~10 in=10b] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e996) 2024-06-03T09:05:17.235 INFO:tasks.ceph.osd.3.smithi088.stderr:2024-06-03T09:05:17.235+0000 ea64640 -1 osd.3 998 get_health_metrics reporting 1 slow ops, oldest is osd_op(client.30047.0:3124 238.1s0 238:8fc3729e:test-rados-api-smithi088-40561-153::bar:head [writefull 0~10 in=10b] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e996) 2024-06-03T09:05:18.276 INFO:tasks.ceph.osd.3.smithi088.stderr:2024-06-03T09:05:18.276+0000 ea64640 -1 osd.3 998 get_health_metrics reporting 1 slow ops, oldest is osd_op(client.30047.0:3124 238.1s0 238:8fc3729e:test-rados-api-smithi088-40561-153::bar:head [writefull 0~10 in=10b] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e996) 2024-06-03T09:05:19.265 INFO:tasks.ceph.osd.3.smithi088.stderr:2024-06-03T09:05:19.264+0000 ea64640 -1 osd.3 998 get_health_metrics reporting 1 slow ops, oldest is osd_op(client.30047.0:3124 238.1s0 238:8fc3729e:test-rados-api-smithi088-40561-153::bar:head [writefull 0~10 in=10b] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e996) 2024-06-03T09:05:20.257 INFO:tasks.ceph.osd.3.smithi088.stderr:2024-06-03T09:05:20.256+0000 ea64640 -1 osd.3 998 get_health_metrics reporting 1 slow ops, oldest is osd_op(client.30047.0:3124 238.1s0 238:8fc3729e:test-rados-api-smithi088-40561-153::bar:head [writefull 0~10 in=10b] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e996) 2024-06-03T09:05:21.217 INFO:tasks.ceph.osd.3.smithi088.stderr:2024-06-03T09:05:21.216+0000 ea64640 -1 osd.3 998 get_health_metrics reporting 1 slow ops, oldest is osd_op(client.30047.0:3124 238.1s0 238:8fc3729e:test-rados-api-smithi088-40561-153::bar:head [writefull 0~10 in=10b] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e996) 2024-06-03T09:05:22.172 INFO:tasks.ceph.osd.3.smithi088.stderr:2024-06-03T09:05:22.171+0000 ea64640 -1 osd.3 998 get_health_metrics reporting 1 slow ops, oldest is osd_op(client.30047.0:3124 238.1s0 238:8fc3729e:test-rados-api-smithi088-40561-153::bar:head [writefull 0~10 in=10b] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e996) 2024-06-03T09:05:23.163 INFO:tasks.ceph.osd.3.smithi088.stderr:2024-06-03T09:05:23.162+0000 ea64640 -1 osd.3 998 get_health_metrics reporting 1 slow ops, oldest is osd_op(client.30047.0:3124 238.1s0 238:8fc3729e:test-rados-api-smithi088-40561-153::bar:head [writefull 0~10 in=10b] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e996) 2024-06-03T09:05:24.188 INFO:tasks.ceph.osd.3.smithi088.stderr:2024-06-03T09:05:24.187+0000 ea64640 -1 osd.3 998 get_health_metrics reporting 1 slow ops, oldest is osd_op(client.30047.0:3124 238.1s0 238:8fc3729e:test-rados-api-smithi088-40561-153::bar:head [writefull 0~10 in=10b] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e996) 2024-06-03T09:05:25.190 INFO:tasks.ceph.osd.3.smithi088.stderr:2024-06-03T09:05:25.190+0000 ea64640 -1 osd.3 998 get_health_metrics reporting 1 slow ops, oldest is osd_op(client.30047.0:3124 238.1s0 238:8fc3729e:test-rados-api-smithi088-40561-153::bar:head [writefull 0~10 in=10b] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e996) 2024-06-03T09:05:26.157 INFO:tasks.ceph.osd.3.smithi088.stderr:2024-06-03T09:05:26.157+0000 ea64640 -1 osd.3 998 get_health_metrics reporting 1 slow ops, oldest is osd_op(client.30047.0:3124 238.1s0 238:8fc3729e:test-rados-api-smithi088-40561-153::bar:head [writefull 0~10 in=10b] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e996) 2024-06-03T09:05:27.161 INFO:tasks.ceph.osd.3.smithi088.stderr:2024-06-03T09:05:27.159+0000 ea64640 -1 osd.3 998 get_health_metrics reporting 1 slow ops, oldest is osd_op(client.30047.0:3124 238.1s0 238:8fc3729e:test-rados-api-smithi088-40561-153::bar:head [writefull 0~10 in=10b] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e996) 2024-06-03T09:05:28.127 INFO:tasks.ceph.osd.3.smithi088.stderr:2024-06-03T09:05:28.126+0000 ea64640 -1 osd.3 998 get_health_metrics reporting 1 slow ops, oldest is osd_op(client.30047.0:3124 238.1s0 238:8fc3729e:test-rados-api-smithi088-40561-153::bar:head [writefull 0~10 in=10b] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e996) 2024-06-03T09:05:28.140 INFO:tasks.ceph.mon.a.smithi088.stderr:==00:00:30:23.466 34743== Warning: unimplemented fcntl command: 1036 2024-06-03T09:05:28.144 INFO:tasks.ceph.mon.a.smithi088.stderr:==00:00:30:23.470 34743== Warning: unimplemented fcntl command: 1036 2024-06-03T09:05:28.160 INFO:tasks.ceph.mon.a.smithi088.stderr:==00:00:30:23.486 34743== Warning: unimplemented fcntl command: 1036 2024-06-03T09:05:28.193 INFO:tasks.ceph.mon.c.smithi088.stderr:==00:00:30:23.519 34749== Warning: unimplemented fcntl command: 1036 2024-06-03T09:05:28.194 INFO:tasks.ceph.mon.b.smithi192.stderr:==00:00:30:23.496 34228== Warning: unimplemented fcntl command: 1036 2024-06-03T09:05:28.195 INFO:tasks.ceph.mon.b.smithi192.stderr:==00:00:30:23.498 34228== Warning: unimplemented fcntl command: 1036 2024-06-03T09:05:28.196 INFO:tasks.ceph.mon.c.smithi088.stderr:==00:00:30:23.522 34749== Warning: unimplemented fcntl command: 1036 2024-06-03T09:05:28.211 INFO:tasks.ceph.mon.b.smithi192.stderr:==00:00:30:23.514 34228== Warning: unimplemented fcntl command: 1036 2024-06-03T09:05:28.215 INFO:tasks.ceph.mon.c.smithi088.stderr:==00:00:30:23.541 34749== Warning: unimplemented fcntl command: 1036 2024-06-03T09:05:29.108 INFO:tasks.ceph.osd.3.smithi088.stderr:2024-06-03T09:05:29.107+0000 ea64640 -1 osd.3 998 get_health_metrics reporting 1 slow ops, oldest is osd_op(client.30047.0:3124 238.1s0 238:8fc3729e:test-rados-api-smithi088-40561-153::bar:head [writefull 0~10 in=10b] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e996) 2024-06-03T09:05:30.097 INFO:tasks.ceph.osd.3.smithi088.stderr:2024-06-03T09:05:30.096+0000 ea64640 -1 osd.3 998 get_health_metrics reporting 1 slow ops, oldest is osd_op(client.30047.0:3124 238.1s0 238:8fc3729e:test-rados-api-smithi088-40561-153::bar:head [writefull 0~10 in=10b] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e996) 2024-06-03T09:05:31.057 INFO:tasks.ceph.osd.3.smithi088.stderr:2024-06-03T09:05:31.057+0000 ea64640 -1 osd.3 998 get_health_metrics reporting 1 slow ops, oldest is osd_op(client.30047.0:3124 238.1s0 238:8fc3729e:test-rados-api-smithi088-40561-153::bar:head [writefull 0~10 in=10b] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e996) 2024-06-03T09:05:32.019 INFO:tasks.ceph.osd.3.smithi088.stderr:2024-06-03T09:05:32.018+0000 ea64640 -1 osd.3 998 get_health_metrics reporting 1 slow ops, oldest is osd_op(client.30047.0:3124 238.1s0 238:8fc3729e:test-rados-api-smithi088-40561-153::bar:head [writefull 0~10 in=10b] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e996) 2024-06-03T09:05:33.018 INFO:tasks.ceph.osd.3.smithi088.stderr:2024-06-03T09:05:33.018+0000 ea64640 -1 osd.3 998 get_health_metrics reporting 1 slow ops, oldest is osd_op(client.30047.0:3124 238.1s0 238:8fc3729e:test-rados-api-smithi088-40561-153::bar:head [writefull 0~10 in=10b] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e996) 2024-06-03T09:05:33.168 INFO:tasks.ceph.mon.a.smithi088.stderr:==00:00:30:28.494 34743== Warning: unimplemented fcntl command: 1036 2024-06-03T09:05:33.216 INFO:tasks.ceph.mon.a.smithi088.stderr:==00:00:30:28.541 34743== Warning: unimplemented fcntl command: 1036 2024-06-03T09:05:33.228 INFO:tasks.ceph.mon.a.smithi088.stderr:==00:00:30:28.554 34743== Warning: unimplemented fcntl command: 1036 2024-06-03T09:05:33.263 INFO:tasks.ceph.mon.c.smithi088.stderr:==00:00:30:28.588 34749== Warning: unimplemented fcntl command: 1036 2024-06-03T09:05:33.263 INFO:tasks.ceph.mon.b.smithi192.stderr:==00:00:30:28.565 34228== Warning: unimplemented fcntl command: 1036 2024-06-03T09:05:33.264 INFO:tasks.ceph.mon.b.smithi192.stderr:==00:00:30:28.567 34228== Warning: unimplemented fcntl command: 1036 2024-06-03T09:05:33.270 INFO:tasks.ceph.mon.c.smithi088.stderr:==00:00:30:28.596 34749== Warning: unimplemented fcntl command: 1036 2024-06-03T09:05:33.274 INFO:tasks.ceph.mon.b.smithi192.stderr:==00:00:30:28.576 34228== Warning: unimplemented fcntl command: 1036 2024-06-03T09:05:33.280 INFO:tasks.ceph.mon.c.smithi088.stderr:==00:00:30:28.606 34749== Warning: unimplemented fcntl command: 1036 2024-06-03T09:05:34.066 INFO:tasks.ceph.osd.3.smithi088.stderr:2024-06-03T09:05:34.065+0000 ea64640 -1 osd.3 998 get_health_metrics reporting 1 slow ops, oldest is osd_op(client.30047.0:3124 238.1s0 238:8fc3729e:test-rados-api-smithi088-40561-153::bar:head [writefull 0~10 in=10b] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e996) 2024-06-03T09:05:35.114 INFO:tasks.ceph.osd.3.smithi088.stderr:2024-06-03T09:05:35.113+0000 ea64640 -1 osd.3 998 get_health_metrics reporting 1 slow ops, oldest is osd_op(client.30047.0:3124 238.1s0 238:8fc3729e:test-rados-api-smithi088-40561-153::bar:head [writefull 0~10 in=10b] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e996) 2024-06-03T09:05:36.162 INFO:tasks.ceph.osd.3.smithi088.stderr:2024-06-03T09:05:36.161+0000 ea64640 -1 osd.3 998 get_health_metrics reporting 1 slow ops, oldest is osd_op(client.30047.0:3124 238.1s0 238:8fc3729e:test-rados-api-smithi088-40561-153::bar:head [writefull 0~10 in=10b] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e996) 2024-06-03T09:05:37.214 INFO:tasks.ceph.osd.3.smithi088.stderr:2024-06-03T09:05:37.213+0000 ea64640 -1 osd.3 998 get_health_metrics reporting 1 slow ops, oldest is osd_op(client.30047.0:3124 238.1s0 238:8fc3729e:test-rados-api-smithi088-40561-153::bar:head [writefull 0~10 in=10b] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e996) 2024-06-03T09:05:38.253 INFO:tasks.ceph.osd.3.smithi088.stderr:2024-06-03T09:05:38.252+0000 ea64640 -1 osd.3 998 get_health_metrics reporting 1 slow ops, oldest is osd_op(client.30047.0:3124 238.1s0 238:8fc3729e:test-rados-api-smithi088-40561-153::bar:head [writefull 0~10 in=10b] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e996) 2024-06-03T09:05:39.243 INFO:tasks.ceph.osd.3.smithi088.stderr:2024-06-03T09:05:39.242+0000 ea64640 -1 osd.3 998 get_health_metrics reporting 1 slow ops, oldest is osd_op(client.30047.0:3124 238.1s0 238:8fc3729e:test-rados-api-smithi088-40561-153::bar:head [writefull 0~10 in=10b] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e996) 2024-06-03T09:05:40.264 INFO:tasks.ceph.osd.3.smithi088.stderr:2024-06-03T09:05:40.263+0000 ea64640 -1 osd.3 998 get_health_metrics reporting 1 slow ops, oldest is osd_op(client.30047.0:3124 238.1s0 238:8fc3729e:test-rados-api-smithi088-40561-153::bar:head [writefull 0~10 in=10b] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e996) 2024-06-03T09:05:41.263 INFO:tasks.ceph.osd.3.smithi088.stderr:2024-06-03T09:05:41.262+0000 ea64640 -1 osd.3 998 get_health_metrics reporting 1 slow ops, oldest is osd_op(client.30047.0:3124 238.1s0 238:8fc3729e:test-rados-api-smithi088-40561-153::bar:head [writefull 0~10 in=10b] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e996) 2024-06-03T09:05:42.251 INFO:tasks.ceph.osd.3.smithi088.stderr:2024-06-03T09:05:42.250+0000 ea64640 -1 osd.3 998 get_health_metrics reporting 1 slow ops, oldest is osd_op(client.30047.0:3124 238.1s0 238:8fc3729e:test-rados-api-smithi088-40561-153::bar:head [writefull 0~10 in=10b] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e996) 2024-06-03T09:05:43.219 INFO:tasks.ceph.mon.a.smithi088.stderr:==00:00:30:38.545 34743== Warning: unimplemented fcntl command: 1036 2024-06-03T09:05:43.224 INFO:tasks.ceph.mon.a.smithi088.stderr:==00:00:30:38.549 34743== Warning: unimplemented fcntl command: 1036 2024-06-03T09:05:43.237 INFO:tasks.ceph.mon.a.smithi088.stderr:==00:00:30:38.563 34743== Warning: unimplemented fcntl command: 1036 2024-06-03T09:05:43.270 INFO:tasks.ceph.mon.b.smithi192.stderr:==00:00:30:38.572 34228== Warning: unimplemented fcntl command: 1036 2024-06-03T09:05:43.271 INFO:tasks.ceph.mon.b.smithi192.stderr:==00:00:30:38.574 34228== Warning: unimplemented fcntl command: 1036 2024-06-03T09:05:43.274 INFO:tasks.ceph.mon.c.smithi088.stderr:==00:00:30:38.600 34749== Warning: unimplemented fcntl command: 1036 2024-06-03T09:05:43.278 INFO:tasks.ceph.mon.c.smithi088.stderr:==00:00:30:38.604 34749== Warning: unimplemented fcntl command: 1036 2024-06-03T09:05:43.284 INFO:tasks.ceph.mon.b.smithi192.stderr:==00:00:30:38.587 34228== Warning: unimplemented fcntl command: 1036 2024-06-03T09:05:43.291 INFO:tasks.ceph.mon.c.smithi088.stderr:==00:00:30:38.617 34749== Warning: unimplemented fcntl command: 1036 2024-06-03T09:05:43.302 INFO:tasks.ceph.osd.3.smithi088.stderr:2024-06-03T09:05:43.301+0000 ea64640 -1 osd.3 998 get_health_metrics reporting 1 slow ops, oldest is osd_op(client.30047.0:3124 238.1s0 238:8fc3729e:test-rados-api-smithi088-40561-153::bar:head [writefull 0~10 in=10b] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e996) 2024-06-03T09:05:44.322 INFO:tasks.ceph.osd.3.smithi088.stderr:2024-06-03T09:05:44.321+0000 ea64640 -1 osd.3 998 get_health_metrics reporting 1 slow ops, oldest is osd_op(client.30047.0:3124 238.1s0 238:8fc3729e:test-rados-api-smithi088-40561-153::bar:head [writefull 0~10 in=10b] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e996) 2024-06-03T09:05:45.351 INFO:tasks.ceph.osd.3.smithi088.stderr:2024-06-03T09:05:45.351+0000 ea64640 -1 osd.3 998 get_health_metrics reporting 1 slow ops, oldest is osd_op(client.30047.0:3124 238.1s0 238:8fc3729e:test-rados-api-smithi088-40561-153::bar:head [writefull 0~10 in=10b] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e996) 2024-06-03T09:05:46.350 INFO:tasks.ceph.osd.3.smithi088.stderr:2024-06-03T09:05:46.349+0000 ea64640 -1 osd.3 998 get_health_metrics reporting 1 slow ops, oldest is osd_op(client.30047.0:3124 238.1s0 238:8fc3729e:test-rados-api-smithi088-40561-153::bar:head [writefull 0~10 in=10b] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e996) 2024-06-03T09:05:47.332 INFO:tasks.ceph.osd.3.smithi088.stderr:2024-06-03T09:05:47.332+0000 ea64640 -1 osd.3 998 get_health_metrics reporting 1 slow ops, oldest is osd_op(client.30047.0:3124 238.1s0 238:8fc3729e:test-rados-api-smithi088-40561-153::bar:head [writefull 0~10 in=10b] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e996) 2024-06-03T09:05:48.321 INFO:tasks.ceph.osd.3.smithi088.stderr:2024-06-03T09:05:48.321+0000 ea64640 -1 osd.3 998 get_health_metrics reporting 1 slow ops, oldest is osd_op(client.30047.0:3124 238.1s0 238:8fc3729e:test-rados-api-smithi088-40561-153::bar:head [writefull 0~10 in=10b] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e996) 2024-06-03T09:05:49.330 INFO:tasks.ceph.osd.3.smithi088.stderr:2024-06-03T09:05:49.329+0000 ea64640 -1 osd.3 998 get_health_metrics reporting 1 slow ops, oldest is osd_op(client.30047.0:3124 238.1s0 238:8fc3729e:test-rados-api-smithi088-40561-153::bar:head [writefull 0~10 in=10b] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e996) 2024-06-03T09:05:50.333 INFO:tasks.ceph.osd.3.smithi088.stderr:2024-06-03T09:05:50.332+0000 ea64640 -1 osd.3 998 get_health_metrics reporting 1 slow ops, oldest is osd_op(client.30047.0:3124 238.1s0 238:8fc3729e:test-rados-api-smithi088-40561-153::bar:head [writefull 0~10 in=10b] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e996) 2024-06-03T09:05:51.356 INFO:tasks.ceph.osd.3.smithi088.stderr:2024-06-03T09:05:51.356+0000 ea64640 -1 osd.3 998 get_health_metrics reporting 1 slow ops, oldest is osd_op(client.30047.0:3124 238.1s0 238:8fc3729e:test-rados-api-smithi088-40561-153::bar:head [writefull 0~10 in=10b] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e996) 2024-06-03T09:05:52.385 INFO:tasks.ceph.osd.3.smithi088.stderr:2024-06-03T09:05:52.384+0000 ea64640 -1 osd.3 998 get_health_metrics reporting 1 slow ops, oldest is osd_op(client.30047.0:3124 238.1s0 238:8fc3729e:test-rados-api-smithi088-40561-153::bar:head [writefull 0~10 in=10b] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e996) 2024-06-03T09:05:53.246 INFO:tasks.ceph.mon.a.smithi088.stderr:==00:00:30:48.571 34743== Warning: unimplemented fcntl command: 1036 2024-06-03T09:05:53.248 INFO:tasks.ceph.mon.a.smithi088.stderr:==00:00:30:48.574 34743== Warning: unimplemented fcntl command: 1036 2024-06-03T09:05:53.272 INFO:tasks.ceph.mon.a.smithi088.stderr:==00:00:30:48.598 34743== Warning: unimplemented fcntl command: 1036 2024-06-03T09:05:53.295 INFO:tasks.ceph.mon.b.smithi192.stderr:==00:00:30:48.598 34228== Warning: unimplemented fcntl command: 1036 2024-06-03T09:05:53.297 INFO:tasks.ceph.mon.b.smithi192.stderr:==00:00:30:48.600 34228== Warning: unimplemented fcntl command: 1036 2024-06-03T09:05:53.305 INFO:tasks.ceph.mon.c.smithi088.stderr:==00:00:30:48.631 34749== Warning: unimplemented fcntl command: 1036 2024-06-03T09:05:53.309 INFO:tasks.ceph.mon.c.smithi088.stderr:==00:00:30:48.635 34749== Warning: unimplemented fcntl command: 1036 2024-06-03T09:05:53.311 INFO:tasks.ceph.mon.b.smithi192.stderr:==00:00:30:48.614 34228== Warning: unimplemented fcntl command: 1036 2024-06-03T09:05:53.326 INFO:tasks.ceph.mon.c.smithi088.stderr:==00:00:30:48.652 34749== Warning: unimplemented fcntl command: 1036 2024-06-03T09:05:53.340 INFO:tasks.ceph.osd.3.smithi088.stderr:2024-06-03T09:05:53.340+0000 ea64640 -1 osd.3 998 get_health_metrics reporting 1 slow ops, oldest is osd_op(client.30047.0:3124 238.1s0 238:8fc3729e:test-rados-api-smithi088-40561-153::bar:head [writefull 0~10 in=10b] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e996) 2024-06-03T09:05:53.897 INFO:tasks.ceph.mon.a.smithi088.stderr:==00:00:30:49.223 34743== Warning: unimplemented fcntl command: 1036 2024-06-03T09:05:53.898 INFO:tasks.ceph.mon.a.smithi088.stderr:==00:00:30:49.224 34743== Warning: unimplemented fcntl command: 1036 2024-06-03T09:05:53.907 INFO:tasks.ceph.mon.a.smithi088.stderr:==00:00:30:49.233 34743== Warning: unimplemented fcntl command: 1036 2024-06-03T09:05:53.921 INFO:tasks.ceph.mon.b.smithi192.stderr:==00:00:30:49.224 34228== Warning: unimplemented fcntl command: 1036 2024-06-03T09:05:53.922 INFO:tasks.ceph.mon.b.smithi192.stderr:==00:00:30:49.225 34228== Warning: unimplemented fcntl command: 1036 2024-06-03T09:05:53.923 INFO:tasks.ceph.mon.c.smithi088.stderr:==00:00:30:49.249 34749== Warning: unimplemented fcntl command: 1036 2024-06-03T09:05:53.924 INFO:tasks.ceph.mon.c.smithi088.stderr:==00:00:30:49.250 34749== Warning: unimplemented fcntl command: 1036 2024-06-03T09:05:53.933 INFO:tasks.ceph.mon.b.smithi192.stderr:==00:00:30:49.236 34228== Warning: unimplemented fcntl command: 1036 2024-06-03T09:05:53.934 INFO:tasks.ceph.mon.c.smithi088.stderr:==00:00:30:49.260 34749== Warning: unimplemented fcntl command: 1036 2024-06-03T09:05:54.345 INFO:tasks.ceph.osd.3.smithi088.stderr:2024-06-03T09:05:54.345+0000 ea64640 -1 osd.3 998 get_health_metrics reporting 1 slow ops, oldest is osd_op(client.30047.0:3124 238.1s0 238:8fc3729e:test-rados-api-smithi088-40561-153::bar:head [writefull 0~10 in=10b] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e996) 2024-06-03T09:05:55.334 INFO:tasks.ceph.osd.3.smithi088.stderr:2024-06-03T09:05:55.333+0000 ea64640 -1 osd.3 998 get_health_metrics reporting 1 slow ops, oldest is osd_op(client.30047.0:3124 238.1s0 238:8fc3729e:test-rados-api-smithi088-40561-153::bar:head [writefull 0~10 in=10b] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e996) 2024-06-03T09:05:56.329 INFO:tasks.ceph.osd.3.smithi088.stderr:2024-06-03T09:05:56.328+0000 ea64640 -1 osd.3 998 get_health_metrics reporting 1 slow ops, oldest is osd_op(client.30047.0:3124 238.1s0 238:8fc3729e:test-rados-api-smithi088-40561-153::bar:head [writefull 0~10 in=10b] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e996) 2024-06-03T09:05:57.162 INFO:tasks.ceph.osd.4.smithi192.stderr:2024-06-03T09:05:57.158+0000 12a6c640 -1 osd.4 998 heartbeat_check: no reply from 172.21.15.192:6815 osd.6 since back 2024-06-03T09:04:35.968606+0000 front 2024-06-03T09:04:35.965594+0000 (oldest deadline 2024-06-03T09:05:56.876396+0000) 2024-06-03T09:05:57.342 INFO:tasks.ceph.osd.3.smithi088.stderr:2024-06-03T09:05:57.342+0000 ea64640 -1 osd.3 998 get_health_metrics reporting 1 slow ops, oldest is osd_op(client.30047.0:3124 238.1s0 238:8fc3729e:test-rados-api-smithi088-40561-153::bar:head [writefull 0~10 in=10b] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e996) 2024-06-03T09:05:58.201 INFO:tasks.ceph.osd.4.smithi192.stderr:2024-06-03T09:05:58.200+0000 12a6c640 -1 osd.4 998 heartbeat_check: no reply from 172.21.15.192:6815 osd.6 since back 2024-06-03T09:04:35.968606+0000 front 2024-06-03T09:04:35.965594+0000 (oldest deadline 2024-06-03T09:05:56.876396+0000) 2024-06-03T09:05:58.300 INFO:tasks.ceph.osd.3.smithi088.stderr:2024-06-03T09:05:58.299+0000 ea64640 -1 osd.3 998 get_health_metrics reporting 1 slow ops, oldest is osd_op(client.30047.0:3124 238.1s0 238:8fc3729e:test-rados-api-smithi088-40561-153::bar:head [writefull 0~10 in=10b] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e996) 2024-06-03T09:05:58.468 INFO:tasks.ceph.osd.7.smithi192.stderr:2024-06-03T09:05:58.468+0000 12a6c640 -1 osd.7 998 heartbeat_check: no reply from 172.21.15.192:6815 osd.6 since back 2024-06-03T09:04:34.096087+0000 front 2024-06-03T09:04:34.101737+0000 (oldest deadline 2024-06-03T09:05:58.193329+0000) 2024-06-03T09:05:58.801 INFO:tasks.ceph.osd.2.smithi088.stderr:2024-06-03T09:05:58.800+0000 12a6c640 -1 osd.2 998 heartbeat_check: no reply from 172.21.15.192:6815 osd.6 since back 2024-06-03T09:04:32.694500+0000 front 2024-06-03T09:04:32.695096+0000 (oldest deadline 2024-06-03T09:05:57.862618+0000) 2024-06-03T09:05:59.174 INFO:tasks.ceph.osd.4.smithi192.stderr:2024-06-03T09:05:59.171+0000 12a6c640 -1 osd.4 998 heartbeat_check: no reply from 172.21.15.192:6815 osd.6 since back 2024-06-03T09:04:35.968606+0000 front 2024-06-03T09:04:35.965594+0000 (oldest deadline 2024-06-03T09:05:56.876396+0000) 2024-06-03T09:05:59.341 INFO:tasks.ceph.osd.3.smithi088.stderr:2024-06-03T09:05:59.340+0000 ea64640 -1 osd.3 998 get_health_metrics reporting 1 slow ops, oldest is osd_op(client.30047.0:3124 238.1s0 238:8fc3729e:test-rados-api-smithi088-40561-153::bar:head [writefull 0~10 in=10b] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e996) 2024-06-03T09:05:59.433 INFO:tasks.ceph.osd.5.smithi192.stderr:2024-06-03T09:05:59.432+0000 12a6c640 -1 osd.5 998 heartbeat_check: no reply from 172.21.15.192:6815 osd.6 since back 2024-06-03T09:04:35.792954+0000 front 2024-06-03T09:04:35.783252+0000 (oldest deadline 2024-06-03T09:05:59.143987+0000) 2024-06-03T09:05:59.444 INFO:tasks.ceph.osd.7.smithi192.stderr:2024-06-03T09:05:59.444+0000 12a6c640 -1 osd.7 998 heartbeat_check: no reply from 172.21.15.192:6815 osd.6 since back 2024-06-03T09:04:34.096087+0000 front 2024-06-03T09:04:34.101737+0000 (oldest deadline 2024-06-03T09:05:58.193329+0000) 2024-06-03T09:05:59.797 INFO:tasks.ceph.osd.2.smithi088.stderr:2024-06-03T09:05:59.792+0000 12a6c640 -1 osd.2 998 heartbeat_check: no reply from 172.21.15.192:6815 osd.6 since back 2024-06-03T09:04:32.694500+0000 front 2024-06-03T09:04:32.695096+0000 (oldest deadline 2024-06-03T09:05:57.862618+0000) 2024-06-03T09:06:00.038 INFO:tasks.ceph.osd.0.smithi088.stderr:2024-06-03T09:06:00.037+0000 ea64640 -1 osd.0 998 heartbeat_check: no reply from 172.21.15.192:6815 osd.6 since back 2024-06-03T09:04:34.632930+0000 front 2024-06-03T09:04:34.631678+0000 (oldest deadline 2024-06-03T09:05:59.318618+0000) 2024-06-03T09:06:00.144 INFO:tasks.ceph.osd.4.smithi192.stderr:2024-06-03T09:06:00.143+0000 12a6c640 -1 osd.4 998 heartbeat_check: no reply from 172.21.15.192:6815 osd.6 since back 2024-06-03T09:04:35.968606+0000 front 2024-06-03T09:04:35.965594+0000 (oldest deadline 2024-06-03T09:05:56.876396+0000) 2024-06-03T09:06:00.326 INFO:tasks.ceph.osd.3.smithi088.stderr:2024-06-03T09:06:00.326+0000 ea64640 -1 osd.3 998 get_health_metrics reporting 1 slow ops, oldest is osd_op(client.30047.0:3124 238.1s0 238:8fc3729e:test-rados-api-smithi088-40561-153::bar:head [writefull 0~10 in=10b] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e996) 2024-06-03T09:06:00.462 INFO:tasks.ceph.osd.5.smithi192.stderr:2024-06-03T09:06:00.462+0000 12a6c640 -1 osd.5 998 heartbeat_check: no reply from 172.21.15.192:6815 osd.6 since back 2024-06-03T09:04:35.792954+0000 front 2024-06-03T09:04:35.783252+0000 (oldest deadline 2024-06-03T09:05:59.143987+0000) 2024-06-03T09:06:00.482 INFO:tasks.ceph.osd.7.smithi192.stderr:2024-06-03T09:06:00.481+0000 12a6c640 -1 osd.7 998 heartbeat_check: no reply from 172.21.15.192:6815 osd.6 since back 2024-06-03T09:04:34.096087+0000 front 2024-06-03T09:04:34.101737+0000 (oldest deadline 2024-06-03T09:05:58.193329+0000) 2024-06-03T09:06:00.790 INFO:tasks.ceph.osd.2.smithi088.stderr:2024-06-03T09:06:00.789+0000 12a6c640 -1 osd.2 998 heartbeat_check: no reply from 172.21.15.192:6815 osd.6 since back 2024-06-03T09:04:32.694500+0000 front 2024-06-03T09:04:32.695096+0000 (oldest deadline 2024-06-03T09:05:57.862618+0000) 2024-06-03T09:06:00.997 INFO:tasks.ceph.osd.0.smithi088.stderr:2024-06-03T09:06:00.996+0000 ea64640 -1 osd.0 998 heartbeat_check: no reply from 172.21.15.192:6815 osd.6 since back 2024-06-03T09:04:34.632930+0000 front 2024-06-03T09:04:34.631678+0000 (oldest deadline 2024-06-03T09:05:59.318618+0000) 2024-06-03T09:06:01.159 INFO:tasks.ceph.osd.4.smithi192.stderr:2024-06-03T09:06:01.159+0000 12a6c640 -1 osd.4 998 heartbeat_check: no reply from 172.21.15.192:6815 osd.6 since back 2024-06-03T09:04:35.968606+0000 front 2024-06-03T09:04:35.965594+0000 (oldest deadline 2024-06-03T09:05:56.876396+0000) 2024-06-03T09:06:01.350 INFO:tasks.ceph.osd.3.smithi088.stderr:2024-06-03T09:06:01.348+0000 ea64640 -1 osd.3 998 heartbeat_check: no reply from 172.21.15.192:6815 osd.6 since back 2024-06-03T09:04:35.437883+0000 front 2024-06-03T09:04:35.426724+0000 (oldest deadline 2024-06-03T09:06:00.722533+0000) 2024-06-03T09:06:01.350 INFO:tasks.ceph.osd.3.smithi088.stderr:2024-06-03T09:06:01.350+0000 ea64640 -1 osd.3 998 get_health_metrics reporting 2 slow ops, oldest is osd_op(client.30047.0:3124 238.1s0 238:8fc3729e:test-rados-api-smithi088-40561-153::bar:head [writefull 0~10 in=10b] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e996) 2024-06-03T09:06:01.456 INFO:tasks.ceph.osd.7.smithi192.stderr:2024-06-03T09:06:01.453+0000 12a6c640 -1 osd.7 998 heartbeat_check: no reply from 172.21.15.192:6815 osd.6 since back 2024-06-03T09:04:34.096087+0000 front 2024-06-03T09:04:34.101737+0000 (oldest deadline 2024-06-03T09:05:58.193329+0000) 2024-06-03T09:06:01.508 INFO:tasks.ceph.osd.5.smithi192.stderr:2024-06-03T09:06:01.507+0000 12a6c640 -1 osd.5 998 heartbeat_check: no reply from 172.21.15.192:6815 osd.6 since back 2024-06-03T09:04:35.792954+0000 front 2024-06-03T09:04:35.783252+0000 (oldest deadline 2024-06-03T09:05:59.143987+0000) 2024-06-03T09:06:01.630 INFO:tasks.ceph.osd.1.smithi088.stderr:2024-06-03T09:06:01.628+0000 ea64640 -1 osd.1 998 heartbeat_check: no reply from 172.21.15.192:6815 osd.6 since back 2024-06-03T09:04:36.267437+0000 front 2024-06-03T09:04:36.421830+0000 (oldest deadline 2024-06-03T09:06:01.553059+0000) 2024-06-03T09:06:01.748 INFO:tasks.ceph.osd.2.smithi088.stderr:2024-06-03T09:06:01.747+0000 12a6c640 -1 osd.2 998 heartbeat_check: no reply from 172.21.15.192:6815 osd.6 since back 2024-06-03T09:04:32.694500+0000 front 2024-06-03T09:04:32.695096+0000 (oldest deadline 2024-06-03T09:05:57.862618+0000) 2024-06-03T09:06:02.042 INFO:tasks.ceph.osd.0.smithi088.stderr:2024-06-03T09:06:02.040+0000 ea64640 -1 osd.0 998 heartbeat_check: no reply from 172.21.15.192:6815 osd.6 since back 2024-06-03T09:04:34.632930+0000 front 2024-06-03T09:04:34.631678+0000 (oldest deadline 2024-06-03T09:05:59.318618+0000) 2024-06-03T09:06:02.137 INFO:tasks.ceph.osd.4.smithi192.stderr:2024-06-03T09:06:02.137+0000 12a6c640 -1 osd.4 998 heartbeat_check: no reply from 172.21.15.192:6815 osd.6 since back 2024-06-03T09:04:35.968606+0000 front 2024-06-03T09:04:35.965594+0000 (oldest deadline 2024-06-03T09:05:56.876396+0000) 2024-06-03T09:06:02.390 INFO:tasks.ceph.osd.3.smithi088.stderr:2024-06-03T09:06:02.389+0000 ea64640 -1 osd.3 998 heartbeat_check: no reply from 172.21.15.192:6815 osd.6 since back 2024-06-03T09:04:35.437883+0000 front 2024-06-03T09:04:35.426724+0000 (oldest deadline 2024-06-03T09:06:00.722533+0000) 2024-06-03T09:06:02.391 INFO:tasks.ceph.osd.3.smithi088.stderr:2024-06-03T09:06:02.390+0000 ea64640 -1 osd.3 998 get_health_metrics reporting 2 slow ops, oldest is osd_op(client.30047.0:3124 238.1s0 238:8fc3729e:test-rados-api-smithi088-40561-153::bar:head [writefull 0~10 in=10b] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e996) 2024-06-03T09:06:02.420 INFO:tasks.ceph.osd.7.smithi192.stderr:2024-06-03T09:06:02.419+0000 12a6c640 -1 osd.7 998 heartbeat_check: no reply from 172.21.15.192:6815 osd.6 since back 2024-06-03T09:04:34.096087+0000 front 2024-06-03T09:04:34.101737+0000 (oldest deadline 2024-06-03T09:05:58.193329+0000) 2024-06-03T09:06:02.506 INFO:tasks.ceph.osd.5.smithi192.stderr:2024-06-03T09:06:02.505+0000 12a6c640 -1 osd.5 998 heartbeat_check: no reply from 172.21.15.192:6815 osd.6 since back 2024-06-03T09:04:35.792954+0000 front 2024-06-03T09:04:35.783252+0000 (oldest deadline 2024-06-03T09:05:59.143987+0000) 2024-06-03T09:06:02.659 INFO:tasks.ceph.osd.1.smithi088.stderr:2024-06-03T09:06:02.657+0000 ea64640 -1 osd.1 998 heartbeat_check: no reply from 172.21.15.192:6815 osd.6 since back 2024-06-03T09:04:36.267437+0000 front 2024-06-03T09:04:36.421830+0000 (oldest deadline 2024-06-03T09:06:01.553059+0000) 2024-06-03T09:06:08.314 INFO:tasks.ceph.mon.a.smithi088.stderr:==00:00:31:03.640 34743== Warning: unimplemented fcntl command: 1036 2024-06-03T09:06:08.324 INFO:tasks.ceph.mon.a.smithi088.stderr:==00:00:31:03.650 34743== Warning: unimplemented fcntl command: 1036 2024-06-03T09:06:08.358 INFO:tasks.ceph.mon.a.smithi088.stderr:==00:00:31:03.684 34743== Warning: unimplemented fcntl command: 1036 2024-06-03T09:06:08.362 INFO:tasks.ceph.mon.b.smithi192.stderr:==00:00:31:03.665 34228== Warning: unimplemented fcntl command: 1036 2024-06-03T09:06:08.369 INFO:tasks.ceph.mon.b.smithi192.stderr:==00:00:31:03.671 34228== Warning: unimplemented fcntl command: 1036 2024-06-03T09:06:08.384 INFO:tasks.ceph.mon.c.smithi088.stderr:==00:00:31:03.710 34749== Warning: unimplemented fcntl command: 1036 2024-06-03T09:06:08.386 INFO:tasks.ceph.mon.c.smithi088.stderr:==00:00:31:03.712 34749== Warning: unimplemented fcntl command: 1036 2024-06-03T09:06:08.398 INFO:tasks.ceph.mon.b.smithi192.stderr:==00:00:31:03.701 34228== Warning: unimplemented fcntl command: 1036 2024-06-03T09:06:08.416 INFO:tasks.ceph.mon.c.smithi088.stderr:==00:00:31:03.741 34749== Warning: unimplemented fcntl command: 1036 2024-06-03T09:06:13.309 INFO:tasks.ceph.mon.a.smithi088.stderr:==00:00:31:08.634 34743== Warning: unimplemented fcntl command: 1036 2024-06-03T09:06:13.310 INFO:tasks.ceph.mon.a.smithi088.stderr:==00:00:31:08.636 34743== Warning: unimplemented fcntl command: 1036 2024-06-03T09:06:13.335 INFO:tasks.ceph.mon.a.smithi088.stderr:==00:00:31:08.661 34743== Warning: unimplemented fcntl command: 1036 2024-06-03T09:06:13.358 INFO:tasks.ceph.mon.b.smithi192.stderr:==00:00:31:08.661 34228== Warning: unimplemented fcntl command: 1036 2024-06-03T09:06:13.361 INFO:tasks.ceph.mon.b.smithi192.stderr:==00:00:31:08.664 34228== Warning: unimplemented fcntl command: 1036 2024-06-03T09:06:13.367 INFO:tasks.ceph.mon.c.smithi088.stderr:==00:00:31:08.693 34749== Warning: unimplemented fcntl command: 1036 2024-06-03T09:06:13.369 INFO:tasks.ceph.mon.c.smithi088.stderr:==00:00:31:08.695 34749== Warning: unimplemented fcntl command: 1036 2024-06-03T09:06:13.375 INFO:tasks.ceph.mon.b.smithi192.stderr:==00:00:31:08.678 34228== Warning: unimplemented fcntl command: 1036 2024-06-03T09:06:13.383 INFO:tasks.ceph.mon.c.smithi088.stderr:==00:00:31:08.709 34749== Warning: unimplemented fcntl command: 1036 2024-06-03T09:06:13.820 INFO:tasks.workunit.client.0.smithi088.stdout: api_tier_pp: [ RUN ] LibRadosTwoPoolsECPP.HitSetTrim 2024-06-03T09:06:13.821 INFO:tasks.workunit.client.0.smithi088.stdout: api_tier_pp: got ls 1717405358,0 2024-06-03T09:06:13.821 INFO:tasks.workunit.client.0.smithi088.stdout: api_tier_pp: first is 1717405358 2024-06-03T09:06:13.821 INFO:tasks.workunit.client.0.smithi088.stdout: api_tier_pp: got ls 1717405358,0 2024-06-03T09:06:13.821 INFO:tasks.workunit.client.0.smithi088.stdout: api_tier_pp: got ls 1717405358,0 2024-06-03T09:06:13.821 INFO:tasks.workunit.client.0.smithi088.stdout: api_tier_pp: got ls 1717405358,1717405361,1717405362,0 2024-06-03T09:06:13.821 INFO:tasks.workunit.client.0.smithi088.stdout: api_tier_pp: got ls 1717405358,1717405361,1717405362,0 2024-06-03T09:06:13.821 INFO:tasks.workunit.client.0.smithi088.stdout: api_tier_pp: got ls 1717405358,1717405361,1717405362,0 2024-06-03T09:06:13.821 INFO:tasks.workunit.client.0.smithi088.stdout: api_tier_pp: got ls 1717405358,1717405361,1717405362,1717405364,1717405365,0 2024-06-03T09:06:13.821 INFO:tasks.workunit.client.0.smithi088.stdout: api_tier_pp: got ls 1717405358,1717405361,1717405362,1717405364,1717405365,0 2024-06-03T09:06:13.821 INFO:tasks.workunit.client.0.smithi088.stdout: api_tier_pp: got ls 1717405358,1717405361,1717405362,1717405364,1717405365,0 2024-06-03T09:06:13.821 INFO:tasks.workunit.client.0.smithi088.stdout: api_tier_pp: got ls 1717405358,1717405361,1717405362,1717405364,1717405365,1717405367,1717405368,0 2024-06-03T09:06:13.822 INFO:tasks.workunit.client.0.smithi088.stdout: api_tier_pp: got ls 1717405358,1717405361,1717405362,1717405364,1717405365,1717405367,1717405368,0 2024-06-03T09:06:13.822 INFO:tasks.workunit.client.0.smithi088.stdout: api_tier_pp: got ls 1717405358,1717405361,1717405362,1717405364,1717405365,1717405367,1717405368,0 2024-06-03T09:06:13.822 INFO:tasks.workunit.client.0.smithi088.stdout: api_tier_pp: got ls 1717405362,1717405364,1717405365,1717405367,1717405368,1717405370,1717405371,0 2024-06-03T09:06:13.822 INFO:tasks.workunit.client.0.smithi088.stdout: api_tier_pp: first now 1717405362, trimmed 2024-06-03T09:06:13.822 INFO:tasks.workunit.client.0.smithi088.stdout: api_tier_pp: [ OK ] LibRadosTwoPoolsECPP.HitSetTrim (21906 ms) 2024-06-03T09:06:13.822 INFO:tasks.workunit.client.0.smithi088.stdout: api_tier_pp: [ RUN ] LibRadosTwoPoolsECPP.PromoteOn2ndRead 2024-06-03T09:06:13.822 INFO:tasks.workunit.client.0.smithi088.stdout: api_tier_pp: foo0 2024-06-03T09:06:13.822 INFO:tasks.workunit.client.0.smithi088.stdout: api_tier_pp: verifying foo0 is eventually promoted 2024-06-03T09:06:13.822 INFO:tasks.workunit.client.0.smithi088.stdout: api_tier_pp: [ OK ] LibRadosTwoPoolsECPP.PromoteOn2ndRead (15605 ms) 2024-06-03T09:06:13.822 INFO:tasks.workunit.client.0.smithi088.stdout: api_tier_pp: [ RUN ] LibRadosTwoPoolsECPP.ProxyRead 2024-06-03T09:06:13.822 INFO:tasks.workunit.client.0.smithi088.stdout: api_tier_pp: [ OK ] LibRadosTwoPoolsECPP.ProxyRead (26463 ms) 2024-06-03T09:06:13.822 INFO:tasks.workunit.client.0.smithi088.stdout: api_tier_pp: [ RUN ] LibRadosTwoPoolsECPP.CachePin 2024-06-03T09:06:13.822 INFO:tasks.workunit.client.0.smithi088.stdout: api_tier_pp: [ OK ] LibRadosTwoPoolsECPP.CachePin (50061 ms) 2024-06-03T09:06:13.823 INFO:tasks.workunit.client.0.smithi088.stdout: api_tier_pp: [ RUN ] LibRadosTwoPoolsECPP.SetRedirectRead 2024-06-03T09:06:13.823 INFO:tasks.workunit.client.0.smithi088.stdout: api_tier_pp: [ OK ] LibRadosTwoPoolsECPP.SetRedirectRead (5672 ms) 2024-06-03T09:06:13.823 INFO:tasks.workunit.client.0.smithi088.stdout: api_tier_pp: [ RUN ] LibRadosTwoPoolsECPP.SetChunkRead 2024-06-03T09:06:13.823 INFO:tasks.workunit.client.0.smithi088.stdout: api_tier_pp: [ OK ] LibRadosTwoPoolsECPP.SetChunkRead (4624 ms) 2024-06-03T09:06:13.823 INFO:tasks.workunit.client.0.smithi088.stdout: api_tier_pp: [ RUN ] LibRadosTwoPoolsECPP.ManifestPromoteRead 2024-06-03T09:06:13.823 INFO:tasks.workunit.client.0.smithi088.stdout: api_tier_pp: [ OK ] LibRadosTwoPoolsECPP.ManifestPromoteRead (90669 ms) 2024-06-03T09:06:13.823 INFO:tasks.workunit.client.0.smithi088.stdout: api_tier_pp: [ RUN ] LibRadosTwoPoolsECPP.TrySetDedupTier 2024-06-03T09:06:13.823 INFO:tasks.workunit.client.0.smithi088.stdout: api_tier_pp: [ OK ] LibRadosTwoPoolsECPP.TrySetDedupTier (4555 ms) 2024-06-03T09:06:13.823 INFO:tasks.workunit.client.0.smithi088.stdout: api_tier_pp: [----------] 22 tests from LibRadosTwoPoolsECPP (409779 ms total) 2024-06-03T09:06:13.823 INFO:tasks.workunit.client.0.smithi088.stdout: api_tier_pp: 2024-06-03T09:06:13.823 INFO:tasks.workunit.client.0.smithi088.stdout: api_tier_pp: [----------] Global test environment tear-down 2024-06-03T09:06:13.823 INFO:tasks.workunit.client.0.smithi088.stdout: api_tier_pp: [==========] 77 tests from 4 test suites ran. (1700330 ms total) 2024-06-03T09:06:13.823 INFO:tasks.workunit.client.0.smithi088.stdout: api_tier_pp: [ PASSED ] 77 tests. 2024-06-03T09:06:13.827 INFO:tasks.workunit.client.0.smithi088.stderr:+ for t in "${!pids[@]}" 2024-06-03T09:06:13.827 INFO:tasks.workunit.client.0.smithi088.stderr:+ pid=40246 2024-06-03T09:06:13.827 INFO:tasks.workunit.client.0.smithi088.stderr:+ wait 40246 2024-06-03T09:06:13.827 INFO:tasks.workunit.client.0.smithi088.stderr:+ for t in "${!pids[@]}" 2024-06-03T09:06:13.827 INFO:tasks.workunit.client.0.smithi088.stderr:+ pid=40415 2024-06-03T09:06:13.827 INFO:tasks.workunit.client.0.smithi088.stderr:+ wait 40415 2024-06-03T09:06:13.827 INFO:tasks.workunit.client.0.smithi088.stderr:+ for t in "${!pids[@]}" 2024-06-03T09:06:13.827 INFO:tasks.workunit.client.0.smithi088.stderr:+ pid=40950 2024-06-03T09:06:13.827 INFO:tasks.workunit.client.0.smithi088.stderr:+ wait 40950 2024-06-03T09:06:13.828 INFO:tasks.workunit.client.0.smithi088.stderr:+ for t in "${!pids[@]}" 2024-06-03T09:06:13.828 INFO:tasks.workunit.client.0.smithi088.stderr:+ pid=40648 2024-06-03T09:06:13.828 INFO:tasks.workunit.client.0.smithi088.stderr:+ wait 40648 2024-06-03T09:06:13.828 INFO:tasks.workunit.client.0.smithi088.stderr:+ for t in "${!pids[@]}" 2024-06-03T09:06:13.828 INFO:tasks.workunit.client.0.smithi088.stderr:+ pid=40564 2024-06-03T09:06:13.828 INFO:tasks.workunit.client.0.smithi088.stderr:+ wait 40564 2024-06-03T09:06:13.828 INFO:tasks.workunit.client.0.smithi088.stderr:+ for t in "${!pids[@]}" 2024-06-03T09:06:13.828 INFO:tasks.workunit.client.0.smithi088.stderr:+ pid=40322 2024-06-03T09:06:13.828 INFO:tasks.workunit.client.0.smithi088.stderr:+ wait 40322 2024-06-03T09:06:13.828 INFO:tasks.workunit.client.0.smithi088.stderr:+ for t in "${!pids[@]}" 2024-06-03T09:06:13.828 INFO:tasks.workunit.client.0.smithi088.stderr:+ pid=40610 2024-06-03T09:06:13.828 INFO:tasks.workunit.client.0.smithi088.stderr:+ wait 40610 2024-06-03T09:06:13.828 INFO:tasks.workunit.client.0.smithi088.stderr:+ for t in "${!pids[@]}" 2024-06-03T09:06:13.829 INFO:tasks.workunit.client.0.smithi088.stderr:+ pid=40347 2024-06-03T09:06:13.829 INFO:tasks.workunit.client.0.smithi088.stderr:+ wait 40347 2024-06-03T09:06:13.829 INFO:tasks.workunit.client.0.smithi088.stderr:+ for t in "${!pids[@]}" 2024-06-03T09:06:13.829 INFO:tasks.workunit.client.0.smithi088.stderr:+ pid=40886 2024-06-03T09:06:13.829 INFO:tasks.workunit.client.0.smithi088.stderr:+ wait 40886 2024-06-03T09:06:13.829 INFO:tasks.workunit.client.0.smithi088.stderr:+ for t in "${!pids[@]}" 2024-06-03T09:06:13.829 INFO:tasks.workunit.client.0.smithi088.stderr:+ pid=40221 2024-06-03T09:06:13.829 INFO:tasks.workunit.client.0.smithi088.stderr:+ wait 40221 2024-06-03T09:06:13.829 INFO:tasks.workunit.client.0.smithi088.stderr:+ for t in "${!pids[@]}" 2024-06-03T09:06:13.829 INFO:tasks.workunit.client.0.smithi088.stderr:+ pid=40284 2024-06-03T09:06:13.829 INFO:tasks.workunit.client.0.smithi088.stderr:+ wait 40284 2024-06-03T09:06:13.829 INFO:tasks.workunit.client.0.smithi088.stderr:+ for t in "${!pids[@]}" 2024-06-03T09:06:13.830 INFO:tasks.workunit.client.0.smithi088.stderr:+ pid=40899 2024-06-03T09:06:13.830 INFO:tasks.workunit.client.0.smithi088.stderr:+ wait 40899 2024-06-03T09:06:13.830 INFO:tasks.workunit.client.0.smithi088.stderr:+ for t in "${!pids[@]}" 2024-06-03T09:06:13.830 INFO:tasks.workunit.client.0.smithi088.stderr:+ pid=40212 2024-06-03T09:06:13.830 INFO:tasks.workunit.client.0.smithi088.stderr:+ wait 40212 2024-06-03T09:06:13.830 INFO:tasks.workunit.client.0.smithi088.stderr:+ for t in "${!pids[@]}" 2024-06-03T09:06:13.830 INFO:tasks.workunit.client.0.smithi088.stderr:+ pid=40232 2024-06-03T09:06:13.830 INFO:tasks.workunit.client.0.smithi088.stderr:+ wait 40232 2024-06-03T09:06:13.830 INFO:tasks.workunit.client.0.smithi088.stderr:+ for t in "${!pids[@]}" 2024-06-03T09:06:13.830 INFO:tasks.workunit.client.0.smithi088.stderr:+ pid=40461 2024-06-03T09:06:13.830 INFO:tasks.workunit.client.0.smithi088.stderr:+ wait 40461 2024-06-03T09:06:13.830 INFO:tasks.workunit.client.0.smithi088.stderr:+ for t in "${!pids[@]}" 2024-06-03T09:06:13.830 INFO:tasks.workunit.client.0.smithi088.stderr:+ pid=40590 2024-06-03T09:06:13.831 INFO:tasks.workunit.client.0.smithi088.stderr:+ wait 40590 2024-06-03T09:06:13.831 INFO:tasks.workunit.client.0.smithi088.stderr:+ for t in "${!pids[@]}" 2024-06-03T09:06:13.831 INFO:tasks.workunit.client.0.smithi088.stderr:+ pid=40512 2024-06-03T09:06:13.831 INFO:tasks.workunit.client.0.smithi088.stderr:+ wait 40512 2024-06-03T09:06:13.831 INFO:tasks.workunit.client.0.smithi088.stderr:+ for t in "${!pids[@]}" 2024-06-03T09:06:13.831 INFO:tasks.workunit.client.0.smithi088.stderr:+ pid=40936 2024-06-03T09:06:13.831 INFO:tasks.workunit.client.0.smithi088.stderr:+ wait 40936 2024-06-03T09:06:13.831 INFO:tasks.workunit.client.0.smithi088.stderr:+ for t in "${!pids[@]}" 2024-06-03T09:06:13.831 INFO:tasks.workunit.client.0.smithi088.stderr:+ pid=40836 2024-06-03T09:06:13.831 INFO:tasks.workunit.client.0.smithi088.stderr:+ wait 40836 2024-06-03T09:06:13.831 INFO:tasks.workunit.client.0.smithi088.stderr:+ for t in "${!pids[@]}" 2024-06-03T09:06:13.831 INFO:tasks.workunit.client.0.smithi088.stderr:+ pid=40375 2024-06-03T09:06:13.831 INFO:tasks.workunit.client.0.smithi088.stderr:+ wait 40375 2024-06-03T09:06:13.832 INFO:tasks.workunit.client.0.smithi088.stderr:+ for t in "${!pids[@]}" 2024-06-03T09:06:13.832 INFO:tasks.workunit.client.0.smithi088.stderr:+ pid=40685 2024-06-03T09:06:13.832 INFO:tasks.workunit.client.0.smithi088.stderr:+ wait 40685 2024-06-03T09:06:13.832 INFO:tasks.workunit.client.0.smithi088.stderr:+ for t in "${!pids[@]}" 2024-06-03T09:06:13.832 INFO:tasks.workunit.client.0.smithi088.stderr:+ pid=40205 2024-06-03T09:06:13.832 INFO:tasks.workunit.client.0.smithi088.stderr:+ wait 40205 2024-06-03T09:06:13.832 INFO:tasks.workunit.client.0.smithi088.stderr:+ for t in "${!pids[@]}" 2024-06-03T09:06:13.832 INFO:tasks.workunit.client.0.smithi088.stderr:+ pid=40263 2024-06-03T09:06:13.832 INFO:tasks.workunit.client.0.smithi088.stderr:+ wait 40263 2024-06-03T09:06:13.832 INFO:tasks.workunit.client.0.smithi088.stderr:+ for t in "${!pids[@]}" 2024-06-03T09:06:13.832 INFO:tasks.workunit.client.0.smithi088.stderr:+ pid=40634 2024-06-03T09:06:13.832 INFO:tasks.workunit.client.0.smithi088.stderr:+ wait 40634 2024-06-03T09:06:13.832 INFO:tasks.workunit.client.0.smithi088.stderr:+ for t in "${!pids[@]}" 2024-06-03T09:06:13.832 INFO:tasks.workunit.client.0.smithi088.stderr:+ pid=40304 2024-06-03T09:06:13.833 INFO:tasks.workunit.client.0.smithi088.stderr:+ wait 40304 2024-06-03T09:06:13.833 INFO:tasks.workunit.client.0.smithi088.stderr:+ for t in "${!pids[@]}" 2024-06-03T09:06:13.833 INFO:tasks.workunit.client.0.smithi088.stderr:+ pid=40723 2024-06-03T09:06:13.833 INFO:tasks.workunit.client.0.smithi088.stderr:+ wait 40723 2024-06-03T09:06:13.833 INFO:tasks.workunit.client.0.smithi088.stderr:+ for t in "${!pids[@]}" 2024-06-03T09:06:13.833 INFO:tasks.workunit.client.0.smithi088.stderr:+ pid=40764 2024-06-03T09:06:13.833 INFO:tasks.workunit.client.0.smithi088.stderr:+ wait 40764 2024-06-03T09:06:13.833 INFO:tasks.workunit.client.0.smithi088.stderr:+ exit 0 2024-06-03T09:06:13.833 INFO:tasks.workunit.client.0.smithi088.stderr:+ cleanup 2024-06-03T09:06:13.833 INFO:tasks.workunit.client.0.smithi088.stderr:+ pkill -P 40200 2024-06-03T09:06:13.911 INFO:tasks.workunit.client.0.smithi088.stderr:+ true 2024-06-03T09:06:13.912 INFO:teuthology.orchestra.run:Running command with timeout 3600 2024-06-03T09:06:13.912 DEBUG:teuthology.orchestra.run.smithi088:> sudo rm -rf -- /home/ubuntu/cephtest/mnt.0/client.0/tmp 2024-06-03T09:06:13.941 INFO:tasks.workunit:Stopping ['rados/test.sh'] on client.0... 2024-06-03T09:06:13.941 DEBUG:teuthology.orchestra.run.smithi088:> sudo rm -rf -- /home/ubuntu/cephtest/workunits.list.client.0 /home/ubuntu/cephtest/clone.client.0 2024-06-03T09:06:14.266 DEBUG:teuthology.parallel:result is None 2024-06-03T09:06:14.266 DEBUG:teuthology.orchestra.run.smithi088:> sudo rm -rf -- /home/ubuntu/cephtest/mnt.0/client.0 2024-06-03T09:06:14.291 INFO:tasks.workunit:Deleted dir /home/ubuntu/cephtest/mnt.0/client.0 2024-06-03T09:06:14.291 DEBUG:teuthology.orchestra.run.smithi088:> rmdir -- /home/ubuntu/cephtest/mnt.0 2024-06-03T09:06:14.355 INFO:tasks.workunit:Deleted artificial mount point /home/ubuntu/cephtest/mnt.0/client.0 2024-06-03T09:06:14.355 DEBUG:teuthology.run_tasks:Unwinding manager ceph 2024-06-03T09:06:14.365 INFO:tasks.ceph.ceph_manager.ceph:waiting for clean 2024-06-03T09:06:14.365 DEBUG:teuthology.orchestra.run.smithi088:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph pg dump --format=json 2024-06-03T09:06:14.503 INFO:teuthology.orchestra.run.smithi088.stderr:2024-06-03T09:06:14.503+0000 7f1d33687640 10 monclient: get_monmap_and_config 2024-06-03T09:06:14.504 INFO:teuthology.orchestra.run.smithi088.stderr:2024-06-03T09:06:14.503+0000 7f1d33687640 10 monclient: build_initial_monmap 2024-06-03T09:06:14.504 INFO:teuthology.orchestra.run.smithi088.stderr:2024-06-03T09:06:14.503+0000 7f1d33687640 10 monclient: monmap: 2024-06-03T09:06:14.504 INFO:teuthology.orchestra.run.smithi088.stderr:epoch 0 2024-06-03T09:06:14.504 INFO:teuthology.orchestra.run.smithi088.stderr:fsid ec3c3fdd-3ac6-4d7d-9412-683130ed8b04 2024-06-03T09:06:14.504 INFO:teuthology.orchestra.run.smithi088.stderr:last_changed 2024-06-03T09:06:14.504924+0000 2024-06-03T09:06:14.504 INFO:teuthology.orchestra.run.smithi088.stderr:created 2024-06-03T09:06:14.504924+0000 2024-06-03T09:06:14.504 INFO:teuthology.orchestra.run.smithi088.stderr:min_mon_release 0 (unknown) 2024-06-03T09:06:14.505 INFO:teuthology.orchestra.run.smithi088.stderr:election_strategy: 3 2024-06-03T09:06:14.505 INFO:teuthology.orchestra.run.smithi088.stderr:0: v1:172.21.15.88:6789/0 mon.noname-a 2024-06-03T09:06:14.505 INFO:teuthology.orchestra.run.smithi088.stderr:1: v1:172.21.15.192:6789/0 mon.noname-c 2024-06-03T09:06:14.506 INFO:teuthology.orchestra.run.smithi088.stderr:2: v1:172.21.15.88:6790/0 mon.noname-b 2024-06-03T09:06:14.506 INFO:teuthology.orchestra.run.smithi088.stderr: 2024-06-03T09:06:14.507 INFO:teuthology.orchestra.run.smithi088.stderr:2024-06-03T09:06:14.504+0000 7f1d33687640 1 Processor -- start 2024-06-03T09:06:14.507 INFO:teuthology.orchestra.run.smithi088.stderr:2024-06-03T09:06:14.504+0000 7f1d33687640 1 -- start start 2024-06-03T09:06:14.507 INFO:teuthology.orchestra.run.smithi088.stderr:2024-06-03T09:06:14.504+0000 7f1d33687640 10 monclient: init 2024-06-03T09:06:14.507 INFO:teuthology.orchestra.run.smithi088.stderr:2024-06-03T09:06:14.505+0000 7f1d33687640 10 monclient: _reopen_session rank -1 2024-06-03T09:06:14.507 INFO:teuthology.orchestra.run.smithi088.stderr:2024-06-03T09:06:14.505+0000 7f1d33687640 10 monclient: _add_conns ranks=[0,2,1] 2024-06-03T09:06:14.507 INFO:teuthology.orchestra.run.smithi088.stderr:2024-06-03T09:06:14.505+0000 7f1d33687640 10 monclient(hunting): picked mon.noname-a con 0x7f1d2c0adb90 addr v1:172.21.15.88:6789/0 2024-06-03T09:06:14.507 INFO:teuthology.orchestra.run.smithi088.stderr:2024-06-03T09:06:14.505+0000 7f1d33687640 10 monclient(hunting): picked mon.noname-b con 0x7f1d2c06ee00 addr v1:172.21.15.88:6790/0 2024-06-03T09:06:14.507 INFO:teuthology.orchestra.run.smithi088.stderr:2024-06-03T09:06:14.505+0000 7f1d33687640 10 monclient(hunting): picked mon.noname-c con 0x7f1d2c1383e0 addr v1:172.21.15.192:6789/0 2024-06-03T09:06:14.507 INFO:teuthology.orchestra.run.smithi088.stderr:2024-06-03T09:06:14.505+0000 7f1d33687640 1 -- --> v1:172.21.15.88:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f1d2c06f2c0 con 0x7f1d2c0adb90 2024-06-03T09:06:14.507 INFO:teuthology.orchestra.run.smithi088.stderr:2024-06-03T09:06:14.505+0000 7f1d33687640 1 -- --> v1:172.21.15.192:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f1d2c0710c0 con 0x7f1d2c1383e0 2024-06-03T09:06:14.508 INFO:teuthology.orchestra.run.smithi088.stderr:2024-06-03T09:06:14.505+0000 7f1d33687640 1 -- --> v1:172.21.15.88:6790/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f1d2c071fd0 con 0x7f1d2c06ee00 2024-06-03T09:06:14.508 INFO:teuthology.orchestra.run.smithi088.stderr:2024-06-03T09:06:14.505+0000 7f1d33687640 10 monclient(hunting): _renew_subs 2024-06-03T09:06:14.508 INFO:teuthology.orchestra.run.smithi088.stderr:2024-06-03T09:06:14.505+0000 7f1d33687640 10 monclient(hunting): authenticate will time out at 3030.315389s 2024-06-03T09:06:14.508 INFO:teuthology.orchestra.run.smithi088.stderr:2024-06-03T09:06:14.505+0000 7f1d31423640 1 --1- >> v1:172.21.15.88:6789/0 conn(0x7f1d2c0adb90 0x7f1d2c0718c0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.88:6789/0 says I am v1:172.21.15.88:53898/0 (socket says 172.21.15.88:53898) 2024-06-03T09:06:14.508 INFO:teuthology.orchestra.run.smithi088.stderr:2024-06-03T09:06:14.505+0000 7f1d31423640 1 -- 172.21.15.88:0/2792702018 learned_addr learned my addr 172.21.15.88:0/2792702018 (peer_addr_for_me v1:172.21.15.88:0/0) 2024-06-03T09:06:14.509 INFO:teuthology.orchestra.run.smithi088.stderr:2024-06-03T09:06:14.508+0000 7f1d23fff640 1 -- 172.21.15.88:0/2792702018 <== mon.0 v1:172.21.15.88:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 2022297320 0 0) 0x7f1d2c06f2c0 con 0x7f1d2c0adb90 2024-06-03T09:06:14.509 INFO:teuthology.orchestra.run.smithi088.stderr:2024-06-03T09:06:14.508+0000 7f1d23fff640 10 monclient(hunting): _negotiate 2024-06-03T09:06:14.509 INFO:teuthology.orchestra.run.smithi088.stderr:2024-06-03T09:06:14.508+0000 7f1d23fff640 10 monclient(hunting): _init_auth method 2 2024-06-03T09:06:14.509 INFO:teuthology.orchestra.run.smithi088.stderr:2024-06-03T09:06:14.508+0000 7f1d23fff640 10 monclient(hunting): _init_auth creating new auth 2024-06-03T09:06:14.509 INFO:teuthology.orchestra.run.smithi088.stderr:2024-06-03T09:06:14.508+0000 7f1d23fff640 10 monclient(hunting): my global_id is 30549 2024-06-03T09:06:14.509 INFO:teuthology.orchestra.run.smithi088.stderr:2024-06-03T09:06:14.508+0000 7f1d23fff640 1 -- 172.21.15.88:0/2792702018 --> v1:172.21.15.88:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f1d18003520 con 0x7f1d2c0adb90 2024-06-03T09:06:14.509 INFO:teuthology.orchestra.run.smithi088.stderr:2024-06-03T09:06:14.508+0000 7f1d23fff640 1 -- 172.21.15.88:0/2792702018 <== mon.2 v1:172.21.15.88:6790/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 4155466041 0 0) 0x7f1d2c071fd0 con 0x7f1d2c06ee00 2024-06-03T09:06:14.509 INFO:teuthology.orchestra.run.smithi088.stderr:2024-06-03T09:06:14.508+0000 7f1d23fff640 10 monclient(hunting): _negotiate 2024-06-03T09:06:14.509 INFO:teuthology.orchestra.run.smithi088.stderr:2024-06-03T09:06:14.508+0000 7f1d23fff640 10 monclient(hunting): _init_auth method 2 2024-06-03T09:06:14.510 INFO:teuthology.orchestra.run.smithi088.stderr:2024-06-03T09:06:14.508+0000 7f1d23fff640 10 monclient(hunting): _init_auth creating new auth 2024-06-03T09:06:14.510 INFO:teuthology.orchestra.run.smithi088.stderr:2024-06-03T09:06:14.508+0000 7f1d23fff640 10 monclient(hunting): my global_id is 30050 2024-06-03T09:06:14.510 INFO:teuthology.orchestra.run.smithi088.stderr:2024-06-03T09:06:14.508+0000 7f1d23fff640 1 -- 172.21.15.88:0/2792702018 --> v1:172.21.15.88:6790/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f1d2c06f2c0 con 0x7f1d2c06ee00 2024-06-03T09:06:14.510 INFO:teuthology.orchestra.run.smithi088.stderr:2024-06-03T09:06:14.508+0000 7f1d23fff640 1 -- 172.21.15.88:0/2792702018 <== mon.1 v1:172.21.15.192:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 3540337830 0 0) 0x7f1d2c0710c0 con 0x7f1d2c1383e0 2024-06-03T09:06:14.510 INFO:teuthology.orchestra.run.smithi088.stderr:2024-06-03T09:06:14.508+0000 7f1d23fff640 10 monclient(hunting): _negotiate 2024-06-03T09:06:14.510 INFO:teuthology.orchestra.run.smithi088.stderr:2024-06-03T09:06:14.508+0000 7f1d23fff640 10 monclient(hunting): _init_auth method 2 2024-06-03T09:06:14.510 INFO:teuthology.orchestra.run.smithi088.stderr:2024-06-03T09:06:14.508+0000 7f1d23fff640 10 monclient(hunting): _init_auth creating new auth 2024-06-03T09:06:14.510 INFO:teuthology.orchestra.run.smithi088.stderr:2024-06-03T09:06:14.508+0000 7f1d23fff640 10 monclient(hunting): my global_id is 30127 2024-06-03T09:06:14.510 INFO:teuthology.orchestra.run.smithi088.stderr:2024-06-03T09:06:14.508+0000 7f1d23fff640 1 -- 172.21.15.88:0/2792702018 --> v1:172.21.15.192:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f1d2c071fd0 con 0x7f1d2c1383e0 2024-06-03T09:06:14.512 INFO:teuthology.orchestra.run.smithi088.stderr:2024-06-03T09:06:14.512+0000 7f1d23fff640 1 -- 172.21.15.88:0/2792702018 <== mon.1 v1:172.21.15.192:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 214+0+0 (unknown 1539244066 0 0) 0x7f1d2c071fd0 con 0x7f1d2c1383e0 2024-06-03T09:06:14.512 INFO:teuthology.orchestra.run.smithi088.stderr:2024-06-03T09:06:14.512+0000 7f1d23fff640 1 -- 172.21.15.88:0/2792702018 >> v1:172.21.15.88:6790/0 conn(0x7f1d2c06ee00 legacy=0x7f1d2c137cd0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-06-03T09:06:14.512 INFO:teuthology.orchestra.run.smithi088.stderr:2024-06-03T09:06:14.512+0000 7f1d23fff640 1 -- 172.21.15.88:0/2792702018 >> v1:172.21.15.88:6789/0 conn(0x7f1d2c0adb90 legacy=0x7f1d2c0718c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-06-03T09:06:14.512 INFO:teuthology.orchestra.run.smithi088.stderr:2024-06-03T09:06:14.512+0000 7f1d23fff640 10 monclient: _finish_hunting 0 2024-06-03T09:06:14.512 INFO:teuthology.orchestra.run.smithi088.stderr:2024-06-03T09:06:14.512+0000 7f1d23fff640 1 monclient: found mon.noname-c 2024-06-03T09:06:14.512 INFO:teuthology.orchestra.run.smithi088.stderr:2024-06-03T09:06:14.512+0000 7f1d23fff640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-06-03T09:06:14.512 INFO:teuthology.orchestra.run.smithi088.stderr:2024-06-03T09:06:14.512+0000 7f1d23fff640 10 monclient: _send_mon_message to mon.noname-c at v1:172.21.15.192:6789/0 2024-06-03T09:06:14.512 INFO:teuthology.orchestra.run.smithi088.stderr:2024-06-03T09:06:14.512+0000 7f1d23fff640 1 -- 172.21.15.88:0/2792702018 --> v1:172.21.15.192:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f1d2c141750 con 0x7f1d2c1383e0 2024-06-03T09:06:14.512 INFO:teuthology.orchestra.run.smithi088.stderr:2024-06-03T09:06:14.512+0000 7f1d23fff640 10 monclient: _finish_auth 0 2024-06-03T09:06:14.513 INFO:teuthology.orchestra.run.smithi088.stderr:2024-06-03T09:06:14.512+0000 7f1d23fff640 10 monclient: _check_auth_tickets 2024-06-03T09:06:14.513 INFO:teuthology.orchestra.run.smithi088.stderr:2024-06-03T09:06:14.512+0000 7f1d23fff640 20 monclient: _check_auth_rotating not needed by client.admin 2024-06-03T09:06:14.513 INFO:teuthology.orchestra.run.smithi088.stderr:2024-06-03T09:06:14.512+0000 7f1d23fff640 1 -- 172.21.15.88:0/2792702018 <== mon.1 v1:172.21.15.192:6789/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 2234374065 0 0) 0x7f1d24001930 con 0x7f1d2c1383e0 2024-06-03T09:06:14.513 INFO:teuthology.orchestra.run.smithi088.stderr:2024-06-03T09:06:14.512+0000 7f1d23fff640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-06-03T09:06:14.513 INFO:teuthology.orchestra.run.smithi088.stderr:2024-06-03T09:06:14.512+0000 7f1d23fff640 10 monclient: got monmap 1 from mon.noname-c (according to old e1) 2024-06-03T09:06:14.513 INFO:teuthology.orchestra.run.smithi088.stderr:2024-06-03T09:06:14.512+0000 7f1d23fff640 10 monclient: dump: 2024-06-03T09:06:14.513 INFO:teuthology.orchestra.run.smithi088.stderr:epoch 1 2024-06-03T09:06:14.513 INFO:teuthology.orchestra.run.smithi088.stderr:fsid ec3c3fdd-3ac6-4d7d-9412-683130ed8b04 2024-06-03T09:06:14.513 INFO:teuthology.orchestra.run.smithi088.stderr:last_changed 2024-06-03T08:34:31.885125+0000 2024-06-03T09:06:14.513 INFO:teuthology.orchestra.run.smithi088.stderr:created 2024-06-03T08:34:31.885125+0000 2024-06-03T09:06:14.513 INFO:teuthology.orchestra.run.smithi088.stderr:min_mon_release 18 (reef) 2024-06-03T09:06:14.513 INFO:teuthology.orchestra.run.smithi088.stderr:election_strategy: 3 2024-06-03T09:06:14.514 INFO:teuthology.orchestra.run.smithi088.stderr:0: v1:172.21.15.88:6789/0 mon.a 2024-06-03T09:06:14.514 INFO:teuthology.orchestra.run.smithi088.stderr:1: v1:172.21.15.192:6789/0 mon.b 2024-06-03T09:06:14.514 INFO:teuthology.orchestra.run.smithi088.stderr:2: v1:172.21.15.88:6790/0 mon.c 2024-06-03T09:06:14.514 INFO:teuthology.orchestra.run.smithi088.stderr: 2024-06-03T09:06:14.514 INFO:teuthology.orchestra.run.smithi088.stderr:2024-06-03T09:06:14.512+0000 7f1d33687640 5 monclient: authenticate success, global_id 30127 2024-06-03T09:06:14.514 INFO:teuthology.orchestra.run.smithi088.stderr:2024-06-03T09:06:14.512+0000 7f1d33687640 20 monclient: get_monmap_and_config waiting for monmap|config 2024-06-03T09:06:14.517 INFO:teuthology.orchestra.run.smithi088.stderr:2024-06-03T09:06:14.513+0000 7f1d23fff640 1 -- 172.21.15.88:0/2792702018 <== mon.1 v1:172.21.15.192:6789/0 4 ==== config(0 keys) v1 ==== 4+0+0 (unknown 0 0 0) 0x7f1d24001c10 con 0x7f1d2c1383e0 2024-06-03T09:06:14.517 INFO:teuthology.orchestra.run.smithi088.stderr:2024-06-03T09:06:14.513+0000 7f1d23fff640 10 monclient: handle_config config(0 keys) v1 2024-06-03T09:06:14.517 INFO:teuthology.orchestra.run.smithi088.stderr:2024-06-03T09:06:14.513+0000 7f1d33687640 10 monclient: get_monmap_and_config success 2024-06-03T09:06:14.517 INFO:teuthology.orchestra.run.smithi088.stderr:2024-06-03T09:06:14.513+0000 7f1d33687640 4 set_mon_vals no callback set 2024-06-03T09:06:14.517 INFO:teuthology.orchestra.run.smithi088.stderr:2024-06-03T09:06:14.514+0000 7f1d23fff640 1 -- 172.21.15.88:0/2792702018 <== mon.1 v1:172.21.15.192:6789/0 5 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 2234374065 0 0) 0x7f1d24001fa0 con 0x7f1d2c1383e0 2024-06-03T09:06:14.517 INFO:teuthology.orchestra.run.smithi088.stderr:2024-06-03T09:06:14.514+0000 7f1d23fff640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-06-03T09:06:14.517 INFO:teuthology.orchestra.run.smithi088.stderr:2024-06-03T09:06:14.514+0000 7f1d23fff640 10 monclient: got monmap 1 from mon.b (according to old e1) 2024-06-03T09:06:14.518 INFO:teuthology.orchestra.run.smithi088.stderr:2024-06-03T09:06:14.514+0000 7f1d23fff640 10 monclient: dump: 2024-06-03T09:06:14.518 INFO:teuthology.orchestra.run.smithi088.stderr:epoch 1 2024-06-03T09:06:14.518 INFO:teuthology.orchestra.run.smithi088.stderr:fsid ec3c3fdd-3ac6-4d7d-9412-683130ed8b04 2024-06-03T09:06:14.518 INFO:teuthology.orchestra.run.smithi088.stderr:last_changed 2024-06-03T08:34:31.885125+0000 2024-06-03T09:06:14.518 INFO:teuthology.orchestra.run.smithi088.stderr:created 2024-06-03T08:34:31.885125+0000 2024-06-03T09:06:14.518 INFO:teuthology.orchestra.run.smithi088.stderr:min_mon_release 18 (reef) 2024-06-03T09:06:14.518 INFO:teuthology.orchestra.run.smithi088.stderr:election_strategy: 3 2024-06-03T09:06:14.518 INFO:teuthology.orchestra.run.smithi088.stderr:0: v1:172.21.15.88:6789/0 mon.a 2024-06-03T09:06:14.518 INFO:teuthology.orchestra.run.smithi088.stderr:1: v1:172.21.15.192:6789/0 mon.b 2024-06-03T09:06:14.518 INFO:teuthology.orchestra.run.smithi088.stderr:2: v1:172.21.15.88:6790/0 mon.c 2024-06-03T09:06:14.518 INFO:teuthology.orchestra.run.smithi088.stderr: 2024-06-03T09:06:14.518 INFO:teuthology.orchestra.run.smithi088.stderr:2024-06-03T09:06:14.514+0000 7f1d33687640 10 monclient: shutdown 2024-06-03T09:06:14.518 INFO:teuthology.orchestra.run.smithi088.stderr:2024-06-03T09:06:14.514+0000 7f1d33687640 20 monclient: shutdown discarding 0 pending message(s) 2024-06-03T09:06:14.519 INFO:teuthology.orchestra.run.smithi088.stderr:2024-06-03T09:06:14.514+0000 7f1d33687640 1 -- 172.21.15.88:0/2792702018 >> v1:172.21.15.192:6789/0 conn(0x7f1d2c1383e0 legacy=0x7f1d2c149af0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-06-03T09:06:14.519 INFO:teuthology.orchestra.run.smithi088.stderr:2024-06-03T09:06:14.514+0000 7f1d33687640 1 -- 172.21.15.88:0/2792702018 shutdown_connections 2024-06-03T09:06:14.519 INFO:teuthology.orchestra.run.smithi088.stderr:2024-06-03T09:06:14.514+0000 7f1d33687640 1 -- 172.21.15.88:0/2792702018 >> 172.21.15.88:0/2792702018 conn(0x7f1d2c0b2f20 msgr2=0x7f1d2c0b3300 unknown :-1 s=STATE_NONE l=0).mark_down 2024-06-03T09:06:14.519 INFO:teuthology.orchestra.run.smithi088.stderr:2024-06-03T09:06:14.514+0000 7f1d33687640 1 -- 172.21.15.88:0/2792702018 shutdown_connections 2024-06-03T09:06:14.519 INFO:teuthology.orchestra.run.smithi088.stderr:2024-06-03T09:06:14.514+0000 7f1d33687640 1 -- 172.21.15.88:0/2792702018 wait complete. 2024-06-03T09:06:14.519 INFO:teuthology.orchestra.run.smithi088.stderr:2024-06-03T09:06:14.514+0000 7f1d33687640 10 monclient: build_initial_monmap 2024-06-03T09:06:14.519 INFO:teuthology.orchestra.run.smithi088.stderr:2024-06-03T09:06:14.514+0000 7f1d33687640 10 monclient: monmap: 2024-06-03T09:06:14.519 INFO:teuthology.orchestra.run.smithi088.stderr:epoch 0 2024-06-03T09:06:14.519 INFO:teuthology.orchestra.run.smithi088.stderr:fsid 00000000-0000-0000-0000-000000000000 2024-06-03T09:06:14.519 INFO:teuthology.orchestra.run.smithi088.stderr:last_changed 0.000000 2024-06-03T09:06:14.519 INFO:teuthology.orchestra.run.smithi088.stderr:created 0.000000 2024-06-03T09:06:14.519 INFO:teuthology.orchestra.run.smithi088.stderr:min_mon_release 0 (unknown) 2024-06-03T09:06:14.519 INFO:teuthology.orchestra.run.smithi088.stderr:election_strategy: 1 2024-06-03T09:06:14.520 INFO:teuthology.orchestra.run.smithi088.stderr:0: v1:172.21.15.88:6789/0 mon.noname-a 2024-06-03T09:06:14.520 INFO:teuthology.orchestra.run.smithi088.stderr:1: v1:172.21.15.192:6789/0 mon.noname-b 2024-06-03T09:06:14.520 INFO:teuthology.orchestra.run.smithi088.stderr:2: v1:172.21.15.88:6790/0 mon.noname-c 2024-06-03T09:06:14.520 INFO:teuthology.orchestra.run.smithi088.stderr: 2024-06-03T09:06:14.520 INFO:teuthology.orchestra.run.smithi088.stderr:2024-06-03T09:06:14.515+0000 7f1d33687640 1 librados: starting msgr at 2024-06-03T09:06:14.520 INFO:teuthology.orchestra.run.smithi088.stderr:2024-06-03T09:06:14.515+0000 7f1d33687640 1 librados: starting objecter 2024-06-03T09:06:14.520 INFO:teuthology.orchestra.run.smithi088.stderr:2024-06-03T09:06:14.515+0000 7f1d33687640 1 Processor -- start 2024-06-03T09:06:14.520 INFO:teuthology.orchestra.run.smithi088.stderr:2024-06-03T09:06:14.515+0000 7f1d33687640 1 -- start start 2024-06-03T09:06:14.520 INFO:teuthology.orchestra.run.smithi088.stderr:2024-06-03T09:06:14.515+0000 7f1d33687640 1 librados: setting wanted keys 2024-06-03T09:06:14.520 INFO:teuthology.orchestra.run.smithi088.stderr:2024-06-03T09:06:14.515+0000 7f1d33687640 1 librados: calling monclient init 2024-06-03T09:06:14.520 INFO:teuthology.orchestra.run.smithi088.stderr:2024-06-03T09:06:14.515+0000 7f1d33687640 10 monclient: init 2024-06-03T09:06:14.520 INFO:teuthology.orchestra.run.smithi088.stderr:2024-06-03T09:06:14.515+0000 7f1d33687640 10 monclient: _reopen_session rank -1 2024-06-03T09:06:14.520 INFO:teuthology.orchestra.run.smithi088.stderr:2024-06-03T09:06:14.515+0000 7f1d33687640 10 monclient: _add_conns ranks=[0,2,1] 2024-06-03T09:06:14.520 INFO:teuthology.orchestra.run.smithi088.stderr:2024-06-03T09:06:14.515+0000 7f1d33687640 10 monclient(hunting): picked mon.noname-a con 0x7f1d2c06ee00 addr v1:172.21.15.88:6789/0 2024-06-03T09:06:14.521 INFO:teuthology.orchestra.run.smithi088.stderr:2024-06-03T09:06:14.515+0000 7f1d33687640 10 monclient(hunting): picked mon.noname-c con 0x7f1d2c0adb90 addr v1:172.21.15.88:6790/0 2024-06-03T09:06:14.521 INFO:teuthology.orchestra.run.smithi088.stderr:2024-06-03T09:06:14.515+0000 7f1d33687640 10 monclient(hunting): picked mon.noname-b con 0x7f1d2c1383e0 addr v1:172.21.15.192:6789/0 2024-06-03T09:06:14.521 INFO:teuthology.orchestra.run.smithi088.stderr:2024-06-03T09:06:14.515+0000 7f1d33687640 1 -- --> v1:172.21.15.88:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f1d2c13ef10 con 0x7f1d2c06ee00 2024-06-03T09:06:14.521 INFO:teuthology.orchestra.run.smithi088.stderr:2024-06-03T09:06:14.515+0000 7f1d33687640 1 -- --> v1:172.21.15.192:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f1d2c13f3a0 con 0x7f1d2c1383e0 2024-06-03T09:06:14.521 INFO:teuthology.orchestra.run.smithi088.stderr:2024-06-03T09:06:14.515+0000 7f1d33687640 1 -- --> v1:172.21.15.88:6790/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f1d2c0aa030 con 0x7f1d2c0adb90 2024-06-03T09:06:14.521 INFO:teuthology.orchestra.run.smithi088.stderr:2024-06-03T09:06:14.515+0000 7f1d33687640 10 monclient(hunting): _renew_subs 2024-06-03T09:06:14.521 INFO:teuthology.orchestra.run.smithi088.stderr:2024-06-03T09:06:14.515+0000 7f1d33687640 10 monclient(hunting): authenticate will time out at 3030.325875s 2024-06-03T09:06:14.521 INFO:teuthology.orchestra.run.smithi088.stderr:2024-06-03T09:06:14.516+0000 7f1d31423640 1 --1- >> v1:172.21.15.88:6789/0 conn(0x7f1d2c06ee00 0x7f1d2c0a86c0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=1).handle_server_banner_and_identify peer v1:172.21.15.88:6789/0 says I am v1:172.21.15.88:53914/0 (socket says 172.21.15.88:53914) 2024-06-03T09:06:14.521 INFO:teuthology.orchestra.run.smithi088.stderr:2024-06-03T09:06:14.516+0000 7f1d31423640 1 -- 172.21.15.88:0/1801295370 learned_addr learned my addr 172.21.15.88:0/1801295370 (peer_addr_for_me v1:172.21.15.88:0/0) 2024-06-03T09:06:14.521 INFO:teuthology.orchestra.run.smithi088.stderr:2024-06-03T09:06:14.516+0000 7f1d21ffb640 10 client.?.objecter ms_handle_connect 0x7f1d2c06ee00 2024-06-03T09:06:14.521 INFO:teuthology.orchestra.run.smithi088.stderr:2024-06-03T09:06:14.516+0000 7f1d21ffb640 10 client.?.objecter resend_mon_ops 2024-06-03T09:06:14.521 INFO:teuthology.orchestra.run.smithi088.stderr:2024-06-03T09:06:14.516+0000 7f1d21ffb640 10 client.?.objecter ms_handle_connect 0x7f1d2c1383e0 2024-06-03T09:06:14.521 INFO:teuthology.orchestra.run.smithi088.stderr:2024-06-03T09:06:14.516+0000 7f1d21ffb640 10 client.?.objecter resend_mon_ops 2024-06-03T09:06:14.521 INFO:teuthology.orchestra.run.smithi088.stderr:2024-06-03T09:06:14.516+0000 7f1d21ffb640 10 client.?.objecter ms_handle_connect 0x7f1d2c0adb90 2024-06-03T09:06:14.522 INFO:teuthology.orchestra.run.smithi088.stderr:2024-06-03T09:06:14.516+0000 7f1d21ffb640 10 client.?.objecter resend_mon_ops 2024-06-03T09:06:14.523 INFO:teuthology.orchestra.run.smithi088.stderr:2024-06-03T09:06:14.518+0000 7f1d21ffb640 1 -- 172.21.15.88:0/1801295370 <== mon.2 v1:172.21.15.88:6790/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 2796032034 0 0) 0x7f1d2c0aa030 con 0x7f1d2c0adb90 2024-06-03T09:06:14.523 INFO:teuthology.orchestra.run.smithi088.stderr:2024-06-03T09:06:14.518+0000 7f1d21ffb640 10 monclient(hunting): _negotiate 2024-06-03T09:06:14.523 INFO:teuthology.orchestra.run.smithi088.stderr:2024-06-03T09:06:14.518+0000 7f1d21ffb640 10 monclient(hunting): _init_auth method 2 2024-06-03T09:06:14.523 INFO:teuthology.orchestra.run.smithi088.stderr:2024-06-03T09:06:14.518+0000 7f1d21ffb640 10 monclient(hunting): _init_auth creating new auth 2024-06-03T09:06:14.523 INFO:teuthology.orchestra.run.smithi088.stderr:2024-06-03T09:06:14.518+0000 7f1d21ffb640 10 monclient(hunting): my global_id is 30053 2024-06-03T09:06:14.523 INFO:teuthology.orchestra.run.smithi088.stderr:2024-06-03T09:06:14.518+0000 7f1d21ffb640 1 -- 172.21.15.88:0/1801295370 --> v1:172.21.15.88:6790/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f1d10004eb0 con 0x7f1d2c0adb90 2024-06-03T09:06:14.523 INFO:teuthology.orchestra.run.smithi088.stderr:2024-06-03T09:06:14.518+0000 7f1d21ffb640 1 -- 172.21.15.88:0/1801295370 <== mon.1 v1:172.21.15.192:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 1150881809 0 0) 0x7f1d2c13f3a0 con 0x7f1d2c1383e0 2024-06-03T09:06:14.523 INFO:teuthology.orchestra.run.smithi088.stderr:2024-06-03T09:06:14.518+0000 7f1d21ffb640 10 monclient(hunting): _negotiate 2024-06-03T09:06:14.523 INFO:teuthology.orchestra.run.smithi088.stderr:2024-06-03T09:06:14.518+0000 7f1d21ffb640 10 monclient(hunting): _init_auth method 2 2024-06-03T09:06:14.524 INFO:teuthology.orchestra.run.smithi088.stderr:2024-06-03T09:06:14.518+0000 7f1d21ffb640 10 monclient(hunting): _init_auth creating new auth 2024-06-03T09:06:14.524 INFO:teuthology.orchestra.run.smithi088.stderr:2024-06-03T09:06:14.518+0000 7f1d21ffb640 10 monclient(hunting): my global_id is 30130 2024-06-03T09:06:14.524 INFO:teuthology.orchestra.run.smithi088.stderr:2024-06-03T09:06:14.518+0000 7f1d21ffb640 1 -- 172.21.15.88:0/1801295370 --> v1:172.21.15.192:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f1d2c0aa030 con 0x7f1d2c1383e0 2024-06-03T09:06:14.524 INFO:teuthology.orchestra.run.smithi088.stderr:2024-06-03T09:06:14.518+0000 7f1d21ffb640 1 -- 172.21.15.88:0/1801295370 <== mon.0 v1:172.21.15.88:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 2235494016 0 0) 0x7f1d2c13ef10 con 0x7f1d2c06ee00 2024-06-03T09:06:14.524 INFO:teuthology.orchestra.run.smithi088.stderr:2024-06-03T09:06:14.518+0000 7f1d21ffb640 10 monclient(hunting): _negotiate 2024-06-03T09:06:14.524 INFO:teuthology.orchestra.run.smithi088.stderr:2024-06-03T09:06:14.518+0000 7f1d21ffb640 10 monclient(hunting): _init_auth method 2 2024-06-03T09:06:14.524 INFO:teuthology.orchestra.run.smithi088.stderr:2024-06-03T09:06:14.518+0000 7f1d21ffb640 10 monclient(hunting): _init_auth creating new auth 2024-06-03T09:06:14.524 INFO:teuthology.orchestra.run.smithi088.stderr:2024-06-03T09:06:14.518+0000 7f1d21ffb640 10 monclient(hunting): my global_id is 30552 2024-06-03T09:06:14.524 INFO:teuthology.orchestra.run.smithi088.stderr:2024-06-03T09:06:14.518+0000 7f1d21ffb640 1 -- 172.21.15.88:0/1801295370 --> v1:172.21.15.88:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f1d2c13f3a0 con 0x7f1d2c06ee00 2024-06-03T09:06:14.524 INFO:teuthology.orchestra.run.smithi088.stderr:2024-06-03T09:06:14.523+0000 7f1d21ffb640 1 -- 172.21.15.88:0/1801295370 <== mon.1 v1:172.21.15.192:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 764+0+0 (unknown 1862297585 0 0) 0x7f1d2c0aa030 con 0x7f1d2c1383e0 2024-06-03T09:06:14.524 INFO:teuthology.orchestra.run.smithi088.stderr:2024-06-03T09:06:14.523+0000 7f1d21ffb640 1 -- 172.21.15.88:0/1801295370 --> v1:172.21.15.192:6789/0 -- auth(proto 2 165 bytes epoch 0) v1 -- 0x7f1d2c13ef10 con 0x7f1d2c1383e0 2024-06-03T09:06:14.524 INFO:teuthology.orchestra.run.smithi088.stderr:2024-06-03T09:06:14.523+0000 7f1d21ffb640 1 -- 172.21.15.88:0/1801295370 <== mon.1 v1:172.21.15.192:6789/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 2234374065 0 0) 0x7f1d240018f0 con 0x7f1d2c1383e0 2024-06-03T09:06:14.524 INFO:teuthology.orchestra.run.smithi088.stderr:2024-06-03T09:06:14.523+0000 7f1d21ffb640 10 monclient(hunting): handle_monmap mon_map magic: 0 v1 2024-06-03T09:06:14.525 INFO:teuthology.orchestra.run.smithi088.stderr:2024-06-03T09:06:14.523+0000 7f1d21ffb640 10 monclient(hunting): got monmap 1 from mon.noname-b (according to old e1) 2024-06-03T09:06:14.525 INFO:teuthology.orchestra.run.smithi088.stderr:2024-06-03T09:06:14.523+0000 7f1d21ffb640 10 monclient(hunting): dump: 2024-06-03T09:06:14.525 INFO:teuthology.orchestra.run.smithi088.stderr:epoch 1 2024-06-03T09:06:14.525 INFO:teuthology.orchestra.run.smithi088.stderr:fsid ec3c3fdd-3ac6-4d7d-9412-683130ed8b04 2024-06-03T09:06:14.525 INFO:teuthology.orchestra.run.smithi088.stderr:last_changed 2024-06-03T08:34:31.885125+0000 2024-06-03T09:06:14.525 INFO:teuthology.orchestra.run.smithi088.stderr:created 2024-06-03T08:34:31.885125+0000 2024-06-03T09:06:14.525 INFO:teuthology.orchestra.run.smithi088.stderr:min_mon_release 18 (reef) 2024-06-03T09:06:14.525 INFO:teuthology.orchestra.run.smithi088.stderr:election_strategy: 3 2024-06-03T09:06:14.525 INFO:teuthology.orchestra.run.smithi088.stderr:0: v1:172.21.15.88:6789/0 mon.a 2024-06-03T09:06:14.525 INFO:teuthology.orchestra.run.smithi088.stderr:1: v1:172.21.15.192:6789/0 mon.b 2024-06-03T09:06:14.525 INFO:teuthology.orchestra.run.smithi088.stderr:2: v1:172.21.15.88:6790/0 mon.c 2024-06-03T09:06:14.525 INFO:teuthology.orchestra.run.smithi088.stderr: 2024-06-03T09:06:14.525 INFO:teuthology.orchestra.run.smithi088.stderr:2024-06-03T09:06:14.523+0000 7f1d21ffb640 10 monclient(hunting): _finish_auth 0 2024-06-03T09:06:14.527 INFO:teuthology.orchestra.run.smithi088.stderr:2024-06-03T09:06:14.523+0000 7f1d21ffb640 1 -- 172.21.15.88:0/1801295370 <== mon.2 v1:172.21.15.88:6790/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 764+0+0 (unknown 3649754580 0 0) 0x7f1d10004eb0 con 0x7f1d2c0adb90 2024-06-03T09:06:14.527 INFO:teuthology.orchestra.run.smithi088.stderr:2024-06-03T09:06:14.523+0000 7f1d21ffb640 1 -- 172.21.15.88:0/1801295370 --> v1:172.21.15.88:6790/0 -- auth(proto 2 165 bytes epoch 0) v1 -- 0x7f1d2c0aa030 con 0x7f1d2c0adb90 2024-06-03T09:06:14.527 INFO:teuthology.orchestra.run.smithi088.stderr:2024-06-03T09:06:14.523+0000 7f1d21ffb640 1 -- 172.21.15.88:0/1801295370 <== mon.2 v1:172.21.15.88:6790/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 2234374065 0 0) 0x7f1d1c0018b0 con 0x7f1d2c0adb90 2024-06-03T09:06:14.527 INFO:teuthology.orchestra.run.smithi088.stderr:2024-06-03T09:06:14.523+0000 7f1d21ffb640 10 monclient(hunting): handle_monmap mon_map magic: 0 v1 2024-06-03T09:06:14.527 INFO:teuthology.orchestra.run.smithi088.stderr:2024-06-03T09:06:14.523+0000 7f1d21ffb640 10 monclient(hunting): got monmap 1 from mon.c (according to old e1) 2024-06-03T09:06:14.527 INFO:teuthology.orchestra.run.smithi088.stderr:2024-06-03T09:06:14.523+0000 7f1d21ffb640 10 monclient(hunting): dump: 2024-06-03T09:06:14.527 INFO:teuthology.orchestra.run.smithi088.stderr:epoch 1 2024-06-03T09:06:14.528 INFO:teuthology.orchestra.run.smithi088.stderr:fsid ec3c3fdd-3ac6-4d7d-9412-683130ed8b04 2024-06-03T09:06:14.528 INFO:teuthology.orchestra.run.smithi088.stderr:last_changed 2024-06-03T08:34:31.885125+0000 2024-06-03T09:06:14.528 INFO:teuthology.orchestra.run.smithi088.stderr:created 2024-06-03T08:34:31.885125+0000 2024-06-03T09:06:14.528 INFO:teuthology.orchestra.run.smithi088.stderr:min_mon_release 18 (reef) 2024-06-03T09:06:14.528 INFO:teuthology.orchestra.run.smithi088.stderr:election_strategy: 3 2024-06-03T09:06:14.528 INFO:teuthology.orchestra.run.smithi088.stderr:0: v1:172.21.15.88:6789/0 mon.a 2024-06-03T09:06:14.528 INFO:teuthology.orchestra.run.smithi088.stderr:1: v1:172.21.15.192:6789/0 mon.b 2024-06-03T09:06:14.528 INFO:teuthology.orchestra.run.smithi088.stderr:2: v1:172.21.15.88:6790/0 mon.c 2024-06-03T09:06:14.528 INFO:teuthology.orchestra.run.smithi088.stderr: 2024-06-03T09:06:14.528 INFO:teuthology.orchestra.run.smithi088.stderr:2024-06-03T09:06:14.523+0000 7f1d21ffb640 1 -- 172.21.15.88:0/1801295370 <== mon.0 v1:172.21.15.88:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 764+0+0 (unknown 1720086364 0 0) 0x7f1d2c13f3a0 con 0x7f1d2c06ee00 2024-06-03T09:06:14.528 INFO:teuthology.orchestra.run.smithi088.stderr:2024-06-03T09:06:14.523+0000 7f1d21ffb640 1 -- 172.21.15.88:0/1801295370 --> v1:172.21.15.88:6789/0 -- auth(proto 2 165 bytes epoch 0) v1 -- 0x7f1d10004eb0 con 0x7f1d2c06ee00 2024-06-03T09:06:14.528 INFO:teuthology.orchestra.run.smithi088.stderr:2024-06-03T09:06:14.523+0000 7f1d21ffb640 1 -- 172.21.15.88:0/1801295370 <== mon.0 v1:172.21.15.88:6789/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 2234374065 0 0) 0x7f1d280018d0 con 0x7f1d2c06ee00 2024-06-03T09:06:14.528 INFO:teuthology.orchestra.run.smithi088.stderr:2024-06-03T09:06:14.523+0000 7f1d21ffb640 10 monclient(hunting): handle_monmap mon_map magic: 0 v1 2024-06-03T09:06:14.529 INFO:teuthology.orchestra.run.smithi088.stderr:2024-06-03T09:06:14.523+0000 7f1d21ffb640 10 monclient(hunting): got monmap 1 from mon.a (according to old e1) 2024-06-03T09:06:14.529 INFO:teuthology.orchestra.run.smithi088.stderr:2024-06-03T09:06:14.523+0000 7f1d21ffb640 10 monclient(hunting): dump: 2024-06-03T09:06:14.529 INFO:teuthology.orchestra.run.smithi088.stderr:epoch 1 2024-06-03T09:06:14.529 INFO:teuthology.orchestra.run.smithi088.stderr:fsid ec3c3fdd-3ac6-4d7d-9412-683130ed8b04 2024-06-03T09:06:14.529 INFO:teuthology.orchestra.run.smithi088.stderr:last_changed 2024-06-03T08:34:31.885125+0000 2024-06-03T09:06:14.529 INFO:teuthology.orchestra.run.smithi088.stderr:created 2024-06-03T08:34:31.885125+0000 2024-06-03T09:06:14.529 INFO:teuthology.orchestra.run.smithi088.stderr:min_mon_release 18 (reef) 2024-06-03T09:06:14.529 INFO:teuthology.orchestra.run.smithi088.stderr:election_strategy: 3 2024-06-03T09:06:14.529 INFO:teuthology.orchestra.run.smithi088.stderr:0: v1:172.21.15.88:6789/0 mon.a 2024-06-03T09:06:14.529 INFO:teuthology.orchestra.run.smithi088.stderr:1: v1:172.21.15.192:6789/0 mon.b 2024-06-03T09:06:14.529 INFO:teuthology.orchestra.run.smithi088.stderr:2: v1:172.21.15.88:6790/0 mon.c 2024-06-03T09:06:14.529 INFO:teuthology.orchestra.run.smithi088.stderr: 2024-06-03T09:06:14.529 INFO:teuthology.orchestra.run.smithi088.stderr:2024-06-03T09:06:14.526+0000 7f1d21ffb640 1 -- 172.21.15.88:0/1801295370 <== mon.1 v1:172.21.15.192:6789/0 4 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 580+0+0 (unknown 180503021 0 0) 0x7f1d2c13ef10 con 0x7f1d2c1383e0 2024-06-03T09:06:14.529 INFO:teuthology.orchestra.run.smithi088.stderr:2024-06-03T09:06:14.526+0000 7f1d21ffb640 1 -- 172.21.15.88:0/1801295370 >> v1:172.21.15.88:6790/0 conn(0x7f1d2c0adb90 legacy=0x7f1d2c13c550 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-06-03T09:06:14.530 INFO:teuthology.orchestra.run.smithi088.stderr:2024-06-03T09:06:14.526+0000 7f1d21ffb640 1 -- 172.21.15.88:0/1801295370 >> v1:172.21.15.88:6789/0 conn(0x7f1d2c06ee00 legacy=0x7f1d2c0a86c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-06-03T09:06:14.530 INFO:teuthology.orchestra.run.smithi088.stderr:2024-06-03T09:06:14.526+0000 7f1d21ffb640 10 monclient: _finish_hunting 0 2024-06-03T09:06:14.530 INFO:teuthology.orchestra.run.smithi088.stderr:2024-06-03T09:06:14.526+0000 7f1d21ffb640 1 monclient: found mon.b 2024-06-03T09:06:14.530 INFO:teuthology.orchestra.run.smithi088.stderr:2024-06-03T09:06:14.526+0000 7f1d21ffb640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-06-03T09:06:14.530 INFO:teuthology.orchestra.run.smithi088.stderr:2024-06-03T09:06:14.526+0000 7f1d21ffb640 10 monclient: _send_mon_message to mon.b at v1:172.21.15.192:6789/0 2024-06-03T09:06:14.530 INFO:teuthology.orchestra.run.smithi088.stderr:2024-06-03T09:06:14.526+0000 7f1d21ffb640 1 -- 172.21.15.88:0/1801295370 --> v1:172.21.15.192:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f1d2c0aa310 con 0x7f1d2c1383e0 2024-06-03T09:06:14.530 INFO:teuthology.orchestra.run.smithi088.stderr:2024-06-03T09:06:14.526+0000 7f1d21ffb640 10 monclient: _finish_auth 0 2024-06-03T09:06:14.530 INFO:teuthology.orchestra.run.smithi088.stderr:2024-06-03T09:06:14.526+0000 7f1d21ffb640 10 monclient: _check_auth_tickets 2024-06-03T09:06:14.530 INFO:teuthology.orchestra.run.smithi088.stderr:2024-06-03T09:06:14.526+0000 7f1d21ffb640 20 monclient: _check_auth_rotating not needed by client.admin 2024-06-03T09:06:14.530 INFO:teuthology.orchestra.run.smithi088.stderr:2024-06-03T09:06:14.526+0000 7f1d33687640 5 monclient: authenticate success, global_id 30130 2024-06-03T09:06:14.530 INFO:teuthology.orchestra.run.smithi088.stderr:2024-06-03T09:06:14.526+0000 7f1d33687640 10 monclient: _renew_subs 2024-06-03T09:06:14.530 INFO:teuthology.orchestra.run.smithi088.stderr:2024-06-03T09:06:14.526+0000 7f1d33687640 10 monclient: _send_mon_message to mon.b at v1:172.21.15.192:6789/0 2024-06-03T09:06:14.530 INFO:teuthology.orchestra.run.smithi088.stderr:2024-06-03T09:06:14.526+0000 7f1d33687640 1 -- 172.21.15.88:0/1801295370 --> v1:172.21.15.192:6789/0 -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f1d2c13e2f0 con 0x7f1d2c1383e0 2024-06-03T09:06:14.531 INFO:teuthology.orchestra.run.smithi088.stderr:2024-06-03T09:06:14.526+0000 7f1d33687640 10 client.30130.objecter _maybe_request_map subscribing (onetime) to next osd map 2024-06-03T09:06:14.531 INFO:teuthology.orchestra.run.smithi088.stderr:2024-06-03T09:06:14.526+0000 7f1d33687640 10 monclient: _renew_subs 2024-06-03T09:06:14.531 INFO:teuthology.orchestra.run.smithi088.stderr:2024-06-03T09:06:14.526+0000 7f1d33687640 10 monclient: _send_mon_message to mon.b at v1:172.21.15.192:6789/0 2024-06-03T09:06:14.531 INFO:teuthology.orchestra.run.smithi088.stderr:2024-06-03T09:06:14.526+0000 7f1d33687640 1 -- 172.21.15.88:0/1801295370 --> v1:172.21.15.192:6789/0 -- mon_subscribe({osdmap=0}) v3 -- 0x7f1d2c13cc60 con 0x7f1d2c1383e0 2024-06-03T09:06:14.531 INFO:teuthology.orchestra.run.smithi088.stderr:2024-06-03T09:06:14.526+0000 7f1d33687640 1 librados: init done 2024-06-03T09:06:14.531 INFO:teuthology.orchestra.run.smithi088.stderr:2024-06-03T09:06:14.527+0000 7f1d33687640 10 start_mon_command cmd=[{"prefix": "get_command_descriptions"}] 2024-06-03T09:06:14.532 INFO:teuthology.orchestra.run.smithi088.stderr:2024-06-03T09:06:14.527+0000 7f1d33687640 10 monclient: _send_command 1 [{"prefix": "get_command_descriptions"}] 2024-06-03T09:06:14.532 INFO:teuthology.orchestra.run.smithi088.stderr:2024-06-03T09:06:14.527+0000 7f1d33687640 10 monclient: _send_mon_message to mon.b at v1:172.21.15.192:6789/0 2024-06-03T09:06:14.532 INFO:teuthology.orchestra.run.smithi088.stderr:2024-06-03T09:06:14.527+0000 7f1d33687640 1 -- 172.21.15.88:0/1801295370 --> v1:172.21.15.192:6789/0 -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f1d2c04ea60 con 0x7f1d2c1383e0 2024-06-03T09:06:14.739 INFO:teuthology.orchestra.run.smithi088.stderr:2024-06-03T09:06:14.739+0000 7f1d21ffb640 1 -- 172.21.15.88:0/1801295370 <== mon.1 v1:172.21.15.192:6789/0 5 ==== config(0 keys) v1 ==== 4+0+0 (unknown 0 0 0) 0x7f1d24003c20 con 0x7f1d2c1383e0 2024-06-03T09:06:14.739 INFO:teuthology.orchestra.run.smithi088.stderr:2024-06-03T09:06:14.739+0000 7f1d21ffb640 10 monclient: handle_config config(0 keys) v1 2024-06-03T09:06:14.739 INFO:teuthology.orchestra.run.smithi088.stderr:2024-06-03T09:06:14.739+0000 7f1d237fe640 4 set_mon_vals no callback set 2024-06-03T09:06:14.779 INFO:teuthology.orchestra.run.smithi088.stderr:2024-06-03T09:06:14.779+0000 7f1d21ffb640 1 -- 172.21.15.88:0/1801295370 <== mon.1 v1:172.21.15.192:6789/0 6 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 2234374065 0 0) 0x7f1d24003f30 con 0x7f1d2c1383e0 2024-06-03T09:06:14.779 INFO:teuthology.orchestra.run.smithi088.stderr:2024-06-03T09:06:14.779+0000 7f1d21ffb640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-06-03T09:06:14.779 INFO:teuthology.orchestra.run.smithi088.stderr:2024-06-03T09:06:14.779+0000 7f1d21ffb640 10 monclient: got monmap 1 from mon.b (according to old e1) 2024-06-03T09:06:14.779 INFO:teuthology.orchestra.run.smithi088.stderr:2024-06-03T09:06:14.779+0000 7f1d21ffb640 10 monclient: dump: 2024-06-03T09:06:14.779 INFO:teuthology.orchestra.run.smithi088.stderr:epoch 1 2024-06-03T09:06:14.779 INFO:teuthology.orchestra.run.smithi088.stderr:fsid ec3c3fdd-3ac6-4d7d-9412-683130ed8b04 2024-06-03T09:06:14.779 INFO:teuthology.orchestra.run.smithi088.stderr:last_changed 2024-06-03T08:34:31.885125+0000 2024-06-03T09:06:14.779 INFO:teuthology.orchestra.run.smithi088.stderr:created 2024-06-03T08:34:31.885125+0000 2024-06-03T09:06:14.779 INFO:teuthology.orchestra.run.smithi088.stderr:min_mon_release 18 (reef) 2024-06-03T09:06:14.780 INFO:teuthology.orchestra.run.smithi088.stderr:election_strategy: 3 2024-06-03T09:06:14.780 INFO:teuthology.orchestra.run.smithi088.stderr:0: v1:172.21.15.88:6789/0 mon.a 2024-06-03T09:06:14.780 INFO:teuthology.orchestra.run.smithi088.stderr:1: v1:172.21.15.192:6789/0 mon.b 2024-06-03T09:06:14.780 INFO:teuthology.orchestra.run.smithi088.stderr:2: v1:172.21.15.88:6790/0 mon.c 2024-06-03T09:06:14.780 INFO:teuthology.orchestra.run.smithi088.stderr: 2024-06-03T09:06:14.781 INFO:teuthology.orchestra.run.smithi088.stderr:2024-06-03T09:06:14.781+0000 7f1d21ffb640 1 -- 172.21.15.88:0/1801295370 <== mon.1 v1:172.21.15.192:6789/0 7 ==== mgrmap(e 5) v1 ==== 89110+0+0 (unknown 873533973 0 0) 0x7f1d24019cb0 con 0x7f1d2c1383e0 2024-06-03T09:06:14.782 INFO:teuthology.orchestra.run.smithi088.stderr:2024-06-03T09:06:14.782+0000 7f1d31423640 10 monclient: get_auth_request con 0x7f1d1006d390 auth_method 0 2024-06-03T09:06:14.782 INFO:teuthology.orchestra.run.smithi088.stderr:2024-06-03T09:06:14.782+0000 7f1d21ffb640 10 client.30130.objecter ms_handle_connect 0x7f1d1006d390 2024-06-03T09:06:14.783 INFO:teuthology.orchestra.run.smithi088.stderr:2024-06-03T09:06:14.783+0000 7f1d21ffb640 1 -- 172.21.15.88:0/1801295370 <== mon.1 v1:172.21.15.192:6789/0 8 ==== osd_map(1009..1009 src has 622..1009) v4 ==== 5229+0+0 (unknown 553880507 0 0) 0x7f1d240532e0 con 0x7f1d2c1383e0 2024-06-03T09:06:14.783 INFO:teuthology.orchestra.run.smithi088.stderr:2024-06-03T09:06:14.783+0000 7f1d21ffb640 10 client.30130.objecter ms_dispatch 0x7f1d2c000b90 osd_map(1009..1009 src has 622..1009) v4 2024-06-03T09:06:14.783 INFO:teuthology.orchestra.run.smithi088.stderr:2024-06-03T09:06:14.783+0000 7f1d21ffb640 3 client.30130.objecter handle_osd_map got epochs [1009,1009] > 0 2024-06-03T09:06:14.784 INFO:teuthology.orchestra.run.smithi088.stderr:2024-06-03T09:06:14.783+0000 7f1d21ffb640 3 client.30130.objecter handle_osd_map decoding full epoch 1009 2024-06-03T09:06:14.784 INFO:teuthology.orchestra.run.smithi088.stderr:2024-06-03T09:06:14.783+0000 7f1d21ffb640 20 client.30130.objecter dump_active .. 0 homeless 2024-06-03T09:06:14.785 INFO:teuthology.orchestra.run.smithi088.stderr:2024-06-03T09:06:14.786+0000 7f1d21ffb640 1 -- 172.21.15.88:0/1801295370 <== mon.1 v1:172.21.15.192:6789/0 9 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+182498 (unknown 1092875540 0 3016760700) 0x7f1d2404ecc0 con 0x7f1d2c1383e0 2024-06-03T09:06:14.785 INFO:teuthology.orchestra.run.smithi088.stderr:2024-06-03T09:06:14.786+0000 7f1d21ffb640 10 monclient: handle_mon_command_ack 1 [{"prefix": "get_command_descriptions"}] 2024-06-03T09:06:14.786 INFO:teuthology.orchestra.run.smithi088.stderr:2024-06-03T09:06:14.786+0000 7f1d21ffb640 10 monclient: _finish_command 1 = system:0 2024-06-03T09:06:14.963 INFO:teuthology.orchestra.run.smithi088.stderr:2024-06-03T09:06:14.963+0000 7f1d33687640 1 -- 172.21.15.88:0/1801295370 --> v1:172.21.15.192:6800/34229 -- mgr_command(tid 0: {"prefix": "pg dump", "target": ["mon-mgr", ""], "format": "json"}) v1 -- 0x7f1d2c06ff80 con 0x7f1d1006d390 2024-06-03T09:06:14.968 INFO:teuthology.orchestra.run.smithi088.stderr:2024-06-03T09:06:14.968+0000 7f1d21ffb640 1 -- 172.21.15.88:0/1801295370 <== mgr.4110 v1:172.21.15.192:6800/34229 1 ==== mgr_command_reply(tid 0: 0 dumped all) v1 ==== 18+0+151852 (unknown 2965378022 0 300975265) 0x7f1d28007810 con 0x7f1d1006d390 2024-06-03T09:06:14.970 INFO:teuthology.orchestra.run.smithi088.stdout: 2024-06-03T09:06:14.971 INFO:teuthology.orchestra.run.smithi088.stderr:dumped all 2024-06-03T09:06:14.973 INFO:teuthology.orchestra.run.smithi088.stderr:2024-06-03T09:06:14.970+0000 7f1d33687640 10 librados: watch_flush enter 2024-06-03T09:06:14.973 INFO:teuthology.orchestra.run.smithi088.stderr:2024-06-03T09:06:14.971+0000 7f1d33687640 10 librados: watch_flush exit 2024-06-03T09:06:14.973 INFO:teuthology.orchestra.run.smithi088.stderr:2024-06-03T09:06:14.971+0000 7f1d33687640 20 client.30130.objecter shutdown clearing up homeless session... 2024-06-03T09:06:14.973 INFO:teuthology.orchestra.run.smithi088.stderr:2024-06-03T09:06:14.971+0000 7f1d33687640 10 client.30130.objecter successfully canceled tick 2024-06-03T09:06:14.973 INFO:teuthology.orchestra.run.smithi088.stderr:2024-06-03T09:06:14.971+0000 7f1d33687640 1 -- 172.21.15.88:0/1801295370 >> v1:172.21.15.192:6800/34229 conn(0x7f1d1006d390 legacy=0x7f1d1006f850 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-06-03T09:06:14.973 INFO:teuthology.orchestra.run.smithi088.stderr:2024-06-03T09:06:14.971+0000 7f1d33687640 10 monclient: shutdown 2024-06-03T09:06:14.973 INFO:teuthology.orchestra.run.smithi088.stderr:2024-06-03T09:06:14.971+0000 7f1d33687640 20 monclient: shutdown discarding 0 pending message(s) 2024-06-03T09:06:14.973 INFO:teuthology.orchestra.run.smithi088.stderr:2024-06-03T09:06:14.971+0000 7f1d33687640 1 -- 172.21.15.88:0/1801295370 >> v1:172.21.15.192:6789/0 conn(0x7f1d2c1383e0 legacy=0x7f1d2c1ddca0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-06-03T09:06:14.974 INFO:teuthology.orchestra.run.smithi088.stderr:2024-06-03T09:06:14.971+0000 7f1d33687640 1 -- 172.21.15.88:0/1801295370 shutdown_connections 2024-06-03T09:06:14.974 INFO:teuthology.orchestra.run.smithi088.stderr:2024-06-03T09:06:14.971+0000 7f1d33687640 1 -- 172.21.15.88:0/1801295370 >> 172.21.15.88:0/1801295370 conn(0x7f1d2c0b2f20 msgr2=0x7f1d2c140080 unknown :-1 s=STATE_NONE l=0).mark_down 2024-06-03T09:06:14.974 INFO:teuthology.orchestra.run.smithi088.stderr:2024-06-03T09:06:14.971+0000 7f1d33687640 1 -- 172.21.15.88:0/1801295370 shutdown_connections 2024-06-03T09:06:14.974 INFO:teuthology.orchestra.run.smithi088.stderr:2024-06-03T09:06:14.971+0000 7f1d33687640 1 -- 172.21.15.88:0/1801295370 wait complete. 2024-06-03T09:06:14.974 INFO:teuthology.orchestra.run.smithi088.stderr:2024-06-03T09:06:14.971+0000 7f1d33687640 1 librados: shutdown 2024-06-03T09:06:14.981 INFO:teuthology.orchestra.run.smithi088.stdout:{"pg_ready":true,"pg_map":{"version":1911,"stamp":"2024-06-03T09:06:13.680365+0000","last_osdmap_epoch":0,"last_pg_scan":0,"pg_stats_sum":{"stat_sum":{"num_bytes":459285,"num_objects":3,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":1,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":3,"num_whiteouts":0,"num_read":102,"num_read_kb":88,"num_write":120,"num_write_kb":1379,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2,"num_bytes_recovered":397840,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":83,"ondisk_log_size":83,"up":68,"acting":68,"num_store_stats":0},"osd_stats_sum":{"up_from":0,"seq":0,"num_pgs":167,"num_osds":8,"num_per_pool_osds":8,"num_per_pool_omap_osds":8,"kb":754974720,"kb_used":494384,"kb_used_data":46136,"kb_used_omap":12,"kb_used_meta":448115,"kb_avail":754480336,"statfs":{"total":773094113280,"available":772587864064,"internally_reserved":0,"allocated":47243264,"data_stored":22795623,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":12744,"internal_metadata":458870328},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":528,"apply_latency_ms":528,"commit_latency_ns":528000000,"apply_latency_ns":528000000},"alerts":[],"network_ping_times":[]},"pg_stats_delta":{"stat_sum":{"num_bytes":-4000299,"num_objects":-44,"num_object_clones":-1,"num_object_copies":-132,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":-32,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":-44,"num_whiteouts":0,"num_read":-31,"num_read_kb":-11,"num_write":-71,"num_write_kb":-3969,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":-1,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":0,"acting":0,"num_store_stats":0,"stamp_delta":"4.584097"},"pg_stats":[{"pgid":"17.d","version":"0'0","reported_seq":939,"reported_epoch":1005,"state":"active+clean","last_fresh":"2024-06-03T09:06:09.228054+0000","last_change":"2024-06-03T08:38:19.260548+0000","last_active":"2024-06-03T09:06:09.228054+0000","last_peered":"2024-06-03T09:06:09.228054+0000","last_clean":"2024-06-03T09:06:09.228054+0000","last_became_active":"2024-06-03T08:38:19.253559+0000","last_became_peered":"2024-06-03T08:38:19.253559+0000","last_unstale":"2024-06-03T09:06:09.228054+0000","last_undegraded":"2024-06-03T09:06:09.228054+0000","last_fullsized":"2024-06-03T09:06:09.228054+0000","mapping_epoch":28,"log_start":"0'0","ondisk_log_start":"0'0","created":28,"last_epoch_clean":45,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-06-03T08:37:55.614497+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-06-03T08:37:55.614497+0000","last_clean_scrub_stamp":"2024-06-03T08:37:55.614497+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-06-04T19:00:12.470979+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1],"acting":[0,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"17.c","version":"0'0","reported_seq":950,"reported_epoch":1006,"state":"active+undersized","last_fresh":"2024-06-03T09:06:10.302316+0000","last_change":"2024-06-03T09:06:03.440074+0000","last_active":"2024-06-03T09:06:10.302316+0000","last_peered":"2024-06-03T09:06:10.302316+0000","last_clean":"2024-06-03T09:05:02.105638+0000","last_became_active":"2024-06-03T09:06:03.440074+0000","last_became_peered":"2024-06-03T09:06:03.440074+0000","last_unstale":"2024-06-03T09:06:10.302316+0000","last_undegraded":"2024-06-03T09:06:10.302316+0000","last_fullsized":"2024-06-03T09:06:03.377982+0000","mapping_epoch":999,"log_start":"0'0","ondisk_log_start":"0'0","created":28,"last_epoch_clean":255,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-06-03T08:37:55.614497+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-06-03T08:37:55.614497+0000","last_clean_scrub_stamp":"2024-06-03T08:37:55.614497+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-06-04T10:34:20.808557+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3],"acting":[3],"avail_no_missing":["3"],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"17.f","version":"0'0","reported_seq":144,"reported_epoch":1004,"state":"active+undersized","last_fresh":"2024-06-03T09:06:08.016575+0000","last_change":"2024-06-03T09:06:03.420285+0000","last_active":"2024-06-03T09:06:08.016575+0000","last_peered":"2024-06-03T09:06:08.016575+0000","last_clean":"2024-06-03T08:37:55.614497+0000","last_became_active":"2024-06-03T09:06:03.420285+0000","last_became_peered":"2024-06-03T09:06:03.420285+0000","last_unstale":"2024-06-03T09:06:08.016575+0000","last_undegraded":"2024-06-03T09:06:08.016575+0000","last_fullsized":"2024-06-03T09:06:03.380616+0000","mapping_epoch":999,"log_start":"0'0","ondisk_log_start":"0'0","created":28,"last_epoch_clean":255,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-06-03T08:37:55.614497+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-06-03T08:37:55.614497+0000","last_clean_scrub_stamp":"2024-06-03T08:37:55.614497+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-06-04T15:39:41.187731+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4],"acting":[4],"avail_no_missing":["4"],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"17.e","version":"0'0","reported_seq":144,"reported_epoch":1003,"state":"active+undersized","last_fresh":"2024-06-03T09:06:07.084889+0000","last_change":"2024-06-03T09:06:03.412887+0000","last_active":"2024-06-03T09:06:07.084889+0000","last_peered":"2024-06-03T09:06:07.084889+0000","last_clean":"2024-06-03T08:37:55.614497+0000","last_became_active":"2024-06-03T09:06:03.412887+0000","last_became_peered":"2024-06-03T09:06:03.412887+0000","last_unstale":"2024-06-03T09:06:07.084889+0000","last_undegraded":"2024-06-03T09:06:07.084889+0000","last_fullsized":"2024-06-03T09:06:03.332482+0000","mapping_epoch":999,"log_start":"0'0","ondisk_log_start":"0'0","created":28,"last_epoch_clean":255,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-06-03T08:37:55.614497+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-06-03T08:37:55.614497+0000","last_clean_scrub_stamp":"2024-06-03T08:37:55.614497+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-06-04T12:23:22.511601+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5],"acting":[5],"avail_no_missing":["5"],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"17.9","version":"0'0","reported_seq":941,"reported_epoch":1004,"state":"active+clean","last_fresh":"2024-06-03T09:06:07.906214+0000","last_change":"2024-06-03T08:38:04.772260+0000","last_active":"2024-06-03T09:06:07.906214+0000","last_peered":"2024-06-03T09:06:07.906214+0000","last_clean":"2024-06-03T09:06:07.906214+0000","last_became_active":"2024-06-03T08:38:04.763428+0000","last_became_peered":"2024-06-03T08:38:04.763428+0000","last_unstale":"2024-06-03T09:06:07.906214+0000","last_undegraded":"2024-06-03T09:06:07.906214+0000","last_fullsized":"2024-06-03T09:06:07.906214+0000","mapping_epoch":28,"log_start":"0'0","ondisk_log_start":"0'0","created":28,"last_epoch_clean":33,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-06-03T08:37:55.614497+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-06-03T08:37:55.614497+0000","last_clean_scrub_stamp":"2024-06-03T08:37:55.614497+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-06-04T10:55:10.773936+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,4],"acting":[7,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"17.8","version":"0'0","reported_seq":958,"reported_epoch":1007,"state":"active+undersized","last_fresh":"2024-06-03T09:06:11.350501+0000","last_change":"2024-06-03T09:06:03.388786+0000","last_active":"2024-06-03T09:06:11.350501+0000","last_peered":"2024-06-03T09:06:11.350501+0000","last_clean":"2024-06-03T09:04:43.297780+0000","last_became_active":"2024-06-03T09:06:03.388786+0000","last_became_peered":"2024-06-03T09:06:03.388786+0000","last_unstale":"2024-06-03T09:06:11.350501+0000","last_undegraded":"2024-06-03T09:06:11.350501+0000","last_fullsized":"2024-06-03T09:06:03.331754+0000","mapping_epoch":999,"log_start":"0'0","ondisk_log_start":"0'0","created":28,"last_epoch_clean":255,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-06-03T08:37:55.614497+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-06-03T08:37:55.614497+0000","last_clean_scrub_stamp":"2024-06-03T08:37:55.614497+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-06-04T09:18:44.765547+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0],"acting":[0],"avail_no_missing":["0"],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"17.b","version":"0'0","reported_seq":145,"reported_epoch":1005,"state":"active+undersized","last_fresh":"2024-06-03T09:06:09.149286+0000","last_change":"2024-06-03T09:06:03.451203+0000","last_active":"2024-06-03T09:06:09.149286+0000","last_peered":"2024-06-03T09:06:09.149286+0000","last_clean":"2024-06-03T08:37:55.614497+0000","last_became_active":"2024-06-03T09:06:03.451203+0000","last_became_peered":"2024-06-03T09:06:03.451203+0000","last_unstale":"2024-06-03T09:06:09.149286+0000","last_undegraded":"2024-06-03T09:06:09.149286+0000","last_fullsized":"2024-06-03T09:06:03.375523+0000","mapping_epoch":999,"log_start":"0'0","ondisk_log_start":"0'0","created":28,"last_epoch_clean":255,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-06-03T08:37:55.614497+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-06-03T08:37:55.614497+0000","last_clean_scrub_stamp":"2024-06-03T08:37:55.614497+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-06-04T10:16:19.813495+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2],"acting":[2],"avail_no_missing":["2"],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"17.a","version":"0'0","reported_seq":920,"reported_epoch":1004,"state":"active+clean","last_fresh":"2024-06-03T09:06:07.894224+0000","last_change":"2024-06-03T08:38:05.843142+0000","last_active":"2024-06-03T09:06:07.894224+0000","last_peered":"2024-06-03T09:06:07.894224+0000","last_clean":"2024-06-03T09:06:07.894224+0000","last_became_active":"2024-06-03T08:38:05.809257+0000","last_became_peered":"2024-06-03T08:38:05.809257+0000","last_unstale":"2024-06-03T09:06:07.894224+0000","last_undegraded":"2024-06-03T09:06:07.894224+0000","last_fullsized":"2024-06-03T09:06:07.894224+0000","mapping_epoch":28,"log_start":"0'0","ondisk_log_start":"0'0","created":28,"last_epoch_clean":33,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-06-03T08:37:55.614497+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-06-03T08:37:55.614497+0000","last_clean_scrub_stamp":"2024-06-03T08:37:55.614497+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-06-04T12:12:03.686838+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2],"acting":[1,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"17.5","version":"0'0","reported_seq":940,"reported_epoch":1005,"state":"active+clean","last_fresh":"2024-06-03T09:06:09.225607+0000","last_change":"2024-06-03T08:38:23.466051+0000","last_active":"2024-06-03T09:06:09.225607+0000","last_peered":"2024-06-03T09:06:09.225607+0000","last_clean":"2024-06-03T09:06:09.225607+0000","last_became_active":"2024-06-03T08:38:23.458039+0000","last_became_peered":"2024-06-03T08:38:23.458039+0000","last_unstale":"2024-06-03T09:06:09.225607+0000","last_undegraded":"2024-06-03T09:06:09.225607+0000","last_fullsized":"2024-06-03T09:06:09.225607+0000","mapping_epoch":28,"log_start":"0'0","ondisk_log_start":"0'0","created":28,"last_epoch_clean":49,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-06-03T08:37:55.614497+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-06-03T08:37:55.614497+0000","last_clean_scrub_stamp":"2024-06-03T08:37:55.614497+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-06-04T12:27:02.532436+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,4],"acting":[0,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"17.4","version":"0'0","reported_seq":145,"reported_epoch":1004,"state":"active+undersized","last_fresh":"2024-06-03T09:06:07.942233+0000","last_change":"2024-06-03T09:06:03.441889+0000","last_active":"2024-06-03T09:06:07.942233+0000","last_peered":"2024-06-03T09:06:07.942233+0000","last_clean":"2024-06-03T08:37:55.614497+0000","last_became_active":"2024-06-03T09:06:03.441889+0000","last_became_peered":"2024-06-03T09:06:03.441889+0000","last_unstale":"2024-06-03T09:06:07.942233+0000","last_undegraded":"2024-06-03T09:06:07.942233+0000","last_fullsized":"2024-06-03T09:06:03.359582+0000","mapping_epoch":999,"log_start":"0'0","ondisk_log_start":"0'0","created":28,"last_epoch_clean":255,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-06-03T08:37:55.614497+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-06-03T08:37:55.614497+0000","last_clean_scrub_stamp":"2024-06-03T08:37:55.614497+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-06-04T12:49:22.907141+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7],"acting":[7],"avail_no_missing":["7"],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"17.7","version":"0'0","reported_seq":924,"reported_epoch":1003,"state":"active+clean","last_fresh":"2024-06-03T09:06:07.105928+0000","last_change":"2024-06-03T08:38:04.837083+0000","last_active":"2024-06-03T09:06:07.105928+0000","last_peered":"2024-06-03T09:06:07.105928+0000","last_clean":"2024-06-03T09:06:07.105928+0000","last_became_active":"2024-06-03T08:38:04.818246+0000","last_became_peered":"2024-06-03T08:38:04.818246+0000","last_unstale":"2024-06-03T09:06:07.105928+0000","last_undegraded":"2024-06-03T09:06:07.105928+0000","last_fullsized":"2024-06-03T09:06:07.105928+0000","mapping_epoch":28,"log_start":"0'0","ondisk_log_start":"0'0","created":28,"last_epoch_clean":34,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-06-03T08:37:55.614497+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-06-03T08:37:55.614497+0000","last_clean_scrub_stamp":"2024-06-03T08:37:55.614497+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-06-04T17:02:38.204813+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,2],"acting":[5,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"17.6","version":"0'0","reported_seq":957,"reported_epoch":1007,"state":"active+undersized","last_fresh":"2024-06-03T09:06:11.430379+0000","last_change":"2024-06-03T09:06:03.428677+0000","last_active":"2024-06-03T09:06:11.430379+0000","last_peered":"2024-06-03T09:06:11.430379+0000","last_clean":"2024-06-03T09:04:43.312352+0000","last_became_active":"2024-06-03T09:06:03.428677+0000","last_became_peered":"2024-06-03T09:06:03.428677+0000","last_unstale":"2024-06-03T09:06:11.430379+0000","last_undegraded":"2024-06-03T09:06:11.430379+0000","last_fullsized":"2024-06-03T09:06:03.359909+0000","mapping_epoch":999,"log_start":"0'0","ondisk_log_start":"0'0","created":28,"last_epoch_clean":255,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-06-03T08:37:55.614497+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-06-03T08:37:55.614497+0000","last_clean_scrub_stamp":"2024-06-03T08:37:55.614497+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-06-04T15:38:02.180241+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0],"acting":[0],"avail_no_missing":["0"],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"2.7","version":"0'0","reported_seq":937,"reported_epoch":1006,"state":"active+clean","last_fresh":"2024-06-03T09:06:10.413619+0000","last_change":"2024-06-03T08:36:30.385405+0000","last_active":"2024-06-03T09:06:10.413619+0000","last_peered":"2024-06-03T09:06:10.413619+0000","last_clean":"2024-06-03T09:06:10.413619+0000","last_became_active":"2024-06-03T08:36:30.380302+0000","last_became_peered":"2024-06-03T08:36:30.380302+0000","last_unstale":"2024-06-03T09:06:10.413619+0000","last_undegraded":"2024-06-03T09:06:10.413619+0000","last_fullsized":"2024-06-03T09:06:10.413619+0000","mapping_epoch":23,"log_start":"0'0","ondisk_log_start":"0'0","created":23,"last_epoch_clean":24,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-06-03T08:36:28.444764+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-06-03T08:36:28.444764+0000","last_clean_scrub_stamp":"2024-06-03T08:36:28.444764+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-06-04T18:46:33.475181+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0],"acting":[3,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"17.14","version":"0'0","reported_seq":921,"reported_epoch":1003,"state":"active+clean","last_fresh":"2024-06-03T09:06:07.099645+0000","last_change":"2024-06-03T08:38:04.547565+0000","last_active":"2024-06-03T09:06:07.099645+0000","last_peered":"2024-06-03T09:06:07.099645+0000","last_clean":"2024-06-03T09:06:07.099645+0000","last_became_active":"2024-06-03T08:38:04.541781+0000","last_became_peered":"2024-06-03T08:38:04.541781+0000","last_unstale":"2024-06-03T09:06:07.099645+0000","last_undegraded":"2024-06-03T09:06:07.099645+0000","last_fullsized":"2024-06-03T09:06:07.099645+0000","mapping_epoch":28,"log_start":"0'0","ondisk_log_start":"0'0","created":28,"last_epoch_clean":34,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-06-03T08:37:55.614497+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-06-03T08:37:55.614497+0000","last_clean_scrub_stamp":"2024-06-03T08:37:55.614497+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-06-04T16:41:55.170134+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,2],"acting":[5,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"2.6","version":"0'0","reported_seq":945,"reported_epoch":1004,"state":"active+clean","last_fresh":"2024-06-03T09:06:07.984455+0000","last_change":"2024-06-03T08:36:30.430760+0000","last_active":"2024-06-03T09:06:07.984455+0000","last_peered":"2024-06-03T09:06:07.984455+0000","last_clean":"2024-06-03T09:06:07.984455+0000","last_became_active":"2024-06-03T08:36:30.404665+0000","last_became_peered":"2024-06-03T08:36:30.404665+0000","last_unstale":"2024-06-03T09:06:07.984455+0000","last_undegraded":"2024-06-03T09:06:07.984455+0000","last_fullsized":"2024-06-03T09:06:07.984455+0000","mapping_epoch":23,"log_start":"0'0","ondisk_log_start":"0'0","created":23,"last_epoch_clean":24,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-06-03T08:36:28.444764+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-06-03T08:36:28.444764+0000","last_clean_scrub_stamp":"2024-06-03T08:36:28.444764+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-06-04T10:37:37.746074+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,0],"acting":[4,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"17.15","version":"0'0","reported_seq":931,"reported_epoch":1006,"state":"active+clean","last_fresh":"2024-06-03T09:06:10.348071+0000","last_change":"2024-06-03T08:38:20.999903+0000","last_active":"2024-06-03T09:06:10.348071+0000","last_peered":"2024-06-03T09:06:10.348071+0000","last_clean":"2024-06-03T09:06:10.348071+0000","last_became_active":"2024-06-03T08:38:20.979523+0000","last_became_peered":"2024-06-03T08:38:20.979523+0000","last_unstale":"2024-06-03T09:06:10.348071+0000","last_undegraded":"2024-06-03T09:06:10.348071+0000","last_fullsized":"2024-06-03T09:06:10.348071+0000","mapping_epoch":28,"log_start":"0'0","ondisk_log_start":"0'0","created":28,"last_epoch_clean":47,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-06-03T08:37:55.614497+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-06-03T08:37:55.614497+0000","last_clean_scrub_stamp":"2024-06-03T08:37:55.614497+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-06-04T20:29:58.314289+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,1],"acting":[3,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"2.5","version":"0'0","reported_seq":938,"reported_epoch":1006,"state":"active+clean","last_fresh":"2024-06-03T09:06:10.349618+0000","last_change":"2024-06-03T08:36:30.024235+0000","last_active":"2024-06-03T09:06:10.349618+0000","last_peered":"2024-06-03T09:06:10.349618+0000","last_clean":"2024-06-03T09:06:10.349618+0000","last_became_active":"2024-06-03T08:36:30.019040+0000","last_became_peered":"2024-06-03T08:36:30.019040+0000","last_unstale":"2024-06-03T09:06:10.349618+0000","last_undegraded":"2024-06-03T09:06:10.349618+0000","last_fullsized":"2024-06-03T09:06:10.349618+0000","mapping_epoch":23,"log_start":"0'0","ondisk_log_start":"0'0","created":23,"last_epoch_clean":24,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-06-03T08:36:28.444764+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-06-03T08:36:28.444764+0000","last_clean_scrub_stamp":"2024-06-03T08:36:28.444764+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-06-04T15:46:47.109502+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,4],"acting":[3,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"17.16","version":"0'0","reported_seq":968,"reported_epoch":1005,"state":"active+undersized","last_fresh":"2024-06-03T09:06:09.191972+0000","last_change":"2024-06-03T09:06:03.478615+0000","last_active":"2024-06-03T09:06:09.191972+0000","last_peered":"2024-06-03T09:06:09.191972+0000","last_clean":"2024-06-03T09:04:43.328415+0000","last_became_active":"2024-06-03T09:06:03.478615+0000","last_became_peered":"2024-06-03T09:06:03.478615+0000","last_unstale":"2024-06-03T09:06:09.191972+0000","last_undegraded":"2024-06-03T09:06:09.191972+0000","last_fullsized":"2024-06-03T09:06:03.402504+0000","mapping_epoch":999,"log_start":"0'0","ondisk_log_start":"0'0","created":28,"last_epoch_clean":255,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-06-03T08:37:55.614497+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-06-03T08:37:55.614497+0000","last_clean_scrub_stamp":"2024-06-03T08:37:55.614497+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-06-04T08:47:08.079233+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2],"acting":[2],"avail_no_missing":["2"],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"2.4","version":"0'0","reported_seq":946,"reported_epoch":1003,"state":"active+undersized","last_fresh":"2024-06-03T09:06:07.101449+0000","last_change":"2024-06-03T09:06:03.387171+0000","last_active":"2024-06-03T09:06:07.101449+0000","last_peered":"2024-06-03T09:06:07.101449+0000","last_clean":"2024-06-03T09:04:43.202645+0000","last_became_active":"2024-06-03T09:06:03.387171+0000","last_became_peered":"2024-06-03T09:06:03.387171+0000","last_unstale":"2024-06-03T09:06:07.101449+0000","last_undegraded":"2024-06-03T09:06:07.101449+0000","last_fullsized":"2024-06-03T09:06:03.283787+0000","mapping_epoch":999,"log_start":"0'0","ondisk_log_start":"0'0","created":23,"last_epoch_clean":255,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-06-03T08:36:28.444764+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-06-03T08:36:28.444764+0000","last_clean_scrub_stamp":"2024-06-03T08:36:28.444764+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-06-04T20:17:43.485023+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5],"acting":[5],"avail_no_missing":["5"],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"17.17","version":"0'0","reported_seq":950,"reported_epoch":1005,"state":"active+clean","last_fresh":"2024-06-03T09:06:09.179648+0000","last_change":"2024-06-03T08:38:38.483275+0000","last_active":"2024-06-03T09:06:09.179648+0000","last_peered":"2024-06-03T09:06:09.179648+0000","last_clean":"2024-06-03T09:06:09.179648+0000","last_became_active":"2024-06-03T08:38:38.467916+0000","last_became_peered":"2024-06-03T08:38:38.467916+0000","last_unstale":"2024-06-03T09:06:09.179648+0000","last_undegraded":"2024-06-03T09:06:09.179648+0000","last_fullsized":"2024-06-03T09:06:09.179648+0000","mapping_epoch":28,"log_start":"0'0","ondisk_log_start":"0'0","created":28,"last_epoch_clean":61,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-06-03T08:37:55.614497+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-06-03T08:37:55.614497+0000","last_clean_scrub_stamp":"2024-06-03T08:37:55.614497+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-06-04T10:02:23.036420+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,7],"acting":[2,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"2.2","version":"0'0","reported_seq":960,"reported_epoch":1006,"state":"active+undersized","last_fresh":"2024-06-03T09:06:10.282512+0000","last_change":"2024-06-03T09:06:03.410020+0000","last_active":"2024-06-03T09:06:10.282512+0000","last_peered":"2024-06-03T09:06:10.282512+0000","last_clean":"2024-06-03T09:05:37.145889+0000","last_became_active":"2024-06-03T09:06:03.410020+0000","last_became_peered":"2024-06-03T09:06:03.410020+0000","last_unstale":"2024-06-03T09:06:10.282512+0000","last_undegraded":"2024-06-03T09:06:10.282512+0000","last_fullsized":"2024-06-03T09:06:03.346864+0000","mapping_epoch":999,"log_start":"0'0","ondisk_log_start":"0'0","created":23,"last_epoch_clean":255,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-06-03T08:36:28.444764+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-06-03T08:36:28.444764+0000","last_clean_scrub_stamp":"2024-06-03T08:36:28.444764+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-06-04T09:57:12.423360+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3],"acting":[3],"avail_no_missing":["3"],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"17.11","version":"0'0","reported_seq":940,"reported_epoch":1004,"state":"active+clean","last_fresh":"2024-06-03T09:06:07.943878+0000","last_change":"2024-06-03T08:38:06.502302+0000","last_active":"2024-06-03T09:06:07.943878+0000","last_peered":"2024-06-03T09:06:07.943878+0000","last_clean":"2024-06-03T09:06:07.943878+0000","last_became_active":"2024-06-03T08:38:06.490204+0000","last_became_peered":"2024-06-03T08:38:06.490204+0000","last_unstale":"2024-06-03T09:06:07.943878+0000","last_undegraded":"2024-06-03T09:06:07.943878+0000","last_fullsized":"2024-06-03T09:06:07.943878+0000","mapping_epoch":28,"log_start":"0'0","ondisk_log_start":"0'0","created":28,"last_epoch_clean":32,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-06-03T08:37:55.614497+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-06-03T08:37:55.614497+0000","last_clean_scrub_stamp":"2024-06-03T08:37:55.614497+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-06-04T10:47:05.988958+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,2],"acting":[4,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"2.1","version":"0'0","reported_seq":948,"reported_epoch":1004,"state":"active+clean","last_fresh":"2024-06-03T09:06:07.907718+0000","last_change":"2024-06-03T08:36:30.883360+0000","last_active":"2024-06-03T09:06:07.907718+0000","last_peered":"2024-06-03T09:06:07.907718+0000","last_clean":"2024-06-03T09:06:07.907718+0000","last_became_active":"2024-06-03T08:36:30.855073+0000","last_became_peered":"2024-06-03T08:36:30.855073+0000","last_unstale":"2024-06-03T09:06:07.907718+0000","last_undegraded":"2024-06-03T09:06:07.907718+0000","last_fullsized":"2024-06-03T09:06:07.907718+0000","mapping_epoch":23,"log_start":"0'0","ondisk_log_start":"0'0","created":23,"last_epoch_clean":25,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-06-03T08:36:28.444764+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-06-03T08:36:28.444764+0000","last_clean_scrub_stamp":"2024-06-03T08:36:28.444764+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-06-04T18:12:22.363473+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,5],"acting":[7,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"17.12","version":"0'0","reported_seq":941,"reported_epoch":1004,"state":"active+clean","last_fresh":"2024-06-03T09:06:07.925825+0000","last_change":"2024-06-03T08:38:01.751562+0000","last_active":"2024-06-03T09:06:07.925825+0000","last_peered":"2024-06-03T09:06:07.925825+0000","last_clean":"2024-06-03T09:06:07.925825+0000","last_became_active":"2024-06-03T08:38:01.745027+0000","last_became_peered":"2024-06-03T08:38:01.745027+0000","last_unstale":"2024-06-03T09:06:07.925825+0000","last_undegraded":"2024-06-03T09:06:07.925825+0000","last_fullsized":"2024-06-03T09:06:07.925825+0000","mapping_epoch":28,"log_start":"0'0","ondisk_log_start":"0'0","created":28,"last_epoch_clean":32,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-06-03T08:37:55.614497+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-06-03T08:37:55.614497+0000","last_clean_scrub_stamp":"2024-06-03T08:37:55.614497+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-06-04T17:54:43.785326+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,3],"acting":[7,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"2.0","version":"0'0","reported_seq":940,"reported_epoch":1006,"state":"active+clean","last_fresh":"2024-06-03T09:06:10.230383+0000","last_change":"2024-06-03T08:36:29.984096+0000","last_active":"2024-06-03T09:06:10.230383+0000","last_peered":"2024-06-03T09:06:10.230383+0000","last_clean":"2024-06-03T09:06:10.230383+0000","last_became_active":"2024-06-03T08:36:29.957875+0000","last_became_peered":"2024-06-03T08:36:29.957875+0000","last_unstale":"2024-06-03T09:06:10.230383+0000","last_undegraded":"2024-06-03T09:06:10.230383+0000","last_fullsized":"2024-06-03T09:06:10.230383+0000","mapping_epoch":23,"log_start":"0'0","ondisk_log_start":"0'0","created":23,"last_epoch_clean":24,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-06-03T08:36:28.444764+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-06-03T08:36:28.444764+0000","last_clean_scrub_stamp":"2024-06-03T08:36:28.444764+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-06-04T08:41:33.679479+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,5],"acting":[3,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"17.13","version":"0'0","reported_seq":939,"reported_epoch":1005,"state":"active+clean","last_fresh":"2024-06-03T09:06:09.209721+0000","last_change":"2024-06-03T08:38:26.838316+0000","last_active":"2024-06-03T09:06:09.209721+0000","last_peered":"2024-06-03T09:06:09.209721+0000","last_clean":"2024-06-03T09:06:09.209721+0000","last_became_active":"2024-06-03T08:38:26.833214+0000","last_became_peered":"2024-06-03T08:38:26.833214+0000","last_unstale":"2024-06-03T09:06:09.209721+0000","last_undegraded":"2024-06-03T09:06:09.209721+0000","last_fullsized":"2024-06-03T09:06:09.209721+0000","mapping_epoch":28,"log_start":"0'0","ondisk_log_start":"0'0","created":28,"last_epoch_clean":52,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-06-03T08:37:55.614497+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-06-03T08:37:55.614497+0000","last_clean_scrub_stamp":"2024-06-03T08:37:55.614497+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-06-04T12:58:51.617998+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,0],"acting":[4,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"2.3","version":"0'0","reported_seq":939,"reported_epoch":1006,"state":"active+clean","last_fresh":"2024-06-03T09:06:10.288262+0000","last_change":"2024-06-03T08:36:30.351670+0000","last_active":"2024-06-03T09:06:10.288262+0000","last_peered":"2024-06-03T09:06:10.288262+0000","last_clean":"2024-06-03T09:06:10.288262+0000","last_became_active":"2024-06-03T08:36:30.343560+0000","last_became_peered":"2024-06-03T08:36:30.343560+0000","last_unstale":"2024-06-03T09:06:10.288262+0000","last_undegraded":"2024-06-03T09:06:10.288262+0000","last_fullsized":"2024-06-03T09:06:10.288262+0000","mapping_epoch":23,"log_start":"0'0","ondisk_log_start":"0'0","created":23,"last_epoch_clean":24,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-06-03T08:36:28.444764+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-06-03T08:36:28.444764+0000","last_clean_scrub_stamp":"2024-06-03T08:36:28.444764+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-06-04T12:02:24.765135+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,2],"acting":[3,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.0","version":"19'76","reported_seq":1047,"reported_epoch":1004,"state":"active+clean","last_fresh":"2024-06-03T09:06:07.848668+0000","last_change":"2024-06-03T08:36:26.702702+0000","last_active":"2024-06-03T09:06:07.848668+0000","last_peered":"2024-06-03T09:06:07.848668+0000","last_clean":"2024-06-03T09:06:07.848668+0000","last_became_active":"2024-06-03T08:36:26.128378+0000","last_became_peered":"2024-06-03T08:36:26.128378+0000","last_unstale":"2024-06-03T09:06:07.848668+0000","last_undegraded":"2024-06-03T09:06:07.848668+0000","last_fullsized":"2024-06-03T09:06:07.848668+0000","mapping_epoch":20,"log_start":"0'0","ondisk_log_start":"0'0","created":16,"last_epoch_clean":22,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-06-03T08:36:15.542635+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-06-03T08:36:15.542635+0000","last_clean_scrub_stamp":"2024-06-03T08:36:15.542635+0000","objects_scrubbed":0,"log_size":76,"log_dups_size":0,"ondisk_log_size":76,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-06-04T12:41:53.731360+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":459280,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":96,"num_read_kb":82,"num_write":113,"num_write_kb":1372,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2,"num_bytes_recovered":397840,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,5],"acting":[1,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"17.10","version":"0'0","reported_seq":940,"reported_epoch":1004,"state":"active+clean","last_fresh":"2024-06-03T09:06:07.939453+0000","last_change":"2024-06-03T08:38:15.399189+0000","last_active":"2024-06-03T09:06:07.939453+0000","last_peered":"2024-06-03T09:06:07.939453+0000","last_clean":"2024-06-03T09:06:07.939453+0000","last_became_active":"2024-06-03T08:38:15.337723+0000","last_became_peered":"2024-06-03T08:38:15.337723+0000","last_unstale":"2024-06-03T09:06:07.939453+0000","last_undegraded":"2024-06-03T09:06:07.939453+0000","last_fullsized":"2024-06-03T09:06:07.939453+0000","mapping_epoch":28,"log_start":"0'0","ondisk_log_start":"0'0","created":28,"last_epoch_clean":42,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-06-03T08:37:55.614497+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-06-03T08:37:55.614497+0000","last_clean_scrub_stamp":"2024-06-03T08:37:55.614497+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-06-04T09:29:46.427317+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,3],"acting":[4,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"17.1a","version":"0'0","reported_seq":956,"reported_epoch":1004,"state":"active+undersized","last_fresh":"2024-06-03T09:06:07.954840+0000","last_change":"2024-06-03T09:06:03.390734+0000","last_active":"2024-06-03T09:06:07.954840+0000","last_peered":"2024-06-03T09:06:07.954840+0000","last_clean":"2024-06-03T09:05:20.322313+0000","last_became_active":"2024-06-03T09:06:03.390734+0000","last_became_peered":"2024-06-03T09:06:03.390734+0000","last_unstale":"2024-06-03T09:06:07.954840+0000","last_undegraded":"2024-06-03T09:06:07.954840+0000","last_fullsized":"2024-06-03T09:06:03.343197+0000","mapping_epoch":999,"log_start":"0'0","ondisk_log_start":"0'0","created":28,"last_epoch_clean":254,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-06-03T08:37:55.614497+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-06-03T08:37:55.614497+0000","last_clean_scrub_stamp":"2024-06-03T08:37:55.614497+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-06-04T15:56:47.628875+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4],"acting":[4],"avail_no_missing":["4"],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"17.1b","version":"0'0","reported_seq":932,"reported_epoch":1006,"state":"active+clean","last_fresh":"2024-06-03T09:06:10.286656+0000","last_change":"2024-06-03T08:38:03.115748+0000","last_active":"2024-06-03T09:06:10.286656+0000","last_peered":"2024-06-03T09:06:10.286656+0000","last_clean":"2024-06-03T09:06:10.286656+0000","last_became_active":"2024-06-03T08:38:03.110533+0000","last_became_peered":"2024-06-03T08:38:03.110533+0000","last_unstale":"2024-06-03T09:06:10.286656+0000","last_undegraded":"2024-06-03T09:06:10.286656+0000","last_fullsized":"2024-06-03T09:06:10.286656+0000","mapping_epoch":28,"log_start":"0'0","ondisk_log_start":"0'0","created":28,"last_epoch_clean":32,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-06-03T08:37:55.614497+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-06-03T08:37:55.614497+0000","last_clean_scrub_stamp":"2024-06-03T08:37:55.614497+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-06-04T12:36:47.471609+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,7],"acting":[3,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"17.18","version":"0'0","reported_seq":940,"reported_epoch":1004,"state":"active+clean","last_fresh":"2024-06-03T09:06:07.900211+0000","last_change":"2024-06-03T08:38:04.632864+0000","last_active":"2024-06-03T09:06:07.900211+0000","last_peered":"2024-06-03T09:06:07.900211+0000","last_clean":"2024-06-03T09:06:07.900211+0000","last_became_active":"2024-06-03T08:38:04.627856+0000","last_became_peered":"2024-06-03T08:38:04.627856+0000","last_unstale":"2024-06-03T09:06:07.900211+0000","last_undegraded":"2024-06-03T09:06:07.900211+0000","last_fullsized":"2024-06-03T09:06:07.900211+0000","mapping_epoch":28,"log_start":"0'0","ondisk_log_start":"0'0","created":28,"last_epoch_clean":33,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-06-03T08:37:55.614497+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-06-03T08:37:55.614497+0000","last_clean_scrub_stamp":"2024-06-03T08:37:55.614497+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-06-04T19:55:51.181598+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,4],"acting":[7,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"17.19","version":"0'0","reported_seq":931,"reported_epoch":1006,"state":"active+clean","last_fresh":"2024-06-03T09:06:10.258391+0000","last_change":"2024-06-03T08:38:01.839491+0000","last_active":"2024-06-03T09:06:10.258391+0000","last_peered":"2024-06-03T09:06:10.258391+0000","last_clean":"2024-06-03T09:06:10.258391+0000","last_became_active":"2024-06-03T08:38:01.829539+0000","last_became_peered":"2024-06-03T08:38:01.829539+0000","last_unstale":"2024-06-03T09:06:10.258391+0000","last_undegraded":"2024-06-03T09:06:10.258391+0000","last_fullsized":"2024-06-03T09:06:10.258391+0000","mapping_epoch":28,"log_start":"0'0","ondisk_log_start":"0'0","created":28,"last_epoch_clean":30,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-06-03T08:37:55.614497+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-06-03T08:37:55.614497+0000","last_clean_scrub_stamp":"2024-06-03T08:37:55.614497+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-06-04T09:10:43.860229+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,2],"acting":[3,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"17.1e","version":"0'0","reported_seq":144,"reported_epoch":1004,"state":"active+undersized","last_fresh":"2024-06-03T09:06:07.982178+0000","last_change":"2024-06-03T09:06:03.406181+0000","last_active":"2024-06-03T09:06:07.982178+0000","last_peered":"2024-06-03T09:06:07.982178+0000","last_clean":"2024-06-03T08:37:55.614497+0000","last_became_active":"2024-06-03T09:06:03.406181+0000","last_became_peered":"2024-06-03T09:06:03.406181+0000","last_unstale":"2024-06-03T09:06:07.982178+0000","last_undegraded":"2024-06-03T09:06:07.982178+0000","last_fullsized":"2024-06-03T09:06:03.363739+0000","mapping_epoch":999,"log_start":"0'0","ondisk_log_start":"0'0","created":28,"last_epoch_clean":255,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-06-03T08:37:55.614497+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-06-03T08:37:55.614497+0000","last_clean_scrub_stamp":"2024-06-03T08:37:55.614497+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-06-04T11:45:24.486600+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4],"acting":[4],"avail_no_missing":["4"],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"17.1f","version":"0'0","reported_seq":938,"reported_epoch":1005,"state":"active+clean","last_fresh":"2024-06-03T09:06:09.243583+0000","last_change":"2024-06-03T08:38:04.383538+0000","last_active":"2024-06-03T09:06:09.243583+0000","last_peered":"2024-06-03T09:06:09.243583+0000","last_clean":"2024-06-03T09:06:09.243583+0000","last_became_active":"2024-06-03T08:38:04.367278+0000","last_became_peered":"2024-06-03T08:38:04.367278+0000","last_unstale":"2024-06-03T09:06:09.243583+0000","last_undegraded":"2024-06-03T09:06:09.243583+0000","last_fullsized":"2024-06-03T09:06:09.243583+0000","mapping_epoch":28,"log_start":"0'0","ondisk_log_start":"0'0","created":28,"last_epoch_clean":32,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-06-03T08:37:55.614497+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-06-03T08:37:55.614497+0000","last_clean_scrub_stamp":"2024-06-03T08:37:55.614497+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-06-04T10:39:26.552282+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,2],"acting":[0,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"17.1c","version":"0'0","reported_seq":938,"reported_epoch":1003,"state":"active+undersized","last_fresh":"2024-06-03T09:06:07.096736+0000","last_change":"2024-06-03T09:06:03.382210+0000","last_active":"2024-06-03T09:06:07.096736+0000","last_peered":"2024-06-03T09:06:07.096736+0000","last_clean":"2024-06-03T09:04:43.199660+0000","last_became_active":"2024-06-03T09:06:03.382210+0000","last_became_peered":"2024-06-03T09:06:03.382210+0000","last_unstale":"2024-06-03T09:06:07.096736+0000","last_undegraded":"2024-06-03T09:06:07.096736+0000","last_fullsized":"2024-06-03T09:06:03.277874+0000","mapping_epoch":999,"log_start":"0'0","ondisk_log_start":"0'0","created":28,"last_epoch_clean":255,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-06-03T08:37:55.614497+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-06-03T08:37:55.614497+0000","last_clean_scrub_stamp":"2024-06-03T08:37:55.614497+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-06-04T12:18:50.013884+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5],"acting":[5],"avail_no_missing":["5"],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"17.1d","version":"0'0","reported_seq":919,"reported_epoch":1004,"state":"active+clean","last_fresh":"2024-06-03T09:06:07.907920+0000","last_change":"2024-06-03T08:38:06.794630+0000","last_active":"2024-06-03T09:06:07.907920+0000","last_peered":"2024-06-03T09:06:07.907920+0000","last_clean":"2024-06-03T09:06:07.907920+0000","last_became_active":"2024-06-03T08:38:06.592317+0000","last_became_peered":"2024-06-03T08:38:06.592317+0000","last_unstale":"2024-06-03T09:06:07.907920+0000","last_undegraded":"2024-06-03T09:06:07.907920+0000","last_fullsized":"2024-06-03T09:06:07.907920+0000","mapping_epoch":28,"log_start":"0'0","ondisk_log_start":"0'0","created":28,"last_epoch_clean":33,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-06-03T08:37:55.614497+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-06-03T08:37:55.614497+0000","last_clean_scrub_stamp":"2024-06-03T08:37:55.614497+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-06-04T12:33:30.785369+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,7],"acting":[1,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"17.2","version":"0'0","reported_seq":921,"reported_epoch":1004,"state":"active+clean","last_fresh":"2024-06-03T09:06:07.892197+0000","last_change":"2024-06-03T08:38:14.981410+0000","last_active":"2024-06-03T09:06:07.892197+0000","last_peered":"2024-06-03T09:06:07.892197+0000","last_clean":"2024-06-03T09:06:07.892197+0000","last_became_active":"2024-06-03T08:38:14.973825+0000","last_became_peered":"2024-06-03T08:38:14.973825+0000","last_unstale":"2024-06-03T09:06:07.892197+0000","last_undegraded":"2024-06-03T09:06:07.892197+0000","last_fullsized":"2024-06-03T09:06:07.892197+0000","mapping_epoch":28,"log_start":"0'0","ondisk_log_start":"0'0","created":28,"last_epoch_clean":41,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-06-03T08:37:55.614497+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-06-03T08:37:55.614497+0000","last_clean_scrub_stamp":"2024-06-03T08:37:55.614497+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-06-04T17:01:05.518613+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,4],"acting":[1,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"17.3","version":"0'0","reported_seq":940,"reported_epoch":1005,"state":"active+clean","last_fresh":"2024-06-03T09:06:09.215956+0000","last_change":"2024-06-03T08:38:14.619745+0000","last_active":"2024-06-03T09:06:09.215956+0000","last_peered":"2024-06-03T09:06:09.215956+0000","last_clean":"2024-06-03T09:06:09.215956+0000","last_became_active":"2024-06-03T08:38:14.510665+0000","last_became_peered":"2024-06-03T08:38:14.510665+0000","last_unstale":"2024-06-03T09:06:09.215956+0000","last_undegraded":"2024-06-03T09:06:09.215956+0000","last_fullsized":"2024-06-03T09:06:09.215956+0000","mapping_epoch":28,"log_start":"0'0","ondisk_log_start":"0'0","created":28,"last_epoch_clean":42,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-06-03T08:37:55.614497+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-06-03T08:37:55.614497+0000","last_clean_scrub_stamp":"2024-06-03T08:37:55.614497+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-06-04T12:03:47.392064+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,7],"acting":[0,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"17.0","version":"0'0","reported_seq":941,"reported_epoch":1007,"state":"active+clean","last_fresh":"2024-06-03T09:06:11.290247+0000","last_change":"2024-06-03T08:38:04.237118+0000","last_active":"2024-06-03T09:06:11.290247+0000","last_peered":"2024-06-03T09:06:11.290247+0000","last_clean":"2024-06-03T09:06:11.290247+0000","last_became_active":"2024-06-03T08:38:04.231954+0000","last_became_peered":"2024-06-03T08:38:04.231954+0000","last_unstale":"2024-06-03T09:06:11.290247+0000","last_undegraded":"2024-06-03T09:06:11.290247+0000","last_fullsized":"2024-06-03T09:06:11.290247+0000","mapping_epoch":28,"log_start":"0'0","ondisk_log_start":"0'0","created":28,"last_epoch_clean":32,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-06-03T08:37:55.614497+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-06-03T08:37:55.614497+0000","last_clean_scrub_stamp":"2024-06-03T08:37:55.614497+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-06-04T09:56:10.741637+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,2],"acting":[0,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"17.1","version":"65'7","reported_seq":954,"reported_epoch":1003,"state":"active+undersized+degraded","last_fresh":"2024-06-03T09:06:07.053648+0000","last_change":"2024-06-03T09:06:03.370660+0000","last_active":"2024-06-03T09:06:07.053648+0000","last_peered":"2024-06-03T09:06:07.053648+0000","last_clean":"2024-06-03T09:05:03.733864+0000","last_became_active":"2024-06-03T09:06:03.370660+0000","last_became_peered":"2024-06-03T09:06:03.370660+0000","last_unstale":"2024-06-03T09:06:07.053648+0000","last_undegraded":"2024-06-03T09:06:03.266076+0000","last_fullsized":"2024-06-03T09:06:03.264525+0000","mapping_epoch":999,"log_start":"0'0","ondisk_log_start":"0'0","created":28,"last_epoch_clean":255,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-06-03T08:37:55.614497+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-06-03T08:37:55.614497+0000","last_clean_scrub_stamp":"2024-06-03T08:37:55.614497+0000","objects_scrubbed":0,"log_size":7,"log_dups_size":0,"ondisk_log_size":7,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-06-04T11:55:18.194283+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":5,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":1,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":6,"num_read_kb":6,"num_write":7,"num_write_kb":7,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5],"acting":[5],"avail_no_missing":["5"],"object_location_counts":[{"shards":"5","objects":1}],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]}],"pool_stats":[{"poolid":2,"num_pg":8,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":14,"acting":14,"num_store_stats":7},{"poolid":1,"num_pg":1,"stat_sum":{"num_bytes":459280,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":96,"num_read_kb":82,"num_write":113,"num_write_kb":1372,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2,"num_bytes_recovered":397840,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":802816,"data_stored":795680,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":76,"ondisk_log_size":76,"up":2,"acting":2,"num_store_stats":2},{"poolid":17,"num_pg":32,"stat_sum":{"num_bytes":5,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":1,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":6,"num_read_kb":6,"num_write":7,"num_write_kb":7,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":8192,"data_stored":10,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":7,"ondisk_log_size":7,"up":52,"acting":52,"num_store_stats":8}],"osd_stats":[{"osd":6,"up_from":253,"seq":1086626726200,"num_pgs":32,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":54772,"kb_used_data":5608,"kb_used_omap":1,"kb_used_meta":49150,"kb_avail":94317068,"statfs":{"total":96636764160,"available":96580677632,"internally_reserved":0,"allocated":5742592,"data_stored":2767180,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1593,"internal_metadata":50330055},"hb_peers":[0,1,2,3,4,5,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":109,"apply_latency_ms":109,"commit_latency_ns":109000000,"apply_latency_ns":109000000},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Mon Jun 3 09:04:02 2024","interfaces":[{"interface":"back","average":{"1min":1582.4780000000001,"5min":332.72699999999998,"15min":124.113},"min":{"1min":4.4829999999999997,"5min":4.1959999999999997,"15min":4.1959999999999997},"max":{"1min":22766.882000000001,"5min":22766.882000000001,"15min":22766.882000000001},"last":21.917999999999999},{"interface":"front","average":{"1min":1579.5409999999999,"5min":334.94499999999999,"15min":125.84999999999999},"min":{"1min":4.8129999999999997,"5min":4.7539999999999996,"15min":4.4530000000000003},"max":{"1min":22766.289000000001,"5min":22766.289000000001,"15min":22766.289000000001},"last":25.408999999999999}]},{"osd":1,"last update":"Mon Jun 3 09:03:40 2024","interfaces":[{"interface":"back","average":{"1min":36.420999999999999,"5min":77.054000000000002,"15min":132.84800000000001},"min":{"1min":6.0380000000000003,"5min":5.8799999999999999,"15min":5.484},"max":{"1min":589.64099999999996,"5min":4138.1679999999997,"15min":9955.1299999999992},"last":26.84},{"interface":"front","average":{"1min":40.240000000000002,"5min":79.674999999999997,"15min":133.38999999999999},"min":{"1min":8.4589999999999996,"5min":8.4090000000000007,"15min":6.1390000000000002},"max":{"1min":588.60799999999995,"5min":4137.6530000000002,"15min":9827.5509999999995},"last":26.263000000000002}]},{"osd":2,"last update":"Mon Jun 3 09:03:40 2024","interfaces":[{"interface":"back","average":{"1min":12.613,"5min":24.600999999999999,"15min":21.088000000000001},"min":{"1min":6.9740000000000002,"5min":6.9740000000000002,"15min":6.5330000000000004},"max":{"1min":26.076000000000001,"5min":376.24900000000002,"15min":376.24900000000002},"last":24.899000000000001},{"interface":"front","average":{"1min":12.493,"5min":22.207999999999998,"15min":21.219000000000001},"min":{"1min":6.5810000000000004,"5min":6.5110000000000001,"15min":6.0750000000000002},"max":{"1min":29.690999999999999,"5min":271.51100000000002,"15min":434.77300000000002},"last":40.372}]},{"osd":3,"last update":"Mon Jun 3 09:03:40 2024","interfaces":[{"interface":"back","average":{"1min":14.093999999999999,"5min":24.143999999999998,"15min":29.376999999999999},"min":{"1min":10.196999999999999,"5min":6.2919999999999998,"15min":6.1719999999999997},"max":{"1min":24.832000000000001,"5min":197.84899999999999,"15min":1235.5799999999999},"last":24.353999999999999},{"interface":"front","average":{"1min":9.3260000000000005,"5min":18.463999999999999,"15min":27.631},"min":{"1min":6.125,"5min":6.125,"15min":6.125},"max":{"1min":28.166,"5min":200.173,"15min":1157.6780000000001},"last":23.454000000000001}]},{"osd":4,"last update":"Mon Jun 3 09:04:29 2024","interfaces":[{"interface":"back","average":{"1min":10.279,"5min":1253.809,"15min":450.79199999999997},"min":{"1min":7.2300000000000004,"5min":6.093,"15min":6.093},"max":{"1min":15.000999999999999,"5min":28093.096000000001,"15min":28093.096000000001},"last":101.45399999999999},{"interface":"front","average":{"1min":15.693,"5min":1279.759,"15min":463.87099999999998},"min":{"1min":10.551,"5min":7.7480000000000002,"15min":7.7480000000000002},"max":{"1min":27.898,"5min":28323.837,"15min":28323.837},"last":102.262}]},{"osd":5,"last update":"Mon Jun 3 09:04:10 2024","interfaces":[{"interface":"back","average":{"1min":13.728999999999999,"5min":24.670000000000002,"15min":23.815999999999999},"min":{"1min":10.401,"5min":7.516,"15min":7.516},"max":{"1min":25.138000000000002,"5min":320.85000000000002,"15min":445.041},"last":103.52500000000001},{"interface":"front","average":{"1min":13.847,"5min":29.283000000000001,"15min":23.393999999999998},"min":{"1min":7.944,"5min":6.266,"15min":6.266},"max":{"1min":18.817,"5min":530.56700000000001,"15min":530.56700000000001},"last":29.263999999999999}]},{"osd":7,"last update":"Mon Jun 3 09:03:40 2024","interfaces":[{"interface":"back","average":{"1min":20.568999999999999,"5min":30.286999999999999,"15min":27.977},"min":{"1min":13.874000000000001,"5min":10.58,"15min":9.2859999999999996},"max":{"1min":86.959000000000003,"5min":222.07599999999999,"15min":255.08799999999999},"last":37.618000000000002},{"interface":"front","average":{"1min":19.102,"5min":29.550000000000001,"15min":27.763000000000002},"min":{"1min":10.052,"5min":8.6709999999999994,"15min":8.5429999999999993},"max":{"1min":67.968999999999994,"5min":293.62099999999998,"15min":438.14600000000002},"last":111.985}]}]},{"osd":7,"up_from":20,"seq":85899346283,"num_pgs":19,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":58960,"kb_used_data":5696,"kb_used_omap":1,"kb_used_meta":53246,"kb_avail":94312880,"statfs":{"total":96636764160,"available":96576389120,"internally_reserved":0,"allocated":5832704,"data_stored":2771720,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1589,"internal_metadata":54524363},"hb_peers":[0,1,2,3,4,5],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":44,"apply_latency_ms":44,"commit_latency_ns":44000000,"apply_latency_ns":44000000},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Mon Jun 3 09:06:09 2024","interfaces":[{"interface":"back","average":{"1min":14.073,"5min":894.22400000000005,"15min":308.93400000000003},"min":{"1min":5.5709999999999997,"5min":4.218,"15min":4.0599999999999996},"max":{"1min":90.757999999999996,"5min":21729.001,"15min":21729.001},"last":8.2750000000000004},{"interface":"front","average":{"1min":9.2550000000000008,"5min":906.23500000000001,"15min":314.06},"min":{"1min":5.5789999999999997,"5min":4.8810000000000002,"15min":4.8810000000000002},"max":{"1min":25.937000000000001,"5min":21899.732,"15min":21899.732},"last":6.9189999999999996}]},{"osd":1,"last update":"Mon Jun 3 09:06:09 2024","interfaces":[{"interface":"back","average":{"1min":15.182,"5min":69.183999999999997,"15min":219.173},"min":{"1min":7.3639999999999999,"5min":6.6200000000000001,"15min":6.1319999999999997},"max":{"1min":76.926000000000002,"5min":2795.1329999999998,"15min":18373.257000000001},"last":13.550000000000001},{"interface":"front","average":{"1min":21.495999999999999,"5min":74.703999999999994,"15min":225.285},"min":{"1min":8.8399999999999999,"5min":8.8399999999999999,"15min":8.5129999999999999},"max":{"1min":94.977999999999994,"5min":2803.3670000000002,"15min":18374.601999999999},"last":15.414}]},{"osd":2,"last update":"Mon Jun 3 09:06:09 2024","interfaces":[{"interface":"back","average":{"1min":19.164000000000001,"5min":26.902000000000001,"15min":22.920999999999999},"min":{"1min":11.207000000000001,"5min":8.5030000000000001,"15min":8.5030000000000001},"max":{"1min":91.319999999999993,"5min":333.99200000000002,"15min":333.99200000000002},"last":22.468},{"interface":"front","average":{"1min":11.125999999999999,"5min":18.132000000000001,"15min":16.353000000000002},"min":{"1min":6.202,"5min":5.8369999999999997,"15min":5.5069999999999997},"max":{"1min":28.649000000000001,"5min":226.18100000000001,"15min":226.18100000000001},"last":28.649000000000001}]},{"osd":3,"last update":"Mon Jun 3 09:06:09 2024","interfaces":[{"interface":"back","average":{"1min":22.291,"5min":26.289999999999999,"15min":31.074999999999999},"min":{"1min":11.714,"5min":6.8959999999999999,"15min":6.3760000000000003},"max":{"1min":83.102000000000004,"5min":316.64400000000001,"15min":2039.9059999999999},"last":48.323},{"interface":"front","average":{"1min":13.324999999999999,"5min":17.963000000000001,"15min":25.969999999999999},"min":{"1min":6.1200000000000001,"5min":5.2089999999999996,"15min":5.2089999999999996},"max":{"1min":49.908999999999999,"5min":233.542,"15min":2037.5550000000001},"last":41.325000000000003}]},{"osd":4,"last update":"Mon Jun 3 09:05:28 2024","interfaces":[{"interface":"back","average":{"1min":22.209,"5min":1379.51,"15min":1803.278},"min":{"1min":5.9089999999999998,"5min":5.9089999999999998,"15min":5.9089999999999998},"max":{"1min":224.54300000000001,"5min":30073.917000000001,"15min":62560.868000000002},"last":18.774999999999999},{"interface":"front","average":{"1min":22.712,"5min":1401.0719999999999,"15min":1806.759},"min":{"1min":7.6449999999999996,"5min":6.7930000000000001,"15min":5.9340000000000002},"max":{"1min":223.58600000000001,"5min":30348.021000000001,"15min":62451.446000000004},"last":5.9279999999999999}]},{"osd":5,"last update":"Mon Jun 3 09:06:02 2024","interfaces":[{"interface":"back","average":{"1min":24.372,"5min":26.856000000000002,"15min":23.571999999999999},"min":{"1min":12.278,"5min":9.2460000000000004,"15min":6.5650000000000004},"max":{"1min":183.64099999999999,"5min":317.572,"15min":317.572},"last":12.349},{"interface":"front","average":{"1min":21.259,"5min":23.364000000000001,"15min":21.193999999999999},"min":{"1min":7.8499999999999996,"5min":7.0999999999999996,"15min":7.008},"max":{"1min":184.53399999999999,"5min":306.60599999999999,"15min":306.60599999999999},"last":9.327}]},{"osd":6,"last update":"Mon Jun 3 09:04:31 2024","interfaces":[{"interface":"back","average":{"1min":44.061,"5min":26.821000000000002,"15min":91.176000000000002},"min":{"1min":9.0800000000000001,"5min":7.9390000000000001,"15min":7.9390000000000001},"max":{"1min":268.029,"5min":308.916,"15min":8734.4639999999999},"last":6.6769999999999996},{"interface":"front","average":{"1min":46.643000000000001,"5min":23.879999999999999,"15min":88.765000000000001},"min":{"1min":8.8119999999999994,"5min":6.9240000000000004,"15min":6.742},"max":{"1min":305.64600000000002,"5min":305.64600000000002,"15min":8735.5319999999992},"last":12.327}]}]},{"osd":5,"up_from":20,"seq":85899346257,"num_pgs":18,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":59388,"kb_used_data":6128,"kb_used_omap":1,"kb_used_meta":53246,"kb_avail":94312452,"statfs":{"total":96636764160,"available":96575950848,"internally_reserved":0,"allocated":6275072,"data_stored":3211775,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1594,"internal_metadata":54524358},"hb_peers":[0,1,2,3,4,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":75,"apply_latency_ms":75,"commit_latency_ns":75000000,"apply_latency_ns":75000000},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Mon Jun 3 09:06:06 2024","interfaces":[{"interface":"back","average":{"1min":7.468,"5min":1703.037,"15min":580.51900000000001},"min":{"1min":4.6509999999999998,"5min":4.6509999999999998,"15min":4.4020000000000001},"max":{"1min":14.711,"5min":26433.764999999999,"15min":26433.764999999999},"last":5.5960000000000001},{"interface":"front","average":{"1min":9.9429999999999996,"5min":1725.27,"15min":587.87800000000004},"min":{"1min":5.5999999999999996,"5min":5.1319999999999997,"15min":4.8170000000000002},"max":{"1min":13.590999999999999,"5min":26704.698,"15min":26704.698},"last":9.7089999999999996}]},{"osd":1,"last update":"Mon Jun 3 09:06:00 2024","interfaces":[{"interface":"back","average":{"1min":10.862,"5min":67.731999999999999,"15min":217.364},"min":{"1min":7.1769999999999996,"5min":6.0890000000000004,"15min":5.4930000000000003},"max":{"1min":16.998999999999999,"5min":3923.4090000000001,"15min":18811.696},"last":11.817},{"interface":"front","average":{"1min":10.417,"5min":65.141999999999996,"15min":220.75700000000001},"min":{"1min":5.3209999999999997,"5min":5.3209999999999997,"15min":5.0209999999999999},"max":{"1min":15.821999999999999,"5min":4019.6129999999998,"15min":18968.365000000002},"last":11.131}]},{"osd":2,"last update":"Mon Jun 3 09:06:00 2024","interfaces":[{"interface":"back","average":{"1min":11.366,"5min":24.18,"15min":21.137},"min":{"1min":5.6369999999999996,"5min":5.6369999999999996,"15min":5.0510000000000002},"max":{"1min":15.529999999999999,"5min":332.48200000000003,"15min":332.48200000000003},"last":15.108000000000001},{"interface":"front","average":{"1min":10.089,"5min":23.117000000000001,"15min":24.050000000000001},"min":{"1min":6.6769999999999996,"5min":6.6769999999999996,"15min":5.8739999999999997},"max":{"1min":15.821,"5min":353.03199999999998,"15min":353.03199999999998},"last":9.1140000000000008}]},{"osd":3,"last update":"Mon Jun 3 09:06:00 2024","interfaces":[{"interface":"back","average":{"1min":13.472,"5min":24.283999999999999,"15min":31.666},"min":{"1min":10.111000000000001,"5min":7.4720000000000004,"15min":7.4210000000000003},"max":{"1min":16.271999999999998,"5min":353.57499999999999,"15min":2855.8519999999999},"last":17.591000000000001},{"interface":"front","average":{"1min":9.5899999999999999,"5min":15.875,"15min":25.834},"min":{"1min":6.6609999999999996,"5min":5.7729999999999997,"15min":5.1909999999999998},"max":{"1min":15.112,"5min":203.804,"15min":2869.7860000000001},"last":13.867000000000001}]},{"osd":4,"last update":"Mon Jun 3 09:05:27 2024","interfaces":[{"interface":"back","average":{"1min":19.155000000000001,"5min":1084.961,"15min":1346.9949999999999},"min":{"1min":7.6130000000000004,"5min":7.6130000000000004,"15min":6.1340000000000003},"max":{"1min":156.55199999999999,"5min":28619.825000000001,"15min":57999.453999999998},"last":12.958},{"interface":"front","average":{"1min":28.068999999999999,"5min":1087.106,"15min":1330.289},"min":{"1min":8.3949999999999996,"5min":6.5030000000000001,"15min":6.5030000000000001},"max":{"1min":254.91300000000001,"5min":28621.294000000002,"15min":57434.631999999998},"last":16.262}]},{"osd":6,"last update":"Mon Jun 3 09:03:57 2024","interfaces":[{"interface":"back","average":{"1min":23.536000000000001,"5min":36.457000000000001,"15min":83.799999999999997},"min":{"1min":9.4800000000000004,"5min":9.4800000000000004,"15min":8.0630000000000006},"max":{"1min":89.442999999999998,"5min":508.77999999999997,"15min":9193.5169999999998},"last":157.79300000000001},{"interface":"front","average":{"1min":18.891999999999999,"5min":22.100999999999999,"15min":75.489999999999995},"min":{"1min":5.923,"5min":5.923,"15min":5.923},"max":{"1min":78.424999999999997,"5min":464.62,"15min":9191.4429999999993},"last":148.09}]},{"osd":7,"last update":"Mon Jun 3 09:06:00 2024","interfaces":[{"interface":"back","average":{"1min":13.257,"5min":20.917999999999999,"15min":21.088999999999999},"min":{"1min":10.475,"5min":6.7629999999999999,"15min":6.7629999999999999},"max":{"1min":17.245000000000001,"5min":211.22999999999999,"15min":211.22999999999999},"last":28.895},{"interface":"front","average":{"1min":12.872999999999999,"5min":20.587,"15min":24.021000000000001},"min":{"1min":8.5760000000000005,"5min":7.7709999999999999,"15min":6.8570000000000002},"max":{"1min":18.359999999999999,"5min":223.191,"15min":286.69499999999999},"last":36.515000000000001}]}]},{"osd":4,"up_from":20,"seq":85899346265,"num_pgs":20,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":63000,"kb_used_data":5644,"kb_used_omap":1,"kb_used_meta":57342,"kb_avail":94308840,"statfs":{"total":96636764160,"available":96572252160,"internally_reserved":0,"allocated":5779456,"data_stored":2718472,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1594,"internal_metadata":58718662},"hb_peers":[0,1,2,3,5,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":50,"apply_latency_ms":50,"commit_latency_ns":50000000,"apply_latency_ns":50000000},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Mon Jun 3 09:05:34 2024","interfaces":[{"interface":"back","average":{"1min":12.416,"5min":1099.4649999999999,"15min":382.15199999999999},"min":{"1min":4.5279999999999996,"5min":4.5279999999999996,"15min":4.3220000000000001},"max":{"1min":123.878,"5min":21726.743999999999,"15min":21726.743999999999},"last":52.241},{"interface":"front","average":{"1min":16.462,"5min":1108.894,"15min":387.70299999999997},"min":{"1min":6.4089999999999998,"5min":5.8380000000000001,"15min":5.4909999999999997},"max":{"1min":130.185,"5min":21727.993999999999,"15min":21727.993999999999},"last":53.148000000000003}]},{"osd":1,"last update":"Mon Jun 3 09:05:34 2024","interfaces":[{"interface":"back","average":{"1min":14.727,"5min":56.755000000000003,"15min":341.09399999999999},"min":{"1min":4.6529999999999996,"5min":4.1970000000000001,"15min":4.1970000000000001},"max":{"1min":126.899,"5min":3346.4389999999999,"15min":22256.098000000002},"last":47.073},{"interface":"front","average":{"1min":13.472,"5min":57.503,"15min":340.303},"min":{"1min":5.0590000000000002,"5min":5.0590000000000002,"15min":4.2640000000000002},"max":{"1min":126.096,"5min":3347.181,"15min":22247.317999999999},"last":49.558}]},{"osd":2,"last update":"Mon Jun 3 09:05:34 2024","interfaces":[{"interface":"back","average":{"1min":26.088999999999999,"5min":23.866,"15min":20.620999999999999},"min":{"1min":7.0869999999999997,"5min":6.8239999999999998,"15min":6.3150000000000004},"max":{"1min":337.60399999999998,"5min":393.58600000000001,"15min":393.58600000000001},"last":85.340000000000003},{"interface":"front","average":{"1min":15.800000000000001,"5min":22.584,"15min":23.245999999999999},"min":{"1min":6.0629999999999997,"5min":6.0629999999999997,"15min":6.0629999999999997},"max":{"1min":127.913,"5min":394.471,"15min":394.471},"last":82.120000000000005}]},{"osd":3,"last update":"Mon Jun 3 09:05:34 2024","interfaces":[{"interface":"back","average":{"1min":17.544,"5min":19.143999999999998,"15min":18.742000000000001},"min":{"1min":5.8719999999999999,"5min":5.4100000000000001,"15min":5.4000000000000004},"max":{"1min":127.43600000000001,"5min":322.63,"15min":322.63},"last":84.555999999999997},{"interface":"front","average":{"1min":21.327000000000002,"5min":23.559000000000001,"15min":23.797000000000001},"min":{"1min":10.023,"5min":7.484,"15min":7.484},"max":{"1min":204.517,"5min":392.67500000000001,"15min":392.67500000000001},"last":71.394000000000005}]},{"osd":5,"last update":"Mon Jun 3 09:05:34 2024","interfaces":[{"interface":"back","average":{"1min":24.274999999999999,"5min":21.274999999999999,"15min":20.988},"min":{"1min":6.7489999999999997,"5min":6.7489999999999997,"15min":6.7489999999999997},"max":{"1min":268.10199999999998,"5min":344.23399999999998,"15min":344.23399999999998},"last":51.326000000000001},{"interface":"front","average":{"1min":26.361999999999998,"5min":25.218,"15min":26.084},"min":{"1min":12.228,"5min":8.5180000000000007,"15min":8.5180000000000007},"max":{"1min":264.75,"5min":345.18700000000001,"15min":345.18700000000001},"last":40.829000000000001}]},{"osd":6,"last update":"Mon Jun 3 09:04:32 2024","interfaces":[{"interface":"back","average":{"1min":15.548,"5min":32.957999999999998,"15min":152.738},"min":{"1min":9.141,"5min":9.141,"15min":8.2319999999999993},"max":{"1min":52.57,"5min":395.238,"15min":12496.335999999999},"last":208.90000000000001},{"interface":"front","average":{"1min":12.680999999999999,"5min":25.428000000000001,"15min":149.30600000000001},"min":{"1min":6.4379999999999997,"5min":5.6319999999999997,"15min":5.6319999999999997},"max":{"1min":46.283000000000001,"5min":377.74299999999999,"15min":12616.297},"last":205.88900000000001}]},{"osd":7,"last update":"Mon Jun 3 09:05:34 2024","interfaces":[{"interface":"back","average":{"1min":18.295999999999999,"5min":19.585999999999999,"15min":21.062000000000001},"min":{"1min":6.2110000000000003,"5min":6.2110000000000003,"15min":5.8010000000000002},"max":{"1min":207.25299999999999,"5min":358.94900000000001,"15min":358.94900000000001},"last":14.252000000000001},{"interface":"front","average":{"1min":23.712,"5min":24.024999999999999,"15min":24.640000000000001},"min":{"1min":11.824999999999999,"5min":7.7789999999999999,"15min":7.7789999999999999},"max":{"1min":209.32900000000001,"5min":346.25200000000001,"15min":380.06900000000002},"last":45.639000000000003}]}]},{"osd":2,"up_from":20,"seq":85899346285,"num_pgs":24,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":58952,"kb_used_data":5688,"kb_used_omap":1,"kb_used_meta":53246,"kb_avail":94312888,"statfs":{"total":96636764160,"available":96576397312,"internally_reserved":0,"allocated":5824512,"data_stored":2763528,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1594,"internal_metadata":54524358},"hb_peers":[0,1,3,4,5,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":79,"apply_latency_ms":79,"commit_latency_ns":79000000,"apply_latency_ns":79000000},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Mon Jun 3 09:05:21 2024","interfaces":[{"interface":"back","average":{"1min":16.364000000000001,"5min":947.59799999999996,"15min":326.58999999999997},"min":{"1min":6.0830000000000002,"5min":4.7050000000000001,"15min":4.1420000000000003},"max":{"1min":157.60900000000001,"5min":23465.18,"15min":23465.18},"last":8.3209999999999997},{"interface":"front","average":{"1min":12.276,"5min":945.29899999999998,"15min":326.78399999999999},"min":{"1min":5.1989999999999998,"5min":4.7439999999999998,"15min":4.6950000000000003},"max":{"1min":102.557,"5min":23465.966,"15min":23465.966},"last":7.2119999999999997}]},{"osd":1,"last update":"Mon Jun 3 09:05:21 2024","interfaces":[{"interface":"back","average":{"1min":17.803999999999998,"5min":77.658000000000001,"15min":294.96899999999999},"min":{"1min":7.2130000000000001,"5min":5.4199999999999999,"15min":5.4199999999999999},"max":{"1min":159.624,"5min":4220.5290000000005,"15min":18954.987000000001},"last":6.4089999999999998},{"interface":"front","average":{"1min":21.542999999999999,"5min":83.528999999999996,"15min":303.11700000000002},"min":{"1min":8.8200000000000003,"5min":7.5949999999999998,"15min":7.5949999999999998},"max":{"1min":170.74299999999999,"5min":4325.9489999999996,"15min":19143.834999999999},"last":11.071}]},{"osd":3,"last update":"Mon Jun 3 09:05:21 2024","interfaces":[{"interface":"back","average":{"1min":19.164999999999999,"5min":13.284000000000001,"15min":28.751999999999999},"min":{"1min":7.3710000000000004,"5min":5.9290000000000003,"15min":5.2450000000000001},"max":{"1min":118.913,"5min":118.913,"15min":2594.8150000000001},"last":8.8699999999999992},{"interface":"front","average":{"1min":22.094000000000001,"5min":17.244,"15min":32.173999999999999},"min":{"1min":9.516,"5min":8.0760000000000005,"15min":6.0780000000000003},"max":{"1min":156.34700000000001,"5min":156.34700000000001,"15min":2706.5120000000002},"last":11.682}]},{"osd":4,"last update":"Mon Jun 3 09:05:57 2024","interfaces":[{"interface":"back","average":{"1min":10.573,"5min":1266.894,"15min":438.154},"min":{"1min":8.6379999999999999,"5min":7.8789999999999996,"15min":6.2039999999999997},"max":{"1min":15.105,"5min":26660.861000000001,"15min":26660.861000000001},"last":14.795},{"interface":"front","average":{"1min":12.407999999999999,"5min":1266.7460000000001,"15min":435.75400000000002},"min":{"1min":10.266999999999999,"5min":10.266999999999999,"15min":6.2720000000000002},"max":{"1min":15.228,"5min":26616.243999999999,"15min":26616.243999999999},"last":15.5}]},{"osd":5,"last update":"Mon Jun 3 09:05:21 2024","interfaces":[{"interface":"back","average":{"1min":25.632000000000001,"5min":20.437999999999999,"15min":21.472999999999999},"min":{"1min":9.2159999999999993,"5min":9.2159999999999993,"15min":6.2050000000000001},"max":{"1min":252.31100000000001,"5min":252.31100000000001,"15min":252.31100000000001},"last":13.285},{"interface":"front","average":{"1min":24.824000000000002,"5min":18.521000000000001,"15min":21.027000000000001},"min":{"1min":8.0449999999999999,"5min":7.4850000000000003,"15min":6.8810000000000002},"max":{"1min":258.31700000000001,"5min":258.31700000000001,"15min":258.31700000000001},"last":9.5879999999999992}]},{"osd":6,"last update":"Mon Jun 3 09:04:28 2024","interfaces":[{"interface":"back","average":{"1min":14.063000000000001,"5min":24.850000000000001,"15min":106.755},"min":{"1min":9.0820000000000007,"5min":8.9649999999999999,"15min":8.9649999999999999},"max":{"1min":24.666,"5min":188.06700000000001,"15min":9521.0010000000002},"last":183.60900000000001},{"interface":"front","average":{"1min":13.933999999999999,"5min":21.081,"15min":106.142},"min":{"1min":5.8719999999999999,"5min":5.8719999999999999,"15min":5.8719999999999999},"max":{"1min":117.551,"5min":193.887,"15min":9585.6620000000003},"last":184.20599999999999}]},{"osd":7,"last update":"Mon Jun 3 09:05:21 2024","interfaces":[{"interface":"back","average":{"1min":25.097000000000001,"5min":22.579999999999998,"15min":24.388000000000002},"min":{"1min":7.8959999999999999,"5min":6.6130000000000004,"15min":6.6130000000000004},"max":{"1min":242.56800000000001,"5min":242.56800000000001,"15min":242.56800000000001},"last":12.548},{"interface":"front","average":{"1min":24.414999999999999,"5min":23.995999999999999,"15min":26.614999999999998},"min":{"1min":12.43,"5min":9.0660000000000007,"15min":9.0660000000000007},"max":{"1min":181.56399999999999,"5min":182.5,"15min":214.33799999999999},"last":13.968999999999999}]}]},{"osd":3,"up_from":14,"seq":60129542499,"num_pgs":25,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":58944,"kb_used_data":5668,"kb_used_omap":1,"kb_used_meta":53246,"kb_avail":94312896,"statfs":{"total":96636764160,"available":96576405504,"internally_reserved":0,"allocated":5804032,"data_stored":2723318,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1592,"internal_metadata":54524360},"hb_peers":[0,1,2,4,5,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":88,"apply_latency_ms":88,"commit_latency_ns":88000000,"apply_latency_ns":88000000},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Mon Jun 3 09:05:25 2024","interfaces":[{"interface":"back","average":{"1min":8.4809999999999999,"5min":1064.2270000000001,"15min":367.88600000000002},"min":{"1min":6.0529999999999999,"5min":5.173,"15min":4.6929999999999996},"max":{"1min":12.010999999999999,"5min":23275.167000000001,"15min":23275.167000000001},"last":7.6280000000000001},{"interface":"front","average":{"1min":13.885,"5min":1070.3710000000001,"15min":373.53500000000003},"min":{"1min":9.6679999999999993,"5min":7.2089999999999996,"15min":5.9880000000000004},"max":{"1min":32.426000000000002,"5min":23278.536,"15min":23278.536},"last":8.7599999999999998}]},{"osd":1,"last update":"Mon Jun 3 09:05:25 2024","interfaces":[{"interface":"back","average":{"1min":11.324,"5min":19.920000000000002,"15min":390.07900000000001},"min":{"1min":6.6379999999999999,"5min":4.8200000000000003,"15min":4.8200000000000003},"max":{"1min":26.449000000000002,"5min":250.72300000000001,"15min":21658.205999999998},"last":6.4059999999999997},{"interface":"front","average":{"1min":9.468,"5min":20.966999999999999,"15min":381.18799999999999},"min":{"1min":5.4969999999999999,"5min":5.2779999999999996,"15min":5.2779999999999996},"max":{"1min":25.643000000000001,"5min":335.33699999999999,"15min":21307.069},"last":11.488}]},{"osd":2,"last update":"Mon Jun 3 09:05:25 2024","interfaces":[{"interface":"back","average":{"1min":13.215,"5min":26.649000000000001,"15min":23.783999999999999},"min":{"1min":7.8710000000000004,"5min":6.4900000000000002,"15min":5.875},"max":{"1min":34.322000000000003,"5min":377.803,"15min":377.803},"last":10.364000000000001},{"interface":"front","average":{"1min":13.105,"5min":22.292999999999999,"15min":20.573},"min":{"1min":7.327,"5min":6.125,"15min":5.9020000000000001},"max":{"1min":32.756999999999998,"5min":371.601,"15min":371.601},"last":12.686999999999999}]},{"osd":4,"last update":"Mon Jun 3 09:05:27 2024","interfaces":[{"interface":"back","average":{"1min":10.444000000000001,"5min":1211.4970000000001,"15min":1639.1579999999999},"min":{"1min":6.4429999999999996,"5min":6.2560000000000002,"15min":6.0309999999999997},"max":{"1min":21.263000000000002,"5min":30144.939999999999,"15min":62679.622000000003},"last":12.101000000000001},{"interface":"front","average":{"1min":11.363,"5min":1219.6400000000001,"15min":1644.127},"min":{"1min":7.8109999999999999,"5min":7.8109999999999999,"15min":6.2439999999999998},"max":{"1min":20.108000000000001,"5min":30180.689999999999,"15min":62681.021999999997},"last":16.518000000000001}]},{"osd":5,"last update":"Mon Jun 3 09:06:07 2024","interfaces":[{"interface":"back","average":{"1min":12.802,"5min":19.640999999999998,"15min":22.318000000000001},"min":{"1min":9.0890000000000004,"5min":6.8810000000000002,"15min":6.7690000000000001},"max":{"1min":16.745999999999999,"5min":249.88,"15min":327.34899999999999},"last":9.7550000000000008},{"interface":"front","average":{"1min":15.002000000000001,"5min":24.003,"15min":26.649999999999999},"min":{"1min":11.984,"5min":9.9149999999999991,"15min":8.9879999999999995},"max":{"1min":18.407,"5min":251.47800000000001,"15min":331.86799999999999},"last":13.369999999999999}]},{"osd":6,"last update":"Mon Jun 3 09:04:04 2024","interfaces":[{"interface":"back","average":{"1min":22.815999999999999,"5min":31.513999999999999,"15min":181.74299999999999},"min":{"1min":9.4809999999999999,"5min":7.2240000000000002,"15min":7.2240000000000002},"max":{"1min":160.56999999999999,"5min":380.613,"15min":11893.74},"last":19.359999999999999},{"interface":"front","average":{"1min":25.359999999999999,"5min":30.981000000000002,"15min":179.81399999999999},"min":{"1min":9.8870000000000005,"5min":7.3360000000000003,"15min":6.0899999999999999},"max":{"1min":193.11099999999999,"5min":379.80700000000002,"15min":11893.069},"last":8.202}]},{"osd":7,"last update":"Mon Jun 3 09:05:25 2024","interfaces":[{"interface":"back","average":{"1min":12.177,"5min":21.126000000000001,"15min":22.972000000000001},"min":{"1min":8.6769999999999996,"5min":7.4279999999999999,"15min":7.4279999999999999},"max":{"1min":17.376999999999999,"5min":164.27500000000001,"15min":178.76300000000001},"last":14.525},{"interface":"front","average":{"1min":15.675000000000001,"5min":22.722000000000001,"15min":25.238},"min":{"1min":11.233000000000001,"5min":10.555999999999999,"15min":7.6440000000000001},"max":{"1min":29.280000000000001,"5min":148.13800000000001,"15min":417.22800000000001},"last":15.449}]}]},{"osd":1,"up_from":14,"seq":60129542497,"num_pgs":17,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":59296,"kb_used_data":6036,"kb_used_omap":1,"kb_used_meta":53246,"kb_avail":94312544,"statfs":{"total":96636764160,"available":96576045056,"internally_reserved":0,"allocated":6180864,"data_stored":3116312,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":54524357},"hb_peers":[0,2,3,4,5,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":38,"apply_latency_ms":38,"commit_latency_ns":38000000,"apply_latency_ns":38000000},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Mon Jun 3 09:06:07 2024","interfaces":[{"interface":"back","average":{"1min":7.7530000000000001,"5min":185.66999999999999,"15min":77.347999999999999},"min":{"1min":5.0229999999999997,"5min":4.9279999999999999,"15min":4.5449999999999999},"max":{"1min":11.896000000000001,"5min":7470.616,"15min":7470.616},"last":5.0229999999999997},{"interface":"front","average":{"1min":9.9220000000000006,"5min":187.14599999999999,"15min":76.325999999999993},"min":{"1min":5.3159999999999998,"5min":5.3159999999999998,"15min":5.3159999999999998},"max":{"1min":18.338000000000001,"5min":7477.0829999999996,"15min":7477.0829999999996},"last":8.5079999999999991}]},{"osd":2,"last update":"Mon Jun 3 09:05:18 2024","interfaces":[{"interface":"back","average":{"1min":10.954000000000001,"5min":18.295000000000002,"15min":24.126999999999999},"min":{"1min":6.8739999999999997,"5min":5.8650000000000002,"15min":5.8650000000000002},"max":{"1min":21.122,"5min":228,"15min":657.38},"last":7.9340000000000002},{"interface":"front","average":{"1min":13.694000000000001,"5min":24.446999999999999,"15min":32.055999999999997},"min":{"1min":9.9860000000000007,"5min":9.1229999999999993,"15min":7.2160000000000002},"max":{"1min":26.507999999999999,"5min":229.465,"15min":700.06500000000005},"last":9.077}]},{"osd":3,"last update":"Mon Jun 3 09:06:04 2024","interfaces":[{"interface":"back","average":{"1min":8.6110000000000007,"5min":15.733000000000001,"15min":26.646999999999998},"min":{"1min":5.827,"5min":5.4640000000000004,"15min":5.4640000000000004},"max":{"1min":13.105,"5min":121.619,"15min":1676.009},"last":10.237},{"interface":"front","average":{"1min":9.5850000000000009,"5min":16.161999999999999,"15min":29.065999999999999},"min":{"1min":6.3630000000000004,"5min":5.7699999999999996,"15min":5.7699999999999996},"max":{"1min":15.166,"5min":132.036,"15min":1766.5840000000001},"last":6.0709999999999997}]},{"osd":4,"last update":"Mon Jun 3 09:05:31 2024","interfaces":[{"interface":"back","average":{"1min":10.996,"5min":954.46500000000003,"15min":1281.6859999999999},"min":{"1min":7.702,"5min":6.1760000000000002,"15min":6.0970000000000004},"max":{"1min":21.538,"5min":28044.862000000001,"15min":59438.017999999996},"last":7.1989999999999998},{"interface":"front","average":{"1min":13.646000000000001,"5min":961.24599999999998,"15min":1285.26},"min":{"1min":9.7200000000000006,"5min":9.3940000000000001,"15min":6.4640000000000004},"max":{"1min":23.277999999999999,"5min":28045.624,"15min":59439.283000000003},"last":13.891}]},{"osd":5,"last update":"Mon Jun 3 09:06:04 2024","interfaces":[{"interface":"back","average":{"1min":12.478999999999999,"5min":20.228000000000002,"15min":25.997},"min":{"1min":8.6430000000000007,"5min":7.9720000000000004,"15min":7.234},"max":{"1min":17.442,"5min":139.43199999999999,"15min":701.22000000000003},"last":11.071},{"interface":"front","average":{"1min":13.507999999999999,"5min":25.760999999999999,"15min":27.553999999999998},"min":{"1min":10.042999999999999,"5min":7.258,"15min":6.242},"max":{"1min":20.611000000000001,"5min":278.63,"15min":598.35000000000002},"last":12.228999999999999}]},{"osd":6,"last update":"Mon Jun 3 09:03:54 2024","interfaces":[{"interface":"back","average":{"1min":28.524000000000001,"5min":20.654,"15min":23.878},"min":{"1min":6.0309999999999997,"5min":6.0309999999999997,"15min":6.0309999999999997},"max":{"1min":167.221,"5min":167.221,"15min":704.245},"last":18.721},{"interface":"front","average":{"1min":27.606000000000002,"5min":30.131,"15min":28.628},"min":{"1min":9.8650000000000002,"5min":9.7200000000000006,"15min":8.3580000000000005},"max":{"1min":123.938,"5min":358.23000000000002,"15min":634.05499999999995},"last":173.113}]},{"osd":7,"last update":"Mon Jun 3 09:05:18 2024","interfaces":[{"interface":"back","average":{"1min":21.853999999999999,"5min":24.245999999999999,"15min":32.801000000000002},"min":{"1min":9.5020000000000007,"5min":8.0559999999999992,"15min":8.0559999999999992},"max":{"1min":142.245,"5min":270.95100000000002,"15min":703.07799999999997},"last":11.706},{"interface":"front","average":{"1min":21.547000000000001,"5min":29.977,"15min":30.513999999999999},"min":{"1min":11.167,"5min":8.2089999999999996,"15min":7.7599999999999998},"max":{"1min":119.348,"5min":360.99900000000002,"15min":608.44100000000003},"last":12.759}]}]},{"osd":0,"up_from":14,"seq":60129542499,"num_pgs":12,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":81072,"kb_used_data":5668,"kb_used_omap":1,"kb_used_meta":75390,"kb_avail":94290768,"statfs":{"total":96636764160,"available":96553746432,"internally_reserved":0,"allocated":5804032,"data_stored":2723318,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1593,"internal_metadata":77199815},"hb_peers":[1,2,3,4,5,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":45,"apply_latency_ms":45,"commit_latency_ns":45000000,"apply_latency_ns":45000000},"alerts":[],"network_ping_times":[{"osd":1,"last update":"Mon Jun 3 09:06:04 2024","interfaces":[{"interface":"back","average":{"1min":6.9960000000000004,"5min":56.997,"15min":339.29700000000003},"min":{"1min":4.4530000000000003,"5min":4.4530000000000003,"15min":4.4100000000000001},"max":{"1min":11.720000000000001,"5min":2414.4749999999999,"15min":20691.453000000001},"last":9.875},{"interface":"front","average":{"1min":9.548,"5min":57.258000000000003,"15min":337.791},"min":{"1min":5.2300000000000004,"5min":5.2300000000000004,"15min":4.6050000000000004},"max":{"1min":15.337999999999999,"5min":2412.9549999999999,"15min":20640.792000000001},"last":4.7199999999999998}]},{"osd":2,"last update":"Mon Jun 3 09:05:22 2024","interfaces":[{"interface":"back","average":{"1min":9.5820000000000007,"5min":27.760000000000002,"15min":23.097999999999999},"min":{"1min":5.766,"5min":5.766,"15min":5.766},"max":{"1min":46.584000000000003,"5min":627.40700000000004,"15min":627.40700000000004},"last":6.3840000000000003},{"interface":"front","average":{"1min":13.574999999999999,"5min":35.847999999999999,"15min":30.065000000000001},"min":{"1min":8.6419999999999995,"5min":7.8630000000000004,"15min":6.9260000000000002},"max":{"1min":54.68,"5min":635.62599999999998,"15min":635.62599999999998},"last":8.5039999999999996}]},{"osd":3,"last update":"Mon Jun 3 09:06:04 2024","interfaces":[{"interface":"back","average":{"1min":12.105,"5min":26.925000000000001,"15min":26.584},"min":{"1min":7.8410000000000002,"5min":7.1280000000000001,"15min":6.1909999999999998},"max":{"1min":18.425999999999998,"5min":389.72699999999998,"15min":389.72699999999998},"last":10.986000000000001},{"interface":"front","average":{"1min":13.794,"5min":29.771999999999998,"15min":28.815000000000001},"min":{"1min":10.084,"5min":9.4920000000000009,"15min":6.1849999999999996},"max":{"1min":17.373999999999999,"5min":390.23200000000003,"15min":494.92899999999997},"last":16.532}]},{"osd":4,"last update":"Mon Jun 3 09:05:27 2024","interfaces":[{"interface":"back","average":{"1min":13.478,"5min":1032.126,"15min":1389.729},"min":{"1min":6.5119999999999996,"5min":6.5119999999999996,"15min":6.5119999999999996},"max":{"1min":55.354999999999997,"5min":25555.212,"15min":60989.964},"last":12.74},{"interface":"front","average":{"1min":16.204999999999998,"5min":1030.8389999999999,"15min":1387.778},"min":{"1min":10.282999999999999,"5min":7.1429999999999998,"15min":7.1429999999999998},"max":{"1min":49.945999999999998,"5min":25556.159,"15min":61000.017},"last":13.673}]},{"osd":5,"last update":"Mon Jun 3 09:05:13 2024","interfaces":[{"interface":"back","average":{"1min":13.778,"5min":31.690999999999999,"15min":29.962},"min":{"1min":8.5030000000000001,"5min":7.7030000000000003,"15min":7.7030000000000003},"max":{"1min":60.216999999999999,"5min":455.78699999999998,"15min":605.05899999999997},"last":14.942},{"interface":"front","average":{"1min":17.823,"5min":30.498999999999999,"15min":27.434999999999999},"min":{"1min":11.547000000000001,"5min":6.2030000000000003,"15min":6.2030000000000003},"max":{"1min":59.677,"5min":391.31200000000001,"15min":541.11000000000001},"last":17.879000000000001}]},{"osd":6,"last update":"Mon Jun 3 09:04:30 2024","interfaces":[{"interface":"back","average":{"1min":52.496000000000002,"5min":33.139000000000003,"15min":160.83199999999999},"min":{"1min":7.5289999999999999,"5min":7.1040000000000001,"15min":6.3739999999999997},"max":{"1min":457.91699999999997,"5min":457.91699999999997,"15min":11115.258},"last":18.959},{"interface":"front","average":{"1min":49.670000000000002,"5min":33.704999999999998,"15min":162.791},"min":{"1min":12.247999999999999,"5min":11.019,"15min":9.9760000000000009},"max":{"1min":407.05099999999999,"5min":407.05099999999999,"15min":11092.194},"last":17.707000000000001}]},{"osd":7,"last update":"Mon Jun 3 09:05:22 2024","interfaces":[{"interface":"back","average":{"1min":14.503,"5min":29.489000000000001,"15min":33.134999999999998},"min":{"1min":8.5690000000000008,"5min":7.5149999999999997,"15min":6.9859999999999998},"max":{"1min":48.526000000000003,"5min":461.41699999999997,"15min":598.33699999999999},"last":11.83},{"interface":"front","average":{"1min":17.050000000000001,"5min":33.313000000000002,"15min":38.603999999999999},"min":{"1min":11.182,"5min":10.263999999999999,"15min":10.263999999999999},"max":{"1min":58.683,"5min":463.10599999999999,"15min":592.80499999999995},"last":14.407}]}]}],"pool_statfs":[{"poolid":1,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":401408,"data_stored":397840,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":5,"total":0,"available":0,"internally_reserved":0,"allocated":401408,"data_stored":397840,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":4,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":5,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":6,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":17,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":17,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":17,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":17,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":17,"osd":4,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":17,"osd":5,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":5,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":17,"osd":6,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":5,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":17,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0}]}} 2024-06-03T09:06:14.985 DEBUG:teuthology.orchestra.run.smithi088:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph pg dump --format=json 2024-06-03T09:06:15.084 INFO:teuthology.orchestra.run.smithi088.stderr:2024-06-03T09:06:15.084+0000 7feeed5e2640 10 monclient: get_monmap_and_config 2024-06-03T09:06:15.084 INFO:teuthology.orchestra.run.smithi088.stderr:2024-06-03T09:06:15.084+0000 7feeed5e2640 10 monclient: build_initial_monmap 2024-06-03T09:06:15.084 INFO:teuthology.orchestra.run.smithi088.stderr:2024-06-03T09:06:15.084+0000 7feeed5e2640 10 monclient: monmap: 2024-06-03T09:06:15.084 INFO:teuthology.orchestra.run.smithi088.stderr:epoch 0 2024-06-03T09:06:15.084 INFO:teuthology.orchestra.run.smithi088.stderr:fsid ec3c3fdd-3ac6-4d7d-9412-683130ed8b04 2024-06-03T09:06:15.085 INFO:teuthology.orchestra.run.smithi088.stderr:last_changed 2024-06-03T09:06:15.085464+0000 2024-06-03T09:06:15.085 INFO:teuthology.orchestra.run.smithi088.stderr:created 2024-06-03T09:06:15.085464+0000 2024-06-03T09:06:15.085 INFO:teuthology.orchestra.run.smithi088.stderr:min_mon_release 0 (unknown) 2024-06-03T09:06:15.085 INFO:teuthology.orchestra.run.smithi088.stderr:election_strategy: 3 2024-06-03T09:06:15.085 INFO:teuthology.orchestra.run.smithi088.stderr:0: v1:172.21.15.88:6789/0 mon.noname-a 2024-06-03T09:06:15.085 INFO:teuthology.orchestra.run.smithi088.stderr:1: v1:172.21.15.192:6789/0 mon.noname-c 2024-06-03T09:06:15.085 INFO:teuthology.orchestra.run.smithi088.stderr:2: v1:172.21.15.88:6790/0 mon.noname-b 2024-06-03T09:06:15.085 INFO:teuthology.orchestra.run.smithi088.stderr: 2024-06-03T09:06:15.086 INFO:teuthology.orchestra.run.smithi088.stderr:2024-06-03T09:06:15.085+0000 7feeed5e2640 1 Processor -- start 2024-06-03T09:06:15.086 INFO:teuthology.orchestra.run.smithi088.stderr:2024-06-03T09:06:15.085+0000 7feeed5e2640 1 -- start start 2024-06-03T09:06:15.086 INFO:teuthology.orchestra.run.smithi088.stderr:2024-06-03T09:06:15.085+0000 7feeed5e2640 10 monclient: init 2024-06-03T09:06:15.086 INFO:teuthology.orchestra.run.smithi088.stderr:2024-06-03T09:06:15.085+0000 7feeed5e2640 10 monclient: _reopen_session rank -1 2024-06-03T09:06:15.086 INFO:teuthology.orchestra.run.smithi088.stderr:2024-06-03T09:06:15.085+0000 7feeed5e2640 10 monclient: _add_conns ranks=[0,2,1] 2024-06-03T09:06:15.087 INFO:teuthology.orchestra.run.smithi088.stderr:2024-06-03T09:06:15.085+0000 7feeed5e2640 10 monclient(hunting): picked mon.noname-a con 0x7feee806fc70 addr v1:172.21.15.88:6789/0 2024-06-03T09:06:15.087 INFO:teuthology.orchestra.run.smithi088.stderr:2024-06-03T09:06:15.085+0000 7feeed5e2640 10 monclient(hunting): picked mon.noname-b con 0x7feee80ad920 addr v1:172.21.15.88:6790/0 2024-06-03T09:06:15.087 INFO:teuthology.orchestra.run.smithi088.stderr:2024-06-03T09:06:15.085+0000 7feeed5e2640 10 monclient(hunting): picked mon.noname-c con 0x7feee81393f0 addr v1:172.21.15.192:6789/0 2024-06-03T09:06:15.087 INFO:teuthology.orchestra.run.smithi088.stderr:2024-06-03T09:06:15.085+0000 7feeed5e2640 1 -- --> v1:172.21.15.88:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7feee8070420 con 0x7feee806fc70 2024-06-03T09:06:15.087 INFO:teuthology.orchestra.run.smithi088.stderr:2024-06-03T09:06:15.085+0000 7feeed5e2640 1 -- --> v1:172.21.15.192:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7feee8071050 con 0x7feee81393f0 2024-06-03T09:06:15.087 INFO:teuthology.orchestra.run.smithi088.stderr:2024-06-03T09:06:15.085+0000 7feeed5e2640 1 -- --> v1:172.21.15.88:6790/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7feee80add60 con 0x7feee80ad920 2024-06-03T09:06:15.087 INFO:teuthology.orchestra.run.smithi088.stderr:2024-06-03T09:06:15.085+0000 7feeed5e2640 10 monclient(hunting): _renew_subs 2024-06-03T09:06:15.087 INFO:teuthology.orchestra.run.smithi088.stderr:2024-06-03T09:06:15.085+0000 7feeed5e2640 10 monclient(hunting): authenticate will time out at 3030.895782s 2024-06-03T09:06:15.087 INFO:teuthology.orchestra.run.smithi088.stderr:2024-06-03T09:06:15.086+0000 7feee6ffd640 1 --1- >> v1:172.21.15.88:6789/0 conn(0x7feee806fc70 0x7feee813fbb0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.88:6789/0 says I am v1:172.21.15.88:53918/0 (socket says 172.21.15.88:53918) 2024-06-03T09:06:15.087 INFO:teuthology.orchestra.run.smithi088.stderr:2024-06-03T09:06:15.086+0000 7feee6ffd640 1 -- 172.21.15.88:0/1407469252 learned_addr learned my addr 172.21.15.88:0/1407469252 (peer_addr_for_me v1:172.21.15.88:0/0) 2024-06-03T09:06:15.089 INFO:teuthology.orchestra.run.smithi088.stderr:2024-06-03T09:06:15.088+0000 7feee67fc640 1 -- 172.21.15.88:0/1407469252 <== mon.1 v1:172.21.15.192:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 2163296713 0 0) 0x7feee8071050 con 0x7feee81393f0 2024-06-03T09:06:15.089 INFO:teuthology.orchestra.run.smithi088.stderr:2024-06-03T09:06:15.088+0000 7feee67fc640 10 monclient(hunting): _negotiate 2024-06-03T09:06:15.089 INFO:teuthology.orchestra.run.smithi088.stderr:2024-06-03T09:06:15.088+0000 7feee67fc640 10 monclient(hunting): _init_auth method 2 2024-06-03T09:06:15.089 INFO:teuthology.orchestra.run.smithi088.stderr:2024-06-03T09:06:15.088+0000 7feee67fc640 10 monclient(hunting): _init_auth creating new auth 2024-06-03T09:06:15.089 INFO:teuthology.orchestra.run.smithi088.stderr:2024-06-03T09:06:15.088+0000 7feee67fc640 10 monclient(hunting): my global_id is 30133 2024-06-03T09:06:15.089 INFO:teuthology.orchestra.run.smithi088.stderr:2024-06-03T09:06:15.088+0000 7feee67fc640 1 -- 172.21.15.88:0/1407469252 --> v1:172.21.15.192:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7feed4003520 con 0x7feee81393f0 2024-06-03T09:06:15.089 INFO:teuthology.orchestra.run.smithi088.stderr:2024-06-03T09:06:15.088+0000 7feee67fc640 1 -- 172.21.15.88:0/1407469252 <== mon.0 v1:172.21.15.88:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 1659593558 0 0) 0x7feee8070420 con 0x7feee806fc70 2024-06-03T09:06:15.089 INFO:teuthology.orchestra.run.smithi088.stderr:2024-06-03T09:06:15.088+0000 7feee67fc640 10 monclient(hunting): _negotiate 2024-06-03T09:06:15.089 INFO:teuthology.orchestra.run.smithi088.stderr:2024-06-03T09:06:15.088+0000 7feee67fc640 10 monclient(hunting): _init_auth method 2 2024-06-03T09:06:15.090 INFO:teuthology.orchestra.run.smithi088.stderr:2024-06-03T09:06:15.088+0000 7feee67fc640 10 monclient(hunting): _init_auth creating new auth 2024-06-03T09:06:15.090 INFO:teuthology.orchestra.run.smithi088.stderr:2024-06-03T09:06:15.088+0000 7feee67fc640 10 monclient(hunting): my global_id is 30555 2024-06-03T09:06:15.090 INFO:teuthology.orchestra.run.smithi088.stderr:2024-06-03T09:06:15.088+0000 7feee67fc640 1 -- 172.21.15.88:0/1407469252 --> v1:172.21.15.88:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7feee8071050 con 0x7feee806fc70 2024-06-03T09:06:15.090 INFO:teuthology.orchestra.run.smithi088.stderr:2024-06-03T09:06:15.089+0000 7feee67fc640 1 -- 172.21.15.88:0/1407469252 <== mon.2 v1:172.21.15.88:6790/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 1538124124 0 0) 0x7feee80add60 con 0x7feee80ad920 2024-06-03T09:06:15.090 INFO:teuthology.orchestra.run.smithi088.stderr:2024-06-03T09:06:15.089+0000 7feee67fc640 10 monclient(hunting): _negotiate 2024-06-03T09:06:15.090 INFO:teuthology.orchestra.run.smithi088.stderr:2024-06-03T09:06:15.089+0000 7feee67fc640 10 monclient(hunting): _init_auth method 2 2024-06-03T09:06:15.090 INFO:teuthology.orchestra.run.smithi088.stderr:2024-06-03T09:06:15.089+0000 7feee67fc640 10 monclient(hunting): _init_auth creating new auth 2024-06-03T09:06:15.090 INFO:teuthology.orchestra.run.smithi088.stderr:2024-06-03T09:06:15.089+0000 7feee67fc640 10 monclient(hunting): my global_id is 30056 2024-06-03T09:06:15.090 INFO:teuthology.orchestra.run.smithi088.stderr:2024-06-03T09:06:15.089+0000 7feee67fc640 1 -- 172.21.15.88:0/1407469252 --> v1:172.21.15.88:6790/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7feee8070420 con 0x7feee80ad920 2024-06-03T09:06:15.091 INFO:teuthology.orchestra.run.smithi088.stderr:2024-06-03T09:06:15.091+0000 7feee67fc640 1 -- 172.21.15.88:0/1407469252 <== mon.1 v1:172.21.15.192:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 214+0+0 (unknown 3812539968 0 0) 0x7feed4003520 con 0x7feee81393f0 2024-06-03T09:06:15.092 INFO:teuthology.orchestra.run.smithi088.stderr:2024-06-03T09:06:15.091+0000 7feee67fc640 1 -- 172.21.15.88:0/1407469252 >> v1:172.21.15.88:6790/0 conn(0x7feee80ad920 legacy=0x7feee8138ce0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-06-03T09:06:15.092 INFO:teuthology.orchestra.run.smithi088.stderr:2024-06-03T09:06:15.091+0000 7feee67fc640 1 -- 172.21.15.88:0/1407469252 >> v1:172.21.15.88:6789/0 conn(0x7feee806fc70 legacy=0x7feee813fbb0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-06-03T09:06:15.092 INFO:teuthology.orchestra.run.smithi088.stderr:2024-06-03T09:06:15.091+0000 7feee67fc640 10 monclient: _finish_hunting 0 2024-06-03T09:06:15.092 INFO:teuthology.orchestra.run.smithi088.stderr:2024-06-03T09:06:15.091+0000 7feee67fc640 1 monclient: found mon.noname-c 2024-06-03T09:06:15.092 INFO:teuthology.orchestra.run.smithi088.stderr:2024-06-03T09:06:15.091+0000 7feee67fc640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-06-03T09:06:15.092 INFO:teuthology.orchestra.run.smithi088.stderr:2024-06-03T09:06:15.091+0000 7feee67fc640 10 monclient: _send_mon_message to mon.noname-c at v1:172.21.15.192:6789/0 2024-06-03T09:06:15.092 INFO:teuthology.orchestra.run.smithi088.stderr:2024-06-03T09:06:15.091+0000 7feee67fc640 1 -- 172.21.15.88:0/1407469252 --> v1:172.21.15.192:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7feee8071be0 con 0x7feee81393f0 2024-06-03T09:06:15.092 INFO:teuthology.orchestra.run.smithi088.stderr:2024-06-03T09:06:15.091+0000 7feee67fc640 10 monclient: _finish_auth 0 2024-06-03T09:06:15.092 INFO:teuthology.orchestra.run.smithi088.stderr:2024-06-03T09:06:15.091+0000 7feee67fc640 10 monclient: _check_auth_tickets 2024-06-03T09:06:15.092 INFO:teuthology.orchestra.run.smithi088.stderr:2024-06-03T09:06:15.091+0000 7feee67fc640 20 monclient: _check_auth_rotating not needed by client.admin 2024-06-03T09:06:15.092 INFO:teuthology.orchestra.run.smithi088.stderr:2024-06-03T09:06:15.091+0000 7feeed5e2640 5 monclient: authenticate success, global_id 30133 2024-06-03T09:06:15.092 INFO:teuthology.orchestra.run.smithi088.stderr:2024-06-03T09:06:15.091+0000 7feeed5e2640 20 monclient: get_monmap_and_config waiting for monmap|config 2024-06-03T09:06:15.093 INFO:teuthology.orchestra.run.smithi088.stderr:2024-06-03T09:06:15.091+0000 7feee67fc640 1 -- 172.21.15.88:0/1407469252 <== mon.1 v1:172.21.15.192:6789/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 2234374065 0 0) 0x7feee0001990 con 0x7feee81393f0 2024-06-03T09:06:15.093 INFO:teuthology.orchestra.run.smithi088.stderr:2024-06-03T09:06:15.091+0000 7feee67fc640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-06-03T09:06:15.093 INFO:teuthology.orchestra.run.smithi088.stderr:2024-06-03T09:06:15.091+0000 7feee67fc640 10 monclient: got monmap 1 from mon.noname-c (according to old e1) 2024-06-03T09:06:15.093 INFO:teuthology.orchestra.run.smithi088.stderr:2024-06-03T09:06:15.091+0000 7feee67fc640 10 monclient: dump: 2024-06-03T09:06:15.093 INFO:teuthology.orchestra.run.smithi088.stderr:epoch 1 2024-06-03T09:06:15.093 INFO:teuthology.orchestra.run.smithi088.stderr:fsid ec3c3fdd-3ac6-4d7d-9412-683130ed8b04 2024-06-03T09:06:15.093 INFO:teuthology.orchestra.run.smithi088.stderr:last_changed 2024-06-03T08:34:31.885125+0000 2024-06-03T09:06:15.093 INFO:teuthology.orchestra.run.smithi088.stderr:created 2024-06-03T08:34:31.885125+0000 2024-06-03T09:06:15.093 INFO:teuthology.orchestra.run.smithi088.stderr:min_mon_release 18 (reef) 2024-06-03T09:06:15.093 INFO:teuthology.orchestra.run.smithi088.stderr:election_strategy: 3 2024-06-03T09:06:15.093 INFO:teuthology.orchestra.run.smithi088.stderr:0: v1:172.21.15.88:6789/0 mon.a 2024-06-03T09:06:15.093 INFO:teuthology.orchestra.run.smithi088.stderr:1: v1:172.21.15.192:6789/0 mon.b 2024-06-03T09:06:15.093 INFO:teuthology.orchestra.run.smithi088.stderr:2: v1:172.21.15.88:6790/0 mon.c 2024-06-03T09:06:15.093 INFO:teuthology.orchestra.run.smithi088.stderr: 2024-06-03T09:06:15.093 INFO:teuthology.orchestra.run.smithi088.stderr:2024-06-03T09:06:15.091+0000 7feeed5e2640 20 monclient: get_monmap_and_config waiting for monmap|config 2024-06-03T09:06:15.094 INFO:teuthology.orchestra.run.smithi088.stderr:2024-06-03T09:06:15.093+0000 7feee67fc640 1 -- 172.21.15.88:0/1407469252 <== mon.1 v1:172.21.15.192:6789/0 4 ==== config(0 keys) v1 ==== 4+0+0 (unknown 0 0 0) 0x7feee0001b70 con 0x7feee81393f0 2024-06-03T09:06:15.095 INFO:teuthology.orchestra.run.smithi088.stderr:2024-06-03T09:06:15.093+0000 7feee67fc640 10 monclient: handle_config config(0 keys) v1 2024-06-03T09:06:15.095 INFO:teuthology.orchestra.run.smithi088.stderr:2024-06-03T09:06:15.093+0000 7feee67fc640 1 -- 172.21.15.88:0/1407469252 <== mon.1 v1:172.21.15.192:6789/0 5 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 2234374065 0 0) 0x7feee0001f00 con 0x7feee81393f0 2024-06-03T09:06:15.095 INFO:teuthology.orchestra.run.smithi088.stderr:2024-06-03T09:06:15.093+0000 7feee67fc640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-06-03T09:06:15.095 INFO:teuthology.orchestra.run.smithi088.stderr:2024-06-03T09:06:15.093+0000 7feee67fc640 10 monclient: got monmap 1 from mon.b (according to old e1) 2024-06-03T09:06:15.095 INFO:teuthology.orchestra.run.smithi088.stderr:2024-06-03T09:06:15.093+0000 7feee67fc640 10 monclient: dump: 2024-06-03T09:06:15.095 INFO:teuthology.orchestra.run.smithi088.stderr:epoch 1 2024-06-03T09:06:15.095 INFO:teuthology.orchestra.run.smithi088.stderr:fsid ec3c3fdd-3ac6-4d7d-9412-683130ed8b04 2024-06-03T09:06:15.095 INFO:teuthology.orchestra.run.smithi088.stderr:last_changed 2024-06-03T08:34:31.885125+0000 2024-06-03T09:06:15.095 INFO:teuthology.orchestra.run.smithi088.stderr:created 2024-06-03T08:34:31.885125+0000 2024-06-03T09:06:15.095 INFO:teuthology.orchestra.run.smithi088.stderr:min_mon_release 18 (reef) 2024-06-03T09:06:15.095 INFO:teuthology.orchestra.run.smithi088.stderr:election_strategy: 3 2024-06-03T09:06:15.095 INFO:teuthology.orchestra.run.smithi088.stderr:0: v1:172.21.15.88:6789/0 mon.a 2024-06-03T09:06:15.095 INFO:teuthology.orchestra.run.smithi088.stderr:1: v1:172.21.15.192:6789/0 mon.b 2024-06-03T09:06:15.096 INFO:teuthology.orchestra.run.smithi088.stderr:2: v1:172.21.15.88:6790/0 mon.c 2024-06-03T09:06:15.096 INFO:teuthology.orchestra.run.smithi088.stderr: 2024-06-03T09:06:15.096 INFO:teuthology.orchestra.run.smithi088.stderr:2024-06-03T09:06:15.093+0000 7feeed5e2640 10 monclient: get_monmap_and_config success 2024-06-03T09:06:15.096 INFO:teuthology.orchestra.run.smithi088.stderr:2024-06-03T09:06:15.093+0000 7feeed5e2640 4 set_mon_vals no callback set 2024-06-03T09:06:15.096 INFO:teuthology.orchestra.run.smithi088.stderr:2024-06-03T09:06:15.094+0000 7feeed5e2640 10 monclient: shutdown 2024-06-03T09:06:15.096 INFO:teuthology.orchestra.run.smithi088.stderr:2024-06-03T09:06:15.094+0000 7feeed5e2640 20 monclient: shutdown discarding 0 pending message(s) 2024-06-03T09:06:15.096 INFO:teuthology.orchestra.run.smithi088.stderr:2024-06-03T09:06:15.094+0000 7feeed5e2640 1 -- 172.21.15.88:0/1407469252 >> v1:172.21.15.192:6789/0 conn(0x7feee81393f0 legacy=0x7feee814ac50 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-06-03T09:06:15.096 INFO:teuthology.orchestra.run.smithi088.stderr:2024-06-03T09:06:15.094+0000 7feeed5e2640 1 -- 172.21.15.88:0/1407469252 shutdown_connections 2024-06-03T09:06:15.096 INFO:teuthology.orchestra.run.smithi088.stderr:2024-06-03T09:06:15.094+0000 7feeed5e2640 1 -- 172.21.15.88:0/1407469252 >> 172.21.15.88:0/1407469252 conn(0x7feee80b2f20 msgr2=0x7feee80b3300 unknown :-1 s=STATE_NONE l=0).mark_down 2024-06-03T09:06:15.096 INFO:teuthology.orchestra.run.smithi088.stderr:2024-06-03T09:06:15.094+0000 7feeed5e2640 1 -- 172.21.15.88:0/1407469252 shutdown_connections 2024-06-03T09:06:15.096 INFO:teuthology.orchestra.run.smithi088.stderr:2024-06-03T09:06:15.094+0000 7feeed5e2640 1 -- 172.21.15.88:0/1407469252 wait complete. 2024-06-03T09:06:15.098 INFO:teuthology.orchestra.run.smithi088.stderr:2024-06-03T09:06:15.095+0000 7feeed5e2640 10 monclient: build_initial_monmap 2024-06-03T09:06:15.098 INFO:teuthology.orchestra.run.smithi088.stderr:2024-06-03T09:06:15.095+0000 7feeed5e2640 10 monclient: monmap: 2024-06-03T09:06:15.098 INFO:teuthology.orchestra.run.smithi088.stderr:epoch 0 2024-06-03T09:06:15.098 INFO:teuthology.orchestra.run.smithi088.stderr:fsid 00000000-0000-0000-0000-000000000000 2024-06-03T09:06:15.098 INFO:teuthology.orchestra.run.smithi088.stderr:last_changed 0.000000 2024-06-03T09:06:15.098 INFO:teuthology.orchestra.run.smithi088.stderr:created 0.000000 2024-06-03T09:06:15.099 INFO:teuthology.orchestra.run.smithi088.stderr:min_mon_release 0 (unknown) 2024-06-03T09:06:15.099 INFO:teuthology.orchestra.run.smithi088.stderr:election_strategy: 1 2024-06-03T09:06:15.099 INFO:teuthology.orchestra.run.smithi088.stderr:0: v1:172.21.15.88:6789/0 mon.noname-a 2024-06-03T09:06:15.099 INFO:teuthology.orchestra.run.smithi088.stderr:1: v1:172.21.15.192:6789/0 mon.noname-b 2024-06-03T09:06:15.099 INFO:teuthology.orchestra.run.smithi088.stderr:2: v1:172.21.15.88:6790/0 mon.noname-c 2024-06-03T09:06:15.099 INFO:teuthology.orchestra.run.smithi088.stderr: 2024-06-03T09:06:15.099 INFO:teuthology.orchestra.run.smithi088.stderr:2024-06-03T09:06:15.095+0000 7feeed5e2640 1 librados: starting msgr at 2024-06-03T09:06:15.099 INFO:teuthology.orchestra.run.smithi088.stderr:2024-06-03T09:06:15.095+0000 7feeed5e2640 1 librados: starting objecter 2024-06-03T09:06:15.099 INFO:teuthology.orchestra.run.smithi088.stderr:2024-06-03T09:06:15.095+0000 7feeed5e2640 1 Processor -- start 2024-06-03T09:06:15.099 INFO:teuthology.orchestra.run.smithi088.stderr:2024-06-03T09:06:15.096+0000 7feeed5e2640 1 -- start start 2024-06-03T09:06:15.099 INFO:teuthology.orchestra.run.smithi088.stderr:2024-06-03T09:06:15.096+0000 7feeed5e2640 1 librados: setting wanted keys 2024-06-03T09:06:15.099 INFO:teuthology.orchestra.run.smithi088.stderr:2024-06-03T09:06:15.096+0000 7feeed5e2640 1 librados: calling monclient init 2024-06-03T09:06:15.099 INFO:teuthology.orchestra.run.smithi088.stderr:2024-06-03T09:06:15.096+0000 7feeed5e2640 10 monclient: init 2024-06-03T09:06:15.099 INFO:teuthology.orchestra.run.smithi088.stderr:2024-06-03T09:06:15.096+0000 7feeed5e2640 10 monclient: _reopen_session rank -1 2024-06-03T09:06:15.099 INFO:teuthology.orchestra.run.smithi088.stderr:2024-06-03T09:06:15.096+0000 7feeed5e2640 10 monclient: _add_conns ranks=[1,0,2] 2024-06-03T09:06:15.100 INFO:teuthology.orchestra.run.smithi088.stderr:2024-06-03T09:06:15.096+0000 7feeed5e2640 10 monclient(hunting): picked mon.noname-b con 0x7feee806fc70 addr v1:172.21.15.192:6789/0 2024-06-03T09:06:15.100 INFO:teuthology.orchestra.run.smithi088.stderr:2024-06-03T09:06:15.096+0000 7feeed5e2640 10 monclient(hunting): picked mon.noname-a con 0x7feee80ad920 addr v1:172.21.15.88:6789/0 2024-06-03T09:06:15.100 INFO:teuthology.orchestra.run.smithi088.stderr:2024-06-03T09:06:15.096+0000 7feeed5e2640 10 monclient(hunting): picked mon.noname-c con 0x7feee81393f0 addr v1:172.21.15.88:6790/0 2024-06-03T09:06:15.100 INFO:teuthology.orchestra.run.smithi088.stderr:2024-06-03T09:06:15.096+0000 7feeed5e2640 1 -- --> v1:172.21.15.88:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7feee8142a70 con 0x7feee80ad920 2024-06-03T09:06:15.100 INFO:teuthology.orchestra.run.smithi088.stderr:2024-06-03T09:06:15.096+0000 7feeed5e2640 1 -- --> v1:172.21.15.192:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7feee81435d0 con 0x7feee806fc70 2024-06-03T09:06:15.100 INFO:teuthology.orchestra.run.smithi088.stderr:2024-06-03T09:06:15.096+0000 7feeed5e2640 1 -- --> v1:172.21.15.88:6790/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7feee8143760 con 0x7feee81393f0 2024-06-03T09:06:15.100 INFO:teuthology.orchestra.run.smithi088.stderr:2024-06-03T09:06:15.096+0000 7feeed5e2640 10 monclient(hunting): _renew_subs 2024-06-03T09:06:15.100 INFO:teuthology.orchestra.run.smithi088.stderr:2024-06-03T09:06:15.096+0000 7feeed5e2640 10 monclient(hunting): authenticate will time out at 3030.906772s 2024-06-03T09:06:15.100 INFO:teuthology.orchestra.run.smithi088.stderr:2024-06-03T09:06:15.097+0000 7feedffff640 1 --1- >> v1:172.21.15.88:6789/0 conn(0x7feee80ad920 0x7feee80a8b50 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=1).handle_server_banner_and_identify peer v1:172.21.15.88:6789/0 says I am v1:172.21.15.88:53920/0 (socket says 172.21.15.88:53920) 2024-06-03T09:06:15.100 INFO:teuthology.orchestra.run.smithi088.stderr:2024-06-03T09:06:15.097+0000 7feedffff640 1 -- 172.21.15.88:0/746948843 learned_addr learned my addr 172.21.15.88:0/746948843 (peer_addr_for_me v1:172.21.15.88:0/0) 2024-06-03T09:06:15.100 INFO:teuthology.orchestra.run.smithi088.stderr:2024-06-03T09:06:15.097+0000 7feedf7fe640 10 client.?.objecter ms_handle_connect 0x7feee80ad920 2024-06-03T09:06:15.100 INFO:teuthology.orchestra.run.smithi088.stderr:2024-06-03T09:06:15.097+0000 7feedf7fe640 10 client.?.objecter resend_mon_ops 2024-06-03T09:06:15.100 INFO:teuthology.orchestra.run.smithi088.stderr:2024-06-03T09:06:15.097+0000 7feedf7fe640 10 client.?.objecter ms_handle_connect 0x7feee81393f0 2024-06-03T09:06:15.100 INFO:teuthology.orchestra.run.smithi088.stderr:2024-06-03T09:06:15.097+0000 7feedf7fe640 10 client.?.objecter resend_mon_ops 2024-06-03T09:06:15.101 INFO:teuthology.orchestra.run.smithi088.stderr:2024-06-03T09:06:15.099+0000 7feedf7fe640 10 client.?.objecter ms_handle_connect 0x7feee806fc70 2024-06-03T09:06:15.101 INFO:teuthology.orchestra.run.smithi088.stderr:2024-06-03T09:06:15.099+0000 7feedf7fe640 10 client.?.objecter resend_mon_ops 2024-06-03T09:06:15.101 INFO:teuthology.orchestra.run.smithi088.stderr:2024-06-03T09:06:15.099+0000 7feedf7fe640 1 -- 172.21.15.88:0/746948843 <== mon.0 v1:172.21.15.88:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 3875796020 0 0) 0x7feee8142a70 con 0x7feee80ad920 2024-06-03T09:06:15.102 INFO:teuthology.orchestra.run.smithi088.stderr:2024-06-03T09:06:15.100+0000 7feedf7fe640 10 monclient(hunting): _negotiate 2024-06-03T09:06:15.102 INFO:teuthology.orchestra.run.smithi088.stderr:2024-06-03T09:06:15.100+0000 7feedf7fe640 10 monclient(hunting): _init_auth method 2 2024-06-03T09:06:15.102 INFO:teuthology.orchestra.run.smithi088.stderr:2024-06-03T09:06:15.100+0000 7feedf7fe640 10 monclient(hunting): _init_auth creating new auth 2024-06-03T09:06:15.102 INFO:teuthology.orchestra.run.smithi088.stderr:2024-06-03T09:06:15.100+0000 7feedf7fe640 10 monclient(hunting): my global_id is 30558 2024-06-03T09:06:15.102 INFO:teuthology.orchestra.run.smithi088.stderr:2024-06-03T09:06:15.100+0000 7feedf7fe640 1 -- 172.21.15.88:0/746948843 --> v1:172.21.15.88:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7feec8004eb0 con 0x7feee80ad920 2024-06-03T09:06:15.102 INFO:teuthology.orchestra.run.smithi088.stderr:2024-06-03T09:06:15.100+0000 7feedf7fe640 1 -- 172.21.15.88:0/746948843 <== mon.2 v1:172.21.15.88:6790/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 810276211 0 0) 0x7feee8143760 con 0x7feee81393f0 2024-06-03T09:06:15.102 INFO:teuthology.orchestra.run.smithi088.stderr:2024-06-03T09:06:15.100+0000 7feedf7fe640 10 monclient(hunting): _negotiate 2024-06-03T09:06:15.102 INFO:teuthology.orchestra.run.smithi088.stderr:2024-06-03T09:06:15.100+0000 7feedf7fe640 10 monclient(hunting): _init_auth method 2 2024-06-03T09:06:15.102 INFO:teuthology.orchestra.run.smithi088.stderr:2024-06-03T09:06:15.100+0000 7feedf7fe640 10 monclient(hunting): _init_auth creating new auth 2024-06-03T09:06:15.102 INFO:teuthology.orchestra.run.smithi088.stderr:2024-06-03T09:06:15.100+0000 7feedf7fe640 10 monclient(hunting): my global_id is 30059 2024-06-03T09:06:15.102 INFO:teuthology.orchestra.run.smithi088.stderr:2024-06-03T09:06:15.100+0000 7feedf7fe640 1 -- 172.21.15.88:0/746948843 --> v1:172.21.15.88:6790/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7feee8142a70 con 0x7feee81393f0 2024-06-03T09:06:15.102 INFO:teuthology.orchestra.run.smithi088.stderr:2024-06-03T09:06:15.101+0000 7feedf7fe640 1 -- 172.21.15.88:0/746948843 <== mon.1 v1:172.21.15.192:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 320905 0 0) 0x7feee81435d0 con 0x7feee806fc70 2024-06-03T09:06:15.103 INFO:teuthology.orchestra.run.smithi088.stderr:2024-06-03T09:06:15.101+0000 7feedf7fe640 10 monclient(hunting): _negotiate 2024-06-03T09:06:15.103 INFO:teuthology.orchestra.run.smithi088.stderr:2024-06-03T09:06:15.101+0000 7feedf7fe640 10 monclient(hunting): _init_auth method 2 2024-06-03T09:06:15.103 INFO:teuthology.orchestra.run.smithi088.stderr:2024-06-03T09:06:15.101+0000 7feedf7fe640 10 monclient(hunting): _init_auth creating new auth 2024-06-03T09:06:15.103 INFO:teuthology.orchestra.run.smithi088.stderr:2024-06-03T09:06:15.101+0000 7feedf7fe640 10 monclient(hunting): my global_id is 30136 2024-06-03T09:06:15.103 INFO:teuthology.orchestra.run.smithi088.stderr:2024-06-03T09:06:15.101+0000 7feedf7fe640 1 -- 172.21.15.88:0/746948843 --> v1:172.21.15.192:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7feee8143760 con 0x7feee806fc70 2024-06-03T09:06:15.104 INFO:teuthology.orchestra.run.smithi088.stderr:2024-06-03T09:06:15.104+0000 7feedf7fe640 1 -- 172.21.15.88:0/746948843 <== mon.2 v1:172.21.15.88:6790/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 764+0+0 (unknown 1669384014 0 0) 0x7feee8142a70 con 0x7feee81393f0 2024-06-03T09:06:15.105 INFO:teuthology.orchestra.run.smithi088.stderr:2024-06-03T09:06:15.105+0000 7feedf7fe640 1 -- 172.21.15.88:0/746948843 --> v1:172.21.15.88:6790/0 -- auth(proto 2 165 bytes epoch 0) v1 -- 0x7feee81435d0 con 0x7feee81393f0 2024-06-03T09:06:15.105 INFO:teuthology.orchestra.run.smithi088.stderr:2024-06-03T09:06:15.105+0000 7feedf7fe640 1 -- 172.21.15.88:0/746948843 <== mon.2 v1:172.21.15.88:6790/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 2234374065 0 0) 0x7feee0001e80 con 0x7feee81393f0 2024-06-03T09:06:15.105 INFO:teuthology.orchestra.run.smithi088.stderr:2024-06-03T09:06:15.105+0000 7feedf7fe640 10 monclient(hunting): handle_monmap mon_map magic: 0 v1 2024-06-03T09:06:15.105 INFO:teuthology.orchestra.run.smithi088.stderr:2024-06-03T09:06:15.105+0000 7feedf7fe640 10 monclient(hunting): got monmap 1 from mon.noname-c (according to old e1) 2024-06-03T09:06:15.105 INFO:teuthology.orchestra.run.smithi088.stderr:2024-06-03T09:06:15.105+0000 7feedf7fe640 10 monclient(hunting): dump: 2024-06-03T09:06:15.105 INFO:teuthology.orchestra.run.smithi088.stderr:epoch 1 2024-06-03T09:06:15.105 INFO:teuthology.orchestra.run.smithi088.stderr:fsid ec3c3fdd-3ac6-4d7d-9412-683130ed8b04 2024-06-03T09:06:15.105 INFO:teuthology.orchestra.run.smithi088.stderr:last_changed 2024-06-03T08:34:31.885125+0000 2024-06-03T09:06:15.105 INFO:teuthology.orchestra.run.smithi088.stderr:created 2024-06-03T08:34:31.885125+0000 2024-06-03T09:06:15.105 INFO:teuthology.orchestra.run.smithi088.stderr:min_mon_release 18 (reef) 2024-06-03T09:06:15.105 INFO:teuthology.orchestra.run.smithi088.stderr:election_strategy: 3 2024-06-03T09:06:15.105 INFO:teuthology.orchestra.run.smithi088.stderr:0: v1:172.21.15.88:6789/0 mon.a 2024-06-03T09:06:15.106 INFO:teuthology.orchestra.run.smithi088.stderr:1: v1:172.21.15.192:6789/0 mon.b 2024-06-03T09:06:15.106 INFO:teuthology.orchestra.run.smithi088.stderr:2: v1:172.21.15.88:6790/0 mon.c 2024-06-03T09:06:15.106 INFO:teuthology.orchestra.run.smithi088.stderr: 2024-06-03T09:06:15.106 INFO:teuthology.orchestra.run.smithi088.stderr:2024-06-03T09:06:15.105+0000 7feedf7fe640 10 monclient(hunting): _finish_auth 0 2024-06-03T09:06:15.106 INFO:teuthology.orchestra.run.smithi088.stderr:2024-06-03T09:06:15.105+0000 7feedf7fe640 1 -- 172.21.15.88:0/746948843 <== mon.1 v1:172.21.15.192:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 764+0+0 (unknown 1578532331 0 0) 0x7feee8143760 con 0x7feee806fc70 2024-06-03T09:06:15.106 INFO:teuthology.orchestra.run.smithi088.stderr:2024-06-03T09:06:15.105+0000 7feedf7fe640 1 -- 172.21.15.88:0/746948843 --> v1:172.21.15.192:6789/0 -- auth(proto 2 165 bytes epoch 0) v1 -- 0x7feee8142a70 con 0x7feee806fc70 2024-06-03T09:06:15.107 INFO:teuthology.orchestra.run.smithi088.stderr:2024-06-03T09:06:15.105+0000 7feedf7fe640 1 -- 172.21.15.88:0/746948843 <== mon.1 v1:172.21.15.192:6789/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 2234374065 0 0) 0x7feed80016e0 con 0x7feee806fc70 2024-06-03T09:06:15.107 INFO:teuthology.orchestra.run.smithi088.stderr:2024-06-03T09:06:15.105+0000 7feedf7fe640 10 monclient(hunting): handle_monmap mon_map magic: 0 v1 2024-06-03T09:06:15.107 INFO:teuthology.orchestra.run.smithi088.stderr:2024-06-03T09:06:15.105+0000 7feedf7fe640 10 monclient(hunting): got monmap 1 from mon.b (according to old e1) 2024-06-03T09:06:15.107 INFO:teuthology.orchestra.run.smithi088.stderr:2024-06-03T09:06:15.105+0000 7feedf7fe640 10 monclient(hunting): dump: 2024-06-03T09:06:15.107 INFO:teuthology.orchestra.run.smithi088.stderr:epoch 1 2024-06-03T09:06:15.107 INFO:teuthology.orchestra.run.smithi088.stderr:fsid ec3c3fdd-3ac6-4d7d-9412-683130ed8b04 2024-06-03T09:06:15.107 INFO:teuthology.orchestra.run.smithi088.stderr:last_changed 2024-06-03T08:34:31.885125+0000 2024-06-03T09:06:15.107 INFO:teuthology.orchestra.run.smithi088.stderr:created 2024-06-03T08:34:31.885125+0000 2024-06-03T09:06:15.107 INFO:teuthology.orchestra.run.smithi088.stderr:min_mon_release 18 (reef) 2024-06-03T09:06:15.107 INFO:teuthology.orchestra.run.smithi088.stderr:election_strategy: 3 2024-06-03T09:06:15.107 INFO:teuthology.orchestra.run.smithi088.stderr:0: v1:172.21.15.88:6789/0 mon.a 2024-06-03T09:06:15.107 INFO:teuthology.orchestra.run.smithi088.stderr:1: v1:172.21.15.192:6789/0 mon.b 2024-06-03T09:06:15.107 INFO:teuthology.orchestra.run.smithi088.stderr:2: v1:172.21.15.88:6790/0 mon.c 2024-06-03T09:06:15.107 INFO:teuthology.orchestra.run.smithi088.stderr: 2024-06-03T09:06:15.108 INFO:teuthology.orchestra.run.smithi088.stderr:2024-06-03T09:06:15.107+0000 7feedf7fe640 1 -- 172.21.15.88:0/746948843 <== mon.0 v1:172.21.15.88:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 764+0+0 (unknown 1908062045 0 0) 0x7feec8004eb0 con 0x7feee80ad920 2024-06-03T09:06:15.108 INFO:teuthology.orchestra.run.smithi088.stderr:2024-06-03T09:06:15.107+0000 7feedf7fe640 1 -- 172.21.15.88:0/746948843 --> v1:172.21.15.88:6789/0 -- auth(proto 2 165 bytes epoch 0) v1 -- 0x7feee8143760 con 0x7feee80ad920 2024-06-03T09:06:15.108 INFO:teuthology.orchestra.run.smithi088.stderr:2024-06-03T09:06:15.107+0000 7feedf7fe640 1 -- 172.21.15.88:0/746948843 <== mon.0 v1:172.21.15.88:6789/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 2234374065 0 0) 0x7feed0001990 con 0x7feee80ad920 2024-06-03T09:06:15.108 INFO:teuthology.orchestra.run.smithi088.stderr:2024-06-03T09:06:15.107+0000 7feedf7fe640 10 monclient(hunting): handle_monmap mon_map magic: 0 v1 2024-06-03T09:06:15.108 INFO:teuthology.orchestra.run.smithi088.stderr:2024-06-03T09:06:15.107+0000 7feedf7fe640 10 monclient(hunting): got monmap 1 from mon.a (according to old e1) 2024-06-03T09:06:15.108 INFO:teuthology.orchestra.run.smithi088.stderr:2024-06-03T09:06:15.107+0000 7feedf7fe640 10 monclient(hunting): dump: 2024-06-03T09:06:15.108 INFO:teuthology.orchestra.run.smithi088.stderr:epoch 1 2024-06-03T09:06:15.109 INFO:teuthology.orchestra.run.smithi088.stderr:fsid ec3c3fdd-3ac6-4d7d-9412-683130ed8b04 2024-06-03T09:06:15.109 INFO:teuthology.orchestra.run.smithi088.stderr:last_changed 2024-06-03T08:34:31.885125+0000 2024-06-03T09:06:15.109 INFO:teuthology.orchestra.run.smithi088.stderr:created 2024-06-03T08:34:31.885125+0000 2024-06-03T09:06:15.109 INFO:teuthology.orchestra.run.smithi088.stderr:min_mon_release 18 (reef) 2024-06-03T09:06:15.109 INFO:teuthology.orchestra.run.smithi088.stderr:election_strategy: 3 2024-06-03T09:06:15.109 INFO:teuthology.orchestra.run.smithi088.stderr:0: v1:172.21.15.88:6789/0 mon.a 2024-06-03T09:06:15.109 INFO:teuthology.orchestra.run.smithi088.stderr:1: v1:172.21.15.192:6789/0 mon.b 2024-06-03T09:06:15.109 INFO:teuthology.orchestra.run.smithi088.stderr:2: v1:172.21.15.88:6790/0 mon.c 2024-06-03T09:06:15.109 INFO:teuthology.orchestra.run.smithi088.stderr: 2024-06-03T09:06:15.110 INFO:teuthology.orchestra.run.smithi088.stderr:2024-06-03T09:06:15.109+0000 7feedf7fe640 1 -- 172.21.15.88:0/746948843 <== mon.1 v1:172.21.15.192:6789/0 4 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 580+0+0 (unknown 2228420502 0 0) 0x7feee8142a70 con 0x7feee806fc70 2024-06-03T09:06:15.110 INFO:teuthology.orchestra.run.smithi088.stderr:2024-06-03T09:06:15.109+0000 7feedf7fe640 1 -- 172.21.15.88:0/746948843 >> v1:172.21.15.88:6790/0 conn(0x7feee81393f0 legacy=0x7feee80a9260 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-06-03T09:06:15.110 INFO:teuthology.orchestra.run.smithi088.stderr:2024-06-03T09:06:15.109+0000 7feedf7fe640 1 -- 172.21.15.88:0/746948843 >> v1:172.21.15.88:6789/0 conn(0x7feee80ad920 legacy=0x7feee80a8b50 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-06-03T09:06:15.110 INFO:teuthology.orchestra.run.smithi088.stderr:2024-06-03T09:06:15.109+0000 7feedf7fe640 10 monclient: _finish_hunting 0 2024-06-03T09:06:15.111 INFO:teuthology.orchestra.run.smithi088.stderr:2024-06-03T09:06:15.109+0000 7feedf7fe640 1 monclient: found mon.b 2024-06-03T09:06:15.111 INFO:teuthology.orchestra.run.smithi088.stderr:2024-06-03T09:06:15.109+0000 7feedf7fe640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-06-03T09:06:15.111 INFO:teuthology.orchestra.run.smithi088.stderr:2024-06-03T09:06:15.109+0000 7feedf7fe640 10 monclient: _send_mon_message to mon.b at v1:172.21.15.192:6789/0 2024-06-03T09:06:15.111 INFO:teuthology.orchestra.run.smithi088.stderr:2024-06-03T09:06:15.109+0000 7feedf7fe640 1 -- 172.21.15.88:0/746948843 --> v1:172.21.15.192:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7feee81da8b0 con 0x7feee806fc70 2024-06-03T09:06:15.111 INFO:teuthology.orchestra.run.smithi088.stderr:2024-06-03T09:06:15.109+0000 7feedf7fe640 10 monclient: _finish_auth 0 2024-06-03T09:06:15.111 INFO:teuthology.orchestra.run.smithi088.stderr:2024-06-03T09:06:15.109+0000 7feedf7fe640 10 monclient: _check_auth_tickets 2024-06-03T09:06:15.111 INFO:teuthology.orchestra.run.smithi088.stderr:2024-06-03T09:06:15.109+0000 7feedf7fe640 20 monclient: _check_auth_rotating not needed by client.admin 2024-06-03T09:06:15.111 INFO:teuthology.orchestra.run.smithi088.stderr:2024-06-03T09:06:15.109+0000 7feeed5e2640 5 monclient: authenticate success, global_id 30136 2024-06-03T09:06:15.111 INFO:teuthology.orchestra.run.smithi088.stderr:2024-06-03T09:06:15.109+0000 7feeed5e2640 10 monclient: _renew_subs 2024-06-03T09:06:15.111 INFO:teuthology.orchestra.run.smithi088.stderr:2024-06-03T09:06:15.109+0000 7feeed5e2640 10 monclient: _send_mon_message to mon.b at v1:172.21.15.192:6789/0 2024-06-03T09:06:15.111 INFO:teuthology.orchestra.run.smithi088.stderr:2024-06-03T09:06:15.109+0000 7feeed5e2640 1 -- 172.21.15.88:0/746948843 --> v1:172.21.15.192:6789/0 -- mon_subscribe({mgrmap=0+}) v3 -- 0x7feee81dab00 con 0x7feee806fc70 2024-06-03T09:06:15.111 INFO:teuthology.orchestra.run.smithi088.stderr:2024-06-03T09:06:15.109+0000 7feeed5e2640 10 client.30136.objecter _maybe_request_map subscribing (onetime) to next osd map 2024-06-03T09:06:15.111 INFO:teuthology.orchestra.run.smithi088.stderr:2024-06-03T09:06:15.109+0000 7feeed5e2640 10 monclient: _renew_subs 2024-06-03T09:06:15.112 INFO:teuthology.orchestra.run.smithi088.stderr:2024-06-03T09:06:15.109+0000 7feeed5e2640 10 monclient: _send_mon_message to mon.b at v1:172.21.15.192:6789/0 2024-06-03T09:06:15.112 INFO:teuthology.orchestra.run.smithi088.stderr:2024-06-03T09:06:15.109+0000 7feeed5e2640 1 -- 172.21.15.88:0/746948843 --> v1:172.21.15.192:6789/0 -- mon_subscribe({osdmap=0}) v3 -- 0x7feee81e2660 con 0x7feee806fc70 2024-06-03T09:06:15.112 INFO:teuthology.orchestra.run.smithi088.stderr:2024-06-03T09:06:15.109+0000 7feeed5e2640 1 librados: init done 2024-06-03T09:06:15.112 INFO:teuthology.orchestra.run.smithi088.stderr:2024-06-03T09:06:15.110+0000 7feeed5e2640 10 start_mon_command cmd=[{"prefix": "get_command_descriptions"}] 2024-06-03T09:06:15.112 INFO:teuthology.orchestra.run.smithi088.stderr:2024-06-03T09:06:15.110+0000 7feeed5e2640 10 monclient: _send_command 1 [{"prefix": "get_command_descriptions"}] 2024-06-03T09:06:15.112 INFO:teuthology.orchestra.run.smithi088.stderr:2024-06-03T09:06:15.110+0000 7feeed5e2640 10 monclient: _send_mon_message to mon.b at v1:172.21.15.192:6789/0 2024-06-03T09:06:15.112 INFO:teuthology.orchestra.run.smithi088.stderr:2024-06-03T09:06:15.110+0000 7feeed5e2640 1 -- 172.21.15.88:0/746948843 --> v1:172.21.15.192:6789/0 -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7feee804ea60 con 0x7feee806fc70 2024-06-03T09:06:15.124 INFO:teuthology.orchestra.run.smithi088.stderr:2024-06-03T09:06:15.125+0000 7feedf7fe640 1 -- 172.21.15.88:0/746948843 <== mon.1 v1:172.21.15.192:6789/0 5 ==== config(0 keys) v1 ==== 4+0+0 (unknown 0 0 0) 0x7feed8001c10 con 0x7feee806fc70 2024-06-03T09:06:15.125 INFO:teuthology.orchestra.run.smithi088.stderr:2024-06-03T09:06:15.125+0000 7feedf7fe640 10 monclient: handle_config config(0 keys) v1 2024-06-03T09:06:15.125 INFO:teuthology.orchestra.run.smithi088.stderr:2024-06-03T09:06:15.125+0000 7feedf7fe640 1 -- 172.21.15.88:0/746948843 <== mon.1 v1:172.21.15.192:6789/0 6 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 2234374065 0 0) 0x7feed8004980 con 0x7feee806fc70 2024-06-03T09:06:15.125 INFO:teuthology.orchestra.run.smithi088.stderr:2024-06-03T09:06:15.125+0000 7feedf7fe640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-06-03T09:06:15.125 INFO:teuthology.orchestra.run.smithi088.stderr:2024-06-03T09:06:15.125+0000 7feedf7fe640 10 monclient: got monmap 1 from mon.b (according to old e1) 2024-06-03T09:06:15.125 INFO:teuthology.orchestra.run.smithi088.stderr:2024-06-03T09:06:15.125+0000 7feedf7fe640 10 monclient: dump: 2024-06-03T09:06:15.125 INFO:teuthology.orchestra.run.smithi088.stderr:epoch 1 2024-06-03T09:06:15.125 INFO:teuthology.orchestra.run.smithi088.stderr:fsid ec3c3fdd-3ac6-4d7d-9412-683130ed8b04 2024-06-03T09:06:15.125 INFO:teuthology.orchestra.run.smithi088.stderr:last_changed 2024-06-03T08:34:31.885125+0000 2024-06-03T09:06:15.125 INFO:teuthology.orchestra.run.smithi088.stderr:created 2024-06-03T08:34:31.885125+0000 2024-06-03T09:06:15.125 INFO:teuthology.orchestra.run.smithi088.stderr:min_mon_release 18 (reef) 2024-06-03T09:06:15.125 INFO:teuthology.orchestra.run.smithi088.stderr:election_strategy: 3 2024-06-03T09:06:15.125 INFO:teuthology.orchestra.run.smithi088.stderr:0: v1:172.21.15.88:6789/0 mon.a 2024-06-03T09:06:15.125 INFO:teuthology.orchestra.run.smithi088.stderr:1: v1:172.21.15.192:6789/0 mon.b 2024-06-03T09:06:15.126 INFO:teuthology.orchestra.run.smithi088.stderr:2: v1:172.21.15.88:6790/0 mon.c 2024-06-03T09:06:15.126 INFO:teuthology.orchestra.run.smithi088.stderr: 2024-06-03T09:06:15.126 INFO:teuthology.orchestra.run.smithi088.stderr:2024-06-03T09:06:15.125+0000 7feee5ffb640 4 set_mon_vals no callback set 2024-06-03T09:06:15.126 INFO:teuthology.orchestra.run.smithi088.stderr:2024-06-03T09:06:15.126+0000 7feedf7fe640 1 -- 172.21.15.88:0/746948843 <== mon.1 v1:172.21.15.192:6789/0 7 ==== mgrmap(e 5) v1 ==== 89110+0+0 (unknown 873533973 0 0) 0x7feed801a700 con 0x7feee806fc70 2024-06-03T09:06:15.126 INFO:teuthology.orchestra.run.smithi088.stderr:2024-06-03T09:06:15.126+0000 7feedffff640 10 monclient: get_auth_request con 0x7feec806d3c0 auth_method 0 2024-06-03T09:06:15.126 INFO:teuthology.orchestra.run.smithi088.stderr:2024-06-03T09:06:15.126+0000 7feedf7fe640 10 client.30136.objecter ms_handle_connect 0x7feec806d3c0 2024-06-03T09:06:15.127 INFO:teuthology.orchestra.run.smithi088.stderr:2024-06-03T09:06:15.127+0000 7feedf7fe640 1 -- 172.21.15.88:0/746948843 <== mon.1 v1:172.21.15.192:6789/0 8 ==== osd_map(1009..1009 src has 622..1009) v4 ==== 5229+0+0 (unknown 553880507 0 0) 0x7feed8053eb0 con 0x7feee806fc70 2024-06-03T09:06:15.127 INFO:teuthology.orchestra.run.smithi088.stderr:2024-06-03T09:06:15.127+0000 7feedf7fe640 10 client.30136.objecter ms_dispatch 0x7feee8000b90 osd_map(1009..1009 src has 622..1009) v4 2024-06-03T09:06:15.127 INFO:teuthology.orchestra.run.smithi088.stderr:2024-06-03T09:06:15.127+0000 7feedf7fe640 3 client.30136.objecter handle_osd_map got epochs [1009,1009] > 0 2024-06-03T09:06:15.127 INFO:teuthology.orchestra.run.smithi088.stderr:2024-06-03T09:06:15.127+0000 7feedf7fe640 3 client.30136.objecter handle_osd_map decoding full epoch 1009 2024-06-03T09:06:15.127 INFO:teuthology.orchestra.run.smithi088.stderr:2024-06-03T09:06:15.127+0000 7feedf7fe640 20 client.30136.objecter dump_active .. 0 homeless 2024-06-03T09:06:15.364 INFO:teuthology.orchestra.run.smithi088.stderr:2024-06-03T09:06:15.365+0000 7feedf7fe640 1 -- 172.21.15.88:0/746948843 <== mon.1 v1:172.21.15.192:6789/0 9 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+182498 (unknown 1092875540 0 3016760700) 0x7feed804f890 con 0x7feee806fc70 2024-06-03T09:06:15.365 INFO:teuthology.orchestra.run.smithi088.stderr:2024-06-03T09:06:15.365+0000 7feedf7fe640 10 monclient: handle_mon_command_ack 1 [{"prefix": "get_command_descriptions"}] 2024-06-03T09:06:15.365 INFO:teuthology.orchestra.run.smithi088.stderr:2024-06-03T09:06:15.365+0000 7feedf7fe640 10 monclient: _finish_command 1 = system:0 2024-06-03T09:06:15.532 INFO:teuthology.orchestra.run.smithi088.stderr:2024-06-03T09:06:15.532+0000 7feeed5e2640 1 -- 172.21.15.88:0/746948843 --> v1:172.21.15.192:6800/34229 -- mgr_command(tid 0: {"prefix": "pg dump", "target": ["mon-mgr", ""], "format": "json"}) v1 -- 0x7feee813ce70 con 0x7feec806d3c0 2024-06-03T09:06:15.537 INFO:teuthology.orchestra.run.smithi088.stderr:2024-06-03T09:06:15.537+0000 7feedf7fe640 1 -- 172.21.15.88:0/746948843 <== mgr.4110 v1:172.21.15.192:6800/34229 1 ==== mgr_command_reply(tid 0: 0 dumped all) v1 ==== 18+0+151820 (unknown 2965378022 0 2920618137) 0x7feed00077e0 con 0x7feec806d3c0 2024-06-03T09:06:15.539 INFO:teuthology.orchestra.run.smithi088.stdout: 2024-06-03T09:06:15.539 INFO:teuthology.orchestra.run.smithi088.stderr:dumped all 2024-06-03T09:06:15.541 INFO:teuthology.orchestra.run.smithi088.stderr:2024-06-03T09:06:15.539+0000 7feeed5e2640 10 librados: watch_flush enter 2024-06-03T09:06:15.541 INFO:teuthology.orchestra.run.smithi088.stderr:2024-06-03T09:06:15.540+0000 7feeed5e2640 10 librados: watch_flush exit 2024-06-03T09:06:15.541 INFO:teuthology.orchestra.run.smithi088.stderr:2024-06-03T09:06:15.540+0000 7feeed5e2640 20 client.30136.objecter shutdown clearing up homeless session... 2024-06-03T09:06:15.541 INFO:teuthology.orchestra.run.smithi088.stderr:2024-06-03T09:06:15.540+0000 7feeed5e2640 10 client.30136.objecter successfully canceled tick 2024-06-03T09:06:15.542 INFO:teuthology.orchestra.run.smithi088.stderr:2024-06-03T09:06:15.540+0000 7feeed5e2640 1 -- 172.21.15.88:0/746948843 >> v1:172.21.15.192:6800/34229 conn(0x7feec806d3c0 legacy=0x7feec806f880 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-06-03T09:06:15.542 INFO:teuthology.orchestra.run.smithi088.stderr:2024-06-03T09:06:15.540+0000 7feeed5e2640 10 monclient: shutdown 2024-06-03T09:06:15.542 INFO:teuthology.orchestra.run.smithi088.stderr:2024-06-03T09:06:15.540+0000 7feeed5e2640 20 monclient: shutdown discarding 0 pending message(s) 2024-06-03T09:06:15.542 INFO:teuthology.orchestra.run.smithi088.stderr:2024-06-03T09:06:15.540+0000 7feeed5e2640 1 -- 172.21.15.88:0/746948843 >> v1:172.21.15.192:6789/0 conn(0x7feee806fc70 legacy=0x7feee81da1a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-06-03T09:06:15.542 INFO:teuthology.orchestra.run.smithi088.stderr:2024-06-03T09:06:15.540+0000 7feeed5e2640 1 -- 172.21.15.88:0/746948843 shutdown_connections 2024-06-03T09:06:15.542 INFO:teuthology.orchestra.run.smithi088.stderr:2024-06-03T09:06:15.540+0000 7feeed5e2640 1 -- 172.21.15.88:0/746948843 >> 172.21.15.88:0/746948843 conn(0x7feee80b2f20 msgr2=0x7feee813dad0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-06-03T09:06:15.542 INFO:teuthology.orchestra.run.smithi088.stderr:2024-06-03T09:06:15.540+0000 7feeed5e2640 1 -- 172.21.15.88:0/746948843 shutdown_connections 2024-06-03T09:06:15.542 INFO:teuthology.orchestra.run.smithi088.stderr:2024-06-03T09:06:15.540+0000 7feeed5e2640 1 -- 172.21.15.88:0/746948843 wait complete. 2024-06-03T09:06:15.542 INFO:teuthology.orchestra.run.smithi088.stderr:2024-06-03T09:06:15.540+0000 7feeed5e2640 1 librados: shutdown 2024-06-03T09:06:15.551 INFO:teuthology.orchestra.run.smithi088.stdout:{"pg_ready":true,"pg_map":{"version":1912,"stamp":"2024-06-03T09:06:15.097444+0000","last_osdmap_epoch":0,"last_pg_scan":0,"pg_stats_sum":{"stat_sum":{"num_bytes":459285,"num_objects":3,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":1,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":3,"num_whiteouts":0,"num_read":102,"num_read_kb":88,"num_write":120,"num_write_kb":1379,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2,"num_bytes_recovered":397840,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":83,"ondisk_log_size":83,"up":68,"acting":68,"num_store_stats":0},"osd_stats_sum":{"up_from":0,"seq":0,"num_pgs":126,"num_osds":8,"num_per_pool_osds":8,"num_per_pool_omap_osds":8,"kb":754974720,"kb_used":494376,"kb_used_data":46140,"kb_used_omap":12,"kb_used_meta":448115,"kb_avail":754480344,"statfs":{"total":773094113280,"available":772587872256,"internally_reserved":0,"allocated":47247360,"data_stored":22695791,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":12744,"internal_metadata":458870328},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":433,"apply_latency_ms":433,"commit_latency_ns":433000000,"apply_latency_ns":433000000},"alerts":[],"network_ping_times":[]},"pg_stats_delta":{"stat_sum":{"num_bytes":-4000299,"num_objects":-44,"num_object_clones":-1,"num_object_copies":-132,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":-32,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":-44,"num_whiteouts":0,"num_read":-31,"num_read_kb":-11,"num_write":-71,"num_write_kb":-3969,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":-1,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":0,"acting":0,"num_store_stats":0,"stamp_delta":"4.897601"},"pg_stats":[{"pgid":"17.d","version":"0'0","reported_seq":940,"reported_epoch":1007,"state":"active+clean","last_fresh":"2024-06-03T09:06:11.402997+0000","last_change":"2024-06-03T08:38:19.260548+0000","last_active":"2024-06-03T09:06:11.402997+0000","last_peered":"2024-06-03T09:06:11.402997+0000","last_clean":"2024-06-03T09:06:11.402997+0000","last_became_active":"2024-06-03T08:38:19.253559+0000","last_became_peered":"2024-06-03T08:38:19.253559+0000","last_unstale":"2024-06-03T09:06:11.402997+0000","last_undegraded":"2024-06-03T09:06:11.402997+0000","last_fullsized":"2024-06-03T09:06:11.402997+0000","mapping_epoch":28,"log_start":"0'0","ondisk_log_start":"0'0","created":28,"last_epoch_clean":45,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-06-03T08:37:55.614497+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-06-03T08:37:55.614497+0000","last_clean_scrub_stamp":"2024-06-03T08:37:55.614497+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-06-04T19:00:12.470979+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1],"acting":[0,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"17.c","version":"0'0","reported_seq":952,"reported_epoch":1008,"state":"active+undersized","last_fresh":"2024-06-03T09:06:12.368416+0000","last_change":"2024-06-03T09:06:03.440074+0000","last_active":"2024-06-03T09:06:12.368416+0000","last_peered":"2024-06-03T09:06:12.368416+0000","last_clean":"2024-06-03T09:05:02.105638+0000","last_became_active":"2024-06-03T09:06:03.440074+0000","last_became_peered":"2024-06-03T09:06:03.440074+0000","last_unstale":"2024-06-03T09:06:12.368416+0000","last_undegraded":"2024-06-03T09:06:12.368416+0000","last_fullsized":"2024-06-03T09:06:03.377982+0000","mapping_epoch":999,"log_start":"0'0","ondisk_log_start":"0'0","created":28,"last_epoch_clean":255,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-06-03T08:37:55.614497+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-06-03T08:37:55.614497+0000","last_clean_scrub_stamp":"2024-06-03T08:37:55.614497+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-06-04T10:34:20.808557+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3],"acting":[3],"avail_no_missing":["3"],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"17.f","version":"0'0","reported_seq":144,"reported_epoch":1004,"state":"active+undersized","last_fresh":"2024-06-03T09:06:08.016575+0000","last_change":"2024-06-03T09:06:03.420285+0000","last_active":"2024-06-03T09:06:08.016575+0000","last_peered":"2024-06-03T09:06:08.016575+0000","last_clean":"2024-06-03T08:37:55.614497+0000","last_became_active":"2024-06-03T09:06:03.420285+0000","last_became_peered":"2024-06-03T09:06:03.420285+0000","last_unstale":"2024-06-03T09:06:08.016575+0000","last_undegraded":"2024-06-03T09:06:08.016575+0000","last_fullsized":"2024-06-03T09:06:03.380616+0000","mapping_epoch":999,"log_start":"0'0","ondisk_log_start":"0'0","created":28,"last_epoch_clean":255,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-06-03T08:37:55.614497+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-06-03T08:37:55.614497+0000","last_clean_scrub_stamp":"2024-06-03T08:37:55.614497+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-06-04T15:39:41.187731+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4],"acting":[4],"avail_no_missing":["4"],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"17.e","version":"0'0","reported_seq":148,"reported_epoch":1008,"state":"active+undersized","last_fresh":"2024-06-03T09:06:12.895353+0000","last_change":"2024-06-03T09:06:03.412887+0000","last_active":"2024-06-03T09:06:12.895353+0000","last_peered":"2024-06-03T09:06:12.895353+0000","last_clean":"2024-06-03T08:37:55.614497+0000","last_became_active":"2024-06-03T09:06:03.412887+0000","last_became_peered":"2024-06-03T09:06:03.412887+0000","last_unstale":"2024-06-03T09:06:12.895353+0000","last_undegraded":"2024-06-03T09:06:12.895353+0000","last_fullsized":"2024-06-03T09:06:03.332482+0000","mapping_epoch":999,"log_start":"0'0","ondisk_log_start":"0'0","created":28,"last_epoch_clean":255,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-06-03T08:37:55.614497+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-06-03T08:37:55.614497+0000","last_clean_scrub_stamp":"2024-06-03T08:37:55.614497+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-06-04T12:23:22.511601+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5],"acting":[5],"avail_no_missing":["5"],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"17.9","version":"0'0","reported_seq":945,"reported_epoch":1009,"state":"active+clean","last_fresh":"2024-06-03T09:06:13.928215+0000","last_change":"2024-06-03T08:38:04.772260+0000","last_active":"2024-06-03T09:06:13.928215+0000","last_peered":"2024-06-03T09:06:13.928215+0000","last_clean":"2024-06-03T09:06:13.928215+0000","last_became_active":"2024-06-03T08:38:04.763428+0000","last_became_peered":"2024-06-03T08:38:04.763428+0000","last_unstale":"2024-06-03T09:06:13.928215+0000","last_undegraded":"2024-06-03T09:06:13.928215+0000","last_fullsized":"2024-06-03T09:06:13.928215+0000","mapping_epoch":28,"log_start":"0'0","ondisk_log_start":"0'0","created":28,"last_epoch_clean":33,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-06-03T08:37:55.614497+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-06-03T08:37:55.614497+0000","last_clean_scrub_stamp":"2024-06-03T08:37:55.614497+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-06-04T10:55:10.773936+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,4],"acting":[7,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"17.8","version":"0'0","reported_seq":959,"reported_epoch":1007,"state":"active+undersized","last_fresh":"2024-06-03T09:06:12.708781+0000","last_change":"2024-06-03T09:06:03.388786+0000","last_active":"2024-06-03T09:06:12.708781+0000","last_peered":"2024-06-03T09:06:12.708781+0000","last_clean":"2024-06-03T09:04:43.297780+0000","last_became_active":"2024-06-03T09:06:03.388786+0000","last_became_peered":"2024-06-03T09:06:03.388786+0000","last_unstale":"2024-06-03T09:06:12.708781+0000","last_undegraded":"2024-06-03T09:06:12.708781+0000","last_fullsized":"2024-06-03T09:06:03.331754+0000","mapping_epoch":999,"log_start":"0'0","ondisk_log_start":"0'0","created":28,"last_epoch_clean":255,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-06-03T08:37:55.614497+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-06-03T08:37:55.614497+0000","last_clean_scrub_stamp":"2024-06-03T08:37:55.614497+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-06-04T09:18:44.765547+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0],"acting":[0],"avail_no_missing":["0"],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"17.b","version":"0'0","reported_seq":145,"reported_epoch":1005,"state":"active+undersized","last_fresh":"2024-06-03T09:06:09.149286+0000","last_change":"2024-06-03T09:06:03.451203+0000","last_active":"2024-06-03T09:06:09.149286+0000","last_peered":"2024-06-03T09:06:09.149286+0000","last_clean":"2024-06-03T08:37:55.614497+0000","last_became_active":"2024-06-03T09:06:03.451203+0000","last_became_peered":"2024-06-03T09:06:03.451203+0000","last_unstale":"2024-06-03T09:06:09.149286+0000","last_undegraded":"2024-06-03T09:06:09.149286+0000","last_fullsized":"2024-06-03T09:06:03.375523+0000","mapping_epoch":999,"log_start":"0'0","ondisk_log_start":"0'0","created":28,"last_epoch_clean":255,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-06-03T08:37:55.614497+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-06-03T08:37:55.614497+0000","last_clean_scrub_stamp":"2024-06-03T08:37:55.614497+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-06-04T10:16:19.813495+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2],"acting":[2],"avail_no_missing":["2"],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"17.a","version":"0'0","reported_seq":923,"reported_epoch":1008,"state":"active+clean","last_fresh":"2024-06-03T09:06:12.871052+0000","last_change":"2024-06-03T08:38:05.843142+0000","last_active":"2024-06-03T09:06:12.871052+0000","last_peered":"2024-06-03T09:06:12.871052+0000","last_clean":"2024-06-03T09:06:12.871052+0000","last_became_active":"2024-06-03T08:38:05.809257+0000","last_became_peered":"2024-06-03T08:38:05.809257+0000","last_unstale":"2024-06-03T09:06:12.871052+0000","last_undegraded":"2024-06-03T09:06:12.871052+0000","last_fullsized":"2024-06-03T09:06:12.871052+0000","mapping_epoch":28,"log_start":"0'0","ondisk_log_start":"0'0","created":28,"last_epoch_clean":33,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-06-03T08:37:55.614497+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-06-03T08:37:55.614497+0000","last_clean_scrub_stamp":"2024-06-03T08:37:55.614497+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-06-04T12:12:03.686838+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2],"acting":[1,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"17.5","version":"0'0","reported_seq":941,"reported_epoch":1007,"state":"active+clean","last_fresh":"2024-06-03T09:06:11.400979+0000","last_change":"2024-06-03T08:38:23.466051+0000","last_active":"2024-06-03T09:06:11.400979+0000","last_peered":"2024-06-03T09:06:11.400979+0000","last_clean":"2024-06-03T09:06:11.400979+0000","last_became_active":"2024-06-03T08:38:23.458039+0000","last_became_peered":"2024-06-03T08:38:23.458039+0000","last_unstale":"2024-06-03T09:06:11.400979+0000","last_undegraded":"2024-06-03T09:06:11.400979+0000","last_fullsized":"2024-06-03T09:06:11.400979+0000","mapping_epoch":28,"log_start":"0'0","ondisk_log_start":"0'0","created":28,"last_epoch_clean":49,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-06-03T08:37:55.614497+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-06-03T08:37:55.614497+0000","last_clean_scrub_stamp":"2024-06-03T08:37:55.614497+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-06-04T12:27:02.532436+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,4],"acting":[0,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"17.4","version":"0'0","reported_seq":148,"reported_epoch":1008,"state":"active+undersized","last_fresh":"2024-06-03T09:06:12.883827+0000","last_change":"2024-06-03T09:06:03.441889+0000","last_active":"2024-06-03T09:06:12.883827+0000","last_peered":"2024-06-03T09:06:12.883827+0000","last_clean":"2024-06-03T08:37:55.614497+0000","last_became_active":"2024-06-03T09:06:03.441889+0000","last_became_peered":"2024-06-03T09:06:03.441889+0000","last_unstale":"2024-06-03T09:06:12.883827+0000","last_undegraded":"2024-06-03T09:06:12.883827+0000","last_fullsized":"2024-06-03T09:06:03.359582+0000","mapping_epoch":999,"log_start":"0'0","ondisk_log_start":"0'0","created":28,"last_epoch_clean":255,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-06-03T08:37:55.614497+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-06-03T08:37:55.614497+0000","last_clean_scrub_stamp":"2024-06-03T08:37:55.614497+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-06-04T12:49:22.907141+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7],"acting":[7],"avail_no_missing":["7"],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"17.7","version":"0'0","reported_seq":928,"reported_epoch":1008,"state":"active+clean","last_fresh":"2024-06-03T09:06:12.906543+0000","last_change":"2024-06-03T08:38:04.837083+0000","last_active":"2024-06-03T09:06:12.906543+0000","last_peered":"2024-06-03T09:06:12.906543+0000","last_clean":"2024-06-03T09:06:12.906543+0000","last_became_active":"2024-06-03T08:38:04.818246+0000","last_became_peered":"2024-06-03T08:38:04.818246+0000","last_unstale":"2024-06-03T09:06:12.906543+0000","last_undegraded":"2024-06-03T09:06:12.906543+0000","last_fullsized":"2024-06-03T09:06:12.906543+0000","mapping_epoch":28,"log_start":"0'0","ondisk_log_start":"0'0","created":28,"last_epoch_clean":34,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-06-03T08:37:55.614497+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-06-03T08:37:55.614497+0000","last_clean_scrub_stamp":"2024-06-03T08:37:55.614497+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-06-04T17:02:38.204813+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,2],"acting":[5,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"17.6","version":"0'0","reported_seq":958,"reported_epoch":1007,"state":"active+undersized","last_fresh":"2024-06-03T09:06:12.710313+0000","last_change":"2024-06-03T09:06:03.428677+0000","last_active":"2024-06-03T09:06:12.710313+0000","last_peered":"2024-06-03T09:06:12.710313+0000","last_clean":"2024-06-03T09:04:43.312352+0000","last_became_active":"2024-06-03T09:06:03.428677+0000","last_became_peered":"2024-06-03T09:06:03.428677+0000","last_unstale":"2024-06-03T09:06:12.710313+0000","last_undegraded":"2024-06-03T09:06:12.710313+0000","last_fullsized":"2024-06-03T09:06:03.359909+0000","mapping_epoch":999,"log_start":"0'0","ondisk_log_start":"0'0","created":28,"last_epoch_clean":255,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-06-03T08:37:55.614497+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-06-03T08:37:55.614497+0000","last_clean_scrub_stamp":"2024-06-03T08:37:55.614497+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-06-04T15:38:02.180241+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0],"acting":[0],"avail_no_missing":["0"],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"2.7","version":"0'0","reported_seq":939,"reported_epoch":1008,"state":"active+clean","last_fresh":"2024-06-03T09:06:12.364540+0000","last_change":"2024-06-03T08:36:30.385405+0000","last_active":"2024-06-03T09:06:12.364540+0000","last_peered":"2024-06-03T09:06:12.364540+0000","last_clean":"2024-06-03T09:06:12.364540+0000","last_became_active":"2024-06-03T08:36:30.380302+0000","last_became_peered":"2024-06-03T08:36:30.380302+0000","last_unstale":"2024-06-03T09:06:12.364540+0000","last_undegraded":"2024-06-03T09:06:12.364540+0000","last_fullsized":"2024-06-03T09:06:12.364540+0000","mapping_epoch":23,"log_start":"0'0","ondisk_log_start":"0'0","created":23,"last_epoch_clean":24,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-06-03T08:36:28.444764+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-06-03T08:36:28.444764+0000","last_clean_scrub_stamp":"2024-06-03T08:36:28.444764+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-06-04T18:46:33.475181+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0],"acting":[3,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"17.14","version":"0'0","reported_seq":925,"reported_epoch":1008,"state":"active+clean","last_fresh":"2024-06-03T09:06:12.884449+0000","last_change":"2024-06-03T08:38:04.547565+0000","last_active":"2024-06-03T09:06:12.884449+0000","last_peered":"2024-06-03T09:06:12.884449+0000","last_clean":"2024-06-03T09:06:12.884449+0000","last_became_active":"2024-06-03T08:38:04.541781+0000","last_became_peered":"2024-06-03T08:38:04.541781+0000","last_unstale":"2024-06-03T09:06:12.884449+0000","last_undegraded":"2024-06-03T09:06:12.884449+0000","last_fullsized":"2024-06-03T09:06:12.884449+0000","mapping_epoch":28,"log_start":"0'0","ondisk_log_start":"0'0","created":28,"last_epoch_clean":34,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-06-03T08:37:55.614497+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-06-03T08:37:55.614497+0000","last_clean_scrub_stamp":"2024-06-03T08:37:55.614497+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-06-04T16:41:55.170134+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,2],"acting":[5,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"2.6","version":"0'0","reported_seq":945,"reported_epoch":1004,"state":"active+clean","last_fresh":"2024-06-03T09:06:07.984455+0000","last_change":"2024-06-03T08:36:30.430760+0000","last_active":"2024-06-03T09:06:07.984455+0000","last_peered":"2024-06-03T09:06:07.984455+0000","last_clean":"2024-06-03T09:06:07.984455+0000","last_became_active":"2024-06-03T08:36:30.404665+0000","last_became_peered":"2024-06-03T08:36:30.404665+0000","last_unstale":"2024-06-03T09:06:07.984455+0000","last_undegraded":"2024-06-03T09:06:07.984455+0000","last_fullsized":"2024-06-03T09:06:07.984455+0000","mapping_epoch":23,"log_start":"0'0","ondisk_log_start":"0'0","created":23,"last_epoch_clean":24,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-06-03T08:36:28.444764+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-06-03T08:36:28.444764+0000","last_clean_scrub_stamp":"2024-06-03T08:36:28.444764+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-06-04T10:37:37.746074+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,0],"acting":[4,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"17.15","version":"0'0","reported_seq":933,"reported_epoch":1008,"state":"active+clean","last_fresh":"2024-06-03T09:06:12.371160+0000","last_change":"2024-06-03T08:38:20.999903+0000","last_active":"2024-06-03T09:06:12.371160+0000","last_peered":"2024-06-03T09:06:12.371160+0000","last_clean":"2024-06-03T09:06:12.371160+0000","last_became_active":"2024-06-03T08:38:20.979523+0000","last_became_peered":"2024-06-03T08:38:20.979523+0000","last_unstale":"2024-06-03T09:06:12.371160+0000","last_undegraded":"2024-06-03T09:06:12.371160+0000","last_fullsized":"2024-06-03T09:06:12.371160+0000","mapping_epoch":28,"log_start":"0'0","ondisk_log_start":"0'0","created":28,"last_epoch_clean":47,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-06-03T08:37:55.614497+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-06-03T08:37:55.614497+0000","last_clean_scrub_stamp":"2024-06-03T08:37:55.614497+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-06-04T20:29:58.314289+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,1],"acting":[3,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"2.5","version":"0'0","reported_seq":940,"reported_epoch":1008,"state":"active+clean","last_fresh":"2024-06-03T09:06:12.372623+0000","last_change":"2024-06-03T08:36:30.024235+0000","last_active":"2024-06-03T09:06:12.372623+0000","last_peered":"2024-06-03T09:06:12.372623+0000","last_clean":"2024-06-03T09:06:12.372623+0000","last_became_active":"2024-06-03T08:36:30.019040+0000","last_became_peered":"2024-06-03T08:36:30.019040+0000","last_unstale":"2024-06-03T09:06:12.372623+0000","last_undegraded":"2024-06-03T09:06:12.372623+0000","last_fullsized":"2024-06-03T09:06:12.372623+0000","mapping_epoch":23,"log_start":"0'0","ondisk_log_start":"0'0","created":23,"last_epoch_clean":24,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-06-03T08:36:28.444764+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-06-03T08:36:28.444764+0000","last_clean_scrub_stamp":"2024-06-03T08:36:28.444764+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-06-04T15:46:47.109502+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,4],"acting":[3,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"17.16","version":"0'0","reported_seq":968,"reported_epoch":1005,"state":"active+undersized","last_fresh":"2024-06-03T09:06:09.191972+0000","last_change":"2024-06-03T09:06:03.478615+0000","last_active":"2024-06-03T09:06:09.191972+0000","last_peered":"2024-06-03T09:06:09.191972+0000","last_clean":"2024-06-03T09:04:43.328415+0000","last_became_active":"2024-06-03T09:06:03.478615+0000","last_became_peered":"2024-06-03T09:06:03.478615+0000","last_unstale":"2024-06-03T09:06:09.191972+0000","last_undegraded":"2024-06-03T09:06:09.191972+0000","last_fullsized":"2024-06-03T09:06:03.402504+0000","mapping_epoch":999,"log_start":"0'0","ondisk_log_start":"0'0","created":28,"last_epoch_clean":255,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-06-03T08:37:55.614497+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-06-03T08:37:55.614497+0000","last_clean_scrub_stamp":"2024-06-03T08:37:55.614497+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-06-04T08:47:08.079233+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2],"acting":[2],"avail_no_missing":["2"],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"2.4","version":"0'0","reported_seq":950,"reported_epoch":1008,"state":"active+undersized","last_fresh":"2024-06-03T09:06:12.887507+0000","last_change":"2024-06-03T09:06:03.387171+0000","last_active":"2024-06-03T09:06:12.887507+0000","last_peered":"2024-06-03T09:06:12.887507+0000","last_clean":"2024-06-03T09:04:43.202645+0000","last_became_active":"2024-06-03T09:06:03.387171+0000","last_became_peered":"2024-06-03T09:06:03.387171+0000","last_unstale":"2024-06-03T09:06:12.887507+0000","last_undegraded":"2024-06-03T09:06:12.887507+0000","last_fullsized":"2024-06-03T09:06:03.283787+0000","mapping_epoch":999,"log_start":"0'0","ondisk_log_start":"0'0","created":23,"last_epoch_clean":255,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-06-03T08:36:28.444764+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-06-03T08:36:28.444764+0000","last_clean_scrub_stamp":"2024-06-03T08:36:28.444764+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-06-04T20:17:43.485023+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5],"acting":[5],"avail_no_missing":["5"],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"17.17","version":"0'0","reported_seq":950,"reported_epoch":1005,"state":"active+clean","last_fresh":"2024-06-03T09:06:09.179648+0000","last_change":"2024-06-03T08:38:38.483275+0000","last_active":"2024-06-03T09:06:09.179648+0000","last_peered":"2024-06-03T09:06:09.179648+0000","last_clean":"2024-06-03T09:06:09.179648+0000","last_became_active":"2024-06-03T08:38:38.467916+0000","last_became_peered":"2024-06-03T08:38:38.467916+0000","last_unstale":"2024-06-03T09:06:09.179648+0000","last_undegraded":"2024-06-03T09:06:09.179648+0000","last_fullsized":"2024-06-03T09:06:09.179648+0000","mapping_epoch":28,"log_start":"0'0","ondisk_log_start":"0'0","created":28,"last_epoch_clean":61,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-06-03T08:37:55.614497+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-06-03T08:37:55.614497+0000","last_clean_scrub_stamp":"2024-06-03T08:37:55.614497+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-06-04T10:02:23.036420+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,7],"acting":[2,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"2.2","version":"0'0","reported_seq":962,"reported_epoch":1008,"state":"active+undersized","last_fresh":"2024-06-03T09:06:12.352260+0000","last_change":"2024-06-03T09:06:03.410020+0000","last_active":"2024-06-03T09:06:12.352260+0000","last_peered":"2024-06-03T09:06:12.352260+0000","last_clean":"2024-06-03T09:05:37.145889+0000","last_became_active":"2024-06-03T09:06:03.410020+0000","last_became_peered":"2024-06-03T09:06:03.410020+0000","last_unstale":"2024-06-03T09:06:12.352260+0000","last_undegraded":"2024-06-03T09:06:12.352260+0000","last_fullsized":"2024-06-03T09:06:03.346864+0000","mapping_epoch":999,"log_start":"0'0","ondisk_log_start":"0'0","created":23,"last_epoch_clean":255,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-06-03T08:36:28.444764+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-06-03T08:36:28.444764+0000","last_clean_scrub_stamp":"2024-06-03T08:36:28.444764+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-06-04T09:57:12.423360+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3],"acting":[3],"avail_no_missing":["3"],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"17.11","version":"0'0","reported_seq":940,"reported_epoch":1004,"state":"active+clean","last_fresh":"2024-06-03T09:06:07.943878+0000","last_change":"2024-06-03T08:38:06.502302+0000","last_active":"2024-06-03T09:06:07.943878+0000","last_peered":"2024-06-03T09:06:07.943878+0000","last_clean":"2024-06-03T09:06:07.943878+0000","last_became_active":"2024-06-03T08:38:06.490204+0000","last_became_peered":"2024-06-03T08:38:06.490204+0000","last_unstale":"2024-06-03T09:06:07.943878+0000","last_undegraded":"2024-06-03T09:06:07.943878+0000","last_fullsized":"2024-06-03T09:06:07.943878+0000","mapping_epoch":28,"log_start":"0'0","ondisk_log_start":"0'0","created":28,"last_epoch_clean":32,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-06-03T08:37:55.614497+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-06-03T08:37:55.614497+0000","last_clean_scrub_stamp":"2024-06-03T08:37:55.614497+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-06-04T10:47:05.988958+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,2],"acting":[4,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"2.1","version":"0'0","reported_seq":951,"reported_epoch":1008,"state":"active+clean","last_fresh":"2024-06-03T09:06:12.872310+0000","last_change":"2024-06-03T08:36:30.883360+0000","last_active":"2024-06-03T09:06:12.872310+0000","last_peered":"2024-06-03T09:06:12.872310+0000","last_clean":"2024-06-03T09:06:12.872310+0000","last_became_active":"2024-06-03T08:36:30.855073+0000","last_became_peered":"2024-06-03T08:36:30.855073+0000","last_unstale":"2024-06-03T09:06:12.872310+0000","last_undegraded":"2024-06-03T09:06:12.872310+0000","last_fullsized":"2024-06-03T09:06:12.872310+0000","mapping_epoch":23,"log_start":"0'0","ondisk_log_start":"0'0","created":23,"last_epoch_clean":25,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-06-03T08:36:28.444764+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-06-03T08:36:28.444764+0000","last_clean_scrub_stamp":"2024-06-03T08:36:28.444764+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-06-04T18:12:22.363473+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,5],"acting":[7,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"17.12","version":"0'0","reported_seq":944,"reported_epoch":1008,"state":"active+clean","last_fresh":"2024-06-03T09:06:12.878527+0000","last_change":"2024-06-03T08:38:01.751562+0000","last_active":"2024-06-03T09:06:12.878527+0000","last_peered":"2024-06-03T09:06:12.878527+0000","last_clean":"2024-06-03T09:06:12.878527+0000","last_became_active":"2024-06-03T08:38:01.745027+0000","last_became_peered":"2024-06-03T08:38:01.745027+0000","last_unstale":"2024-06-03T09:06:12.878527+0000","last_undegraded":"2024-06-03T09:06:12.878527+0000","last_fullsized":"2024-06-03T09:06:12.878527+0000","mapping_epoch":28,"log_start":"0'0","ondisk_log_start":"0'0","created":28,"last_epoch_clean":32,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-06-03T08:37:55.614497+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-06-03T08:37:55.614497+0000","last_clean_scrub_stamp":"2024-06-03T08:37:55.614497+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-06-04T17:54:43.785326+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,3],"acting":[7,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"2.0","version":"0'0","reported_seq":942,"reported_epoch":1008,"state":"active+clean","last_fresh":"2024-06-03T09:06:12.341224+0000","last_change":"2024-06-03T08:36:29.984096+0000","last_active":"2024-06-03T09:06:12.341224+0000","last_peered":"2024-06-03T09:06:12.341224+0000","last_clean":"2024-06-03T09:06:12.341224+0000","last_became_active":"2024-06-03T08:36:29.957875+0000","last_became_peered":"2024-06-03T08:36:29.957875+0000","last_unstale":"2024-06-03T09:06:12.341224+0000","last_undegraded":"2024-06-03T09:06:12.341224+0000","last_fullsized":"2024-06-03T09:06:12.341224+0000","mapping_epoch":23,"log_start":"0'0","ondisk_log_start":"0'0","created":23,"last_epoch_clean":24,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-06-03T08:36:28.444764+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-06-03T08:36:28.444764+0000","last_clean_scrub_stamp":"2024-06-03T08:36:28.444764+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-06-04T08:41:33.679479+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,5],"acting":[3,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"17.13","version":"0'0","reported_seq":939,"reported_epoch":1005,"state":"active+clean","last_fresh":"2024-06-03T09:06:09.209721+0000","last_change":"2024-06-03T08:38:26.838316+0000","last_active":"2024-06-03T09:06:09.209721+0000","last_peered":"2024-06-03T09:06:09.209721+0000","last_clean":"2024-06-03T09:06:09.209721+0000","last_became_active":"2024-06-03T08:38:26.833214+0000","last_became_peered":"2024-06-03T08:38:26.833214+0000","last_unstale":"2024-06-03T09:06:09.209721+0000","last_undegraded":"2024-06-03T09:06:09.209721+0000","last_fullsized":"2024-06-03T09:06:09.209721+0000","mapping_epoch":28,"log_start":"0'0","ondisk_log_start":"0'0","created":28,"last_epoch_clean":52,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-06-03T08:37:55.614497+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-06-03T08:37:55.614497+0000","last_clean_scrub_stamp":"2024-06-03T08:37:55.614497+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-06-04T12:58:51.617998+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,0],"acting":[4,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"2.3","version":"0'0","reported_seq":941,"reported_epoch":1008,"state":"active+clean","last_fresh":"2024-06-03T09:06:12.356955+0000","last_change":"2024-06-03T08:36:30.351670+0000","last_active":"2024-06-03T09:06:12.356955+0000","last_peered":"2024-06-03T09:06:12.356955+0000","last_clean":"2024-06-03T09:06:12.356955+0000","last_became_active":"2024-06-03T08:36:30.343560+0000","last_became_peered":"2024-06-03T08:36:30.343560+0000","last_unstale":"2024-06-03T09:06:12.356955+0000","last_undegraded":"2024-06-03T09:06:12.356955+0000","last_fullsized":"2024-06-03T09:06:12.356955+0000","mapping_epoch":23,"log_start":"0'0","ondisk_log_start":"0'0","created":23,"last_epoch_clean":24,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-06-03T08:36:28.444764+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-06-03T08:36:28.444764+0000","last_clean_scrub_stamp":"2024-06-03T08:36:28.444764+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-06-04T12:02:24.765135+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,2],"acting":[3,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.0","version":"19'76","reported_seq":1050,"reported_epoch":1008,"state":"active+clean","last_fresh":"2024-06-03T09:06:12.865199+0000","last_change":"2024-06-03T08:36:26.702702+0000","last_active":"2024-06-03T09:06:12.865199+0000","last_peered":"2024-06-03T09:06:12.865199+0000","last_clean":"2024-06-03T09:06:12.865199+0000","last_became_active":"2024-06-03T08:36:26.128378+0000","last_became_peered":"2024-06-03T08:36:26.128378+0000","last_unstale":"2024-06-03T09:06:12.865199+0000","last_undegraded":"2024-06-03T09:06:12.865199+0000","last_fullsized":"2024-06-03T09:06:12.865199+0000","mapping_epoch":20,"log_start":"0'0","ondisk_log_start":"0'0","created":16,"last_epoch_clean":22,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-06-03T08:36:15.542635+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-06-03T08:36:15.542635+0000","last_clean_scrub_stamp":"2024-06-03T08:36:15.542635+0000","objects_scrubbed":0,"log_size":76,"log_dups_size":0,"ondisk_log_size":76,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-06-04T12:41:53.731360+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":459280,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":96,"num_read_kb":82,"num_write":113,"num_write_kb":1372,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2,"num_bytes_recovered":397840,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,5],"acting":[1,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"17.10","version":"0'0","reported_seq":940,"reported_epoch":1004,"state":"active+clean","last_fresh":"2024-06-03T09:06:07.939453+0000","last_change":"2024-06-03T08:38:15.399189+0000","last_active":"2024-06-03T09:06:07.939453+0000","last_peered":"2024-06-03T09:06:07.939453+0000","last_clean":"2024-06-03T09:06:07.939453+0000","last_became_active":"2024-06-03T08:38:15.337723+0000","last_became_peered":"2024-06-03T08:38:15.337723+0000","last_unstale":"2024-06-03T09:06:07.939453+0000","last_undegraded":"2024-06-03T09:06:07.939453+0000","last_fullsized":"2024-06-03T09:06:07.939453+0000","mapping_epoch":28,"log_start":"0'0","ondisk_log_start":"0'0","created":28,"last_epoch_clean":42,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-06-03T08:37:55.614497+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-06-03T08:37:55.614497+0000","last_clean_scrub_stamp":"2024-06-03T08:37:55.614497+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-06-04T09:29:46.427317+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,3],"acting":[4,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"17.1a","version":"0'0","reported_seq":956,"reported_epoch":1004,"state":"active+undersized","last_fresh":"2024-06-03T09:06:07.954840+0000","last_change":"2024-06-03T09:06:03.390734+0000","last_active":"2024-06-03T09:06:07.954840+0000","last_peered":"2024-06-03T09:06:07.954840+0000","last_clean":"2024-06-03T09:05:20.322313+0000","last_became_active":"2024-06-03T09:06:03.390734+0000","last_became_peered":"2024-06-03T09:06:03.390734+0000","last_unstale":"2024-06-03T09:06:07.954840+0000","last_undegraded":"2024-06-03T09:06:07.954840+0000","last_fullsized":"2024-06-03T09:06:03.343197+0000","mapping_epoch":999,"log_start":"0'0","ondisk_log_start":"0'0","created":28,"last_epoch_clean":254,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-06-03T08:37:55.614497+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-06-03T08:37:55.614497+0000","last_clean_scrub_stamp":"2024-06-03T08:37:55.614497+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-06-04T15:56:47.628875+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4],"acting":[4],"avail_no_missing":["4"],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"17.1b","version":"0'0","reported_seq":934,"reported_epoch":1008,"state":"active+clean","last_fresh":"2024-06-03T09:06:12.355403+0000","last_change":"2024-06-03T08:38:03.115748+0000","last_active":"2024-06-03T09:06:12.355403+0000","last_peered":"2024-06-03T09:06:12.355403+0000","last_clean":"2024-06-03T09:06:12.355403+0000","last_became_active":"2024-06-03T08:38:03.110533+0000","last_became_peered":"2024-06-03T08:38:03.110533+0000","last_unstale":"2024-06-03T09:06:12.355403+0000","last_undegraded":"2024-06-03T09:06:12.355403+0000","last_fullsized":"2024-06-03T09:06:12.355403+0000","mapping_epoch":28,"log_start":"0'0","ondisk_log_start":"0'0","created":28,"last_epoch_clean":32,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-06-03T08:37:55.614497+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-06-03T08:37:55.614497+0000","last_clean_scrub_stamp":"2024-06-03T08:37:55.614497+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-06-04T12:36:47.471609+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,7],"acting":[3,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"17.18","version":"0'0","reported_seq":943,"reported_epoch":1008,"state":"active+clean","last_fresh":"2024-06-03T09:06:12.868791+0000","last_change":"2024-06-03T08:38:04.632864+0000","last_active":"2024-06-03T09:06:12.868791+0000","last_peered":"2024-06-03T09:06:12.868791+0000","last_clean":"2024-06-03T09:06:12.868791+0000","last_became_active":"2024-06-03T08:38:04.627856+0000","last_became_peered":"2024-06-03T08:38:04.627856+0000","last_unstale":"2024-06-03T09:06:12.868791+0000","last_undegraded":"2024-06-03T09:06:12.868791+0000","last_fullsized":"2024-06-03T09:06:12.868791+0000","mapping_epoch":28,"log_start":"0'0","ondisk_log_start":"0'0","created":28,"last_epoch_clean":33,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-06-03T08:37:55.614497+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-06-03T08:37:55.614497+0000","last_clean_scrub_stamp":"2024-06-03T08:37:55.614497+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-06-04T19:55:51.181598+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,4],"acting":[7,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"17.19","version":"0'0","reported_seq":933,"reported_epoch":1008,"state":"active+clean","last_fresh":"2024-06-03T09:06:12.336384+0000","last_change":"2024-06-03T08:38:01.839491+0000","last_active":"2024-06-03T09:06:12.336384+0000","last_peered":"2024-06-03T09:06:12.336384+0000","last_clean":"2024-06-03T09:06:12.336384+0000","last_became_active":"2024-06-03T08:38:01.829539+0000","last_became_peered":"2024-06-03T08:38:01.829539+0000","last_unstale":"2024-06-03T09:06:12.336384+0000","last_undegraded":"2024-06-03T09:06:12.336384+0000","last_fullsized":"2024-06-03T09:06:12.336384+0000","mapping_epoch":28,"log_start":"0'0","ondisk_log_start":"0'0","created":28,"last_epoch_clean":30,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-06-03T08:37:55.614497+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-06-03T08:37:55.614497+0000","last_clean_scrub_stamp":"2024-06-03T08:37:55.614497+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-06-04T09:10:43.860229+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,2],"acting":[3,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"17.1e","version":"0'0","reported_seq":144,"reported_epoch":1004,"state":"active+undersized","last_fresh":"2024-06-03T09:06:07.982178+0000","last_change":"2024-06-03T09:06:03.406181+0000","last_active":"2024-06-03T09:06:07.982178+0000","last_peered":"2024-06-03T09:06:07.982178+0000","last_clean":"2024-06-03T08:37:55.614497+0000","last_became_active":"2024-06-03T09:06:03.406181+0000","last_became_peered":"2024-06-03T09:06:03.406181+0000","last_unstale":"2024-06-03T09:06:07.982178+0000","last_undegraded":"2024-06-03T09:06:07.982178+0000","last_fullsized":"2024-06-03T09:06:03.363739+0000","mapping_epoch":999,"log_start":"0'0","ondisk_log_start":"0'0","created":28,"last_epoch_clean":255,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-06-03T08:37:55.614497+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-06-03T08:37:55.614497+0000","last_clean_scrub_stamp":"2024-06-03T08:37:55.614497+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-06-04T11:45:24.486600+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4],"acting":[4],"avail_no_missing":["4"],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"17.1f","version":"0'0","reported_seq":939,"reported_epoch":1007,"state":"active+clean","last_fresh":"2024-06-03T09:06:11.314307+0000","last_change":"2024-06-03T08:38:04.383538+0000","last_active":"2024-06-03T09:06:11.314307+0000","last_peered":"2024-06-03T09:06:11.314307+0000","last_clean":"2024-06-03T09:06:11.314307+0000","last_became_active":"2024-06-03T08:38:04.367278+0000","last_became_peered":"2024-06-03T08:38:04.367278+0000","last_unstale":"2024-06-03T09:06:11.314307+0000","last_undegraded":"2024-06-03T09:06:11.314307+0000","last_fullsized":"2024-06-03T09:06:11.314307+0000","mapping_epoch":28,"log_start":"0'0","ondisk_log_start":"0'0","created":28,"last_epoch_clean":32,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-06-03T08:37:55.614497+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-06-03T08:37:55.614497+0000","last_clean_scrub_stamp":"2024-06-03T08:37:55.614497+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-06-04T10:39:26.552282+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,2],"acting":[0,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"17.1c","version":"0'0","reported_seq":942,"reported_epoch":1008,"state":"active+undersized","last_fresh":"2024-06-03T09:06:12.880704+0000","last_change":"2024-06-03T09:06:03.382210+0000","last_active":"2024-06-03T09:06:12.880704+0000","last_peered":"2024-06-03T09:06:12.880704+0000","last_clean":"2024-06-03T09:04:43.199660+0000","last_became_active":"2024-06-03T09:06:03.382210+0000","last_became_peered":"2024-06-03T09:06:03.382210+0000","last_unstale":"2024-06-03T09:06:12.880704+0000","last_undegraded":"2024-06-03T09:06:12.880704+0000","last_fullsized":"2024-06-03T09:06:03.277874+0000","mapping_epoch":999,"log_start":"0'0","ondisk_log_start":"0'0","created":28,"last_epoch_clean":255,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-06-03T08:37:55.614497+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-06-03T08:37:55.614497+0000","last_clean_scrub_stamp":"2024-06-03T08:37:55.614497+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-06-04T12:18:50.013884+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5],"acting":[5],"avail_no_missing":["5"],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"17.1d","version":"0'0","reported_seq":922,"reported_epoch":1008,"state":"active+clean","last_fresh":"2024-06-03T09:06:12.881860+0000","last_change":"2024-06-03T08:38:06.794630+0000","last_active":"2024-06-03T09:06:12.881860+0000","last_peered":"2024-06-03T09:06:12.881860+0000","last_clean":"2024-06-03T09:06:12.881860+0000","last_became_active":"2024-06-03T08:38:06.592317+0000","last_became_peered":"2024-06-03T08:38:06.592317+0000","last_unstale":"2024-06-03T09:06:12.881860+0000","last_undegraded":"2024-06-03T09:06:12.881860+0000","last_fullsized":"2024-06-03T09:06:12.881860+0000","mapping_epoch":28,"log_start":"0'0","ondisk_log_start":"0'0","created":28,"last_epoch_clean":33,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-06-03T08:37:55.614497+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-06-03T08:37:55.614497+0000","last_clean_scrub_stamp":"2024-06-03T08:37:55.614497+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-06-04T12:33:30.785369+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,7],"acting":[1,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"17.2","version":"0'0","reported_seq":924,"reported_epoch":1008,"state":"active+clean","last_fresh":"2024-06-03T09:06:12.869501+0000","last_change":"2024-06-03T08:38:14.981410+0000","last_active":"2024-06-03T09:06:12.869501+0000","last_peered":"2024-06-03T09:06:12.869501+0000","last_clean":"2024-06-03T09:06:12.869501+0000","last_became_active":"2024-06-03T08:38:14.973825+0000","last_became_peered":"2024-06-03T08:38:14.973825+0000","last_unstale":"2024-06-03T09:06:12.869501+0000","last_undegraded":"2024-06-03T09:06:12.869501+0000","last_fullsized":"2024-06-03T09:06:12.869501+0000","mapping_epoch":28,"log_start":"0'0","ondisk_log_start":"0'0","created":28,"last_epoch_clean":41,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-06-03T08:37:55.614497+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-06-03T08:37:55.614497+0000","last_clean_scrub_stamp":"2024-06-03T08:37:55.614497+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-06-04T17:01:05.518613+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,4],"acting":[1,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"17.3","version":"0'0","reported_seq":941,"reported_epoch":1007,"state":"active+clean","last_fresh":"2024-06-03T09:06:11.338843+0000","last_change":"2024-06-03T08:38:14.619745+0000","last_active":"2024-06-03T09:06:11.338843+0000","last_peered":"2024-06-03T09:06:11.338843+0000","last_clean":"2024-06-03T09:06:11.338843+0000","last_became_active":"2024-06-03T08:38:14.510665+0000","last_became_peered":"2024-06-03T08:38:14.510665+0000","last_unstale":"2024-06-03T09:06:11.338843+0000","last_undegraded":"2024-06-03T09:06:11.338843+0000","last_fullsized":"2024-06-03T09:06:11.338843+0000","mapping_epoch":28,"log_start":"0'0","ondisk_log_start":"0'0","created":28,"last_epoch_clean":42,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-06-03T08:37:55.614497+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-06-03T08:37:55.614497+0000","last_clean_scrub_stamp":"2024-06-03T08:37:55.614497+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-06-04T12:03:47.392064+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,7],"acting":[0,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"17.0","version":"0'0","reported_seq":942,"reported_epoch":1007,"state":"active+clean","last_fresh":"2024-06-03T09:06:12.708393+0000","last_change":"2024-06-03T08:38:04.237118+0000","last_active":"2024-06-03T09:06:12.708393+0000","last_peered":"2024-06-03T09:06:12.708393+0000","last_clean":"2024-06-03T09:06:12.708393+0000","last_became_active":"2024-06-03T08:38:04.231954+0000","last_became_peered":"2024-06-03T08:38:04.231954+0000","last_unstale":"2024-06-03T09:06:12.708393+0000","last_undegraded":"2024-06-03T09:06:12.708393+0000","last_fullsized":"2024-06-03T09:06:12.708393+0000","mapping_epoch":28,"log_start":"0'0","ondisk_log_start":"0'0","created":28,"last_epoch_clean":32,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-06-03T08:37:55.614497+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-06-03T08:37:55.614497+0000","last_clean_scrub_stamp":"2024-06-03T08:37:55.614497+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-06-04T09:56:10.741637+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,2],"acting":[0,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"17.1","version":"65'7","reported_seq":958,"reported_epoch":1008,"state":"active+undersized+degraded","last_fresh":"2024-06-03T09:06:12.872100+0000","last_change":"2024-06-03T09:06:03.370660+0000","last_active":"2024-06-03T09:06:12.872100+0000","last_peered":"2024-06-03T09:06:12.872100+0000","last_clean":"2024-06-03T09:05:03.733864+0000","last_became_active":"2024-06-03T09:06:03.370660+0000","last_became_peered":"2024-06-03T09:06:03.370660+0000","last_unstale":"2024-06-03T09:06:12.872100+0000","last_undegraded":"2024-06-03T09:06:03.266076+0000","last_fullsized":"2024-06-03T09:06:03.264525+0000","mapping_epoch":999,"log_start":"0'0","ondisk_log_start":"0'0","created":28,"last_epoch_clean":255,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-06-03T08:37:55.614497+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-06-03T08:37:55.614497+0000","last_clean_scrub_stamp":"2024-06-03T08:37:55.614497+0000","objects_scrubbed":0,"log_size":7,"log_dups_size":0,"ondisk_log_size":7,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-06-04T11:55:18.194283+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":5,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":1,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":6,"num_read_kb":6,"num_write":7,"num_write_kb":7,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5],"acting":[5],"avail_no_missing":["5"],"object_location_counts":[{"shards":"5","objects":1}],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]}],"pool_stats":[{"poolid":2,"num_pg":8,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":14,"acting":14,"num_store_stats":7},{"poolid":1,"num_pg":1,"stat_sum":{"num_bytes":459280,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":96,"num_read_kb":82,"num_write":113,"num_write_kb":1372,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2,"num_bytes_recovered":397840,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":802816,"data_stored":795680,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":76,"ondisk_log_size":76,"up":2,"acting":2,"num_store_stats":2},{"poolid":17,"num_pg":32,"stat_sum":{"num_bytes":5,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":1,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":6,"num_read_kb":6,"num_write":7,"num_write_kb":7,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":8192,"data_stored":10,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":7,"ondisk_log_size":7,"up":52,"acting":52,"num_store_stats":8}],"osd_stats":[{"osd":6,"up_from":253,"seq":1086626726200,"num_pgs":32,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":54772,"kb_used_data":5608,"kb_used_omap":1,"kb_used_meta":49150,"kb_avail":94317068,"statfs":{"total":96636764160,"available":96580677632,"internally_reserved":0,"allocated":5742592,"data_stored":2767180,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1593,"internal_metadata":50330055},"hb_peers":[0,1,2,3,4,5,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":109,"apply_latency_ms":109,"commit_latency_ns":109000000,"apply_latency_ns":109000000},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Mon Jun 3 09:04:02 2024","interfaces":[{"interface":"back","average":{"1min":1582.4780000000001,"5min":332.72699999999998,"15min":124.113},"min":{"1min":4.4829999999999997,"5min":4.1959999999999997,"15min":4.1959999999999997},"max":{"1min":22766.882000000001,"5min":22766.882000000001,"15min":22766.882000000001},"last":21.917999999999999},{"interface":"front","average":{"1min":1579.5409999999999,"5min":334.94499999999999,"15min":125.84999999999999},"min":{"1min":4.8129999999999997,"5min":4.7539999999999996,"15min":4.4530000000000003},"max":{"1min":22766.289000000001,"5min":22766.289000000001,"15min":22766.289000000001},"last":25.408999999999999}]},{"osd":1,"last update":"Mon Jun 3 09:03:40 2024","interfaces":[{"interface":"back","average":{"1min":36.420999999999999,"5min":77.054000000000002,"15min":132.84800000000001},"min":{"1min":6.0380000000000003,"5min":5.8799999999999999,"15min":5.484},"max":{"1min":589.64099999999996,"5min":4138.1679999999997,"15min":9955.1299999999992},"last":26.84},{"interface":"front","average":{"1min":40.240000000000002,"5min":79.674999999999997,"15min":133.38999999999999},"min":{"1min":8.4589999999999996,"5min":8.4090000000000007,"15min":6.1390000000000002},"max":{"1min":588.60799999999995,"5min":4137.6530000000002,"15min":9827.5509999999995},"last":26.263000000000002}]},{"osd":2,"last update":"Mon Jun 3 09:03:40 2024","interfaces":[{"interface":"back","average":{"1min":12.613,"5min":24.600999999999999,"15min":21.088000000000001},"min":{"1min":6.9740000000000002,"5min":6.9740000000000002,"15min":6.5330000000000004},"max":{"1min":26.076000000000001,"5min":376.24900000000002,"15min":376.24900000000002},"last":24.899000000000001},{"interface":"front","average":{"1min":12.493,"5min":22.207999999999998,"15min":21.219000000000001},"min":{"1min":6.5810000000000004,"5min":6.5110000000000001,"15min":6.0750000000000002},"max":{"1min":29.690999999999999,"5min":271.51100000000002,"15min":434.77300000000002},"last":40.372}]},{"osd":3,"last update":"Mon Jun 3 09:03:40 2024","interfaces":[{"interface":"back","average":{"1min":14.093999999999999,"5min":24.143999999999998,"15min":29.376999999999999},"min":{"1min":10.196999999999999,"5min":6.2919999999999998,"15min":6.1719999999999997},"max":{"1min":24.832000000000001,"5min":197.84899999999999,"15min":1235.5799999999999},"last":24.353999999999999},{"interface":"front","average":{"1min":9.3260000000000005,"5min":18.463999999999999,"15min":27.631},"min":{"1min":6.125,"5min":6.125,"15min":6.125},"max":{"1min":28.166,"5min":200.173,"15min":1157.6780000000001},"last":23.454000000000001}]},{"osd":4,"last update":"Mon Jun 3 09:04:29 2024","interfaces":[{"interface":"back","average":{"1min":10.279,"5min":1253.809,"15min":450.79199999999997},"min":{"1min":7.2300000000000004,"5min":6.093,"15min":6.093},"max":{"1min":15.000999999999999,"5min":28093.096000000001,"15min":28093.096000000001},"last":101.45399999999999},{"interface":"front","average":{"1min":15.693,"5min":1279.759,"15min":463.87099999999998},"min":{"1min":10.551,"5min":7.7480000000000002,"15min":7.7480000000000002},"max":{"1min":27.898,"5min":28323.837,"15min":28323.837},"last":102.262}]},{"osd":5,"last update":"Mon Jun 3 09:04:10 2024","interfaces":[{"interface":"back","average":{"1min":13.728999999999999,"5min":24.670000000000002,"15min":23.815999999999999},"min":{"1min":10.401,"5min":7.516,"15min":7.516},"max":{"1min":25.138000000000002,"5min":320.85000000000002,"15min":445.041},"last":103.52500000000001},{"interface":"front","average":{"1min":13.847,"5min":29.283000000000001,"15min":23.393999999999998},"min":{"1min":7.944,"5min":6.266,"15min":6.266},"max":{"1min":18.817,"5min":530.56700000000001,"15min":530.56700000000001},"last":29.263999999999999}]},{"osd":7,"last update":"Mon Jun 3 09:03:40 2024","interfaces":[{"interface":"back","average":{"1min":20.568999999999999,"5min":30.286999999999999,"15min":27.977},"min":{"1min":13.874000000000001,"5min":10.58,"15min":9.2859999999999996},"max":{"1min":86.959000000000003,"5min":222.07599999999999,"15min":255.08799999999999},"last":37.618000000000002},{"interface":"front","average":{"1min":19.102,"5min":29.550000000000001,"15min":27.763000000000002},"min":{"1min":10.052,"5min":8.6709999999999994,"15min":8.5429999999999993},"max":{"1min":67.968999999999994,"5min":293.62099999999998,"15min":438.14600000000002},"last":111.985}]}]},{"osd":7,"up_from":20,"seq":85899346284,"num_pgs":9,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":58952,"kb_used_data":5692,"kb_used_omap":1,"kb_used_meta":53246,"kb_avail":94312888,"statfs":{"total":96636764160,"available":96576397312,"internally_reserved":0,"allocated":5828608,"data_stored":2735298,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1589,"internal_metadata":54524363},"hb_peers":[0,1,2,3,4,5],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":55,"apply_latency_ms":55,"commit_latency_ns":55000000,"apply_latency_ns":55000000},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Mon Jun 3 09:06:09 2024","interfaces":[{"interface":"back","average":{"1min":14.073,"5min":894.22400000000005,"15min":308.93400000000003},"min":{"1min":5.5709999999999997,"5min":4.218,"15min":4.0599999999999996},"max":{"1min":90.757999999999996,"5min":21729.001,"15min":21729.001},"last":11.128},{"interface":"front","average":{"1min":9.2550000000000008,"5min":906.23500000000001,"15min":314.06},"min":{"1min":5.5789999999999997,"5min":4.8810000000000002,"15min":4.8810000000000002},"max":{"1min":25.937000000000001,"5min":21899.732,"15min":21899.732},"last":16.053999999999998}]},{"osd":1,"last update":"Mon Jun 3 09:06:09 2024","interfaces":[{"interface":"back","average":{"1min":15.182,"5min":69.183999999999997,"15min":219.173},"min":{"1min":7.3639999999999999,"5min":6.6200000000000001,"15min":6.1319999999999997},"max":{"1min":76.926000000000002,"5min":2795.1329999999998,"15min":18373.257000000001},"last":8.6799999999999997},{"interface":"front","average":{"1min":21.495999999999999,"5min":74.703999999999994,"15min":225.285},"min":{"1min":8.8399999999999999,"5min":8.8399999999999999,"15min":8.5129999999999999},"max":{"1min":94.977999999999994,"5min":2803.3670000000002,"15min":18374.601999999999},"last":13.189}]},{"osd":2,"last update":"Mon Jun 3 09:06:09 2024","interfaces":[{"interface":"back","average":{"1min":19.164000000000001,"5min":26.902000000000001,"15min":22.920999999999999},"min":{"1min":11.207000000000001,"5min":8.5030000000000001,"15min":8.5030000000000001},"max":{"1min":91.319999999999993,"5min":333.99200000000002,"15min":333.99200000000002},"last":18.968},{"interface":"front","average":{"1min":11.125999999999999,"5min":18.132000000000001,"15min":16.353000000000002},"min":{"1min":6.202,"5min":5.8369999999999997,"15min":5.5069999999999997},"max":{"1min":28.649000000000001,"5min":226.18100000000001,"15min":226.18100000000001},"last":13.875}]},{"osd":3,"last update":"Mon Jun 3 09:06:09 2024","interfaces":[{"interface":"back","average":{"1min":22.291,"5min":26.289999999999999,"15min":31.074999999999999},"min":{"1min":11.714,"5min":6.8959999999999999,"15min":6.3760000000000003},"max":{"1min":83.102000000000004,"5min":316.64400000000001,"15min":2039.9059999999999},"last":15.42},{"interface":"front","average":{"1min":13.324999999999999,"5min":17.963000000000001,"15min":25.969999999999999},"min":{"1min":6.1200000000000001,"5min":5.2089999999999996,"15min":5.2089999999999996},"max":{"1min":49.908999999999999,"5min":233.542,"15min":2037.5550000000001},"last":7.6879999999999997}]},{"osd":4,"last update":"Mon Jun 3 09:05:28 2024","interfaces":[{"interface":"back","average":{"1min":22.209,"5min":1379.51,"15min":1803.278},"min":{"1min":5.9089999999999998,"5min":5.9089999999999998,"15min":5.9089999999999998},"max":{"1min":224.54300000000001,"5min":30073.917000000001,"15min":62560.868000000002},"last":10.279},{"interface":"front","average":{"1min":22.712,"5min":1401.0719999999999,"15min":1806.759},"min":{"1min":7.6449999999999996,"5min":6.7930000000000001,"15min":5.9340000000000002},"max":{"1min":223.58600000000001,"5min":30348.021000000001,"15min":62451.446000000004},"last":16.579000000000001}]},{"osd":5,"last update":"Mon Jun 3 09:06:02 2024","interfaces":[{"interface":"back","average":{"1min":24.372,"5min":26.856000000000002,"15min":23.571999999999999},"min":{"1min":12.278,"5min":9.2460000000000004,"15min":6.5650000000000004},"max":{"1min":183.64099999999999,"5min":317.572,"15min":317.572},"last":10.933999999999999},{"interface":"front","average":{"1min":21.259,"5min":23.364000000000001,"15min":21.193999999999999},"min":{"1min":7.8499999999999996,"5min":7.0999999999999996,"15min":7.008},"max":{"1min":184.53399999999999,"5min":306.60599999999999,"15min":306.60599999999999},"last":18.036000000000001}]},{"osd":6,"last update":"Mon Jun 3 09:04:31 2024","interfaces":[{"interface":"back","average":{"1min":44.061,"5min":26.821000000000002,"15min":91.176000000000002},"min":{"1min":9.0800000000000001,"5min":7.9390000000000001,"15min":7.9390000000000001},"max":{"1min":268.029,"5min":308.916,"15min":8734.4639999999999},"last":6.6769999999999996},{"interface":"front","average":{"1min":46.643000000000001,"5min":23.879999999999999,"15min":88.765000000000001},"min":{"1min":8.8119999999999994,"5min":6.9240000000000004,"15min":6.742},"max":{"1min":305.64600000000002,"5min":305.64600000000002,"15min":8735.5319999999992},"last":12.327}]}]},{"osd":5,"up_from":20,"seq":85899346258,"num_pgs":9,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":59340,"kb_used_data":6076,"kb_used_omap":1,"kb_used_meta":53246,"kb_avail":94312500,"statfs":{"total":96636764160,"available":96576000000,"internally_reserved":0,"allocated":6221824,"data_stored":3126752,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1594,"internal_metadata":54524358},"hb_peers":[0,1,2,3,4,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":51,"apply_latency_ms":51,"commit_latency_ns":51000000,"apply_latency_ns":51000000},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Mon Jun 3 09:06:06 2024","interfaces":[{"interface":"back","average":{"1min":7.468,"5min":1703.037,"15min":580.51900000000001},"min":{"1min":4.6509999999999998,"5min":4.6509999999999998,"15min":4.4020000000000001},"max":{"1min":14.711,"5min":26433.764999999999,"15min":26433.764999999999},"last":7.9260000000000002},{"interface":"front","average":{"1min":9.9429999999999996,"5min":1725.27,"15min":587.87800000000004},"min":{"1min":5.5999999999999996,"5min":5.1319999999999997,"15min":4.8170000000000002},"max":{"1min":13.590999999999999,"5min":26704.698,"15min":26704.698},"last":10.76}]},{"osd":1,"last update":"Mon Jun 3 09:06:00 2024","interfaces":[{"interface":"back","average":{"1min":10.862,"5min":67.731999999999999,"15min":217.364},"min":{"1min":7.1769999999999996,"5min":6.0890000000000004,"15min":5.4930000000000003},"max":{"1min":16.998999999999999,"5min":3923.4090000000001,"15min":18811.696},"last":11.622999999999999},{"interface":"front","average":{"1min":10.417,"5min":65.141999999999996,"15min":220.75700000000001},"min":{"1min":5.3209999999999997,"5min":5.3209999999999997,"15min":5.0209999999999999},"max":{"1min":15.821999999999999,"5min":4019.6129999999998,"15min":18968.365000000002},"last":12.172000000000001}]},{"osd":2,"last update":"Mon Jun 3 09:06:00 2024","interfaces":[{"interface":"back","average":{"1min":11.366,"5min":24.18,"15min":21.137},"min":{"1min":5.6369999999999996,"5min":5.6369999999999996,"15min":5.0510000000000002},"max":{"1min":15.529999999999999,"5min":332.48200000000003,"15min":332.48200000000003},"last":13.997},{"interface":"front","average":{"1min":10.089,"5min":23.117000000000001,"15min":24.050000000000001},"min":{"1min":6.6769999999999996,"5min":6.6769999999999996,"15min":5.8739999999999997},"max":{"1min":15.821,"5min":353.03199999999998,"15min":353.03199999999998},"last":9.0280000000000005}]},{"osd":3,"last update":"Mon Jun 3 09:06:00 2024","interfaces":[{"interface":"back","average":{"1min":13.472,"5min":24.283999999999999,"15min":31.666},"min":{"1min":10.111000000000001,"5min":7.4720000000000004,"15min":7.4210000000000003},"max":{"1min":16.271999999999998,"5min":353.57499999999999,"15min":2855.8519999999999},"last":19.434999999999999},{"interface":"front","average":{"1min":9.5899999999999999,"5min":15.875,"15min":25.834},"min":{"1min":6.6609999999999996,"5min":5.7729999999999997,"15min":5.1909999999999998},"max":{"1min":15.112,"5min":203.804,"15min":2869.7860000000001},"last":18.408000000000001}]},{"osd":4,"last update":"Mon Jun 3 09:05:27 2024","interfaces":[{"interface":"back","average":{"1min":19.155000000000001,"5min":1084.961,"15min":1346.9949999999999},"min":{"1min":7.6130000000000004,"5min":7.6130000000000004,"15min":6.1340000000000003},"max":{"1min":156.55199999999999,"5min":28619.825000000001,"15min":57999.453999999998},"last":13.683},{"interface":"front","average":{"1min":28.068999999999999,"5min":1087.106,"15min":1330.289},"min":{"1min":8.3949999999999996,"5min":6.5030000000000001,"15min":6.5030000000000001},"max":{"1min":254.91300000000001,"5min":28621.294000000002,"15min":57434.631999999998},"last":14.867000000000001}]},{"osd":6,"last update":"Mon Jun 3 09:03:57 2024","interfaces":[{"interface":"back","average":{"1min":23.536000000000001,"5min":36.457000000000001,"15min":83.799999999999997},"min":{"1min":9.4800000000000004,"5min":9.4800000000000004,"15min":8.0630000000000006},"max":{"1min":89.442999999999998,"5min":508.77999999999997,"15min":9193.5169999999998},"last":157.79300000000001},{"interface":"front","average":{"1min":18.891999999999999,"5min":22.100999999999999,"15min":75.489999999999995},"min":{"1min":5.923,"5min":5.923,"15min":5.923},"max":{"1min":78.424999999999997,"5min":464.62,"15min":9191.4429999999993},"last":148.09}]},{"osd":7,"last update":"Mon Jun 3 09:06:00 2024","interfaces":[{"interface":"back","average":{"1min":13.257,"5min":20.917999999999999,"15min":21.088999999999999},"min":{"1min":10.475,"5min":6.7629999999999999,"15min":6.7629999999999999},"max":{"1min":17.245000000000001,"5min":211.22999999999999,"15min":211.22999999999999},"last":15.555},{"interface":"front","average":{"1min":12.872999999999999,"5min":20.587,"15min":24.021000000000001},"min":{"1min":8.5760000000000005,"5min":7.7709999999999999,"15min":6.8570000000000002},"max":{"1min":18.359999999999999,"5min":223.191,"15min":286.69499999999999},"last":9.7189999999999994}]}]},{"osd":4,"up_from":20,"seq":85899346265,"num_pgs":20,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":63000,"kb_used_data":5644,"kb_used_omap":1,"kb_used_meta":57342,"kb_avail":94308840,"statfs":{"total":96636764160,"available":96572252160,"internally_reserved":0,"allocated":5779456,"data_stored":2718472,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1594,"internal_metadata":58718662},"hb_peers":[0,1,2,3,5,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":50,"apply_latency_ms":50,"commit_latency_ns":50000000,"apply_latency_ns":50000000},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Mon Jun 3 09:05:34 2024","interfaces":[{"interface":"back","average":{"1min":12.416,"5min":1099.4649999999999,"15min":382.15199999999999},"min":{"1min":4.5279999999999996,"5min":4.5279999999999996,"15min":4.3220000000000001},"max":{"1min":123.878,"5min":21726.743999999999,"15min":21726.743999999999},"last":52.241},{"interface":"front","average":{"1min":16.462,"5min":1108.894,"15min":387.70299999999997},"min":{"1min":6.4089999999999998,"5min":5.8380000000000001,"15min":5.4909999999999997},"max":{"1min":130.185,"5min":21727.993999999999,"15min":21727.993999999999},"last":53.148000000000003}]},{"osd":1,"last update":"Mon Jun 3 09:05:34 2024","interfaces":[{"interface":"back","average":{"1min":14.727,"5min":56.755000000000003,"15min":341.09399999999999},"min":{"1min":4.6529999999999996,"5min":4.1970000000000001,"15min":4.1970000000000001},"max":{"1min":126.899,"5min":3346.4389999999999,"15min":22256.098000000002},"last":47.073},{"interface":"front","average":{"1min":13.472,"5min":57.503,"15min":340.303},"min":{"1min":5.0590000000000002,"5min":5.0590000000000002,"15min":4.2640000000000002},"max":{"1min":126.096,"5min":3347.181,"15min":22247.317999999999},"last":49.558}]},{"osd":2,"last update":"Mon Jun 3 09:05:34 2024","interfaces":[{"interface":"back","average":{"1min":26.088999999999999,"5min":23.866,"15min":20.620999999999999},"min":{"1min":7.0869999999999997,"5min":6.8239999999999998,"15min":6.3150000000000004},"max":{"1min":337.60399999999998,"5min":393.58600000000001,"15min":393.58600000000001},"last":85.340000000000003},{"interface":"front","average":{"1min":15.800000000000001,"5min":22.584,"15min":23.245999999999999},"min":{"1min":6.0629999999999997,"5min":6.0629999999999997,"15min":6.0629999999999997},"max":{"1min":127.913,"5min":394.471,"15min":394.471},"last":82.120000000000005}]},{"osd":3,"last update":"Mon Jun 3 09:05:34 2024","interfaces":[{"interface":"back","average":{"1min":17.544,"5min":19.143999999999998,"15min":18.742000000000001},"min":{"1min":5.8719999999999999,"5min":5.4100000000000001,"15min":5.4000000000000004},"max":{"1min":127.43600000000001,"5min":322.63,"15min":322.63},"last":84.555999999999997},{"interface":"front","average":{"1min":21.327000000000002,"5min":23.559000000000001,"15min":23.797000000000001},"min":{"1min":10.023,"5min":7.484,"15min":7.484},"max":{"1min":204.517,"5min":392.67500000000001,"15min":392.67500000000001},"last":71.394000000000005}]},{"osd":5,"last update":"Mon Jun 3 09:05:34 2024","interfaces":[{"interface":"back","average":{"1min":24.274999999999999,"5min":21.274999999999999,"15min":20.988},"min":{"1min":6.7489999999999997,"5min":6.7489999999999997,"15min":6.7489999999999997},"max":{"1min":268.10199999999998,"5min":344.23399999999998,"15min":344.23399999999998},"last":51.326000000000001},{"interface":"front","average":{"1min":26.361999999999998,"5min":25.218,"15min":26.084},"min":{"1min":12.228,"5min":8.5180000000000007,"15min":8.5180000000000007},"max":{"1min":264.75,"5min":345.18700000000001,"15min":345.18700000000001},"last":40.829000000000001}]},{"osd":6,"last update":"Mon Jun 3 09:04:32 2024","interfaces":[{"interface":"back","average":{"1min":15.548,"5min":32.957999999999998,"15min":152.738},"min":{"1min":9.141,"5min":9.141,"15min":8.2319999999999993},"max":{"1min":52.57,"5min":395.238,"15min":12496.335999999999},"last":208.90000000000001},{"interface":"front","average":{"1min":12.680999999999999,"5min":25.428000000000001,"15min":149.30600000000001},"min":{"1min":6.4379999999999997,"5min":5.6319999999999997,"15min":5.6319999999999997},"max":{"1min":46.283000000000001,"5min":377.74299999999999,"15min":12616.297},"last":205.88900000000001}]},{"osd":7,"last update":"Mon Jun 3 09:05:34 2024","interfaces":[{"interface":"back","average":{"1min":18.295999999999999,"5min":19.585999999999999,"15min":21.062000000000001},"min":{"1min":6.2110000000000003,"5min":6.2110000000000003,"15min":5.8010000000000002},"max":{"1min":207.25299999999999,"5min":358.94900000000001,"15min":358.94900000000001},"last":14.252000000000001},{"interface":"front","average":{"1min":23.712,"5min":24.024999999999999,"15min":24.640000000000001},"min":{"1min":11.824999999999999,"5min":7.7789999999999999,"15min":7.7789999999999999},"max":{"1min":209.32900000000001,"5min":346.25200000000001,"15min":380.06900000000002},"last":45.639000000000003}]}]},{"osd":2,"up_from":20,"seq":85899346285,"num_pgs":24,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":58952,"kb_used_data":5688,"kb_used_omap":1,"kb_used_meta":53246,"kb_avail":94312888,"statfs":{"total":96636764160,"available":96576397312,"internally_reserved":0,"allocated":5824512,"data_stored":2763528,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1594,"internal_metadata":54524358},"hb_peers":[0,1,3,4,5,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":79,"apply_latency_ms":79,"commit_latency_ns":79000000,"apply_latency_ns":79000000},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Mon Jun 3 09:05:21 2024","interfaces":[{"interface":"back","average":{"1min":16.364000000000001,"5min":947.59799999999996,"15min":326.58999999999997},"min":{"1min":6.0830000000000002,"5min":4.7050000000000001,"15min":4.1420000000000003},"max":{"1min":157.60900000000001,"5min":23465.18,"15min":23465.18},"last":8.3209999999999997},{"interface":"front","average":{"1min":12.276,"5min":945.29899999999998,"15min":326.78399999999999},"min":{"1min":5.1989999999999998,"5min":4.7439999999999998,"15min":4.6950000000000003},"max":{"1min":102.557,"5min":23465.966,"15min":23465.966},"last":7.2119999999999997}]},{"osd":1,"last update":"Mon Jun 3 09:05:21 2024","interfaces":[{"interface":"back","average":{"1min":17.803999999999998,"5min":77.658000000000001,"15min":294.96899999999999},"min":{"1min":7.2130000000000001,"5min":5.4199999999999999,"15min":5.4199999999999999},"max":{"1min":159.624,"5min":4220.5290000000005,"15min":18954.987000000001},"last":6.4089999999999998},{"interface":"front","average":{"1min":21.542999999999999,"5min":83.528999999999996,"15min":303.11700000000002},"min":{"1min":8.8200000000000003,"5min":7.5949999999999998,"15min":7.5949999999999998},"max":{"1min":170.74299999999999,"5min":4325.9489999999996,"15min":19143.834999999999},"last":11.071}]},{"osd":3,"last update":"Mon Jun 3 09:05:21 2024","interfaces":[{"interface":"back","average":{"1min":19.164999999999999,"5min":13.284000000000001,"15min":28.751999999999999},"min":{"1min":7.3710000000000004,"5min":5.9290000000000003,"15min":5.2450000000000001},"max":{"1min":118.913,"5min":118.913,"15min":2594.8150000000001},"last":8.8699999999999992},{"interface":"front","average":{"1min":22.094000000000001,"5min":17.244,"15min":32.173999999999999},"min":{"1min":9.516,"5min":8.0760000000000005,"15min":6.0780000000000003},"max":{"1min":156.34700000000001,"5min":156.34700000000001,"15min":2706.5120000000002},"last":11.682}]},{"osd":4,"last update":"Mon Jun 3 09:05:57 2024","interfaces":[{"interface":"back","average":{"1min":10.573,"5min":1266.894,"15min":438.154},"min":{"1min":8.6379999999999999,"5min":7.8789999999999996,"15min":6.2039999999999997},"max":{"1min":15.105,"5min":26660.861000000001,"15min":26660.861000000001},"last":14.795},{"interface":"front","average":{"1min":12.407999999999999,"5min":1266.7460000000001,"15min":435.75400000000002},"min":{"1min":10.266999999999999,"5min":10.266999999999999,"15min":6.2720000000000002},"max":{"1min":15.228,"5min":26616.243999999999,"15min":26616.243999999999},"last":15.5}]},{"osd":5,"last update":"Mon Jun 3 09:05:21 2024","interfaces":[{"interface":"back","average":{"1min":25.632000000000001,"5min":20.437999999999999,"15min":21.472999999999999},"min":{"1min":9.2159999999999993,"5min":9.2159999999999993,"15min":6.2050000000000001},"max":{"1min":252.31100000000001,"5min":252.31100000000001,"15min":252.31100000000001},"last":13.285},{"interface":"front","average":{"1min":24.824000000000002,"5min":18.521000000000001,"15min":21.027000000000001},"min":{"1min":8.0449999999999999,"5min":7.4850000000000003,"15min":6.8810000000000002},"max":{"1min":258.31700000000001,"5min":258.31700000000001,"15min":258.31700000000001},"last":9.5879999999999992}]},{"osd":6,"last update":"Mon Jun 3 09:04:28 2024","interfaces":[{"interface":"back","average":{"1min":14.063000000000001,"5min":24.850000000000001,"15min":106.755},"min":{"1min":9.0820000000000007,"5min":8.9649999999999999,"15min":8.9649999999999999},"max":{"1min":24.666,"5min":188.06700000000001,"15min":9521.0010000000002},"last":183.60900000000001},{"interface":"front","average":{"1min":13.933999999999999,"5min":21.081,"15min":106.142},"min":{"1min":5.8719999999999999,"5min":5.8719999999999999,"15min":5.8719999999999999},"max":{"1min":117.551,"5min":193.887,"15min":9585.6620000000003},"last":184.20599999999999}]},{"osd":7,"last update":"Mon Jun 3 09:05:21 2024","interfaces":[{"interface":"back","average":{"1min":25.097000000000001,"5min":22.579999999999998,"15min":24.388000000000002},"min":{"1min":7.8959999999999999,"5min":6.6130000000000004,"15min":6.6130000000000004},"max":{"1min":242.56800000000001,"5min":242.56800000000001,"15min":242.56800000000001},"last":12.548},{"interface":"front","average":{"1min":24.414999999999999,"5min":23.995999999999999,"15min":26.614999999999998},"min":{"1min":12.43,"5min":9.0660000000000007,"15min":9.0660000000000007},"max":{"1min":181.56399999999999,"5min":182.5,"15min":214.33799999999999},"last":13.968999999999999}]}]},{"osd":3,"up_from":14,"seq":60129542500,"num_pgs":11,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":58940,"kb_used_data":5680,"kb_used_omap":1,"kb_used_meta":53246,"kb_avail":94312900,"statfs":{"total":96636764160,"available":96576409600,"internally_reserved":0,"allocated":5816320,"data_stored":2728907,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1592,"internal_metadata":54524360},"hb_peers":[0,1,2,4,5,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":20,"apply_latency_ms":20,"commit_latency_ns":20000000,"apply_latency_ns":20000000},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Mon Jun 3 09:05:25 2024","interfaces":[{"interface":"back","average":{"1min":8.4809999999999999,"5min":1064.2270000000001,"15min":367.88600000000002},"min":{"1min":6.0529999999999999,"5min":5.173,"15min":4.6929999999999996},"max":{"1min":12.010999999999999,"5min":23275.167000000001,"15min":23275.167000000001},"last":8.0489999999999995},{"interface":"front","average":{"1min":13.885,"5min":1070.3710000000001,"15min":373.53500000000003},"min":{"1min":9.6679999999999993,"5min":7.2089999999999996,"15min":5.9880000000000004},"max":{"1min":32.426000000000002,"5min":23278.536,"15min":23278.536},"last":10.086}]},{"osd":1,"last update":"Mon Jun 3 09:05:25 2024","interfaces":[{"interface":"back","average":{"1min":11.324,"5min":19.920000000000002,"15min":390.07900000000001},"min":{"1min":6.6379999999999999,"5min":4.8200000000000003,"15min":4.8200000000000003},"max":{"1min":26.449000000000002,"5min":250.72300000000001,"15min":21658.205999999998},"last":4.335},{"interface":"front","average":{"1min":9.468,"5min":20.966999999999999,"15min":381.18799999999999},"min":{"1min":5.4969999999999999,"5min":5.2779999999999996,"15min":5.2779999999999996},"max":{"1min":25.643000000000001,"5min":335.33699999999999,"15min":21307.069},"last":7.3360000000000003}]},{"osd":2,"last update":"Mon Jun 3 09:05:25 2024","interfaces":[{"interface":"back","average":{"1min":13.215,"5min":26.649000000000001,"15min":23.783999999999999},"min":{"1min":7.8710000000000004,"5min":6.4900000000000002,"15min":5.875},"max":{"1min":34.322000000000003,"5min":377.803,"15min":377.803},"last":11.356999999999999},{"interface":"front","average":{"1min":13.105,"5min":22.292999999999999,"15min":20.573},"min":{"1min":7.327,"5min":6.125,"15min":5.9020000000000001},"max":{"1min":32.756999999999998,"5min":371.601,"15min":371.601},"last":8.9670000000000005}]},{"osd":4,"last update":"Mon Jun 3 09:05:27 2024","interfaces":[{"interface":"back","average":{"1min":10.444000000000001,"5min":1211.4970000000001,"15min":1639.1579999999999},"min":{"1min":6.4429999999999996,"5min":6.2560000000000002,"15min":6.0309999999999997},"max":{"1min":21.263000000000002,"5min":30144.939999999999,"15min":62679.622000000003},"last":14.975},{"interface":"front","average":{"1min":11.363,"5min":1219.6400000000001,"15min":1644.127},"min":{"1min":7.8109999999999999,"5min":7.8109999999999999,"15min":6.2439999999999998},"max":{"1min":20.108000000000001,"5min":30180.689999999999,"15min":62681.021999999997},"last":14.266}]},{"osd":5,"last update":"Mon Jun 3 09:06:07 2024","interfaces":[{"interface":"back","average":{"1min":12.802,"5min":19.640999999999998,"15min":22.318000000000001},"min":{"1min":9.0890000000000004,"5min":6.8810000000000002,"15min":6.7690000000000001},"max":{"1min":16.745999999999999,"5min":249.88,"15min":327.34899999999999},"last":11.952},{"interface":"front","average":{"1min":15.002000000000001,"5min":24.003,"15min":26.649999999999999},"min":{"1min":11.984,"5min":9.9149999999999991,"15min":8.9879999999999995},"max":{"1min":18.407,"5min":251.47800000000001,"15min":331.86799999999999},"last":13.717000000000001}]},{"osd":6,"last update":"Mon Jun 3 09:04:04 2024","interfaces":[{"interface":"back","average":{"1min":22.815999999999999,"5min":31.513999999999999,"15min":181.74299999999999},"min":{"1min":9.4809999999999999,"5min":7.2240000000000002,"15min":7.2240000000000002},"max":{"1min":160.56999999999999,"5min":380.613,"15min":11893.74},"last":19.359999999999999},{"interface":"front","average":{"1min":25.359999999999999,"5min":30.981000000000002,"15min":179.81399999999999},"min":{"1min":9.8870000000000005,"5min":7.3360000000000003,"15min":6.0899999999999999},"max":{"1min":193.11099999999999,"5min":379.80700000000002,"15min":11893.069},"last":8.202}]},{"osd":7,"last update":"Mon Jun 3 09:05:25 2024","interfaces":[{"interface":"back","average":{"1min":12.177,"5min":21.126000000000001,"15min":22.972000000000001},"min":{"1min":8.6769999999999996,"5min":7.4279999999999999,"15min":7.4279999999999999},"max":{"1min":17.376999999999999,"5min":164.27500000000001,"15min":178.76300000000001},"last":9.5540000000000003},{"interface":"front","average":{"1min":15.675000000000001,"5min":22.722000000000001,"15min":25.238},"min":{"1min":11.233000000000001,"5min":10.555999999999999,"15min":7.6440000000000001},"max":{"1min":29.280000000000001,"5min":148.13800000000001,"15min":417.22800000000001},"last":12.855}]}]},{"osd":1,"up_from":14,"seq":60129542498,"num_pgs":9,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":59332,"kb_used_data":6072,"kb_used_omap":1,"kb_used_meta":53246,"kb_avail":94312508,"statfs":{"total":96636764160,"available":96576008192,"internally_reserved":0,"allocated":6217728,"data_stored":3126747,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":54524357},"hb_peers":[0,2,3,4,5,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":65,"apply_latency_ms":65,"commit_latency_ns":65000000,"apply_latency_ns":65000000},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Mon Jun 3 09:06:07 2024","interfaces":[{"interface":"back","average":{"1min":7.7530000000000001,"5min":185.66999999999999,"15min":77.347999999999999},"min":{"1min":5.0229999999999997,"5min":4.9279999999999999,"15min":4.5449999999999999},"max":{"1min":11.896000000000001,"5min":7470.616,"15min":7470.616},"last":10.471},{"interface":"front","average":{"1min":9.9220000000000006,"5min":187.14599999999999,"15min":76.325999999999993},"min":{"1min":5.3159999999999998,"5min":5.3159999999999998,"15min":5.3159999999999998},"max":{"1min":18.338000000000001,"5min":7477.0829999999996,"15min":7477.0829999999996},"last":5.7300000000000004}]},{"osd":2,"last update":"Mon Jun 3 09:05:18 2024","interfaces":[{"interface":"back","average":{"1min":10.954000000000001,"5min":18.295000000000002,"15min":24.126999999999999},"min":{"1min":6.8739999999999997,"5min":5.8650000000000002,"15min":5.8650000000000002},"max":{"1min":21.122,"5min":228,"15min":657.38},"last":8.5600000000000005},{"interface":"front","average":{"1min":13.694000000000001,"5min":24.446999999999999,"15min":32.055999999999997},"min":{"1min":9.9860000000000007,"5min":9.1229999999999993,"15min":7.2160000000000002},"max":{"1min":26.507999999999999,"5min":229.465,"15min":700.06500000000005},"last":12.885999999999999}]},{"osd":3,"last update":"Mon Jun 3 09:06:04 2024","interfaces":[{"interface":"back","average":{"1min":8.6110000000000007,"5min":15.733000000000001,"15min":26.646999999999998},"min":{"1min":5.827,"5min":5.4640000000000004,"15min":5.4640000000000004},"max":{"1min":13.105,"5min":121.619,"15min":1676.009},"last":6.3609999999999998},{"interface":"front","average":{"1min":9.5850000000000009,"5min":16.161999999999999,"15min":29.065999999999999},"min":{"1min":6.3630000000000004,"5min":5.7699999999999996,"15min":5.7699999999999996},"max":{"1min":15.166,"5min":132.036,"15min":1766.5840000000001},"last":12.067}]},{"osd":4,"last update":"Mon Jun 3 09:05:31 2024","interfaces":[{"interface":"back","average":{"1min":10.996,"5min":954.46500000000003,"15min":1281.6859999999999},"min":{"1min":7.702,"5min":6.1760000000000002,"15min":6.0970000000000004},"max":{"1min":21.538,"5min":28044.862000000001,"15min":59438.017999999996},"last":7.4029999999999996},{"interface":"front","average":{"1min":13.646000000000001,"5min":961.24599999999998,"15min":1285.26},"min":{"1min":9.7200000000000006,"5min":9.3940000000000001,"15min":6.4640000000000004},"max":{"1min":23.277999999999999,"5min":28045.624,"15min":59439.283000000003},"last":9.8949999999999996}]},{"osd":5,"last update":"Mon Jun 3 09:06:04 2024","interfaces":[{"interface":"back","average":{"1min":12.478999999999999,"5min":20.228000000000002,"15min":25.997},"min":{"1min":8.6430000000000007,"5min":7.9720000000000004,"15min":7.234},"max":{"1min":17.442,"5min":139.43199999999999,"15min":701.22000000000003},"last":7.9050000000000002},{"interface":"front","average":{"1min":13.507999999999999,"5min":25.760999999999999,"15min":27.553999999999998},"min":{"1min":10.042999999999999,"5min":7.258,"15min":6.242},"max":{"1min":20.611000000000001,"5min":278.63,"15min":598.35000000000002},"last":11.395}]},{"osd":6,"last update":"Mon Jun 3 09:03:54 2024","interfaces":[{"interface":"back","average":{"1min":28.524000000000001,"5min":20.654,"15min":23.878},"min":{"1min":6.0309999999999997,"5min":6.0309999999999997,"15min":6.0309999999999997},"max":{"1min":167.221,"5min":167.221,"15min":704.245},"last":18.721},{"interface":"front","average":{"1min":27.606000000000002,"5min":30.131,"15min":28.628},"min":{"1min":9.8650000000000002,"5min":9.7200000000000006,"15min":8.3580000000000005},"max":{"1min":123.938,"5min":358.23000000000002,"15min":634.05499999999995},"last":173.113}]},{"osd":7,"last update":"Mon Jun 3 09:05:18 2024","interfaces":[{"interface":"back","average":{"1min":21.853999999999999,"5min":24.245999999999999,"15min":32.801000000000002},"min":{"1min":9.5020000000000007,"5min":8.0559999999999992,"15min":8.0559999999999992},"max":{"1min":142.245,"5min":270.95100000000002,"15min":703.07799999999997},"last":15.696},{"interface":"front","average":{"1min":21.547000000000001,"5min":29.977,"15min":30.513999999999999},"min":{"1min":11.167,"5min":8.2089999999999996,"15min":7.7599999999999998},"max":{"1min":119.348,"5min":360.99900000000002,"15min":608.44100000000003},"last":14.506}]}]},{"osd":0,"up_from":14,"seq":60129542500,"num_pgs":12,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":81088,"kb_used_data":5680,"kb_used_omap":1,"kb_used_meta":75390,"kb_avail":94290752,"statfs":{"total":96636764160,"available":96553730048,"internally_reserved":0,"allocated":5816320,"data_stored":2728907,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1593,"internal_metadata":77199815},"hb_peers":[1,2,3,4,5,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":4,"apply_latency_ms":4,"commit_latency_ns":4000000,"apply_latency_ns":4000000},"alerts":[],"network_ping_times":[{"osd":1,"last update":"Mon Jun 3 09:06:04 2024","interfaces":[{"interface":"back","average":{"1min":6.9960000000000004,"5min":56.997,"15min":339.29700000000003},"min":{"1min":4.4530000000000003,"5min":4.4530000000000003,"15min":4.4100000000000001},"max":{"1min":11.720000000000001,"5min":2414.4749999999999,"15min":20691.453000000001},"last":9.875},{"interface":"front","average":{"1min":9.548,"5min":57.258000000000003,"15min":337.791},"min":{"1min":5.2300000000000004,"5min":5.2300000000000004,"15min":4.6050000000000004},"max":{"1min":15.337999999999999,"5min":2412.9549999999999,"15min":20640.792000000001},"last":4.7199999999999998}]},{"osd":2,"last update":"Mon Jun 3 09:05:22 2024","interfaces":[{"interface":"back","average":{"1min":9.5820000000000007,"5min":27.760000000000002,"15min":23.097999999999999},"min":{"1min":5.766,"5min":5.766,"15min":5.766},"max":{"1min":46.584000000000003,"5min":627.40700000000004,"15min":627.40700000000004},"last":6.3840000000000003},{"interface":"front","average":{"1min":13.574999999999999,"5min":35.847999999999999,"15min":30.065000000000001},"min":{"1min":8.6419999999999995,"5min":7.8630000000000004,"15min":6.9260000000000002},"max":{"1min":54.68,"5min":635.62599999999998,"15min":635.62599999999998},"last":8.5039999999999996}]},{"osd":3,"last update":"Mon Jun 3 09:06:04 2024","interfaces":[{"interface":"back","average":{"1min":12.105,"5min":26.925000000000001,"15min":26.584},"min":{"1min":7.8410000000000002,"5min":7.1280000000000001,"15min":6.1909999999999998},"max":{"1min":18.425999999999998,"5min":389.72699999999998,"15min":389.72699999999998},"last":10.986000000000001},{"interface":"front","average":{"1min":13.794,"5min":29.771999999999998,"15min":28.815000000000001},"min":{"1min":10.084,"5min":9.4920000000000009,"15min":6.1849999999999996},"max":{"1min":17.373999999999999,"5min":390.23200000000003,"15min":494.92899999999997},"last":16.532}]},{"osd":4,"last update":"Mon Jun 3 09:05:27 2024","interfaces":[{"interface":"back","average":{"1min":13.478,"5min":1032.126,"15min":1389.729},"min":{"1min":6.5119999999999996,"5min":6.5119999999999996,"15min":6.5119999999999996},"max":{"1min":55.354999999999997,"5min":25555.212,"15min":60989.964},"last":12.74},{"interface":"front","average":{"1min":16.204999999999998,"5min":1030.8389999999999,"15min":1387.778},"min":{"1min":10.282999999999999,"5min":7.1429999999999998,"15min":7.1429999999999998},"max":{"1min":49.945999999999998,"5min":25556.159,"15min":61000.017},"last":13.673}]},{"osd":5,"last update":"Mon Jun 3 09:05:13 2024","interfaces":[{"interface":"back","average":{"1min":13.778,"5min":31.690999999999999,"15min":29.962},"min":{"1min":8.5030000000000001,"5min":7.7030000000000003,"15min":7.7030000000000003},"max":{"1min":60.216999999999999,"5min":455.78699999999998,"15min":605.05899999999997},"last":14.942},{"interface":"front","average":{"1min":17.823,"5min":30.498999999999999,"15min":27.434999999999999},"min":{"1min":11.547000000000001,"5min":6.2030000000000003,"15min":6.2030000000000003},"max":{"1min":59.677,"5min":391.31200000000001,"15min":541.11000000000001},"last":17.879000000000001}]},{"osd":6,"last update":"Mon Jun 3 09:04:30 2024","interfaces":[{"interface":"back","average":{"1min":52.496000000000002,"5min":33.139000000000003,"15min":160.83199999999999},"min":{"1min":7.5289999999999999,"5min":7.1040000000000001,"15min":6.3739999999999997},"max":{"1min":457.91699999999997,"5min":457.91699999999997,"15min":11115.258},"last":18.959},{"interface":"front","average":{"1min":49.670000000000002,"5min":33.704999999999998,"15min":162.791},"min":{"1min":12.247999999999999,"5min":11.019,"15min":9.9760000000000009},"max":{"1min":407.05099999999999,"5min":407.05099999999999,"15min":11092.194},"last":17.707000000000001}]},{"osd":7,"last update":"Mon Jun 3 09:05:22 2024","interfaces":[{"interface":"back","average":{"1min":14.503,"5min":29.489000000000001,"15min":33.134999999999998},"min":{"1min":8.5690000000000008,"5min":7.5149999999999997,"15min":6.9859999999999998},"max":{"1min":48.526000000000003,"5min":461.41699999999997,"15min":598.33699999999999},"last":11.83},{"interface":"front","average":{"1min":17.050000000000001,"5min":33.313000000000002,"15min":38.603999999999999},"min":{"1min":11.182,"5min":10.263999999999999,"15min":10.263999999999999},"max":{"1min":58.683,"5min":463.10599999999999,"15min":592.80499999999995},"last":14.407}]}]}],"pool_statfs":[{"poolid":1,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":401408,"data_stored":397840,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":5,"total":0,"available":0,"internally_reserved":0,"allocated":401408,"data_stored":397840,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":4,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":5,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":6,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":17,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":17,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":17,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":17,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":17,"osd":4,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":17,"osd":5,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":5,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":17,"osd":6,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":5,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":17,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0}]}} 2024-06-03T09:06:15.554 DEBUG:teuthology.orchestra.run.smithi088:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph pg dump --format=json 2024-06-03T09:06:15.652 INFO:teuthology.orchestra.run.smithi088.stderr:2024-06-03T09:06:15.652+0000 7f4bea384640 10 monclient: get_monmap_and_config 2024-06-03T09:06:15.652 INFO:teuthology.orchestra.run.smithi088.stderr:2024-06-03T09:06:15.652+0000 7f4bea384640 10 monclient: build_initial_monmap 2024-06-03T09:06:15.652 INFO:teuthology.orchestra.run.smithi088.stderr:2024-06-03T09:06:15.652+0000 7f4bea384640 10 monclient: monmap: 2024-06-03T09:06:15.653 INFO:teuthology.orchestra.run.smithi088.stderr:epoch 0 2024-06-03T09:06:15.653 INFO:teuthology.orchestra.run.smithi088.stderr:fsid ec3c3fdd-3ac6-4d7d-9412-683130ed8b04 2024-06-03T09:06:15.653 INFO:teuthology.orchestra.run.smithi088.stderr:last_changed 2024-06-03T09:06:15.653563+0000 2024-06-03T09:06:15.653 INFO:teuthology.orchestra.run.smithi088.stderr:created 2024-06-03T09:06:15.653563+0000 2024-06-03T09:06:15.653 INFO:teuthology.orchestra.run.smithi088.stderr:min_mon_release 0 (unknown) 2024-06-03T09:06:15.653 INFO:teuthology.orchestra.run.smithi088.stderr:election_strategy: 3 2024-06-03T09:06:15.653 INFO:teuthology.orchestra.run.smithi088.stderr:0: v1:172.21.15.88:6789/0 mon.noname-a 2024-06-03T09:06:15.653 INFO:teuthology.orchestra.run.smithi088.stderr:1: v1:172.21.15.192:6789/0 mon.noname-c 2024-06-03T09:06:15.653 INFO:teuthology.orchestra.run.smithi088.stderr:2: v1:172.21.15.88:6790/0 mon.noname-b 2024-06-03T09:06:15.653 INFO:teuthology.orchestra.run.smithi088.stderr: 2024-06-03T09:06:15.654 INFO:teuthology.orchestra.run.smithi088.stderr:2024-06-03T09:06:15.653+0000 7f4bea384640 1 Processor -- start 2024-06-03T09:06:15.654 INFO:teuthology.orchestra.run.smithi088.stderr:2024-06-03T09:06:15.653+0000 7f4bea384640 1 -- start start 2024-06-03T09:06:15.655 INFO:teuthology.orchestra.run.smithi088.stderr:2024-06-03T09:06:15.653+0000 7f4bea384640 10 monclient: init 2024-06-03T09:06:15.655 INFO:teuthology.orchestra.run.smithi088.stderr:2024-06-03T09:06:15.653+0000 7f4bea384640 10 monclient: _reopen_session rank -1 2024-06-03T09:06:15.655 INFO:teuthology.orchestra.run.smithi088.stderr:2024-06-03T09:06:15.653+0000 7f4bea384640 10 monclient: _add_conns ranks=[1,2,0] 2024-06-03T09:06:15.655 INFO:teuthology.orchestra.run.smithi088.stderr:2024-06-03T09:06:15.653+0000 7f4bea384640 10 monclient(hunting): picked mon.noname-c con 0x7f4be406fc70 addr v1:172.21.15.192:6789/0 2024-06-03T09:06:15.655 INFO:teuthology.orchestra.run.smithi088.stderr:2024-06-03T09:06:15.653+0000 7f4bea384640 10 monclient(hunting): picked mon.noname-b con 0x7f4be40ad920 addr v1:172.21.15.88:6790/0 2024-06-03T09:06:15.655 INFO:teuthology.orchestra.run.smithi088.stderr:2024-06-03T09:06:15.653+0000 7f4bea384640 10 monclient(hunting): picked mon.noname-a con 0x7f4be41393f0 addr v1:172.21.15.88:6789/0 2024-06-03T09:06:15.655 INFO:teuthology.orchestra.run.smithi088.stderr:2024-06-03T09:06:15.653+0000 7f4bea384640 1 -- --> v1:172.21.15.88:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f4be4070420 con 0x7f4be41393f0 2024-06-03T09:06:15.655 INFO:teuthology.orchestra.run.smithi088.stderr:2024-06-03T09:06:15.653+0000 7f4bea384640 1 -- --> v1:172.21.15.192:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f4be4071050 con 0x7f4be406fc70 2024-06-03T09:06:15.655 INFO:teuthology.orchestra.run.smithi088.stderr:2024-06-03T09:06:15.653+0000 7f4bea384640 1 -- --> v1:172.21.15.88:6790/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f4be40add60 con 0x7f4be40ad920 2024-06-03T09:06:15.655 INFO:teuthology.orchestra.run.smithi088.stderr:2024-06-03T09:06:15.653+0000 7f4bea384640 10 monclient(hunting): _renew_subs 2024-06-03T09:06:15.655 INFO:teuthology.orchestra.run.smithi088.stderr:2024-06-03T09:06:15.653+0000 7f4bea384640 10 monclient(hunting): authenticate will time out at 3031.463894s 2024-06-03T09:06:15.655 INFO:teuthology.orchestra.run.smithi088.stderr:2024-06-03T09:06:15.654+0000 7f4be8921640 1 --1- >> v1:172.21.15.88:6789/0 conn(0x7f4be41393f0 0x7f4be414ac50 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.88:6789/0 says I am v1:172.21.15.88:53924/0 (socket says 172.21.15.88:53924) 2024-06-03T09:06:15.655 INFO:teuthology.orchestra.run.smithi088.stderr:2024-06-03T09:06:15.654+0000 7f4be8921640 1 -- 172.21.15.88:0/2281387306 learned_addr learned my addr 172.21.15.88:0/2281387306 (peer_addr_for_me v1:172.21.15.88:0/0) 2024-06-03T09:06:15.655 INFO:teuthology.orchestra.run.smithi088.stderr:2024-06-03T09:06:15.654+0000 7f4be37fe640 1 --1- >> v1:172.21.15.88:6790/0 conn(0x7f4be40ad920 0x7f4be4138ce0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.88:6790/0 says I am v1:172.21.15.88:60418/0 (socket says 172.21.15.88:60418) 2024-06-03T09:06:15.656 INFO:teuthology.orchestra.run.smithi088.stderr:2024-06-03T09:06:15.656+0000 7f4be2ffd640 1 -- 172.21.15.88:0/2281387306 <== mon.1 v1:172.21.15.192:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 1993320034 0 0) 0x7f4be4071050 con 0x7f4be406fc70 2024-06-03T09:06:15.656 INFO:teuthology.orchestra.run.smithi088.stderr:2024-06-03T09:06:15.656+0000 7f4be2ffd640 10 monclient(hunting): _negotiate 2024-06-03T09:06:15.656 INFO:teuthology.orchestra.run.smithi088.stderr:2024-06-03T09:06:15.656+0000 7f4be2ffd640 10 monclient(hunting): _init_auth method 2 2024-06-03T09:06:15.656 INFO:teuthology.orchestra.run.smithi088.stderr:2024-06-03T09:06:15.656+0000 7f4be2ffd640 10 monclient(hunting): _init_auth creating new auth 2024-06-03T09:06:15.656 INFO:teuthology.orchestra.run.smithi088.stderr:2024-06-03T09:06:15.656+0000 7f4be2ffd640 10 monclient(hunting): my global_id is 30139 2024-06-03T09:06:15.657 INFO:teuthology.orchestra.run.smithi088.stderr:2024-06-03T09:06:15.656+0000 7f4be2ffd640 1 -- 172.21.15.88:0/2281387306 --> v1:172.21.15.192:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f4bcc003520 con 0x7f4be406fc70 2024-06-03T09:06:15.657 INFO:teuthology.orchestra.run.smithi088.stderr:2024-06-03T09:06:15.657+0000 7f4be2ffd640 1 -- 172.21.15.88:0/2281387306 <== mon.0 v1:172.21.15.88:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 3775204059 0 0) 0x7f4be4070420 con 0x7f4be41393f0 2024-06-03T09:06:15.657 INFO:teuthology.orchestra.run.smithi088.stderr:2024-06-03T09:06:15.657+0000 7f4be2ffd640 10 monclient(hunting): _negotiate 2024-06-03T09:06:15.657 INFO:teuthology.orchestra.run.smithi088.stderr:2024-06-03T09:06:15.657+0000 7f4be2ffd640 10 monclient(hunting): _init_auth method 2 2024-06-03T09:06:15.657 INFO:teuthology.orchestra.run.smithi088.stderr:2024-06-03T09:06:15.657+0000 7f4be2ffd640 10 monclient(hunting): _init_auth creating new auth 2024-06-03T09:06:15.658 INFO:teuthology.orchestra.run.smithi088.stderr:2024-06-03T09:06:15.657+0000 7f4be2ffd640 10 monclient(hunting): my global_id is 30561 2024-06-03T09:06:15.658 INFO:teuthology.orchestra.run.smithi088.stderr:2024-06-03T09:06:15.657+0000 7f4be2ffd640 1 -- 172.21.15.88:0/2281387306 --> v1:172.21.15.88:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f4be4071050 con 0x7f4be41393f0 2024-06-03T09:06:15.658 INFO:teuthology.orchestra.run.smithi088.stderr:2024-06-03T09:06:15.657+0000 7f4be2ffd640 1 -- 172.21.15.88:0/2281387306 <== mon.2 v1:172.21.15.88:6790/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 960260022 0 0) 0x7f4be40add60 con 0x7f4be40ad920 2024-06-03T09:06:15.658 INFO:teuthology.orchestra.run.smithi088.stderr:2024-06-03T09:06:15.657+0000 7f4be2ffd640 10 monclient(hunting): _negotiate 2024-06-03T09:06:15.658 INFO:teuthology.orchestra.run.smithi088.stderr:2024-06-03T09:06:15.657+0000 7f4be2ffd640 10 monclient(hunting): _init_auth method 2 2024-06-03T09:06:15.658 INFO:teuthology.orchestra.run.smithi088.stderr:2024-06-03T09:06:15.657+0000 7f4be2ffd640 10 monclient(hunting): _init_auth creating new auth 2024-06-03T09:06:15.658 INFO:teuthology.orchestra.run.smithi088.stderr:2024-06-03T09:06:15.657+0000 7f4be2ffd640 10 monclient(hunting): my global_id is 30062 2024-06-03T09:06:15.658 INFO:teuthology.orchestra.run.smithi088.stderr:2024-06-03T09:06:15.657+0000 7f4be2ffd640 1 -- 172.21.15.88:0/2281387306 --> v1:172.21.15.88:6790/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f4be4070420 con 0x7f4be40ad920 2024-06-03T09:06:15.660 INFO:teuthology.orchestra.run.smithi088.stderr:2024-06-03T09:06:15.659+0000 7f4be2ffd640 1 -- 172.21.15.88:0/2281387306 <== mon.1 v1:172.21.15.192:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 214+0+0 (unknown 2684556146 0 0) 0x7f4bcc003520 con 0x7f4be406fc70 2024-06-03T09:06:15.660 INFO:teuthology.orchestra.run.smithi088.stderr:2024-06-03T09:06:15.659+0000 7f4be2ffd640 1 -- 172.21.15.88:0/2281387306 >> v1:172.21.15.88:6790/0 conn(0x7f4be40ad920 legacy=0x7f4be4138ce0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-06-03T09:06:15.660 INFO:teuthology.orchestra.run.smithi088.stderr:2024-06-03T09:06:15.659+0000 7f4be2ffd640 1 -- 172.21.15.88:0/2281387306 >> v1:172.21.15.88:6789/0 conn(0x7f4be41393f0 legacy=0x7f4be414ac50 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-06-03T09:06:15.660 INFO:teuthology.orchestra.run.smithi088.stderr:2024-06-03T09:06:15.659+0000 7f4be2ffd640 10 monclient: _finish_hunting 0 2024-06-03T09:06:15.660 INFO:teuthology.orchestra.run.smithi088.stderr:2024-06-03T09:06:15.659+0000 7f4be2ffd640 1 monclient: found mon.noname-c 2024-06-03T09:06:15.660 INFO:teuthology.orchestra.run.smithi088.stderr:2024-06-03T09:06:15.659+0000 7f4be2ffd640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-06-03T09:06:15.660 INFO:teuthology.orchestra.run.smithi088.stderr:2024-06-03T09:06:15.659+0000 7f4be2ffd640 10 monclient: _send_mon_message to mon.noname-c at v1:172.21.15.192:6789/0 2024-06-03T09:06:15.660 INFO:teuthology.orchestra.run.smithi088.stderr:2024-06-03T09:06:15.659+0000 7f4be2ffd640 1 -- 172.21.15.88:0/2281387306 --> v1:172.21.15.192:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f4be4071be0 con 0x7f4be406fc70 2024-06-03T09:06:15.660 INFO:teuthology.orchestra.run.smithi088.stderr:2024-06-03T09:06:15.659+0000 7f4be2ffd640 10 monclient: _finish_auth 0 2024-06-03T09:06:15.660 INFO:teuthology.orchestra.run.smithi088.stderr:2024-06-03T09:06:15.659+0000 7f4be2ffd640 10 monclient: _check_auth_tickets 2024-06-03T09:06:15.661 INFO:teuthology.orchestra.run.smithi088.stderr:2024-06-03T09:06:15.659+0000 7f4be2ffd640 20 monclient: _check_auth_rotating not needed by client.admin 2024-06-03T09:06:15.661 INFO:teuthology.orchestra.run.smithi088.stderr:2024-06-03T09:06:15.659+0000 7f4be2ffd640 1 -- 172.21.15.88:0/2281387306 <== mon.1 v1:172.21.15.192:6789/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 2234374065 0 0) 0x7f4bd4001540 con 0x7f4be406fc70 2024-06-03T09:06:15.661 INFO:teuthology.orchestra.run.smithi088.stderr:2024-06-03T09:06:15.659+0000 7f4be2ffd640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-06-03T09:06:15.661 INFO:teuthology.orchestra.run.smithi088.stderr:2024-06-03T09:06:15.659+0000 7f4be2ffd640 10 monclient: got monmap 1 from mon.noname-c (according to old e1) 2024-06-03T09:06:15.661 INFO:teuthology.orchestra.run.smithi088.stderr:2024-06-03T09:06:15.659+0000 7f4be2ffd640 10 monclient: dump: 2024-06-03T09:06:15.661 INFO:teuthology.orchestra.run.smithi088.stderr:epoch 1 2024-06-03T09:06:15.661 INFO:teuthology.orchestra.run.smithi088.stderr:fsid ec3c3fdd-3ac6-4d7d-9412-683130ed8b04 2024-06-03T09:06:15.661 INFO:teuthology.orchestra.run.smithi088.stderr:last_changed 2024-06-03T08:34:31.885125+0000 2024-06-03T09:06:15.661 INFO:teuthology.orchestra.run.smithi088.stderr:created 2024-06-03T08:34:31.885125+0000 2024-06-03T09:06:15.661 INFO:teuthology.orchestra.run.smithi088.stderr:min_mon_release 18 (reef) 2024-06-03T09:06:15.661 INFO:teuthology.orchestra.run.smithi088.stderr:election_strategy: 3 2024-06-03T09:06:15.661 INFO:teuthology.orchestra.run.smithi088.stderr:0: v1:172.21.15.88:6789/0 mon.a 2024-06-03T09:06:15.661 INFO:teuthology.orchestra.run.smithi088.stderr:1: v1:172.21.15.192:6789/0 mon.b 2024-06-03T09:06:15.661 INFO:teuthology.orchestra.run.smithi088.stderr:2: v1:172.21.15.88:6790/0 mon.c 2024-06-03T09:06:15.662 INFO:teuthology.orchestra.run.smithi088.stderr: 2024-06-03T09:06:15.662 INFO:teuthology.orchestra.run.smithi088.stderr:2024-06-03T09:06:15.660+0000 7f4bea384640 5 monclient: authenticate success, global_id 30139 2024-06-03T09:06:15.662 INFO:teuthology.orchestra.run.smithi088.stderr:2024-06-03T09:06:15.660+0000 7f4bea384640 20 monclient: get_monmap_and_config waiting for monmap|config 2024-06-03T09:06:15.664 INFO:teuthology.orchestra.run.smithi088.stderr:2024-06-03T09:06:15.661+0000 7f4be2ffd640 1 -- 172.21.15.88:0/2281387306 <== mon.1 v1:172.21.15.192:6789/0 4 ==== config(0 keys) v1 ==== 4+0+0 (unknown 0 0 0) 0x7f4bd4001760 con 0x7f4be406fc70 2024-06-03T09:06:15.665 INFO:teuthology.orchestra.run.smithi088.stderr:2024-06-03T09:06:15.661+0000 7f4be2ffd640 10 monclient: handle_config config(0 keys) v1 2024-06-03T09:06:15.665 INFO:teuthology.orchestra.run.smithi088.stderr:2024-06-03T09:06:15.661+0000 7f4be2ffd640 1 -- 172.21.15.88:0/2281387306 <== mon.1 v1:172.21.15.192:6789/0 5 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 2234374065 0 0) 0x7f4bd4001a70 con 0x7f4be406fc70 2024-06-03T09:06:15.665 INFO:teuthology.orchestra.run.smithi088.stderr:2024-06-03T09:06:15.661+0000 7f4be2ffd640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-06-03T09:06:15.665 INFO:teuthology.orchestra.run.smithi088.stderr:2024-06-03T09:06:15.661+0000 7f4be2ffd640 10 monclient: got monmap 1 from mon.b (according to old e1) 2024-06-03T09:06:15.665 INFO:teuthology.orchestra.run.smithi088.stderr:2024-06-03T09:06:15.661+0000 7f4be2ffd640 10 monclient: dump: 2024-06-03T09:06:15.665 INFO:teuthology.orchestra.run.smithi088.stderr:epoch 1 2024-06-03T09:06:15.665 INFO:teuthology.orchestra.run.smithi088.stderr:fsid ec3c3fdd-3ac6-4d7d-9412-683130ed8b04 2024-06-03T09:06:15.665 INFO:teuthology.orchestra.run.smithi088.stderr:last_changed 2024-06-03T08:34:31.885125+0000 2024-06-03T09:06:15.665 INFO:teuthology.orchestra.run.smithi088.stderr:created 2024-06-03T08:34:31.885125+0000 2024-06-03T09:06:15.665 INFO:teuthology.orchestra.run.smithi088.stderr:min_mon_release 18 (reef) 2024-06-03T09:06:15.665 INFO:teuthology.orchestra.run.smithi088.stderr:election_strategy: 3 2024-06-03T09:06:15.666 INFO:teuthology.orchestra.run.smithi088.stderr:0: v1:172.21.15.88:6789/0 mon.a 2024-06-03T09:06:15.666 INFO:teuthology.orchestra.run.smithi088.stderr:1: v1:172.21.15.192:6789/0 mon.b 2024-06-03T09:06:15.666 INFO:teuthology.orchestra.run.smithi088.stderr:2: v1:172.21.15.88:6790/0 mon.c 2024-06-03T09:06:15.666 INFO:teuthology.orchestra.run.smithi088.stderr: 2024-06-03T09:06:15.666 INFO:teuthology.orchestra.run.smithi088.stderr:2024-06-03T09:06:15.661+0000 7f4bea384640 10 monclient: get_monmap_and_config success 2024-06-03T09:06:15.666 INFO:teuthology.orchestra.run.smithi088.stderr:2024-06-03T09:06:15.661+0000 7f4bea384640 4 set_mon_vals no callback set 2024-06-03T09:06:15.666 INFO:teuthology.orchestra.run.smithi088.stderr:2024-06-03T09:06:15.662+0000 7f4bea384640 10 monclient: shutdown 2024-06-03T09:06:15.666 INFO:teuthology.orchestra.run.smithi088.stderr:2024-06-03T09:06:15.662+0000 7f4bea384640 20 monclient: shutdown discarding 0 pending message(s) 2024-06-03T09:06:15.666 INFO:teuthology.orchestra.run.smithi088.stderr:2024-06-03T09:06:15.662+0000 7f4bea384640 1 -- 172.21.15.88:0/2281387306 >> v1:172.21.15.192:6789/0 conn(0x7f4be406fc70 legacy=0x7f4be413fbb0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-06-03T09:06:15.666 INFO:teuthology.orchestra.run.smithi088.stderr:2024-06-03T09:06:15.662+0000 7f4bea384640 1 -- 172.21.15.88:0/2281387306 shutdown_connections 2024-06-03T09:06:15.666 INFO:teuthology.orchestra.run.smithi088.stderr:2024-06-03T09:06:15.662+0000 7f4bea384640 1 -- 172.21.15.88:0/2281387306 >> 172.21.15.88:0/2281387306 conn(0x7f4be40b2f20 msgr2=0x7f4be40b3300 unknown :-1 s=STATE_NONE l=0).mark_down 2024-06-03T09:06:15.666 INFO:teuthology.orchestra.run.smithi088.stderr:2024-06-03T09:06:15.662+0000 7f4bea384640 1 -- 172.21.15.88:0/2281387306 shutdown_connections 2024-06-03T09:06:15.666 INFO:teuthology.orchestra.run.smithi088.stderr:2024-06-03T09:06:15.662+0000 7f4bea384640 1 -- 172.21.15.88:0/2281387306 wait complete. 2024-06-03T09:06:15.667 INFO:teuthology.orchestra.run.smithi088.stderr:2024-06-03T09:06:15.663+0000 7f4bea384640 10 monclient: build_initial_monmap 2024-06-03T09:06:15.667 INFO:teuthology.orchestra.run.smithi088.stderr:2024-06-03T09:06:15.663+0000 7f4bea384640 10 monclient: monmap: 2024-06-03T09:06:15.667 INFO:teuthology.orchestra.run.smithi088.stderr:epoch 0 2024-06-03T09:06:15.667 INFO:teuthology.orchestra.run.smithi088.stderr:fsid 00000000-0000-0000-0000-000000000000 2024-06-03T09:06:15.667 INFO:teuthology.orchestra.run.smithi088.stderr:last_changed 0.000000 2024-06-03T09:06:15.667 INFO:teuthology.orchestra.run.smithi088.stderr:created 0.000000 2024-06-03T09:06:15.667 INFO:teuthology.orchestra.run.smithi088.stderr:min_mon_release 0 (unknown) 2024-06-03T09:06:15.667 INFO:teuthology.orchestra.run.smithi088.stderr:election_strategy: 1 2024-06-03T09:06:15.667 INFO:teuthology.orchestra.run.smithi088.stderr:0: v1:172.21.15.88:6789/0 mon.noname-a 2024-06-03T09:06:15.667 INFO:teuthology.orchestra.run.smithi088.stderr:1: v1:172.21.15.192:6789/0 mon.noname-b 2024-06-03T09:06:15.667 INFO:teuthology.orchestra.run.smithi088.stderr:2: v1:172.21.15.88:6790/0 mon.noname-c 2024-06-03T09:06:15.667 INFO:teuthology.orchestra.run.smithi088.stderr: 2024-06-03T09:06:15.667 INFO:teuthology.orchestra.run.smithi088.stderr:2024-06-03T09:06:15.663+0000 7f4bea384640 1 librados: starting msgr at 2024-06-03T09:06:15.667 INFO:teuthology.orchestra.run.smithi088.stderr:2024-06-03T09:06:15.663+0000 7f4bea384640 1 librados: starting objecter 2024-06-03T09:06:15.668 INFO:teuthology.orchestra.run.smithi088.stderr:2024-06-03T09:06:15.663+0000 7f4bea384640 1 Processor -- start 2024-06-03T09:06:15.668 INFO:teuthology.orchestra.run.smithi088.stderr:2024-06-03T09:06:15.663+0000 7f4bea384640 1 -- start start 2024-06-03T09:06:15.668 INFO:teuthology.orchestra.run.smithi088.stderr:2024-06-03T09:06:15.663+0000 7f4bea384640 1 librados: setting wanted keys 2024-06-03T09:06:15.668 INFO:teuthology.orchestra.run.smithi088.stderr:2024-06-03T09:06:15.663+0000 7f4bea384640 1 librados: calling monclient init 2024-06-03T09:06:15.668 INFO:teuthology.orchestra.run.smithi088.stderr:2024-06-03T09:06:15.663+0000 7f4bea384640 10 monclient: init 2024-06-03T09:06:15.668 INFO:teuthology.orchestra.run.smithi088.stderr:2024-06-03T09:06:15.664+0000 7f4bea384640 10 monclient: _reopen_session rank -1 2024-06-03T09:06:15.668 INFO:teuthology.orchestra.run.smithi088.stderr:2024-06-03T09:06:15.664+0000 7f4bea384640 10 monclient: _add_conns ranks=[2,0,1] 2024-06-03T09:06:15.668 INFO:teuthology.orchestra.run.smithi088.stderr:2024-06-03T09:06:15.664+0000 7f4bea384640 10 monclient(hunting): picked mon.noname-c con 0x7f4be406fc70 addr v1:172.21.15.88:6790/0 2024-06-03T09:06:15.668 INFO:teuthology.orchestra.run.smithi088.stderr:2024-06-03T09:06:15.664+0000 7f4bea384640 10 monclient(hunting): picked mon.noname-a con 0x7f4be40ad920 addr v1:172.21.15.88:6789/0 2024-06-03T09:06:15.668 INFO:teuthology.orchestra.run.smithi088.stderr:2024-06-03T09:06:15.664+0000 7f4bea384640 10 monclient(hunting): picked mon.noname-b con 0x7f4be41393f0 addr v1:172.21.15.192:6789/0 2024-06-03T09:06:15.668 INFO:teuthology.orchestra.run.smithi088.stderr:2024-06-03T09:06:15.664+0000 7f4bea384640 1 -- --> v1:172.21.15.88:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f4be41da710 con 0x7f4be40ad920 2024-06-03T09:06:15.668 INFO:teuthology.orchestra.run.smithi088.stderr:2024-06-03T09:06:15.664+0000 7f4bea384640 1 -- --> v1:172.21.15.192:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f4be41da990 con 0x7f4be41393f0 2024-06-03T09:06:15.668 INFO:teuthology.orchestra.run.smithi088.stderr:2024-06-03T09:06:15.664+0000 7f4bea384640 1 -- --> v1:172.21.15.88:6790/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f4be40ac600 con 0x7f4be406fc70 2024-06-03T09:06:15.668 INFO:teuthology.orchestra.run.smithi088.stderr:2024-06-03T09:06:15.664+0000 7f4bea384640 10 monclient(hunting): _renew_subs 2024-06-03T09:06:15.668 INFO:teuthology.orchestra.run.smithi088.stderr:2024-06-03T09:06:15.664+0000 7f4bea384640 10 monclient(hunting): authenticate will time out at 3031.474312s 2024-06-03T09:06:15.669 INFO:teuthology.orchestra.run.smithi088.stderr:2024-06-03T09:06:15.664+0000 7f4be37fe640 1 --1- >> v1:172.21.15.88:6789/0 conn(0x7f4be40ad920 0x7f4be40aae50 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=1).handle_server_banner_and_identify peer v1:172.21.15.88:6789/0 says I am v1:172.21.15.88:53926/0 (socket says 172.21.15.88:53926) 2024-06-03T09:06:15.669 INFO:teuthology.orchestra.run.smithi088.stderr:2024-06-03T09:06:15.664+0000 7f4be37fe640 1 -- 172.21.15.88:0/193553391 learned_addr learned my addr 172.21.15.88:0/193553391 (peer_addr_for_me v1:172.21.15.88:0/0) 2024-06-03T09:06:15.669 INFO:teuthology.orchestra.run.smithi088.stderr:2024-06-03T09:06:15.665+0000 7f4be0ff9640 10 client.?.objecter ms_handle_connect 0x7f4be40ad920 2024-06-03T09:06:15.669 INFO:teuthology.orchestra.run.smithi088.stderr:2024-06-03T09:06:15.665+0000 7f4be0ff9640 10 client.?.objecter resend_mon_ops 2024-06-03T09:06:15.669 INFO:teuthology.orchestra.run.smithi088.stderr:2024-06-03T09:06:15.665+0000 7f4be0ff9640 10 client.?.objecter ms_handle_connect 0x7f4be41393f0 2024-06-03T09:06:15.669 INFO:teuthology.orchestra.run.smithi088.stderr:2024-06-03T09:06:15.665+0000 7f4be0ff9640 10 client.?.objecter resend_mon_ops 2024-06-03T09:06:15.669 INFO:teuthology.orchestra.run.smithi088.stderr:2024-06-03T09:06:15.665+0000 7f4be0ff9640 10 client.?.objecter ms_handle_connect 0x7f4be406fc70 2024-06-03T09:06:15.669 INFO:teuthology.orchestra.run.smithi088.stderr:2024-06-03T09:06:15.665+0000 7f4be0ff9640 10 client.?.objecter resend_mon_ops 2024-06-03T09:06:15.670 INFO:teuthology.orchestra.run.smithi088.stderr:2024-06-03T09:06:15.666+0000 7f4be0ff9640 1 -- 172.21.15.88:0/193553391 <== mon.1 v1:172.21.15.192:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 2469568908 0 0) 0x7f4be41da990 con 0x7f4be41393f0 2024-06-03T09:06:15.670 INFO:teuthology.orchestra.run.smithi088.stderr:2024-06-03T09:06:15.666+0000 7f4be0ff9640 10 monclient(hunting): _negotiate 2024-06-03T09:06:15.670 INFO:teuthology.orchestra.run.smithi088.stderr:2024-06-03T09:06:15.666+0000 7f4be0ff9640 10 monclient(hunting): _init_auth method 2 2024-06-03T09:06:15.670 INFO:teuthology.orchestra.run.smithi088.stderr:2024-06-03T09:06:15.666+0000 7f4be0ff9640 10 monclient(hunting): _init_auth creating new auth 2024-06-03T09:06:15.670 INFO:teuthology.orchestra.run.smithi088.stderr:2024-06-03T09:06:15.666+0000 7f4be0ff9640 10 monclient(hunting): my global_id is 30142 2024-06-03T09:06:15.670 INFO:teuthology.orchestra.run.smithi088.stderr:2024-06-03T09:06:15.666+0000 7f4be0ff9640 1 -- 172.21.15.88:0/193553391 --> v1:172.21.15.192:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f4bd0004eb0 con 0x7f4be41393f0 2024-06-03T09:06:15.671 INFO:teuthology.orchestra.run.smithi088.stderr:2024-06-03T09:06:15.667+0000 7f4be0ff9640 1 -- 172.21.15.88:0/193553391 <== mon.0 v1:172.21.15.88:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 933691766 0 0) 0x7f4be41da710 con 0x7f4be40ad920 2024-06-03T09:06:15.671 INFO:teuthology.orchestra.run.smithi088.stderr:2024-06-03T09:06:15.667+0000 7f4be0ff9640 10 monclient(hunting): _negotiate 2024-06-03T09:06:15.671 INFO:teuthology.orchestra.run.smithi088.stderr:2024-06-03T09:06:15.667+0000 7f4be0ff9640 10 monclient(hunting): _init_auth method 2 2024-06-03T09:06:15.671 INFO:teuthology.orchestra.run.smithi088.stderr:2024-06-03T09:06:15.667+0000 7f4be0ff9640 10 monclient(hunting): _init_auth creating new auth 2024-06-03T09:06:15.671 INFO:teuthology.orchestra.run.smithi088.stderr:2024-06-03T09:06:15.667+0000 7f4be0ff9640 10 monclient(hunting): my global_id is 30564 2024-06-03T09:06:15.671 INFO:teuthology.orchestra.run.smithi088.stderr:2024-06-03T09:06:15.667+0000 7f4be0ff9640 1 -- 172.21.15.88:0/193553391 --> v1:172.21.15.88:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f4be41da990 con 0x7f4be40ad920 2024-06-03T09:06:15.671 INFO:teuthology.orchestra.run.smithi088.stderr:2024-06-03T09:06:15.667+0000 7f4be0ff9640 1 -- 172.21.15.88:0/193553391 <== mon.2 v1:172.21.15.88:6790/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 2136851174 0 0) 0x7f4be40ac600 con 0x7f4be406fc70 2024-06-03T09:06:15.671 INFO:teuthology.orchestra.run.smithi088.stderr:2024-06-03T09:06:15.667+0000 7f4be0ff9640 10 monclient(hunting): _negotiate 2024-06-03T09:06:15.671 INFO:teuthology.orchestra.run.smithi088.stderr:2024-06-03T09:06:15.667+0000 7f4be0ff9640 10 monclient(hunting): _init_auth method 2 2024-06-03T09:06:15.671 INFO:teuthology.orchestra.run.smithi088.stderr:2024-06-03T09:06:15.667+0000 7f4be0ff9640 10 monclient(hunting): _init_auth creating new auth 2024-06-03T09:06:15.671 INFO:teuthology.orchestra.run.smithi088.stderr:2024-06-03T09:06:15.667+0000 7f4be0ff9640 10 monclient(hunting): my global_id is 30065 2024-06-03T09:06:15.671 INFO:teuthology.orchestra.run.smithi088.stderr:2024-06-03T09:06:15.667+0000 7f4be0ff9640 1 -- 172.21.15.88:0/193553391 --> v1:172.21.15.88:6790/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f4be41da710 con 0x7f4be406fc70 2024-06-03T09:06:15.672 INFO:teuthology.orchestra.run.smithi088.stderr:2024-06-03T09:06:15.672+0000 7f4be0ff9640 1 -- 172.21.15.88:0/193553391 <== mon.0 v1:172.21.15.88:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 764+0+0 (unknown 2644071714 0 0) 0x7f4be41da990 con 0x7f4be40ad920 2024-06-03T09:06:15.672 INFO:teuthology.orchestra.run.smithi088.stderr:2024-06-03T09:06:15.672+0000 7f4be0ff9640 1 -- 172.21.15.88:0/193553391 --> v1:172.21.15.88:6789/0 -- auth(proto 2 165 bytes epoch 0) v1 -- 0x7f4be40ac600 con 0x7f4be40ad920 2024-06-03T09:06:15.673 INFO:teuthology.orchestra.run.smithi088.stderr:2024-06-03T09:06:15.672+0000 7f4be0ff9640 1 -- 172.21.15.88:0/193553391 <== mon.0 v1:172.21.15.88:6789/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 2234374065 0 0) 0x7f4bd80018f0 con 0x7f4be40ad920 2024-06-03T09:06:15.673 INFO:teuthology.orchestra.run.smithi088.stderr:2024-06-03T09:06:15.672+0000 7f4be0ff9640 10 monclient(hunting): handle_monmap mon_map magic: 0 v1 2024-06-03T09:06:15.673 INFO:teuthology.orchestra.run.smithi088.stderr:2024-06-03T09:06:15.672+0000 7f4be0ff9640 10 monclient(hunting): got monmap 1 from mon.noname-a (according to old e1) 2024-06-03T09:06:15.673 INFO:teuthology.orchestra.run.smithi088.stderr:2024-06-03T09:06:15.672+0000 7f4be0ff9640 10 monclient(hunting): dump: 2024-06-03T09:06:15.673 INFO:teuthology.orchestra.run.smithi088.stderr:epoch 1 2024-06-03T09:06:15.673 INFO:teuthology.orchestra.run.smithi088.stderr:fsid ec3c3fdd-3ac6-4d7d-9412-683130ed8b04 2024-06-03T09:06:15.673 INFO:teuthology.orchestra.run.smithi088.stderr:last_changed 2024-06-03T08:34:31.885125+0000 2024-06-03T09:06:15.673 INFO:teuthology.orchestra.run.smithi088.stderr:created 2024-06-03T08:34:31.885125+0000 2024-06-03T09:06:15.673 INFO:teuthology.orchestra.run.smithi088.stderr:min_mon_release 18 (reef) 2024-06-03T09:06:15.673 INFO:teuthology.orchestra.run.smithi088.stderr:election_strategy: 3 2024-06-03T09:06:15.673 INFO:teuthology.orchestra.run.smithi088.stderr:0: v1:172.21.15.88:6789/0 mon.a 2024-06-03T09:06:15.673 INFO:teuthology.orchestra.run.smithi088.stderr:1: v1:172.21.15.192:6789/0 mon.b 2024-06-03T09:06:15.673 INFO:teuthology.orchestra.run.smithi088.stderr:2: v1:172.21.15.88:6790/0 mon.c 2024-06-03T09:06:15.674 INFO:teuthology.orchestra.run.smithi088.stderr: 2024-06-03T09:06:15.674 INFO:teuthology.orchestra.run.smithi088.stderr:2024-06-03T09:06:15.672+0000 7f4be0ff9640 10 monclient(hunting): _finish_auth 0 2024-06-03T09:06:15.674 INFO:teuthology.orchestra.run.smithi088.stderr:2024-06-03T09:06:15.672+0000 7f4be0ff9640 1 -- 172.21.15.88:0/193553391 <== mon.2 v1:172.21.15.88:6790/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 764+0+0 (unknown 1065130177 0 0) 0x7f4be41da710 con 0x7f4be406fc70 2024-06-03T09:06:15.674 INFO:teuthology.orchestra.run.smithi088.stderr:2024-06-03T09:06:15.672+0000 7f4be0ff9640 1 -- 172.21.15.88:0/193553391 --> v1:172.21.15.88:6790/0 -- auth(proto 2 165 bytes epoch 0) v1 -- 0x7f4be41da990 con 0x7f4be406fc70 2024-06-03T09:06:15.674 INFO:teuthology.orchestra.run.smithi088.stderr:2024-06-03T09:06:15.672+0000 7f4be0ff9640 1 -- 172.21.15.88:0/193553391 <== mon.2 v1:172.21.15.88:6790/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 2234374065 0 0) 0x7f4bd4001a70 con 0x7f4be406fc70 2024-06-03T09:06:15.674 INFO:teuthology.orchestra.run.smithi088.stderr:2024-06-03T09:06:15.672+0000 7f4be0ff9640 10 monclient(hunting): handle_monmap mon_map magic: 0 v1 2024-06-03T09:06:15.674 INFO:teuthology.orchestra.run.smithi088.stderr:2024-06-03T09:06:15.672+0000 7f4be0ff9640 10 monclient(hunting): got monmap 1 from mon.c (according to old e1) 2024-06-03T09:06:15.674 INFO:teuthology.orchestra.run.smithi088.stderr:2024-06-03T09:06:15.672+0000 7f4be0ff9640 10 monclient(hunting): dump: 2024-06-03T09:06:15.674 INFO:teuthology.orchestra.run.smithi088.stderr:epoch 1 2024-06-03T09:06:15.674 INFO:teuthology.orchestra.run.smithi088.stderr:fsid ec3c3fdd-3ac6-4d7d-9412-683130ed8b04 2024-06-03T09:06:15.674 INFO:teuthology.orchestra.run.smithi088.stderr:last_changed 2024-06-03T08:34:31.885125+0000 2024-06-03T09:06:15.674 INFO:teuthology.orchestra.run.smithi088.stderr:created 2024-06-03T08:34:31.885125+0000 2024-06-03T09:06:15.674 INFO:teuthology.orchestra.run.smithi088.stderr:min_mon_release 18 (reef) 2024-06-03T09:06:15.674 INFO:teuthology.orchestra.run.smithi088.stderr:election_strategy: 3 2024-06-03T09:06:15.675 INFO:teuthology.orchestra.run.smithi088.stderr:0: v1:172.21.15.88:6789/0 mon.a 2024-06-03T09:06:15.675 INFO:teuthology.orchestra.run.smithi088.stderr:1: v1:172.21.15.192:6789/0 mon.b 2024-06-03T09:06:15.675 INFO:teuthology.orchestra.run.smithi088.stderr:2: v1:172.21.15.88:6790/0 mon.c 2024-06-03T09:06:15.675 INFO:teuthology.orchestra.run.smithi088.stderr: 2024-06-03T09:06:15.676 INFO:teuthology.orchestra.run.smithi088.stderr:2024-06-03T09:06:15.674+0000 7f4be0ff9640 1 -- 172.21.15.88:0/193553391 <== mon.1 v1:172.21.15.192:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 764+0+0 (unknown 2525656705 0 0) 0x7f4bd0004eb0 con 0x7f4be41393f0 2024-06-03T09:06:15.676 INFO:teuthology.orchestra.run.smithi088.stderr:2024-06-03T09:06:15.674+0000 7f4be0ff9640 1 -- 172.21.15.88:0/193553391 --> v1:172.21.15.192:6789/0 -- auth(proto 2 165 bytes epoch 0) v1 -- 0x7f4be41da710 con 0x7f4be41393f0 2024-06-03T09:06:15.676 INFO:teuthology.orchestra.run.smithi088.stderr:2024-06-03T09:06:15.674+0000 7f4be0ff9640 1 -- 172.21.15.88:0/193553391 <== mon.1 v1:172.21.15.192:6789/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 2234374065 0 0) 0x7f4bdc001c30 con 0x7f4be41393f0 2024-06-03T09:06:15.676 INFO:teuthology.orchestra.run.smithi088.stderr:2024-06-03T09:06:15.674+0000 7f4be0ff9640 10 monclient(hunting): handle_monmap mon_map magic: 0 v1 2024-06-03T09:06:15.677 INFO:teuthology.orchestra.run.smithi088.stderr:2024-06-03T09:06:15.674+0000 7f4be0ff9640 10 monclient(hunting): got monmap 1 from mon.b (according to old e1) 2024-06-03T09:06:15.677 INFO:teuthology.orchestra.run.smithi088.stderr:2024-06-03T09:06:15.674+0000 7f4be0ff9640 10 monclient(hunting): dump: 2024-06-03T09:06:15.677 INFO:teuthology.orchestra.run.smithi088.stderr:epoch 1 2024-06-03T09:06:15.677 INFO:teuthology.orchestra.run.smithi088.stderr:fsid ec3c3fdd-3ac6-4d7d-9412-683130ed8b04 2024-06-03T09:06:15.677 INFO:teuthology.orchestra.run.smithi088.stderr:last_changed 2024-06-03T08:34:31.885125+0000 2024-06-03T09:06:15.677 INFO:teuthology.orchestra.run.smithi088.stderr:created 2024-06-03T08:34:31.885125+0000 2024-06-03T09:06:15.677 INFO:teuthology.orchestra.run.smithi088.stderr:min_mon_release 18 (reef) 2024-06-03T09:06:15.677 INFO:teuthology.orchestra.run.smithi088.stderr:election_strategy: 3 2024-06-03T09:06:15.677 INFO:teuthology.orchestra.run.smithi088.stderr:0: v1:172.21.15.88:6789/0 mon.a 2024-06-03T09:06:15.677 INFO:teuthology.orchestra.run.smithi088.stderr:1: v1:172.21.15.192:6789/0 mon.b 2024-06-03T09:06:15.677 INFO:teuthology.orchestra.run.smithi088.stderr:2: v1:172.21.15.88:6790/0 mon.c 2024-06-03T09:06:15.677 INFO:teuthology.orchestra.run.smithi088.stderr: 2024-06-03T09:06:15.677 INFO:teuthology.orchestra.run.smithi088.stderr:2024-06-03T09:06:15.675+0000 7f4be0ff9640 1 -- 172.21.15.88:0/193553391 <== mon.0 v1:172.21.15.88:6789/0 4 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 580+0+0 (unknown 787451937 0 0) 0x7f4be40ac600 con 0x7f4be40ad920 2024-06-03T09:06:15.678 INFO:teuthology.orchestra.run.smithi088.stderr:2024-06-03T09:06:15.675+0000 7f4be0ff9640 1 -- 172.21.15.88:0/193553391 >> v1:172.21.15.88:6790/0 conn(0x7f4be406fc70 legacy=0x7f4be41da000 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-06-03T09:06:15.678 INFO:teuthology.orchestra.run.smithi088.stderr:2024-06-03T09:06:15.675+0000 7f4be0ff9640 1 -- 172.21.15.88:0/193553391 >> v1:172.21.15.192:6789/0 conn(0x7f4be41393f0 legacy=0x7f4be4141070 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-06-03T09:06:15.678 INFO:teuthology.orchestra.run.smithi088.stderr:2024-06-03T09:06:15.675+0000 7f4be0ff9640 10 monclient: _finish_hunting 0 2024-06-03T09:06:15.678 INFO:teuthology.orchestra.run.smithi088.stderr:2024-06-03T09:06:15.675+0000 7f4be0ff9640 1 monclient: found mon.a 2024-06-03T09:06:15.678 INFO:teuthology.orchestra.run.smithi088.stderr:2024-06-03T09:06:15.675+0000 7f4be0ff9640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-06-03T09:06:15.678 INFO:teuthology.orchestra.run.smithi088.stderr:2024-06-03T09:06:15.675+0000 7f4be0ff9640 10 monclient: _send_mon_message to mon.a at v1:172.21.15.88:6789/0 2024-06-03T09:06:15.678 INFO:teuthology.orchestra.run.smithi088.stderr:2024-06-03T09:06:15.675+0000 7f4be0ff9640 1 -- 172.21.15.88:0/193553391 --> v1:172.21.15.88:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f4be40abb60 con 0x7f4be40ad920 2024-06-03T09:06:15.678 INFO:teuthology.orchestra.run.smithi088.stderr:2024-06-03T09:06:15.675+0000 7f4be0ff9640 10 monclient: _finish_auth 0 2024-06-03T09:06:15.678 INFO:teuthology.orchestra.run.smithi088.stderr:2024-06-03T09:06:15.675+0000 7f4be0ff9640 10 monclient: _check_auth_tickets 2024-06-03T09:06:15.678 INFO:teuthology.orchestra.run.smithi088.stderr:2024-06-03T09:06:15.675+0000 7f4be0ff9640 20 monclient: _check_auth_rotating not needed by client.admin 2024-06-03T09:06:15.678 INFO:teuthology.orchestra.run.smithi088.stderr:2024-06-03T09:06:15.675+0000 7f4bea384640 5 monclient: authenticate success, global_id 30564 2024-06-03T09:06:15.678 INFO:teuthology.orchestra.run.smithi088.stderr:2024-06-03T09:06:15.675+0000 7f4bea384640 10 monclient: _renew_subs 2024-06-03T09:06:15.678 INFO:teuthology.orchestra.run.smithi088.stderr:2024-06-03T09:06:15.675+0000 7f4bea384640 10 monclient: _send_mon_message to mon.a at v1:172.21.15.88:6789/0 2024-06-03T09:06:15.678 INFO:teuthology.orchestra.run.smithi088.stderr:2024-06-03T09:06:15.675+0000 7f4bea384640 1 -- 172.21.15.88:0/193553391 --> v1:172.21.15.88:6789/0 -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f4be40abe10 con 0x7f4be40ad920 2024-06-03T09:06:15.679 INFO:teuthology.orchestra.run.smithi088.stderr:2024-06-03T09:06:15.675+0000 7f4bea384640 10 client.30564.objecter _maybe_request_map subscribing (onetime) to next osd map 2024-06-03T09:06:15.679 INFO:teuthology.orchestra.run.smithi088.stderr:2024-06-03T09:06:15.675+0000 7f4bea384640 10 monclient: _renew_subs 2024-06-03T09:06:15.679 INFO:teuthology.orchestra.run.smithi088.stderr:2024-06-03T09:06:15.675+0000 7f4bea384640 10 monclient: _send_mon_message to mon.a at v1:172.21.15.88:6789/0 2024-06-03T09:06:15.679 INFO:teuthology.orchestra.run.smithi088.stderr:2024-06-03T09:06:15.675+0000 7f4bea384640 1 -- 172.21.15.88:0/193553391 --> v1:172.21.15.88:6789/0 -- mon_subscribe({osdmap=0}) v3 -- 0x7f4be40ab740 con 0x7f4be40ad920 2024-06-03T09:06:15.679 INFO:teuthology.orchestra.run.smithi088.stderr:2024-06-03T09:06:15.675+0000 7f4bea384640 1 librados: init done 2024-06-03T09:06:15.679 INFO:teuthology.orchestra.run.smithi088.stderr:2024-06-03T09:06:15.676+0000 7f4bea384640 10 start_mon_command cmd=[{"prefix": "get_command_descriptions"}] 2024-06-03T09:06:15.679 INFO:teuthology.orchestra.run.smithi088.stderr:2024-06-03T09:06:15.676+0000 7f4bea384640 10 monclient: _send_command 1 [{"prefix": "get_command_descriptions"}] 2024-06-03T09:06:15.679 INFO:teuthology.orchestra.run.smithi088.stderr:2024-06-03T09:06:15.676+0000 7f4bea384640 10 monclient: _send_mon_message to mon.a at v1:172.21.15.88:6789/0 2024-06-03T09:06:15.679 INFO:teuthology.orchestra.run.smithi088.stderr:2024-06-03T09:06:15.676+0000 7f4bea384640 1 -- 172.21.15.88:0/193553391 --> v1:172.21.15.88:6789/0 -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f4be41dab20 con 0x7f4be40ad920 2024-06-03T09:06:15.891 INFO:teuthology.orchestra.run.smithi088.stderr:2024-06-03T09:06:15.891+0000 7f4be0ff9640 1 -- 172.21.15.88:0/193553391 <== mon.0 v1:172.21.15.88:6789/0 5 ==== config(0 keys) v1 ==== 4+0+0 (unknown 0 0 0) 0x7f4bd80047e0 con 0x7f4be40ad920 2024-06-03T09:06:15.891 INFO:teuthology.orchestra.run.smithi088.stderr:2024-06-03T09:06:15.891+0000 7f4be0ff9640 10 monclient: handle_config config(0 keys) v1 2024-06-03T09:06:15.891 INFO:teuthology.orchestra.run.smithi088.stderr:2024-06-03T09:06:15.891+0000 7f4be0ff9640 1 -- 172.21.15.88:0/193553391 <== mon.0 v1:172.21.15.88:6789/0 6 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 2234374065 0 0) 0x7f4bd8004af0 con 0x7f4be40ad920 2024-06-03T09:06:15.891 INFO:teuthology.orchestra.run.smithi088.stderr:2024-06-03T09:06:15.891+0000 7f4be0ff9640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-06-03T09:06:15.891 INFO:teuthology.orchestra.run.smithi088.stderr:2024-06-03T09:06:15.891+0000 7f4be0ff9640 10 monclient: got monmap 1 from mon.a (according to old e1) 2024-06-03T09:06:15.891 INFO:teuthology.orchestra.run.smithi088.stderr:2024-06-03T09:06:15.891+0000 7f4be0ff9640 10 monclient: dump: 2024-06-03T09:06:15.891 INFO:teuthology.orchestra.run.smithi088.stderr:epoch 1 2024-06-03T09:06:15.892 INFO:teuthology.orchestra.run.smithi088.stderr:fsid ec3c3fdd-3ac6-4d7d-9412-683130ed8b04 2024-06-03T09:06:15.892 INFO:teuthology.orchestra.run.smithi088.stderr:last_changed 2024-06-03T08:34:31.885125+0000 2024-06-03T09:06:15.892 INFO:teuthology.orchestra.run.smithi088.stderr:created 2024-06-03T08:34:31.885125+0000 2024-06-03T09:06:15.892 INFO:teuthology.orchestra.run.smithi088.stderr:min_mon_release 18 (reef) 2024-06-03T09:06:15.892 INFO:teuthology.orchestra.run.smithi088.stderr:election_strategy: 3 2024-06-03T09:06:15.892 INFO:teuthology.orchestra.run.smithi088.stderr:0: v1:172.21.15.88:6789/0 mon.a 2024-06-03T09:06:15.892 INFO:teuthology.orchestra.run.smithi088.stderr:1: v1:172.21.15.192:6789/0 mon.b 2024-06-03T09:06:15.892 INFO:teuthology.orchestra.run.smithi088.stderr:2: v1:172.21.15.88:6790/0 mon.c 2024-06-03T09:06:15.892 INFO:teuthology.orchestra.run.smithi088.stderr: 2024-06-03T09:06:15.892 INFO:teuthology.orchestra.run.smithi088.stderr:2024-06-03T09:06:15.891+0000 7f4be27fc640 4 set_mon_vals no callback set 2024-06-03T09:06:15.922 INFO:teuthology.orchestra.run.smithi088.stderr:2024-06-03T09:06:15.922+0000 7f4be0ff9640 1 -- 172.21.15.88:0/193553391 <== mon.0 v1:172.21.15.88:6789/0 7 ==== mgrmap(e 5) v1 ==== 89110+0+0 (unknown 873533973 0 0) 0x7f4bd801a870 con 0x7f4be40ad920 2024-06-03T09:06:15.923 INFO:teuthology.orchestra.run.smithi088.stderr:2024-06-03T09:06:15.923+0000 7f4be3fff640 10 monclient: get_auth_request con 0x7f4bd006d450 auth_method 0 2024-06-03T09:06:15.923 INFO:teuthology.orchestra.run.smithi088.stderr:2024-06-03T09:06:15.923+0000 7f4be0ff9640 10 client.30564.objecter ms_handle_connect 0x7f4bd006d450 2024-06-03T09:06:15.928 INFO:teuthology.orchestra.run.smithi088.stderr:2024-06-03T09:06:15.928+0000 7f4be0ff9640 1 -- 172.21.15.88:0/193553391 <== mon.0 v1:172.21.15.88:6789/0 8 ==== osd_map(1009..1009 src has 622..1009) v4 ==== 5229+0+0 (unknown 553880507 0 0) 0x7f4bd8054140 con 0x7f4be40ad920 2024-06-03T09:06:15.929 INFO:teuthology.orchestra.run.smithi088.stderr:2024-06-03T09:06:15.928+0000 7f4be0ff9640 10 client.30564.objecter ms_dispatch 0x7f4be4000b90 osd_map(1009..1009 src has 622..1009) v4 2024-06-03T09:06:15.929 INFO:teuthology.orchestra.run.smithi088.stderr:2024-06-03T09:06:15.928+0000 7f4be0ff9640 3 client.30564.objecter handle_osd_map got epochs [1009,1009] > 0 2024-06-03T09:06:15.929 INFO:teuthology.orchestra.run.smithi088.stderr:2024-06-03T09:06:15.928+0000 7f4be0ff9640 3 client.30564.objecter handle_osd_map decoding full epoch 1009 2024-06-03T09:06:15.929 INFO:teuthology.orchestra.run.smithi088.stderr:2024-06-03T09:06:15.928+0000 7f4be0ff9640 20 client.30564.objecter dump_active .. 0 homeless 2024-06-03T09:06:15.929 INFO:teuthology.orchestra.run.smithi088.stderr:2024-06-03T09:06:15.928+0000 7f4be0ff9640 1 -- 172.21.15.88:0/193553391 <== mon.0 v1:172.21.15.88:6789/0 9 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+182498 (unknown 1092875540 0 3016760700) 0x7f4bd804fcf0 con 0x7f4be40ad920 2024-06-03T09:06:15.929 INFO:teuthology.orchestra.run.smithi088.stderr:2024-06-03T09:06:15.928+0000 7f4be0ff9640 10 monclient: handle_mon_command_ack 1 [{"prefix": "get_command_descriptions"}] 2024-06-03T09:06:15.929 INFO:teuthology.orchestra.run.smithi088.stderr:2024-06-03T09:06:15.928+0000 7f4be0ff9640 10 monclient: _finish_command 1 = system:0 2024-06-03T09:06:16.115 INFO:teuthology.orchestra.run.smithi088.stderr:2024-06-03T09:06:16.115+0000 7f4bea384640 1 -- 172.21.15.88:0/193553391 --> v1:172.21.15.192:6800/34229 -- mgr_command(tid 0: {"prefix": "pg dump", "target": ["mon-mgr", ""], "format": "json"}) v1 -- 0x7f4be413d230 con 0x7f4bd006d450 2024-06-03T09:06:16.120 INFO:teuthology.orchestra.run.smithi088.stderr:2024-06-03T09:06:16.120+0000 7f4be0ff9640 1 -- 172.21.15.88:0/193553391 <== mgr.4110 v1:172.21.15.192:6800/34229 1 ==== mgr_command_reply(tid 0: 0 dumped all) v1 ==== 18+0+151820 (unknown 2965378022 0 2920618137) 0x7f4be413d230 con 0x7f4bd006d450 2024-06-03T09:06:16.121 INFO:teuthology.orchestra.run.smithi088.stdout: 2024-06-03T09:06:16.122 INFO:teuthology.orchestra.run.smithi088.stderr:dumped all 2024-06-03T09:06:16.123 INFO:teuthology.orchestra.run.smithi088.stderr:2024-06-03T09:06:16.122+0000 7f4bea384640 10 librados: watch_flush enter 2024-06-03T09:06:16.123 INFO:teuthology.orchestra.run.smithi088.stderr:2024-06-03T09:06:16.122+0000 7f4bea384640 10 librados: watch_flush exit 2024-06-03T09:06:16.124 INFO:teuthology.orchestra.run.smithi088.stderr:2024-06-03T09:06:16.122+0000 7f4bea384640 20 client.30564.objecter shutdown clearing up homeless session... 2024-06-03T09:06:16.124 INFO:teuthology.orchestra.run.smithi088.stderr:2024-06-03T09:06:16.122+0000 7f4bea384640 10 client.30564.objecter successfully canceled tick 2024-06-03T09:06:16.124 INFO:teuthology.orchestra.run.smithi088.stderr:2024-06-03T09:06:16.122+0000 7f4bea384640 1 -- 172.21.15.88:0/193553391 >> v1:172.21.15.192:6800/34229 conn(0x7f4bd006d450 legacy=0x7f4bd006f910 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-06-03T09:06:16.124 INFO:teuthology.orchestra.run.smithi088.stderr:2024-06-03T09:06:16.122+0000 7f4bea384640 10 monclient: shutdown 2024-06-03T09:06:16.124 INFO:teuthology.orchestra.run.smithi088.stderr:2024-06-03T09:06:16.122+0000 7f4bea384640 20 monclient: shutdown discarding 0 pending message(s) 2024-06-03T09:06:16.124 INFO:teuthology.orchestra.run.smithi088.stderr:2024-06-03T09:06:16.122+0000 7f4bea384640 1 -- 172.21.15.88:0/193553391 >> v1:172.21.15.88:6789/0 conn(0x7f4be40ad920 legacy=0x7f4be40aae50 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-06-03T09:06:16.124 INFO:teuthology.orchestra.run.smithi088.stderr:2024-06-03T09:06:16.122+0000 7f4bea384640 1 -- 172.21.15.88:0/193553391 shutdown_connections 2024-06-03T09:06:16.124 INFO:teuthology.orchestra.run.smithi088.stderr:2024-06-03T09:06:16.122+0000 7f4bea384640 1 -- 172.21.15.88:0/193553391 >> 172.21.15.88:0/193553391 conn(0x7f4be40b2f20 msgr2=0x7f4be40adf80 unknown :-1 s=STATE_NONE l=0).mark_down 2024-06-03T09:06:16.124 INFO:teuthology.orchestra.run.smithi088.stderr:2024-06-03T09:06:16.122+0000 7f4bea384640 1 -- 172.21.15.88:0/193553391 shutdown_connections 2024-06-03T09:06:16.124 INFO:teuthology.orchestra.run.smithi088.stderr:2024-06-03T09:06:16.122+0000 7f4bea384640 1 -- 172.21.15.88:0/193553391 wait complete. 2024-06-03T09:06:16.124 INFO:teuthology.orchestra.run.smithi088.stderr:2024-06-03T09:06:16.122+0000 7f4bea384640 1 librados: shutdown 2024-06-03T09:06:16.133 INFO:teuthology.orchestra.run.smithi088.stdout:{"pg_ready":true,"pg_map":{"version":1912,"stamp":"2024-06-03T09:06:15.097444+0000","last_osdmap_epoch":0,"last_pg_scan":0,"pg_stats_sum":{"stat_sum":{"num_bytes":459285,"num_objects":3,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":1,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":3,"num_whiteouts":0,"num_read":102,"num_read_kb":88,"num_write":120,"num_write_kb":1379,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2,"num_bytes_recovered":397840,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":83,"ondisk_log_size":83,"up":68,"acting":68,"num_store_stats":0},"osd_stats_sum":{"up_from":0,"seq":0,"num_pgs":126,"num_osds":8,"num_per_pool_osds":8,"num_per_pool_omap_osds":8,"kb":754974720,"kb_used":494376,"kb_used_data":46140,"kb_used_omap":12,"kb_used_meta":448115,"kb_avail":754480344,"statfs":{"total":773094113280,"available":772587872256,"internally_reserved":0,"allocated":47247360,"data_stored":22695791,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":12744,"internal_metadata":458870328},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":433,"apply_latency_ms":433,"commit_latency_ns":433000000,"apply_latency_ns":433000000},"alerts":[],"network_ping_times":[]},"pg_stats_delta":{"stat_sum":{"num_bytes":-4000299,"num_objects":-44,"num_object_clones":-1,"num_object_copies":-132,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":-32,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":-44,"num_whiteouts":0,"num_read":-31,"num_read_kb":-11,"num_write":-71,"num_write_kb":-3969,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":-1,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":0,"acting":0,"num_store_stats":0,"stamp_delta":"4.897601"},"pg_stats":[{"pgid":"17.d","version":"0'0","reported_seq":940,"reported_epoch":1007,"state":"active+clean","last_fresh":"2024-06-03T09:06:11.402997+0000","last_change":"2024-06-03T08:38:19.260548+0000","last_active":"2024-06-03T09:06:11.402997+0000","last_peered":"2024-06-03T09:06:11.402997+0000","last_clean":"2024-06-03T09:06:11.402997+0000","last_became_active":"2024-06-03T08:38:19.253559+0000","last_became_peered":"2024-06-03T08:38:19.253559+0000","last_unstale":"2024-06-03T09:06:11.402997+0000","last_undegraded":"2024-06-03T09:06:11.402997+0000","last_fullsized":"2024-06-03T09:06:11.402997+0000","mapping_epoch":28,"log_start":"0'0","ondisk_log_start":"0'0","created":28,"last_epoch_clean":45,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-06-03T08:37:55.614497+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-06-03T08:37:55.614497+0000","last_clean_scrub_stamp":"2024-06-03T08:37:55.614497+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-06-04T19:00:12.470979+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1],"acting":[0,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"17.c","version":"0'0","reported_seq":952,"reported_epoch":1008,"state":"active+undersized","last_fresh":"2024-06-03T09:06:12.368416+0000","last_change":"2024-06-03T09:06:03.440074+0000","last_active":"2024-06-03T09:06:12.368416+0000","last_peered":"2024-06-03T09:06:12.368416+0000","last_clean":"2024-06-03T09:05:02.105638+0000","last_became_active":"2024-06-03T09:06:03.440074+0000","last_became_peered":"2024-06-03T09:06:03.440074+0000","last_unstale":"2024-06-03T09:06:12.368416+0000","last_undegraded":"2024-06-03T09:06:12.368416+0000","last_fullsized":"2024-06-03T09:06:03.377982+0000","mapping_epoch":999,"log_start":"0'0","ondisk_log_start":"0'0","created":28,"last_epoch_clean":255,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-06-03T08:37:55.614497+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-06-03T08:37:55.614497+0000","last_clean_scrub_stamp":"2024-06-03T08:37:55.614497+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-06-04T10:34:20.808557+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3],"acting":[3],"avail_no_missing":["3"],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"17.f","version":"0'0","reported_seq":144,"reported_epoch":1004,"state":"active+undersized","last_fresh":"2024-06-03T09:06:08.016575+0000","last_change":"2024-06-03T09:06:03.420285+0000","last_active":"2024-06-03T09:06:08.016575+0000","last_peered":"2024-06-03T09:06:08.016575+0000","last_clean":"2024-06-03T08:37:55.614497+0000","last_became_active":"2024-06-03T09:06:03.420285+0000","last_became_peered":"2024-06-03T09:06:03.420285+0000","last_unstale":"2024-06-03T09:06:08.016575+0000","last_undegraded":"2024-06-03T09:06:08.016575+0000","last_fullsized":"2024-06-03T09:06:03.380616+0000","mapping_epoch":999,"log_start":"0'0","ondisk_log_start":"0'0","created":28,"last_epoch_clean":255,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-06-03T08:37:55.614497+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-06-03T08:37:55.614497+0000","last_clean_scrub_stamp":"2024-06-03T08:37:55.614497+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-06-04T15:39:41.187731+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4],"acting":[4],"avail_no_missing":["4"],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"17.e","version":"0'0","reported_seq":148,"reported_epoch":1008,"state":"active+undersized","last_fresh":"2024-06-03T09:06:12.895353+0000","last_change":"2024-06-03T09:06:03.412887+0000","last_active":"2024-06-03T09:06:12.895353+0000","last_peered":"2024-06-03T09:06:12.895353+0000","last_clean":"2024-06-03T08:37:55.614497+0000","last_became_active":"2024-06-03T09:06:03.412887+0000","last_became_peered":"2024-06-03T09:06:03.412887+0000","last_unstale":"2024-06-03T09:06:12.895353+0000","last_undegraded":"2024-06-03T09:06:12.895353+0000","last_fullsized":"2024-06-03T09:06:03.332482+0000","mapping_epoch":999,"log_start":"0'0","ondisk_log_start":"0'0","created":28,"last_epoch_clean":255,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-06-03T08:37:55.614497+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-06-03T08:37:55.614497+0000","last_clean_scrub_stamp":"2024-06-03T08:37:55.614497+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-06-04T12:23:22.511601+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5],"acting":[5],"avail_no_missing":["5"],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"17.9","version":"0'0","reported_seq":945,"reported_epoch":1009,"state":"active+clean","last_fresh":"2024-06-03T09:06:13.928215+0000","last_change":"2024-06-03T08:38:04.772260+0000","last_active":"2024-06-03T09:06:13.928215+0000","last_peered":"2024-06-03T09:06:13.928215+0000","last_clean":"2024-06-03T09:06:13.928215+0000","last_became_active":"2024-06-03T08:38:04.763428+0000","last_became_peered":"2024-06-03T08:38:04.763428+0000","last_unstale":"2024-06-03T09:06:13.928215+0000","last_undegraded":"2024-06-03T09:06:13.928215+0000","last_fullsized":"2024-06-03T09:06:13.928215+0000","mapping_epoch":28,"log_start":"0'0","ondisk_log_start":"0'0","created":28,"last_epoch_clean":33,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-06-03T08:37:55.614497+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-06-03T08:37:55.614497+0000","last_clean_scrub_stamp":"2024-06-03T08:37:55.614497+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-06-04T10:55:10.773936+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,4],"acting":[7,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"17.8","version":"0'0","reported_seq":959,"reported_epoch":1007,"state":"active+undersized","last_fresh":"2024-06-03T09:06:12.708781+0000","last_change":"2024-06-03T09:06:03.388786+0000","last_active":"2024-06-03T09:06:12.708781+0000","last_peered":"2024-06-03T09:06:12.708781+0000","last_clean":"2024-06-03T09:04:43.297780+0000","last_became_active":"2024-06-03T09:06:03.388786+0000","last_became_peered":"2024-06-03T09:06:03.388786+0000","last_unstale":"2024-06-03T09:06:12.708781+0000","last_undegraded":"2024-06-03T09:06:12.708781+0000","last_fullsized":"2024-06-03T09:06:03.331754+0000","mapping_epoch":999,"log_start":"0'0","ondisk_log_start":"0'0","created":28,"last_epoch_clean":255,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-06-03T08:37:55.614497+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-06-03T08:37:55.614497+0000","last_clean_scrub_stamp":"2024-06-03T08:37:55.614497+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-06-04T09:18:44.765547+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0],"acting":[0],"avail_no_missing":["0"],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"17.b","version":"0'0","reported_seq":145,"reported_epoch":1005,"state":"active+undersized","last_fresh":"2024-06-03T09:06:09.149286+0000","last_change":"2024-06-03T09:06:03.451203+0000","last_active":"2024-06-03T09:06:09.149286+0000","last_peered":"2024-06-03T09:06:09.149286+0000","last_clean":"2024-06-03T08:37:55.614497+0000","last_became_active":"2024-06-03T09:06:03.451203+0000","last_became_peered":"2024-06-03T09:06:03.451203+0000","last_unstale":"2024-06-03T09:06:09.149286+0000","last_undegraded":"2024-06-03T09:06:09.149286+0000","last_fullsized":"2024-06-03T09:06:03.375523+0000","mapping_epoch":999,"log_start":"0'0","ondisk_log_start":"0'0","created":28,"last_epoch_clean":255,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-06-03T08:37:55.614497+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-06-03T08:37:55.614497+0000","last_clean_scrub_stamp":"2024-06-03T08:37:55.614497+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-06-04T10:16:19.813495+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2],"acting":[2],"avail_no_missing":["2"],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"17.a","version":"0'0","reported_seq":923,"reported_epoch":1008,"state":"active+clean","last_fresh":"2024-06-03T09:06:12.871052+0000","last_change":"2024-06-03T08:38:05.843142+0000","last_active":"2024-06-03T09:06:12.871052+0000","last_peered":"2024-06-03T09:06:12.871052+0000","last_clean":"2024-06-03T09:06:12.871052+0000","last_became_active":"2024-06-03T08:38:05.809257+0000","last_became_peered":"2024-06-03T08:38:05.809257+0000","last_unstale":"2024-06-03T09:06:12.871052+0000","last_undegraded":"2024-06-03T09:06:12.871052+0000","last_fullsized":"2024-06-03T09:06:12.871052+0000","mapping_epoch":28,"log_start":"0'0","ondisk_log_start":"0'0","created":28,"last_epoch_clean":33,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-06-03T08:37:55.614497+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-06-03T08:37:55.614497+0000","last_clean_scrub_stamp":"2024-06-03T08:37:55.614497+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-06-04T12:12:03.686838+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2],"acting":[1,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"17.5","version":"0'0","reported_seq":941,"reported_epoch":1007,"state":"active+clean","last_fresh":"2024-06-03T09:06:11.400979+0000","last_change":"2024-06-03T08:38:23.466051+0000","last_active":"2024-06-03T09:06:11.400979+0000","last_peered":"2024-06-03T09:06:11.400979+0000","last_clean":"2024-06-03T09:06:11.400979+0000","last_became_active":"2024-06-03T08:38:23.458039+0000","last_became_peered":"2024-06-03T08:38:23.458039+0000","last_unstale":"2024-06-03T09:06:11.400979+0000","last_undegraded":"2024-06-03T09:06:11.400979+0000","last_fullsized":"2024-06-03T09:06:11.400979+0000","mapping_epoch":28,"log_start":"0'0","ondisk_log_start":"0'0","created":28,"last_epoch_clean":49,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-06-03T08:37:55.614497+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-06-03T08:37:55.614497+0000","last_clean_scrub_stamp":"2024-06-03T08:37:55.614497+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-06-04T12:27:02.532436+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,4],"acting":[0,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"17.4","version":"0'0","reported_seq":148,"reported_epoch":1008,"state":"active+undersized","last_fresh":"2024-06-03T09:06:12.883827+0000","last_change":"2024-06-03T09:06:03.441889+0000","last_active":"2024-06-03T09:06:12.883827+0000","last_peered":"2024-06-03T09:06:12.883827+0000","last_clean":"2024-06-03T08:37:55.614497+0000","last_became_active":"2024-06-03T09:06:03.441889+0000","last_became_peered":"2024-06-03T09:06:03.441889+0000","last_unstale":"2024-06-03T09:06:12.883827+0000","last_undegraded":"2024-06-03T09:06:12.883827+0000","last_fullsized":"2024-06-03T09:06:03.359582+0000","mapping_epoch":999,"log_start":"0'0","ondisk_log_start":"0'0","created":28,"last_epoch_clean":255,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-06-03T08:37:55.614497+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-06-03T08:37:55.614497+0000","last_clean_scrub_stamp":"2024-06-03T08:37:55.614497+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-06-04T12:49:22.907141+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7],"acting":[7],"avail_no_missing":["7"],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"17.7","version":"0'0","reported_seq":928,"reported_epoch":1008,"state":"active+clean","last_fresh":"2024-06-03T09:06:12.906543+0000","last_change":"2024-06-03T08:38:04.837083+0000","last_active":"2024-06-03T09:06:12.906543+0000","last_peered":"2024-06-03T09:06:12.906543+0000","last_clean":"2024-06-03T09:06:12.906543+0000","last_became_active":"2024-06-03T08:38:04.818246+0000","last_became_peered":"2024-06-03T08:38:04.818246+0000","last_unstale":"2024-06-03T09:06:12.906543+0000","last_undegraded":"2024-06-03T09:06:12.906543+0000","last_fullsized":"2024-06-03T09:06:12.906543+0000","mapping_epoch":28,"log_start":"0'0","ondisk_log_start":"0'0","created":28,"last_epoch_clean":34,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-06-03T08:37:55.614497+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-06-03T08:37:55.614497+0000","last_clean_scrub_stamp":"2024-06-03T08:37:55.614497+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-06-04T17:02:38.204813+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,2],"acting":[5,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"17.6","version":"0'0","reported_seq":958,"reported_epoch":1007,"state":"active+undersized","last_fresh":"2024-06-03T09:06:12.710313+0000","last_change":"2024-06-03T09:06:03.428677+0000","last_active":"2024-06-03T09:06:12.710313+0000","last_peered":"2024-06-03T09:06:12.710313+0000","last_clean":"2024-06-03T09:04:43.312352+0000","last_became_active":"2024-06-03T09:06:03.428677+0000","last_became_peered":"2024-06-03T09:06:03.428677+0000","last_unstale":"2024-06-03T09:06:12.710313+0000","last_undegraded":"2024-06-03T09:06:12.710313+0000","last_fullsized":"2024-06-03T09:06:03.359909+0000","mapping_epoch":999,"log_start":"0'0","ondisk_log_start":"0'0","created":28,"last_epoch_clean":255,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-06-03T08:37:55.614497+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-06-03T08:37:55.614497+0000","last_clean_scrub_stamp":"2024-06-03T08:37:55.614497+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-06-04T15:38:02.180241+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0],"acting":[0],"avail_no_missing":["0"],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"2.7","version":"0'0","reported_seq":939,"reported_epoch":1008,"state":"active+clean","last_fresh":"2024-06-03T09:06:12.364540+0000","last_change":"2024-06-03T08:36:30.385405+0000","last_active":"2024-06-03T09:06:12.364540+0000","last_peered":"2024-06-03T09:06:12.364540+0000","last_clean":"2024-06-03T09:06:12.364540+0000","last_became_active":"2024-06-03T08:36:30.380302+0000","last_became_peered":"2024-06-03T08:36:30.380302+0000","last_unstale":"2024-06-03T09:06:12.364540+0000","last_undegraded":"2024-06-03T09:06:12.364540+0000","last_fullsized":"2024-06-03T09:06:12.364540+0000","mapping_epoch":23,"log_start":"0'0","ondisk_log_start":"0'0","created":23,"last_epoch_clean":24,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-06-03T08:36:28.444764+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-06-03T08:36:28.444764+0000","last_clean_scrub_stamp":"2024-06-03T08:36:28.444764+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-06-04T18:46:33.475181+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0],"acting":[3,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"17.14","version":"0'0","reported_seq":925,"reported_epoch":1008,"state":"active+clean","last_fresh":"2024-06-03T09:06:12.884449+0000","last_change":"2024-06-03T08:38:04.547565+0000","last_active":"2024-06-03T09:06:12.884449+0000","last_peered":"2024-06-03T09:06:12.884449+0000","last_clean":"2024-06-03T09:06:12.884449+0000","last_became_active":"2024-06-03T08:38:04.541781+0000","last_became_peered":"2024-06-03T08:38:04.541781+0000","last_unstale":"2024-06-03T09:06:12.884449+0000","last_undegraded":"2024-06-03T09:06:12.884449+0000","last_fullsized":"2024-06-03T09:06:12.884449+0000","mapping_epoch":28,"log_start":"0'0","ondisk_log_start":"0'0","created":28,"last_epoch_clean":34,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-06-03T08:37:55.614497+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-06-03T08:37:55.614497+0000","last_clean_scrub_stamp":"2024-06-03T08:37:55.614497+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-06-04T16:41:55.170134+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,2],"acting":[5,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"2.6","version":"0'0","reported_seq":945,"reported_epoch":1004,"state":"active+clean","last_fresh":"2024-06-03T09:06:07.984455+0000","last_change":"2024-06-03T08:36:30.430760+0000","last_active":"2024-06-03T09:06:07.984455+0000","last_peered":"2024-06-03T09:06:07.984455+0000","last_clean":"2024-06-03T09:06:07.984455+0000","last_became_active":"2024-06-03T08:36:30.404665+0000","last_became_peered":"2024-06-03T08:36:30.404665+0000","last_unstale":"2024-06-03T09:06:07.984455+0000","last_undegraded":"2024-06-03T09:06:07.984455+0000","last_fullsized":"2024-06-03T09:06:07.984455+0000","mapping_epoch":23,"log_start":"0'0","ondisk_log_start":"0'0","created":23,"last_epoch_clean":24,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-06-03T08:36:28.444764+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-06-03T08:36:28.444764+0000","last_clean_scrub_stamp":"2024-06-03T08:36:28.444764+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-06-04T10:37:37.746074+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,0],"acting":[4,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"17.15","version":"0'0","reported_seq":933,"reported_epoch":1008,"state":"active+clean","last_fresh":"2024-06-03T09:06:12.371160+0000","last_change":"2024-06-03T08:38:20.999903+0000","last_active":"2024-06-03T09:06:12.371160+0000","last_peered":"2024-06-03T09:06:12.371160+0000","last_clean":"2024-06-03T09:06:12.371160+0000","last_became_active":"2024-06-03T08:38:20.979523+0000","last_became_peered":"2024-06-03T08:38:20.979523+0000","last_unstale":"2024-06-03T09:06:12.371160+0000","last_undegraded":"2024-06-03T09:06:12.371160+0000","last_fullsized":"2024-06-03T09:06:12.371160+0000","mapping_epoch":28,"log_start":"0'0","ondisk_log_start":"0'0","created":28,"last_epoch_clean":47,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-06-03T08:37:55.614497+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-06-03T08:37:55.614497+0000","last_clean_scrub_stamp":"2024-06-03T08:37:55.614497+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-06-04T20:29:58.314289+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,1],"acting":[3,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"2.5","version":"0'0","reported_seq":940,"reported_epoch":1008,"state":"active+clean","last_fresh":"2024-06-03T09:06:12.372623+0000","last_change":"2024-06-03T08:36:30.024235+0000","last_active":"2024-06-03T09:06:12.372623+0000","last_peered":"2024-06-03T09:06:12.372623+0000","last_clean":"2024-06-03T09:06:12.372623+0000","last_became_active":"2024-06-03T08:36:30.019040+0000","last_became_peered":"2024-06-03T08:36:30.019040+0000","last_unstale":"2024-06-03T09:06:12.372623+0000","last_undegraded":"2024-06-03T09:06:12.372623+0000","last_fullsized":"2024-06-03T09:06:12.372623+0000","mapping_epoch":23,"log_start":"0'0","ondisk_log_start":"0'0","created":23,"last_epoch_clean":24,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-06-03T08:36:28.444764+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-06-03T08:36:28.444764+0000","last_clean_scrub_stamp":"2024-06-03T08:36:28.444764+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-06-04T15:46:47.109502+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,4],"acting":[3,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"17.16","version":"0'0","reported_seq":968,"reported_epoch":1005,"state":"active+undersized","last_fresh":"2024-06-03T09:06:09.191972+0000","last_change":"2024-06-03T09:06:03.478615+0000","last_active":"2024-06-03T09:06:09.191972+0000","last_peered":"2024-06-03T09:06:09.191972+0000","last_clean":"2024-06-03T09:04:43.328415+0000","last_became_active":"2024-06-03T09:06:03.478615+0000","last_became_peered":"2024-06-03T09:06:03.478615+0000","last_unstale":"2024-06-03T09:06:09.191972+0000","last_undegraded":"2024-06-03T09:06:09.191972+0000","last_fullsized":"2024-06-03T09:06:03.402504+0000","mapping_epoch":999,"log_start":"0'0","ondisk_log_start":"0'0","created":28,"last_epoch_clean":255,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-06-03T08:37:55.614497+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-06-03T08:37:55.614497+0000","last_clean_scrub_stamp":"2024-06-03T08:37:55.614497+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-06-04T08:47:08.079233+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2],"acting":[2],"avail_no_missing":["2"],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"2.4","version":"0'0","reported_seq":950,"reported_epoch":1008,"state":"active+undersized","last_fresh":"2024-06-03T09:06:12.887507+0000","last_change":"2024-06-03T09:06:03.387171+0000","last_active":"2024-06-03T09:06:12.887507+0000","last_peered":"2024-06-03T09:06:12.887507+0000","last_clean":"2024-06-03T09:04:43.202645+0000","last_became_active":"2024-06-03T09:06:03.387171+0000","last_became_peered":"2024-06-03T09:06:03.387171+0000","last_unstale":"2024-06-03T09:06:12.887507+0000","last_undegraded":"2024-06-03T09:06:12.887507+0000","last_fullsized":"2024-06-03T09:06:03.283787+0000","mapping_epoch":999,"log_start":"0'0","ondisk_log_start":"0'0","created":23,"last_epoch_clean":255,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-06-03T08:36:28.444764+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-06-03T08:36:28.444764+0000","last_clean_scrub_stamp":"2024-06-03T08:36:28.444764+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-06-04T20:17:43.485023+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5],"acting":[5],"avail_no_missing":["5"],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"17.17","version":"0'0","reported_seq":950,"reported_epoch":1005,"state":"active+clean","last_fresh":"2024-06-03T09:06:09.179648+0000","last_change":"2024-06-03T08:38:38.483275+0000","last_active":"2024-06-03T09:06:09.179648+0000","last_peered":"2024-06-03T09:06:09.179648+0000","last_clean":"2024-06-03T09:06:09.179648+0000","last_became_active":"2024-06-03T08:38:38.467916+0000","last_became_peered":"2024-06-03T08:38:38.467916+0000","last_unstale":"2024-06-03T09:06:09.179648+0000","last_undegraded":"2024-06-03T09:06:09.179648+0000","last_fullsized":"2024-06-03T09:06:09.179648+0000","mapping_epoch":28,"log_start":"0'0","ondisk_log_start":"0'0","created":28,"last_epoch_clean":61,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-06-03T08:37:55.614497+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-06-03T08:37:55.614497+0000","last_clean_scrub_stamp":"2024-06-03T08:37:55.614497+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-06-04T10:02:23.036420+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,7],"acting":[2,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"2.2","version":"0'0","reported_seq":962,"reported_epoch":1008,"state":"active+undersized","last_fresh":"2024-06-03T09:06:12.352260+0000","last_change":"2024-06-03T09:06:03.410020+0000","last_active":"2024-06-03T09:06:12.352260+0000","last_peered":"2024-06-03T09:06:12.352260+0000","last_clean":"2024-06-03T09:05:37.145889+0000","last_became_active":"2024-06-03T09:06:03.410020+0000","last_became_peered":"2024-06-03T09:06:03.410020+0000","last_unstale":"2024-06-03T09:06:12.352260+0000","last_undegraded":"2024-06-03T09:06:12.352260+0000","last_fullsized":"2024-06-03T09:06:03.346864+0000","mapping_epoch":999,"log_start":"0'0","ondisk_log_start":"0'0","created":23,"last_epoch_clean":255,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-06-03T08:36:28.444764+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-06-03T08:36:28.444764+0000","last_clean_scrub_stamp":"2024-06-03T08:36:28.444764+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-06-04T09:57:12.423360+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3],"acting":[3],"avail_no_missing":["3"],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"17.11","version":"0'0","reported_seq":940,"reported_epoch":1004,"state":"active+clean","last_fresh":"2024-06-03T09:06:07.943878+0000","last_change":"2024-06-03T08:38:06.502302+0000","last_active":"2024-06-03T09:06:07.943878+0000","last_peered":"2024-06-03T09:06:07.943878+0000","last_clean":"2024-06-03T09:06:07.943878+0000","last_became_active":"2024-06-03T08:38:06.490204+0000","last_became_peered":"2024-06-03T08:38:06.490204+0000","last_unstale":"2024-06-03T09:06:07.943878+0000","last_undegraded":"2024-06-03T09:06:07.943878+0000","last_fullsized":"2024-06-03T09:06:07.943878+0000","mapping_epoch":28,"log_start":"0'0","ondisk_log_start":"0'0","created":28,"last_epoch_clean":32,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-06-03T08:37:55.614497+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-06-03T08:37:55.614497+0000","last_clean_scrub_stamp":"2024-06-03T08:37:55.614497+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-06-04T10:47:05.988958+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,2],"acting":[4,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"2.1","version":"0'0","reported_seq":951,"reported_epoch":1008,"state":"active+clean","last_fresh":"2024-06-03T09:06:12.872310+0000","last_change":"2024-06-03T08:36:30.883360+0000","last_active":"2024-06-03T09:06:12.872310+0000","last_peered":"2024-06-03T09:06:12.872310+0000","last_clean":"2024-06-03T09:06:12.872310+0000","last_became_active":"2024-06-03T08:36:30.855073+0000","last_became_peered":"2024-06-03T08:36:30.855073+0000","last_unstale":"2024-06-03T09:06:12.872310+0000","last_undegraded":"2024-06-03T09:06:12.872310+0000","last_fullsized":"2024-06-03T09:06:12.872310+0000","mapping_epoch":23,"log_start":"0'0","ondisk_log_start":"0'0","created":23,"last_epoch_clean":25,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-06-03T08:36:28.444764+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-06-03T08:36:28.444764+0000","last_clean_scrub_stamp":"2024-06-03T08:36:28.444764+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-06-04T18:12:22.363473+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,5],"acting":[7,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"17.12","version":"0'0","reported_seq":944,"reported_epoch":1008,"state":"active+clean","last_fresh":"2024-06-03T09:06:12.878527+0000","last_change":"2024-06-03T08:38:01.751562+0000","last_active":"2024-06-03T09:06:12.878527+0000","last_peered":"2024-06-03T09:06:12.878527+0000","last_clean":"2024-06-03T09:06:12.878527+0000","last_became_active":"2024-06-03T08:38:01.745027+0000","last_became_peered":"2024-06-03T08:38:01.745027+0000","last_unstale":"2024-06-03T09:06:12.878527+0000","last_undegraded":"2024-06-03T09:06:12.878527+0000","last_fullsized":"2024-06-03T09:06:12.878527+0000","mapping_epoch":28,"log_start":"0'0","ondisk_log_start":"0'0","created":28,"last_epoch_clean":32,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-06-03T08:37:55.614497+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-06-03T08:37:55.614497+0000","last_clean_scrub_stamp":"2024-06-03T08:37:55.614497+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-06-04T17:54:43.785326+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,3],"acting":[7,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"2.0","version":"0'0","reported_seq":942,"reported_epoch":1008,"state":"active+clean","last_fresh":"2024-06-03T09:06:12.341224+0000","last_change":"2024-06-03T08:36:29.984096+0000","last_active":"2024-06-03T09:06:12.341224+0000","last_peered":"2024-06-03T09:06:12.341224+0000","last_clean":"2024-06-03T09:06:12.341224+0000","last_became_active":"2024-06-03T08:36:29.957875+0000","last_became_peered":"2024-06-03T08:36:29.957875+0000","last_unstale":"2024-06-03T09:06:12.341224+0000","last_undegraded":"2024-06-03T09:06:12.341224+0000","last_fullsized":"2024-06-03T09:06:12.341224+0000","mapping_epoch":23,"log_start":"0'0","ondisk_log_start":"0'0","created":23,"last_epoch_clean":24,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-06-03T08:36:28.444764+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-06-03T08:36:28.444764+0000","last_clean_scrub_stamp":"2024-06-03T08:36:28.444764+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-06-04T08:41:33.679479+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,5],"acting":[3,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"17.13","version":"0'0","reported_seq":939,"reported_epoch":1005,"state":"active+clean","last_fresh":"2024-06-03T09:06:09.209721+0000","last_change":"2024-06-03T08:38:26.838316+0000","last_active":"2024-06-03T09:06:09.209721+0000","last_peered":"2024-06-03T09:06:09.209721+0000","last_clean":"2024-06-03T09:06:09.209721+0000","last_became_active":"2024-06-03T08:38:26.833214+0000","last_became_peered":"2024-06-03T08:38:26.833214+0000","last_unstale":"2024-06-03T09:06:09.209721+0000","last_undegraded":"2024-06-03T09:06:09.209721+0000","last_fullsized":"2024-06-03T09:06:09.209721+0000","mapping_epoch":28,"log_start":"0'0","ondisk_log_start":"0'0","created":28,"last_epoch_clean":52,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-06-03T08:37:55.614497+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-06-03T08:37:55.614497+0000","last_clean_scrub_stamp":"2024-06-03T08:37:55.614497+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-06-04T12:58:51.617998+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,0],"acting":[4,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"2.3","version":"0'0","reported_seq":941,"reported_epoch":1008,"state":"active+clean","last_fresh":"2024-06-03T09:06:12.356955+0000","last_change":"2024-06-03T08:36:30.351670+0000","last_active":"2024-06-03T09:06:12.356955+0000","last_peered":"2024-06-03T09:06:12.356955+0000","last_clean":"2024-06-03T09:06:12.356955+0000","last_became_active":"2024-06-03T08:36:30.343560+0000","last_became_peered":"2024-06-03T08:36:30.343560+0000","last_unstale":"2024-06-03T09:06:12.356955+0000","last_undegraded":"2024-06-03T09:06:12.356955+0000","last_fullsized":"2024-06-03T09:06:12.356955+0000","mapping_epoch":23,"log_start":"0'0","ondisk_log_start":"0'0","created":23,"last_epoch_clean":24,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-06-03T08:36:28.444764+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-06-03T08:36:28.444764+0000","last_clean_scrub_stamp":"2024-06-03T08:36:28.444764+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-06-04T12:02:24.765135+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,2],"acting":[3,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.0","version":"19'76","reported_seq":1050,"reported_epoch":1008,"state":"active+clean","last_fresh":"2024-06-03T09:06:12.865199+0000","last_change":"2024-06-03T08:36:26.702702+0000","last_active":"2024-06-03T09:06:12.865199+0000","last_peered":"2024-06-03T09:06:12.865199+0000","last_clean":"2024-06-03T09:06:12.865199+0000","last_became_active":"2024-06-03T08:36:26.128378+0000","last_became_peered":"2024-06-03T08:36:26.128378+0000","last_unstale":"2024-06-03T09:06:12.865199+0000","last_undegraded":"2024-06-03T09:06:12.865199+0000","last_fullsized":"2024-06-03T09:06:12.865199+0000","mapping_epoch":20,"log_start":"0'0","ondisk_log_start":"0'0","created":16,"last_epoch_clean":22,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-06-03T08:36:15.542635+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-06-03T08:36:15.542635+0000","last_clean_scrub_stamp":"2024-06-03T08:36:15.542635+0000","objects_scrubbed":0,"log_size":76,"log_dups_size":0,"ondisk_log_size":76,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-06-04T12:41:53.731360+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":459280,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":96,"num_read_kb":82,"num_write":113,"num_write_kb":1372,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2,"num_bytes_recovered":397840,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,5],"acting":[1,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"17.10","version":"0'0","reported_seq":940,"reported_epoch":1004,"state":"active+clean","last_fresh":"2024-06-03T09:06:07.939453+0000","last_change":"2024-06-03T08:38:15.399189+0000","last_active":"2024-06-03T09:06:07.939453+0000","last_peered":"2024-06-03T09:06:07.939453+0000","last_clean":"2024-06-03T09:06:07.939453+0000","last_became_active":"2024-06-03T08:38:15.337723+0000","last_became_peered":"2024-06-03T08:38:15.337723+0000","last_unstale":"2024-06-03T09:06:07.939453+0000","last_undegraded":"2024-06-03T09:06:07.939453+0000","last_fullsized":"2024-06-03T09:06:07.939453+0000","mapping_epoch":28,"log_start":"0'0","ondisk_log_start":"0'0","created":28,"last_epoch_clean":42,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-06-03T08:37:55.614497+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-06-03T08:37:55.614497+0000","last_clean_scrub_stamp":"2024-06-03T08:37:55.614497+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-06-04T09:29:46.427317+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,3],"acting":[4,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"17.1a","version":"0'0","reported_seq":956,"reported_epoch":1004,"state":"active+undersized","last_fresh":"2024-06-03T09:06:07.954840+0000","last_change":"2024-06-03T09:06:03.390734+0000","last_active":"2024-06-03T09:06:07.954840+0000","last_peered":"2024-06-03T09:06:07.954840+0000","last_clean":"2024-06-03T09:05:20.322313+0000","last_became_active":"2024-06-03T09:06:03.390734+0000","last_became_peered":"2024-06-03T09:06:03.390734+0000","last_unstale":"2024-06-03T09:06:07.954840+0000","last_undegraded":"2024-06-03T09:06:07.954840+0000","last_fullsized":"2024-06-03T09:06:03.343197+0000","mapping_epoch":999,"log_start":"0'0","ondisk_log_start":"0'0","created":28,"last_epoch_clean":254,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-06-03T08:37:55.614497+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-06-03T08:37:55.614497+0000","last_clean_scrub_stamp":"2024-06-03T08:37:55.614497+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-06-04T15:56:47.628875+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4],"acting":[4],"avail_no_missing":["4"],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"17.1b","version":"0'0","reported_seq":934,"reported_epoch":1008,"state":"active+clean","last_fresh":"2024-06-03T09:06:12.355403+0000","last_change":"2024-06-03T08:38:03.115748+0000","last_active":"2024-06-03T09:06:12.355403+0000","last_peered":"2024-06-03T09:06:12.355403+0000","last_clean":"2024-06-03T09:06:12.355403+0000","last_became_active":"2024-06-03T08:38:03.110533+0000","last_became_peered":"2024-06-03T08:38:03.110533+0000","last_unstale":"2024-06-03T09:06:12.355403+0000","last_undegraded":"2024-06-03T09:06:12.355403+0000","last_fullsized":"2024-06-03T09:06:12.355403+0000","mapping_epoch":28,"log_start":"0'0","ondisk_log_start":"0'0","created":28,"last_epoch_clean":32,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-06-03T08:37:55.614497+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-06-03T08:37:55.614497+0000","last_clean_scrub_stamp":"2024-06-03T08:37:55.614497+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-06-04T12:36:47.471609+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,7],"acting":[3,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"17.18","version":"0'0","reported_seq":943,"reported_epoch":1008,"state":"active+clean","last_fresh":"2024-06-03T09:06:12.868791+0000","last_change":"2024-06-03T08:38:04.632864+0000","last_active":"2024-06-03T09:06:12.868791+0000","last_peered":"2024-06-03T09:06:12.868791+0000","last_clean":"2024-06-03T09:06:12.868791+0000","last_became_active":"2024-06-03T08:38:04.627856+0000","last_became_peered":"2024-06-03T08:38:04.627856+0000","last_unstale":"2024-06-03T09:06:12.868791+0000","last_undegraded":"2024-06-03T09:06:12.868791+0000","last_fullsized":"2024-06-03T09:06:12.868791+0000","mapping_epoch":28,"log_start":"0'0","ondisk_log_start":"0'0","created":28,"last_epoch_clean":33,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-06-03T08:37:55.614497+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-06-03T08:37:55.614497+0000","last_clean_scrub_stamp":"2024-06-03T08:37:55.614497+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-06-04T19:55:51.181598+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,4],"acting":[7,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"17.19","version":"0'0","reported_seq":933,"reported_epoch":1008,"state":"active+clean","last_fresh":"2024-06-03T09:06:12.336384+0000","last_change":"2024-06-03T08:38:01.839491+0000","last_active":"2024-06-03T09:06:12.336384+0000","last_peered":"2024-06-03T09:06:12.336384+0000","last_clean":"2024-06-03T09:06:12.336384+0000","last_became_active":"2024-06-03T08:38:01.829539+0000","last_became_peered":"2024-06-03T08:38:01.829539+0000","last_unstale":"2024-06-03T09:06:12.336384+0000","last_undegraded":"2024-06-03T09:06:12.336384+0000","last_fullsized":"2024-06-03T09:06:12.336384+0000","mapping_epoch":28,"log_start":"0'0","ondisk_log_start":"0'0","created":28,"last_epoch_clean":30,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-06-03T08:37:55.614497+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-06-03T08:37:55.614497+0000","last_clean_scrub_stamp":"2024-06-03T08:37:55.614497+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-06-04T09:10:43.860229+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,2],"acting":[3,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"17.1e","version":"0'0","reported_seq":144,"reported_epoch":1004,"state":"active+undersized","last_fresh":"2024-06-03T09:06:07.982178+0000","last_change":"2024-06-03T09:06:03.406181+0000","last_active":"2024-06-03T09:06:07.982178+0000","last_peered":"2024-06-03T09:06:07.982178+0000","last_clean":"2024-06-03T08:37:55.614497+0000","last_became_active":"2024-06-03T09:06:03.406181+0000","last_became_peered":"2024-06-03T09:06:03.406181+0000","last_unstale":"2024-06-03T09:06:07.982178+0000","last_undegraded":"2024-06-03T09:06:07.982178+0000","last_fullsized":"2024-06-03T09:06:03.363739+0000","mapping_epoch":999,"log_start":"0'0","ondisk_log_start":"0'0","created":28,"last_epoch_clean":255,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-06-03T08:37:55.614497+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-06-03T08:37:55.614497+0000","last_clean_scrub_stamp":"2024-06-03T08:37:55.614497+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-06-04T11:45:24.486600+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4],"acting":[4],"avail_no_missing":["4"],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"17.1f","version":"0'0","reported_seq":939,"reported_epoch":1007,"state":"active+clean","last_fresh":"2024-06-03T09:06:11.314307+0000","last_change":"2024-06-03T08:38:04.383538+0000","last_active":"2024-06-03T09:06:11.314307+0000","last_peered":"2024-06-03T09:06:11.314307+0000","last_clean":"2024-06-03T09:06:11.314307+0000","last_became_active":"2024-06-03T08:38:04.367278+0000","last_became_peered":"2024-06-03T08:38:04.367278+0000","last_unstale":"2024-06-03T09:06:11.314307+0000","last_undegraded":"2024-06-03T09:06:11.314307+0000","last_fullsized":"2024-06-03T09:06:11.314307+0000","mapping_epoch":28,"log_start":"0'0","ondisk_log_start":"0'0","created":28,"last_epoch_clean":32,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-06-03T08:37:55.614497+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-06-03T08:37:55.614497+0000","last_clean_scrub_stamp":"2024-06-03T08:37:55.614497+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-06-04T10:39:26.552282+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,2],"acting":[0,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"17.1c","version":"0'0","reported_seq":942,"reported_epoch":1008,"state":"active+undersized","last_fresh":"2024-06-03T09:06:12.880704+0000","last_change":"2024-06-03T09:06:03.382210+0000","last_active":"2024-06-03T09:06:12.880704+0000","last_peered":"2024-06-03T09:06:12.880704+0000","last_clean":"2024-06-03T09:04:43.199660+0000","last_became_active":"2024-06-03T09:06:03.382210+0000","last_became_peered":"2024-06-03T09:06:03.382210+0000","last_unstale":"2024-06-03T09:06:12.880704+0000","last_undegraded":"2024-06-03T09:06:12.880704+0000","last_fullsized":"2024-06-03T09:06:03.277874+0000","mapping_epoch":999,"log_start":"0'0","ondisk_log_start":"0'0","created":28,"last_epoch_clean":255,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-06-03T08:37:55.614497+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-06-03T08:37:55.614497+0000","last_clean_scrub_stamp":"2024-06-03T08:37:55.614497+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-06-04T12:18:50.013884+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5],"acting":[5],"avail_no_missing":["5"],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"17.1d","version":"0'0","reported_seq":922,"reported_epoch":1008,"state":"active+clean","last_fresh":"2024-06-03T09:06:12.881860+0000","last_change":"2024-06-03T08:38:06.794630+0000","last_active":"2024-06-03T09:06:12.881860+0000","last_peered":"2024-06-03T09:06:12.881860+0000","last_clean":"2024-06-03T09:06:12.881860+0000","last_became_active":"2024-06-03T08:38:06.592317+0000","last_became_peered":"2024-06-03T08:38:06.592317+0000","last_unstale":"2024-06-03T09:06:12.881860+0000","last_undegraded":"2024-06-03T09:06:12.881860+0000","last_fullsized":"2024-06-03T09:06:12.881860+0000","mapping_epoch":28,"log_start":"0'0","ondisk_log_start":"0'0","created":28,"last_epoch_clean":33,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-06-03T08:37:55.614497+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-06-03T08:37:55.614497+0000","last_clean_scrub_stamp":"2024-06-03T08:37:55.614497+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-06-04T12:33:30.785369+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,7],"acting":[1,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"17.2","version":"0'0","reported_seq":924,"reported_epoch":1008,"state":"active+clean","last_fresh":"2024-06-03T09:06:12.869501+0000","last_change":"2024-06-03T08:38:14.981410+0000","last_active":"2024-06-03T09:06:12.869501+0000","last_peered":"2024-06-03T09:06:12.869501+0000","last_clean":"2024-06-03T09:06:12.869501+0000","last_became_active":"2024-06-03T08:38:14.973825+0000","last_became_peered":"2024-06-03T08:38:14.973825+0000","last_unstale":"2024-06-03T09:06:12.869501+0000","last_undegraded":"2024-06-03T09:06:12.869501+0000","last_fullsized":"2024-06-03T09:06:12.869501+0000","mapping_epoch":28,"log_start":"0'0","ondisk_log_start":"0'0","created":28,"last_epoch_clean":41,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-06-03T08:37:55.614497+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-06-03T08:37:55.614497+0000","last_clean_scrub_stamp":"2024-06-03T08:37:55.614497+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-06-04T17:01:05.518613+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,4],"acting":[1,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"17.3","version":"0'0","reported_seq":941,"reported_epoch":1007,"state":"active+clean","last_fresh":"2024-06-03T09:06:11.338843+0000","last_change":"2024-06-03T08:38:14.619745+0000","last_active":"2024-06-03T09:06:11.338843+0000","last_peered":"2024-06-03T09:06:11.338843+0000","last_clean":"2024-06-03T09:06:11.338843+0000","last_became_active":"2024-06-03T08:38:14.510665+0000","last_became_peered":"2024-06-03T08:38:14.510665+0000","last_unstale":"2024-06-03T09:06:11.338843+0000","last_undegraded":"2024-06-03T09:06:11.338843+0000","last_fullsized":"2024-06-03T09:06:11.338843+0000","mapping_epoch":28,"log_start":"0'0","ondisk_log_start":"0'0","created":28,"last_epoch_clean":42,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-06-03T08:37:55.614497+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-06-03T08:37:55.614497+0000","last_clean_scrub_stamp":"2024-06-03T08:37:55.614497+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-06-04T12:03:47.392064+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,7],"acting":[0,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"17.0","version":"0'0","reported_seq":942,"reported_epoch":1007,"state":"active+clean","last_fresh":"2024-06-03T09:06:12.708393+0000","last_change":"2024-06-03T08:38:04.237118+0000","last_active":"2024-06-03T09:06:12.708393+0000","last_peered":"2024-06-03T09:06:12.708393+0000","last_clean":"2024-06-03T09:06:12.708393+0000","last_became_active":"2024-06-03T08:38:04.231954+0000","last_became_peered":"2024-06-03T08:38:04.231954+0000","last_unstale":"2024-06-03T09:06:12.708393+0000","last_undegraded":"2024-06-03T09:06:12.708393+0000","last_fullsized":"2024-06-03T09:06:12.708393+0000","mapping_epoch":28,"log_start":"0'0","ondisk_log_start":"0'0","created":28,"last_epoch_clean":32,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-06-03T08:37:55.614497+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-06-03T08:37:55.614497+0000","last_clean_scrub_stamp":"2024-06-03T08:37:55.614497+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-06-04T09:56:10.741637+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,2],"acting":[0,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"17.1","version":"65'7","reported_seq":958,"reported_epoch":1008,"state":"active+undersized+degraded","last_fresh":"2024-06-03T09:06:12.872100+0000","last_change":"2024-06-03T09:06:03.370660+0000","last_active":"2024-06-03T09:06:12.872100+0000","last_peered":"2024-06-03T09:06:12.872100+0000","last_clean":"2024-06-03T09:05:03.733864+0000","last_became_active":"2024-06-03T09:06:03.370660+0000","last_became_peered":"2024-06-03T09:06:03.370660+0000","last_unstale":"2024-06-03T09:06:12.872100+0000","last_undegraded":"2024-06-03T09:06:03.266076+0000","last_fullsized":"2024-06-03T09:06:03.264525+0000","mapping_epoch":999,"log_start":"0'0","ondisk_log_start":"0'0","created":28,"last_epoch_clean":255,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-06-03T08:37:55.614497+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-06-03T08:37:55.614497+0000","last_clean_scrub_stamp":"2024-06-03T08:37:55.614497+0000","objects_scrubbed":0,"log_size":7,"log_dups_size":0,"ondisk_log_size":7,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-06-04T11:55:18.194283+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":5,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":1,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":6,"num_read_kb":6,"num_write":7,"num_write_kb":7,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5],"acting":[5],"avail_no_missing":["5"],"object_location_counts":[{"shards":"5","objects":1}],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]}],"pool_stats":[{"poolid":2,"num_pg":8,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":14,"acting":14,"num_store_stats":7},{"poolid":1,"num_pg":1,"stat_sum":{"num_bytes":459280,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":96,"num_read_kb":82,"num_write":113,"num_write_kb":1372,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2,"num_bytes_recovered":397840,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":802816,"data_stored":795680,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":76,"ondisk_log_size":76,"up":2,"acting":2,"num_store_stats":2},{"poolid":17,"num_pg":32,"stat_sum":{"num_bytes":5,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":1,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":6,"num_read_kb":6,"num_write":7,"num_write_kb":7,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":8192,"data_stored":10,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":7,"ondisk_log_size":7,"up":52,"acting":52,"num_store_stats":8}],"osd_stats":[{"osd":6,"up_from":253,"seq":1086626726200,"num_pgs":32,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":54772,"kb_used_data":5608,"kb_used_omap":1,"kb_used_meta":49150,"kb_avail":94317068,"statfs":{"total":96636764160,"available":96580677632,"internally_reserved":0,"allocated":5742592,"data_stored":2767180,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1593,"internal_metadata":50330055},"hb_peers":[0,1,2,3,4,5,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":109,"apply_latency_ms":109,"commit_latency_ns":109000000,"apply_latency_ns":109000000},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Mon Jun 3 09:04:02 2024","interfaces":[{"interface":"back","average":{"1min":1582.4780000000001,"5min":332.72699999999998,"15min":124.113},"min":{"1min":4.4829999999999997,"5min":4.1959999999999997,"15min":4.1959999999999997},"max":{"1min":22766.882000000001,"5min":22766.882000000001,"15min":22766.882000000001},"last":21.917999999999999},{"interface":"front","average":{"1min":1579.5409999999999,"5min":334.94499999999999,"15min":125.84999999999999},"min":{"1min":4.8129999999999997,"5min":4.7539999999999996,"15min":4.4530000000000003},"max":{"1min":22766.289000000001,"5min":22766.289000000001,"15min":22766.289000000001},"last":25.408999999999999}]},{"osd":1,"last update":"Mon Jun 3 09:03:40 2024","interfaces":[{"interface":"back","average":{"1min":36.420999999999999,"5min":77.054000000000002,"15min":132.84800000000001},"min":{"1min":6.0380000000000003,"5min":5.8799999999999999,"15min":5.484},"max":{"1min":589.64099999999996,"5min":4138.1679999999997,"15min":9955.1299999999992},"last":26.84},{"interface":"front","average":{"1min":40.240000000000002,"5min":79.674999999999997,"15min":133.38999999999999},"min":{"1min":8.4589999999999996,"5min":8.4090000000000007,"15min":6.1390000000000002},"max":{"1min":588.60799999999995,"5min":4137.6530000000002,"15min":9827.5509999999995},"last":26.263000000000002}]},{"osd":2,"last update":"Mon Jun 3 09:03:40 2024","interfaces":[{"interface":"back","average":{"1min":12.613,"5min":24.600999999999999,"15min":21.088000000000001},"min":{"1min":6.9740000000000002,"5min":6.9740000000000002,"15min":6.5330000000000004},"max":{"1min":26.076000000000001,"5min":376.24900000000002,"15min":376.24900000000002},"last":24.899000000000001},{"interface":"front","average":{"1min":12.493,"5min":22.207999999999998,"15min":21.219000000000001},"min":{"1min":6.5810000000000004,"5min":6.5110000000000001,"15min":6.0750000000000002},"max":{"1min":29.690999999999999,"5min":271.51100000000002,"15min":434.77300000000002},"last":40.372}]},{"osd":3,"last update":"Mon Jun 3 09:03:40 2024","interfaces":[{"interface":"back","average":{"1min":14.093999999999999,"5min":24.143999999999998,"15min":29.376999999999999},"min":{"1min":10.196999999999999,"5min":6.2919999999999998,"15min":6.1719999999999997},"max":{"1min":24.832000000000001,"5min":197.84899999999999,"15min":1235.5799999999999},"last":24.353999999999999},{"interface":"front","average":{"1min":9.3260000000000005,"5min":18.463999999999999,"15min":27.631},"min":{"1min":6.125,"5min":6.125,"15min":6.125},"max":{"1min":28.166,"5min":200.173,"15min":1157.6780000000001},"last":23.454000000000001}]},{"osd":4,"last update":"Mon Jun 3 09:04:29 2024","interfaces":[{"interface":"back","average":{"1min":10.279,"5min":1253.809,"15min":450.79199999999997},"min":{"1min":7.2300000000000004,"5min":6.093,"15min":6.093},"max":{"1min":15.000999999999999,"5min":28093.096000000001,"15min":28093.096000000001},"last":101.45399999999999},{"interface":"front","average":{"1min":15.693,"5min":1279.759,"15min":463.87099999999998},"min":{"1min":10.551,"5min":7.7480000000000002,"15min":7.7480000000000002},"max":{"1min":27.898,"5min":28323.837,"15min":28323.837},"last":102.262}]},{"osd":5,"last update":"Mon Jun 3 09:04:10 2024","interfaces":[{"interface":"back","average":{"1min":13.728999999999999,"5min":24.670000000000002,"15min":23.815999999999999},"min":{"1min":10.401,"5min":7.516,"15min":7.516},"max":{"1min":25.138000000000002,"5min":320.85000000000002,"15min":445.041},"last":103.52500000000001},{"interface":"front","average":{"1min":13.847,"5min":29.283000000000001,"15min":23.393999999999998},"min":{"1min":7.944,"5min":6.266,"15min":6.266},"max":{"1min":18.817,"5min":530.56700000000001,"15min":530.56700000000001},"last":29.263999999999999}]},{"osd":7,"last update":"Mon Jun 3 09:03:40 2024","interfaces":[{"interface":"back","average":{"1min":20.568999999999999,"5min":30.286999999999999,"15min":27.977},"min":{"1min":13.874000000000001,"5min":10.58,"15min":9.2859999999999996},"max":{"1min":86.959000000000003,"5min":222.07599999999999,"15min":255.08799999999999},"last":37.618000000000002},{"interface":"front","average":{"1min":19.102,"5min":29.550000000000001,"15min":27.763000000000002},"min":{"1min":10.052,"5min":8.6709999999999994,"15min":8.5429999999999993},"max":{"1min":67.968999999999994,"5min":293.62099999999998,"15min":438.14600000000002},"last":111.985}]}]},{"osd":7,"up_from":20,"seq":85899346284,"num_pgs":9,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":58952,"kb_used_data":5692,"kb_used_omap":1,"kb_used_meta":53246,"kb_avail":94312888,"statfs":{"total":96636764160,"available":96576397312,"internally_reserved":0,"allocated":5828608,"data_stored":2735298,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1589,"internal_metadata":54524363},"hb_peers":[0,1,2,3,4,5],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":55,"apply_latency_ms":55,"commit_latency_ns":55000000,"apply_latency_ns":55000000},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Mon Jun 3 09:06:09 2024","interfaces":[{"interface":"back","average":{"1min":14.073,"5min":894.22400000000005,"15min":308.93400000000003},"min":{"1min":5.5709999999999997,"5min":4.218,"15min":4.0599999999999996},"max":{"1min":90.757999999999996,"5min":21729.001,"15min":21729.001},"last":11.128},{"interface":"front","average":{"1min":9.2550000000000008,"5min":906.23500000000001,"15min":314.06},"min":{"1min":5.5789999999999997,"5min":4.8810000000000002,"15min":4.8810000000000002},"max":{"1min":25.937000000000001,"5min":21899.732,"15min":21899.732},"last":16.053999999999998}]},{"osd":1,"last update":"Mon Jun 3 09:06:09 2024","interfaces":[{"interface":"back","average":{"1min":15.182,"5min":69.183999999999997,"15min":219.173},"min":{"1min":7.3639999999999999,"5min":6.6200000000000001,"15min":6.1319999999999997},"max":{"1min":76.926000000000002,"5min":2795.1329999999998,"15min":18373.257000000001},"last":8.6799999999999997},{"interface":"front","average":{"1min":21.495999999999999,"5min":74.703999999999994,"15min":225.285},"min":{"1min":8.8399999999999999,"5min":8.8399999999999999,"15min":8.5129999999999999},"max":{"1min":94.977999999999994,"5min":2803.3670000000002,"15min":18374.601999999999},"last":13.189}]},{"osd":2,"last update":"Mon Jun 3 09:06:09 2024","interfaces":[{"interface":"back","average":{"1min":19.164000000000001,"5min":26.902000000000001,"15min":22.920999999999999},"min":{"1min":11.207000000000001,"5min":8.5030000000000001,"15min":8.5030000000000001},"max":{"1min":91.319999999999993,"5min":333.99200000000002,"15min":333.99200000000002},"last":18.968},{"interface":"front","average":{"1min":11.125999999999999,"5min":18.132000000000001,"15min":16.353000000000002},"min":{"1min":6.202,"5min":5.8369999999999997,"15min":5.5069999999999997},"max":{"1min":28.649000000000001,"5min":226.18100000000001,"15min":226.18100000000001},"last":13.875}]},{"osd":3,"last update":"Mon Jun 3 09:06:09 2024","interfaces":[{"interface":"back","average":{"1min":22.291,"5min":26.289999999999999,"15min":31.074999999999999},"min":{"1min":11.714,"5min":6.8959999999999999,"15min":6.3760000000000003},"max":{"1min":83.102000000000004,"5min":316.64400000000001,"15min":2039.9059999999999},"last":15.42},{"interface":"front","average":{"1min":13.324999999999999,"5min":17.963000000000001,"15min":25.969999999999999},"min":{"1min":6.1200000000000001,"5min":5.2089999999999996,"15min":5.2089999999999996},"max":{"1min":49.908999999999999,"5min":233.542,"15min":2037.5550000000001},"last":7.6879999999999997}]},{"osd":4,"last update":"Mon Jun 3 09:05:28 2024","interfaces":[{"interface":"back","average":{"1min":22.209,"5min":1379.51,"15min":1803.278},"min":{"1min":5.9089999999999998,"5min":5.9089999999999998,"15min":5.9089999999999998},"max":{"1min":224.54300000000001,"5min":30073.917000000001,"15min":62560.868000000002},"last":10.279},{"interface":"front","average":{"1min":22.712,"5min":1401.0719999999999,"15min":1806.759},"min":{"1min":7.6449999999999996,"5min":6.7930000000000001,"15min":5.9340000000000002},"max":{"1min":223.58600000000001,"5min":30348.021000000001,"15min":62451.446000000004},"last":16.579000000000001}]},{"osd":5,"last update":"Mon Jun 3 09:06:02 2024","interfaces":[{"interface":"back","average":{"1min":24.372,"5min":26.856000000000002,"15min":23.571999999999999},"min":{"1min":12.278,"5min":9.2460000000000004,"15min":6.5650000000000004},"max":{"1min":183.64099999999999,"5min":317.572,"15min":317.572},"last":10.933999999999999},{"interface":"front","average":{"1min":21.259,"5min":23.364000000000001,"15min":21.193999999999999},"min":{"1min":7.8499999999999996,"5min":7.0999999999999996,"15min":7.008},"max":{"1min":184.53399999999999,"5min":306.60599999999999,"15min":306.60599999999999},"last":18.036000000000001}]},{"osd":6,"last update":"Mon Jun 3 09:04:31 2024","interfaces":[{"interface":"back","average":{"1min":44.061,"5min":26.821000000000002,"15min":91.176000000000002},"min":{"1min":9.0800000000000001,"5min":7.9390000000000001,"15min":7.9390000000000001},"max":{"1min":268.029,"5min":308.916,"15min":8734.4639999999999},"last":6.6769999999999996},{"interface":"front","average":{"1min":46.643000000000001,"5min":23.879999999999999,"15min":88.765000000000001},"min":{"1min":8.8119999999999994,"5min":6.9240000000000004,"15min":6.742},"max":{"1min":305.64600000000002,"5min":305.64600000000002,"15min":8735.5319999999992},"last":12.327}]}]},{"osd":5,"up_from":20,"seq":85899346258,"num_pgs":9,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":59340,"kb_used_data":6076,"kb_used_omap":1,"kb_used_meta":53246,"kb_avail":94312500,"statfs":{"total":96636764160,"available":96576000000,"internally_reserved":0,"allocated":6221824,"data_stored":3126752,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1594,"internal_metadata":54524358},"hb_peers":[0,1,2,3,4,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":51,"apply_latency_ms":51,"commit_latency_ns":51000000,"apply_latency_ns":51000000},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Mon Jun 3 09:06:06 2024","interfaces":[{"interface":"back","average":{"1min":7.468,"5min":1703.037,"15min":580.51900000000001},"min":{"1min":4.6509999999999998,"5min":4.6509999999999998,"15min":4.4020000000000001},"max":{"1min":14.711,"5min":26433.764999999999,"15min":26433.764999999999},"last":7.9260000000000002},{"interface":"front","average":{"1min":9.9429999999999996,"5min":1725.27,"15min":587.87800000000004},"min":{"1min":5.5999999999999996,"5min":5.1319999999999997,"15min":4.8170000000000002},"max":{"1min":13.590999999999999,"5min":26704.698,"15min":26704.698},"last":10.76}]},{"osd":1,"last update":"Mon Jun 3 09:06:00 2024","interfaces":[{"interface":"back","average":{"1min":10.862,"5min":67.731999999999999,"15min":217.364},"min":{"1min":7.1769999999999996,"5min":6.0890000000000004,"15min":5.4930000000000003},"max":{"1min":16.998999999999999,"5min":3923.4090000000001,"15min":18811.696},"last":11.622999999999999},{"interface":"front","average":{"1min":10.417,"5min":65.141999999999996,"15min":220.75700000000001},"min":{"1min":5.3209999999999997,"5min":5.3209999999999997,"15min":5.0209999999999999},"max":{"1min":15.821999999999999,"5min":4019.6129999999998,"15min":18968.365000000002},"last":12.172000000000001}]},{"osd":2,"last update":"Mon Jun 3 09:06:00 2024","interfaces":[{"interface":"back","average":{"1min":11.366,"5min":24.18,"15min":21.137},"min":{"1min":5.6369999999999996,"5min":5.6369999999999996,"15min":5.0510000000000002},"max":{"1min":15.529999999999999,"5min":332.48200000000003,"15min":332.48200000000003},"last":13.997},{"interface":"front","average":{"1min":10.089,"5min":23.117000000000001,"15min":24.050000000000001},"min":{"1min":6.6769999999999996,"5min":6.6769999999999996,"15min":5.8739999999999997},"max":{"1min":15.821,"5min":353.03199999999998,"15min":353.03199999999998},"last":9.0280000000000005}]},{"osd":3,"last update":"Mon Jun 3 09:06:00 2024","interfaces":[{"interface":"back","average":{"1min":13.472,"5min":24.283999999999999,"15min":31.666},"min":{"1min":10.111000000000001,"5min":7.4720000000000004,"15min":7.4210000000000003},"max":{"1min":16.271999999999998,"5min":353.57499999999999,"15min":2855.8519999999999},"last":19.434999999999999},{"interface":"front","average":{"1min":9.5899999999999999,"5min":15.875,"15min":25.834},"min":{"1min":6.6609999999999996,"5min":5.7729999999999997,"15min":5.1909999999999998},"max":{"1min":15.112,"5min":203.804,"15min":2869.7860000000001},"last":18.408000000000001}]},{"osd":4,"last update":"Mon Jun 3 09:05:27 2024","interfaces":[{"interface":"back","average":{"1min":19.155000000000001,"5min":1084.961,"15min":1346.9949999999999},"min":{"1min":7.6130000000000004,"5min":7.6130000000000004,"15min":6.1340000000000003},"max":{"1min":156.55199999999999,"5min":28619.825000000001,"15min":57999.453999999998},"last":13.683},{"interface":"front","average":{"1min":28.068999999999999,"5min":1087.106,"15min":1330.289},"min":{"1min":8.3949999999999996,"5min":6.5030000000000001,"15min":6.5030000000000001},"max":{"1min":254.91300000000001,"5min":28621.294000000002,"15min":57434.631999999998},"last":14.867000000000001}]},{"osd":6,"last update":"Mon Jun 3 09:03:57 2024","interfaces":[{"interface":"back","average":{"1min":23.536000000000001,"5min":36.457000000000001,"15min":83.799999999999997},"min":{"1min":9.4800000000000004,"5min":9.4800000000000004,"15min":8.0630000000000006},"max":{"1min":89.442999999999998,"5min":508.77999999999997,"15min":9193.5169999999998},"last":157.79300000000001},{"interface":"front","average":{"1min":18.891999999999999,"5min":22.100999999999999,"15min":75.489999999999995},"min":{"1min":5.923,"5min":5.923,"15min":5.923},"max":{"1min":78.424999999999997,"5min":464.62,"15min":9191.4429999999993},"last":148.09}]},{"osd":7,"last update":"Mon Jun 3 09:06:00 2024","interfaces":[{"interface":"back","average":{"1min":13.257,"5min":20.917999999999999,"15min":21.088999999999999},"min":{"1min":10.475,"5min":6.7629999999999999,"15min":6.7629999999999999},"max":{"1min":17.245000000000001,"5min":211.22999999999999,"15min":211.22999999999999},"last":15.555},{"interface":"front","average":{"1min":12.872999999999999,"5min":20.587,"15min":24.021000000000001},"min":{"1min":8.5760000000000005,"5min":7.7709999999999999,"15min":6.8570000000000002},"max":{"1min":18.359999999999999,"5min":223.191,"15min":286.69499999999999},"last":9.7189999999999994}]}]},{"osd":4,"up_from":20,"seq":85899346265,"num_pgs":20,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":63000,"kb_used_data":5644,"kb_used_omap":1,"kb_used_meta":57342,"kb_avail":94308840,"statfs":{"total":96636764160,"available":96572252160,"internally_reserved":0,"allocated":5779456,"data_stored":2718472,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1594,"internal_metadata":58718662},"hb_peers":[0,1,2,3,5,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":50,"apply_latency_ms":50,"commit_latency_ns":50000000,"apply_latency_ns":50000000},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Mon Jun 3 09:05:34 2024","interfaces":[{"interface":"back","average":{"1min":12.416,"5min":1099.4649999999999,"15min":382.15199999999999},"min":{"1min":4.5279999999999996,"5min":4.5279999999999996,"15min":4.3220000000000001},"max":{"1min":123.878,"5min":21726.743999999999,"15min":21726.743999999999},"last":52.241},{"interface":"front","average":{"1min":16.462,"5min":1108.894,"15min":387.70299999999997},"min":{"1min":6.4089999999999998,"5min":5.8380000000000001,"15min":5.4909999999999997},"max":{"1min":130.185,"5min":21727.993999999999,"15min":21727.993999999999},"last":53.148000000000003}]},{"osd":1,"last update":"Mon Jun 3 09:05:34 2024","interfaces":[{"interface":"back","average":{"1min":14.727,"5min":56.755000000000003,"15min":341.09399999999999},"min":{"1min":4.6529999999999996,"5min":4.1970000000000001,"15min":4.1970000000000001},"max":{"1min":126.899,"5min":3346.4389999999999,"15min":22256.098000000002},"last":47.073},{"interface":"front","average":{"1min":13.472,"5min":57.503,"15min":340.303},"min":{"1min":5.0590000000000002,"5min":5.0590000000000002,"15min":4.2640000000000002},"max":{"1min":126.096,"5min":3347.181,"15min":22247.317999999999},"last":49.558}]},{"osd":2,"last update":"Mon Jun 3 09:05:34 2024","interfaces":[{"interface":"back","average":{"1min":26.088999999999999,"5min":23.866,"15min":20.620999999999999},"min":{"1min":7.0869999999999997,"5min":6.8239999999999998,"15min":6.3150000000000004},"max":{"1min":337.60399999999998,"5min":393.58600000000001,"15min":393.58600000000001},"last":85.340000000000003},{"interface":"front","average":{"1min":15.800000000000001,"5min":22.584,"15min":23.245999999999999},"min":{"1min":6.0629999999999997,"5min":6.0629999999999997,"15min":6.0629999999999997},"max":{"1min":127.913,"5min":394.471,"15min":394.471},"last":82.120000000000005}]},{"osd":3,"last update":"Mon Jun 3 09:05:34 2024","interfaces":[{"interface":"back","average":{"1min":17.544,"5min":19.143999999999998,"15min":18.742000000000001},"min":{"1min":5.8719999999999999,"5min":5.4100000000000001,"15min":5.4000000000000004},"max":{"1min":127.43600000000001,"5min":322.63,"15min":322.63},"last":84.555999999999997},{"interface":"front","average":{"1min":21.327000000000002,"5min":23.559000000000001,"15min":23.797000000000001},"min":{"1min":10.023,"5min":7.484,"15min":7.484},"max":{"1min":204.517,"5min":392.67500000000001,"15min":392.67500000000001},"last":71.394000000000005}]},{"osd":5,"last update":"Mon Jun 3 09:05:34 2024","interfaces":[{"interface":"back","average":{"1min":24.274999999999999,"5min":21.274999999999999,"15min":20.988},"min":{"1min":6.7489999999999997,"5min":6.7489999999999997,"15min":6.7489999999999997},"max":{"1min":268.10199999999998,"5min":344.23399999999998,"15min":344.23399999999998},"last":51.326000000000001},{"interface":"front","average":{"1min":26.361999999999998,"5min":25.218,"15min":26.084},"min":{"1min":12.228,"5min":8.5180000000000007,"15min":8.5180000000000007},"max":{"1min":264.75,"5min":345.18700000000001,"15min":345.18700000000001},"last":40.829000000000001}]},{"osd":6,"last update":"Mon Jun 3 09:04:32 2024","interfaces":[{"interface":"back","average":{"1min":15.548,"5min":32.957999999999998,"15min":152.738},"min":{"1min":9.141,"5min":9.141,"15min":8.2319999999999993},"max":{"1min":52.57,"5min":395.238,"15min":12496.335999999999},"last":208.90000000000001},{"interface":"front","average":{"1min":12.680999999999999,"5min":25.428000000000001,"15min":149.30600000000001},"min":{"1min":6.4379999999999997,"5min":5.6319999999999997,"15min":5.6319999999999997},"max":{"1min":46.283000000000001,"5min":377.74299999999999,"15min":12616.297},"last":205.88900000000001}]},{"osd":7,"last update":"Mon Jun 3 09:05:34 2024","interfaces":[{"interface":"back","average":{"1min":18.295999999999999,"5min":19.585999999999999,"15min":21.062000000000001},"min":{"1min":6.2110000000000003,"5min":6.2110000000000003,"15min":5.8010000000000002},"max":{"1min":207.25299999999999,"5min":358.94900000000001,"15min":358.94900000000001},"last":14.252000000000001},{"interface":"front","average":{"1min":23.712,"5min":24.024999999999999,"15min":24.640000000000001},"min":{"1min":11.824999999999999,"5min":7.7789999999999999,"15min":7.7789999999999999},"max":{"1min":209.32900000000001,"5min":346.25200000000001,"15min":380.06900000000002},"last":45.639000000000003}]}]},{"osd":2,"up_from":20,"seq":85899346285,"num_pgs":24,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":58952,"kb_used_data":5688,"kb_used_omap":1,"kb_used_meta":53246,"kb_avail":94312888,"statfs":{"total":96636764160,"available":96576397312,"internally_reserved":0,"allocated":5824512,"data_stored":2763528,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1594,"internal_metadata":54524358},"hb_peers":[0,1,3,4,5,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":79,"apply_latency_ms":79,"commit_latency_ns":79000000,"apply_latency_ns":79000000},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Mon Jun 3 09:05:21 2024","interfaces":[{"interface":"back","average":{"1min":16.364000000000001,"5min":947.59799999999996,"15min":326.58999999999997},"min":{"1min":6.0830000000000002,"5min":4.7050000000000001,"15min":4.1420000000000003},"max":{"1min":157.60900000000001,"5min":23465.18,"15min":23465.18},"last":8.3209999999999997},{"interface":"front","average":{"1min":12.276,"5min":945.29899999999998,"15min":326.78399999999999},"min":{"1min":5.1989999999999998,"5min":4.7439999999999998,"15min":4.6950000000000003},"max":{"1min":102.557,"5min":23465.966,"15min":23465.966},"last":7.2119999999999997}]},{"osd":1,"last update":"Mon Jun 3 09:05:21 2024","interfaces":[{"interface":"back","average":{"1min":17.803999999999998,"5min":77.658000000000001,"15min":294.96899999999999},"min":{"1min":7.2130000000000001,"5min":5.4199999999999999,"15min":5.4199999999999999},"max":{"1min":159.624,"5min":4220.5290000000005,"15min":18954.987000000001},"last":6.4089999999999998},{"interface":"front","average":{"1min":21.542999999999999,"5min":83.528999999999996,"15min":303.11700000000002},"min":{"1min":8.8200000000000003,"5min":7.5949999999999998,"15min":7.5949999999999998},"max":{"1min":170.74299999999999,"5min":4325.9489999999996,"15min":19143.834999999999},"last":11.071}]},{"osd":3,"last update":"Mon Jun 3 09:05:21 2024","interfaces":[{"interface":"back","average":{"1min":19.164999999999999,"5min":13.284000000000001,"15min":28.751999999999999},"min":{"1min":7.3710000000000004,"5min":5.9290000000000003,"15min":5.2450000000000001},"max":{"1min":118.913,"5min":118.913,"15min":2594.8150000000001},"last":8.8699999999999992},{"interface":"front","average":{"1min":22.094000000000001,"5min":17.244,"15min":32.173999999999999},"min":{"1min":9.516,"5min":8.0760000000000005,"15min":6.0780000000000003},"max":{"1min":156.34700000000001,"5min":156.34700000000001,"15min":2706.5120000000002},"last":11.682}]},{"osd":4,"last update":"Mon Jun 3 09:05:57 2024","interfaces":[{"interface":"back","average":{"1min":10.573,"5min":1266.894,"15min":438.154},"min":{"1min":8.6379999999999999,"5min":7.8789999999999996,"15min":6.2039999999999997},"max":{"1min":15.105,"5min":26660.861000000001,"15min":26660.861000000001},"last":14.795},{"interface":"front","average":{"1min":12.407999999999999,"5min":1266.7460000000001,"15min":435.75400000000002},"min":{"1min":10.266999999999999,"5min":10.266999999999999,"15min":6.2720000000000002},"max":{"1min":15.228,"5min":26616.243999999999,"15min":26616.243999999999},"last":15.5}]},{"osd":5,"last update":"Mon Jun 3 09:05:21 2024","interfaces":[{"interface":"back","average":{"1min":25.632000000000001,"5min":20.437999999999999,"15min":21.472999999999999},"min":{"1min":9.2159999999999993,"5min":9.2159999999999993,"15min":6.2050000000000001},"max":{"1min":252.31100000000001,"5min":252.31100000000001,"15min":252.31100000000001},"last":13.285},{"interface":"front","average":{"1min":24.824000000000002,"5min":18.521000000000001,"15min":21.027000000000001},"min":{"1min":8.0449999999999999,"5min":7.4850000000000003,"15min":6.8810000000000002},"max":{"1min":258.31700000000001,"5min":258.31700000000001,"15min":258.31700000000001},"last":9.5879999999999992}]},{"osd":6,"last update":"Mon Jun 3 09:04:28 2024","interfaces":[{"interface":"back","average":{"1min":14.063000000000001,"5min":24.850000000000001,"15min":106.755},"min":{"1min":9.0820000000000007,"5min":8.9649999999999999,"15min":8.9649999999999999},"max":{"1min":24.666,"5min":188.06700000000001,"15min":9521.0010000000002},"last":183.60900000000001},{"interface":"front","average":{"1min":13.933999999999999,"5min":21.081,"15min":106.142},"min":{"1min":5.8719999999999999,"5min":5.8719999999999999,"15min":5.8719999999999999},"max":{"1min":117.551,"5min":193.887,"15min":9585.6620000000003},"last":184.20599999999999}]},{"osd":7,"last update":"Mon Jun 3 09:05:21 2024","interfaces":[{"interface":"back","average":{"1min":25.097000000000001,"5min":22.579999999999998,"15min":24.388000000000002},"min":{"1min":7.8959999999999999,"5min":6.6130000000000004,"15min":6.6130000000000004},"max":{"1min":242.56800000000001,"5min":242.56800000000001,"15min":242.56800000000001},"last":12.548},{"interface":"front","average":{"1min":24.414999999999999,"5min":23.995999999999999,"15min":26.614999999999998},"min":{"1min":12.43,"5min":9.0660000000000007,"15min":9.0660000000000007},"max":{"1min":181.56399999999999,"5min":182.5,"15min":214.33799999999999},"last":13.968999999999999}]}]},{"osd":3,"up_from":14,"seq":60129542500,"num_pgs":11,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":58940,"kb_used_data":5680,"kb_used_omap":1,"kb_used_meta":53246,"kb_avail":94312900,"statfs":{"total":96636764160,"available":96576409600,"internally_reserved":0,"allocated":5816320,"data_stored":2728907,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1592,"internal_metadata":54524360},"hb_peers":[0,1,2,4,5,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":20,"apply_latency_ms":20,"commit_latency_ns":20000000,"apply_latency_ns":20000000},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Mon Jun 3 09:05:25 2024","interfaces":[{"interface":"back","average":{"1min":8.4809999999999999,"5min":1064.2270000000001,"15min":367.88600000000002},"min":{"1min":6.0529999999999999,"5min":5.173,"15min":4.6929999999999996},"max":{"1min":12.010999999999999,"5min":23275.167000000001,"15min":23275.167000000001},"last":8.0489999999999995},{"interface":"front","average":{"1min":13.885,"5min":1070.3710000000001,"15min":373.53500000000003},"min":{"1min":9.6679999999999993,"5min":7.2089999999999996,"15min":5.9880000000000004},"max":{"1min":32.426000000000002,"5min":23278.536,"15min":23278.536},"last":10.086}]},{"osd":1,"last update":"Mon Jun 3 09:05:25 2024","interfaces":[{"interface":"back","average":{"1min":11.324,"5min":19.920000000000002,"15min":390.07900000000001},"min":{"1min":6.6379999999999999,"5min":4.8200000000000003,"15min":4.8200000000000003},"max":{"1min":26.449000000000002,"5min":250.72300000000001,"15min":21658.205999999998},"last":4.335},{"interface":"front","average":{"1min":9.468,"5min":20.966999999999999,"15min":381.18799999999999},"min":{"1min":5.4969999999999999,"5min":5.2779999999999996,"15min":5.2779999999999996},"max":{"1min":25.643000000000001,"5min":335.33699999999999,"15min":21307.069},"last":7.3360000000000003}]},{"osd":2,"last update":"Mon Jun 3 09:05:25 2024","interfaces":[{"interface":"back","average":{"1min":13.215,"5min":26.649000000000001,"15min":23.783999999999999},"min":{"1min":7.8710000000000004,"5min":6.4900000000000002,"15min":5.875},"max":{"1min":34.322000000000003,"5min":377.803,"15min":377.803},"last":11.356999999999999},{"interface":"front","average":{"1min":13.105,"5min":22.292999999999999,"15min":20.573},"min":{"1min":7.327,"5min":6.125,"15min":5.9020000000000001},"max":{"1min":32.756999999999998,"5min":371.601,"15min":371.601},"last":8.9670000000000005}]},{"osd":4,"last update":"Mon Jun 3 09:05:27 2024","interfaces":[{"interface":"back","average":{"1min":10.444000000000001,"5min":1211.4970000000001,"15min":1639.1579999999999},"min":{"1min":6.4429999999999996,"5min":6.2560000000000002,"15min":6.0309999999999997},"max":{"1min":21.263000000000002,"5min":30144.939999999999,"15min":62679.622000000003},"last":14.975},{"interface":"front","average":{"1min":11.363,"5min":1219.6400000000001,"15min":1644.127},"min":{"1min":7.8109999999999999,"5min":7.8109999999999999,"15min":6.2439999999999998},"max":{"1min":20.108000000000001,"5min":30180.689999999999,"15min":62681.021999999997},"last":14.266}]},{"osd":5,"last update":"Mon Jun 3 09:06:07 2024","interfaces":[{"interface":"back","average":{"1min":12.802,"5min":19.640999999999998,"15min":22.318000000000001},"min":{"1min":9.0890000000000004,"5min":6.8810000000000002,"15min":6.7690000000000001},"max":{"1min":16.745999999999999,"5min":249.88,"15min":327.34899999999999},"last":11.952},{"interface":"front","average":{"1min":15.002000000000001,"5min":24.003,"15min":26.649999999999999},"min":{"1min":11.984,"5min":9.9149999999999991,"15min":8.9879999999999995},"max":{"1min":18.407,"5min":251.47800000000001,"15min":331.86799999999999},"last":13.717000000000001}]},{"osd":6,"last update":"Mon Jun 3 09:04:04 2024","interfaces":[{"interface":"back","average":{"1min":22.815999999999999,"5min":31.513999999999999,"15min":181.74299999999999},"min":{"1min":9.4809999999999999,"5min":7.2240000000000002,"15min":7.2240000000000002},"max":{"1min":160.56999999999999,"5min":380.613,"15min":11893.74},"last":19.359999999999999},{"interface":"front","average":{"1min":25.359999999999999,"5min":30.981000000000002,"15min":179.81399999999999},"min":{"1min":9.8870000000000005,"5min":7.3360000000000003,"15min":6.0899999999999999},"max":{"1min":193.11099999999999,"5min":379.80700000000002,"15min":11893.069},"last":8.202}]},{"osd":7,"last update":"Mon Jun 3 09:05:25 2024","interfaces":[{"interface":"back","average":{"1min":12.177,"5min":21.126000000000001,"15min":22.972000000000001},"min":{"1min":8.6769999999999996,"5min":7.4279999999999999,"15min":7.4279999999999999},"max":{"1min":17.376999999999999,"5min":164.27500000000001,"15min":178.76300000000001},"last":9.5540000000000003},{"interface":"front","average":{"1min":15.675000000000001,"5min":22.722000000000001,"15min":25.238},"min":{"1min":11.233000000000001,"5min":10.555999999999999,"15min":7.6440000000000001},"max":{"1min":29.280000000000001,"5min":148.13800000000001,"15min":417.22800000000001},"last":12.855}]}]},{"osd":1,"up_from":14,"seq":60129542498,"num_pgs":9,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":59332,"kb_used_data":6072,"kb_used_omap":1,"kb_used_meta":53246,"kb_avail":94312508,"statfs":{"total":96636764160,"available":96576008192,"internally_reserved":0,"allocated":6217728,"data_stored":3126747,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":54524357},"hb_peers":[0,2,3,4,5,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":65,"apply_latency_ms":65,"commit_latency_ns":65000000,"apply_latency_ns":65000000},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Mon Jun 3 09:06:07 2024","interfaces":[{"interface":"back","average":{"1min":7.7530000000000001,"5min":185.66999999999999,"15min":77.347999999999999},"min":{"1min":5.0229999999999997,"5min":4.9279999999999999,"15min":4.5449999999999999},"max":{"1min":11.896000000000001,"5min":7470.616,"15min":7470.616},"last":10.471},{"interface":"front","average":{"1min":9.9220000000000006,"5min":187.14599999999999,"15min":76.325999999999993},"min":{"1min":5.3159999999999998,"5min":5.3159999999999998,"15min":5.3159999999999998},"max":{"1min":18.338000000000001,"5min":7477.0829999999996,"15min":7477.0829999999996},"last":5.7300000000000004}]},{"osd":2,"last update":"Mon Jun 3 09:05:18 2024","interfaces":[{"interface":"back","average":{"1min":10.954000000000001,"5min":18.295000000000002,"15min":24.126999999999999},"min":{"1min":6.8739999999999997,"5min":5.8650000000000002,"15min":5.8650000000000002},"max":{"1min":21.122,"5min":228,"15min":657.38},"last":8.5600000000000005},{"interface":"front","average":{"1min":13.694000000000001,"5min":24.446999999999999,"15min":32.055999999999997},"min":{"1min":9.9860000000000007,"5min":9.1229999999999993,"15min":7.2160000000000002},"max":{"1min":26.507999999999999,"5min":229.465,"15min":700.06500000000005},"last":12.885999999999999}]},{"osd":3,"last update":"Mon Jun 3 09:06:04 2024","interfaces":[{"interface":"back","average":{"1min":8.6110000000000007,"5min":15.733000000000001,"15min":26.646999999999998},"min":{"1min":5.827,"5min":5.4640000000000004,"15min":5.4640000000000004},"max":{"1min":13.105,"5min":121.619,"15min":1676.009},"last":6.3609999999999998},{"interface":"front","average":{"1min":9.5850000000000009,"5min":16.161999999999999,"15min":29.065999999999999},"min":{"1min":6.3630000000000004,"5min":5.7699999999999996,"15min":5.7699999999999996},"max":{"1min":15.166,"5min":132.036,"15min":1766.5840000000001},"last":12.067}]},{"osd":4,"last update":"Mon Jun 3 09:05:31 2024","interfaces":[{"interface":"back","average":{"1min":10.996,"5min":954.46500000000003,"15min":1281.6859999999999},"min":{"1min":7.702,"5min":6.1760000000000002,"15min":6.0970000000000004},"max":{"1min":21.538,"5min":28044.862000000001,"15min":59438.017999999996},"last":7.4029999999999996},{"interface":"front","average":{"1min":13.646000000000001,"5min":961.24599999999998,"15min":1285.26},"min":{"1min":9.7200000000000006,"5min":9.3940000000000001,"15min":6.4640000000000004},"max":{"1min":23.277999999999999,"5min":28045.624,"15min":59439.283000000003},"last":9.8949999999999996}]},{"osd":5,"last update":"Mon Jun 3 09:06:04 2024","interfaces":[{"interface":"back","average":{"1min":12.478999999999999,"5min":20.228000000000002,"15min":25.997},"min":{"1min":8.6430000000000007,"5min":7.9720000000000004,"15min":7.234},"max":{"1min":17.442,"5min":139.43199999999999,"15min":701.22000000000003},"last":7.9050000000000002},{"interface":"front","average":{"1min":13.507999999999999,"5min":25.760999999999999,"15min":27.553999999999998},"min":{"1min":10.042999999999999,"5min":7.258,"15min":6.242},"max":{"1min":20.611000000000001,"5min":278.63,"15min":598.35000000000002},"last":11.395}]},{"osd":6,"last update":"Mon Jun 3 09:03:54 2024","interfaces":[{"interface":"back","average":{"1min":28.524000000000001,"5min":20.654,"15min":23.878},"min":{"1min":6.0309999999999997,"5min":6.0309999999999997,"15min":6.0309999999999997},"max":{"1min":167.221,"5min":167.221,"15min":704.245},"last":18.721},{"interface":"front","average":{"1min":27.606000000000002,"5min":30.131,"15min":28.628},"min":{"1min":9.8650000000000002,"5min":9.7200000000000006,"15min":8.3580000000000005},"max":{"1min":123.938,"5min":358.23000000000002,"15min":634.05499999999995},"last":173.113}]},{"osd":7,"last update":"Mon Jun 3 09:05:18 2024","interfaces":[{"interface":"back","average":{"1min":21.853999999999999,"5min":24.245999999999999,"15min":32.801000000000002},"min":{"1min":9.5020000000000007,"5min":8.0559999999999992,"15min":8.0559999999999992},"max":{"1min":142.245,"5min":270.95100000000002,"15min":703.07799999999997},"last":15.696},{"interface":"front","average":{"1min":21.547000000000001,"5min":29.977,"15min":30.513999999999999},"min":{"1min":11.167,"5min":8.2089999999999996,"15min":7.7599999999999998},"max":{"1min":119.348,"5min":360.99900000000002,"15min":608.44100000000003},"last":14.506}]}]},{"osd":0,"up_from":14,"seq":60129542500,"num_pgs":12,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":81088,"kb_used_data":5680,"kb_used_omap":1,"kb_used_meta":75390,"kb_avail":94290752,"statfs":{"total":96636764160,"available":96553730048,"internally_reserved":0,"allocated":5816320,"data_stored":2728907,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1593,"internal_metadata":77199815},"hb_peers":[1,2,3,4,5,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":4,"apply_latency_ms":4,"commit_latency_ns":4000000,"apply_latency_ns":4000000},"alerts":[],"network_ping_times":[{"osd":1,"last update":"Mon Jun 3 09:06:04 2024","interfaces":[{"interface":"back","average":{"1min":6.9960000000000004,"5min":56.997,"15min":339.29700000000003},"min":{"1min":4.4530000000000003,"5min":4.4530000000000003,"15min":4.4100000000000001},"max":{"1min":11.720000000000001,"5min":2414.4749999999999,"15min":20691.453000000001},"last":9.875},{"interface":"front","average":{"1min":9.548,"5min":57.258000000000003,"15min":337.791},"min":{"1min":5.2300000000000004,"5min":5.2300000000000004,"15min":4.6050000000000004},"max":{"1min":15.337999999999999,"5min":2412.9549999999999,"15min":20640.792000000001},"last":4.7199999999999998}]},{"osd":2,"last update":"Mon Jun 3 09:05:22 2024","interfaces":[{"interface":"back","average":{"1min":9.5820000000000007,"5min":27.760000000000002,"15min":23.097999999999999},"min":{"1min":5.766,"5min":5.766,"15min":5.766},"max":{"1min":46.584000000000003,"5min":627.40700000000004,"15min":627.40700000000004},"last":6.3840000000000003},{"interface":"front","average":{"1min":13.574999999999999,"5min":35.847999999999999,"15min":30.065000000000001},"min":{"1min":8.6419999999999995,"5min":7.8630000000000004,"15min":6.9260000000000002},"max":{"1min":54.68,"5min":635.62599999999998,"15min":635.62599999999998},"last":8.5039999999999996}]},{"osd":3,"last update":"Mon Jun 3 09:06:04 2024","interfaces":[{"interface":"back","average":{"1min":12.105,"5min":26.925000000000001,"15min":26.584},"min":{"1min":7.8410000000000002,"5min":7.1280000000000001,"15min":6.1909999999999998},"max":{"1min":18.425999999999998,"5min":389.72699999999998,"15min":389.72699999999998},"last":10.986000000000001},{"interface":"front","average":{"1min":13.794,"5min":29.771999999999998,"15min":28.815000000000001},"min":{"1min":10.084,"5min":9.4920000000000009,"15min":6.1849999999999996},"max":{"1min":17.373999999999999,"5min":390.23200000000003,"15min":494.92899999999997},"last":16.532}]},{"osd":4,"last update":"Mon Jun 3 09:05:27 2024","interfaces":[{"interface":"back","average":{"1min":13.478,"5min":1032.126,"15min":1389.729},"min":{"1min":6.5119999999999996,"5min":6.5119999999999996,"15min":6.5119999999999996},"max":{"1min":55.354999999999997,"5min":25555.212,"15min":60989.964},"last":12.74},{"interface":"front","average":{"1min":16.204999999999998,"5min":1030.8389999999999,"15min":1387.778},"min":{"1min":10.282999999999999,"5min":7.1429999999999998,"15min":7.1429999999999998},"max":{"1min":49.945999999999998,"5min":25556.159,"15min":61000.017},"last":13.673}]},{"osd":5,"last update":"Mon Jun 3 09:05:13 2024","interfaces":[{"interface":"back","average":{"1min":13.778,"5min":31.690999999999999,"15min":29.962},"min":{"1min":8.5030000000000001,"5min":7.7030000000000003,"15min":7.7030000000000003},"max":{"1min":60.216999999999999,"5min":455.78699999999998,"15min":605.05899999999997},"last":14.942},{"interface":"front","average":{"1min":17.823,"5min":30.498999999999999,"15min":27.434999999999999},"min":{"1min":11.547000000000001,"5min":6.2030000000000003,"15min":6.2030000000000003},"max":{"1min":59.677,"5min":391.31200000000001,"15min":541.11000000000001},"last":17.879000000000001}]},{"osd":6,"last update":"Mon Jun 3 09:04:30 2024","interfaces":[{"interface":"back","average":{"1min":52.496000000000002,"5min":33.139000000000003,"15min":160.83199999999999},"min":{"1min":7.5289999999999999,"5min":7.1040000000000001,"15min":6.3739999999999997},"max":{"1min":457.91699999999997,"5min":457.91699999999997,"15min":11115.258},"last":18.959},{"interface":"front","average":{"1min":49.670000000000002,"5min":33.704999999999998,"15min":162.791},"min":{"1min":12.247999999999999,"5min":11.019,"15min":9.9760000000000009},"max":{"1min":407.05099999999999,"5min":407.05099999999999,"15min":11092.194},"last":17.707000000000001}]},{"osd":7,"last update":"Mon Jun 3 09:05:22 2024","interfaces":[{"interface":"back","average":{"1min":14.503,"5min":29.489000000000001,"15min":33.134999999999998},"min":{"1min":8.5690000000000008,"5min":7.5149999999999997,"15min":6.9859999999999998},"max":{"1min":48.526000000000003,"5min":461.41699999999997,"15min":598.33699999999999},"last":11.83},{"interface":"front","average":{"1min":17.050000000000001,"5min":33.313000000000002,"15min":38.603999999999999},"min":{"1min":11.182,"5min":10.263999999999999,"15min":10.263999999999999},"max":{"1min":58.683,"5min":463.10599999999999,"15min":592.80499999999995},"last":14.407}]}]}],"pool_statfs":[{"poolid":1,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":401408,"data_stored":397840,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":5,"total":0,"available":0,"internally_reserved":0,"allocated":401408,"data_stored":397840,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":4,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":5,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":6,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":17,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":17,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":17,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":17,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":17,"osd":4,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":17,"osd":5,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":5,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":17,"osd":6,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":5,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":17,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0}]}} 2024-06-03T09:06:16.136 INFO:tasks.ceph.ceph_manager.ceph:PG 17.c is not active+clean 2024-06-03T09:06:16.136 INFO:tasks.ceph.ceph_manager.ceph:{'pgid': '17.c', 'version': "0'0", 'reported_seq': 952, 'reported_epoch': 1008, 'state': 'active+undersized', 'last_fresh': '2024-06-03T09:06:12.368416+0000', 'last_change': '2024-06-03T09:06:03.440074+0000', 'last_active': '2024-06-03T09:06:12.368416+0000', 'last_peered': '2024-06-03T09:06:12.368416+0000', 'last_clean': '2024-06-03T09:05:02.105638+0000', 'last_became_active': '2024-06-03T09:06:03.440074+0000', 'last_became_peered': '2024-06-03T09:06:03.440074+0000', 'last_unstale': '2024-06-03T09:06:12.368416+0000', 'last_undegraded': '2024-06-03T09:06:12.368416+0000', 'last_fullsized': '2024-06-03T09:06:03.377982+0000', 'mapping_epoch': 999, 'log_start': "0'0", 'ondisk_log_start': "0'0", 'created': 28, 'last_epoch_clean': 255, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2024-06-03T08:37:55.614497+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2024-06-03T08:37:55.614497+0000', 'last_clean_scrub_stamp': '2024-06-03T08:37:55.614497+0000', 'objects_scrubbed': 0, 'log_size': 0, 'log_dups_size': 0, 'ondisk_log_size': 0, 'stats_invalid': False, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': 'periodic scrub scheduled @ 2024-06-04T10:34:20.808557+0000', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 0, 'num_objects': 0, 'num_object_clones': 0, 'num_object_copies': 0, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 0, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 0, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 0, 'num_write_kb': 0, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 0, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [3], 'acting': [3], 'avail_no_missing': ['3'], 'object_location_counts': [], 'blocked_by': [], 'up_primary': 3, 'acting_primary': 3, 'purged_snaps': []} 2024-06-03T09:06:16.136 INFO:tasks.ceph.ceph_manager.ceph:PG 17.f is not active+clean 2024-06-03T09:06:16.136 INFO:tasks.ceph.ceph_manager.ceph:{'pgid': '17.f', 'version': "0'0", 'reported_seq': 144, 'reported_epoch': 1004, 'state': 'active+undersized', 'last_fresh': '2024-06-03T09:06:08.016575+0000', 'last_change': '2024-06-03T09:06:03.420285+0000', 'last_active': '2024-06-03T09:06:08.016575+0000', 'last_peered': '2024-06-03T09:06:08.016575+0000', 'last_clean': '2024-06-03T08:37:55.614497+0000', 'last_became_active': '2024-06-03T09:06:03.420285+0000', 'last_became_peered': '2024-06-03T09:06:03.420285+0000', 'last_unstale': '2024-06-03T09:06:08.016575+0000', 'last_undegraded': '2024-06-03T09:06:08.016575+0000', 'last_fullsized': '2024-06-03T09:06:03.380616+0000', 'mapping_epoch': 999, 'log_start': "0'0", 'ondisk_log_start': "0'0", 'created': 28, 'last_epoch_clean': 255, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2024-06-03T08:37:55.614497+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2024-06-03T08:37:55.614497+0000', 'last_clean_scrub_stamp': '2024-06-03T08:37:55.614497+0000', 'objects_scrubbed': 0, 'log_size': 0, 'log_dups_size': 0, 'ondisk_log_size': 0, 'stats_invalid': False, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': 'periodic scrub scheduled @ 2024-06-04T15:39:41.187731+0000', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 0, 'num_objects': 0, 'num_object_clones': 0, 'num_object_copies': 0, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 0, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 0, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 0, 'num_write_kb': 0, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 0, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [4], 'acting': [4], 'avail_no_missing': ['4'], 'object_location_counts': [], 'blocked_by': [], 'up_primary': 4, 'acting_primary': 4, 'purged_snaps': []} 2024-06-03T09:06:16.136 INFO:tasks.ceph.ceph_manager.ceph:PG 17.e is not active+clean 2024-06-03T09:06:16.136 INFO:tasks.ceph.ceph_manager.ceph:{'pgid': '17.e', 'version': "0'0", 'reported_seq': 148, 'reported_epoch': 1008, 'state': 'active+undersized', 'last_fresh': '2024-06-03T09:06:12.895353+0000', 'last_change': '2024-06-03T09:06:03.412887+0000', 'last_active': '2024-06-03T09:06:12.895353+0000', 'last_peered': '2024-06-03T09:06:12.895353+0000', 'last_clean': '2024-06-03T08:37:55.614497+0000', 'last_became_active': '2024-06-03T09:06:03.412887+0000', 'last_became_peered': '2024-06-03T09:06:03.412887+0000', 'last_unstale': '2024-06-03T09:06:12.895353+0000', 'last_undegraded': '2024-06-03T09:06:12.895353+0000', 'last_fullsized': '2024-06-03T09:06:03.332482+0000', 'mapping_epoch': 999, 'log_start': "0'0", 'ondisk_log_start': "0'0", 'created': 28, 'last_epoch_clean': 255, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2024-06-03T08:37:55.614497+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2024-06-03T08:37:55.614497+0000', 'last_clean_scrub_stamp': '2024-06-03T08:37:55.614497+0000', 'objects_scrubbed': 0, 'log_size': 0, 'log_dups_size': 0, 'ondisk_log_size': 0, 'stats_invalid': False, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': 'periodic scrub scheduled @ 2024-06-04T12:23:22.511601+0000', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 0, 'num_objects': 0, 'num_object_clones': 0, 'num_object_copies': 0, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 0, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 0, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 0, 'num_write_kb': 0, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 0, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [5], 'acting': [5], 'avail_no_missing': ['5'], 'object_location_counts': [], 'blocked_by': [], 'up_primary': 5, 'acting_primary': 5, 'purged_snaps': []} 2024-06-03T09:06:16.137 INFO:tasks.ceph.ceph_manager.ceph:PG 17.8 is not active+clean 2024-06-03T09:06:16.137 INFO:tasks.ceph.ceph_manager.ceph:{'pgid': '17.8', 'version': "0'0", 'reported_seq': 959, 'reported_epoch': 1007, 'state': 'active+undersized', 'last_fresh': '2024-06-03T09:06:12.708781+0000', 'last_change': '2024-06-03T09:06:03.388786+0000', 'last_active': '2024-06-03T09:06:12.708781+0000', 'last_peered': '2024-06-03T09:06:12.708781+0000', 'last_clean': '2024-06-03T09:04:43.297780+0000', 'last_became_active': '2024-06-03T09:06:03.388786+0000', 'last_became_peered': '2024-06-03T09:06:03.388786+0000', 'last_unstale': '2024-06-03T09:06:12.708781+0000', 'last_undegraded': '2024-06-03T09:06:12.708781+0000', 'last_fullsized': '2024-06-03T09:06:03.331754+0000', 'mapping_epoch': 999, 'log_start': "0'0", 'ondisk_log_start': "0'0", 'created': 28, 'last_epoch_clean': 255, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2024-06-03T08:37:55.614497+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2024-06-03T08:37:55.614497+0000', 'last_clean_scrub_stamp': '2024-06-03T08:37:55.614497+0000', 'objects_scrubbed': 0, 'log_size': 0, 'log_dups_size': 0, 'ondisk_log_size': 0, 'stats_invalid': False, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': 'periodic scrub scheduled @ 2024-06-04T09:18:44.765547+0000', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 0, 'num_objects': 0, 'num_object_clones': 0, 'num_object_copies': 0, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 0, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 0, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 0, 'num_write_kb': 0, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 0, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [0], 'acting': [0], 'avail_no_missing': ['0'], 'object_location_counts': [], 'blocked_by': [], 'up_primary': 0, 'acting_primary': 0, 'purged_snaps': []} 2024-06-03T09:06:16.137 INFO:tasks.ceph.ceph_manager.ceph:PG 17.b is not active+clean 2024-06-03T09:06:16.137 INFO:tasks.ceph.ceph_manager.ceph:{'pgid': '17.b', 'version': "0'0", 'reported_seq': 145, 'reported_epoch': 1005, 'state': 'active+undersized', 'last_fresh': '2024-06-03T09:06:09.149286+0000', 'last_change': '2024-06-03T09:06:03.451203+0000', 'last_active': '2024-06-03T09:06:09.149286+0000', 'last_peered': '2024-06-03T09:06:09.149286+0000', 'last_clean': '2024-06-03T08:37:55.614497+0000', 'last_became_active': '2024-06-03T09:06:03.451203+0000', 'last_became_peered': '2024-06-03T09:06:03.451203+0000', 'last_unstale': '2024-06-03T09:06:09.149286+0000', 'last_undegraded': '2024-06-03T09:06:09.149286+0000', 'last_fullsized': '2024-06-03T09:06:03.375523+0000', 'mapping_epoch': 999, 'log_start': "0'0", 'ondisk_log_start': "0'0", 'created': 28, 'last_epoch_clean': 255, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2024-06-03T08:37:55.614497+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2024-06-03T08:37:55.614497+0000', 'last_clean_scrub_stamp': '2024-06-03T08:37:55.614497+0000', 'objects_scrubbed': 0, 'log_size': 0, 'log_dups_size': 0, 'ondisk_log_size': 0, 'stats_invalid': False, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': 'periodic scrub scheduled @ 2024-06-04T10:16:19.813495+0000', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 0, 'num_objects': 0, 'num_object_clones': 0, 'num_object_copies': 0, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 0, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 0, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 0, 'num_write_kb': 0, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 0, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [2], 'acting': [2], 'avail_no_missing': ['2'], 'object_location_counts': [], 'blocked_by': [], 'up_primary': 2, 'acting_primary': 2, 'purged_snaps': []} 2024-06-03T09:06:16.137 INFO:tasks.ceph.ceph_manager.ceph:PG 17.4 is not active+clean 2024-06-03T09:06:16.137 INFO:tasks.ceph.ceph_manager.ceph:{'pgid': '17.4', 'version': "0'0", 'reported_seq': 148, 'reported_epoch': 1008, 'state': 'active+undersized', 'last_fresh': '2024-06-03T09:06:12.883827+0000', 'last_change': '2024-06-03T09:06:03.441889+0000', 'last_active': '2024-06-03T09:06:12.883827+0000', 'last_peered': '2024-06-03T09:06:12.883827+0000', 'last_clean': '2024-06-03T08:37:55.614497+0000', 'last_became_active': '2024-06-03T09:06:03.441889+0000', 'last_became_peered': '2024-06-03T09:06:03.441889+0000', 'last_unstale': '2024-06-03T09:06:12.883827+0000', 'last_undegraded': '2024-06-03T09:06:12.883827+0000', 'last_fullsized': '2024-06-03T09:06:03.359582+0000', 'mapping_epoch': 999, 'log_start': "0'0", 'ondisk_log_start': "0'0", 'created': 28, 'last_epoch_clean': 255, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2024-06-03T08:37:55.614497+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2024-06-03T08:37:55.614497+0000', 'last_clean_scrub_stamp': '2024-06-03T08:37:55.614497+0000', 'objects_scrubbed': 0, 'log_size': 0, 'log_dups_size': 0, 'ondisk_log_size': 0, 'stats_invalid': False, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': 'periodic scrub scheduled @ 2024-06-04T12:49:22.907141+0000', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 0, 'num_objects': 0, 'num_object_clones': 0, 'num_object_copies': 0, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 0, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 0, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 0, 'num_write_kb': 0, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 0, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [7], 'acting': [7], 'avail_no_missing': ['7'], 'object_location_counts': [], 'blocked_by': [], 'up_primary': 7, 'acting_primary': 7, 'purged_snaps': []} 2024-06-03T09:06:16.137 INFO:tasks.ceph.ceph_manager.ceph:PG 17.6 is not active+clean 2024-06-03T09:06:16.137 INFO:tasks.ceph.ceph_manager.ceph:{'pgid': '17.6', 'version': "0'0", 'reported_seq': 958, 'reported_epoch': 1007, 'state': 'active+undersized', 'last_fresh': '2024-06-03T09:06:12.710313+0000', 'last_change': '2024-06-03T09:06:03.428677+0000', 'last_active': '2024-06-03T09:06:12.710313+0000', 'last_peered': '2024-06-03T09:06:12.710313+0000', 'last_clean': '2024-06-03T09:04:43.312352+0000', 'last_became_active': '2024-06-03T09:06:03.428677+0000', 'last_became_peered': '2024-06-03T09:06:03.428677+0000', 'last_unstale': '2024-06-03T09:06:12.710313+0000', 'last_undegraded': '2024-06-03T09:06:12.710313+0000', 'last_fullsized': '2024-06-03T09:06:03.359909+0000', 'mapping_epoch': 999, 'log_start': "0'0", 'ondisk_log_start': "0'0", 'created': 28, 'last_epoch_clean': 255, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2024-06-03T08:37:55.614497+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2024-06-03T08:37:55.614497+0000', 'last_clean_scrub_stamp': '2024-06-03T08:37:55.614497+0000', 'objects_scrubbed': 0, 'log_size': 0, 'log_dups_size': 0, 'ondisk_log_size': 0, 'stats_invalid': False, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': 'periodic scrub scheduled @ 2024-06-04T15:38:02.180241+0000', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 0, 'num_objects': 0, 'num_object_clones': 0, 'num_object_copies': 0, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 0, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 0, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 0, 'num_write_kb': 0, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 0, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [0], 'acting': [0], 'avail_no_missing': ['0'], 'object_location_counts': [], 'blocked_by': [], 'up_primary': 0, 'acting_primary': 0, 'purged_snaps': []} 2024-06-03T09:06:16.137 INFO:tasks.ceph.ceph_manager.ceph:PG 17.16 is not active+clean 2024-06-03T09:06:16.137 INFO:tasks.ceph.ceph_manager.ceph:{'pgid': '17.16', 'version': "0'0", 'reported_seq': 968, 'reported_epoch': 1005, 'state': 'active+undersized', 'last_fresh': '2024-06-03T09:06:09.191972+0000', 'last_change': '2024-06-03T09:06:03.478615+0000', 'last_active': '2024-06-03T09:06:09.191972+0000', 'last_peered': '2024-06-03T09:06:09.191972+0000', 'last_clean': '2024-06-03T09:04:43.328415+0000', 'last_became_active': '2024-06-03T09:06:03.478615+0000', 'last_became_peered': '2024-06-03T09:06:03.478615+0000', 'last_unstale': '2024-06-03T09:06:09.191972+0000', 'last_undegraded': '2024-06-03T09:06:09.191972+0000', 'last_fullsized': '2024-06-03T09:06:03.402504+0000', 'mapping_epoch': 999, 'log_start': "0'0", 'ondisk_log_start': "0'0", 'created': 28, 'last_epoch_clean': 255, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2024-06-03T08:37:55.614497+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2024-06-03T08:37:55.614497+0000', 'last_clean_scrub_stamp': '2024-06-03T08:37:55.614497+0000', 'objects_scrubbed': 0, 'log_size': 0, 'log_dups_size': 0, 'ondisk_log_size': 0, 'stats_invalid': False, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': 'periodic scrub scheduled @ 2024-06-04T08:47:08.079233+0000', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 0, 'num_objects': 0, 'num_object_clones': 0, 'num_object_copies': 0, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 0, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 0, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 0, 'num_write_kb': 0, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 0, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [2], 'acting': [2], 'avail_no_missing': ['2'], 'object_location_counts': [], 'blocked_by': [], 'up_primary': 2, 'acting_primary': 2, 'purged_snaps': []} 2024-06-03T09:06:16.137 INFO:tasks.ceph.ceph_manager.ceph:PG 2.4 is not active+clean 2024-06-03T09:06:16.138 INFO:tasks.ceph.ceph_manager.ceph:{'pgid': '2.4', 'version': "0'0", 'reported_seq': 950, 'reported_epoch': 1008, 'state': 'active+undersized', 'last_fresh': '2024-06-03T09:06:12.887507+0000', 'last_change': '2024-06-03T09:06:03.387171+0000', 'last_active': '2024-06-03T09:06:12.887507+0000', 'last_peered': '2024-06-03T09:06:12.887507+0000', 'last_clean': '2024-06-03T09:04:43.202645+0000', 'last_became_active': '2024-06-03T09:06:03.387171+0000', 'last_became_peered': '2024-06-03T09:06:03.387171+0000', 'last_unstale': '2024-06-03T09:06:12.887507+0000', 'last_undegraded': '2024-06-03T09:06:12.887507+0000', 'last_fullsized': '2024-06-03T09:06:03.283787+0000', 'mapping_epoch': 999, 'log_start': "0'0", 'ondisk_log_start': "0'0", 'created': 23, 'last_epoch_clean': 255, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2024-06-03T08:36:28.444764+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2024-06-03T08:36:28.444764+0000', 'last_clean_scrub_stamp': '2024-06-03T08:36:28.444764+0000', 'objects_scrubbed': 0, 'log_size': 0, 'log_dups_size': 0, 'ondisk_log_size': 0, 'stats_invalid': False, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': 'periodic scrub scheduled @ 2024-06-04T20:17:43.485023+0000', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 0, 'num_objects': 0, 'num_object_clones': 0, 'num_object_copies': 0, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 0, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 0, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 0, 'num_write_kb': 0, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 0, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [5], 'acting': [5], 'avail_no_missing': ['5'], 'object_location_counts': [], 'blocked_by': [], 'up_primary': 5, 'acting_primary': 5, 'purged_snaps': []} 2024-06-03T09:06:16.138 INFO:tasks.ceph.ceph_manager.ceph:PG 2.2 is not active+clean 2024-06-03T09:06:16.138 INFO:tasks.ceph.ceph_manager.ceph:{'pgid': '2.2', 'version': "0'0", 'reported_seq': 962, 'reported_epoch': 1008, 'state': 'active+undersized', 'last_fresh': '2024-06-03T09:06:12.352260+0000', 'last_change': '2024-06-03T09:06:03.410020+0000', 'last_active': '2024-06-03T09:06:12.352260+0000', 'last_peered': '2024-06-03T09:06:12.352260+0000', 'last_clean': '2024-06-03T09:05:37.145889+0000', 'last_became_active': '2024-06-03T09:06:03.410020+0000', 'last_became_peered': '2024-06-03T09:06:03.410020+0000', 'last_unstale': '2024-06-03T09:06:12.352260+0000', 'last_undegraded': '2024-06-03T09:06:12.352260+0000', 'last_fullsized': '2024-06-03T09:06:03.346864+0000', 'mapping_epoch': 999, 'log_start': "0'0", 'ondisk_log_start': "0'0", 'created': 23, 'last_epoch_clean': 255, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2024-06-03T08:36:28.444764+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2024-06-03T08:36:28.444764+0000', 'last_clean_scrub_stamp': '2024-06-03T08:36:28.444764+0000', 'objects_scrubbed': 0, 'log_size': 0, 'log_dups_size': 0, 'ondisk_log_size': 0, 'stats_invalid': False, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': 'periodic scrub scheduled @ 2024-06-04T09:57:12.423360+0000', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 0, 'num_objects': 0, 'num_object_clones': 0, 'num_object_copies': 0, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 0, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 0, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 0, 'num_write_kb': 0, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 0, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [3], 'acting': [3], 'avail_no_missing': ['3'], 'object_location_counts': [], 'blocked_by': [], 'up_primary': 3, 'acting_primary': 3, 'purged_snaps': []} 2024-06-03T09:06:16.138 INFO:tasks.ceph.ceph_manager.ceph:PG 17.1a is not active+clean 2024-06-03T09:06:16.138 INFO:tasks.ceph.ceph_manager.ceph:{'pgid': '17.1a', 'version': "0'0", 'reported_seq': 956, 'reported_epoch': 1004, 'state': 'active+undersized', 'last_fresh': '2024-06-03T09:06:07.954840+0000', 'last_change': '2024-06-03T09:06:03.390734+0000', 'last_active': '2024-06-03T09:06:07.954840+0000', 'last_peered': '2024-06-03T09:06:07.954840+0000', 'last_clean': '2024-06-03T09:05:20.322313+0000', 'last_became_active': '2024-06-03T09:06:03.390734+0000', 'last_became_peered': '2024-06-03T09:06:03.390734+0000', 'last_unstale': '2024-06-03T09:06:07.954840+0000', 'last_undegraded': '2024-06-03T09:06:07.954840+0000', 'last_fullsized': '2024-06-03T09:06:03.343197+0000', 'mapping_epoch': 999, 'log_start': "0'0", 'ondisk_log_start': "0'0", 'created': 28, 'last_epoch_clean': 254, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2024-06-03T08:37:55.614497+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2024-06-03T08:37:55.614497+0000', 'last_clean_scrub_stamp': '2024-06-03T08:37:55.614497+0000', 'objects_scrubbed': 0, 'log_size': 0, 'log_dups_size': 0, 'ondisk_log_size': 0, 'stats_invalid': False, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': 'periodic scrub scheduled @ 2024-06-04T15:56:47.628875+0000', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 0, 'num_objects': 0, 'num_object_clones': 0, 'num_object_copies': 0, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 0, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 0, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 0, 'num_write_kb': 0, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 0, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [4], 'acting': [4], 'avail_no_missing': ['4'], 'object_location_counts': [], 'blocked_by': [], 'up_primary': 4, 'acting_primary': 4, 'purged_snaps': []} 2024-06-03T09:06:16.138 INFO:tasks.ceph.ceph_manager.ceph:PG 17.1e is not active+clean 2024-06-03T09:06:16.138 INFO:tasks.ceph.ceph_manager.ceph:{'pgid': '17.1e', 'version': "0'0", 'reported_seq': 144, 'reported_epoch': 1004, 'state': 'active+undersized', 'last_fresh': '2024-06-03T09:06:07.982178+0000', 'last_change': '2024-06-03T09:06:03.406181+0000', 'last_active': '2024-06-03T09:06:07.982178+0000', 'last_peered': '2024-06-03T09:06:07.982178+0000', 'last_clean': '2024-06-03T08:37:55.614497+0000', 'last_became_active': '2024-06-03T09:06:03.406181+0000', 'last_became_peered': '2024-06-03T09:06:03.406181+0000', 'last_unstale': '2024-06-03T09:06:07.982178+0000', 'last_undegraded': '2024-06-03T09:06:07.982178+0000', 'last_fullsized': '2024-06-03T09:06:03.363739+0000', 'mapping_epoch': 999, 'log_start': "0'0", 'ondisk_log_start': "0'0", 'created': 28, 'last_epoch_clean': 255, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2024-06-03T08:37:55.614497+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2024-06-03T08:37:55.614497+0000', 'last_clean_scrub_stamp': '2024-06-03T08:37:55.614497+0000', 'objects_scrubbed': 0, 'log_size': 0, 'log_dups_size': 0, 'ondisk_log_size': 0, 'stats_invalid': False, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': 'periodic scrub scheduled @ 2024-06-04T11:45:24.486600+0000', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 0, 'num_objects': 0, 'num_object_clones': 0, 'num_object_copies': 0, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 0, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 0, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 0, 'num_write_kb': 0, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 0, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [4], 'acting': [4], 'avail_no_missing': ['4'], 'object_location_counts': [], 'blocked_by': [], 'up_primary': 4, 'acting_primary': 4, 'purged_snaps': []} 2024-06-03T09:06:16.138 INFO:tasks.ceph.ceph_manager.ceph:PG 17.1c is not active+clean 2024-06-03T09:06:16.138 INFO:tasks.ceph.ceph_manager.ceph:{'pgid': '17.1c', 'version': "0'0", 'reported_seq': 942, 'reported_epoch': 1008, 'state': 'active+undersized', 'last_fresh': '2024-06-03T09:06:12.880704+0000', 'last_change': '2024-06-03T09:06:03.382210+0000', 'last_active': '2024-06-03T09:06:12.880704+0000', 'last_peered': '2024-06-03T09:06:12.880704+0000', 'last_clean': '2024-06-03T09:04:43.199660+0000', 'last_became_active': '2024-06-03T09:06:03.382210+0000', 'last_became_peered': '2024-06-03T09:06:03.382210+0000', 'last_unstale': '2024-06-03T09:06:12.880704+0000', 'last_undegraded': '2024-06-03T09:06:12.880704+0000', 'last_fullsized': '2024-06-03T09:06:03.277874+0000', 'mapping_epoch': 999, 'log_start': "0'0", 'ondisk_log_start': "0'0", 'created': 28, 'last_epoch_clean': 255, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2024-06-03T08:37:55.614497+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2024-06-03T08:37:55.614497+0000', 'last_clean_scrub_stamp': '2024-06-03T08:37:55.614497+0000', 'objects_scrubbed': 0, 'log_size': 0, 'log_dups_size': 0, 'ondisk_log_size': 0, 'stats_invalid': False, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': 'periodic scrub scheduled @ 2024-06-04T12:18:50.013884+0000', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 0, 'num_objects': 0, 'num_object_clones': 0, 'num_object_copies': 0, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 0, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 0, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 0, 'num_write_kb': 0, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 0, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [5], 'acting': [5], 'avail_no_missing': ['5'], 'object_location_counts': [], 'blocked_by': [], 'up_primary': 5, 'acting_primary': 5, 'purged_snaps': []} 2024-06-03T09:06:16.138 INFO:tasks.ceph.ceph_manager.ceph:PG 17.1 is not active+clean 2024-06-03T09:06:16.139 INFO:tasks.ceph.ceph_manager.ceph:{'pgid': '17.1', 'version': "65'7", 'reported_seq': 958, 'reported_epoch': 1008, 'state': 'active+undersized+degraded', 'last_fresh': '2024-06-03T09:06:12.872100+0000', 'last_change': '2024-06-03T09:06:03.370660+0000', 'last_active': '2024-06-03T09:06:12.872100+0000', 'last_peered': '2024-06-03T09:06:12.872100+0000', 'last_clean': '2024-06-03T09:05:03.733864+0000', 'last_became_active': '2024-06-03T09:06:03.370660+0000', 'last_became_peered': '2024-06-03T09:06:03.370660+0000', 'last_unstale': '2024-06-03T09:06:12.872100+0000', 'last_undegraded': '2024-06-03T09:06:03.266076+0000', 'last_fullsized': '2024-06-03T09:06:03.264525+0000', 'mapping_epoch': 999, 'log_start': "0'0", 'ondisk_log_start': "0'0", 'created': 28, 'last_epoch_clean': 255, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2024-06-03T08:37:55.614497+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2024-06-03T08:37:55.614497+0000', 'last_clean_scrub_stamp': '2024-06-03T08:37:55.614497+0000', 'objects_scrubbed': 0, 'log_size': 7, 'log_dups_size': 0, 'ondisk_log_size': 7, 'stats_invalid': False, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': 'periodic scrub scheduled @ 2024-06-04T11:55:18.194283+0000', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 5, 'num_objects': 1, 'num_object_clones': 0, 'num_object_copies': 2, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 1, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 1, 'num_whiteouts': 0, 'num_read': 6, 'num_read_kb': 6, 'num_write': 7, 'num_write_kb': 7, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 0, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [5], 'acting': [5], 'avail_no_missing': ['5'], 'object_location_counts': [{'shards': '5', 'objects': 1}], 'blocked_by': [], 'up_primary': 5, 'acting_primary': 5, 'purged_snaps': []} 2024-06-03T09:06:16.139 DEBUG:teuthology.orchestra.run.smithi088:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph status --format=json 2024-06-03T09:06:16.235 INFO:teuthology.orchestra.run.smithi088.stderr:2024-06-03T09:06:16.235+0000 7fa0d201c640 10 monclient: get_monmap_and_config 2024-06-03T09:06:16.235 INFO:teuthology.orchestra.run.smithi088.stderr:2024-06-03T09:06:16.235+0000 7fa0d201c640 10 monclient: build_initial_monmap 2024-06-03T09:06:16.235 INFO:teuthology.orchestra.run.smithi088.stderr:2024-06-03T09:06:16.235+0000 7fa0d201c640 10 monclient: monmap: 2024-06-03T09:06:16.236 INFO:teuthology.orchestra.run.smithi088.stderr:epoch 0 2024-06-03T09:06:16.236 INFO:teuthology.orchestra.run.smithi088.stderr:fsid ec3c3fdd-3ac6-4d7d-9412-683130ed8b04 2024-06-03T09:06:16.236 INFO:teuthology.orchestra.run.smithi088.stderr:last_changed 2024-06-03T09:06:16.236629+0000 2024-06-03T09:06:16.236 INFO:teuthology.orchestra.run.smithi088.stderr:created 2024-06-03T09:06:16.236629+0000 2024-06-03T09:06:16.236 INFO:teuthology.orchestra.run.smithi088.stderr:min_mon_release 0 (unknown) 2024-06-03T09:06:16.236 INFO:teuthology.orchestra.run.smithi088.stderr:election_strategy: 3 2024-06-03T09:06:16.236 INFO:teuthology.orchestra.run.smithi088.stderr:0: v1:172.21.15.88:6789/0 mon.noname-a 2024-06-03T09:06:16.236 INFO:teuthology.orchestra.run.smithi088.stderr:1: v1:172.21.15.192:6789/0 mon.noname-c 2024-06-03T09:06:16.236 INFO:teuthology.orchestra.run.smithi088.stderr:2: v1:172.21.15.88:6790/0 mon.noname-b 2024-06-03T09:06:16.236 INFO:teuthology.orchestra.run.smithi088.stderr: 2024-06-03T09:06:16.238 INFO:teuthology.orchestra.run.smithi088.stderr:2024-06-03T09:06:16.236+0000 7fa0d201c640 1 Processor -- start 2024-06-03T09:06:16.238 INFO:teuthology.orchestra.run.smithi088.stderr:2024-06-03T09:06:16.236+0000 7fa0d201c640 1 -- start start 2024-06-03T09:06:16.238 INFO:teuthology.orchestra.run.smithi088.stderr:2024-06-03T09:06:16.236+0000 7fa0d201c640 10 monclient: init 2024-06-03T09:06:16.238 INFO:teuthology.orchestra.run.smithi088.stderr:2024-06-03T09:06:16.237+0000 7fa0d201c640 10 monclient: _reopen_session rank -1 2024-06-03T09:06:16.238 INFO:teuthology.orchestra.run.smithi088.stderr:2024-06-03T09:06:16.237+0000 7fa0d201c640 10 monclient: _add_conns ranks=[0,1,2] 2024-06-03T09:06:16.238 INFO:teuthology.orchestra.run.smithi088.stderr:2024-06-03T09:06:16.237+0000 7fa0d201c640 10 monclient(hunting): picked mon.noname-a con 0x7fa0cc06ea80 addr v1:172.21.15.88:6789/0 2024-06-03T09:06:16.238 INFO:teuthology.orchestra.run.smithi088.stderr:2024-06-03T09:06:16.237+0000 7fa0d201c640 10 monclient(hunting): picked mon.noname-c con 0x7fa0cc06dde0 addr v1:172.21.15.192:6789/0 2024-06-03T09:06:16.238 INFO:teuthology.orchestra.run.smithi088.stderr:2024-06-03T09:06:16.237+0000 7fa0d201c640 10 monclient(hunting): picked mon.noname-b con 0x7fa0cc13d960 addr v1:172.21.15.88:6790/0 2024-06-03T09:06:16.238 INFO:teuthology.orchestra.run.smithi088.stderr:2024-06-03T09:06:16.237+0000 7fa0d201c640 1 -- --> v1:172.21.15.88:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fa0cc13e8e0 con 0x7fa0cc06ea80 2024-06-03T09:06:16.238 INFO:teuthology.orchestra.run.smithi088.stderr:2024-06-03T09:06:16.237+0000 7fa0d201c640 1 -- --> v1:172.21.15.192:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fa0cc13b560 con 0x7fa0cc06dde0 2024-06-03T09:06:16.238 INFO:teuthology.orchestra.run.smithi088.stderr:2024-06-03T09:06:16.237+0000 7fa0d201c640 1 -- --> v1:172.21.15.88:6790/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fa0cc070230 con 0x7fa0cc13d960 2024-06-03T09:06:16.238 INFO:teuthology.orchestra.run.smithi088.stderr:2024-06-03T09:06:16.237+0000 7fa0d201c640 10 monclient(hunting): _renew_subs 2024-06-03T09:06:16.239 INFO:teuthology.orchestra.run.smithi088.stderr:2024-06-03T09:06:16.237+0000 7fa0d201c640 10 monclient(hunting): authenticate will time out at 3032.047109s 2024-06-03T09:06:16.239 INFO:teuthology.orchestra.run.smithi088.stderr:2024-06-03T09:06:16.237+0000 7fa0cb7fe640 1 --1- >> v1:172.21.15.88:6789/0 conn(0x7fa0cc06ea80 0x7fa0cc13ae50 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.88:6789/0 says I am v1:172.21.15.88:53936/0 (socket says 172.21.15.88:53936) 2024-06-03T09:06:16.239 INFO:teuthology.orchestra.run.smithi088.stderr:2024-06-03T09:06:16.237+0000 7fa0cb7fe640 1 -- 172.21.15.88:0/3703080539 learned_addr learned my addr 172.21.15.88:0/3703080539 (peer_addr_for_me v1:172.21.15.88:0/0) 2024-06-03T09:06:16.239 INFO:teuthology.orchestra.run.smithi088.stderr:2024-06-03T09:06:16.239+0000 7fa0ca7fc640 1 -- 172.21.15.88:0/3703080539 <== mon.1 v1:172.21.15.192:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 4221395827 0 0) 0x7fa0cc13b560 con 0x7fa0cc06dde0 2024-06-03T09:06:16.240 INFO:teuthology.orchestra.run.smithi088.stderr:2024-06-03T09:06:16.239+0000 7fa0ca7fc640 10 monclient(hunting): _negotiate 2024-06-03T09:06:16.240 INFO:teuthology.orchestra.run.smithi088.stderr:2024-06-03T09:06:16.239+0000 7fa0ca7fc640 10 monclient(hunting): _init_auth method 2 2024-06-03T09:06:16.240 INFO:teuthology.orchestra.run.smithi088.stderr:2024-06-03T09:06:16.239+0000 7fa0ca7fc640 10 monclient(hunting): _init_auth creating new auth 2024-06-03T09:06:16.240 INFO:teuthology.orchestra.run.smithi088.stderr:2024-06-03T09:06:16.239+0000 7fa0ca7fc640 10 monclient(hunting): my global_id is 30145 2024-06-03T09:06:16.240 INFO:teuthology.orchestra.run.smithi088.stderr:2024-06-03T09:06:16.239+0000 7fa0ca7fc640 1 -- 172.21.15.88:0/3703080539 --> v1:172.21.15.192:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7fa0b4005730 con 0x7fa0cc06dde0 2024-06-03T09:06:16.240 INFO:teuthology.orchestra.run.smithi088.stderr:2024-06-03T09:06:16.239+0000 7fa0ca7fc640 1 -- 172.21.15.88:0/3703080539 <== mon.2 v1:172.21.15.88:6790/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 221393757 0 0) 0x7fa0cc070230 con 0x7fa0cc13d960 2024-06-03T09:06:16.240 INFO:teuthology.orchestra.run.smithi088.stderr:2024-06-03T09:06:16.240+0000 7fa0ca7fc640 10 monclient(hunting): _negotiate 2024-06-03T09:06:16.240 INFO:teuthology.orchestra.run.smithi088.stderr:2024-06-03T09:06:16.240+0000 7fa0ca7fc640 10 monclient(hunting): _init_auth method 2 2024-06-03T09:06:16.240 INFO:teuthology.orchestra.run.smithi088.stderr:2024-06-03T09:06:16.240+0000 7fa0ca7fc640 10 monclient(hunting): _init_auth creating new auth 2024-06-03T09:06:16.240 INFO:teuthology.orchestra.run.smithi088.stderr:2024-06-03T09:06:16.240+0000 7fa0ca7fc640 10 monclient(hunting): my global_id is 30068 2024-06-03T09:06:16.240 INFO:teuthology.orchestra.run.smithi088.stderr:2024-06-03T09:06:16.240+0000 7fa0ca7fc640 1 -- 172.21.15.88:0/3703080539 --> v1:172.21.15.88:6790/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7fa0cc13b560 con 0x7fa0cc13d960 2024-06-03T09:06:16.240 INFO:teuthology.orchestra.run.smithi088.stderr:2024-06-03T09:06:16.240+0000 7fa0ca7fc640 1 -- 172.21.15.88:0/3703080539 <== mon.0 v1:172.21.15.88:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 676682507 0 0) 0x7fa0cc13e8e0 con 0x7fa0cc06ea80 2024-06-03T09:06:16.240 INFO:teuthology.orchestra.run.smithi088.stderr:2024-06-03T09:06:16.240+0000 7fa0ca7fc640 10 monclient(hunting): _negotiate 2024-06-03T09:06:16.241 INFO:teuthology.orchestra.run.smithi088.stderr:2024-06-03T09:06:16.240+0000 7fa0ca7fc640 10 monclient(hunting): _init_auth method 2 2024-06-03T09:06:16.241 INFO:teuthology.orchestra.run.smithi088.stderr:2024-06-03T09:06:16.240+0000 7fa0ca7fc640 10 monclient(hunting): _init_auth creating new auth 2024-06-03T09:06:16.241 INFO:teuthology.orchestra.run.smithi088.stderr:2024-06-03T09:06:16.240+0000 7fa0ca7fc640 10 monclient(hunting): my global_id is 30567 2024-06-03T09:06:16.241 INFO:teuthology.orchestra.run.smithi088.stderr:2024-06-03T09:06:16.240+0000 7fa0ca7fc640 1 -- 172.21.15.88:0/3703080539 --> v1:172.21.15.88:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7fa0cc070230 con 0x7fa0cc06ea80 2024-06-03T09:06:16.243 INFO:teuthology.orchestra.run.smithi088.stderr:2024-06-03T09:06:16.243+0000 7fa0ca7fc640 1 -- 172.21.15.88:0/3703080539 <== mon.0 v1:172.21.15.88:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 214+0+0 (unknown 1421093442 0 0) 0x7fa0cc070230 con 0x7fa0cc06ea80 2024-06-03T09:06:16.243 INFO:teuthology.orchestra.run.smithi088.stderr:2024-06-03T09:06:16.243+0000 7fa0ca7fc640 1 -- 172.21.15.88:0/3703080539 >> v1:172.21.15.88:6790/0 conn(0x7fa0cc13d960 legacy=0x7fa0cc149260 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-06-03T09:06:16.243 INFO:teuthology.orchestra.run.smithi088.stderr:2024-06-03T09:06:16.243+0000 7fa0ca7fc640 1 -- 172.21.15.88:0/3703080539 >> v1:172.21.15.192:6789/0 conn(0x7fa0cc06dde0 legacy=0x7fa0cc070ec0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-06-03T09:06:16.243 INFO:teuthology.orchestra.run.smithi088.stderr:2024-06-03T09:06:16.243+0000 7fa0ca7fc640 10 monclient: _finish_hunting 0 2024-06-03T09:06:16.243 INFO:teuthology.orchestra.run.smithi088.stderr:2024-06-03T09:06:16.243+0000 7fa0ca7fc640 1 monclient: found mon.noname-a 2024-06-03T09:06:16.243 INFO:teuthology.orchestra.run.smithi088.stderr:2024-06-03T09:06:16.243+0000 7fa0ca7fc640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-06-03T09:06:16.244 INFO:teuthology.orchestra.run.smithi088.stderr:2024-06-03T09:06:16.243+0000 7fa0ca7fc640 10 monclient: _send_mon_message to mon.noname-a at v1:172.21.15.88:6789/0 2024-06-03T09:06:16.244 INFO:teuthology.orchestra.run.smithi088.stderr:2024-06-03T09:06:16.243+0000 7fa0ca7fc640 1 -- 172.21.15.88:0/3703080539 --> v1:172.21.15.88:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fa0cc070930 con 0x7fa0cc06ea80 2024-06-03T09:06:16.244 INFO:teuthology.orchestra.run.smithi088.stderr:2024-06-03T09:06:16.243+0000 7fa0ca7fc640 10 monclient: _finish_auth 0 2024-06-03T09:06:16.244 INFO:teuthology.orchestra.run.smithi088.stderr:2024-06-03T09:06:16.243+0000 7fa0ca7fc640 10 monclient: _check_auth_tickets 2024-06-03T09:06:16.244 INFO:teuthology.orchestra.run.smithi088.stderr:2024-06-03T09:06:16.243+0000 7fa0ca7fc640 20 monclient: _check_auth_rotating not needed by client.admin 2024-06-03T09:06:16.244 INFO:teuthology.orchestra.run.smithi088.stderr:2024-06-03T09:06:16.243+0000 7fa0d201c640 5 monclient: authenticate success, global_id 30567 2024-06-03T09:06:16.244 INFO:teuthology.orchestra.run.smithi088.stderr:2024-06-03T09:06:16.243+0000 7fa0d201c640 20 monclient: get_monmap_and_config waiting for monmap|config 2024-06-03T09:06:16.244 INFO:teuthology.orchestra.run.smithi088.stderr:2024-06-03T09:06:16.243+0000 7fa0ca7fc640 1 -- 172.21.15.88:0/3703080539 <== mon.0 v1:172.21.15.88:6789/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 2234374065 0 0) 0x7fa0bc0015b0 con 0x7fa0cc06ea80 2024-06-03T09:06:16.244 INFO:teuthology.orchestra.run.smithi088.stderr:2024-06-03T09:06:16.243+0000 7fa0ca7fc640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-06-03T09:06:16.244 INFO:teuthology.orchestra.run.smithi088.stderr:2024-06-03T09:06:16.243+0000 7fa0ca7fc640 10 monclient: got monmap 1 from mon.noname-a (according to old e1) 2024-06-03T09:06:16.244 INFO:teuthology.orchestra.run.smithi088.stderr:2024-06-03T09:06:16.243+0000 7fa0ca7fc640 10 monclient: dump: 2024-06-03T09:06:16.244 INFO:teuthology.orchestra.run.smithi088.stderr:epoch 1 2024-06-03T09:06:16.244 INFO:teuthology.orchestra.run.smithi088.stderr:fsid ec3c3fdd-3ac6-4d7d-9412-683130ed8b04 2024-06-03T09:06:16.244 INFO:teuthology.orchestra.run.smithi088.stderr:last_changed 2024-06-03T08:34:31.885125+0000 2024-06-03T09:06:16.245 INFO:teuthology.orchestra.run.smithi088.stderr:created 2024-06-03T08:34:31.885125+0000 2024-06-03T09:06:16.245 INFO:teuthology.orchestra.run.smithi088.stderr:min_mon_release 18 (reef) 2024-06-03T09:06:16.245 INFO:teuthology.orchestra.run.smithi088.stderr:election_strategy: 3 2024-06-03T09:06:16.245 INFO:teuthology.orchestra.run.smithi088.stderr:0: v1:172.21.15.88:6789/0 mon.a 2024-06-03T09:06:16.245 INFO:teuthology.orchestra.run.smithi088.stderr:1: v1:172.21.15.192:6789/0 mon.b 2024-06-03T09:06:16.245 INFO:teuthology.orchestra.run.smithi088.stderr:2: v1:172.21.15.88:6790/0 mon.c 2024-06-03T09:06:16.245 INFO:teuthology.orchestra.run.smithi088.stderr: 2024-06-03T09:06:16.245 INFO:teuthology.orchestra.run.smithi088.stderr:2024-06-03T09:06:16.243+0000 7fa0d201c640 20 monclient: get_monmap_and_config waiting for monmap|config 2024-06-03T09:06:16.246 INFO:teuthology.orchestra.run.smithi088.stderr:2024-06-03T09:06:16.245+0000 7fa0ca7fc640 1 -- 172.21.15.88:0/3703080539 <== mon.0 v1:172.21.15.88:6789/0 4 ==== config(0 keys) v1 ==== 4+0+0 (unknown 0 0 0) 0x7fa0bc001800 con 0x7fa0cc06ea80 2024-06-03T09:06:16.247 INFO:teuthology.orchestra.run.smithi088.stderr:2024-06-03T09:06:16.245+0000 7fa0ca7fc640 10 monclient: handle_config config(0 keys) v1 2024-06-03T09:06:16.247 INFO:teuthology.orchestra.run.smithi088.stderr:2024-06-03T09:06:16.245+0000 7fa0d201c640 10 monclient: get_monmap_and_config success 2024-06-03T09:06:16.247 INFO:teuthology.orchestra.run.smithi088.stderr:2024-06-03T09:06:16.245+0000 7fa0d201c640 4 set_mon_vals no callback set 2024-06-03T09:06:16.247 INFO:teuthology.orchestra.run.smithi088.stderr:2024-06-03T09:06:16.245+0000 7fa0ca7fc640 1 -- 172.21.15.88:0/3703080539 <== mon.0 v1:172.21.15.88:6789/0 5 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 2234374065 0 0) 0x7fa0bc001b90 con 0x7fa0cc06ea80 2024-06-03T09:06:16.247 INFO:teuthology.orchestra.run.smithi088.stderr:2024-06-03T09:06:16.245+0000 7fa0ca7fc640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-06-03T09:06:16.247 INFO:teuthology.orchestra.run.smithi088.stderr:2024-06-03T09:06:16.245+0000 7fa0ca7fc640 10 monclient: got monmap 1 from mon.a (according to old e1) 2024-06-03T09:06:16.247 INFO:teuthology.orchestra.run.smithi088.stderr:2024-06-03T09:06:16.245+0000 7fa0ca7fc640 10 monclient: dump: 2024-06-03T09:06:16.247 INFO:teuthology.orchestra.run.smithi088.stderr:epoch 1 2024-06-03T09:06:16.247 INFO:teuthology.orchestra.run.smithi088.stderr:fsid ec3c3fdd-3ac6-4d7d-9412-683130ed8b04 2024-06-03T09:06:16.247 INFO:teuthology.orchestra.run.smithi088.stderr:last_changed 2024-06-03T08:34:31.885125+0000 2024-06-03T09:06:16.247 INFO:teuthology.orchestra.run.smithi088.stderr:created 2024-06-03T08:34:31.885125+0000 2024-06-03T09:06:16.247 INFO:teuthology.orchestra.run.smithi088.stderr:min_mon_release 18 (reef) 2024-06-03T09:06:16.247 INFO:teuthology.orchestra.run.smithi088.stderr:election_strategy: 3 2024-06-03T09:06:16.248 INFO:teuthology.orchestra.run.smithi088.stderr:0: v1:172.21.15.88:6789/0 mon.a 2024-06-03T09:06:16.248 INFO:teuthology.orchestra.run.smithi088.stderr:1: v1:172.21.15.192:6789/0 mon.b 2024-06-03T09:06:16.248 INFO:teuthology.orchestra.run.smithi088.stderr:2: v1:172.21.15.88:6790/0 mon.c 2024-06-03T09:06:16.248 INFO:teuthology.orchestra.run.smithi088.stderr: 2024-06-03T09:06:16.248 INFO:teuthology.orchestra.run.smithi088.stderr:2024-06-03T09:06:16.246+0000 7fa0d201c640 10 monclient: shutdown 2024-06-03T09:06:16.248 INFO:teuthology.orchestra.run.smithi088.stderr:2024-06-03T09:06:16.246+0000 7fa0d201c640 20 monclient: shutdown discarding 0 pending message(s) 2024-06-03T09:06:16.248 INFO:teuthology.orchestra.run.smithi088.stderr:2024-06-03T09:06:16.246+0000 7fa0d201c640 1 -- 172.21.15.88:0/3703080539 >> v1:172.21.15.88:6789/0 conn(0x7fa0cc06ea80 legacy=0x7fa0cc13ae50 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-06-03T09:06:16.248 INFO:teuthology.orchestra.run.smithi088.stderr:2024-06-03T09:06:16.246+0000 7fa0d201c640 1 -- 172.21.15.88:0/3703080539 shutdown_connections 2024-06-03T09:06:16.248 INFO:teuthology.orchestra.run.smithi088.stderr:2024-06-03T09:06:16.246+0000 7fa0d201c640 1 -- 172.21.15.88:0/3703080539 >> 172.21.15.88:0/3703080539 conn(0x7fa0cc0b2e40 msgr2=0x7fa0cc0b3250 unknown :-1 s=STATE_NONE l=0).mark_down 2024-06-03T09:06:16.248 INFO:teuthology.orchestra.run.smithi088.stderr:2024-06-03T09:06:16.246+0000 7fa0d201c640 1 -- 172.21.15.88:0/3703080539 shutdown_connections 2024-06-03T09:06:16.248 INFO:teuthology.orchestra.run.smithi088.stderr:2024-06-03T09:06:16.246+0000 7fa0d201c640 1 -- 172.21.15.88:0/3703080539 wait complete. 2024-06-03T09:06:16.248 INFO:teuthology.orchestra.run.smithi088.stderr:2024-06-03T09:06:16.247+0000 7fa0d201c640 10 monclient: build_initial_monmap 2024-06-03T09:06:16.248 INFO:teuthology.orchestra.run.smithi088.stderr:2024-06-03T09:06:16.247+0000 7fa0d201c640 10 monclient: monmap: 2024-06-03T09:06:16.248 INFO:teuthology.orchestra.run.smithi088.stderr:epoch 0 2024-06-03T09:06:16.249 INFO:teuthology.orchestra.run.smithi088.stderr:fsid 00000000-0000-0000-0000-000000000000 2024-06-03T09:06:16.249 INFO:teuthology.orchestra.run.smithi088.stderr:last_changed 0.000000 2024-06-03T09:06:16.249 INFO:teuthology.orchestra.run.smithi088.stderr:created 0.000000 2024-06-03T09:06:16.249 INFO:teuthology.orchestra.run.smithi088.stderr:min_mon_release 0 (unknown) 2024-06-03T09:06:16.249 INFO:teuthology.orchestra.run.smithi088.stderr:election_strategy: 1 2024-06-03T09:06:16.249 INFO:teuthology.orchestra.run.smithi088.stderr:0: v1:172.21.15.88:6789/0 mon.noname-a 2024-06-03T09:06:16.249 INFO:teuthology.orchestra.run.smithi088.stderr:1: v1:172.21.15.192:6789/0 mon.noname-b 2024-06-03T09:06:16.249 INFO:teuthology.orchestra.run.smithi088.stderr:2: v1:172.21.15.88:6790/0 mon.noname-c 2024-06-03T09:06:16.249 INFO:teuthology.orchestra.run.smithi088.stderr: 2024-06-03T09:06:16.250 INFO:teuthology.orchestra.run.smithi088.stderr:2024-06-03T09:06:16.247+0000 7fa0d201c640 1 librados: starting msgr at 2024-06-03T09:06:16.250 INFO:teuthology.orchestra.run.smithi088.stderr:2024-06-03T09:06:16.247+0000 7fa0d201c640 1 librados: starting objecter 2024-06-03T09:06:16.251 INFO:teuthology.orchestra.run.smithi088.stderr:2024-06-03T09:06:16.247+0000 7fa0d201c640 1 Processor -- start 2024-06-03T09:06:16.251 INFO:teuthology.orchestra.run.smithi088.stderr:2024-06-03T09:06:16.247+0000 7fa0d201c640 1 -- start start 2024-06-03T09:06:16.251 INFO:teuthology.orchestra.run.smithi088.stderr:2024-06-03T09:06:16.247+0000 7fa0d201c640 1 librados: setting wanted keys 2024-06-03T09:06:16.251 INFO:teuthology.orchestra.run.smithi088.stderr:2024-06-03T09:06:16.247+0000 7fa0d201c640 1 librados: calling monclient init 2024-06-03T09:06:16.251 INFO:teuthology.orchestra.run.smithi088.stderr:2024-06-03T09:06:16.247+0000 7fa0d201c640 10 monclient: init 2024-06-03T09:06:16.251 INFO:teuthology.orchestra.run.smithi088.stderr:2024-06-03T09:06:16.248+0000 7fa0d201c640 10 monclient: _reopen_session rank -1 2024-06-03T09:06:16.251 INFO:teuthology.orchestra.run.smithi088.stderr:2024-06-03T09:06:16.248+0000 7fa0d201c640 10 monclient: _add_conns ranks=[0,2,1] 2024-06-03T09:06:16.251 INFO:teuthology.orchestra.run.smithi088.stderr:2024-06-03T09:06:16.248+0000 7fa0d201c640 10 monclient(hunting): picked mon.noname-a con 0x7fa0cc06dde0 addr v1:172.21.15.88:6789/0 2024-06-03T09:06:16.251 INFO:teuthology.orchestra.run.smithi088.stderr:2024-06-03T09:06:16.248+0000 7fa0d201c640 10 monclient(hunting): picked mon.noname-c con 0x7fa0cc06ea80 addr v1:172.21.15.88:6790/0 2024-06-03T09:06:16.251 INFO:teuthology.orchestra.run.smithi088.stderr:2024-06-03T09:06:16.248+0000 7fa0d201c640 10 monclient(hunting): picked mon.noname-b con 0x7fa0cc13d960 addr v1:172.21.15.192:6789/0 2024-06-03T09:06:16.251 INFO:teuthology.orchestra.run.smithi088.stderr:2024-06-03T09:06:16.248+0000 7fa0d201c640 1 -- --> v1:172.21.15.88:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fa0cc141930 con 0x7fa0cc06dde0 2024-06-03T09:06:16.251 INFO:teuthology.orchestra.run.smithi088.stderr:2024-06-03T09:06:16.248+0000 7fa0d201c640 1 -- --> v1:172.21.15.192:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fa0cc0aa3c0 con 0x7fa0cc13d960 2024-06-03T09:06:16.251 INFO:teuthology.orchestra.run.smithi088.stderr:2024-06-03T09:06:16.248+0000 7fa0d201c640 1 -- --> v1:172.21.15.88:6790/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fa0cc0aa070 con 0x7fa0cc06ea80 2024-06-03T09:06:16.252 INFO:teuthology.orchestra.run.smithi088.stderr:2024-06-03T09:06:16.248+0000 7fa0d201c640 10 monclient(hunting): _renew_subs 2024-06-03T09:06:16.252 INFO:teuthology.orchestra.run.smithi088.stderr:2024-06-03T09:06:16.248+0000 7fa0d201c640 10 monclient(hunting): authenticate will time out at 3032.058382s 2024-06-03T09:06:16.252 INFO:teuthology.orchestra.run.smithi088.stderr:2024-06-03T09:06:16.248+0000 7fa0cb7fe640 1 --1- >> v1:172.21.15.88:6789/0 conn(0x7fa0cc06dde0 0x7fa0cc0aa8e0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=1).handle_server_banner_and_identify peer v1:172.21.15.88:6789/0 says I am v1:172.21.15.88:53946/0 (socket says 172.21.15.88:53946) 2024-06-03T09:06:16.252 INFO:teuthology.orchestra.run.smithi088.stderr:2024-06-03T09:06:16.248+0000 7fa0cb7fe640 1 -- 172.21.15.88:0/3785650794 learned_addr learned my addr 172.21.15.88:0/3785650794 (peer_addr_for_me v1:172.21.15.88:0/0) 2024-06-03T09:06:16.252 INFO:teuthology.orchestra.run.smithi088.stderr:2024-06-03T09:06:16.249+0000 7fa0b3fff640 10 client.?.objecter ms_handle_connect 0x7fa0cc06dde0 2024-06-03T09:06:16.252 INFO:teuthology.orchestra.run.smithi088.stderr:2024-06-03T09:06:16.249+0000 7fa0b3fff640 10 client.?.objecter resend_mon_ops 2024-06-03T09:06:16.252 INFO:teuthology.orchestra.run.smithi088.stderr:2024-06-03T09:06:16.249+0000 7fa0b3fff640 10 client.?.objecter ms_handle_connect 0x7fa0cc13d960 2024-06-03T09:06:16.252 INFO:teuthology.orchestra.run.smithi088.stderr:2024-06-03T09:06:16.249+0000 7fa0b3fff640 10 client.?.objecter resend_mon_ops 2024-06-03T09:06:16.252 INFO:teuthology.orchestra.run.smithi088.stderr:2024-06-03T09:06:16.249+0000 7fa0b3fff640 10 client.?.objecter ms_handle_connect 0x7fa0cc06ea80 2024-06-03T09:06:16.252 INFO:teuthology.orchestra.run.smithi088.stderr:2024-06-03T09:06:16.249+0000 7fa0b3fff640 10 client.?.objecter resend_mon_ops 2024-06-03T09:06:16.253 INFO:teuthology.orchestra.run.smithi088.stderr:2024-06-03T09:06:16.252+0000 7fa0b3fff640 1 -- 172.21.15.88:0/3785650794 <== mon.0 v1:172.21.15.88:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 2915444527 0 0) 0x7fa0cc141930 con 0x7fa0cc06dde0 2024-06-03T09:06:16.253 INFO:teuthology.orchestra.run.smithi088.stderr:2024-06-03T09:06:16.252+0000 7fa0b3fff640 10 monclient(hunting): _negotiate 2024-06-03T09:06:16.253 INFO:teuthology.orchestra.run.smithi088.stderr:2024-06-03T09:06:16.252+0000 7fa0b3fff640 10 monclient(hunting): _init_auth method 2 2024-06-03T09:06:16.253 INFO:teuthology.orchestra.run.smithi088.stderr:2024-06-03T09:06:16.252+0000 7fa0b3fff640 10 monclient(hunting): _init_auth creating new auth 2024-06-03T09:06:16.253 INFO:teuthology.orchestra.run.smithi088.stderr:2024-06-03T09:06:16.252+0000 7fa0b3fff640 10 monclient(hunting): my global_id is 30570 2024-06-03T09:06:16.253 INFO:teuthology.orchestra.run.smithi088.stderr:2024-06-03T09:06:16.252+0000 7fa0b3fff640 1 -- 172.21.15.88:0/3785650794 --> v1:172.21.15.88:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7fa0b8004eb0 con 0x7fa0cc06dde0 2024-06-03T09:06:16.253 INFO:teuthology.orchestra.run.smithi088.stderr:2024-06-03T09:06:16.252+0000 7fa0b3fff640 1 -- 172.21.15.88:0/3785650794 <== mon.2 v1:172.21.15.88:6790/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 4043335587 0 0) 0x7fa0cc0aa070 con 0x7fa0cc06ea80 2024-06-03T09:06:16.254 INFO:teuthology.orchestra.run.smithi088.stderr:2024-06-03T09:06:16.252+0000 7fa0b3fff640 10 monclient(hunting): _negotiate 2024-06-03T09:06:16.254 INFO:teuthology.orchestra.run.smithi088.stderr:2024-06-03T09:06:16.252+0000 7fa0b3fff640 10 monclient(hunting): _init_auth method 2 2024-06-03T09:06:16.254 INFO:teuthology.orchestra.run.smithi088.stderr:2024-06-03T09:06:16.252+0000 7fa0b3fff640 10 monclient(hunting): _init_auth creating new auth 2024-06-03T09:06:16.254 INFO:teuthology.orchestra.run.smithi088.stderr:2024-06-03T09:06:16.252+0000 7fa0b3fff640 10 monclient(hunting): my global_id is 30071 2024-06-03T09:06:16.254 INFO:teuthology.orchestra.run.smithi088.stderr:2024-06-03T09:06:16.252+0000 7fa0b3fff640 1 -- 172.21.15.88:0/3785650794 --> v1:172.21.15.88:6790/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7fa0cc141930 con 0x7fa0cc06ea80 2024-06-03T09:06:16.254 INFO:teuthology.orchestra.run.smithi088.stderr:2024-06-03T09:06:16.252+0000 7fa0b3fff640 1 -- 172.21.15.88:0/3785650794 <== mon.1 v1:172.21.15.192:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 438196843 0 0) 0x7fa0cc0aa3c0 con 0x7fa0cc13d960 2024-06-03T09:06:16.254 INFO:teuthology.orchestra.run.smithi088.stderr:2024-06-03T09:06:16.252+0000 7fa0b3fff640 10 monclient(hunting): _negotiate 2024-06-03T09:06:16.254 INFO:teuthology.orchestra.run.smithi088.stderr:2024-06-03T09:06:16.252+0000 7fa0b3fff640 10 monclient(hunting): _init_auth method 2 2024-06-03T09:06:16.254 INFO:teuthology.orchestra.run.smithi088.stderr:2024-06-03T09:06:16.252+0000 7fa0b3fff640 10 monclient(hunting): _init_auth creating new auth 2024-06-03T09:06:16.254 INFO:teuthology.orchestra.run.smithi088.stderr:2024-06-03T09:06:16.252+0000 7fa0b3fff640 10 monclient(hunting): my global_id is 30148 2024-06-03T09:06:16.254 INFO:teuthology.orchestra.run.smithi088.stderr:2024-06-03T09:06:16.252+0000 7fa0b3fff640 1 -- 172.21.15.88:0/3785650794 --> v1:172.21.15.192:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7fa0cc0aa070 con 0x7fa0cc13d960 2024-06-03T09:06:16.257 INFO:teuthology.orchestra.run.smithi088.stderr:2024-06-03T09:06:16.256+0000 7fa0b3fff640 1 -- 172.21.15.88:0/3785650794 <== mon.0 v1:172.21.15.88:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 764+0+0 (unknown 1868483812 0 0) 0x7fa0b8004eb0 con 0x7fa0cc06dde0 2024-06-03T09:06:16.257 INFO:teuthology.orchestra.run.smithi088.stderr:2024-06-03T09:06:16.256+0000 7fa0b3fff640 1 -- 172.21.15.88:0/3785650794 --> v1:172.21.15.88:6789/0 -- auth(proto 2 165 bytes epoch 0) v1 -- 0x7fa0cc0aa3c0 con 0x7fa0cc06dde0 2024-06-03T09:06:16.257 INFO:teuthology.orchestra.run.smithi088.stderr:2024-06-03T09:06:16.256+0000 7fa0b3fff640 1 -- 172.21.15.88:0/3785650794 <== mon.0 v1:172.21.15.88:6789/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 2234374065 0 0) 0x7fa0bc001630 con 0x7fa0cc06dde0 2024-06-03T09:06:16.257 INFO:teuthology.orchestra.run.smithi088.stderr:2024-06-03T09:06:16.256+0000 7fa0b3fff640 10 monclient(hunting): handle_monmap mon_map magic: 0 v1 2024-06-03T09:06:16.257 INFO:teuthology.orchestra.run.smithi088.stderr:2024-06-03T09:06:16.256+0000 7fa0b3fff640 10 monclient(hunting): got monmap 1 from mon.noname-a (according to old e1) 2024-06-03T09:06:16.257 INFO:teuthology.orchestra.run.smithi088.stderr:2024-06-03T09:06:16.256+0000 7fa0b3fff640 10 monclient(hunting): dump: 2024-06-03T09:06:16.257 INFO:teuthology.orchestra.run.smithi088.stderr:epoch 1 2024-06-03T09:06:16.257 INFO:teuthology.orchestra.run.smithi088.stderr:fsid ec3c3fdd-3ac6-4d7d-9412-683130ed8b04 2024-06-03T09:06:16.257 INFO:teuthology.orchestra.run.smithi088.stderr:last_changed 2024-06-03T08:34:31.885125+0000 2024-06-03T09:06:16.257 INFO:teuthology.orchestra.run.smithi088.stderr:created 2024-06-03T08:34:31.885125+0000 2024-06-03T09:06:16.258 INFO:teuthology.orchestra.run.smithi088.stderr:min_mon_release 18 (reef) 2024-06-03T09:06:16.258 INFO:teuthology.orchestra.run.smithi088.stderr:election_strategy: 3 2024-06-03T09:06:16.258 INFO:teuthology.orchestra.run.smithi088.stderr:0: v1:172.21.15.88:6789/0 mon.a 2024-06-03T09:06:16.258 INFO:teuthology.orchestra.run.smithi088.stderr:1: v1:172.21.15.192:6789/0 mon.b 2024-06-03T09:06:16.258 INFO:teuthology.orchestra.run.smithi088.stderr:2: v1:172.21.15.88:6790/0 mon.c 2024-06-03T09:06:16.258 INFO:teuthology.orchestra.run.smithi088.stderr: 2024-06-03T09:06:16.258 INFO:teuthology.orchestra.run.smithi088.stderr:2024-06-03T09:06:16.256+0000 7fa0b3fff640 10 monclient(hunting): _finish_auth 0 2024-06-03T09:06:16.258 INFO:teuthology.orchestra.run.smithi088.stderr:2024-06-03T09:06:16.256+0000 7fa0b3fff640 1 -- 172.21.15.88:0/3785650794 <== mon.2 v1:172.21.15.88:6790/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 764+0+0 (unknown 305025884 0 0) 0x7fa0cc141930 con 0x7fa0cc06ea80 2024-06-03T09:06:16.258 INFO:teuthology.orchestra.run.smithi088.stderr:2024-06-03T09:06:16.256+0000 7fa0b3fff640 1 -- 172.21.15.88:0/3785650794 --> v1:172.21.15.88:6790/0 -- auth(proto 2 165 bytes epoch 0) v1 -- 0x7fa0b8004eb0 con 0x7fa0cc06ea80 2024-06-03T09:06:16.258 INFO:teuthology.orchestra.run.smithi088.stderr:2024-06-03T09:06:16.256+0000 7fa0b3fff640 1 -- 172.21.15.88:0/3785650794 <== mon.2 v1:172.21.15.88:6790/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 2234374065 0 0) 0x7fa0c0001bd0 con 0x7fa0cc06ea80 2024-06-03T09:06:16.258 INFO:teuthology.orchestra.run.smithi088.stderr:2024-06-03T09:06:16.256+0000 7fa0b3fff640 10 monclient(hunting): handle_monmap mon_map magic: 0 v1 2024-06-03T09:06:16.258 INFO:teuthology.orchestra.run.smithi088.stderr:2024-06-03T09:06:16.256+0000 7fa0b3fff640 10 monclient(hunting): got monmap 1 from mon.c (according to old e1) 2024-06-03T09:06:16.258 INFO:teuthology.orchestra.run.smithi088.stderr:2024-06-03T09:06:16.256+0000 7fa0b3fff640 10 monclient(hunting): dump: 2024-06-03T09:06:16.258 INFO:teuthology.orchestra.run.smithi088.stderr:epoch 1 2024-06-03T09:06:16.259 INFO:teuthology.orchestra.run.smithi088.stderr:fsid ec3c3fdd-3ac6-4d7d-9412-683130ed8b04 2024-06-03T09:06:16.259 INFO:teuthology.orchestra.run.smithi088.stderr:last_changed 2024-06-03T08:34:31.885125+0000 2024-06-03T09:06:16.259 INFO:teuthology.orchestra.run.smithi088.stderr:created 2024-06-03T08:34:31.885125+0000 2024-06-03T09:06:16.259 INFO:teuthology.orchestra.run.smithi088.stderr:min_mon_release 18 (reef) 2024-06-03T09:06:16.259 INFO:teuthology.orchestra.run.smithi088.stderr:election_strategy: 3 2024-06-03T09:06:16.259 INFO:teuthology.orchestra.run.smithi088.stderr:0: v1:172.21.15.88:6789/0 mon.a 2024-06-03T09:06:16.259 INFO:teuthology.orchestra.run.smithi088.stderr:1: v1:172.21.15.192:6789/0 mon.b 2024-06-03T09:06:16.259 INFO:teuthology.orchestra.run.smithi088.stderr:2: v1:172.21.15.88:6790/0 mon.c 2024-06-03T09:06:16.259 INFO:teuthology.orchestra.run.smithi088.stderr: 2024-06-03T09:06:16.260 INFO:teuthology.orchestra.run.smithi088.stderr:2024-06-03T09:06:16.257+0000 7fa0b3fff640 1 -- 172.21.15.88:0/3785650794 <== mon.1 v1:172.21.15.192:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 764+0+0 (unknown 1849957493 0 0) 0x7fa0cc0aa070 con 0x7fa0cc13d960 2024-06-03T09:06:16.260 INFO:teuthology.orchestra.run.smithi088.stderr:2024-06-03T09:06:16.257+0000 7fa0b3fff640 1 -- 172.21.15.88:0/3785650794 --> v1:172.21.15.192:6789/0 -- auth(proto 2 165 bytes epoch 0) v1 -- 0x7fa0cc141930 con 0x7fa0cc13d960 2024-06-03T09:06:16.261 INFO:teuthology.orchestra.run.smithi088.stderr:2024-06-03T09:06:16.257+0000 7fa0b3fff640 1 -- 172.21.15.88:0/3785650794 <== mon.1 v1:172.21.15.192:6789/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 2234374065 0 0) 0x7fa0c4001750 con 0x7fa0cc13d960 2024-06-03T09:06:16.261 INFO:teuthology.orchestra.run.smithi088.stderr:2024-06-03T09:06:16.257+0000 7fa0b3fff640 10 monclient(hunting): handle_monmap mon_map magic: 0 v1 2024-06-03T09:06:16.261 INFO:teuthology.orchestra.run.smithi088.stderr:2024-06-03T09:06:16.257+0000 7fa0b3fff640 10 monclient(hunting): got monmap 1 from mon.b (according to old e1) 2024-06-03T09:06:16.261 INFO:teuthology.orchestra.run.smithi088.stderr:2024-06-03T09:06:16.257+0000 7fa0b3fff640 10 monclient(hunting): dump: 2024-06-03T09:06:16.261 INFO:teuthology.orchestra.run.smithi088.stderr:epoch 1 2024-06-03T09:06:16.261 INFO:teuthology.orchestra.run.smithi088.stderr:fsid ec3c3fdd-3ac6-4d7d-9412-683130ed8b04 2024-06-03T09:06:16.261 INFO:teuthology.orchestra.run.smithi088.stderr:last_changed 2024-06-03T08:34:31.885125+0000 2024-06-03T09:06:16.261 INFO:teuthology.orchestra.run.smithi088.stderr:created 2024-06-03T08:34:31.885125+0000 2024-06-03T09:06:16.261 INFO:teuthology.orchestra.run.smithi088.stderr:min_mon_release 18 (reef) 2024-06-03T09:06:16.261 INFO:teuthology.orchestra.run.smithi088.stderr:election_strategy: 3 2024-06-03T09:06:16.261 INFO:teuthology.orchestra.run.smithi088.stderr:0: v1:172.21.15.88:6789/0 mon.a 2024-06-03T09:06:16.261 INFO:teuthology.orchestra.run.smithi088.stderr:1: v1:172.21.15.192:6789/0 mon.b 2024-06-03T09:06:16.261 INFO:teuthology.orchestra.run.smithi088.stderr:2: v1:172.21.15.88:6790/0 mon.c 2024-06-03T09:06:16.261 INFO:teuthology.orchestra.run.smithi088.stderr: 2024-06-03T09:06:16.262 INFO:teuthology.orchestra.run.smithi088.stderr:2024-06-03T09:06:16.260+0000 7fa0b3fff640 1 -- 172.21.15.88:0/3785650794 <== mon.0 v1:172.21.15.88:6789/0 4 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 580+0+0 (unknown 4178562496 0 0) 0x7fa0cc0aa3c0 con 0x7fa0cc06dde0 2024-06-03T09:06:16.262 INFO:teuthology.orchestra.run.smithi088.stderr:2024-06-03T09:06:16.260+0000 7fa0b3fff640 1 -- 172.21.15.88:0/3785650794 >> v1:172.21.15.88:6790/0 conn(0x7fa0cc06ea80 legacy=0x7fa0cc0a9900 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-06-03T09:06:16.262 INFO:teuthology.orchestra.run.smithi088.stderr:2024-06-03T09:06:16.260+0000 7fa0b3fff640 1 -- 172.21.15.88:0/3785650794 >> v1:172.21.15.192:6789/0 conn(0x7fa0cc13d960 legacy=0x7fa0cc1d5610 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-06-03T09:06:16.262 INFO:teuthology.orchestra.run.smithi088.stderr:2024-06-03T09:06:16.260+0000 7fa0b3fff640 10 monclient: _finish_hunting 0 2024-06-03T09:06:16.262 INFO:teuthology.orchestra.run.smithi088.stderr:2024-06-03T09:06:16.260+0000 7fa0b3fff640 1 monclient: found mon.a 2024-06-03T09:06:16.262 INFO:teuthology.orchestra.run.smithi088.stderr:2024-06-03T09:06:16.260+0000 7fa0b3fff640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-06-03T09:06:16.262 INFO:teuthology.orchestra.run.smithi088.stderr:2024-06-03T09:06:16.260+0000 7fa0b3fff640 10 monclient: _send_mon_message to mon.a at v1:172.21.15.88:6789/0 2024-06-03T09:06:16.262 INFO:teuthology.orchestra.run.smithi088.stderr:2024-06-03T09:06:16.260+0000 7fa0b3fff640 1 -- 172.21.15.88:0/3785650794 --> v1:172.21.15.88:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fa0cc140d10 con 0x7fa0cc06dde0 2024-06-03T09:06:16.262 INFO:teuthology.orchestra.run.smithi088.stderr:2024-06-03T09:06:16.260+0000 7fa0b3fff640 10 monclient: _finish_auth 0 2024-06-03T09:06:16.262 INFO:teuthology.orchestra.run.smithi088.stderr:2024-06-03T09:06:16.260+0000 7fa0b3fff640 10 monclient: _check_auth_tickets 2024-06-03T09:06:16.262 INFO:teuthology.orchestra.run.smithi088.stderr:2024-06-03T09:06:16.260+0000 7fa0b3fff640 20 monclient: _check_auth_rotating not needed by client.admin 2024-06-03T09:06:16.262 INFO:teuthology.orchestra.run.smithi088.stderr:2024-06-03T09:06:16.260+0000 7fa0d201c640 5 monclient: authenticate success, global_id 30570 2024-06-03T09:06:16.262 INFO:teuthology.orchestra.run.smithi088.stderr:2024-06-03T09:06:16.260+0000 7fa0d201c640 10 monclient: _renew_subs 2024-06-03T09:06:16.263 INFO:teuthology.orchestra.run.smithi088.stderr:2024-06-03T09:06:16.260+0000 7fa0d201c640 10 monclient: _send_mon_message to mon.a at v1:172.21.15.88:6789/0 2024-06-03T09:06:16.263 INFO:teuthology.orchestra.run.smithi088.stderr:2024-06-03T09:06:16.260+0000 7fa0d201c640 1 -- 172.21.15.88:0/3785650794 --> v1:172.21.15.88:6789/0 -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fa0cc140f60 con 0x7fa0cc06dde0 2024-06-03T09:06:16.263 INFO:teuthology.orchestra.run.smithi088.stderr:2024-06-03T09:06:16.260+0000 7fa0d201c640 10 client.30570.objecter _maybe_request_map subscribing (onetime) to next osd map 2024-06-03T09:06:16.263 INFO:teuthology.orchestra.run.smithi088.stderr:2024-06-03T09:06:16.260+0000 7fa0d201c640 10 monclient: _renew_subs 2024-06-03T09:06:16.263 INFO:teuthology.orchestra.run.smithi088.stderr:2024-06-03T09:06:16.260+0000 7fa0d201c640 10 monclient: _send_mon_message to mon.a at v1:172.21.15.88:6789/0 2024-06-03T09:06:16.263 INFO:teuthology.orchestra.run.smithi088.stderr:2024-06-03T09:06:16.260+0000 7fa0d201c640 1 -- 172.21.15.88:0/3785650794 --> v1:172.21.15.88:6789/0 -- mon_subscribe({osdmap=0}) v3 -- 0x7fa0cc138220 con 0x7fa0cc06dde0 2024-06-03T09:06:16.263 INFO:teuthology.orchestra.run.smithi088.stderr:2024-06-03T09:06:16.260+0000 7fa0d201c640 1 librados: init done 2024-06-03T09:06:16.263 INFO:teuthology.orchestra.run.smithi088.stderr:2024-06-03T09:06:16.261+0000 7fa0d201c640 10 start_mon_command cmd=[{"prefix": "get_command_descriptions"}] 2024-06-03T09:06:16.264 INFO:teuthology.orchestra.run.smithi088.stderr:2024-06-03T09:06:16.261+0000 7fa0d201c640 10 monclient: _send_command 1 [{"prefix": "get_command_descriptions"}] 2024-06-03T09:06:16.264 INFO:teuthology.orchestra.run.smithi088.stderr:2024-06-03T09:06:16.261+0000 7fa0d201c640 10 monclient: _send_mon_message to mon.a at v1:172.21.15.88:6789/0 2024-06-03T09:06:16.264 INFO:teuthology.orchestra.run.smithi088.stderr:2024-06-03T09:06:16.261+0000 7fa0d201c640 1 -- 172.21.15.88:0/3785650794 --> v1:172.21.15.88:6789/0 -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fa0cc0aa200 con 0x7fa0cc06dde0 2024-06-03T09:06:16.503 INFO:teuthology.orchestra.run.smithi088.stderr:2024-06-03T09:06:16.503+0000 7fa0b3fff640 1 -- 172.21.15.88:0/3785650794 <== mon.0 v1:172.21.15.88:6789/0 5 ==== config(0 keys) v1 ==== 4+0+0 (unknown 0 0 0) 0x7fa0bc004b10 con 0x7fa0cc06dde0 2024-06-03T09:06:16.503 INFO:teuthology.orchestra.run.smithi088.stderr:2024-06-03T09:06:16.503+0000 7fa0b3fff640 10 monclient: handle_config config(0 keys) v1 2024-06-03T09:06:16.503 INFO:teuthology.orchestra.run.smithi088.stderr:2024-06-03T09:06:16.503+0000 7fa0b3fff640 1 -- 172.21.15.88:0/3785650794 <== mon.0 v1:172.21.15.88:6789/0 6 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 2234374065 0 0) 0x7fa0bc004ea0 con 0x7fa0cc06dde0 2024-06-03T09:06:16.503 INFO:teuthology.orchestra.run.smithi088.stderr:2024-06-03T09:06:16.503+0000 7fa0b3fff640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-06-03T09:06:16.504 INFO:teuthology.orchestra.run.smithi088.stderr:2024-06-03T09:06:16.503+0000 7fa0b3fff640 10 monclient: got monmap 1 from mon.a (according to old e1) 2024-06-03T09:06:16.504 INFO:teuthology.orchestra.run.smithi088.stderr:2024-06-03T09:06:16.503+0000 7fa0b3fff640 10 monclient: dump: 2024-06-03T09:06:16.504 INFO:teuthology.orchestra.run.smithi088.stderr:epoch 1 2024-06-03T09:06:16.504 INFO:teuthology.orchestra.run.smithi088.stderr:fsid ec3c3fdd-3ac6-4d7d-9412-683130ed8b04 2024-06-03T09:06:16.504 INFO:teuthology.orchestra.run.smithi088.stderr:last_changed 2024-06-03T08:34:31.885125+0000 2024-06-03T09:06:16.504 INFO:teuthology.orchestra.run.smithi088.stderr:created 2024-06-03T08:34:31.885125+0000 2024-06-03T09:06:16.504 INFO:teuthology.orchestra.run.smithi088.stderr:min_mon_release 18 (reef) 2024-06-03T09:06:16.504 INFO:teuthology.orchestra.run.smithi088.stderr:election_strategy: 3 2024-06-03T09:06:16.504 INFO:teuthology.orchestra.run.smithi088.stderr:0: v1:172.21.15.88:6789/0 mon.a 2024-06-03T09:06:16.504 INFO:teuthology.orchestra.run.smithi088.stderr:1: v1:172.21.15.192:6789/0 mon.b 2024-06-03T09:06:16.504 INFO:teuthology.orchestra.run.smithi088.stderr:2: v1:172.21.15.88:6790/0 mon.c 2024-06-03T09:06:16.504 INFO:teuthology.orchestra.run.smithi088.stderr: 2024-06-03T09:06:16.504 INFO:teuthology.orchestra.run.smithi088.stderr:2024-06-03T09:06:16.503+0000 7fa0c9ffb640 4 set_mon_vals no callback set 2024-06-03T09:06:16.505 INFO:teuthology.orchestra.run.smithi088.stderr:2024-06-03T09:06:16.503+0000 7fa0b3fff640 1 -- 172.21.15.88:0/3785650794 <== mon.0 v1:172.21.15.88:6789/0 7 ==== mgrmap(e 5) v1 ==== 89110+0+0 (unknown 873533973 0 0) 0x7fa0bc005000 con 0x7fa0cc06dde0 2024-06-03T09:06:16.505 INFO:teuthology.orchestra.run.smithi088.stderr:2024-06-03T09:06:16.504+0000 7fa0caffd640 10 monclient: get_auth_request con 0x7fa0b806d3e0 auth_method 0 2024-06-03T09:06:16.505 INFO:teuthology.orchestra.run.smithi088.stderr:2024-06-03T09:06:16.504+0000 7fa0b3fff640 10 client.30570.objecter ms_handle_connect 0x7fa0b806d3e0 2024-06-03T09:06:16.508 INFO:teuthology.orchestra.run.smithi088.stderr:2024-06-03T09:06:16.508+0000 7fa0b3fff640 1 -- 172.21.15.88:0/3785650794 <== mon.0 v1:172.21.15.88:6789/0 8 ==== osd_map(1009..1009 src has 622..1009) v4 ==== 5229+0+0 (unknown 553880507 0 0) 0x7fa0bc0542f0 con 0x7fa0cc06dde0 2024-06-03T09:06:16.508 INFO:teuthology.orchestra.run.smithi088.stderr:2024-06-03T09:06:16.508+0000 7fa0b3fff640 10 client.30570.objecter ms_dispatch 0x7fa0cc000b90 osd_map(1009..1009 src has 622..1009) v4 2024-06-03T09:06:16.508 INFO:teuthology.orchestra.run.smithi088.stderr:2024-06-03T09:06:16.508+0000 7fa0b3fff640 3 client.30570.objecter handle_osd_map got epochs [1009,1009] > 0 2024-06-03T09:06:16.508 INFO:teuthology.orchestra.run.smithi088.stderr:2024-06-03T09:06:16.508+0000 7fa0b3fff640 3 client.30570.objecter handle_osd_map decoding full epoch 1009 2024-06-03T09:06:16.508 INFO:teuthology.orchestra.run.smithi088.stderr:2024-06-03T09:06:16.508+0000 7fa0b3fff640 20 client.30570.objecter dump_active .. 0 homeless 2024-06-03T09:06:16.508 INFO:teuthology.orchestra.run.smithi088.stderr:2024-06-03T09:06:16.508+0000 7fa0b3fff640 1 -- 172.21.15.88:0/3785650794 <== mon.0 v1:172.21.15.88:6789/0 9 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+182498 (unknown 1092875540 0 3016760700) 0x7fa0bc04fea0 con 0x7fa0cc06dde0 2024-06-03T09:06:16.508 INFO:teuthology.orchestra.run.smithi088.stderr:2024-06-03T09:06:16.508+0000 7fa0b3fff640 10 monclient: handle_mon_command_ack 1 [{"prefix": "get_command_descriptions"}] 2024-06-03T09:06:16.508 INFO:teuthology.orchestra.run.smithi088.stderr:2024-06-03T09:06:16.508+0000 7fa0b3fff640 10 monclient: _finish_command 1 = system:0 2024-06-03T09:06:16.750 INFO:teuthology.orchestra.run.smithi088.stderr:2024-06-03T09:06:16.750+0000 7fa0d201c640 10 start_mon_command cmd=[{"prefix": "status", "format": "json"}] 2024-06-03T09:06:16.750 INFO:teuthology.orchestra.run.smithi088.stderr:2024-06-03T09:06:16.750+0000 7fa0d201c640 10 monclient: _send_command 2 [{"prefix": "status", "format": "json"}] 2024-06-03T09:06:16.751 INFO:teuthology.orchestra.run.smithi088.stderr:2024-06-03T09:06:16.750+0000 7fa0d201c640 10 monclient: _send_mon_message to mon.a at v1:172.21.15.88:6789/0 2024-06-03T09:06:16.751 INFO:teuthology.orchestra.run.smithi088.stderr:2024-06-03T09:06:16.750+0000 7fa0d201c640 1 -- 172.21.15.88:0/3785650794 --> v1:172.21.15.88:6789/0 -- mon_command({"prefix": "status", "format": "json"} v 0) v1 -- 0x7fa0cc04ea60 con 0x7fa0cc06dde0 2024-06-03T09:06:16.768 INFO:teuthology.orchestra.run.smithi088.stderr:2024-06-03T09:06:16.769+0000 7fa0b3fff640 1 -- 172.21.15.88:0/3785650794 <== mon.0 v1:172.21.15.88:6789/0 10 ==== mon_command_ack([{"prefix": "status", "format": "json"}]=0 v0) v1 ==== 72+0+1431 (unknown 3226119948 0 942995608) 0x7fa0bc051dd0 con 0x7fa0cc06dde0 2024-06-03T09:06:16.769 INFO:teuthology.orchestra.run.smithi088.stderr:2024-06-03T09:06:16.769+0000 7fa0b3fff640 10 monclient: handle_mon_command_ack 2 [{"prefix": "status", "format": "json"}] 2024-06-03T09:06:16.769 INFO:teuthology.orchestra.run.smithi088.stderr:2024-06-03T09:06:16.769+0000 7fa0b3fff640 10 monclient: _finish_command 2 = system:0 2024-06-03T09:06:16.769 INFO:teuthology.orchestra.run.smithi088.stdout: 2024-06-03T09:06:16.769 INFO:teuthology.orchestra.run.smithi088.stdout:{"fsid":"ec3c3fdd-3ac6-4d7d-9412-683130ed8b04","health":{"status":"HEALTH_WARN","checks":{"OSD_DOWN":{"severity":"HEALTH_WARN","summary":{"message":"1 osds down","count":1},"muted":false},"PG_DEGRADED":{"severity":"HEALTH_WARN","summary":{"message":"Degraded data redundancy: 1/6 objects degraded (16.667%), 1 pg degraded","count":1},"muted":false},"POOL_APP_NOT_ENABLED":{"severity":"HEALTH_WARN","summary":{"message":"1 pool(s) do not have an application enabled","count":1},"muted":false}},"mutes":[]},"election_epoch":10,"quorum":[0,1,2],"quorum_names":["a","b","c"],"quorum_age":1850,"monmap":{"epoch":1,"min_mon_release_name":"reef","num_mons":3},"osdmap":{"epoch":1009,"num_osds":8,"num_up_osds":7,"osd_up_since":1717405562,"num_in_osds":8,"osd_in_since":1717403732,"num_remapped_pgs":0},"pgmap":{"pgs_by_state":[{"state_name":"active+clean","count":27},{"state_name":"active+undersized","count":13},{"state_name":"active+undersized+degraded","count":1}],"num_pgs":41,"num_pools":3,"num_objects":3,"data_bytes":459285,"bytes_used":506241024,"bytes_avail":772587872256,"bytes_total":773094113280,"degraded_objects":1,"degraded_total":6,"degraded_ratio":0.16666666666666663},"fsmap":{"epoch":1,"by_rank":[],"up:standby":0},"mgrmap":{"available":true,"num_standbys":1,"modules":["iostat","nfs","restful"],"services":{}},"servicemap":{"epoch":23,"modified":"2024-06-03T08:53:48.910215+0000","services":{}},"progress_events":{}} 2024-06-03T09:06:16.771 INFO:teuthology.orchestra.run.smithi088.stderr:2024-06-03T09:06:16.771+0000 7fa0d201c640 10 librados: watch_flush enter 2024-06-03T09:06:16.771 INFO:teuthology.orchestra.run.smithi088.stderr:2024-06-03T09:06:16.771+0000 7fa0d201c640 10 librados: watch_flush exit 2024-06-03T09:06:16.771 INFO:teuthology.orchestra.run.smithi088.stderr:2024-06-03T09:06:16.771+0000 7fa0d201c640 20 client.30570.objecter shutdown clearing up homeless session... 2024-06-03T09:06:16.771 INFO:teuthology.orchestra.run.smithi088.stderr:2024-06-03T09:06:16.771+0000 7fa0d201c640 10 client.30570.objecter successfully canceled tick 2024-06-03T09:06:16.771 INFO:teuthology.orchestra.run.smithi088.stderr:2024-06-03T09:06:16.771+0000 7fa0d201c640 1 -- 172.21.15.88:0/3785650794 >> v1:172.21.15.192:6800/34229 conn(0x7fa0b806d3e0 legacy=0x7fa0b806f8a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-06-03T09:06:16.772 INFO:teuthology.orchestra.run.smithi088.stderr:2024-06-03T09:06:16.771+0000 7fa0d201c640 10 monclient: shutdown 2024-06-03T09:06:16.772 INFO:teuthology.orchestra.run.smithi088.stderr:2024-06-03T09:06:16.771+0000 7fa0d201c640 20 monclient: shutdown discarding 0 pending message(s) 2024-06-03T09:06:16.772 INFO:teuthology.orchestra.run.smithi088.stderr:2024-06-03T09:06:16.771+0000 7fa0d201c640 1 -- 172.21.15.88:0/3785650794 >> v1:172.21.15.88:6789/0 conn(0x7fa0cc06dde0 legacy=0x7fa0cc0aa8e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-06-03T09:06:16.772 INFO:teuthology.orchestra.run.smithi088.stderr:2024-06-03T09:06:16.771+0000 7fa0d201c640 1 -- 172.21.15.88:0/3785650794 shutdown_connections 2024-06-03T09:06:16.772 INFO:teuthology.orchestra.run.smithi088.stderr:2024-06-03T09:06:16.771+0000 7fa0d201c640 1 -- 172.21.15.88:0/3785650794 >> 172.21.15.88:0/3785650794 conn(0x7fa0cc0b2e40 msgr2=0x7fa0cc0737f0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-06-03T09:06:16.772 INFO:teuthology.orchestra.run.smithi088.stderr:2024-06-03T09:06:16.771+0000 7fa0d201c640 1 -- 172.21.15.88:0/3785650794 shutdown_connections 2024-06-03T09:06:16.772 INFO:teuthology.orchestra.run.smithi088.stderr:2024-06-03T09:06:16.771+0000 7fa0d201c640 1 -- 172.21.15.88:0/3785650794 wait complete. 2024-06-03T09:06:16.772 INFO:teuthology.orchestra.run.smithi088.stderr:2024-06-03T09:06:16.771+0000 7fa0d201c640 1 librados: shutdown 2024-06-03T09:06:16.782 INFO:tasks.ceph.ceph_manager.ceph:no progress seen, keeping timeout for now 2024-06-03T09:06:16.782 DEBUG:teuthology.orchestra.run.smithi088:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph pg dump --format=json 2024-06-03T09:06:16.879 INFO:teuthology.orchestra.run.smithi088.stderr:2024-06-03T09:06:16.879+0000 7fb509173640 10 monclient: get_monmap_and_config 2024-06-03T09:06:16.879 INFO:teuthology.orchestra.run.smithi088.stderr:2024-06-03T09:06:16.879+0000 7fb509173640 10 monclient: build_initial_monmap 2024-06-03T09:06:16.879 INFO:teuthology.orchestra.run.smithi088.stderr:2024-06-03T09:06:16.879+0000 7fb509173640 10 monclient: monmap: 2024-06-03T09:06:16.879 INFO:teuthology.orchestra.run.smithi088.stderr:epoch 0 2024-06-03T09:06:16.879 INFO:teuthology.orchestra.run.smithi088.stderr:fsid ec3c3fdd-3ac6-4d7d-9412-683130ed8b04 2024-06-03T09:06:16.879 INFO:teuthology.orchestra.run.smithi088.stderr:last_changed 2024-06-03T09:06:16.880172+0000 2024-06-03T09:06:16.879 INFO:teuthology.orchestra.run.smithi088.stderr:created 2024-06-03T09:06:16.880172+0000 2024-06-03T09:06:16.879 INFO:teuthology.orchestra.run.smithi088.stderr:min_mon_release 0 (unknown) 2024-06-03T09:06:16.879 INFO:teuthology.orchestra.run.smithi088.stderr:election_strategy: 3 2024-06-03T09:06:16.880 INFO:teuthology.orchestra.run.smithi088.stderr:0: v1:172.21.15.88:6789/0 mon.noname-a 2024-06-03T09:06:16.880 INFO:teuthology.orchestra.run.smithi088.stderr:1: v1:172.21.15.192:6789/0 mon.noname-c 2024-06-03T09:06:16.880 INFO:teuthology.orchestra.run.smithi088.stderr:2: v1:172.21.15.88:6790/0 mon.noname-b 2024-06-03T09:06:16.880 INFO:teuthology.orchestra.run.smithi088.stderr: 2024-06-03T09:06:16.881 INFO:teuthology.orchestra.run.smithi088.stderr:2024-06-03T09:06:16.879+0000 7fb509173640 1 Processor -- start 2024-06-03T09:06:16.881 INFO:teuthology.orchestra.run.smithi088.stderr:2024-06-03T09:06:16.879+0000 7fb509173640 1 -- start start 2024-06-03T09:06:16.881 INFO:teuthology.orchestra.run.smithi088.stderr:2024-06-03T09:06:16.879+0000 7fb509173640 10 monclient: init 2024-06-03T09:06:16.882 INFO:teuthology.orchestra.run.smithi088.stderr:2024-06-03T09:06:16.880+0000 7fb509173640 10 monclient: _reopen_session rank -1 2024-06-03T09:06:16.882 INFO:teuthology.orchestra.run.smithi088.stderr:2024-06-03T09:06:16.880+0000 7fb509173640 10 monclient: _add_conns ranks=[0,2,1] 2024-06-03T09:06:16.882 INFO:teuthology.orchestra.run.smithi088.stderr:2024-06-03T09:06:16.880+0000 7fb509173640 10 monclient(hunting): picked mon.noname-a con 0x7fb5040a8960 addr v1:172.21.15.88:6789/0 2024-06-03T09:06:16.882 INFO:teuthology.orchestra.run.smithi088.stderr:2024-06-03T09:06:16.880+0000 7fb509173640 10 monclient(hunting): picked mon.noname-b con 0x7fb5041450f0 addr v1:172.21.15.88:6790/0 2024-06-03T09:06:16.882 INFO:teuthology.orchestra.run.smithi088.stderr:2024-06-03T09:06:16.880+0000 7fb509173640 10 monclient(hunting): picked mon.noname-c con 0x7fb50413eec0 addr v1:172.21.15.192:6789/0 2024-06-03T09:06:16.882 INFO:teuthology.orchestra.run.smithi088.stderr:2024-06-03T09:06:16.880+0000 7fb509173640 1 -- --> v1:172.21.15.88:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fb504144c60 con 0x7fb5040a8960 2024-06-03T09:06:16.882 INFO:teuthology.orchestra.run.smithi088.stderr:2024-06-03T09:06:16.880+0000 7fb509173640 1 -- --> v1:172.21.15.192:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fb5040b3930 con 0x7fb50413eec0 2024-06-03T09:06:16.882 INFO:teuthology.orchestra.run.smithi088.stderr:2024-06-03T09:06:16.880+0000 7fb509173640 1 -- --> v1:172.21.15.88:6790/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fb5040b1ad0 con 0x7fb5041450f0 2024-06-03T09:06:16.882 INFO:teuthology.orchestra.run.smithi088.stderr:2024-06-03T09:06:16.880+0000 7fb509173640 10 monclient(hunting): _renew_subs 2024-06-03T09:06:16.882 INFO:teuthology.orchestra.run.smithi088.stderr:2024-06-03T09:06:16.880+0000 7fb509173640 10 monclient(hunting): authenticate will time out at 3032.690464s 2024-06-03T09:06:16.882 INFO:teuthology.orchestra.run.smithi088.stderr:2024-06-03T09:06:16.881+0000 7fb50259c640 1 --1- >> v1:172.21.15.88:6790/0 conn(0x7fb5041450f0 0x7fb50413e7b0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.88:6790/0 says I am v1:172.21.15.88:60438/0 (socket says 172.21.15.88:60438) 2024-06-03T09:06:16.882 INFO:teuthology.orchestra.run.smithi088.stderr:2024-06-03T09:06:16.881+0000 7fb50259c640 1 -- 172.21.15.88:0/1262741288 learned_addr learned my addr 172.21.15.88:0/1262741288 (peer_addr_for_me v1:172.21.15.88:0/0) 2024-06-03T09:06:16.883 INFO:teuthology.orchestra.run.smithi088.stderr:2024-06-03T09:06:16.883+0000 7fb501d9b640 1 -- 172.21.15.88:0/1262741288 <== mon.2 v1:172.21.15.88:6790/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 1754110192 0 0) 0x7fb5040b1ad0 con 0x7fb5041450f0 2024-06-03T09:06:16.883 INFO:teuthology.orchestra.run.smithi088.stderr:2024-06-03T09:06:16.883+0000 7fb501d9b640 10 monclient(hunting): _negotiate 2024-06-03T09:06:16.883 INFO:teuthology.orchestra.run.smithi088.stderr:2024-06-03T09:06:16.883+0000 7fb501d9b640 10 monclient(hunting): _init_auth method 2 2024-06-03T09:06:16.883 INFO:teuthology.orchestra.run.smithi088.stderr:2024-06-03T09:06:16.883+0000 7fb501d9b640 10 monclient(hunting): _init_auth creating new auth 2024-06-03T09:06:16.883 INFO:teuthology.orchestra.run.smithi088.stderr:2024-06-03T09:06:16.883+0000 7fb501d9b640 10 monclient(hunting): my global_id is 30074 2024-06-03T09:06:16.883 INFO:teuthology.orchestra.run.smithi088.stderr:2024-06-03T09:06:16.883+0000 7fb501d9b640 1 -- 172.21.15.88:0/1262741288 --> v1:172.21.15.88:6790/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7fb4f0003520 con 0x7fb5041450f0 2024-06-03T09:06:16.883 INFO:teuthology.orchestra.run.smithi088.stderr:2024-06-03T09:06:16.883+0000 7fb501d9b640 1 -- 172.21.15.88:0/1262741288 <== mon.1 v1:172.21.15.192:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 2365617799 0 0) 0x7fb5040b3930 con 0x7fb50413eec0 2024-06-03T09:06:16.884 INFO:teuthology.orchestra.run.smithi088.stderr:2024-06-03T09:06:16.883+0000 7fb501d9b640 10 monclient(hunting): _negotiate 2024-06-03T09:06:16.884 INFO:teuthology.orchestra.run.smithi088.stderr:2024-06-03T09:06:16.883+0000 7fb501d9b640 10 monclient(hunting): _init_auth method 2 2024-06-03T09:06:16.884 INFO:teuthology.orchestra.run.smithi088.stderr:2024-06-03T09:06:16.883+0000 7fb501d9b640 10 monclient(hunting): _init_auth creating new auth 2024-06-03T09:06:16.884 INFO:teuthology.orchestra.run.smithi088.stderr:2024-06-03T09:06:16.883+0000 7fb501d9b640 10 monclient(hunting): my global_id is 30151 2024-06-03T09:06:16.884 INFO:teuthology.orchestra.run.smithi088.stderr:2024-06-03T09:06:16.883+0000 7fb501d9b640 1 -- 172.21.15.88:0/1262741288 --> v1:172.21.15.192:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7fb5040b1ad0 con 0x7fb50413eec0 2024-06-03T09:06:16.886 INFO:teuthology.orchestra.run.smithi088.stderr:2024-06-03T09:06:16.886+0000 7fb501d9b640 1 -- 172.21.15.88:0/1262741288 <== mon.1 v1:172.21.15.192:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 214+0+0 (unknown 1180340376 0 0) 0x7fb5040b1ad0 con 0x7fb50413eec0 2024-06-03T09:06:16.886 INFO:teuthology.orchestra.run.smithi088.stderr:2024-06-03T09:06:16.886+0000 7fb501d9b640 1 -- 172.21.15.88:0/1262741288 >> v1:172.21.15.88:6790/0 conn(0x7fb5041450f0 legacy=0x7fb50413e7b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-06-03T09:06:16.886 INFO:teuthology.orchestra.run.smithi088.stderr:2024-06-03T09:06:16.886+0000 7fb501d9b640 1 -- 172.21.15.88:0/1262741288 >> v1:172.21.15.88:6789/0 conn(0x7fb5040a8960 legacy=0x7fb5040a8d40 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-06-03T09:06:16.886 INFO:teuthology.orchestra.run.smithi088.stderr:2024-06-03T09:06:16.886+0000 7fb501d9b640 10 monclient: _finish_hunting 0 2024-06-03T09:06:16.886 INFO:teuthology.orchestra.run.smithi088.stderr:2024-06-03T09:06:16.886+0000 7fb501d9b640 1 monclient: found mon.noname-c 2024-06-03T09:06:16.886 INFO:teuthology.orchestra.run.smithi088.stderr:2024-06-03T09:06:16.886+0000 7fb501d9b640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-06-03T09:06:16.886 INFO:teuthology.orchestra.run.smithi088.stderr:2024-06-03T09:06:16.886+0000 7fb501d9b640 10 monclient: _send_mon_message to mon.noname-c at v1:172.21.15.192:6789/0 2024-06-03T09:06:16.887 INFO:teuthology.orchestra.run.smithi088.stderr:2024-06-03T09:06:16.886+0000 7fb501d9b640 1 -- 172.21.15.88:0/1262741288 --> v1:172.21.15.192:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fb5040b1c60 con 0x7fb50413eec0 2024-06-03T09:06:16.887 INFO:teuthology.orchestra.run.smithi088.stderr:2024-06-03T09:06:16.886+0000 7fb501d9b640 10 monclient: _finish_auth 0 2024-06-03T09:06:16.887 INFO:teuthology.orchestra.run.smithi088.stderr:2024-06-03T09:06:16.886+0000 7fb501d9b640 10 monclient: _check_auth_tickets 2024-06-03T09:06:16.887 INFO:teuthology.orchestra.run.smithi088.stderr:2024-06-03T09:06:16.886+0000 7fb501d9b640 20 monclient: _check_auth_rotating not needed by client.admin 2024-06-03T09:06:16.887 INFO:teuthology.orchestra.run.smithi088.stderr:2024-06-03T09:06:16.886+0000 7fb501d9b640 1 -- 172.21.15.88:0/1262741288 <== mon.1 v1:172.21.15.192:6789/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 2234374065 0 0) 0x7fb4fc001910 con 0x7fb50413eec0 2024-06-03T09:06:16.887 INFO:teuthology.orchestra.run.smithi088.stderr:2024-06-03T09:06:16.886+0000 7fb501d9b640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-06-03T09:06:16.887 INFO:teuthology.orchestra.run.smithi088.stderr:2024-06-03T09:06:16.886+0000 7fb501d9b640 10 monclient: got monmap 1 from mon.noname-c (according to old e1) 2024-06-03T09:06:16.887 INFO:teuthology.orchestra.run.smithi088.stderr:2024-06-03T09:06:16.886+0000 7fb501d9b640 10 monclient: dump: 2024-06-03T09:06:16.887 INFO:teuthology.orchestra.run.smithi088.stderr:epoch 1 2024-06-03T09:06:16.887 INFO:teuthology.orchestra.run.smithi088.stderr:fsid ec3c3fdd-3ac6-4d7d-9412-683130ed8b04 2024-06-03T09:06:16.888 INFO:teuthology.orchestra.run.smithi088.stderr:last_changed 2024-06-03T08:34:31.885125+0000 2024-06-03T09:06:16.888 INFO:teuthology.orchestra.run.smithi088.stderr:created 2024-06-03T08:34:31.885125+0000 2024-06-03T09:06:16.888 INFO:teuthology.orchestra.run.smithi088.stderr:min_mon_release 18 (reef) 2024-06-03T09:06:16.888 INFO:teuthology.orchestra.run.smithi088.stderr:election_strategy: 3 2024-06-03T09:06:16.888 INFO:teuthology.orchestra.run.smithi088.stderr:0: v1:172.21.15.88:6789/0 mon.a 2024-06-03T09:06:16.888 INFO:teuthology.orchestra.run.smithi088.stderr:1: v1:172.21.15.192:6789/0 mon.b 2024-06-03T09:06:16.888 INFO:teuthology.orchestra.run.smithi088.stderr:2: v1:172.21.15.88:6790/0 mon.c 2024-06-03T09:06:16.888 INFO:teuthology.orchestra.run.smithi088.stderr: 2024-06-03T09:06:16.888 INFO:teuthology.orchestra.run.smithi088.stderr:2024-06-03T09:06:16.886+0000 7fb509173640 5 monclient: authenticate success, global_id 30151 2024-06-03T09:06:16.888 INFO:teuthology.orchestra.run.smithi088.stderr:2024-06-03T09:06:16.886+0000 7fb509173640 20 monclient: get_monmap_and_config waiting for monmap|config 2024-06-03T09:06:16.890 INFO:teuthology.orchestra.run.smithi088.stderr:2024-06-03T09:06:16.888+0000 7fb501d9b640 1 -- 172.21.15.88:0/1262741288 <== mon.1 v1:172.21.15.192:6789/0 4 ==== config(0 keys) v1 ==== 4+0+0 (unknown 0 0 0) 0x7fb4fc001b30 con 0x7fb50413eec0 2024-06-03T09:06:16.890 INFO:teuthology.orchestra.run.smithi088.stderr:2024-06-03T09:06:16.888+0000 7fb501d9b640 10 monclient: handle_config config(0 keys) v1 2024-06-03T09:06:16.890 INFO:teuthology.orchestra.run.smithi088.stderr:2024-06-03T09:06:16.888+0000 7fb501d9b640 1 -- 172.21.15.88:0/1262741288 <== mon.1 v1:172.21.15.192:6789/0 5 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 2234374065 0 0) 0x7fb4fc001e40 con 0x7fb50413eec0 2024-06-03T09:06:16.890 INFO:teuthology.orchestra.run.smithi088.stderr:2024-06-03T09:06:16.888+0000 7fb501d9b640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-06-03T09:06:16.890 INFO:teuthology.orchestra.run.smithi088.stderr:2024-06-03T09:06:16.888+0000 7fb501d9b640 10 monclient: got monmap 1 from mon.b (according to old e1) 2024-06-03T09:06:16.890 INFO:teuthology.orchestra.run.smithi088.stderr:2024-06-03T09:06:16.888+0000 7fb501d9b640 10 monclient: dump: 2024-06-03T09:06:16.890 INFO:teuthology.orchestra.run.smithi088.stderr:epoch 1 2024-06-03T09:06:16.890 INFO:teuthology.orchestra.run.smithi088.stderr:fsid ec3c3fdd-3ac6-4d7d-9412-683130ed8b04 2024-06-03T09:06:16.890 INFO:teuthology.orchestra.run.smithi088.stderr:last_changed 2024-06-03T08:34:31.885125+0000 2024-06-03T09:06:16.890 INFO:teuthology.orchestra.run.smithi088.stderr:created 2024-06-03T08:34:31.885125+0000 2024-06-03T09:06:16.890 INFO:teuthology.orchestra.run.smithi088.stderr:min_mon_release 18 (reef) 2024-06-03T09:06:16.891 INFO:teuthology.orchestra.run.smithi088.stderr:election_strategy: 3 2024-06-03T09:06:16.891 INFO:teuthology.orchestra.run.smithi088.stderr:0: v1:172.21.15.88:6789/0 mon.a 2024-06-03T09:06:16.891 INFO:teuthology.orchestra.run.smithi088.stderr:1: v1:172.21.15.192:6789/0 mon.b 2024-06-03T09:06:16.891 INFO:teuthology.orchestra.run.smithi088.stderr:2: v1:172.21.15.88:6790/0 mon.c 2024-06-03T09:06:16.891 INFO:teuthology.orchestra.run.smithi088.stderr: 2024-06-03T09:06:16.891 INFO:teuthology.orchestra.run.smithi088.stderr:2024-06-03T09:06:16.888+0000 7fb509173640 10 monclient: get_monmap_and_config success 2024-06-03T09:06:16.891 INFO:teuthology.orchestra.run.smithi088.stderr:2024-06-03T09:06:16.888+0000 7fb509173640 4 set_mon_vals no callback set 2024-06-03T09:06:16.891 INFO:teuthology.orchestra.run.smithi088.stderr:2024-06-03T09:06:16.889+0000 7fb509173640 10 monclient: shutdown 2024-06-03T09:06:16.891 INFO:teuthology.orchestra.run.smithi088.stderr:2024-06-03T09:06:16.889+0000 7fb509173640 20 monclient: shutdown discarding 0 pending message(s) 2024-06-03T09:06:16.891 INFO:teuthology.orchestra.run.smithi088.stderr:2024-06-03T09:06:16.889+0000 7fb509173640 1 -- 172.21.15.88:0/1262741288 >> v1:172.21.15.192:6789/0 conn(0x7fb50413eec0 legacy=0x7fb50414ce60 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-06-03T09:06:16.891 INFO:teuthology.orchestra.run.smithi088.stderr:2024-06-03T09:06:16.889+0000 7fb509173640 1 -- 172.21.15.88:0/1262741288 shutdown_connections 2024-06-03T09:06:16.891 INFO:teuthology.orchestra.run.smithi088.stderr:2024-06-03T09:06:16.889+0000 7fb509173640 1 -- 172.21.15.88:0/1262741288 >> 172.21.15.88:0/1262741288 conn(0x7fb5040b70d0 msgr2=0x7fb5040b74e0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-06-03T09:06:16.891 INFO:teuthology.orchestra.run.smithi088.stderr:2024-06-03T09:06:16.889+0000 7fb509173640 1 -- 172.21.15.88:0/1262741288 shutdown_connections 2024-06-03T09:06:16.892 INFO:teuthology.orchestra.run.smithi088.stderr:2024-06-03T09:06:16.889+0000 7fb509173640 1 -- 172.21.15.88:0/1262741288 wait complete. 2024-06-03T09:06:16.892 INFO:teuthology.orchestra.run.smithi088.stderr:2024-06-03T09:06:16.889+0000 7fb509173640 10 monclient: build_initial_monmap 2024-06-03T09:06:16.892 INFO:teuthology.orchestra.run.smithi088.stderr:2024-06-03T09:06:16.890+0000 7fb509173640 10 monclient: monmap: 2024-06-03T09:06:16.892 INFO:teuthology.orchestra.run.smithi088.stderr:epoch 0 2024-06-03T09:06:16.892 INFO:teuthology.orchestra.run.smithi088.stderr:fsid 00000000-0000-0000-0000-000000000000 2024-06-03T09:06:16.892 INFO:teuthology.orchestra.run.smithi088.stderr:last_changed 0.000000 2024-06-03T09:06:16.892 INFO:teuthology.orchestra.run.smithi088.stderr:created 0.000000 2024-06-03T09:06:16.892 INFO:teuthology.orchestra.run.smithi088.stderr:min_mon_release 0 (unknown) 2024-06-03T09:06:16.892 INFO:teuthology.orchestra.run.smithi088.stderr:election_strategy: 1 2024-06-03T09:06:16.892 INFO:teuthology.orchestra.run.smithi088.stderr:0: v1:172.21.15.88:6789/0 mon.noname-a 2024-06-03T09:06:16.892 INFO:teuthology.orchestra.run.smithi088.stderr:1: v1:172.21.15.192:6789/0 mon.noname-b 2024-06-03T09:06:16.892 INFO:teuthology.orchestra.run.smithi088.stderr:2: v1:172.21.15.88:6790/0 mon.noname-c 2024-06-03T09:06:16.892 INFO:teuthology.orchestra.run.smithi088.stderr: 2024-06-03T09:06:16.892 INFO:teuthology.orchestra.run.smithi088.stderr:2024-06-03T09:06:16.890+0000 7fb509173640 1 librados: starting msgr at 2024-06-03T09:06:16.893 INFO:teuthology.orchestra.run.smithi088.stderr:2024-06-03T09:06:16.890+0000 7fb509173640 1 librados: starting objecter 2024-06-03T09:06:16.895 INFO:teuthology.orchestra.run.smithi088.stderr:2024-06-03T09:06:16.890+0000 7fb509173640 1 Processor -- start 2024-06-03T09:06:16.895 INFO:teuthology.orchestra.run.smithi088.stderr:2024-06-03T09:06:16.890+0000 7fb509173640 1 -- start start 2024-06-03T09:06:16.895 INFO:teuthology.orchestra.run.smithi088.stderr:2024-06-03T09:06:16.890+0000 7fb509173640 1 librados: setting wanted keys 2024-06-03T09:06:16.895 INFO:teuthology.orchestra.run.smithi088.stderr:2024-06-03T09:06:16.890+0000 7fb509173640 1 librados: calling monclient init 2024-06-03T09:06:16.895 INFO:teuthology.orchestra.run.smithi088.stderr:2024-06-03T09:06:16.890+0000 7fb509173640 10 monclient: init 2024-06-03T09:06:16.895 INFO:teuthology.orchestra.run.smithi088.stderr:2024-06-03T09:06:16.891+0000 7fb509173640 10 monclient: _reopen_session rank -1 2024-06-03T09:06:16.895 INFO:teuthology.orchestra.run.smithi088.stderr:2024-06-03T09:06:16.891+0000 7fb509173640 10 monclient: _add_conns ranks=[0,1,2] 2024-06-03T09:06:16.895 INFO:teuthology.orchestra.run.smithi088.stderr:2024-06-03T09:06:16.891+0000 7fb509173640 10 monclient(hunting): picked mon.noname-a con 0x7fb5040a8960 addr v1:172.21.15.88:6789/0 2024-06-03T09:06:16.895 INFO:teuthology.orchestra.run.smithi088.stderr:2024-06-03T09:06:16.891+0000 7fb509173640 10 monclient(hunting): picked mon.noname-b con 0x7fb50413eec0 addr v1:172.21.15.192:6789/0 2024-06-03T09:06:16.896 INFO:teuthology.orchestra.run.smithi088.stderr:2024-06-03T09:06:16.891+0000 7fb509173640 10 monclient(hunting): picked mon.noname-c con 0x7fb5041450f0 addr v1:172.21.15.88:6790/0 2024-06-03T09:06:16.896 INFO:teuthology.orchestra.run.smithi088.stderr:2024-06-03T09:06:16.891+0000 7fb509173640 1 -- --> v1:172.21.15.88:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fb5041d63e0 con 0x7fb5040a8960 2024-06-03T09:06:16.896 INFO:teuthology.orchestra.run.smithi088.stderr:2024-06-03T09:06:16.891+0000 7fb509173640 1 -- --> v1:172.21.15.192:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fb5040af020 con 0x7fb50413eec0 2024-06-03T09:06:16.896 INFO:teuthology.orchestra.run.smithi088.stderr:2024-06-03T09:06:16.891+0000 7fb509173640 1 -- --> v1:172.21.15.88:6790/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fb5040af720 con 0x7fb5041450f0 2024-06-03T09:06:16.896 INFO:teuthology.orchestra.run.smithi088.stderr:2024-06-03T09:06:16.891+0000 7fb509173640 10 monclient(hunting): _renew_subs 2024-06-03T09:06:16.896 INFO:teuthology.orchestra.run.smithi088.stderr:2024-06-03T09:06:16.891+0000 7fb509173640 10 monclient(hunting): authenticate will time out at 3032.701220s 2024-06-03T09:06:16.896 INFO:teuthology.orchestra.run.smithi088.stderr:2024-06-03T09:06:16.891+0000 7fb502d9d640 1 --1- >> v1:172.21.15.88:6789/0 conn(0x7fb5040a8960 0x7fb5041d6870 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=1).handle_server_banner_and_identify peer v1:172.21.15.88:6789/0 says I am v1:172.21.15.88:53974/0 (socket says 172.21.15.88:53974) 2024-06-03T09:06:16.896 INFO:teuthology.orchestra.run.smithi088.stderr:2024-06-03T09:06:16.891+0000 7fb502d9d640 1 -- 172.21.15.88:0/247611366 learned_addr learned my addr 172.21.15.88:0/247611366 (peer_addr_for_me v1:172.21.15.88:0/0) 2024-06-03T09:06:16.896 INFO:teuthology.orchestra.run.smithi088.stderr:2024-06-03T09:06:16.891+0000 7fb4fb7fe640 10 client.?.objecter ms_handle_connect 0x7fb5040a8960 2024-06-03T09:06:16.896 INFO:teuthology.orchestra.run.smithi088.stderr:2024-06-03T09:06:16.891+0000 7fb4fb7fe640 10 client.?.objecter resend_mon_ops 2024-06-03T09:06:16.896 INFO:teuthology.orchestra.run.smithi088.stderr:2024-06-03T09:06:16.892+0000 7fb4fb7fe640 10 client.?.objecter ms_handle_connect 0x7fb5041450f0 2024-06-03T09:06:16.896 INFO:teuthology.orchestra.run.smithi088.stderr:2024-06-03T09:06:16.892+0000 7fb4fb7fe640 10 client.?.objecter resend_mon_ops 2024-06-03T09:06:16.896 INFO:teuthology.orchestra.run.smithi088.stderr:2024-06-03T09:06:16.892+0000 7fb4fb7fe640 10 client.?.objecter ms_handle_connect 0x7fb50413eec0 2024-06-03T09:06:16.897 INFO:teuthology.orchestra.run.smithi088.stderr:2024-06-03T09:06:16.892+0000 7fb4fb7fe640 10 client.?.objecter resend_mon_ops 2024-06-03T09:06:16.897 INFO:teuthology.orchestra.run.smithi088.stderr:2024-06-03T09:06:16.894+0000 7fb4fb7fe640 1 -- 172.21.15.88:0/247611366 <== mon.0 v1:172.21.15.88:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 2559444485 0 0) 0x7fb5041d63e0 con 0x7fb5040a8960 2024-06-03T09:06:16.897 INFO:teuthology.orchestra.run.smithi088.stderr:2024-06-03T09:06:16.894+0000 7fb4fb7fe640 10 monclient(hunting): _negotiate 2024-06-03T09:06:16.897 INFO:teuthology.orchestra.run.smithi088.stderr:2024-06-03T09:06:16.894+0000 7fb4fb7fe640 10 monclient(hunting): _init_auth method 2 2024-06-03T09:06:16.897 INFO:teuthology.orchestra.run.smithi088.stderr:2024-06-03T09:06:16.894+0000 7fb4fb7fe640 10 monclient(hunting): _init_auth creating new auth 2024-06-03T09:06:16.897 INFO:teuthology.orchestra.run.smithi088.stderr:2024-06-03T09:06:16.894+0000 7fb4fb7fe640 10 monclient(hunting): my global_id is 30573 2024-06-03T09:06:16.897 INFO:teuthology.orchestra.run.smithi088.stderr:2024-06-03T09:06:16.894+0000 7fb4fb7fe640 1 -- 172.21.15.88:0/247611366 --> v1:172.21.15.88:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7fb4e4004eb0 con 0x7fb5040a8960 2024-06-03T09:06:16.897 INFO:teuthology.orchestra.run.smithi088.stderr:2024-06-03T09:06:16.894+0000 7fb4fb7fe640 1 -- 172.21.15.88:0/247611366 <== mon.2 v1:172.21.15.88:6790/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 3892130789 0 0) 0x7fb5040af720 con 0x7fb5041450f0 2024-06-03T09:06:16.897 INFO:teuthology.orchestra.run.smithi088.stderr:2024-06-03T09:06:16.894+0000 7fb4fb7fe640 10 monclient(hunting): _negotiate 2024-06-03T09:06:16.897 INFO:teuthology.orchestra.run.smithi088.stderr:2024-06-03T09:06:16.894+0000 7fb4fb7fe640 10 monclient(hunting): _init_auth method 2 2024-06-03T09:06:16.897 INFO:teuthology.orchestra.run.smithi088.stderr:2024-06-03T09:06:16.894+0000 7fb4fb7fe640 10 monclient(hunting): _init_auth creating new auth 2024-06-03T09:06:16.897 INFO:teuthology.orchestra.run.smithi088.stderr:2024-06-03T09:06:16.894+0000 7fb4fb7fe640 10 monclient(hunting): my global_id is 30077 2024-06-03T09:06:16.897 INFO:teuthology.orchestra.run.smithi088.stderr:2024-06-03T09:06:16.894+0000 7fb4fb7fe640 1 -- 172.21.15.88:0/247611366 --> v1:172.21.15.88:6790/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7fb5041d63e0 con 0x7fb5041450f0 2024-06-03T09:06:16.898 INFO:teuthology.orchestra.run.smithi088.stderr:2024-06-03T09:06:16.894+0000 7fb4fb7fe640 1 -- 172.21.15.88:0/247611366 <== mon.1 v1:172.21.15.192:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 2824577006 0 0) 0x7fb5040af020 con 0x7fb50413eec0 2024-06-03T09:06:16.898 INFO:teuthology.orchestra.run.smithi088.stderr:2024-06-03T09:06:16.894+0000 7fb4fb7fe640 10 monclient(hunting): _negotiate 2024-06-03T09:06:16.898 INFO:teuthology.orchestra.run.smithi088.stderr:2024-06-03T09:06:16.894+0000 7fb4fb7fe640 10 monclient(hunting): _init_auth method 2 2024-06-03T09:06:16.898 INFO:teuthology.orchestra.run.smithi088.stderr:2024-06-03T09:06:16.894+0000 7fb4fb7fe640 10 monclient(hunting): _init_auth creating new auth 2024-06-03T09:06:16.898 INFO:teuthology.orchestra.run.smithi088.stderr:2024-06-03T09:06:16.894+0000 7fb4fb7fe640 10 monclient(hunting): my global_id is 30154 2024-06-03T09:06:16.898 INFO:teuthology.orchestra.run.smithi088.stderr:2024-06-03T09:06:16.894+0000 7fb4fb7fe640 1 -- 172.21.15.88:0/247611366 --> v1:172.21.15.192:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7fb5040af720 con 0x7fb50413eec0 2024-06-03T09:06:16.899 INFO:teuthology.orchestra.run.smithi088.stderr:2024-06-03T09:06:16.898+0000 7fb4fb7fe640 1 -- 172.21.15.88:0/247611366 <== mon.0 v1:172.21.15.88:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 764+0+0 (unknown 3705259877 0 0) 0x7fb4e4004eb0 con 0x7fb5040a8960 2024-06-03T09:06:16.899 INFO:teuthology.orchestra.run.smithi088.stderr:2024-06-03T09:06:16.898+0000 7fb4fb7fe640 1 -- 172.21.15.88:0/247611366 --> v1:172.21.15.88:6789/0 -- auth(proto 2 165 bytes epoch 0) v1 -- 0x7fb5040af020 con 0x7fb5040a8960 2024-06-03T09:06:16.899 INFO:teuthology.orchestra.run.smithi088.stderr:2024-06-03T09:06:16.898+0000 7fb4fb7fe640 1 -- 172.21.15.88:0/247611366 <== mon.0 v1:172.21.15.88:6789/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 2234374065 0 0) 0x7fb4f4004000 con 0x7fb5040a8960 2024-06-03T09:06:16.899 INFO:teuthology.orchestra.run.smithi088.stderr:2024-06-03T09:06:16.898+0000 7fb4fb7fe640 10 monclient(hunting): handle_monmap mon_map magic: 0 v1 2024-06-03T09:06:16.899 INFO:teuthology.orchestra.run.smithi088.stderr:2024-06-03T09:06:16.898+0000 7fb4fb7fe640 10 monclient(hunting): got monmap 1 from mon.noname-a (according to old e1) 2024-06-03T09:06:16.900 INFO:teuthology.orchestra.run.smithi088.stderr:2024-06-03T09:06:16.898+0000 7fb4fb7fe640 10 monclient(hunting): dump: 2024-06-03T09:06:16.900 INFO:teuthology.orchestra.run.smithi088.stderr:epoch 1 2024-06-03T09:06:16.900 INFO:teuthology.orchestra.run.smithi088.stderr:fsid ec3c3fdd-3ac6-4d7d-9412-683130ed8b04 2024-06-03T09:06:16.900 INFO:teuthology.orchestra.run.smithi088.stderr:last_changed 2024-06-03T08:34:31.885125+0000 2024-06-03T09:06:16.900 INFO:teuthology.orchestra.run.smithi088.stderr:created 2024-06-03T08:34:31.885125+0000 2024-06-03T09:06:16.900 INFO:teuthology.orchestra.run.smithi088.stderr:min_mon_release 18 (reef) 2024-06-03T09:06:16.900 INFO:teuthology.orchestra.run.smithi088.stderr:election_strategy: 3 2024-06-03T09:06:16.900 INFO:teuthology.orchestra.run.smithi088.stderr:0: v1:172.21.15.88:6789/0 mon.a 2024-06-03T09:06:16.900 INFO:teuthology.orchestra.run.smithi088.stderr:1: v1:172.21.15.192:6789/0 mon.b 2024-06-03T09:06:16.900 INFO:teuthology.orchestra.run.smithi088.stderr:2: v1:172.21.15.88:6790/0 mon.c 2024-06-03T09:06:16.900 INFO:teuthology.orchestra.run.smithi088.stderr: 2024-06-03T09:06:16.900 INFO:teuthology.orchestra.run.smithi088.stderr:2024-06-03T09:06:16.898+0000 7fb4fb7fe640 10 monclient(hunting): _finish_auth 0 2024-06-03T09:06:16.900 INFO:teuthology.orchestra.run.smithi088.stderr:2024-06-03T09:06:16.899+0000 7fb4fb7fe640 1 -- 172.21.15.88:0/247611366 <== mon.2 v1:172.21.15.88:6790/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 764+0+0 (unknown 1162326600 0 0) 0x7fb5041d63e0 con 0x7fb5041450f0 2024-06-03T09:06:16.901 INFO:teuthology.orchestra.run.smithi088.stderr:2024-06-03T09:06:16.899+0000 7fb4fb7fe640 1 -- 172.21.15.88:0/247611366 --> v1:172.21.15.88:6790/0 -- auth(proto 2 165 bytes epoch 0) v1 -- 0x7fb4e4004eb0 con 0x7fb5041450f0 2024-06-03T09:06:16.901 INFO:teuthology.orchestra.run.smithi088.stderr:2024-06-03T09:06:16.899+0000 7fb4fb7fe640 1 -- 172.21.15.88:0/247611366 <== mon.2 v1:172.21.15.88:6790/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 2234374065 0 0) 0x7fb4fc001670 con 0x7fb5041450f0 2024-06-03T09:06:16.901 INFO:teuthology.orchestra.run.smithi088.stderr:2024-06-03T09:06:16.899+0000 7fb4fb7fe640 10 monclient(hunting): handle_monmap mon_map magic: 0 v1 2024-06-03T09:06:16.901 INFO:teuthology.orchestra.run.smithi088.stderr:2024-06-03T09:06:16.899+0000 7fb4fb7fe640 10 monclient(hunting): got monmap 1 from mon.c (according to old e1) 2024-06-03T09:06:16.901 INFO:teuthology.orchestra.run.smithi088.stderr:2024-06-03T09:06:16.899+0000 7fb4fb7fe640 10 monclient(hunting): dump: 2024-06-03T09:06:16.901 INFO:teuthology.orchestra.run.smithi088.stderr:epoch 1 2024-06-03T09:06:16.901 INFO:teuthology.orchestra.run.smithi088.stderr:fsid ec3c3fdd-3ac6-4d7d-9412-683130ed8b04 2024-06-03T09:06:16.901 INFO:teuthology.orchestra.run.smithi088.stderr:last_changed 2024-06-03T08:34:31.885125+0000 2024-06-03T09:06:16.901 INFO:teuthology.orchestra.run.smithi088.stderr:created 2024-06-03T08:34:31.885125+0000 2024-06-03T09:06:16.901 INFO:teuthology.orchestra.run.smithi088.stderr:min_mon_release 18 (reef) 2024-06-03T09:06:16.901 INFO:teuthology.orchestra.run.smithi088.stderr:election_strategy: 3 2024-06-03T09:06:16.901 INFO:teuthology.orchestra.run.smithi088.stderr:0: v1:172.21.15.88:6789/0 mon.a 2024-06-03T09:06:16.901 INFO:teuthology.orchestra.run.smithi088.stderr:1: v1:172.21.15.192:6789/0 mon.b 2024-06-03T09:06:16.901 INFO:teuthology.orchestra.run.smithi088.stderr:2: v1:172.21.15.88:6790/0 mon.c 2024-06-03T09:06:16.902 INFO:teuthology.orchestra.run.smithi088.stderr: 2024-06-03T09:06:16.902 INFO:teuthology.orchestra.run.smithi088.stderr:2024-06-03T09:06:16.899+0000 7fb4fb7fe640 1 -- 172.21.15.88:0/247611366 <== mon.1 v1:172.21.15.192:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 764+0+0 (unknown 1871333910 0 0) 0x7fb5040af720 con 0x7fb50413eec0 2024-06-03T09:06:16.902 INFO:teuthology.orchestra.run.smithi088.stderr:2024-06-03T09:06:16.899+0000 7fb4fb7fe640 1 -- 172.21.15.88:0/247611366 --> v1:172.21.15.192:6789/0 -- auth(proto 2 165 bytes epoch 0) v1 -- 0x7fb5041d63e0 con 0x7fb50413eec0 2024-06-03T09:06:16.902 INFO:teuthology.orchestra.run.smithi088.stderr:2024-06-03T09:06:16.899+0000 7fb4fb7fe640 1 -- 172.21.15.88:0/247611366 <== mon.1 v1:172.21.15.192:6789/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 2234374065 0 0) 0x7fb4ec001810 con 0x7fb50413eec0 2024-06-03T09:06:16.902 INFO:teuthology.orchestra.run.smithi088.stderr:2024-06-03T09:06:16.899+0000 7fb4fb7fe640 10 monclient(hunting): handle_monmap mon_map magic: 0 v1 2024-06-03T09:06:16.902 INFO:teuthology.orchestra.run.smithi088.stderr:2024-06-03T09:06:16.899+0000 7fb4fb7fe640 10 monclient(hunting): got monmap 1 from mon.b (according to old e1) 2024-06-03T09:06:16.902 INFO:teuthology.orchestra.run.smithi088.stderr:2024-06-03T09:06:16.899+0000 7fb4fb7fe640 10 monclient(hunting): dump: 2024-06-03T09:06:16.902 INFO:teuthology.orchestra.run.smithi088.stderr:epoch 1 2024-06-03T09:06:16.902 INFO:teuthology.orchestra.run.smithi088.stderr:fsid ec3c3fdd-3ac6-4d7d-9412-683130ed8b04 2024-06-03T09:06:16.902 INFO:teuthology.orchestra.run.smithi088.stderr:last_changed 2024-06-03T08:34:31.885125+0000 2024-06-03T09:06:16.902 INFO:teuthology.orchestra.run.smithi088.stderr:created 2024-06-03T08:34:31.885125+0000 2024-06-03T09:06:16.902 INFO:teuthology.orchestra.run.smithi088.stderr:min_mon_release 18 (reef) 2024-06-03T09:06:16.903 INFO:teuthology.orchestra.run.smithi088.stderr:election_strategy: 3 2024-06-03T09:06:16.903 INFO:teuthology.orchestra.run.smithi088.stderr:0: v1:172.21.15.88:6789/0 mon.a 2024-06-03T09:06:16.903 INFO:teuthology.orchestra.run.smithi088.stderr:1: v1:172.21.15.192:6789/0 mon.b 2024-06-03T09:06:16.903 INFO:teuthology.orchestra.run.smithi088.stderr:2: v1:172.21.15.88:6790/0 mon.c 2024-06-03T09:06:16.903 INFO:teuthology.orchestra.run.smithi088.stderr: 2024-06-03T09:06:16.904 INFO:teuthology.orchestra.run.smithi088.stderr:2024-06-03T09:06:16.902+0000 7fb4fb7fe640 1 -- 172.21.15.88:0/247611366 <== mon.1 v1:172.21.15.192:6789/0 4 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 580+0+0 (unknown 3012582993 0 0) 0x7fb5041d63e0 con 0x7fb50413eec0 2024-06-03T09:06:16.904 INFO:teuthology.orchestra.run.smithi088.stderr:2024-06-03T09:06:16.902+0000 7fb4fb7fe640 1 -- 172.21.15.88:0/247611366 >> v1:172.21.15.88:6790/0 conn(0x7fb5041450f0 legacy=0x7fb5041e0e40 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-06-03T09:06:16.904 INFO:teuthology.orchestra.run.smithi088.stderr:2024-06-03T09:06:16.902+0000 7fb4fb7fe640 1 -- 172.21.15.88:0/247611366 >> v1:172.21.15.88:6789/0 conn(0x7fb5040a8960 legacy=0x7fb5041d6870 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-06-03T09:06:16.904 INFO:teuthology.orchestra.run.smithi088.stderr:2024-06-03T09:06:16.902+0000 7fb4fb7fe640 10 monclient: _finish_hunting 0 2024-06-03T09:06:16.904 INFO:teuthology.orchestra.run.smithi088.stderr:2024-06-03T09:06:16.902+0000 7fb4fb7fe640 1 monclient: found mon.b 2024-06-03T09:06:16.904 INFO:teuthology.orchestra.run.smithi088.stderr:2024-06-03T09:06:16.902+0000 7fb4fb7fe640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-06-03T09:06:16.905 INFO:teuthology.orchestra.run.smithi088.stderr:2024-06-03T09:06:16.902+0000 7fb4fb7fe640 10 monclient: _send_mon_message to mon.b at v1:172.21.15.192:6789/0 2024-06-03T09:06:16.905 INFO:teuthology.orchestra.run.smithi088.stderr:2024-06-03T09:06:16.902+0000 7fb4fb7fe640 1 -- 172.21.15.88:0/247611366 --> v1:172.21.15.192:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fb5040af8b0 con 0x7fb50413eec0 2024-06-03T09:06:16.905 INFO:teuthology.orchestra.run.smithi088.stderr:2024-06-03T09:06:16.902+0000 7fb4fb7fe640 10 monclient: _finish_auth 0 2024-06-03T09:06:16.905 INFO:teuthology.orchestra.run.smithi088.stderr:2024-06-03T09:06:16.902+0000 7fb4fb7fe640 10 monclient: _check_auth_tickets 2024-06-03T09:06:16.905 INFO:teuthology.orchestra.run.smithi088.stderr:2024-06-03T09:06:16.902+0000 7fb4fb7fe640 20 monclient: _check_auth_rotating not needed by client.admin 2024-06-03T09:06:16.905 INFO:teuthology.orchestra.run.smithi088.stderr:2024-06-03T09:06:16.902+0000 7fb509173640 5 monclient: authenticate success, global_id 30154 2024-06-03T09:06:16.905 INFO:teuthology.orchestra.run.smithi088.stderr:2024-06-03T09:06:16.902+0000 7fb509173640 10 monclient: _renew_subs 2024-06-03T09:06:16.905 INFO:teuthology.orchestra.run.smithi088.stderr:2024-06-03T09:06:16.902+0000 7fb509173640 10 monclient: _send_mon_message to mon.b at v1:172.21.15.192:6789/0 2024-06-03T09:06:16.905 INFO:teuthology.orchestra.run.smithi088.stderr:2024-06-03T09:06:16.902+0000 7fb509173640 1 -- 172.21.15.88:0/247611366 --> v1:172.21.15.192:6789/0 -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fb5041d6f80 con 0x7fb50413eec0 2024-06-03T09:06:16.905 INFO:teuthology.orchestra.run.smithi088.stderr:2024-06-03T09:06:16.902+0000 7fb509173640 10 client.30154.objecter _maybe_request_map subscribing (onetime) to next osd map 2024-06-03T09:06:16.905 INFO:teuthology.orchestra.run.smithi088.stderr:2024-06-03T09:06:16.902+0000 7fb509173640 10 monclient: _renew_subs 2024-06-03T09:06:16.905 INFO:teuthology.orchestra.run.smithi088.stderr:2024-06-03T09:06:16.902+0000 7fb509173640 10 monclient: _send_mon_message to mon.b at v1:172.21.15.192:6789/0 2024-06-03T09:06:16.905 INFO:teuthology.orchestra.run.smithi088.stderr:2024-06-03T09:06:16.902+0000 7fb509173640 1 -- 172.21.15.88:0/247611366 --> v1:172.21.15.192:6789/0 -- mon_subscribe({osdmap=0}) v3 -- 0x7fb5040ade00 con 0x7fb50413eec0 2024-06-03T09:06:16.905 INFO:teuthology.orchestra.run.smithi088.stderr:2024-06-03T09:06:16.902+0000 7fb509173640 1 librados: init done 2024-06-03T09:06:16.906 INFO:teuthology.orchestra.run.smithi088.stderr:2024-06-03T09:06:16.903+0000 7fb509173640 10 start_mon_command cmd=[{"prefix": "get_command_descriptions"}] 2024-06-03T09:06:16.906 INFO:teuthology.orchestra.run.smithi088.stderr:2024-06-03T09:06:16.903+0000 7fb509173640 10 monclient: _send_command 1 [{"prefix": "get_command_descriptions"}] 2024-06-03T09:06:16.906 INFO:teuthology.orchestra.run.smithi088.stderr:2024-06-03T09:06:16.903+0000 7fb509173640 10 monclient: _send_mon_message to mon.b at v1:172.21.15.192:6789/0 2024-06-03T09:06:16.906 INFO:teuthology.orchestra.run.smithi088.stderr:2024-06-03T09:06:16.903+0000 7fb509173640 1 -- 172.21.15.88:0/247611366 --> v1:172.21.15.192:6789/0 -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fb5040a9e60 con 0x7fb50413eec0 2024-06-03T09:06:17.155 INFO:teuthology.orchestra.run.smithi088.stderr:2024-06-03T09:06:17.155+0000 7fb4fb7fe640 1 -- 172.21.15.88:0/247611366 <== mon.1 v1:172.21.15.192:6789/0 5 ==== config(0 keys) v1 ==== 4+0+0 (unknown 0 0 0) 0x7fb4ec001d60 con 0x7fb50413eec0 2024-06-03T09:06:17.155 INFO:teuthology.orchestra.run.smithi088.stderr:2024-06-03T09:06:17.155+0000 7fb4fb7fe640 10 monclient: handle_config config(0 keys) v1 2024-06-03T09:06:17.155 INFO:teuthology.orchestra.run.smithi088.stderr:2024-06-03T09:06:17.155+0000 7fb4fb7fe640 1 -- 172.21.15.88:0/247611366 <== mon.1 v1:172.21.15.192:6789/0 6 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 2234374065 0 0) 0x7fb4ec004a00 con 0x7fb50413eec0 2024-06-03T09:06:17.155 INFO:teuthology.orchestra.run.smithi088.stderr:2024-06-03T09:06:17.155+0000 7fb4fb7fe640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-06-03T09:06:17.155 INFO:teuthology.orchestra.run.smithi088.stderr:2024-06-03T09:06:17.155+0000 7fb4fb7fe640 10 monclient: got monmap 1 from mon.b (according to old e1) 2024-06-03T09:06:17.155 INFO:teuthology.orchestra.run.smithi088.stderr:2024-06-03T09:06:17.155+0000 7fb4fb7fe640 10 monclient: dump: 2024-06-03T09:06:17.156 INFO:teuthology.orchestra.run.smithi088.stderr:epoch 1 2024-06-03T09:06:17.156 INFO:teuthology.orchestra.run.smithi088.stderr:fsid ec3c3fdd-3ac6-4d7d-9412-683130ed8b04 2024-06-03T09:06:17.156 INFO:teuthology.orchestra.run.smithi088.stderr:last_changed 2024-06-03T08:34:31.885125+0000 2024-06-03T09:06:17.156 INFO:teuthology.orchestra.run.smithi088.stderr:created 2024-06-03T08:34:31.885125+0000 2024-06-03T09:06:17.156 INFO:teuthology.orchestra.run.smithi088.stderr:min_mon_release 18 (reef) 2024-06-03T09:06:17.156 INFO:teuthology.orchestra.run.smithi088.stderr:election_strategy: 3 2024-06-03T09:06:17.156 INFO:teuthology.orchestra.run.smithi088.stderr:0: v1:172.21.15.88:6789/0 mon.a 2024-06-03T09:06:17.156 INFO:teuthology.orchestra.run.smithi088.stderr:1: v1:172.21.15.192:6789/0 mon.b 2024-06-03T09:06:17.156 INFO:teuthology.orchestra.run.smithi088.stderr:2: v1:172.21.15.88:6790/0 mon.c 2024-06-03T09:06:17.156 INFO:teuthology.orchestra.run.smithi088.stderr: 2024-06-03T09:06:17.156 INFO:teuthology.orchestra.run.smithi088.stderr:2024-06-03T09:06:17.155+0000 7fb50159a640 4 set_mon_vals no callback set 2024-06-03T09:06:17.157 INFO:teuthology.orchestra.run.smithi088.stderr:2024-06-03T09:06:17.156+0000 7fb4fb7fe640 1 -- 172.21.15.88:0/247611366 <== mon.1 v1:172.21.15.192:6789/0 7 ==== mgrmap(e 5) v1 ==== 89110+0+0 (unknown 873533973 0 0) 0x7fb4ec01a780 con 0x7fb50413eec0 2024-06-03T09:06:17.157 INFO:teuthology.orchestra.run.smithi088.stderr:2024-06-03T09:06:17.156+0000 7fb502d9d640 10 monclient: get_auth_request con 0x7fb4e406d470 auth_method 0 2024-06-03T09:06:17.157 INFO:teuthology.orchestra.run.smithi088.stderr:2024-06-03T09:06:17.156+0000 7fb4fb7fe640 10 client.30154.objecter ms_handle_connect 0x7fb4e406d470 2024-06-03T09:06:17.158 INFO:teuthology.orchestra.run.smithi088.stderr:2024-06-03T09:06:17.158+0000 7fb4fb7fe640 1 -- 172.21.15.88:0/247611366 <== mon.1 v1:172.21.15.192:6789/0 8 ==== osd_map(1009..1009 src has 622..1009) v4 ==== 5229+0+0 (unknown 553880507 0 0) 0x7fb4ec054020 con 0x7fb50413eec0 2024-06-03T09:06:17.158 INFO:teuthology.orchestra.run.smithi088.stderr:2024-06-03T09:06:17.158+0000 7fb4fb7fe640 10 client.30154.objecter ms_dispatch 0x7fb504000b90 osd_map(1009..1009 src has 622..1009) v4 2024-06-03T09:06:17.158 INFO:teuthology.orchestra.run.smithi088.stderr:2024-06-03T09:06:17.158+0000 7fb4fb7fe640 3 client.30154.objecter handle_osd_map got epochs [1009,1009] > 0 2024-06-03T09:06:17.159 INFO:teuthology.orchestra.run.smithi088.stderr:2024-06-03T09:06:17.158+0000 7fb4fb7fe640 3 client.30154.objecter handle_osd_map decoding full epoch 1009 2024-06-03T09:06:17.159 INFO:teuthology.orchestra.run.smithi088.stderr:2024-06-03T09:06:17.158+0000 7fb4fb7fe640 20 client.30154.objecter dump_active .. 0 homeless 2024-06-03T09:06:17.160 INFO:teuthology.orchestra.run.smithi088.stderr:2024-06-03T09:06:17.161+0000 7fb4fb7fe640 1 -- 172.21.15.88:0/247611366 <== mon.1 v1:172.21.15.192:6789/0 9 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+182498 (unknown 1092875540 0 3016760700) 0x7fb4ec04fa00 con 0x7fb50413eec0 2024-06-03T09:06:17.161 INFO:teuthology.orchestra.run.smithi088.stderr:2024-06-03T09:06:17.161+0000 7fb4fb7fe640 10 monclient: handle_mon_command_ack 1 [{"prefix": "get_command_descriptions"}] 2024-06-03T09:06:17.161 INFO:teuthology.orchestra.run.smithi088.stderr:2024-06-03T09:06:17.161+0000 7fb4fb7fe640 10 monclient: _finish_command 1 = system:0 2024-06-03T09:06:17.365 INFO:teuthology.orchestra.run.smithi088.stderr:2024-06-03T09:06:17.365+0000 7fb509173640 1 -- 172.21.15.88:0/247611366 --> v1:172.21.15.192:6800/34229 -- mgr_command(tid 0: {"prefix": "pg dump", "target": ["mon-mgr", ""], "format": "json"}) v1 -- 0x7fb5041d6590 con 0x7fb4e406d470 2024-06-03T09:06:17.374 INFO:teuthology.orchestra.run.smithi088.stderr:2024-06-03T09:06:17.374+0000 7fb4fb7fe640 1 -- 172.21.15.88:0/247611366 <== mgr.4110 v1:172.21.15.192:6800/34229 1 ==== mgr_command_reply(tid 0: 0 dumped all) v1 ==== 18+0+151834 (unknown 2965378022 0 525065400) 0x7fb5041d6590 con 0x7fb4e406d470 2024-06-03T09:06:17.376 INFO:teuthology.orchestra.run.smithi088.stdout: 2024-06-03T09:06:17.376 INFO:teuthology.orchestra.run.smithi088.stderr:dumped all 2024-06-03T09:06:17.378 INFO:teuthology.orchestra.run.smithi088.stderr:2024-06-03T09:06:17.376+0000 7fb509173640 10 librados: watch_flush enter 2024-06-03T09:06:17.378 INFO:teuthology.orchestra.run.smithi088.stderr:2024-06-03T09:06:17.376+0000 7fb509173640 10 librados: watch_flush exit 2024-06-03T09:06:17.378 INFO:teuthology.orchestra.run.smithi088.stderr:2024-06-03T09:06:17.376+0000 7fb509173640 20 client.30154.objecter shutdown clearing up homeless session... 2024-06-03T09:06:17.378 INFO:teuthology.orchestra.run.smithi088.stderr:2024-06-03T09:06:17.376+0000 7fb509173640 10 client.30154.objecter successfully canceled tick 2024-06-03T09:06:17.378 INFO:teuthology.orchestra.run.smithi088.stderr:2024-06-03T09:06:17.377+0000 7fb509173640 1 -- 172.21.15.88:0/247611366 >> v1:172.21.15.192:6800/34229 conn(0x7fb4e406d470 legacy=0x7fb4e406f930 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-06-03T09:06:17.378 INFO:teuthology.orchestra.run.smithi088.stderr:2024-06-03T09:06:17.377+0000 7fb509173640 10 monclient: shutdown 2024-06-03T09:06:17.378 INFO:teuthology.orchestra.run.smithi088.stderr:2024-06-03T09:06:17.377+0000 7fb509173640 20 monclient: shutdown discarding 0 pending message(s) 2024-06-03T09:06:17.378 INFO:teuthology.orchestra.run.smithi088.stderr:2024-06-03T09:06:17.377+0000 7fb509173640 1 -- 172.21.15.88:0/247611366 >> v1:172.21.15.192:6789/0 conn(0x7fb50413eec0 legacy=0x7fb5040ad510 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-06-03T09:06:17.378 INFO:teuthology.orchestra.run.smithi088.stderr:2024-06-03T09:06:17.377+0000 7fb509173640 1 -- 172.21.15.88:0/247611366 shutdown_connections 2024-06-03T09:06:17.379 INFO:teuthology.orchestra.run.smithi088.stderr:2024-06-03T09:06:17.377+0000 7fb509173640 1 -- 172.21.15.88:0/247611366 >> 172.21.15.88:0/247611366 conn(0x7fb5040b70d0 msgr2=0x7fb5040b67c0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-06-03T09:06:17.379 INFO:teuthology.orchestra.run.smithi088.stderr:2024-06-03T09:06:17.377+0000 7fb509173640 1 -- 172.21.15.88:0/247611366 shutdown_connections 2024-06-03T09:06:17.379 INFO:teuthology.orchestra.run.smithi088.stderr:2024-06-03T09:06:17.377+0000 7fb509173640 1 -- 172.21.15.88:0/247611366 wait complete. 2024-06-03T09:06:17.379 INFO:teuthology.orchestra.run.smithi088.stderr:2024-06-03T09:06:17.377+0000 7fb509173640 1 librados: shutdown 2024-06-03T09:06:17.387 INFO:teuthology.orchestra.run.smithi088.stdout:{"pg_ready":true,"pg_map":{"version":1913,"stamp":"2024-06-03T09:06:17.097856+0000","last_osdmap_epoch":0,"last_pg_scan":0,"pg_stats_sum":{"stat_sum":{"num_bytes":459285,"num_objects":3,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":1,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":3,"num_whiteouts":0,"num_read":102,"num_read_kb":88,"num_write":120,"num_write_kb":1379,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2,"num_bytes_recovered":397840,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":83,"ondisk_log_size":83,"up":68,"acting":68,"num_store_stats":0},"osd_stats_sum":{"up_from":0,"seq":0,"num_pgs":107,"num_osds":8,"num_per_pool_osds":8,"num_per_pool_omap_osds":8,"kb":754974720,"kb_used":494428,"kb_used_data":46192,"kb_used_omap":12,"kb_used_meta":448115,"kb_avail":754480292,"statfs":{"total":773094113280,"available":772587819008,"internally_reserved":0,"allocated":47300608,"data_stored":22684387,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":12744,"internal_metadata":458870328},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":371,"apply_latency_ms":371,"commit_latency_ns":371000000,"apply_latency_ns":371000000},"alerts":[],"network_ping_times":[]},"pg_stats_delta":{"stat_sum":{"num_bytes":-4000299,"num_objects":-44,"num_object_clones":-1,"num_object_copies":-132,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":-32,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":-44,"num_whiteouts":0,"num_read":-31,"num_read_kb":-11,"num_write":-71,"num_write_kb":-3969,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":-1,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":0,"acting":0,"num_store_stats":0,"stamp_delta":"6.001219"},"pg_stats":[{"pgid":"17.d","version":"0'0","reported_seq":940,"reported_epoch":1007,"state":"active+clean","last_fresh":"2024-06-03T09:06:11.402997+0000","last_change":"2024-06-03T08:38:19.260548+0000","last_active":"2024-06-03T09:06:11.402997+0000","last_peered":"2024-06-03T09:06:11.402997+0000","last_clean":"2024-06-03T09:06:11.402997+0000","last_became_active":"2024-06-03T08:38:19.253559+0000","last_became_peered":"2024-06-03T08:38:19.253559+0000","last_unstale":"2024-06-03T09:06:11.402997+0000","last_undegraded":"2024-06-03T09:06:11.402997+0000","last_fullsized":"2024-06-03T09:06:11.402997+0000","mapping_epoch":28,"log_start":"0'0","ondisk_log_start":"0'0","created":28,"last_epoch_clean":45,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-06-03T08:37:55.614497+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-06-03T08:37:55.614497+0000","last_clean_scrub_stamp":"2024-06-03T08:37:55.614497+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-06-04T19:00:12.470979+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1],"acting":[0,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"17.c","version":"0'0","reported_seq":952,"reported_epoch":1008,"state":"active+undersized","last_fresh":"2024-06-03T09:06:12.368416+0000","last_change":"2024-06-03T09:06:03.440074+0000","last_active":"2024-06-03T09:06:12.368416+0000","last_peered":"2024-06-03T09:06:12.368416+0000","last_clean":"2024-06-03T09:05:02.105638+0000","last_became_active":"2024-06-03T09:06:03.440074+0000","last_became_peered":"2024-06-03T09:06:03.440074+0000","last_unstale":"2024-06-03T09:06:12.368416+0000","last_undegraded":"2024-06-03T09:06:12.368416+0000","last_fullsized":"2024-06-03T09:06:03.377982+0000","mapping_epoch":999,"log_start":"0'0","ondisk_log_start":"0'0","created":28,"last_epoch_clean":255,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-06-03T08:37:55.614497+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-06-03T08:37:55.614497+0000","last_clean_scrub_stamp":"2024-06-03T08:37:55.614497+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-06-04T10:34:20.808557+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3],"acting":[3],"avail_no_missing":["3"],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"17.f","version":"0'0","reported_seq":147,"reported_epoch":1008,"state":"active+undersized","last_fresh":"2024-06-03T09:06:12.378079+0000","last_change":"2024-06-03T09:06:03.420285+0000","last_active":"2024-06-03T09:06:12.378079+0000","last_peered":"2024-06-03T09:06:12.378079+0000","last_clean":"2024-06-03T08:37:55.614497+0000","last_became_active":"2024-06-03T09:06:03.420285+0000","last_became_peered":"2024-06-03T09:06:03.420285+0000","last_unstale":"2024-06-03T09:06:12.378079+0000","last_undegraded":"2024-06-03T09:06:12.378079+0000","last_fullsized":"2024-06-03T09:06:03.380616+0000","mapping_epoch":999,"log_start":"0'0","ondisk_log_start":"0'0","created":28,"last_epoch_clean":255,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-06-03T08:37:55.614497+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-06-03T08:37:55.614497+0000","last_clean_scrub_stamp":"2024-06-03T08:37:55.614497+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-06-04T15:39:41.187731+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4],"acting":[4],"avail_no_missing":["4"],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"17.e","version":"0'0","reported_seq":148,"reported_epoch":1008,"state":"active+undersized","last_fresh":"2024-06-03T09:06:12.895353+0000","last_change":"2024-06-03T09:06:03.412887+0000","last_active":"2024-06-03T09:06:12.895353+0000","last_peered":"2024-06-03T09:06:12.895353+0000","last_clean":"2024-06-03T08:37:55.614497+0000","last_became_active":"2024-06-03T09:06:03.412887+0000","last_became_peered":"2024-06-03T09:06:03.412887+0000","last_unstale":"2024-06-03T09:06:12.895353+0000","last_undegraded":"2024-06-03T09:06:12.895353+0000","last_fullsized":"2024-06-03T09:06:03.332482+0000","mapping_epoch":999,"log_start":"0'0","ondisk_log_start":"0'0","created":28,"last_epoch_clean":255,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-06-03T08:37:55.614497+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-06-03T08:37:55.614497+0000","last_clean_scrub_stamp":"2024-06-03T08:37:55.614497+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-06-04T12:23:22.511601+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5],"acting":[5],"avail_no_missing":["5"],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"17.9","version":"0'0","reported_seq":945,"reported_epoch":1009,"state":"active+clean","last_fresh":"2024-06-03T09:06:13.928215+0000","last_change":"2024-06-03T08:38:04.772260+0000","last_active":"2024-06-03T09:06:13.928215+0000","last_peered":"2024-06-03T09:06:13.928215+0000","last_clean":"2024-06-03T09:06:13.928215+0000","last_became_active":"2024-06-03T08:38:04.763428+0000","last_became_peered":"2024-06-03T08:38:04.763428+0000","last_unstale":"2024-06-03T09:06:13.928215+0000","last_undegraded":"2024-06-03T09:06:13.928215+0000","last_fullsized":"2024-06-03T09:06:13.928215+0000","mapping_epoch":28,"log_start":"0'0","ondisk_log_start":"0'0","created":28,"last_epoch_clean":33,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-06-03T08:37:55.614497+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-06-03T08:37:55.614497+0000","last_clean_scrub_stamp":"2024-06-03T08:37:55.614497+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-06-04T10:55:10.773936+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,4],"acting":[7,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"17.8","version":"0'0","reported_seq":959,"reported_epoch":1007,"state":"active+undersized","last_fresh":"2024-06-03T09:06:12.708781+0000","last_change":"2024-06-03T09:06:03.388786+0000","last_active":"2024-06-03T09:06:12.708781+0000","last_peered":"2024-06-03T09:06:12.708781+0000","last_clean":"2024-06-03T09:04:43.297780+0000","last_became_active":"2024-06-03T09:06:03.388786+0000","last_became_peered":"2024-06-03T09:06:03.388786+0000","last_unstale":"2024-06-03T09:06:12.708781+0000","last_undegraded":"2024-06-03T09:06:12.708781+0000","last_fullsized":"2024-06-03T09:06:03.331754+0000","mapping_epoch":999,"log_start":"0'0","ondisk_log_start":"0'0","created":28,"last_epoch_clean":255,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-06-03T08:37:55.614497+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-06-03T08:37:55.614497+0000","last_clean_scrub_stamp":"2024-06-03T08:37:55.614497+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-06-04T09:18:44.765547+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0],"acting":[0],"avail_no_missing":["0"],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"17.b","version":"0'0","reported_seq":148,"reported_epoch":1008,"state":"active+undersized","last_fresh":"2024-06-03T09:06:12.873670+0000","last_change":"2024-06-03T09:06:03.451203+0000","last_active":"2024-06-03T09:06:12.873670+0000","last_peered":"2024-06-03T09:06:12.873670+0000","last_clean":"2024-06-03T08:37:55.614497+0000","last_became_active":"2024-06-03T09:06:03.451203+0000","last_became_peered":"2024-06-03T09:06:03.451203+0000","last_unstale":"2024-06-03T09:06:12.873670+0000","last_undegraded":"2024-06-03T09:06:12.873670+0000","last_fullsized":"2024-06-03T09:06:03.375523+0000","mapping_epoch":999,"log_start":"0'0","ondisk_log_start":"0'0","created":28,"last_epoch_clean":255,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-06-03T08:37:55.614497+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-06-03T08:37:55.614497+0000","last_clean_scrub_stamp":"2024-06-03T08:37:55.614497+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-06-04T10:16:19.813495+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2],"acting":[2],"avail_no_missing":["2"],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"17.a","version":"0'0","reported_seq":923,"reported_epoch":1008,"state":"active+clean","last_fresh":"2024-06-03T09:06:12.871052+0000","last_change":"2024-06-03T08:38:05.843142+0000","last_active":"2024-06-03T09:06:12.871052+0000","last_peered":"2024-06-03T09:06:12.871052+0000","last_clean":"2024-06-03T09:06:12.871052+0000","last_became_active":"2024-06-03T08:38:05.809257+0000","last_became_peered":"2024-06-03T08:38:05.809257+0000","last_unstale":"2024-06-03T09:06:12.871052+0000","last_undegraded":"2024-06-03T09:06:12.871052+0000","last_fullsized":"2024-06-03T09:06:12.871052+0000","mapping_epoch":28,"log_start":"0'0","ondisk_log_start":"0'0","created":28,"last_epoch_clean":33,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-06-03T08:37:55.614497+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-06-03T08:37:55.614497+0000","last_clean_scrub_stamp":"2024-06-03T08:37:55.614497+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-06-04T12:12:03.686838+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2],"acting":[1,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"17.5","version":"0'0","reported_seq":941,"reported_epoch":1007,"state":"active+clean","last_fresh":"2024-06-03T09:06:11.400979+0000","last_change":"2024-06-03T08:38:23.466051+0000","last_active":"2024-06-03T09:06:11.400979+0000","last_peered":"2024-06-03T09:06:11.400979+0000","last_clean":"2024-06-03T09:06:11.400979+0000","last_became_active":"2024-06-03T08:38:23.458039+0000","last_became_peered":"2024-06-03T08:38:23.458039+0000","last_unstale":"2024-06-03T09:06:11.400979+0000","last_undegraded":"2024-06-03T09:06:11.400979+0000","last_fullsized":"2024-06-03T09:06:11.400979+0000","mapping_epoch":28,"log_start":"0'0","ondisk_log_start":"0'0","created":28,"last_epoch_clean":49,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-06-03T08:37:55.614497+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-06-03T08:37:55.614497+0000","last_clean_scrub_stamp":"2024-06-03T08:37:55.614497+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-06-04T12:27:02.532436+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,4],"acting":[0,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"17.4","version":"0'0","reported_seq":148,"reported_epoch":1008,"state":"active+undersized","last_fresh":"2024-06-03T09:06:12.883827+0000","last_change":"2024-06-03T09:06:03.441889+0000","last_active":"2024-06-03T09:06:12.883827+0000","last_peered":"2024-06-03T09:06:12.883827+0000","last_clean":"2024-06-03T08:37:55.614497+0000","last_became_active":"2024-06-03T09:06:03.441889+0000","last_became_peered":"2024-06-03T09:06:03.441889+0000","last_unstale":"2024-06-03T09:06:12.883827+0000","last_undegraded":"2024-06-03T09:06:12.883827+0000","last_fullsized":"2024-06-03T09:06:03.359582+0000","mapping_epoch":999,"log_start":"0'0","ondisk_log_start":"0'0","created":28,"last_epoch_clean":255,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-06-03T08:37:55.614497+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-06-03T08:37:55.614497+0000","last_clean_scrub_stamp":"2024-06-03T08:37:55.614497+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-06-04T12:49:22.907141+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7],"acting":[7],"avail_no_missing":["7"],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"17.7","version":"0'0","reported_seq":928,"reported_epoch":1008,"state":"active+clean","last_fresh":"2024-06-03T09:06:12.906543+0000","last_change":"2024-06-03T08:38:04.837083+0000","last_active":"2024-06-03T09:06:12.906543+0000","last_peered":"2024-06-03T09:06:12.906543+0000","last_clean":"2024-06-03T09:06:12.906543+0000","last_became_active":"2024-06-03T08:38:04.818246+0000","last_became_peered":"2024-06-03T08:38:04.818246+0000","last_unstale":"2024-06-03T09:06:12.906543+0000","last_undegraded":"2024-06-03T09:06:12.906543+0000","last_fullsized":"2024-06-03T09:06:12.906543+0000","mapping_epoch":28,"log_start":"0'0","ondisk_log_start":"0'0","created":28,"last_epoch_clean":34,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-06-03T08:37:55.614497+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-06-03T08:37:55.614497+0000","last_clean_scrub_stamp":"2024-06-03T08:37:55.614497+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-06-04T17:02:38.204813+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,2],"acting":[5,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"17.6","version":"0'0","reported_seq":958,"reported_epoch":1007,"state":"active+undersized","last_fresh":"2024-06-03T09:06:12.710313+0000","last_change":"2024-06-03T09:06:03.428677+0000","last_active":"2024-06-03T09:06:12.710313+0000","last_peered":"2024-06-03T09:06:12.710313+0000","last_clean":"2024-06-03T09:04:43.312352+0000","last_became_active":"2024-06-03T09:06:03.428677+0000","last_became_peered":"2024-06-03T09:06:03.428677+0000","last_unstale":"2024-06-03T09:06:12.710313+0000","last_undegraded":"2024-06-03T09:06:12.710313+0000","last_fullsized":"2024-06-03T09:06:03.359909+0000","mapping_epoch":999,"log_start":"0'0","ondisk_log_start":"0'0","created":28,"last_epoch_clean":255,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-06-03T08:37:55.614497+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-06-03T08:37:55.614497+0000","last_clean_scrub_stamp":"2024-06-03T08:37:55.614497+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-06-04T15:38:02.180241+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0],"acting":[0],"avail_no_missing":["0"],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"2.7","version":"0'0","reported_seq":939,"reported_epoch":1008,"state":"active+clean","last_fresh":"2024-06-03T09:06:12.364540+0000","last_change":"2024-06-03T08:36:30.385405+0000","last_active":"2024-06-03T09:06:12.364540+0000","last_peered":"2024-06-03T09:06:12.364540+0000","last_clean":"2024-06-03T09:06:12.364540+0000","last_became_active":"2024-06-03T08:36:30.380302+0000","last_became_peered":"2024-06-03T08:36:30.380302+0000","last_unstale":"2024-06-03T09:06:12.364540+0000","last_undegraded":"2024-06-03T09:06:12.364540+0000","last_fullsized":"2024-06-03T09:06:12.364540+0000","mapping_epoch":23,"log_start":"0'0","ondisk_log_start":"0'0","created":23,"last_epoch_clean":24,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-06-03T08:36:28.444764+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-06-03T08:36:28.444764+0000","last_clean_scrub_stamp":"2024-06-03T08:36:28.444764+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-06-04T18:46:33.475181+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0],"acting":[3,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"17.14","version":"0'0","reported_seq":925,"reported_epoch":1008,"state":"active+clean","last_fresh":"2024-06-03T09:06:12.884449+0000","last_change":"2024-06-03T08:38:04.547565+0000","last_active":"2024-06-03T09:06:12.884449+0000","last_peered":"2024-06-03T09:06:12.884449+0000","last_clean":"2024-06-03T09:06:12.884449+0000","last_became_active":"2024-06-03T08:38:04.541781+0000","last_became_peered":"2024-06-03T08:38:04.541781+0000","last_unstale":"2024-06-03T09:06:12.884449+0000","last_undegraded":"2024-06-03T09:06:12.884449+0000","last_fullsized":"2024-06-03T09:06:12.884449+0000","mapping_epoch":28,"log_start":"0'0","ondisk_log_start":"0'0","created":28,"last_epoch_clean":34,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-06-03T08:37:55.614497+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-06-03T08:37:55.614497+0000","last_clean_scrub_stamp":"2024-06-03T08:37:55.614497+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-06-04T16:41:55.170134+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,2],"acting":[5,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"2.6","version":"0'0","reported_seq":948,"reported_epoch":1008,"state":"active+clean","last_fresh":"2024-06-03T09:06:12.385918+0000","last_change":"2024-06-03T08:36:30.430760+0000","last_active":"2024-06-03T09:06:12.385918+0000","last_peered":"2024-06-03T09:06:12.385918+0000","last_clean":"2024-06-03T09:06:12.385918+0000","last_became_active":"2024-06-03T08:36:30.404665+0000","last_became_peered":"2024-06-03T08:36:30.404665+0000","last_unstale":"2024-06-03T09:06:12.385918+0000","last_undegraded":"2024-06-03T09:06:12.385918+0000","last_fullsized":"2024-06-03T09:06:12.385918+0000","mapping_epoch":23,"log_start":"0'0","ondisk_log_start":"0'0","created":23,"last_epoch_clean":24,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-06-03T08:36:28.444764+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-06-03T08:36:28.444764+0000","last_clean_scrub_stamp":"2024-06-03T08:36:28.444764+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-06-04T10:37:37.746074+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,0],"acting":[4,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"17.15","version":"0'0","reported_seq":933,"reported_epoch":1008,"state":"active+clean","last_fresh":"2024-06-03T09:06:12.371160+0000","last_change":"2024-06-03T08:38:20.999903+0000","last_active":"2024-06-03T09:06:12.371160+0000","last_peered":"2024-06-03T09:06:12.371160+0000","last_clean":"2024-06-03T09:06:12.371160+0000","last_became_active":"2024-06-03T08:38:20.979523+0000","last_became_peered":"2024-06-03T08:38:20.979523+0000","last_unstale":"2024-06-03T09:06:12.371160+0000","last_undegraded":"2024-06-03T09:06:12.371160+0000","last_fullsized":"2024-06-03T09:06:12.371160+0000","mapping_epoch":28,"log_start":"0'0","ondisk_log_start":"0'0","created":28,"last_epoch_clean":47,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-06-03T08:37:55.614497+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-06-03T08:37:55.614497+0000","last_clean_scrub_stamp":"2024-06-03T08:37:55.614497+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-06-04T20:29:58.314289+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,1],"acting":[3,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"2.5","version":"0'0","reported_seq":940,"reported_epoch":1008,"state":"active+clean","last_fresh":"2024-06-03T09:06:12.372623+0000","last_change":"2024-06-03T08:36:30.024235+0000","last_active":"2024-06-03T09:06:12.372623+0000","last_peered":"2024-06-03T09:06:12.372623+0000","last_clean":"2024-06-03T09:06:12.372623+0000","last_became_active":"2024-06-03T08:36:30.019040+0000","last_became_peered":"2024-06-03T08:36:30.019040+0000","last_unstale":"2024-06-03T09:06:12.372623+0000","last_undegraded":"2024-06-03T09:06:12.372623+0000","last_fullsized":"2024-06-03T09:06:12.372623+0000","mapping_epoch":23,"log_start":"0'0","ondisk_log_start":"0'0","created":23,"last_epoch_clean":24,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-06-03T08:36:28.444764+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-06-03T08:36:28.444764+0000","last_clean_scrub_stamp":"2024-06-03T08:36:28.444764+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-06-04T15:46:47.109502+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,4],"acting":[3,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"17.16","version":"0'0","reported_seq":971,"reported_epoch":1008,"state":"active+undersized","last_fresh":"2024-06-03T09:06:12.882676+0000","last_change":"2024-06-03T09:06:03.478615+0000","last_active":"2024-06-03T09:06:12.882676+0000","last_peered":"2024-06-03T09:06:12.882676+0000","last_clean":"2024-06-03T09:04:43.328415+0000","last_became_active":"2024-06-03T09:06:03.478615+0000","last_became_peered":"2024-06-03T09:06:03.478615+0000","last_unstale":"2024-06-03T09:06:12.882676+0000","last_undegraded":"2024-06-03T09:06:12.882676+0000","last_fullsized":"2024-06-03T09:06:03.402504+0000","mapping_epoch":999,"log_start":"0'0","ondisk_log_start":"0'0","created":28,"last_epoch_clean":255,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-06-03T08:37:55.614497+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-06-03T08:37:55.614497+0000","last_clean_scrub_stamp":"2024-06-03T08:37:55.614497+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-06-04T08:47:08.079233+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2],"acting":[2],"avail_no_missing":["2"],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"2.4","version":"0'0","reported_seq":950,"reported_epoch":1008,"state":"active+undersized","last_fresh":"2024-06-03T09:06:12.887507+0000","last_change":"2024-06-03T09:06:03.387171+0000","last_active":"2024-06-03T09:06:12.887507+0000","last_peered":"2024-06-03T09:06:12.887507+0000","last_clean":"2024-06-03T09:04:43.202645+0000","last_became_active":"2024-06-03T09:06:03.387171+0000","last_became_peered":"2024-06-03T09:06:03.387171+0000","last_unstale":"2024-06-03T09:06:12.887507+0000","last_undegraded":"2024-06-03T09:06:12.887507+0000","last_fullsized":"2024-06-03T09:06:03.283787+0000","mapping_epoch":999,"log_start":"0'0","ondisk_log_start":"0'0","created":23,"last_epoch_clean":255,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-06-03T08:36:28.444764+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-06-03T08:36:28.444764+0000","last_clean_scrub_stamp":"2024-06-03T08:36:28.444764+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-06-04T20:17:43.485023+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5],"acting":[5],"avail_no_missing":["5"],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"17.17","version":"0'0","reported_seq":953,"reported_epoch":1008,"state":"active+clean","last_fresh":"2024-06-03T09:06:12.889836+0000","last_change":"2024-06-03T08:38:38.483275+0000","last_active":"2024-06-03T09:06:12.889836+0000","last_peered":"2024-06-03T09:06:12.889836+0000","last_clean":"2024-06-03T09:06:12.889836+0000","last_became_active":"2024-06-03T08:38:38.467916+0000","last_became_peered":"2024-06-03T08:38:38.467916+0000","last_unstale":"2024-06-03T09:06:12.889836+0000","last_undegraded":"2024-06-03T09:06:12.889836+0000","last_fullsized":"2024-06-03T09:06:12.889836+0000","mapping_epoch":28,"log_start":"0'0","ondisk_log_start":"0'0","created":28,"last_epoch_clean":61,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-06-03T08:37:55.614497+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-06-03T08:37:55.614497+0000","last_clean_scrub_stamp":"2024-06-03T08:37:55.614497+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-06-04T10:02:23.036420+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,7],"acting":[2,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"2.2","version":"0'0","reported_seq":962,"reported_epoch":1008,"state":"active+undersized","last_fresh":"2024-06-03T09:06:12.352260+0000","last_change":"2024-06-03T09:06:03.410020+0000","last_active":"2024-06-03T09:06:12.352260+0000","last_peered":"2024-06-03T09:06:12.352260+0000","last_clean":"2024-06-03T09:05:37.145889+0000","last_became_active":"2024-06-03T09:06:03.410020+0000","last_became_peered":"2024-06-03T09:06:03.410020+0000","last_unstale":"2024-06-03T09:06:12.352260+0000","last_undegraded":"2024-06-03T09:06:12.352260+0000","last_fullsized":"2024-06-03T09:06:03.346864+0000","mapping_epoch":999,"log_start":"0'0","ondisk_log_start":"0'0","created":23,"last_epoch_clean":255,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-06-03T08:36:28.444764+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-06-03T08:36:28.444764+0000","last_clean_scrub_stamp":"2024-06-03T08:36:28.444764+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-06-04T09:57:12.423360+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3],"acting":[3],"avail_no_missing":["3"],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"17.11","version":"0'0","reported_seq":943,"reported_epoch":1008,"state":"active+clean","last_fresh":"2024-06-03T09:06:12.354057+0000","last_change":"2024-06-03T08:38:06.502302+0000","last_active":"2024-06-03T09:06:12.354057+0000","last_peered":"2024-06-03T09:06:12.354057+0000","last_clean":"2024-06-03T09:06:12.354057+0000","last_became_active":"2024-06-03T08:38:06.490204+0000","last_became_peered":"2024-06-03T08:38:06.490204+0000","last_unstale":"2024-06-03T09:06:12.354057+0000","last_undegraded":"2024-06-03T09:06:12.354057+0000","last_fullsized":"2024-06-03T09:06:12.354057+0000","mapping_epoch":28,"log_start":"0'0","ondisk_log_start":"0'0","created":28,"last_epoch_clean":32,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-06-03T08:37:55.614497+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-06-03T08:37:55.614497+0000","last_clean_scrub_stamp":"2024-06-03T08:37:55.614497+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-06-04T10:47:05.988958+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,2],"acting":[4,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"2.1","version":"0'0","reported_seq":951,"reported_epoch":1008,"state":"active+clean","last_fresh":"2024-06-03T09:06:12.872310+0000","last_change":"2024-06-03T08:36:30.883360+0000","last_active":"2024-06-03T09:06:12.872310+0000","last_peered":"2024-06-03T09:06:12.872310+0000","last_clean":"2024-06-03T09:06:12.872310+0000","last_became_active":"2024-06-03T08:36:30.855073+0000","last_became_peered":"2024-06-03T08:36:30.855073+0000","last_unstale":"2024-06-03T09:06:12.872310+0000","last_undegraded":"2024-06-03T09:06:12.872310+0000","last_fullsized":"2024-06-03T09:06:12.872310+0000","mapping_epoch":23,"log_start":"0'0","ondisk_log_start":"0'0","created":23,"last_epoch_clean":25,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-06-03T08:36:28.444764+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-06-03T08:36:28.444764+0000","last_clean_scrub_stamp":"2024-06-03T08:36:28.444764+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-06-04T18:12:22.363473+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,5],"acting":[7,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"17.12","version":"0'0","reported_seq":944,"reported_epoch":1008,"state":"active+clean","last_fresh":"2024-06-03T09:06:12.878527+0000","last_change":"2024-06-03T08:38:01.751562+0000","last_active":"2024-06-03T09:06:12.878527+0000","last_peered":"2024-06-03T09:06:12.878527+0000","last_clean":"2024-06-03T09:06:12.878527+0000","last_became_active":"2024-06-03T08:38:01.745027+0000","last_became_peered":"2024-06-03T08:38:01.745027+0000","last_unstale":"2024-06-03T09:06:12.878527+0000","last_undegraded":"2024-06-03T09:06:12.878527+0000","last_fullsized":"2024-06-03T09:06:12.878527+0000","mapping_epoch":28,"log_start":"0'0","ondisk_log_start":"0'0","created":28,"last_epoch_clean":32,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-06-03T08:37:55.614497+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-06-03T08:37:55.614497+0000","last_clean_scrub_stamp":"2024-06-03T08:37:55.614497+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-06-04T17:54:43.785326+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,3],"acting":[7,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"2.0","version":"0'0","reported_seq":942,"reported_epoch":1008,"state":"active+clean","last_fresh":"2024-06-03T09:06:12.341224+0000","last_change":"2024-06-03T08:36:29.984096+0000","last_active":"2024-06-03T09:06:12.341224+0000","last_peered":"2024-06-03T09:06:12.341224+0000","last_clean":"2024-06-03T09:06:12.341224+0000","last_became_active":"2024-06-03T08:36:29.957875+0000","last_became_peered":"2024-06-03T08:36:29.957875+0000","last_unstale":"2024-06-03T09:06:12.341224+0000","last_undegraded":"2024-06-03T09:06:12.341224+0000","last_fullsized":"2024-06-03T09:06:12.341224+0000","mapping_epoch":23,"log_start":"0'0","ondisk_log_start":"0'0","created":23,"last_epoch_clean":24,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-06-03T08:36:28.444764+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-06-03T08:36:28.444764+0000","last_clean_scrub_stamp":"2024-06-03T08:36:28.444764+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-06-04T08:41:33.679479+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,5],"acting":[3,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"17.13","version":"0'0","reported_seq":942,"reported_epoch":1008,"state":"active+clean","last_fresh":"2024-06-03T09:06:12.364319+0000","last_change":"2024-06-03T08:38:26.838316+0000","last_active":"2024-06-03T09:06:12.364319+0000","last_peered":"2024-06-03T09:06:12.364319+0000","last_clean":"2024-06-03T09:06:12.364319+0000","last_became_active":"2024-06-03T08:38:26.833214+0000","last_became_peered":"2024-06-03T08:38:26.833214+0000","last_unstale":"2024-06-03T09:06:12.364319+0000","last_undegraded":"2024-06-03T09:06:12.364319+0000","last_fullsized":"2024-06-03T09:06:12.364319+0000","mapping_epoch":28,"log_start":"0'0","ondisk_log_start":"0'0","created":28,"last_epoch_clean":52,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-06-03T08:37:55.614497+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-06-03T08:37:55.614497+0000","last_clean_scrub_stamp":"2024-06-03T08:37:55.614497+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-06-04T12:58:51.617998+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,0],"acting":[4,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"2.3","version":"0'0","reported_seq":941,"reported_epoch":1008,"state":"active+clean","last_fresh":"2024-06-03T09:06:12.356955+0000","last_change":"2024-06-03T08:36:30.351670+0000","last_active":"2024-06-03T09:06:12.356955+0000","last_peered":"2024-06-03T09:06:12.356955+0000","last_clean":"2024-06-03T09:06:12.356955+0000","last_became_active":"2024-06-03T08:36:30.343560+0000","last_became_peered":"2024-06-03T08:36:30.343560+0000","last_unstale":"2024-06-03T09:06:12.356955+0000","last_undegraded":"2024-06-03T09:06:12.356955+0000","last_fullsized":"2024-06-03T09:06:12.356955+0000","mapping_epoch":23,"log_start":"0'0","ondisk_log_start":"0'0","created":23,"last_epoch_clean":24,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-06-03T08:36:28.444764+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-06-03T08:36:28.444764+0000","last_clean_scrub_stamp":"2024-06-03T08:36:28.444764+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-06-04T12:02:24.765135+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,2],"acting":[3,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.0","version":"19'76","reported_seq":1050,"reported_epoch":1008,"state":"active+clean","last_fresh":"2024-06-03T09:06:12.865199+0000","last_change":"2024-06-03T08:36:26.702702+0000","last_active":"2024-06-03T09:06:12.865199+0000","last_peered":"2024-06-03T09:06:12.865199+0000","last_clean":"2024-06-03T09:06:12.865199+0000","last_became_active":"2024-06-03T08:36:26.128378+0000","last_became_peered":"2024-06-03T08:36:26.128378+0000","last_unstale":"2024-06-03T09:06:12.865199+0000","last_undegraded":"2024-06-03T09:06:12.865199+0000","last_fullsized":"2024-06-03T09:06:12.865199+0000","mapping_epoch":20,"log_start":"0'0","ondisk_log_start":"0'0","created":16,"last_epoch_clean":22,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-06-03T08:36:15.542635+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-06-03T08:36:15.542635+0000","last_clean_scrub_stamp":"2024-06-03T08:36:15.542635+0000","objects_scrubbed":0,"log_size":76,"log_dups_size":0,"ondisk_log_size":76,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-06-04T12:41:53.731360+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":459280,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":96,"num_read_kb":82,"num_write":113,"num_write_kb":1372,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2,"num_bytes_recovered":397840,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,5],"acting":[1,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"17.10","version":"0'0","reported_seq":943,"reported_epoch":1008,"state":"active+clean","last_fresh":"2024-06-03T09:06:12.369433+0000","last_change":"2024-06-03T08:38:15.399189+0000","last_active":"2024-06-03T09:06:12.369433+0000","last_peered":"2024-06-03T09:06:12.369433+0000","last_clean":"2024-06-03T09:06:12.369433+0000","last_became_active":"2024-06-03T08:38:15.337723+0000","last_became_peered":"2024-06-03T08:38:15.337723+0000","last_unstale":"2024-06-03T09:06:12.369433+0000","last_undegraded":"2024-06-03T09:06:12.369433+0000","last_fullsized":"2024-06-03T09:06:12.369433+0000","mapping_epoch":28,"log_start":"0'0","ondisk_log_start":"0'0","created":28,"last_epoch_clean":42,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-06-03T08:37:55.614497+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-06-03T08:37:55.614497+0000","last_clean_scrub_stamp":"2024-06-03T08:37:55.614497+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-06-04T09:29:46.427317+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,3],"acting":[4,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"17.1a","version":"0'0","reported_seq":959,"reported_epoch":1008,"state":"active+undersized","last_fresh":"2024-06-03T09:06:12.360277+0000","last_change":"2024-06-03T09:06:03.390734+0000","last_active":"2024-06-03T09:06:12.360277+0000","last_peered":"2024-06-03T09:06:12.360277+0000","last_clean":"2024-06-03T09:05:20.322313+0000","last_became_active":"2024-06-03T09:06:03.390734+0000","last_became_peered":"2024-06-03T09:06:03.390734+0000","last_unstale":"2024-06-03T09:06:12.360277+0000","last_undegraded":"2024-06-03T09:06:12.360277+0000","last_fullsized":"2024-06-03T09:06:03.343197+0000","mapping_epoch":999,"log_start":"0'0","ondisk_log_start":"0'0","created":28,"last_epoch_clean":254,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-06-03T08:37:55.614497+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-06-03T08:37:55.614497+0000","last_clean_scrub_stamp":"2024-06-03T08:37:55.614497+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-06-04T15:56:47.628875+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4],"acting":[4],"avail_no_missing":["4"],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"17.1b","version":"0'0","reported_seq":934,"reported_epoch":1008,"state":"active+clean","last_fresh":"2024-06-03T09:06:12.355403+0000","last_change":"2024-06-03T08:38:03.115748+0000","last_active":"2024-06-03T09:06:12.355403+0000","last_peered":"2024-06-03T09:06:12.355403+0000","last_clean":"2024-06-03T09:06:12.355403+0000","last_became_active":"2024-06-03T08:38:03.110533+0000","last_became_peered":"2024-06-03T08:38:03.110533+0000","last_unstale":"2024-06-03T09:06:12.355403+0000","last_undegraded":"2024-06-03T09:06:12.355403+0000","last_fullsized":"2024-06-03T09:06:12.355403+0000","mapping_epoch":28,"log_start":"0'0","ondisk_log_start":"0'0","created":28,"last_epoch_clean":32,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-06-03T08:37:55.614497+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-06-03T08:37:55.614497+0000","last_clean_scrub_stamp":"2024-06-03T08:37:55.614497+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-06-04T12:36:47.471609+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,7],"acting":[3,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"17.18","version":"0'0","reported_seq":943,"reported_epoch":1008,"state":"active+clean","last_fresh":"2024-06-03T09:06:12.868791+0000","last_change":"2024-06-03T08:38:04.632864+0000","last_active":"2024-06-03T09:06:12.868791+0000","last_peered":"2024-06-03T09:06:12.868791+0000","last_clean":"2024-06-03T09:06:12.868791+0000","last_became_active":"2024-06-03T08:38:04.627856+0000","last_became_peered":"2024-06-03T08:38:04.627856+0000","last_unstale":"2024-06-03T09:06:12.868791+0000","last_undegraded":"2024-06-03T09:06:12.868791+0000","last_fullsized":"2024-06-03T09:06:12.868791+0000","mapping_epoch":28,"log_start":"0'0","ondisk_log_start":"0'0","created":28,"last_epoch_clean":33,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-06-03T08:37:55.614497+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-06-03T08:37:55.614497+0000","last_clean_scrub_stamp":"2024-06-03T08:37:55.614497+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-06-04T19:55:51.181598+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,4],"acting":[7,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"17.19","version":"0'0","reported_seq":933,"reported_epoch":1008,"state":"active+clean","last_fresh":"2024-06-03T09:06:12.336384+0000","last_change":"2024-06-03T08:38:01.839491+0000","last_active":"2024-06-03T09:06:12.336384+0000","last_peered":"2024-06-03T09:06:12.336384+0000","last_clean":"2024-06-03T09:06:12.336384+0000","last_became_active":"2024-06-03T08:38:01.829539+0000","last_became_peered":"2024-06-03T08:38:01.829539+0000","last_unstale":"2024-06-03T09:06:12.336384+0000","last_undegraded":"2024-06-03T09:06:12.336384+0000","last_fullsized":"2024-06-03T09:06:12.336384+0000","mapping_epoch":28,"log_start":"0'0","ondisk_log_start":"0'0","created":28,"last_epoch_clean":30,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-06-03T08:37:55.614497+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-06-03T08:37:55.614497+0000","last_clean_scrub_stamp":"2024-06-03T08:37:55.614497+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-06-04T09:10:43.860229+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,2],"acting":[3,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"17.1e","version":"0'0","reported_seq":147,"reported_epoch":1008,"state":"active+undersized","last_fresh":"2024-06-03T09:06:12.382089+0000","last_change":"2024-06-03T09:06:03.406181+0000","last_active":"2024-06-03T09:06:12.382089+0000","last_peered":"2024-06-03T09:06:12.382089+0000","last_clean":"2024-06-03T08:37:55.614497+0000","last_became_active":"2024-06-03T09:06:03.406181+0000","last_became_peered":"2024-06-03T09:06:03.406181+0000","last_unstale":"2024-06-03T09:06:12.382089+0000","last_undegraded":"2024-06-03T09:06:12.382089+0000","last_fullsized":"2024-06-03T09:06:03.363739+0000","mapping_epoch":999,"log_start":"0'0","ondisk_log_start":"0'0","created":28,"last_epoch_clean":255,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-06-03T08:37:55.614497+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-06-03T08:37:55.614497+0000","last_clean_scrub_stamp":"2024-06-03T08:37:55.614497+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-06-04T11:45:24.486600+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4],"acting":[4],"avail_no_missing":["4"],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"17.1f","version":"0'0","reported_seq":939,"reported_epoch":1007,"state":"active+clean","last_fresh":"2024-06-03T09:06:11.314307+0000","last_change":"2024-06-03T08:38:04.383538+0000","last_active":"2024-06-03T09:06:11.314307+0000","last_peered":"2024-06-03T09:06:11.314307+0000","last_clean":"2024-06-03T09:06:11.314307+0000","last_became_active":"2024-06-03T08:38:04.367278+0000","last_became_peered":"2024-06-03T08:38:04.367278+0000","last_unstale":"2024-06-03T09:06:11.314307+0000","last_undegraded":"2024-06-03T09:06:11.314307+0000","last_fullsized":"2024-06-03T09:06:11.314307+0000","mapping_epoch":28,"log_start":"0'0","ondisk_log_start":"0'0","created":28,"last_epoch_clean":32,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-06-03T08:37:55.614497+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-06-03T08:37:55.614497+0000","last_clean_scrub_stamp":"2024-06-03T08:37:55.614497+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-06-04T10:39:26.552282+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,2],"acting":[0,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"17.1c","version":"0'0","reported_seq":942,"reported_epoch":1008,"state":"active+undersized","last_fresh":"2024-06-03T09:06:12.880704+0000","last_change":"2024-06-03T09:06:03.382210+0000","last_active":"2024-06-03T09:06:12.880704+0000","last_peered":"2024-06-03T09:06:12.880704+0000","last_clean":"2024-06-03T09:04:43.199660+0000","last_became_active":"2024-06-03T09:06:03.382210+0000","last_became_peered":"2024-06-03T09:06:03.382210+0000","last_unstale":"2024-06-03T09:06:12.880704+0000","last_undegraded":"2024-06-03T09:06:12.880704+0000","last_fullsized":"2024-06-03T09:06:03.277874+0000","mapping_epoch":999,"log_start":"0'0","ondisk_log_start":"0'0","created":28,"last_epoch_clean":255,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-06-03T08:37:55.614497+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-06-03T08:37:55.614497+0000","last_clean_scrub_stamp":"2024-06-03T08:37:55.614497+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-06-04T12:18:50.013884+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5],"acting":[5],"avail_no_missing":["5"],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"17.1d","version":"0'0","reported_seq":922,"reported_epoch":1008,"state":"active+clean","last_fresh":"2024-06-03T09:06:12.881860+0000","last_change":"2024-06-03T08:38:06.794630+0000","last_active":"2024-06-03T09:06:12.881860+0000","last_peered":"2024-06-03T09:06:12.881860+0000","last_clean":"2024-06-03T09:06:12.881860+0000","last_became_active":"2024-06-03T08:38:06.592317+0000","last_became_peered":"2024-06-03T08:38:06.592317+0000","last_unstale":"2024-06-03T09:06:12.881860+0000","last_undegraded":"2024-06-03T09:06:12.881860+0000","last_fullsized":"2024-06-03T09:06:12.881860+0000","mapping_epoch":28,"log_start":"0'0","ondisk_log_start":"0'0","created":28,"last_epoch_clean":33,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-06-03T08:37:55.614497+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-06-03T08:37:55.614497+0000","last_clean_scrub_stamp":"2024-06-03T08:37:55.614497+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-06-04T12:33:30.785369+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,7],"acting":[1,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"17.2","version":"0'0","reported_seq":924,"reported_epoch":1008,"state":"active+clean","last_fresh":"2024-06-03T09:06:12.869501+0000","last_change":"2024-06-03T08:38:14.981410+0000","last_active":"2024-06-03T09:06:12.869501+0000","last_peered":"2024-06-03T09:06:12.869501+0000","last_clean":"2024-06-03T09:06:12.869501+0000","last_became_active":"2024-06-03T08:38:14.973825+0000","last_became_peered":"2024-06-03T08:38:14.973825+0000","last_unstale":"2024-06-03T09:06:12.869501+0000","last_undegraded":"2024-06-03T09:06:12.869501+0000","last_fullsized":"2024-06-03T09:06:12.869501+0000","mapping_epoch":28,"log_start":"0'0","ondisk_log_start":"0'0","created":28,"last_epoch_clean":41,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-06-03T08:37:55.614497+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-06-03T08:37:55.614497+0000","last_clean_scrub_stamp":"2024-06-03T08:37:55.614497+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-06-04T17:01:05.518613+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,4],"acting":[1,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"17.3","version":"0'0","reported_seq":941,"reported_epoch":1007,"state":"active+clean","last_fresh":"2024-06-03T09:06:11.338843+0000","last_change":"2024-06-03T08:38:14.619745+0000","last_active":"2024-06-03T09:06:11.338843+0000","last_peered":"2024-06-03T09:06:11.338843+0000","last_clean":"2024-06-03T09:06:11.338843+0000","last_became_active":"2024-06-03T08:38:14.510665+0000","last_became_peered":"2024-06-03T08:38:14.510665+0000","last_unstale":"2024-06-03T09:06:11.338843+0000","last_undegraded":"2024-06-03T09:06:11.338843+0000","last_fullsized":"2024-06-03T09:06:11.338843+0000","mapping_epoch":28,"log_start":"0'0","ondisk_log_start":"0'0","created":28,"last_epoch_clean":42,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-06-03T08:37:55.614497+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-06-03T08:37:55.614497+0000","last_clean_scrub_stamp":"2024-06-03T08:37:55.614497+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-06-04T12:03:47.392064+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,7],"acting":[0,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"17.0","version":"0'0","reported_seq":942,"reported_epoch":1007,"state":"active+clean","last_fresh":"2024-06-03T09:06:12.708393+0000","last_change":"2024-06-03T08:38:04.237118+0000","last_active":"2024-06-03T09:06:12.708393+0000","last_peered":"2024-06-03T09:06:12.708393+0000","last_clean":"2024-06-03T09:06:12.708393+0000","last_became_active":"2024-06-03T08:38:04.231954+0000","last_became_peered":"2024-06-03T08:38:04.231954+0000","last_unstale":"2024-06-03T09:06:12.708393+0000","last_undegraded":"2024-06-03T09:06:12.708393+0000","last_fullsized":"2024-06-03T09:06:12.708393+0000","mapping_epoch":28,"log_start":"0'0","ondisk_log_start":"0'0","created":28,"last_epoch_clean":32,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-06-03T08:37:55.614497+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-06-03T08:37:55.614497+0000","last_clean_scrub_stamp":"2024-06-03T08:37:55.614497+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-06-04T09:56:10.741637+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,2],"acting":[0,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"17.1","version":"65'7","reported_seq":958,"reported_epoch":1008,"state":"active+undersized+degraded","last_fresh":"2024-06-03T09:06:12.872100+0000","last_change":"2024-06-03T09:06:03.370660+0000","last_active":"2024-06-03T09:06:12.872100+0000","last_peered":"2024-06-03T09:06:12.872100+0000","last_clean":"2024-06-03T09:05:03.733864+0000","last_became_active":"2024-06-03T09:06:03.370660+0000","last_became_peered":"2024-06-03T09:06:03.370660+0000","last_unstale":"2024-06-03T09:06:12.872100+0000","last_undegraded":"2024-06-03T09:06:03.266076+0000","last_fullsized":"2024-06-03T09:06:03.264525+0000","mapping_epoch":999,"log_start":"0'0","ondisk_log_start":"0'0","created":28,"last_epoch_clean":255,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-06-03T08:37:55.614497+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-06-03T08:37:55.614497+0000","last_clean_scrub_stamp":"2024-06-03T08:37:55.614497+0000","objects_scrubbed":0,"log_size":7,"log_dups_size":0,"ondisk_log_size":7,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-06-04T11:55:18.194283+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":5,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":1,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":6,"num_read_kb":6,"num_write":7,"num_write_kb":7,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5],"acting":[5],"avail_no_missing":["5"],"object_location_counts":[{"shards":"5","objects":1}],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]}],"pool_stats":[{"poolid":2,"num_pg":8,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":14,"acting":14,"num_store_stats":7},{"poolid":1,"num_pg":1,"stat_sum":{"num_bytes":459280,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":96,"num_read_kb":82,"num_write":113,"num_write_kb":1372,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2,"num_bytes_recovered":397840,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":802816,"data_stored":795680,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":76,"ondisk_log_size":76,"up":2,"acting":2,"num_store_stats":2},{"poolid":17,"num_pg":32,"stat_sum":{"num_bytes":5,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":1,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":6,"num_read_kb":6,"num_write":7,"num_write_kb":7,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":8192,"data_stored":10,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":7,"ondisk_log_size":7,"up":52,"acting":52,"num_store_stats":8}],"osd_stats":[{"osd":6,"up_from":253,"seq":1086626726200,"num_pgs":32,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":54772,"kb_used_data":5608,"kb_used_omap":1,"kb_used_meta":49150,"kb_avail":94317068,"statfs":{"total":96636764160,"available":96580677632,"internally_reserved":0,"allocated":5742592,"data_stored":2767180,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1593,"internal_metadata":50330055},"hb_peers":[0,1,2,3,4,5,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":109,"apply_latency_ms":109,"commit_latency_ns":109000000,"apply_latency_ns":109000000},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Mon Jun 3 09:04:02 2024","interfaces":[{"interface":"back","average":{"1min":1582.4780000000001,"5min":332.72699999999998,"15min":124.113},"min":{"1min":4.4829999999999997,"5min":4.1959999999999997,"15min":4.1959999999999997},"max":{"1min":22766.882000000001,"5min":22766.882000000001,"15min":22766.882000000001},"last":21.917999999999999},{"interface":"front","average":{"1min":1579.5409999999999,"5min":334.94499999999999,"15min":125.84999999999999},"min":{"1min":4.8129999999999997,"5min":4.7539999999999996,"15min":4.4530000000000003},"max":{"1min":22766.289000000001,"5min":22766.289000000001,"15min":22766.289000000001},"last":25.408999999999999}]},{"osd":1,"last update":"Mon Jun 3 09:03:40 2024","interfaces":[{"interface":"back","average":{"1min":36.420999999999999,"5min":77.054000000000002,"15min":132.84800000000001},"min":{"1min":6.0380000000000003,"5min":5.8799999999999999,"15min":5.484},"max":{"1min":589.64099999999996,"5min":4138.1679999999997,"15min":9955.1299999999992},"last":26.84},{"interface":"front","average":{"1min":40.240000000000002,"5min":79.674999999999997,"15min":133.38999999999999},"min":{"1min":8.4589999999999996,"5min":8.4090000000000007,"15min":6.1390000000000002},"max":{"1min":588.60799999999995,"5min":4137.6530000000002,"15min":9827.5509999999995},"last":26.263000000000002}]},{"osd":2,"last update":"Mon Jun 3 09:03:40 2024","interfaces":[{"interface":"back","average":{"1min":12.613,"5min":24.600999999999999,"15min":21.088000000000001},"min":{"1min":6.9740000000000002,"5min":6.9740000000000002,"15min":6.5330000000000004},"max":{"1min":26.076000000000001,"5min":376.24900000000002,"15min":376.24900000000002},"last":24.899000000000001},{"interface":"front","average":{"1min":12.493,"5min":22.207999999999998,"15min":21.219000000000001},"min":{"1min":6.5810000000000004,"5min":6.5110000000000001,"15min":6.0750000000000002},"max":{"1min":29.690999999999999,"5min":271.51100000000002,"15min":434.77300000000002},"last":40.372}]},{"osd":3,"last update":"Mon Jun 3 09:03:40 2024","interfaces":[{"interface":"back","average":{"1min":14.093999999999999,"5min":24.143999999999998,"15min":29.376999999999999},"min":{"1min":10.196999999999999,"5min":6.2919999999999998,"15min":6.1719999999999997},"max":{"1min":24.832000000000001,"5min":197.84899999999999,"15min":1235.5799999999999},"last":24.353999999999999},{"interface":"front","average":{"1min":9.3260000000000005,"5min":18.463999999999999,"15min":27.631},"min":{"1min":6.125,"5min":6.125,"15min":6.125},"max":{"1min":28.166,"5min":200.173,"15min":1157.6780000000001},"last":23.454000000000001}]},{"osd":4,"last update":"Mon Jun 3 09:04:29 2024","interfaces":[{"interface":"back","average":{"1min":10.279,"5min":1253.809,"15min":450.79199999999997},"min":{"1min":7.2300000000000004,"5min":6.093,"15min":6.093},"max":{"1min":15.000999999999999,"5min":28093.096000000001,"15min":28093.096000000001},"last":101.45399999999999},{"interface":"front","average":{"1min":15.693,"5min":1279.759,"15min":463.87099999999998},"min":{"1min":10.551,"5min":7.7480000000000002,"15min":7.7480000000000002},"max":{"1min":27.898,"5min":28323.837,"15min":28323.837},"last":102.262}]},{"osd":5,"last update":"Mon Jun 3 09:04:10 2024","interfaces":[{"interface":"back","average":{"1min":13.728999999999999,"5min":24.670000000000002,"15min":23.815999999999999},"min":{"1min":10.401,"5min":7.516,"15min":7.516},"max":{"1min":25.138000000000002,"5min":320.85000000000002,"15min":445.041},"last":103.52500000000001},{"interface":"front","average":{"1min":13.847,"5min":29.283000000000001,"15min":23.393999999999998},"min":{"1min":7.944,"5min":6.266,"15min":6.266},"max":{"1min":18.817,"5min":530.56700000000001,"15min":530.56700000000001},"last":29.263999999999999}]},{"osd":7,"last update":"Mon Jun 3 09:03:40 2024","interfaces":[{"interface":"back","average":{"1min":20.568999999999999,"5min":30.286999999999999,"15min":27.977},"min":{"1min":13.874000000000001,"5min":10.58,"15min":9.2859999999999996},"max":{"1min":86.959000000000003,"5min":222.07599999999999,"15min":255.08799999999999},"last":37.618000000000002},{"interface":"front","average":{"1min":19.102,"5min":29.550000000000001,"15min":27.763000000000002},"min":{"1min":10.052,"5min":8.6709999999999994,"15min":8.5429999999999993},"max":{"1min":67.968999999999994,"5min":293.62099999999998,"15min":438.14600000000002},"last":111.985}]}]},{"osd":7,"up_from":20,"seq":85899346284,"num_pgs":9,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":58952,"kb_used_data":5692,"kb_used_omap":1,"kb_used_meta":53246,"kb_avail":94312888,"statfs":{"total":96636764160,"available":96576397312,"internally_reserved":0,"allocated":5828608,"data_stored":2735298,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1589,"internal_metadata":54524363},"hb_peers":[0,1,2,3,4,5],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":55,"apply_latency_ms":55,"commit_latency_ns":55000000,"apply_latency_ns":55000000},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Mon Jun 3 09:06:09 2024","interfaces":[{"interface":"back","average":{"1min":14.073,"5min":894.22400000000005,"15min":308.93400000000003},"min":{"1min":5.5709999999999997,"5min":4.218,"15min":4.0599999999999996},"max":{"1min":90.757999999999996,"5min":21729.001,"15min":21729.001},"last":11.128},{"interface":"front","average":{"1min":9.2550000000000008,"5min":906.23500000000001,"15min":314.06},"min":{"1min":5.5789999999999997,"5min":4.8810000000000002,"15min":4.8810000000000002},"max":{"1min":25.937000000000001,"5min":21899.732,"15min":21899.732},"last":16.053999999999998}]},{"osd":1,"last update":"Mon Jun 3 09:06:09 2024","interfaces":[{"interface":"back","average":{"1min":15.182,"5min":69.183999999999997,"15min":219.173},"min":{"1min":7.3639999999999999,"5min":6.6200000000000001,"15min":6.1319999999999997},"max":{"1min":76.926000000000002,"5min":2795.1329999999998,"15min":18373.257000000001},"last":8.6799999999999997},{"interface":"front","average":{"1min":21.495999999999999,"5min":74.703999999999994,"15min":225.285},"min":{"1min":8.8399999999999999,"5min":8.8399999999999999,"15min":8.5129999999999999},"max":{"1min":94.977999999999994,"5min":2803.3670000000002,"15min":18374.601999999999},"last":13.189}]},{"osd":2,"last update":"Mon Jun 3 09:06:09 2024","interfaces":[{"interface":"back","average":{"1min":19.164000000000001,"5min":26.902000000000001,"15min":22.920999999999999},"min":{"1min":11.207000000000001,"5min":8.5030000000000001,"15min":8.5030000000000001},"max":{"1min":91.319999999999993,"5min":333.99200000000002,"15min":333.99200000000002},"last":18.968},{"interface":"front","average":{"1min":11.125999999999999,"5min":18.132000000000001,"15min":16.353000000000002},"min":{"1min":6.202,"5min":5.8369999999999997,"15min":5.5069999999999997},"max":{"1min":28.649000000000001,"5min":226.18100000000001,"15min":226.18100000000001},"last":13.875}]},{"osd":3,"last update":"Mon Jun 3 09:06:09 2024","interfaces":[{"interface":"back","average":{"1min":22.291,"5min":26.289999999999999,"15min":31.074999999999999},"min":{"1min":11.714,"5min":6.8959999999999999,"15min":6.3760000000000003},"max":{"1min":83.102000000000004,"5min":316.64400000000001,"15min":2039.9059999999999},"last":15.42},{"interface":"front","average":{"1min":13.324999999999999,"5min":17.963000000000001,"15min":25.969999999999999},"min":{"1min":6.1200000000000001,"5min":5.2089999999999996,"15min":5.2089999999999996},"max":{"1min":49.908999999999999,"5min":233.542,"15min":2037.5550000000001},"last":7.6879999999999997}]},{"osd":4,"last update":"Mon Jun 3 09:05:28 2024","interfaces":[{"interface":"back","average":{"1min":22.209,"5min":1379.51,"15min":1803.278},"min":{"1min":5.9089999999999998,"5min":5.9089999999999998,"15min":5.9089999999999998},"max":{"1min":224.54300000000001,"5min":30073.917000000001,"15min":62560.868000000002},"last":10.279},{"interface":"front","average":{"1min":22.712,"5min":1401.0719999999999,"15min":1806.759},"min":{"1min":7.6449999999999996,"5min":6.7930000000000001,"15min":5.9340000000000002},"max":{"1min":223.58600000000001,"5min":30348.021000000001,"15min":62451.446000000004},"last":16.579000000000001}]},{"osd":5,"last update":"Mon Jun 3 09:06:02 2024","interfaces":[{"interface":"back","average":{"1min":24.372,"5min":26.856000000000002,"15min":23.571999999999999},"min":{"1min":12.278,"5min":9.2460000000000004,"15min":6.5650000000000004},"max":{"1min":183.64099999999999,"5min":317.572,"15min":317.572},"last":10.933999999999999},{"interface":"front","average":{"1min":21.259,"5min":23.364000000000001,"15min":21.193999999999999},"min":{"1min":7.8499999999999996,"5min":7.0999999999999996,"15min":7.008},"max":{"1min":184.53399999999999,"5min":306.60599999999999,"15min":306.60599999999999},"last":18.036000000000001}]},{"osd":6,"last update":"Mon Jun 3 09:04:31 2024","interfaces":[{"interface":"back","average":{"1min":44.061,"5min":26.821000000000002,"15min":91.176000000000002},"min":{"1min":9.0800000000000001,"5min":7.9390000000000001,"15min":7.9390000000000001},"max":{"1min":268.029,"5min":308.916,"15min":8734.4639999999999},"last":6.6769999999999996},{"interface":"front","average":{"1min":46.643000000000001,"5min":23.879999999999999,"15min":88.765000000000001},"min":{"1min":8.8119999999999994,"5min":6.9240000000000004,"15min":6.742},"max":{"1min":305.64600000000002,"5min":305.64600000000002,"15min":8735.5319999999992},"last":12.327}]}]},{"osd":5,"up_from":20,"seq":85899346258,"num_pgs":9,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":59340,"kb_used_data":6076,"kb_used_omap":1,"kb_used_meta":53246,"kb_avail":94312500,"statfs":{"total":96636764160,"available":96576000000,"internally_reserved":0,"allocated":6221824,"data_stored":3126752,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1594,"internal_metadata":54524358},"hb_peers":[0,1,2,3,4,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":51,"apply_latency_ms":51,"commit_latency_ns":51000000,"apply_latency_ns":51000000},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Mon Jun 3 09:06:06 2024","interfaces":[{"interface":"back","average":{"1min":7.468,"5min":1703.037,"15min":580.51900000000001},"min":{"1min":4.6509999999999998,"5min":4.6509999999999998,"15min":4.4020000000000001},"max":{"1min":14.711,"5min":26433.764999999999,"15min":26433.764999999999},"last":7.9260000000000002},{"interface":"front","average":{"1min":9.9429999999999996,"5min":1725.27,"15min":587.87800000000004},"min":{"1min":5.5999999999999996,"5min":5.1319999999999997,"15min":4.8170000000000002},"max":{"1min":13.590999999999999,"5min":26704.698,"15min":26704.698},"last":10.76}]},{"osd":1,"last update":"Mon Jun 3 09:06:00 2024","interfaces":[{"interface":"back","average":{"1min":10.862,"5min":67.731999999999999,"15min":217.364},"min":{"1min":7.1769999999999996,"5min":6.0890000000000004,"15min":5.4930000000000003},"max":{"1min":16.998999999999999,"5min":3923.4090000000001,"15min":18811.696},"last":11.622999999999999},{"interface":"front","average":{"1min":10.417,"5min":65.141999999999996,"15min":220.75700000000001},"min":{"1min":5.3209999999999997,"5min":5.3209999999999997,"15min":5.0209999999999999},"max":{"1min":15.821999999999999,"5min":4019.6129999999998,"15min":18968.365000000002},"last":12.172000000000001}]},{"osd":2,"last update":"Mon Jun 3 09:06:00 2024","interfaces":[{"interface":"back","average":{"1min":11.366,"5min":24.18,"15min":21.137},"min":{"1min":5.6369999999999996,"5min":5.6369999999999996,"15min":5.0510000000000002},"max":{"1min":15.529999999999999,"5min":332.48200000000003,"15min":332.48200000000003},"last":13.997},{"interface":"front","average":{"1min":10.089,"5min":23.117000000000001,"15min":24.050000000000001},"min":{"1min":6.6769999999999996,"5min":6.6769999999999996,"15min":5.8739999999999997},"max":{"1min":15.821,"5min":353.03199999999998,"15min":353.03199999999998},"last":9.0280000000000005}]},{"osd":3,"last update":"Mon Jun 3 09:06:00 2024","interfaces":[{"interface":"back","average":{"1min":13.472,"5min":24.283999999999999,"15min":31.666},"min":{"1min":10.111000000000001,"5min":7.4720000000000004,"15min":7.4210000000000003},"max":{"1min":16.271999999999998,"5min":353.57499999999999,"15min":2855.8519999999999},"last":19.434999999999999},{"interface":"front","average":{"1min":9.5899999999999999,"5min":15.875,"15min":25.834},"min":{"1min":6.6609999999999996,"5min":5.7729999999999997,"15min":5.1909999999999998},"max":{"1min":15.112,"5min":203.804,"15min":2869.7860000000001},"last":18.408000000000001}]},{"osd":4,"last update":"Mon Jun 3 09:05:27 2024","interfaces":[{"interface":"back","average":{"1min":19.155000000000001,"5min":1084.961,"15min":1346.9949999999999},"min":{"1min":7.6130000000000004,"5min":7.6130000000000004,"15min":6.1340000000000003},"max":{"1min":156.55199999999999,"5min":28619.825000000001,"15min":57999.453999999998},"last":13.683},{"interface":"front","average":{"1min":28.068999999999999,"5min":1087.106,"15min":1330.289},"min":{"1min":8.3949999999999996,"5min":6.5030000000000001,"15min":6.5030000000000001},"max":{"1min":254.91300000000001,"5min":28621.294000000002,"15min":57434.631999999998},"last":14.867000000000001}]},{"osd":6,"last update":"Mon Jun 3 09:03:57 2024","interfaces":[{"interface":"back","average":{"1min":23.536000000000001,"5min":36.457000000000001,"15min":83.799999999999997},"min":{"1min":9.4800000000000004,"5min":9.4800000000000004,"15min":8.0630000000000006},"max":{"1min":89.442999999999998,"5min":508.77999999999997,"15min":9193.5169999999998},"last":157.79300000000001},{"interface":"front","average":{"1min":18.891999999999999,"5min":22.100999999999999,"15min":75.489999999999995},"min":{"1min":5.923,"5min":5.923,"15min":5.923},"max":{"1min":78.424999999999997,"5min":464.62,"15min":9191.4429999999993},"last":148.09}]},{"osd":7,"last update":"Mon Jun 3 09:06:00 2024","interfaces":[{"interface":"back","average":{"1min":13.257,"5min":20.917999999999999,"15min":21.088999999999999},"min":{"1min":10.475,"5min":6.7629999999999999,"15min":6.7629999999999999},"max":{"1min":17.245000000000001,"5min":211.22999999999999,"15min":211.22999999999999},"last":15.555},{"interface":"front","average":{"1min":12.872999999999999,"5min":20.587,"15min":24.021000000000001},"min":{"1min":8.5760000000000005,"5min":7.7709999999999999,"15min":6.8570000000000002},"max":{"1min":18.359999999999999,"5min":223.191,"15min":286.69499999999999},"last":9.7189999999999994}]}]},{"osd":4,"up_from":20,"seq":85899346266,"num_pgs":12,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":63052,"kb_used_data":5692,"kb_used_omap":1,"kb_used_meta":57342,"kb_avail":94308788,"statfs":{"total":96636764160,"available":96572198912,"internally_reserved":0,"allocated":5828608,"data_stored":2735298,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1594,"internal_metadata":58718662},"hb_peers":[0,1,2,3,5,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":51,"apply_latency_ms":51,"commit_latency_ns":51000000,"apply_latency_ns":51000000},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Mon Jun 3 09:05:34 2024","interfaces":[{"interface":"back","average":{"1min":12.416,"5min":1099.4649999999999,"15min":382.15199999999999},"min":{"1min":4.5279999999999996,"5min":4.5279999999999996,"15min":4.3220000000000001},"max":{"1min":123.878,"5min":21726.743999999999,"15min":21726.743999999999},"last":5.6609999999999996},{"interface":"front","average":{"1min":16.462,"5min":1108.894,"15min":387.70299999999997},"min":{"1min":6.4089999999999998,"5min":5.8380000000000001,"15min":5.4909999999999997},"max":{"1min":130.185,"5min":21727.993999999999,"15min":21727.993999999999},"last":6.1950000000000003}]},{"osd":1,"last update":"Mon Jun 3 09:05:34 2024","interfaces":[{"interface":"back","average":{"1min":14.727,"5min":56.755000000000003,"15min":341.09399999999999},"min":{"1min":4.6529999999999996,"5min":4.1970000000000001,"15min":4.1970000000000001},"max":{"1min":126.899,"5min":3346.4389999999999,"15min":22256.098000000002},"last":5.0839999999999996},{"interface":"front","average":{"1min":13.472,"5min":57.503,"15min":340.303},"min":{"1min":5.0590000000000002,"5min":5.0590000000000002,"15min":4.2640000000000002},"max":{"1min":126.096,"5min":3347.181,"15min":22247.317999999999},"last":10.532999999999999}]},{"osd":2,"last update":"Mon Jun 3 09:05:34 2024","interfaces":[{"interface":"back","average":{"1min":26.088999999999999,"5min":23.866,"15min":20.620999999999999},"min":{"1min":7.0869999999999997,"5min":6.8239999999999998,"15min":6.3150000000000004},"max":{"1min":337.60399999999998,"5min":393.58600000000001,"15min":393.58600000000001},"last":9.5120000000000005},{"interface":"front","average":{"1min":15.800000000000001,"5min":22.584,"15min":23.245999999999999},"min":{"1min":6.0629999999999997,"5min":6.0629999999999997,"15min":6.0629999999999997},"max":{"1min":127.913,"5min":394.471,"15min":394.471},"last":12.246}]},{"osd":3,"last update":"Mon Jun 3 09:05:34 2024","interfaces":[{"interface":"back","average":{"1min":17.544,"5min":19.143999999999998,"15min":18.742000000000001},"min":{"1min":5.8719999999999999,"5min":5.4100000000000001,"15min":5.4000000000000004},"max":{"1min":127.43600000000001,"5min":322.63,"15min":322.63},"last":13.051},{"interface":"front","average":{"1min":21.327000000000002,"5min":23.559000000000001,"15min":23.797000000000001},"min":{"1min":10.023,"5min":7.484,"15min":7.484},"max":{"1min":204.517,"5min":392.67500000000001,"15min":392.67500000000001},"last":11.057}]},{"osd":5,"last update":"Mon Jun 3 09:05:34 2024","interfaces":[{"interface":"back","average":{"1min":24.274999999999999,"5min":21.274999999999999,"15min":20.988},"min":{"1min":6.7489999999999997,"5min":6.7489999999999997,"15min":6.7489999999999997},"max":{"1min":268.10199999999998,"5min":344.23399999999998,"15min":344.23399999999998},"last":13.826000000000001},{"interface":"front","average":{"1min":26.361999999999998,"5min":25.218,"15min":26.084},"min":{"1min":12.228,"5min":8.5180000000000007,"15min":8.5180000000000007},"max":{"1min":264.75,"5min":345.18700000000001,"15min":345.18700000000001},"last":14.75}]},{"osd":6,"last update":"Mon Jun 3 09:04:32 2024","interfaces":[{"interface":"back","average":{"1min":15.548,"5min":32.957999999999998,"15min":152.738},"min":{"1min":9.141,"5min":9.141,"15min":8.2319999999999993},"max":{"1min":52.57,"5min":395.238,"15min":12496.335999999999},"last":208.90000000000001},{"interface":"front","average":{"1min":12.680999999999999,"5min":25.428000000000001,"15min":149.30600000000001},"min":{"1min":6.4379999999999997,"5min":5.6319999999999997,"15min":5.6319999999999997},"max":{"1min":46.283000000000001,"5min":377.74299999999999,"15min":12616.297},"last":205.88900000000001}]},{"osd":7,"last update":"Mon Jun 3 09:05:34 2024","interfaces":[{"interface":"back","average":{"1min":18.295999999999999,"5min":19.585999999999999,"15min":21.062000000000001},"min":{"1min":6.2110000000000003,"5min":6.2110000000000003,"15min":5.8010000000000002},"max":{"1min":207.25299999999999,"5min":358.94900000000001,"15min":358.94900000000001},"last":7.0279999999999996},{"interface":"front","average":{"1min":23.712,"5min":24.024999999999999,"15min":24.640000000000001},"min":{"1min":11.824999999999999,"5min":7.7789999999999999,"15min":7.7789999999999999},"max":{"1min":209.32900000000001,"5min":346.25200000000001,"15min":380.06900000000002},"last":15.519}]}]},{"osd":2,"up_from":20,"seq":85899346286,"num_pgs":13,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":58952,"kb_used_data":5692,"kb_used_omap":1,"kb_used_meta":53246,"kb_avail":94312888,"statfs":{"total":96636764160,"available":96576397312,"internally_reserved":0,"allocated":5828608,"data_stored":2735298,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1594,"internal_metadata":54524358},"hb_peers":[0,1,3,4,5,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":16,"apply_latency_ms":16,"commit_latency_ns":16000000,"apply_latency_ns":16000000},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Mon Jun 3 09:05:21 2024","interfaces":[{"interface":"back","average":{"1min":16.364000000000001,"5min":947.59799999999996,"15min":326.58999999999997},"min":{"1min":6.0830000000000002,"5min":4.7050000000000001,"15min":4.1420000000000003},"max":{"1min":157.60900000000001,"5min":23465.18,"15min":23465.18},"last":26.015000000000001},{"interface":"front","average":{"1min":12.276,"5min":945.29899999999998,"15min":326.78399999999999},"min":{"1min":5.1989999999999998,"5min":4.7439999999999998,"15min":4.6950000000000003},"max":{"1min":102.557,"5min":23465.966,"15min":23465.966},"last":20.802}]},{"osd":1,"last update":"Mon Jun 3 09:05:21 2024","interfaces":[{"interface":"back","average":{"1min":17.803999999999998,"5min":77.658000000000001,"15min":294.96899999999999},"min":{"1min":7.2130000000000001,"5min":5.4199999999999999,"15min":5.4199999999999999},"max":{"1min":159.624,"5min":4220.5290000000005,"15min":18954.987000000001},"last":85.430000000000007},{"interface":"front","average":{"1min":21.542999999999999,"5min":83.528999999999996,"15min":303.11700000000002},"min":{"1min":8.8200000000000003,"5min":7.5949999999999998,"15min":7.5949999999999998},"max":{"1min":170.74299999999999,"5min":4325.9489999999996,"15min":19143.834999999999},"last":98.498000000000005}]},{"osd":3,"last update":"Mon Jun 3 09:05:21 2024","interfaces":[{"interface":"back","average":{"1min":19.164999999999999,"5min":13.284000000000001,"15min":28.751999999999999},"min":{"1min":7.3710000000000004,"5min":5.9290000000000003,"15min":5.2450000000000001},"max":{"1min":118.913,"5min":118.913,"15min":2594.8150000000001},"last":28.164000000000001},{"interface":"front","average":{"1min":22.094000000000001,"5min":17.244,"15min":32.173999999999999},"min":{"1min":9.516,"5min":8.0760000000000005,"15min":6.0780000000000003},"max":{"1min":156.34700000000001,"5min":156.34700000000001,"15min":2706.5120000000002},"last":32.360999999999997}]},{"osd":4,"last update":"Mon Jun 3 09:05:57 2024","interfaces":[{"interface":"back","average":{"1min":10.573,"5min":1266.894,"15min":438.154},"min":{"1min":8.6379999999999999,"5min":7.8789999999999996,"15min":6.2039999999999997},"max":{"1min":15.105,"5min":26660.861000000001,"15min":26660.861000000001},"last":30.163},{"interface":"front","average":{"1min":12.407999999999999,"5min":1266.7460000000001,"15min":435.75400000000002},"min":{"1min":10.266999999999999,"5min":10.266999999999999,"15min":6.2720000000000002},"max":{"1min":15.228,"5min":26616.243999999999,"15min":26616.243999999999},"last":33.402999999999999}]},{"osd":5,"last update":"Mon Jun 3 09:05:21 2024","interfaces":[{"interface":"back","average":{"1min":25.632000000000001,"5min":20.437999999999999,"15min":21.472999999999999},"min":{"1min":9.2159999999999993,"5min":9.2159999999999993,"15min":6.2050000000000001},"max":{"1min":252.31100000000001,"5min":252.31100000000001,"15min":252.31100000000001},"last":32.890000000000001},{"interface":"front","average":{"1min":24.824000000000002,"5min":18.521000000000001,"15min":21.027000000000001},"min":{"1min":8.0449999999999999,"5min":7.4850000000000003,"15min":6.8810000000000002},"max":{"1min":258.31700000000001,"5min":258.31700000000001,"15min":258.31700000000001},"last":29.228999999999999}]},{"osd":6,"last update":"Mon Jun 3 09:04:28 2024","interfaces":[{"interface":"back","average":{"1min":14.063000000000001,"5min":24.850000000000001,"15min":106.755},"min":{"1min":9.0820000000000007,"5min":8.9649999999999999,"15min":8.9649999999999999},"max":{"1min":24.666,"5min":188.06700000000001,"15min":9521.0010000000002},"last":183.60900000000001},{"interface":"front","average":{"1min":13.933999999999999,"5min":21.081,"15min":106.142},"min":{"1min":5.8719999999999999,"5min":5.8719999999999999,"15min":5.8719999999999999},"max":{"1min":117.551,"5min":193.887,"15min":9585.6620000000003},"last":184.20599999999999}]},{"osd":7,"last update":"Mon Jun 3 09:05:21 2024","interfaces":[{"interface":"back","average":{"1min":25.097000000000001,"5min":22.579999999999998,"15min":24.388000000000002},"min":{"1min":7.8959999999999999,"5min":6.6130000000000004,"15min":6.6130000000000004},"max":{"1min":242.56800000000001,"5min":242.56800000000001,"15min":242.56800000000001},"last":88.974999999999994},{"interface":"front","average":{"1min":24.414999999999999,"5min":23.995999999999999,"15min":26.614999999999998},"min":{"1min":12.43,"5min":9.0660000000000007,"15min":9.0660000000000007},"max":{"1min":181.56399999999999,"5min":182.5,"15min":214.33799999999999},"last":102.229}]}]},{"osd":3,"up_from":14,"seq":60129542500,"num_pgs":11,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":58940,"kb_used_data":5680,"kb_used_omap":1,"kb_used_meta":53246,"kb_avail":94312900,"statfs":{"total":96636764160,"available":96576409600,"internally_reserved":0,"allocated":5816320,"data_stored":2728907,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1592,"internal_metadata":54524360},"hb_peers":[0,1,2,4,5,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":20,"apply_latency_ms":20,"commit_latency_ns":20000000,"apply_latency_ns":20000000},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Mon Jun 3 09:05:25 2024","interfaces":[{"interface":"back","average":{"1min":8.4809999999999999,"5min":1064.2270000000001,"15min":367.88600000000002},"min":{"1min":6.0529999999999999,"5min":5.173,"15min":4.6929999999999996},"max":{"1min":12.010999999999999,"5min":23275.167000000001,"15min":23275.167000000001},"last":8.0489999999999995},{"interface":"front","average":{"1min":13.885,"5min":1070.3710000000001,"15min":373.53500000000003},"min":{"1min":9.6679999999999993,"5min":7.2089999999999996,"15min":5.9880000000000004},"max":{"1min":32.426000000000002,"5min":23278.536,"15min":23278.536},"last":10.086}]},{"osd":1,"last update":"Mon Jun 3 09:05:25 2024","interfaces":[{"interface":"back","average":{"1min":11.324,"5min":19.920000000000002,"15min":390.07900000000001},"min":{"1min":6.6379999999999999,"5min":4.8200000000000003,"15min":4.8200000000000003},"max":{"1min":26.449000000000002,"5min":250.72300000000001,"15min":21658.205999999998},"last":4.335},{"interface":"front","average":{"1min":9.468,"5min":20.966999999999999,"15min":381.18799999999999},"min":{"1min":5.4969999999999999,"5min":5.2779999999999996,"15min":5.2779999999999996},"max":{"1min":25.643000000000001,"5min":335.33699999999999,"15min":21307.069},"last":7.3360000000000003}]},{"osd":2,"last update":"Mon Jun 3 09:05:25 2024","interfaces":[{"interface":"back","average":{"1min":13.215,"5min":26.649000000000001,"15min":23.783999999999999},"min":{"1min":7.8710000000000004,"5min":6.4900000000000002,"15min":5.875},"max":{"1min":34.322000000000003,"5min":377.803,"15min":377.803},"last":11.356999999999999},{"interface":"front","average":{"1min":13.105,"5min":22.292999999999999,"15min":20.573},"min":{"1min":7.327,"5min":6.125,"15min":5.9020000000000001},"max":{"1min":32.756999999999998,"5min":371.601,"15min":371.601},"last":8.9670000000000005}]},{"osd":4,"last update":"Mon Jun 3 09:05:27 2024","interfaces":[{"interface":"back","average":{"1min":10.444000000000001,"5min":1211.4970000000001,"15min":1639.1579999999999},"min":{"1min":6.4429999999999996,"5min":6.2560000000000002,"15min":6.0309999999999997},"max":{"1min":21.263000000000002,"5min":30144.939999999999,"15min":62679.622000000003},"last":14.975},{"interface":"front","average":{"1min":11.363,"5min":1219.6400000000001,"15min":1644.127},"min":{"1min":7.8109999999999999,"5min":7.8109999999999999,"15min":6.2439999999999998},"max":{"1min":20.108000000000001,"5min":30180.689999999999,"15min":62681.021999999997},"last":14.266}]},{"osd":5,"last update":"Mon Jun 3 09:06:07 2024","interfaces":[{"interface":"back","average":{"1min":12.802,"5min":19.640999999999998,"15min":22.318000000000001},"min":{"1min":9.0890000000000004,"5min":6.8810000000000002,"15min":6.7690000000000001},"max":{"1min":16.745999999999999,"5min":249.88,"15min":327.34899999999999},"last":11.952},{"interface":"front","average":{"1min":15.002000000000001,"5min":24.003,"15min":26.649999999999999},"min":{"1min":11.984,"5min":9.9149999999999991,"15min":8.9879999999999995},"max":{"1min":18.407,"5min":251.47800000000001,"15min":331.86799999999999},"last":13.717000000000001}]},{"osd":6,"last update":"Mon Jun 3 09:04:04 2024","interfaces":[{"interface":"back","average":{"1min":22.815999999999999,"5min":31.513999999999999,"15min":181.74299999999999},"min":{"1min":9.4809999999999999,"5min":7.2240000000000002,"15min":7.2240000000000002},"max":{"1min":160.56999999999999,"5min":380.613,"15min":11893.74},"last":19.359999999999999},{"interface":"front","average":{"1min":25.359999999999999,"5min":30.981000000000002,"15min":179.81399999999999},"min":{"1min":9.8870000000000005,"5min":7.3360000000000003,"15min":6.0899999999999999},"max":{"1min":193.11099999999999,"5min":379.80700000000002,"15min":11893.069},"last":8.202}]},{"osd":7,"last update":"Mon Jun 3 09:05:25 2024","interfaces":[{"interface":"back","average":{"1min":12.177,"5min":21.126000000000001,"15min":22.972000000000001},"min":{"1min":8.6769999999999996,"5min":7.4279999999999999,"15min":7.4279999999999999},"max":{"1min":17.376999999999999,"5min":164.27500000000001,"15min":178.76300000000001},"last":9.5540000000000003},{"interface":"front","average":{"1min":15.675000000000001,"5min":22.722000000000001,"15min":25.238},"min":{"1min":11.233000000000001,"5min":10.555999999999999,"15min":7.6440000000000001},"max":{"1min":29.280000000000001,"5min":148.13800000000001,"15min":417.22800000000001},"last":12.855}]}]},{"osd":1,"up_from":14,"seq":60129542498,"num_pgs":9,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":59332,"kb_used_data":6072,"kb_used_omap":1,"kb_used_meta":53246,"kb_avail":94312508,"statfs":{"total":96636764160,"available":96576008192,"internally_reserved":0,"allocated":6217728,"data_stored":3126747,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":54524357},"hb_peers":[0,2,3,4,5,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":65,"apply_latency_ms":65,"commit_latency_ns":65000000,"apply_latency_ns":65000000},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Mon Jun 3 09:06:07 2024","interfaces":[{"interface":"back","average":{"1min":7.7530000000000001,"5min":185.66999999999999,"15min":77.347999999999999},"min":{"1min":5.0229999999999997,"5min":4.9279999999999999,"15min":4.5449999999999999},"max":{"1min":11.896000000000001,"5min":7470.616,"15min":7470.616},"last":10.471},{"interface":"front","average":{"1min":9.9220000000000006,"5min":187.14599999999999,"15min":76.325999999999993},"min":{"1min":5.3159999999999998,"5min":5.3159999999999998,"15min":5.3159999999999998},"max":{"1min":18.338000000000001,"5min":7477.0829999999996,"15min":7477.0829999999996},"last":5.7300000000000004}]},{"osd":2,"last update":"Mon Jun 3 09:05:18 2024","interfaces":[{"interface":"back","average":{"1min":10.954000000000001,"5min":18.295000000000002,"15min":24.126999999999999},"min":{"1min":6.8739999999999997,"5min":5.8650000000000002,"15min":5.8650000000000002},"max":{"1min":21.122,"5min":228,"15min":657.38},"last":8.5600000000000005},{"interface":"front","average":{"1min":13.694000000000001,"5min":24.446999999999999,"15min":32.055999999999997},"min":{"1min":9.9860000000000007,"5min":9.1229999999999993,"15min":7.2160000000000002},"max":{"1min":26.507999999999999,"5min":229.465,"15min":700.06500000000005},"last":12.885999999999999}]},{"osd":3,"last update":"Mon Jun 3 09:06:04 2024","interfaces":[{"interface":"back","average":{"1min":8.6110000000000007,"5min":15.733000000000001,"15min":26.646999999999998},"min":{"1min":5.827,"5min":5.4640000000000004,"15min":5.4640000000000004},"max":{"1min":13.105,"5min":121.619,"15min":1676.009},"last":6.3609999999999998},{"interface":"front","average":{"1min":9.5850000000000009,"5min":16.161999999999999,"15min":29.065999999999999},"min":{"1min":6.3630000000000004,"5min":5.7699999999999996,"15min":5.7699999999999996},"max":{"1min":15.166,"5min":132.036,"15min":1766.5840000000001},"last":12.067}]},{"osd":4,"last update":"Mon Jun 3 09:05:31 2024","interfaces":[{"interface":"back","average":{"1min":10.996,"5min":954.46500000000003,"15min":1281.6859999999999},"min":{"1min":7.702,"5min":6.1760000000000002,"15min":6.0970000000000004},"max":{"1min":21.538,"5min":28044.862000000001,"15min":59438.017999999996},"last":7.4029999999999996},{"interface":"front","average":{"1min":13.646000000000001,"5min":961.24599999999998,"15min":1285.26},"min":{"1min":9.7200000000000006,"5min":9.3940000000000001,"15min":6.4640000000000004},"max":{"1min":23.277999999999999,"5min":28045.624,"15min":59439.283000000003},"last":9.8949999999999996}]},{"osd":5,"last update":"Mon Jun 3 09:06:04 2024","interfaces":[{"interface":"back","average":{"1min":12.478999999999999,"5min":20.228000000000002,"15min":25.997},"min":{"1min":8.6430000000000007,"5min":7.9720000000000004,"15min":7.234},"max":{"1min":17.442,"5min":139.43199999999999,"15min":701.22000000000003},"last":7.9050000000000002},{"interface":"front","average":{"1min":13.507999999999999,"5min":25.760999999999999,"15min":27.553999999999998},"min":{"1min":10.042999999999999,"5min":7.258,"15min":6.242},"max":{"1min":20.611000000000001,"5min":278.63,"15min":598.35000000000002},"last":11.395}]},{"osd":6,"last update":"Mon Jun 3 09:03:54 2024","interfaces":[{"interface":"back","average":{"1min":28.524000000000001,"5min":20.654,"15min":23.878},"min":{"1min":6.0309999999999997,"5min":6.0309999999999997,"15min":6.0309999999999997},"max":{"1min":167.221,"5min":167.221,"15min":704.245},"last":18.721},{"interface":"front","average":{"1min":27.606000000000002,"5min":30.131,"15min":28.628},"min":{"1min":9.8650000000000002,"5min":9.7200000000000006,"15min":8.3580000000000005},"max":{"1min":123.938,"5min":358.23000000000002,"15min":634.05499999999995},"last":173.113}]},{"osd":7,"last update":"Mon Jun 3 09:05:18 2024","interfaces":[{"interface":"back","average":{"1min":21.853999999999999,"5min":24.245999999999999,"15min":32.801000000000002},"min":{"1min":9.5020000000000007,"5min":8.0559999999999992,"15min":8.0559999999999992},"max":{"1min":142.245,"5min":270.95100000000002,"15min":703.07799999999997},"last":15.696},{"interface":"front","average":{"1min":21.547000000000001,"5min":29.977,"15min":30.513999999999999},"min":{"1min":11.167,"5min":8.2089999999999996,"15min":7.7599999999999998},"max":{"1min":119.348,"5min":360.99900000000002,"15min":608.44100000000003},"last":14.506}]}]},{"osd":0,"up_from":14,"seq":60129542500,"num_pgs":12,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":81088,"kb_used_data":5680,"kb_used_omap":1,"kb_used_meta":75390,"kb_avail":94290752,"statfs":{"total":96636764160,"available":96553730048,"internally_reserved":0,"allocated":5816320,"data_stored":2728907,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1593,"internal_metadata":77199815},"hb_peers":[1,2,3,4,5,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":4,"apply_latency_ms":4,"commit_latency_ns":4000000,"apply_latency_ns":4000000},"alerts":[],"network_ping_times":[{"osd":1,"last update":"Mon Jun 3 09:06:04 2024","interfaces":[{"interface":"back","average":{"1min":6.9960000000000004,"5min":56.997,"15min":339.29700000000003},"min":{"1min":4.4530000000000003,"5min":4.4530000000000003,"15min":4.4100000000000001},"max":{"1min":11.720000000000001,"5min":2414.4749999999999,"15min":20691.453000000001},"last":9.875},{"interface":"front","average":{"1min":9.548,"5min":57.258000000000003,"15min":337.791},"min":{"1min":5.2300000000000004,"5min":5.2300000000000004,"15min":4.6050000000000004},"max":{"1min":15.337999999999999,"5min":2412.9549999999999,"15min":20640.792000000001},"last":4.7199999999999998}]},{"osd":2,"last update":"Mon Jun 3 09:05:22 2024","interfaces":[{"interface":"back","average":{"1min":9.5820000000000007,"5min":27.760000000000002,"15min":23.097999999999999},"min":{"1min":5.766,"5min":5.766,"15min":5.766},"max":{"1min":46.584000000000003,"5min":627.40700000000004,"15min":627.40700000000004},"last":6.3840000000000003},{"interface":"front","average":{"1min":13.574999999999999,"5min":35.847999999999999,"15min":30.065000000000001},"min":{"1min":8.6419999999999995,"5min":7.8630000000000004,"15min":6.9260000000000002},"max":{"1min":54.68,"5min":635.62599999999998,"15min":635.62599999999998},"last":8.5039999999999996}]},{"osd":3,"last update":"Mon Jun 3 09:06:04 2024","interfaces":[{"interface":"back","average":{"1min":12.105,"5min":26.925000000000001,"15min":26.584},"min":{"1min":7.8410000000000002,"5min":7.1280000000000001,"15min":6.1909999999999998},"max":{"1min":18.425999999999998,"5min":389.72699999999998,"15min":389.72699999999998},"last":10.986000000000001},{"interface":"front","average":{"1min":13.794,"5min":29.771999999999998,"15min":28.815000000000001},"min":{"1min":10.084,"5min":9.4920000000000009,"15min":6.1849999999999996},"max":{"1min":17.373999999999999,"5min":390.23200000000003,"15min":494.92899999999997},"last":16.532}]},{"osd":4,"last update":"Mon Jun 3 09:05:27 2024","interfaces":[{"interface":"back","average":{"1min":13.478,"5min":1032.126,"15min":1389.729},"min":{"1min":6.5119999999999996,"5min":6.5119999999999996,"15min":6.5119999999999996},"max":{"1min":55.354999999999997,"5min":25555.212,"15min":60989.964},"last":12.74},{"interface":"front","average":{"1min":16.204999999999998,"5min":1030.8389999999999,"15min":1387.778},"min":{"1min":10.282999999999999,"5min":7.1429999999999998,"15min":7.1429999999999998},"max":{"1min":49.945999999999998,"5min":25556.159,"15min":61000.017},"last":13.673}]},{"osd":5,"last update":"Mon Jun 3 09:05:13 2024","interfaces":[{"interface":"back","average":{"1min":13.778,"5min":31.690999999999999,"15min":29.962},"min":{"1min":8.5030000000000001,"5min":7.7030000000000003,"15min":7.7030000000000003},"max":{"1min":60.216999999999999,"5min":455.78699999999998,"15min":605.05899999999997},"last":14.942},{"interface":"front","average":{"1min":17.823,"5min":30.498999999999999,"15min":27.434999999999999},"min":{"1min":11.547000000000001,"5min":6.2030000000000003,"15min":6.2030000000000003},"max":{"1min":59.677,"5min":391.31200000000001,"15min":541.11000000000001},"last":17.879000000000001}]},{"osd":6,"last update":"Mon Jun 3 09:04:30 2024","interfaces":[{"interface":"back","average":{"1min":52.496000000000002,"5min":33.139000000000003,"15min":160.83199999999999},"min":{"1min":7.5289999999999999,"5min":7.1040000000000001,"15min":6.3739999999999997},"max":{"1min":457.91699999999997,"5min":457.91699999999997,"15min":11115.258},"last":18.959},{"interface":"front","average":{"1min":49.670000000000002,"5min":33.704999999999998,"15min":162.791},"min":{"1min":12.247999999999999,"5min":11.019,"15min":9.9760000000000009},"max":{"1min":407.05099999999999,"5min":407.05099999999999,"15min":11092.194},"last":17.707000000000001}]},{"osd":7,"last update":"Mon Jun 3 09:05:22 2024","interfaces":[{"interface":"back","average":{"1min":14.503,"5min":29.489000000000001,"15min":33.134999999999998},"min":{"1min":8.5690000000000008,"5min":7.5149999999999997,"15min":6.9859999999999998},"max":{"1min":48.526000000000003,"5min":461.41699999999997,"15min":598.33699999999999},"last":11.83},{"interface":"front","average":{"1min":17.050000000000001,"5min":33.313000000000002,"15min":38.603999999999999},"min":{"1min":11.182,"5min":10.263999999999999,"15min":10.263999999999999},"max":{"1min":58.683,"5min":463.10599999999999,"15min":592.80499999999995},"last":14.407}]}]}],"pool_statfs":[{"poolid":1,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":401408,"data_stored":397840,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":5,"total":0,"available":0,"internally_reserved":0,"allocated":401408,"data_stored":397840,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":4,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":5,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":6,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":17,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":17,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":17,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":17,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":17,"osd":4,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":17,"osd":5,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":5,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":17,"osd":6,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":5,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":17,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0}]}} 2024-06-03T09:06:20.392 DEBUG:teuthology.orchestra.run.smithi088:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph pg dump --format=json 2024-06-03T09:06:20.512 INFO:teuthology.orchestra.run.smithi088.stderr:2024-06-03T09:06:20.512+0000 7fdb1c416640 10 monclient: get_monmap_and_config 2024-06-03T09:06:20.513 INFO:teuthology.orchestra.run.smithi088.stderr:2024-06-03T09:06:20.512+0000 7fdb1c416640 10 monclient: build_initial_monmap 2024-06-03T09:06:20.513 INFO:teuthology.orchestra.run.smithi088.stderr:2024-06-03T09:06:20.512+0000 7fdb1c416640 10 monclient: monmap: 2024-06-03T09:06:20.513 INFO:teuthology.orchestra.run.smithi088.stderr:epoch 0 2024-06-03T09:06:20.513 INFO:teuthology.orchestra.run.smithi088.stderr:fsid ec3c3fdd-3ac6-4d7d-9412-683130ed8b04 2024-06-03T09:06:20.513 INFO:teuthology.orchestra.run.smithi088.stderr:last_changed 2024-06-03T09:06:20.513709+0000 2024-06-03T09:06:20.513 INFO:teuthology.orchestra.run.smithi088.stderr:created 2024-06-03T09:06:20.513709+0000 2024-06-03T09:06:20.513 INFO:teuthology.orchestra.run.smithi088.stderr:min_mon_release 0 (unknown) 2024-06-03T09:06:20.513 INFO:teuthology.orchestra.run.smithi088.stderr:election_strategy: 3 2024-06-03T09:06:20.513 INFO:teuthology.orchestra.run.smithi088.stderr:0: v1:172.21.15.88:6789/0 mon.noname-a 2024-06-03T09:06:20.513 INFO:teuthology.orchestra.run.smithi088.stderr:1: v1:172.21.15.192:6789/0 mon.noname-c 2024-06-03T09:06:20.514 INFO:teuthology.orchestra.run.smithi088.stderr:2: v1:172.21.15.88:6790/0 mon.noname-b 2024-06-03T09:06:20.514 INFO:teuthology.orchestra.run.smithi088.stderr: 2024-06-03T09:06:20.515 INFO:teuthology.orchestra.run.smithi088.stderr:2024-06-03T09:06:20.513+0000 7fdb1c416640 1 Processor -- start 2024-06-03T09:06:20.515 INFO:teuthology.orchestra.run.smithi088.stderr:2024-06-03T09:06:20.513+0000 7fdb1c416640 1 -- start start 2024-06-03T09:06:20.515 INFO:teuthology.orchestra.run.smithi088.stderr:2024-06-03T09:06:20.513+0000 7fdb1c416640 10 monclient: init 2024-06-03T09:06:20.516 INFO:teuthology.orchestra.run.smithi088.stderr:2024-06-03T09:06:20.514+0000 7fdb1c416640 10 monclient: _reopen_session rank -1 2024-06-03T09:06:20.516 INFO:teuthology.orchestra.run.smithi088.stderr:2024-06-03T09:06:20.514+0000 7fdb1c416640 10 monclient: _add_conns ranks=[0,1,2] 2024-06-03T09:06:20.516 INFO:teuthology.orchestra.run.smithi088.stderr:2024-06-03T09:06:20.514+0000 7fdb1c416640 10 monclient(hunting): picked mon.noname-a con 0x7fdb1406fc70 addr v1:172.21.15.88:6789/0 2024-06-03T09:06:20.516 INFO:teuthology.orchestra.run.smithi088.stderr:2024-06-03T09:06:20.514+0000 7fdb1c416640 10 monclient(hunting): picked mon.noname-c con 0x7fdb140ad920 addr v1:172.21.15.192:6789/0 2024-06-03T09:06:20.516 INFO:teuthology.orchestra.run.smithi088.stderr:2024-06-03T09:06:20.514+0000 7fdb1c416640 10 monclient(hunting): picked mon.noname-b con 0x7fdb141393f0 addr v1:172.21.15.88:6790/0 2024-06-03T09:06:20.516 INFO:teuthology.orchestra.run.smithi088.stderr:2024-06-03T09:06:20.514+0000 7fdb1c416640 1 -- --> v1:172.21.15.88:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fdb14070420 con 0x7fdb1406fc70 2024-06-03T09:06:20.516 INFO:teuthology.orchestra.run.smithi088.stderr:2024-06-03T09:06:20.514+0000 7fdb1c416640 1 -- --> v1:172.21.15.192:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fdb14071050 con 0x7fdb140ad920 2024-06-03T09:06:20.516 INFO:teuthology.orchestra.run.smithi088.stderr:2024-06-03T09:06:20.514+0000 7fdb1c416640 1 -- --> v1:172.21.15.88:6790/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fdb140add60 con 0x7fdb141393f0 2024-06-03T09:06:20.517 INFO:teuthology.orchestra.run.smithi088.stderr:2024-06-03T09:06:20.514+0000 7fdb1c416640 10 monclient(hunting): _renew_subs 2024-06-03T09:06:20.517 INFO:teuthology.orchestra.run.smithi088.stderr:2024-06-03T09:06:20.514+0000 7fdb1c416640 10 monclient(hunting): authenticate will time out at 3036.324562s 2024-06-03T09:06:20.538 INFO:teuthology.orchestra.run.smithi088.stderr:2024-06-03T09:06:20.538+0000 7fdb199b1640 1 --1- >> v1:172.21.15.192:6789/0 conn(0x7fdb140ad920 0x7fdb14138ce0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.192:6789/0 says I am v1:172.21.15.88:45546/0 (socket says 172.21.15.88:45546) 2024-06-03T09:06:20.538 INFO:teuthology.orchestra.run.smithi088.stderr:2024-06-03T09:06:20.538+0000 7fdb199b1640 1 -- 172.21.15.88:0/236507320 learned_addr learned my addr 172.21.15.88:0/236507320 (peer_addr_for_me v1:172.21.15.88:0/0) 2024-06-03T09:06:20.542 INFO:teuthology.orchestra.run.smithi088.stderr:2024-06-03T09:06:20.542+0000 7fdb191b0640 1 -- 172.21.15.88:0/236507320 <== mon.0 v1:172.21.15.88:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 487775663 0 0) 0x7fdb14070420 con 0x7fdb1406fc70 2024-06-03T09:06:20.542 INFO:teuthology.orchestra.run.smithi088.stderr:2024-06-03T09:06:20.542+0000 7fdb191b0640 10 monclient(hunting): _negotiate 2024-06-03T09:06:20.542 INFO:teuthology.orchestra.run.smithi088.stderr:2024-06-03T09:06:20.542+0000 7fdb191b0640 10 monclient(hunting): _init_auth method 2 2024-06-03T09:06:20.542 INFO:teuthology.orchestra.run.smithi088.stderr:2024-06-03T09:06:20.542+0000 7fdb191b0640 10 monclient(hunting): _init_auth creating new auth 2024-06-03T09:06:20.542 INFO:teuthology.orchestra.run.smithi088.stderr:2024-06-03T09:06:20.542+0000 7fdb191b0640 10 monclient(hunting): my global_id is 30576 2024-06-03T09:06:20.543 INFO:teuthology.orchestra.run.smithi088.stderr:2024-06-03T09:06:20.542+0000 7fdb191b0640 1 -- 172.21.15.88:0/236507320 --> v1:172.21.15.88:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7fdafc003520 con 0x7fdb1406fc70 2024-06-03T09:06:20.545 INFO:teuthology.orchestra.run.smithi088.stderr:2024-06-03T09:06:20.544+0000 7fdb191b0640 1 -- 172.21.15.88:0/236507320 <== mon.1 v1:172.21.15.192:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 283642176 0 0) 0x7fdb14071050 con 0x7fdb140ad920 2024-06-03T09:06:20.545 INFO:teuthology.orchestra.run.smithi088.stderr:2024-06-03T09:06:20.544+0000 7fdb191b0640 10 monclient(hunting): _negotiate 2024-06-03T09:06:20.545 INFO:teuthology.orchestra.run.smithi088.stderr:2024-06-03T09:06:20.544+0000 7fdb191b0640 10 monclient(hunting): _init_auth method 2 2024-06-03T09:06:20.545 INFO:teuthology.orchestra.run.smithi088.stderr:2024-06-03T09:06:20.544+0000 7fdb191b0640 10 monclient(hunting): _init_auth creating new auth 2024-06-03T09:06:20.545 INFO:teuthology.orchestra.run.smithi088.stderr:2024-06-03T09:06:20.544+0000 7fdb191b0640 10 monclient(hunting): my global_id is 30157 2024-06-03T09:06:20.545 INFO:teuthology.orchestra.run.smithi088.stderr:2024-06-03T09:06:20.544+0000 7fdb191b0640 1 -- 172.21.15.88:0/236507320 --> v1:172.21.15.192:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7fdb14070420 con 0x7fdb140ad920 2024-06-03T09:06:20.547 INFO:teuthology.orchestra.run.smithi088.stderr:2024-06-03T09:06:20.546+0000 7fdb191b0640 1 -- 172.21.15.88:0/236507320 <== mon.2 v1:172.21.15.88:6790/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 2384108943 0 0) 0x7fdb140add60 con 0x7fdb141393f0 2024-06-03T09:06:20.547 INFO:teuthology.orchestra.run.smithi088.stderr:2024-06-03T09:06:20.546+0000 7fdb191b0640 10 monclient(hunting): _negotiate 2024-06-03T09:06:20.547 INFO:teuthology.orchestra.run.smithi088.stderr:2024-06-03T09:06:20.546+0000 7fdb191b0640 10 monclient(hunting): _init_auth method 2 2024-06-03T09:06:20.547 INFO:teuthology.orchestra.run.smithi088.stderr:2024-06-03T09:06:20.546+0000 7fdb191b0640 10 monclient(hunting): _init_auth creating new auth 2024-06-03T09:06:20.547 INFO:teuthology.orchestra.run.smithi088.stderr:2024-06-03T09:06:20.546+0000 7fdb191b0640 10 monclient(hunting): my global_id is 30080 2024-06-03T09:06:20.547 INFO:teuthology.orchestra.run.smithi088.stderr:2024-06-03T09:06:20.546+0000 7fdb191b0640 1 -- 172.21.15.88:0/236507320 --> v1:172.21.15.88:6790/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7fdb14071050 con 0x7fdb141393f0 2024-06-03T09:06:20.547 INFO:teuthology.orchestra.run.smithi088.stderr:2024-06-03T09:06:20.546+0000 7fdb191b0640 1 -- 172.21.15.88:0/236507320 <== mon.0 v1:172.21.15.88:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 214+0+0 (unknown 2039923568 0 0) 0x7fdafc003520 con 0x7fdb1406fc70 2024-06-03T09:06:20.547 INFO:teuthology.orchestra.run.smithi088.stderr:2024-06-03T09:06:20.546+0000 7fdb191b0640 1 -- 172.21.15.88:0/236507320 >> v1:172.21.15.88:6790/0 conn(0x7fdb141393f0 legacy=0x7fdb1414ac50 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-06-03T09:06:20.548 INFO:teuthology.orchestra.run.smithi088.stderr:2024-06-03T09:06:20.546+0000 7fdb191b0640 1 -- 172.21.15.88:0/236507320 >> v1:172.21.15.192:6789/0 conn(0x7fdb140ad920 legacy=0x7fdb14138ce0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-06-03T09:06:20.548 INFO:teuthology.orchestra.run.smithi088.stderr:2024-06-03T09:06:20.546+0000 7fdb191b0640 10 monclient: _finish_hunting 0 2024-06-03T09:06:20.548 INFO:teuthology.orchestra.run.smithi088.stderr:2024-06-03T09:06:20.546+0000 7fdb191b0640 1 monclient: found mon.noname-a 2024-06-03T09:06:20.548 INFO:teuthology.orchestra.run.smithi088.stderr:2024-06-03T09:06:20.546+0000 7fdb191b0640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-06-03T09:06:20.548 INFO:teuthology.orchestra.run.smithi088.stderr:2024-06-03T09:06:20.546+0000 7fdb191b0640 10 monclient: _send_mon_message to mon.noname-a at v1:172.21.15.88:6789/0 2024-06-03T09:06:20.548 INFO:teuthology.orchestra.run.smithi088.stderr:2024-06-03T09:06:20.546+0000 7fdb191b0640 1 -- 172.21.15.88:0/236507320 --> v1:172.21.15.88:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fdb14071be0 con 0x7fdb1406fc70 2024-06-03T09:06:20.548 INFO:teuthology.orchestra.run.smithi088.stderr:2024-06-03T09:06:20.546+0000 7fdb191b0640 10 monclient: _finish_auth 0 2024-06-03T09:06:20.548 INFO:teuthology.orchestra.run.smithi088.stderr:2024-06-03T09:06:20.546+0000 7fdb191b0640 10 monclient: _check_auth_tickets 2024-06-03T09:06:20.548 INFO:teuthology.orchestra.run.smithi088.stderr:2024-06-03T09:06:20.546+0000 7fdb191b0640 20 monclient: _check_auth_rotating not needed by client.admin 2024-06-03T09:06:20.548 INFO:teuthology.orchestra.run.smithi088.stderr:2024-06-03T09:06:20.546+0000 7fdb191b0640 1 -- 172.21.15.88:0/236507320 <== mon.0 v1:172.21.15.88:6789/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 2234374065 0 0) 0x7fdb04001990 con 0x7fdb1406fc70 2024-06-03T09:06:20.548 INFO:teuthology.orchestra.run.smithi088.stderr:2024-06-03T09:06:20.546+0000 7fdb191b0640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-06-03T09:06:20.548 INFO:teuthology.orchestra.run.smithi088.stderr:2024-06-03T09:06:20.546+0000 7fdb191b0640 10 monclient: got monmap 1 from mon.noname-a (according to old e1) 2024-06-03T09:06:20.548 INFO:teuthology.orchestra.run.smithi088.stderr:2024-06-03T09:06:20.546+0000 7fdb191b0640 10 monclient: dump: 2024-06-03T09:06:20.548 INFO:teuthology.orchestra.run.smithi088.stderr:epoch 1 2024-06-03T09:06:20.549 INFO:teuthology.orchestra.run.smithi088.stderr:fsid ec3c3fdd-3ac6-4d7d-9412-683130ed8b04 2024-06-03T09:06:20.549 INFO:teuthology.orchestra.run.smithi088.stderr:last_changed 2024-06-03T08:34:31.885125+0000 2024-06-03T09:06:20.549 INFO:teuthology.orchestra.run.smithi088.stderr:created 2024-06-03T08:34:31.885125+0000 2024-06-03T09:06:20.549 INFO:teuthology.orchestra.run.smithi088.stderr:min_mon_release 18 (reef) 2024-06-03T09:06:20.549 INFO:teuthology.orchestra.run.smithi088.stderr:election_strategy: 3 2024-06-03T09:06:20.549 INFO:teuthology.orchestra.run.smithi088.stderr:0: v1:172.21.15.88:6789/0 mon.a 2024-06-03T09:06:20.549 INFO:teuthology.orchestra.run.smithi088.stderr:1: v1:172.21.15.192:6789/0 mon.b 2024-06-03T09:06:20.549 INFO:teuthology.orchestra.run.smithi088.stderr:2: v1:172.21.15.88:6790/0 mon.c 2024-06-03T09:06:20.549 INFO:teuthology.orchestra.run.smithi088.stderr: 2024-06-03T09:06:20.549 INFO:teuthology.orchestra.run.smithi088.stderr:2024-06-03T09:06:20.546+0000 7fdb1c416640 5 monclient: authenticate success, global_id 30576 2024-06-03T09:06:20.549 INFO:teuthology.orchestra.run.smithi088.stderr:2024-06-03T09:06:20.546+0000 7fdb1c416640 20 monclient: get_monmap_and_config waiting for monmap|config 2024-06-03T09:06:20.551 INFO:teuthology.orchestra.run.smithi088.stderr:2024-06-03T09:06:20.551+0000 7fdb191b0640 1 -- 172.21.15.88:0/236507320 <== mon.0 v1:172.21.15.88:6789/0 4 ==== config(0 keys) v1 ==== 4+0+0 (unknown 0 0 0) 0x7fdb04001bb0 con 0x7fdb1406fc70 2024-06-03T09:06:20.551 INFO:teuthology.orchestra.run.smithi088.stderr:2024-06-03T09:06:20.551+0000 7fdb191b0640 10 monclient: handle_config config(0 keys) v1 2024-06-03T09:06:20.552 INFO:teuthology.orchestra.run.smithi088.stderr:2024-06-03T09:06:20.551+0000 7fdb191b0640 1 -- 172.21.15.88:0/236507320 <== mon.0 v1:172.21.15.88:6789/0 5 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 2234374065 0 0) 0x7fdb04001ec0 con 0x7fdb1406fc70 2024-06-03T09:06:20.552 INFO:teuthology.orchestra.run.smithi088.stderr:2024-06-03T09:06:20.551+0000 7fdb1c416640 10 monclient: get_monmap_and_config success 2024-06-03T09:06:20.552 INFO:teuthology.orchestra.run.smithi088.stderr:2024-06-03T09:06:20.551+0000 7fdb1c416640 4 set_mon_vals no callback set 2024-06-03T09:06:20.552 INFO:teuthology.orchestra.run.smithi088.stderr:2024-06-03T09:06:20.551+0000 7fdb191b0640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-06-03T09:06:20.552 INFO:teuthology.orchestra.run.smithi088.stderr:2024-06-03T09:06:20.551+0000 7fdb191b0640 10 monclient: got monmap 1 from mon.a (according to old e1) 2024-06-03T09:06:20.552 INFO:teuthology.orchestra.run.smithi088.stderr:2024-06-03T09:06:20.551+0000 7fdb191b0640 10 monclient: dump: 2024-06-03T09:06:20.552 INFO:teuthology.orchestra.run.smithi088.stderr:epoch 1 2024-06-03T09:06:20.552 INFO:teuthology.orchestra.run.smithi088.stderr:fsid ec3c3fdd-3ac6-4d7d-9412-683130ed8b04 2024-06-03T09:06:20.552 INFO:teuthology.orchestra.run.smithi088.stderr:last_changed 2024-06-03T08:34:31.885125+0000 2024-06-03T09:06:20.552 INFO:teuthology.orchestra.run.smithi088.stderr:created 2024-06-03T08:34:31.885125+0000 2024-06-03T09:06:20.552 INFO:teuthology.orchestra.run.smithi088.stderr:min_mon_release 18 (reef) 2024-06-03T09:06:20.552 INFO:teuthology.orchestra.run.smithi088.stderr:election_strategy: 3 2024-06-03T09:06:20.552 INFO:teuthology.orchestra.run.smithi088.stderr:0: v1:172.21.15.88:6789/0 mon.a 2024-06-03T09:06:20.552 INFO:teuthology.orchestra.run.smithi088.stderr:1: v1:172.21.15.192:6789/0 mon.b 2024-06-03T09:06:20.553 INFO:teuthology.orchestra.run.smithi088.stderr:2: v1:172.21.15.88:6790/0 mon.c 2024-06-03T09:06:20.553 INFO:teuthology.orchestra.run.smithi088.stderr: 2024-06-03T09:06:20.555 INFO:teuthology.orchestra.run.smithi088.stderr:2024-06-03T09:06:20.552+0000 7fdb1c416640 10 monclient: shutdown 2024-06-03T09:06:20.555 INFO:teuthology.orchestra.run.smithi088.stderr:2024-06-03T09:06:20.552+0000 7fdb1c416640 20 monclient: shutdown discarding 0 pending message(s) 2024-06-03T09:06:20.555 INFO:teuthology.orchestra.run.smithi088.stderr:2024-06-03T09:06:20.552+0000 7fdb1c416640 1 -- 172.21.15.88:0/236507320 >> v1:172.21.15.88:6789/0 conn(0x7fdb1406fc70 legacy=0x7fdb1413fbb0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-06-03T09:06:20.555 INFO:teuthology.orchestra.run.smithi088.stderr:2024-06-03T09:06:20.552+0000 7fdb1c416640 1 -- 172.21.15.88:0/236507320 shutdown_connections 2024-06-03T09:06:20.555 INFO:teuthology.orchestra.run.smithi088.stderr:2024-06-03T09:06:20.552+0000 7fdb1c416640 1 -- 172.21.15.88:0/236507320 >> 172.21.15.88:0/236507320 conn(0x7fdb140b2f20 msgr2=0x7fdb140b3300 unknown :-1 s=STATE_NONE l=0).mark_down 2024-06-03T09:06:20.555 INFO:teuthology.orchestra.run.smithi088.stderr:2024-06-03T09:06:20.552+0000 7fdb1c416640 1 -- 172.21.15.88:0/236507320 shutdown_connections 2024-06-03T09:06:20.555 INFO:teuthology.orchestra.run.smithi088.stderr:2024-06-03T09:06:20.552+0000 7fdb1c416640 1 -- 172.21.15.88:0/236507320 wait complete. 2024-06-03T09:06:20.556 INFO:teuthology.orchestra.run.smithi088.stderr:2024-06-03T09:06:20.553+0000 7fdb1c416640 10 monclient: build_initial_monmap 2024-06-03T09:06:20.556 INFO:teuthology.orchestra.run.smithi088.stderr:2024-06-03T09:06:20.553+0000 7fdb1c416640 10 monclient: monmap: 2024-06-03T09:06:20.556 INFO:teuthology.orchestra.run.smithi088.stderr:epoch 0 2024-06-03T09:06:20.556 INFO:teuthology.orchestra.run.smithi088.stderr:fsid 00000000-0000-0000-0000-000000000000 2024-06-03T09:06:20.556 INFO:teuthology.orchestra.run.smithi088.stderr:last_changed 0.000000 2024-06-03T09:06:20.556 INFO:teuthology.orchestra.run.smithi088.stderr:created 0.000000 2024-06-03T09:06:20.556 INFO:teuthology.orchestra.run.smithi088.stderr:min_mon_release 0 (unknown) 2024-06-03T09:06:20.556 INFO:teuthology.orchestra.run.smithi088.stderr:election_strategy: 1 2024-06-03T09:06:20.556 INFO:teuthology.orchestra.run.smithi088.stderr:0: v1:172.21.15.88:6789/0 mon.noname-a 2024-06-03T09:06:20.556 INFO:teuthology.orchestra.run.smithi088.stderr:1: v1:172.21.15.192:6789/0 mon.noname-b 2024-06-03T09:06:20.556 INFO:teuthology.orchestra.run.smithi088.stderr:2: v1:172.21.15.88:6790/0 mon.noname-c 2024-06-03T09:06:20.556 INFO:teuthology.orchestra.run.smithi088.stderr: 2024-06-03T09:06:20.556 INFO:teuthology.orchestra.run.smithi088.stderr:2024-06-03T09:06:20.553+0000 7fdb1c416640 1 librados: starting msgr at 2024-06-03T09:06:20.557 INFO:teuthology.orchestra.run.smithi088.stderr:2024-06-03T09:06:20.553+0000 7fdb1c416640 1 librados: starting objecter 2024-06-03T09:06:20.557 INFO:teuthology.orchestra.run.smithi088.stderr:2024-06-03T09:06:20.554+0000 7fdb1c416640 1 Processor -- start 2024-06-03T09:06:20.557 INFO:teuthology.orchestra.run.smithi088.stderr:2024-06-03T09:06:20.554+0000 7fdb1c416640 1 -- start start 2024-06-03T09:06:20.557 INFO:teuthology.orchestra.run.smithi088.stderr:2024-06-03T09:06:20.554+0000 7fdb1c416640 1 librados: setting wanted keys 2024-06-03T09:06:20.557 INFO:teuthology.orchestra.run.smithi088.stderr:2024-06-03T09:06:20.554+0000 7fdb1c416640 1 librados: calling monclient init 2024-06-03T09:06:20.557 INFO:teuthology.orchestra.run.smithi088.stderr:2024-06-03T09:06:20.554+0000 7fdb1c416640 10 monclient: init 2024-06-03T09:06:20.557 INFO:teuthology.orchestra.run.smithi088.stderr:2024-06-03T09:06:20.554+0000 7fdb1c416640 10 monclient: _reopen_session rank -1 2024-06-03T09:06:20.557 INFO:teuthology.orchestra.run.smithi088.stderr:2024-06-03T09:06:20.554+0000 7fdb1c416640 10 monclient: _add_conns ranks=[2,1,0] 2024-06-03T09:06:20.557 INFO:teuthology.orchestra.run.smithi088.stderr:2024-06-03T09:06:20.554+0000 7fdb1c416640 10 monclient(hunting): picked mon.noname-c con 0x7fdb1406fc70 addr v1:172.21.15.88:6790/0 2024-06-03T09:06:20.557 INFO:teuthology.orchestra.run.smithi088.stderr:2024-06-03T09:06:20.554+0000 7fdb1c416640 10 monclient(hunting): picked mon.noname-b con 0x7fdb140ad920 addr v1:172.21.15.192:6789/0 2024-06-03T09:06:20.557 INFO:teuthology.orchestra.run.smithi088.stderr:2024-06-03T09:06:20.554+0000 7fdb1c416640 10 monclient(hunting): picked mon.noname-a con 0x7fdb141393f0 addr v1:172.21.15.88:6789/0 2024-06-03T09:06:20.557 INFO:teuthology.orchestra.run.smithi088.stderr:2024-06-03T09:06:20.554+0000 7fdb1c416640 1 -- --> v1:172.21.15.88:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fdb14140aa0 con 0x7fdb141393f0 2024-06-03T09:06:20.557 INFO:teuthology.orchestra.run.smithi088.stderr:2024-06-03T09:06:20.554+0000 7fdb1c416640 1 -- --> v1:172.21.15.192:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fdb140aa470 con 0x7fdb140ad920 2024-06-03T09:06:20.558 INFO:teuthology.orchestra.run.smithi088.stderr:2024-06-03T09:06:20.554+0000 7fdb1c416640 1 -- --> v1:172.21.15.88:6790/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fdb140aa750 con 0x7fdb1406fc70 2024-06-03T09:06:20.558 INFO:teuthology.orchestra.run.smithi088.stderr:2024-06-03T09:06:20.554+0000 7fdb1c416640 10 monclient(hunting): _renew_subs 2024-06-03T09:06:20.558 INFO:teuthology.orchestra.run.smithi088.stderr:2024-06-03T09:06:20.554+0000 7fdb1c416640 10 monclient(hunting): authenticate will time out at 3036.364628s 2024-06-03T09:06:20.558 INFO:teuthology.orchestra.run.smithi088.stderr:2024-06-03T09:06:20.555+0000 7fdb1a9b3640 1 --1- >> v1:172.21.15.88:6789/0 conn(0x7fdb141393f0 0x7fdb14141640 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=1).handle_server_banner_and_identify peer v1:172.21.15.88:6789/0 says I am v1:172.21.15.88:43908/0 (socket says 172.21.15.88:43908) 2024-06-03T09:06:20.558 INFO:teuthology.orchestra.run.smithi088.stderr:2024-06-03T09:06:20.555+0000 7fdb1a9b3640 1 -- 172.21.15.88:0/470026592 learned_addr learned my addr 172.21.15.88:0/470026592 (peer_addr_for_me v1:172.21.15.88:0/0) 2024-06-03T09:06:20.558 INFO:teuthology.orchestra.run.smithi088.stderr:2024-06-03T09:06:20.555+0000 7fdb12ffd640 10 client.?.objecter ms_handle_connect 0x7fdb141393f0 2024-06-03T09:06:20.558 INFO:teuthology.orchestra.run.smithi088.stderr:2024-06-03T09:06:20.555+0000 7fdb12ffd640 10 client.?.objecter resend_mon_ops 2024-06-03T09:06:20.558 INFO:teuthology.orchestra.run.smithi088.stderr:2024-06-03T09:06:20.555+0000 7fdb12ffd640 10 client.?.objecter ms_handle_connect 0x7fdb1406fc70 2024-06-03T09:06:20.558 INFO:teuthology.orchestra.run.smithi088.stderr:2024-06-03T09:06:20.555+0000 7fdb12ffd640 10 client.?.objecter resend_mon_ops 2024-06-03T09:06:20.558 INFO:teuthology.orchestra.run.smithi088.stderr:2024-06-03T09:06:20.555+0000 7fdb12ffd640 10 client.?.objecter ms_handle_connect 0x7fdb140ad920 2024-06-03T09:06:20.558 INFO:teuthology.orchestra.run.smithi088.stderr:2024-06-03T09:06:20.555+0000 7fdb12ffd640 10 client.?.objecter resend_mon_ops 2024-06-03T09:06:20.559 INFO:teuthology.orchestra.run.smithi088.stderr:2024-06-03T09:06:20.557+0000 7fdb12ffd640 1 -- 172.21.15.88:0/470026592 <== mon.1 v1:172.21.15.192:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 2774739168 0 0) 0x7fdb140aa470 con 0x7fdb140ad920 2024-06-03T09:06:20.559 INFO:teuthology.orchestra.run.smithi088.stderr:2024-06-03T09:06:20.557+0000 7fdb12ffd640 10 monclient(hunting): _negotiate 2024-06-03T09:06:20.559 INFO:teuthology.orchestra.run.smithi088.stderr:2024-06-03T09:06:20.557+0000 7fdb12ffd640 10 monclient(hunting): _init_auth method 2 2024-06-03T09:06:20.559 INFO:teuthology.orchestra.run.smithi088.stderr:2024-06-03T09:06:20.557+0000 7fdb12ffd640 10 monclient(hunting): _init_auth creating new auth 2024-06-03T09:06:20.560 INFO:teuthology.orchestra.run.smithi088.stderr:2024-06-03T09:06:20.557+0000 7fdb12ffd640 10 monclient(hunting): my global_id is 30160 2024-06-03T09:06:20.560 INFO:teuthology.orchestra.run.smithi088.stderr:2024-06-03T09:06:20.557+0000 7fdb12ffd640 1 -- 172.21.15.88:0/470026592 --> v1:172.21.15.192:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7fdb00004eb0 con 0x7fdb140ad920 2024-06-03T09:06:20.560 INFO:teuthology.orchestra.run.smithi088.stderr:2024-06-03T09:06:20.557+0000 7fdb12ffd640 1 -- 172.21.15.88:0/470026592 <== mon.0 v1:172.21.15.88:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 1695580037 0 0) 0x7fdb14140aa0 con 0x7fdb141393f0 2024-06-03T09:06:20.560 INFO:teuthology.orchestra.run.smithi088.stderr:2024-06-03T09:06:20.557+0000 7fdb12ffd640 10 monclient(hunting): _negotiate 2024-06-03T09:06:20.560 INFO:teuthology.orchestra.run.smithi088.stderr:2024-06-03T09:06:20.557+0000 7fdb12ffd640 10 monclient(hunting): _init_auth method 2 2024-06-03T09:06:20.560 INFO:teuthology.orchestra.run.smithi088.stderr:2024-06-03T09:06:20.557+0000 7fdb12ffd640 10 monclient(hunting): _init_auth creating new auth 2024-06-03T09:06:20.560 INFO:teuthology.orchestra.run.smithi088.stderr:2024-06-03T09:06:20.557+0000 7fdb12ffd640 10 monclient(hunting): my global_id is 30579 2024-06-03T09:06:20.560 INFO:teuthology.orchestra.run.smithi088.stderr:2024-06-03T09:06:20.557+0000 7fdb12ffd640 1 -- 172.21.15.88:0/470026592 --> v1:172.21.15.88:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7fdb140aa470 con 0x7fdb141393f0 2024-06-03T09:06:20.560 INFO:teuthology.orchestra.run.smithi088.stderr:2024-06-03T09:06:20.558+0000 7fdb12ffd640 1 -- 172.21.15.88:0/470026592 <== mon.2 v1:172.21.15.88:6790/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 478811200 0 0) 0x7fdb140aa750 con 0x7fdb1406fc70 2024-06-03T09:06:20.560 INFO:teuthology.orchestra.run.smithi088.stderr:2024-06-03T09:06:20.558+0000 7fdb12ffd640 10 monclient(hunting): _negotiate 2024-06-03T09:06:20.560 INFO:teuthology.orchestra.run.smithi088.stderr:2024-06-03T09:06:20.558+0000 7fdb12ffd640 10 monclient(hunting): _init_auth method 2 2024-06-03T09:06:20.560 INFO:teuthology.orchestra.run.smithi088.stderr:2024-06-03T09:06:20.558+0000 7fdb12ffd640 10 monclient(hunting): _init_auth creating new auth 2024-06-03T09:06:20.560 INFO:teuthology.orchestra.run.smithi088.stderr:2024-06-03T09:06:20.558+0000 7fdb12ffd640 10 monclient(hunting): my global_id is 30083 2024-06-03T09:06:20.561 INFO:teuthology.orchestra.run.smithi088.stderr:2024-06-03T09:06:20.558+0000 7fdb12ffd640 1 -- 172.21.15.88:0/470026592 --> v1:172.21.15.88:6790/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7fdb14140aa0 con 0x7fdb1406fc70 2024-06-03T09:06:20.562 INFO:teuthology.orchestra.run.smithi088.stderr:2024-06-03T09:06:20.562+0000 7fdb12ffd640 1 -- 172.21.15.88:0/470026592 <== mon.1 v1:172.21.15.192:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 764+0+0 (unknown 3411367800 0 0) 0x7fdb00004eb0 con 0x7fdb140ad920 2024-06-03T09:06:20.563 INFO:teuthology.orchestra.run.smithi088.stderr:2024-06-03T09:06:20.562+0000 7fdb12ffd640 1 -- 172.21.15.88:0/470026592 --> v1:172.21.15.192:6789/0 -- auth(proto 2 165 bytes epoch 0) v1 -- 0x7fdb140aa750 con 0x7fdb140ad920 2024-06-03T09:06:20.563 INFO:teuthology.orchestra.run.smithi088.stderr:2024-06-03T09:06:20.562+0000 7fdb12ffd640 1 -- 172.21.15.88:0/470026592 <== mon.1 v1:172.21.15.192:6789/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 2234374065 0 0) 0x7fdb080018b0 con 0x7fdb140ad920 2024-06-03T09:06:20.563 INFO:teuthology.orchestra.run.smithi088.stderr:2024-06-03T09:06:20.562+0000 7fdb12ffd640 10 monclient(hunting): handle_monmap mon_map magic: 0 v1 2024-06-03T09:06:20.563 INFO:teuthology.orchestra.run.smithi088.stderr:2024-06-03T09:06:20.562+0000 7fdb12ffd640 10 monclient(hunting): got monmap 1 from mon.noname-b (according to old e1) 2024-06-03T09:06:20.563 INFO:teuthology.orchestra.run.smithi088.stderr:2024-06-03T09:06:20.562+0000 7fdb12ffd640 10 monclient(hunting): dump: 2024-06-03T09:06:20.563 INFO:teuthology.orchestra.run.smithi088.stderr:epoch 1 2024-06-03T09:06:20.563 INFO:teuthology.orchestra.run.smithi088.stderr:fsid ec3c3fdd-3ac6-4d7d-9412-683130ed8b04 2024-06-03T09:06:20.563 INFO:teuthology.orchestra.run.smithi088.stderr:last_changed 2024-06-03T08:34:31.885125+0000 2024-06-03T09:06:20.563 INFO:teuthology.orchestra.run.smithi088.stderr:created 2024-06-03T08:34:31.885125+0000 2024-06-03T09:06:20.563 INFO:teuthology.orchestra.run.smithi088.stderr:min_mon_release 18 (reef) 2024-06-03T09:06:20.563 INFO:teuthology.orchestra.run.smithi088.stderr:election_strategy: 3 2024-06-03T09:06:20.563 INFO:teuthology.orchestra.run.smithi088.stderr:0: v1:172.21.15.88:6789/0 mon.a 2024-06-03T09:06:20.563 INFO:teuthology.orchestra.run.smithi088.stderr:1: v1:172.21.15.192:6789/0 mon.b 2024-06-03T09:06:20.564 INFO:teuthology.orchestra.run.smithi088.stderr:2: v1:172.21.15.88:6790/0 mon.c 2024-06-03T09:06:20.564 INFO:teuthology.orchestra.run.smithi088.stderr: 2024-06-03T09:06:20.564 INFO:teuthology.orchestra.run.smithi088.stderr:2024-06-03T09:06:20.562+0000 7fdb12ffd640 10 monclient(hunting): _finish_auth 0 2024-06-03T09:06:20.565 INFO:teuthology.orchestra.run.smithi088.stderr:2024-06-03T09:06:20.564+0000 7fdb12ffd640 1 -- 172.21.15.88:0/470026592 <== mon.0 v1:172.21.15.88:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 764+0+0 (unknown 1050569811 0 0) 0x7fdb140aa470 con 0x7fdb141393f0 2024-06-03T09:06:20.565 INFO:teuthology.orchestra.run.smithi088.stderr:2024-06-03T09:06:20.564+0000 7fdb12ffd640 1 -- 172.21.15.88:0/470026592 --> v1:172.21.15.88:6789/0 -- auth(proto 2 165 bytes epoch 0) v1 -- 0x7fdb00004eb0 con 0x7fdb141393f0 2024-06-03T09:06:20.565 INFO:teuthology.orchestra.run.smithi088.stderr:2024-06-03T09:06:20.564+0000 7fdb12ffd640 1 -- 172.21.15.88:0/470026592 <== mon.2 v1:172.21.15.88:6790/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 764+0+0 (unknown 230398550 0 0) 0x7fdb14140aa0 con 0x7fdb1406fc70 2024-06-03T09:06:20.565 INFO:teuthology.orchestra.run.smithi088.stderr:2024-06-03T09:06:20.564+0000 7fdb12ffd640 1 -- 172.21.15.88:0/470026592 --> v1:172.21.15.88:6790/0 -- auth(proto 2 165 bytes epoch 0) v1 -- 0x7fdb140aa470 con 0x7fdb1406fc70 2024-06-03T09:06:20.565 INFO:teuthology.orchestra.run.smithi088.stderr:2024-06-03T09:06:20.564+0000 7fdb12ffd640 1 -- 172.21.15.88:0/470026592 <== mon.2 v1:172.21.15.88:6790/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 2234374065 0 0) 0x7fdb04001ec0 con 0x7fdb1406fc70 2024-06-03T09:06:20.565 INFO:teuthology.orchestra.run.smithi088.stderr:2024-06-03T09:06:20.564+0000 7fdb12ffd640 10 monclient(hunting): handle_monmap mon_map magic: 0 v1 2024-06-03T09:06:20.565 INFO:teuthology.orchestra.run.smithi088.stderr:2024-06-03T09:06:20.564+0000 7fdb12ffd640 10 monclient(hunting): got monmap 1 from mon.c (according to old e1) 2024-06-03T09:06:20.565 INFO:teuthology.orchestra.run.smithi088.stderr:2024-06-03T09:06:20.564+0000 7fdb12ffd640 10 monclient(hunting): dump: 2024-06-03T09:06:20.565 INFO:teuthology.orchestra.run.smithi088.stderr:epoch 1 2024-06-03T09:06:20.565 INFO:teuthology.orchestra.run.smithi088.stderr:fsid ec3c3fdd-3ac6-4d7d-9412-683130ed8b04 2024-06-03T09:06:20.565 INFO:teuthology.orchestra.run.smithi088.stderr:last_changed 2024-06-03T08:34:31.885125+0000 2024-06-03T09:06:20.565 INFO:teuthology.orchestra.run.smithi088.stderr:created 2024-06-03T08:34:31.885125+0000 2024-06-03T09:06:20.565 INFO:teuthology.orchestra.run.smithi088.stderr:min_mon_release 18 (reef) 2024-06-03T09:06:20.566 INFO:teuthology.orchestra.run.smithi088.stderr:election_strategy: 3 2024-06-03T09:06:20.566 INFO:teuthology.orchestra.run.smithi088.stderr:0: v1:172.21.15.88:6789/0 mon.a 2024-06-03T09:06:20.566 INFO:teuthology.orchestra.run.smithi088.stderr:1: v1:172.21.15.192:6789/0 mon.b 2024-06-03T09:06:20.566 INFO:teuthology.orchestra.run.smithi088.stderr:2: v1:172.21.15.88:6790/0 mon.c 2024-06-03T09:06:20.566 INFO:teuthology.orchestra.run.smithi088.stderr: 2024-06-03T09:06:20.567 INFO:teuthology.orchestra.run.smithi088.stderr:2024-06-03T09:06:20.566+0000 7fdb12ffd640 1 -- 172.21.15.88:0/470026592 <== mon.1 v1:172.21.15.192:6789/0 4 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 580+0+0 (unknown 1513867899 0 0) 0x7fdb140aa750 con 0x7fdb140ad920 2024-06-03T09:06:20.567 INFO:teuthology.orchestra.run.smithi088.stderr:2024-06-03T09:06:20.566+0000 7fdb12ffd640 1 -- 172.21.15.88:0/470026592 >> v1:172.21.15.88:6790/0 conn(0x7fdb1406fc70 legacy=0x7fdb14135290 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-06-03T09:06:20.567 INFO:teuthology.orchestra.run.smithi088.stderr:2024-06-03T09:06:20.566+0000 7fdb12ffd640 1 -- 172.21.15.88:0/470026592 >> v1:172.21.15.88:6789/0 conn(0x7fdb141393f0 legacy=0x7fdb14141640 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-06-03T09:06:20.567 INFO:teuthology.orchestra.run.smithi088.stderr:2024-06-03T09:06:20.566+0000 7fdb12ffd640 10 monclient: _finish_hunting 0 2024-06-03T09:06:20.567 INFO:teuthology.orchestra.run.smithi088.stderr:2024-06-03T09:06:20.566+0000 7fdb12ffd640 1 monclient: found mon.b 2024-06-03T09:06:20.567 INFO:teuthology.orchestra.run.smithi088.stderr:2024-06-03T09:06:20.566+0000 7fdb12ffd640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-06-03T09:06:20.567 INFO:teuthology.orchestra.run.smithi088.stderr:2024-06-03T09:06:20.566+0000 7fdb12ffd640 10 monclient: _send_mon_message to mon.b at v1:172.21.15.192:6789/0 2024-06-03T09:06:20.567 INFO:teuthology.orchestra.run.smithi088.stderr:2024-06-03T09:06:20.566+0000 7fdb12ffd640 1 -- 172.21.15.88:0/470026592 --> v1:172.21.15.192:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fdb14142420 con 0x7fdb140ad920 2024-06-03T09:06:20.567 INFO:teuthology.orchestra.run.smithi088.stderr:2024-06-03T09:06:20.566+0000 7fdb12ffd640 10 monclient: _finish_auth 0 2024-06-03T09:06:20.567 INFO:teuthology.orchestra.run.smithi088.stderr:2024-06-03T09:06:20.566+0000 7fdb12ffd640 10 monclient: _check_auth_tickets 2024-06-03T09:06:20.567 INFO:teuthology.orchestra.run.smithi088.stderr:2024-06-03T09:06:20.566+0000 7fdb12ffd640 20 monclient: _check_auth_rotating not needed by client.admin 2024-06-03T09:06:20.568 INFO:teuthology.orchestra.run.smithi088.stderr:2024-06-03T09:06:20.566+0000 7fdb1c416640 5 monclient: authenticate success, global_id 30160 2024-06-03T09:06:20.568 INFO:teuthology.orchestra.run.smithi088.stderr:2024-06-03T09:06:20.566+0000 7fdb1c416640 10 monclient: _renew_subs 2024-06-03T09:06:20.568 INFO:teuthology.orchestra.run.smithi088.stderr:2024-06-03T09:06:20.566+0000 7fdb1c416640 10 monclient: _send_mon_message to mon.b at v1:172.21.15.192:6789/0 2024-06-03T09:06:20.568 INFO:teuthology.orchestra.run.smithi088.stderr:2024-06-03T09:06:20.566+0000 7fdb1c416640 1 -- 172.21.15.88:0/470026592 --> v1:172.21.15.192:6789/0 -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fdb140a8820 con 0x7fdb140ad920 2024-06-03T09:06:20.568 INFO:teuthology.orchestra.run.smithi088.stderr:2024-06-03T09:06:20.566+0000 7fdb1c416640 10 client.30160.objecter _maybe_request_map subscribing (onetime) to next osd map 2024-06-03T09:06:20.568 INFO:teuthology.orchestra.run.smithi088.stderr:2024-06-03T09:06:20.566+0000 7fdb1c416640 10 monclient: _renew_subs 2024-06-03T09:06:20.568 INFO:teuthology.orchestra.run.smithi088.stderr:2024-06-03T09:06:20.566+0000 7fdb1c416640 10 monclient: _send_mon_message to mon.b at v1:172.21.15.192:6789/0 2024-06-03T09:06:20.568 INFO:teuthology.orchestra.run.smithi088.stderr:2024-06-03T09:06:20.566+0000 7fdb1c416640 1 -- 172.21.15.88:0/470026592 --> v1:172.21.15.192:6789/0 -- mon_subscribe({osdmap=0}) v3 -- 0x7fdb14143440 con 0x7fdb140ad920 2024-06-03T09:06:20.568 INFO:teuthology.orchestra.run.smithi088.stderr:2024-06-03T09:06:20.566+0000 7fdb1c416640 1 librados: init done 2024-06-03T09:06:20.568 INFO:teuthology.orchestra.run.smithi088.stderr:2024-06-03T09:06:20.567+0000 7fdb1c416640 10 start_mon_command cmd=[{"prefix": "get_command_descriptions"}] 2024-06-03T09:06:20.569 INFO:teuthology.orchestra.run.smithi088.stderr:2024-06-03T09:06:20.567+0000 7fdb1c416640 10 monclient: _send_command 1 [{"prefix": "get_command_descriptions"}] 2024-06-03T09:06:20.569 INFO:teuthology.orchestra.run.smithi088.stderr:2024-06-03T09:06:20.567+0000 7fdb1c416640 10 monclient: _send_mon_message to mon.b at v1:172.21.15.192:6789/0 2024-06-03T09:06:20.569 INFO:teuthology.orchestra.run.smithi088.stderr:2024-06-03T09:06:20.567+0000 7fdb1c416640 1 -- 172.21.15.88:0/470026592 --> v1:172.21.15.192:6789/0 -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fdb1404ea60 con 0x7fdb140ad920 2024-06-03T09:06:20.591 INFO:teuthology.orchestra.run.smithi088.stderr:2024-06-03T09:06:20.590+0000 7fdb12ffd640 1 -- 172.21.15.88:0/470026592 <== mon.1 v1:172.21.15.192:6789/0 5 ==== config(0 keys) v1 ==== 4+0+0 (unknown 0 0 0) 0x7fdb080047e0 con 0x7fdb140ad920 2024-06-03T09:06:20.591 INFO:teuthology.orchestra.run.smithi088.stderr:2024-06-03T09:06:20.591+0000 7fdb12ffd640 10 monclient: handle_config config(0 keys) v1 2024-06-03T09:06:20.591 INFO:teuthology.orchestra.run.smithi088.stderr:2024-06-03T09:06:20.591+0000 7fdb12ffd640 1 -- 172.21.15.88:0/470026592 <== mon.1 v1:172.21.15.192:6789/0 6 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 2234374065 0 0) 0x7fdb08004af0 con 0x7fdb140ad920 2024-06-03T09:06:20.591 INFO:teuthology.orchestra.run.smithi088.stderr:2024-06-03T09:06:20.591+0000 7fdb12ffd640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-06-03T09:06:20.591 INFO:teuthology.orchestra.run.smithi088.stderr:2024-06-03T09:06:20.591+0000 7fdb12ffd640 10 monclient: got monmap 1 from mon.b (according to old e1) 2024-06-03T09:06:20.591 INFO:teuthology.orchestra.run.smithi088.stderr:2024-06-03T09:06:20.591+0000 7fdb12ffd640 10 monclient: dump: 2024-06-03T09:06:20.591 INFO:teuthology.orchestra.run.smithi088.stderr:epoch 1 2024-06-03T09:06:20.591 INFO:teuthology.orchestra.run.smithi088.stderr:fsid ec3c3fdd-3ac6-4d7d-9412-683130ed8b04 2024-06-03T09:06:20.591 INFO:teuthology.orchestra.run.smithi088.stderr:last_changed 2024-06-03T08:34:31.885125+0000 2024-06-03T09:06:20.591 INFO:teuthology.orchestra.run.smithi088.stderr:created 2024-06-03T08:34:31.885125+0000 2024-06-03T09:06:20.591 INFO:teuthology.orchestra.run.smithi088.stderr:min_mon_release 18 (reef) 2024-06-03T09:06:20.591 INFO:teuthology.orchestra.run.smithi088.stderr:election_strategy: 3 2024-06-03T09:06:20.592 INFO:teuthology.orchestra.run.smithi088.stderr:0: v1:172.21.15.88:6789/0 mon.a 2024-06-03T09:06:20.592 INFO:teuthology.orchestra.run.smithi088.stderr:1: v1:172.21.15.192:6789/0 mon.b 2024-06-03T09:06:20.592 INFO:teuthology.orchestra.run.smithi088.stderr:2: v1:172.21.15.88:6790/0 mon.c 2024-06-03T09:06:20.592 INFO:teuthology.orchestra.run.smithi088.stderr: 2024-06-03T09:06:20.592 INFO:teuthology.orchestra.run.smithi088.stderr:2024-06-03T09:06:20.591+0000 7fdb189af640 4 set_mon_vals no callback set 2024-06-03T09:06:20.592 INFO:teuthology.orchestra.run.smithi088.stderr:2024-06-03T09:06:20.591+0000 7fdb12ffd640 1 -- 172.21.15.88:0/470026592 <== mon.1 v1:172.21.15.192:6789/0 7 ==== mgrmap(e 5) v1 ==== 89110+0+0 (unknown 873533973 0 0) 0x7fdb0801a870 con 0x7fdb140ad920 2024-06-03T09:06:20.592 INFO:teuthology.orchestra.run.smithi088.stderr:2024-06-03T09:06:20.592+0000 7fdb1a1b2640 10 monclient: get_auth_request con 0x7fdb0006d4f0 auth_method 0 2024-06-03T09:06:20.593 INFO:teuthology.orchestra.run.smithi088.stderr:2024-06-03T09:06:20.592+0000 7fdb12ffd640 10 client.30160.objecter ms_handle_connect 0x7fdb0006d4f0 2024-06-03T09:06:20.596 INFO:teuthology.orchestra.run.smithi088.stderr:2024-06-03T09:06:20.596+0000 7fdb12ffd640 1 -- 172.21.15.88:0/470026592 <== mon.1 v1:172.21.15.192:6789/0 8 ==== osd_map(1010..1010 src has 622..1010) v4 ==== 5229+0+0 (unknown 1105883993 0 0) 0x7fdb08054060 con 0x7fdb140ad920 2024-06-03T09:06:20.596 INFO:teuthology.orchestra.run.smithi088.stderr:2024-06-03T09:06:20.596+0000 7fdb12ffd640 10 client.30160.objecter ms_dispatch 0x7fdb14000b90 osd_map(1010..1010 src has 622..1010) v4 2024-06-03T09:06:20.596 INFO:teuthology.orchestra.run.smithi088.stderr:2024-06-03T09:06:20.596+0000 7fdb12ffd640 3 client.30160.objecter handle_osd_map got epochs [1010,1010] > 0 2024-06-03T09:06:20.596 INFO:teuthology.orchestra.run.smithi088.stderr:2024-06-03T09:06:20.596+0000 7fdb12ffd640 3 client.30160.objecter handle_osd_map decoding full epoch 1010 2024-06-03T09:06:20.596 INFO:teuthology.orchestra.run.smithi088.stderr:2024-06-03T09:06:20.596+0000 7fdb12ffd640 20 client.30160.objecter dump_active .. 0 homeless 2024-06-03T09:06:20.836 INFO:teuthology.orchestra.run.smithi088.stderr:2024-06-03T09:06:20.837+0000 7fdb12ffd640 1 -- 172.21.15.88:0/470026592 <== mon.1 v1:172.21.15.192:6789/0 9 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+182498 (unknown 1092875540 0 3016760700) 0x7fdb0804fa40 con 0x7fdb140ad920 2024-06-03T09:06:20.837 INFO:teuthology.orchestra.run.smithi088.stderr:2024-06-03T09:06:20.837+0000 7fdb12ffd640 10 monclient: handle_mon_command_ack 1 [{"prefix": "get_command_descriptions"}] 2024-06-03T09:06:20.837 INFO:teuthology.orchestra.run.smithi088.stderr:2024-06-03T09:06:20.837+0000 7fdb12ffd640 10 monclient: _finish_command 1 = system:0 2024-06-03T09:06:21.012 INFO:teuthology.orchestra.run.smithi088.stderr:2024-06-03T09:06:21.012+0000 7fdb1c416640 1 -- 172.21.15.88:0/470026592 --> v1:172.21.15.192:6800/34229 -- mgr_command(tid 0: {"prefix": "pg dump", "target": ["mon-mgr", ""], "format": "json"}) v1 -- 0x7fdb1413ce70 con 0x7fdb0006d4f0 2024-06-03T09:06:21.020 INFO:teuthology.orchestra.run.smithi088.stderr:2024-06-03T09:06:21.021+0000 7fdb12ffd640 1 -- 172.21.15.88:0/470026592 <== mgr.4110 v1:172.21.15.192:6800/34229 1 ==== mgr_command_reply(tid 0: 0 dumped all) v1 ==== 18+0+151791 (unknown 2965378022 0 3344124681) 0x7fdb04007b20 con 0x7fdb0006d4f0 2024-06-03T09:06:21.022 INFO:teuthology.orchestra.run.smithi088.stdout: 2024-06-03T09:06:21.022 INFO:teuthology.orchestra.run.smithi088.stderr:dumped all 2024-06-03T09:06:21.024 INFO:teuthology.orchestra.run.smithi088.stderr:2024-06-03T09:06:21.022+0000 7fdb1c416640 10 librados: watch_flush enter 2024-06-03T09:06:21.024 INFO:teuthology.orchestra.run.smithi088.stderr:2024-06-03T09:06:21.022+0000 7fdb1c416640 10 librados: watch_flush exit 2024-06-03T09:06:21.025 INFO:teuthology.orchestra.run.smithi088.stderr:2024-06-03T09:06:21.022+0000 7fdb1c416640 20 client.30160.objecter shutdown clearing up homeless session... 2024-06-03T09:06:21.025 INFO:teuthology.orchestra.run.smithi088.stderr:2024-06-03T09:06:21.022+0000 7fdb1c416640 10 client.30160.objecter successfully canceled tick 2024-06-03T09:06:21.025 INFO:teuthology.orchestra.run.smithi088.stderr:2024-06-03T09:06:21.022+0000 7fdb1c416640 1 -- 172.21.15.88:0/470026592 >> v1:172.21.15.192:6800/34229 conn(0x7fdb0006d4f0 legacy=0x7fdb0006f9b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-06-03T09:06:21.025 INFO:teuthology.orchestra.run.smithi088.stderr:2024-06-03T09:06:21.023+0000 7fdb1c416640 10 monclient: shutdown 2024-06-03T09:06:21.025 INFO:teuthology.orchestra.run.smithi088.stderr:2024-06-03T09:06:21.023+0000 7fdb1c416640 20 monclient: shutdown discarding 0 pending message(s) 2024-06-03T09:06:21.025 INFO:teuthology.orchestra.run.smithi088.stderr:2024-06-03T09:06:21.023+0000 7fdb1c416640 1 -- 172.21.15.88:0/470026592 >> v1:172.21.15.192:6789/0 conn(0x7fdb140ad920 legacy=0x7fdb14140f30 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-06-03T09:06:21.025 INFO:teuthology.orchestra.run.smithi088.stderr:2024-06-03T09:06:21.023+0000 7fdb1c416640 1 -- 172.21.15.88:0/470026592 shutdown_connections 2024-06-03T09:06:21.025 INFO:teuthology.orchestra.run.smithi088.stderr:2024-06-03T09:06:21.023+0000 7fdb1c416640 1 -- 172.21.15.88:0/470026592 >> 172.21.15.88:0/470026592 conn(0x7fdb140b2f20 msgr2=0x7fdb140adf80 unknown :-1 s=STATE_NONE l=0).mark_down 2024-06-03T09:06:21.025 INFO:teuthology.orchestra.run.smithi088.stderr:2024-06-03T09:06:21.023+0000 7fdb1c416640 1 -- 172.21.15.88:0/470026592 shutdown_connections 2024-06-03T09:06:21.025 INFO:teuthology.orchestra.run.smithi088.stderr:2024-06-03T09:06:21.023+0000 7fdb1c416640 1 -- 172.21.15.88:0/470026592 wait complete. 2024-06-03T09:06:21.026 INFO:teuthology.orchestra.run.smithi088.stderr:2024-06-03T09:06:21.023+0000 7fdb1c416640 1 librados: shutdown 2024-06-03T09:06:21.033 INFO:teuthology.orchestra.run.smithi088.stdout:{"pg_ready":true,"pg_map":{"version":1915,"stamp":"2024-06-03T09:06:19.098350+0000","last_osdmap_epoch":0,"last_pg_scan":0,"pg_stats_sum":{"stat_sum":{"num_bytes":459285,"num_objects":3,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":1,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":3,"num_whiteouts":0,"num_read":102,"num_read_kb":88,"num_write":120,"num_write_kb":1379,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2,"num_bytes_recovered":397840,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":83,"ondisk_log_size":83,"up":68,"acting":68,"num_store_stats":0},"osd_stats_sum":{"up_from":0,"seq":0,"num_pgs":107,"num_osds":8,"num_per_pool_osds":8,"num_per_pool_omap_osds":8,"kb":754974720,"kb_used":494448,"kb_used_data":46216,"kb_used_omap":12,"kb_used_meta":448115,"kb_avail":754480272,"statfs":{"total":773094113280,"available":772587798528,"internally_reserved":0,"allocated":47325184,"data_stored":22696207,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":12744,"internal_metadata":458870328},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":325,"apply_latency_ms":325,"commit_latency_ns":325000000,"apply_latency_ns":325000000},"alerts":[],"network_ping_times":[]},"pg_stats_delta":{"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":0,"acting":0,"num_store_stats":0,"stamp_delta":"6.767667"},"pg_stats":[{"pgid":"17.d","version":"0'0","reported_seq":940,"reported_epoch":1007,"state":"active+clean","last_fresh":"2024-06-03T09:06:11.402997+0000","last_change":"2024-06-03T08:38:19.260548+0000","last_active":"2024-06-03T09:06:11.402997+0000","last_peered":"2024-06-03T09:06:11.402997+0000","last_clean":"2024-06-03T09:06:11.402997+0000","last_became_active":"2024-06-03T08:38:19.253559+0000","last_became_peered":"2024-06-03T08:38:19.253559+0000","last_unstale":"2024-06-03T09:06:11.402997+0000","last_undegraded":"2024-06-03T09:06:11.402997+0000","last_fullsized":"2024-06-03T09:06:11.402997+0000","mapping_epoch":28,"log_start":"0'0","ondisk_log_start":"0'0","created":28,"last_epoch_clean":45,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-06-03T08:37:55.614497+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-06-03T08:37:55.614497+0000","last_clean_scrub_stamp":"2024-06-03T08:37:55.614497+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-06-04T19:00:12.470979+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1],"acting":[0,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"17.c","version":"0'0","reported_seq":952,"reported_epoch":1008,"state":"active+undersized","last_fresh":"2024-06-03T09:06:12.368416+0000","last_change":"2024-06-03T09:06:03.440074+0000","last_active":"2024-06-03T09:06:12.368416+0000","last_peered":"2024-06-03T09:06:12.368416+0000","last_clean":"2024-06-03T09:05:02.105638+0000","last_became_active":"2024-06-03T09:06:03.440074+0000","last_became_peered":"2024-06-03T09:06:03.440074+0000","last_unstale":"2024-06-03T09:06:12.368416+0000","last_undegraded":"2024-06-03T09:06:12.368416+0000","last_fullsized":"2024-06-03T09:06:03.377982+0000","mapping_epoch":999,"log_start":"0'0","ondisk_log_start":"0'0","created":28,"last_epoch_clean":255,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-06-03T08:37:55.614497+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-06-03T08:37:55.614497+0000","last_clean_scrub_stamp":"2024-06-03T08:37:55.614497+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-06-04T10:34:20.808557+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3],"acting":[3],"avail_no_missing":["3"],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"17.f","version":"0'0","reported_seq":147,"reported_epoch":1008,"state":"active+undersized","last_fresh":"2024-06-03T09:06:12.378079+0000","last_change":"2024-06-03T09:06:03.420285+0000","last_active":"2024-06-03T09:06:12.378079+0000","last_peered":"2024-06-03T09:06:12.378079+0000","last_clean":"2024-06-03T08:37:55.614497+0000","last_became_active":"2024-06-03T09:06:03.420285+0000","last_became_peered":"2024-06-03T09:06:03.420285+0000","last_unstale":"2024-06-03T09:06:12.378079+0000","last_undegraded":"2024-06-03T09:06:12.378079+0000","last_fullsized":"2024-06-03T09:06:03.380616+0000","mapping_epoch":999,"log_start":"0'0","ondisk_log_start":"0'0","created":28,"last_epoch_clean":255,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-06-03T08:37:55.614497+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-06-03T08:37:55.614497+0000","last_clean_scrub_stamp":"2024-06-03T08:37:55.614497+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-06-04T15:39:41.187731+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4],"acting":[4],"avail_no_missing":["4"],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"17.e","version":"0'0","reported_seq":149,"reported_epoch":1009,"state":"active+undersized","last_fresh":"2024-06-03T09:06:13.726992+0000","last_change":"2024-06-03T09:06:03.412887+0000","last_active":"2024-06-03T09:06:13.726992+0000","last_peered":"2024-06-03T09:06:13.726992+0000","last_clean":"2024-06-03T08:37:55.614497+0000","last_became_active":"2024-06-03T09:06:03.412887+0000","last_became_peered":"2024-06-03T09:06:03.412887+0000","last_unstale":"2024-06-03T09:06:13.726992+0000","last_undegraded":"2024-06-03T09:06:13.726992+0000","last_fullsized":"2024-06-03T09:06:03.332482+0000","mapping_epoch":999,"log_start":"0'0","ondisk_log_start":"0'0","created":28,"last_epoch_clean":255,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-06-03T08:37:55.614497+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-06-03T08:37:55.614497+0000","last_clean_scrub_stamp":"2024-06-03T08:37:55.614497+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-06-04T12:23:22.511601+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5],"acting":[5],"avail_no_missing":["5"],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"17.9","version":"0'0","reported_seq":945,"reported_epoch":1009,"state":"active+clean","last_fresh":"2024-06-03T09:06:13.928215+0000","last_change":"2024-06-03T08:38:04.772260+0000","last_active":"2024-06-03T09:06:13.928215+0000","last_peered":"2024-06-03T09:06:13.928215+0000","last_clean":"2024-06-03T09:06:13.928215+0000","last_became_active":"2024-06-03T08:38:04.763428+0000","last_became_peered":"2024-06-03T08:38:04.763428+0000","last_unstale":"2024-06-03T09:06:13.928215+0000","last_undegraded":"2024-06-03T09:06:13.928215+0000","last_fullsized":"2024-06-03T09:06:13.928215+0000","mapping_epoch":28,"log_start":"0'0","ondisk_log_start":"0'0","created":28,"last_epoch_clean":33,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-06-03T08:37:55.614497+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-06-03T08:37:55.614497+0000","last_clean_scrub_stamp":"2024-06-03T08:37:55.614497+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-06-04T10:55:10.773936+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,4],"acting":[7,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"17.8","version":"0'0","reported_seq":959,"reported_epoch":1007,"state":"active+undersized","last_fresh":"2024-06-03T09:06:12.708781+0000","last_change":"2024-06-03T09:06:03.388786+0000","last_active":"2024-06-03T09:06:12.708781+0000","last_peered":"2024-06-03T09:06:12.708781+0000","last_clean":"2024-06-03T09:04:43.297780+0000","last_became_active":"2024-06-03T09:06:03.388786+0000","last_became_peered":"2024-06-03T09:06:03.388786+0000","last_unstale":"2024-06-03T09:06:12.708781+0000","last_undegraded":"2024-06-03T09:06:12.708781+0000","last_fullsized":"2024-06-03T09:06:03.331754+0000","mapping_epoch":999,"log_start":"0'0","ondisk_log_start":"0'0","created":28,"last_epoch_clean":255,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-06-03T08:37:55.614497+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-06-03T08:37:55.614497+0000","last_clean_scrub_stamp":"2024-06-03T08:37:55.614497+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-06-04T09:18:44.765547+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0],"acting":[0],"avail_no_missing":["0"],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"17.b","version":"0'0","reported_seq":148,"reported_epoch":1008,"state":"active+undersized","last_fresh":"2024-06-03T09:06:12.873670+0000","last_change":"2024-06-03T09:06:03.451203+0000","last_active":"2024-06-03T09:06:12.873670+0000","last_peered":"2024-06-03T09:06:12.873670+0000","last_clean":"2024-06-03T08:37:55.614497+0000","last_became_active":"2024-06-03T09:06:03.451203+0000","last_became_peered":"2024-06-03T09:06:03.451203+0000","last_unstale":"2024-06-03T09:06:12.873670+0000","last_undegraded":"2024-06-03T09:06:12.873670+0000","last_fullsized":"2024-06-03T09:06:03.375523+0000","mapping_epoch":999,"log_start":"0'0","ondisk_log_start":"0'0","created":28,"last_epoch_clean":255,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-06-03T08:37:55.614497+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-06-03T08:37:55.614497+0000","last_clean_scrub_stamp":"2024-06-03T08:37:55.614497+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-06-04T10:16:19.813495+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2],"acting":[2],"avail_no_missing":["2"],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"17.a","version":"0'0","reported_seq":923,"reported_epoch":1008,"state":"active+clean","last_fresh":"2024-06-03T09:06:12.871052+0000","last_change":"2024-06-03T08:38:05.843142+0000","last_active":"2024-06-03T09:06:12.871052+0000","last_peered":"2024-06-03T09:06:12.871052+0000","last_clean":"2024-06-03T09:06:12.871052+0000","last_became_active":"2024-06-03T08:38:05.809257+0000","last_became_peered":"2024-06-03T08:38:05.809257+0000","last_unstale":"2024-06-03T09:06:12.871052+0000","last_undegraded":"2024-06-03T09:06:12.871052+0000","last_fullsized":"2024-06-03T09:06:12.871052+0000","mapping_epoch":28,"log_start":"0'0","ondisk_log_start":"0'0","created":28,"last_epoch_clean":33,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-06-03T08:37:55.614497+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-06-03T08:37:55.614497+0000","last_clean_scrub_stamp":"2024-06-03T08:37:55.614497+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-06-04T12:12:03.686838+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2],"acting":[1,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"17.5","version":"0'0","reported_seq":941,"reported_epoch":1007,"state":"active+clean","last_fresh":"2024-06-03T09:06:11.400979+0000","last_change":"2024-06-03T08:38:23.466051+0000","last_active":"2024-06-03T09:06:11.400979+0000","last_peered":"2024-06-03T09:06:11.400979+0000","last_clean":"2024-06-03T09:06:11.400979+0000","last_became_active":"2024-06-03T08:38:23.458039+0000","last_became_peered":"2024-06-03T08:38:23.458039+0000","last_unstale":"2024-06-03T09:06:11.400979+0000","last_undegraded":"2024-06-03T09:06:11.400979+0000","last_fullsized":"2024-06-03T09:06:11.400979+0000","mapping_epoch":28,"log_start":"0'0","ondisk_log_start":"0'0","created":28,"last_epoch_clean":49,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-06-03T08:37:55.614497+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-06-03T08:37:55.614497+0000","last_clean_scrub_stamp":"2024-06-03T08:37:55.614497+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-06-04T12:27:02.532436+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,4],"acting":[0,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"17.4","version":"0'0","reported_seq":148,"reported_epoch":1008,"state":"active+undersized","last_fresh":"2024-06-03T09:06:12.883827+0000","last_change":"2024-06-03T09:06:03.441889+0000","last_active":"2024-06-03T09:06:12.883827+0000","last_peered":"2024-06-03T09:06:12.883827+0000","last_clean":"2024-06-03T08:37:55.614497+0000","last_became_active":"2024-06-03T09:06:03.441889+0000","last_became_peered":"2024-06-03T09:06:03.441889+0000","last_unstale":"2024-06-03T09:06:12.883827+0000","last_undegraded":"2024-06-03T09:06:12.883827+0000","last_fullsized":"2024-06-03T09:06:03.359582+0000","mapping_epoch":999,"log_start":"0'0","ondisk_log_start":"0'0","created":28,"last_epoch_clean":255,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-06-03T08:37:55.614497+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-06-03T08:37:55.614497+0000","last_clean_scrub_stamp":"2024-06-03T08:37:55.614497+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-06-04T12:49:22.907141+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7],"acting":[7],"avail_no_missing":["7"],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"17.7","version":"0'0","reported_seq":929,"reported_epoch":1009,"state":"active+clean","last_fresh":"2024-06-03T09:06:13.736920+0000","last_change":"2024-06-03T08:38:04.837083+0000","last_active":"2024-06-03T09:06:13.736920+0000","last_peered":"2024-06-03T09:06:13.736920+0000","last_clean":"2024-06-03T09:06:13.736920+0000","last_became_active":"2024-06-03T08:38:04.818246+0000","last_became_peered":"2024-06-03T08:38:04.818246+0000","last_unstale":"2024-06-03T09:06:13.736920+0000","last_undegraded":"2024-06-03T09:06:13.736920+0000","last_fullsized":"2024-06-03T09:06:13.736920+0000","mapping_epoch":28,"log_start":"0'0","ondisk_log_start":"0'0","created":28,"last_epoch_clean":34,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-06-03T08:37:55.614497+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-06-03T08:37:55.614497+0000","last_clean_scrub_stamp":"2024-06-03T08:37:55.614497+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-06-04T17:02:38.204813+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,2],"acting":[5,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"17.6","version":"0'0","reported_seq":958,"reported_epoch":1007,"state":"active+undersized","last_fresh":"2024-06-03T09:06:12.710313+0000","last_change":"2024-06-03T09:06:03.428677+0000","last_active":"2024-06-03T09:06:12.710313+0000","last_peered":"2024-06-03T09:06:12.710313+0000","last_clean":"2024-06-03T09:04:43.312352+0000","last_became_active":"2024-06-03T09:06:03.428677+0000","last_became_peered":"2024-06-03T09:06:03.428677+0000","last_unstale":"2024-06-03T09:06:12.710313+0000","last_undegraded":"2024-06-03T09:06:12.710313+0000","last_fullsized":"2024-06-03T09:06:03.359909+0000","mapping_epoch":999,"log_start":"0'0","ondisk_log_start":"0'0","created":28,"last_epoch_clean":255,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-06-03T08:37:55.614497+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-06-03T08:37:55.614497+0000","last_clean_scrub_stamp":"2024-06-03T08:37:55.614497+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-06-04T15:38:02.180241+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0],"acting":[0],"avail_no_missing":["0"],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"2.7","version":"0'0","reported_seq":939,"reported_epoch":1008,"state":"active+clean","last_fresh":"2024-06-03T09:06:12.364540+0000","last_change":"2024-06-03T08:36:30.385405+0000","last_active":"2024-06-03T09:06:12.364540+0000","last_peered":"2024-06-03T09:06:12.364540+0000","last_clean":"2024-06-03T09:06:12.364540+0000","last_became_active":"2024-06-03T08:36:30.380302+0000","last_became_peered":"2024-06-03T08:36:30.380302+0000","last_unstale":"2024-06-03T09:06:12.364540+0000","last_undegraded":"2024-06-03T09:06:12.364540+0000","last_fullsized":"2024-06-03T09:06:12.364540+0000","mapping_epoch":23,"log_start":"0'0","ondisk_log_start":"0'0","created":23,"last_epoch_clean":24,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-06-03T08:36:28.444764+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-06-03T08:36:28.444764+0000","last_clean_scrub_stamp":"2024-06-03T08:36:28.444764+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-06-04T18:46:33.475181+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0],"acting":[3,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"17.14","version":"0'0","reported_seq":926,"reported_epoch":1009,"state":"active+clean","last_fresh":"2024-06-03T09:06:13.721460+0000","last_change":"2024-06-03T08:38:04.547565+0000","last_active":"2024-06-03T09:06:13.721460+0000","last_peered":"2024-06-03T09:06:13.721460+0000","last_clean":"2024-06-03T09:06:13.721460+0000","last_became_active":"2024-06-03T08:38:04.541781+0000","last_became_peered":"2024-06-03T08:38:04.541781+0000","last_unstale":"2024-06-03T09:06:13.721460+0000","last_undegraded":"2024-06-03T09:06:13.721460+0000","last_fullsized":"2024-06-03T09:06:13.721460+0000","mapping_epoch":28,"log_start":"0'0","ondisk_log_start":"0'0","created":28,"last_epoch_clean":34,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-06-03T08:37:55.614497+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-06-03T08:37:55.614497+0000","last_clean_scrub_stamp":"2024-06-03T08:37:55.614497+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-06-04T16:41:55.170134+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,2],"acting":[5,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"2.6","version":"0'0","reported_seq":948,"reported_epoch":1008,"state":"active+clean","last_fresh":"2024-06-03T09:06:12.385918+0000","last_change":"2024-06-03T08:36:30.430760+0000","last_active":"2024-06-03T09:06:12.385918+0000","last_peered":"2024-06-03T09:06:12.385918+0000","last_clean":"2024-06-03T09:06:12.385918+0000","last_became_active":"2024-06-03T08:36:30.404665+0000","last_became_peered":"2024-06-03T08:36:30.404665+0000","last_unstale":"2024-06-03T09:06:12.385918+0000","last_undegraded":"2024-06-03T09:06:12.385918+0000","last_fullsized":"2024-06-03T09:06:12.385918+0000","mapping_epoch":23,"log_start":"0'0","ondisk_log_start":"0'0","created":23,"last_epoch_clean":24,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-06-03T08:36:28.444764+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-06-03T08:36:28.444764+0000","last_clean_scrub_stamp":"2024-06-03T08:36:28.444764+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-06-04T10:37:37.746074+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,0],"acting":[4,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"17.15","version":"0'0","reported_seq":933,"reported_epoch":1008,"state":"active+clean","last_fresh":"2024-06-03T09:06:12.371160+0000","last_change":"2024-06-03T08:38:20.999903+0000","last_active":"2024-06-03T09:06:12.371160+0000","last_peered":"2024-06-03T09:06:12.371160+0000","last_clean":"2024-06-03T09:06:12.371160+0000","last_became_active":"2024-06-03T08:38:20.979523+0000","last_became_peered":"2024-06-03T08:38:20.979523+0000","last_unstale":"2024-06-03T09:06:12.371160+0000","last_undegraded":"2024-06-03T09:06:12.371160+0000","last_fullsized":"2024-06-03T09:06:12.371160+0000","mapping_epoch":28,"log_start":"0'0","ondisk_log_start":"0'0","created":28,"last_epoch_clean":47,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-06-03T08:37:55.614497+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-06-03T08:37:55.614497+0000","last_clean_scrub_stamp":"2024-06-03T08:37:55.614497+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-06-04T20:29:58.314289+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,1],"acting":[3,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"2.5","version":"0'0","reported_seq":940,"reported_epoch":1008,"state":"active+clean","last_fresh":"2024-06-03T09:06:12.372623+0000","last_change":"2024-06-03T08:36:30.024235+0000","last_active":"2024-06-03T09:06:12.372623+0000","last_peered":"2024-06-03T09:06:12.372623+0000","last_clean":"2024-06-03T09:06:12.372623+0000","last_became_active":"2024-06-03T08:36:30.019040+0000","last_became_peered":"2024-06-03T08:36:30.019040+0000","last_unstale":"2024-06-03T09:06:12.372623+0000","last_undegraded":"2024-06-03T09:06:12.372623+0000","last_fullsized":"2024-06-03T09:06:12.372623+0000","mapping_epoch":23,"log_start":"0'0","ondisk_log_start":"0'0","created":23,"last_epoch_clean":24,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-06-03T08:36:28.444764+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-06-03T08:36:28.444764+0000","last_clean_scrub_stamp":"2024-06-03T08:36:28.444764+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-06-04T15:46:47.109502+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,4],"acting":[3,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"17.16","version":"0'0","reported_seq":971,"reported_epoch":1008,"state":"active+undersized","last_fresh":"2024-06-03T09:06:12.882676+0000","last_change":"2024-06-03T09:06:03.478615+0000","last_active":"2024-06-03T09:06:12.882676+0000","last_peered":"2024-06-03T09:06:12.882676+0000","last_clean":"2024-06-03T09:04:43.328415+0000","last_became_active":"2024-06-03T09:06:03.478615+0000","last_became_peered":"2024-06-03T09:06:03.478615+0000","last_unstale":"2024-06-03T09:06:12.882676+0000","last_undegraded":"2024-06-03T09:06:12.882676+0000","last_fullsized":"2024-06-03T09:06:03.402504+0000","mapping_epoch":999,"log_start":"0'0","ondisk_log_start":"0'0","created":28,"last_epoch_clean":255,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-06-03T08:37:55.614497+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-06-03T08:37:55.614497+0000","last_clean_scrub_stamp":"2024-06-03T08:37:55.614497+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-06-04T08:47:08.079233+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2],"acting":[2],"avail_no_missing":["2"],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"2.4","version":"0'0","reported_seq":952,"reported_epoch":1010,"state":"active+undersized","last_fresh":"2024-06-03T09:06:18.331485+0000","last_change":"2024-06-03T09:06:03.387171+0000","last_active":"2024-06-03T09:06:18.331485+0000","last_peered":"2024-06-03T09:06:18.331485+0000","last_clean":"2024-06-03T09:04:43.202645+0000","last_became_active":"2024-06-03T09:06:03.387171+0000","last_became_peered":"2024-06-03T09:06:03.387171+0000","last_unstale":"2024-06-03T09:06:18.331485+0000","last_undegraded":"2024-06-03T09:06:18.331485+0000","last_fullsized":"2024-06-03T09:06:03.283787+0000","mapping_epoch":999,"log_start":"0'0","ondisk_log_start":"0'0","created":23,"last_epoch_clean":255,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-06-03T08:36:28.444764+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-06-03T08:36:28.444764+0000","last_clean_scrub_stamp":"2024-06-03T08:36:28.444764+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-06-04T20:17:43.485023+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5],"acting":[5],"avail_no_missing":["5"],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"17.17","version":"0'0","reported_seq":953,"reported_epoch":1008,"state":"active+clean","last_fresh":"2024-06-03T09:06:12.889836+0000","last_change":"2024-06-03T08:38:38.483275+0000","last_active":"2024-06-03T09:06:12.889836+0000","last_peered":"2024-06-03T09:06:12.889836+0000","last_clean":"2024-06-03T09:06:12.889836+0000","last_became_active":"2024-06-03T08:38:38.467916+0000","last_became_peered":"2024-06-03T08:38:38.467916+0000","last_unstale":"2024-06-03T09:06:12.889836+0000","last_undegraded":"2024-06-03T09:06:12.889836+0000","last_fullsized":"2024-06-03T09:06:12.889836+0000","mapping_epoch":28,"log_start":"0'0","ondisk_log_start":"0'0","created":28,"last_epoch_clean":61,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-06-03T08:37:55.614497+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-06-03T08:37:55.614497+0000","last_clean_scrub_stamp":"2024-06-03T08:37:55.614497+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-06-04T10:02:23.036420+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,7],"acting":[2,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"2.2","version":"0'0","reported_seq":962,"reported_epoch":1008,"state":"active+undersized","last_fresh":"2024-06-03T09:06:12.352260+0000","last_change":"2024-06-03T09:06:03.410020+0000","last_active":"2024-06-03T09:06:12.352260+0000","last_peered":"2024-06-03T09:06:12.352260+0000","last_clean":"2024-06-03T09:05:37.145889+0000","last_became_active":"2024-06-03T09:06:03.410020+0000","last_became_peered":"2024-06-03T09:06:03.410020+0000","last_unstale":"2024-06-03T09:06:12.352260+0000","last_undegraded":"2024-06-03T09:06:12.352260+0000","last_fullsized":"2024-06-03T09:06:03.346864+0000","mapping_epoch":999,"log_start":"0'0","ondisk_log_start":"0'0","created":23,"last_epoch_clean":255,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-06-03T08:36:28.444764+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-06-03T08:36:28.444764+0000","last_clean_scrub_stamp":"2024-06-03T08:36:28.444764+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-06-04T09:57:12.423360+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3],"acting":[3],"avail_no_missing":["3"],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"17.11","version":"0'0","reported_seq":943,"reported_epoch":1008,"state":"active+clean","last_fresh":"2024-06-03T09:06:12.354057+0000","last_change":"2024-06-03T08:38:06.502302+0000","last_active":"2024-06-03T09:06:12.354057+0000","last_peered":"2024-06-03T09:06:12.354057+0000","last_clean":"2024-06-03T09:06:12.354057+0000","last_became_active":"2024-06-03T08:38:06.490204+0000","last_became_peered":"2024-06-03T08:38:06.490204+0000","last_unstale":"2024-06-03T09:06:12.354057+0000","last_undegraded":"2024-06-03T09:06:12.354057+0000","last_fullsized":"2024-06-03T09:06:12.354057+0000","mapping_epoch":28,"log_start":"0'0","ondisk_log_start":"0'0","created":28,"last_epoch_clean":32,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-06-03T08:37:55.614497+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-06-03T08:37:55.614497+0000","last_clean_scrub_stamp":"2024-06-03T08:37:55.614497+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-06-04T10:47:05.988958+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,2],"acting":[4,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"2.1","version":"0'0","reported_seq":951,"reported_epoch":1008,"state":"active+clean","last_fresh":"2024-06-03T09:06:12.872310+0000","last_change":"2024-06-03T08:36:30.883360+0000","last_active":"2024-06-03T09:06:12.872310+0000","last_peered":"2024-06-03T09:06:12.872310+0000","last_clean":"2024-06-03T09:06:12.872310+0000","last_became_active":"2024-06-03T08:36:30.855073+0000","last_became_peered":"2024-06-03T08:36:30.855073+0000","last_unstale":"2024-06-03T09:06:12.872310+0000","last_undegraded":"2024-06-03T09:06:12.872310+0000","last_fullsized":"2024-06-03T09:06:12.872310+0000","mapping_epoch":23,"log_start":"0'0","ondisk_log_start":"0'0","created":23,"last_epoch_clean":25,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-06-03T08:36:28.444764+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-06-03T08:36:28.444764+0000","last_clean_scrub_stamp":"2024-06-03T08:36:28.444764+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-06-04T18:12:22.363473+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,5],"acting":[7,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"17.12","version":"0'0","reported_seq":944,"reported_epoch":1008,"state":"active+clean","last_fresh":"2024-06-03T09:06:12.878527+0000","last_change":"2024-06-03T08:38:01.751562+0000","last_active":"2024-06-03T09:06:12.878527+0000","last_peered":"2024-06-03T09:06:12.878527+0000","last_clean":"2024-06-03T09:06:12.878527+0000","last_became_active":"2024-06-03T08:38:01.745027+0000","last_became_peered":"2024-06-03T08:38:01.745027+0000","last_unstale":"2024-06-03T09:06:12.878527+0000","last_undegraded":"2024-06-03T09:06:12.878527+0000","last_fullsized":"2024-06-03T09:06:12.878527+0000","mapping_epoch":28,"log_start":"0'0","ondisk_log_start":"0'0","created":28,"last_epoch_clean":32,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-06-03T08:37:55.614497+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-06-03T08:37:55.614497+0000","last_clean_scrub_stamp":"2024-06-03T08:37:55.614497+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-06-04T17:54:43.785326+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,3],"acting":[7,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"2.0","version":"0'0","reported_seq":942,"reported_epoch":1008,"state":"active+clean","last_fresh":"2024-06-03T09:06:12.341224+0000","last_change":"2024-06-03T08:36:29.984096+0000","last_active":"2024-06-03T09:06:12.341224+0000","last_peered":"2024-06-03T09:06:12.341224+0000","last_clean":"2024-06-03T09:06:12.341224+0000","last_became_active":"2024-06-03T08:36:29.957875+0000","last_became_peered":"2024-06-03T08:36:29.957875+0000","last_unstale":"2024-06-03T09:06:12.341224+0000","last_undegraded":"2024-06-03T09:06:12.341224+0000","last_fullsized":"2024-06-03T09:06:12.341224+0000","mapping_epoch":23,"log_start":"0'0","ondisk_log_start":"0'0","created":23,"last_epoch_clean":24,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-06-03T08:36:28.444764+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-06-03T08:36:28.444764+0000","last_clean_scrub_stamp":"2024-06-03T08:36:28.444764+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-06-04T08:41:33.679479+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,5],"acting":[3,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"17.13","version":"0'0","reported_seq":942,"reported_epoch":1008,"state":"active+clean","last_fresh":"2024-06-03T09:06:12.364319+0000","last_change":"2024-06-03T08:38:26.838316+0000","last_active":"2024-06-03T09:06:12.364319+0000","last_peered":"2024-06-03T09:06:12.364319+0000","last_clean":"2024-06-03T09:06:12.364319+0000","last_became_active":"2024-06-03T08:38:26.833214+0000","last_became_peered":"2024-06-03T08:38:26.833214+0000","last_unstale":"2024-06-03T09:06:12.364319+0000","last_undegraded":"2024-06-03T09:06:12.364319+0000","last_fullsized":"2024-06-03T09:06:12.364319+0000","mapping_epoch":28,"log_start":"0'0","ondisk_log_start":"0'0","created":28,"last_epoch_clean":52,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-06-03T08:37:55.614497+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-06-03T08:37:55.614497+0000","last_clean_scrub_stamp":"2024-06-03T08:37:55.614497+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-06-04T12:58:51.617998+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,0],"acting":[4,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"2.3","version":"0'0","reported_seq":941,"reported_epoch":1008,"state":"active+clean","last_fresh":"2024-06-03T09:06:12.356955+0000","last_change":"2024-06-03T08:36:30.351670+0000","last_active":"2024-06-03T09:06:12.356955+0000","last_peered":"2024-06-03T09:06:12.356955+0000","last_clean":"2024-06-03T09:06:12.356955+0000","last_became_active":"2024-06-03T08:36:30.343560+0000","last_became_peered":"2024-06-03T08:36:30.343560+0000","last_unstale":"2024-06-03T09:06:12.356955+0000","last_undegraded":"2024-06-03T09:06:12.356955+0000","last_fullsized":"2024-06-03T09:06:12.356955+0000","mapping_epoch":23,"log_start":"0'0","ondisk_log_start":"0'0","created":23,"last_epoch_clean":24,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-06-03T08:36:28.444764+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-06-03T08:36:28.444764+0000","last_clean_scrub_stamp":"2024-06-03T08:36:28.444764+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-06-04T12:02:24.765135+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,2],"acting":[3,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.0","version":"19'76","reported_seq":1050,"reported_epoch":1008,"state":"active+clean","last_fresh":"2024-06-03T09:06:12.865199+0000","last_change":"2024-06-03T08:36:26.702702+0000","last_active":"2024-06-03T09:06:12.865199+0000","last_peered":"2024-06-03T09:06:12.865199+0000","last_clean":"2024-06-03T09:06:12.865199+0000","last_became_active":"2024-06-03T08:36:26.128378+0000","last_became_peered":"2024-06-03T08:36:26.128378+0000","last_unstale":"2024-06-03T09:06:12.865199+0000","last_undegraded":"2024-06-03T09:06:12.865199+0000","last_fullsized":"2024-06-03T09:06:12.865199+0000","mapping_epoch":20,"log_start":"0'0","ondisk_log_start":"0'0","created":16,"last_epoch_clean":22,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-06-03T08:36:15.542635+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-06-03T08:36:15.542635+0000","last_clean_scrub_stamp":"2024-06-03T08:36:15.542635+0000","objects_scrubbed":0,"log_size":76,"log_dups_size":0,"ondisk_log_size":76,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-06-04T12:41:53.731360+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":459280,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":96,"num_read_kb":82,"num_write":113,"num_write_kb":1372,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2,"num_bytes_recovered":397840,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,5],"acting":[1,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"17.10","version":"0'0","reported_seq":943,"reported_epoch":1008,"state":"active+clean","last_fresh":"2024-06-03T09:06:12.369433+0000","last_change":"2024-06-03T08:38:15.399189+0000","last_active":"2024-06-03T09:06:12.369433+0000","last_peered":"2024-06-03T09:06:12.369433+0000","last_clean":"2024-06-03T09:06:12.369433+0000","last_became_active":"2024-06-03T08:38:15.337723+0000","last_became_peered":"2024-06-03T08:38:15.337723+0000","last_unstale":"2024-06-03T09:06:12.369433+0000","last_undegraded":"2024-06-03T09:06:12.369433+0000","last_fullsized":"2024-06-03T09:06:12.369433+0000","mapping_epoch":28,"log_start":"0'0","ondisk_log_start":"0'0","created":28,"last_epoch_clean":42,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-06-03T08:37:55.614497+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-06-03T08:37:55.614497+0000","last_clean_scrub_stamp":"2024-06-03T08:37:55.614497+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-06-04T09:29:46.427317+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,3],"acting":[4,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"17.1a","version":"0'0","reported_seq":959,"reported_epoch":1008,"state":"active+undersized","last_fresh":"2024-06-03T09:06:12.360277+0000","last_change":"2024-06-03T09:06:03.390734+0000","last_active":"2024-06-03T09:06:12.360277+0000","last_peered":"2024-06-03T09:06:12.360277+0000","last_clean":"2024-06-03T09:05:20.322313+0000","last_became_active":"2024-06-03T09:06:03.390734+0000","last_became_peered":"2024-06-03T09:06:03.390734+0000","last_unstale":"2024-06-03T09:06:12.360277+0000","last_undegraded":"2024-06-03T09:06:12.360277+0000","last_fullsized":"2024-06-03T09:06:03.343197+0000","mapping_epoch":999,"log_start":"0'0","ondisk_log_start":"0'0","created":28,"last_epoch_clean":254,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-06-03T08:37:55.614497+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-06-03T08:37:55.614497+0000","last_clean_scrub_stamp":"2024-06-03T08:37:55.614497+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-06-04T15:56:47.628875+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4],"acting":[4],"avail_no_missing":["4"],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"17.1b","version":"0'0","reported_seq":934,"reported_epoch":1008,"state":"active+clean","last_fresh":"2024-06-03T09:06:12.355403+0000","last_change":"2024-06-03T08:38:03.115748+0000","last_active":"2024-06-03T09:06:12.355403+0000","last_peered":"2024-06-03T09:06:12.355403+0000","last_clean":"2024-06-03T09:06:12.355403+0000","last_became_active":"2024-06-03T08:38:03.110533+0000","last_became_peered":"2024-06-03T08:38:03.110533+0000","last_unstale":"2024-06-03T09:06:12.355403+0000","last_undegraded":"2024-06-03T09:06:12.355403+0000","last_fullsized":"2024-06-03T09:06:12.355403+0000","mapping_epoch":28,"log_start":"0'0","ondisk_log_start":"0'0","created":28,"last_epoch_clean":32,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-06-03T08:37:55.614497+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-06-03T08:37:55.614497+0000","last_clean_scrub_stamp":"2024-06-03T08:37:55.614497+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-06-04T12:36:47.471609+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,7],"acting":[3,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"17.18","version":"0'0","reported_seq":943,"reported_epoch":1008,"state":"active+clean","last_fresh":"2024-06-03T09:06:12.868791+0000","last_change":"2024-06-03T08:38:04.632864+0000","last_active":"2024-06-03T09:06:12.868791+0000","last_peered":"2024-06-03T09:06:12.868791+0000","last_clean":"2024-06-03T09:06:12.868791+0000","last_became_active":"2024-06-03T08:38:04.627856+0000","last_became_peered":"2024-06-03T08:38:04.627856+0000","last_unstale":"2024-06-03T09:06:12.868791+0000","last_undegraded":"2024-06-03T09:06:12.868791+0000","last_fullsized":"2024-06-03T09:06:12.868791+0000","mapping_epoch":28,"log_start":"0'0","ondisk_log_start":"0'0","created":28,"last_epoch_clean":33,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-06-03T08:37:55.614497+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-06-03T08:37:55.614497+0000","last_clean_scrub_stamp":"2024-06-03T08:37:55.614497+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-06-04T19:55:51.181598+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,4],"acting":[7,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"17.19","version":"0'0","reported_seq":933,"reported_epoch":1008,"state":"active+clean","last_fresh":"2024-06-03T09:06:12.336384+0000","last_change":"2024-06-03T08:38:01.839491+0000","last_active":"2024-06-03T09:06:12.336384+0000","last_peered":"2024-06-03T09:06:12.336384+0000","last_clean":"2024-06-03T09:06:12.336384+0000","last_became_active":"2024-06-03T08:38:01.829539+0000","last_became_peered":"2024-06-03T08:38:01.829539+0000","last_unstale":"2024-06-03T09:06:12.336384+0000","last_undegraded":"2024-06-03T09:06:12.336384+0000","last_fullsized":"2024-06-03T09:06:12.336384+0000","mapping_epoch":28,"log_start":"0'0","ondisk_log_start":"0'0","created":28,"last_epoch_clean":30,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-06-03T08:37:55.614497+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-06-03T08:37:55.614497+0000","last_clean_scrub_stamp":"2024-06-03T08:37:55.614497+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-06-04T09:10:43.860229+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,2],"acting":[3,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"17.1e","version":"0'0","reported_seq":147,"reported_epoch":1008,"state":"active+undersized","last_fresh":"2024-06-03T09:06:12.382089+0000","last_change":"2024-06-03T09:06:03.406181+0000","last_active":"2024-06-03T09:06:12.382089+0000","last_peered":"2024-06-03T09:06:12.382089+0000","last_clean":"2024-06-03T08:37:55.614497+0000","last_became_active":"2024-06-03T09:06:03.406181+0000","last_became_peered":"2024-06-03T09:06:03.406181+0000","last_unstale":"2024-06-03T09:06:12.382089+0000","last_undegraded":"2024-06-03T09:06:12.382089+0000","last_fullsized":"2024-06-03T09:06:03.363739+0000","mapping_epoch":999,"log_start":"0'0","ondisk_log_start":"0'0","created":28,"last_epoch_clean":255,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-06-03T08:37:55.614497+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-06-03T08:37:55.614497+0000","last_clean_scrub_stamp":"2024-06-03T08:37:55.614497+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-06-04T11:45:24.486600+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4],"acting":[4],"avail_no_missing":["4"],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"17.1f","version":"0'0","reported_seq":939,"reported_epoch":1007,"state":"active+clean","last_fresh":"2024-06-03T09:06:11.314307+0000","last_change":"2024-06-03T08:38:04.383538+0000","last_active":"2024-06-03T09:06:11.314307+0000","last_peered":"2024-06-03T09:06:11.314307+0000","last_clean":"2024-06-03T09:06:11.314307+0000","last_became_active":"2024-06-03T08:38:04.367278+0000","last_became_peered":"2024-06-03T08:38:04.367278+0000","last_unstale":"2024-06-03T09:06:11.314307+0000","last_undegraded":"2024-06-03T09:06:11.314307+0000","last_fullsized":"2024-06-03T09:06:11.314307+0000","mapping_epoch":28,"log_start":"0'0","ondisk_log_start":"0'0","created":28,"last_epoch_clean":32,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-06-03T08:37:55.614497+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-06-03T08:37:55.614497+0000","last_clean_scrub_stamp":"2024-06-03T08:37:55.614497+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-06-04T10:39:26.552282+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,2],"acting":[0,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"17.1c","version":"0'0","reported_seq":943,"reported_epoch":1009,"state":"active+undersized","last_fresh":"2024-06-03T09:06:13.719240+0000","last_change":"2024-06-03T09:06:03.382210+0000","last_active":"2024-06-03T09:06:13.719240+0000","last_peered":"2024-06-03T09:06:13.719240+0000","last_clean":"2024-06-03T09:04:43.199660+0000","last_became_active":"2024-06-03T09:06:03.382210+0000","last_became_peered":"2024-06-03T09:06:03.382210+0000","last_unstale":"2024-06-03T09:06:13.719240+0000","last_undegraded":"2024-06-03T09:06:13.719240+0000","last_fullsized":"2024-06-03T09:06:03.277874+0000","mapping_epoch":999,"log_start":"0'0","ondisk_log_start":"0'0","created":28,"last_epoch_clean":255,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-06-03T08:37:55.614497+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-06-03T08:37:55.614497+0000","last_clean_scrub_stamp":"2024-06-03T08:37:55.614497+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-06-04T12:18:50.013884+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5],"acting":[5],"avail_no_missing":["5"],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"17.1d","version":"0'0","reported_seq":922,"reported_epoch":1008,"state":"active+clean","last_fresh":"2024-06-03T09:06:12.881860+0000","last_change":"2024-06-03T08:38:06.794630+0000","last_active":"2024-06-03T09:06:12.881860+0000","last_peered":"2024-06-03T09:06:12.881860+0000","last_clean":"2024-06-03T09:06:12.881860+0000","last_became_active":"2024-06-03T08:38:06.592317+0000","last_became_peered":"2024-06-03T08:38:06.592317+0000","last_unstale":"2024-06-03T09:06:12.881860+0000","last_undegraded":"2024-06-03T09:06:12.881860+0000","last_fullsized":"2024-06-03T09:06:12.881860+0000","mapping_epoch":28,"log_start":"0'0","ondisk_log_start":"0'0","created":28,"last_epoch_clean":33,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-06-03T08:37:55.614497+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-06-03T08:37:55.614497+0000","last_clean_scrub_stamp":"2024-06-03T08:37:55.614497+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-06-04T12:33:30.785369+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,7],"acting":[1,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"17.2","version":"0'0","reported_seq":924,"reported_epoch":1008,"state":"active+clean","last_fresh":"2024-06-03T09:06:12.869501+0000","last_change":"2024-06-03T08:38:14.981410+0000","last_active":"2024-06-03T09:06:12.869501+0000","last_peered":"2024-06-03T09:06:12.869501+0000","last_clean":"2024-06-03T09:06:12.869501+0000","last_became_active":"2024-06-03T08:38:14.973825+0000","last_became_peered":"2024-06-03T08:38:14.973825+0000","last_unstale":"2024-06-03T09:06:12.869501+0000","last_undegraded":"2024-06-03T09:06:12.869501+0000","last_fullsized":"2024-06-03T09:06:12.869501+0000","mapping_epoch":28,"log_start":"0'0","ondisk_log_start":"0'0","created":28,"last_epoch_clean":41,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-06-03T08:37:55.614497+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-06-03T08:37:55.614497+0000","last_clean_scrub_stamp":"2024-06-03T08:37:55.614497+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-06-04T17:01:05.518613+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,4],"acting":[1,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"17.3","version":"0'0","reported_seq":941,"reported_epoch":1007,"state":"active+clean","last_fresh":"2024-06-03T09:06:11.338843+0000","last_change":"2024-06-03T08:38:14.619745+0000","last_active":"2024-06-03T09:06:11.338843+0000","last_peered":"2024-06-03T09:06:11.338843+0000","last_clean":"2024-06-03T09:06:11.338843+0000","last_became_active":"2024-06-03T08:38:14.510665+0000","last_became_peered":"2024-06-03T08:38:14.510665+0000","last_unstale":"2024-06-03T09:06:11.338843+0000","last_undegraded":"2024-06-03T09:06:11.338843+0000","last_fullsized":"2024-06-03T09:06:11.338843+0000","mapping_epoch":28,"log_start":"0'0","ondisk_log_start":"0'0","created":28,"last_epoch_clean":42,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-06-03T08:37:55.614497+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-06-03T08:37:55.614497+0000","last_clean_scrub_stamp":"2024-06-03T08:37:55.614497+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-06-04T12:03:47.392064+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,7],"acting":[0,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"17.0","version":"0'0","reported_seq":942,"reported_epoch":1007,"state":"active+clean","last_fresh":"2024-06-03T09:06:12.708393+0000","last_change":"2024-06-03T08:38:04.237118+0000","last_active":"2024-06-03T09:06:12.708393+0000","last_peered":"2024-06-03T09:06:12.708393+0000","last_clean":"2024-06-03T09:06:12.708393+0000","last_became_active":"2024-06-03T08:38:04.231954+0000","last_became_peered":"2024-06-03T08:38:04.231954+0000","last_unstale":"2024-06-03T09:06:12.708393+0000","last_undegraded":"2024-06-03T09:06:12.708393+0000","last_fullsized":"2024-06-03T09:06:12.708393+0000","mapping_epoch":28,"log_start":"0'0","ondisk_log_start":"0'0","created":28,"last_epoch_clean":32,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-06-03T08:37:55.614497+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-06-03T08:37:55.614497+0000","last_clean_scrub_stamp":"2024-06-03T08:37:55.614497+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-06-04T09:56:10.741637+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,2],"acting":[0,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"17.1","version":"65'7","reported_seq":959,"reported_epoch":1009,"state":"active+undersized+degraded","last_fresh":"2024-06-03T09:06:13.709209+0000","last_change":"2024-06-03T09:06:03.370660+0000","last_active":"2024-06-03T09:06:13.709209+0000","last_peered":"2024-06-03T09:06:13.709209+0000","last_clean":"2024-06-03T09:05:03.733864+0000","last_became_active":"2024-06-03T09:06:03.370660+0000","last_became_peered":"2024-06-03T09:06:03.370660+0000","last_unstale":"2024-06-03T09:06:13.709209+0000","last_undegraded":"2024-06-03T09:06:03.266076+0000","last_fullsized":"2024-06-03T09:06:03.264525+0000","mapping_epoch":999,"log_start":"0'0","ondisk_log_start":"0'0","created":28,"last_epoch_clean":255,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-06-03T08:37:55.614497+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-06-03T08:37:55.614497+0000","last_clean_scrub_stamp":"2024-06-03T08:37:55.614497+0000","objects_scrubbed":0,"log_size":7,"log_dups_size":0,"ondisk_log_size":7,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-06-04T11:55:18.194283+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":5,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":1,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":6,"num_read_kb":6,"num_write":7,"num_write_kb":7,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5],"acting":[5],"avail_no_missing":["5"],"object_location_counts":[{"shards":"5","objects":1}],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]}],"pool_stats":[{"poolid":2,"num_pg":8,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":14,"acting":14,"num_store_stats":7},{"poolid":1,"num_pg":1,"stat_sum":{"num_bytes":459280,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":96,"num_read_kb":82,"num_write":113,"num_write_kb":1372,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2,"num_bytes_recovered":397840,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":802816,"data_stored":795680,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":76,"ondisk_log_size":76,"up":2,"acting":2,"num_store_stats":2},{"poolid":17,"num_pg":32,"stat_sum":{"num_bytes":5,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":1,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":6,"num_read_kb":6,"num_write":7,"num_write_kb":7,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":8192,"data_stored":10,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":7,"ondisk_log_size":7,"up":52,"acting":52,"num_store_stats":8}],"osd_stats":[{"osd":6,"up_from":253,"seq":1086626726200,"num_pgs":32,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":54772,"kb_used_data":5608,"kb_used_omap":1,"kb_used_meta":49150,"kb_avail":94317068,"statfs":{"total":96636764160,"available":96580677632,"internally_reserved":0,"allocated":5742592,"data_stored":2767180,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1593,"internal_metadata":50330055},"hb_peers":[0,1,2,3,4,5,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":109,"apply_latency_ms":109,"commit_latency_ns":109000000,"apply_latency_ns":109000000},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Mon Jun 3 09:04:02 2024","interfaces":[{"interface":"back","average":{"1min":1582.4780000000001,"5min":332.72699999999998,"15min":124.113},"min":{"1min":4.4829999999999997,"5min":4.1959999999999997,"15min":4.1959999999999997},"max":{"1min":22766.882000000001,"5min":22766.882000000001,"15min":22766.882000000001},"last":21.917999999999999},{"interface":"front","average":{"1min":1579.5409999999999,"5min":334.94499999999999,"15min":125.84999999999999},"min":{"1min":4.8129999999999997,"5min":4.7539999999999996,"15min":4.4530000000000003},"max":{"1min":22766.289000000001,"5min":22766.289000000001,"15min":22766.289000000001},"last":25.408999999999999}]},{"osd":1,"last update":"Mon Jun 3 09:03:40 2024","interfaces":[{"interface":"back","average":{"1min":36.420999999999999,"5min":77.054000000000002,"15min":132.84800000000001},"min":{"1min":6.0380000000000003,"5min":5.8799999999999999,"15min":5.484},"max":{"1min":589.64099999999996,"5min":4138.1679999999997,"15min":9955.1299999999992},"last":26.84},{"interface":"front","average":{"1min":40.240000000000002,"5min":79.674999999999997,"15min":133.38999999999999},"min":{"1min":8.4589999999999996,"5min":8.4090000000000007,"15min":6.1390000000000002},"max":{"1min":588.60799999999995,"5min":4137.6530000000002,"15min":9827.5509999999995},"last":26.263000000000002}]},{"osd":2,"last update":"Mon Jun 3 09:03:40 2024","interfaces":[{"interface":"back","average":{"1min":12.613,"5min":24.600999999999999,"15min":21.088000000000001},"min":{"1min":6.9740000000000002,"5min":6.9740000000000002,"15min":6.5330000000000004},"max":{"1min":26.076000000000001,"5min":376.24900000000002,"15min":376.24900000000002},"last":24.899000000000001},{"interface":"front","average":{"1min":12.493,"5min":22.207999999999998,"15min":21.219000000000001},"min":{"1min":6.5810000000000004,"5min":6.5110000000000001,"15min":6.0750000000000002},"max":{"1min":29.690999999999999,"5min":271.51100000000002,"15min":434.77300000000002},"last":40.372}]},{"osd":3,"last update":"Mon Jun 3 09:03:40 2024","interfaces":[{"interface":"back","average":{"1min":14.093999999999999,"5min":24.143999999999998,"15min":29.376999999999999},"min":{"1min":10.196999999999999,"5min":6.2919999999999998,"15min":6.1719999999999997},"max":{"1min":24.832000000000001,"5min":197.84899999999999,"15min":1235.5799999999999},"last":24.353999999999999},{"interface":"front","average":{"1min":9.3260000000000005,"5min":18.463999999999999,"15min":27.631},"min":{"1min":6.125,"5min":6.125,"15min":6.125},"max":{"1min":28.166,"5min":200.173,"15min":1157.6780000000001},"last":23.454000000000001}]},{"osd":4,"last update":"Mon Jun 3 09:04:29 2024","interfaces":[{"interface":"back","average":{"1min":10.279,"5min":1253.809,"15min":450.79199999999997},"min":{"1min":7.2300000000000004,"5min":6.093,"15min":6.093},"max":{"1min":15.000999999999999,"5min":28093.096000000001,"15min":28093.096000000001},"last":101.45399999999999},{"interface":"front","average":{"1min":15.693,"5min":1279.759,"15min":463.87099999999998},"min":{"1min":10.551,"5min":7.7480000000000002,"15min":7.7480000000000002},"max":{"1min":27.898,"5min":28323.837,"15min":28323.837},"last":102.262}]},{"osd":5,"last update":"Mon Jun 3 09:04:10 2024","interfaces":[{"interface":"back","average":{"1min":13.728999999999999,"5min":24.670000000000002,"15min":23.815999999999999},"min":{"1min":10.401,"5min":7.516,"15min":7.516},"max":{"1min":25.138000000000002,"5min":320.85000000000002,"15min":445.041},"last":103.52500000000001},{"interface":"front","average":{"1min":13.847,"5min":29.283000000000001,"15min":23.393999999999998},"min":{"1min":7.944,"5min":6.266,"15min":6.266},"max":{"1min":18.817,"5min":530.56700000000001,"15min":530.56700000000001},"last":29.263999999999999}]},{"osd":7,"last update":"Mon Jun 3 09:03:40 2024","interfaces":[{"interface":"back","average":{"1min":20.568999999999999,"5min":30.286999999999999,"15min":27.977},"min":{"1min":13.874000000000001,"5min":10.58,"15min":9.2859999999999996},"max":{"1min":86.959000000000003,"5min":222.07599999999999,"15min":255.08799999999999},"last":37.618000000000002},{"interface":"front","average":{"1min":19.102,"5min":29.550000000000001,"15min":27.763000000000002},"min":{"1min":10.052,"5min":8.6709999999999994,"15min":8.5429999999999993},"max":{"1min":67.968999999999994,"5min":293.62099999999998,"15min":438.14600000000002},"last":111.985}]}]},{"osd":7,"up_from":20,"seq":85899346284,"num_pgs":9,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":58952,"kb_used_data":5692,"kb_used_omap":1,"kb_used_meta":53246,"kb_avail":94312888,"statfs":{"total":96636764160,"available":96576397312,"internally_reserved":0,"allocated":5828608,"data_stored":2735298,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1589,"internal_metadata":54524363},"hb_peers":[0,1,2,3,4,5],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":55,"apply_latency_ms":55,"commit_latency_ns":55000000,"apply_latency_ns":55000000},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Mon Jun 3 09:06:09 2024","interfaces":[{"interface":"back","average":{"1min":14.073,"5min":894.22400000000005,"15min":308.93400000000003},"min":{"1min":5.5709999999999997,"5min":4.218,"15min":4.0599999999999996},"max":{"1min":90.757999999999996,"5min":21729.001,"15min":21729.001},"last":11.128},{"interface":"front","average":{"1min":9.2550000000000008,"5min":906.23500000000001,"15min":314.06},"min":{"1min":5.5789999999999997,"5min":4.8810000000000002,"15min":4.8810000000000002},"max":{"1min":25.937000000000001,"5min":21899.732,"15min":21899.732},"last":16.053999999999998}]},{"osd":1,"last update":"Mon Jun 3 09:06:09 2024","interfaces":[{"interface":"back","average":{"1min":15.182,"5min":69.183999999999997,"15min":219.173},"min":{"1min":7.3639999999999999,"5min":6.6200000000000001,"15min":6.1319999999999997},"max":{"1min":76.926000000000002,"5min":2795.1329999999998,"15min":18373.257000000001},"last":8.6799999999999997},{"interface":"front","average":{"1min":21.495999999999999,"5min":74.703999999999994,"15min":225.285},"min":{"1min":8.8399999999999999,"5min":8.8399999999999999,"15min":8.5129999999999999},"max":{"1min":94.977999999999994,"5min":2803.3670000000002,"15min":18374.601999999999},"last":13.189}]},{"osd":2,"last update":"Mon Jun 3 09:06:09 2024","interfaces":[{"interface":"back","average":{"1min":19.164000000000001,"5min":26.902000000000001,"15min":22.920999999999999},"min":{"1min":11.207000000000001,"5min":8.5030000000000001,"15min":8.5030000000000001},"max":{"1min":91.319999999999993,"5min":333.99200000000002,"15min":333.99200000000002},"last":18.968},{"interface":"front","average":{"1min":11.125999999999999,"5min":18.132000000000001,"15min":16.353000000000002},"min":{"1min":6.202,"5min":5.8369999999999997,"15min":5.5069999999999997},"max":{"1min":28.649000000000001,"5min":226.18100000000001,"15min":226.18100000000001},"last":13.875}]},{"osd":3,"last update":"Mon Jun 3 09:06:09 2024","interfaces":[{"interface":"back","average":{"1min":22.291,"5min":26.289999999999999,"15min":31.074999999999999},"min":{"1min":11.714,"5min":6.8959999999999999,"15min":6.3760000000000003},"max":{"1min":83.102000000000004,"5min":316.64400000000001,"15min":2039.9059999999999},"last":15.42},{"interface":"front","average":{"1min":13.324999999999999,"5min":17.963000000000001,"15min":25.969999999999999},"min":{"1min":6.1200000000000001,"5min":5.2089999999999996,"15min":5.2089999999999996},"max":{"1min":49.908999999999999,"5min":233.542,"15min":2037.5550000000001},"last":7.6879999999999997}]},{"osd":4,"last update":"Mon Jun 3 09:05:28 2024","interfaces":[{"interface":"back","average":{"1min":22.209,"5min":1379.51,"15min":1803.278},"min":{"1min":5.9089999999999998,"5min":5.9089999999999998,"15min":5.9089999999999998},"max":{"1min":224.54300000000001,"5min":30073.917000000001,"15min":62560.868000000002},"last":10.279},{"interface":"front","average":{"1min":22.712,"5min":1401.0719999999999,"15min":1806.759},"min":{"1min":7.6449999999999996,"5min":6.7930000000000001,"15min":5.9340000000000002},"max":{"1min":223.58600000000001,"5min":30348.021000000001,"15min":62451.446000000004},"last":16.579000000000001}]},{"osd":5,"last update":"Mon Jun 3 09:06:02 2024","interfaces":[{"interface":"back","average":{"1min":24.372,"5min":26.856000000000002,"15min":23.571999999999999},"min":{"1min":12.278,"5min":9.2460000000000004,"15min":6.5650000000000004},"max":{"1min":183.64099999999999,"5min":317.572,"15min":317.572},"last":10.933999999999999},{"interface":"front","average":{"1min":21.259,"5min":23.364000000000001,"15min":21.193999999999999},"min":{"1min":7.8499999999999996,"5min":7.0999999999999996,"15min":7.008},"max":{"1min":184.53399999999999,"5min":306.60599999999999,"15min":306.60599999999999},"last":18.036000000000001}]},{"osd":6,"last update":"Mon Jun 3 09:04:31 2024","interfaces":[{"interface":"back","average":{"1min":44.061,"5min":26.821000000000002,"15min":91.176000000000002},"min":{"1min":9.0800000000000001,"5min":7.9390000000000001,"15min":7.9390000000000001},"max":{"1min":268.029,"5min":308.916,"15min":8734.4639999999999},"last":6.6769999999999996},{"interface":"front","average":{"1min":46.643000000000001,"5min":23.879999999999999,"15min":88.765000000000001},"min":{"1min":8.8119999999999994,"5min":6.9240000000000004,"15min":6.742},"max":{"1min":305.64600000000002,"5min":305.64600000000002,"15min":8735.5319999999992},"last":12.327}]}]},{"osd":5,"up_from":20,"seq":85899346259,"num_pgs":9,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":59360,"kb_used_data":6100,"kb_used_omap":1,"kb_used_meta":53246,"kb_avail":94312480,"statfs":{"total":96636764160,"available":96575979520,"internally_reserved":0,"allocated":6246400,"data_stored":3138572,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1594,"internal_metadata":54524358},"hb_peers":[0,1,2,3,4,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":5,"apply_latency_ms":5,"commit_latency_ns":5000000,"apply_latency_ns":5000000},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Mon Jun 3 09:06:06 2024","interfaces":[{"interface":"back","average":{"1min":7.468,"5min":1703.037,"15min":580.51900000000001},"min":{"1min":4.6509999999999998,"5min":4.6509999999999998,"15min":4.4020000000000001},"max":{"1min":14.711,"5min":26433.764999999999,"15min":26433.764999999999},"last":6.1859999999999999},{"interface":"front","average":{"1min":9.9429999999999996,"5min":1725.27,"15min":587.87800000000004},"min":{"1min":5.5999999999999996,"5min":5.1319999999999997,"15min":4.8170000000000002},"max":{"1min":13.590999999999999,"5min":26704.698,"15min":26704.698},"last":7.6970000000000001}]},{"osd":1,"last update":"Mon Jun 3 09:06:00 2024","interfaces":[{"interface":"back","average":{"1min":10.862,"5min":67.731999999999999,"15min":217.364},"min":{"1min":7.1769999999999996,"5min":6.0890000000000004,"15min":5.4930000000000003},"max":{"1min":16.998999999999999,"5min":3923.4090000000001,"15min":18811.696},"last":12.917999999999999},{"interface":"front","average":{"1min":10.417,"5min":65.141999999999996,"15min":220.75700000000001},"min":{"1min":5.3209999999999997,"5min":5.3209999999999997,"15min":5.0209999999999999},"max":{"1min":15.821999999999999,"5min":4019.6129999999998,"15min":18968.365000000002},"last":8.6590000000000007}]},{"osd":2,"last update":"Mon Jun 3 09:06:00 2024","interfaces":[{"interface":"back","average":{"1min":11.366,"5min":24.18,"15min":21.137},"min":{"1min":5.6369999999999996,"5min":5.6369999999999996,"15min":5.0510000000000002},"max":{"1min":15.529999999999999,"5min":332.48200000000003,"15min":332.48200000000003},"last":14.346},{"interface":"front","average":{"1min":10.089,"5min":23.117000000000001,"15min":24.050000000000001},"min":{"1min":6.6769999999999996,"5min":6.6769999999999996,"15min":5.8739999999999997},"max":{"1min":15.821,"5min":353.03199999999998,"15min":353.03199999999998},"last":8.2629999999999999}]},{"osd":3,"last update":"Mon Jun 3 09:06:00 2024","interfaces":[{"interface":"back","average":{"1min":13.472,"5min":24.283999999999999,"15min":31.666},"min":{"1min":10.111000000000001,"5min":7.4720000000000004,"15min":7.4210000000000003},"max":{"1min":16.271999999999998,"5min":353.57499999999999,"15min":2855.8519999999999},"last":13.637},{"interface":"front","average":{"1min":9.5899999999999999,"5min":15.875,"15min":25.834},"min":{"1min":6.6609999999999996,"5min":5.7729999999999997,"15min":5.1909999999999998},"max":{"1min":15.112,"5min":203.804,"15min":2869.7860000000001},"last":10.704000000000001}]},{"osd":4,"last update":"Mon Jun 3 09:05:27 2024","interfaces":[{"interface":"back","average":{"1min":19.155000000000001,"5min":1084.961,"15min":1346.9949999999999},"min":{"1min":7.6130000000000004,"5min":7.6130000000000004,"15min":6.1340000000000003},"max":{"1min":156.55199999999999,"5min":28619.825000000001,"15min":57999.453999999998},"last":10.016},{"interface":"front","average":{"1min":28.068999999999999,"5min":1087.106,"15min":1330.289},"min":{"1min":8.3949999999999996,"5min":6.5030000000000001,"15min":6.5030000000000001},"max":{"1min":254.91300000000001,"5min":28621.294000000002,"15min":57434.631999999998},"last":15.166}]},{"osd":6,"last update":"Mon Jun 3 09:03:57 2024","interfaces":[{"interface":"back","average":{"1min":23.536000000000001,"5min":36.457000000000001,"15min":83.799999999999997},"min":{"1min":9.4800000000000004,"5min":9.4800000000000004,"15min":8.0630000000000006},"max":{"1min":89.442999999999998,"5min":508.77999999999997,"15min":9193.5169999999998},"last":157.79300000000001},{"interface":"front","average":{"1min":18.891999999999999,"5min":22.100999999999999,"15min":75.489999999999995},"min":{"1min":5.923,"5min":5.923,"15min":5.923},"max":{"1min":78.424999999999997,"5min":464.62,"15min":9191.4429999999993},"last":148.09}]},{"osd":7,"last update":"Mon Jun 3 09:06:00 2024","interfaces":[{"interface":"back","average":{"1min":13.257,"5min":20.917999999999999,"15min":21.088999999999999},"min":{"1min":10.475,"5min":6.7629999999999999,"15min":6.7629999999999999},"max":{"1min":17.245000000000001,"5min":211.22999999999999,"15min":211.22999999999999},"last":11.826000000000001},{"interface":"front","average":{"1min":12.872999999999999,"5min":20.587,"15min":24.021000000000001},"min":{"1min":8.5760000000000005,"5min":7.7709999999999999,"15min":6.8570000000000002},"max":{"1min":18.359999999999999,"5min":223.191,"15min":286.69499999999999},"last":11.209}]}]},{"osd":4,"up_from":20,"seq":85899346266,"num_pgs":12,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":63052,"kb_used_data":5692,"kb_used_omap":1,"kb_used_meta":57342,"kb_avail":94308788,"statfs":{"total":96636764160,"available":96572198912,"internally_reserved":0,"allocated":5828608,"data_stored":2735298,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1594,"internal_metadata":58718662},"hb_peers":[0,1,2,3,5,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":51,"apply_latency_ms":51,"commit_latency_ns":51000000,"apply_latency_ns":51000000},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Mon Jun 3 09:05:34 2024","interfaces":[{"interface":"back","average":{"1min":12.416,"5min":1099.4649999999999,"15min":382.15199999999999},"min":{"1min":4.5279999999999996,"5min":4.5279999999999996,"15min":4.3220000000000001},"max":{"1min":123.878,"5min":21726.743999999999,"15min":21726.743999999999},"last":5.6609999999999996},{"interface":"front","average":{"1min":16.462,"5min":1108.894,"15min":387.70299999999997},"min":{"1min":6.4089999999999998,"5min":5.8380000000000001,"15min":5.4909999999999997},"max":{"1min":130.185,"5min":21727.993999999999,"15min":21727.993999999999},"last":6.1950000000000003}]},{"osd":1,"last update":"Mon Jun 3 09:05:34 2024","interfaces":[{"interface":"back","average":{"1min":14.727,"5min":56.755000000000003,"15min":341.09399999999999},"min":{"1min":4.6529999999999996,"5min":4.1970000000000001,"15min":4.1970000000000001},"max":{"1min":126.899,"5min":3346.4389999999999,"15min":22256.098000000002},"last":5.0839999999999996},{"interface":"front","average":{"1min":13.472,"5min":57.503,"15min":340.303},"min":{"1min":5.0590000000000002,"5min":5.0590000000000002,"15min":4.2640000000000002},"max":{"1min":126.096,"5min":3347.181,"15min":22247.317999999999},"last":10.532999999999999}]},{"osd":2,"last update":"Mon Jun 3 09:05:34 2024","interfaces":[{"interface":"back","average":{"1min":26.088999999999999,"5min":23.866,"15min":20.620999999999999},"min":{"1min":7.0869999999999997,"5min":6.8239999999999998,"15min":6.3150000000000004},"max":{"1min":337.60399999999998,"5min":393.58600000000001,"15min":393.58600000000001},"last":9.5120000000000005},{"interface":"front","average":{"1min":15.800000000000001,"5min":22.584,"15min":23.245999999999999},"min":{"1min":6.0629999999999997,"5min":6.0629999999999997,"15min":6.0629999999999997},"max":{"1min":127.913,"5min":394.471,"15min":394.471},"last":12.246}]},{"osd":3,"last update":"Mon Jun 3 09:05:34 2024","interfaces":[{"interface":"back","average":{"1min":17.544,"5min":19.143999999999998,"15min":18.742000000000001},"min":{"1min":5.8719999999999999,"5min":5.4100000000000001,"15min":5.4000000000000004},"max":{"1min":127.43600000000001,"5min":322.63,"15min":322.63},"last":13.051},{"interface":"front","average":{"1min":21.327000000000002,"5min":23.559000000000001,"15min":23.797000000000001},"min":{"1min":10.023,"5min":7.484,"15min":7.484},"max":{"1min":204.517,"5min":392.67500000000001,"15min":392.67500000000001},"last":11.057}]},{"osd":5,"last update":"Mon Jun 3 09:05:34 2024","interfaces":[{"interface":"back","average":{"1min":24.274999999999999,"5min":21.274999999999999,"15min":20.988},"min":{"1min":6.7489999999999997,"5min":6.7489999999999997,"15min":6.7489999999999997},"max":{"1min":268.10199999999998,"5min":344.23399999999998,"15min":344.23399999999998},"last":13.826000000000001},{"interface":"front","average":{"1min":26.361999999999998,"5min":25.218,"15min":26.084},"min":{"1min":12.228,"5min":8.5180000000000007,"15min":8.5180000000000007},"max":{"1min":264.75,"5min":345.18700000000001,"15min":345.18700000000001},"last":14.75}]},{"osd":6,"last update":"Mon Jun 3 09:04:32 2024","interfaces":[{"interface":"back","average":{"1min":15.548,"5min":32.957999999999998,"15min":152.738},"min":{"1min":9.141,"5min":9.141,"15min":8.2319999999999993},"max":{"1min":52.57,"5min":395.238,"15min":12496.335999999999},"last":208.90000000000001},{"interface":"front","average":{"1min":12.680999999999999,"5min":25.428000000000001,"15min":149.30600000000001},"min":{"1min":6.4379999999999997,"5min":5.6319999999999997,"15min":5.6319999999999997},"max":{"1min":46.283000000000001,"5min":377.74299999999999,"15min":12616.297},"last":205.88900000000001}]},{"osd":7,"last update":"Mon Jun 3 09:05:34 2024","interfaces":[{"interface":"back","average":{"1min":18.295999999999999,"5min":19.585999999999999,"15min":21.062000000000001},"min":{"1min":6.2110000000000003,"5min":6.2110000000000003,"15min":5.8010000000000002},"max":{"1min":207.25299999999999,"5min":358.94900000000001,"15min":358.94900000000001},"last":7.0279999999999996},{"interface":"front","average":{"1min":23.712,"5min":24.024999999999999,"15min":24.640000000000001},"min":{"1min":11.824999999999999,"5min":7.7789999999999999,"15min":7.7789999999999999},"max":{"1min":209.32900000000001,"5min":346.25200000000001,"15min":380.06900000000002},"last":15.519}]}]},{"osd":2,"up_from":20,"seq":85899346286,"num_pgs":13,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":58952,"kb_used_data":5692,"kb_used_omap":1,"kb_used_meta":53246,"kb_avail":94312888,"statfs":{"total":96636764160,"available":96576397312,"internally_reserved":0,"allocated":5828608,"data_stored":2735298,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1594,"internal_metadata":54524358},"hb_peers":[0,1,3,4,5,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":16,"apply_latency_ms":16,"commit_latency_ns":16000000,"apply_latency_ns":16000000},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Mon Jun 3 09:05:21 2024","interfaces":[{"interface":"back","average":{"1min":16.364000000000001,"5min":947.59799999999996,"15min":326.58999999999997},"min":{"1min":6.0830000000000002,"5min":4.7050000000000001,"15min":4.1420000000000003},"max":{"1min":157.60900000000001,"5min":23465.18,"15min":23465.18},"last":26.015000000000001},{"interface":"front","average":{"1min":12.276,"5min":945.29899999999998,"15min":326.78399999999999},"min":{"1min":5.1989999999999998,"5min":4.7439999999999998,"15min":4.6950000000000003},"max":{"1min":102.557,"5min":23465.966,"15min":23465.966},"last":20.802}]},{"osd":1,"last update":"Mon Jun 3 09:05:21 2024","interfaces":[{"interface":"back","average":{"1min":17.803999999999998,"5min":77.658000000000001,"15min":294.96899999999999},"min":{"1min":7.2130000000000001,"5min":5.4199999999999999,"15min":5.4199999999999999},"max":{"1min":159.624,"5min":4220.5290000000005,"15min":18954.987000000001},"last":85.430000000000007},{"interface":"front","average":{"1min":21.542999999999999,"5min":83.528999999999996,"15min":303.11700000000002},"min":{"1min":8.8200000000000003,"5min":7.5949999999999998,"15min":7.5949999999999998},"max":{"1min":170.74299999999999,"5min":4325.9489999999996,"15min":19143.834999999999},"last":98.498000000000005}]},{"osd":3,"last update":"Mon Jun 3 09:05:21 2024","interfaces":[{"interface":"back","average":{"1min":19.164999999999999,"5min":13.284000000000001,"15min":28.751999999999999},"min":{"1min":7.3710000000000004,"5min":5.9290000000000003,"15min":5.2450000000000001},"max":{"1min":118.913,"5min":118.913,"15min":2594.8150000000001},"last":28.164000000000001},{"interface":"front","average":{"1min":22.094000000000001,"5min":17.244,"15min":32.173999999999999},"min":{"1min":9.516,"5min":8.0760000000000005,"15min":6.0780000000000003},"max":{"1min":156.34700000000001,"5min":156.34700000000001,"15min":2706.5120000000002},"last":32.360999999999997}]},{"osd":4,"last update":"Mon Jun 3 09:05:57 2024","interfaces":[{"interface":"back","average":{"1min":10.573,"5min":1266.894,"15min":438.154},"min":{"1min":8.6379999999999999,"5min":7.8789999999999996,"15min":6.2039999999999997},"max":{"1min":15.105,"5min":26660.861000000001,"15min":26660.861000000001},"last":30.163},{"interface":"front","average":{"1min":12.407999999999999,"5min":1266.7460000000001,"15min":435.75400000000002},"min":{"1min":10.266999999999999,"5min":10.266999999999999,"15min":6.2720000000000002},"max":{"1min":15.228,"5min":26616.243999999999,"15min":26616.243999999999},"last":33.402999999999999}]},{"osd":5,"last update":"Mon Jun 3 09:05:21 2024","interfaces":[{"interface":"back","average":{"1min":25.632000000000001,"5min":20.437999999999999,"15min":21.472999999999999},"min":{"1min":9.2159999999999993,"5min":9.2159999999999993,"15min":6.2050000000000001},"max":{"1min":252.31100000000001,"5min":252.31100000000001,"15min":252.31100000000001},"last":32.890000000000001},{"interface":"front","average":{"1min":24.824000000000002,"5min":18.521000000000001,"15min":21.027000000000001},"min":{"1min":8.0449999999999999,"5min":7.4850000000000003,"15min":6.8810000000000002},"max":{"1min":258.31700000000001,"5min":258.31700000000001,"15min":258.31700000000001},"last":29.228999999999999}]},{"osd":6,"last update":"Mon Jun 3 09:04:28 2024","interfaces":[{"interface":"back","average":{"1min":14.063000000000001,"5min":24.850000000000001,"15min":106.755},"min":{"1min":9.0820000000000007,"5min":8.9649999999999999,"15min":8.9649999999999999},"max":{"1min":24.666,"5min":188.06700000000001,"15min":9521.0010000000002},"last":183.60900000000001},{"interface":"front","average":{"1min":13.933999999999999,"5min":21.081,"15min":106.142},"min":{"1min":5.8719999999999999,"5min":5.8719999999999999,"15min":5.8719999999999999},"max":{"1min":117.551,"5min":193.887,"15min":9585.6620000000003},"last":184.20599999999999}]},{"osd":7,"last update":"Mon Jun 3 09:05:21 2024","interfaces":[{"interface":"back","average":{"1min":25.097000000000001,"5min":22.579999999999998,"15min":24.388000000000002},"min":{"1min":7.8959999999999999,"5min":6.6130000000000004,"15min":6.6130000000000004},"max":{"1min":242.56800000000001,"5min":242.56800000000001,"15min":242.56800000000001},"last":88.974999999999994},{"interface":"front","average":{"1min":24.414999999999999,"5min":23.995999999999999,"15min":26.614999999999998},"min":{"1min":12.43,"5min":9.0660000000000007,"15min":9.0660000000000007},"max":{"1min":181.56399999999999,"5min":182.5,"15min":214.33799999999999},"last":102.229}]}]},{"osd":3,"up_from":14,"seq":60129542500,"num_pgs":11,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":58940,"kb_used_data":5680,"kb_used_omap":1,"kb_used_meta":53246,"kb_avail":94312900,"statfs":{"total":96636764160,"available":96576409600,"internally_reserved":0,"allocated":5816320,"data_stored":2728907,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1592,"internal_metadata":54524360},"hb_peers":[0,1,2,4,5,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":20,"apply_latency_ms":20,"commit_latency_ns":20000000,"apply_latency_ns":20000000},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Mon Jun 3 09:05:25 2024","interfaces":[{"interface":"back","average":{"1min":8.4809999999999999,"5min":1064.2270000000001,"15min":367.88600000000002},"min":{"1min":6.0529999999999999,"5min":5.173,"15min":4.6929999999999996},"max":{"1min":12.010999999999999,"5min":23275.167000000001,"15min":23275.167000000001},"last":8.0489999999999995},{"interface":"front","average":{"1min":13.885,"5min":1070.3710000000001,"15min":373.53500000000003},"min":{"1min":9.6679999999999993,"5min":7.2089999999999996,"15min":5.9880000000000004},"max":{"1min":32.426000000000002,"5min":23278.536,"15min":23278.536},"last":10.086}]},{"osd":1,"last update":"Mon Jun 3 09:05:25 2024","interfaces":[{"interface":"back","average":{"1min":11.324,"5min":19.920000000000002,"15min":390.07900000000001},"min":{"1min":6.6379999999999999,"5min":4.8200000000000003,"15min":4.8200000000000003},"max":{"1min":26.449000000000002,"5min":250.72300000000001,"15min":21658.205999999998},"last":4.335},{"interface":"front","average":{"1min":9.468,"5min":20.966999999999999,"15min":381.18799999999999},"min":{"1min":5.4969999999999999,"5min":5.2779999999999996,"15min":5.2779999999999996},"max":{"1min":25.643000000000001,"5min":335.33699999999999,"15min":21307.069},"last":7.3360000000000003}]},{"osd":2,"last update":"Mon Jun 3 09:05:25 2024","interfaces":[{"interface":"back","average":{"1min":13.215,"5min":26.649000000000001,"15min":23.783999999999999},"min":{"1min":7.8710000000000004,"5min":6.4900000000000002,"15min":5.875},"max":{"1min":34.322000000000003,"5min":377.803,"15min":377.803},"last":11.356999999999999},{"interface":"front","average":{"1min":13.105,"5min":22.292999999999999,"15min":20.573},"min":{"1min":7.327,"5min":6.125,"15min":5.9020000000000001},"max":{"1min":32.756999999999998,"5min":371.601,"15min":371.601},"last":8.9670000000000005}]},{"osd":4,"last update":"Mon Jun 3 09:05:27 2024","interfaces":[{"interface":"back","average":{"1min":10.444000000000001,"5min":1211.4970000000001,"15min":1639.1579999999999},"min":{"1min":6.4429999999999996,"5min":6.2560000000000002,"15min":6.0309999999999997},"max":{"1min":21.263000000000002,"5min":30144.939999999999,"15min":62679.622000000003},"last":14.975},{"interface":"front","average":{"1min":11.363,"5min":1219.6400000000001,"15min":1644.127},"min":{"1min":7.8109999999999999,"5min":7.8109999999999999,"15min":6.2439999999999998},"max":{"1min":20.108000000000001,"5min":30180.689999999999,"15min":62681.021999999997},"last":14.266}]},{"osd":5,"last update":"Mon Jun 3 09:06:07 2024","interfaces":[{"interface":"back","average":{"1min":12.802,"5min":19.640999999999998,"15min":22.318000000000001},"min":{"1min":9.0890000000000004,"5min":6.8810000000000002,"15min":6.7690000000000001},"max":{"1min":16.745999999999999,"5min":249.88,"15min":327.34899999999999},"last":11.952},{"interface":"front","average":{"1min":15.002000000000001,"5min":24.003,"15min":26.649999999999999},"min":{"1min":11.984,"5min":9.9149999999999991,"15min":8.9879999999999995},"max":{"1min":18.407,"5min":251.47800000000001,"15min":331.86799999999999},"last":13.717000000000001}]},{"osd":6,"last update":"Mon Jun 3 09:04:04 2024","interfaces":[{"interface":"back","average":{"1min":22.815999999999999,"5min":31.513999999999999,"15min":181.74299999999999},"min":{"1min":9.4809999999999999,"5min":7.2240000000000002,"15min":7.2240000000000002},"max":{"1min":160.56999999999999,"5min":380.613,"15min":11893.74},"last":19.359999999999999},{"interface":"front","average":{"1min":25.359999999999999,"5min":30.981000000000002,"15min":179.81399999999999},"min":{"1min":9.8870000000000005,"5min":7.3360000000000003,"15min":6.0899999999999999},"max":{"1min":193.11099999999999,"5min":379.80700000000002,"15min":11893.069},"last":8.202}]},{"osd":7,"last update":"Mon Jun 3 09:05:25 2024","interfaces":[{"interface":"back","average":{"1min":12.177,"5min":21.126000000000001,"15min":22.972000000000001},"min":{"1min":8.6769999999999996,"5min":7.4279999999999999,"15min":7.4279999999999999},"max":{"1min":17.376999999999999,"5min":164.27500000000001,"15min":178.76300000000001},"last":9.5540000000000003},{"interface":"front","average":{"1min":15.675000000000001,"5min":22.722000000000001,"15min":25.238},"min":{"1min":11.233000000000001,"5min":10.555999999999999,"15min":7.6440000000000001},"max":{"1min":29.280000000000001,"5min":148.13800000000001,"15min":417.22800000000001},"last":12.855}]}]},{"osd":1,"up_from":14,"seq":60129542498,"num_pgs":9,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":59332,"kb_used_data":6072,"kb_used_omap":1,"kb_used_meta":53246,"kb_avail":94312508,"statfs":{"total":96636764160,"available":96576008192,"internally_reserved":0,"allocated":6217728,"data_stored":3126747,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":54524357},"hb_peers":[0,2,3,4,5,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":65,"apply_latency_ms":65,"commit_latency_ns":65000000,"apply_latency_ns":65000000},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Mon Jun 3 09:06:07 2024","interfaces":[{"interface":"back","average":{"1min":7.7530000000000001,"5min":185.66999999999999,"15min":77.347999999999999},"min":{"1min":5.0229999999999997,"5min":4.9279999999999999,"15min":4.5449999999999999},"max":{"1min":11.896000000000001,"5min":7470.616,"15min":7470.616},"last":10.471},{"interface":"front","average":{"1min":9.9220000000000006,"5min":187.14599999999999,"15min":76.325999999999993},"min":{"1min":5.3159999999999998,"5min":5.3159999999999998,"15min":5.3159999999999998},"max":{"1min":18.338000000000001,"5min":7477.0829999999996,"15min":7477.0829999999996},"last":5.7300000000000004}]},{"osd":2,"last update":"Mon Jun 3 09:05:18 2024","interfaces":[{"interface":"back","average":{"1min":10.954000000000001,"5min":18.295000000000002,"15min":24.126999999999999},"min":{"1min":6.8739999999999997,"5min":5.8650000000000002,"15min":5.8650000000000002},"max":{"1min":21.122,"5min":228,"15min":657.38},"last":8.5600000000000005},{"interface":"front","average":{"1min":13.694000000000001,"5min":24.446999999999999,"15min":32.055999999999997},"min":{"1min":9.9860000000000007,"5min":9.1229999999999993,"15min":7.2160000000000002},"max":{"1min":26.507999999999999,"5min":229.465,"15min":700.06500000000005},"last":12.885999999999999}]},{"osd":3,"last update":"Mon Jun 3 09:06:04 2024","interfaces":[{"interface":"back","average":{"1min":8.6110000000000007,"5min":15.733000000000001,"15min":26.646999999999998},"min":{"1min":5.827,"5min":5.4640000000000004,"15min":5.4640000000000004},"max":{"1min":13.105,"5min":121.619,"15min":1676.009},"last":6.3609999999999998},{"interface":"front","average":{"1min":9.5850000000000009,"5min":16.161999999999999,"15min":29.065999999999999},"min":{"1min":6.3630000000000004,"5min":5.7699999999999996,"15min":5.7699999999999996},"max":{"1min":15.166,"5min":132.036,"15min":1766.5840000000001},"last":12.067}]},{"osd":4,"last update":"Mon Jun 3 09:05:31 2024","interfaces":[{"interface":"back","average":{"1min":10.996,"5min":954.46500000000003,"15min":1281.6859999999999},"min":{"1min":7.702,"5min":6.1760000000000002,"15min":6.0970000000000004},"max":{"1min":21.538,"5min":28044.862000000001,"15min":59438.017999999996},"last":7.4029999999999996},{"interface":"front","average":{"1min":13.646000000000001,"5min":961.24599999999998,"15min":1285.26},"min":{"1min":9.7200000000000006,"5min":9.3940000000000001,"15min":6.4640000000000004},"max":{"1min":23.277999999999999,"5min":28045.624,"15min":59439.283000000003},"last":9.8949999999999996}]},{"osd":5,"last update":"Mon Jun 3 09:06:04 2024","interfaces":[{"interface":"back","average":{"1min":12.478999999999999,"5min":20.228000000000002,"15min":25.997},"min":{"1min":8.6430000000000007,"5min":7.9720000000000004,"15min":7.234},"max":{"1min":17.442,"5min":139.43199999999999,"15min":701.22000000000003},"last":7.9050000000000002},{"interface":"front","average":{"1min":13.507999999999999,"5min":25.760999999999999,"15min":27.553999999999998},"min":{"1min":10.042999999999999,"5min":7.258,"15min":6.242},"max":{"1min":20.611000000000001,"5min":278.63,"15min":598.35000000000002},"last":11.395}]},{"osd":6,"last update":"Mon Jun 3 09:03:54 2024","interfaces":[{"interface":"back","average":{"1min":28.524000000000001,"5min":20.654,"15min":23.878},"min":{"1min":6.0309999999999997,"5min":6.0309999999999997,"15min":6.0309999999999997},"max":{"1min":167.221,"5min":167.221,"15min":704.245},"last":18.721},{"interface":"front","average":{"1min":27.606000000000002,"5min":30.131,"15min":28.628},"min":{"1min":9.8650000000000002,"5min":9.7200000000000006,"15min":8.3580000000000005},"max":{"1min":123.938,"5min":358.23000000000002,"15min":634.05499999999995},"last":173.113}]},{"osd":7,"last update":"Mon Jun 3 09:05:18 2024","interfaces":[{"interface":"back","average":{"1min":21.853999999999999,"5min":24.245999999999999,"15min":32.801000000000002},"min":{"1min":9.5020000000000007,"5min":8.0559999999999992,"15min":8.0559999999999992},"max":{"1min":142.245,"5min":270.95100000000002,"15min":703.07799999999997},"last":15.696},{"interface":"front","average":{"1min":21.547000000000001,"5min":29.977,"15min":30.513999999999999},"min":{"1min":11.167,"5min":8.2089999999999996,"15min":7.7599999999999998},"max":{"1min":119.348,"5min":360.99900000000002,"15min":608.44100000000003},"last":14.506}]}]},{"osd":0,"up_from":14,"seq":60129542500,"num_pgs":12,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":81088,"kb_used_data":5680,"kb_used_omap":1,"kb_used_meta":75390,"kb_avail":94290752,"statfs":{"total":96636764160,"available":96553730048,"internally_reserved":0,"allocated":5816320,"data_stored":2728907,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1593,"internal_metadata":77199815},"hb_peers":[1,2,3,4,5,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":4,"apply_latency_ms":4,"commit_latency_ns":4000000,"apply_latency_ns":4000000},"alerts":[],"network_ping_times":[{"osd":1,"last update":"Mon Jun 3 09:06:04 2024","interfaces":[{"interface":"back","average":{"1min":6.9960000000000004,"5min":56.997,"15min":339.29700000000003},"min":{"1min":4.4530000000000003,"5min":4.4530000000000003,"15min":4.4100000000000001},"max":{"1min":11.720000000000001,"5min":2414.4749999999999,"15min":20691.453000000001},"last":9.875},{"interface":"front","average":{"1min":9.548,"5min":57.258000000000003,"15min":337.791},"min":{"1min":5.2300000000000004,"5min":5.2300000000000004,"15min":4.6050000000000004},"max":{"1min":15.337999999999999,"5min":2412.9549999999999,"15min":20640.792000000001},"last":4.7199999999999998}]},{"osd":2,"last update":"Mon Jun 3 09:05:22 2024","interfaces":[{"interface":"back","average":{"1min":9.5820000000000007,"5min":27.760000000000002,"15min":23.097999999999999},"min":{"1min":5.766,"5min":5.766,"15min":5.766},"max":{"1min":46.584000000000003,"5min":627.40700000000004,"15min":627.40700000000004},"last":6.3840000000000003},{"interface":"front","average":{"1min":13.574999999999999,"5min":35.847999999999999,"15min":30.065000000000001},"min":{"1min":8.6419999999999995,"5min":7.8630000000000004,"15min":6.9260000000000002},"max":{"1min":54.68,"5min":635.62599999999998,"15min":635.62599999999998},"last":8.5039999999999996}]},{"osd":3,"last update":"Mon Jun 3 09:06:04 2024","interfaces":[{"interface":"back","average":{"1min":12.105,"5min":26.925000000000001,"15min":26.584},"min":{"1min":7.8410000000000002,"5min":7.1280000000000001,"15min":6.1909999999999998},"max":{"1min":18.425999999999998,"5min":389.72699999999998,"15min":389.72699999999998},"last":10.986000000000001},{"interface":"front","average":{"1min":13.794,"5min":29.771999999999998,"15min":28.815000000000001},"min":{"1min":10.084,"5min":9.4920000000000009,"15min":6.1849999999999996},"max":{"1min":17.373999999999999,"5min":390.23200000000003,"15min":494.92899999999997},"last":16.532}]},{"osd":4,"last update":"Mon Jun 3 09:05:27 2024","interfaces":[{"interface":"back","average":{"1min":13.478,"5min":1032.126,"15min":1389.729},"min":{"1min":6.5119999999999996,"5min":6.5119999999999996,"15min":6.5119999999999996},"max":{"1min":55.354999999999997,"5min":25555.212,"15min":60989.964},"last":12.74},{"interface":"front","average":{"1min":16.204999999999998,"5min":1030.8389999999999,"15min":1387.778},"min":{"1min":10.282999999999999,"5min":7.1429999999999998,"15min":7.1429999999999998},"max":{"1min":49.945999999999998,"5min":25556.159,"15min":61000.017},"last":13.673}]},{"osd":5,"last update":"Mon Jun 3 09:05:13 2024","interfaces":[{"interface":"back","average":{"1min":13.778,"5min":31.690999999999999,"15min":29.962},"min":{"1min":8.5030000000000001,"5min":7.7030000000000003,"15min":7.7030000000000003},"max":{"1min":60.216999999999999,"5min":455.78699999999998,"15min":605.05899999999997},"last":14.942},{"interface":"front","average":{"1min":17.823,"5min":30.498999999999999,"15min":27.434999999999999},"min":{"1min":11.547000000000001,"5min":6.2030000000000003,"15min":6.2030000000000003},"max":{"1min":59.677,"5min":391.31200000000001,"15min":541.11000000000001},"last":17.879000000000001}]},{"osd":6,"last update":"Mon Jun 3 09:04:30 2024","interfaces":[{"interface":"back","average":{"1min":52.496000000000002,"5min":33.139000000000003,"15min":160.83199999999999},"min":{"1min":7.5289999999999999,"5min":7.1040000000000001,"15min":6.3739999999999997},"max":{"1min":457.91699999999997,"5min":457.91699999999997,"15min":11115.258},"last":18.959},{"interface":"front","average":{"1min":49.670000000000002,"5min":33.704999999999998,"15min":162.791},"min":{"1min":12.247999999999999,"5min":11.019,"15min":9.9760000000000009},"max":{"1min":407.05099999999999,"5min":407.05099999999999,"15min":11092.194},"last":17.707000000000001}]},{"osd":7,"last update":"Mon Jun 3 09:05:22 2024","interfaces":[{"interface":"back","average":{"1min":14.503,"5min":29.489000000000001,"15min":33.134999999999998},"min":{"1min":8.5690000000000008,"5min":7.5149999999999997,"15min":6.9859999999999998},"max":{"1min":48.526000000000003,"5min":461.41699999999997,"15min":598.33699999999999},"last":11.83},{"interface":"front","average":{"1min":17.050000000000001,"5min":33.313000000000002,"15min":38.603999999999999},"min":{"1min":11.182,"5min":10.263999999999999,"15min":10.263999999999999},"max":{"1min":58.683,"5min":463.10599999999999,"15min":592.80499999999995},"last":14.407}]}]}],"pool_statfs":[{"poolid":1,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":401408,"data_stored":397840,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":5,"total":0,"available":0,"internally_reserved":0,"allocated":401408,"data_stored":397840,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":4,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":5,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":6,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":17,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":17,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":17,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":17,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":17,"osd":4,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":17,"osd":5,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":5,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":17,"osd":6,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":5,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":17,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0}]}} 2024-06-03T09:06:21.036 DEBUG:teuthology.orchestra.run.smithi088:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph pg dump --format=json 2024-06-03T09:06:21.136 INFO:teuthology.orchestra.run.smithi088.stderr:2024-06-03T09:06:21.136+0000 7f462fa0d640 10 monclient: get_monmap_and_config 2024-06-03T09:06:21.137 INFO:teuthology.orchestra.run.smithi088.stderr:2024-06-03T09:06:21.136+0000 7f462fa0d640 10 monclient: build_initial_monmap 2024-06-03T09:06:21.137 INFO:teuthology.orchestra.run.smithi088.stderr:2024-06-03T09:06:21.136+0000 7f462fa0d640 10 monclient: monmap: 2024-06-03T09:06:21.137 INFO:teuthology.orchestra.run.smithi088.stderr:epoch 0 2024-06-03T09:06:21.137 INFO:teuthology.orchestra.run.smithi088.stderr:fsid ec3c3fdd-3ac6-4d7d-9412-683130ed8b04 2024-06-03T09:06:21.137 INFO:teuthology.orchestra.run.smithi088.stderr:last_changed 2024-06-03T09:06:21.137938+0000 2024-06-03T09:06:21.137 INFO:teuthology.orchestra.run.smithi088.stderr:created 2024-06-03T09:06:21.137938+0000 2024-06-03T09:06:21.137 INFO:teuthology.orchestra.run.smithi088.stderr:min_mon_release 0 (unknown) 2024-06-03T09:06:21.137 INFO:teuthology.orchestra.run.smithi088.stderr:election_strategy: 3 2024-06-03T09:06:21.137 INFO:teuthology.orchestra.run.smithi088.stderr:0: v1:172.21.15.88:6789/0 mon.noname-a 2024-06-03T09:06:21.137 INFO:teuthology.orchestra.run.smithi088.stderr:1: v1:172.21.15.192:6789/0 mon.noname-c 2024-06-03T09:06:21.137 INFO:teuthology.orchestra.run.smithi088.stderr:2: v1:172.21.15.88:6790/0 mon.noname-b 2024-06-03T09:06:21.137 INFO:teuthology.orchestra.run.smithi088.stderr: 2024-06-03T09:06:21.139 INFO:teuthology.orchestra.run.smithi088.stderr:2024-06-03T09:06:21.137+0000 7f462fa0d640 1 Processor -- start 2024-06-03T09:06:21.139 INFO:teuthology.orchestra.run.smithi088.stderr:2024-06-03T09:06:21.137+0000 7f462fa0d640 1 -- start start 2024-06-03T09:06:21.139 INFO:teuthology.orchestra.run.smithi088.stderr:2024-06-03T09:06:21.137+0000 7f462fa0d640 10 monclient: init 2024-06-03T09:06:21.139 INFO:teuthology.orchestra.run.smithi088.stderr:2024-06-03T09:06:21.138+0000 7f462fa0d640 10 monclient: _reopen_session rank -1 2024-06-03T09:06:21.139 INFO:teuthology.orchestra.run.smithi088.stderr:2024-06-03T09:06:21.138+0000 7f462fa0d640 10 monclient: _add_conns ranks=[1,2,0] 2024-06-03T09:06:21.139 INFO:teuthology.orchestra.run.smithi088.stderr:2024-06-03T09:06:21.138+0000 7f462fa0d640 10 monclient(hunting): picked mon.noname-c con 0x7f462806e050 addr v1:172.21.15.192:6789/0 2024-06-03T09:06:21.139 INFO:teuthology.orchestra.run.smithi088.stderr:2024-06-03T09:06:21.138+0000 7f462fa0d640 10 monclient(hunting): picked mon.noname-b con 0x7f462806e430 addr v1:172.21.15.88:6790/0 2024-06-03T09:06:21.139 INFO:teuthology.orchestra.run.smithi088.stderr:2024-06-03T09:06:21.138+0000 7f462fa0d640 10 monclient(hunting): picked mon.noname-a con 0x7f46280717c0 addr v1:172.21.15.88:6789/0 2024-06-03T09:06:21.139 INFO:teuthology.orchestra.run.smithi088.stderr:2024-06-03T09:06:21.138+0000 7f462fa0d640 1 -- --> v1:172.21.15.88:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f4628070c10 con 0x7f46280717c0 2024-06-03T09:06:21.139 INFO:teuthology.orchestra.run.smithi088.stderr:2024-06-03T09:06:21.138+0000 7f462fa0d640 1 -- --> v1:172.21.15.192:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f4628070230 con 0x7f462806e050 2024-06-03T09:06:21.139 INFO:teuthology.orchestra.run.smithi088.stderr:2024-06-03T09:06:21.138+0000 7f462fa0d640 1 -- --> v1:172.21.15.88:6790/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f4628070510 con 0x7f462806e430 2024-06-03T09:06:21.139 INFO:teuthology.orchestra.run.smithi088.stderr:2024-06-03T09:06:21.138+0000 7f462fa0d640 10 monclient(hunting): _renew_subs 2024-06-03T09:06:21.140 INFO:teuthology.orchestra.run.smithi088.stderr:2024-06-03T09:06:21.138+0000 7f462fa0d640 10 monclient(hunting): authenticate will time out at 3036.948358s 2024-06-03T09:06:21.140 INFO:teuthology.orchestra.run.smithi088.stderr:2024-06-03T09:06:21.138+0000 7f462dfaa640 1 --1- >> v1:172.21.15.88:6789/0 conn(0x7f46280717c0 0x7f4628149240 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.88:6789/0 says I am v1:172.21.15.88:43916/0 (socket says 172.21.15.88:43916) 2024-06-03T09:06:21.140 INFO:teuthology.orchestra.run.smithi088.stderr:2024-06-03T09:06:21.138+0000 7f462dfaa640 1 -- 172.21.15.88:0/3313730164 learned_addr learned my addr 172.21.15.88:0/3313730164 (peer_addr_for_me v1:172.21.15.88:0/0) 2024-06-03T09:06:21.141 INFO:teuthology.orchestra.run.smithi088.stderr:2024-06-03T09:06:21.141+0000 7f461ffff640 1 -- 172.21.15.88:0/3313730164 <== mon.1 v1:172.21.15.192:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 564537583 0 0) 0x7f4628070230 con 0x7f462806e050 2024-06-03T09:06:21.141 INFO:teuthology.orchestra.run.smithi088.stderr:2024-06-03T09:06:21.141+0000 7f461ffff640 10 monclient(hunting): _negotiate 2024-06-03T09:06:21.141 INFO:teuthology.orchestra.run.smithi088.stderr:2024-06-03T09:06:21.141+0000 7f461ffff640 10 monclient(hunting): _init_auth method 2 2024-06-03T09:06:21.141 INFO:teuthology.orchestra.run.smithi088.stderr:2024-06-03T09:06:21.141+0000 7f461ffff640 10 monclient(hunting): _init_auth creating new auth 2024-06-03T09:06:21.141 INFO:teuthology.orchestra.run.smithi088.stderr:2024-06-03T09:06:21.141+0000 7f461ffff640 10 monclient(hunting): my global_id is 30163 2024-06-03T09:06:21.141 INFO:teuthology.orchestra.run.smithi088.stderr:2024-06-03T09:06:21.141+0000 7f461ffff640 1 -- 172.21.15.88:0/3313730164 --> v1:172.21.15.192:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f4614004630 con 0x7f462806e050 2024-06-03T09:06:21.141 INFO:teuthology.orchestra.run.smithi088.stderr:2024-06-03T09:06:21.141+0000 7f461ffff640 1 -- 172.21.15.88:0/3313730164 <== mon.0 v1:172.21.15.88:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 1743501569 0 0) 0x7f4628070c10 con 0x7f46280717c0 2024-06-03T09:06:21.142 INFO:teuthology.orchestra.run.smithi088.stderr:2024-06-03T09:06:21.141+0000 7f461ffff640 10 monclient(hunting): _negotiate 2024-06-03T09:06:21.142 INFO:teuthology.orchestra.run.smithi088.stderr:2024-06-03T09:06:21.141+0000 7f461ffff640 10 monclient(hunting): _init_auth method 2 2024-06-03T09:06:21.142 INFO:teuthology.orchestra.run.smithi088.stderr:2024-06-03T09:06:21.141+0000 7f461ffff640 10 monclient(hunting): _init_auth creating new auth 2024-06-03T09:06:21.142 INFO:teuthology.orchestra.run.smithi088.stderr:2024-06-03T09:06:21.141+0000 7f461ffff640 10 monclient(hunting): my global_id is 30582 2024-06-03T09:06:21.142 INFO:teuthology.orchestra.run.smithi088.stderr:2024-06-03T09:06:21.141+0000 7f461ffff640 1 -- 172.21.15.88:0/3313730164 --> v1:172.21.15.88:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f4628070230 con 0x7f46280717c0 2024-06-03T09:06:21.142 INFO:teuthology.orchestra.run.smithi088.stderr:2024-06-03T09:06:21.141+0000 7f461ffff640 1 -- 172.21.15.88:0/3313730164 <== mon.2 v1:172.21.15.88:6790/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 2805176460 0 0) 0x7f4628070510 con 0x7f462806e430 2024-06-03T09:06:21.142 INFO:teuthology.orchestra.run.smithi088.stderr:2024-06-03T09:06:21.141+0000 7f461ffff640 10 monclient(hunting): _negotiate 2024-06-03T09:06:21.142 INFO:teuthology.orchestra.run.smithi088.stderr:2024-06-03T09:06:21.141+0000 7f461ffff640 10 monclient(hunting): _init_auth method 2 2024-06-03T09:06:21.142 INFO:teuthology.orchestra.run.smithi088.stderr:2024-06-03T09:06:21.141+0000 7f461ffff640 10 monclient(hunting): _init_auth creating new auth 2024-06-03T09:06:21.142 INFO:teuthology.orchestra.run.smithi088.stderr:2024-06-03T09:06:21.141+0000 7f461ffff640 10 monclient(hunting): my global_id is 30086 2024-06-03T09:06:21.142 INFO:teuthology.orchestra.run.smithi088.stderr:2024-06-03T09:06:21.141+0000 7f461ffff640 1 -- 172.21.15.88:0/3313730164 --> v1:172.21.15.88:6790/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f4628070c10 con 0x7f462806e430 2024-06-03T09:06:21.144 INFO:teuthology.orchestra.run.smithi088.stderr:2024-06-03T09:06:21.144+0000 7f461ffff640 1 -- 172.21.15.88:0/3313730164 <== mon.1 v1:172.21.15.192:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 214+0+0 (unknown 93455435 0 0) 0x7f4614004630 con 0x7f462806e050 2024-06-03T09:06:21.144 INFO:teuthology.orchestra.run.smithi088.stderr:2024-06-03T09:06:21.144+0000 7f461ffff640 1 -- 172.21.15.88:0/3313730164 >> v1:172.21.15.88:6790/0 conn(0x7f462806e430 legacy=0x7f46280710b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-06-03T09:06:21.144 INFO:teuthology.orchestra.run.smithi088.stderr:2024-06-03T09:06:21.144+0000 7f461ffff640 1 -- 172.21.15.88:0/3313730164 >> v1:172.21.15.88:6789/0 conn(0x7f46280717c0 legacy=0x7f4628149240 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-06-03T09:06:21.144 INFO:teuthology.orchestra.run.smithi088.stderr:2024-06-03T09:06:21.144+0000 7f461ffff640 10 monclient: _finish_hunting 0 2024-06-03T09:06:21.144 INFO:teuthology.orchestra.run.smithi088.stderr:2024-06-03T09:06:21.144+0000 7f461ffff640 1 monclient: found mon.noname-c 2024-06-03T09:06:21.144 INFO:teuthology.orchestra.run.smithi088.stderr:2024-06-03T09:06:21.144+0000 7f461ffff640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-06-03T09:06:21.144 INFO:teuthology.orchestra.run.smithi088.stderr:2024-06-03T09:06:21.144+0000 7f461ffff640 10 monclient: _send_mon_message to mon.noname-c at v1:172.21.15.192:6789/0 2024-06-03T09:06:21.145 INFO:teuthology.orchestra.run.smithi088.stderr:2024-06-03T09:06:21.144+0000 7f461ffff640 1 -- 172.21.15.88:0/3313730164 --> v1:172.21.15.192:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f46280707f0 con 0x7f462806e050 2024-06-03T09:06:21.145 INFO:teuthology.orchestra.run.smithi088.stderr:2024-06-03T09:06:21.144+0000 7f461ffff640 10 monclient: _finish_auth 0 2024-06-03T09:06:21.145 INFO:teuthology.orchestra.run.smithi088.stderr:2024-06-03T09:06:21.144+0000 7f461ffff640 10 monclient: _check_auth_tickets 2024-06-03T09:06:21.145 INFO:teuthology.orchestra.run.smithi088.stderr:2024-06-03T09:06:21.144+0000 7f461ffff640 20 monclient: _check_auth_rotating not needed by client.admin 2024-06-03T09:06:21.145 INFO:teuthology.orchestra.run.smithi088.stderr:2024-06-03T09:06:21.144+0000 7f461ffff640 1 -- 172.21.15.88:0/3313730164 <== mon.1 v1:172.21.15.192:6789/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 2234374065 0 0) 0x7f4624001610 con 0x7f462806e050 2024-06-03T09:06:21.145 INFO:teuthology.orchestra.run.smithi088.stderr:2024-06-03T09:06:21.144+0000 7f461ffff640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-06-03T09:06:21.145 INFO:teuthology.orchestra.run.smithi088.stderr:2024-06-03T09:06:21.144+0000 7f461ffff640 10 monclient: got monmap 1 from mon.noname-c (according to old e1) 2024-06-03T09:06:21.145 INFO:teuthology.orchestra.run.smithi088.stderr:2024-06-03T09:06:21.144+0000 7f461ffff640 10 monclient: dump: 2024-06-03T09:06:21.145 INFO:teuthology.orchestra.run.smithi088.stderr:epoch 1 2024-06-03T09:06:21.145 INFO:teuthology.orchestra.run.smithi088.stderr:fsid ec3c3fdd-3ac6-4d7d-9412-683130ed8b04 2024-06-03T09:06:21.145 INFO:teuthology.orchestra.run.smithi088.stderr:last_changed 2024-06-03T08:34:31.885125+0000 2024-06-03T09:06:21.145 INFO:teuthology.orchestra.run.smithi088.stderr:created 2024-06-03T08:34:31.885125+0000 2024-06-03T09:06:21.145 INFO:teuthology.orchestra.run.smithi088.stderr:min_mon_release 18 (reef) 2024-06-03T09:06:21.146 INFO:teuthology.orchestra.run.smithi088.stderr:election_strategy: 3 2024-06-03T09:06:21.146 INFO:teuthology.orchestra.run.smithi088.stderr:0: v1:172.21.15.88:6789/0 mon.a 2024-06-03T09:06:21.146 INFO:teuthology.orchestra.run.smithi088.stderr:1: v1:172.21.15.192:6789/0 mon.b 2024-06-03T09:06:21.146 INFO:teuthology.orchestra.run.smithi088.stderr:2: v1:172.21.15.88:6790/0 mon.c 2024-06-03T09:06:21.146 INFO:teuthology.orchestra.run.smithi088.stderr: 2024-06-03T09:06:21.146 INFO:teuthology.orchestra.run.smithi088.stderr:2024-06-03T09:06:21.144+0000 7f462fa0d640 5 monclient: authenticate success, global_id 30163 2024-06-03T09:06:21.146 INFO:teuthology.orchestra.run.smithi088.stderr:2024-06-03T09:06:21.144+0000 7f462fa0d640 20 monclient: get_monmap_and_config waiting for monmap|config 2024-06-03T09:06:21.148 INFO:teuthology.orchestra.run.smithi088.stderr:2024-06-03T09:06:21.146+0000 7f461ffff640 1 -- 172.21.15.88:0/3313730164 <== mon.1 v1:172.21.15.192:6789/0 4 ==== config(0 keys) v1 ==== 4+0+0 (unknown 0 0 0) 0x7f4624001830 con 0x7f462806e050 2024-06-03T09:06:21.148 INFO:teuthology.orchestra.run.smithi088.stderr:2024-06-03T09:06:21.146+0000 7f461ffff640 10 monclient: handle_config config(0 keys) v1 2024-06-03T09:06:21.148 INFO:teuthology.orchestra.run.smithi088.stderr:2024-06-03T09:06:21.146+0000 7f461ffff640 1 -- 172.21.15.88:0/3313730164 <== mon.1 v1:172.21.15.192:6789/0 5 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 2234374065 0 0) 0x7f4624001bc0 con 0x7f462806e050 2024-06-03T09:06:21.148 INFO:teuthology.orchestra.run.smithi088.stderr:2024-06-03T09:06:21.146+0000 7f461ffff640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-06-03T09:06:21.148 INFO:teuthology.orchestra.run.smithi088.stderr:2024-06-03T09:06:21.146+0000 7f461ffff640 10 monclient: got monmap 1 from mon.b (according to old e1) 2024-06-03T09:06:21.148 INFO:teuthology.orchestra.run.smithi088.stderr:2024-06-03T09:06:21.146+0000 7f461ffff640 10 monclient: dump: 2024-06-03T09:06:21.148 INFO:teuthology.orchestra.run.smithi088.stderr:epoch 1 2024-06-03T09:06:21.149 INFO:teuthology.orchestra.run.smithi088.stderr:fsid ec3c3fdd-3ac6-4d7d-9412-683130ed8b04 2024-06-03T09:06:21.149 INFO:teuthology.orchestra.run.smithi088.stderr:last_changed 2024-06-03T08:34:31.885125+0000 2024-06-03T09:06:21.149 INFO:teuthology.orchestra.run.smithi088.stderr:created 2024-06-03T08:34:31.885125+0000 2024-06-03T09:06:21.149 INFO:teuthology.orchestra.run.smithi088.stderr:min_mon_release 18 (reef) 2024-06-03T09:06:21.149 INFO:teuthology.orchestra.run.smithi088.stderr:election_strategy: 3 2024-06-03T09:06:21.149 INFO:teuthology.orchestra.run.smithi088.stderr:0: v1:172.21.15.88:6789/0 mon.a 2024-06-03T09:06:21.149 INFO:teuthology.orchestra.run.smithi088.stderr:1: v1:172.21.15.192:6789/0 mon.b 2024-06-03T09:06:21.149 INFO:teuthology.orchestra.run.smithi088.stderr:2: v1:172.21.15.88:6790/0 mon.c 2024-06-03T09:06:21.149 INFO:teuthology.orchestra.run.smithi088.stderr: 2024-06-03T09:06:21.149 INFO:teuthology.orchestra.run.smithi088.stderr:2024-06-03T09:06:21.146+0000 7f462fa0d640 10 monclient: get_monmap_and_config success 2024-06-03T09:06:21.149 INFO:teuthology.orchestra.run.smithi088.stderr:2024-06-03T09:06:21.146+0000 7f462fa0d640 4 set_mon_vals no callback set 2024-06-03T09:06:21.149 INFO:teuthology.orchestra.run.smithi088.stderr:2024-06-03T09:06:21.146+0000 7f462fa0d640 10 monclient: shutdown 2024-06-03T09:06:21.149 INFO:teuthology.orchestra.run.smithi088.stderr:2024-06-03T09:06:21.146+0000 7f462fa0d640 20 monclient: shutdown discarding 0 pending message(s) 2024-06-03T09:06:21.150 INFO:teuthology.orchestra.run.smithi088.stderr:2024-06-03T09:06:21.146+0000 7f462fa0d640 1 -- 172.21.15.88:0/3313730164 >> v1:172.21.15.192:6789/0 conn(0x7f462806e050 legacy=0x7f462813ad00 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-06-03T09:06:21.150 INFO:teuthology.orchestra.run.smithi088.stderr:2024-06-03T09:06:21.147+0000 7f462fa0d640 1 -- 172.21.15.88:0/3313730164 shutdown_connections 2024-06-03T09:06:21.150 INFO:teuthology.orchestra.run.smithi088.stderr:2024-06-03T09:06:21.147+0000 7f462fa0d640 1 -- 172.21.15.88:0/3313730164 >> 172.21.15.88:0/3313730164 conn(0x7f46280b2f20 msgr2=0x7f46280b3300 unknown :-1 s=STATE_NONE l=0).mark_down 2024-06-03T09:06:21.150 INFO:teuthology.orchestra.run.smithi088.stderr:2024-06-03T09:06:21.147+0000 7f462fa0d640 1 -- 172.21.15.88:0/3313730164 shutdown_connections 2024-06-03T09:06:21.150 INFO:teuthology.orchestra.run.smithi088.stderr:2024-06-03T09:06:21.147+0000 7f462fa0d640 1 -- 172.21.15.88:0/3313730164 wait complete. 2024-06-03T09:06:21.150 INFO:teuthology.orchestra.run.smithi088.stderr:2024-06-03T09:06:21.147+0000 7f462fa0d640 10 monclient: build_initial_monmap 2024-06-03T09:06:21.150 INFO:teuthology.orchestra.run.smithi088.stderr:2024-06-03T09:06:21.147+0000 7f462fa0d640 10 monclient: monmap: 2024-06-03T09:06:21.150 INFO:teuthology.orchestra.run.smithi088.stderr:epoch 0 2024-06-03T09:06:21.150 INFO:teuthology.orchestra.run.smithi088.stderr:fsid 00000000-0000-0000-0000-000000000000 2024-06-03T09:06:21.150 INFO:teuthology.orchestra.run.smithi088.stderr:last_changed 0.000000 2024-06-03T09:06:21.150 INFO:teuthology.orchestra.run.smithi088.stderr:created 0.000000 2024-06-03T09:06:21.150 INFO:teuthology.orchestra.run.smithi088.stderr:min_mon_release 0 (unknown) 2024-06-03T09:06:21.150 INFO:teuthology.orchestra.run.smithi088.stderr:election_strategy: 1 2024-06-03T09:06:21.151 INFO:teuthology.orchestra.run.smithi088.stderr:0: v1:172.21.15.88:6789/0 mon.noname-a 2024-06-03T09:06:21.151 INFO:teuthology.orchestra.run.smithi088.stderr:1: v1:172.21.15.192:6789/0 mon.noname-b 2024-06-03T09:06:21.151 INFO:teuthology.orchestra.run.smithi088.stderr:2: v1:172.21.15.88:6790/0 mon.noname-c 2024-06-03T09:06:21.151 INFO:teuthology.orchestra.run.smithi088.stderr: 2024-06-03T09:06:21.151 INFO:teuthology.orchestra.run.smithi088.stderr:2024-06-03T09:06:21.148+0000 7f462fa0d640 1 librados: starting msgr at 2024-06-03T09:06:21.151 INFO:teuthology.orchestra.run.smithi088.stderr:2024-06-03T09:06:21.148+0000 7f462fa0d640 1 librados: starting objecter 2024-06-03T09:06:21.151 INFO:teuthology.orchestra.run.smithi088.stderr:2024-06-03T09:06:21.148+0000 7f462fa0d640 1 Processor -- start 2024-06-03T09:06:21.151 INFO:teuthology.orchestra.run.smithi088.stderr:2024-06-03T09:06:21.148+0000 7f462fa0d640 1 -- start start 2024-06-03T09:06:21.151 INFO:teuthology.orchestra.run.smithi088.stderr:2024-06-03T09:06:21.148+0000 7f462fa0d640 1 librados: setting wanted keys 2024-06-03T09:06:21.151 INFO:teuthology.orchestra.run.smithi088.stderr:2024-06-03T09:06:21.148+0000 7f462fa0d640 1 librados: calling monclient init 2024-06-03T09:06:21.151 INFO:teuthology.orchestra.run.smithi088.stderr:2024-06-03T09:06:21.148+0000 7f462fa0d640 10 monclient: init 2024-06-03T09:06:21.153 INFO:teuthology.orchestra.run.smithi088.stderr:2024-06-03T09:06:21.148+0000 7f462fa0d640 10 monclient: _reopen_session rank -1 2024-06-03T09:06:21.153 INFO:teuthology.orchestra.run.smithi088.stderr:2024-06-03T09:06:21.148+0000 7f462fa0d640 10 monclient: _add_conns ranks=[1,2,0] 2024-06-03T09:06:21.153 INFO:teuthology.orchestra.run.smithi088.stderr:2024-06-03T09:06:21.148+0000 7f462fa0d640 10 monclient(hunting): picked mon.noname-b con 0x7f462806e050 addr v1:172.21.15.192:6789/0 2024-06-03T09:06:21.153 INFO:teuthology.orchestra.run.smithi088.stderr:2024-06-03T09:06:21.148+0000 7f462fa0d640 10 monclient(hunting): picked mon.noname-c con 0x7f46280717c0 addr v1:172.21.15.88:6790/0 2024-06-03T09:06:21.154 INFO:teuthology.orchestra.run.smithi088.stderr:2024-06-03T09:06:21.148+0000 7f462fa0d640 10 monclient(hunting): picked mon.noname-a con 0x7f46280ac7a0 addr v1:172.21.15.88:6789/0 2024-06-03T09:06:21.154 INFO:teuthology.orchestra.run.smithi088.stderr:2024-06-03T09:06:21.148+0000 7f462fa0d640 1 -- --> v1:172.21.15.88:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f46280ac4a0 con 0x7f46280ac7a0 2024-06-03T09:06:21.154 INFO:teuthology.orchestra.run.smithi088.stderr:2024-06-03T09:06:21.148+0000 7f462fa0d640 1 -- --> v1:172.21.15.192:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f462813e7d0 con 0x7f462806e050 2024-06-03T09:06:21.154 INFO:teuthology.orchestra.run.smithi088.stderr:2024-06-03T09:06:21.148+0000 7f462fa0d640 1 -- --> v1:172.21.15.88:6790/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f46280ac050 con 0x7f46280717c0 2024-06-03T09:06:21.154 INFO:teuthology.orchestra.run.smithi088.stderr:2024-06-03T09:06:21.148+0000 7f462fa0d640 10 monclient(hunting): _renew_subs 2024-06-03T09:06:21.154 INFO:teuthology.orchestra.run.smithi088.stderr:2024-06-03T09:06:21.148+0000 7f462fa0d640 10 monclient(hunting): authenticate will time out at 3036.958697s 2024-06-03T09:06:21.154 INFO:teuthology.orchestra.run.smithi088.stderr:2024-06-03T09:06:21.149+0000 7f462dfaa640 1 --1- >> v1:172.21.15.88:6789/0 conn(0x7f46280ac7a0 0x7f46281d2a60 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=1).handle_server_banner_and_identify peer v1:172.21.15.88:6789/0 says I am v1:172.21.15.88:43922/0 (socket says 172.21.15.88:43922) 2024-06-03T09:06:21.154 INFO:teuthology.orchestra.run.smithi088.stderr:2024-06-03T09:06:21.149+0000 7f462dfaa640 1 -- 172.21.15.88:0/2463024474 learned_addr learned my addr 172.21.15.88:0/2463024474 (peer_addr_for_me v1:172.21.15.88:0/0) 2024-06-03T09:06:21.154 INFO:teuthology.orchestra.run.smithi088.stderr:2024-06-03T09:06:21.149+0000 7f461dffb640 10 client.?.objecter ms_handle_connect 0x7f46280ac7a0 2024-06-03T09:06:21.154 INFO:teuthology.orchestra.run.smithi088.stderr:2024-06-03T09:06:21.149+0000 7f461dffb640 10 client.?.objecter resend_mon_ops 2024-06-03T09:06:21.154 INFO:teuthology.orchestra.run.smithi088.stderr:2024-06-03T09:06:21.149+0000 7f461dffb640 10 client.?.objecter ms_handle_connect 0x7f46280717c0 2024-06-03T09:06:21.154 INFO:teuthology.orchestra.run.smithi088.stderr:2024-06-03T09:06:21.149+0000 7f461dffb640 10 client.?.objecter resend_mon_ops 2024-06-03T09:06:21.155 INFO:teuthology.orchestra.run.smithi088.stderr:2024-06-03T09:06:21.149+0000 7f461dffb640 10 client.?.objecter ms_handle_connect 0x7f462806e050 2024-06-03T09:06:21.155 INFO:teuthology.orchestra.run.smithi088.stderr:2024-06-03T09:06:21.149+0000 7f461dffb640 10 client.?.objecter resend_mon_ops 2024-06-03T09:06:21.155 INFO:teuthology.orchestra.run.smithi088.stderr:2024-06-03T09:06:21.151+0000 7f461dffb640 1 -- 172.21.15.88:0/2463024474 <== mon.0 v1:172.21.15.88:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 1799156065 0 0) 0x7f46280ac4a0 con 0x7f46280ac7a0 2024-06-03T09:06:21.155 INFO:teuthology.orchestra.run.smithi088.stderr:2024-06-03T09:06:21.151+0000 7f461dffb640 10 monclient(hunting): _negotiate 2024-06-03T09:06:21.155 INFO:teuthology.orchestra.run.smithi088.stderr:2024-06-03T09:06:21.151+0000 7f461dffb640 10 monclient(hunting): _init_auth method 2 2024-06-03T09:06:21.155 INFO:teuthology.orchestra.run.smithi088.stderr:2024-06-03T09:06:21.151+0000 7f461dffb640 10 monclient(hunting): _init_auth creating new auth 2024-06-03T09:06:21.155 INFO:teuthology.orchestra.run.smithi088.stderr:2024-06-03T09:06:21.151+0000 7f461dffb640 10 monclient(hunting): my global_id is 30585 2024-06-03T09:06:21.155 INFO:teuthology.orchestra.run.smithi088.stderr:2024-06-03T09:06:21.151+0000 7f461dffb640 1 -- 172.21.15.88:0/2463024474 --> v1:172.21.15.88:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f460c004eb0 con 0x7f46280ac7a0 2024-06-03T09:06:21.155 INFO:teuthology.orchestra.run.smithi088.stderr:2024-06-03T09:06:21.151+0000 7f461dffb640 1 -- 172.21.15.88:0/2463024474 <== mon.2 v1:172.21.15.88:6790/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 2432773653 0 0) 0x7f46280ac050 con 0x7f46280717c0 2024-06-03T09:06:21.155 INFO:teuthology.orchestra.run.smithi088.stderr:2024-06-03T09:06:21.151+0000 7f461dffb640 10 monclient(hunting): _negotiate 2024-06-03T09:06:21.155 INFO:teuthology.orchestra.run.smithi088.stderr:2024-06-03T09:06:21.151+0000 7f461dffb640 10 monclient(hunting): _init_auth method 2 2024-06-03T09:06:21.155 INFO:teuthology.orchestra.run.smithi088.stderr:2024-06-03T09:06:21.151+0000 7f461dffb640 10 monclient(hunting): _init_auth creating new auth 2024-06-03T09:06:21.155 INFO:teuthology.orchestra.run.smithi088.stderr:2024-06-03T09:06:21.151+0000 7f461dffb640 10 monclient(hunting): my global_id is 30089 2024-06-03T09:06:21.155 INFO:teuthology.orchestra.run.smithi088.stderr:2024-06-03T09:06:21.151+0000 7f461dffb640 1 -- 172.21.15.88:0/2463024474 --> v1:172.21.15.88:6790/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f46280ac4a0 con 0x7f46280717c0 2024-06-03T09:06:21.156 INFO:teuthology.orchestra.run.smithi088.stderr:2024-06-03T09:06:21.151+0000 7f461dffb640 1 -- 172.21.15.88:0/2463024474 <== mon.1 v1:172.21.15.192:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 1306404773 0 0) 0x7f462813e7d0 con 0x7f462806e050 2024-06-03T09:06:21.156 INFO:teuthology.orchestra.run.smithi088.stderr:2024-06-03T09:06:21.151+0000 7f461dffb640 10 monclient(hunting): _negotiate 2024-06-03T09:06:21.156 INFO:teuthology.orchestra.run.smithi088.stderr:2024-06-03T09:06:21.151+0000 7f461dffb640 10 monclient(hunting): _init_auth method 2 2024-06-03T09:06:21.156 INFO:teuthology.orchestra.run.smithi088.stderr:2024-06-03T09:06:21.151+0000 7f461dffb640 10 monclient(hunting): _init_auth creating new auth 2024-06-03T09:06:21.156 INFO:teuthology.orchestra.run.smithi088.stderr:2024-06-03T09:06:21.151+0000 7f461dffb640 10 monclient(hunting): my global_id is 30166 2024-06-03T09:06:21.156 INFO:teuthology.orchestra.run.smithi088.stderr:2024-06-03T09:06:21.151+0000 7f461dffb640 1 -- 172.21.15.88:0/2463024474 --> v1:172.21.15.192:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f46280ac050 con 0x7f462806e050 2024-06-03T09:06:21.157 INFO:teuthology.orchestra.run.smithi088.stderr:2024-06-03T09:06:21.156+0000 7f461dffb640 1 -- 172.21.15.88:0/2463024474 <== mon.1 v1:172.21.15.192:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 764+0+0 (unknown 804287583 0 0) 0x7f46280ac050 con 0x7f462806e050 2024-06-03T09:06:21.157 INFO:teuthology.orchestra.run.smithi088.stderr:2024-06-03T09:06:21.156+0000 7f461dffb640 1 -- 172.21.15.88:0/2463024474 --> v1:172.21.15.192:6789/0 -- auth(proto 2 165 bytes epoch 0) v1 -- 0x7f462813e7d0 con 0x7f462806e050 2024-06-03T09:06:21.157 INFO:teuthology.orchestra.run.smithi088.stderr:2024-06-03T09:06:21.156+0000 7f461dffb640 1 -- 172.21.15.88:0/2463024474 <== mon.1 v1:172.21.15.192:6789/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 2234374065 0 0) 0x7f4624001b40 con 0x7f462806e050 2024-06-03T09:06:21.157 INFO:teuthology.orchestra.run.smithi088.stderr:2024-06-03T09:06:21.156+0000 7f461dffb640 10 monclient(hunting): handle_monmap mon_map magic: 0 v1 2024-06-03T09:06:21.157 INFO:teuthology.orchestra.run.smithi088.stderr:2024-06-03T09:06:21.156+0000 7f461dffb640 10 monclient(hunting): got monmap 1 from mon.noname-b (according to old e1) 2024-06-03T09:06:21.158 INFO:teuthology.orchestra.run.smithi088.stderr:2024-06-03T09:06:21.156+0000 7f461dffb640 10 monclient(hunting): dump: 2024-06-03T09:06:21.158 INFO:teuthology.orchestra.run.smithi088.stderr:epoch 1 2024-06-03T09:06:21.158 INFO:teuthology.orchestra.run.smithi088.stderr:fsid ec3c3fdd-3ac6-4d7d-9412-683130ed8b04 2024-06-03T09:06:21.158 INFO:teuthology.orchestra.run.smithi088.stderr:last_changed 2024-06-03T08:34:31.885125+0000 2024-06-03T09:06:21.158 INFO:teuthology.orchestra.run.smithi088.stderr:created 2024-06-03T08:34:31.885125+0000 2024-06-03T09:06:21.158 INFO:teuthology.orchestra.run.smithi088.stderr:min_mon_release 18 (reef) 2024-06-03T09:06:21.158 INFO:teuthology.orchestra.run.smithi088.stderr:election_strategy: 3 2024-06-03T09:06:21.158 INFO:teuthology.orchestra.run.smithi088.stderr:0: v1:172.21.15.88:6789/0 mon.a 2024-06-03T09:06:21.158 INFO:teuthology.orchestra.run.smithi088.stderr:1: v1:172.21.15.192:6789/0 mon.b 2024-06-03T09:06:21.158 INFO:teuthology.orchestra.run.smithi088.stderr:2: v1:172.21.15.88:6790/0 mon.c 2024-06-03T09:06:21.158 INFO:teuthology.orchestra.run.smithi088.stderr: 2024-06-03T09:06:21.158 INFO:teuthology.orchestra.run.smithi088.stderr:2024-06-03T09:06:21.156+0000 7f461dffb640 10 monclient(hunting): _finish_auth 0 2024-06-03T09:06:21.158 INFO:teuthology.orchestra.run.smithi088.stderr:2024-06-03T09:06:21.156+0000 7f461dffb640 1 -- 172.21.15.88:0/2463024474 <== mon.2 v1:172.21.15.88:6790/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 764+0+0 (unknown 2502684817 0 0) 0x7f46280ac4a0 con 0x7f46280717c0 2024-06-03T09:06:21.159 INFO:teuthology.orchestra.run.smithi088.stderr:2024-06-03T09:06:21.156+0000 7f461dffb640 1 -- 172.21.15.88:0/2463024474 --> v1:172.21.15.88:6790/0 -- auth(proto 2 165 bytes epoch 0) v1 -- 0x7f46280ac050 con 0x7f46280717c0 2024-06-03T09:06:21.159 INFO:teuthology.orchestra.run.smithi088.stderr:2024-06-03T09:06:21.156+0000 7f461dffb640 1 -- 172.21.15.88:0/2463024474 <== mon.2 v1:172.21.15.88:6790/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 2234374065 0 0) 0x7f46180018b0 con 0x7f46280717c0 2024-06-03T09:06:21.159 INFO:teuthology.orchestra.run.smithi088.stderr:2024-06-03T09:06:21.156+0000 7f461dffb640 10 monclient(hunting): handle_monmap mon_map magic: 0 v1 2024-06-03T09:06:21.159 INFO:teuthology.orchestra.run.smithi088.stderr:2024-06-03T09:06:21.156+0000 7f461dffb640 10 monclient(hunting): got monmap 1 from mon.c (according to old e1) 2024-06-03T09:06:21.159 INFO:teuthology.orchestra.run.smithi088.stderr:2024-06-03T09:06:21.156+0000 7f461dffb640 10 monclient(hunting): dump: 2024-06-03T09:06:21.159 INFO:teuthology.orchestra.run.smithi088.stderr:epoch 1 2024-06-03T09:06:21.159 INFO:teuthology.orchestra.run.smithi088.stderr:fsid ec3c3fdd-3ac6-4d7d-9412-683130ed8b04 2024-06-03T09:06:21.159 INFO:teuthology.orchestra.run.smithi088.stderr:last_changed 2024-06-03T08:34:31.885125+0000 2024-06-03T09:06:21.159 INFO:teuthology.orchestra.run.smithi088.stderr:created 2024-06-03T08:34:31.885125+0000 2024-06-03T09:06:21.159 INFO:teuthology.orchestra.run.smithi088.stderr:min_mon_release 18 (reef) 2024-06-03T09:06:21.159 INFO:teuthology.orchestra.run.smithi088.stderr:election_strategy: 3 2024-06-03T09:06:21.159 INFO:teuthology.orchestra.run.smithi088.stderr:0: v1:172.21.15.88:6789/0 mon.a 2024-06-03T09:06:21.159 INFO:teuthology.orchestra.run.smithi088.stderr:1: v1:172.21.15.192:6789/0 mon.b 2024-06-03T09:06:21.160 INFO:teuthology.orchestra.run.smithi088.stderr:2: v1:172.21.15.88:6790/0 mon.c 2024-06-03T09:06:21.160 INFO:teuthology.orchestra.run.smithi088.stderr: 2024-06-03T09:06:21.160 INFO:teuthology.orchestra.run.smithi088.stderr:2024-06-03T09:06:21.156+0000 7f461dffb640 1 -- 172.21.15.88:0/2463024474 <== mon.0 v1:172.21.15.88:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 764+0+0 (unknown 59110949 0 0) 0x7f460c004eb0 con 0x7f46280ac7a0 2024-06-03T09:06:21.160 INFO:teuthology.orchestra.run.smithi088.stderr:2024-06-03T09:06:21.156+0000 7f461dffb640 1 -- 172.21.15.88:0/2463024474 --> v1:172.21.15.88:6789/0 -- auth(proto 2 165 bytes epoch 0) v1 -- 0x7f46280ac4a0 con 0x7f46280ac7a0 2024-06-03T09:06:21.160 INFO:teuthology.orchestra.run.smithi088.stderr:2024-06-03T09:06:21.156+0000 7f461dffb640 1 -- 172.21.15.88:0/2463024474 <== mon.0 v1:172.21.15.88:6789/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 2234374065 0 0) 0x7f4620001bb0 con 0x7f46280ac7a0 2024-06-03T09:06:21.160 INFO:teuthology.orchestra.run.smithi088.stderr:2024-06-03T09:06:21.156+0000 7f461dffb640 10 monclient(hunting): handle_monmap mon_map magic: 0 v1 2024-06-03T09:06:21.160 INFO:teuthology.orchestra.run.smithi088.stderr:2024-06-03T09:06:21.156+0000 7f461dffb640 10 monclient(hunting): got monmap 1 from mon.a (according to old e1) 2024-06-03T09:06:21.160 INFO:teuthology.orchestra.run.smithi088.stderr:2024-06-03T09:06:21.156+0000 7f461dffb640 10 monclient(hunting): dump: 2024-06-03T09:06:21.160 INFO:teuthology.orchestra.run.smithi088.stderr:epoch 1 2024-06-03T09:06:21.160 INFO:teuthology.orchestra.run.smithi088.stderr:fsid ec3c3fdd-3ac6-4d7d-9412-683130ed8b04 2024-06-03T09:06:21.160 INFO:teuthology.orchestra.run.smithi088.stderr:last_changed 2024-06-03T08:34:31.885125+0000 2024-06-03T09:06:21.160 INFO:teuthology.orchestra.run.smithi088.stderr:created 2024-06-03T08:34:31.885125+0000 2024-06-03T09:06:21.160 INFO:teuthology.orchestra.run.smithi088.stderr:min_mon_release 18 (reef) 2024-06-03T09:06:21.160 INFO:teuthology.orchestra.run.smithi088.stderr:election_strategy: 3 2024-06-03T09:06:21.161 INFO:teuthology.orchestra.run.smithi088.stderr:0: v1:172.21.15.88:6789/0 mon.a 2024-06-03T09:06:21.161 INFO:teuthology.orchestra.run.smithi088.stderr:1: v1:172.21.15.192:6789/0 mon.b 2024-06-03T09:06:21.161 INFO:teuthology.orchestra.run.smithi088.stderr:2: v1:172.21.15.88:6790/0 mon.c 2024-06-03T09:06:21.161 INFO:teuthology.orchestra.run.smithi088.stderr: 2024-06-03T09:06:21.162 INFO:teuthology.orchestra.run.smithi088.stderr:2024-06-03T09:06:21.159+0000 7f461dffb640 1 -- 172.21.15.88:0/2463024474 <== mon.0 v1:172.21.15.88:6789/0 4 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 580+0+0 (unknown 2459101352 0 0) 0x7f46280ac4a0 con 0x7f46280ac7a0 2024-06-03T09:06:21.162 INFO:teuthology.orchestra.run.smithi088.stderr:2024-06-03T09:06:21.159+0000 7f461dffb640 1 -- 172.21.15.88:0/2463024474 >> v1:172.21.15.88:6790/0 conn(0x7f46280717c0 legacy=0x7f46281d2350 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-06-03T09:06:21.162 INFO:teuthology.orchestra.run.smithi088.stderr:2024-06-03T09:06:21.159+0000 7f461dffb640 1 -- 172.21.15.88:0/2463024474 >> v1:172.21.15.192:6789/0 conn(0x7f462806e050 legacy=0x7f46280ab940 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-06-03T09:06:21.162 INFO:teuthology.orchestra.run.smithi088.stderr:2024-06-03T09:06:21.159+0000 7f461dffb640 10 monclient: _finish_hunting 0 2024-06-03T09:06:21.162 INFO:teuthology.orchestra.run.smithi088.stderr:2024-06-03T09:06:21.159+0000 7f461dffb640 1 monclient: found mon.a 2024-06-03T09:06:21.162 INFO:teuthology.orchestra.run.smithi088.stderr:2024-06-03T09:06:21.159+0000 7f461dffb640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-06-03T09:06:21.162 INFO:teuthology.orchestra.run.smithi088.stderr:2024-06-03T09:06:21.159+0000 7f461dffb640 10 monclient: _send_mon_message to mon.a at v1:172.21.15.88:6789/0 2024-06-03T09:06:21.163 INFO:teuthology.orchestra.run.smithi088.stderr:2024-06-03T09:06:21.159+0000 7f461dffb640 1 -- 172.21.15.88:0/2463024474 --> v1:172.21.15.88:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f46280ac1e0 con 0x7f46280ac7a0 2024-06-03T09:06:21.163 INFO:teuthology.orchestra.run.smithi088.stderr:2024-06-03T09:06:21.159+0000 7f461dffb640 10 monclient: _finish_auth 0 2024-06-03T09:06:21.163 INFO:teuthology.orchestra.run.smithi088.stderr:2024-06-03T09:06:21.159+0000 7f461dffb640 10 monclient: _check_auth_tickets 2024-06-03T09:06:21.163 INFO:teuthology.orchestra.run.smithi088.stderr:2024-06-03T09:06:21.159+0000 7f461dffb640 20 monclient: _check_auth_rotating not needed by client.admin 2024-06-03T09:06:21.163 INFO:teuthology.orchestra.run.smithi088.stderr:2024-06-03T09:06:21.159+0000 7f462fa0d640 5 monclient: authenticate success, global_id 30585 2024-06-03T09:06:21.163 INFO:teuthology.orchestra.run.smithi088.stderr:2024-06-03T09:06:21.159+0000 7f462fa0d640 10 monclient: _renew_subs 2024-06-03T09:06:21.163 INFO:teuthology.orchestra.run.smithi088.stderr:2024-06-03T09:06:21.159+0000 7f462fa0d640 10 monclient: _send_mon_message to mon.a at v1:172.21.15.88:6789/0 2024-06-03T09:06:21.163 INFO:teuthology.orchestra.run.smithi088.stderr:2024-06-03T09:06:21.159+0000 7f462fa0d640 1 -- 172.21.15.88:0/2463024474 --> v1:172.21.15.88:6789/0 -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f46281ddbf0 con 0x7f46280ac7a0 2024-06-03T09:06:21.163 INFO:teuthology.orchestra.run.smithi088.stderr:2024-06-03T09:06:21.159+0000 7f462fa0d640 10 client.30585.objecter _maybe_request_map subscribing (onetime) to next osd map 2024-06-03T09:06:21.163 INFO:teuthology.orchestra.run.smithi088.stderr:2024-06-03T09:06:21.159+0000 7f462fa0d640 10 monclient: _renew_subs 2024-06-03T09:06:21.163 INFO:teuthology.orchestra.run.smithi088.stderr:2024-06-03T09:06:21.159+0000 7f462fa0d640 10 monclient: _send_mon_message to mon.a at v1:172.21.15.88:6789/0 2024-06-03T09:06:21.163 INFO:teuthology.orchestra.run.smithi088.stderr:2024-06-03T09:06:21.159+0000 7f462fa0d640 1 -- 172.21.15.88:0/2463024474 --> v1:172.21.15.88:6789/0 -- mon_subscribe({osdmap=0}) v3 -- 0x7f46281ddfc0 con 0x7f46280ac7a0 2024-06-03T09:06:21.163 INFO:teuthology.orchestra.run.smithi088.stderr:2024-06-03T09:06:21.159+0000 7f462fa0d640 1 librados: init done 2024-06-03T09:06:21.164 INFO:teuthology.orchestra.run.smithi088.stderr:2024-06-03T09:06:21.160+0000 7f462fa0d640 10 start_mon_command cmd=[{"prefix": "get_command_descriptions"}] 2024-06-03T09:06:21.164 INFO:teuthology.orchestra.run.smithi088.stderr:2024-06-03T09:06:21.160+0000 7f462fa0d640 10 monclient: _send_command 1 [{"prefix": "get_command_descriptions"}] 2024-06-03T09:06:21.164 INFO:teuthology.orchestra.run.smithi088.stderr:2024-06-03T09:06:21.160+0000 7f462fa0d640 10 monclient: _send_mon_message to mon.a at v1:172.21.15.88:6789/0 2024-06-03T09:06:21.164 INFO:teuthology.orchestra.run.smithi088.stderr:2024-06-03T09:06:21.160+0000 7f462fa0d640 1 -- 172.21.15.88:0/2463024474 --> v1:172.21.15.88:6789/0 -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f462804ea60 con 0x7f46280ac7a0 2024-06-03T09:06:21.176 INFO:teuthology.orchestra.run.smithi088.stderr:2024-06-03T09:06:21.176+0000 7f461dffb640 1 -- 172.21.15.88:0/2463024474 <== mon.0 v1:172.21.15.88:6789/0 5 ==== config(0 keys) v1 ==== 4+0+0 (unknown 0 0 0) 0x7f46200020c0 con 0x7f46280ac7a0 2024-06-03T09:06:21.176 INFO:teuthology.orchestra.run.smithi088.stderr:2024-06-03T09:06:21.176+0000 7f461dffb640 10 monclient: handle_config config(0 keys) v1 2024-06-03T09:06:21.176 INFO:teuthology.orchestra.run.smithi088.stderr:2024-06-03T09:06:21.176+0000 7f461dffb640 1 -- 172.21.15.88:0/2463024474 <== mon.0 v1:172.21.15.88:6789/0 6 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 2234374065 0 0) 0x7f4620004d80 con 0x7f46280ac7a0 2024-06-03T09:06:21.176 INFO:teuthology.orchestra.run.smithi088.stderr:2024-06-03T09:06:21.176+0000 7f461dffb640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-06-03T09:06:21.176 INFO:teuthology.orchestra.run.smithi088.stderr:2024-06-03T09:06:21.176+0000 7f461dffb640 10 monclient: got monmap 1 from mon.a (according to old e1) 2024-06-03T09:06:21.176 INFO:teuthology.orchestra.run.smithi088.stderr:2024-06-03T09:06:21.176+0000 7f461dffb640 10 monclient: dump: 2024-06-03T09:06:21.176 INFO:teuthology.orchestra.run.smithi088.stderr:epoch 1 2024-06-03T09:06:21.176 INFO:teuthology.orchestra.run.smithi088.stderr:fsid ec3c3fdd-3ac6-4d7d-9412-683130ed8b04 2024-06-03T09:06:21.177 INFO:teuthology.orchestra.run.smithi088.stderr:last_changed 2024-06-03T08:34:31.885125+0000 2024-06-03T09:06:21.177 INFO:teuthology.orchestra.run.smithi088.stderr:created 2024-06-03T08:34:31.885125+0000 2024-06-03T09:06:21.177 INFO:teuthology.orchestra.run.smithi088.stderr:min_mon_release 18 (reef) 2024-06-03T09:06:21.177 INFO:teuthology.orchestra.run.smithi088.stderr:election_strategy: 3 2024-06-03T09:06:21.177 INFO:teuthology.orchestra.run.smithi088.stderr:0: v1:172.21.15.88:6789/0 mon.a 2024-06-03T09:06:21.177 INFO:teuthology.orchestra.run.smithi088.stderr:1: v1:172.21.15.192:6789/0 mon.b 2024-06-03T09:06:21.177 INFO:teuthology.orchestra.run.smithi088.stderr:2: v1:172.21.15.88:6790/0 mon.c 2024-06-03T09:06:21.177 INFO:teuthology.orchestra.run.smithi088.stderr: 2024-06-03T09:06:21.177 INFO:teuthology.orchestra.run.smithi088.stderr:2024-06-03T09:06:21.176+0000 7f461f7fe640 4 set_mon_vals no callback set 2024-06-03T09:06:21.178 INFO:teuthology.orchestra.run.smithi088.stderr:2024-06-03T09:06:21.176+0000 7f461dffb640 1 -- 172.21.15.88:0/2463024474 <== mon.0 v1:172.21.15.88:6789/0 7 ==== mgrmap(e 5) v1 ==== 89110+0+0 (unknown 873533973 0 0) 0x7f462001ab00 con 0x7f46280ac7a0 2024-06-03T09:06:21.178 INFO:teuthology.orchestra.run.smithi088.stderr:2024-06-03T09:06:21.177+0000 7f462d7a9640 10 monclient: get_auth_request con 0x7f460c06d460 auth_method 0 2024-06-03T09:06:21.178 INFO:teuthology.orchestra.run.smithi088.stderr:2024-06-03T09:06:21.178+0000 7f461dffb640 10 client.30585.objecter ms_handle_connect 0x7f460c06d460 2024-06-03T09:06:21.179 INFO:teuthology.orchestra.run.smithi088.stderr:2024-06-03T09:06:21.179+0000 7f461dffb640 1 -- 172.21.15.88:0/2463024474 <== mon.0 v1:172.21.15.88:6789/0 8 ==== osd_map(1010..1010 src has 622..1010) v4 ==== 5229+0+0 (unknown 1105883993 0 0) 0x7f4620054340 con 0x7f46280ac7a0 2024-06-03T09:06:21.179 INFO:teuthology.orchestra.run.smithi088.stderr:2024-06-03T09:06:21.179+0000 7f461dffb640 10 client.30585.objecter ms_dispatch 0x7f4628000b90 osd_map(1010..1010 src has 622..1010) v4 2024-06-03T09:06:21.179 INFO:teuthology.orchestra.run.smithi088.stderr:2024-06-03T09:06:21.179+0000 7f461dffb640 3 client.30585.objecter handle_osd_map got epochs [1010,1010] > 0 2024-06-03T09:06:21.179 INFO:teuthology.orchestra.run.smithi088.stderr:2024-06-03T09:06:21.179+0000 7f461dffb640 3 client.30585.objecter handle_osd_map decoding full epoch 1010 2024-06-03T09:06:21.179 INFO:teuthology.orchestra.run.smithi088.stderr:2024-06-03T09:06:21.179+0000 7f461dffb640 20 client.30585.objecter dump_active .. 0 homeless 2024-06-03T09:06:21.412 INFO:teuthology.orchestra.run.smithi088.stderr:2024-06-03T09:06:21.412+0000 7f461dffb640 1 -- 172.21.15.88:0/2463024474 <== mon.0 v1:172.21.15.88:6789/0 9 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+182498 (unknown 1092875540 0 3016760700) 0x7f462004fd20 con 0x7f46280ac7a0 2024-06-03T09:06:21.412 INFO:teuthology.orchestra.run.smithi088.stderr:2024-06-03T09:06:21.412+0000 7f461dffb640 10 monclient: handle_mon_command_ack 1 [{"prefix": "get_command_descriptions"}] 2024-06-03T09:06:21.412 INFO:teuthology.orchestra.run.smithi088.stderr:2024-06-03T09:06:21.412+0000 7f461dffb640 10 monclient: _finish_command 1 = system:0 2024-06-03T09:06:21.617 INFO:teuthology.orchestra.run.smithi088.stderr:2024-06-03T09:06:21.617+0000 7f462fa0d640 1 -- 172.21.15.88:0/2463024474 --> v1:172.21.15.192:6800/34229 -- mgr_command(tid 0: {"prefix": "pg dump", "target": ["mon-mgr", ""], "format": "json"}) v1 -- 0x7f462806eeb0 con 0x7f460c06d460 2024-06-03T09:06:21.626 INFO:teuthology.orchestra.run.smithi088.stderr:2024-06-03T09:06:21.626+0000 7f461dffb640 1 -- 172.21.15.88:0/2463024474 <== mgr.4110 v1:172.21.15.192:6800/34229 1 ==== mgr_command_reply(tid 0: 0 dumped all) v1 ==== 18+0+151744 (unknown 2965378022 0 78872612) 0x7f46240077a0 con 0x7f460c06d460 2024-06-03T09:06:21.627 INFO:teuthology.orchestra.run.smithi088.stdout: 2024-06-03T09:06:21.628 INFO:teuthology.orchestra.run.smithi088.stderr:dumped all 2024-06-03T09:06:21.631 INFO:teuthology.orchestra.run.smithi088.stderr:2024-06-03T09:06:21.628+0000 7f462fa0d640 10 librados: watch_flush enter 2024-06-03T09:06:21.631 INFO:teuthology.orchestra.run.smithi088.stderr:2024-06-03T09:06:21.628+0000 7f462fa0d640 10 librados: watch_flush exit 2024-06-03T09:06:21.631 INFO:teuthology.orchestra.run.smithi088.stderr:2024-06-03T09:06:21.628+0000 7f462fa0d640 20 client.30585.objecter shutdown clearing up homeless session... 2024-06-03T09:06:21.631 INFO:teuthology.orchestra.run.smithi088.stderr:2024-06-03T09:06:21.628+0000 7f462fa0d640 10 client.30585.objecter successfully canceled tick 2024-06-03T09:06:21.631 INFO:teuthology.orchestra.run.smithi088.stderr:2024-06-03T09:06:21.628+0000 7f462fa0d640 1 -- 172.21.15.88:0/2463024474 >> v1:172.21.15.192:6800/34229 conn(0x7f460c06d460 legacy=0x7f460c06f920 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-06-03T09:06:21.631 INFO:teuthology.orchestra.run.smithi088.stderr:2024-06-03T09:06:21.628+0000 7f462fa0d640 10 monclient: shutdown 2024-06-03T09:06:21.631 INFO:teuthology.orchestra.run.smithi088.stderr:2024-06-03T09:06:21.628+0000 7f462fa0d640 20 monclient: shutdown discarding 0 pending message(s) 2024-06-03T09:06:21.631 INFO:teuthology.orchestra.run.smithi088.stderr:2024-06-03T09:06:21.628+0000 7f462fa0d640 1 -- 172.21.15.88:0/2463024474 >> v1:172.21.15.88:6789/0 conn(0x7f46280ac7a0 legacy=0x7f46281d2a60 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-06-03T09:06:21.631 INFO:teuthology.orchestra.run.smithi088.stderr:2024-06-03T09:06:21.628+0000 7f462fa0d640 1 -- 172.21.15.88:0/2463024474 shutdown_connections 2024-06-03T09:06:21.631 INFO:teuthology.orchestra.run.smithi088.stderr:2024-06-03T09:06:21.628+0000 7f462fa0d640 1 -- 172.21.15.88:0/2463024474 >> 172.21.15.88:0/2463024474 conn(0x7f46280b2f20 msgr2=0x7f46280b3300 unknown :-1 s=STATE_NONE l=0).mark_down 2024-06-03T09:06:21.631 INFO:teuthology.orchestra.run.smithi088.stderr:2024-06-03T09:06:21.628+0000 7f462fa0d640 1 -- 172.21.15.88:0/2463024474 shutdown_connections 2024-06-03T09:06:21.632 INFO:teuthology.orchestra.run.smithi088.stderr:2024-06-03T09:06:21.628+0000 7f462fa0d640 1 -- 172.21.15.88:0/2463024474 wait complete. 2024-06-03T09:06:21.632 INFO:teuthology.orchestra.run.smithi088.stderr:2024-06-03T09:06:21.628+0000 7f462fa0d640 1 librados: shutdown 2024-06-03T09:06:21.639 INFO:teuthology.orchestra.run.smithi088.stdout:{"pg_ready":true,"pg_map":{"version":1916,"stamp":"2024-06-03T09:06:21.098664+0000","last_osdmap_epoch":0,"last_pg_scan":0,"pg_stats_sum":{"stat_sum":{"num_bytes":459285,"num_objects":3,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":1,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":3,"num_whiteouts":0,"num_read":102,"num_read_kb":88,"num_write":120,"num_write_kb":1379,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2,"num_bytes_recovered":397840,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":83,"ondisk_log_size":83,"up":68,"acting":68,"num_store_stats":0},"osd_stats_sum":{"up_from":0,"seq":0,"num_pgs":100,"num_osds":8,"num_per_pool_osds":8,"num_per_pool_omap_osds":8,"kb":754974720,"kb_used":494540,"kb_used_data":46312,"kb_used_omap":12,"kb_used_meta":448115,"kb_avail":754480180,"statfs":{"total":773094113280,"available":772587704320,"internally_reserved":0,"allocated":47423488,"data_stored":22742525,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":12744,"internal_metadata":458870328},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":146,"apply_latency_ms":146,"commit_latency_ns":146000000,"apply_latency_ns":146000000},"alerts":[],"network_ping_times":[]},"pg_stats_delta":{"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":0,"acting":0,"num_store_stats":0,"stamp_delta":"8.001643"},"pg_stats":[{"pgid":"17.d","version":"0'0","reported_seq":942,"reported_epoch":1009,"state":"active+clean","last_fresh":"2024-06-03T09:06:14.443917+0000","last_change":"2024-06-03T08:38:19.260548+0000","last_active":"2024-06-03T09:06:14.443917+0000","last_peered":"2024-06-03T09:06:14.443917+0000","last_clean":"2024-06-03T09:06:14.443917+0000","last_became_active":"2024-06-03T08:38:19.253559+0000","last_became_peered":"2024-06-03T08:38:19.253559+0000","last_unstale":"2024-06-03T09:06:14.443917+0000","last_undegraded":"2024-06-03T09:06:14.443917+0000","last_fullsized":"2024-06-03T09:06:14.443917+0000","mapping_epoch":28,"log_start":"0'0","ondisk_log_start":"0'0","created":28,"last_epoch_clean":45,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-06-03T08:37:55.614497+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-06-03T08:37:55.614497+0000","last_clean_scrub_stamp":"2024-06-03T08:37:55.614497+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-06-04T19:00:12.470979+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1],"acting":[0,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"17.c","version":"0'0","reported_seq":953,"reported_epoch":1009,"state":"active+undersized","last_fresh":"2024-06-03T09:06:13.668156+0000","last_change":"2024-06-03T09:06:03.440074+0000","last_active":"2024-06-03T09:06:13.668156+0000","last_peered":"2024-06-03T09:06:13.668156+0000","last_clean":"2024-06-03T09:05:02.105638+0000","last_became_active":"2024-06-03T09:06:03.440074+0000","last_became_peered":"2024-06-03T09:06:03.440074+0000","last_unstale":"2024-06-03T09:06:13.668156+0000","last_undegraded":"2024-06-03T09:06:13.668156+0000","last_fullsized":"2024-06-03T09:06:03.377982+0000","mapping_epoch":999,"log_start":"0'0","ondisk_log_start":"0'0","created":28,"last_epoch_clean":255,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-06-03T08:37:55.614497+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-06-03T08:37:55.614497+0000","last_clean_scrub_stamp":"2024-06-03T08:37:55.614497+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-06-04T10:34:20.808557+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3],"acting":[3],"avail_no_missing":["3"],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"17.f","version":"0'0","reported_seq":148,"reported_epoch":1009,"state":"active+undersized","last_fresh":"2024-06-03T09:06:14.450543+0000","last_change":"2024-06-03T09:06:03.420285+0000","last_active":"2024-06-03T09:06:14.450543+0000","last_peered":"2024-06-03T09:06:14.450543+0000","last_clean":"2024-06-03T08:37:55.614497+0000","last_became_active":"2024-06-03T09:06:03.420285+0000","last_became_peered":"2024-06-03T09:06:03.420285+0000","last_unstale":"2024-06-03T09:06:14.450543+0000","last_undegraded":"2024-06-03T09:06:14.450543+0000","last_fullsized":"2024-06-03T09:06:03.380616+0000","mapping_epoch":999,"log_start":"0'0","ondisk_log_start":"0'0","created":28,"last_epoch_clean":255,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-06-03T08:37:55.614497+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-06-03T08:37:55.614497+0000","last_clean_scrub_stamp":"2024-06-03T08:37:55.614497+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-06-04T15:39:41.187731+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4],"acting":[4],"avail_no_missing":["4"],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"17.e","version":"0'0","reported_seq":149,"reported_epoch":1009,"state":"active+undersized","last_fresh":"2024-06-03T09:06:13.726992+0000","last_change":"2024-06-03T09:06:03.412887+0000","last_active":"2024-06-03T09:06:13.726992+0000","last_peered":"2024-06-03T09:06:13.726992+0000","last_clean":"2024-06-03T08:37:55.614497+0000","last_became_active":"2024-06-03T09:06:03.412887+0000","last_became_peered":"2024-06-03T09:06:03.412887+0000","last_unstale":"2024-06-03T09:06:13.726992+0000","last_undegraded":"2024-06-03T09:06:13.726992+0000","last_fullsized":"2024-06-03T09:06:03.332482+0000","mapping_epoch":999,"log_start":"0'0","ondisk_log_start":"0'0","created":28,"last_epoch_clean":255,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-06-03T08:37:55.614497+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-06-03T08:37:55.614497+0000","last_clean_scrub_stamp":"2024-06-03T08:37:55.614497+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-06-04T12:23:22.511601+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5],"acting":[5],"avail_no_missing":["5"],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"17.9","version":"0'0","reported_seq":946,"reported_epoch":1009,"state":"active+clean","last_fresh":"2024-06-03T09:06:14.999299+0000","last_change":"2024-06-03T08:38:04.772260+0000","last_active":"2024-06-03T09:06:14.999299+0000","last_peered":"2024-06-03T09:06:14.999299+0000","last_clean":"2024-06-03T09:06:14.999299+0000","last_became_active":"2024-06-03T08:38:04.763428+0000","last_became_peered":"2024-06-03T08:38:04.763428+0000","last_unstale":"2024-06-03T09:06:14.999299+0000","last_undegraded":"2024-06-03T09:06:14.999299+0000","last_fullsized":"2024-06-03T09:06:14.999299+0000","mapping_epoch":28,"log_start":"0'0","ondisk_log_start":"0'0","created":28,"last_epoch_clean":33,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-06-03T08:37:55.614497+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-06-03T08:37:55.614497+0000","last_clean_scrub_stamp":"2024-06-03T08:37:55.614497+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-06-04T10:55:10.773936+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,4],"acting":[7,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"17.8","version":"0'0","reported_seq":961,"reported_epoch":1009,"state":"active+undersized","last_fresh":"2024-06-03T09:06:14.430375+0000","last_change":"2024-06-03T09:06:03.388786+0000","last_active":"2024-06-03T09:06:14.430375+0000","last_peered":"2024-06-03T09:06:14.430375+0000","last_clean":"2024-06-03T09:04:43.297780+0000","last_became_active":"2024-06-03T09:06:03.388786+0000","last_became_peered":"2024-06-03T09:06:03.388786+0000","last_unstale":"2024-06-03T09:06:14.430375+0000","last_undegraded":"2024-06-03T09:06:14.430375+0000","last_fullsized":"2024-06-03T09:06:03.331754+0000","mapping_epoch":999,"log_start":"0'0","ondisk_log_start":"0'0","created":28,"last_epoch_clean":255,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-06-03T08:37:55.614497+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-06-03T08:37:55.614497+0000","last_clean_scrub_stamp":"2024-06-03T08:37:55.614497+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-06-04T09:18:44.765547+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0],"acting":[0],"avail_no_missing":["0"],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"17.b","version":"0'0","reported_seq":149,"reported_epoch":1009,"state":"active+undersized","last_fresh":"2024-06-03T09:06:14.433735+0000","last_change":"2024-06-03T09:06:03.451203+0000","last_active":"2024-06-03T09:06:14.433735+0000","last_peered":"2024-06-03T09:06:14.433735+0000","last_clean":"2024-06-03T08:37:55.614497+0000","last_became_active":"2024-06-03T09:06:03.451203+0000","last_became_peered":"2024-06-03T09:06:03.451203+0000","last_unstale":"2024-06-03T09:06:14.433735+0000","last_undegraded":"2024-06-03T09:06:14.433735+0000","last_fullsized":"2024-06-03T09:06:03.375523+0000","mapping_epoch":999,"log_start":"0'0","ondisk_log_start":"0'0","created":28,"last_epoch_clean":255,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-06-03T08:37:55.614497+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-06-03T08:37:55.614497+0000","last_clean_scrub_stamp":"2024-06-03T08:37:55.614497+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-06-04T10:16:19.813495+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2],"acting":[2],"avail_no_missing":["2"],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"17.a","version":"0'0","reported_seq":924,"reported_epoch":1009,"state":"active+clean","last_fresh":"2024-06-03T09:06:13.710307+0000","last_change":"2024-06-03T08:38:05.843142+0000","last_active":"2024-06-03T09:06:13.710307+0000","last_peered":"2024-06-03T09:06:13.710307+0000","last_clean":"2024-06-03T09:06:13.710307+0000","last_became_active":"2024-06-03T08:38:05.809257+0000","last_became_peered":"2024-06-03T08:38:05.809257+0000","last_unstale":"2024-06-03T09:06:13.710307+0000","last_undegraded":"2024-06-03T09:06:13.710307+0000","last_fullsized":"2024-06-03T09:06:13.710307+0000","mapping_epoch":28,"log_start":"0'0","ondisk_log_start":"0'0","created":28,"last_epoch_clean":33,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-06-03T08:37:55.614497+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-06-03T08:37:55.614497+0000","last_clean_scrub_stamp":"2024-06-03T08:37:55.614497+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-06-04T12:12:03.686838+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2],"acting":[1,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"17.5","version":"0'0","reported_seq":943,"reported_epoch":1009,"state":"active+clean","last_fresh":"2024-06-03T09:06:14.442357+0000","last_change":"2024-06-03T08:38:23.466051+0000","last_active":"2024-06-03T09:06:14.442357+0000","last_peered":"2024-06-03T09:06:14.442357+0000","last_clean":"2024-06-03T09:06:14.442357+0000","last_became_active":"2024-06-03T08:38:23.458039+0000","last_became_peered":"2024-06-03T08:38:23.458039+0000","last_unstale":"2024-06-03T09:06:14.442357+0000","last_undegraded":"2024-06-03T09:06:14.442357+0000","last_fullsized":"2024-06-03T09:06:14.442357+0000","mapping_epoch":28,"log_start":"0'0","ondisk_log_start":"0'0","created":28,"last_epoch_clean":49,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-06-03T08:37:55.614497+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-06-03T08:37:55.614497+0000","last_clean_scrub_stamp":"2024-06-03T08:37:55.614497+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-06-04T12:27:02.532436+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,4],"acting":[0,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"17.4","version":"0'0","reported_seq":149,"reported_epoch":1009,"state":"active+undersized","last_fresh":"2024-06-03T09:06:13.941522+0000","last_change":"2024-06-03T09:06:03.441889+0000","last_active":"2024-06-03T09:06:13.941522+0000","last_peered":"2024-06-03T09:06:13.941522+0000","last_clean":"2024-06-03T08:37:55.614497+0000","last_became_active":"2024-06-03T09:06:03.441889+0000","last_became_peered":"2024-06-03T09:06:03.441889+0000","last_unstale":"2024-06-03T09:06:13.941522+0000","last_undegraded":"2024-06-03T09:06:13.941522+0000","last_fullsized":"2024-06-03T09:06:03.359582+0000","mapping_epoch":999,"log_start":"0'0","ondisk_log_start":"0'0","created":28,"last_epoch_clean":255,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-06-03T08:37:55.614497+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-06-03T08:37:55.614497+0000","last_clean_scrub_stamp":"2024-06-03T08:37:55.614497+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-06-04T12:49:22.907141+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7],"acting":[7],"avail_no_missing":["7"],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"17.7","version":"0'0","reported_seq":929,"reported_epoch":1009,"state":"active+clean","last_fresh":"2024-06-03T09:06:13.736920+0000","last_change":"2024-06-03T08:38:04.837083+0000","last_active":"2024-06-03T09:06:13.736920+0000","last_peered":"2024-06-03T09:06:13.736920+0000","last_clean":"2024-06-03T09:06:13.736920+0000","last_became_active":"2024-06-03T08:38:04.818246+0000","last_became_peered":"2024-06-03T08:38:04.818246+0000","last_unstale":"2024-06-03T09:06:13.736920+0000","last_undegraded":"2024-06-03T09:06:13.736920+0000","last_fullsized":"2024-06-03T09:06:13.736920+0000","mapping_epoch":28,"log_start":"0'0","ondisk_log_start":"0'0","created":28,"last_epoch_clean":34,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-06-03T08:37:55.614497+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-06-03T08:37:55.614497+0000","last_clean_scrub_stamp":"2024-06-03T08:37:55.614497+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-06-04T17:02:38.204813+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,2],"acting":[5,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"17.6","version":"0'0","reported_seq":960,"reported_epoch":1009,"state":"active+undersized","last_fresh":"2024-06-03T09:06:14.453110+0000","last_change":"2024-06-03T09:06:03.428677+0000","last_active":"2024-06-03T09:06:14.453110+0000","last_peered":"2024-06-03T09:06:14.453110+0000","last_clean":"2024-06-03T09:04:43.312352+0000","last_became_active":"2024-06-03T09:06:03.428677+0000","last_became_peered":"2024-06-03T09:06:03.428677+0000","last_unstale":"2024-06-03T09:06:14.453110+0000","last_undegraded":"2024-06-03T09:06:14.453110+0000","last_fullsized":"2024-06-03T09:06:03.359909+0000","mapping_epoch":999,"log_start":"0'0","ondisk_log_start":"0'0","created":28,"last_epoch_clean":255,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-06-03T08:37:55.614497+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-06-03T08:37:55.614497+0000","last_clean_scrub_stamp":"2024-06-03T08:37:55.614497+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-06-04T15:38:02.180241+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0],"acting":[0],"avail_no_missing":["0"],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"2.7","version":"0'0","reported_seq":940,"reported_epoch":1009,"state":"active+clean","last_fresh":"2024-06-03T09:06:13.684065+0000","last_change":"2024-06-03T08:36:30.385405+0000","last_active":"2024-06-03T09:06:13.684065+0000","last_peered":"2024-06-03T09:06:13.684065+0000","last_clean":"2024-06-03T09:06:13.684065+0000","last_became_active":"2024-06-03T08:36:30.380302+0000","last_became_peered":"2024-06-03T08:36:30.380302+0000","last_unstale":"2024-06-03T09:06:13.684065+0000","last_undegraded":"2024-06-03T09:06:13.684065+0000","last_fullsized":"2024-06-03T09:06:13.684065+0000","mapping_epoch":23,"log_start":"0'0","ondisk_log_start":"0'0","created":23,"last_epoch_clean":24,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-06-03T08:36:28.444764+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-06-03T08:36:28.444764+0000","last_clean_scrub_stamp":"2024-06-03T08:36:28.444764+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-06-04T18:46:33.475181+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0],"acting":[3,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"17.14","version":"0'0","reported_seq":926,"reported_epoch":1009,"state":"active+clean","last_fresh":"2024-06-03T09:06:13.721460+0000","last_change":"2024-06-03T08:38:04.547565+0000","last_active":"2024-06-03T09:06:13.721460+0000","last_peered":"2024-06-03T09:06:13.721460+0000","last_clean":"2024-06-03T09:06:13.721460+0000","last_became_active":"2024-06-03T08:38:04.541781+0000","last_became_peered":"2024-06-03T08:38:04.541781+0000","last_unstale":"2024-06-03T09:06:13.721460+0000","last_undegraded":"2024-06-03T09:06:13.721460+0000","last_fullsized":"2024-06-03T09:06:13.721460+0000","mapping_epoch":28,"log_start":"0'0","ondisk_log_start":"0'0","created":28,"last_epoch_clean":34,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-06-03T08:37:55.614497+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-06-03T08:37:55.614497+0000","last_clean_scrub_stamp":"2024-06-03T08:37:55.614497+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-06-04T16:41:55.170134+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,2],"acting":[5,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"2.6","version":"0'0","reported_seq":949,"reported_epoch":1009,"state":"active+clean","last_fresh":"2024-06-03T09:06:14.448100+0000","last_change":"2024-06-03T08:36:30.430760+0000","last_active":"2024-06-03T09:06:14.448100+0000","last_peered":"2024-06-03T09:06:14.448100+0000","last_clean":"2024-06-03T09:06:14.448100+0000","last_became_active":"2024-06-03T08:36:30.404665+0000","last_became_peered":"2024-06-03T08:36:30.404665+0000","last_unstale":"2024-06-03T09:06:14.448100+0000","last_undegraded":"2024-06-03T09:06:14.448100+0000","last_fullsized":"2024-06-03T09:06:14.448100+0000","mapping_epoch":23,"log_start":"0'0","ondisk_log_start":"0'0","created":23,"last_epoch_clean":24,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-06-03T08:36:28.444764+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-06-03T08:36:28.444764+0000","last_clean_scrub_stamp":"2024-06-03T08:36:28.444764+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-06-04T10:37:37.746074+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,0],"acting":[4,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"17.15","version":"0'0","reported_seq":934,"reported_epoch":1009,"state":"active+clean","last_fresh":"2024-06-03T09:06:13.672940+0000","last_change":"2024-06-03T08:38:20.999903+0000","last_active":"2024-06-03T09:06:13.672940+0000","last_peered":"2024-06-03T09:06:13.672940+0000","last_clean":"2024-06-03T09:06:13.672940+0000","last_became_active":"2024-06-03T08:38:20.979523+0000","last_became_peered":"2024-06-03T08:38:20.979523+0000","last_unstale":"2024-06-03T09:06:13.672940+0000","last_undegraded":"2024-06-03T09:06:13.672940+0000","last_fullsized":"2024-06-03T09:06:13.672940+0000","mapping_epoch":28,"log_start":"0'0","ondisk_log_start":"0'0","created":28,"last_epoch_clean":47,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-06-03T08:37:55.614497+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-06-03T08:37:55.614497+0000","last_clean_scrub_stamp":"2024-06-03T08:37:55.614497+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-06-04T20:29:58.314289+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,1],"acting":[3,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"2.5","version":"0'0","reported_seq":941,"reported_epoch":1009,"state":"active+clean","last_fresh":"2024-06-03T09:06:13.674542+0000","last_change":"2024-06-03T08:36:30.024235+0000","last_active":"2024-06-03T09:06:13.674542+0000","last_peered":"2024-06-03T09:06:13.674542+0000","last_clean":"2024-06-03T09:06:13.674542+0000","last_became_active":"2024-06-03T08:36:30.019040+0000","last_became_peered":"2024-06-03T08:36:30.019040+0000","last_unstale":"2024-06-03T09:06:13.674542+0000","last_undegraded":"2024-06-03T09:06:13.674542+0000","last_fullsized":"2024-06-03T09:06:13.674542+0000","mapping_epoch":23,"log_start":"0'0","ondisk_log_start":"0'0","created":23,"last_epoch_clean":24,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-06-03T08:36:28.444764+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-06-03T08:36:28.444764+0000","last_clean_scrub_stamp":"2024-06-03T08:36:28.444764+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-06-04T15:46:47.109502+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,4],"acting":[3,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"17.16","version":"0'0","reported_seq":972,"reported_epoch":1009,"state":"active+undersized","last_fresh":"2024-06-03T09:06:14.441665+0000","last_change":"2024-06-03T09:06:03.478615+0000","last_active":"2024-06-03T09:06:14.441665+0000","last_peered":"2024-06-03T09:06:14.441665+0000","last_clean":"2024-06-03T09:04:43.328415+0000","last_became_active":"2024-06-03T09:06:03.478615+0000","last_became_peered":"2024-06-03T09:06:03.478615+0000","last_unstale":"2024-06-03T09:06:14.441665+0000","last_undegraded":"2024-06-03T09:06:14.441665+0000","last_fullsized":"2024-06-03T09:06:03.402504+0000","mapping_epoch":999,"log_start":"0'0","ondisk_log_start":"0'0","created":28,"last_epoch_clean":255,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-06-03T08:37:55.614497+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-06-03T08:37:55.614497+0000","last_clean_scrub_stamp":"2024-06-03T08:37:55.614497+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-06-04T08:47:08.079233+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2],"acting":[2],"avail_no_missing":["2"],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"2.4","version":"0'0","reported_seq":952,"reported_epoch":1010,"state":"active+undersized","last_fresh":"2024-06-03T09:06:18.331485+0000","last_change":"2024-06-03T09:06:03.387171+0000","last_active":"2024-06-03T09:06:18.331485+0000","last_peered":"2024-06-03T09:06:18.331485+0000","last_clean":"2024-06-03T09:04:43.202645+0000","last_became_active":"2024-06-03T09:06:03.387171+0000","last_became_peered":"2024-06-03T09:06:03.387171+0000","last_unstale":"2024-06-03T09:06:18.331485+0000","last_undegraded":"2024-06-03T09:06:18.331485+0000","last_fullsized":"2024-06-03T09:06:03.283787+0000","mapping_epoch":999,"log_start":"0'0","ondisk_log_start":"0'0","created":23,"last_epoch_clean":255,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-06-03T08:36:28.444764+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-06-03T08:36:28.444764+0000","last_clean_scrub_stamp":"2024-06-03T08:36:28.444764+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-06-04T20:17:43.485023+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5],"acting":[5],"avail_no_missing":["5"],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"17.17","version":"0'0","reported_seq":954,"reported_epoch":1009,"state":"active+clean","last_fresh":"2024-06-03T09:06:14.447046+0000","last_change":"2024-06-03T08:38:38.483275+0000","last_active":"2024-06-03T09:06:14.447046+0000","last_peered":"2024-06-03T09:06:14.447046+0000","last_clean":"2024-06-03T09:06:14.447046+0000","last_became_active":"2024-06-03T08:38:38.467916+0000","last_became_peered":"2024-06-03T08:38:38.467916+0000","last_unstale":"2024-06-03T09:06:14.447046+0000","last_undegraded":"2024-06-03T09:06:14.447046+0000","last_fullsized":"2024-06-03T09:06:14.447046+0000","mapping_epoch":28,"log_start":"0'0","ondisk_log_start":"0'0","created":28,"last_epoch_clean":61,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-06-03T08:37:55.614497+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-06-03T08:37:55.614497+0000","last_clean_scrub_stamp":"2024-06-03T08:37:55.614497+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-06-04T10:02:23.036420+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,7],"acting":[2,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"2.2","version":"0'0","reported_seq":963,"reported_epoch":1009,"state":"active+undersized","last_fresh":"2024-06-03T09:06:13.657892+0000","last_change":"2024-06-03T09:06:03.410020+0000","last_active":"2024-06-03T09:06:13.657892+0000","last_peered":"2024-06-03T09:06:13.657892+0000","last_clean":"2024-06-03T09:05:37.145889+0000","last_became_active":"2024-06-03T09:06:03.410020+0000","last_became_peered":"2024-06-03T09:06:03.410020+0000","last_unstale":"2024-06-03T09:06:13.657892+0000","last_undegraded":"2024-06-03T09:06:13.657892+0000","last_fullsized":"2024-06-03T09:06:03.346864+0000","mapping_epoch":999,"log_start":"0'0","ondisk_log_start":"0'0","created":23,"last_epoch_clean":255,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-06-03T08:36:28.444764+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-06-03T08:36:28.444764+0000","last_clean_scrub_stamp":"2024-06-03T08:36:28.444764+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-06-04T09:57:12.423360+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3],"acting":[3],"avail_no_missing":["3"],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"17.11","version":"0'0","reported_seq":944,"reported_epoch":1009,"state":"active+clean","last_fresh":"2024-06-03T09:06:14.426670+0000","last_change":"2024-06-03T08:38:06.502302+0000","last_active":"2024-06-03T09:06:14.426670+0000","last_peered":"2024-06-03T09:06:14.426670+0000","last_clean":"2024-06-03T09:06:14.426670+0000","last_became_active":"2024-06-03T08:38:06.490204+0000","last_became_peered":"2024-06-03T08:38:06.490204+0000","last_unstale":"2024-06-03T09:06:14.426670+0000","last_undegraded":"2024-06-03T09:06:14.426670+0000","last_fullsized":"2024-06-03T09:06:14.426670+0000","mapping_epoch":28,"log_start":"0'0","ondisk_log_start":"0'0","created":28,"last_epoch_clean":32,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-06-03T08:37:55.614497+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-06-03T08:37:55.614497+0000","last_clean_scrub_stamp":"2024-06-03T08:37:55.614497+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-06-04T10:47:05.988958+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,2],"acting":[4,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"2.1","version":"0'0","reported_seq":952,"reported_epoch":1009,"state":"active+clean","last_fresh":"2024-06-03T09:06:13.929725+0000","last_change":"2024-06-03T08:36:30.883360+0000","last_active":"2024-06-03T09:06:13.929725+0000","last_peered":"2024-06-03T09:06:13.929725+0000","last_clean":"2024-06-03T09:06:13.929725+0000","last_became_active":"2024-06-03T08:36:30.855073+0000","last_became_peered":"2024-06-03T08:36:30.855073+0000","last_unstale":"2024-06-03T09:06:13.929725+0000","last_undegraded":"2024-06-03T09:06:13.929725+0000","last_fullsized":"2024-06-03T09:06:13.929725+0000","mapping_epoch":23,"log_start":"0'0","ondisk_log_start":"0'0","created":23,"last_epoch_clean":25,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-06-03T08:36:28.444764+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-06-03T08:36:28.444764+0000","last_clean_scrub_stamp":"2024-06-03T08:36:28.444764+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-06-04T18:12:22.363473+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,5],"acting":[7,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"17.12","version":"0'0","reported_seq":945,"reported_epoch":1009,"state":"active+clean","last_fresh":"2024-06-03T09:06:13.935141+0000","last_change":"2024-06-03T08:38:01.751562+0000","last_active":"2024-06-03T09:06:13.935141+0000","last_peered":"2024-06-03T09:06:13.935141+0000","last_clean":"2024-06-03T09:06:13.935141+0000","last_became_active":"2024-06-03T08:38:01.745027+0000","last_became_peered":"2024-06-03T08:38:01.745027+0000","last_unstale":"2024-06-03T09:06:13.935141+0000","last_undegraded":"2024-06-03T09:06:13.935141+0000","last_fullsized":"2024-06-03T09:06:13.935141+0000","mapping_epoch":28,"log_start":"0'0","ondisk_log_start":"0'0","created":28,"last_epoch_clean":32,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-06-03T08:37:55.614497+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-06-03T08:37:55.614497+0000","last_clean_scrub_stamp":"2024-06-03T08:37:55.614497+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-06-04T17:54:43.785326+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,3],"acting":[7,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"2.0","version":"0'0","reported_seq":943,"reported_epoch":1009,"state":"active+clean","last_fresh":"2024-06-03T09:06:13.648112+0000","last_change":"2024-06-03T08:36:29.984096+0000","last_active":"2024-06-03T09:06:13.648112+0000","last_peered":"2024-06-03T09:06:13.648112+0000","last_clean":"2024-06-03T09:06:13.648112+0000","last_became_active":"2024-06-03T08:36:29.957875+0000","last_became_peered":"2024-06-03T08:36:29.957875+0000","last_unstale":"2024-06-03T09:06:13.648112+0000","last_undegraded":"2024-06-03T09:06:13.648112+0000","last_fullsized":"2024-06-03T09:06:13.648112+0000","mapping_epoch":23,"log_start":"0'0","ondisk_log_start":"0'0","created":23,"last_epoch_clean":24,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-06-03T08:36:28.444764+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-06-03T08:36:28.444764+0000","last_clean_scrub_stamp":"2024-06-03T08:36:28.444764+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-06-04T08:41:33.679479+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,5],"acting":[3,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"17.13","version":"0'0","reported_seq":943,"reported_epoch":1009,"state":"active+clean","last_fresh":"2024-06-03T09:06:14.438399+0000","last_change":"2024-06-03T08:38:26.838316+0000","last_active":"2024-06-03T09:06:14.438399+0000","last_peered":"2024-06-03T09:06:14.438399+0000","last_clean":"2024-06-03T09:06:14.438399+0000","last_became_active":"2024-06-03T08:38:26.833214+0000","last_became_peered":"2024-06-03T08:38:26.833214+0000","last_unstale":"2024-06-03T09:06:14.438399+0000","last_undegraded":"2024-06-03T09:06:14.438399+0000","last_fullsized":"2024-06-03T09:06:14.438399+0000","mapping_epoch":28,"log_start":"0'0","ondisk_log_start":"0'0","created":28,"last_epoch_clean":52,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-06-03T08:37:55.614497+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-06-03T08:37:55.614497+0000","last_clean_scrub_stamp":"2024-06-03T08:37:55.614497+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-06-04T12:58:51.617998+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,0],"acting":[4,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"2.3","version":"0'0","reported_seq":942,"reported_epoch":1009,"state":"active+clean","last_fresh":"2024-06-03T09:06:13.664847+0000","last_change":"2024-06-03T08:36:30.351670+0000","last_active":"2024-06-03T09:06:13.664847+0000","last_peered":"2024-06-03T09:06:13.664847+0000","last_clean":"2024-06-03T09:06:13.664847+0000","last_became_active":"2024-06-03T08:36:30.343560+0000","last_became_peered":"2024-06-03T08:36:30.343560+0000","last_unstale":"2024-06-03T09:06:13.664847+0000","last_undegraded":"2024-06-03T09:06:13.664847+0000","last_fullsized":"2024-06-03T09:06:13.664847+0000","mapping_epoch":23,"log_start":"0'0","ondisk_log_start":"0'0","created":23,"last_epoch_clean":24,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-06-03T08:36:28.444764+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-06-03T08:36:28.444764+0000","last_clean_scrub_stamp":"2024-06-03T08:36:28.444764+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-06-04T12:02:24.765135+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,2],"acting":[3,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.0","version":"19'76","reported_seq":1051,"reported_epoch":1009,"state":"active+clean","last_fresh":"2024-06-03T09:06:13.703685+0000","last_change":"2024-06-03T08:36:26.702702+0000","last_active":"2024-06-03T09:06:13.703685+0000","last_peered":"2024-06-03T09:06:13.703685+0000","last_clean":"2024-06-03T09:06:13.703685+0000","last_became_active":"2024-06-03T08:36:26.128378+0000","last_became_peered":"2024-06-03T08:36:26.128378+0000","last_unstale":"2024-06-03T09:06:13.703685+0000","last_undegraded":"2024-06-03T09:06:13.703685+0000","last_fullsized":"2024-06-03T09:06:13.703685+0000","mapping_epoch":20,"log_start":"0'0","ondisk_log_start":"0'0","created":16,"last_epoch_clean":22,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-06-03T08:36:15.542635+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-06-03T08:36:15.542635+0000","last_clean_scrub_stamp":"2024-06-03T08:36:15.542635+0000","objects_scrubbed":0,"log_size":76,"log_dups_size":0,"ondisk_log_size":76,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-06-04T12:41:53.731360+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":459280,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":96,"num_read_kb":82,"num_write":113,"num_write_kb":1372,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2,"num_bytes_recovered":397840,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,5],"acting":[1,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"17.10","version":"0'0","reported_seq":944,"reported_epoch":1009,"state":"active+clean","last_fresh":"2024-06-03T09:06:14.422035+0000","last_change":"2024-06-03T08:38:15.399189+0000","last_active":"2024-06-03T09:06:14.422035+0000","last_peered":"2024-06-03T09:06:14.422035+0000","last_clean":"2024-06-03T09:06:14.422035+0000","last_became_active":"2024-06-03T08:38:15.337723+0000","last_became_peered":"2024-06-03T08:38:15.337723+0000","last_unstale":"2024-06-03T09:06:14.422035+0000","last_undegraded":"2024-06-03T09:06:14.422035+0000","last_fullsized":"2024-06-03T09:06:14.422035+0000","mapping_epoch":28,"log_start":"0'0","ondisk_log_start":"0'0","created":28,"last_epoch_clean":42,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-06-03T08:37:55.614497+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-06-03T08:37:55.614497+0000","last_clean_scrub_stamp":"2024-06-03T08:37:55.614497+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-06-04T09:29:46.427317+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,3],"acting":[4,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"17.1a","version":"0'0","reported_seq":960,"reported_epoch":1009,"state":"active+undersized","last_fresh":"2024-06-03T09:06:14.431224+0000","last_change":"2024-06-03T09:06:03.390734+0000","last_active":"2024-06-03T09:06:14.431224+0000","last_peered":"2024-06-03T09:06:14.431224+0000","last_clean":"2024-06-03T09:05:20.322313+0000","last_became_active":"2024-06-03T09:06:03.390734+0000","last_became_peered":"2024-06-03T09:06:03.390734+0000","last_unstale":"2024-06-03T09:06:14.431224+0000","last_undegraded":"2024-06-03T09:06:14.431224+0000","last_fullsized":"2024-06-03T09:06:03.343197+0000","mapping_epoch":999,"log_start":"0'0","ondisk_log_start":"0'0","created":28,"last_epoch_clean":254,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-06-03T08:37:55.614497+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-06-03T08:37:55.614497+0000","last_clean_scrub_stamp":"2024-06-03T08:37:55.614497+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-06-04T15:56:47.628875+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4],"acting":[4],"avail_no_missing":["4"],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"17.1b","version":"0'0","reported_seq":935,"reported_epoch":1009,"state":"active+clean","last_fresh":"2024-06-03T09:06:13.663279+0000","last_change":"2024-06-03T08:38:03.115748+0000","last_active":"2024-06-03T09:06:13.663279+0000","last_peered":"2024-06-03T09:06:13.663279+0000","last_clean":"2024-06-03T09:06:13.663279+0000","last_became_active":"2024-06-03T08:38:03.110533+0000","last_became_peered":"2024-06-03T08:38:03.110533+0000","last_unstale":"2024-06-03T09:06:13.663279+0000","last_undegraded":"2024-06-03T09:06:13.663279+0000","last_fullsized":"2024-06-03T09:06:13.663279+0000","mapping_epoch":28,"log_start":"0'0","ondisk_log_start":"0'0","created":28,"last_epoch_clean":32,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-06-03T08:37:55.614497+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-06-03T08:37:55.614497+0000","last_clean_scrub_stamp":"2024-06-03T08:37:55.614497+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-06-04T12:36:47.471609+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,7],"acting":[3,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"17.18","version":"0'0","reported_seq":944,"reported_epoch":1009,"state":"active+clean","last_fresh":"2024-06-03T09:06:13.924799+0000","last_change":"2024-06-03T08:38:04.632864+0000","last_active":"2024-06-03T09:06:13.924799+0000","last_peered":"2024-06-03T09:06:13.924799+0000","last_clean":"2024-06-03T09:06:13.924799+0000","last_became_active":"2024-06-03T08:38:04.627856+0000","last_became_peered":"2024-06-03T08:38:04.627856+0000","last_unstale":"2024-06-03T09:06:13.924799+0000","last_undegraded":"2024-06-03T09:06:13.924799+0000","last_fullsized":"2024-06-03T09:06:13.924799+0000","mapping_epoch":28,"log_start":"0'0","ondisk_log_start":"0'0","created":28,"last_epoch_clean":33,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-06-03T08:37:55.614497+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-06-03T08:37:55.614497+0000","last_clean_scrub_stamp":"2024-06-03T08:37:55.614497+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-06-04T19:55:51.181598+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,4],"acting":[7,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"17.19","version":"0'0","reported_seq":934,"reported_epoch":1009,"state":"active+clean","last_fresh":"2024-06-03T09:06:13.652943+0000","last_change":"2024-06-03T08:38:01.839491+0000","last_active":"2024-06-03T09:06:13.652943+0000","last_peered":"2024-06-03T09:06:13.652943+0000","last_clean":"2024-06-03T09:06:13.652943+0000","last_became_active":"2024-06-03T08:38:01.829539+0000","last_became_peered":"2024-06-03T08:38:01.829539+0000","last_unstale":"2024-06-03T09:06:13.652943+0000","last_undegraded":"2024-06-03T09:06:13.652943+0000","last_fullsized":"2024-06-03T09:06:13.652943+0000","mapping_epoch":28,"log_start":"0'0","ondisk_log_start":"0'0","created":28,"last_epoch_clean":30,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-06-03T08:37:55.614497+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-06-03T08:37:55.614497+0000","last_clean_scrub_stamp":"2024-06-03T08:37:55.614497+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-06-04T09:10:43.860229+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,2],"acting":[3,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"17.1e","version":"0'0","reported_seq":148,"reported_epoch":1009,"state":"active+undersized","last_fresh":"2024-06-03T09:06:14.445917+0000","last_change":"2024-06-03T09:06:03.406181+0000","last_active":"2024-06-03T09:06:14.445917+0000","last_peered":"2024-06-03T09:06:14.445917+0000","last_clean":"2024-06-03T08:37:55.614497+0000","last_became_active":"2024-06-03T09:06:03.406181+0000","last_became_peered":"2024-06-03T09:06:03.406181+0000","last_unstale":"2024-06-03T09:06:14.445917+0000","last_undegraded":"2024-06-03T09:06:14.445917+0000","last_fullsized":"2024-06-03T09:06:03.363739+0000","mapping_epoch":999,"log_start":"0'0","ondisk_log_start":"0'0","created":28,"last_epoch_clean":255,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-06-03T08:37:55.614497+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-06-03T08:37:55.614497+0000","last_clean_scrub_stamp":"2024-06-03T08:37:55.614497+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-06-04T11:45:24.486600+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4],"acting":[4],"avail_no_missing":["4"],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"17.1f","version":"0'0","reported_seq":941,"reported_epoch":1009,"state":"active+clean","last_fresh":"2024-06-03T09:06:14.450053+0000","last_change":"2024-06-03T08:38:04.383538+0000","last_active":"2024-06-03T09:06:14.450053+0000","last_peered":"2024-06-03T09:06:14.450053+0000","last_clean":"2024-06-03T09:06:14.450053+0000","last_became_active":"2024-06-03T08:38:04.367278+0000","last_became_peered":"2024-06-03T08:38:04.367278+0000","last_unstale":"2024-06-03T09:06:14.450053+0000","last_undegraded":"2024-06-03T09:06:14.450053+0000","last_fullsized":"2024-06-03T09:06:14.450053+0000","mapping_epoch":28,"log_start":"0'0","ondisk_log_start":"0'0","created":28,"last_epoch_clean":32,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-06-03T08:37:55.614497+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-06-03T08:37:55.614497+0000","last_clean_scrub_stamp":"2024-06-03T08:37:55.614497+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-06-04T10:39:26.552282+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,2],"acting":[0,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"17.1c","version":"0'0","reported_seq":943,"reported_epoch":1009,"state":"active+undersized","last_fresh":"2024-06-03T09:06:13.719240+0000","last_change":"2024-06-03T09:06:03.382210+0000","last_active":"2024-06-03T09:06:13.719240+0000","last_peered":"2024-06-03T09:06:13.719240+0000","last_clean":"2024-06-03T09:04:43.199660+0000","last_became_active":"2024-06-03T09:06:03.382210+0000","last_became_peered":"2024-06-03T09:06:03.382210+0000","last_unstale":"2024-06-03T09:06:13.719240+0000","last_undegraded":"2024-06-03T09:06:13.719240+0000","last_fullsized":"2024-06-03T09:06:03.277874+0000","mapping_epoch":999,"log_start":"0'0","ondisk_log_start":"0'0","created":28,"last_epoch_clean":255,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-06-03T08:37:55.614497+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-06-03T08:37:55.614497+0000","last_clean_scrub_stamp":"2024-06-03T08:37:55.614497+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-06-04T12:18:50.013884+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5],"acting":[5],"avail_no_missing":["5"],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"17.1d","version":"0'0","reported_seq":923,"reported_epoch":1009,"state":"active+clean","last_fresh":"2024-06-03T09:06:13.715906+0000","last_change":"2024-06-03T08:38:06.794630+0000","last_active":"2024-06-03T09:06:13.715906+0000","last_peered":"2024-06-03T09:06:13.715906+0000","last_clean":"2024-06-03T09:06:13.715906+0000","last_became_active":"2024-06-03T08:38:06.592317+0000","last_became_peered":"2024-06-03T08:38:06.592317+0000","last_unstale":"2024-06-03T09:06:13.715906+0000","last_undegraded":"2024-06-03T09:06:13.715906+0000","last_fullsized":"2024-06-03T09:06:13.715906+0000","mapping_epoch":28,"log_start":"0'0","ondisk_log_start":"0'0","created":28,"last_epoch_clean":33,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-06-03T08:37:55.614497+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-06-03T08:37:55.614497+0000","last_clean_scrub_stamp":"2024-06-03T08:37:55.614497+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-06-04T12:33:30.785369+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,7],"acting":[1,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"17.2","version":"0'0","reported_seq":925,"reported_epoch":1009,"state":"active+clean","last_fresh":"2024-06-03T09:06:13.708688+0000","last_change":"2024-06-03T08:38:14.981410+0000","last_active":"2024-06-03T09:06:13.708688+0000","last_peered":"2024-06-03T09:06:13.708688+0000","last_clean":"2024-06-03T09:06:13.708688+0000","last_became_active":"2024-06-03T08:38:14.973825+0000","last_became_peered":"2024-06-03T08:38:14.973825+0000","last_unstale":"2024-06-03T09:06:13.708688+0000","last_undegraded":"2024-06-03T09:06:13.708688+0000","last_fullsized":"2024-06-03T09:06:13.708688+0000","mapping_epoch":28,"log_start":"0'0","ondisk_log_start":"0'0","created":28,"last_epoch_clean":41,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-06-03T08:37:55.614497+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-06-03T08:37:55.614497+0000","last_clean_scrub_stamp":"2024-06-03T08:37:55.614497+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-06-04T17:01:05.518613+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,4],"acting":[1,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"17.3","version":"0'0","reported_seq":943,"reported_epoch":1009,"state":"active+clean","last_fresh":"2024-06-03T09:06:14.437636+0000","last_change":"2024-06-03T08:38:14.619745+0000","last_active":"2024-06-03T09:06:14.437636+0000","last_peered":"2024-06-03T09:06:14.437636+0000","last_clean":"2024-06-03T09:06:14.437636+0000","last_became_active":"2024-06-03T08:38:14.510665+0000","last_became_peered":"2024-06-03T08:38:14.510665+0000","last_unstale":"2024-06-03T09:06:14.437636+0000","last_undegraded":"2024-06-03T09:06:14.437636+0000","last_fullsized":"2024-06-03T09:06:14.437636+0000","mapping_epoch":28,"log_start":"0'0","ondisk_log_start":"0'0","created":28,"last_epoch_clean":42,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-06-03T08:37:55.614497+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-06-03T08:37:55.614497+0000","last_clean_scrub_stamp":"2024-06-03T08:37:55.614497+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-06-04T12:03:47.392064+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,7],"acting":[0,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"17.0","version":"0'0","reported_seq":944,"reported_epoch":1009,"state":"active+clean","last_fresh":"2024-06-03T09:06:14.427123+0000","last_change":"2024-06-03T08:38:04.237118+0000","last_active":"2024-06-03T09:06:14.427123+0000","last_peered":"2024-06-03T09:06:14.427123+0000","last_clean":"2024-06-03T09:06:14.427123+0000","last_became_active":"2024-06-03T08:38:04.231954+0000","last_became_peered":"2024-06-03T08:38:04.231954+0000","last_unstale":"2024-06-03T09:06:14.427123+0000","last_undegraded":"2024-06-03T09:06:14.427123+0000","last_fullsized":"2024-06-03T09:06:14.427123+0000","mapping_epoch":28,"log_start":"0'0","ondisk_log_start":"0'0","created":28,"last_epoch_clean":32,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-06-03T08:37:55.614497+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-06-03T08:37:55.614497+0000","last_clean_scrub_stamp":"2024-06-03T08:37:55.614497+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-06-04T09:56:10.741637+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,2],"acting":[0,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"17.1","version":"65'7","reported_seq":959,"reported_epoch":1009,"state":"active+undersized+degraded","last_fresh":"2024-06-03T09:06:13.709209+0000","last_change":"2024-06-03T09:06:03.370660+0000","last_active":"2024-06-03T09:06:13.709209+0000","last_peered":"2024-06-03T09:06:13.709209+0000","last_clean":"2024-06-03T09:05:03.733864+0000","last_became_active":"2024-06-03T09:06:03.370660+0000","last_became_peered":"2024-06-03T09:06:03.370660+0000","last_unstale":"2024-06-03T09:06:13.709209+0000","last_undegraded":"2024-06-03T09:06:03.266076+0000","last_fullsized":"2024-06-03T09:06:03.264525+0000","mapping_epoch":999,"log_start":"0'0","ondisk_log_start":"0'0","created":28,"last_epoch_clean":255,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-06-03T08:37:55.614497+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-06-03T08:37:55.614497+0000","last_clean_scrub_stamp":"2024-06-03T08:37:55.614497+0000","objects_scrubbed":0,"log_size":7,"log_dups_size":0,"ondisk_log_size":7,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-06-04T11:55:18.194283+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":5,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":1,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":6,"num_read_kb":6,"num_write":7,"num_write_kb":7,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5],"acting":[5],"avail_no_missing":["5"],"object_location_counts":[{"shards":"5","objects":1}],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]}],"pool_stats":[{"poolid":2,"num_pg":8,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":14,"acting":14,"num_store_stats":7},{"poolid":1,"num_pg":1,"stat_sum":{"num_bytes":459280,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":96,"num_read_kb":82,"num_write":113,"num_write_kb":1372,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2,"num_bytes_recovered":397840,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":802816,"data_stored":795680,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":76,"ondisk_log_size":76,"up":2,"acting":2,"num_store_stats":2},{"poolid":17,"num_pg":32,"stat_sum":{"num_bytes":5,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":1,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":6,"num_read_kb":6,"num_write":7,"num_write_kb":7,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":8192,"data_stored":10,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":7,"ondisk_log_size":7,"up":52,"acting":52,"num_store_stats":8}],"osd_stats":[{"osd":6,"up_from":253,"seq":1086626726200,"num_pgs":32,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":54772,"kb_used_data":5608,"kb_used_omap":1,"kb_used_meta":49150,"kb_avail":94317068,"statfs":{"total":96636764160,"available":96580677632,"internally_reserved":0,"allocated":5742592,"data_stored":2767180,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1593,"internal_metadata":50330055},"hb_peers":[0,1,2,3,4,5,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":109,"apply_latency_ms":109,"commit_latency_ns":109000000,"apply_latency_ns":109000000},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Mon Jun 3 09:04:02 2024","interfaces":[{"interface":"back","average":{"1min":1582.4780000000001,"5min":332.72699999999998,"15min":124.113},"min":{"1min":4.4829999999999997,"5min":4.1959999999999997,"15min":4.1959999999999997},"max":{"1min":22766.882000000001,"5min":22766.882000000001,"15min":22766.882000000001},"last":21.917999999999999},{"interface":"front","average":{"1min":1579.5409999999999,"5min":334.94499999999999,"15min":125.84999999999999},"min":{"1min":4.8129999999999997,"5min":4.7539999999999996,"15min":4.4530000000000003},"max":{"1min":22766.289000000001,"5min":22766.289000000001,"15min":22766.289000000001},"last":25.408999999999999}]},{"osd":1,"last update":"Mon Jun 3 09:03:40 2024","interfaces":[{"interface":"back","average":{"1min":36.420999999999999,"5min":77.054000000000002,"15min":132.84800000000001},"min":{"1min":6.0380000000000003,"5min":5.8799999999999999,"15min":5.484},"max":{"1min":589.64099999999996,"5min":4138.1679999999997,"15min":9955.1299999999992},"last":26.84},{"interface":"front","average":{"1min":40.240000000000002,"5min":79.674999999999997,"15min":133.38999999999999},"min":{"1min":8.4589999999999996,"5min":8.4090000000000007,"15min":6.1390000000000002},"max":{"1min":588.60799999999995,"5min":4137.6530000000002,"15min":9827.5509999999995},"last":26.263000000000002}]},{"osd":2,"last update":"Mon Jun 3 09:03:40 2024","interfaces":[{"interface":"back","average":{"1min":12.613,"5min":24.600999999999999,"15min":21.088000000000001},"min":{"1min":6.9740000000000002,"5min":6.9740000000000002,"15min":6.5330000000000004},"max":{"1min":26.076000000000001,"5min":376.24900000000002,"15min":376.24900000000002},"last":24.899000000000001},{"interface":"front","average":{"1min":12.493,"5min":22.207999999999998,"15min":21.219000000000001},"min":{"1min":6.5810000000000004,"5min":6.5110000000000001,"15min":6.0750000000000002},"max":{"1min":29.690999999999999,"5min":271.51100000000002,"15min":434.77300000000002},"last":40.372}]},{"osd":3,"last update":"Mon Jun 3 09:03:40 2024","interfaces":[{"interface":"back","average":{"1min":14.093999999999999,"5min":24.143999999999998,"15min":29.376999999999999},"min":{"1min":10.196999999999999,"5min":6.2919999999999998,"15min":6.1719999999999997},"max":{"1min":24.832000000000001,"5min":197.84899999999999,"15min":1235.5799999999999},"last":24.353999999999999},{"interface":"front","average":{"1min":9.3260000000000005,"5min":18.463999999999999,"15min":27.631},"min":{"1min":6.125,"5min":6.125,"15min":6.125},"max":{"1min":28.166,"5min":200.173,"15min":1157.6780000000001},"last":23.454000000000001}]},{"osd":4,"last update":"Mon Jun 3 09:04:29 2024","interfaces":[{"interface":"back","average":{"1min":10.279,"5min":1253.809,"15min":450.79199999999997},"min":{"1min":7.2300000000000004,"5min":6.093,"15min":6.093},"max":{"1min":15.000999999999999,"5min":28093.096000000001,"15min":28093.096000000001},"last":101.45399999999999},{"interface":"front","average":{"1min":15.693,"5min":1279.759,"15min":463.87099999999998},"min":{"1min":10.551,"5min":7.7480000000000002,"15min":7.7480000000000002},"max":{"1min":27.898,"5min":28323.837,"15min":28323.837},"last":102.262}]},{"osd":5,"last update":"Mon Jun 3 09:04:10 2024","interfaces":[{"interface":"back","average":{"1min":13.728999999999999,"5min":24.670000000000002,"15min":23.815999999999999},"min":{"1min":10.401,"5min":7.516,"15min":7.516},"max":{"1min":25.138000000000002,"5min":320.85000000000002,"15min":445.041},"last":103.52500000000001},{"interface":"front","average":{"1min":13.847,"5min":29.283000000000001,"15min":23.393999999999998},"min":{"1min":7.944,"5min":6.266,"15min":6.266},"max":{"1min":18.817,"5min":530.56700000000001,"15min":530.56700000000001},"last":29.263999999999999}]},{"osd":7,"last update":"Mon Jun 3 09:03:40 2024","interfaces":[{"interface":"back","average":{"1min":20.568999999999999,"5min":30.286999999999999,"15min":27.977},"min":{"1min":13.874000000000001,"5min":10.58,"15min":9.2859999999999996},"max":{"1min":86.959000000000003,"5min":222.07599999999999,"15min":255.08799999999999},"last":37.618000000000002},{"interface":"front","average":{"1min":19.102,"5min":29.550000000000001,"15min":27.763000000000002},"min":{"1min":10.052,"5min":8.6709999999999994,"15min":8.5429999999999993},"max":{"1min":67.968999999999994,"5min":293.62099999999998,"15min":438.14600000000002},"last":111.985}]}]},{"osd":7,"up_from":20,"seq":85899346285,"num_pgs":9,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":58964,"kb_used_data":5704,"kb_used_omap":1,"kb_used_meta":53246,"kb_avail":94312876,"statfs":{"total":96636764160,"available":96576385024,"internally_reserved":0,"allocated":5840896,"data_stored":2740727,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1589,"internal_metadata":54524363},"hb_peers":[0,1,2,3,4,5],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":4,"apply_latency_ms":4,"commit_latency_ns":4000000,"apply_latency_ns":4000000},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Mon Jun 3 09:06:09 2024","interfaces":[{"interface":"back","average":{"1min":14.073,"5min":894.22400000000005,"15min":308.93400000000003},"min":{"1min":5.5709999999999997,"5min":4.218,"15min":4.0599999999999996},"max":{"1min":90.757999999999996,"5min":21729.001,"15min":21729.001},"last":17.491},{"interface":"front","average":{"1min":9.2550000000000008,"5min":906.23500000000001,"15min":314.06},"min":{"1min":5.5789999999999997,"5min":4.8810000000000002,"15min":4.8810000000000002},"max":{"1min":25.937000000000001,"5min":21899.732,"15min":21899.732},"last":6.1420000000000003}]},{"osd":1,"last update":"Mon Jun 3 09:06:09 2024","interfaces":[{"interface":"back","average":{"1min":15.182,"5min":69.183999999999997,"15min":219.173},"min":{"1min":7.3639999999999999,"5min":6.6200000000000001,"15min":6.1319999999999997},"max":{"1min":76.926000000000002,"5min":2795.1329999999998,"15min":18373.257000000001},"last":7.444},{"interface":"front","average":{"1min":21.495999999999999,"5min":74.703999999999994,"15min":225.285},"min":{"1min":8.8399999999999999,"5min":8.8399999999999999,"15min":8.5129999999999999},"max":{"1min":94.977999999999994,"5min":2803.3670000000002,"15min":18374.601999999999},"last":12.337999999999999}]},{"osd":2,"last update":"Mon Jun 3 09:06:09 2024","interfaces":[{"interface":"back","average":{"1min":19.164000000000001,"5min":26.902000000000001,"15min":22.920999999999999},"min":{"1min":11.207000000000001,"5min":8.5030000000000001,"15min":8.5030000000000001},"max":{"1min":91.319999999999993,"5min":333.99200000000002,"15min":333.99200000000002},"last":11.518000000000001},{"interface":"front","average":{"1min":11.125999999999999,"5min":18.132000000000001,"15min":16.353000000000002},"min":{"1min":6.202,"5min":5.8369999999999997,"15min":5.5069999999999997},"max":{"1min":28.649000000000001,"5min":226.18100000000001,"15min":226.18100000000001},"last":9.077}]},{"osd":3,"last update":"Mon Jun 3 09:06:09 2024","interfaces":[{"interface":"back","average":{"1min":22.291,"5min":26.289999999999999,"15min":31.074999999999999},"min":{"1min":11.714,"5min":6.8959999999999999,"15min":6.3760000000000003},"max":{"1min":83.102000000000004,"5min":316.64400000000001,"15min":2039.9059999999999},"last":10.698},{"interface":"front","average":{"1min":13.324999999999999,"5min":17.963000000000001,"15min":25.969999999999999},"min":{"1min":6.1200000000000001,"5min":5.2089999999999996,"15min":5.2089999999999996},"max":{"1min":49.908999999999999,"5min":233.542,"15min":2037.5550000000001},"last":9.9540000000000006}]},{"osd":4,"last update":"Mon Jun 3 09:05:28 2024","interfaces":[{"interface":"back","average":{"1min":22.209,"5min":1379.51,"15min":1803.278},"min":{"1min":5.9089999999999998,"5min":5.9089999999999998,"15min":5.9089999999999998},"max":{"1min":224.54300000000001,"5min":30073.917000000001,"15min":62560.868000000002},"last":16.802},{"interface":"front","average":{"1min":22.712,"5min":1401.0719999999999,"15min":1806.759},"min":{"1min":7.6449999999999996,"5min":6.7930000000000001,"15min":5.9340000000000002},"max":{"1min":223.58600000000001,"5min":30348.021000000001,"15min":62451.446000000004},"last":16.009}]},{"osd":5,"last update":"Mon Jun 3 09:06:02 2024","interfaces":[{"interface":"back","average":{"1min":24.372,"5min":26.856000000000002,"15min":23.571999999999999},"min":{"1min":12.278,"5min":9.2460000000000004,"15min":6.5650000000000004},"max":{"1min":183.64099999999999,"5min":317.572,"15min":317.572},"last":13.161},{"interface":"front","average":{"1min":21.259,"5min":23.364000000000001,"15min":21.193999999999999},"min":{"1min":7.8499999999999996,"5min":7.0999999999999996,"15min":7.008},"max":{"1min":184.53399999999999,"5min":306.60599999999999,"15min":306.60599999999999},"last":15.401}]},{"osd":6,"last update":"Mon Jun 3 09:04:31 2024","interfaces":[{"interface":"back","average":{"1min":44.061,"5min":26.821000000000002,"15min":91.176000000000002},"min":{"1min":9.0800000000000001,"5min":7.9390000000000001,"15min":7.9390000000000001},"max":{"1min":268.029,"5min":308.916,"15min":8734.4639999999999},"last":6.6769999999999996},{"interface":"front","average":{"1min":46.643000000000001,"5min":23.879999999999999,"15min":88.765000000000001},"min":{"1min":8.8119999999999994,"5min":6.9240000000000004,"15min":6.742},"max":{"1min":305.64600000000002,"5min":305.64600000000002,"15min":8735.5319999999992},"last":12.327}]}]},{"osd":5,"up_from":20,"seq":85899346259,"num_pgs":9,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":59360,"kb_used_data":6100,"kb_used_omap":1,"kb_used_meta":53246,"kb_avail":94312480,"statfs":{"total":96636764160,"available":96575979520,"internally_reserved":0,"allocated":6246400,"data_stored":3138572,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1594,"internal_metadata":54524358},"hb_peers":[0,1,2,3,4,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":5,"apply_latency_ms":5,"commit_latency_ns":5000000,"apply_latency_ns":5000000},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Mon Jun 3 09:06:06 2024","interfaces":[{"interface":"back","average":{"1min":7.468,"5min":1703.037,"15min":580.51900000000001},"min":{"1min":4.6509999999999998,"5min":4.6509999999999998,"15min":4.4020000000000001},"max":{"1min":14.711,"5min":26433.764999999999,"15min":26433.764999999999},"last":6.1859999999999999},{"interface":"front","average":{"1min":9.9429999999999996,"5min":1725.27,"15min":587.87800000000004},"min":{"1min":5.5999999999999996,"5min":5.1319999999999997,"15min":4.8170000000000002},"max":{"1min":13.590999999999999,"5min":26704.698,"15min":26704.698},"last":7.6970000000000001}]},{"osd":1,"last update":"Mon Jun 3 09:06:00 2024","interfaces":[{"interface":"back","average":{"1min":10.862,"5min":67.731999999999999,"15min":217.364},"min":{"1min":7.1769999999999996,"5min":6.0890000000000004,"15min":5.4930000000000003},"max":{"1min":16.998999999999999,"5min":3923.4090000000001,"15min":18811.696},"last":12.917999999999999},{"interface":"front","average":{"1min":10.417,"5min":65.141999999999996,"15min":220.75700000000001},"min":{"1min":5.3209999999999997,"5min":5.3209999999999997,"15min":5.0209999999999999},"max":{"1min":15.821999999999999,"5min":4019.6129999999998,"15min":18968.365000000002},"last":8.6590000000000007}]},{"osd":2,"last update":"Mon Jun 3 09:06:00 2024","interfaces":[{"interface":"back","average":{"1min":11.366,"5min":24.18,"15min":21.137},"min":{"1min":5.6369999999999996,"5min":5.6369999999999996,"15min":5.0510000000000002},"max":{"1min":15.529999999999999,"5min":332.48200000000003,"15min":332.48200000000003},"last":14.346},{"interface":"front","average":{"1min":10.089,"5min":23.117000000000001,"15min":24.050000000000001},"min":{"1min":6.6769999999999996,"5min":6.6769999999999996,"15min":5.8739999999999997},"max":{"1min":15.821,"5min":353.03199999999998,"15min":353.03199999999998},"last":8.2629999999999999}]},{"osd":3,"last update":"Mon Jun 3 09:06:00 2024","interfaces":[{"interface":"back","average":{"1min":13.472,"5min":24.283999999999999,"15min":31.666},"min":{"1min":10.111000000000001,"5min":7.4720000000000004,"15min":7.4210000000000003},"max":{"1min":16.271999999999998,"5min":353.57499999999999,"15min":2855.8519999999999},"last":13.637},{"interface":"front","average":{"1min":9.5899999999999999,"5min":15.875,"15min":25.834},"min":{"1min":6.6609999999999996,"5min":5.7729999999999997,"15min":5.1909999999999998},"max":{"1min":15.112,"5min":203.804,"15min":2869.7860000000001},"last":10.704000000000001}]},{"osd":4,"last update":"Mon Jun 3 09:05:27 2024","interfaces":[{"interface":"back","average":{"1min":19.155000000000001,"5min":1084.961,"15min":1346.9949999999999},"min":{"1min":7.6130000000000004,"5min":7.6130000000000004,"15min":6.1340000000000003},"max":{"1min":156.55199999999999,"5min":28619.825000000001,"15min":57999.453999999998},"last":10.016},{"interface":"front","average":{"1min":28.068999999999999,"5min":1087.106,"15min":1330.289},"min":{"1min":8.3949999999999996,"5min":6.5030000000000001,"15min":6.5030000000000001},"max":{"1min":254.91300000000001,"5min":28621.294000000002,"15min":57434.631999999998},"last":15.166}]},{"osd":6,"last update":"Mon Jun 3 09:03:57 2024","interfaces":[{"interface":"back","average":{"1min":23.536000000000001,"5min":36.457000000000001,"15min":83.799999999999997},"min":{"1min":9.4800000000000004,"5min":9.4800000000000004,"15min":8.0630000000000006},"max":{"1min":89.442999999999998,"5min":508.77999999999997,"15min":9193.5169999999998},"last":157.79300000000001},{"interface":"front","average":{"1min":18.891999999999999,"5min":22.100999999999999,"15min":75.489999999999995},"min":{"1min":5.923,"5min":5.923,"15min":5.923},"max":{"1min":78.424999999999997,"5min":464.62,"15min":9191.4429999999993},"last":148.09}]},{"osd":7,"last update":"Mon Jun 3 09:06:00 2024","interfaces":[{"interface":"back","average":{"1min":13.257,"5min":20.917999999999999,"15min":21.088999999999999},"min":{"1min":10.475,"5min":6.7629999999999999,"15min":6.7629999999999999},"max":{"1min":17.245000000000001,"5min":211.22999999999999,"15min":211.22999999999999},"last":11.826000000000001},{"interface":"front","average":{"1min":12.872999999999999,"5min":20.587,"15min":24.021000000000001},"min":{"1min":8.5760000000000005,"5min":7.7709999999999999,"15min":6.8570000000000002},"max":{"1min":18.359999999999999,"5min":223.191,"15min":286.69499999999999},"last":11.209}]}]},{"osd":4,"up_from":20,"seq":85899346267,"num_pgs":12,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":63060,"kb_used_data":5704,"kb_used_omap":1,"kb_used_meta":57342,"kb_avail":94308780,"statfs":{"total":96636764160,"available":96572190720,"internally_reserved":0,"allocated":5840896,"data_stored":2740727,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1594,"internal_metadata":58718662},"hb_peers":[0,1,2,3,5,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":4,"apply_latency_ms":4,"commit_latency_ns":4000000,"apply_latency_ns":4000000},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Mon Jun 3 09:05:34 2024","interfaces":[{"interface":"back","average":{"1min":12.416,"5min":1099.4649999999999,"15min":382.15199999999999},"min":{"1min":4.5279999999999996,"5min":4.5279999999999996,"15min":4.3220000000000001},"max":{"1min":123.878,"5min":21726.743999999999,"15min":21726.743999999999},"last":4.6900000000000004},{"interface":"front","average":{"1min":16.462,"5min":1108.894,"15min":387.70299999999997},"min":{"1min":6.4089999999999998,"5min":5.8380000000000001,"15min":5.4909999999999997},"max":{"1min":130.185,"5min":21727.993999999999,"15min":21727.993999999999},"last":7.4349999999999996}]},{"osd":1,"last update":"Mon Jun 3 09:05:34 2024","interfaces":[{"interface":"back","average":{"1min":14.727,"5min":56.755000000000003,"15min":341.09399999999999},"min":{"1min":4.6529999999999996,"5min":4.1970000000000001,"15min":4.1970000000000001},"max":{"1min":126.899,"5min":3346.4389999999999,"15min":22256.098000000002},"last":8.7590000000000003},{"interface":"front","average":{"1min":13.472,"5min":57.503,"15min":340.303},"min":{"1min":5.0590000000000002,"5min":5.0590000000000002,"15min":4.2640000000000002},"max":{"1min":126.096,"5min":3347.181,"15min":22247.317999999999},"last":7.7999999999999998}]},{"osd":2,"last update":"Mon Jun 3 09:05:34 2024","interfaces":[{"interface":"back","average":{"1min":26.088999999999999,"5min":23.866,"15min":20.620999999999999},"min":{"1min":7.0869999999999997,"5min":6.8239999999999998,"15min":6.3150000000000004},"max":{"1min":337.60399999999998,"5min":393.58600000000001,"15min":393.58600000000001},"last":9.5820000000000007},{"interface":"front","average":{"1min":15.800000000000001,"5min":22.584,"15min":23.245999999999999},"min":{"1min":6.0629999999999997,"5min":6.0629999999999997,"15min":6.0629999999999997},"max":{"1min":127.913,"5min":394.471,"15min":394.471},"last":10.23}]},{"osd":3,"last update":"Mon Jun 3 09:05:34 2024","interfaces":[{"interface":"back","average":{"1min":17.544,"5min":19.143999999999998,"15min":18.742000000000001},"min":{"1min":5.8719999999999999,"5min":5.4100000000000001,"15min":5.4000000000000004},"max":{"1min":127.43600000000001,"5min":322.63,"15min":322.63},"last":11.765000000000001},{"interface":"front","average":{"1min":21.327000000000002,"5min":23.559000000000001,"15min":23.797000000000001},"min":{"1min":10.023,"5min":7.484,"15min":7.484},"max":{"1min":204.517,"5min":392.67500000000001,"15min":392.67500000000001},"last":11.102}]},{"osd":5,"last update":"Mon Jun 3 09:05:34 2024","interfaces":[{"interface":"back","average":{"1min":24.274999999999999,"5min":21.274999999999999,"15min":20.988},"min":{"1min":6.7489999999999997,"5min":6.7489999999999997,"15min":6.7489999999999997},"max":{"1min":268.10199999999998,"5min":344.23399999999998,"15min":344.23399999999998},"last":13.635},{"interface":"front","average":{"1min":26.361999999999998,"5min":25.218,"15min":26.084},"min":{"1min":12.228,"5min":8.5180000000000007,"15min":8.5180000000000007},"max":{"1min":264.75,"5min":345.18700000000001,"15min":345.18700000000001},"last":14.532}]},{"osd":6,"last update":"Mon Jun 3 09:04:32 2024","interfaces":[{"interface":"back","average":{"1min":15.548,"5min":32.957999999999998,"15min":152.738},"min":{"1min":9.141,"5min":9.141,"15min":8.2319999999999993},"max":{"1min":52.57,"5min":395.238,"15min":12496.335999999999},"last":208.90000000000001},{"interface":"front","average":{"1min":12.680999999999999,"5min":25.428000000000001,"15min":149.30600000000001},"min":{"1min":6.4379999999999997,"5min":5.6319999999999997,"15min":5.6319999999999997},"max":{"1min":46.283000000000001,"5min":377.74299999999999,"15min":12616.297},"last":205.88900000000001}]},{"osd":7,"last update":"Mon Jun 3 09:05:34 2024","interfaces":[{"interface":"back","average":{"1min":18.295999999999999,"5min":19.585999999999999,"15min":21.062000000000001},"min":{"1min":6.2110000000000003,"5min":6.2110000000000003,"15min":5.8010000000000002},"max":{"1min":207.25299999999999,"5min":358.94900000000001,"15min":358.94900000000001},"last":12.619},{"interface":"front","average":{"1min":23.712,"5min":24.024999999999999,"15min":24.640000000000001},"min":{"1min":11.824999999999999,"5min":7.7789999999999999,"15min":7.7789999999999999},"max":{"1min":209.32900000000001,"5min":346.25200000000001,"15min":380.06900000000002},"last":15.253}]}]},{"osd":2,"up_from":20,"seq":85899346287,"num_pgs":11,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":58964,"kb_used_data":5704,"kb_used_omap":1,"kb_used_meta":53246,"kb_avail":94312876,"statfs":{"total":96636764160,"available":96576385024,"internally_reserved":0,"allocated":5840896,"data_stored":2740727,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1594,"internal_metadata":54524358},"hb_peers":[0,1,3,4,5,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":4,"apply_latency_ms":4,"commit_latency_ns":4000000,"apply_latency_ns":4000000},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Mon Jun 3 09:05:21 2024","interfaces":[{"interface":"back","average":{"1min":16.364000000000001,"5min":947.59799999999996,"15min":326.58999999999997},"min":{"1min":6.0830000000000002,"5min":4.7050000000000001,"15min":4.1420000000000003},"max":{"1min":157.60900000000001,"5min":23465.18,"15min":23465.18},"last":6.8499999999999996},{"interface":"front","average":{"1min":12.276,"5min":945.29899999999998,"15min":326.78399999999999},"min":{"1min":5.1989999999999998,"5min":4.7439999999999998,"15min":4.6950000000000003},"max":{"1min":102.557,"5min":23465.966,"15min":23465.966},"last":5.8600000000000003}]},{"osd":1,"last update":"Mon Jun 3 09:05:21 2024","interfaces":[{"interface":"back","average":{"1min":17.803999999999998,"5min":77.658000000000001,"15min":294.96899999999999},"min":{"1min":7.2130000000000001,"5min":5.4199999999999999,"15min":5.4199999999999999},"max":{"1min":159.624,"5min":4220.5290000000005,"15min":18954.987000000001},"last":10.423999999999999},{"interface":"front","average":{"1min":21.542999999999999,"5min":83.528999999999996,"15min":303.11700000000002},"min":{"1min":8.8200000000000003,"5min":7.5949999999999998,"15min":7.5949999999999998},"max":{"1min":170.74299999999999,"5min":4325.9489999999996,"15min":19143.834999999999},"last":11.566000000000001}]},{"osd":3,"last update":"Mon Jun 3 09:05:21 2024","interfaces":[{"interface":"back","average":{"1min":19.164999999999999,"5min":13.284000000000001,"15min":28.751999999999999},"min":{"1min":7.3710000000000004,"5min":5.9290000000000003,"15min":5.2450000000000001},"max":{"1min":118.913,"5min":118.913,"15min":2594.8150000000001},"last":8.3040000000000003},{"interface":"front","average":{"1min":22.094000000000001,"5min":17.244,"15min":32.173999999999999},"min":{"1min":9.516,"5min":8.0760000000000005,"15min":6.0780000000000003},"max":{"1min":156.34700000000001,"5min":156.34700000000001,"15min":2706.5120000000002},"last":13.273999999999999}]},{"osd":4,"last update":"Mon Jun 3 09:05:57 2024","interfaces":[{"interface":"back","average":{"1min":10.573,"5min":1266.894,"15min":438.154},"min":{"1min":8.6379999999999999,"5min":7.8789999999999996,"15min":6.2039999999999997},"max":{"1min":15.105,"5min":26660.861000000001,"15min":26660.861000000001},"last":9.673},{"interface":"front","average":{"1min":12.407999999999999,"5min":1266.7460000000001,"15min":435.75400000000002},"min":{"1min":10.266999999999999,"5min":10.266999999999999,"15min":6.2720000000000002},"max":{"1min":15.228,"5min":26616.243999999999,"15min":26616.243999999999},"last":14.26}]},{"osd":5,"last update":"Mon Jun 3 09:05:21 2024","interfaces":[{"interface":"back","average":{"1min":25.632000000000001,"5min":20.437999999999999,"15min":21.472999999999999},"min":{"1min":9.2159999999999993,"5min":9.2159999999999993,"15min":6.2050000000000001},"max":{"1min":252.31100000000001,"5min":252.31100000000001,"15min":252.31100000000001},"last":14.960000000000001},{"interface":"front","average":{"1min":24.824000000000002,"5min":18.521000000000001,"15min":21.027000000000001},"min":{"1min":8.0449999999999999,"5min":7.4850000000000003,"15min":6.8810000000000002},"max":{"1min":258.31700000000001,"5min":258.31700000000001,"15min":258.31700000000001},"last":12.413}]},{"osd":6,"last update":"Mon Jun 3 09:04:28 2024","interfaces":[{"interface":"back","average":{"1min":14.063000000000001,"5min":24.850000000000001,"15min":106.755},"min":{"1min":9.0820000000000007,"5min":8.9649999999999999,"15min":8.9649999999999999},"max":{"1min":24.666,"5min":188.06700000000001,"15min":9521.0010000000002},"last":183.60900000000001},{"interface":"front","average":{"1min":13.933999999999999,"5min":21.081,"15min":106.142},"min":{"1min":5.8719999999999999,"5min":5.8719999999999999,"15min":5.8719999999999999},"max":{"1min":117.551,"5min":193.887,"15min":9585.6620000000003},"last":184.20599999999999}]},{"osd":7,"last update":"Mon Jun 3 09:05:21 2024","interfaces":[{"interface":"back","average":{"1min":25.097000000000001,"5min":22.579999999999998,"15min":24.388000000000002},"min":{"1min":7.8959999999999999,"5min":6.6130000000000004,"15min":6.6130000000000004},"max":{"1min":242.56800000000001,"5min":242.56800000000001,"15min":242.56800000000001},"last":15.646000000000001},{"interface":"front","average":{"1min":24.414999999999999,"5min":23.995999999999999,"15min":26.614999999999998},"min":{"1min":12.43,"5min":9.0660000000000007,"15min":9.0660000000000007},"max":{"1min":181.56399999999999,"5min":182.5,"15min":214.33799999999999},"last":16.282}]}]},{"osd":3,"up_from":14,"seq":60129542501,"num_pgs":11,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":58964,"kb_used_data":5704,"kb_used_omap":1,"kb_used_meta":53246,"kb_avail":94312876,"statfs":{"total":96636764160,"available":96576385024,"internally_reserved":0,"allocated":5840896,"data_stored":2740727,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1592,"internal_metadata":54524360},"hb_peers":[0,1,2,4,5,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":4,"apply_latency_ms":4,"commit_latency_ns":4000000,"apply_latency_ns":4000000},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Mon Jun 3 09:05:25 2024","interfaces":[{"interface":"back","average":{"1min":8.4809999999999999,"5min":1064.2270000000001,"15min":367.88600000000002},"min":{"1min":6.0529999999999999,"5min":5.173,"15min":4.6929999999999996},"max":{"1min":12.010999999999999,"5min":23275.167000000001,"15min":23275.167000000001},"last":7.3920000000000003},{"interface":"front","average":{"1min":13.885,"5min":1070.3710000000001,"15min":373.53500000000003},"min":{"1min":9.6679999999999993,"5min":7.2089999999999996,"15min":5.9880000000000004},"max":{"1min":32.426000000000002,"5min":23278.536,"15min":23278.536},"last":13.382}]},{"osd":1,"last update":"Mon Jun 3 09:05:25 2024","interfaces":[{"interface":"back","average":{"1min":11.324,"5min":19.920000000000002,"15min":390.07900000000001},"min":{"1min":6.6379999999999999,"5min":4.8200000000000003,"15min":4.8200000000000003},"max":{"1min":26.449000000000002,"5min":250.72300000000001,"15min":21658.205999999998},"last":5.9640000000000004},{"interface":"front","average":{"1min":9.468,"5min":20.966999999999999,"15min":381.18799999999999},"min":{"1min":5.4969999999999999,"5min":5.2779999999999996,"15min":5.2779999999999996},"max":{"1min":25.643000000000001,"5min":335.33699999999999,"15min":21307.069},"last":8.484}]},{"osd":2,"last update":"Mon Jun 3 09:05:25 2024","interfaces":[{"interface":"back","average":{"1min":13.215,"5min":26.649000000000001,"15min":23.783999999999999},"min":{"1min":7.8710000000000004,"5min":6.4900000000000002,"15min":5.875},"max":{"1min":34.322000000000003,"5min":377.803,"15min":377.803},"last":14.013999999999999},{"interface":"front","average":{"1min":13.105,"5min":22.292999999999999,"15min":20.573},"min":{"1min":7.327,"5min":6.125,"15min":5.9020000000000001},"max":{"1min":32.756999999999998,"5min":371.601,"15min":371.601},"last":9.2789999999999999}]},{"osd":4,"last update":"Mon Jun 3 09:05:27 2024","interfaces":[{"interface":"back","average":{"1min":10.444000000000001,"5min":1211.4970000000001,"15min":1639.1579999999999},"min":{"1min":6.4429999999999996,"5min":6.2560000000000002,"15min":6.0309999999999997},"max":{"1min":21.263000000000002,"5min":30144.939999999999,"15min":62679.622000000003},"last":6.734},{"interface":"front","average":{"1min":11.363,"5min":1219.6400000000001,"15min":1644.127},"min":{"1min":7.8109999999999999,"5min":7.8109999999999999,"15min":6.2439999999999998},"max":{"1min":20.108000000000001,"5min":30180.689999999999,"15min":62681.021999999997},"last":9.9329999999999998}]},{"osd":5,"last update":"Mon Jun 3 09:06:07 2024","interfaces":[{"interface":"back","average":{"1min":12.802,"5min":19.640999999999998,"15min":22.318000000000001},"min":{"1min":9.0890000000000004,"5min":6.8810000000000002,"15min":6.7690000000000001},"max":{"1min":16.745999999999999,"5min":249.88,"15min":327.34899999999999},"last":15.218},{"interface":"front","average":{"1min":15.002000000000001,"5min":24.003,"15min":26.649999999999999},"min":{"1min":11.984,"5min":9.9149999999999991,"15min":8.9879999999999995},"max":{"1min":18.407,"5min":251.47800000000001,"15min":331.86799999999999},"last":16.097000000000001}]},{"osd":6,"last update":"Mon Jun 3 09:04:04 2024","interfaces":[{"interface":"back","average":{"1min":22.815999999999999,"5min":31.513999999999999,"15min":181.74299999999999},"min":{"1min":9.4809999999999999,"5min":7.2240000000000002,"15min":7.2240000000000002},"max":{"1min":160.56999999999999,"5min":380.613,"15min":11893.74},"last":19.359999999999999},{"interface":"front","average":{"1min":25.359999999999999,"5min":30.981000000000002,"15min":179.81399999999999},"min":{"1min":9.8870000000000005,"5min":7.3360000000000003,"15min":6.0899999999999999},"max":{"1min":193.11099999999999,"5min":379.80700000000002,"15min":11893.069},"last":8.202}]},{"osd":7,"last update":"Mon Jun 3 09:05:25 2024","interfaces":[{"interface":"back","average":{"1min":12.177,"5min":21.126000000000001,"15min":22.972000000000001},"min":{"1min":8.6769999999999996,"5min":7.4279999999999999,"15min":7.4279999999999999},"max":{"1min":17.376999999999999,"5min":164.27500000000001,"15min":178.76300000000001},"last":10.628},{"interface":"front","average":{"1min":15.675000000000001,"5min":22.722000000000001,"15min":25.238},"min":{"1min":11.233000000000001,"5min":10.555999999999999,"15min":7.6440000000000001},"max":{"1min":29.280000000000001,"5min":148.13800000000001,"15min":417.22800000000001},"last":11.492000000000001}]}]},{"osd":1,"up_from":14,"seq":60129542499,"num_pgs":6,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":59348,"kb_used_data":6084,"kb_used_omap":1,"kb_used_meta":53246,"kb_avail":94312492,"statfs":{"total":96636764160,"available":96575991808,"internally_reserved":0,"allocated":6230016,"data_stored":3133138,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":54524357},"hb_peers":[0,2,3,4,5,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":9,"apply_latency_ms":9,"commit_latency_ns":9000000,"apply_latency_ns":9000000},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Mon Jun 3 09:06:07 2024","interfaces":[{"interface":"back","average":{"1min":7.7530000000000001,"5min":185.66999999999999,"15min":77.347999999999999},"min":{"1min":5.0229999999999997,"5min":4.9279999999999999,"15min":4.5449999999999999},"max":{"1min":11.896000000000001,"5min":7470.616,"15min":7470.616},"last":6.056},{"interface":"front","average":{"1min":9.9220000000000006,"5min":187.14599999999999,"15min":76.325999999999993},"min":{"1min":5.3159999999999998,"5min":5.3159999999999998,"15min":5.3159999999999998},"max":{"1min":18.338000000000001,"5min":7477.0829999999996,"15min":7477.0829999999996},"last":5.5709999999999997}]},{"osd":2,"last update":"Mon Jun 3 09:05:18 2024","interfaces":[{"interface":"back","average":{"1min":10.954000000000001,"5min":18.295000000000002,"15min":24.126999999999999},"min":{"1min":6.8739999999999997,"5min":5.8650000000000002,"15min":5.8650000000000002},"max":{"1min":21.122,"5min":228,"15min":657.38},"last":6.9740000000000002},{"interface":"front","average":{"1min":13.694000000000001,"5min":24.446999999999999,"15min":32.055999999999997},"min":{"1min":9.9860000000000007,"5min":9.1229999999999993,"15min":7.2160000000000002},"max":{"1min":26.507999999999999,"5min":229.465,"15min":700.06500000000005},"last":12.847}]},{"osd":3,"last update":"Mon Jun 3 09:06:04 2024","interfaces":[{"interface":"back","average":{"1min":8.6110000000000007,"5min":15.733000000000001,"15min":26.646999999999998},"min":{"1min":5.827,"5min":5.4640000000000004,"15min":5.4640000000000004},"max":{"1min":13.105,"5min":121.619,"15min":1676.009},"last":8.2050000000000001},{"interface":"front","average":{"1min":9.5850000000000009,"5min":16.161999999999999,"15min":29.065999999999999},"min":{"1min":6.3630000000000004,"5min":5.7699999999999996,"15min":5.7699999999999996},"max":{"1min":15.166,"5min":132.036,"15min":1766.5840000000001},"last":11.587}]},{"osd":4,"last update":"Mon Jun 3 09:05:31 2024","interfaces":[{"interface":"back","average":{"1min":10.996,"5min":954.46500000000003,"15min":1281.6859999999999},"min":{"1min":7.702,"5min":6.1760000000000002,"15min":6.0970000000000004},"max":{"1min":21.538,"5min":28044.862000000001,"15min":59438.017999999996},"last":9.1210000000000004},{"interface":"front","average":{"1min":13.646000000000001,"5min":961.24599999999998,"15min":1285.26},"min":{"1min":9.7200000000000006,"5min":9.3940000000000001,"15min":6.4640000000000004},"max":{"1min":23.277999999999999,"5min":28045.624,"15min":59439.283000000003},"last":10.907}]},{"osd":5,"last update":"Mon Jun 3 09:06:04 2024","interfaces":[{"interface":"back","average":{"1min":12.478999999999999,"5min":20.228000000000002,"15min":25.997},"min":{"1min":8.6430000000000007,"5min":7.9720000000000004,"15min":7.234},"max":{"1min":17.442,"5min":139.43199999999999,"15min":701.22000000000003},"last":9.7400000000000002},{"interface":"front","average":{"1min":13.507999999999999,"5min":25.760999999999999,"15min":27.553999999999998},"min":{"1min":10.042999999999999,"5min":7.258,"15min":6.242},"max":{"1min":20.611000000000001,"5min":278.63,"15min":598.35000000000002},"last":14.231}]},{"osd":6,"last update":"Mon Jun 3 09:03:54 2024","interfaces":[{"interface":"back","average":{"1min":28.524000000000001,"5min":20.654,"15min":23.878},"min":{"1min":6.0309999999999997,"5min":6.0309999999999997,"15min":6.0309999999999997},"max":{"1min":167.221,"5min":167.221,"15min":704.245},"last":18.721},{"interface":"front","average":{"1min":27.606000000000002,"5min":30.131,"15min":28.628},"min":{"1min":9.8650000000000002,"5min":9.7200000000000006,"15min":8.3580000000000005},"max":{"1min":123.938,"5min":358.23000000000002,"15min":634.05499999999995},"last":173.113}]},{"osd":7,"last update":"Mon Jun 3 09:05:18 2024","interfaces":[{"interface":"back","average":{"1min":21.853999999999999,"5min":24.245999999999999,"15min":32.801000000000002},"min":{"1min":9.5020000000000007,"5min":8.0559999999999992,"15min":8.0559999999999992},"max":{"1min":142.245,"5min":270.95100000000002,"15min":703.07799999999997},"last":13.542},{"interface":"front","average":{"1min":21.547000000000001,"5min":29.977,"15min":30.513999999999999},"min":{"1min":11.167,"5min":8.2089999999999996,"15min":7.7599999999999998},"max":{"1min":119.348,"5min":360.99900000000002,"15min":608.44100000000003},"last":14.951000000000001}]}]},{"osd":0,"up_from":14,"seq":60129542501,"num_pgs":10,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":81108,"kb_used_data":5704,"kb_used_omap":1,"kb_used_meta":75390,"kb_avail":94290732,"statfs":{"total":96636764160,"available":96553709568,"internally_reserved":0,"allocated":5840896,"data_stored":2740727,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1593,"internal_metadata":77199815},"hb_peers":[1,2,3,4,5,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":7,"apply_latency_ms":7,"commit_latency_ns":7000000,"apply_latency_ns":7000000},"alerts":[],"network_ping_times":[{"osd":1,"last update":"Mon Jun 3 09:06:04 2024","interfaces":[{"interface":"back","average":{"1min":6.9960000000000004,"5min":56.997,"15min":339.29700000000003},"min":{"1min":4.4530000000000003,"5min":4.4530000000000003,"15min":4.4100000000000001},"max":{"1min":11.720000000000001,"5min":2414.4749999999999,"15min":20691.453000000001},"last":6.8019999999999996},{"interface":"front","average":{"1min":9.548,"5min":57.258000000000003,"15min":337.791},"min":{"1min":5.2300000000000004,"5min":5.2300000000000004,"15min":4.6050000000000004},"max":{"1min":15.337999999999999,"5min":2412.9549999999999,"15min":20640.792000000001},"last":9.7210000000000001}]},{"osd":2,"last update":"Mon Jun 3 09:05:22 2024","interfaces":[{"interface":"back","average":{"1min":9.5820000000000007,"5min":27.760000000000002,"15min":23.097999999999999},"min":{"1min":5.766,"5min":5.766,"15min":5.766},"max":{"1min":46.584000000000003,"5min":627.40700000000004,"15min":627.40700000000004},"last":5.9109999999999996},{"interface":"front","average":{"1min":13.574999999999999,"5min":35.847999999999999,"15min":30.065000000000001},"min":{"1min":8.6419999999999995,"5min":7.8630000000000004,"15min":6.9260000000000002},"max":{"1min":54.68,"5min":635.62599999999998,"15min":635.62599999999998},"last":10.257}]},{"osd":3,"last update":"Mon Jun 3 09:06:04 2024","interfaces":[{"interface":"back","average":{"1min":12.105,"5min":26.925000000000001,"15min":26.584},"min":{"1min":7.8410000000000002,"5min":7.1280000000000001,"15min":6.1909999999999998},"max":{"1min":18.425999999999998,"5min":389.72699999999998,"15min":389.72699999999998},"last":7.6399999999999997},{"interface":"front","average":{"1min":13.794,"5min":29.771999999999998,"15min":28.815000000000001},"min":{"1min":10.084,"5min":9.4920000000000009,"15min":6.1849999999999996},"max":{"1min":17.373999999999999,"5min":390.23200000000003,"15min":494.92899999999997},"last":12.619}]},{"osd":4,"last update":"Mon Jun 3 09:05:27 2024","interfaces":[{"interface":"back","average":{"1min":13.478,"5min":1032.126,"15min":1389.729},"min":{"1min":6.5119999999999996,"5min":6.5119999999999996,"15min":6.5119999999999996},"max":{"1min":55.354999999999997,"5min":25555.212,"15min":60989.964},"last":9.1739999999999995},{"interface":"front","average":{"1min":16.204999999999998,"5min":1030.8389999999999,"15min":1387.778},"min":{"1min":10.282999999999999,"5min":7.1429999999999998,"15min":7.1429999999999998},"max":{"1min":49.945999999999998,"5min":25556.159,"15min":61000.017},"last":11.471}]},{"osd":5,"last update":"Mon Jun 3 09:06:17 2024","interfaces":[{"interface":"back","average":{"1min":9.7040000000000006,"5min":26.533999999999999,"15min":28.683},"min":{"1min":6.8129999999999997,"5min":6.8129999999999997,"15min":6.8129999999999997},"max":{"1min":14.942,"5min":455.78699999999998,"15min":605.05899999999997},"last":13.146000000000001},{"interface":"front","average":{"1min":14.279,"5min":28.398,"15min":26.192},"min":{"1min":11.696,"5min":6.4420000000000002,"15min":6.2030000000000003},"max":{"1min":17.879000000000001,"5min":391.31200000000001,"15min":541.11000000000001},"last":14.859999999999999}]},{"osd":6,"last update":"Mon Jun 3 09:04:30 2024","interfaces":[{"interface":"back","average":{"1min":52.496000000000002,"5min":33.139000000000003,"15min":160.83199999999999},"min":{"1min":7.5289999999999999,"5min":7.1040000000000001,"15min":6.3739999999999997},"max":{"1min":457.91699999999997,"5min":457.91699999999997,"15min":11115.258},"last":18.959},{"interface":"front","average":{"1min":49.670000000000002,"5min":33.704999999999998,"15min":162.791},"min":{"1min":12.247999999999999,"5min":11.019,"15min":9.9760000000000009},"max":{"1min":407.05099999999999,"5min":407.05099999999999,"15min":11092.194},"last":17.707000000000001}]},{"osd":7,"last update":"Mon Jun 3 09:05:22 2024","interfaces":[{"interface":"back","average":{"1min":14.503,"5min":29.489000000000001,"15min":33.134999999999998},"min":{"1min":8.5690000000000008,"5min":7.5149999999999997,"15min":6.9859999999999998},"max":{"1min":48.526000000000003,"5min":461.41699999999997,"15min":598.33699999999999},"last":11.723000000000001},{"interface":"front","average":{"1min":17.050000000000001,"5min":33.313000000000002,"15min":38.603999999999999},"min":{"1min":11.182,"5min":10.263999999999999,"15min":10.263999999999999},"max":{"1min":58.683,"5min":463.10599999999999,"15min":592.80499999999995},"last":14.053000000000001}]}]}],"pool_statfs":[{"poolid":1,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":401408,"data_stored":397840,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":5,"total":0,"available":0,"internally_reserved":0,"allocated":401408,"data_stored":397840,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":4,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":5,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":6,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":17,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":17,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":17,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":17,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":17,"osd":4,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":17,"osd":5,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":5,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":17,"osd":6,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":5,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":17,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0}]}} 2024-06-03T09:06:21.642 INFO:tasks.ceph.ceph_manager.ceph:PG 17.c is not active+clean 2024-06-03T09:06:21.642 INFO:tasks.ceph.ceph_manager.ceph:{'pgid': '17.c', 'version': "0'0", 'reported_seq': 953, 'reported_epoch': 1009, 'state': 'active+undersized', 'last_fresh': '2024-06-03T09:06:13.668156+0000', 'last_change': '2024-06-03T09:06:03.440074+0000', 'last_active': '2024-06-03T09:06:13.668156+0000', 'last_peered': '2024-06-03T09:06:13.668156+0000', 'last_clean': '2024-06-03T09:05:02.105638+0000', 'last_became_active': '2024-06-03T09:06:03.440074+0000', 'last_became_peered': '2024-06-03T09:06:03.440074+0000', 'last_unstale': '2024-06-03T09:06:13.668156+0000', 'last_undegraded': '2024-06-03T09:06:13.668156+0000', 'last_fullsized': '2024-06-03T09:06:03.377982+0000', 'mapping_epoch': 999, 'log_start': "0'0", 'ondisk_log_start': "0'0", 'created': 28, 'last_epoch_clean': 255, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2024-06-03T08:37:55.614497+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2024-06-03T08:37:55.614497+0000', 'last_clean_scrub_stamp': '2024-06-03T08:37:55.614497+0000', 'objects_scrubbed': 0, 'log_size': 0, 'log_dups_size': 0, 'ondisk_log_size': 0, 'stats_invalid': False, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': 'periodic scrub scheduled @ 2024-06-04T10:34:20.808557+0000', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 0, 'num_objects': 0, 'num_object_clones': 0, 'num_object_copies': 0, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 0, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 0, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 0, 'num_write_kb': 0, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 0, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [3], 'acting': [3], 'avail_no_missing': ['3'], 'object_location_counts': [], 'blocked_by': [], 'up_primary': 3, 'acting_primary': 3, 'purged_snaps': []} 2024-06-03T09:06:21.643 INFO:tasks.ceph.ceph_manager.ceph:PG 17.f is not active+clean 2024-06-03T09:06:21.643 INFO:tasks.ceph.ceph_manager.ceph:{'pgid': '17.f', 'version': "0'0", 'reported_seq': 148, 'reported_epoch': 1009, 'state': 'active+undersized', 'last_fresh': '2024-06-03T09:06:14.450543+0000', 'last_change': '2024-06-03T09:06:03.420285+0000', 'last_active': '2024-06-03T09:06:14.450543+0000', 'last_peered': '2024-06-03T09:06:14.450543+0000', 'last_clean': '2024-06-03T08:37:55.614497+0000', 'last_became_active': '2024-06-03T09:06:03.420285+0000', 'last_became_peered': '2024-06-03T09:06:03.420285+0000', 'last_unstale': '2024-06-03T09:06:14.450543+0000', 'last_undegraded': '2024-06-03T09:06:14.450543+0000', 'last_fullsized': '2024-06-03T09:06:03.380616+0000', 'mapping_epoch': 999, 'log_start': "0'0", 'ondisk_log_start': "0'0", 'created': 28, 'last_epoch_clean': 255, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2024-06-03T08:37:55.614497+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2024-06-03T08:37:55.614497+0000', 'last_clean_scrub_stamp': '2024-06-03T08:37:55.614497+0000', 'objects_scrubbed': 0, 'log_size': 0, 'log_dups_size': 0, 'ondisk_log_size': 0, 'stats_invalid': False, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': 'periodic scrub scheduled @ 2024-06-04T15:39:41.187731+0000', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 0, 'num_objects': 0, 'num_object_clones': 0, 'num_object_copies': 0, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 0, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 0, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 0, 'num_write_kb': 0, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 0, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [4], 'acting': [4], 'avail_no_missing': ['4'], 'object_location_counts': [], 'blocked_by': [], 'up_primary': 4, 'acting_primary': 4, 'purged_snaps': []} 2024-06-03T09:06:21.643 INFO:tasks.ceph.ceph_manager.ceph:PG 17.e is not active+clean 2024-06-03T09:06:21.643 INFO:tasks.ceph.ceph_manager.ceph:{'pgid': '17.e', 'version': "0'0", 'reported_seq': 149, 'reported_epoch': 1009, 'state': 'active+undersized', 'last_fresh': '2024-06-03T09:06:13.726992+0000', 'last_change': '2024-06-03T09:06:03.412887+0000', 'last_active': '2024-06-03T09:06:13.726992+0000', 'last_peered': '2024-06-03T09:06:13.726992+0000', 'last_clean': '2024-06-03T08:37:55.614497+0000', 'last_became_active': '2024-06-03T09:06:03.412887+0000', 'last_became_peered': '2024-06-03T09:06:03.412887+0000', 'last_unstale': '2024-06-03T09:06:13.726992+0000', 'last_undegraded': '2024-06-03T09:06:13.726992+0000', 'last_fullsized': '2024-06-03T09:06:03.332482+0000', 'mapping_epoch': 999, 'log_start': "0'0", 'ondisk_log_start': "0'0", 'created': 28, 'last_epoch_clean': 255, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2024-06-03T08:37:55.614497+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2024-06-03T08:37:55.614497+0000', 'last_clean_scrub_stamp': '2024-06-03T08:37:55.614497+0000', 'objects_scrubbed': 0, 'log_size': 0, 'log_dups_size': 0, 'ondisk_log_size': 0, 'stats_invalid': False, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': 'periodic scrub scheduled @ 2024-06-04T12:23:22.511601+0000', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 0, 'num_objects': 0, 'num_object_clones': 0, 'num_object_copies': 0, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 0, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 0, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 0, 'num_write_kb': 0, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 0, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [5], 'acting': [5], 'avail_no_missing': ['5'], 'object_location_counts': [], 'blocked_by': [], 'up_primary': 5, 'acting_primary': 5, 'purged_snaps': []} 2024-06-03T09:06:21.643 INFO:tasks.ceph.ceph_manager.ceph:PG 17.8 is not active+clean 2024-06-03T09:06:21.643 INFO:tasks.ceph.ceph_manager.ceph:{'pgid': '17.8', 'version': "0'0", 'reported_seq': 961, 'reported_epoch': 1009, 'state': 'active+undersized', 'last_fresh': '2024-06-03T09:06:14.430375+0000', 'last_change': '2024-06-03T09:06:03.388786+0000', 'last_active': '2024-06-03T09:06:14.430375+0000', 'last_peered': '2024-06-03T09:06:14.430375+0000', 'last_clean': '2024-06-03T09:04:43.297780+0000', 'last_became_active': '2024-06-03T09:06:03.388786+0000', 'last_became_peered': '2024-06-03T09:06:03.388786+0000', 'last_unstale': '2024-06-03T09:06:14.430375+0000', 'last_undegraded': '2024-06-03T09:06:14.430375+0000', 'last_fullsized': '2024-06-03T09:06:03.331754+0000', 'mapping_epoch': 999, 'log_start': "0'0", 'ondisk_log_start': "0'0", 'created': 28, 'last_epoch_clean': 255, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2024-06-03T08:37:55.614497+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2024-06-03T08:37:55.614497+0000', 'last_clean_scrub_stamp': '2024-06-03T08:37:55.614497+0000', 'objects_scrubbed': 0, 'log_size': 0, 'log_dups_size': 0, 'ondisk_log_size': 0, 'stats_invalid': False, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': 'periodic scrub scheduled @ 2024-06-04T09:18:44.765547+0000', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 0, 'num_objects': 0, 'num_object_clones': 0, 'num_object_copies': 0, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 0, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 0, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 0, 'num_write_kb': 0, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 0, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [0], 'acting': [0], 'avail_no_missing': ['0'], 'object_location_counts': [], 'blocked_by': [], 'up_primary': 0, 'acting_primary': 0, 'purged_snaps': []} 2024-06-03T09:06:21.643 INFO:tasks.ceph.ceph_manager.ceph:PG 17.b is not active+clean 2024-06-03T09:06:21.643 INFO:tasks.ceph.ceph_manager.ceph:{'pgid': '17.b', 'version': "0'0", 'reported_seq': 149, 'reported_epoch': 1009, 'state': 'active+undersized', 'last_fresh': '2024-06-03T09:06:14.433735+0000', 'last_change': '2024-06-03T09:06:03.451203+0000', 'last_active': '2024-06-03T09:06:14.433735+0000', 'last_peered': '2024-06-03T09:06:14.433735+0000', 'last_clean': '2024-06-03T08:37:55.614497+0000', 'last_became_active': '2024-06-03T09:06:03.451203+0000', 'last_became_peered': '2024-06-03T09:06:03.451203+0000', 'last_unstale': '2024-06-03T09:06:14.433735+0000', 'last_undegraded': '2024-06-03T09:06:14.433735+0000', 'last_fullsized': '2024-06-03T09:06:03.375523+0000', 'mapping_epoch': 999, 'log_start': "0'0", 'ondisk_log_start': "0'0", 'created': 28, 'last_epoch_clean': 255, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2024-06-03T08:37:55.614497+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2024-06-03T08:37:55.614497+0000', 'last_clean_scrub_stamp': '2024-06-03T08:37:55.614497+0000', 'objects_scrubbed': 0, 'log_size': 0, 'log_dups_size': 0, 'ondisk_log_size': 0, 'stats_invalid': False, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': 'periodic scrub scheduled @ 2024-06-04T10:16:19.813495+0000', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 0, 'num_objects': 0, 'num_object_clones': 0, 'num_object_copies': 0, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 0, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 0, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 0, 'num_write_kb': 0, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 0, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [2], 'acting': [2], 'avail_no_missing': ['2'], 'object_location_counts': [], 'blocked_by': [], 'up_primary': 2, 'acting_primary': 2, 'purged_snaps': []} 2024-06-03T09:06:21.643 INFO:tasks.ceph.ceph_manager.ceph:PG 17.4 is not active+clean 2024-06-03T09:06:21.644 INFO:tasks.ceph.ceph_manager.ceph:{'pgid': '17.4', 'version': "0'0", 'reported_seq': 149, 'reported_epoch': 1009, 'state': 'active+undersized', 'last_fresh': '2024-06-03T09:06:13.941522+0000', 'last_change': '2024-06-03T09:06:03.441889+0000', 'last_active': '2024-06-03T09:06:13.941522+0000', 'last_peered': '2024-06-03T09:06:13.941522+0000', 'last_clean': '2024-06-03T08:37:55.614497+0000', 'last_became_active': '2024-06-03T09:06:03.441889+0000', 'last_became_peered': '2024-06-03T09:06:03.441889+0000', 'last_unstale': '2024-06-03T09:06:13.941522+0000', 'last_undegraded': '2024-06-03T09:06:13.941522+0000', 'last_fullsized': '2024-06-03T09:06:03.359582+0000', 'mapping_epoch': 999, 'log_start': "0'0", 'ondisk_log_start': "0'0", 'created': 28, 'last_epoch_clean': 255, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2024-06-03T08:37:55.614497+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2024-06-03T08:37:55.614497+0000', 'last_clean_scrub_stamp': '2024-06-03T08:37:55.614497+0000', 'objects_scrubbed': 0, 'log_size': 0, 'log_dups_size': 0, 'ondisk_log_size': 0, 'stats_invalid': False, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': 'periodic scrub scheduled @ 2024-06-04T12:49:22.907141+0000', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 0, 'num_objects': 0, 'num_object_clones': 0, 'num_object_copies': 0, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 0, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 0, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 0, 'num_write_kb': 0, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 0, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [7], 'acting': [7], 'avail_no_missing': ['7'], 'object_location_counts': [], 'blocked_by': [], 'up_primary': 7, 'acting_primary': 7, 'purged_snaps': []} 2024-06-03T09:06:21.644 INFO:tasks.ceph.ceph_manager.ceph:PG 17.6 is not active+clean 2024-06-03T09:06:21.644 INFO:tasks.ceph.ceph_manager.ceph:{'pgid': '17.6', 'version': "0'0", 'reported_seq': 960, 'reported_epoch': 1009, 'state': 'active+undersized', 'last_fresh': '2024-06-03T09:06:14.453110+0000', 'last_change': '2024-06-03T09:06:03.428677+0000', 'last_active': '2024-06-03T09:06:14.453110+0000', 'last_peered': '2024-06-03T09:06:14.453110+0000', 'last_clean': '2024-06-03T09:04:43.312352+0000', 'last_became_active': '2024-06-03T09:06:03.428677+0000', 'last_became_peered': '2024-06-03T09:06:03.428677+0000', 'last_unstale': '2024-06-03T09:06:14.453110+0000', 'last_undegraded': '2024-06-03T09:06:14.453110+0000', 'last_fullsized': '2024-06-03T09:06:03.359909+0000', 'mapping_epoch': 999, 'log_start': "0'0", 'ondisk_log_start': "0'0", 'created': 28, 'last_epoch_clean': 255, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2024-06-03T08:37:55.614497+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2024-06-03T08:37:55.614497+0000', 'last_clean_scrub_stamp': '2024-06-03T08:37:55.614497+0000', 'objects_scrubbed': 0, 'log_size': 0, 'log_dups_size': 0, 'ondisk_log_size': 0, 'stats_invalid': False, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': 'periodic scrub scheduled @ 2024-06-04T15:38:02.180241+0000', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 0, 'num_objects': 0, 'num_object_clones': 0, 'num_object_copies': 0, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 0, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 0, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 0, 'num_write_kb': 0, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 0, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [0], 'acting': [0], 'avail_no_missing': ['0'], 'object_location_counts': [], 'blocked_by': [], 'up_primary': 0, 'acting_primary': 0, 'purged_snaps': []} 2024-06-03T09:06:21.644 INFO:tasks.ceph.ceph_manager.ceph:PG 17.16 is not active+clean 2024-06-03T09:06:21.644 INFO:tasks.ceph.ceph_manager.ceph:{'pgid': '17.16', 'version': "0'0", 'reported_seq': 972, 'reported_epoch': 1009, 'state': 'active+undersized', 'last_fresh': '2024-06-03T09:06:14.441665+0000', 'last_change': '2024-06-03T09:06:03.478615+0000', 'last_active': '2024-06-03T09:06:14.441665+0000', 'last_peered': '2024-06-03T09:06:14.441665+0000', 'last_clean': '2024-06-03T09:04:43.328415+0000', 'last_became_active': '2024-06-03T09:06:03.478615+0000', 'last_became_peered': '2024-06-03T09:06:03.478615+0000', 'last_unstale': '2024-06-03T09:06:14.441665+0000', 'last_undegraded': '2024-06-03T09:06:14.441665+0000', 'last_fullsized': '2024-06-03T09:06:03.402504+0000', 'mapping_epoch': 999, 'log_start': "0'0", 'ondisk_log_start': "0'0", 'created': 28, 'last_epoch_clean': 255, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2024-06-03T08:37:55.614497+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2024-06-03T08:37:55.614497+0000', 'last_clean_scrub_stamp': '2024-06-03T08:37:55.614497+0000', 'objects_scrubbed': 0, 'log_size': 0, 'log_dups_size': 0, 'ondisk_log_size': 0, 'stats_invalid': False, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': 'periodic scrub scheduled @ 2024-06-04T08:47:08.079233+0000', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 0, 'num_objects': 0, 'num_object_clones': 0, 'num_object_copies': 0, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 0, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 0, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 0, 'num_write_kb': 0, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 0, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [2], 'acting': [2], 'avail_no_missing': ['2'], 'object_location_counts': [], 'blocked_by': [], 'up_primary': 2, 'acting_primary': 2, 'purged_snaps': []} 2024-06-03T09:06:21.644 INFO:tasks.ceph.ceph_manager.ceph:PG 2.4 is not active+clean 2024-06-03T09:06:21.644 INFO:tasks.ceph.ceph_manager.ceph:{'pgid': '2.4', 'version': "0'0", 'reported_seq': 952, 'reported_epoch': 1010, 'state': 'active+undersized', 'last_fresh': '2024-06-03T09:06:18.331485+0000', 'last_change': '2024-06-03T09:06:03.387171+0000', 'last_active': '2024-06-03T09:06:18.331485+0000', 'last_peered': '2024-06-03T09:06:18.331485+0000', 'last_clean': '2024-06-03T09:04:43.202645+0000', 'last_became_active': '2024-06-03T09:06:03.387171+0000', 'last_became_peered': '2024-06-03T09:06:03.387171+0000', 'last_unstale': '2024-06-03T09:06:18.331485+0000', 'last_undegraded': '2024-06-03T09:06:18.331485+0000', 'last_fullsized': '2024-06-03T09:06:03.283787+0000', 'mapping_epoch': 999, 'log_start': "0'0", 'ondisk_log_start': "0'0", 'created': 23, 'last_epoch_clean': 255, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2024-06-03T08:36:28.444764+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2024-06-03T08:36:28.444764+0000', 'last_clean_scrub_stamp': '2024-06-03T08:36:28.444764+0000', 'objects_scrubbed': 0, 'log_size': 0, 'log_dups_size': 0, 'ondisk_log_size': 0, 'stats_invalid': False, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': 'periodic scrub scheduled @ 2024-06-04T20:17:43.485023+0000', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 0, 'num_objects': 0, 'num_object_clones': 0, 'num_object_copies': 0, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 0, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 0, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 0, 'num_write_kb': 0, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 0, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [5], 'acting': [5], 'avail_no_missing': ['5'], 'object_location_counts': [], 'blocked_by': [], 'up_primary': 5, 'acting_primary': 5, 'purged_snaps': []} 2024-06-03T09:06:21.644 INFO:tasks.ceph.ceph_manager.ceph:PG 2.2 is not active+clean 2024-06-03T09:06:21.644 INFO:tasks.ceph.ceph_manager.ceph:{'pgid': '2.2', 'version': "0'0", 'reported_seq': 963, 'reported_epoch': 1009, 'state': 'active+undersized', 'last_fresh': '2024-06-03T09:06:13.657892+0000', 'last_change': '2024-06-03T09:06:03.410020+0000', 'last_active': '2024-06-03T09:06:13.657892+0000', 'last_peered': '2024-06-03T09:06:13.657892+0000', 'last_clean': '2024-06-03T09:05:37.145889+0000', 'last_became_active': '2024-06-03T09:06:03.410020+0000', 'last_became_peered': '2024-06-03T09:06:03.410020+0000', 'last_unstale': '2024-06-03T09:06:13.657892+0000', 'last_undegraded': '2024-06-03T09:06:13.657892+0000', 'last_fullsized': '2024-06-03T09:06:03.346864+0000', 'mapping_epoch': 999, 'log_start': "0'0", 'ondisk_log_start': "0'0", 'created': 23, 'last_epoch_clean': 255, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2024-06-03T08:36:28.444764+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2024-06-03T08:36:28.444764+0000', 'last_clean_scrub_stamp': '2024-06-03T08:36:28.444764+0000', 'objects_scrubbed': 0, 'log_size': 0, 'log_dups_size': 0, 'ondisk_log_size': 0, 'stats_invalid': False, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': 'periodic scrub scheduled @ 2024-06-04T09:57:12.423360+0000', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 0, 'num_objects': 0, 'num_object_clones': 0, 'num_object_copies': 0, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 0, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 0, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 0, 'num_write_kb': 0, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 0, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [3], 'acting': [3], 'avail_no_missing': ['3'], 'object_location_counts': [], 'blocked_by': [], 'up_primary': 3, 'acting_primary': 3, 'purged_snaps': []} 2024-06-03T09:06:21.644 INFO:tasks.ceph.ceph_manager.ceph:PG 17.1a is not active+clean 2024-06-03T09:06:21.644 INFO:tasks.ceph.ceph_manager.ceph:{'pgid': '17.1a', 'version': "0'0", 'reported_seq': 960, 'reported_epoch': 1009, 'state': 'active+undersized', 'last_fresh': '2024-06-03T09:06:14.431224+0000', 'last_change': '2024-06-03T09:06:03.390734+0000', 'last_active': '2024-06-03T09:06:14.431224+0000', 'last_peered': '2024-06-03T09:06:14.431224+0000', 'last_clean': '2024-06-03T09:05:20.322313+0000', 'last_became_active': '2024-06-03T09:06:03.390734+0000', 'last_became_peered': '2024-06-03T09:06:03.390734+0000', 'last_unstale': '2024-06-03T09:06:14.431224+0000', 'last_undegraded': '2024-06-03T09:06:14.431224+0000', 'last_fullsized': '2024-06-03T09:06:03.343197+0000', 'mapping_epoch': 999, 'log_start': "0'0", 'ondisk_log_start': "0'0", 'created': 28, 'last_epoch_clean': 254, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2024-06-03T08:37:55.614497+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2024-06-03T08:37:55.614497+0000', 'last_clean_scrub_stamp': '2024-06-03T08:37:55.614497+0000', 'objects_scrubbed': 0, 'log_size': 0, 'log_dups_size': 0, 'ondisk_log_size': 0, 'stats_invalid': False, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': 'periodic scrub scheduled @ 2024-06-04T15:56:47.628875+0000', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 0, 'num_objects': 0, 'num_object_clones': 0, 'num_object_copies': 0, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 0, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 0, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 0, 'num_write_kb': 0, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 0, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [4], 'acting': [4], 'avail_no_missing': ['4'], 'object_location_counts': [], 'blocked_by': [], 'up_primary': 4, 'acting_primary': 4, 'purged_snaps': []} 2024-06-03T09:06:21.645 INFO:tasks.ceph.ceph_manager.ceph:PG 17.1e is not active+clean 2024-06-03T09:06:21.645 INFO:tasks.ceph.ceph_manager.ceph:{'pgid': '17.1e', 'version': "0'0", 'reported_seq': 148, 'reported_epoch': 1009, 'state': 'active+undersized', 'last_fresh': '2024-06-03T09:06:14.445917+0000', 'last_change': '2024-06-03T09:06:03.406181+0000', 'last_active': '2024-06-03T09:06:14.445917+0000', 'last_peered': '2024-06-03T09:06:14.445917+0000', 'last_clean': '2024-06-03T08:37:55.614497+0000', 'last_became_active': '2024-06-03T09:06:03.406181+0000', 'last_became_peered': '2024-06-03T09:06:03.406181+0000', 'last_unstale': '2024-06-03T09:06:14.445917+0000', 'last_undegraded': '2024-06-03T09:06:14.445917+0000', 'last_fullsized': '2024-06-03T09:06:03.363739+0000', 'mapping_epoch': 999, 'log_start': "0'0", 'ondisk_log_start': "0'0", 'created': 28, 'last_epoch_clean': 255, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2024-06-03T08:37:55.614497+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2024-06-03T08:37:55.614497+0000', 'last_clean_scrub_stamp': '2024-06-03T08:37:55.614497+0000', 'objects_scrubbed': 0, 'log_size': 0, 'log_dups_size': 0, 'ondisk_log_size': 0, 'stats_invalid': False, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': 'periodic scrub scheduled @ 2024-06-04T11:45:24.486600+0000', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 0, 'num_objects': 0, 'num_object_clones': 0, 'num_object_copies': 0, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 0, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 0, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 0, 'num_write_kb': 0, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 0, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [4], 'acting': [4], 'avail_no_missing': ['4'], 'object_location_counts': [], 'blocked_by': [], 'up_primary': 4, 'acting_primary': 4, 'purged_snaps': []} 2024-06-03T09:06:21.645 INFO:tasks.ceph.ceph_manager.ceph:PG 17.1c is not active+clean 2024-06-03T09:06:21.645 INFO:tasks.ceph.ceph_manager.ceph:{'pgid': '17.1c', 'version': "0'0", 'reported_seq': 943, 'reported_epoch': 1009, 'state': 'active+undersized', 'last_fresh': '2024-06-03T09:06:13.719240+0000', 'last_change': '2024-06-03T09:06:03.382210+0000', 'last_active': '2024-06-03T09:06:13.719240+0000', 'last_peered': '2024-06-03T09:06:13.719240+0000', 'last_clean': '2024-06-03T09:04:43.199660+0000', 'last_became_active': '2024-06-03T09:06:03.382210+0000', 'last_became_peered': '2024-06-03T09:06:03.382210+0000', 'last_unstale': '2024-06-03T09:06:13.719240+0000', 'last_undegraded': '2024-06-03T09:06:13.719240+0000', 'last_fullsized': '2024-06-03T09:06:03.277874+0000', 'mapping_epoch': 999, 'log_start': "0'0", 'ondisk_log_start': "0'0", 'created': 28, 'last_epoch_clean': 255, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2024-06-03T08:37:55.614497+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2024-06-03T08:37:55.614497+0000', 'last_clean_scrub_stamp': '2024-06-03T08:37:55.614497+0000', 'objects_scrubbed': 0, 'log_size': 0, 'log_dups_size': 0, 'ondisk_log_size': 0, 'stats_invalid': False, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': 'periodic scrub scheduled @ 2024-06-04T12:18:50.013884+0000', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 0, 'num_objects': 0, 'num_object_clones': 0, 'num_object_copies': 0, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 0, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 0, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 0, 'num_write_kb': 0, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 0, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [5], 'acting': [5], 'avail_no_missing': ['5'], 'object_location_counts': [], 'blocked_by': [], 'up_primary': 5, 'acting_primary': 5, 'purged_snaps': []} 2024-06-03T09:06:21.645 INFO:tasks.ceph.ceph_manager.ceph:PG 17.1 is not active+clean 2024-06-03T09:06:21.645 INFO:tasks.ceph.ceph_manager.ceph:{'pgid': '17.1', 'version': "65'7", 'reported_seq': 959, 'reported_epoch': 1009, 'state': 'active+undersized+degraded', 'last_fresh': '2024-06-03T09:06:13.709209+0000', 'last_change': '2024-06-03T09:06:03.370660+0000', 'last_active': '2024-06-03T09:06:13.709209+0000', 'last_peered': '2024-06-03T09:06:13.709209+0000', 'last_clean': '2024-06-03T09:05:03.733864+0000', 'last_became_active': '2024-06-03T09:06:03.370660+0000', 'last_became_peered': '2024-06-03T09:06:03.370660+0000', 'last_unstale': '2024-06-03T09:06:13.709209+0000', 'last_undegraded': '2024-06-03T09:06:03.266076+0000', 'last_fullsized': '2024-06-03T09:06:03.264525+0000', 'mapping_epoch': 999, 'log_start': "0'0", 'ondisk_log_start': "0'0", 'created': 28, 'last_epoch_clean': 255, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2024-06-03T08:37:55.614497+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2024-06-03T08:37:55.614497+0000', 'last_clean_scrub_stamp': '2024-06-03T08:37:55.614497+0000', 'objects_scrubbed': 0, 'log_size': 7, 'log_dups_size': 0, 'ondisk_log_size': 7, 'stats_invalid': False, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': 'periodic scrub scheduled @ 2024-06-04T11:55:18.194283+0000', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 5, 'num_objects': 1, 'num_object_clones': 0, 'num_object_copies': 2, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 1, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 1, 'num_whiteouts': 0, 'num_read': 6, 'num_read_kb': 6, 'num_write': 7, 'num_write_kb': 7, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 0, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [5], 'acting': [5], 'avail_no_missing': ['5'], 'object_location_counts': [{'shards': '5', 'objects': 1}], 'blocked_by': [], 'up_primary': 5, 'acting_primary': 5, 'purged_snaps': []} 2024-06-03T09:06:21.645 DEBUG:teuthology.orchestra.run.smithi088:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph status --format=json 2024-06-03T09:06:21.741 INFO:teuthology.orchestra.run.smithi088.stderr:2024-06-03T09:06:21.741+0000 7fae7b7a9640 10 monclient: get_monmap_and_config 2024-06-03T09:06:21.742 INFO:teuthology.orchestra.run.smithi088.stderr:2024-06-03T09:06:21.741+0000 7fae7b7a9640 10 monclient: build_initial_monmap 2024-06-03T09:06:21.742 INFO:teuthology.orchestra.run.smithi088.stderr:2024-06-03T09:06:21.741+0000 7fae7b7a9640 10 monclient: monmap: 2024-06-03T09:06:21.742 INFO:teuthology.orchestra.run.smithi088.stderr:epoch 0 2024-06-03T09:06:21.742 INFO:teuthology.orchestra.run.smithi088.stderr:fsid ec3c3fdd-3ac6-4d7d-9412-683130ed8b04 2024-06-03T09:06:21.742 INFO:teuthology.orchestra.run.smithi088.stderr:last_changed 2024-06-03T09:06:21.742955+0000 2024-06-03T09:06:21.743 INFO:teuthology.orchestra.run.smithi088.stderr:created 2024-06-03T09:06:21.742955+0000 2024-06-03T09:06:21.743 INFO:teuthology.orchestra.run.smithi088.stderr:min_mon_release 0 (unknown) 2024-06-03T09:06:21.743 INFO:teuthology.orchestra.run.smithi088.stderr:election_strategy: 3 2024-06-03T09:06:21.743 INFO:teuthology.orchestra.run.smithi088.stderr:0: v1:172.21.15.88:6789/0 mon.noname-a 2024-06-03T09:06:21.743 INFO:teuthology.orchestra.run.smithi088.stderr:1: v1:172.21.15.192:6789/0 mon.noname-c 2024-06-03T09:06:21.743 INFO:teuthology.orchestra.run.smithi088.stderr:2: v1:172.21.15.88:6790/0 mon.noname-b 2024-06-03T09:06:21.743 INFO:teuthology.orchestra.run.smithi088.stderr: 2024-06-03T09:06:21.744 INFO:teuthology.orchestra.run.smithi088.stderr:2024-06-03T09:06:21.742+0000 7fae7b7a9640 1 Processor -- start 2024-06-03T09:06:21.744 INFO:teuthology.orchestra.run.smithi088.stderr:2024-06-03T09:06:21.742+0000 7fae7b7a9640 1 -- start start 2024-06-03T09:06:21.744 INFO:teuthology.orchestra.run.smithi088.stderr:2024-06-03T09:06:21.742+0000 7fae7b7a9640 10 monclient: init 2024-06-03T09:06:21.744 INFO:teuthology.orchestra.run.smithi088.stderr:2024-06-03T09:06:21.743+0000 7fae7b7a9640 10 monclient: _reopen_session rank -1 2024-06-03T09:06:21.745 INFO:teuthology.orchestra.run.smithi088.stderr:2024-06-03T09:06:21.743+0000 7fae7b7a9640 10 monclient: _add_conns ranks=[2,1,0] 2024-06-03T09:06:21.745 INFO:teuthology.orchestra.run.smithi088.stderr:2024-06-03T09:06:21.743+0000 7fae7b7a9640 10 monclient(hunting): picked mon.noname-b con 0x7fae7406f970 addr v1:172.21.15.88:6790/0 2024-06-03T09:06:21.745 INFO:teuthology.orchestra.run.smithi088.stderr:2024-06-03T09:06:21.743+0000 7fae7b7a9640 10 monclient(hunting): picked mon.noname-c con 0x7fae740ad840 addr v1:172.21.15.192:6789/0 2024-06-03T09:06:21.745 INFO:teuthology.orchestra.run.smithi088.stderr:2024-06-03T09:06:21.743+0000 7fae7b7a9640 10 monclient(hunting): picked mon.noname-a con 0x7fae7406fd50 addr v1:172.21.15.88:6789/0 2024-06-03T09:06:21.745 INFO:teuthology.orchestra.run.smithi088.stderr:2024-06-03T09:06:21.743+0000 7fae7b7a9640 1 -- --> v1:172.21.15.88:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fae74071be0 con 0x7fae7406fd50 2024-06-03T09:06:21.745 INFO:teuthology.orchestra.run.smithi088.stderr:2024-06-03T09:06:21.743+0000 7fae7b7a9640 1 -- --> v1:172.21.15.192:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fae74071ec0 con 0x7fae740ad840 2024-06-03T09:06:21.745 INFO:teuthology.orchestra.run.smithi088.stderr:2024-06-03T09:06:21.743+0000 7fae7b7a9640 1 -- --> v1:172.21.15.88:6790/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fae740708e0 con 0x7fae7406f970 2024-06-03T09:06:21.745 INFO:teuthology.orchestra.run.smithi088.stderr:2024-06-03T09:06:21.743+0000 7fae7b7a9640 10 monclient(hunting): _renew_subs 2024-06-03T09:06:21.745 INFO:teuthology.orchestra.run.smithi088.stderr:2024-06-03T09:06:21.743+0000 7fae7b7a9640 10 monclient(hunting): authenticate will time out at 3037.553310s 2024-06-03T09:06:21.745 INFO:teuthology.orchestra.run.smithi088.stderr:2024-06-03T09:06:21.743+0000 7fae79d46640 1 --1- >> v1:172.21.15.88:6789/0 conn(0x7fae7406fd50 0x7fae7414abf0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.88:6789/0 says I am v1:172.21.15.88:43928/0 (socket says 172.21.15.88:43928) 2024-06-03T09:06:21.745 INFO:teuthology.orchestra.run.smithi088.stderr:2024-06-03T09:06:21.743+0000 7fae79d46640 1 -- 172.21.15.88:0/3152404876 learned_addr learned my addr 172.21.15.88:0/3152404876 (peer_addr_for_me v1:172.21.15.88:0/0) 2024-06-03T09:06:21.746 INFO:teuthology.orchestra.run.smithi088.stderr:2024-06-03T09:06:21.746+0000 7fae6bfff640 1 -- 172.21.15.88:0/3152404876 <== mon.1 v1:172.21.15.192:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 870009727 0 0) 0x7fae74071ec0 con 0x7fae740ad840 2024-06-03T09:06:21.746 INFO:teuthology.orchestra.run.smithi088.stderr:2024-06-03T09:06:21.746+0000 7fae6bfff640 10 monclient(hunting): _negotiate 2024-06-03T09:06:21.746 INFO:teuthology.orchestra.run.smithi088.stderr:2024-06-03T09:06:21.746+0000 7fae6bfff640 10 monclient(hunting): _init_auth method 2 2024-06-03T09:06:21.746 INFO:teuthology.orchestra.run.smithi088.stderr:2024-06-03T09:06:21.746+0000 7fae6bfff640 10 monclient(hunting): _init_auth creating new auth 2024-06-03T09:06:21.746 INFO:teuthology.orchestra.run.smithi088.stderr:2024-06-03T09:06:21.746+0000 7fae6bfff640 10 monclient(hunting): my global_id is 30169 2024-06-03T09:06:21.746 INFO:teuthology.orchestra.run.smithi088.stderr:2024-06-03T09:06:21.746+0000 7fae6bfff640 1 -- 172.21.15.88:0/3152404876 --> v1:172.21.15.192:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7fae60003520 con 0x7fae740ad840 2024-06-03T09:06:21.746 INFO:teuthology.orchestra.run.smithi088.stderr:2024-06-03T09:06:21.746+0000 7fae6bfff640 1 -- 172.21.15.88:0/3152404876 <== mon.0 v1:172.21.15.88:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 3889585458 0 0) 0x7fae74071be0 con 0x7fae7406fd50 2024-06-03T09:06:21.747 INFO:teuthology.orchestra.run.smithi088.stderr:2024-06-03T09:06:21.746+0000 7fae6bfff640 10 monclient(hunting): _negotiate 2024-06-03T09:06:21.747 INFO:teuthology.orchestra.run.smithi088.stderr:2024-06-03T09:06:21.746+0000 7fae6bfff640 10 monclient(hunting): _init_auth method 2 2024-06-03T09:06:21.747 INFO:teuthology.orchestra.run.smithi088.stderr:2024-06-03T09:06:21.746+0000 7fae6bfff640 10 monclient(hunting): _init_auth creating new auth 2024-06-03T09:06:21.747 INFO:teuthology.orchestra.run.smithi088.stderr:2024-06-03T09:06:21.746+0000 7fae6bfff640 10 monclient(hunting): my global_id is 30588 2024-06-03T09:06:21.747 INFO:teuthology.orchestra.run.smithi088.stderr:2024-06-03T09:06:21.746+0000 7fae6bfff640 1 -- 172.21.15.88:0/3152404876 --> v1:172.21.15.88:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7fae74071ec0 con 0x7fae7406fd50 2024-06-03T09:06:21.747 INFO:teuthology.orchestra.run.smithi088.stderr:2024-06-03T09:06:21.746+0000 7fae6bfff640 1 -- 172.21.15.88:0/3152404876 <== mon.2 v1:172.21.15.88:6790/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 3137498580 0 0) 0x7fae740708e0 con 0x7fae7406f970 2024-06-03T09:06:21.747 INFO:teuthology.orchestra.run.smithi088.stderr:2024-06-03T09:06:21.746+0000 7fae6bfff640 10 monclient(hunting): _negotiate 2024-06-03T09:06:21.747 INFO:teuthology.orchestra.run.smithi088.stderr:2024-06-03T09:06:21.746+0000 7fae6bfff640 10 monclient(hunting): _init_auth method 2 2024-06-03T09:06:21.747 INFO:teuthology.orchestra.run.smithi088.stderr:2024-06-03T09:06:21.746+0000 7fae6bfff640 10 monclient(hunting): _init_auth creating new auth 2024-06-03T09:06:21.747 INFO:teuthology.orchestra.run.smithi088.stderr:2024-06-03T09:06:21.746+0000 7fae6bfff640 10 monclient(hunting): my global_id is 30092 2024-06-03T09:06:21.747 INFO:teuthology.orchestra.run.smithi088.stderr:2024-06-03T09:06:21.746+0000 7fae6bfff640 1 -- 172.21.15.88:0/3152404876 --> v1:172.21.15.88:6790/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7fae74071be0 con 0x7fae7406f970 2024-06-03T09:06:21.749 INFO:teuthology.orchestra.run.smithi088.stderr:2024-06-03T09:06:21.749+0000 7fae6bfff640 1 -- 172.21.15.88:0/3152404876 <== mon.0 v1:172.21.15.88:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 214+0+0 (unknown 3117644616 0 0) 0x7fae74071ec0 con 0x7fae7406fd50 2024-06-03T09:06:21.749 INFO:teuthology.orchestra.run.smithi088.stderr:2024-06-03T09:06:21.749+0000 7fae6bfff640 1 -- 172.21.15.88:0/3152404876 >> v1:172.21.15.88:6790/0 conn(0x7fae7406f970 legacy=0x7fae7413fa70 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-06-03T09:06:21.749 INFO:teuthology.orchestra.run.smithi088.stderr:2024-06-03T09:06:21.749+0000 7fae6bfff640 1 -- 172.21.15.88:0/3152404876 >> v1:172.21.15.192:6789/0 conn(0x7fae740ad840 legacy=0x7fae7413c770 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-06-03T09:06:21.749 INFO:teuthology.orchestra.run.smithi088.stderr:2024-06-03T09:06:21.749+0000 7fae6bfff640 10 monclient: _finish_hunting 0 2024-06-03T09:06:21.749 INFO:teuthology.orchestra.run.smithi088.stderr:2024-06-03T09:06:21.749+0000 7fae6bfff640 1 monclient: found mon.noname-a 2024-06-03T09:06:21.749 INFO:teuthology.orchestra.run.smithi088.stderr:2024-06-03T09:06:21.749+0000 7fae6bfff640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-06-03T09:06:21.749 INFO:teuthology.orchestra.run.smithi088.stderr:2024-06-03T09:06:21.749+0000 7fae6bfff640 10 monclient: _send_mon_message to mon.noname-a at v1:172.21.15.88:6789/0 2024-06-03T09:06:21.749 INFO:teuthology.orchestra.run.smithi088.stderr:2024-06-03T09:06:21.749+0000 7fae6bfff640 1 -- 172.21.15.88:0/3152404876 --> v1:172.21.15.88:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fae7413f550 con 0x7fae7406fd50 2024-06-03T09:06:21.749 INFO:teuthology.orchestra.run.smithi088.stderr:2024-06-03T09:06:21.749+0000 7fae6bfff640 10 monclient: _finish_auth 0 2024-06-03T09:06:21.749 INFO:teuthology.orchestra.run.smithi088.stderr:2024-06-03T09:06:21.749+0000 7fae6bfff640 10 monclient: _check_auth_tickets 2024-06-03T09:06:21.749 INFO:teuthology.orchestra.run.smithi088.stderr:2024-06-03T09:06:21.749+0000 7fae6bfff640 20 monclient: _check_auth_rotating not needed by client.admin 2024-06-03T09:06:21.750 INFO:teuthology.orchestra.run.smithi088.stderr:2024-06-03T09:06:21.749+0000 7fae7b7a9640 5 monclient: authenticate success, global_id 30588 2024-06-03T09:06:21.750 INFO:teuthology.orchestra.run.smithi088.stderr:2024-06-03T09:06:21.749+0000 7fae7b7a9640 20 monclient: get_monmap_and_config waiting for monmap|config 2024-06-03T09:06:21.750 INFO:teuthology.orchestra.run.smithi088.stderr:2024-06-03T09:06:21.749+0000 7fae6bfff640 1 -- 172.21.15.88:0/3152404876 <== mon.0 v1:172.21.15.88:6789/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 2234374065 0 0) 0x7fae6c001970 con 0x7fae7406fd50 2024-06-03T09:06:21.751 INFO:teuthology.orchestra.run.smithi088.stderr:2024-06-03T09:06:21.749+0000 7fae6bfff640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-06-03T09:06:21.751 INFO:teuthology.orchestra.run.smithi088.stderr:2024-06-03T09:06:21.749+0000 7fae6bfff640 10 monclient: got monmap 1 from mon.noname-a (according to old e1) 2024-06-03T09:06:21.751 INFO:teuthology.orchestra.run.smithi088.stderr:2024-06-03T09:06:21.749+0000 7fae6bfff640 10 monclient: dump: 2024-06-03T09:06:21.751 INFO:teuthology.orchestra.run.smithi088.stderr:epoch 1 2024-06-03T09:06:21.751 INFO:teuthology.orchestra.run.smithi088.stderr:fsid ec3c3fdd-3ac6-4d7d-9412-683130ed8b04 2024-06-03T09:06:21.751 INFO:teuthology.orchestra.run.smithi088.stderr:last_changed 2024-06-03T08:34:31.885125+0000 2024-06-03T09:06:21.751 INFO:teuthology.orchestra.run.smithi088.stderr:created 2024-06-03T08:34:31.885125+0000 2024-06-03T09:06:21.751 INFO:teuthology.orchestra.run.smithi088.stderr:min_mon_release 18 (reef) 2024-06-03T09:06:21.751 INFO:teuthology.orchestra.run.smithi088.stderr:election_strategy: 3 2024-06-03T09:06:21.751 INFO:teuthology.orchestra.run.smithi088.stderr:0: v1:172.21.15.88:6789/0 mon.a 2024-06-03T09:06:21.751 INFO:teuthology.orchestra.run.smithi088.stderr:1: v1:172.21.15.192:6789/0 mon.b 2024-06-03T09:06:21.751 INFO:teuthology.orchestra.run.smithi088.stderr:2: v1:172.21.15.88:6790/0 mon.c 2024-06-03T09:06:21.751 INFO:teuthology.orchestra.run.smithi088.stderr: 2024-06-03T09:06:21.751 INFO:teuthology.orchestra.run.smithi088.stderr:2024-06-03T09:06:21.749+0000 7fae7b7a9640 20 monclient: get_monmap_and_config waiting for monmap|config 2024-06-03T09:06:21.752 INFO:teuthology.orchestra.run.smithi088.stderr:2024-06-03T09:06:21.751+0000 7fae6bfff640 1 -- 172.21.15.88:0/3152404876 <== mon.0 v1:172.21.15.88:6789/0 4 ==== config(0 keys) v1 ==== 4+0+0 (unknown 0 0 0) 0x7fae6c001bc0 con 0x7fae7406fd50 2024-06-03T09:06:21.752 INFO:teuthology.orchestra.run.smithi088.stderr:2024-06-03T09:06:21.751+0000 7fae6bfff640 10 monclient: handle_config config(0 keys) v1 2024-06-03T09:06:21.752 INFO:teuthology.orchestra.run.smithi088.stderr:2024-06-03T09:06:21.751+0000 7fae6bfff640 1 -- 172.21.15.88:0/3152404876 <== mon.0 v1:172.21.15.88:6789/0 5 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 2234374065 0 0) 0x7fae6c001f90 con 0x7fae7406fd50 2024-06-03T09:06:21.752 INFO:teuthology.orchestra.run.smithi088.stderr:2024-06-03T09:06:21.751+0000 7fae7b7a9640 10 monclient: get_monmap_and_config success 2024-06-03T09:06:21.752 INFO:teuthology.orchestra.run.smithi088.stderr:2024-06-03T09:06:21.751+0000 7fae7b7a9640 4 set_mon_vals no callback set 2024-06-03T09:06:21.752 INFO:teuthology.orchestra.run.smithi088.stderr:2024-06-03T09:06:21.751+0000 7fae6bfff640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-06-03T09:06:21.752 INFO:teuthology.orchestra.run.smithi088.stderr:2024-06-03T09:06:21.751+0000 7fae6bfff640 10 monclient: got monmap 1 from mon.a (according to old e1) 2024-06-03T09:06:21.752 INFO:teuthology.orchestra.run.smithi088.stderr:2024-06-03T09:06:21.751+0000 7fae6bfff640 10 monclient: dump: 2024-06-03T09:06:21.752 INFO:teuthology.orchestra.run.smithi088.stderr:epoch 1 2024-06-03T09:06:21.752 INFO:teuthology.orchestra.run.smithi088.stderr:fsid ec3c3fdd-3ac6-4d7d-9412-683130ed8b04 2024-06-03T09:06:21.752 INFO:teuthology.orchestra.run.smithi088.stderr:last_changed 2024-06-03T08:34:31.885125+0000 2024-06-03T09:06:21.752 INFO:teuthology.orchestra.run.smithi088.stderr:created 2024-06-03T08:34:31.885125+0000 2024-06-03T09:06:21.752 INFO:teuthology.orchestra.run.smithi088.stderr:min_mon_release 18 (reef) 2024-06-03T09:06:21.752 INFO:teuthology.orchestra.run.smithi088.stderr:election_strategy: 3 2024-06-03T09:06:21.753 INFO:teuthology.orchestra.run.smithi088.stderr:0: v1:172.21.15.88:6789/0 mon.a 2024-06-03T09:06:21.753 INFO:teuthology.orchestra.run.smithi088.stderr:1: v1:172.21.15.192:6789/0 mon.b 2024-06-03T09:06:21.753 INFO:teuthology.orchestra.run.smithi088.stderr:2: v1:172.21.15.88:6790/0 mon.c 2024-06-03T09:06:21.753 INFO:teuthology.orchestra.run.smithi088.stderr: 2024-06-03T09:06:21.756 INFO:teuthology.orchestra.run.smithi088.stderr:2024-06-03T09:06:21.751+0000 7fae7b7a9640 10 monclient: shutdown 2024-06-03T09:06:21.756 INFO:teuthology.orchestra.run.smithi088.stderr:2024-06-03T09:06:21.751+0000 7fae7b7a9640 20 monclient: shutdown discarding 0 pending message(s) 2024-06-03T09:06:21.756 INFO:teuthology.orchestra.run.smithi088.stderr:2024-06-03T09:06:21.751+0000 7fae7b7a9640 1 -- 172.21.15.88:0/3152404876 >> v1:172.21.15.88:6789/0 conn(0x7fae7406fd50 legacy=0x7fae7414abf0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-06-03T09:06:21.756 INFO:teuthology.orchestra.run.smithi088.stderr:2024-06-03T09:06:21.751+0000 7fae7b7a9640 1 -- 172.21.15.88:0/3152404876 shutdown_connections 2024-06-03T09:06:21.756 INFO:teuthology.orchestra.run.smithi088.stderr:2024-06-03T09:06:21.751+0000 7fae7b7a9640 1 -- 172.21.15.88:0/3152404876 >> 172.21.15.88:0/3152404876 conn(0x7fae740b2e40 msgr2=0x7fae740b3250 unknown :-1 s=STATE_NONE l=0).mark_down 2024-06-03T09:06:21.756 INFO:teuthology.orchestra.run.smithi088.stderr:2024-06-03T09:06:21.751+0000 7fae7b7a9640 1 -- 172.21.15.88:0/3152404876 shutdown_connections 2024-06-03T09:06:21.756 INFO:teuthology.orchestra.run.smithi088.stderr:2024-06-03T09:06:21.751+0000 7fae7b7a9640 1 -- 172.21.15.88:0/3152404876 wait complete. 2024-06-03T09:06:21.756 INFO:teuthology.orchestra.run.smithi088.stderr:2024-06-03T09:06:21.752+0000 7fae7b7a9640 10 monclient: build_initial_monmap 2024-06-03T09:06:21.756 INFO:teuthology.orchestra.run.smithi088.stderr:2024-06-03T09:06:21.752+0000 7fae7b7a9640 10 monclient: monmap: 2024-06-03T09:06:21.756 INFO:teuthology.orchestra.run.smithi088.stderr:epoch 0 2024-06-03T09:06:21.757 INFO:teuthology.orchestra.run.smithi088.stderr:fsid 00000000-0000-0000-0000-000000000000 2024-06-03T09:06:21.757 INFO:teuthology.orchestra.run.smithi088.stderr:last_changed 0.000000 2024-06-03T09:06:21.757 INFO:teuthology.orchestra.run.smithi088.stderr:created 0.000000 2024-06-03T09:06:21.757 INFO:teuthology.orchestra.run.smithi088.stderr:min_mon_release 0 (unknown) 2024-06-03T09:06:21.757 INFO:teuthology.orchestra.run.smithi088.stderr:election_strategy: 1 2024-06-03T09:06:21.757 INFO:teuthology.orchestra.run.smithi088.stderr:0: v1:172.21.15.88:6789/0 mon.noname-a 2024-06-03T09:06:21.757 INFO:teuthology.orchestra.run.smithi088.stderr:1: v1:172.21.15.192:6789/0 mon.noname-b 2024-06-03T09:06:21.757 INFO:teuthology.orchestra.run.smithi088.stderr:2: v1:172.21.15.88:6790/0 mon.noname-c 2024-06-03T09:06:21.757 INFO:teuthology.orchestra.run.smithi088.stderr: 2024-06-03T09:06:21.757 INFO:teuthology.orchestra.run.smithi088.stderr:2024-06-03T09:06:21.752+0000 7fae7b7a9640 1 librados: starting msgr at 2024-06-03T09:06:21.757 INFO:teuthology.orchestra.run.smithi088.stderr:2024-06-03T09:06:21.752+0000 7fae7b7a9640 1 librados: starting objecter 2024-06-03T09:06:21.757 INFO:teuthology.orchestra.run.smithi088.stderr:2024-06-03T09:06:21.752+0000 7fae7b7a9640 1 Processor -- start 2024-06-03T09:06:21.757 INFO:teuthology.orchestra.run.smithi088.stderr:2024-06-03T09:06:21.752+0000 7fae7b7a9640 1 -- start start 2024-06-03T09:06:21.758 INFO:teuthology.orchestra.run.smithi088.stderr:2024-06-03T09:06:21.752+0000 7fae7b7a9640 1 librados: setting wanted keys 2024-06-03T09:06:21.758 INFO:teuthology.orchestra.run.smithi088.stderr:2024-06-03T09:06:21.752+0000 7fae7b7a9640 1 librados: calling monclient init 2024-06-03T09:06:21.758 INFO:teuthology.orchestra.run.smithi088.stderr:2024-06-03T09:06:21.752+0000 7fae7b7a9640 10 monclient: init 2024-06-03T09:06:21.758 INFO:teuthology.orchestra.run.smithi088.stderr:2024-06-03T09:06:21.753+0000 7fae7b7a9640 10 monclient: _reopen_session rank -1 2024-06-03T09:06:21.758 INFO:teuthology.orchestra.run.smithi088.stderr:2024-06-03T09:06:21.753+0000 7fae7b7a9640 10 monclient: _add_conns ranks=[1,2,0] 2024-06-03T09:06:21.758 INFO:teuthology.orchestra.run.smithi088.stderr:2024-06-03T09:06:21.753+0000 7fae7b7a9640 10 monclient(hunting): picked mon.noname-b con 0x7fae7406f970 addr v1:172.21.15.192:6789/0 2024-06-03T09:06:21.758 INFO:teuthology.orchestra.run.smithi088.stderr:2024-06-03T09:06:21.753+0000 7fae7b7a9640 10 monclient(hunting): picked mon.noname-c con 0x7fae7406fd50 addr v1:172.21.15.88:6790/0 2024-06-03T09:06:21.758 INFO:teuthology.orchestra.run.smithi088.stderr:2024-06-03T09:06:21.753+0000 7fae7b7a9640 10 monclient(hunting): picked mon.noname-a con 0x7fae740ad840 addr v1:172.21.15.88:6789/0 2024-06-03T09:06:21.758 INFO:teuthology.orchestra.run.smithi088.stderr:2024-06-03T09:06:21.753+0000 7fae7b7a9640 1 -- --> v1:172.21.15.88:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fae7413d4e0 con 0x7fae740ad840 2024-06-03T09:06:21.758 INFO:teuthology.orchestra.run.smithi088.stderr:2024-06-03T09:06:21.753+0000 7fae7b7a9640 1 -- --> v1:172.21.15.192:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fae741d9fe0 con 0x7fae7406f970 2024-06-03T09:06:21.758 INFO:teuthology.orchestra.run.smithi088.stderr:2024-06-03T09:06:21.753+0000 7fae7b7a9640 1 -- --> v1:172.21.15.88:6790/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fae7412f2a0 con 0x7fae7406fd50 2024-06-03T09:06:21.758 INFO:teuthology.orchestra.run.smithi088.stderr:2024-06-03T09:06:21.753+0000 7fae7b7a9640 10 monclient(hunting): _renew_subs 2024-06-03T09:06:21.758 INFO:teuthology.orchestra.run.smithi088.stderr:2024-06-03T09:06:21.753+0000 7fae7b7a9640 10 monclient(hunting): authenticate will time out at 3037.563369s 2024-06-03T09:06:21.758 INFO:teuthology.orchestra.run.smithi088.stderr:2024-06-03T09:06:21.753+0000 7fae79d46640 1 --1- >> v1:172.21.15.88:6789/0 conn(0x7fae740ad840 0x7fae741e1c50 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=1).handle_server_banner_and_identify peer v1:172.21.15.88:6789/0 says I am v1:172.21.15.88:43944/0 (socket says 172.21.15.88:43944) 2024-06-03T09:06:21.759 INFO:teuthology.orchestra.run.smithi088.stderr:2024-06-03T09:06:21.753+0000 7fae79d46640 1 -- 172.21.15.88:0/1615676086 learned_addr learned my addr 172.21.15.88:0/1615676086 (peer_addr_for_me v1:172.21.15.88:0/0) 2024-06-03T09:06:21.759 INFO:teuthology.orchestra.run.smithi088.stderr:2024-06-03T09:06:21.754+0000 7fae69ffb640 10 client.?.objecter ms_handle_connect 0x7fae740ad840 2024-06-03T09:06:21.759 INFO:teuthology.orchestra.run.smithi088.stderr:2024-06-03T09:06:21.754+0000 7fae69ffb640 10 client.?.objecter resend_mon_ops 2024-06-03T09:06:21.759 INFO:teuthology.orchestra.run.smithi088.stderr:2024-06-03T09:06:21.754+0000 7fae69ffb640 10 client.?.objecter ms_handle_connect 0x7fae7406fd50 2024-06-03T09:06:21.759 INFO:teuthology.orchestra.run.smithi088.stderr:2024-06-03T09:06:21.754+0000 7fae69ffb640 10 client.?.objecter resend_mon_ops 2024-06-03T09:06:21.759 INFO:teuthology.orchestra.run.smithi088.stderr:2024-06-03T09:06:21.754+0000 7fae69ffb640 10 client.?.objecter ms_handle_connect 0x7fae7406f970 2024-06-03T09:06:21.759 INFO:teuthology.orchestra.run.smithi088.stderr:2024-06-03T09:06:21.754+0000 7fae69ffb640 10 client.?.objecter resend_mon_ops 2024-06-03T09:06:21.759 INFO:teuthology.orchestra.run.smithi088.stderr:2024-06-03T09:06:21.755+0000 7fae69ffb640 1 -- 172.21.15.88:0/1615676086 <== mon.0 v1:172.21.15.88:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 3752038481 0 0) 0x7fae7413d4e0 con 0x7fae740ad840 2024-06-03T09:06:21.759 INFO:teuthology.orchestra.run.smithi088.stderr:2024-06-03T09:06:21.755+0000 7fae69ffb640 10 monclient(hunting): _negotiate 2024-06-03T09:06:21.759 INFO:teuthology.orchestra.run.smithi088.stderr:2024-06-03T09:06:21.755+0000 7fae69ffb640 10 monclient(hunting): _init_auth method 2 2024-06-03T09:06:21.759 INFO:teuthology.orchestra.run.smithi088.stderr:2024-06-03T09:06:21.755+0000 7fae69ffb640 10 monclient(hunting): _init_auth creating new auth 2024-06-03T09:06:21.759 INFO:teuthology.orchestra.run.smithi088.stderr:2024-06-03T09:06:21.755+0000 7fae69ffb640 10 monclient(hunting): my global_id is 30591 2024-06-03T09:06:21.759 INFO:teuthology.orchestra.run.smithi088.stderr:2024-06-03T09:06:21.755+0000 7fae69ffb640 1 -- 172.21.15.88:0/1615676086 --> v1:172.21.15.88:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7fae58004eb0 con 0x7fae740ad840 2024-06-03T09:06:21.759 INFO:teuthology.orchestra.run.smithi088.stderr:2024-06-03T09:06:21.755+0000 7fae69ffb640 1 -- 172.21.15.88:0/1615676086 <== mon.1 v1:172.21.15.192:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 2495772995 0 0) 0x7fae741d9fe0 con 0x7fae7406f970 2024-06-03T09:06:21.760 INFO:teuthology.orchestra.run.smithi088.stderr:2024-06-03T09:06:21.755+0000 7fae69ffb640 10 monclient(hunting): _negotiate 2024-06-03T09:06:21.760 INFO:teuthology.orchestra.run.smithi088.stderr:2024-06-03T09:06:21.755+0000 7fae69ffb640 10 monclient(hunting): _init_auth method 2 2024-06-03T09:06:21.760 INFO:teuthology.orchestra.run.smithi088.stderr:2024-06-03T09:06:21.756+0000 7fae69ffb640 10 monclient(hunting): _init_auth creating new auth 2024-06-03T09:06:21.760 INFO:teuthology.orchestra.run.smithi088.stderr:2024-06-03T09:06:21.756+0000 7fae69ffb640 10 monclient(hunting): my global_id is 30172 2024-06-03T09:06:21.760 INFO:teuthology.orchestra.run.smithi088.stderr:2024-06-03T09:06:21.756+0000 7fae69ffb640 1 -- 172.21.15.88:0/1615676086 --> v1:172.21.15.192:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7fae7413d4e0 con 0x7fae7406f970 2024-06-03T09:06:21.760 INFO:teuthology.orchestra.run.smithi088.stderr:2024-06-03T09:06:21.756+0000 7fae69ffb640 1 -- 172.21.15.88:0/1615676086 <== mon.2 v1:172.21.15.88:6790/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 4160733263 0 0) 0x7fae7412f2a0 con 0x7fae7406fd50 2024-06-03T09:06:21.760 INFO:teuthology.orchestra.run.smithi088.stderr:2024-06-03T09:06:21.756+0000 7fae69ffb640 10 monclient(hunting): _negotiate 2024-06-03T09:06:21.760 INFO:teuthology.orchestra.run.smithi088.stderr:2024-06-03T09:06:21.756+0000 7fae69ffb640 10 monclient(hunting): _init_auth method 2 2024-06-03T09:06:21.760 INFO:teuthology.orchestra.run.smithi088.stderr:2024-06-03T09:06:21.756+0000 7fae69ffb640 10 monclient(hunting): _init_auth creating new auth 2024-06-03T09:06:21.761 INFO:teuthology.orchestra.run.smithi088.stderr:2024-06-03T09:06:21.756+0000 7fae69ffb640 10 monclient(hunting): my global_id is 30095 2024-06-03T09:06:21.761 INFO:teuthology.orchestra.run.smithi088.stderr:2024-06-03T09:06:21.756+0000 7fae69ffb640 1 -- 172.21.15.88:0/1615676086 --> v1:172.21.15.88:6790/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7fae741d9fe0 con 0x7fae7406fd50 2024-06-03T09:06:21.762 INFO:teuthology.orchestra.run.smithi088.stderr:2024-06-03T09:06:21.760+0000 7fae69ffb640 1 -- 172.21.15.88:0/1615676086 <== mon.1 v1:172.21.15.192:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 764+0+0 (unknown 3059013833 0 0) 0x7fae7413d4e0 con 0x7fae7406f970 2024-06-03T09:06:21.762 INFO:teuthology.orchestra.run.smithi088.stderr:2024-06-03T09:06:21.760+0000 7fae69ffb640 1 -- 172.21.15.88:0/1615676086 --> v1:172.21.15.192:6789/0 -- auth(proto 2 165 bytes epoch 0) v1 -- 0x7fae7412f2a0 con 0x7fae7406f970 2024-06-03T09:06:21.762 INFO:teuthology.orchestra.run.smithi088.stderr:2024-06-03T09:06:21.760+0000 7fae69ffb640 1 -- 172.21.15.88:0/1615676086 <== mon.1 v1:172.21.15.192:6789/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 2234374065 0 0) 0x7fae700017f0 con 0x7fae7406f970 2024-06-03T09:06:21.762 INFO:teuthology.orchestra.run.smithi088.stderr:2024-06-03T09:06:21.760+0000 7fae69ffb640 10 monclient(hunting): handle_monmap mon_map magic: 0 v1 2024-06-03T09:06:21.762 INFO:teuthology.orchestra.run.smithi088.stderr:2024-06-03T09:06:21.760+0000 7fae69ffb640 10 monclient(hunting): got monmap 1 from mon.noname-b (according to old e1) 2024-06-03T09:06:21.762 INFO:teuthology.orchestra.run.smithi088.stderr:2024-06-03T09:06:21.760+0000 7fae69ffb640 10 monclient(hunting): dump: 2024-06-03T09:06:21.762 INFO:teuthology.orchestra.run.smithi088.stderr:epoch 1 2024-06-03T09:06:21.763 INFO:teuthology.orchestra.run.smithi088.stderr:fsid ec3c3fdd-3ac6-4d7d-9412-683130ed8b04 2024-06-03T09:06:21.763 INFO:teuthology.orchestra.run.smithi088.stderr:last_changed 2024-06-03T08:34:31.885125+0000 2024-06-03T09:06:21.763 INFO:teuthology.orchestra.run.smithi088.stderr:created 2024-06-03T08:34:31.885125+0000 2024-06-03T09:06:21.763 INFO:teuthology.orchestra.run.smithi088.stderr:min_mon_release 18 (reef) 2024-06-03T09:06:21.763 INFO:teuthology.orchestra.run.smithi088.stderr:election_strategy: 3 2024-06-03T09:06:21.763 INFO:teuthology.orchestra.run.smithi088.stderr:0: v1:172.21.15.88:6789/0 mon.a 2024-06-03T09:06:21.763 INFO:teuthology.orchestra.run.smithi088.stderr:1: v1:172.21.15.192:6789/0 mon.b 2024-06-03T09:06:21.763 INFO:teuthology.orchestra.run.smithi088.stderr:2: v1:172.21.15.88:6790/0 mon.c 2024-06-03T09:06:21.763 INFO:teuthology.orchestra.run.smithi088.stderr: 2024-06-03T09:06:21.763 INFO:teuthology.orchestra.run.smithi088.stderr:2024-06-03T09:06:21.760+0000 7fae69ffb640 10 monclient(hunting): _finish_auth 0 2024-06-03T09:06:21.763 INFO:teuthology.orchestra.run.smithi088.stderr:2024-06-03T09:06:21.760+0000 7fae69ffb640 1 -- 172.21.15.88:0/1615676086 <== mon.0 v1:172.21.15.88:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 764+0+0 (unknown 321230310 0 0) 0x7fae58004eb0 con 0x7fae740ad840 2024-06-03T09:06:21.763 INFO:teuthology.orchestra.run.smithi088.stderr:2024-06-03T09:06:21.760+0000 7fae69ffb640 1 -- 172.21.15.88:0/1615676086 --> v1:172.21.15.88:6789/0 -- auth(proto 2 165 bytes epoch 0) v1 -- 0x7fae7413d4e0 con 0x7fae740ad840 2024-06-03T09:06:21.763 INFO:teuthology.orchestra.run.smithi088.stderr:2024-06-03T09:06:21.760+0000 7fae69ffb640 1 -- 172.21.15.88:0/1615676086 <== mon.0 v1:172.21.15.88:6789/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 2234374065 0 0) 0x7fae6c001ed0 con 0x7fae740ad840 2024-06-03T09:06:21.764 INFO:teuthology.orchestra.run.smithi088.stderr:2024-06-03T09:06:21.760+0000 7fae69ffb640 10 monclient(hunting): handle_monmap mon_map magic: 0 v1 2024-06-03T09:06:21.764 INFO:teuthology.orchestra.run.smithi088.stderr:2024-06-03T09:06:21.760+0000 7fae69ffb640 10 monclient(hunting): got monmap 1 from mon.a (according to old e1) 2024-06-03T09:06:21.764 INFO:teuthology.orchestra.run.smithi088.stderr:2024-06-03T09:06:21.760+0000 7fae69ffb640 10 monclient(hunting): dump: 2024-06-03T09:06:21.764 INFO:teuthology.orchestra.run.smithi088.stderr:epoch 1 2024-06-03T09:06:21.764 INFO:teuthology.orchestra.run.smithi088.stderr:fsid ec3c3fdd-3ac6-4d7d-9412-683130ed8b04 2024-06-03T09:06:21.764 INFO:teuthology.orchestra.run.smithi088.stderr:last_changed 2024-06-03T08:34:31.885125+0000 2024-06-03T09:06:21.764 INFO:teuthology.orchestra.run.smithi088.stderr:created 2024-06-03T08:34:31.885125+0000 2024-06-03T09:06:21.764 INFO:teuthology.orchestra.run.smithi088.stderr:min_mon_release 18 (reef) 2024-06-03T09:06:21.764 INFO:teuthology.orchestra.run.smithi088.stderr:election_strategy: 3 2024-06-03T09:06:21.764 INFO:teuthology.orchestra.run.smithi088.stderr:0: v1:172.21.15.88:6789/0 mon.a 2024-06-03T09:06:21.764 INFO:teuthology.orchestra.run.smithi088.stderr:1: v1:172.21.15.192:6789/0 mon.b 2024-06-03T09:06:21.764 INFO:teuthology.orchestra.run.smithi088.stderr:2: v1:172.21.15.88:6790/0 mon.c 2024-06-03T09:06:21.764 INFO:teuthology.orchestra.run.smithi088.stderr: 2024-06-03T09:06:21.764 INFO:teuthology.orchestra.run.smithi088.stderr:2024-06-03T09:06:21.760+0000 7fae69ffb640 1 -- 172.21.15.88:0/1615676086 <== mon.2 v1:172.21.15.88:6790/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 764+0+0 (unknown 1505420458 0 0) 0x7fae741d9fe0 con 0x7fae7406fd50 2024-06-03T09:06:21.765 INFO:teuthology.orchestra.run.smithi088.stderr:2024-06-03T09:06:21.760+0000 7fae69ffb640 1 -- 172.21.15.88:0/1615676086 --> v1:172.21.15.88:6790/0 -- auth(proto 2 165 bytes epoch 0) v1 -- 0x7fae58004eb0 con 0x7fae7406fd50 2024-06-03T09:06:21.765 INFO:teuthology.orchestra.run.smithi088.stderr:2024-06-03T09:06:21.760+0000 7fae69ffb640 1 -- 172.21.15.88:0/1615676086 <== mon.2 v1:172.21.15.88:6790/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 2234374065 0 0) 0x7fae64001810 con 0x7fae7406fd50 2024-06-03T09:06:21.765 INFO:teuthology.orchestra.run.smithi088.stderr:2024-06-03T09:06:21.760+0000 7fae69ffb640 10 monclient(hunting): handle_monmap mon_map magic: 0 v1 2024-06-03T09:06:21.765 INFO:teuthology.orchestra.run.smithi088.stderr:2024-06-03T09:06:21.760+0000 7fae69ffb640 10 monclient(hunting): got monmap 1 from mon.c (according to old e1) 2024-06-03T09:06:21.765 INFO:teuthology.orchestra.run.smithi088.stderr:2024-06-03T09:06:21.760+0000 7fae69ffb640 10 monclient(hunting): dump: 2024-06-03T09:06:21.765 INFO:teuthology.orchestra.run.smithi088.stderr:epoch 1 2024-06-03T09:06:21.765 INFO:teuthology.orchestra.run.smithi088.stderr:fsid ec3c3fdd-3ac6-4d7d-9412-683130ed8b04 2024-06-03T09:06:21.765 INFO:teuthology.orchestra.run.smithi088.stderr:last_changed 2024-06-03T08:34:31.885125+0000 2024-06-03T09:06:21.765 INFO:teuthology.orchestra.run.smithi088.stderr:created 2024-06-03T08:34:31.885125+0000 2024-06-03T09:06:21.765 INFO:teuthology.orchestra.run.smithi088.stderr:min_mon_release 18 (reef) 2024-06-03T09:06:21.765 INFO:teuthology.orchestra.run.smithi088.stderr:election_strategy: 3 2024-06-03T09:06:21.765 INFO:teuthology.orchestra.run.smithi088.stderr:0: v1:172.21.15.88:6789/0 mon.a 2024-06-03T09:06:21.765 INFO:teuthology.orchestra.run.smithi088.stderr:1: v1:172.21.15.192:6789/0 mon.b 2024-06-03T09:06:21.766 INFO:teuthology.orchestra.run.smithi088.stderr:2: v1:172.21.15.88:6790/0 mon.c 2024-06-03T09:06:21.766 INFO:teuthology.orchestra.run.smithi088.stderr: 2024-06-03T09:06:21.767 INFO:teuthology.orchestra.run.smithi088.stderr:2024-06-03T09:06:21.764+0000 7fae69ffb640 1 -- 172.21.15.88:0/1615676086 <== mon.1 v1:172.21.15.192:6789/0 4 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 580+0+0 (unknown 4210522493 0 0) 0x7fae7412f2a0 con 0x7fae7406f970 2024-06-03T09:06:21.767 INFO:teuthology.orchestra.run.smithi088.stderr:2024-06-03T09:06:21.764+0000 7fae69ffb640 1 -- 172.21.15.88:0/1615676086 >> v1:172.21.15.88:6790/0 conn(0x7fae7406fd50 legacy=0x7fae7412c440 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-06-03T09:06:21.767 INFO:teuthology.orchestra.run.smithi088.stderr:2024-06-03T09:06:21.764+0000 7fae69ffb640 1 -- 172.21.15.88:0/1615676086 >> v1:172.21.15.88:6789/0 conn(0x7fae740ad840 legacy=0x7fae741e1c50 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-06-03T09:06:21.767 INFO:teuthology.orchestra.run.smithi088.stderr:2024-06-03T09:06:21.764+0000 7fae69ffb640 10 monclient: _finish_hunting 0 2024-06-03T09:06:21.767 INFO:teuthology.orchestra.run.smithi088.stderr:2024-06-03T09:06:21.764+0000 7fae69ffb640 1 monclient: found mon.b 2024-06-03T09:06:21.767 INFO:teuthology.orchestra.run.smithi088.stderr:2024-06-03T09:06:21.764+0000 7fae69ffb640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-06-03T09:06:21.768 INFO:teuthology.orchestra.run.smithi088.stderr:2024-06-03T09:06:21.764+0000 7fae69ffb640 10 monclient: _send_mon_message to mon.b at v1:172.21.15.192:6789/0 2024-06-03T09:06:21.768 INFO:teuthology.orchestra.run.smithi088.stderr:2024-06-03T09:06:21.764+0000 7fae69ffb640 1 -- 172.21.15.88:0/1615676086 --> v1:172.21.15.192:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fae7413d1b0 con 0x7fae7406f970 2024-06-03T09:06:21.768 INFO:teuthology.orchestra.run.smithi088.stderr:2024-06-03T09:06:21.764+0000 7fae69ffb640 10 monclient: _finish_auth 0 2024-06-03T09:06:21.768 INFO:teuthology.orchestra.run.smithi088.stderr:2024-06-03T09:06:21.764+0000 7fae69ffb640 10 monclient: _check_auth_tickets 2024-06-03T09:06:21.768 INFO:teuthology.orchestra.run.smithi088.stderr:2024-06-03T09:06:21.764+0000 7fae69ffb640 20 monclient: _check_auth_rotating not needed by client.admin 2024-06-03T09:06:21.768 INFO:teuthology.orchestra.run.smithi088.stderr:2024-06-03T09:06:21.764+0000 7fae7b7a9640 5 monclient: authenticate success, global_id 30172 2024-06-03T09:06:21.768 INFO:teuthology.orchestra.run.smithi088.stderr:2024-06-03T09:06:21.764+0000 7fae7b7a9640 10 monclient: _renew_subs 2024-06-03T09:06:21.768 INFO:teuthology.orchestra.run.smithi088.stderr:2024-06-03T09:06:21.764+0000 7fae7b7a9640 10 monclient: _send_mon_message to mon.b at v1:172.21.15.192:6789/0 2024-06-03T09:06:21.768 INFO:teuthology.orchestra.run.smithi088.stderr:2024-06-03T09:06:21.764+0000 7fae7b7a9640 1 -- 172.21.15.88:0/1615676086 --> v1:172.21.15.192:6789/0 -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fae7412f520 con 0x7fae7406f970 2024-06-03T09:06:21.768 INFO:teuthology.orchestra.run.smithi088.stderr:2024-06-03T09:06:21.764+0000 7fae7b7a9640 10 client.30172.objecter _maybe_request_map subscribing (onetime) to next osd map 2024-06-03T09:06:21.768 INFO:teuthology.orchestra.run.smithi088.stderr:2024-06-03T09:06:21.764+0000 7fae7b7a9640 10 monclient: _renew_subs 2024-06-03T09:06:21.768 INFO:teuthology.orchestra.run.smithi088.stderr:2024-06-03T09:06:21.764+0000 7fae7b7a9640 10 monclient: _send_mon_message to mon.b at v1:172.21.15.192:6789/0 2024-06-03T09:06:21.768 INFO:teuthology.orchestra.run.smithi088.stderr:2024-06-03T09:06:21.764+0000 7fae7b7a9640 1 -- 172.21.15.88:0/1615676086 --> v1:172.21.15.192:6789/0 -- mon_subscribe({osdmap=0}) v3 -- 0x7fae7412ce50 con 0x7fae7406f970 2024-06-03T09:06:21.769 INFO:teuthology.orchestra.run.smithi088.stderr:2024-06-03T09:06:21.764+0000 7fae7b7a9640 1 librados: init done 2024-06-03T09:06:21.769 INFO:teuthology.orchestra.run.smithi088.stderr:2024-06-03T09:06:21.765+0000 7fae7b7a9640 10 start_mon_command cmd=[{"prefix": "get_command_descriptions"}] 2024-06-03T09:06:21.769 INFO:teuthology.orchestra.run.smithi088.stderr:2024-06-03T09:06:21.765+0000 7fae7b7a9640 10 monclient: _send_command 1 [{"prefix": "get_command_descriptions"}] 2024-06-03T09:06:21.769 INFO:teuthology.orchestra.run.smithi088.stderr:2024-06-03T09:06:21.765+0000 7fae7b7a9640 10 monclient: _send_mon_message to mon.b at v1:172.21.15.192:6789/0 2024-06-03T09:06:21.769 INFO:teuthology.orchestra.run.smithi088.stderr:2024-06-03T09:06:21.765+0000 7fae7b7a9640 1 -- 172.21.15.88:0/1615676086 --> v1:172.21.15.192:6789/0 -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fae7404ea60 con 0x7fae7406f970 2024-06-03T09:06:21.979 INFO:teuthology.orchestra.run.smithi088.stderr:2024-06-03T09:06:21.979+0000 7fae69ffb640 1 -- 172.21.15.88:0/1615676086 <== mon.1 v1:172.21.15.192:6789/0 5 ==== config(0 keys) v1 ==== 4+0+0 (unknown 0 0 0) 0x7fae70003610 con 0x7fae7406f970 2024-06-03T09:06:21.979 INFO:teuthology.orchestra.run.smithi088.stderr:2024-06-03T09:06:21.979+0000 7fae69ffb640 10 monclient: handle_config config(0 keys) v1 2024-06-03T09:06:21.979 INFO:teuthology.orchestra.run.smithi088.stderr:2024-06-03T09:06:21.979+0000 7fae69ffb640 1 -- 172.21.15.88:0/1615676086 <== mon.1 v1:172.21.15.192:6789/0 6 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 2234374065 0 0) 0x7fae70001d80 con 0x7fae7406f970 2024-06-03T09:06:21.979 INFO:teuthology.orchestra.run.smithi088.stderr:2024-06-03T09:06:21.979+0000 7fae69ffb640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-06-03T09:06:21.979 INFO:teuthology.orchestra.run.smithi088.stderr:2024-06-03T09:06:21.979+0000 7fae69ffb640 10 monclient: got monmap 1 from mon.b (according to old e1) 2024-06-03T09:06:21.979 INFO:teuthology.orchestra.run.smithi088.stderr:2024-06-03T09:06:21.979+0000 7fae69ffb640 10 monclient: dump: 2024-06-03T09:06:21.979 INFO:teuthology.orchestra.run.smithi088.stderr:epoch 1 2024-06-03T09:06:21.979 INFO:teuthology.orchestra.run.smithi088.stderr:fsid ec3c3fdd-3ac6-4d7d-9412-683130ed8b04 2024-06-03T09:06:21.979 INFO:teuthology.orchestra.run.smithi088.stderr:last_changed 2024-06-03T08:34:31.885125+0000 2024-06-03T09:06:21.980 INFO:teuthology.orchestra.run.smithi088.stderr:created 2024-06-03T08:34:31.885125+0000 2024-06-03T09:06:21.980 INFO:teuthology.orchestra.run.smithi088.stderr:min_mon_release 18 (reef) 2024-06-03T09:06:21.980 INFO:teuthology.orchestra.run.smithi088.stderr:election_strategy: 3 2024-06-03T09:06:21.980 INFO:teuthology.orchestra.run.smithi088.stderr:0: v1:172.21.15.88:6789/0 mon.a 2024-06-03T09:06:21.980 INFO:teuthology.orchestra.run.smithi088.stderr:1: v1:172.21.15.192:6789/0 mon.b 2024-06-03T09:06:21.980 INFO:teuthology.orchestra.run.smithi088.stderr:2: v1:172.21.15.88:6790/0 mon.c 2024-06-03T09:06:21.980 INFO:teuthology.orchestra.run.smithi088.stderr: 2024-06-03T09:06:21.980 INFO:teuthology.orchestra.run.smithi088.stderr:2024-06-03T09:06:21.979+0000 7fae6affd640 4 set_mon_vals no callback set 2024-06-03T09:06:22.026 INFO:teuthology.orchestra.run.smithi088.stderr:2024-06-03T09:06:22.026+0000 7fae69ffb640 1 -- 172.21.15.88:0/1615676086 <== mon.1 v1:172.21.15.192:6789/0 7 ==== mgrmap(e 5) v1 ==== 89110+0+0 (unknown 873533973 0 0) 0x7fae70019540 con 0x7fae7406f970 2024-06-03T09:06:22.027 INFO:teuthology.orchestra.run.smithi088.stderr:2024-06-03T09:06:22.027+0000 7fae78d44640 10 monclient: get_auth_request con 0x7fae5806d440 auth_method 0 2024-06-03T09:06:22.027 INFO:teuthology.orchestra.run.smithi088.stderr:2024-06-03T09:06:22.027+0000 7fae69ffb640 10 client.30172.objecter ms_handle_connect 0x7fae5806d440 2024-06-03T09:06:22.030 INFO:teuthology.orchestra.run.smithi088.stderr:2024-06-03T09:06:22.030+0000 7fae69ffb640 1 -- 172.21.15.88:0/1615676086 <== mon.1 v1:172.21.15.192:6789/0 8 ==== osd_map(1010..1010 src has 622..1010) v4 ==== 5229+0+0 (unknown 1105883993 0 0) 0x7fae70052df0 con 0x7fae7406f970 2024-06-03T09:06:22.030 INFO:teuthology.orchestra.run.smithi088.stderr:2024-06-03T09:06:22.030+0000 7fae69ffb640 10 client.30172.objecter ms_dispatch 0x7fae74000b90 osd_map(1010..1010 src has 622..1010) v4 2024-06-03T09:06:22.030 INFO:teuthology.orchestra.run.smithi088.stderr:2024-06-03T09:06:22.030+0000 7fae69ffb640 3 client.30172.objecter handle_osd_map got epochs [1010,1010] > 0 2024-06-03T09:06:22.030 INFO:teuthology.orchestra.run.smithi088.stderr:2024-06-03T09:06:22.030+0000 7fae69ffb640 3 client.30172.objecter handle_osd_map decoding full epoch 1010 2024-06-03T09:06:22.030 INFO:teuthology.orchestra.run.smithi088.stderr:2024-06-03T09:06:22.030+0000 7fae69ffb640 20 client.30172.objecter dump_active .. 0 homeless 2024-06-03T09:06:22.031 INFO:teuthology.orchestra.run.smithi088.stderr:2024-06-03T09:06:22.031+0000 7fae69ffb640 1 -- 172.21.15.88:0/1615676086 <== mon.1 v1:172.21.15.192:6789/0 9 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+182498 (unknown 1092875540 0 3016760700) 0x7fae7004e9a0 con 0x7fae7406f970 2024-06-03T09:06:22.031 INFO:teuthology.orchestra.run.smithi088.stderr:2024-06-03T09:06:22.031+0000 7fae69ffb640 10 monclient: handle_mon_command_ack 1 [{"prefix": "get_command_descriptions"}] 2024-06-03T09:06:22.031 INFO:teuthology.orchestra.run.smithi088.stderr:2024-06-03T09:06:22.031+0000 7fae69ffb640 10 monclient: _finish_command 1 = system:0 2024-06-03T09:06:22.272 INFO:teuthology.orchestra.run.smithi088.stderr:2024-06-03T09:06:22.272+0000 7fae7b7a9640 10 start_mon_command cmd=[{"prefix": "status", "format": "json"}] 2024-06-03T09:06:22.272 INFO:teuthology.orchestra.run.smithi088.stderr:2024-06-03T09:06:22.272+0000 7fae7b7a9640 10 monclient: _send_command 2 [{"prefix": "status", "format": "json"}] 2024-06-03T09:06:22.272 INFO:teuthology.orchestra.run.smithi088.stderr:2024-06-03T09:06:22.272+0000 7fae7b7a9640 10 monclient: _send_mon_message to mon.b at v1:172.21.15.192:6789/0 2024-06-03T09:06:22.272 INFO:teuthology.orchestra.run.smithi088.stderr:2024-06-03T09:06:22.272+0000 7fae7b7a9640 1 -- 172.21.15.88:0/1615676086 --> v1:172.21.15.192:6789/0 -- mon_command({"prefix": "status", "format": "json"} v 0) v1 -- 0x7fae7406a0d0 con 0x7fae7406f970 2024-06-03T09:06:22.278 INFO:teuthology.orchestra.run.smithi088.stderr:2024-06-03T09:06:22.278+0000 7fae69ffb640 1 -- 172.21.15.88:0/1615676086 <== mon.1 v1:172.21.15.192:6789/0 10 ==== mon_command_ack([{"prefix": "status", "format": "json"}]=0 v0) v1 ==== 72+0+1431 (unknown 3226119948 0 465274944) 0x7fae700508d0 con 0x7fae7406f970 2024-06-03T09:06:22.278 INFO:teuthology.orchestra.run.smithi088.stderr:2024-06-03T09:06:22.278+0000 7fae69ffb640 10 monclient: handle_mon_command_ack 2 [{"prefix": "status", "format": "json"}] 2024-06-03T09:06:22.278 INFO:teuthology.orchestra.run.smithi088.stderr:2024-06-03T09:06:22.278+0000 7fae69ffb640 10 monclient: _finish_command 2 = system:0 2024-06-03T09:06:22.278 INFO:teuthology.orchestra.run.smithi088.stdout: 2024-06-03T09:06:22.278 INFO:teuthology.orchestra.run.smithi088.stdout:{"fsid":"ec3c3fdd-3ac6-4d7d-9412-683130ed8b04","health":{"status":"HEALTH_WARN","checks":{"OSD_DOWN":{"severity":"HEALTH_WARN","summary":{"message":"1 osds down","count":1},"muted":false},"PG_DEGRADED":{"severity":"HEALTH_WARN","summary":{"message":"Degraded data redundancy: 1/6 objects degraded (16.667%), 1 pg degraded","count":1},"muted":false},"POOL_APP_NOT_ENABLED":{"severity":"HEALTH_WARN","summary":{"message":"1 pool(s) do not have an application enabled","count":1},"muted":false}},"mutes":[]},"election_epoch":10,"quorum":[0,1,2],"quorum_names":["a","b","c"],"quorum_age":1855,"monmap":{"epoch":1,"min_mon_release_name":"reef","num_mons":3},"osdmap":{"epoch":1010,"num_osds":8,"num_up_osds":7,"osd_up_since":1717405562,"num_in_osds":8,"osd_in_since":1717403732,"num_remapped_pgs":0},"pgmap":{"pgs_by_state":[{"state_name":"active+clean","count":27},{"state_name":"active+undersized","count":13},{"state_name":"active+undersized+degraded","count":1}],"num_pgs":41,"num_pools":3,"num_objects":3,"data_bytes":459285,"bytes_used":506408960,"bytes_avail":772587704320,"bytes_total":773094113280,"degraded_objects":1,"degraded_total":6,"degraded_ratio":0.16666666666666663},"fsmap":{"epoch":1,"by_rank":[],"up:standby":0},"mgrmap":{"available":true,"num_standbys":1,"modules":["iostat","nfs","restful"],"services":{}},"servicemap":{"epoch":23,"modified":"2024-06-03T08:53:48.910215+0000","services":{}},"progress_events":{}} 2024-06-03T09:06:22.280 INFO:teuthology.orchestra.run.smithi088.stderr:2024-06-03T09:06:22.279+0000 7fae7b7a9640 10 librados: watch_flush enter 2024-06-03T09:06:22.280 INFO:teuthology.orchestra.run.smithi088.stderr:2024-06-03T09:06:22.279+0000 7fae7b7a9640 10 librados: watch_flush exit 2024-06-03T09:06:22.280 INFO:teuthology.orchestra.run.smithi088.stderr:2024-06-03T09:06:22.279+0000 7fae7b7a9640 20 client.30172.objecter shutdown clearing up homeless session... 2024-06-03T09:06:22.280 INFO:teuthology.orchestra.run.smithi088.stderr:2024-06-03T09:06:22.279+0000 7fae7b7a9640 10 client.30172.objecter successfully canceled tick 2024-06-03T09:06:22.280 INFO:teuthology.orchestra.run.smithi088.stderr:2024-06-03T09:06:22.279+0000 7fae7b7a9640 1 -- 172.21.15.88:0/1615676086 >> v1:172.21.15.192:6800/34229 conn(0x7fae5806d440 legacy=0x7fae5806f900 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-06-03T09:06:22.280 INFO:teuthology.orchestra.run.smithi088.stderr:2024-06-03T09:06:22.279+0000 7fae7b7a9640 10 monclient: shutdown 2024-06-03T09:06:22.280 INFO:teuthology.orchestra.run.smithi088.stderr:2024-06-03T09:06:22.279+0000 7fae7b7a9640 20 monclient: shutdown discarding 0 pending message(s) 2024-06-03T09:06:22.280 INFO:teuthology.orchestra.run.smithi088.stderr:2024-06-03T09:06:22.279+0000 7fae7b7a9640 1 -- 172.21.15.88:0/1615676086 >> v1:172.21.15.192:6789/0 conn(0x7fae7406f970 legacy=0x7fae741d98d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-06-03T09:06:22.280 INFO:teuthology.orchestra.run.smithi088.stderr:2024-06-03T09:06:22.279+0000 7fae7b7a9640 1 -- 172.21.15.88:0/1615676086 shutdown_connections 2024-06-03T09:06:22.280 INFO:teuthology.orchestra.run.smithi088.stderr:2024-06-03T09:06:22.279+0000 7fae7b7a9640 1 -- 172.21.15.88:0/1615676086 >> 172.21.15.88:0/1615676086 conn(0x7fae740b2e40 msgr2=0x7fae740737f0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-06-03T09:06:22.280 INFO:teuthology.orchestra.run.smithi088.stderr:2024-06-03T09:06:22.279+0000 7fae7b7a9640 1 -- 172.21.15.88:0/1615676086 shutdown_connections 2024-06-03T09:06:22.280 INFO:teuthology.orchestra.run.smithi088.stderr:2024-06-03T09:06:22.279+0000 7fae7b7a9640 1 -- 172.21.15.88:0/1615676086 wait complete. 2024-06-03T09:06:22.281 INFO:teuthology.orchestra.run.smithi088.stderr:2024-06-03T09:06:22.279+0000 7fae7b7a9640 1 librados: shutdown 2024-06-03T09:06:22.291 INFO:tasks.ceph.ceph_manager.ceph:no progress seen, keeping timeout for now 2024-06-03T09:06:22.291 DEBUG:teuthology.orchestra.run.smithi088:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph pg dump --format=json 2024-06-03T09:06:22.390 INFO:teuthology.orchestra.run.smithi088.stderr:2024-06-03T09:06:22.390+0000 7fad74d27640 10 monclient: get_monmap_and_config 2024-06-03T09:06:22.390 INFO:teuthology.orchestra.run.smithi088.stderr:2024-06-03T09:06:22.390+0000 7fad74d27640 10 monclient: build_initial_monmap 2024-06-03T09:06:22.390 INFO:teuthology.orchestra.run.smithi088.stderr:2024-06-03T09:06:22.390+0000 7fad74d27640 10 monclient: monmap: 2024-06-03T09:06:22.390 INFO:teuthology.orchestra.run.smithi088.stderr:epoch 0 2024-06-03T09:06:22.390 INFO:teuthology.orchestra.run.smithi088.stderr:fsid ec3c3fdd-3ac6-4d7d-9412-683130ed8b04 2024-06-03T09:06:22.390 INFO:teuthology.orchestra.run.smithi088.stderr:last_changed 2024-06-03T09:06:22.391267+0000 2024-06-03T09:06:22.390 INFO:teuthology.orchestra.run.smithi088.stderr:created 2024-06-03T09:06:22.391267+0000 2024-06-03T09:06:22.390 INFO:teuthology.orchestra.run.smithi088.stderr:min_mon_release 0 (unknown) 2024-06-03T09:06:22.391 INFO:teuthology.orchestra.run.smithi088.stderr:election_strategy: 3 2024-06-03T09:06:22.391 INFO:teuthology.orchestra.run.smithi088.stderr:0: v1:172.21.15.88:6789/0 mon.noname-a 2024-06-03T09:06:22.391 INFO:teuthology.orchestra.run.smithi088.stderr:1: v1:172.21.15.192:6789/0 mon.noname-c 2024-06-03T09:06:22.391 INFO:teuthology.orchestra.run.smithi088.stderr:2: v1:172.21.15.88:6790/0 mon.noname-b 2024-06-03T09:06:22.391 INFO:teuthology.orchestra.run.smithi088.stderr: 2024-06-03T09:06:22.392 INFO:teuthology.orchestra.run.smithi088.stderr:2024-06-03T09:06:22.390+0000 7fad74d27640 1 Processor -- start 2024-06-03T09:06:22.392 INFO:teuthology.orchestra.run.smithi088.stderr:2024-06-03T09:06:22.390+0000 7fad74d27640 1 -- start start 2024-06-03T09:06:22.392 INFO:teuthology.orchestra.run.smithi088.stderr:2024-06-03T09:06:22.390+0000 7fad74d27640 10 monclient: init 2024-06-03T09:06:22.392 INFO:teuthology.orchestra.run.smithi088.stderr:2024-06-03T09:06:22.391+0000 7fad74d27640 10 monclient: _reopen_session rank -1 2024-06-03T09:06:22.392 INFO:teuthology.orchestra.run.smithi088.stderr:2024-06-03T09:06:22.391+0000 7fad74d27640 10 monclient: _add_conns ranks=[0,1,2] 2024-06-03T09:06:22.392 INFO:teuthology.orchestra.run.smithi088.stderr:2024-06-03T09:06:22.391+0000 7fad74d27640 10 monclient(hunting): picked mon.noname-a con 0x7fad700a8960 addr v1:172.21.15.88:6789/0 2024-06-03T09:06:22.393 INFO:teuthology.orchestra.run.smithi088.stderr:2024-06-03T09:06:22.391+0000 7fad74d27640 10 monclient(hunting): picked mon.noname-c con 0x7fad701450f0 addr v1:172.21.15.192:6789/0 2024-06-03T09:06:22.393 INFO:teuthology.orchestra.run.smithi088.stderr:2024-06-03T09:06:22.391+0000 7fad74d27640 10 monclient(hunting): picked mon.noname-b con 0x7fad7013eec0 addr v1:172.21.15.88:6790/0 2024-06-03T09:06:22.393 INFO:teuthology.orchestra.run.smithi088.stderr:2024-06-03T09:06:22.391+0000 7fad74d27640 1 -- --> v1:172.21.15.88:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fad70144c60 con 0x7fad700a8960 2024-06-03T09:06:22.393 INFO:teuthology.orchestra.run.smithi088.stderr:2024-06-03T09:06:22.391+0000 7fad74d27640 1 -- --> v1:172.21.15.192:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fad700b3930 con 0x7fad701450f0 2024-06-03T09:06:22.393 INFO:teuthology.orchestra.run.smithi088.stderr:2024-06-03T09:06:22.391+0000 7fad74d27640 1 -- --> v1:172.21.15.88:6790/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fad700b1ad0 con 0x7fad7013eec0 2024-06-03T09:06:22.393 INFO:teuthology.orchestra.run.smithi088.stderr:2024-06-03T09:06:22.391+0000 7fad74d27640 10 monclient(hunting): _renew_subs 2024-06-03T09:06:22.393 INFO:teuthology.orchestra.run.smithi088.stderr:2024-06-03T09:06:22.391+0000 7fad74d27640 10 monclient(hunting): authenticate will time out at 3038.201601s 2024-06-03T09:06:22.393 INFO:teuthology.orchestra.run.smithi088.stderr:2024-06-03T09:06:22.392+0000 7fad6e59c640 1 --1- >> v1:172.21.15.88:6789/0 conn(0x7fad700a8960 0x7fad700a8d40 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.88:6789/0 says I am v1:172.21.15.88:43948/0 (socket says 172.21.15.88:43948) 2024-06-03T09:06:22.393 INFO:teuthology.orchestra.run.smithi088.stderr:2024-06-03T09:06:22.392+0000 7fad6e59c640 1 -- 172.21.15.88:0/363291984 learned_addr learned my addr 172.21.15.88:0/363291984 (peer_addr_for_me v1:172.21.15.88:0/0) 2024-06-03T09:06:22.394 INFO:teuthology.orchestra.run.smithi088.stderr:2024-06-03T09:06:22.394+0000 7fad6d59a640 1 -- 172.21.15.88:0/363291984 <== mon.0 v1:172.21.15.88:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 3087220455 0 0) 0x7fad70144c60 con 0x7fad700a8960 2024-06-03T09:06:22.394 INFO:teuthology.orchestra.run.smithi088.stderr:2024-06-03T09:06:22.394+0000 7fad6d59a640 10 monclient(hunting): _negotiate 2024-06-03T09:06:22.394 INFO:teuthology.orchestra.run.smithi088.stderr:2024-06-03T09:06:22.394+0000 7fad6d59a640 10 monclient(hunting): _init_auth method 2 2024-06-03T09:06:22.394 INFO:teuthology.orchestra.run.smithi088.stderr:2024-06-03T09:06:22.394+0000 7fad6d59a640 10 monclient(hunting): _init_auth creating new auth 2024-06-03T09:06:22.395 INFO:teuthology.orchestra.run.smithi088.stderr:2024-06-03T09:06:22.394+0000 7fad6d59a640 10 monclient(hunting): my global_id is 30594 2024-06-03T09:06:22.395 INFO:teuthology.orchestra.run.smithi088.stderr:2024-06-03T09:06:22.394+0000 7fad6d59a640 1 -- 172.21.15.88:0/363291984 --> v1:172.21.15.88:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7fad5c003520 con 0x7fad700a8960 2024-06-03T09:06:22.395 INFO:teuthology.orchestra.run.smithi088.stderr:2024-06-03T09:06:22.394+0000 7fad6d59a640 1 -- 172.21.15.88:0/363291984 <== mon.1 v1:172.21.15.192:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 1032248317 0 0) 0x7fad700b3930 con 0x7fad701450f0 2024-06-03T09:06:22.395 INFO:teuthology.orchestra.run.smithi088.stderr:2024-06-03T09:06:22.394+0000 7fad6d59a640 10 monclient(hunting): _negotiate 2024-06-03T09:06:22.395 INFO:teuthology.orchestra.run.smithi088.stderr:2024-06-03T09:06:22.394+0000 7fad6d59a640 10 monclient(hunting): _init_auth method 2 2024-06-03T09:06:22.395 INFO:teuthology.orchestra.run.smithi088.stderr:2024-06-03T09:06:22.394+0000 7fad6d59a640 10 monclient(hunting): _init_auth creating new auth 2024-06-03T09:06:22.395 INFO:teuthology.orchestra.run.smithi088.stderr:2024-06-03T09:06:22.394+0000 7fad6d59a640 10 monclient(hunting): my global_id is 30175 2024-06-03T09:06:22.395 INFO:teuthology.orchestra.run.smithi088.stderr:2024-06-03T09:06:22.394+0000 7fad6d59a640 1 -- 172.21.15.88:0/363291984 --> v1:172.21.15.192:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7fad70144c60 con 0x7fad701450f0 2024-06-03T09:06:22.395 INFO:teuthology.orchestra.run.smithi088.stderr:2024-06-03T09:06:22.394+0000 7fad6d59a640 1 -- 172.21.15.88:0/363291984 <== mon.2 v1:172.21.15.88:6790/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 2134544215 0 0) 0x7fad700b1ad0 con 0x7fad7013eec0 2024-06-03T09:06:22.395 INFO:teuthology.orchestra.run.smithi088.stderr:2024-06-03T09:06:22.394+0000 7fad6d59a640 10 monclient(hunting): _negotiate 2024-06-03T09:06:22.395 INFO:teuthology.orchestra.run.smithi088.stderr:2024-06-03T09:06:22.394+0000 7fad6d59a640 10 monclient(hunting): _init_auth method 2 2024-06-03T09:06:22.395 INFO:teuthology.orchestra.run.smithi088.stderr:2024-06-03T09:06:22.394+0000 7fad6d59a640 10 monclient(hunting): _init_auth creating new auth 2024-06-03T09:06:22.396 INFO:teuthology.orchestra.run.smithi088.stderr:2024-06-03T09:06:22.394+0000 7fad6d59a640 10 monclient(hunting): my global_id is 30098 2024-06-03T09:06:22.396 INFO:teuthology.orchestra.run.smithi088.stderr:2024-06-03T09:06:22.394+0000 7fad6d59a640 1 -- 172.21.15.88:0/363291984 --> v1:172.21.15.88:6790/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7fad700b3930 con 0x7fad7013eec0 2024-06-03T09:06:22.397 INFO:teuthology.orchestra.run.smithi088.stderr:2024-06-03T09:06:22.397+0000 7fad6d59a640 1 -- 172.21.15.88:0/363291984 <== mon.2 v1:172.21.15.88:6790/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 214+0+0 (unknown 457363416 0 0) 0x7fad700b3930 con 0x7fad7013eec0 2024-06-03T09:06:22.398 INFO:teuthology.orchestra.run.smithi088.stderr:2024-06-03T09:06:22.397+0000 7fad6d59a640 1 -- 172.21.15.88:0/363291984 >> v1:172.21.15.192:6789/0 conn(0x7fad701450f0 legacy=0x7fad7013e7b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-06-03T09:06:22.398 INFO:teuthology.orchestra.run.smithi088.stderr:2024-06-03T09:06:22.397+0000 7fad6d59a640 1 -- 172.21.15.88:0/363291984 >> v1:172.21.15.88:6789/0 conn(0x7fad700a8960 legacy=0x7fad700a8d40 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-06-03T09:06:22.398 INFO:teuthology.orchestra.run.smithi088.stderr:2024-06-03T09:06:22.397+0000 7fad6d59a640 10 monclient: _finish_hunting 0 2024-06-03T09:06:22.398 INFO:teuthology.orchestra.run.smithi088.stderr:2024-06-03T09:06:22.397+0000 7fad6d59a640 1 monclient: found mon.noname-b 2024-06-03T09:06:22.398 INFO:teuthology.orchestra.run.smithi088.stderr:2024-06-03T09:06:22.397+0000 7fad6d59a640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-06-03T09:06:22.398 INFO:teuthology.orchestra.run.smithi088.stderr:2024-06-03T09:06:22.397+0000 7fad6d59a640 10 monclient: _send_mon_message to mon.noname-b at v1:172.21.15.88:6790/0 2024-06-03T09:06:22.398 INFO:teuthology.orchestra.run.smithi088.stderr:2024-06-03T09:06:22.397+0000 7fad6d59a640 1 -- 172.21.15.88:0/363291984 --> v1:172.21.15.88:6790/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fad700b1c60 con 0x7fad7013eec0 2024-06-03T09:06:22.398 INFO:teuthology.orchestra.run.smithi088.stderr:2024-06-03T09:06:22.397+0000 7fad6d59a640 10 monclient: _finish_auth 0 2024-06-03T09:06:22.398 INFO:teuthology.orchestra.run.smithi088.stderr:2024-06-03T09:06:22.397+0000 7fad6d59a640 10 monclient: _check_auth_tickets 2024-06-03T09:06:22.398 INFO:teuthology.orchestra.run.smithi088.stderr:2024-06-03T09:06:22.397+0000 7fad6d59a640 20 monclient: _check_auth_rotating not needed by client.admin 2024-06-03T09:06:22.399 INFO:teuthology.orchestra.run.smithi088.stderr:2024-06-03T09:06:22.397+0000 7fad74d27640 5 monclient: authenticate success, global_id 30098 2024-06-03T09:06:22.399 INFO:teuthology.orchestra.run.smithi088.stderr:2024-06-03T09:06:22.397+0000 7fad74d27640 20 monclient: get_monmap_and_config waiting for monmap|config 2024-06-03T09:06:22.399 INFO:teuthology.orchestra.run.smithi088.stderr:2024-06-03T09:06:22.397+0000 7fad6d59a640 1 -- 172.21.15.88:0/363291984 <== mon.2 v1:172.21.15.88:6790/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 2234374065 0 0) 0x7fad680018b0 con 0x7fad7013eec0 2024-06-03T09:06:22.399 INFO:teuthology.orchestra.run.smithi088.stderr:2024-06-03T09:06:22.397+0000 7fad6d59a640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-06-03T09:06:22.399 INFO:teuthology.orchestra.run.smithi088.stderr:2024-06-03T09:06:22.397+0000 7fad6d59a640 10 monclient: got monmap 1 from mon.noname-b (according to old e1) 2024-06-03T09:06:22.399 INFO:teuthology.orchestra.run.smithi088.stderr:2024-06-03T09:06:22.397+0000 7fad6d59a640 10 monclient: dump: 2024-06-03T09:06:22.399 INFO:teuthology.orchestra.run.smithi088.stderr:epoch 1 2024-06-03T09:06:22.399 INFO:teuthology.orchestra.run.smithi088.stderr:fsid ec3c3fdd-3ac6-4d7d-9412-683130ed8b04 2024-06-03T09:06:22.399 INFO:teuthology.orchestra.run.smithi088.stderr:last_changed 2024-06-03T08:34:31.885125+0000 2024-06-03T09:06:22.399 INFO:teuthology.orchestra.run.smithi088.stderr:created 2024-06-03T08:34:31.885125+0000 2024-06-03T09:06:22.399 INFO:teuthology.orchestra.run.smithi088.stderr:min_mon_release 18 (reef) 2024-06-03T09:06:22.399 INFO:teuthology.orchestra.run.smithi088.stderr:election_strategy: 3 2024-06-03T09:06:22.399 INFO:teuthology.orchestra.run.smithi088.stderr:0: v1:172.21.15.88:6789/0 mon.a 2024-06-03T09:06:22.400 INFO:teuthology.orchestra.run.smithi088.stderr:1: v1:172.21.15.192:6789/0 mon.b 2024-06-03T09:06:22.400 INFO:teuthology.orchestra.run.smithi088.stderr:2: v1:172.21.15.88:6790/0 mon.c 2024-06-03T09:06:22.400 INFO:teuthology.orchestra.run.smithi088.stderr: 2024-06-03T09:06:22.400 INFO:teuthology.orchestra.run.smithi088.stderr:2024-06-03T09:06:22.397+0000 7fad74d27640 20 monclient: get_monmap_and_config waiting for monmap|config 2024-06-03T09:06:22.403 INFO:teuthology.orchestra.run.smithi088.stderr:2024-06-03T09:06:22.399+0000 7fad6d59a640 1 -- 172.21.15.88:0/363291984 <== mon.2 v1:172.21.15.88:6790/0 4 ==== config(0 keys) v1 ==== 4+0+0 (unknown 0 0 0) 0x7fad68001a50 con 0x7fad7013eec0 2024-06-03T09:06:22.403 INFO:teuthology.orchestra.run.smithi088.stderr:2024-06-03T09:06:22.399+0000 7fad6d59a640 10 monclient: handle_config config(0 keys) v1 2024-06-03T09:06:22.403 INFO:teuthology.orchestra.run.smithi088.stderr:2024-06-03T09:06:22.399+0000 7fad6d59a640 1 -- 172.21.15.88:0/363291984 <== mon.2 v1:172.21.15.88:6790/0 5 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 2234374065 0 0) 0x7fad68001e20 con 0x7fad7013eec0 2024-06-03T09:06:22.403 INFO:teuthology.orchestra.run.smithi088.stderr:2024-06-03T09:06:22.399+0000 7fad74d27640 10 monclient: get_monmap_and_config success 2024-06-03T09:06:22.403 INFO:teuthology.orchestra.run.smithi088.stderr:2024-06-03T09:06:22.399+0000 7fad74d27640 4 set_mon_vals no callback set 2024-06-03T09:06:22.403 INFO:teuthology.orchestra.run.smithi088.stderr:2024-06-03T09:06:22.399+0000 7fad6d59a640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-06-03T09:06:22.403 INFO:teuthology.orchestra.run.smithi088.stderr:2024-06-03T09:06:22.399+0000 7fad6d59a640 10 monclient: got monmap 1 from mon.c (according to old e1) 2024-06-03T09:06:22.403 INFO:teuthology.orchestra.run.smithi088.stderr:2024-06-03T09:06:22.399+0000 7fad6d59a640 10 monclient: dump: 2024-06-03T09:06:22.403 INFO:teuthology.orchestra.run.smithi088.stderr:epoch 1 2024-06-03T09:06:22.403 INFO:teuthology.orchestra.run.smithi088.stderr:fsid ec3c3fdd-3ac6-4d7d-9412-683130ed8b04 2024-06-03T09:06:22.403 INFO:teuthology.orchestra.run.smithi088.stderr:last_changed 2024-06-03T08:34:31.885125+0000 2024-06-03T09:06:22.403 INFO:teuthology.orchestra.run.smithi088.stderr:created 2024-06-03T08:34:31.885125+0000 2024-06-03T09:06:22.404 INFO:teuthology.orchestra.run.smithi088.stderr:min_mon_release 18 (reef) 2024-06-03T09:06:22.404 INFO:teuthology.orchestra.run.smithi088.stderr:election_strategy: 3 2024-06-03T09:06:22.404 INFO:teuthology.orchestra.run.smithi088.stderr:0: v1:172.21.15.88:6789/0 mon.a 2024-06-03T09:06:22.404 INFO:teuthology.orchestra.run.smithi088.stderr:1: v1:172.21.15.192:6789/0 mon.b 2024-06-03T09:06:22.404 INFO:teuthology.orchestra.run.smithi088.stderr:2: v1:172.21.15.88:6790/0 mon.c 2024-06-03T09:06:22.404 INFO:teuthology.orchestra.run.smithi088.stderr: 2024-06-03T09:06:22.404 INFO:teuthology.orchestra.run.smithi088.stderr:2024-06-03T09:06:22.400+0000 7fad74d27640 10 monclient: shutdown 2024-06-03T09:06:22.404 INFO:teuthology.orchestra.run.smithi088.stderr:2024-06-03T09:06:22.400+0000 7fad74d27640 20 monclient: shutdown discarding 0 pending message(s) 2024-06-03T09:06:22.404 INFO:teuthology.orchestra.run.smithi088.stderr:2024-06-03T09:06:22.400+0000 7fad74d27640 1 -- 172.21.15.88:0/363291984 >> v1:172.21.15.88:6790/0 conn(0x7fad7013eec0 legacy=0x7fad7014ce60 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-06-03T09:06:22.404 INFO:teuthology.orchestra.run.smithi088.stderr:2024-06-03T09:06:22.400+0000 7fad74d27640 1 -- 172.21.15.88:0/363291984 shutdown_connections 2024-06-03T09:06:22.404 INFO:teuthology.orchestra.run.smithi088.stderr:2024-06-03T09:06:22.400+0000 7fad74d27640 1 -- 172.21.15.88:0/363291984 >> 172.21.15.88:0/363291984 conn(0x7fad700b70d0 msgr2=0x7fad700b74e0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-06-03T09:06:22.404 INFO:teuthology.orchestra.run.smithi088.stderr:2024-06-03T09:06:22.400+0000 7fad74d27640 1 -- 172.21.15.88:0/363291984 shutdown_connections 2024-06-03T09:06:22.404 INFO:teuthology.orchestra.run.smithi088.stderr:2024-06-03T09:06:22.400+0000 7fad74d27640 1 -- 172.21.15.88:0/363291984 wait complete. 2024-06-03T09:06:22.405 INFO:teuthology.orchestra.run.smithi088.stderr:2024-06-03T09:06:22.401+0000 7fad74d27640 10 monclient: build_initial_monmap 2024-06-03T09:06:22.405 INFO:teuthology.orchestra.run.smithi088.stderr:2024-06-03T09:06:22.401+0000 7fad74d27640 10 monclient: monmap: 2024-06-03T09:06:22.405 INFO:teuthology.orchestra.run.smithi088.stderr:epoch 0 2024-06-03T09:06:22.405 INFO:teuthology.orchestra.run.smithi088.stderr:fsid 00000000-0000-0000-0000-000000000000 2024-06-03T09:06:22.405 INFO:teuthology.orchestra.run.smithi088.stderr:last_changed 0.000000 2024-06-03T09:06:22.405 INFO:teuthology.orchestra.run.smithi088.stderr:created 0.000000 2024-06-03T09:06:22.405 INFO:teuthology.orchestra.run.smithi088.stderr:min_mon_release 0 (unknown) 2024-06-03T09:06:22.405 INFO:teuthology.orchestra.run.smithi088.stderr:election_strategy: 1 2024-06-03T09:06:22.405 INFO:teuthology.orchestra.run.smithi088.stderr:0: v1:172.21.15.88:6789/0 mon.noname-a 2024-06-03T09:06:22.405 INFO:teuthology.orchestra.run.smithi088.stderr:1: v1:172.21.15.192:6789/0 mon.noname-b 2024-06-03T09:06:22.405 INFO:teuthology.orchestra.run.smithi088.stderr:2: v1:172.21.15.88:6790/0 mon.noname-c 2024-06-03T09:06:22.405 INFO:teuthology.orchestra.run.smithi088.stderr: 2024-06-03T09:06:22.405 INFO:teuthology.orchestra.run.smithi088.stderr:2024-06-03T09:06:22.401+0000 7fad74d27640 1 librados: starting msgr at 2024-06-03T09:06:22.405 INFO:teuthology.orchestra.run.smithi088.stderr:2024-06-03T09:06:22.401+0000 7fad74d27640 1 librados: starting objecter 2024-06-03T09:06:22.406 INFO:teuthology.orchestra.run.smithi088.stderr:2024-06-03T09:06:22.401+0000 7fad74d27640 1 Processor -- start 2024-06-03T09:06:22.406 INFO:teuthology.orchestra.run.smithi088.stderr:2024-06-03T09:06:22.401+0000 7fad74d27640 1 -- start start 2024-06-03T09:06:22.406 INFO:teuthology.orchestra.run.smithi088.stderr:2024-06-03T09:06:22.401+0000 7fad74d27640 1 librados: setting wanted keys 2024-06-03T09:06:22.406 INFO:teuthology.orchestra.run.smithi088.stderr:2024-06-03T09:06:22.401+0000 7fad74d27640 1 librados: calling monclient init 2024-06-03T09:06:22.406 INFO:teuthology.orchestra.run.smithi088.stderr:2024-06-03T09:06:22.401+0000 7fad74d27640 10 monclient: init 2024-06-03T09:06:22.406 INFO:teuthology.orchestra.run.smithi088.stderr:2024-06-03T09:06:22.402+0000 7fad74d27640 10 monclient: _reopen_session rank -1 2024-06-03T09:06:22.406 INFO:teuthology.orchestra.run.smithi088.stderr:2024-06-03T09:06:22.402+0000 7fad74d27640 10 monclient: _add_conns ranks=[2,0,1] 2024-06-03T09:06:22.406 INFO:teuthology.orchestra.run.smithi088.stderr:2024-06-03T09:06:22.402+0000 7fad74d27640 10 monclient(hunting): picked mon.noname-c con 0x7fad700a8960 addr v1:172.21.15.88:6790/0 2024-06-03T09:06:22.406 INFO:teuthology.orchestra.run.smithi088.stderr:2024-06-03T09:06:22.402+0000 7fad74d27640 10 monclient(hunting): picked mon.noname-a con 0x7fad7013eec0 addr v1:172.21.15.88:6789/0 2024-06-03T09:06:22.406 INFO:teuthology.orchestra.run.smithi088.stderr:2024-06-03T09:06:22.402+0000 7fad74d27640 10 monclient(hunting): picked mon.noname-b con 0x7fad701450f0 addr v1:172.21.15.192:6789/0 2024-06-03T09:06:22.406 INFO:teuthology.orchestra.run.smithi088.stderr:2024-06-03T09:06:22.402+0000 7fad74d27640 1 -- --> v1:172.21.15.88:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fad700acd90 con 0x7fad7013eec0 2024-06-03T09:06:22.406 INFO:teuthology.orchestra.run.smithi088.stderr:2024-06-03T09:06:22.402+0000 7fad74d27640 1 -- --> v1:172.21.15.192:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fad70141640 con 0x7fad701450f0 2024-06-03T09:06:22.406 INFO:teuthology.orchestra.run.smithi088.stderr:2024-06-03T09:06:22.402+0000 7fad74d27640 1 -- --> v1:172.21.15.88:6790/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fad701417d0 con 0x7fad700a8960 2024-06-03T09:06:22.407 INFO:teuthology.orchestra.run.smithi088.stderr:2024-06-03T09:06:22.402+0000 7fad74d27640 10 monclient(hunting): _renew_subs 2024-06-03T09:06:22.407 INFO:teuthology.orchestra.run.smithi088.stderr:2024-06-03T09:06:22.402+0000 7fad74d27640 10 monclient(hunting): authenticate will time out at 3038.212392s 2024-06-03T09:06:22.407 INFO:teuthology.orchestra.run.smithi088.stderr:2024-06-03T09:06:22.402+0000 7fad6dd9b640 1 --1- >> v1:172.21.15.88:6789/0 conn(0x7fad7013eec0 0x7fad701d97d0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=1).handle_server_banner_and_identify peer v1:172.21.15.88:6789/0 says I am v1:172.21.15.88:43962/0 (socket says 172.21.15.88:43962) 2024-06-03T09:06:22.407 INFO:teuthology.orchestra.run.smithi088.stderr:2024-06-03T09:06:22.402+0000 7fad6dd9b640 1 -- 172.21.15.88:0/2287369957 learned_addr learned my addr 172.21.15.88:0/2287369957 (peer_addr_for_me v1:172.21.15.88:0/0) 2024-06-03T09:06:22.407 INFO:teuthology.orchestra.run.smithi088.stderr:2024-06-03T09:06:22.403+0000 7fad66ffd640 10 client.?.objecter ms_handle_connect 0x7fad7013eec0 2024-06-03T09:06:22.407 INFO:teuthology.orchestra.run.smithi088.stderr:2024-06-03T09:06:22.403+0000 7fad66ffd640 10 client.?.objecter resend_mon_ops 2024-06-03T09:06:22.407 INFO:teuthology.orchestra.run.smithi088.stderr:2024-06-03T09:06:22.403+0000 7fad66ffd640 10 client.?.objecter ms_handle_connect 0x7fad700a8960 2024-06-03T09:06:22.407 INFO:teuthology.orchestra.run.smithi088.stderr:2024-06-03T09:06:22.403+0000 7fad66ffd640 10 client.?.objecter resend_mon_ops 2024-06-03T09:06:22.407 INFO:teuthology.orchestra.run.smithi088.stderr:2024-06-03T09:06:22.403+0000 7fad66ffd640 10 client.?.objecter ms_handle_connect 0x7fad701450f0 2024-06-03T09:06:22.407 INFO:teuthology.orchestra.run.smithi088.stderr:2024-06-03T09:06:22.403+0000 7fad66ffd640 10 client.?.objecter resend_mon_ops 2024-06-03T09:06:22.408 INFO:teuthology.orchestra.run.smithi088.stderr:2024-06-03T09:06:22.405+0000 7fad66ffd640 1 -- 172.21.15.88:0/2287369957 <== mon.1 v1:172.21.15.192:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 744573773 0 0) 0x7fad70141640 con 0x7fad701450f0 2024-06-03T09:06:22.408 INFO:teuthology.orchestra.run.smithi088.stderr:2024-06-03T09:06:22.405+0000 7fad66ffd640 10 monclient(hunting): _negotiate 2024-06-03T09:06:22.408 INFO:teuthology.orchestra.run.smithi088.stderr:2024-06-03T09:06:22.405+0000 7fad66ffd640 10 monclient(hunting): _init_auth method 2 2024-06-03T09:06:22.408 INFO:teuthology.orchestra.run.smithi088.stderr:2024-06-03T09:06:22.405+0000 7fad66ffd640 10 monclient(hunting): _init_auth creating new auth 2024-06-03T09:06:22.408 INFO:teuthology.orchestra.run.smithi088.stderr:2024-06-03T09:06:22.405+0000 7fad66ffd640 10 monclient(hunting): my global_id is 30178 2024-06-03T09:06:22.409 INFO:teuthology.orchestra.run.smithi088.stderr:2024-06-03T09:06:22.405+0000 7fad66ffd640 1 -- 172.21.15.88:0/2287369957 --> v1:172.21.15.192:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7fad50004eb0 con 0x7fad701450f0 2024-06-03T09:06:22.409 INFO:teuthology.orchestra.run.smithi088.stderr:2024-06-03T09:06:22.405+0000 7fad66ffd640 1 -- 172.21.15.88:0/2287369957 <== mon.2 v1:172.21.15.88:6790/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 2810619704 0 0) 0x7fad701417d0 con 0x7fad700a8960 2024-06-03T09:06:22.409 INFO:teuthology.orchestra.run.smithi088.stderr:2024-06-03T09:06:22.405+0000 7fad66ffd640 10 monclient(hunting): _negotiate 2024-06-03T09:06:22.409 INFO:teuthology.orchestra.run.smithi088.stderr:2024-06-03T09:06:22.405+0000 7fad66ffd640 10 monclient(hunting): _init_auth method 2 2024-06-03T09:06:22.409 INFO:teuthology.orchestra.run.smithi088.stderr:2024-06-03T09:06:22.405+0000 7fad66ffd640 10 monclient(hunting): _init_auth creating new auth 2024-06-03T09:06:22.409 INFO:teuthology.orchestra.run.smithi088.stderr:2024-06-03T09:06:22.405+0000 7fad66ffd640 10 monclient(hunting): my global_id is 30101 2024-06-03T09:06:22.409 INFO:teuthology.orchestra.run.smithi088.stderr:2024-06-03T09:06:22.405+0000 7fad66ffd640 1 -- 172.21.15.88:0/2287369957 --> v1:172.21.15.88:6790/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7fad70141640 con 0x7fad700a8960 2024-06-03T09:06:22.409 INFO:teuthology.orchestra.run.smithi088.stderr:2024-06-03T09:06:22.405+0000 7fad66ffd640 1 -- 172.21.15.88:0/2287369957 <== mon.0 v1:172.21.15.88:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 955380273 0 0) 0x7fad700acd90 con 0x7fad7013eec0 2024-06-03T09:06:22.409 INFO:teuthology.orchestra.run.smithi088.stderr:2024-06-03T09:06:22.405+0000 7fad66ffd640 10 monclient(hunting): _negotiate 2024-06-03T09:06:22.409 INFO:teuthology.orchestra.run.smithi088.stderr:2024-06-03T09:06:22.405+0000 7fad66ffd640 10 monclient(hunting): _init_auth method 2 2024-06-03T09:06:22.409 INFO:teuthology.orchestra.run.smithi088.stderr:2024-06-03T09:06:22.405+0000 7fad66ffd640 10 monclient(hunting): _init_auth creating new auth 2024-06-03T09:06:22.409 INFO:teuthology.orchestra.run.smithi088.stderr:2024-06-03T09:06:22.405+0000 7fad66ffd640 10 monclient(hunting): my global_id is 30597 2024-06-03T09:06:22.410 INFO:teuthology.orchestra.run.smithi088.stderr:2024-06-03T09:06:22.405+0000 7fad66ffd640 1 -- 172.21.15.88:0/2287369957 --> v1:172.21.15.88:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7fad701417d0 con 0x7fad7013eec0 2024-06-03T09:06:22.410 INFO:teuthology.orchestra.run.smithi088.stderr:2024-06-03T09:06:22.409+0000 7fad66ffd640 1 -- 172.21.15.88:0/2287369957 <== mon.1 v1:172.21.15.192:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 764+0+0 (unknown 1319228477 0 0) 0x7fad50004eb0 con 0x7fad701450f0 2024-06-03T09:06:22.411 INFO:teuthology.orchestra.run.smithi088.stderr:2024-06-03T09:06:22.409+0000 7fad66ffd640 1 -- 172.21.15.88:0/2287369957 --> v1:172.21.15.192:6789/0 -- auth(proto 2 165 bytes epoch 0) v1 -- 0x7fad700acd90 con 0x7fad701450f0 2024-06-03T09:06:22.411 INFO:teuthology.orchestra.run.smithi088.stderr:2024-06-03T09:06:22.409+0000 7fad66ffd640 1 -- 172.21.15.88:0/2287369957 <== mon.1 v1:172.21.15.192:6789/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 2234374065 0 0) 0x7fad68001d60 con 0x7fad701450f0 2024-06-03T09:06:22.411 INFO:teuthology.orchestra.run.smithi088.stderr:2024-06-03T09:06:22.409+0000 7fad66ffd640 10 monclient(hunting): handle_monmap mon_map magic: 0 v1 2024-06-03T09:06:22.411 INFO:teuthology.orchestra.run.smithi088.stderr:2024-06-03T09:06:22.409+0000 7fad66ffd640 10 monclient(hunting): got monmap 1 from mon.noname-b (according to old e1) 2024-06-03T09:06:22.411 INFO:teuthology.orchestra.run.smithi088.stderr:2024-06-03T09:06:22.409+0000 7fad66ffd640 10 monclient(hunting): dump: 2024-06-03T09:06:22.411 INFO:teuthology.orchestra.run.smithi088.stderr:epoch 1 2024-06-03T09:06:22.411 INFO:teuthology.orchestra.run.smithi088.stderr:fsid ec3c3fdd-3ac6-4d7d-9412-683130ed8b04 2024-06-03T09:06:22.411 INFO:teuthology.orchestra.run.smithi088.stderr:last_changed 2024-06-03T08:34:31.885125+0000 2024-06-03T09:06:22.411 INFO:teuthology.orchestra.run.smithi088.stderr:created 2024-06-03T08:34:31.885125+0000 2024-06-03T09:06:22.411 INFO:teuthology.orchestra.run.smithi088.stderr:min_mon_release 18 (reef) 2024-06-03T09:06:22.411 INFO:teuthology.orchestra.run.smithi088.stderr:election_strategy: 3 2024-06-03T09:06:22.411 INFO:teuthology.orchestra.run.smithi088.stderr:0: v1:172.21.15.88:6789/0 mon.a 2024-06-03T09:06:22.411 INFO:teuthology.orchestra.run.smithi088.stderr:1: v1:172.21.15.192:6789/0 mon.b 2024-06-03T09:06:22.412 INFO:teuthology.orchestra.run.smithi088.stderr:2: v1:172.21.15.88:6790/0 mon.c 2024-06-03T09:06:22.412 INFO:teuthology.orchestra.run.smithi088.stderr: 2024-06-03T09:06:22.412 INFO:teuthology.orchestra.run.smithi088.stderr:2024-06-03T09:06:22.409+0000 7fad66ffd640 10 monclient(hunting): _finish_auth 0 2024-06-03T09:06:22.412 INFO:teuthology.orchestra.run.smithi088.stderr:2024-06-03T09:06:22.410+0000 7fad66ffd640 1 -- 172.21.15.88:0/2287369957 <== mon.2 v1:172.21.15.88:6790/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 764+0+0 (unknown 3210435185 0 0) 0x7fad70141640 con 0x7fad700a8960 2024-06-03T09:06:22.412 INFO:teuthology.orchestra.run.smithi088.stderr:2024-06-03T09:06:22.410+0000 7fad66ffd640 1 -- 172.21.15.88:0/2287369957 --> v1:172.21.15.88:6790/0 -- auth(proto 2 165 bytes epoch 0) v1 -- 0x7fad50004eb0 con 0x7fad700a8960 2024-06-03T09:06:22.412 INFO:teuthology.orchestra.run.smithi088.stderr:2024-06-03T09:06:22.410+0000 7fad66ffd640 1 -- 172.21.15.88:0/2287369957 <== mon.2 v1:172.21.15.88:6790/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 2234374065 0 0) 0x7fad580018d0 con 0x7fad700a8960 2024-06-03T09:06:22.412 INFO:teuthology.orchestra.run.smithi088.stderr:2024-06-03T09:06:22.410+0000 7fad66ffd640 10 monclient(hunting): handle_monmap mon_map magic: 0 v1 2024-06-03T09:06:22.412 INFO:teuthology.orchestra.run.smithi088.stderr:2024-06-03T09:06:22.410+0000 7fad66ffd640 10 monclient(hunting): got monmap 1 from mon.c (according to old e1) 2024-06-03T09:06:22.412 INFO:teuthology.orchestra.run.smithi088.stderr:2024-06-03T09:06:22.410+0000 7fad66ffd640 10 monclient(hunting): dump: 2024-06-03T09:06:22.412 INFO:teuthology.orchestra.run.smithi088.stderr:epoch 1 2024-06-03T09:06:22.412 INFO:teuthology.orchestra.run.smithi088.stderr:fsid ec3c3fdd-3ac6-4d7d-9412-683130ed8b04 2024-06-03T09:06:22.412 INFO:teuthology.orchestra.run.smithi088.stderr:last_changed 2024-06-03T08:34:31.885125+0000 2024-06-03T09:06:22.412 INFO:teuthology.orchestra.run.smithi088.stderr:created 2024-06-03T08:34:31.885125+0000 2024-06-03T09:06:22.413 INFO:teuthology.orchestra.run.smithi088.stderr:min_mon_release 18 (reef) 2024-06-03T09:06:22.413 INFO:teuthology.orchestra.run.smithi088.stderr:election_strategy: 3 2024-06-03T09:06:22.413 INFO:teuthology.orchestra.run.smithi088.stderr:0: v1:172.21.15.88:6789/0 mon.a 2024-06-03T09:06:22.413 INFO:teuthology.orchestra.run.smithi088.stderr:1: v1:172.21.15.192:6789/0 mon.b 2024-06-03T09:06:22.413 INFO:teuthology.orchestra.run.smithi088.stderr:2: v1:172.21.15.88:6790/0 mon.c 2024-06-03T09:06:22.413 INFO:teuthology.orchestra.run.smithi088.stderr: 2024-06-03T09:06:22.415 INFO:teuthology.orchestra.run.smithi088.stderr:2024-06-03T09:06:22.412+0000 7fad66ffd640 1 -- 172.21.15.88:0/2287369957 <== mon.0 v1:172.21.15.88:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 764+0+0 (unknown 3331228401 0 0) 0x7fad701417d0 con 0x7fad7013eec0 2024-06-03T09:06:22.415 INFO:teuthology.orchestra.run.smithi088.stderr:2024-06-03T09:06:22.412+0000 7fad66ffd640 1 -- 172.21.15.88:0/2287369957 --> v1:172.21.15.88:6789/0 -- auth(proto 2 165 bytes epoch 0) v1 -- 0x7fad70141640 con 0x7fad7013eec0 2024-06-03T09:06:22.415 INFO:teuthology.orchestra.run.smithi088.stderr:2024-06-03T09:06:22.412+0000 7fad66ffd640 1 -- 172.21.15.88:0/2287369957 <== mon.0 v1:172.21.15.88:6789/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 2234374065 0 0) 0x7fad60001810 con 0x7fad7013eec0 2024-06-03T09:06:22.415 INFO:teuthology.orchestra.run.smithi088.stderr:2024-06-03T09:06:22.412+0000 7fad66ffd640 10 monclient(hunting): handle_monmap mon_map magic: 0 v1 2024-06-03T09:06:22.415 INFO:teuthology.orchestra.run.smithi088.stderr:2024-06-03T09:06:22.412+0000 7fad66ffd640 10 monclient(hunting): got monmap 1 from mon.a (according to old e1) 2024-06-03T09:06:22.415 INFO:teuthology.orchestra.run.smithi088.stderr:2024-06-03T09:06:22.412+0000 7fad66ffd640 10 monclient(hunting): dump: 2024-06-03T09:06:22.415 INFO:teuthology.orchestra.run.smithi088.stderr:epoch 1 2024-06-03T09:06:22.415 INFO:teuthology.orchestra.run.smithi088.stderr:fsid ec3c3fdd-3ac6-4d7d-9412-683130ed8b04 2024-06-03T09:06:22.415 INFO:teuthology.orchestra.run.smithi088.stderr:last_changed 2024-06-03T08:34:31.885125+0000 2024-06-03T09:06:22.415 INFO:teuthology.orchestra.run.smithi088.stderr:created 2024-06-03T08:34:31.885125+0000 2024-06-03T09:06:22.416 INFO:teuthology.orchestra.run.smithi088.stderr:min_mon_release 18 (reef) 2024-06-03T09:06:22.416 INFO:teuthology.orchestra.run.smithi088.stderr:election_strategy: 3 2024-06-03T09:06:22.416 INFO:teuthology.orchestra.run.smithi088.stderr:0: v1:172.21.15.88:6789/0 mon.a 2024-06-03T09:06:22.416 INFO:teuthology.orchestra.run.smithi088.stderr:1: v1:172.21.15.192:6789/0 mon.b 2024-06-03T09:06:22.416 INFO:teuthology.orchestra.run.smithi088.stderr:2: v1:172.21.15.88:6790/0 mon.c 2024-06-03T09:06:22.416 INFO:teuthology.orchestra.run.smithi088.stderr: 2024-06-03T09:06:22.416 INFO:teuthology.orchestra.run.smithi088.stderr:2024-06-03T09:06:22.413+0000 7fad66ffd640 1 -- 172.21.15.88:0/2287369957 <== mon.1 v1:172.21.15.192:6789/0 4 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 580+0+0 (unknown 2040296863 0 0) 0x7fad700acd90 con 0x7fad701450f0 2024-06-03T09:06:22.416 INFO:teuthology.orchestra.run.smithi088.stderr:2024-06-03T09:06:22.413+0000 7fad66ffd640 1 -- 172.21.15.88:0/2287369957 >> v1:172.21.15.88:6790/0 conn(0x7fad700a8960 legacy=0x7fad70139210 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-06-03T09:06:22.416 INFO:teuthology.orchestra.run.smithi088.stderr:2024-06-03T09:06:22.413+0000 7fad66ffd640 1 -- 172.21.15.88:0/2287369957 >> v1:172.21.15.88:6789/0 conn(0x7fad7013eec0 legacy=0x7fad701d97d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-06-03T09:06:22.416 INFO:teuthology.orchestra.run.smithi088.stderr:2024-06-03T09:06:22.413+0000 7fad66ffd640 10 monclient: _finish_hunting 0 2024-06-03T09:06:22.416 INFO:teuthology.orchestra.run.smithi088.stderr:2024-06-03T09:06:22.413+0000 7fad66ffd640 1 monclient: found mon.b 2024-06-03T09:06:22.416 INFO:teuthology.orchestra.run.smithi088.stderr:2024-06-03T09:06:22.413+0000 7fad66ffd640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-06-03T09:06:22.416 INFO:teuthology.orchestra.run.smithi088.stderr:2024-06-03T09:06:22.413+0000 7fad66ffd640 10 monclient: _send_mon_message to mon.b at v1:172.21.15.192:6789/0 2024-06-03T09:06:22.417 INFO:teuthology.orchestra.run.smithi088.stderr:2024-06-03T09:06:22.413+0000 7fad66ffd640 1 -- 172.21.15.88:0/2287369957 --> v1:172.21.15.192:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fad70141960 con 0x7fad701450f0 2024-06-03T09:06:22.417 INFO:teuthology.orchestra.run.smithi088.stderr:2024-06-03T09:06:22.413+0000 7fad66ffd640 10 monclient: _finish_auth 0 2024-06-03T09:06:22.417 INFO:teuthology.orchestra.run.smithi088.stderr:2024-06-03T09:06:22.413+0000 7fad66ffd640 10 monclient: _check_auth_tickets 2024-06-03T09:06:22.417 INFO:teuthology.orchestra.run.smithi088.stderr:2024-06-03T09:06:22.413+0000 7fad66ffd640 20 monclient: _check_auth_rotating not needed by client.admin 2024-06-03T09:06:22.417 INFO:teuthology.orchestra.run.smithi088.stderr:2024-06-03T09:06:22.413+0000 7fad74d27640 5 monclient: authenticate success, global_id 30178 2024-06-03T09:06:22.417 INFO:teuthology.orchestra.run.smithi088.stderr:2024-06-03T09:06:22.413+0000 7fad74d27640 10 monclient: _renew_subs 2024-06-03T09:06:22.417 INFO:teuthology.orchestra.run.smithi088.stderr:2024-06-03T09:06:22.413+0000 7fad74d27640 10 monclient: _send_mon_message to mon.b at v1:172.21.15.192:6789/0 2024-06-03T09:06:22.417 INFO:teuthology.orchestra.run.smithi088.stderr:2024-06-03T09:06:22.413+0000 7fad74d27640 1 -- 172.21.15.88:0/2287369957 --> v1:172.21.15.192:6789/0 -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fad700ac7b0 con 0x7fad701450f0 2024-06-03T09:06:22.417 INFO:teuthology.orchestra.run.smithi088.stderr:2024-06-03T09:06:22.413+0000 7fad74d27640 10 client.30178.objecter _maybe_request_map subscribing (onetime) to next osd map 2024-06-03T09:06:22.417 INFO:teuthology.orchestra.run.smithi088.stderr:2024-06-03T09:06:22.413+0000 7fad74d27640 10 monclient: _renew_subs 2024-06-03T09:06:22.417 INFO:teuthology.orchestra.run.smithi088.stderr:2024-06-03T09:06:22.413+0000 7fad74d27640 10 monclient: _send_mon_message to mon.b at v1:172.21.15.192:6789/0 2024-06-03T09:06:22.417 INFO:teuthology.orchestra.run.smithi088.stderr:2024-06-03T09:06:22.413+0000 7fad74d27640 1 -- 172.21.15.88:0/2287369957 --> v1:172.21.15.192:6789/0 -- mon_subscribe({osdmap=0}) v3 -- 0x7fad70139920 con 0x7fad701450f0 2024-06-03T09:06:22.417 INFO:teuthology.orchestra.run.smithi088.stderr:2024-06-03T09:06:22.413+0000 7fad74d27640 1 librados: init done 2024-06-03T09:06:22.418 INFO:teuthology.orchestra.run.smithi088.stderr:2024-06-03T09:06:22.414+0000 7fad74d27640 10 start_mon_command cmd=[{"prefix": "get_command_descriptions"}] 2024-06-03T09:06:22.418 INFO:teuthology.orchestra.run.smithi088.stderr:2024-06-03T09:06:22.414+0000 7fad74d27640 10 monclient: _send_command 1 [{"prefix": "get_command_descriptions"}] 2024-06-03T09:06:22.418 INFO:teuthology.orchestra.run.smithi088.stderr:2024-06-03T09:06:22.414+0000 7fad74d27640 10 monclient: _send_mon_message to mon.b at v1:172.21.15.192:6789/0 2024-06-03T09:06:22.418 INFO:teuthology.orchestra.run.smithi088.stderr:2024-06-03T09:06:22.414+0000 7fad74d27640 1 -- 172.21.15.88:0/2287369957 --> v1:172.21.15.192:6789/0 -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fad700a9e60 con 0x7fad701450f0 2024-06-03T09:06:22.429 INFO:teuthology.orchestra.run.smithi088.stderr:2024-06-03T09:06:22.429+0000 7fad66ffd640 1 -- 172.21.15.88:0/2287369957 <== mon.1 v1:172.21.15.192:6789/0 5 ==== config(0 keys) v1 ==== 4+0+0 (unknown 0 0 0) 0x7fad68003c20 con 0x7fad701450f0 2024-06-03T09:06:22.429 INFO:teuthology.orchestra.run.smithi088.stderr:2024-06-03T09:06:22.429+0000 7fad66ffd640 10 monclient: handle_config config(0 keys) v1 2024-06-03T09:06:22.429 INFO:teuthology.orchestra.run.smithi088.stderr:2024-06-03T09:06:22.429+0000 7fad66ffd640 1 -- 172.21.15.88:0/2287369957 <== mon.1 v1:172.21.15.192:6789/0 6 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 2234374065 0 0) 0x7fad68003f30 con 0x7fad701450f0 2024-06-03T09:06:22.429 INFO:teuthology.orchestra.run.smithi088.stderr:2024-06-03T09:06:22.429+0000 7fad66ffd640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-06-03T09:06:22.429 INFO:teuthology.orchestra.run.smithi088.stderr:2024-06-03T09:06:22.429+0000 7fad66ffd640 10 monclient: got monmap 1 from mon.b (according to old e1) 2024-06-03T09:06:22.429 INFO:teuthology.orchestra.run.smithi088.stderr:2024-06-03T09:06:22.429+0000 7fad66ffd640 10 monclient: dump: 2024-06-03T09:06:22.429 INFO:teuthology.orchestra.run.smithi088.stderr:epoch 1 2024-06-03T09:06:22.429 INFO:teuthology.orchestra.run.smithi088.stderr:fsid ec3c3fdd-3ac6-4d7d-9412-683130ed8b04 2024-06-03T09:06:22.430 INFO:teuthology.orchestra.run.smithi088.stderr:last_changed 2024-06-03T08:34:31.885125+0000 2024-06-03T09:06:22.430 INFO:teuthology.orchestra.run.smithi088.stderr:created 2024-06-03T08:34:31.885125+0000 2024-06-03T09:06:22.430 INFO:teuthology.orchestra.run.smithi088.stderr:min_mon_release 18 (reef) 2024-06-03T09:06:22.430 INFO:teuthology.orchestra.run.smithi088.stderr:election_strategy: 3 2024-06-03T09:06:22.430 INFO:teuthology.orchestra.run.smithi088.stderr:0: v1:172.21.15.88:6789/0 mon.a 2024-06-03T09:06:22.430 INFO:teuthology.orchestra.run.smithi088.stderr:1: v1:172.21.15.192:6789/0 mon.b 2024-06-03T09:06:22.430 INFO:teuthology.orchestra.run.smithi088.stderr:2: v1:172.21.15.88:6790/0 mon.c 2024-06-03T09:06:22.430 INFO:teuthology.orchestra.run.smithi088.stderr: 2024-06-03T09:06:22.430 INFO:teuthology.orchestra.run.smithi088.stderr:2024-06-03T09:06:22.429+0000 7fad6cd99640 4 set_mon_vals no callback set 2024-06-03T09:06:22.431 INFO:teuthology.orchestra.run.smithi088.stderr:2024-06-03T09:06:22.429+0000 7fad66ffd640 1 -- 172.21.15.88:0/2287369957 <== mon.1 v1:172.21.15.192:6789/0 7 ==== mgrmap(e 5) v1 ==== 89110+0+0 (unknown 873533973 0 0) 0x7fad68019cb0 con 0x7fad701450f0 2024-06-03T09:06:22.431 INFO:teuthology.orchestra.run.smithi088.stderr:2024-06-03T09:06:22.430+0000 7fad6e59c640 10 monclient: get_auth_request con 0x7fad5006d430 auth_method 0 2024-06-03T09:06:22.431 INFO:teuthology.orchestra.run.smithi088.stderr:2024-06-03T09:06:22.430+0000 7fad66ffd640 10 client.30178.objecter ms_handle_connect 0x7fad5006d430 2024-06-03T09:06:22.431 INFO:teuthology.orchestra.run.smithi088.stderr:2024-06-03T09:06:22.431+0000 7fad66ffd640 1 -- 172.21.15.88:0/2287369957 <== mon.1 v1:172.21.15.192:6789/0 8 ==== osd_map(1010..1010 src has 622..1010) v4 ==== 5229+0+0 (unknown 1105883993 0 0) 0x7fad68053320 con 0x7fad701450f0 2024-06-03T09:06:22.432 INFO:teuthology.orchestra.run.smithi088.stderr:2024-06-03T09:06:22.432+0000 7fad66ffd640 10 client.30178.objecter ms_dispatch 0x7fad70000b90 osd_map(1010..1010 src has 622..1010) v4 2024-06-03T09:06:22.432 INFO:teuthology.orchestra.run.smithi088.stderr:2024-06-03T09:06:22.432+0000 7fad66ffd640 3 client.30178.objecter handle_osd_map got epochs [1010,1010] > 0 2024-06-03T09:06:22.432 INFO:teuthology.orchestra.run.smithi088.stderr:2024-06-03T09:06:22.432+0000 7fad66ffd640 3 client.30178.objecter handle_osd_map decoding full epoch 1010 2024-06-03T09:06:22.432 INFO:teuthology.orchestra.run.smithi088.stderr:2024-06-03T09:06:22.432+0000 7fad66ffd640 20 client.30178.objecter dump_active .. 0 homeless 2024-06-03T09:06:22.672 INFO:teuthology.orchestra.run.smithi088.stderr:2024-06-03T09:06:22.673+0000 7fad66ffd640 1 -- 172.21.15.88:0/2287369957 <== mon.1 v1:172.21.15.192:6789/0 9 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+182498 (unknown 1092875540 0 3016760700) 0x7fad6804eed0 con 0x7fad701450f0 2024-06-03T09:06:22.672 INFO:teuthology.orchestra.run.smithi088.stderr:2024-06-03T09:06:22.673+0000 7fad66ffd640 10 monclient: handle_mon_command_ack 1 [{"prefix": "get_command_descriptions"}] 2024-06-03T09:06:22.673 INFO:teuthology.orchestra.run.smithi088.stderr:2024-06-03T09:06:22.673+0000 7fad66ffd640 10 monclient: _finish_command 1 = system:0 2024-06-03T09:06:22.850 INFO:teuthology.orchestra.run.smithi088.stderr:2024-06-03T09:06:22.850+0000 7fad74d27640 1 -- 172.21.15.88:0/2287369957 --> v1:172.21.15.192:6800/34229 -- mgr_command(tid 0: {"prefix": "pg dump", "target": ["mon-mgr", ""], "format": "json"}) v1 -- 0x7fad700acf20 con 0x7fad5006d430 2024-06-03T09:06:22.859 INFO:teuthology.orchestra.run.smithi088.stderr:2024-06-03T09:06:22.859+0000 7fad66ffd640 1 -- 172.21.15.88:0/2287369957 <== mgr.4110 v1:172.21.15.192:6800/34229 1 ==== mgr_command_reply(tid 0: 0 dumped all) v1 ==== 18+0+151744 (unknown 2965378022 0 78872612) 0x7fad700acf20 con 0x7fad5006d430 2024-06-03T09:06:22.860 INFO:teuthology.orchestra.run.smithi088.stdout: 2024-06-03T09:06:22.861 INFO:teuthology.orchestra.run.smithi088.stderr:dumped all 2024-06-03T09:06:22.863 INFO:teuthology.orchestra.run.smithi088.stderr:2024-06-03T09:06:22.861+0000 7fad74d27640 10 librados: watch_flush enter 2024-06-03T09:06:22.863 INFO:teuthology.orchestra.run.smithi088.stderr:2024-06-03T09:06:22.861+0000 7fad74d27640 10 librados: watch_flush exit 2024-06-03T09:06:22.863 INFO:teuthology.orchestra.run.smithi088.stderr:2024-06-03T09:06:22.861+0000 7fad74d27640 20 client.30178.objecter shutdown clearing up homeless session... 2024-06-03T09:06:22.863 INFO:teuthology.orchestra.run.smithi088.stderr:2024-06-03T09:06:22.861+0000 7fad74d27640 10 client.30178.objecter successfully canceled tick 2024-06-03T09:06:22.863 INFO:teuthology.orchestra.run.smithi088.stderr:2024-06-03T09:06:22.861+0000 7fad74d27640 1 -- 172.21.15.88:0/2287369957 >> v1:172.21.15.192:6800/34229 conn(0x7fad5006d430 legacy=0x7fad5006f8f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-06-03T09:06:22.863 INFO:teuthology.orchestra.run.smithi088.stderr:2024-06-03T09:06:22.861+0000 7fad74d27640 10 monclient: shutdown 2024-06-03T09:06:22.864 INFO:teuthology.orchestra.run.smithi088.stderr:2024-06-03T09:06:22.861+0000 7fad74d27640 20 monclient: shutdown discarding 0 pending message(s) 2024-06-03T09:06:22.864 INFO:teuthology.orchestra.run.smithi088.stderr:2024-06-03T09:06:22.861+0000 7fad74d27640 1 -- 172.21.15.88:0/2287369957 >> v1:172.21.15.192:6789/0 conn(0x7fad701450f0 legacy=0x7fad701e1b70 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-06-03T09:06:22.864 INFO:teuthology.orchestra.run.smithi088.stderr:2024-06-03T09:06:22.861+0000 7fad74d27640 1 -- 172.21.15.88:0/2287369957 shutdown_connections 2024-06-03T09:06:22.864 INFO:teuthology.orchestra.run.smithi088.stderr:2024-06-03T09:06:22.861+0000 7fad74d27640 1 -- 172.21.15.88:0/2287369957 >> 172.21.15.88:0/2287369957 conn(0x7fad700b70d0 msgr2=0x7fad7013e3f0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-06-03T09:06:22.864 INFO:teuthology.orchestra.run.smithi088.stderr:2024-06-03T09:06:22.861+0000 7fad74d27640 1 -- 172.21.15.88:0/2287369957 shutdown_connections 2024-06-03T09:06:22.864 INFO:teuthology.orchestra.run.smithi088.stderr:2024-06-03T09:06:22.861+0000 7fad74d27640 1 -- 172.21.15.88:0/2287369957 wait complete. 2024-06-03T09:06:22.864 INFO:teuthology.orchestra.run.smithi088.stderr:2024-06-03T09:06:22.861+0000 7fad74d27640 1 librados: shutdown 2024-06-03T09:06:22.872 INFO:teuthology.orchestra.run.smithi088.stdout:{"pg_ready":true,"pg_map":{"version":1916,"stamp":"2024-06-03T09:06:21.098664+0000","last_osdmap_epoch":0,"last_pg_scan":0,"pg_stats_sum":{"stat_sum":{"num_bytes":459285,"num_objects":3,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":1,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":3,"num_whiteouts":0,"num_read":102,"num_read_kb":88,"num_write":120,"num_write_kb":1379,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2,"num_bytes_recovered":397840,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":83,"ondisk_log_size":83,"up":68,"acting":68,"num_store_stats":0},"osd_stats_sum":{"up_from":0,"seq":0,"num_pgs":100,"num_osds":8,"num_per_pool_osds":8,"num_per_pool_omap_osds":8,"kb":754974720,"kb_used":494540,"kb_used_data":46312,"kb_used_omap":12,"kb_used_meta":448115,"kb_avail":754480180,"statfs":{"total":773094113280,"available":772587704320,"internally_reserved":0,"allocated":47423488,"data_stored":22742525,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":12744,"internal_metadata":458870328},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":146,"apply_latency_ms":146,"commit_latency_ns":146000000,"apply_latency_ns":146000000},"alerts":[],"network_ping_times":[]},"pg_stats_delta":{"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":0,"acting":0,"num_store_stats":0,"stamp_delta":"8.001643"},"pg_stats":[{"pgid":"17.d","version":"0'0","reported_seq":942,"reported_epoch":1009,"state":"active+clean","last_fresh":"2024-06-03T09:06:14.443917+0000","last_change":"2024-06-03T08:38:19.260548+0000","last_active":"2024-06-03T09:06:14.443917+0000","last_peered":"2024-06-03T09:06:14.443917+0000","last_clean":"2024-06-03T09:06:14.443917+0000","last_became_active":"2024-06-03T08:38:19.253559+0000","last_became_peered":"2024-06-03T08:38:19.253559+0000","last_unstale":"2024-06-03T09:06:14.443917+0000","last_undegraded":"2024-06-03T09:06:14.443917+0000","last_fullsized":"2024-06-03T09:06:14.443917+0000","mapping_epoch":28,"log_start":"0'0","ondisk_log_start":"0'0","created":28,"last_epoch_clean":45,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-06-03T08:37:55.614497+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-06-03T08:37:55.614497+0000","last_clean_scrub_stamp":"2024-06-03T08:37:55.614497+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-06-04T19:00:12.470979+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1],"acting":[0,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"17.c","version":"0'0","reported_seq":953,"reported_epoch":1009,"state":"active+undersized","last_fresh":"2024-06-03T09:06:13.668156+0000","last_change":"2024-06-03T09:06:03.440074+0000","last_active":"2024-06-03T09:06:13.668156+0000","last_peered":"2024-06-03T09:06:13.668156+0000","last_clean":"2024-06-03T09:05:02.105638+0000","last_became_active":"2024-06-03T09:06:03.440074+0000","last_became_peered":"2024-06-03T09:06:03.440074+0000","last_unstale":"2024-06-03T09:06:13.668156+0000","last_undegraded":"2024-06-03T09:06:13.668156+0000","last_fullsized":"2024-06-03T09:06:03.377982+0000","mapping_epoch":999,"log_start":"0'0","ondisk_log_start":"0'0","created":28,"last_epoch_clean":255,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-06-03T08:37:55.614497+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-06-03T08:37:55.614497+0000","last_clean_scrub_stamp":"2024-06-03T08:37:55.614497+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-06-04T10:34:20.808557+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3],"acting":[3],"avail_no_missing":["3"],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"17.f","version":"0'0","reported_seq":148,"reported_epoch":1009,"state":"active+undersized","last_fresh":"2024-06-03T09:06:14.450543+0000","last_change":"2024-06-03T09:06:03.420285+0000","last_active":"2024-06-03T09:06:14.450543+0000","last_peered":"2024-06-03T09:06:14.450543+0000","last_clean":"2024-06-03T08:37:55.614497+0000","last_became_active":"2024-06-03T09:06:03.420285+0000","last_became_peered":"2024-06-03T09:06:03.420285+0000","last_unstale":"2024-06-03T09:06:14.450543+0000","last_undegraded":"2024-06-03T09:06:14.450543+0000","last_fullsized":"2024-06-03T09:06:03.380616+0000","mapping_epoch":999,"log_start":"0'0","ondisk_log_start":"0'0","created":28,"last_epoch_clean":255,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-06-03T08:37:55.614497+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-06-03T08:37:55.614497+0000","last_clean_scrub_stamp":"2024-06-03T08:37:55.614497+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-06-04T15:39:41.187731+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4],"acting":[4],"avail_no_missing":["4"],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"17.e","version":"0'0","reported_seq":149,"reported_epoch":1009,"state":"active+undersized","last_fresh":"2024-06-03T09:06:13.726992+0000","last_change":"2024-06-03T09:06:03.412887+0000","last_active":"2024-06-03T09:06:13.726992+0000","last_peered":"2024-06-03T09:06:13.726992+0000","last_clean":"2024-06-03T08:37:55.614497+0000","last_became_active":"2024-06-03T09:06:03.412887+0000","last_became_peered":"2024-06-03T09:06:03.412887+0000","last_unstale":"2024-06-03T09:06:13.726992+0000","last_undegraded":"2024-06-03T09:06:13.726992+0000","last_fullsized":"2024-06-03T09:06:03.332482+0000","mapping_epoch":999,"log_start":"0'0","ondisk_log_start":"0'0","created":28,"last_epoch_clean":255,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-06-03T08:37:55.614497+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-06-03T08:37:55.614497+0000","last_clean_scrub_stamp":"2024-06-03T08:37:55.614497+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-06-04T12:23:22.511601+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5],"acting":[5],"avail_no_missing":["5"],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"17.9","version":"0'0","reported_seq":946,"reported_epoch":1009,"state":"active+clean","last_fresh":"2024-06-03T09:06:14.999299+0000","last_change":"2024-06-03T08:38:04.772260+0000","last_active":"2024-06-03T09:06:14.999299+0000","last_peered":"2024-06-03T09:06:14.999299+0000","last_clean":"2024-06-03T09:06:14.999299+0000","last_became_active":"2024-06-03T08:38:04.763428+0000","last_became_peered":"2024-06-03T08:38:04.763428+0000","last_unstale":"2024-06-03T09:06:14.999299+0000","last_undegraded":"2024-06-03T09:06:14.999299+0000","last_fullsized":"2024-06-03T09:06:14.999299+0000","mapping_epoch":28,"log_start":"0'0","ondisk_log_start":"0'0","created":28,"last_epoch_clean":33,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-06-03T08:37:55.614497+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-06-03T08:37:55.614497+0000","last_clean_scrub_stamp":"2024-06-03T08:37:55.614497+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-06-04T10:55:10.773936+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,4],"acting":[7,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"17.8","version":"0'0","reported_seq":961,"reported_epoch":1009,"state":"active+undersized","last_fresh":"2024-06-03T09:06:14.430375+0000","last_change":"2024-06-03T09:06:03.388786+0000","last_active":"2024-06-03T09:06:14.430375+0000","last_peered":"2024-06-03T09:06:14.430375+0000","last_clean":"2024-06-03T09:04:43.297780+0000","last_became_active":"2024-06-03T09:06:03.388786+0000","last_became_peered":"2024-06-03T09:06:03.388786+0000","last_unstale":"2024-06-03T09:06:14.430375+0000","last_undegraded":"2024-06-03T09:06:14.430375+0000","last_fullsized":"2024-06-03T09:06:03.331754+0000","mapping_epoch":999,"log_start":"0'0","ondisk_log_start":"0'0","created":28,"last_epoch_clean":255,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-06-03T08:37:55.614497+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-06-03T08:37:55.614497+0000","last_clean_scrub_stamp":"2024-06-03T08:37:55.614497+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-06-04T09:18:44.765547+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0],"acting":[0],"avail_no_missing":["0"],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"17.b","version":"0'0","reported_seq":149,"reported_epoch":1009,"state":"active+undersized","last_fresh":"2024-06-03T09:06:14.433735+0000","last_change":"2024-06-03T09:06:03.451203+0000","last_active":"2024-06-03T09:06:14.433735+0000","last_peered":"2024-06-03T09:06:14.433735+0000","last_clean":"2024-06-03T08:37:55.614497+0000","last_became_active":"2024-06-03T09:06:03.451203+0000","last_became_peered":"2024-06-03T09:06:03.451203+0000","last_unstale":"2024-06-03T09:06:14.433735+0000","last_undegraded":"2024-06-03T09:06:14.433735+0000","last_fullsized":"2024-06-03T09:06:03.375523+0000","mapping_epoch":999,"log_start":"0'0","ondisk_log_start":"0'0","created":28,"last_epoch_clean":255,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-06-03T08:37:55.614497+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-06-03T08:37:55.614497+0000","last_clean_scrub_stamp":"2024-06-03T08:37:55.614497+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-06-04T10:16:19.813495+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2],"acting":[2],"avail_no_missing":["2"],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"17.a","version":"0'0","reported_seq":924,"reported_epoch":1009,"state":"active+clean","last_fresh":"2024-06-03T09:06:13.710307+0000","last_change":"2024-06-03T08:38:05.843142+0000","last_active":"2024-06-03T09:06:13.710307+0000","last_peered":"2024-06-03T09:06:13.710307+0000","last_clean":"2024-06-03T09:06:13.710307+0000","last_became_active":"2024-06-03T08:38:05.809257+0000","last_became_peered":"2024-06-03T08:38:05.809257+0000","last_unstale":"2024-06-03T09:06:13.710307+0000","last_undegraded":"2024-06-03T09:06:13.710307+0000","last_fullsized":"2024-06-03T09:06:13.710307+0000","mapping_epoch":28,"log_start":"0'0","ondisk_log_start":"0'0","created":28,"last_epoch_clean":33,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-06-03T08:37:55.614497+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-06-03T08:37:55.614497+0000","last_clean_scrub_stamp":"2024-06-03T08:37:55.614497+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-06-04T12:12:03.686838+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2],"acting":[1,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"17.5","version":"0'0","reported_seq":943,"reported_epoch":1009,"state":"active+clean","last_fresh":"2024-06-03T09:06:14.442357+0000","last_change":"2024-06-03T08:38:23.466051+0000","last_active":"2024-06-03T09:06:14.442357+0000","last_peered":"2024-06-03T09:06:14.442357+0000","last_clean":"2024-06-03T09:06:14.442357+0000","last_became_active":"2024-06-03T08:38:23.458039+0000","last_became_peered":"2024-06-03T08:38:23.458039+0000","last_unstale":"2024-06-03T09:06:14.442357+0000","last_undegraded":"2024-06-03T09:06:14.442357+0000","last_fullsized":"2024-06-03T09:06:14.442357+0000","mapping_epoch":28,"log_start":"0'0","ondisk_log_start":"0'0","created":28,"last_epoch_clean":49,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-06-03T08:37:55.614497+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-06-03T08:37:55.614497+0000","last_clean_scrub_stamp":"2024-06-03T08:37:55.614497+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-06-04T12:27:02.532436+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,4],"acting":[0,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"17.4","version":"0'0","reported_seq":149,"reported_epoch":1009,"state":"active+undersized","last_fresh":"2024-06-03T09:06:13.941522+0000","last_change":"2024-06-03T09:06:03.441889+0000","last_active":"2024-06-03T09:06:13.941522+0000","last_peered":"2024-06-03T09:06:13.941522+0000","last_clean":"2024-06-03T08:37:55.614497+0000","last_became_active":"2024-06-03T09:06:03.441889+0000","last_became_peered":"2024-06-03T09:06:03.441889+0000","last_unstale":"2024-06-03T09:06:13.941522+0000","last_undegraded":"2024-06-03T09:06:13.941522+0000","last_fullsized":"2024-06-03T09:06:03.359582+0000","mapping_epoch":999,"log_start":"0'0","ondisk_log_start":"0'0","created":28,"last_epoch_clean":255,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-06-03T08:37:55.614497+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-06-03T08:37:55.614497+0000","last_clean_scrub_stamp":"2024-06-03T08:37:55.614497+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-06-04T12:49:22.907141+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7],"acting":[7],"avail_no_missing":["7"],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"17.7","version":"0'0","reported_seq":929,"reported_epoch":1009,"state":"active+clean","last_fresh":"2024-06-03T09:06:13.736920+0000","last_change":"2024-06-03T08:38:04.837083+0000","last_active":"2024-06-03T09:06:13.736920+0000","last_peered":"2024-06-03T09:06:13.736920+0000","last_clean":"2024-06-03T09:06:13.736920+0000","last_became_active":"2024-06-03T08:38:04.818246+0000","last_became_peered":"2024-06-03T08:38:04.818246+0000","last_unstale":"2024-06-03T09:06:13.736920+0000","last_undegraded":"2024-06-03T09:06:13.736920+0000","last_fullsized":"2024-06-03T09:06:13.736920+0000","mapping_epoch":28,"log_start":"0'0","ondisk_log_start":"0'0","created":28,"last_epoch_clean":34,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-06-03T08:37:55.614497+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-06-03T08:37:55.614497+0000","last_clean_scrub_stamp":"2024-06-03T08:37:55.614497+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-06-04T17:02:38.204813+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,2],"acting":[5,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"17.6","version":"0'0","reported_seq":960,"reported_epoch":1009,"state":"active+undersized","last_fresh":"2024-06-03T09:06:14.453110+0000","last_change":"2024-06-03T09:06:03.428677+0000","last_active":"2024-06-03T09:06:14.453110+0000","last_peered":"2024-06-03T09:06:14.453110+0000","last_clean":"2024-06-03T09:04:43.312352+0000","last_became_active":"2024-06-03T09:06:03.428677+0000","last_became_peered":"2024-06-03T09:06:03.428677+0000","last_unstale":"2024-06-03T09:06:14.453110+0000","last_undegraded":"2024-06-03T09:06:14.453110+0000","last_fullsized":"2024-06-03T09:06:03.359909+0000","mapping_epoch":999,"log_start":"0'0","ondisk_log_start":"0'0","created":28,"last_epoch_clean":255,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-06-03T08:37:55.614497+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-06-03T08:37:55.614497+0000","last_clean_scrub_stamp":"2024-06-03T08:37:55.614497+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-06-04T15:38:02.180241+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0],"acting":[0],"avail_no_missing":["0"],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"2.7","version":"0'0","reported_seq":940,"reported_epoch":1009,"state":"active+clean","last_fresh":"2024-06-03T09:06:13.684065+0000","last_change":"2024-06-03T08:36:30.385405+0000","last_active":"2024-06-03T09:06:13.684065+0000","last_peered":"2024-06-03T09:06:13.684065+0000","last_clean":"2024-06-03T09:06:13.684065+0000","last_became_active":"2024-06-03T08:36:30.380302+0000","last_became_peered":"2024-06-03T08:36:30.380302+0000","last_unstale":"2024-06-03T09:06:13.684065+0000","last_undegraded":"2024-06-03T09:06:13.684065+0000","last_fullsized":"2024-06-03T09:06:13.684065+0000","mapping_epoch":23,"log_start":"0'0","ondisk_log_start":"0'0","created":23,"last_epoch_clean":24,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-06-03T08:36:28.444764+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-06-03T08:36:28.444764+0000","last_clean_scrub_stamp":"2024-06-03T08:36:28.444764+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-06-04T18:46:33.475181+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0],"acting":[3,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"17.14","version":"0'0","reported_seq":926,"reported_epoch":1009,"state":"active+clean","last_fresh":"2024-06-03T09:06:13.721460+0000","last_change":"2024-06-03T08:38:04.547565+0000","last_active":"2024-06-03T09:06:13.721460+0000","last_peered":"2024-06-03T09:06:13.721460+0000","last_clean":"2024-06-03T09:06:13.721460+0000","last_became_active":"2024-06-03T08:38:04.541781+0000","last_became_peered":"2024-06-03T08:38:04.541781+0000","last_unstale":"2024-06-03T09:06:13.721460+0000","last_undegraded":"2024-06-03T09:06:13.721460+0000","last_fullsized":"2024-06-03T09:06:13.721460+0000","mapping_epoch":28,"log_start":"0'0","ondisk_log_start":"0'0","created":28,"last_epoch_clean":34,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-06-03T08:37:55.614497+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-06-03T08:37:55.614497+0000","last_clean_scrub_stamp":"2024-06-03T08:37:55.614497+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-06-04T16:41:55.170134+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,2],"acting":[5,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"2.6","version":"0'0","reported_seq":949,"reported_epoch":1009,"state":"active+clean","last_fresh":"2024-06-03T09:06:14.448100+0000","last_change":"2024-06-03T08:36:30.430760+0000","last_active":"2024-06-03T09:06:14.448100+0000","last_peered":"2024-06-03T09:06:14.448100+0000","last_clean":"2024-06-03T09:06:14.448100+0000","last_became_active":"2024-06-03T08:36:30.404665+0000","last_became_peered":"2024-06-03T08:36:30.404665+0000","last_unstale":"2024-06-03T09:06:14.448100+0000","last_undegraded":"2024-06-03T09:06:14.448100+0000","last_fullsized":"2024-06-03T09:06:14.448100+0000","mapping_epoch":23,"log_start":"0'0","ondisk_log_start":"0'0","created":23,"last_epoch_clean":24,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-06-03T08:36:28.444764+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-06-03T08:36:28.444764+0000","last_clean_scrub_stamp":"2024-06-03T08:36:28.444764+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-06-04T10:37:37.746074+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,0],"acting":[4,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"17.15","version":"0'0","reported_seq":934,"reported_epoch":1009,"state":"active+clean","last_fresh":"2024-06-03T09:06:13.672940+0000","last_change":"2024-06-03T08:38:20.999903+0000","last_active":"2024-06-03T09:06:13.672940+0000","last_peered":"2024-06-03T09:06:13.672940+0000","last_clean":"2024-06-03T09:06:13.672940+0000","last_became_active":"2024-06-03T08:38:20.979523+0000","last_became_peered":"2024-06-03T08:38:20.979523+0000","last_unstale":"2024-06-03T09:06:13.672940+0000","last_undegraded":"2024-06-03T09:06:13.672940+0000","last_fullsized":"2024-06-03T09:06:13.672940+0000","mapping_epoch":28,"log_start":"0'0","ondisk_log_start":"0'0","created":28,"last_epoch_clean":47,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-06-03T08:37:55.614497+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-06-03T08:37:55.614497+0000","last_clean_scrub_stamp":"2024-06-03T08:37:55.614497+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-06-04T20:29:58.314289+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,1],"acting":[3,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"2.5","version":"0'0","reported_seq":941,"reported_epoch":1009,"state":"active+clean","last_fresh":"2024-06-03T09:06:13.674542+0000","last_change":"2024-06-03T08:36:30.024235+0000","last_active":"2024-06-03T09:06:13.674542+0000","last_peered":"2024-06-03T09:06:13.674542+0000","last_clean":"2024-06-03T09:06:13.674542+0000","last_became_active":"2024-06-03T08:36:30.019040+0000","last_became_peered":"2024-06-03T08:36:30.019040+0000","last_unstale":"2024-06-03T09:06:13.674542+0000","last_undegraded":"2024-06-03T09:06:13.674542+0000","last_fullsized":"2024-06-03T09:06:13.674542+0000","mapping_epoch":23,"log_start":"0'0","ondisk_log_start":"0'0","created":23,"last_epoch_clean":24,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-06-03T08:36:28.444764+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-06-03T08:36:28.444764+0000","last_clean_scrub_stamp":"2024-06-03T08:36:28.444764+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-06-04T15:46:47.109502+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,4],"acting":[3,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"17.16","version":"0'0","reported_seq":972,"reported_epoch":1009,"state":"active+undersized","last_fresh":"2024-06-03T09:06:14.441665+0000","last_change":"2024-06-03T09:06:03.478615+0000","last_active":"2024-06-03T09:06:14.441665+0000","last_peered":"2024-06-03T09:06:14.441665+0000","last_clean":"2024-06-03T09:04:43.328415+0000","last_became_active":"2024-06-03T09:06:03.478615+0000","last_became_peered":"2024-06-03T09:06:03.478615+0000","last_unstale":"2024-06-03T09:06:14.441665+0000","last_undegraded":"2024-06-03T09:06:14.441665+0000","last_fullsized":"2024-06-03T09:06:03.402504+0000","mapping_epoch":999,"log_start":"0'0","ondisk_log_start":"0'0","created":28,"last_epoch_clean":255,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-06-03T08:37:55.614497+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-06-03T08:37:55.614497+0000","last_clean_scrub_stamp":"2024-06-03T08:37:55.614497+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-06-04T08:47:08.079233+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2],"acting":[2],"avail_no_missing":["2"],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"2.4","version":"0'0","reported_seq":952,"reported_epoch":1010,"state":"active+undersized","last_fresh":"2024-06-03T09:06:18.331485+0000","last_change":"2024-06-03T09:06:03.387171+0000","last_active":"2024-06-03T09:06:18.331485+0000","last_peered":"2024-06-03T09:06:18.331485+0000","last_clean":"2024-06-03T09:04:43.202645+0000","last_became_active":"2024-06-03T09:06:03.387171+0000","last_became_peered":"2024-06-03T09:06:03.387171+0000","last_unstale":"2024-06-03T09:06:18.331485+0000","last_undegraded":"2024-06-03T09:06:18.331485+0000","last_fullsized":"2024-06-03T09:06:03.283787+0000","mapping_epoch":999,"log_start":"0'0","ondisk_log_start":"0'0","created":23,"last_epoch_clean":255,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-06-03T08:36:28.444764+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-06-03T08:36:28.444764+0000","last_clean_scrub_stamp":"2024-06-03T08:36:28.444764+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-06-04T20:17:43.485023+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5],"acting":[5],"avail_no_missing":["5"],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"17.17","version":"0'0","reported_seq":954,"reported_epoch":1009,"state":"active+clean","last_fresh":"2024-06-03T09:06:14.447046+0000","last_change":"2024-06-03T08:38:38.483275+0000","last_active":"2024-06-03T09:06:14.447046+0000","last_peered":"2024-06-03T09:06:14.447046+0000","last_clean":"2024-06-03T09:06:14.447046+0000","last_became_active":"2024-06-03T08:38:38.467916+0000","last_became_peered":"2024-06-03T08:38:38.467916+0000","last_unstale":"2024-06-03T09:06:14.447046+0000","last_undegraded":"2024-06-03T09:06:14.447046+0000","last_fullsized":"2024-06-03T09:06:14.447046+0000","mapping_epoch":28,"log_start":"0'0","ondisk_log_start":"0'0","created":28,"last_epoch_clean":61,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-06-03T08:37:55.614497+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-06-03T08:37:55.614497+0000","last_clean_scrub_stamp":"2024-06-03T08:37:55.614497+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-06-04T10:02:23.036420+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,7],"acting":[2,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"2.2","version":"0'0","reported_seq":963,"reported_epoch":1009,"state":"active+undersized","last_fresh":"2024-06-03T09:06:13.657892+0000","last_change":"2024-06-03T09:06:03.410020+0000","last_active":"2024-06-03T09:06:13.657892+0000","last_peered":"2024-06-03T09:06:13.657892+0000","last_clean":"2024-06-03T09:05:37.145889+0000","last_became_active":"2024-06-03T09:06:03.410020+0000","last_became_peered":"2024-06-03T09:06:03.410020+0000","last_unstale":"2024-06-03T09:06:13.657892+0000","last_undegraded":"2024-06-03T09:06:13.657892+0000","last_fullsized":"2024-06-03T09:06:03.346864+0000","mapping_epoch":999,"log_start":"0'0","ondisk_log_start":"0'0","created":23,"last_epoch_clean":255,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-06-03T08:36:28.444764+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-06-03T08:36:28.444764+0000","last_clean_scrub_stamp":"2024-06-03T08:36:28.444764+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-06-04T09:57:12.423360+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3],"acting":[3],"avail_no_missing":["3"],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"17.11","version":"0'0","reported_seq":944,"reported_epoch":1009,"state":"active+clean","last_fresh":"2024-06-03T09:06:14.426670+0000","last_change":"2024-06-03T08:38:06.502302+0000","last_active":"2024-06-03T09:06:14.426670+0000","last_peered":"2024-06-03T09:06:14.426670+0000","last_clean":"2024-06-03T09:06:14.426670+0000","last_became_active":"2024-06-03T08:38:06.490204+0000","last_became_peered":"2024-06-03T08:38:06.490204+0000","last_unstale":"2024-06-03T09:06:14.426670+0000","last_undegraded":"2024-06-03T09:06:14.426670+0000","last_fullsized":"2024-06-03T09:06:14.426670+0000","mapping_epoch":28,"log_start":"0'0","ondisk_log_start":"0'0","created":28,"last_epoch_clean":32,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-06-03T08:37:55.614497+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-06-03T08:37:55.614497+0000","last_clean_scrub_stamp":"2024-06-03T08:37:55.614497+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-06-04T10:47:05.988958+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,2],"acting":[4,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"2.1","version":"0'0","reported_seq":952,"reported_epoch":1009,"state":"active+clean","last_fresh":"2024-06-03T09:06:13.929725+0000","last_change":"2024-06-03T08:36:30.883360+0000","last_active":"2024-06-03T09:06:13.929725+0000","last_peered":"2024-06-03T09:06:13.929725+0000","last_clean":"2024-06-03T09:06:13.929725+0000","last_became_active":"2024-06-03T08:36:30.855073+0000","last_became_peered":"2024-06-03T08:36:30.855073+0000","last_unstale":"2024-06-03T09:06:13.929725+0000","last_undegraded":"2024-06-03T09:06:13.929725+0000","last_fullsized":"2024-06-03T09:06:13.929725+0000","mapping_epoch":23,"log_start":"0'0","ondisk_log_start":"0'0","created":23,"last_epoch_clean":25,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-06-03T08:36:28.444764+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-06-03T08:36:28.444764+0000","last_clean_scrub_stamp":"2024-06-03T08:36:28.444764+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-06-04T18:12:22.363473+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,5],"acting":[7,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"17.12","version":"0'0","reported_seq":945,"reported_epoch":1009,"state":"active+clean","last_fresh":"2024-06-03T09:06:13.935141+0000","last_change":"2024-06-03T08:38:01.751562+0000","last_active":"2024-06-03T09:06:13.935141+0000","last_peered":"2024-06-03T09:06:13.935141+0000","last_clean":"2024-06-03T09:06:13.935141+0000","last_became_active":"2024-06-03T08:38:01.745027+0000","last_became_peered":"2024-06-03T08:38:01.745027+0000","last_unstale":"2024-06-03T09:06:13.935141+0000","last_undegraded":"2024-06-03T09:06:13.935141+0000","last_fullsized":"2024-06-03T09:06:13.935141+0000","mapping_epoch":28,"log_start":"0'0","ondisk_log_start":"0'0","created":28,"last_epoch_clean":32,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-06-03T08:37:55.614497+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-06-03T08:37:55.614497+0000","last_clean_scrub_stamp":"2024-06-03T08:37:55.614497+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-06-04T17:54:43.785326+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,3],"acting":[7,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"2.0","version":"0'0","reported_seq":943,"reported_epoch":1009,"state":"active+clean","last_fresh":"2024-06-03T09:06:13.648112+0000","last_change":"2024-06-03T08:36:29.984096+0000","last_active":"2024-06-03T09:06:13.648112+0000","last_peered":"2024-06-03T09:06:13.648112+0000","last_clean":"2024-06-03T09:06:13.648112+0000","last_became_active":"2024-06-03T08:36:29.957875+0000","last_became_peered":"2024-06-03T08:36:29.957875+0000","last_unstale":"2024-06-03T09:06:13.648112+0000","last_undegraded":"2024-06-03T09:06:13.648112+0000","last_fullsized":"2024-06-03T09:06:13.648112+0000","mapping_epoch":23,"log_start":"0'0","ondisk_log_start":"0'0","created":23,"last_epoch_clean":24,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-06-03T08:36:28.444764+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-06-03T08:36:28.444764+0000","last_clean_scrub_stamp":"2024-06-03T08:36:28.444764+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-06-04T08:41:33.679479+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,5],"acting":[3,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"17.13","version":"0'0","reported_seq":943,"reported_epoch":1009,"state":"active+clean","last_fresh":"2024-06-03T09:06:14.438399+0000","last_change":"2024-06-03T08:38:26.838316+0000","last_active":"2024-06-03T09:06:14.438399+0000","last_peered":"2024-06-03T09:06:14.438399+0000","last_clean":"2024-06-03T09:06:14.438399+0000","last_became_active":"2024-06-03T08:38:26.833214+0000","last_became_peered":"2024-06-03T08:38:26.833214+0000","last_unstale":"2024-06-03T09:06:14.438399+0000","last_undegraded":"2024-06-03T09:06:14.438399+0000","last_fullsized":"2024-06-03T09:06:14.438399+0000","mapping_epoch":28,"log_start":"0'0","ondisk_log_start":"0'0","created":28,"last_epoch_clean":52,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-06-03T08:37:55.614497+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-06-03T08:37:55.614497+0000","last_clean_scrub_stamp":"2024-06-03T08:37:55.614497+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-06-04T12:58:51.617998+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,0],"acting":[4,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"2.3","version":"0'0","reported_seq":942,"reported_epoch":1009,"state":"active+clean","last_fresh":"2024-06-03T09:06:13.664847+0000","last_change":"2024-06-03T08:36:30.351670+0000","last_active":"2024-06-03T09:06:13.664847+0000","last_peered":"2024-06-03T09:06:13.664847+0000","last_clean":"2024-06-03T09:06:13.664847+0000","last_became_active":"2024-06-03T08:36:30.343560+0000","last_became_peered":"2024-06-03T08:36:30.343560+0000","last_unstale":"2024-06-03T09:06:13.664847+0000","last_undegraded":"2024-06-03T09:06:13.664847+0000","last_fullsized":"2024-06-03T09:06:13.664847+0000","mapping_epoch":23,"log_start":"0'0","ondisk_log_start":"0'0","created":23,"last_epoch_clean":24,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-06-03T08:36:28.444764+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-06-03T08:36:28.444764+0000","last_clean_scrub_stamp":"2024-06-03T08:36:28.444764+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-06-04T12:02:24.765135+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,2],"acting":[3,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.0","version":"19'76","reported_seq":1051,"reported_epoch":1009,"state":"active+clean","last_fresh":"2024-06-03T09:06:13.703685+0000","last_change":"2024-06-03T08:36:26.702702+0000","last_active":"2024-06-03T09:06:13.703685+0000","last_peered":"2024-06-03T09:06:13.703685+0000","last_clean":"2024-06-03T09:06:13.703685+0000","last_became_active":"2024-06-03T08:36:26.128378+0000","last_became_peered":"2024-06-03T08:36:26.128378+0000","last_unstale":"2024-06-03T09:06:13.703685+0000","last_undegraded":"2024-06-03T09:06:13.703685+0000","last_fullsized":"2024-06-03T09:06:13.703685+0000","mapping_epoch":20,"log_start":"0'0","ondisk_log_start":"0'0","created":16,"last_epoch_clean":22,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-06-03T08:36:15.542635+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-06-03T08:36:15.542635+0000","last_clean_scrub_stamp":"2024-06-03T08:36:15.542635+0000","objects_scrubbed":0,"log_size":76,"log_dups_size":0,"ondisk_log_size":76,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-06-04T12:41:53.731360+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":459280,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":96,"num_read_kb":82,"num_write":113,"num_write_kb":1372,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2,"num_bytes_recovered":397840,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,5],"acting":[1,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"17.10","version":"0'0","reported_seq":944,"reported_epoch":1009,"state":"active+clean","last_fresh":"2024-06-03T09:06:14.422035+0000","last_change":"2024-06-03T08:38:15.399189+0000","last_active":"2024-06-03T09:06:14.422035+0000","last_peered":"2024-06-03T09:06:14.422035+0000","last_clean":"2024-06-03T09:06:14.422035+0000","last_became_active":"2024-06-03T08:38:15.337723+0000","last_became_peered":"2024-06-03T08:38:15.337723+0000","last_unstale":"2024-06-03T09:06:14.422035+0000","last_undegraded":"2024-06-03T09:06:14.422035+0000","last_fullsized":"2024-06-03T09:06:14.422035+0000","mapping_epoch":28,"log_start":"0'0","ondisk_log_start":"0'0","created":28,"last_epoch_clean":42,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-06-03T08:37:55.614497+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-06-03T08:37:55.614497+0000","last_clean_scrub_stamp":"2024-06-03T08:37:55.614497+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-06-04T09:29:46.427317+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,3],"acting":[4,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"17.1a","version":"0'0","reported_seq":960,"reported_epoch":1009,"state":"active+undersized","last_fresh":"2024-06-03T09:06:14.431224+0000","last_change":"2024-06-03T09:06:03.390734+0000","last_active":"2024-06-03T09:06:14.431224+0000","last_peered":"2024-06-03T09:06:14.431224+0000","last_clean":"2024-06-03T09:05:20.322313+0000","last_became_active":"2024-06-03T09:06:03.390734+0000","last_became_peered":"2024-06-03T09:06:03.390734+0000","last_unstale":"2024-06-03T09:06:14.431224+0000","last_undegraded":"2024-06-03T09:06:14.431224+0000","last_fullsized":"2024-06-03T09:06:03.343197+0000","mapping_epoch":999,"log_start":"0'0","ondisk_log_start":"0'0","created":28,"last_epoch_clean":254,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-06-03T08:37:55.614497+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-06-03T08:37:55.614497+0000","last_clean_scrub_stamp":"2024-06-03T08:37:55.614497+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-06-04T15:56:47.628875+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4],"acting":[4],"avail_no_missing":["4"],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"17.1b","version":"0'0","reported_seq":935,"reported_epoch":1009,"state":"active+clean","last_fresh":"2024-06-03T09:06:13.663279+0000","last_change":"2024-06-03T08:38:03.115748+0000","last_active":"2024-06-03T09:06:13.663279+0000","last_peered":"2024-06-03T09:06:13.663279+0000","last_clean":"2024-06-03T09:06:13.663279+0000","last_became_active":"2024-06-03T08:38:03.110533+0000","last_became_peered":"2024-06-03T08:38:03.110533+0000","last_unstale":"2024-06-03T09:06:13.663279+0000","last_undegraded":"2024-06-03T09:06:13.663279+0000","last_fullsized":"2024-06-03T09:06:13.663279+0000","mapping_epoch":28,"log_start":"0'0","ondisk_log_start":"0'0","created":28,"last_epoch_clean":32,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-06-03T08:37:55.614497+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-06-03T08:37:55.614497+0000","last_clean_scrub_stamp":"2024-06-03T08:37:55.614497+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-06-04T12:36:47.471609+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,7],"acting":[3,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"17.18","version":"0'0","reported_seq":944,"reported_epoch":1009,"state":"active+clean","last_fresh":"2024-06-03T09:06:13.924799+0000","last_change":"2024-06-03T08:38:04.632864+0000","last_active":"2024-06-03T09:06:13.924799+0000","last_peered":"2024-06-03T09:06:13.924799+0000","last_clean":"2024-06-03T09:06:13.924799+0000","last_became_active":"2024-06-03T08:38:04.627856+0000","last_became_peered":"2024-06-03T08:38:04.627856+0000","last_unstale":"2024-06-03T09:06:13.924799+0000","last_undegraded":"2024-06-03T09:06:13.924799+0000","last_fullsized":"2024-06-03T09:06:13.924799+0000","mapping_epoch":28,"log_start":"0'0","ondisk_log_start":"0'0","created":28,"last_epoch_clean":33,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-06-03T08:37:55.614497+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-06-03T08:37:55.614497+0000","last_clean_scrub_stamp":"2024-06-03T08:37:55.614497+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-06-04T19:55:51.181598+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,4],"acting":[7,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"17.19","version":"0'0","reported_seq":934,"reported_epoch":1009,"state":"active+clean","last_fresh":"2024-06-03T09:06:13.652943+0000","last_change":"2024-06-03T08:38:01.839491+0000","last_active":"2024-06-03T09:06:13.652943+0000","last_peered":"2024-06-03T09:06:13.652943+0000","last_clean":"2024-06-03T09:06:13.652943+0000","last_became_active":"2024-06-03T08:38:01.829539+0000","last_became_peered":"2024-06-03T08:38:01.829539+0000","last_unstale":"2024-06-03T09:06:13.652943+0000","last_undegraded":"2024-06-03T09:06:13.652943+0000","last_fullsized":"2024-06-03T09:06:13.652943+0000","mapping_epoch":28,"log_start":"0'0","ondisk_log_start":"0'0","created":28,"last_epoch_clean":30,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-06-03T08:37:55.614497+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-06-03T08:37:55.614497+0000","last_clean_scrub_stamp":"2024-06-03T08:37:55.614497+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-06-04T09:10:43.860229+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,2],"acting":[3,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"17.1e","version":"0'0","reported_seq":148,"reported_epoch":1009,"state":"active+undersized","last_fresh":"2024-06-03T09:06:14.445917+0000","last_change":"2024-06-03T09:06:03.406181+0000","last_active":"2024-06-03T09:06:14.445917+0000","last_peered":"2024-06-03T09:06:14.445917+0000","last_clean":"2024-06-03T08:37:55.614497+0000","last_became_active":"2024-06-03T09:06:03.406181+0000","last_became_peered":"2024-06-03T09:06:03.406181+0000","last_unstale":"2024-06-03T09:06:14.445917+0000","last_undegraded":"2024-06-03T09:06:14.445917+0000","last_fullsized":"2024-06-03T09:06:03.363739+0000","mapping_epoch":999,"log_start":"0'0","ondisk_log_start":"0'0","created":28,"last_epoch_clean":255,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-06-03T08:37:55.614497+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-06-03T08:37:55.614497+0000","last_clean_scrub_stamp":"2024-06-03T08:37:55.614497+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-06-04T11:45:24.486600+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4],"acting":[4],"avail_no_missing":["4"],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"17.1f","version":"0'0","reported_seq":941,"reported_epoch":1009,"state":"active+clean","last_fresh":"2024-06-03T09:06:14.450053+0000","last_change":"2024-06-03T08:38:04.383538+0000","last_active":"2024-06-03T09:06:14.450053+0000","last_peered":"2024-06-03T09:06:14.450053+0000","last_clean":"2024-06-03T09:06:14.450053+0000","last_became_active":"2024-06-03T08:38:04.367278+0000","last_became_peered":"2024-06-03T08:38:04.367278+0000","last_unstale":"2024-06-03T09:06:14.450053+0000","last_undegraded":"2024-06-03T09:06:14.450053+0000","last_fullsized":"2024-06-03T09:06:14.450053+0000","mapping_epoch":28,"log_start":"0'0","ondisk_log_start":"0'0","created":28,"last_epoch_clean":32,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-06-03T08:37:55.614497+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-06-03T08:37:55.614497+0000","last_clean_scrub_stamp":"2024-06-03T08:37:55.614497+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-06-04T10:39:26.552282+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,2],"acting":[0,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"17.1c","version":"0'0","reported_seq":943,"reported_epoch":1009,"state":"active+undersized","last_fresh":"2024-06-03T09:06:13.719240+0000","last_change":"2024-06-03T09:06:03.382210+0000","last_active":"2024-06-03T09:06:13.719240+0000","last_peered":"2024-06-03T09:06:13.719240+0000","last_clean":"2024-06-03T09:04:43.199660+0000","last_became_active":"2024-06-03T09:06:03.382210+0000","last_became_peered":"2024-06-03T09:06:03.382210+0000","last_unstale":"2024-06-03T09:06:13.719240+0000","last_undegraded":"2024-06-03T09:06:13.719240+0000","last_fullsized":"2024-06-03T09:06:03.277874+0000","mapping_epoch":999,"log_start":"0'0","ondisk_log_start":"0'0","created":28,"last_epoch_clean":255,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-06-03T08:37:55.614497+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-06-03T08:37:55.614497+0000","last_clean_scrub_stamp":"2024-06-03T08:37:55.614497+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-06-04T12:18:50.013884+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5],"acting":[5],"avail_no_missing":["5"],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"17.1d","version":"0'0","reported_seq":923,"reported_epoch":1009,"state":"active+clean","last_fresh":"2024-06-03T09:06:13.715906+0000","last_change":"2024-06-03T08:38:06.794630+0000","last_active":"2024-06-03T09:06:13.715906+0000","last_peered":"2024-06-03T09:06:13.715906+0000","last_clean":"2024-06-03T09:06:13.715906+0000","last_became_active":"2024-06-03T08:38:06.592317+0000","last_became_peered":"2024-06-03T08:38:06.592317+0000","last_unstale":"2024-06-03T09:06:13.715906+0000","last_undegraded":"2024-06-03T09:06:13.715906+0000","last_fullsized":"2024-06-03T09:06:13.715906+0000","mapping_epoch":28,"log_start":"0'0","ondisk_log_start":"0'0","created":28,"last_epoch_clean":33,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-06-03T08:37:55.614497+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-06-03T08:37:55.614497+0000","last_clean_scrub_stamp":"2024-06-03T08:37:55.614497+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-06-04T12:33:30.785369+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,7],"acting":[1,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"17.2","version":"0'0","reported_seq":925,"reported_epoch":1009,"state":"active+clean","last_fresh":"2024-06-03T09:06:13.708688+0000","last_change":"2024-06-03T08:38:14.981410+0000","last_active":"2024-06-03T09:06:13.708688+0000","last_peered":"2024-06-03T09:06:13.708688+0000","last_clean":"2024-06-03T09:06:13.708688+0000","last_became_active":"2024-06-03T08:38:14.973825+0000","last_became_peered":"2024-06-03T08:38:14.973825+0000","last_unstale":"2024-06-03T09:06:13.708688+0000","last_undegraded":"2024-06-03T09:06:13.708688+0000","last_fullsized":"2024-06-03T09:06:13.708688+0000","mapping_epoch":28,"log_start":"0'0","ondisk_log_start":"0'0","created":28,"last_epoch_clean":41,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-06-03T08:37:55.614497+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-06-03T08:37:55.614497+0000","last_clean_scrub_stamp":"2024-06-03T08:37:55.614497+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-06-04T17:01:05.518613+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,4],"acting":[1,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"17.3","version":"0'0","reported_seq":943,"reported_epoch":1009,"state":"active+clean","last_fresh":"2024-06-03T09:06:14.437636+0000","last_change":"2024-06-03T08:38:14.619745+0000","last_active":"2024-06-03T09:06:14.437636+0000","last_peered":"2024-06-03T09:06:14.437636+0000","last_clean":"2024-06-03T09:06:14.437636+0000","last_became_active":"2024-06-03T08:38:14.510665+0000","last_became_peered":"2024-06-03T08:38:14.510665+0000","last_unstale":"2024-06-03T09:06:14.437636+0000","last_undegraded":"2024-06-03T09:06:14.437636+0000","last_fullsized":"2024-06-03T09:06:14.437636+0000","mapping_epoch":28,"log_start":"0'0","ondisk_log_start":"0'0","created":28,"last_epoch_clean":42,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-06-03T08:37:55.614497+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-06-03T08:37:55.614497+0000","last_clean_scrub_stamp":"2024-06-03T08:37:55.614497+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-06-04T12:03:47.392064+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,7],"acting":[0,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"17.0","version":"0'0","reported_seq":944,"reported_epoch":1009,"state":"active+clean","last_fresh":"2024-06-03T09:06:14.427123+0000","last_change":"2024-06-03T08:38:04.237118+0000","last_active":"2024-06-03T09:06:14.427123+0000","last_peered":"2024-06-03T09:06:14.427123+0000","last_clean":"2024-06-03T09:06:14.427123+0000","last_became_active":"2024-06-03T08:38:04.231954+0000","last_became_peered":"2024-06-03T08:38:04.231954+0000","last_unstale":"2024-06-03T09:06:14.427123+0000","last_undegraded":"2024-06-03T09:06:14.427123+0000","last_fullsized":"2024-06-03T09:06:14.427123+0000","mapping_epoch":28,"log_start":"0'0","ondisk_log_start":"0'0","created":28,"last_epoch_clean":32,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-06-03T08:37:55.614497+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-06-03T08:37:55.614497+0000","last_clean_scrub_stamp":"2024-06-03T08:37:55.614497+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-06-04T09:56:10.741637+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,2],"acting":[0,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"17.1","version":"65'7","reported_seq":959,"reported_epoch":1009,"state":"active+undersized+degraded","last_fresh":"2024-06-03T09:06:13.709209+0000","last_change":"2024-06-03T09:06:03.370660+0000","last_active":"2024-06-03T09:06:13.709209+0000","last_peered":"2024-06-03T09:06:13.709209+0000","last_clean":"2024-06-03T09:05:03.733864+0000","last_became_active":"2024-06-03T09:06:03.370660+0000","last_became_peered":"2024-06-03T09:06:03.370660+0000","last_unstale":"2024-06-03T09:06:13.709209+0000","last_undegraded":"2024-06-03T09:06:03.266076+0000","last_fullsized":"2024-06-03T09:06:03.264525+0000","mapping_epoch":999,"log_start":"0'0","ondisk_log_start":"0'0","created":28,"last_epoch_clean":255,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-06-03T08:37:55.614497+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-06-03T08:37:55.614497+0000","last_clean_scrub_stamp":"2024-06-03T08:37:55.614497+0000","objects_scrubbed":0,"log_size":7,"log_dups_size":0,"ondisk_log_size":7,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-06-04T11:55:18.194283+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":5,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":1,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":6,"num_read_kb":6,"num_write":7,"num_write_kb":7,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5],"acting":[5],"avail_no_missing":["5"],"object_location_counts":[{"shards":"5","objects":1}],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]}],"pool_stats":[{"poolid":2,"num_pg":8,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":14,"acting":14,"num_store_stats":7},{"poolid":1,"num_pg":1,"stat_sum":{"num_bytes":459280,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":96,"num_read_kb":82,"num_write":113,"num_write_kb":1372,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2,"num_bytes_recovered":397840,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":802816,"data_stored":795680,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":76,"ondisk_log_size":76,"up":2,"acting":2,"num_store_stats":2},{"poolid":17,"num_pg":32,"stat_sum":{"num_bytes":5,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":1,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":6,"num_read_kb":6,"num_write":7,"num_write_kb":7,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":8192,"data_stored":10,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":7,"ondisk_log_size":7,"up":52,"acting":52,"num_store_stats":8}],"osd_stats":[{"osd":6,"up_from":253,"seq":1086626726200,"num_pgs":32,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":54772,"kb_used_data":5608,"kb_used_omap":1,"kb_used_meta":49150,"kb_avail":94317068,"statfs":{"total":96636764160,"available":96580677632,"internally_reserved":0,"allocated":5742592,"data_stored":2767180,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1593,"internal_metadata":50330055},"hb_peers":[0,1,2,3,4,5,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":109,"apply_latency_ms":109,"commit_latency_ns":109000000,"apply_latency_ns":109000000},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Mon Jun 3 09:04:02 2024","interfaces":[{"interface":"back","average":{"1min":1582.4780000000001,"5min":332.72699999999998,"15min":124.113},"min":{"1min":4.4829999999999997,"5min":4.1959999999999997,"15min":4.1959999999999997},"max":{"1min":22766.882000000001,"5min":22766.882000000001,"15min":22766.882000000001},"last":21.917999999999999},{"interface":"front","average":{"1min":1579.5409999999999,"5min":334.94499999999999,"15min":125.84999999999999},"min":{"1min":4.8129999999999997,"5min":4.7539999999999996,"15min":4.4530000000000003},"max":{"1min":22766.289000000001,"5min":22766.289000000001,"15min":22766.289000000001},"last":25.408999999999999}]},{"osd":1,"last update":"Mon Jun 3 09:03:40 2024","interfaces":[{"interface":"back","average":{"1min":36.420999999999999,"5min":77.054000000000002,"15min":132.84800000000001},"min":{"1min":6.0380000000000003,"5min":5.8799999999999999,"15min":5.484},"max":{"1min":589.64099999999996,"5min":4138.1679999999997,"15min":9955.1299999999992},"last":26.84},{"interface":"front","average":{"1min":40.240000000000002,"5min":79.674999999999997,"15min":133.38999999999999},"min":{"1min":8.4589999999999996,"5min":8.4090000000000007,"15min":6.1390000000000002},"max":{"1min":588.60799999999995,"5min":4137.6530000000002,"15min":9827.5509999999995},"last":26.263000000000002}]},{"osd":2,"last update":"Mon Jun 3 09:03:40 2024","interfaces":[{"interface":"back","average":{"1min":12.613,"5min":24.600999999999999,"15min":21.088000000000001},"min":{"1min":6.9740000000000002,"5min":6.9740000000000002,"15min":6.5330000000000004},"max":{"1min":26.076000000000001,"5min":376.24900000000002,"15min":376.24900000000002},"last":24.899000000000001},{"interface":"front","average":{"1min":12.493,"5min":22.207999999999998,"15min":21.219000000000001},"min":{"1min":6.5810000000000004,"5min":6.5110000000000001,"15min":6.0750000000000002},"max":{"1min":29.690999999999999,"5min":271.51100000000002,"15min":434.77300000000002},"last":40.372}]},{"osd":3,"last update":"Mon Jun 3 09:03:40 2024","interfaces":[{"interface":"back","average":{"1min":14.093999999999999,"5min":24.143999999999998,"15min":29.376999999999999},"min":{"1min":10.196999999999999,"5min":6.2919999999999998,"15min":6.1719999999999997},"max":{"1min":24.832000000000001,"5min":197.84899999999999,"15min":1235.5799999999999},"last":24.353999999999999},{"interface":"front","average":{"1min":9.3260000000000005,"5min":18.463999999999999,"15min":27.631},"min":{"1min":6.125,"5min":6.125,"15min":6.125},"max":{"1min":28.166,"5min":200.173,"15min":1157.6780000000001},"last":23.454000000000001}]},{"osd":4,"last update":"Mon Jun 3 09:04:29 2024","interfaces":[{"interface":"back","average":{"1min":10.279,"5min":1253.809,"15min":450.79199999999997},"min":{"1min":7.2300000000000004,"5min":6.093,"15min":6.093},"max":{"1min":15.000999999999999,"5min":28093.096000000001,"15min":28093.096000000001},"last":101.45399999999999},{"interface":"front","average":{"1min":15.693,"5min":1279.759,"15min":463.87099999999998},"min":{"1min":10.551,"5min":7.7480000000000002,"15min":7.7480000000000002},"max":{"1min":27.898,"5min":28323.837,"15min":28323.837},"last":102.262}]},{"osd":5,"last update":"Mon Jun 3 09:04:10 2024","interfaces":[{"interface":"back","average":{"1min":13.728999999999999,"5min":24.670000000000002,"15min":23.815999999999999},"min":{"1min":10.401,"5min":7.516,"15min":7.516},"max":{"1min":25.138000000000002,"5min":320.85000000000002,"15min":445.041},"last":103.52500000000001},{"interface":"front","average":{"1min":13.847,"5min":29.283000000000001,"15min":23.393999999999998},"min":{"1min":7.944,"5min":6.266,"15min":6.266},"max":{"1min":18.817,"5min":530.56700000000001,"15min":530.56700000000001},"last":29.263999999999999}]},{"osd":7,"last update":"Mon Jun 3 09:03:40 2024","interfaces":[{"interface":"back","average":{"1min":20.568999999999999,"5min":30.286999999999999,"15min":27.977},"min":{"1min":13.874000000000001,"5min":10.58,"15min":9.2859999999999996},"max":{"1min":86.959000000000003,"5min":222.07599999999999,"15min":255.08799999999999},"last":37.618000000000002},{"interface":"front","average":{"1min":19.102,"5min":29.550000000000001,"15min":27.763000000000002},"min":{"1min":10.052,"5min":8.6709999999999994,"15min":8.5429999999999993},"max":{"1min":67.968999999999994,"5min":293.62099999999998,"15min":438.14600000000002},"last":111.985}]}]},{"osd":7,"up_from":20,"seq":85899346285,"num_pgs":9,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":58964,"kb_used_data":5704,"kb_used_omap":1,"kb_used_meta":53246,"kb_avail":94312876,"statfs":{"total":96636764160,"available":96576385024,"internally_reserved":0,"allocated":5840896,"data_stored":2740727,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1589,"internal_metadata":54524363},"hb_peers":[0,1,2,3,4,5],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":4,"apply_latency_ms":4,"commit_latency_ns":4000000,"apply_latency_ns":4000000},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Mon Jun 3 09:06:09 2024","interfaces":[{"interface":"back","average":{"1min":14.073,"5min":894.22400000000005,"15min":308.93400000000003},"min":{"1min":5.5709999999999997,"5min":4.218,"15min":4.0599999999999996},"max":{"1min":90.757999999999996,"5min":21729.001,"15min":21729.001},"last":17.491},{"interface":"front","average":{"1min":9.2550000000000008,"5min":906.23500000000001,"15min":314.06},"min":{"1min":5.5789999999999997,"5min":4.8810000000000002,"15min":4.8810000000000002},"max":{"1min":25.937000000000001,"5min":21899.732,"15min":21899.732},"last":6.1420000000000003}]},{"osd":1,"last update":"Mon Jun 3 09:06:09 2024","interfaces":[{"interface":"back","average":{"1min":15.182,"5min":69.183999999999997,"15min":219.173},"min":{"1min":7.3639999999999999,"5min":6.6200000000000001,"15min":6.1319999999999997},"max":{"1min":76.926000000000002,"5min":2795.1329999999998,"15min":18373.257000000001},"last":7.444},{"interface":"front","average":{"1min":21.495999999999999,"5min":74.703999999999994,"15min":225.285},"min":{"1min":8.8399999999999999,"5min":8.8399999999999999,"15min":8.5129999999999999},"max":{"1min":94.977999999999994,"5min":2803.3670000000002,"15min":18374.601999999999},"last":12.337999999999999}]},{"osd":2,"last update":"Mon Jun 3 09:06:09 2024","interfaces":[{"interface":"back","average":{"1min":19.164000000000001,"5min":26.902000000000001,"15min":22.920999999999999},"min":{"1min":11.207000000000001,"5min":8.5030000000000001,"15min":8.5030000000000001},"max":{"1min":91.319999999999993,"5min":333.99200000000002,"15min":333.99200000000002},"last":11.518000000000001},{"interface":"front","average":{"1min":11.125999999999999,"5min":18.132000000000001,"15min":16.353000000000002},"min":{"1min":6.202,"5min":5.8369999999999997,"15min":5.5069999999999997},"max":{"1min":28.649000000000001,"5min":226.18100000000001,"15min":226.18100000000001},"last":9.077}]},{"osd":3,"last update":"Mon Jun 3 09:06:09 2024","interfaces":[{"interface":"back","average":{"1min":22.291,"5min":26.289999999999999,"15min":31.074999999999999},"min":{"1min":11.714,"5min":6.8959999999999999,"15min":6.3760000000000003},"max":{"1min":83.102000000000004,"5min":316.64400000000001,"15min":2039.9059999999999},"last":10.698},{"interface":"front","average":{"1min":13.324999999999999,"5min":17.963000000000001,"15min":25.969999999999999},"min":{"1min":6.1200000000000001,"5min":5.2089999999999996,"15min":5.2089999999999996},"max":{"1min":49.908999999999999,"5min":233.542,"15min":2037.5550000000001},"last":9.9540000000000006}]},{"osd":4,"last update":"Mon Jun 3 09:05:28 2024","interfaces":[{"interface":"back","average":{"1min":22.209,"5min":1379.51,"15min":1803.278},"min":{"1min":5.9089999999999998,"5min":5.9089999999999998,"15min":5.9089999999999998},"max":{"1min":224.54300000000001,"5min":30073.917000000001,"15min":62560.868000000002},"last":16.802},{"interface":"front","average":{"1min":22.712,"5min":1401.0719999999999,"15min":1806.759},"min":{"1min":7.6449999999999996,"5min":6.7930000000000001,"15min":5.9340000000000002},"max":{"1min":223.58600000000001,"5min":30348.021000000001,"15min":62451.446000000004},"last":16.009}]},{"osd":5,"last update":"Mon Jun 3 09:06:02 2024","interfaces":[{"interface":"back","average":{"1min":24.372,"5min":26.856000000000002,"15min":23.571999999999999},"min":{"1min":12.278,"5min":9.2460000000000004,"15min":6.5650000000000004},"max":{"1min":183.64099999999999,"5min":317.572,"15min":317.572},"last":13.161},{"interface":"front","average":{"1min":21.259,"5min":23.364000000000001,"15min":21.193999999999999},"min":{"1min":7.8499999999999996,"5min":7.0999999999999996,"15min":7.008},"max":{"1min":184.53399999999999,"5min":306.60599999999999,"15min":306.60599999999999},"last":15.401}]},{"osd":6,"last update":"Mon Jun 3 09:04:31 2024","interfaces":[{"interface":"back","average":{"1min":44.061,"5min":26.821000000000002,"15min":91.176000000000002},"min":{"1min":9.0800000000000001,"5min":7.9390000000000001,"15min":7.9390000000000001},"max":{"1min":268.029,"5min":308.916,"15min":8734.4639999999999},"last":6.6769999999999996},{"interface":"front","average":{"1min":46.643000000000001,"5min":23.879999999999999,"15min":88.765000000000001},"min":{"1min":8.8119999999999994,"5min":6.9240000000000004,"15min":6.742},"max":{"1min":305.64600000000002,"5min":305.64600000000002,"15min":8735.5319999999992},"last":12.327}]}]},{"osd":5,"up_from":20,"seq":85899346259,"num_pgs":9,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":59360,"kb_used_data":6100,"kb_used_omap":1,"kb_used_meta":53246,"kb_avail":94312480,"statfs":{"total":96636764160,"available":96575979520,"internally_reserved":0,"allocated":6246400,"data_stored":3138572,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1594,"internal_metadata":54524358},"hb_peers":[0,1,2,3,4,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":5,"apply_latency_ms":5,"commit_latency_ns":5000000,"apply_latency_ns":5000000},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Mon Jun 3 09:06:06 2024","interfaces":[{"interface":"back","average":{"1min":7.468,"5min":1703.037,"15min":580.51900000000001},"min":{"1min":4.6509999999999998,"5min":4.6509999999999998,"15min":4.4020000000000001},"max":{"1min":14.711,"5min":26433.764999999999,"15min":26433.764999999999},"last":6.1859999999999999},{"interface":"front","average":{"1min":9.9429999999999996,"5min":1725.27,"15min":587.87800000000004},"min":{"1min":5.5999999999999996,"5min":5.1319999999999997,"15min":4.8170000000000002},"max":{"1min":13.590999999999999,"5min":26704.698,"15min":26704.698},"last":7.6970000000000001}]},{"osd":1,"last update":"Mon Jun 3 09:06:00 2024","interfaces":[{"interface":"back","average":{"1min":10.862,"5min":67.731999999999999,"15min":217.364},"min":{"1min":7.1769999999999996,"5min":6.0890000000000004,"15min":5.4930000000000003},"max":{"1min":16.998999999999999,"5min":3923.4090000000001,"15min":18811.696},"last":12.917999999999999},{"interface":"front","average":{"1min":10.417,"5min":65.141999999999996,"15min":220.75700000000001},"min":{"1min":5.3209999999999997,"5min":5.3209999999999997,"15min":5.0209999999999999},"max":{"1min":15.821999999999999,"5min":4019.6129999999998,"15min":18968.365000000002},"last":8.6590000000000007}]},{"osd":2,"last update":"Mon Jun 3 09:06:00 2024","interfaces":[{"interface":"back","average":{"1min":11.366,"5min":24.18,"15min":21.137},"min":{"1min":5.6369999999999996,"5min":5.6369999999999996,"15min":5.0510000000000002},"max":{"1min":15.529999999999999,"5min":332.48200000000003,"15min":332.48200000000003},"last":14.346},{"interface":"front","average":{"1min":10.089,"5min":23.117000000000001,"15min":24.050000000000001},"min":{"1min":6.6769999999999996,"5min":6.6769999999999996,"15min":5.8739999999999997},"max":{"1min":15.821,"5min":353.03199999999998,"15min":353.03199999999998},"last":8.2629999999999999}]},{"osd":3,"last update":"Mon Jun 3 09:06:00 2024","interfaces":[{"interface":"back","average":{"1min":13.472,"5min":24.283999999999999,"15min":31.666},"min":{"1min":10.111000000000001,"5min":7.4720000000000004,"15min":7.4210000000000003},"max":{"1min":16.271999999999998,"5min":353.57499999999999,"15min":2855.8519999999999},"last":13.637},{"interface":"front","average":{"1min":9.5899999999999999,"5min":15.875,"15min":25.834},"min":{"1min":6.6609999999999996,"5min":5.7729999999999997,"15min":5.1909999999999998},"max":{"1min":15.112,"5min":203.804,"15min":2869.7860000000001},"last":10.704000000000001}]},{"osd":4,"last update":"Mon Jun 3 09:05:27 2024","interfaces":[{"interface":"back","average":{"1min":19.155000000000001,"5min":1084.961,"15min":1346.9949999999999},"min":{"1min":7.6130000000000004,"5min":7.6130000000000004,"15min":6.1340000000000003},"max":{"1min":156.55199999999999,"5min":28619.825000000001,"15min":57999.453999999998},"last":10.016},{"interface":"front","average":{"1min":28.068999999999999,"5min":1087.106,"15min":1330.289},"min":{"1min":8.3949999999999996,"5min":6.5030000000000001,"15min":6.5030000000000001},"max":{"1min":254.91300000000001,"5min":28621.294000000002,"15min":57434.631999999998},"last":15.166}]},{"osd":6,"last update":"Mon Jun 3 09:03:57 2024","interfaces":[{"interface":"back","average":{"1min":23.536000000000001,"5min":36.457000000000001,"15min":83.799999999999997},"min":{"1min":9.4800000000000004,"5min":9.4800000000000004,"15min":8.0630000000000006},"max":{"1min":89.442999999999998,"5min":508.77999999999997,"15min":9193.5169999999998},"last":157.79300000000001},{"interface":"front","average":{"1min":18.891999999999999,"5min":22.100999999999999,"15min":75.489999999999995},"min":{"1min":5.923,"5min":5.923,"15min":5.923},"max":{"1min":78.424999999999997,"5min":464.62,"15min":9191.4429999999993},"last":148.09}]},{"osd":7,"last update":"Mon Jun 3 09:06:00 2024","interfaces":[{"interface":"back","average":{"1min":13.257,"5min":20.917999999999999,"15min":21.088999999999999},"min":{"1min":10.475,"5min":6.7629999999999999,"15min":6.7629999999999999},"max":{"1min":17.245000000000001,"5min":211.22999999999999,"15min":211.22999999999999},"last":11.826000000000001},{"interface":"front","average":{"1min":12.872999999999999,"5min":20.587,"15min":24.021000000000001},"min":{"1min":8.5760000000000005,"5min":7.7709999999999999,"15min":6.8570000000000002},"max":{"1min":18.359999999999999,"5min":223.191,"15min":286.69499999999999},"last":11.209}]}]},{"osd":4,"up_from":20,"seq":85899346267,"num_pgs":12,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":63060,"kb_used_data":5704,"kb_used_omap":1,"kb_used_meta":57342,"kb_avail":94308780,"statfs":{"total":96636764160,"available":96572190720,"internally_reserved":0,"allocated":5840896,"data_stored":2740727,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1594,"internal_metadata":58718662},"hb_peers":[0,1,2,3,5,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":4,"apply_latency_ms":4,"commit_latency_ns":4000000,"apply_latency_ns":4000000},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Mon Jun 3 09:05:34 2024","interfaces":[{"interface":"back","average":{"1min":12.416,"5min":1099.4649999999999,"15min":382.15199999999999},"min":{"1min":4.5279999999999996,"5min":4.5279999999999996,"15min":4.3220000000000001},"max":{"1min":123.878,"5min":21726.743999999999,"15min":21726.743999999999},"last":4.6900000000000004},{"interface":"front","average":{"1min":16.462,"5min":1108.894,"15min":387.70299999999997},"min":{"1min":6.4089999999999998,"5min":5.8380000000000001,"15min":5.4909999999999997},"max":{"1min":130.185,"5min":21727.993999999999,"15min":21727.993999999999},"last":7.4349999999999996}]},{"osd":1,"last update":"Mon Jun 3 09:05:34 2024","interfaces":[{"interface":"back","average":{"1min":14.727,"5min":56.755000000000003,"15min":341.09399999999999},"min":{"1min":4.6529999999999996,"5min":4.1970000000000001,"15min":4.1970000000000001},"max":{"1min":126.899,"5min":3346.4389999999999,"15min":22256.098000000002},"last":8.7590000000000003},{"interface":"front","average":{"1min":13.472,"5min":57.503,"15min":340.303},"min":{"1min":5.0590000000000002,"5min":5.0590000000000002,"15min":4.2640000000000002},"max":{"1min":126.096,"5min":3347.181,"15min":22247.317999999999},"last":7.7999999999999998}]},{"osd":2,"last update":"Mon Jun 3 09:05:34 2024","interfaces":[{"interface":"back","average":{"1min":26.088999999999999,"5min":23.866,"15min":20.620999999999999},"min":{"1min":7.0869999999999997,"5min":6.8239999999999998,"15min":6.3150000000000004},"max":{"1min":337.60399999999998,"5min":393.58600000000001,"15min":393.58600000000001},"last":9.5820000000000007},{"interface":"front","average":{"1min":15.800000000000001,"5min":22.584,"15min":23.245999999999999},"min":{"1min":6.0629999999999997,"5min":6.0629999999999997,"15min":6.0629999999999997},"max":{"1min":127.913,"5min":394.471,"15min":394.471},"last":10.23}]},{"osd":3,"last update":"Mon Jun 3 09:05:34 2024","interfaces":[{"interface":"back","average":{"1min":17.544,"5min":19.143999999999998,"15min":18.742000000000001},"min":{"1min":5.8719999999999999,"5min":5.4100000000000001,"15min":5.4000000000000004},"max":{"1min":127.43600000000001,"5min":322.63,"15min":322.63},"last":11.765000000000001},{"interface":"front","average":{"1min":21.327000000000002,"5min":23.559000000000001,"15min":23.797000000000001},"min":{"1min":10.023,"5min":7.484,"15min":7.484},"max":{"1min":204.517,"5min":392.67500000000001,"15min":392.67500000000001},"last":11.102}]},{"osd":5,"last update":"Mon Jun 3 09:05:34 2024","interfaces":[{"interface":"back","average":{"1min":24.274999999999999,"5min":21.274999999999999,"15min":20.988},"min":{"1min":6.7489999999999997,"5min":6.7489999999999997,"15min":6.7489999999999997},"max":{"1min":268.10199999999998,"5min":344.23399999999998,"15min":344.23399999999998},"last":13.635},{"interface":"front","average":{"1min":26.361999999999998,"5min":25.218,"15min":26.084},"min":{"1min":12.228,"5min":8.5180000000000007,"15min":8.5180000000000007},"max":{"1min":264.75,"5min":345.18700000000001,"15min":345.18700000000001},"last":14.532}]},{"osd":6,"last update":"Mon Jun 3 09:04:32 2024","interfaces":[{"interface":"back","average":{"1min":15.548,"5min":32.957999999999998,"15min":152.738},"min":{"1min":9.141,"5min":9.141,"15min":8.2319999999999993},"max":{"1min":52.57,"5min":395.238,"15min":12496.335999999999},"last":208.90000000000001},{"interface":"front","average":{"1min":12.680999999999999,"5min":25.428000000000001,"15min":149.30600000000001},"min":{"1min":6.4379999999999997,"5min":5.6319999999999997,"15min":5.6319999999999997},"max":{"1min":46.283000000000001,"5min":377.74299999999999,"15min":12616.297},"last":205.88900000000001}]},{"osd":7,"last update":"Mon Jun 3 09:05:34 2024","interfaces":[{"interface":"back","average":{"1min":18.295999999999999,"5min":19.585999999999999,"15min":21.062000000000001},"min":{"1min":6.2110000000000003,"5min":6.2110000000000003,"15min":5.8010000000000002},"max":{"1min":207.25299999999999,"5min":358.94900000000001,"15min":358.94900000000001},"last":12.619},{"interface":"front","average":{"1min":23.712,"5min":24.024999999999999,"15min":24.640000000000001},"min":{"1min":11.824999999999999,"5min":7.7789999999999999,"15min":7.7789999999999999},"max":{"1min":209.32900000000001,"5min":346.25200000000001,"15min":380.06900000000002},"last":15.253}]}]},{"osd":2,"up_from":20,"seq":85899346287,"num_pgs":11,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":58964,"kb_used_data":5704,"kb_used_omap":1,"kb_used_meta":53246,"kb_avail":94312876,"statfs":{"total":96636764160,"available":96576385024,"internally_reserved":0,"allocated":5840896,"data_stored":2740727,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1594,"internal_metadata":54524358},"hb_peers":[0,1,3,4,5,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":4,"apply_latency_ms":4,"commit_latency_ns":4000000,"apply_latency_ns":4000000},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Mon Jun 3 09:05:21 2024","interfaces":[{"interface":"back","average":{"1min":16.364000000000001,"5min":947.59799999999996,"15min":326.58999999999997},"min":{"1min":6.0830000000000002,"5min":4.7050000000000001,"15min":4.1420000000000003},"max":{"1min":157.60900000000001,"5min":23465.18,"15min":23465.18},"last":6.8499999999999996},{"interface":"front","average":{"1min":12.276,"5min":945.29899999999998,"15min":326.78399999999999},"min":{"1min":5.1989999999999998,"5min":4.7439999999999998,"15min":4.6950000000000003},"max":{"1min":102.557,"5min":23465.966,"15min":23465.966},"last":5.8600000000000003}]},{"osd":1,"last update":"Mon Jun 3 09:05:21 2024","interfaces":[{"interface":"back","average":{"1min":17.803999999999998,"5min":77.658000000000001,"15min":294.96899999999999},"min":{"1min":7.2130000000000001,"5min":5.4199999999999999,"15min":5.4199999999999999},"max":{"1min":159.624,"5min":4220.5290000000005,"15min":18954.987000000001},"last":10.423999999999999},{"interface":"front","average":{"1min":21.542999999999999,"5min":83.528999999999996,"15min":303.11700000000002},"min":{"1min":8.8200000000000003,"5min":7.5949999999999998,"15min":7.5949999999999998},"max":{"1min":170.74299999999999,"5min":4325.9489999999996,"15min":19143.834999999999},"last":11.566000000000001}]},{"osd":3,"last update":"Mon Jun 3 09:05:21 2024","interfaces":[{"interface":"back","average":{"1min":19.164999999999999,"5min":13.284000000000001,"15min":28.751999999999999},"min":{"1min":7.3710000000000004,"5min":5.9290000000000003,"15min":5.2450000000000001},"max":{"1min":118.913,"5min":118.913,"15min":2594.8150000000001},"last":8.3040000000000003},{"interface":"front","average":{"1min":22.094000000000001,"5min":17.244,"15min":32.173999999999999},"min":{"1min":9.516,"5min":8.0760000000000005,"15min":6.0780000000000003},"max":{"1min":156.34700000000001,"5min":156.34700000000001,"15min":2706.5120000000002},"last":13.273999999999999}]},{"osd":4,"last update":"Mon Jun 3 09:05:57 2024","interfaces":[{"interface":"back","average":{"1min":10.573,"5min":1266.894,"15min":438.154},"min":{"1min":8.6379999999999999,"5min":7.8789999999999996,"15min":6.2039999999999997},"max":{"1min":15.105,"5min":26660.861000000001,"15min":26660.861000000001},"last":9.673},{"interface":"front","average":{"1min":12.407999999999999,"5min":1266.7460000000001,"15min":435.75400000000002},"min":{"1min":10.266999999999999,"5min":10.266999999999999,"15min":6.2720000000000002},"max":{"1min":15.228,"5min":26616.243999999999,"15min":26616.243999999999},"last":14.26}]},{"osd":5,"last update":"Mon Jun 3 09:05:21 2024","interfaces":[{"interface":"back","average":{"1min":25.632000000000001,"5min":20.437999999999999,"15min":21.472999999999999},"min":{"1min":9.2159999999999993,"5min":9.2159999999999993,"15min":6.2050000000000001},"max":{"1min":252.31100000000001,"5min":252.31100000000001,"15min":252.31100000000001},"last":14.960000000000001},{"interface":"front","average":{"1min":24.824000000000002,"5min":18.521000000000001,"15min":21.027000000000001},"min":{"1min":8.0449999999999999,"5min":7.4850000000000003,"15min":6.8810000000000002},"max":{"1min":258.31700000000001,"5min":258.31700000000001,"15min":258.31700000000001},"last":12.413}]},{"osd":6,"last update":"Mon Jun 3 09:04:28 2024","interfaces":[{"interface":"back","average":{"1min":14.063000000000001,"5min":24.850000000000001,"15min":106.755},"min":{"1min":9.0820000000000007,"5min":8.9649999999999999,"15min":8.9649999999999999},"max":{"1min":24.666,"5min":188.06700000000001,"15min":9521.0010000000002},"last":183.60900000000001},{"interface":"front","average":{"1min":13.933999999999999,"5min":21.081,"15min":106.142},"min":{"1min":5.8719999999999999,"5min":5.8719999999999999,"15min":5.8719999999999999},"max":{"1min":117.551,"5min":193.887,"15min":9585.6620000000003},"last":184.20599999999999}]},{"osd":7,"last update":"Mon Jun 3 09:05:21 2024","interfaces":[{"interface":"back","average":{"1min":25.097000000000001,"5min":22.579999999999998,"15min":24.388000000000002},"min":{"1min":7.8959999999999999,"5min":6.6130000000000004,"15min":6.6130000000000004},"max":{"1min":242.56800000000001,"5min":242.56800000000001,"15min":242.56800000000001},"last":15.646000000000001},{"interface":"front","average":{"1min":24.414999999999999,"5min":23.995999999999999,"15min":26.614999999999998},"min":{"1min":12.43,"5min":9.0660000000000007,"15min":9.0660000000000007},"max":{"1min":181.56399999999999,"5min":182.5,"15min":214.33799999999999},"last":16.282}]}]},{"osd":3,"up_from":14,"seq":60129542501,"num_pgs":11,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":58964,"kb_used_data":5704,"kb_used_omap":1,"kb_used_meta":53246,"kb_avail":94312876,"statfs":{"total":96636764160,"available":96576385024,"internally_reserved":0,"allocated":5840896,"data_stored":2740727,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1592,"internal_metadata":54524360},"hb_peers":[0,1,2,4,5,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":4,"apply_latency_ms":4,"commit_latency_ns":4000000,"apply_latency_ns":4000000},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Mon Jun 3 09:05:25 2024","interfaces":[{"interface":"back","average":{"1min":8.4809999999999999,"5min":1064.2270000000001,"15min":367.88600000000002},"min":{"1min":6.0529999999999999,"5min":5.173,"15min":4.6929999999999996},"max":{"1min":12.010999999999999,"5min":23275.167000000001,"15min":23275.167000000001},"last":7.3920000000000003},{"interface":"front","average":{"1min":13.885,"5min":1070.3710000000001,"15min":373.53500000000003},"min":{"1min":9.6679999999999993,"5min":7.2089999999999996,"15min":5.9880000000000004},"max":{"1min":32.426000000000002,"5min":23278.536,"15min":23278.536},"last":13.382}]},{"osd":1,"last update":"Mon Jun 3 09:05:25 2024","interfaces":[{"interface":"back","average":{"1min":11.324,"5min":19.920000000000002,"15min":390.07900000000001},"min":{"1min":6.6379999999999999,"5min":4.8200000000000003,"15min":4.8200000000000003},"max":{"1min":26.449000000000002,"5min":250.72300000000001,"15min":21658.205999999998},"last":5.9640000000000004},{"interface":"front","average":{"1min":9.468,"5min":20.966999999999999,"15min":381.18799999999999},"min":{"1min":5.4969999999999999,"5min":5.2779999999999996,"15min":5.2779999999999996},"max":{"1min":25.643000000000001,"5min":335.33699999999999,"15min":21307.069},"last":8.484}]},{"osd":2,"last update":"Mon Jun 3 09:05:25 2024","interfaces":[{"interface":"back","average":{"1min":13.215,"5min":26.649000000000001,"15min":23.783999999999999},"min":{"1min":7.8710000000000004,"5min":6.4900000000000002,"15min":5.875},"max":{"1min":34.322000000000003,"5min":377.803,"15min":377.803},"last":14.013999999999999},{"interface":"front","average":{"1min":13.105,"5min":22.292999999999999,"15min":20.573},"min":{"1min":7.327,"5min":6.125,"15min":5.9020000000000001},"max":{"1min":32.756999999999998,"5min":371.601,"15min":371.601},"last":9.2789999999999999}]},{"osd":4,"last update":"Mon Jun 3 09:05:27 2024","interfaces":[{"interface":"back","average":{"1min":10.444000000000001,"5min":1211.4970000000001,"15min":1639.1579999999999},"min":{"1min":6.4429999999999996,"5min":6.2560000000000002,"15min":6.0309999999999997},"max":{"1min":21.263000000000002,"5min":30144.939999999999,"15min":62679.622000000003},"last":6.734},{"interface":"front","average":{"1min":11.363,"5min":1219.6400000000001,"15min":1644.127},"min":{"1min":7.8109999999999999,"5min":7.8109999999999999,"15min":6.2439999999999998},"max":{"1min":20.108000000000001,"5min":30180.689999999999,"15min":62681.021999999997},"last":9.9329999999999998}]},{"osd":5,"last update":"Mon Jun 3 09:06:07 2024","interfaces":[{"interface":"back","average":{"1min":12.802,"5min":19.640999999999998,"15min":22.318000000000001},"min":{"1min":9.0890000000000004,"5min":6.8810000000000002,"15min":6.7690000000000001},"max":{"1min":16.745999999999999,"5min":249.88,"15min":327.34899999999999},"last":15.218},{"interface":"front","average":{"1min":15.002000000000001,"5min":24.003,"15min":26.649999999999999},"min":{"1min":11.984,"5min":9.9149999999999991,"15min":8.9879999999999995},"max":{"1min":18.407,"5min":251.47800000000001,"15min":331.86799999999999},"last":16.097000000000001}]},{"osd":6,"last update":"Mon Jun 3 09:04:04 2024","interfaces":[{"interface":"back","average":{"1min":22.815999999999999,"5min":31.513999999999999,"15min":181.74299999999999},"min":{"1min":9.4809999999999999,"5min":7.2240000000000002,"15min":7.2240000000000002},"max":{"1min":160.56999999999999,"5min":380.613,"15min":11893.74},"last":19.359999999999999},{"interface":"front","average":{"1min":25.359999999999999,"5min":30.981000000000002,"15min":179.81399999999999},"min":{"1min":9.8870000000000005,"5min":7.3360000000000003,"15min":6.0899999999999999},"max":{"1min":193.11099999999999,"5min":379.80700000000002,"15min":11893.069},"last":8.202}]},{"osd":7,"last update":"Mon Jun 3 09:05:25 2024","interfaces":[{"interface":"back","average":{"1min":12.177,"5min":21.126000000000001,"15min":22.972000000000001},"min":{"1min":8.6769999999999996,"5min":7.4279999999999999,"15min":7.4279999999999999},"max":{"1min":17.376999999999999,"5min":164.27500000000001,"15min":178.76300000000001},"last":10.628},{"interface":"front","average":{"1min":15.675000000000001,"5min":22.722000000000001,"15min":25.238},"min":{"1min":11.233000000000001,"5min":10.555999999999999,"15min":7.6440000000000001},"max":{"1min":29.280000000000001,"5min":148.13800000000001,"15min":417.22800000000001},"last":11.492000000000001}]}]},{"osd":1,"up_from":14,"seq":60129542499,"num_pgs":6,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":59348,"kb_used_data":6084,"kb_used_omap":1,"kb_used_meta":53246,"kb_avail":94312492,"statfs":{"total":96636764160,"available":96575991808,"internally_reserved":0,"allocated":6230016,"data_stored":3133138,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":54524357},"hb_peers":[0,2,3,4,5,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":9,"apply_latency_ms":9,"commit_latency_ns":9000000,"apply_latency_ns":9000000},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Mon Jun 3 09:06:07 2024","interfaces":[{"interface":"back","average":{"1min":7.7530000000000001,"5min":185.66999999999999,"15min":77.347999999999999},"min":{"1min":5.0229999999999997,"5min":4.9279999999999999,"15min":4.5449999999999999},"max":{"1min":11.896000000000001,"5min":7470.616,"15min":7470.616},"last":6.056},{"interface":"front","average":{"1min":9.9220000000000006,"5min":187.14599999999999,"15min":76.325999999999993},"min":{"1min":5.3159999999999998,"5min":5.3159999999999998,"15min":5.3159999999999998},"max":{"1min":18.338000000000001,"5min":7477.0829999999996,"15min":7477.0829999999996},"last":5.5709999999999997}]},{"osd":2,"last update":"Mon Jun 3 09:05:18 2024","interfaces":[{"interface":"back","average":{"1min":10.954000000000001,"5min":18.295000000000002,"15min":24.126999999999999},"min":{"1min":6.8739999999999997,"5min":5.8650000000000002,"15min":5.8650000000000002},"max":{"1min":21.122,"5min":228,"15min":657.38},"last":6.9740000000000002},{"interface":"front","average":{"1min":13.694000000000001,"5min":24.446999999999999,"15min":32.055999999999997},"min":{"1min":9.9860000000000007,"5min":9.1229999999999993,"15min":7.2160000000000002},"max":{"1min":26.507999999999999,"5min":229.465,"15min":700.06500000000005},"last":12.847}]},{"osd":3,"last update":"Mon Jun 3 09:06:04 2024","interfaces":[{"interface":"back","average":{"1min":8.6110000000000007,"5min":15.733000000000001,"15min":26.646999999999998},"min":{"1min":5.827,"5min":5.4640000000000004,"15min":5.4640000000000004},"max":{"1min":13.105,"5min":121.619,"15min":1676.009},"last":8.2050000000000001},{"interface":"front","average":{"1min":9.5850000000000009,"5min":16.161999999999999,"15min":29.065999999999999},"min":{"1min":6.3630000000000004,"5min":5.7699999999999996,"15min":5.7699999999999996},"max":{"1min":15.166,"5min":132.036,"15min":1766.5840000000001},"last":11.587}]},{"osd":4,"last update":"Mon Jun 3 09:05:31 2024","interfaces":[{"interface":"back","average":{"1min":10.996,"5min":954.46500000000003,"15min":1281.6859999999999},"min":{"1min":7.702,"5min":6.1760000000000002,"15min":6.0970000000000004},"max":{"1min":21.538,"5min":28044.862000000001,"15min":59438.017999999996},"last":9.1210000000000004},{"interface":"front","average":{"1min":13.646000000000001,"5min":961.24599999999998,"15min":1285.26},"min":{"1min":9.7200000000000006,"5min":9.3940000000000001,"15min":6.4640000000000004},"max":{"1min":23.277999999999999,"5min":28045.624,"15min":59439.283000000003},"last":10.907}]},{"osd":5,"last update":"Mon Jun 3 09:06:04 2024","interfaces":[{"interface":"back","average":{"1min":12.478999999999999,"5min":20.228000000000002,"15min":25.997},"min":{"1min":8.6430000000000007,"5min":7.9720000000000004,"15min":7.234},"max":{"1min":17.442,"5min":139.43199999999999,"15min":701.22000000000003},"last":9.7400000000000002},{"interface":"front","average":{"1min":13.507999999999999,"5min":25.760999999999999,"15min":27.553999999999998},"min":{"1min":10.042999999999999,"5min":7.258,"15min":6.242},"max":{"1min":20.611000000000001,"5min":278.63,"15min":598.35000000000002},"last":14.231}]},{"osd":6,"last update":"Mon Jun 3 09:03:54 2024","interfaces":[{"interface":"back","average":{"1min":28.524000000000001,"5min":20.654,"15min":23.878},"min":{"1min":6.0309999999999997,"5min":6.0309999999999997,"15min":6.0309999999999997},"max":{"1min":167.221,"5min":167.221,"15min":704.245},"last":18.721},{"interface":"front","average":{"1min":27.606000000000002,"5min":30.131,"15min":28.628},"min":{"1min":9.8650000000000002,"5min":9.7200000000000006,"15min":8.3580000000000005},"max":{"1min":123.938,"5min":358.23000000000002,"15min":634.05499999999995},"last":173.113}]},{"osd":7,"last update":"Mon Jun 3 09:05:18 2024","interfaces":[{"interface":"back","average":{"1min":21.853999999999999,"5min":24.245999999999999,"15min":32.801000000000002},"min":{"1min":9.5020000000000007,"5min":8.0559999999999992,"15min":8.0559999999999992},"max":{"1min":142.245,"5min":270.95100000000002,"15min":703.07799999999997},"last":13.542},{"interface":"front","average":{"1min":21.547000000000001,"5min":29.977,"15min":30.513999999999999},"min":{"1min":11.167,"5min":8.2089999999999996,"15min":7.7599999999999998},"max":{"1min":119.348,"5min":360.99900000000002,"15min":608.44100000000003},"last":14.951000000000001}]}]},{"osd":0,"up_from":14,"seq":60129542501,"num_pgs":10,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":81108,"kb_used_data":5704,"kb_used_omap":1,"kb_used_meta":75390,"kb_avail":94290732,"statfs":{"total":96636764160,"available":96553709568,"internally_reserved":0,"allocated":5840896,"data_stored":2740727,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1593,"internal_metadata":77199815},"hb_peers":[1,2,3,4,5,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":7,"apply_latency_ms":7,"commit_latency_ns":7000000,"apply_latency_ns":7000000},"alerts":[],"network_ping_times":[{"osd":1,"last update":"Mon Jun 3 09:06:04 2024","interfaces":[{"interface":"back","average":{"1min":6.9960000000000004,"5min":56.997,"15min":339.29700000000003},"min":{"1min":4.4530000000000003,"5min":4.4530000000000003,"15min":4.4100000000000001},"max":{"1min":11.720000000000001,"5min":2414.4749999999999,"15min":20691.453000000001},"last":6.8019999999999996},{"interface":"front","average":{"1min":9.548,"5min":57.258000000000003,"15min":337.791},"min":{"1min":5.2300000000000004,"5min":5.2300000000000004,"15min":4.6050000000000004},"max":{"1min":15.337999999999999,"5min":2412.9549999999999,"15min":20640.792000000001},"last":9.7210000000000001}]},{"osd":2,"last update":"Mon Jun 3 09:05:22 2024","interfaces":[{"interface":"back","average":{"1min":9.5820000000000007,"5min":27.760000000000002,"15min":23.097999999999999},"min":{"1min":5.766,"5min":5.766,"15min":5.766},"max":{"1min":46.584000000000003,"5min":627.40700000000004,"15min":627.40700000000004},"last":5.9109999999999996},{"interface":"front","average":{"1min":13.574999999999999,"5min":35.847999999999999,"15min":30.065000000000001},"min":{"1min":8.6419999999999995,"5min":7.8630000000000004,"15min":6.9260000000000002},"max":{"1min":54.68,"5min":635.62599999999998,"15min":635.62599999999998},"last":10.257}]},{"osd":3,"last update":"Mon Jun 3 09:06:04 2024","interfaces":[{"interface":"back","average":{"1min":12.105,"5min":26.925000000000001,"15min":26.584},"min":{"1min":7.8410000000000002,"5min":7.1280000000000001,"15min":6.1909999999999998},"max":{"1min":18.425999999999998,"5min":389.72699999999998,"15min":389.72699999999998},"last":7.6399999999999997},{"interface":"front","average":{"1min":13.794,"5min":29.771999999999998,"15min":28.815000000000001},"min":{"1min":10.084,"5min":9.4920000000000009,"15min":6.1849999999999996},"max":{"1min":17.373999999999999,"5min":390.23200000000003,"15min":494.92899999999997},"last":12.619}]},{"osd":4,"last update":"Mon Jun 3 09:05:27 2024","interfaces":[{"interface":"back","average":{"1min":13.478,"5min":1032.126,"15min":1389.729},"min":{"1min":6.5119999999999996,"5min":6.5119999999999996,"15min":6.5119999999999996},"max":{"1min":55.354999999999997,"5min":25555.212,"15min":60989.964},"last":9.1739999999999995},{"interface":"front","average":{"1min":16.204999999999998,"5min":1030.8389999999999,"15min":1387.778},"min":{"1min":10.282999999999999,"5min":7.1429999999999998,"15min":7.1429999999999998},"max":{"1min":49.945999999999998,"5min":25556.159,"15min":61000.017},"last":11.471}]},{"osd":5,"last update":"Mon Jun 3 09:06:17 2024","interfaces":[{"interface":"back","average":{"1min":9.7040000000000006,"5min":26.533999999999999,"15min":28.683},"min":{"1min":6.8129999999999997,"5min":6.8129999999999997,"15min":6.8129999999999997},"max":{"1min":14.942,"5min":455.78699999999998,"15min":605.05899999999997},"last":13.146000000000001},{"interface":"front","average":{"1min":14.279,"5min":28.398,"15min":26.192},"min":{"1min":11.696,"5min":6.4420000000000002,"15min":6.2030000000000003},"max":{"1min":17.879000000000001,"5min":391.31200000000001,"15min":541.11000000000001},"last":14.859999999999999}]},{"osd":6,"last update":"Mon Jun 3 09:04:30 2024","interfaces":[{"interface":"back","average":{"1min":52.496000000000002,"5min":33.139000000000003,"15min":160.83199999999999},"min":{"1min":7.5289999999999999,"5min":7.1040000000000001,"15min":6.3739999999999997},"max":{"1min":457.91699999999997,"5min":457.91699999999997,"15min":11115.258},"last":18.959},{"interface":"front","average":{"1min":49.670000000000002,"5min":33.704999999999998,"15min":162.791},"min":{"1min":12.247999999999999,"5min":11.019,"15min":9.9760000000000009},"max":{"1min":407.05099999999999,"5min":407.05099999999999,"15min":11092.194},"last":17.707000000000001}]},{"osd":7,"last update":"Mon Jun 3 09:05:22 2024","interfaces":[{"interface":"back","average":{"1min":14.503,"5min":29.489000000000001,"15min":33.134999999999998},"min":{"1min":8.5690000000000008,"5min":7.5149999999999997,"15min":6.9859999999999998},"max":{"1min":48.526000000000003,"5min":461.41699999999997,"15min":598.33699999999999},"last":11.723000000000001},{"interface":"front","average":{"1min":17.050000000000001,"5min":33.313000000000002,"15min":38.603999999999999},"min":{"1min":11.182,"5min":10.263999999999999,"15min":10.263999999999999},"max":{"1min":58.683,"5min":463.10599999999999,"15min":592.80499999999995},"last":14.053000000000001}]}]}],"pool_statfs":[{"poolid":1,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":401408,"data_stored":397840,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":5,"total":0,"available":0,"internally_reserved":0,"allocated":401408,"data_stored":397840,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":4,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":5,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":6,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":17,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":17,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":17,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":17,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":17,"osd":4,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":17,"osd":5,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":5,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":17,"osd":6,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":5,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":17,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0}]}} 2024-06-03T09:06:23.348 INFO:tasks.ceph.mon.a.smithi088.stderr:==00:00:31:18.674 34743== Warning: unimplemented fcntl command: 1036 2024-06-03T09:06:23.350 INFO:tasks.ceph.mon.a.smithi088.stderr:==00:00:31:18.676 34743== Warning: unimplemented fcntl command: 1036 2024-06-03T09:06:23.365 INFO:tasks.ceph.mon.a.smithi088.stderr:==00:00:31:18.691 34743== Warning: unimplemented fcntl command: 1036 2024-06-03T09:06:23.400 INFO:tasks.ceph.mon.c.smithi088.stderr:==00:00:31:18.726 34749== Warning: unimplemented fcntl command: 1036 2024-06-03T09:06:23.401 INFO:tasks.ceph.mon.c.smithi088.stderr:==00:00:31:18.727 34749== Warning: unimplemented fcntl command: 1036 2024-06-03T09:06:23.406 INFO:tasks.ceph.mon.b.smithi192.stderr:==00:00:31:18.709 34228== Warning: unimplemented fcntl command: 1036 2024-06-03T09:06:23.407 INFO:tasks.ceph.mon.b.smithi192.stderr:==00:00:31:18.710 34228== Warning: unimplemented fcntl command: 1036 2024-06-03T09:06:23.419 INFO:tasks.ceph.mon.c.smithi088.stderr:==00:00:31:18.745 34749== Warning: unimplemented fcntl command: 1036 2024-06-03T09:06:23.422 INFO:tasks.ceph.mon.b.smithi192.stderr:==00:00:31:18.725 34228== Warning: unimplemented fcntl command: 1036 2024-06-03T09:06:25.876 DEBUG:teuthology.orchestra.run.smithi088:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph pg dump --format=json 2024-06-03T09:06:25.982 INFO:teuthology.orchestra.run.smithi088.stderr:2024-06-03T09:06:25.982+0000 7fab520af640 10 monclient: get_monmap_and_config 2024-06-03T09:06:25.982 INFO:teuthology.orchestra.run.smithi088.stderr:2024-06-03T09:06:25.982+0000 7fab520af640 10 monclient: build_initial_monmap 2024-06-03T09:06:25.982 INFO:teuthology.orchestra.run.smithi088.stderr:2024-06-03T09:06:25.982+0000 7fab520af640 10 monclient: monmap: 2024-06-03T09:06:25.982 INFO:teuthology.orchestra.run.smithi088.stderr:epoch 0 2024-06-03T09:06:25.982 INFO:teuthology.orchestra.run.smithi088.stderr:fsid ec3c3fdd-3ac6-4d7d-9412-683130ed8b04 2024-06-03T09:06:25.982 INFO:teuthology.orchestra.run.smithi088.stderr:last_changed 2024-06-03T09:06:25.983201+0000 2024-06-03T09:06:25.982 INFO:teuthology.orchestra.run.smithi088.stderr:created 2024-06-03T09:06:25.983201+0000 2024-06-03T09:06:25.982 INFO:teuthology.orchestra.run.smithi088.stderr:min_mon_release 0 (unknown) 2024-06-03T09:06:25.983 INFO:teuthology.orchestra.run.smithi088.stderr:election_strategy: 3 2024-06-03T09:06:25.983 INFO:teuthology.orchestra.run.smithi088.stderr:0: v1:172.21.15.88:6789/0 mon.noname-a 2024-06-03T09:06:25.983 INFO:teuthology.orchestra.run.smithi088.stderr:1: v1:172.21.15.192:6789/0 mon.noname-c 2024-06-03T09:06:25.983 INFO:teuthology.orchestra.run.smithi088.stderr:2: v1:172.21.15.88:6790/0 mon.noname-b 2024-06-03T09:06:25.983 INFO:teuthology.orchestra.run.smithi088.stderr: 2024-06-03T09:06:25.984 INFO:teuthology.orchestra.run.smithi088.stderr:2024-06-03T09:06:25.982+0000 7fab520af640 1 Processor -- start 2024-06-03T09:06:25.984 INFO:teuthology.orchestra.run.smithi088.stderr:2024-06-03T09:06:25.983+0000 7fab520af640 1 -- start start 2024-06-03T09:06:25.984 INFO:teuthology.orchestra.run.smithi088.stderr:2024-06-03T09:06:25.983+0000 7fab520af640 10 monclient: init 2024-06-03T09:06:25.984 INFO:teuthology.orchestra.run.smithi088.stderr:2024-06-03T09:06:25.983+0000 7fab520af640 10 monclient: _reopen_session rank -1 2024-06-03T09:06:25.984 INFO:teuthology.orchestra.run.smithi088.stderr:2024-06-03T09:06:25.983+0000 7fab520af640 10 monclient: _add_conns ranks=[1,2,0] 2024-06-03T09:06:25.984 INFO:teuthology.orchestra.run.smithi088.stderr:2024-06-03T09:06:25.983+0000 7fab520af640 10 monclient(hunting): picked mon.noname-c con 0x7fab4c0adb90 addr v1:172.21.15.192:6789/0 2024-06-03T09:06:25.985 INFO:teuthology.orchestra.run.smithi088.stderr:2024-06-03T09:06:25.983+0000 7fab520af640 10 monclient(hunting): picked mon.noname-b con 0x7fab4c06ee00 addr v1:172.21.15.88:6790/0 2024-06-03T09:06:25.985 INFO:teuthology.orchestra.run.smithi088.stderr:2024-06-03T09:06:25.983+0000 7fab520af640 10 monclient(hunting): picked mon.noname-a con 0x7fab4c1383e0 addr v1:172.21.15.88:6789/0 2024-06-03T09:06:25.985 INFO:teuthology.orchestra.run.smithi088.stderr:2024-06-03T09:06:25.983+0000 7fab520af640 1 -- --> v1:172.21.15.88:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fab4c06f2c0 con 0x7fab4c1383e0 2024-06-03T09:06:25.985 INFO:teuthology.orchestra.run.smithi088.stderr:2024-06-03T09:06:25.983+0000 7fab520af640 1 -- --> v1:172.21.15.192:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fab4c0710c0 con 0x7fab4c0adb90 2024-06-03T09:06:25.985 INFO:teuthology.orchestra.run.smithi088.stderr:2024-06-03T09:06:25.983+0000 7fab520af640 1 -- --> v1:172.21.15.88:6790/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fab4c071fd0 con 0x7fab4c06ee00 2024-06-03T09:06:25.985 INFO:teuthology.orchestra.run.smithi088.stderr:2024-06-03T09:06:25.983+0000 7fab520af640 10 monclient(hunting): _renew_subs 2024-06-03T09:06:25.985 INFO:teuthology.orchestra.run.smithi088.stderr:2024-06-03T09:06:25.983+0000 7fab520af640 10 monclient(hunting): authenticate will time out at 3041.793677s 2024-06-03T09:06:25.985 INFO:teuthology.orchestra.run.smithi088.stderr:2024-06-03T09:06:25.984+0000 7fab4bfff640 1 --1- >> v1:172.21.15.88:6789/0 conn(0x7fab4c1383e0 0x7fab4c149af0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.88:6789/0 says I am v1:172.21.15.88:43970/0 (socket says 172.21.15.88:43970) 2024-06-03T09:06:25.985 INFO:teuthology.orchestra.run.smithi088.stderr:2024-06-03T09:06:25.984+0000 7fab4bfff640 1 -- 172.21.15.88:0/1971501106 learned_addr learned my addr 172.21.15.88:0/1971501106 (peer_addr_for_me v1:172.21.15.88:0/0) 2024-06-03T09:06:25.986 INFO:teuthology.orchestra.run.smithi088.stderr:2024-06-03T09:06:25.986+0000 7fab4a7fc640 1 -- 172.21.15.88:0/1971501106 <== mon.1 v1:172.21.15.192:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 2086065709 0 0) 0x7fab4c0710c0 con 0x7fab4c0adb90 2024-06-03T09:06:25.986 INFO:teuthology.orchestra.run.smithi088.stderr:2024-06-03T09:06:25.986+0000 7fab4a7fc640 10 monclient(hunting): _negotiate 2024-06-03T09:06:25.986 INFO:teuthology.orchestra.run.smithi088.stderr:2024-06-03T09:06:25.986+0000 7fab4a7fc640 10 monclient(hunting): _init_auth method 2 2024-06-03T09:06:25.986 INFO:teuthology.orchestra.run.smithi088.stderr:2024-06-03T09:06:25.986+0000 7fab4a7fc640 10 monclient(hunting): _init_auth creating new auth 2024-06-03T09:06:25.986 INFO:teuthology.orchestra.run.smithi088.stderr:2024-06-03T09:06:25.986+0000 7fab4a7fc640 10 monclient(hunting): my global_id is 30181 2024-06-03T09:06:25.986 INFO:teuthology.orchestra.run.smithi088.stderr:2024-06-03T09:06:25.986+0000 7fab4a7fc640 1 -- 172.21.15.88:0/1971501106 --> v1:172.21.15.192:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7fab34005730 con 0x7fab4c0adb90 2024-06-03T09:06:25.987 INFO:teuthology.orchestra.run.smithi088.stderr:2024-06-03T09:06:25.986+0000 7fab4a7fc640 1 -- 172.21.15.88:0/1971501106 <== mon.0 v1:172.21.15.88:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 4194289410 0 0) 0x7fab4c06f2c0 con 0x7fab4c1383e0 2024-06-03T09:06:25.987 INFO:teuthology.orchestra.run.smithi088.stderr:2024-06-03T09:06:25.986+0000 7fab4a7fc640 10 monclient(hunting): _negotiate 2024-06-03T09:06:25.987 INFO:teuthology.orchestra.run.smithi088.stderr:2024-06-03T09:06:25.986+0000 7fab4a7fc640 10 monclient(hunting): _init_auth method 2 2024-06-03T09:06:25.987 INFO:teuthology.orchestra.run.smithi088.stderr:2024-06-03T09:06:25.986+0000 7fab4a7fc640 10 monclient(hunting): _init_auth creating new auth 2024-06-03T09:06:25.987 INFO:teuthology.orchestra.run.smithi088.stderr:2024-06-03T09:06:25.986+0000 7fab4a7fc640 10 monclient(hunting): my global_id is 30600 2024-06-03T09:06:25.988 INFO:teuthology.orchestra.run.smithi088.stderr:2024-06-03T09:06:25.986+0000 7fab4a7fc640 1 -- 172.21.15.88:0/1971501106 --> v1:172.21.15.88:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7fab4c0710c0 con 0x7fab4c1383e0 2024-06-03T09:06:25.988 INFO:teuthology.orchestra.run.smithi088.stderr:2024-06-03T09:06:25.987+0000 7fab4a7fc640 1 -- 172.21.15.88:0/1971501106 <== mon.2 v1:172.21.15.88:6790/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 89900738 0 0) 0x7fab4c071fd0 con 0x7fab4c06ee00 2024-06-03T09:06:25.988 INFO:teuthology.orchestra.run.smithi088.stderr:2024-06-03T09:06:25.987+0000 7fab4a7fc640 10 monclient(hunting): _negotiate 2024-06-03T09:06:25.988 INFO:teuthology.orchestra.run.smithi088.stderr:2024-06-03T09:06:25.987+0000 7fab4a7fc640 10 monclient(hunting): _init_auth method 2 2024-06-03T09:06:25.988 INFO:teuthology.orchestra.run.smithi088.stderr:2024-06-03T09:06:25.987+0000 7fab4a7fc640 10 monclient(hunting): _init_auth creating new auth 2024-06-03T09:06:25.988 INFO:teuthology.orchestra.run.smithi088.stderr:2024-06-03T09:06:25.987+0000 7fab4a7fc640 10 monclient(hunting): my global_id is 30104 2024-06-03T09:06:25.988 INFO:teuthology.orchestra.run.smithi088.stderr:2024-06-03T09:06:25.987+0000 7fab4a7fc640 1 -- 172.21.15.88:0/1971501106 --> v1:172.21.15.88:6790/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7fab4c06f2c0 con 0x7fab4c06ee00 2024-06-03T09:06:25.989 INFO:teuthology.orchestra.run.smithi088.stderr:2024-06-03T09:06:25.989+0000 7fab4a7fc640 1 -- 172.21.15.88:0/1971501106 <== mon.1 v1:172.21.15.192:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 214+0+0 (unknown 1797684233 0 0) 0x7fab34005730 con 0x7fab4c0adb90 2024-06-03T09:06:25.989 INFO:teuthology.orchestra.run.smithi088.stderr:2024-06-03T09:06:25.989+0000 7fab4a7fc640 1 -- 172.21.15.88:0/1971501106 >> v1:172.21.15.88:6790/0 conn(0x7fab4c06ee00 legacy=0x7fab4c137cd0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-06-03T09:06:25.990 INFO:teuthology.orchestra.run.smithi088.stderr:2024-06-03T09:06:25.989+0000 7fab4a7fc640 1 -- 172.21.15.88:0/1971501106 >> v1:172.21.15.88:6789/0 conn(0x7fab4c1383e0 legacy=0x7fab4c149af0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-06-03T09:06:25.990 INFO:teuthology.orchestra.run.smithi088.stderr:2024-06-03T09:06:25.989+0000 7fab4a7fc640 10 monclient: _finish_hunting 0 2024-06-03T09:06:25.990 INFO:teuthology.orchestra.run.smithi088.stderr:2024-06-03T09:06:25.989+0000 7fab4a7fc640 1 monclient: found mon.noname-c 2024-06-03T09:06:25.990 INFO:teuthology.orchestra.run.smithi088.stderr:2024-06-03T09:06:25.989+0000 7fab4a7fc640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-06-03T09:06:25.990 INFO:teuthology.orchestra.run.smithi088.stderr:2024-06-03T09:06:25.989+0000 7fab4a7fc640 10 monclient: _send_mon_message to mon.noname-c at v1:172.21.15.192:6789/0 2024-06-03T09:06:25.990 INFO:teuthology.orchestra.run.smithi088.stderr:2024-06-03T09:06:25.989+0000 7fab4a7fc640 1 -- 172.21.15.88:0/1971501106 --> v1:172.21.15.192:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fab4c141750 con 0x7fab4c0adb90 2024-06-03T09:06:25.990 INFO:teuthology.orchestra.run.smithi088.stderr:2024-06-03T09:06:25.989+0000 7fab4a7fc640 10 monclient: _finish_auth 0 2024-06-03T09:06:25.990 INFO:teuthology.orchestra.run.smithi088.stderr:2024-06-03T09:06:25.989+0000 7fab4a7fc640 10 monclient: _check_auth_tickets 2024-06-03T09:06:25.990 INFO:teuthology.orchestra.run.smithi088.stderr:2024-06-03T09:06:25.989+0000 7fab4a7fc640 20 monclient: _check_auth_rotating not needed by client.admin 2024-06-03T09:06:25.990 INFO:teuthology.orchestra.run.smithi088.stderr:2024-06-03T09:06:25.989+0000 7fab4a7fc640 1 -- 172.21.15.88:0/1971501106 <== mon.1 v1:172.21.15.192:6789/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 2234374065 0 0) 0x7fab3c001610 con 0x7fab4c0adb90 2024-06-03T09:06:25.990 INFO:teuthology.orchestra.run.smithi088.stderr:2024-06-03T09:06:25.989+0000 7fab4a7fc640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-06-03T09:06:25.990 INFO:teuthology.orchestra.run.smithi088.stderr:2024-06-03T09:06:25.989+0000 7fab4a7fc640 10 monclient: got monmap 1 from mon.noname-c (according to old e1) 2024-06-03T09:06:25.990 INFO:teuthology.orchestra.run.smithi088.stderr:2024-06-03T09:06:25.989+0000 7fab4a7fc640 10 monclient: dump: 2024-06-03T09:06:25.991 INFO:teuthology.orchestra.run.smithi088.stderr:epoch 1 2024-06-03T09:06:25.991 INFO:teuthology.orchestra.run.smithi088.stderr:fsid ec3c3fdd-3ac6-4d7d-9412-683130ed8b04 2024-06-03T09:06:25.991 INFO:teuthology.orchestra.run.smithi088.stderr:last_changed 2024-06-03T08:34:31.885125+0000 2024-06-03T09:06:25.991 INFO:teuthology.orchestra.run.smithi088.stderr:created 2024-06-03T08:34:31.885125+0000 2024-06-03T09:06:25.991 INFO:teuthology.orchestra.run.smithi088.stderr:min_mon_release 18 (reef) 2024-06-03T09:06:25.991 INFO:teuthology.orchestra.run.smithi088.stderr:election_strategy: 3 2024-06-03T09:06:25.991 INFO:teuthology.orchestra.run.smithi088.stderr:0: v1:172.21.15.88:6789/0 mon.a 2024-06-03T09:06:25.991 INFO:teuthology.orchestra.run.smithi088.stderr:1: v1:172.21.15.192:6789/0 mon.b 2024-06-03T09:06:25.991 INFO:teuthology.orchestra.run.smithi088.stderr:2: v1:172.21.15.88:6790/0 mon.c 2024-06-03T09:06:25.991 INFO:teuthology.orchestra.run.smithi088.stderr: 2024-06-03T09:06:25.991 INFO:teuthology.orchestra.run.smithi088.stderr:2024-06-03T09:06:25.989+0000 7fab520af640 5 monclient: authenticate success, global_id 30181 2024-06-03T09:06:25.991 INFO:teuthology.orchestra.run.smithi088.stderr:2024-06-03T09:06:25.989+0000 7fab520af640 20 monclient: get_monmap_and_config waiting for monmap|config 2024-06-03T09:06:25.993 INFO:teuthology.orchestra.run.smithi088.stderr:2024-06-03T09:06:25.991+0000 7fab4a7fc640 1 -- 172.21.15.88:0/1971501106 <== mon.1 v1:172.21.15.192:6789/0 4 ==== config(0 keys) v1 ==== 4+0+0 (unknown 0 0 0) 0x7fab3c001830 con 0x7fab4c0adb90 2024-06-03T09:06:25.993 INFO:teuthology.orchestra.run.smithi088.stderr:2024-06-03T09:06:25.991+0000 7fab4a7fc640 10 monclient: handle_config config(0 keys) v1 2024-06-03T09:06:25.993 INFO:teuthology.orchestra.run.smithi088.stderr:2024-06-03T09:06:25.991+0000 7fab4a7fc640 1 -- 172.21.15.88:0/1971501106 <== mon.1 v1:172.21.15.192:6789/0 5 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 2234374065 0 0) 0x7fab3c001bc0 con 0x7fab4c0adb90 2024-06-03T09:06:25.993 INFO:teuthology.orchestra.run.smithi088.stderr:2024-06-03T09:06:25.991+0000 7fab520af640 10 monclient: get_monmap_and_config success 2024-06-03T09:06:25.993 INFO:teuthology.orchestra.run.smithi088.stderr:2024-06-03T09:06:25.991+0000 7fab520af640 4 set_mon_vals no callback set 2024-06-03T09:06:25.993 INFO:teuthology.orchestra.run.smithi088.stderr:2024-06-03T09:06:25.991+0000 7fab4a7fc640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-06-03T09:06:25.993 INFO:teuthology.orchestra.run.smithi088.stderr:2024-06-03T09:06:25.991+0000 7fab4a7fc640 10 monclient: got monmap 1 from mon.b (according to old e1) 2024-06-03T09:06:25.994 INFO:teuthology.orchestra.run.smithi088.stderr:2024-06-03T09:06:25.991+0000 7fab4a7fc640 10 monclient: dump: 2024-06-03T09:06:25.994 INFO:teuthology.orchestra.run.smithi088.stderr:epoch 1 2024-06-03T09:06:25.994 INFO:teuthology.orchestra.run.smithi088.stderr:fsid ec3c3fdd-3ac6-4d7d-9412-683130ed8b04 2024-06-03T09:06:25.994 INFO:teuthology.orchestra.run.smithi088.stderr:last_changed 2024-06-03T08:34:31.885125+0000 2024-06-03T09:06:25.994 INFO:teuthology.orchestra.run.smithi088.stderr:created 2024-06-03T08:34:31.885125+0000 2024-06-03T09:06:25.994 INFO:teuthology.orchestra.run.smithi088.stderr:min_mon_release 18 (reef) 2024-06-03T09:06:25.994 INFO:teuthology.orchestra.run.smithi088.stderr:election_strategy: 3 2024-06-03T09:06:25.994 INFO:teuthology.orchestra.run.smithi088.stderr:0: v1:172.21.15.88:6789/0 mon.a 2024-06-03T09:06:25.994 INFO:teuthology.orchestra.run.smithi088.stderr:1: v1:172.21.15.192:6789/0 mon.b 2024-06-03T09:06:25.994 INFO:teuthology.orchestra.run.smithi088.stderr:2: v1:172.21.15.88:6790/0 mon.c 2024-06-03T09:06:25.994 INFO:teuthology.orchestra.run.smithi088.stderr: 2024-06-03T09:06:25.994 INFO:teuthology.orchestra.run.smithi088.stderr:2024-06-03T09:06:25.992+0000 7fab520af640 10 monclient: shutdown 2024-06-03T09:06:25.994 INFO:teuthology.orchestra.run.smithi088.stderr:2024-06-03T09:06:25.992+0000 7fab520af640 20 monclient: shutdown discarding 0 pending message(s) 2024-06-03T09:06:25.995 INFO:teuthology.orchestra.run.smithi088.stderr:2024-06-03T09:06:25.992+0000 7fab520af640 1 -- 172.21.15.88:0/1971501106 >> v1:172.21.15.192:6789/0 conn(0x7fab4c0adb90 legacy=0x7fab4c0718c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-06-03T09:06:25.995 INFO:teuthology.orchestra.run.smithi088.stderr:2024-06-03T09:06:25.992+0000 7fab520af640 1 -- 172.21.15.88:0/1971501106 shutdown_connections 2024-06-03T09:06:25.995 INFO:teuthology.orchestra.run.smithi088.stderr:2024-06-03T09:06:25.992+0000 7fab520af640 1 -- 172.21.15.88:0/1971501106 >> 172.21.15.88:0/1971501106 conn(0x7fab4c0b2f20 msgr2=0x7fab4c0b3300 unknown :-1 s=STATE_NONE l=0).mark_down 2024-06-03T09:06:25.995 INFO:teuthology.orchestra.run.smithi088.stderr:2024-06-03T09:06:25.992+0000 7fab520af640 1 -- 172.21.15.88:0/1971501106 shutdown_connections 2024-06-03T09:06:25.995 INFO:teuthology.orchestra.run.smithi088.stderr:2024-06-03T09:06:25.992+0000 7fab520af640 1 -- 172.21.15.88:0/1971501106 wait complete. 2024-06-03T09:06:25.995 INFO:teuthology.orchestra.run.smithi088.stderr:2024-06-03T09:06:25.992+0000 7fab520af640 10 monclient: build_initial_monmap 2024-06-03T09:06:25.995 INFO:teuthology.orchestra.run.smithi088.stderr:2024-06-03T09:06:25.992+0000 7fab520af640 10 monclient: monmap: 2024-06-03T09:06:25.995 INFO:teuthology.orchestra.run.smithi088.stderr:epoch 0 2024-06-03T09:06:25.995 INFO:teuthology.orchestra.run.smithi088.stderr:fsid 00000000-0000-0000-0000-000000000000 2024-06-03T09:06:25.995 INFO:teuthology.orchestra.run.smithi088.stderr:last_changed 0.000000 2024-06-03T09:06:25.995 INFO:teuthology.orchestra.run.smithi088.stderr:created 0.000000 2024-06-03T09:06:25.995 INFO:teuthology.orchestra.run.smithi088.stderr:min_mon_release 0 (unknown) 2024-06-03T09:06:25.995 INFO:teuthology.orchestra.run.smithi088.stderr:election_strategy: 1 2024-06-03T09:06:25.995 INFO:teuthology.orchestra.run.smithi088.stderr:0: v1:172.21.15.88:6789/0 mon.noname-a 2024-06-03T09:06:25.996 INFO:teuthology.orchestra.run.smithi088.stderr:1: v1:172.21.15.192:6789/0 mon.noname-b 2024-06-03T09:06:25.996 INFO:teuthology.orchestra.run.smithi088.stderr:2: v1:172.21.15.88:6790/0 mon.noname-c 2024-06-03T09:06:25.996 INFO:teuthology.orchestra.run.smithi088.stderr: 2024-06-03T09:06:25.996 INFO:teuthology.orchestra.run.smithi088.stderr:2024-06-03T09:06:25.993+0000 7fab520af640 1 librados: starting msgr at 2024-06-03T09:06:25.996 INFO:teuthology.orchestra.run.smithi088.stderr:2024-06-03T09:06:25.993+0000 7fab520af640 1 librados: starting objecter 2024-06-03T09:06:25.996 INFO:teuthology.orchestra.run.smithi088.stderr:2024-06-03T09:06:25.993+0000 7fab520af640 1 Processor -- start 2024-06-03T09:06:25.996 INFO:teuthology.orchestra.run.smithi088.stderr:2024-06-03T09:06:25.993+0000 7fab520af640 1 -- start start 2024-06-03T09:06:25.996 INFO:teuthology.orchestra.run.smithi088.stderr:2024-06-03T09:06:25.993+0000 7fab520af640 1 librados: setting wanted keys 2024-06-03T09:06:25.996 INFO:teuthology.orchestra.run.smithi088.stderr:2024-06-03T09:06:25.993+0000 7fab520af640 1 librados: calling monclient init 2024-06-03T09:06:25.996 INFO:teuthology.orchestra.run.smithi088.stderr:2024-06-03T09:06:25.993+0000 7fab520af640 10 monclient: init 2024-06-03T09:06:25.998 INFO:teuthology.orchestra.run.smithi088.stderr:2024-06-03T09:06:25.993+0000 7fab520af640 10 monclient: _reopen_session rank -1 2024-06-03T09:06:25.998 INFO:teuthology.orchestra.run.smithi088.stderr:2024-06-03T09:06:25.993+0000 7fab520af640 10 monclient: _add_conns ranks=[0,2,1] 2024-06-03T09:06:25.998 INFO:teuthology.orchestra.run.smithi088.stderr:2024-06-03T09:06:25.993+0000 7fab520af640 10 monclient(hunting): picked mon.noname-a con 0x7fab4c06ee00 addr v1:172.21.15.88:6789/0 2024-06-03T09:06:25.999 INFO:teuthology.orchestra.run.smithi088.stderr:2024-06-03T09:06:25.993+0000 7fab520af640 10 monclient(hunting): picked mon.noname-c con 0x7fab4c0adb90 addr v1:172.21.15.88:6790/0 2024-06-03T09:06:25.999 INFO:teuthology.orchestra.run.smithi088.stderr:2024-06-03T09:06:25.993+0000 7fab520af640 10 monclient(hunting): picked mon.noname-b con 0x7fab4c1383e0 addr v1:172.21.15.192:6789/0 2024-06-03T09:06:25.999 INFO:teuthology.orchestra.run.smithi088.stderr:2024-06-03T09:06:25.993+0000 7fab520af640 1 -- --> v1:172.21.15.88:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fab4c13ed70 con 0x7fab4c06ee00 2024-06-03T09:06:25.999 INFO:teuthology.orchestra.run.smithi088.stderr:2024-06-03T09:06:25.994+0000 7fab520af640 1 -- --> v1:172.21.15.192:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fab4c13fa90 con 0x7fab4c1383e0 2024-06-03T09:06:25.999 INFO:teuthology.orchestra.run.smithi088.stderr:2024-06-03T09:06:25.994+0000 7fab520af640 1 -- --> v1:172.21.15.88:6790/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fab4c1cfd30 con 0x7fab4c0adb90 2024-06-03T09:06:25.999 INFO:teuthology.orchestra.run.smithi088.stderr:2024-06-03T09:06:25.994+0000 7fab520af640 10 monclient(hunting): _renew_subs 2024-06-03T09:06:25.999 INFO:teuthology.orchestra.run.smithi088.stderr:2024-06-03T09:06:25.994+0000 7fab520af640 10 monclient(hunting): authenticate will time out at 3041.803960s 2024-06-03T09:06:25.999 INFO:teuthology.orchestra.run.smithi088.stderr:2024-06-03T09:06:25.994+0000 7fab4b7fe640 1 --1- >> v1:172.21.15.88:6789/0 conn(0x7fab4c06ee00 0x7fab4c0ac850 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=1).handle_server_banner_and_identify peer v1:172.21.15.88:6789/0 says I am v1:172.21.15.88:43986/0 (socket says 172.21.15.88:43986) 2024-06-03T09:06:25.999 INFO:teuthology.orchestra.run.smithi088.stderr:2024-06-03T09:06:25.994+0000 7fab4b7fe640 1 -- 172.21.15.88:0/1827382178 learned_addr learned my addr 172.21.15.88:0/1827382178 (peer_addr_for_me v1:172.21.15.88:0/0) 2024-06-03T09:06:25.999 INFO:teuthology.orchestra.run.smithi088.stderr:2024-06-03T09:06:25.994+0000 7fab33fff640 10 client.?.objecter ms_handle_connect 0x7fab4c06ee00 2024-06-03T09:06:25.999 INFO:teuthology.orchestra.run.smithi088.stderr:2024-06-03T09:06:25.994+0000 7fab33fff640 10 client.?.objecter resend_mon_ops 2024-06-03T09:06:25.999 INFO:teuthology.orchestra.run.smithi088.stderr:2024-06-03T09:06:25.994+0000 7fab33fff640 10 client.?.objecter ms_handle_connect 0x7fab4c1383e0 2024-06-03T09:06:25.999 INFO:teuthology.orchestra.run.smithi088.stderr:2024-06-03T09:06:25.994+0000 7fab33fff640 10 client.?.objecter resend_mon_ops 2024-06-03T09:06:26.000 INFO:teuthology.orchestra.run.smithi088.stderr:2024-06-03T09:06:25.994+0000 7fab33fff640 10 client.?.objecter ms_handle_connect 0x7fab4c0adb90 2024-06-03T09:06:26.000 INFO:teuthology.orchestra.run.smithi088.stderr:2024-06-03T09:06:25.994+0000 7fab33fff640 10 client.?.objecter resend_mon_ops 2024-06-03T09:06:26.000 INFO:teuthology.orchestra.run.smithi088.stderr:2024-06-03T09:06:25.996+0000 7fab33fff640 1 -- 172.21.15.88:0/1827382178 <== mon.1 v1:172.21.15.192:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 4107598206 0 0) 0x7fab4c13fa90 con 0x7fab4c1383e0 2024-06-03T09:06:26.000 INFO:teuthology.orchestra.run.smithi088.stderr:2024-06-03T09:06:25.996+0000 7fab33fff640 10 monclient(hunting): _negotiate 2024-06-03T09:06:26.000 INFO:teuthology.orchestra.run.smithi088.stderr:2024-06-03T09:06:25.996+0000 7fab33fff640 10 monclient(hunting): _init_auth method 2 2024-06-03T09:06:26.000 INFO:teuthology.orchestra.run.smithi088.stderr:2024-06-03T09:06:25.996+0000 7fab33fff640 10 monclient(hunting): _init_auth creating new auth 2024-06-03T09:06:26.000 INFO:teuthology.orchestra.run.smithi088.stderr:2024-06-03T09:06:25.996+0000 7fab33fff640 10 monclient(hunting): my global_id is 30184 2024-06-03T09:06:26.000 INFO:teuthology.orchestra.run.smithi088.stderr:2024-06-03T09:06:25.996+0000 7fab33fff640 1 -- 172.21.15.88:0/1827382178 --> v1:172.21.15.192:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7fab38004eb0 con 0x7fab4c1383e0 2024-06-03T09:06:26.000 INFO:teuthology.orchestra.run.smithi088.stderr:2024-06-03T09:06:25.996+0000 7fab33fff640 1 -- 172.21.15.88:0/1827382178 <== mon.0 v1:172.21.15.88:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 694384616 0 0) 0x7fab4c13ed70 con 0x7fab4c06ee00 2024-06-03T09:06:26.000 INFO:teuthology.orchestra.run.smithi088.stderr:2024-06-03T09:06:25.996+0000 7fab33fff640 10 monclient(hunting): _negotiate 2024-06-03T09:06:26.000 INFO:teuthology.orchestra.run.smithi088.stderr:2024-06-03T09:06:25.996+0000 7fab33fff640 10 monclient(hunting): _init_auth method 2 2024-06-03T09:06:26.000 INFO:teuthology.orchestra.run.smithi088.stderr:2024-06-03T09:06:25.996+0000 7fab33fff640 10 monclient(hunting): _init_auth creating new auth 2024-06-03T09:06:26.000 INFO:teuthology.orchestra.run.smithi088.stderr:2024-06-03T09:06:25.996+0000 7fab33fff640 10 monclient(hunting): my global_id is 30603 2024-06-03T09:06:26.001 INFO:teuthology.orchestra.run.smithi088.stderr:2024-06-03T09:06:25.996+0000 7fab33fff640 1 -- 172.21.15.88:0/1827382178 --> v1:172.21.15.88:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7fab4c13fa90 con 0x7fab4c06ee00 2024-06-03T09:06:26.001 INFO:teuthology.orchestra.run.smithi088.stderr:2024-06-03T09:06:25.996+0000 7fab33fff640 1 -- 172.21.15.88:0/1827382178 <== mon.2 v1:172.21.15.88:6790/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 2471139192 0 0) 0x7fab4c1cfd30 con 0x7fab4c0adb90 2024-06-03T09:06:26.001 INFO:teuthology.orchestra.run.smithi088.stderr:2024-06-03T09:06:25.996+0000 7fab33fff640 10 monclient(hunting): _negotiate 2024-06-03T09:06:26.001 INFO:teuthology.orchestra.run.smithi088.stderr:2024-06-03T09:06:25.996+0000 7fab33fff640 10 monclient(hunting): _init_auth method 2 2024-06-03T09:06:26.001 INFO:teuthology.orchestra.run.smithi088.stderr:2024-06-03T09:06:25.996+0000 7fab33fff640 10 monclient(hunting): _init_auth creating new auth 2024-06-03T09:06:26.001 INFO:teuthology.orchestra.run.smithi088.stderr:2024-06-03T09:06:25.996+0000 7fab33fff640 10 monclient(hunting): my global_id is 30107 2024-06-03T09:06:26.001 INFO:teuthology.orchestra.run.smithi088.stderr:2024-06-03T09:06:25.996+0000 7fab33fff640 1 -- 172.21.15.88:0/1827382178 --> v1:172.21.15.88:6790/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7fab4c13ed70 con 0x7fab4c0adb90 2024-06-03T09:06:26.002 INFO:teuthology.orchestra.run.smithi088.stderr:2024-06-03T09:06:26.001+0000 7fab33fff640 1 -- 172.21.15.88:0/1827382178 <== mon.1 v1:172.21.15.192:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 764+0+0 (unknown 2635923902 0 0) 0x7fab38004eb0 con 0x7fab4c1383e0 2024-06-03T09:06:26.002 INFO:teuthology.orchestra.run.smithi088.stderr:2024-06-03T09:06:26.001+0000 7fab33fff640 1 -- 172.21.15.88:0/1827382178 --> v1:172.21.15.192:6789/0 -- auth(proto 2 165 bytes epoch 0) v1 -- 0x7fab4c1cfd30 con 0x7fab4c1383e0 2024-06-03T09:06:26.003 INFO:teuthology.orchestra.run.smithi088.stderr:2024-06-03T09:06:26.001+0000 7fab33fff640 1 -- 172.21.15.88:0/1827382178 <== mon.1 v1:172.21.15.192:6789/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 2234374065 0 0) 0x7fab440017d0 con 0x7fab4c1383e0 2024-06-03T09:06:26.003 INFO:teuthology.orchestra.run.smithi088.stderr:2024-06-03T09:06:26.001+0000 7fab33fff640 10 monclient(hunting): handle_monmap mon_map magic: 0 v1 2024-06-03T09:06:26.003 INFO:teuthology.orchestra.run.smithi088.stderr:2024-06-03T09:06:26.001+0000 7fab33fff640 10 monclient(hunting): got monmap 1 from mon.noname-b (according to old e1) 2024-06-03T09:06:26.003 INFO:teuthology.orchestra.run.smithi088.stderr:2024-06-03T09:06:26.001+0000 7fab33fff640 10 monclient(hunting): dump: 2024-06-03T09:06:26.003 INFO:teuthology.orchestra.run.smithi088.stderr:epoch 1 2024-06-03T09:06:26.003 INFO:teuthology.orchestra.run.smithi088.stderr:fsid ec3c3fdd-3ac6-4d7d-9412-683130ed8b04 2024-06-03T09:06:26.003 INFO:teuthology.orchestra.run.smithi088.stderr:last_changed 2024-06-03T08:34:31.885125+0000 2024-06-03T09:06:26.003 INFO:teuthology.orchestra.run.smithi088.stderr:created 2024-06-03T08:34:31.885125+0000 2024-06-03T09:06:26.003 INFO:teuthology.orchestra.run.smithi088.stderr:min_mon_release 18 (reef) 2024-06-03T09:06:26.003 INFO:teuthology.orchestra.run.smithi088.stderr:election_strategy: 3 2024-06-03T09:06:26.003 INFO:teuthology.orchestra.run.smithi088.stderr:0: v1:172.21.15.88:6789/0 mon.a 2024-06-03T09:06:26.003 INFO:teuthology.orchestra.run.smithi088.stderr:1: v1:172.21.15.192:6789/0 mon.b 2024-06-03T09:06:26.003 INFO:teuthology.orchestra.run.smithi088.stderr:2: v1:172.21.15.88:6790/0 mon.c 2024-06-03T09:06:26.003 INFO:teuthology.orchestra.run.smithi088.stderr: 2024-06-03T09:06:26.004 INFO:teuthology.orchestra.run.smithi088.stderr:2024-06-03T09:06:26.001+0000 7fab33fff640 10 monclient(hunting): _finish_auth 0 2024-06-03T09:06:26.004 INFO:teuthology.orchestra.run.smithi088.stderr:2024-06-03T09:06:26.001+0000 7fab33fff640 1 -- 172.21.15.88:0/1827382178 <== mon.0 v1:172.21.15.88:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 764+0+0 (unknown 1142503285 0 0) 0x7fab4c13fa90 con 0x7fab4c06ee00 2024-06-03T09:06:26.004 INFO:teuthology.orchestra.run.smithi088.stderr:2024-06-03T09:06:26.001+0000 7fab33fff640 1 -- 172.21.15.88:0/1827382178 --> v1:172.21.15.88:6789/0 -- auth(proto 2 165 bytes epoch 0) v1 -- 0x7fab38004eb0 con 0x7fab4c06ee00 2024-06-03T09:06:26.004 INFO:teuthology.orchestra.run.smithi088.stderr:2024-06-03T09:06:26.001+0000 7fab33fff640 1 -- 172.21.15.88:0/1827382178 <== mon.0 v1:172.21.15.88:6789/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 2234374065 0 0) 0x7fab3c001b40 con 0x7fab4c06ee00 2024-06-03T09:06:26.004 INFO:teuthology.orchestra.run.smithi088.stderr:2024-06-03T09:06:26.001+0000 7fab33fff640 10 monclient(hunting): handle_monmap mon_map magic: 0 v1 2024-06-03T09:06:26.004 INFO:teuthology.orchestra.run.smithi088.stderr:2024-06-03T09:06:26.001+0000 7fab33fff640 10 monclient(hunting): got monmap 1 from mon.a (according to old e1) 2024-06-03T09:06:26.004 INFO:teuthology.orchestra.run.smithi088.stderr:2024-06-03T09:06:26.001+0000 7fab33fff640 10 monclient(hunting): dump: 2024-06-03T09:06:26.004 INFO:teuthology.orchestra.run.smithi088.stderr:epoch 1 2024-06-03T09:06:26.004 INFO:teuthology.orchestra.run.smithi088.stderr:fsid ec3c3fdd-3ac6-4d7d-9412-683130ed8b04 2024-06-03T09:06:26.004 INFO:teuthology.orchestra.run.smithi088.stderr:last_changed 2024-06-03T08:34:31.885125+0000 2024-06-03T09:06:26.004 INFO:teuthology.orchestra.run.smithi088.stderr:created 2024-06-03T08:34:31.885125+0000 2024-06-03T09:06:26.004 INFO:teuthology.orchestra.run.smithi088.stderr:min_mon_release 18 (reef) 2024-06-03T09:06:26.004 INFO:teuthology.orchestra.run.smithi088.stderr:election_strategy: 3 2024-06-03T09:06:26.005 INFO:teuthology.orchestra.run.smithi088.stderr:0: v1:172.21.15.88:6789/0 mon.a 2024-06-03T09:06:26.005 INFO:teuthology.orchestra.run.smithi088.stderr:1: v1:172.21.15.192:6789/0 mon.b 2024-06-03T09:06:26.005 INFO:teuthology.orchestra.run.smithi088.stderr:2: v1:172.21.15.88:6790/0 mon.c 2024-06-03T09:06:26.005 INFO:teuthology.orchestra.run.smithi088.stderr: 2024-06-03T09:06:26.005 INFO:teuthology.orchestra.run.smithi088.stderr:2024-06-03T09:06:26.002+0000 7fab33fff640 1 -- 172.21.15.88:0/1827382178 <== mon.2 v1:172.21.15.88:6790/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 764+0+0 (unknown 4033369261 0 0) 0x7fab4c13ed70 con 0x7fab4c0adb90 2024-06-03T09:06:26.005 INFO:teuthology.orchestra.run.smithi088.stderr:2024-06-03T09:06:26.002+0000 7fab33fff640 1 -- 172.21.15.88:0/1827382178 --> v1:172.21.15.88:6790/0 -- auth(proto 2 165 bytes epoch 0) v1 -- 0x7fab4c13fa90 con 0x7fab4c0adb90 2024-06-03T09:06:26.005 INFO:teuthology.orchestra.run.smithi088.stderr:2024-06-03T09:06:26.002+0000 7fab33fff640 1 -- 172.21.15.88:0/1827382178 <== mon.2 v1:172.21.15.88:6790/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 2234374065 0 0) 0x7fab40001c30 con 0x7fab4c0adb90 2024-06-03T09:06:26.005 INFO:teuthology.orchestra.run.smithi088.stderr:2024-06-03T09:06:26.002+0000 7fab33fff640 10 monclient(hunting): handle_monmap mon_map magic: 0 v1 2024-06-03T09:06:26.005 INFO:teuthology.orchestra.run.smithi088.stderr:2024-06-03T09:06:26.002+0000 7fab33fff640 10 monclient(hunting): got monmap 1 from mon.c (according to old e1) 2024-06-03T09:06:26.005 INFO:teuthology.orchestra.run.smithi088.stderr:2024-06-03T09:06:26.002+0000 7fab33fff640 10 monclient(hunting): dump: 2024-06-03T09:06:26.006 INFO:teuthology.orchestra.run.smithi088.stderr:epoch 1 2024-06-03T09:06:26.006 INFO:teuthology.orchestra.run.smithi088.stderr:fsid ec3c3fdd-3ac6-4d7d-9412-683130ed8b04 2024-06-03T09:06:26.006 INFO:teuthology.orchestra.run.smithi088.stderr:last_changed 2024-06-03T08:34:31.885125+0000 2024-06-03T09:06:26.006 INFO:teuthology.orchestra.run.smithi088.stderr:created 2024-06-03T08:34:31.885125+0000 2024-06-03T09:06:26.006 INFO:teuthology.orchestra.run.smithi088.stderr:min_mon_release 18 (reef) 2024-06-03T09:06:26.006 INFO:teuthology.orchestra.run.smithi088.stderr:election_strategy: 3 2024-06-03T09:06:26.006 INFO:teuthology.orchestra.run.smithi088.stderr:0: v1:172.21.15.88:6789/0 mon.a 2024-06-03T09:06:26.006 INFO:teuthology.orchestra.run.smithi088.stderr:1: v1:172.21.15.192:6789/0 mon.b 2024-06-03T09:06:26.006 INFO:teuthology.orchestra.run.smithi088.stderr:2: v1:172.21.15.88:6790/0 mon.c 2024-06-03T09:06:26.006 INFO:teuthology.orchestra.run.smithi088.stderr: 2024-06-03T09:06:26.008 INFO:teuthology.orchestra.run.smithi088.stderr:2024-06-03T09:06:26.004+0000 7fab33fff640 1 -- 172.21.15.88:0/1827382178 <== mon.1 v1:172.21.15.192:6789/0 4 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 580+0+0 (unknown 3798171257 0 0) 0x7fab4c1cfd30 con 0x7fab4c1383e0 2024-06-03T09:06:26.008 INFO:teuthology.orchestra.run.smithi088.stderr:2024-06-03T09:06:26.004+0000 7fab33fff640 1 -- 172.21.15.88:0/1827382178 >> v1:172.21.15.88:6790/0 conn(0x7fab4c0adb90 legacy=0x7fab4c13c210 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-06-03T09:06:26.008 INFO:teuthology.orchestra.run.smithi088.stderr:2024-06-03T09:06:26.004+0000 7fab33fff640 1 -- 172.21.15.88:0/1827382178 >> v1:172.21.15.88:6789/0 conn(0x7fab4c06ee00 legacy=0x7fab4c0ac850 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-06-03T09:06:26.008 INFO:teuthology.orchestra.run.smithi088.stderr:2024-06-03T09:06:26.004+0000 7fab33fff640 10 monclient: _finish_hunting 0 2024-06-03T09:06:26.008 INFO:teuthology.orchestra.run.smithi088.stderr:2024-06-03T09:06:26.004+0000 7fab33fff640 1 monclient: found mon.b 2024-06-03T09:06:26.008 INFO:teuthology.orchestra.run.smithi088.stderr:2024-06-03T09:06:26.004+0000 7fab33fff640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-06-03T09:06:26.008 INFO:teuthology.orchestra.run.smithi088.stderr:2024-06-03T09:06:26.004+0000 7fab33fff640 10 monclient: _send_mon_message to mon.b at v1:172.21.15.192:6789/0 2024-06-03T09:06:26.008 INFO:teuthology.orchestra.run.smithi088.stderr:2024-06-03T09:06:26.004+0000 7fab33fff640 1 -- 172.21.15.88:0/1827382178 --> v1:172.21.15.192:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fab4c13d940 con 0x7fab4c1383e0 2024-06-03T09:06:26.008 INFO:teuthology.orchestra.run.smithi088.stderr:2024-06-03T09:06:26.004+0000 7fab33fff640 10 monclient: _finish_auth 0 2024-06-03T09:06:26.008 INFO:teuthology.orchestra.run.smithi088.stderr:2024-06-03T09:06:26.004+0000 7fab33fff640 10 monclient: _check_auth_tickets 2024-06-03T09:06:26.009 INFO:teuthology.orchestra.run.smithi088.stderr:2024-06-03T09:06:26.004+0000 7fab33fff640 20 monclient: _check_auth_rotating not needed by client.admin 2024-06-03T09:06:26.009 INFO:teuthology.orchestra.run.smithi088.stderr:2024-06-03T09:06:26.004+0000 7fab520af640 5 monclient: authenticate success, global_id 30184 2024-06-03T09:06:26.009 INFO:teuthology.orchestra.run.smithi088.stderr:2024-06-03T09:06:26.004+0000 7fab520af640 10 monclient: _renew_subs 2024-06-03T09:06:26.009 INFO:teuthology.orchestra.run.smithi088.stderr:2024-06-03T09:06:26.004+0000 7fab520af640 10 monclient: _send_mon_message to mon.b at v1:172.21.15.192:6789/0 2024-06-03T09:06:26.009 INFO:teuthology.orchestra.run.smithi088.stderr:2024-06-03T09:06:26.004+0000 7fab520af640 1 -- 172.21.15.88:0/1827382178 --> v1:172.21.15.192:6789/0 -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fab4c13f400 con 0x7fab4c1383e0 2024-06-03T09:06:26.009 INFO:teuthology.orchestra.run.smithi088.stderr:2024-06-03T09:06:26.004+0000 7fab520af640 10 client.30184.objecter _maybe_request_map subscribing (onetime) to next osd map 2024-06-03T09:06:26.009 INFO:teuthology.orchestra.run.smithi088.stderr:2024-06-03T09:06:26.004+0000 7fab520af640 10 monclient: _renew_subs 2024-06-03T09:06:26.009 INFO:teuthology.orchestra.run.smithi088.stderr:2024-06-03T09:06:26.004+0000 7fab520af640 10 monclient: _send_mon_message to mon.b at v1:172.21.15.192:6789/0 2024-06-03T09:06:26.009 INFO:teuthology.orchestra.run.smithi088.stderr:2024-06-03T09:06:26.004+0000 7fab520af640 1 -- 172.21.15.88:0/1827382178 --> v1:172.21.15.192:6789/0 -- mon_subscribe({osdmap=0}) v3 -- 0x7fab4c13e780 con 0x7fab4c1383e0 2024-06-03T09:06:26.009 INFO:teuthology.orchestra.run.smithi088.stderr:2024-06-03T09:06:26.004+0000 7fab520af640 1 librados: init done 2024-06-03T09:06:26.009 INFO:teuthology.orchestra.run.smithi088.stderr:2024-06-03T09:06:26.005+0000 7fab520af640 10 start_mon_command cmd=[{"prefix": "get_command_descriptions"}] 2024-06-03T09:06:26.009 INFO:teuthology.orchestra.run.smithi088.stderr:2024-06-03T09:06:26.005+0000 7fab520af640 10 monclient: _send_command 1 [{"prefix": "get_command_descriptions"}] 2024-06-03T09:06:26.009 INFO:teuthology.orchestra.run.smithi088.stderr:2024-06-03T09:06:26.005+0000 7fab520af640 10 monclient: _send_mon_message to mon.b at v1:172.21.15.192:6789/0 2024-06-03T09:06:26.009 INFO:teuthology.orchestra.run.smithi088.stderr:2024-06-03T09:06:26.005+0000 7fab520af640 1 -- 172.21.15.88:0/1827382178 --> v1:172.21.15.192:6789/0 -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fab4c13f890 con 0x7fab4c1383e0 2024-06-03T09:06:26.020 INFO:teuthology.orchestra.run.smithi088.stderr:2024-06-03T09:06:26.020+0000 7fab33fff640 1 -- 172.21.15.88:0/1827382178 <== mon.1 v1:172.21.15.192:6789/0 5 ==== config(0 keys) v1 ==== 4+0+0 (unknown 0 0 0) 0x7fab44001d00 con 0x7fab4c1383e0 2024-06-03T09:06:26.020 INFO:teuthology.orchestra.run.smithi088.stderr:2024-06-03T09:06:26.020+0000 7fab33fff640 10 monclient: handle_config config(0 keys) v1 2024-06-03T09:06:26.020 INFO:teuthology.orchestra.run.smithi088.stderr:2024-06-03T09:06:26.020+0000 7fab33fff640 1 -- 172.21.15.88:0/1827382178 <== mon.1 v1:172.21.15.192:6789/0 6 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 2234374065 0 0) 0x7fab44004a00 con 0x7fab4c1383e0 2024-06-03T09:06:26.020 INFO:teuthology.orchestra.run.smithi088.stderr:2024-06-03T09:06:26.020+0000 7fab33fff640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-06-03T09:06:26.020 INFO:teuthology.orchestra.run.smithi088.stderr:2024-06-03T09:06:26.020+0000 7fab33fff640 10 monclient: got monmap 1 from mon.b (according to old e1) 2024-06-03T09:06:26.021 INFO:teuthology.orchestra.run.smithi088.stderr:2024-06-03T09:06:26.020+0000 7fab33fff640 10 monclient: dump: 2024-06-03T09:06:26.021 INFO:teuthology.orchestra.run.smithi088.stderr:epoch 1 2024-06-03T09:06:26.021 INFO:teuthology.orchestra.run.smithi088.stderr:fsid ec3c3fdd-3ac6-4d7d-9412-683130ed8b04 2024-06-03T09:06:26.021 INFO:teuthology.orchestra.run.smithi088.stderr:last_changed 2024-06-03T08:34:31.885125+0000 2024-06-03T09:06:26.021 INFO:teuthology.orchestra.run.smithi088.stderr:created 2024-06-03T08:34:31.885125+0000 2024-06-03T09:06:26.021 INFO:teuthology.orchestra.run.smithi088.stderr:min_mon_release 18 (reef) 2024-06-03T09:06:26.021 INFO:teuthology.orchestra.run.smithi088.stderr:election_strategy: 3 2024-06-03T09:06:26.021 INFO:teuthology.orchestra.run.smithi088.stderr:0: v1:172.21.15.88:6789/0 mon.a 2024-06-03T09:06:26.021 INFO:teuthology.orchestra.run.smithi088.stderr:1: v1:172.21.15.192:6789/0 mon.b 2024-06-03T09:06:26.021 INFO:teuthology.orchestra.run.smithi088.stderr:2: v1:172.21.15.88:6790/0 mon.c 2024-06-03T09:06:26.021 INFO:teuthology.orchestra.run.smithi088.stderr: 2024-06-03T09:06:26.021 INFO:teuthology.orchestra.run.smithi088.stderr:2024-06-03T09:06:26.020+0000 7fab49ffb640 4 set_mon_vals no callback set 2024-06-03T09:06:26.022 INFO:teuthology.orchestra.run.smithi088.stderr:2024-06-03T09:06:26.021+0000 7fab33fff640 1 -- 172.21.15.88:0/1827382178 <== mon.1 v1:172.21.15.192:6789/0 7 ==== mgrmap(e 5) v1 ==== 89110+0+0 (unknown 873533973 0 0) 0x7fab4401a780 con 0x7fab4c1383e0 2024-06-03T09:06:26.022 INFO:teuthology.orchestra.run.smithi088.stderr:2024-06-03T09:06:26.021+0000 7fab4b7fe640 10 monclient: get_auth_request con 0x7fab3806d420 auth_method 0 2024-06-03T09:06:26.023 INFO:teuthology.orchestra.run.smithi088.stderr:2024-06-03T09:06:26.022+0000 7fab33fff640 10 client.30184.objecter ms_handle_connect 0x7fab3806d420 2024-06-03T09:06:26.023 INFO:teuthology.orchestra.run.smithi088.stderr:2024-06-03T09:06:26.022+0000 7fab33fff640 1 -- 172.21.15.88:0/1827382178 <== mon.1 v1:172.21.15.192:6789/0 8 ==== osd_map(1010..1010 src has 622..1010) v4 ==== 5229+0+0 (unknown 1105883993 0 0) 0x7fab44053ef0 con 0x7fab4c1383e0 2024-06-03T09:06:26.023 INFO:teuthology.orchestra.run.smithi088.stderr:2024-06-03T09:06:26.022+0000 7fab33fff640 10 client.30184.objecter ms_dispatch 0x7fab4c000b90 osd_map(1010..1010 src has 622..1010) v4 2024-06-03T09:06:26.023 INFO:teuthology.orchestra.run.smithi088.stderr:2024-06-03T09:06:26.022+0000 7fab33fff640 3 client.30184.objecter handle_osd_map got epochs [1010,1010] > 0 2024-06-03T09:06:26.023 INFO:teuthology.orchestra.run.smithi088.stderr:2024-06-03T09:06:26.022+0000 7fab33fff640 3 client.30184.objecter handle_osd_map decoding full epoch 1010 2024-06-03T09:06:26.023 INFO:teuthology.orchestra.run.smithi088.stderr:2024-06-03T09:06:26.023+0000 7fab33fff640 20 client.30184.objecter dump_active .. 0 homeless 2024-06-03T09:06:26.258 INFO:teuthology.orchestra.run.smithi088.stderr:2024-06-03T09:06:26.258+0000 7fab33fff640 1 -- 172.21.15.88:0/1827382178 <== mon.1 v1:172.21.15.192:6789/0 9 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+182498 (unknown 1092875540 0 3016760700) 0x7fab4404faa0 con 0x7fab4c1383e0 2024-06-03T09:06:26.259 INFO:teuthology.orchestra.run.smithi088.stderr:2024-06-03T09:06:26.258+0000 7fab33fff640 10 monclient: handle_mon_command_ack 1 [{"prefix": "get_command_descriptions"}] 2024-06-03T09:06:26.259 INFO:teuthology.orchestra.run.smithi088.stderr:2024-06-03T09:06:26.258+0000 7fab33fff640 10 monclient: _finish_command 1 = system:0 2024-06-03T09:06:26.439 INFO:teuthology.orchestra.run.smithi088.stderr:2024-06-03T09:06:26.439+0000 7fab520af640 1 -- 172.21.15.88:0/1827382178 --> v1:172.21.15.192:6800/34229 -- mgr_command(tid 0: {"prefix": "pg dump", "target": ["mon-mgr", ""], "format": "json"}) v1 -- 0x7fab4c0adf70 con 0x7fab3806d420 2024-06-03T09:06:26.443 INFO:teuthology.orchestra.run.smithi088.stderr:2024-06-03T09:06:26.444+0000 7fab33fff640 1 -- 172.21.15.88:0/1827382178 <== mgr.4110 v1:172.21.15.192:6800/34229 1 ==== mgr_command_reply(tid 0: 0 dumped all) v1 ==== 18+0+151766 (unknown 2965378022 0 2002554278) 0x7fab3c0077c0 con 0x7fab3806d420 2024-06-03T09:06:26.445 INFO:teuthology.orchestra.run.smithi088.stdout: 2024-06-03T09:06:26.446 INFO:teuthology.orchestra.run.smithi088.stderr:dumped all 2024-06-03T09:06:26.448 INFO:teuthology.orchestra.run.smithi088.stderr:2024-06-03T09:06:26.446+0000 7fab520af640 10 librados: watch_flush enter 2024-06-03T09:06:26.448 INFO:teuthology.orchestra.run.smithi088.stderr:2024-06-03T09:06:26.446+0000 7fab520af640 10 librados: watch_flush exit 2024-06-03T09:06:26.448 INFO:teuthology.orchestra.run.smithi088.stderr:2024-06-03T09:06:26.446+0000 7fab520af640 20 client.30184.objecter shutdown clearing up homeless session... 2024-06-03T09:06:26.448 INFO:teuthology.orchestra.run.smithi088.stderr:2024-06-03T09:06:26.446+0000 7fab520af640 10 client.30184.objecter successfully canceled tick 2024-06-03T09:06:26.448 INFO:teuthology.orchestra.run.smithi088.stderr:2024-06-03T09:06:26.446+0000 7fab520af640 1 -- 172.21.15.88:0/1827382178 >> v1:172.21.15.192:6800/34229 conn(0x7fab3806d420 legacy=0x7fab3806f8e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-06-03T09:06:26.448 INFO:teuthology.orchestra.run.smithi088.stderr:2024-06-03T09:06:26.446+0000 7fab520af640 10 monclient: shutdown 2024-06-03T09:06:26.448 INFO:teuthology.orchestra.run.smithi088.stderr:2024-06-03T09:06:26.446+0000 7fab520af640 20 monclient: shutdown discarding 0 pending message(s) 2024-06-03T09:06:26.448 INFO:teuthology.orchestra.run.smithi088.stderr:2024-06-03T09:06:26.446+0000 7fab520af640 1 -- 172.21.15.88:0/1827382178 >> v1:172.21.15.192:6789/0 conn(0x7fab4c1383e0 legacy=0x7fab4c13c920 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-06-03T09:06:26.449 INFO:teuthology.orchestra.run.smithi088.stderr:2024-06-03T09:06:26.446+0000 7fab520af640 1 -- 172.21.15.88:0/1827382178 shutdown_connections 2024-06-03T09:06:26.449 INFO:teuthology.orchestra.run.smithi088.stderr:2024-06-03T09:06:26.446+0000 7fab520af640 1 -- 172.21.15.88:0/1827382178 >> 172.21.15.88:0/1827382178 conn(0x7fab4c0b2f20 msgr2=0x7fab4c0aea20 unknown :-1 s=STATE_NONE l=0).mark_down 2024-06-03T09:06:26.449 INFO:teuthology.orchestra.run.smithi088.stderr:2024-06-03T09:06:26.446+0000 7fab520af640 1 -- 172.21.15.88:0/1827382178 shutdown_connections 2024-06-03T09:06:26.449 INFO:teuthology.orchestra.run.smithi088.stderr:2024-06-03T09:06:26.446+0000 7fab520af640 1 -- 172.21.15.88:0/1827382178 wait complete. 2024-06-03T09:06:26.449 INFO:teuthology.orchestra.run.smithi088.stderr:2024-06-03T09:06:26.446+0000 7fab520af640 1 librados: shutdown 2024-06-03T09:06:26.457 INFO:teuthology.orchestra.run.smithi088.stdout:{"pg_ready":true,"pg_map":{"version":1918,"stamp":"2024-06-03T09:06:25.099450+0000","last_osdmap_epoch":0,"last_pg_scan":0,"pg_stats_sum":{"stat_sum":{"num_bytes":459285,"num_objects":3,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":1,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":3,"num_whiteouts":0,"num_read":102,"num_read_kb":88,"num_write":120,"num_write_kb":1379,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2,"num_bytes_recovered":397840,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":83,"ondisk_log_size":83,"up":68,"acting":68,"num_store_stats":0},"osd_stats_sum":{"up_from":0,"seq":0,"num_pgs":100,"num_osds":8,"num_per_pool_osds":8,"num_per_pool_omap_osds":8,"kb":754974720,"kb_used":494548,"kb_used_data":46324,"kb_used_omap":12,"kb_used_meta":448115,"kb_avail":754480172,"statfs":{"total":773094113280,"available":772587696128,"internally_reserved":0,"allocated":47435776,"data_stored":22747954,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":12744,"internal_metadata":458870328},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":113,"apply_latency_ms":113,"commit_latency_ns":113000000,"apply_latency_ns":113000000},"alerts":[],"network_ping_times":[]},"pg_stats_delta":{"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":0,"acting":0,"num_store_stats":0,"stamp_delta":"10.002005"},"pg_stats":[{"pgid":"17.d","version":"0'0","reported_seq":942,"reported_epoch":1009,"state":"active+clean","last_fresh":"2024-06-03T09:06:14.443917+0000","last_change":"2024-06-03T08:38:19.260548+0000","last_active":"2024-06-03T09:06:14.443917+0000","last_peered":"2024-06-03T09:06:14.443917+0000","last_clean":"2024-06-03T09:06:14.443917+0000","last_became_active":"2024-06-03T08:38:19.253559+0000","last_became_peered":"2024-06-03T08:38:19.253559+0000","last_unstale":"2024-06-03T09:06:14.443917+0000","last_undegraded":"2024-06-03T09:06:14.443917+0000","last_fullsized":"2024-06-03T09:06:14.443917+0000","mapping_epoch":28,"log_start":"0'0","ondisk_log_start":"0'0","created":28,"last_epoch_clean":45,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-06-03T08:37:55.614497+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-06-03T08:37:55.614497+0000","last_clean_scrub_stamp":"2024-06-03T08:37:55.614497+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-06-04T19:00:12.470979+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1],"acting":[0,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"17.c","version":"0'0","reported_seq":953,"reported_epoch":1009,"state":"active+undersized","last_fresh":"2024-06-03T09:06:13.668156+0000","last_change":"2024-06-03T09:06:03.440074+0000","last_active":"2024-06-03T09:06:13.668156+0000","last_peered":"2024-06-03T09:06:13.668156+0000","last_clean":"2024-06-03T09:05:02.105638+0000","last_became_active":"2024-06-03T09:06:03.440074+0000","last_became_peered":"2024-06-03T09:06:03.440074+0000","last_unstale":"2024-06-03T09:06:13.668156+0000","last_undegraded":"2024-06-03T09:06:13.668156+0000","last_fullsized":"2024-06-03T09:06:03.377982+0000","mapping_epoch":999,"log_start":"0'0","ondisk_log_start":"0'0","created":28,"last_epoch_clean":255,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-06-03T08:37:55.614497+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-06-03T08:37:55.614497+0000","last_clean_scrub_stamp":"2024-06-03T08:37:55.614497+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-06-04T10:34:20.808557+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3],"acting":[3],"avail_no_missing":["3"],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"17.f","version":"0'0","reported_seq":148,"reported_epoch":1009,"state":"active+undersized","last_fresh":"2024-06-03T09:06:14.450543+0000","last_change":"2024-06-03T09:06:03.420285+0000","last_active":"2024-06-03T09:06:14.450543+0000","last_peered":"2024-06-03T09:06:14.450543+0000","last_clean":"2024-06-03T08:37:55.614497+0000","last_became_active":"2024-06-03T09:06:03.420285+0000","last_became_peered":"2024-06-03T09:06:03.420285+0000","last_unstale":"2024-06-03T09:06:14.450543+0000","last_undegraded":"2024-06-03T09:06:14.450543+0000","last_fullsized":"2024-06-03T09:06:03.380616+0000","mapping_epoch":999,"log_start":"0'0","ondisk_log_start":"0'0","created":28,"last_epoch_clean":255,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-06-03T08:37:55.614497+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-06-03T08:37:55.614497+0000","last_clean_scrub_stamp":"2024-06-03T08:37:55.614497+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-06-04T15:39:41.187731+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4],"acting":[4],"avail_no_missing":["4"],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"17.e","version":"0'0","reported_seq":149,"reported_epoch":1009,"state":"active+undersized","last_fresh":"2024-06-03T09:06:13.726992+0000","last_change":"2024-06-03T09:06:03.412887+0000","last_active":"2024-06-03T09:06:13.726992+0000","last_peered":"2024-06-03T09:06:13.726992+0000","last_clean":"2024-06-03T08:37:55.614497+0000","last_became_active":"2024-06-03T09:06:03.412887+0000","last_became_peered":"2024-06-03T09:06:03.412887+0000","last_unstale":"2024-06-03T09:06:13.726992+0000","last_undegraded":"2024-06-03T09:06:13.726992+0000","last_fullsized":"2024-06-03T09:06:03.332482+0000","mapping_epoch":999,"log_start":"0'0","ondisk_log_start":"0'0","created":28,"last_epoch_clean":255,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-06-03T08:37:55.614497+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-06-03T08:37:55.614497+0000","last_clean_scrub_stamp":"2024-06-03T08:37:55.614497+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-06-04T12:23:22.511601+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5],"acting":[5],"avail_no_missing":["5"],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"17.9","version":"0'0","reported_seq":946,"reported_epoch":1009,"state":"active+clean","last_fresh":"2024-06-03T09:06:14.999299+0000","last_change":"2024-06-03T08:38:04.772260+0000","last_active":"2024-06-03T09:06:14.999299+0000","last_peered":"2024-06-03T09:06:14.999299+0000","last_clean":"2024-06-03T09:06:14.999299+0000","last_became_active":"2024-06-03T08:38:04.763428+0000","last_became_peered":"2024-06-03T08:38:04.763428+0000","last_unstale":"2024-06-03T09:06:14.999299+0000","last_undegraded":"2024-06-03T09:06:14.999299+0000","last_fullsized":"2024-06-03T09:06:14.999299+0000","mapping_epoch":28,"log_start":"0'0","ondisk_log_start":"0'0","created":28,"last_epoch_clean":33,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-06-03T08:37:55.614497+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-06-03T08:37:55.614497+0000","last_clean_scrub_stamp":"2024-06-03T08:37:55.614497+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-06-04T10:55:10.773936+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,4],"acting":[7,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"17.8","version":"0'0","reported_seq":961,"reported_epoch":1009,"state":"active+undersized","last_fresh":"2024-06-03T09:06:14.430375+0000","last_change":"2024-06-03T09:06:03.388786+0000","last_active":"2024-06-03T09:06:14.430375+0000","last_peered":"2024-06-03T09:06:14.430375+0000","last_clean":"2024-06-03T09:04:43.297780+0000","last_became_active":"2024-06-03T09:06:03.388786+0000","last_became_peered":"2024-06-03T09:06:03.388786+0000","last_unstale":"2024-06-03T09:06:14.430375+0000","last_undegraded":"2024-06-03T09:06:14.430375+0000","last_fullsized":"2024-06-03T09:06:03.331754+0000","mapping_epoch":999,"log_start":"0'0","ondisk_log_start":"0'0","created":28,"last_epoch_clean":255,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-06-03T08:37:55.614497+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-06-03T08:37:55.614497+0000","last_clean_scrub_stamp":"2024-06-03T08:37:55.614497+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-06-04T09:18:44.765547+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0],"acting":[0],"avail_no_missing":["0"],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"17.b","version":"0'0","reported_seq":149,"reported_epoch":1009,"state":"active+undersized","last_fresh":"2024-06-03T09:06:14.433735+0000","last_change":"2024-06-03T09:06:03.451203+0000","last_active":"2024-06-03T09:06:14.433735+0000","last_peered":"2024-06-03T09:06:14.433735+0000","last_clean":"2024-06-03T08:37:55.614497+0000","last_became_active":"2024-06-03T09:06:03.451203+0000","last_became_peered":"2024-06-03T09:06:03.451203+0000","last_unstale":"2024-06-03T09:06:14.433735+0000","last_undegraded":"2024-06-03T09:06:14.433735+0000","last_fullsized":"2024-06-03T09:06:03.375523+0000","mapping_epoch":999,"log_start":"0'0","ondisk_log_start":"0'0","created":28,"last_epoch_clean":255,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-06-03T08:37:55.614497+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-06-03T08:37:55.614497+0000","last_clean_scrub_stamp":"2024-06-03T08:37:55.614497+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-06-04T10:16:19.813495+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2],"acting":[2],"avail_no_missing":["2"],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"17.a","version":"0'0","reported_seq":925,"reported_epoch":1010,"state":"active+clean","last_fresh":"2024-06-03T09:06:18.651697+0000","last_change":"2024-06-03T08:38:05.843142+0000","last_active":"2024-06-03T09:06:18.651697+0000","last_peered":"2024-06-03T09:06:18.651697+0000","last_clean":"2024-06-03T09:06:18.651697+0000","last_became_active":"2024-06-03T08:38:05.809257+0000","last_became_peered":"2024-06-03T08:38:05.809257+0000","last_unstale":"2024-06-03T09:06:18.651697+0000","last_undegraded":"2024-06-03T09:06:18.651697+0000","last_fullsized":"2024-06-03T09:06:18.651697+0000","mapping_epoch":28,"log_start":"0'0","ondisk_log_start":"0'0","created":28,"last_epoch_clean":33,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-06-03T08:37:55.614497+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-06-03T08:37:55.614497+0000","last_clean_scrub_stamp":"2024-06-03T08:37:55.614497+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-06-04T12:12:03.686838+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2],"acting":[1,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"17.5","version":"0'0","reported_seq":943,"reported_epoch":1009,"state":"active+clean","last_fresh":"2024-06-03T09:06:14.442357+0000","last_change":"2024-06-03T08:38:23.466051+0000","last_active":"2024-06-03T09:06:14.442357+0000","last_peered":"2024-06-03T09:06:14.442357+0000","last_clean":"2024-06-03T09:06:14.442357+0000","last_became_active":"2024-06-03T08:38:23.458039+0000","last_became_peered":"2024-06-03T08:38:23.458039+0000","last_unstale":"2024-06-03T09:06:14.442357+0000","last_undegraded":"2024-06-03T09:06:14.442357+0000","last_fullsized":"2024-06-03T09:06:14.442357+0000","mapping_epoch":28,"log_start":"0'0","ondisk_log_start":"0'0","created":28,"last_epoch_clean":49,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-06-03T08:37:55.614497+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-06-03T08:37:55.614497+0000","last_clean_scrub_stamp":"2024-06-03T08:37:55.614497+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-06-04T12:27:02.532436+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,4],"acting":[0,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"17.4","version":"0'0","reported_seq":149,"reported_epoch":1009,"state":"active+undersized","last_fresh":"2024-06-03T09:06:13.941522+0000","last_change":"2024-06-03T09:06:03.441889+0000","last_active":"2024-06-03T09:06:13.941522+0000","last_peered":"2024-06-03T09:06:13.941522+0000","last_clean":"2024-06-03T08:37:55.614497+0000","last_became_active":"2024-06-03T09:06:03.441889+0000","last_became_peered":"2024-06-03T09:06:03.441889+0000","last_unstale":"2024-06-03T09:06:13.941522+0000","last_undegraded":"2024-06-03T09:06:13.941522+0000","last_fullsized":"2024-06-03T09:06:03.359582+0000","mapping_epoch":999,"log_start":"0'0","ondisk_log_start":"0'0","created":28,"last_epoch_clean":255,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-06-03T08:37:55.614497+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-06-03T08:37:55.614497+0000","last_clean_scrub_stamp":"2024-06-03T08:37:55.614497+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-06-04T12:49:22.907141+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7],"acting":[7],"avail_no_missing":["7"],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"17.7","version":"0'0","reported_seq":929,"reported_epoch":1009,"state":"active+clean","last_fresh":"2024-06-03T09:06:13.736920+0000","last_change":"2024-06-03T08:38:04.837083+0000","last_active":"2024-06-03T09:06:13.736920+0000","last_peered":"2024-06-03T09:06:13.736920+0000","last_clean":"2024-06-03T09:06:13.736920+0000","last_became_active":"2024-06-03T08:38:04.818246+0000","last_became_peered":"2024-06-03T08:38:04.818246+0000","last_unstale":"2024-06-03T09:06:13.736920+0000","last_undegraded":"2024-06-03T09:06:13.736920+0000","last_fullsized":"2024-06-03T09:06:13.736920+0000","mapping_epoch":28,"log_start":"0'0","ondisk_log_start":"0'0","created":28,"last_epoch_clean":34,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-06-03T08:37:55.614497+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-06-03T08:37:55.614497+0000","last_clean_scrub_stamp":"2024-06-03T08:37:55.614497+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-06-04T17:02:38.204813+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,2],"acting":[5,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"17.6","version":"0'0","reported_seq":960,"reported_epoch":1009,"state":"active+undersized","last_fresh":"2024-06-03T09:06:14.453110+0000","last_change":"2024-06-03T09:06:03.428677+0000","last_active":"2024-06-03T09:06:14.453110+0000","last_peered":"2024-06-03T09:06:14.453110+0000","last_clean":"2024-06-03T09:04:43.312352+0000","last_became_active":"2024-06-03T09:06:03.428677+0000","last_became_peered":"2024-06-03T09:06:03.428677+0000","last_unstale":"2024-06-03T09:06:14.453110+0000","last_undegraded":"2024-06-03T09:06:14.453110+0000","last_fullsized":"2024-06-03T09:06:03.359909+0000","mapping_epoch":999,"log_start":"0'0","ondisk_log_start":"0'0","created":28,"last_epoch_clean":255,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-06-03T08:37:55.614497+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-06-03T08:37:55.614497+0000","last_clean_scrub_stamp":"2024-06-03T08:37:55.614497+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-06-04T15:38:02.180241+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0],"acting":[0],"avail_no_missing":["0"],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"2.7","version":"0'0","reported_seq":940,"reported_epoch":1009,"state":"active+clean","last_fresh":"2024-06-03T09:06:13.684065+0000","last_change":"2024-06-03T08:36:30.385405+0000","last_active":"2024-06-03T09:06:13.684065+0000","last_peered":"2024-06-03T09:06:13.684065+0000","last_clean":"2024-06-03T09:06:13.684065+0000","last_became_active":"2024-06-03T08:36:30.380302+0000","last_became_peered":"2024-06-03T08:36:30.380302+0000","last_unstale":"2024-06-03T09:06:13.684065+0000","last_undegraded":"2024-06-03T09:06:13.684065+0000","last_fullsized":"2024-06-03T09:06:13.684065+0000","mapping_epoch":23,"log_start":"0'0","ondisk_log_start":"0'0","created":23,"last_epoch_clean":24,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-06-03T08:36:28.444764+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-06-03T08:36:28.444764+0000","last_clean_scrub_stamp":"2024-06-03T08:36:28.444764+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-06-04T18:46:33.475181+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0],"acting":[3,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"17.14","version":"0'0","reported_seq":926,"reported_epoch":1009,"state":"active+clean","last_fresh":"2024-06-03T09:06:13.721460+0000","last_change":"2024-06-03T08:38:04.547565+0000","last_active":"2024-06-03T09:06:13.721460+0000","last_peered":"2024-06-03T09:06:13.721460+0000","last_clean":"2024-06-03T09:06:13.721460+0000","last_became_active":"2024-06-03T08:38:04.541781+0000","last_became_peered":"2024-06-03T08:38:04.541781+0000","last_unstale":"2024-06-03T09:06:13.721460+0000","last_undegraded":"2024-06-03T09:06:13.721460+0000","last_fullsized":"2024-06-03T09:06:13.721460+0000","mapping_epoch":28,"log_start":"0'0","ondisk_log_start":"0'0","created":28,"last_epoch_clean":34,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-06-03T08:37:55.614497+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-06-03T08:37:55.614497+0000","last_clean_scrub_stamp":"2024-06-03T08:37:55.614497+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-06-04T16:41:55.170134+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,2],"acting":[5,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"2.6","version":"0'0","reported_seq":949,"reported_epoch":1009,"state":"active+clean","last_fresh":"2024-06-03T09:06:14.448100+0000","last_change":"2024-06-03T08:36:30.430760+0000","last_active":"2024-06-03T09:06:14.448100+0000","last_peered":"2024-06-03T09:06:14.448100+0000","last_clean":"2024-06-03T09:06:14.448100+0000","last_became_active":"2024-06-03T08:36:30.404665+0000","last_became_peered":"2024-06-03T08:36:30.404665+0000","last_unstale":"2024-06-03T09:06:14.448100+0000","last_undegraded":"2024-06-03T09:06:14.448100+0000","last_fullsized":"2024-06-03T09:06:14.448100+0000","mapping_epoch":23,"log_start":"0'0","ondisk_log_start":"0'0","created":23,"last_epoch_clean":24,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-06-03T08:36:28.444764+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-06-03T08:36:28.444764+0000","last_clean_scrub_stamp":"2024-06-03T08:36:28.444764+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-06-04T10:37:37.746074+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,0],"acting":[4,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"17.15","version":"0'0","reported_seq":934,"reported_epoch":1009,"state":"active+clean","last_fresh":"2024-06-03T09:06:13.672940+0000","last_change":"2024-06-03T08:38:20.999903+0000","last_active":"2024-06-03T09:06:13.672940+0000","last_peered":"2024-06-03T09:06:13.672940+0000","last_clean":"2024-06-03T09:06:13.672940+0000","last_became_active":"2024-06-03T08:38:20.979523+0000","last_became_peered":"2024-06-03T08:38:20.979523+0000","last_unstale":"2024-06-03T09:06:13.672940+0000","last_undegraded":"2024-06-03T09:06:13.672940+0000","last_fullsized":"2024-06-03T09:06:13.672940+0000","mapping_epoch":28,"log_start":"0'0","ondisk_log_start":"0'0","created":28,"last_epoch_clean":47,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-06-03T08:37:55.614497+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-06-03T08:37:55.614497+0000","last_clean_scrub_stamp":"2024-06-03T08:37:55.614497+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-06-04T20:29:58.314289+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,1],"acting":[3,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"2.5","version":"0'0","reported_seq":942,"reported_epoch":1010,"state":"active+clean","last_fresh":"2024-06-03T09:06:18.382767+0000","last_change":"2024-06-03T08:36:30.024235+0000","last_active":"2024-06-03T09:06:18.382767+0000","last_peered":"2024-06-03T09:06:18.382767+0000","last_clean":"2024-06-03T09:06:18.382767+0000","last_became_active":"2024-06-03T08:36:30.019040+0000","last_became_peered":"2024-06-03T08:36:30.019040+0000","last_unstale":"2024-06-03T09:06:18.382767+0000","last_undegraded":"2024-06-03T09:06:18.382767+0000","last_fullsized":"2024-06-03T09:06:18.382767+0000","mapping_epoch":23,"log_start":"0'0","ondisk_log_start":"0'0","created":23,"last_epoch_clean":24,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-06-03T08:36:28.444764+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-06-03T08:36:28.444764+0000","last_clean_scrub_stamp":"2024-06-03T08:36:28.444764+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-06-04T15:46:47.109502+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,4],"acting":[3,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"17.16","version":"0'0","reported_seq":972,"reported_epoch":1009,"state":"active+undersized","last_fresh":"2024-06-03T09:06:14.441665+0000","last_change":"2024-06-03T09:06:03.478615+0000","last_active":"2024-06-03T09:06:14.441665+0000","last_peered":"2024-06-03T09:06:14.441665+0000","last_clean":"2024-06-03T09:04:43.328415+0000","last_became_active":"2024-06-03T09:06:03.478615+0000","last_became_peered":"2024-06-03T09:06:03.478615+0000","last_unstale":"2024-06-03T09:06:14.441665+0000","last_undegraded":"2024-06-03T09:06:14.441665+0000","last_fullsized":"2024-06-03T09:06:03.402504+0000","mapping_epoch":999,"log_start":"0'0","ondisk_log_start":"0'0","created":28,"last_epoch_clean":255,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-06-03T08:37:55.614497+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-06-03T08:37:55.614497+0000","last_clean_scrub_stamp":"2024-06-03T08:37:55.614497+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-06-04T08:47:08.079233+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2],"acting":[2],"avail_no_missing":["2"],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"2.4","version":"0'0","reported_seq":952,"reported_epoch":1010,"state":"active+undersized","last_fresh":"2024-06-03T09:06:18.331485+0000","last_change":"2024-06-03T09:06:03.387171+0000","last_active":"2024-06-03T09:06:18.331485+0000","last_peered":"2024-06-03T09:06:18.331485+0000","last_clean":"2024-06-03T09:04:43.202645+0000","last_became_active":"2024-06-03T09:06:03.387171+0000","last_became_peered":"2024-06-03T09:06:03.387171+0000","last_unstale":"2024-06-03T09:06:18.331485+0000","last_undegraded":"2024-06-03T09:06:18.331485+0000","last_fullsized":"2024-06-03T09:06:03.283787+0000","mapping_epoch":999,"log_start":"0'0","ondisk_log_start":"0'0","created":23,"last_epoch_clean":255,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-06-03T08:36:28.444764+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-06-03T08:36:28.444764+0000","last_clean_scrub_stamp":"2024-06-03T08:36:28.444764+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-06-04T20:17:43.485023+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5],"acting":[5],"avail_no_missing":["5"],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"17.17","version":"0'0","reported_seq":954,"reported_epoch":1009,"state":"active+clean","last_fresh":"2024-06-03T09:06:14.447046+0000","last_change":"2024-06-03T08:38:38.483275+0000","last_active":"2024-06-03T09:06:14.447046+0000","last_peered":"2024-06-03T09:06:14.447046+0000","last_clean":"2024-06-03T09:06:14.447046+0000","last_became_active":"2024-06-03T08:38:38.467916+0000","last_became_peered":"2024-06-03T08:38:38.467916+0000","last_unstale":"2024-06-03T09:06:14.447046+0000","last_undegraded":"2024-06-03T09:06:14.447046+0000","last_fullsized":"2024-06-03T09:06:14.447046+0000","mapping_epoch":28,"log_start":"0'0","ondisk_log_start":"0'0","created":28,"last_epoch_clean":61,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-06-03T08:37:55.614497+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-06-03T08:37:55.614497+0000","last_clean_scrub_stamp":"2024-06-03T08:37:55.614497+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-06-04T10:02:23.036420+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,7],"acting":[2,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"2.2","version":"0'0","reported_seq":963,"reported_epoch":1009,"state":"active+undersized","last_fresh":"2024-06-03T09:06:13.657892+0000","last_change":"2024-06-03T09:06:03.410020+0000","last_active":"2024-06-03T09:06:13.657892+0000","last_peered":"2024-06-03T09:06:13.657892+0000","last_clean":"2024-06-03T09:05:37.145889+0000","last_became_active":"2024-06-03T09:06:03.410020+0000","last_became_peered":"2024-06-03T09:06:03.410020+0000","last_unstale":"2024-06-03T09:06:13.657892+0000","last_undegraded":"2024-06-03T09:06:13.657892+0000","last_fullsized":"2024-06-03T09:06:03.346864+0000","mapping_epoch":999,"log_start":"0'0","ondisk_log_start":"0'0","created":23,"last_epoch_clean":255,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-06-03T08:36:28.444764+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-06-03T08:36:28.444764+0000","last_clean_scrub_stamp":"2024-06-03T08:36:28.444764+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-06-04T09:57:12.423360+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3],"acting":[3],"avail_no_missing":["3"],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"17.11","version":"0'0","reported_seq":944,"reported_epoch":1009,"state":"active+clean","last_fresh":"2024-06-03T09:06:14.426670+0000","last_change":"2024-06-03T08:38:06.502302+0000","last_active":"2024-06-03T09:06:14.426670+0000","last_peered":"2024-06-03T09:06:14.426670+0000","last_clean":"2024-06-03T09:06:14.426670+0000","last_became_active":"2024-06-03T08:38:06.490204+0000","last_became_peered":"2024-06-03T08:38:06.490204+0000","last_unstale":"2024-06-03T09:06:14.426670+0000","last_undegraded":"2024-06-03T09:06:14.426670+0000","last_fullsized":"2024-06-03T09:06:14.426670+0000","mapping_epoch":28,"log_start":"0'0","ondisk_log_start":"0'0","created":28,"last_epoch_clean":32,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-06-03T08:37:55.614497+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-06-03T08:37:55.614497+0000","last_clean_scrub_stamp":"2024-06-03T08:37:55.614497+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-06-04T10:47:05.988958+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,2],"acting":[4,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"2.1","version":"0'0","reported_seq":952,"reported_epoch":1009,"state":"active+clean","last_fresh":"2024-06-03T09:06:13.929725+0000","last_change":"2024-06-03T08:36:30.883360+0000","last_active":"2024-06-03T09:06:13.929725+0000","last_peered":"2024-06-03T09:06:13.929725+0000","last_clean":"2024-06-03T09:06:13.929725+0000","last_became_active":"2024-06-03T08:36:30.855073+0000","last_became_peered":"2024-06-03T08:36:30.855073+0000","last_unstale":"2024-06-03T09:06:13.929725+0000","last_undegraded":"2024-06-03T09:06:13.929725+0000","last_fullsized":"2024-06-03T09:06:13.929725+0000","mapping_epoch":23,"log_start":"0'0","ondisk_log_start":"0'0","created":23,"last_epoch_clean":25,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-06-03T08:36:28.444764+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-06-03T08:36:28.444764+0000","last_clean_scrub_stamp":"2024-06-03T08:36:28.444764+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-06-04T18:12:22.363473+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,5],"acting":[7,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"17.12","version":"0'0","reported_seq":945,"reported_epoch":1009,"state":"active+clean","last_fresh":"2024-06-03T09:06:13.935141+0000","last_change":"2024-06-03T08:38:01.751562+0000","last_active":"2024-06-03T09:06:13.935141+0000","last_peered":"2024-06-03T09:06:13.935141+0000","last_clean":"2024-06-03T09:06:13.935141+0000","last_became_active":"2024-06-03T08:38:01.745027+0000","last_became_peered":"2024-06-03T08:38:01.745027+0000","last_unstale":"2024-06-03T09:06:13.935141+0000","last_undegraded":"2024-06-03T09:06:13.935141+0000","last_fullsized":"2024-06-03T09:06:13.935141+0000","mapping_epoch":28,"log_start":"0'0","ondisk_log_start":"0'0","created":28,"last_epoch_clean":32,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-06-03T08:37:55.614497+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-06-03T08:37:55.614497+0000","last_clean_scrub_stamp":"2024-06-03T08:37:55.614497+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-06-04T17:54:43.785326+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,3],"acting":[7,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"2.0","version":"0'0","reported_seq":943,"reported_epoch":1009,"state":"active+clean","last_fresh":"2024-06-03T09:06:13.648112+0000","last_change":"2024-06-03T08:36:29.984096+0000","last_active":"2024-06-03T09:06:13.648112+0000","last_peered":"2024-06-03T09:06:13.648112+0000","last_clean":"2024-06-03T09:06:13.648112+0000","last_became_active":"2024-06-03T08:36:29.957875+0000","last_became_peered":"2024-06-03T08:36:29.957875+0000","last_unstale":"2024-06-03T09:06:13.648112+0000","last_undegraded":"2024-06-03T09:06:13.648112+0000","last_fullsized":"2024-06-03T09:06:13.648112+0000","mapping_epoch":23,"log_start":"0'0","ondisk_log_start":"0'0","created":23,"last_epoch_clean":24,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-06-03T08:36:28.444764+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-06-03T08:36:28.444764+0000","last_clean_scrub_stamp":"2024-06-03T08:36:28.444764+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-06-04T08:41:33.679479+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,5],"acting":[3,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"17.13","version":"0'0","reported_seq":943,"reported_epoch":1009,"state":"active+clean","last_fresh":"2024-06-03T09:06:14.438399+0000","last_change":"2024-06-03T08:38:26.838316+0000","last_active":"2024-06-03T09:06:14.438399+0000","last_peered":"2024-06-03T09:06:14.438399+0000","last_clean":"2024-06-03T09:06:14.438399+0000","last_became_active":"2024-06-03T08:38:26.833214+0000","last_became_peered":"2024-06-03T08:38:26.833214+0000","last_unstale":"2024-06-03T09:06:14.438399+0000","last_undegraded":"2024-06-03T09:06:14.438399+0000","last_fullsized":"2024-06-03T09:06:14.438399+0000","mapping_epoch":28,"log_start":"0'0","ondisk_log_start":"0'0","created":28,"last_epoch_clean":52,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-06-03T08:37:55.614497+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-06-03T08:37:55.614497+0000","last_clean_scrub_stamp":"2024-06-03T08:37:55.614497+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-06-04T12:58:51.617998+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,0],"acting":[4,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"2.3","version":"0'0","reported_seq":943,"reported_epoch":1010,"state":"active+clean","last_fresh":"2024-06-03T09:06:18.362614+0000","last_change":"2024-06-03T08:36:30.351670+0000","last_active":"2024-06-03T09:06:18.362614+0000","last_peered":"2024-06-03T09:06:18.362614+0000","last_clean":"2024-06-03T09:06:18.362614+0000","last_became_active":"2024-06-03T08:36:30.343560+0000","last_became_peered":"2024-06-03T08:36:30.343560+0000","last_unstale":"2024-06-03T09:06:18.362614+0000","last_undegraded":"2024-06-03T09:06:18.362614+0000","last_fullsized":"2024-06-03T09:06:18.362614+0000","mapping_epoch":23,"log_start":"0'0","ondisk_log_start":"0'0","created":23,"last_epoch_clean":24,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-06-03T08:36:28.444764+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-06-03T08:36:28.444764+0000","last_clean_scrub_stamp":"2024-06-03T08:36:28.444764+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-06-04T12:02:24.765135+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,2],"acting":[3,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.0","version":"19'76","reported_seq":1051,"reported_epoch":1009,"state":"active+clean","last_fresh":"2024-06-03T09:06:13.703685+0000","last_change":"2024-06-03T08:36:26.702702+0000","last_active":"2024-06-03T09:06:13.703685+0000","last_peered":"2024-06-03T09:06:13.703685+0000","last_clean":"2024-06-03T09:06:13.703685+0000","last_became_active":"2024-06-03T08:36:26.128378+0000","last_became_peered":"2024-06-03T08:36:26.128378+0000","last_unstale":"2024-06-03T09:06:13.703685+0000","last_undegraded":"2024-06-03T09:06:13.703685+0000","last_fullsized":"2024-06-03T09:06:13.703685+0000","mapping_epoch":20,"log_start":"0'0","ondisk_log_start":"0'0","created":16,"last_epoch_clean":22,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-06-03T08:36:15.542635+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-06-03T08:36:15.542635+0000","last_clean_scrub_stamp":"2024-06-03T08:36:15.542635+0000","objects_scrubbed":0,"log_size":76,"log_dups_size":0,"ondisk_log_size":76,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-06-04T12:41:53.731360+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":459280,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":96,"num_read_kb":82,"num_write":113,"num_write_kb":1372,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2,"num_bytes_recovered":397840,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,5],"acting":[1,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"17.10","version":"0'0","reported_seq":944,"reported_epoch":1009,"state":"active+clean","last_fresh":"2024-06-03T09:06:14.422035+0000","last_change":"2024-06-03T08:38:15.399189+0000","last_active":"2024-06-03T09:06:14.422035+0000","last_peered":"2024-06-03T09:06:14.422035+0000","last_clean":"2024-06-03T09:06:14.422035+0000","last_became_active":"2024-06-03T08:38:15.337723+0000","last_became_peered":"2024-06-03T08:38:15.337723+0000","last_unstale":"2024-06-03T09:06:14.422035+0000","last_undegraded":"2024-06-03T09:06:14.422035+0000","last_fullsized":"2024-06-03T09:06:14.422035+0000","mapping_epoch":28,"log_start":"0'0","ondisk_log_start":"0'0","created":28,"last_epoch_clean":42,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-06-03T08:37:55.614497+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-06-03T08:37:55.614497+0000","last_clean_scrub_stamp":"2024-06-03T08:37:55.614497+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-06-04T09:29:46.427317+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,3],"acting":[4,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"17.1a","version":"0'0","reported_seq":960,"reported_epoch":1009,"state":"active+undersized","last_fresh":"2024-06-03T09:06:14.431224+0000","last_change":"2024-06-03T09:06:03.390734+0000","last_active":"2024-06-03T09:06:14.431224+0000","last_peered":"2024-06-03T09:06:14.431224+0000","last_clean":"2024-06-03T09:05:20.322313+0000","last_became_active":"2024-06-03T09:06:03.390734+0000","last_became_peered":"2024-06-03T09:06:03.390734+0000","last_unstale":"2024-06-03T09:06:14.431224+0000","last_undegraded":"2024-06-03T09:06:14.431224+0000","last_fullsized":"2024-06-03T09:06:03.343197+0000","mapping_epoch":999,"log_start":"0'0","ondisk_log_start":"0'0","created":28,"last_epoch_clean":254,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-06-03T08:37:55.614497+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-06-03T08:37:55.614497+0000","last_clean_scrub_stamp":"2024-06-03T08:37:55.614497+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-06-04T15:56:47.628875+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4],"acting":[4],"avail_no_missing":["4"],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"17.1b","version":"0'0","reported_seq":935,"reported_epoch":1009,"state":"active+clean","last_fresh":"2024-06-03T09:06:13.663279+0000","last_change":"2024-06-03T08:38:03.115748+0000","last_active":"2024-06-03T09:06:13.663279+0000","last_peered":"2024-06-03T09:06:13.663279+0000","last_clean":"2024-06-03T09:06:13.663279+0000","last_became_active":"2024-06-03T08:38:03.110533+0000","last_became_peered":"2024-06-03T08:38:03.110533+0000","last_unstale":"2024-06-03T09:06:13.663279+0000","last_undegraded":"2024-06-03T09:06:13.663279+0000","last_fullsized":"2024-06-03T09:06:13.663279+0000","mapping_epoch":28,"log_start":"0'0","ondisk_log_start":"0'0","created":28,"last_epoch_clean":32,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-06-03T08:37:55.614497+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-06-03T08:37:55.614497+0000","last_clean_scrub_stamp":"2024-06-03T08:37:55.614497+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-06-04T12:36:47.471609+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,7],"acting":[3,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"17.18","version":"0'0","reported_seq":944,"reported_epoch":1009,"state":"active+clean","last_fresh":"2024-06-03T09:06:13.924799+0000","last_change":"2024-06-03T08:38:04.632864+0000","last_active":"2024-06-03T09:06:13.924799+0000","last_peered":"2024-06-03T09:06:13.924799+0000","last_clean":"2024-06-03T09:06:13.924799+0000","last_became_active":"2024-06-03T08:38:04.627856+0000","last_became_peered":"2024-06-03T08:38:04.627856+0000","last_unstale":"2024-06-03T09:06:13.924799+0000","last_undegraded":"2024-06-03T09:06:13.924799+0000","last_fullsized":"2024-06-03T09:06:13.924799+0000","mapping_epoch":28,"log_start":"0'0","ondisk_log_start":"0'0","created":28,"last_epoch_clean":33,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-06-03T08:37:55.614497+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-06-03T08:37:55.614497+0000","last_clean_scrub_stamp":"2024-06-03T08:37:55.614497+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-06-04T19:55:51.181598+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,4],"acting":[7,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"17.19","version":"0'0","reported_seq":934,"reported_epoch":1009,"state":"active+clean","last_fresh":"2024-06-03T09:06:13.652943+0000","last_change":"2024-06-03T08:38:01.839491+0000","last_active":"2024-06-03T09:06:13.652943+0000","last_peered":"2024-06-03T09:06:13.652943+0000","last_clean":"2024-06-03T09:06:13.652943+0000","last_became_active":"2024-06-03T08:38:01.829539+0000","last_became_peered":"2024-06-03T08:38:01.829539+0000","last_unstale":"2024-06-03T09:06:13.652943+0000","last_undegraded":"2024-06-03T09:06:13.652943+0000","last_fullsized":"2024-06-03T09:06:13.652943+0000","mapping_epoch":28,"log_start":"0'0","ondisk_log_start":"0'0","created":28,"last_epoch_clean":30,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-06-03T08:37:55.614497+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-06-03T08:37:55.614497+0000","last_clean_scrub_stamp":"2024-06-03T08:37:55.614497+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-06-04T09:10:43.860229+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,2],"acting":[3,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"17.1e","version":"0'0","reported_seq":148,"reported_epoch":1009,"state":"active+undersized","last_fresh":"2024-06-03T09:06:14.445917+0000","last_change":"2024-06-03T09:06:03.406181+0000","last_active":"2024-06-03T09:06:14.445917+0000","last_peered":"2024-06-03T09:06:14.445917+0000","last_clean":"2024-06-03T08:37:55.614497+0000","last_became_active":"2024-06-03T09:06:03.406181+0000","last_became_peered":"2024-06-03T09:06:03.406181+0000","last_unstale":"2024-06-03T09:06:14.445917+0000","last_undegraded":"2024-06-03T09:06:14.445917+0000","last_fullsized":"2024-06-03T09:06:03.363739+0000","mapping_epoch":999,"log_start":"0'0","ondisk_log_start":"0'0","created":28,"last_epoch_clean":255,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-06-03T08:37:55.614497+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-06-03T08:37:55.614497+0000","last_clean_scrub_stamp":"2024-06-03T08:37:55.614497+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-06-04T11:45:24.486600+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4],"acting":[4],"avail_no_missing":["4"],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"17.1f","version":"0'0","reported_seq":941,"reported_epoch":1009,"state":"active+clean","last_fresh":"2024-06-03T09:06:14.450053+0000","last_change":"2024-06-03T08:38:04.383538+0000","last_active":"2024-06-03T09:06:14.450053+0000","last_peered":"2024-06-03T09:06:14.450053+0000","last_clean":"2024-06-03T09:06:14.450053+0000","last_became_active":"2024-06-03T08:38:04.367278+0000","last_became_peered":"2024-06-03T08:38:04.367278+0000","last_unstale":"2024-06-03T09:06:14.450053+0000","last_undegraded":"2024-06-03T09:06:14.450053+0000","last_fullsized":"2024-06-03T09:06:14.450053+0000","mapping_epoch":28,"log_start":"0'0","ondisk_log_start":"0'0","created":28,"last_epoch_clean":32,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-06-03T08:37:55.614497+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-06-03T08:37:55.614497+0000","last_clean_scrub_stamp":"2024-06-03T08:37:55.614497+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-06-04T10:39:26.552282+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,2],"acting":[0,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"17.1c","version":"0'0","reported_seq":944,"reported_epoch":1010,"state":"active+undersized","last_fresh":"2024-06-03T09:06:18.327492+0000","last_change":"2024-06-03T09:06:03.382210+0000","last_active":"2024-06-03T09:06:18.327492+0000","last_peered":"2024-06-03T09:06:18.327492+0000","last_clean":"2024-06-03T09:04:43.199660+0000","last_became_active":"2024-06-03T09:06:03.382210+0000","last_became_peered":"2024-06-03T09:06:03.382210+0000","last_unstale":"2024-06-03T09:06:18.327492+0000","last_undegraded":"2024-06-03T09:06:18.327492+0000","last_fullsized":"2024-06-03T09:06:03.277874+0000","mapping_epoch":999,"log_start":"0'0","ondisk_log_start":"0'0","created":28,"last_epoch_clean":255,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-06-03T08:37:55.614497+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-06-03T08:37:55.614497+0000","last_clean_scrub_stamp":"2024-06-03T08:37:55.614497+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-06-04T12:18:50.013884+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5],"acting":[5],"avail_no_missing":["5"],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"17.1d","version":"0'0","reported_seq":923,"reported_epoch":1009,"state":"active+clean","last_fresh":"2024-06-03T09:06:13.715906+0000","last_change":"2024-06-03T08:38:06.794630+0000","last_active":"2024-06-03T09:06:13.715906+0000","last_peered":"2024-06-03T09:06:13.715906+0000","last_clean":"2024-06-03T09:06:13.715906+0000","last_became_active":"2024-06-03T08:38:06.592317+0000","last_became_peered":"2024-06-03T08:38:06.592317+0000","last_unstale":"2024-06-03T09:06:13.715906+0000","last_undegraded":"2024-06-03T09:06:13.715906+0000","last_fullsized":"2024-06-03T09:06:13.715906+0000","mapping_epoch":28,"log_start":"0'0","ondisk_log_start":"0'0","created":28,"last_epoch_clean":33,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-06-03T08:37:55.614497+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-06-03T08:37:55.614497+0000","last_clean_scrub_stamp":"2024-06-03T08:37:55.614497+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-06-04T12:33:30.785369+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,7],"acting":[1,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"17.2","version":"0'0","reported_seq":925,"reported_epoch":1009,"state":"active+clean","last_fresh":"2024-06-03T09:06:13.708688+0000","last_change":"2024-06-03T08:38:14.981410+0000","last_active":"2024-06-03T09:06:13.708688+0000","last_peered":"2024-06-03T09:06:13.708688+0000","last_clean":"2024-06-03T09:06:13.708688+0000","last_became_active":"2024-06-03T08:38:14.973825+0000","last_became_peered":"2024-06-03T08:38:14.973825+0000","last_unstale":"2024-06-03T09:06:13.708688+0000","last_undegraded":"2024-06-03T09:06:13.708688+0000","last_fullsized":"2024-06-03T09:06:13.708688+0000","mapping_epoch":28,"log_start":"0'0","ondisk_log_start":"0'0","created":28,"last_epoch_clean":41,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-06-03T08:37:55.614497+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-06-03T08:37:55.614497+0000","last_clean_scrub_stamp":"2024-06-03T08:37:55.614497+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-06-04T17:01:05.518613+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,4],"acting":[1,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"17.3","version":"0'0","reported_seq":943,"reported_epoch":1009,"state":"active+clean","last_fresh":"2024-06-03T09:06:14.437636+0000","last_change":"2024-06-03T08:38:14.619745+0000","last_active":"2024-06-03T09:06:14.437636+0000","last_peered":"2024-06-03T09:06:14.437636+0000","last_clean":"2024-06-03T09:06:14.437636+0000","last_became_active":"2024-06-03T08:38:14.510665+0000","last_became_peered":"2024-06-03T08:38:14.510665+0000","last_unstale":"2024-06-03T09:06:14.437636+0000","last_undegraded":"2024-06-03T09:06:14.437636+0000","last_fullsized":"2024-06-03T09:06:14.437636+0000","mapping_epoch":28,"log_start":"0'0","ondisk_log_start":"0'0","created":28,"last_epoch_clean":42,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-06-03T08:37:55.614497+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-06-03T08:37:55.614497+0000","last_clean_scrub_stamp":"2024-06-03T08:37:55.614497+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-06-04T12:03:47.392064+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,7],"acting":[0,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"17.0","version":"0'0","reported_seq":944,"reported_epoch":1009,"state":"active+clean","last_fresh":"2024-06-03T09:06:14.427123+0000","last_change":"2024-06-03T08:38:04.237118+0000","last_active":"2024-06-03T09:06:14.427123+0000","last_peered":"2024-06-03T09:06:14.427123+0000","last_clean":"2024-06-03T09:06:14.427123+0000","last_became_active":"2024-06-03T08:38:04.231954+0000","last_became_peered":"2024-06-03T08:38:04.231954+0000","last_unstale":"2024-06-03T09:06:14.427123+0000","last_undegraded":"2024-06-03T09:06:14.427123+0000","last_fullsized":"2024-06-03T09:06:14.427123+0000","mapping_epoch":28,"log_start":"0'0","ondisk_log_start":"0'0","created":28,"last_epoch_clean":32,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-06-03T08:37:55.614497+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-06-03T08:37:55.614497+0000","last_clean_scrub_stamp":"2024-06-03T08:37:55.614497+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-06-04T09:56:10.741637+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,2],"acting":[0,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"17.1","version":"65'7","reported_seq":959,"reported_epoch":1009,"state":"active+undersized+degraded","last_fresh":"2024-06-03T09:06:13.709209+0000","last_change":"2024-06-03T09:06:03.370660+0000","last_active":"2024-06-03T09:06:13.709209+0000","last_peered":"2024-06-03T09:06:13.709209+0000","last_clean":"2024-06-03T09:05:03.733864+0000","last_became_active":"2024-06-03T09:06:03.370660+0000","last_became_peered":"2024-06-03T09:06:03.370660+0000","last_unstale":"2024-06-03T09:06:13.709209+0000","last_undegraded":"2024-06-03T09:06:03.266076+0000","last_fullsized":"2024-06-03T09:06:03.264525+0000","mapping_epoch":999,"log_start":"0'0","ondisk_log_start":"0'0","created":28,"last_epoch_clean":255,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-06-03T08:37:55.614497+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-06-03T08:37:55.614497+0000","last_clean_scrub_stamp":"2024-06-03T08:37:55.614497+0000","objects_scrubbed":0,"log_size":7,"log_dups_size":0,"ondisk_log_size":7,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-06-04T11:55:18.194283+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":5,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":1,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":6,"num_read_kb":6,"num_write":7,"num_write_kb":7,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5],"acting":[5],"avail_no_missing":["5"],"object_location_counts":[{"shards":"5","objects":1}],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]}],"pool_stats":[{"poolid":2,"num_pg":8,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":14,"acting":14,"num_store_stats":7},{"poolid":1,"num_pg":1,"stat_sum":{"num_bytes":459280,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":96,"num_read_kb":82,"num_write":113,"num_write_kb":1372,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2,"num_bytes_recovered":397840,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":802816,"data_stored":795680,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":76,"ondisk_log_size":76,"up":2,"acting":2,"num_store_stats":2},{"poolid":17,"num_pg":32,"stat_sum":{"num_bytes":5,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":1,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":6,"num_read_kb":6,"num_write":7,"num_write_kb":7,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":8192,"data_stored":10,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":7,"ondisk_log_size":7,"up":52,"acting":52,"num_store_stats":8}],"osd_stats":[{"osd":6,"up_from":253,"seq":1086626726200,"num_pgs":32,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":54772,"kb_used_data":5608,"kb_used_omap":1,"kb_used_meta":49150,"kb_avail":94317068,"statfs":{"total":96636764160,"available":96580677632,"internally_reserved":0,"allocated":5742592,"data_stored":2767180,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1593,"internal_metadata":50330055},"hb_peers":[0,1,2,3,4,5,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":109,"apply_latency_ms":109,"commit_latency_ns":109000000,"apply_latency_ns":109000000},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Mon Jun 3 09:04:02 2024","interfaces":[{"interface":"back","average":{"1min":1582.4780000000001,"5min":332.72699999999998,"15min":124.113},"min":{"1min":4.4829999999999997,"5min":4.1959999999999997,"15min":4.1959999999999997},"max":{"1min":22766.882000000001,"5min":22766.882000000001,"15min":22766.882000000001},"last":21.917999999999999},{"interface":"front","average":{"1min":1579.5409999999999,"5min":334.94499999999999,"15min":125.84999999999999},"min":{"1min":4.8129999999999997,"5min":4.7539999999999996,"15min":4.4530000000000003},"max":{"1min":22766.289000000001,"5min":22766.289000000001,"15min":22766.289000000001},"last":25.408999999999999}]},{"osd":1,"last update":"Mon Jun 3 09:03:40 2024","interfaces":[{"interface":"back","average":{"1min":36.420999999999999,"5min":77.054000000000002,"15min":132.84800000000001},"min":{"1min":6.0380000000000003,"5min":5.8799999999999999,"15min":5.484},"max":{"1min":589.64099999999996,"5min":4138.1679999999997,"15min":9955.1299999999992},"last":26.84},{"interface":"front","average":{"1min":40.240000000000002,"5min":79.674999999999997,"15min":133.38999999999999},"min":{"1min":8.4589999999999996,"5min":8.4090000000000007,"15min":6.1390000000000002},"max":{"1min":588.60799999999995,"5min":4137.6530000000002,"15min":9827.5509999999995},"last":26.263000000000002}]},{"osd":2,"last update":"Mon Jun 3 09:03:40 2024","interfaces":[{"interface":"back","average":{"1min":12.613,"5min":24.600999999999999,"15min":21.088000000000001},"min":{"1min":6.9740000000000002,"5min":6.9740000000000002,"15min":6.5330000000000004},"max":{"1min":26.076000000000001,"5min":376.24900000000002,"15min":376.24900000000002},"last":24.899000000000001},{"interface":"front","average":{"1min":12.493,"5min":22.207999999999998,"15min":21.219000000000001},"min":{"1min":6.5810000000000004,"5min":6.5110000000000001,"15min":6.0750000000000002},"max":{"1min":29.690999999999999,"5min":271.51100000000002,"15min":434.77300000000002},"last":40.372}]},{"osd":3,"last update":"Mon Jun 3 09:03:40 2024","interfaces":[{"interface":"back","average":{"1min":14.093999999999999,"5min":24.143999999999998,"15min":29.376999999999999},"min":{"1min":10.196999999999999,"5min":6.2919999999999998,"15min":6.1719999999999997},"max":{"1min":24.832000000000001,"5min":197.84899999999999,"15min":1235.5799999999999},"last":24.353999999999999},{"interface":"front","average":{"1min":9.3260000000000005,"5min":18.463999999999999,"15min":27.631},"min":{"1min":6.125,"5min":6.125,"15min":6.125},"max":{"1min":28.166,"5min":200.173,"15min":1157.6780000000001},"last":23.454000000000001}]},{"osd":4,"last update":"Mon Jun 3 09:04:29 2024","interfaces":[{"interface":"back","average":{"1min":10.279,"5min":1253.809,"15min":450.79199999999997},"min":{"1min":7.2300000000000004,"5min":6.093,"15min":6.093},"max":{"1min":15.000999999999999,"5min":28093.096000000001,"15min":28093.096000000001},"last":101.45399999999999},{"interface":"front","average":{"1min":15.693,"5min":1279.759,"15min":463.87099999999998},"min":{"1min":10.551,"5min":7.7480000000000002,"15min":7.7480000000000002},"max":{"1min":27.898,"5min":28323.837,"15min":28323.837},"last":102.262}]},{"osd":5,"last update":"Mon Jun 3 09:04:10 2024","interfaces":[{"interface":"back","average":{"1min":13.728999999999999,"5min":24.670000000000002,"15min":23.815999999999999},"min":{"1min":10.401,"5min":7.516,"15min":7.516},"max":{"1min":25.138000000000002,"5min":320.85000000000002,"15min":445.041},"last":103.52500000000001},{"interface":"front","average":{"1min":13.847,"5min":29.283000000000001,"15min":23.393999999999998},"min":{"1min":7.944,"5min":6.266,"15min":6.266},"max":{"1min":18.817,"5min":530.56700000000001,"15min":530.56700000000001},"last":29.263999999999999}]},{"osd":7,"last update":"Mon Jun 3 09:03:40 2024","interfaces":[{"interface":"back","average":{"1min":20.568999999999999,"5min":30.286999999999999,"15min":27.977},"min":{"1min":13.874000000000001,"5min":10.58,"15min":9.2859999999999996},"max":{"1min":86.959000000000003,"5min":222.07599999999999,"15min":255.08799999999999},"last":37.618000000000002},{"interface":"front","average":{"1min":19.102,"5min":29.550000000000001,"15min":27.763000000000002},"min":{"1min":10.052,"5min":8.6709999999999994,"15min":8.5429999999999993},"max":{"1min":67.968999999999994,"5min":293.62099999999998,"15min":438.14600000000002},"last":111.985}]}]},{"osd":7,"up_from":20,"seq":85899346286,"num_pgs":9,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":58964,"kb_used_data":5704,"kb_used_omap":1,"kb_used_meta":53246,"kb_avail":94312876,"statfs":{"total":96636764160,"available":96576385024,"internally_reserved":0,"allocated":5840896,"data_stored":2740727,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1589,"internal_metadata":54524363},"hb_peers":[0,1,2,3,4,5],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Mon Jun 3 09:06:09 2024","interfaces":[{"interface":"back","average":{"1min":14.073,"5min":894.22400000000005,"15min":308.93400000000003},"min":{"1min":5.5709999999999997,"5min":4.218,"15min":4.0599999999999996},"max":{"1min":90.757999999999996,"5min":21729.001,"15min":21729.001},"last":4.032},{"interface":"front","average":{"1min":9.2550000000000008,"5min":906.23500000000001,"15min":314.06},"min":{"1min":5.5789999999999997,"5min":4.8810000000000002,"15min":4.8810000000000002},"max":{"1min":25.937000000000001,"5min":21899.732,"15min":21899.732},"last":7.5819999999999999}]},{"osd":1,"last update":"Mon Jun 3 09:06:09 2024","interfaces":[{"interface":"back","average":{"1min":15.182,"5min":69.183999999999997,"15min":219.173},"min":{"1min":7.3639999999999999,"5min":6.6200000000000001,"15min":6.1319999999999997},"max":{"1min":76.926000000000002,"5min":2795.1329999999998,"15min":18373.257000000001},"last":9.7349999999999994},{"interface":"front","average":{"1min":21.495999999999999,"5min":74.703999999999994,"15min":225.285},"min":{"1min":8.8399999999999999,"5min":8.8399999999999999,"15min":8.5129999999999999},"max":{"1min":94.977999999999994,"5min":2803.3670000000002,"15min":18374.601999999999},"last":13.657999999999999}]},{"osd":2,"last update":"Mon Jun 3 09:06:09 2024","interfaces":[{"interface":"back","average":{"1min":19.164000000000001,"5min":26.902000000000001,"15min":22.920999999999999},"min":{"1min":11.207000000000001,"5min":8.5030000000000001,"15min":8.5030000000000001},"max":{"1min":91.319999999999993,"5min":333.99200000000002,"15min":333.99200000000002},"last":12.694000000000001},{"interface":"front","average":{"1min":11.125999999999999,"5min":18.132000000000001,"15min":16.353000000000002},"min":{"1min":6.202,"5min":5.8369999999999997,"15min":5.5069999999999997},"max":{"1min":28.649000000000001,"5min":226.18100000000001,"15min":226.18100000000001},"last":10.028}]},{"osd":3,"last update":"Mon Jun 3 09:06:09 2024","interfaces":[{"interface":"back","average":{"1min":22.291,"5min":26.289999999999999,"15min":31.074999999999999},"min":{"1min":11.714,"5min":6.8959999999999999,"15min":6.3760000000000003},"max":{"1min":83.102000000000004,"5min":316.64400000000001,"15min":2039.9059999999999},"last":14.345000000000001},{"interface":"front","average":{"1min":13.324999999999999,"5min":17.963000000000001,"15min":25.969999999999999},"min":{"1min":6.1200000000000001,"5min":5.2089999999999996,"15min":5.2089999999999996},"max":{"1min":49.908999999999999,"5min":233.542,"15min":2037.5550000000001},"last":9.5060000000000002}]},{"osd":4,"last update":"Mon Jun 3 09:05:28 2024","interfaces":[{"interface":"back","average":{"1min":22.209,"5min":1379.51,"15min":1803.278},"min":{"1min":5.9089999999999998,"5min":5.9089999999999998,"15min":5.9089999999999998},"max":{"1min":224.54300000000001,"5min":30073.917000000001,"15min":62560.868000000002},"last":11.138},{"interface":"front","average":{"1min":22.712,"5min":1401.0719999999999,"15min":1806.759},"min":{"1min":7.6449999999999996,"5min":6.7930000000000001,"15min":5.9340000000000002},"max":{"1min":223.58600000000001,"5min":30348.021000000001,"15min":62451.446000000004},"last":6.569}]},{"osd":5,"last update":"Mon Jun 3 09:06:02 2024","interfaces":[{"interface":"back","average":{"1min":24.372,"5min":26.856000000000002,"15min":23.571999999999999},"min":{"1min":12.278,"5min":9.2460000000000004,"15min":6.5650000000000004},"max":{"1min":183.64099999999999,"5min":317.572,"15min":317.572},"last":15.047000000000001},{"interface":"front","average":{"1min":21.259,"5min":23.364000000000001,"15min":21.193999999999999},"min":{"1min":7.8499999999999996,"5min":7.0999999999999996,"15min":7.008},"max":{"1min":184.53399999999999,"5min":306.60599999999999,"15min":306.60599999999999},"last":11.656000000000001}]},{"osd":6,"last update":"Mon Jun 3 09:04:31 2024","interfaces":[{"interface":"back","average":{"1min":44.061,"5min":26.821000000000002,"15min":91.176000000000002},"min":{"1min":9.0800000000000001,"5min":7.9390000000000001,"15min":7.9390000000000001},"max":{"1min":268.029,"5min":308.916,"15min":8734.4639999999999},"last":6.6769999999999996},{"interface":"front","average":{"1min":46.643000000000001,"5min":23.879999999999999,"15min":88.765000000000001},"min":{"1min":8.8119999999999994,"5min":6.9240000000000004,"15min":6.742},"max":{"1min":305.64600000000002,"5min":305.64600000000002,"15min":8735.5319999999992},"last":12.327}]}]},{"osd":5,"up_from":20,"seq":85899346260,"num_pgs":9,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":59360,"kb_used_data":6100,"kb_used_omap":1,"kb_used_meta":53246,"kb_avail":94312480,"statfs":{"total":96636764160,"available":96575979520,"internally_reserved":0,"allocated":6246400,"data_stored":3138572,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1594,"internal_metadata":54524358},"hb_peers":[0,1,2,3,4,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Mon Jun 3 09:06:06 2024","interfaces":[{"interface":"back","average":{"1min":7.468,"5min":1703.037,"15min":580.51900000000001},"min":{"1min":4.6509999999999998,"5min":4.6509999999999998,"15min":4.4020000000000001},"max":{"1min":14.711,"5min":26433.764999999999,"15min":26433.764999999999},"last":9.3849999999999998},{"interface":"front","average":{"1min":9.9429999999999996,"5min":1725.27,"15min":587.87800000000004},"min":{"1min":5.5999999999999996,"5min":5.1319999999999997,"15min":4.8170000000000002},"max":{"1min":13.590999999999999,"5min":26704.698,"15min":26704.698},"last":12.407999999999999}]},{"osd":1,"last update":"Mon Jun 3 09:06:00 2024","interfaces":[{"interface":"back","average":{"1min":10.862,"5min":67.731999999999999,"15min":217.364},"min":{"1min":7.1769999999999996,"5min":6.0890000000000004,"15min":5.4930000000000003},"max":{"1min":16.998999999999999,"5min":3923.4090000000001,"15min":18811.696},"last":12.917999999999999},{"interface":"front","average":{"1min":10.417,"5min":65.141999999999996,"15min":220.75700000000001},"min":{"1min":5.3209999999999997,"5min":5.3209999999999997,"15min":5.0209999999999999},"max":{"1min":15.821999999999999,"5min":4019.6129999999998,"15min":18968.365000000002},"last":8.6590000000000007}]},{"osd":2,"last update":"Mon Jun 3 09:06:00 2024","interfaces":[{"interface":"back","average":{"1min":11.366,"5min":24.18,"15min":21.137},"min":{"1min":5.6369999999999996,"5min":5.6369999999999996,"15min":5.0510000000000002},"max":{"1min":15.529999999999999,"5min":332.48200000000003,"15min":332.48200000000003},"last":13.375},{"interface":"front","average":{"1min":10.089,"5min":23.117000000000001,"15min":24.050000000000001},"min":{"1min":6.6769999999999996,"5min":6.6769999999999996,"15min":5.8739999999999997},"max":{"1min":15.821,"5min":353.03199999999998,"15min":353.03199999999998},"last":8.5139999999999993}]},{"osd":3,"last update":"Mon Jun 3 09:06:00 2024","interfaces":[{"interface":"back","average":{"1min":13.472,"5min":24.283999999999999,"15min":31.666},"min":{"1min":10.111000000000001,"5min":7.4720000000000004,"15min":7.4210000000000003},"max":{"1min":16.271999999999998,"5min":353.57499999999999,"15min":2855.8519999999999},"last":9.9960000000000004},{"interface":"front","average":{"1min":9.5899999999999999,"5min":15.875,"15min":25.834},"min":{"1min":6.6609999999999996,"5min":5.7729999999999997,"15min":5.1909999999999998},"max":{"1min":15.112,"5min":203.804,"15min":2869.7860000000001},"last":6.3769999999999998}]},{"osd":4,"last update":"Mon Jun 3 09:05:27 2024","interfaces":[{"interface":"back","average":{"1min":19.155000000000001,"5min":1084.961,"15min":1346.9949999999999},"min":{"1min":7.6130000000000004,"5min":7.6130000000000004,"15min":6.1340000000000003},"max":{"1min":156.55199999999999,"5min":28619.825000000001,"15min":57999.453999999998},"last":14.244999999999999},{"interface":"front","average":{"1min":28.068999999999999,"5min":1087.106,"15min":1330.289},"min":{"1min":8.3949999999999996,"5min":6.5030000000000001,"15min":6.5030000000000001},"max":{"1min":254.91300000000001,"5min":28621.294000000002,"15min":57434.631999999998},"last":14.859}]},{"osd":6,"last update":"Mon Jun 3 09:03:57 2024","interfaces":[{"interface":"back","average":{"1min":23.536000000000001,"5min":36.457000000000001,"15min":83.799999999999997},"min":{"1min":9.4800000000000004,"5min":9.4800000000000004,"15min":8.0630000000000006},"max":{"1min":89.442999999999998,"5min":508.77999999999997,"15min":9193.5169999999998},"last":157.79300000000001},{"interface":"front","average":{"1min":18.891999999999999,"5min":22.100999999999999,"15min":75.489999999999995},"min":{"1min":5.923,"5min":5.923,"15min":5.923},"max":{"1min":78.424999999999997,"5min":464.62,"15min":9191.4429999999993},"last":148.09}]},{"osd":7,"last update":"Mon Jun 3 09:06:00 2024","interfaces":[{"interface":"back","average":{"1min":13.257,"5min":20.917999999999999,"15min":21.088999999999999},"min":{"1min":10.475,"5min":6.7629999999999999,"15min":6.7629999999999999},"max":{"1min":17.245000000000001,"5min":211.22999999999999,"15min":211.22999999999999},"last":15.929},{"interface":"front","average":{"1min":12.872999999999999,"5min":20.587,"15min":24.021000000000001},"min":{"1min":8.5760000000000005,"5min":7.7709999999999999,"15min":6.8570000000000002},"max":{"1min":18.359999999999999,"5min":223.191,"15min":286.69499999999999},"last":10.608000000000001}]}]},{"osd":4,"up_from":20,"seq":85899346268,"num_pgs":12,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":63060,"kb_used_data":5704,"kb_used_omap":1,"kb_used_meta":57342,"kb_avail":94308780,"statfs":{"total":96636764160,"available":96572190720,"internally_reserved":0,"allocated":5840896,"data_stored":2740727,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1594,"internal_metadata":58718662},"hb_peers":[0,1,2,3,5,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Mon Jun 3 09:05:34 2024","interfaces":[{"interface":"back","average":{"1min":12.416,"5min":1099.4649999999999,"15min":382.15199999999999},"min":{"1min":4.5279999999999996,"5min":4.5279999999999996,"15min":4.3220000000000001},"max":{"1min":123.878,"5min":21726.743999999999,"15min":21726.743999999999},"last":6.7329999999999997},{"interface":"front","average":{"1min":16.462,"5min":1108.894,"15min":387.70299999999997},"min":{"1min":6.4089999999999998,"5min":5.8380000000000001,"15min":5.4909999999999997},"max":{"1min":130.185,"5min":21727.993999999999,"15min":21727.993999999999},"last":9.3309999999999995}]},{"osd":1,"last update":"Mon Jun 3 09:05:34 2024","interfaces":[{"interface":"back","average":{"1min":14.727,"5min":56.755000000000003,"15min":341.09399999999999},"min":{"1min":4.6529999999999996,"5min":4.1970000000000001,"15min":4.1970000000000001},"max":{"1min":126.899,"5min":3346.4389999999999,"15min":22256.098000000002},"last":15.904},{"interface":"front","average":{"1min":13.472,"5min":57.503,"15min":340.303},"min":{"1min":5.0590000000000002,"5min":5.0590000000000002,"15min":4.2640000000000002},"max":{"1min":126.096,"5min":3347.181,"15min":22247.317999999999},"last":8.0199999999999996}]},{"osd":2,"last update":"Mon Jun 3 09:05:34 2024","interfaces":[{"interface":"back","average":{"1min":26.088999999999999,"5min":23.866,"15min":20.620999999999999},"min":{"1min":7.0869999999999997,"5min":6.8239999999999998,"15min":6.3150000000000004},"max":{"1min":337.60399999999998,"5min":393.58600000000001,"15min":393.58600000000001},"last":9.9429999999999996},{"interface":"front","average":{"1min":15.800000000000001,"5min":22.584,"15min":23.245999999999999},"min":{"1min":6.0629999999999997,"5min":6.0629999999999997,"15min":6.0629999999999997},"max":{"1min":127.913,"5min":394.471,"15min":394.471},"last":6.4889999999999999}]},{"osd":3,"last update":"Mon Jun 3 09:05:34 2024","interfaces":[{"interface":"back","average":{"1min":17.544,"5min":19.143999999999998,"15min":18.742000000000001},"min":{"1min":5.8719999999999999,"5min":5.4100000000000001,"15min":5.4000000000000004},"max":{"1min":127.43600000000001,"5min":322.63,"15min":322.63},"last":11.32},{"interface":"front","average":{"1min":21.327000000000002,"5min":23.559000000000001,"15min":23.797000000000001},"min":{"1min":10.023,"5min":7.484,"15min":7.484},"max":{"1min":204.517,"5min":392.67500000000001,"15min":392.67500000000001},"last":13.484999999999999}]},{"osd":5,"last update":"Mon Jun 3 09:05:34 2024","interfaces":[{"interface":"back","average":{"1min":24.274999999999999,"5min":21.274999999999999,"15min":20.988},"min":{"1min":6.7489999999999997,"5min":6.7489999999999997,"15min":6.7489999999999997},"max":{"1min":268.10199999999998,"5min":344.23399999999998,"15min":344.23399999999998},"last":8.8209999999999997},{"interface":"front","average":{"1min":26.361999999999998,"5min":25.218,"15min":26.084},"min":{"1min":12.228,"5min":8.5180000000000007,"15min":8.5180000000000007},"max":{"1min":264.75,"5min":345.18700000000001,"15min":345.18700000000001},"last":12.875}]},{"osd":6,"last update":"Mon Jun 3 09:04:32 2024","interfaces":[{"interface":"back","average":{"1min":15.548,"5min":32.957999999999998,"15min":152.738},"min":{"1min":9.141,"5min":9.141,"15min":8.2319999999999993},"max":{"1min":52.57,"5min":395.238,"15min":12496.335999999999},"last":208.90000000000001},{"interface":"front","average":{"1min":12.680999999999999,"5min":25.428000000000001,"15min":149.30600000000001},"min":{"1min":6.4379999999999997,"5min":5.6319999999999997,"15min":5.6319999999999997},"max":{"1min":46.283000000000001,"5min":377.74299999999999,"15min":12616.297},"last":205.88900000000001}]},{"osd":7,"last update":"Mon Jun 3 09:05:34 2024","interfaces":[{"interface":"back","average":{"1min":18.295999999999999,"5min":19.585999999999999,"15min":21.062000000000001},"min":{"1min":6.2110000000000003,"5min":6.2110000000000003,"15min":5.8010000000000002},"max":{"1min":207.25299999999999,"5min":358.94900000000001,"15min":358.94900000000001},"last":14.076000000000001},{"interface":"front","average":{"1min":23.712,"5min":24.024999999999999,"15min":24.640000000000001},"min":{"1min":11.824999999999999,"5min":7.7789999999999999,"15min":7.7789999999999999},"max":{"1min":209.32900000000001,"5min":346.25200000000001,"15min":380.06900000000002},"last":14.923}]}]},{"osd":2,"up_from":20,"seq":85899346287,"num_pgs":11,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":58964,"kb_used_data":5704,"kb_used_omap":1,"kb_used_meta":53246,"kb_avail":94312876,"statfs":{"total":96636764160,"available":96576385024,"internally_reserved":0,"allocated":5840896,"data_stored":2740727,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1594,"internal_metadata":54524358},"hb_peers":[0,1,3,4,5,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":4,"apply_latency_ms":4,"commit_latency_ns":4000000,"apply_latency_ns":4000000},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Mon Jun 3 09:05:21 2024","interfaces":[{"interface":"back","average":{"1min":16.364000000000001,"5min":947.59799999999996,"15min":326.58999999999997},"min":{"1min":6.0830000000000002,"5min":4.7050000000000001,"15min":4.1420000000000003},"max":{"1min":157.60900000000001,"5min":23465.18,"15min":23465.18},"last":6.8499999999999996},{"interface":"front","average":{"1min":12.276,"5min":945.29899999999998,"15min":326.78399999999999},"min":{"1min":5.1989999999999998,"5min":4.7439999999999998,"15min":4.6950000000000003},"max":{"1min":102.557,"5min":23465.966,"15min":23465.966},"last":5.8600000000000003}]},{"osd":1,"last update":"Mon Jun 3 09:05:21 2024","interfaces":[{"interface":"back","average":{"1min":17.803999999999998,"5min":77.658000000000001,"15min":294.96899999999999},"min":{"1min":7.2130000000000001,"5min":5.4199999999999999,"15min":5.4199999999999999},"max":{"1min":159.624,"5min":4220.5290000000005,"15min":18954.987000000001},"last":10.423999999999999},{"interface":"front","average":{"1min":21.542999999999999,"5min":83.528999999999996,"15min":303.11700000000002},"min":{"1min":8.8200000000000003,"5min":7.5949999999999998,"15min":7.5949999999999998},"max":{"1min":170.74299999999999,"5min":4325.9489999999996,"15min":19143.834999999999},"last":11.566000000000001}]},{"osd":3,"last update":"Mon Jun 3 09:05:21 2024","interfaces":[{"interface":"back","average":{"1min":19.164999999999999,"5min":13.284000000000001,"15min":28.751999999999999},"min":{"1min":7.3710000000000004,"5min":5.9290000000000003,"15min":5.2450000000000001},"max":{"1min":118.913,"5min":118.913,"15min":2594.8150000000001},"last":8.3040000000000003},{"interface":"front","average":{"1min":22.094000000000001,"5min":17.244,"15min":32.173999999999999},"min":{"1min":9.516,"5min":8.0760000000000005,"15min":6.0780000000000003},"max":{"1min":156.34700000000001,"5min":156.34700000000001,"15min":2706.5120000000002},"last":13.273999999999999}]},{"osd":4,"last update":"Mon Jun 3 09:05:57 2024","interfaces":[{"interface":"back","average":{"1min":10.573,"5min":1266.894,"15min":438.154},"min":{"1min":8.6379999999999999,"5min":7.8789999999999996,"15min":6.2039999999999997},"max":{"1min":15.105,"5min":26660.861000000001,"15min":26660.861000000001},"last":9.673},{"interface":"front","average":{"1min":12.407999999999999,"5min":1266.7460000000001,"15min":435.75400000000002},"min":{"1min":10.266999999999999,"5min":10.266999999999999,"15min":6.2720000000000002},"max":{"1min":15.228,"5min":26616.243999999999,"15min":26616.243999999999},"last":14.26}]},{"osd":5,"last update":"Mon Jun 3 09:05:21 2024","interfaces":[{"interface":"back","average":{"1min":25.632000000000001,"5min":20.437999999999999,"15min":21.472999999999999},"min":{"1min":9.2159999999999993,"5min":9.2159999999999993,"15min":6.2050000000000001},"max":{"1min":252.31100000000001,"5min":252.31100000000001,"15min":252.31100000000001},"last":14.960000000000001},{"interface":"front","average":{"1min":24.824000000000002,"5min":18.521000000000001,"15min":21.027000000000001},"min":{"1min":8.0449999999999999,"5min":7.4850000000000003,"15min":6.8810000000000002},"max":{"1min":258.31700000000001,"5min":258.31700000000001,"15min":258.31700000000001},"last":12.413}]},{"osd":6,"last update":"Mon Jun 3 09:04:28 2024","interfaces":[{"interface":"back","average":{"1min":14.063000000000001,"5min":24.850000000000001,"15min":106.755},"min":{"1min":9.0820000000000007,"5min":8.9649999999999999,"15min":8.9649999999999999},"max":{"1min":24.666,"5min":188.06700000000001,"15min":9521.0010000000002},"last":183.60900000000001},{"interface":"front","average":{"1min":13.933999999999999,"5min":21.081,"15min":106.142},"min":{"1min":5.8719999999999999,"5min":5.8719999999999999,"15min":5.8719999999999999},"max":{"1min":117.551,"5min":193.887,"15min":9585.6620000000003},"last":184.20599999999999}]},{"osd":7,"last update":"Mon Jun 3 09:05:21 2024","interfaces":[{"interface":"back","average":{"1min":25.097000000000001,"5min":22.579999999999998,"15min":24.388000000000002},"min":{"1min":7.8959999999999999,"5min":6.6130000000000004,"15min":6.6130000000000004},"max":{"1min":242.56800000000001,"5min":242.56800000000001,"15min":242.56800000000001},"last":15.646000000000001},{"interface":"front","average":{"1min":24.414999999999999,"5min":23.995999999999999,"15min":26.614999999999998},"min":{"1min":12.43,"5min":9.0660000000000007,"15min":9.0660000000000007},"max":{"1min":181.56399999999999,"5min":182.5,"15min":214.33799999999999},"last":16.282}]}]},{"osd":3,"up_from":14,"seq":60129542502,"num_pgs":11,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":58964,"kb_used_data":5704,"kb_used_omap":1,"kb_used_meta":53246,"kb_avail":94312876,"statfs":{"total":96636764160,"available":96576385024,"internally_reserved":0,"allocated":5840896,"data_stored":2740727,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1592,"internal_metadata":54524360},"hb_peers":[0,1,2,4,5,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Mon Jun 3 09:05:25 2024","interfaces":[{"interface":"back","average":{"1min":8.4809999999999999,"5min":1064.2270000000001,"15min":367.88600000000002},"min":{"1min":6.0529999999999999,"5min":5.173,"15min":4.6929999999999996},"max":{"1min":12.010999999999999,"5min":23275.167000000001,"15min":23275.167000000001},"last":8.7289999999999992},{"interface":"front","average":{"1min":13.885,"5min":1070.3710000000001,"15min":373.53500000000003},"min":{"1min":9.6679999999999993,"5min":7.2089999999999996,"15min":5.9880000000000004},"max":{"1min":32.426000000000002,"5min":23278.536,"15min":23278.536},"last":12.621}]},{"osd":1,"last update":"Mon Jun 3 09:05:25 2024","interfaces":[{"interface":"back","average":{"1min":11.324,"5min":19.920000000000002,"15min":390.07900000000001},"min":{"1min":6.6379999999999999,"5min":4.8200000000000003,"15min":4.8200000000000003},"max":{"1min":26.449000000000002,"5min":250.72300000000001,"15min":21658.205999999998},"last":9.7889999999999997},{"interface":"front","average":{"1min":9.468,"5min":20.966999999999999,"15min":381.18799999999999},"min":{"1min":5.4969999999999999,"5min":5.2779999999999996,"15min":5.2779999999999996},"max":{"1min":25.643000000000001,"5min":335.33699999999999,"15min":21307.069},"last":7.8179999999999996}]},{"osd":2,"last update":"Mon Jun 3 09:05:25 2024","interfaces":[{"interface":"back","average":{"1min":13.215,"5min":26.649000000000001,"15min":23.783999999999999},"min":{"1min":7.8710000000000004,"5min":6.4900000000000002,"15min":5.875},"max":{"1min":34.322000000000003,"5min":377.803,"15min":377.803},"last":14.018000000000001},{"interface":"front","average":{"1min":13.105,"5min":22.292999999999999,"15min":20.573},"min":{"1min":7.327,"5min":6.125,"15min":5.9020000000000001},"max":{"1min":32.756999999999998,"5min":371.601,"15min":371.601},"last":6.9630000000000001}]},{"osd":4,"last update":"Mon Jun 3 09:05:27 2024","interfaces":[{"interface":"back","average":{"1min":10.444000000000001,"5min":1211.4970000000001,"15min":1639.1579999999999},"min":{"1min":6.4429999999999996,"5min":6.2560000000000002,"15min":6.0309999999999997},"max":{"1min":21.263000000000002,"5min":30144.939999999999,"15min":62679.622000000003},"last":5.6539999999999999},{"interface":"front","average":{"1min":11.363,"5min":1219.6400000000001,"15min":1644.127},"min":{"1min":7.8109999999999999,"5min":7.8109999999999999,"15min":6.2439999999999998},"max":{"1min":20.108000000000001,"5min":30180.689999999999,"15min":62681.021999999997},"last":11.103999999999999}]},{"osd":5,"last update":"Mon Jun 3 09:06:07 2024","interfaces":[{"interface":"back","average":{"1min":12.802,"5min":19.640999999999998,"15min":22.318000000000001},"min":{"1min":9.0890000000000004,"5min":6.8810000000000002,"15min":6.7690000000000001},"max":{"1min":16.745999999999999,"5min":249.88,"15min":327.34899999999999},"last":13.329000000000001},{"interface":"front","average":{"1min":15.002000000000001,"5min":24.003,"15min":26.649999999999999},"min":{"1min":11.984,"5min":9.9149999999999991,"15min":8.9879999999999995},"max":{"1min":18.407,"5min":251.47800000000001,"15min":331.86799999999999},"last":14.683999999999999}]},{"osd":6,"last update":"Mon Jun 3 09:04:04 2024","interfaces":[{"interface":"back","average":{"1min":22.815999999999999,"5min":31.513999999999999,"15min":181.74299999999999},"min":{"1min":9.4809999999999999,"5min":7.2240000000000002,"15min":7.2240000000000002},"max":{"1min":160.56999999999999,"5min":380.613,"15min":11893.74},"last":19.359999999999999},{"interface":"front","average":{"1min":25.359999999999999,"5min":30.981000000000002,"15min":179.81399999999999},"min":{"1min":9.8870000000000005,"5min":7.3360000000000003,"15min":6.0899999999999999},"max":{"1min":193.11099999999999,"5min":379.80700000000002,"15min":11893.069},"last":8.202}]},{"osd":7,"last update":"Mon Jun 3 09:05:25 2024","interfaces":[{"interface":"back","average":{"1min":12.177,"5min":21.126000000000001,"15min":22.972000000000001},"min":{"1min":8.6769999999999996,"5min":7.4279999999999999,"15min":7.4279999999999999},"max":{"1min":17.376999999999999,"5min":164.27500000000001,"15min":178.76300000000001},"last":9.3070000000000004},{"interface":"front","average":{"1min":15.675000000000001,"5min":22.722000000000001,"15min":25.238},"min":{"1min":11.233000000000001,"5min":10.555999999999999,"15min":7.6440000000000001},"max":{"1min":29.280000000000001,"5min":148.13800000000001,"15min":417.22800000000001},"last":10.32}]}]},{"osd":1,"up_from":14,"seq":60129542500,"num_pgs":6,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":59356,"kb_used_data":6096,"kb_used_omap":1,"kb_used_meta":53246,"kb_avail":94312484,"statfs":{"total":96636764160,"available":96575983616,"internally_reserved":0,"allocated":6242304,"data_stored":3138567,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":54524357},"hb_peers":[0,2,3,4,5,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Mon Jun 3 09:06:07 2024","interfaces":[{"interface":"back","average":{"1min":7.7530000000000001,"5min":185.66999999999999,"15min":77.347999999999999},"min":{"1min":5.0229999999999997,"5min":4.9279999999999999,"15min":4.5449999999999999},"max":{"1min":11.896000000000001,"5min":7470.616,"15min":7470.616},"last":7.2610000000000001},{"interface":"front","average":{"1min":9.9220000000000006,"5min":187.14599999999999,"15min":76.325999999999993},"min":{"1min":5.3159999999999998,"5min":5.3159999999999998,"15min":5.3159999999999998},"max":{"1min":18.338000000000001,"5min":7477.0829999999996,"15min":7477.0829999999996},"last":13.295999999999999}]},{"osd":2,"last update":"Mon Jun 3 09:06:20 2024","interfaces":[{"interface":"back","average":{"1min":10.340999999999999,"5min":17.369,"15min":23.670999999999999},"min":{"1min":6.9740000000000002,"5min":5.8650000000000002,"15min":5.8650000000000002},"max":{"1min":15.452,"5min":228,"15min":657.38},"last":9.1630000000000003},{"interface":"front","average":{"1min":13.387,"5min":23.355,"15min":31.524000000000001},"min":{"1min":9.077,"5min":9.077,"15min":7.2160000000000002},"max":{"1min":19.701000000000001,"5min":229.465,"15min":700.06500000000005},"last":14.991}]},{"osd":3,"last update":"Mon Jun 3 09:06:04 2024","interfaces":[{"interface":"back","average":{"1min":8.6110000000000007,"5min":15.733000000000001,"15min":26.646999999999998},"min":{"1min":5.827,"5min":5.4640000000000004,"15min":5.4640000000000004},"max":{"1min":13.105,"5min":121.619,"15min":1676.009},"last":6.2830000000000004},{"interface":"front","average":{"1min":9.5850000000000009,"5min":16.161999999999999,"15min":29.065999999999999},"min":{"1min":6.3630000000000004,"5min":5.7699999999999996,"15min":5.7699999999999996},"max":{"1min":15.166,"5min":132.036,"15min":1766.5840000000001},"last":11.605}]},{"osd":4,"last update":"Mon Jun 3 09:05:31 2024","interfaces":[{"interface":"back","average":{"1min":10.996,"5min":954.46500000000003,"15min":1281.6859999999999},"min":{"1min":7.702,"5min":6.1760000000000002,"15min":6.0970000000000004},"max":{"1min":21.538,"5min":28044.862000000001,"15min":59438.017999999996},"last":8.6010000000000009},{"interface":"front","average":{"1min":13.646000000000001,"5min":961.24599999999998,"15min":1285.26},"min":{"1min":9.7200000000000006,"5min":9.3940000000000001,"15min":6.4640000000000004},"max":{"1min":23.277999999999999,"5min":28045.624,"15min":59439.283000000003},"last":10.781000000000001}]},{"osd":5,"last update":"Mon Jun 3 09:06:04 2024","interfaces":[{"interface":"back","average":{"1min":12.478999999999999,"5min":20.228000000000002,"15min":25.997},"min":{"1min":8.6430000000000007,"5min":7.9720000000000004,"15min":7.234},"max":{"1min":17.442,"5min":139.43199999999999,"15min":701.22000000000003},"last":10.263999999999999},{"interface":"front","average":{"1min":13.507999999999999,"5min":25.760999999999999,"15min":27.553999999999998},"min":{"1min":10.042999999999999,"5min":7.258,"15min":6.242},"max":{"1min":20.611000000000001,"5min":278.63,"15min":598.35000000000002},"last":15.890000000000001}]},{"osd":6,"last update":"Mon Jun 3 09:03:54 2024","interfaces":[{"interface":"back","average":{"1min":28.524000000000001,"5min":20.654,"15min":23.878},"min":{"1min":6.0309999999999997,"5min":6.0309999999999997,"15min":6.0309999999999997},"max":{"1min":167.221,"5min":167.221,"15min":704.245},"last":18.721},{"interface":"front","average":{"1min":27.606000000000002,"5min":30.131,"15min":28.628},"min":{"1min":9.8650000000000002,"5min":9.7200000000000006,"15min":8.3580000000000005},"max":{"1min":123.938,"5min":358.23000000000002,"15min":634.05499999999995},"last":173.113}]},{"osd":7,"last update":"Mon Jun 3 09:06:20 2024","interfaces":[{"interface":"back","average":{"1min":14.66,"5min":23.234999999999999,"15min":31.097000000000001},"min":{"1min":11.250999999999999,"5min":8.0559999999999992,"15min":8.0559999999999992},"max":{"1min":21.507999999999999,"5min":270.95100000000002,"15min":703.07799999999997},"last":14.182},{"interface":"front","average":{"1min":15.548,"5min":28.760000000000002,"15min":28.925000000000001},"min":{"1min":12.759,"5min":8.2089999999999996,"15min":7.7599999999999998},"max":{"1min":22.408000000000001,"5min":360.99900000000002,"15min":608.44100000000003},"last":16.588000000000001}]}]},{"osd":0,"up_from":14,"seq":60129542502,"num_pgs":10,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":81108,"kb_used_data":5704,"kb_used_omap":1,"kb_used_meta":75390,"kb_avail":94290732,"statfs":{"total":96636764160,"available":96553709568,"internally_reserved":0,"allocated":5840896,"data_stored":2740727,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1593,"internal_metadata":77199815},"hb_peers":[1,2,3,4,5,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[{"osd":1,"last update":"Mon Jun 3 09:06:04 2024","interfaces":[{"interface":"back","average":{"1min":6.9960000000000004,"5min":56.997,"15min":339.29700000000003},"min":{"1min":4.4530000000000003,"5min":4.4530000000000003,"15min":4.4100000000000001},"max":{"1min":11.720000000000001,"5min":2414.4749999999999,"15min":20691.453000000001},"last":4.6040000000000001},{"interface":"front","average":{"1min":9.548,"5min":57.258000000000003,"15min":337.791},"min":{"1min":5.2300000000000004,"5min":5.2300000000000004,"15min":4.6050000000000004},"max":{"1min":15.337999999999999,"5min":2412.9549999999999,"15min":20640.792000000001},"last":7.7910000000000004}]},{"osd":2,"last update":"Mon Jun 3 09:06:22 2024","interfaces":[{"interface":"back","average":{"1min":11.154,"5min":26.515999999999998,"15min":22.882000000000001},"min":{"1min":5.7869999999999999,"5min":5.766,"15min":5.766},"max":{"1min":72.269999999999996,"5min":627.40700000000004,"15min":627.40700000000004},"last":8.6430000000000007},{"interface":"front","average":{"1min":14.353999999999999,"5min":33.304000000000002,"15min":29.885000000000002},"min":{"1min":8.2149999999999999,"5min":7.8630000000000004,"15min":7.4089999999999998},"max":{"1min":73.418000000000006,"5min":635.62599999999998,"15min":635.62599999999998},"last":10.872999999999999}]},{"osd":3,"last update":"Mon Jun 3 09:06:04 2024","interfaces":[{"interface":"back","average":{"1min":12.105,"5min":26.925000000000001,"15min":26.584},"min":{"1min":7.8410000000000002,"5min":7.1280000000000001,"15min":6.1909999999999998},"max":{"1min":18.425999999999998,"5min":389.72699999999998,"15min":389.72699999999998},"last":9.484},{"interface":"front","average":{"1min":13.794,"5min":29.771999999999998,"15min":28.815000000000001},"min":{"1min":10.084,"5min":9.4920000000000009,"15min":6.1849999999999996},"max":{"1min":17.373999999999999,"5min":390.23200000000003,"15min":494.92899999999997},"last":15.166}]},{"osd":4,"last update":"Mon Jun 3 09:05:27 2024","interfaces":[{"interface":"back","average":{"1min":13.478,"5min":1032.126,"15min":1389.729},"min":{"1min":6.5119999999999996,"5min":6.5119999999999996,"15min":6.5119999999999996},"max":{"1min":55.354999999999997,"5min":25555.212,"15min":60989.964},"last":11.698},{"interface":"front","average":{"1min":16.204999999999998,"5min":1030.8389999999999,"15min":1387.778},"min":{"1min":10.282999999999999,"5min":7.1429999999999998,"15min":7.1429999999999998},"max":{"1min":49.945999999999998,"5min":25556.159,"15min":61000.017},"last":12.278}]},{"osd":5,"last update":"Mon Jun 3 09:06:17 2024","interfaces":[{"interface":"back","average":{"1min":9.7040000000000006,"5min":26.533999999999999,"15min":28.683},"min":{"1min":6.8129999999999997,"5min":6.8129999999999997,"15min":6.8129999999999997},"max":{"1min":14.942,"5min":455.78699999999998,"15min":605.05899999999997},"last":7.2480000000000002},{"interface":"front","average":{"1min":14.279,"5min":28.398,"15min":26.192},"min":{"1min":11.696,"5min":6.4420000000000002,"15min":6.2030000000000003},"max":{"1min":17.879000000000001,"5min":391.31200000000001,"15min":541.11000000000001},"last":14.297000000000001}]},{"osd":6,"last update":"Mon Jun 3 09:04:30 2024","interfaces":[{"interface":"back","average":{"1min":52.496000000000002,"5min":33.139000000000003,"15min":160.83199999999999},"min":{"1min":7.5289999999999999,"5min":7.1040000000000001,"15min":6.3739999999999997},"max":{"1min":457.91699999999997,"5min":457.91699999999997,"15min":11115.258},"last":18.959},{"interface":"front","average":{"1min":49.670000000000002,"5min":33.704999999999998,"15min":162.791},"min":{"1min":12.247999999999999,"5min":11.019,"15min":9.9760000000000009},"max":{"1min":407.05099999999999,"5min":407.05099999999999,"15min":11092.194},"last":17.707000000000001}]},{"osd":7,"last update":"Mon Jun 3 09:06:22 2024","interfaces":[{"interface":"back","average":{"1min":11.625999999999999,"5min":26.375,"15min":28.972999999999999},"min":{"1min":6.4480000000000004,"5min":6.4480000000000004,"15min":6.4480000000000004},"max":{"1min":15.352,"5min":461.41699999999997,"15min":598.33699999999999},"last":10.250999999999999},{"interface":"front","average":{"1min":14.013,"5min":29.696000000000002,"15min":33.630000000000003},"min":{"1min":10.938000000000001,"5min":10.58,"15min":10.263999999999999},"max":{"1min":16.603000000000002,"5min":463.10599999999999,"15min":592.80499999999995},"last":13.254}]}]}],"pool_statfs":[{"poolid":1,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":401408,"data_stored":397840,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":5,"total":0,"available":0,"internally_reserved":0,"allocated":401408,"data_stored":397840,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":4,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":5,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":6,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":17,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":17,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":17,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":17,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":17,"osd":4,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":17,"osd":5,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":5,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":17,"osd":6,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":5,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":17,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0}]}} 2024-06-03T09:06:26.460 DEBUG:teuthology.orchestra.run.smithi088:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph pg dump --format=json 2024-06-03T09:06:26.559 INFO:teuthology.orchestra.run.smithi088.stderr:2024-06-03T09:06:26.559+0000 7fb07518b640 10 monclient: get_monmap_and_config 2024-06-03T09:06:26.559 INFO:teuthology.orchestra.run.smithi088.stderr:2024-06-03T09:06:26.559+0000 7fb07518b640 10 monclient: build_initial_monmap 2024-06-03T09:06:26.560 INFO:teuthology.orchestra.run.smithi088.stderr:2024-06-03T09:06:26.559+0000 7fb07518b640 10 monclient: monmap: 2024-06-03T09:06:26.560 INFO:teuthology.orchestra.run.smithi088.stderr:epoch 0 2024-06-03T09:06:26.560 INFO:teuthology.orchestra.run.smithi088.stderr:fsid ec3c3fdd-3ac6-4d7d-9412-683130ed8b04 2024-06-03T09:06:26.560 INFO:teuthology.orchestra.run.smithi088.stderr:last_changed 2024-06-03T09:06:26.560551+0000 2024-06-03T09:06:26.560 INFO:teuthology.orchestra.run.smithi088.stderr:created 2024-06-03T09:06:26.560551+0000 2024-06-03T09:06:26.560 INFO:teuthology.orchestra.run.smithi088.stderr:min_mon_release 0 (unknown) 2024-06-03T09:06:26.560 INFO:teuthology.orchestra.run.smithi088.stderr:election_strategy: 3 2024-06-03T09:06:26.560 INFO:teuthology.orchestra.run.smithi088.stderr:0: v1:172.21.15.88:6789/0 mon.noname-a 2024-06-03T09:06:26.560 INFO:teuthology.orchestra.run.smithi088.stderr:1: v1:172.21.15.192:6789/0 mon.noname-c 2024-06-03T09:06:26.560 INFO:teuthology.orchestra.run.smithi088.stderr:2: v1:172.21.15.88:6790/0 mon.noname-b 2024-06-03T09:06:26.560 INFO:teuthology.orchestra.run.smithi088.stderr: 2024-06-03T09:06:26.562 INFO:teuthology.orchestra.run.smithi088.stderr:2024-06-03T09:06:26.560+0000 7fb07518b640 1 Processor -- start 2024-06-03T09:06:26.562 INFO:teuthology.orchestra.run.smithi088.stderr:2024-06-03T09:06:26.560+0000 7fb07518b640 1 -- start start 2024-06-03T09:06:26.562 INFO:teuthology.orchestra.run.smithi088.stderr:2024-06-03T09:06:26.560+0000 7fb07518b640 10 monclient: init 2024-06-03T09:06:26.562 INFO:teuthology.orchestra.run.smithi088.stderr:2024-06-03T09:06:26.560+0000 7fb07518b640 10 monclient: _reopen_session rank -1 2024-06-03T09:06:26.562 INFO:teuthology.orchestra.run.smithi088.stderr:2024-06-03T09:06:26.560+0000 7fb07518b640 10 monclient: _add_conns ranks=[1,2,0] 2024-06-03T09:06:26.562 INFO:teuthology.orchestra.run.smithi088.stderr:2024-06-03T09:06:26.560+0000 7fb07518b640 10 monclient(hunting): picked mon.noname-c con 0x7fb0700adb90 addr v1:172.21.15.192:6789/0 2024-06-03T09:06:26.562 INFO:teuthology.orchestra.run.smithi088.stderr:2024-06-03T09:06:26.560+0000 7fb07518b640 10 monclient(hunting): picked mon.noname-b con 0x7fb07006ee00 addr v1:172.21.15.88:6790/0 2024-06-03T09:06:26.563 INFO:teuthology.orchestra.run.smithi088.stderr:2024-06-03T09:06:26.560+0000 7fb07518b640 10 monclient(hunting): picked mon.noname-a con 0x7fb0701383e0 addr v1:172.21.15.88:6789/0 2024-06-03T09:06:26.563 INFO:teuthology.orchestra.run.smithi088.stderr:2024-06-03T09:06:26.560+0000 7fb07518b640 1 -- --> v1:172.21.15.88:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fb07006f2c0 con 0x7fb0701383e0 2024-06-03T09:06:26.563 INFO:teuthology.orchestra.run.smithi088.stderr:2024-06-03T09:06:26.560+0000 7fb07518b640 1 -- --> v1:172.21.15.192:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fb0700710c0 con 0x7fb0700adb90 2024-06-03T09:06:26.563 INFO:teuthology.orchestra.run.smithi088.stderr:2024-06-03T09:06:26.560+0000 7fb07518b640 1 -- --> v1:172.21.15.88:6790/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fb070071fd0 con 0x7fb07006ee00 2024-06-03T09:06:26.563 INFO:teuthology.orchestra.run.smithi088.stderr:2024-06-03T09:06:26.560+0000 7fb07518b640 10 monclient(hunting): _renew_subs 2024-06-03T09:06:26.563 INFO:teuthology.orchestra.run.smithi088.stderr:2024-06-03T09:06:26.560+0000 7fb07518b640 10 monclient(hunting): authenticate will time out at 3042.370869s 2024-06-03T09:06:26.563 INFO:teuthology.orchestra.run.smithi088.stderr:2024-06-03T09:06:26.561+0000 7fb06f59e640 1 --1- >> v1:172.21.15.88:6789/0 conn(0x7fb0701383e0 0x7fb070149af0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.88:6789/0 says I am v1:172.21.15.88:43996/0 (socket says 172.21.15.88:43996) 2024-06-03T09:06:26.563 INFO:teuthology.orchestra.run.smithi088.stderr:2024-06-03T09:06:26.561+0000 7fb06f59e640 1 -- 172.21.15.88:0/2949248270 learned_addr learned my addr 172.21.15.88:0/2949248270 (peer_addr_for_me v1:172.21.15.88:0/0) 2024-06-03T09:06:26.564 INFO:teuthology.orchestra.run.smithi088.stderr:2024-06-03T09:06:26.563+0000 7fb06e59c640 1 -- 172.21.15.88:0/2949248270 <== mon.1 v1:172.21.15.192:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 4187091455 0 0) 0x7fb0700710c0 con 0x7fb0700adb90 2024-06-03T09:06:26.564 INFO:teuthology.orchestra.run.smithi088.stderr:2024-06-03T09:06:26.563+0000 7fb06e59c640 10 monclient(hunting): _negotiate 2024-06-03T09:06:26.564 INFO:teuthology.orchestra.run.smithi088.stderr:2024-06-03T09:06:26.563+0000 7fb06e59c640 10 monclient(hunting): _init_auth method 2 2024-06-03T09:06:26.565 INFO:teuthology.orchestra.run.smithi088.stderr:2024-06-03T09:06:26.563+0000 7fb06e59c640 10 monclient(hunting): _init_auth creating new auth 2024-06-03T09:06:26.565 INFO:teuthology.orchestra.run.smithi088.stderr:2024-06-03T09:06:26.563+0000 7fb06e59c640 10 monclient(hunting): my global_id is 30187 2024-06-03T09:06:26.565 INFO:teuthology.orchestra.run.smithi088.stderr:2024-06-03T09:06:26.563+0000 7fb06e59c640 1 -- 172.21.15.88:0/2949248270 --> v1:172.21.15.192:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7fb05c003520 con 0x7fb0700adb90 2024-06-03T09:06:26.565 INFO:teuthology.orchestra.run.smithi088.stderr:2024-06-03T09:06:26.563+0000 7fb06e59c640 1 -- 172.21.15.88:0/2949248270 <== mon.2 v1:172.21.15.88:6790/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 2692914875 0 0) 0x7fb070071fd0 con 0x7fb07006ee00 2024-06-03T09:06:26.565 INFO:teuthology.orchestra.run.smithi088.stderr:2024-06-03T09:06:26.563+0000 7fb06e59c640 10 monclient(hunting): _negotiate 2024-06-03T09:06:26.565 INFO:teuthology.orchestra.run.smithi088.stderr:2024-06-03T09:06:26.563+0000 7fb06e59c640 10 monclient(hunting): _init_auth method 2 2024-06-03T09:06:26.565 INFO:teuthology.orchestra.run.smithi088.stderr:2024-06-03T09:06:26.563+0000 7fb06e59c640 10 monclient(hunting): _init_auth creating new auth 2024-06-03T09:06:26.565 INFO:teuthology.orchestra.run.smithi088.stderr:2024-06-03T09:06:26.563+0000 7fb06e59c640 10 monclient(hunting): my global_id is 30110 2024-06-03T09:06:26.565 INFO:teuthology.orchestra.run.smithi088.stderr:2024-06-03T09:06:26.563+0000 7fb06e59c640 1 -- 172.21.15.88:0/2949248270 --> v1:172.21.15.88:6790/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7fb0700710c0 con 0x7fb07006ee00 2024-06-03T09:06:26.565 INFO:teuthology.orchestra.run.smithi088.stderr:2024-06-03T09:06:26.563+0000 7fb06e59c640 1 -- 172.21.15.88:0/2949248270 <== mon.0 v1:172.21.15.88:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 3805278780 0 0) 0x7fb07006f2c0 con 0x7fb0701383e0 2024-06-03T09:06:26.565 INFO:teuthology.orchestra.run.smithi088.stderr:2024-06-03T09:06:26.563+0000 7fb06e59c640 10 monclient(hunting): _negotiate 2024-06-03T09:06:26.565 INFO:teuthology.orchestra.run.smithi088.stderr:2024-06-03T09:06:26.563+0000 7fb06e59c640 10 monclient(hunting): _init_auth method 2 2024-06-03T09:06:26.565 INFO:teuthology.orchestra.run.smithi088.stderr:2024-06-03T09:06:26.563+0000 7fb06e59c640 10 monclient(hunting): _init_auth creating new auth 2024-06-03T09:06:26.566 INFO:teuthology.orchestra.run.smithi088.stderr:2024-06-03T09:06:26.563+0000 7fb06e59c640 10 monclient(hunting): my global_id is 30606 2024-06-03T09:06:26.566 INFO:teuthology.orchestra.run.smithi088.stderr:2024-06-03T09:06:26.563+0000 7fb06e59c640 1 -- 172.21.15.88:0/2949248270 --> v1:172.21.15.88:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7fb070071fd0 con 0x7fb0701383e0 2024-06-03T09:06:26.567 INFO:teuthology.orchestra.run.smithi088.stderr:2024-06-03T09:06:26.566+0000 7fb06e59c640 1 -- 172.21.15.88:0/2949248270 <== mon.0 v1:172.21.15.88:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 214+0+0 (unknown 2498161954 0 0) 0x7fb070071fd0 con 0x7fb0701383e0 2024-06-03T09:06:26.567 INFO:teuthology.orchestra.run.smithi088.stderr:2024-06-03T09:06:26.566+0000 7fb06e59c640 1 -- 172.21.15.88:0/2949248270 >> v1:172.21.15.88:6790/0 conn(0x7fb07006ee00 legacy=0x7fb070137cd0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-06-03T09:06:26.567 INFO:teuthology.orchestra.run.smithi088.stderr:2024-06-03T09:06:26.566+0000 7fb06e59c640 1 -- 172.21.15.88:0/2949248270 >> v1:172.21.15.192:6789/0 conn(0x7fb0700adb90 legacy=0x7fb0700718c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-06-03T09:06:26.567 INFO:teuthology.orchestra.run.smithi088.stderr:2024-06-03T09:06:26.566+0000 7fb06e59c640 10 monclient: _finish_hunting 0 2024-06-03T09:06:26.567 INFO:teuthology.orchestra.run.smithi088.stderr:2024-06-03T09:06:26.566+0000 7fb06e59c640 1 monclient: found mon.noname-a 2024-06-03T09:06:26.567 INFO:teuthology.orchestra.run.smithi088.stderr:2024-06-03T09:06:26.566+0000 7fb06e59c640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-06-03T09:06:26.567 INFO:teuthology.orchestra.run.smithi088.stderr:2024-06-03T09:06:26.566+0000 7fb06e59c640 10 monclient: _send_mon_message to mon.noname-a at v1:172.21.15.88:6789/0 2024-06-03T09:06:26.567 INFO:teuthology.orchestra.run.smithi088.stderr:2024-06-03T09:06:26.566+0000 7fb06e59c640 1 -- 172.21.15.88:0/2949248270 --> v1:172.21.15.88:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fb070141750 con 0x7fb0701383e0 2024-06-03T09:06:26.568 INFO:teuthology.orchestra.run.smithi088.stderr:2024-06-03T09:06:26.566+0000 7fb06e59c640 10 monclient: _finish_auth 0 2024-06-03T09:06:26.568 INFO:teuthology.orchestra.run.smithi088.stderr:2024-06-03T09:06:26.566+0000 7fb06e59c640 10 monclient: _check_auth_tickets 2024-06-03T09:06:26.568 INFO:teuthology.orchestra.run.smithi088.stderr:2024-06-03T09:06:26.566+0000 7fb06e59c640 20 monclient: _check_auth_rotating not needed by client.admin 2024-06-03T09:06:26.568 INFO:teuthology.orchestra.run.smithi088.stderr:2024-06-03T09:06:26.566+0000 7fb07518b640 5 monclient: authenticate success, global_id 30606 2024-06-03T09:06:26.568 INFO:teuthology.orchestra.run.smithi088.stderr:2024-06-03T09:06:26.566+0000 7fb07518b640 20 monclient: get_monmap_and_config waiting for monmap|config 2024-06-03T09:06:26.568 INFO:teuthology.orchestra.run.smithi088.stderr:2024-06-03T09:06:26.567+0000 7fb06e59c640 1 -- 172.21.15.88:0/2949248270 <== mon.0 v1:172.21.15.88:6789/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 2234374065 0 0) 0x7fb068001930 con 0x7fb0701383e0 2024-06-03T09:06:26.568 INFO:teuthology.orchestra.run.smithi088.stderr:2024-06-03T09:06:26.567+0000 7fb06e59c640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-06-03T09:06:26.568 INFO:teuthology.orchestra.run.smithi088.stderr:2024-06-03T09:06:26.567+0000 7fb06e59c640 10 monclient: got monmap 1 from mon.noname-a (according to old e1) 2024-06-03T09:06:26.568 INFO:teuthology.orchestra.run.smithi088.stderr:2024-06-03T09:06:26.567+0000 7fb06e59c640 10 monclient: dump: 2024-06-03T09:06:26.568 INFO:teuthology.orchestra.run.smithi088.stderr:epoch 1 2024-06-03T09:06:26.568 INFO:teuthology.orchestra.run.smithi088.stderr:fsid ec3c3fdd-3ac6-4d7d-9412-683130ed8b04 2024-06-03T09:06:26.568 INFO:teuthology.orchestra.run.smithi088.stderr:last_changed 2024-06-03T08:34:31.885125+0000 2024-06-03T09:06:26.569 INFO:teuthology.orchestra.run.smithi088.stderr:created 2024-06-03T08:34:31.885125+0000 2024-06-03T09:06:26.569 INFO:teuthology.orchestra.run.smithi088.stderr:min_mon_release 18 (reef) 2024-06-03T09:06:26.569 INFO:teuthology.orchestra.run.smithi088.stderr:election_strategy: 3 2024-06-03T09:06:26.569 INFO:teuthology.orchestra.run.smithi088.stderr:0: v1:172.21.15.88:6789/0 mon.a 2024-06-03T09:06:26.569 INFO:teuthology.orchestra.run.smithi088.stderr:1: v1:172.21.15.192:6789/0 mon.b 2024-06-03T09:06:26.569 INFO:teuthology.orchestra.run.smithi088.stderr:2: v1:172.21.15.88:6790/0 mon.c 2024-06-03T09:06:26.569 INFO:teuthology.orchestra.run.smithi088.stderr: 2024-06-03T09:06:26.569 INFO:teuthology.orchestra.run.smithi088.stderr:2024-06-03T09:06:26.567+0000 7fb07518b640 20 monclient: get_monmap_and_config waiting for monmap|config 2024-06-03T09:06:26.572 INFO:teuthology.orchestra.run.smithi088.stderr:2024-06-03T09:06:26.569+0000 7fb06e59c640 1 -- 172.21.15.88:0/2949248270 <== mon.0 v1:172.21.15.88:6789/0 4 ==== config(0 keys) v1 ==== 4+0+0 (unknown 0 0 0) 0x7fb068001b80 con 0x7fb0701383e0 2024-06-03T09:06:26.572 INFO:teuthology.orchestra.run.smithi088.stderr:2024-06-03T09:06:26.569+0000 7fb06e59c640 10 monclient: handle_config config(0 keys) v1 2024-06-03T09:06:26.572 INFO:teuthology.orchestra.run.smithi088.stderr:2024-06-03T09:06:26.569+0000 7fb06e59c640 1 -- 172.21.15.88:0/2949248270 <== mon.0 v1:172.21.15.88:6789/0 5 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 2234374065 0 0) 0x7fb068001f10 con 0x7fb0701383e0 2024-06-03T09:06:26.572 INFO:teuthology.orchestra.run.smithi088.stderr:2024-06-03T09:06:26.569+0000 7fb06e59c640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-06-03T09:06:26.572 INFO:teuthology.orchestra.run.smithi088.stderr:2024-06-03T09:06:26.569+0000 7fb06e59c640 10 monclient: got monmap 1 from mon.a (according to old e1) 2024-06-03T09:06:26.572 INFO:teuthology.orchestra.run.smithi088.stderr:2024-06-03T09:06:26.569+0000 7fb06e59c640 10 monclient: dump: 2024-06-03T09:06:26.572 INFO:teuthology.orchestra.run.smithi088.stderr:epoch 1 2024-06-03T09:06:26.573 INFO:teuthology.orchestra.run.smithi088.stderr:fsid ec3c3fdd-3ac6-4d7d-9412-683130ed8b04 2024-06-03T09:06:26.573 INFO:teuthology.orchestra.run.smithi088.stderr:last_changed 2024-06-03T08:34:31.885125+0000 2024-06-03T09:06:26.573 INFO:teuthology.orchestra.run.smithi088.stderr:created 2024-06-03T08:34:31.885125+0000 2024-06-03T09:06:26.573 INFO:teuthology.orchestra.run.smithi088.stderr:min_mon_release 18 (reef) 2024-06-03T09:06:26.573 INFO:teuthology.orchestra.run.smithi088.stderr:election_strategy: 3 2024-06-03T09:06:26.573 INFO:teuthology.orchestra.run.smithi088.stderr:0: v1:172.21.15.88:6789/0 mon.a 2024-06-03T09:06:26.573 INFO:teuthology.orchestra.run.smithi088.stderr:1: v1:172.21.15.192:6789/0 mon.b 2024-06-03T09:06:26.573 INFO:teuthology.orchestra.run.smithi088.stderr:2: v1:172.21.15.88:6790/0 mon.c 2024-06-03T09:06:26.573 INFO:teuthology.orchestra.run.smithi088.stderr: 2024-06-03T09:06:26.573 INFO:teuthology.orchestra.run.smithi088.stderr:2024-06-03T09:06:26.569+0000 7fb07518b640 10 monclient: get_monmap_and_config success 2024-06-03T09:06:26.573 INFO:teuthology.orchestra.run.smithi088.stderr:2024-06-03T09:06:26.569+0000 7fb07518b640 4 set_mon_vals no callback set 2024-06-03T09:06:26.573 INFO:teuthology.orchestra.run.smithi088.stderr:2024-06-03T09:06:26.569+0000 7fb07518b640 10 monclient: shutdown 2024-06-03T09:06:26.574 INFO:teuthology.orchestra.run.smithi088.stderr:2024-06-03T09:06:26.569+0000 7fb07518b640 20 monclient: shutdown discarding 0 pending message(s) 2024-06-03T09:06:26.574 INFO:teuthology.orchestra.run.smithi088.stderr:2024-06-03T09:06:26.569+0000 7fb07518b640 1 -- 172.21.15.88:0/2949248270 >> v1:172.21.15.88:6789/0 conn(0x7fb0701383e0 legacy=0x7fb070149af0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-06-03T09:06:26.574 INFO:teuthology.orchestra.run.smithi088.stderr:2024-06-03T09:06:26.569+0000 7fb07518b640 1 -- 172.21.15.88:0/2949248270 shutdown_connections 2024-06-03T09:06:26.574 INFO:teuthology.orchestra.run.smithi088.stderr:2024-06-03T09:06:26.569+0000 7fb07518b640 1 -- 172.21.15.88:0/2949248270 >> 172.21.15.88:0/2949248270 conn(0x7fb0700b2f20 msgr2=0x7fb0700b3300 unknown :-1 s=STATE_NONE l=0).mark_down 2024-06-03T09:06:26.574 INFO:teuthology.orchestra.run.smithi088.stderr:2024-06-03T09:06:26.569+0000 7fb07518b640 1 -- 172.21.15.88:0/2949248270 shutdown_connections 2024-06-03T09:06:26.574 INFO:teuthology.orchestra.run.smithi088.stderr:2024-06-03T09:06:26.569+0000 7fb07518b640 1 -- 172.21.15.88:0/2949248270 wait complete. 2024-06-03T09:06:26.574 INFO:teuthology.orchestra.run.smithi088.stderr:2024-06-03T09:06:26.570+0000 7fb07518b640 10 monclient: build_initial_monmap 2024-06-03T09:06:26.574 INFO:teuthology.orchestra.run.smithi088.stderr:2024-06-03T09:06:26.570+0000 7fb07518b640 10 monclient: monmap: 2024-06-03T09:06:26.574 INFO:teuthology.orchestra.run.smithi088.stderr:epoch 0 2024-06-03T09:06:26.574 INFO:teuthology.orchestra.run.smithi088.stderr:fsid 00000000-0000-0000-0000-000000000000 2024-06-03T09:06:26.574 INFO:teuthology.orchestra.run.smithi088.stderr:last_changed 0.000000 2024-06-03T09:06:26.574 INFO:teuthology.orchestra.run.smithi088.stderr:created 0.000000 2024-06-03T09:06:26.574 INFO:teuthology.orchestra.run.smithi088.stderr:min_mon_release 0 (unknown) 2024-06-03T09:06:26.575 INFO:teuthology.orchestra.run.smithi088.stderr:election_strategy: 1 2024-06-03T09:06:26.575 INFO:teuthology.orchestra.run.smithi088.stderr:0: v1:172.21.15.88:6789/0 mon.noname-a 2024-06-03T09:06:26.575 INFO:teuthology.orchestra.run.smithi088.stderr:1: v1:172.21.15.192:6789/0 mon.noname-b 2024-06-03T09:06:26.575 INFO:teuthology.orchestra.run.smithi088.stderr:2: v1:172.21.15.88:6790/0 mon.noname-c 2024-06-03T09:06:26.575 INFO:teuthology.orchestra.run.smithi088.stderr: 2024-06-03T09:06:26.575 INFO:teuthology.orchestra.run.smithi088.stderr:2024-06-03T09:06:26.570+0000 7fb07518b640 1 librados: starting msgr at 2024-06-03T09:06:26.575 INFO:teuthology.orchestra.run.smithi088.stderr:2024-06-03T09:06:26.570+0000 7fb07518b640 1 librados: starting objecter 2024-06-03T09:06:26.575 INFO:teuthology.orchestra.run.smithi088.stderr:2024-06-03T09:06:26.571+0000 7fb07518b640 1 Processor -- start 2024-06-03T09:06:26.575 INFO:teuthology.orchestra.run.smithi088.stderr:2024-06-03T09:06:26.571+0000 7fb07518b640 1 -- start start 2024-06-03T09:06:26.575 INFO:teuthology.orchestra.run.smithi088.stderr:2024-06-03T09:06:26.571+0000 7fb07518b640 1 librados: setting wanted keys 2024-06-03T09:06:26.575 INFO:teuthology.orchestra.run.smithi088.stderr:2024-06-03T09:06:26.571+0000 7fb07518b640 1 librados: calling monclient init 2024-06-03T09:06:26.575 INFO:teuthology.orchestra.run.smithi088.stderr:2024-06-03T09:06:26.571+0000 7fb07518b640 10 monclient: init 2024-06-03T09:06:26.575 INFO:teuthology.orchestra.run.smithi088.stderr:2024-06-03T09:06:26.571+0000 7fb07518b640 10 monclient: _reopen_session rank -1 2024-06-03T09:06:26.576 INFO:teuthology.orchestra.run.smithi088.stderr:2024-06-03T09:06:26.571+0000 7fb07518b640 10 monclient: _add_conns ranks=[1,2,0] 2024-06-03T09:06:26.576 INFO:teuthology.orchestra.run.smithi088.stderr:2024-06-03T09:06:26.571+0000 7fb07518b640 10 monclient(hunting): picked mon.noname-b con 0x7fb07006ee00 addr v1:172.21.15.192:6789/0 2024-06-03T09:06:26.576 INFO:teuthology.orchestra.run.smithi088.stderr:2024-06-03T09:06:26.571+0000 7fb07518b640 10 monclient(hunting): picked mon.noname-c con 0x7fb0700adb90 addr v1:172.21.15.88:6790/0 2024-06-03T09:06:26.576 INFO:teuthology.orchestra.run.smithi088.stderr:2024-06-03T09:06:26.571+0000 7fb07518b640 10 monclient(hunting): picked mon.noname-a con 0x7fb0701383e0 addr v1:172.21.15.88:6789/0 2024-06-03T09:06:26.576 INFO:teuthology.orchestra.run.smithi088.stderr:2024-06-03T09:06:26.571+0000 7fb07518b640 1 -- --> v1:172.21.15.88:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fb0700aaa30 con 0x7fb0701383e0 2024-06-03T09:06:26.576 INFO:teuthology.orchestra.run.smithi088.stderr:2024-06-03T09:06:26.571+0000 7fb07518b640 1 -- --> v1:172.21.15.192:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fb0700aaec0 con 0x7fb07006ee00 2024-06-03T09:06:26.576 INFO:teuthology.orchestra.run.smithi088.stderr:2024-06-03T09:06:26.571+0000 7fb07518b640 1 -- --> v1:172.21.15.88:6790/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fb0700aa3c0 con 0x7fb0700adb90 2024-06-03T09:06:26.576 INFO:teuthology.orchestra.run.smithi088.stderr:2024-06-03T09:06:26.571+0000 7fb07518b640 10 monclient(hunting): _renew_subs 2024-06-03T09:06:26.576 INFO:teuthology.orchestra.run.smithi088.stderr:2024-06-03T09:06:26.571+0000 7fb07518b640 10 monclient(hunting): authenticate will time out at 3042.381703s 2024-06-03T09:06:26.576 INFO:teuthology.orchestra.run.smithi088.stderr:2024-06-03T09:06:26.572+0000 7fb06f59e640 1 --1- >> v1:172.21.15.88:6789/0 conn(0x7fb0701383e0 0x7fb0701dfa10 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=1).handle_server_banner_and_identify peer v1:172.21.15.88:6789/0 says I am v1:172.21.15.88:43998/0 (socket says 172.21.15.88:43998) 2024-06-03T09:06:26.576 INFO:teuthology.orchestra.run.smithi088.stderr:2024-06-03T09:06:26.572+0000 7fb06f59e640 1 -- 172.21.15.88:0/42259004 learned_addr learned my addr 172.21.15.88:0/42259004 (peer_addr_for_me v1:172.21.15.88:0/0) 2024-06-03T09:06:26.576 INFO:teuthology.orchestra.run.smithi088.stderr:2024-06-03T09:06:26.572+0000 7fb0677fe640 10 client.?.objecter ms_handle_connect 0x7fb0701383e0 2024-06-03T09:06:26.576 INFO:teuthology.orchestra.run.smithi088.stderr:2024-06-03T09:06:26.572+0000 7fb0677fe640 10 client.?.objecter resend_mon_ops 2024-06-03T09:06:26.577 INFO:teuthology.orchestra.run.smithi088.stderr:2024-06-03T09:06:26.572+0000 7fb0677fe640 10 client.?.objecter ms_handle_connect 0x7fb07006ee00 2024-06-03T09:06:26.577 INFO:teuthology.orchestra.run.smithi088.stderr:2024-06-03T09:06:26.572+0000 7fb0677fe640 10 client.?.objecter resend_mon_ops 2024-06-03T09:06:26.577 INFO:teuthology.orchestra.run.smithi088.stderr:2024-06-03T09:06:26.572+0000 7fb0677fe640 10 client.?.objecter ms_handle_connect 0x7fb0700adb90 2024-06-03T09:06:26.577 INFO:teuthology.orchestra.run.smithi088.stderr:2024-06-03T09:06:26.572+0000 7fb0677fe640 10 client.?.objecter resend_mon_ops 2024-06-03T09:06:26.578 INFO:teuthology.orchestra.run.smithi088.stderr:2024-06-03T09:06:26.574+0000 7fb0677fe640 1 -- 172.21.15.88:0/42259004 <== mon.1 v1:172.21.15.192:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 1352901820 0 0) 0x7fb0700aaec0 con 0x7fb07006ee00 2024-06-03T09:06:26.578 INFO:teuthology.orchestra.run.smithi088.stderr:2024-06-03T09:06:26.574+0000 7fb0677fe640 10 monclient(hunting): _negotiate 2024-06-03T09:06:26.578 INFO:teuthology.orchestra.run.smithi088.stderr:2024-06-03T09:06:26.574+0000 7fb0677fe640 10 monclient(hunting): _init_auth method 2 2024-06-03T09:06:26.578 INFO:teuthology.orchestra.run.smithi088.stderr:2024-06-03T09:06:26.574+0000 7fb0677fe640 10 monclient(hunting): _init_auth creating new auth 2024-06-03T09:06:26.578 INFO:teuthology.orchestra.run.smithi088.stderr:2024-06-03T09:06:26.574+0000 7fb0677fe640 10 monclient(hunting): my global_id is 30190 2024-06-03T09:06:26.578 INFO:teuthology.orchestra.run.smithi088.stderr:2024-06-03T09:06:26.574+0000 7fb0677fe640 1 -- 172.21.15.88:0/42259004 --> v1:172.21.15.192:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7fb050004eb0 con 0x7fb07006ee00 2024-06-03T09:06:26.578 INFO:teuthology.orchestra.run.smithi088.stderr:2024-06-03T09:06:26.574+0000 7fb0677fe640 1 -- 172.21.15.88:0/42259004 <== mon.0 v1:172.21.15.88:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 565047788 0 0) 0x7fb0700aaa30 con 0x7fb0701383e0 2024-06-03T09:06:26.578 INFO:teuthology.orchestra.run.smithi088.stderr:2024-06-03T09:06:26.574+0000 7fb0677fe640 10 monclient(hunting): _negotiate 2024-06-03T09:06:26.578 INFO:teuthology.orchestra.run.smithi088.stderr:2024-06-03T09:06:26.574+0000 7fb0677fe640 10 monclient(hunting): _init_auth method 2 2024-06-03T09:06:26.579 INFO:teuthology.orchestra.run.smithi088.stderr:2024-06-03T09:06:26.574+0000 7fb0677fe640 10 monclient(hunting): _init_auth creating new auth 2024-06-03T09:06:26.579 INFO:teuthology.orchestra.run.smithi088.stderr:2024-06-03T09:06:26.574+0000 7fb0677fe640 10 monclient(hunting): my global_id is 30609 2024-06-03T09:06:26.579 INFO:teuthology.orchestra.run.smithi088.stderr:2024-06-03T09:06:26.574+0000 7fb0677fe640 1 -- 172.21.15.88:0/42259004 --> v1:172.21.15.88:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7fb0700aaec0 con 0x7fb0701383e0 2024-06-03T09:06:26.579 INFO:teuthology.orchestra.run.smithi088.stderr:2024-06-03T09:06:26.574+0000 7fb0677fe640 1 -- 172.21.15.88:0/42259004 <== mon.2 v1:172.21.15.88:6790/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 2979088056 0 0) 0x7fb0700aa3c0 con 0x7fb0700adb90 2024-06-03T09:06:26.579 INFO:teuthology.orchestra.run.smithi088.stderr:2024-06-03T09:06:26.574+0000 7fb0677fe640 10 monclient(hunting): _negotiate 2024-06-03T09:06:26.579 INFO:teuthology.orchestra.run.smithi088.stderr:2024-06-03T09:06:26.574+0000 7fb0677fe640 10 monclient(hunting): _init_auth method 2 2024-06-03T09:06:26.579 INFO:teuthology.orchestra.run.smithi088.stderr:2024-06-03T09:06:26.574+0000 7fb0677fe640 10 monclient(hunting): _init_auth creating new auth 2024-06-03T09:06:26.579 INFO:teuthology.orchestra.run.smithi088.stderr:2024-06-03T09:06:26.574+0000 7fb0677fe640 10 monclient(hunting): my global_id is 30113 2024-06-03T09:06:26.579 INFO:teuthology.orchestra.run.smithi088.stderr:2024-06-03T09:06:26.574+0000 7fb0677fe640 1 -- 172.21.15.88:0/42259004 --> v1:172.21.15.88:6790/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7fb0700aaa30 con 0x7fb0700adb90 2024-06-03T09:06:26.580 INFO:teuthology.orchestra.run.smithi088.stderr:2024-06-03T09:06:26.578+0000 7fb0677fe640 1 -- 172.21.15.88:0/42259004 <== mon.1 v1:172.21.15.192:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 764+0+0 (unknown 2042079617 0 0) 0x7fb050004eb0 con 0x7fb07006ee00 2024-06-03T09:06:26.580 INFO:teuthology.orchestra.run.smithi088.stderr:2024-06-03T09:06:26.578+0000 7fb0677fe640 1 -- 172.21.15.88:0/42259004 --> v1:172.21.15.192:6789/0 -- auth(proto 2 165 bytes epoch 0) v1 -- 0x7fb0700aa3c0 con 0x7fb07006ee00 2024-06-03T09:06:26.581 INFO:teuthology.orchestra.run.smithi088.stderr:2024-06-03T09:06:26.578+0000 7fb0677fe640 1 -- 172.21.15.88:0/42259004 <== mon.1 v1:172.21.15.192:6789/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 2234374065 0 0) 0x7fb0600018d0 con 0x7fb07006ee00 2024-06-03T09:06:26.581 INFO:teuthology.orchestra.run.smithi088.stderr:2024-06-03T09:06:26.578+0000 7fb0677fe640 10 monclient(hunting): handle_monmap mon_map magic: 0 v1 2024-06-03T09:06:26.581 INFO:teuthology.orchestra.run.smithi088.stderr:2024-06-03T09:06:26.578+0000 7fb0677fe640 10 monclient(hunting): got monmap 1 from mon.noname-b (according to old e1) 2024-06-03T09:06:26.581 INFO:teuthology.orchestra.run.smithi088.stderr:2024-06-03T09:06:26.578+0000 7fb0677fe640 10 monclient(hunting): dump: 2024-06-03T09:06:26.581 INFO:teuthology.orchestra.run.smithi088.stderr:epoch 1 2024-06-03T09:06:26.581 INFO:teuthology.orchestra.run.smithi088.stderr:fsid ec3c3fdd-3ac6-4d7d-9412-683130ed8b04 2024-06-03T09:06:26.581 INFO:teuthology.orchestra.run.smithi088.stderr:last_changed 2024-06-03T08:34:31.885125+0000 2024-06-03T09:06:26.581 INFO:teuthology.orchestra.run.smithi088.stderr:created 2024-06-03T08:34:31.885125+0000 2024-06-03T09:06:26.581 INFO:teuthology.orchestra.run.smithi088.stderr:min_mon_release 18 (reef) 2024-06-03T09:06:26.581 INFO:teuthology.orchestra.run.smithi088.stderr:election_strategy: 3 2024-06-03T09:06:26.581 INFO:teuthology.orchestra.run.smithi088.stderr:0: v1:172.21.15.88:6789/0 mon.a 2024-06-03T09:06:26.581 INFO:teuthology.orchestra.run.smithi088.stderr:1: v1:172.21.15.192:6789/0 mon.b 2024-06-03T09:06:26.581 INFO:teuthology.orchestra.run.smithi088.stderr:2: v1:172.21.15.88:6790/0 mon.c 2024-06-03T09:06:26.581 INFO:teuthology.orchestra.run.smithi088.stderr: 2024-06-03T09:06:26.582 INFO:teuthology.orchestra.run.smithi088.stderr:2024-06-03T09:06:26.578+0000 7fb0677fe640 10 monclient(hunting): _finish_auth 0 2024-06-03T09:06:26.582 INFO:teuthology.orchestra.run.smithi088.stderr:2024-06-03T09:06:26.579+0000 7fb0677fe640 1 -- 172.21.15.88:0/42259004 <== mon.0 v1:172.21.15.88:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 764+0+0 (unknown 17067870 0 0) 0x7fb0700aaec0 con 0x7fb0701383e0 2024-06-03T09:06:26.582 INFO:teuthology.orchestra.run.smithi088.stderr:2024-06-03T09:06:26.579+0000 7fb0677fe640 1 -- 172.21.15.88:0/42259004 --> v1:172.21.15.88:6789/0 -- auth(proto 2 165 bytes epoch 0) v1 -- 0x7fb050004eb0 con 0x7fb0701383e0 2024-06-03T09:06:26.582 INFO:teuthology.orchestra.run.smithi088.stderr:2024-06-03T09:06:26.579+0000 7fb0677fe640 1 -- 172.21.15.88:0/42259004 <== mon.0 v1:172.21.15.88:6789/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 2234374065 0 0) 0x7fb068001e90 con 0x7fb0701383e0 2024-06-03T09:06:26.582 INFO:teuthology.orchestra.run.smithi088.stderr:2024-06-03T09:06:26.579+0000 7fb0677fe640 10 monclient(hunting): handle_monmap mon_map magic: 0 v1 2024-06-03T09:06:26.582 INFO:teuthology.orchestra.run.smithi088.stderr:2024-06-03T09:06:26.579+0000 7fb0677fe640 10 monclient(hunting): got monmap 1 from mon.a (according to old e1) 2024-06-03T09:06:26.582 INFO:teuthology.orchestra.run.smithi088.stderr:2024-06-03T09:06:26.579+0000 7fb0677fe640 10 monclient(hunting): dump: 2024-06-03T09:06:26.582 INFO:teuthology.orchestra.run.smithi088.stderr:epoch 1 2024-06-03T09:06:26.582 INFO:teuthology.orchestra.run.smithi088.stderr:fsid ec3c3fdd-3ac6-4d7d-9412-683130ed8b04 2024-06-03T09:06:26.582 INFO:teuthology.orchestra.run.smithi088.stderr:last_changed 2024-06-03T08:34:31.885125+0000 2024-06-03T09:06:26.582 INFO:teuthology.orchestra.run.smithi088.stderr:created 2024-06-03T08:34:31.885125+0000 2024-06-03T09:06:26.582 INFO:teuthology.orchestra.run.smithi088.stderr:min_mon_release 18 (reef) 2024-06-03T09:06:26.582 INFO:teuthology.orchestra.run.smithi088.stderr:election_strategy: 3 2024-06-03T09:06:26.583 INFO:teuthology.orchestra.run.smithi088.stderr:0: v1:172.21.15.88:6789/0 mon.a 2024-06-03T09:06:26.583 INFO:teuthology.orchestra.run.smithi088.stderr:1: v1:172.21.15.192:6789/0 mon.b 2024-06-03T09:06:26.583 INFO:teuthology.orchestra.run.smithi088.stderr:2: v1:172.21.15.88:6790/0 mon.c 2024-06-03T09:06:26.583 INFO:teuthology.orchestra.run.smithi088.stderr: 2024-06-03T09:06:26.583 INFO:teuthology.orchestra.run.smithi088.stderr:2024-06-03T09:06:26.579+0000 7fb0677fe640 1 -- 172.21.15.88:0/42259004 <== mon.2 v1:172.21.15.88:6790/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 764+0+0 (unknown 2659652530 0 0) 0x7fb0700aaa30 con 0x7fb0700adb90 2024-06-03T09:06:26.583 INFO:teuthology.orchestra.run.smithi088.stderr:2024-06-03T09:06:26.579+0000 7fb0677fe640 1 -- 172.21.15.88:0/42259004 --> v1:172.21.15.88:6790/0 -- auth(proto 2 165 bytes epoch 0) v1 -- 0x7fb0700aaec0 con 0x7fb0700adb90 2024-06-03T09:06:26.583 INFO:teuthology.orchestra.run.smithi088.stderr:2024-06-03T09:06:26.579+0000 7fb0677fe640 1 -- 172.21.15.88:0/42259004 <== mon.2 v1:172.21.15.88:6790/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 2234374065 0 0) 0x7fb058001870 con 0x7fb0700adb90 2024-06-03T09:06:26.583 INFO:teuthology.orchestra.run.smithi088.stderr:2024-06-03T09:06:26.579+0000 7fb0677fe640 10 monclient(hunting): handle_monmap mon_map magic: 0 v1 2024-06-03T09:06:26.583 INFO:teuthology.orchestra.run.smithi088.stderr:2024-06-03T09:06:26.579+0000 7fb0677fe640 10 monclient(hunting): got monmap 1 from mon.c (according to old e1) 2024-06-03T09:06:26.583 INFO:teuthology.orchestra.run.smithi088.stderr:2024-06-03T09:06:26.579+0000 7fb0677fe640 10 monclient(hunting): dump: 2024-06-03T09:06:26.583 INFO:teuthology.orchestra.run.smithi088.stderr:epoch 1 2024-06-03T09:06:26.583 INFO:teuthology.orchestra.run.smithi088.stderr:fsid ec3c3fdd-3ac6-4d7d-9412-683130ed8b04 2024-06-03T09:06:26.583 INFO:teuthology.orchestra.run.smithi088.stderr:last_changed 2024-06-03T08:34:31.885125+0000 2024-06-03T09:06:26.583 INFO:teuthology.orchestra.run.smithi088.stderr:created 2024-06-03T08:34:31.885125+0000 2024-06-03T09:06:26.584 INFO:teuthology.orchestra.run.smithi088.stderr:min_mon_release 18 (reef) 2024-06-03T09:06:26.584 INFO:teuthology.orchestra.run.smithi088.stderr:election_strategy: 3 2024-06-03T09:06:26.584 INFO:teuthology.orchestra.run.smithi088.stderr:0: v1:172.21.15.88:6789/0 mon.a 2024-06-03T09:06:26.584 INFO:teuthology.orchestra.run.smithi088.stderr:1: v1:172.21.15.192:6789/0 mon.b 2024-06-03T09:06:26.584 INFO:teuthology.orchestra.run.smithi088.stderr:2: v1:172.21.15.88:6790/0 mon.c 2024-06-03T09:06:26.584 INFO:teuthology.orchestra.run.smithi088.stderr: 2024-06-03T09:06:26.585 INFO:teuthology.orchestra.run.smithi088.stderr:2024-06-03T09:06:26.582+0000 7fb0677fe640 1 -- 172.21.15.88:0/42259004 <== mon.2 v1:172.21.15.88:6790/0 4 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 580+0+0 (unknown 3170287575 0 0) 0x7fb0700aaec0 con 0x7fb0700adb90 2024-06-03T09:06:26.585 INFO:teuthology.orchestra.run.smithi088.stderr:2024-06-03T09:06:26.582+0000 7fb0677fe640 1 -- 172.21.15.88:0/42259004 >> v1:172.21.15.192:6789/0 conn(0x7fb07006ee00 legacy=0x7fb070135080 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-06-03T09:06:26.585 INFO:teuthology.orchestra.run.smithi088.stderr:2024-06-03T09:06:26.582+0000 7fb0677fe640 1 -- 172.21.15.88:0/42259004 >> v1:172.21.15.88:6789/0 conn(0x7fb0701383e0 legacy=0x7fb0701dfa10 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-06-03T09:06:26.585 INFO:teuthology.orchestra.run.smithi088.stderr:2024-06-03T09:06:26.582+0000 7fb0677fe640 10 monclient: _finish_hunting 0 2024-06-03T09:06:26.585 INFO:teuthology.orchestra.run.smithi088.stderr:2024-06-03T09:06:26.582+0000 7fb0677fe640 1 monclient: found mon.c 2024-06-03T09:06:26.585 INFO:teuthology.orchestra.run.smithi088.stderr:2024-06-03T09:06:26.582+0000 7fb0677fe640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-06-03T09:06:26.586 INFO:teuthology.orchestra.run.smithi088.stderr:2024-06-03T09:06:26.582+0000 7fb0677fe640 10 monclient: _send_mon_message to mon.c at v1:172.21.15.88:6790/0 2024-06-03T09:06:26.586 INFO:teuthology.orchestra.run.smithi088.stderr:2024-06-03T09:06:26.582+0000 7fb0677fe640 1 -- 172.21.15.88:0/42259004 --> v1:172.21.15.88:6790/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fb0700aa550 con 0x7fb0700adb90 2024-06-03T09:06:26.586 INFO:teuthology.orchestra.run.smithi088.stderr:2024-06-03T09:06:26.582+0000 7fb0677fe640 10 monclient: _finish_auth 0 2024-06-03T09:06:26.586 INFO:teuthology.orchestra.run.smithi088.stderr:2024-06-03T09:06:26.582+0000 7fb0677fe640 10 monclient: _check_auth_tickets 2024-06-03T09:06:26.586 INFO:teuthology.orchestra.run.smithi088.stderr:2024-06-03T09:06:26.582+0000 7fb0677fe640 20 monclient: _check_auth_rotating not needed by client.admin 2024-06-03T09:06:26.586 INFO:teuthology.orchestra.run.smithi088.stderr:2024-06-03T09:06:26.582+0000 7fb07518b640 5 monclient: authenticate success, global_id 30113 2024-06-03T09:06:26.586 INFO:teuthology.orchestra.run.smithi088.stderr:2024-06-03T09:06:26.582+0000 7fb07518b640 10 monclient: _renew_subs 2024-06-03T09:06:26.586 INFO:teuthology.orchestra.run.smithi088.stderr:2024-06-03T09:06:26.582+0000 7fb07518b640 10 monclient: _send_mon_message to mon.c at v1:172.21.15.88:6790/0 2024-06-03T09:06:26.586 INFO:teuthology.orchestra.run.smithi088.stderr:2024-06-03T09:06:26.582+0000 7fb07518b640 1 -- 172.21.15.88:0/42259004 --> v1:172.21.15.88:6790/0 -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fb07013aa80 con 0x7fb0700adb90 2024-06-03T09:06:26.586 INFO:teuthology.orchestra.run.smithi088.stderr:2024-06-03T09:06:26.582+0000 7fb07518b640 10 client.30113.objecter _maybe_request_map subscribing (onetime) to next osd map 2024-06-03T09:06:26.586 INFO:teuthology.orchestra.run.smithi088.stderr:2024-06-03T09:06:26.582+0000 7fb07518b640 10 monclient: _renew_subs 2024-06-03T09:06:26.586 INFO:teuthology.orchestra.run.smithi088.stderr:2024-06-03T09:06:26.582+0000 7fb07518b640 10 monclient: _send_mon_message to mon.c at v1:172.21.15.88:6790/0 2024-06-03T09:06:26.587 INFO:teuthology.orchestra.run.smithi088.stderr:2024-06-03T09:06:26.582+0000 7fb07518b640 1 -- 172.21.15.88:0/42259004 --> v1:172.21.15.88:6790/0 -- mon_subscribe({osdmap=0}) v3 -- 0x7fb0700a9e00 con 0x7fb0700adb90 2024-06-03T09:06:26.587 INFO:teuthology.orchestra.run.smithi088.stderr:2024-06-03T09:06:26.582+0000 7fb07518b640 1 librados: init done 2024-06-03T09:06:26.587 INFO:teuthology.orchestra.run.smithi088.stderr:2024-06-03T09:06:26.583+0000 7fb07518b640 10 start_mon_command cmd=[{"prefix": "get_command_descriptions"}] 2024-06-03T09:06:26.587 INFO:teuthology.orchestra.run.smithi088.stderr:2024-06-03T09:06:26.583+0000 7fb07518b640 10 monclient: _send_command 1 [{"prefix": "get_command_descriptions"}] 2024-06-03T09:06:26.587 INFO:teuthology.orchestra.run.smithi088.stderr:2024-06-03T09:06:26.583+0000 7fb07518b640 10 monclient: _send_mon_message to mon.c at v1:172.21.15.88:6790/0 2024-06-03T09:06:26.587 INFO:teuthology.orchestra.run.smithi088.stderr:2024-06-03T09:06:26.583+0000 7fb07518b640 1 -- 172.21.15.88:0/42259004 --> v1:172.21.15.88:6790/0 -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fb0700aa7a0 con 0x7fb0700adb90 2024-06-03T09:06:26.825 INFO:teuthology.orchestra.run.smithi088.stderr:2024-06-03T09:06:26.825+0000 7fb0677fe640 1 -- 172.21.15.88:0/42259004 <== mon.2 v1:172.21.15.88:6790/0 5 ==== config(0 keys) v1 ==== 4+0+0 (unknown 0 0 0) 0x7fb058003610 con 0x7fb0700adb90 2024-06-03T09:06:26.826 INFO:teuthology.orchestra.run.smithi088.stderr:2024-06-03T09:06:26.825+0000 7fb0677fe640 10 monclient: handle_config config(0 keys) v1 2024-06-03T09:06:26.826 INFO:teuthology.orchestra.run.smithi088.stderr:2024-06-03T09:06:26.825+0000 7fb0677fe640 1 -- 172.21.15.88:0/42259004 <== mon.2 v1:172.21.15.88:6790/0 6 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 2234374065 0 0) 0x7fb058003920 con 0x7fb0700adb90 2024-06-03T09:06:26.826 INFO:teuthology.orchestra.run.smithi088.stderr:2024-06-03T09:06:26.825+0000 7fb0677fe640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-06-03T09:06:26.826 INFO:teuthology.orchestra.run.smithi088.stderr:2024-06-03T09:06:26.825+0000 7fb0677fe640 10 monclient: got monmap 1 from mon.c (according to old e1) 2024-06-03T09:06:26.826 INFO:teuthology.orchestra.run.smithi088.stderr:2024-06-03T09:06:26.825+0000 7fb0677fe640 10 monclient: dump: 2024-06-03T09:06:26.826 INFO:teuthology.orchestra.run.smithi088.stderr:epoch 1 2024-06-03T09:06:26.826 INFO:teuthology.orchestra.run.smithi088.stderr:fsid ec3c3fdd-3ac6-4d7d-9412-683130ed8b04 2024-06-03T09:06:26.826 INFO:teuthology.orchestra.run.smithi088.stderr:last_changed 2024-06-03T08:34:31.885125+0000 2024-06-03T09:06:26.826 INFO:teuthology.orchestra.run.smithi088.stderr:created 2024-06-03T08:34:31.885125+0000 2024-06-03T09:06:26.826 INFO:teuthology.orchestra.run.smithi088.stderr:min_mon_release 18 (reef) 2024-06-03T09:06:26.826 INFO:teuthology.orchestra.run.smithi088.stderr:election_strategy: 3 2024-06-03T09:06:26.827 INFO:teuthology.orchestra.run.smithi088.stderr:0: v1:172.21.15.88:6789/0 mon.a 2024-06-03T09:06:26.827 INFO:teuthology.orchestra.run.smithi088.stderr:1: v1:172.21.15.192:6789/0 mon.b 2024-06-03T09:06:26.827 INFO:teuthology.orchestra.run.smithi088.stderr:2: v1:172.21.15.88:6790/0 mon.c 2024-06-03T09:06:26.827 INFO:teuthology.orchestra.run.smithi088.stderr: 2024-06-03T09:06:26.827 INFO:teuthology.orchestra.run.smithi088.stderr:2024-06-03T09:06:26.825+0000 7fb06dd9b640 4 set_mon_vals no callback set 2024-06-03T09:06:26.827 INFO:teuthology.orchestra.run.smithi088.stderr:2024-06-03T09:06:26.826+0000 7fb0677fe640 1 -- 172.21.15.88:0/42259004 <== mon.2 v1:172.21.15.88:6790/0 7 ==== mgrmap(e 5) v1 ==== 89110+0+0 (unknown 873533973 0 0) 0x7fb0580196a0 con 0x7fb0700adb90 2024-06-03T09:06:26.827 INFO:teuthology.orchestra.run.smithi088.stderr:2024-06-03T09:06:26.827+0000 7fb06ed9d640 10 monclient: get_auth_request con 0x7fb05006d300 auth_method 0 2024-06-03T09:06:26.828 INFO:teuthology.orchestra.run.smithi088.stderr:2024-06-03T09:06:26.827+0000 7fb0677fe640 10 client.30113.objecter ms_handle_connect 0x7fb05006d300 2024-06-03T09:06:26.831 INFO:teuthology.orchestra.run.smithi088.stderr:2024-06-03T09:06:26.831+0000 7fb0677fe640 1 -- 172.21.15.88:0/42259004 <== mon.2 v1:172.21.15.88:6790/0 8 ==== osd_map(1010..1010 src has 622..1010) v4 ==== 5229+0+0 (unknown 1105883993 0 0) 0x7fb058052e40 con 0x7fb0700adb90 2024-06-03T09:06:26.831 INFO:teuthology.orchestra.run.smithi088.stderr:2024-06-03T09:06:26.831+0000 7fb0677fe640 10 client.30113.objecter ms_dispatch 0x7fb070000b90 osd_map(1010..1010 src has 622..1010) v4 2024-06-03T09:06:26.831 INFO:teuthology.orchestra.run.smithi088.stderr:2024-06-03T09:06:26.831+0000 7fb0677fe640 3 client.30113.objecter handle_osd_map got epochs [1010,1010] > 0 2024-06-03T09:06:26.831 INFO:teuthology.orchestra.run.smithi088.stderr:2024-06-03T09:06:26.831+0000 7fb0677fe640 3 client.30113.objecter handle_osd_map decoding full epoch 1010 2024-06-03T09:06:26.831 INFO:teuthology.orchestra.run.smithi088.stderr:2024-06-03T09:06:26.831+0000 7fb0677fe640 20 client.30113.objecter dump_active .. 0 homeless 2024-06-03T09:06:26.831 INFO:teuthology.orchestra.run.smithi088.stderr:2024-06-03T09:06:26.831+0000 7fb0677fe640 1 -- 172.21.15.88:0/42259004 <== mon.2 v1:172.21.15.88:6790/0 9 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+182498 (unknown 1092875540 0 3016760700) 0x7fb05804e9f0 con 0x7fb0700adb90 2024-06-03T09:06:26.831 INFO:teuthology.orchestra.run.smithi088.stderr:2024-06-03T09:06:26.831+0000 7fb0677fe640 10 monclient: handle_mon_command_ack 1 [{"prefix": "get_command_descriptions"}] 2024-06-03T09:06:26.832 INFO:teuthology.orchestra.run.smithi088.stderr:2024-06-03T09:06:26.831+0000 7fb0677fe640 10 monclient: _finish_command 1 = system:0 2024-06-03T09:06:27.018 INFO:teuthology.orchestra.run.smithi088.stderr:2024-06-03T09:06:27.018+0000 7fb07518b640 1 -- 172.21.15.88:0/42259004 --> v1:172.21.15.192:6800/34229 -- mgr_command(tid 0: {"prefix": "pg dump", "target": ["mon-mgr", ""], "format": "json"}) v1 -- 0x7fb07006ff80 con 0x7fb05006d300 2024-06-03T09:06:27.023 INFO:teuthology.orchestra.run.smithi088.stderr:2024-06-03T09:06:27.023+0000 7fb0677fe640 1 -- 172.21.15.88:0/42259004 <== mgr.4110 v1:172.21.15.192:6800/34229 1 ==== mgr_command_reply(tid 0: 0 dumped all) v1 ==== 18+0+151766 (unknown 2965378022 0 2002554278) 0x7fb0600078d0 con 0x7fb05006d300 2024-06-03T09:06:27.025 INFO:teuthology.orchestra.run.smithi088.stdout: 2024-06-03T09:06:27.026 INFO:teuthology.orchestra.run.smithi088.stderr:dumped all 2024-06-03T09:06:27.027 INFO:teuthology.orchestra.run.smithi088.stderr:2024-06-03T09:06:27.026+0000 7fb07518b640 10 librados: watch_flush enter 2024-06-03T09:06:27.028 INFO:teuthology.orchestra.run.smithi088.stderr:2024-06-03T09:06:27.026+0000 7fb07518b640 10 librados: watch_flush exit 2024-06-03T09:06:27.028 INFO:teuthology.orchestra.run.smithi088.stderr:2024-06-03T09:06:27.026+0000 7fb07518b640 20 client.30113.objecter shutdown clearing up homeless session... 2024-06-03T09:06:27.028 INFO:teuthology.orchestra.run.smithi088.stderr:2024-06-03T09:06:27.026+0000 7fb07518b640 10 client.30113.objecter successfully canceled tick 2024-06-03T09:06:27.028 INFO:teuthology.orchestra.run.smithi088.stderr:2024-06-03T09:06:27.026+0000 7fb07518b640 1 -- 172.21.15.88:0/42259004 >> v1:172.21.15.192:6800/34229 conn(0x7fb05006d300 legacy=0x7fb05006f7c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-06-03T09:06:27.028 INFO:teuthology.orchestra.run.smithi088.stderr:2024-06-03T09:06:27.026+0000 7fb07518b640 10 monclient: shutdown 2024-06-03T09:06:27.028 INFO:teuthology.orchestra.run.smithi088.stderr:2024-06-03T09:06:27.026+0000 7fb07518b640 20 monclient: shutdown discarding 0 pending message(s) 2024-06-03T09:06:27.028 INFO:teuthology.orchestra.run.smithi088.stderr:2024-06-03T09:06:27.026+0000 7fb07518b640 1 -- 172.21.15.88:0/42259004 >> v1:172.21.15.88:6790/0 conn(0x7fb0700adb90 legacy=0x7fb0700ac410 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-06-03T09:06:27.028 INFO:teuthology.orchestra.run.smithi088.stderr:2024-06-03T09:06:27.026+0000 7fb07518b640 1 -- 172.21.15.88:0/42259004 shutdown_connections 2024-06-03T09:06:27.028 INFO:teuthology.orchestra.run.smithi088.stderr:2024-06-03T09:06:27.026+0000 7fb07518b640 1 -- 172.21.15.88:0/42259004 >> 172.21.15.88:0/42259004 conn(0x7fb0700b2f20 msgr2=0x7fb07013e910 unknown :-1 s=STATE_NONE l=0).mark_down 2024-06-03T09:06:27.028 INFO:teuthology.orchestra.run.smithi088.stderr:2024-06-03T09:06:27.026+0000 7fb07518b640 1 -- 172.21.15.88:0/42259004 shutdown_connections 2024-06-03T09:06:27.028 INFO:teuthology.orchestra.run.smithi088.stderr:2024-06-03T09:06:27.026+0000 7fb07518b640 1 -- 172.21.15.88:0/42259004 wait complete. 2024-06-03T09:06:27.028 INFO:teuthology.orchestra.run.smithi088.stderr:2024-06-03T09:06:27.026+0000 7fb07518b640 1 librados: shutdown 2024-06-03T09:06:27.036 INFO:teuthology.orchestra.run.smithi088.stdout:{"pg_ready":true,"pg_map":{"version":1918,"stamp":"2024-06-03T09:06:25.099450+0000","last_osdmap_epoch":0,"last_pg_scan":0,"pg_stats_sum":{"stat_sum":{"num_bytes":459285,"num_objects":3,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":1,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":3,"num_whiteouts":0,"num_read":102,"num_read_kb":88,"num_write":120,"num_write_kb":1379,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2,"num_bytes_recovered":397840,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":83,"ondisk_log_size":83,"up":68,"acting":68,"num_store_stats":0},"osd_stats_sum":{"up_from":0,"seq":0,"num_pgs":100,"num_osds":8,"num_per_pool_osds":8,"num_per_pool_omap_osds":8,"kb":754974720,"kb_used":494548,"kb_used_data":46324,"kb_used_omap":12,"kb_used_meta":448115,"kb_avail":754480172,"statfs":{"total":773094113280,"available":772587696128,"internally_reserved":0,"allocated":47435776,"data_stored":22747954,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":12744,"internal_metadata":458870328},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":113,"apply_latency_ms":113,"commit_latency_ns":113000000,"apply_latency_ns":113000000},"alerts":[],"network_ping_times":[]},"pg_stats_delta":{"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":0,"acting":0,"num_store_stats":0,"stamp_delta":"10.002005"},"pg_stats":[{"pgid":"17.d","version":"0'0","reported_seq":942,"reported_epoch":1009,"state":"active+clean","last_fresh":"2024-06-03T09:06:14.443917+0000","last_change":"2024-06-03T08:38:19.260548+0000","last_active":"2024-06-03T09:06:14.443917+0000","last_peered":"2024-06-03T09:06:14.443917+0000","last_clean":"2024-06-03T09:06:14.443917+0000","last_became_active":"2024-06-03T08:38:19.253559+0000","last_became_peered":"2024-06-03T08:38:19.253559+0000","last_unstale":"2024-06-03T09:06:14.443917+0000","last_undegraded":"2024-06-03T09:06:14.443917+0000","last_fullsized":"2024-06-03T09:06:14.443917+0000","mapping_epoch":28,"log_start":"0'0","ondisk_log_start":"0'0","created":28,"last_epoch_clean":45,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-06-03T08:37:55.614497+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-06-03T08:37:55.614497+0000","last_clean_scrub_stamp":"2024-06-03T08:37:55.614497+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-06-04T19:00:12.470979+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1],"acting":[0,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"17.c","version":"0'0","reported_seq":953,"reported_epoch":1009,"state":"active+undersized","last_fresh":"2024-06-03T09:06:13.668156+0000","last_change":"2024-06-03T09:06:03.440074+0000","last_active":"2024-06-03T09:06:13.668156+0000","last_peered":"2024-06-03T09:06:13.668156+0000","last_clean":"2024-06-03T09:05:02.105638+0000","last_became_active":"2024-06-03T09:06:03.440074+0000","last_became_peered":"2024-06-03T09:06:03.440074+0000","last_unstale":"2024-06-03T09:06:13.668156+0000","last_undegraded":"2024-06-03T09:06:13.668156+0000","last_fullsized":"2024-06-03T09:06:03.377982+0000","mapping_epoch":999,"log_start":"0'0","ondisk_log_start":"0'0","created":28,"last_epoch_clean":255,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-06-03T08:37:55.614497+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-06-03T08:37:55.614497+0000","last_clean_scrub_stamp":"2024-06-03T08:37:55.614497+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-06-04T10:34:20.808557+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3],"acting":[3],"avail_no_missing":["3"],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"17.f","version":"0'0","reported_seq":148,"reported_epoch":1009,"state":"active+undersized","last_fresh":"2024-06-03T09:06:14.450543+0000","last_change":"2024-06-03T09:06:03.420285+0000","last_active":"2024-06-03T09:06:14.450543+0000","last_peered":"2024-06-03T09:06:14.450543+0000","last_clean":"2024-06-03T08:37:55.614497+0000","last_became_active":"2024-06-03T09:06:03.420285+0000","last_became_peered":"2024-06-03T09:06:03.420285+0000","last_unstale":"2024-06-03T09:06:14.450543+0000","last_undegraded":"2024-06-03T09:06:14.450543+0000","last_fullsized":"2024-06-03T09:06:03.380616+0000","mapping_epoch":999,"log_start":"0'0","ondisk_log_start":"0'0","created":28,"last_epoch_clean":255,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-06-03T08:37:55.614497+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-06-03T08:37:55.614497+0000","last_clean_scrub_stamp":"2024-06-03T08:37:55.614497+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-06-04T15:39:41.187731+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4],"acting":[4],"avail_no_missing":["4"],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"17.e","version":"0'0","reported_seq":149,"reported_epoch":1009,"state":"active+undersized","last_fresh":"2024-06-03T09:06:13.726992+0000","last_change":"2024-06-03T09:06:03.412887+0000","last_active":"2024-06-03T09:06:13.726992+0000","last_peered":"2024-06-03T09:06:13.726992+0000","last_clean":"2024-06-03T08:37:55.614497+0000","last_became_active":"2024-06-03T09:06:03.412887+0000","last_became_peered":"2024-06-03T09:06:03.412887+0000","last_unstale":"2024-06-03T09:06:13.726992+0000","last_undegraded":"2024-06-03T09:06:13.726992+0000","last_fullsized":"2024-06-03T09:06:03.332482+0000","mapping_epoch":999,"log_start":"0'0","ondisk_log_start":"0'0","created":28,"last_epoch_clean":255,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-06-03T08:37:55.614497+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-06-03T08:37:55.614497+0000","last_clean_scrub_stamp":"2024-06-03T08:37:55.614497+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-06-04T12:23:22.511601+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5],"acting":[5],"avail_no_missing":["5"],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"17.9","version":"0'0","reported_seq":946,"reported_epoch":1009,"state":"active+clean","last_fresh":"2024-06-03T09:06:14.999299+0000","last_change":"2024-06-03T08:38:04.772260+0000","last_active":"2024-06-03T09:06:14.999299+0000","last_peered":"2024-06-03T09:06:14.999299+0000","last_clean":"2024-06-03T09:06:14.999299+0000","last_became_active":"2024-06-03T08:38:04.763428+0000","last_became_peered":"2024-06-03T08:38:04.763428+0000","last_unstale":"2024-06-03T09:06:14.999299+0000","last_undegraded":"2024-06-03T09:06:14.999299+0000","last_fullsized":"2024-06-03T09:06:14.999299+0000","mapping_epoch":28,"log_start":"0'0","ondisk_log_start":"0'0","created":28,"last_epoch_clean":33,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-06-03T08:37:55.614497+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-06-03T08:37:55.614497+0000","last_clean_scrub_stamp":"2024-06-03T08:37:55.614497+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-06-04T10:55:10.773936+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,4],"acting":[7,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"17.8","version":"0'0","reported_seq":961,"reported_epoch":1009,"state":"active+undersized","last_fresh":"2024-06-03T09:06:14.430375+0000","last_change":"2024-06-03T09:06:03.388786+0000","last_active":"2024-06-03T09:06:14.430375+0000","last_peered":"2024-06-03T09:06:14.430375+0000","last_clean":"2024-06-03T09:04:43.297780+0000","last_became_active":"2024-06-03T09:06:03.388786+0000","last_became_peered":"2024-06-03T09:06:03.388786+0000","last_unstale":"2024-06-03T09:06:14.430375+0000","last_undegraded":"2024-06-03T09:06:14.430375+0000","last_fullsized":"2024-06-03T09:06:03.331754+0000","mapping_epoch":999,"log_start":"0'0","ondisk_log_start":"0'0","created":28,"last_epoch_clean":255,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-06-03T08:37:55.614497+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-06-03T08:37:55.614497+0000","last_clean_scrub_stamp":"2024-06-03T08:37:55.614497+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-06-04T09:18:44.765547+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0],"acting":[0],"avail_no_missing":["0"],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"17.b","version":"0'0","reported_seq":149,"reported_epoch":1009,"state":"active+undersized","last_fresh":"2024-06-03T09:06:14.433735+0000","last_change":"2024-06-03T09:06:03.451203+0000","last_active":"2024-06-03T09:06:14.433735+0000","last_peered":"2024-06-03T09:06:14.433735+0000","last_clean":"2024-06-03T08:37:55.614497+0000","last_became_active":"2024-06-03T09:06:03.451203+0000","last_became_peered":"2024-06-03T09:06:03.451203+0000","last_unstale":"2024-06-03T09:06:14.433735+0000","last_undegraded":"2024-06-03T09:06:14.433735+0000","last_fullsized":"2024-06-03T09:06:03.375523+0000","mapping_epoch":999,"log_start":"0'0","ondisk_log_start":"0'0","created":28,"last_epoch_clean":255,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-06-03T08:37:55.614497+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-06-03T08:37:55.614497+0000","last_clean_scrub_stamp":"2024-06-03T08:37:55.614497+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-06-04T10:16:19.813495+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2],"acting":[2],"avail_no_missing":["2"],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"17.a","version":"0'0","reported_seq":925,"reported_epoch":1010,"state":"active+clean","last_fresh":"2024-06-03T09:06:18.651697+0000","last_change":"2024-06-03T08:38:05.843142+0000","last_active":"2024-06-03T09:06:18.651697+0000","last_peered":"2024-06-03T09:06:18.651697+0000","last_clean":"2024-06-03T09:06:18.651697+0000","last_became_active":"2024-06-03T08:38:05.809257+0000","last_became_peered":"2024-06-03T08:38:05.809257+0000","last_unstale":"2024-06-03T09:06:18.651697+0000","last_undegraded":"2024-06-03T09:06:18.651697+0000","last_fullsized":"2024-06-03T09:06:18.651697+0000","mapping_epoch":28,"log_start":"0'0","ondisk_log_start":"0'0","created":28,"last_epoch_clean":33,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-06-03T08:37:55.614497+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-06-03T08:37:55.614497+0000","last_clean_scrub_stamp":"2024-06-03T08:37:55.614497+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-06-04T12:12:03.686838+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2],"acting":[1,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"17.5","version":"0'0","reported_seq":943,"reported_epoch":1009,"state":"active+clean","last_fresh":"2024-06-03T09:06:14.442357+0000","last_change":"2024-06-03T08:38:23.466051+0000","last_active":"2024-06-03T09:06:14.442357+0000","last_peered":"2024-06-03T09:06:14.442357+0000","last_clean":"2024-06-03T09:06:14.442357+0000","last_became_active":"2024-06-03T08:38:23.458039+0000","last_became_peered":"2024-06-03T08:38:23.458039+0000","last_unstale":"2024-06-03T09:06:14.442357+0000","last_undegraded":"2024-06-03T09:06:14.442357+0000","last_fullsized":"2024-06-03T09:06:14.442357+0000","mapping_epoch":28,"log_start":"0'0","ondisk_log_start":"0'0","created":28,"last_epoch_clean":49,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-06-03T08:37:55.614497+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-06-03T08:37:55.614497+0000","last_clean_scrub_stamp":"2024-06-03T08:37:55.614497+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-06-04T12:27:02.532436+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,4],"acting":[0,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"17.4","version":"0'0","reported_seq":149,"reported_epoch":1009,"state":"active+undersized","last_fresh":"2024-06-03T09:06:13.941522+0000","last_change":"2024-06-03T09:06:03.441889+0000","last_active":"2024-06-03T09:06:13.941522+0000","last_peered":"2024-06-03T09:06:13.941522+0000","last_clean":"2024-06-03T08:37:55.614497+0000","last_became_active":"2024-06-03T09:06:03.441889+0000","last_became_peered":"2024-06-03T09:06:03.441889+0000","last_unstale":"2024-06-03T09:06:13.941522+0000","last_undegraded":"2024-06-03T09:06:13.941522+0000","last_fullsized":"2024-06-03T09:06:03.359582+0000","mapping_epoch":999,"log_start":"0'0","ondisk_log_start":"0'0","created":28,"last_epoch_clean":255,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-06-03T08:37:55.614497+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-06-03T08:37:55.614497+0000","last_clean_scrub_stamp":"2024-06-03T08:37:55.614497+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-06-04T12:49:22.907141+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7],"acting":[7],"avail_no_missing":["7"],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"17.7","version":"0'0","reported_seq":929,"reported_epoch":1009,"state":"active+clean","last_fresh":"2024-06-03T09:06:13.736920+0000","last_change":"2024-06-03T08:38:04.837083+0000","last_active":"2024-06-03T09:06:13.736920+0000","last_peered":"2024-06-03T09:06:13.736920+0000","last_clean":"2024-06-03T09:06:13.736920+0000","last_became_active":"2024-06-03T08:38:04.818246+0000","last_became_peered":"2024-06-03T08:38:04.818246+0000","last_unstale":"2024-06-03T09:06:13.736920+0000","last_undegraded":"2024-06-03T09:06:13.736920+0000","last_fullsized":"2024-06-03T09:06:13.736920+0000","mapping_epoch":28,"log_start":"0'0","ondisk_log_start":"0'0","created":28,"last_epoch_clean":34,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-06-03T08:37:55.614497+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-06-03T08:37:55.614497+0000","last_clean_scrub_stamp":"2024-06-03T08:37:55.614497+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-06-04T17:02:38.204813+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,2],"acting":[5,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"17.6","version":"0'0","reported_seq":960,"reported_epoch":1009,"state":"active+undersized","last_fresh":"2024-06-03T09:06:14.453110+0000","last_change":"2024-06-03T09:06:03.428677+0000","last_active":"2024-06-03T09:06:14.453110+0000","last_peered":"2024-06-03T09:06:14.453110+0000","last_clean":"2024-06-03T09:04:43.312352+0000","last_became_active":"2024-06-03T09:06:03.428677+0000","last_became_peered":"2024-06-03T09:06:03.428677+0000","last_unstale":"2024-06-03T09:06:14.453110+0000","last_undegraded":"2024-06-03T09:06:14.453110+0000","last_fullsized":"2024-06-03T09:06:03.359909+0000","mapping_epoch":999,"log_start":"0'0","ondisk_log_start":"0'0","created":28,"last_epoch_clean":255,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-06-03T08:37:55.614497+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-06-03T08:37:55.614497+0000","last_clean_scrub_stamp":"2024-06-03T08:37:55.614497+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-06-04T15:38:02.180241+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0],"acting":[0],"avail_no_missing":["0"],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"2.7","version":"0'0","reported_seq":940,"reported_epoch":1009,"state":"active+clean","last_fresh":"2024-06-03T09:06:13.684065+0000","last_change":"2024-06-03T08:36:30.385405+0000","last_active":"2024-06-03T09:06:13.684065+0000","last_peered":"2024-06-03T09:06:13.684065+0000","last_clean":"2024-06-03T09:06:13.684065+0000","last_became_active":"2024-06-03T08:36:30.380302+0000","last_became_peered":"2024-06-03T08:36:30.380302+0000","last_unstale":"2024-06-03T09:06:13.684065+0000","last_undegraded":"2024-06-03T09:06:13.684065+0000","last_fullsized":"2024-06-03T09:06:13.684065+0000","mapping_epoch":23,"log_start":"0'0","ondisk_log_start":"0'0","created":23,"last_epoch_clean":24,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-06-03T08:36:28.444764+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-06-03T08:36:28.444764+0000","last_clean_scrub_stamp":"2024-06-03T08:36:28.444764+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-06-04T18:46:33.475181+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0],"acting":[3,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"17.14","version":"0'0","reported_seq":926,"reported_epoch":1009,"state":"active+clean","last_fresh":"2024-06-03T09:06:13.721460+0000","last_change":"2024-06-03T08:38:04.547565+0000","last_active":"2024-06-03T09:06:13.721460+0000","last_peered":"2024-06-03T09:06:13.721460+0000","last_clean":"2024-06-03T09:06:13.721460+0000","last_became_active":"2024-06-03T08:38:04.541781+0000","last_became_peered":"2024-06-03T08:38:04.541781+0000","last_unstale":"2024-06-03T09:06:13.721460+0000","last_undegraded":"2024-06-03T09:06:13.721460+0000","last_fullsized":"2024-06-03T09:06:13.721460+0000","mapping_epoch":28,"log_start":"0'0","ondisk_log_start":"0'0","created":28,"last_epoch_clean":34,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-06-03T08:37:55.614497+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-06-03T08:37:55.614497+0000","last_clean_scrub_stamp":"2024-06-03T08:37:55.614497+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-06-04T16:41:55.170134+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,2],"acting":[5,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"2.6","version":"0'0","reported_seq":949,"reported_epoch":1009,"state":"active+clean","last_fresh":"2024-06-03T09:06:14.448100+0000","last_change":"2024-06-03T08:36:30.430760+0000","last_active":"2024-06-03T09:06:14.448100+0000","last_peered":"2024-06-03T09:06:14.448100+0000","last_clean":"2024-06-03T09:06:14.448100+0000","last_became_active":"2024-06-03T08:36:30.404665+0000","last_became_peered":"2024-06-03T08:36:30.404665+0000","last_unstale":"2024-06-03T09:06:14.448100+0000","last_undegraded":"2024-06-03T09:06:14.448100+0000","last_fullsized":"2024-06-03T09:06:14.448100+0000","mapping_epoch":23,"log_start":"0'0","ondisk_log_start":"0'0","created":23,"last_epoch_clean":24,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-06-03T08:36:28.444764+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-06-03T08:36:28.444764+0000","last_clean_scrub_stamp":"2024-06-03T08:36:28.444764+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-06-04T10:37:37.746074+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,0],"acting":[4,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"17.15","version":"0'0","reported_seq":934,"reported_epoch":1009,"state":"active+clean","last_fresh":"2024-06-03T09:06:13.672940+0000","last_change":"2024-06-03T08:38:20.999903+0000","last_active":"2024-06-03T09:06:13.672940+0000","last_peered":"2024-06-03T09:06:13.672940+0000","last_clean":"2024-06-03T09:06:13.672940+0000","last_became_active":"2024-06-03T08:38:20.979523+0000","last_became_peered":"2024-06-03T08:38:20.979523+0000","last_unstale":"2024-06-03T09:06:13.672940+0000","last_undegraded":"2024-06-03T09:06:13.672940+0000","last_fullsized":"2024-06-03T09:06:13.672940+0000","mapping_epoch":28,"log_start":"0'0","ondisk_log_start":"0'0","created":28,"last_epoch_clean":47,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-06-03T08:37:55.614497+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-06-03T08:37:55.614497+0000","last_clean_scrub_stamp":"2024-06-03T08:37:55.614497+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-06-04T20:29:58.314289+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,1],"acting":[3,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"2.5","version":"0'0","reported_seq":942,"reported_epoch":1010,"state":"active+clean","last_fresh":"2024-06-03T09:06:18.382767+0000","last_change":"2024-06-03T08:36:30.024235+0000","last_active":"2024-06-03T09:06:18.382767+0000","last_peered":"2024-06-03T09:06:18.382767+0000","last_clean":"2024-06-03T09:06:18.382767+0000","last_became_active":"2024-06-03T08:36:30.019040+0000","last_became_peered":"2024-06-03T08:36:30.019040+0000","last_unstale":"2024-06-03T09:06:18.382767+0000","last_undegraded":"2024-06-03T09:06:18.382767+0000","last_fullsized":"2024-06-03T09:06:18.382767+0000","mapping_epoch":23,"log_start":"0'0","ondisk_log_start":"0'0","created":23,"last_epoch_clean":24,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-06-03T08:36:28.444764+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-06-03T08:36:28.444764+0000","last_clean_scrub_stamp":"2024-06-03T08:36:28.444764+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-06-04T15:46:47.109502+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,4],"acting":[3,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"17.16","version":"0'0","reported_seq":972,"reported_epoch":1009,"state":"active+undersized","last_fresh":"2024-06-03T09:06:14.441665+0000","last_change":"2024-06-03T09:06:03.478615+0000","last_active":"2024-06-03T09:06:14.441665+0000","last_peered":"2024-06-03T09:06:14.441665+0000","last_clean":"2024-06-03T09:04:43.328415+0000","last_became_active":"2024-06-03T09:06:03.478615+0000","last_became_peered":"2024-06-03T09:06:03.478615+0000","last_unstale":"2024-06-03T09:06:14.441665+0000","last_undegraded":"2024-06-03T09:06:14.441665+0000","last_fullsized":"2024-06-03T09:06:03.402504+0000","mapping_epoch":999,"log_start":"0'0","ondisk_log_start":"0'0","created":28,"last_epoch_clean":255,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-06-03T08:37:55.614497+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-06-03T08:37:55.614497+0000","last_clean_scrub_stamp":"2024-06-03T08:37:55.614497+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-06-04T08:47:08.079233+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2],"acting":[2],"avail_no_missing":["2"],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"2.4","version":"0'0","reported_seq":952,"reported_epoch":1010,"state":"active+undersized","last_fresh":"2024-06-03T09:06:18.331485+0000","last_change":"2024-06-03T09:06:03.387171+0000","last_active":"2024-06-03T09:06:18.331485+0000","last_peered":"2024-06-03T09:06:18.331485+0000","last_clean":"2024-06-03T09:04:43.202645+0000","last_became_active":"2024-06-03T09:06:03.387171+0000","last_became_peered":"2024-06-03T09:06:03.387171+0000","last_unstale":"2024-06-03T09:06:18.331485+0000","last_undegraded":"2024-06-03T09:06:18.331485+0000","last_fullsized":"2024-06-03T09:06:03.283787+0000","mapping_epoch":999,"log_start":"0'0","ondisk_log_start":"0'0","created":23,"last_epoch_clean":255,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-06-03T08:36:28.444764+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-06-03T08:36:28.444764+0000","last_clean_scrub_stamp":"2024-06-03T08:36:28.444764+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-06-04T20:17:43.485023+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5],"acting":[5],"avail_no_missing":["5"],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"17.17","version":"0'0","reported_seq":954,"reported_epoch":1009,"state":"active+clean","last_fresh":"2024-06-03T09:06:14.447046+0000","last_change":"2024-06-03T08:38:38.483275+0000","last_active":"2024-06-03T09:06:14.447046+0000","last_peered":"2024-06-03T09:06:14.447046+0000","last_clean":"2024-06-03T09:06:14.447046+0000","last_became_active":"2024-06-03T08:38:38.467916+0000","last_became_peered":"2024-06-03T08:38:38.467916+0000","last_unstale":"2024-06-03T09:06:14.447046+0000","last_undegraded":"2024-06-03T09:06:14.447046+0000","last_fullsized":"2024-06-03T09:06:14.447046+0000","mapping_epoch":28,"log_start":"0'0","ondisk_log_start":"0'0","created":28,"last_epoch_clean":61,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-06-03T08:37:55.614497+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-06-03T08:37:55.614497+0000","last_clean_scrub_stamp":"2024-06-03T08:37:55.614497+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-06-04T10:02:23.036420+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,7],"acting":[2,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"2.2","version":"0'0","reported_seq":963,"reported_epoch":1009,"state":"active+undersized","last_fresh":"2024-06-03T09:06:13.657892+0000","last_change":"2024-06-03T09:06:03.410020+0000","last_active":"2024-06-03T09:06:13.657892+0000","last_peered":"2024-06-03T09:06:13.657892+0000","last_clean":"2024-06-03T09:05:37.145889+0000","last_became_active":"2024-06-03T09:06:03.410020+0000","last_became_peered":"2024-06-03T09:06:03.410020+0000","last_unstale":"2024-06-03T09:06:13.657892+0000","last_undegraded":"2024-06-03T09:06:13.657892+0000","last_fullsized":"2024-06-03T09:06:03.346864+0000","mapping_epoch":999,"log_start":"0'0","ondisk_log_start":"0'0","created":23,"last_epoch_clean":255,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-06-03T08:36:28.444764+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-06-03T08:36:28.444764+0000","last_clean_scrub_stamp":"2024-06-03T08:36:28.444764+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-06-04T09:57:12.423360+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3],"acting":[3],"avail_no_missing":["3"],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"17.11","version":"0'0","reported_seq":944,"reported_epoch":1009,"state":"active+clean","last_fresh":"2024-06-03T09:06:14.426670+0000","last_change":"2024-06-03T08:38:06.502302+0000","last_active":"2024-06-03T09:06:14.426670+0000","last_peered":"2024-06-03T09:06:14.426670+0000","last_clean":"2024-06-03T09:06:14.426670+0000","last_became_active":"2024-06-03T08:38:06.490204+0000","last_became_peered":"2024-06-03T08:38:06.490204+0000","last_unstale":"2024-06-03T09:06:14.426670+0000","last_undegraded":"2024-06-03T09:06:14.426670+0000","last_fullsized":"2024-06-03T09:06:14.426670+0000","mapping_epoch":28,"log_start":"0'0","ondisk_log_start":"0'0","created":28,"last_epoch_clean":32,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-06-03T08:37:55.614497+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-06-03T08:37:55.614497+0000","last_clean_scrub_stamp":"2024-06-03T08:37:55.614497+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-06-04T10:47:05.988958+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,2],"acting":[4,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"2.1","version":"0'0","reported_seq":952,"reported_epoch":1009,"state":"active+clean","last_fresh":"2024-06-03T09:06:13.929725+0000","last_change":"2024-06-03T08:36:30.883360+0000","last_active":"2024-06-03T09:06:13.929725+0000","last_peered":"2024-06-03T09:06:13.929725+0000","last_clean":"2024-06-03T09:06:13.929725+0000","last_became_active":"2024-06-03T08:36:30.855073+0000","last_became_peered":"2024-06-03T08:36:30.855073+0000","last_unstale":"2024-06-03T09:06:13.929725+0000","last_undegraded":"2024-06-03T09:06:13.929725+0000","last_fullsized":"2024-06-03T09:06:13.929725+0000","mapping_epoch":23,"log_start":"0'0","ondisk_log_start":"0'0","created":23,"last_epoch_clean":25,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-06-03T08:36:28.444764+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-06-03T08:36:28.444764+0000","last_clean_scrub_stamp":"2024-06-03T08:36:28.444764+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-06-04T18:12:22.363473+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,5],"acting":[7,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"17.12","version":"0'0","reported_seq":945,"reported_epoch":1009,"state":"active+clean","last_fresh":"2024-06-03T09:06:13.935141+0000","last_change":"2024-06-03T08:38:01.751562+0000","last_active":"2024-06-03T09:06:13.935141+0000","last_peered":"2024-06-03T09:06:13.935141+0000","last_clean":"2024-06-03T09:06:13.935141+0000","last_became_active":"2024-06-03T08:38:01.745027+0000","last_became_peered":"2024-06-03T08:38:01.745027+0000","last_unstale":"2024-06-03T09:06:13.935141+0000","last_undegraded":"2024-06-03T09:06:13.935141+0000","last_fullsized":"2024-06-03T09:06:13.935141+0000","mapping_epoch":28,"log_start":"0'0","ondisk_log_start":"0'0","created":28,"last_epoch_clean":32,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-06-03T08:37:55.614497+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-06-03T08:37:55.614497+0000","last_clean_scrub_stamp":"2024-06-03T08:37:55.614497+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-06-04T17:54:43.785326+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,3],"acting":[7,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"2.0","version":"0'0","reported_seq":943,"reported_epoch":1009,"state":"active+clean","last_fresh":"2024-06-03T09:06:13.648112+0000","last_change":"2024-06-03T08:36:29.984096+0000","last_active":"2024-06-03T09:06:13.648112+0000","last_peered":"2024-06-03T09:06:13.648112+0000","last_clean":"2024-06-03T09:06:13.648112+0000","last_became_active":"2024-06-03T08:36:29.957875+0000","last_became_peered":"2024-06-03T08:36:29.957875+0000","last_unstale":"2024-06-03T09:06:13.648112+0000","last_undegraded":"2024-06-03T09:06:13.648112+0000","last_fullsized":"2024-06-03T09:06:13.648112+0000","mapping_epoch":23,"log_start":"0'0","ondisk_log_start":"0'0","created":23,"last_epoch_clean":24,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-06-03T08:36:28.444764+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-06-03T08:36:28.444764+0000","last_clean_scrub_stamp":"2024-06-03T08:36:28.444764+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-06-04T08:41:33.679479+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,5],"acting":[3,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"17.13","version":"0'0","reported_seq":943,"reported_epoch":1009,"state":"active+clean","last_fresh":"2024-06-03T09:06:14.438399+0000","last_change":"2024-06-03T08:38:26.838316+0000","last_active":"2024-06-03T09:06:14.438399+0000","last_peered":"2024-06-03T09:06:14.438399+0000","last_clean":"2024-06-03T09:06:14.438399+0000","last_became_active":"2024-06-03T08:38:26.833214+0000","last_became_peered":"2024-06-03T08:38:26.833214+0000","last_unstale":"2024-06-03T09:06:14.438399+0000","last_undegraded":"2024-06-03T09:06:14.438399+0000","last_fullsized":"2024-06-03T09:06:14.438399+0000","mapping_epoch":28,"log_start":"0'0","ondisk_log_start":"0'0","created":28,"last_epoch_clean":52,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-06-03T08:37:55.614497+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-06-03T08:37:55.614497+0000","last_clean_scrub_stamp":"2024-06-03T08:37:55.614497+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-06-04T12:58:51.617998+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,0],"acting":[4,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"2.3","version":"0'0","reported_seq":943,"reported_epoch":1010,"state":"active+clean","last_fresh":"2024-06-03T09:06:18.362614+0000","last_change":"2024-06-03T08:36:30.351670+0000","last_active":"2024-06-03T09:06:18.362614+0000","last_peered":"2024-06-03T09:06:18.362614+0000","last_clean":"2024-06-03T09:06:18.362614+0000","last_became_active":"2024-06-03T08:36:30.343560+0000","last_became_peered":"2024-06-03T08:36:30.343560+0000","last_unstale":"2024-06-03T09:06:18.362614+0000","last_undegraded":"2024-06-03T09:06:18.362614+0000","last_fullsized":"2024-06-03T09:06:18.362614+0000","mapping_epoch":23,"log_start":"0'0","ondisk_log_start":"0'0","created":23,"last_epoch_clean":24,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-06-03T08:36:28.444764+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-06-03T08:36:28.444764+0000","last_clean_scrub_stamp":"2024-06-03T08:36:28.444764+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-06-04T12:02:24.765135+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,2],"acting":[3,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.0","version":"19'76","reported_seq":1051,"reported_epoch":1009,"state":"active+clean","last_fresh":"2024-06-03T09:06:13.703685+0000","last_change":"2024-06-03T08:36:26.702702+0000","last_active":"2024-06-03T09:06:13.703685+0000","last_peered":"2024-06-03T09:06:13.703685+0000","last_clean":"2024-06-03T09:06:13.703685+0000","last_became_active":"2024-06-03T08:36:26.128378+0000","last_became_peered":"2024-06-03T08:36:26.128378+0000","last_unstale":"2024-06-03T09:06:13.703685+0000","last_undegraded":"2024-06-03T09:06:13.703685+0000","last_fullsized":"2024-06-03T09:06:13.703685+0000","mapping_epoch":20,"log_start":"0'0","ondisk_log_start":"0'0","created":16,"last_epoch_clean":22,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-06-03T08:36:15.542635+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-06-03T08:36:15.542635+0000","last_clean_scrub_stamp":"2024-06-03T08:36:15.542635+0000","objects_scrubbed":0,"log_size":76,"log_dups_size":0,"ondisk_log_size":76,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-06-04T12:41:53.731360+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":459280,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":96,"num_read_kb":82,"num_write":113,"num_write_kb":1372,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2,"num_bytes_recovered":397840,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,5],"acting":[1,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"17.10","version":"0'0","reported_seq":944,"reported_epoch":1009,"state":"active+clean","last_fresh":"2024-06-03T09:06:14.422035+0000","last_change":"2024-06-03T08:38:15.399189+0000","last_active":"2024-06-03T09:06:14.422035+0000","last_peered":"2024-06-03T09:06:14.422035+0000","last_clean":"2024-06-03T09:06:14.422035+0000","last_became_active":"2024-06-03T08:38:15.337723+0000","last_became_peered":"2024-06-03T08:38:15.337723+0000","last_unstale":"2024-06-03T09:06:14.422035+0000","last_undegraded":"2024-06-03T09:06:14.422035+0000","last_fullsized":"2024-06-03T09:06:14.422035+0000","mapping_epoch":28,"log_start":"0'0","ondisk_log_start":"0'0","created":28,"last_epoch_clean":42,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-06-03T08:37:55.614497+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-06-03T08:37:55.614497+0000","last_clean_scrub_stamp":"2024-06-03T08:37:55.614497+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-06-04T09:29:46.427317+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,3],"acting":[4,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"17.1a","version":"0'0","reported_seq":960,"reported_epoch":1009,"state":"active+undersized","last_fresh":"2024-06-03T09:06:14.431224+0000","last_change":"2024-06-03T09:06:03.390734+0000","last_active":"2024-06-03T09:06:14.431224+0000","last_peered":"2024-06-03T09:06:14.431224+0000","last_clean":"2024-06-03T09:05:20.322313+0000","last_became_active":"2024-06-03T09:06:03.390734+0000","last_became_peered":"2024-06-03T09:06:03.390734+0000","last_unstale":"2024-06-03T09:06:14.431224+0000","last_undegraded":"2024-06-03T09:06:14.431224+0000","last_fullsized":"2024-06-03T09:06:03.343197+0000","mapping_epoch":999,"log_start":"0'0","ondisk_log_start":"0'0","created":28,"last_epoch_clean":254,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-06-03T08:37:55.614497+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-06-03T08:37:55.614497+0000","last_clean_scrub_stamp":"2024-06-03T08:37:55.614497+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-06-04T15:56:47.628875+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4],"acting":[4],"avail_no_missing":["4"],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"17.1b","version":"0'0","reported_seq":935,"reported_epoch":1009,"state":"active+clean","last_fresh":"2024-06-03T09:06:13.663279+0000","last_change":"2024-06-03T08:38:03.115748+0000","last_active":"2024-06-03T09:06:13.663279+0000","last_peered":"2024-06-03T09:06:13.663279+0000","last_clean":"2024-06-03T09:06:13.663279+0000","last_became_active":"2024-06-03T08:38:03.110533+0000","last_became_peered":"2024-06-03T08:38:03.110533+0000","last_unstale":"2024-06-03T09:06:13.663279+0000","last_undegraded":"2024-06-03T09:06:13.663279+0000","last_fullsized":"2024-06-03T09:06:13.663279+0000","mapping_epoch":28,"log_start":"0'0","ondisk_log_start":"0'0","created":28,"last_epoch_clean":32,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-06-03T08:37:55.614497+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-06-03T08:37:55.614497+0000","last_clean_scrub_stamp":"2024-06-03T08:37:55.614497+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-06-04T12:36:47.471609+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,7],"acting":[3,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"17.18","version":"0'0","reported_seq":944,"reported_epoch":1009,"state":"active+clean","last_fresh":"2024-06-03T09:06:13.924799+0000","last_change":"2024-06-03T08:38:04.632864+0000","last_active":"2024-06-03T09:06:13.924799+0000","last_peered":"2024-06-03T09:06:13.924799+0000","last_clean":"2024-06-03T09:06:13.924799+0000","last_became_active":"2024-06-03T08:38:04.627856+0000","last_became_peered":"2024-06-03T08:38:04.627856+0000","last_unstale":"2024-06-03T09:06:13.924799+0000","last_undegraded":"2024-06-03T09:06:13.924799+0000","last_fullsized":"2024-06-03T09:06:13.924799+0000","mapping_epoch":28,"log_start":"0'0","ondisk_log_start":"0'0","created":28,"last_epoch_clean":33,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-06-03T08:37:55.614497+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-06-03T08:37:55.614497+0000","last_clean_scrub_stamp":"2024-06-03T08:37:55.614497+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-06-04T19:55:51.181598+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,4],"acting":[7,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"17.19","version":"0'0","reported_seq":934,"reported_epoch":1009,"state":"active+clean","last_fresh":"2024-06-03T09:06:13.652943+0000","last_change":"2024-06-03T08:38:01.839491+0000","last_active":"2024-06-03T09:06:13.652943+0000","last_peered":"2024-06-03T09:06:13.652943+0000","last_clean":"2024-06-03T09:06:13.652943+0000","last_became_active":"2024-06-03T08:38:01.829539+0000","last_became_peered":"2024-06-03T08:38:01.829539+0000","last_unstale":"2024-06-03T09:06:13.652943+0000","last_undegraded":"2024-06-03T09:06:13.652943+0000","last_fullsized":"2024-06-03T09:06:13.652943+0000","mapping_epoch":28,"log_start":"0'0","ondisk_log_start":"0'0","created":28,"last_epoch_clean":30,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-06-03T08:37:55.614497+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-06-03T08:37:55.614497+0000","last_clean_scrub_stamp":"2024-06-03T08:37:55.614497+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-06-04T09:10:43.860229+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,2],"acting":[3,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"17.1e","version":"0'0","reported_seq":148,"reported_epoch":1009,"state":"active+undersized","last_fresh":"2024-06-03T09:06:14.445917+0000","last_change":"2024-06-03T09:06:03.406181+0000","last_active":"2024-06-03T09:06:14.445917+0000","last_peered":"2024-06-03T09:06:14.445917+0000","last_clean":"2024-06-03T08:37:55.614497+0000","last_became_active":"2024-06-03T09:06:03.406181+0000","last_became_peered":"2024-06-03T09:06:03.406181+0000","last_unstale":"2024-06-03T09:06:14.445917+0000","last_undegraded":"2024-06-03T09:06:14.445917+0000","last_fullsized":"2024-06-03T09:06:03.363739+0000","mapping_epoch":999,"log_start":"0'0","ondisk_log_start":"0'0","created":28,"last_epoch_clean":255,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-06-03T08:37:55.614497+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-06-03T08:37:55.614497+0000","last_clean_scrub_stamp":"2024-06-03T08:37:55.614497+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-06-04T11:45:24.486600+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4],"acting":[4],"avail_no_missing":["4"],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"17.1f","version":"0'0","reported_seq":941,"reported_epoch":1009,"state":"active+clean","last_fresh":"2024-06-03T09:06:14.450053+0000","last_change":"2024-06-03T08:38:04.383538+0000","last_active":"2024-06-03T09:06:14.450053+0000","last_peered":"2024-06-03T09:06:14.450053+0000","last_clean":"2024-06-03T09:06:14.450053+0000","last_became_active":"2024-06-03T08:38:04.367278+0000","last_became_peered":"2024-06-03T08:38:04.367278+0000","last_unstale":"2024-06-03T09:06:14.450053+0000","last_undegraded":"2024-06-03T09:06:14.450053+0000","last_fullsized":"2024-06-03T09:06:14.450053+0000","mapping_epoch":28,"log_start":"0'0","ondisk_log_start":"0'0","created":28,"last_epoch_clean":32,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-06-03T08:37:55.614497+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-06-03T08:37:55.614497+0000","last_clean_scrub_stamp":"2024-06-03T08:37:55.614497+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-06-04T10:39:26.552282+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,2],"acting":[0,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"17.1c","version":"0'0","reported_seq":944,"reported_epoch":1010,"state":"active+undersized","last_fresh":"2024-06-03T09:06:18.327492+0000","last_change":"2024-06-03T09:06:03.382210+0000","last_active":"2024-06-03T09:06:18.327492+0000","last_peered":"2024-06-03T09:06:18.327492+0000","last_clean":"2024-06-03T09:04:43.199660+0000","last_became_active":"2024-06-03T09:06:03.382210+0000","last_became_peered":"2024-06-03T09:06:03.382210+0000","last_unstale":"2024-06-03T09:06:18.327492+0000","last_undegraded":"2024-06-03T09:06:18.327492+0000","last_fullsized":"2024-06-03T09:06:03.277874+0000","mapping_epoch":999,"log_start":"0'0","ondisk_log_start":"0'0","created":28,"last_epoch_clean":255,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-06-03T08:37:55.614497+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-06-03T08:37:55.614497+0000","last_clean_scrub_stamp":"2024-06-03T08:37:55.614497+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-06-04T12:18:50.013884+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5],"acting":[5],"avail_no_missing":["5"],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"17.1d","version":"0'0","reported_seq":923,"reported_epoch":1009,"state":"active+clean","last_fresh":"2024-06-03T09:06:13.715906+0000","last_change":"2024-06-03T08:38:06.794630+0000","last_active":"2024-06-03T09:06:13.715906+0000","last_peered":"2024-06-03T09:06:13.715906+0000","last_clean":"2024-06-03T09:06:13.715906+0000","last_became_active":"2024-06-03T08:38:06.592317+0000","last_became_peered":"2024-06-03T08:38:06.592317+0000","last_unstale":"2024-06-03T09:06:13.715906+0000","last_undegraded":"2024-06-03T09:06:13.715906+0000","last_fullsized":"2024-06-03T09:06:13.715906+0000","mapping_epoch":28,"log_start":"0'0","ondisk_log_start":"0'0","created":28,"last_epoch_clean":33,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-06-03T08:37:55.614497+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-06-03T08:37:55.614497+0000","last_clean_scrub_stamp":"2024-06-03T08:37:55.614497+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-06-04T12:33:30.785369+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,7],"acting":[1,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"17.2","version":"0'0","reported_seq":925,"reported_epoch":1009,"state":"active+clean","last_fresh":"2024-06-03T09:06:13.708688+0000","last_change":"2024-06-03T08:38:14.981410+0000","last_active":"2024-06-03T09:06:13.708688+0000","last_peered":"2024-06-03T09:06:13.708688+0000","last_clean":"2024-06-03T09:06:13.708688+0000","last_became_active":"2024-06-03T08:38:14.973825+0000","last_became_peered":"2024-06-03T08:38:14.973825+0000","last_unstale":"2024-06-03T09:06:13.708688+0000","last_undegraded":"2024-06-03T09:06:13.708688+0000","last_fullsized":"2024-06-03T09:06:13.708688+0000","mapping_epoch":28,"log_start":"0'0","ondisk_log_start":"0'0","created":28,"last_epoch_clean":41,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-06-03T08:37:55.614497+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-06-03T08:37:55.614497+0000","last_clean_scrub_stamp":"2024-06-03T08:37:55.614497+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-06-04T17:01:05.518613+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,4],"acting":[1,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"17.3","version":"0'0","reported_seq":943,"reported_epoch":1009,"state":"active+clean","last_fresh":"2024-06-03T09:06:14.437636+0000","last_change":"2024-06-03T08:38:14.619745+0000","last_active":"2024-06-03T09:06:14.437636+0000","last_peered":"2024-06-03T09:06:14.437636+0000","last_clean":"2024-06-03T09:06:14.437636+0000","last_became_active":"2024-06-03T08:38:14.510665+0000","last_became_peered":"2024-06-03T08:38:14.510665+0000","last_unstale":"2024-06-03T09:06:14.437636+0000","last_undegraded":"2024-06-03T09:06:14.437636+0000","last_fullsized":"2024-06-03T09:06:14.437636+0000","mapping_epoch":28,"log_start":"0'0","ondisk_log_start":"0'0","created":28,"last_epoch_clean":42,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-06-03T08:37:55.614497+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-06-03T08:37:55.614497+0000","last_clean_scrub_stamp":"2024-06-03T08:37:55.614497+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-06-04T12:03:47.392064+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,7],"acting":[0,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"17.0","version":"0'0","reported_seq":944,"reported_epoch":1009,"state":"active+clean","last_fresh":"2024-06-03T09:06:14.427123+0000","last_change":"2024-06-03T08:38:04.237118+0000","last_active":"2024-06-03T09:06:14.427123+0000","last_peered":"2024-06-03T09:06:14.427123+0000","last_clean":"2024-06-03T09:06:14.427123+0000","last_became_active":"2024-06-03T08:38:04.231954+0000","last_became_peered":"2024-06-03T08:38:04.231954+0000","last_unstale":"2024-06-03T09:06:14.427123+0000","last_undegraded":"2024-06-03T09:06:14.427123+0000","last_fullsized":"2024-06-03T09:06:14.427123+0000","mapping_epoch":28,"log_start":"0'0","ondisk_log_start":"0'0","created":28,"last_epoch_clean":32,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-06-03T08:37:55.614497+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-06-03T08:37:55.614497+0000","last_clean_scrub_stamp":"2024-06-03T08:37:55.614497+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-06-04T09:56:10.741637+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,2],"acting":[0,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"17.1","version":"65'7","reported_seq":959,"reported_epoch":1009,"state":"active+undersized+degraded","last_fresh":"2024-06-03T09:06:13.709209+0000","last_change":"2024-06-03T09:06:03.370660+0000","last_active":"2024-06-03T09:06:13.709209+0000","last_peered":"2024-06-03T09:06:13.709209+0000","last_clean":"2024-06-03T09:05:03.733864+0000","last_became_active":"2024-06-03T09:06:03.370660+0000","last_became_peered":"2024-06-03T09:06:03.370660+0000","last_unstale":"2024-06-03T09:06:13.709209+0000","last_undegraded":"2024-06-03T09:06:03.266076+0000","last_fullsized":"2024-06-03T09:06:03.264525+0000","mapping_epoch":999,"log_start":"0'0","ondisk_log_start":"0'0","created":28,"last_epoch_clean":255,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-06-03T08:37:55.614497+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-06-03T08:37:55.614497+0000","last_clean_scrub_stamp":"2024-06-03T08:37:55.614497+0000","objects_scrubbed":0,"log_size":7,"log_dups_size":0,"ondisk_log_size":7,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-06-04T11:55:18.194283+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":5,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":1,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":6,"num_read_kb":6,"num_write":7,"num_write_kb":7,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5],"acting":[5],"avail_no_missing":["5"],"object_location_counts":[{"shards":"5","objects":1}],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]}],"pool_stats":[{"poolid":2,"num_pg":8,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":14,"acting":14,"num_store_stats":7},{"poolid":1,"num_pg":1,"stat_sum":{"num_bytes":459280,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":96,"num_read_kb":82,"num_write":113,"num_write_kb":1372,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2,"num_bytes_recovered":397840,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":802816,"data_stored":795680,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":76,"ondisk_log_size":76,"up":2,"acting":2,"num_store_stats":2},{"poolid":17,"num_pg":32,"stat_sum":{"num_bytes":5,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":1,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":6,"num_read_kb":6,"num_write":7,"num_write_kb":7,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":8192,"data_stored":10,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":7,"ondisk_log_size":7,"up":52,"acting":52,"num_store_stats":8}],"osd_stats":[{"osd":6,"up_from":253,"seq":1086626726200,"num_pgs":32,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":54772,"kb_used_data":5608,"kb_used_omap":1,"kb_used_meta":49150,"kb_avail":94317068,"statfs":{"total":96636764160,"available":96580677632,"internally_reserved":0,"allocated":5742592,"data_stored":2767180,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1593,"internal_metadata":50330055},"hb_peers":[0,1,2,3,4,5,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":109,"apply_latency_ms":109,"commit_latency_ns":109000000,"apply_latency_ns":109000000},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Mon Jun 3 09:04:02 2024","interfaces":[{"interface":"back","average":{"1min":1582.4780000000001,"5min":332.72699999999998,"15min":124.113},"min":{"1min":4.4829999999999997,"5min":4.1959999999999997,"15min":4.1959999999999997},"max":{"1min":22766.882000000001,"5min":22766.882000000001,"15min":22766.882000000001},"last":21.917999999999999},{"interface":"front","average":{"1min":1579.5409999999999,"5min":334.94499999999999,"15min":125.84999999999999},"min":{"1min":4.8129999999999997,"5min":4.7539999999999996,"15min":4.4530000000000003},"max":{"1min":22766.289000000001,"5min":22766.289000000001,"15min":22766.289000000001},"last":25.408999999999999}]},{"osd":1,"last update":"Mon Jun 3 09:03:40 2024","interfaces":[{"interface":"back","average":{"1min":36.420999999999999,"5min":77.054000000000002,"15min":132.84800000000001},"min":{"1min":6.0380000000000003,"5min":5.8799999999999999,"15min":5.484},"max":{"1min":589.64099999999996,"5min":4138.1679999999997,"15min":9955.1299999999992},"last":26.84},{"interface":"front","average":{"1min":40.240000000000002,"5min":79.674999999999997,"15min":133.38999999999999},"min":{"1min":8.4589999999999996,"5min":8.4090000000000007,"15min":6.1390000000000002},"max":{"1min":588.60799999999995,"5min":4137.6530000000002,"15min":9827.5509999999995},"last":26.263000000000002}]},{"osd":2,"last update":"Mon Jun 3 09:03:40 2024","interfaces":[{"interface":"back","average":{"1min":12.613,"5min":24.600999999999999,"15min":21.088000000000001},"min":{"1min":6.9740000000000002,"5min":6.9740000000000002,"15min":6.5330000000000004},"max":{"1min":26.076000000000001,"5min":376.24900000000002,"15min":376.24900000000002},"last":24.899000000000001},{"interface":"front","average":{"1min":12.493,"5min":22.207999999999998,"15min":21.219000000000001},"min":{"1min":6.5810000000000004,"5min":6.5110000000000001,"15min":6.0750000000000002},"max":{"1min":29.690999999999999,"5min":271.51100000000002,"15min":434.77300000000002},"last":40.372}]},{"osd":3,"last update":"Mon Jun 3 09:03:40 2024","interfaces":[{"interface":"back","average":{"1min":14.093999999999999,"5min":24.143999999999998,"15min":29.376999999999999},"min":{"1min":10.196999999999999,"5min":6.2919999999999998,"15min":6.1719999999999997},"max":{"1min":24.832000000000001,"5min":197.84899999999999,"15min":1235.5799999999999},"last":24.353999999999999},{"interface":"front","average":{"1min":9.3260000000000005,"5min":18.463999999999999,"15min":27.631},"min":{"1min":6.125,"5min":6.125,"15min":6.125},"max":{"1min":28.166,"5min":200.173,"15min":1157.6780000000001},"last":23.454000000000001}]},{"osd":4,"last update":"Mon Jun 3 09:04:29 2024","interfaces":[{"interface":"back","average":{"1min":10.279,"5min":1253.809,"15min":450.79199999999997},"min":{"1min":7.2300000000000004,"5min":6.093,"15min":6.093},"max":{"1min":15.000999999999999,"5min":28093.096000000001,"15min":28093.096000000001},"last":101.45399999999999},{"interface":"front","average":{"1min":15.693,"5min":1279.759,"15min":463.87099999999998},"min":{"1min":10.551,"5min":7.7480000000000002,"15min":7.7480000000000002},"max":{"1min":27.898,"5min":28323.837,"15min":28323.837},"last":102.262}]},{"osd":5,"last update":"Mon Jun 3 09:04:10 2024","interfaces":[{"interface":"back","average":{"1min":13.728999999999999,"5min":24.670000000000002,"15min":23.815999999999999},"min":{"1min":10.401,"5min":7.516,"15min":7.516},"max":{"1min":25.138000000000002,"5min":320.85000000000002,"15min":445.041},"last":103.52500000000001},{"interface":"front","average":{"1min":13.847,"5min":29.283000000000001,"15min":23.393999999999998},"min":{"1min":7.944,"5min":6.266,"15min":6.266},"max":{"1min":18.817,"5min":530.56700000000001,"15min":530.56700000000001},"last":29.263999999999999}]},{"osd":7,"last update":"Mon Jun 3 09:03:40 2024","interfaces":[{"interface":"back","average":{"1min":20.568999999999999,"5min":30.286999999999999,"15min":27.977},"min":{"1min":13.874000000000001,"5min":10.58,"15min":9.2859999999999996},"max":{"1min":86.959000000000003,"5min":222.07599999999999,"15min":255.08799999999999},"last":37.618000000000002},{"interface":"front","average":{"1min":19.102,"5min":29.550000000000001,"15min":27.763000000000002},"min":{"1min":10.052,"5min":8.6709999999999994,"15min":8.5429999999999993},"max":{"1min":67.968999999999994,"5min":293.62099999999998,"15min":438.14600000000002},"last":111.985}]}]},{"osd":7,"up_from":20,"seq":85899346286,"num_pgs":9,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":58964,"kb_used_data":5704,"kb_used_omap":1,"kb_used_meta":53246,"kb_avail":94312876,"statfs":{"total":96636764160,"available":96576385024,"internally_reserved":0,"allocated":5840896,"data_stored":2740727,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1589,"internal_metadata":54524363},"hb_peers":[0,1,2,3,4,5],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Mon Jun 3 09:06:09 2024","interfaces":[{"interface":"back","average":{"1min":14.073,"5min":894.22400000000005,"15min":308.93400000000003},"min":{"1min":5.5709999999999997,"5min":4.218,"15min":4.0599999999999996},"max":{"1min":90.757999999999996,"5min":21729.001,"15min":21729.001},"last":4.032},{"interface":"front","average":{"1min":9.2550000000000008,"5min":906.23500000000001,"15min":314.06},"min":{"1min":5.5789999999999997,"5min":4.8810000000000002,"15min":4.8810000000000002},"max":{"1min":25.937000000000001,"5min":21899.732,"15min":21899.732},"last":7.5819999999999999}]},{"osd":1,"last update":"Mon Jun 3 09:06:09 2024","interfaces":[{"interface":"back","average":{"1min":15.182,"5min":69.183999999999997,"15min":219.173},"min":{"1min":7.3639999999999999,"5min":6.6200000000000001,"15min":6.1319999999999997},"max":{"1min":76.926000000000002,"5min":2795.1329999999998,"15min":18373.257000000001},"last":9.7349999999999994},{"interface":"front","average":{"1min":21.495999999999999,"5min":74.703999999999994,"15min":225.285},"min":{"1min":8.8399999999999999,"5min":8.8399999999999999,"15min":8.5129999999999999},"max":{"1min":94.977999999999994,"5min":2803.3670000000002,"15min":18374.601999999999},"last":13.657999999999999}]},{"osd":2,"last update":"Mon Jun 3 09:06:09 2024","interfaces":[{"interface":"back","average":{"1min":19.164000000000001,"5min":26.902000000000001,"15min":22.920999999999999},"min":{"1min":11.207000000000001,"5min":8.5030000000000001,"15min":8.5030000000000001},"max":{"1min":91.319999999999993,"5min":333.99200000000002,"15min":333.99200000000002},"last":12.694000000000001},{"interface":"front","average":{"1min":11.125999999999999,"5min":18.132000000000001,"15min":16.353000000000002},"min":{"1min":6.202,"5min":5.8369999999999997,"15min":5.5069999999999997},"max":{"1min":28.649000000000001,"5min":226.18100000000001,"15min":226.18100000000001},"last":10.028}]},{"osd":3,"last update":"Mon Jun 3 09:06:09 2024","interfaces":[{"interface":"back","average":{"1min":22.291,"5min":26.289999999999999,"15min":31.074999999999999},"min":{"1min":11.714,"5min":6.8959999999999999,"15min":6.3760000000000003},"max":{"1min":83.102000000000004,"5min":316.64400000000001,"15min":2039.9059999999999},"last":14.345000000000001},{"interface":"front","average":{"1min":13.324999999999999,"5min":17.963000000000001,"15min":25.969999999999999},"min":{"1min":6.1200000000000001,"5min":5.2089999999999996,"15min":5.2089999999999996},"max":{"1min":49.908999999999999,"5min":233.542,"15min":2037.5550000000001},"last":9.5060000000000002}]},{"osd":4,"last update":"Mon Jun 3 09:05:28 2024","interfaces":[{"interface":"back","average":{"1min":22.209,"5min":1379.51,"15min":1803.278},"min":{"1min":5.9089999999999998,"5min":5.9089999999999998,"15min":5.9089999999999998},"max":{"1min":224.54300000000001,"5min":30073.917000000001,"15min":62560.868000000002},"last":11.138},{"interface":"front","average":{"1min":22.712,"5min":1401.0719999999999,"15min":1806.759},"min":{"1min":7.6449999999999996,"5min":6.7930000000000001,"15min":5.9340000000000002},"max":{"1min":223.58600000000001,"5min":30348.021000000001,"15min":62451.446000000004},"last":6.569}]},{"osd":5,"last update":"Mon Jun 3 09:06:02 2024","interfaces":[{"interface":"back","average":{"1min":24.372,"5min":26.856000000000002,"15min":23.571999999999999},"min":{"1min":12.278,"5min":9.2460000000000004,"15min":6.5650000000000004},"max":{"1min":183.64099999999999,"5min":317.572,"15min":317.572},"last":15.047000000000001},{"interface":"front","average":{"1min":21.259,"5min":23.364000000000001,"15min":21.193999999999999},"min":{"1min":7.8499999999999996,"5min":7.0999999999999996,"15min":7.008},"max":{"1min":184.53399999999999,"5min":306.60599999999999,"15min":306.60599999999999},"last":11.656000000000001}]},{"osd":6,"last update":"Mon Jun 3 09:04:31 2024","interfaces":[{"interface":"back","average":{"1min":44.061,"5min":26.821000000000002,"15min":91.176000000000002},"min":{"1min":9.0800000000000001,"5min":7.9390000000000001,"15min":7.9390000000000001},"max":{"1min":268.029,"5min":308.916,"15min":8734.4639999999999},"last":6.6769999999999996},{"interface":"front","average":{"1min":46.643000000000001,"5min":23.879999999999999,"15min":88.765000000000001},"min":{"1min":8.8119999999999994,"5min":6.9240000000000004,"15min":6.742},"max":{"1min":305.64600000000002,"5min":305.64600000000002,"15min":8735.5319999999992},"last":12.327}]}]},{"osd":5,"up_from":20,"seq":85899346260,"num_pgs":9,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":59360,"kb_used_data":6100,"kb_used_omap":1,"kb_used_meta":53246,"kb_avail":94312480,"statfs":{"total":96636764160,"available":96575979520,"internally_reserved":0,"allocated":6246400,"data_stored":3138572,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1594,"internal_metadata":54524358},"hb_peers":[0,1,2,3,4,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Mon Jun 3 09:06:06 2024","interfaces":[{"interface":"back","average":{"1min":7.468,"5min":1703.037,"15min":580.51900000000001},"min":{"1min":4.6509999999999998,"5min":4.6509999999999998,"15min":4.4020000000000001},"max":{"1min":14.711,"5min":26433.764999999999,"15min":26433.764999999999},"last":9.3849999999999998},{"interface":"front","average":{"1min":9.9429999999999996,"5min":1725.27,"15min":587.87800000000004},"min":{"1min":5.5999999999999996,"5min":5.1319999999999997,"15min":4.8170000000000002},"max":{"1min":13.590999999999999,"5min":26704.698,"15min":26704.698},"last":12.407999999999999}]},{"osd":1,"last update":"Mon Jun 3 09:06:00 2024","interfaces":[{"interface":"back","average":{"1min":10.862,"5min":67.731999999999999,"15min":217.364},"min":{"1min":7.1769999999999996,"5min":6.0890000000000004,"15min":5.4930000000000003},"max":{"1min":16.998999999999999,"5min":3923.4090000000001,"15min":18811.696},"last":12.917999999999999},{"interface":"front","average":{"1min":10.417,"5min":65.141999999999996,"15min":220.75700000000001},"min":{"1min":5.3209999999999997,"5min":5.3209999999999997,"15min":5.0209999999999999},"max":{"1min":15.821999999999999,"5min":4019.6129999999998,"15min":18968.365000000002},"last":8.6590000000000007}]},{"osd":2,"last update":"Mon Jun 3 09:06:00 2024","interfaces":[{"interface":"back","average":{"1min":11.366,"5min":24.18,"15min":21.137},"min":{"1min":5.6369999999999996,"5min":5.6369999999999996,"15min":5.0510000000000002},"max":{"1min":15.529999999999999,"5min":332.48200000000003,"15min":332.48200000000003},"last":13.375},{"interface":"front","average":{"1min":10.089,"5min":23.117000000000001,"15min":24.050000000000001},"min":{"1min":6.6769999999999996,"5min":6.6769999999999996,"15min":5.8739999999999997},"max":{"1min":15.821,"5min":353.03199999999998,"15min":353.03199999999998},"last":8.5139999999999993}]},{"osd":3,"last update":"Mon Jun 3 09:06:00 2024","interfaces":[{"interface":"back","average":{"1min":13.472,"5min":24.283999999999999,"15min":31.666},"min":{"1min":10.111000000000001,"5min":7.4720000000000004,"15min":7.4210000000000003},"max":{"1min":16.271999999999998,"5min":353.57499999999999,"15min":2855.8519999999999},"last":9.9960000000000004},{"interface":"front","average":{"1min":9.5899999999999999,"5min":15.875,"15min":25.834},"min":{"1min":6.6609999999999996,"5min":5.7729999999999997,"15min":5.1909999999999998},"max":{"1min":15.112,"5min":203.804,"15min":2869.7860000000001},"last":6.3769999999999998}]},{"osd":4,"last update":"Mon Jun 3 09:05:27 2024","interfaces":[{"interface":"back","average":{"1min":19.155000000000001,"5min":1084.961,"15min":1346.9949999999999},"min":{"1min":7.6130000000000004,"5min":7.6130000000000004,"15min":6.1340000000000003},"max":{"1min":156.55199999999999,"5min":28619.825000000001,"15min":57999.453999999998},"last":14.244999999999999},{"interface":"front","average":{"1min":28.068999999999999,"5min":1087.106,"15min":1330.289},"min":{"1min":8.3949999999999996,"5min":6.5030000000000001,"15min":6.5030000000000001},"max":{"1min":254.91300000000001,"5min":28621.294000000002,"15min":57434.631999999998},"last":14.859}]},{"osd":6,"last update":"Mon Jun 3 09:03:57 2024","interfaces":[{"interface":"back","average":{"1min":23.536000000000001,"5min":36.457000000000001,"15min":83.799999999999997},"min":{"1min":9.4800000000000004,"5min":9.4800000000000004,"15min":8.0630000000000006},"max":{"1min":89.442999999999998,"5min":508.77999999999997,"15min":9193.5169999999998},"last":157.79300000000001},{"interface":"front","average":{"1min":18.891999999999999,"5min":22.100999999999999,"15min":75.489999999999995},"min":{"1min":5.923,"5min":5.923,"15min":5.923},"max":{"1min":78.424999999999997,"5min":464.62,"15min":9191.4429999999993},"last":148.09}]},{"osd":7,"last update":"Mon Jun 3 09:06:00 2024","interfaces":[{"interface":"back","average":{"1min":13.257,"5min":20.917999999999999,"15min":21.088999999999999},"min":{"1min":10.475,"5min":6.7629999999999999,"15min":6.7629999999999999},"max":{"1min":17.245000000000001,"5min":211.22999999999999,"15min":211.22999999999999},"last":15.929},{"interface":"front","average":{"1min":12.872999999999999,"5min":20.587,"15min":24.021000000000001},"min":{"1min":8.5760000000000005,"5min":7.7709999999999999,"15min":6.8570000000000002},"max":{"1min":18.359999999999999,"5min":223.191,"15min":286.69499999999999},"last":10.608000000000001}]}]},{"osd":4,"up_from":20,"seq":85899346268,"num_pgs":12,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":63060,"kb_used_data":5704,"kb_used_omap":1,"kb_used_meta":57342,"kb_avail":94308780,"statfs":{"total":96636764160,"available":96572190720,"internally_reserved":0,"allocated":5840896,"data_stored":2740727,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1594,"internal_metadata":58718662},"hb_peers":[0,1,2,3,5,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Mon Jun 3 09:05:34 2024","interfaces":[{"interface":"back","average":{"1min":12.416,"5min":1099.4649999999999,"15min":382.15199999999999},"min":{"1min":4.5279999999999996,"5min":4.5279999999999996,"15min":4.3220000000000001},"max":{"1min":123.878,"5min":21726.743999999999,"15min":21726.743999999999},"last":6.7329999999999997},{"interface":"front","average":{"1min":16.462,"5min":1108.894,"15min":387.70299999999997},"min":{"1min":6.4089999999999998,"5min":5.8380000000000001,"15min":5.4909999999999997},"max":{"1min":130.185,"5min":21727.993999999999,"15min":21727.993999999999},"last":9.3309999999999995}]},{"osd":1,"last update":"Mon Jun 3 09:05:34 2024","interfaces":[{"interface":"back","average":{"1min":14.727,"5min":56.755000000000003,"15min":341.09399999999999},"min":{"1min":4.6529999999999996,"5min":4.1970000000000001,"15min":4.1970000000000001},"max":{"1min":126.899,"5min":3346.4389999999999,"15min":22256.098000000002},"last":15.904},{"interface":"front","average":{"1min":13.472,"5min":57.503,"15min":340.303},"min":{"1min":5.0590000000000002,"5min":5.0590000000000002,"15min":4.2640000000000002},"max":{"1min":126.096,"5min":3347.181,"15min":22247.317999999999},"last":8.0199999999999996}]},{"osd":2,"last update":"Mon Jun 3 09:05:34 2024","interfaces":[{"interface":"back","average":{"1min":26.088999999999999,"5min":23.866,"15min":20.620999999999999},"min":{"1min":7.0869999999999997,"5min":6.8239999999999998,"15min":6.3150000000000004},"max":{"1min":337.60399999999998,"5min":393.58600000000001,"15min":393.58600000000001},"last":9.9429999999999996},{"interface":"front","average":{"1min":15.800000000000001,"5min":22.584,"15min":23.245999999999999},"min":{"1min":6.0629999999999997,"5min":6.0629999999999997,"15min":6.0629999999999997},"max":{"1min":127.913,"5min":394.471,"15min":394.471},"last":6.4889999999999999}]},{"osd":3,"last update":"Mon Jun 3 09:05:34 2024","interfaces":[{"interface":"back","average":{"1min":17.544,"5min":19.143999999999998,"15min":18.742000000000001},"min":{"1min":5.8719999999999999,"5min":5.4100000000000001,"15min":5.4000000000000004},"max":{"1min":127.43600000000001,"5min":322.63,"15min":322.63},"last":11.32},{"interface":"front","average":{"1min":21.327000000000002,"5min":23.559000000000001,"15min":23.797000000000001},"min":{"1min":10.023,"5min":7.484,"15min":7.484},"max":{"1min":204.517,"5min":392.67500000000001,"15min":392.67500000000001},"last":13.484999999999999}]},{"osd":5,"last update":"Mon Jun 3 09:05:34 2024","interfaces":[{"interface":"back","average":{"1min":24.274999999999999,"5min":21.274999999999999,"15min":20.988},"min":{"1min":6.7489999999999997,"5min":6.7489999999999997,"15min":6.7489999999999997},"max":{"1min":268.10199999999998,"5min":344.23399999999998,"15min":344.23399999999998},"last":8.8209999999999997},{"interface":"front","average":{"1min":26.361999999999998,"5min":25.218,"15min":26.084},"min":{"1min":12.228,"5min":8.5180000000000007,"15min":8.5180000000000007},"max":{"1min":264.75,"5min":345.18700000000001,"15min":345.18700000000001},"last":12.875}]},{"osd":6,"last update":"Mon Jun 3 09:04:32 2024","interfaces":[{"interface":"back","average":{"1min":15.548,"5min":32.957999999999998,"15min":152.738},"min":{"1min":9.141,"5min":9.141,"15min":8.2319999999999993},"max":{"1min":52.57,"5min":395.238,"15min":12496.335999999999},"last":208.90000000000001},{"interface":"front","average":{"1min":12.680999999999999,"5min":25.428000000000001,"15min":149.30600000000001},"min":{"1min":6.4379999999999997,"5min":5.6319999999999997,"15min":5.6319999999999997},"max":{"1min":46.283000000000001,"5min":377.74299999999999,"15min":12616.297},"last":205.88900000000001}]},{"osd":7,"last update":"Mon Jun 3 09:05:34 2024","interfaces":[{"interface":"back","average":{"1min":18.295999999999999,"5min":19.585999999999999,"15min":21.062000000000001},"min":{"1min":6.2110000000000003,"5min":6.2110000000000003,"15min":5.8010000000000002},"max":{"1min":207.25299999999999,"5min":358.94900000000001,"15min":358.94900000000001},"last":14.076000000000001},{"interface":"front","average":{"1min":23.712,"5min":24.024999999999999,"15min":24.640000000000001},"min":{"1min":11.824999999999999,"5min":7.7789999999999999,"15min":7.7789999999999999},"max":{"1min":209.32900000000001,"5min":346.25200000000001,"15min":380.06900000000002},"last":14.923}]}]},{"osd":2,"up_from":20,"seq":85899346287,"num_pgs":11,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":58964,"kb_used_data":5704,"kb_used_omap":1,"kb_used_meta":53246,"kb_avail":94312876,"statfs":{"total":96636764160,"available":96576385024,"internally_reserved":0,"allocated":5840896,"data_stored":2740727,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1594,"internal_metadata":54524358},"hb_peers":[0,1,3,4,5,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":4,"apply_latency_ms":4,"commit_latency_ns":4000000,"apply_latency_ns":4000000},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Mon Jun 3 09:05:21 2024","interfaces":[{"interface":"back","average":{"1min":16.364000000000001,"5min":947.59799999999996,"15min":326.58999999999997},"min":{"1min":6.0830000000000002,"5min":4.7050000000000001,"15min":4.1420000000000003},"max":{"1min":157.60900000000001,"5min":23465.18,"15min":23465.18},"last":6.8499999999999996},{"interface":"front","average":{"1min":12.276,"5min":945.29899999999998,"15min":326.78399999999999},"min":{"1min":5.1989999999999998,"5min":4.7439999999999998,"15min":4.6950000000000003},"max":{"1min":102.557,"5min":23465.966,"15min":23465.966},"last":5.8600000000000003}]},{"osd":1,"last update":"Mon Jun 3 09:05:21 2024","interfaces":[{"interface":"back","average":{"1min":17.803999999999998,"5min":77.658000000000001,"15min":294.96899999999999},"min":{"1min":7.2130000000000001,"5min":5.4199999999999999,"15min":5.4199999999999999},"max":{"1min":159.624,"5min":4220.5290000000005,"15min":18954.987000000001},"last":10.423999999999999},{"interface":"front","average":{"1min":21.542999999999999,"5min":83.528999999999996,"15min":303.11700000000002},"min":{"1min":8.8200000000000003,"5min":7.5949999999999998,"15min":7.5949999999999998},"max":{"1min":170.74299999999999,"5min":4325.9489999999996,"15min":19143.834999999999},"last":11.566000000000001}]},{"osd":3,"last update":"Mon Jun 3 09:05:21 2024","interfaces":[{"interface":"back","average":{"1min":19.164999999999999,"5min":13.284000000000001,"15min":28.751999999999999},"min":{"1min":7.3710000000000004,"5min":5.9290000000000003,"15min":5.2450000000000001},"max":{"1min":118.913,"5min":118.913,"15min":2594.8150000000001},"last":8.3040000000000003},{"interface":"front","average":{"1min":22.094000000000001,"5min":17.244,"15min":32.173999999999999},"min":{"1min":9.516,"5min":8.0760000000000005,"15min":6.0780000000000003},"max":{"1min":156.34700000000001,"5min":156.34700000000001,"15min":2706.5120000000002},"last":13.273999999999999}]},{"osd":4,"last update":"Mon Jun 3 09:05:57 2024","interfaces":[{"interface":"back","average":{"1min":10.573,"5min":1266.894,"15min":438.154},"min":{"1min":8.6379999999999999,"5min":7.8789999999999996,"15min":6.2039999999999997},"max":{"1min":15.105,"5min":26660.861000000001,"15min":26660.861000000001},"last":9.673},{"interface":"front","average":{"1min":12.407999999999999,"5min":1266.7460000000001,"15min":435.75400000000002},"min":{"1min":10.266999999999999,"5min":10.266999999999999,"15min":6.2720000000000002},"max":{"1min":15.228,"5min":26616.243999999999,"15min":26616.243999999999},"last":14.26}]},{"osd":5,"last update":"Mon Jun 3 09:05:21 2024","interfaces":[{"interface":"back","average":{"1min":25.632000000000001,"5min":20.437999999999999,"15min":21.472999999999999},"min":{"1min":9.2159999999999993,"5min":9.2159999999999993,"15min":6.2050000000000001},"max":{"1min":252.31100000000001,"5min":252.31100000000001,"15min":252.31100000000001},"last":14.960000000000001},{"interface":"front","average":{"1min":24.824000000000002,"5min":18.521000000000001,"15min":21.027000000000001},"min":{"1min":8.0449999999999999,"5min":7.4850000000000003,"15min":6.8810000000000002},"max":{"1min":258.31700000000001,"5min":258.31700000000001,"15min":258.31700000000001},"last":12.413}]},{"osd":6,"last update":"Mon Jun 3 09:04:28 2024","interfaces":[{"interface":"back","average":{"1min":14.063000000000001,"5min":24.850000000000001,"15min":106.755},"min":{"1min":9.0820000000000007,"5min":8.9649999999999999,"15min":8.9649999999999999},"max":{"1min":24.666,"5min":188.06700000000001,"15min":9521.0010000000002},"last":183.60900000000001},{"interface":"front","average":{"1min":13.933999999999999,"5min":21.081,"15min":106.142},"min":{"1min":5.8719999999999999,"5min":5.8719999999999999,"15min":5.8719999999999999},"max":{"1min":117.551,"5min":193.887,"15min":9585.6620000000003},"last":184.20599999999999}]},{"osd":7,"last update":"Mon Jun 3 09:05:21 2024","interfaces":[{"interface":"back","average":{"1min":25.097000000000001,"5min":22.579999999999998,"15min":24.388000000000002},"min":{"1min":7.8959999999999999,"5min":6.6130000000000004,"15min":6.6130000000000004},"max":{"1min":242.56800000000001,"5min":242.56800000000001,"15min":242.56800000000001},"last":15.646000000000001},{"interface":"front","average":{"1min":24.414999999999999,"5min":23.995999999999999,"15min":26.614999999999998},"min":{"1min":12.43,"5min":9.0660000000000007,"15min":9.0660000000000007},"max":{"1min":181.56399999999999,"5min":182.5,"15min":214.33799999999999},"last":16.282}]}]},{"osd":3,"up_from":14,"seq":60129542502,"num_pgs":11,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":58964,"kb_used_data":5704,"kb_used_omap":1,"kb_used_meta":53246,"kb_avail":94312876,"statfs":{"total":96636764160,"available":96576385024,"internally_reserved":0,"allocated":5840896,"data_stored":2740727,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1592,"internal_metadata":54524360},"hb_peers":[0,1,2,4,5,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Mon Jun 3 09:05:25 2024","interfaces":[{"interface":"back","average":{"1min":8.4809999999999999,"5min":1064.2270000000001,"15min":367.88600000000002},"min":{"1min":6.0529999999999999,"5min":5.173,"15min":4.6929999999999996},"max":{"1min":12.010999999999999,"5min":23275.167000000001,"15min":23275.167000000001},"last":8.7289999999999992},{"interface":"front","average":{"1min":13.885,"5min":1070.3710000000001,"15min":373.53500000000003},"min":{"1min":9.6679999999999993,"5min":7.2089999999999996,"15min":5.9880000000000004},"max":{"1min":32.426000000000002,"5min":23278.536,"15min":23278.536},"last":12.621}]},{"osd":1,"last update":"Mon Jun 3 09:05:25 2024","interfaces":[{"interface":"back","average":{"1min":11.324,"5min":19.920000000000002,"15min":390.07900000000001},"min":{"1min":6.6379999999999999,"5min":4.8200000000000003,"15min":4.8200000000000003},"max":{"1min":26.449000000000002,"5min":250.72300000000001,"15min":21658.205999999998},"last":9.7889999999999997},{"interface":"front","average":{"1min":9.468,"5min":20.966999999999999,"15min":381.18799999999999},"min":{"1min":5.4969999999999999,"5min":5.2779999999999996,"15min":5.2779999999999996},"max":{"1min":25.643000000000001,"5min":335.33699999999999,"15min":21307.069},"last":7.8179999999999996}]},{"osd":2,"last update":"Mon Jun 3 09:05:25 2024","interfaces":[{"interface":"back","average":{"1min":13.215,"5min":26.649000000000001,"15min":23.783999999999999},"min":{"1min":7.8710000000000004,"5min":6.4900000000000002,"15min":5.875},"max":{"1min":34.322000000000003,"5min":377.803,"15min":377.803},"last":14.018000000000001},{"interface":"front","average":{"1min":13.105,"5min":22.292999999999999,"15min":20.573},"min":{"1min":7.327,"5min":6.125,"15min":5.9020000000000001},"max":{"1min":32.756999999999998,"5min":371.601,"15min":371.601},"last":6.9630000000000001}]},{"osd":4,"last update":"Mon Jun 3 09:05:27 2024","interfaces":[{"interface":"back","average":{"1min":10.444000000000001,"5min":1211.4970000000001,"15min":1639.1579999999999},"min":{"1min":6.4429999999999996,"5min":6.2560000000000002,"15min":6.0309999999999997},"max":{"1min":21.263000000000002,"5min":30144.939999999999,"15min":62679.622000000003},"last":5.6539999999999999},{"interface":"front","average":{"1min":11.363,"5min":1219.6400000000001,"15min":1644.127},"min":{"1min":7.8109999999999999,"5min":7.8109999999999999,"15min":6.2439999999999998},"max":{"1min":20.108000000000001,"5min":30180.689999999999,"15min":62681.021999999997},"last":11.103999999999999}]},{"osd":5,"last update":"Mon Jun 3 09:06:07 2024","interfaces":[{"interface":"back","average":{"1min":12.802,"5min":19.640999999999998,"15min":22.318000000000001},"min":{"1min":9.0890000000000004,"5min":6.8810000000000002,"15min":6.7690000000000001},"max":{"1min":16.745999999999999,"5min":249.88,"15min":327.34899999999999},"last":13.329000000000001},{"interface":"front","average":{"1min":15.002000000000001,"5min":24.003,"15min":26.649999999999999},"min":{"1min":11.984,"5min":9.9149999999999991,"15min":8.9879999999999995},"max":{"1min":18.407,"5min":251.47800000000001,"15min":331.86799999999999},"last":14.683999999999999}]},{"osd":6,"last update":"Mon Jun 3 09:04:04 2024","interfaces":[{"interface":"back","average":{"1min":22.815999999999999,"5min":31.513999999999999,"15min":181.74299999999999},"min":{"1min":9.4809999999999999,"5min":7.2240000000000002,"15min":7.2240000000000002},"max":{"1min":160.56999999999999,"5min":380.613,"15min":11893.74},"last":19.359999999999999},{"interface":"front","average":{"1min":25.359999999999999,"5min":30.981000000000002,"15min":179.81399999999999},"min":{"1min":9.8870000000000005,"5min":7.3360000000000003,"15min":6.0899999999999999},"max":{"1min":193.11099999999999,"5min":379.80700000000002,"15min":11893.069},"last":8.202}]},{"osd":7,"last update":"Mon Jun 3 09:05:25 2024","interfaces":[{"interface":"back","average":{"1min":12.177,"5min":21.126000000000001,"15min":22.972000000000001},"min":{"1min":8.6769999999999996,"5min":7.4279999999999999,"15min":7.4279999999999999},"max":{"1min":17.376999999999999,"5min":164.27500000000001,"15min":178.76300000000001},"last":9.3070000000000004},{"interface":"front","average":{"1min":15.675000000000001,"5min":22.722000000000001,"15min":25.238},"min":{"1min":11.233000000000001,"5min":10.555999999999999,"15min":7.6440000000000001},"max":{"1min":29.280000000000001,"5min":148.13800000000001,"15min":417.22800000000001},"last":10.32}]}]},{"osd":1,"up_from":14,"seq":60129542500,"num_pgs":6,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":59356,"kb_used_data":6096,"kb_used_omap":1,"kb_used_meta":53246,"kb_avail":94312484,"statfs":{"total":96636764160,"available":96575983616,"internally_reserved":0,"allocated":6242304,"data_stored":3138567,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":54524357},"hb_peers":[0,2,3,4,5,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Mon Jun 3 09:06:07 2024","interfaces":[{"interface":"back","average":{"1min":7.7530000000000001,"5min":185.66999999999999,"15min":77.347999999999999},"min":{"1min":5.0229999999999997,"5min":4.9279999999999999,"15min":4.5449999999999999},"max":{"1min":11.896000000000001,"5min":7470.616,"15min":7470.616},"last":7.2610000000000001},{"interface":"front","average":{"1min":9.9220000000000006,"5min":187.14599999999999,"15min":76.325999999999993},"min":{"1min":5.3159999999999998,"5min":5.3159999999999998,"15min":5.3159999999999998},"max":{"1min":18.338000000000001,"5min":7477.0829999999996,"15min":7477.0829999999996},"last":13.295999999999999}]},{"osd":2,"last update":"Mon Jun 3 09:06:20 2024","interfaces":[{"interface":"back","average":{"1min":10.340999999999999,"5min":17.369,"15min":23.670999999999999},"min":{"1min":6.9740000000000002,"5min":5.8650000000000002,"15min":5.8650000000000002},"max":{"1min":15.452,"5min":228,"15min":657.38},"last":9.1630000000000003},{"interface":"front","average":{"1min":13.387,"5min":23.355,"15min":31.524000000000001},"min":{"1min":9.077,"5min":9.077,"15min":7.2160000000000002},"max":{"1min":19.701000000000001,"5min":229.465,"15min":700.06500000000005},"last":14.991}]},{"osd":3,"last update":"Mon Jun 3 09:06:04 2024","interfaces":[{"interface":"back","average":{"1min":8.6110000000000007,"5min":15.733000000000001,"15min":26.646999999999998},"min":{"1min":5.827,"5min":5.4640000000000004,"15min":5.4640000000000004},"max":{"1min":13.105,"5min":121.619,"15min":1676.009},"last":6.2830000000000004},{"interface":"front","average":{"1min":9.5850000000000009,"5min":16.161999999999999,"15min":29.065999999999999},"min":{"1min":6.3630000000000004,"5min":5.7699999999999996,"15min":5.7699999999999996},"max":{"1min":15.166,"5min":132.036,"15min":1766.5840000000001},"last":11.605}]},{"osd":4,"last update":"Mon Jun 3 09:05:31 2024","interfaces":[{"interface":"back","average":{"1min":10.996,"5min":954.46500000000003,"15min":1281.6859999999999},"min":{"1min":7.702,"5min":6.1760000000000002,"15min":6.0970000000000004},"max":{"1min":21.538,"5min":28044.862000000001,"15min":59438.017999999996},"last":8.6010000000000009},{"interface":"front","average":{"1min":13.646000000000001,"5min":961.24599999999998,"15min":1285.26},"min":{"1min":9.7200000000000006,"5min":9.3940000000000001,"15min":6.4640000000000004},"max":{"1min":23.277999999999999,"5min":28045.624,"15min":59439.283000000003},"last":10.781000000000001}]},{"osd":5,"last update":"Mon Jun 3 09:06:04 2024","interfaces":[{"interface":"back","average":{"1min":12.478999999999999,"5min":20.228000000000002,"15min":25.997},"min":{"1min":8.6430000000000007,"5min":7.9720000000000004,"15min":7.234},"max":{"1min":17.442,"5min":139.43199999999999,"15min":701.22000000000003},"last":10.263999999999999},{"interface":"front","average":{"1min":13.507999999999999,"5min":25.760999999999999,"15min":27.553999999999998},"min":{"1min":10.042999999999999,"5min":7.258,"15min":6.242},"max":{"1min":20.611000000000001,"5min":278.63,"15min":598.35000000000002},"last":15.890000000000001}]},{"osd":6,"last update":"Mon Jun 3 09:03:54 2024","interfaces":[{"interface":"back","average":{"1min":28.524000000000001,"5min":20.654,"15min":23.878},"min":{"1min":6.0309999999999997,"5min":6.0309999999999997,"15min":6.0309999999999997},"max":{"1min":167.221,"5min":167.221,"15min":704.245},"last":18.721},{"interface":"front","average":{"1min":27.606000000000002,"5min":30.131,"15min":28.628},"min":{"1min":9.8650000000000002,"5min":9.7200000000000006,"15min":8.3580000000000005},"max":{"1min":123.938,"5min":358.23000000000002,"15min":634.05499999999995},"last":173.113}]},{"osd":7,"last update":"Mon Jun 3 09:06:20 2024","interfaces":[{"interface":"back","average":{"1min":14.66,"5min":23.234999999999999,"15min":31.097000000000001},"min":{"1min":11.250999999999999,"5min":8.0559999999999992,"15min":8.0559999999999992},"max":{"1min":21.507999999999999,"5min":270.95100000000002,"15min":703.07799999999997},"last":14.182},{"interface":"front","average":{"1min":15.548,"5min":28.760000000000002,"15min":28.925000000000001},"min":{"1min":12.759,"5min":8.2089999999999996,"15min":7.7599999999999998},"max":{"1min":22.408000000000001,"5min":360.99900000000002,"15min":608.44100000000003},"last":16.588000000000001}]}]},{"osd":0,"up_from":14,"seq":60129542502,"num_pgs":10,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":81108,"kb_used_data":5704,"kb_used_omap":1,"kb_used_meta":75390,"kb_avail":94290732,"statfs":{"total":96636764160,"available":96553709568,"internally_reserved":0,"allocated":5840896,"data_stored":2740727,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1593,"internal_metadata":77199815},"hb_peers":[1,2,3,4,5,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[{"osd":1,"last update":"Mon Jun 3 09:06:04 2024","interfaces":[{"interface":"back","average":{"1min":6.9960000000000004,"5min":56.997,"15min":339.29700000000003},"min":{"1min":4.4530000000000003,"5min":4.4530000000000003,"15min":4.4100000000000001},"max":{"1min":11.720000000000001,"5min":2414.4749999999999,"15min":20691.453000000001},"last":4.6040000000000001},{"interface":"front","average":{"1min":9.548,"5min":57.258000000000003,"15min":337.791},"min":{"1min":5.2300000000000004,"5min":5.2300000000000004,"15min":4.6050000000000004},"max":{"1min":15.337999999999999,"5min":2412.9549999999999,"15min":20640.792000000001},"last":7.7910000000000004}]},{"osd":2,"last update":"Mon Jun 3 09:06:22 2024","interfaces":[{"interface":"back","average":{"1min":11.154,"5min":26.515999999999998,"15min":22.882000000000001},"min":{"1min":5.7869999999999999,"5min":5.766,"15min":5.766},"max":{"1min":72.269999999999996,"5min":627.40700000000004,"15min":627.40700000000004},"last":8.6430000000000007},{"interface":"front","average":{"1min":14.353999999999999,"5min":33.304000000000002,"15min":29.885000000000002},"min":{"1min":8.2149999999999999,"5min":7.8630000000000004,"15min":7.4089999999999998},"max":{"1min":73.418000000000006,"5min":635.62599999999998,"15min":635.62599999999998},"last":10.872999999999999}]},{"osd":3,"last update":"Mon Jun 3 09:06:04 2024","interfaces":[{"interface":"back","average":{"1min":12.105,"5min":26.925000000000001,"15min":26.584},"min":{"1min":7.8410000000000002,"5min":7.1280000000000001,"15min":6.1909999999999998},"max":{"1min":18.425999999999998,"5min":389.72699999999998,"15min":389.72699999999998},"last":9.484},{"interface":"front","average":{"1min":13.794,"5min":29.771999999999998,"15min":28.815000000000001},"min":{"1min":10.084,"5min":9.4920000000000009,"15min":6.1849999999999996},"max":{"1min":17.373999999999999,"5min":390.23200000000003,"15min":494.92899999999997},"last":15.166}]},{"osd":4,"last update":"Mon Jun 3 09:05:27 2024","interfaces":[{"interface":"back","average":{"1min":13.478,"5min":1032.126,"15min":1389.729},"min":{"1min":6.5119999999999996,"5min":6.5119999999999996,"15min":6.5119999999999996},"max":{"1min":55.354999999999997,"5min":25555.212,"15min":60989.964},"last":11.698},{"interface":"front","average":{"1min":16.204999999999998,"5min":1030.8389999999999,"15min":1387.778},"min":{"1min":10.282999999999999,"5min":7.1429999999999998,"15min":7.1429999999999998},"max":{"1min":49.945999999999998,"5min":25556.159,"15min":61000.017},"last":12.278}]},{"osd":5,"last update":"Mon Jun 3 09:06:17 2024","interfaces":[{"interface":"back","average":{"1min":9.7040000000000006,"5min":26.533999999999999,"15min":28.683},"min":{"1min":6.8129999999999997,"5min":6.8129999999999997,"15min":6.8129999999999997},"max":{"1min":14.942,"5min":455.78699999999998,"15min":605.05899999999997},"last":7.2480000000000002},{"interface":"front","average":{"1min":14.279,"5min":28.398,"15min":26.192},"min":{"1min":11.696,"5min":6.4420000000000002,"15min":6.2030000000000003},"max":{"1min":17.879000000000001,"5min":391.31200000000001,"15min":541.11000000000001},"last":14.297000000000001}]},{"osd":6,"last update":"Mon Jun 3 09:04:30 2024","interfaces":[{"interface":"back","average":{"1min":52.496000000000002,"5min":33.139000000000003,"15min":160.83199999999999},"min":{"1min":7.5289999999999999,"5min":7.1040000000000001,"15min":6.3739999999999997},"max":{"1min":457.91699999999997,"5min":457.91699999999997,"15min":11115.258},"last":18.959},{"interface":"front","average":{"1min":49.670000000000002,"5min":33.704999999999998,"15min":162.791},"min":{"1min":12.247999999999999,"5min":11.019,"15min":9.9760000000000009},"max":{"1min":407.05099999999999,"5min":407.05099999999999,"15min":11092.194},"last":17.707000000000001}]},{"osd":7,"last update":"Mon Jun 3 09:06:22 2024","interfaces":[{"interface":"back","average":{"1min":11.625999999999999,"5min":26.375,"15min":28.972999999999999},"min":{"1min":6.4480000000000004,"5min":6.4480000000000004,"15min":6.4480000000000004},"max":{"1min":15.352,"5min":461.41699999999997,"15min":598.33699999999999},"last":10.250999999999999},{"interface":"front","average":{"1min":14.013,"5min":29.696000000000002,"15min":33.630000000000003},"min":{"1min":10.938000000000001,"5min":10.58,"15min":10.263999999999999},"max":{"1min":16.603000000000002,"5min":463.10599999999999,"15min":592.80499999999995},"last":13.254}]}]}],"pool_statfs":[{"poolid":1,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":401408,"data_stored":397840,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":5,"total":0,"available":0,"internally_reserved":0,"allocated":401408,"data_stored":397840,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":4,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":5,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":6,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":17,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":17,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":17,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":17,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":17,"osd":4,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":17,"osd":5,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":5,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":17,"osd":6,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":5,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":17,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0}]}} 2024-06-03T09:06:27.039 INFO:tasks.ceph.ceph_manager.ceph:PG 17.c is not active+clean 2024-06-03T09:06:27.039 INFO:tasks.ceph.ceph_manager.ceph:{'pgid': '17.c', 'version': "0'0", 'reported_seq': 953, 'reported_epoch': 1009, 'state': 'active+undersized', 'last_fresh': '2024-06-03T09:06:13.668156+0000', 'last_change': '2024-06-03T09:06:03.440074+0000', 'last_active': '2024-06-03T09:06:13.668156+0000', 'last_peered': '2024-06-03T09:06:13.668156+0000', 'last_clean': '2024-06-03T09:05:02.105638+0000', 'last_became_active': '2024-06-03T09:06:03.440074+0000', 'last_became_peered': '2024-06-03T09:06:03.440074+0000', 'last_unstale': '2024-06-03T09:06:13.668156+0000', 'last_undegraded': '2024-06-03T09:06:13.668156+0000', 'last_fullsized': '2024-06-03T09:06:03.377982+0000', 'mapping_epoch': 999, 'log_start': "0'0", 'ondisk_log_start': "0'0", 'created': 28, 'last_epoch_clean': 255, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2024-06-03T08:37:55.614497+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2024-06-03T08:37:55.614497+0000', 'last_clean_scrub_stamp': '2024-06-03T08:37:55.614497+0000', 'objects_scrubbed': 0, 'log_size': 0, 'log_dups_size': 0, 'ondisk_log_size': 0, 'stats_invalid': False, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': 'periodic scrub scheduled @ 2024-06-04T10:34:20.808557+0000', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 0, 'num_objects': 0, 'num_object_clones': 0, 'num_object_copies': 0, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 0, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 0, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 0, 'num_write_kb': 0, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 0, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [3], 'acting': [3], 'avail_no_missing': ['3'], 'object_location_counts': [], 'blocked_by': [], 'up_primary': 3, 'acting_primary': 3, 'purged_snaps': []} 2024-06-03T09:06:27.039 INFO:tasks.ceph.ceph_manager.ceph:PG 17.f is not active+clean 2024-06-03T09:06:27.039 INFO:tasks.ceph.ceph_manager.ceph:{'pgid': '17.f', 'version': "0'0", 'reported_seq': 148, 'reported_epoch': 1009, 'state': 'active+undersized', 'last_fresh': '2024-06-03T09:06:14.450543+0000', 'last_change': '2024-06-03T09:06:03.420285+0000', 'last_active': '2024-06-03T09:06:14.450543+0000', 'last_peered': '2024-06-03T09:06:14.450543+0000', 'last_clean': '2024-06-03T08:37:55.614497+0000', 'last_became_active': '2024-06-03T09:06:03.420285+0000', 'last_became_peered': '2024-06-03T09:06:03.420285+0000', 'last_unstale': '2024-06-03T09:06:14.450543+0000', 'last_undegraded': '2024-06-03T09:06:14.450543+0000', 'last_fullsized': '2024-06-03T09:06:03.380616+0000', 'mapping_epoch': 999, 'log_start': "0'0", 'ondisk_log_start': "0'0", 'created': 28, 'last_epoch_clean': 255, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2024-06-03T08:37:55.614497+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2024-06-03T08:37:55.614497+0000', 'last_clean_scrub_stamp': '2024-06-03T08:37:55.614497+0000', 'objects_scrubbed': 0, 'log_size': 0, 'log_dups_size': 0, 'ondisk_log_size': 0, 'stats_invalid': False, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': 'periodic scrub scheduled @ 2024-06-04T15:39:41.187731+0000', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 0, 'num_objects': 0, 'num_object_clones': 0, 'num_object_copies': 0, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 0, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 0, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 0, 'num_write_kb': 0, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 0, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [4], 'acting': [4], 'avail_no_missing': ['4'], 'object_location_counts': [], 'blocked_by': [], 'up_primary': 4, 'acting_primary': 4, 'purged_snaps': []} 2024-06-03T09:06:27.040 INFO:tasks.ceph.ceph_manager.ceph:PG 17.e is not active+clean 2024-06-03T09:06:27.040 INFO:tasks.ceph.ceph_manager.ceph:{'pgid': '17.e', 'version': "0'0", 'reported_seq': 149, 'reported_epoch': 1009, 'state': 'active+undersized', 'last_fresh': '2024-06-03T09:06:13.726992+0000', 'last_change': '2024-06-03T09:06:03.412887+0000', 'last_active': '2024-06-03T09:06:13.726992+0000', 'last_peered': '2024-06-03T09:06:13.726992+0000', 'last_clean': '2024-06-03T08:37:55.614497+0000', 'last_became_active': '2024-06-03T09:06:03.412887+0000', 'last_became_peered': '2024-06-03T09:06:03.412887+0000', 'last_unstale': '2024-06-03T09:06:13.726992+0000', 'last_undegraded': '2024-06-03T09:06:13.726992+0000', 'last_fullsized': '2024-06-03T09:06:03.332482+0000', 'mapping_epoch': 999, 'log_start': "0'0", 'ondisk_log_start': "0'0", 'created': 28, 'last_epoch_clean': 255, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2024-06-03T08:37:55.614497+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2024-06-03T08:37:55.614497+0000', 'last_clean_scrub_stamp': '2024-06-03T08:37:55.614497+0000', 'objects_scrubbed': 0, 'log_size': 0, 'log_dups_size': 0, 'ondisk_log_size': 0, 'stats_invalid': False, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': 'periodic scrub scheduled @ 2024-06-04T12:23:22.511601+0000', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 0, 'num_objects': 0, 'num_object_clones': 0, 'num_object_copies': 0, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 0, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 0, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 0, 'num_write_kb': 0, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 0, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [5], 'acting': [5], 'avail_no_missing': ['5'], 'object_location_counts': [], 'blocked_by': [], 'up_primary': 5, 'acting_primary': 5, 'purged_snaps': []} 2024-06-03T09:06:27.040 INFO:tasks.ceph.ceph_manager.ceph:PG 17.8 is not active+clean 2024-06-03T09:06:27.040 INFO:tasks.ceph.ceph_manager.ceph:{'pgid': '17.8', 'version': "0'0", 'reported_seq': 961, 'reported_epoch': 1009, 'state': 'active+undersized', 'last_fresh': '2024-06-03T09:06:14.430375+0000', 'last_change': '2024-06-03T09:06:03.388786+0000', 'last_active': '2024-06-03T09:06:14.430375+0000', 'last_peered': '2024-06-03T09:06:14.430375+0000', 'last_clean': '2024-06-03T09:04:43.297780+0000', 'last_became_active': '2024-06-03T09:06:03.388786+0000', 'last_became_peered': '2024-06-03T09:06:03.388786+0000', 'last_unstale': '2024-06-03T09:06:14.430375+0000', 'last_undegraded': '2024-06-03T09:06:14.430375+0000', 'last_fullsized': '2024-06-03T09:06:03.331754+0000', 'mapping_epoch': 999, 'log_start': "0'0", 'ondisk_log_start': "0'0", 'created': 28, 'last_epoch_clean': 255, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2024-06-03T08:37:55.614497+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2024-06-03T08:37:55.614497+0000', 'last_clean_scrub_stamp': '2024-06-03T08:37:55.614497+0000', 'objects_scrubbed': 0, 'log_size': 0, 'log_dups_size': 0, 'ondisk_log_size': 0, 'stats_invalid': False, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': 'periodic scrub scheduled @ 2024-06-04T09:18:44.765547+0000', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 0, 'num_objects': 0, 'num_object_clones': 0, 'num_object_copies': 0, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 0, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 0, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 0, 'num_write_kb': 0, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 0, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [0], 'acting': [0], 'avail_no_missing': ['0'], 'object_location_counts': [], 'blocked_by': [], 'up_primary': 0, 'acting_primary': 0, 'purged_snaps': []} 2024-06-03T09:06:27.040 INFO:tasks.ceph.ceph_manager.ceph:PG 17.b is not active+clean 2024-06-03T09:06:27.040 INFO:tasks.ceph.ceph_manager.ceph:{'pgid': '17.b', 'version': "0'0", 'reported_seq': 149, 'reported_epoch': 1009, 'state': 'active+undersized', 'last_fresh': '2024-06-03T09:06:14.433735+0000', 'last_change': '2024-06-03T09:06:03.451203+0000', 'last_active': '2024-06-03T09:06:14.433735+0000', 'last_peered': '2024-06-03T09:06:14.433735+0000', 'last_clean': '2024-06-03T08:37:55.614497+0000', 'last_became_active': '2024-06-03T09:06:03.451203+0000', 'last_became_peered': '2024-06-03T09:06:03.451203+0000', 'last_unstale': '2024-06-03T09:06:14.433735+0000', 'last_undegraded': '2024-06-03T09:06:14.433735+0000', 'last_fullsized': '2024-06-03T09:06:03.375523+0000', 'mapping_epoch': 999, 'log_start': "0'0", 'ondisk_log_start': "0'0", 'created': 28, 'last_epoch_clean': 255, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2024-06-03T08:37:55.614497+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2024-06-03T08:37:55.614497+0000', 'last_clean_scrub_stamp': '2024-06-03T08:37:55.614497+0000', 'objects_scrubbed': 0, 'log_size': 0, 'log_dups_size': 0, 'ondisk_log_size': 0, 'stats_invalid': False, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': 'periodic scrub scheduled @ 2024-06-04T10:16:19.813495+0000', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 0, 'num_objects': 0, 'num_object_clones': 0, 'num_object_copies': 0, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 0, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 0, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 0, 'num_write_kb': 0, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 0, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [2], 'acting': [2], 'avail_no_missing': ['2'], 'object_location_counts': [], 'blocked_by': [], 'up_primary': 2, 'acting_primary': 2, 'purged_snaps': []} 2024-06-03T09:06:27.040 INFO:tasks.ceph.ceph_manager.ceph:PG 17.4 is not active+clean 2024-06-03T09:06:27.040 INFO:tasks.ceph.ceph_manager.ceph:{'pgid': '17.4', 'version': "0'0", 'reported_seq': 149, 'reported_epoch': 1009, 'state': 'active+undersized', 'last_fresh': '2024-06-03T09:06:13.941522+0000', 'last_change': '2024-06-03T09:06:03.441889+0000', 'last_active': '2024-06-03T09:06:13.941522+0000', 'last_peered': '2024-06-03T09:06:13.941522+0000', 'last_clean': '2024-06-03T08:37:55.614497+0000', 'last_became_active': '2024-06-03T09:06:03.441889+0000', 'last_became_peered': '2024-06-03T09:06:03.441889+0000', 'last_unstale': '2024-06-03T09:06:13.941522+0000', 'last_undegraded': '2024-06-03T09:06:13.941522+0000', 'last_fullsized': '2024-06-03T09:06:03.359582+0000', 'mapping_epoch': 999, 'log_start': "0'0", 'ondisk_log_start': "0'0", 'created': 28, 'last_epoch_clean': 255, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2024-06-03T08:37:55.614497+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2024-06-03T08:37:55.614497+0000', 'last_clean_scrub_stamp': '2024-06-03T08:37:55.614497+0000', 'objects_scrubbed': 0, 'log_size': 0, 'log_dups_size': 0, 'ondisk_log_size': 0, 'stats_invalid': False, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': 'periodic scrub scheduled @ 2024-06-04T12:49:22.907141+0000', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 0, 'num_objects': 0, 'num_object_clones': 0, 'num_object_copies': 0, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 0, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 0, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 0, 'num_write_kb': 0, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 0, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [7], 'acting': [7], 'avail_no_missing': ['7'], 'object_location_counts': [], 'blocked_by': [], 'up_primary': 7, 'acting_primary': 7, 'purged_snaps': []} 2024-06-03T09:06:27.040 INFO:tasks.ceph.ceph_manager.ceph:PG 17.6 is not active+clean 2024-06-03T09:06:27.040 INFO:tasks.ceph.ceph_manager.ceph:{'pgid': '17.6', 'version': "0'0", 'reported_seq': 960, 'reported_epoch': 1009, 'state': 'active+undersized', 'last_fresh': '2024-06-03T09:06:14.453110+0000', 'last_change': '2024-06-03T09:06:03.428677+0000', 'last_active': '2024-06-03T09:06:14.453110+0000', 'last_peered': '2024-06-03T09:06:14.453110+0000', 'last_clean': '2024-06-03T09:04:43.312352+0000', 'last_became_active': '2024-06-03T09:06:03.428677+0000', 'last_became_peered': '2024-06-03T09:06:03.428677+0000', 'last_unstale': '2024-06-03T09:06:14.453110+0000', 'last_undegraded': '2024-06-03T09:06:14.453110+0000', 'last_fullsized': '2024-06-03T09:06:03.359909+0000', 'mapping_epoch': 999, 'log_start': "0'0", 'ondisk_log_start': "0'0", 'created': 28, 'last_epoch_clean': 255, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2024-06-03T08:37:55.614497+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2024-06-03T08:37:55.614497+0000', 'last_clean_scrub_stamp': '2024-06-03T08:37:55.614497+0000', 'objects_scrubbed': 0, 'log_size': 0, 'log_dups_size': 0, 'ondisk_log_size': 0, 'stats_invalid': False, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': 'periodic scrub scheduled @ 2024-06-04T15:38:02.180241+0000', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 0, 'num_objects': 0, 'num_object_clones': 0, 'num_object_copies': 0, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 0, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 0, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 0, 'num_write_kb': 0, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 0, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [0], 'acting': [0], 'avail_no_missing': ['0'], 'object_location_counts': [], 'blocked_by': [], 'up_primary': 0, 'acting_primary': 0, 'purged_snaps': []} 2024-06-03T09:06:27.040 INFO:tasks.ceph.ceph_manager.ceph:PG 17.16 is not active+clean 2024-06-03T09:06:27.040 INFO:tasks.ceph.ceph_manager.ceph:{'pgid': '17.16', 'version': "0'0", 'reported_seq': 972, 'reported_epoch': 1009, 'state': 'active+undersized', 'last_fresh': '2024-06-03T09:06:14.441665+0000', 'last_change': '2024-06-03T09:06:03.478615+0000', 'last_active': '2024-06-03T09:06:14.441665+0000', 'last_peered': '2024-06-03T09:06:14.441665+0000', 'last_clean': '2024-06-03T09:04:43.328415+0000', 'last_became_active': '2024-06-03T09:06:03.478615+0000', 'last_became_peered': '2024-06-03T09:06:03.478615+0000', 'last_unstale': '2024-06-03T09:06:14.441665+0000', 'last_undegraded': '2024-06-03T09:06:14.441665+0000', 'last_fullsized': '2024-06-03T09:06:03.402504+0000', 'mapping_epoch': 999, 'log_start': "0'0", 'ondisk_log_start': "0'0", 'created': 28, 'last_epoch_clean': 255, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2024-06-03T08:37:55.614497+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2024-06-03T08:37:55.614497+0000', 'last_clean_scrub_stamp': '2024-06-03T08:37:55.614497+0000', 'objects_scrubbed': 0, 'log_size': 0, 'log_dups_size': 0, 'ondisk_log_size': 0, 'stats_invalid': False, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': 'periodic scrub scheduled @ 2024-06-04T08:47:08.079233+0000', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 0, 'num_objects': 0, 'num_object_clones': 0, 'num_object_copies': 0, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 0, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 0, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 0, 'num_write_kb': 0, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 0, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [2], 'acting': [2], 'avail_no_missing': ['2'], 'object_location_counts': [], 'blocked_by': [], 'up_primary': 2, 'acting_primary': 2, 'purged_snaps': []} 2024-06-03T09:06:27.041 INFO:tasks.ceph.ceph_manager.ceph:PG 2.4 is not active+clean 2024-06-03T09:06:27.041 INFO:tasks.ceph.ceph_manager.ceph:{'pgid': '2.4', 'version': "0'0", 'reported_seq': 952, 'reported_epoch': 1010, 'state': 'active+undersized', 'last_fresh': '2024-06-03T09:06:18.331485+0000', 'last_change': '2024-06-03T09:06:03.387171+0000', 'last_active': '2024-06-03T09:06:18.331485+0000', 'last_peered': '2024-06-03T09:06:18.331485+0000', 'last_clean': '2024-06-03T09:04:43.202645+0000', 'last_became_active': '2024-06-03T09:06:03.387171+0000', 'last_became_peered': '2024-06-03T09:06:03.387171+0000', 'last_unstale': '2024-06-03T09:06:18.331485+0000', 'last_undegraded': '2024-06-03T09:06:18.331485+0000', 'last_fullsized': '2024-06-03T09:06:03.283787+0000', 'mapping_epoch': 999, 'log_start': "0'0", 'ondisk_log_start': "0'0", 'created': 23, 'last_epoch_clean': 255, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2024-06-03T08:36:28.444764+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2024-06-03T08:36:28.444764+0000', 'last_clean_scrub_stamp': '2024-06-03T08:36:28.444764+0000', 'objects_scrubbed': 0, 'log_size': 0, 'log_dups_size': 0, 'ondisk_log_size': 0, 'stats_invalid': False, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': 'periodic scrub scheduled @ 2024-06-04T20:17:43.485023+0000', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 0, 'num_objects': 0, 'num_object_clones': 0, 'num_object_copies': 0, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 0, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 0, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 0, 'num_write_kb': 0, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 0, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [5], 'acting': [5], 'avail_no_missing': ['5'], 'object_location_counts': [], 'blocked_by': [], 'up_primary': 5, 'acting_primary': 5, 'purged_snaps': []} 2024-06-03T09:06:27.041 INFO:tasks.ceph.ceph_manager.ceph:PG 2.2 is not active+clean 2024-06-03T09:06:27.041 INFO:tasks.ceph.ceph_manager.ceph:{'pgid': '2.2', 'version': "0'0", 'reported_seq': 963, 'reported_epoch': 1009, 'state': 'active+undersized', 'last_fresh': '2024-06-03T09:06:13.657892+0000', 'last_change': '2024-06-03T09:06:03.410020+0000', 'last_active': '2024-06-03T09:06:13.657892+0000', 'last_peered': '2024-06-03T09:06:13.657892+0000', 'last_clean': '2024-06-03T09:05:37.145889+0000', 'last_became_active': '2024-06-03T09:06:03.410020+0000', 'last_became_peered': '2024-06-03T09:06:03.410020+0000', 'last_unstale': '2024-06-03T09:06:13.657892+0000', 'last_undegraded': '2024-06-03T09:06:13.657892+0000', 'last_fullsized': '2024-06-03T09:06:03.346864+0000', 'mapping_epoch': 999, 'log_start': "0'0", 'ondisk_log_start': "0'0", 'created': 23, 'last_epoch_clean': 255, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2024-06-03T08:36:28.444764+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2024-06-03T08:36:28.444764+0000', 'last_clean_scrub_stamp': '2024-06-03T08:36:28.444764+0000', 'objects_scrubbed': 0, 'log_size': 0, 'log_dups_size': 0, 'ondisk_log_size': 0, 'stats_invalid': False, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': 'periodic scrub scheduled @ 2024-06-04T09:57:12.423360+0000', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 0, 'num_objects': 0, 'num_object_clones': 0, 'num_object_copies': 0, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 0, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 0, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 0, 'num_write_kb': 0, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 0, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [3], 'acting': [3], 'avail_no_missing': ['3'], 'object_location_counts': [], 'blocked_by': [], 'up_primary': 3, 'acting_primary': 3, 'purged_snaps': []} 2024-06-03T09:06:27.041 INFO:tasks.ceph.ceph_manager.ceph:PG 17.1a is not active+clean 2024-06-03T09:06:27.041 INFO:tasks.ceph.ceph_manager.ceph:{'pgid': '17.1a', 'version': "0'0", 'reported_seq': 960, 'reported_epoch': 1009, 'state': 'active+undersized', 'last_fresh': '2024-06-03T09:06:14.431224+0000', 'last_change': '2024-06-03T09:06:03.390734+0000', 'last_active': '2024-06-03T09:06:14.431224+0000', 'last_peered': '2024-06-03T09:06:14.431224+0000', 'last_clean': '2024-06-03T09:05:20.322313+0000', 'last_became_active': '2024-06-03T09:06:03.390734+0000', 'last_became_peered': '2024-06-03T09:06:03.390734+0000', 'last_unstale': '2024-06-03T09:06:14.431224+0000', 'last_undegraded': '2024-06-03T09:06:14.431224+0000', 'last_fullsized': '2024-06-03T09:06:03.343197+0000', 'mapping_epoch': 999, 'log_start': "0'0", 'ondisk_log_start': "0'0", 'created': 28, 'last_epoch_clean': 254, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2024-06-03T08:37:55.614497+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2024-06-03T08:37:55.614497+0000', 'last_clean_scrub_stamp': '2024-06-03T08:37:55.614497+0000', 'objects_scrubbed': 0, 'log_size': 0, 'log_dups_size': 0, 'ondisk_log_size': 0, 'stats_invalid': False, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': 'periodic scrub scheduled @ 2024-06-04T15:56:47.628875+0000', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 0, 'num_objects': 0, 'num_object_clones': 0, 'num_object_copies': 0, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 0, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 0, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 0, 'num_write_kb': 0, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 0, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [4], 'acting': [4], 'avail_no_missing': ['4'], 'object_location_counts': [], 'blocked_by': [], 'up_primary': 4, 'acting_primary': 4, 'purged_snaps': []} 2024-06-03T09:06:27.041 INFO:tasks.ceph.ceph_manager.ceph:PG 17.1e is not active+clean 2024-06-03T09:06:27.041 INFO:tasks.ceph.ceph_manager.ceph:{'pgid': '17.1e', 'version': "0'0", 'reported_seq': 148, 'reported_epoch': 1009, 'state': 'active+undersized', 'last_fresh': '2024-06-03T09:06:14.445917+0000', 'last_change': '2024-06-03T09:06:03.406181+0000', 'last_active': '2024-06-03T09:06:14.445917+0000', 'last_peered': '2024-06-03T09:06:14.445917+0000', 'last_clean': '2024-06-03T08:37:55.614497+0000', 'last_became_active': '2024-06-03T09:06:03.406181+0000', 'last_became_peered': '2024-06-03T09:06:03.406181+0000', 'last_unstale': '2024-06-03T09:06:14.445917+0000', 'last_undegraded': '2024-06-03T09:06:14.445917+0000', 'last_fullsized': '2024-06-03T09:06:03.363739+0000', 'mapping_epoch': 999, 'log_start': "0'0", 'ondisk_log_start': "0'0", 'created': 28, 'last_epoch_clean': 255, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2024-06-03T08:37:55.614497+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2024-06-03T08:37:55.614497+0000', 'last_clean_scrub_stamp': '2024-06-03T08:37:55.614497+0000', 'objects_scrubbed': 0, 'log_size': 0, 'log_dups_size': 0, 'ondisk_log_size': 0, 'stats_invalid': False, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': 'periodic scrub scheduled @ 2024-06-04T11:45:24.486600+0000', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 0, 'num_objects': 0, 'num_object_clones': 0, 'num_object_copies': 0, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 0, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 0, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 0, 'num_write_kb': 0, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 0, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [4], 'acting': [4], 'avail_no_missing': ['4'], 'object_location_counts': [], 'blocked_by': [], 'up_primary': 4, 'acting_primary': 4, 'purged_snaps': []} 2024-06-03T09:06:27.041 INFO:tasks.ceph.ceph_manager.ceph:PG 17.1c is not active+clean 2024-06-03T09:06:27.041 INFO:tasks.ceph.ceph_manager.ceph:{'pgid': '17.1c', 'version': "0'0", 'reported_seq': 944, 'reported_epoch': 1010, 'state': 'active+undersized', 'last_fresh': '2024-06-03T09:06:18.327492+0000', 'last_change': '2024-06-03T09:06:03.382210+0000', 'last_active': '2024-06-03T09:06:18.327492+0000', 'last_peered': '2024-06-03T09:06:18.327492+0000', 'last_clean': '2024-06-03T09:04:43.199660+0000', 'last_became_active': '2024-06-03T09:06:03.382210+0000', 'last_became_peered': '2024-06-03T09:06:03.382210+0000', 'last_unstale': '2024-06-03T09:06:18.327492+0000', 'last_undegraded': '2024-06-03T09:06:18.327492+0000', 'last_fullsized': '2024-06-03T09:06:03.277874+0000', 'mapping_epoch': 999, 'log_start': "0'0", 'ondisk_log_start': "0'0", 'created': 28, 'last_epoch_clean': 255, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2024-06-03T08:37:55.614497+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2024-06-03T08:37:55.614497+0000', 'last_clean_scrub_stamp': '2024-06-03T08:37:55.614497+0000', 'objects_scrubbed': 0, 'log_size': 0, 'log_dups_size': 0, 'ondisk_log_size': 0, 'stats_invalid': False, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': 'periodic scrub scheduled @ 2024-06-04T12:18:50.013884+0000', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 0, 'num_objects': 0, 'num_object_clones': 0, 'num_object_copies': 0, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 0, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 0, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 0, 'num_write_kb': 0, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 0, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [5], 'acting': [5], 'avail_no_missing': ['5'], 'object_location_counts': [], 'blocked_by': [], 'up_primary': 5, 'acting_primary': 5, 'purged_snaps': []} 2024-06-03T09:06:27.041 INFO:tasks.ceph.ceph_manager.ceph:PG 17.1 is not active+clean 2024-06-03T09:06:27.041 INFO:tasks.ceph.ceph_manager.ceph:{'pgid': '17.1', 'version': "65'7", 'reported_seq': 959, 'reported_epoch': 1009, 'state': 'active+undersized+degraded', 'last_fresh': '2024-06-03T09:06:13.709209+0000', 'last_change': '2024-06-03T09:06:03.370660+0000', 'last_active': '2024-06-03T09:06:13.709209+0000', 'last_peered': '2024-06-03T09:06:13.709209+0000', 'last_clean': '2024-06-03T09:05:03.733864+0000', 'last_became_active': '2024-06-03T09:06:03.370660+0000', 'last_became_peered': '2024-06-03T09:06:03.370660+0000', 'last_unstale': '2024-06-03T09:06:13.709209+0000', 'last_undegraded': '2024-06-03T09:06:03.266076+0000', 'last_fullsized': '2024-06-03T09:06:03.264525+0000', 'mapping_epoch': 999, 'log_start': "0'0", 'ondisk_log_start': "0'0", 'created': 28, 'last_epoch_clean': 255, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2024-06-03T08:37:55.614497+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2024-06-03T08:37:55.614497+0000', 'last_clean_scrub_stamp': '2024-06-03T08:37:55.614497+0000', 'objects_scrubbed': 0, 'log_size': 7, 'log_dups_size': 0, 'ondisk_log_size': 7, 'stats_invalid': False, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': 'periodic scrub scheduled @ 2024-06-04T11:55:18.194283+0000', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 5, 'num_objects': 1, 'num_object_clones': 0, 'num_object_copies': 2, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 1, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 1, 'num_whiteouts': 0, 'num_read': 6, 'num_read_kb': 6, 'num_write': 7, 'num_write_kb': 7, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 0, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [5], 'acting': [5], 'avail_no_missing': ['5'], 'object_location_counts': [{'shards': '5', 'objects': 1}], 'blocked_by': [], 'up_primary': 5, 'acting_primary': 5, 'purged_snaps': []} 2024-06-03T09:06:27.042 DEBUG:teuthology.orchestra.run.smithi088:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph status --format=json 2024-06-03T09:06:27.142 INFO:teuthology.orchestra.run.smithi088.stderr:2024-06-03T09:06:27.141+0000 7f0ff3516640 10 monclient: get_monmap_and_config 2024-06-03T09:06:27.142 INFO:teuthology.orchestra.run.smithi088.stderr:2024-06-03T09:06:27.141+0000 7f0ff3516640 10 monclient: build_initial_monmap 2024-06-03T09:06:27.142 INFO:teuthology.orchestra.run.smithi088.stderr:2024-06-03T09:06:27.142+0000 7f0ff3516640 10 monclient: monmap: 2024-06-03T09:06:27.142 INFO:teuthology.orchestra.run.smithi088.stderr:epoch 0 2024-06-03T09:06:27.142 INFO:teuthology.orchestra.run.smithi088.stderr:fsid ec3c3fdd-3ac6-4d7d-9412-683130ed8b04 2024-06-03T09:06:27.142 INFO:teuthology.orchestra.run.smithi088.stderr:last_changed 2024-06-03T09:06:27.143012+0000 2024-06-03T09:06:27.142 INFO:teuthology.orchestra.run.smithi088.stderr:created 2024-06-03T09:06:27.143012+0000 2024-06-03T09:06:27.142 INFO:teuthology.orchestra.run.smithi088.stderr:min_mon_release 0 (unknown) 2024-06-03T09:06:27.142 INFO:teuthology.orchestra.run.smithi088.stderr:election_strategy: 3 2024-06-03T09:06:27.143 INFO:teuthology.orchestra.run.smithi088.stderr:0: v1:172.21.15.88:6789/0 mon.noname-a 2024-06-03T09:06:27.143 INFO:teuthology.orchestra.run.smithi088.stderr:1: v1:172.21.15.192:6789/0 mon.noname-c 2024-06-03T09:06:27.143 INFO:teuthology.orchestra.run.smithi088.stderr:2: v1:172.21.15.88:6790/0 mon.noname-b 2024-06-03T09:06:27.143 INFO:teuthology.orchestra.run.smithi088.stderr: 2024-06-03T09:06:27.144 INFO:teuthology.orchestra.run.smithi088.stderr:2024-06-03T09:06:27.142+0000 7f0ff3516640 1 Processor -- start 2024-06-03T09:06:27.144 INFO:teuthology.orchestra.run.smithi088.stderr:2024-06-03T09:06:27.142+0000 7f0ff3516640 1 -- start start 2024-06-03T09:06:27.144 INFO:teuthology.orchestra.run.smithi088.stderr:2024-06-03T09:06:27.142+0000 7f0ff3516640 10 monclient: init 2024-06-03T09:06:27.144 INFO:teuthology.orchestra.run.smithi088.stderr:2024-06-03T09:06:27.143+0000 7f0ff3516640 10 monclient: _reopen_session rank -1 2024-06-03T09:06:27.144 INFO:teuthology.orchestra.run.smithi088.stderr:2024-06-03T09:06:27.143+0000 7f0ff3516640 10 monclient: _add_conns ranks=[2,1,0] 2024-06-03T09:06:27.144 INFO:teuthology.orchestra.run.smithi088.stderr:2024-06-03T09:06:27.143+0000 7f0ff3516640 10 monclient(hunting): picked mon.noname-b con 0x7f0fec06f970 addr v1:172.21.15.88:6790/0 2024-06-03T09:06:27.144 INFO:teuthology.orchestra.run.smithi088.stderr:2024-06-03T09:06:27.143+0000 7f0ff3516640 10 monclient(hunting): picked mon.noname-c con 0x7f0fec0ad840 addr v1:172.21.15.192:6789/0 2024-06-03T09:06:27.145 INFO:teuthology.orchestra.run.smithi088.stderr:2024-06-03T09:06:27.143+0000 7f0ff3516640 10 monclient(hunting): picked mon.noname-a con 0x7f0fec06fd50 addr v1:172.21.15.88:6789/0 2024-06-03T09:06:27.145 INFO:teuthology.orchestra.run.smithi088.stderr:2024-06-03T09:06:27.143+0000 7f0ff3516640 1 -- --> v1:172.21.15.88:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f0fec071be0 con 0x7f0fec06fd50 2024-06-03T09:06:27.145 INFO:teuthology.orchestra.run.smithi088.stderr:2024-06-03T09:06:27.143+0000 7f0ff3516640 1 -- --> v1:172.21.15.192:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f0fec071ec0 con 0x7f0fec0ad840 2024-06-03T09:06:27.145 INFO:teuthology.orchestra.run.smithi088.stderr:2024-06-03T09:06:27.143+0000 7f0ff3516640 1 -- --> v1:172.21.15.88:6790/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f0fec0708e0 con 0x7f0fec06f970 2024-06-03T09:06:27.145 INFO:teuthology.orchestra.run.smithi088.stderr:2024-06-03T09:06:27.143+0000 7f0ff3516640 10 monclient(hunting): _renew_subs 2024-06-03T09:06:27.145 INFO:teuthology.orchestra.run.smithi088.stderr:2024-06-03T09:06:27.143+0000 7f0ff3516640 10 monclient(hunting): authenticate will time out at 3042.953359s 2024-06-03T09:06:27.145 INFO:teuthology.orchestra.run.smithi088.stderr:2024-06-03T09:06:27.143+0000 7f0ff1ab3640 1 --1- >> v1:172.21.15.88:6789/0 conn(0x7f0fec06fd50 0x7f0fec14abf0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.88:6789/0 says I am v1:172.21.15.88:41536/0 (socket says 172.21.15.88:41536) 2024-06-03T09:06:27.145 INFO:teuthology.orchestra.run.smithi088.stderr:2024-06-03T09:06:27.143+0000 7f0ff1ab3640 1 -- 172.21.15.88:0/3064106962 learned_addr learned my addr 172.21.15.88:0/3064106962 (peer_addr_for_me v1:172.21.15.88:0/0) 2024-06-03T09:06:27.146 INFO:teuthology.orchestra.run.smithi088.stderr:2024-06-03T09:06:27.146+0000 7f0febfff640 1 -- 172.21.15.88:0/3064106962 <== mon.1 v1:172.21.15.192:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 2988811960 0 0) 0x7f0fec071ec0 con 0x7f0fec0ad840 2024-06-03T09:06:27.146 INFO:teuthology.orchestra.run.smithi088.stderr:2024-06-03T09:06:27.146+0000 7f0febfff640 10 monclient(hunting): _negotiate 2024-06-03T09:06:27.146 INFO:teuthology.orchestra.run.smithi088.stderr:2024-06-03T09:06:27.146+0000 7f0febfff640 10 monclient(hunting): _init_auth method 2 2024-06-03T09:06:27.146 INFO:teuthology.orchestra.run.smithi088.stderr:2024-06-03T09:06:27.146+0000 7f0febfff640 10 monclient(hunting): _init_auth creating new auth 2024-06-03T09:06:27.146 INFO:teuthology.orchestra.run.smithi088.stderr:2024-06-03T09:06:27.146+0000 7f0febfff640 10 monclient(hunting): my global_id is 30193 2024-06-03T09:06:27.146 INFO:teuthology.orchestra.run.smithi088.stderr:2024-06-03T09:06:27.146+0000 7f0febfff640 1 -- 172.21.15.88:0/3064106962 --> v1:172.21.15.192:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f0fd4003520 con 0x7f0fec0ad840 2024-06-03T09:06:27.146 INFO:teuthology.orchestra.run.smithi088.stderr:2024-06-03T09:06:27.146+0000 7f0febfff640 1 -- 172.21.15.88:0/3064106962 <== mon.0 v1:172.21.15.88:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 1565239170 0 0) 0x7f0fec071be0 con 0x7f0fec06fd50 2024-06-03T09:06:27.146 INFO:teuthology.orchestra.run.smithi088.stderr:2024-06-03T09:06:27.146+0000 7f0febfff640 10 monclient(hunting): _negotiate 2024-06-03T09:06:27.147 INFO:teuthology.orchestra.run.smithi088.stderr:2024-06-03T09:06:27.146+0000 7f0febfff640 10 monclient(hunting): _init_auth method 2 2024-06-03T09:06:27.147 INFO:teuthology.orchestra.run.smithi088.stderr:2024-06-03T09:06:27.146+0000 7f0febfff640 10 monclient(hunting): _init_auth creating new auth 2024-06-03T09:06:27.147 INFO:teuthology.orchestra.run.smithi088.stderr:2024-06-03T09:06:27.146+0000 7f0febfff640 10 monclient(hunting): my global_id is 30612 2024-06-03T09:06:27.147 INFO:teuthology.orchestra.run.smithi088.stderr:2024-06-03T09:06:27.146+0000 7f0febfff640 1 -- 172.21.15.88:0/3064106962 --> v1:172.21.15.88:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f0fec071ec0 con 0x7f0fec06fd50 2024-06-03T09:06:27.147 INFO:teuthology.orchestra.run.smithi088.stderr:2024-06-03T09:06:27.146+0000 7f0febfff640 1 -- 172.21.15.88:0/3064106962 <== mon.2 v1:172.21.15.88:6790/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 245950379 0 0) 0x7f0fec0708e0 con 0x7f0fec06f970 2024-06-03T09:06:27.147 INFO:teuthology.orchestra.run.smithi088.stderr:2024-06-03T09:06:27.146+0000 7f0febfff640 10 monclient(hunting): _negotiate 2024-06-03T09:06:27.147 INFO:teuthology.orchestra.run.smithi088.stderr:2024-06-03T09:06:27.146+0000 7f0febfff640 10 monclient(hunting): _init_auth method 2 2024-06-03T09:06:27.148 INFO:teuthology.orchestra.run.smithi088.stderr:2024-06-03T09:06:27.146+0000 7f0febfff640 10 monclient(hunting): _init_auth creating new auth 2024-06-03T09:06:27.148 INFO:teuthology.orchestra.run.smithi088.stderr:2024-06-03T09:06:27.146+0000 7f0febfff640 10 monclient(hunting): my global_id is 30116 2024-06-03T09:06:27.148 INFO:teuthology.orchestra.run.smithi088.stderr:2024-06-03T09:06:27.146+0000 7f0febfff640 1 -- 172.21.15.88:0/3064106962 --> v1:172.21.15.88:6790/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f0fec071be0 con 0x7f0fec06f970 2024-06-03T09:06:27.149 INFO:teuthology.orchestra.run.smithi088.stderr:2024-06-03T09:06:27.149+0000 7f0febfff640 1 -- 172.21.15.88:0/3064106962 <== mon.1 v1:172.21.15.192:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 214+0+0 (unknown 1955242099 0 0) 0x7f0fd4003520 con 0x7f0fec0ad840 2024-06-03T09:06:27.149 INFO:teuthology.orchestra.run.smithi088.stderr:2024-06-03T09:06:27.149+0000 7f0febfff640 1 -- 172.21.15.88:0/3064106962 >> v1:172.21.15.88:6790/0 conn(0x7f0fec06f970 legacy=0x7f0fec13fa70 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-06-03T09:06:27.149 INFO:teuthology.orchestra.run.smithi088.stderr:2024-06-03T09:06:27.149+0000 7f0febfff640 1 -- 172.21.15.88:0/3064106962 >> v1:172.21.15.88:6789/0 conn(0x7f0fec06fd50 legacy=0x7f0fec14abf0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-06-03T09:06:27.149 INFO:teuthology.orchestra.run.smithi088.stderr:2024-06-03T09:06:27.149+0000 7f0febfff640 10 monclient: _finish_hunting 0 2024-06-03T09:06:27.150 INFO:teuthology.orchestra.run.smithi088.stderr:2024-06-03T09:06:27.149+0000 7f0febfff640 1 monclient: found mon.noname-c 2024-06-03T09:06:27.150 INFO:teuthology.orchestra.run.smithi088.stderr:2024-06-03T09:06:27.149+0000 7f0febfff640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-06-03T09:06:27.150 INFO:teuthology.orchestra.run.smithi088.stderr:2024-06-03T09:06:27.149+0000 7f0febfff640 10 monclient: _send_mon_message to mon.noname-c at v1:172.21.15.192:6789/0 2024-06-03T09:06:27.150 INFO:teuthology.orchestra.run.smithi088.stderr:2024-06-03T09:06:27.149+0000 7f0febfff640 1 -- 172.21.15.88:0/3064106962 --> v1:172.21.15.192:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f0fec13f550 con 0x7f0fec0ad840 2024-06-03T09:06:27.150 INFO:teuthology.orchestra.run.smithi088.stderr:2024-06-03T09:06:27.149+0000 7f0febfff640 10 monclient: _finish_auth 0 2024-06-03T09:06:27.150 INFO:teuthology.orchestra.run.smithi088.stderr:2024-06-03T09:06:27.149+0000 7f0febfff640 10 monclient: _check_auth_tickets 2024-06-03T09:06:27.150 INFO:teuthology.orchestra.run.smithi088.stderr:2024-06-03T09:06:27.149+0000 7f0febfff640 20 monclient: _check_auth_rotating not needed by client.admin 2024-06-03T09:06:27.150 INFO:teuthology.orchestra.run.smithi088.stderr:2024-06-03T09:06:27.149+0000 7f0febfff640 1 -- 172.21.15.88:0/3064106962 <== mon.1 v1:172.21.15.192:6789/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 2234374065 0 0) 0x7f0fe0001610 con 0x7f0fec0ad840 2024-06-03T09:06:27.150 INFO:teuthology.orchestra.run.smithi088.stderr:2024-06-03T09:06:27.149+0000 7f0febfff640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-06-03T09:06:27.150 INFO:teuthology.orchestra.run.smithi088.stderr:2024-06-03T09:06:27.149+0000 7f0febfff640 10 monclient: got monmap 1 from mon.noname-c (according to old e1) 2024-06-03T09:06:27.150 INFO:teuthology.orchestra.run.smithi088.stderr:2024-06-03T09:06:27.149+0000 7f0febfff640 10 monclient: dump: 2024-06-03T09:06:27.150 INFO:teuthology.orchestra.run.smithi088.stderr:epoch 1 2024-06-03T09:06:27.151 INFO:teuthology.orchestra.run.smithi088.stderr:fsid ec3c3fdd-3ac6-4d7d-9412-683130ed8b04 2024-06-03T09:06:27.151 INFO:teuthology.orchestra.run.smithi088.stderr:last_changed 2024-06-03T08:34:31.885125+0000 2024-06-03T09:06:27.151 INFO:teuthology.orchestra.run.smithi088.stderr:created 2024-06-03T08:34:31.885125+0000 2024-06-03T09:06:27.151 INFO:teuthology.orchestra.run.smithi088.stderr:min_mon_release 18 (reef) 2024-06-03T09:06:27.151 INFO:teuthology.orchestra.run.smithi088.stderr:election_strategy: 3 2024-06-03T09:06:27.151 INFO:teuthology.orchestra.run.smithi088.stderr:0: v1:172.21.15.88:6789/0 mon.a 2024-06-03T09:06:27.151 INFO:teuthology.orchestra.run.smithi088.stderr:1: v1:172.21.15.192:6789/0 mon.b 2024-06-03T09:06:27.151 INFO:teuthology.orchestra.run.smithi088.stderr:2: v1:172.21.15.88:6790/0 mon.c 2024-06-03T09:06:27.151 INFO:teuthology.orchestra.run.smithi088.stderr: 2024-06-03T09:06:27.151 INFO:teuthology.orchestra.run.smithi088.stderr:2024-06-03T09:06:27.149+0000 7f0ff3516640 5 monclient: authenticate success, global_id 30193 2024-06-03T09:06:27.151 INFO:teuthology.orchestra.run.smithi088.stderr:2024-06-03T09:06:27.149+0000 7f0ff3516640 20 monclient: get_monmap_and_config waiting for monmap|config 2024-06-03T09:06:27.154 INFO:teuthology.orchestra.run.smithi088.stderr:2024-06-03T09:06:27.151+0000 7f0febfff640 1 -- 172.21.15.88:0/3064106962 <== mon.1 v1:172.21.15.192:6789/0 4 ==== config(0 keys) v1 ==== 4+0+0 (unknown 0 0 0) 0x7f0fe0001830 con 0x7f0fec0ad840 2024-06-03T09:06:27.154 INFO:teuthology.orchestra.run.smithi088.stderr:2024-06-03T09:06:27.151+0000 7f0febfff640 10 monclient: handle_config config(0 keys) v1 2024-06-03T09:06:27.154 INFO:teuthology.orchestra.run.smithi088.stderr:2024-06-03T09:06:27.151+0000 7f0febfff640 1 -- 172.21.15.88:0/3064106962 <== mon.1 v1:172.21.15.192:6789/0 5 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 2234374065 0 0) 0x7f0fe0001bc0 con 0x7f0fec0ad840 2024-06-03T09:06:27.154 INFO:teuthology.orchestra.run.smithi088.stderr:2024-06-03T09:06:27.151+0000 7f0febfff640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-06-03T09:06:27.154 INFO:teuthology.orchestra.run.smithi088.stderr:2024-06-03T09:06:27.151+0000 7f0febfff640 10 monclient: got monmap 1 from mon.b (according to old e1) 2024-06-03T09:06:27.155 INFO:teuthology.orchestra.run.smithi088.stderr:2024-06-03T09:06:27.151+0000 7f0febfff640 10 monclient: dump: 2024-06-03T09:06:27.155 INFO:teuthology.orchestra.run.smithi088.stderr:epoch 1 2024-06-03T09:06:27.155 INFO:teuthology.orchestra.run.smithi088.stderr:fsid ec3c3fdd-3ac6-4d7d-9412-683130ed8b04 2024-06-03T09:06:27.155 INFO:teuthology.orchestra.run.smithi088.stderr:last_changed 2024-06-03T08:34:31.885125+0000 2024-06-03T09:06:27.155 INFO:teuthology.orchestra.run.smithi088.stderr:created 2024-06-03T08:34:31.885125+0000 2024-06-03T09:06:27.155 INFO:teuthology.orchestra.run.smithi088.stderr:min_mon_release 18 (reef) 2024-06-03T09:06:27.155 INFO:teuthology.orchestra.run.smithi088.stderr:election_strategy: 3 2024-06-03T09:06:27.155 INFO:teuthology.orchestra.run.smithi088.stderr:0: v1:172.21.15.88:6789/0 mon.a 2024-06-03T09:06:27.155 INFO:teuthology.orchestra.run.smithi088.stderr:1: v1:172.21.15.192:6789/0 mon.b 2024-06-03T09:06:27.155 INFO:teuthology.orchestra.run.smithi088.stderr:2: v1:172.21.15.88:6790/0 mon.c 2024-06-03T09:06:27.155 INFO:teuthology.orchestra.run.smithi088.stderr: 2024-06-03T09:06:27.155 INFO:teuthology.orchestra.run.smithi088.stderr:2024-06-03T09:06:27.151+0000 7f0ff3516640 10 monclient: get_monmap_and_config success 2024-06-03T09:06:27.155 INFO:teuthology.orchestra.run.smithi088.stderr:2024-06-03T09:06:27.151+0000 7f0ff3516640 4 set_mon_vals no callback set 2024-06-03T09:06:27.156 INFO:teuthology.orchestra.run.smithi088.stderr:2024-06-03T09:06:27.151+0000 7f0ff3516640 10 monclient: shutdown 2024-06-03T09:06:27.156 INFO:teuthology.orchestra.run.smithi088.stderr:2024-06-03T09:06:27.151+0000 7f0ff3516640 20 monclient: shutdown discarding 0 pending message(s) 2024-06-03T09:06:27.156 INFO:teuthology.orchestra.run.smithi088.stderr:2024-06-03T09:06:27.151+0000 7f0ff3516640 1 -- 172.21.15.88:0/3064106962 >> v1:172.21.15.192:6789/0 conn(0x7f0fec0ad840 legacy=0x7f0fec13c770 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-06-03T09:06:27.156 INFO:teuthology.orchestra.run.smithi088.stderr:2024-06-03T09:06:27.151+0000 7f0ff3516640 1 -- 172.21.15.88:0/3064106962 shutdown_connections 2024-06-03T09:06:27.156 INFO:teuthology.orchestra.run.smithi088.stderr:2024-06-03T09:06:27.151+0000 7f0ff3516640 1 -- 172.21.15.88:0/3064106962 >> 172.21.15.88:0/3064106962 conn(0x7f0fec0b2e40 msgr2=0x7f0fec0b3250 unknown :-1 s=STATE_NONE l=0).mark_down 2024-06-03T09:06:27.156 INFO:teuthology.orchestra.run.smithi088.stderr:2024-06-03T09:06:27.151+0000 7f0ff3516640 1 -- 172.21.15.88:0/3064106962 shutdown_connections 2024-06-03T09:06:27.156 INFO:teuthology.orchestra.run.smithi088.stderr:2024-06-03T09:06:27.151+0000 7f0ff3516640 1 -- 172.21.15.88:0/3064106962 wait complete. 2024-06-03T09:06:27.156 INFO:teuthology.orchestra.run.smithi088.stderr:2024-06-03T09:06:27.152+0000 7f0ff3516640 10 monclient: build_initial_monmap 2024-06-03T09:06:27.156 INFO:teuthology.orchestra.run.smithi088.stderr:2024-06-03T09:06:27.152+0000 7f0ff3516640 10 monclient: monmap: 2024-06-03T09:06:27.156 INFO:teuthology.orchestra.run.smithi088.stderr:epoch 0 2024-06-03T09:06:27.156 INFO:teuthology.orchestra.run.smithi088.stderr:fsid 00000000-0000-0000-0000-000000000000 2024-06-03T09:06:27.156 INFO:teuthology.orchestra.run.smithi088.stderr:last_changed 0.000000 2024-06-03T09:06:27.156 INFO:teuthology.orchestra.run.smithi088.stderr:created 0.000000 2024-06-03T09:06:27.156 INFO:teuthology.orchestra.run.smithi088.stderr:min_mon_release 0 (unknown) 2024-06-03T09:06:27.157 INFO:teuthology.orchestra.run.smithi088.stderr:election_strategy: 1 2024-06-03T09:06:27.157 INFO:teuthology.orchestra.run.smithi088.stderr:0: v1:172.21.15.88:6789/0 mon.noname-a 2024-06-03T09:06:27.157 INFO:teuthology.orchestra.run.smithi088.stderr:1: v1:172.21.15.192:6789/0 mon.noname-b 2024-06-03T09:06:27.157 INFO:teuthology.orchestra.run.smithi088.stderr:2: v1:172.21.15.88:6790/0 mon.noname-c 2024-06-03T09:06:27.157 INFO:teuthology.orchestra.run.smithi088.stderr: 2024-06-03T09:06:27.157 INFO:teuthology.orchestra.run.smithi088.stderr:2024-06-03T09:06:27.152+0000 7f0ff3516640 1 librados: starting msgr at 2024-06-03T09:06:27.157 INFO:teuthology.orchestra.run.smithi088.stderr:2024-06-03T09:06:27.152+0000 7f0ff3516640 1 librados: starting objecter 2024-06-03T09:06:27.157 INFO:teuthology.orchestra.run.smithi088.stderr:2024-06-03T09:06:27.152+0000 7f0ff3516640 1 Processor -- start 2024-06-03T09:06:27.158 INFO:teuthology.orchestra.run.smithi088.stderr:2024-06-03T09:06:27.153+0000 7f0ff3516640 1 -- start start 2024-06-03T09:06:27.158 INFO:teuthology.orchestra.run.smithi088.stderr:2024-06-03T09:06:27.153+0000 7f0ff3516640 1 librados: setting wanted keys 2024-06-03T09:06:27.158 INFO:teuthology.orchestra.run.smithi088.stderr:2024-06-03T09:06:27.153+0000 7f0ff3516640 1 librados: calling monclient init 2024-06-03T09:06:27.158 INFO:teuthology.orchestra.run.smithi088.stderr:2024-06-03T09:06:27.153+0000 7f0ff3516640 10 monclient: init 2024-06-03T09:06:27.158 INFO:teuthology.orchestra.run.smithi088.stderr:2024-06-03T09:06:27.153+0000 7f0ff3516640 10 monclient: _reopen_session rank -1 2024-06-03T09:06:27.158 INFO:teuthology.orchestra.run.smithi088.stderr:2024-06-03T09:06:27.153+0000 7f0ff3516640 10 monclient: _add_conns ranks=[0,1,2] 2024-06-03T09:06:27.158 INFO:teuthology.orchestra.run.smithi088.stderr:2024-06-03T09:06:27.153+0000 7f0ff3516640 10 monclient(hunting): picked mon.noname-a con 0x7f0fec06f970 addr v1:172.21.15.88:6789/0 2024-06-03T09:06:27.158 INFO:teuthology.orchestra.run.smithi088.stderr:2024-06-03T09:06:27.153+0000 7f0ff3516640 10 monclient(hunting): picked mon.noname-b con 0x7f0fec06fd50 addr v1:172.21.15.192:6789/0 2024-06-03T09:06:27.158 INFO:teuthology.orchestra.run.smithi088.stderr:2024-06-03T09:06:27.153+0000 7f0ff3516640 10 monclient(hunting): picked mon.noname-c con 0x7f0fec0ad840 addr v1:172.21.15.88:6790/0 2024-06-03T09:06:27.158 INFO:teuthology.orchestra.run.smithi088.stderr:2024-06-03T09:06:27.153+0000 7f0ff3516640 1 -- --> v1:172.21.15.88:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f0fec1d5360 con 0x7f0fec06f970 2024-06-03T09:06:27.158 INFO:teuthology.orchestra.run.smithi088.stderr:2024-06-03T09:06:27.153+0000 7f0ff3516640 1 -- --> v1:172.21.15.192:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f0fec1d4450 con 0x7f0fec06fd50 2024-06-03T09:06:27.158 INFO:teuthology.orchestra.run.smithi088.stderr:2024-06-03T09:06:27.153+0000 7f0ff3516640 1 -- --> v1:172.21.15.88:6790/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f0fec12fe30 con 0x7f0fec0ad840 2024-06-03T09:06:27.159 INFO:teuthology.orchestra.run.smithi088.stderr:2024-06-03T09:06:27.153+0000 7f0ff3516640 10 monclient(hunting): _renew_subs 2024-06-03T09:06:27.159 INFO:teuthology.orchestra.run.smithi088.stderr:2024-06-03T09:06:27.153+0000 7f0ff3516640 10 monclient(hunting): authenticate will time out at 3042.963460s 2024-06-03T09:06:27.159 INFO:teuthology.orchestra.run.smithi088.stderr:2024-06-03T09:06:27.153+0000 7f0ff1ab3640 1 --1- >> v1:172.21.15.88:6790/0 conn(0x7f0fec0ad840 0x7f0fec12cf20 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=1).handle_server_banner_and_identify peer v1:172.21.15.88:6790/0 says I am v1:172.21.15.88:36316/0 (socket says 172.21.15.88:36316) 2024-06-03T09:06:27.159 INFO:teuthology.orchestra.run.smithi088.stderr:2024-06-03T09:06:27.153+0000 7f0ff1ab3640 1 -- 172.21.15.88:0/4140293942 learned_addr learned my addr 172.21.15.88:0/4140293942 (peer_addr_for_me v1:172.21.15.88:0/0) 2024-06-03T09:06:27.159 INFO:teuthology.orchestra.run.smithi088.stderr:2024-06-03T09:06:27.154+0000 7f0fe9ffb640 10 client.?.objecter ms_handle_connect 0x7f0fec06f970 2024-06-03T09:06:27.159 INFO:teuthology.orchestra.run.smithi088.stderr:2024-06-03T09:06:27.154+0000 7f0fe9ffb640 10 client.?.objecter resend_mon_ops 2024-06-03T09:06:27.159 INFO:teuthology.orchestra.run.smithi088.stderr:2024-06-03T09:06:27.154+0000 7f0fe9ffb640 10 client.?.objecter ms_handle_connect 0x7f0fec06fd50 2024-06-03T09:06:27.159 INFO:teuthology.orchestra.run.smithi088.stderr:2024-06-03T09:06:27.154+0000 7f0fe9ffb640 10 client.?.objecter resend_mon_ops 2024-06-03T09:06:27.159 INFO:teuthology.orchestra.run.smithi088.stderr:2024-06-03T09:06:27.154+0000 7f0fe9ffb640 10 client.?.objecter ms_handle_connect 0x7f0fec0ad840 2024-06-03T09:06:27.159 INFO:teuthology.orchestra.run.smithi088.stderr:2024-06-03T09:06:27.154+0000 7f0fe9ffb640 10 client.?.objecter resend_mon_ops 2024-06-03T09:06:27.161 INFO:teuthology.orchestra.run.smithi088.stderr:2024-06-03T09:06:27.156+0000 7f0fe9ffb640 1 -- 172.21.15.88:0/4140293942 <== mon.1 v1:172.21.15.192:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 802787400 0 0) 0x7f0fec1d4450 con 0x7f0fec06fd50 2024-06-03T09:06:27.161 INFO:teuthology.orchestra.run.smithi088.stderr:2024-06-03T09:06:27.156+0000 7f0fe9ffb640 10 monclient(hunting): _negotiate 2024-06-03T09:06:27.161 INFO:teuthology.orchestra.run.smithi088.stderr:2024-06-03T09:06:27.156+0000 7f0fe9ffb640 10 monclient(hunting): _init_auth method 2 2024-06-03T09:06:27.161 INFO:teuthology.orchestra.run.smithi088.stderr:2024-06-03T09:06:27.156+0000 7f0fe9ffb640 10 monclient(hunting): _init_auth creating new auth 2024-06-03T09:06:27.162 INFO:teuthology.orchestra.run.smithi088.stderr:2024-06-03T09:06:27.156+0000 7f0fe9ffb640 10 monclient(hunting): my global_id is 30196 2024-06-03T09:06:27.162 INFO:teuthology.orchestra.run.smithi088.stderr:2024-06-03T09:06:27.156+0000 7f0fe9ffb640 1 -- 172.21.15.88:0/4140293942 --> v1:172.21.15.192:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f0fd8004eb0 con 0x7f0fec06fd50 2024-06-03T09:06:27.162 INFO:teuthology.orchestra.run.smithi088.stderr:2024-06-03T09:06:27.156+0000 7f0fe9ffb640 1 -- 172.21.15.88:0/4140293942 <== mon.2 v1:172.21.15.88:6790/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 319094395 0 0) 0x7f0fec12fe30 con 0x7f0fec0ad840 2024-06-03T09:06:27.162 INFO:teuthology.orchestra.run.smithi088.stderr:2024-06-03T09:06:27.156+0000 7f0fe9ffb640 10 monclient(hunting): _negotiate 2024-06-03T09:06:27.162 INFO:teuthology.orchestra.run.smithi088.stderr:2024-06-03T09:06:27.156+0000 7f0fe9ffb640 10 monclient(hunting): _init_auth method 2 2024-06-03T09:06:27.162 INFO:teuthology.orchestra.run.smithi088.stderr:2024-06-03T09:06:27.156+0000 7f0fe9ffb640 10 monclient(hunting): _init_auth creating new auth 2024-06-03T09:06:27.162 INFO:teuthology.orchestra.run.smithi088.stderr:2024-06-03T09:06:27.156+0000 7f0fe9ffb640 10 monclient(hunting): my global_id is 30119 2024-06-03T09:06:27.162 INFO:teuthology.orchestra.run.smithi088.stderr:2024-06-03T09:06:27.156+0000 7f0fe9ffb640 1 -- 172.21.15.88:0/4140293942 --> v1:172.21.15.88:6790/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f0fec1d4450 con 0x7f0fec0ad840 2024-06-03T09:06:27.162 INFO:teuthology.orchestra.run.smithi088.stderr:2024-06-03T09:06:27.156+0000 7f0fe9ffb640 1 -- 172.21.15.88:0/4140293942 <== mon.0 v1:172.21.15.88:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 3373471018 0 0) 0x7f0fec1d5360 con 0x7f0fec06f970 2024-06-03T09:06:27.162 INFO:teuthology.orchestra.run.smithi088.stderr:2024-06-03T09:06:27.156+0000 7f0fe9ffb640 10 monclient(hunting): _negotiate 2024-06-03T09:06:27.162 INFO:teuthology.orchestra.run.smithi088.stderr:2024-06-03T09:06:27.156+0000 7f0fe9ffb640 10 monclient(hunting): _init_auth method 2 2024-06-03T09:06:27.162 INFO:teuthology.orchestra.run.smithi088.stderr:2024-06-03T09:06:27.156+0000 7f0fe9ffb640 10 monclient(hunting): _init_auth creating new auth 2024-06-03T09:06:27.162 INFO:teuthology.orchestra.run.smithi088.stderr:2024-06-03T09:06:27.156+0000 7f0fe9ffb640 10 monclient(hunting): my global_id is 30615 2024-06-03T09:06:27.163 INFO:teuthology.orchestra.run.smithi088.stderr:2024-06-03T09:06:27.156+0000 7f0fe9ffb640 1 -- 172.21.15.88:0/4140293942 --> v1:172.21.15.88:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f0fec12fe30 con 0x7f0fec06f970 2024-06-03T09:06:27.163 INFO:teuthology.orchestra.run.smithi088.stderr:2024-06-03T09:06:27.160+0000 7f0fe9ffb640 1 -- 172.21.15.88:0/4140293942 <== mon.1 v1:172.21.15.192:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 764+0+0 (unknown 3203562018 0 0) 0x7f0fd8004eb0 con 0x7f0fec06fd50 2024-06-03T09:06:27.163 INFO:teuthology.orchestra.run.smithi088.stderr:2024-06-03T09:06:27.160+0000 7f0fe9ffb640 1 -- 172.21.15.88:0/4140293942 --> v1:172.21.15.192:6789/0 -- auth(proto 2 165 bytes epoch 0) v1 -- 0x7f0fec1d5360 con 0x7f0fec06fd50 2024-06-03T09:06:27.163 INFO:teuthology.orchestra.run.smithi088.stderr:2024-06-03T09:06:27.160+0000 7f0fe9ffb640 1 -- 172.21.15.88:0/4140293942 <== mon.1 v1:172.21.15.192:6789/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 2234374065 0 0) 0x7f0fe0001b40 con 0x7f0fec06fd50 2024-06-03T09:06:27.163 INFO:teuthology.orchestra.run.smithi088.stderr:2024-06-03T09:06:27.160+0000 7f0fe9ffb640 10 monclient(hunting): handle_monmap mon_map magic: 0 v1 2024-06-03T09:06:27.163 INFO:teuthology.orchestra.run.smithi088.stderr:2024-06-03T09:06:27.160+0000 7f0fe9ffb640 10 monclient(hunting): got monmap 1 from mon.noname-b (according to old e1) 2024-06-03T09:06:27.163 INFO:teuthology.orchestra.run.smithi088.stderr:2024-06-03T09:06:27.160+0000 7f0fe9ffb640 10 monclient(hunting): dump: 2024-06-03T09:06:27.163 INFO:teuthology.orchestra.run.smithi088.stderr:epoch 1 2024-06-03T09:06:27.163 INFO:teuthology.orchestra.run.smithi088.stderr:fsid ec3c3fdd-3ac6-4d7d-9412-683130ed8b04 2024-06-03T09:06:27.163 INFO:teuthology.orchestra.run.smithi088.stderr:last_changed 2024-06-03T08:34:31.885125+0000 2024-06-03T09:06:27.163 INFO:teuthology.orchestra.run.smithi088.stderr:created 2024-06-03T08:34:31.885125+0000 2024-06-03T09:06:27.163 INFO:teuthology.orchestra.run.smithi088.stderr:min_mon_release 18 (reef) 2024-06-03T09:06:27.164 INFO:teuthology.orchestra.run.smithi088.stderr:election_strategy: 3 2024-06-03T09:06:27.164 INFO:teuthology.orchestra.run.smithi088.stderr:0: v1:172.21.15.88:6789/0 mon.a 2024-06-03T09:06:27.164 INFO:teuthology.orchestra.run.smithi088.stderr:1: v1:172.21.15.192:6789/0 mon.b 2024-06-03T09:06:27.164 INFO:teuthology.orchestra.run.smithi088.stderr:2: v1:172.21.15.88:6790/0 mon.c 2024-06-03T09:06:27.164 INFO:teuthology.orchestra.run.smithi088.stderr: 2024-06-03T09:06:27.164 INFO:teuthology.orchestra.run.smithi088.stderr:2024-06-03T09:06:27.160+0000 7f0fe9ffb640 10 monclient(hunting): _finish_auth 0 2024-06-03T09:06:27.164 INFO:teuthology.orchestra.run.smithi088.stderr:2024-06-03T09:06:27.161+0000 7f0fe9ffb640 1 -- 172.21.15.88:0/4140293942 <== mon.2 v1:172.21.15.88:6790/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 764+0+0 (unknown 1001781773 0 0) 0x7f0fec1d4450 con 0x7f0fec0ad840 2024-06-03T09:06:27.164 INFO:teuthology.orchestra.run.smithi088.stderr:2024-06-03T09:06:27.161+0000 7f0fe9ffb640 1 -- 172.21.15.88:0/4140293942 --> v1:172.21.15.88:6790/0 -- auth(proto 2 165 bytes epoch 0) v1 -- 0x7f0fd8004eb0 con 0x7f0fec0ad840 2024-06-03T09:06:27.164 INFO:teuthology.orchestra.run.smithi088.stderr:2024-06-03T09:06:27.161+0000 7f0fe9ffb640 1 -- 172.21.15.88:0/4140293942 <== mon.2 v1:172.21.15.88:6790/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 2234374065 0 0) 0x7f0fdc001bf0 con 0x7f0fec0ad840 2024-06-03T09:06:27.164 INFO:teuthology.orchestra.run.smithi088.stderr:2024-06-03T09:06:27.161+0000 7f0fe9ffb640 10 monclient(hunting): handle_monmap mon_map magic: 0 v1 2024-06-03T09:06:27.164 INFO:teuthology.orchestra.run.smithi088.stderr:2024-06-03T09:06:27.161+0000 7f0fe9ffb640 10 monclient(hunting): got monmap 1 from mon.c (according to old e1) 2024-06-03T09:06:27.164 INFO:teuthology.orchestra.run.smithi088.stderr:2024-06-03T09:06:27.161+0000 7f0fe9ffb640 10 monclient(hunting): dump: 2024-06-03T09:06:27.164 INFO:teuthology.orchestra.run.smithi088.stderr:epoch 1 2024-06-03T09:06:27.165 INFO:teuthology.orchestra.run.smithi088.stderr:fsid ec3c3fdd-3ac6-4d7d-9412-683130ed8b04 2024-06-03T09:06:27.165 INFO:teuthology.orchestra.run.smithi088.stderr:last_changed 2024-06-03T08:34:31.885125+0000 2024-06-03T09:06:27.165 INFO:teuthology.orchestra.run.smithi088.stderr:created 2024-06-03T08:34:31.885125+0000 2024-06-03T09:06:27.165 INFO:teuthology.orchestra.run.smithi088.stderr:min_mon_release 18 (reef) 2024-06-03T09:06:27.165 INFO:teuthology.orchestra.run.smithi088.stderr:election_strategy: 3 2024-06-03T09:06:27.165 INFO:teuthology.orchestra.run.smithi088.stderr:0: v1:172.21.15.88:6789/0 mon.a 2024-06-03T09:06:27.165 INFO:teuthology.orchestra.run.smithi088.stderr:1: v1:172.21.15.192:6789/0 mon.b 2024-06-03T09:06:27.165 INFO:teuthology.orchestra.run.smithi088.stderr:2: v1:172.21.15.88:6790/0 mon.c 2024-06-03T09:06:27.165 INFO:teuthology.orchestra.run.smithi088.stderr: 2024-06-03T09:06:27.165 INFO:teuthology.orchestra.run.smithi088.stderr:2024-06-03T09:06:27.161+0000 7f0fe9ffb640 1 -- 172.21.15.88:0/4140293942 <== mon.0 v1:172.21.15.88:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 764+0+0 (unknown 4134979898 0 0) 0x7f0fec12fe30 con 0x7f0fec06f970 2024-06-03T09:06:27.165 INFO:teuthology.orchestra.run.smithi088.stderr:2024-06-03T09:06:27.161+0000 7f0fe9ffb640 1 -- 172.21.15.88:0/4140293942 --> v1:172.21.15.88:6789/0 -- auth(proto 2 165 bytes epoch 0) v1 -- 0x7f0fec1d4450 con 0x7f0fec06f970 2024-06-03T09:06:27.165 INFO:teuthology.orchestra.run.smithi088.stderr:2024-06-03T09:06:27.161+0000 7f0fe9ffb640 1 -- 172.21.15.88:0/4140293942 <== mon.0 v1:172.21.15.88:6789/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 2234374065 0 0) 0x7f0fe40018b0 con 0x7f0fec06f970 2024-06-03T09:06:27.165 INFO:teuthology.orchestra.run.smithi088.stderr:2024-06-03T09:06:27.161+0000 7f0fe9ffb640 10 monclient(hunting): handle_monmap mon_map magic: 0 v1 2024-06-03T09:06:27.165 INFO:teuthology.orchestra.run.smithi088.stderr:2024-06-03T09:06:27.161+0000 7f0fe9ffb640 10 monclient(hunting): got monmap 1 from mon.a (according to old e1) 2024-06-03T09:06:27.166 INFO:teuthology.orchestra.run.smithi088.stderr:2024-06-03T09:06:27.161+0000 7f0fe9ffb640 10 monclient(hunting): dump: 2024-06-03T09:06:27.166 INFO:teuthology.orchestra.run.smithi088.stderr:epoch 1 2024-06-03T09:06:27.166 INFO:teuthology.orchestra.run.smithi088.stderr:fsid ec3c3fdd-3ac6-4d7d-9412-683130ed8b04 2024-06-03T09:06:27.166 INFO:teuthology.orchestra.run.smithi088.stderr:last_changed 2024-06-03T08:34:31.885125+0000 2024-06-03T09:06:27.166 INFO:teuthology.orchestra.run.smithi088.stderr:created 2024-06-03T08:34:31.885125+0000 2024-06-03T09:06:27.166 INFO:teuthology.orchestra.run.smithi088.stderr:min_mon_release 18 (reef) 2024-06-03T09:06:27.166 INFO:teuthology.orchestra.run.smithi088.stderr:election_strategy: 3 2024-06-03T09:06:27.166 INFO:teuthology.orchestra.run.smithi088.stderr:0: v1:172.21.15.88:6789/0 mon.a 2024-06-03T09:06:27.166 INFO:teuthology.orchestra.run.smithi088.stderr:1: v1:172.21.15.192:6789/0 mon.b 2024-06-03T09:06:27.166 INFO:teuthology.orchestra.run.smithi088.stderr:2: v1:172.21.15.88:6790/0 mon.c 2024-06-03T09:06:27.166 INFO:teuthology.orchestra.run.smithi088.stderr: 2024-06-03T09:06:27.168 INFO:teuthology.orchestra.run.smithi088.stderr:2024-06-03T09:06:27.164+0000 7f0fe9ffb640 1 -- 172.21.15.88:0/4140293942 <== mon.1 v1:172.21.15.192:6789/0 4 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 580+0+0 (unknown 2218228871 0 0) 0x7f0fec1d5360 con 0x7f0fec06fd50 2024-06-03T09:06:27.168 INFO:teuthology.orchestra.run.smithi088.stderr:2024-06-03T09:06:27.164+0000 7f0fe9ffb640 1 -- 172.21.15.88:0/4140293942 >> v1:172.21.15.88:6790/0 conn(0x7f0fec0ad840 legacy=0x7f0fec12cf20 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-06-03T09:06:27.168 INFO:teuthology.orchestra.run.smithi088.stderr:2024-06-03T09:06:27.164+0000 7f0fe9ffb640 1 -- 172.21.15.88:0/4140293942 >> v1:172.21.15.88:6789/0 conn(0x7f0fec06f970 legacy=0x7f0fec1d4c50 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-06-03T09:06:27.168 INFO:teuthology.orchestra.run.smithi088.stderr:2024-06-03T09:06:27.164+0000 7f0fe9ffb640 10 monclient: _finish_hunting 0 2024-06-03T09:06:27.168 INFO:teuthology.orchestra.run.smithi088.stderr:2024-06-03T09:06:27.164+0000 7f0fe9ffb640 1 monclient: found mon.b 2024-06-03T09:06:27.168 INFO:teuthology.orchestra.run.smithi088.stderr:2024-06-03T09:06:27.164+0000 7f0fe9ffb640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-06-03T09:06:27.168 INFO:teuthology.orchestra.run.smithi088.stderr:2024-06-03T09:06:27.164+0000 7f0fe9ffb640 10 monclient: _send_mon_message to mon.b at v1:172.21.15.192:6789/0 2024-06-03T09:06:27.168 INFO:teuthology.orchestra.run.smithi088.stderr:2024-06-03T09:06:27.164+0000 7f0fe9ffb640 1 -- 172.21.15.88:0/4140293942 --> v1:172.21.15.192:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f0fec12ffc0 con 0x7f0fec06fd50 2024-06-03T09:06:27.168 INFO:teuthology.orchestra.run.smithi088.stderr:2024-06-03T09:06:27.164+0000 7f0fe9ffb640 10 monclient: _finish_auth 0 2024-06-03T09:06:27.168 INFO:teuthology.orchestra.run.smithi088.stderr:2024-06-03T09:06:27.164+0000 7f0fe9ffb640 10 monclient: _check_auth_tickets 2024-06-03T09:06:27.168 INFO:teuthology.orchestra.run.smithi088.stderr:2024-06-03T09:06:27.164+0000 7f0fe9ffb640 20 monclient: _check_auth_rotating not needed by client.admin 2024-06-03T09:06:27.168 INFO:teuthology.orchestra.run.smithi088.stderr:2024-06-03T09:06:27.164+0000 7f0ff3516640 5 monclient: authenticate success, global_id 30196 2024-06-03T09:06:27.169 INFO:teuthology.orchestra.run.smithi088.stderr:2024-06-03T09:06:27.164+0000 7f0ff3516640 10 monclient: _renew_subs 2024-06-03T09:06:27.169 INFO:teuthology.orchestra.run.smithi088.stderr:2024-06-03T09:06:27.164+0000 7f0ff3516640 10 monclient: _send_mon_message to mon.b at v1:172.21.15.192:6789/0 2024-06-03T09:06:27.169 INFO:teuthology.orchestra.run.smithi088.stderr:2024-06-03T09:06:27.164+0000 7f0ff3516640 1 -- 172.21.15.88:0/4140293942 --> v1:172.21.15.192:6789/0 -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f0fec12ef40 con 0x7f0fec06fd50 2024-06-03T09:06:27.169 INFO:teuthology.orchestra.run.smithi088.stderr:2024-06-03T09:06:27.164+0000 7f0ff3516640 10 client.30196.objecter _maybe_request_map subscribing (onetime) to next osd map 2024-06-03T09:06:27.169 INFO:teuthology.orchestra.run.smithi088.stderr:2024-06-03T09:06:27.164+0000 7f0ff3516640 10 monclient: _renew_subs 2024-06-03T09:06:27.169 INFO:teuthology.orchestra.run.smithi088.stderr:2024-06-03T09:06:27.164+0000 7f0ff3516640 10 monclient: _send_mon_message to mon.b at v1:172.21.15.192:6789/0 2024-06-03T09:06:27.169 INFO:teuthology.orchestra.run.smithi088.stderr:2024-06-03T09:06:27.164+0000 7f0ff3516640 1 -- 172.21.15.88:0/4140293942 --> v1:172.21.15.192:6789/0 -- mon_subscribe({osdmap=0}) v3 -- 0x7f0fec12d810 con 0x7f0fec06fd50 2024-06-03T09:06:27.169 INFO:teuthology.orchestra.run.smithi088.stderr:2024-06-03T09:06:27.164+0000 7f0ff3516640 1 librados: init done 2024-06-03T09:06:27.169 INFO:teuthology.orchestra.run.smithi088.stderr:2024-06-03T09:06:27.165+0000 7f0ff3516640 10 start_mon_command cmd=[{"prefix": "get_command_descriptions"}] 2024-06-03T09:06:27.169 INFO:teuthology.orchestra.run.smithi088.stderr:2024-06-03T09:06:27.165+0000 7f0ff3516640 10 monclient: _send_command 1 [{"prefix": "get_command_descriptions"}] 2024-06-03T09:06:27.169 INFO:teuthology.orchestra.run.smithi088.stderr:2024-06-03T09:06:27.165+0000 7f0ff3516640 10 monclient: _send_mon_message to mon.b at v1:172.21.15.192:6789/0 2024-06-03T09:06:27.169 INFO:teuthology.orchestra.run.smithi088.stderr:2024-06-03T09:06:27.165+0000 7f0ff3516640 1 -- 172.21.15.88:0/4140293942 --> v1:172.21.15.192:6789/0 -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f0fec12dac0 con 0x7f0fec06fd50 2024-06-03T09:06:27.379 INFO:teuthology.orchestra.run.smithi088.stderr:2024-06-03T09:06:27.379+0000 7f0fe9ffb640 1 -- 172.21.15.88:0/4140293942 <== mon.1 v1:172.21.15.192:6789/0 5 ==== config(0 keys) v1 ==== 4+0+0 (unknown 0 0 0) 0x7f0fe00038a0 con 0x7f0fec06fd50 2024-06-03T09:06:27.379 INFO:teuthology.orchestra.run.smithi088.stderr:2024-06-03T09:06:27.379+0000 7f0fe9ffb640 10 monclient: handle_config config(0 keys) v1 2024-06-03T09:06:27.379 INFO:teuthology.orchestra.run.smithi088.stderr:2024-06-03T09:06:27.379+0000 7f0feb7fe640 4 set_mon_vals no callback set 2024-06-03T09:06:27.417 INFO:teuthology.orchestra.run.smithi088.stderr:2024-06-03T09:06:27.417+0000 7f0fe9ffb640 1 -- 172.21.15.88:0/4140293942 <== mon.1 v1:172.21.15.192:6789/0 6 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 2234374065 0 0) 0x7f0fe0003bb0 con 0x7f0fec06fd50 2024-06-03T09:06:27.417 INFO:teuthology.orchestra.run.smithi088.stderr:2024-06-03T09:06:27.417+0000 7f0fe9ffb640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-06-03T09:06:27.417 INFO:teuthology.orchestra.run.smithi088.stderr:2024-06-03T09:06:27.417+0000 7f0fe9ffb640 10 monclient: got monmap 1 from mon.b (according to old e1) 2024-06-03T09:06:27.417 INFO:teuthology.orchestra.run.smithi088.stderr:2024-06-03T09:06:27.417+0000 7f0fe9ffb640 10 monclient: dump: 2024-06-03T09:06:27.417 INFO:teuthology.orchestra.run.smithi088.stderr:epoch 1 2024-06-03T09:06:27.417 INFO:teuthology.orchestra.run.smithi088.stderr:fsid ec3c3fdd-3ac6-4d7d-9412-683130ed8b04 2024-06-03T09:06:27.418 INFO:teuthology.orchestra.run.smithi088.stderr:last_changed 2024-06-03T08:34:31.885125+0000 2024-06-03T09:06:27.418 INFO:teuthology.orchestra.run.smithi088.stderr:created 2024-06-03T08:34:31.885125+0000 2024-06-03T09:06:27.418 INFO:teuthology.orchestra.run.smithi088.stderr:min_mon_release 18 (reef) 2024-06-03T09:06:27.418 INFO:teuthology.orchestra.run.smithi088.stderr:election_strategy: 3 2024-06-03T09:06:27.418 INFO:teuthology.orchestra.run.smithi088.stderr:0: v1:172.21.15.88:6789/0 mon.a 2024-06-03T09:06:27.418 INFO:teuthology.orchestra.run.smithi088.stderr:1: v1:172.21.15.192:6789/0 mon.b 2024-06-03T09:06:27.418 INFO:teuthology.orchestra.run.smithi088.stderr:2: v1:172.21.15.88:6790/0 mon.c 2024-06-03T09:06:27.418 INFO:teuthology.orchestra.run.smithi088.stderr: 2024-06-03T09:06:27.419 INFO:teuthology.orchestra.run.smithi088.stderr:2024-06-03T09:06:27.418+0000 7f0fe9ffb640 1 -- 172.21.15.88:0/4140293942 <== mon.1 v1:172.21.15.192:6789/0 7 ==== mgrmap(e 5) v1 ==== 89110+0+0 (unknown 873533973 0 0) 0x7f0fe0019930 con 0x7f0fec06fd50 2024-06-03T09:06:27.419 INFO:teuthology.orchestra.run.smithi088.stderr:2024-06-03T09:06:27.418+0000 7f0ff12b2640 10 monclient: get_auth_request con 0x7f0fd806d2a0 auth_method 0 2024-06-03T09:06:27.419 INFO:teuthology.orchestra.run.smithi088.stderr:2024-06-03T09:06:27.419+0000 7f0fe9ffb640 10 client.30196.objecter ms_handle_connect 0x7f0fd806d2a0 2024-06-03T09:06:27.421 INFO:teuthology.orchestra.run.smithi088.stderr:2024-06-03T09:06:27.421+0000 7f0fe9ffb640 1 -- 172.21.15.88:0/4140293942 <== mon.1 v1:172.21.15.192:6789/0 8 ==== osd_map(1010..1010 src has 622..1010) v4 ==== 5229+0+0 (unknown 1105883993 0 0) 0x7f0fe00011a0 con 0x7f0fec06fd50 2024-06-03T09:06:27.421 INFO:teuthology.orchestra.run.smithi088.stderr:2024-06-03T09:06:27.421+0000 7f0fe9ffb640 10 client.30196.objecter ms_dispatch 0x7f0fec000b90 osd_map(1010..1010 src has 622..1010) v4 2024-06-03T09:06:27.421 INFO:teuthology.orchestra.run.smithi088.stderr:2024-06-03T09:06:27.421+0000 7f0fe9ffb640 3 client.30196.objecter handle_osd_map got epochs [1010,1010] > 0 2024-06-03T09:06:27.421 INFO:teuthology.orchestra.run.smithi088.stderr:2024-06-03T09:06:27.421+0000 7f0fe9ffb640 3 client.30196.objecter handle_osd_map decoding full epoch 1010 2024-06-03T09:06:27.421 INFO:teuthology.orchestra.run.smithi088.stderr:2024-06-03T09:06:27.421+0000 7f0fe9ffb640 20 client.30196.objecter dump_active .. 0 homeless 2024-06-03T09:06:27.423 INFO:teuthology.orchestra.run.smithi088.stderr:2024-06-03T09:06:27.424+0000 7f0fe9ffb640 1 -- 172.21.15.88:0/4140293942 <== mon.1 v1:172.21.15.192:6789/0 9 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+182498 (unknown 1092875540 0 3016760700) 0x7f0fe004e960 con 0x7f0fec06fd50 2024-06-03T09:06:27.423 INFO:teuthology.orchestra.run.smithi088.stderr:2024-06-03T09:06:27.424+0000 7f0fe9ffb640 10 monclient: handle_mon_command_ack 1 [{"prefix": "get_command_descriptions"}] 2024-06-03T09:06:27.423 INFO:teuthology.orchestra.run.smithi088.stderr:2024-06-03T09:06:27.424+0000 7f0fe9ffb640 10 monclient: _finish_command 1 = system:0 2024-06-03T09:06:27.695 INFO:teuthology.orchestra.run.smithi088.stderr:2024-06-03T09:06:27.695+0000 7f0ff3516640 10 start_mon_command cmd=[{"prefix": "status", "format": "json"}] 2024-06-03T09:06:27.695 INFO:teuthology.orchestra.run.smithi088.stderr:2024-06-03T09:06:27.695+0000 7f0ff3516640 10 monclient: _send_command 2 [{"prefix": "status", "format": "json"}] 2024-06-03T09:06:27.695 INFO:teuthology.orchestra.run.smithi088.stderr:2024-06-03T09:06:27.695+0000 7f0ff3516640 10 monclient: _send_mon_message to mon.b at v1:172.21.15.192:6789/0 2024-06-03T09:06:27.695 INFO:teuthology.orchestra.run.smithi088.stderr:2024-06-03T09:06:27.695+0000 7f0ff3516640 1 -- 172.21.15.88:0/4140293942 --> v1:172.21.15.192:6789/0 -- mon_command({"prefix": "status", "format": "json"} v 0) v1 -- 0x7f0fec04ea60 con 0x7f0fec06fd50 2024-06-03T09:06:27.701 INFO:teuthology.orchestra.run.smithi088.stderr:2024-06-03T09:06:27.702+0000 7f0fe9ffb640 1 -- 172.21.15.88:0/4140293942 <== mon.1 v1:172.21.15.192:6789/0 10 ==== mon_command_ack([{"prefix": "status", "format": "json"}]=0 v0) v1 ==== 72+0+1431 (unknown 3226119948 0 4104475570) 0x7f0fe0050a60 con 0x7f0fec06fd50 2024-06-03T09:06:27.701 INFO:teuthology.orchestra.run.smithi088.stderr:2024-06-03T09:06:27.702+0000 7f0fe9ffb640 10 monclient: handle_mon_command_ack 2 [{"prefix": "status", "format": "json"}] 2024-06-03T09:06:27.701 INFO:teuthology.orchestra.run.smithi088.stderr:2024-06-03T09:06:27.702+0000 7f0fe9ffb640 10 monclient: _finish_command 2 = system:0 2024-06-03T09:06:27.702 INFO:teuthology.orchestra.run.smithi088.stdout: 2024-06-03T09:06:27.702 INFO:teuthology.orchestra.run.smithi088.stdout:{"fsid":"ec3c3fdd-3ac6-4d7d-9412-683130ed8b04","health":{"status":"HEALTH_WARN","checks":{"OSD_DOWN":{"severity":"HEALTH_WARN","summary":{"message":"1 osds down","count":1},"muted":false},"PG_DEGRADED":{"severity":"HEALTH_WARN","summary":{"message":"Degraded data redundancy: 1/6 objects degraded (16.667%), 1 pg degraded","count":1},"muted":false},"POOL_APP_NOT_ENABLED":{"severity":"HEALTH_WARN","summary":{"message":"1 pool(s) do not have an application enabled","count":1},"muted":false}},"mutes":[]},"election_epoch":10,"quorum":[0,1,2],"quorum_names":["a","b","c"],"quorum_age":1861,"monmap":{"epoch":1,"min_mon_release_name":"reef","num_mons":3},"osdmap":{"epoch":1010,"num_osds":8,"num_up_osds":7,"osd_up_since":1717405562,"num_in_osds":8,"osd_in_since":1717403732,"num_remapped_pgs":0},"pgmap":{"pgs_by_state":[{"state_name":"active+clean","count":27},{"state_name":"active+undersized","count":13},{"state_name":"active+undersized+degraded","count":1}],"num_pgs":41,"num_pools":3,"num_objects":3,"data_bytes":459285,"bytes_used":506417152,"bytes_avail":772587696128,"bytes_total":773094113280,"degraded_objects":1,"degraded_total":6,"degraded_ratio":0.16666666666666663},"fsmap":{"epoch":1,"by_rank":[],"up:standby":0},"mgrmap":{"available":true,"num_standbys":1,"modules":["iostat","nfs","restful"],"services":{}},"servicemap":{"epoch":23,"modified":"2024-06-03T08:53:48.910215+0000","services":{}},"progress_events":{}} 2024-06-03T09:06:27.704 INFO:teuthology.orchestra.run.smithi088.stderr:2024-06-03T09:06:27.703+0000 7f0ff3516640 10 librados: watch_flush enter 2024-06-03T09:06:27.704 INFO:teuthology.orchestra.run.smithi088.stderr:2024-06-03T09:06:27.703+0000 7f0ff3516640 10 librados: watch_flush exit 2024-06-03T09:06:27.704 INFO:teuthology.orchestra.run.smithi088.stderr:2024-06-03T09:06:27.703+0000 7f0ff3516640 20 client.30196.objecter shutdown clearing up homeless session... 2024-06-03T09:06:27.704 INFO:teuthology.orchestra.run.smithi088.stderr:2024-06-03T09:06:27.703+0000 7f0ff3516640 10 client.30196.objecter successfully canceled tick 2024-06-03T09:06:27.704 INFO:teuthology.orchestra.run.smithi088.stderr:2024-06-03T09:06:27.703+0000 7f0ff3516640 1 -- 172.21.15.88:0/4140293942 >> v1:172.21.15.192:6800/34229 conn(0x7f0fd806d2a0 legacy=0x7f0fd806f760 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-06-03T09:06:27.704 INFO:teuthology.orchestra.run.smithi088.stderr:2024-06-03T09:06:27.703+0000 7f0ff3516640 10 monclient: shutdown 2024-06-03T09:06:27.704 INFO:teuthology.orchestra.run.smithi088.stderr:2024-06-03T09:06:27.703+0000 7f0ff3516640 20 monclient: shutdown discarding 0 pending message(s) 2024-06-03T09:06:27.704 INFO:teuthology.orchestra.run.smithi088.stderr:2024-06-03T09:06:27.703+0000 7f0ff3516640 1 -- 172.21.15.88:0/4140293942 >> v1:172.21.15.192:6789/0 conn(0x7f0fec06fd50 legacy=0x7f0fec12f720 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-06-03T09:06:27.705 INFO:teuthology.orchestra.run.smithi088.stderr:2024-06-03T09:06:27.703+0000 7f0ff3516640 1 -- 172.21.15.88:0/4140293942 shutdown_connections 2024-06-03T09:06:27.705 INFO:teuthology.orchestra.run.smithi088.stderr:2024-06-03T09:06:27.703+0000 7f0ff3516640 1 -- 172.21.15.88:0/4140293942 >> 172.21.15.88:0/4140293942 conn(0x7f0fec0b2e40 msgr2=0x7f0fec13c720 unknown :-1 s=STATE_NONE l=0).mark_down 2024-06-03T09:06:27.705 INFO:teuthology.orchestra.run.smithi088.stderr:2024-06-03T09:06:27.703+0000 7f0ff3516640 1 -- 172.21.15.88:0/4140293942 shutdown_connections 2024-06-03T09:06:27.705 INFO:teuthology.orchestra.run.smithi088.stderr:2024-06-03T09:06:27.703+0000 7f0ff3516640 1 -- 172.21.15.88:0/4140293942 wait complete. 2024-06-03T09:06:27.705 INFO:teuthology.orchestra.run.smithi088.stderr:2024-06-03T09:06:27.703+0000 7f0ff3516640 1 librados: shutdown 2024-06-03T09:06:27.714 INFO:tasks.ceph.ceph_manager.ceph:no progress seen, keeping timeout for now 2024-06-03T09:06:27.714 DEBUG:teuthology.orchestra.run.smithi088:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph pg dump --format=json 2024-06-03T09:06:27.813 INFO:teuthology.orchestra.run.smithi088.stderr:2024-06-03T09:06:27.813+0000 7f88e35c8640 10 monclient: get_monmap_and_config 2024-06-03T09:06:27.813 INFO:teuthology.orchestra.run.smithi088.stderr:2024-06-03T09:06:27.813+0000 7f88e35c8640 10 monclient: build_initial_monmap 2024-06-03T09:06:27.813 INFO:teuthology.orchestra.run.smithi088.stderr:2024-06-03T09:06:27.813+0000 7f88e35c8640 10 monclient: monmap: 2024-06-03T09:06:27.813 INFO:teuthology.orchestra.run.smithi088.stderr:epoch 0 2024-06-03T09:06:27.813 INFO:teuthology.orchestra.run.smithi088.stderr:fsid ec3c3fdd-3ac6-4d7d-9412-683130ed8b04 2024-06-03T09:06:27.813 INFO:teuthology.orchestra.run.smithi088.stderr:last_changed 2024-06-03T09:06:27.814216+0000 2024-06-03T09:06:27.813 INFO:teuthology.orchestra.run.smithi088.stderr:created 2024-06-03T09:06:27.814216+0000 2024-06-03T09:06:27.814 INFO:teuthology.orchestra.run.smithi088.stderr:min_mon_release 0 (unknown) 2024-06-03T09:06:27.814 INFO:teuthology.orchestra.run.smithi088.stderr:election_strategy: 3 2024-06-03T09:06:27.814 INFO:teuthology.orchestra.run.smithi088.stderr:0: v1:172.21.15.88:6789/0 mon.noname-a 2024-06-03T09:06:27.814 INFO:teuthology.orchestra.run.smithi088.stderr:1: v1:172.21.15.192:6789/0 mon.noname-c 2024-06-03T09:06:27.814 INFO:teuthology.orchestra.run.smithi088.stderr:2: v1:172.21.15.88:6790/0 mon.noname-b 2024-06-03T09:06:27.814 INFO:teuthology.orchestra.run.smithi088.stderr: 2024-06-03T09:06:27.815 INFO:teuthology.orchestra.run.smithi088.stderr:2024-06-03T09:06:27.813+0000 7f88e35c8640 1 Processor -- start 2024-06-03T09:06:27.815 INFO:teuthology.orchestra.run.smithi088.stderr:2024-06-03T09:06:27.814+0000 7f88e35c8640 1 -- start start 2024-06-03T09:06:27.815 INFO:teuthology.orchestra.run.smithi088.stderr:2024-06-03T09:06:27.814+0000 7f88e35c8640 10 monclient: init 2024-06-03T09:06:27.815 INFO:teuthology.orchestra.run.smithi088.stderr:2024-06-03T09:06:27.814+0000 7f88e35c8640 10 monclient: _reopen_session rank -1 2024-06-03T09:06:27.816 INFO:teuthology.orchestra.run.smithi088.stderr:2024-06-03T09:06:27.814+0000 7f88e35c8640 10 monclient: _add_conns ranks=[1,0,2] 2024-06-03T09:06:27.816 INFO:teuthology.orchestra.run.smithi088.stderr:2024-06-03T09:06:27.814+0000 7f88e35c8640 10 monclient(hunting): picked mon.noname-c con 0x7f88dc06e050 addr v1:172.21.15.192:6789/0 2024-06-03T09:06:27.816 INFO:teuthology.orchestra.run.smithi088.stderr:2024-06-03T09:06:27.814+0000 7f88e35c8640 10 monclient(hunting): picked mon.noname-a con 0x7f88dc06e430 addr v1:172.21.15.88:6789/0 2024-06-03T09:06:27.816 INFO:teuthology.orchestra.run.smithi088.stderr:2024-06-03T09:06:27.814+0000 7f88e35c8640 10 monclient(hunting): picked mon.noname-b con 0x7f88dc0717c0 addr v1:172.21.15.88:6790/0 2024-06-03T09:06:27.816 INFO:teuthology.orchestra.run.smithi088.stderr:2024-06-03T09:06:27.814+0000 7f88e35c8640 1 -- --> v1:172.21.15.88:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f88dc070c10 con 0x7f88dc06e430 2024-06-03T09:06:27.816 INFO:teuthology.orchestra.run.smithi088.stderr:2024-06-03T09:06:27.814+0000 7f88e35c8640 1 -- --> v1:172.21.15.192:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f88dc070230 con 0x7f88dc06e050 2024-06-03T09:06:27.816 INFO:teuthology.orchestra.run.smithi088.stderr:2024-06-03T09:06:27.814+0000 7f88e35c8640 1 -- --> v1:172.21.15.88:6790/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f88dc070510 con 0x7f88dc0717c0 2024-06-03T09:06:27.816 INFO:teuthology.orchestra.run.smithi088.stderr:2024-06-03T09:06:27.814+0000 7f88e35c8640 10 monclient(hunting): _renew_subs 2024-06-03T09:06:27.816 INFO:teuthology.orchestra.run.smithi088.stderr:2024-06-03T09:06:27.814+0000 7f88e35c8640 10 monclient(hunting): authenticate will time out at 3043.624573s 2024-06-03T09:06:27.816 INFO:teuthology.orchestra.run.smithi088.stderr:2024-06-03T09:06:27.815+0000 7f88e0b63640 1 --1- >> v1:172.21.15.88:6789/0 conn(0x7f88dc06e430 0x7f88dc0710b0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.88:6789/0 says I am v1:172.21.15.88:41546/0 (socket says 172.21.15.88:41546) 2024-06-03T09:06:27.816 INFO:teuthology.orchestra.run.smithi088.stderr:2024-06-03T09:06:27.815+0000 7f88e0b63640 1 -- 172.21.15.88:0/2010293054 learned_addr learned my addr 172.21.15.88:0/2010293054 (peer_addr_for_me v1:172.21.15.88:0/0) 2024-06-03T09:06:27.817 INFO:teuthology.orchestra.run.smithi088.stderr:2024-06-03T09:06:27.817+0000 7f88d3fff640 1 -- 172.21.15.88:0/2010293054 <== mon.2 v1:172.21.15.88:6790/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 2070974290 0 0) 0x7f88dc070510 con 0x7f88dc0717c0 2024-06-03T09:06:27.817 INFO:teuthology.orchestra.run.smithi088.stderr:2024-06-03T09:06:27.817+0000 7f88d3fff640 10 monclient(hunting): _negotiate 2024-06-03T09:06:27.817 INFO:teuthology.orchestra.run.smithi088.stderr:2024-06-03T09:06:27.817+0000 7f88d3fff640 10 monclient(hunting): _init_auth method 2 2024-06-03T09:06:27.818 INFO:teuthology.orchestra.run.smithi088.stderr:2024-06-03T09:06:27.817+0000 7f88d3fff640 10 monclient(hunting): _init_auth creating new auth 2024-06-03T09:06:27.818 INFO:teuthology.orchestra.run.smithi088.stderr:2024-06-03T09:06:27.817+0000 7f88d3fff640 10 monclient(hunting): my global_id is 30122 2024-06-03T09:06:27.818 INFO:teuthology.orchestra.run.smithi088.stderr:2024-06-03T09:06:27.817+0000 7f88d3fff640 1 -- 172.21.15.88:0/2010293054 --> v1:172.21.15.88:6790/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f88c8003520 con 0x7f88dc0717c0 2024-06-03T09:06:27.818 INFO:teuthology.orchestra.run.smithi088.stderr:2024-06-03T09:06:27.817+0000 7f88d3fff640 1 -- 172.21.15.88:0/2010293054 <== mon.1 v1:172.21.15.192:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 3261728386 0 0) 0x7f88dc070230 con 0x7f88dc06e050 2024-06-03T09:06:27.818 INFO:teuthology.orchestra.run.smithi088.stderr:2024-06-03T09:06:27.817+0000 7f88d3fff640 10 monclient(hunting): _negotiate 2024-06-03T09:06:27.818 INFO:teuthology.orchestra.run.smithi088.stderr:2024-06-03T09:06:27.817+0000 7f88d3fff640 10 monclient(hunting): _init_auth method 2 2024-06-03T09:06:27.818 INFO:teuthology.orchestra.run.smithi088.stderr:2024-06-03T09:06:27.817+0000 7f88d3fff640 10 monclient(hunting): _init_auth creating new auth 2024-06-03T09:06:27.818 INFO:teuthology.orchestra.run.smithi088.stderr:2024-06-03T09:06:27.817+0000 7f88d3fff640 10 monclient(hunting): my global_id is 30199 2024-06-03T09:06:27.818 INFO:teuthology.orchestra.run.smithi088.stderr:2024-06-03T09:06:27.817+0000 7f88d3fff640 1 -- 172.21.15.88:0/2010293054 --> v1:172.21.15.192:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f88dc070510 con 0x7f88dc06e050 2024-06-03T09:06:27.818 INFO:teuthology.orchestra.run.smithi088.stderr:2024-06-03T09:06:27.817+0000 7f88d3fff640 1 -- 172.21.15.88:0/2010293054 <== mon.0 v1:172.21.15.88:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 1117003152 0 0) 0x7f88dc070c10 con 0x7f88dc06e430 2024-06-03T09:06:27.818 INFO:teuthology.orchestra.run.smithi088.stderr:2024-06-03T09:06:27.817+0000 7f88d3fff640 10 monclient(hunting): _negotiate 2024-06-03T09:06:27.818 INFO:teuthology.orchestra.run.smithi088.stderr:2024-06-03T09:06:27.817+0000 7f88d3fff640 10 monclient(hunting): _init_auth method 2 2024-06-03T09:06:27.818 INFO:teuthology.orchestra.run.smithi088.stderr:2024-06-03T09:06:27.817+0000 7f88d3fff640 10 monclient(hunting): _init_auth creating new auth 2024-06-03T09:06:27.819 INFO:teuthology.orchestra.run.smithi088.stderr:2024-06-03T09:06:27.817+0000 7f88d3fff640 10 monclient(hunting): my global_id is 30618 2024-06-03T09:06:27.819 INFO:teuthology.orchestra.run.smithi088.stderr:2024-06-03T09:06:27.817+0000 7f88d3fff640 1 -- 172.21.15.88:0/2010293054 --> v1:172.21.15.88:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f88dc070230 con 0x7f88dc06e430 2024-06-03T09:06:27.821 INFO:teuthology.orchestra.run.smithi088.stderr:2024-06-03T09:06:27.820+0000 7f88d3fff640 1 -- 172.21.15.88:0/2010293054 <== mon.1 v1:172.21.15.192:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 214+0+0 (unknown 1470801544 0 0) 0x7f88dc070510 con 0x7f88dc06e050 2024-06-03T09:06:27.821 INFO:teuthology.orchestra.run.smithi088.stderr:2024-06-03T09:06:27.820+0000 7f88d3fff640 1 -- 172.21.15.88:0/2010293054 >> v1:172.21.15.88:6790/0 conn(0x7f88dc0717c0 legacy=0x7f88dc149240 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-06-03T09:06:27.821 INFO:teuthology.orchestra.run.smithi088.stderr:2024-06-03T09:06:27.820+0000 7f88d3fff640 1 -- 172.21.15.88:0/2010293054 >> v1:172.21.15.88:6789/0 conn(0x7f88dc06e430 legacy=0x7f88dc0710b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-06-03T09:06:27.821 INFO:teuthology.orchestra.run.smithi088.stderr:2024-06-03T09:06:27.820+0000 7f88d3fff640 10 monclient: _finish_hunting 0 2024-06-03T09:06:27.821 INFO:teuthology.orchestra.run.smithi088.stderr:2024-06-03T09:06:27.820+0000 7f88d3fff640 1 monclient: found mon.noname-c 2024-06-03T09:06:27.821 INFO:teuthology.orchestra.run.smithi088.stderr:2024-06-03T09:06:27.820+0000 7f88d3fff640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-06-03T09:06:27.821 INFO:teuthology.orchestra.run.smithi088.stderr:2024-06-03T09:06:27.820+0000 7f88d3fff640 10 monclient: _send_mon_message to mon.noname-c at v1:172.21.15.192:6789/0 2024-06-03T09:06:27.821 INFO:teuthology.orchestra.run.smithi088.stderr:2024-06-03T09:06:27.820+0000 7f88d3fff640 1 -- 172.21.15.88:0/2010293054 --> v1:172.21.15.192:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f88dc0707f0 con 0x7f88dc06e050 2024-06-03T09:06:27.821 INFO:teuthology.orchestra.run.smithi088.stderr:2024-06-03T09:06:27.820+0000 7f88d3fff640 10 monclient: _finish_auth 0 2024-06-03T09:06:27.821 INFO:teuthology.orchestra.run.smithi088.stderr:2024-06-03T09:06:27.820+0000 7f88d3fff640 10 monclient: _check_auth_tickets 2024-06-03T09:06:27.821 INFO:teuthology.orchestra.run.smithi088.stderr:2024-06-03T09:06:27.820+0000 7f88d3fff640 20 monclient: _check_auth_rotating not needed by client.admin 2024-06-03T09:06:27.822 INFO:teuthology.orchestra.run.smithi088.stderr:2024-06-03T09:06:27.820+0000 7f88d3fff640 1 -- 172.21.15.88:0/2010293054 <== mon.1 v1:172.21.15.192:6789/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 2234374065 0 0) 0x7f88cc0015f0 con 0x7f88dc06e050 2024-06-03T09:06:27.822 INFO:teuthology.orchestra.run.smithi088.stderr:2024-06-03T09:06:27.820+0000 7f88d3fff640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-06-03T09:06:27.822 INFO:teuthology.orchestra.run.smithi088.stderr:2024-06-03T09:06:27.820+0000 7f88d3fff640 10 monclient: got monmap 1 from mon.noname-c (according to old e1) 2024-06-03T09:06:27.822 INFO:teuthology.orchestra.run.smithi088.stderr:2024-06-03T09:06:27.820+0000 7f88d3fff640 10 monclient: dump: 2024-06-03T09:06:27.822 INFO:teuthology.orchestra.run.smithi088.stderr:epoch 1 2024-06-03T09:06:27.822 INFO:teuthology.orchestra.run.smithi088.stderr:fsid ec3c3fdd-3ac6-4d7d-9412-683130ed8b04 2024-06-03T09:06:27.822 INFO:teuthology.orchestra.run.smithi088.stderr:last_changed 2024-06-03T08:34:31.885125+0000 2024-06-03T09:06:27.822 INFO:teuthology.orchestra.run.smithi088.stderr:created 2024-06-03T08:34:31.885125+0000 2024-06-03T09:06:27.822 INFO:teuthology.orchestra.run.smithi088.stderr:min_mon_release 18 (reef) 2024-06-03T09:06:27.823 INFO:teuthology.orchestra.run.smithi088.stderr:election_strategy: 3 2024-06-03T09:06:27.823 INFO:teuthology.orchestra.run.smithi088.stderr:0: v1:172.21.15.88:6789/0 mon.a 2024-06-03T09:06:27.823 INFO:teuthology.orchestra.run.smithi088.stderr:1: v1:172.21.15.192:6789/0 mon.b 2024-06-03T09:06:27.823 INFO:teuthology.orchestra.run.smithi088.stderr:2: v1:172.21.15.88:6790/0 mon.c 2024-06-03T09:06:27.823 INFO:teuthology.orchestra.run.smithi088.stderr: 2024-06-03T09:06:27.823 INFO:teuthology.orchestra.run.smithi088.stderr:2024-06-03T09:06:27.820+0000 7f88e35c8640 5 monclient: authenticate success, global_id 30199 2024-06-03T09:06:27.823 INFO:teuthology.orchestra.run.smithi088.stderr:2024-06-03T09:06:27.820+0000 7f88e35c8640 20 monclient: get_monmap_and_config waiting for monmap|config 2024-06-03T09:06:27.826 INFO:teuthology.orchestra.run.smithi088.stderr:2024-06-03T09:06:27.822+0000 7f88d3fff640 1 -- 172.21.15.88:0/2010293054 <== mon.1 v1:172.21.15.192:6789/0 4 ==== config(0 keys) v1 ==== 4+0+0 (unknown 0 0 0) 0x7f88cc001810 con 0x7f88dc06e050 2024-06-03T09:06:27.827 INFO:teuthology.orchestra.run.smithi088.stderr:2024-06-03T09:06:27.822+0000 7f88d3fff640 10 monclient: handle_config config(0 keys) v1 2024-06-03T09:06:27.827 INFO:teuthology.orchestra.run.smithi088.stderr:2024-06-03T09:06:27.822+0000 7f88d3fff640 1 -- 172.21.15.88:0/2010293054 <== mon.1 v1:172.21.15.192:6789/0 5 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 2234374065 0 0) 0x7f88cc001be0 con 0x7f88dc06e050 2024-06-03T09:06:27.827 INFO:teuthology.orchestra.run.smithi088.stderr:2024-06-03T09:06:27.822+0000 7f88d3fff640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-06-03T09:06:27.827 INFO:teuthology.orchestra.run.smithi088.stderr:2024-06-03T09:06:27.822+0000 7f88d3fff640 10 monclient: got monmap 1 from mon.b (according to old e1) 2024-06-03T09:06:27.827 INFO:teuthology.orchestra.run.smithi088.stderr:2024-06-03T09:06:27.822+0000 7f88d3fff640 10 monclient: dump: 2024-06-03T09:06:27.827 INFO:teuthology.orchestra.run.smithi088.stderr:epoch 1 2024-06-03T09:06:27.827 INFO:teuthology.orchestra.run.smithi088.stderr:fsid ec3c3fdd-3ac6-4d7d-9412-683130ed8b04 2024-06-03T09:06:27.827 INFO:teuthology.orchestra.run.smithi088.stderr:last_changed 2024-06-03T08:34:31.885125+0000 2024-06-03T09:06:27.827 INFO:teuthology.orchestra.run.smithi088.stderr:created 2024-06-03T08:34:31.885125+0000 2024-06-03T09:06:27.827 INFO:teuthology.orchestra.run.smithi088.stderr:min_mon_release 18 (reef) 2024-06-03T09:06:27.827 INFO:teuthology.orchestra.run.smithi088.stderr:election_strategy: 3 2024-06-03T09:06:27.828 INFO:teuthology.orchestra.run.smithi088.stderr:0: v1:172.21.15.88:6789/0 mon.a 2024-06-03T09:06:27.828 INFO:teuthology.orchestra.run.smithi088.stderr:1: v1:172.21.15.192:6789/0 mon.b 2024-06-03T09:06:27.828 INFO:teuthology.orchestra.run.smithi088.stderr:2: v1:172.21.15.88:6790/0 mon.c 2024-06-03T09:06:27.828 INFO:teuthology.orchestra.run.smithi088.stderr: 2024-06-03T09:06:27.828 INFO:teuthology.orchestra.run.smithi088.stderr:2024-06-03T09:06:27.822+0000 7f88e35c8640 10 monclient: get_monmap_and_config success 2024-06-03T09:06:27.828 INFO:teuthology.orchestra.run.smithi088.stderr:2024-06-03T09:06:27.822+0000 7f88e35c8640 4 set_mon_vals no callback set 2024-06-03T09:06:27.828 INFO:teuthology.orchestra.run.smithi088.stderr:2024-06-03T09:06:27.823+0000 7f88e35c8640 10 monclient: shutdown 2024-06-03T09:06:27.829 INFO:teuthology.orchestra.run.smithi088.stderr:2024-06-03T09:06:27.823+0000 7f88e35c8640 20 monclient: shutdown discarding 0 pending message(s) 2024-06-03T09:06:27.829 INFO:teuthology.orchestra.run.smithi088.stderr:2024-06-03T09:06:27.823+0000 7f88e35c8640 1 -- 172.21.15.88:0/2010293054 >> v1:172.21.15.192:6789/0 conn(0x7f88dc06e050 legacy=0x7f88dc13ad00 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-06-03T09:06:27.829 INFO:teuthology.orchestra.run.smithi088.stderr:2024-06-03T09:06:27.823+0000 7f88e35c8640 1 -- 172.21.15.88:0/2010293054 shutdown_connections 2024-06-03T09:06:27.829 INFO:teuthology.orchestra.run.smithi088.stderr:2024-06-03T09:06:27.823+0000 7f88e35c8640 1 -- 172.21.15.88:0/2010293054 >> 172.21.15.88:0/2010293054 conn(0x7f88dc0b2f20 msgr2=0x7f88dc0b3300 unknown :-1 s=STATE_NONE l=0).mark_down 2024-06-03T09:06:27.829 INFO:teuthology.orchestra.run.smithi088.stderr:2024-06-03T09:06:27.823+0000 7f88e35c8640 1 -- 172.21.15.88:0/2010293054 shutdown_connections 2024-06-03T09:06:27.829 INFO:teuthology.orchestra.run.smithi088.stderr:2024-06-03T09:06:27.823+0000 7f88e35c8640 1 -- 172.21.15.88:0/2010293054 wait complete. 2024-06-03T09:06:27.829 INFO:teuthology.orchestra.run.smithi088.stderr:2024-06-03T09:06:27.823+0000 7f88e35c8640 10 monclient: build_initial_monmap 2024-06-03T09:06:27.829 INFO:teuthology.orchestra.run.smithi088.stderr:2024-06-03T09:06:27.823+0000 7f88e35c8640 10 monclient: monmap: 2024-06-03T09:06:27.829 INFO:teuthology.orchestra.run.smithi088.stderr:epoch 0 2024-06-03T09:06:27.829 INFO:teuthology.orchestra.run.smithi088.stderr:fsid 00000000-0000-0000-0000-000000000000 2024-06-03T09:06:27.829 INFO:teuthology.orchestra.run.smithi088.stderr:last_changed 0.000000 2024-06-03T09:06:27.829 INFO:teuthology.orchestra.run.smithi088.stderr:created 0.000000 2024-06-03T09:06:27.829 INFO:teuthology.orchestra.run.smithi088.stderr:min_mon_release 0 (unknown) 2024-06-03T09:06:27.830 INFO:teuthology.orchestra.run.smithi088.stderr:election_strategy: 1 2024-06-03T09:06:27.830 INFO:teuthology.orchestra.run.smithi088.stderr:0: v1:172.21.15.88:6789/0 mon.noname-a 2024-06-03T09:06:27.830 INFO:teuthology.orchestra.run.smithi088.stderr:1: v1:172.21.15.192:6789/0 mon.noname-b 2024-06-03T09:06:27.830 INFO:teuthology.orchestra.run.smithi088.stderr:2: v1:172.21.15.88:6790/0 mon.noname-c 2024-06-03T09:06:27.830 INFO:teuthology.orchestra.run.smithi088.stderr: 2024-06-03T09:06:27.830 INFO:teuthology.orchestra.run.smithi088.stderr:2024-06-03T09:06:27.824+0000 7f88e35c8640 1 librados: starting msgr at 2024-06-03T09:06:27.830 INFO:teuthology.orchestra.run.smithi088.stderr:2024-06-03T09:06:27.824+0000 7f88e35c8640 1 librados: starting objecter 2024-06-03T09:06:27.830 INFO:teuthology.orchestra.run.smithi088.stderr:2024-06-03T09:06:27.824+0000 7f88e35c8640 1 Processor -- start 2024-06-03T09:06:27.830 INFO:teuthology.orchestra.run.smithi088.stderr:2024-06-03T09:06:27.824+0000 7f88e35c8640 1 -- start start 2024-06-03T09:06:27.830 INFO:teuthology.orchestra.run.smithi088.stderr:2024-06-03T09:06:27.824+0000 7f88e35c8640 1 librados: setting wanted keys 2024-06-03T09:06:27.830 INFO:teuthology.orchestra.run.smithi088.stderr:2024-06-03T09:06:27.824+0000 7f88e35c8640 1 librados: calling monclient init 2024-06-03T09:06:27.831 INFO:teuthology.orchestra.run.smithi088.stderr:2024-06-03T09:06:27.824+0000 7f88e35c8640 10 monclient: init 2024-06-03T09:06:27.831 INFO:teuthology.orchestra.run.smithi088.stderr:2024-06-03T09:06:27.824+0000 7f88e35c8640 10 monclient: _reopen_session rank -1 2024-06-03T09:06:27.831 INFO:teuthology.orchestra.run.smithi088.stderr:2024-06-03T09:06:27.824+0000 7f88e35c8640 10 monclient: _add_conns ranks=[1,0,2] 2024-06-03T09:06:27.831 INFO:teuthology.orchestra.run.smithi088.stderr:2024-06-03T09:06:27.824+0000 7f88e35c8640 10 monclient(hunting): picked mon.noname-b con 0x7f88dc06e050 addr v1:172.21.15.192:6789/0 2024-06-03T09:06:27.831 INFO:teuthology.orchestra.run.smithi088.stderr:2024-06-03T09:06:27.824+0000 7f88e35c8640 10 monclient(hunting): picked mon.noname-a con 0x7f88dc0717c0 addr v1:172.21.15.88:6789/0 2024-06-03T09:06:27.831 INFO:teuthology.orchestra.run.smithi088.stderr:2024-06-03T09:06:27.824+0000 7f88e35c8640 10 monclient(hunting): picked mon.noname-c con 0x7f88dc0acad0 addr v1:172.21.15.88:6790/0 2024-06-03T09:06:27.831 INFO:teuthology.orchestra.run.smithi088.stderr:2024-06-03T09:06:27.824+0000 7f88e35c8640 1 -- --> v1:172.21.15.88:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f88dc13dd30 con 0x7f88dc0717c0 2024-06-03T09:06:27.831 INFO:teuthology.orchestra.run.smithi088.stderr:2024-06-03T09:06:27.824+0000 7f88e35c8640 1 -- --> v1:172.21.15.192:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f88dc11b3a0 con 0x7f88dc06e050 2024-06-03T09:06:27.831 INFO:teuthology.orchestra.run.smithi088.stderr:2024-06-03T09:06:27.824+0000 7f88e35c8640 1 -- --> v1:172.21.15.88:6790/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f88dc139260 con 0x7f88dc0acad0 2024-06-03T09:06:27.831 INFO:teuthology.orchestra.run.smithi088.stderr:2024-06-03T09:06:27.824+0000 7f88e35c8640 10 monclient(hunting): _renew_subs 2024-06-03T09:06:27.831 INFO:teuthology.orchestra.run.smithi088.stderr:2024-06-03T09:06:27.824+0000 7f88e35c8640 10 monclient(hunting): authenticate will time out at 3043.634846s 2024-06-03T09:06:27.831 INFO:teuthology.orchestra.run.smithi088.stderr:2024-06-03T09:06:27.825+0000 7f88e0b63640 1 --1- >> v1:172.21.15.88:6789/0 conn(0x7f88dc0717c0 0x7f88dc138b50 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=1).handle_server_banner_and_identify peer v1:172.21.15.88:6789/0 says I am v1:172.21.15.88:41552/0 (socket says 172.21.15.88:41552) 2024-06-03T09:06:27.832 INFO:teuthology.orchestra.run.smithi088.stderr:2024-06-03T09:06:27.825+0000 7f88e0b63640 1 -- 172.21.15.88:0/2567601929 learned_addr learned my addr 172.21.15.88:0/2567601929 (peer_addr_for_me v1:172.21.15.88:0/0) 2024-06-03T09:06:27.832 INFO:teuthology.orchestra.run.smithi088.stderr:2024-06-03T09:06:27.825+0000 7f88d1ffb640 10 client.?.objecter ms_handle_connect 0x7f88dc0717c0 2024-06-03T09:06:27.832 INFO:teuthology.orchestra.run.smithi088.stderr:2024-06-03T09:06:27.825+0000 7f88d1ffb640 10 client.?.objecter resend_mon_ops 2024-06-03T09:06:27.832 INFO:teuthology.orchestra.run.smithi088.stderr:2024-06-03T09:06:27.825+0000 7f88d1ffb640 10 client.?.objecter ms_handle_connect 0x7f88dc06e050 2024-06-03T09:06:27.832 INFO:teuthology.orchestra.run.smithi088.stderr:2024-06-03T09:06:27.825+0000 7f88d1ffb640 10 client.?.objecter resend_mon_ops 2024-06-03T09:06:27.832 INFO:teuthology.orchestra.run.smithi088.stderr:2024-06-03T09:06:27.825+0000 7f88d1ffb640 10 client.?.objecter ms_handle_connect 0x7f88dc0acad0 2024-06-03T09:06:27.832 INFO:teuthology.orchestra.run.smithi088.stderr:2024-06-03T09:06:27.825+0000 7f88d1ffb640 10 client.?.objecter resend_mon_ops 2024-06-03T09:06:27.832 INFO:teuthology.orchestra.run.smithi088.stderr:2024-06-03T09:06:27.827+0000 7f88d1ffb640 1 -- 172.21.15.88:0/2567601929 <== mon.1 v1:172.21.15.192:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 3226770211 0 0) 0x7f88dc11b3a0 con 0x7f88dc06e050 2024-06-03T09:06:27.832 INFO:teuthology.orchestra.run.smithi088.stderr:2024-06-03T09:06:27.827+0000 7f88d1ffb640 10 monclient(hunting): _negotiate 2024-06-03T09:06:27.832 INFO:teuthology.orchestra.run.smithi088.stderr:2024-06-03T09:06:27.827+0000 7f88d1ffb640 10 monclient(hunting): _init_auth method 2 2024-06-03T09:06:27.832 INFO:teuthology.orchestra.run.smithi088.stderr:2024-06-03T09:06:27.827+0000 7f88d1ffb640 10 monclient(hunting): _init_auth creating new auth 2024-06-03T09:06:27.832 INFO:teuthology.orchestra.run.smithi088.stderr:2024-06-03T09:06:27.827+0000 7f88d1ffb640 10 monclient(hunting): my global_id is 30202 2024-06-03T09:06:27.832 INFO:teuthology.orchestra.run.smithi088.stderr:2024-06-03T09:06:27.827+0000 7f88d1ffb640 1 -- 172.21.15.88:0/2567601929 --> v1:172.21.15.192:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f88c0004eb0 con 0x7f88dc06e050 2024-06-03T09:06:27.834 INFO:teuthology.orchestra.run.smithi088.stderr:2024-06-03T09:06:27.827+0000 7f88d1ffb640 1 -- 172.21.15.88:0/2567601929 <== mon.2 v1:172.21.15.88:6790/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 2723720291 0 0) 0x7f88dc139260 con 0x7f88dc0acad0 2024-06-03T09:06:27.835 INFO:teuthology.orchestra.run.smithi088.stderr:2024-06-03T09:06:27.827+0000 7f88d1ffb640 10 monclient(hunting): _negotiate 2024-06-03T09:06:27.835 INFO:teuthology.orchestra.run.smithi088.stderr:2024-06-03T09:06:27.827+0000 7f88d1ffb640 10 monclient(hunting): _init_auth method 2 2024-06-03T09:06:27.835 INFO:teuthology.orchestra.run.smithi088.stderr:2024-06-03T09:06:27.827+0000 7f88d1ffb640 10 monclient(hunting): _init_auth creating new auth 2024-06-03T09:06:27.835 INFO:teuthology.orchestra.run.smithi088.stderr:2024-06-03T09:06:27.827+0000 7f88d1ffb640 10 monclient(hunting): my global_id is 30125 2024-06-03T09:06:27.835 INFO:teuthology.orchestra.run.smithi088.stderr:2024-06-03T09:06:27.827+0000 7f88d1ffb640 1 -- 172.21.15.88:0/2567601929 --> v1:172.21.15.88:6790/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f88dc11b3a0 con 0x7f88dc0acad0 2024-06-03T09:06:27.835 INFO:teuthology.orchestra.run.smithi088.stderr:2024-06-03T09:06:27.827+0000 7f88d1ffb640 1 -- 172.21.15.88:0/2567601929 <== mon.0 v1:172.21.15.88:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 796085113 0 0) 0x7f88dc13dd30 con 0x7f88dc0717c0 2024-06-03T09:06:27.835 INFO:teuthology.orchestra.run.smithi088.stderr:2024-06-03T09:06:27.827+0000 7f88d1ffb640 10 monclient(hunting): _negotiate 2024-06-03T09:06:27.835 INFO:teuthology.orchestra.run.smithi088.stderr:2024-06-03T09:06:27.827+0000 7f88d1ffb640 10 monclient(hunting): _init_auth method 2 2024-06-03T09:06:27.835 INFO:teuthology.orchestra.run.smithi088.stderr:2024-06-03T09:06:27.827+0000 7f88d1ffb640 10 monclient(hunting): _init_auth creating new auth 2024-06-03T09:06:27.835 INFO:teuthology.orchestra.run.smithi088.stderr:2024-06-03T09:06:27.827+0000 7f88d1ffb640 10 monclient(hunting): my global_id is 30621 2024-06-03T09:06:27.835 INFO:teuthology.orchestra.run.smithi088.stderr:2024-06-03T09:06:27.827+0000 7f88d1ffb640 1 -- 172.21.15.88:0/2567601929 --> v1:172.21.15.88:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f88dc139260 con 0x7f88dc0717c0 2024-06-03T09:06:27.835 INFO:teuthology.orchestra.run.smithi088.stderr:2024-06-03T09:06:27.831+0000 7f88d1ffb640 1 -- 172.21.15.88:0/2567601929 <== mon.1 v1:172.21.15.192:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 764+0+0 (unknown 971801959 0 0) 0x7f88c0004eb0 con 0x7f88dc06e050 2024-06-03T09:06:27.836 INFO:teuthology.orchestra.run.smithi088.stderr:2024-06-03T09:06:27.831+0000 7f88d1ffb640 1 -- 172.21.15.88:0/2567601929 --> v1:172.21.15.192:6789/0 -- auth(proto 2 165 bytes epoch 0) v1 -- 0x7f88dc13dd30 con 0x7f88dc06e050 2024-06-03T09:06:27.836 INFO:teuthology.orchestra.run.smithi088.stderr:2024-06-03T09:06:27.831+0000 7f88d1ffb640 1 -- 172.21.15.88:0/2567601929 <== mon.1 v1:172.21.15.192:6789/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 2234374065 0 0) 0x7f88cc001b20 con 0x7f88dc06e050 2024-06-03T09:06:27.836 INFO:teuthology.orchestra.run.smithi088.stderr:2024-06-03T09:06:27.831+0000 7f88d1ffb640 10 monclient(hunting): handle_monmap mon_map magic: 0 v1 2024-06-03T09:06:27.836 INFO:teuthology.orchestra.run.smithi088.stderr:2024-06-03T09:06:27.831+0000 7f88d1ffb640 10 monclient(hunting): got monmap 1 from mon.noname-b (according to old e1) 2024-06-03T09:06:27.836 INFO:teuthology.orchestra.run.smithi088.stderr:2024-06-03T09:06:27.831+0000 7f88d1ffb640 10 monclient(hunting): dump: 2024-06-03T09:06:27.836 INFO:teuthology.orchestra.run.smithi088.stderr:epoch 1 2024-06-03T09:06:27.836 INFO:teuthology.orchestra.run.smithi088.stderr:fsid ec3c3fdd-3ac6-4d7d-9412-683130ed8b04 2024-06-03T09:06:27.836 INFO:teuthology.orchestra.run.smithi088.stderr:last_changed 2024-06-03T08:34:31.885125+0000 2024-06-03T09:06:27.836 INFO:teuthology.orchestra.run.smithi088.stderr:created 2024-06-03T08:34:31.885125+0000 2024-06-03T09:06:27.836 INFO:teuthology.orchestra.run.smithi088.stderr:min_mon_release 18 (reef) 2024-06-03T09:06:27.836 INFO:teuthology.orchestra.run.smithi088.stderr:election_strategy: 3 2024-06-03T09:06:27.836 INFO:teuthology.orchestra.run.smithi088.stderr:0: v1:172.21.15.88:6789/0 mon.a 2024-06-03T09:06:27.836 INFO:teuthology.orchestra.run.smithi088.stderr:1: v1:172.21.15.192:6789/0 mon.b 2024-06-03T09:06:27.837 INFO:teuthology.orchestra.run.smithi088.stderr:2: v1:172.21.15.88:6790/0 mon.c 2024-06-03T09:06:27.837 INFO:teuthology.orchestra.run.smithi088.stderr: 2024-06-03T09:06:27.837 INFO:teuthology.orchestra.run.smithi088.stderr:2024-06-03T09:06:27.831+0000 7f88d1ffb640 10 monclient(hunting): _finish_auth 0 2024-06-03T09:06:27.837 INFO:teuthology.orchestra.run.smithi088.stderr:2024-06-03T09:06:27.832+0000 7f88d1ffb640 1 -- 172.21.15.88:0/2567601929 <== mon.2 v1:172.21.15.88:6790/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 764+0+0 (unknown 152501792 0 0) 0x7f88dc11b3a0 con 0x7f88dc0acad0 2024-06-03T09:06:27.837 INFO:teuthology.orchestra.run.smithi088.stderr:2024-06-03T09:06:27.832+0000 7f88d1ffb640 1 -- 172.21.15.88:0/2567601929 --> v1:172.21.15.88:6790/0 -- auth(proto 2 165 bytes epoch 0) v1 -- 0x7f88c0004eb0 con 0x7f88dc0acad0 2024-06-03T09:06:27.837 INFO:teuthology.orchestra.run.smithi088.stderr:2024-06-03T09:06:27.832+0000 7f88d1ffb640 1 -- 172.21.15.88:0/2567601929 <== mon.2 v1:172.21.15.88:6790/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 2234374065 0 0) 0x7f88d4001b90 con 0x7f88dc0acad0 2024-06-03T09:06:27.837 INFO:teuthology.orchestra.run.smithi088.stderr:2024-06-03T09:06:27.832+0000 7f88d1ffb640 10 monclient(hunting): handle_monmap mon_map magic: 0 v1 2024-06-03T09:06:27.837 INFO:teuthology.orchestra.run.smithi088.stderr:2024-06-03T09:06:27.832+0000 7f88d1ffb640 10 monclient(hunting): got monmap 1 from mon.c (according to old e1) 2024-06-03T09:06:27.837 INFO:teuthology.orchestra.run.smithi088.stderr:2024-06-03T09:06:27.832+0000 7f88d1ffb640 10 monclient(hunting): dump: 2024-06-03T09:06:27.837 INFO:teuthology.orchestra.run.smithi088.stderr:epoch 1 2024-06-03T09:06:27.837 INFO:teuthology.orchestra.run.smithi088.stderr:fsid ec3c3fdd-3ac6-4d7d-9412-683130ed8b04 2024-06-03T09:06:27.837 INFO:teuthology.orchestra.run.smithi088.stderr:last_changed 2024-06-03T08:34:31.885125+0000 2024-06-03T09:06:27.838 INFO:teuthology.orchestra.run.smithi088.stderr:created 2024-06-03T08:34:31.885125+0000 2024-06-03T09:06:27.838 INFO:teuthology.orchestra.run.smithi088.stderr:min_mon_release 18 (reef) 2024-06-03T09:06:27.838 INFO:teuthology.orchestra.run.smithi088.stderr:election_strategy: 3 2024-06-03T09:06:27.838 INFO:teuthology.orchestra.run.smithi088.stderr:0: v1:172.21.15.88:6789/0 mon.a 2024-06-03T09:06:27.838 INFO:teuthology.orchestra.run.smithi088.stderr:1: v1:172.21.15.192:6789/0 mon.b 2024-06-03T09:06:27.838 INFO:teuthology.orchestra.run.smithi088.stderr:2: v1:172.21.15.88:6790/0 mon.c 2024-06-03T09:06:27.838 INFO:teuthology.orchestra.run.smithi088.stderr: 2024-06-03T09:06:27.838 INFO:teuthology.orchestra.run.smithi088.stderr:2024-06-03T09:06:27.832+0000 7f88d1ffb640 1 -- 172.21.15.88:0/2567601929 <== mon.0 v1:172.21.15.88:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 764+0+0 (unknown 2619599450 0 0) 0x7f88dc139260 con 0x7f88dc0717c0 2024-06-03T09:06:27.838 INFO:teuthology.orchestra.run.smithi088.stderr:2024-06-03T09:06:27.832+0000 7f88d1ffb640 1 -- 172.21.15.88:0/2567601929 --> v1:172.21.15.88:6789/0 -- auth(proto 2 165 bytes epoch 0) v1 -- 0x7f88dc11b3a0 con 0x7f88dc0717c0 2024-06-03T09:06:27.838 INFO:teuthology.orchestra.run.smithi088.stderr:2024-06-03T09:06:27.832+0000 7f88d1ffb640 1 -- 172.21.15.88:0/2567601929 <== mon.0 v1:172.21.15.88:6789/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 2234374065 0 0) 0x7f88d80018b0 con 0x7f88dc0717c0 2024-06-03T09:06:27.838 INFO:teuthology.orchestra.run.smithi088.stderr:2024-06-03T09:06:27.832+0000 7f88d1ffb640 10 monclient(hunting): handle_monmap mon_map magic: 0 v1 2024-06-03T09:06:27.838 INFO:teuthology.orchestra.run.smithi088.stderr:2024-06-03T09:06:27.832+0000 7f88d1ffb640 10 monclient(hunting): got monmap 1 from mon.a (according to old e1) 2024-06-03T09:06:27.838 INFO:teuthology.orchestra.run.smithi088.stderr:2024-06-03T09:06:27.832+0000 7f88d1ffb640 10 monclient(hunting): dump: 2024-06-03T09:06:27.839 INFO:teuthology.orchestra.run.smithi088.stderr:epoch 1 2024-06-03T09:06:27.839 INFO:teuthology.orchestra.run.smithi088.stderr:fsid ec3c3fdd-3ac6-4d7d-9412-683130ed8b04 2024-06-03T09:06:27.839 INFO:teuthology.orchestra.run.smithi088.stderr:last_changed 2024-06-03T08:34:31.885125+0000 2024-06-03T09:06:27.839 INFO:teuthology.orchestra.run.smithi088.stderr:created 2024-06-03T08:34:31.885125+0000 2024-06-03T09:06:27.839 INFO:teuthology.orchestra.run.smithi088.stderr:min_mon_release 18 (reef) 2024-06-03T09:06:27.839 INFO:teuthology.orchestra.run.smithi088.stderr:election_strategy: 3 2024-06-03T09:06:27.839 INFO:teuthology.orchestra.run.smithi088.stderr:0: v1:172.21.15.88:6789/0 mon.a 2024-06-03T09:06:27.839 INFO:teuthology.orchestra.run.smithi088.stderr:1: v1:172.21.15.192:6789/0 mon.b 2024-06-03T09:06:27.839 INFO:teuthology.orchestra.run.smithi088.stderr:2: v1:172.21.15.88:6790/0 mon.c 2024-06-03T09:06:27.839 INFO:teuthology.orchestra.run.smithi088.stderr: 2024-06-03T09:06:27.840 INFO:teuthology.orchestra.run.smithi088.stderr:2024-06-03T09:06:27.835+0000 7f88d1ffb640 1 -- 172.21.15.88:0/2567601929 <== mon.1 v1:172.21.15.192:6789/0 4 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 580+0+0 (unknown 2081500335 0 0) 0x7f88dc13dd30 con 0x7f88dc06e050 2024-06-03T09:06:27.840 INFO:teuthology.orchestra.run.smithi088.stderr:2024-06-03T09:06:27.835+0000 7f88d1ffb640 1 -- 172.21.15.88:0/2567601929 >> v1:172.21.15.88:6790/0 conn(0x7f88dc0acad0 legacy=0x7f88dc1e1ff0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-06-03T09:06:27.841 INFO:teuthology.orchestra.run.smithi088.stderr:2024-06-03T09:06:27.835+0000 7f88d1ffb640 1 -- 172.21.15.88:0/2567601929 >> v1:172.21.15.88:6789/0 conn(0x7f88dc0717c0 legacy=0x7f88dc138b50 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-06-03T09:06:27.841 INFO:teuthology.orchestra.run.smithi088.stderr:2024-06-03T09:06:27.835+0000 7f88d1ffb640 10 monclient: _finish_hunting 0 2024-06-03T09:06:27.841 INFO:teuthology.orchestra.run.smithi088.stderr:2024-06-03T09:06:27.835+0000 7f88d1ffb640 1 monclient: found mon.b 2024-06-03T09:06:27.841 INFO:teuthology.orchestra.run.smithi088.stderr:2024-06-03T09:06:27.835+0000 7f88d1ffb640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-06-03T09:06:27.841 INFO:teuthology.orchestra.run.smithi088.stderr:2024-06-03T09:06:27.835+0000 7f88d1ffb640 10 monclient: _send_mon_message to mon.b at v1:172.21.15.192:6789/0 2024-06-03T09:06:27.841 INFO:teuthology.orchestra.run.smithi088.stderr:2024-06-03T09:06:27.835+0000 7f88d1ffb640 1 -- 172.21.15.88:0/2567601929 --> v1:172.21.15.192:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f88dc13e820 con 0x7f88dc06e050 2024-06-03T09:06:27.841 INFO:teuthology.orchestra.run.smithi088.stderr:2024-06-03T09:06:27.835+0000 7f88d1ffb640 10 monclient: _finish_auth 0 2024-06-03T09:06:27.841 INFO:teuthology.orchestra.run.smithi088.stderr:2024-06-03T09:06:27.835+0000 7f88d1ffb640 10 monclient: _check_auth_tickets 2024-06-03T09:06:27.841 INFO:teuthology.orchestra.run.smithi088.stderr:2024-06-03T09:06:27.835+0000 7f88d1ffb640 20 monclient: _check_auth_rotating not needed by client.admin 2024-06-03T09:06:27.841 INFO:teuthology.orchestra.run.smithi088.stderr:2024-06-03T09:06:27.835+0000 7f88e35c8640 5 monclient: authenticate success, global_id 30202 2024-06-03T09:06:27.841 INFO:teuthology.orchestra.run.smithi088.stderr:2024-06-03T09:06:27.835+0000 7f88e35c8640 10 monclient: _renew_subs 2024-06-03T09:06:27.842 INFO:teuthology.orchestra.run.smithi088.stderr:2024-06-03T09:06:27.835+0000 7f88e35c8640 10 monclient: _send_mon_message to mon.b at v1:172.21.15.192:6789/0 2024-06-03T09:06:27.842 INFO:teuthology.orchestra.run.smithi088.stderr:2024-06-03T09:06:27.835+0000 7f88e35c8640 1 -- 172.21.15.88:0/2567601929 --> v1:172.21.15.192:6789/0 -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f88dc11ab00 con 0x7f88dc06e050 2024-06-03T09:06:27.842 INFO:teuthology.orchestra.run.smithi088.stderr:2024-06-03T09:06:27.835+0000 7f88e35c8640 10 client.30202.objecter _maybe_request_map subscribing (onetime) to next osd map 2024-06-03T09:06:27.842 INFO:teuthology.orchestra.run.smithi088.stderr:2024-06-03T09:06:27.835+0000 7f88e35c8640 10 monclient: _renew_subs 2024-06-03T09:06:27.842 INFO:teuthology.orchestra.run.smithi088.stderr:2024-06-03T09:06:27.835+0000 7f88e35c8640 10 monclient: _send_mon_message to mon.b at v1:172.21.15.192:6789/0 2024-06-03T09:06:27.842 INFO:teuthology.orchestra.run.smithi088.stderr:2024-06-03T09:06:27.835+0000 7f88e35c8640 1 -- 172.21.15.88:0/2567601929 --> v1:172.21.15.192:6789/0 -- mon_subscribe({osdmap=0}) v3 -- 0x7f88dc11af90 con 0x7f88dc06e050 2024-06-03T09:06:27.842 INFO:teuthology.orchestra.run.smithi088.stderr:2024-06-03T09:06:27.835+0000 7f88e35c8640 1 librados: init done 2024-06-03T09:06:27.842 INFO:teuthology.orchestra.run.smithi088.stderr:2024-06-03T09:06:27.836+0000 7f88e35c8640 10 start_mon_command cmd=[{"prefix": "get_command_descriptions"}] 2024-06-03T09:06:27.842 INFO:teuthology.orchestra.run.smithi088.stderr:2024-06-03T09:06:27.836+0000 7f88e35c8640 10 monclient: _send_command 1 [{"prefix": "get_command_descriptions"}] 2024-06-03T09:06:27.842 INFO:teuthology.orchestra.run.smithi088.stderr:2024-06-03T09:06:27.836+0000 7f88e35c8640 10 monclient: _send_mon_message to mon.b at v1:172.21.15.192:6789/0 2024-06-03T09:06:27.842 INFO:teuthology.orchestra.run.smithi088.stderr:2024-06-03T09:06:27.836+0000 7f88e35c8640 1 -- 172.21.15.88:0/2567601929 --> v1:172.21.15.192:6789/0 -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f88dc11b1f0 con 0x7f88dc06e050 2024-06-03T09:06:28.051 INFO:teuthology.orchestra.run.smithi088.stderr:2024-06-03T09:06:28.051+0000 7f88d1ffb640 1 -- 172.21.15.88:0/2567601929 <== mon.1 v1:172.21.15.192:6789/0 5 ==== config(0 keys) v1 ==== 4+0+0 (unknown 0 0 0) 0x7f88cc0038a0 con 0x7f88dc06e050 2024-06-03T09:06:28.051 INFO:teuthology.orchestra.run.smithi088.stderr:2024-06-03T09:06:28.051+0000 7f88d1ffb640 10 monclient: handle_config config(0 keys) v1 2024-06-03T09:06:28.051 INFO:teuthology.orchestra.run.smithi088.stderr:2024-06-03T09:06:28.051+0000 7f88d1ffb640 1 -- 172.21.15.88:0/2567601929 <== mon.1 v1:172.21.15.192:6789/0 6 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 2234374065 0 0) 0x7f88cc003bb0 con 0x7f88dc06e050 2024-06-03T09:06:28.051 INFO:teuthology.orchestra.run.smithi088.stderr:2024-06-03T09:06:28.051+0000 7f88d1ffb640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-06-03T09:06:28.051 INFO:teuthology.orchestra.run.smithi088.stderr:2024-06-03T09:06:28.051+0000 7f88d1ffb640 10 monclient: got monmap 1 from mon.b (according to old e1) 2024-06-03T09:06:28.051 INFO:teuthology.orchestra.run.smithi088.stderr:2024-06-03T09:06:28.051+0000 7f88d1ffb640 10 monclient: dump: 2024-06-03T09:06:28.051 INFO:teuthology.orchestra.run.smithi088.stderr:epoch 1 2024-06-03T09:06:28.051 INFO:teuthology.orchestra.run.smithi088.stderr:fsid ec3c3fdd-3ac6-4d7d-9412-683130ed8b04 2024-06-03T09:06:28.052 INFO:teuthology.orchestra.run.smithi088.stderr:last_changed 2024-06-03T08:34:31.885125+0000 2024-06-03T09:06:28.052 INFO:teuthology.orchestra.run.smithi088.stderr:created 2024-06-03T08:34:31.885125+0000 2024-06-03T09:06:28.052 INFO:teuthology.orchestra.run.smithi088.stderr:min_mon_release 18 (reef) 2024-06-03T09:06:28.052 INFO:teuthology.orchestra.run.smithi088.stderr:election_strategy: 3 2024-06-03T09:06:28.052 INFO:teuthology.orchestra.run.smithi088.stderr:0: v1:172.21.15.88:6789/0 mon.a 2024-06-03T09:06:28.052 INFO:teuthology.orchestra.run.smithi088.stderr:1: v1:172.21.15.192:6789/0 mon.b 2024-06-03T09:06:28.052 INFO:teuthology.orchestra.run.smithi088.stderr:2: v1:172.21.15.88:6790/0 mon.c 2024-06-03T09:06:28.052 INFO:teuthology.orchestra.run.smithi088.stderr: 2024-06-03T09:06:28.052 INFO:teuthology.orchestra.run.smithi088.stderr:2024-06-03T09:06:28.051+0000 7f88d37fe640 4 set_mon_vals no callback set 2024-06-03T09:06:28.093 INFO:teuthology.orchestra.run.smithi088.stderr:2024-06-03T09:06:28.093+0000 7f88d1ffb640 1 -- 172.21.15.88:0/2567601929 <== mon.1 v1:172.21.15.192:6789/0 7 ==== mgrmap(e 5) v1 ==== 89110+0+0 (unknown 873533973 0 0) 0x7f88cc019930 con 0x7f88dc06e050 2024-06-03T09:06:28.093 INFO:teuthology.orchestra.run.smithi088.stderr:2024-06-03T09:06:28.093+0000 7f88e0b63640 10 monclient: get_auth_request con 0x7f88c006d450 auth_method 0 2024-06-03T09:06:28.093 INFO:teuthology.orchestra.run.smithi088.stderr:2024-06-03T09:06:28.094+0000 7f88d1ffb640 10 client.30202.objecter ms_handle_connect 0x7f88c006d450 2024-06-03T09:06:28.095 INFO:teuthology.orchestra.run.smithi088.stderr:2024-06-03T09:06:28.095+0000 7f88d1ffb640 1 -- 172.21.15.88:0/2567601929 <== mon.1 v1:172.21.15.192:6789/0 8 ==== osd_map(1010..1010 src has 622..1010) v4 ==== 5229+0+0 (unknown 1105883993 0 0) 0x7f88cc052fc0 con 0x7f88dc06e050 2024-06-03T09:06:28.095 INFO:teuthology.orchestra.run.smithi088.stderr:2024-06-03T09:06:28.095+0000 7f88d1ffb640 10 client.30202.objecter ms_dispatch 0x7f88dc000b90 osd_map(1010..1010 src has 622..1010) v4 2024-06-03T09:06:28.095 INFO:teuthology.orchestra.run.smithi088.stderr:2024-06-03T09:06:28.095+0000 7f88d1ffb640 3 client.30202.objecter handle_osd_map got epochs [1010,1010] > 0 2024-06-03T09:06:28.095 INFO:teuthology.orchestra.run.smithi088.stderr:2024-06-03T09:06:28.095+0000 7f88d1ffb640 3 client.30202.objecter handle_osd_map decoding full epoch 1010 2024-06-03T09:06:28.096 INFO:teuthology.orchestra.run.smithi088.stderr:2024-06-03T09:06:28.095+0000 7f88d1ffb640 20 client.30202.objecter dump_active .. 0 homeless 2024-06-03T09:06:28.097 INFO:teuthology.orchestra.run.smithi088.stderr:2024-06-03T09:06:28.097+0000 7f88d1ffb640 1 -- 172.21.15.88:0/2567601929 <== mon.1 v1:172.21.15.192:6789/0 9 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+182498 (unknown 1092875540 0 3016760700) 0x7f88cc04e9a0 con 0x7f88dc06e050 2024-06-03T09:06:28.097 INFO:teuthology.orchestra.run.smithi088.stderr:2024-06-03T09:06:28.097+0000 7f88d1ffb640 10 monclient: handle_mon_command_ack 1 [{"prefix": "get_command_descriptions"}] 2024-06-03T09:06:28.097 INFO:teuthology.orchestra.run.smithi088.stderr:2024-06-03T09:06:28.097+0000 7f88d1ffb640 10 monclient: _finish_command 1 = system:0 2024-06-03T09:06:28.263 INFO:teuthology.orchestra.run.smithi088.stderr:2024-06-03T09:06:28.263+0000 7f88e35c8640 1 -- 172.21.15.88:0/2567601929 --> v1:172.21.15.192:6800/34229 -- mgr_command(tid 0: {"prefix": "pg dump", "target": ["mon-mgr", ""], "format": "json"}) v1 -- 0x7f88dc1393f0 con 0x7f88c006d450 2024-06-03T09:06:28.268 INFO:teuthology.orchestra.run.smithi088.stderr:2024-06-03T09:06:28.268+0000 7f88d1ffb640 1 -- 172.21.15.88:0/2567601929 <== mgr.4110 v1:172.21.15.192:6800/34229 1 ==== mgr_command_reply(tid 0: 0 dumped all) v1 ==== 18+0+151639 (unknown 2965378022 0 2281109479) 0x7f88dc1393f0 con 0x7f88c006d450 2024-06-03T09:06:28.270 INFO:teuthology.orchestra.run.smithi088.stdout: 2024-06-03T09:06:28.270 INFO:teuthology.orchestra.run.smithi088.stderr:dumped all 2024-06-03T09:06:28.272 INFO:teuthology.orchestra.run.smithi088.stderr:2024-06-03T09:06:28.270+0000 7f88e35c8640 10 librados: watch_flush enter 2024-06-03T09:06:28.272 INFO:teuthology.orchestra.run.smithi088.stderr:2024-06-03T09:06:28.270+0000 7f88e35c8640 10 librados: watch_flush exit 2024-06-03T09:06:28.272 INFO:teuthology.orchestra.run.smithi088.stderr:2024-06-03T09:06:28.270+0000 7f88e35c8640 20 client.30202.objecter shutdown clearing up homeless session... 2024-06-03T09:06:28.272 INFO:teuthology.orchestra.run.smithi088.stderr:2024-06-03T09:06:28.270+0000 7f88e35c8640 10 client.30202.objecter successfully canceled tick 2024-06-03T09:06:28.272 INFO:teuthology.orchestra.run.smithi088.stderr:2024-06-03T09:06:28.270+0000 7f88e35c8640 1 -- 172.21.15.88:0/2567601929 >> v1:172.21.15.192:6800/34229 conn(0x7f88c006d450 legacy=0x7f88c006f910 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-06-03T09:06:28.272 INFO:teuthology.orchestra.run.smithi088.stderr:2024-06-03T09:06:28.270+0000 7f88e35c8640 10 monclient: shutdown 2024-06-03T09:06:28.272 INFO:teuthology.orchestra.run.smithi088.stderr:2024-06-03T09:06:28.270+0000 7f88e35c8640 20 monclient: shutdown discarding 0 pending message(s) 2024-06-03T09:06:28.272 INFO:teuthology.orchestra.run.smithi088.stderr:2024-06-03T09:06:28.270+0000 7f88e35c8640 1 -- 172.21.15.88:0/2567601929 >> v1:172.21.15.192:6789/0 conn(0x7f88dc06e050 legacy=0x7f88dc1d9c90 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-06-03T09:06:28.272 INFO:teuthology.orchestra.run.smithi088.stderr:2024-06-03T09:06:28.270+0000 7f88e35c8640 1 -- 172.21.15.88:0/2567601929 shutdown_connections 2024-06-03T09:06:28.273 INFO:teuthology.orchestra.run.smithi088.stderr:2024-06-03T09:06:28.270+0000 7f88e35c8640 1 -- 172.21.15.88:0/2567601929 >> 172.21.15.88:0/2567601929 conn(0x7f88dc0b2f20 msgr2=0x7f88dc0b0b10 unknown :-1 s=STATE_NONE l=0).mark_down 2024-06-03T09:06:28.273 INFO:teuthology.orchestra.run.smithi088.stderr:2024-06-03T09:06:28.271+0000 7f88e35c8640 1 -- 172.21.15.88:0/2567601929 shutdown_connections 2024-06-03T09:06:28.273 INFO:teuthology.orchestra.run.smithi088.stderr:2024-06-03T09:06:28.271+0000 7f88e35c8640 1 -- 172.21.15.88:0/2567601929 wait complete. 2024-06-03T09:06:28.273 INFO:teuthology.orchestra.run.smithi088.stderr:2024-06-03T09:06:28.271+0000 7f88e35c8640 1 librados: shutdown 2024-06-03T09:06:28.281 INFO:teuthology.orchestra.run.smithi088.stdout:{"pg_ready":true,"pg_map":{"version":1919,"stamp":"2024-06-03T09:06:27.099877+0000","last_osdmap_epoch":0,"last_pg_scan":0,"pg_stats_sum":{"stat_sum":{"num_bytes":459285,"num_objects":3,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":1,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":3,"num_whiteouts":0,"num_read":102,"num_read_kb":88,"num_write":120,"num_write_kb":1379,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2,"num_bytes_recovered":397840,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":83,"ondisk_log_size":83,"up":68,"acting":68,"num_store_stats":0},"osd_stats_sum":{"up_from":0,"seq":0,"num_pgs":100,"num_osds":8,"num_per_pool_osds":8,"num_per_pool_omap_osds":8,"kb":754974720,"kb_used":494548,"kb_used_data":46324,"kb_used_omap":12,"kb_used_meta":448115,"kb_avail":754480172,"statfs":{"total":773094113280,"available":772587696128,"internally_reserved":0,"allocated":47435776,"data_stored":22747954,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":12744,"internal_metadata":458870328},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":109,"apply_latency_ms":109,"commit_latency_ns":109000000,"apply_latency_ns":109000000},"alerts":[],"network_ping_times":[]},"pg_stats_delta":{"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":0,"acting":0,"num_store_stats":0,"stamp_delta":"10.002020"},"pg_stats":[{"pgid":"17.d","version":"0'0","reported_seq":942,"reported_epoch":1009,"state":"active+clean","last_fresh":"2024-06-03T09:06:14.443917+0000","last_change":"2024-06-03T08:38:19.260548+0000","last_active":"2024-06-03T09:06:14.443917+0000","last_peered":"2024-06-03T09:06:14.443917+0000","last_clean":"2024-06-03T09:06:14.443917+0000","last_became_active":"2024-06-03T08:38:19.253559+0000","last_became_peered":"2024-06-03T08:38:19.253559+0000","last_unstale":"2024-06-03T09:06:14.443917+0000","last_undegraded":"2024-06-03T09:06:14.443917+0000","last_fullsized":"2024-06-03T09:06:14.443917+0000","mapping_epoch":28,"log_start":"0'0","ondisk_log_start":"0'0","created":28,"last_epoch_clean":45,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-06-03T08:37:55.614497+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-06-03T08:37:55.614497+0000","last_clean_scrub_stamp":"2024-06-03T08:37:55.614497+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-06-04T19:00:12.470979+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1],"acting":[0,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"17.c","version":"0'0","reported_seq":953,"reported_epoch":1009,"state":"active+undersized","last_fresh":"2024-06-03T09:06:13.668156+0000","last_change":"2024-06-03T09:06:03.440074+0000","last_active":"2024-06-03T09:06:13.668156+0000","last_peered":"2024-06-03T09:06:13.668156+0000","last_clean":"2024-06-03T09:05:02.105638+0000","last_became_active":"2024-06-03T09:06:03.440074+0000","last_became_peered":"2024-06-03T09:06:03.440074+0000","last_unstale":"2024-06-03T09:06:13.668156+0000","last_undegraded":"2024-06-03T09:06:13.668156+0000","last_fullsized":"2024-06-03T09:06:03.377982+0000","mapping_epoch":999,"log_start":"0'0","ondisk_log_start":"0'0","created":28,"last_epoch_clean":255,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-06-03T08:37:55.614497+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-06-03T08:37:55.614497+0000","last_clean_scrub_stamp":"2024-06-03T08:37:55.614497+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-06-04T10:34:20.808557+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3],"acting":[3],"avail_no_missing":["3"],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"17.f","version":"0'0","reported_seq":148,"reported_epoch":1009,"state":"active+undersized","last_fresh":"2024-06-03T09:06:14.450543+0000","last_change":"2024-06-03T09:06:03.420285+0000","last_active":"2024-06-03T09:06:14.450543+0000","last_peered":"2024-06-03T09:06:14.450543+0000","last_clean":"2024-06-03T08:37:55.614497+0000","last_became_active":"2024-06-03T09:06:03.420285+0000","last_became_peered":"2024-06-03T09:06:03.420285+0000","last_unstale":"2024-06-03T09:06:14.450543+0000","last_undegraded":"2024-06-03T09:06:14.450543+0000","last_fullsized":"2024-06-03T09:06:03.380616+0000","mapping_epoch":999,"log_start":"0'0","ondisk_log_start":"0'0","created":28,"last_epoch_clean":255,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-06-03T08:37:55.614497+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-06-03T08:37:55.614497+0000","last_clean_scrub_stamp":"2024-06-03T08:37:55.614497+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-06-04T15:39:41.187731+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4],"acting":[4],"avail_no_missing":["4"],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"17.e","version":"0'0","reported_seq":149,"reported_epoch":1009,"state":"active+undersized","last_fresh":"2024-06-03T09:06:13.726992+0000","last_change":"2024-06-03T09:06:03.412887+0000","last_active":"2024-06-03T09:06:13.726992+0000","last_peered":"2024-06-03T09:06:13.726992+0000","last_clean":"2024-06-03T08:37:55.614497+0000","last_became_active":"2024-06-03T09:06:03.412887+0000","last_became_peered":"2024-06-03T09:06:03.412887+0000","last_unstale":"2024-06-03T09:06:13.726992+0000","last_undegraded":"2024-06-03T09:06:13.726992+0000","last_fullsized":"2024-06-03T09:06:03.332482+0000","mapping_epoch":999,"log_start":"0'0","ondisk_log_start":"0'0","created":28,"last_epoch_clean":255,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-06-03T08:37:55.614497+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-06-03T08:37:55.614497+0000","last_clean_scrub_stamp":"2024-06-03T08:37:55.614497+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-06-04T12:23:22.511601+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5],"acting":[5],"avail_no_missing":["5"],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"17.9","version":"0'0","reported_seq":946,"reported_epoch":1009,"state":"active+clean","last_fresh":"2024-06-03T09:06:14.999299+0000","last_change":"2024-06-03T08:38:04.772260+0000","last_active":"2024-06-03T09:06:14.999299+0000","last_peered":"2024-06-03T09:06:14.999299+0000","last_clean":"2024-06-03T09:06:14.999299+0000","last_became_active":"2024-06-03T08:38:04.763428+0000","last_became_peered":"2024-06-03T08:38:04.763428+0000","last_unstale":"2024-06-03T09:06:14.999299+0000","last_undegraded":"2024-06-03T09:06:14.999299+0000","last_fullsized":"2024-06-03T09:06:14.999299+0000","mapping_epoch":28,"log_start":"0'0","ondisk_log_start":"0'0","created":28,"last_epoch_clean":33,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-06-03T08:37:55.614497+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-06-03T08:37:55.614497+0000","last_clean_scrub_stamp":"2024-06-03T08:37:55.614497+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-06-04T10:55:10.773936+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,4],"acting":[7,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"17.8","version":"0'0","reported_seq":961,"reported_epoch":1009,"state":"active+undersized","last_fresh":"2024-06-03T09:06:14.430375+0000","last_change":"2024-06-03T09:06:03.388786+0000","last_active":"2024-06-03T09:06:14.430375+0000","last_peered":"2024-06-03T09:06:14.430375+0000","last_clean":"2024-06-03T09:04:43.297780+0000","last_became_active":"2024-06-03T09:06:03.388786+0000","last_became_peered":"2024-06-03T09:06:03.388786+0000","last_unstale":"2024-06-03T09:06:14.430375+0000","last_undegraded":"2024-06-03T09:06:14.430375+0000","last_fullsized":"2024-06-03T09:06:03.331754+0000","mapping_epoch":999,"log_start":"0'0","ondisk_log_start":"0'0","created":28,"last_epoch_clean":255,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-06-03T08:37:55.614497+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-06-03T08:37:55.614497+0000","last_clean_scrub_stamp":"2024-06-03T08:37:55.614497+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-06-04T09:18:44.765547+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0],"acting":[0],"avail_no_missing":["0"],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"17.b","version":"0'0","reported_seq":149,"reported_epoch":1009,"state":"active+undersized","last_fresh":"2024-06-03T09:06:14.433735+0000","last_change":"2024-06-03T09:06:03.451203+0000","last_active":"2024-06-03T09:06:14.433735+0000","last_peered":"2024-06-03T09:06:14.433735+0000","last_clean":"2024-06-03T08:37:55.614497+0000","last_became_active":"2024-06-03T09:06:03.451203+0000","last_became_peered":"2024-06-03T09:06:03.451203+0000","last_unstale":"2024-06-03T09:06:14.433735+0000","last_undegraded":"2024-06-03T09:06:14.433735+0000","last_fullsized":"2024-06-03T09:06:03.375523+0000","mapping_epoch":999,"log_start":"0'0","ondisk_log_start":"0'0","created":28,"last_epoch_clean":255,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-06-03T08:37:55.614497+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-06-03T08:37:55.614497+0000","last_clean_scrub_stamp":"2024-06-03T08:37:55.614497+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-06-04T10:16:19.813495+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2],"acting":[2],"avail_no_missing":["2"],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"17.a","version":"0'0","reported_seq":925,"reported_epoch":1010,"state":"active+clean","last_fresh":"2024-06-03T09:06:18.651697+0000","last_change":"2024-06-03T08:38:05.843142+0000","last_active":"2024-06-03T09:06:18.651697+0000","last_peered":"2024-06-03T09:06:18.651697+0000","last_clean":"2024-06-03T09:06:18.651697+0000","last_became_active":"2024-06-03T08:38:05.809257+0000","last_became_peered":"2024-06-03T08:38:05.809257+0000","last_unstale":"2024-06-03T09:06:18.651697+0000","last_undegraded":"2024-06-03T09:06:18.651697+0000","last_fullsized":"2024-06-03T09:06:18.651697+0000","mapping_epoch":28,"log_start":"0'0","ondisk_log_start":"0'0","created":28,"last_epoch_clean":33,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-06-03T08:37:55.614497+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-06-03T08:37:55.614497+0000","last_clean_scrub_stamp":"2024-06-03T08:37:55.614497+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-06-04T12:12:03.686838+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2],"acting":[1,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"17.5","version":"0'0","reported_seq":943,"reported_epoch":1009,"state":"active+clean","last_fresh":"2024-06-03T09:06:14.442357+0000","last_change":"2024-06-03T08:38:23.466051+0000","last_active":"2024-06-03T09:06:14.442357+0000","last_peered":"2024-06-03T09:06:14.442357+0000","last_clean":"2024-06-03T09:06:14.442357+0000","last_became_active":"2024-06-03T08:38:23.458039+0000","last_became_peered":"2024-06-03T08:38:23.458039+0000","last_unstale":"2024-06-03T09:06:14.442357+0000","last_undegraded":"2024-06-03T09:06:14.442357+0000","last_fullsized":"2024-06-03T09:06:14.442357+0000","mapping_epoch":28,"log_start":"0'0","ondisk_log_start":"0'0","created":28,"last_epoch_clean":49,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-06-03T08:37:55.614497+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-06-03T08:37:55.614497+0000","last_clean_scrub_stamp":"2024-06-03T08:37:55.614497+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-06-04T12:27:02.532436+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,4],"acting":[0,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"17.4","version":"0'0","reported_seq":149,"reported_epoch":1009,"state":"active+undersized","last_fresh":"2024-06-03T09:06:13.941522+0000","last_change":"2024-06-03T09:06:03.441889+0000","last_active":"2024-06-03T09:06:13.941522+0000","last_peered":"2024-06-03T09:06:13.941522+0000","last_clean":"2024-06-03T08:37:55.614497+0000","last_became_active":"2024-06-03T09:06:03.441889+0000","last_became_peered":"2024-06-03T09:06:03.441889+0000","last_unstale":"2024-06-03T09:06:13.941522+0000","last_undegraded":"2024-06-03T09:06:13.941522+0000","last_fullsized":"2024-06-03T09:06:03.359582+0000","mapping_epoch":999,"log_start":"0'0","ondisk_log_start":"0'0","created":28,"last_epoch_clean":255,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-06-03T08:37:55.614497+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-06-03T08:37:55.614497+0000","last_clean_scrub_stamp":"2024-06-03T08:37:55.614497+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-06-04T12:49:22.907141+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7],"acting":[7],"avail_no_missing":["7"],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"17.7","version":"0'0","reported_seq":929,"reported_epoch":1009,"state":"active+clean","last_fresh":"2024-06-03T09:06:13.736920+0000","last_change":"2024-06-03T08:38:04.837083+0000","last_active":"2024-06-03T09:06:13.736920+0000","last_peered":"2024-06-03T09:06:13.736920+0000","last_clean":"2024-06-03T09:06:13.736920+0000","last_became_active":"2024-06-03T08:38:04.818246+0000","last_became_peered":"2024-06-03T08:38:04.818246+0000","last_unstale":"2024-06-03T09:06:13.736920+0000","last_undegraded":"2024-06-03T09:06:13.736920+0000","last_fullsized":"2024-06-03T09:06:13.736920+0000","mapping_epoch":28,"log_start":"0'0","ondisk_log_start":"0'0","created":28,"last_epoch_clean":34,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-06-03T08:37:55.614497+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-06-03T08:37:55.614497+0000","last_clean_scrub_stamp":"2024-06-03T08:37:55.614497+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-06-04T17:02:38.204813+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,2],"acting":[5,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"17.6","version":"0'0","reported_seq":960,"reported_epoch":1009,"state":"active+undersized","last_fresh":"2024-06-03T09:06:14.453110+0000","last_change":"2024-06-03T09:06:03.428677+0000","last_active":"2024-06-03T09:06:14.453110+0000","last_peered":"2024-06-03T09:06:14.453110+0000","last_clean":"2024-06-03T09:04:43.312352+0000","last_became_active":"2024-06-03T09:06:03.428677+0000","last_became_peered":"2024-06-03T09:06:03.428677+0000","last_unstale":"2024-06-03T09:06:14.453110+0000","last_undegraded":"2024-06-03T09:06:14.453110+0000","last_fullsized":"2024-06-03T09:06:03.359909+0000","mapping_epoch":999,"log_start":"0'0","ondisk_log_start":"0'0","created":28,"last_epoch_clean":255,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-06-03T08:37:55.614497+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-06-03T08:37:55.614497+0000","last_clean_scrub_stamp":"2024-06-03T08:37:55.614497+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-06-04T15:38:02.180241+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0],"acting":[0],"avail_no_missing":["0"],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"2.7","version":"0'0","reported_seq":940,"reported_epoch":1009,"state":"active+clean","last_fresh":"2024-06-03T09:06:13.684065+0000","last_change":"2024-06-03T08:36:30.385405+0000","last_active":"2024-06-03T09:06:13.684065+0000","last_peered":"2024-06-03T09:06:13.684065+0000","last_clean":"2024-06-03T09:06:13.684065+0000","last_became_active":"2024-06-03T08:36:30.380302+0000","last_became_peered":"2024-06-03T08:36:30.380302+0000","last_unstale":"2024-06-03T09:06:13.684065+0000","last_undegraded":"2024-06-03T09:06:13.684065+0000","last_fullsized":"2024-06-03T09:06:13.684065+0000","mapping_epoch":23,"log_start":"0'0","ondisk_log_start":"0'0","created":23,"last_epoch_clean":24,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-06-03T08:36:28.444764+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-06-03T08:36:28.444764+0000","last_clean_scrub_stamp":"2024-06-03T08:36:28.444764+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-06-04T18:46:33.475181+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0],"acting":[3,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"17.14","version":"0'0","reported_seq":926,"reported_epoch":1009,"state":"active+clean","last_fresh":"2024-06-03T09:06:13.721460+0000","last_change":"2024-06-03T08:38:04.547565+0000","last_active":"2024-06-03T09:06:13.721460+0000","last_peered":"2024-06-03T09:06:13.721460+0000","last_clean":"2024-06-03T09:06:13.721460+0000","last_became_active":"2024-06-03T08:38:04.541781+0000","last_became_peered":"2024-06-03T08:38:04.541781+0000","last_unstale":"2024-06-03T09:06:13.721460+0000","last_undegraded":"2024-06-03T09:06:13.721460+0000","last_fullsized":"2024-06-03T09:06:13.721460+0000","mapping_epoch":28,"log_start":"0'0","ondisk_log_start":"0'0","created":28,"last_epoch_clean":34,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-06-03T08:37:55.614497+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-06-03T08:37:55.614497+0000","last_clean_scrub_stamp":"2024-06-03T08:37:55.614497+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-06-04T16:41:55.170134+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,2],"acting":[5,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"2.6","version":"0'0","reported_seq":949,"reported_epoch":1009,"state":"active+clean","last_fresh":"2024-06-03T09:06:14.448100+0000","last_change":"2024-06-03T08:36:30.430760+0000","last_active":"2024-06-03T09:06:14.448100+0000","last_peered":"2024-06-03T09:06:14.448100+0000","last_clean":"2024-06-03T09:06:14.448100+0000","last_became_active":"2024-06-03T08:36:30.404665+0000","last_became_peered":"2024-06-03T08:36:30.404665+0000","last_unstale":"2024-06-03T09:06:14.448100+0000","last_undegraded":"2024-06-03T09:06:14.448100+0000","last_fullsized":"2024-06-03T09:06:14.448100+0000","mapping_epoch":23,"log_start":"0'0","ondisk_log_start":"0'0","created":23,"last_epoch_clean":24,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-06-03T08:36:28.444764+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-06-03T08:36:28.444764+0000","last_clean_scrub_stamp":"2024-06-03T08:36:28.444764+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-06-04T10:37:37.746074+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,0],"acting":[4,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"17.15","version":"0'0","reported_seq":934,"reported_epoch":1009,"state":"active+clean","last_fresh":"2024-06-03T09:06:13.672940+0000","last_change":"2024-06-03T08:38:20.999903+0000","last_active":"2024-06-03T09:06:13.672940+0000","last_peered":"2024-06-03T09:06:13.672940+0000","last_clean":"2024-06-03T09:06:13.672940+0000","last_became_active":"2024-06-03T08:38:20.979523+0000","last_became_peered":"2024-06-03T08:38:20.979523+0000","last_unstale":"2024-06-03T09:06:13.672940+0000","last_undegraded":"2024-06-03T09:06:13.672940+0000","last_fullsized":"2024-06-03T09:06:13.672940+0000","mapping_epoch":28,"log_start":"0'0","ondisk_log_start":"0'0","created":28,"last_epoch_clean":47,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-06-03T08:37:55.614497+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-06-03T08:37:55.614497+0000","last_clean_scrub_stamp":"2024-06-03T08:37:55.614497+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-06-04T20:29:58.314289+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,1],"acting":[3,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"2.5","version":"0'0","reported_seq":942,"reported_epoch":1010,"state":"active+clean","last_fresh":"2024-06-03T09:06:18.382767+0000","last_change":"2024-06-03T08:36:30.024235+0000","last_active":"2024-06-03T09:06:18.382767+0000","last_peered":"2024-06-03T09:06:18.382767+0000","last_clean":"2024-06-03T09:06:18.382767+0000","last_became_active":"2024-06-03T08:36:30.019040+0000","last_became_peered":"2024-06-03T08:36:30.019040+0000","last_unstale":"2024-06-03T09:06:18.382767+0000","last_undegraded":"2024-06-03T09:06:18.382767+0000","last_fullsized":"2024-06-03T09:06:18.382767+0000","mapping_epoch":23,"log_start":"0'0","ondisk_log_start":"0'0","created":23,"last_epoch_clean":24,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-06-03T08:36:28.444764+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-06-03T08:36:28.444764+0000","last_clean_scrub_stamp":"2024-06-03T08:36:28.444764+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-06-04T15:46:47.109502+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,4],"acting":[3,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"17.16","version":"0'0","reported_seq":972,"reported_epoch":1009,"state":"active+undersized","last_fresh":"2024-06-03T09:06:14.441665+0000","last_change":"2024-06-03T09:06:03.478615+0000","last_active":"2024-06-03T09:06:14.441665+0000","last_peered":"2024-06-03T09:06:14.441665+0000","last_clean":"2024-06-03T09:04:43.328415+0000","last_became_active":"2024-06-03T09:06:03.478615+0000","last_became_peered":"2024-06-03T09:06:03.478615+0000","last_unstale":"2024-06-03T09:06:14.441665+0000","last_undegraded":"2024-06-03T09:06:14.441665+0000","last_fullsized":"2024-06-03T09:06:03.402504+0000","mapping_epoch":999,"log_start":"0'0","ondisk_log_start":"0'0","created":28,"last_epoch_clean":255,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-06-03T08:37:55.614497+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-06-03T08:37:55.614497+0000","last_clean_scrub_stamp":"2024-06-03T08:37:55.614497+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-06-04T08:47:08.079233+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2],"acting":[2],"avail_no_missing":["2"],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"2.4","version":"0'0","reported_seq":952,"reported_epoch":1010,"state":"active+undersized","last_fresh":"2024-06-03T09:06:18.331485+0000","last_change":"2024-06-03T09:06:03.387171+0000","last_active":"2024-06-03T09:06:18.331485+0000","last_peered":"2024-06-03T09:06:18.331485+0000","last_clean":"2024-06-03T09:04:43.202645+0000","last_became_active":"2024-06-03T09:06:03.387171+0000","last_became_peered":"2024-06-03T09:06:03.387171+0000","last_unstale":"2024-06-03T09:06:18.331485+0000","last_undegraded":"2024-06-03T09:06:18.331485+0000","last_fullsized":"2024-06-03T09:06:03.283787+0000","mapping_epoch":999,"log_start":"0'0","ondisk_log_start":"0'0","created":23,"last_epoch_clean":255,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-06-03T08:36:28.444764+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-06-03T08:36:28.444764+0000","last_clean_scrub_stamp":"2024-06-03T08:36:28.444764+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-06-04T20:17:43.485023+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5],"acting":[5],"avail_no_missing":["5"],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"17.17","version":"0'0","reported_seq":954,"reported_epoch":1009,"state":"active+clean","last_fresh":"2024-06-03T09:06:14.447046+0000","last_change":"2024-06-03T08:38:38.483275+0000","last_active":"2024-06-03T09:06:14.447046+0000","last_peered":"2024-06-03T09:06:14.447046+0000","last_clean":"2024-06-03T09:06:14.447046+0000","last_became_active":"2024-06-03T08:38:38.467916+0000","last_became_peered":"2024-06-03T08:38:38.467916+0000","last_unstale":"2024-06-03T09:06:14.447046+0000","last_undegraded":"2024-06-03T09:06:14.447046+0000","last_fullsized":"2024-06-03T09:06:14.447046+0000","mapping_epoch":28,"log_start":"0'0","ondisk_log_start":"0'0","created":28,"last_epoch_clean":61,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-06-03T08:37:55.614497+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-06-03T08:37:55.614497+0000","last_clean_scrub_stamp":"2024-06-03T08:37:55.614497+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-06-04T10:02:23.036420+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,7],"acting":[2,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"2.2","version":"0'0","reported_seq":963,"reported_epoch":1009,"state":"active+undersized","last_fresh":"2024-06-03T09:06:13.657892+0000","last_change":"2024-06-03T09:06:03.410020+0000","last_active":"2024-06-03T09:06:13.657892+0000","last_peered":"2024-06-03T09:06:13.657892+0000","last_clean":"2024-06-03T09:05:37.145889+0000","last_became_active":"2024-06-03T09:06:03.410020+0000","last_became_peered":"2024-06-03T09:06:03.410020+0000","last_unstale":"2024-06-03T09:06:13.657892+0000","last_undegraded":"2024-06-03T09:06:13.657892+0000","last_fullsized":"2024-06-03T09:06:03.346864+0000","mapping_epoch":999,"log_start":"0'0","ondisk_log_start":"0'0","created":23,"last_epoch_clean":255,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-06-03T08:36:28.444764+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-06-03T08:36:28.444764+0000","last_clean_scrub_stamp":"2024-06-03T08:36:28.444764+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-06-04T09:57:12.423360+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3],"acting":[3],"avail_no_missing":["3"],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"17.11","version":"0'0","reported_seq":944,"reported_epoch":1009,"state":"active+clean","last_fresh":"2024-06-03T09:06:14.426670+0000","last_change":"2024-06-03T08:38:06.502302+0000","last_active":"2024-06-03T09:06:14.426670+0000","last_peered":"2024-06-03T09:06:14.426670+0000","last_clean":"2024-06-03T09:06:14.426670+0000","last_became_active":"2024-06-03T08:38:06.490204+0000","last_became_peered":"2024-06-03T08:38:06.490204+0000","last_unstale":"2024-06-03T09:06:14.426670+0000","last_undegraded":"2024-06-03T09:06:14.426670+0000","last_fullsized":"2024-06-03T09:06:14.426670+0000","mapping_epoch":28,"log_start":"0'0","ondisk_log_start":"0'0","created":28,"last_epoch_clean":32,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-06-03T08:37:55.614497+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-06-03T08:37:55.614497+0000","last_clean_scrub_stamp":"2024-06-03T08:37:55.614497+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-06-04T10:47:05.988958+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,2],"acting":[4,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"2.1","version":"0'0","reported_seq":952,"reported_epoch":1009,"state":"active+clean","last_fresh":"2024-06-03T09:06:13.929725+0000","last_change":"2024-06-03T08:36:30.883360+0000","last_active":"2024-06-03T09:06:13.929725+0000","last_peered":"2024-06-03T09:06:13.929725+0000","last_clean":"2024-06-03T09:06:13.929725+0000","last_became_active":"2024-06-03T08:36:30.855073+0000","last_became_peered":"2024-06-03T08:36:30.855073+0000","last_unstale":"2024-06-03T09:06:13.929725+0000","last_undegraded":"2024-06-03T09:06:13.929725+0000","last_fullsized":"2024-06-03T09:06:13.929725+0000","mapping_epoch":23,"log_start":"0'0","ondisk_log_start":"0'0","created":23,"last_epoch_clean":25,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-06-03T08:36:28.444764+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-06-03T08:36:28.444764+0000","last_clean_scrub_stamp":"2024-06-03T08:36:28.444764+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-06-04T18:12:22.363473+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,5],"acting":[7,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"17.12","version":"0'0","reported_seq":945,"reported_epoch":1009,"state":"active+clean","last_fresh":"2024-06-03T09:06:13.935141+0000","last_change":"2024-06-03T08:38:01.751562+0000","last_active":"2024-06-03T09:06:13.935141+0000","last_peered":"2024-06-03T09:06:13.935141+0000","last_clean":"2024-06-03T09:06:13.935141+0000","last_became_active":"2024-06-03T08:38:01.745027+0000","last_became_peered":"2024-06-03T08:38:01.745027+0000","last_unstale":"2024-06-03T09:06:13.935141+0000","last_undegraded":"2024-06-03T09:06:13.935141+0000","last_fullsized":"2024-06-03T09:06:13.935141+0000","mapping_epoch":28,"log_start":"0'0","ondisk_log_start":"0'0","created":28,"last_epoch_clean":32,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-06-03T08:37:55.614497+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-06-03T08:37:55.614497+0000","last_clean_scrub_stamp":"2024-06-03T08:37:55.614497+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-06-04T17:54:43.785326+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,3],"acting":[7,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"2.0","version":"0'0","reported_seq":943,"reported_epoch":1009,"state":"active+clean","last_fresh":"2024-06-03T09:06:13.648112+0000","last_change":"2024-06-03T08:36:29.984096+0000","last_active":"2024-06-03T09:06:13.648112+0000","last_peered":"2024-06-03T09:06:13.648112+0000","last_clean":"2024-06-03T09:06:13.648112+0000","last_became_active":"2024-06-03T08:36:29.957875+0000","last_became_peered":"2024-06-03T08:36:29.957875+0000","last_unstale":"2024-06-03T09:06:13.648112+0000","last_undegraded":"2024-06-03T09:06:13.648112+0000","last_fullsized":"2024-06-03T09:06:13.648112+0000","mapping_epoch":23,"log_start":"0'0","ondisk_log_start":"0'0","created":23,"last_epoch_clean":24,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-06-03T08:36:28.444764+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-06-03T08:36:28.444764+0000","last_clean_scrub_stamp":"2024-06-03T08:36:28.444764+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-06-04T08:41:33.679479+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,5],"acting":[3,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"17.13","version":"0'0","reported_seq":943,"reported_epoch":1009,"state":"active+clean","last_fresh":"2024-06-03T09:06:14.438399+0000","last_change":"2024-06-03T08:38:26.838316+0000","last_active":"2024-06-03T09:06:14.438399+0000","last_peered":"2024-06-03T09:06:14.438399+0000","last_clean":"2024-06-03T09:06:14.438399+0000","last_became_active":"2024-06-03T08:38:26.833214+0000","last_became_peered":"2024-06-03T08:38:26.833214+0000","last_unstale":"2024-06-03T09:06:14.438399+0000","last_undegraded":"2024-06-03T09:06:14.438399+0000","last_fullsized":"2024-06-03T09:06:14.438399+0000","mapping_epoch":28,"log_start":"0'0","ondisk_log_start":"0'0","created":28,"last_epoch_clean":52,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-06-03T08:37:55.614497+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-06-03T08:37:55.614497+0000","last_clean_scrub_stamp":"2024-06-03T08:37:55.614497+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-06-04T12:58:51.617998+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,0],"acting":[4,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"2.3","version":"0'0","reported_seq":943,"reported_epoch":1010,"state":"active+clean","last_fresh":"2024-06-03T09:06:18.362614+0000","last_change":"2024-06-03T08:36:30.351670+0000","last_active":"2024-06-03T09:06:18.362614+0000","last_peered":"2024-06-03T09:06:18.362614+0000","last_clean":"2024-06-03T09:06:18.362614+0000","last_became_active":"2024-06-03T08:36:30.343560+0000","last_became_peered":"2024-06-03T08:36:30.343560+0000","last_unstale":"2024-06-03T09:06:18.362614+0000","last_undegraded":"2024-06-03T09:06:18.362614+0000","last_fullsized":"2024-06-03T09:06:18.362614+0000","mapping_epoch":23,"log_start":"0'0","ondisk_log_start":"0'0","created":23,"last_epoch_clean":24,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-06-03T08:36:28.444764+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-06-03T08:36:28.444764+0000","last_clean_scrub_stamp":"2024-06-03T08:36:28.444764+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-06-04T12:02:24.765135+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,2],"acting":[3,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.0","version":"19'76","reported_seq":1051,"reported_epoch":1009,"state":"active+clean","last_fresh":"2024-06-03T09:06:13.703685+0000","last_change":"2024-06-03T08:36:26.702702+0000","last_active":"2024-06-03T09:06:13.703685+0000","last_peered":"2024-06-03T09:06:13.703685+0000","last_clean":"2024-06-03T09:06:13.703685+0000","last_became_active":"2024-06-03T08:36:26.128378+0000","last_became_peered":"2024-06-03T08:36:26.128378+0000","last_unstale":"2024-06-03T09:06:13.703685+0000","last_undegraded":"2024-06-03T09:06:13.703685+0000","last_fullsized":"2024-06-03T09:06:13.703685+0000","mapping_epoch":20,"log_start":"0'0","ondisk_log_start":"0'0","created":16,"last_epoch_clean":22,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-06-03T08:36:15.542635+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-06-03T08:36:15.542635+0000","last_clean_scrub_stamp":"2024-06-03T08:36:15.542635+0000","objects_scrubbed":0,"log_size":76,"log_dups_size":0,"ondisk_log_size":76,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-06-04T12:41:53.731360+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":459280,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":96,"num_read_kb":82,"num_write":113,"num_write_kb":1372,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2,"num_bytes_recovered":397840,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,5],"acting":[1,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"17.10","version":"0'0","reported_seq":944,"reported_epoch":1009,"state":"active+clean","last_fresh":"2024-06-03T09:06:14.422035+0000","last_change":"2024-06-03T08:38:15.399189+0000","last_active":"2024-06-03T09:06:14.422035+0000","last_peered":"2024-06-03T09:06:14.422035+0000","last_clean":"2024-06-03T09:06:14.422035+0000","last_became_active":"2024-06-03T08:38:15.337723+0000","last_became_peered":"2024-06-03T08:38:15.337723+0000","last_unstale":"2024-06-03T09:06:14.422035+0000","last_undegraded":"2024-06-03T09:06:14.422035+0000","last_fullsized":"2024-06-03T09:06:14.422035+0000","mapping_epoch":28,"log_start":"0'0","ondisk_log_start":"0'0","created":28,"last_epoch_clean":42,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-06-03T08:37:55.614497+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-06-03T08:37:55.614497+0000","last_clean_scrub_stamp":"2024-06-03T08:37:55.614497+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-06-04T09:29:46.427317+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,3],"acting":[4,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"17.1a","version":"0'0","reported_seq":960,"reported_epoch":1009,"state":"active+undersized","last_fresh":"2024-06-03T09:06:14.431224+0000","last_change":"2024-06-03T09:06:03.390734+0000","last_active":"2024-06-03T09:06:14.431224+0000","last_peered":"2024-06-03T09:06:14.431224+0000","last_clean":"2024-06-03T09:05:20.322313+0000","last_became_active":"2024-06-03T09:06:03.390734+0000","last_became_peered":"2024-06-03T09:06:03.390734+0000","last_unstale":"2024-06-03T09:06:14.431224+0000","last_undegraded":"2024-06-03T09:06:14.431224+0000","last_fullsized":"2024-06-03T09:06:03.343197+0000","mapping_epoch":999,"log_start":"0'0","ondisk_log_start":"0'0","created":28,"last_epoch_clean":254,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-06-03T08:37:55.614497+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-06-03T08:37:55.614497+0000","last_clean_scrub_stamp":"2024-06-03T08:37:55.614497+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-06-04T15:56:47.628875+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4],"acting":[4],"avail_no_missing":["4"],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"17.1b","version":"0'0","reported_seq":935,"reported_epoch":1009,"state":"active+clean","last_fresh":"2024-06-03T09:06:13.663279+0000","last_change":"2024-06-03T08:38:03.115748+0000","last_active":"2024-06-03T09:06:13.663279+0000","last_peered":"2024-06-03T09:06:13.663279+0000","last_clean":"2024-06-03T09:06:13.663279+0000","last_became_active":"2024-06-03T08:38:03.110533+0000","last_became_peered":"2024-06-03T08:38:03.110533+0000","last_unstale":"2024-06-03T09:06:13.663279+0000","last_undegraded":"2024-06-03T09:06:13.663279+0000","last_fullsized":"2024-06-03T09:06:13.663279+0000","mapping_epoch":28,"log_start":"0'0","ondisk_log_start":"0'0","created":28,"last_epoch_clean":32,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-06-03T08:37:55.614497+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-06-03T08:37:55.614497+0000","last_clean_scrub_stamp":"2024-06-03T08:37:55.614497+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-06-04T12:36:47.471609+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,7],"acting":[3,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"17.18","version":"0'0","reported_seq":944,"reported_epoch":1009,"state":"active+clean","last_fresh":"2024-06-03T09:06:13.924799+0000","last_change":"2024-06-03T08:38:04.632864+0000","last_active":"2024-06-03T09:06:13.924799+0000","last_peered":"2024-06-03T09:06:13.924799+0000","last_clean":"2024-06-03T09:06:13.924799+0000","last_became_active":"2024-06-03T08:38:04.627856+0000","last_became_peered":"2024-06-03T08:38:04.627856+0000","last_unstale":"2024-06-03T09:06:13.924799+0000","last_undegraded":"2024-06-03T09:06:13.924799+0000","last_fullsized":"2024-06-03T09:06:13.924799+0000","mapping_epoch":28,"log_start":"0'0","ondisk_log_start":"0'0","created":28,"last_epoch_clean":33,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-06-03T08:37:55.614497+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-06-03T08:37:55.614497+0000","last_clean_scrub_stamp":"2024-06-03T08:37:55.614497+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-06-04T19:55:51.181598+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,4],"acting":[7,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"17.19","version":"0'0","reported_seq":934,"reported_epoch":1009,"state":"active+clean","last_fresh":"2024-06-03T09:06:13.652943+0000","last_change":"2024-06-03T08:38:01.839491+0000","last_active":"2024-06-03T09:06:13.652943+0000","last_peered":"2024-06-03T09:06:13.652943+0000","last_clean":"2024-06-03T09:06:13.652943+0000","last_became_active":"2024-06-03T08:38:01.829539+0000","last_became_peered":"2024-06-03T08:38:01.829539+0000","last_unstale":"2024-06-03T09:06:13.652943+0000","last_undegraded":"2024-06-03T09:06:13.652943+0000","last_fullsized":"2024-06-03T09:06:13.652943+0000","mapping_epoch":28,"log_start":"0'0","ondisk_log_start":"0'0","created":28,"last_epoch_clean":30,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-06-03T08:37:55.614497+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-06-03T08:37:55.614497+0000","last_clean_scrub_stamp":"2024-06-03T08:37:55.614497+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-06-04T09:10:43.860229+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,2],"acting":[3,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"17.1e","version":"0'0","reported_seq":148,"reported_epoch":1009,"state":"active+undersized","last_fresh":"2024-06-03T09:06:14.445917+0000","last_change":"2024-06-03T09:06:03.406181+0000","last_active":"2024-06-03T09:06:14.445917+0000","last_peered":"2024-06-03T09:06:14.445917+0000","last_clean":"2024-06-03T08:37:55.614497+0000","last_became_active":"2024-06-03T09:06:03.406181+0000","last_became_peered":"2024-06-03T09:06:03.406181+0000","last_unstale":"2024-06-03T09:06:14.445917+0000","last_undegraded":"2024-06-03T09:06:14.445917+0000","last_fullsized":"2024-06-03T09:06:03.363739+0000","mapping_epoch":999,"log_start":"0'0","ondisk_log_start":"0'0","created":28,"last_epoch_clean":255,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-06-03T08:37:55.614497+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-06-03T08:37:55.614497+0000","last_clean_scrub_stamp":"2024-06-03T08:37:55.614497+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-06-04T11:45:24.486600+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4],"acting":[4],"avail_no_missing":["4"],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"17.1f","version":"0'0","reported_seq":941,"reported_epoch":1009,"state":"active+clean","last_fresh":"2024-06-03T09:06:14.450053+0000","last_change":"2024-06-03T08:38:04.383538+0000","last_active":"2024-06-03T09:06:14.450053+0000","last_peered":"2024-06-03T09:06:14.450053+0000","last_clean":"2024-06-03T09:06:14.450053+0000","last_became_active":"2024-06-03T08:38:04.367278+0000","last_became_peered":"2024-06-03T08:38:04.367278+0000","last_unstale":"2024-06-03T09:06:14.450053+0000","last_undegraded":"2024-06-03T09:06:14.450053+0000","last_fullsized":"2024-06-03T09:06:14.450053+0000","mapping_epoch":28,"log_start":"0'0","ondisk_log_start":"0'0","created":28,"last_epoch_clean":32,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-06-03T08:37:55.614497+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-06-03T08:37:55.614497+0000","last_clean_scrub_stamp":"2024-06-03T08:37:55.614497+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-06-04T10:39:26.552282+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,2],"acting":[0,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"17.1c","version":"0'0","reported_seq":944,"reported_epoch":1010,"state":"active+undersized","last_fresh":"2024-06-03T09:06:18.327492+0000","last_change":"2024-06-03T09:06:03.382210+0000","last_active":"2024-06-03T09:06:18.327492+0000","last_peered":"2024-06-03T09:06:18.327492+0000","last_clean":"2024-06-03T09:04:43.199660+0000","last_became_active":"2024-06-03T09:06:03.382210+0000","last_became_peered":"2024-06-03T09:06:03.382210+0000","last_unstale":"2024-06-03T09:06:18.327492+0000","last_undegraded":"2024-06-03T09:06:18.327492+0000","last_fullsized":"2024-06-03T09:06:03.277874+0000","mapping_epoch":999,"log_start":"0'0","ondisk_log_start":"0'0","created":28,"last_epoch_clean":255,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-06-03T08:37:55.614497+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-06-03T08:37:55.614497+0000","last_clean_scrub_stamp":"2024-06-03T08:37:55.614497+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-06-04T12:18:50.013884+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5],"acting":[5],"avail_no_missing":["5"],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"17.1d","version":"0'0","reported_seq":923,"reported_epoch":1009,"state":"active+clean","last_fresh":"2024-06-03T09:06:13.715906+0000","last_change":"2024-06-03T08:38:06.794630+0000","last_active":"2024-06-03T09:06:13.715906+0000","last_peered":"2024-06-03T09:06:13.715906+0000","last_clean":"2024-06-03T09:06:13.715906+0000","last_became_active":"2024-06-03T08:38:06.592317+0000","last_became_peered":"2024-06-03T08:38:06.592317+0000","last_unstale":"2024-06-03T09:06:13.715906+0000","last_undegraded":"2024-06-03T09:06:13.715906+0000","last_fullsized":"2024-06-03T09:06:13.715906+0000","mapping_epoch":28,"log_start":"0'0","ondisk_log_start":"0'0","created":28,"last_epoch_clean":33,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-06-03T08:37:55.614497+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-06-03T08:37:55.614497+0000","last_clean_scrub_stamp":"2024-06-03T08:37:55.614497+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-06-04T12:33:30.785369+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,7],"acting":[1,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"17.2","version":"0'0","reported_seq":925,"reported_epoch":1009,"state":"active+clean","last_fresh":"2024-06-03T09:06:13.708688+0000","last_change":"2024-06-03T08:38:14.981410+0000","last_active":"2024-06-03T09:06:13.708688+0000","last_peered":"2024-06-03T09:06:13.708688+0000","last_clean":"2024-06-03T09:06:13.708688+0000","last_became_active":"2024-06-03T08:38:14.973825+0000","last_became_peered":"2024-06-03T08:38:14.973825+0000","last_unstale":"2024-06-03T09:06:13.708688+0000","last_undegraded":"2024-06-03T09:06:13.708688+0000","last_fullsized":"2024-06-03T09:06:13.708688+0000","mapping_epoch":28,"log_start":"0'0","ondisk_log_start":"0'0","created":28,"last_epoch_clean":41,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-06-03T08:37:55.614497+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-06-03T08:37:55.614497+0000","last_clean_scrub_stamp":"2024-06-03T08:37:55.614497+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-06-04T17:01:05.518613+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,4],"acting":[1,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"17.3","version":"0'0","reported_seq":943,"reported_epoch":1009,"state":"active+clean","last_fresh":"2024-06-03T09:06:14.437636+0000","last_change":"2024-06-03T08:38:14.619745+0000","last_active":"2024-06-03T09:06:14.437636+0000","last_peered":"2024-06-03T09:06:14.437636+0000","last_clean":"2024-06-03T09:06:14.437636+0000","last_became_active":"2024-06-03T08:38:14.510665+0000","last_became_peered":"2024-06-03T08:38:14.510665+0000","last_unstale":"2024-06-03T09:06:14.437636+0000","last_undegraded":"2024-06-03T09:06:14.437636+0000","last_fullsized":"2024-06-03T09:06:14.437636+0000","mapping_epoch":28,"log_start":"0'0","ondisk_log_start":"0'0","created":28,"last_epoch_clean":42,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-06-03T08:37:55.614497+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-06-03T08:37:55.614497+0000","last_clean_scrub_stamp":"2024-06-03T08:37:55.614497+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-06-04T12:03:47.392064+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,7],"acting":[0,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"17.0","version":"0'0","reported_seq":944,"reported_epoch":1009,"state":"active+clean","last_fresh":"2024-06-03T09:06:14.427123+0000","last_change":"2024-06-03T08:38:04.237118+0000","last_active":"2024-06-03T09:06:14.427123+0000","last_peered":"2024-06-03T09:06:14.427123+0000","last_clean":"2024-06-03T09:06:14.427123+0000","last_became_active":"2024-06-03T08:38:04.231954+0000","last_became_peered":"2024-06-03T08:38:04.231954+0000","last_unstale":"2024-06-03T09:06:14.427123+0000","last_undegraded":"2024-06-03T09:06:14.427123+0000","last_fullsized":"2024-06-03T09:06:14.427123+0000","mapping_epoch":28,"log_start":"0'0","ondisk_log_start":"0'0","created":28,"last_epoch_clean":32,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-06-03T08:37:55.614497+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-06-03T08:37:55.614497+0000","last_clean_scrub_stamp":"2024-06-03T08:37:55.614497+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-06-04T09:56:10.741637+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,2],"acting":[0,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"17.1","version":"65'7","reported_seq":959,"reported_epoch":1009,"state":"active+undersized+degraded","last_fresh":"2024-06-03T09:06:13.709209+0000","last_change":"2024-06-03T09:06:03.370660+0000","last_active":"2024-06-03T09:06:13.709209+0000","last_peered":"2024-06-03T09:06:13.709209+0000","last_clean":"2024-06-03T09:05:03.733864+0000","last_became_active":"2024-06-03T09:06:03.370660+0000","last_became_peered":"2024-06-03T09:06:03.370660+0000","last_unstale":"2024-06-03T09:06:13.709209+0000","last_undegraded":"2024-06-03T09:06:03.266076+0000","last_fullsized":"2024-06-03T09:06:03.264525+0000","mapping_epoch":999,"log_start":"0'0","ondisk_log_start":"0'0","created":28,"last_epoch_clean":255,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-06-03T08:37:55.614497+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-06-03T08:37:55.614497+0000","last_clean_scrub_stamp":"2024-06-03T08:37:55.614497+0000","objects_scrubbed":0,"log_size":7,"log_dups_size":0,"ondisk_log_size":7,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-06-04T11:55:18.194283+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":5,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":1,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":6,"num_read_kb":6,"num_write":7,"num_write_kb":7,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5],"acting":[5],"avail_no_missing":["5"],"object_location_counts":[{"shards":"5","objects":1}],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]}],"pool_stats":[{"poolid":2,"num_pg":8,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":14,"acting":14,"num_store_stats":7},{"poolid":1,"num_pg":1,"stat_sum":{"num_bytes":459280,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":96,"num_read_kb":82,"num_write":113,"num_write_kb":1372,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2,"num_bytes_recovered":397840,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":802816,"data_stored":795680,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":76,"ondisk_log_size":76,"up":2,"acting":2,"num_store_stats":2},{"poolid":17,"num_pg":32,"stat_sum":{"num_bytes":5,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":1,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":6,"num_read_kb":6,"num_write":7,"num_write_kb":7,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":8192,"data_stored":10,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":7,"ondisk_log_size":7,"up":52,"acting":52,"num_store_stats":8}],"osd_stats":[{"osd":6,"up_from":253,"seq":1086626726200,"num_pgs":32,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":54772,"kb_used_data":5608,"kb_used_omap":1,"kb_used_meta":49150,"kb_avail":94317068,"statfs":{"total":96636764160,"available":96580677632,"internally_reserved":0,"allocated":5742592,"data_stored":2767180,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1593,"internal_metadata":50330055},"hb_peers":[0,1,2,3,4,5,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":109,"apply_latency_ms":109,"commit_latency_ns":109000000,"apply_latency_ns":109000000},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Mon Jun 3 09:04:02 2024","interfaces":[{"interface":"back","average":{"1min":1582.4780000000001,"5min":332.72699999999998,"15min":124.113},"min":{"1min":4.4829999999999997,"5min":4.1959999999999997,"15min":4.1959999999999997},"max":{"1min":22766.882000000001,"5min":22766.882000000001,"15min":22766.882000000001},"last":21.917999999999999},{"interface":"front","average":{"1min":1579.5409999999999,"5min":334.94499999999999,"15min":125.84999999999999},"min":{"1min":4.8129999999999997,"5min":4.7539999999999996,"15min":4.4530000000000003},"max":{"1min":22766.289000000001,"5min":22766.289000000001,"15min":22766.289000000001},"last":25.408999999999999}]},{"osd":1,"last update":"Mon Jun 3 09:03:40 2024","interfaces":[{"interface":"back","average":{"1min":36.420999999999999,"5min":77.054000000000002,"15min":132.84800000000001},"min":{"1min":6.0380000000000003,"5min":5.8799999999999999,"15min":5.484},"max":{"1min":589.64099999999996,"5min":4138.1679999999997,"15min":9955.1299999999992},"last":26.84},{"interface":"front","average":{"1min":40.240000000000002,"5min":79.674999999999997,"15min":133.38999999999999},"min":{"1min":8.4589999999999996,"5min":8.4090000000000007,"15min":6.1390000000000002},"max":{"1min":588.60799999999995,"5min":4137.6530000000002,"15min":9827.5509999999995},"last":26.263000000000002}]},{"osd":2,"last update":"Mon Jun 3 09:03:40 2024","interfaces":[{"interface":"back","average":{"1min":12.613,"5min":24.600999999999999,"15min":21.088000000000001},"min":{"1min":6.9740000000000002,"5min":6.9740000000000002,"15min":6.5330000000000004},"max":{"1min":26.076000000000001,"5min":376.24900000000002,"15min":376.24900000000002},"last":24.899000000000001},{"interface":"front","average":{"1min":12.493,"5min":22.207999999999998,"15min":21.219000000000001},"min":{"1min":6.5810000000000004,"5min":6.5110000000000001,"15min":6.0750000000000002},"max":{"1min":29.690999999999999,"5min":271.51100000000002,"15min":434.77300000000002},"last":40.372}]},{"osd":3,"last update":"Mon Jun 3 09:03:40 2024","interfaces":[{"interface":"back","average":{"1min":14.093999999999999,"5min":24.143999999999998,"15min":29.376999999999999},"min":{"1min":10.196999999999999,"5min":6.2919999999999998,"15min":6.1719999999999997},"max":{"1min":24.832000000000001,"5min":197.84899999999999,"15min":1235.5799999999999},"last":24.353999999999999},{"interface":"front","average":{"1min":9.3260000000000005,"5min":18.463999999999999,"15min":27.631},"min":{"1min":6.125,"5min":6.125,"15min":6.125},"max":{"1min":28.166,"5min":200.173,"15min":1157.6780000000001},"last":23.454000000000001}]},{"osd":4,"last update":"Mon Jun 3 09:04:29 2024","interfaces":[{"interface":"back","average":{"1min":10.279,"5min":1253.809,"15min":450.79199999999997},"min":{"1min":7.2300000000000004,"5min":6.093,"15min":6.093},"max":{"1min":15.000999999999999,"5min":28093.096000000001,"15min":28093.096000000001},"last":101.45399999999999},{"interface":"front","average":{"1min":15.693,"5min":1279.759,"15min":463.87099999999998},"min":{"1min":10.551,"5min":7.7480000000000002,"15min":7.7480000000000002},"max":{"1min":27.898,"5min":28323.837,"15min":28323.837},"last":102.262}]},{"osd":5,"last update":"Mon Jun 3 09:04:10 2024","interfaces":[{"interface":"back","average":{"1min":13.728999999999999,"5min":24.670000000000002,"15min":23.815999999999999},"min":{"1min":10.401,"5min":7.516,"15min":7.516},"max":{"1min":25.138000000000002,"5min":320.85000000000002,"15min":445.041},"last":103.52500000000001},{"interface":"front","average":{"1min":13.847,"5min":29.283000000000001,"15min":23.393999999999998},"min":{"1min":7.944,"5min":6.266,"15min":6.266},"max":{"1min":18.817,"5min":530.56700000000001,"15min":530.56700000000001},"last":29.263999999999999}]},{"osd":7,"last update":"Mon Jun 3 09:03:40 2024","interfaces":[{"interface":"back","average":{"1min":20.568999999999999,"5min":30.286999999999999,"15min":27.977},"min":{"1min":13.874000000000001,"5min":10.58,"15min":9.2859999999999996},"max":{"1min":86.959000000000003,"5min":222.07599999999999,"15min":255.08799999999999},"last":37.618000000000002},{"interface":"front","average":{"1min":19.102,"5min":29.550000000000001,"15min":27.763000000000002},"min":{"1min":10.052,"5min":8.6709999999999994,"15min":8.5429999999999993},"max":{"1min":67.968999999999994,"5min":293.62099999999998,"15min":438.14600000000002},"last":111.985}]}]},{"osd":7,"up_from":20,"seq":85899346286,"num_pgs":9,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":58964,"kb_used_data":5704,"kb_used_omap":1,"kb_used_meta":53246,"kb_avail":94312876,"statfs":{"total":96636764160,"available":96576385024,"internally_reserved":0,"allocated":5840896,"data_stored":2740727,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1589,"internal_metadata":54524363},"hb_peers":[0,1,2,3,4,5],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Mon Jun 3 09:06:09 2024","interfaces":[{"interface":"back","average":{"1min":14.073,"5min":894.22400000000005,"15min":308.93400000000003},"min":{"1min":5.5709999999999997,"5min":4.218,"15min":4.0599999999999996},"max":{"1min":90.757999999999996,"5min":21729.001,"15min":21729.001},"last":4.032},{"interface":"front","average":{"1min":9.2550000000000008,"5min":906.23500000000001,"15min":314.06},"min":{"1min":5.5789999999999997,"5min":4.8810000000000002,"15min":4.8810000000000002},"max":{"1min":25.937000000000001,"5min":21899.732,"15min":21899.732},"last":7.5819999999999999}]},{"osd":1,"last update":"Mon Jun 3 09:06:09 2024","interfaces":[{"interface":"back","average":{"1min":15.182,"5min":69.183999999999997,"15min":219.173},"min":{"1min":7.3639999999999999,"5min":6.6200000000000001,"15min":6.1319999999999997},"max":{"1min":76.926000000000002,"5min":2795.1329999999998,"15min":18373.257000000001},"last":9.7349999999999994},{"interface":"front","average":{"1min":21.495999999999999,"5min":74.703999999999994,"15min":225.285},"min":{"1min":8.8399999999999999,"5min":8.8399999999999999,"15min":8.5129999999999999},"max":{"1min":94.977999999999994,"5min":2803.3670000000002,"15min":18374.601999999999},"last":13.657999999999999}]},{"osd":2,"last update":"Mon Jun 3 09:06:09 2024","interfaces":[{"interface":"back","average":{"1min":19.164000000000001,"5min":26.902000000000001,"15min":22.920999999999999},"min":{"1min":11.207000000000001,"5min":8.5030000000000001,"15min":8.5030000000000001},"max":{"1min":91.319999999999993,"5min":333.99200000000002,"15min":333.99200000000002},"last":12.694000000000001},{"interface":"front","average":{"1min":11.125999999999999,"5min":18.132000000000001,"15min":16.353000000000002},"min":{"1min":6.202,"5min":5.8369999999999997,"15min":5.5069999999999997},"max":{"1min":28.649000000000001,"5min":226.18100000000001,"15min":226.18100000000001},"last":10.028}]},{"osd":3,"last update":"Mon Jun 3 09:06:09 2024","interfaces":[{"interface":"back","average":{"1min":22.291,"5min":26.289999999999999,"15min":31.074999999999999},"min":{"1min":11.714,"5min":6.8959999999999999,"15min":6.3760000000000003},"max":{"1min":83.102000000000004,"5min":316.64400000000001,"15min":2039.9059999999999},"last":14.345000000000001},{"interface":"front","average":{"1min":13.324999999999999,"5min":17.963000000000001,"15min":25.969999999999999},"min":{"1min":6.1200000000000001,"5min":5.2089999999999996,"15min":5.2089999999999996},"max":{"1min":49.908999999999999,"5min":233.542,"15min":2037.5550000000001},"last":9.5060000000000002}]},{"osd":4,"last update":"Mon Jun 3 09:05:28 2024","interfaces":[{"interface":"back","average":{"1min":22.209,"5min":1379.51,"15min":1803.278},"min":{"1min":5.9089999999999998,"5min":5.9089999999999998,"15min":5.9089999999999998},"max":{"1min":224.54300000000001,"5min":30073.917000000001,"15min":62560.868000000002},"last":11.138},{"interface":"front","average":{"1min":22.712,"5min":1401.0719999999999,"15min":1806.759},"min":{"1min":7.6449999999999996,"5min":6.7930000000000001,"15min":5.9340000000000002},"max":{"1min":223.58600000000001,"5min":30348.021000000001,"15min":62451.446000000004},"last":6.569}]},{"osd":5,"last update":"Mon Jun 3 09:06:02 2024","interfaces":[{"interface":"back","average":{"1min":24.372,"5min":26.856000000000002,"15min":23.571999999999999},"min":{"1min":12.278,"5min":9.2460000000000004,"15min":6.5650000000000004},"max":{"1min":183.64099999999999,"5min":317.572,"15min":317.572},"last":15.047000000000001},{"interface":"front","average":{"1min":21.259,"5min":23.364000000000001,"15min":21.193999999999999},"min":{"1min":7.8499999999999996,"5min":7.0999999999999996,"15min":7.008},"max":{"1min":184.53399999999999,"5min":306.60599999999999,"15min":306.60599999999999},"last":11.656000000000001}]},{"osd":6,"last update":"Mon Jun 3 09:04:31 2024","interfaces":[{"interface":"back","average":{"1min":44.061,"5min":26.821000000000002,"15min":91.176000000000002},"min":{"1min":9.0800000000000001,"5min":7.9390000000000001,"15min":7.9390000000000001},"max":{"1min":268.029,"5min":308.916,"15min":8734.4639999999999},"last":6.6769999999999996},{"interface":"front","average":{"1min":46.643000000000001,"5min":23.879999999999999,"15min":88.765000000000001},"min":{"1min":8.8119999999999994,"5min":6.9240000000000004,"15min":6.742},"max":{"1min":305.64600000000002,"5min":305.64600000000002,"15min":8735.5319999999992},"last":12.327}]}]},{"osd":5,"up_from":20,"seq":85899346260,"num_pgs":9,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":59360,"kb_used_data":6100,"kb_used_omap":1,"kb_used_meta":53246,"kb_avail":94312480,"statfs":{"total":96636764160,"available":96575979520,"internally_reserved":0,"allocated":6246400,"data_stored":3138572,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1594,"internal_metadata":54524358},"hb_peers":[0,1,2,3,4,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Mon Jun 3 09:06:06 2024","interfaces":[{"interface":"back","average":{"1min":7.468,"5min":1703.037,"15min":580.51900000000001},"min":{"1min":4.6509999999999998,"5min":4.6509999999999998,"15min":4.4020000000000001},"max":{"1min":14.711,"5min":26433.764999999999,"15min":26433.764999999999},"last":9.3849999999999998},{"interface":"front","average":{"1min":9.9429999999999996,"5min":1725.27,"15min":587.87800000000004},"min":{"1min":5.5999999999999996,"5min":5.1319999999999997,"15min":4.8170000000000002},"max":{"1min":13.590999999999999,"5min":26704.698,"15min":26704.698},"last":12.407999999999999}]},{"osd":1,"last update":"Mon Jun 3 09:06:00 2024","interfaces":[{"interface":"back","average":{"1min":10.862,"5min":67.731999999999999,"15min":217.364},"min":{"1min":7.1769999999999996,"5min":6.0890000000000004,"15min":5.4930000000000003},"max":{"1min":16.998999999999999,"5min":3923.4090000000001,"15min":18811.696},"last":12.917999999999999},{"interface":"front","average":{"1min":10.417,"5min":65.141999999999996,"15min":220.75700000000001},"min":{"1min":5.3209999999999997,"5min":5.3209999999999997,"15min":5.0209999999999999},"max":{"1min":15.821999999999999,"5min":4019.6129999999998,"15min":18968.365000000002},"last":8.6590000000000007}]},{"osd":2,"last update":"Mon Jun 3 09:06:00 2024","interfaces":[{"interface":"back","average":{"1min":11.366,"5min":24.18,"15min":21.137},"min":{"1min":5.6369999999999996,"5min":5.6369999999999996,"15min":5.0510000000000002},"max":{"1min":15.529999999999999,"5min":332.48200000000003,"15min":332.48200000000003},"last":13.375},{"interface":"front","average":{"1min":10.089,"5min":23.117000000000001,"15min":24.050000000000001},"min":{"1min":6.6769999999999996,"5min":6.6769999999999996,"15min":5.8739999999999997},"max":{"1min":15.821,"5min":353.03199999999998,"15min":353.03199999999998},"last":8.5139999999999993}]},{"osd":3,"last update":"Mon Jun 3 09:06:00 2024","interfaces":[{"interface":"back","average":{"1min":13.472,"5min":24.283999999999999,"15min":31.666},"min":{"1min":10.111000000000001,"5min":7.4720000000000004,"15min":7.4210000000000003},"max":{"1min":16.271999999999998,"5min":353.57499999999999,"15min":2855.8519999999999},"last":9.9960000000000004},{"interface":"front","average":{"1min":9.5899999999999999,"5min":15.875,"15min":25.834},"min":{"1min":6.6609999999999996,"5min":5.7729999999999997,"15min":5.1909999999999998},"max":{"1min":15.112,"5min":203.804,"15min":2869.7860000000001},"last":6.3769999999999998}]},{"osd":4,"last update":"Mon Jun 3 09:05:27 2024","interfaces":[{"interface":"back","average":{"1min":19.155000000000001,"5min":1084.961,"15min":1346.9949999999999},"min":{"1min":7.6130000000000004,"5min":7.6130000000000004,"15min":6.1340000000000003},"max":{"1min":156.55199999999999,"5min":28619.825000000001,"15min":57999.453999999998},"last":14.244999999999999},{"interface":"front","average":{"1min":28.068999999999999,"5min":1087.106,"15min":1330.289},"min":{"1min":8.3949999999999996,"5min":6.5030000000000001,"15min":6.5030000000000001},"max":{"1min":254.91300000000001,"5min":28621.294000000002,"15min":57434.631999999998},"last":14.859}]},{"osd":6,"last update":"Mon Jun 3 09:03:57 2024","interfaces":[{"interface":"back","average":{"1min":23.536000000000001,"5min":36.457000000000001,"15min":83.799999999999997},"min":{"1min":9.4800000000000004,"5min":9.4800000000000004,"15min":8.0630000000000006},"max":{"1min":89.442999999999998,"5min":508.77999999999997,"15min":9193.5169999999998},"last":157.79300000000001},{"interface":"front","average":{"1min":18.891999999999999,"5min":22.100999999999999,"15min":75.489999999999995},"min":{"1min":5.923,"5min":5.923,"15min":5.923},"max":{"1min":78.424999999999997,"5min":464.62,"15min":9191.4429999999993},"last":148.09}]},{"osd":7,"last update":"Mon Jun 3 09:06:00 2024","interfaces":[{"interface":"back","average":{"1min":13.257,"5min":20.917999999999999,"15min":21.088999999999999},"min":{"1min":10.475,"5min":6.7629999999999999,"15min":6.7629999999999999},"max":{"1min":17.245000000000001,"5min":211.22999999999999,"15min":211.22999999999999},"last":15.929},{"interface":"front","average":{"1min":12.872999999999999,"5min":20.587,"15min":24.021000000000001},"min":{"1min":8.5760000000000005,"5min":7.7709999999999999,"15min":6.8570000000000002},"max":{"1min":18.359999999999999,"5min":223.191,"15min":286.69499999999999},"last":10.608000000000001}]}]},{"osd":4,"up_from":20,"seq":85899346268,"num_pgs":12,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":63060,"kb_used_data":5704,"kb_used_omap":1,"kb_used_meta":57342,"kb_avail":94308780,"statfs":{"total":96636764160,"available":96572190720,"internally_reserved":0,"allocated":5840896,"data_stored":2740727,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1594,"internal_metadata":58718662},"hb_peers":[0,1,2,3,5,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Mon Jun 3 09:05:34 2024","interfaces":[{"interface":"back","average":{"1min":12.416,"5min":1099.4649999999999,"15min":382.15199999999999},"min":{"1min":4.5279999999999996,"5min":4.5279999999999996,"15min":4.3220000000000001},"max":{"1min":123.878,"5min":21726.743999999999,"15min":21726.743999999999},"last":6.7329999999999997},{"interface":"front","average":{"1min":16.462,"5min":1108.894,"15min":387.70299999999997},"min":{"1min":6.4089999999999998,"5min":5.8380000000000001,"15min":5.4909999999999997},"max":{"1min":130.185,"5min":21727.993999999999,"15min":21727.993999999999},"last":9.3309999999999995}]},{"osd":1,"last update":"Mon Jun 3 09:05:34 2024","interfaces":[{"interface":"back","average":{"1min":14.727,"5min":56.755000000000003,"15min":341.09399999999999},"min":{"1min":4.6529999999999996,"5min":4.1970000000000001,"15min":4.1970000000000001},"max":{"1min":126.899,"5min":3346.4389999999999,"15min":22256.098000000002},"last":15.904},{"interface":"front","average":{"1min":13.472,"5min":57.503,"15min":340.303},"min":{"1min":5.0590000000000002,"5min":5.0590000000000002,"15min":4.2640000000000002},"max":{"1min":126.096,"5min":3347.181,"15min":22247.317999999999},"last":8.0199999999999996}]},{"osd":2,"last update":"Mon Jun 3 09:05:34 2024","interfaces":[{"interface":"back","average":{"1min":26.088999999999999,"5min":23.866,"15min":20.620999999999999},"min":{"1min":7.0869999999999997,"5min":6.8239999999999998,"15min":6.3150000000000004},"max":{"1min":337.60399999999998,"5min":393.58600000000001,"15min":393.58600000000001},"last":9.9429999999999996},{"interface":"front","average":{"1min":15.800000000000001,"5min":22.584,"15min":23.245999999999999},"min":{"1min":6.0629999999999997,"5min":6.0629999999999997,"15min":6.0629999999999997},"max":{"1min":127.913,"5min":394.471,"15min":394.471},"last":6.4889999999999999}]},{"osd":3,"last update":"Mon Jun 3 09:05:34 2024","interfaces":[{"interface":"back","average":{"1min":17.544,"5min":19.143999999999998,"15min":18.742000000000001},"min":{"1min":5.8719999999999999,"5min":5.4100000000000001,"15min":5.4000000000000004},"max":{"1min":127.43600000000001,"5min":322.63,"15min":322.63},"last":11.32},{"interface":"front","average":{"1min":21.327000000000002,"5min":23.559000000000001,"15min":23.797000000000001},"min":{"1min":10.023,"5min":7.484,"15min":7.484},"max":{"1min":204.517,"5min":392.67500000000001,"15min":392.67500000000001},"last":13.484999999999999}]},{"osd":5,"last update":"Mon Jun 3 09:05:34 2024","interfaces":[{"interface":"back","average":{"1min":24.274999999999999,"5min":21.274999999999999,"15min":20.988},"min":{"1min":6.7489999999999997,"5min":6.7489999999999997,"15min":6.7489999999999997},"max":{"1min":268.10199999999998,"5min":344.23399999999998,"15min":344.23399999999998},"last":8.8209999999999997},{"interface":"front","average":{"1min":26.361999999999998,"5min":25.218,"15min":26.084},"min":{"1min":12.228,"5min":8.5180000000000007,"15min":8.5180000000000007},"max":{"1min":264.75,"5min":345.18700000000001,"15min":345.18700000000001},"last":12.875}]},{"osd":6,"last update":"Mon Jun 3 09:04:32 2024","interfaces":[{"interface":"back","average":{"1min":15.548,"5min":32.957999999999998,"15min":152.738},"min":{"1min":9.141,"5min":9.141,"15min":8.2319999999999993},"max":{"1min":52.57,"5min":395.238,"15min":12496.335999999999},"last":208.90000000000001},{"interface":"front","average":{"1min":12.680999999999999,"5min":25.428000000000001,"15min":149.30600000000001},"min":{"1min":6.4379999999999997,"5min":5.6319999999999997,"15min":5.6319999999999997},"max":{"1min":46.283000000000001,"5min":377.74299999999999,"15min":12616.297},"last":205.88900000000001}]},{"osd":7,"last update":"Mon Jun 3 09:05:34 2024","interfaces":[{"interface":"back","average":{"1min":18.295999999999999,"5min":19.585999999999999,"15min":21.062000000000001},"min":{"1min":6.2110000000000003,"5min":6.2110000000000003,"15min":5.8010000000000002},"max":{"1min":207.25299999999999,"5min":358.94900000000001,"15min":358.94900000000001},"last":14.076000000000001},{"interface":"front","average":{"1min":23.712,"5min":24.024999999999999,"15min":24.640000000000001},"min":{"1min":11.824999999999999,"5min":7.7789999999999999,"15min":7.7789999999999999},"max":{"1min":209.32900000000001,"5min":346.25200000000001,"15min":380.06900000000002},"last":14.923}]}]},{"osd":2,"up_from":20,"seq":85899346288,"num_pgs":11,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":58964,"kb_used_data":5704,"kb_used_omap":1,"kb_used_meta":53246,"kb_avail":94312876,"statfs":{"total":96636764160,"available":96576385024,"internally_reserved":0,"allocated":5840896,"data_stored":2740727,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1594,"internal_metadata":54524358},"hb_peers":[0,1,3,4,5,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Mon Jun 3 09:06:23 2024","interfaces":[{"interface":"back","average":{"1min":8.9629999999999992,"5min":946.25400000000002,"15min":326.36099999999999},"min":{"1min":5.3479999999999999,"5min":4.7050000000000001,"15min":4.1420000000000003},"max":{"1min":26.015000000000001,"5min":23465.18,"15min":23465.18},"last":7.2699999999999996},{"interface":"front","average":{"1min":7.6130000000000004,"5min":944.39800000000002,"15min":326.178},"min":{"1min":4.6790000000000003,"5min":4.6790000000000003,"15min":4.6790000000000003},"max":{"1min":20.802,"5min":23465.966,"15min":23465.966},"last":5.548}]},{"osd":1,"last update":"Mon Jun 3 09:06:23 2024","interfaces":[{"interface":"back","average":{"1min":13.007,"5min":76.507000000000005,"15min":294.72899999999998},"min":{"1min":6.4089999999999998,"5min":5.4199999999999999,"15min":5.4199999999999999},"max":{"1min":85.430000000000007,"5min":4220.5290000000005,"15min":18954.987000000001},"last":9.6739999999999995},{"interface":"front","average":{"1min":15.894,"5min":82.119,"15min":302.92399999999998},"min":{"1min":7.367,"5min":7.367,"15min":7.367},"max":{"1min":98.498000000000005,"5min":4325.9489999999996,"15min":19143.834999999999},"last":9.1620000000000008}]},{"osd":3,"last update":"Mon Jun 3 09:06:23 2024","interfaces":[{"interface":"back","average":{"1min":9.7579999999999991,"5min":12.444000000000001,"15min":28.065000000000001},"min":{"1min":7,"5min":5.9290000000000003,"15min":5.2450000000000001},"max":{"1min":28.164000000000001,"5min":118.913,"15min":2594.8150000000001},"last":10.461},{"interface":"front","average":{"1min":12.321999999999999,"5min":15.25,"15min":31.349},"min":{"1min":9.1240000000000006,"5min":8.0760000000000005,"15min":6.0780000000000003},"max":{"1min":32.360999999999997,"5min":156.34700000000001,"15min":2706.5120000000002},"last":12.317}]},{"osd":4,"last update":"Mon Jun 3 09:05:57 2024","interfaces":[{"interface":"back","average":{"1min":10.573,"5min":1266.894,"15min":438.154},"min":{"1min":8.6379999999999999,"5min":7.8789999999999996,"15min":6.2039999999999997},"max":{"1min":15.105,"5min":26660.861000000001,"15min":26660.861000000001},"last":11.643000000000001},{"interface":"front","average":{"1min":12.407999999999999,"5min":1266.7460000000001,"15min":435.75400000000002},"min":{"1min":10.266999999999999,"5min":10.266999999999999,"15min":6.2720000000000002},"max":{"1min":15.228,"5min":26616.243999999999,"15min":26616.243999999999},"last":13.852}]},{"osd":5,"last update":"Mon Jun 3 09:06:23 2024","interfaces":[{"interface":"back","average":{"1min":14.103,"5min":18.091000000000001,"15min":21.119},"min":{"1min":9.6349999999999998,"5min":9.2159999999999993,"15min":6.2050000000000001},"max":{"1min":32.890000000000001,"5min":252.31100000000001,"15min":252.31100000000001},"last":13.047000000000001},{"interface":"front","average":{"1min":11.545999999999999,"5min":16.170999999999999,"15min":20.254000000000001},"min":{"1min":7.6070000000000002,"5min":7.4850000000000003,"15min":6.8810000000000002},"max":{"1min":29.228999999999999,"5min":258.31700000000001,"15min":258.31700000000001},"last":8.6129999999999995}]},{"osd":6,"last update":"Mon Jun 3 09:04:28 2024","interfaces":[{"interface":"back","average":{"1min":14.063000000000001,"5min":24.850000000000001,"15min":106.755},"min":{"1min":9.0820000000000007,"5min":8.9649999999999999,"15min":8.9649999999999999},"max":{"1min":24.666,"5min":188.06700000000001,"15min":9521.0010000000002},"last":183.60900000000001},{"interface":"front","average":{"1min":13.933999999999999,"5min":21.081,"15min":106.142},"min":{"1min":5.8719999999999999,"5min":5.8719999999999999,"15min":5.8719999999999999},"max":{"1min":117.551,"5min":193.887,"15min":9585.6620000000003},"last":184.20599999999999}]},{"osd":7,"last update":"Mon Jun 3 09:06:23 2024","interfaces":[{"interface":"back","average":{"1min":17.056000000000001,"5min":20.055,"15min":23.268999999999998},"min":{"1min":10.353999999999999,"5min":6.6130000000000004,"15min":6.6130000000000004},"max":{"1min":88.974999999999994,"5min":242.56800000000001,"15min":242.56800000000001},"last":14.375999999999999},{"interface":"front","average":{"1min":18.885000000000002,"5min":21.460000000000001,"15min":25.489999999999998},"min":{"1min":12.497,"5min":9.0660000000000007,"15min":9.0660000000000007},"max":{"1min":102.229,"5min":181.56399999999999,"15min":214.33799999999999},"last":15.35}]}]},{"osd":3,"up_from":14,"seq":60129542502,"num_pgs":11,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":58964,"kb_used_data":5704,"kb_used_omap":1,"kb_used_meta":53246,"kb_avail":94312876,"statfs":{"total":96636764160,"available":96576385024,"internally_reserved":0,"allocated":5840896,"data_stored":2740727,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1592,"internal_metadata":54524360},"hb_peers":[0,1,2,4,5,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Mon Jun 3 09:05:25 2024","interfaces":[{"interface":"back","average":{"1min":8.4809999999999999,"5min":1064.2270000000001,"15min":367.88600000000002},"min":{"1min":6.0529999999999999,"5min":5.173,"15min":4.6929999999999996},"max":{"1min":12.010999999999999,"5min":23275.167000000001,"15min":23275.167000000001},"last":8.7289999999999992},{"interface":"front","average":{"1min":13.885,"5min":1070.3710000000001,"15min":373.53500000000003},"min":{"1min":9.6679999999999993,"5min":7.2089999999999996,"15min":5.9880000000000004},"max":{"1min":32.426000000000002,"5min":23278.536,"15min":23278.536},"last":12.621}]},{"osd":1,"last update":"Mon Jun 3 09:05:25 2024","interfaces":[{"interface":"back","average":{"1min":11.324,"5min":19.920000000000002,"15min":390.07900000000001},"min":{"1min":6.6379999999999999,"5min":4.8200000000000003,"15min":4.8200000000000003},"max":{"1min":26.449000000000002,"5min":250.72300000000001,"15min":21658.205999999998},"last":9.7889999999999997},{"interface":"front","average":{"1min":9.468,"5min":20.966999999999999,"15min":381.18799999999999},"min":{"1min":5.4969999999999999,"5min":5.2779999999999996,"15min":5.2779999999999996},"max":{"1min":25.643000000000001,"5min":335.33699999999999,"15min":21307.069},"last":7.8179999999999996}]},{"osd":2,"last update":"Mon Jun 3 09:05:25 2024","interfaces":[{"interface":"back","average":{"1min":13.215,"5min":26.649000000000001,"15min":23.783999999999999},"min":{"1min":7.8710000000000004,"5min":6.4900000000000002,"15min":5.875},"max":{"1min":34.322000000000003,"5min":377.803,"15min":377.803},"last":14.018000000000001},{"interface":"front","average":{"1min":13.105,"5min":22.292999999999999,"15min":20.573},"min":{"1min":7.327,"5min":6.125,"15min":5.9020000000000001},"max":{"1min":32.756999999999998,"5min":371.601,"15min":371.601},"last":6.9630000000000001}]},{"osd":4,"last update":"Mon Jun 3 09:05:27 2024","interfaces":[{"interface":"back","average":{"1min":10.444000000000001,"5min":1211.4970000000001,"15min":1639.1579999999999},"min":{"1min":6.4429999999999996,"5min":6.2560000000000002,"15min":6.0309999999999997},"max":{"1min":21.263000000000002,"5min":30144.939999999999,"15min":62679.622000000003},"last":5.6539999999999999},{"interface":"front","average":{"1min":11.363,"5min":1219.6400000000001,"15min":1644.127},"min":{"1min":7.8109999999999999,"5min":7.8109999999999999,"15min":6.2439999999999998},"max":{"1min":20.108000000000001,"5min":30180.689999999999,"15min":62681.021999999997},"last":11.103999999999999}]},{"osd":5,"last update":"Mon Jun 3 09:06:07 2024","interfaces":[{"interface":"back","average":{"1min":12.802,"5min":19.640999999999998,"15min":22.318000000000001},"min":{"1min":9.0890000000000004,"5min":6.8810000000000002,"15min":6.7690000000000001},"max":{"1min":16.745999999999999,"5min":249.88,"15min":327.34899999999999},"last":13.329000000000001},{"interface":"front","average":{"1min":15.002000000000001,"5min":24.003,"15min":26.649999999999999},"min":{"1min":11.984,"5min":9.9149999999999991,"15min":8.9879999999999995},"max":{"1min":18.407,"5min":251.47800000000001,"15min":331.86799999999999},"last":14.683999999999999}]},{"osd":6,"last update":"Mon Jun 3 09:04:04 2024","interfaces":[{"interface":"back","average":{"1min":22.815999999999999,"5min":31.513999999999999,"15min":181.74299999999999},"min":{"1min":9.4809999999999999,"5min":7.2240000000000002,"15min":7.2240000000000002},"max":{"1min":160.56999999999999,"5min":380.613,"15min":11893.74},"last":19.359999999999999},{"interface":"front","average":{"1min":25.359999999999999,"5min":30.981000000000002,"15min":179.81399999999999},"min":{"1min":9.8870000000000005,"5min":7.3360000000000003,"15min":6.0899999999999999},"max":{"1min":193.11099999999999,"5min":379.80700000000002,"15min":11893.069},"last":8.202}]},{"osd":7,"last update":"Mon Jun 3 09:05:25 2024","interfaces":[{"interface":"back","average":{"1min":12.177,"5min":21.126000000000001,"15min":22.972000000000001},"min":{"1min":8.6769999999999996,"5min":7.4279999999999999,"15min":7.4279999999999999},"max":{"1min":17.376999999999999,"5min":164.27500000000001,"15min":178.76300000000001},"last":9.3070000000000004},{"interface":"front","average":{"1min":15.675000000000001,"5min":22.722000000000001,"15min":25.238},"min":{"1min":11.233000000000001,"5min":10.555999999999999,"15min":7.6440000000000001},"max":{"1min":29.280000000000001,"5min":148.13800000000001,"15min":417.22800000000001},"last":10.32}]}]},{"osd":1,"up_from":14,"seq":60129542500,"num_pgs":6,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":59356,"kb_used_data":6096,"kb_used_omap":1,"kb_used_meta":53246,"kb_avail":94312484,"statfs":{"total":96636764160,"available":96575983616,"internally_reserved":0,"allocated":6242304,"data_stored":3138567,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":54524357},"hb_peers":[0,2,3,4,5,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Mon Jun 3 09:06:07 2024","interfaces":[{"interface":"back","average":{"1min":7.7530000000000001,"5min":185.66999999999999,"15min":77.347999999999999},"min":{"1min":5.0229999999999997,"5min":4.9279999999999999,"15min":4.5449999999999999},"max":{"1min":11.896000000000001,"5min":7470.616,"15min":7470.616},"last":7.2610000000000001},{"interface":"front","average":{"1min":9.9220000000000006,"5min":187.14599999999999,"15min":76.325999999999993},"min":{"1min":5.3159999999999998,"5min":5.3159999999999998,"15min":5.3159999999999998},"max":{"1min":18.338000000000001,"5min":7477.0829999999996,"15min":7477.0829999999996},"last":13.295999999999999}]},{"osd":2,"last update":"Mon Jun 3 09:06:20 2024","interfaces":[{"interface":"back","average":{"1min":10.340999999999999,"5min":17.369,"15min":23.670999999999999},"min":{"1min":6.9740000000000002,"5min":5.8650000000000002,"15min":5.8650000000000002},"max":{"1min":15.452,"5min":228,"15min":657.38},"last":9.1630000000000003},{"interface":"front","average":{"1min":13.387,"5min":23.355,"15min":31.524000000000001},"min":{"1min":9.077,"5min":9.077,"15min":7.2160000000000002},"max":{"1min":19.701000000000001,"5min":229.465,"15min":700.06500000000005},"last":14.991}]},{"osd":3,"last update":"Mon Jun 3 09:06:04 2024","interfaces":[{"interface":"back","average":{"1min":8.6110000000000007,"5min":15.733000000000001,"15min":26.646999999999998},"min":{"1min":5.827,"5min":5.4640000000000004,"15min":5.4640000000000004},"max":{"1min":13.105,"5min":121.619,"15min":1676.009},"last":6.2830000000000004},{"interface":"front","average":{"1min":9.5850000000000009,"5min":16.161999999999999,"15min":29.065999999999999},"min":{"1min":6.3630000000000004,"5min":5.7699999999999996,"15min":5.7699999999999996},"max":{"1min":15.166,"5min":132.036,"15min":1766.5840000000001},"last":11.605}]},{"osd":4,"last update":"Mon Jun 3 09:05:31 2024","interfaces":[{"interface":"back","average":{"1min":10.996,"5min":954.46500000000003,"15min":1281.6859999999999},"min":{"1min":7.702,"5min":6.1760000000000002,"15min":6.0970000000000004},"max":{"1min":21.538,"5min":28044.862000000001,"15min":59438.017999999996},"last":8.6010000000000009},{"interface":"front","average":{"1min":13.646000000000001,"5min":961.24599999999998,"15min":1285.26},"min":{"1min":9.7200000000000006,"5min":9.3940000000000001,"15min":6.4640000000000004},"max":{"1min":23.277999999999999,"5min":28045.624,"15min":59439.283000000003},"last":10.781000000000001}]},{"osd":5,"last update":"Mon Jun 3 09:06:04 2024","interfaces":[{"interface":"back","average":{"1min":12.478999999999999,"5min":20.228000000000002,"15min":25.997},"min":{"1min":8.6430000000000007,"5min":7.9720000000000004,"15min":7.234},"max":{"1min":17.442,"5min":139.43199999999999,"15min":701.22000000000003},"last":10.263999999999999},{"interface":"front","average":{"1min":13.507999999999999,"5min":25.760999999999999,"15min":27.553999999999998},"min":{"1min":10.042999999999999,"5min":7.258,"15min":6.242},"max":{"1min":20.611000000000001,"5min":278.63,"15min":598.35000000000002},"last":15.890000000000001}]},{"osd":6,"last update":"Mon Jun 3 09:03:54 2024","interfaces":[{"interface":"back","average":{"1min":28.524000000000001,"5min":20.654,"15min":23.878},"min":{"1min":6.0309999999999997,"5min":6.0309999999999997,"15min":6.0309999999999997},"max":{"1min":167.221,"5min":167.221,"15min":704.245},"last":18.721},{"interface":"front","average":{"1min":27.606000000000002,"5min":30.131,"15min":28.628},"min":{"1min":9.8650000000000002,"5min":9.7200000000000006,"15min":8.3580000000000005},"max":{"1min":123.938,"5min":358.23000000000002,"15min":634.05499999999995},"last":173.113}]},{"osd":7,"last update":"Mon Jun 3 09:06:20 2024","interfaces":[{"interface":"back","average":{"1min":14.66,"5min":23.234999999999999,"15min":31.097000000000001},"min":{"1min":11.250999999999999,"5min":8.0559999999999992,"15min":8.0559999999999992},"max":{"1min":21.507999999999999,"5min":270.95100000000002,"15min":703.07799999999997},"last":14.182},{"interface":"front","average":{"1min":15.548,"5min":28.760000000000002,"15min":28.925000000000001},"min":{"1min":12.759,"5min":8.2089999999999996,"15min":7.7599999999999998},"max":{"1min":22.408000000000001,"5min":360.99900000000002,"15min":608.44100000000003},"last":16.588000000000001}]}]},{"osd":0,"up_from":14,"seq":60129542502,"num_pgs":10,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":81108,"kb_used_data":5704,"kb_used_omap":1,"kb_used_meta":75390,"kb_avail":94290732,"statfs":{"total":96636764160,"available":96553709568,"internally_reserved":0,"allocated":5840896,"data_stored":2740727,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1593,"internal_metadata":77199815},"hb_peers":[1,2,3,4,5,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[{"osd":1,"last update":"Mon Jun 3 09:06:04 2024","interfaces":[{"interface":"back","average":{"1min":6.9960000000000004,"5min":56.997,"15min":339.29700000000003},"min":{"1min":4.4530000000000003,"5min":4.4530000000000003,"15min":4.4100000000000001},"max":{"1min":11.720000000000001,"5min":2414.4749999999999,"15min":20691.453000000001},"last":4.6040000000000001},{"interface":"front","average":{"1min":9.548,"5min":57.258000000000003,"15min":337.791},"min":{"1min":5.2300000000000004,"5min":5.2300000000000004,"15min":4.6050000000000004},"max":{"1min":15.337999999999999,"5min":2412.9549999999999,"15min":20640.792000000001},"last":7.7910000000000004}]},{"osd":2,"last update":"Mon Jun 3 09:06:22 2024","interfaces":[{"interface":"back","average":{"1min":11.154,"5min":26.515999999999998,"15min":22.882000000000001},"min":{"1min":5.7869999999999999,"5min":5.766,"15min":5.766},"max":{"1min":72.269999999999996,"5min":627.40700000000004,"15min":627.40700000000004},"last":8.6430000000000007},{"interface":"front","average":{"1min":14.353999999999999,"5min":33.304000000000002,"15min":29.885000000000002},"min":{"1min":8.2149999999999999,"5min":7.8630000000000004,"15min":7.4089999999999998},"max":{"1min":73.418000000000006,"5min":635.62599999999998,"15min":635.62599999999998},"last":10.872999999999999}]},{"osd":3,"last update":"Mon Jun 3 09:06:04 2024","interfaces":[{"interface":"back","average":{"1min":12.105,"5min":26.925000000000001,"15min":26.584},"min":{"1min":7.8410000000000002,"5min":7.1280000000000001,"15min":6.1909999999999998},"max":{"1min":18.425999999999998,"5min":389.72699999999998,"15min":389.72699999999998},"last":9.484},{"interface":"front","average":{"1min":13.794,"5min":29.771999999999998,"15min":28.815000000000001},"min":{"1min":10.084,"5min":9.4920000000000009,"15min":6.1849999999999996},"max":{"1min":17.373999999999999,"5min":390.23200000000003,"15min":494.92899999999997},"last":15.166}]},{"osd":4,"last update":"Mon Jun 3 09:05:27 2024","interfaces":[{"interface":"back","average":{"1min":13.478,"5min":1032.126,"15min":1389.729},"min":{"1min":6.5119999999999996,"5min":6.5119999999999996,"15min":6.5119999999999996},"max":{"1min":55.354999999999997,"5min":25555.212,"15min":60989.964},"last":11.698},{"interface":"front","average":{"1min":16.204999999999998,"5min":1030.8389999999999,"15min":1387.778},"min":{"1min":10.282999999999999,"5min":7.1429999999999998,"15min":7.1429999999999998},"max":{"1min":49.945999999999998,"5min":25556.159,"15min":61000.017},"last":12.278}]},{"osd":5,"last update":"Mon Jun 3 09:06:17 2024","interfaces":[{"interface":"back","average":{"1min":9.7040000000000006,"5min":26.533999999999999,"15min":28.683},"min":{"1min":6.8129999999999997,"5min":6.8129999999999997,"15min":6.8129999999999997},"max":{"1min":14.942,"5min":455.78699999999998,"15min":605.05899999999997},"last":7.2480000000000002},{"interface":"front","average":{"1min":14.279,"5min":28.398,"15min":26.192},"min":{"1min":11.696,"5min":6.4420000000000002,"15min":6.2030000000000003},"max":{"1min":17.879000000000001,"5min":391.31200000000001,"15min":541.11000000000001},"last":14.297000000000001}]},{"osd":6,"last update":"Mon Jun 3 09:04:30 2024","interfaces":[{"interface":"back","average":{"1min":52.496000000000002,"5min":33.139000000000003,"15min":160.83199999999999},"min":{"1min":7.5289999999999999,"5min":7.1040000000000001,"15min":6.3739999999999997},"max":{"1min":457.91699999999997,"5min":457.91699999999997,"15min":11115.258},"last":18.959},{"interface":"front","average":{"1min":49.670000000000002,"5min":33.704999999999998,"15min":162.791},"min":{"1min":12.247999999999999,"5min":11.019,"15min":9.9760000000000009},"max":{"1min":407.05099999999999,"5min":407.05099999999999,"15min":11092.194},"last":17.707000000000001}]},{"osd":7,"last update":"Mon Jun 3 09:06:22 2024","interfaces":[{"interface":"back","average":{"1min":11.625999999999999,"5min":26.375,"15min":28.972999999999999},"min":{"1min":6.4480000000000004,"5min":6.4480000000000004,"15min":6.4480000000000004},"max":{"1min":15.352,"5min":461.41699999999997,"15min":598.33699999999999},"last":10.250999999999999},{"interface":"front","average":{"1min":14.013,"5min":29.696000000000002,"15min":33.630000000000003},"min":{"1min":10.938000000000001,"5min":10.58,"15min":10.263999999999999},"max":{"1min":16.603000000000002,"5min":463.10599999999999,"15min":592.80499999999995},"last":13.254}]}]}],"pool_statfs":[{"poolid":1,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":401408,"data_stored":397840,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":5,"total":0,"available":0,"internally_reserved":0,"allocated":401408,"data_stored":397840,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":4,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":5,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":6,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":17,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":17,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":17,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":17,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":17,"osd":4,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":17,"osd":5,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":5,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":17,"osd":6,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":5,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":17,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0}]}} 2024-06-03T09:06:31.285 DEBUG:teuthology.orchestra.run.smithi088:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph pg dump --format=json 2024-06-03T09:06:31.414 INFO:teuthology.orchestra.run.smithi088.stderr:2024-06-03T09:06:31.414+0000 7f96a7180640 10 monclient: get_monmap_and_config 2024-06-03T09:06:31.414 INFO:teuthology.orchestra.run.smithi088.stderr:2024-06-03T09:06:31.414+0000 7f96a7180640 10 monclient: build_initial_monmap 2024-06-03T09:06:31.414 INFO:teuthology.orchestra.run.smithi088.stderr:2024-06-03T09:06:31.414+0000 7f96a7180640 10 monclient: monmap: 2024-06-03T09:06:31.415 INFO:teuthology.orchestra.run.smithi088.stderr:epoch 0 2024-06-03T09:06:31.415 INFO:teuthology.orchestra.run.smithi088.stderr:fsid ec3c3fdd-3ac6-4d7d-9412-683130ed8b04 2024-06-03T09:06:31.415 INFO:teuthology.orchestra.run.smithi088.stderr:last_changed 2024-06-03T09:06:31.415535+0000 2024-06-03T09:06:31.415 INFO:teuthology.orchestra.run.smithi088.stderr:created 2024-06-03T09:06:31.415535+0000 2024-06-03T09:06:31.415 INFO:teuthology.orchestra.run.smithi088.stderr:min_mon_release 0 (unknown) 2024-06-03T09:06:31.415 INFO:teuthology.orchestra.run.smithi088.stderr:election_strategy: 3 2024-06-03T09:06:31.415 INFO:teuthology.orchestra.run.smithi088.stderr:0: v1:172.21.15.88:6789/0 mon.noname-a 2024-06-03T09:06:31.415 INFO:teuthology.orchestra.run.smithi088.stderr:1: v1:172.21.15.192:6789/0 mon.noname-c 2024-06-03T09:06:31.415 INFO:teuthology.orchestra.run.smithi088.stderr:2: v1:172.21.15.88:6790/0 mon.noname-b 2024-06-03T09:06:31.415 INFO:teuthology.orchestra.run.smithi088.stderr: 2024-06-03T09:06:31.416 INFO:teuthology.orchestra.run.smithi088.stderr:2024-06-03T09:06:31.415+0000 7f96a7180640 1 Processor -- start 2024-06-03T09:06:31.416 INFO:teuthology.orchestra.run.smithi088.stderr:2024-06-03T09:06:31.415+0000 7f96a7180640 1 -- start start 2024-06-03T09:06:31.416 INFO:teuthology.orchestra.run.smithi088.stderr:2024-06-03T09:06:31.415+0000 7f96a7180640 10 monclient: init 2024-06-03T09:06:31.416 INFO:teuthology.orchestra.run.smithi088.stderr:2024-06-03T09:06:31.415+0000 7f96a7180640 10 monclient: _reopen_session rank -1 2024-06-03T09:06:31.417 INFO:teuthology.orchestra.run.smithi088.stderr:2024-06-03T09:06:31.415+0000 7f96a7180640 10 monclient: _add_conns ranks=[0,2,1] 2024-06-03T09:06:31.417 INFO:teuthology.orchestra.run.smithi088.stderr:2024-06-03T09:06:31.415+0000 7f96a7180640 10 monclient(hunting): picked mon.noname-a con 0x7f96a006fc70 addr v1:172.21.15.88:6789/0 2024-06-03T09:06:31.417 INFO:teuthology.orchestra.run.smithi088.stderr:2024-06-03T09:06:31.415+0000 7f96a7180640 10 monclient(hunting): picked mon.noname-b con 0x7f96a00ad920 addr v1:172.21.15.88:6790/0 2024-06-03T09:06:31.417 INFO:teuthology.orchestra.run.smithi088.stderr:2024-06-03T09:06:31.415+0000 7f96a7180640 10 monclient(hunting): picked mon.noname-c con 0x7f96a01393f0 addr v1:172.21.15.192:6789/0 2024-06-03T09:06:31.417 INFO:teuthology.orchestra.run.smithi088.stderr:2024-06-03T09:06:31.415+0000 7f96a7180640 1 -- --> v1:172.21.15.88:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f96a0070420 con 0x7f96a006fc70 2024-06-03T09:06:31.417 INFO:teuthology.orchestra.run.smithi088.stderr:2024-06-03T09:06:31.416+0000 7f96a7180640 1 -- --> v1:172.21.15.192:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f96a0071050 con 0x7f96a01393f0 2024-06-03T09:06:31.417 INFO:teuthology.orchestra.run.smithi088.stderr:2024-06-03T09:06:31.416+0000 7f96a7180640 1 -- --> v1:172.21.15.88:6790/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f96a00add60 con 0x7f96a00ad920 2024-06-03T09:06:31.417 INFO:teuthology.orchestra.run.smithi088.stderr:2024-06-03T09:06:31.416+0000 7f96a7180640 10 monclient(hunting): _renew_subs 2024-06-03T09:06:31.417 INFO:teuthology.orchestra.run.smithi088.stderr:2024-06-03T09:06:31.416+0000 7f96a7180640 10 monclient(hunting): authenticate will time out at 3047.225954s 2024-06-03T09:06:31.417 INFO:teuthology.orchestra.run.smithi088.stderr:2024-06-03T09:06:31.416+0000 7f96a4f1c640 1 --1- >> v1:172.21.15.88:6789/0 conn(0x7f96a006fc70 0x7f96a013fbb0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.88:6789/0 says I am v1:172.21.15.88:41556/0 (socket says 172.21.15.88:41556) 2024-06-03T09:06:31.417 INFO:teuthology.orchestra.run.smithi088.stderr:2024-06-03T09:06:31.416+0000 7f96a4f1c640 1 -- 172.21.15.88:0/2089866776 learned_addr learned my addr 172.21.15.88:0/2089866776 (peer_addr_for_me v1:172.21.15.88:0/0) 2024-06-03T09:06:31.418 INFO:teuthology.orchestra.run.smithi088.stderr:2024-06-03T09:06:31.418+0000 7f969f7fe640 1 -- 172.21.15.88:0/2089866776 <== mon.2 v1:172.21.15.88:6790/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 3174150504 0 0) 0x7f96a00add60 con 0x7f96a00ad920 2024-06-03T09:06:31.419 INFO:teuthology.orchestra.run.smithi088.stderr:2024-06-03T09:06:31.418+0000 7f969f7fe640 10 monclient(hunting): _negotiate 2024-06-03T09:06:31.419 INFO:teuthology.orchestra.run.smithi088.stderr:2024-06-03T09:06:31.418+0000 7f969f7fe640 10 monclient(hunting): _init_auth method 2 2024-06-03T09:06:31.419 INFO:teuthology.orchestra.run.smithi088.stderr:2024-06-03T09:06:31.418+0000 7f969f7fe640 10 monclient(hunting): _init_auth creating new auth 2024-06-03T09:06:31.419 INFO:teuthology.orchestra.run.smithi088.stderr:2024-06-03T09:06:31.418+0000 7f969f7fe640 10 monclient(hunting): my global_id is 30128 2024-06-03T09:06:31.419 INFO:teuthology.orchestra.run.smithi088.stderr:2024-06-03T09:06:31.418+0000 7f969f7fe640 1 -- 172.21.15.88:0/2089866776 --> v1:172.21.15.88:6790/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f9690003520 con 0x7f96a00ad920 2024-06-03T09:06:31.419 INFO:teuthology.orchestra.run.smithi088.stderr:2024-06-03T09:06:31.419+0000 7f969f7fe640 1 -- 172.21.15.88:0/2089866776 <== mon.0 v1:172.21.15.88:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 3859613168 0 0) 0x7f96a0070420 con 0x7f96a006fc70 2024-06-03T09:06:31.419 INFO:teuthology.orchestra.run.smithi088.stderr:2024-06-03T09:06:31.419+0000 7f969f7fe640 10 monclient(hunting): _negotiate 2024-06-03T09:06:31.419 INFO:teuthology.orchestra.run.smithi088.stderr:2024-06-03T09:06:31.419+0000 7f969f7fe640 10 monclient(hunting): _init_auth method 2 2024-06-03T09:06:31.419 INFO:teuthology.orchestra.run.smithi088.stderr:2024-06-03T09:06:31.419+0000 7f969f7fe640 10 monclient(hunting): _init_auth creating new auth 2024-06-03T09:06:31.419 INFO:teuthology.orchestra.run.smithi088.stderr:2024-06-03T09:06:31.419+0000 7f969f7fe640 10 monclient(hunting): my global_id is 30624 2024-06-03T09:06:31.419 INFO:teuthology.orchestra.run.smithi088.stderr:2024-06-03T09:06:31.419+0000 7f969f7fe640 1 -- 172.21.15.88:0/2089866776 --> v1:172.21.15.88:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f96a00add60 con 0x7f96a006fc70 2024-06-03T09:06:31.419 INFO:teuthology.orchestra.run.smithi088.stderr:2024-06-03T09:06:31.419+0000 7f969f7fe640 1 -- 172.21.15.88:0/2089866776 <== mon.1 v1:172.21.15.192:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 4171889366 0 0) 0x7f96a0071050 con 0x7f96a01393f0 2024-06-03T09:06:31.419 INFO:teuthology.orchestra.run.smithi088.stderr:2024-06-03T09:06:31.419+0000 7f969f7fe640 10 monclient(hunting): _negotiate 2024-06-03T09:06:31.420 INFO:teuthology.orchestra.run.smithi088.stderr:2024-06-03T09:06:31.419+0000 7f969f7fe640 10 monclient(hunting): _init_auth method 2 2024-06-03T09:06:31.420 INFO:teuthology.orchestra.run.smithi088.stderr:2024-06-03T09:06:31.419+0000 7f969f7fe640 10 monclient(hunting): _init_auth creating new auth 2024-06-03T09:06:31.420 INFO:teuthology.orchestra.run.smithi088.stderr:2024-06-03T09:06:31.419+0000 7f969f7fe640 10 monclient(hunting): my global_id is 30205 2024-06-03T09:06:31.420 INFO:teuthology.orchestra.run.smithi088.stderr:2024-06-03T09:06:31.419+0000 7f969f7fe640 1 -- 172.21.15.88:0/2089866776 --> v1:172.21.15.192:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f96a0070420 con 0x7f96a01393f0 2024-06-03T09:06:31.422 INFO:teuthology.orchestra.run.smithi088.stderr:2024-06-03T09:06:31.422+0000 7f969f7fe640 1 -- 172.21.15.88:0/2089866776 <== mon.1 v1:172.21.15.192:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 214+0+0 (unknown 3444660370 0 0) 0x7f96a0070420 con 0x7f96a01393f0 2024-06-03T09:06:31.422 INFO:teuthology.orchestra.run.smithi088.stderr:2024-06-03T09:06:31.422+0000 7f969f7fe640 1 -- 172.21.15.88:0/2089866776 >> v1:172.21.15.88:6790/0 conn(0x7f96a00ad920 legacy=0x7f96a0138ce0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-06-03T09:06:31.422 INFO:teuthology.orchestra.run.smithi088.stderr:2024-06-03T09:06:31.422+0000 7f969f7fe640 1 -- 172.21.15.88:0/2089866776 >> v1:172.21.15.88:6789/0 conn(0x7f96a006fc70 legacy=0x7f96a013fbb0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-06-03T09:06:31.422 INFO:teuthology.orchestra.run.smithi088.stderr:2024-06-03T09:06:31.422+0000 7f969f7fe640 10 monclient: _finish_hunting 0 2024-06-03T09:06:31.422 INFO:teuthology.orchestra.run.smithi088.stderr:2024-06-03T09:06:31.422+0000 7f969f7fe640 1 monclient: found mon.noname-c 2024-06-03T09:06:31.422 INFO:teuthology.orchestra.run.smithi088.stderr:2024-06-03T09:06:31.422+0000 7f969f7fe640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-06-03T09:06:31.423 INFO:teuthology.orchestra.run.smithi088.stderr:2024-06-03T09:06:31.422+0000 7f969f7fe640 10 monclient: _send_mon_message to mon.noname-c at v1:172.21.15.192:6789/0 2024-06-03T09:06:31.423 INFO:teuthology.orchestra.run.smithi088.stderr:2024-06-03T09:06:31.422+0000 7f969f7fe640 1 -- 172.21.15.88:0/2089866776 --> v1:172.21.15.192:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f96a0071be0 con 0x7f96a01393f0 2024-06-03T09:06:31.423 INFO:teuthology.orchestra.run.smithi088.stderr:2024-06-03T09:06:31.422+0000 7f969f7fe640 10 monclient: _finish_auth 0 2024-06-03T09:06:31.423 INFO:teuthology.orchestra.run.smithi088.stderr:2024-06-03T09:06:31.422+0000 7f969f7fe640 10 monclient: _check_auth_tickets 2024-06-03T09:06:31.423 INFO:teuthology.orchestra.run.smithi088.stderr:2024-06-03T09:06:31.422+0000 7f969f7fe640 20 monclient: _check_auth_rotating not needed by client.admin 2024-06-03T09:06:31.423 INFO:teuthology.orchestra.run.smithi088.stderr:2024-06-03T09:06:31.422+0000 7f969f7fe640 1 -- 172.21.15.88:0/2089866776 <== mon.1 v1:172.21.15.192:6789/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 2234374065 0 0) 0x7f9698001930 con 0x7f96a01393f0 2024-06-03T09:06:31.423 INFO:teuthology.orchestra.run.smithi088.stderr:2024-06-03T09:06:31.422+0000 7f969f7fe640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-06-03T09:06:31.423 INFO:teuthology.orchestra.run.smithi088.stderr:2024-06-03T09:06:31.422+0000 7f969f7fe640 10 monclient: got monmap 1 from mon.noname-c (according to old e1) 2024-06-03T09:06:31.423 INFO:teuthology.orchestra.run.smithi088.stderr:2024-06-03T09:06:31.422+0000 7f969f7fe640 10 monclient: dump: 2024-06-03T09:06:31.423 INFO:teuthology.orchestra.run.smithi088.stderr:epoch 1 2024-06-03T09:06:31.423 INFO:teuthology.orchestra.run.smithi088.stderr:fsid ec3c3fdd-3ac6-4d7d-9412-683130ed8b04 2024-06-03T09:06:31.423 INFO:teuthology.orchestra.run.smithi088.stderr:last_changed 2024-06-03T08:34:31.885125+0000 2024-06-03T09:06:31.423 INFO:teuthology.orchestra.run.smithi088.stderr:created 2024-06-03T08:34:31.885125+0000 2024-06-03T09:06:31.423 INFO:teuthology.orchestra.run.smithi088.stderr:min_mon_release 18 (reef) 2024-06-03T09:06:31.424 INFO:teuthology.orchestra.run.smithi088.stderr:election_strategy: 3 2024-06-03T09:06:31.424 INFO:teuthology.orchestra.run.smithi088.stderr:0: v1:172.21.15.88:6789/0 mon.a 2024-06-03T09:06:31.424 INFO:teuthology.orchestra.run.smithi088.stderr:1: v1:172.21.15.192:6789/0 mon.b 2024-06-03T09:06:31.424 INFO:teuthology.orchestra.run.smithi088.stderr:2: v1:172.21.15.88:6790/0 mon.c 2024-06-03T09:06:31.424 INFO:teuthology.orchestra.run.smithi088.stderr: 2024-06-03T09:06:31.424 INFO:teuthology.orchestra.run.smithi088.stderr:2024-06-03T09:06:31.422+0000 7f96a7180640 5 monclient: authenticate success, global_id 30205 2024-06-03T09:06:31.424 INFO:teuthology.orchestra.run.smithi088.stderr:2024-06-03T09:06:31.422+0000 7f96a7180640 20 monclient: get_monmap_and_config waiting for monmap|config 2024-06-03T09:06:31.425 INFO:teuthology.orchestra.run.smithi088.stderr:2024-06-03T09:06:31.424+0000 7f969f7fe640 1 -- 172.21.15.88:0/2089866776 <== mon.1 v1:172.21.15.192:6789/0 4 ==== config(0 keys) v1 ==== 4+0+0 (unknown 0 0 0) 0x7f9698001b50 con 0x7f96a01393f0 2024-06-03T09:06:31.426 INFO:teuthology.orchestra.run.smithi088.stderr:2024-06-03T09:06:31.424+0000 7f969f7fe640 10 monclient: handle_config config(0 keys) v1 2024-06-03T09:06:31.426 INFO:teuthology.orchestra.run.smithi088.stderr:2024-06-03T09:06:31.424+0000 7f969f7fe640 1 -- 172.21.15.88:0/2089866776 <== mon.1 v1:172.21.15.192:6789/0 5 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 2234374065 0 0) 0x7f9698001e60 con 0x7f96a01393f0 2024-06-03T09:06:31.426 INFO:teuthology.orchestra.run.smithi088.stderr:2024-06-03T09:06:31.424+0000 7f969f7fe640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-06-03T09:06:31.426 INFO:teuthology.orchestra.run.smithi088.stderr:2024-06-03T09:06:31.424+0000 7f969f7fe640 10 monclient: got monmap 1 from mon.b (according to old e1) 2024-06-03T09:06:31.426 INFO:teuthology.orchestra.run.smithi088.stderr:2024-06-03T09:06:31.424+0000 7f969f7fe640 10 monclient: dump: 2024-06-03T09:06:31.426 INFO:teuthology.orchestra.run.smithi088.stderr:epoch 1 2024-06-03T09:06:31.426 INFO:teuthology.orchestra.run.smithi088.stderr:fsid ec3c3fdd-3ac6-4d7d-9412-683130ed8b04 2024-06-03T09:06:31.426 INFO:teuthology.orchestra.run.smithi088.stderr:last_changed 2024-06-03T08:34:31.885125+0000 2024-06-03T09:06:31.426 INFO:teuthology.orchestra.run.smithi088.stderr:created 2024-06-03T08:34:31.885125+0000 2024-06-03T09:06:31.426 INFO:teuthology.orchestra.run.smithi088.stderr:min_mon_release 18 (reef) 2024-06-03T09:06:31.426 INFO:teuthology.orchestra.run.smithi088.stderr:election_strategy: 3 2024-06-03T09:06:31.426 INFO:teuthology.orchestra.run.smithi088.stderr:0: v1:172.21.15.88:6789/0 mon.a 2024-06-03T09:06:31.426 INFO:teuthology.orchestra.run.smithi088.stderr:1: v1:172.21.15.192:6789/0 mon.b 2024-06-03T09:06:31.427 INFO:teuthology.orchestra.run.smithi088.stderr:2: v1:172.21.15.88:6790/0 mon.c 2024-06-03T09:06:31.427 INFO:teuthology.orchestra.run.smithi088.stderr: 2024-06-03T09:06:31.427 INFO:teuthology.orchestra.run.smithi088.stderr:2024-06-03T09:06:31.424+0000 7f96a7180640 10 monclient: get_monmap_and_config success 2024-06-03T09:06:31.427 INFO:teuthology.orchestra.run.smithi088.stderr:2024-06-03T09:06:31.424+0000 7f96a7180640 4 set_mon_vals no callback set 2024-06-03T09:06:31.427 INFO:teuthology.orchestra.run.smithi088.stderr:2024-06-03T09:06:31.424+0000 7f96a7180640 10 monclient: shutdown 2024-06-03T09:06:31.427 INFO:teuthology.orchestra.run.smithi088.stderr:2024-06-03T09:06:31.424+0000 7f96a7180640 20 monclient: shutdown discarding 0 pending message(s) 2024-06-03T09:06:31.427 INFO:teuthology.orchestra.run.smithi088.stderr:2024-06-03T09:06:31.424+0000 7f96a7180640 1 -- 172.21.15.88:0/2089866776 >> v1:172.21.15.192:6789/0 conn(0x7f96a01393f0 legacy=0x7f96a014ac50 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-06-03T09:06:31.427 INFO:teuthology.orchestra.run.smithi088.stderr:2024-06-03T09:06:31.424+0000 7f96a7180640 1 -- 172.21.15.88:0/2089866776 shutdown_connections 2024-06-03T09:06:31.427 INFO:teuthology.orchestra.run.smithi088.stderr:2024-06-03T09:06:31.424+0000 7f96a7180640 1 -- 172.21.15.88:0/2089866776 >> 172.21.15.88:0/2089866776 conn(0x7f96a00b2f20 msgr2=0x7f96a00b3300 unknown :-1 s=STATE_NONE l=0).mark_down 2024-06-03T09:06:31.427 INFO:teuthology.orchestra.run.smithi088.stderr:2024-06-03T09:06:31.424+0000 7f96a7180640 1 -- 172.21.15.88:0/2089866776 shutdown_connections 2024-06-03T09:06:31.428 INFO:teuthology.orchestra.run.smithi088.stderr:2024-06-03T09:06:31.424+0000 7f96a7180640 1 -- 172.21.15.88:0/2089866776 wait complete. 2024-06-03T09:06:31.428 INFO:teuthology.orchestra.run.smithi088.stderr:2024-06-03T09:06:31.425+0000 7f96a7180640 10 monclient: build_initial_monmap 2024-06-03T09:06:31.428 INFO:teuthology.orchestra.run.smithi088.stderr:2024-06-03T09:06:31.425+0000 7f96a7180640 10 monclient: monmap: 2024-06-03T09:06:31.428 INFO:teuthology.orchestra.run.smithi088.stderr:epoch 0 2024-06-03T09:06:31.428 INFO:teuthology.orchestra.run.smithi088.stderr:fsid 00000000-0000-0000-0000-000000000000 2024-06-03T09:06:31.428 INFO:teuthology.orchestra.run.smithi088.stderr:last_changed 0.000000 2024-06-03T09:06:31.428 INFO:teuthology.orchestra.run.smithi088.stderr:created 0.000000 2024-06-03T09:06:31.428 INFO:teuthology.orchestra.run.smithi088.stderr:min_mon_release 0 (unknown) 2024-06-03T09:06:31.428 INFO:teuthology.orchestra.run.smithi088.stderr:election_strategy: 1 2024-06-03T09:06:31.428 INFO:teuthology.orchestra.run.smithi088.stderr:0: v1:172.21.15.88:6789/0 mon.noname-a 2024-06-03T09:06:31.428 INFO:teuthology.orchestra.run.smithi088.stderr:1: v1:172.21.15.192:6789/0 mon.noname-b 2024-06-03T09:06:31.428 INFO:teuthology.orchestra.run.smithi088.stderr:2: v1:172.21.15.88:6790/0 mon.noname-c 2024-06-03T09:06:31.428 INFO:teuthology.orchestra.run.smithi088.stderr: 2024-06-03T09:06:31.429 INFO:teuthology.orchestra.run.smithi088.stderr:2024-06-03T09:06:31.425+0000 7f96a7180640 1 librados: starting msgr at 2024-06-03T09:06:31.429 INFO:teuthology.orchestra.run.smithi088.stderr:2024-06-03T09:06:31.425+0000 7f96a7180640 1 librados: starting objecter 2024-06-03T09:06:31.429 INFO:teuthology.orchestra.run.smithi088.stderr:2024-06-03T09:06:31.425+0000 7f96a7180640 1 Processor -- start 2024-06-03T09:06:31.429 INFO:teuthology.orchestra.run.smithi088.stderr:2024-06-03T09:06:31.426+0000 7f96a7180640 1 -- start start 2024-06-03T09:06:31.429 INFO:teuthology.orchestra.run.smithi088.stderr:2024-06-03T09:06:31.426+0000 7f96a7180640 1 librados: setting wanted keys 2024-06-03T09:06:31.429 INFO:teuthology.orchestra.run.smithi088.stderr:2024-06-03T09:06:31.426+0000 7f96a7180640 1 librados: calling monclient init 2024-06-03T09:06:31.429 INFO:teuthology.orchestra.run.smithi088.stderr:2024-06-03T09:06:31.426+0000 7f96a7180640 10 monclient: init 2024-06-03T09:06:31.431 INFO:teuthology.orchestra.run.smithi088.stderr:2024-06-03T09:06:31.426+0000 7f96a7180640 10 monclient: _reopen_session rank -1 2024-06-03T09:06:31.432 INFO:teuthology.orchestra.run.smithi088.stderr:2024-06-03T09:06:31.426+0000 7f96a7180640 10 monclient: _add_conns ranks=[0,1,2] 2024-06-03T09:06:31.432 INFO:teuthology.orchestra.run.smithi088.stderr:2024-06-03T09:06:31.426+0000 7f96a7180640 10 monclient(hunting): picked mon.noname-a con 0x7f96a006fc70 addr v1:172.21.15.88:6789/0 2024-06-03T09:06:31.432 INFO:teuthology.orchestra.run.smithi088.stderr:2024-06-03T09:06:31.426+0000 7f96a7180640 10 monclient(hunting): picked mon.noname-b con 0x7f96a00ad920 addr v1:172.21.15.192:6789/0 2024-06-03T09:06:31.432 INFO:teuthology.orchestra.run.smithi088.stderr:2024-06-03T09:06:31.426+0000 7f96a7180640 10 monclient(hunting): picked mon.noname-c con 0x7f96a01393f0 addr v1:172.21.15.88:6790/0 2024-06-03T09:06:31.432 INFO:teuthology.orchestra.run.smithi088.stderr:2024-06-03T09:06:31.426+0000 7f96a7180640 1 -- --> v1:172.21.15.88:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f96a01d4b80 con 0x7f96a006fc70 2024-06-03T09:06:31.432 INFO:teuthology.orchestra.run.smithi088.stderr:2024-06-03T09:06:31.426+0000 7f96a7180640 1 -- --> v1:172.21.15.192:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f96a01d5a00 con 0x7f96a00ad920 2024-06-03T09:06:31.432 INFO:teuthology.orchestra.run.smithi088.stderr:2024-06-03T09:06:31.426+0000 7f96a7180640 1 -- --> v1:172.21.15.88:6790/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f96a00aaa10 con 0x7f96a01393f0 2024-06-03T09:06:31.432 INFO:teuthology.orchestra.run.smithi088.stderr:2024-06-03T09:06:31.426+0000 7f96a7180640 10 monclient(hunting): _renew_subs 2024-06-03T09:06:31.432 INFO:teuthology.orchestra.run.smithi088.stderr:2024-06-03T09:06:31.426+0000 7f96a7180640 10 monclient(hunting): authenticate will time out at 3047.236775s 2024-06-03T09:06:31.432 INFO:teuthology.orchestra.run.smithi088.stderr:2024-06-03T09:06:31.427+0000 7f96a4f1c640 1 --1- >> v1:172.21.15.88:6789/0 conn(0x7f96a006fc70 0x7f96a01d52f0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=1).handle_server_banner_and_identify peer v1:172.21.15.88:6789/0 says I am v1:172.21.15.88:41560/0 (socket says 172.21.15.88:41560) 2024-06-03T09:06:31.432 INFO:teuthology.orchestra.run.smithi088.stderr:2024-06-03T09:06:31.427+0000 7f96a4f1c640 1 -- 172.21.15.88:0/2599947638 learned_addr learned my addr 172.21.15.88:0/2599947638 (peer_addr_for_me v1:172.21.15.88:0/0) 2024-06-03T09:06:31.432 INFO:teuthology.orchestra.run.smithi088.stderr:2024-06-03T09:06:31.427+0000 7f969d7fa640 10 client.?.objecter ms_handle_connect 0x7f96a006fc70 2024-06-03T09:06:31.432 INFO:teuthology.orchestra.run.smithi088.stderr:2024-06-03T09:06:31.427+0000 7f969d7fa640 10 client.?.objecter resend_mon_ops 2024-06-03T09:06:31.433 INFO:teuthology.orchestra.run.smithi088.stderr:2024-06-03T09:06:31.427+0000 7f969d7fa640 10 client.?.objecter ms_handle_connect 0x7f96a00ad920 2024-06-03T09:06:31.433 INFO:teuthology.orchestra.run.smithi088.stderr:2024-06-03T09:06:31.427+0000 7f969d7fa640 10 client.?.objecter resend_mon_ops 2024-06-03T09:06:31.433 INFO:teuthology.orchestra.run.smithi088.stderr:2024-06-03T09:06:31.427+0000 7f969d7fa640 10 client.?.objecter ms_handle_connect 0x7f96a01393f0 2024-06-03T09:06:31.433 INFO:teuthology.orchestra.run.smithi088.stderr:2024-06-03T09:06:31.427+0000 7f969d7fa640 10 client.?.objecter resend_mon_ops 2024-06-03T09:06:31.433 INFO:teuthology.orchestra.run.smithi088.stderr:2024-06-03T09:06:31.429+0000 7f969d7fa640 1 -- 172.21.15.88:0/2599947638 <== mon.1 v1:172.21.15.192:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 1499094079 0 0) 0x7f96a01d5a00 con 0x7f96a00ad920 2024-06-03T09:06:31.433 INFO:teuthology.orchestra.run.smithi088.stderr:2024-06-03T09:06:31.429+0000 7f969d7fa640 10 monclient(hunting): _negotiate 2024-06-03T09:06:31.433 INFO:teuthology.orchestra.run.smithi088.stderr:2024-06-03T09:06:31.429+0000 7f969d7fa640 10 monclient(hunting): _init_auth method 2 2024-06-03T09:06:31.433 INFO:teuthology.orchestra.run.smithi088.stderr:2024-06-03T09:06:31.429+0000 7f969d7fa640 10 monclient(hunting): _init_auth creating new auth 2024-06-03T09:06:31.433 INFO:teuthology.orchestra.run.smithi088.stderr:2024-06-03T09:06:31.429+0000 7f969d7fa640 10 monclient(hunting): my global_id is 30208 2024-06-03T09:06:31.433 INFO:teuthology.orchestra.run.smithi088.stderr:2024-06-03T09:06:31.429+0000 7f969d7fa640 1 -- 172.21.15.88:0/2599947638 --> v1:172.21.15.192:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f9684004eb0 con 0x7f96a00ad920 2024-06-03T09:06:31.433 INFO:teuthology.orchestra.run.smithi088.stderr:2024-06-03T09:06:31.429+0000 7f969d7fa640 1 -- 172.21.15.88:0/2599947638 <== mon.0 v1:172.21.15.88:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 3464196164 0 0) 0x7f96a01d4b80 con 0x7f96a006fc70 2024-06-03T09:06:31.433 INFO:teuthology.orchestra.run.smithi088.stderr:2024-06-03T09:06:31.429+0000 7f969d7fa640 10 monclient(hunting): _negotiate 2024-06-03T09:06:31.433 INFO:teuthology.orchestra.run.smithi088.stderr:2024-06-03T09:06:31.429+0000 7f969d7fa640 10 monclient(hunting): _init_auth method 2 2024-06-03T09:06:31.433 INFO:teuthology.orchestra.run.smithi088.stderr:2024-06-03T09:06:31.429+0000 7f969d7fa640 10 monclient(hunting): _init_auth creating new auth 2024-06-03T09:06:31.434 INFO:teuthology.orchestra.run.smithi088.stderr:2024-06-03T09:06:31.429+0000 7f969d7fa640 10 monclient(hunting): my global_id is 30627 2024-06-03T09:06:31.434 INFO:teuthology.orchestra.run.smithi088.stderr:2024-06-03T09:06:31.429+0000 7f969d7fa640 1 -- 172.21.15.88:0/2599947638 --> v1:172.21.15.88:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f96a01d5a00 con 0x7f96a006fc70 2024-06-03T09:06:31.434 INFO:teuthology.orchestra.run.smithi088.stderr:2024-06-03T09:06:31.430+0000 7f969d7fa640 1 -- 172.21.15.88:0/2599947638 <== mon.2 v1:172.21.15.88:6790/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 1208397560 0 0) 0x7f96a00aaa10 con 0x7f96a01393f0 2024-06-03T09:06:31.434 INFO:teuthology.orchestra.run.smithi088.stderr:2024-06-03T09:06:31.430+0000 7f969d7fa640 10 monclient(hunting): _negotiate 2024-06-03T09:06:31.434 INFO:teuthology.orchestra.run.smithi088.stderr:2024-06-03T09:06:31.430+0000 7f969d7fa640 10 monclient(hunting): _init_auth method 2 2024-06-03T09:06:31.434 INFO:teuthology.orchestra.run.smithi088.stderr:2024-06-03T09:06:31.430+0000 7f969d7fa640 10 monclient(hunting): _init_auth creating new auth 2024-06-03T09:06:31.434 INFO:teuthology.orchestra.run.smithi088.stderr:2024-06-03T09:06:31.430+0000 7f969d7fa640 10 monclient(hunting): my global_id is 30131 2024-06-03T09:06:31.434 INFO:teuthology.orchestra.run.smithi088.stderr:2024-06-03T09:06:31.430+0000 7f969d7fa640 1 -- 172.21.15.88:0/2599947638 --> v1:172.21.15.88:6790/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f96a01d4b80 con 0x7f96a01393f0 2024-06-03T09:06:31.435 INFO:teuthology.orchestra.run.smithi088.stderr:2024-06-03T09:06:31.434+0000 7f969d7fa640 1 -- 172.21.15.88:0/2599947638 <== mon.1 v1:172.21.15.192:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 764+0+0 (unknown 1334979111 0 0) 0x7f9684004eb0 con 0x7f96a00ad920 2024-06-03T09:06:31.435 INFO:teuthology.orchestra.run.smithi088.stderr:2024-06-03T09:06:31.434+0000 7f969d7fa640 1 -- 172.21.15.88:0/2599947638 --> v1:172.21.15.192:6789/0 -- auth(proto 2 165 bytes epoch 0) v1 -- 0x7f96a00aaa10 con 0x7f96a00ad920 2024-06-03T09:06:31.436 INFO:teuthology.orchestra.run.smithi088.stderr:2024-06-03T09:06:31.434+0000 7f969d7fa640 1 -- 172.21.15.88:0/2599947638 <== mon.1 v1:172.21.15.192:6789/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 2234374065 0 0) 0x7f968c0018d0 con 0x7f96a00ad920 2024-06-03T09:06:31.436 INFO:teuthology.orchestra.run.smithi088.stderr:2024-06-03T09:06:31.434+0000 7f969d7fa640 10 monclient(hunting): handle_monmap mon_map magic: 0 v1 2024-06-03T09:06:31.436 INFO:teuthology.orchestra.run.smithi088.stderr:2024-06-03T09:06:31.434+0000 7f969d7fa640 10 monclient(hunting): got monmap 1 from mon.noname-b (according to old e1) 2024-06-03T09:06:31.436 INFO:teuthology.orchestra.run.smithi088.stderr:2024-06-03T09:06:31.434+0000 7f969d7fa640 10 monclient(hunting): dump: 2024-06-03T09:06:31.436 INFO:teuthology.orchestra.run.smithi088.stderr:epoch 1 2024-06-03T09:06:31.436 INFO:teuthology.orchestra.run.smithi088.stderr:fsid ec3c3fdd-3ac6-4d7d-9412-683130ed8b04 2024-06-03T09:06:31.436 INFO:teuthology.orchestra.run.smithi088.stderr:last_changed 2024-06-03T08:34:31.885125+0000 2024-06-03T09:06:31.436 INFO:teuthology.orchestra.run.smithi088.stderr:created 2024-06-03T08:34:31.885125+0000 2024-06-03T09:06:31.436 INFO:teuthology.orchestra.run.smithi088.stderr:min_mon_release 18 (reef) 2024-06-03T09:06:31.436 INFO:teuthology.orchestra.run.smithi088.stderr:election_strategy: 3 2024-06-03T09:06:31.436 INFO:teuthology.orchestra.run.smithi088.stderr:0: v1:172.21.15.88:6789/0 mon.a 2024-06-03T09:06:31.436 INFO:teuthology.orchestra.run.smithi088.stderr:1: v1:172.21.15.192:6789/0 mon.b 2024-06-03T09:06:31.436 INFO:teuthology.orchestra.run.smithi088.stderr:2: v1:172.21.15.88:6790/0 mon.c 2024-06-03T09:06:31.437 INFO:teuthology.orchestra.run.smithi088.stderr: 2024-06-03T09:06:31.437 INFO:teuthology.orchestra.run.smithi088.stderr:2024-06-03T09:06:31.434+0000 7f969d7fa640 10 monclient(hunting): _finish_auth 0 2024-06-03T09:06:31.437 INFO:teuthology.orchestra.run.smithi088.stderr:2024-06-03T09:06:31.434+0000 7f969d7fa640 1 -- 172.21.15.88:0/2599947638 <== mon.0 v1:172.21.15.88:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 764+0+0 (unknown 35497135 0 0) 0x7f96a01d5a00 con 0x7f96a006fc70 2024-06-03T09:06:31.437 INFO:teuthology.orchestra.run.smithi088.stderr:2024-06-03T09:06:31.434+0000 7f969d7fa640 1 -- 172.21.15.88:0/2599947638 --> v1:172.21.15.88:6789/0 -- auth(proto 2 165 bytes epoch 0) v1 -- 0x7f9684004eb0 con 0x7f96a006fc70 2024-06-03T09:06:31.437 INFO:teuthology.orchestra.run.smithi088.stderr:2024-06-03T09:06:31.434+0000 7f969d7fa640 1 -- 172.21.15.88:0/2599947638 <== mon.0 v1:172.21.15.88:6789/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 2234374065 0 0) 0x7f9694001890 con 0x7f96a006fc70 2024-06-03T09:06:31.437 INFO:teuthology.orchestra.run.smithi088.stderr:2024-06-03T09:06:31.434+0000 7f969d7fa640 10 monclient(hunting): handle_monmap mon_map magic: 0 v1 2024-06-03T09:06:31.437 INFO:teuthology.orchestra.run.smithi088.stderr:2024-06-03T09:06:31.434+0000 7f969d7fa640 10 monclient(hunting): got monmap 1 from mon.a (according to old e1) 2024-06-03T09:06:31.437 INFO:teuthology.orchestra.run.smithi088.stderr:2024-06-03T09:06:31.434+0000 7f969d7fa640 10 monclient(hunting): dump: 2024-06-03T09:06:31.437 INFO:teuthology.orchestra.run.smithi088.stderr:epoch 1 2024-06-03T09:06:31.437 INFO:teuthology.orchestra.run.smithi088.stderr:fsid ec3c3fdd-3ac6-4d7d-9412-683130ed8b04 2024-06-03T09:06:31.437 INFO:teuthology.orchestra.run.smithi088.stderr:last_changed 2024-06-03T08:34:31.885125+0000 2024-06-03T09:06:31.437 INFO:teuthology.orchestra.run.smithi088.stderr:created 2024-06-03T08:34:31.885125+0000 2024-06-03T09:06:31.437 INFO:teuthology.orchestra.run.smithi088.stderr:min_mon_release 18 (reef) 2024-06-03T09:06:31.437 INFO:teuthology.orchestra.run.smithi088.stderr:election_strategy: 3 2024-06-03T09:06:31.438 INFO:teuthology.orchestra.run.smithi088.stderr:0: v1:172.21.15.88:6789/0 mon.a 2024-06-03T09:06:31.438 INFO:teuthology.orchestra.run.smithi088.stderr:1: v1:172.21.15.192:6789/0 mon.b 2024-06-03T09:06:31.438 INFO:teuthology.orchestra.run.smithi088.stderr:2: v1:172.21.15.88:6790/0 mon.c 2024-06-03T09:06:31.438 INFO:teuthology.orchestra.run.smithi088.stderr: 2024-06-03T09:06:31.438 INFO:teuthology.orchestra.run.smithi088.stderr:2024-06-03T09:06:31.435+0000 7f969d7fa640 1 -- 172.21.15.88:0/2599947638 <== mon.2 v1:172.21.15.88:6790/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 764+0+0 (unknown 938834130 0 0) 0x7f96a01d4b80 con 0x7f96a01393f0 2024-06-03T09:06:31.438 INFO:teuthology.orchestra.run.smithi088.stderr:2024-06-03T09:06:31.435+0000 7f969d7fa640 1 -- 172.21.15.88:0/2599947638 --> v1:172.21.15.88:6790/0 -- auth(proto 2 165 bytes epoch 0) v1 -- 0x7f96a01d5a00 con 0x7f96a01393f0 2024-06-03T09:06:31.438 INFO:teuthology.orchestra.run.smithi088.stderr:2024-06-03T09:06:31.435+0000 7f969d7fa640 1 -- 172.21.15.88:0/2599947638 <== mon.2 v1:172.21.15.88:6790/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 2234374065 0 0) 0x7f9698001870 con 0x7f96a01393f0 2024-06-03T09:06:31.438 INFO:teuthology.orchestra.run.smithi088.stderr:2024-06-03T09:06:31.435+0000 7f969d7fa640 10 monclient(hunting): handle_monmap mon_map magic: 0 v1 2024-06-03T09:06:31.438 INFO:teuthology.orchestra.run.smithi088.stderr:2024-06-03T09:06:31.435+0000 7f969d7fa640 10 monclient(hunting): got monmap 1 from mon.c (according to old e1) 2024-06-03T09:06:31.438 INFO:teuthology.orchestra.run.smithi088.stderr:2024-06-03T09:06:31.435+0000 7f969d7fa640 10 monclient(hunting): dump: 2024-06-03T09:06:31.438 INFO:teuthology.orchestra.run.smithi088.stderr:epoch 1 2024-06-03T09:06:31.438 INFO:teuthology.orchestra.run.smithi088.stderr:fsid ec3c3fdd-3ac6-4d7d-9412-683130ed8b04 2024-06-03T09:06:31.438 INFO:teuthology.orchestra.run.smithi088.stderr:last_changed 2024-06-03T08:34:31.885125+0000 2024-06-03T09:06:31.439 INFO:teuthology.orchestra.run.smithi088.stderr:created 2024-06-03T08:34:31.885125+0000 2024-06-03T09:06:31.439 INFO:teuthology.orchestra.run.smithi088.stderr:min_mon_release 18 (reef) 2024-06-03T09:06:31.439 INFO:teuthology.orchestra.run.smithi088.stderr:election_strategy: 3 2024-06-03T09:06:31.439 INFO:teuthology.orchestra.run.smithi088.stderr:0: v1:172.21.15.88:6789/0 mon.a 2024-06-03T09:06:31.439 INFO:teuthology.orchestra.run.smithi088.stderr:1: v1:172.21.15.192:6789/0 mon.b 2024-06-03T09:06:31.439 INFO:teuthology.orchestra.run.smithi088.stderr:2: v1:172.21.15.88:6790/0 mon.c 2024-06-03T09:06:31.439 INFO:teuthology.orchestra.run.smithi088.stderr: 2024-06-03T09:06:31.440 INFO:teuthology.orchestra.run.smithi088.stderr:2024-06-03T09:06:31.437+0000 7f969d7fa640 1 -- 172.21.15.88:0/2599947638 <== mon.1 v1:172.21.15.192:6789/0 4 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 580+0+0 (unknown 1361672638 0 0) 0x7f96a00aaa10 con 0x7f96a00ad920 2024-06-03T09:06:31.440 INFO:teuthology.orchestra.run.smithi088.stderr:2024-06-03T09:06:31.437+0000 7f969d7fa640 1 -- 172.21.15.88:0/2599947638 >> v1:172.21.15.88:6790/0 conn(0x7f96a01393f0 legacy=0x7f96a01e03c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-06-03T09:06:31.440 INFO:teuthology.orchestra.run.smithi088.stderr:2024-06-03T09:06:31.437+0000 7f969d7fa640 1 -- 172.21.15.88:0/2599947638 >> v1:172.21.15.88:6789/0 conn(0x7f96a006fc70 legacy=0x7f96a01d52f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-06-03T09:06:31.440 INFO:teuthology.orchestra.run.smithi088.stderr:2024-06-03T09:06:31.437+0000 7f969d7fa640 10 monclient: _finish_hunting 0 2024-06-03T09:06:31.440 INFO:teuthology.orchestra.run.smithi088.stderr:2024-06-03T09:06:31.437+0000 7f969d7fa640 1 monclient: found mon.b 2024-06-03T09:06:31.441 INFO:teuthology.orchestra.run.smithi088.stderr:2024-06-03T09:06:31.437+0000 7f969d7fa640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-06-03T09:06:31.441 INFO:teuthology.orchestra.run.smithi088.stderr:2024-06-03T09:06:31.437+0000 7f969d7fa640 10 monclient: _send_mon_message to mon.b at v1:172.21.15.192:6789/0 2024-06-03T09:06:31.441 INFO:teuthology.orchestra.run.smithi088.stderr:2024-06-03T09:06:31.437+0000 7f969d7fa640 1 -- 172.21.15.88:0/2599947638 --> v1:172.21.15.192:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f96a0142c10 con 0x7f96a00ad920 2024-06-03T09:06:31.441 INFO:teuthology.orchestra.run.smithi088.stderr:2024-06-03T09:06:31.437+0000 7f969d7fa640 10 monclient: _finish_auth 0 2024-06-03T09:06:31.441 INFO:teuthology.orchestra.run.smithi088.stderr:2024-06-03T09:06:31.437+0000 7f969d7fa640 10 monclient: _check_auth_tickets 2024-06-03T09:06:31.441 INFO:teuthology.orchestra.run.smithi088.stderr:2024-06-03T09:06:31.437+0000 7f969d7fa640 20 monclient: _check_auth_rotating not needed by client.admin 2024-06-03T09:06:31.441 INFO:teuthology.orchestra.run.smithi088.stderr:2024-06-03T09:06:31.437+0000 7f96a7180640 5 monclient: authenticate success, global_id 30208 2024-06-03T09:06:31.441 INFO:teuthology.orchestra.run.smithi088.stderr:2024-06-03T09:06:31.437+0000 7f96a7180640 10 monclient: _renew_subs 2024-06-03T09:06:31.441 INFO:teuthology.orchestra.run.smithi088.stderr:2024-06-03T09:06:31.437+0000 7f96a7180640 10 monclient: _send_mon_message to mon.b at v1:172.21.15.192:6789/0 2024-06-03T09:06:31.441 INFO:teuthology.orchestra.run.smithi088.stderr:2024-06-03T09:06:31.437+0000 7f96a7180640 1 -- 172.21.15.88:0/2599947638 --> v1:172.21.15.192:6789/0 -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f96a01d4dd0 con 0x7f96a00ad920 2024-06-03T09:06:31.441 INFO:teuthology.orchestra.run.smithi088.stderr:2024-06-03T09:06:31.437+0000 7f96a7180640 10 client.30208.objecter _maybe_request_map subscribing (onetime) to next osd map 2024-06-03T09:06:31.441 INFO:teuthology.orchestra.run.smithi088.stderr:2024-06-03T09:06:31.437+0000 7f96a7180640 10 monclient: _renew_subs 2024-06-03T09:06:31.441 INFO:teuthology.orchestra.run.smithi088.stderr:2024-06-03T09:06:31.437+0000 7f96a7180640 10 monclient: _send_mon_message to mon.b at v1:172.21.15.192:6789/0 2024-06-03T09:06:31.442 INFO:teuthology.orchestra.run.smithi088.stderr:2024-06-03T09:06:31.437+0000 7f96a7180640 1 -- 172.21.15.88:0/2599947638 --> v1:172.21.15.192:6789/0 -- mon_subscribe({osdmap=0}) v3 -- 0x7f96a00a8d30 con 0x7f96a00ad920 2024-06-03T09:06:31.442 INFO:teuthology.orchestra.run.smithi088.stderr:2024-06-03T09:06:31.437+0000 7f96a7180640 1 librados: init done 2024-06-03T09:06:31.442 INFO:teuthology.orchestra.run.smithi088.stderr:2024-06-03T09:06:31.438+0000 7f96a7180640 10 start_mon_command cmd=[{"prefix": "get_command_descriptions"}] 2024-06-03T09:06:31.442 INFO:teuthology.orchestra.run.smithi088.stderr:2024-06-03T09:06:31.438+0000 7f96a7180640 10 monclient: _send_command 1 [{"prefix": "get_command_descriptions"}] 2024-06-03T09:06:31.442 INFO:teuthology.orchestra.run.smithi088.stderr:2024-06-03T09:06:31.438+0000 7f96a7180640 10 monclient: _send_mon_message to mon.b at v1:172.21.15.192:6789/0 2024-06-03T09:06:31.442 INFO:teuthology.orchestra.run.smithi088.stderr:2024-06-03T09:06:31.438+0000 7f96a7180640 1 -- 172.21.15.88:0/2599947638 --> v1:172.21.15.192:6789/0 -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f96a01d4fc0 con 0x7f96a00ad920 2024-06-03T09:06:31.651 INFO:teuthology.orchestra.run.smithi088.stderr:2024-06-03T09:06:31.651+0000 7f969d7fa640 1 -- 172.21.15.88:0/2599947638 <== mon.1 v1:172.21.15.192:6789/0 5 ==== config(0 keys) v1 ==== 4+0+0 (unknown 0 0 0) 0x7f968c0035f0 con 0x7f96a00ad920 2024-06-03T09:06:31.651 INFO:teuthology.orchestra.run.smithi088.stderr:2024-06-03T09:06:31.651+0000 7f969d7fa640 10 monclient: handle_config config(0 keys) v1 2024-06-03T09:06:31.651 INFO:teuthology.orchestra.run.smithi088.stderr:2024-06-03T09:06:31.651+0000 7f969effd640 4 set_mon_vals no callback set 2024-06-03T09:06:31.684 INFO:teuthology.orchestra.run.smithi088.stderr:2024-06-03T09:06:31.684+0000 7f969d7fa640 1 -- 172.21.15.88:0/2599947638 <== mon.1 v1:172.21.15.192:6789/0 6 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 2234374065 0 0) 0x7f968c003900 con 0x7f96a00ad920 2024-06-03T09:06:31.684 INFO:teuthology.orchestra.run.smithi088.stderr:2024-06-03T09:06:31.684+0000 7f969d7fa640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-06-03T09:06:31.684 INFO:teuthology.orchestra.run.smithi088.stderr:2024-06-03T09:06:31.684+0000 7f969d7fa640 10 monclient: got monmap 1 from mon.b (according to old e1) 2024-06-03T09:06:31.684 INFO:teuthology.orchestra.run.smithi088.stderr:2024-06-03T09:06:31.684+0000 7f969d7fa640 10 monclient: dump: 2024-06-03T09:06:31.685 INFO:teuthology.orchestra.run.smithi088.stderr:epoch 1 2024-06-03T09:06:31.685 INFO:teuthology.orchestra.run.smithi088.stderr:fsid ec3c3fdd-3ac6-4d7d-9412-683130ed8b04 2024-06-03T09:06:31.685 INFO:teuthology.orchestra.run.smithi088.stderr:last_changed 2024-06-03T08:34:31.885125+0000 2024-06-03T09:06:31.685 INFO:teuthology.orchestra.run.smithi088.stderr:created 2024-06-03T08:34:31.885125+0000 2024-06-03T09:06:31.685 INFO:teuthology.orchestra.run.smithi088.stderr:min_mon_release 18 (reef) 2024-06-03T09:06:31.685 INFO:teuthology.orchestra.run.smithi088.stderr:election_strategy: 3 2024-06-03T09:06:31.685 INFO:teuthology.orchestra.run.smithi088.stderr:0: v1:172.21.15.88:6789/0 mon.a 2024-06-03T09:06:31.685 INFO:teuthology.orchestra.run.smithi088.stderr:1: v1:172.21.15.192:6789/0 mon.b 2024-06-03T09:06:31.685 INFO:teuthology.orchestra.run.smithi088.stderr:2: v1:172.21.15.88:6790/0 mon.c 2024-06-03T09:06:31.685 INFO:teuthology.orchestra.run.smithi088.stderr: 2024-06-03T09:06:31.686 INFO:teuthology.orchestra.run.smithi088.stderr:2024-06-03T09:06:31.684+0000 7f969d7fa640 1 -- 172.21.15.88:0/2599947638 <== mon.1 v1:172.21.15.192:6789/0 7 ==== mgrmap(e 5) v1 ==== 89110+0+0 (unknown 873533973 0 0) 0x7f968c019680 con 0x7f96a00ad920 2024-06-03T09:06:31.686 INFO:teuthology.orchestra.run.smithi088.stderr:2024-06-03T09:06:31.685+0000 7f96a4f1c640 10 monclient: get_auth_request con 0x7f968406d400 auth_method 0 2024-06-03T09:06:31.686 INFO:teuthology.orchestra.run.smithi088.stderr:2024-06-03T09:06:31.685+0000 7f969d7fa640 10 client.30208.objecter ms_handle_connect 0x7f968406d400 2024-06-03T09:06:31.687 INFO:teuthology.orchestra.run.smithi088.stderr:2024-06-03T09:06:31.687+0000 7f969d7fa640 1 -- 172.21.15.88:0/2599947638 <== mon.1 v1:172.21.15.192:6789/0 8 ==== osd_map(1010..1010 src has 622..1010) v4 ==== 5229+0+0 (unknown 1105883993 0 0) 0x7f968c052ee0 con 0x7f96a00ad920 2024-06-03T09:06:31.687 INFO:teuthology.orchestra.run.smithi088.stderr:2024-06-03T09:06:31.687+0000 7f969d7fa640 10 client.30208.objecter ms_dispatch 0x7f96a0000b90 osd_map(1010..1010 src has 622..1010) v4 2024-06-03T09:06:31.687 INFO:teuthology.orchestra.run.smithi088.stderr:2024-06-03T09:06:31.687+0000 7f969d7fa640 3 client.30208.objecter handle_osd_map got epochs [1010,1010] > 0 2024-06-03T09:06:31.687 INFO:teuthology.orchestra.run.smithi088.stderr:2024-06-03T09:06:31.687+0000 7f969d7fa640 3 client.30208.objecter handle_osd_map decoding full epoch 1010 2024-06-03T09:06:31.687 INFO:teuthology.orchestra.run.smithi088.stderr:2024-06-03T09:06:31.687+0000 7f969d7fa640 20 client.30208.objecter dump_active .. 0 homeless 2024-06-03T09:06:31.689 INFO:teuthology.orchestra.run.smithi088.stderr:2024-06-03T09:06:31.689+0000 7f969d7fa640 1 -- 172.21.15.88:0/2599947638 <== mon.1 v1:172.21.15.192:6789/0 9 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+182498 (unknown 1092875540 0 3016760700) 0x7f968c04e8c0 con 0x7f96a00ad920 2024-06-03T09:06:31.689 INFO:teuthology.orchestra.run.smithi088.stderr:2024-06-03T09:06:31.689+0000 7f969d7fa640 10 monclient: handle_mon_command_ack 1 [{"prefix": "get_command_descriptions"}] 2024-06-03T09:06:31.689 INFO:teuthology.orchestra.run.smithi088.stderr:2024-06-03T09:06:31.689+0000 7f969d7fa640 10 monclient: _finish_command 1 = system:0 2024-06-03T09:06:31.911 INFO:teuthology.orchestra.run.smithi088.stderr:2024-06-03T09:06:31.910+0000 7f96a7180640 1 -- 172.21.15.88:0/2599947638 --> v1:172.21.15.192:6800/34229 -- mgr_command(tid 0: {"prefix": "pg dump", "target": ["mon-mgr", ""], "format": "json"}) v1 -- 0x7f96a013ce70 con 0x7f968406d400 2024-06-03T09:06:31.914 INFO:teuthology.orchestra.run.smithi088.stderr:2024-06-03T09:06:31.915+0000 7f969d7fa640 1 -- 172.21.15.88:0/2599947638 <== mgr.4110 v1:172.21.15.192:6800/34229 1 ==== mgr_command_reply(tid 0: 0 dumped all) v1 ==== 18+0+151509 (unknown 2965378022 0 1476458159) 0x7f9694007810 con 0x7f968406d400 2024-06-03T09:06:31.917 INFO:teuthology.orchestra.run.smithi088.stdout: 2024-06-03T09:06:31.917 INFO:teuthology.orchestra.run.smithi088.stderr:dumped all 2024-06-03T09:06:31.919 INFO:teuthology.orchestra.run.smithi088.stderr:2024-06-03T09:06:31.917+0000 7f96a7180640 10 librados: watch_flush enter 2024-06-03T09:06:31.919 INFO:teuthology.orchestra.run.smithi088.stderr:2024-06-03T09:06:31.917+0000 7f96a7180640 10 librados: watch_flush exit 2024-06-03T09:06:31.919 INFO:teuthology.orchestra.run.smithi088.stderr:2024-06-03T09:06:31.917+0000 7f96a7180640 20 client.30208.objecter shutdown clearing up homeless session... 2024-06-03T09:06:31.919 INFO:teuthology.orchestra.run.smithi088.stderr:2024-06-03T09:06:31.917+0000 7f96a7180640 10 client.30208.objecter successfully canceled tick 2024-06-03T09:06:31.919 INFO:teuthology.orchestra.run.smithi088.stderr:2024-06-03T09:06:31.917+0000 7f96a7180640 1 -- 172.21.15.88:0/2599947638 >> v1:172.21.15.192:6800/34229 conn(0x7f968406d400 legacy=0x7f968406f8c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-06-03T09:06:31.919 INFO:teuthology.orchestra.run.smithi088.stderr:2024-06-03T09:06:31.917+0000 7f96a7180640 10 monclient: shutdown 2024-06-03T09:06:31.920 INFO:teuthology.orchestra.run.smithi088.stderr:2024-06-03T09:06:31.917+0000 7f96a7180640 20 monclient: shutdown discarding 0 pending message(s) 2024-06-03T09:06:31.920 INFO:teuthology.orchestra.run.smithi088.stderr:2024-06-03T09:06:31.917+0000 7f96a7180640 1 -- 172.21.15.88:0/2599947638 >> v1:172.21.15.192:6789/0 conn(0x7f96a00ad920 legacy=0x7f96a0135390 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-06-03T09:06:31.920 INFO:teuthology.orchestra.run.smithi088.stderr:2024-06-03T09:06:31.917+0000 7f96a7180640 1 -- 172.21.15.88:0/2599947638 shutdown_connections 2024-06-03T09:06:31.920 INFO:teuthology.orchestra.run.smithi088.stderr:2024-06-03T09:06:31.917+0000 7f96a7180640 1 -- 172.21.15.88:0/2599947638 >> 172.21.15.88:0/2599947638 conn(0x7f96a00b2f20 msgr2=0x7f96a013d9f0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-06-03T09:06:31.920 INFO:teuthology.orchestra.run.smithi088.stderr:2024-06-03T09:06:31.917+0000 7f96a7180640 1 -- 172.21.15.88:0/2599947638 shutdown_connections 2024-06-03T09:06:31.920 INFO:teuthology.orchestra.run.smithi088.stderr:2024-06-03T09:06:31.917+0000 7f96a7180640 1 -- 172.21.15.88:0/2599947638 wait complete. 2024-06-03T09:06:31.920 INFO:teuthology.orchestra.run.smithi088.stderr:2024-06-03T09:06:31.918+0000 7f96a7180640 1 librados: shutdown 2024-06-03T09:06:31.929 INFO:teuthology.orchestra.run.smithi088.stdout:{"pg_ready":true,"pg_map":{"version":1921,"stamp":"2024-06-03T09:06:31.100641+0000","last_osdmap_epoch":0,"last_pg_scan":0,"pg_stats_sum":{"stat_sum":{"num_bytes":459285,"num_objects":3,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":1,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":3,"num_whiteouts":0,"num_read":102,"num_read_kb":88,"num_write":120,"num_write_kb":1379,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2,"num_bytes_recovered":397840,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":83,"ondisk_log_size":83,"up":68,"acting":68,"num_store_stats":0},"osd_stats_sum":{"up_from":0,"seq":0,"num_pgs":100,"num_osds":8,"num_per_pool_osds":8,"num_per_pool_omap_osds":8,"kb":754974720,"kb_used":494548,"kb_used_data":46324,"kb_used_omap":12,"kb_used_meta":448115,"kb_avail":754480172,"statfs":{"total":773094113280,"available":772587696128,"internally_reserved":0,"allocated":47435776,"data_stored":22747954,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":12744,"internal_metadata":458870328},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":109,"apply_latency_ms":109,"commit_latency_ns":109000000,"apply_latency_ns":109000000},"alerts":[],"network_ping_times":[]},"pg_stats_delta":{"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":0,"acting":0,"num_store_stats":0,"stamp_delta":"12.002291"},"pg_stats":[{"pgid":"17.d","version":"0'0","reported_seq":942,"reported_epoch":1009,"state":"active+clean","last_fresh":"2024-06-03T09:06:14.443917+0000","last_change":"2024-06-03T08:38:19.260548+0000","last_active":"2024-06-03T09:06:14.443917+0000","last_peered":"2024-06-03T09:06:14.443917+0000","last_clean":"2024-06-03T09:06:14.443917+0000","last_became_active":"2024-06-03T08:38:19.253559+0000","last_became_peered":"2024-06-03T08:38:19.253559+0000","last_unstale":"2024-06-03T09:06:14.443917+0000","last_undegraded":"2024-06-03T09:06:14.443917+0000","last_fullsized":"2024-06-03T09:06:14.443917+0000","mapping_epoch":28,"log_start":"0'0","ondisk_log_start":"0'0","created":28,"last_epoch_clean":45,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-06-03T08:37:55.614497+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-06-03T08:37:55.614497+0000","last_clean_scrub_stamp":"2024-06-03T08:37:55.614497+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-06-04T19:00:12.470979+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1],"acting":[0,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"17.c","version":"0'0","reported_seq":953,"reported_epoch":1009,"state":"active+undersized","last_fresh":"2024-06-03T09:06:13.668156+0000","last_change":"2024-06-03T09:06:03.440074+0000","last_active":"2024-06-03T09:06:13.668156+0000","last_peered":"2024-06-03T09:06:13.668156+0000","last_clean":"2024-06-03T09:05:02.105638+0000","last_became_active":"2024-06-03T09:06:03.440074+0000","last_became_peered":"2024-06-03T09:06:03.440074+0000","last_unstale":"2024-06-03T09:06:13.668156+0000","last_undegraded":"2024-06-03T09:06:13.668156+0000","last_fullsized":"2024-06-03T09:06:03.377982+0000","mapping_epoch":999,"log_start":"0'0","ondisk_log_start":"0'0","created":28,"last_epoch_clean":255,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-06-03T08:37:55.614497+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-06-03T08:37:55.614497+0000","last_clean_scrub_stamp":"2024-06-03T08:37:55.614497+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-06-04T10:34:20.808557+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3],"acting":[3],"avail_no_missing":["3"],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"17.f","version":"0'0","reported_seq":148,"reported_epoch":1009,"state":"active+undersized","last_fresh":"2024-06-03T09:06:14.450543+0000","last_change":"2024-06-03T09:06:03.420285+0000","last_active":"2024-06-03T09:06:14.450543+0000","last_peered":"2024-06-03T09:06:14.450543+0000","last_clean":"2024-06-03T08:37:55.614497+0000","last_became_active":"2024-06-03T09:06:03.420285+0000","last_became_peered":"2024-06-03T09:06:03.420285+0000","last_unstale":"2024-06-03T09:06:14.450543+0000","last_undegraded":"2024-06-03T09:06:14.450543+0000","last_fullsized":"2024-06-03T09:06:03.380616+0000","mapping_epoch":999,"log_start":"0'0","ondisk_log_start":"0'0","created":28,"last_epoch_clean":255,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-06-03T08:37:55.614497+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-06-03T08:37:55.614497+0000","last_clean_scrub_stamp":"2024-06-03T08:37:55.614497+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-06-04T15:39:41.187731+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4],"acting":[4],"avail_no_missing":["4"],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"17.e","version":"0'0","reported_seq":149,"reported_epoch":1009,"state":"active+undersized","last_fresh":"2024-06-03T09:06:13.726992+0000","last_change":"2024-06-03T09:06:03.412887+0000","last_active":"2024-06-03T09:06:13.726992+0000","last_peered":"2024-06-03T09:06:13.726992+0000","last_clean":"2024-06-03T08:37:55.614497+0000","last_became_active":"2024-06-03T09:06:03.412887+0000","last_became_peered":"2024-06-03T09:06:03.412887+0000","last_unstale":"2024-06-03T09:06:13.726992+0000","last_undegraded":"2024-06-03T09:06:13.726992+0000","last_fullsized":"2024-06-03T09:06:03.332482+0000","mapping_epoch":999,"log_start":"0'0","ondisk_log_start":"0'0","created":28,"last_epoch_clean":255,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-06-03T08:37:55.614497+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-06-03T08:37:55.614497+0000","last_clean_scrub_stamp":"2024-06-03T08:37:55.614497+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-06-04T12:23:22.511601+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5],"acting":[5],"avail_no_missing":["5"],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"17.9","version":"0'0","reported_seq":946,"reported_epoch":1009,"state":"active+clean","last_fresh":"2024-06-03T09:06:14.999299+0000","last_change":"2024-06-03T08:38:04.772260+0000","last_active":"2024-06-03T09:06:14.999299+0000","last_peered":"2024-06-03T09:06:14.999299+0000","last_clean":"2024-06-03T09:06:14.999299+0000","last_became_active":"2024-06-03T08:38:04.763428+0000","last_became_peered":"2024-06-03T08:38:04.763428+0000","last_unstale":"2024-06-03T09:06:14.999299+0000","last_undegraded":"2024-06-03T09:06:14.999299+0000","last_fullsized":"2024-06-03T09:06:14.999299+0000","mapping_epoch":28,"log_start":"0'0","ondisk_log_start":"0'0","created":28,"last_epoch_clean":33,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-06-03T08:37:55.614497+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-06-03T08:37:55.614497+0000","last_clean_scrub_stamp":"2024-06-03T08:37:55.614497+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-06-04T10:55:10.773936+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,4],"acting":[7,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"17.8","version":"0'0","reported_seq":961,"reported_epoch":1009,"state":"active+undersized","last_fresh":"2024-06-03T09:06:14.430375+0000","last_change":"2024-06-03T09:06:03.388786+0000","last_active":"2024-06-03T09:06:14.430375+0000","last_peered":"2024-06-03T09:06:14.430375+0000","last_clean":"2024-06-03T09:04:43.297780+0000","last_became_active":"2024-06-03T09:06:03.388786+0000","last_became_peered":"2024-06-03T09:06:03.388786+0000","last_unstale":"2024-06-03T09:06:14.430375+0000","last_undegraded":"2024-06-03T09:06:14.430375+0000","last_fullsized":"2024-06-03T09:06:03.331754+0000","mapping_epoch":999,"log_start":"0'0","ondisk_log_start":"0'0","created":28,"last_epoch_clean":255,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-06-03T08:37:55.614497+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-06-03T08:37:55.614497+0000","last_clean_scrub_stamp":"2024-06-03T08:37:55.614497+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-06-04T09:18:44.765547+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0],"acting":[0],"avail_no_missing":["0"],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"17.b","version":"0'0","reported_seq":149,"reported_epoch":1009,"state":"active+undersized","last_fresh":"2024-06-03T09:06:14.433735+0000","last_change":"2024-06-03T09:06:03.451203+0000","last_active":"2024-06-03T09:06:14.433735+0000","last_peered":"2024-06-03T09:06:14.433735+0000","last_clean":"2024-06-03T08:37:55.614497+0000","last_became_active":"2024-06-03T09:06:03.451203+0000","last_became_peered":"2024-06-03T09:06:03.451203+0000","last_unstale":"2024-06-03T09:06:14.433735+0000","last_undegraded":"2024-06-03T09:06:14.433735+0000","last_fullsized":"2024-06-03T09:06:03.375523+0000","mapping_epoch":999,"log_start":"0'0","ondisk_log_start":"0'0","created":28,"last_epoch_clean":255,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-06-03T08:37:55.614497+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-06-03T08:37:55.614497+0000","last_clean_scrub_stamp":"2024-06-03T08:37:55.614497+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-06-04T10:16:19.813495+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2],"acting":[2],"avail_no_missing":["2"],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"17.a","version":"0'0","reported_seq":925,"reported_epoch":1010,"state":"active+clean","last_fresh":"2024-06-03T09:06:18.651697+0000","last_change":"2024-06-03T08:38:05.843142+0000","last_active":"2024-06-03T09:06:18.651697+0000","last_peered":"2024-06-03T09:06:18.651697+0000","last_clean":"2024-06-03T09:06:18.651697+0000","last_became_active":"2024-06-03T08:38:05.809257+0000","last_became_peered":"2024-06-03T08:38:05.809257+0000","last_unstale":"2024-06-03T09:06:18.651697+0000","last_undegraded":"2024-06-03T09:06:18.651697+0000","last_fullsized":"2024-06-03T09:06:18.651697+0000","mapping_epoch":28,"log_start":"0'0","ondisk_log_start":"0'0","created":28,"last_epoch_clean":33,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-06-03T08:37:55.614497+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-06-03T08:37:55.614497+0000","last_clean_scrub_stamp":"2024-06-03T08:37:55.614497+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-06-04T12:12:03.686838+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2],"acting":[1,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"17.5","version":"0'0","reported_seq":943,"reported_epoch":1009,"state":"active+clean","last_fresh":"2024-06-03T09:06:14.442357+0000","last_change":"2024-06-03T08:38:23.466051+0000","last_active":"2024-06-03T09:06:14.442357+0000","last_peered":"2024-06-03T09:06:14.442357+0000","last_clean":"2024-06-03T09:06:14.442357+0000","last_became_active":"2024-06-03T08:38:23.458039+0000","last_became_peered":"2024-06-03T08:38:23.458039+0000","last_unstale":"2024-06-03T09:06:14.442357+0000","last_undegraded":"2024-06-03T09:06:14.442357+0000","last_fullsized":"2024-06-03T09:06:14.442357+0000","mapping_epoch":28,"log_start":"0'0","ondisk_log_start":"0'0","created":28,"last_epoch_clean":49,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-06-03T08:37:55.614497+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-06-03T08:37:55.614497+0000","last_clean_scrub_stamp":"2024-06-03T08:37:55.614497+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-06-04T12:27:02.532436+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,4],"acting":[0,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"17.4","version":"0'0","reported_seq":149,"reported_epoch":1009,"state":"active+undersized","last_fresh":"2024-06-03T09:06:13.941522+0000","last_change":"2024-06-03T09:06:03.441889+0000","last_active":"2024-06-03T09:06:13.941522+0000","last_peered":"2024-06-03T09:06:13.941522+0000","last_clean":"2024-06-03T08:37:55.614497+0000","last_became_active":"2024-06-03T09:06:03.441889+0000","last_became_peered":"2024-06-03T09:06:03.441889+0000","last_unstale":"2024-06-03T09:06:13.941522+0000","last_undegraded":"2024-06-03T09:06:13.941522+0000","last_fullsized":"2024-06-03T09:06:03.359582+0000","mapping_epoch":999,"log_start":"0'0","ondisk_log_start":"0'0","created":28,"last_epoch_clean":255,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-06-03T08:37:55.614497+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-06-03T08:37:55.614497+0000","last_clean_scrub_stamp":"2024-06-03T08:37:55.614497+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-06-04T12:49:22.907141+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7],"acting":[7],"avail_no_missing":["7"],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"17.7","version":"0'0","reported_seq":929,"reported_epoch":1009,"state":"active+clean","last_fresh":"2024-06-03T09:06:13.736920+0000","last_change":"2024-06-03T08:38:04.837083+0000","last_active":"2024-06-03T09:06:13.736920+0000","last_peered":"2024-06-03T09:06:13.736920+0000","last_clean":"2024-06-03T09:06:13.736920+0000","last_became_active":"2024-06-03T08:38:04.818246+0000","last_became_peered":"2024-06-03T08:38:04.818246+0000","last_unstale":"2024-06-03T09:06:13.736920+0000","last_undegraded":"2024-06-03T09:06:13.736920+0000","last_fullsized":"2024-06-03T09:06:13.736920+0000","mapping_epoch":28,"log_start":"0'0","ondisk_log_start":"0'0","created":28,"last_epoch_clean":34,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-06-03T08:37:55.614497+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-06-03T08:37:55.614497+0000","last_clean_scrub_stamp":"2024-06-03T08:37:55.614497+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-06-04T17:02:38.204813+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,2],"acting":[5,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"17.6","version":"0'0","reported_seq":960,"reported_epoch":1009,"state":"active+undersized","last_fresh":"2024-06-03T09:06:14.453110+0000","last_change":"2024-06-03T09:06:03.428677+0000","last_active":"2024-06-03T09:06:14.453110+0000","last_peered":"2024-06-03T09:06:14.453110+0000","last_clean":"2024-06-03T09:04:43.312352+0000","last_became_active":"2024-06-03T09:06:03.428677+0000","last_became_peered":"2024-06-03T09:06:03.428677+0000","last_unstale":"2024-06-03T09:06:14.453110+0000","last_undegraded":"2024-06-03T09:06:14.453110+0000","last_fullsized":"2024-06-03T09:06:03.359909+0000","mapping_epoch":999,"log_start":"0'0","ondisk_log_start":"0'0","created":28,"last_epoch_clean":255,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-06-03T08:37:55.614497+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-06-03T08:37:55.614497+0000","last_clean_scrub_stamp":"2024-06-03T08:37:55.614497+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-06-04T15:38:02.180241+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0],"acting":[0],"avail_no_missing":["0"],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"2.7","version":"0'0","reported_seq":940,"reported_epoch":1009,"state":"active+clean","last_fresh":"2024-06-03T09:06:13.684065+0000","last_change":"2024-06-03T08:36:30.385405+0000","last_active":"2024-06-03T09:06:13.684065+0000","last_peered":"2024-06-03T09:06:13.684065+0000","last_clean":"2024-06-03T09:06:13.684065+0000","last_became_active":"2024-06-03T08:36:30.380302+0000","last_became_peered":"2024-06-03T08:36:30.380302+0000","last_unstale":"2024-06-03T09:06:13.684065+0000","last_undegraded":"2024-06-03T09:06:13.684065+0000","last_fullsized":"2024-06-03T09:06:13.684065+0000","mapping_epoch":23,"log_start":"0'0","ondisk_log_start":"0'0","created":23,"last_epoch_clean":24,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-06-03T08:36:28.444764+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-06-03T08:36:28.444764+0000","last_clean_scrub_stamp":"2024-06-03T08:36:28.444764+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-06-04T18:46:33.475181+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0],"acting":[3,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"17.14","version":"0'0","reported_seq":926,"reported_epoch":1009,"state":"active+clean","last_fresh":"2024-06-03T09:06:13.721460+0000","last_change":"2024-06-03T08:38:04.547565+0000","last_active":"2024-06-03T09:06:13.721460+0000","last_peered":"2024-06-03T09:06:13.721460+0000","last_clean":"2024-06-03T09:06:13.721460+0000","last_became_active":"2024-06-03T08:38:04.541781+0000","last_became_peered":"2024-06-03T08:38:04.541781+0000","last_unstale":"2024-06-03T09:06:13.721460+0000","last_undegraded":"2024-06-03T09:06:13.721460+0000","last_fullsized":"2024-06-03T09:06:13.721460+0000","mapping_epoch":28,"log_start":"0'0","ondisk_log_start":"0'0","created":28,"last_epoch_clean":34,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-06-03T08:37:55.614497+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-06-03T08:37:55.614497+0000","last_clean_scrub_stamp":"2024-06-03T08:37:55.614497+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-06-04T16:41:55.170134+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,2],"acting":[5,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"2.6","version":"0'0","reported_seq":949,"reported_epoch":1009,"state":"active+clean","last_fresh":"2024-06-03T09:06:14.448100+0000","last_change":"2024-06-03T08:36:30.430760+0000","last_active":"2024-06-03T09:06:14.448100+0000","last_peered":"2024-06-03T09:06:14.448100+0000","last_clean":"2024-06-03T09:06:14.448100+0000","last_became_active":"2024-06-03T08:36:30.404665+0000","last_became_peered":"2024-06-03T08:36:30.404665+0000","last_unstale":"2024-06-03T09:06:14.448100+0000","last_undegraded":"2024-06-03T09:06:14.448100+0000","last_fullsized":"2024-06-03T09:06:14.448100+0000","mapping_epoch":23,"log_start":"0'0","ondisk_log_start":"0'0","created":23,"last_epoch_clean":24,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-06-03T08:36:28.444764+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-06-03T08:36:28.444764+0000","last_clean_scrub_stamp":"2024-06-03T08:36:28.444764+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-06-04T10:37:37.746074+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,0],"acting":[4,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"17.15","version":"0'0","reported_seq":934,"reported_epoch":1009,"state":"active+clean","last_fresh":"2024-06-03T09:06:13.672940+0000","last_change":"2024-06-03T08:38:20.999903+0000","last_active":"2024-06-03T09:06:13.672940+0000","last_peered":"2024-06-03T09:06:13.672940+0000","last_clean":"2024-06-03T09:06:13.672940+0000","last_became_active":"2024-06-03T08:38:20.979523+0000","last_became_peered":"2024-06-03T08:38:20.979523+0000","last_unstale":"2024-06-03T09:06:13.672940+0000","last_undegraded":"2024-06-03T09:06:13.672940+0000","last_fullsized":"2024-06-03T09:06:13.672940+0000","mapping_epoch":28,"log_start":"0'0","ondisk_log_start":"0'0","created":28,"last_epoch_clean":47,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-06-03T08:37:55.614497+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-06-03T08:37:55.614497+0000","last_clean_scrub_stamp":"2024-06-03T08:37:55.614497+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-06-04T20:29:58.314289+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,1],"acting":[3,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"2.5","version":"0'0","reported_seq":942,"reported_epoch":1010,"state":"active+clean","last_fresh":"2024-06-03T09:06:18.382767+0000","last_change":"2024-06-03T08:36:30.024235+0000","last_active":"2024-06-03T09:06:18.382767+0000","last_peered":"2024-06-03T09:06:18.382767+0000","last_clean":"2024-06-03T09:06:18.382767+0000","last_became_active":"2024-06-03T08:36:30.019040+0000","last_became_peered":"2024-06-03T08:36:30.019040+0000","last_unstale":"2024-06-03T09:06:18.382767+0000","last_undegraded":"2024-06-03T09:06:18.382767+0000","last_fullsized":"2024-06-03T09:06:18.382767+0000","mapping_epoch":23,"log_start":"0'0","ondisk_log_start":"0'0","created":23,"last_epoch_clean":24,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-06-03T08:36:28.444764+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-06-03T08:36:28.444764+0000","last_clean_scrub_stamp":"2024-06-03T08:36:28.444764+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-06-04T15:46:47.109502+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,4],"acting":[3,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"17.16","version":"0'0","reported_seq":972,"reported_epoch":1009,"state":"active+undersized","last_fresh":"2024-06-03T09:06:14.441665+0000","last_change":"2024-06-03T09:06:03.478615+0000","last_active":"2024-06-03T09:06:14.441665+0000","last_peered":"2024-06-03T09:06:14.441665+0000","last_clean":"2024-06-03T09:04:43.328415+0000","last_became_active":"2024-06-03T09:06:03.478615+0000","last_became_peered":"2024-06-03T09:06:03.478615+0000","last_unstale":"2024-06-03T09:06:14.441665+0000","last_undegraded":"2024-06-03T09:06:14.441665+0000","last_fullsized":"2024-06-03T09:06:03.402504+0000","mapping_epoch":999,"log_start":"0'0","ondisk_log_start":"0'0","created":28,"last_epoch_clean":255,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-06-03T08:37:55.614497+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-06-03T08:37:55.614497+0000","last_clean_scrub_stamp":"2024-06-03T08:37:55.614497+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-06-04T08:47:08.079233+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2],"acting":[2],"avail_no_missing":["2"],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"2.4","version":"0'0","reported_seq":952,"reported_epoch":1010,"state":"active+undersized","last_fresh":"2024-06-03T09:06:18.331485+0000","last_change":"2024-06-03T09:06:03.387171+0000","last_active":"2024-06-03T09:06:18.331485+0000","last_peered":"2024-06-03T09:06:18.331485+0000","last_clean":"2024-06-03T09:04:43.202645+0000","last_became_active":"2024-06-03T09:06:03.387171+0000","last_became_peered":"2024-06-03T09:06:03.387171+0000","last_unstale":"2024-06-03T09:06:18.331485+0000","last_undegraded":"2024-06-03T09:06:18.331485+0000","last_fullsized":"2024-06-03T09:06:03.283787+0000","mapping_epoch":999,"log_start":"0'0","ondisk_log_start":"0'0","created":23,"last_epoch_clean":255,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-06-03T08:36:28.444764+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-06-03T08:36:28.444764+0000","last_clean_scrub_stamp":"2024-06-03T08:36:28.444764+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-06-04T20:17:43.485023+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5],"acting":[5],"avail_no_missing":["5"],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"17.17","version":"0'0","reported_seq":954,"reported_epoch":1009,"state":"active+clean","last_fresh":"2024-06-03T09:06:14.447046+0000","last_change":"2024-06-03T08:38:38.483275+0000","last_active":"2024-06-03T09:06:14.447046+0000","last_peered":"2024-06-03T09:06:14.447046+0000","last_clean":"2024-06-03T09:06:14.447046+0000","last_became_active":"2024-06-03T08:38:38.467916+0000","last_became_peered":"2024-06-03T08:38:38.467916+0000","last_unstale":"2024-06-03T09:06:14.447046+0000","last_undegraded":"2024-06-03T09:06:14.447046+0000","last_fullsized":"2024-06-03T09:06:14.447046+0000","mapping_epoch":28,"log_start":"0'0","ondisk_log_start":"0'0","created":28,"last_epoch_clean":61,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-06-03T08:37:55.614497+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-06-03T08:37:55.614497+0000","last_clean_scrub_stamp":"2024-06-03T08:37:55.614497+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-06-04T10:02:23.036420+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,7],"acting":[2,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"2.2","version":"0'0","reported_seq":963,"reported_epoch":1009,"state":"active+undersized","last_fresh":"2024-06-03T09:06:13.657892+0000","last_change":"2024-06-03T09:06:03.410020+0000","last_active":"2024-06-03T09:06:13.657892+0000","last_peered":"2024-06-03T09:06:13.657892+0000","last_clean":"2024-06-03T09:05:37.145889+0000","last_became_active":"2024-06-03T09:06:03.410020+0000","last_became_peered":"2024-06-03T09:06:03.410020+0000","last_unstale":"2024-06-03T09:06:13.657892+0000","last_undegraded":"2024-06-03T09:06:13.657892+0000","last_fullsized":"2024-06-03T09:06:03.346864+0000","mapping_epoch":999,"log_start":"0'0","ondisk_log_start":"0'0","created":23,"last_epoch_clean":255,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-06-03T08:36:28.444764+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-06-03T08:36:28.444764+0000","last_clean_scrub_stamp":"2024-06-03T08:36:28.444764+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-06-04T09:57:12.423360+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3],"acting":[3],"avail_no_missing":["3"],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"17.11","version":"0'0","reported_seq":944,"reported_epoch":1009,"state":"active+clean","last_fresh":"2024-06-03T09:06:14.426670+0000","last_change":"2024-06-03T08:38:06.502302+0000","last_active":"2024-06-03T09:06:14.426670+0000","last_peered":"2024-06-03T09:06:14.426670+0000","last_clean":"2024-06-03T09:06:14.426670+0000","last_became_active":"2024-06-03T08:38:06.490204+0000","last_became_peered":"2024-06-03T08:38:06.490204+0000","last_unstale":"2024-06-03T09:06:14.426670+0000","last_undegraded":"2024-06-03T09:06:14.426670+0000","last_fullsized":"2024-06-03T09:06:14.426670+0000","mapping_epoch":28,"log_start":"0'0","ondisk_log_start":"0'0","created":28,"last_epoch_clean":32,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-06-03T08:37:55.614497+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-06-03T08:37:55.614497+0000","last_clean_scrub_stamp":"2024-06-03T08:37:55.614497+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-06-04T10:47:05.988958+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,2],"acting":[4,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"2.1","version":"0'0","reported_seq":952,"reported_epoch":1009,"state":"active+clean","last_fresh":"2024-06-03T09:06:13.929725+0000","last_change":"2024-06-03T08:36:30.883360+0000","last_active":"2024-06-03T09:06:13.929725+0000","last_peered":"2024-06-03T09:06:13.929725+0000","last_clean":"2024-06-03T09:06:13.929725+0000","last_became_active":"2024-06-03T08:36:30.855073+0000","last_became_peered":"2024-06-03T08:36:30.855073+0000","last_unstale":"2024-06-03T09:06:13.929725+0000","last_undegraded":"2024-06-03T09:06:13.929725+0000","last_fullsized":"2024-06-03T09:06:13.929725+0000","mapping_epoch":23,"log_start":"0'0","ondisk_log_start":"0'0","created":23,"last_epoch_clean":25,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-06-03T08:36:28.444764+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-06-03T08:36:28.444764+0000","last_clean_scrub_stamp":"2024-06-03T08:36:28.444764+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-06-04T18:12:22.363473+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,5],"acting":[7,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"17.12","version":"0'0","reported_seq":945,"reported_epoch":1009,"state":"active+clean","last_fresh":"2024-06-03T09:06:13.935141+0000","last_change":"2024-06-03T08:38:01.751562+0000","last_active":"2024-06-03T09:06:13.935141+0000","last_peered":"2024-06-03T09:06:13.935141+0000","last_clean":"2024-06-03T09:06:13.935141+0000","last_became_active":"2024-06-03T08:38:01.745027+0000","last_became_peered":"2024-06-03T08:38:01.745027+0000","last_unstale":"2024-06-03T09:06:13.935141+0000","last_undegraded":"2024-06-03T09:06:13.935141+0000","last_fullsized":"2024-06-03T09:06:13.935141+0000","mapping_epoch":28,"log_start":"0'0","ondisk_log_start":"0'0","created":28,"last_epoch_clean":32,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-06-03T08:37:55.614497+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-06-03T08:37:55.614497+0000","last_clean_scrub_stamp":"2024-06-03T08:37:55.614497+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-06-04T17:54:43.785326+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,3],"acting":[7,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"2.0","version":"0'0","reported_seq":943,"reported_epoch":1009,"state":"active+clean","last_fresh":"2024-06-03T09:06:13.648112+0000","last_change":"2024-06-03T08:36:29.984096+0000","last_active":"2024-06-03T09:06:13.648112+0000","last_peered":"2024-06-03T09:06:13.648112+0000","last_clean":"2024-06-03T09:06:13.648112+0000","last_became_active":"2024-06-03T08:36:29.957875+0000","last_became_peered":"2024-06-03T08:36:29.957875+0000","last_unstale":"2024-06-03T09:06:13.648112+0000","last_undegraded":"2024-06-03T09:06:13.648112+0000","last_fullsized":"2024-06-03T09:06:13.648112+0000","mapping_epoch":23,"log_start":"0'0","ondisk_log_start":"0'0","created":23,"last_epoch_clean":24,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-06-03T08:36:28.444764+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-06-03T08:36:28.444764+0000","last_clean_scrub_stamp":"2024-06-03T08:36:28.444764+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-06-04T08:41:33.679479+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,5],"acting":[3,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"17.13","version":"0'0","reported_seq":943,"reported_epoch":1009,"state":"active+clean","last_fresh":"2024-06-03T09:06:14.438399+0000","last_change":"2024-06-03T08:38:26.838316+0000","last_active":"2024-06-03T09:06:14.438399+0000","last_peered":"2024-06-03T09:06:14.438399+0000","last_clean":"2024-06-03T09:06:14.438399+0000","last_became_active":"2024-06-03T08:38:26.833214+0000","last_became_peered":"2024-06-03T08:38:26.833214+0000","last_unstale":"2024-06-03T09:06:14.438399+0000","last_undegraded":"2024-06-03T09:06:14.438399+0000","last_fullsized":"2024-06-03T09:06:14.438399+0000","mapping_epoch":28,"log_start":"0'0","ondisk_log_start":"0'0","created":28,"last_epoch_clean":52,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-06-03T08:37:55.614497+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-06-03T08:37:55.614497+0000","last_clean_scrub_stamp":"2024-06-03T08:37:55.614497+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-06-04T12:58:51.617998+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,0],"acting":[4,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"2.3","version":"0'0","reported_seq":943,"reported_epoch":1010,"state":"active+clean","last_fresh":"2024-06-03T09:06:18.362614+0000","last_change":"2024-06-03T08:36:30.351670+0000","last_active":"2024-06-03T09:06:18.362614+0000","last_peered":"2024-06-03T09:06:18.362614+0000","last_clean":"2024-06-03T09:06:18.362614+0000","last_became_active":"2024-06-03T08:36:30.343560+0000","last_became_peered":"2024-06-03T08:36:30.343560+0000","last_unstale":"2024-06-03T09:06:18.362614+0000","last_undegraded":"2024-06-03T09:06:18.362614+0000","last_fullsized":"2024-06-03T09:06:18.362614+0000","mapping_epoch":23,"log_start":"0'0","ondisk_log_start":"0'0","created":23,"last_epoch_clean":24,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-06-03T08:36:28.444764+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-06-03T08:36:28.444764+0000","last_clean_scrub_stamp":"2024-06-03T08:36:28.444764+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-06-04T12:02:24.765135+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,2],"acting":[3,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.0","version":"19'76","reported_seq":1051,"reported_epoch":1009,"state":"active+clean","last_fresh":"2024-06-03T09:06:13.703685+0000","last_change":"2024-06-03T08:36:26.702702+0000","last_active":"2024-06-03T09:06:13.703685+0000","last_peered":"2024-06-03T09:06:13.703685+0000","last_clean":"2024-06-03T09:06:13.703685+0000","last_became_active":"2024-06-03T08:36:26.128378+0000","last_became_peered":"2024-06-03T08:36:26.128378+0000","last_unstale":"2024-06-03T09:06:13.703685+0000","last_undegraded":"2024-06-03T09:06:13.703685+0000","last_fullsized":"2024-06-03T09:06:13.703685+0000","mapping_epoch":20,"log_start":"0'0","ondisk_log_start":"0'0","created":16,"last_epoch_clean":22,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-06-03T08:36:15.542635+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-06-03T08:36:15.542635+0000","last_clean_scrub_stamp":"2024-06-03T08:36:15.542635+0000","objects_scrubbed":0,"log_size":76,"log_dups_size":0,"ondisk_log_size":76,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-06-04T12:41:53.731360+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":459280,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":96,"num_read_kb":82,"num_write":113,"num_write_kb":1372,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2,"num_bytes_recovered":397840,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,5],"acting":[1,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"17.10","version":"0'0","reported_seq":944,"reported_epoch":1009,"state":"active+clean","last_fresh":"2024-06-03T09:06:14.422035+0000","last_change":"2024-06-03T08:38:15.399189+0000","last_active":"2024-06-03T09:06:14.422035+0000","last_peered":"2024-06-03T09:06:14.422035+0000","last_clean":"2024-06-03T09:06:14.422035+0000","last_became_active":"2024-06-03T08:38:15.337723+0000","last_became_peered":"2024-06-03T08:38:15.337723+0000","last_unstale":"2024-06-03T09:06:14.422035+0000","last_undegraded":"2024-06-03T09:06:14.422035+0000","last_fullsized":"2024-06-03T09:06:14.422035+0000","mapping_epoch":28,"log_start":"0'0","ondisk_log_start":"0'0","created":28,"last_epoch_clean":42,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-06-03T08:37:55.614497+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-06-03T08:37:55.614497+0000","last_clean_scrub_stamp":"2024-06-03T08:37:55.614497+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-06-04T09:29:46.427317+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,3],"acting":[4,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"17.1a","version":"0'0","reported_seq":960,"reported_epoch":1009,"state":"active+undersized","last_fresh":"2024-06-03T09:06:14.431224+0000","last_change":"2024-06-03T09:06:03.390734+0000","last_active":"2024-06-03T09:06:14.431224+0000","last_peered":"2024-06-03T09:06:14.431224+0000","last_clean":"2024-06-03T09:05:20.322313+0000","last_became_active":"2024-06-03T09:06:03.390734+0000","last_became_peered":"2024-06-03T09:06:03.390734+0000","last_unstale":"2024-06-03T09:06:14.431224+0000","last_undegraded":"2024-06-03T09:06:14.431224+0000","last_fullsized":"2024-06-03T09:06:03.343197+0000","mapping_epoch":999,"log_start":"0'0","ondisk_log_start":"0'0","created":28,"last_epoch_clean":254,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-06-03T08:37:55.614497+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-06-03T08:37:55.614497+0000","last_clean_scrub_stamp":"2024-06-03T08:37:55.614497+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-06-04T15:56:47.628875+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4],"acting":[4],"avail_no_missing":["4"],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"17.1b","version":"0'0","reported_seq":935,"reported_epoch":1009,"state":"active+clean","last_fresh":"2024-06-03T09:06:13.663279+0000","last_change":"2024-06-03T08:38:03.115748+0000","last_active":"2024-06-03T09:06:13.663279+0000","last_peered":"2024-06-03T09:06:13.663279+0000","last_clean":"2024-06-03T09:06:13.663279+0000","last_became_active":"2024-06-03T08:38:03.110533+0000","last_became_peered":"2024-06-03T08:38:03.110533+0000","last_unstale":"2024-06-03T09:06:13.663279+0000","last_undegraded":"2024-06-03T09:06:13.663279+0000","last_fullsized":"2024-06-03T09:06:13.663279+0000","mapping_epoch":28,"log_start":"0'0","ondisk_log_start":"0'0","created":28,"last_epoch_clean":32,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-06-03T08:37:55.614497+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-06-03T08:37:55.614497+0000","last_clean_scrub_stamp":"2024-06-03T08:37:55.614497+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-06-04T12:36:47.471609+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,7],"acting":[3,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"17.18","version":"0'0","reported_seq":944,"reported_epoch":1009,"state":"active+clean","last_fresh":"2024-06-03T09:06:13.924799+0000","last_change":"2024-06-03T08:38:04.632864+0000","last_active":"2024-06-03T09:06:13.924799+0000","last_peered":"2024-06-03T09:06:13.924799+0000","last_clean":"2024-06-03T09:06:13.924799+0000","last_became_active":"2024-06-03T08:38:04.627856+0000","last_became_peered":"2024-06-03T08:38:04.627856+0000","last_unstale":"2024-06-03T09:06:13.924799+0000","last_undegraded":"2024-06-03T09:06:13.924799+0000","last_fullsized":"2024-06-03T09:06:13.924799+0000","mapping_epoch":28,"log_start":"0'0","ondisk_log_start":"0'0","created":28,"last_epoch_clean":33,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-06-03T08:37:55.614497+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-06-03T08:37:55.614497+0000","last_clean_scrub_stamp":"2024-06-03T08:37:55.614497+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-06-04T19:55:51.181598+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,4],"acting":[7,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"17.19","version":"0'0","reported_seq":934,"reported_epoch":1009,"state":"active+clean","last_fresh":"2024-06-03T09:06:13.652943+0000","last_change":"2024-06-03T08:38:01.839491+0000","last_active":"2024-06-03T09:06:13.652943+0000","last_peered":"2024-06-03T09:06:13.652943+0000","last_clean":"2024-06-03T09:06:13.652943+0000","last_became_active":"2024-06-03T08:38:01.829539+0000","last_became_peered":"2024-06-03T08:38:01.829539+0000","last_unstale":"2024-06-03T09:06:13.652943+0000","last_undegraded":"2024-06-03T09:06:13.652943+0000","last_fullsized":"2024-06-03T09:06:13.652943+0000","mapping_epoch":28,"log_start":"0'0","ondisk_log_start":"0'0","created":28,"last_epoch_clean":30,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-06-03T08:37:55.614497+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-06-03T08:37:55.614497+0000","last_clean_scrub_stamp":"2024-06-03T08:37:55.614497+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-06-04T09:10:43.860229+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,2],"acting":[3,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"17.1e","version":"0'0","reported_seq":148,"reported_epoch":1009,"state":"active+undersized","last_fresh":"2024-06-03T09:06:14.445917+0000","last_change":"2024-06-03T09:06:03.406181+0000","last_active":"2024-06-03T09:06:14.445917+0000","last_peered":"2024-06-03T09:06:14.445917+0000","last_clean":"2024-06-03T08:37:55.614497+0000","last_became_active":"2024-06-03T09:06:03.406181+0000","last_became_peered":"2024-06-03T09:06:03.406181+0000","last_unstale":"2024-06-03T09:06:14.445917+0000","last_undegraded":"2024-06-03T09:06:14.445917+0000","last_fullsized":"2024-06-03T09:06:03.363739+0000","mapping_epoch":999,"log_start":"0'0","ondisk_log_start":"0'0","created":28,"last_epoch_clean":255,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-06-03T08:37:55.614497+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-06-03T08:37:55.614497+0000","last_clean_scrub_stamp":"2024-06-03T08:37:55.614497+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-06-04T11:45:24.486600+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4],"acting":[4],"avail_no_missing":["4"],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"17.1f","version":"0'0","reported_seq":941,"reported_epoch":1009,"state":"active+clean","last_fresh":"2024-06-03T09:06:14.450053+0000","last_change":"2024-06-03T08:38:04.383538+0000","last_active":"2024-06-03T09:06:14.450053+0000","last_peered":"2024-06-03T09:06:14.450053+0000","last_clean":"2024-06-03T09:06:14.450053+0000","last_became_active":"2024-06-03T08:38:04.367278+0000","last_became_peered":"2024-06-03T08:38:04.367278+0000","last_unstale":"2024-06-03T09:06:14.450053+0000","last_undegraded":"2024-06-03T09:06:14.450053+0000","last_fullsized":"2024-06-03T09:06:14.450053+0000","mapping_epoch":28,"log_start":"0'0","ondisk_log_start":"0'0","created":28,"last_epoch_clean":32,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-06-03T08:37:55.614497+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-06-03T08:37:55.614497+0000","last_clean_scrub_stamp":"2024-06-03T08:37:55.614497+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-06-04T10:39:26.552282+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,2],"acting":[0,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"17.1c","version":"0'0","reported_seq":944,"reported_epoch":1010,"state":"active+undersized","last_fresh":"2024-06-03T09:06:18.327492+0000","last_change":"2024-06-03T09:06:03.382210+0000","last_active":"2024-06-03T09:06:18.327492+0000","last_peered":"2024-06-03T09:06:18.327492+0000","last_clean":"2024-06-03T09:04:43.199660+0000","last_became_active":"2024-06-03T09:06:03.382210+0000","last_became_peered":"2024-06-03T09:06:03.382210+0000","last_unstale":"2024-06-03T09:06:18.327492+0000","last_undegraded":"2024-06-03T09:06:18.327492+0000","last_fullsized":"2024-06-03T09:06:03.277874+0000","mapping_epoch":999,"log_start":"0'0","ondisk_log_start":"0'0","created":28,"last_epoch_clean":255,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-06-03T08:37:55.614497+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-06-03T08:37:55.614497+0000","last_clean_scrub_stamp":"2024-06-03T08:37:55.614497+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-06-04T12:18:50.013884+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5],"acting":[5],"avail_no_missing":["5"],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"17.1d","version":"0'0","reported_seq":923,"reported_epoch":1009,"state":"active+clean","last_fresh":"2024-06-03T09:06:13.715906+0000","last_change":"2024-06-03T08:38:06.794630+0000","last_active":"2024-06-03T09:06:13.715906+0000","last_peered":"2024-06-03T09:06:13.715906+0000","last_clean":"2024-06-03T09:06:13.715906+0000","last_became_active":"2024-06-03T08:38:06.592317+0000","last_became_peered":"2024-06-03T08:38:06.592317+0000","last_unstale":"2024-06-03T09:06:13.715906+0000","last_undegraded":"2024-06-03T09:06:13.715906+0000","last_fullsized":"2024-06-03T09:06:13.715906+0000","mapping_epoch":28,"log_start":"0'0","ondisk_log_start":"0'0","created":28,"last_epoch_clean":33,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-06-03T08:37:55.614497+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-06-03T08:37:55.614497+0000","last_clean_scrub_stamp":"2024-06-03T08:37:55.614497+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-06-04T12:33:30.785369+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,7],"acting":[1,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"17.2","version":"0'0","reported_seq":925,"reported_epoch":1009,"state":"active+clean","last_fresh":"2024-06-03T09:06:13.708688+0000","last_change":"2024-06-03T08:38:14.981410+0000","last_active":"2024-06-03T09:06:13.708688+0000","last_peered":"2024-06-03T09:06:13.708688+0000","last_clean":"2024-06-03T09:06:13.708688+0000","last_became_active":"2024-06-03T08:38:14.973825+0000","last_became_peered":"2024-06-03T08:38:14.973825+0000","last_unstale":"2024-06-03T09:06:13.708688+0000","last_undegraded":"2024-06-03T09:06:13.708688+0000","last_fullsized":"2024-06-03T09:06:13.708688+0000","mapping_epoch":28,"log_start":"0'0","ondisk_log_start":"0'0","created":28,"last_epoch_clean":41,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-06-03T08:37:55.614497+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-06-03T08:37:55.614497+0000","last_clean_scrub_stamp":"2024-06-03T08:37:55.614497+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-06-04T17:01:05.518613+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,4],"acting":[1,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"17.3","version":"0'0","reported_seq":943,"reported_epoch":1009,"state":"active+clean","last_fresh":"2024-06-03T09:06:14.437636+0000","last_change":"2024-06-03T08:38:14.619745+0000","last_active":"2024-06-03T09:06:14.437636+0000","last_peered":"2024-06-03T09:06:14.437636+0000","last_clean":"2024-06-03T09:06:14.437636+0000","last_became_active":"2024-06-03T08:38:14.510665+0000","last_became_peered":"2024-06-03T08:38:14.510665+0000","last_unstale":"2024-06-03T09:06:14.437636+0000","last_undegraded":"2024-06-03T09:06:14.437636+0000","last_fullsized":"2024-06-03T09:06:14.437636+0000","mapping_epoch":28,"log_start":"0'0","ondisk_log_start":"0'0","created":28,"last_epoch_clean":42,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-06-03T08:37:55.614497+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-06-03T08:37:55.614497+0000","last_clean_scrub_stamp":"2024-06-03T08:37:55.614497+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-06-04T12:03:47.392064+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,7],"acting":[0,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"17.0","version":"0'0","reported_seq":944,"reported_epoch":1009,"state":"active+clean","last_fresh":"2024-06-03T09:06:14.427123+0000","last_change":"2024-06-03T08:38:04.237118+0000","last_active":"2024-06-03T09:06:14.427123+0000","last_peered":"2024-06-03T09:06:14.427123+0000","last_clean":"2024-06-03T09:06:14.427123+0000","last_became_active":"2024-06-03T08:38:04.231954+0000","last_became_peered":"2024-06-03T08:38:04.231954+0000","last_unstale":"2024-06-03T09:06:14.427123+0000","last_undegraded":"2024-06-03T09:06:14.427123+0000","last_fullsized":"2024-06-03T09:06:14.427123+0000","mapping_epoch":28,"log_start":"0'0","ondisk_log_start":"0'0","created":28,"last_epoch_clean":32,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-06-03T08:37:55.614497+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-06-03T08:37:55.614497+0000","last_clean_scrub_stamp":"2024-06-03T08:37:55.614497+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-06-04T09:56:10.741637+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,2],"acting":[0,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"17.1","version":"65'7","reported_seq":959,"reported_epoch":1009,"state":"active+undersized+degraded","last_fresh":"2024-06-03T09:06:13.709209+0000","last_change":"2024-06-03T09:06:03.370660+0000","last_active":"2024-06-03T09:06:13.709209+0000","last_peered":"2024-06-03T09:06:13.709209+0000","last_clean":"2024-06-03T09:05:03.733864+0000","last_became_active":"2024-06-03T09:06:03.370660+0000","last_became_peered":"2024-06-03T09:06:03.370660+0000","last_unstale":"2024-06-03T09:06:13.709209+0000","last_undegraded":"2024-06-03T09:06:03.266076+0000","last_fullsized":"2024-06-03T09:06:03.264525+0000","mapping_epoch":999,"log_start":"0'0","ondisk_log_start":"0'0","created":28,"last_epoch_clean":255,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-06-03T08:37:55.614497+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-06-03T08:37:55.614497+0000","last_clean_scrub_stamp":"2024-06-03T08:37:55.614497+0000","objects_scrubbed":0,"log_size":7,"log_dups_size":0,"ondisk_log_size":7,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-06-04T11:55:18.194283+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":5,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":1,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":6,"num_read_kb":6,"num_write":7,"num_write_kb":7,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5],"acting":[5],"avail_no_missing":["5"],"object_location_counts":[{"shards":"5","objects":1}],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]}],"pool_stats":[{"poolid":2,"num_pg":8,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":14,"acting":14,"num_store_stats":7},{"poolid":1,"num_pg":1,"stat_sum":{"num_bytes":459280,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":96,"num_read_kb":82,"num_write":113,"num_write_kb":1372,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2,"num_bytes_recovered":397840,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":802816,"data_stored":795680,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":76,"ondisk_log_size":76,"up":2,"acting":2,"num_store_stats":2},{"poolid":17,"num_pg":32,"stat_sum":{"num_bytes":5,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":1,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":6,"num_read_kb":6,"num_write":7,"num_write_kb":7,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":8192,"data_stored":10,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":7,"ondisk_log_size":7,"up":52,"acting":52,"num_store_stats":8}],"osd_stats":[{"osd":6,"up_from":253,"seq":1086626726200,"num_pgs":32,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":54772,"kb_used_data":5608,"kb_used_omap":1,"kb_used_meta":49150,"kb_avail":94317068,"statfs":{"total":96636764160,"available":96580677632,"internally_reserved":0,"allocated":5742592,"data_stored":2767180,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1593,"internal_metadata":50330055},"hb_peers":[0,1,2,3,4,5,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":109,"apply_latency_ms":109,"commit_latency_ns":109000000,"apply_latency_ns":109000000},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Mon Jun 3 09:04:02 2024","interfaces":[{"interface":"back","average":{"1min":1582.4780000000001,"5min":332.72699999999998,"15min":124.113},"min":{"1min":4.4829999999999997,"5min":4.1959999999999997,"15min":4.1959999999999997},"max":{"1min":22766.882000000001,"5min":22766.882000000001,"15min":22766.882000000001},"last":21.917999999999999},{"interface":"front","average":{"1min":1579.5409999999999,"5min":334.94499999999999,"15min":125.84999999999999},"min":{"1min":4.8129999999999997,"5min":4.7539999999999996,"15min":4.4530000000000003},"max":{"1min":22766.289000000001,"5min":22766.289000000001,"15min":22766.289000000001},"last":25.408999999999999}]},{"osd":1,"last update":"Mon Jun 3 09:03:40 2024","interfaces":[{"interface":"back","average":{"1min":36.420999999999999,"5min":77.054000000000002,"15min":132.84800000000001},"min":{"1min":6.0380000000000003,"5min":5.8799999999999999,"15min":5.484},"max":{"1min":589.64099999999996,"5min":4138.1679999999997,"15min":9955.1299999999992},"last":26.84},{"interface":"front","average":{"1min":40.240000000000002,"5min":79.674999999999997,"15min":133.38999999999999},"min":{"1min":8.4589999999999996,"5min":8.4090000000000007,"15min":6.1390000000000002},"max":{"1min":588.60799999999995,"5min":4137.6530000000002,"15min":9827.5509999999995},"last":26.263000000000002}]},{"osd":2,"last update":"Mon Jun 3 09:03:40 2024","interfaces":[{"interface":"back","average":{"1min":12.613,"5min":24.600999999999999,"15min":21.088000000000001},"min":{"1min":6.9740000000000002,"5min":6.9740000000000002,"15min":6.5330000000000004},"max":{"1min":26.076000000000001,"5min":376.24900000000002,"15min":376.24900000000002},"last":24.899000000000001},{"interface":"front","average":{"1min":12.493,"5min":22.207999999999998,"15min":21.219000000000001},"min":{"1min":6.5810000000000004,"5min":6.5110000000000001,"15min":6.0750000000000002},"max":{"1min":29.690999999999999,"5min":271.51100000000002,"15min":434.77300000000002},"last":40.372}]},{"osd":3,"last update":"Mon Jun 3 09:03:40 2024","interfaces":[{"interface":"back","average":{"1min":14.093999999999999,"5min":24.143999999999998,"15min":29.376999999999999},"min":{"1min":10.196999999999999,"5min":6.2919999999999998,"15min":6.1719999999999997},"max":{"1min":24.832000000000001,"5min":197.84899999999999,"15min":1235.5799999999999},"last":24.353999999999999},{"interface":"front","average":{"1min":9.3260000000000005,"5min":18.463999999999999,"15min":27.631},"min":{"1min":6.125,"5min":6.125,"15min":6.125},"max":{"1min":28.166,"5min":200.173,"15min":1157.6780000000001},"last":23.454000000000001}]},{"osd":4,"last update":"Mon Jun 3 09:04:29 2024","interfaces":[{"interface":"back","average":{"1min":10.279,"5min":1253.809,"15min":450.79199999999997},"min":{"1min":7.2300000000000004,"5min":6.093,"15min":6.093},"max":{"1min":15.000999999999999,"5min":28093.096000000001,"15min":28093.096000000001},"last":101.45399999999999},{"interface":"front","average":{"1min":15.693,"5min":1279.759,"15min":463.87099999999998},"min":{"1min":10.551,"5min":7.7480000000000002,"15min":7.7480000000000002},"max":{"1min":27.898,"5min":28323.837,"15min":28323.837},"last":102.262}]},{"osd":5,"last update":"Mon Jun 3 09:04:10 2024","interfaces":[{"interface":"back","average":{"1min":13.728999999999999,"5min":24.670000000000002,"15min":23.815999999999999},"min":{"1min":10.401,"5min":7.516,"15min":7.516},"max":{"1min":25.138000000000002,"5min":320.85000000000002,"15min":445.041},"last":103.52500000000001},{"interface":"front","average":{"1min":13.847,"5min":29.283000000000001,"15min":23.393999999999998},"min":{"1min":7.944,"5min":6.266,"15min":6.266},"max":{"1min":18.817,"5min":530.56700000000001,"15min":530.56700000000001},"last":29.263999999999999}]},{"osd":7,"last update":"Mon Jun 3 09:03:40 2024","interfaces":[{"interface":"back","average":{"1min":20.568999999999999,"5min":30.286999999999999,"15min":27.977},"min":{"1min":13.874000000000001,"5min":10.58,"15min":9.2859999999999996},"max":{"1min":86.959000000000003,"5min":222.07599999999999,"15min":255.08799999999999},"last":37.618000000000002},{"interface":"front","average":{"1min":19.102,"5min":29.550000000000001,"15min":27.763000000000002},"min":{"1min":10.052,"5min":8.6709999999999994,"15min":8.5429999999999993},"max":{"1min":67.968999999999994,"5min":293.62099999999998,"15min":438.14600000000002},"last":111.985}]}]},{"osd":7,"up_from":20,"seq":85899346287,"num_pgs":9,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":58964,"kb_used_data":5704,"kb_used_omap":1,"kb_used_meta":53246,"kb_avail":94312876,"statfs":{"total":96636764160,"available":96576385024,"internally_reserved":0,"allocated":5840896,"data_stored":2740727,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1589,"internal_metadata":54524363},"hb_peers":[0,1,2,3,4,5],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Mon Jun 3 09:06:09 2024","interfaces":[{"interface":"back","average":{"1min":14.073,"5min":894.22400000000005,"15min":308.93400000000003},"min":{"1min":5.5709999999999997,"5min":4.218,"15min":4.0599999999999996},"max":{"1min":90.757999999999996,"5min":21729.001,"15min":21729.001},"last":9.1039999999999992},{"interface":"front","average":{"1min":9.2550000000000008,"5min":906.23500000000001,"15min":314.06},"min":{"1min":5.5789999999999997,"5min":4.8810000000000002,"15min":4.8810000000000002},"max":{"1min":25.937000000000001,"5min":21899.732,"15min":21899.732},"last":5.7240000000000002}]},{"osd":1,"last update":"Mon Jun 3 09:06:09 2024","interfaces":[{"interface":"back","average":{"1min":15.182,"5min":69.183999999999997,"15min":219.173},"min":{"1min":7.3639999999999999,"5min":6.6200000000000001,"15min":6.1319999999999997},"max":{"1min":76.926000000000002,"5min":2795.1329999999998,"15min":18373.257000000001},"last":9.8119999999999994},{"interface":"front","average":{"1min":21.495999999999999,"5min":74.703999999999994,"15min":225.285},"min":{"1min":8.8399999999999999,"5min":8.8399999999999999,"15min":8.5129999999999999},"max":{"1min":94.977999999999994,"5min":2803.3670000000002,"15min":18374.601999999999},"last":12.17}]},{"osd":2,"last update":"Mon Jun 3 09:06:09 2024","interfaces":[{"interface":"back","average":{"1min":19.164000000000001,"5min":26.902000000000001,"15min":22.920999999999999},"min":{"1min":11.207000000000001,"5min":8.5030000000000001,"15min":8.5030000000000001},"max":{"1min":91.319999999999993,"5min":333.99200000000002,"15min":333.99200000000002},"last":14.196999999999999},{"interface":"front","average":{"1min":11.125999999999999,"5min":18.132000000000001,"15min":16.353000000000002},"min":{"1min":6.202,"5min":5.8369999999999997,"15min":5.5069999999999997},"max":{"1min":28.649000000000001,"5min":226.18100000000001,"15min":226.18100000000001},"last":11.329000000000001}]},{"osd":3,"last update":"Mon Jun 3 09:06:09 2024","interfaces":[{"interface":"back","average":{"1min":22.291,"5min":26.289999999999999,"15min":31.074999999999999},"min":{"1min":11.714,"5min":6.8959999999999999,"15min":6.3760000000000003},"max":{"1min":83.102000000000004,"5min":316.64400000000001,"15min":2039.9059999999999},"last":15.954000000000001},{"interface":"front","average":{"1min":13.324999999999999,"5min":17.963000000000001,"15min":25.969999999999999},"min":{"1min":6.1200000000000001,"5min":5.2089999999999996,"15min":5.2089999999999996},"max":{"1min":49.908999999999999,"5min":233.542,"15min":2037.5550000000001},"last":13.622999999999999}]},{"osd":4,"last update":"Mon Jun 3 09:06:29 2024","interfaces":[{"interface":"back","average":{"1min":17.675999999999998,"5min":1026.325,"15min":472.47199999999998},"min":{"1min":7.306,"5min":5.9089999999999998,"15min":5.9089999999999998},"max":{"1min":140.03299999999999,"5min":28969.369999999999,"15min":30073.917000000001},"last":8.7119999999999997},{"interface":"front","average":{"1min":17.890999999999998,"5min":1044.5139999999999,"15min":480.226},"min":{"1min":5.9279999999999999,"5min":5.9279999999999999,"15min":5.9279999999999999},"max":{"1min":141.179,"5min":29244.138999999999,"15min":30348.021000000001},"last":10.571}]},{"osd":5,"last update":"Mon Jun 3 09:06:02 2024","interfaces":[{"interface":"back","average":{"1min":24.372,"5min":26.856000000000002,"15min":23.571999999999999},"min":{"1min":12.278,"5min":9.2460000000000004,"15min":6.5650000000000004},"max":{"1min":183.64099999999999,"5min":317.572,"15min":317.572},"last":15.128},{"interface":"front","average":{"1min":21.259,"5min":23.364000000000001,"15min":21.193999999999999},"min":{"1min":7.8499999999999996,"5min":7.0999999999999996,"15min":7.008},"max":{"1min":184.53399999999999,"5min":306.60599999999999,"15min":306.60599999999999},"last":12.704000000000001}]},{"osd":6,"last update":"Mon Jun 3 09:04:31 2024","interfaces":[{"interface":"back","average":{"1min":44.061,"5min":26.821000000000002,"15min":91.176000000000002},"min":{"1min":9.0800000000000001,"5min":7.9390000000000001,"15min":7.9390000000000001},"max":{"1min":268.029,"5min":308.916,"15min":8734.4639999999999},"last":6.6769999999999996},{"interface":"front","average":{"1min":46.643000000000001,"5min":23.879999999999999,"15min":88.765000000000001},"min":{"1min":8.8119999999999994,"5min":6.9240000000000004,"15min":6.742},"max":{"1min":305.64600000000002,"5min":305.64600000000002,"15min":8735.5319999999992},"last":12.327}]}]},{"osd":5,"up_from":20,"seq":85899346261,"num_pgs":9,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":59360,"kb_used_data":6100,"kb_used_omap":1,"kb_used_meta":53246,"kb_avail":94312480,"statfs":{"total":96636764160,"available":96575979520,"internally_reserved":0,"allocated":6246400,"data_stored":3138572,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1594,"internal_metadata":54524358},"hb_peers":[0,1,2,3,4,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Mon Jun 3 09:06:06 2024","interfaces":[{"interface":"back","average":{"1min":7.468,"5min":1703.037,"15min":580.51900000000001},"min":{"1min":4.6509999999999998,"5min":4.6509999999999998,"15min":4.4020000000000001},"max":{"1min":14.711,"5min":26433.764999999999,"15min":26433.764999999999},"last":8.4039999999999999},{"interface":"front","average":{"1min":9.9429999999999996,"5min":1725.27,"15min":587.87800000000004},"min":{"1min":5.5999999999999996,"5min":5.1319999999999997,"15min":4.8170000000000002},"max":{"1min":13.590999999999999,"5min":26704.698,"15min":26704.698},"last":9.7420000000000009}]},{"osd":1,"last update":"Mon Jun 3 09:06:00 2024","interfaces":[{"interface":"back","average":{"1min":10.862,"5min":67.731999999999999,"15min":217.364},"min":{"1min":7.1769999999999996,"5min":6.0890000000000004,"15min":5.4930000000000003},"max":{"1min":16.998999999999999,"5min":3923.4090000000001,"15min":18811.696},"last":6.1040000000000001},{"interface":"front","average":{"1min":10.417,"5min":65.141999999999996,"15min":220.75700000000001},"min":{"1min":5.3209999999999997,"5min":5.3209999999999997,"15min":5.0209999999999999},"max":{"1min":15.821999999999999,"5min":4019.6129999999998,"15min":18968.365000000002},"last":9.2059999999999995}]},{"osd":2,"last update":"Mon Jun 3 09:06:00 2024","interfaces":[{"interface":"back","average":{"1min":11.366,"5min":24.18,"15min":21.137},"min":{"1min":5.6369999999999996,"5min":5.6369999999999996,"15min":5.0510000000000002},"max":{"1min":15.529999999999999,"5min":332.48200000000003,"15min":332.48200000000003},"last":10.680999999999999},{"interface":"front","average":{"1min":10.089,"5min":23.117000000000001,"15min":24.050000000000001},"min":{"1min":6.6769999999999996,"5min":6.6769999999999996,"15min":5.8739999999999997},"max":{"1min":15.821,"5min":353.03199999999998,"15min":353.03199999999998},"last":12.638999999999999}]},{"osd":3,"last update":"Mon Jun 3 09:06:00 2024","interfaces":[{"interface":"back","average":{"1min":13.472,"5min":24.283999999999999,"15min":31.666},"min":{"1min":10.111000000000001,"5min":7.4720000000000004,"15min":7.4210000000000003},"max":{"1min":16.271999999999998,"5min":353.57499999999999,"15min":2855.8519999999999},"last":11.196999999999999},{"interface":"front","average":{"1min":9.5899999999999999,"5min":15.875,"15min":25.834},"min":{"1min":6.6609999999999996,"5min":5.7729999999999997,"15min":5.1909999999999998},"max":{"1min":15.112,"5min":203.804,"15min":2869.7860000000001},"last":12.1}]},{"osd":4,"last update":"Mon Jun 3 09:05:27 2024","interfaces":[{"interface":"back","average":{"1min":19.155000000000001,"5min":1084.961,"15min":1346.9949999999999},"min":{"1min":7.6130000000000004,"5min":7.6130000000000004,"15min":6.1340000000000003},"max":{"1min":156.55199999999999,"5min":28619.825000000001,"15min":57999.453999999998},"last":6.8789999999999996},{"interface":"front","average":{"1min":28.068999999999999,"5min":1087.106,"15min":1330.289},"min":{"1min":8.3949999999999996,"5min":6.5030000000000001,"15min":6.5030000000000001},"max":{"1min":254.91300000000001,"5min":28621.294000000002,"15min":57434.631999999998},"last":14.146000000000001}]},{"osd":6,"last update":"Mon Jun 3 09:03:57 2024","interfaces":[{"interface":"back","average":{"1min":23.536000000000001,"5min":36.457000000000001,"15min":83.799999999999997},"min":{"1min":9.4800000000000004,"5min":9.4800000000000004,"15min":8.0630000000000006},"max":{"1min":89.442999999999998,"5min":508.77999999999997,"15min":9193.5169999999998},"last":157.79300000000001},{"interface":"front","average":{"1min":18.891999999999999,"5min":22.100999999999999,"15min":75.489999999999995},"min":{"1min":5.923,"5min":5.923,"15min":5.923},"max":{"1min":78.424999999999997,"5min":464.62,"15min":9191.4429999999993},"last":148.09}]},{"osd":7,"last update":"Mon Jun 3 09:06:00 2024","interfaces":[{"interface":"back","average":{"1min":13.257,"5min":20.917999999999999,"15min":21.088999999999999},"min":{"1min":10.475,"5min":6.7629999999999999,"15min":6.7629999999999999},"max":{"1min":17.245000000000001,"5min":211.22999999999999,"15min":211.22999999999999},"last":14.390000000000001},{"interface":"front","average":{"1min":12.872999999999999,"5min":20.587,"15min":24.021000000000001},"min":{"1min":8.5760000000000005,"5min":7.7709999999999999,"15min":6.8570000000000002},"max":{"1min":18.359999999999999,"5min":223.191,"15min":286.69499999999999},"last":7.3949999999999996}]}]},{"osd":4,"up_from":20,"seq":85899346269,"num_pgs":12,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":63060,"kb_used_data":5704,"kb_used_omap":1,"kb_used_meta":57342,"kb_avail":94308780,"statfs":{"total":96636764160,"available":96572190720,"internally_reserved":0,"allocated":5840896,"data_stored":2740727,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1594,"internal_metadata":58718662},"hb_peers":[0,1,2,3,5,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Mon Jun 3 09:05:34 2024","interfaces":[{"interface":"back","average":{"1min":12.416,"5min":1099.4649999999999,"15min":382.15199999999999},"min":{"1min":4.5279999999999996,"5min":4.5279999999999996,"15min":4.3220000000000001},"max":{"1min":123.878,"5min":21726.743999999999,"15min":21726.743999999999},"last":7.5579999999999998},{"interface":"front","average":{"1min":16.462,"5min":1108.894,"15min":387.70299999999997},"min":{"1min":6.4089999999999998,"5min":5.8380000000000001,"15min":5.4909999999999997},"max":{"1min":130.185,"5min":21727.993999999999,"15min":21727.993999999999},"last":11.199}]},{"osd":1,"last update":"Mon Jun 3 09:05:34 2024","interfaces":[{"interface":"back","average":{"1min":14.727,"5min":56.755000000000003,"15min":341.09399999999999},"min":{"1min":4.6529999999999996,"5min":4.1970000000000001,"15min":4.1970000000000001},"max":{"1min":126.899,"5min":3346.4389999999999,"15min":22256.098000000002},"last":6.7679999999999998},{"interface":"front","average":{"1min":13.472,"5min":57.503,"15min":340.303},"min":{"1min":5.0590000000000002,"5min":5.0590000000000002,"15min":4.2640000000000002},"max":{"1min":126.096,"5min":3347.181,"15min":22247.317999999999},"last":12.481}]},{"osd":2,"last update":"Mon Jun 3 09:05:34 2024","interfaces":[{"interface":"back","average":{"1min":26.088999999999999,"5min":23.866,"15min":20.620999999999999},"min":{"1min":7.0869999999999997,"5min":6.8239999999999998,"15min":6.3150000000000004},"max":{"1min":337.60399999999998,"5min":393.58600000000001,"15min":393.58600000000001},"last":10.336},{"interface":"front","average":{"1min":15.800000000000001,"5min":22.584,"15min":23.245999999999999},"min":{"1min":6.0629999999999997,"5min":6.0629999999999997,"15min":6.0629999999999997},"max":{"1min":127.913,"5min":394.471,"15min":394.471},"last":14.75}]},{"osd":3,"last update":"Mon Jun 3 09:05:34 2024","interfaces":[{"interface":"back","average":{"1min":17.544,"5min":19.143999999999998,"15min":18.742000000000001},"min":{"1min":5.8719999999999999,"5min":5.4100000000000001,"15min":5.4000000000000004},"max":{"1min":127.43600000000001,"5min":322.63,"15min":322.63},"last":15.779},{"interface":"front","average":{"1min":21.327000000000002,"5min":23.559000000000001,"15min":23.797000000000001},"min":{"1min":10.023,"5min":7.484,"15min":7.484},"max":{"1min":204.517,"5min":392.67500000000001,"15min":392.67500000000001},"last":11.742000000000001}]},{"osd":5,"last update":"Mon Jun 3 09:05:34 2024","interfaces":[{"interface":"back","average":{"1min":24.274999999999999,"5min":21.274999999999999,"15min":20.988},"min":{"1min":6.7489999999999997,"5min":6.7489999999999997,"15min":6.7489999999999997},"max":{"1min":268.10199999999998,"5min":344.23399999999998,"15min":344.23399999999998},"last":13.760999999999999},{"interface":"front","average":{"1min":26.361999999999998,"5min":25.218,"15min":26.084},"min":{"1min":12.228,"5min":8.5180000000000007,"15min":8.5180000000000007},"max":{"1min":264.75,"5min":345.18700000000001,"15min":345.18700000000001},"last":17.739000000000001}]},{"osd":6,"last update":"Mon Jun 3 09:04:32 2024","interfaces":[{"interface":"back","average":{"1min":15.548,"5min":32.957999999999998,"15min":152.738},"min":{"1min":9.141,"5min":9.141,"15min":8.2319999999999993},"max":{"1min":52.57,"5min":395.238,"15min":12496.335999999999},"last":208.90000000000001},{"interface":"front","average":{"1min":12.680999999999999,"5min":25.428000000000001,"15min":149.30600000000001},"min":{"1min":6.4379999999999997,"5min":5.6319999999999997,"15min":5.6319999999999997},"max":{"1min":46.283000000000001,"5min":377.74299999999999,"15min":12616.297},"last":205.88900000000001}]},{"osd":7,"last update":"Mon Jun 3 09:05:34 2024","interfaces":[{"interface":"back","average":{"1min":18.295999999999999,"5min":19.585999999999999,"15min":21.062000000000001},"min":{"1min":6.2110000000000003,"5min":6.2110000000000003,"15min":5.8010000000000002},"max":{"1min":207.25299999999999,"5min":358.94900000000001,"15min":358.94900000000001},"last":9.2669999999999995},{"interface":"front","average":{"1min":23.712,"5min":24.024999999999999,"15min":24.640000000000001},"min":{"1min":11.824999999999999,"5min":7.7789999999999999,"15min":7.7789999999999999},"max":{"1min":209.32900000000001,"5min":346.25200000000001,"15min":380.06900000000002},"last":16.763999999999999}]}]},{"osd":2,"up_from":20,"seq":85899346289,"num_pgs":11,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":58964,"kb_used_data":5704,"kb_used_omap":1,"kb_used_meta":53246,"kb_avail":94312876,"statfs":{"total":96636764160,"available":96576385024,"internally_reserved":0,"allocated":5840896,"data_stored":2740727,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1594,"internal_metadata":54524358},"hb_peers":[0,1,3,4,5,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Mon Jun 3 09:06:23 2024","interfaces":[{"interface":"back","average":{"1min":8.9629999999999992,"5min":946.25400000000002,"15min":326.36099999999999},"min":{"1min":5.3479999999999999,"5min":4.7050000000000001,"15min":4.1420000000000003},"max":{"1min":26.015000000000001,"5min":23465.18,"15min":23465.18},"last":6.6070000000000002},{"interface":"front","average":{"1min":7.6130000000000004,"5min":944.39800000000002,"15min":326.178},"min":{"1min":4.6790000000000003,"5min":4.6790000000000003,"15min":4.6790000000000003},"max":{"1min":20.802,"5min":23465.966,"15min":23465.966},"last":5.1609999999999996}]},{"osd":1,"last update":"Mon Jun 3 09:06:23 2024","interfaces":[{"interface":"back","average":{"1min":13.007,"5min":76.507000000000005,"15min":294.72899999999998},"min":{"1min":6.4089999999999998,"5min":5.4199999999999999,"15min":5.4199999999999999},"max":{"1min":85.430000000000007,"5min":4220.5290000000005,"15min":18954.987000000001},"last":7.2859999999999996},{"interface":"front","average":{"1min":15.894,"5min":82.119,"15min":302.92399999999998},"min":{"1min":7.367,"5min":7.367,"15min":7.367},"max":{"1min":98.498000000000005,"5min":4325.9489999999996,"15min":19143.834999999999},"last":9.2759999999999998}]},{"osd":3,"last update":"Mon Jun 3 09:06:23 2024","interfaces":[{"interface":"back","average":{"1min":9.7579999999999991,"5min":12.444000000000001,"15min":28.065000000000001},"min":{"1min":7,"5min":5.9290000000000003,"15min":5.2450000000000001},"max":{"1min":28.164000000000001,"5min":118.913,"15min":2594.8150000000001},"last":10.537000000000001},{"interface":"front","average":{"1min":12.321999999999999,"5min":15.25,"15min":31.349},"min":{"1min":9.1240000000000006,"5min":8.0760000000000005,"15min":6.0780000000000003},"max":{"1min":32.360999999999997,"5min":156.34700000000001,"15min":2706.5120000000002},"last":12.038}]},{"osd":4,"last update":"Mon Jun 3 09:05:57 2024","interfaces":[{"interface":"back","average":{"1min":10.573,"5min":1266.894,"15min":438.154},"min":{"1min":8.6379999999999999,"5min":7.8789999999999996,"15min":6.2039999999999997},"max":{"1min":15.105,"5min":26660.861000000001,"15min":26660.861000000001},"last":11.036},{"interface":"front","average":{"1min":12.407999999999999,"5min":1266.7460000000001,"15min":435.75400000000002},"min":{"1min":10.266999999999999,"5min":10.266999999999999,"15min":6.2720000000000002},"max":{"1min":15.228,"5min":26616.243999999999,"15min":26616.243999999999},"last":12.824}]},{"osd":5,"last update":"Mon Jun 3 09:06:23 2024","interfaces":[{"interface":"back","average":{"1min":14.103,"5min":18.091000000000001,"15min":21.119},"min":{"1min":9.6349999999999998,"5min":9.2159999999999993,"15min":6.2050000000000001},"max":{"1min":32.890000000000001,"5min":252.31100000000001,"15min":252.31100000000001},"last":13.518000000000001},{"interface":"front","average":{"1min":11.545999999999999,"5min":16.170999999999999,"15min":20.254000000000001},"min":{"1min":7.6070000000000002,"5min":7.4850000000000003,"15min":6.8810000000000002},"max":{"1min":29.228999999999999,"5min":258.31700000000001,"15min":258.31700000000001},"last":8.6509999999999998}]},{"osd":6,"last update":"Mon Jun 3 09:04:28 2024","interfaces":[{"interface":"back","average":{"1min":14.063000000000001,"5min":24.850000000000001,"15min":106.755},"min":{"1min":9.0820000000000007,"5min":8.9649999999999999,"15min":8.9649999999999999},"max":{"1min":24.666,"5min":188.06700000000001,"15min":9521.0010000000002},"last":183.60900000000001},{"interface":"front","average":{"1min":13.933999999999999,"5min":21.081,"15min":106.142},"min":{"1min":5.8719999999999999,"5min":5.8719999999999999,"15min":5.8719999999999999},"max":{"1min":117.551,"5min":193.887,"15min":9585.6620000000003},"last":184.20599999999999}]},{"osd":7,"last update":"Mon Jun 3 09:06:23 2024","interfaces":[{"interface":"back","average":{"1min":17.056000000000001,"5min":20.055,"15min":23.268999999999998},"min":{"1min":10.353999999999999,"5min":6.6130000000000004,"15min":6.6130000000000004},"max":{"1min":88.974999999999994,"5min":242.56800000000001,"15min":242.56800000000001},"last":14.315},{"interface":"front","average":{"1min":18.885000000000002,"5min":21.460000000000001,"15min":25.489999999999998},"min":{"1min":12.497,"5min":9.0660000000000007,"15min":9.0660000000000007},"max":{"1min":102.229,"5min":181.56399999999999,"15min":214.33799999999999},"last":15.112}]}]},{"osd":3,"up_from":14,"seq":60129542503,"num_pgs":11,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":58964,"kb_used_data":5704,"kb_used_omap":1,"kb_used_meta":53246,"kb_avail":94312876,"statfs":{"total":96636764160,"available":96576385024,"internally_reserved":0,"allocated":5840896,"data_stored":2740727,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1592,"internal_metadata":54524360},"hb_peers":[0,1,2,4,5,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Mon Jun 3 09:06:25 2024","interfaces":[{"interface":"back","average":{"1min":7.79,"5min":1060.1959999999999,"15min":367.01799999999997},"min":{"1min":5.6909999999999998,"5min":5.173,"15min":4.6929999999999996},"max":{"1min":10.59,"5min":23275.167000000001,"15min":23275.167000000001},"last":8.5299999999999994},{"interface":"front","average":{"1min":11.215999999999999,"5min":1066.5799999999999,"15min":372.476},"min":{"1min":8.7080000000000002,"5min":7.2510000000000003,"15min":5.9880000000000004},"max":{"1min":13.499000000000001,"5min":23278.536,"15min":23278.536},"last":14.012}]},{"osd":1,"last update":"Mon Jun 3 09:06:25 2024","interfaces":[{"interface":"back","average":{"1min":7.3760000000000003,"5min":15.614000000000001,"15min":389.83100000000002},"min":{"1min":4.335,"5min":4.335,"15min":4.335},"max":{"1min":11.289,"5min":250.72300000000001,"15min":21658.205999999998},"last":10.667},{"interface":"front","average":{"1min":8.9499999999999993,"5min":14.135,"15min":381.10300000000001},"min":{"1min":6.5609999999999999,"5min":5.4100000000000001,"15min":5.2779999999999996},"max":{"1min":14.51,"5min":180.44499999999999,"15min":21307.069},"last":6.109}]},{"osd":2,"last update":"Mon Jun 3 09:06:25 2024","interfaces":[{"interface":"back","average":{"1min":12.343999999999999,"5min":22.059999999999999,"15min":23.715},"min":{"1min":7.9199999999999999,"5min":6.4900000000000002,"15min":6.056},"max":{"1min":15.601000000000001,"5min":377.803,"15min":377.803},"last":9.2889999999999997},{"interface":"front","average":{"1min":10.319000000000001,"5min":19.364999999999998,"15min":20.242999999999999},"min":{"1min":6.891,"5min":6.1749999999999998,"15min":5.9020000000000001},"max":{"1min":15.432,"5min":371.601,"15min":371.601},"last":11.827999999999999}]},{"osd":4,"last update":"Mon Jun 3 09:05:27 2024","interfaces":[{"interface":"back","average":{"1min":10.444000000000001,"5min":1211.4970000000001,"15min":1639.1579999999999},"min":{"1min":6.4429999999999996,"5min":6.2560000000000002,"15min":6.0309999999999997},"max":{"1min":21.263000000000002,"5min":30144.939999999999,"15min":62679.622000000003},"last":7.3719999999999999},{"interface":"front","average":{"1min":11.363,"5min":1219.6400000000001,"15min":1644.127},"min":{"1min":7.8109999999999999,"5min":7.8109999999999999,"15min":6.2439999999999998},"max":{"1min":20.108000000000001,"5min":30180.689999999999,"15min":62681.021999999997},"last":9.9800000000000004}]},{"osd":5,"last update":"Mon Jun 3 09:06:07 2024","interfaces":[{"interface":"back","average":{"1min":12.802,"5min":19.640999999999998,"15min":22.318000000000001},"min":{"1min":9.0890000000000004,"5min":6.8810000000000002,"15min":6.7690000000000001},"max":{"1min":16.745999999999999,"5min":249.88,"15min":327.34899999999999},"last":14.826000000000001},{"interface":"front","average":{"1min":15.002000000000001,"5min":24.003,"15min":26.649999999999999},"min":{"1min":11.984,"5min":9.9149999999999991,"15min":8.9879999999999995},"max":{"1min":18.407,"5min":251.47800000000001,"15min":331.86799999999999},"last":15.525}]},{"osd":6,"last update":"Mon Jun 3 09:04:04 2024","interfaces":[{"interface":"back","average":{"1min":22.815999999999999,"5min":31.513999999999999,"15min":181.74299999999999},"min":{"1min":9.4809999999999999,"5min":7.2240000000000002,"15min":7.2240000000000002},"max":{"1min":160.56999999999999,"5min":380.613,"15min":11893.74},"last":19.359999999999999},{"interface":"front","average":{"1min":25.359999999999999,"5min":30.981000000000002,"15min":179.81399999999999},"min":{"1min":9.8870000000000005,"5min":7.3360000000000003,"15min":6.0899999999999999},"max":{"1min":193.11099999999999,"5min":379.80700000000002,"15min":11893.069},"last":8.202}]},{"osd":7,"last update":"Mon Jun 3 09:06:25 2024","interfaces":[{"interface":"back","average":{"1min":11.321999999999999,"5min":18.443999999999999,"15min":21.568999999999999},"min":{"1min":8.109,"5min":8.109,"15min":7.4279999999999999},"max":{"1min":17.113,"5min":146.267,"15min":178.76300000000001},"last":11.212999999999999},{"interface":"front","average":{"1min":12.512,"5min":20.065999999999999,"15min":24.190000000000001},"min":{"1min":7.4989999999999997,"5min":7.4989999999999997,"15min":7.4989999999999997},"max":{"1min":18.047999999999998,"5min":148.13800000000001,"15min":417.22800000000001},"last":12.34}]}]},{"osd":1,"up_from":14,"seq":60129542501,"num_pgs":6,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":59356,"kb_used_data":6096,"kb_used_omap":1,"kb_used_meta":53246,"kb_avail":94312484,"statfs":{"total":96636764160,"available":96575983616,"internally_reserved":0,"allocated":6242304,"data_stored":3138567,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":54524357},"hb_peers":[0,2,3,4,5,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Mon Jun 3 09:06:07 2024","interfaces":[{"interface":"back","average":{"1min":7.7530000000000001,"5min":185.66999999999999,"15min":77.347999999999999},"min":{"1min":5.0229999999999997,"5min":4.9279999999999999,"15min":4.5449999999999999},"max":{"1min":11.896000000000001,"5min":7470.616,"15min":7470.616},"last":7.7709999999999999},{"interface":"front","average":{"1min":9.9220000000000006,"5min":187.14599999999999,"15min":76.325999999999993},"min":{"1min":5.3159999999999998,"5min":5.3159999999999998,"15min":5.3159999999999998},"max":{"1min":18.338000000000001,"5min":7477.0829999999996,"15min":7477.0829999999996},"last":10.446999999999999}]},{"osd":2,"last update":"Mon Jun 3 09:06:20 2024","interfaces":[{"interface":"back","average":{"1min":10.340999999999999,"5min":17.369,"15min":23.670999999999999},"min":{"1min":6.9740000000000002,"5min":5.8650000000000002,"15min":5.8650000000000002},"max":{"1min":15.452,"5min":228,"15min":657.38},"last":11.384},{"interface":"front","average":{"1min":13.387,"5min":23.355,"15min":31.524000000000001},"min":{"1min":9.077,"5min":9.077,"15min":7.2160000000000002},"max":{"1min":19.701000000000001,"5min":229.465,"15min":700.06500000000005},"last":13.276}]},{"osd":3,"last update":"Mon Jun 3 09:06:04 2024","interfaces":[{"interface":"back","average":{"1min":8.6110000000000007,"5min":15.733000000000001,"15min":26.646999999999998},"min":{"1min":5.827,"5min":5.4640000000000004,"15min":5.4640000000000004},"max":{"1min":13.105,"5min":121.619,"15min":1676.009},"last":7.0140000000000002},{"interface":"front","average":{"1min":9.5850000000000009,"5min":16.161999999999999,"15min":29.065999999999999},"min":{"1min":6.3630000000000004,"5min":5.7699999999999996,"15min":5.7699999999999996},"max":{"1min":15.166,"5min":132.036,"15min":1766.5840000000001},"last":8.3870000000000005}]},{"osd":4,"last update":"Mon Jun 3 09:05:31 2024","interfaces":[{"interface":"back","average":{"1min":10.996,"5min":954.46500000000003,"15min":1281.6859999999999},"min":{"1min":7.702,"5min":6.1760000000000002,"15min":6.0970000000000004},"max":{"1min":21.538,"5min":28044.862000000001,"15min":59438.017999999996},"last":9.5630000000000006},{"interface":"front","average":{"1min":13.646000000000001,"5min":961.24599999999998,"15min":1285.26},"min":{"1min":9.7200000000000006,"5min":9.3940000000000001,"15min":6.4640000000000004},"max":{"1min":23.277999999999999,"5min":28045.624,"15min":59439.283000000003},"last":12.569000000000001}]},{"osd":5,"last update":"Mon Jun 3 09:06:04 2024","interfaces":[{"interface":"back","average":{"1min":12.478999999999999,"5min":20.228000000000002,"15min":25.997},"min":{"1min":8.6430000000000007,"5min":7.9720000000000004,"15min":7.234},"max":{"1min":17.442,"5min":139.43199999999999,"15min":701.22000000000003},"last":17.462},{"interface":"front","average":{"1min":13.507999999999999,"5min":25.760999999999999,"15min":27.553999999999998},"min":{"1min":10.042999999999999,"5min":7.258,"15min":6.242},"max":{"1min":20.611000000000001,"5min":278.63,"15min":598.35000000000002},"last":14.523999999999999}]},{"osd":6,"last update":"Mon Jun 3 09:03:54 2024","interfaces":[{"interface":"back","average":{"1min":28.524000000000001,"5min":20.654,"15min":23.878},"min":{"1min":6.0309999999999997,"5min":6.0309999999999997,"15min":6.0309999999999997},"max":{"1min":167.221,"5min":167.221,"15min":704.245},"last":18.721},{"interface":"front","average":{"1min":27.606000000000002,"5min":30.131,"15min":28.628},"min":{"1min":9.8650000000000002,"5min":9.7200000000000006,"15min":8.3580000000000005},"max":{"1min":123.938,"5min":358.23000000000002,"15min":634.05499999999995},"last":173.113}]},{"osd":7,"last update":"Mon Jun 3 09:06:20 2024","interfaces":[{"interface":"back","average":{"1min":14.66,"5min":23.234999999999999,"15min":31.097000000000001},"min":{"1min":11.250999999999999,"5min":8.0559999999999992,"15min":8.0559999999999992},"max":{"1min":21.507999999999999,"5min":270.95100000000002,"15min":703.07799999999997},"last":15.736000000000001},{"interface":"front","average":{"1min":15.548,"5min":28.760000000000002,"15min":28.925000000000001},"min":{"1min":12.759,"5min":8.2089999999999996,"15min":7.7599999999999998},"max":{"1min":22.408000000000001,"5min":360.99900000000002,"15min":608.44100000000003},"last":16.59}]}]},{"osd":0,"up_from":14,"seq":60129542503,"num_pgs":10,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":81108,"kb_used_data":5704,"kb_used_omap":1,"kb_used_meta":75390,"kb_avail":94290732,"statfs":{"total":96636764160,"available":96553709568,"internally_reserved":0,"allocated":5840896,"data_stored":2740727,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1593,"internal_metadata":77199815},"hb_peers":[1,2,3,4,5,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[{"osd":1,"last update":"Mon Jun 3 09:06:04 2024","interfaces":[{"interface":"back","average":{"1min":6.9960000000000004,"5min":56.997,"15min":339.29700000000003},"min":{"1min":4.4530000000000003,"5min":4.4530000000000003,"15min":4.4100000000000001},"max":{"1min":11.720000000000001,"5min":2414.4749999999999,"15min":20691.453000000001},"last":4.915},{"interface":"front","average":{"1min":9.548,"5min":57.258000000000003,"15min":337.791},"min":{"1min":5.2300000000000004,"5min":5.2300000000000004,"15min":4.6050000000000004},"max":{"1min":15.337999999999999,"5min":2412.9549999999999,"15min":20640.792000000001},"last":9.7650000000000006}]},{"osd":2,"last update":"Mon Jun 3 09:06:22 2024","interfaces":[{"interface":"back","average":{"1min":11.154,"5min":26.515999999999998,"15min":22.882000000000001},"min":{"1min":5.7869999999999999,"5min":5.766,"15min":5.766},"max":{"1min":72.269999999999996,"5min":627.40700000000004,"15min":627.40700000000004},"last":7.2930000000000001},{"interface":"front","average":{"1min":14.353999999999999,"5min":33.304000000000002,"15min":29.885000000000002},"min":{"1min":8.2149999999999999,"5min":7.8630000000000004,"15min":7.4089999999999998},"max":{"1min":73.418000000000006,"5min":635.62599999999998,"15min":635.62599999999998},"last":9.0449999999999999}]},{"osd":3,"last update":"Mon Jun 3 09:06:04 2024","interfaces":[{"interface":"back","average":{"1min":12.105,"5min":26.925000000000001,"15min":26.584},"min":{"1min":7.8410000000000002,"5min":7.1280000000000001,"15min":6.1909999999999998},"max":{"1min":18.425999999999998,"5min":389.72699999999998,"15min":389.72699999999998},"last":12.268000000000001},{"interface":"front","average":{"1min":13.794,"5min":29.771999999999998,"15min":28.815000000000001},"min":{"1min":10.084,"5min":9.4920000000000009,"15min":6.1849999999999996},"max":{"1min":17.373999999999999,"5min":390.23200000000003,"15min":494.92899999999997},"last":14.003}]},{"osd":4,"last update":"Mon Jun 3 09:06:27 2024","interfaces":[{"interface":"back","average":{"1min":11.352,"5min":745.30600000000004,"15min":363.48099999999999},"min":{"1min":6.415,"5min":6.415,"15min":6.415},"max":{"1min":16.724,"5min":20248.616000000002,"15min":25555.212},"last":10.99},{"interface":"front","average":{"1min":13.545,"5min":744.76999999999998,"15min":361.03100000000001},"min":{"1min":9.3889999999999993,"5min":8.8940000000000001,"15min":7.1429999999999998},"max":{"1min":18.411000000000001,"5min":20250.195,"15min":25556.159},"last":14.521000000000001}]},{"osd":5,"last update":"Mon Jun 3 09:06:17 2024","interfaces":[{"interface":"back","average":{"1min":9.7040000000000006,"5min":26.533999999999999,"15min":28.683},"min":{"1min":6.8129999999999997,"5min":6.8129999999999997,"15min":6.8129999999999997},"max":{"1min":14.942,"5min":455.78699999999998,"15min":605.05899999999997},"last":9.4299999999999997},{"interface":"front","average":{"1min":14.279,"5min":28.398,"15min":26.192},"min":{"1min":11.696,"5min":6.4420000000000002,"15min":6.2030000000000003},"max":{"1min":17.879000000000001,"5min":391.31200000000001,"15min":541.11000000000001},"last":15.06}]},{"osd":6,"last update":"Mon Jun 3 09:04:30 2024","interfaces":[{"interface":"back","average":{"1min":52.496000000000002,"5min":33.139000000000003,"15min":160.83199999999999},"min":{"1min":7.5289999999999999,"5min":7.1040000000000001,"15min":6.3739999999999997},"max":{"1min":457.91699999999997,"5min":457.91699999999997,"15min":11115.258},"last":18.959},{"interface":"front","average":{"1min":49.670000000000002,"5min":33.704999999999998,"15min":162.791},"min":{"1min":12.247999999999999,"5min":11.019,"15min":9.9760000000000009},"max":{"1min":407.05099999999999,"5min":407.05099999999999,"15min":11092.194},"last":17.707000000000001}]},{"osd":7,"last update":"Mon Jun 3 09:06:22 2024","interfaces":[{"interface":"back","average":{"1min":11.625999999999999,"5min":26.375,"15min":28.972999999999999},"min":{"1min":6.4480000000000004,"5min":6.4480000000000004,"15min":6.4480000000000004},"max":{"1min":15.352,"5min":461.41699999999997,"15min":598.33699999999999},"last":10.305999999999999},{"interface":"front","average":{"1min":14.013,"5min":29.696000000000002,"15min":33.630000000000003},"min":{"1min":10.938000000000001,"5min":10.58,"15min":10.263999999999999},"max":{"1min":16.603000000000002,"5min":463.10599999999999,"15min":592.80499999999995},"last":11.743}]}]}],"pool_statfs":[{"poolid":1,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":401408,"data_stored":397840,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":5,"total":0,"available":0,"internally_reserved":0,"allocated":401408,"data_stored":397840,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":4,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":5,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":6,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":17,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":17,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":17,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":17,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":17,"osd":4,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":17,"osd":5,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":5,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":17,"osd":6,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":5,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":17,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0}]}} 2024-06-03T09:06:31.934 DEBUG:teuthology.orchestra.run.smithi088:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph pg dump --format=json 2024-06-03T09:06:32.034 INFO:teuthology.orchestra.run.smithi088.stderr:2024-06-03T09:06:32.033+0000 7fd2ca380640 10 monclient: get_monmap_and_config 2024-06-03T09:06:32.034 INFO:teuthology.orchestra.run.smithi088.stderr:2024-06-03T09:06:32.033+0000 7fd2ca380640 10 monclient: build_initial_monmap 2024-06-03T09:06:32.034 INFO:teuthology.orchestra.run.smithi088.stderr:2024-06-03T09:06:32.033+0000 7fd2ca380640 10 monclient: monmap: 2024-06-03T09:06:32.034 INFO:teuthology.orchestra.run.smithi088.stderr:epoch 0 2024-06-03T09:06:32.035 INFO:teuthology.orchestra.run.smithi088.stderr:fsid ec3c3fdd-3ac6-4d7d-9412-683130ed8b04 2024-06-03T09:06:32.035 INFO:teuthology.orchestra.run.smithi088.stderr:last_changed 2024-06-03T09:06:32.034722+0000 2024-06-03T09:06:32.035 INFO:teuthology.orchestra.run.smithi088.stderr:created 2024-06-03T09:06:32.034722+0000 2024-06-03T09:06:32.035 INFO:teuthology.orchestra.run.smithi088.stderr:min_mon_release 0 (unknown) 2024-06-03T09:06:32.035 INFO:teuthology.orchestra.run.smithi088.stderr:election_strategy: 3 2024-06-03T09:06:32.035 INFO:teuthology.orchestra.run.smithi088.stderr:0: v1:172.21.15.88:6789/0 mon.noname-a 2024-06-03T09:06:32.036 INFO:teuthology.orchestra.run.smithi088.stderr:1: v1:172.21.15.192:6789/0 mon.noname-c 2024-06-03T09:06:32.036 INFO:teuthology.orchestra.run.smithi088.stderr:2: v1:172.21.15.88:6790/0 mon.noname-b 2024-06-03T09:06:32.036 INFO:teuthology.orchestra.run.smithi088.stderr: 2024-06-03T09:06:32.043 INFO:teuthology.orchestra.run.smithi088.stderr:2024-06-03T09:06:32.034+0000 7fd2ca380640 1 Processor -- start 2024-06-03T09:06:32.043 INFO:teuthology.orchestra.run.smithi088.stderr:2024-06-03T09:06:32.034+0000 7fd2ca380640 1 -- start start 2024-06-03T09:06:32.043 INFO:teuthology.orchestra.run.smithi088.stderr:2024-06-03T09:06:32.034+0000 7fd2ca380640 10 monclient: init 2024-06-03T09:06:32.043 INFO:teuthology.orchestra.run.smithi088.stderr:2024-06-03T09:06:32.035+0000 7fd2ca380640 10 monclient: _reopen_session rank -1 2024-06-03T09:06:32.043 INFO:teuthology.orchestra.run.smithi088.stderr:2024-06-03T09:06:32.035+0000 7fd2ca380640 10 monclient: _add_conns ranks=[0,2,1] 2024-06-03T09:06:32.044 INFO:teuthology.orchestra.run.smithi088.stderr:2024-06-03T09:06:32.035+0000 7fd2ca380640 10 monclient(hunting): picked mon.noname-a con 0x7fd2c406fc70 addr v1:172.21.15.88:6789/0 2024-06-03T09:06:32.044 INFO:teuthology.orchestra.run.smithi088.stderr:2024-06-03T09:06:32.035+0000 7fd2ca380640 10 monclient(hunting): picked mon.noname-b con 0x7fd2c4070ed0 addr v1:172.21.15.88:6790/0 2024-06-03T09:06:32.044 INFO:teuthology.orchestra.run.smithi088.stderr:2024-06-03T09:06:32.035+0000 7fd2ca380640 10 monclient(hunting): picked mon.noname-c con 0x7fd2c413fa00 addr v1:172.21.15.192:6789/0 2024-06-03T09:06:32.044 INFO:teuthology.orchestra.run.smithi088.stderr:2024-06-03T09:06:32.035+0000 7fd2ca380640 1 -- --> v1:172.21.15.88:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fd2c4071f30 con 0x7fd2c406fc70 2024-06-03T09:06:32.044 INFO:teuthology.orchestra.run.smithi088.stderr:2024-06-03T09:06:32.035+0000 7fd2ca380640 1 -- --> v1:172.21.15.192:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fd2c40acce0 con 0x7fd2c413fa00 2024-06-03T09:06:32.044 INFO:teuthology.orchestra.run.smithi088.stderr:2024-06-03T09:06:32.035+0000 7fd2ca380640 1 -- --> v1:172.21.15.88:6790/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fd2c4070050 con 0x7fd2c4070ed0 2024-06-03T09:06:32.045 INFO:teuthology.orchestra.run.smithi088.stderr:2024-06-03T09:06:32.035+0000 7fd2ca380640 10 monclient(hunting): _renew_subs 2024-06-03T09:06:32.045 INFO:teuthology.orchestra.run.smithi088.stderr:2024-06-03T09:06:32.035+0000 7fd2ca380640 10 monclient(hunting): authenticate will time out at 3047.845123s 2024-06-03T09:06:32.045 INFO:teuthology.orchestra.run.smithi088.stderr:2024-06-03T09:06:32.035+0000 7fd2c3fff640 1 --1- >> v1:172.21.15.88:6789/0 conn(0x7fd2c406fc70 0x7fd2c413c720 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.88:6789/0 says I am v1:172.21.15.88:41568/0 (socket says 172.21.15.88:41568) 2024-06-03T09:06:32.045 INFO:teuthology.orchestra.run.smithi088.stderr:2024-06-03T09:06:32.035+0000 7fd2c3fff640 1 -- 172.21.15.88:0/3501757994 learned_addr learned my addr 172.21.15.88:0/3501757994 (peer_addr_for_me v1:172.21.15.88:0/0) 2024-06-03T09:06:32.045 INFO:teuthology.orchestra.run.smithi088.stderr:2024-06-03T09:06:32.038+0000 7fd2c2ffd640 1 -- 172.21.15.88:0/3501757994 <== mon.0 v1:172.21.15.88:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 4265501651 0 0) 0x7fd2c4071f30 con 0x7fd2c406fc70 2024-06-03T09:06:32.046 INFO:teuthology.orchestra.run.smithi088.stderr:2024-06-03T09:06:32.038+0000 7fd2c2ffd640 10 monclient(hunting): _negotiate 2024-06-03T09:06:32.046 INFO:teuthology.orchestra.run.smithi088.stderr:2024-06-03T09:06:32.038+0000 7fd2c2ffd640 10 monclient(hunting): _init_auth method 2 2024-06-03T09:06:32.046 INFO:teuthology.orchestra.run.smithi088.stderr:2024-06-03T09:06:32.038+0000 7fd2c2ffd640 10 monclient(hunting): _init_auth creating new auth 2024-06-03T09:06:32.046 INFO:teuthology.orchestra.run.smithi088.stderr:2024-06-03T09:06:32.038+0000 7fd2c2ffd640 10 monclient(hunting): my global_id is 30630 2024-06-03T09:06:32.046 INFO:teuthology.orchestra.run.smithi088.stderr:2024-06-03T09:06:32.038+0000 7fd2c2ffd640 1 -- 172.21.15.88:0/3501757994 --> v1:172.21.15.88:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7fd2ac003520 con 0x7fd2c406fc70 2024-06-03T09:06:32.047 INFO:teuthology.orchestra.run.smithi088.stderr:2024-06-03T09:06:32.038+0000 7fd2c2ffd640 1 -- 172.21.15.88:0/3501757994 <== mon.2 v1:172.21.15.88:6790/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 2139040034 0 0) 0x7fd2c4070050 con 0x7fd2c4070ed0 2024-06-03T09:06:32.047 INFO:teuthology.orchestra.run.smithi088.stderr:2024-06-03T09:06:32.038+0000 7fd2c2ffd640 10 monclient(hunting): _negotiate 2024-06-03T09:06:32.047 INFO:teuthology.orchestra.run.smithi088.stderr:2024-06-03T09:06:32.038+0000 7fd2c2ffd640 10 monclient(hunting): _init_auth method 2 2024-06-03T09:06:32.047 INFO:teuthology.orchestra.run.smithi088.stderr:2024-06-03T09:06:32.038+0000 7fd2c2ffd640 10 monclient(hunting): _init_auth creating new auth 2024-06-03T09:06:32.047 INFO:teuthology.orchestra.run.smithi088.stderr:2024-06-03T09:06:32.038+0000 7fd2c2ffd640 10 monclient(hunting): my global_id is 30134 2024-06-03T09:06:32.047 INFO:teuthology.orchestra.run.smithi088.stderr:2024-06-03T09:06:32.038+0000 7fd2c2ffd640 1 -- 172.21.15.88:0/3501757994 --> v1:172.21.15.88:6790/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7fd2c4071f30 con 0x7fd2c4070ed0 2024-06-03T09:06:32.048 INFO:teuthology.orchestra.run.smithi088.stderr:2024-06-03T09:06:32.038+0000 7fd2c2ffd640 1 -- 172.21.15.88:0/3501757994 <== mon.1 v1:172.21.15.192:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 1389885937 0 0) 0x7fd2c40acce0 con 0x7fd2c413fa00 2024-06-03T09:06:32.048 INFO:teuthology.orchestra.run.smithi088.stderr:2024-06-03T09:06:32.038+0000 7fd2c2ffd640 10 monclient(hunting): _negotiate 2024-06-03T09:06:32.048 INFO:teuthology.orchestra.run.smithi088.stderr:2024-06-03T09:06:32.038+0000 7fd2c2ffd640 10 monclient(hunting): _init_auth method 2 2024-06-03T09:06:32.048 INFO:teuthology.orchestra.run.smithi088.stderr:2024-06-03T09:06:32.038+0000 7fd2c2ffd640 10 monclient(hunting): _init_auth creating new auth 2024-06-03T09:06:32.048 INFO:teuthology.orchestra.run.smithi088.stderr:2024-06-03T09:06:32.038+0000 7fd2c2ffd640 10 monclient(hunting): my global_id is 30211 2024-06-03T09:06:32.049 INFO:teuthology.orchestra.run.smithi088.stderr:2024-06-03T09:06:32.038+0000 7fd2c2ffd640 1 -- 172.21.15.88:0/3501757994 --> v1:172.21.15.192:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7fd2c4070050 con 0x7fd2c413fa00 2024-06-03T09:06:32.049 INFO:teuthology.orchestra.run.smithi088.stderr:2024-06-03T09:06:32.041+0000 7fd2c2ffd640 1 -- 172.21.15.88:0/3501757994 <== mon.0 v1:172.21.15.88:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 214+0+0 (unknown 2030704035 0 0) 0x7fd2ac003520 con 0x7fd2c406fc70 2024-06-03T09:06:32.049 INFO:teuthology.orchestra.run.smithi088.stderr:2024-06-03T09:06:32.041+0000 7fd2c2ffd640 1 -- 172.21.15.88:0/3501757994 >> v1:172.21.15.88:6790/0 conn(0x7fd2c4070ed0 legacy=0x7fd2c413f2f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-06-03T09:06:32.049 INFO:teuthology.orchestra.run.smithi088.stderr:2024-06-03T09:06:32.041+0000 7fd2c2ffd640 1 -- 172.21.15.88:0/3501757994 >> v1:172.21.15.192:6789/0 conn(0x7fd2c413fa00 legacy=0x7fd2c414abf0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-06-03T09:06:32.049 INFO:teuthology.orchestra.run.smithi088.stderr:2024-06-03T09:06:32.041+0000 7fd2c2ffd640 10 monclient: _finish_hunting 0 2024-06-03T09:06:32.050 INFO:teuthology.orchestra.run.smithi088.stderr:2024-06-03T09:06:32.041+0000 7fd2c2ffd640 1 monclient: found mon.noname-a 2024-06-03T09:06:32.050 INFO:teuthology.orchestra.run.smithi088.stderr:2024-06-03T09:06:32.041+0000 7fd2c2ffd640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-06-03T09:06:32.050 INFO:teuthology.orchestra.run.smithi088.stderr:2024-06-03T09:06:32.041+0000 7fd2c2ffd640 10 monclient: _send_mon_message to mon.noname-a at v1:172.21.15.88:6789/0 2024-06-03T09:06:32.050 INFO:teuthology.orchestra.run.smithi088.stderr:2024-06-03T09:06:32.041+0000 7fd2c2ffd640 1 -- 172.21.15.88:0/3501757994 --> v1:172.21.15.88:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fd2c413ce30 con 0x7fd2c406fc70 2024-06-03T09:06:32.050 INFO:teuthology.orchestra.run.smithi088.stderr:2024-06-03T09:06:32.041+0000 7fd2c2ffd640 10 monclient: _finish_auth 0 2024-06-03T09:06:32.051 INFO:teuthology.orchestra.run.smithi088.stderr:2024-06-03T09:06:32.041+0000 7fd2c2ffd640 10 monclient: _check_auth_tickets 2024-06-03T09:06:32.051 INFO:teuthology.orchestra.run.smithi088.stderr:2024-06-03T09:06:32.041+0000 7fd2c2ffd640 20 monclient: _check_auth_rotating not needed by client.admin 2024-06-03T09:06:32.051 INFO:teuthology.orchestra.run.smithi088.stderr:2024-06-03T09:06:32.041+0000 7fd2c2ffd640 1 -- 172.21.15.88:0/3501757994 <== mon.0 v1:172.21.15.88:6789/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 2234374065 0 0) 0x7fd2b4001610 con 0x7fd2c406fc70 2024-06-03T09:06:32.051 INFO:teuthology.orchestra.run.smithi088.stderr:2024-06-03T09:06:32.041+0000 7fd2ca380640 5 monclient: authenticate success, global_id 30630 2024-06-03T09:06:32.051 INFO:teuthology.orchestra.run.smithi088.stderr:2024-06-03T09:06:32.041+0000 7fd2ca380640 20 monclient: get_monmap_and_config waiting for monmap|config 2024-06-03T09:06:32.052 INFO:teuthology.orchestra.run.smithi088.stderr:2024-06-03T09:06:32.041+0000 7fd2c2ffd640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-06-03T09:06:32.052 INFO:teuthology.orchestra.run.smithi088.stderr:2024-06-03T09:06:32.041+0000 7fd2c2ffd640 10 monclient: got monmap 1 from mon.noname-a (according to old e1) 2024-06-03T09:06:32.052 INFO:teuthology.orchestra.run.smithi088.stderr:2024-06-03T09:06:32.041+0000 7fd2c2ffd640 10 monclient: dump: 2024-06-03T09:06:32.052 INFO:teuthology.orchestra.run.smithi088.stderr:epoch 1 2024-06-03T09:06:32.052 INFO:teuthology.orchestra.run.smithi088.stderr:fsid ec3c3fdd-3ac6-4d7d-9412-683130ed8b04 2024-06-03T09:06:32.052 INFO:teuthology.orchestra.run.smithi088.stderr:last_changed 2024-06-03T08:34:31.885125+0000 2024-06-03T09:06:32.053 INFO:teuthology.orchestra.run.smithi088.stderr:created 2024-06-03T08:34:31.885125+0000 2024-06-03T09:06:32.053 INFO:teuthology.orchestra.run.smithi088.stderr:min_mon_release 18 (reef) 2024-06-03T09:06:32.053 INFO:teuthology.orchestra.run.smithi088.stderr:election_strategy: 3 2024-06-03T09:06:32.053 INFO:teuthology.orchestra.run.smithi088.stderr:0: v1:172.21.15.88:6789/0 mon.a 2024-06-03T09:06:32.053 INFO:teuthology.orchestra.run.smithi088.stderr:1: v1:172.21.15.192:6789/0 mon.b 2024-06-03T09:06:32.054 INFO:teuthology.orchestra.run.smithi088.stderr:2: v1:172.21.15.88:6790/0 mon.c 2024-06-03T09:06:32.054 INFO:teuthology.orchestra.run.smithi088.stderr: 2024-06-03T09:06:32.054 INFO:teuthology.orchestra.run.smithi088.stderr:2024-06-03T09:06:32.041+0000 7fd2ca380640 20 monclient: get_monmap_and_config waiting for monmap|config 2024-06-03T09:06:32.054 INFO:teuthology.orchestra.run.smithi088.stderr:2024-06-03T09:06:32.043+0000 7fd2c2ffd640 1 -- 172.21.15.88:0/3501757994 <== mon.0 v1:172.21.15.88:6789/0 4 ==== config(0 keys) v1 ==== 4+0+0 (unknown 0 0 0) 0x7fd2b4001830 con 0x7fd2c406fc70 2024-06-03T09:06:32.054 INFO:teuthology.orchestra.run.smithi088.stderr:2024-06-03T09:06:32.043+0000 7fd2c2ffd640 10 monclient: handle_config config(0 keys) v1 2024-06-03T09:06:32.054 INFO:teuthology.orchestra.run.smithi088.stderr:2024-06-03T09:06:32.043+0000 7fd2c2ffd640 1 -- 172.21.15.88:0/3501757994 <== mon.0 v1:172.21.15.88:6789/0 5 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 2234374065 0 0) 0x7fd2b4001bc0 con 0x7fd2c406fc70 2024-06-03T09:06:32.055 INFO:teuthology.orchestra.run.smithi088.stderr:2024-06-03T09:06:32.043+0000 7fd2c2ffd640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-06-03T09:06:32.055 INFO:teuthology.orchestra.run.smithi088.stderr:2024-06-03T09:06:32.043+0000 7fd2c2ffd640 10 monclient: got monmap 1 from mon.a (according to old e1) 2024-06-03T09:06:32.055 INFO:teuthology.orchestra.run.smithi088.stderr:2024-06-03T09:06:32.043+0000 7fd2c2ffd640 10 monclient: dump: 2024-06-03T09:06:32.055 INFO:teuthology.orchestra.run.smithi088.stderr:epoch 1 2024-06-03T09:06:32.055 INFO:teuthology.orchestra.run.smithi088.stderr:fsid ec3c3fdd-3ac6-4d7d-9412-683130ed8b04 2024-06-03T09:06:32.056 INFO:teuthology.orchestra.run.smithi088.stderr:last_changed 2024-06-03T08:34:31.885125+0000 2024-06-03T09:06:32.056 INFO:teuthology.orchestra.run.smithi088.stderr:created 2024-06-03T08:34:31.885125+0000 2024-06-03T09:06:32.056 INFO:teuthology.orchestra.run.smithi088.stderr:min_mon_release 18 (reef) 2024-06-03T09:06:32.056 INFO:teuthology.orchestra.run.smithi088.stderr:election_strategy: 3 2024-06-03T09:06:32.056 INFO:teuthology.orchestra.run.smithi088.stderr:0: v1:172.21.15.88:6789/0 mon.a 2024-06-03T09:06:32.057 INFO:teuthology.orchestra.run.smithi088.stderr:1: v1:172.21.15.192:6789/0 mon.b 2024-06-03T09:06:32.057 INFO:teuthology.orchestra.run.smithi088.stderr:2: v1:172.21.15.88:6790/0 mon.c 2024-06-03T09:06:32.057 INFO:teuthology.orchestra.run.smithi088.stderr: 2024-06-03T09:06:32.057 INFO:teuthology.orchestra.run.smithi088.stderr:2024-06-03T09:06:32.043+0000 7fd2ca380640 10 monclient: get_monmap_and_config success 2024-06-03T09:06:32.057 INFO:teuthology.orchestra.run.smithi088.stderr:2024-06-03T09:06:32.043+0000 7fd2ca380640 4 set_mon_vals no callback set 2024-06-03T09:06:32.062 INFO:teuthology.orchestra.run.smithi088.stderr:2024-06-03T09:06:32.043+0000 7fd2ca380640 10 monclient: shutdown 2024-06-03T09:06:32.062 INFO:teuthology.orchestra.run.smithi088.stderr:2024-06-03T09:06:32.043+0000 7fd2ca380640 20 monclient: shutdown discarding 0 pending message(s) 2024-06-03T09:06:32.062 INFO:teuthology.orchestra.run.smithi088.stderr:2024-06-03T09:06:32.043+0000 7fd2ca380640 1 -- 172.21.15.88:0/3501757994 >> v1:172.21.15.88:6789/0 conn(0x7fd2c406fc70 legacy=0x7fd2c413c720 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-06-03T09:06:32.063 INFO:teuthology.orchestra.run.smithi088.stderr:2024-06-03T09:06:32.043+0000 7fd2ca380640 1 -- 172.21.15.88:0/3501757994 shutdown_connections 2024-06-03T09:06:32.063 INFO:teuthology.orchestra.run.smithi088.stderr:2024-06-03T09:06:32.043+0000 7fd2ca380640 1 -- 172.21.15.88:0/3501757994 >> 172.21.15.88:0/3501757994 conn(0x7fd2c40b2f20 msgr2=0x7fd2c40b3300 unknown :-1 s=STATE_NONE l=0).mark_down 2024-06-03T09:06:32.063 INFO:teuthology.orchestra.run.smithi088.stderr:2024-06-03T09:06:32.043+0000 7fd2ca380640 1 -- 172.21.15.88:0/3501757994 shutdown_connections 2024-06-03T09:06:32.063 INFO:teuthology.orchestra.run.smithi088.stderr:2024-06-03T09:06:32.043+0000 7fd2ca380640 1 -- 172.21.15.88:0/3501757994 wait complete. 2024-06-03T09:06:32.063 INFO:teuthology.orchestra.run.smithi088.stderr:2024-06-03T09:06:32.044+0000 7fd2ca380640 10 monclient: build_initial_monmap 2024-06-03T09:06:32.063 INFO:teuthology.orchestra.run.smithi088.stderr:2024-06-03T09:06:32.044+0000 7fd2ca380640 10 monclient: monmap: 2024-06-03T09:06:32.063 INFO:teuthology.orchestra.run.smithi088.stderr:epoch 0 2024-06-03T09:06:32.063 INFO:teuthology.orchestra.run.smithi088.stderr:fsid 00000000-0000-0000-0000-000000000000 2024-06-03T09:06:32.063 INFO:teuthology.orchestra.run.smithi088.stderr:last_changed 0.000000 2024-06-03T09:06:32.063 INFO:teuthology.orchestra.run.smithi088.stderr:created 0.000000 2024-06-03T09:06:32.063 INFO:teuthology.orchestra.run.smithi088.stderr:min_mon_release 0 (unknown) 2024-06-03T09:06:32.063 INFO:teuthology.orchestra.run.smithi088.stderr:election_strategy: 1 2024-06-03T09:06:32.064 INFO:teuthology.orchestra.run.smithi088.stderr:0: v1:172.21.15.88:6789/0 mon.noname-a 2024-06-03T09:06:32.064 INFO:teuthology.orchestra.run.smithi088.stderr:1: v1:172.21.15.192:6789/0 mon.noname-b 2024-06-03T09:06:32.064 INFO:teuthology.orchestra.run.smithi088.stderr:2: v1:172.21.15.88:6790/0 mon.noname-c 2024-06-03T09:06:32.064 INFO:teuthology.orchestra.run.smithi088.stderr: 2024-06-03T09:06:32.064 INFO:teuthology.orchestra.run.smithi088.stderr:2024-06-03T09:06:32.044+0000 7fd2ca380640 1 librados: starting msgr at 2024-06-03T09:06:32.064 INFO:teuthology.orchestra.run.smithi088.stderr:2024-06-03T09:06:32.044+0000 7fd2ca380640 1 librados: starting objecter 2024-06-03T09:06:32.064 INFO:teuthology.orchestra.run.smithi088.stderr:2024-06-03T09:06:32.044+0000 7fd2ca380640 1 Processor -- start 2024-06-03T09:06:32.064 INFO:teuthology.orchestra.run.smithi088.stderr:2024-06-03T09:06:32.045+0000 7fd2ca380640 1 -- start start 2024-06-03T09:06:32.064 INFO:teuthology.orchestra.run.smithi088.stderr:2024-06-03T09:06:32.045+0000 7fd2ca380640 1 librados: setting wanted keys 2024-06-03T09:06:32.064 INFO:teuthology.orchestra.run.smithi088.stderr:2024-06-03T09:06:32.045+0000 7fd2ca380640 1 librados: calling monclient init 2024-06-03T09:06:32.064 INFO:teuthology.orchestra.run.smithi088.stderr:2024-06-03T09:06:32.045+0000 7fd2ca380640 10 monclient: init 2024-06-03T09:06:32.064 INFO:teuthology.orchestra.run.smithi088.stderr:2024-06-03T09:06:32.045+0000 7fd2ca380640 10 monclient: _reopen_session rank -1 2024-06-03T09:06:32.064 INFO:teuthology.orchestra.run.smithi088.stderr:2024-06-03T09:06:32.045+0000 7fd2ca380640 10 monclient: _add_conns ranks=[0,1,2] 2024-06-03T09:06:32.064 INFO:teuthology.orchestra.run.smithi088.stderr:2024-06-03T09:06:32.045+0000 7fd2ca380640 10 monclient(hunting): picked mon.noname-a con 0x7fd2c406fc70 addr v1:172.21.15.88:6789/0 2024-06-03T09:06:32.065 INFO:teuthology.orchestra.run.smithi088.stderr:2024-06-03T09:06:32.045+0000 7fd2ca380640 10 monclient(hunting): picked mon.noname-b con 0x7fd2c4070ed0 addr v1:172.21.15.192:6789/0 2024-06-03T09:06:32.065 INFO:teuthology.orchestra.run.smithi088.stderr:2024-06-03T09:06:32.045+0000 7fd2ca380640 10 monclient(hunting): picked mon.noname-c con 0x7fd2c413fa00 addr v1:172.21.15.88:6790/0 2024-06-03T09:06:32.065 INFO:teuthology.orchestra.run.smithi088.stderr:2024-06-03T09:06:32.045+0000 7fd2ca380640 1 -- --> v1:172.21.15.88:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fd2c4130700 con 0x7fd2c406fc70 2024-06-03T09:06:32.065 INFO:teuthology.orchestra.run.smithi088.stderr:2024-06-03T09:06:32.045+0000 7fd2ca380640 1 -- --> v1:172.21.15.192:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fd2c40abbe0 con 0x7fd2c4070ed0 2024-06-03T09:06:32.065 INFO:teuthology.orchestra.run.smithi088.stderr:2024-06-03T09:06:32.045+0000 7fd2ca380640 1 -- --> v1:172.21.15.88:6790/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fd2c41d1f70 con 0x7fd2c413fa00 2024-06-03T09:06:32.065 INFO:teuthology.orchestra.run.smithi088.stderr:2024-06-03T09:06:32.045+0000 7fd2ca380640 10 monclient(hunting): _renew_subs 2024-06-03T09:06:32.065 INFO:teuthology.orchestra.run.smithi088.stderr:2024-06-03T09:06:32.045+0000 7fd2ca380640 10 monclient(hunting): authenticate will time out at 3047.855592s 2024-06-03T09:06:32.065 INFO:teuthology.orchestra.run.smithi088.stderr:2024-06-03T09:06:32.045+0000 7fd2c3fff640 1 --1- >> v1:172.21.15.88:6789/0 conn(0x7fd2c406fc70 0x7fd2c412fff0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=1).handle_server_banner_and_identify peer v1:172.21.15.88:6789/0 says I am v1:172.21.15.88:41574/0 (socket says 172.21.15.88:41574) 2024-06-03T09:06:32.065 INFO:teuthology.orchestra.run.smithi088.stderr:2024-06-03T09:06:32.045+0000 7fd2c3fff640 1 -- 172.21.15.88:0/1705515602 learned_addr learned my addr 172.21.15.88:0/1705515602 (peer_addr_for_me v1:172.21.15.88:0/0) 2024-06-03T09:06:32.065 INFO:teuthology.orchestra.run.smithi088.stderr:2024-06-03T09:06:32.046+0000 7fd2c0ff9640 10 client.?.objecter ms_handle_connect 0x7fd2c406fc70 2024-06-03T09:06:32.066 INFO:teuthology.orchestra.run.smithi088.stderr:2024-06-03T09:06:32.046+0000 7fd2c0ff9640 10 client.?.objecter resend_mon_ops 2024-06-03T09:06:32.066 INFO:teuthology.orchestra.run.smithi088.stderr:2024-06-03T09:06:32.046+0000 7fd2c0ff9640 10 client.?.objecter ms_handle_connect 0x7fd2c413fa00 2024-06-03T09:06:32.066 INFO:teuthology.orchestra.run.smithi088.stderr:2024-06-03T09:06:32.046+0000 7fd2c0ff9640 10 client.?.objecter resend_mon_ops 2024-06-03T09:06:32.066 INFO:teuthology.orchestra.run.smithi088.stderr:2024-06-03T09:06:32.046+0000 7fd2c0ff9640 10 client.?.objecter ms_handle_connect 0x7fd2c4070ed0 2024-06-03T09:06:32.066 INFO:teuthology.orchestra.run.smithi088.stderr:2024-06-03T09:06:32.046+0000 7fd2c0ff9640 10 client.?.objecter resend_mon_ops 2024-06-03T09:06:32.067 INFO:teuthology.orchestra.run.smithi088.stderr:2024-06-03T09:06:32.048+0000 7fd2c0ff9640 1 -- 172.21.15.88:0/1705515602 <== mon.1 v1:172.21.15.192:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 2313675106 0 0) 0x7fd2c40abbe0 con 0x7fd2c4070ed0 2024-06-03T09:06:32.067 INFO:teuthology.orchestra.run.smithi088.stderr:2024-06-03T09:06:32.048+0000 7fd2c0ff9640 10 monclient(hunting): _negotiate 2024-06-03T09:06:32.067 INFO:teuthology.orchestra.run.smithi088.stderr:2024-06-03T09:06:32.048+0000 7fd2c0ff9640 10 monclient(hunting): _init_auth method 2 2024-06-03T09:06:32.067 INFO:teuthology.orchestra.run.smithi088.stderr:2024-06-03T09:06:32.048+0000 7fd2c0ff9640 10 monclient(hunting): _init_auth creating new auth 2024-06-03T09:06:32.067 INFO:teuthology.orchestra.run.smithi088.stderr:2024-06-03T09:06:32.048+0000 7fd2c0ff9640 10 monclient(hunting): my global_id is 30214 2024-06-03T09:06:32.067 INFO:teuthology.orchestra.run.smithi088.stderr:2024-06-03T09:06:32.048+0000 7fd2c0ff9640 1 -- 172.21.15.88:0/1705515602 --> v1:172.21.15.192:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7fd2b0004eb0 con 0x7fd2c4070ed0 2024-06-03T09:06:32.068 INFO:teuthology.orchestra.run.smithi088.stderr:2024-06-03T09:06:32.048+0000 7fd2c0ff9640 1 -- 172.21.15.88:0/1705515602 <== mon.0 v1:172.21.15.88:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 3554398065 0 0) 0x7fd2c4130700 con 0x7fd2c406fc70 2024-06-03T09:06:32.068 INFO:teuthology.orchestra.run.smithi088.stderr:2024-06-03T09:06:32.048+0000 7fd2c0ff9640 10 monclient(hunting): _negotiate 2024-06-03T09:06:32.068 INFO:teuthology.orchestra.run.smithi088.stderr:2024-06-03T09:06:32.048+0000 7fd2c0ff9640 10 monclient(hunting): _init_auth method 2 2024-06-03T09:06:32.068 INFO:teuthology.orchestra.run.smithi088.stderr:2024-06-03T09:06:32.048+0000 7fd2c0ff9640 10 monclient(hunting): _init_auth creating new auth 2024-06-03T09:06:32.068 INFO:teuthology.orchestra.run.smithi088.stderr:2024-06-03T09:06:32.048+0000 7fd2c0ff9640 10 monclient(hunting): my global_id is 30633 2024-06-03T09:06:32.068 INFO:teuthology.orchestra.run.smithi088.stderr:2024-06-03T09:06:32.048+0000 7fd2c0ff9640 1 -- 172.21.15.88:0/1705515602 --> v1:172.21.15.88:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7fd2c40abbe0 con 0x7fd2c406fc70 2024-06-03T09:06:32.068 INFO:teuthology.orchestra.run.smithi088.stderr:2024-06-03T09:06:32.048+0000 7fd2c0ff9640 1 -- 172.21.15.88:0/1705515602 <== mon.2 v1:172.21.15.88:6790/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 528275272 0 0) 0x7fd2c41d1f70 con 0x7fd2c413fa00 2024-06-03T09:06:32.068 INFO:teuthology.orchestra.run.smithi088.stderr:2024-06-03T09:06:32.048+0000 7fd2c0ff9640 10 monclient(hunting): _negotiate 2024-06-03T09:06:32.068 INFO:teuthology.orchestra.run.smithi088.stderr:2024-06-03T09:06:32.048+0000 7fd2c0ff9640 10 monclient(hunting): _init_auth method 2 2024-06-03T09:06:32.069 INFO:teuthology.orchestra.run.smithi088.stderr:2024-06-03T09:06:32.048+0000 7fd2c0ff9640 10 monclient(hunting): _init_auth creating new auth 2024-06-03T09:06:32.069 INFO:teuthology.orchestra.run.smithi088.stderr:2024-06-03T09:06:32.048+0000 7fd2c0ff9640 10 monclient(hunting): my global_id is 30137 2024-06-03T09:06:32.069 INFO:teuthology.orchestra.run.smithi088.stderr:2024-06-03T09:06:32.048+0000 7fd2c0ff9640 1 -- 172.21.15.88:0/1705515602 --> v1:172.21.15.88:6790/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7fd2c4130700 con 0x7fd2c413fa00 2024-06-03T09:06:32.069 INFO:teuthology.orchestra.run.smithi088.stderr:2024-06-03T09:06:32.053+0000 7fd2c0ff9640 1 -- 172.21.15.88:0/1705515602 <== mon.0 v1:172.21.15.88:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 764+0+0 (unknown 145402126 0 0) 0x7fd2c40abbe0 con 0x7fd2c406fc70 2024-06-03T09:06:32.069 INFO:teuthology.orchestra.run.smithi088.stderr:2024-06-03T09:06:32.053+0000 7fd2c0ff9640 1 -- 172.21.15.88:0/1705515602 --> v1:172.21.15.88:6789/0 -- auth(proto 2 165 bytes epoch 0) v1 -- 0x7fd2c41d1f70 con 0x7fd2c406fc70 2024-06-03T09:06:32.069 INFO:teuthology.orchestra.run.smithi088.stderr:2024-06-03T09:06:32.053+0000 7fd2c0ff9640 1 -- 172.21.15.88:0/1705515602 <== mon.0 v1:172.21.15.88:6789/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 2234374065 0 0) 0x7fd2b4001b40 con 0x7fd2c406fc70 2024-06-03T09:06:32.069 INFO:teuthology.orchestra.run.smithi088.stderr:2024-06-03T09:06:32.053+0000 7fd2c0ff9640 10 monclient(hunting): handle_monmap mon_map magic: 0 v1 2024-06-03T09:06:32.069 INFO:teuthology.orchestra.run.smithi088.stderr:2024-06-03T09:06:32.053+0000 7fd2c0ff9640 10 monclient(hunting): got monmap 1 from mon.noname-a (according to old e1) 2024-06-03T09:06:32.069 INFO:teuthology.orchestra.run.smithi088.stderr:2024-06-03T09:06:32.053+0000 7fd2c0ff9640 10 monclient(hunting): dump: 2024-06-03T09:06:32.069 INFO:teuthology.orchestra.run.smithi088.stderr:epoch 1 2024-06-03T09:06:32.069 INFO:teuthology.orchestra.run.smithi088.stderr:fsid ec3c3fdd-3ac6-4d7d-9412-683130ed8b04 2024-06-03T09:06:32.069 INFO:teuthology.orchestra.run.smithi088.stderr:last_changed 2024-06-03T08:34:31.885125+0000 2024-06-03T09:06:32.069 INFO:teuthology.orchestra.run.smithi088.stderr:created 2024-06-03T08:34:31.885125+0000 2024-06-03T09:06:32.069 INFO:teuthology.orchestra.run.smithi088.stderr:min_mon_release 18 (reef) 2024-06-03T09:06:32.070 INFO:teuthology.orchestra.run.smithi088.stderr:election_strategy: 3 2024-06-03T09:06:32.070 INFO:teuthology.orchestra.run.smithi088.stderr:0: v1:172.21.15.88:6789/0 mon.a 2024-06-03T09:06:32.070 INFO:teuthology.orchestra.run.smithi088.stderr:1: v1:172.21.15.192:6789/0 mon.b 2024-06-03T09:06:32.070 INFO:teuthology.orchestra.run.smithi088.stderr:2: v1:172.21.15.88:6790/0 mon.c 2024-06-03T09:06:32.070 INFO:teuthology.orchestra.run.smithi088.stderr: 2024-06-03T09:06:32.070 INFO:teuthology.orchestra.run.smithi088.stderr:2024-06-03T09:06:32.053+0000 7fd2c0ff9640 10 monclient(hunting): _finish_auth 0 2024-06-03T09:06:32.070 INFO:teuthology.orchestra.run.smithi088.stderr:2024-06-03T09:06:32.053+0000 7fd2c0ff9640 1 -- 172.21.15.88:0/1705515602 <== mon.1 v1:172.21.15.192:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 764+0+0 (unknown 17844131 0 0) 0x7fd2b0004eb0 con 0x7fd2c4070ed0 2024-06-03T09:06:32.070 INFO:teuthology.orchestra.run.smithi088.stderr:2024-06-03T09:06:32.053+0000 7fd2c0ff9640 1 -- 172.21.15.88:0/1705515602 --> v1:172.21.15.192:6789/0 -- auth(proto 2 165 bytes epoch 0) v1 -- 0x7fd2c40abbe0 con 0x7fd2c4070ed0 2024-06-03T09:06:32.070 INFO:teuthology.orchestra.run.smithi088.stderr:2024-06-03T09:06:32.053+0000 7fd2c0ff9640 1 -- 172.21.15.88:0/1705515602 <== mon.2 v1:172.21.15.88:6790/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 764+0+0 (unknown 3034122097 0 0) 0x7fd2c4130700 con 0x7fd2c413fa00 2024-06-03T09:06:32.070 INFO:teuthology.orchestra.run.smithi088.stderr:2024-06-03T09:06:32.053+0000 7fd2c0ff9640 1 -- 172.21.15.88:0/1705515602 --> v1:172.21.15.88:6790/0 -- auth(proto 2 165 bytes epoch 0) v1 -- 0x7fd2b0004eb0 con 0x7fd2c413fa00 2024-06-03T09:06:32.070 INFO:teuthology.orchestra.run.smithi088.stderr:2024-06-03T09:06:32.053+0000 7fd2c0ff9640 1 -- 172.21.15.88:0/1705515602 <== mon.2 v1:172.21.15.88:6790/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 2234374065 0 0) 0x7fd2bc001b10 con 0x7fd2c413fa00 2024-06-03T09:06:32.070 INFO:teuthology.orchestra.run.smithi088.stderr:2024-06-03T09:06:32.053+0000 7fd2c0ff9640 10 monclient(hunting): handle_monmap mon_map magic: 0 v1 2024-06-03T09:06:32.071 INFO:teuthology.orchestra.run.smithi088.stderr:2024-06-03T09:06:32.053+0000 7fd2c0ff9640 10 monclient(hunting): got monmap 1 from mon.c (according to old e1) 2024-06-03T09:06:32.071 INFO:teuthology.orchestra.run.smithi088.stderr:2024-06-03T09:06:32.053+0000 7fd2c0ff9640 10 monclient(hunting): dump: 2024-06-03T09:06:32.071 INFO:teuthology.orchestra.run.smithi088.stderr:epoch 1 2024-06-03T09:06:32.071 INFO:teuthology.orchestra.run.smithi088.stderr:fsid ec3c3fdd-3ac6-4d7d-9412-683130ed8b04 2024-06-03T09:06:32.071 INFO:teuthology.orchestra.run.smithi088.stderr:last_changed 2024-06-03T08:34:31.885125+0000 2024-06-03T09:06:32.071 INFO:teuthology.orchestra.run.smithi088.stderr:created 2024-06-03T08:34:31.885125+0000 2024-06-03T09:06:32.071 INFO:teuthology.orchestra.run.smithi088.stderr:min_mon_release 18 (reef) 2024-06-03T09:06:32.071 INFO:teuthology.orchestra.run.smithi088.stderr:election_strategy: 3 2024-06-03T09:06:32.071 INFO:teuthology.orchestra.run.smithi088.stderr:0: v1:172.21.15.88:6789/0 mon.a 2024-06-03T09:06:32.071 INFO:teuthology.orchestra.run.smithi088.stderr:1: v1:172.21.15.192:6789/0 mon.b 2024-06-03T09:06:32.071 INFO:teuthology.orchestra.run.smithi088.stderr:2: v1:172.21.15.88:6790/0 mon.c 2024-06-03T09:06:32.071 INFO:teuthology.orchestra.run.smithi088.stderr: 2024-06-03T09:06:32.071 INFO:teuthology.orchestra.run.smithi088.stderr:2024-06-03T09:06:32.053+0000 7fd2c0ff9640 1 -- 172.21.15.88:0/1705515602 <== mon.1 v1:172.21.15.192:6789/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 2234374065 0 0) 0x7fd2b80018f0 con 0x7fd2c4070ed0 2024-06-03T09:06:32.072 INFO:teuthology.orchestra.run.smithi088.stderr:2024-06-03T09:06:32.053+0000 7fd2c0ff9640 10 monclient(hunting): handle_monmap mon_map magic: 0 v1 2024-06-03T09:06:32.072 INFO:teuthology.orchestra.run.smithi088.stderr:2024-06-03T09:06:32.053+0000 7fd2c0ff9640 10 monclient(hunting): got monmap 1 from mon.b (according to old e1) 2024-06-03T09:06:32.072 INFO:teuthology.orchestra.run.smithi088.stderr:2024-06-03T09:06:32.053+0000 7fd2c0ff9640 10 monclient(hunting): dump: 2024-06-03T09:06:32.072 INFO:teuthology.orchestra.run.smithi088.stderr:epoch 1 2024-06-03T09:06:32.072 INFO:teuthology.orchestra.run.smithi088.stderr:fsid ec3c3fdd-3ac6-4d7d-9412-683130ed8b04 2024-06-03T09:06:32.072 INFO:teuthology.orchestra.run.smithi088.stderr:last_changed 2024-06-03T08:34:31.885125+0000 2024-06-03T09:06:32.072 INFO:teuthology.orchestra.run.smithi088.stderr:created 2024-06-03T08:34:31.885125+0000 2024-06-03T09:06:32.072 INFO:teuthology.orchestra.run.smithi088.stderr:min_mon_release 18 (reef) 2024-06-03T09:06:32.072 INFO:teuthology.orchestra.run.smithi088.stderr:election_strategy: 3 2024-06-03T09:06:32.072 INFO:teuthology.orchestra.run.smithi088.stderr:0: v1:172.21.15.88:6789/0 mon.a 2024-06-03T09:06:32.072 INFO:teuthology.orchestra.run.smithi088.stderr:1: v1:172.21.15.192:6789/0 mon.b 2024-06-03T09:06:32.072 INFO:teuthology.orchestra.run.smithi088.stderr:2: v1:172.21.15.88:6790/0 mon.c 2024-06-03T09:06:32.072 INFO:teuthology.orchestra.run.smithi088.stderr: 2024-06-03T09:06:32.073 INFO:teuthology.orchestra.run.smithi088.stderr:2024-06-03T09:06:32.056+0000 7fd2c0ff9640 1 -- 172.21.15.88:0/1705515602 <== mon.0 v1:172.21.15.88:6789/0 4 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 580+0+0 (unknown 631772507 0 0) 0x7fd2c41d1f70 con 0x7fd2c406fc70 2024-06-03T09:06:32.073 INFO:teuthology.orchestra.run.smithi088.stderr:2024-06-03T09:06:32.056+0000 7fd2c0ff9640 1 -- 172.21.15.88:0/1705515602 >> v1:172.21.15.88:6790/0 conn(0x7fd2c413fa00 legacy=0x7fd2c41d7b90 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-06-03T09:06:32.073 INFO:teuthology.orchestra.run.smithi088.stderr:2024-06-03T09:06:32.056+0000 7fd2c0ff9640 1 -- 172.21.15.88:0/1705515602 >> v1:172.21.15.192:6789/0 conn(0x7fd2c4070ed0 legacy=0x7fd2c41d11a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-06-03T09:06:32.073 INFO:teuthology.orchestra.run.smithi088.stderr:2024-06-03T09:06:32.056+0000 7fd2c0ff9640 10 monclient: _finish_hunting 0 2024-06-03T09:06:32.073 INFO:teuthology.orchestra.run.smithi088.stderr:2024-06-03T09:06:32.056+0000 7fd2c0ff9640 1 monclient: found mon.a 2024-06-03T09:06:32.073 INFO:teuthology.orchestra.run.smithi088.stderr:2024-06-03T09:06:32.056+0000 7fd2c0ff9640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-06-03T09:06:32.074 INFO:teuthology.orchestra.run.smithi088.stderr:2024-06-03T09:06:32.056+0000 7fd2c0ff9640 10 monclient: _send_mon_message to mon.a at v1:172.21.15.88:6789/0 2024-06-03T09:06:32.074 INFO:teuthology.orchestra.run.smithi088.stderr:2024-06-03T09:06:32.056+0000 7fd2c0ff9640 1 -- 172.21.15.88:0/1705515602 --> v1:172.21.15.88:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fd2c40a8c20 con 0x7fd2c406fc70 2024-06-03T09:06:32.074 INFO:teuthology.orchestra.run.smithi088.stderr:2024-06-03T09:06:32.056+0000 7fd2c0ff9640 10 monclient: _finish_auth 0 2024-06-03T09:06:32.074 INFO:teuthology.orchestra.run.smithi088.stderr:2024-06-03T09:06:32.056+0000 7fd2c0ff9640 10 monclient: _check_auth_tickets 2024-06-03T09:06:32.074 INFO:teuthology.orchestra.run.smithi088.stderr:2024-06-03T09:06:32.056+0000 7fd2c0ff9640 20 monclient: _check_auth_rotating not needed by client.admin 2024-06-03T09:06:32.075 INFO:teuthology.orchestra.run.smithi088.stderr:2024-06-03T09:06:32.056+0000 7fd2ca380640 5 monclient: authenticate success, global_id 30633 2024-06-03T09:06:32.075 INFO:teuthology.orchestra.run.smithi088.stderr:2024-06-03T09:06:32.056+0000 7fd2ca380640 10 monclient: _renew_subs 2024-06-03T09:06:32.075 INFO:teuthology.orchestra.run.smithi088.stderr:2024-06-03T09:06:32.056+0000 7fd2ca380640 10 monclient: _send_mon_message to mon.a at v1:172.21.15.88:6789/0 2024-06-03T09:06:32.075 INFO:teuthology.orchestra.run.smithi088.stderr:2024-06-03T09:06:32.056+0000 7fd2ca380640 1 -- 172.21.15.88:0/1705515602 --> v1:172.21.15.88:6789/0 -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fd2c40b2af0 con 0x7fd2c406fc70 2024-06-03T09:06:32.075 INFO:teuthology.orchestra.run.smithi088.stderr:2024-06-03T09:06:32.056+0000 7fd2ca380640 10 client.30633.objecter _maybe_request_map subscribing (onetime) to next osd map 2024-06-03T09:06:32.075 INFO:teuthology.orchestra.run.smithi088.stderr:2024-06-03T09:06:32.056+0000 7fd2ca380640 10 monclient: _renew_subs 2024-06-03T09:06:32.076 INFO:teuthology.orchestra.run.smithi088.stderr:2024-06-03T09:06:32.056+0000 7fd2ca380640 10 monclient: _send_mon_message to mon.a at v1:172.21.15.88:6789/0 2024-06-03T09:06:32.076 INFO:teuthology.orchestra.run.smithi088.stderr:2024-06-03T09:06:32.056+0000 7fd2ca380640 1 -- 172.21.15.88:0/1705515602 --> v1:172.21.15.88:6789/0 -- mon_subscribe({osdmap=0}) v3 -- 0x7fd2c41d1a90 con 0x7fd2c406fc70 2024-06-03T09:06:32.076 INFO:teuthology.orchestra.run.smithi088.stderr:2024-06-03T09:06:32.056+0000 7fd2ca380640 1 librados: init done 2024-06-03T09:06:32.076 INFO:teuthology.orchestra.run.smithi088.stderr:2024-06-03T09:06:32.057+0000 7fd2ca380640 10 start_mon_command cmd=[{"prefix": "get_command_descriptions"}] 2024-06-03T09:06:32.076 INFO:teuthology.orchestra.run.smithi088.stderr:2024-06-03T09:06:32.057+0000 7fd2ca380640 10 monclient: _send_command 1 [{"prefix": "get_command_descriptions"}] 2024-06-03T09:06:32.076 INFO:teuthology.orchestra.run.smithi088.stderr:2024-06-03T09:06:32.057+0000 7fd2ca380640 10 monclient: _send_mon_message to mon.a at v1:172.21.15.88:6789/0 2024-06-03T09:06:32.077 INFO:teuthology.orchestra.run.smithi088.stderr:2024-06-03T09:06:32.057+0000 7fd2ca380640 1 -- 172.21.15.88:0/1705515602 --> v1:172.21.15.88:6789/0 -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fd2c404ea60 con 0x7fd2c406fc70 2024-06-03T09:06:32.305 INFO:teuthology.orchestra.run.smithi088.stderr:2024-06-03T09:06:32.305+0000 7fd2c0ff9640 1 -- 172.21.15.88:0/1705515602 <== mon.0 v1:172.21.15.88:6789/0 5 ==== config(0 keys) v1 ==== 4+0+0 (unknown 0 0 0) 0x7fd2b4004a90 con 0x7fd2c406fc70 2024-06-03T09:06:32.305 INFO:teuthology.orchestra.run.smithi088.stderr:2024-06-03T09:06:32.305+0000 7fd2c0ff9640 10 monclient: handle_config config(0 keys) v1 2024-06-03T09:06:32.306 INFO:teuthology.orchestra.run.smithi088.stderr:2024-06-03T09:06:32.305+0000 7fd2c0ff9640 1 -- 172.21.15.88:0/1705515602 <== mon.0 v1:172.21.15.88:6789/0 6 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 2234374065 0 0) 0x7fd2b4004da0 con 0x7fd2c406fc70 2024-06-03T09:06:32.306 INFO:teuthology.orchestra.run.smithi088.stderr:2024-06-03T09:06:32.305+0000 7fd2c0ff9640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-06-03T09:06:32.306 INFO:teuthology.orchestra.run.smithi088.stderr:2024-06-03T09:06:32.305+0000 7fd2c0ff9640 10 monclient: got monmap 1 from mon.a (according to old e1) 2024-06-03T09:06:32.306 INFO:teuthology.orchestra.run.smithi088.stderr:2024-06-03T09:06:32.305+0000 7fd2c0ff9640 10 monclient: dump: 2024-06-03T09:06:32.306 INFO:teuthology.orchestra.run.smithi088.stderr:epoch 1 2024-06-03T09:06:32.306 INFO:teuthology.orchestra.run.smithi088.stderr:fsid ec3c3fdd-3ac6-4d7d-9412-683130ed8b04 2024-06-03T09:06:32.306 INFO:teuthology.orchestra.run.smithi088.stderr:last_changed 2024-06-03T08:34:31.885125+0000 2024-06-03T09:06:32.306 INFO:teuthology.orchestra.run.smithi088.stderr:created 2024-06-03T08:34:31.885125+0000 2024-06-03T09:06:32.306 INFO:teuthology.orchestra.run.smithi088.stderr:min_mon_release 18 (reef) 2024-06-03T09:06:32.307 INFO:teuthology.orchestra.run.smithi088.stderr:election_strategy: 3 2024-06-03T09:06:32.307 INFO:teuthology.orchestra.run.smithi088.stderr:0: v1:172.21.15.88:6789/0 mon.a 2024-06-03T09:06:32.307 INFO:teuthology.orchestra.run.smithi088.stderr:1: v1:172.21.15.192:6789/0 mon.b 2024-06-03T09:06:32.307 INFO:teuthology.orchestra.run.smithi088.stderr:2: v1:172.21.15.88:6790/0 mon.c 2024-06-03T09:06:32.307 INFO:teuthology.orchestra.run.smithi088.stderr: 2024-06-03T09:06:32.307 INFO:teuthology.orchestra.run.smithi088.stderr:2024-06-03T09:06:32.305+0000 7fd2c27fc640 4 set_mon_vals no callback set 2024-06-03T09:06:32.308 INFO:teuthology.orchestra.run.smithi088.stderr:2024-06-03T09:06:32.306+0000 7fd2c0ff9640 1 -- 172.21.15.88:0/1705515602 <== mon.0 v1:172.21.15.88:6789/0 7 ==== mgrmap(e 5) v1 ==== 89110+0+0 (unknown 873533973 0 0) 0x7fd2b401ab20 con 0x7fd2c406fc70 2024-06-03T09:06:32.308 INFO:teuthology.orchestra.run.smithi088.stderr:2024-06-03T09:06:32.306+0000 7fd2c37fe640 10 monclient: get_auth_request con 0x7fd2b006d350 auth_method 0 2024-06-03T09:06:32.308 INFO:teuthology.orchestra.run.smithi088.stderr:2024-06-03T09:06:32.307+0000 7fd2c0ff9640 10 client.30633.objecter ms_handle_connect 0x7fd2b006d350 2024-06-03T09:06:32.311 INFO:teuthology.orchestra.run.smithi088.stderr:2024-06-03T09:06:32.311+0000 7fd2c0ff9640 1 -- 172.21.15.88:0/1705515602 <== mon.0 v1:172.21.15.88:6789/0 8 ==== osd_map(1010..1010 src has 622..1010) v4 ==== 5229+0+0 (unknown 1105883993 0 0) 0x7fd2b40541d0 con 0x7fd2c406fc70 2024-06-03T09:06:32.311 INFO:teuthology.orchestra.run.smithi088.stderr:2024-06-03T09:06:32.311+0000 7fd2c0ff9640 10 client.30633.objecter ms_dispatch 0x7fd2c4000b90 osd_map(1010..1010 src has 622..1010) v4 2024-06-03T09:06:32.311 INFO:teuthology.orchestra.run.smithi088.stderr:2024-06-03T09:06:32.311+0000 7fd2c0ff9640 3 client.30633.objecter handle_osd_map got epochs [1010,1010] > 0 2024-06-03T09:06:32.311 INFO:teuthology.orchestra.run.smithi088.stderr:2024-06-03T09:06:32.311+0000 7fd2c0ff9640 3 client.30633.objecter handle_osd_map decoding full epoch 1010 2024-06-03T09:06:32.311 INFO:teuthology.orchestra.run.smithi088.stderr:2024-06-03T09:06:32.311+0000 7fd2c0ff9640 20 client.30633.objecter dump_active .. 0 homeless 2024-06-03T09:06:32.312 INFO:teuthology.orchestra.run.smithi088.stderr:2024-06-03T09:06:32.311+0000 7fd2c0ff9640 1 -- 172.21.15.88:0/1705515602 <== mon.0 v1:172.21.15.88:6789/0 9 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+182498 (unknown 1092875540 0 3016760700) 0x7fd2b404fd80 con 0x7fd2c406fc70 2024-06-03T09:06:32.312 INFO:teuthology.orchestra.run.smithi088.stderr:2024-06-03T09:06:32.311+0000 7fd2c0ff9640 10 monclient: handle_mon_command_ack 1 [{"prefix": "get_command_descriptions"}] 2024-06-03T09:06:32.312 INFO:teuthology.orchestra.run.smithi088.stderr:2024-06-03T09:06:32.311+0000 7fd2c0ff9640 10 monclient: _finish_command 1 = system:0 2024-06-03T09:06:32.480 INFO:teuthology.orchestra.run.smithi088.stderr:2024-06-03T09:06:32.480+0000 7fd2ca380640 1 -- 172.21.15.88:0/1705515602 --> v1:172.21.15.192:6800/34229 -- mgr_command(tid 0: {"prefix": "pg dump", "target": ["mon-mgr", ""], "format": "json"}) v1 -- 0x7fd2c414c310 con 0x7fd2b006d350 2024-06-03T09:06:32.489 INFO:teuthology.orchestra.run.smithi088.stderr:2024-06-03T09:06:32.489+0000 7fd2c0ff9640 1 -- 172.21.15.88:0/1705515602 <== mgr.4110 v1:172.21.15.192:6800/34229 1 ==== mgr_command_reply(tid 0: 0 dumped all) v1 ==== 18+0+151509 (unknown 2965378022 0 1476458159) 0x7fd2b80077a0 con 0x7fd2b006d350 2024-06-03T09:06:32.489 INFO:teuthology.orchestra.run.smithi088.stdout: 2024-06-03T09:06:32.492 INFO:teuthology.orchestra.run.smithi088.stderr:dumped all 2024-06-03T09:06:32.494 INFO:teuthology.orchestra.run.smithi088.stderr:2024-06-03T09:06:32.491+0000 7fd2ca380640 10 librados: watch_flush enter 2024-06-03T09:06:32.494 INFO:teuthology.orchestra.run.smithi088.stderr:2024-06-03T09:06:32.491+0000 7fd2ca380640 10 librados: watch_flush exit 2024-06-03T09:06:32.494 INFO:teuthology.orchestra.run.smithi088.stderr:2024-06-03T09:06:32.491+0000 7fd2ca380640 20 client.30633.objecter shutdown clearing up homeless session... 2024-06-03T09:06:32.494 INFO:teuthology.orchestra.run.smithi088.stderr:2024-06-03T09:06:32.491+0000 7fd2ca380640 10 client.30633.objecter successfully canceled tick 2024-06-03T09:06:32.494 INFO:teuthology.orchestra.run.smithi088.stderr:2024-06-03T09:06:32.491+0000 7fd2ca380640 1 -- 172.21.15.88:0/1705515602 >> v1:172.21.15.192:6800/34229 conn(0x7fd2b006d350 legacy=0x7fd2b006f810 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-06-03T09:06:32.494 INFO:teuthology.orchestra.run.smithi088.stderr:2024-06-03T09:06:32.491+0000 7fd2ca380640 10 monclient: shutdown 2024-06-03T09:06:32.494 INFO:teuthology.orchestra.run.smithi088.stderr:2024-06-03T09:06:32.491+0000 7fd2ca380640 20 monclient: shutdown discarding 0 pending message(s) 2024-06-03T09:06:32.494 INFO:teuthology.orchestra.run.smithi088.stderr:2024-06-03T09:06:32.491+0000 7fd2ca380640 1 -- 172.21.15.88:0/1705515602 >> v1:172.21.15.88:6789/0 conn(0x7fd2c406fc70 legacy=0x7fd2c412fff0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-06-03T09:06:32.494 INFO:teuthology.orchestra.run.smithi088.stderr:2024-06-03T09:06:32.491+0000 7fd2ca380640 1 -- 172.21.15.88:0/1705515602 shutdown_connections 2024-06-03T09:06:32.495 INFO:teuthology.orchestra.run.smithi088.stderr:2024-06-03T09:06:32.491+0000 7fd2ca380640 1 -- 172.21.15.88:0/1705515602 >> 172.21.15.88:0/1705515602 conn(0x7fd2c40b2f20 msgr2=0x7fd2c40b03a0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-06-03T09:06:32.495 INFO:teuthology.orchestra.run.smithi088.stderr:2024-06-03T09:06:32.491+0000 7fd2ca380640 1 -- 172.21.15.88:0/1705515602 shutdown_connections 2024-06-03T09:06:32.495 INFO:teuthology.orchestra.run.smithi088.stderr:2024-06-03T09:06:32.491+0000 7fd2ca380640 1 -- 172.21.15.88:0/1705515602 wait complete. 2024-06-03T09:06:32.495 INFO:teuthology.orchestra.run.smithi088.stderr:2024-06-03T09:06:32.491+0000 7fd2ca380640 1 librados: shutdown 2024-06-03T09:06:32.502 INFO:teuthology.orchestra.run.smithi088.stdout:{"pg_ready":true,"pg_map":{"version":1921,"stamp":"2024-06-03T09:06:31.100641+0000","last_osdmap_epoch":0,"last_pg_scan":0,"pg_stats_sum":{"stat_sum":{"num_bytes":459285,"num_objects":3,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":1,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":3,"num_whiteouts":0,"num_read":102,"num_read_kb":88,"num_write":120,"num_write_kb":1379,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2,"num_bytes_recovered":397840,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":83,"ondisk_log_size":83,"up":68,"acting":68,"num_store_stats":0},"osd_stats_sum":{"up_from":0,"seq":0,"num_pgs":100,"num_osds":8,"num_per_pool_osds":8,"num_per_pool_omap_osds":8,"kb":754974720,"kb_used":494548,"kb_used_data":46324,"kb_used_omap":12,"kb_used_meta":448115,"kb_avail":754480172,"statfs":{"total":773094113280,"available":772587696128,"internally_reserved":0,"allocated":47435776,"data_stored":22747954,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":12744,"internal_metadata":458870328},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":109,"apply_latency_ms":109,"commit_latency_ns":109000000,"apply_latency_ns":109000000},"alerts":[],"network_ping_times":[]},"pg_stats_delta":{"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":0,"acting":0,"num_store_stats":0,"stamp_delta":"12.002291"},"pg_stats":[{"pgid":"17.d","version":"0'0","reported_seq":942,"reported_epoch":1009,"state":"active+clean","last_fresh":"2024-06-03T09:06:14.443917+0000","last_change":"2024-06-03T08:38:19.260548+0000","last_active":"2024-06-03T09:06:14.443917+0000","last_peered":"2024-06-03T09:06:14.443917+0000","last_clean":"2024-06-03T09:06:14.443917+0000","last_became_active":"2024-06-03T08:38:19.253559+0000","last_became_peered":"2024-06-03T08:38:19.253559+0000","last_unstale":"2024-06-03T09:06:14.443917+0000","last_undegraded":"2024-06-03T09:06:14.443917+0000","last_fullsized":"2024-06-03T09:06:14.443917+0000","mapping_epoch":28,"log_start":"0'0","ondisk_log_start":"0'0","created":28,"last_epoch_clean":45,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-06-03T08:37:55.614497+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-06-03T08:37:55.614497+0000","last_clean_scrub_stamp":"2024-06-03T08:37:55.614497+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-06-04T19:00:12.470979+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1],"acting":[0,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"17.c","version":"0'0","reported_seq":953,"reported_epoch":1009,"state":"active+undersized","last_fresh":"2024-06-03T09:06:13.668156+0000","last_change":"2024-06-03T09:06:03.440074+0000","last_active":"2024-06-03T09:06:13.668156+0000","last_peered":"2024-06-03T09:06:13.668156+0000","last_clean":"2024-06-03T09:05:02.105638+0000","last_became_active":"2024-06-03T09:06:03.440074+0000","last_became_peered":"2024-06-03T09:06:03.440074+0000","last_unstale":"2024-06-03T09:06:13.668156+0000","last_undegraded":"2024-06-03T09:06:13.668156+0000","last_fullsized":"2024-06-03T09:06:03.377982+0000","mapping_epoch":999,"log_start":"0'0","ondisk_log_start":"0'0","created":28,"last_epoch_clean":255,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-06-03T08:37:55.614497+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-06-03T08:37:55.614497+0000","last_clean_scrub_stamp":"2024-06-03T08:37:55.614497+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-06-04T10:34:20.808557+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3],"acting":[3],"avail_no_missing":["3"],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"17.f","version":"0'0","reported_seq":148,"reported_epoch":1009,"state":"active+undersized","last_fresh":"2024-06-03T09:06:14.450543+0000","last_change":"2024-06-03T09:06:03.420285+0000","last_active":"2024-06-03T09:06:14.450543+0000","last_peered":"2024-06-03T09:06:14.450543+0000","last_clean":"2024-06-03T08:37:55.614497+0000","last_became_active":"2024-06-03T09:06:03.420285+0000","last_became_peered":"2024-06-03T09:06:03.420285+0000","last_unstale":"2024-06-03T09:06:14.450543+0000","last_undegraded":"2024-06-03T09:06:14.450543+0000","last_fullsized":"2024-06-03T09:06:03.380616+0000","mapping_epoch":999,"log_start":"0'0","ondisk_log_start":"0'0","created":28,"last_epoch_clean":255,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-06-03T08:37:55.614497+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-06-03T08:37:55.614497+0000","last_clean_scrub_stamp":"2024-06-03T08:37:55.614497+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-06-04T15:39:41.187731+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4],"acting":[4],"avail_no_missing":["4"],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"17.e","version":"0'0","reported_seq":149,"reported_epoch":1009,"state":"active+undersized","last_fresh":"2024-06-03T09:06:13.726992+0000","last_change":"2024-06-03T09:06:03.412887+0000","last_active":"2024-06-03T09:06:13.726992+0000","last_peered":"2024-06-03T09:06:13.726992+0000","last_clean":"2024-06-03T08:37:55.614497+0000","last_became_active":"2024-06-03T09:06:03.412887+0000","last_became_peered":"2024-06-03T09:06:03.412887+0000","last_unstale":"2024-06-03T09:06:13.726992+0000","last_undegraded":"2024-06-03T09:06:13.726992+0000","last_fullsized":"2024-06-03T09:06:03.332482+0000","mapping_epoch":999,"log_start":"0'0","ondisk_log_start":"0'0","created":28,"last_epoch_clean":255,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-06-03T08:37:55.614497+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-06-03T08:37:55.614497+0000","last_clean_scrub_stamp":"2024-06-03T08:37:55.614497+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-06-04T12:23:22.511601+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5],"acting":[5],"avail_no_missing":["5"],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"17.9","version":"0'0","reported_seq":946,"reported_epoch":1009,"state":"active+clean","last_fresh":"2024-06-03T09:06:14.999299+0000","last_change":"2024-06-03T08:38:04.772260+0000","last_active":"2024-06-03T09:06:14.999299+0000","last_peered":"2024-06-03T09:06:14.999299+0000","last_clean":"2024-06-03T09:06:14.999299+0000","last_became_active":"2024-06-03T08:38:04.763428+0000","last_became_peered":"2024-06-03T08:38:04.763428+0000","last_unstale":"2024-06-03T09:06:14.999299+0000","last_undegraded":"2024-06-03T09:06:14.999299+0000","last_fullsized":"2024-06-03T09:06:14.999299+0000","mapping_epoch":28,"log_start":"0'0","ondisk_log_start":"0'0","created":28,"last_epoch_clean":33,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-06-03T08:37:55.614497+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-06-03T08:37:55.614497+0000","last_clean_scrub_stamp":"2024-06-03T08:37:55.614497+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-06-04T10:55:10.773936+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,4],"acting":[7,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"17.8","version":"0'0","reported_seq":961,"reported_epoch":1009,"state":"active+undersized","last_fresh":"2024-06-03T09:06:14.430375+0000","last_change":"2024-06-03T09:06:03.388786+0000","last_active":"2024-06-03T09:06:14.430375+0000","last_peered":"2024-06-03T09:06:14.430375+0000","last_clean":"2024-06-03T09:04:43.297780+0000","last_became_active":"2024-06-03T09:06:03.388786+0000","last_became_peered":"2024-06-03T09:06:03.388786+0000","last_unstale":"2024-06-03T09:06:14.430375+0000","last_undegraded":"2024-06-03T09:06:14.430375+0000","last_fullsized":"2024-06-03T09:06:03.331754+0000","mapping_epoch":999,"log_start":"0'0","ondisk_log_start":"0'0","created":28,"last_epoch_clean":255,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-06-03T08:37:55.614497+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-06-03T08:37:55.614497+0000","last_clean_scrub_stamp":"2024-06-03T08:37:55.614497+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-06-04T09:18:44.765547+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0],"acting":[0],"avail_no_missing":["0"],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"17.b","version":"0'0","reported_seq":149,"reported_epoch":1009,"state":"active+undersized","last_fresh":"2024-06-03T09:06:14.433735+0000","last_change":"2024-06-03T09:06:03.451203+0000","last_active":"2024-06-03T09:06:14.433735+0000","last_peered":"2024-06-03T09:06:14.433735+0000","last_clean":"2024-06-03T08:37:55.614497+0000","last_became_active":"2024-06-03T09:06:03.451203+0000","last_became_peered":"2024-06-03T09:06:03.451203+0000","last_unstale":"2024-06-03T09:06:14.433735+0000","last_undegraded":"2024-06-03T09:06:14.433735+0000","last_fullsized":"2024-06-03T09:06:03.375523+0000","mapping_epoch":999,"log_start":"0'0","ondisk_log_start":"0'0","created":28,"last_epoch_clean":255,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-06-03T08:37:55.614497+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-06-03T08:37:55.614497+0000","last_clean_scrub_stamp":"2024-06-03T08:37:55.614497+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-06-04T10:16:19.813495+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2],"acting":[2],"avail_no_missing":["2"],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"17.a","version":"0'0","reported_seq":925,"reported_epoch":1010,"state":"active+clean","last_fresh":"2024-06-03T09:06:18.651697+0000","last_change":"2024-06-03T08:38:05.843142+0000","last_active":"2024-06-03T09:06:18.651697+0000","last_peered":"2024-06-03T09:06:18.651697+0000","last_clean":"2024-06-03T09:06:18.651697+0000","last_became_active":"2024-06-03T08:38:05.809257+0000","last_became_peered":"2024-06-03T08:38:05.809257+0000","last_unstale":"2024-06-03T09:06:18.651697+0000","last_undegraded":"2024-06-03T09:06:18.651697+0000","last_fullsized":"2024-06-03T09:06:18.651697+0000","mapping_epoch":28,"log_start":"0'0","ondisk_log_start":"0'0","created":28,"last_epoch_clean":33,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-06-03T08:37:55.614497+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-06-03T08:37:55.614497+0000","last_clean_scrub_stamp":"2024-06-03T08:37:55.614497+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-06-04T12:12:03.686838+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2],"acting":[1,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"17.5","version":"0'0","reported_seq":943,"reported_epoch":1009,"state":"active+clean","last_fresh":"2024-06-03T09:06:14.442357+0000","last_change":"2024-06-03T08:38:23.466051+0000","last_active":"2024-06-03T09:06:14.442357+0000","last_peered":"2024-06-03T09:06:14.442357+0000","last_clean":"2024-06-03T09:06:14.442357+0000","last_became_active":"2024-06-03T08:38:23.458039+0000","last_became_peered":"2024-06-03T08:38:23.458039+0000","last_unstale":"2024-06-03T09:06:14.442357+0000","last_undegraded":"2024-06-03T09:06:14.442357+0000","last_fullsized":"2024-06-03T09:06:14.442357+0000","mapping_epoch":28,"log_start":"0'0","ondisk_log_start":"0'0","created":28,"last_epoch_clean":49,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-06-03T08:37:55.614497+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-06-03T08:37:55.614497+0000","last_clean_scrub_stamp":"2024-06-03T08:37:55.614497+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-06-04T12:27:02.532436+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,4],"acting":[0,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"17.4","version":"0'0","reported_seq":149,"reported_epoch":1009,"state":"active+undersized","last_fresh":"2024-06-03T09:06:13.941522+0000","last_change":"2024-06-03T09:06:03.441889+0000","last_active":"2024-06-03T09:06:13.941522+0000","last_peered":"2024-06-03T09:06:13.941522+0000","last_clean":"2024-06-03T08:37:55.614497+0000","last_became_active":"2024-06-03T09:06:03.441889+0000","last_became_peered":"2024-06-03T09:06:03.441889+0000","last_unstale":"2024-06-03T09:06:13.941522+0000","last_undegraded":"2024-06-03T09:06:13.941522+0000","last_fullsized":"2024-06-03T09:06:03.359582+0000","mapping_epoch":999,"log_start":"0'0","ondisk_log_start":"0'0","created":28,"last_epoch_clean":255,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-06-03T08:37:55.614497+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-06-03T08:37:55.614497+0000","last_clean_scrub_stamp":"2024-06-03T08:37:55.614497+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-06-04T12:49:22.907141+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7],"acting":[7],"avail_no_missing":["7"],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"17.7","version":"0'0","reported_seq":929,"reported_epoch":1009,"state":"active+clean","last_fresh":"2024-06-03T09:06:13.736920+0000","last_change":"2024-06-03T08:38:04.837083+0000","last_active":"2024-06-03T09:06:13.736920+0000","last_peered":"2024-06-03T09:06:13.736920+0000","last_clean":"2024-06-03T09:06:13.736920+0000","last_became_active":"2024-06-03T08:38:04.818246+0000","last_became_peered":"2024-06-03T08:38:04.818246+0000","last_unstale":"2024-06-03T09:06:13.736920+0000","last_undegraded":"2024-06-03T09:06:13.736920+0000","last_fullsized":"2024-06-03T09:06:13.736920+0000","mapping_epoch":28,"log_start":"0'0","ondisk_log_start":"0'0","created":28,"last_epoch_clean":34,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-06-03T08:37:55.614497+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-06-03T08:37:55.614497+0000","last_clean_scrub_stamp":"2024-06-03T08:37:55.614497+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-06-04T17:02:38.204813+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,2],"acting":[5,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"17.6","version":"0'0","reported_seq":960,"reported_epoch":1009,"state":"active+undersized","last_fresh":"2024-06-03T09:06:14.453110+0000","last_change":"2024-06-03T09:06:03.428677+0000","last_active":"2024-06-03T09:06:14.453110+0000","last_peered":"2024-06-03T09:06:14.453110+0000","last_clean":"2024-06-03T09:04:43.312352+0000","last_became_active":"2024-06-03T09:06:03.428677+0000","last_became_peered":"2024-06-03T09:06:03.428677+0000","last_unstale":"2024-06-03T09:06:14.453110+0000","last_undegraded":"2024-06-03T09:06:14.453110+0000","last_fullsized":"2024-06-03T09:06:03.359909+0000","mapping_epoch":999,"log_start":"0'0","ondisk_log_start":"0'0","created":28,"last_epoch_clean":255,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-06-03T08:37:55.614497+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-06-03T08:37:55.614497+0000","last_clean_scrub_stamp":"2024-06-03T08:37:55.614497+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-06-04T15:38:02.180241+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0],"acting":[0],"avail_no_missing":["0"],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"2.7","version":"0'0","reported_seq":940,"reported_epoch":1009,"state":"active+clean","last_fresh":"2024-06-03T09:06:13.684065+0000","last_change":"2024-06-03T08:36:30.385405+0000","last_active":"2024-06-03T09:06:13.684065+0000","last_peered":"2024-06-03T09:06:13.684065+0000","last_clean":"2024-06-03T09:06:13.684065+0000","last_became_active":"2024-06-03T08:36:30.380302+0000","last_became_peered":"2024-06-03T08:36:30.380302+0000","last_unstale":"2024-06-03T09:06:13.684065+0000","last_undegraded":"2024-06-03T09:06:13.684065+0000","last_fullsized":"2024-06-03T09:06:13.684065+0000","mapping_epoch":23,"log_start":"0'0","ondisk_log_start":"0'0","created":23,"last_epoch_clean":24,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-06-03T08:36:28.444764+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-06-03T08:36:28.444764+0000","last_clean_scrub_stamp":"2024-06-03T08:36:28.444764+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-06-04T18:46:33.475181+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0],"acting":[3,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"17.14","version":"0'0","reported_seq":926,"reported_epoch":1009,"state":"active+clean","last_fresh":"2024-06-03T09:06:13.721460+0000","last_change":"2024-06-03T08:38:04.547565+0000","last_active":"2024-06-03T09:06:13.721460+0000","last_peered":"2024-06-03T09:06:13.721460+0000","last_clean":"2024-06-03T09:06:13.721460+0000","last_became_active":"2024-06-03T08:38:04.541781+0000","last_became_peered":"2024-06-03T08:38:04.541781+0000","last_unstale":"2024-06-03T09:06:13.721460+0000","last_undegraded":"2024-06-03T09:06:13.721460+0000","last_fullsized":"2024-06-03T09:06:13.721460+0000","mapping_epoch":28,"log_start":"0'0","ondisk_log_start":"0'0","created":28,"last_epoch_clean":34,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-06-03T08:37:55.614497+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-06-03T08:37:55.614497+0000","last_clean_scrub_stamp":"2024-06-03T08:37:55.614497+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-06-04T16:41:55.170134+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,2],"acting":[5,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"2.6","version":"0'0","reported_seq":949,"reported_epoch":1009,"state":"active+clean","last_fresh":"2024-06-03T09:06:14.448100+0000","last_change":"2024-06-03T08:36:30.430760+0000","last_active":"2024-06-03T09:06:14.448100+0000","last_peered":"2024-06-03T09:06:14.448100+0000","last_clean":"2024-06-03T09:06:14.448100+0000","last_became_active":"2024-06-03T08:36:30.404665+0000","last_became_peered":"2024-06-03T08:36:30.404665+0000","last_unstale":"2024-06-03T09:06:14.448100+0000","last_undegraded":"2024-06-03T09:06:14.448100+0000","last_fullsized":"2024-06-03T09:06:14.448100+0000","mapping_epoch":23,"log_start":"0'0","ondisk_log_start":"0'0","created":23,"last_epoch_clean":24,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-06-03T08:36:28.444764+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-06-03T08:36:28.444764+0000","last_clean_scrub_stamp":"2024-06-03T08:36:28.444764+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-06-04T10:37:37.746074+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,0],"acting":[4,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"17.15","version":"0'0","reported_seq":934,"reported_epoch":1009,"state":"active+clean","last_fresh":"2024-06-03T09:06:13.672940+0000","last_change":"2024-06-03T08:38:20.999903+0000","last_active":"2024-06-03T09:06:13.672940+0000","last_peered":"2024-06-03T09:06:13.672940+0000","last_clean":"2024-06-03T09:06:13.672940+0000","last_became_active":"2024-06-03T08:38:20.979523+0000","last_became_peered":"2024-06-03T08:38:20.979523+0000","last_unstale":"2024-06-03T09:06:13.672940+0000","last_undegraded":"2024-06-03T09:06:13.672940+0000","last_fullsized":"2024-06-03T09:06:13.672940+0000","mapping_epoch":28,"log_start":"0'0","ondisk_log_start":"0'0","created":28,"last_epoch_clean":47,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-06-03T08:37:55.614497+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-06-03T08:37:55.614497+0000","last_clean_scrub_stamp":"2024-06-03T08:37:55.614497+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-06-04T20:29:58.314289+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,1],"acting":[3,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"2.5","version":"0'0","reported_seq":942,"reported_epoch":1010,"state":"active+clean","last_fresh":"2024-06-03T09:06:18.382767+0000","last_change":"2024-06-03T08:36:30.024235+0000","last_active":"2024-06-03T09:06:18.382767+0000","last_peered":"2024-06-03T09:06:18.382767+0000","last_clean":"2024-06-03T09:06:18.382767+0000","last_became_active":"2024-06-03T08:36:30.019040+0000","last_became_peered":"2024-06-03T08:36:30.019040+0000","last_unstale":"2024-06-03T09:06:18.382767+0000","last_undegraded":"2024-06-03T09:06:18.382767+0000","last_fullsized":"2024-06-03T09:06:18.382767+0000","mapping_epoch":23,"log_start":"0'0","ondisk_log_start":"0'0","created":23,"last_epoch_clean":24,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-06-03T08:36:28.444764+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-06-03T08:36:28.444764+0000","last_clean_scrub_stamp":"2024-06-03T08:36:28.444764+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-06-04T15:46:47.109502+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,4],"acting":[3,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"17.16","version":"0'0","reported_seq":972,"reported_epoch":1009,"state":"active+undersized","last_fresh":"2024-06-03T09:06:14.441665+0000","last_change":"2024-06-03T09:06:03.478615+0000","last_active":"2024-06-03T09:06:14.441665+0000","last_peered":"2024-06-03T09:06:14.441665+0000","last_clean":"2024-06-03T09:04:43.328415+0000","last_became_active":"2024-06-03T09:06:03.478615+0000","last_became_peered":"2024-06-03T09:06:03.478615+0000","last_unstale":"2024-06-03T09:06:14.441665+0000","last_undegraded":"2024-06-03T09:06:14.441665+0000","last_fullsized":"2024-06-03T09:06:03.402504+0000","mapping_epoch":999,"log_start":"0'0","ondisk_log_start":"0'0","created":28,"last_epoch_clean":255,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-06-03T08:37:55.614497+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-06-03T08:37:55.614497+0000","last_clean_scrub_stamp":"2024-06-03T08:37:55.614497+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-06-04T08:47:08.079233+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2],"acting":[2],"avail_no_missing":["2"],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"2.4","version":"0'0","reported_seq":952,"reported_epoch":1010,"state":"active+undersized","last_fresh":"2024-06-03T09:06:18.331485+0000","last_change":"2024-06-03T09:06:03.387171+0000","last_active":"2024-06-03T09:06:18.331485+0000","last_peered":"2024-06-03T09:06:18.331485+0000","last_clean":"2024-06-03T09:04:43.202645+0000","last_became_active":"2024-06-03T09:06:03.387171+0000","last_became_peered":"2024-06-03T09:06:03.387171+0000","last_unstale":"2024-06-03T09:06:18.331485+0000","last_undegraded":"2024-06-03T09:06:18.331485+0000","last_fullsized":"2024-06-03T09:06:03.283787+0000","mapping_epoch":999,"log_start":"0'0","ondisk_log_start":"0'0","created":23,"last_epoch_clean":255,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-06-03T08:36:28.444764+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-06-03T08:36:28.444764+0000","last_clean_scrub_stamp":"2024-06-03T08:36:28.444764+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-06-04T20:17:43.485023+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5],"acting":[5],"avail_no_missing":["5"],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"17.17","version":"0'0","reported_seq":954,"reported_epoch":1009,"state":"active+clean","last_fresh":"2024-06-03T09:06:14.447046+0000","last_change":"2024-06-03T08:38:38.483275+0000","last_active":"2024-06-03T09:06:14.447046+0000","last_peered":"2024-06-03T09:06:14.447046+0000","last_clean":"2024-06-03T09:06:14.447046+0000","last_became_active":"2024-06-03T08:38:38.467916+0000","last_became_peered":"2024-06-03T08:38:38.467916+0000","last_unstale":"2024-06-03T09:06:14.447046+0000","last_undegraded":"2024-06-03T09:06:14.447046+0000","last_fullsized":"2024-06-03T09:06:14.447046+0000","mapping_epoch":28,"log_start":"0'0","ondisk_log_start":"0'0","created":28,"last_epoch_clean":61,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-06-03T08:37:55.614497+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-06-03T08:37:55.614497+0000","last_clean_scrub_stamp":"2024-06-03T08:37:55.614497+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-06-04T10:02:23.036420+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,7],"acting":[2,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"2.2","version":"0'0","reported_seq":963,"reported_epoch":1009,"state":"active+undersized","last_fresh":"2024-06-03T09:06:13.657892+0000","last_change":"2024-06-03T09:06:03.410020+0000","last_active":"2024-06-03T09:06:13.657892+0000","last_peered":"2024-06-03T09:06:13.657892+0000","last_clean":"2024-06-03T09:05:37.145889+0000","last_became_active":"2024-06-03T09:06:03.410020+0000","last_became_peered":"2024-06-03T09:06:03.410020+0000","last_unstale":"2024-06-03T09:06:13.657892+0000","last_undegraded":"2024-06-03T09:06:13.657892+0000","last_fullsized":"2024-06-03T09:06:03.346864+0000","mapping_epoch":999,"log_start":"0'0","ondisk_log_start":"0'0","created":23,"last_epoch_clean":255,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-06-03T08:36:28.444764+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-06-03T08:36:28.444764+0000","last_clean_scrub_stamp":"2024-06-03T08:36:28.444764+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-06-04T09:57:12.423360+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3],"acting":[3],"avail_no_missing":["3"],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"17.11","version":"0'0","reported_seq":944,"reported_epoch":1009,"state":"active+clean","last_fresh":"2024-06-03T09:06:14.426670+0000","last_change":"2024-06-03T08:38:06.502302+0000","last_active":"2024-06-03T09:06:14.426670+0000","last_peered":"2024-06-03T09:06:14.426670+0000","last_clean":"2024-06-03T09:06:14.426670+0000","last_became_active":"2024-06-03T08:38:06.490204+0000","last_became_peered":"2024-06-03T08:38:06.490204+0000","last_unstale":"2024-06-03T09:06:14.426670+0000","last_undegraded":"2024-06-03T09:06:14.426670+0000","last_fullsized":"2024-06-03T09:06:14.426670+0000","mapping_epoch":28,"log_start":"0'0","ondisk_log_start":"0'0","created":28,"last_epoch_clean":32,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-06-03T08:37:55.614497+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-06-03T08:37:55.614497+0000","last_clean_scrub_stamp":"2024-06-03T08:37:55.614497+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-06-04T10:47:05.988958+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,2],"acting":[4,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"2.1","version":"0'0","reported_seq":952,"reported_epoch":1009,"state":"active+clean","last_fresh":"2024-06-03T09:06:13.929725+0000","last_change":"2024-06-03T08:36:30.883360+0000","last_active":"2024-06-03T09:06:13.929725+0000","last_peered":"2024-06-03T09:06:13.929725+0000","last_clean":"2024-06-03T09:06:13.929725+0000","last_became_active":"2024-06-03T08:36:30.855073+0000","last_became_peered":"2024-06-03T08:36:30.855073+0000","last_unstale":"2024-06-03T09:06:13.929725+0000","last_undegraded":"2024-06-03T09:06:13.929725+0000","last_fullsized":"2024-06-03T09:06:13.929725+0000","mapping_epoch":23,"log_start":"0'0","ondisk_log_start":"0'0","created":23,"last_epoch_clean":25,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-06-03T08:36:28.444764+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-06-03T08:36:28.444764+0000","last_clean_scrub_stamp":"2024-06-03T08:36:28.444764+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-06-04T18:12:22.363473+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,5],"acting":[7,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"17.12","version":"0'0","reported_seq":945,"reported_epoch":1009,"state":"active+clean","last_fresh":"2024-06-03T09:06:13.935141+0000","last_change":"2024-06-03T08:38:01.751562+0000","last_active":"2024-06-03T09:06:13.935141+0000","last_peered":"2024-06-03T09:06:13.935141+0000","last_clean":"2024-06-03T09:06:13.935141+0000","last_became_active":"2024-06-03T08:38:01.745027+0000","last_became_peered":"2024-06-03T08:38:01.745027+0000","last_unstale":"2024-06-03T09:06:13.935141+0000","last_undegraded":"2024-06-03T09:06:13.935141+0000","last_fullsized":"2024-06-03T09:06:13.935141+0000","mapping_epoch":28,"log_start":"0'0","ondisk_log_start":"0'0","created":28,"last_epoch_clean":32,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-06-03T08:37:55.614497+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-06-03T08:37:55.614497+0000","last_clean_scrub_stamp":"2024-06-03T08:37:55.614497+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-06-04T17:54:43.785326+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,3],"acting":[7,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"2.0","version":"0'0","reported_seq":943,"reported_epoch":1009,"state":"active+clean","last_fresh":"2024-06-03T09:06:13.648112+0000","last_change":"2024-06-03T08:36:29.984096+0000","last_active":"2024-06-03T09:06:13.648112+0000","last_peered":"2024-06-03T09:06:13.648112+0000","last_clean":"2024-06-03T09:06:13.648112+0000","last_became_active":"2024-06-03T08:36:29.957875+0000","last_became_peered":"2024-06-03T08:36:29.957875+0000","last_unstale":"2024-06-03T09:06:13.648112+0000","last_undegraded":"2024-06-03T09:06:13.648112+0000","last_fullsized":"2024-06-03T09:06:13.648112+0000","mapping_epoch":23,"log_start":"0'0","ondisk_log_start":"0'0","created":23,"last_epoch_clean":24,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-06-03T08:36:28.444764+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-06-03T08:36:28.444764+0000","last_clean_scrub_stamp":"2024-06-03T08:36:28.444764+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-06-04T08:41:33.679479+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,5],"acting":[3,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"17.13","version":"0'0","reported_seq":943,"reported_epoch":1009,"state":"active+clean","last_fresh":"2024-06-03T09:06:14.438399+0000","last_change":"2024-06-03T08:38:26.838316+0000","last_active":"2024-06-03T09:06:14.438399+0000","last_peered":"2024-06-03T09:06:14.438399+0000","last_clean":"2024-06-03T09:06:14.438399+0000","last_became_active":"2024-06-03T08:38:26.833214+0000","last_became_peered":"2024-06-03T08:38:26.833214+0000","last_unstale":"2024-06-03T09:06:14.438399+0000","last_undegraded":"2024-06-03T09:06:14.438399+0000","last_fullsized":"2024-06-03T09:06:14.438399+0000","mapping_epoch":28,"log_start":"0'0","ondisk_log_start":"0'0","created":28,"last_epoch_clean":52,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-06-03T08:37:55.614497+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-06-03T08:37:55.614497+0000","last_clean_scrub_stamp":"2024-06-03T08:37:55.614497+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-06-04T12:58:51.617998+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,0],"acting":[4,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"2.3","version":"0'0","reported_seq":943,"reported_epoch":1010,"state":"active+clean","last_fresh":"2024-06-03T09:06:18.362614+0000","last_change":"2024-06-03T08:36:30.351670+0000","last_active":"2024-06-03T09:06:18.362614+0000","last_peered":"2024-06-03T09:06:18.362614+0000","last_clean":"2024-06-03T09:06:18.362614+0000","last_became_active":"2024-06-03T08:36:30.343560+0000","last_became_peered":"2024-06-03T08:36:30.343560+0000","last_unstale":"2024-06-03T09:06:18.362614+0000","last_undegraded":"2024-06-03T09:06:18.362614+0000","last_fullsized":"2024-06-03T09:06:18.362614+0000","mapping_epoch":23,"log_start":"0'0","ondisk_log_start":"0'0","created":23,"last_epoch_clean":24,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-06-03T08:36:28.444764+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-06-03T08:36:28.444764+0000","last_clean_scrub_stamp":"2024-06-03T08:36:28.444764+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-06-04T12:02:24.765135+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,2],"acting":[3,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.0","version":"19'76","reported_seq":1051,"reported_epoch":1009,"state":"active+clean","last_fresh":"2024-06-03T09:06:13.703685+0000","last_change":"2024-06-03T08:36:26.702702+0000","last_active":"2024-06-03T09:06:13.703685+0000","last_peered":"2024-06-03T09:06:13.703685+0000","last_clean":"2024-06-03T09:06:13.703685+0000","last_became_active":"2024-06-03T08:36:26.128378+0000","last_became_peered":"2024-06-03T08:36:26.128378+0000","last_unstale":"2024-06-03T09:06:13.703685+0000","last_undegraded":"2024-06-03T09:06:13.703685+0000","last_fullsized":"2024-06-03T09:06:13.703685+0000","mapping_epoch":20,"log_start":"0'0","ondisk_log_start":"0'0","created":16,"last_epoch_clean":22,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-06-03T08:36:15.542635+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-06-03T08:36:15.542635+0000","last_clean_scrub_stamp":"2024-06-03T08:36:15.542635+0000","objects_scrubbed":0,"log_size":76,"log_dups_size":0,"ondisk_log_size":76,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-06-04T12:41:53.731360+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":459280,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":96,"num_read_kb":82,"num_write":113,"num_write_kb":1372,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2,"num_bytes_recovered":397840,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,5],"acting":[1,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"17.10","version":"0'0","reported_seq":944,"reported_epoch":1009,"state":"active+clean","last_fresh":"2024-06-03T09:06:14.422035+0000","last_change":"2024-06-03T08:38:15.399189+0000","last_active":"2024-06-03T09:06:14.422035+0000","last_peered":"2024-06-03T09:06:14.422035+0000","last_clean":"2024-06-03T09:06:14.422035+0000","last_became_active":"2024-06-03T08:38:15.337723+0000","last_became_peered":"2024-06-03T08:38:15.337723+0000","last_unstale":"2024-06-03T09:06:14.422035+0000","last_undegraded":"2024-06-03T09:06:14.422035+0000","last_fullsized":"2024-06-03T09:06:14.422035+0000","mapping_epoch":28,"log_start":"0'0","ondisk_log_start":"0'0","created":28,"last_epoch_clean":42,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-06-03T08:37:55.614497+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-06-03T08:37:55.614497+0000","last_clean_scrub_stamp":"2024-06-03T08:37:55.614497+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-06-04T09:29:46.427317+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,3],"acting":[4,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"17.1a","version":"0'0","reported_seq":960,"reported_epoch":1009,"state":"active+undersized","last_fresh":"2024-06-03T09:06:14.431224+0000","last_change":"2024-06-03T09:06:03.390734+0000","last_active":"2024-06-03T09:06:14.431224+0000","last_peered":"2024-06-03T09:06:14.431224+0000","last_clean":"2024-06-03T09:05:20.322313+0000","last_became_active":"2024-06-03T09:06:03.390734+0000","last_became_peered":"2024-06-03T09:06:03.390734+0000","last_unstale":"2024-06-03T09:06:14.431224+0000","last_undegraded":"2024-06-03T09:06:14.431224+0000","last_fullsized":"2024-06-03T09:06:03.343197+0000","mapping_epoch":999,"log_start":"0'0","ondisk_log_start":"0'0","created":28,"last_epoch_clean":254,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-06-03T08:37:55.614497+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-06-03T08:37:55.614497+0000","last_clean_scrub_stamp":"2024-06-03T08:37:55.614497+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-06-04T15:56:47.628875+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4],"acting":[4],"avail_no_missing":["4"],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"17.1b","version":"0'0","reported_seq":935,"reported_epoch":1009,"state":"active+clean","last_fresh":"2024-06-03T09:06:13.663279+0000","last_change":"2024-06-03T08:38:03.115748+0000","last_active":"2024-06-03T09:06:13.663279+0000","last_peered":"2024-06-03T09:06:13.663279+0000","last_clean":"2024-06-03T09:06:13.663279+0000","last_became_active":"2024-06-03T08:38:03.110533+0000","last_became_peered":"2024-06-03T08:38:03.110533+0000","last_unstale":"2024-06-03T09:06:13.663279+0000","last_undegraded":"2024-06-03T09:06:13.663279+0000","last_fullsized":"2024-06-03T09:06:13.663279+0000","mapping_epoch":28,"log_start":"0'0","ondisk_log_start":"0'0","created":28,"last_epoch_clean":32,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-06-03T08:37:55.614497+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-06-03T08:37:55.614497+0000","last_clean_scrub_stamp":"2024-06-03T08:37:55.614497+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-06-04T12:36:47.471609+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,7],"acting":[3,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"17.18","version":"0'0","reported_seq":944,"reported_epoch":1009,"state":"active+clean","last_fresh":"2024-06-03T09:06:13.924799+0000","last_change":"2024-06-03T08:38:04.632864+0000","last_active":"2024-06-03T09:06:13.924799+0000","last_peered":"2024-06-03T09:06:13.924799+0000","last_clean":"2024-06-03T09:06:13.924799+0000","last_became_active":"2024-06-03T08:38:04.627856+0000","last_became_peered":"2024-06-03T08:38:04.627856+0000","last_unstale":"2024-06-03T09:06:13.924799+0000","last_undegraded":"2024-06-03T09:06:13.924799+0000","last_fullsized":"2024-06-03T09:06:13.924799+0000","mapping_epoch":28,"log_start":"0'0","ondisk_log_start":"0'0","created":28,"last_epoch_clean":33,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-06-03T08:37:55.614497+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-06-03T08:37:55.614497+0000","last_clean_scrub_stamp":"2024-06-03T08:37:55.614497+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-06-04T19:55:51.181598+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,4],"acting":[7,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"17.19","version":"0'0","reported_seq":934,"reported_epoch":1009,"state":"active+clean","last_fresh":"2024-06-03T09:06:13.652943+0000","last_change":"2024-06-03T08:38:01.839491+0000","last_active":"2024-06-03T09:06:13.652943+0000","last_peered":"2024-06-03T09:06:13.652943+0000","last_clean":"2024-06-03T09:06:13.652943+0000","last_became_active":"2024-06-03T08:38:01.829539+0000","last_became_peered":"2024-06-03T08:38:01.829539+0000","last_unstale":"2024-06-03T09:06:13.652943+0000","last_undegraded":"2024-06-03T09:06:13.652943+0000","last_fullsized":"2024-06-03T09:06:13.652943+0000","mapping_epoch":28,"log_start":"0'0","ondisk_log_start":"0'0","created":28,"last_epoch_clean":30,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-06-03T08:37:55.614497+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-06-03T08:37:55.614497+0000","last_clean_scrub_stamp":"2024-06-03T08:37:55.614497+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-06-04T09:10:43.860229+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,2],"acting":[3,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"17.1e","version":"0'0","reported_seq":148,"reported_epoch":1009,"state":"active+undersized","last_fresh":"2024-06-03T09:06:14.445917+0000","last_change":"2024-06-03T09:06:03.406181+0000","last_active":"2024-06-03T09:06:14.445917+0000","last_peered":"2024-06-03T09:06:14.445917+0000","last_clean":"2024-06-03T08:37:55.614497+0000","last_became_active":"2024-06-03T09:06:03.406181+0000","last_became_peered":"2024-06-03T09:06:03.406181+0000","last_unstale":"2024-06-03T09:06:14.445917+0000","last_undegraded":"2024-06-03T09:06:14.445917+0000","last_fullsized":"2024-06-03T09:06:03.363739+0000","mapping_epoch":999,"log_start":"0'0","ondisk_log_start":"0'0","created":28,"last_epoch_clean":255,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-06-03T08:37:55.614497+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-06-03T08:37:55.614497+0000","last_clean_scrub_stamp":"2024-06-03T08:37:55.614497+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-06-04T11:45:24.486600+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4],"acting":[4],"avail_no_missing":["4"],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"17.1f","version":"0'0","reported_seq":941,"reported_epoch":1009,"state":"active+clean","last_fresh":"2024-06-03T09:06:14.450053+0000","last_change":"2024-06-03T08:38:04.383538+0000","last_active":"2024-06-03T09:06:14.450053+0000","last_peered":"2024-06-03T09:06:14.450053+0000","last_clean":"2024-06-03T09:06:14.450053+0000","last_became_active":"2024-06-03T08:38:04.367278+0000","last_became_peered":"2024-06-03T08:38:04.367278+0000","last_unstale":"2024-06-03T09:06:14.450053+0000","last_undegraded":"2024-06-03T09:06:14.450053+0000","last_fullsized":"2024-06-03T09:06:14.450053+0000","mapping_epoch":28,"log_start":"0'0","ondisk_log_start":"0'0","created":28,"last_epoch_clean":32,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-06-03T08:37:55.614497+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-06-03T08:37:55.614497+0000","last_clean_scrub_stamp":"2024-06-03T08:37:55.614497+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-06-04T10:39:26.552282+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,2],"acting":[0,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"17.1c","version":"0'0","reported_seq":944,"reported_epoch":1010,"state":"active+undersized","last_fresh":"2024-06-03T09:06:18.327492+0000","last_change":"2024-06-03T09:06:03.382210+0000","last_active":"2024-06-03T09:06:18.327492+0000","last_peered":"2024-06-03T09:06:18.327492+0000","last_clean":"2024-06-03T09:04:43.199660+0000","last_became_active":"2024-06-03T09:06:03.382210+0000","last_became_peered":"2024-06-03T09:06:03.382210+0000","last_unstale":"2024-06-03T09:06:18.327492+0000","last_undegraded":"2024-06-03T09:06:18.327492+0000","last_fullsized":"2024-06-03T09:06:03.277874+0000","mapping_epoch":999,"log_start":"0'0","ondisk_log_start":"0'0","created":28,"last_epoch_clean":255,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-06-03T08:37:55.614497+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-06-03T08:37:55.614497+0000","last_clean_scrub_stamp":"2024-06-03T08:37:55.614497+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-06-04T12:18:50.013884+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5],"acting":[5],"avail_no_missing":["5"],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"17.1d","version":"0'0","reported_seq":923,"reported_epoch":1009,"state":"active+clean","last_fresh":"2024-06-03T09:06:13.715906+0000","last_change":"2024-06-03T08:38:06.794630+0000","last_active":"2024-06-03T09:06:13.715906+0000","last_peered":"2024-06-03T09:06:13.715906+0000","last_clean":"2024-06-03T09:06:13.715906+0000","last_became_active":"2024-06-03T08:38:06.592317+0000","last_became_peered":"2024-06-03T08:38:06.592317+0000","last_unstale":"2024-06-03T09:06:13.715906+0000","last_undegraded":"2024-06-03T09:06:13.715906+0000","last_fullsized":"2024-06-03T09:06:13.715906+0000","mapping_epoch":28,"log_start":"0'0","ondisk_log_start":"0'0","created":28,"last_epoch_clean":33,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-06-03T08:37:55.614497+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-06-03T08:37:55.614497+0000","last_clean_scrub_stamp":"2024-06-03T08:37:55.614497+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-06-04T12:33:30.785369+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,7],"acting":[1,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"17.2","version":"0'0","reported_seq":925,"reported_epoch":1009,"state":"active+clean","last_fresh":"2024-06-03T09:06:13.708688+0000","last_change":"2024-06-03T08:38:14.981410+0000","last_active":"2024-06-03T09:06:13.708688+0000","last_peered":"2024-06-03T09:06:13.708688+0000","last_clean":"2024-06-03T09:06:13.708688+0000","last_became_active":"2024-06-03T08:38:14.973825+0000","last_became_peered":"2024-06-03T08:38:14.973825+0000","last_unstale":"2024-06-03T09:06:13.708688+0000","last_undegraded":"2024-06-03T09:06:13.708688+0000","last_fullsized":"2024-06-03T09:06:13.708688+0000","mapping_epoch":28,"log_start":"0'0","ondisk_log_start":"0'0","created":28,"last_epoch_clean":41,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-06-03T08:37:55.614497+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-06-03T08:37:55.614497+0000","last_clean_scrub_stamp":"2024-06-03T08:37:55.614497+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-06-04T17:01:05.518613+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,4],"acting":[1,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"17.3","version":"0'0","reported_seq":943,"reported_epoch":1009,"state":"active+clean","last_fresh":"2024-06-03T09:06:14.437636+0000","last_change":"2024-06-03T08:38:14.619745+0000","last_active":"2024-06-03T09:06:14.437636+0000","last_peered":"2024-06-03T09:06:14.437636+0000","last_clean":"2024-06-03T09:06:14.437636+0000","last_became_active":"2024-06-03T08:38:14.510665+0000","last_became_peered":"2024-06-03T08:38:14.510665+0000","last_unstale":"2024-06-03T09:06:14.437636+0000","last_undegraded":"2024-06-03T09:06:14.437636+0000","last_fullsized":"2024-06-03T09:06:14.437636+0000","mapping_epoch":28,"log_start":"0'0","ondisk_log_start":"0'0","created":28,"last_epoch_clean":42,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-06-03T08:37:55.614497+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-06-03T08:37:55.614497+0000","last_clean_scrub_stamp":"2024-06-03T08:37:55.614497+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-06-04T12:03:47.392064+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,7],"acting":[0,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"17.0","version":"0'0","reported_seq":944,"reported_epoch":1009,"state":"active+clean","last_fresh":"2024-06-03T09:06:14.427123+0000","last_change":"2024-06-03T08:38:04.237118+0000","last_active":"2024-06-03T09:06:14.427123+0000","last_peered":"2024-06-03T09:06:14.427123+0000","last_clean":"2024-06-03T09:06:14.427123+0000","last_became_active":"2024-06-03T08:38:04.231954+0000","last_became_peered":"2024-06-03T08:38:04.231954+0000","last_unstale":"2024-06-03T09:06:14.427123+0000","last_undegraded":"2024-06-03T09:06:14.427123+0000","last_fullsized":"2024-06-03T09:06:14.427123+0000","mapping_epoch":28,"log_start":"0'0","ondisk_log_start":"0'0","created":28,"last_epoch_clean":32,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-06-03T08:37:55.614497+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-06-03T08:37:55.614497+0000","last_clean_scrub_stamp":"2024-06-03T08:37:55.614497+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-06-04T09:56:10.741637+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,2],"acting":[0,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"17.1","version":"65'7","reported_seq":959,"reported_epoch":1009,"state":"active+undersized+degraded","last_fresh":"2024-06-03T09:06:13.709209+0000","last_change":"2024-06-03T09:06:03.370660+0000","last_active":"2024-06-03T09:06:13.709209+0000","last_peered":"2024-06-03T09:06:13.709209+0000","last_clean":"2024-06-03T09:05:03.733864+0000","last_became_active":"2024-06-03T09:06:03.370660+0000","last_became_peered":"2024-06-03T09:06:03.370660+0000","last_unstale":"2024-06-03T09:06:13.709209+0000","last_undegraded":"2024-06-03T09:06:03.266076+0000","last_fullsized":"2024-06-03T09:06:03.264525+0000","mapping_epoch":999,"log_start":"0'0","ondisk_log_start":"0'0","created":28,"last_epoch_clean":255,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-06-03T08:37:55.614497+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-06-03T08:37:55.614497+0000","last_clean_scrub_stamp":"2024-06-03T08:37:55.614497+0000","objects_scrubbed":0,"log_size":7,"log_dups_size":0,"ondisk_log_size":7,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-06-04T11:55:18.194283+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":5,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":1,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":6,"num_read_kb":6,"num_write":7,"num_write_kb":7,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5],"acting":[5],"avail_no_missing":["5"],"object_location_counts":[{"shards":"5","objects":1}],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]}],"pool_stats":[{"poolid":2,"num_pg":8,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":14,"acting":14,"num_store_stats":7},{"poolid":1,"num_pg":1,"stat_sum":{"num_bytes":459280,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":96,"num_read_kb":82,"num_write":113,"num_write_kb":1372,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2,"num_bytes_recovered":397840,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":802816,"data_stored":795680,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":76,"ondisk_log_size":76,"up":2,"acting":2,"num_store_stats":2},{"poolid":17,"num_pg":32,"stat_sum":{"num_bytes":5,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":1,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":6,"num_read_kb":6,"num_write":7,"num_write_kb":7,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":8192,"data_stored":10,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":7,"ondisk_log_size":7,"up":52,"acting":52,"num_store_stats":8}],"osd_stats":[{"osd":6,"up_from":253,"seq":1086626726200,"num_pgs":32,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":54772,"kb_used_data":5608,"kb_used_omap":1,"kb_used_meta":49150,"kb_avail":94317068,"statfs":{"total":96636764160,"available":96580677632,"internally_reserved":0,"allocated":5742592,"data_stored":2767180,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1593,"internal_metadata":50330055},"hb_peers":[0,1,2,3,4,5,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":109,"apply_latency_ms":109,"commit_latency_ns":109000000,"apply_latency_ns":109000000},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Mon Jun 3 09:04:02 2024","interfaces":[{"interface":"back","average":{"1min":1582.4780000000001,"5min":332.72699999999998,"15min":124.113},"min":{"1min":4.4829999999999997,"5min":4.1959999999999997,"15min":4.1959999999999997},"max":{"1min":22766.882000000001,"5min":22766.882000000001,"15min":22766.882000000001},"last":21.917999999999999},{"interface":"front","average":{"1min":1579.5409999999999,"5min":334.94499999999999,"15min":125.84999999999999},"min":{"1min":4.8129999999999997,"5min":4.7539999999999996,"15min":4.4530000000000003},"max":{"1min":22766.289000000001,"5min":22766.289000000001,"15min":22766.289000000001},"last":25.408999999999999}]},{"osd":1,"last update":"Mon Jun 3 09:03:40 2024","interfaces":[{"interface":"back","average":{"1min":36.420999999999999,"5min":77.054000000000002,"15min":132.84800000000001},"min":{"1min":6.0380000000000003,"5min":5.8799999999999999,"15min":5.484},"max":{"1min":589.64099999999996,"5min":4138.1679999999997,"15min":9955.1299999999992},"last":26.84},{"interface":"front","average":{"1min":40.240000000000002,"5min":79.674999999999997,"15min":133.38999999999999},"min":{"1min":8.4589999999999996,"5min":8.4090000000000007,"15min":6.1390000000000002},"max":{"1min":588.60799999999995,"5min":4137.6530000000002,"15min":9827.5509999999995},"last":26.263000000000002}]},{"osd":2,"last update":"Mon Jun 3 09:03:40 2024","interfaces":[{"interface":"back","average":{"1min":12.613,"5min":24.600999999999999,"15min":21.088000000000001},"min":{"1min":6.9740000000000002,"5min":6.9740000000000002,"15min":6.5330000000000004},"max":{"1min":26.076000000000001,"5min":376.24900000000002,"15min":376.24900000000002},"last":24.899000000000001},{"interface":"front","average":{"1min":12.493,"5min":22.207999999999998,"15min":21.219000000000001},"min":{"1min":6.5810000000000004,"5min":6.5110000000000001,"15min":6.0750000000000002},"max":{"1min":29.690999999999999,"5min":271.51100000000002,"15min":434.77300000000002},"last":40.372}]},{"osd":3,"last update":"Mon Jun 3 09:03:40 2024","interfaces":[{"interface":"back","average":{"1min":14.093999999999999,"5min":24.143999999999998,"15min":29.376999999999999},"min":{"1min":10.196999999999999,"5min":6.2919999999999998,"15min":6.1719999999999997},"max":{"1min":24.832000000000001,"5min":197.84899999999999,"15min":1235.5799999999999},"last":24.353999999999999},{"interface":"front","average":{"1min":9.3260000000000005,"5min":18.463999999999999,"15min":27.631},"min":{"1min":6.125,"5min":6.125,"15min":6.125},"max":{"1min":28.166,"5min":200.173,"15min":1157.6780000000001},"last":23.454000000000001}]},{"osd":4,"last update":"Mon Jun 3 09:04:29 2024","interfaces":[{"interface":"back","average":{"1min":10.279,"5min":1253.809,"15min":450.79199999999997},"min":{"1min":7.2300000000000004,"5min":6.093,"15min":6.093},"max":{"1min":15.000999999999999,"5min":28093.096000000001,"15min":28093.096000000001},"last":101.45399999999999},{"interface":"front","average":{"1min":15.693,"5min":1279.759,"15min":463.87099999999998},"min":{"1min":10.551,"5min":7.7480000000000002,"15min":7.7480000000000002},"max":{"1min":27.898,"5min":28323.837,"15min":28323.837},"last":102.262}]},{"osd":5,"last update":"Mon Jun 3 09:04:10 2024","interfaces":[{"interface":"back","average":{"1min":13.728999999999999,"5min":24.670000000000002,"15min":23.815999999999999},"min":{"1min":10.401,"5min":7.516,"15min":7.516},"max":{"1min":25.138000000000002,"5min":320.85000000000002,"15min":445.041},"last":103.52500000000001},{"interface":"front","average":{"1min":13.847,"5min":29.283000000000001,"15min":23.393999999999998},"min":{"1min":7.944,"5min":6.266,"15min":6.266},"max":{"1min":18.817,"5min":530.56700000000001,"15min":530.56700000000001},"last":29.263999999999999}]},{"osd":7,"last update":"Mon Jun 3 09:03:40 2024","interfaces":[{"interface":"back","average":{"1min":20.568999999999999,"5min":30.286999999999999,"15min":27.977},"min":{"1min":13.874000000000001,"5min":10.58,"15min":9.2859999999999996},"max":{"1min":86.959000000000003,"5min":222.07599999999999,"15min":255.08799999999999},"last":37.618000000000002},{"interface":"front","average":{"1min":19.102,"5min":29.550000000000001,"15min":27.763000000000002},"min":{"1min":10.052,"5min":8.6709999999999994,"15min":8.5429999999999993},"max":{"1min":67.968999999999994,"5min":293.62099999999998,"15min":438.14600000000002},"last":111.985}]}]},{"osd":7,"up_from":20,"seq":85899346287,"num_pgs":9,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":58964,"kb_used_data":5704,"kb_used_omap":1,"kb_used_meta":53246,"kb_avail":94312876,"statfs":{"total":96636764160,"available":96576385024,"internally_reserved":0,"allocated":5840896,"data_stored":2740727,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1589,"internal_metadata":54524363},"hb_peers":[0,1,2,3,4,5],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Mon Jun 3 09:06:09 2024","interfaces":[{"interface":"back","average":{"1min":14.073,"5min":894.22400000000005,"15min":308.93400000000003},"min":{"1min":5.5709999999999997,"5min":4.218,"15min":4.0599999999999996},"max":{"1min":90.757999999999996,"5min":21729.001,"15min":21729.001},"last":9.1039999999999992},{"interface":"front","average":{"1min":9.2550000000000008,"5min":906.23500000000001,"15min":314.06},"min":{"1min":5.5789999999999997,"5min":4.8810000000000002,"15min":4.8810000000000002},"max":{"1min":25.937000000000001,"5min":21899.732,"15min":21899.732},"last":5.7240000000000002}]},{"osd":1,"last update":"Mon Jun 3 09:06:09 2024","interfaces":[{"interface":"back","average":{"1min":15.182,"5min":69.183999999999997,"15min":219.173},"min":{"1min":7.3639999999999999,"5min":6.6200000000000001,"15min":6.1319999999999997},"max":{"1min":76.926000000000002,"5min":2795.1329999999998,"15min":18373.257000000001},"last":9.8119999999999994},{"interface":"front","average":{"1min":21.495999999999999,"5min":74.703999999999994,"15min":225.285},"min":{"1min":8.8399999999999999,"5min":8.8399999999999999,"15min":8.5129999999999999},"max":{"1min":94.977999999999994,"5min":2803.3670000000002,"15min":18374.601999999999},"last":12.17}]},{"osd":2,"last update":"Mon Jun 3 09:06:09 2024","interfaces":[{"interface":"back","average":{"1min":19.164000000000001,"5min":26.902000000000001,"15min":22.920999999999999},"min":{"1min":11.207000000000001,"5min":8.5030000000000001,"15min":8.5030000000000001},"max":{"1min":91.319999999999993,"5min":333.99200000000002,"15min":333.99200000000002},"last":14.196999999999999},{"interface":"front","average":{"1min":11.125999999999999,"5min":18.132000000000001,"15min":16.353000000000002},"min":{"1min":6.202,"5min":5.8369999999999997,"15min":5.5069999999999997},"max":{"1min":28.649000000000001,"5min":226.18100000000001,"15min":226.18100000000001},"last":11.329000000000001}]},{"osd":3,"last update":"Mon Jun 3 09:06:09 2024","interfaces":[{"interface":"back","average":{"1min":22.291,"5min":26.289999999999999,"15min":31.074999999999999},"min":{"1min":11.714,"5min":6.8959999999999999,"15min":6.3760000000000003},"max":{"1min":83.102000000000004,"5min":316.64400000000001,"15min":2039.9059999999999},"last":15.954000000000001},{"interface":"front","average":{"1min":13.324999999999999,"5min":17.963000000000001,"15min":25.969999999999999},"min":{"1min":6.1200000000000001,"5min":5.2089999999999996,"15min":5.2089999999999996},"max":{"1min":49.908999999999999,"5min":233.542,"15min":2037.5550000000001},"last":13.622999999999999}]},{"osd":4,"last update":"Mon Jun 3 09:06:29 2024","interfaces":[{"interface":"back","average":{"1min":17.675999999999998,"5min":1026.325,"15min":472.47199999999998},"min":{"1min":7.306,"5min":5.9089999999999998,"15min":5.9089999999999998},"max":{"1min":140.03299999999999,"5min":28969.369999999999,"15min":30073.917000000001},"last":8.7119999999999997},{"interface":"front","average":{"1min":17.890999999999998,"5min":1044.5139999999999,"15min":480.226},"min":{"1min":5.9279999999999999,"5min":5.9279999999999999,"15min":5.9279999999999999},"max":{"1min":141.179,"5min":29244.138999999999,"15min":30348.021000000001},"last":10.571}]},{"osd":5,"last update":"Mon Jun 3 09:06:02 2024","interfaces":[{"interface":"back","average":{"1min":24.372,"5min":26.856000000000002,"15min":23.571999999999999},"min":{"1min":12.278,"5min":9.2460000000000004,"15min":6.5650000000000004},"max":{"1min":183.64099999999999,"5min":317.572,"15min":317.572},"last":15.128},{"interface":"front","average":{"1min":21.259,"5min":23.364000000000001,"15min":21.193999999999999},"min":{"1min":7.8499999999999996,"5min":7.0999999999999996,"15min":7.008},"max":{"1min":184.53399999999999,"5min":306.60599999999999,"15min":306.60599999999999},"last":12.704000000000001}]},{"osd":6,"last update":"Mon Jun 3 09:04:31 2024","interfaces":[{"interface":"back","average":{"1min":44.061,"5min":26.821000000000002,"15min":91.176000000000002},"min":{"1min":9.0800000000000001,"5min":7.9390000000000001,"15min":7.9390000000000001},"max":{"1min":268.029,"5min":308.916,"15min":8734.4639999999999},"last":6.6769999999999996},{"interface":"front","average":{"1min":46.643000000000001,"5min":23.879999999999999,"15min":88.765000000000001},"min":{"1min":8.8119999999999994,"5min":6.9240000000000004,"15min":6.742},"max":{"1min":305.64600000000002,"5min":305.64600000000002,"15min":8735.5319999999992},"last":12.327}]}]},{"osd":5,"up_from":20,"seq":85899346261,"num_pgs":9,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":59360,"kb_used_data":6100,"kb_used_omap":1,"kb_used_meta":53246,"kb_avail":94312480,"statfs":{"total":96636764160,"available":96575979520,"internally_reserved":0,"allocated":6246400,"data_stored":3138572,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1594,"internal_metadata":54524358},"hb_peers":[0,1,2,3,4,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Mon Jun 3 09:06:06 2024","interfaces":[{"interface":"back","average":{"1min":7.468,"5min":1703.037,"15min":580.51900000000001},"min":{"1min":4.6509999999999998,"5min":4.6509999999999998,"15min":4.4020000000000001},"max":{"1min":14.711,"5min":26433.764999999999,"15min":26433.764999999999},"last":8.4039999999999999},{"interface":"front","average":{"1min":9.9429999999999996,"5min":1725.27,"15min":587.87800000000004},"min":{"1min":5.5999999999999996,"5min":5.1319999999999997,"15min":4.8170000000000002},"max":{"1min":13.590999999999999,"5min":26704.698,"15min":26704.698},"last":9.7420000000000009}]},{"osd":1,"last update":"Mon Jun 3 09:06:00 2024","interfaces":[{"interface":"back","average":{"1min":10.862,"5min":67.731999999999999,"15min":217.364},"min":{"1min":7.1769999999999996,"5min":6.0890000000000004,"15min":5.4930000000000003},"max":{"1min":16.998999999999999,"5min":3923.4090000000001,"15min":18811.696},"last":6.1040000000000001},{"interface":"front","average":{"1min":10.417,"5min":65.141999999999996,"15min":220.75700000000001},"min":{"1min":5.3209999999999997,"5min":5.3209999999999997,"15min":5.0209999999999999},"max":{"1min":15.821999999999999,"5min":4019.6129999999998,"15min":18968.365000000002},"last":9.2059999999999995}]},{"osd":2,"last update":"Mon Jun 3 09:06:00 2024","interfaces":[{"interface":"back","average":{"1min":11.366,"5min":24.18,"15min":21.137},"min":{"1min":5.6369999999999996,"5min":5.6369999999999996,"15min":5.0510000000000002},"max":{"1min":15.529999999999999,"5min":332.48200000000003,"15min":332.48200000000003},"last":10.680999999999999},{"interface":"front","average":{"1min":10.089,"5min":23.117000000000001,"15min":24.050000000000001},"min":{"1min":6.6769999999999996,"5min":6.6769999999999996,"15min":5.8739999999999997},"max":{"1min":15.821,"5min":353.03199999999998,"15min":353.03199999999998},"last":12.638999999999999}]},{"osd":3,"last update":"Mon Jun 3 09:06:00 2024","interfaces":[{"interface":"back","average":{"1min":13.472,"5min":24.283999999999999,"15min":31.666},"min":{"1min":10.111000000000001,"5min":7.4720000000000004,"15min":7.4210000000000003},"max":{"1min":16.271999999999998,"5min":353.57499999999999,"15min":2855.8519999999999},"last":11.196999999999999},{"interface":"front","average":{"1min":9.5899999999999999,"5min":15.875,"15min":25.834},"min":{"1min":6.6609999999999996,"5min":5.7729999999999997,"15min":5.1909999999999998},"max":{"1min":15.112,"5min":203.804,"15min":2869.7860000000001},"last":12.1}]},{"osd":4,"last update":"Mon Jun 3 09:05:27 2024","interfaces":[{"interface":"back","average":{"1min":19.155000000000001,"5min":1084.961,"15min":1346.9949999999999},"min":{"1min":7.6130000000000004,"5min":7.6130000000000004,"15min":6.1340000000000003},"max":{"1min":156.55199999999999,"5min":28619.825000000001,"15min":57999.453999999998},"last":6.8789999999999996},{"interface":"front","average":{"1min":28.068999999999999,"5min":1087.106,"15min":1330.289},"min":{"1min":8.3949999999999996,"5min":6.5030000000000001,"15min":6.5030000000000001},"max":{"1min":254.91300000000001,"5min":28621.294000000002,"15min":57434.631999999998},"last":14.146000000000001}]},{"osd":6,"last update":"Mon Jun 3 09:03:57 2024","interfaces":[{"interface":"back","average":{"1min":23.536000000000001,"5min":36.457000000000001,"15min":83.799999999999997},"min":{"1min":9.4800000000000004,"5min":9.4800000000000004,"15min":8.0630000000000006},"max":{"1min":89.442999999999998,"5min":508.77999999999997,"15min":9193.5169999999998},"last":157.79300000000001},{"interface":"front","average":{"1min":18.891999999999999,"5min":22.100999999999999,"15min":75.489999999999995},"min":{"1min":5.923,"5min":5.923,"15min":5.923},"max":{"1min":78.424999999999997,"5min":464.62,"15min":9191.4429999999993},"last":148.09}]},{"osd":7,"last update":"Mon Jun 3 09:06:00 2024","interfaces":[{"interface":"back","average":{"1min":13.257,"5min":20.917999999999999,"15min":21.088999999999999},"min":{"1min":10.475,"5min":6.7629999999999999,"15min":6.7629999999999999},"max":{"1min":17.245000000000001,"5min":211.22999999999999,"15min":211.22999999999999},"last":14.390000000000001},{"interface":"front","average":{"1min":12.872999999999999,"5min":20.587,"15min":24.021000000000001},"min":{"1min":8.5760000000000005,"5min":7.7709999999999999,"15min":6.8570000000000002},"max":{"1min":18.359999999999999,"5min":223.191,"15min":286.69499999999999},"last":7.3949999999999996}]}]},{"osd":4,"up_from":20,"seq":85899346269,"num_pgs":12,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":63060,"kb_used_data":5704,"kb_used_omap":1,"kb_used_meta":57342,"kb_avail":94308780,"statfs":{"total":96636764160,"available":96572190720,"internally_reserved":0,"allocated":5840896,"data_stored":2740727,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1594,"internal_metadata":58718662},"hb_peers":[0,1,2,3,5,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Mon Jun 3 09:05:34 2024","interfaces":[{"interface":"back","average":{"1min":12.416,"5min":1099.4649999999999,"15min":382.15199999999999},"min":{"1min":4.5279999999999996,"5min":4.5279999999999996,"15min":4.3220000000000001},"max":{"1min":123.878,"5min":21726.743999999999,"15min":21726.743999999999},"last":7.5579999999999998},{"interface":"front","average":{"1min":16.462,"5min":1108.894,"15min":387.70299999999997},"min":{"1min":6.4089999999999998,"5min":5.8380000000000001,"15min":5.4909999999999997},"max":{"1min":130.185,"5min":21727.993999999999,"15min":21727.993999999999},"last":11.199}]},{"osd":1,"last update":"Mon Jun 3 09:05:34 2024","interfaces":[{"interface":"back","average":{"1min":14.727,"5min":56.755000000000003,"15min":341.09399999999999},"min":{"1min":4.6529999999999996,"5min":4.1970000000000001,"15min":4.1970000000000001},"max":{"1min":126.899,"5min":3346.4389999999999,"15min":22256.098000000002},"last":6.7679999999999998},{"interface":"front","average":{"1min":13.472,"5min":57.503,"15min":340.303},"min":{"1min":5.0590000000000002,"5min":5.0590000000000002,"15min":4.2640000000000002},"max":{"1min":126.096,"5min":3347.181,"15min":22247.317999999999},"last":12.481}]},{"osd":2,"last update":"Mon Jun 3 09:05:34 2024","interfaces":[{"interface":"back","average":{"1min":26.088999999999999,"5min":23.866,"15min":20.620999999999999},"min":{"1min":7.0869999999999997,"5min":6.8239999999999998,"15min":6.3150000000000004},"max":{"1min":337.60399999999998,"5min":393.58600000000001,"15min":393.58600000000001},"last":10.336},{"interface":"front","average":{"1min":15.800000000000001,"5min":22.584,"15min":23.245999999999999},"min":{"1min":6.0629999999999997,"5min":6.0629999999999997,"15min":6.0629999999999997},"max":{"1min":127.913,"5min":394.471,"15min":394.471},"last":14.75}]},{"osd":3,"last update":"Mon Jun 3 09:05:34 2024","interfaces":[{"interface":"back","average":{"1min":17.544,"5min":19.143999999999998,"15min":18.742000000000001},"min":{"1min":5.8719999999999999,"5min":5.4100000000000001,"15min":5.4000000000000004},"max":{"1min":127.43600000000001,"5min":322.63,"15min":322.63},"last":15.779},{"interface":"front","average":{"1min":21.327000000000002,"5min":23.559000000000001,"15min":23.797000000000001},"min":{"1min":10.023,"5min":7.484,"15min":7.484},"max":{"1min":204.517,"5min":392.67500000000001,"15min":392.67500000000001},"last":11.742000000000001}]},{"osd":5,"last update":"Mon Jun 3 09:05:34 2024","interfaces":[{"interface":"back","average":{"1min":24.274999999999999,"5min":21.274999999999999,"15min":20.988},"min":{"1min":6.7489999999999997,"5min":6.7489999999999997,"15min":6.7489999999999997},"max":{"1min":268.10199999999998,"5min":344.23399999999998,"15min":344.23399999999998},"last":13.760999999999999},{"interface":"front","average":{"1min":26.361999999999998,"5min":25.218,"15min":26.084},"min":{"1min":12.228,"5min":8.5180000000000007,"15min":8.5180000000000007},"max":{"1min":264.75,"5min":345.18700000000001,"15min":345.18700000000001},"last":17.739000000000001}]},{"osd":6,"last update":"Mon Jun 3 09:04:32 2024","interfaces":[{"interface":"back","average":{"1min":15.548,"5min":32.957999999999998,"15min":152.738},"min":{"1min":9.141,"5min":9.141,"15min":8.2319999999999993},"max":{"1min":52.57,"5min":395.238,"15min":12496.335999999999},"last":208.90000000000001},{"interface":"front","average":{"1min":12.680999999999999,"5min":25.428000000000001,"15min":149.30600000000001},"min":{"1min":6.4379999999999997,"5min":5.6319999999999997,"15min":5.6319999999999997},"max":{"1min":46.283000000000001,"5min":377.74299999999999,"15min":12616.297},"last":205.88900000000001}]},{"osd":7,"last update":"Mon Jun 3 09:05:34 2024","interfaces":[{"interface":"back","average":{"1min":18.295999999999999,"5min":19.585999999999999,"15min":21.062000000000001},"min":{"1min":6.2110000000000003,"5min":6.2110000000000003,"15min":5.8010000000000002},"max":{"1min":207.25299999999999,"5min":358.94900000000001,"15min":358.94900000000001},"last":9.2669999999999995},{"interface":"front","average":{"1min":23.712,"5min":24.024999999999999,"15min":24.640000000000001},"min":{"1min":11.824999999999999,"5min":7.7789999999999999,"15min":7.7789999999999999},"max":{"1min":209.32900000000001,"5min":346.25200000000001,"15min":380.06900000000002},"last":16.763999999999999}]}]},{"osd":2,"up_from":20,"seq":85899346289,"num_pgs":11,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":58964,"kb_used_data":5704,"kb_used_omap":1,"kb_used_meta":53246,"kb_avail":94312876,"statfs":{"total":96636764160,"available":96576385024,"internally_reserved":0,"allocated":5840896,"data_stored":2740727,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1594,"internal_metadata":54524358},"hb_peers":[0,1,3,4,5,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Mon Jun 3 09:06:23 2024","interfaces":[{"interface":"back","average":{"1min":8.9629999999999992,"5min":946.25400000000002,"15min":326.36099999999999},"min":{"1min":5.3479999999999999,"5min":4.7050000000000001,"15min":4.1420000000000003},"max":{"1min":26.015000000000001,"5min":23465.18,"15min":23465.18},"last":6.6070000000000002},{"interface":"front","average":{"1min":7.6130000000000004,"5min":944.39800000000002,"15min":326.178},"min":{"1min":4.6790000000000003,"5min":4.6790000000000003,"15min":4.6790000000000003},"max":{"1min":20.802,"5min":23465.966,"15min":23465.966},"last":5.1609999999999996}]},{"osd":1,"last update":"Mon Jun 3 09:06:23 2024","interfaces":[{"interface":"back","average":{"1min":13.007,"5min":76.507000000000005,"15min":294.72899999999998},"min":{"1min":6.4089999999999998,"5min":5.4199999999999999,"15min":5.4199999999999999},"max":{"1min":85.430000000000007,"5min":4220.5290000000005,"15min":18954.987000000001},"last":7.2859999999999996},{"interface":"front","average":{"1min":15.894,"5min":82.119,"15min":302.92399999999998},"min":{"1min":7.367,"5min":7.367,"15min":7.367},"max":{"1min":98.498000000000005,"5min":4325.9489999999996,"15min":19143.834999999999},"last":9.2759999999999998}]},{"osd":3,"last update":"Mon Jun 3 09:06:23 2024","interfaces":[{"interface":"back","average":{"1min":9.7579999999999991,"5min":12.444000000000001,"15min":28.065000000000001},"min":{"1min":7,"5min":5.9290000000000003,"15min":5.2450000000000001},"max":{"1min":28.164000000000001,"5min":118.913,"15min":2594.8150000000001},"last":10.537000000000001},{"interface":"front","average":{"1min":12.321999999999999,"5min":15.25,"15min":31.349},"min":{"1min":9.1240000000000006,"5min":8.0760000000000005,"15min":6.0780000000000003},"max":{"1min":32.360999999999997,"5min":156.34700000000001,"15min":2706.5120000000002},"last":12.038}]},{"osd":4,"last update":"Mon Jun 3 09:05:57 2024","interfaces":[{"interface":"back","average":{"1min":10.573,"5min":1266.894,"15min":438.154},"min":{"1min":8.6379999999999999,"5min":7.8789999999999996,"15min":6.2039999999999997},"max":{"1min":15.105,"5min":26660.861000000001,"15min":26660.861000000001},"last":11.036},{"interface":"front","average":{"1min":12.407999999999999,"5min":1266.7460000000001,"15min":435.75400000000002},"min":{"1min":10.266999999999999,"5min":10.266999999999999,"15min":6.2720000000000002},"max":{"1min":15.228,"5min":26616.243999999999,"15min":26616.243999999999},"last":12.824}]},{"osd":5,"last update":"Mon Jun 3 09:06:23 2024","interfaces":[{"interface":"back","average":{"1min":14.103,"5min":18.091000000000001,"15min":21.119},"min":{"1min":9.6349999999999998,"5min":9.2159999999999993,"15min":6.2050000000000001},"max":{"1min":32.890000000000001,"5min":252.31100000000001,"15min":252.31100000000001},"last":13.518000000000001},{"interface":"front","average":{"1min":11.545999999999999,"5min":16.170999999999999,"15min":20.254000000000001},"min":{"1min":7.6070000000000002,"5min":7.4850000000000003,"15min":6.8810000000000002},"max":{"1min":29.228999999999999,"5min":258.31700000000001,"15min":258.31700000000001},"last":8.6509999999999998}]},{"osd":6,"last update":"Mon Jun 3 09:04:28 2024","interfaces":[{"interface":"back","average":{"1min":14.063000000000001,"5min":24.850000000000001,"15min":106.755},"min":{"1min":9.0820000000000007,"5min":8.9649999999999999,"15min":8.9649999999999999},"max":{"1min":24.666,"5min":188.06700000000001,"15min":9521.0010000000002},"last":183.60900000000001},{"interface":"front","average":{"1min":13.933999999999999,"5min":21.081,"15min":106.142},"min":{"1min":5.8719999999999999,"5min":5.8719999999999999,"15min":5.8719999999999999},"max":{"1min":117.551,"5min":193.887,"15min":9585.6620000000003},"last":184.20599999999999}]},{"osd":7,"last update":"Mon Jun 3 09:06:23 2024","interfaces":[{"interface":"back","average":{"1min":17.056000000000001,"5min":20.055,"15min":23.268999999999998},"min":{"1min":10.353999999999999,"5min":6.6130000000000004,"15min":6.6130000000000004},"max":{"1min":88.974999999999994,"5min":242.56800000000001,"15min":242.56800000000001},"last":14.315},{"interface":"front","average":{"1min":18.885000000000002,"5min":21.460000000000001,"15min":25.489999999999998},"min":{"1min":12.497,"5min":9.0660000000000007,"15min":9.0660000000000007},"max":{"1min":102.229,"5min":181.56399999999999,"15min":214.33799999999999},"last":15.112}]}]},{"osd":3,"up_from":14,"seq":60129542503,"num_pgs":11,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":58964,"kb_used_data":5704,"kb_used_omap":1,"kb_used_meta":53246,"kb_avail":94312876,"statfs":{"total":96636764160,"available":96576385024,"internally_reserved":0,"allocated":5840896,"data_stored":2740727,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1592,"internal_metadata":54524360},"hb_peers":[0,1,2,4,5,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Mon Jun 3 09:06:25 2024","interfaces":[{"interface":"back","average":{"1min":7.79,"5min":1060.1959999999999,"15min":367.01799999999997},"min":{"1min":5.6909999999999998,"5min":5.173,"15min":4.6929999999999996},"max":{"1min":10.59,"5min":23275.167000000001,"15min":23275.167000000001},"last":8.5299999999999994},{"interface":"front","average":{"1min":11.215999999999999,"5min":1066.5799999999999,"15min":372.476},"min":{"1min":8.7080000000000002,"5min":7.2510000000000003,"15min":5.9880000000000004},"max":{"1min":13.499000000000001,"5min":23278.536,"15min":23278.536},"last":14.012}]},{"osd":1,"last update":"Mon Jun 3 09:06:25 2024","interfaces":[{"interface":"back","average":{"1min":7.3760000000000003,"5min":15.614000000000001,"15min":389.83100000000002},"min":{"1min":4.335,"5min":4.335,"15min":4.335},"max":{"1min":11.289,"5min":250.72300000000001,"15min":21658.205999999998},"last":10.667},{"interface":"front","average":{"1min":8.9499999999999993,"5min":14.135,"15min":381.10300000000001},"min":{"1min":6.5609999999999999,"5min":5.4100000000000001,"15min":5.2779999999999996},"max":{"1min":14.51,"5min":180.44499999999999,"15min":21307.069},"last":6.109}]},{"osd":2,"last update":"Mon Jun 3 09:06:25 2024","interfaces":[{"interface":"back","average":{"1min":12.343999999999999,"5min":22.059999999999999,"15min":23.715},"min":{"1min":7.9199999999999999,"5min":6.4900000000000002,"15min":6.056},"max":{"1min":15.601000000000001,"5min":377.803,"15min":377.803},"last":9.2889999999999997},{"interface":"front","average":{"1min":10.319000000000001,"5min":19.364999999999998,"15min":20.242999999999999},"min":{"1min":6.891,"5min":6.1749999999999998,"15min":5.9020000000000001},"max":{"1min":15.432,"5min":371.601,"15min":371.601},"last":11.827999999999999}]},{"osd":4,"last update":"Mon Jun 3 09:05:27 2024","interfaces":[{"interface":"back","average":{"1min":10.444000000000001,"5min":1211.4970000000001,"15min":1639.1579999999999},"min":{"1min":6.4429999999999996,"5min":6.2560000000000002,"15min":6.0309999999999997},"max":{"1min":21.263000000000002,"5min":30144.939999999999,"15min":62679.622000000003},"last":7.3719999999999999},{"interface":"front","average":{"1min":11.363,"5min":1219.6400000000001,"15min":1644.127},"min":{"1min":7.8109999999999999,"5min":7.8109999999999999,"15min":6.2439999999999998},"max":{"1min":20.108000000000001,"5min":30180.689999999999,"15min":62681.021999999997},"last":9.9800000000000004}]},{"osd":5,"last update":"Mon Jun 3 09:06:07 2024","interfaces":[{"interface":"back","average":{"1min":12.802,"5min":19.640999999999998,"15min":22.318000000000001},"min":{"1min":9.0890000000000004,"5min":6.8810000000000002,"15min":6.7690000000000001},"max":{"1min":16.745999999999999,"5min":249.88,"15min":327.34899999999999},"last":14.826000000000001},{"interface":"front","average":{"1min":15.002000000000001,"5min":24.003,"15min":26.649999999999999},"min":{"1min":11.984,"5min":9.9149999999999991,"15min":8.9879999999999995},"max":{"1min":18.407,"5min":251.47800000000001,"15min":331.86799999999999},"last":15.525}]},{"osd":6,"last update":"Mon Jun 3 09:04:04 2024","interfaces":[{"interface":"back","average":{"1min":22.815999999999999,"5min":31.513999999999999,"15min":181.74299999999999},"min":{"1min":9.4809999999999999,"5min":7.2240000000000002,"15min":7.2240000000000002},"max":{"1min":160.56999999999999,"5min":380.613,"15min":11893.74},"last":19.359999999999999},{"interface":"front","average":{"1min":25.359999999999999,"5min":30.981000000000002,"15min":179.81399999999999},"min":{"1min":9.8870000000000005,"5min":7.3360000000000003,"15min":6.0899999999999999},"max":{"1min":193.11099999999999,"5min":379.80700000000002,"15min":11893.069},"last":8.202}]},{"osd":7,"last update":"Mon Jun 3 09:06:25 2024","interfaces":[{"interface":"back","average":{"1min":11.321999999999999,"5min":18.443999999999999,"15min":21.568999999999999},"min":{"1min":8.109,"5min":8.109,"15min":7.4279999999999999},"max":{"1min":17.113,"5min":146.267,"15min":178.76300000000001},"last":11.212999999999999},{"interface":"front","average":{"1min":12.512,"5min":20.065999999999999,"15min":24.190000000000001},"min":{"1min":7.4989999999999997,"5min":7.4989999999999997,"15min":7.4989999999999997},"max":{"1min":18.047999999999998,"5min":148.13800000000001,"15min":417.22800000000001},"last":12.34}]}]},{"osd":1,"up_from":14,"seq":60129542501,"num_pgs":6,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":59356,"kb_used_data":6096,"kb_used_omap":1,"kb_used_meta":53246,"kb_avail":94312484,"statfs":{"total":96636764160,"available":96575983616,"internally_reserved":0,"allocated":6242304,"data_stored":3138567,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":54524357},"hb_peers":[0,2,3,4,5,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Mon Jun 3 09:06:07 2024","interfaces":[{"interface":"back","average":{"1min":7.7530000000000001,"5min":185.66999999999999,"15min":77.347999999999999},"min":{"1min":5.0229999999999997,"5min":4.9279999999999999,"15min":4.5449999999999999},"max":{"1min":11.896000000000001,"5min":7470.616,"15min":7470.616},"last":7.7709999999999999},{"interface":"front","average":{"1min":9.9220000000000006,"5min":187.14599999999999,"15min":76.325999999999993},"min":{"1min":5.3159999999999998,"5min":5.3159999999999998,"15min":5.3159999999999998},"max":{"1min":18.338000000000001,"5min":7477.0829999999996,"15min":7477.0829999999996},"last":10.446999999999999}]},{"osd":2,"last update":"Mon Jun 3 09:06:20 2024","interfaces":[{"interface":"back","average":{"1min":10.340999999999999,"5min":17.369,"15min":23.670999999999999},"min":{"1min":6.9740000000000002,"5min":5.8650000000000002,"15min":5.8650000000000002},"max":{"1min":15.452,"5min":228,"15min":657.38},"last":11.384},{"interface":"front","average":{"1min":13.387,"5min":23.355,"15min":31.524000000000001},"min":{"1min":9.077,"5min":9.077,"15min":7.2160000000000002},"max":{"1min":19.701000000000001,"5min":229.465,"15min":700.06500000000005},"last":13.276}]},{"osd":3,"last update":"Mon Jun 3 09:06:04 2024","interfaces":[{"interface":"back","average":{"1min":8.6110000000000007,"5min":15.733000000000001,"15min":26.646999999999998},"min":{"1min":5.827,"5min":5.4640000000000004,"15min":5.4640000000000004},"max":{"1min":13.105,"5min":121.619,"15min":1676.009},"last":7.0140000000000002},{"interface":"front","average":{"1min":9.5850000000000009,"5min":16.161999999999999,"15min":29.065999999999999},"min":{"1min":6.3630000000000004,"5min":5.7699999999999996,"15min":5.7699999999999996},"max":{"1min":15.166,"5min":132.036,"15min":1766.5840000000001},"last":8.3870000000000005}]},{"osd":4,"last update":"Mon Jun 3 09:05:31 2024","interfaces":[{"interface":"back","average":{"1min":10.996,"5min":954.46500000000003,"15min":1281.6859999999999},"min":{"1min":7.702,"5min":6.1760000000000002,"15min":6.0970000000000004},"max":{"1min":21.538,"5min":28044.862000000001,"15min":59438.017999999996},"last":9.5630000000000006},{"interface":"front","average":{"1min":13.646000000000001,"5min":961.24599999999998,"15min":1285.26},"min":{"1min":9.7200000000000006,"5min":9.3940000000000001,"15min":6.4640000000000004},"max":{"1min":23.277999999999999,"5min":28045.624,"15min":59439.283000000003},"last":12.569000000000001}]},{"osd":5,"last update":"Mon Jun 3 09:06:04 2024","interfaces":[{"interface":"back","average":{"1min":12.478999999999999,"5min":20.228000000000002,"15min":25.997},"min":{"1min":8.6430000000000007,"5min":7.9720000000000004,"15min":7.234},"max":{"1min":17.442,"5min":139.43199999999999,"15min":701.22000000000003},"last":17.462},{"interface":"front","average":{"1min":13.507999999999999,"5min":25.760999999999999,"15min":27.553999999999998},"min":{"1min":10.042999999999999,"5min":7.258,"15min":6.242},"max":{"1min":20.611000000000001,"5min":278.63,"15min":598.35000000000002},"last":14.523999999999999}]},{"osd":6,"last update":"Mon Jun 3 09:03:54 2024","interfaces":[{"interface":"back","average":{"1min":28.524000000000001,"5min":20.654,"15min":23.878},"min":{"1min":6.0309999999999997,"5min":6.0309999999999997,"15min":6.0309999999999997},"max":{"1min":167.221,"5min":167.221,"15min":704.245},"last":18.721},{"interface":"front","average":{"1min":27.606000000000002,"5min":30.131,"15min":28.628},"min":{"1min":9.8650000000000002,"5min":9.7200000000000006,"15min":8.3580000000000005},"max":{"1min":123.938,"5min":358.23000000000002,"15min":634.05499999999995},"last":173.113}]},{"osd":7,"last update":"Mon Jun 3 09:06:20 2024","interfaces":[{"interface":"back","average":{"1min":14.66,"5min":23.234999999999999,"15min":31.097000000000001},"min":{"1min":11.250999999999999,"5min":8.0559999999999992,"15min":8.0559999999999992},"max":{"1min":21.507999999999999,"5min":270.95100000000002,"15min":703.07799999999997},"last":15.736000000000001},{"interface":"front","average":{"1min":15.548,"5min":28.760000000000002,"15min":28.925000000000001},"min":{"1min":12.759,"5min":8.2089999999999996,"15min":7.7599999999999998},"max":{"1min":22.408000000000001,"5min":360.99900000000002,"15min":608.44100000000003},"last":16.59}]}]},{"osd":0,"up_from":14,"seq":60129542503,"num_pgs":10,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":81108,"kb_used_data":5704,"kb_used_omap":1,"kb_used_meta":75390,"kb_avail":94290732,"statfs":{"total":96636764160,"available":96553709568,"internally_reserved":0,"allocated":5840896,"data_stored":2740727,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1593,"internal_metadata":77199815},"hb_peers":[1,2,3,4,5,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[{"osd":1,"last update":"Mon Jun 3 09:06:04 2024","interfaces":[{"interface":"back","average":{"1min":6.9960000000000004,"5min":56.997,"15min":339.29700000000003},"min":{"1min":4.4530000000000003,"5min":4.4530000000000003,"15min":4.4100000000000001},"max":{"1min":11.720000000000001,"5min":2414.4749999999999,"15min":20691.453000000001},"last":4.915},{"interface":"front","average":{"1min":9.548,"5min":57.258000000000003,"15min":337.791},"min":{"1min":5.2300000000000004,"5min":5.2300000000000004,"15min":4.6050000000000004},"max":{"1min":15.337999999999999,"5min":2412.9549999999999,"15min":20640.792000000001},"last":9.7650000000000006}]},{"osd":2,"last update":"Mon Jun 3 09:06:22 2024","interfaces":[{"interface":"back","average":{"1min":11.154,"5min":26.515999999999998,"15min":22.882000000000001},"min":{"1min":5.7869999999999999,"5min":5.766,"15min":5.766},"max":{"1min":72.269999999999996,"5min":627.40700000000004,"15min":627.40700000000004},"last":7.2930000000000001},{"interface":"front","average":{"1min":14.353999999999999,"5min":33.304000000000002,"15min":29.885000000000002},"min":{"1min":8.2149999999999999,"5min":7.8630000000000004,"15min":7.4089999999999998},"max":{"1min":73.418000000000006,"5min":635.62599999999998,"15min":635.62599999999998},"last":9.0449999999999999}]},{"osd":3,"last update":"Mon Jun 3 09:06:04 2024","interfaces":[{"interface":"back","average":{"1min":12.105,"5min":26.925000000000001,"15min":26.584},"min":{"1min":7.8410000000000002,"5min":7.1280000000000001,"15min":6.1909999999999998},"max":{"1min":18.425999999999998,"5min":389.72699999999998,"15min":389.72699999999998},"last":12.268000000000001},{"interface":"front","average":{"1min":13.794,"5min":29.771999999999998,"15min":28.815000000000001},"min":{"1min":10.084,"5min":9.4920000000000009,"15min":6.1849999999999996},"max":{"1min":17.373999999999999,"5min":390.23200000000003,"15min":494.92899999999997},"last":14.003}]},{"osd":4,"last update":"Mon Jun 3 09:06:27 2024","interfaces":[{"interface":"back","average":{"1min":11.352,"5min":745.30600000000004,"15min":363.48099999999999},"min":{"1min":6.415,"5min":6.415,"15min":6.415},"max":{"1min":16.724,"5min":20248.616000000002,"15min":25555.212},"last":10.99},{"interface":"front","average":{"1min":13.545,"5min":744.76999999999998,"15min":361.03100000000001},"min":{"1min":9.3889999999999993,"5min":8.8940000000000001,"15min":7.1429999999999998},"max":{"1min":18.411000000000001,"5min":20250.195,"15min":25556.159},"last":14.521000000000001}]},{"osd":5,"last update":"Mon Jun 3 09:06:17 2024","interfaces":[{"interface":"back","average":{"1min":9.7040000000000006,"5min":26.533999999999999,"15min":28.683},"min":{"1min":6.8129999999999997,"5min":6.8129999999999997,"15min":6.8129999999999997},"max":{"1min":14.942,"5min":455.78699999999998,"15min":605.05899999999997},"last":9.4299999999999997},{"interface":"front","average":{"1min":14.279,"5min":28.398,"15min":26.192},"min":{"1min":11.696,"5min":6.4420000000000002,"15min":6.2030000000000003},"max":{"1min":17.879000000000001,"5min":391.31200000000001,"15min":541.11000000000001},"last":15.06}]},{"osd":6,"last update":"Mon Jun 3 09:04:30 2024","interfaces":[{"interface":"back","average":{"1min":52.496000000000002,"5min":33.139000000000003,"15min":160.83199999999999},"min":{"1min":7.5289999999999999,"5min":7.1040000000000001,"15min":6.3739999999999997},"max":{"1min":457.91699999999997,"5min":457.91699999999997,"15min":11115.258},"last":18.959},{"interface":"front","average":{"1min":49.670000000000002,"5min":33.704999999999998,"15min":162.791},"min":{"1min":12.247999999999999,"5min":11.019,"15min":9.9760000000000009},"max":{"1min":407.05099999999999,"5min":407.05099999999999,"15min":11092.194},"last":17.707000000000001}]},{"osd":7,"last update":"Mon Jun 3 09:06:22 2024","interfaces":[{"interface":"back","average":{"1min":11.625999999999999,"5min":26.375,"15min":28.972999999999999},"min":{"1min":6.4480000000000004,"5min":6.4480000000000004,"15min":6.4480000000000004},"max":{"1min":15.352,"5min":461.41699999999997,"15min":598.33699999999999},"last":10.305999999999999},{"interface":"front","average":{"1min":14.013,"5min":29.696000000000002,"15min":33.630000000000003},"min":{"1min":10.938000000000001,"5min":10.58,"15min":10.263999999999999},"max":{"1min":16.603000000000002,"5min":463.10599999999999,"15min":592.80499999999995},"last":11.743}]}]}],"pool_statfs":[{"poolid":1,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":401408,"data_stored":397840,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":5,"total":0,"available":0,"internally_reserved":0,"allocated":401408,"data_stored":397840,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":4,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":5,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":6,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":17,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":17,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":17,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":17,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":17,"osd":4,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":17,"osd":5,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":5,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":17,"osd":6,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":5,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":17,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0}]}} 2024-06-03T09:06:32.505 INFO:tasks.ceph.ceph_manager.ceph:PG 17.c is not active+clean 2024-06-03T09:06:32.505 INFO:tasks.ceph.ceph_manager.ceph:{'pgid': '17.c', 'version': "0'0", 'reported_seq': 953, 'reported_epoch': 1009, 'state': 'active+undersized', 'last_fresh': '2024-06-03T09:06:13.668156+0000', 'last_change': '2024-06-03T09:06:03.440074+0000', 'last_active': '2024-06-03T09:06:13.668156+0000', 'last_peered': '2024-06-03T09:06:13.668156+0000', 'last_clean': '2024-06-03T09:05:02.105638+0000', 'last_became_active': '2024-06-03T09:06:03.440074+0000', 'last_became_peered': '2024-06-03T09:06:03.440074+0000', 'last_unstale': '2024-06-03T09:06:13.668156+0000', 'last_undegraded': '2024-06-03T09:06:13.668156+0000', 'last_fullsized': '2024-06-03T09:06:03.377982+0000', 'mapping_epoch': 999, 'log_start': "0'0", 'ondisk_log_start': "0'0", 'created': 28, 'last_epoch_clean': 255, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2024-06-03T08:37:55.614497+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2024-06-03T08:37:55.614497+0000', 'last_clean_scrub_stamp': '2024-06-03T08:37:55.614497+0000', 'objects_scrubbed': 0, 'log_size': 0, 'log_dups_size': 0, 'ondisk_log_size': 0, 'stats_invalid': False, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': 'periodic scrub scheduled @ 2024-06-04T10:34:20.808557+0000', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 0, 'num_objects': 0, 'num_object_clones': 0, 'num_object_copies': 0, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 0, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 0, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 0, 'num_write_kb': 0, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 0, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [3], 'acting': [3], 'avail_no_missing': ['3'], 'object_location_counts': [], 'blocked_by': [], 'up_primary': 3, 'acting_primary': 3, 'purged_snaps': []} 2024-06-03T09:06:32.505 INFO:tasks.ceph.ceph_manager.ceph:PG 17.f is not active+clean 2024-06-03T09:06:32.505 INFO:tasks.ceph.ceph_manager.ceph:{'pgid': '17.f', 'version': "0'0", 'reported_seq': 148, 'reported_epoch': 1009, 'state': 'active+undersized', 'last_fresh': '2024-06-03T09:06:14.450543+0000', 'last_change': '2024-06-03T09:06:03.420285+0000', 'last_active': '2024-06-03T09:06:14.450543+0000', 'last_peered': '2024-06-03T09:06:14.450543+0000', 'last_clean': '2024-06-03T08:37:55.614497+0000', 'last_became_active': '2024-06-03T09:06:03.420285+0000', 'last_became_peered': '2024-06-03T09:06:03.420285+0000', 'last_unstale': '2024-06-03T09:06:14.450543+0000', 'last_undegraded': '2024-06-03T09:06:14.450543+0000', 'last_fullsized': '2024-06-03T09:06:03.380616+0000', 'mapping_epoch': 999, 'log_start': "0'0", 'ondisk_log_start': "0'0", 'created': 28, 'last_epoch_clean': 255, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2024-06-03T08:37:55.614497+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2024-06-03T08:37:55.614497+0000', 'last_clean_scrub_stamp': '2024-06-03T08:37:55.614497+0000', 'objects_scrubbed': 0, 'log_size': 0, 'log_dups_size': 0, 'ondisk_log_size': 0, 'stats_invalid': False, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': 'periodic scrub scheduled @ 2024-06-04T15:39:41.187731+0000', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 0, 'num_objects': 0, 'num_object_clones': 0, 'num_object_copies': 0, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 0, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 0, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 0, 'num_write_kb': 0, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 0, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [4], 'acting': [4], 'avail_no_missing': ['4'], 'object_location_counts': [], 'blocked_by': [], 'up_primary': 4, 'acting_primary': 4, 'purged_snaps': []} 2024-06-03T09:06:32.505 INFO:tasks.ceph.ceph_manager.ceph:PG 17.e is not active+clean 2024-06-03T09:06:32.505 INFO:tasks.ceph.ceph_manager.ceph:{'pgid': '17.e', 'version': "0'0", 'reported_seq': 149, 'reported_epoch': 1009, 'state': 'active+undersized', 'last_fresh': '2024-06-03T09:06:13.726992+0000', 'last_change': '2024-06-03T09:06:03.412887+0000', 'last_active': '2024-06-03T09:06:13.726992+0000', 'last_peered': '2024-06-03T09:06:13.726992+0000', 'last_clean': '2024-06-03T08:37:55.614497+0000', 'last_became_active': '2024-06-03T09:06:03.412887+0000', 'last_became_peered': '2024-06-03T09:06:03.412887+0000', 'last_unstale': '2024-06-03T09:06:13.726992+0000', 'last_undegraded': '2024-06-03T09:06:13.726992+0000', 'last_fullsized': '2024-06-03T09:06:03.332482+0000', 'mapping_epoch': 999, 'log_start': "0'0", 'ondisk_log_start': "0'0", 'created': 28, 'last_epoch_clean': 255, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2024-06-03T08:37:55.614497+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2024-06-03T08:37:55.614497+0000', 'last_clean_scrub_stamp': '2024-06-03T08:37:55.614497+0000', 'objects_scrubbed': 0, 'log_size': 0, 'log_dups_size': 0, 'ondisk_log_size': 0, 'stats_invalid': False, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': 'periodic scrub scheduled @ 2024-06-04T12:23:22.511601+0000', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 0, 'num_objects': 0, 'num_object_clones': 0, 'num_object_copies': 0, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 0, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 0, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 0, 'num_write_kb': 0, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 0, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [5], 'acting': [5], 'avail_no_missing': ['5'], 'object_location_counts': [], 'blocked_by': [], 'up_primary': 5, 'acting_primary': 5, 'purged_snaps': []} 2024-06-03T09:06:32.505 INFO:tasks.ceph.ceph_manager.ceph:PG 17.8 is not active+clean 2024-06-03T09:06:32.506 INFO:tasks.ceph.ceph_manager.ceph:{'pgid': '17.8', 'version': "0'0", 'reported_seq': 961, 'reported_epoch': 1009, 'state': 'active+undersized', 'last_fresh': '2024-06-03T09:06:14.430375+0000', 'last_change': '2024-06-03T09:06:03.388786+0000', 'last_active': '2024-06-03T09:06:14.430375+0000', 'last_peered': '2024-06-03T09:06:14.430375+0000', 'last_clean': '2024-06-03T09:04:43.297780+0000', 'last_became_active': '2024-06-03T09:06:03.388786+0000', 'last_became_peered': '2024-06-03T09:06:03.388786+0000', 'last_unstale': '2024-06-03T09:06:14.430375+0000', 'last_undegraded': '2024-06-03T09:06:14.430375+0000', 'last_fullsized': '2024-06-03T09:06:03.331754+0000', 'mapping_epoch': 999, 'log_start': "0'0", 'ondisk_log_start': "0'0", 'created': 28, 'last_epoch_clean': 255, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2024-06-03T08:37:55.614497+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2024-06-03T08:37:55.614497+0000', 'last_clean_scrub_stamp': '2024-06-03T08:37:55.614497+0000', 'objects_scrubbed': 0, 'log_size': 0, 'log_dups_size': 0, 'ondisk_log_size': 0, 'stats_invalid': False, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': 'periodic scrub scheduled @ 2024-06-04T09:18:44.765547+0000', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 0, 'num_objects': 0, 'num_object_clones': 0, 'num_object_copies': 0, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 0, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 0, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 0, 'num_write_kb': 0, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 0, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [0], 'acting': [0], 'avail_no_missing': ['0'], 'object_location_counts': [], 'blocked_by': [], 'up_primary': 0, 'acting_primary': 0, 'purged_snaps': []} 2024-06-03T09:06:32.506 INFO:tasks.ceph.ceph_manager.ceph:PG 17.b is not active+clean 2024-06-03T09:06:32.506 INFO:tasks.ceph.ceph_manager.ceph:{'pgid': '17.b', 'version': "0'0", 'reported_seq': 149, 'reported_epoch': 1009, 'state': 'active+undersized', 'last_fresh': '2024-06-03T09:06:14.433735+0000', 'last_change': '2024-06-03T09:06:03.451203+0000', 'last_active': '2024-06-03T09:06:14.433735+0000', 'last_peered': '2024-06-03T09:06:14.433735+0000', 'last_clean': '2024-06-03T08:37:55.614497+0000', 'last_became_active': '2024-06-03T09:06:03.451203+0000', 'last_became_peered': '2024-06-03T09:06:03.451203+0000', 'last_unstale': '2024-06-03T09:06:14.433735+0000', 'last_undegraded': '2024-06-03T09:06:14.433735+0000', 'last_fullsized': '2024-06-03T09:06:03.375523+0000', 'mapping_epoch': 999, 'log_start': "0'0", 'ondisk_log_start': "0'0", 'created': 28, 'last_epoch_clean': 255, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2024-06-03T08:37:55.614497+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2024-06-03T08:37:55.614497+0000', 'last_clean_scrub_stamp': '2024-06-03T08:37:55.614497+0000', 'objects_scrubbed': 0, 'log_size': 0, 'log_dups_size': 0, 'ondisk_log_size': 0, 'stats_invalid': False, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': 'periodic scrub scheduled @ 2024-06-04T10:16:19.813495+0000', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 0, 'num_objects': 0, 'num_object_clones': 0, 'num_object_copies': 0, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 0, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 0, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 0, 'num_write_kb': 0, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 0, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [2], 'acting': [2], 'avail_no_missing': ['2'], 'object_location_counts': [], 'blocked_by': [], 'up_primary': 2, 'acting_primary': 2, 'purged_snaps': []} 2024-06-03T09:06:32.506 INFO:tasks.ceph.ceph_manager.ceph:PG 17.4 is not active+clean 2024-06-03T09:06:32.506 INFO:tasks.ceph.ceph_manager.ceph:{'pgid': '17.4', 'version': "0'0", 'reported_seq': 149, 'reported_epoch': 1009, 'state': 'active+undersized', 'last_fresh': '2024-06-03T09:06:13.941522+0000', 'last_change': '2024-06-03T09:06:03.441889+0000', 'last_active': '2024-06-03T09:06:13.941522+0000', 'last_peered': '2024-06-03T09:06:13.941522+0000', 'last_clean': '2024-06-03T08:37:55.614497+0000', 'last_became_active': '2024-06-03T09:06:03.441889+0000', 'last_became_peered': '2024-06-03T09:06:03.441889+0000', 'last_unstale': '2024-06-03T09:06:13.941522+0000', 'last_undegraded': '2024-06-03T09:06:13.941522+0000', 'last_fullsized': '2024-06-03T09:06:03.359582+0000', 'mapping_epoch': 999, 'log_start': "0'0", 'ondisk_log_start': "0'0", 'created': 28, 'last_epoch_clean': 255, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2024-06-03T08:37:55.614497+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2024-06-03T08:37:55.614497+0000', 'last_clean_scrub_stamp': '2024-06-03T08:37:55.614497+0000', 'objects_scrubbed': 0, 'log_size': 0, 'log_dups_size': 0, 'ondisk_log_size': 0, 'stats_invalid': False, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': 'periodic scrub scheduled @ 2024-06-04T12:49:22.907141+0000', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 0, 'num_objects': 0, 'num_object_clones': 0, 'num_object_copies': 0, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 0, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 0, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 0, 'num_write_kb': 0, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 0, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [7], 'acting': [7], 'avail_no_missing': ['7'], 'object_location_counts': [], 'blocked_by': [], 'up_primary': 7, 'acting_primary': 7, 'purged_snaps': []} 2024-06-03T09:06:32.506 INFO:tasks.ceph.ceph_manager.ceph:PG 17.6 is not active+clean 2024-06-03T09:06:32.506 INFO:tasks.ceph.ceph_manager.ceph:{'pgid': '17.6', 'version': "0'0", 'reported_seq': 960, 'reported_epoch': 1009, 'state': 'active+undersized', 'last_fresh': '2024-06-03T09:06:14.453110+0000', 'last_change': '2024-06-03T09:06:03.428677+0000', 'last_active': '2024-06-03T09:06:14.453110+0000', 'last_peered': '2024-06-03T09:06:14.453110+0000', 'last_clean': '2024-06-03T09:04:43.312352+0000', 'last_became_active': '2024-06-03T09:06:03.428677+0000', 'last_became_peered': '2024-06-03T09:06:03.428677+0000', 'last_unstale': '2024-06-03T09:06:14.453110+0000', 'last_undegraded': '2024-06-03T09:06:14.453110+0000', 'last_fullsized': '2024-06-03T09:06:03.359909+0000', 'mapping_epoch': 999, 'log_start': "0'0", 'ondisk_log_start': "0'0", 'created': 28, 'last_epoch_clean': 255, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2024-06-03T08:37:55.614497+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2024-06-03T08:37:55.614497+0000', 'last_clean_scrub_stamp': '2024-06-03T08:37:55.614497+0000', 'objects_scrubbed': 0, 'log_size': 0, 'log_dups_size': 0, 'ondisk_log_size': 0, 'stats_invalid': False, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': 'periodic scrub scheduled @ 2024-06-04T15:38:02.180241+0000', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 0, 'num_objects': 0, 'num_object_clones': 0, 'num_object_copies': 0, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 0, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 0, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 0, 'num_write_kb': 0, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 0, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [0], 'acting': [0], 'avail_no_missing': ['0'], 'object_location_counts': [], 'blocked_by': [], 'up_primary': 0, 'acting_primary': 0, 'purged_snaps': []} 2024-06-03T09:06:32.506 INFO:tasks.ceph.ceph_manager.ceph:PG 17.16 is not active+clean 2024-06-03T09:06:32.506 INFO:tasks.ceph.ceph_manager.ceph:{'pgid': '17.16', 'version': "0'0", 'reported_seq': 972, 'reported_epoch': 1009, 'state': 'active+undersized', 'last_fresh': '2024-06-03T09:06:14.441665+0000', 'last_change': '2024-06-03T09:06:03.478615+0000', 'last_active': '2024-06-03T09:06:14.441665+0000', 'last_peered': '2024-06-03T09:06:14.441665+0000', 'last_clean': '2024-06-03T09:04:43.328415+0000', 'last_became_active': '2024-06-03T09:06:03.478615+0000', 'last_became_peered': '2024-06-03T09:06:03.478615+0000', 'last_unstale': '2024-06-03T09:06:14.441665+0000', 'last_undegraded': '2024-06-03T09:06:14.441665+0000', 'last_fullsized': '2024-06-03T09:06:03.402504+0000', 'mapping_epoch': 999, 'log_start': "0'0", 'ondisk_log_start': "0'0", 'created': 28, 'last_epoch_clean': 255, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2024-06-03T08:37:55.614497+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2024-06-03T08:37:55.614497+0000', 'last_clean_scrub_stamp': '2024-06-03T08:37:55.614497+0000', 'objects_scrubbed': 0, 'log_size': 0, 'log_dups_size': 0, 'ondisk_log_size': 0, 'stats_invalid': False, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': 'periodic scrub scheduled @ 2024-06-04T08:47:08.079233+0000', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 0, 'num_objects': 0, 'num_object_clones': 0, 'num_object_copies': 0, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 0, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 0, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 0, 'num_write_kb': 0, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 0, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [2], 'acting': [2], 'avail_no_missing': ['2'], 'object_location_counts': [], 'blocked_by': [], 'up_primary': 2, 'acting_primary': 2, 'purged_snaps': []} 2024-06-03T09:06:32.507 INFO:tasks.ceph.ceph_manager.ceph:PG 2.4 is not active+clean 2024-06-03T09:06:32.507 INFO:tasks.ceph.ceph_manager.ceph:{'pgid': '2.4', 'version': "0'0", 'reported_seq': 952, 'reported_epoch': 1010, 'state': 'active+undersized', 'last_fresh': '2024-06-03T09:06:18.331485+0000', 'last_change': '2024-06-03T09:06:03.387171+0000', 'last_active': '2024-06-03T09:06:18.331485+0000', 'last_peered': '2024-06-03T09:06:18.331485+0000', 'last_clean': '2024-06-03T09:04:43.202645+0000', 'last_became_active': '2024-06-03T09:06:03.387171+0000', 'last_became_peered': '2024-06-03T09:06:03.387171+0000', 'last_unstale': '2024-06-03T09:06:18.331485+0000', 'last_undegraded': '2024-06-03T09:06:18.331485+0000', 'last_fullsized': '2024-06-03T09:06:03.283787+0000', 'mapping_epoch': 999, 'log_start': "0'0", 'ondisk_log_start': "0'0", 'created': 23, 'last_epoch_clean': 255, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2024-06-03T08:36:28.444764+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2024-06-03T08:36:28.444764+0000', 'last_clean_scrub_stamp': '2024-06-03T08:36:28.444764+0000', 'objects_scrubbed': 0, 'log_size': 0, 'log_dups_size': 0, 'ondisk_log_size': 0, 'stats_invalid': False, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': 'periodic scrub scheduled @ 2024-06-04T20:17:43.485023+0000', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 0, 'num_objects': 0, 'num_object_clones': 0, 'num_object_copies': 0, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 0, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 0, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 0, 'num_write_kb': 0, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 0, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [5], 'acting': [5], 'avail_no_missing': ['5'], 'object_location_counts': [], 'blocked_by': [], 'up_primary': 5, 'acting_primary': 5, 'purged_snaps': []} 2024-06-03T09:06:32.507 INFO:tasks.ceph.ceph_manager.ceph:PG 2.2 is not active+clean 2024-06-03T09:06:32.507 INFO:tasks.ceph.ceph_manager.ceph:{'pgid': '2.2', 'version': "0'0", 'reported_seq': 963, 'reported_epoch': 1009, 'state': 'active+undersized', 'last_fresh': '2024-06-03T09:06:13.657892+0000', 'last_change': '2024-06-03T09:06:03.410020+0000', 'last_active': '2024-06-03T09:06:13.657892+0000', 'last_peered': '2024-06-03T09:06:13.657892+0000', 'last_clean': '2024-06-03T09:05:37.145889+0000', 'last_became_active': '2024-06-03T09:06:03.410020+0000', 'last_became_peered': '2024-06-03T09:06:03.410020+0000', 'last_unstale': '2024-06-03T09:06:13.657892+0000', 'last_undegraded': '2024-06-03T09:06:13.657892+0000', 'last_fullsized': '2024-06-03T09:06:03.346864+0000', 'mapping_epoch': 999, 'log_start': "0'0", 'ondisk_log_start': "0'0", 'created': 23, 'last_epoch_clean': 255, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2024-06-03T08:36:28.444764+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2024-06-03T08:36:28.444764+0000', 'last_clean_scrub_stamp': '2024-06-03T08:36:28.444764+0000', 'objects_scrubbed': 0, 'log_size': 0, 'log_dups_size': 0, 'ondisk_log_size': 0, 'stats_invalid': False, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': 'periodic scrub scheduled @ 2024-06-04T09:57:12.423360+0000', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 0, 'num_objects': 0, 'num_object_clones': 0, 'num_object_copies': 0, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 0, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 0, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 0, 'num_write_kb': 0, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 0, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [3], 'acting': [3], 'avail_no_missing': ['3'], 'object_location_counts': [], 'blocked_by': [], 'up_primary': 3, 'acting_primary': 3, 'purged_snaps': []} 2024-06-03T09:06:32.507 INFO:tasks.ceph.ceph_manager.ceph:PG 17.1a is not active+clean 2024-06-03T09:06:32.507 INFO:tasks.ceph.ceph_manager.ceph:{'pgid': '17.1a', 'version': "0'0", 'reported_seq': 960, 'reported_epoch': 1009, 'state': 'active+undersized', 'last_fresh': '2024-06-03T09:06:14.431224+0000', 'last_change': '2024-06-03T09:06:03.390734+0000', 'last_active': '2024-06-03T09:06:14.431224+0000', 'last_peered': '2024-06-03T09:06:14.431224+0000', 'last_clean': '2024-06-03T09:05:20.322313+0000', 'last_became_active': '2024-06-03T09:06:03.390734+0000', 'last_became_peered': '2024-06-03T09:06:03.390734+0000', 'last_unstale': '2024-06-03T09:06:14.431224+0000', 'last_undegraded': '2024-06-03T09:06:14.431224+0000', 'last_fullsized': '2024-06-03T09:06:03.343197+0000', 'mapping_epoch': 999, 'log_start': "0'0", 'ondisk_log_start': "0'0", 'created': 28, 'last_epoch_clean': 254, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2024-06-03T08:37:55.614497+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2024-06-03T08:37:55.614497+0000', 'last_clean_scrub_stamp': '2024-06-03T08:37:55.614497+0000', 'objects_scrubbed': 0, 'log_size': 0, 'log_dups_size': 0, 'ondisk_log_size': 0, 'stats_invalid': False, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': 'periodic scrub scheduled @ 2024-06-04T15:56:47.628875+0000', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 0, 'num_objects': 0, 'num_object_clones': 0, 'num_object_copies': 0, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 0, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 0, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 0, 'num_write_kb': 0, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 0, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [4], 'acting': [4], 'avail_no_missing': ['4'], 'object_location_counts': [], 'blocked_by': [], 'up_primary': 4, 'acting_primary': 4, 'purged_snaps': []} 2024-06-03T09:06:32.507 INFO:tasks.ceph.ceph_manager.ceph:PG 17.1e is not active+clean 2024-06-03T09:06:32.507 INFO:tasks.ceph.ceph_manager.ceph:{'pgid': '17.1e', 'version': "0'0", 'reported_seq': 148, 'reported_epoch': 1009, 'state': 'active+undersized', 'last_fresh': '2024-06-03T09:06:14.445917+0000', 'last_change': '2024-06-03T09:06:03.406181+0000', 'last_active': '2024-06-03T09:06:14.445917+0000', 'last_peered': '2024-06-03T09:06:14.445917+0000', 'last_clean': '2024-06-03T08:37:55.614497+0000', 'last_became_active': '2024-06-03T09:06:03.406181+0000', 'last_became_peered': '2024-06-03T09:06:03.406181+0000', 'last_unstale': '2024-06-03T09:06:14.445917+0000', 'last_undegraded': '2024-06-03T09:06:14.445917+0000', 'last_fullsized': '2024-06-03T09:06:03.363739+0000', 'mapping_epoch': 999, 'log_start': "0'0", 'ondisk_log_start': "0'0", 'created': 28, 'last_epoch_clean': 255, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2024-06-03T08:37:55.614497+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2024-06-03T08:37:55.614497+0000', 'last_clean_scrub_stamp': '2024-06-03T08:37:55.614497+0000', 'objects_scrubbed': 0, 'log_size': 0, 'log_dups_size': 0, 'ondisk_log_size': 0, 'stats_invalid': False, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': 'periodic scrub scheduled @ 2024-06-04T11:45:24.486600+0000', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 0, 'num_objects': 0, 'num_object_clones': 0, 'num_object_copies': 0, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 0, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 0, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 0, 'num_write_kb': 0, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 0, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [4], 'acting': [4], 'avail_no_missing': ['4'], 'object_location_counts': [], 'blocked_by': [], 'up_primary': 4, 'acting_primary': 4, 'purged_snaps': []} 2024-06-03T09:06:32.507 INFO:tasks.ceph.ceph_manager.ceph:PG 17.1c is not active+clean 2024-06-03T09:06:32.507 INFO:tasks.ceph.ceph_manager.ceph:{'pgid': '17.1c', 'version': "0'0", 'reported_seq': 944, 'reported_epoch': 1010, 'state': 'active+undersized', 'last_fresh': '2024-06-03T09:06:18.327492+0000', 'last_change': '2024-06-03T09:06:03.382210+0000', 'last_active': '2024-06-03T09:06:18.327492+0000', 'last_peered': '2024-06-03T09:06:18.327492+0000', 'last_clean': '2024-06-03T09:04:43.199660+0000', 'last_became_active': '2024-06-03T09:06:03.382210+0000', 'last_became_peered': '2024-06-03T09:06:03.382210+0000', 'last_unstale': '2024-06-03T09:06:18.327492+0000', 'last_undegraded': '2024-06-03T09:06:18.327492+0000', 'last_fullsized': '2024-06-03T09:06:03.277874+0000', 'mapping_epoch': 999, 'log_start': "0'0", 'ondisk_log_start': "0'0", 'created': 28, 'last_epoch_clean': 255, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2024-06-03T08:37:55.614497+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2024-06-03T08:37:55.614497+0000', 'last_clean_scrub_stamp': '2024-06-03T08:37:55.614497+0000', 'objects_scrubbed': 0, 'log_size': 0, 'log_dups_size': 0, 'ondisk_log_size': 0, 'stats_invalid': False, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': 'periodic scrub scheduled @ 2024-06-04T12:18:50.013884+0000', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 0, 'num_objects': 0, 'num_object_clones': 0, 'num_object_copies': 0, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 0, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 0, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 0, 'num_write_kb': 0, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 0, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [5], 'acting': [5], 'avail_no_missing': ['5'], 'object_location_counts': [], 'blocked_by': [], 'up_primary': 5, 'acting_primary': 5, 'purged_snaps': []} 2024-06-03T09:06:32.507 INFO:tasks.ceph.ceph_manager.ceph:PG 17.1 is not active+clean 2024-06-03T09:06:32.508 INFO:tasks.ceph.ceph_manager.ceph:{'pgid': '17.1', 'version': "65'7", 'reported_seq': 959, 'reported_epoch': 1009, 'state': 'active+undersized+degraded', 'last_fresh': '2024-06-03T09:06:13.709209+0000', 'last_change': '2024-06-03T09:06:03.370660+0000', 'last_active': '2024-06-03T09:06:13.709209+0000', 'last_peered': '2024-06-03T09:06:13.709209+0000', 'last_clean': '2024-06-03T09:05:03.733864+0000', 'last_became_active': '2024-06-03T09:06:03.370660+0000', 'last_became_peered': '2024-06-03T09:06:03.370660+0000', 'last_unstale': '2024-06-03T09:06:13.709209+0000', 'last_undegraded': '2024-06-03T09:06:03.266076+0000', 'last_fullsized': '2024-06-03T09:06:03.264525+0000', 'mapping_epoch': 999, 'log_start': "0'0", 'ondisk_log_start': "0'0", 'created': 28, 'last_epoch_clean': 255, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2024-06-03T08:37:55.614497+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2024-06-03T08:37:55.614497+0000', 'last_clean_scrub_stamp': '2024-06-03T08:37:55.614497+0000', 'objects_scrubbed': 0, 'log_size': 7, 'log_dups_size': 0, 'ondisk_log_size': 7, 'stats_invalid': False, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': 'periodic scrub scheduled @ 2024-06-04T11:55:18.194283+0000', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 5, 'num_objects': 1, 'num_object_clones': 0, 'num_object_copies': 2, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 1, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 1, 'num_whiteouts': 0, 'num_read': 6, 'num_read_kb': 6, 'num_write': 7, 'num_write_kb': 7, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 0, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [5], 'acting': [5], 'avail_no_missing': ['5'], 'object_location_counts': [{'shards': '5', 'objects': 1}], 'blocked_by': [], 'up_primary': 5, 'acting_primary': 5, 'purged_snaps': []} 2024-06-03T09:06:32.508 DEBUG:teuthology.orchestra.run.smithi088:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph status --format=json 2024-06-03T09:06:32.605 INFO:teuthology.orchestra.run.smithi088.stderr:2024-06-03T09:06:32.605+0000 7f800062d640 10 monclient: get_monmap_and_config 2024-06-03T09:06:32.605 INFO:teuthology.orchestra.run.smithi088.stderr:2024-06-03T09:06:32.605+0000 7f800062d640 10 monclient: build_initial_monmap 2024-06-03T09:06:32.605 INFO:teuthology.orchestra.run.smithi088.stderr:2024-06-03T09:06:32.605+0000 7f800062d640 10 monclient: monmap: 2024-06-03T09:06:32.605 INFO:teuthology.orchestra.run.smithi088.stderr:epoch 0 2024-06-03T09:06:32.605 INFO:teuthology.orchestra.run.smithi088.stderr:fsid ec3c3fdd-3ac6-4d7d-9412-683130ed8b04 2024-06-03T09:06:32.605 INFO:teuthology.orchestra.run.smithi088.stderr:last_changed 2024-06-03T09:06:32.606158+0000 2024-06-03T09:06:32.606 INFO:teuthology.orchestra.run.smithi088.stderr:created 2024-06-03T09:06:32.606158+0000 2024-06-03T09:06:32.606 INFO:teuthology.orchestra.run.smithi088.stderr:min_mon_release 0 (unknown) 2024-06-03T09:06:32.606 INFO:teuthology.orchestra.run.smithi088.stderr:election_strategy: 3 2024-06-03T09:06:32.606 INFO:teuthology.orchestra.run.smithi088.stderr:0: v1:172.21.15.88:6789/0 mon.noname-a 2024-06-03T09:06:32.606 INFO:teuthology.orchestra.run.smithi088.stderr:1: v1:172.21.15.192:6789/0 mon.noname-c 2024-06-03T09:06:32.606 INFO:teuthology.orchestra.run.smithi088.stderr:2: v1:172.21.15.88:6790/0 mon.noname-b 2024-06-03T09:06:32.606 INFO:teuthology.orchestra.run.smithi088.stderr: 2024-06-03T09:06:32.607 INFO:teuthology.orchestra.run.smithi088.stderr:2024-06-03T09:06:32.605+0000 7f800062d640 1 Processor -- start 2024-06-03T09:06:32.607 INFO:teuthology.orchestra.run.smithi088.stderr:2024-06-03T09:06:32.605+0000 7f800062d640 1 -- start start 2024-06-03T09:06:32.607 INFO:teuthology.orchestra.run.smithi088.stderr:2024-06-03T09:06:32.605+0000 7f800062d640 10 monclient: init 2024-06-03T09:06:32.608 INFO:teuthology.orchestra.run.smithi088.stderr:2024-06-03T09:06:32.606+0000 7f800062d640 10 monclient: _reopen_session rank -1 2024-06-03T09:06:32.608 INFO:teuthology.orchestra.run.smithi088.stderr:2024-06-03T09:06:32.606+0000 7f800062d640 10 monclient: _add_conns ranks=[0,2,1] 2024-06-03T09:06:32.608 INFO:teuthology.orchestra.run.smithi088.stderr:2024-06-03T09:06:32.606+0000 7f800062d640 10 monclient(hunting): picked mon.noname-a con 0x7f7ff80a87f0 addr v1:172.21.15.88:6789/0 2024-06-03T09:06:32.608 INFO:teuthology.orchestra.run.smithi088.stderr:2024-06-03T09:06:32.606+0000 7f800062d640 10 monclient(hunting): picked mon.noname-b con 0x7f7ff80a8bd0 addr v1:172.21.15.88:6790/0 2024-06-03T09:06:32.608 INFO:teuthology.orchestra.run.smithi088.stderr:2024-06-03T09:06:32.606+0000 7f800062d640 10 monclient(hunting): picked mon.noname-c con 0x7f7ff81414c0 addr v1:172.21.15.192:6789/0 2024-06-03T09:06:32.608 INFO:teuthology.orchestra.run.smithi088.stderr:2024-06-03T09:06:32.606+0000 7f800062d640 1 -- --> v1:172.21.15.88:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f7ff80aa350 con 0x7f7ff80a87f0 2024-06-03T09:06:32.608 INFO:teuthology.orchestra.run.smithi088.stderr:2024-06-03T09:06:32.606+0000 7f800062d640 1 -- --> v1:172.21.15.192:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f7ff80aa5c0 con 0x7f7ff81414c0 2024-06-03T09:06:32.608 INFO:teuthology.orchestra.run.smithi088.stderr:2024-06-03T09:06:32.606+0000 7f800062d640 1 -- --> v1:172.21.15.88:6790/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f7ff8141950 con 0x7f7ff80a8bd0 2024-06-03T09:06:32.608 INFO:teuthology.orchestra.run.smithi088.stderr:2024-06-03T09:06:32.606+0000 7f800062d640 10 monclient(hunting): _renew_subs 2024-06-03T09:06:32.608 INFO:teuthology.orchestra.run.smithi088.stderr:2024-06-03T09:06:32.606+0000 7f800062d640 10 monclient(hunting): authenticate will time out at 3048.416560s 2024-06-03T09:06:32.609 INFO:teuthology.orchestra.run.smithi088.stderr:2024-06-03T09:06:32.607+0000 7f7ffe3c9640 1 --1- >> v1:172.21.15.88:6789/0 conn(0x7f7ff80a87f0 0x7f7ff80b1be0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.88:6789/0 says I am v1:172.21.15.88:41576/0 (socket says 172.21.15.88:41576) 2024-06-03T09:06:32.609 INFO:teuthology.orchestra.run.smithi088.stderr:2024-06-03T09:06:32.607+0000 7f7ffe3c9640 1 -- 172.21.15.88:0/1763688704 learned_addr learned my addr 172.21.15.88:0/1763688704 (peer_addr_for_me v1:172.21.15.88:0/0) 2024-06-03T09:06:32.609 INFO:teuthology.orchestra.run.smithi088.stderr:2024-06-03T09:06:32.609+0000 7f7ffd3c7640 1 -- 172.21.15.88:0/1763688704 <== mon.0 v1:172.21.15.88:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 3787687573 0 0) 0x7f7ff80aa350 con 0x7f7ff80a87f0 2024-06-03T09:06:32.610 INFO:teuthology.orchestra.run.smithi088.stderr:2024-06-03T09:06:32.609+0000 7f7ffd3c7640 10 monclient(hunting): _negotiate 2024-06-03T09:06:32.610 INFO:teuthology.orchestra.run.smithi088.stderr:2024-06-03T09:06:32.609+0000 7f7ffd3c7640 10 monclient(hunting): _init_auth method 2 2024-06-03T09:06:32.610 INFO:teuthology.orchestra.run.smithi088.stderr:2024-06-03T09:06:32.609+0000 7f7ffd3c7640 10 monclient(hunting): _init_auth creating new auth 2024-06-03T09:06:32.610 INFO:teuthology.orchestra.run.smithi088.stderr:2024-06-03T09:06:32.609+0000 7f7ffd3c7640 10 monclient(hunting): my global_id is 30636 2024-06-03T09:06:32.610 INFO:teuthology.orchestra.run.smithi088.stderr:2024-06-03T09:06:32.609+0000 7f7ffd3c7640 1 -- 172.21.15.88:0/1763688704 --> v1:172.21.15.88:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f7fe0003520 con 0x7f7ff80a87f0 2024-06-03T09:06:32.610 INFO:teuthology.orchestra.run.smithi088.stderr:2024-06-03T09:06:32.609+0000 7f7ffd3c7640 1 -- 172.21.15.88:0/1763688704 <== mon.2 v1:172.21.15.88:6790/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 2374965752 0 0) 0x7f7ff8141950 con 0x7f7ff80a8bd0 2024-06-03T09:06:32.610 INFO:teuthology.orchestra.run.smithi088.stderr:2024-06-03T09:06:32.609+0000 7f7ffd3c7640 10 monclient(hunting): _negotiate 2024-06-03T09:06:32.610 INFO:teuthology.orchestra.run.smithi088.stderr:2024-06-03T09:06:32.609+0000 7f7ffd3c7640 10 monclient(hunting): _init_auth method 2 2024-06-03T09:06:32.611 INFO:teuthology.orchestra.run.smithi088.stderr:2024-06-03T09:06:32.609+0000 7f7ffd3c7640 10 monclient(hunting): _init_auth creating new auth 2024-06-03T09:06:32.611 INFO:teuthology.orchestra.run.smithi088.stderr:2024-06-03T09:06:32.609+0000 7f7ffd3c7640 10 monclient(hunting): my global_id is 30140 2024-06-03T09:06:32.611 INFO:teuthology.orchestra.run.smithi088.stderr:2024-06-03T09:06:32.609+0000 7f7ffd3c7640 1 -- 172.21.15.88:0/1763688704 --> v1:172.21.15.88:6790/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f7ff80aa350 con 0x7f7ff80a8bd0 2024-06-03T09:06:32.611 INFO:teuthology.orchestra.run.smithi088.stderr:2024-06-03T09:06:32.609+0000 7f7ffd3c7640 1 -- 172.21.15.88:0/1763688704 <== mon.1 v1:172.21.15.192:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 669907107 0 0) 0x7f7ff80aa5c0 con 0x7f7ff81414c0 2024-06-03T09:06:32.611 INFO:teuthology.orchestra.run.smithi088.stderr:2024-06-03T09:06:32.609+0000 7f7ffd3c7640 10 monclient(hunting): _negotiate 2024-06-03T09:06:32.611 INFO:teuthology.orchestra.run.smithi088.stderr:2024-06-03T09:06:32.609+0000 7f7ffd3c7640 10 monclient(hunting): _init_auth method 2 2024-06-03T09:06:32.611 INFO:teuthology.orchestra.run.smithi088.stderr:2024-06-03T09:06:32.609+0000 7f7ffd3c7640 10 monclient(hunting): _init_auth creating new auth 2024-06-03T09:06:32.611 INFO:teuthology.orchestra.run.smithi088.stderr:2024-06-03T09:06:32.609+0000 7f7ffd3c7640 10 monclient(hunting): my global_id is 30217 2024-06-03T09:06:32.611 INFO:teuthology.orchestra.run.smithi088.stderr:2024-06-03T09:06:32.609+0000 7f7ffd3c7640 1 -- 172.21.15.88:0/1763688704 --> v1:172.21.15.192:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f7ff8141950 con 0x7f7ff81414c0 2024-06-03T09:06:32.613 INFO:teuthology.orchestra.run.smithi088.stderr:2024-06-03T09:06:32.612+0000 7f7ffd3c7640 1 -- 172.21.15.88:0/1763688704 <== mon.2 v1:172.21.15.88:6790/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 214+0+0 (unknown 836600167 0 0) 0x7f7ff80aa350 con 0x7f7ff80a8bd0 2024-06-03T09:06:32.613 INFO:teuthology.orchestra.run.smithi088.stderr:2024-06-03T09:06:32.612+0000 7f7ffd3c7640 1 -- 172.21.15.88:0/1763688704 >> v1:172.21.15.192:6789/0 conn(0x7f7ff81414c0 legacy=0x7f7ff813b1b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-06-03T09:06:32.613 INFO:teuthology.orchestra.run.smithi088.stderr:2024-06-03T09:06:32.612+0000 7f7ffd3c7640 1 -- 172.21.15.88:0/1763688704 >> v1:172.21.15.88:6789/0 conn(0x7f7ff80a87f0 legacy=0x7f7ff80b1be0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-06-03T09:06:32.613 INFO:teuthology.orchestra.run.smithi088.stderr:2024-06-03T09:06:32.612+0000 7f7ffd3c7640 10 monclient: _finish_hunting 0 2024-06-03T09:06:32.613 INFO:teuthology.orchestra.run.smithi088.stderr:2024-06-03T09:06:32.612+0000 7f7ffd3c7640 1 monclient: found mon.noname-b 2024-06-03T09:06:32.613 INFO:teuthology.orchestra.run.smithi088.stderr:2024-06-03T09:06:32.612+0000 7f7ffd3c7640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-06-03T09:06:32.613 INFO:teuthology.orchestra.run.smithi088.stderr:2024-06-03T09:06:32.612+0000 7f7ffd3c7640 10 monclient: _send_mon_message to mon.noname-b at v1:172.21.15.88:6790/0 2024-06-03T09:06:32.613 INFO:teuthology.orchestra.run.smithi088.stderr:2024-06-03T09:06:32.612+0000 7f7ffd3c7640 1 -- 172.21.15.88:0/1763688704 --> v1:172.21.15.88:6790/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f7ff813b8c0 con 0x7f7ff80a8bd0 2024-06-03T09:06:32.613 INFO:teuthology.orchestra.run.smithi088.stderr:2024-06-03T09:06:32.612+0000 7f7ffd3c7640 10 monclient: _finish_auth 0 2024-06-03T09:06:32.614 INFO:teuthology.orchestra.run.smithi088.stderr:2024-06-03T09:06:32.612+0000 7f7ffd3c7640 10 monclient: _check_auth_tickets 2024-06-03T09:06:32.614 INFO:teuthology.orchestra.run.smithi088.stderr:2024-06-03T09:06:32.612+0000 7f7ffd3c7640 20 monclient: _check_auth_rotating not needed by client.admin 2024-06-03T09:06:32.614 INFO:teuthology.orchestra.run.smithi088.stderr:2024-06-03T09:06:32.612+0000 7f7ffd3c7640 1 -- 172.21.15.88:0/1763688704 <== mon.2 v1:172.21.15.88:6790/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 2234374065 0 0) 0x7f7fec0015f0 con 0x7f7ff80a8bd0 2024-06-03T09:06:32.614 INFO:teuthology.orchestra.run.smithi088.stderr:2024-06-03T09:06:32.612+0000 7f7ffd3c7640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-06-03T09:06:32.614 INFO:teuthology.orchestra.run.smithi088.stderr:2024-06-03T09:06:32.612+0000 7f7ffd3c7640 10 monclient: got monmap 1 from mon.noname-b (according to old e1) 2024-06-03T09:06:32.614 INFO:teuthology.orchestra.run.smithi088.stderr:2024-06-03T09:06:32.612+0000 7f7ffd3c7640 10 monclient: dump: 2024-06-03T09:06:32.614 INFO:teuthology.orchestra.run.smithi088.stderr:epoch 1 2024-06-03T09:06:32.614 INFO:teuthology.orchestra.run.smithi088.stderr:fsid ec3c3fdd-3ac6-4d7d-9412-683130ed8b04 2024-06-03T09:06:32.615 INFO:teuthology.orchestra.run.smithi088.stderr:last_changed 2024-06-03T08:34:31.885125+0000 2024-06-03T09:06:32.615 INFO:teuthology.orchestra.run.smithi088.stderr:created 2024-06-03T08:34:31.885125+0000 2024-06-03T09:06:32.615 INFO:teuthology.orchestra.run.smithi088.stderr:min_mon_release 18 (reef) 2024-06-03T09:06:32.615 INFO:teuthology.orchestra.run.smithi088.stderr:election_strategy: 3 2024-06-03T09:06:32.615 INFO:teuthology.orchestra.run.smithi088.stderr:0: v1:172.21.15.88:6789/0 mon.a 2024-06-03T09:06:32.615 INFO:teuthology.orchestra.run.smithi088.stderr:1: v1:172.21.15.192:6789/0 mon.b 2024-06-03T09:06:32.615 INFO:teuthology.orchestra.run.smithi088.stderr:2: v1:172.21.15.88:6790/0 mon.c 2024-06-03T09:06:32.615 INFO:teuthology.orchestra.run.smithi088.stderr: 2024-06-03T09:06:32.615 INFO:teuthology.orchestra.run.smithi088.stderr:2024-06-03T09:06:32.612+0000 7f800062d640 5 monclient: authenticate success, global_id 30140 2024-06-03T09:06:32.615 INFO:teuthology.orchestra.run.smithi088.stderr:2024-06-03T09:06:32.612+0000 7f800062d640 20 monclient: get_monmap_and_config waiting for monmap|config 2024-06-03T09:06:32.618 INFO:teuthology.orchestra.run.smithi088.stderr:2024-06-03T09:06:32.614+0000 7f7ffd3c7640 1 -- 172.21.15.88:0/1763688704 <== mon.2 v1:172.21.15.88:6790/0 4 ==== config(0 keys) v1 ==== 4+0+0 (unknown 0 0 0) 0x7f7fec001810 con 0x7f7ff80a8bd0 2024-06-03T09:06:32.618 INFO:teuthology.orchestra.run.smithi088.stderr:2024-06-03T09:06:32.614+0000 7f7ffd3c7640 10 monclient: handle_config config(0 keys) v1 2024-06-03T09:06:32.618 INFO:teuthology.orchestra.run.smithi088.stderr:2024-06-03T09:06:32.614+0000 7f7ffd3c7640 1 -- 172.21.15.88:0/1763688704 <== mon.2 v1:172.21.15.88:6790/0 5 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 2234374065 0 0) 0x7f7fec001be0 con 0x7f7ff80a8bd0 2024-06-03T09:06:32.618 INFO:teuthology.orchestra.run.smithi088.stderr:2024-06-03T09:06:32.614+0000 7f7ffd3c7640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-06-03T09:06:32.618 INFO:teuthology.orchestra.run.smithi088.stderr:2024-06-03T09:06:32.614+0000 7f7ffd3c7640 10 monclient: got monmap 1 from mon.c (according to old e1) 2024-06-03T09:06:32.618 INFO:teuthology.orchestra.run.smithi088.stderr:2024-06-03T09:06:32.614+0000 7f7ffd3c7640 10 monclient: dump: 2024-06-03T09:06:32.618 INFO:teuthology.orchestra.run.smithi088.stderr:epoch 1 2024-06-03T09:06:32.618 INFO:teuthology.orchestra.run.smithi088.stderr:fsid ec3c3fdd-3ac6-4d7d-9412-683130ed8b04 2024-06-03T09:06:32.619 INFO:teuthology.orchestra.run.smithi088.stderr:last_changed 2024-06-03T08:34:31.885125+0000 2024-06-03T09:06:32.619 INFO:teuthology.orchestra.run.smithi088.stderr:created 2024-06-03T08:34:31.885125+0000 2024-06-03T09:06:32.619 INFO:teuthology.orchestra.run.smithi088.stderr:min_mon_release 18 (reef) 2024-06-03T09:06:32.619 INFO:teuthology.orchestra.run.smithi088.stderr:election_strategy: 3 2024-06-03T09:06:32.619 INFO:teuthology.orchestra.run.smithi088.stderr:0: v1:172.21.15.88:6789/0 mon.a 2024-06-03T09:06:32.619 INFO:teuthology.orchestra.run.smithi088.stderr:1: v1:172.21.15.192:6789/0 mon.b 2024-06-03T09:06:32.619 INFO:teuthology.orchestra.run.smithi088.stderr:2: v1:172.21.15.88:6790/0 mon.c 2024-06-03T09:06:32.619 INFO:teuthology.orchestra.run.smithi088.stderr: 2024-06-03T09:06:32.619 INFO:teuthology.orchestra.run.smithi088.stderr:2024-06-03T09:06:32.614+0000 7f800062d640 10 monclient: get_monmap_and_config success 2024-06-03T09:06:32.619 INFO:teuthology.orchestra.run.smithi088.stderr:2024-06-03T09:06:32.614+0000 7f800062d640 4 set_mon_vals no callback set 2024-06-03T09:06:32.619 INFO:teuthology.orchestra.run.smithi088.stderr:2024-06-03T09:06:32.615+0000 7f800062d640 10 monclient: shutdown 2024-06-03T09:06:32.619 INFO:teuthology.orchestra.run.smithi088.stderr:2024-06-03T09:06:32.615+0000 7f800062d640 20 monclient: shutdown discarding 0 pending message(s) 2024-06-03T09:06:32.619 INFO:teuthology.orchestra.run.smithi088.stderr:2024-06-03T09:06:32.615+0000 7f800062d640 1 -- 172.21.15.88:0/1763688704 >> v1:172.21.15.88:6790/0 conn(0x7f7ff80a8bd0 legacy=0x7f7ff8141e70 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-06-03T09:06:32.619 INFO:teuthology.orchestra.run.smithi088.stderr:2024-06-03T09:06:32.615+0000 7f800062d640 1 -- 172.21.15.88:0/1763688704 shutdown_connections 2024-06-03T09:06:32.620 INFO:teuthology.orchestra.run.smithi088.stderr:2024-06-03T09:06:32.615+0000 7f800062d640 1 -- 172.21.15.88:0/1763688704 >> 172.21.15.88:0/1763688704 conn(0x7f7ff80b71e0 msgr2=0x7f7ff80b75c0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-06-03T09:06:32.620 INFO:teuthology.orchestra.run.smithi088.stderr:2024-06-03T09:06:32.615+0000 7f800062d640 1 -- 172.21.15.88:0/1763688704 shutdown_connections 2024-06-03T09:06:32.620 INFO:teuthology.orchestra.run.smithi088.stderr:2024-06-03T09:06:32.615+0000 7f800062d640 1 -- 172.21.15.88:0/1763688704 wait complete. 2024-06-03T09:06:32.620 INFO:teuthology.orchestra.run.smithi088.stderr:2024-06-03T09:06:32.616+0000 7f800062d640 10 monclient: build_initial_monmap 2024-06-03T09:06:32.620 INFO:teuthology.orchestra.run.smithi088.stderr:2024-06-03T09:06:32.616+0000 7f800062d640 10 monclient: monmap: 2024-06-03T09:06:32.620 INFO:teuthology.orchestra.run.smithi088.stderr:epoch 0 2024-06-03T09:06:32.620 INFO:teuthology.orchestra.run.smithi088.stderr:fsid 00000000-0000-0000-0000-000000000000 2024-06-03T09:06:32.620 INFO:teuthology.orchestra.run.smithi088.stderr:last_changed 0.000000 2024-06-03T09:06:32.620 INFO:teuthology.orchestra.run.smithi088.stderr:created 0.000000 2024-06-03T09:06:32.620 INFO:teuthology.orchestra.run.smithi088.stderr:min_mon_release 0 (unknown) 2024-06-03T09:06:32.620 INFO:teuthology.orchestra.run.smithi088.stderr:election_strategy: 1 2024-06-03T09:06:32.620 INFO:teuthology.orchestra.run.smithi088.stderr:0: v1:172.21.15.88:6789/0 mon.noname-a 2024-06-03T09:06:32.620 INFO:teuthology.orchestra.run.smithi088.stderr:1: v1:172.21.15.192:6789/0 mon.noname-b 2024-06-03T09:06:32.620 INFO:teuthology.orchestra.run.smithi088.stderr:2: v1:172.21.15.88:6790/0 mon.noname-c 2024-06-03T09:06:32.621 INFO:teuthology.orchestra.run.smithi088.stderr: 2024-06-03T09:06:32.621 INFO:teuthology.orchestra.run.smithi088.stderr:2024-06-03T09:06:32.616+0000 7f800062d640 1 librados: starting msgr at 2024-06-03T09:06:32.621 INFO:teuthology.orchestra.run.smithi088.stderr:2024-06-03T09:06:32.616+0000 7f800062d640 1 librados: starting objecter 2024-06-03T09:06:32.621 INFO:teuthology.orchestra.run.smithi088.stderr:2024-06-03T09:06:32.616+0000 7f800062d640 1 Processor -- start 2024-06-03T09:06:32.621 INFO:teuthology.orchestra.run.smithi088.stderr:2024-06-03T09:06:32.616+0000 7f800062d640 1 -- start start 2024-06-03T09:06:32.621 INFO:teuthology.orchestra.run.smithi088.stderr:2024-06-03T09:06:32.616+0000 7f800062d640 1 librados: setting wanted keys 2024-06-03T09:06:32.621 INFO:teuthology.orchestra.run.smithi088.stderr:2024-06-03T09:06:32.616+0000 7f800062d640 1 librados: calling monclient init 2024-06-03T09:06:32.621 INFO:teuthology.orchestra.run.smithi088.stderr:2024-06-03T09:06:32.616+0000 7f800062d640 10 monclient: init 2024-06-03T09:06:32.621 INFO:teuthology.orchestra.run.smithi088.stderr:2024-06-03T09:06:32.617+0000 7f800062d640 10 monclient: _reopen_session rank -1 2024-06-03T09:06:32.621 INFO:teuthology.orchestra.run.smithi088.stderr:2024-06-03T09:06:32.617+0000 7f800062d640 10 monclient: _add_conns ranks=[1,0,2] 2024-06-03T09:06:32.621 INFO:teuthology.orchestra.run.smithi088.stderr:2024-06-03T09:06:32.617+0000 7f800062d640 10 monclient(hunting): picked mon.noname-b con 0x7f7ff80a87f0 addr v1:172.21.15.192:6789/0 2024-06-03T09:06:32.621 INFO:teuthology.orchestra.run.smithi088.stderr:2024-06-03T09:06:32.617+0000 7f800062d640 10 monclient(hunting): picked mon.noname-a con 0x7f7ff8144410 addr v1:172.21.15.88:6789/0 2024-06-03T09:06:32.622 INFO:teuthology.orchestra.run.smithi088.stderr:2024-06-03T09:06:32.617+0000 7f800062d640 10 monclient(hunting): picked mon.noname-c con 0x7f7ff8143970 addr v1:172.21.15.88:6790/0 2024-06-03T09:06:32.622 INFO:teuthology.orchestra.run.smithi088.stderr:2024-06-03T09:06:32.617+0000 7f800062d640 1 -- --> v1:172.21.15.88:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f7ff8130e50 con 0x7f7ff8144410 2024-06-03T09:06:32.622 INFO:teuthology.orchestra.run.smithi088.stderr:2024-06-03T09:06:32.617+0000 7f800062d640 1 -- --> v1:172.21.15.192:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f7ff812ecd0 con 0x7f7ff80a87f0 2024-06-03T09:06:32.622 INFO:teuthology.orchestra.run.smithi088.stderr:2024-06-03T09:06:32.617+0000 7f800062d640 1 -- --> v1:172.21.15.88:6790/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f7ff8130280 con 0x7f7ff8143970 2024-06-03T09:06:32.622 INFO:teuthology.orchestra.run.smithi088.stderr:2024-06-03T09:06:32.617+0000 7f800062d640 10 monclient(hunting): _renew_subs 2024-06-03T09:06:32.622 INFO:teuthology.orchestra.run.smithi088.stderr:2024-06-03T09:06:32.617+0000 7f800062d640 10 monclient(hunting): authenticate will time out at 3048.427234s 2024-06-03T09:06:32.622 INFO:teuthology.orchestra.run.smithi088.stderr:2024-06-03T09:06:32.617+0000 7f7ffdbc8640 1 --1- >> v1:172.21.15.88:6789/0 conn(0x7f7ff8144410 0x7f7ff812deb0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=1).handle_server_banner_and_identify peer v1:172.21.15.88:6789/0 says I am v1:172.21.15.88:41584/0 (socket says 172.21.15.88:41584) 2024-06-03T09:06:32.622 INFO:teuthology.orchestra.run.smithi088.stderr:2024-06-03T09:06:32.617+0000 7f7ffdbc8640 1 -- 172.21.15.88:0/2418070986 learned_addr learned my addr 172.21.15.88:0/2418070986 (peer_addr_for_me v1:172.21.15.88:0/0) 2024-06-03T09:06:32.622 INFO:teuthology.orchestra.run.smithi088.stderr:2024-06-03T09:06:32.618+0000 7f7ff6ffd640 10 client.?.objecter ms_handle_connect 0x7f7ff8144410 2024-06-03T09:06:32.622 INFO:teuthology.orchestra.run.smithi088.stderr:2024-06-03T09:06:32.618+0000 7f7ff6ffd640 10 client.?.objecter resend_mon_ops 2024-06-03T09:06:32.622 INFO:teuthology.orchestra.run.smithi088.stderr:2024-06-03T09:06:32.618+0000 7f7ff6ffd640 10 client.?.objecter ms_handle_connect 0x7f7ff80a87f0 2024-06-03T09:06:32.622 INFO:teuthology.orchestra.run.smithi088.stderr:2024-06-03T09:06:32.618+0000 7f7ff6ffd640 10 client.?.objecter resend_mon_ops 2024-06-03T09:06:32.622 INFO:teuthology.orchestra.run.smithi088.stderr:2024-06-03T09:06:32.618+0000 7f7ff6ffd640 10 client.?.objecter ms_handle_connect 0x7f7ff8143970 2024-06-03T09:06:32.623 INFO:teuthology.orchestra.run.smithi088.stderr:2024-06-03T09:06:32.618+0000 7f7ff6ffd640 10 client.?.objecter resend_mon_ops 2024-06-03T09:06:32.624 INFO:teuthology.orchestra.run.smithi088.stderr:2024-06-03T09:06:32.619+0000 7f7ff6ffd640 1 -- 172.21.15.88:0/2418070986 <== mon.1 v1:172.21.15.192:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 2351630793 0 0) 0x7f7ff812ecd0 con 0x7f7ff80a87f0 2024-06-03T09:06:32.624 INFO:teuthology.orchestra.run.smithi088.stderr:2024-06-03T09:06:32.619+0000 7f7ff6ffd640 10 monclient(hunting): _negotiate 2024-06-03T09:06:32.624 INFO:teuthology.orchestra.run.smithi088.stderr:2024-06-03T09:06:32.619+0000 7f7ff6ffd640 10 monclient(hunting): _init_auth method 2 2024-06-03T09:06:32.624 INFO:teuthology.orchestra.run.smithi088.stderr:2024-06-03T09:06:32.619+0000 7f7ff6ffd640 10 monclient(hunting): _init_auth creating new auth 2024-06-03T09:06:32.624 INFO:teuthology.orchestra.run.smithi088.stderr:2024-06-03T09:06:32.619+0000 7f7ff6ffd640 10 monclient(hunting): my global_id is 30220 2024-06-03T09:06:32.624 INFO:teuthology.orchestra.run.smithi088.stderr:2024-06-03T09:06:32.619+0000 7f7ff6ffd640 1 -- 172.21.15.88:0/2418070986 --> v1:172.21.15.192:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f7fe4004eb0 con 0x7f7ff80a87f0 2024-06-03T09:06:32.624 INFO:teuthology.orchestra.run.smithi088.stderr:2024-06-03T09:06:32.620+0000 7f7ff6ffd640 1 -- 172.21.15.88:0/2418070986 <== mon.2 v1:172.21.15.88:6790/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 1975894346 0 0) 0x7f7ff8130280 con 0x7f7ff8143970 2024-06-03T09:06:32.624 INFO:teuthology.orchestra.run.smithi088.stderr:2024-06-03T09:06:32.620+0000 7f7ff6ffd640 10 monclient(hunting): _negotiate 2024-06-03T09:06:32.624 INFO:teuthology.orchestra.run.smithi088.stderr:2024-06-03T09:06:32.620+0000 7f7ff6ffd640 10 monclient(hunting): _init_auth method 2 2024-06-03T09:06:32.625 INFO:teuthology.orchestra.run.smithi088.stderr:2024-06-03T09:06:32.620+0000 7f7ff6ffd640 10 monclient(hunting): _init_auth creating new auth 2024-06-03T09:06:32.625 INFO:teuthology.orchestra.run.smithi088.stderr:2024-06-03T09:06:32.620+0000 7f7ff6ffd640 10 monclient(hunting): my global_id is 30143 2024-06-03T09:06:32.625 INFO:teuthology.orchestra.run.smithi088.stderr:2024-06-03T09:06:32.620+0000 7f7ff6ffd640 1 -- 172.21.15.88:0/2418070986 --> v1:172.21.15.88:6790/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f7ff812ecd0 con 0x7f7ff8143970 2024-06-03T09:06:32.625 INFO:teuthology.orchestra.run.smithi088.stderr:2024-06-03T09:06:32.620+0000 7f7ff6ffd640 1 -- 172.21.15.88:0/2418070986 <== mon.0 v1:172.21.15.88:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 3079656716 0 0) 0x7f7ff8130e50 con 0x7f7ff8144410 2024-06-03T09:06:32.625 INFO:teuthology.orchestra.run.smithi088.stderr:2024-06-03T09:06:32.620+0000 7f7ff6ffd640 10 monclient(hunting): _negotiate 2024-06-03T09:06:32.625 INFO:teuthology.orchestra.run.smithi088.stderr:2024-06-03T09:06:32.620+0000 7f7ff6ffd640 10 monclient(hunting): _init_auth method 2 2024-06-03T09:06:32.625 INFO:teuthology.orchestra.run.smithi088.stderr:2024-06-03T09:06:32.620+0000 7f7ff6ffd640 10 monclient(hunting): _init_auth creating new auth 2024-06-03T09:06:32.625 INFO:teuthology.orchestra.run.smithi088.stderr:2024-06-03T09:06:32.620+0000 7f7ff6ffd640 10 monclient(hunting): my global_id is 30639 2024-06-03T09:06:32.625 INFO:teuthology.orchestra.run.smithi088.stderr:2024-06-03T09:06:32.620+0000 7f7ff6ffd640 1 -- 172.21.15.88:0/2418070986 --> v1:172.21.15.88:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f7ff8130280 con 0x7f7ff8144410 2024-06-03T09:06:32.625 INFO:teuthology.orchestra.run.smithi088.stderr:2024-06-03T09:06:32.624+0000 7f7ff6ffd640 1 -- 172.21.15.88:0/2418070986 <== mon.1 v1:172.21.15.192:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 764+0+0 (unknown 400516288 0 0) 0x7f7fe4004eb0 con 0x7f7ff80a87f0 2024-06-03T09:06:32.625 INFO:teuthology.orchestra.run.smithi088.stderr:2024-06-03T09:06:32.624+0000 7f7ff6ffd640 1 -- 172.21.15.88:0/2418070986 --> v1:172.21.15.192:6789/0 -- auth(proto 2 165 bytes epoch 0) v1 -- 0x7f7ff8130e50 con 0x7f7ff80a87f0 2024-06-03T09:06:32.625 INFO:teuthology.orchestra.run.smithi088.stderr:2024-06-03T09:06:32.624+0000 7f7ff6ffd640 1 -- 172.21.15.88:0/2418070986 <== mon.1 v1:172.21.15.192:6789/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 2234374065 0 0) 0x7f7fe8001b70 con 0x7f7ff80a87f0 2024-06-03T09:06:32.625 INFO:teuthology.orchestra.run.smithi088.stderr:2024-06-03T09:06:32.624+0000 7f7ff6ffd640 10 monclient(hunting): handle_monmap mon_map magic: 0 v1 2024-06-03T09:06:32.626 INFO:teuthology.orchestra.run.smithi088.stderr:2024-06-03T09:06:32.624+0000 7f7ff6ffd640 10 monclient(hunting): got monmap 1 from mon.noname-b (according to old e1) 2024-06-03T09:06:32.626 INFO:teuthology.orchestra.run.smithi088.stderr:2024-06-03T09:06:32.624+0000 7f7ff6ffd640 10 monclient(hunting): dump: 2024-06-03T09:06:32.626 INFO:teuthology.orchestra.run.smithi088.stderr:epoch 1 2024-06-03T09:06:32.626 INFO:teuthology.orchestra.run.smithi088.stderr:fsid ec3c3fdd-3ac6-4d7d-9412-683130ed8b04 2024-06-03T09:06:32.626 INFO:teuthology.orchestra.run.smithi088.stderr:last_changed 2024-06-03T08:34:31.885125+0000 2024-06-03T09:06:32.626 INFO:teuthology.orchestra.run.smithi088.stderr:created 2024-06-03T08:34:31.885125+0000 2024-06-03T09:06:32.626 INFO:teuthology.orchestra.run.smithi088.stderr:min_mon_release 18 (reef) 2024-06-03T09:06:32.626 INFO:teuthology.orchestra.run.smithi088.stderr:election_strategy: 3 2024-06-03T09:06:32.626 INFO:teuthology.orchestra.run.smithi088.stderr:0: v1:172.21.15.88:6789/0 mon.a 2024-06-03T09:06:32.626 INFO:teuthology.orchestra.run.smithi088.stderr:1: v1:172.21.15.192:6789/0 mon.b 2024-06-03T09:06:32.626 INFO:teuthology.orchestra.run.smithi088.stderr:2: v1:172.21.15.88:6790/0 mon.c 2024-06-03T09:06:32.626 INFO:teuthology.orchestra.run.smithi088.stderr: 2024-06-03T09:06:32.627 INFO:teuthology.orchestra.run.smithi088.stderr:2024-06-03T09:06:32.624+0000 7f7ff6ffd640 10 monclient(hunting): _finish_auth 0 2024-06-03T09:06:32.628 INFO:teuthology.orchestra.run.smithi088.stderr:2024-06-03T09:06:32.624+0000 7f7ff6ffd640 1 -- 172.21.15.88:0/2418070986 <== mon.2 v1:172.21.15.88:6790/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 764+0+0 (unknown 1244657484 0 0) 0x7f7ff812ecd0 con 0x7f7ff8143970 2024-06-03T09:06:32.629 INFO:teuthology.orchestra.run.smithi088.stderr:2024-06-03T09:06:32.624+0000 7f7ff6ffd640 1 -- 172.21.15.88:0/2418070986 --> v1:172.21.15.88:6790/0 -- auth(proto 2 165 bytes epoch 0) v1 -- 0x7f7fe4004eb0 con 0x7f7ff8143970 2024-06-03T09:06:32.629 INFO:teuthology.orchestra.run.smithi088.stderr:2024-06-03T09:06:32.624+0000 7f7ff6ffd640 1 -- 172.21.15.88:0/2418070986 <== mon.2 v1:172.21.15.88:6790/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 2234374065 0 0) 0x7f7ff0001830 con 0x7f7ff8143970 2024-06-03T09:06:32.629 INFO:teuthology.orchestra.run.smithi088.stderr:2024-06-03T09:06:32.624+0000 7f7ff6ffd640 10 monclient(hunting): handle_monmap mon_map magic: 0 v1 2024-06-03T09:06:32.629 INFO:teuthology.orchestra.run.smithi088.stderr:2024-06-03T09:06:32.624+0000 7f7ff6ffd640 10 monclient(hunting): got monmap 1 from mon.c (according to old e1) 2024-06-03T09:06:32.629 INFO:teuthology.orchestra.run.smithi088.stderr:2024-06-03T09:06:32.624+0000 7f7ff6ffd640 10 monclient(hunting): dump: 2024-06-03T09:06:32.629 INFO:teuthology.orchestra.run.smithi088.stderr:epoch 1 2024-06-03T09:06:32.629 INFO:teuthology.orchestra.run.smithi088.stderr:fsid ec3c3fdd-3ac6-4d7d-9412-683130ed8b04 2024-06-03T09:06:32.629 INFO:teuthology.orchestra.run.smithi088.stderr:last_changed 2024-06-03T08:34:31.885125+0000 2024-06-03T09:06:32.629 INFO:teuthology.orchestra.run.smithi088.stderr:created 2024-06-03T08:34:31.885125+0000 2024-06-03T09:06:32.629 INFO:teuthology.orchestra.run.smithi088.stderr:min_mon_release 18 (reef) 2024-06-03T09:06:32.630 INFO:teuthology.orchestra.run.smithi088.stderr:election_strategy: 3 2024-06-03T09:06:32.630 INFO:teuthology.orchestra.run.smithi088.stderr:0: v1:172.21.15.88:6789/0 mon.a 2024-06-03T09:06:32.630 INFO:teuthology.orchestra.run.smithi088.stderr:1: v1:172.21.15.192:6789/0 mon.b 2024-06-03T09:06:32.630 INFO:teuthology.orchestra.run.smithi088.stderr:2: v1:172.21.15.88:6790/0 mon.c 2024-06-03T09:06:32.630 INFO:teuthology.orchestra.run.smithi088.stderr: 2024-06-03T09:06:32.630 INFO:teuthology.orchestra.run.smithi088.stderr:2024-06-03T09:06:32.625+0000 7f7ff6ffd640 1 -- 172.21.15.88:0/2418070986 <== mon.0 v1:172.21.15.88:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 764+0+0 (unknown 2190952245 0 0) 0x7f7ff8130280 con 0x7f7ff8144410 2024-06-03T09:06:32.630 INFO:teuthology.orchestra.run.smithi088.stderr:2024-06-03T09:06:32.625+0000 7f7ff6ffd640 1 -- 172.21.15.88:0/2418070986 --> v1:172.21.15.88:6789/0 -- auth(proto 2 165 bytes epoch 0) v1 -- 0x7f7ff812ecd0 con 0x7f7ff8144410 2024-06-03T09:06:32.630 INFO:teuthology.orchestra.run.smithi088.stderr:2024-06-03T09:06:32.625+0000 7f7ff6ffd640 1 -- 172.21.15.88:0/2418070986 <== mon.0 v1:172.21.15.88:6789/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 2234374065 0 0) 0x7f7fec0015b0 con 0x7f7ff8144410 2024-06-03T09:06:32.630 INFO:teuthology.orchestra.run.smithi088.stderr:2024-06-03T09:06:32.625+0000 7f7ff6ffd640 10 monclient(hunting): handle_monmap mon_map magic: 0 v1 2024-06-03T09:06:32.630 INFO:teuthology.orchestra.run.smithi088.stderr:2024-06-03T09:06:32.625+0000 7f7ff6ffd640 10 monclient(hunting): got monmap 1 from mon.a (according to old e1) 2024-06-03T09:06:32.630 INFO:teuthology.orchestra.run.smithi088.stderr:2024-06-03T09:06:32.625+0000 7f7ff6ffd640 10 monclient(hunting): dump: 2024-06-03T09:06:32.630 INFO:teuthology.orchestra.run.smithi088.stderr:epoch 1 2024-06-03T09:06:32.631 INFO:teuthology.orchestra.run.smithi088.stderr:fsid ec3c3fdd-3ac6-4d7d-9412-683130ed8b04 2024-06-03T09:06:32.631 INFO:teuthology.orchestra.run.smithi088.stderr:last_changed 2024-06-03T08:34:31.885125+0000 2024-06-03T09:06:32.631 INFO:teuthology.orchestra.run.smithi088.stderr:created 2024-06-03T08:34:31.885125+0000 2024-06-03T09:06:32.631 INFO:teuthology.orchestra.run.smithi088.stderr:min_mon_release 18 (reef) 2024-06-03T09:06:32.631 INFO:teuthology.orchestra.run.smithi088.stderr:election_strategy: 3 2024-06-03T09:06:32.631 INFO:teuthology.orchestra.run.smithi088.stderr:0: v1:172.21.15.88:6789/0 mon.a 2024-06-03T09:06:32.631 INFO:teuthology.orchestra.run.smithi088.stderr:1: v1:172.21.15.192:6789/0 mon.b 2024-06-03T09:06:32.631 INFO:teuthology.orchestra.run.smithi088.stderr:2: v1:172.21.15.88:6790/0 mon.c 2024-06-03T09:06:32.631 INFO:teuthology.orchestra.run.smithi088.stderr: 2024-06-03T09:06:32.631 INFO:teuthology.orchestra.run.smithi088.stderr:2024-06-03T09:06:32.627+0000 7f7ff6ffd640 1 -- 172.21.15.88:0/2418070986 <== mon.2 v1:172.21.15.88:6790/0 4 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 580+0+0 (unknown 2061019112 0 0) 0x7f7fe4004eb0 con 0x7f7ff8143970 2024-06-03T09:06:32.631 INFO:teuthology.orchestra.run.smithi088.stderr:2024-06-03T09:06:32.627+0000 7f7ff6ffd640 1 -- 172.21.15.88:0/2418070986 >> v1:172.21.15.192:6789/0 conn(0x7f7ff80a87f0 legacy=0x7f7ff81d6360 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-06-03T09:06:32.631 INFO:teuthology.orchestra.run.smithi088.stderr:2024-06-03T09:06:32.627+0000 7f7ff6ffd640 1 -- 172.21.15.88:0/2418070986 >> v1:172.21.15.88:6789/0 conn(0x7f7ff8144410 legacy=0x7f7ff812deb0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-06-03T09:06:32.631 INFO:teuthology.orchestra.run.smithi088.stderr:2024-06-03T09:06:32.627+0000 7f7ff6ffd640 10 monclient: _finish_hunting 0 2024-06-03T09:06:32.632 INFO:teuthology.orchestra.run.smithi088.stderr:2024-06-03T09:06:32.627+0000 7f7ff6ffd640 1 monclient: found mon.c 2024-06-03T09:06:32.632 INFO:teuthology.orchestra.run.smithi088.stderr:2024-06-03T09:06:32.627+0000 7f7ff6ffd640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-06-03T09:06:32.632 INFO:teuthology.orchestra.run.smithi088.stderr:2024-06-03T09:06:32.627+0000 7f7ff6ffd640 10 monclient: _send_mon_message to mon.c at v1:172.21.15.88:6790/0 2024-06-03T09:06:32.632 INFO:teuthology.orchestra.run.smithi088.stderr:2024-06-03T09:06:32.627+0000 7f7ff6ffd640 1 -- 172.21.15.88:0/2418070986 --> v1:172.21.15.88:6790/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f7ff8130410 con 0x7f7ff8143970 2024-06-03T09:06:32.632 INFO:teuthology.orchestra.run.smithi088.stderr:2024-06-03T09:06:32.627+0000 7f7ff6ffd640 10 monclient: _finish_auth 0 2024-06-03T09:06:32.632 INFO:teuthology.orchestra.run.smithi088.stderr:2024-06-03T09:06:32.627+0000 7f7ff6ffd640 10 monclient: _check_auth_tickets 2024-06-03T09:06:32.632 INFO:teuthology.orchestra.run.smithi088.stderr:2024-06-03T09:06:32.627+0000 7f7ff6ffd640 20 monclient: _check_auth_rotating not needed by client.admin 2024-06-03T09:06:32.632 INFO:teuthology.orchestra.run.smithi088.stderr:2024-06-03T09:06:32.627+0000 7f800062d640 5 monclient: authenticate success, global_id 30143 2024-06-03T09:06:32.632 INFO:teuthology.orchestra.run.smithi088.stderr:2024-06-03T09:06:32.627+0000 7f800062d640 10 monclient: _renew_subs 2024-06-03T09:06:32.632 INFO:teuthology.orchestra.run.smithi088.stderr:2024-06-03T09:06:32.627+0000 7f800062d640 10 monclient: _send_mon_message to mon.c at v1:172.21.15.88:6790/0 2024-06-03T09:06:32.632 INFO:teuthology.orchestra.run.smithi088.stderr:2024-06-03T09:06:32.627+0000 7f800062d640 1 -- 172.21.15.88:0/2418070986 --> v1:172.21.15.88:6790/0 -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f7ff81d6a70 con 0x7f7ff8143970 2024-06-03T09:06:32.632 INFO:teuthology.orchestra.run.smithi088.stderr:2024-06-03T09:06:32.627+0000 7f800062d640 10 client.30143.objecter _maybe_request_map subscribing (onetime) to next osd map 2024-06-03T09:06:32.632 INFO:teuthology.orchestra.run.smithi088.stderr:2024-06-03T09:06:32.627+0000 7f800062d640 10 monclient: _renew_subs 2024-06-03T09:06:32.632 INFO:teuthology.orchestra.run.smithi088.stderr:2024-06-03T09:06:32.627+0000 7f800062d640 10 monclient: _send_mon_message to mon.c at v1:172.21.15.88:6790/0 2024-06-03T09:06:32.633 INFO:teuthology.orchestra.run.smithi088.stderr:2024-06-03T09:06:32.627+0000 7f800062d640 1 -- 172.21.15.88:0/2418070986 --> v1:172.21.15.88:6790/0 -- mon_subscribe({osdmap=0}) v3 -- 0x7f7ff81d6f00 con 0x7f7ff8143970 2024-06-03T09:06:32.633 INFO:teuthology.orchestra.run.smithi088.stderr:2024-06-03T09:06:32.627+0000 7f800062d640 1 librados: init done 2024-06-03T09:06:32.633 INFO:teuthology.orchestra.run.smithi088.stderr:2024-06-03T09:06:32.628+0000 7f800062d640 10 start_mon_command cmd=[{"prefix": "get_command_descriptions"}] 2024-06-03T09:06:32.633 INFO:teuthology.orchestra.run.smithi088.stderr:2024-06-03T09:06:32.628+0000 7f800062d640 10 monclient: _send_command 1 [{"prefix": "get_command_descriptions"}] 2024-06-03T09:06:32.633 INFO:teuthology.orchestra.run.smithi088.stderr:2024-06-03T09:06:32.628+0000 7f800062d640 10 monclient: _send_mon_message to mon.c at v1:172.21.15.88:6790/0 2024-06-03T09:06:32.633 INFO:teuthology.orchestra.run.smithi088.stderr:2024-06-03T09:06:32.628+0000 7f800062d640 1 -- 172.21.15.88:0/2418070986 --> v1:172.21.15.88:6790/0 -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f7ff8130600 con 0x7f7ff8143970 2024-06-03T09:06:32.869 INFO:teuthology.orchestra.run.smithi088.stderr:2024-06-03T09:06:32.869+0000 7f7ff6ffd640 1 -- 172.21.15.88:0/2418070986 <== mon.2 v1:172.21.15.88:6790/0 5 ==== config(0 keys) v1 ==== 4+0+0 (unknown 0 0 0) 0x7f7ff0001d00 con 0x7f7ff8143970 2024-06-03T09:06:32.869 INFO:teuthology.orchestra.run.smithi088.stderr:2024-06-03T09:06:32.869+0000 7f7ff6ffd640 10 monclient: handle_config config(0 keys) v1 2024-06-03T09:06:32.870 INFO:teuthology.orchestra.run.smithi088.stderr:2024-06-03T09:06:32.869+0000 7f7ff6ffd640 1 -- 172.21.15.88:0/2418070986 <== mon.2 v1:172.21.15.88:6790/0 6 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 2234374065 0 0) 0x7f7ff0003790 con 0x7f7ff8143970 2024-06-03T09:06:32.870 INFO:teuthology.orchestra.run.smithi088.stderr:2024-06-03T09:06:32.869+0000 7f7ff6ffd640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-06-03T09:06:32.870 INFO:teuthology.orchestra.run.smithi088.stderr:2024-06-03T09:06:32.869+0000 7f7ff6ffd640 10 monclient: got monmap 1 from mon.c (according to old e1) 2024-06-03T09:06:32.870 INFO:teuthology.orchestra.run.smithi088.stderr:2024-06-03T09:06:32.869+0000 7f7ff6ffd640 10 monclient: dump: 2024-06-03T09:06:32.870 INFO:teuthology.orchestra.run.smithi088.stderr:epoch 1 2024-06-03T09:06:32.870 INFO:teuthology.orchestra.run.smithi088.stderr:fsid ec3c3fdd-3ac6-4d7d-9412-683130ed8b04 2024-06-03T09:06:32.870 INFO:teuthology.orchestra.run.smithi088.stderr:last_changed 2024-06-03T08:34:31.885125+0000 2024-06-03T09:06:32.870 INFO:teuthology.orchestra.run.smithi088.stderr:created 2024-06-03T08:34:31.885125+0000 2024-06-03T09:06:32.870 INFO:teuthology.orchestra.run.smithi088.stderr:min_mon_release 18 (reef) 2024-06-03T09:06:32.870 INFO:teuthology.orchestra.run.smithi088.stderr:election_strategy: 3 2024-06-03T09:06:32.870 INFO:teuthology.orchestra.run.smithi088.stderr:0: v1:172.21.15.88:6789/0 mon.a 2024-06-03T09:06:32.870 INFO:teuthology.orchestra.run.smithi088.stderr:1: v1:172.21.15.192:6789/0 mon.b 2024-06-03T09:06:32.870 INFO:teuthology.orchestra.run.smithi088.stderr:2: v1:172.21.15.88:6790/0 mon.c 2024-06-03T09:06:32.871 INFO:teuthology.orchestra.run.smithi088.stderr: 2024-06-03T09:06:32.871 INFO:teuthology.orchestra.run.smithi088.stderr:2024-06-03T09:06:32.869+0000 7f7ffcbc6640 4 set_mon_vals no callback set 2024-06-03T09:06:32.871 INFO:teuthology.orchestra.run.smithi088.stderr:2024-06-03T09:06:32.870+0000 7f7ff6ffd640 1 -- 172.21.15.88:0/2418070986 <== mon.2 v1:172.21.15.88:6790/0 7 ==== mgrmap(e 5) v1 ==== 89110+0+0 (unknown 873533973 0 0) 0x7f7ff0019510 con 0x7f7ff8143970 2024-06-03T09:06:32.871 INFO:teuthology.orchestra.run.smithi088.stderr:2024-06-03T09:06:32.870+0000 7f7ffe3c9640 10 monclient: get_auth_request con 0x7f7fe406d3d0 auth_method 0 2024-06-03T09:06:32.871 INFO:teuthology.orchestra.run.smithi088.stderr:2024-06-03T09:06:32.871+0000 7f7ff6ffd640 10 client.30143.objecter ms_handle_connect 0x7f7fe406d3d0 2024-06-03T09:06:32.877 INFO:teuthology.orchestra.run.smithi088.stderr:2024-06-03T09:06:32.877+0000 7f7ff6ffd640 1 -- 172.21.15.88:0/2418070986 <== mon.2 v1:172.21.15.88:6790/0 8 ==== osd_map(1010..1010 src has 622..1010) v4 ==== 5229+0+0 (unknown 1105883993 0 0) 0x7f7ff0052d40 con 0x7f7ff8143970 2024-06-03T09:06:32.877 INFO:teuthology.orchestra.run.smithi088.stderr:2024-06-03T09:06:32.877+0000 7f7ff6ffd640 10 client.30143.objecter ms_dispatch 0x7f7ff8000b90 osd_map(1010..1010 src has 622..1010) v4 2024-06-03T09:06:32.877 INFO:teuthology.orchestra.run.smithi088.stderr:2024-06-03T09:06:32.877+0000 7f7ff6ffd640 3 client.30143.objecter handle_osd_map got epochs [1010,1010] > 0 2024-06-03T09:06:32.877 INFO:teuthology.orchestra.run.smithi088.stderr:2024-06-03T09:06:32.877+0000 7f7ff6ffd640 3 client.30143.objecter handle_osd_map decoding full epoch 1010 2024-06-03T09:06:32.877 INFO:teuthology.orchestra.run.smithi088.stderr:2024-06-03T09:06:32.877+0000 7f7ff6ffd640 20 client.30143.objecter dump_active .. 0 homeless 2024-06-03T09:06:32.877 INFO:teuthology.orchestra.run.smithi088.stderr:2024-06-03T09:06:32.877+0000 7f7ff6ffd640 1 -- 172.21.15.88:0/2418070986 <== mon.2 v1:172.21.15.88:6790/0 9 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+182498 (unknown 1092875540 0 3016760700) 0x7f7ff004e8f0 con 0x7f7ff8143970 2024-06-03T09:06:32.877 INFO:teuthology.orchestra.run.smithi088.stderr:2024-06-03T09:06:32.877+0000 7f7ff6ffd640 10 monclient: handle_mon_command_ack 1 [{"prefix": "get_command_descriptions"}] 2024-06-03T09:06:32.877 INFO:teuthology.orchestra.run.smithi088.stderr:2024-06-03T09:06:32.877+0000 7f7ff6ffd640 10 monclient: _finish_command 1 = system:0 2024-06-03T09:06:33.124 INFO:teuthology.orchestra.run.smithi088.stderr:2024-06-03T09:06:33.124+0000 7f800062d640 10 start_mon_command cmd=[{"prefix": "status", "format": "json"}] 2024-06-03T09:06:33.124 INFO:teuthology.orchestra.run.smithi088.stderr:2024-06-03T09:06:33.124+0000 7f800062d640 10 monclient: _send_command 2 [{"prefix": "status", "format": "json"}] 2024-06-03T09:06:33.124 INFO:teuthology.orchestra.run.smithi088.stderr:2024-06-03T09:06:33.124+0000 7f800062d640 10 monclient: _send_mon_message to mon.c at v1:172.21.15.88:6790/0 2024-06-03T09:06:33.124 INFO:teuthology.orchestra.run.smithi088.stderr:2024-06-03T09:06:33.124+0000 7f800062d640 1 -- 172.21.15.88:0/2418070986 --> v1:172.21.15.88:6790/0 -- mon_command({"prefix": "status", "format": "json"} v 0) v1 -- 0x7f7ff804ea60 con 0x7f7ff8143970 2024-06-03T09:06:33.140 INFO:teuthology.orchestra.run.smithi088.stderr:2024-06-03T09:06:33.140+0000 7f7ff6ffd640 1 -- 172.21.15.88:0/2418070986 <== mon.2 v1:172.21.15.88:6790/0 10 ==== mon_command_ack([{"prefix": "status", "format": "json"}]=0 v0) v1 ==== 72+0+1431 (unknown 3226119948 0 746728493) 0x7f7ff0050820 con 0x7f7ff8143970 2024-06-03T09:06:33.140 INFO:teuthology.orchestra.run.smithi088.stderr:2024-06-03T09:06:33.140+0000 7f7ff6ffd640 10 monclient: handle_mon_command_ack 2 [{"prefix": "status", "format": "json"}] 2024-06-03T09:06:33.140 INFO:teuthology.orchestra.run.smithi088.stderr:2024-06-03T09:06:33.140+0000 7f7ff6ffd640 10 monclient: _finish_command 2 = system:0 2024-06-03T09:06:33.141 INFO:teuthology.orchestra.run.smithi088.stdout: 2024-06-03T09:06:33.141 INFO:teuthology.orchestra.run.smithi088.stdout:{"fsid":"ec3c3fdd-3ac6-4d7d-9412-683130ed8b04","health":{"status":"HEALTH_WARN","checks":{"OSD_DOWN":{"severity":"HEALTH_WARN","summary":{"message":"1 osds down","count":1},"muted":false},"PG_DEGRADED":{"severity":"HEALTH_WARN","summary":{"message":"Degraded data redundancy: 1/6 objects degraded (16.667%), 1 pg degraded","count":1},"muted":false},"POOL_APP_NOT_ENABLED":{"severity":"HEALTH_WARN","summary":{"message":"1 pool(s) do not have an application enabled","count":1},"muted":false}},"mutes":[]},"election_epoch":10,"quorum":[0,1,2],"quorum_names":["a","b","c"],"quorum_age":1866,"monmap":{"epoch":1,"min_mon_release_name":"reef","num_mons":3},"osdmap":{"epoch":1010,"num_osds":8,"num_up_osds":7,"osd_up_since":1717405562,"num_in_osds":8,"osd_in_since":1717403732,"num_remapped_pgs":0},"pgmap":{"pgs_by_state":[{"state_name":"active+clean","count":27},{"state_name":"active+undersized","count":13},{"state_name":"active+undersized+degraded","count":1}],"num_pgs":41,"num_pools":3,"num_objects":3,"data_bytes":459285,"bytes_used":506417152,"bytes_avail":772587696128,"bytes_total":773094113280,"degraded_objects":1,"degraded_total":6,"degraded_ratio":0.16666666666666663},"fsmap":{"epoch":1,"by_rank":[],"up:standby":0},"mgrmap":{"available":true,"num_standbys":1,"modules":["iostat","nfs","restful"],"services":{}},"servicemap":{"epoch":23,"modified":"2024-06-03T08:53:48.910215+0000","services":{}},"progress_events":{}} 2024-06-03T09:06:33.147 INFO:teuthology.orchestra.run.smithi088.stderr:2024-06-03T09:06:33.143+0000 7f800062d640 10 librados: watch_flush enter 2024-06-03T09:06:33.147 INFO:teuthology.orchestra.run.smithi088.stderr:2024-06-03T09:06:33.143+0000 7f800062d640 10 librados: watch_flush exit 2024-06-03T09:06:33.147 INFO:teuthology.orchestra.run.smithi088.stderr:2024-06-03T09:06:33.143+0000 7f800062d640 20 client.30143.objecter shutdown clearing up homeless session... 2024-06-03T09:06:33.147 INFO:teuthology.orchestra.run.smithi088.stderr:2024-06-03T09:06:33.143+0000 7f800062d640 10 client.30143.objecter successfully canceled tick 2024-06-03T09:06:33.147 INFO:teuthology.orchestra.run.smithi088.stderr:2024-06-03T09:06:33.143+0000 7f800062d640 1 -- 172.21.15.88:0/2418070986 >> v1:172.21.15.192:6800/34229 conn(0x7f7fe406d3d0 legacy=0x7f7fe406f890 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-06-03T09:06:33.148 INFO:teuthology.orchestra.run.smithi088.stderr:2024-06-03T09:06:33.143+0000 7f800062d640 10 monclient: shutdown 2024-06-03T09:06:33.148 INFO:teuthology.orchestra.run.smithi088.stderr:2024-06-03T09:06:33.143+0000 7f800062d640 20 monclient: shutdown discarding 0 pending message(s) 2024-06-03T09:06:33.148 INFO:teuthology.orchestra.run.smithi088.stderr:2024-06-03T09:06:33.143+0000 7f800062d640 1 -- 172.21.15.88:0/2418070986 >> v1:172.21.15.88:6790/0 conn(0x7f7ff8143970 legacy=0x7f7ff812e5c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-06-03T09:06:33.148 INFO:teuthology.orchestra.run.smithi088.stderr:2024-06-03T09:06:33.143+0000 7f800062d640 1 -- 172.21.15.88:0/2418070986 shutdown_connections 2024-06-03T09:06:33.148 INFO:teuthology.orchestra.run.smithi088.stderr:2024-06-03T09:06:33.143+0000 7f800062d640 1 -- 172.21.15.88:0/2418070986 >> 172.21.15.88:0/2418070986 conn(0x7f7ff80b71e0 msgr2=0x7f7ff80b75c0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-06-03T09:06:33.148 INFO:teuthology.orchestra.run.smithi088.stderr:2024-06-03T09:06:33.143+0000 7f800062d640 1 -- 172.21.15.88:0/2418070986 shutdown_connections 2024-06-03T09:06:33.148 INFO:teuthology.orchestra.run.smithi088.stderr:2024-06-03T09:06:33.143+0000 7f800062d640 1 -- 172.21.15.88:0/2418070986 wait complete. 2024-06-03T09:06:33.148 INFO:teuthology.orchestra.run.smithi088.stderr:2024-06-03T09:06:33.143+0000 7f800062d640 1 librados: shutdown 2024-06-03T09:06:33.154 INFO:tasks.ceph.ceph_manager.ceph:no progress seen, keeping timeout for now 2024-06-03T09:06:33.154 DEBUG:teuthology.orchestra.run.smithi088:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph pg dump --format=json 2024-06-03T09:06:33.252 INFO:teuthology.orchestra.run.smithi088.stderr:2024-06-03T09:06:33.251+0000 7f045bf1d640 10 monclient: get_monmap_and_config 2024-06-03T09:06:33.252 INFO:teuthology.orchestra.run.smithi088.stderr:2024-06-03T09:06:33.251+0000 7f045bf1d640 10 monclient: build_initial_monmap 2024-06-03T09:06:33.252 INFO:teuthology.orchestra.run.smithi088.stderr:2024-06-03T09:06:33.251+0000 7f045bf1d640 10 monclient: monmap: 2024-06-03T09:06:33.252 INFO:teuthology.orchestra.run.smithi088.stderr:epoch 0 2024-06-03T09:06:33.252 INFO:teuthology.orchestra.run.smithi088.stderr:fsid ec3c3fdd-3ac6-4d7d-9412-683130ed8b04 2024-06-03T09:06:33.252 INFO:teuthology.orchestra.run.smithi088.stderr:last_changed 2024-06-03T09:06:33.252961+0000 2024-06-03T09:06:33.252 INFO:teuthology.orchestra.run.smithi088.stderr:created 2024-06-03T09:06:33.252961+0000 2024-06-03T09:06:33.252 INFO:teuthology.orchestra.run.smithi088.stderr:min_mon_release 0 (unknown) 2024-06-03T09:06:33.252 INFO:teuthology.orchestra.run.smithi088.stderr:election_strategy: 3 2024-06-03T09:06:33.252 INFO:teuthology.orchestra.run.smithi088.stderr:0: v1:172.21.15.88:6789/0 mon.noname-a 2024-06-03T09:06:33.252 INFO:teuthology.orchestra.run.smithi088.stderr:1: v1:172.21.15.192:6789/0 mon.noname-c 2024-06-03T09:06:33.253 INFO:teuthology.orchestra.run.smithi088.stderr:2: v1:172.21.15.88:6790/0 mon.noname-b 2024-06-03T09:06:33.253 INFO:teuthology.orchestra.run.smithi088.stderr: 2024-06-03T09:06:33.254 INFO:teuthology.orchestra.run.smithi088.stderr:2024-06-03T09:06:33.252+0000 7f045bf1d640 1 Processor -- start 2024-06-03T09:06:33.254 INFO:teuthology.orchestra.run.smithi088.stderr:2024-06-03T09:06:33.252+0000 7f045bf1d640 1 -- start start 2024-06-03T09:06:33.254 INFO:teuthology.orchestra.run.smithi088.stderr:2024-06-03T09:06:33.252+0000 7f045bf1d640 10 monclient: init 2024-06-03T09:06:33.254 INFO:teuthology.orchestra.run.smithi088.stderr:2024-06-03T09:06:33.253+0000 7f045bf1d640 10 monclient: _reopen_session rank -1 2024-06-03T09:06:33.254 INFO:teuthology.orchestra.run.smithi088.stderr:2024-06-03T09:06:33.253+0000 7f045bf1d640 10 monclient: _add_conns ranks=[0,1,2] 2024-06-03T09:06:33.254 INFO:teuthology.orchestra.run.smithi088.stderr:2024-06-03T09:06:33.253+0000 7f045bf1d640 10 monclient(hunting): picked mon.noname-a con 0x7f04540adb90 addr v1:172.21.15.88:6789/0 2024-06-03T09:06:33.254 INFO:teuthology.orchestra.run.smithi088.stderr:2024-06-03T09:06:33.253+0000 7f045bf1d640 10 monclient(hunting): picked mon.noname-c con 0x7f045406ee00 addr v1:172.21.15.192:6789/0 2024-06-03T09:06:33.254 INFO:teuthology.orchestra.run.smithi088.stderr:2024-06-03T09:06:33.253+0000 7f045bf1d640 10 monclient(hunting): picked mon.noname-b con 0x7f04541383e0 addr v1:172.21.15.88:6790/0 2024-06-03T09:06:33.254 INFO:teuthology.orchestra.run.smithi088.stderr:2024-06-03T09:06:33.253+0000 7f045bf1d640 1 -- --> v1:172.21.15.88:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f045406f2c0 con 0x7f04540adb90 2024-06-03T09:06:33.254 INFO:teuthology.orchestra.run.smithi088.stderr:2024-06-03T09:06:33.253+0000 7f045bf1d640 1 -- --> v1:172.21.15.192:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f04540710c0 con 0x7f045406ee00 2024-06-03T09:06:33.255 INFO:teuthology.orchestra.run.smithi088.stderr:2024-06-03T09:06:33.253+0000 7f045bf1d640 1 -- --> v1:172.21.15.88:6790/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f0454071fd0 con 0x7f04541383e0 2024-06-03T09:06:33.255 INFO:teuthology.orchestra.run.smithi088.stderr:2024-06-03T09:06:33.253+0000 7f045bf1d640 10 monclient(hunting): _renew_subs 2024-06-03T09:06:33.255 INFO:teuthology.orchestra.run.smithi088.stderr:2024-06-03T09:06:33.253+0000 7f045bf1d640 10 monclient(hunting): authenticate will time out at 3049.063368s 2024-06-03T09:06:33.255 INFO:teuthology.orchestra.run.smithi088.stderr:2024-06-03T09:06:33.253+0000 7f0459cb9640 1 --1- >> v1:172.21.15.88:6789/0 conn(0x7f04540adb90 0x7f04540718c0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.88:6789/0 says I am v1:172.21.15.88:41598/0 (socket says 172.21.15.88:41598) 2024-06-03T09:06:33.255 INFO:teuthology.orchestra.run.smithi088.stderr:2024-06-03T09:06:33.253+0000 7f0459cb9640 1 -- 172.21.15.88:0/2540818699 learned_addr learned my addr 172.21.15.88:0/2540818699 (peer_addr_for_me v1:172.21.15.88:0/0) 2024-06-03T09:06:33.256 INFO:teuthology.orchestra.run.smithi088.stderr:2024-06-03T09:06:33.256+0000 7f0458cb7640 1 -- 172.21.15.88:0/2540818699 <== mon.1 v1:172.21.15.192:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 134382754 0 0) 0x7f04540710c0 con 0x7f045406ee00 2024-06-03T09:06:33.256 INFO:teuthology.orchestra.run.smithi088.stderr:2024-06-03T09:06:33.256+0000 7f0458cb7640 10 monclient(hunting): _negotiate 2024-06-03T09:06:33.256 INFO:teuthology.orchestra.run.smithi088.stderr:2024-06-03T09:06:33.256+0000 7f0458cb7640 10 monclient(hunting): _init_auth method 2 2024-06-03T09:06:33.256 INFO:teuthology.orchestra.run.smithi088.stderr:2024-06-03T09:06:33.256+0000 7f0458cb7640 10 monclient(hunting): _init_auth creating new auth 2024-06-03T09:06:33.256 INFO:teuthology.orchestra.run.smithi088.stderr:2024-06-03T09:06:33.256+0000 7f0458cb7640 10 monclient(hunting): my global_id is 30223 2024-06-03T09:06:33.256 INFO:teuthology.orchestra.run.smithi088.stderr:2024-06-03T09:06:33.256+0000 7f0458cb7640 1 -- 172.21.15.88:0/2540818699 --> v1:172.21.15.192:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f0440003520 con 0x7f045406ee00 2024-06-03T09:06:33.256 INFO:teuthology.orchestra.run.smithi088.stderr:2024-06-03T09:06:33.256+0000 7f0458cb7640 1 -- 172.21.15.88:0/2540818699 <== mon.0 v1:172.21.15.88:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 3764281388 0 0) 0x7f045406f2c0 con 0x7f04540adb90 2024-06-03T09:06:33.256 INFO:teuthology.orchestra.run.smithi088.stderr:2024-06-03T09:06:33.256+0000 7f0458cb7640 10 monclient(hunting): _negotiate 2024-06-03T09:06:33.256 INFO:teuthology.orchestra.run.smithi088.stderr:2024-06-03T09:06:33.256+0000 7f0458cb7640 10 monclient(hunting): _init_auth method 2 2024-06-03T09:06:33.257 INFO:teuthology.orchestra.run.smithi088.stderr:2024-06-03T09:06:33.256+0000 7f0458cb7640 10 monclient(hunting): _init_auth creating new auth 2024-06-03T09:06:33.257 INFO:teuthology.orchestra.run.smithi088.stderr:2024-06-03T09:06:33.256+0000 7f0458cb7640 10 monclient(hunting): my global_id is 30642 2024-06-03T09:06:33.257 INFO:teuthology.orchestra.run.smithi088.stderr:2024-06-03T09:06:33.256+0000 7f0458cb7640 1 -- 172.21.15.88:0/2540818699 --> v1:172.21.15.88:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f04540710c0 con 0x7f04540adb90 2024-06-03T09:06:33.257 INFO:teuthology.orchestra.run.smithi088.stderr:2024-06-03T09:06:33.256+0000 7f0458cb7640 1 -- 172.21.15.88:0/2540818699 <== mon.2 v1:172.21.15.88:6790/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 932466853 0 0) 0x7f0454071fd0 con 0x7f04541383e0 2024-06-03T09:06:33.257 INFO:teuthology.orchestra.run.smithi088.stderr:2024-06-03T09:06:33.256+0000 7f0458cb7640 10 monclient(hunting): _negotiate 2024-06-03T09:06:33.257 INFO:teuthology.orchestra.run.smithi088.stderr:2024-06-03T09:06:33.256+0000 7f0458cb7640 10 monclient(hunting): _init_auth method 2 2024-06-03T09:06:33.257 INFO:teuthology.orchestra.run.smithi088.stderr:2024-06-03T09:06:33.256+0000 7f0458cb7640 10 monclient(hunting): _init_auth creating new auth 2024-06-03T09:06:33.258 INFO:teuthology.orchestra.run.smithi088.stderr:2024-06-03T09:06:33.256+0000 7f0458cb7640 10 monclient(hunting): my global_id is 30146 2024-06-03T09:06:33.258 INFO:teuthology.orchestra.run.smithi088.stderr:2024-06-03T09:06:33.256+0000 7f0458cb7640 1 -- 172.21.15.88:0/2540818699 --> v1:172.21.15.88:6790/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f045406f2c0 con 0x7f04541383e0 2024-06-03T09:06:33.259 INFO:teuthology.orchestra.run.smithi088.stderr:2024-06-03T09:06:33.259+0000 7f0458cb7640 1 -- 172.21.15.88:0/2540818699 <== mon.1 v1:172.21.15.192:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 214+0+0 (unknown 615974754 0 0) 0x7f0440003520 con 0x7f045406ee00 2024-06-03T09:06:33.259 INFO:teuthology.orchestra.run.smithi088.stderr:2024-06-03T09:06:33.259+0000 7f0458cb7640 1 -- 172.21.15.88:0/2540818699 >> v1:172.21.15.88:6790/0 conn(0x7f04541383e0 legacy=0x7f0454149af0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-06-03T09:06:33.259 INFO:teuthology.orchestra.run.smithi088.stderr:2024-06-03T09:06:33.259+0000 7f0458cb7640 1 -- 172.21.15.88:0/2540818699 >> v1:172.21.15.88:6789/0 conn(0x7f04540adb90 legacy=0x7f04540718c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-06-03T09:06:33.259 INFO:teuthology.orchestra.run.smithi088.stderr:2024-06-03T09:06:33.259+0000 7f0458cb7640 10 monclient: _finish_hunting 0 2024-06-03T09:06:33.259 INFO:teuthology.orchestra.run.smithi088.stderr:2024-06-03T09:06:33.259+0000 7f0458cb7640 1 monclient: found mon.noname-c 2024-06-03T09:06:33.260 INFO:teuthology.orchestra.run.smithi088.stderr:2024-06-03T09:06:33.259+0000 7f0458cb7640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-06-03T09:06:33.260 INFO:teuthology.orchestra.run.smithi088.stderr:2024-06-03T09:06:33.259+0000 7f0458cb7640 10 monclient: _send_mon_message to mon.noname-c at v1:172.21.15.192:6789/0 2024-06-03T09:06:33.260 INFO:teuthology.orchestra.run.smithi088.stderr:2024-06-03T09:06:33.259+0000 7f0458cb7640 1 -- 172.21.15.88:0/2540818699 --> v1:172.21.15.192:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f0454141750 con 0x7f045406ee00 2024-06-03T09:06:33.260 INFO:teuthology.orchestra.run.smithi088.stderr:2024-06-03T09:06:33.259+0000 7f0458cb7640 10 monclient: _finish_auth 0 2024-06-03T09:06:33.260 INFO:teuthology.orchestra.run.smithi088.stderr:2024-06-03T09:06:33.259+0000 7f0458cb7640 10 monclient: _check_auth_tickets 2024-06-03T09:06:33.260 INFO:teuthology.orchestra.run.smithi088.stderr:2024-06-03T09:06:33.259+0000 7f0458cb7640 20 monclient: _check_auth_rotating not needed by client.admin 2024-06-03T09:06:33.260 INFO:teuthology.orchestra.run.smithi088.stderr:2024-06-03T09:06:33.259+0000 7f0458cb7640 1 -- 172.21.15.88:0/2540818699 <== mon.1 v1:172.21.15.192:6789/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 2234374065 0 0) 0x7f0444001610 con 0x7f045406ee00 2024-06-03T09:06:33.260 INFO:teuthology.orchestra.run.smithi088.stderr:2024-06-03T09:06:33.259+0000 7f045bf1d640 5 monclient: authenticate success, global_id 30223 2024-06-03T09:06:33.260 INFO:teuthology.orchestra.run.smithi088.stderr:2024-06-03T09:06:33.259+0000 7f045bf1d640 20 monclient: get_monmap_and_config waiting for monmap|config 2024-06-03T09:06:33.260 INFO:teuthology.orchestra.run.smithi088.stderr:2024-06-03T09:06:33.259+0000 7f0458cb7640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-06-03T09:06:33.260 INFO:teuthology.orchestra.run.smithi088.stderr:2024-06-03T09:06:33.259+0000 7f0458cb7640 10 monclient: got monmap 1 from mon.noname-c (according to old e1) 2024-06-03T09:06:33.260 INFO:teuthology.orchestra.run.smithi088.stderr:2024-06-03T09:06:33.259+0000 7f0458cb7640 10 monclient: dump: 2024-06-03T09:06:33.260 INFO:teuthology.orchestra.run.smithi088.stderr:epoch 1 2024-06-03T09:06:33.261 INFO:teuthology.orchestra.run.smithi088.stderr:fsid ec3c3fdd-3ac6-4d7d-9412-683130ed8b04 2024-06-03T09:06:33.261 INFO:teuthology.orchestra.run.smithi088.stderr:last_changed 2024-06-03T08:34:31.885125+0000 2024-06-03T09:06:33.261 INFO:teuthology.orchestra.run.smithi088.stderr:created 2024-06-03T08:34:31.885125+0000 2024-06-03T09:06:33.261 INFO:teuthology.orchestra.run.smithi088.stderr:min_mon_release 18 (reef) 2024-06-03T09:06:33.261 INFO:teuthology.orchestra.run.smithi088.stderr:election_strategy: 3 2024-06-03T09:06:33.261 INFO:teuthology.orchestra.run.smithi088.stderr:0: v1:172.21.15.88:6789/0 mon.a 2024-06-03T09:06:33.261 INFO:teuthology.orchestra.run.smithi088.stderr:1: v1:172.21.15.192:6789/0 mon.b 2024-06-03T09:06:33.261 INFO:teuthology.orchestra.run.smithi088.stderr:2: v1:172.21.15.88:6790/0 mon.c 2024-06-03T09:06:33.261 INFO:teuthology.orchestra.run.smithi088.stderr: 2024-06-03T09:06:33.261 INFO:teuthology.orchestra.run.smithi088.stderr:2024-06-03T09:06:33.259+0000 7f045bf1d640 20 monclient: get_monmap_and_config waiting for monmap|config 2024-06-03T09:06:33.264 INFO:teuthology.orchestra.run.smithi088.stderr:2024-06-03T09:06:33.261+0000 7f0458cb7640 1 -- 172.21.15.88:0/2540818699 <== mon.1 v1:172.21.15.192:6789/0 4 ==== config(0 keys) v1 ==== 4+0+0 (unknown 0 0 0) 0x7f0444001830 con 0x7f045406ee00 2024-06-03T09:06:33.264 INFO:teuthology.orchestra.run.smithi088.stderr:2024-06-03T09:06:33.261+0000 7f0458cb7640 10 monclient: handle_config config(0 keys) v1 2024-06-03T09:06:33.264 INFO:teuthology.orchestra.run.smithi088.stderr:2024-06-03T09:06:33.261+0000 7f0458cb7640 1 -- 172.21.15.88:0/2540818699 <== mon.1 v1:172.21.15.192:6789/0 5 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 2234374065 0 0) 0x7f0444001bc0 con 0x7f045406ee00 2024-06-03T09:06:33.264 INFO:teuthology.orchestra.run.smithi088.stderr:2024-06-03T09:06:33.261+0000 7f0458cb7640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-06-03T09:06:33.264 INFO:teuthology.orchestra.run.smithi088.stderr:2024-06-03T09:06:33.261+0000 7f0458cb7640 10 monclient: got monmap 1 from mon.b (according to old e1) 2024-06-03T09:06:33.264 INFO:teuthology.orchestra.run.smithi088.stderr:2024-06-03T09:06:33.261+0000 7f0458cb7640 10 monclient: dump: 2024-06-03T09:06:33.264 INFO:teuthology.orchestra.run.smithi088.stderr:epoch 1 2024-06-03T09:06:33.264 INFO:teuthology.orchestra.run.smithi088.stderr:fsid ec3c3fdd-3ac6-4d7d-9412-683130ed8b04 2024-06-03T09:06:33.265 INFO:teuthology.orchestra.run.smithi088.stderr:last_changed 2024-06-03T08:34:31.885125+0000 2024-06-03T09:06:33.265 INFO:teuthology.orchestra.run.smithi088.stderr:created 2024-06-03T08:34:31.885125+0000 2024-06-03T09:06:33.265 INFO:teuthology.orchestra.run.smithi088.stderr:min_mon_release 18 (reef) 2024-06-03T09:06:33.265 INFO:teuthology.orchestra.run.smithi088.stderr:election_strategy: 3 2024-06-03T09:06:33.265 INFO:teuthology.orchestra.run.smithi088.stderr:0: v1:172.21.15.88:6789/0 mon.a 2024-06-03T09:06:33.265 INFO:teuthology.orchestra.run.smithi088.stderr:1: v1:172.21.15.192:6789/0 mon.b 2024-06-03T09:06:33.265 INFO:teuthology.orchestra.run.smithi088.stderr:2: v1:172.21.15.88:6790/0 mon.c 2024-06-03T09:06:33.265 INFO:teuthology.orchestra.run.smithi088.stderr: 2024-06-03T09:06:33.265 INFO:teuthology.orchestra.run.smithi088.stderr:2024-06-03T09:06:33.261+0000 7f045bf1d640 10 monclient: get_monmap_and_config success 2024-06-03T09:06:33.265 INFO:teuthology.orchestra.run.smithi088.stderr:2024-06-03T09:06:33.261+0000 7f045bf1d640 4 set_mon_vals no callback set 2024-06-03T09:06:33.265 INFO:teuthology.orchestra.run.smithi088.stderr:2024-06-03T09:06:33.261+0000 7f045bf1d640 10 monclient: shutdown 2024-06-03T09:06:33.265 INFO:teuthology.orchestra.run.smithi088.stderr:2024-06-03T09:06:33.261+0000 7f045bf1d640 20 monclient: shutdown discarding 0 pending message(s) 2024-06-03T09:06:33.266 INFO:teuthology.orchestra.run.smithi088.stderr:2024-06-03T09:06:33.261+0000 7f045bf1d640 1 -- 172.21.15.88:0/2540818699 >> v1:172.21.15.192:6789/0 conn(0x7f045406ee00 legacy=0x7f0454137cd0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-06-03T09:06:33.266 INFO:teuthology.orchestra.run.smithi088.stderr:2024-06-03T09:06:33.261+0000 7f045bf1d640 1 -- 172.21.15.88:0/2540818699 shutdown_connections 2024-06-03T09:06:33.266 INFO:teuthology.orchestra.run.smithi088.stderr:2024-06-03T09:06:33.261+0000 7f045bf1d640 1 -- 172.21.15.88:0/2540818699 >> 172.21.15.88:0/2540818699 conn(0x7f04540b2f20 msgr2=0x7f04540b3300 unknown :-1 s=STATE_NONE l=0).mark_down 2024-06-03T09:06:33.266 INFO:teuthology.orchestra.run.smithi088.stderr:2024-06-03T09:06:33.261+0000 7f045bf1d640 1 -- 172.21.15.88:0/2540818699 shutdown_connections 2024-06-03T09:06:33.266 INFO:teuthology.orchestra.run.smithi088.stderr:2024-06-03T09:06:33.261+0000 7f045bf1d640 1 -- 172.21.15.88:0/2540818699 wait complete. 2024-06-03T09:06:33.266 INFO:teuthology.orchestra.run.smithi088.stderr:2024-06-03T09:06:33.262+0000 7f045bf1d640 10 monclient: build_initial_monmap 2024-06-03T09:06:33.266 INFO:teuthology.orchestra.run.smithi088.stderr:2024-06-03T09:06:33.262+0000 7f045bf1d640 10 monclient: monmap: 2024-06-03T09:06:33.266 INFO:teuthology.orchestra.run.smithi088.stderr:epoch 0 2024-06-03T09:06:33.266 INFO:teuthology.orchestra.run.smithi088.stderr:fsid 00000000-0000-0000-0000-000000000000 2024-06-03T09:06:33.266 INFO:teuthology.orchestra.run.smithi088.stderr:last_changed 0.000000 2024-06-03T09:06:33.266 INFO:teuthology.orchestra.run.smithi088.stderr:created 0.000000 2024-06-03T09:06:33.266 INFO:teuthology.orchestra.run.smithi088.stderr:min_mon_release 0 (unknown) 2024-06-03T09:06:33.267 INFO:teuthology.orchestra.run.smithi088.stderr:election_strategy: 1 2024-06-03T09:06:33.267 INFO:teuthology.orchestra.run.smithi088.stderr:0: v1:172.21.15.88:6789/0 mon.noname-a 2024-06-03T09:06:33.267 INFO:teuthology.orchestra.run.smithi088.stderr:1: v1:172.21.15.192:6789/0 mon.noname-b 2024-06-03T09:06:33.267 INFO:teuthology.orchestra.run.smithi088.stderr:2: v1:172.21.15.88:6790/0 mon.noname-c 2024-06-03T09:06:33.267 INFO:teuthology.orchestra.run.smithi088.stderr: 2024-06-03T09:06:33.267 INFO:teuthology.orchestra.run.smithi088.stderr:2024-06-03T09:06:33.262+0000 7f045bf1d640 1 librados: starting msgr at 2024-06-03T09:06:33.267 INFO:teuthology.orchestra.run.smithi088.stderr:2024-06-03T09:06:33.262+0000 7f045bf1d640 1 librados: starting objecter 2024-06-03T09:06:33.267 INFO:teuthology.orchestra.run.smithi088.stderr:2024-06-03T09:06:33.262+0000 7f045bf1d640 1 Processor -- start 2024-06-03T09:06:33.267 INFO:teuthology.orchestra.run.smithi088.stderr:2024-06-03T09:06:33.263+0000 7f045bf1d640 1 -- start start 2024-06-03T09:06:33.267 INFO:teuthology.orchestra.run.smithi088.stderr:2024-06-03T09:06:33.263+0000 7f045bf1d640 1 librados: setting wanted keys 2024-06-03T09:06:33.267 INFO:teuthology.orchestra.run.smithi088.stderr:2024-06-03T09:06:33.263+0000 7f045bf1d640 1 librados: calling monclient init 2024-06-03T09:06:33.267 INFO:teuthology.orchestra.run.smithi088.stderr:2024-06-03T09:06:33.263+0000 7f045bf1d640 10 monclient: init 2024-06-03T09:06:33.267 INFO:teuthology.orchestra.run.smithi088.stderr:2024-06-03T09:06:33.263+0000 7f045bf1d640 10 monclient: _reopen_session rank -1 2024-06-03T09:06:33.268 INFO:teuthology.orchestra.run.smithi088.stderr:2024-06-03T09:06:33.263+0000 7f045bf1d640 10 monclient: _add_conns ranks=[0,2,1] 2024-06-03T09:06:33.268 INFO:teuthology.orchestra.run.smithi088.stderr:2024-06-03T09:06:33.263+0000 7f045bf1d640 10 monclient(hunting): picked mon.noname-a con 0x7f045406ee00 addr v1:172.21.15.88:6789/0 2024-06-03T09:06:33.268 INFO:teuthology.orchestra.run.smithi088.stderr:2024-06-03T09:06:33.263+0000 7f045bf1d640 10 monclient(hunting): picked mon.noname-c con 0x7f04540adb90 addr v1:172.21.15.88:6790/0 2024-06-03T09:06:33.268 INFO:teuthology.orchestra.run.smithi088.stderr:2024-06-03T09:06:33.263+0000 7f045bf1d640 10 monclient(hunting): picked mon.noname-b con 0x7f04541383e0 addr v1:172.21.15.192:6789/0 2024-06-03T09:06:33.268 INFO:teuthology.orchestra.run.smithi088.stderr:2024-06-03T09:06:33.263+0000 7f045bf1d640 1 -- --> v1:172.21.15.88:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f04541d26b0 con 0x7f045406ee00 2024-06-03T09:06:33.268 INFO:teuthology.orchestra.run.smithi088.stderr:2024-06-03T09:06:33.263+0000 7f045bf1d640 1 -- --> v1:172.21.15.192:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f0454138cd0 con 0x7f04541383e0 2024-06-03T09:06:33.268 INFO:teuthology.orchestra.run.smithi088.stderr:2024-06-03T09:06:33.263+0000 7f045bf1d640 1 -- --> v1:172.21.15.88:6790/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f045413ae20 con 0x7f04540adb90 2024-06-03T09:06:33.268 INFO:teuthology.orchestra.run.smithi088.stderr:2024-06-03T09:06:33.263+0000 7f045bf1d640 10 monclient(hunting): _renew_subs 2024-06-03T09:06:33.268 INFO:teuthology.orchestra.run.smithi088.stderr:2024-06-03T09:06:33.263+0000 7f045bf1d640 10 monclient(hunting): authenticate will time out at 3049.073536s 2024-06-03T09:06:33.268 INFO:teuthology.orchestra.run.smithi088.stderr:2024-06-03T09:06:33.263+0000 7f0459cb9640 1 --1- >> v1:172.21.15.88:6789/0 conn(0x7f045406ee00 0x7f045413a710 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=1).handle_server_banner_and_identify peer v1:172.21.15.88:6789/0 says I am v1:172.21.15.88:41612/0 (socket says 172.21.15.88:41612) 2024-06-03T09:06:33.268 INFO:teuthology.orchestra.run.smithi088.stderr:2024-06-03T09:06:33.263+0000 7f0459cb9640 1 -- 172.21.15.88:0/1352477224 learned_addr learned my addr 172.21.15.88:0/1352477224 (peer_addr_for_me v1:172.21.15.88:0/0) 2024-06-03T09:06:33.268 INFO:teuthology.orchestra.run.smithi088.stderr:2024-06-03T09:06:33.264+0000 7f044a7fc640 10 client.?.objecter ms_handle_connect 0x7f045406ee00 2024-06-03T09:06:33.268 INFO:teuthology.orchestra.run.smithi088.stderr:2024-06-03T09:06:33.264+0000 7f044a7fc640 10 client.?.objecter resend_mon_ops 2024-06-03T09:06:33.268 INFO:teuthology.orchestra.run.smithi088.stderr:2024-06-03T09:06:33.264+0000 7f044a7fc640 10 client.?.objecter ms_handle_connect 0x7f04541383e0 2024-06-03T09:06:33.269 INFO:teuthology.orchestra.run.smithi088.stderr:2024-06-03T09:06:33.264+0000 7f044a7fc640 10 client.?.objecter resend_mon_ops 2024-06-03T09:06:33.269 INFO:teuthology.orchestra.run.smithi088.stderr:2024-06-03T09:06:33.264+0000 7f044a7fc640 10 client.?.objecter ms_handle_connect 0x7f04540adb90 2024-06-03T09:06:33.269 INFO:teuthology.orchestra.run.smithi088.stderr:2024-06-03T09:06:33.264+0000 7f044a7fc640 10 client.?.objecter resend_mon_ops 2024-06-03T09:06:33.270 INFO:teuthology.orchestra.run.smithi088.stderr:2024-06-03T09:06:33.266+0000 7f044a7fc640 1 -- 172.21.15.88:0/1352477224 <== mon.0 v1:172.21.15.88:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 3029489017 0 0) 0x7f04541d26b0 con 0x7f045406ee00 2024-06-03T09:06:33.270 INFO:teuthology.orchestra.run.smithi088.stderr:2024-06-03T09:06:33.266+0000 7f044a7fc640 10 monclient(hunting): _negotiate 2024-06-03T09:06:33.270 INFO:teuthology.orchestra.run.smithi088.stderr:2024-06-03T09:06:33.266+0000 7f044a7fc640 10 monclient(hunting): _init_auth method 2 2024-06-03T09:06:33.270 INFO:teuthology.orchestra.run.smithi088.stderr:2024-06-03T09:06:33.266+0000 7f044a7fc640 10 monclient(hunting): _init_auth creating new auth 2024-06-03T09:06:33.270 INFO:teuthology.orchestra.run.smithi088.stderr:2024-06-03T09:06:33.266+0000 7f044a7fc640 10 monclient(hunting): my global_id is 30645 2024-06-03T09:06:33.270 INFO:teuthology.orchestra.run.smithi088.stderr:2024-06-03T09:06:33.266+0000 7f044a7fc640 1 -- 172.21.15.88:0/1352477224 --> v1:172.21.15.88:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f0438004eb0 con 0x7f045406ee00 2024-06-03T09:06:33.270 INFO:teuthology.orchestra.run.smithi088.stderr:2024-06-03T09:06:33.266+0000 7f044a7fc640 1 -- 172.21.15.88:0/1352477224 <== mon.1 v1:172.21.15.192:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 4292810580 0 0) 0x7f0454138cd0 con 0x7f04541383e0 2024-06-03T09:06:33.270 INFO:teuthology.orchestra.run.smithi088.stderr:2024-06-03T09:06:33.266+0000 7f044a7fc640 10 monclient(hunting): _negotiate 2024-06-03T09:06:33.270 INFO:teuthology.orchestra.run.smithi088.stderr:2024-06-03T09:06:33.266+0000 7f044a7fc640 10 monclient(hunting): _init_auth method 2 2024-06-03T09:06:33.270 INFO:teuthology.orchestra.run.smithi088.stderr:2024-06-03T09:06:33.266+0000 7f044a7fc640 10 monclient(hunting): _init_auth creating new auth 2024-06-03T09:06:33.270 INFO:teuthology.orchestra.run.smithi088.stderr:2024-06-03T09:06:33.266+0000 7f044a7fc640 10 monclient(hunting): my global_id is 30226 2024-06-03T09:06:33.270 INFO:teuthology.orchestra.run.smithi088.stderr:2024-06-03T09:06:33.266+0000 7f044a7fc640 1 -- 172.21.15.88:0/1352477224 --> v1:172.21.15.192:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f04541d26b0 con 0x7f04541383e0 2024-06-03T09:06:33.271 INFO:teuthology.orchestra.run.smithi088.stderr:2024-06-03T09:06:33.266+0000 7f044a7fc640 1 -- 172.21.15.88:0/1352477224 <== mon.2 v1:172.21.15.88:6790/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 1577871563 0 0) 0x7f045413ae20 con 0x7f04540adb90 2024-06-03T09:06:33.271 INFO:teuthology.orchestra.run.smithi088.stderr:2024-06-03T09:06:33.266+0000 7f044a7fc640 10 monclient(hunting): _negotiate 2024-06-03T09:06:33.271 INFO:teuthology.orchestra.run.smithi088.stderr:2024-06-03T09:06:33.266+0000 7f044a7fc640 10 monclient(hunting): _init_auth method 2 2024-06-03T09:06:33.271 INFO:teuthology.orchestra.run.smithi088.stderr:2024-06-03T09:06:33.266+0000 7f044a7fc640 10 monclient(hunting): _init_auth creating new auth 2024-06-03T09:06:33.271 INFO:teuthology.orchestra.run.smithi088.stderr:2024-06-03T09:06:33.266+0000 7f044a7fc640 10 monclient(hunting): my global_id is 30149 2024-06-03T09:06:33.271 INFO:teuthology.orchestra.run.smithi088.stderr:2024-06-03T09:06:33.266+0000 7f044a7fc640 1 -- 172.21.15.88:0/1352477224 --> v1:172.21.15.88:6790/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f0454138cd0 con 0x7f04540adb90 2024-06-03T09:06:33.272 INFO:teuthology.orchestra.run.smithi088.stderr:2024-06-03T09:06:33.270+0000 7f044a7fc640 1 -- 172.21.15.88:0/1352477224 <== mon.1 v1:172.21.15.192:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 764+0+0 (unknown 1495122075 0 0) 0x7f04541d26b0 con 0x7f04541383e0 2024-06-03T09:06:33.272 INFO:teuthology.orchestra.run.smithi088.stderr:2024-06-03T09:06:33.270+0000 7f044a7fc640 1 -- 172.21.15.88:0/1352477224 --> v1:172.21.15.192:6789/0 -- auth(proto 2 165 bytes epoch 0) v1 -- 0x7f045413ae20 con 0x7f04541383e0 2024-06-03T09:06:33.272 INFO:teuthology.orchestra.run.smithi088.stderr:2024-06-03T09:06:33.270+0000 7f044a7fc640 1 -- 172.21.15.88:0/1352477224 <== mon.1 v1:172.21.15.192:6789/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 2234374065 0 0) 0x7f044c001b90 con 0x7f04541383e0 2024-06-03T09:06:33.272 INFO:teuthology.orchestra.run.smithi088.stderr:2024-06-03T09:06:33.270+0000 7f044a7fc640 10 monclient(hunting): handle_monmap mon_map magic: 0 v1 2024-06-03T09:06:33.272 INFO:teuthology.orchestra.run.smithi088.stderr:2024-06-03T09:06:33.270+0000 7f044a7fc640 10 monclient(hunting): got monmap 1 from mon.noname-b (according to old e1) 2024-06-03T09:06:33.272 INFO:teuthology.orchestra.run.smithi088.stderr:2024-06-03T09:06:33.270+0000 7f044a7fc640 10 monclient(hunting): dump: 2024-06-03T09:06:33.272 INFO:teuthology.orchestra.run.smithi088.stderr:epoch 1 2024-06-03T09:06:33.273 INFO:teuthology.orchestra.run.smithi088.stderr:fsid ec3c3fdd-3ac6-4d7d-9412-683130ed8b04 2024-06-03T09:06:33.273 INFO:teuthology.orchestra.run.smithi088.stderr:last_changed 2024-06-03T08:34:31.885125+0000 2024-06-03T09:06:33.273 INFO:teuthology.orchestra.run.smithi088.stderr:created 2024-06-03T08:34:31.885125+0000 2024-06-03T09:06:33.273 INFO:teuthology.orchestra.run.smithi088.stderr:min_mon_release 18 (reef) 2024-06-03T09:06:33.273 INFO:teuthology.orchestra.run.smithi088.stderr:election_strategy: 3 2024-06-03T09:06:33.273 INFO:teuthology.orchestra.run.smithi088.stderr:0: v1:172.21.15.88:6789/0 mon.a 2024-06-03T09:06:33.273 INFO:teuthology.orchestra.run.smithi088.stderr:1: v1:172.21.15.192:6789/0 mon.b 2024-06-03T09:06:33.273 INFO:teuthology.orchestra.run.smithi088.stderr:2: v1:172.21.15.88:6790/0 mon.c 2024-06-03T09:06:33.273 INFO:teuthology.orchestra.run.smithi088.stderr: 2024-06-03T09:06:33.273 INFO:teuthology.orchestra.run.smithi088.stderr:2024-06-03T09:06:33.270+0000 7f044a7fc640 10 monclient(hunting): _finish_auth 0 2024-06-03T09:06:33.273 INFO:teuthology.orchestra.run.smithi088.stderr:2024-06-03T09:06:33.272+0000 7f044a7fc640 1 -- 172.21.15.88:0/1352477224 <== mon.0 v1:172.21.15.88:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 764+0+0 (unknown 3689966307 0 0) 0x7f0438004eb0 con 0x7f045406ee00 2024-06-03T09:06:33.273 INFO:teuthology.orchestra.run.smithi088.stderr:2024-06-03T09:06:33.272+0000 7f044a7fc640 1 -- 172.21.15.88:0/1352477224 --> v1:172.21.15.88:6789/0 -- auth(proto 2 165 bytes epoch 0) v1 -- 0x7f04541d26b0 con 0x7f045406ee00 2024-06-03T09:06:33.273 INFO:teuthology.orchestra.run.smithi088.stderr:2024-06-03T09:06:33.272+0000 7f044a7fc640 1 -- 172.21.15.88:0/1352477224 <== mon.0 v1:172.21.15.88:6789/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 2234374065 0 0) 0x7f04500018f0 con 0x7f045406ee00 2024-06-03T09:06:33.273 INFO:teuthology.orchestra.run.smithi088.stderr:2024-06-03T09:06:33.272+0000 7f044a7fc640 10 monclient(hunting): handle_monmap mon_map magic: 0 v1 2024-06-03T09:06:33.274 INFO:teuthology.orchestra.run.smithi088.stderr:2024-06-03T09:06:33.272+0000 7f044a7fc640 10 monclient(hunting): got monmap 1 from mon.a (according to old e1) 2024-06-03T09:06:33.274 INFO:teuthology.orchestra.run.smithi088.stderr:2024-06-03T09:06:33.272+0000 7f044a7fc640 10 monclient(hunting): dump: 2024-06-03T09:06:33.274 INFO:teuthology.orchestra.run.smithi088.stderr:epoch 1 2024-06-03T09:06:33.274 INFO:teuthology.orchestra.run.smithi088.stderr:fsid ec3c3fdd-3ac6-4d7d-9412-683130ed8b04 2024-06-03T09:06:33.274 INFO:teuthology.orchestra.run.smithi088.stderr:last_changed 2024-06-03T08:34:31.885125+0000 2024-06-03T09:06:33.274 INFO:teuthology.orchestra.run.smithi088.stderr:created 2024-06-03T08:34:31.885125+0000 2024-06-03T09:06:33.274 INFO:teuthology.orchestra.run.smithi088.stderr:min_mon_release 18 (reef) 2024-06-03T09:06:33.274 INFO:teuthology.orchestra.run.smithi088.stderr:election_strategy: 3 2024-06-03T09:06:33.274 INFO:teuthology.orchestra.run.smithi088.stderr:0: v1:172.21.15.88:6789/0 mon.a 2024-06-03T09:06:33.274 INFO:teuthology.orchestra.run.smithi088.stderr:1: v1:172.21.15.192:6789/0 mon.b 2024-06-03T09:06:33.274 INFO:teuthology.orchestra.run.smithi088.stderr:2: v1:172.21.15.88:6790/0 mon.c 2024-06-03T09:06:33.274 INFO:teuthology.orchestra.run.smithi088.stderr: 2024-06-03T09:06:33.274 INFO:teuthology.orchestra.run.smithi088.stderr:2024-06-03T09:06:33.272+0000 7f044a7fc640 1 -- 172.21.15.88:0/1352477224 <== mon.2 v1:172.21.15.88:6790/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 764+0+0 (unknown 3278588001 0 0) 0x7f0454138cd0 con 0x7f04540adb90 2024-06-03T09:06:33.275 INFO:teuthology.orchestra.run.smithi088.stderr:2024-06-03T09:06:33.272+0000 7f044a7fc640 1 -- 172.21.15.88:0/1352477224 --> v1:172.21.15.88:6790/0 -- auth(proto 2 165 bytes epoch 0) v1 -- 0x7f0438004eb0 con 0x7f04540adb90 2024-06-03T09:06:33.275 INFO:teuthology.orchestra.run.smithi088.stderr:2024-06-03T09:06:33.272+0000 7f044a7fc640 1 -- 172.21.15.88:0/1352477224 <== mon.2 v1:172.21.15.88:6790/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 2234374065 0 0) 0x7f0444001b40 con 0x7f04540adb90 2024-06-03T09:06:33.275 INFO:teuthology.orchestra.run.smithi088.stderr:2024-06-03T09:06:33.272+0000 7f044a7fc640 10 monclient(hunting): handle_monmap mon_map magic: 0 v1 2024-06-03T09:06:33.275 INFO:teuthology.orchestra.run.smithi088.stderr:2024-06-03T09:06:33.272+0000 7f044a7fc640 10 monclient(hunting): got monmap 1 from mon.c (according to old e1) 2024-06-03T09:06:33.275 INFO:teuthology.orchestra.run.smithi088.stderr:2024-06-03T09:06:33.272+0000 7f044a7fc640 10 monclient(hunting): dump: 2024-06-03T09:06:33.275 INFO:teuthology.orchestra.run.smithi088.stderr:epoch 1 2024-06-03T09:06:33.275 INFO:teuthology.orchestra.run.smithi088.stderr:fsid ec3c3fdd-3ac6-4d7d-9412-683130ed8b04 2024-06-03T09:06:33.275 INFO:teuthology.orchestra.run.smithi088.stderr:last_changed 2024-06-03T08:34:31.885125+0000 2024-06-03T09:06:33.275 INFO:teuthology.orchestra.run.smithi088.stderr:created 2024-06-03T08:34:31.885125+0000 2024-06-03T09:06:33.275 INFO:teuthology.orchestra.run.smithi088.stderr:min_mon_release 18 (reef) 2024-06-03T09:06:33.275 INFO:teuthology.orchestra.run.smithi088.stderr:election_strategy: 3 2024-06-03T09:06:33.275 INFO:teuthology.orchestra.run.smithi088.stderr:0: v1:172.21.15.88:6789/0 mon.a 2024-06-03T09:06:33.275 INFO:teuthology.orchestra.run.smithi088.stderr:1: v1:172.21.15.192:6789/0 mon.b 2024-06-03T09:06:33.276 INFO:teuthology.orchestra.run.smithi088.stderr:2: v1:172.21.15.88:6790/0 mon.c 2024-06-03T09:06:33.276 INFO:teuthology.orchestra.run.smithi088.stderr: 2024-06-03T09:06:33.277 INFO:teuthology.orchestra.run.smithi088.stderr:2024-06-03T09:06:33.274+0000 7f044a7fc640 1 -- 172.21.15.88:0/1352477224 <== mon.1 v1:172.21.15.192:6789/0 4 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 580+0+0 (unknown 1976852448 0 0) 0x7f045413ae20 con 0x7f04541383e0 2024-06-03T09:06:33.277 INFO:teuthology.orchestra.run.smithi088.stderr:2024-06-03T09:06:33.274+0000 7f044a7fc640 1 -- 172.21.15.88:0/1352477224 >> v1:172.21.15.88:6790/0 conn(0x7f04540adb90 legacy=0x7f04541d2bd0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-06-03T09:06:33.277 INFO:teuthology.orchestra.run.smithi088.stderr:2024-06-03T09:06:33.274+0000 7f044a7fc640 1 -- 172.21.15.88:0/1352477224 >> v1:172.21.15.88:6789/0 conn(0x7f045406ee00 legacy=0x7f045413a710 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-06-03T09:06:33.277 INFO:teuthology.orchestra.run.smithi088.stderr:2024-06-03T09:06:33.274+0000 7f044a7fc640 10 monclient: _finish_hunting 0 2024-06-03T09:06:33.277 INFO:teuthology.orchestra.run.smithi088.stderr:2024-06-03T09:06:33.274+0000 7f044a7fc640 1 monclient: found mon.b 2024-06-03T09:06:33.277 INFO:teuthology.orchestra.run.smithi088.stderr:2024-06-03T09:06:33.274+0000 7f044a7fc640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-06-03T09:06:33.277 INFO:teuthology.orchestra.run.smithi088.stderr:2024-06-03T09:06:33.274+0000 7f044a7fc640 10 monclient: _send_mon_message to mon.b at v1:172.21.15.192:6789/0 2024-06-03T09:06:33.277 INFO:teuthology.orchestra.run.smithi088.stderr:2024-06-03T09:06:33.274+0000 7f044a7fc640 1 -- 172.21.15.88:0/1352477224 --> v1:172.21.15.192:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f04541398a0 con 0x7f04541383e0 2024-06-03T09:06:33.278 INFO:teuthology.orchestra.run.smithi088.stderr:2024-06-03T09:06:33.274+0000 7f044a7fc640 10 monclient: _finish_auth 0 2024-06-03T09:06:33.278 INFO:teuthology.orchestra.run.smithi088.stderr:2024-06-03T09:06:33.274+0000 7f044a7fc640 10 monclient: _check_auth_tickets 2024-06-03T09:06:33.278 INFO:teuthology.orchestra.run.smithi088.stderr:2024-06-03T09:06:33.274+0000 7f044a7fc640 20 monclient: _check_auth_rotating not needed by client.admin 2024-06-03T09:06:33.278 INFO:teuthology.orchestra.run.smithi088.stderr:2024-06-03T09:06:33.274+0000 7f045bf1d640 5 monclient: authenticate success, global_id 30226 2024-06-03T09:06:33.278 INFO:teuthology.orchestra.run.smithi088.stderr:2024-06-03T09:06:33.274+0000 7f045bf1d640 10 monclient: _renew_subs 2024-06-03T09:06:33.278 INFO:teuthology.orchestra.run.smithi088.stderr:2024-06-03T09:06:33.274+0000 7f045bf1d640 10 monclient: _send_mon_message to mon.b at v1:172.21.15.192:6789/0 2024-06-03T09:06:33.278 INFO:teuthology.orchestra.run.smithi088.stderr:2024-06-03T09:06:33.274+0000 7f045bf1d640 1 -- 172.21.15.88:0/1352477224 --> v1:172.21.15.192:6789/0 -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f04541d32e0 con 0x7f04541383e0 2024-06-03T09:06:33.278 INFO:teuthology.orchestra.run.smithi088.stderr:2024-06-03T09:06:33.274+0000 7f045bf1d640 10 client.30226.objecter _maybe_request_map subscribing (onetime) to next osd map 2024-06-03T09:06:33.278 INFO:teuthology.orchestra.run.smithi088.stderr:2024-06-03T09:06:33.274+0000 7f045bf1d640 10 monclient: _renew_subs 2024-06-03T09:06:33.278 INFO:teuthology.orchestra.run.smithi088.stderr:2024-06-03T09:06:33.274+0000 7f045bf1d640 10 monclient: _send_mon_message to mon.b at v1:172.21.15.192:6789/0 2024-06-03T09:06:33.278 INFO:teuthology.orchestra.run.smithi088.stderr:2024-06-03T09:06:33.274+0000 7f045bf1d640 1 -- 172.21.15.88:0/1352477224 --> v1:172.21.15.192:6789/0 -- mon_subscribe({osdmap=0}) v3 -- 0x7f045413db50 con 0x7f04541383e0 2024-06-03T09:06:33.278 INFO:teuthology.orchestra.run.smithi088.stderr:2024-06-03T09:06:33.274+0000 7f045bf1d640 1 librados: init done 2024-06-03T09:06:33.278 INFO:teuthology.orchestra.run.smithi088.stderr:2024-06-03T09:06:33.275+0000 7f045bf1d640 10 start_mon_command cmd=[{"prefix": "get_command_descriptions"}] 2024-06-03T09:06:33.279 INFO:teuthology.orchestra.run.smithi088.stderr:2024-06-03T09:06:33.275+0000 7f045bf1d640 10 monclient: _send_command 1 [{"prefix": "get_command_descriptions"}] 2024-06-03T09:06:33.279 INFO:teuthology.orchestra.run.smithi088.stderr:2024-06-03T09:06:33.275+0000 7f045bf1d640 10 monclient: _send_mon_message to mon.b at v1:172.21.15.192:6789/0 2024-06-03T09:06:33.279 INFO:teuthology.orchestra.run.smithi088.stderr:2024-06-03T09:06:33.275+0000 7f045bf1d640 1 -- 172.21.15.88:0/1352477224 --> v1:172.21.15.192:6789/0 -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f045413afb0 con 0x7f04541383e0 2024-06-03T09:06:33.483 INFO:teuthology.orchestra.run.smithi088.stderr:2024-06-03T09:06:33.483+0000 7f044a7fc640 1 -- 172.21.15.88:0/1352477224 <== mon.1 v1:172.21.15.192:6789/0 5 ==== config(0 keys) v1 ==== 4+0+0 (unknown 0 0 0) 0x7f044c002080 con 0x7f04541383e0 2024-06-03T09:06:33.483 INFO:teuthology.orchestra.run.smithi088.stderr:2024-06-03T09:06:33.483+0000 7f044a7fc640 10 monclient: handle_config config(0 keys) v1 2024-06-03T09:06:33.483 INFO:teuthology.orchestra.run.smithi088.stderr:2024-06-03T09:06:33.483+0000 7f044bfff640 4 set_mon_vals no callback set 2024-06-03T09:06:33.523 INFO:teuthology.orchestra.run.smithi088.stderr:2024-06-03T09:06:33.523+0000 7f044a7fc640 1 -- 172.21.15.88:0/1352477224 <== mon.1 v1:172.21.15.192:6789/0 6 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 2234374065 0 0) 0x7f044c003b30 con 0x7f04541383e0 2024-06-03T09:06:33.523 INFO:teuthology.orchestra.run.smithi088.stderr:2024-06-03T09:06:33.523+0000 7f044a7fc640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-06-03T09:06:33.523 INFO:teuthology.orchestra.run.smithi088.stderr:2024-06-03T09:06:33.523+0000 7f044a7fc640 10 monclient: got monmap 1 from mon.b (according to old e1) 2024-06-03T09:06:33.523 INFO:teuthology.orchestra.run.smithi088.stderr:2024-06-03T09:06:33.523+0000 7f044a7fc640 10 monclient: dump: 2024-06-03T09:06:33.523 INFO:teuthology.orchestra.run.smithi088.stderr:epoch 1 2024-06-03T09:06:33.523 INFO:teuthology.orchestra.run.smithi088.stderr:fsid ec3c3fdd-3ac6-4d7d-9412-683130ed8b04 2024-06-03T09:06:33.523 INFO:teuthology.orchestra.run.smithi088.stderr:last_changed 2024-06-03T08:34:31.885125+0000 2024-06-03T09:06:33.523 INFO:teuthology.orchestra.run.smithi088.stderr:created 2024-06-03T08:34:31.885125+0000 2024-06-03T09:06:33.523 INFO:teuthology.orchestra.run.smithi088.stderr:min_mon_release 18 (reef) 2024-06-03T09:06:33.524 INFO:teuthology.orchestra.run.smithi088.stderr:election_strategy: 3 2024-06-03T09:06:33.524 INFO:teuthology.orchestra.run.smithi088.stderr:0: v1:172.21.15.88:6789/0 mon.a 2024-06-03T09:06:33.524 INFO:teuthology.orchestra.run.smithi088.stderr:1: v1:172.21.15.192:6789/0 mon.b 2024-06-03T09:06:33.524 INFO:teuthology.orchestra.run.smithi088.stderr:2: v1:172.21.15.88:6790/0 mon.c 2024-06-03T09:06:33.524 INFO:teuthology.orchestra.run.smithi088.stderr: 2024-06-03T09:06:33.530 INFO:teuthology.orchestra.run.smithi088.stderr:2024-06-03T09:06:33.530+0000 7f044a7fc640 1 -- 172.21.15.88:0/1352477224 <== mon.1 v1:172.21.15.192:6789/0 7 ==== mgrmap(e 5) v1 ==== 89110+0+0 (unknown 873533973 0 0) 0x7f044c019930 con 0x7f04541383e0 2024-06-03T09:06:33.531 INFO:teuthology.orchestra.run.smithi088.stderr:2024-06-03T09:06:33.531+0000 7f0459cb9640 10 monclient: get_auth_request con 0x7f043806d340 auth_method 0 2024-06-03T09:06:33.531 INFO:teuthology.orchestra.run.smithi088.stderr:2024-06-03T09:06:33.531+0000 7f044a7fc640 10 client.30226.objecter ms_handle_connect 0x7f043806d340 2024-06-03T09:06:33.534 INFO:teuthology.orchestra.run.smithi088.stderr:2024-06-03T09:06:33.534+0000 7f044a7fc640 1 -- 172.21.15.88:0/1352477224 <== mon.1 v1:172.21.15.192:6789/0 8 ==== osd_map(1010..1010 src has 622..1010) v4 ==== 5229+0+0 (unknown 1105883993 0 0) 0x7f044c053160 con 0x7f04541383e0 2024-06-03T09:06:33.534 INFO:teuthology.orchestra.run.smithi088.stderr:2024-06-03T09:06:33.534+0000 7f044a7fc640 10 client.30226.objecter ms_dispatch 0x7f0454000b90 osd_map(1010..1010 src has 622..1010) v4 2024-06-03T09:06:33.534 INFO:teuthology.orchestra.run.smithi088.stderr:2024-06-03T09:06:33.534+0000 7f044a7fc640 3 client.30226.objecter handle_osd_map got epochs [1010,1010] > 0 2024-06-03T09:06:33.534 INFO:teuthology.orchestra.run.smithi088.stderr:2024-06-03T09:06:33.534+0000 7f044a7fc640 3 client.30226.objecter handle_osd_map decoding full epoch 1010 2024-06-03T09:06:33.535 INFO:teuthology.orchestra.run.smithi088.stderr:2024-06-03T09:06:33.534+0000 7f044a7fc640 20 client.30226.objecter dump_active .. 0 homeless 2024-06-03T09:06:33.536 INFO:teuthology.orchestra.run.smithi088.stderr:2024-06-03T09:06:33.537+0000 7f044a7fc640 1 -- 172.21.15.88:0/1352477224 <== mon.1 v1:172.21.15.192:6789/0 9 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+182498 (unknown 1092875540 0 3016760700) 0x7f044c04eb40 con 0x7f04541383e0 2024-06-03T09:06:33.537 INFO:teuthology.orchestra.run.smithi088.stderr:2024-06-03T09:06:33.537+0000 7f044a7fc640 10 monclient: handle_mon_command_ack 1 [{"prefix": "get_command_descriptions"}] 2024-06-03T09:06:33.537 INFO:teuthology.orchestra.run.smithi088.stderr:2024-06-03T09:06:33.537+0000 7f044a7fc640 10 monclient: _finish_command 1 = system:0 2024-06-03T09:06:33.593 INFO:tasks.ceph.mon.a.smithi088.stderr:==00:00:31:28.919 34743== Warning: unimplemented fcntl command: 1036 2024-06-03T09:06:33.596 INFO:tasks.ceph.mon.a.smithi088.stderr:==00:00:31:28.922 34743== Warning: unimplemented fcntl command: 1036 2024-06-03T09:06:33.609 INFO:tasks.ceph.mon.a.smithi088.stderr:==00:00:31:28.935 34743== Warning: unimplemented fcntl command: 1036 2024-06-03T09:06:33.645 INFO:tasks.ceph.mon.b.smithi192.stderr:==00:00:31:28.947 34228== Warning: unimplemented fcntl command: 1036 2024-06-03T09:06:33.646 INFO:tasks.ceph.mon.b.smithi192.stderr:==00:00:31:28.949 34228== Warning: unimplemented fcntl command: 1036 2024-06-03T09:06:33.647 INFO:tasks.ceph.mon.c.smithi088.stderr:==00:00:31:28.973 34749== Warning: unimplemented fcntl command: 1036 2024-06-03T09:06:33.649 INFO:tasks.ceph.mon.c.smithi088.stderr:==00:00:31:28.974 34749== Warning: unimplemented fcntl command: 1036 2024-06-03T09:06:33.658 INFO:tasks.ceph.mon.b.smithi192.stderr:==00:00:31:28.961 34228== Warning: unimplemented fcntl command: 1036 2024-06-03T09:06:33.663 INFO:tasks.ceph.mon.c.smithi088.stderr:==00:00:31:28.989 34749== Warning: unimplemented fcntl command: 1036 2024-06-03T09:06:33.718 INFO:teuthology.orchestra.run.smithi088.stderr:2024-06-03T09:06:33.718+0000 7f045bf1d640 1 -- 172.21.15.88:0/1352477224 --> v1:172.21.15.192:6800/34229 -- mgr_command(tid 0: {"prefix": "pg dump", "target": ["mon-mgr", ""], "format": "json"}) v1 -- 0x7f04540adf70 con 0x7f043806d340 2024-06-03T09:06:33.722 INFO:teuthology.orchestra.run.smithi088.stderr:2024-06-03T09:06:33.723+0000 7f044a7fc640 1 -- 172.21.15.88:0/1352477224 <== mgr.4110 v1:172.21.15.192:6800/34229 1 ==== mgr_command_reply(tid 0: 0 dumped all) v1 ==== 18+0+151421 (unknown 2965378022 0 216613864) 0x7f0450007810 con 0x7f043806d340 2024-06-03T09:06:33.724 INFO:teuthology.orchestra.run.smithi088.stdout: 2024-06-03T09:06:33.725 INFO:teuthology.orchestra.run.smithi088.stderr:dumped all 2024-06-03T09:06:33.727 INFO:teuthology.orchestra.run.smithi088.stderr:2024-06-03T09:06:33.725+0000 7f045bf1d640 10 librados: watch_flush enter 2024-06-03T09:06:33.727 INFO:teuthology.orchestra.run.smithi088.stderr:2024-06-03T09:06:33.725+0000 7f045bf1d640 10 librados: watch_flush exit 2024-06-03T09:06:33.727 INFO:teuthology.orchestra.run.smithi088.stderr:2024-06-03T09:06:33.725+0000 7f045bf1d640 20 client.30226.objecter shutdown clearing up homeless session... 2024-06-03T09:06:33.727 INFO:teuthology.orchestra.run.smithi088.stderr:2024-06-03T09:06:33.725+0000 7f045bf1d640 10 client.30226.objecter successfully canceled tick 2024-06-03T09:06:33.727 INFO:teuthology.orchestra.run.smithi088.stderr:2024-06-03T09:06:33.725+0000 7f045bf1d640 1 -- 172.21.15.88:0/1352477224 >> v1:172.21.15.192:6800/34229 conn(0x7f043806d340 legacy=0x7f043806f800 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-06-03T09:06:33.727 INFO:teuthology.orchestra.run.smithi088.stderr:2024-06-03T09:06:33.725+0000 7f045bf1d640 10 monclient: shutdown 2024-06-03T09:06:33.727 INFO:teuthology.orchestra.run.smithi088.stderr:2024-06-03T09:06:33.725+0000 7f045bf1d640 20 monclient: shutdown discarding 0 pending message(s) 2024-06-03T09:06:33.727 INFO:teuthology.orchestra.run.smithi088.stderr:2024-06-03T09:06:33.725+0000 7f045bf1d640 1 -- 172.21.15.88:0/1352477224 >> v1:172.21.15.192:6789/0 conn(0x7f04541383e0 legacy=0x7f0454133390 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-06-03T09:06:33.727 INFO:teuthology.orchestra.run.smithi088.stderr:2024-06-03T09:06:33.725+0000 7f045bf1d640 1 -- 172.21.15.88:0/1352477224 shutdown_connections 2024-06-03T09:06:33.727 INFO:teuthology.orchestra.run.smithi088.stderr:2024-06-03T09:06:33.726+0000 7f045bf1d640 1 -- 172.21.15.88:0/1352477224 >> 172.21.15.88:0/1352477224 conn(0x7f04540b2f20 msgr2=0x7f04540ae150 unknown :-1 s=STATE_NONE l=0).mark_down 2024-06-03T09:06:33.727 INFO:teuthology.orchestra.run.smithi088.stderr:2024-06-03T09:06:33.726+0000 7f045bf1d640 1 -- 172.21.15.88:0/1352477224 shutdown_connections 2024-06-03T09:06:33.727 INFO:teuthology.orchestra.run.smithi088.stderr:2024-06-03T09:06:33.726+0000 7f045bf1d640 1 -- 172.21.15.88:0/1352477224 wait complete. 2024-06-03T09:06:33.728 INFO:teuthology.orchestra.run.smithi088.stderr:2024-06-03T09:06:33.726+0000 7f045bf1d640 1 librados: shutdown 2024-06-03T09:06:33.736 INFO:teuthology.orchestra.run.smithi088.stdout:{"pg_ready":true,"pg_map":{"version":1922,"stamp":"2024-06-03T09:06:33.101045+0000","last_osdmap_epoch":0,"last_pg_scan":0,"pg_stats_sum":{"stat_sum":{"num_bytes":459285,"num_objects":3,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":1,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":3,"num_whiteouts":0,"num_read":102,"num_read_kb":88,"num_write":120,"num_write_kb":1379,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2,"num_bytes_recovered":397840,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":83,"ondisk_log_size":83,"up":68,"acting":68,"num_store_stats":0},"osd_stats_sum":{"up_from":0,"seq":0,"num_pgs":100,"num_osds":8,"num_per_pool_osds":8,"num_per_pool_omap_osds":8,"kb":754974720,"kb_used":494548,"kb_used_data":46324,"kb_used_omap":12,"kb_used_meta":448115,"kb_avail":754480172,"statfs":{"total":773094113280,"available":772587696128,"internally_reserved":0,"allocated":47435776,"data_stored":22747954,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":12744,"internal_metadata":458870328},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":109,"apply_latency_ms":109,"commit_latency_ns":109000000,"apply_latency_ns":109000000},"alerts":[],"network_ping_times":[]},"pg_stats_delta":{"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":0,"acting":0,"num_store_stats":0,"stamp_delta":"12.002380"},"pg_stats":[{"pgid":"17.d","version":"0'0","reported_seq":942,"reported_epoch":1009,"state":"active+clean","last_fresh":"2024-06-03T09:06:14.443917+0000","last_change":"2024-06-03T08:38:19.260548+0000","last_active":"2024-06-03T09:06:14.443917+0000","last_peered":"2024-06-03T09:06:14.443917+0000","last_clean":"2024-06-03T09:06:14.443917+0000","last_became_active":"2024-06-03T08:38:19.253559+0000","last_became_peered":"2024-06-03T08:38:19.253559+0000","last_unstale":"2024-06-03T09:06:14.443917+0000","last_undegraded":"2024-06-03T09:06:14.443917+0000","last_fullsized":"2024-06-03T09:06:14.443917+0000","mapping_epoch":28,"log_start":"0'0","ondisk_log_start":"0'0","created":28,"last_epoch_clean":45,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-06-03T08:37:55.614497+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-06-03T08:37:55.614497+0000","last_clean_scrub_stamp":"2024-06-03T08:37:55.614497+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-06-04T19:00:12.470979+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1],"acting":[0,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"17.c","version":"0'0","reported_seq":953,"reported_epoch":1009,"state":"active+undersized","last_fresh":"2024-06-03T09:06:13.668156+0000","last_change":"2024-06-03T09:06:03.440074+0000","last_active":"2024-06-03T09:06:13.668156+0000","last_peered":"2024-06-03T09:06:13.668156+0000","last_clean":"2024-06-03T09:05:02.105638+0000","last_became_active":"2024-06-03T09:06:03.440074+0000","last_became_peered":"2024-06-03T09:06:03.440074+0000","last_unstale":"2024-06-03T09:06:13.668156+0000","last_undegraded":"2024-06-03T09:06:13.668156+0000","last_fullsized":"2024-06-03T09:06:03.377982+0000","mapping_epoch":999,"log_start":"0'0","ondisk_log_start":"0'0","created":28,"last_epoch_clean":255,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-06-03T08:37:55.614497+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-06-03T08:37:55.614497+0000","last_clean_scrub_stamp":"2024-06-03T08:37:55.614497+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-06-04T10:34:20.808557+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3],"acting":[3],"avail_no_missing":["3"],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"17.f","version":"0'0","reported_seq":148,"reported_epoch":1009,"state":"active+undersized","last_fresh":"2024-06-03T09:06:14.450543+0000","last_change":"2024-06-03T09:06:03.420285+0000","last_active":"2024-06-03T09:06:14.450543+0000","last_peered":"2024-06-03T09:06:14.450543+0000","last_clean":"2024-06-03T08:37:55.614497+0000","last_became_active":"2024-06-03T09:06:03.420285+0000","last_became_peered":"2024-06-03T09:06:03.420285+0000","last_unstale":"2024-06-03T09:06:14.450543+0000","last_undegraded":"2024-06-03T09:06:14.450543+0000","last_fullsized":"2024-06-03T09:06:03.380616+0000","mapping_epoch":999,"log_start":"0'0","ondisk_log_start":"0'0","created":28,"last_epoch_clean":255,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-06-03T08:37:55.614497+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-06-03T08:37:55.614497+0000","last_clean_scrub_stamp":"2024-06-03T08:37:55.614497+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-06-04T15:39:41.187731+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4],"acting":[4],"avail_no_missing":["4"],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"17.e","version":"0'0","reported_seq":149,"reported_epoch":1009,"state":"active+undersized","last_fresh":"2024-06-03T09:06:13.726992+0000","last_change":"2024-06-03T09:06:03.412887+0000","last_active":"2024-06-03T09:06:13.726992+0000","last_peered":"2024-06-03T09:06:13.726992+0000","last_clean":"2024-06-03T08:37:55.614497+0000","last_became_active":"2024-06-03T09:06:03.412887+0000","last_became_peered":"2024-06-03T09:06:03.412887+0000","last_unstale":"2024-06-03T09:06:13.726992+0000","last_undegraded":"2024-06-03T09:06:13.726992+0000","last_fullsized":"2024-06-03T09:06:03.332482+0000","mapping_epoch":999,"log_start":"0'0","ondisk_log_start":"0'0","created":28,"last_epoch_clean":255,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-06-03T08:37:55.614497+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-06-03T08:37:55.614497+0000","last_clean_scrub_stamp":"2024-06-03T08:37:55.614497+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-06-04T12:23:22.511601+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5],"acting":[5],"avail_no_missing":["5"],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"17.9","version":"0'0","reported_seq":946,"reported_epoch":1009,"state":"active+clean","last_fresh":"2024-06-03T09:06:14.999299+0000","last_change":"2024-06-03T08:38:04.772260+0000","last_active":"2024-06-03T09:06:14.999299+0000","last_peered":"2024-06-03T09:06:14.999299+0000","last_clean":"2024-06-03T09:06:14.999299+0000","last_became_active":"2024-06-03T08:38:04.763428+0000","last_became_peered":"2024-06-03T08:38:04.763428+0000","last_unstale":"2024-06-03T09:06:14.999299+0000","last_undegraded":"2024-06-03T09:06:14.999299+0000","last_fullsized":"2024-06-03T09:06:14.999299+0000","mapping_epoch":28,"log_start":"0'0","ondisk_log_start":"0'0","created":28,"last_epoch_clean":33,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-06-03T08:37:55.614497+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-06-03T08:37:55.614497+0000","last_clean_scrub_stamp":"2024-06-03T08:37:55.614497+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-06-04T10:55:10.773936+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,4],"acting":[7,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"17.8","version":"0'0","reported_seq":961,"reported_epoch":1009,"state":"active+undersized","last_fresh":"2024-06-03T09:06:14.430375+0000","last_change":"2024-06-03T09:06:03.388786+0000","last_active":"2024-06-03T09:06:14.430375+0000","last_peered":"2024-06-03T09:06:14.430375+0000","last_clean":"2024-06-03T09:04:43.297780+0000","last_became_active":"2024-06-03T09:06:03.388786+0000","last_became_peered":"2024-06-03T09:06:03.388786+0000","last_unstale":"2024-06-03T09:06:14.430375+0000","last_undegraded":"2024-06-03T09:06:14.430375+0000","last_fullsized":"2024-06-03T09:06:03.331754+0000","mapping_epoch":999,"log_start":"0'0","ondisk_log_start":"0'0","created":28,"last_epoch_clean":255,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-06-03T08:37:55.614497+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-06-03T08:37:55.614497+0000","last_clean_scrub_stamp":"2024-06-03T08:37:55.614497+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-06-04T09:18:44.765547+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0],"acting":[0],"avail_no_missing":["0"],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"17.b","version":"0'0","reported_seq":149,"reported_epoch":1009,"state":"active+undersized","last_fresh":"2024-06-03T09:06:14.433735+0000","last_change":"2024-06-03T09:06:03.451203+0000","last_active":"2024-06-03T09:06:14.433735+0000","last_peered":"2024-06-03T09:06:14.433735+0000","last_clean":"2024-06-03T08:37:55.614497+0000","last_became_active":"2024-06-03T09:06:03.451203+0000","last_became_peered":"2024-06-03T09:06:03.451203+0000","last_unstale":"2024-06-03T09:06:14.433735+0000","last_undegraded":"2024-06-03T09:06:14.433735+0000","last_fullsized":"2024-06-03T09:06:03.375523+0000","mapping_epoch":999,"log_start":"0'0","ondisk_log_start":"0'0","created":28,"last_epoch_clean":255,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-06-03T08:37:55.614497+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-06-03T08:37:55.614497+0000","last_clean_scrub_stamp":"2024-06-03T08:37:55.614497+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-06-04T10:16:19.813495+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2],"acting":[2],"avail_no_missing":["2"],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"17.a","version":"0'0","reported_seq":925,"reported_epoch":1010,"state":"active+clean","last_fresh":"2024-06-03T09:06:18.651697+0000","last_change":"2024-06-03T08:38:05.843142+0000","last_active":"2024-06-03T09:06:18.651697+0000","last_peered":"2024-06-03T09:06:18.651697+0000","last_clean":"2024-06-03T09:06:18.651697+0000","last_became_active":"2024-06-03T08:38:05.809257+0000","last_became_peered":"2024-06-03T08:38:05.809257+0000","last_unstale":"2024-06-03T09:06:18.651697+0000","last_undegraded":"2024-06-03T09:06:18.651697+0000","last_fullsized":"2024-06-03T09:06:18.651697+0000","mapping_epoch":28,"log_start":"0'0","ondisk_log_start":"0'0","created":28,"last_epoch_clean":33,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-06-03T08:37:55.614497+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-06-03T08:37:55.614497+0000","last_clean_scrub_stamp":"2024-06-03T08:37:55.614497+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-06-04T12:12:03.686838+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2],"acting":[1,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"17.5","version":"0'0","reported_seq":943,"reported_epoch":1009,"state":"active+clean","last_fresh":"2024-06-03T09:06:14.442357+0000","last_change":"2024-06-03T08:38:23.466051+0000","last_active":"2024-06-03T09:06:14.442357+0000","last_peered":"2024-06-03T09:06:14.442357+0000","last_clean":"2024-06-03T09:06:14.442357+0000","last_became_active":"2024-06-03T08:38:23.458039+0000","last_became_peered":"2024-06-03T08:38:23.458039+0000","last_unstale":"2024-06-03T09:06:14.442357+0000","last_undegraded":"2024-06-03T09:06:14.442357+0000","last_fullsized":"2024-06-03T09:06:14.442357+0000","mapping_epoch":28,"log_start":"0'0","ondisk_log_start":"0'0","created":28,"last_epoch_clean":49,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-06-03T08:37:55.614497+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-06-03T08:37:55.614497+0000","last_clean_scrub_stamp":"2024-06-03T08:37:55.614497+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-06-04T12:27:02.532436+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,4],"acting":[0,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"17.4","version":"0'0","reported_seq":149,"reported_epoch":1009,"state":"active+undersized","last_fresh":"2024-06-03T09:06:13.941522+0000","last_change":"2024-06-03T09:06:03.441889+0000","last_active":"2024-06-03T09:06:13.941522+0000","last_peered":"2024-06-03T09:06:13.941522+0000","last_clean":"2024-06-03T08:37:55.614497+0000","last_became_active":"2024-06-03T09:06:03.441889+0000","last_became_peered":"2024-06-03T09:06:03.441889+0000","last_unstale":"2024-06-03T09:06:13.941522+0000","last_undegraded":"2024-06-03T09:06:13.941522+0000","last_fullsized":"2024-06-03T09:06:03.359582+0000","mapping_epoch":999,"log_start":"0'0","ondisk_log_start":"0'0","created":28,"last_epoch_clean":255,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-06-03T08:37:55.614497+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-06-03T08:37:55.614497+0000","last_clean_scrub_stamp":"2024-06-03T08:37:55.614497+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-06-04T12:49:22.907141+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7],"acting":[7],"avail_no_missing":["7"],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"17.7","version":"0'0","reported_seq":929,"reported_epoch":1009,"state":"active+clean","last_fresh":"2024-06-03T09:06:13.736920+0000","last_change":"2024-06-03T08:38:04.837083+0000","last_active":"2024-06-03T09:06:13.736920+0000","last_peered":"2024-06-03T09:06:13.736920+0000","last_clean":"2024-06-03T09:06:13.736920+0000","last_became_active":"2024-06-03T08:38:04.818246+0000","last_became_peered":"2024-06-03T08:38:04.818246+0000","last_unstale":"2024-06-03T09:06:13.736920+0000","last_undegraded":"2024-06-03T09:06:13.736920+0000","last_fullsized":"2024-06-03T09:06:13.736920+0000","mapping_epoch":28,"log_start":"0'0","ondisk_log_start":"0'0","created":28,"last_epoch_clean":34,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-06-03T08:37:55.614497+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-06-03T08:37:55.614497+0000","last_clean_scrub_stamp":"2024-06-03T08:37:55.614497+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-06-04T17:02:38.204813+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,2],"acting":[5,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"17.6","version":"0'0","reported_seq":960,"reported_epoch":1009,"state":"active+undersized","last_fresh":"2024-06-03T09:06:14.453110+0000","last_change":"2024-06-03T09:06:03.428677+0000","last_active":"2024-06-03T09:06:14.453110+0000","last_peered":"2024-06-03T09:06:14.453110+0000","last_clean":"2024-06-03T09:04:43.312352+0000","last_became_active":"2024-06-03T09:06:03.428677+0000","last_became_peered":"2024-06-03T09:06:03.428677+0000","last_unstale":"2024-06-03T09:06:14.453110+0000","last_undegraded":"2024-06-03T09:06:14.453110+0000","last_fullsized":"2024-06-03T09:06:03.359909+0000","mapping_epoch":999,"log_start":"0'0","ondisk_log_start":"0'0","created":28,"last_epoch_clean":255,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-06-03T08:37:55.614497+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-06-03T08:37:55.614497+0000","last_clean_scrub_stamp":"2024-06-03T08:37:55.614497+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-06-04T15:38:02.180241+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0],"acting":[0],"avail_no_missing":["0"],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"2.7","version":"0'0","reported_seq":940,"reported_epoch":1009,"state":"active+clean","last_fresh":"2024-06-03T09:06:13.684065+0000","last_change":"2024-06-03T08:36:30.385405+0000","last_active":"2024-06-03T09:06:13.684065+0000","last_peered":"2024-06-03T09:06:13.684065+0000","last_clean":"2024-06-03T09:06:13.684065+0000","last_became_active":"2024-06-03T08:36:30.380302+0000","last_became_peered":"2024-06-03T08:36:30.380302+0000","last_unstale":"2024-06-03T09:06:13.684065+0000","last_undegraded":"2024-06-03T09:06:13.684065+0000","last_fullsized":"2024-06-03T09:06:13.684065+0000","mapping_epoch":23,"log_start":"0'0","ondisk_log_start":"0'0","created":23,"last_epoch_clean":24,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-06-03T08:36:28.444764+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-06-03T08:36:28.444764+0000","last_clean_scrub_stamp":"2024-06-03T08:36:28.444764+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-06-04T18:46:33.475181+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0],"acting":[3,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"17.14","version":"0'0","reported_seq":926,"reported_epoch":1009,"state":"active+clean","last_fresh":"2024-06-03T09:06:13.721460+0000","last_change":"2024-06-03T08:38:04.547565+0000","last_active":"2024-06-03T09:06:13.721460+0000","last_peered":"2024-06-03T09:06:13.721460+0000","last_clean":"2024-06-03T09:06:13.721460+0000","last_became_active":"2024-06-03T08:38:04.541781+0000","last_became_peered":"2024-06-03T08:38:04.541781+0000","last_unstale":"2024-06-03T09:06:13.721460+0000","last_undegraded":"2024-06-03T09:06:13.721460+0000","last_fullsized":"2024-06-03T09:06:13.721460+0000","mapping_epoch":28,"log_start":"0'0","ondisk_log_start":"0'0","created":28,"last_epoch_clean":34,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-06-03T08:37:55.614497+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-06-03T08:37:55.614497+0000","last_clean_scrub_stamp":"2024-06-03T08:37:55.614497+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-06-04T16:41:55.170134+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,2],"acting":[5,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"2.6","version":"0'0","reported_seq":949,"reported_epoch":1009,"state":"active+clean","last_fresh":"2024-06-03T09:06:14.448100+0000","last_change":"2024-06-03T08:36:30.430760+0000","last_active":"2024-06-03T09:06:14.448100+0000","last_peered":"2024-06-03T09:06:14.448100+0000","last_clean":"2024-06-03T09:06:14.448100+0000","last_became_active":"2024-06-03T08:36:30.404665+0000","last_became_peered":"2024-06-03T08:36:30.404665+0000","last_unstale":"2024-06-03T09:06:14.448100+0000","last_undegraded":"2024-06-03T09:06:14.448100+0000","last_fullsized":"2024-06-03T09:06:14.448100+0000","mapping_epoch":23,"log_start":"0'0","ondisk_log_start":"0'0","created":23,"last_epoch_clean":24,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-06-03T08:36:28.444764+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-06-03T08:36:28.444764+0000","last_clean_scrub_stamp":"2024-06-03T08:36:28.444764+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-06-04T10:37:37.746074+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,0],"acting":[4,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"17.15","version":"0'0","reported_seq":934,"reported_epoch":1009,"state":"active+clean","last_fresh":"2024-06-03T09:06:13.672940+0000","last_change":"2024-06-03T08:38:20.999903+0000","last_active":"2024-06-03T09:06:13.672940+0000","last_peered":"2024-06-03T09:06:13.672940+0000","last_clean":"2024-06-03T09:06:13.672940+0000","last_became_active":"2024-06-03T08:38:20.979523+0000","last_became_peered":"2024-06-03T08:38:20.979523+0000","last_unstale":"2024-06-03T09:06:13.672940+0000","last_undegraded":"2024-06-03T09:06:13.672940+0000","last_fullsized":"2024-06-03T09:06:13.672940+0000","mapping_epoch":28,"log_start":"0'0","ondisk_log_start":"0'0","created":28,"last_epoch_clean":47,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-06-03T08:37:55.614497+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-06-03T08:37:55.614497+0000","last_clean_scrub_stamp":"2024-06-03T08:37:55.614497+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-06-04T20:29:58.314289+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,1],"acting":[3,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"2.5","version":"0'0","reported_seq":942,"reported_epoch":1010,"state":"active+clean","last_fresh":"2024-06-03T09:06:18.382767+0000","last_change":"2024-06-03T08:36:30.024235+0000","last_active":"2024-06-03T09:06:18.382767+0000","last_peered":"2024-06-03T09:06:18.382767+0000","last_clean":"2024-06-03T09:06:18.382767+0000","last_became_active":"2024-06-03T08:36:30.019040+0000","last_became_peered":"2024-06-03T08:36:30.019040+0000","last_unstale":"2024-06-03T09:06:18.382767+0000","last_undegraded":"2024-06-03T09:06:18.382767+0000","last_fullsized":"2024-06-03T09:06:18.382767+0000","mapping_epoch":23,"log_start":"0'0","ondisk_log_start":"0'0","created":23,"last_epoch_clean":24,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-06-03T08:36:28.444764+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-06-03T08:36:28.444764+0000","last_clean_scrub_stamp":"2024-06-03T08:36:28.444764+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-06-04T15:46:47.109502+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,4],"acting":[3,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"17.16","version":"0'0","reported_seq":972,"reported_epoch":1009,"state":"active+undersized","last_fresh":"2024-06-03T09:06:14.441665+0000","last_change":"2024-06-03T09:06:03.478615+0000","last_active":"2024-06-03T09:06:14.441665+0000","last_peered":"2024-06-03T09:06:14.441665+0000","last_clean":"2024-06-03T09:04:43.328415+0000","last_became_active":"2024-06-03T09:06:03.478615+0000","last_became_peered":"2024-06-03T09:06:03.478615+0000","last_unstale":"2024-06-03T09:06:14.441665+0000","last_undegraded":"2024-06-03T09:06:14.441665+0000","last_fullsized":"2024-06-03T09:06:03.402504+0000","mapping_epoch":999,"log_start":"0'0","ondisk_log_start":"0'0","created":28,"last_epoch_clean":255,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-06-03T08:37:55.614497+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-06-03T08:37:55.614497+0000","last_clean_scrub_stamp":"2024-06-03T08:37:55.614497+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-06-04T08:47:08.079233+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2],"acting":[2],"avail_no_missing":["2"],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"2.4","version":"0'0","reported_seq":952,"reported_epoch":1010,"state":"active+undersized","last_fresh":"2024-06-03T09:06:18.331485+0000","last_change":"2024-06-03T09:06:03.387171+0000","last_active":"2024-06-03T09:06:18.331485+0000","last_peered":"2024-06-03T09:06:18.331485+0000","last_clean":"2024-06-03T09:04:43.202645+0000","last_became_active":"2024-06-03T09:06:03.387171+0000","last_became_peered":"2024-06-03T09:06:03.387171+0000","last_unstale":"2024-06-03T09:06:18.331485+0000","last_undegraded":"2024-06-03T09:06:18.331485+0000","last_fullsized":"2024-06-03T09:06:03.283787+0000","mapping_epoch":999,"log_start":"0'0","ondisk_log_start":"0'0","created":23,"last_epoch_clean":255,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-06-03T08:36:28.444764+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-06-03T08:36:28.444764+0000","last_clean_scrub_stamp":"2024-06-03T08:36:28.444764+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-06-04T20:17:43.485023+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5],"acting":[5],"avail_no_missing":["5"],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"17.17","version":"0'0","reported_seq":954,"reported_epoch":1009,"state":"active+clean","last_fresh":"2024-06-03T09:06:14.447046+0000","last_change":"2024-06-03T08:38:38.483275+0000","last_active":"2024-06-03T09:06:14.447046+0000","last_peered":"2024-06-03T09:06:14.447046+0000","last_clean":"2024-06-03T09:06:14.447046+0000","last_became_active":"2024-06-03T08:38:38.467916+0000","last_became_peered":"2024-06-03T08:38:38.467916+0000","last_unstale":"2024-06-03T09:06:14.447046+0000","last_undegraded":"2024-06-03T09:06:14.447046+0000","last_fullsized":"2024-06-03T09:06:14.447046+0000","mapping_epoch":28,"log_start":"0'0","ondisk_log_start":"0'0","created":28,"last_epoch_clean":61,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-06-03T08:37:55.614497+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-06-03T08:37:55.614497+0000","last_clean_scrub_stamp":"2024-06-03T08:37:55.614497+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-06-04T10:02:23.036420+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,7],"acting":[2,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"2.2","version":"0'0","reported_seq":963,"reported_epoch":1009,"state":"active+undersized","last_fresh":"2024-06-03T09:06:13.657892+0000","last_change":"2024-06-03T09:06:03.410020+0000","last_active":"2024-06-03T09:06:13.657892+0000","last_peered":"2024-06-03T09:06:13.657892+0000","last_clean":"2024-06-03T09:05:37.145889+0000","last_became_active":"2024-06-03T09:06:03.410020+0000","last_became_peered":"2024-06-03T09:06:03.410020+0000","last_unstale":"2024-06-03T09:06:13.657892+0000","last_undegraded":"2024-06-03T09:06:13.657892+0000","last_fullsized":"2024-06-03T09:06:03.346864+0000","mapping_epoch":999,"log_start":"0'0","ondisk_log_start":"0'0","created":23,"last_epoch_clean":255,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-06-03T08:36:28.444764+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-06-03T08:36:28.444764+0000","last_clean_scrub_stamp":"2024-06-03T08:36:28.444764+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-06-04T09:57:12.423360+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3],"acting":[3],"avail_no_missing":["3"],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"17.11","version":"0'0","reported_seq":944,"reported_epoch":1009,"state":"active+clean","last_fresh":"2024-06-03T09:06:14.426670+0000","last_change":"2024-06-03T08:38:06.502302+0000","last_active":"2024-06-03T09:06:14.426670+0000","last_peered":"2024-06-03T09:06:14.426670+0000","last_clean":"2024-06-03T09:06:14.426670+0000","last_became_active":"2024-06-03T08:38:06.490204+0000","last_became_peered":"2024-06-03T08:38:06.490204+0000","last_unstale":"2024-06-03T09:06:14.426670+0000","last_undegraded":"2024-06-03T09:06:14.426670+0000","last_fullsized":"2024-06-03T09:06:14.426670+0000","mapping_epoch":28,"log_start":"0'0","ondisk_log_start":"0'0","created":28,"last_epoch_clean":32,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-06-03T08:37:55.614497+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-06-03T08:37:55.614497+0000","last_clean_scrub_stamp":"2024-06-03T08:37:55.614497+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-06-04T10:47:05.988958+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,2],"acting":[4,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"2.1","version":"0'0","reported_seq":952,"reported_epoch":1009,"state":"active+clean","last_fresh":"2024-06-03T09:06:13.929725+0000","last_change":"2024-06-03T08:36:30.883360+0000","last_active":"2024-06-03T09:06:13.929725+0000","last_peered":"2024-06-03T09:06:13.929725+0000","last_clean":"2024-06-03T09:06:13.929725+0000","last_became_active":"2024-06-03T08:36:30.855073+0000","last_became_peered":"2024-06-03T08:36:30.855073+0000","last_unstale":"2024-06-03T09:06:13.929725+0000","last_undegraded":"2024-06-03T09:06:13.929725+0000","last_fullsized":"2024-06-03T09:06:13.929725+0000","mapping_epoch":23,"log_start":"0'0","ondisk_log_start":"0'0","created":23,"last_epoch_clean":25,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-06-03T08:36:28.444764+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-06-03T08:36:28.444764+0000","last_clean_scrub_stamp":"2024-06-03T08:36:28.444764+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-06-04T18:12:22.363473+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,5],"acting":[7,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"17.12","version":"0'0","reported_seq":945,"reported_epoch":1009,"state":"active+clean","last_fresh":"2024-06-03T09:06:13.935141+0000","last_change":"2024-06-03T08:38:01.751562+0000","last_active":"2024-06-03T09:06:13.935141+0000","last_peered":"2024-06-03T09:06:13.935141+0000","last_clean":"2024-06-03T09:06:13.935141+0000","last_became_active":"2024-06-03T08:38:01.745027+0000","last_became_peered":"2024-06-03T08:38:01.745027+0000","last_unstale":"2024-06-03T09:06:13.935141+0000","last_undegraded":"2024-06-03T09:06:13.935141+0000","last_fullsized":"2024-06-03T09:06:13.935141+0000","mapping_epoch":28,"log_start":"0'0","ondisk_log_start":"0'0","created":28,"last_epoch_clean":32,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-06-03T08:37:55.614497+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-06-03T08:37:55.614497+0000","last_clean_scrub_stamp":"2024-06-03T08:37:55.614497+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-06-04T17:54:43.785326+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,3],"acting":[7,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"2.0","version":"0'0","reported_seq":943,"reported_epoch":1009,"state":"active+clean","last_fresh":"2024-06-03T09:06:13.648112+0000","last_change":"2024-06-03T08:36:29.984096+0000","last_active":"2024-06-03T09:06:13.648112+0000","last_peered":"2024-06-03T09:06:13.648112+0000","last_clean":"2024-06-03T09:06:13.648112+0000","last_became_active":"2024-06-03T08:36:29.957875+0000","last_became_peered":"2024-06-03T08:36:29.957875+0000","last_unstale":"2024-06-03T09:06:13.648112+0000","last_undegraded":"2024-06-03T09:06:13.648112+0000","last_fullsized":"2024-06-03T09:06:13.648112+0000","mapping_epoch":23,"log_start":"0'0","ondisk_log_start":"0'0","created":23,"last_epoch_clean":24,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-06-03T08:36:28.444764+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-06-03T08:36:28.444764+0000","last_clean_scrub_stamp":"2024-06-03T08:36:28.444764+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-06-04T08:41:33.679479+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,5],"acting":[3,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"17.13","version":"0'0","reported_seq":943,"reported_epoch":1009,"state":"active+clean","last_fresh":"2024-06-03T09:06:14.438399+0000","last_change":"2024-06-03T08:38:26.838316+0000","last_active":"2024-06-03T09:06:14.438399+0000","last_peered":"2024-06-03T09:06:14.438399+0000","last_clean":"2024-06-03T09:06:14.438399+0000","last_became_active":"2024-06-03T08:38:26.833214+0000","last_became_peered":"2024-06-03T08:38:26.833214+0000","last_unstale":"2024-06-03T09:06:14.438399+0000","last_undegraded":"2024-06-03T09:06:14.438399+0000","last_fullsized":"2024-06-03T09:06:14.438399+0000","mapping_epoch":28,"log_start":"0'0","ondisk_log_start":"0'0","created":28,"last_epoch_clean":52,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-06-03T08:37:55.614497+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-06-03T08:37:55.614497+0000","last_clean_scrub_stamp":"2024-06-03T08:37:55.614497+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-06-04T12:58:51.617998+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,0],"acting":[4,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"2.3","version":"0'0","reported_seq":943,"reported_epoch":1010,"state":"active+clean","last_fresh":"2024-06-03T09:06:18.362614+0000","last_change":"2024-06-03T08:36:30.351670+0000","last_active":"2024-06-03T09:06:18.362614+0000","last_peered":"2024-06-03T09:06:18.362614+0000","last_clean":"2024-06-03T09:06:18.362614+0000","last_became_active":"2024-06-03T08:36:30.343560+0000","last_became_peered":"2024-06-03T08:36:30.343560+0000","last_unstale":"2024-06-03T09:06:18.362614+0000","last_undegraded":"2024-06-03T09:06:18.362614+0000","last_fullsized":"2024-06-03T09:06:18.362614+0000","mapping_epoch":23,"log_start":"0'0","ondisk_log_start":"0'0","created":23,"last_epoch_clean":24,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-06-03T08:36:28.444764+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-06-03T08:36:28.444764+0000","last_clean_scrub_stamp":"2024-06-03T08:36:28.444764+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-06-04T12:02:24.765135+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,2],"acting":[3,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.0","version":"19'76","reported_seq":1051,"reported_epoch":1009,"state":"active+clean","last_fresh":"2024-06-03T09:06:13.703685+0000","last_change":"2024-06-03T08:36:26.702702+0000","last_active":"2024-06-03T09:06:13.703685+0000","last_peered":"2024-06-03T09:06:13.703685+0000","last_clean":"2024-06-03T09:06:13.703685+0000","last_became_active":"2024-06-03T08:36:26.128378+0000","last_became_peered":"2024-06-03T08:36:26.128378+0000","last_unstale":"2024-06-03T09:06:13.703685+0000","last_undegraded":"2024-06-03T09:06:13.703685+0000","last_fullsized":"2024-06-03T09:06:13.703685+0000","mapping_epoch":20,"log_start":"0'0","ondisk_log_start":"0'0","created":16,"last_epoch_clean":22,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-06-03T08:36:15.542635+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-06-03T08:36:15.542635+0000","last_clean_scrub_stamp":"2024-06-03T08:36:15.542635+0000","objects_scrubbed":0,"log_size":76,"log_dups_size":0,"ondisk_log_size":76,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-06-04T12:41:53.731360+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":459280,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":96,"num_read_kb":82,"num_write":113,"num_write_kb":1372,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2,"num_bytes_recovered":397840,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,5],"acting":[1,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"17.10","version":"0'0","reported_seq":944,"reported_epoch":1009,"state":"active+clean","last_fresh":"2024-06-03T09:06:14.422035+0000","last_change":"2024-06-03T08:38:15.399189+0000","last_active":"2024-06-03T09:06:14.422035+0000","last_peered":"2024-06-03T09:06:14.422035+0000","last_clean":"2024-06-03T09:06:14.422035+0000","last_became_active":"2024-06-03T08:38:15.337723+0000","last_became_peered":"2024-06-03T08:38:15.337723+0000","last_unstale":"2024-06-03T09:06:14.422035+0000","last_undegraded":"2024-06-03T09:06:14.422035+0000","last_fullsized":"2024-06-03T09:06:14.422035+0000","mapping_epoch":28,"log_start":"0'0","ondisk_log_start":"0'0","created":28,"last_epoch_clean":42,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-06-03T08:37:55.614497+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-06-03T08:37:55.614497+0000","last_clean_scrub_stamp":"2024-06-03T08:37:55.614497+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-06-04T09:29:46.427317+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,3],"acting":[4,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"17.1a","version":"0'0","reported_seq":960,"reported_epoch":1009,"state":"active+undersized","last_fresh":"2024-06-03T09:06:14.431224+0000","last_change":"2024-06-03T09:06:03.390734+0000","last_active":"2024-06-03T09:06:14.431224+0000","last_peered":"2024-06-03T09:06:14.431224+0000","last_clean":"2024-06-03T09:05:20.322313+0000","last_became_active":"2024-06-03T09:06:03.390734+0000","last_became_peered":"2024-06-03T09:06:03.390734+0000","last_unstale":"2024-06-03T09:06:14.431224+0000","last_undegraded":"2024-06-03T09:06:14.431224+0000","last_fullsized":"2024-06-03T09:06:03.343197+0000","mapping_epoch":999,"log_start":"0'0","ondisk_log_start":"0'0","created":28,"last_epoch_clean":254,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-06-03T08:37:55.614497+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-06-03T08:37:55.614497+0000","last_clean_scrub_stamp":"2024-06-03T08:37:55.614497+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-06-04T15:56:47.628875+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4],"acting":[4],"avail_no_missing":["4"],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"17.1b","version":"0'0","reported_seq":935,"reported_epoch":1009,"state":"active+clean","last_fresh":"2024-06-03T09:06:13.663279+0000","last_change":"2024-06-03T08:38:03.115748+0000","last_active":"2024-06-03T09:06:13.663279+0000","last_peered":"2024-06-03T09:06:13.663279+0000","last_clean":"2024-06-03T09:06:13.663279+0000","last_became_active":"2024-06-03T08:38:03.110533+0000","last_became_peered":"2024-06-03T08:38:03.110533+0000","last_unstale":"2024-06-03T09:06:13.663279+0000","last_undegraded":"2024-06-03T09:06:13.663279+0000","last_fullsized":"2024-06-03T09:06:13.663279+0000","mapping_epoch":28,"log_start":"0'0","ondisk_log_start":"0'0","created":28,"last_epoch_clean":32,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-06-03T08:37:55.614497+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-06-03T08:37:55.614497+0000","last_clean_scrub_stamp":"2024-06-03T08:37:55.614497+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-06-04T12:36:47.471609+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,7],"acting":[3,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"17.18","version":"0'0","reported_seq":944,"reported_epoch":1009,"state":"active+clean","last_fresh":"2024-06-03T09:06:13.924799+0000","last_change":"2024-06-03T08:38:04.632864+0000","last_active":"2024-06-03T09:06:13.924799+0000","last_peered":"2024-06-03T09:06:13.924799+0000","last_clean":"2024-06-03T09:06:13.924799+0000","last_became_active":"2024-06-03T08:38:04.627856+0000","last_became_peered":"2024-06-03T08:38:04.627856+0000","last_unstale":"2024-06-03T09:06:13.924799+0000","last_undegraded":"2024-06-03T09:06:13.924799+0000","last_fullsized":"2024-06-03T09:06:13.924799+0000","mapping_epoch":28,"log_start":"0'0","ondisk_log_start":"0'0","created":28,"last_epoch_clean":33,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-06-03T08:37:55.614497+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-06-03T08:37:55.614497+0000","last_clean_scrub_stamp":"2024-06-03T08:37:55.614497+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-06-04T19:55:51.181598+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,4],"acting":[7,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"17.19","version":"0'0","reported_seq":934,"reported_epoch":1009,"state":"active+clean","last_fresh":"2024-06-03T09:06:13.652943+0000","last_change":"2024-06-03T08:38:01.839491+0000","last_active":"2024-06-03T09:06:13.652943+0000","last_peered":"2024-06-03T09:06:13.652943+0000","last_clean":"2024-06-03T09:06:13.652943+0000","last_became_active":"2024-06-03T08:38:01.829539+0000","last_became_peered":"2024-06-03T08:38:01.829539+0000","last_unstale":"2024-06-03T09:06:13.652943+0000","last_undegraded":"2024-06-03T09:06:13.652943+0000","last_fullsized":"2024-06-03T09:06:13.652943+0000","mapping_epoch":28,"log_start":"0'0","ondisk_log_start":"0'0","created":28,"last_epoch_clean":30,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-06-03T08:37:55.614497+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-06-03T08:37:55.614497+0000","last_clean_scrub_stamp":"2024-06-03T08:37:55.614497+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-06-04T09:10:43.860229+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,2],"acting":[3,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"17.1e","version":"0'0","reported_seq":148,"reported_epoch":1009,"state":"active+undersized","last_fresh":"2024-06-03T09:06:14.445917+0000","last_change":"2024-06-03T09:06:03.406181+0000","last_active":"2024-06-03T09:06:14.445917+0000","last_peered":"2024-06-03T09:06:14.445917+0000","last_clean":"2024-06-03T08:37:55.614497+0000","last_became_active":"2024-06-03T09:06:03.406181+0000","last_became_peered":"2024-06-03T09:06:03.406181+0000","last_unstale":"2024-06-03T09:06:14.445917+0000","last_undegraded":"2024-06-03T09:06:14.445917+0000","last_fullsized":"2024-06-03T09:06:03.363739+0000","mapping_epoch":999,"log_start":"0'0","ondisk_log_start":"0'0","created":28,"last_epoch_clean":255,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-06-03T08:37:55.614497+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-06-03T08:37:55.614497+0000","last_clean_scrub_stamp":"2024-06-03T08:37:55.614497+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-06-04T11:45:24.486600+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4],"acting":[4],"avail_no_missing":["4"],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"17.1f","version":"0'0","reported_seq":941,"reported_epoch":1009,"state":"active+clean","last_fresh":"2024-06-03T09:06:14.450053+0000","last_change":"2024-06-03T08:38:04.383538+0000","last_active":"2024-06-03T09:06:14.450053+0000","last_peered":"2024-06-03T09:06:14.450053+0000","last_clean":"2024-06-03T09:06:14.450053+0000","last_became_active":"2024-06-03T08:38:04.367278+0000","last_became_peered":"2024-06-03T08:38:04.367278+0000","last_unstale":"2024-06-03T09:06:14.450053+0000","last_undegraded":"2024-06-03T09:06:14.450053+0000","last_fullsized":"2024-06-03T09:06:14.450053+0000","mapping_epoch":28,"log_start":"0'0","ondisk_log_start":"0'0","created":28,"last_epoch_clean":32,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-06-03T08:37:55.614497+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-06-03T08:37:55.614497+0000","last_clean_scrub_stamp":"2024-06-03T08:37:55.614497+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-06-04T10:39:26.552282+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,2],"acting":[0,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"17.1c","version":"0'0","reported_seq":944,"reported_epoch":1010,"state":"active+undersized","last_fresh":"2024-06-03T09:06:18.327492+0000","last_change":"2024-06-03T09:06:03.382210+0000","last_active":"2024-06-03T09:06:18.327492+0000","last_peered":"2024-06-03T09:06:18.327492+0000","last_clean":"2024-06-03T09:04:43.199660+0000","last_became_active":"2024-06-03T09:06:03.382210+0000","last_became_peered":"2024-06-03T09:06:03.382210+0000","last_unstale":"2024-06-03T09:06:18.327492+0000","last_undegraded":"2024-06-03T09:06:18.327492+0000","last_fullsized":"2024-06-03T09:06:03.277874+0000","mapping_epoch":999,"log_start":"0'0","ondisk_log_start":"0'0","created":28,"last_epoch_clean":255,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-06-03T08:37:55.614497+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-06-03T08:37:55.614497+0000","last_clean_scrub_stamp":"2024-06-03T08:37:55.614497+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-06-04T12:18:50.013884+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5],"acting":[5],"avail_no_missing":["5"],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"17.1d","version":"0'0","reported_seq":923,"reported_epoch":1009,"state":"active+clean","last_fresh":"2024-06-03T09:06:13.715906+0000","last_change":"2024-06-03T08:38:06.794630+0000","last_active":"2024-06-03T09:06:13.715906+0000","last_peered":"2024-06-03T09:06:13.715906+0000","last_clean":"2024-06-03T09:06:13.715906+0000","last_became_active":"2024-06-03T08:38:06.592317+0000","last_became_peered":"2024-06-03T08:38:06.592317+0000","last_unstale":"2024-06-03T09:06:13.715906+0000","last_undegraded":"2024-06-03T09:06:13.715906+0000","last_fullsized":"2024-06-03T09:06:13.715906+0000","mapping_epoch":28,"log_start":"0'0","ondisk_log_start":"0'0","created":28,"last_epoch_clean":33,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-06-03T08:37:55.614497+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-06-03T08:37:55.614497+0000","last_clean_scrub_stamp":"2024-06-03T08:37:55.614497+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-06-04T12:33:30.785369+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,7],"acting":[1,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"17.2","version":"0'0","reported_seq":925,"reported_epoch":1009,"state":"active+clean","last_fresh":"2024-06-03T09:06:13.708688+0000","last_change":"2024-06-03T08:38:14.981410+0000","last_active":"2024-06-03T09:06:13.708688+0000","last_peered":"2024-06-03T09:06:13.708688+0000","last_clean":"2024-06-03T09:06:13.708688+0000","last_became_active":"2024-06-03T08:38:14.973825+0000","last_became_peered":"2024-06-03T08:38:14.973825+0000","last_unstale":"2024-06-03T09:06:13.708688+0000","last_undegraded":"2024-06-03T09:06:13.708688+0000","last_fullsized":"2024-06-03T09:06:13.708688+0000","mapping_epoch":28,"log_start":"0'0","ondisk_log_start":"0'0","created":28,"last_epoch_clean":41,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-06-03T08:37:55.614497+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-06-03T08:37:55.614497+0000","last_clean_scrub_stamp":"2024-06-03T08:37:55.614497+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-06-04T17:01:05.518613+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,4],"acting":[1,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"17.3","version":"0'0","reported_seq":943,"reported_epoch":1009,"state":"active+clean","last_fresh":"2024-06-03T09:06:14.437636+0000","last_change":"2024-06-03T08:38:14.619745+0000","last_active":"2024-06-03T09:06:14.437636+0000","last_peered":"2024-06-03T09:06:14.437636+0000","last_clean":"2024-06-03T09:06:14.437636+0000","last_became_active":"2024-06-03T08:38:14.510665+0000","last_became_peered":"2024-06-03T08:38:14.510665+0000","last_unstale":"2024-06-03T09:06:14.437636+0000","last_undegraded":"2024-06-03T09:06:14.437636+0000","last_fullsized":"2024-06-03T09:06:14.437636+0000","mapping_epoch":28,"log_start":"0'0","ondisk_log_start":"0'0","created":28,"last_epoch_clean":42,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-06-03T08:37:55.614497+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-06-03T08:37:55.614497+0000","last_clean_scrub_stamp":"2024-06-03T08:37:55.614497+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-06-04T12:03:47.392064+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,7],"acting":[0,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"17.0","version":"0'0","reported_seq":944,"reported_epoch":1009,"state":"active+clean","last_fresh":"2024-06-03T09:06:14.427123+0000","last_change":"2024-06-03T08:38:04.237118+0000","last_active":"2024-06-03T09:06:14.427123+0000","last_peered":"2024-06-03T09:06:14.427123+0000","last_clean":"2024-06-03T09:06:14.427123+0000","last_became_active":"2024-06-03T08:38:04.231954+0000","last_became_peered":"2024-06-03T08:38:04.231954+0000","last_unstale":"2024-06-03T09:06:14.427123+0000","last_undegraded":"2024-06-03T09:06:14.427123+0000","last_fullsized":"2024-06-03T09:06:14.427123+0000","mapping_epoch":28,"log_start":"0'0","ondisk_log_start":"0'0","created":28,"last_epoch_clean":32,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-06-03T08:37:55.614497+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-06-03T08:37:55.614497+0000","last_clean_scrub_stamp":"2024-06-03T08:37:55.614497+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-06-04T09:56:10.741637+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,2],"acting":[0,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"17.1","version":"65'7","reported_seq":959,"reported_epoch":1009,"state":"active+undersized+degraded","last_fresh":"2024-06-03T09:06:13.709209+0000","last_change":"2024-06-03T09:06:03.370660+0000","last_active":"2024-06-03T09:06:13.709209+0000","last_peered":"2024-06-03T09:06:13.709209+0000","last_clean":"2024-06-03T09:05:03.733864+0000","last_became_active":"2024-06-03T09:06:03.370660+0000","last_became_peered":"2024-06-03T09:06:03.370660+0000","last_unstale":"2024-06-03T09:06:13.709209+0000","last_undegraded":"2024-06-03T09:06:03.266076+0000","last_fullsized":"2024-06-03T09:06:03.264525+0000","mapping_epoch":999,"log_start":"0'0","ondisk_log_start":"0'0","created":28,"last_epoch_clean":255,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-06-03T08:37:55.614497+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-06-03T08:37:55.614497+0000","last_clean_scrub_stamp":"2024-06-03T08:37:55.614497+0000","objects_scrubbed":0,"log_size":7,"log_dups_size":0,"ondisk_log_size":7,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-06-04T11:55:18.194283+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":5,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":1,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":6,"num_read_kb":6,"num_write":7,"num_write_kb":7,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5],"acting":[5],"avail_no_missing":["5"],"object_location_counts":[{"shards":"5","objects":1}],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]}],"pool_stats":[{"poolid":2,"num_pg":8,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":14,"acting":14,"num_store_stats":7},{"poolid":1,"num_pg":1,"stat_sum":{"num_bytes":459280,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":96,"num_read_kb":82,"num_write":113,"num_write_kb":1372,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2,"num_bytes_recovered":397840,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":802816,"data_stored":795680,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":76,"ondisk_log_size":76,"up":2,"acting":2,"num_store_stats":2},{"poolid":17,"num_pg":32,"stat_sum":{"num_bytes":5,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":1,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":6,"num_read_kb":6,"num_write":7,"num_write_kb":7,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":8192,"data_stored":10,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":7,"ondisk_log_size":7,"up":52,"acting":52,"num_store_stats":8}],"osd_stats":[{"osd":6,"up_from":253,"seq":1086626726200,"num_pgs":32,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":54772,"kb_used_data":5608,"kb_used_omap":1,"kb_used_meta":49150,"kb_avail":94317068,"statfs":{"total":96636764160,"available":96580677632,"internally_reserved":0,"allocated":5742592,"data_stored":2767180,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1593,"internal_metadata":50330055},"hb_peers":[0,1,2,3,4,5,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":109,"apply_latency_ms":109,"commit_latency_ns":109000000,"apply_latency_ns":109000000},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Mon Jun 3 09:04:02 2024","interfaces":[{"interface":"back","average":{"1min":1582.4780000000001,"5min":332.72699999999998,"15min":124.113},"min":{"1min":4.4829999999999997,"5min":4.1959999999999997,"15min":4.1959999999999997},"max":{"1min":22766.882000000001,"5min":22766.882000000001,"15min":22766.882000000001},"last":21.917999999999999},{"interface":"front","average":{"1min":1579.5409999999999,"5min":334.94499999999999,"15min":125.84999999999999},"min":{"1min":4.8129999999999997,"5min":4.7539999999999996,"15min":4.4530000000000003},"max":{"1min":22766.289000000001,"5min":22766.289000000001,"15min":22766.289000000001},"last":25.408999999999999}]},{"osd":1,"last update":"Mon Jun 3 09:03:40 2024","interfaces":[{"interface":"back","average":{"1min":36.420999999999999,"5min":77.054000000000002,"15min":132.84800000000001},"min":{"1min":6.0380000000000003,"5min":5.8799999999999999,"15min":5.484},"max":{"1min":589.64099999999996,"5min":4138.1679999999997,"15min":9955.1299999999992},"last":26.84},{"interface":"front","average":{"1min":40.240000000000002,"5min":79.674999999999997,"15min":133.38999999999999},"min":{"1min":8.4589999999999996,"5min":8.4090000000000007,"15min":6.1390000000000002},"max":{"1min":588.60799999999995,"5min":4137.6530000000002,"15min":9827.5509999999995},"last":26.263000000000002}]},{"osd":2,"last update":"Mon Jun 3 09:03:40 2024","interfaces":[{"interface":"back","average":{"1min":12.613,"5min":24.600999999999999,"15min":21.088000000000001},"min":{"1min":6.9740000000000002,"5min":6.9740000000000002,"15min":6.5330000000000004},"max":{"1min":26.076000000000001,"5min":376.24900000000002,"15min":376.24900000000002},"last":24.899000000000001},{"interface":"front","average":{"1min":12.493,"5min":22.207999999999998,"15min":21.219000000000001},"min":{"1min":6.5810000000000004,"5min":6.5110000000000001,"15min":6.0750000000000002},"max":{"1min":29.690999999999999,"5min":271.51100000000002,"15min":434.77300000000002},"last":40.372}]},{"osd":3,"last update":"Mon Jun 3 09:03:40 2024","interfaces":[{"interface":"back","average":{"1min":14.093999999999999,"5min":24.143999999999998,"15min":29.376999999999999},"min":{"1min":10.196999999999999,"5min":6.2919999999999998,"15min":6.1719999999999997},"max":{"1min":24.832000000000001,"5min":197.84899999999999,"15min":1235.5799999999999},"last":24.353999999999999},{"interface":"front","average":{"1min":9.3260000000000005,"5min":18.463999999999999,"15min":27.631},"min":{"1min":6.125,"5min":6.125,"15min":6.125},"max":{"1min":28.166,"5min":200.173,"15min":1157.6780000000001},"last":23.454000000000001}]},{"osd":4,"last update":"Mon Jun 3 09:04:29 2024","interfaces":[{"interface":"back","average":{"1min":10.279,"5min":1253.809,"15min":450.79199999999997},"min":{"1min":7.2300000000000004,"5min":6.093,"15min":6.093},"max":{"1min":15.000999999999999,"5min":28093.096000000001,"15min":28093.096000000001},"last":101.45399999999999},{"interface":"front","average":{"1min":15.693,"5min":1279.759,"15min":463.87099999999998},"min":{"1min":10.551,"5min":7.7480000000000002,"15min":7.7480000000000002},"max":{"1min":27.898,"5min":28323.837,"15min":28323.837},"last":102.262}]},{"osd":5,"last update":"Mon Jun 3 09:04:10 2024","interfaces":[{"interface":"back","average":{"1min":13.728999999999999,"5min":24.670000000000002,"15min":23.815999999999999},"min":{"1min":10.401,"5min":7.516,"15min":7.516},"max":{"1min":25.138000000000002,"5min":320.85000000000002,"15min":445.041},"last":103.52500000000001},{"interface":"front","average":{"1min":13.847,"5min":29.283000000000001,"15min":23.393999999999998},"min":{"1min":7.944,"5min":6.266,"15min":6.266},"max":{"1min":18.817,"5min":530.56700000000001,"15min":530.56700000000001},"last":29.263999999999999}]},{"osd":7,"last update":"Mon Jun 3 09:03:40 2024","interfaces":[{"interface":"back","average":{"1min":20.568999999999999,"5min":30.286999999999999,"15min":27.977},"min":{"1min":13.874000000000001,"5min":10.58,"15min":9.2859999999999996},"max":{"1min":86.959000000000003,"5min":222.07599999999999,"15min":255.08799999999999},"last":37.618000000000002},{"interface":"front","average":{"1min":19.102,"5min":29.550000000000001,"15min":27.763000000000002},"min":{"1min":10.052,"5min":8.6709999999999994,"15min":8.5429999999999993},"max":{"1min":67.968999999999994,"5min":293.62099999999998,"15min":438.14600000000002},"last":111.985}]}]},{"osd":7,"up_from":20,"seq":85899346287,"num_pgs":9,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":58964,"kb_used_data":5704,"kb_used_omap":1,"kb_used_meta":53246,"kb_avail":94312876,"statfs":{"total":96636764160,"available":96576385024,"internally_reserved":0,"allocated":5840896,"data_stored":2740727,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1589,"internal_metadata":54524363},"hb_peers":[0,1,2,3,4,5],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Mon Jun 3 09:06:09 2024","interfaces":[{"interface":"back","average":{"1min":14.073,"5min":894.22400000000005,"15min":308.93400000000003},"min":{"1min":5.5709999999999997,"5min":4.218,"15min":4.0599999999999996},"max":{"1min":90.757999999999996,"5min":21729.001,"15min":21729.001},"last":9.1039999999999992},{"interface":"front","average":{"1min":9.2550000000000008,"5min":906.23500000000001,"15min":314.06},"min":{"1min":5.5789999999999997,"5min":4.8810000000000002,"15min":4.8810000000000002},"max":{"1min":25.937000000000001,"5min":21899.732,"15min":21899.732},"last":5.7240000000000002}]},{"osd":1,"last update":"Mon Jun 3 09:06:09 2024","interfaces":[{"interface":"back","average":{"1min":15.182,"5min":69.183999999999997,"15min":219.173},"min":{"1min":7.3639999999999999,"5min":6.6200000000000001,"15min":6.1319999999999997},"max":{"1min":76.926000000000002,"5min":2795.1329999999998,"15min":18373.257000000001},"last":9.8119999999999994},{"interface":"front","average":{"1min":21.495999999999999,"5min":74.703999999999994,"15min":225.285},"min":{"1min":8.8399999999999999,"5min":8.8399999999999999,"15min":8.5129999999999999},"max":{"1min":94.977999999999994,"5min":2803.3670000000002,"15min":18374.601999999999},"last":12.17}]},{"osd":2,"last update":"Mon Jun 3 09:06:09 2024","interfaces":[{"interface":"back","average":{"1min":19.164000000000001,"5min":26.902000000000001,"15min":22.920999999999999},"min":{"1min":11.207000000000001,"5min":8.5030000000000001,"15min":8.5030000000000001},"max":{"1min":91.319999999999993,"5min":333.99200000000002,"15min":333.99200000000002},"last":14.196999999999999},{"interface":"front","average":{"1min":11.125999999999999,"5min":18.132000000000001,"15min":16.353000000000002},"min":{"1min":6.202,"5min":5.8369999999999997,"15min":5.5069999999999997},"max":{"1min":28.649000000000001,"5min":226.18100000000001,"15min":226.18100000000001},"last":11.329000000000001}]},{"osd":3,"last update":"Mon Jun 3 09:06:09 2024","interfaces":[{"interface":"back","average":{"1min":22.291,"5min":26.289999999999999,"15min":31.074999999999999},"min":{"1min":11.714,"5min":6.8959999999999999,"15min":6.3760000000000003},"max":{"1min":83.102000000000004,"5min":316.64400000000001,"15min":2039.9059999999999},"last":15.954000000000001},{"interface":"front","average":{"1min":13.324999999999999,"5min":17.963000000000001,"15min":25.969999999999999},"min":{"1min":6.1200000000000001,"5min":5.2089999999999996,"15min":5.2089999999999996},"max":{"1min":49.908999999999999,"5min":233.542,"15min":2037.5550000000001},"last":13.622999999999999}]},{"osd":4,"last update":"Mon Jun 3 09:06:29 2024","interfaces":[{"interface":"back","average":{"1min":17.675999999999998,"5min":1026.325,"15min":472.47199999999998},"min":{"1min":7.306,"5min":5.9089999999999998,"15min":5.9089999999999998},"max":{"1min":140.03299999999999,"5min":28969.369999999999,"15min":30073.917000000001},"last":8.7119999999999997},{"interface":"front","average":{"1min":17.890999999999998,"5min":1044.5139999999999,"15min":480.226},"min":{"1min":5.9279999999999999,"5min":5.9279999999999999,"15min":5.9279999999999999},"max":{"1min":141.179,"5min":29244.138999999999,"15min":30348.021000000001},"last":10.571}]},{"osd":5,"last update":"Mon Jun 3 09:06:02 2024","interfaces":[{"interface":"back","average":{"1min":24.372,"5min":26.856000000000002,"15min":23.571999999999999},"min":{"1min":12.278,"5min":9.2460000000000004,"15min":6.5650000000000004},"max":{"1min":183.64099999999999,"5min":317.572,"15min":317.572},"last":15.128},{"interface":"front","average":{"1min":21.259,"5min":23.364000000000001,"15min":21.193999999999999},"min":{"1min":7.8499999999999996,"5min":7.0999999999999996,"15min":7.008},"max":{"1min":184.53399999999999,"5min":306.60599999999999,"15min":306.60599999999999},"last":12.704000000000001}]},{"osd":6,"last update":"Mon Jun 3 09:04:31 2024","interfaces":[{"interface":"back","average":{"1min":44.061,"5min":26.821000000000002,"15min":91.176000000000002},"min":{"1min":9.0800000000000001,"5min":7.9390000000000001,"15min":7.9390000000000001},"max":{"1min":268.029,"5min":308.916,"15min":8734.4639999999999},"last":6.6769999999999996},{"interface":"front","average":{"1min":46.643000000000001,"5min":23.879999999999999,"15min":88.765000000000001},"min":{"1min":8.8119999999999994,"5min":6.9240000000000004,"15min":6.742},"max":{"1min":305.64600000000002,"5min":305.64600000000002,"15min":8735.5319999999992},"last":12.327}]}]},{"osd":5,"up_from":20,"seq":85899346262,"num_pgs":9,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":59360,"kb_used_data":6100,"kb_used_omap":1,"kb_used_meta":53246,"kb_avail":94312480,"statfs":{"total":96636764160,"available":96575979520,"internally_reserved":0,"allocated":6246400,"data_stored":3138572,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1594,"internal_metadata":54524358},"hb_peers":[0,1,2,3,4,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Mon Jun 3 09:06:06 2024","interfaces":[{"interface":"back","average":{"1min":7.468,"5min":1703.037,"15min":580.51900000000001},"min":{"1min":4.6509999999999998,"5min":4.6509999999999998,"15min":4.4020000000000001},"max":{"1min":14.711,"5min":26433.764999999999,"15min":26433.764999999999},"last":4.641},{"interface":"front","average":{"1min":9.9429999999999996,"5min":1725.27,"15min":587.87800000000004},"min":{"1min":5.5999999999999996,"5min":5.1319999999999997,"15min":4.8170000000000002},"max":{"1min":13.590999999999999,"5min":26704.698,"15min":26704.698},"last":8.7279999999999998}]},{"osd":1,"last update":"Mon Jun 3 09:06:00 2024","interfaces":[{"interface":"back","average":{"1min":10.862,"5min":67.731999999999999,"15min":217.364},"min":{"1min":7.1769999999999996,"5min":6.0890000000000004,"15min":5.4930000000000003},"max":{"1min":16.998999999999999,"5min":3923.4090000000001,"15min":18811.696},"last":12.723000000000001},{"interface":"front","average":{"1min":10.417,"5min":65.141999999999996,"15min":220.75700000000001},"min":{"1min":5.3209999999999997,"5min":5.3209999999999997,"15min":5.0209999999999999},"max":{"1min":15.821999999999999,"5min":4019.6129999999998,"15min":18968.365000000002},"last":7.766}]},{"osd":2,"last update":"Mon Jun 3 09:06:00 2024","interfaces":[{"interface":"back","average":{"1min":11.366,"5min":24.18,"15min":21.137},"min":{"1min":5.6369999999999996,"5min":5.6369999999999996,"15min":5.0510000000000002},"max":{"1min":15.529999999999999,"5min":332.48200000000003,"15min":332.48200000000003},"last":11.538},{"interface":"front","average":{"1min":10.089,"5min":23.117000000000001,"15min":24.050000000000001},"min":{"1min":6.6769999999999996,"5min":6.6769999999999996,"15min":5.8739999999999997},"max":{"1min":15.821,"5min":353.03199999999998,"15min":353.03199999999998},"last":6.9349999999999996}]},{"osd":3,"last update":"Mon Jun 3 09:06:00 2024","interfaces":[{"interface":"back","average":{"1min":13.472,"5min":24.283999999999999,"15min":31.666},"min":{"1min":10.111000000000001,"5min":7.4720000000000004,"15min":7.4210000000000003},"max":{"1min":16.271999999999998,"5min":353.57499999999999,"15min":2855.8519999999999},"last":14.273999999999999},{"interface":"front","average":{"1min":9.5899999999999999,"5min":15.875,"15min":25.834},"min":{"1min":6.6609999999999996,"5min":5.7729999999999997,"15min":5.1909999999999998},"max":{"1min":15.112,"5min":203.804,"15min":2869.7860000000001},"last":13.554}]},{"osd":4,"last update":"Mon Jun 3 09:06:29 2024","interfaces":[{"interface":"back","average":{"1min":11.067,"5min":780.21000000000004,"15min":377.28399999999999},"min":{"1min":5.4779999999999998,"5min":5.4779999999999998,"15min":5.4779999999999998},"max":{"1min":14.654999999999999,"5min":23315.438999999998,"15min":28619.825000000001},"last":11.292},{"interface":"front","average":{"1min":14.349,"5min":782.97299999999996,"15min":377.93400000000003},"min":{"1min":7.492,"5min":6.5030000000000001,"15min":6.5030000000000001},"max":{"1min":17.707000000000001,"5min":23317.502,"15min":28621.294000000002},"last":15.134}]},{"osd":6,"last update":"Mon Jun 3 09:03:57 2024","interfaces":[{"interface":"back","average":{"1min":23.536000000000001,"5min":36.457000000000001,"15min":83.799999999999997},"min":{"1min":9.4800000000000004,"5min":9.4800000000000004,"15min":8.0630000000000006},"max":{"1min":89.442999999999998,"5min":508.77999999999997,"15min":9193.5169999999998},"last":157.79300000000001},{"interface":"front","average":{"1min":18.891999999999999,"5min":22.100999999999999,"15min":75.489999999999995},"min":{"1min":5.923,"5min":5.923,"15min":5.923},"max":{"1min":78.424999999999997,"5min":464.62,"15min":9191.4429999999993},"last":148.09}]},{"osd":7,"last update":"Mon Jun 3 09:06:00 2024","interfaces":[{"interface":"back","average":{"1min":13.257,"5min":20.917999999999999,"15min":21.088999999999999},"min":{"1min":10.475,"5min":6.7629999999999999,"15min":6.7629999999999999},"max":{"1min":17.245000000000001,"5min":211.22999999999999,"15min":211.22999999999999},"last":12.083},{"interface":"front","average":{"1min":12.872999999999999,"5min":20.587,"15min":24.021000000000001},"min":{"1min":8.5760000000000005,"5min":7.7709999999999999,"15min":6.8570000000000002},"max":{"1min":18.359999999999999,"5min":223.191,"15min":286.69499999999999},"last":9.3849999999999998}]}]},{"osd":4,"up_from":20,"seq":85899346269,"num_pgs":12,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":63060,"kb_used_data":5704,"kb_used_omap":1,"kb_used_meta":57342,"kb_avail":94308780,"statfs":{"total":96636764160,"available":96572190720,"internally_reserved":0,"allocated":5840896,"data_stored":2740727,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1594,"internal_metadata":58718662},"hb_peers":[0,1,2,3,5,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Mon Jun 3 09:05:34 2024","interfaces":[{"interface":"back","average":{"1min":12.416,"5min":1099.4649999999999,"15min":382.15199999999999},"min":{"1min":4.5279999999999996,"5min":4.5279999999999996,"15min":4.3220000000000001},"max":{"1min":123.878,"5min":21726.743999999999,"15min":21726.743999999999},"last":7.5579999999999998},{"interface":"front","average":{"1min":16.462,"5min":1108.894,"15min":387.70299999999997},"min":{"1min":6.4089999999999998,"5min":5.8380000000000001,"15min":5.4909999999999997},"max":{"1min":130.185,"5min":21727.993999999999,"15min":21727.993999999999},"last":11.199}]},{"osd":1,"last update":"Mon Jun 3 09:05:34 2024","interfaces":[{"interface":"back","average":{"1min":14.727,"5min":56.755000000000003,"15min":341.09399999999999},"min":{"1min":4.6529999999999996,"5min":4.1970000000000001,"15min":4.1970000000000001},"max":{"1min":126.899,"5min":3346.4389999999999,"15min":22256.098000000002},"last":6.7679999999999998},{"interface":"front","average":{"1min":13.472,"5min":57.503,"15min":340.303},"min":{"1min":5.0590000000000002,"5min":5.0590000000000002,"15min":4.2640000000000002},"max":{"1min":126.096,"5min":3347.181,"15min":22247.317999999999},"last":12.481}]},{"osd":2,"last update":"Mon Jun 3 09:05:34 2024","interfaces":[{"interface":"back","average":{"1min":26.088999999999999,"5min":23.866,"15min":20.620999999999999},"min":{"1min":7.0869999999999997,"5min":6.8239999999999998,"15min":6.3150000000000004},"max":{"1min":337.60399999999998,"5min":393.58600000000001,"15min":393.58600000000001},"last":10.336},{"interface":"front","average":{"1min":15.800000000000001,"5min":22.584,"15min":23.245999999999999},"min":{"1min":6.0629999999999997,"5min":6.0629999999999997,"15min":6.0629999999999997},"max":{"1min":127.913,"5min":394.471,"15min":394.471},"last":14.75}]},{"osd":3,"last update":"Mon Jun 3 09:05:34 2024","interfaces":[{"interface":"back","average":{"1min":17.544,"5min":19.143999999999998,"15min":18.742000000000001},"min":{"1min":5.8719999999999999,"5min":5.4100000000000001,"15min":5.4000000000000004},"max":{"1min":127.43600000000001,"5min":322.63,"15min":322.63},"last":15.779},{"interface":"front","average":{"1min":21.327000000000002,"5min":23.559000000000001,"15min":23.797000000000001},"min":{"1min":10.023,"5min":7.484,"15min":7.484},"max":{"1min":204.517,"5min":392.67500000000001,"15min":392.67500000000001},"last":11.742000000000001}]},{"osd":5,"last update":"Mon Jun 3 09:05:34 2024","interfaces":[{"interface":"back","average":{"1min":24.274999999999999,"5min":21.274999999999999,"15min":20.988},"min":{"1min":6.7489999999999997,"5min":6.7489999999999997,"15min":6.7489999999999997},"max":{"1min":268.10199999999998,"5min":344.23399999999998,"15min":344.23399999999998},"last":13.760999999999999},{"interface":"front","average":{"1min":26.361999999999998,"5min":25.218,"15min":26.084},"min":{"1min":12.228,"5min":8.5180000000000007,"15min":8.5180000000000007},"max":{"1min":264.75,"5min":345.18700000000001,"15min":345.18700000000001},"last":17.739000000000001}]},{"osd":6,"last update":"Mon Jun 3 09:04:32 2024","interfaces":[{"interface":"back","average":{"1min":15.548,"5min":32.957999999999998,"15min":152.738},"min":{"1min":9.141,"5min":9.141,"15min":8.2319999999999993},"max":{"1min":52.57,"5min":395.238,"15min":12496.335999999999},"last":208.90000000000001},{"interface":"front","average":{"1min":12.680999999999999,"5min":25.428000000000001,"15min":149.30600000000001},"min":{"1min":6.4379999999999997,"5min":5.6319999999999997,"15min":5.6319999999999997},"max":{"1min":46.283000000000001,"5min":377.74299999999999,"15min":12616.297},"last":205.88900000000001}]},{"osd":7,"last update":"Mon Jun 3 09:05:34 2024","interfaces":[{"interface":"back","average":{"1min":18.295999999999999,"5min":19.585999999999999,"15min":21.062000000000001},"min":{"1min":6.2110000000000003,"5min":6.2110000000000003,"15min":5.8010000000000002},"max":{"1min":207.25299999999999,"5min":358.94900000000001,"15min":358.94900000000001},"last":9.2669999999999995},{"interface":"front","average":{"1min":23.712,"5min":24.024999999999999,"15min":24.640000000000001},"min":{"1min":11.824999999999999,"5min":7.7789999999999999,"15min":7.7789999999999999},"max":{"1min":209.32900000000001,"5min":346.25200000000001,"15min":380.06900000000002},"last":16.763999999999999}]}]},{"osd":2,"up_from":20,"seq":85899346289,"num_pgs":11,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":58964,"kb_used_data":5704,"kb_used_omap":1,"kb_used_meta":53246,"kb_avail":94312876,"statfs":{"total":96636764160,"available":96576385024,"internally_reserved":0,"allocated":5840896,"data_stored":2740727,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1594,"internal_metadata":54524358},"hb_peers":[0,1,3,4,5,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Mon Jun 3 09:06:23 2024","interfaces":[{"interface":"back","average":{"1min":8.9629999999999992,"5min":946.25400000000002,"15min":326.36099999999999},"min":{"1min":5.3479999999999999,"5min":4.7050000000000001,"15min":4.1420000000000003},"max":{"1min":26.015000000000001,"5min":23465.18,"15min":23465.18},"last":6.6070000000000002},{"interface":"front","average":{"1min":7.6130000000000004,"5min":944.39800000000002,"15min":326.178},"min":{"1min":4.6790000000000003,"5min":4.6790000000000003,"15min":4.6790000000000003},"max":{"1min":20.802,"5min":23465.966,"15min":23465.966},"last":5.1609999999999996}]},{"osd":1,"last update":"Mon Jun 3 09:06:23 2024","interfaces":[{"interface":"back","average":{"1min":13.007,"5min":76.507000000000005,"15min":294.72899999999998},"min":{"1min":6.4089999999999998,"5min":5.4199999999999999,"15min":5.4199999999999999},"max":{"1min":85.430000000000007,"5min":4220.5290000000005,"15min":18954.987000000001},"last":7.2859999999999996},{"interface":"front","average":{"1min":15.894,"5min":82.119,"15min":302.92399999999998},"min":{"1min":7.367,"5min":7.367,"15min":7.367},"max":{"1min":98.498000000000005,"5min":4325.9489999999996,"15min":19143.834999999999},"last":9.2759999999999998}]},{"osd":3,"last update":"Mon Jun 3 09:06:23 2024","interfaces":[{"interface":"back","average":{"1min":9.7579999999999991,"5min":12.444000000000001,"15min":28.065000000000001},"min":{"1min":7,"5min":5.9290000000000003,"15min":5.2450000000000001},"max":{"1min":28.164000000000001,"5min":118.913,"15min":2594.8150000000001},"last":10.537000000000001},{"interface":"front","average":{"1min":12.321999999999999,"5min":15.25,"15min":31.349},"min":{"1min":9.1240000000000006,"5min":8.0760000000000005,"15min":6.0780000000000003},"max":{"1min":32.360999999999997,"5min":156.34700000000001,"15min":2706.5120000000002},"last":12.038}]},{"osd":4,"last update":"Mon Jun 3 09:05:57 2024","interfaces":[{"interface":"back","average":{"1min":10.573,"5min":1266.894,"15min":438.154},"min":{"1min":8.6379999999999999,"5min":7.8789999999999996,"15min":6.2039999999999997},"max":{"1min":15.105,"5min":26660.861000000001,"15min":26660.861000000001},"last":11.036},{"interface":"front","average":{"1min":12.407999999999999,"5min":1266.7460000000001,"15min":435.75400000000002},"min":{"1min":10.266999999999999,"5min":10.266999999999999,"15min":6.2720000000000002},"max":{"1min":15.228,"5min":26616.243999999999,"15min":26616.243999999999},"last":12.824}]},{"osd":5,"last update":"Mon Jun 3 09:06:23 2024","interfaces":[{"interface":"back","average":{"1min":14.103,"5min":18.091000000000001,"15min":21.119},"min":{"1min":9.6349999999999998,"5min":9.2159999999999993,"15min":6.2050000000000001},"max":{"1min":32.890000000000001,"5min":252.31100000000001,"15min":252.31100000000001},"last":13.518000000000001},{"interface":"front","average":{"1min":11.545999999999999,"5min":16.170999999999999,"15min":20.254000000000001},"min":{"1min":7.6070000000000002,"5min":7.4850000000000003,"15min":6.8810000000000002},"max":{"1min":29.228999999999999,"5min":258.31700000000001,"15min":258.31700000000001},"last":8.6509999999999998}]},{"osd":6,"last update":"Mon Jun 3 09:04:28 2024","interfaces":[{"interface":"back","average":{"1min":14.063000000000001,"5min":24.850000000000001,"15min":106.755},"min":{"1min":9.0820000000000007,"5min":8.9649999999999999,"15min":8.9649999999999999},"max":{"1min":24.666,"5min":188.06700000000001,"15min":9521.0010000000002},"last":183.60900000000001},{"interface":"front","average":{"1min":13.933999999999999,"5min":21.081,"15min":106.142},"min":{"1min":5.8719999999999999,"5min":5.8719999999999999,"15min":5.8719999999999999},"max":{"1min":117.551,"5min":193.887,"15min":9585.6620000000003},"last":184.20599999999999}]},{"osd":7,"last update":"Mon Jun 3 09:06:23 2024","interfaces":[{"interface":"back","average":{"1min":17.056000000000001,"5min":20.055,"15min":23.268999999999998},"min":{"1min":10.353999999999999,"5min":6.6130000000000004,"15min":6.6130000000000004},"max":{"1min":88.974999999999994,"5min":242.56800000000001,"15min":242.56800000000001},"last":14.315},{"interface":"front","average":{"1min":18.885000000000002,"5min":21.460000000000001,"15min":25.489999999999998},"min":{"1min":12.497,"5min":9.0660000000000007,"15min":9.0660000000000007},"max":{"1min":102.229,"5min":181.56399999999999,"15min":214.33799999999999},"last":15.112}]}]},{"osd":3,"up_from":14,"seq":60129542503,"num_pgs":11,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":58964,"kb_used_data":5704,"kb_used_omap":1,"kb_used_meta":53246,"kb_avail":94312876,"statfs":{"total":96636764160,"available":96576385024,"internally_reserved":0,"allocated":5840896,"data_stored":2740727,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1592,"internal_metadata":54524360},"hb_peers":[0,1,2,4,5,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Mon Jun 3 09:06:25 2024","interfaces":[{"interface":"back","average":{"1min":7.79,"5min":1060.1959999999999,"15min":367.01799999999997},"min":{"1min":5.6909999999999998,"5min":5.173,"15min":4.6929999999999996},"max":{"1min":10.59,"5min":23275.167000000001,"15min":23275.167000000001},"last":8.5299999999999994},{"interface":"front","average":{"1min":11.215999999999999,"5min":1066.5799999999999,"15min":372.476},"min":{"1min":8.7080000000000002,"5min":7.2510000000000003,"15min":5.9880000000000004},"max":{"1min":13.499000000000001,"5min":23278.536,"15min":23278.536},"last":14.012}]},{"osd":1,"last update":"Mon Jun 3 09:06:25 2024","interfaces":[{"interface":"back","average":{"1min":7.3760000000000003,"5min":15.614000000000001,"15min":389.83100000000002},"min":{"1min":4.335,"5min":4.335,"15min":4.335},"max":{"1min":11.289,"5min":250.72300000000001,"15min":21658.205999999998},"last":10.667},{"interface":"front","average":{"1min":8.9499999999999993,"5min":14.135,"15min":381.10300000000001},"min":{"1min":6.5609999999999999,"5min":5.4100000000000001,"15min":5.2779999999999996},"max":{"1min":14.51,"5min":180.44499999999999,"15min":21307.069},"last":6.109}]},{"osd":2,"last update":"Mon Jun 3 09:06:25 2024","interfaces":[{"interface":"back","average":{"1min":12.343999999999999,"5min":22.059999999999999,"15min":23.715},"min":{"1min":7.9199999999999999,"5min":6.4900000000000002,"15min":6.056},"max":{"1min":15.601000000000001,"5min":377.803,"15min":377.803},"last":9.2889999999999997},{"interface":"front","average":{"1min":10.319000000000001,"5min":19.364999999999998,"15min":20.242999999999999},"min":{"1min":6.891,"5min":6.1749999999999998,"15min":5.9020000000000001},"max":{"1min":15.432,"5min":371.601,"15min":371.601},"last":11.827999999999999}]},{"osd":4,"last update":"Mon Jun 3 09:05:27 2024","interfaces":[{"interface":"back","average":{"1min":10.444000000000001,"5min":1211.4970000000001,"15min":1639.1579999999999},"min":{"1min":6.4429999999999996,"5min":6.2560000000000002,"15min":6.0309999999999997},"max":{"1min":21.263000000000002,"5min":30144.939999999999,"15min":62679.622000000003},"last":7.3719999999999999},{"interface":"front","average":{"1min":11.363,"5min":1219.6400000000001,"15min":1644.127},"min":{"1min":7.8109999999999999,"5min":7.8109999999999999,"15min":6.2439999999999998},"max":{"1min":20.108000000000001,"5min":30180.689999999999,"15min":62681.021999999997},"last":9.9800000000000004}]},{"osd":5,"last update":"Mon Jun 3 09:06:07 2024","interfaces":[{"interface":"back","average":{"1min":12.802,"5min":19.640999999999998,"15min":22.318000000000001},"min":{"1min":9.0890000000000004,"5min":6.8810000000000002,"15min":6.7690000000000001},"max":{"1min":16.745999999999999,"5min":249.88,"15min":327.34899999999999},"last":14.826000000000001},{"interface":"front","average":{"1min":15.002000000000001,"5min":24.003,"15min":26.649999999999999},"min":{"1min":11.984,"5min":9.9149999999999991,"15min":8.9879999999999995},"max":{"1min":18.407,"5min":251.47800000000001,"15min":331.86799999999999},"last":15.525}]},{"osd":6,"last update":"Mon Jun 3 09:04:04 2024","interfaces":[{"interface":"back","average":{"1min":22.815999999999999,"5min":31.513999999999999,"15min":181.74299999999999},"min":{"1min":9.4809999999999999,"5min":7.2240000000000002,"15min":7.2240000000000002},"max":{"1min":160.56999999999999,"5min":380.613,"15min":11893.74},"last":19.359999999999999},{"interface":"front","average":{"1min":25.359999999999999,"5min":30.981000000000002,"15min":179.81399999999999},"min":{"1min":9.8870000000000005,"5min":7.3360000000000003,"15min":6.0899999999999999},"max":{"1min":193.11099999999999,"5min":379.80700000000002,"15min":11893.069},"last":8.202}]},{"osd":7,"last update":"Mon Jun 3 09:06:25 2024","interfaces":[{"interface":"back","average":{"1min":11.321999999999999,"5min":18.443999999999999,"15min":21.568999999999999},"min":{"1min":8.109,"5min":8.109,"15min":7.4279999999999999},"max":{"1min":17.113,"5min":146.267,"15min":178.76300000000001},"last":11.212999999999999},{"interface":"front","average":{"1min":12.512,"5min":20.065999999999999,"15min":24.190000000000001},"min":{"1min":7.4989999999999997,"5min":7.4989999999999997,"15min":7.4989999999999997},"max":{"1min":18.047999999999998,"5min":148.13800000000001,"15min":417.22800000000001},"last":12.34}]}]},{"osd":1,"up_from":14,"seq":60129542501,"num_pgs":6,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":59356,"kb_used_data":6096,"kb_used_omap":1,"kb_used_meta":53246,"kb_avail":94312484,"statfs":{"total":96636764160,"available":96575983616,"internally_reserved":0,"allocated":6242304,"data_stored":3138567,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":54524357},"hb_peers":[0,2,3,4,5,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Mon Jun 3 09:06:07 2024","interfaces":[{"interface":"back","average":{"1min":7.7530000000000001,"5min":185.66999999999999,"15min":77.347999999999999},"min":{"1min":5.0229999999999997,"5min":4.9279999999999999,"15min":4.5449999999999999},"max":{"1min":11.896000000000001,"5min":7470.616,"15min":7470.616},"last":7.7709999999999999},{"interface":"front","average":{"1min":9.9220000000000006,"5min":187.14599999999999,"15min":76.325999999999993},"min":{"1min":5.3159999999999998,"5min":5.3159999999999998,"15min":5.3159999999999998},"max":{"1min":18.338000000000001,"5min":7477.0829999999996,"15min":7477.0829999999996},"last":10.446999999999999}]},{"osd":2,"last update":"Mon Jun 3 09:06:20 2024","interfaces":[{"interface":"back","average":{"1min":10.340999999999999,"5min":17.369,"15min":23.670999999999999},"min":{"1min":6.9740000000000002,"5min":5.8650000000000002,"15min":5.8650000000000002},"max":{"1min":15.452,"5min":228,"15min":657.38},"last":11.384},{"interface":"front","average":{"1min":13.387,"5min":23.355,"15min":31.524000000000001},"min":{"1min":9.077,"5min":9.077,"15min":7.2160000000000002},"max":{"1min":19.701000000000001,"5min":229.465,"15min":700.06500000000005},"last":13.276}]},{"osd":3,"last update":"Mon Jun 3 09:06:04 2024","interfaces":[{"interface":"back","average":{"1min":8.6110000000000007,"5min":15.733000000000001,"15min":26.646999999999998},"min":{"1min":5.827,"5min":5.4640000000000004,"15min":5.4640000000000004},"max":{"1min":13.105,"5min":121.619,"15min":1676.009},"last":7.0140000000000002},{"interface":"front","average":{"1min":9.5850000000000009,"5min":16.161999999999999,"15min":29.065999999999999},"min":{"1min":6.3630000000000004,"5min":5.7699999999999996,"15min":5.7699999999999996},"max":{"1min":15.166,"5min":132.036,"15min":1766.5840000000001},"last":8.3870000000000005}]},{"osd":4,"last update":"Mon Jun 3 09:05:31 2024","interfaces":[{"interface":"back","average":{"1min":10.996,"5min":954.46500000000003,"15min":1281.6859999999999},"min":{"1min":7.702,"5min":6.1760000000000002,"15min":6.0970000000000004},"max":{"1min":21.538,"5min":28044.862000000001,"15min":59438.017999999996},"last":9.5630000000000006},{"interface":"front","average":{"1min":13.646000000000001,"5min":961.24599999999998,"15min":1285.26},"min":{"1min":9.7200000000000006,"5min":9.3940000000000001,"15min":6.4640000000000004},"max":{"1min":23.277999999999999,"5min":28045.624,"15min":59439.283000000003},"last":12.569000000000001}]},{"osd":5,"last update":"Mon Jun 3 09:06:04 2024","interfaces":[{"interface":"back","average":{"1min":12.478999999999999,"5min":20.228000000000002,"15min":25.997},"min":{"1min":8.6430000000000007,"5min":7.9720000000000004,"15min":7.234},"max":{"1min":17.442,"5min":139.43199999999999,"15min":701.22000000000003},"last":17.462},{"interface":"front","average":{"1min":13.507999999999999,"5min":25.760999999999999,"15min":27.553999999999998},"min":{"1min":10.042999999999999,"5min":7.258,"15min":6.242},"max":{"1min":20.611000000000001,"5min":278.63,"15min":598.35000000000002},"last":14.523999999999999}]},{"osd":6,"last update":"Mon Jun 3 09:03:54 2024","interfaces":[{"interface":"back","average":{"1min":28.524000000000001,"5min":20.654,"15min":23.878},"min":{"1min":6.0309999999999997,"5min":6.0309999999999997,"15min":6.0309999999999997},"max":{"1min":167.221,"5min":167.221,"15min":704.245},"last":18.721},{"interface":"front","average":{"1min":27.606000000000002,"5min":30.131,"15min":28.628},"min":{"1min":9.8650000000000002,"5min":9.7200000000000006,"15min":8.3580000000000005},"max":{"1min":123.938,"5min":358.23000000000002,"15min":634.05499999999995},"last":173.113}]},{"osd":7,"last update":"Mon Jun 3 09:06:20 2024","interfaces":[{"interface":"back","average":{"1min":14.66,"5min":23.234999999999999,"15min":31.097000000000001},"min":{"1min":11.250999999999999,"5min":8.0559999999999992,"15min":8.0559999999999992},"max":{"1min":21.507999999999999,"5min":270.95100000000002,"15min":703.07799999999997},"last":15.736000000000001},{"interface":"front","average":{"1min":15.548,"5min":28.760000000000002,"15min":28.925000000000001},"min":{"1min":12.759,"5min":8.2089999999999996,"15min":7.7599999999999998},"max":{"1min":22.408000000000001,"5min":360.99900000000002,"15min":608.44100000000003},"last":16.59}]}]},{"osd":0,"up_from":14,"seq":60129542503,"num_pgs":10,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":81108,"kb_used_data":5704,"kb_used_omap":1,"kb_used_meta":75390,"kb_avail":94290732,"statfs":{"total":96636764160,"available":96553709568,"internally_reserved":0,"allocated":5840896,"data_stored":2740727,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1593,"internal_metadata":77199815},"hb_peers":[1,2,3,4,5,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[{"osd":1,"last update":"Mon Jun 3 09:06:04 2024","interfaces":[{"interface":"back","average":{"1min":6.9960000000000004,"5min":56.997,"15min":339.29700000000003},"min":{"1min":4.4530000000000003,"5min":4.4530000000000003,"15min":4.4100000000000001},"max":{"1min":11.720000000000001,"5min":2414.4749999999999,"15min":20691.453000000001},"last":4.915},{"interface":"front","average":{"1min":9.548,"5min":57.258000000000003,"15min":337.791},"min":{"1min":5.2300000000000004,"5min":5.2300000000000004,"15min":4.6050000000000004},"max":{"1min":15.337999999999999,"5min":2412.9549999999999,"15min":20640.792000000001},"last":9.7650000000000006}]},{"osd":2,"last update":"Mon Jun 3 09:06:22 2024","interfaces":[{"interface":"back","average":{"1min":11.154,"5min":26.515999999999998,"15min":22.882000000000001},"min":{"1min":5.7869999999999999,"5min":5.766,"15min":5.766},"max":{"1min":72.269999999999996,"5min":627.40700000000004,"15min":627.40700000000004},"last":7.2930000000000001},{"interface":"front","average":{"1min":14.353999999999999,"5min":33.304000000000002,"15min":29.885000000000002},"min":{"1min":8.2149999999999999,"5min":7.8630000000000004,"15min":7.4089999999999998},"max":{"1min":73.418000000000006,"5min":635.62599999999998,"15min":635.62599999999998},"last":9.0449999999999999}]},{"osd":3,"last update":"Mon Jun 3 09:06:04 2024","interfaces":[{"interface":"back","average":{"1min":12.105,"5min":26.925000000000001,"15min":26.584},"min":{"1min":7.8410000000000002,"5min":7.1280000000000001,"15min":6.1909999999999998},"max":{"1min":18.425999999999998,"5min":389.72699999999998,"15min":389.72699999999998},"last":12.268000000000001},{"interface":"front","average":{"1min":13.794,"5min":29.771999999999998,"15min":28.815000000000001},"min":{"1min":10.084,"5min":9.4920000000000009,"15min":6.1849999999999996},"max":{"1min":17.373999999999999,"5min":390.23200000000003,"15min":494.92899999999997},"last":14.003}]},{"osd":4,"last update":"Mon Jun 3 09:06:27 2024","interfaces":[{"interface":"back","average":{"1min":11.352,"5min":745.30600000000004,"15min":363.48099999999999},"min":{"1min":6.415,"5min":6.415,"15min":6.415},"max":{"1min":16.724,"5min":20248.616000000002,"15min":25555.212},"last":10.99},{"interface":"front","average":{"1min":13.545,"5min":744.76999999999998,"15min":361.03100000000001},"min":{"1min":9.3889999999999993,"5min":8.8940000000000001,"15min":7.1429999999999998},"max":{"1min":18.411000000000001,"5min":20250.195,"15min":25556.159},"last":14.521000000000001}]},{"osd":5,"last update":"Mon Jun 3 09:06:17 2024","interfaces":[{"interface":"back","average":{"1min":9.7040000000000006,"5min":26.533999999999999,"15min":28.683},"min":{"1min":6.8129999999999997,"5min":6.8129999999999997,"15min":6.8129999999999997},"max":{"1min":14.942,"5min":455.78699999999998,"15min":605.05899999999997},"last":9.4299999999999997},{"interface":"front","average":{"1min":14.279,"5min":28.398,"15min":26.192},"min":{"1min":11.696,"5min":6.4420000000000002,"15min":6.2030000000000003},"max":{"1min":17.879000000000001,"5min":391.31200000000001,"15min":541.11000000000001},"last":15.06}]},{"osd":6,"last update":"Mon Jun 3 09:04:30 2024","interfaces":[{"interface":"back","average":{"1min":52.496000000000002,"5min":33.139000000000003,"15min":160.83199999999999},"min":{"1min":7.5289999999999999,"5min":7.1040000000000001,"15min":6.3739999999999997},"max":{"1min":457.91699999999997,"5min":457.91699999999997,"15min":11115.258},"last":18.959},{"interface":"front","average":{"1min":49.670000000000002,"5min":33.704999999999998,"15min":162.791},"min":{"1min":12.247999999999999,"5min":11.019,"15min":9.9760000000000009},"max":{"1min":407.05099999999999,"5min":407.05099999999999,"15min":11092.194},"last":17.707000000000001}]},{"osd":7,"last update":"Mon Jun 3 09:06:22 2024","interfaces":[{"interface":"back","average":{"1min":11.625999999999999,"5min":26.375,"15min":28.972999999999999},"min":{"1min":6.4480000000000004,"5min":6.4480000000000004,"15min":6.4480000000000004},"max":{"1min":15.352,"5min":461.41699999999997,"15min":598.33699999999999},"last":10.305999999999999},{"interface":"front","average":{"1min":14.013,"5min":29.696000000000002,"15min":33.630000000000003},"min":{"1min":10.938000000000001,"5min":10.58,"15min":10.263999999999999},"max":{"1min":16.603000000000002,"5min":463.10599999999999,"15min":592.80499999999995},"last":11.743}]}]}],"pool_statfs":[{"poolid":1,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":401408,"data_stored":397840,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":5,"total":0,"available":0,"internally_reserved":0,"allocated":401408,"data_stored":397840,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":4,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":5,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":6,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":17,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":17,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":17,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":17,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":17,"osd":4,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":17,"osd":5,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":5,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":17,"osd":6,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":5,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":17,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0}]}} 2024-06-03T09:06:36.741 DEBUG:teuthology.orchestra.run.smithi088:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph pg dump --format=json 2024-06-03T09:06:36.839 INFO:teuthology.orchestra.run.smithi088.stderr:2024-06-03T09:06:36.840+0000 7f74eb2a1640 10 monclient: get_monmap_and_config 2024-06-03T09:06:36.840 INFO:teuthology.orchestra.run.smithi088.stderr:2024-06-03T09:06:36.840+0000 7f74eb2a1640 10 monclient: build_initial_monmap 2024-06-03T09:06:36.840 INFO:teuthology.orchestra.run.smithi088.stderr:2024-06-03T09:06:36.840+0000 7f74eb2a1640 10 monclient: monmap: 2024-06-03T09:06:36.840 INFO:teuthology.orchestra.run.smithi088.stderr:epoch 0 2024-06-03T09:06:36.840 INFO:teuthology.orchestra.run.smithi088.stderr:fsid ec3c3fdd-3ac6-4d7d-9412-683130ed8b04 2024-06-03T09:06:36.840 INFO:teuthology.orchestra.run.smithi088.stderr:last_changed 2024-06-03T09:06:36.840523+0000 2024-06-03T09:06:36.840 INFO:teuthology.orchestra.run.smithi088.stderr:created 2024-06-03T09:06:36.840523+0000 2024-06-03T09:06:36.840 INFO:teuthology.orchestra.run.smithi088.stderr:min_mon_release 0 (unknown) 2024-06-03T09:06:36.840 INFO:teuthology.orchestra.run.smithi088.stderr:election_strategy: 3 2024-06-03T09:06:36.840 INFO:teuthology.orchestra.run.smithi088.stderr:0: v1:172.21.15.88:6789/0 mon.noname-a 2024-06-03T09:06:36.840 INFO:teuthology.orchestra.run.smithi088.stderr:1: v1:172.21.15.192:6789/0 mon.noname-c 2024-06-03T09:06:36.840 INFO:teuthology.orchestra.run.smithi088.stderr:2: v1:172.21.15.88:6790/0 mon.noname-b 2024-06-03T09:06:36.841 INFO:teuthology.orchestra.run.smithi088.stderr: 2024-06-03T09:06:36.841 INFO:teuthology.orchestra.run.smithi088.stderr:2024-06-03T09:06:36.841+0000 7f74eb2a1640 1 Processor -- start 2024-06-03T09:06:36.841 INFO:teuthology.orchestra.run.smithi088.stderr:2024-06-03T09:06:36.841+0000 7f74eb2a1640 1 -- start start 2024-06-03T09:06:36.841 INFO:teuthology.orchestra.run.smithi088.stderr:2024-06-03T09:06:36.841+0000 7f74eb2a1640 10 monclient: init 2024-06-03T09:06:36.842 INFO:teuthology.orchestra.run.smithi088.stderr:2024-06-03T09:06:36.841+0000 7f74eb2a1640 10 monclient: _reopen_session rank -1 2024-06-03T09:06:36.842 INFO:teuthology.orchestra.run.smithi088.stderr:2024-06-03T09:06:36.841+0000 7f74eb2a1640 10 monclient: _add_conns ranks=[1,0,2] 2024-06-03T09:06:36.842 INFO:teuthology.orchestra.run.smithi088.stderr:2024-06-03T09:06:36.841+0000 7f74eb2a1640 10 monclient(hunting): picked mon.noname-c con 0x7f74e40adb90 addr v1:172.21.15.192:6789/0 2024-06-03T09:06:36.842 INFO:teuthology.orchestra.run.smithi088.stderr:2024-06-03T09:06:36.841+0000 7f74eb2a1640 10 monclient(hunting): picked mon.noname-a con 0x7f74e406f020 addr v1:172.21.15.88:6789/0 2024-06-03T09:06:36.842 INFO:teuthology.orchestra.run.smithi088.stderr:2024-06-03T09:06:36.841+0000 7f74eb2a1640 10 monclient(hunting): picked mon.noname-b con 0x7f74e406fc80 addr v1:172.21.15.88:6790/0 2024-06-03T09:06:36.842 INFO:teuthology.orchestra.run.smithi088.stderr:2024-06-03T09:06:36.841+0000 7f74eb2a1640 1 -- --> v1:172.21.15.88:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f74e40716a0 con 0x7f74e406f020 2024-06-03T09:06:36.842 INFO:teuthology.orchestra.run.smithi088.stderr:2024-06-03T09:06:36.841+0000 7f74eb2a1640 1 -- --> v1:172.21.15.192:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f74e40af7d0 con 0x7f74e40adb90 2024-06-03T09:06:36.842 INFO:teuthology.orchestra.run.smithi088.stderr:2024-06-03T09:06:36.841+0000 7f74eb2a1640 1 -- --> v1:172.21.15.88:6790/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f74e413ba00 con 0x7f74e406fc80 2024-06-03T09:06:36.842 INFO:teuthology.orchestra.run.smithi088.stderr:2024-06-03T09:06:36.841+0000 7f74eb2a1640 10 monclient(hunting): _renew_subs 2024-06-03T09:06:36.843 INFO:teuthology.orchestra.run.smithi088.stderr:2024-06-03T09:06:36.841+0000 7f74eb2a1640 10 monclient(hunting): authenticate will time out at 3052.650859s 2024-06-03T09:06:36.843 INFO:teuthology.orchestra.run.smithi088.stderr:2024-06-03T09:06:36.842+0000 7f74e883c640 1 --1- >> v1:172.21.15.88:6789/0 conn(0x7f74e406f020 0x7f74e4141730 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.88:6789/0 says I am v1:172.21.15.88:41628/0 (socket says 172.21.15.88:41628) 2024-06-03T09:06:36.843 INFO:teuthology.orchestra.run.smithi088.stderr:2024-06-03T09:06:36.842+0000 7f74e883c640 1 -- 172.21.15.88:0/2116143032 learned_addr learned my addr 172.21.15.88:0/2116143032 (peer_addr_for_me v1:172.21.15.88:0/0) 2024-06-03T09:06:36.843 INFO:teuthology.orchestra.run.smithi088.stderr:2024-06-03T09:06:36.844+0000 7f74d7fff640 1 -- 172.21.15.88:0/2116143032 <== mon.1 v1:172.21.15.192:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 2948135305 0 0) 0x7f74e40af7d0 con 0x7f74e40adb90 2024-06-03T09:06:36.843 INFO:teuthology.orchestra.run.smithi088.stderr:2024-06-03T09:06:36.844+0000 7f74d7fff640 10 monclient(hunting): _negotiate 2024-06-03T09:06:36.843 INFO:teuthology.orchestra.run.smithi088.stderr:2024-06-03T09:06:36.844+0000 7f74d7fff640 10 monclient(hunting): _init_auth method 2 2024-06-03T09:06:36.843 INFO:teuthology.orchestra.run.smithi088.stderr:2024-06-03T09:06:36.844+0000 7f74d7fff640 10 monclient(hunting): _init_auth creating new auth 2024-06-03T09:06:36.844 INFO:teuthology.orchestra.run.smithi088.stderr:2024-06-03T09:06:36.844+0000 7f74d7fff640 10 monclient(hunting): my global_id is 30229 2024-06-03T09:06:36.844 INFO:teuthology.orchestra.run.smithi088.stderr:2024-06-03T09:06:36.844+0000 7f74d7fff640 1 -- 172.21.15.88:0/2116143032 --> v1:172.21.15.192:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f74d0003520 con 0x7f74e40adb90 2024-06-03T09:06:36.844 INFO:teuthology.orchestra.run.smithi088.stderr:2024-06-03T09:06:36.845+0000 7f74d7fff640 1 -- 172.21.15.88:0/2116143032 <== mon.0 v1:172.21.15.88:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 962911907 0 0) 0x7f74e40716a0 con 0x7f74e406f020 2024-06-03T09:06:36.844 INFO:teuthology.orchestra.run.smithi088.stderr:2024-06-03T09:06:36.845+0000 7f74d7fff640 10 monclient(hunting): _negotiate 2024-06-03T09:06:36.845 INFO:teuthology.orchestra.run.smithi088.stderr:2024-06-03T09:06:36.845+0000 7f74d7fff640 10 monclient(hunting): _init_auth method 2 2024-06-03T09:06:36.845 INFO:teuthology.orchestra.run.smithi088.stderr:2024-06-03T09:06:36.845+0000 7f74d7fff640 10 monclient(hunting): _init_auth creating new auth 2024-06-03T09:06:36.845 INFO:teuthology.orchestra.run.smithi088.stderr:2024-06-03T09:06:36.845+0000 7f74d7fff640 10 monclient(hunting): my global_id is 30648 2024-06-03T09:06:36.845 INFO:teuthology.orchestra.run.smithi088.stderr:2024-06-03T09:06:36.845+0000 7f74d7fff640 1 -- 172.21.15.88:0/2116143032 --> v1:172.21.15.88:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f74e40af7d0 con 0x7f74e406f020 2024-06-03T09:06:36.845 INFO:teuthology.orchestra.run.smithi088.stderr:2024-06-03T09:06:36.845+0000 7f74d7fff640 1 -- 172.21.15.88:0/2116143032 <== mon.2 v1:172.21.15.88:6790/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 2151728070 0 0) 0x7f74e413ba00 con 0x7f74e406fc80 2024-06-03T09:06:36.845 INFO:teuthology.orchestra.run.smithi088.stderr:2024-06-03T09:06:36.845+0000 7f74d7fff640 10 monclient(hunting): _negotiate 2024-06-03T09:06:36.845 INFO:teuthology.orchestra.run.smithi088.stderr:2024-06-03T09:06:36.845+0000 7f74d7fff640 10 monclient(hunting): _init_auth method 2 2024-06-03T09:06:36.845 INFO:teuthology.orchestra.run.smithi088.stderr:2024-06-03T09:06:36.845+0000 7f74d7fff640 10 monclient(hunting): _init_auth creating new auth 2024-06-03T09:06:36.845 INFO:teuthology.orchestra.run.smithi088.stderr:2024-06-03T09:06:36.845+0000 7f74d7fff640 10 monclient(hunting): my global_id is 30152 2024-06-03T09:06:36.845 INFO:teuthology.orchestra.run.smithi088.stderr:2024-06-03T09:06:36.845+0000 7f74d7fff640 1 -- 172.21.15.88:0/2116143032 --> v1:172.21.15.88:6790/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f74e40716a0 con 0x7f74e406fc80 2024-06-03T09:06:36.847 INFO:teuthology.orchestra.run.smithi088.stderr:2024-06-03T09:06:36.848+0000 7f74d7fff640 1 -- 172.21.15.88:0/2116143032 <== mon.1 v1:172.21.15.192:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 214+0+0 (unknown 2662951071 0 0) 0x7f74d0003520 con 0x7f74e40adb90 2024-06-03T09:06:36.847 INFO:teuthology.orchestra.run.smithi088.stderr:2024-06-03T09:06:36.848+0000 7f74d7fff640 1 -- 172.21.15.88:0/2116143032 >> v1:172.21.15.88:6790/0 conn(0x7f74e406fc80 legacy=0x7f74e413b2f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-06-03T09:06:36.847 INFO:teuthology.orchestra.run.smithi088.stderr:2024-06-03T09:06:36.848+0000 7f74d7fff640 1 -- 172.21.15.88:0/2116143032 >> v1:172.21.15.88:6789/0 conn(0x7f74e406f020 legacy=0x7f74e4141730 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-06-03T09:06:36.848 INFO:teuthology.orchestra.run.smithi088.stderr:2024-06-03T09:06:36.848+0000 7f74d7fff640 10 monclient: _finish_hunting 0 2024-06-03T09:06:36.848 INFO:teuthology.orchestra.run.smithi088.stderr:2024-06-03T09:06:36.848+0000 7f74d7fff640 1 monclient: found mon.noname-c 2024-06-03T09:06:36.848 INFO:teuthology.orchestra.run.smithi088.stderr:2024-06-03T09:06:36.848+0000 7f74d7fff640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-06-03T09:06:36.848 INFO:teuthology.orchestra.run.smithi088.stderr:2024-06-03T09:06:36.848+0000 7f74d7fff640 10 monclient: _send_mon_message to mon.noname-c at v1:172.21.15.192:6789/0 2024-06-03T09:06:36.848 INFO:teuthology.orchestra.run.smithi088.stderr:2024-06-03T09:06:36.848+0000 7f74d7fff640 1 -- 172.21.15.88:0/2116143032 --> v1:172.21.15.192:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f74e413bb90 con 0x7f74e40adb90 2024-06-03T09:06:36.848 INFO:teuthology.orchestra.run.smithi088.stderr:2024-06-03T09:06:36.848+0000 7f74d7fff640 10 monclient: _finish_auth 0 2024-06-03T09:06:36.848 INFO:teuthology.orchestra.run.smithi088.stderr:2024-06-03T09:06:36.848+0000 7f74d7fff640 10 monclient: _check_auth_tickets 2024-06-03T09:06:36.848 INFO:teuthology.orchestra.run.smithi088.stderr:2024-06-03T09:06:36.848+0000 7f74d7fff640 20 monclient: _check_auth_rotating not needed by client.admin 2024-06-03T09:06:36.848 INFO:teuthology.orchestra.run.smithi088.stderr:2024-06-03T09:06:36.848+0000 7f74d7fff640 1 -- 172.21.15.88:0/2116143032 <== mon.1 v1:172.21.15.192:6789/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 2234374065 0 0) 0x7f74e0001610 con 0x7f74e40adb90 2024-06-03T09:06:36.848 INFO:teuthology.orchestra.run.smithi088.stderr:2024-06-03T09:06:36.848+0000 7f74eb2a1640 5 monclient: authenticate success, global_id 30229 2024-06-03T09:06:36.848 INFO:teuthology.orchestra.run.smithi088.stderr:2024-06-03T09:06:36.848+0000 7f74eb2a1640 20 monclient: get_monmap_and_config waiting for monmap|config 2024-06-03T09:06:36.849 INFO:teuthology.orchestra.run.smithi088.stderr:2024-06-03T09:06:36.848+0000 7f74d7fff640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-06-03T09:06:36.849 INFO:teuthology.orchestra.run.smithi088.stderr:2024-06-03T09:06:36.848+0000 7f74d7fff640 10 monclient: got monmap 1 from mon.noname-c (according to old e1) 2024-06-03T09:06:36.849 INFO:teuthology.orchestra.run.smithi088.stderr:2024-06-03T09:06:36.848+0000 7f74d7fff640 10 monclient: dump: 2024-06-03T09:06:36.849 INFO:teuthology.orchestra.run.smithi088.stderr:epoch 1 2024-06-03T09:06:36.849 INFO:teuthology.orchestra.run.smithi088.stderr:fsid ec3c3fdd-3ac6-4d7d-9412-683130ed8b04 2024-06-03T09:06:36.849 INFO:teuthology.orchestra.run.smithi088.stderr:last_changed 2024-06-03T08:34:31.885125+0000 2024-06-03T09:06:36.849 INFO:teuthology.orchestra.run.smithi088.stderr:created 2024-06-03T08:34:31.885125+0000 2024-06-03T09:06:36.849 INFO:teuthology.orchestra.run.smithi088.stderr:min_mon_release 18 (reef) 2024-06-03T09:06:36.849 INFO:teuthology.orchestra.run.smithi088.stderr:election_strategy: 3 2024-06-03T09:06:36.849 INFO:teuthology.orchestra.run.smithi088.stderr:0: v1:172.21.15.88:6789/0 mon.a 2024-06-03T09:06:36.849 INFO:teuthology.orchestra.run.smithi088.stderr:1: v1:172.21.15.192:6789/0 mon.b 2024-06-03T09:06:36.849 INFO:teuthology.orchestra.run.smithi088.stderr:2: v1:172.21.15.88:6790/0 mon.c 2024-06-03T09:06:36.850 INFO:teuthology.orchestra.run.smithi088.stderr: 2024-06-03T09:06:36.850 INFO:teuthology.orchestra.run.smithi088.stderr:2024-06-03T09:06:36.848+0000 7f74eb2a1640 20 monclient: get_monmap_and_config waiting for monmap|config 2024-06-03T09:06:36.853 INFO:teuthology.orchestra.run.smithi088.stderr:2024-06-03T09:06:36.849+0000 7f74d7fff640 1 -- 172.21.15.88:0/2116143032 <== mon.1 v1:172.21.15.192:6789/0 4 ==== config(0 keys) v1 ==== 4+0+0 (unknown 0 0 0) 0x7f74e0001830 con 0x7f74e40adb90 2024-06-03T09:06:36.853 INFO:teuthology.orchestra.run.smithi088.stderr:2024-06-03T09:06:36.849+0000 7f74d7fff640 10 monclient: handle_config config(0 keys) v1 2024-06-03T09:06:36.854 INFO:teuthology.orchestra.run.smithi088.stderr:2024-06-03T09:06:36.849+0000 7f74eb2a1640 10 monclient: get_monmap_and_config success 2024-06-03T09:06:36.854 INFO:teuthology.orchestra.run.smithi088.stderr:2024-06-03T09:06:36.849+0000 7f74eb2a1640 4 set_mon_vals no callback set 2024-06-03T09:06:36.854 INFO:teuthology.orchestra.run.smithi088.stderr:2024-06-03T09:06:36.849+0000 7f74d7fff640 1 -- 172.21.15.88:0/2116143032 <== mon.1 v1:172.21.15.192:6789/0 5 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 2234374065 0 0) 0x7f74e0001bc0 con 0x7f74e40adb90 2024-06-03T09:06:36.854 INFO:teuthology.orchestra.run.smithi088.stderr:2024-06-03T09:06:36.849+0000 7f74d7fff640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-06-03T09:06:36.854 INFO:teuthology.orchestra.run.smithi088.stderr:2024-06-03T09:06:36.849+0000 7f74d7fff640 10 monclient: got monmap 1 from mon.b (according to old e1) 2024-06-03T09:06:36.854 INFO:teuthology.orchestra.run.smithi088.stderr:2024-06-03T09:06:36.849+0000 7f74d7fff640 10 monclient: dump: 2024-06-03T09:06:36.854 INFO:teuthology.orchestra.run.smithi088.stderr:epoch 1 2024-06-03T09:06:36.854 INFO:teuthology.orchestra.run.smithi088.stderr:fsid ec3c3fdd-3ac6-4d7d-9412-683130ed8b04 2024-06-03T09:06:36.854 INFO:teuthology.orchestra.run.smithi088.stderr:last_changed 2024-06-03T08:34:31.885125+0000 2024-06-03T09:06:36.854 INFO:teuthology.orchestra.run.smithi088.stderr:created 2024-06-03T08:34:31.885125+0000 2024-06-03T09:06:36.854 INFO:teuthology.orchestra.run.smithi088.stderr:min_mon_release 18 (reef) 2024-06-03T09:06:36.855 INFO:teuthology.orchestra.run.smithi088.stderr:election_strategy: 3 2024-06-03T09:06:36.855 INFO:teuthology.orchestra.run.smithi088.stderr:0: v1:172.21.15.88:6789/0 mon.a 2024-06-03T09:06:36.855 INFO:teuthology.orchestra.run.smithi088.stderr:1: v1:172.21.15.192:6789/0 mon.b 2024-06-03T09:06:36.855 INFO:teuthology.orchestra.run.smithi088.stderr:2: v1:172.21.15.88:6790/0 mon.c 2024-06-03T09:06:36.855 INFO:teuthology.orchestra.run.smithi088.stderr: 2024-06-03T09:06:36.855 INFO:teuthology.orchestra.run.smithi088.stderr:2024-06-03T09:06:36.850+0000 7f74eb2a1640 10 monclient: shutdown 2024-06-03T09:06:36.855 INFO:teuthology.orchestra.run.smithi088.stderr:2024-06-03T09:06:36.850+0000 7f74eb2a1640 20 monclient: shutdown discarding 0 pending message(s) 2024-06-03T09:06:36.855 INFO:teuthology.orchestra.run.smithi088.stderr:2024-06-03T09:06:36.850+0000 7f74eb2a1640 1 -- 172.21.15.88:0/2116143032 >> v1:172.21.15.192:6789/0 conn(0x7f74e40adb90 legacy=0x7f74e4071b30 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-06-03T09:06:36.855 INFO:teuthology.orchestra.run.smithi088.stderr:2024-06-03T09:06:36.850+0000 7f74eb2a1640 1 -- 172.21.15.88:0/2116143032 shutdown_connections 2024-06-03T09:06:36.855 INFO:teuthology.orchestra.run.smithi088.stderr:2024-06-03T09:06:36.850+0000 7f74eb2a1640 1 -- 172.21.15.88:0/2116143032 >> 172.21.15.88:0/2116143032 conn(0x7f74e40667e0 msgr2=0x7f74e40b2f50 unknown :-1 s=STATE_NONE l=0).mark_down 2024-06-03T09:06:36.856 INFO:teuthology.orchestra.run.smithi088.stderr:2024-06-03T09:06:36.850+0000 7f74eb2a1640 1 -- 172.21.15.88:0/2116143032 shutdown_connections 2024-06-03T09:06:36.856 INFO:teuthology.orchestra.run.smithi088.stderr:2024-06-03T09:06:36.850+0000 7f74eb2a1640 1 -- 172.21.15.88:0/2116143032 wait complete. 2024-06-03T09:06:36.856 INFO:teuthology.orchestra.run.smithi088.stderr:2024-06-03T09:06:36.850+0000 7f74eb2a1640 10 monclient: build_initial_monmap 2024-06-03T09:06:36.856 INFO:teuthology.orchestra.run.smithi088.stderr:2024-06-03T09:06:36.850+0000 7f74eb2a1640 10 monclient: monmap: 2024-06-03T09:06:36.856 INFO:teuthology.orchestra.run.smithi088.stderr:epoch 0 2024-06-03T09:06:36.856 INFO:teuthology.orchestra.run.smithi088.stderr:fsid 00000000-0000-0000-0000-000000000000 2024-06-03T09:06:36.856 INFO:teuthology.orchestra.run.smithi088.stderr:last_changed 0.000000 2024-06-03T09:06:36.856 INFO:teuthology.orchestra.run.smithi088.stderr:created 0.000000 2024-06-03T09:06:36.856 INFO:teuthology.orchestra.run.smithi088.stderr:min_mon_release 0 (unknown) 2024-06-03T09:06:36.856 INFO:teuthology.orchestra.run.smithi088.stderr:election_strategy: 1 2024-06-03T09:06:36.856 INFO:teuthology.orchestra.run.smithi088.stderr:0: v1:172.21.15.88:6789/0 mon.noname-a 2024-06-03T09:06:36.856 INFO:teuthology.orchestra.run.smithi088.stderr:1: v1:172.21.15.192:6789/0 mon.noname-b 2024-06-03T09:06:36.857 INFO:teuthology.orchestra.run.smithi088.stderr:2: v1:172.21.15.88:6790/0 mon.noname-c 2024-06-03T09:06:36.857 INFO:teuthology.orchestra.run.smithi088.stderr: 2024-06-03T09:06:36.857 INFO:teuthology.orchestra.run.smithi088.stderr:2024-06-03T09:06:36.851+0000 7f74eb2a1640 1 librados: starting msgr at 2024-06-03T09:06:36.857 INFO:teuthology.orchestra.run.smithi088.stderr:2024-06-03T09:06:36.851+0000 7f74eb2a1640 1 librados: starting objecter 2024-06-03T09:06:36.857 INFO:teuthology.orchestra.run.smithi088.stderr:2024-06-03T09:06:36.851+0000 7f74eb2a1640 1 Processor -- start 2024-06-03T09:06:36.857 INFO:teuthology.orchestra.run.smithi088.stderr:2024-06-03T09:06:36.851+0000 7f74eb2a1640 1 -- start start 2024-06-03T09:06:36.857 INFO:teuthology.orchestra.run.smithi088.stderr:2024-06-03T09:06:36.851+0000 7f74eb2a1640 1 librados: setting wanted keys 2024-06-03T09:06:36.857 INFO:teuthology.orchestra.run.smithi088.stderr:2024-06-03T09:06:36.851+0000 7f74eb2a1640 1 librados: calling monclient init 2024-06-03T09:06:36.857 INFO:teuthology.orchestra.run.smithi088.stderr:2024-06-03T09:06:36.851+0000 7f74eb2a1640 10 monclient: init 2024-06-03T09:06:36.857 INFO:teuthology.orchestra.run.smithi088.stderr:2024-06-03T09:06:36.851+0000 7f74eb2a1640 10 monclient: _reopen_session rank -1 2024-06-03T09:06:36.858 INFO:teuthology.orchestra.run.smithi088.stderr:2024-06-03T09:06:36.851+0000 7f74eb2a1640 10 monclient: _add_conns ranks=[0,2,1] 2024-06-03T09:06:36.858 INFO:teuthology.orchestra.run.smithi088.stderr:2024-06-03T09:06:36.851+0000 7f74eb2a1640 10 monclient(hunting): picked mon.noname-a con 0x7f74e406f020 addr v1:172.21.15.88:6789/0 2024-06-03T09:06:36.858 INFO:teuthology.orchestra.run.smithi088.stderr:2024-06-03T09:06:36.851+0000 7f74eb2a1640 10 monclient(hunting): picked mon.noname-c con 0x7f74e406fc80 addr v1:172.21.15.88:6790/0 2024-06-03T09:06:36.858 INFO:teuthology.orchestra.run.smithi088.stderr:2024-06-03T09:06:36.851+0000 7f74eb2a1640 10 monclient(hunting): picked mon.noname-b con 0x7f74e40adb90 addr v1:172.21.15.192:6789/0 2024-06-03T09:06:36.858 INFO:teuthology.orchestra.run.smithi088.stderr:2024-06-03T09:06:36.851+0000 7f74eb2a1640 1 -- --> v1:172.21.15.88:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f74e41da230 con 0x7f74e406f020 2024-06-03T09:06:36.858 INFO:teuthology.orchestra.run.smithi088.stderr:2024-06-03T09:06:36.851+0000 7f74eb2a1640 1 -- --> v1:172.21.15.192:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f74e4133260 con 0x7f74e40adb90 2024-06-03T09:06:36.858 INFO:teuthology.orchestra.run.smithi088.stderr:2024-06-03T09:06:36.851+0000 7f74eb2a1640 1 -- --> v1:172.21.15.88:6790/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f74e4132300 con 0x7f74e406fc80 2024-06-03T09:06:36.858 INFO:teuthology.orchestra.run.smithi088.stderr:2024-06-03T09:06:36.851+0000 7f74eb2a1640 10 monclient(hunting): _renew_subs 2024-06-03T09:06:36.858 INFO:teuthology.orchestra.run.smithi088.stderr:2024-06-03T09:06:36.851+0000 7f74eb2a1640 10 monclient(hunting): authenticate will time out at 3052.660769s 2024-06-03T09:06:36.859 INFO:teuthology.orchestra.run.smithi088.stderr:2024-06-03T09:06:36.852+0000 7f74e883c640 1 --1- >> v1:172.21.15.88:6790/0 conn(0x7f74e406fc80 0x7f74e412f020 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=1).handle_server_banner_and_identify peer v1:172.21.15.88:6790/0 says I am v1:172.21.15.88:36428/0 (socket says 172.21.15.88:36428) 2024-06-03T09:06:36.859 INFO:teuthology.orchestra.run.smithi088.stderr:2024-06-03T09:06:36.852+0000 7f74e903d640 1 --1- >> v1:172.21.15.88:6789/0 conn(0x7f74e406f020 0x7f74e41d9b20 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=1).handle_server_banner_and_identify peer v1:172.21.15.88:6789/0 says I am v1:172.21.15.88:41630/0 (socket says 172.21.15.88:41630) 2024-06-03T09:06:36.859 INFO:teuthology.orchestra.run.smithi088.stderr:2024-06-03T09:06:36.852+0000 7f74e883c640 1 -- 172.21.15.88:0/2134435251 learned_addr learned my addr 172.21.15.88:0/2134435251 (peer_addr_for_me v1:172.21.15.88:0/0) 2024-06-03T09:06:36.859 INFO:teuthology.orchestra.run.smithi088.stderr:2024-06-03T09:06:36.852+0000 7f74d5ffb640 10 client.?.objecter ms_handle_connect 0x7f74e406f020 2024-06-03T09:06:36.859 INFO:teuthology.orchestra.run.smithi088.stderr:2024-06-03T09:06:36.852+0000 7f74d5ffb640 10 client.?.objecter resend_mon_ops 2024-06-03T09:06:36.859 INFO:teuthology.orchestra.run.smithi088.stderr:2024-06-03T09:06:36.852+0000 7f74d5ffb640 10 client.?.objecter ms_handle_connect 0x7f74e406fc80 2024-06-03T09:06:36.859 INFO:teuthology.orchestra.run.smithi088.stderr:2024-06-03T09:06:36.852+0000 7f74d5ffb640 10 client.?.objecter resend_mon_ops 2024-06-03T09:06:36.859 INFO:teuthology.orchestra.run.smithi088.stderr:2024-06-03T09:06:36.852+0000 7f74d5ffb640 10 client.?.objecter ms_handle_connect 0x7f74e40adb90 2024-06-03T09:06:36.859 INFO:teuthology.orchestra.run.smithi088.stderr:2024-06-03T09:06:36.852+0000 7f74d5ffb640 10 client.?.objecter resend_mon_ops 2024-06-03T09:06:36.860 INFO:teuthology.orchestra.run.smithi088.stderr:2024-06-03T09:06:36.854+0000 7f74d5ffb640 1 -- 172.21.15.88:0/2134435251 <== mon.1 v1:172.21.15.192:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 720752278 0 0) 0x7f74e4133260 con 0x7f74e40adb90 2024-06-03T09:06:36.860 INFO:teuthology.orchestra.run.smithi088.stderr:2024-06-03T09:06:36.854+0000 7f74d5ffb640 10 monclient(hunting): _negotiate 2024-06-03T09:06:36.860 INFO:teuthology.orchestra.run.smithi088.stderr:2024-06-03T09:06:36.854+0000 7f74d5ffb640 10 monclient(hunting): _init_auth method 2 2024-06-03T09:06:36.860 INFO:teuthology.orchestra.run.smithi088.stderr:2024-06-03T09:06:36.854+0000 7f74d5ffb640 10 monclient(hunting): _init_auth creating new auth 2024-06-03T09:06:36.860 INFO:teuthology.orchestra.run.smithi088.stderr:2024-06-03T09:06:36.854+0000 7f74d5ffb640 10 monclient(hunting): my global_id is 30232 2024-06-03T09:06:36.860 INFO:teuthology.orchestra.run.smithi088.stderr:2024-06-03T09:06:36.854+0000 7f74d5ffb640 1 -- 172.21.15.88:0/2134435251 --> v1:172.21.15.192:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f74c8004eb0 con 0x7f74e40adb90 2024-06-03T09:06:36.860 INFO:teuthology.orchestra.run.smithi088.stderr:2024-06-03T09:06:36.854+0000 7f74d5ffb640 1 -- 172.21.15.88:0/2134435251 <== mon.0 v1:172.21.15.88:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 4032127753 0 0) 0x7f74e41da230 con 0x7f74e406f020 2024-06-03T09:06:36.860 INFO:teuthology.orchestra.run.smithi088.stderr:2024-06-03T09:06:36.854+0000 7f74d5ffb640 10 monclient(hunting): _negotiate 2024-06-03T09:06:36.860 INFO:teuthology.orchestra.run.smithi088.stderr:2024-06-03T09:06:36.854+0000 7f74d5ffb640 10 monclient(hunting): _init_auth method 2 2024-06-03T09:06:36.860 INFO:teuthology.orchestra.run.smithi088.stderr:2024-06-03T09:06:36.854+0000 7f74d5ffb640 10 monclient(hunting): _init_auth creating new auth 2024-06-03T09:06:36.860 INFO:teuthology.orchestra.run.smithi088.stderr:2024-06-03T09:06:36.854+0000 7f74d5ffb640 10 monclient(hunting): my global_id is 30651 2024-06-03T09:06:36.860 INFO:teuthology.orchestra.run.smithi088.stderr:2024-06-03T09:06:36.854+0000 7f74d5ffb640 1 -- 172.21.15.88:0/2134435251 --> v1:172.21.15.88:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f74e4133260 con 0x7f74e406f020 2024-06-03T09:06:36.860 INFO:teuthology.orchestra.run.smithi088.stderr:2024-06-03T09:06:36.854+0000 7f74d5ffb640 1 -- 172.21.15.88:0/2134435251 <== mon.2 v1:172.21.15.88:6790/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 112368710 0 0) 0x7f74e4132300 con 0x7f74e406fc80 2024-06-03T09:06:36.861 INFO:teuthology.orchestra.run.smithi088.stderr:2024-06-03T09:06:36.854+0000 7f74d5ffb640 10 monclient(hunting): _negotiate 2024-06-03T09:06:36.861 INFO:teuthology.orchestra.run.smithi088.stderr:2024-06-03T09:06:36.854+0000 7f74d5ffb640 10 monclient(hunting): _init_auth method 2 2024-06-03T09:06:36.861 INFO:teuthology.orchestra.run.smithi088.stderr:2024-06-03T09:06:36.854+0000 7f74d5ffb640 10 monclient(hunting): _init_auth creating new auth 2024-06-03T09:06:36.861 INFO:teuthology.orchestra.run.smithi088.stderr:2024-06-03T09:06:36.854+0000 7f74d5ffb640 10 monclient(hunting): my global_id is 30155 2024-06-03T09:06:36.861 INFO:teuthology.orchestra.run.smithi088.stderr:2024-06-03T09:06:36.854+0000 7f74d5ffb640 1 -- 172.21.15.88:0/2134435251 --> v1:172.21.15.88:6790/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f74e41da230 con 0x7f74e406fc80 2024-06-03T09:06:36.863 INFO:teuthology.orchestra.run.smithi088.stderr:2024-06-03T09:06:36.858+0000 7f74d5ffb640 1 -- 172.21.15.88:0/2134435251 <== mon.1 v1:172.21.15.192:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 764+0+0 (unknown 4080616999 0 0) 0x7f74c8004eb0 con 0x7f74e40adb90 2024-06-03T09:06:36.863 INFO:teuthology.orchestra.run.smithi088.stderr:2024-06-03T09:06:36.858+0000 7f74d5ffb640 1 -- 172.21.15.88:0/2134435251 --> v1:172.21.15.192:6789/0 -- auth(proto 2 165 bytes epoch 0) v1 -- 0x7f74e4132300 con 0x7f74e40adb90 2024-06-03T09:06:36.863 INFO:teuthology.orchestra.run.smithi088.stderr:2024-06-03T09:06:36.858+0000 7f74d5ffb640 1 -- 172.21.15.88:0/2134435251 <== mon.1 v1:172.21.15.192:6789/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 2234374065 0 0) 0x7f74dc001c30 con 0x7f74e40adb90 2024-06-03T09:06:36.863 INFO:teuthology.orchestra.run.smithi088.stderr:2024-06-03T09:06:36.858+0000 7f74d5ffb640 10 monclient(hunting): handle_monmap mon_map magic: 0 v1 2024-06-03T09:06:36.863 INFO:teuthology.orchestra.run.smithi088.stderr:2024-06-03T09:06:36.858+0000 7f74d5ffb640 10 monclient(hunting): got monmap 1 from mon.noname-b (according to old e1) 2024-06-03T09:06:36.863 INFO:teuthology.orchestra.run.smithi088.stderr:2024-06-03T09:06:36.859+0000 7f74d5ffb640 10 monclient(hunting): dump: 2024-06-03T09:06:36.863 INFO:teuthology.orchestra.run.smithi088.stderr:epoch 1 2024-06-03T09:06:36.864 INFO:teuthology.orchestra.run.smithi088.stderr:fsid ec3c3fdd-3ac6-4d7d-9412-683130ed8b04 2024-06-03T09:06:36.864 INFO:teuthology.orchestra.run.smithi088.stderr:last_changed 2024-06-03T08:34:31.885125+0000 2024-06-03T09:06:36.864 INFO:teuthology.orchestra.run.smithi088.stderr:created 2024-06-03T08:34:31.885125+0000 2024-06-03T09:06:36.864 INFO:teuthology.orchestra.run.smithi088.stderr:min_mon_release 18 (reef) 2024-06-03T09:06:36.864 INFO:teuthology.orchestra.run.smithi088.stderr:election_strategy: 3 2024-06-03T09:06:36.864 INFO:teuthology.orchestra.run.smithi088.stderr:0: v1:172.21.15.88:6789/0 mon.a 2024-06-03T09:06:36.864 INFO:teuthology.orchestra.run.smithi088.stderr:1: v1:172.21.15.192:6789/0 mon.b 2024-06-03T09:06:36.864 INFO:teuthology.orchestra.run.smithi088.stderr:2: v1:172.21.15.88:6790/0 mon.c 2024-06-03T09:06:36.864 INFO:teuthology.orchestra.run.smithi088.stderr: 2024-06-03T09:06:36.864 INFO:teuthology.orchestra.run.smithi088.stderr:2024-06-03T09:06:36.859+0000 7f74d5ffb640 10 monclient(hunting): _finish_auth 0 2024-06-03T09:06:36.864 INFO:teuthology.orchestra.run.smithi088.stderr:2024-06-03T09:06:36.859+0000 7f74d5ffb640 1 -- 172.21.15.88:0/2134435251 <== mon.2 v1:172.21.15.88:6790/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 764+0+0 (unknown 2410667732 0 0) 0x7f74e41da230 con 0x7f74e406fc80 2024-06-03T09:06:36.864 INFO:teuthology.orchestra.run.smithi088.stderr:2024-06-03T09:06:36.859+0000 7f74d5ffb640 1 -- 172.21.15.88:0/2134435251 --> v1:172.21.15.88:6790/0 -- auth(proto 2 165 bytes epoch 0) v1 -- 0x7f74c8004eb0 con 0x7f74e406fc80 2024-06-03T09:06:36.865 INFO:teuthology.orchestra.run.smithi088.stderr:2024-06-03T09:06:36.859+0000 7f74d5ffb640 1 -- 172.21.15.88:0/2134435251 <== mon.2 v1:172.21.15.88:6790/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 2234374065 0 0) 0x7f74d80018b0 con 0x7f74e406fc80 2024-06-03T09:06:36.865 INFO:teuthology.orchestra.run.smithi088.stderr:2024-06-03T09:06:36.859+0000 7f74d5ffb640 10 monclient(hunting): handle_monmap mon_map magic: 0 v1 2024-06-03T09:06:36.865 INFO:teuthology.orchestra.run.smithi088.stderr:2024-06-03T09:06:36.859+0000 7f74d5ffb640 10 monclient(hunting): got monmap 1 from mon.c (according to old e1) 2024-06-03T09:06:36.865 INFO:teuthology.orchestra.run.smithi088.stderr:2024-06-03T09:06:36.859+0000 7f74d5ffb640 10 monclient(hunting): dump: 2024-06-03T09:06:36.865 INFO:teuthology.orchestra.run.smithi088.stderr:epoch 1 2024-06-03T09:06:36.865 INFO:teuthology.orchestra.run.smithi088.stderr:fsid ec3c3fdd-3ac6-4d7d-9412-683130ed8b04 2024-06-03T09:06:36.865 INFO:teuthology.orchestra.run.smithi088.stderr:last_changed 2024-06-03T08:34:31.885125+0000 2024-06-03T09:06:36.866 INFO:teuthology.orchestra.run.smithi088.stderr:created 2024-06-03T08:34:31.885125+0000 2024-06-03T09:06:36.866 INFO:teuthology.orchestra.run.smithi088.stderr:min_mon_release 18 (reef) 2024-06-03T09:06:36.866 INFO:teuthology.orchestra.run.smithi088.stderr:election_strategy: 3 2024-06-03T09:06:36.866 INFO:teuthology.orchestra.run.smithi088.stderr:0: v1:172.21.15.88:6789/0 mon.a 2024-06-03T09:06:36.866 INFO:teuthology.orchestra.run.smithi088.stderr:1: v1:172.21.15.192:6789/0 mon.b 2024-06-03T09:06:36.866 INFO:teuthology.orchestra.run.smithi088.stderr:2: v1:172.21.15.88:6790/0 mon.c 2024-06-03T09:06:36.866 INFO:teuthology.orchestra.run.smithi088.stderr: 2024-06-03T09:06:36.866 INFO:teuthology.orchestra.run.smithi088.stderr:2024-06-03T09:06:36.859+0000 7f74d5ffb640 1 -- 172.21.15.88:0/2134435251 <== mon.0 v1:172.21.15.88:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 764+0+0 (unknown 1752751724 0 0) 0x7f74e4133260 con 0x7f74e406f020 2024-06-03T09:06:36.866 INFO:teuthology.orchestra.run.smithi088.stderr:2024-06-03T09:06:36.859+0000 7f74d5ffb640 1 -- 172.21.15.88:0/2134435251 --> v1:172.21.15.88:6789/0 -- auth(proto 2 165 bytes epoch 0) v1 -- 0x7f74e41da230 con 0x7f74e406f020 2024-06-03T09:06:36.866 INFO:teuthology.orchestra.run.smithi088.stderr:2024-06-03T09:06:36.859+0000 7f74d5ffb640 1 -- 172.21.15.88:0/2134435251 <== mon.0 v1:172.21.15.88:6789/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 2234374065 0 0) 0x7f74e0001510 con 0x7f74e406f020 2024-06-03T09:06:36.867 INFO:teuthology.orchestra.run.smithi088.stderr:2024-06-03T09:06:36.859+0000 7f74d5ffb640 10 monclient(hunting): handle_monmap mon_map magic: 0 v1 2024-06-03T09:06:36.867 INFO:teuthology.orchestra.run.smithi088.stderr:2024-06-03T09:06:36.859+0000 7f74d5ffb640 10 monclient(hunting): got monmap 1 from mon.a (according to old e1) 2024-06-03T09:06:36.867 INFO:teuthology.orchestra.run.smithi088.stderr:2024-06-03T09:06:36.859+0000 7f74d5ffb640 10 monclient(hunting): dump: 2024-06-03T09:06:36.867 INFO:teuthology.orchestra.run.smithi088.stderr:epoch 1 2024-06-03T09:06:36.867 INFO:teuthology.orchestra.run.smithi088.stderr:fsid ec3c3fdd-3ac6-4d7d-9412-683130ed8b04 2024-06-03T09:06:36.867 INFO:teuthology.orchestra.run.smithi088.stderr:last_changed 2024-06-03T08:34:31.885125+0000 2024-06-03T09:06:36.867 INFO:teuthology.orchestra.run.smithi088.stderr:created 2024-06-03T08:34:31.885125+0000 2024-06-03T09:06:36.867 INFO:teuthology.orchestra.run.smithi088.stderr:min_mon_release 18 (reef) 2024-06-03T09:06:36.867 INFO:teuthology.orchestra.run.smithi088.stderr:election_strategy: 3 2024-06-03T09:06:36.867 INFO:teuthology.orchestra.run.smithi088.stderr:0: v1:172.21.15.88:6789/0 mon.a 2024-06-03T09:06:36.867 INFO:teuthology.orchestra.run.smithi088.stderr:1: v1:172.21.15.192:6789/0 mon.b 2024-06-03T09:06:36.867 INFO:teuthology.orchestra.run.smithi088.stderr:2: v1:172.21.15.88:6790/0 mon.c 2024-06-03T09:06:36.868 INFO:teuthology.orchestra.run.smithi088.stderr: 2024-06-03T09:06:36.868 INFO:teuthology.orchestra.run.smithi088.stderr:2024-06-03T09:06:36.862+0000 7f74d5ffb640 1 -- 172.21.15.88:0/2134435251 <== mon.1 v1:172.21.15.192:6789/0 4 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 580+0+0 (unknown 4100396887 0 0) 0x7f74e4132300 con 0x7f74e40adb90 2024-06-03T09:06:36.868 INFO:teuthology.orchestra.run.smithi088.stderr:2024-06-03T09:06:36.862+0000 7f74d5ffb640 1 -- 172.21.15.88:0/2134435251 >> v1:172.21.15.88:6790/0 conn(0x7f74e406fc80 legacy=0x7f74e412f020 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-06-03T09:06:36.868 INFO:teuthology.orchestra.run.smithi088.stderr:2024-06-03T09:06:36.862+0000 7f74d5ffb640 1 -- 172.21.15.88:0/2134435251 >> v1:172.21.15.88:6789/0 conn(0x7f74e406f020 legacy=0x7f74e41d9b20 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-06-03T09:06:36.868 INFO:teuthology.orchestra.run.smithi088.stderr:2024-06-03T09:06:36.862+0000 7f74d5ffb640 10 monclient: _finish_hunting 0 2024-06-03T09:06:36.868 INFO:teuthology.orchestra.run.smithi088.stderr:2024-06-03T09:06:36.862+0000 7f74d5ffb640 1 monclient: found mon.b 2024-06-03T09:06:36.868 INFO:teuthology.orchestra.run.smithi088.stderr:2024-06-03T09:06:36.862+0000 7f74d5ffb640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-06-03T09:06:36.868 INFO:teuthology.orchestra.run.smithi088.stderr:2024-06-03T09:06:36.862+0000 7f74d5ffb640 10 monclient: _send_mon_message to mon.b at v1:172.21.15.192:6789/0 2024-06-03T09:06:36.868 INFO:teuthology.orchestra.run.smithi088.stderr:2024-06-03T09:06:36.862+0000 7f74d5ffb640 1 -- 172.21.15.88:0/2134435251 --> v1:172.21.15.192:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f74e4132490 con 0x7f74e40adb90 2024-06-03T09:06:36.868 INFO:teuthology.orchestra.run.smithi088.stderr:2024-06-03T09:06:36.862+0000 7f74d5ffb640 10 monclient: _finish_auth 0 2024-06-03T09:06:36.868 INFO:teuthology.orchestra.run.smithi088.stderr:2024-06-03T09:06:36.862+0000 7f74d5ffb640 10 monclient: _check_auth_tickets 2024-06-03T09:06:36.868 INFO:teuthology.orchestra.run.smithi088.stderr:2024-06-03T09:06:36.862+0000 7f74d5ffb640 20 monclient: _check_auth_rotating not needed by client.admin 2024-06-03T09:06:36.868 INFO:teuthology.orchestra.run.smithi088.stderr:2024-06-03T09:06:36.862+0000 7f74eb2a1640 5 monclient: authenticate success, global_id 30232 2024-06-03T09:06:36.869 INFO:teuthology.orchestra.run.smithi088.stderr:2024-06-03T09:06:36.862+0000 7f74eb2a1640 10 monclient: _renew_subs 2024-06-03T09:06:36.869 INFO:teuthology.orchestra.run.smithi088.stderr:2024-06-03T09:06:36.862+0000 7f74eb2a1640 10 monclient: _send_mon_message to mon.b at v1:172.21.15.192:6789/0 2024-06-03T09:06:36.869 INFO:teuthology.orchestra.run.smithi088.stderr:2024-06-03T09:06:36.862+0000 7f74eb2a1640 1 -- 172.21.15.88:0/2134435251 --> v1:172.21.15.192:6789/0 -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f74e4132dc0 con 0x7f74e40adb90 2024-06-03T09:06:36.869 INFO:teuthology.orchestra.run.smithi088.stderr:2024-06-03T09:06:36.862+0000 7f74eb2a1640 10 client.30232.objecter _maybe_request_map subscribing (onetime) to next osd map 2024-06-03T09:06:36.869 INFO:teuthology.orchestra.run.smithi088.stderr:2024-06-03T09:06:36.862+0000 7f74eb2a1640 10 monclient: _renew_subs 2024-06-03T09:06:36.869 INFO:teuthology.orchestra.run.smithi088.stderr:2024-06-03T09:06:36.862+0000 7f74eb2a1640 10 monclient: _send_mon_message to mon.b at v1:172.21.15.192:6789/0 2024-06-03T09:06:36.869 INFO:teuthology.orchestra.run.smithi088.stderr:2024-06-03T09:06:36.862+0000 7f74eb2a1640 1 -- 172.21.15.88:0/2134435251 --> v1:172.21.15.192:6789/0 -- mon_subscribe({osdmap=0}) v3 -- 0x7f74e40accb0 con 0x7f74e40adb90 2024-06-03T09:06:36.869 INFO:teuthology.orchestra.run.smithi088.stderr:2024-06-03T09:06:36.862+0000 7f74eb2a1640 1 librados: init done 2024-06-03T09:06:36.870 INFO:teuthology.orchestra.run.smithi088.stderr:2024-06-03T09:06:36.863+0000 7f74eb2a1640 10 start_mon_command cmd=[{"prefix": "get_command_descriptions"}] 2024-06-03T09:06:36.870 INFO:teuthology.orchestra.run.smithi088.stderr:2024-06-03T09:06:36.863+0000 7f74eb2a1640 10 monclient: _send_command 1 [{"prefix": "get_command_descriptions"}] 2024-06-03T09:06:36.870 INFO:teuthology.orchestra.run.smithi088.stderr:2024-06-03T09:06:36.863+0000 7f74eb2a1640 10 monclient: _send_mon_message to mon.b at v1:172.21.15.192:6789/0 2024-06-03T09:06:36.870 INFO:teuthology.orchestra.run.smithi088.stderr:2024-06-03T09:06:36.863+0000 7f74eb2a1640 1 -- 172.21.15.88:0/2134435251 --> v1:172.21.15.192:6789/0 -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f74e41333f0 con 0x7f74e40adb90 2024-06-03T09:06:37.124 INFO:teuthology.orchestra.run.smithi088.stderr:2024-06-03T09:06:37.124+0000 7f74d5ffb640 1 -- 172.21.15.88:0/2134435251 <== mon.1 v1:172.21.15.192:6789/0 5 ==== config(0 keys) v1 ==== 4+0+0 (unknown 0 0 0) 0x7f74dc003990 con 0x7f74e40adb90 2024-06-03T09:06:37.124 INFO:teuthology.orchestra.run.smithi088.stderr:2024-06-03T09:06:37.124+0000 7f74d5ffb640 10 monclient: handle_config config(0 keys) v1 2024-06-03T09:06:37.124 INFO:teuthology.orchestra.run.smithi088.stderr:2024-06-03T09:06:37.124+0000 7f74d5ffb640 1 -- 172.21.15.88:0/2134435251 <== mon.1 v1:172.21.15.192:6789/0 6 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 2234374065 0 0) 0x7f74dc003ca0 con 0x7f74e40adb90 2024-06-03T09:06:37.125 INFO:teuthology.orchestra.run.smithi088.stderr:2024-06-03T09:06:37.124+0000 7f74d5ffb640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-06-03T09:06:37.125 INFO:teuthology.orchestra.run.smithi088.stderr:2024-06-03T09:06:37.124+0000 7f74d5ffb640 10 monclient: got monmap 1 from mon.b (according to old e1) 2024-06-03T09:06:37.125 INFO:teuthology.orchestra.run.smithi088.stderr:2024-06-03T09:06:37.124+0000 7f74d5ffb640 10 monclient: dump: 2024-06-03T09:06:37.125 INFO:teuthology.orchestra.run.smithi088.stderr:epoch 1 2024-06-03T09:06:37.125 INFO:teuthology.orchestra.run.smithi088.stderr:fsid ec3c3fdd-3ac6-4d7d-9412-683130ed8b04 2024-06-03T09:06:37.125 INFO:teuthology.orchestra.run.smithi088.stderr:last_changed 2024-06-03T08:34:31.885125+0000 2024-06-03T09:06:37.125 INFO:teuthology.orchestra.run.smithi088.stderr:created 2024-06-03T08:34:31.885125+0000 2024-06-03T09:06:37.125 INFO:teuthology.orchestra.run.smithi088.stderr:min_mon_release 18 (reef) 2024-06-03T09:06:37.125 INFO:teuthology.orchestra.run.smithi088.stderr:election_strategy: 3 2024-06-03T09:06:37.125 INFO:teuthology.orchestra.run.smithi088.stderr:0: v1:172.21.15.88:6789/0 mon.a 2024-06-03T09:06:37.125 INFO:teuthology.orchestra.run.smithi088.stderr:1: v1:172.21.15.192:6789/0 mon.b 2024-06-03T09:06:37.126 INFO:teuthology.orchestra.run.smithi088.stderr:2: v1:172.21.15.88:6790/0 mon.c 2024-06-03T09:06:37.126 INFO:teuthology.orchestra.run.smithi088.stderr: 2024-06-03T09:06:37.126 INFO:teuthology.orchestra.run.smithi088.stderr:2024-06-03T09:06:37.124+0000 7f74d77fe640 4 set_mon_vals no callback set 2024-06-03T09:06:37.126 INFO:teuthology.orchestra.run.smithi088.stderr:2024-06-03T09:06:37.125+0000 7f74d5ffb640 1 -- 172.21.15.88:0/2134435251 <== mon.1 v1:172.21.15.192:6789/0 7 ==== mgrmap(e 5) v1 ==== 89110+0+0 (unknown 873533973 0 0) 0x7f74dc019a20 con 0x7f74e40adb90 2024-06-03T09:06:37.126 INFO:teuthology.orchestra.run.smithi088.stderr:2024-06-03T09:06:37.125+0000 7f74e903d640 10 monclient: get_auth_request con 0x7f74c806d340 auth_method 0 2024-06-03T09:06:37.127 INFO:teuthology.orchestra.run.smithi088.stderr:2024-06-03T09:06:37.125+0000 7f74d5ffb640 10 client.30232.objecter ms_handle_connect 0x7f74c806d340 2024-06-03T09:06:37.130 INFO:teuthology.orchestra.run.smithi088.stderr:2024-06-03T09:06:37.130+0000 7f74d5ffb640 1 -- 172.21.15.88:0/2134435251 <== mon.1 v1:172.21.15.192:6789/0 8 ==== osd_map(1010..1010 src has 622..1010) v4 ==== 5229+0+0 (unknown 1105883993 0 0) 0x7f74dc0531b0 con 0x7f74e40adb90 2024-06-03T09:06:37.130 INFO:teuthology.orchestra.run.smithi088.stderr:2024-06-03T09:06:37.130+0000 7f74d5ffb640 10 client.30232.objecter ms_dispatch 0x7f74e4000b90 osd_map(1010..1010 src has 622..1010) v4 2024-06-03T09:06:37.131 INFO:teuthology.orchestra.run.smithi088.stderr:2024-06-03T09:06:37.130+0000 7f74d5ffb640 3 client.30232.objecter handle_osd_map got epochs [1010,1010] > 0 2024-06-03T09:06:37.131 INFO:teuthology.orchestra.run.smithi088.stderr:2024-06-03T09:06:37.130+0000 7f74d5ffb640 3 client.30232.objecter handle_osd_map decoding full epoch 1010 2024-06-03T09:06:37.131 INFO:teuthology.orchestra.run.smithi088.stderr:2024-06-03T09:06:37.130+0000 7f74d5ffb640 20 client.30232.objecter dump_active .. 0 homeless 2024-06-03T09:06:37.132 INFO:teuthology.orchestra.run.smithi088.stderr:2024-06-03T09:06:37.132+0000 7f74d5ffb640 1 -- 172.21.15.88:0/2134435251 <== mon.1 v1:172.21.15.192:6789/0 9 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+182498 (unknown 1092875540 0 3016760700) 0x7f74dc04eb90 con 0x7f74e40adb90 2024-06-03T09:06:37.132 INFO:teuthology.orchestra.run.smithi088.stderr:2024-06-03T09:06:37.132+0000 7f74d5ffb640 10 monclient: handle_mon_command_ack 1 [{"prefix": "get_command_descriptions"}] 2024-06-03T09:06:37.132 INFO:teuthology.orchestra.run.smithi088.stderr:2024-06-03T09:06:37.132+0000 7f74d5ffb640 10 monclient: _finish_command 1 = system:0 2024-06-03T09:06:37.319 INFO:teuthology.orchestra.run.smithi088.stderr:2024-06-03T09:06:37.319+0000 7f74eb2a1640 1 -- 172.21.15.88:0/2134435251 --> v1:172.21.15.192:6800/34229 -- mgr_command(tid 0: {"prefix": "pg dump", "target": ["mon-mgr", ""], "format": "json"}) v1 -- 0x7f74e40adf70 con 0x7f74c806d340 2024-06-03T09:06:37.328 INFO:teuthology.orchestra.run.smithi088.stderr:2024-06-03T09:06:37.328+0000 7f74d5ffb640 1 -- 172.21.15.88:0/2134435251 <== mgr.4110 v1:172.21.15.192:6800/34229 1 ==== mgr_command_reply(tid 0: 0 dumped all) v1 ==== 18+0+151434 (unknown 2965378022 0 729038708) 0x7f74e00077a0 con 0x7f74c806d340 2024-06-03T09:06:37.330 INFO:teuthology.orchestra.run.smithi088.stdout: 2024-06-03T09:06:37.330 INFO:teuthology.orchestra.run.smithi088.stderr:dumped all 2024-06-03T09:06:37.332 INFO:teuthology.orchestra.run.smithi088.stderr:2024-06-03T09:06:37.330+0000 7f74eb2a1640 10 librados: watch_flush enter 2024-06-03T09:06:37.332 INFO:teuthology.orchestra.run.smithi088.stderr:2024-06-03T09:06:37.330+0000 7f74eb2a1640 10 librados: watch_flush exit 2024-06-03T09:06:37.332 INFO:teuthology.orchestra.run.smithi088.stderr:2024-06-03T09:06:37.330+0000 7f74eb2a1640 20 client.30232.objecter shutdown clearing up homeless session... 2024-06-03T09:06:37.332 INFO:teuthology.orchestra.run.smithi088.stderr:2024-06-03T09:06:37.330+0000 7f74eb2a1640 10 client.30232.objecter successfully canceled tick 2024-06-03T09:06:37.332 INFO:teuthology.orchestra.run.smithi088.stderr:2024-06-03T09:06:37.331+0000 7f74eb2a1640 1 -- 172.21.15.88:0/2134435251 >> v1:172.21.15.192:6800/34229 conn(0x7f74c806d340 legacy=0x7f74c806f800 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-06-03T09:06:37.332 INFO:teuthology.orchestra.run.smithi088.stderr:2024-06-03T09:06:37.331+0000 7f74eb2a1640 10 monclient: shutdown 2024-06-03T09:06:37.333 INFO:teuthology.orchestra.run.smithi088.stderr:2024-06-03T09:06:37.331+0000 7f74eb2a1640 20 monclient: shutdown discarding 0 pending message(s) 2024-06-03T09:06:37.333 INFO:teuthology.orchestra.run.smithi088.stderr:2024-06-03T09:06:37.331+0000 7f74eb2a1640 1 -- 172.21.15.88:0/2134435251 >> v1:172.21.15.192:6789/0 conn(0x7f74e40adb90 legacy=0x7f74e41e1ec0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-06-03T09:06:37.333 INFO:teuthology.orchestra.run.smithi088.stderr:2024-06-03T09:06:37.331+0000 7f74eb2a1640 1 -- 172.21.15.88:0/2134435251 shutdown_connections 2024-06-03T09:06:37.333 INFO:teuthology.orchestra.run.smithi088.stderr:2024-06-03T09:06:37.331+0000 7f74eb2a1640 1 -- 172.21.15.88:0/2134435251 >> 172.21.15.88:0/2134435251 conn(0x7f74e40667e0 msgr2=0x7f74e406e670 unknown :-1 s=STATE_NONE l=0).mark_down 2024-06-03T09:06:37.333 INFO:teuthology.orchestra.run.smithi088.stderr:2024-06-03T09:06:37.331+0000 7f74eb2a1640 1 -- 172.21.15.88:0/2134435251 shutdown_connections 2024-06-03T09:06:37.333 INFO:teuthology.orchestra.run.smithi088.stderr:2024-06-03T09:06:37.331+0000 7f74eb2a1640 1 -- 172.21.15.88:0/2134435251 wait complete. 2024-06-03T09:06:37.333 INFO:teuthology.orchestra.run.smithi088.stderr:2024-06-03T09:06:37.331+0000 7f74eb2a1640 1 librados: shutdown 2024-06-03T09:06:37.341 INFO:teuthology.orchestra.run.smithi088.stdout:{"pg_ready":true,"pg_map":{"version":1924,"stamp":"2024-06-03T09:06:37.101775+0000","last_osdmap_epoch":0,"last_pg_scan":0,"pg_stats_sum":{"stat_sum":{"num_bytes":459285,"num_objects":3,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":1,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":3,"num_whiteouts":0,"num_read":102,"num_read_kb":88,"num_write":120,"num_write_kb":1379,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2,"num_bytes_recovered":397840,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":83,"ondisk_log_size":83,"up":68,"acting":68,"num_store_stats":0},"osd_stats_sum":{"up_from":0,"seq":0,"num_pgs":100,"num_osds":8,"num_per_pool_osds":8,"num_per_pool_omap_osds":8,"kb":754974720,"kb_used":494548,"kb_used_data":46324,"kb_used_omap":12,"kb_used_meta":448115,"kb_avail":754480172,"statfs":{"total":773094113280,"available":772587696128,"internally_reserved":0,"allocated":47435776,"data_stored":22747954,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":12744,"internal_metadata":458870328},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":109,"apply_latency_ms":109,"commit_latency_ns":109000000,"apply_latency_ns":109000000},"alerts":[],"network_ping_times":[]},"pg_stats_delta":{"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":0,"acting":0,"num_store_stats":0,"stamp_delta":"12.002325"},"pg_stats":[{"pgid":"17.d","version":"0'0","reported_seq":942,"reported_epoch":1009,"state":"active+clean","last_fresh":"2024-06-03T09:06:14.443917+0000","last_change":"2024-06-03T08:38:19.260548+0000","last_active":"2024-06-03T09:06:14.443917+0000","last_peered":"2024-06-03T09:06:14.443917+0000","last_clean":"2024-06-03T09:06:14.443917+0000","last_became_active":"2024-06-03T08:38:19.253559+0000","last_became_peered":"2024-06-03T08:38:19.253559+0000","last_unstale":"2024-06-03T09:06:14.443917+0000","last_undegraded":"2024-06-03T09:06:14.443917+0000","last_fullsized":"2024-06-03T09:06:14.443917+0000","mapping_epoch":28,"log_start":"0'0","ondisk_log_start":"0'0","created":28,"last_epoch_clean":45,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-06-03T08:37:55.614497+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-06-03T08:37:55.614497+0000","last_clean_scrub_stamp":"2024-06-03T08:37:55.614497+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-06-04T19:00:12.470979+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1],"acting":[0,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"17.c","version":"0'0","reported_seq":953,"reported_epoch":1009,"state":"active+undersized","last_fresh":"2024-06-03T09:06:13.668156+0000","last_change":"2024-06-03T09:06:03.440074+0000","last_active":"2024-06-03T09:06:13.668156+0000","last_peered":"2024-06-03T09:06:13.668156+0000","last_clean":"2024-06-03T09:05:02.105638+0000","last_became_active":"2024-06-03T09:06:03.440074+0000","last_became_peered":"2024-06-03T09:06:03.440074+0000","last_unstale":"2024-06-03T09:06:13.668156+0000","last_undegraded":"2024-06-03T09:06:13.668156+0000","last_fullsized":"2024-06-03T09:06:03.377982+0000","mapping_epoch":999,"log_start":"0'0","ondisk_log_start":"0'0","created":28,"last_epoch_clean":255,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-06-03T08:37:55.614497+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-06-03T08:37:55.614497+0000","last_clean_scrub_stamp":"2024-06-03T08:37:55.614497+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-06-04T10:34:20.808557+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3],"acting":[3],"avail_no_missing":["3"],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"17.f","version":"0'0","reported_seq":148,"reported_epoch":1009,"state":"active+undersized","last_fresh":"2024-06-03T09:06:14.450543+0000","last_change":"2024-06-03T09:06:03.420285+0000","last_active":"2024-06-03T09:06:14.450543+0000","last_peered":"2024-06-03T09:06:14.450543+0000","last_clean":"2024-06-03T08:37:55.614497+0000","last_became_active":"2024-06-03T09:06:03.420285+0000","last_became_peered":"2024-06-03T09:06:03.420285+0000","last_unstale":"2024-06-03T09:06:14.450543+0000","last_undegraded":"2024-06-03T09:06:14.450543+0000","last_fullsized":"2024-06-03T09:06:03.380616+0000","mapping_epoch":999,"log_start":"0'0","ondisk_log_start":"0'0","created":28,"last_epoch_clean":255,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-06-03T08:37:55.614497+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-06-03T08:37:55.614497+0000","last_clean_scrub_stamp":"2024-06-03T08:37:55.614497+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-06-04T15:39:41.187731+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4],"acting":[4],"avail_no_missing":["4"],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"17.e","version":"0'0","reported_seq":149,"reported_epoch":1009,"state":"active+undersized","last_fresh":"2024-06-03T09:06:13.726992+0000","last_change":"2024-06-03T09:06:03.412887+0000","last_active":"2024-06-03T09:06:13.726992+0000","last_peered":"2024-06-03T09:06:13.726992+0000","last_clean":"2024-06-03T08:37:55.614497+0000","last_became_active":"2024-06-03T09:06:03.412887+0000","last_became_peered":"2024-06-03T09:06:03.412887+0000","last_unstale":"2024-06-03T09:06:13.726992+0000","last_undegraded":"2024-06-03T09:06:13.726992+0000","last_fullsized":"2024-06-03T09:06:03.332482+0000","mapping_epoch":999,"log_start":"0'0","ondisk_log_start":"0'0","created":28,"last_epoch_clean":255,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-06-03T08:37:55.614497+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-06-03T08:37:55.614497+0000","last_clean_scrub_stamp":"2024-06-03T08:37:55.614497+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-06-04T12:23:22.511601+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5],"acting":[5],"avail_no_missing":["5"],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"17.9","version":"0'0","reported_seq":946,"reported_epoch":1009,"state":"active+clean","last_fresh":"2024-06-03T09:06:14.999299+0000","last_change":"2024-06-03T08:38:04.772260+0000","last_active":"2024-06-03T09:06:14.999299+0000","last_peered":"2024-06-03T09:06:14.999299+0000","last_clean":"2024-06-03T09:06:14.999299+0000","last_became_active":"2024-06-03T08:38:04.763428+0000","last_became_peered":"2024-06-03T08:38:04.763428+0000","last_unstale":"2024-06-03T09:06:14.999299+0000","last_undegraded":"2024-06-03T09:06:14.999299+0000","last_fullsized":"2024-06-03T09:06:14.999299+0000","mapping_epoch":28,"log_start":"0'0","ondisk_log_start":"0'0","created":28,"last_epoch_clean":33,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-06-03T08:37:55.614497+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-06-03T08:37:55.614497+0000","last_clean_scrub_stamp":"2024-06-03T08:37:55.614497+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-06-04T10:55:10.773936+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,4],"acting":[7,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"17.8","version":"0'0","reported_seq":961,"reported_epoch":1009,"state":"active+undersized","last_fresh":"2024-06-03T09:06:14.430375+0000","last_change":"2024-06-03T09:06:03.388786+0000","last_active":"2024-06-03T09:06:14.430375+0000","last_peered":"2024-06-03T09:06:14.430375+0000","last_clean":"2024-06-03T09:04:43.297780+0000","last_became_active":"2024-06-03T09:06:03.388786+0000","last_became_peered":"2024-06-03T09:06:03.388786+0000","last_unstale":"2024-06-03T09:06:14.430375+0000","last_undegraded":"2024-06-03T09:06:14.430375+0000","last_fullsized":"2024-06-03T09:06:03.331754+0000","mapping_epoch":999,"log_start":"0'0","ondisk_log_start":"0'0","created":28,"last_epoch_clean":255,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-06-03T08:37:55.614497+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-06-03T08:37:55.614497+0000","last_clean_scrub_stamp":"2024-06-03T08:37:55.614497+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-06-04T09:18:44.765547+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0],"acting":[0],"avail_no_missing":["0"],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"17.b","version":"0'0","reported_seq":149,"reported_epoch":1009,"state":"active+undersized","last_fresh":"2024-06-03T09:06:14.433735+0000","last_change":"2024-06-03T09:06:03.451203+0000","last_active":"2024-06-03T09:06:14.433735+0000","last_peered":"2024-06-03T09:06:14.433735+0000","last_clean":"2024-06-03T08:37:55.614497+0000","last_became_active":"2024-06-03T09:06:03.451203+0000","last_became_peered":"2024-06-03T09:06:03.451203+0000","last_unstale":"2024-06-03T09:06:14.433735+0000","last_undegraded":"2024-06-03T09:06:14.433735+0000","last_fullsized":"2024-06-03T09:06:03.375523+0000","mapping_epoch":999,"log_start":"0'0","ondisk_log_start":"0'0","created":28,"last_epoch_clean":255,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-06-03T08:37:55.614497+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-06-03T08:37:55.614497+0000","last_clean_scrub_stamp":"2024-06-03T08:37:55.614497+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-06-04T10:16:19.813495+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2],"acting":[2],"avail_no_missing":["2"],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"17.a","version":"0'0","reported_seq":925,"reported_epoch":1010,"state":"active+clean","last_fresh":"2024-06-03T09:06:18.651697+0000","last_change":"2024-06-03T08:38:05.843142+0000","last_active":"2024-06-03T09:06:18.651697+0000","last_peered":"2024-06-03T09:06:18.651697+0000","last_clean":"2024-06-03T09:06:18.651697+0000","last_became_active":"2024-06-03T08:38:05.809257+0000","last_became_peered":"2024-06-03T08:38:05.809257+0000","last_unstale":"2024-06-03T09:06:18.651697+0000","last_undegraded":"2024-06-03T09:06:18.651697+0000","last_fullsized":"2024-06-03T09:06:18.651697+0000","mapping_epoch":28,"log_start":"0'0","ondisk_log_start":"0'0","created":28,"last_epoch_clean":33,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-06-03T08:37:55.614497+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-06-03T08:37:55.614497+0000","last_clean_scrub_stamp":"2024-06-03T08:37:55.614497+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-06-04T12:12:03.686838+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2],"acting":[1,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"17.5","version":"0'0","reported_seq":943,"reported_epoch":1009,"state":"active+clean","last_fresh":"2024-06-03T09:06:14.442357+0000","last_change":"2024-06-03T08:38:23.466051+0000","last_active":"2024-06-03T09:06:14.442357+0000","last_peered":"2024-06-03T09:06:14.442357+0000","last_clean":"2024-06-03T09:06:14.442357+0000","last_became_active":"2024-06-03T08:38:23.458039+0000","last_became_peered":"2024-06-03T08:38:23.458039+0000","last_unstale":"2024-06-03T09:06:14.442357+0000","last_undegraded":"2024-06-03T09:06:14.442357+0000","last_fullsized":"2024-06-03T09:06:14.442357+0000","mapping_epoch":28,"log_start":"0'0","ondisk_log_start":"0'0","created":28,"last_epoch_clean":49,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-06-03T08:37:55.614497+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-06-03T08:37:55.614497+0000","last_clean_scrub_stamp":"2024-06-03T08:37:55.614497+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-06-04T12:27:02.532436+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,4],"acting":[0,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"17.4","version":"0'0","reported_seq":149,"reported_epoch":1009,"state":"active+undersized","last_fresh":"2024-06-03T09:06:13.941522+0000","last_change":"2024-06-03T09:06:03.441889+0000","last_active":"2024-06-03T09:06:13.941522+0000","last_peered":"2024-06-03T09:06:13.941522+0000","last_clean":"2024-06-03T08:37:55.614497+0000","last_became_active":"2024-06-03T09:06:03.441889+0000","last_became_peered":"2024-06-03T09:06:03.441889+0000","last_unstale":"2024-06-03T09:06:13.941522+0000","last_undegraded":"2024-06-03T09:06:13.941522+0000","last_fullsized":"2024-06-03T09:06:03.359582+0000","mapping_epoch":999,"log_start":"0'0","ondisk_log_start":"0'0","created":28,"last_epoch_clean":255,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-06-03T08:37:55.614497+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-06-03T08:37:55.614497+0000","last_clean_scrub_stamp":"2024-06-03T08:37:55.614497+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-06-04T12:49:22.907141+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7],"acting":[7],"avail_no_missing":["7"],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"17.7","version":"0'0","reported_seq":929,"reported_epoch":1009,"state":"active+clean","last_fresh":"2024-06-03T09:06:13.736920+0000","last_change":"2024-06-03T08:38:04.837083+0000","last_active":"2024-06-03T09:06:13.736920+0000","last_peered":"2024-06-03T09:06:13.736920+0000","last_clean":"2024-06-03T09:06:13.736920+0000","last_became_active":"2024-06-03T08:38:04.818246+0000","last_became_peered":"2024-06-03T08:38:04.818246+0000","last_unstale":"2024-06-03T09:06:13.736920+0000","last_undegraded":"2024-06-03T09:06:13.736920+0000","last_fullsized":"2024-06-03T09:06:13.736920+0000","mapping_epoch":28,"log_start":"0'0","ondisk_log_start":"0'0","created":28,"last_epoch_clean":34,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-06-03T08:37:55.614497+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-06-03T08:37:55.614497+0000","last_clean_scrub_stamp":"2024-06-03T08:37:55.614497+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-06-04T17:02:38.204813+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,2],"acting":[5,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"17.6","version":"0'0","reported_seq":960,"reported_epoch":1009,"state":"active+undersized","last_fresh":"2024-06-03T09:06:14.453110+0000","last_change":"2024-06-03T09:06:03.428677+0000","last_active":"2024-06-03T09:06:14.453110+0000","last_peered":"2024-06-03T09:06:14.453110+0000","last_clean":"2024-06-03T09:04:43.312352+0000","last_became_active":"2024-06-03T09:06:03.428677+0000","last_became_peered":"2024-06-03T09:06:03.428677+0000","last_unstale":"2024-06-03T09:06:14.453110+0000","last_undegraded":"2024-06-03T09:06:14.453110+0000","last_fullsized":"2024-06-03T09:06:03.359909+0000","mapping_epoch":999,"log_start":"0'0","ondisk_log_start":"0'0","created":28,"last_epoch_clean":255,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-06-03T08:37:55.614497+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-06-03T08:37:55.614497+0000","last_clean_scrub_stamp":"2024-06-03T08:37:55.614497+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-06-04T15:38:02.180241+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0],"acting":[0],"avail_no_missing":["0"],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"2.7","version":"0'0","reported_seq":940,"reported_epoch":1009,"state":"active+clean","last_fresh":"2024-06-03T09:06:13.684065+0000","last_change":"2024-06-03T08:36:30.385405+0000","last_active":"2024-06-03T09:06:13.684065+0000","last_peered":"2024-06-03T09:06:13.684065+0000","last_clean":"2024-06-03T09:06:13.684065+0000","last_became_active":"2024-06-03T08:36:30.380302+0000","last_became_peered":"2024-06-03T08:36:30.380302+0000","last_unstale":"2024-06-03T09:06:13.684065+0000","last_undegraded":"2024-06-03T09:06:13.684065+0000","last_fullsized":"2024-06-03T09:06:13.684065+0000","mapping_epoch":23,"log_start":"0'0","ondisk_log_start":"0'0","created":23,"last_epoch_clean":24,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-06-03T08:36:28.444764+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-06-03T08:36:28.444764+0000","last_clean_scrub_stamp":"2024-06-03T08:36:28.444764+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-06-04T18:46:33.475181+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0],"acting":[3,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"17.14","version":"0'0","reported_seq":926,"reported_epoch":1009,"state":"active+clean","last_fresh":"2024-06-03T09:06:13.721460+0000","last_change":"2024-06-03T08:38:04.547565+0000","last_active":"2024-06-03T09:06:13.721460+0000","last_peered":"2024-06-03T09:06:13.721460+0000","last_clean":"2024-06-03T09:06:13.721460+0000","last_became_active":"2024-06-03T08:38:04.541781+0000","last_became_peered":"2024-06-03T08:38:04.541781+0000","last_unstale":"2024-06-03T09:06:13.721460+0000","last_undegraded":"2024-06-03T09:06:13.721460+0000","last_fullsized":"2024-06-03T09:06:13.721460+0000","mapping_epoch":28,"log_start":"0'0","ondisk_log_start":"0'0","created":28,"last_epoch_clean":34,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-06-03T08:37:55.614497+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-06-03T08:37:55.614497+0000","last_clean_scrub_stamp":"2024-06-03T08:37:55.614497+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-06-04T16:41:55.170134+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,2],"acting":[5,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"2.6","version":"0'0","reported_seq":949,"reported_epoch":1009,"state":"active+clean","last_fresh":"2024-06-03T09:06:14.448100+0000","last_change":"2024-06-03T08:36:30.430760+0000","last_active":"2024-06-03T09:06:14.448100+0000","last_peered":"2024-06-03T09:06:14.448100+0000","last_clean":"2024-06-03T09:06:14.448100+0000","last_became_active":"2024-06-03T08:36:30.404665+0000","last_became_peered":"2024-06-03T08:36:30.404665+0000","last_unstale":"2024-06-03T09:06:14.448100+0000","last_undegraded":"2024-06-03T09:06:14.448100+0000","last_fullsized":"2024-06-03T09:06:14.448100+0000","mapping_epoch":23,"log_start":"0'0","ondisk_log_start":"0'0","created":23,"last_epoch_clean":24,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-06-03T08:36:28.444764+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-06-03T08:36:28.444764+0000","last_clean_scrub_stamp":"2024-06-03T08:36:28.444764+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-06-04T10:37:37.746074+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,0],"acting":[4,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"17.15","version":"0'0","reported_seq":934,"reported_epoch":1009,"state":"active+clean","last_fresh":"2024-06-03T09:06:13.672940+0000","last_change":"2024-06-03T08:38:20.999903+0000","last_active":"2024-06-03T09:06:13.672940+0000","last_peered":"2024-06-03T09:06:13.672940+0000","last_clean":"2024-06-03T09:06:13.672940+0000","last_became_active":"2024-06-03T08:38:20.979523+0000","last_became_peered":"2024-06-03T08:38:20.979523+0000","last_unstale":"2024-06-03T09:06:13.672940+0000","last_undegraded":"2024-06-03T09:06:13.672940+0000","last_fullsized":"2024-06-03T09:06:13.672940+0000","mapping_epoch":28,"log_start":"0'0","ondisk_log_start":"0'0","created":28,"last_epoch_clean":47,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-06-03T08:37:55.614497+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-06-03T08:37:55.614497+0000","last_clean_scrub_stamp":"2024-06-03T08:37:55.614497+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-06-04T20:29:58.314289+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,1],"acting":[3,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"2.5","version":"0'0","reported_seq":942,"reported_epoch":1010,"state":"active+clean","last_fresh":"2024-06-03T09:06:18.382767+0000","last_change":"2024-06-03T08:36:30.024235+0000","last_active":"2024-06-03T09:06:18.382767+0000","last_peered":"2024-06-03T09:06:18.382767+0000","last_clean":"2024-06-03T09:06:18.382767+0000","last_became_active":"2024-06-03T08:36:30.019040+0000","last_became_peered":"2024-06-03T08:36:30.019040+0000","last_unstale":"2024-06-03T09:06:18.382767+0000","last_undegraded":"2024-06-03T09:06:18.382767+0000","last_fullsized":"2024-06-03T09:06:18.382767+0000","mapping_epoch":23,"log_start":"0'0","ondisk_log_start":"0'0","created":23,"last_epoch_clean":24,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-06-03T08:36:28.444764+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-06-03T08:36:28.444764+0000","last_clean_scrub_stamp":"2024-06-03T08:36:28.444764+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-06-04T15:46:47.109502+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,4],"acting":[3,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"17.16","version":"0'0","reported_seq":972,"reported_epoch":1009,"state":"active+undersized","last_fresh":"2024-06-03T09:06:14.441665+0000","last_change":"2024-06-03T09:06:03.478615+0000","last_active":"2024-06-03T09:06:14.441665+0000","last_peered":"2024-06-03T09:06:14.441665+0000","last_clean":"2024-06-03T09:04:43.328415+0000","last_became_active":"2024-06-03T09:06:03.478615+0000","last_became_peered":"2024-06-03T09:06:03.478615+0000","last_unstale":"2024-06-03T09:06:14.441665+0000","last_undegraded":"2024-06-03T09:06:14.441665+0000","last_fullsized":"2024-06-03T09:06:03.402504+0000","mapping_epoch":999,"log_start":"0'0","ondisk_log_start":"0'0","created":28,"last_epoch_clean":255,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-06-03T08:37:55.614497+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-06-03T08:37:55.614497+0000","last_clean_scrub_stamp":"2024-06-03T08:37:55.614497+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-06-04T08:47:08.079233+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2],"acting":[2],"avail_no_missing":["2"],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"2.4","version":"0'0","reported_seq":952,"reported_epoch":1010,"state":"active+undersized","last_fresh":"2024-06-03T09:06:18.331485+0000","last_change":"2024-06-03T09:06:03.387171+0000","last_active":"2024-06-03T09:06:18.331485+0000","last_peered":"2024-06-03T09:06:18.331485+0000","last_clean":"2024-06-03T09:04:43.202645+0000","last_became_active":"2024-06-03T09:06:03.387171+0000","last_became_peered":"2024-06-03T09:06:03.387171+0000","last_unstale":"2024-06-03T09:06:18.331485+0000","last_undegraded":"2024-06-03T09:06:18.331485+0000","last_fullsized":"2024-06-03T09:06:03.283787+0000","mapping_epoch":999,"log_start":"0'0","ondisk_log_start":"0'0","created":23,"last_epoch_clean":255,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-06-03T08:36:28.444764+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-06-03T08:36:28.444764+0000","last_clean_scrub_stamp":"2024-06-03T08:36:28.444764+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-06-04T20:17:43.485023+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5],"acting":[5],"avail_no_missing":["5"],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"17.17","version":"0'0","reported_seq":954,"reported_epoch":1009,"state":"active+clean","last_fresh":"2024-06-03T09:06:14.447046+0000","last_change":"2024-06-03T08:38:38.483275+0000","last_active":"2024-06-03T09:06:14.447046+0000","last_peered":"2024-06-03T09:06:14.447046+0000","last_clean":"2024-06-03T09:06:14.447046+0000","last_became_active":"2024-06-03T08:38:38.467916+0000","last_became_peered":"2024-06-03T08:38:38.467916+0000","last_unstale":"2024-06-03T09:06:14.447046+0000","last_undegraded":"2024-06-03T09:06:14.447046+0000","last_fullsized":"2024-06-03T09:06:14.447046+0000","mapping_epoch":28,"log_start":"0'0","ondisk_log_start":"0'0","created":28,"last_epoch_clean":61,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-06-03T08:37:55.614497+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-06-03T08:37:55.614497+0000","last_clean_scrub_stamp":"2024-06-03T08:37:55.614497+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-06-04T10:02:23.036420+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,7],"acting":[2,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"2.2","version":"0'0","reported_seq":963,"reported_epoch":1009,"state":"active+undersized","last_fresh":"2024-06-03T09:06:13.657892+0000","last_change":"2024-06-03T09:06:03.410020+0000","last_active":"2024-06-03T09:06:13.657892+0000","last_peered":"2024-06-03T09:06:13.657892+0000","last_clean":"2024-06-03T09:05:37.145889+0000","last_became_active":"2024-06-03T09:06:03.410020+0000","last_became_peered":"2024-06-03T09:06:03.410020+0000","last_unstale":"2024-06-03T09:06:13.657892+0000","last_undegraded":"2024-06-03T09:06:13.657892+0000","last_fullsized":"2024-06-03T09:06:03.346864+0000","mapping_epoch":999,"log_start":"0'0","ondisk_log_start":"0'0","created":23,"last_epoch_clean":255,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-06-03T08:36:28.444764+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-06-03T08:36:28.444764+0000","last_clean_scrub_stamp":"2024-06-03T08:36:28.444764+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-06-04T09:57:12.423360+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3],"acting":[3],"avail_no_missing":["3"],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"17.11","version":"0'0","reported_seq":944,"reported_epoch":1009,"state":"active+clean","last_fresh":"2024-06-03T09:06:14.426670+0000","last_change":"2024-06-03T08:38:06.502302+0000","last_active":"2024-06-03T09:06:14.426670+0000","last_peered":"2024-06-03T09:06:14.426670+0000","last_clean":"2024-06-03T09:06:14.426670+0000","last_became_active":"2024-06-03T08:38:06.490204+0000","last_became_peered":"2024-06-03T08:38:06.490204+0000","last_unstale":"2024-06-03T09:06:14.426670+0000","last_undegraded":"2024-06-03T09:06:14.426670+0000","last_fullsized":"2024-06-03T09:06:14.426670+0000","mapping_epoch":28,"log_start":"0'0","ondisk_log_start":"0'0","created":28,"last_epoch_clean":32,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-06-03T08:37:55.614497+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-06-03T08:37:55.614497+0000","last_clean_scrub_stamp":"2024-06-03T08:37:55.614497+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-06-04T10:47:05.988958+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,2],"acting":[4,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"2.1","version":"0'0","reported_seq":952,"reported_epoch":1009,"state":"active+clean","last_fresh":"2024-06-03T09:06:13.929725+0000","last_change":"2024-06-03T08:36:30.883360+0000","last_active":"2024-06-03T09:06:13.929725+0000","last_peered":"2024-06-03T09:06:13.929725+0000","last_clean":"2024-06-03T09:06:13.929725+0000","last_became_active":"2024-06-03T08:36:30.855073+0000","last_became_peered":"2024-06-03T08:36:30.855073+0000","last_unstale":"2024-06-03T09:06:13.929725+0000","last_undegraded":"2024-06-03T09:06:13.929725+0000","last_fullsized":"2024-06-03T09:06:13.929725+0000","mapping_epoch":23,"log_start":"0'0","ondisk_log_start":"0'0","created":23,"last_epoch_clean":25,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-06-03T08:36:28.444764+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-06-03T08:36:28.444764+0000","last_clean_scrub_stamp":"2024-06-03T08:36:28.444764+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-06-04T18:12:22.363473+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,5],"acting":[7,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"17.12","version":"0'0","reported_seq":945,"reported_epoch":1009,"state":"active+clean","last_fresh":"2024-06-03T09:06:13.935141+0000","last_change":"2024-06-03T08:38:01.751562+0000","last_active":"2024-06-03T09:06:13.935141+0000","last_peered":"2024-06-03T09:06:13.935141+0000","last_clean":"2024-06-03T09:06:13.935141+0000","last_became_active":"2024-06-03T08:38:01.745027+0000","last_became_peered":"2024-06-03T08:38:01.745027+0000","last_unstale":"2024-06-03T09:06:13.935141+0000","last_undegraded":"2024-06-03T09:06:13.935141+0000","last_fullsized":"2024-06-03T09:06:13.935141+0000","mapping_epoch":28,"log_start":"0'0","ondisk_log_start":"0'0","created":28,"last_epoch_clean":32,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-06-03T08:37:55.614497+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-06-03T08:37:55.614497+0000","last_clean_scrub_stamp":"2024-06-03T08:37:55.614497+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-06-04T17:54:43.785326+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,3],"acting":[7,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"2.0","version":"0'0","reported_seq":943,"reported_epoch":1009,"state":"active+clean","last_fresh":"2024-06-03T09:06:13.648112+0000","last_change":"2024-06-03T08:36:29.984096+0000","last_active":"2024-06-03T09:06:13.648112+0000","last_peered":"2024-06-03T09:06:13.648112+0000","last_clean":"2024-06-03T09:06:13.648112+0000","last_became_active":"2024-06-03T08:36:29.957875+0000","last_became_peered":"2024-06-03T08:36:29.957875+0000","last_unstale":"2024-06-03T09:06:13.648112+0000","last_undegraded":"2024-06-03T09:06:13.648112+0000","last_fullsized":"2024-06-03T09:06:13.648112+0000","mapping_epoch":23,"log_start":"0'0","ondisk_log_start":"0'0","created":23,"last_epoch_clean":24,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-06-03T08:36:28.444764+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-06-03T08:36:28.444764+0000","last_clean_scrub_stamp":"2024-06-03T08:36:28.444764+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-06-04T08:41:33.679479+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,5],"acting":[3,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"17.13","version":"0'0","reported_seq":943,"reported_epoch":1009,"state":"active+clean","last_fresh":"2024-06-03T09:06:14.438399+0000","last_change":"2024-06-03T08:38:26.838316+0000","last_active":"2024-06-03T09:06:14.438399+0000","last_peered":"2024-06-03T09:06:14.438399+0000","last_clean":"2024-06-03T09:06:14.438399+0000","last_became_active":"2024-06-03T08:38:26.833214+0000","last_became_peered":"2024-06-03T08:38:26.833214+0000","last_unstale":"2024-06-03T09:06:14.438399+0000","last_undegraded":"2024-06-03T09:06:14.438399+0000","last_fullsized":"2024-06-03T09:06:14.438399+0000","mapping_epoch":28,"log_start":"0'0","ondisk_log_start":"0'0","created":28,"last_epoch_clean":52,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-06-03T08:37:55.614497+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-06-03T08:37:55.614497+0000","last_clean_scrub_stamp":"2024-06-03T08:37:55.614497+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-06-04T12:58:51.617998+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,0],"acting":[4,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"2.3","version":"0'0","reported_seq":943,"reported_epoch":1010,"state":"active+clean","last_fresh":"2024-06-03T09:06:18.362614+0000","last_change":"2024-06-03T08:36:30.351670+0000","last_active":"2024-06-03T09:06:18.362614+0000","last_peered":"2024-06-03T09:06:18.362614+0000","last_clean":"2024-06-03T09:06:18.362614+0000","last_became_active":"2024-06-03T08:36:30.343560+0000","last_became_peered":"2024-06-03T08:36:30.343560+0000","last_unstale":"2024-06-03T09:06:18.362614+0000","last_undegraded":"2024-06-03T09:06:18.362614+0000","last_fullsized":"2024-06-03T09:06:18.362614+0000","mapping_epoch":23,"log_start":"0'0","ondisk_log_start":"0'0","created":23,"last_epoch_clean":24,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-06-03T08:36:28.444764+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-06-03T08:36:28.444764+0000","last_clean_scrub_stamp":"2024-06-03T08:36:28.444764+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-06-04T12:02:24.765135+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,2],"acting":[3,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.0","version":"19'76","reported_seq":1051,"reported_epoch":1009,"state":"active+clean","last_fresh":"2024-06-03T09:06:13.703685+0000","last_change":"2024-06-03T08:36:26.702702+0000","last_active":"2024-06-03T09:06:13.703685+0000","last_peered":"2024-06-03T09:06:13.703685+0000","last_clean":"2024-06-03T09:06:13.703685+0000","last_became_active":"2024-06-03T08:36:26.128378+0000","last_became_peered":"2024-06-03T08:36:26.128378+0000","last_unstale":"2024-06-03T09:06:13.703685+0000","last_undegraded":"2024-06-03T09:06:13.703685+0000","last_fullsized":"2024-06-03T09:06:13.703685+0000","mapping_epoch":20,"log_start":"0'0","ondisk_log_start":"0'0","created":16,"last_epoch_clean":22,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-06-03T08:36:15.542635+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-06-03T08:36:15.542635+0000","last_clean_scrub_stamp":"2024-06-03T08:36:15.542635+0000","objects_scrubbed":0,"log_size":76,"log_dups_size":0,"ondisk_log_size":76,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-06-04T12:41:53.731360+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":459280,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":96,"num_read_kb":82,"num_write":113,"num_write_kb":1372,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2,"num_bytes_recovered":397840,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,5],"acting":[1,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"17.10","version":"0'0","reported_seq":944,"reported_epoch":1009,"state":"active+clean","last_fresh":"2024-06-03T09:06:14.422035+0000","last_change":"2024-06-03T08:38:15.399189+0000","last_active":"2024-06-03T09:06:14.422035+0000","last_peered":"2024-06-03T09:06:14.422035+0000","last_clean":"2024-06-03T09:06:14.422035+0000","last_became_active":"2024-06-03T08:38:15.337723+0000","last_became_peered":"2024-06-03T08:38:15.337723+0000","last_unstale":"2024-06-03T09:06:14.422035+0000","last_undegraded":"2024-06-03T09:06:14.422035+0000","last_fullsized":"2024-06-03T09:06:14.422035+0000","mapping_epoch":28,"log_start":"0'0","ondisk_log_start":"0'0","created":28,"last_epoch_clean":42,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-06-03T08:37:55.614497+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-06-03T08:37:55.614497+0000","last_clean_scrub_stamp":"2024-06-03T08:37:55.614497+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-06-04T09:29:46.427317+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,3],"acting":[4,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"17.1a","version":"0'0","reported_seq":960,"reported_epoch":1009,"state":"active+undersized","last_fresh":"2024-06-03T09:06:14.431224+0000","last_change":"2024-06-03T09:06:03.390734+0000","last_active":"2024-06-03T09:06:14.431224+0000","last_peered":"2024-06-03T09:06:14.431224+0000","last_clean":"2024-06-03T09:05:20.322313+0000","last_became_active":"2024-06-03T09:06:03.390734+0000","last_became_peered":"2024-06-03T09:06:03.390734+0000","last_unstale":"2024-06-03T09:06:14.431224+0000","last_undegraded":"2024-06-03T09:06:14.431224+0000","last_fullsized":"2024-06-03T09:06:03.343197+0000","mapping_epoch":999,"log_start":"0'0","ondisk_log_start":"0'0","created":28,"last_epoch_clean":254,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-06-03T08:37:55.614497+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-06-03T08:37:55.614497+0000","last_clean_scrub_stamp":"2024-06-03T08:37:55.614497+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-06-04T15:56:47.628875+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4],"acting":[4],"avail_no_missing":["4"],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"17.1b","version":"0'0","reported_seq":935,"reported_epoch":1009,"state":"active+clean","last_fresh":"2024-06-03T09:06:13.663279+0000","last_change":"2024-06-03T08:38:03.115748+0000","last_active":"2024-06-03T09:06:13.663279+0000","last_peered":"2024-06-03T09:06:13.663279+0000","last_clean":"2024-06-03T09:06:13.663279+0000","last_became_active":"2024-06-03T08:38:03.110533+0000","last_became_peered":"2024-06-03T08:38:03.110533+0000","last_unstale":"2024-06-03T09:06:13.663279+0000","last_undegraded":"2024-06-03T09:06:13.663279+0000","last_fullsized":"2024-06-03T09:06:13.663279+0000","mapping_epoch":28,"log_start":"0'0","ondisk_log_start":"0'0","created":28,"last_epoch_clean":32,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-06-03T08:37:55.614497+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-06-03T08:37:55.614497+0000","last_clean_scrub_stamp":"2024-06-03T08:37:55.614497+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-06-04T12:36:47.471609+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,7],"acting":[3,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"17.18","version":"0'0","reported_seq":944,"reported_epoch":1009,"state":"active+clean","last_fresh":"2024-06-03T09:06:13.924799+0000","last_change":"2024-06-03T08:38:04.632864+0000","last_active":"2024-06-03T09:06:13.924799+0000","last_peered":"2024-06-03T09:06:13.924799+0000","last_clean":"2024-06-03T09:06:13.924799+0000","last_became_active":"2024-06-03T08:38:04.627856+0000","last_became_peered":"2024-06-03T08:38:04.627856+0000","last_unstale":"2024-06-03T09:06:13.924799+0000","last_undegraded":"2024-06-03T09:06:13.924799+0000","last_fullsized":"2024-06-03T09:06:13.924799+0000","mapping_epoch":28,"log_start":"0'0","ondisk_log_start":"0'0","created":28,"last_epoch_clean":33,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-06-03T08:37:55.614497+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-06-03T08:37:55.614497+0000","last_clean_scrub_stamp":"2024-06-03T08:37:55.614497+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-06-04T19:55:51.181598+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,4],"acting":[7,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"17.19","version":"0'0","reported_seq":934,"reported_epoch":1009,"state":"active+clean","last_fresh":"2024-06-03T09:06:13.652943+0000","last_change":"2024-06-03T08:38:01.839491+0000","last_active":"2024-06-03T09:06:13.652943+0000","last_peered":"2024-06-03T09:06:13.652943+0000","last_clean":"2024-06-03T09:06:13.652943+0000","last_became_active":"2024-06-03T08:38:01.829539+0000","last_became_peered":"2024-06-03T08:38:01.829539+0000","last_unstale":"2024-06-03T09:06:13.652943+0000","last_undegraded":"2024-06-03T09:06:13.652943+0000","last_fullsized":"2024-06-03T09:06:13.652943+0000","mapping_epoch":28,"log_start":"0'0","ondisk_log_start":"0'0","created":28,"last_epoch_clean":30,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-06-03T08:37:55.614497+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-06-03T08:37:55.614497+0000","last_clean_scrub_stamp":"2024-06-03T08:37:55.614497+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-06-04T09:10:43.860229+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,2],"acting":[3,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"17.1e","version":"0'0","reported_seq":148,"reported_epoch":1009,"state":"active+undersized","last_fresh":"2024-06-03T09:06:14.445917+0000","last_change":"2024-06-03T09:06:03.406181+0000","last_active":"2024-06-03T09:06:14.445917+0000","last_peered":"2024-06-03T09:06:14.445917+0000","last_clean":"2024-06-03T08:37:55.614497+0000","last_became_active":"2024-06-03T09:06:03.406181+0000","last_became_peered":"2024-06-03T09:06:03.406181+0000","last_unstale":"2024-06-03T09:06:14.445917+0000","last_undegraded":"2024-06-03T09:06:14.445917+0000","last_fullsized":"2024-06-03T09:06:03.363739+0000","mapping_epoch":999,"log_start":"0'0","ondisk_log_start":"0'0","created":28,"last_epoch_clean":255,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-06-03T08:37:55.614497+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-06-03T08:37:55.614497+0000","last_clean_scrub_stamp":"2024-06-03T08:37:55.614497+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-06-04T11:45:24.486600+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4],"acting":[4],"avail_no_missing":["4"],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"17.1f","version":"0'0","reported_seq":941,"reported_epoch":1009,"state":"active+clean","last_fresh":"2024-06-03T09:06:14.450053+0000","last_change":"2024-06-03T08:38:04.383538+0000","last_active":"2024-06-03T09:06:14.450053+0000","last_peered":"2024-06-03T09:06:14.450053+0000","last_clean":"2024-06-03T09:06:14.450053+0000","last_became_active":"2024-06-03T08:38:04.367278+0000","last_became_peered":"2024-06-03T08:38:04.367278+0000","last_unstale":"2024-06-03T09:06:14.450053+0000","last_undegraded":"2024-06-03T09:06:14.450053+0000","last_fullsized":"2024-06-03T09:06:14.450053+0000","mapping_epoch":28,"log_start":"0'0","ondisk_log_start":"0'0","created":28,"last_epoch_clean":32,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-06-03T08:37:55.614497+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-06-03T08:37:55.614497+0000","last_clean_scrub_stamp":"2024-06-03T08:37:55.614497+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-06-04T10:39:26.552282+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,2],"acting":[0,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"17.1c","version":"0'0","reported_seq":944,"reported_epoch":1010,"state":"active+undersized","last_fresh":"2024-06-03T09:06:18.327492+0000","last_change":"2024-06-03T09:06:03.382210+0000","last_active":"2024-06-03T09:06:18.327492+0000","last_peered":"2024-06-03T09:06:18.327492+0000","last_clean":"2024-06-03T09:04:43.199660+0000","last_became_active":"2024-06-03T09:06:03.382210+0000","last_became_peered":"2024-06-03T09:06:03.382210+0000","last_unstale":"2024-06-03T09:06:18.327492+0000","last_undegraded":"2024-06-03T09:06:18.327492+0000","last_fullsized":"2024-06-03T09:06:03.277874+0000","mapping_epoch":999,"log_start":"0'0","ondisk_log_start":"0'0","created":28,"last_epoch_clean":255,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-06-03T08:37:55.614497+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-06-03T08:37:55.614497+0000","last_clean_scrub_stamp":"2024-06-03T08:37:55.614497+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-06-04T12:18:50.013884+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5],"acting":[5],"avail_no_missing":["5"],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"17.1d","version":"0'0","reported_seq":923,"reported_epoch":1009,"state":"active+clean","last_fresh":"2024-06-03T09:06:13.715906+0000","last_change":"2024-06-03T08:38:06.794630+0000","last_active":"2024-06-03T09:06:13.715906+0000","last_peered":"2024-06-03T09:06:13.715906+0000","last_clean":"2024-06-03T09:06:13.715906+0000","last_became_active":"2024-06-03T08:38:06.592317+0000","last_became_peered":"2024-06-03T08:38:06.592317+0000","last_unstale":"2024-06-03T09:06:13.715906+0000","last_undegraded":"2024-06-03T09:06:13.715906+0000","last_fullsized":"2024-06-03T09:06:13.715906+0000","mapping_epoch":28,"log_start":"0'0","ondisk_log_start":"0'0","created":28,"last_epoch_clean":33,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-06-03T08:37:55.614497+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-06-03T08:37:55.614497+0000","last_clean_scrub_stamp":"2024-06-03T08:37:55.614497+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-06-04T12:33:30.785369+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,7],"acting":[1,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"17.2","version":"0'0","reported_seq":925,"reported_epoch":1009,"state":"active+clean","last_fresh":"2024-06-03T09:06:13.708688+0000","last_change":"2024-06-03T08:38:14.981410+0000","last_active":"2024-06-03T09:06:13.708688+0000","last_peered":"2024-06-03T09:06:13.708688+0000","last_clean":"2024-06-03T09:06:13.708688+0000","last_became_active":"2024-06-03T08:38:14.973825+0000","last_became_peered":"2024-06-03T08:38:14.973825+0000","last_unstale":"2024-06-03T09:06:13.708688+0000","last_undegraded":"2024-06-03T09:06:13.708688+0000","last_fullsized":"2024-06-03T09:06:13.708688+0000","mapping_epoch":28,"log_start":"0'0","ondisk_log_start":"0'0","created":28,"last_epoch_clean":41,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-06-03T08:37:55.614497+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-06-03T08:37:55.614497+0000","last_clean_scrub_stamp":"2024-06-03T08:37:55.614497+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-06-04T17:01:05.518613+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,4],"acting":[1,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"17.3","version":"0'0","reported_seq":943,"reported_epoch":1009,"state":"active+clean","last_fresh":"2024-06-03T09:06:14.437636+0000","last_change":"2024-06-03T08:38:14.619745+0000","last_active":"2024-06-03T09:06:14.437636+0000","last_peered":"2024-06-03T09:06:14.437636+0000","last_clean":"2024-06-03T09:06:14.437636+0000","last_became_active":"2024-06-03T08:38:14.510665+0000","last_became_peered":"2024-06-03T08:38:14.510665+0000","last_unstale":"2024-06-03T09:06:14.437636+0000","last_undegraded":"2024-06-03T09:06:14.437636+0000","last_fullsized":"2024-06-03T09:06:14.437636+0000","mapping_epoch":28,"log_start":"0'0","ondisk_log_start":"0'0","created":28,"last_epoch_clean":42,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-06-03T08:37:55.614497+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-06-03T08:37:55.614497+0000","last_clean_scrub_stamp":"2024-06-03T08:37:55.614497+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-06-04T12:03:47.392064+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,7],"acting":[0,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"17.0","version":"0'0","reported_seq":944,"reported_epoch":1009,"state":"active+clean","last_fresh":"2024-06-03T09:06:14.427123+0000","last_change":"2024-06-03T08:38:04.237118+0000","last_active":"2024-06-03T09:06:14.427123+0000","last_peered":"2024-06-03T09:06:14.427123+0000","last_clean":"2024-06-03T09:06:14.427123+0000","last_became_active":"2024-06-03T08:38:04.231954+0000","last_became_peered":"2024-06-03T08:38:04.231954+0000","last_unstale":"2024-06-03T09:06:14.427123+0000","last_undegraded":"2024-06-03T09:06:14.427123+0000","last_fullsized":"2024-06-03T09:06:14.427123+0000","mapping_epoch":28,"log_start":"0'0","ondisk_log_start":"0'0","created":28,"last_epoch_clean":32,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-06-03T08:37:55.614497+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-06-03T08:37:55.614497+0000","last_clean_scrub_stamp":"2024-06-03T08:37:55.614497+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-06-04T09:56:10.741637+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,2],"acting":[0,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"17.1","version":"65'7","reported_seq":959,"reported_epoch":1009,"state":"active+undersized+degraded","last_fresh":"2024-06-03T09:06:13.709209+0000","last_change":"2024-06-03T09:06:03.370660+0000","last_active":"2024-06-03T09:06:13.709209+0000","last_peered":"2024-06-03T09:06:13.709209+0000","last_clean":"2024-06-03T09:05:03.733864+0000","last_became_active":"2024-06-03T09:06:03.370660+0000","last_became_peered":"2024-06-03T09:06:03.370660+0000","last_unstale":"2024-06-03T09:06:13.709209+0000","last_undegraded":"2024-06-03T09:06:03.266076+0000","last_fullsized":"2024-06-03T09:06:03.264525+0000","mapping_epoch":999,"log_start":"0'0","ondisk_log_start":"0'0","created":28,"last_epoch_clean":255,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-06-03T08:37:55.614497+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-06-03T08:37:55.614497+0000","last_clean_scrub_stamp":"2024-06-03T08:37:55.614497+0000","objects_scrubbed":0,"log_size":7,"log_dups_size":0,"ondisk_log_size":7,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-06-04T11:55:18.194283+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":5,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":1,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":6,"num_read_kb":6,"num_write":7,"num_write_kb":7,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5],"acting":[5],"avail_no_missing":["5"],"object_location_counts":[{"shards":"5","objects":1}],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]}],"pool_stats":[{"poolid":2,"num_pg":8,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":14,"acting":14,"num_store_stats":7},{"poolid":1,"num_pg":1,"stat_sum":{"num_bytes":459280,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":96,"num_read_kb":82,"num_write":113,"num_write_kb":1372,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2,"num_bytes_recovered":397840,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":802816,"data_stored":795680,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":76,"ondisk_log_size":76,"up":2,"acting":2,"num_store_stats":2},{"poolid":17,"num_pg":32,"stat_sum":{"num_bytes":5,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":1,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":6,"num_read_kb":6,"num_write":7,"num_write_kb":7,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":8192,"data_stored":10,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":7,"ondisk_log_size":7,"up":52,"acting":52,"num_store_stats":8}],"osd_stats":[{"osd":6,"up_from":253,"seq":1086626726200,"num_pgs":32,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":54772,"kb_used_data":5608,"kb_used_omap":1,"kb_used_meta":49150,"kb_avail":94317068,"statfs":{"total":96636764160,"available":96580677632,"internally_reserved":0,"allocated":5742592,"data_stored":2767180,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1593,"internal_metadata":50330055},"hb_peers":[0,1,2,3,4,5,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":109,"apply_latency_ms":109,"commit_latency_ns":109000000,"apply_latency_ns":109000000},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Mon Jun 3 09:04:02 2024","interfaces":[{"interface":"back","average":{"1min":1582.4780000000001,"5min":332.72699999999998,"15min":124.113},"min":{"1min":4.4829999999999997,"5min":4.1959999999999997,"15min":4.1959999999999997},"max":{"1min":22766.882000000001,"5min":22766.882000000001,"15min":22766.882000000001},"last":21.917999999999999},{"interface":"front","average":{"1min":1579.5409999999999,"5min":334.94499999999999,"15min":125.84999999999999},"min":{"1min":4.8129999999999997,"5min":4.7539999999999996,"15min":4.4530000000000003},"max":{"1min":22766.289000000001,"5min":22766.289000000001,"15min":22766.289000000001},"last":25.408999999999999}]},{"osd":1,"last update":"Mon Jun 3 09:03:40 2024","interfaces":[{"interface":"back","average":{"1min":36.420999999999999,"5min":77.054000000000002,"15min":132.84800000000001},"min":{"1min":6.0380000000000003,"5min":5.8799999999999999,"15min":5.484},"max":{"1min":589.64099999999996,"5min":4138.1679999999997,"15min":9955.1299999999992},"last":26.84},{"interface":"front","average":{"1min":40.240000000000002,"5min":79.674999999999997,"15min":133.38999999999999},"min":{"1min":8.4589999999999996,"5min":8.4090000000000007,"15min":6.1390000000000002},"max":{"1min":588.60799999999995,"5min":4137.6530000000002,"15min":9827.5509999999995},"last":26.263000000000002}]},{"osd":2,"last update":"Mon Jun 3 09:03:40 2024","interfaces":[{"interface":"back","average":{"1min":12.613,"5min":24.600999999999999,"15min":21.088000000000001},"min":{"1min":6.9740000000000002,"5min":6.9740000000000002,"15min":6.5330000000000004},"max":{"1min":26.076000000000001,"5min":376.24900000000002,"15min":376.24900000000002},"last":24.899000000000001},{"interface":"front","average":{"1min":12.493,"5min":22.207999999999998,"15min":21.219000000000001},"min":{"1min":6.5810000000000004,"5min":6.5110000000000001,"15min":6.0750000000000002},"max":{"1min":29.690999999999999,"5min":271.51100000000002,"15min":434.77300000000002},"last":40.372}]},{"osd":3,"last update":"Mon Jun 3 09:03:40 2024","interfaces":[{"interface":"back","average":{"1min":14.093999999999999,"5min":24.143999999999998,"15min":29.376999999999999},"min":{"1min":10.196999999999999,"5min":6.2919999999999998,"15min":6.1719999999999997},"max":{"1min":24.832000000000001,"5min":197.84899999999999,"15min":1235.5799999999999},"last":24.353999999999999},{"interface":"front","average":{"1min":9.3260000000000005,"5min":18.463999999999999,"15min":27.631},"min":{"1min":6.125,"5min":6.125,"15min":6.125},"max":{"1min":28.166,"5min":200.173,"15min":1157.6780000000001},"last":23.454000000000001}]},{"osd":4,"last update":"Mon Jun 3 09:04:29 2024","interfaces":[{"interface":"back","average":{"1min":10.279,"5min":1253.809,"15min":450.79199999999997},"min":{"1min":7.2300000000000004,"5min":6.093,"15min":6.093},"max":{"1min":15.000999999999999,"5min":28093.096000000001,"15min":28093.096000000001},"last":101.45399999999999},{"interface":"front","average":{"1min":15.693,"5min":1279.759,"15min":463.87099999999998},"min":{"1min":10.551,"5min":7.7480000000000002,"15min":7.7480000000000002},"max":{"1min":27.898,"5min":28323.837,"15min":28323.837},"last":102.262}]},{"osd":5,"last update":"Mon Jun 3 09:04:10 2024","interfaces":[{"interface":"back","average":{"1min":13.728999999999999,"5min":24.670000000000002,"15min":23.815999999999999},"min":{"1min":10.401,"5min":7.516,"15min":7.516},"max":{"1min":25.138000000000002,"5min":320.85000000000002,"15min":445.041},"last":103.52500000000001},{"interface":"front","average":{"1min":13.847,"5min":29.283000000000001,"15min":23.393999999999998},"min":{"1min":7.944,"5min":6.266,"15min":6.266},"max":{"1min":18.817,"5min":530.56700000000001,"15min":530.56700000000001},"last":29.263999999999999}]},{"osd":7,"last update":"Mon Jun 3 09:03:40 2024","interfaces":[{"interface":"back","average":{"1min":20.568999999999999,"5min":30.286999999999999,"15min":27.977},"min":{"1min":13.874000000000001,"5min":10.58,"15min":9.2859999999999996},"max":{"1min":86.959000000000003,"5min":222.07599999999999,"15min":255.08799999999999},"last":37.618000000000002},{"interface":"front","average":{"1min":19.102,"5min":29.550000000000001,"15min":27.763000000000002},"min":{"1min":10.052,"5min":8.6709999999999994,"15min":8.5429999999999993},"max":{"1min":67.968999999999994,"5min":293.62099999999998,"15min":438.14600000000002},"last":111.985}]}]},{"osd":7,"up_from":20,"seq":85899346288,"num_pgs":9,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":58964,"kb_used_data":5704,"kb_used_omap":1,"kb_used_meta":53246,"kb_avail":94312876,"statfs":{"total":96636764160,"available":96576385024,"internally_reserved":0,"allocated":5840896,"data_stored":2740727,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1589,"internal_metadata":54524363},"hb_peers":[0,1,2,3,4,5],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Mon Jun 3 09:06:09 2024","interfaces":[{"interface":"back","average":{"1min":14.073,"5min":894.22400000000005,"15min":308.93400000000003},"min":{"1min":5.5709999999999997,"5min":4.218,"15min":4.0599999999999996},"max":{"1min":90.757999999999996,"5min":21729.001,"15min":21729.001},"last":6.1630000000000003},{"interface":"front","average":{"1min":9.2550000000000008,"5min":906.23500000000001,"15min":314.06},"min":{"1min":5.5789999999999997,"5min":4.8810000000000002,"15min":4.8810000000000002},"max":{"1min":25.937000000000001,"5min":21899.732,"15min":21899.732},"last":7.109}]},{"osd":1,"last update":"Mon Jun 3 09:06:09 2024","interfaces":[{"interface":"back","average":{"1min":15.182,"5min":69.183999999999997,"15min":219.173},"min":{"1min":7.3639999999999999,"5min":6.6200000000000001,"15min":6.1319999999999997},"max":{"1min":76.926000000000002,"5min":2795.1329999999998,"15min":18373.257000000001},"last":9.9030000000000005},{"interface":"front","average":{"1min":21.495999999999999,"5min":74.703999999999994,"15min":225.285},"min":{"1min":8.8399999999999999,"5min":8.8399999999999999,"15min":8.5129999999999999},"max":{"1min":94.977999999999994,"5min":2803.3670000000002,"15min":18374.601999999999},"last":12.782}]},{"osd":2,"last update":"Mon Jun 3 09:06:09 2024","interfaces":[{"interface":"back","average":{"1min":19.164000000000001,"5min":26.902000000000001,"15min":22.920999999999999},"min":{"1min":11.207000000000001,"5min":8.5030000000000001,"15min":8.5030000000000001},"max":{"1min":91.319999999999993,"5min":333.99200000000002,"15min":333.99200000000002},"last":14.696},{"interface":"front","average":{"1min":11.125999999999999,"5min":18.132000000000001,"15min":16.353000000000002},"min":{"1min":6.202,"5min":5.8369999999999997,"15min":5.5069999999999997},"max":{"1min":28.649000000000001,"5min":226.18100000000001,"15min":226.18100000000001},"last":8.0749999999999993}]},{"osd":3,"last update":"Mon Jun 3 09:06:09 2024","interfaces":[{"interface":"back","average":{"1min":22.291,"5min":26.289999999999999,"15min":31.074999999999999},"min":{"1min":11.714,"5min":6.8959999999999999,"15min":6.3760000000000003},"max":{"1min":83.102000000000004,"5min":316.64400000000001,"15min":2039.9059999999999},"last":13.901},{"interface":"front","average":{"1min":13.324999999999999,"5min":17.963000000000001,"15min":25.969999999999999},"min":{"1min":6.1200000000000001,"5min":5.2089999999999996,"15min":5.2089999999999996},"max":{"1min":49.908999999999999,"5min":233.542,"15min":2037.5550000000001},"last":9.3919999999999995}]},{"osd":4,"last update":"Mon Jun 3 09:06:29 2024","interfaces":[{"interface":"back","average":{"1min":17.675999999999998,"5min":1026.325,"15min":472.47199999999998},"min":{"1min":7.306,"5min":5.9089999999999998,"15min":5.9089999999999998},"max":{"1min":140.03299999999999,"5min":28969.369999999999,"15min":30073.917000000001},"last":8.8829999999999991},{"interface":"front","average":{"1min":17.890999999999998,"5min":1044.5139999999999,"15min":480.226},"min":{"1min":5.9279999999999999,"5min":5.9279999999999999,"15min":5.9279999999999999},"max":{"1min":141.179,"5min":29244.138999999999,"15min":30348.021000000001},"last":12.252000000000001}]},{"osd":5,"last update":"Mon Jun 3 09:06:02 2024","interfaces":[{"interface":"back","average":{"1min":24.372,"5min":26.856000000000002,"15min":23.571999999999999},"min":{"1min":12.278,"5min":9.2460000000000004,"15min":6.5650000000000004},"max":{"1min":183.64099999999999,"5min":317.572,"15min":317.572},"last":15.528},{"interface":"front","average":{"1min":21.259,"5min":23.364000000000001,"15min":21.193999999999999},"min":{"1min":7.8499999999999996,"5min":7.0999999999999996,"15min":7.008},"max":{"1min":184.53399999999999,"5min":306.60599999999999,"15min":306.60599999999999},"last":10.446999999999999}]},{"osd":6,"last update":"Mon Jun 3 09:04:31 2024","interfaces":[{"interface":"back","average":{"1min":44.061,"5min":26.821000000000002,"15min":91.176000000000002},"min":{"1min":9.0800000000000001,"5min":7.9390000000000001,"15min":7.9390000000000001},"max":{"1min":268.029,"5min":308.916,"15min":8734.4639999999999},"last":6.6769999999999996},{"interface":"front","average":{"1min":46.643000000000001,"5min":23.879999999999999,"15min":88.765000000000001},"min":{"1min":8.8119999999999994,"5min":6.9240000000000004,"15min":6.742},"max":{"1min":305.64600000000002,"5min":305.64600000000002,"15min":8735.5319999999992},"last":12.327}]}]},{"osd":5,"up_from":20,"seq":85899346262,"num_pgs":9,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":59360,"kb_used_data":6100,"kb_used_omap":1,"kb_used_meta":53246,"kb_avail":94312480,"statfs":{"total":96636764160,"available":96575979520,"internally_reserved":0,"allocated":6246400,"data_stored":3138572,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1594,"internal_metadata":54524358},"hb_peers":[0,1,2,3,4,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Mon Jun 3 09:06:06 2024","interfaces":[{"interface":"back","average":{"1min":7.468,"5min":1703.037,"15min":580.51900000000001},"min":{"1min":4.6509999999999998,"5min":4.6509999999999998,"15min":4.4020000000000001},"max":{"1min":14.711,"5min":26433.764999999999,"15min":26433.764999999999},"last":4.641},{"interface":"front","average":{"1min":9.9429999999999996,"5min":1725.27,"15min":587.87800000000004},"min":{"1min":5.5999999999999996,"5min":5.1319999999999997,"15min":4.8170000000000002},"max":{"1min":13.590999999999999,"5min":26704.698,"15min":26704.698},"last":8.7279999999999998}]},{"osd":1,"last update":"Mon Jun 3 09:06:00 2024","interfaces":[{"interface":"back","average":{"1min":10.862,"5min":67.731999999999999,"15min":217.364},"min":{"1min":7.1769999999999996,"5min":6.0890000000000004,"15min":5.4930000000000003},"max":{"1min":16.998999999999999,"5min":3923.4090000000001,"15min":18811.696},"last":12.723000000000001},{"interface":"front","average":{"1min":10.417,"5min":65.141999999999996,"15min":220.75700000000001},"min":{"1min":5.3209999999999997,"5min":5.3209999999999997,"15min":5.0209999999999999},"max":{"1min":15.821999999999999,"5min":4019.6129999999998,"15min":18968.365000000002},"last":7.766}]},{"osd":2,"last update":"Mon Jun 3 09:06:00 2024","interfaces":[{"interface":"back","average":{"1min":11.366,"5min":24.18,"15min":21.137},"min":{"1min":5.6369999999999996,"5min":5.6369999999999996,"15min":5.0510000000000002},"max":{"1min":15.529999999999999,"5min":332.48200000000003,"15min":332.48200000000003},"last":11.538},{"interface":"front","average":{"1min":10.089,"5min":23.117000000000001,"15min":24.050000000000001},"min":{"1min":6.6769999999999996,"5min":6.6769999999999996,"15min":5.8739999999999997},"max":{"1min":15.821,"5min":353.03199999999998,"15min":353.03199999999998},"last":6.9349999999999996}]},{"osd":3,"last update":"Mon Jun 3 09:06:00 2024","interfaces":[{"interface":"back","average":{"1min":13.472,"5min":24.283999999999999,"15min":31.666},"min":{"1min":10.111000000000001,"5min":7.4720000000000004,"15min":7.4210000000000003},"max":{"1min":16.271999999999998,"5min":353.57499999999999,"15min":2855.8519999999999},"last":14.273999999999999},{"interface":"front","average":{"1min":9.5899999999999999,"5min":15.875,"15min":25.834},"min":{"1min":6.6609999999999996,"5min":5.7729999999999997,"15min":5.1909999999999998},"max":{"1min":15.112,"5min":203.804,"15min":2869.7860000000001},"last":13.554}]},{"osd":4,"last update":"Mon Jun 3 09:06:29 2024","interfaces":[{"interface":"back","average":{"1min":11.067,"5min":780.21000000000004,"15min":377.28399999999999},"min":{"1min":5.4779999999999998,"5min":5.4779999999999998,"15min":5.4779999999999998},"max":{"1min":14.654999999999999,"5min":23315.438999999998,"15min":28619.825000000001},"last":11.292},{"interface":"front","average":{"1min":14.349,"5min":782.97299999999996,"15min":377.93400000000003},"min":{"1min":7.492,"5min":6.5030000000000001,"15min":6.5030000000000001},"max":{"1min":17.707000000000001,"5min":23317.502,"15min":28621.294000000002},"last":15.134}]},{"osd":6,"last update":"Mon Jun 3 09:03:57 2024","interfaces":[{"interface":"back","average":{"1min":23.536000000000001,"5min":36.457000000000001,"15min":83.799999999999997},"min":{"1min":9.4800000000000004,"5min":9.4800000000000004,"15min":8.0630000000000006},"max":{"1min":89.442999999999998,"5min":508.77999999999997,"15min":9193.5169999999998},"last":157.79300000000001},{"interface":"front","average":{"1min":18.891999999999999,"5min":22.100999999999999,"15min":75.489999999999995},"min":{"1min":5.923,"5min":5.923,"15min":5.923},"max":{"1min":78.424999999999997,"5min":464.62,"15min":9191.4429999999993},"last":148.09}]},{"osd":7,"last update":"Mon Jun 3 09:06:00 2024","interfaces":[{"interface":"back","average":{"1min":13.257,"5min":20.917999999999999,"15min":21.088999999999999},"min":{"1min":10.475,"5min":6.7629999999999999,"15min":6.7629999999999999},"max":{"1min":17.245000000000001,"5min":211.22999999999999,"15min":211.22999999999999},"last":12.083},{"interface":"front","average":{"1min":12.872999999999999,"5min":20.587,"15min":24.021000000000001},"min":{"1min":8.5760000000000005,"5min":7.7709999999999999,"15min":6.8570000000000002},"max":{"1min":18.359999999999999,"5min":223.191,"15min":286.69499999999999},"last":9.3849999999999998}]}]},{"osd":4,"up_from":20,"seq":85899346270,"num_pgs":12,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":63060,"kb_used_data":5704,"kb_used_omap":1,"kb_used_meta":57342,"kb_avail":94308780,"statfs":{"total":96636764160,"available":96572190720,"internally_reserved":0,"allocated":5840896,"data_stored":2740727,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1594,"internal_metadata":58718662},"hb_peers":[0,1,2,3,5,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Mon Jun 3 09:05:34 2024","interfaces":[{"interface":"back","average":{"1min":12.416,"5min":1099.4649999999999,"15min":382.15199999999999},"min":{"1min":4.5279999999999996,"5min":4.5279999999999996,"15min":4.3220000000000001},"max":{"1min":123.878,"5min":21726.743999999999,"15min":21726.743999999999},"last":6.8849999999999998},{"interface":"front","average":{"1min":16.462,"5min":1108.894,"15min":387.70299999999997},"min":{"1min":6.4089999999999998,"5min":5.8380000000000001,"15min":5.4909999999999997},"max":{"1min":130.185,"5min":21727.993999999999,"15min":21727.993999999999},"last":8.8729999999999993}]},{"osd":1,"last update":"Mon Jun 3 09:05:34 2024","interfaces":[{"interface":"back","average":{"1min":14.727,"5min":56.755000000000003,"15min":341.09399999999999},"min":{"1min":4.6529999999999996,"5min":4.1970000000000001,"15min":4.1970000000000001},"max":{"1min":126.899,"5min":3346.4389999999999,"15min":22256.098000000002},"last":6.5860000000000003},{"interface":"front","average":{"1min":13.472,"5min":57.503,"15min":340.303},"min":{"1min":5.0590000000000002,"5min":5.0590000000000002,"15min":4.2640000000000002},"max":{"1min":126.096,"5min":3347.181,"15min":22247.317999999999},"last":11.868}]},{"osd":2,"last update":"Mon Jun 3 09:05:34 2024","interfaces":[{"interface":"back","average":{"1min":26.088999999999999,"5min":23.866,"15min":20.620999999999999},"min":{"1min":7.0869999999999997,"5min":6.8239999999999998,"15min":6.3150000000000004},"max":{"1min":337.60399999999998,"5min":393.58600000000001,"15min":393.58600000000001},"last":11.247999999999999},{"interface":"front","average":{"1min":15.800000000000001,"5min":22.584,"15min":23.245999999999999},"min":{"1min":6.0629999999999997,"5min":6.0629999999999997,"15min":6.0629999999999997},"max":{"1min":127.913,"5min":394.471,"15min":394.471},"last":13.949999999999999}]},{"osd":3,"last update":"Mon Jun 3 09:05:34 2024","interfaces":[{"interface":"back","average":{"1min":17.544,"5min":19.143999999999998,"15min":18.742000000000001},"min":{"1min":5.8719999999999999,"5min":5.4100000000000001,"15min":5.4000000000000004},"max":{"1min":127.43600000000001,"5min":322.63,"15min":322.63},"last":14.968},{"interface":"front","average":{"1min":21.327000000000002,"5min":23.559000000000001,"15min":23.797000000000001},"min":{"1min":10.023,"5min":7.484,"15min":7.484},"max":{"1min":204.517,"5min":392.67500000000001,"15min":392.67500000000001},"last":13.297000000000001}]},{"osd":5,"last update":"Mon Jun 3 09:05:34 2024","interfaces":[{"interface":"back","average":{"1min":24.274999999999999,"5min":21.274999999999999,"15min":20.988},"min":{"1min":6.7489999999999997,"5min":6.7489999999999997,"15min":6.7489999999999997},"max":{"1min":268.10199999999998,"5min":344.23399999999998,"15min":344.23399999999998},"last":12.752000000000001},{"interface":"front","average":{"1min":26.361999999999998,"5min":25.218,"15min":26.084},"min":{"1min":12.228,"5min":8.5180000000000007,"15min":8.5180000000000007},"max":{"1min":264.75,"5min":345.18700000000001,"15min":345.18700000000001},"last":15.800000000000001}]},{"osd":6,"last update":"Mon Jun 3 09:04:32 2024","interfaces":[{"interface":"back","average":{"1min":15.548,"5min":32.957999999999998,"15min":152.738},"min":{"1min":9.141,"5min":9.141,"15min":8.2319999999999993},"max":{"1min":52.57,"5min":395.238,"15min":12496.335999999999},"last":208.90000000000001},{"interface":"front","average":{"1min":12.680999999999999,"5min":25.428000000000001,"15min":149.30600000000001},"min":{"1min":6.4379999999999997,"5min":5.6319999999999997,"15min":5.6319999999999997},"max":{"1min":46.283000000000001,"5min":377.74299999999999,"15min":12616.297},"last":205.88900000000001}]},{"osd":7,"last update":"Mon Jun 3 09:05:34 2024","interfaces":[{"interface":"back","average":{"1min":18.295999999999999,"5min":19.585999999999999,"15min":21.062000000000001},"min":{"1min":6.2110000000000003,"5min":6.2110000000000003,"15min":5.8010000000000002},"max":{"1min":207.25299999999999,"5min":358.94900000000001,"15min":358.94900000000001},"last":9.4960000000000004},{"interface":"front","average":{"1min":23.712,"5min":24.024999999999999,"15min":24.640000000000001},"min":{"1min":11.824999999999999,"5min":7.7789999999999999,"15min":7.7789999999999999},"max":{"1min":209.32900000000001,"5min":346.25200000000001,"15min":380.06900000000002},"last":16.488}]}]},{"osd":2,"up_from":20,"seq":85899346290,"num_pgs":11,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":58964,"kb_used_data":5704,"kb_used_omap":1,"kb_used_meta":53246,"kb_avail":94312876,"statfs":{"total":96636764160,"available":96576385024,"internally_reserved":0,"allocated":5840896,"data_stored":2740727,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1594,"internal_metadata":54524358},"hb_peers":[0,1,3,4,5,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Mon Jun 3 09:06:23 2024","interfaces":[{"interface":"back","average":{"1min":8.9629999999999992,"5min":946.25400000000002,"15min":326.36099999999999},"min":{"1min":5.3479999999999999,"5min":4.7050000000000001,"15min":4.1420000000000003},"max":{"1min":26.015000000000001,"5min":23465.18,"15min":23465.18},"last":6.3940000000000001},{"interface":"front","average":{"1min":7.6130000000000004,"5min":944.39800000000002,"15min":326.178},"min":{"1min":4.6790000000000003,"5min":4.6790000000000003,"15min":4.6790000000000003},"max":{"1min":20.802,"5min":23465.966,"15min":23465.966},"last":4.492}]},{"osd":1,"last update":"Mon Jun 3 09:06:23 2024","interfaces":[{"interface":"back","average":{"1min":13.007,"5min":76.507000000000005,"15min":294.72899999999998},"min":{"1min":6.4089999999999998,"5min":5.4199999999999999,"15min":5.4199999999999999},"max":{"1min":85.430000000000007,"5min":4220.5290000000005,"15min":18954.987000000001},"last":7.4249999999999998},{"interface":"front","average":{"1min":15.894,"5min":82.119,"15min":302.92399999999998},"min":{"1min":7.367,"5min":7.367,"15min":7.367},"max":{"1min":98.498000000000005,"5min":4325.9489999999996,"15min":19143.834999999999},"last":9.6929999999999996}]},{"osd":3,"last update":"Mon Jun 3 09:06:23 2024","interfaces":[{"interface":"back","average":{"1min":9.7579999999999991,"5min":12.444000000000001,"15min":28.065000000000001},"min":{"1min":7,"5min":5.9290000000000003,"15min":5.2450000000000001},"max":{"1min":28.164000000000001,"5min":118.913,"15min":2594.8150000000001},"last":8.8740000000000006},{"interface":"front","average":{"1min":12.321999999999999,"5min":15.25,"15min":31.349},"min":{"1min":9.1240000000000006,"5min":8.0760000000000005,"15min":6.0780000000000003},"max":{"1min":32.360999999999997,"5min":156.34700000000001,"15min":2706.5120000000002},"last":11.180999999999999}]},{"osd":4,"last update":"Mon Jun 3 09:05:57 2024","interfaces":[{"interface":"back","average":{"1min":10.573,"5min":1266.894,"15min":438.154},"min":{"1min":8.6379999999999999,"5min":7.8789999999999996,"15min":6.2039999999999997},"max":{"1min":15.105,"5min":26660.861000000001,"15min":26660.861000000001},"last":10.538},{"interface":"front","average":{"1min":12.407999999999999,"5min":1266.7460000000001,"15min":435.75400000000002},"min":{"1min":10.266999999999999,"5min":10.266999999999999,"15min":6.2720000000000002},"max":{"1min":15.228,"5min":26616.243999999999,"15min":26616.243999999999},"last":11.879}]},{"osd":5,"last update":"Mon Jun 3 09:06:23 2024","interfaces":[{"interface":"back","average":{"1min":14.103,"5min":18.091000000000001,"15min":21.119},"min":{"1min":9.6349999999999998,"5min":9.2159999999999993,"15min":6.2050000000000001},"max":{"1min":32.890000000000001,"5min":252.31100000000001,"15min":252.31100000000001},"last":12.696},{"interface":"front","average":{"1min":11.545999999999999,"5min":16.170999999999999,"15min":20.254000000000001},"min":{"1min":7.6070000000000002,"5min":7.4850000000000003,"15min":6.8810000000000002},"max":{"1min":29.228999999999999,"5min":258.31700000000001,"15min":258.31700000000001},"last":8.6699999999999999}]},{"osd":6,"last update":"Mon Jun 3 09:04:28 2024","interfaces":[{"interface":"back","average":{"1min":14.063000000000001,"5min":24.850000000000001,"15min":106.755},"min":{"1min":9.0820000000000007,"5min":8.9649999999999999,"15min":8.9649999999999999},"max":{"1min":24.666,"5min":188.06700000000001,"15min":9521.0010000000002},"last":183.60900000000001},{"interface":"front","average":{"1min":13.933999999999999,"5min":21.081,"15min":106.142},"min":{"1min":5.8719999999999999,"5min":5.8719999999999999,"15min":5.8719999999999999},"max":{"1min":117.551,"5min":193.887,"15min":9585.6620000000003},"last":184.20599999999999}]},{"osd":7,"last update":"Mon Jun 3 09:06:23 2024","interfaces":[{"interface":"back","average":{"1min":17.056000000000001,"5min":20.055,"15min":23.268999999999998},"min":{"1min":10.353999999999999,"5min":6.6130000000000004,"15min":6.6130000000000004},"max":{"1min":88.974999999999994,"5min":242.56800000000001,"15min":242.56800000000001},"last":13.343},{"interface":"front","average":{"1min":18.885000000000002,"5min":21.460000000000001,"15min":25.489999999999998},"min":{"1min":12.497,"5min":9.0660000000000007,"15min":9.0660000000000007},"max":{"1min":102.229,"5min":181.56399999999999,"15min":214.33799999999999},"last":14.042}]}]},{"osd":3,"up_from":14,"seq":60129542504,"num_pgs":11,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":58964,"kb_used_data":5704,"kb_used_omap":1,"kb_used_meta":53246,"kb_avail":94312876,"statfs":{"total":96636764160,"available":96576385024,"internally_reserved":0,"allocated":5840896,"data_stored":2740727,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1592,"internal_metadata":54524360},"hb_peers":[0,1,2,4,5,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Mon Jun 3 09:06:25 2024","interfaces":[{"interface":"back","average":{"1min":7.79,"5min":1060.1959999999999,"15min":367.01799999999997},"min":{"1min":5.6909999999999998,"5min":5.173,"15min":4.6929999999999996},"max":{"1min":10.59,"5min":23275.167000000001,"15min":23275.167000000001},"last":5.1989999999999998},{"interface":"front","average":{"1min":11.215999999999999,"5min":1066.5799999999999,"15min":372.476},"min":{"1min":8.7080000000000002,"5min":7.2510000000000003,"15min":5.9880000000000004},"max":{"1min":13.499000000000001,"5min":23278.536,"15min":23278.536},"last":10.922000000000001}]},{"osd":1,"last update":"Mon Jun 3 09:06:25 2024","interfaces":[{"interface":"back","average":{"1min":7.3760000000000003,"5min":15.614000000000001,"15min":389.83100000000002},"min":{"1min":4.335,"5min":4.335,"15min":4.335},"max":{"1min":11.289,"5min":250.72300000000001,"15min":21658.205999999998},"last":8.5549999999999997},{"interface":"front","average":{"1min":8.9499999999999993,"5min":14.135,"15min":381.10300000000001},"min":{"1min":6.5609999999999999,"5min":5.4100000000000001,"15min":5.2779999999999996},"max":{"1min":14.51,"5min":180.44499999999999,"15min":21307.069},"last":7.04}]},{"osd":2,"last update":"Mon Jun 3 09:06:25 2024","interfaces":[{"interface":"back","average":{"1min":12.343999999999999,"5min":22.059999999999999,"15min":23.715},"min":{"1min":7.9199999999999999,"5min":6.4900000000000002,"15min":6.056},"max":{"1min":15.601000000000001,"5min":377.803,"15min":377.803},"last":12.581},{"interface":"front","average":{"1min":10.319000000000001,"5min":19.364999999999998,"15min":20.242999999999999},"min":{"1min":6.891,"5min":6.1749999999999998,"15min":5.9020000000000001},"max":{"1min":15.432,"5min":371.601,"15min":371.601},"last":10.368}]},{"osd":4,"last update":"Mon Jun 3 09:06:31 2024","interfaces":[{"interface":"back","average":{"1min":9.7460000000000004,"5min":891.23000000000002,"15min":416.48099999999999},"min":{"1min":5.6539999999999999,"5min":5.6539999999999999,"15min":5.6539999999999999},"max":{"1min":15.445,"5min":29040.635999999999,"15min":30144.939999999999},"last":8.0259999999999998},{"interface":"front","average":{"1min":12.398999999999999,"5min":897.58500000000004,"15min":421.27499999999998},"min":{"1min":8.9469999999999992,"5min":7.8109999999999999,"15min":6.2439999999999998},"max":{"1min":16.518000000000001,"5min":29076.955999999998,"15min":30180.689999999999},"last":9.1539999999999999}]},{"osd":5,"last update":"Mon Jun 3 09:06:07 2024","interfaces":[{"interface":"back","average":{"1min":12.802,"5min":19.640999999999998,"15min":22.318000000000001},"min":{"1min":9.0890000000000004,"5min":6.8810000000000002,"15min":6.7690000000000001},"max":{"1min":16.745999999999999,"5min":249.88,"15min":327.34899999999999},"last":14.225},{"interface":"front","average":{"1min":15.002000000000001,"5min":24.003,"15min":26.649999999999999},"min":{"1min":11.984,"5min":9.9149999999999991,"15min":8.9879999999999995},"max":{"1min":18.407,"5min":251.47800000000001,"15min":331.86799999999999},"last":14.91}]},{"osd":6,"last update":"Mon Jun 3 09:04:04 2024","interfaces":[{"interface":"back","average":{"1min":22.815999999999999,"5min":31.513999999999999,"15min":181.74299999999999},"min":{"1min":9.4809999999999999,"5min":7.2240000000000002,"15min":7.2240000000000002},"max":{"1min":160.56999999999999,"5min":380.613,"15min":11893.74},"last":19.359999999999999},{"interface":"front","average":{"1min":25.359999999999999,"5min":30.981000000000002,"15min":179.81399999999999},"min":{"1min":9.8870000000000005,"5min":7.3360000000000003,"15min":6.0899999999999999},"max":{"1min":193.11099999999999,"5min":379.80700000000002,"15min":11893.069},"last":8.202}]},{"osd":7,"last update":"Mon Jun 3 09:06:25 2024","interfaces":[{"interface":"back","average":{"1min":11.321999999999999,"5min":18.443999999999999,"15min":21.568999999999999},"min":{"1min":8.109,"5min":8.109,"15min":7.4279999999999999},"max":{"1min":17.113,"5min":146.267,"15min":178.76300000000001},"last":11.492000000000001},{"interface":"front","average":{"1min":12.512,"5min":20.065999999999999,"15min":24.190000000000001},"min":{"1min":7.4989999999999997,"5min":7.4989999999999997,"15min":7.4989999999999997},"max":{"1min":18.047999999999998,"5min":148.13800000000001,"15min":417.22800000000001},"last":13.419}]}]},{"osd":1,"up_from":14,"seq":60129542502,"num_pgs":6,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":59356,"kb_used_data":6096,"kb_used_omap":1,"kb_used_meta":53246,"kb_avail":94312484,"statfs":{"total":96636764160,"available":96575983616,"internally_reserved":0,"allocated":6242304,"data_stored":3138567,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":54524357},"hb_peers":[0,2,3,4,5,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Mon Jun 3 09:06:07 2024","interfaces":[{"interface":"back","average":{"1min":7.7530000000000001,"5min":185.66999999999999,"15min":77.347999999999999},"min":{"1min":5.0229999999999997,"5min":4.9279999999999999,"15min":4.5449999999999999},"max":{"1min":11.896000000000001,"5min":7470.616,"15min":7470.616},"last":10.537000000000001},{"interface":"front","average":{"1min":9.9220000000000006,"5min":187.14599999999999,"15min":76.325999999999993},"min":{"1min":5.3159999999999998,"5min":5.3159999999999998,"15min":5.3159999999999998},"max":{"1min":18.338000000000001,"5min":7477.0829999999996,"15min":7477.0829999999996},"last":9.5649999999999995}]},{"osd":2,"last update":"Mon Jun 3 09:06:20 2024","interfaces":[{"interface":"back","average":{"1min":10.340999999999999,"5min":17.369,"15min":23.670999999999999},"min":{"1min":6.9740000000000002,"5min":5.8650000000000002,"15min":5.8650000000000002},"max":{"1min":15.452,"5min":228,"15min":657.38},"last":11.122999999999999},{"interface":"front","average":{"1min":13.387,"5min":23.355,"15min":31.524000000000001},"min":{"1min":9.077,"5min":9.077,"15min":7.2160000000000002},"max":{"1min":19.701000000000001,"5min":229.465,"15min":700.06500000000005},"last":15.970000000000001}]},{"osd":3,"last update":"Mon Jun 3 09:06:04 2024","interfaces":[{"interface":"back","average":{"1min":8.6110000000000007,"5min":15.733000000000001,"15min":26.646999999999998},"min":{"1min":5.827,"5min":5.4640000000000004,"15min":5.4640000000000004},"max":{"1min":13.105,"5min":121.619,"15min":1676.009},"last":11.763},{"interface":"front","average":{"1min":9.5850000000000009,"5min":16.161999999999999,"15min":29.065999999999999},"min":{"1min":6.3630000000000004,"5min":5.7699999999999996,"15min":5.7699999999999996},"max":{"1min":15.166,"5min":132.036,"15min":1766.5840000000001},"last":8.7479999999999993}]},{"osd":4,"last update":"Mon Jun 3 09:05:31 2024","interfaces":[{"interface":"back","average":{"1min":10.996,"5min":954.46500000000003,"15min":1281.6859999999999},"min":{"1min":7.702,"5min":6.1760000000000002,"15min":6.0970000000000004},"max":{"1min":21.538,"5min":28044.862000000001,"15min":59438.017999999996},"last":9.3219999999999992},{"interface":"front","average":{"1min":13.646000000000001,"5min":961.24599999999998,"15min":1285.26},"min":{"1min":9.7200000000000006,"5min":9.3940000000000001,"15min":6.4640000000000004},"max":{"1min":23.277999999999999,"5min":28045.624,"15min":59439.283000000003},"last":12.798}]},{"osd":5,"last update":"Mon Jun 3 09:06:04 2024","interfaces":[{"interface":"back","average":{"1min":12.478999999999999,"5min":20.228000000000002,"15min":25.997},"min":{"1min":8.6430000000000007,"5min":7.9720000000000004,"15min":7.234},"max":{"1min":17.442,"5min":139.43199999999999,"15min":701.22000000000003},"last":13.678000000000001},{"interface":"front","average":{"1min":13.507999999999999,"5min":25.760999999999999,"15min":27.553999999999998},"min":{"1min":10.042999999999999,"5min":7.258,"15min":6.242},"max":{"1min":20.611000000000001,"5min":278.63,"15min":598.35000000000002},"last":14.173}]},{"osd":6,"last update":"Mon Jun 3 09:03:54 2024","interfaces":[{"interface":"back","average":{"1min":28.524000000000001,"5min":20.654,"15min":23.878},"min":{"1min":6.0309999999999997,"5min":6.0309999999999997,"15min":6.0309999999999997},"max":{"1min":167.221,"5min":167.221,"15min":704.245},"last":18.721},{"interface":"front","average":{"1min":27.606000000000002,"5min":30.131,"15min":28.628},"min":{"1min":9.8650000000000002,"5min":9.7200000000000006,"15min":8.3580000000000005},"max":{"1min":123.938,"5min":358.23000000000002,"15min":634.05499999999995},"last":173.113}]},{"osd":7,"last update":"Mon Jun 3 09:06:20 2024","interfaces":[{"interface":"back","average":{"1min":14.66,"5min":23.234999999999999,"15min":31.097000000000001},"min":{"1min":11.250999999999999,"5min":8.0559999999999992,"15min":8.0559999999999992},"max":{"1min":21.507999999999999,"5min":270.95100000000002,"15min":703.07799999999997},"last":15.116},{"interface":"front","average":{"1min":15.548,"5min":28.760000000000002,"15min":28.925000000000001},"min":{"1min":12.759,"5min":8.2089999999999996,"15min":7.7599999999999998},"max":{"1min":22.408000000000001,"5min":360.99900000000002,"15min":608.44100000000003},"last":16.640000000000001}]}]},{"osd":0,"up_from":14,"seq":60129542504,"num_pgs":10,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":81108,"kb_used_data":5704,"kb_used_omap":1,"kb_used_meta":75390,"kb_avail":94290732,"statfs":{"total":96636764160,"available":96553709568,"internally_reserved":0,"allocated":5840896,"data_stored":2740727,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1593,"internal_metadata":77199815},"hb_peers":[1,2,3,4,5,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[{"osd":1,"last update":"Mon Jun 3 09:06:04 2024","interfaces":[{"interface":"back","average":{"1min":6.9960000000000004,"5min":56.997,"15min":339.29700000000003},"min":{"1min":4.4530000000000003,"5min":4.4530000000000003,"15min":4.4100000000000001},"max":{"1min":11.720000000000001,"5min":2414.4749999999999,"15min":20691.453000000001},"last":5.7039999999999997},{"interface":"front","average":{"1min":9.548,"5min":57.258000000000003,"15min":337.791},"min":{"1min":5.2300000000000004,"5min":5.2300000000000004,"15min":4.6050000000000004},"max":{"1min":15.337999999999999,"5min":2412.9549999999999,"15min":20640.792000000001},"last":8.2159999999999993}]},{"osd":2,"last update":"Mon Jun 3 09:06:22 2024","interfaces":[{"interface":"back","average":{"1min":11.154,"5min":26.515999999999998,"15min":22.882000000000001},"min":{"1min":5.7869999999999999,"5min":5.766,"15min":5.766},"max":{"1min":72.269999999999996,"5min":627.40700000000004,"15min":627.40700000000004},"last":11.311},{"interface":"front","average":{"1min":14.353999999999999,"5min":33.304000000000002,"15min":29.885000000000002},"min":{"1min":8.2149999999999999,"5min":7.8630000000000004,"15min":7.4089999999999998},"max":{"1min":73.418000000000006,"5min":635.62599999999998,"15min":635.62599999999998},"last":13.371}]},{"osd":3,"last update":"Mon Jun 3 09:06:04 2024","interfaces":[{"interface":"back","average":{"1min":12.105,"5min":26.925000000000001,"15min":26.584},"min":{"1min":7.8410000000000002,"5min":7.1280000000000001,"15min":6.1909999999999998},"max":{"1min":18.425999999999998,"5min":389.72699999999998,"15min":389.72699999999998},"last":9.4990000000000006},{"interface":"front","average":{"1min":13.794,"5min":29.771999999999998,"15min":28.815000000000001},"min":{"1min":10.084,"5min":9.4920000000000009,"15min":6.1849999999999996},"max":{"1min":17.373999999999999,"5min":390.23200000000003,"15min":494.92899999999997},"last":10.595000000000001}]},{"osd":4,"last update":"Mon Jun 3 09:06:27 2024","interfaces":[{"interface":"back","average":{"1min":11.352,"5min":745.30600000000004,"15min":363.48099999999999},"min":{"1min":6.415,"5min":6.415,"15min":6.415},"max":{"1min":16.724,"5min":20248.616000000002,"15min":25555.212},"last":12.867000000000001},{"interface":"front","average":{"1min":13.545,"5min":744.76999999999998,"15min":361.03100000000001},"min":{"1min":9.3889999999999993,"5min":8.8940000000000001,"15min":7.1429999999999998},"max":{"1min":18.411000000000001,"5min":20250.195,"15min":25556.159},"last":14.318}]},{"osd":5,"last update":"Mon Jun 3 09:06:17 2024","interfaces":[{"interface":"back","average":{"1min":9.7040000000000006,"5min":26.533999999999999,"15min":28.683},"min":{"1min":6.8129999999999997,"5min":6.8129999999999997,"15min":6.8129999999999997},"max":{"1min":14.942,"5min":455.78699999999998,"15min":605.05899999999997},"last":6.6539999999999999},{"interface":"front","average":{"1min":14.279,"5min":28.398,"15min":26.192},"min":{"1min":11.696,"5min":6.4420000000000002,"15min":6.2030000000000003},"max":{"1min":17.879000000000001,"5min":391.31200000000001,"15min":541.11000000000001},"last":11.938000000000001}]},{"osd":6,"last update":"Mon Jun 3 09:04:30 2024","interfaces":[{"interface":"back","average":{"1min":52.496000000000002,"5min":33.139000000000003,"15min":160.83199999999999},"min":{"1min":7.5289999999999999,"5min":7.1040000000000001,"15min":6.3739999999999997},"max":{"1min":457.91699999999997,"5min":457.91699999999997,"15min":11115.258},"last":18.959},{"interface":"front","average":{"1min":49.670000000000002,"5min":33.704999999999998,"15min":162.791},"min":{"1min":12.247999999999999,"5min":11.019,"15min":9.9760000000000009},"max":{"1min":407.05099999999999,"5min":407.05099999999999,"15min":11092.194},"last":17.707000000000001}]},{"osd":7,"last update":"Mon Jun 3 09:06:22 2024","interfaces":[{"interface":"back","average":{"1min":11.625999999999999,"5min":26.375,"15min":28.972999999999999},"min":{"1min":6.4480000000000004,"5min":6.4480000000000004,"15min":6.4480000000000004},"max":{"1min":15.352,"5min":461.41699999999997,"15min":598.33699999999999},"last":7.1369999999999996},{"interface":"front","average":{"1min":14.013,"5min":29.696000000000002,"15min":33.630000000000003},"min":{"1min":10.938000000000001,"5min":10.58,"15min":10.263999999999999},"max":{"1min":16.603000000000002,"5min":463.10599999999999,"15min":592.80499999999995},"last":8.7810000000000006}]}]}],"pool_statfs":[{"poolid":1,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":401408,"data_stored":397840,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":5,"total":0,"available":0,"internally_reserved":0,"allocated":401408,"data_stored":397840,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":4,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":5,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":6,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":17,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":17,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":17,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":17,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":17,"osd":4,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":17,"osd":5,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":5,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":17,"osd":6,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":5,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":17,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0}]}} 2024-06-03T09:06:37.349 DEBUG:teuthology.orchestra.run.smithi088:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph pg dump --format=json 2024-06-03T09:06:37.449 INFO:teuthology.orchestra.run.smithi088.stderr:2024-06-03T09:06:37.449+0000 7f46c4a3b640 10 monclient: get_monmap_and_config 2024-06-03T09:06:37.449 INFO:teuthology.orchestra.run.smithi088.stderr:2024-06-03T09:06:37.449+0000 7f46c4a3b640 10 monclient: build_initial_monmap 2024-06-03T09:06:37.449 INFO:teuthology.orchestra.run.smithi088.stderr:2024-06-03T09:06:37.449+0000 7f46c4a3b640 10 monclient: monmap: 2024-06-03T09:06:37.449 INFO:teuthology.orchestra.run.smithi088.stderr:epoch 0 2024-06-03T09:06:37.449 INFO:teuthology.orchestra.run.smithi088.stderr:fsid ec3c3fdd-3ac6-4d7d-9412-683130ed8b04 2024-06-03T09:06:37.450 INFO:teuthology.orchestra.run.smithi088.stderr:last_changed 2024-06-03T09:06:37.450428+0000 2024-06-03T09:06:37.450 INFO:teuthology.orchestra.run.smithi088.stderr:created 2024-06-03T09:06:37.450428+0000 2024-06-03T09:06:37.450 INFO:teuthology.orchestra.run.smithi088.stderr:min_mon_release 0 (unknown) 2024-06-03T09:06:37.450 INFO:teuthology.orchestra.run.smithi088.stderr:election_strategy: 3 2024-06-03T09:06:37.450 INFO:teuthology.orchestra.run.smithi088.stderr:0: v1:172.21.15.88:6789/0 mon.noname-a 2024-06-03T09:06:37.450 INFO:teuthology.orchestra.run.smithi088.stderr:1: v1:172.21.15.192:6789/0 mon.noname-c 2024-06-03T09:06:37.450 INFO:teuthology.orchestra.run.smithi088.stderr:2: v1:172.21.15.88:6790/0 mon.noname-b 2024-06-03T09:06:37.450 INFO:teuthology.orchestra.run.smithi088.stderr: 2024-06-03T09:06:37.451 INFO:teuthology.orchestra.run.smithi088.stderr:2024-06-03T09:06:37.449+0000 7f46c4a3b640 1 Processor -- start 2024-06-03T09:06:37.451 INFO:teuthology.orchestra.run.smithi088.stderr:2024-06-03T09:06:37.450+0000 7f46c4a3b640 1 -- start start 2024-06-03T09:06:37.451 INFO:teuthology.orchestra.run.smithi088.stderr:2024-06-03T09:06:37.450+0000 7f46c4a3b640 10 monclient: init 2024-06-03T09:06:37.451 INFO:teuthology.orchestra.run.smithi088.stderr:2024-06-03T09:06:37.450+0000 7f46c4a3b640 10 monclient: _reopen_session rank -1 2024-06-03T09:06:37.452 INFO:teuthology.orchestra.run.smithi088.stderr:2024-06-03T09:06:37.450+0000 7f46c4a3b640 10 monclient: _add_conns ranks=[2,1,0] 2024-06-03T09:06:37.452 INFO:teuthology.orchestra.run.smithi088.stderr:2024-06-03T09:06:37.450+0000 7f46c4a3b640 10 monclient(hunting): picked mon.noname-b con 0x7f46c006e050 addr v1:172.21.15.88:6790/0 2024-06-03T09:06:37.452 INFO:teuthology.orchestra.run.smithi088.stderr:2024-06-03T09:06:37.450+0000 7f46c4a3b640 10 monclient(hunting): picked mon.noname-c con 0x7f46c006e430 addr v1:172.21.15.192:6789/0 2024-06-03T09:06:37.452 INFO:teuthology.orchestra.run.smithi088.stderr:2024-06-03T09:06:37.450+0000 7f46c4a3b640 10 monclient(hunting): picked mon.noname-a con 0x7f46c00717c0 addr v1:172.21.15.88:6789/0 2024-06-03T09:06:37.452 INFO:teuthology.orchestra.run.smithi088.stderr:2024-06-03T09:06:37.450+0000 7f46c4a3b640 1 -- --> v1:172.21.15.88:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f46c0070c10 con 0x7f46c00717c0 2024-06-03T09:06:37.452 INFO:teuthology.orchestra.run.smithi088.stderr:2024-06-03T09:06:37.450+0000 7f46c4a3b640 1 -- --> v1:172.21.15.192:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f46c0070230 con 0x7f46c006e430 2024-06-03T09:06:37.452 INFO:teuthology.orchestra.run.smithi088.stderr:2024-06-03T09:06:37.450+0000 7f46c4a3b640 1 -- --> v1:172.21.15.88:6790/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f46c0070510 con 0x7f46c006e050 2024-06-03T09:06:37.452 INFO:teuthology.orchestra.run.smithi088.stderr:2024-06-03T09:06:37.450+0000 7f46c4a3b640 10 monclient(hunting): _renew_subs 2024-06-03T09:06:37.452 INFO:teuthology.orchestra.run.smithi088.stderr:2024-06-03T09:06:37.450+0000 7f46c4a3b640 10 monclient(hunting): authenticate will time out at 3053.260720s 2024-06-03T09:06:37.452 INFO:teuthology.orchestra.run.smithi088.stderr:2024-06-03T09:06:37.451+0000 7f46bed9d640 1 --1- >> v1:172.21.15.88:6789/0 conn(0x7f46c00717c0 0x7f46c0149240 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.88:6789/0 says I am v1:172.21.15.88:54330/0 (socket says 172.21.15.88:54330) 2024-06-03T09:06:37.452 INFO:teuthology.orchestra.run.smithi088.stderr:2024-06-03T09:06:37.451+0000 7f46bed9d640 1 -- 172.21.15.88:0/156292764 learned_addr learned my addr 172.21.15.88:0/156292764 (peer_addr_for_me v1:172.21.15.88:0/0) 2024-06-03T09:06:37.453 INFO:teuthology.orchestra.run.smithi088.stderr:2024-06-03T09:06:37.453+0000 7f46bd59a640 1 -- 172.21.15.88:0/156292764 <== mon.1 v1:172.21.15.192:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 2073864519 0 0) 0x7f46c0070230 con 0x7f46c006e430 2024-06-03T09:06:37.453 INFO:teuthology.orchestra.run.smithi088.stderr:2024-06-03T09:06:37.453+0000 7f46bd59a640 10 monclient(hunting): _negotiate 2024-06-03T09:06:37.454 INFO:teuthology.orchestra.run.smithi088.stderr:2024-06-03T09:06:37.453+0000 7f46bd59a640 10 monclient(hunting): _init_auth method 2 2024-06-03T09:06:37.454 INFO:teuthology.orchestra.run.smithi088.stderr:2024-06-03T09:06:37.453+0000 7f46bd59a640 10 monclient(hunting): _init_auth creating new auth 2024-06-03T09:06:37.454 INFO:teuthology.orchestra.run.smithi088.stderr:2024-06-03T09:06:37.453+0000 7f46bd59a640 10 monclient(hunting): my global_id is 30235 2024-06-03T09:06:37.454 INFO:teuthology.orchestra.run.smithi088.stderr:2024-06-03T09:06:37.453+0000 7f46bd59a640 1 -- 172.21.15.88:0/156292764 --> v1:172.21.15.192:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f46ac003520 con 0x7f46c006e430 2024-06-03T09:06:37.454 INFO:teuthology.orchestra.run.smithi088.stderr:2024-06-03T09:06:37.453+0000 7f46bd59a640 1 -- 172.21.15.88:0/156292764 <== mon.2 v1:172.21.15.88:6790/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 552440223 0 0) 0x7f46c0070510 con 0x7f46c006e050 2024-06-03T09:06:37.454 INFO:teuthology.orchestra.run.smithi088.stderr:2024-06-03T09:06:37.453+0000 7f46bd59a640 10 monclient(hunting): _negotiate 2024-06-03T09:06:37.454 INFO:teuthology.orchestra.run.smithi088.stderr:2024-06-03T09:06:37.453+0000 7f46bd59a640 10 monclient(hunting): _init_auth method 2 2024-06-03T09:06:37.454 INFO:teuthology.orchestra.run.smithi088.stderr:2024-06-03T09:06:37.453+0000 7f46bd59a640 10 monclient(hunting): _init_auth creating new auth 2024-06-03T09:06:37.454 INFO:teuthology.orchestra.run.smithi088.stderr:2024-06-03T09:06:37.453+0000 7f46bd59a640 10 monclient(hunting): my global_id is 30158 2024-06-03T09:06:37.454 INFO:teuthology.orchestra.run.smithi088.stderr:2024-06-03T09:06:37.453+0000 7f46bd59a640 1 -- 172.21.15.88:0/156292764 --> v1:172.21.15.88:6790/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f46c0070230 con 0x7f46c006e050 2024-06-03T09:06:37.454 INFO:teuthology.orchestra.run.smithi088.stderr:2024-06-03T09:06:37.453+0000 7f46bd59a640 1 -- 172.21.15.88:0/156292764 <== mon.0 v1:172.21.15.88:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 2754808394 0 0) 0x7f46c0070c10 con 0x7f46c00717c0 2024-06-03T09:06:37.454 INFO:teuthology.orchestra.run.smithi088.stderr:2024-06-03T09:06:37.453+0000 7f46bd59a640 10 monclient(hunting): _negotiate 2024-06-03T09:06:37.455 INFO:teuthology.orchestra.run.smithi088.stderr:2024-06-03T09:06:37.453+0000 7f46bd59a640 10 monclient(hunting): _init_auth method 2 2024-06-03T09:06:37.455 INFO:teuthology.orchestra.run.smithi088.stderr:2024-06-03T09:06:37.453+0000 7f46bd59a640 10 monclient(hunting): _init_auth creating new auth 2024-06-03T09:06:37.455 INFO:teuthology.orchestra.run.smithi088.stderr:2024-06-03T09:06:37.453+0000 7f46bd59a640 10 monclient(hunting): my global_id is 30654 2024-06-03T09:06:37.455 INFO:teuthology.orchestra.run.smithi088.stderr:2024-06-03T09:06:37.453+0000 7f46bd59a640 1 -- 172.21.15.88:0/156292764 --> v1:172.21.15.88:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f46c0070510 con 0x7f46c00717c0 2024-06-03T09:06:37.457 INFO:teuthology.orchestra.run.smithi088.stderr:2024-06-03T09:06:37.456+0000 7f46bd59a640 1 -- 172.21.15.88:0/156292764 <== mon.2 v1:172.21.15.88:6790/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 214+0+0 (unknown 1463897183 0 0) 0x7f46c0070230 con 0x7f46c006e050 2024-06-03T09:06:37.457 INFO:teuthology.orchestra.run.smithi088.stderr:2024-06-03T09:06:37.456+0000 7f46bd59a640 1 -- 172.21.15.88:0/156292764 >> v1:172.21.15.192:6789/0 conn(0x7f46c006e430 legacy=0x7f46c00710b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-06-03T09:06:37.457 INFO:teuthology.orchestra.run.smithi088.stderr:2024-06-03T09:06:37.456+0000 7f46bd59a640 1 -- 172.21.15.88:0/156292764 >> v1:172.21.15.88:6789/0 conn(0x7f46c00717c0 legacy=0x7f46c0149240 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-06-03T09:06:37.457 INFO:teuthology.orchestra.run.smithi088.stderr:2024-06-03T09:06:37.456+0000 7f46bd59a640 10 monclient: _finish_hunting 0 2024-06-03T09:06:37.457 INFO:teuthology.orchestra.run.smithi088.stderr:2024-06-03T09:06:37.456+0000 7f46bd59a640 1 monclient: found mon.noname-b 2024-06-03T09:06:37.457 INFO:teuthology.orchestra.run.smithi088.stderr:2024-06-03T09:06:37.456+0000 7f46bd59a640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-06-03T09:06:37.457 INFO:teuthology.orchestra.run.smithi088.stderr:2024-06-03T09:06:37.456+0000 7f46bd59a640 10 monclient: _send_mon_message to mon.noname-b at v1:172.21.15.88:6790/0 2024-06-03T09:06:37.457 INFO:teuthology.orchestra.run.smithi088.stderr:2024-06-03T09:06:37.456+0000 7f46bd59a640 1 -- 172.21.15.88:0/156292764 --> v1:172.21.15.88:6790/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f46c00707f0 con 0x7f46c006e050 2024-06-03T09:06:37.457 INFO:teuthology.orchestra.run.smithi088.stderr:2024-06-03T09:06:37.456+0000 7f46bd59a640 10 monclient: _finish_auth 0 2024-06-03T09:06:37.457 INFO:teuthology.orchestra.run.smithi088.stderr:2024-06-03T09:06:37.456+0000 7f46bd59a640 10 monclient: _check_auth_tickets 2024-06-03T09:06:37.458 INFO:teuthology.orchestra.run.smithi088.stderr:2024-06-03T09:06:37.456+0000 7f46bd59a640 20 monclient: _check_auth_rotating not needed by client.admin 2024-06-03T09:06:37.458 INFO:teuthology.orchestra.run.smithi088.stderr:2024-06-03T09:06:37.456+0000 7f46bd59a640 1 -- 172.21.15.88:0/156292764 <== mon.2 v1:172.21.15.88:6790/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 2234374065 0 0) 0x7f46b80015f0 con 0x7f46c006e050 2024-06-03T09:06:37.458 INFO:teuthology.orchestra.run.smithi088.stderr:2024-06-03T09:06:37.456+0000 7f46c4a3b640 5 monclient: authenticate success, global_id 30158 2024-06-03T09:06:37.458 INFO:teuthology.orchestra.run.smithi088.stderr:2024-06-03T09:06:37.456+0000 7f46c4a3b640 20 monclient: get_monmap_and_config waiting for monmap|config 2024-06-03T09:06:37.458 INFO:teuthology.orchestra.run.smithi088.stderr:2024-06-03T09:06:37.456+0000 7f46bd59a640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-06-03T09:06:37.458 INFO:teuthology.orchestra.run.smithi088.stderr:2024-06-03T09:06:37.456+0000 7f46bd59a640 10 monclient: got monmap 1 from mon.noname-b (according to old e1) 2024-06-03T09:06:37.458 INFO:teuthology.orchestra.run.smithi088.stderr:2024-06-03T09:06:37.456+0000 7f46bd59a640 10 monclient: dump: 2024-06-03T09:06:37.458 INFO:teuthology.orchestra.run.smithi088.stderr:epoch 1 2024-06-03T09:06:37.458 INFO:teuthology.orchestra.run.smithi088.stderr:fsid ec3c3fdd-3ac6-4d7d-9412-683130ed8b04 2024-06-03T09:06:37.458 INFO:teuthology.orchestra.run.smithi088.stderr:last_changed 2024-06-03T08:34:31.885125+0000 2024-06-03T09:06:37.458 INFO:teuthology.orchestra.run.smithi088.stderr:created 2024-06-03T08:34:31.885125+0000 2024-06-03T09:06:37.458 INFO:teuthology.orchestra.run.smithi088.stderr:min_mon_release 18 (reef) 2024-06-03T09:06:37.459 INFO:teuthology.orchestra.run.smithi088.stderr:election_strategy: 3 2024-06-03T09:06:37.459 INFO:teuthology.orchestra.run.smithi088.stderr:0: v1:172.21.15.88:6789/0 mon.a 2024-06-03T09:06:37.459 INFO:teuthology.orchestra.run.smithi088.stderr:1: v1:172.21.15.192:6789/0 mon.b 2024-06-03T09:06:37.459 INFO:teuthology.orchestra.run.smithi088.stderr:2: v1:172.21.15.88:6790/0 mon.c 2024-06-03T09:06:37.459 INFO:teuthology.orchestra.run.smithi088.stderr: 2024-06-03T09:06:37.459 INFO:teuthology.orchestra.run.smithi088.stderr:2024-06-03T09:06:37.457+0000 7f46c4a3b640 20 monclient: get_monmap_and_config waiting for monmap|config 2024-06-03T09:06:37.462 INFO:teuthology.orchestra.run.smithi088.stderr:2024-06-03T09:06:37.458+0000 7f46bd59a640 1 -- 172.21.15.88:0/156292764 <== mon.2 v1:172.21.15.88:6790/0 4 ==== config(0 keys) v1 ==== 4+0+0 (unknown 0 0 0) 0x7f46b8001810 con 0x7f46c006e050 2024-06-03T09:06:37.462 INFO:teuthology.orchestra.run.smithi088.stderr:2024-06-03T09:06:37.458+0000 7f46bd59a640 10 monclient: handle_config config(0 keys) v1 2024-06-03T09:06:37.462 INFO:teuthology.orchestra.run.smithi088.stderr:2024-06-03T09:06:37.458+0000 7f46bd59a640 1 -- 172.21.15.88:0/156292764 <== mon.2 v1:172.21.15.88:6790/0 5 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 2234374065 0 0) 0x7f46b8001c20 con 0x7f46c006e050 2024-06-03T09:06:37.462 INFO:teuthology.orchestra.run.smithi088.stderr:2024-06-03T09:06:37.458+0000 7f46bd59a640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-06-03T09:06:37.462 INFO:teuthology.orchestra.run.smithi088.stderr:2024-06-03T09:06:37.458+0000 7f46bd59a640 10 monclient: got monmap 1 from mon.c (according to old e1) 2024-06-03T09:06:37.462 INFO:teuthology.orchestra.run.smithi088.stderr:2024-06-03T09:06:37.458+0000 7f46bd59a640 10 monclient: dump: 2024-06-03T09:06:37.462 INFO:teuthology.orchestra.run.smithi088.stderr:epoch 1 2024-06-03T09:06:37.463 INFO:teuthology.orchestra.run.smithi088.stderr:fsid ec3c3fdd-3ac6-4d7d-9412-683130ed8b04 2024-06-03T09:06:37.463 INFO:teuthology.orchestra.run.smithi088.stderr:last_changed 2024-06-03T08:34:31.885125+0000 2024-06-03T09:06:37.463 INFO:teuthology.orchestra.run.smithi088.stderr:created 2024-06-03T08:34:31.885125+0000 2024-06-03T09:06:37.463 INFO:teuthology.orchestra.run.smithi088.stderr:min_mon_release 18 (reef) 2024-06-03T09:06:37.463 INFO:teuthology.orchestra.run.smithi088.stderr:election_strategy: 3 2024-06-03T09:06:37.463 INFO:teuthology.orchestra.run.smithi088.stderr:0: v1:172.21.15.88:6789/0 mon.a 2024-06-03T09:06:37.463 INFO:teuthology.orchestra.run.smithi088.stderr:1: v1:172.21.15.192:6789/0 mon.b 2024-06-03T09:06:37.463 INFO:teuthology.orchestra.run.smithi088.stderr:2: v1:172.21.15.88:6790/0 mon.c 2024-06-03T09:06:37.463 INFO:teuthology.orchestra.run.smithi088.stderr: 2024-06-03T09:06:37.463 INFO:teuthology.orchestra.run.smithi088.stderr:2024-06-03T09:06:37.458+0000 7f46c4a3b640 10 monclient: get_monmap_and_config success 2024-06-03T09:06:37.463 INFO:teuthology.orchestra.run.smithi088.stderr:2024-06-03T09:06:37.458+0000 7f46c4a3b640 4 set_mon_vals no callback set 2024-06-03T09:06:37.463 INFO:teuthology.orchestra.run.smithi088.stderr:2024-06-03T09:06:37.459+0000 7f46c4a3b640 10 monclient: shutdown 2024-06-03T09:06:37.463 INFO:teuthology.orchestra.run.smithi088.stderr:2024-06-03T09:06:37.459+0000 7f46c4a3b640 20 monclient: shutdown discarding 0 pending message(s) 2024-06-03T09:06:37.463 INFO:teuthology.orchestra.run.smithi088.stderr:2024-06-03T09:06:37.459+0000 7f46c4a3b640 1 -- 172.21.15.88:0/156292764 >> v1:172.21.15.88:6790/0 conn(0x7f46c006e050 legacy=0x7f46c013ad00 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-06-03T09:06:37.464 INFO:teuthology.orchestra.run.smithi088.stderr:2024-06-03T09:06:37.459+0000 7f46c4a3b640 1 -- 172.21.15.88:0/156292764 shutdown_connections 2024-06-03T09:06:37.464 INFO:teuthology.orchestra.run.smithi088.stderr:2024-06-03T09:06:37.459+0000 7f46c4a3b640 1 -- 172.21.15.88:0/156292764 >> 172.21.15.88:0/156292764 conn(0x7f46c00b2f20 msgr2=0x7f46c00b3300 unknown :-1 s=STATE_NONE l=0).mark_down 2024-06-03T09:06:37.464 INFO:teuthology.orchestra.run.smithi088.stderr:2024-06-03T09:06:37.459+0000 7f46c4a3b640 1 -- 172.21.15.88:0/156292764 shutdown_connections 2024-06-03T09:06:37.464 INFO:teuthology.orchestra.run.smithi088.stderr:2024-06-03T09:06:37.459+0000 7f46c4a3b640 1 -- 172.21.15.88:0/156292764 wait complete. 2024-06-03T09:06:37.464 INFO:teuthology.orchestra.run.smithi088.stderr:2024-06-03T09:06:37.460+0000 7f46c4a3b640 10 monclient: build_initial_monmap 2024-06-03T09:06:37.464 INFO:teuthology.orchestra.run.smithi088.stderr:2024-06-03T09:06:37.460+0000 7f46c4a3b640 10 monclient: monmap: 2024-06-03T09:06:37.464 INFO:teuthology.orchestra.run.smithi088.stderr:epoch 0 2024-06-03T09:06:37.464 INFO:teuthology.orchestra.run.smithi088.stderr:fsid 00000000-0000-0000-0000-000000000000 2024-06-03T09:06:37.464 INFO:teuthology.orchestra.run.smithi088.stderr:last_changed 0.000000 2024-06-03T09:06:37.464 INFO:teuthology.orchestra.run.smithi088.stderr:created 0.000000 2024-06-03T09:06:37.464 INFO:teuthology.orchestra.run.smithi088.stderr:min_mon_release 0 (unknown) 2024-06-03T09:06:37.464 INFO:teuthology.orchestra.run.smithi088.stderr:election_strategy: 1 2024-06-03T09:06:37.464 INFO:teuthology.orchestra.run.smithi088.stderr:0: v1:172.21.15.88:6789/0 mon.noname-a 2024-06-03T09:06:37.464 INFO:teuthology.orchestra.run.smithi088.stderr:1: v1:172.21.15.192:6789/0 mon.noname-b 2024-06-03T09:06:37.464 INFO:teuthology.orchestra.run.smithi088.stderr:2: v1:172.21.15.88:6790/0 mon.noname-c 2024-06-03T09:06:37.465 INFO:teuthology.orchestra.run.smithi088.stderr: 2024-06-03T09:06:37.465 INFO:teuthology.orchestra.run.smithi088.stderr:2024-06-03T09:06:37.460+0000 7f46c4a3b640 1 librados: starting msgr at 2024-06-03T09:06:37.465 INFO:teuthology.orchestra.run.smithi088.stderr:2024-06-03T09:06:37.460+0000 7f46c4a3b640 1 librados: starting objecter 2024-06-03T09:06:37.465 INFO:teuthology.orchestra.run.smithi088.stderr:2024-06-03T09:06:37.460+0000 7f46c4a3b640 1 Processor -- start 2024-06-03T09:06:37.465 INFO:teuthology.orchestra.run.smithi088.stderr:2024-06-03T09:06:37.460+0000 7f46c4a3b640 1 -- start start 2024-06-03T09:06:37.465 INFO:teuthology.orchestra.run.smithi088.stderr:2024-06-03T09:06:37.460+0000 7f46c4a3b640 1 librados: setting wanted keys 2024-06-03T09:06:37.465 INFO:teuthology.orchestra.run.smithi088.stderr:2024-06-03T09:06:37.460+0000 7f46c4a3b640 1 librados: calling monclient init 2024-06-03T09:06:37.465 INFO:teuthology.orchestra.run.smithi088.stderr:2024-06-03T09:06:37.460+0000 7f46c4a3b640 10 monclient: init 2024-06-03T09:06:37.465 INFO:teuthology.orchestra.run.smithi088.stderr:2024-06-03T09:06:37.461+0000 7f46c4a3b640 10 monclient: _reopen_session rank -1 2024-06-03T09:06:37.465 INFO:teuthology.orchestra.run.smithi088.stderr:2024-06-03T09:06:37.461+0000 7f46c4a3b640 10 monclient: _add_conns ranks=[1,2,0] 2024-06-03T09:06:37.465 INFO:teuthology.orchestra.run.smithi088.stderr:2024-06-03T09:06:37.461+0000 7f46c4a3b640 10 monclient(hunting): picked mon.noname-b con 0x7f46c006e050 addr v1:172.21.15.192:6789/0 2024-06-03T09:06:37.465 INFO:teuthology.orchestra.run.smithi088.stderr:2024-06-03T09:06:37.461+0000 7f46c4a3b640 10 monclient(hunting): picked mon.noname-c con 0x7f46c00717c0 addr v1:172.21.15.88:6790/0 2024-06-03T09:06:37.465 INFO:teuthology.orchestra.run.smithi088.stderr:2024-06-03T09:06:37.461+0000 7f46c4a3b640 10 monclient(hunting): picked mon.noname-a con 0x7f46c012f7a0 addr v1:172.21.15.88:6789/0 2024-06-03T09:06:37.465 INFO:teuthology.orchestra.run.smithi088.stderr:2024-06-03T09:06:37.461+0000 7f46c4a3b640 1 -- --> v1:172.21.15.88:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f46c01370e0 con 0x7f46c012f7a0 2024-06-03T09:06:37.465 INFO:teuthology.orchestra.run.smithi088.stderr:2024-06-03T09:06:37.461+0000 7f46c4a3b640 1 -- --> v1:172.21.15.192:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f46c0139380 con 0x7f46c006e050 2024-06-03T09:06:37.465 INFO:teuthology.orchestra.run.smithi088.stderr:2024-06-03T09:06:37.461+0000 7f46c4a3b640 1 -- --> v1:172.21.15.88:6790/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f46c01400d0 con 0x7f46c00717c0 2024-06-03T09:06:37.466 INFO:teuthology.orchestra.run.smithi088.stderr:2024-06-03T09:06:37.461+0000 7f46c4a3b640 10 monclient(hunting): _renew_subs 2024-06-03T09:06:37.466 INFO:teuthology.orchestra.run.smithi088.stderr:2024-06-03T09:06:37.461+0000 7f46c4a3b640 10 monclient(hunting): authenticate will time out at 3053.271220s 2024-06-03T09:06:37.466 INFO:teuthology.orchestra.run.smithi088.stderr:2024-06-03T09:06:37.461+0000 7f46bed9d640 1 --1- >> v1:172.21.15.88:6789/0 conn(0x7f46c012f7a0 0x7f46c01dfd10 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=1).handle_server_banner_and_identify peer v1:172.21.15.88:6789/0 says I am v1:172.21.15.88:54340/0 (socket says 172.21.15.88:54340) 2024-06-03T09:06:37.466 INFO:teuthology.orchestra.run.smithi088.stderr:2024-06-03T09:06:37.461+0000 7f46bed9d640 1 -- 172.21.15.88:0/1643555911 learned_addr learned my addr 172.21.15.88:0/1643555911 (peer_addr_for_me v1:172.21.15.88:0/0) 2024-06-03T09:06:37.466 INFO:teuthology.orchestra.run.smithi088.stderr:2024-06-03T09:06:37.462+0000 7f46b6ffd640 10 client.?.objecter ms_handle_connect 0x7f46c012f7a0 2024-06-03T09:06:37.466 INFO:teuthology.orchestra.run.smithi088.stderr:2024-06-03T09:06:37.462+0000 7f46b6ffd640 10 client.?.objecter resend_mon_ops 2024-06-03T09:06:37.466 INFO:teuthology.orchestra.run.smithi088.stderr:2024-06-03T09:06:37.462+0000 7f46b6ffd640 10 client.?.objecter ms_handle_connect 0x7f46c00717c0 2024-06-03T09:06:37.466 INFO:teuthology.orchestra.run.smithi088.stderr:2024-06-03T09:06:37.462+0000 7f46b6ffd640 10 client.?.objecter resend_mon_ops 2024-06-03T09:06:37.466 INFO:teuthology.orchestra.run.smithi088.stderr:2024-06-03T09:06:37.462+0000 7f46b6ffd640 10 client.?.objecter ms_handle_connect 0x7f46c006e050 2024-06-03T09:06:37.466 INFO:teuthology.orchestra.run.smithi088.stderr:2024-06-03T09:06:37.462+0000 7f46b6ffd640 10 client.?.objecter resend_mon_ops 2024-06-03T09:06:37.467 INFO:teuthology.orchestra.run.smithi088.stderr:2024-06-03T09:06:37.463+0000 7f46b6ffd640 1 -- 172.21.15.88:0/1643555911 <== mon.1 v1:172.21.15.192:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 1548470946 0 0) 0x7f46c0139380 con 0x7f46c006e050 2024-06-03T09:06:37.467 INFO:teuthology.orchestra.run.smithi088.stderr:2024-06-03T09:06:37.463+0000 7f46b6ffd640 10 monclient(hunting): _negotiate 2024-06-03T09:06:37.467 INFO:teuthology.orchestra.run.smithi088.stderr:2024-06-03T09:06:37.463+0000 7f46b6ffd640 10 monclient(hunting): _init_auth method 2 2024-06-03T09:06:37.467 INFO:teuthology.orchestra.run.smithi088.stderr:2024-06-03T09:06:37.463+0000 7f46b6ffd640 10 monclient(hunting): _init_auth creating new auth 2024-06-03T09:06:37.467 INFO:teuthology.orchestra.run.smithi088.stderr:2024-06-03T09:06:37.463+0000 7f46b6ffd640 10 monclient(hunting): my global_id is 30238 2024-06-03T09:06:37.468 INFO:teuthology.orchestra.run.smithi088.stderr:2024-06-03T09:06:37.463+0000 7f46b6ffd640 1 -- 172.21.15.88:0/1643555911 --> v1:172.21.15.192:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f46a0004eb0 con 0x7f46c006e050 2024-06-03T09:06:37.468 INFO:teuthology.orchestra.run.smithi088.stderr:2024-06-03T09:06:37.464+0000 7f46b6ffd640 1 -- 172.21.15.88:0/1643555911 <== mon.0 v1:172.21.15.88:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 3872828562 0 0) 0x7f46c01370e0 con 0x7f46c012f7a0 2024-06-03T09:06:37.468 INFO:teuthology.orchestra.run.smithi088.stderr:2024-06-03T09:06:37.464+0000 7f46b6ffd640 10 monclient(hunting): _negotiate 2024-06-03T09:06:37.468 INFO:teuthology.orchestra.run.smithi088.stderr:2024-06-03T09:06:37.464+0000 7f46b6ffd640 10 monclient(hunting): _init_auth method 2 2024-06-03T09:06:37.468 INFO:teuthology.orchestra.run.smithi088.stderr:2024-06-03T09:06:37.464+0000 7f46b6ffd640 10 monclient(hunting): _init_auth creating new auth 2024-06-03T09:06:37.468 INFO:teuthology.orchestra.run.smithi088.stderr:2024-06-03T09:06:37.464+0000 7f46b6ffd640 10 monclient(hunting): my global_id is 30657 2024-06-03T09:06:37.468 INFO:teuthology.orchestra.run.smithi088.stderr:2024-06-03T09:06:37.464+0000 7f46b6ffd640 1 -- 172.21.15.88:0/1643555911 --> v1:172.21.15.88:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f46c0139380 con 0x7f46c012f7a0 2024-06-03T09:06:37.468 INFO:teuthology.orchestra.run.smithi088.stderr:2024-06-03T09:06:37.464+0000 7f46b6ffd640 1 -- 172.21.15.88:0/1643555911 <== mon.2 v1:172.21.15.88:6790/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 999969755 0 0) 0x7f46c01400d0 con 0x7f46c00717c0 2024-06-03T09:06:37.468 INFO:teuthology.orchestra.run.smithi088.stderr:2024-06-03T09:06:37.464+0000 7f46b6ffd640 10 monclient(hunting): _negotiate 2024-06-03T09:06:37.468 INFO:teuthology.orchestra.run.smithi088.stderr:2024-06-03T09:06:37.464+0000 7f46b6ffd640 10 monclient(hunting): _init_auth method 2 2024-06-03T09:06:37.468 INFO:teuthology.orchestra.run.smithi088.stderr:2024-06-03T09:06:37.464+0000 7f46b6ffd640 10 monclient(hunting): _init_auth creating new auth 2024-06-03T09:06:37.468 INFO:teuthology.orchestra.run.smithi088.stderr:2024-06-03T09:06:37.464+0000 7f46b6ffd640 10 monclient(hunting): my global_id is 30161 2024-06-03T09:06:37.468 INFO:teuthology.orchestra.run.smithi088.stderr:2024-06-03T09:06:37.464+0000 7f46b6ffd640 1 -- 172.21.15.88:0/1643555911 --> v1:172.21.15.88:6790/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f46c01370e0 con 0x7f46c00717c0 2024-06-03T09:06:37.469 INFO:teuthology.orchestra.run.smithi088.stderr:2024-06-03T09:06:37.468+0000 7f46b6ffd640 1 -- 172.21.15.88:0/1643555911 <== mon.1 v1:172.21.15.192:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 764+0+0 (unknown 2634643742 0 0) 0x7f46a0004eb0 con 0x7f46c006e050 2024-06-03T09:06:37.469 INFO:teuthology.orchestra.run.smithi088.stderr:2024-06-03T09:06:37.468+0000 7f46b6ffd640 1 -- 172.21.15.88:0/1643555911 --> v1:172.21.15.192:6789/0 -- auth(proto 2 165 bytes epoch 0) v1 -- 0x7f46c01400d0 con 0x7f46c006e050 2024-06-03T09:06:37.469 INFO:teuthology.orchestra.run.smithi088.stderr:2024-06-03T09:06:37.468+0000 7f46b6ffd640 1 -- 172.21.15.88:0/1643555911 <== mon.1 v1:172.21.15.192:6789/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 2234374065 0 0) 0x7f46b8001be0 con 0x7f46c006e050 2024-06-03T09:06:37.469 INFO:teuthology.orchestra.run.smithi088.stderr:2024-06-03T09:06:37.468+0000 7f46b6ffd640 10 monclient(hunting): handle_monmap mon_map magic: 0 v1 2024-06-03T09:06:37.469 INFO:teuthology.orchestra.run.smithi088.stderr:2024-06-03T09:06:37.468+0000 7f46b6ffd640 10 monclient(hunting): got monmap 1 from mon.noname-b (according to old e1) 2024-06-03T09:06:37.469 INFO:teuthology.orchestra.run.smithi088.stderr:2024-06-03T09:06:37.468+0000 7f46b6ffd640 10 monclient(hunting): dump: 2024-06-03T09:06:37.469 INFO:teuthology.orchestra.run.smithi088.stderr:epoch 1 2024-06-03T09:06:37.469 INFO:teuthology.orchestra.run.smithi088.stderr:fsid ec3c3fdd-3ac6-4d7d-9412-683130ed8b04 2024-06-03T09:06:37.469 INFO:teuthology.orchestra.run.smithi088.stderr:last_changed 2024-06-03T08:34:31.885125+0000 2024-06-03T09:06:37.469 INFO:teuthology.orchestra.run.smithi088.stderr:created 2024-06-03T08:34:31.885125+0000 2024-06-03T09:06:37.470 INFO:teuthology.orchestra.run.smithi088.stderr:min_mon_release 18 (reef) 2024-06-03T09:06:37.470 INFO:teuthology.orchestra.run.smithi088.stderr:election_strategy: 3 2024-06-03T09:06:37.470 INFO:teuthology.orchestra.run.smithi088.stderr:0: v1:172.21.15.88:6789/0 mon.a 2024-06-03T09:06:37.470 INFO:teuthology.orchestra.run.smithi088.stderr:1: v1:172.21.15.192:6789/0 mon.b 2024-06-03T09:06:37.470 INFO:teuthology.orchestra.run.smithi088.stderr:2: v1:172.21.15.88:6790/0 mon.c 2024-06-03T09:06:37.470 INFO:teuthology.orchestra.run.smithi088.stderr: 2024-06-03T09:06:37.470 INFO:teuthology.orchestra.run.smithi088.stderr:2024-06-03T09:06:37.468+0000 7f46b6ffd640 10 monclient(hunting): _finish_auth 0 2024-06-03T09:06:37.471 INFO:teuthology.orchestra.run.smithi088.stderr:2024-06-03T09:06:37.470+0000 7f46b6ffd640 1 -- 172.21.15.88:0/1643555911 <== mon.2 v1:172.21.15.88:6790/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 764+0+0 (unknown 4228243828 0 0) 0x7f46c01370e0 con 0x7f46c00717c0 2024-06-03T09:06:37.472 INFO:teuthology.orchestra.run.smithi088.stderr:2024-06-03T09:06:37.470+0000 7f46b6ffd640 1 -- 172.21.15.88:0/1643555911 --> v1:172.21.15.88:6790/0 -- auth(proto 2 165 bytes epoch 0) v1 -- 0x7f46a0004eb0 con 0x7f46c00717c0 2024-06-03T09:06:37.472 INFO:teuthology.orchestra.run.smithi088.stderr:2024-06-03T09:06:37.470+0000 7f46b6ffd640 1 -- 172.21.15.88:0/1643555911 <== mon.2 v1:172.21.15.88:6790/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 2234374065 0 0) 0x7f46a8001850 con 0x7f46c00717c0 2024-06-03T09:06:37.472 INFO:teuthology.orchestra.run.smithi088.stderr:2024-06-03T09:06:37.470+0000 7f46b6ffd640 10 monclient(hunting): handle_monmap mon_map magic: 0 v1 2024-06-03T09:06:37.472 INFO:teuthology.orchestra.run.smithi088.stderr:2024-06-03T09:06:37.470+0000 7f46b6ffd640 10 monclient(hunting): got monmap 1 from mon.c (according to old e1) 2024-06-03T09:06:37.472 INFO:teuthology.orchestra.run.smithi088.stderr:2024-06-03T09:06:37.470+0000 7f46b6ffd640 10 monclient(hunting): dump: 2024-06-03T09:06:37.472 INFO:teuthology.orchestra.run.smithi088.stderr:epoch 1 2024-06-03T09:06:37.472 INFO:teuthology.orchestra.run.smithi088.stderr:fsid ec3c3fdd-3ac6-4d7d-9412-683130ed8b04 2024-06-03T09:06:37.472 INFO:teuthology.orchestra.run.smithi088.stderr:last_changed 2024-06-03T08:34:31.885125+0000 2024-06-03T09:06:37.472 INFO:teuthology.orchestra.run.smithi088.stderr:created 2024-06-03T08:34:31.885125+0000 2024-06-03T09:06:37.472 INFO:teuthology.orchestra.run.smithi088.stderr:min_mon_release 18 (reef) 2024-06-03T09:06:37.472 INFO:teuthology.orchestra.run.smithi088.stderr:election_strategy: 3 2024-06-03T09:06:37.472 INFO:teuthology.orchestra.run.smithi088.stderr:0: v1:172.21.15.88:6789/0 mon.a 2024-06-03T09:06:37.473 INFO:teuthology.orchestra.run.smithi088.stderr:1: v1:172.21.15.192:6789/0 mon.b 2024-06-03T09:06:37.473 INFO:teuthology.orchestra.run.smithi088.stderr:2: v1:172.21.15.88:6790/0 mon.c 2024-06-03T09:06:37.473 INFO:teuthology.orchestra.run.smithi088.stderr: 2024-06-03T09:06:37.473 INFO:teuthology.orchestra.run.smithi088.stderr:2024-06-03T09:06:37.471+0000 7f46b6ffd640 1 -- 172.21.15.88:0/1643555911 <== mon.0 v1:172.21.15.88:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 764+0+0 (unknown 1184096815 0 0) 0x7f46c0139380 con 0x7f46c012f7a0 2024-06-03T09:06:37.473 INFO:teuthology.orchestra.run.smithi088.stderr:2024-06-03T09:06:37.471+0000 7f46b6ffd640 1 -- 172.21.15.88:0/1643555911 --> v1:172.21.15.88:6789/0 -- auth(proto 2 165 bytes epoch 0) v1 -- 0x7f46c01370e0 con 0x7f46c012f7a0 2024-06-03T09:06:37.473 INFO:teuthology.orchestra.run.smithi088.stderr:2024-06-03T09:06:37.471+0000 7f46b6ffd640 1 -- 172.21.15.88:0/1643555911 <== mon.0 v1:172.21.15.88:6789/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 2234374065 0 0) 0x7f46b0001b90 con 0x7f46c012f7a0 2024-06-03T09:06:37.473 INFO:teuthology.orchestra.run.smithi088.stderr:2024-06-03T09:06:37.471+0000 7f46b6ffd640 10 monclient(hunting): handle_monmap mon_map magic: 0 v1 2024-06-03T09:06:37.473 INFO:teuthology.orchestra.run.smithi088.stderr:2024-06-03T09:06:37.471+0000 7f46b6ffd640 10 monclient(hunting): got monmap 1 from mon.a (according to old e1) 2024-06-03T09:06:37.473 INFO:teuthology.orchestra.run.smithi088.stderr:2024-06-03T09:06:37.471+0000 7f46b6ffd640 10 monclient(hunting): dump: 2024-06-03T09:06:37.473 INFO:teuthology.orchestra.run.smithi088.stderr:epoch 1 2024-06-03T09:06:37.473 INFO:teuthology.orchestra.run.smithi088.stderr:fsid ec3c3fdd-3ac6-4d7d-9412-683130ed8b04 2024-06-03T09:06:37.473 INFO:teuthology.orchestra.run.smithi088.stderr:last_changed 2024-06-03T08:34:31.885125+0000 2024-06-03T09:06:37.473 INFO:teuthology.orchestra.run.smithi088.stderr:created 2024-06-03T08:34:31.885125+0000 2024-06-03T09:06:37.473 INFO:teuthology.orchestra.run.smithi088.stderr:min_mon_release 18 (reef) 2024-06-03T09:06:37.474 INFO:teuthology.orchestra.run.smithi088.stderr:election_strategy: 3 2024-06-03T09:06:37.474 INFO:teuthology.orchestra.run.smithi088.stderr:0: v1:172.21.15.88:6789/0 mon.a 2024-06-03T09:06:37.474 INFO:teuthology.orchestra.run.smithi088.stderr:1: v1:172.21.15.192:6789/0 mon.b 2024-06-03T09:06:37.474 INFO:teuthology.orchestra.run.smithi088.stderr:2: v1:172.21.15.88:6790/0 mon.c 2024-06-03T09:06:37.474 INFO:teuthology.orchestra.run.smithi088.stderr: 2024-06-03T09:06:37.474 INFO:teuthology.orchestra.run.smithi088.stderr:2024-06-03T09:06:37.471+0000 7f46b6ffd640 1 -- 172.21.15.88:0/1643555911 <== mon.1 v1:172.21.15.192:6789/0 4 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 580+0+0 (unknown 2887331104 0 0) 0x7f46c01400d0 con 0x7f46c006e050 2024-06-03T09:06:37.474 INFO:teuthology.orchestra.run.smithi088.stderr:2024-06-03T09:06:37.471+0000 7f46b6ffd640 1 -- 172.21.15.88:0/1643555911 >> v1:172.21.15.88:6790/0 conn(0x7f46c00717c0 legacy=0x7f46c01d79d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-06-03T09:06:37.474 INFO:teuthology.orchestra.run.smithi088.stderr:2024-06-03T09:06:37.471+0000 7f46b6ffd640 1 -- 172.21.15.88:0/1643555911 >> v1:172.21.15.88:6789/0 conn(0x7f46c012f7a0 legacy=0x7f46c01dfd10 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-06-03T09:06:37.474 INFO:teuthology.orchestra.run.smithi088.stderr:2024-06-03T09:06:37.471+0000 7f46b6ffd640 10 monclient: _finish_hunting 0 2024-06-03T09:06:37.474 INFO:teuthology.orchestra.run.smithi088.stderr:2024-06-03T09:06:37.471+0000 7f46b6ffd640 1 monclient: found mon.b 2024-06-03T09:06:37.474 INFO:teuthology.orchestra.run.smithi088.stderr:2024-06-03T09:06:37.471+0000 7f46b6ffd640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-06-03T09:06:37.474 INFO:teuthology.orchestra.run.smithi088.stderr:2024-06-03T09:06:37.471+0000 7f46b6ffd640 10 monclient: _send_mon_message to mon.b at v1:172.21.15.192:6789/0 2024-06-03T09:06:37.474 INFO:teuthology.orchestra.run.smithi088.stderr:2024-06-03T09:06:37.471+0000 7f46b6ffd640 1 -- 172.21.15.88:0/1643555911 --> v1:172.21.15.192:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f46c0137570 con 0x7f46c006e050 2024-06-03T09:06:37.475 INFO:teuthology.orchestra.run.smithi088.stderr:2024-06-03T09:06:37.471+0000 7f46b6ffd640 10 monclient: _finish_auth 0 2024-06-03T09:06:37.475 INFO:teuthology.orchestra.run.smithi088.stderr:2024-06-03T09:06:37.471+0000 7f46b6ffd640 10 monclient: _check_auth_tickets 2024-06-03T09:06:37.475 INFO:teuthology.orchestra.run.smithi088.stderr:2024-06-03T09:06:37.471+0000 7f46b6ffd640 20 monclient: _check_auth_rotating not needed by client.admin 2024-06-03T09:06:37.475 INFO:teuthology.orchestra.run.smithi088.stderr:2024-06-03T09:06:37.471+0000 7f46c4a3b640 5 monclient: authenticate success, global_id 30238 2024-06-03T09:06:37.475 INFO:teuthology.orchestra.run.smithi088.stderr:2024-06-03T09:06:37.471+0000 7f46c4a3b640 10 monclient: _renew_subs 2024-06-03T09:06:37.475 INFO:teuthology.orchestra.run.smithi088.stderr:2024-06-03T09:06:37.471+0000 7f46c4a3b640 10 monclient: _send_mon_message to mon.b at v1:172.21.15.192:6789/0 2024-06-03T09:06:37.475 INFO:teuthology.orchestra.run.smithi088.stderr:2024-06-03T09:06:37.471+0000 7f46c4a3b640 1 -- 172.21.15.88:0/1643555911 --> v1:172.21.15.192:6789/0 -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f46c01d80e0 con 0x7f46c006e050 2024-06-03T09:06:37.475 INFO:teuthology.orchestra.run.smithi088.stderr:2024-06-03T09:06:37.471+0000 7f46c4a3b640 10 client.30238.objecter _maybe_request_map subscribing (onetime) to next osd map 2024-06-03T09:06:37.475 INFO:teuthology.orchestra.run.smithi088.stderr:2024-06-03T09:06:37.471+0000 7f46c4a3b640 10 monclient: _renew_subs 2024-06-03T09:06:37.475 INFO:teuthology.orchestra.run.smithi088.stderr:2024-06-03T09:06:37.471+0000 7f46c4a3b640 10 monclient: _send_mon_message to mon.b at v1:172.21.15.192:6789/0 2024-06-03T09:06:37.475 INFO:teuthology.orchestra.run.smithi088.stderr:2024-06-03T09:06:37.471+0000 7f46c4a3b640 1 -- 172.21.15.88:0/1643555911 --> v1:172.21.15.192:6789/0 -- mon_subscribe({osdmap=0}) v3 -- 0x7f46c0131e40 con 0x7f46c006e050 2024-06-03T09:06:37.476 INFO:teuthology.orchestra.run.smithi088.stderr:2024-06-03T09:06:37.471+0000 7f46c4a3b640 1 librados: init done 2024-06-03T09:06:37.476 INFO:teuthology.orchestra.run.smithi088.stderr:2024-06-03T09:06:37.472+0000 7f46c4a3b640 10 start_mon_command cmd=[{"prefix": "get_command_descriptions"}] 2024-06-03T09:06:37.476 INFO:teuthology.orchestra.run.smithi088.stderr:2024-06-03T09:06:37.472+0000 7f46c4a3b640 10 monclient: _send_command 1 [{"prefix": "get_command_descriptions"}] 2024-06-03T09:06:37.476 INFO:teuthology.orchestra.run.smithi088.stderr:2024-06-03T09:06:37.472+0000 7f46c4a3b640 10 monclient: _send_mon_message to mon.b at v1:172.21.15.192:6789/0 2024-06-03T09:06:37.476 INFO:teuthology.orchestra.run.smithi088.stderr:2024-06-03T09:06:37.472+0000 7f46c4a3b640 1 -- 172.21.15.88:0/1643555911 --> v1:172.21.15.192:6789/0 -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f46c01391f0 con 0x7f46c006e050 2024-06-03T09:06:37.487 INFO:teuthology.orchestra.run.smithi088.stderr:2024-06-03T09:06:37.487+0000 7f46b6ffd640 1 -- 172.21.15.88:0/1643555911 <== mon.1 v1:172.21.15.192:6789/0 5 ==== config(0 keys) v1 ==== 4+0+0 (unknown 0 0 0) 0x7f46b80038a0 con 0x7f46c006e050 2024-06-03T09:06:37.487 INFO:teuthology.orchestra.run.smithi088.stderr:2024-06-03T09:06:37.487+0000 7f46b6ffd640 10 monclient: handle_config config(0 keys) v1 2024-06-03T09:06:37.487 INFO:teuthology.orchestra.run.smithi088.stderr:2024-06-03T09:06:37.487+0000 7f46b6ffd640 1 -- 172.21.15.88:0/1643555911 <== mon.1 v1:172.21.15.192:6789/0 6 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 2234374065 0 0) 0x7f46b8003bb0 con 0x7f46c006e050 2024-06-03T09:06:37.487 INFO:teuthology.orchestra.run.smithi088.stderr:2024-06-03T09:06:37.487+0000 7f46b6ffd640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-06-03T09:06:37.487 INFO:teuthology.orchestra.run.smithi088.stderr:2024-06-03T09:06:37.487+0000 7f46b6ffd640 10 monclient: got monmap 1 from mon.b (according to old e1) 2024-06-03T09:06:37.488 INFO:teuthology.orchestra.run.smithi088.stderr:2024-06-03T09:06:37.487+0000 7f46b6ffd640 10 monclient: dump: 2024-06-03T09:06:37.488 INFO:teuthology.orchestra.run.smithi088.stderr:epoch 1 2024-06-03T09:06:37.488 INFO:teuthology.orchestra.run.smithi088.stderr:fsid ec3c3fdd-3ac6-4d7d-9412-683130ed8b04 2024-06-03T09:06:37.488 INFO:teuthology.orchestra.run.smithi088.stderr:last_changed 2024-06-03T08:34:31.885125+0000 2024-06-03T09:06:37.488 INFO:teuthology.orchestra.run.smithi088.stderr:created 2024-06-03T08:34:31.885125+0000 2024-06-03T09:06:37.488 INFO:teuthology.orchestra.run.smithi088.stderr:min_mon_release 18 (reef) 2024-06-03T09:06:37.488 INFO:teuthology.orchestra.run.smithi088.stderr:election_strategy: 3 2024-06-03T09:06:37.488 INFO:teuthology.orchestra.run.smithi088.stderr:0: v1:172.21.15.88:6789/0 mon.a 2024-06-03T09:06:37.488 INFO:teuthology.orchestra.run.smithi088.stderr:1: v1:172.21.15.192:6789/0 mon.b 2024-06-03T09:06:37.488 INFO:teuthology.orchestra.run.smithi088.stderr:2: v1:172.21.15.88:6790/0 mon.c 2024-06-03T09:06:37.488 INFO:teuthology.orchestra.run.smithi088.stderr: 2024-06-03T09:06:37.488 INFO:teuthology.orchestra.run.smithi088.stderr:2024-06-03T09:06:37.487+0000 7f46bcd99640 4 set_mon_vals no callback set 2024-06-03T09:06:37.489 INFO:teuthology.orchestra.run.smithi088.stderr:2024-06-03T09:06:37.488+0000 7f46b6ffd640 1 -- 172.21.15.88:0/1643555911 <== mon.1 v1:172.21.15.192:6789/0 7 ==== mgrmap(e 5) v1 ==== 89110+0+0 (unknown 873533973 0 0) 0x7f46b8019930 con 0x7f46c006e050 2024-06-03T09:06:37.489 INFO:teuthology.orchestra.run.smithi088.stderr:2024-06-03T09:06:37.489+0000 7f46bdd9b640 10 monclient: get_auth_request con 0x7f46a006d3e0 auth_method 0 2024-06-03T09:06:37.489 INFO:teuthology.orchestra.run.smithi088.stderr:2024-06-03T09:06:37.489+0000 7f46b6ffd640 10 client.30238.objecter ms_handle_connect 0x7f46a006d3e0 2024-06-03T09:06:37.490 INFO:teuthology.orchestra.run.smithi088.stderr:2024-06-03T09:06:37.489+0000 7f46b6ffd640 1 -- 172.21.15.88:0/1643555911 <== mon.1 v1:172.21.15.192:6789/0 8 ==== osd_map(1010..1010 src has 622..1010) v4 ==== 5229+0+0 (unknown 1105883993 0 0) 0x7f46b8053080 con 0x7f46c006e050 2024-06-03T09:06:37.490 INFO:teuthology.orchestra.run.smithi088.stderr:2024-06-03T09:06:37.489+0000 7f46b6ffd640 10 client.30238.objecter ms_dispatch 0x7f46c0000b90 osd_map(1010..1010 src has 622..1010) v4 2024-06-03T09:06:37.490 INFO:teuthology.orchestra.run.smithi088.stderr:2024-06-03T09:06:37.489+0000 7f46b6ffd640 3 client.30238.objecter handle_osd_map got epochs [1010,1010] > 0 2024-06-03T09:06:37.490 INFO:teuthology.orchestra.run.smithi088.stderr:2024-06-03T09:06:37.489+0000 7f46b6ffd640 3 client.30238.objecter handle_osd_map decoding full epoch 1010 2024-06-03T09:06:37.490 INFO:teuthology.orchestra.run.smithi088.stderr:2024-06-03T09:06:37.490+0000 7f46b6ffd640 20 client.30238.objecter dump_active .. 0 homeless 2024-06-03T09:06:37.728 INFO:teuthology.orchestra.run.smithi088.stderr:2024-06-03T09:06:37.728+0000 7f46b6ffd640 1 -- 172.21.15.88:0/1643555911 <== mon.1 v1:172.21.15.192:6789/0 9 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+182498 (unknown 1092875540 0 3016760700) 0x7f46b804ea60 con 0x7f46c006e050 2024-06-03T09:06:37.729 INFO:teuthology.orchestra.run.smithi088.stderr:2024-06-03T09:06:37.728+0000 7f46b6ffd640 10 monclient: handle_mon_command_ack 1 [{"prefix": "get_command_descriptions"}] 2024-06-03T09:06:37.729 INFO:teuthology.orchestra.run.smithi088.stderr:2024-06-03T09:06:37.728+0000 7f46b6ffd640 10 monclient: _finish_command 1 = system:0 2024-06-03T09:06:37.898 INFO:teuthology.orchestra.run.smithi088.stderr:2024-06-03T09:06:37.898+0000 7f46c4a3b640 1 -- 172.21.15.88:0/1643555911 --> v1:172.21.15.192:6800/34229 -- mgr_command(tid 0: {"prefix": "pg dump", "target": ["mon-mgr", ""], "format": "json"}) v1 -- 0x7f46c0136eb0 con 0x7f46a006d3e0 2024-06-03T09:06:37.906 INFO:teuthology.orchestra.run.smithi088.stderr:2024-06-03T09:06:37.906+0000 7f46b6ffd640 1 -- 172.21.15.88:0/1643555911 <== mgr.4110 v1:172.21.15.192:6800/34229 1 ==== mgr_command_reply(tid 0: 0 dumped all) v1 ==== 18+0+151434 (unknown 2965378022 0 729038708) 0x7f46c0136eb0 con 0x7f46a006d3e0 2024-06-03T09:06:37.908 INFO:teuthology.orchestra.run.smithi088.stdout: 2024-06-03T09:06:37.909 INFO:teuthology.orchestra.run.smithi088.stderr:dumped all 2024-06-03T09:06:37.911 INFO:teuthology.orchestra.run.smithi088.stderr:2024-06-03T09:06:37.908+0000 7f46c4a3b640 10 librados: watch_flush enter 2024-06-03T09:06:37.911 INFO:teuthology.orchestra.run.smithi088.stderr:2024-06-03T09:06:37.909+0000 7f46c4a3b640 10 librados: watch_flush exit 2024-06-03T09:06:37.911 INFO:teuthology.orchestra.run.smithi088.stderr:2024-06-03T09:06:37.909+0000 7f46c4a3b640 20 client.30238.objecter shutdown clearing up homeless session... 2024-06-03T09:06:37.911 INFO:teuthology.orchestra.run.smithi088.stderr:2024-06-03T09:06:37.909+0000 7f46c4a3b640 10 client.30238.objecter successfully canceled tick 2024-06-03T09:06:37.911 INFO:teuthology.orchestra.run.smithi088.stderr:2024-06-03T09:06:37.909+0000 7f46c4a3b640 1 -- 172.21.15.88:0/1643555911 >> v1:172.21.15.192:6800/34229 conn(0x7f46a006d3e0 legacy=0x7f46a006f8a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-06-03T09:06:37.911 INFO:teuthology.orchestra.run.smithi088.stderr:2024-06-03T09:06:37.909+0000 7f46c4a3b640 10 monclient: shutdown 2024-06-03T09:06:37.911 INFO:teuthology.orchestra.run.smithi088.stderr:2024-06-03T09:06:37.909+0000 7f46c4a3b640 20 monclient: shutdown discarding 0 pending message(s) 2024-06-03T09:06:37.911 INFO:teuthology.orchestra.run.smithi088.stderr:2024-06-03T09:06:37.909+0000 7f46c4a3b640 1 -- 172.21.15.88:0/1643555911 >> v1:172.21.15.192:6789/0 conn(0x7f46c006e050 legacy=0x7f46c012f090 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-06-03T09:06:37.911 INFO:teuthology.orchestra.run.smithi088.stderr:2024-06-03T09:06:37.909+0000 7f46c4a3b640 1 -- 172.21.15.88:0/1643555911 shutdown_connections 2024-06-03T09:06:37.911 INFO:teuthology.orchestra.run.smithi088.stderr:2024-06-03T09:06:37.909+0000 7f46c4a3b640 1 -- 172.21.15.88:0/1643555911 >> 172.21.15.88:0/1643555911 conn(0x7f46c00b2f20 msgr2=0x7f46c00b0170 unknown :-1 s=STATE_NONE l=0).mark_down 2024-06-03T09:06:37.911 INFO:teuthology.orchestra.run.smithi088.stderr:2024-06-03T09:06:37.909+0000 7f46c4a3b640 1 -- 172.21.15.88:0/1643555911 shutdown_connections 2024-06-03T09:06:37.911 INFO:teuthology.orchestra.run.smithi088.stderr:2024-06-03T09:06:37.909+0000 7f46c4a3b640 1 -- 172.21.15.88:0/1643555911 wait complete. 2024-06-03T09:06:37.912 INFO:teuthology.orchestra.run.smithi088.stderr:2024-06-03T09:06:37.909+0000 7f46c4a3b640 1 librados: shutdown 2024-06-03T09:06:37.920 INFO:teuthology.orchestra.run.smithi088.stdout:{"pg_ready":true,"pg_map":{"version":1924,"stamp":"2024-06-03T09:06:37.101775+0000","last_osdmap_epoch":0,"last_pg_scan":0,"pg_stats_sum":{"stat_sum":{"num_bytes":459285,"num_objects":3,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":1,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":3,"num_whiteouts":0,"num_read":102,"num_read_kb":88,"num_write":120,"num_write_kb":1379,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2,"num_bytes_recovered":397840,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":83,"ondisk_log_size":83,"up":68,"acting":68,"num_store_stats":0},"osd_stats_sum":{"up_from":0,"seq":0,"num_pgs":100,"num_osds":8,"num_per_pool_osds":8,"num_per_pool_omap_osds":8,"kb":754974720,"kb_used":494548,"kb_used_data":46324,"kb_used_omap":12,"kb_used_meta":448115,"kb_avail":754480172,"statfs":{"total":773094113280,"available":772587696128,"internally_reserved":0,"allocated":47435776,"data_stored":22747954,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":12744,"internal_metadata":458870328},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":109,"apply_latency_ms":109,"commit_latency_ns":109000000,"apply_latency_ns":109000000},"alerts":[],"network_ping_times":[]},"pg_stats_delta":{"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":0,"acting":0,"num_store_stats":0,"stamp_delta":"12.002325"},"pg_stats":[{"pgid":"17.d","version":"0'0","reported_seq":942,"reported_epoch":1009,"state":"active+clean","last_fresh":"2024-06-03T09:06:14.443917+0000","last_change":"2024-06-03T08:38:19.260548+0000","last_active":"2024-06-03T09:06:14.443917+0000","last_peered":"2024-06-03T09:06:14.443917+0000","last_clean":"2024-06-03T09:06:14.443917+0000","last_became_active":"2024-06-03T08:38:19.253559+0000","last_became_peered":"2024-06-03T08:38:19.253559+0000","last_unstale":"2024-06-03T09:06:14.443917+0000","last_undegraded":"2024-06-03T09:06:14.443917+0000","last_fullsized":"2024-06-03T09:06:14.443917+0000","mapping_epoch":28,"log_start":"0'0","ondisk_log_start":"0'0","created":28,"last_epoch_clean":45,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-06-03T08:37:55.614497+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-06-03T08:37:55.614497+0000","last_clean_scrub_stamp":"2024-06-03T08:37:55.614497+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-06-04T19:00:12.470979+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1],"acting":[0,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"17.c","version":"0'0","reported_seq":953,"reported_epoch":1009,"state":"active+undersized","last_fresh":"2024-06-03T09:06:13.668156+0000","last_change":"2024-06-03T09:06:03.440074+0000","last_active":"2024-06-03T09:06:13.668156+0000","last_peered":"2024-06-03T09:06:13.668156+0000","last_clean":"2024-06-03T09:05:02.105638+0000","last_became_active":"2024-06-03T09:06:03.440074+0000","last_became_peered":"2024-06-03T09:06:03.440074+0000","last_unstale":"2024-06-03T09:06:13.668156+0000","last_undegraded":"2024-06-03T09:06:13.668156+0000","last_fullsized":"2024-06-03T09:06:03.377982+0000","mapping_epoch":999,"log_start":"0'0","ondisk_log_start":"0'0","created":28,"last_epoch_clean":255,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-06-03T08:37:55.614497+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-06-03T08:37:55.614497+0000","last_clean_scrub_stamp":"2024-06-03T08:37:55.614497+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-06-04T10:34:20.808557+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3],"acting":[3],"avail_no_missing":["3"],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"17.f","version":"0'0","reported_seq":148,"reported_epoch":1009,"state":"active+undersized","last_fresh":"2024-06-03T09:06:14.450543+0000","last_change":"2024-06-03T09:06:03.420285+0000","last_active":"2024-06-03T09:06:14.450543+0000","last_peered":"2024-06-03T09:06:14.450543+0000","last_clean":"2024-06-03T08:37:55.614497+0000","last_became_active":"2024-06-03T09:06:03.420285+0000","last_became_peered":"2024-06-03T09:06:03.420285+0000","last_unstale":"2024-06-03T09:06:14.450543+0000","last_undegraded":"2024-06-03T09:06:14.450543+0000","last_fullsized":"2024-06-03T09:06:03.380616+0000","mapping_epoch":999,"log_start":"0'0","ondisk_log_start":"0'0","created":28,"last_epoch_clean":255,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-06-03T08:37:55.614497+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-06-03T08:37:55.614497+0000","last_clean_scrub_stamp":"2024-06-03T08:37:55.614497+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-06-04T15:39:41.187731+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4],"acting":[4],"avail_no_missing":["4"],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"17.e","version":"0'0","reported_seq":149,"reported_epoch":1009,"state":"active+undersized","last_fresh":"2024-06-03T09:06:13.726992+0000","last_change":"2024-06-03T09:06:03.412887+0000","last_active":"2024-06-03T09:06:13.726992+0000","last_peered":"2024-06-03T09:06:13.726992+0000","last_clean":"2024-06-03T08:37:55.614497+0000","last_became_active":"2024-06-03T09:06:03.412887+0000","last_became_peered":"2024-06-03T09:06:03.412887+0000","last_unstale":"2024-06-03T09:06:13.726992+0000","last_undegraded":"2024-06-03T09:06:13.726992+0000","last_fullsized":"2024-06-03T09:06:03.332482+0000","mapping_epoch":999,"log_start":"0'0","ondisk_log_start":"0'0","created":28,"last_epoch_clean":255,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-06-03T08:37:55.614497+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-06-03T08:37:55.614497+0000","last_clean_scrub_stamp":"2024-06-03T08:37:55.614497+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-06-04T12:23:22.511601+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5],"acting":[5],"avail_no_missing":["5"],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"17.9","version":"0'0","reported_seq":946,"reported_epoch":1009,"state":"active+clean","last_fresh":"2024-06-03T09:06:14.999299+0000","last_change":"2024-06-03T08:38:04.772260+0000","last_active":"2024-06-03T09:06:14.999299+0000","last_peered":"2024-06-03T09:06:14.999299+0000","last_clean":"2024-06-03T09:06:14.999299+0000","last_became_active":"2024-06-03T08:38:04.763428+0000","last_became_peered":"2024-06-03T08:38:04.763428+0000","last_unstale":"2024-06-03T09:06:14.999299+0000","last_undegraded":"2024-06-03T09:06:14.999299+0000","last_fullsized":"2024-06-03T09:06:14.999299+0000","mapping_epoch":28,"log_start":"0'0","ondisk_log_start":"0'0","created":28,"last_epoch_clean":33,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-06-03T08:37:55.614497+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-06-03T08:37:55.614497+0000","last_clean_scrub_stamp":"2024-06-03T08:37:55.614497+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-06-04T10:55:10.773936+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,4],"acting":[7,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"17.8","version":"0'0","reported_seq":961,"reported_epoch":1009,"state":"active+undersized","last_fresh":"2024-06-03T09:06:14.430375+0000","last_change":"2024-06-03T09:06:03.388786+0000","last_active":"2024-06-03T09:06:14.430375+0000","last_peered":"2024-06-03T09:06:14.430375+0000","last_clean":"2024-06-03T09:04:43.297780+0000","last_became_active":"2024-06-03T09:06:03.388786+0000","last_became_peered":"2024-06-03T09:06:03.388786+0000","last_unstale":"2024-06-03T09:06:14.430375+0000","last_undegraded":"2024-06-03T09:06:14.430375+0000","last_fullsized":"2024-06-03T09:06:03.331754+0000","mapping_epoch":999,"log_start":"0'0","ondisk_log_start":"0'0","created":28,"last_epoch_clean":255,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-06-03T08:37:55.614497+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-06-03T08:37:55.614497+0000","last_clean_scrub_stamp":"2024-06-03T08:37:55.614497+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-06-04T09:18:44.765547+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0],"acting":[0],"avail_no_missing":["0"],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"17.b","version":"0'0","reported_seq":149,"reported_epoch":1009,"state":"active+undersized","last_fresh":"2024-06-03T09:06:14.433735+0000","last_change":"2024-06-03T09:06:03.451203+0000","last_active":"2024-06-03T09:06:14.433735+0000","last_peered":"2024-06-03T09:06:14.433735+0000","last_clean":"2024-06-03T08:37:55.614497+0000","last_became_active":"2024-06-03T09:06:03.451203+0000","last_became_peered":"2024-06-03T09:06:03.451203+0000","last_unstale":"2024-06-03T09:06:14.433735+0000","last_undegraded":"2024-06-03T09:06:14.433735+0000","last_fullsized":"2024-06-03T09:06:03.375523+0000","mapping_epoch":999,"log_start":"0'0","ondisk_log_start":"0'0","created":28,"last_epoch_clean":255,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-06-03T08:37:55.614497+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-06-03T08:37:55.614497+0000","last_clean_scrub_stamp":"2024-06-03T08:37:55.614497+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-06-04T10:16:19.813495+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2],"acting":[2],"avail_no_missing":["2"],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"17.a","version":"0'0","reported_seq":925,"reported_epoch":1010,"state":"active+clean","last_fresh":"2024-06-03T09:06:18.651697+0000","last_change":"2024-06-03T08:38:05.843142+0000","last_active":"2024-06-03T09:06:18.651697+0000","last_peered":"2024-06-03T09:06:18.651697+0000","last_clean":"2024-06-03T09:06:18.651697+0000","last_became_active":"2024-06-03T08:38:05.809257+0000","last_became_peered":"2024-06-03T08:38:05.809257+0000","last_unstale":"2024-06-03T09:06:18.651697+0000","last_undegraded":"2024-06-03T09:06:18.651697+0000","last_fullsized":"2024-06-03T09:06:18.651697+0000","mapping_epoch":28,"log_start":"0'0","ondisk_log_start":"0'0","created":28,"last_epoch_clean":33,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-06-03T08:37:55.614497+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-06-03T08:37:55.614497+0000","last_clean_scrub_stamp":"2024-06-03T08:37:55.614497+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-06-04T12:12:03.686838+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2],"acting":[1,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"17.5","version":"0'0","reported_seq":943,"reported_epoch":1009,"state":"active+clean","last_fresh":"2024-06-03T09:06:14.442357+0000","last_change":"2024-06-03T08:38:23.466051+0000","last_active":"2024-06-03T09:06:14.442357+0000","last_peered":"2024-06-03T09:06:14.442357+0000","last_clean":"2024-06-03T09:06:14.442357+0000","last_became_active":"2024-06-03T08:38:23.458039+0000","last_became_peered":"2024-06-03T08:38:23.458039+0000","last_unstale":"2024-06-03T09:06:14.442357+0000","last_undegraded":"2024-06-03T09:06:14.442357+0000","last_fullsized":"2024-06-03T09:06:14.442357+0000","mapping_epoch":28,"log_start":"0'0","ondisk_log_start":"0'0","created":28,"last_epoch_clean":49,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-06-03T08:37:55.614497+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-06-03T08:37:55.614497+0000","last_clean_scrub_stamp":"2024-06-03T08:37:55.614497+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-06-04T12:27:02.532436+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,4],"acting":[0,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"17.4","version":"0'0","reported_seq":149,"reported_epoch":1009,"state":"active+undersized","last_fresh":"2024-06-03T09:06:13.941522+0000","last_change":"2024-06-03T09:06:03.441889+0000","last_active":"2024-06-03T09:06:13.941522+0000","last_peered":"2024-06-03T09:06:13.941522+0000","last_clean":"2024-06-03T08:37:55.614497+0000","last_became_active":"2024-06-03T09:06:03.441889+0000","last_became_peered":"2024-06-03T09:06:03.441889+0000","last_unstale":"2024-06-03T09:06:13.941522+0000","last_undegraded":"2024-06-03T09:06:13.941522+0000","last_fullsized":"2024-06-03T09:06:03.359582+0000","mapping_epoch":999,"log_start":"0'0","ondisk_log_start":"0'0","created":28,"last_epoch_clean":255,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-06-03T08:37:55.614497+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-06-03T08:37:55.614497+0000","last_clean_scrub_stamp":"2024-06-03T08:37:55.614497+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-06-04T12:49:22.907141+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7],"acting":[7],"avail_no_missing":["7"],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"17.7","version":"0'0","reported_seq":929,"reported_epoch":1009,"state":"active+clean","last_fresh":"2024-06-03T09:06:13.736920+0000","last_change":"2024-06-03T08:38:04.837083+0000","last_active":"2024-06-03T09:06:13.736920+0000","last_peered":"2024-06-03T09:06:13.736920+0000","last_clean":"2024-06-03T09:06:13.736920+0000","last_became_active":"2024-06-03T08:38:04.818246+0000","last_became_peered":"2024-06-03T08:38:04.818246+0000","last_unstale":"2024-06-03T09:06:13.736920+0000","last_undegraded":"2024-06-03T09:06:13.736920+0000","last_fullsized":"2024-06-03T09:06:13.736920+0000","mapping_epoch":28,"log_start":"0'0","ondisk_log_start":"0'0","created":28,"last_epoch_clean":34,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-06-03T08:37:55.614497+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-06-03T08:37:55.614497+0000","last_clean_scrub_stamp":"2024-06-03T08:37:55.614497+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-06-04T17:02:38.204813+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,2],"acting":[5,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"17.6","version":"0'0","reported_seq":960,"reported_epoch":1009,"state":"active+undersized","last_fresh":"2024-06-03T09:06:14.453110+0000","last_change":"2024-06-03T09:06:03.428677+0000","last_active":"2024-06-03T09:06:14.453110+0000","last_peered":"2024-06-03T09:06:14.453110+0000","last_clean":"2024-06-03T09:04:43.312352+0000","last_became_active":"2024-06-03T09:06:03.428677+0000","last_became_peered":"2024-06-03T09:06:03.428677+0000","last_unstale":"2024-06-03T09:06:14.453110+0000","last_undegraded":"2024-06-03T09:06:14.453110+0000","last_fullsized":"2024-06-03T09:06:03.359909+0000","mapping_epoch":999,"log_start":"0'0","ondisk_log_start":"0'0","created":28,"last_epoch_clean":255,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-06-03T08:37:55.614497+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-06-03T08:37:55.614497+0000","last_clean_scrub_stamp":"2024-06-03T08:37:55.614497+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-06-04T15:38:02.180241+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0],"acting":[0],"avail_no_missing":["0"],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"2.7","version":"0'0","reported_seq":940,"reported_epoch":1009,"state":"active+clean","last_fresh":"2024-06-03T09:06:13.684065+0000","last_change":"2024-06-03T08:36:30.385405+0000","last_active":"2024-06-03T09:06:13.684065+0000","last_peered":"2024-06-03T09:06:13.684065+0000","last_clean":"2024-06-03T09:06:13.684065+0000","last_became_active":"2024-06-03T08:36:30.380302+0000","last_became_peered":"2024-06-03T08:36:30.380302+0000","last_unstale":"2024-06-03T09:06:13.684065+0000","last_undegraded":"2024-06-03T09:06:13.684065+0000","last_fullsized":"2024-06-03T09:06:13.684065+0000","mapping_epoch":23,"log_start":"0'0","ondisk_log_start":"0'0","created":23,"last_epoch_clean":24,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-06-03T08:36:28.444764+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-06-03T08:36:28.444764+0000","last_clean_scrub_stamp":"2024-06-03T08:36:28.444764+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-06-04T18:46:33.475181+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0],"acting":[3,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"17.14","version":"0'0","reported_seq":926,"reported_epoch":1009,"state":"active+clean","last_fresh":"2024-06-03T09:06:13.721460+0000","last_change":"2024-06-03T08:38:04.547565+0000","last_active":"2024-06-03T09:06:13.721460+0000","last_peered":"2024-06-03T09:06:13.721460+0000","last_clean":"2024-06-03T09:06:13.721460+0000","last_became_active":"2024-06-03T08:38:04.541781+0000","last_became_peered":"2024-06-03T08:38:04.541781+0000","last_unstale":"2024-06-03T09:06:13.721460+0000","last_undegraded":"2024-06-03T09:06:13.721460+0000","last_fullsized":"2024-06-03T09:06:13.721460+0000","mapping_epoch":28,"log_start":"0'0","ondisk_log_start":"0'0","created":28,"last_epoch_clean":34,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-06-03T08:37:55.614497+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-06-03T08:37:55.614497+0000","last_clean_scrub_stamp":"2024-06-03T08:37:55.614497+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-06-04T16:41:55.170134+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,2],"acting":[5,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"2.6","version":"0'0","reported_seq":949,"reported_epoch":1009,"state":"active+clean","last_fresh":"2024-06-03T09:06:14.448100+0000","last_change":"2024-06-03T08:36:30.430760+0000","last_active":"2024-06-03T09:06:14.448100+0000","last_peered":"2024-06-03T09:06:14.448100+0000","last_clean":"2024-06-03T09:06:14.448100+0000","last_became_active":"2024-06-03T08:36:30.404665+0000","last_became_peered":"2024-06-03T08:36:30.404665+0000","last_unstale":"2024-06-03T09:06:14.448100+0000","last_undegraded":"2024-06-03T09:06:14.448100+0000","last_fullsized":"2024-06-03T09:06:14.448100+0000","mapping_epoch":23,"log_start":"0'0","ondisk_log_start":"0'0","created":23,"last_epoch_clean":24,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-06-03T08:36:28.444764+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-06-03T08:36:28.444764+0000","last_clean_scrub_stamp":"2024-06-03T08:36:28.444764+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-06-04T10:37:37.746074+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,0],"acting":[4,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"17.15","version":"0'0","reported_seq":934,"reported_epoch":1009,"state":"active+clean","last_fresh":"2024-06-03T09:06:13.672940+0000","last_change":"2024-06-03T08:38:20.999903+0000","last_active":"2024-06-03T09:06:13.672940+0000","last_peered":"2024-06-03T09:06:13.672940+0000","last_clean":"2024-06-03T09:06:13.672940+0000","last_became_active":"2024-06-03T08:38:20.979523+0000","last_became_peered":"2024-06-03T08:38:20.979523+0000","last_unstale":"2024-06-03T09:06:13.672940+0000","last_undegraded":"2024-06-03T09:06:13.672940+0000","last_fullsized":"2024-06-03T09:06:13.672940+0000","mapping_epoch":28,"log_start":"0'0","ondisk_log_start":"0'0","created":28,"last_epoch_clean":47,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-06-03T08:37:55.614497+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-06-03T08:37:55.614497+0000","last_clean_scrub_stamp":"2024-06-03T08:37:55.614497+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-06-04T20:29:58.314289+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,1],"acting":[3,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"2.5","version":"0'0","reported_seq":942,"reported_epoch":1010,"state":"active+clean","last_fresh":"2024-06-03T09:06:18.382767+0000","last_change":"2024-06-03T08:36:30.024235+0000","last_active":"2024-06-03T09:06:18.382767+0000","last_peered":"2024-06-03T09:06:18.382767+0000","last_clean":"2024-06-03T09:06:18.382767+0000","last_became_active":"2024-06-03T08:36:30.019040+0000","last_became_peered":"2024-06-03T08:36:30.019040+0000","last_unstale":"2024-06-03T09:06:18.382767+0000","last_undegraded":"2024-06-03T09:06:18.382767+0000","last_fullsized":"2024-06-03T09:06:18.382767+0000","mapping_epoch":23,"log_start":"0'0","ondisk_log_start":"0'0","created":23,"last_epoch_clean":24,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-06-03T08:36:28.444764+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-06-03T08:36:28.444764+0000","last_clean_scrub_stamp":"2024-06-03T08:36:28.444764+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-06-04T15:46:47.109502+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,4],"acting":[3,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"17.16","version":"0'0","reported_seq":972,"reported_epoch":1009,"state":"active+undersized","last_fresh":"2024-06-03T09:06:14.441665+0000","last_change":"2024-06-03T09:06:03.478615+0000","last_active":"2024-06-03T09:06:14.441665+0000","last_peered":"2024-06-03T09:06:14.441665+0000","last_clean":"2024-06-03T09:04:43.328415+0000","last_became_active":"2024-06-03T09:06:03.478615+0000","last_became_peered":"2024-06-03T09:06:03.478615+0000","last_unstale":"2024-06-03T09:06:14.441665+0000","last_undegraded":"2024-06-03T09:06:14.441665+0000","last_fullsized":"2024-06-03T09:06:03.402504+0000","mapping_epoch":999,"log_start":"0'0","ondisk_log_start":"0'0","created":28,"last_epoch_clean":255,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-06-03T08:37:55.614497+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-06-03T08:37:55.614497+0000","last_clean_scrub_stamp":"2024-06-03T08:37:55.614497+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-06-04T08:47:08.079233+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2],"acting":[2],"avail_no_missing":["2"],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"2.4","version":"0'0","reported_seq":952,"reported_epoch":1010,"state":"active+undersized","last_fresh":"2024-06-03T09:06:18.331485+0000","last_change":"2024-06-03T09:06:03.387171+0000","last_active":"2024-06-03T09:06:18.331485+0000","last_peered":"2024-06-03T09:06:18.331485+0000","last_clean":"2024-06-03T09:04:43.202645+0000","last_became_active":"2024-06-03T09:06:03.387171+0000","last_became_peered":"2024-06-03T09:06:03.387171+0000","last_unstale":"2024-06-03T09:06:18.331485+0000","last_undegraded":"2024-06-03T09:06:18.331485+0000","last_fullsized":"2024-06-03T09:06:03.283787+0000","mapping_epoch":999,"log_start":"0'0","ondisk_log_start":"0'0","created":23,"last_epoch_clean":255,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-06-03T08:36:28.444764+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-06-03T08:36:28.444764+0000","last_clean_scrub_stamp":"2024-06-03T08:36:28.444764+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-06-04T20:17:43.485023+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5],"acting":[5],"avail_no_missing":["5"],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"17.17","version":"0'0","reported_seq":954,"reported_epoch":1009,"state":"active+clean","last_fresh":"2024-06-03T09:06:14.447046+0000","last_change":"2024-06-03T08:38:38.483275+0000","last_active":"2024-06-03T09:06:14.447046+0000","last_peered":"2024-06-03T09:06:14.447046+0000","last_clean":"2024-06-03T09:06:14.447046+0000","last_became_active":"2024-06-03T08:38:38.467916+0000","last_became_peered":"2024-06-03T08:38:38.467916+0000","last_unstale":"2024-06-03T09:06:14.447046+0000","last_undegraded":"2024-06-03T09:06:14.447046+0000","last_fullsized":"2024-06-03T09:06:14.447046+0000","mapping_epoch":28,"log_start":"0'0","ondisk_log_start":"0'0","created":28,"last_epoch_clean":61,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-06-03T08:37:55.614497+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-06-03T08:37:55.614497+0000","last_clean_scrub_stamp":"2024-06-03T08:37:55.614497+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-06-04T10:02:23.036420+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,7],"acting":[2,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"2.2","version":"0'0","reported_seq":963,"reported_epoch":1009,"state":"active+undersized","last_fresh":"2024-06-03T09:06:13.657892+0000","last_change":"2024-06-03T09:06:03.410020+0000","last_active":"2024-06-03T09:06:13.657892+0000","last_peered":"2024-06-03T09:06:13.657892+0000","last_clean":"2024-06-03T09:05:37.145889+0000","last_became_active":"2024-06-03T09:06:03.410020+0000","last_became_peered":"2024-06-03T09:06:03.410020+0000","last_unstale":"2024-06-03T09:06:13.657892+0000","last_undegraded":"2024-06-03T09:06:13.657892+0000","last_fullsized":"2024-06-03T09:06:03.346864+0000","mapping_epoch":999,"log_start":"0'0","ondisk_log_start":"0'0","created":23,"last_epoch_clean":255,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-06-03T08:36:28.444764+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-06-03T08:36:28.444764+0000","last_clean_scrub_stamp":"2024-06-03T08:36:28.444764+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-06-04T09:57:12.423360+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3],"acting":[3],"avail_no_missing":["3"],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"17.11","version":"0'0","reported_seq":944,"reported_epoch":1009,"state":"active+clean","last_fresh":"2024-06-03T09:06:14.426670+0000","last_change":"2024-06-03T08:38:06.502302+0000","last_active":"2024-06-03T09:06:14.426670+0000","last_peered":"2024-06-03T09:06:14.426670+0000","last_clean":"2024-06-03T09:06:14.426670+0000","last_became_active":"2024-06-03T08:38:06.490204+0000","last_became_peered":"2024-06-03T08:38:06.490204+0000","last_unstale":"2024-06-03T09:06:14.426670+0000","last_undegraded":"2024-06-03T09:06:14.426670+0000","last_fullsized":"2024-06-03T09:06:14.426670+0000","mapping_epoch":28,"log_start":"0'0","ondisk_log_start":"0'0","created":28,"last_epoch_clean":32,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-06-03T08:37:55.614497+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-06-03T08:37:55.614497+0000","last_clean_scrub_stamp":"2024-06-03T08:37:55.614497+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-06-04T10:47:05.988958+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,2],"acting":[4,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"2.1","version":"0'0","reported_seq":952,"reported_epoch":1009,"state":"active+clean","last_fresh":"2024-06-03T09:06:13.929725+0000","last_change":"2024-06-03T08:36:30.883360+0000","last_active":"2024-06-03T09:06:13.929725+0000","last_peered":"2024-06-03T09:06:13.929725+0000","last_clean":"2024-06-03T09:06:13.929725+0000","last_became_active":"2024-06-03T08:36:30.855073+0000","last_became_peered":"2024-06-03T08:36:30.855073+0000","last_unstale":"2024-06-03T09:06:13.929725+0000","last_undegraded":"2024-06-03T09:06:13.929725+0000","last_fullsized":"2024-06-03T09:06:13.929725+0000","mapping_epoch":23,"log_start":"0'0","ondisk_log_start":"0'0","created":23,"last_epoch_clean":25,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-06-03T08:36:28.444764+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-06-03T08:36:28.444764+0000","last_clean_scrub_stamp":"2024-06-03T08:36:28.444764+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-06-04T18:12:22.363473+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,5],"acting":[7,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"17.12","version":"0'0","reported_seq":945,"reported_epoch":1009,"state":"active+clean","last_fresh":"2024-06-03T09:06:13.935141+0000","last_change":"2024-06-03T08:38:01.751562+0000","last_active":"2024-06-03T09:06:13.935141+0000","last_peered":"2024-06-03T09:06:13.935141+0000","last_clean":"2024-06-03T09:06:13.935141+0000","last_became_active":"2024-06-03T08:38:01.745027+0000","last_became_peered":"2024-06-03T08:38:01.745027+0000","last_unstale":"2024-06-03T09:06:13.935141+0000","last_undegraded":"2024-06-03T09:06:13.935141+0000","last_fullsized":"2024-06-03T09:06:13.935141+0000","mapping_epoch":28,"log_start":"0'0","ondisk_log_start":"0'0","created":28,"last_epoch_clean":32,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-06-03T08:37:55.614497+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-06-03T08:37:55.614497+0000","last_clean_scrub_stamp":"2024-06-03T08:37:55.614497+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-06-04T17:54:43.785326+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,3],"acting":[7,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"2.0","version":"0'0","reported_seq":943,"reported_epoch":1009,"state":"active+clean","last_fresh":"2024-06-03T09:06:13.648112+0000","last_change":"2024-06-03T08:36:29.984096+0000","last_active":"2024-06-03T09:06:13.648112+0000","last_peered":"2024-06-03T09:06:13.648112+0000","last_clean":"2024-06-03T09:06:13.648112+0000","last_became_active":"2024-06-03T08:36:29.957875+0000","last_became_peered":"2024-06-03T08:36:29.957875+0000","last_unstale":"2024-06-03T09:06:13.648112+0000","last_undegraded":"2024-06-03T09:06:13.648112+0000","last_fullsized":"2024-06-03T09:06:13.648112+0000","mapping_epoch":23,"log_start":"0'0","ondisk_log_start":"0'0","created":23,"last_epoch_clean":24,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-06-03T08:36:28.444764+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-06-03T08:36:28.444764+0000","last_clean_scrub_stamp":"2024-06-03T08:36:28.444764+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-06-04T08:41:33.679479+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,5],"acting":[3,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"17.13","version":"0'0","reported_seq":943,"reported_epoch":1009,"state":"active+clean","last_fresh":"2024-06-03T09:06:14.438399+0000","last_change":"2024-06-03T08:38:26.838316+0000","last_active":"2024-06-03T09:06:14.438399+0000","last_peered":"2024-06-03T09:06:14.438399+0000","last_clean":"2024-06-03T09:06:14.438399+0000","last_became_active":"2024-06-03T08:38:26.833214+0000","last_became_peered":"2024-06-03T08:38:26.833214+0000","last_unstale":"2024-06-03T09:06:14.438399+0000","last_undegraded":"2024-06-03T09:06:14.438399+0000","last_fullsized":"2024-06-03T09:06:14.438399+0000","mapping_epoch":28,"log_start":"0'0","ondisk_log_start":"0'0","created":28,"last_epoch_clean":52,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-06-03T08:37:55.614497+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-06-03T08:37:55.614497+0000","last_clean_scrub_stamp":"2024-06-03T08:37:55.614497+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-06-04T12:58:51.617998+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,0],"acting":[4,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"2.3","version":"0'0","reported_seq":943,"reported_epoch":1010,"state":"active+clean","last_fresh":"2024-06-03T09:06:18.362614+0000","last_change":"2024-06-03T08:36:30.351670+0000","last_active":"2024-06-03T09:06:18.362614+0000","last_peered":"2024-06-03T09:06:18.362614+0000","last_clean":"2024-06-03T09:06:18.362614+0000","last_became_active":"2024-06-03T08:36:30.343560+0000","last_became_peered":"2024-06-03T08:36:30.343560+0000","last_unstale":"2024-06-03T09:06:18.362614+0000","last_undegraded":"2024-06-03T09:06:18.362614+0000","last_fullsized":"2024-06-03T09:06:18.362614+0000","mapping_epoch":23,"log_start":"0'0","ondisk_log_start":"0'0","created":23,"last_epoch_clean":24,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-06-03T08:36:28.444764+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-06-03T08:36:28.444764+0000","last_clean_scrub_stamp":"2024-06-03T08:36:28.444764+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-06-04T12:02:24.765135+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,2],"acting":[3,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.0","version":"19'76","reported_seq":1051,"reported_epoch":1009,"state":"active+clean","last_fresh":"2024-06-03T09:06:13.703685+0000","last_change":"2024-06-03T08:36:26.702702+0000","last_active":"2024-06-03T09:06:13.703685+0000","last_peered":"2024-06-03T09:06:13.703685+0000","last_clean":"2024-06-03T09:06:13.703685+0000","last_became_active":"2024-06-03T08:36:26.128378+0000","last_became_peered":"2024-06-03T08:36:26.128378+0000","last_unstale":"2024-06-03T09:06:13.703685+0000","last_undegraded":"2024-06-03T09:06:13.703685+0000","last_fullsized":"2024-06-03T09:06:13.703685+0000","mapping_epoch":20,"log_start":"0'0","ondisk_log_start":"0'0","created":16,"last_epoch_clean":22,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-06-03T08:36:15.542635+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-06-03T08:36:15.542635+0000","last_clean_scrub_stamp":"2024-06-03T08:36:15.542635+0000","objects_scrubbed":0,"log_size":76,"log_dups_size":0,"ondisk_log_size":76,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-06-04T12:41:53.731360+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":459280,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":96,"num_read_kb":82,"num_write":113,"num_write_kb":1372,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2,"num_bytes_recovered":397840,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,5],"acting":[1,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"17.10","version":"0'0","reported_seq":944,"reported_epoch":1009,"state":"active+clean","last_fresh":"2024-06-03T09:06:14.422035+0000","last_change":"2024-06-03T08:38:15.399189+0000","last_active":"2024-06-03T09:06:14.422035+0000","last_peered":"2024-06-03T09:06:14.422035+0000","last_clean":"2024-06-03T09:06:14.422035+0000","last_became_active":"2024-06-03T08:38:15.337723+0000","last_became_peered":"2024-06-03T08:38:15.337723+0000","last_unstale":"2024-06-03T09:06:14.422035+0000","last_undegraded":"2024-06-03T09:06:14.422035+0000","last_fullsized":"2024-06-03T09:06:14.422035+0000","mapping_epoch":28,"log_start":"0'0","ondisk_log_start":"0'0","created":28,"last_epoch_clean":42,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-06-03T08:37:55.614497+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-06-03T08:37:55.614497+0000","last_clean_scrub_stamp":"2024-06-03T08:37:55.614497+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-06-04T09:29:46.427317+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,3],"acting":[4,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"17.1a","version":"0'0","reported_seq":960,"reported_epoch":1009,"state":"active+undersized","last_fresh":"2024-06-03T09:06:14.431224+0000","last_change":"2024-06-03T09:06:03.390734+0000","last_active":"2024-06-03T09:06:14.431224+0000","last_peered":"2024-06-03T09:06:14.431224+0000","last_clean":"2024-06-03T09:05:20.322313+0000","last_became_active":"2024-06-03T09:06:03.390734+0000","last_became_peered":"2024-06-03T09:06:03.390734+0000","last_unstale":"2024-06-03T09:06:14.431224+0000","last_undegraded":"2024-06-03T09:06:14.431224+0000","last_fullsized":"2024-06-03T09:06:03.343197+0000","mapping_epoch":999,"log_start":"0'0","ondisk_log_start":"0'0","created":28,"last_epoch_clean":254,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-06-03T08:37:55.614497+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-06-03T08:37:55.614497+0000","last_clean_scrub_stamp":"2024-06-03T08:37:55.614497+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-06-04T15:56:47.628875+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4],"acting":[4],"avail_no_missing":["4"],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"17.1b","version":"0'0","reported_seq":935,"reported_epoch":1009,"state":"active+clean","last_fresh":"2024-06-03T09:06:13.663279+0000","last_change":"2024-06-03T08:38:03.115748+0000","last_active":"2024-06-03T09:06:13.663279+0000","last_peered":"2024-06-03T09:06:13.663279+0000","last_clean":"2024-06-03T09:06:13.663279+0000","last_became_active":"2024-06-03T08:38:03.110533+0000","last_became_peered":"2024-06-03T08:38:03.110533+0000","last_unstale":"2024-06-03T09:06:13.663279+0000","last_undegraded":"2024-06-03T09:06:13.663279+0000","last_fullsized":"2024-06-03T09:06:13.663279+0000","mapping_epoch":28,"log_start":"0'0","ondisk_log_start":"0'0","created":28,"last_epoch_clean":32,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-06-03T08:37:55.614497+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-06-03T08:37:55.614497+0000","last_clean_scrub_stamp":"2024-06-03T08:37:55.614497+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-06-04T12:36:47.471609+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,7],"acting":[3,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"17.18","version":"0'0","reported_seq":944,"reported_epoch":1009,"state":"active+clean","last_fresh":"2024-06-03T09:06:13.924799+0000","last_change":"2024-06-03T08:38:04.632864+0000","last_active":"2024-06-03T09:06:13.924799+0000","last_peered":"2024-06-03T09:06:13.924799+0000","last_clean":"2024-06-03T09:06:13.924799+0000","last_became_active":"2024-06-03T08:38:04.627856+0000","last_became_peered":"2024-06-03T08:38:04.627856+0000","last_unstale":"2024-06-03T09:06:13.924799+0000","last_undegraded":"2024-06-03T09:06:13.924799+0000","last_fullsized":"2024-06-03T09:06:13.924799+0000","mapping_epoch":28,"log_start":"0'0","ondisk_log_start":"0'0","created":28,"last_epoch_clean":33,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-06-03T08:37:55.614497+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-06-03T08:37:55.614497+0000","last_clean_scrub_stamp":"2024-06-03T08:37:55.614497+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-06-04T19:55:51.181598+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,4],"acting":[7,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"17.19","version":"0'0","reported_seq":934,"reported_epoch":1009,"state":"active+clean","last_fresh":"2024-06-03T09:06:13.652943+0000","last_change":"2024-06-03T08:38:01.839491+0000","last_active":"2024-06-03T09:06:13.652943+0000","last_peered":"2024-06-03T09:06:13.652943+0000","last_clean":"2024-06-03T09:06:13.652943+0000","last_became_active":"2024-06-03T08:38:01.829539+0000","last_became_peered":"2024-06-03T08:38:01.829539+0000","last_unstale":"2024-06-03T09:06:13.652943+0000","last_undegraded":"2024-06-03T09:06:13.652943+0000","last_fullsized":"2024-06-03T09:06:13.652943+0000","mapping_epoch":28,"log_start":"0'0","ondisk_log_start":"0'0","created":28,"last_epoch_clean":30,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-06-03T08:37:55.614497+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-06-03T08:37:55.614497+0000","last_clean_scrub_stamp":"2024-06-03T08:37:55.614497+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-06-04T09:10:43.860229+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,2],"acting":[3,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"17.1e","version":"0'0","reported_seq":148,"reported_epoch":1009,"state":"active+undersized","last_fresh":"2024-06-03T09:06:14.445917+0000","last_change":"2024-06-03T09:06:03.406181+0000","last_active":"2024-06-03T09:06:14.445917+0000","last_peered":"2024-06-03T09:06:14.445917+0000","last_clean":"2024-06-03T08:37:55.614497+0000","last_became_active":"2024-06-03T09:06:03.406181+0000","last_became_peered":"2024-06-03T09:06:03.406181+0000","last_unstale":"2024-06-03T09:06:14.445917+0000","last_undegraded":"2024-06-03T09:06:14.445917+0000","last_fullsized":"2024-06-03T09:06:03.363739+0000","mapping_epoch":999,"log_start":"0'0","ondisk_log_start":"0'0","created":28,"last_epoch_clean":255,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-06-03T08:37:55.614497+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-06-03T08:37:55.614497+0000","last_clean_scrub_stamp":"2024-06-03T08:37:55.614497+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-06-04T11:45:24.486600+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4],"acting":[4],"avail_no_missing":["4"],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"17.1f","version":"0'0","reported_seq":941,"reported_epoch":1009,"state":"active+clean","last_fresh":"2024-06-03T09:06:14.450053+0000","last_change":"2024-06-03T08:38:04.383538+0000","last_active":"2024-06-03T09:06:14.450053+0000","last_peered":"2024-06-03T09:06:14.450053+0000","last_clean":"2024-06-03T09:06:14.450053+0000","last_became_active":"2024-06-03T08:38:04.367278+0000","last_became_peered":"2024-06-03T08:38:04.367278+0000","last_unstale":"2024-06-03T09:06:14.450053+0000","last_undegraded":"2024-06-03T09:06:14.450053+0000","last_fullsized":"2024-06-03T09:06:14.450053+0000","mapping_epoch":28,"log_start":"0'0","ondisk_log_start":"0'0","created":28,"last_epoch_clean":32,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-06-03T08:37:55.614497+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-06-03T08:37:55.614497+0000","last_clean_scrub_stamp":"2024-06-03T08:37:55.614497+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-06-04T10:39:26.552282+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,2],"acting":[0,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"17.1c","version":"0'0","reported_seq":944,"reported_epoch":1010,"state":"active+undersized","last_fresh":"2024-06-03T09:06:18.327492+0000","last_change":"2024-06-03T09:06:03.382210+0000","last_active":"2024-06-03T09:06:18.327492+0000","last_peered":"2024-06-03T09:06:18.327492+0000","last_clean":"2024-06-03T09:04:43.199660+0000","last_became_active":"2024-06-03T09:06:03.382210+0000","last_became_peered":"2024-06-03T09:06:03.382210+0000","last_unstale":"2024-06-03T09:06:18.327492+0000","last_undegraded":"2024-06-03T09:06:18.327492+0000","last_fullsized":"2024-06-03T09:06:03.277874+0000","mapping_epoch":999,"log_start":"0'0","ondisk_log_start":"0'0","created":28,"last_epoch_clean":255,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-06-03T08:37:55.614497+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-06-03T08:37:55.614497+0000","last_clean_scrub_stamp":"2024-06-03T08:37:55.614497+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-06-04T12:18:50.013884+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5],"acting":[5],"avail_no_missing":["5"],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"17.1d","version":"0'0","reported_seq":923,"reported_epoch":1009,"state":"active+clean","last_fresh":"2024-06-03T09:06:13.715906+0000","last_change":"2024-06-03T08:38:06.794630+0000","last_active":"2024-06-03T09:06:13.715906+0000","last_peered":"2024-06-03T09:06:13.715906+0000","last_clean":"2024-06-03T09:06:13.715906+0000","last_became_active":"2024-06-03T08:38:06.592317+0000","last_became_peered":"2024-06-03T08:38:06.592317+0000","last_unstale":"2024-06-03T09:06:13.715906+0000","last_undegraded":"2024-06-03T09:06:13.715906+0000","last_fullsized":"2024-06-03T09:06:13.715906+0000","mapping_epoch":28,"log_start":"0'0","ondisk_log_start":"0'0","created":28,"last_epoch_clean":33,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-06-03T08:37:55.614497+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-06-03T08:37:55.614497+0000","last_clean_scrub_stamp":"2024-06-03T08:37:55.614497+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-06-04T12:33:30.785369+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,7],"acting":[1,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"17.2","version":"0'0","reported_seq":925,"reported_epoch":1009,"state":"active+clean","last_fresh":"2024-06-03T09:06:13.708688+0000","last_change":"2024-06-03T08:38:14.981410+0000","last_active":"2024-06-03T09:06:13.708688+0000","last_peered":"2024-06-03T09:06:13.708688+0000","last_clean":"2024-06-03T09:06:13.708688+0000","last_became_active":"2024-06-03T08:38:14.973825+0000","last_became_peered":"2024-06-03T08:38:14.973825+0000","last_unstale":"2024-06-03T09:06:13.708688+0000","last_undegraded":"2024-06-03T09:06:13.708688+0000","last_fullsized":"2024-06-03T09:06:13.708688+0000","mapping_epoch":28,"log_start":"0'0","ondisk_log_start":"0'0","created":28,"last_epoch_clean":41,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-06-03T08:37:55.614497+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-06-03T08:37:55.614497+0000","last_clean_scrub_stamp":"2024-06-03T08:37:55.614497+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-06-04T17:01:05.518613+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,4],"acting":[1,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"17.3","version":"0'0","reported_seq":943,"reported_epoch":1009,"state":"active+clean","last_fresh":"2024-06-03T09:06:14.437636+0000","last_change":"2024-06-03T08:38:14.619745+0000","last_active":"2024-06-03T09:06:14.437636+0000","last_peered":"2024-06-03T09:06:14.437636+0000","last_clean":"2024-06-03T09:06:14.437636+0000","last_became_active":"2024-06-03T08:38:14.510665+0000","last_became_peered":"2024-06-03T08:38:14.510665+0000","last_unstale":"2024-06-03T09:06:14.437636+0000","last_undegraded":"2024-06-03T09:06:14.437636+0000","last_fullsized":"2024-06-03T09:06:14.437636+0000","mapping_epoch":28,"log_start":"0'0","ondisk_log_start":"0'0","created":28,"last_epoch_clean":42,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-06-03T08:37:55.614497+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-06-03T08:37:55.614497+0000","last_clean_scrub_stamp":"2024-06-03T08:37:55.614497+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-06-04T12:03:47.392064+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,7],"acting":[0,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"17.0","version":"0'0","reported_seq":944,"reported_epoch":1009,"state":"active+clean","last_fresh":"2024-06-03T09:06:14.427123+0000","last_change":"2024-06-03T08:38:04.237118+0000","last_active":"2024-06-03T09:06:14.427123+0000","last_peered":"2024-06-03T09:06:14.427123+0000","last_clean":"2024-06-03T09:06:14.427123+0000","last_became_active":"2024-06-03T08:38:04.231954+0000","last_became_peered":"2024-06-03T08:38:04.231954+0000","last_unstale":"2024-06-03T09:06:14.427123+0000","last_undegraded":"2024-06-03T09:06:14.427123+0000","last_fullsized":"2024-06-03T09:06:14.427123+0000","mapping_epoch":28,"log_start":"0'0","ondisk_log_start":"0'0","created":28,"last_epoch_clean":32,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-06-03T08:37:55.614497+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-06-03T08:37:55.614497+0000","last_clean_scrub_stamp":"2024-06-03T08:37:55.614497+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-06-04T09:56:10.741637+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,2],"acting":[0,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"17.1","version":"65'7","reported_seq":959,"reported_epoch":1009,"state":"active+undersized+degraded","last_fresh":"2024-06-03T09:06:13.709209+0000","last_change":"2024-06-03T09:06:03.370660+0000","last_active":"2024-06-03T09:06:13.709209+0000","last_peered":"2024-06-03T09:06:13.709209+0000","last_clean":"2024-06-03T09:05:03.733864+0000","last_became_active":"2024-06-03T09:06:03.370660+0000","last_became_peered":"2024-06-03T09:06:03.370660+0000","last_unstale":"2024-06-03T09:06:13.709209+0000","last_undegraded":"2024-06-03T09:06:03.266076+0000","last_fullsized":"2024-06-03T09:06:03.264525+0000","mapping_epoch":999,"log_start":"0'0","ondisk_log_start":"0'0","created":28,"last_epoch_clean":255,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-06-03T08:37:55.614497+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-06-03T08:37:55.614497+0000","last_clean_scrub_stamp":"2024-06-03T08:37:55.614497+0000","objects_scrubbed":0,"log_size":7,"log_dups_size":0,"ondisk_log_size":7,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-06-04T11:55:18.194283+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":5,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":1,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":6,"num_read_kb":6,"num_write":7,"num_write_kb":7,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5],"acting":[5],"avail_no_missing":["5"],"object_location_counts":[{"shards":"5","objects":1}],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]}],"pool_stats":[{"poolid":2,"num_pg":8,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":14,"acting":14,"num_store_stats":7},{"poolid":1,"num_pg":1,"stat_sum":{"num_bytes":459280,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":96,"num_read_kb":82,"num_write":113,"num_write_kb":1372,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2,"num_bytes_recovered":397840,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":802816,"data_stored":795680,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":76,"ondisk_log_size":76,"up":2,"acting":2,"num_store_stats":2},{"poolid":17,"num_pg":32,"stat_sum":{"num_bytes":5,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":1,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":6,"num_read_kb":6,"num_write":7,"num_write_kb":7,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":8192,"data_stored":10,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":7,"ondisk_log_size":7,"up":52,"acting":52,"num_store_stats":8}],"osd_stats":[{"osd":6,"up_from":253,"seq":1086626726200,"num_pgs":32,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":54772,"kb_used_data":5608,"kb_used_omap":1,"kb_used_meta":49150,"kb_avail":94317068,"statfs":{"total":96636764160,"available":96580677632,"internally_reserved":0,"allocated":5742592,"data_stored":2767180,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1593,"internal_metadata":50330055},"hb_peers":[0,1,2,3,4,5,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":109,"apply_latency_ms":109,"commit_latency_ns":109000000,"apply_latency_ns":109000000},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Mon Jun 3 09:04:02 2024","interfaces":[{"interface":"back","average":{"1min":1582.4780000000001,"5min":332.72699999999998,"15min":124.113},"min":{"1min":4.4829999999999997,"5min":4.1959999999999997,"15min":4.1959999999999997},"max":{"1min":22766.882000000001,"5min":22766.882000000001,"15min":22766.882000000001},"last":21.917999999999999},{"interface":"front","average":{"1min":1579.5409999999999,"5min":334.94499999999999,"15min":125.84999999999999},"min":{"1min":4.8129999999999997,"5min":4.7539999999999996,"15min":4.4530000000000003},"max":{"1min":22766.289000000001,"5min":22766.289000000001,"15min":22766.289000000001},"last":25.408999999999999}]},{"osd":1,"last update":"Mon Jun 3 09:03:40 2024","interfaces":[{"interface":"back","average":{"1min":36.420999999999999,"5min":77.054000000000002,"15min":132.84800000000001},"min":{"1min":6.0380000000000003,"5min":5.8799999999999999,"15min":5.484},"max":{"1min":589.64099999999996,"5min":4138.1679999999997,"15min":9955.1299999999992},"last":26.84},{"interface":"front","average":{"1min":40.240000000000002,"5min":79.674999999999997,"15min":133.38999999999999},"min":{"1min":8.4589999999999996,"5min":8.4090000000000007,"15min":6.1390000000000002},"max":{"1min":588.60799999999995,"5min":4137.6530000000002,"15min":9827.5509999999995},"last":26.263000000000002}]},{"osd":2,"last update":"Mon Jun 3 09:03:40 2024","interfaces":[{"interface":"back","average":{"1min":12.613,"5min":24.600999999999999,"15min":21.088000000000001},"min":{"1min":6.9740000000000002,"5min":6.9740000000000002,"15min":6.5330000000000004},"max":{"1min":26.076000000000001,"5min":376.24900000000002,"15min":376.24900000000002},"last":24.899000000000001},{"interface":"front","average":{"1min":12.493,"5min":22.207999999999998,"15min":21.219000000000001},"min":{"1min":6.5810000000000004,"5min":6.5110000000000001,"15min":6.0750000000000002},"max":{"1min":29.690999999999999,"5min":271.51100000000002,"15min":434.77300000000002},"last":40.372}]},{"osd":3,"last update":"Mon Jun 3 09:03:40 2024","interfaces":[{"interface":"back","average":{"1min":14.093999999999999,"5min":24.143999999999998,"15min":29.376999999999999},"min":{"1min":10.196999999999999,"5min":6.2919999999999998,"15min":6.1719999999999997},"max":{"1min":24.832000000000001,"5min":197.84899999999999,"15min":1235.5799999999999},"last":24.353999999999999},{"interface":"front","average":{"1min":9.3260000000000005,"5min":18.463999999999999,"15min":27.631},"min":{"1min":6.125,"5min":6.125,"15min":6.125},"max":{"1min":28.166,"5min":200.173,"15min":1157.6780000000001},"last":23.454000000000001}]},{"osd":4,"last update":"Mon Jun 3 09:04:29 2024","interfaces":[{"interface":"back","average":{"1min":10.279,"5min":1253.809,"15min":450.79199999999997},"min":{"1min":7.2300000000000004,"5min":6.093,"15min":6.093},"max":{"1min":15.000999999999999,"5min":28093.096000000001,"15min":28093.096000000001},"last":101.45399999999999},{"interface":"front","average":{"1min":15.693,"5min":1279.759,"15min":463.87099999999998},"min":{"1min":10.551,"5min":7.7480000000000002,"15min":7.7480000000000002},"max":{"1min":27.898,"5min":28323.837,"15min":28323.837},"last":102.262}]},{"osd":5,"last update":"Mon Jun 3 09:04:10 2024","interfaces":[{"interface":"back","average":{"1min":13.728999999999999,"5min":24.670000000000002,"15min":23.815999999999999},"min":{"1min":10.401,"5min":7.516,"15min":7.516},"max":{"1min":25.138000000000002,"5min":320.85000000000002,"15min":445.041},"last":103.52500000000001},{"interface":"front","average":{"1min":13.847,"5min":29.283000000000001,"15min":23.393999999999998},"min":{"1min":7.944,"5min":6.266,"15min":6.266},"max":{"1min":18.817,"5min":530.56700000000001,"15min":530.56700000000001},"last":29.263999999999999}]},{"osd":7,"last update":"Mon Jun 3 09:03:40 2024","interfaces":[{"interface":"back","average":{"1min":20.568999999999999,"5min":30.286999999999999,"15min":27.977},"min":{"1min":13.874000000000001,"5min":10.58,"15min":9.2859999999999996},"max":{"1min":86.959000000000003,"5min":222.07599999999999,"15min":255.08799999999999},"last":37.618000000000002},{"interface":"front","average":{"1min":19.102,"5min":29.550000000000001,"15min":27.763000000000002},"min":{"1min":10.052,"5min":8.6709999999999994,"15min":8.5429999999999993},"max":{"1min":67.968999999999994,"5min":293.62099999999998,"15min":438.14600000000002},"last":111.985}]}]},{"osd":7,"up_from":20,"seq":85899346288,"num_pgs":9,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":58964,"kb_used_data":5704,"kb_used_omap":1,"kb_used_meta":53246,"kb_avail":94312876,"statfs":{"total":96636764160,"available":96576385024,"internally_reserved":0,"allocated":5840896,"data_stored":2740727,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1589,"internal_metadata":54524363},"hb_peers":[0,1,2,3,4,5],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Mon Jun 3 09:06:09 2024","interfaces":[{"interface":"back","average":{"1min":14.073,"5min":894.22400000000005,"15min":308.93400000000003},"min":{"1min":5.5709999999999997,"5min":4.218,"15min":4.0599999999999996},"max":{"1min":90.757999999999996,"5min":21729.001,"15min":21729.001},"last":6.1630000000000003},{"interface":"front","average":{"1min":9.2550000000000008,"5min":906.23500000000001,"15min":314.06},"min":{"1min":5.5789999999999997,"5min":4.8810000000000002,"15min":4.8810000000000002},"max":{"1min":25.937000000000001,"5min":21899.732,"15min":21899.732},"last":7.109}]},{"osd":1,"last update":"Mon Jun 3 09:06:09 2024","interfaces":[{"interface":"back","average":{"1min":15.182,"5min":69.183999999999997,"15min":219.173},"min":{"1min":7.3639999999999999,"5min":6.6200000000000001,"15min":6.1319999999999997},"max":{"1min":76.926000000000002,"5min":2795.1329999999998,"15min":18373.257000000001},"last":9.9030000000000005},{"interface":"front","average":{"1min":21.495999999999999,"5min":74.703999999999994,"15min":225.285},"min":{"1min":8.8399999999999999,"5min":8.8399999999999999,"15min":8.5129999999999999},"max":{"1min":94.977999999999994,"5min":2803.3670000000002,"15min":18374.601999999999},"last":12.782}]},{"osd":2,"last update":"Mon Jun 3 09:06:09 2024","interfaces":[{"interface":"back","average":{"1min":19.164000000000001,"5min":26.902000000000001,"15min":22.920999999999999},"min":{"1min":11.207000000000001,"5min":8.5030000000000001,"15min":8.5030000000000001},"max":{"1min":91.319999999999993,"5min":333.99200000000002,"15min":333.99200000000002},"last":14.696},{"interface":"front","average":{"1min":11.125999999999999,"5min":18.132000000000001,"15min":16.353000000000002},"min":{"1min":6.202,"5min":5.8369999999999997,"15min":5.5069999999999997},"max":{"1min":28.649000000000001,"5min":226.18100000000001,"15min":226.18100000000001},"last":8.0749999999999993}]},{"osd":3,"last update":"Mon Jun 3 09:06:09 2024","interfaces":[{"interface":"back","average":{"1min":22.291,"5min":26.289999999999999,"15min":31.074999999999999},"min":{"1min":11.714,"5min":6.8959999999999999,"15min":6.3760000000000003},"max":{"1min":83.102000000000004,"5min":316.64400000000001,"15min":2039.9059999999999},"last":13.901},{"interface":"front","average":{"1min":13.324999999999999,"5min":17.963000000000001,"15min":25.969999999999999},"min":{"1min":6.1200000000000001,"5min":5.2089999999999996,"15min":5.2089999999999996},"max":{"1min":49.908999999999999,"5min":233.542,"15min":2037.5550000000001},"last":9.3919999999999995}]},{"osd":4,"last update":"Mon Jun 3 09:06:29 2024","interfaces":[{"interface":"back","average":{"1min":17.675999999999998,"5min":1026.325,"15min":472.47199999999998},"min":{"1min":7.306,"5min":5.9089999999999998,"15min":5.9089999999999998},"max":{"1min":140.03299999999999,"5min":28969.369999999999,"15min":30073.917000000001},"last":8.8829999999999991},{"interface":"front","average":{"1min":17.890999999999998,"5min":1044.5139999999999,"15min":480.226},"min":{"1min":5.9279999999999999,"5min":5.9279999999999999,"15min":5.9279999999999999},"max":{"1min":141.179,"5min":29244.138999999999,"15min":30348.021000000001},"last":12.252000000000001}]},{"osd":5,"last update":"Mon Jun 3 09:06:02 2024","interfaces":[{"interface":"back","average":{"1min":24.372,"5min":26.856000000000002,"15min":23.571999999999999},"min":{"1min":12.278,"5min":9.2460000000000004,"15min":6.5650000000000004},"max":{"1min":183.64099999999999,"5min":317.572,"15min":317.572},"last":15.528},{"interface":"front","average":{"1min":21.259,"5min":23.364000000000001,"15min":21.193999999999999},"min":{"1min":7.8499999999999996,"5min":7.0999999999999996,"15min":7.008},"max":{"1min":184.53399999999999,"5min":306.60599999999999,"15min":306.60599999999999},"last":10.446999999999999}]},{"osd":6,"last update":"Mon Jun 3 09:04:31 2024","interfaces":[{"interface":"back","average":{"1min":44.061,"5min":26.821000000000002,"15min":91.176000000000002},"min":{"1min":9.0800000000000001,"5min":7.9390000000000001,"15min":7.9390000000000001},"max":{"1min":268.029,"5min":308.916,"15min":8734.4639999999999},"last":6.6769999999999996},{"interface":"front","average":{"1min":46.643000000000001,"5min":23.879999999999999,"15min":88.765000000000001},"min":{"1min":8.8119999999999994,"5min":6.9240000000000004,"15min":6.742},"max":{"1min":305.64600000000002,"5min":305.64600000000002,"15min":8735.5319999999992},"last":12.327}]}]},{"osd":5,"up_from":20,"seq":85899346262,"num_pgs":9,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":59360,"kb_used_data":6100,"kb_used_omap":1,"kb_used_meta":53246,"kb_avail":94312480,"statfs":{"total":96636764160,"available":96575979520,"internally_reserved":0,"allocated":6246400,"data_stored":3138572,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1594,"internal_metadata":54524358},"hb_peers":[0,1,2,3,4,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Mon Jun 3 09:06:06 2024","interfaces":[{"interface":"back","average":{"1min":7.468,"5min":1703.037,"15min":580.51900000000001},"min":{"1min":4.6509999999999998,"5min":4.6509999999999998,"15min":4.4020000000000001},"max":{"1min":14.711,"5min":26433.764999999999,"15min":26433.764999999999},"last":4.641},{"interface":"front","average":{"1min":9.9429999999999996,"5min":1725.27,"15min":587.87800000000004},"min":{"1min":5.5999999999999996,"5min":5.1319999999999997,"15min":4.8170000000000002},"max":{"1min":13.590999999999999,"5min":26704.698,"15min":26704.698},"last":8.7279999999999998}]},{"osd":1,"last update":"Mon Jun 3 09:06:00 2024","interfaces":[{"interface":"back","average":{"1min":10.862,"5min":67.731999999999999,"15min":217.364},"min":{"1min":7.1769999999999996,"5min":6.0890000000000004,"15min":5.4930000000000003},"max":{"1min":16.998999999999999,"5min":3923.4090000000001,"15min":18811.696},"last":12.723000000000001},{"interface":"front","average":{"1min":10.417,"5min":65.141999999999996,"15min":220.75700000000001},"min":{"1min":5.3209999999999997,"5min":5.3209999999999997,"15min":5.0209999999999999},"max":{"1min":15.821999999999999,"5min":4019.6129999999998,"15min":18968.365000000002},"last":7.766}]},{"osd":2,"last update":"Mon Jun 3 09:06:00 2024","interfaces":[{"interface":"back","average":{"1min":11.366,"5min":24.18,"15min":21.137},"min":{"1min":5.6369999999999996,"5min":5.6369999999999996,"15min":5.0510000000000002},"max":{"1min":15.529999999999999,"5min":332.48200000000003,"15min":332.48200000000003},"last":11.538},{"interface":"front","average":{"1min":10.089,"5min":23.117000000000001,"15min":24.050000000000001},"min":{"1min":6.6769999999999996,"5min":6.6769999999999996,"15min":5.8739999999999997},"max":{"1min":15.821,"5min":353.03199999999998,"15min":353.03199999999998},"last":6.9349999999999996}]},{"osd":3,"last update":"Mon Jun 3 09:06:00 2024","interfaces":[{"interface":"back","average":{"1min":13.472,"5min":24.283999999999999,"15min":31.666},"min":{"1min":10.111000000000001,"5min":7.4720000000000004,"15min":7.4210000000000003},"max":{"1min":16.271999999999998,"5min":353.57499999999999,"15min":2855.8519999999999},"last":14.273999999999999},{"interface":"front","average":{"1min":9.5899999999999999,"5min":15.875,"15min":25.834},"min":{"1min":6.6609999999999996,"5min":5.7729999999999997,"15min":5.1909999999999998},"max":{"1min":15.112,"5min":203.804,"15min":2869.7860000000001},"last":13.554}]},{"osd":4,"last update":"Mon Jun 3 09:06:29 2024","interfaces":[{"interface":"back","average":{"1min":11.067,"5min":780.21000000000004,"15min":377.28399999999999},"min":{"1min":5.4779999999999998,"5min":5.4779999999999998,"15min":5.4779999999999998},"max":{"1min":14.654999999999999,"5min":23315.438999999998,"15min":28619.825000000001},"last":11.292},{"interface":"front","average":{"1min":14.349,"5min":782.97299999999996,"15min":377.93400000000003},"min":{"1min":7.492,"5min":6.5030000000000001,"15min":6.5030000000000001},"max":{"1min":17.707000000000001,"5min":23317.502,"15min":28621.294000000002},"last":15.134}]},{"osd":6,"last update":"Mon Jun 3 09:03:57 2024","interfaces":[{"interface":"back","average":{"1min":23.536000000000001,"5min":36.457000000000001,"15min":83.799999999999997},"min":{"1min":9.4800000000000004,"5min":9.4800000000000004,"15min":8.0630000000000006},"max":{"1min":89.442999999999998,"5min":508.77999999999997,"15min":9193.5169999999998},"last":157.79300000000001},{"interface":"front","average":{"1min":18.891999999999999,"5min":22.100999999999999,"15min":75.489999999999995},"min":{"1min":5.923,"5min":5.923,"15min":5.923},"max":{"1min":78.424999999999997,"5min":464.62,"15min":9191.4429999999993},"last":148.09}]},{"osd":7,"last update":"Mon Jun 3 09:06:00 2024","interfaces":[{"interface":"back","average":{"1min":13.257,"5min":20.917999999999999,"15min":21.088999999999999},"min":{"1min":10.475,"5min":6.7629999999999999,"15min":6.7629999999999999},"max":{"1min":17.245000000000001,"5min":211.22999999999999,"15min":211.22999999999999},"last":12.083},{"interface":"front","average":{"1min":12.872999999999999,"5min":20.587,"15min":24.021000000000001},"min":{"1min":8.5760000000000005,"5min":7.7709999999999999,"15min":6.8570000000000002},"max":{"1min":18.359999999999999,"5min":223.191,"15min":286.69499999999999},"last":9.3849999999999998}]}]},{"osd":4,"up_from":20,"seq":85899346270,"num_pgs":12,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":63060,"kb_used_data":5704,"kb_used_omap":1,"kb_used_meta":57342,"kb_avail":94308780,"statfs":{"total":96636764160,"available":96572190720,"internally_reserved":0,"allocated":5840896,"data_stored":2740727,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1594,"internal_metadata":58718662},"hb_peers":[0,1,2,3,5,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Mon Jun 3 09:05:34 2024","interfaces":[{"interface":"back","average":{"1min":12.416,"5min":1099.4649999999999,"15min":382.15199999999999},"min":{"1min":4.5279999999999996,"5min":4.5279999999999996,"15min":4.3220000000000001},"max":{"1min":123.878,"5min":21726.743999999999,"15min":21726.743999999999},"last":6.8849999999999998},{"interface":"front","average":{"1min":16.462,"5min":1108.894,"15min":387.70299999999997},"min":{"1min":6.4089999999999998,"5min":5.8380000000000001,"15min":5.4909999999999997},"max":{"1min":130.185,"5min":21727.993999999999,"15min":21727.993999999999},"last":8.8729999999999993}]},{"osd":1,"last update":"Mon Jun 3 09:05:34 2024","interfaces":[{"interface":"back","average":{"1min":14.727,"5min":56.755000000000003,"15min":341.09399999999999},"min":{"1min":4.6529999999999996,"5min":4.1970000000000001,"15min":4.1970000000000001},"max":{"1min":126.899,"5min":3346.4389999999999,"15min":22256.098000000002},"last":6.5860000000000003},{"interface":"front","average":{"1min":13.472,"5min":57.503,"15min":340.303},"min":{"1min":5.0590000000000002,"5min":5.0590000000000002,"15min":4.2640000000000002},"max":{"1min":126.096,"5min":3347.181,"15min":22247.317999999999},"last":11.868}]},{"osd":2,"last update":"Mon Jun 3 09:05:34 2024","interfaces":[{"interface":"back","average":{"1min":26.088999999999999,"5min":23.866,"15min":20.620999999999999},"min":{"1min":7.0869999999999997,"5min":6.8239999999999998,"15min":6.3150000000000004},"max":{"1min":337.60399999999998,"5min":393.58600000000001,"15min":393.58600000000001},"last":11.247999999999999},{"interface":"front","average":{"1min":15.800000000000001,"5min":22.584,"15min":23.245999999999999},"min":{"1min":6.0629999999999997,"5min":6.0629999999999997,"15min":6.0629999999999997},"max":{"1min":127.913,"5min":394.471,"15min":394.471},"last":13.949999999999999}]},{"osd":3,"last update":"Mon Jun 3 09:05:34 2024","interfaces":[{"interface":"back","average":{"1min":17.544,"5min":19.143999999999998,"15min":18.742000000000001},"min":{"1min":5.8719999999999999,"5min":5.4100000000000001,"15min":5.4000000000000004},"max":{"1min":127.43600000000001,"5min":322.63,"15min":322.63},"last":14.968},{"interface":"front","average":{"1min":21.327000000000002,"5min":23.559000000000001,"15min":23.797000000000001},"min":{"1min":10.023,"5min":7.484,"15min":7.484},"max":{"1min":204.517,"5min":392.67500000000001,"15min":392.67500000000001},"last":13.297000000000001}]},{"osd":5,"last update":"Mon Jun 3 09:05:34 2024","interfaces":[{"interface":"back","average":{"1min":24.274999999999999,"5min":21.274999999999999,"15min":20.988},"min":{"1min":6.7489999999999997,"5min":6.7489999999999997,"15min":6.7489999999999997},"max":{"1min":268.10199999999998,"5min":344.23399999999998,"15min":344.23399999999998},"last":12.752000000000001},{"interface":"front","average":{"1min":26.361999999999998,"5min":25.218,"15min":26.084},"min":{"1min":12.228,"5min":8.5180000000000007,"15min":8.5180000000000007},"max":{"1min":264.75,"5min":345.18700000000001,"15min":345.18700000000001},"last":15.800000000000001}]},{"osd":6,"last update":"Mon Jun 3 09:04:32 2024","interfaces":[{"interface":"back","average":{"1min":15.548,"5min":32.957999999999998,"15min":152.738},"min":{"1min":9.141,"5min":9.141,"15min":8.2319999999999993},"max":{"1min":52.57,"5min":395.238,"15min":12496.335999999999},"last":208.90000000000001},{"interface":"front","average":{"1min":12.680999999999999,"5min":25.428000000000001,"15min":149.30600000000001},"min":{"1min":6.4379999999999997,"5min":5.6319999999999997,"15min":5.6319999999999997},"max":{"1min":46.283000000000001,"5min":377.74299999999999,"15min":12616.297},"last":205.88900000000001}]},{"osd":7,"last update":"Mon Jun 3 09:05:34 2024","interfaces":[{"interface":"back","average":{"1min":18.295999999999999,"5min":19.585999999999999,"15min":21.062000000000001},"min":{"1min":6.2110000000000003,"5min":6.2110000000000003,"15min":5.8010000000000002},"max":{"1min":207.25299999999999,"5min":358.94900000000001,"15min":358.94900000000001},"last":9.4960000000000004},{"interface":"front","average":{"1min":23.712,"5min":24.024999999999999,"15min":24.640000000000001},"min":{"1min":11.824999999999999,"5min":7.7789999999999999,"15min":7.7789999999999999},"max":{"1min":209.32900000000001,"5min":346.25200000000001,"15min":380.06900000000002},"last":16.488}]}]},{"osd":2,"up_from":20,"seq":85899346290,"num_pgs":11,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":58964,"kb_used_data":5704,"kb_used_omap":1,"kb_used_meta":53246,"kb_avail":94312876,"statfs":{"total":96636764160,"available":96576385024,"internally_reserved":0,"allocated":5840896,"data_stored":2740727,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1594,"internal_metadata":54524358},"hb_peers":[0,1,3,4,5,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Mon Jun 3 09:06:23 2024","interfaces":[{"interface":"back","average":{"1min":8.9629999999999992,"5min":946.25400000000002,"15min":326.36099999999999},"min":{"1min":5.3479999999999999,"5min":4.7050000000000001,"15min":4.1420000000000003},"max":{"1min":26.015000000000001,"5min":23465.18,"15min":23465.18},"last":6.3940000000000001},{"interface":"front","average":{"1min":7.6130000000000004,"5min":944.39800000000002,"15min":326.178},"min":{"1min":4.6790000000000003,"5min":4.6790000000000003,"15min":4.6790000000000003},"max":{"1min":20.802,"5min":23465.966,"15min":23465.966},"last":4.492}]},{"osd":1,"last update":"Mon Jun 3 09:06:23 2024","interfaces":[{"interface":"back","average":{"1min":13.007,"5min":76.507000000000005,"15min":294.72899999999998},"min":{"1min":6.4089999999999998,"5min":5.4199999999999999,"15min":5.4199999999999999},"max":{"1min":85.430000000000007,"5min":4220.5290000000005,"15min":18954.987000000001},"last":7.4249999999999998},{"interface":"front","average":{"1min":15.894,"5min":82.119,"15min":302.92399999999998},"min":{"1min":7.367,"5min":7.367,"15min":7.367},"max":{"1min":98.498000000000005,"5min":4325.9489999999996,"15min":19143.834999999999},"last":9.6929999999999996}]},{"osd":3,"last update":"Mon Jun 3 09:06:23 2024","interfaces":[{"interface":"back","average":{"1min":9.7579999999999991,"5min":12.444000000000001,"15min":28.065000000000001},"min":{"1min":7,"5min":5.9290000000000003,"15min":5.2450000000000001},"max":{"1min":28.164000000000001,"5min":118.913,"15min":2594.8150000000001},"last":8.8740000000000006},{"interface":"front","average":{"1min":12.321999999999999,"5min":15.25,"15min":31.349},"min":{"1min":9.1240000000000006,"5min":8.0760000000000005,"15min":6.0780000000000003},"max":{"1min":32.360999999999997,"5min":156.34700000000001,"15min":2706.5120000000002},"last":11.180999999999999}]},{"osd":4,"last update":"Mon Jun 3 09:05:57 2024","interfaces":[{"interface":"back","average":{"1min":10.573,"5min":1266.894,"15min":438.154},"min":{"1min":8.6379999999999999,"5min":7.8789999999999996,"15min":6.2039999999999997},"max":{"1min":15.105,"5min":26660.861000000001,"15min":26660.861000000001},"last":10.538},{"interface":"front","average":{"1min":12.407999999999999,"5min":1266.7460000000001,"15min":435.75400000000002},"min":{"1min":10.266999999999999,"5min":10.266999999999999,"15min":6.2720000000000002},"max":{"1min":15.228,"5min":26616.243999999999,"15min":26616.243999999999},"last":11.879}]},{"osd":5,"last update":"Mon Jun 3 09:06:23 2024","interfaces":[{"interface":"back","average":{"1min":14.103,"5min":18.091000000000001,"15min":21.119},"min":{"1min":9.6349999999999998,"5min":9.2159999999999993,"15min":6.2050000000000001},"max":{"1min":32.890000000000001,"5min":252.31100000000001,"15min":252.31100000000001},"last":12.696},{"interface":"front","average":{"1min":11.545999999999999,"5min":16.170999999999999,"15min":20.254000000000001},"min":{"1min":7.6070000000000002,"5min":7.4850000000000003,"15min":6.8810000000000002},"max":{"1min":29.228999999999999,"5min":258.31700000000001,"15min":258.31700000000001},"last":8.6699999999999999}]},{"osd":6,"last update":"Mon Jun 3 09:04:28 2024","interfaces":[{"interface":"back","average":{"1min":14.063000000000001,"5min":24.850000000000001,"15min":106.755},"min":{"1min":9.0820000000000007,"5min":8.9649999999999999,"15min":8.9649999999999999},"max":{"1min":24.666,"5min":188.06700000000001,"15min":9521.0010000000002},"last":183.60900000000001},{"interface":"front","average":{"1min":13.933999999999999,"5min":21.081,"15min":106.142},"min":{"1min":5.8719999999999999,"5min":5.8719999999999999,"15min":5.8719999999999999},"max":{"1min":117.551,"5min":193.887,"15min":9585.6620000000003},"last":184.20599999999999}]},{"osd":7,"last update":"Mon Jun 3 09:06:23 2024","interfaces":[{"interface":"back","average":{"1min":17.056000000000001,"5min":20.055,"15min":23.268999999999998},"min":{"1min":10.353999999999999,"5min":6.6130000000000004,"15min":6.6130000000000004},"max":{"1min":88.974999999999994,"5min":242.56800000000001,"15min":242.56800000000001},"last":13.343},{"interface":"front","average":{"1min":18.885000000000002,"5min":21.460000000000001,"15min":25.489999999999998},"min":{"1min":12.497,"5min":9.0660000000000007,"15min":9.0660000000000007},"max":{"1min":102.229,"5min":181.56399999999999,"15min":214.33799999999999},"last":14.042}]}]},{"osd":3,"up_from":14,"seq":60129542504,"num_pgs":11,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":58964,"kb_used_data":5704,"kb_used_omap":1,"kb_used_meta":53246,"kb_avail":94312876,"statfs":{"total":96636764160,"available":96576385024,"internally_reserved":0,"allocated":5840896,"data_stored":2740727,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1592,"internal_metadata":54524360},"hb_peers":[0,1,2,4,5,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Mon Jun 3 09:06:25 2024","interfaces":[{"interface":"back","average":{"1min":7.79,"5min":1060.1959999999999,"15min":367.01799999999997},"min":{"1min":5.6909999999999998,"5min":5.173,"15min":4.6929999999999996},"max":{"1min":10.59,"5min":23275.167000000001,"15min":23275.167000000001},"last":5.1989999999999998},{"interface":"front","average":{"1min":11.215999999999999,"5min":1066.5799999999999,"15min":372.476},"min":{"1min":8.7080000000000002,"5min":7.2510000000000003,"15min":5.9880000000000004},"max":{"1min":13.499000000000001,"5min":23278.536,"15min":23278.536},"last":10.922000000000001}]},{"osd":1,"last update":"Mon Jun 3 09:06:25 2024","interfaces":[{"interface":"back","average":{"1min":7.3760000000000003,"5min":15.614000000000001,"15min":389.83100000000002},"min":{"1min":4.335,"5min":4.335,"15min":4.335},"max":{"1min":11.289,"5min":250.72300000000001,"15min":21658.205999999998},"last":8.5549999999999997},{"interface":"front","average":{"1min":8.9499999999999993,"5min":14.135,"15min":381.10300000000001},"min":{"1min":6.5609999999999999,"5min":5.4100000000000001,"15min":5.2779999999999996},"max":{"1min":14.51,"5min":180.44499999999999,"15min":21307.069},"last":7.04}]},{"osd":2,"last update":"Mon Jun 3 09:06:25 2024","interfaces":[{"interface":"back","average":{"1min":12.343999999999999,"5min":22.059999999999999,"15min":23.715},"min":{"1min":7.9199999999999999,"5min":6.4900000000000002,"15min":6.056},"max":{"1min":15.601000000000001,"5min":377.803,"15min":377.803},"last":12.581},{"interface":"front","average":{"1min":10.319000000000001,"5min":19.364999999999998,"15min":20.242999999999999},"min":{"1min":6.891,"5min":6.1749999999999998,"15min":5.9020000000000001},"max":{"1min":15.432,"5min":371.601,"15min":371.601},"last":10.368}]},{"osd":4,"last update":"Mon Jun 3 09:06:31 2024","interfaces":[{"interface":"back","average":{"1min":9.7460000000000004,"5min":891.23000000000002,"15min":416.48099999999999},"min":{"1min":5.6539999999999999,"5min":5.6539999999999999,"15min":5.6539999999999999},"max":{"1min":15.445,"5min":29040.635999999999,"15min":30144.939999999999},"last":8.0259999999999998},{"interface":"front","average":{"1min":12.398999999999999,"5min":897.58500000000004,"15min":421.27499999999998},"min":{"1min":8.9469999999999992,"5min":7.8109999999999999,"15min":6.2439999999999998},"max":{"1min":16.518000000000001,"5min":29076.955999999998,"15min":30180.689999999999},"last":9.1539999999999999}]},{"osd":5,"last update":"Mon Jun 3 09:06:07 2024","interfaces":[{"interface":"back","average":{"1min":12.802,"5min":19.640999999999998,"15min":22.318000000000001},"min":{"1min":9.0890000000000004,"5min":6.8810000000000002,"15min":6.7690000000000001},"max":{"1min":16.745999999999999,"5min":249.88,"15min":327.34899999999999},"last":14.225},{"interface":"front","average":{"1min":15.002000000000001,"5min":24.003,"15min":26.649999999999999},"min":{"1min":11.984,"5min":9.9149999999999991,"15min":8.9879999999999995},"max":{"1min":18.407,"5min":251.47800000000001,"15min":331.86799999999999},"last":14.91}]},{"osd":6,"last update":"Mon Jun 3 09:04:04 2024","interfaces":[{"interface":"back","average":{"1min":22.815999999999999,"5min":31.513999999999999,"15min":181.74299999999999},"min":{"1min":9.4809999999999999,"5min":7.2240000000000002,"15min":7.2240000000000002},"max":{"1min":160.56999999999999,"5min":380.613,"15min":11893.74},"last":19.359999999999999},{"interface":"front","average":{"1min":25.359999999999999,"5min":30.981000000000002,"15min":179.81399999999999},"min":{"1min":9.8870000000000005,"5min":7.3360000000000003,"15min":6.0899999999999999},"max":{"1min":193.11099999999999,"5min":379.80700000000002,"15min":11893.069},"last":8.202}]},{"osd":7,"last update":"Mon Jun 3 09:06:25 2024","interfaces":[{"interface":"back","average":{"1min":11.321999999999999,"5min":18.443999999999999,"15min":21.568999999999999},"min":{"1min":8.109,"5min":8.109,"15min":7.4279999999999999},"max":{"1min":17.113,"5min":146.267,"15min":178.76300000000001},"last":11.492000000000001},{"interface":"front","average":{"1min":12.512,"5min":20.065999999999999,"15min":24.190000000000001},"min":{"1min":7.4989999999999997,"5min":7.4989999999999997,"15min":7.4989999999999997},"max":{"1min":18.047999999999998,"5min":148.13800000000001,"15min":417.22800000000001},"last":13.419}]}]},{"osd":1,"up_from":14,"seq":60129542502,"num_pgs":6,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":59356,"kb_used_data":6096,"kb_used_omap":1,"kb_used_meta":53246,"kb_avail":94312484,"statfs":{"total":96636764160,"available":96575983616,"internally_reserved":0,"allocated":6242304,"data_stored":3138567,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":54524357},"hb_peers":[0,2,3,4,5,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Mon Jun 3 09:06:07 2024","interfaces":[{"interface":"back","average":{"1min":7.7530000000000001,"5min":185.66999999999999,"15min":77.347999999999999},"min":{"1min":5.0229999999999997,"5min":4.9279999999999999,"15min":4.5449999999999999},"max":{"1min":11.896000000000001,"5min":7470.616,"15min":7470.616},"last":10.537000000000001},{"interface":"front","average":{"1min":9.9220000000000006,"5min":187.14599999999999,"15min":76.325999999999993},"min":{"1min":5.3159999999999998,"5min":5.3159999999999998,"15min":5.3159999999999998},"max":{"1min":18.338000000000001,"5min":7477.0829999999996,"15min":7477.0829999999996},"last":9.5649999999999995}]},{"osd":2,"last update":"Mon Jun 3 09:06:20 2024","interfaces":[{"interface":"back","average":{"1min":10.340999999999999,"5min":17.369,"15min":23.670999999999999},"min":{"1min":6.9740000000000002,"5min":5.8650000000000002,"15min":5.8650000000000002},"max":{"1min":15.452,"5min":228,"15min":657.38},"last":11.122999999999999},{"interface":"front","average":{"1min":13.387,"5min":23.355,"15min":31.524000000000001},"min":{"1min":9.077,"5min":9.077,"15min":7.2160000000000002},"max":{"1min":19.701000000000001,"5min":229.465,"15min":700.06500000000005},"last":15.970000000000001}]},{"osd":3,"last update":"Mon Jun 3 09:06:04 2024","interfaces":[{"interface":"back","average":{"1min":8.6110000000000007,"5min":15.733000000000001,"15min":26.646999999999998},"min":{"1min":5.827,"5min":5.4640000000000004,"15min":5.4640000000000004},"max":{"1min":13.105,"5min":121.619,"15min":1676.009},"last":11.763},{"interface":"front","average":{"1min":9.5850000000000009,"5min":16.161999999999999,"15min":29.065999999999999},"min":{"1min":6.3630000000000004,"5min":5.7699999999999996,"15min":5.7699999999999996},"max":{"1min":15.166,"5min":132.036,"15min":1766.5840000000001},"last":8.7479999999999993}]},{"osd":4,"last update":"Mon Jun 3 09:05:31 2024","interfaces":[{"interface":"back","average":{"1min":10.996,"5min":954.46500000000003,"15min":1281.6859999999999},"min":{"1min":7.702,"5min":6.1760000000000002,"15min":6.0970000000000004},"max":{"1min":21.538,"5min":28044.862000000001,"15min":59438.017999999996},"last":9.3219999999999992},{"interface":"front","average":{"1min":13.646000000000001,"5min":961.24599999999998,"15min":1285.26},"min":{"1min":9.7200000000000006,"5min":9.3940000000000001,"15min":6.4640000000000004},"max":{"1min":23.277999999999999,"5min":28045.624,"15min":59439.283000000003},"last":12.798}]},{"osd":5,"last update":"Mon Jun 3 09:06:04 2024","interfaces":[{"interface":"back","average":{"1min":12.478999999999999,"5min":20.228000000000002,"15min":25.997},"min":{"1min":8.6430000000000007,"5min":7.9720000000000004,"15min":7.234},"max":{"1min":17.442,"5min":139.43199999999999,"15min":701.22000000000003},"last":13.678000000000001},{"interface":"front","average":{"1min":13.507999999999999,"5min":25.760999999999999,"15min":27.553999999999998},"min":{"1min":10.042999999999999,"5min":7.258,"15min":6.242},"max":{"1min":20.611000000000001,"5min":278.63,"15min":598.35000000000002},"last":14.173}]},{"osd":6,"last update":"Mon Jun 3 09:03:54 2024","interfaces":[{"interface":"back","average":{"1min":28.524000000000001,"5min":20.654,"15min":23.878},"min":{"1min":6.0309999999999997,"5min":6.0309999999999997,"15min":6.0309999999999997},"max":{"1min":167.221,"5min":167.221,"15min":704.245},"last":18.721},{"interface":"front","average":{"1min":27.606000000000002,"5min":30.131,"15min":28.628},"min":{"1min":9.8650000000000002,"5min":9.7200000000000006,"15min":8.3580000000000005},"max":{"1min":123.938,"5min":358.23000000000002,"15min":634.05499999999995},"last":173.113}]},{"osd":7,"last update":"Mon Jun 3 09:06:20 2024","interfaces":[{"interface":"back","average":{"1min":14.66,"5min":23.234999999999999,"15min":31.097000000000001},"min":{"1min":11.250999999999999,"5min":8.0559999999999992,"15min":8.0559999999999992},"max":{"1min":21.507999999999999,"5min":270.95100000000002,"15min":703.07799999999997},"last":15.116},{"interface":"front","average":{"1min":15.548,"5min":28.760000000000002,"15min":28.925000000000001},"min":{"1min":12.759,"5min":8.2089999999999996,"15min":7.7599999999999998},"max":{"1min":22.408000000000001,"5min":360.99900000000002,"15min":608.44100000000003},"last":16.640000000000001}]}]},{"osd":0,"up_from":14,"seq":60129542504,"num_pgs":10,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":81108,"kb_used_data":5704,"kb_used_omap":1,"kb_used_meta":75390,"kb_avail":94290732,"statfs":{"total":96636764160,"available":96553709568,"internally_reserved":0,"allocated":5840896,"data_stored":2740727,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1593,"internal_metadata":77199815},"hb_peers":[1,2,3,4,5,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[{"osd":1,"last update":"Mon Jun 3 09:06:04 2024","interfaces":[{"interface":"back","average":{"1min":6.9960000000000004,"5min":56.997,"15min":339.29700000000003},"min":{"1min":4.4530000000000003,"5min":4.4530000000000003,"15min":4.4100000000000001},"max":{"1min":11.720000000000001,"5min":2414.4749999999999,"15min":20691.453000000001},"last":5.7039999999999997},{"interface":"front","average":{"1min":9.548,"5min":57.258000000000003,"15min":337.791},"min":{"1min":5.2300000000000004,"5min":5.2300000000000004,"15min":4.6050000000000004},"max":{"1min":15.337999999999999,"5min":2412.9549999999999,"15min":20640.792000000001},"last":8.2159999999999993}]},{"osd":2,"last update":"Mon Jun 3 09:06:22 2024","interfaces":[{"interface":"back","average":{"1min":11.154,"5min":26.515999999999998,"15min":22.882000000000001},"min":{"1min":5.7869999999999999,"5min":5.766,"15min":5.766},"max":{"1min":72.269999999999996,"5min":627.40700000000004,"15min":627.40700000000004},"last":11.311},{"interface":"front","average":{"1min":14.353999999999999,"5min":33.304000000000002,"15min":29.885000000000002},"min":{"1min":8.2149999999999999,"5min":7.8630000000000004,"15min":7.4089999999999998},"max":{"1min":73.418000000000006,"5min":635.62599999999998,"15min":635.62599999999998},"last":13.371}]},{"osd":3,"last update":"Mon Jun 3 09:06:04 2024","interfaces":[{"interface":"back","average":{"1min":12.105,"5min":26.925000000000001,"15min":26.584},"min":{"1min":7.8410000000000002,"5min":7.1280000000000001,"15min":6.1909999999999998},"max":{"1min":18.425999999999998,"5min":389.72699999999998,"15min":389.72699999999998},"last":9.4990000000000006},{"interface":"front","average":{"1min":13.794,"5min":29.771999999999998,"15min":28.815000000000001},"min":{"1min":10.084,"5min":9.4920000000000009,"15min":6.1849999999999996},"max":{"1min":17.373999999999999,"5min":390.23200000000003,"15min":494.92899999999997},"last":10.595000000000001}]},{"osd":4,"last update":"Mon Jun 3 09:06:27 2024","interfaces":[{"interface":"back","average":{"1min":11.352,"5min":745.30600000000004,"15min":363.48099999999999},"min":{"1min":6.415,"5min":6.415,"15min":6.415},"max":{"1min":16.724,"5min":20248.616000000002,"15min":25555.212},"last":12.867000000000001},{"interface":"front","average":{"1min":13.545,"5min":744.76999999999998,"15min":361.03100000000001},"min":{"1min":9.3889999999999993,"5min":8.8940000000000001,"15min":7.1429999999999998},"max":{"1min":18.411000000000001,"5min":20250.195,"15min":25556.159},"last":14.318}]},{"osd":5,"last update":"Mon Jun 3 09:06:17 2024","interfaces":[{"interface":"back","average":{"1min":9.7040000000000006,"5min":26.533999999999999,"15min":28.683},"min":{"1min":6.8129999999999997,"5min":6.8129999999999997,"15min":6.8129999999999997},"max":{"1min":14.942,"5min":455.78699999999998,"15min":605.05899999999997},"last":6.6539999999999999},{"interface":"front","average":{"1min":14.279,"5min":28.398,"15min":26.192},"min":{"1min":11.696,"5min":6.4420000000000002,"15min":6.2030000000000003},"max":{"1min":17.879000000000001,"5min":391.31200000000001,"15min":541.11000000000001},"last":11.938000000000001}]},{"osd":6,"last update":"Mon Jun 3 09:04:30 2024","interfaces":[{"interface":"back","average":{"1min":52.496000000000002,"5min":33.139000000000003,"15min":160.83199999999999},"min":{"1min":7.5289999999999999,"5min":7.1040000000000001,"15min":6.3739999999999997},"max":{"1min":457.91699999999997,"5min":457.91699999999997,"15min":11115.258},"last":18.959},{"interface":"front","average":{"1min":49.670000000000002,"5min":33.704999999999998,"15min":162.791},"min":{"1min":12.247999999999999,"5min":11.019,"15min":9.9760000000000009},"max":{"1min":407.05099999999999,"5min":407.05099999999999,"15min":11092.194},"last":17.707000000000001}]},{"osd":7,"last update":"Mon Jun 3 09:06:22 2024","interfaces":[{"interface":"back","average":{"1min":11.625999999999999,"5min":26.375,"15min":28.972999999999999},"min":{"1min":6.4480000000000004,"5min":6.4480000000000004,"15min":6.4480000000000004},"max":{"1min":15.352,"5min":461.41699999999997,"15min":598.33699999999999},"last":7.1369999999999996},{"interface":"front","average":{"1min":14.013,"5min":29.696000000000002,"15min":33.630000000000003},"min":{"1min":10.938000000000001,"5min":10.58,"15min":10.263999999999999},"max":{"1min":16.603000000000002,"5min":463.10599999999999,"15min":592.80499999999995},"last":8.7810000000000006}]}]}],"pool_statfs":[{"poolid":1,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":401408,"data_stored":397840,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":5,"total":0,"available":0,"internally_reserved":0,"allocated":401408,"data_stored":397840,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":4,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":5,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":6,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":17,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":17,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":17,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":17,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":17,"osd":4,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":17,"osd":5,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":5,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":17,"osd":6,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":5,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":17,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0}]}} 2024-06-03T09:06:37.923 INFO:tasks.ceph.ceph_manager.ceph:PG 17.c is not active+clean 2024-06-03T09:06:37.923 INFO:tasks.ceph.ceph_manager.ceph:{'pgid': '17.c', 'version': "0'0", 'reported_seq': 953, 'reported_epoch': 1009, 'state': 'active+undersized', 'last_fresh': '2024-06-03T09:06:13.668156+0000', 'last_change': '2024-06-03T09:06:03.440074+0000', 'last_active': '2024-06-03T09:06:13.668156+0000', 'last_peered': '2024-06-03T09:06:13.668156+0000', 'last_clean': '2024-06-03T09:05:02.105638+0000', 'last_became_active': '2024-06-03T09:06:03.440074+0000', 'last_became_peered': '2024-06-03T09:06:03.440074+0000', 'last_unstale': '2024-06-03T09:06:13.668156+0000', 'last_undegraded': '2024-06-03T09:06:13.668156+0000', 'last_fullsized': '2024-06-03T09:06:03.377982+0000', 'mapping_epoch': 999, 'log_start': "0'0", 'ondisk_log_start': "0'0", 'created': 28, 'last_epoch_clean': 255, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2024-06-03T08:37:55.614497+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2024-06-03T08:37:55.614497+0000', 'last_clean_scrub_stamp': '2024-06-03T08:37:55.614497+0000', 'objects_scrubbed': 0, 'log_size': 0, 'log_dups_size': 0, 'ondisk_log_size': 0, 'stats_invalid': False, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': 'periodic scrub scheduled @ 2024-06-04T10:34:20.808557+0000', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 0, 'num_objects': 0, 'num_object_clones': 0, 'num_object_copies': 0, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 0, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 0, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 0, 'num_write_kb': 0, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 0, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [3], 'acting': [3], 'avail_no_missing': ['3'], 'object_location_counts': [], 'blocked_by': [], 'up_primary': 3, 'acting_primary': 3, 'purged_snaps': []} 2024-06-03T09:06:37.923 INFO:tasks.ceph.ceph_manager.ceph:PG 17.f is not active+clean 2024-06-03T09:06:37.923 INFO:tasks.ceph.ceph_manager.ceph:{'pgid': '17.f', 'version': "0'0", 'reported_seq': 148, 'reported_epoch': 1009, 'state': 'active+undersized', 'last_fresh': '2024-06-03T09:06:14.450543+0000', 'last_change': '2024-06-03T09:06:03.420285+0000', 'last_active': '2024-06-03T09:06:14.450543+0000', 'last_peered': '2024-06-03T09:06:14.450543+0000', 'last_clean': '2024-06-03T08:37:55.614497+0000', 'last_became_active': '2024-06-03T09:06:03.420285+0000', 'last_became_peered': '2024-06-03T09:06:03.420285+0000', 'last_unstale': '2024-06-03T09:06:14.450543+0000', 'last_undegraded': '2024-06-03T09:06:14.450543+0000', 'last_fullsized': '2024-06-03T09:06:03.380616+0000', 'mapping_epoch': 999, 'log_start': "0'0", 'ondisk_log_start': "0'0", 'created': 28, 'last_epoch_clean': 255, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2024-06-03T08:37:55.614497+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2024-06-03T08:37:55.614497+0000', 'last_clean_scrub_stamp': '2024-06-03T08:37:55.614497+0000', 'objects_scrubbed': 0, 'log_size': 0, 'log_dups_size': 0, 'ondisk_log_size': 0, 'stats_invalid': False, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': 'periodic scrub scheduled @ 2024-06-04T15:39:41.187731+0000', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 0, 'num_objects': 0, 'num_object_clones': 0, 'num_object_copies': 0, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 0, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 0, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 0, 'num_write_kb': 0, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 0, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [4], 'acting': [4], 'avail_no_missing': ['4'], 'object_location_counts': [], 'blocked_by': [], 'up_primary': 4, 'acting_primary': 4, 'purged_snaps': []} 2024-06-03T09:06:37.923 INFO:tasks.ceph.ceph_manager.ceph:PG 17.e is not active+clean 2024-06-03T09:06:37.923 INFO:tasks.ceph.ceph_manager.ceph:{'pgid': '17.e', 'version': "0'0", 'reported_seq': 149, 'reported_epoch': 1009, 'state': 'active+undersized', 'last_fresh': '2024-06-03T09:06:13.726992+0000', 'last_change': '2024-06-03T09:06:03.412887+0000', 'last_active': '2024-06-03T09:06:13.726992+0000', 'last_peered': '2024-06-03T09:06:13.726992+0000', 'last_clean': '2024-06-03T08:37:55.614497+0000', 'last_became_active': '2024-06-03T09:06:03.412887+0000', 'last_became_peered': '2024-06-03T09:06:03.412887+0000', 'last_unstale': '2024-06-03T09:06:13.726992+0000', 'last_undegraded': '2024-06-03T09:06:13.726992+0000', 'last_fullsized': '2024-06-03T09:06:03.332482+0000', 'mapping_epoch': 999, 'log_start': "0'0", 'ondisk_log_start': "0'0", 'created': 28, 'last_epoch_clean': 255, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2024-06-03T08:37:55.614497+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2024-06-03T08:37:55.614497+0000', 'last_clean_scrub_stamp': '2024-06-03T08:37:55.614497+0000', 'objects_scrubbed': 0, 'log_size': 0, 'log_dups_size': 0, 'ondisk_log_size': 0, 'stats_invalid': False, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': 'periodic scrub scheduled @ 2024-06-04T12:23:22.511601+0000', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 0, 'num_objects': 0, 'num_object_clones': 0, 'num_object_copies': 0, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 0, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 0, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 0, 'num_write_kb': 0, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 0, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [5], 'acting': [5], 'avail_no_missing': ['5'], 'object_location_counts': [], 'blocked_by': [], 'up_primary': 5, 'acting_primary': 5, 'purged_snaps': []} 2024-06-03T09:06:37.923 INFO:tasks.ceph.ceph_manager.ceph:PG 17.8 is not active+clean 2024-06-03T09:06:37.923 INFO:tasks.ceph.ceph_manager.ceph:{'pgid': '17.8', 'version': "0'0", 'reported_seq': 961, 'reported_epoch': 1009, 'state': 'active+undersized', 'last_fresh': '2024-06-03T09:06:14.430375+0000', 'last_change': '2024-06-03T09:06:03.388786+0000', 'last_active': '2024-06-03T09:06:14.430375+0000', 'last_peered': '2024-06-03T09:06:14.430375+0000', 'last_clean': '2024-06-03T09:04:43.297780+0000', 'last_became_active': '2024-06-03T09:06:03.388786+0000', 'last_became_peered': '2024-06-03T09:06:03.388786+0000', 'last_unstale': '2024-06-03T09:06:14.430375+0000', 'last_undegraded': '2024-06-03T09:06:14.430375+0000', 'last_fullsized': '2024-06-03T09:06:03.331754+0000', 'mapping_epoch': 999, 'log_start': "0'0", 'ondisk_log_start': "0'0", 'created': 28, 'last_epoch_clean': 255, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2024-06-03T08:37:55.614497+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2024-06-03T08:37:55.614497+0000', 'last_clean_scrub_stamp': '2024-06-03T08:37:55.614497+0000', 'objects_scrubbed': 0, 'log_size': 0, 'log_dups_size': 0, 'ondisk_log_size': 0, 'stats_invalid': False, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': 'periodic scrub scheduled @ 2024-06-04T09:18:44.765547+0000', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 0, 'num_objects': 0, 'num_object_clones': 0, 'num_object_copies': 0, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 0, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 0, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 0, 'num_write_kb': 0, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 0, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [0], 'acting': [0], 'avail_no_missing': ['0'], 'object_location_counts': [], 'blocked_by': [], 'up_primary': 0, 'acting_primary': 0, 'purged_snaps': []} 2024-06-03T09:06:37.923 INFO:tasks.ceph.ceph_manager.ceph:PG 17.b is not active+clean 2024-06-03T09:06:37.923 INFO:tasks.ceph.ceph_manager.ceph:{'pgid': '17.b', 'version': "0'0", 'reported_seq': 149, 'reported_epoch': 1009, 'state': 'active+undersized', 'last_fresh': '2024-06-03T09:06:14.433735+0000', 'last_change': '2024-06-03T09:06:03.451203+0000', 'last_active': '2024-06-03T09:06:14.433735+0000', 'last_peered': '2024-06-03T09:06:14.433735+0000', 'last_clean': '2024-06-03T08:37:55.614497+0000', 'last_became_active': '2024-06-03T09:06:03.451203+0000', 'last_became_peered': '2024-06-03T09:06:03.451203+0000', 'last_unstale': '2024-06-03T09:06:14.433735+0000', 'last_undegraded': '2024-06-03T09:06:14.433735+0000', 'last_fullsized': '2024-06-03T09:06:03.375523+0000', 'mapping_epoch': 999, 'log_start': "0'0", 'ondisk_log_start': "0'0", 'created': 28, 'last_epoch_clean': 255, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2024-06-03T08:37:55.614497+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2024-06-03T08:37:55.614497+0000', 'last_clean_scrub_stamp': '2024-06-03T08:37:55.614497+0000', 'objects_scrubbed': 0, 'log_size': 0, 'log_dups_size': 0, 'ondisk_log_size': 0, 'stats_invalid': False, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': 'periodic scrub scheduled @ 2024-06-04T10:16:19.813495+0000', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 0, 'num_objects': 0, 'num_object_clones': 0, 'num_object_copies': 0, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 0, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 0, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 0, 'num_write_kb': 0, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 0, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [2], 'acting': [2], 'avail_no_missing': ['2'], 'object_location_counts': [], 'blocked_by': [], 'up_primary': 2, 'acting_primary': 2, 'purged_snaps': []} 2024-06-03T09:06:37.924 INFO:tasks.ceph.ceph_manager.ceph:PG 17.4 is not active+clean 2024-06-03T09:06:37.924 INFO:tasks.ceph.ceph_manager.ceph:{'pgid': '17.4', 'version': "0'0", 'reported_seq': 149, 'reported_epoch': 1009, 'state': 'active+undersized', 'last_fresh': '2024-06-03T09:06:13.941522+0000', 'last_change': '2024-06-03T09:06:03.441889+0000', 'last_active': '2024-06-03T09:06:13.941522+0000', 'last_peered': '2024-06-03T09:06:13.941522+0000', 'last_clean': '2024-06-03T08:37:55.614497+0000', 'last_became_active': '2024-06-03T09:06:03.441889+0000', 'last_became_peered': '2024-06-03T09:06:03.441889+0000', 'last_unstale': '2024-06-03T09:06:13.941522+0000', 'last_undegraded': '2024-06-03T09:06:13.941522+0000', 'last_fullsized': '2024-06-03T09:06:03.359582+0000', 'mapping_epoch': 999, 'log_start': "0'0", 'ondisk_log_start': "0'0", 'created': 28, 'last_epoch_clean': 255, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2024-06-03T08:37:55.614497+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2024-06-03T08:37:55.614497+0000', 'last_clean_scrub_stamp': '2024-06-03T08:37:55.614497+0000', 'objects_scrubbed': 0, 'log_size': 0, 'log_dups_size': 0, 'ondisk_log_size': 0, 'stats_invalid': False, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': 'periodic scrub scheduled @ 2024-06-04T12:49:22.907141+0000', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 0, 'num_objects': 0, 'num_object_clones': 0, 'num_object_copies': 0, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 0, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 0, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 0, 'num_write_kb': 0, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 0, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [7], 'acting': [7], 'avail_no_missing': ['7'], 'object_location_counts': [], 'blocked_by': [], 'up_primary': 7, 'acting_primary': 7, 'purged_snaps': []} 2024-06-03T09:06:37.924 INFO:tasks.ceph.ceph_manager.ceph:PG 17.6 is not active+clean 2024-06-03T09:06:37.924 INFO:tasks.ceph.ceph_manager.ceph:{'pgid': '17.6', 'version': "0'0", 'reported_seq': 960, 'reported_epoch': 1009, 'state': 'active+undersized', 'last_fresh': '2024-06-03T09:06:14.453110+0000', 'last_change': '2024-06-03T09:06:03.428677+0000', 'last_active': '2024-06-03T09:06:14.453110+0000', 'last_peered': '2024-06-03T09:06:14.453110+0000', 'last_clean': '2024-06-03T09:04:43.312352+0000', 'last_became_active': '2024-06-03T09:06:03.428677+0000', 'last_became_peered': '2024-06-03T09:06:03.428677+0000', 'last_unstale': '2024-06-03T09:06:14.453110+0000', 'last_undegraded': '2024-06-03T09:06:14.453110+0000', 'last_fullsized': '2024-06-03T09:06:03.359909+0000', 'mapping_epoch': 999, 'log_start': "0'0", 'ondisk_log_start': "0'0", 'created': 28, 'last_epoch_clean': 255, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2024-06-03T08:37:55.614497+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2024-06-03T08:37:55.614497+0000', 'last_clean_scrub_stamp': '2024-06-03T08:37:55.614497+0000', 'objects_scrubbed': 0, 'log_size': 0, 'log_dups_size': 0, 'ondisk_log_size': 0, 'stats_invalid': False, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': 'periodic scrub scheduled @ 2024-06-04T15:38:02.180241+0000', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 0, 'num_objects': 0, 'num_object_clones': 0, 'num_object_copies': 0, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 0, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 0, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 0, 'num_write_kb': 0, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 0, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [0], 'acting': [0], 'avail_no_missing': ['0'], 'object_location_counts': [], 'blocked_by': [], 'up_primary': 0, 'acting_primary': 0, 'purged_snaps': []} 2024-06-03T09:06:37.924 INFO:tasks.ceph.ceph_manager.ceph:PG 17.16 is not active+clean 2024-06-03T09:06:37.924 INFO:tasks.ceph.ceph_manager.ceph:{'pgid': '17.16', 'version': "0'0", 'reported_seq': 972, 'reported_epoch': 1009, 'state': 'active+undersized', 'last_fresh': '2024-06-03T09:06:14.441665+0000', 'last_change': '2024-06-03T09:06:03.478615+0000', 'last_active': '2024-06-03T09:06:14.441665+0000', 'last_peered': '2024-06-03T09:06:14.441665+0000', 'last_clean': '2024-06-03T09:04:43.328415+0000', 'last_became_active': '2024-06-03T09:06:03.478615+0000', 'last_became_peered': '2024-06-03T09:06:03.478615+0000', 'last_unstale': '2024-06-03T09:06:14.441665+0000', 'last_undegraded': '2024-06-03T09:06:14.441665+0000', 'last_fullsized': '2024-06-03T09:06:03.402504+0000', 'mapping_epoch': 999, 'log_start': "0'0", 'ondisk_log_start': "0'0", 'created': 28, 'last_epoch_clean': 255, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2024-06-03T08:37:55.614497+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2024-06-03T08:37:55.614497+0000', 'last_clean_scrub_stamp': '2024-06-03T08:37:55.614497+0000', 'objects_scrubbed': 0, 'log_size': 0, 'log_dups_size': 0, 'ondisk_log_size': 0, 'stats_invalid': False, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': 'periodic scrub scheduled @ 2024-06-04T08:47:08.079233+0000', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 0, 'num_objects': 0, 'num_object_clones': 0, 'num_object_copies': 0, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 0, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 0, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 0, 'num_write_kb': 0, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 0, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [2], 'acting': [2], 'avail_no_missing': ['2'], 'object_location_counts': [], 'blocked_by': [], 'up_primary': 2, 'acting_primary': 2, 'purged_snaps': []} 2024-06-03T09:06:37.924 INFO:tasks.ceph.ceph_manager.ceph:PG 2.4 is not active+clean 2024-06-03T09:06:37.924 INFO:tasks.ceph.ceph_manager.ceph:{'pgid': '2.4', 'version': "0'0", 'reported_seq': 952, 'reported_epoch': 1010, 'state': 'active+undersized', 'last_fresh': '2024-06-03T09:06:18.331485+0000', 'last_change': '2024-06-03T09:06:03.387171+0000', 'last_active': '2024-06-03T09:06:18.331485+0000', 'last_peered': '2024-06-03T09:06:18.331485+0000', 'last_clean': '2024-06-03T09:04:43.202645+0000', 'last_became_active': '2024-06-03T09:06:03.387171+0000', 'last_became_peered': '2024-06-03T09:06:03.387171+0000', 'last_unstale': '2024-06-03T09:06:18.331485+0000', 'last_undegraded': '2024-06-03T09:06:18.331485+0000', 'last_fullsized': '2024-06-03T09:06:03.283787+0000', 'mapping_epoch': 999, 'log_start': "0'0", 'ondisk_log_start': "0'0", 'created': 23, 'last_epoch_clean': 255, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2024-06-03T08:36:28.444764+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2024-06-03T08:36:28.444764+0000', 'last_clean_scrub_stamp': '2024-06-03T08:36:28.444764+0000', 'objects_scrubbed': 0, 'log_size': 0, 'log_dups_size': 0, 'ondisk_log_size': 0, 'stats_invalid': False, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': 'periodic scrub scheduled @ 2024-06-04T20:17:43.485023+0000', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 0, 'num_objects': 0, 'num_object_clones': 0, 'num_object_copies': 0, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 0, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 0, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 0, 'num_write_kb': 0, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 0, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [5], 'acting': [5], 'avail_no_missing': ['5'], 'object_location_counts': [], 'blocked_by': [], 'up_primary': 5, 'acting_primary': 5, 'purged_snaps': []} 2024-06-03T09:06:37.924 INFO:tasks.ceph.ceph_manager.ceph:PG 2.2 is not active+clean 2024-06-03T09:06:37.924 INFO:tasks.ceph.ceph_manager.ceph:{'pgid': '2.2', 'version': "0'0", 'reported_seq': 963, 'reported_epoch': 1009, 'state': 'active+undersized', 'last_fresh': '2024-06-03T09:06:13.657892+0000', 'last_change': '2024-06-03T09:06:03.410020+0000', 'last_active': '2024-06-03T09:06:13.657892+0000', 'last_peered': '2024-06-03T09:06:13.657892+0000', 'last_clean': '2024-06-03T09:05:37.145889+0000', 'last_became_active': '2024-06-03T09:06:03.410020+0000', 'last_became_peered': '2024-06-03T09:06:03.410020+0000', 'last_unstale': '2024-06-03T09:06:13.657892+0000', 'last_undegraded': '2024-06-03T09:06:13.657892+0000', 'last_fullsized': '2024-06-03T09:06:03.346864+0000', 'mapping_epoch': 999, 'log_start': "0'0", 'ondisk_log_start': "0'0", 'created': 23, 'last_epoch_clean': 255, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2024-06-03T08:36:28.444764+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2024-06-03T08:36:28.444764+0000', 'last_clean_scrub_stamp': '2024-06-03T08:36:28.444764+0000', 'objects_scrubbed': 0, 'log_size': 0, 'log_dups_size': 0, 'ondisk_log_size': 0, 'stats_invalid': False, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': 'periodic scrub scheduled @ 2024-06-04T09:57:12.423360+0000', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 0, 'num_objects': 0, 'num_object_clones': 0, 'num_object_copies': 0, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 0, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 0, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 0, 'num_write_kb': 0, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 0, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [3], 'acting': [3], 'avail_no_missing': ['3'], 'object_location_counts': [], 'blocked_by': [], 'up_primary': 3, 'acting_primary': 3, 'purged_snaps': []} 2024-06-03T09:06:37.925 INFO:tasks.ceph.ceph_manager.ceph:PG 17.1a is not active+clean 2024-06-03T09:06:37.925 INFO:tasks.ceph.ceph_manager.ceph:{'pgid': '17.1a', 'version': "0'0", 'reported_seq': 960, 'reported_epoch': 1009, 'state': 'active+undersized', 'last_fresh': '2024-06-03T09:06:14.431224+0000', 'last_change': '2024-06-03T09:06:03.390734+0000', 'last_active': '2024-06-03T09:06:14.431224+0000', 'last_peered': '2024-06-03T09:06:14.431224+0000', 'last_clean': '2024-06-03T09:05:20.322313+0000', 'last_became_active': '2024-06-03T09:06:03.390734+0000', 'last_became_peered': '2024-06-03T09:06:03.390734+0000', 'last_unstale': '2024-06-03T09:06:14.431224+0000', 'last_undegraded': '2024-06-03T09:06:14.431224+0000', 'last_fullsized': '2024-06-03T09:06:03.343197+0000', 'mapping_epoch': 999, 'log_start': "0'0", 'ondisk_log_start': "0'0", 'created': 28, 'last_epoch_clean': 254, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2024-06-03T08:37:55.614497+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2024-06-03T08:37:55.614497+0000', 'last_clean_scrub_stamp': '2024-06-03T08:37:55.614497+0000', 'objects_scrubbed': 0, 'log_size': 0, 'log_dups_size': 0, 'ondisk_log_size': 0, 'stats_invalid': False, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': 'periodic scrub scheduled @ 2024-06-04T15:56:47.628875+0000', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 0, 'num_objects': 0, 'num_object_clones': 0, 'num_object_copies': 0, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 0, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 0, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 0, 'num_write_kb': 0, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 0, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [4], 'acting': [4], 'avail_no_missing': ['4'], 'object_location_counts': [], 'blocked_by': [], 'up_primary': 4, 'acting_primary': 4, 'purged_snaps': []} 2024-06-03T09:06:37.925 INFO:tasks.ceph.ceph_manager.ceph:PG 17.1e is not active+clean 2024-06-03T09:06:37.925 INFO:tasks.ceph.ceph_manager.ceph:{'pgid': '17.1e', 'version': "0'0", 'reported_seq': 148, 'reported_epoch': 1009, 'state': 'active+undersized', 'last_fresh': '2024-06-03T09:06:14.445917+0000', 'last_change': '2024-06-03T09:06:03.406181+0000', 'last_active': '2024-06-03T09:06:14.445917+0000', 'last_peered': '2024-06-03T09:06:14.445917+0000', 'last_clean': '2024-06-03T08:37:55.614497+0000', 'last_became_active': '2024-06-03T09:06:03.406181+0000', 'last_became_peered': '2024-06-03T09:06:03.406181+0000', 'last_unstale': '2024-06-03T09:06:14.445917+0000', 'last_undegraded': '2024-06-03T09:06:14.445917+0000', 'last_fullsized': '2024-06-03T09:06:03.363739+0000', 'mapping_epoch': 999, 'log_start': "0'0", 'ondisk_log_start': "0'0", 'created': 28, 'last_epoch_clean': 255, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2024-06-03T08:37:55.614497+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2024-06-03T08:37:55.614497+0000', 'last_clean_scrub_stamp': '2024-06-03T08:37:55.614497+0000', 'objects_scrubbed': 0, 'log_size': 0, 'log_dups_size': 0, 'ondisk_log_size': 0, 'stats_invalid': False, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': 'periodic scrub scheduled @ 2024-06-04T11:45:24.486600+0000', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 0, 'num_objects': 0, 'num_object_clones': 0, 'num_object_copies': 0, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 0, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 0, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 0, 'num_write_kb': 0, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 0, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [4], 'acting': [4], 'avail_no_missing': ['4'], 'object_location_counts': [], 'blocked_by': [], 'up_primary': 4, 'acting_primary': 4, 'purged_snaps': []} 2024-06-03T09:06:37.925 INFO:tasks.ceph.ceph_manager.ceph:PG 17.1c is not active+clean 2024-06-03T09:06:37.925 INFO:tasks.ceph.ceph_manager.ceph:{'pgid': '17.1c', 'version': "0'0", 'reported_seq': 944, 'reported_epoch': 1010, 'state': 'active+undersized', 'last_fresh': '2024-06-03T09:06:18.327492+0000', 'last_change': '2024-06-03T09:06:03.382210+0000', 'last_active': '2024-06-03T09:06:18.327492+0000', 'last_peered': '2024-06-03T09:06:18.327492+0000', 'last_clean': '2024-06-03T09:04:43.199660+0000', 'last_became_active': '2024-06-03T09:06:03.382210+0000', 'last_became_peered': '2024-06-03T09:06:03.382210+0000', 'last_unstale': '2024-06-03T09:06:18.327492+0000', 'last_undegraded': '2024-06-03T09:06:18.327492+0000', 'last_fullsized': '2024-06-03T09:06:03.277874+0000', 'mapping_epoch': 999, 'log_start': "0'0", 'ondisk_log_start': "0'0", 'created': 28, 'last_epoch_clean': 255, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2024-06-03T08:37:55.614497+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2024-06-03T08:37:55.614497+0000', 'last_clean_scrub_stamp': '2024-06-03T08:37:55.614497+0000', 'objects_scrubbed': 0, 'log_size': 0, 'log_dups_size': 0, 'ondisk_log_size': 0, 'stats_invalid': False, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': 'periodic scrub scheduled @ 2024-06-04T12:18:50.013884+0000', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 0, 'num_objects': 0, 'num_object_clones': 0, 'num_object_copies': 0, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 0, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 0, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 0, 'num_write_kb': 0, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 0, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [5], 'acting': [5], 'avail_no_missing': ['5'], 'object_location_counts': [], 'blocked_by': [], 'up_primary': 5, 'acting_primary': 5, 'purged_snaps': []} 2024-06-03T09:06:37.925 INFO:tasks.ceph.ceph_manager.ceph:PG 17.1 is not active+clean 2024-06-03T09:06:37.925 INFO:tasks.ceph.ceph_manager.ceph:{'pgid': '17.1', 'version': "65'7", 'reported_seq': 959, 'reported_epoch': 1009, 'state': 'active+undersized+degraded', 'last_fresh': '2024-06-03T09:06:13.709209+0000', 'last_change': '2024-06-03T09:06:03.370660+0000', 'last_active': '2024-06-03T09:06:13.709209+0000', 'last_peered': '2024-06-03T09:06:13.709209+0000', 'last_clean': '2024-06-03T09:05:03.733864+0000', 'last_became_active': '2024-06-03T09:06:03.370660+0000', 'last_became_peered': '2024-06-03T09:06:03.370660+0000', 'last_unstale': '2024-06-03T09:06:13.709209+0000', 'last_undegraded': '2024-06-03T09:06:03.266076+0000', 'last_fullsized': '2024-06-03T09:06:03.264525+0000', 'mapping_epoch': 999, 'log_start': "0'0", 'ondisk_log_start': "0'0", 'created': 28, 'last_epoch_clean': 255, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2024-06-03T08:37:55.614497+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2024-06-03T08:37:55.614497+0000', 'last_clean_scrub_stamp': '2024-06-03T08:37:55.614497+0000', 'objects_scrubbed': 0, 'log_size': 7, 'log_dups_size': 0, 'ondisk_log_size': 7, 'stats_invalid': False, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': 'periodic scrub scheduled @ 2024-06-04T11:55:18.194283+0000', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 5, 'num_objects': 1, 'num_object_clones': 0, 'num_object_copies': 2, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 1, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 1, 'num_whiteouts': 0, 'num_read': 6, 'num_read_kb': 6, 'num_write': 7, 'num_write_kb': 7, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 0, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [5], 'acting': [5], 'avail_no_missing': ['5'], 'object_location_counts': [{'shards': '5', 'objects': 1}], 'blocked_by': [], 'up_primary': 5, 'acting_primary': 5, 'purged_snaps': []} 2024-06-03T09:06:37.925 DEBUG:teuthology.orchestra.run.smithi088:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph status --format=json 2024-06-03T09:06:38.022 INFO:teuthology.orchestra.run.smithi088.stderr:2024-06-03T09:06:38.022+0000 7f8d2d0b2640 10 monclient: get_monmap_and_config 2024-06-03T09:06:38.022 INFO:teuthology.orchestra.run.smithi088.stderr:2024-06-03T09:06:38.022+0000 7f8d2d0b2640 10 monclient: build_initial_monmap 2024-06-03T09:06:38.022 INFO:teuthology.orchestra.run.smithi088.stderr:2024-06-03T09:06:38.022+0000 7f8d2d0b2640 10 monclient: monmap: 2024-06-03T09:06:38.022 INFO:teuthology.orchestra.run.smithi088.stderr:epoch 0 2024-06-03T09:06:38.022 INFO:teuthology.orchestra.run.smithi088.stderr:fsid ec3c3fdd-3ac6-4d7d-9412-683130ed8b04 2024-06-03T09:06:38.022 INFO:teuthology.orchestra.run.smithi088.stderr:last_changed 2024-06-03T09:06:38.023060+0000 2024-06-03T09:06:38.022 INFO:teuthology.orchestra.run.smithi088.stderr:created 2024-06-03T09:06:38.023060+0000 2024-06-03T09:06:38.022 INFO:teuthology.orchestra.run.smithi088.stderr:min_mon_release 0 (unknown) 2024-06-03T09:06:38.022 INFO:teuthology.orchestra.run.smithi088.stderr:election_strategy: 3 2024-06-03T09:06:38.023 INFO:teuthology.orchestra.run.smithi088.stderr:0: v1:172.21.15.88:6789/0 mon.noname-a 2024-06-03T09:06:38.023 INFO:teuthology.orchestra.run.smithi088.stderr:1: v1:172.21.15.192:6789/0 mon.noname-c 2024-06-03T09:06:38.023 INFO:teuthology.orchestra.run.smithi088.stderr:2: v1:172.21.15.88:6790/0 mon.noname-b 2024-06-03T09:06:38.023 INFO:teuthology.orchestra.run.smithi088.stderr: 2024-06-03T09:06:38.024 INFO:teuthology.orchestra.run.smithi088.stderr:2024-06-03T09:06:38.022+0000 7f8d2d0b2640 1 Processor -- start 2024-06-03T09:06:38.024 INFO:teuthology.orchestra.run.smithi088.stderr:2024-06-03T09:06:38.022+0000 7f8d2d0b2640 1 -- start start 2024-06-03T09:06:38.024 INFO:teuthology.orchestra.run.smithi088.stderr:2024-06-03T09:06:38.022+0000 7f8d2d0b2640 10 monclient: init 2024-06-03T09:06:38.025 INFO:teuthology.orchestra.run.smithi088.stderr:2024-06-03T09:06:38.023+0000 7f8d2d0b2640 10 monclient: _reopen_session rank -1 2024-06-03T09:06:38.025 INFO:teuthology.orchestra.run.smithi088.stderr:2024-06-03T09:06:38.023+0000 7f8d2d0b2640 10 monclient: _add_conns ranks=[2,0,1] 2024-06-03T09:06:38.025 INFO:teuthology.orchestra.run.smithi088.stderr:2024-06-03T09:06:38.023+0000 7f8d2d0b2640 10 monclient(hunting): picked mon.noname-b con 0x7f8d280a87f0 addr v1:172.21.15.88:6790/0 2024-06-03T09:06:38.025 INFO:teuthology.orchestra.run.smithi088.stderr:2024-06-03T09:06:38.023+0000 7f8d2d0b2640 10 monclient(hunting): picked mon.noname-a con 0x7f8d280a8bd0 addr v1:172.21.15.88:6789/0 2024-06-03T09:06:38.025 INFO:teuthology.orchestra.run.smithi088.stderr:2024-06-03T09:06:38.023+0000 7f8d2d0b2640 10 monclient(hunting): picked mon.noname-c con 0x7f8d281414c0 addr v1:172.21.15.192:6789/0 2024-06-03T09:06:38.025 INFO:teuthology.orchestra.run.smithi088.stderr:2024-06-03T09:06:38.023+0000 7f8d2d0b2640 1 -- --> v1:172.21.15.88:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f8d280aa350 con 0x7f8d280a8bd0 2024-06-03T09:06:38.025 INFO:teuthology.orchestra.run.smithi088.stderr:2024-06-03T09:06:38.023+0000 7f8d2d0b2640 1 -- --> v1:172.21.15.192:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f8d280aa5c0 con 0x7f8d281414c0 2024-06-03T09:06:38.025 INFO:teuthology.orchestra.run.smithi088.stderr:2024-06-03T09:06:38.023+0000 7f8d2d0b2640 1 -- --> v1:172.21.15.88:6790/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f8d28141950 con 0x7f8d280a87f0 2024-06-03T09:06:38.025 INFO:teuthology.orchestra.run.smithi088.stderr:2024-06-03T09:06:38.023+0000 7f8d2d0b2640 10 monclient(hunting): _renew_subs 2024-06-03T09:06:38.025 INFO:teuthology.orchestra.run.smithi088.stderr:2024-06-03T09:06:38.023+0000 7f8d2d0b2640 10 monclient(hunting): authenticate will time out at 3053.833408s 2024-06-03T09:06:38.025 INFO:teuthology.orchestra.run.smithi088.stderr:2024-06-03T09:06:38.023+0000 7f8d2659c640 1 --1- >> v1:172.21.15.88:6789/0 conn(0x7f8d280a8bd0 0x7f8d28141e70 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.88:6789/0 says I am v1:172.21.15.88:54356/0 (socket says 172.21.15.88:54356) 2024-06-03T09:06:38.025 INFO:teuthology.orchestra.run.smithi088.stderr:2024-06-03T09:06:38.023+0000 7f8d26d9d640 1 --1- >> v1:172.21.15.88:6790/0 conn(0x7f8d280a87f0 0x7f8d280b1be0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.88:6790/0 says I am v1:172.21.15.88:41242/0 (socket says 172.21.15.88:41242) 2024-06-03T09:06:38.025 INFO:teuthology.orchestra.run.smithi088.stderr:2024-06-03T09:06:38.023+0000 7f8d2659c640 1 -- 172.21.15.88:0/799820575 learned_addr learned my addr 172.21.15.88:0/799820575 (peer_addr_for_me v1:172.21.15.88:0/0) 2024-06-03T09:06:38.026 INFO:teuthology.orchestra.run.smithi088.stderr:2024-06-03T09:06:38.026+0000 7f8d25d9b640 1 -- 172.21.15.88:0/799820575 <== mon.1 v1:172.21.15.192:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 296826234 0 0) 0x7f8d280aa5c0 con 0x7f8d281414c0 2024-06-03T09:06:38.026 INFO:teuthology.orchestra.run.smithi088.stderr:2024-06-03T09:06:38.026+0000 7f8d25d9b640 10 monclient(hunting): _negotiate 2024-06-03T09:06:38.027 INFO:teuthology.orchestra.run.smithi088.stderr:2024-06-03T09:06:38.026+0000 7f8d25d9b640 10 monclient(hunting): _init_auth method 2 2024-06-03T09:06:38.027 INFO:teuthology.orchestra.run.smithi088.stderr:2024-06-03T09:06:38.026+0000 7f8d25d9b640 10 monclient(hunting): _init_auth creating new auth 2024-06-03T09:06:38.027 INFO:teuthology.orchestra.run.smithi088.stderr:2024-06-03T09:06:38.026+0000 7f8d25d9b640 10 monclient(hunting): my global_id is 30241 2024-06-03T09:06:38.027 INFO:teuthology.orchestra.run.smithi088.stderr:2024-06-03T09:06:38.026+0000 7f8d25d9b640 1 -- 172.21.15.88:0/799820575 --> v1:172.21.15.192:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f8d10003520 con 0x7f8d281414c0 2024-06-03T09:06:38.027 INFO:teuthology.orchestra.run.smithi088.stderr:2024-06-03T09:06:38.026+0000 7f8d25d9b640 1 -- 172.21.15.88:0/799820575 <== mon.0 v1:172.21.15.88:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 3202546662 0 0) 0x7f8d280aa350 con 0x7f8d280a8bd0 2024-06-03T09:06:38.027 INFO:teuthology.orchestra.run.smithi088.stderr:2024-06-03T09:06:38.026+0000 7f8d25d9b640 10 monclient(hunting): _negotiate 2024-06-03T09:06:38.027 INFO:teuthology.orchestra.run.smithi088.stderr:2024-06-03T09:06:38.026+0000 7f8d25d9b640 10 monclient(hunting): _init_auth method 2 2024-06-03T09:06:38.027 INFO:teuthology.orchestra.run.smithi088.stderr:2024-06-03T09:06:38.026+0000 7f8d25d9b640 10 monclient(hunting): _init_auth creating new auth 2024-06-03T09:06:38.027 INFO:teuthology.orchestra.run.smithi088.stderr:2024-06-03T09:06:38.026+0000 7f8d25d9b640 10 monclient(hunting): my global_id is 30660 2024-06-03T09:06:38.027 INFO:teuthology.orchestra.run.smithi088.stderr:2024-06-03T09:06:38.026+0000 7f8d25d9b640 1 -- 172.21.15.88:0/799820575 --> v1:172.21.15.88:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f8d280aa5c0 con 0x7f8d280a8bd0 2024-06-03T09:06:38.027 INFO:teuthology.orchestra.run.smithi088.stderr:2024-06-03T09:06:38.026+0000 7f8d25d9b640 1 -- 172.21.15.88:0/799820575 <== mon.2 v1:172.21.15.88:6790/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 3714711323 0 0) 0x7f8d28141950 con 0x7f8d280a87f0 2024-06-03T09:06:38.027 INFO:teuthology.orchestra.run.smithi088.stderr:2024-06-03T09:06:38.026+0000 7f8d25d9b640 10 monclient(hunting): _negotiate 2024-06-03T09:06:38.027 INFO:teuthology.orchestra.run.smithi088.stderr:2024-06-03T09:06:38.026+0000 7f8d25d9b640 10 monclient(hunting): _init_auth method 2 2024-06-03T09:06:38.028 INFO:teuthology.orchestra.run.smithi088.stderr:2024-06-03T09:06:38.026+0000 7f8d25d9b640 10 monclient(hunting): _init_auth creating new auth 2024-06-03T09:06:38.028 INFO:teuthology.orchestra.run.smithi088.stderr:2024-06-03T09:06:38.026+0000 7f8d25d9b640 10 monclient(hunting): my global_id is 30164 2024-06-03T09:06:38.028 INFO:teuthology.orchestra.run.smithi088.stderr:2024-06-03T09:06:38.026+0000 7f8d25d9b640 1 -- 172.21.15.88:0/799820575 --> v1:172.21.15.88:6790/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f8d280aa350 con 0x7f8d280a87f0 2024-06-03T09:06:38.029 INFO:teuthology.orchestra.run.smithi088.stderr:2024-06-03T09:06:38.029+0000 7f8d25d9b640 1 -- 172.21.15.88:0/799820575 <== mon.1 v1:172.21.15.192:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 214+0+0 (unknown 2389731448 0 0) 0x7f8d10003520 con 0x7f8d281414c0 2024-06-03T09:06:38.029 INFO:teuthology.orchestra.run.smithi088.stderr:2024-06-03T09:06:38.029+0000 7f8d25d9b640 1 -- 172.21.15.88:0/799820575 >> v1:172.21.15.88:6790/0 conn(0x7f8d280a87f0 legacy=0x7f8d280b1be0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-06-03T09:06:38.029 INFO:teuthology.orchestra.run.smithi088.stderr:2024-06-03T09:06:38.029+0000 7f8d25d9b640 1 -- 172.21.15.88:0/799820575 >> v1:172.21.15.88:6789/0 conn(0x7f8d280a8bd0 legacy=0x7f8d28141e70 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-06-03T09:06:38.029 INFO:teuthology.orchestra.run.smithi088.stderr:2024-06-03T09:06:38.029+0000 7f8d25d9b640 10 monclient: _finish_hunting 0 2024-06-03T09:06:38.029 INFO:teuthology.orchestra.run.smithi088.stderr:2024-06-03T09:06:38.029+0000 7f8d25d9b640 1 monclient: found mon.noname-c 2024-06-03T09:06:38.029 INFO:teuthology.orchestra.run.smithi088.stderr:2024-06-03T09:06:38.029+0000 7f8d25d9b640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-06-03T09:06:38.029 INFO:teuthology.orchestra.run.smithi088.stderr:2024-06-03T09:06:38.029+0000 7f8d25d9b640 10 monclient: _send_mon_message to mon.noname-c at v1:172.21.15.192:6789/0 2024-06-03T09:06:38.029 INFO:teuthology.orchestra.run.smithi088.stderr:2024-06-03T09:06:38.029+0000 7f8d25d9b640 1 -- 172.21.15.88:0/799820575 --> v1:172.21.15.192:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f8d2813b8c0 con 0x7f8d281414c0 2024-06-03T09:06:38.030 INFO:teuthology.orchestra.run.smithi088.stderr:2024-06-03T09:06:38.029+0000 7f8d25d9b640 10 monclient: _finish_auth 0 2024-06-03T09:06:38.030 INFO:teuthology.orchestra.run.smithi088.stderr:2024-06-03T09:06:38.029+0000 7f8d25d9b640 10 monclient: _check_auth_tickets 2024-06-03T09:06:38.030 INFO:teuthology.orchestra.run.smithi088.stderr:2024-06-03T09:06:38.029+0000 7f8d25d9b640 20 monclient: _check_auth_rotating not needed by client.admin 2024-06-03T09:06:38.030 INFO:teuthology.orchestra.run.smithi088.stderr:2024-06-03T09:06:38.029+0000 7f8d25d9b640 1 -- 172.21.15.88:0/799820575 <== mon.1 v1:172.21.15.192:6789/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 2234374065 0 0) 0x7f8d20001990 con 0x7f8d281414c0 2024-06-03T09:06:38.030 INFO:teuthology.orchestra.run.smithi088.stderr:2024-06-03T09:06:38.029+0000 7f8d25d9b640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-06-03T09:06:38.030 INFO:teuthology.orchestra.run.smithi088.stderr:2024-06-03T09:06:38.029+0000 7f8d25d9b640 10 monclient: got monmap 1 from mon.noname-c (according to old e1) 2024-06-03T09:06:38.030 INFO:teuthology.orchestra.run.smithi088.stderr:2024-06-03T09:06:38.029+0000 7f8d25d9b640 10 monclient: dump: 2024-06-03T09:06:38.030 INFO:teuthology.orchestra.run.smithi088.stderr:epoch 1 2024-06-03T09:06:38.030 INFO:teuthology.orchestra.run.smithi088.stderr:fsid ec3c3fdd-3ac6-4d7d-9412-683130ed8b04 2024-06-03T09:06:38.030 INFO:teuthology.orchestra.run.smithi088.stderr:last_changed 2024-06-03T08:34:31.885125+0000 2024-06-03T09:06:38.030 INFO:teuthology.orchestra.run.smithi088.stderr:created 2024-06-03T08:34:31.885125+0000 2024-06-03T09:06:38.031 INFO:teuthology.orchestra.run.smithi088.stderr:min_mon_release 18 (reef) 2024-06-03T09:06:38.031 INFO:teuthology.orchestra.run.smithi088.stderr:election_strategy: 3 2024-06-03T09:06:38.031 INFO:teuthology.orchestra.run.smithi088.stderr:0: v1:172.21.15.88:6789/0 mon.a 2024-06-03T09:06:38.031 INFO:teuthology.orchestra.run.smithi088.stderr:1: v1:172.21.15.192:6789/0 mon.b 2024-06-03T09:06:38.031 INFO:teuthology.orchestra.run.smithi088.stderr:2: v1:172.21.15.88:6790/0 mon.c 2024-06-03T09:06:38.031 INFO:teuthology.orchestra.run.smithi088.stderr: 2024-06-03T09:06:38.031 INFO:teuthology.orchestra.run.smithi088.stderr:2024-06-03T09:06:38.029+0000 7f8d2d0b2640 5 monclient: authenticate success, global_id 30241 2024-06-03T09:06:38.031 INFO:teuthology.orchestra.run.smithi088.stderr:2024-06-03T09:06:38.029+0000 7f8d2d0b2640 20 monclient: get_monmap_and_config waiting for monmap|config 2024-06-03T09:06:38.034 INFO:teuthology.orchestra.run.smithi088.stderr:2024-06-03T09:06:38.031+0000 7f8d25d9b640 1 -- 172.21.15.88:0/799820575 <== mon.1 v1:172.21.15.192:6789/0 4 ==== config(0 keys) v1 ==== 4+0+0 (unknown 0 0 0) 0x7f8d20001bb0 con 0x7f8d281414c0 2024-06-03T09:06:38.034 INFO:teuthology.orchestra.run.smithi088.stderr:2024-06-03T09:06:38.031+0000 7f8d25d9b640 10 monclient: handle_config config(0 keys) v1 2024-06-03T09:06:38.034 INFO:teuthology.orchestra.run.smithi088.stderr:2024-06-03T09:06:38.031+0000 7f8d25d9b640 1 -- 172.21.15.88:0/799820575 <== mon.1 v1:172.21.15.192:6789/0 5 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 2234374065 0 0) 0x7f8d20001f40 con 0x7f8d281414c0 2024-06-03T09:06:38.034 INFO:teuthology.orchestra.run.smithi088.stderr:2024-06-03T09:06:38.031+0000 7f8d2d0b2640 10 monclient: get_monmap_and_config success 2024-06-03T09:06:38.034 INFO:teuthology.orchestra.run.smithi088.stderr:2024-06-03T09:06:38.031+0000 7f8d2d0b2640 4 set_mon_vals no callback set 2024-06-03T09:06:38.034 INFO:teuthology.orchestra.run.smithi088.stderr:2024-06-03T09:06:38.031+0000 7f8d25d9b640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-06-03T09:06:38.034 INFO:teuthology.orchestra.run.smithi088.stderr:2024-06-03T09:06:38.031+0000 7f8d25d9b640 10 monclient: got monmap 1 from mon.b (according to old e1) 2024-06-03T09:06:38.034 INFO:teuthology.orchestra.run.smithi088.stderr:2024-06-03T09:06:38.031+0000 7f8d25d9b640 10 monclient: dump: 2024-06-03T09:06:38.035 INFO:teuthology.orchestra.run.smithi088.stderr:epoch 1 2024-06-03T09:06:38.035 INFO:teuthology.orchestra.run.smithi088.stderr:fsid ec3c3fdd-3ac6-4d7d-9412-683130ed8b04 2024-06-03T09:06:38.035 INFO:teuthology.orchestra.run.smithi088.stderr:last_changed 2024-06-03T08:34:31.885125+0000 2024-06-03T09:06:38.035 INFO:teuthology.orchestra.run.smithi088.stderr:created 2024-06-03T08:34:31.885125+0000 2024-06-03T09:06:38.035 INFO:teuthology.orchestra.run.smithi088.stderr:min_mon_release 18 (reef) 2024-06-03T09:06:38.035 INFO:teuthology.orchestra.run.smithi088.stderr:election_strategy: 3 2024-06-03T09:06:38.035 INFO:teuthology.orchestra.run.smithi088.stderr:0: v1:172.21.15.88:6789/0 mon.a 2024-06-03T09:06:38.035 INFO:teuthology.orchestra.run.smithi088.stderr:1: v1:172.21.15.192:6789/0 mon.b 2024-06-03T09:06:38.035 INFO:teuthology.orchestra.run.smithi088.stderr:2: v1:172.21.15.88:6790/0 mon.c 2024-06-03T09:06:38.035 INFO:teuthology.orchestra.run.smithi088.stderr: 2024-06-03T09:06:38.035 INFO:teuthology.orchestra.run.smithi088.stderr:2024-06-03T09:06:38.031+0000 7f8d2d0b2640 10 monclient: shutdown 2024-06-03T09:06:38.035 INFO:teuthology.orchestra.run.smithi088.stderr:2024-06-03T09:06:38.031+0000 7f8d2d0b2640 20 monclient: shutdown discarding 0 pending message(s) 2024-06-03T09:06:38.035 INFO:teuthology.orchestra.run.smithi088.stderr:2024-06-03T09:06:38.031+0000 7f8d2d0b2640 1 -- 172.21.15.88:0/799820575 >> v1:172.21.15.192:6789/0 conn(0x7f8d281414c0 legacy=0x7f8d2813b1b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-06-03T09:06:38.035 INFO:teuthology.orchestra.run.smithi088.stderr:2024-06-03T09:06:38.031+0000 7f8d2d0b2640 1 -- 172.21.15.88:0/799820575 shutdown_connections 2024-06-03T09:06:38.036 INFO:teuthology.orchestra.run.smithi088.stderr:2024-06-03T09:06:38.031+0000 7f8d2d0b2640 1 -- 172.21.15.88:0/799820575 >> 172.21.15.88:0/799820575 conn(0x7f8d280b71e0 msgr2=0x7f8d280b75c0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-06-03T09:06:38.036 INFO:teuthology.orchestra.run.smithi088.stderr:2024-06-03T09:06:38.031+0000 7f8d2d0b2640 1 -- 172.21.15.88:0/799820575 shutdown_connections 2024-06-03T09:06:38.036 INFO:teuthology.orchestra.run.smithi088.stderr:2024-06-03T09:06:38.031+0000 7f8d2d0b2640 1 -- 172.21.15.88:0/799820575 wait complete. 2024-06-03T09:06:38.036 INFO:teuthology.orchestra.run.smithi088.stderr:2024-06-03T09:06:38.032+0000 7f8d2d0b2640 10 monclient: build_initial_monmap 2024-06-03T09:06:38.036 INFO:teuthology.orchestra.run.smithi088.stderr:2024-06-03T09:06:38.032+0000 7f8d2d0b2640 10 monclient: monmap: 2024-06-03T09:06:38.036 INFO:teuthology.orchestra.run.smithi088.stderr:epoch 0 2024-06-03T09:06:38.036 INFO:teuthology.orchestra.run.smithi088.stderr:fsid 00000000-0000-0000-0000-000000000000 2024-06-03T09:06:38.036 INFO:teuthology.orchestra.run.smithi088.stderr:last_changed 0.000000 2024-06-03T09:06:38.036 INFO:teuthology.orchestra.run.smithi088.stderr:created 0.000000 2024-06-03T09:06:38.036 INFO:teuthology.orchestra.run.smithi088.stderr:min_mon_release 0 (unknown) 2024-06-03T09:06:38.036 INFO:teuthology.orchestra.run.smithi088.stderr:election_strategy: 1 2024-06-03T09:06:38.036 INFO:teuthology.orchestra.run.smithi088.stderr:0: v1:172.21.15.88:6789/0 mon.noname-a 2024-06-03T09:06:38.036 INFO:teuthology.orchestra.run.smithi088.stderr:1: v1:172.21.15.192:6789/0 mon.noname-b 2024-06-03T09:06:38.037 INFO:teuthology.orchestra.run.smithi088.stderr:2: v1:172.21.15.88:6790/0 mon.noname-c 2024-06-03T09:06:38.037 INFO:teuthology.orchestra.run.smithi088.stderr: 2024-06-03T09:06:38.037 INFO:teuthology.orchestra.run.smithi088.stderr:2024-06-03T09:06:38.032+0000 7f8d2d0b2640 1 librados: starting msgr at 2024-06-03T09:06:38.037 INFO:teuthology.orchestra.run.smithi088.stderr:2024-06-03T09:06:38.032+0000 7f8d2d0b2640 1 librados: starting objecter 2024-06-03T09:06:38.037 INFO:teuthology.orchestra.run.smithi088.stderr:2024-06-03T09:06:38.033+0000 7f8d2d0b2640 1 Processor -- start 2024-06-03T09:06:38.037 INFO:teuthology.orchestra.run.smithi088.stderr:2024-06-03T09:06:38.033+0000 7f8d2d0b2640 1 -- start start 2024-06-03T09:06:38.037 INFO:teuthology.orchestra.run.smithi088.stderr:2024-06-03T09:06:38.033+0000 7f8d2d0b2640 1 librados: setting wanted keys 2024-06-03T09:06:38.037 INFO:teuthology.orchestra.run.smithi088.stderr:2024-06-03T09:06:38.033+0000 7f8d2d0b2640 1 librados: calling monclient init 2024-06-03T09:06:38.037 INFO:teuthology.orchestra.run.smithi088.stderr:2024-06-03T09:06:38.033+0000 7f8d2d0b2640 10 monclient: init 2024-06-03T09:06:38.037 INFO:teuthology.orchestra.run.smithi088.stderr:2024-06-03T09:06:38.033+0000 7f8d2d0b2640 10 monclient: _reopen_session rank -1 2024-06-03T09:06:38.037 INFO:teuthology.orchestra.run.smithi088.stderr:2024-06-03T09:06:38.033+0000 7f8d2d0b2640 10 monclient: _add_conns ranks=[0,2,1] 2024-06-03T09:06:38.037 INFO:teuthology.orchestra.run.smithi088.stderr:2024-06-03T09:06:38.033+0000 7f8d2d0b2640 10 monclient(hunting): picked mon.noname-a con 0x7f8d280a87f0 addr v1:172.21.15.88:6789/0 2024-06-03T09:06:38.037 INFO:teuthology.orchestra.run.smithi088.stderr:2024-06-03T09:06:38.033+0000 7f8d2d0b2640 10 monclient(hunting): picked mon.noname-c con 0x7f8d28136e60 addr v1:172.21.15.88:6790/0 2024-06-03T09:06:38.038 INFO:teuthology.orchestra.run.smithi088.stderr:2024-06-03T09:06:38.033+0000 7f8d2d0b2640 10 monclient(hunting): picked mon.noname-b con 0x7f8d2813ccb0 addr v1:172.21.15.192:6789/0 2024-06-03T09:06:38.038 INFO:teuthology.orchestra.run.smithi088.stderr:2024-06-03T09:06:38.033+0000 7f8d2d0b2640 1 -- --> v1:172.21.15.88:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f8d28139390 con 0x7f8d280a87f0 2024-06-03T09:06:38.038 INFO:teuthology.orchestra.run.smithi088.stderr:2024-06-03T09:06:38.033+0000 7f8d2d0b2640 1 -- --> v1:172.21.15.192:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f8d28137580 con 0x7f8d2813ccb0 2024-06-03T09:06:38.038 INFO:teuthology.orchestra.run.smithi088.stderr:2024-06-03T09:06:38.033+0000 7f8d2d0b2640 1 -- --> v1:172.21.15.88:6790/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f8d2813d0e0 con 0x7f8d28136e60 2024-06-03T09:06:38.038 INFO:teuthology.orchestra.run.smithi088.stderr:2024-06-03T09:06:38.033+0000 7f8d2d0b2640 10 monclient(hunting): _renew_subs 2024-06-03T09:06:38.038 INFO:teuthology.orchestra.run.smithi088.stderr:2024-06-03T09:06:38.033+0000 7f8d2d0b2640 10 monclient(hunting): authenticate will time out at 3053.843603s 2024-06-03T09:06:38.038 INFO:teuthology.orchestra.run.smithi088.stderr:2024-06-03T09:06:38.034+0000 7f8d26d9d640 1 --1- >> v1:172.21.15.88:6789/0 conn(0x7f8d280a87f0 0x7f8d28138c80 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=1).handle_server_banner_and_identify peer v1:172.21.15.88:6789/0 says I am v1:172.21.15.88:54368/0 (socket says 172.21.15.88:54368) 2024-06-03T09:06:38.038 INFO:teuthology.orchestra.run.smithi088.stderr:2024-06-03T09:06:38.034+0000 7f8d26d9d640 1 -- 172.21.15.88:0/1583682877 learned_addr learned my addr 172.21.15.88:0/1583682877 (peer_addr_for_me v1:172.21.15.88:0/0) 2024-06-03T09:06:38.038 INFO:teuthology.orchestra.run.smithi088.stderr:2024-06-03T09:06:38.034+0000 7f8d0f7fe640 10 client.?.objecter ms_handle_connect 0x7f8d280a87f0 2024-06-03T09:06:38.038 INFO:teuthology.orchestra.run.smithi088.stderr:2024-06-03T09:06:38.034+0000 7f8d0f7fe640 10 client.?.objecter resend_mon_ops 2024-06-03T09:06:38.038 INFO:teuthology.orchestra.run.smithi088.stderr:2024-06-03T09:06:38.034+0000 7f8d0f7fe640 10 client.?.objecter ms_handle_connect 0x7f8d2813ccb0 2024-06-03T09:06:38.038 INFO:teuthology.orchestra.run.smithi088.stderr:2024-06-03T09:06:38.034+0000 7f8d0f7fe640 10 client.?.objecter resend_mon_ops 2024-06-03T09:06:38.039 INFO:teuthology.orchestra.run.smithi088.stderr:2024-06-03T09:06:38.034+0000 7f8d0f7fe640 10 client.?.objecter ms_handle_connect 0x7f8d28136e60 2024-06-03T09:06:38.039 INFO:teuthology.orchestra.run.smithi088.stderr:2024-06-03T09:06:38.034+0000 7f8d0f7fe640 10 client.?.objecter resend_mon_ops 2024-06-03T09:06:38.039 INFO:teuthology.orchestra.run.smithi088.stderr:2024-06-03T09:06:38.036+0000 7f8d0f7fe640 1 -- 172.21.15.88:0/1583682877 <== mon.1 v1:172.21.15.192:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 3712376206 0 0) 0x7f8d28137580 con 0x7f8d2813ccb0 2024-06-03T09:06:38.039 INFO:teuthology.orchestra.run.smithi088.stderr:2024-06-03T09:06:38.036+0000 7f8d0f7fe640 10 monclient(hunting): _negotiate 2024-06-03T09:06:38.040 INFO:teuthology.orchestra.run.smithi088.stderr:2024-06-03T09:06:38.036+0000 7f8d0f7fe640 10 monclient(hunting): _init_auth method 2 2024-06-03T09:06:38.040 INFO:teuthology.orchestra.run.smithi088.stderr:2024-06-03T09:06:38.036+0000 7f8d0f7fe640 10 monclient(hunting): _init_auth creating new auth 2024-06-03T09:06:38.040 INFO:teuthology.orchestra.run.smithi088.stderr:2024-06-03T09:06:38.036+0000 7f8d0f7fe640 10 monclient(hunting): my global_id is 30244 2024-06-03T09:06:38.040 INFO:teuthology.orchestra.run.smithi088.stderr:2024-06-03T09:06:38.036+0000 7f8d0f7fe640 1 -- 172.21.15.88:0/1583682877 --> v1:172.21.15.192:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f8d14004eb0 con 0x7f8d2813ccb0 2024-06-03T09:06:38.040 INFO:teuthology.orchestra.run.smithi088.stderr:2024-06-03T09:06:38.036+0000 7f8d0f7fe640 1 -- 172.21.15.88:0/1583682877 <== mon.0 v1:172.21.15.88:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 2123099798 0 0) 0x7f8d280aa350 con 0x7f8d280a87f0 2024-06-03T09:06:38.040 INFO:teuthology.orchestra.run.smithi088.stderr:2024-06-03T09:06:38.036+0000 7f8d0f7fe640 10 monclient(hunting): _negotiate 2024-06-03T09:06:38.040 INFO:teuthology.orchestra.run.smithi088.stderr:2024-06-03T09:06:38.036+0000 7f8d0f7fe640 10 monclient(hunting): _init_auth method 2 2024-06-03T09:06:38.040 INFO:teuthology.orchestra.run.smithi088.stderr:2024-06-03T09:06:38.036+0000 7f8d0f7fe640 10 monclient(hunting): _init_auth creating new auth 2024-06-03T09:06:38.040 INFO:teuthology.orchestra.run.smithi088.stderr:2024-06-03T09:06:38.036+0000 7f8d0f7fe640 10 monclient(hunting): my global_id is 30663 2024-06-03T09:06:38.040 INFO:teuthology.orchestra.run.smithi088.stderr:2024-06-03T09:06:38.036+0000 7f8d0f7fe640 1 -- 172.21.15.88:0/1583682877 --> v1:172.21.15.88:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f8d28137580 con 0x7f8d280a87f0 2024-06-03T09:06:38.040 INFO:teuthology.orchestra.run.smithi088.stderr:2024-06-03T09:06:38.036+0000 7f8d0f7fe640 1 -- 172.21.15.88:0/1583682877 <== mon.2 v1:172.21.15.88:6790/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 3558560228 0 0) 0x7f8d2813d0e0 con 0x7f8d28136e60 2024-06-03T09:06:38.040 INFO:teuthology.orchestra.run.smithi088.stderr:2024-06-03T09:06:38.036+0000 7f8d0f7fe640 10 monclient(hunting): _negotiate 2024-06-03T09:06:38.040 INFO:teuthology.orchestra.run.smithi088.stderr:2024-06-03T09:06:38.036+0000 7f8d0f7fe640 10 monclient(hunting): _init_auth method 2 2024-06-03T09:06:38.041 INFO:teuthology.orchestra.run.smithi088.stderr:2024-06-03T09:06:38.036+0000 7f8d0f7fe640 10 monclient(hunting): _init_auth creating new auth 2024-06-03T09:06:38.041 INFO:teuthology.orchestra.run.smithi088.stderr:2024-06-03T09:06:38.036+0000 7f8d0f7fe640 10 monclient(hunting): my global_id is 30167 2024-06-03T09:06:38.041 INFO:teuthology.orchestra.run.smithi088.stderr:2024-06-03T09:06:38.036+0000 7f8d0f7fe640 1 -- 172.21.15.88:0/1583682877 --> v1:172.21.15.88:6790/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f8d280aa350 con 0x7f8d28136e60 2024-06-03T09:06:38.042 INFO:teuthology.orchestra.run.smithi088.stderr:2024-06-03T09:06:38.040+0000 7f8d0f7fe640 1 -- 172.21.15.88:0/1583682877 <== mon.1 v1:172.21.15.192:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 764+0+0 (unknown 804200957 0 0) 0x7f8d14004eb0 con 0x7f8d2813ccb0 2024-06-03T09:06:38.042 INFO:teuthology.orchestra.run.smithi088.stderr:2024-06-03T09:06:38.040+0000 7f8d0f7fe640 1 -- 172.21.15.88:0/1583682877 --> v1:172.21.15.192:6789/0 -- auth(proto 2 165 bytes epoch 0) v1 -- 0x7f8d2813d0e0 con 0x7f8d2813ccb0 2024-06-03T09:06:38.042 INFO:teuthology.orchestra.run.smithi088.stderr:2024-06-03T09:06:38.040+0000 7f8d0f7fe640 1 -- 172.21.15.88:0/1583682877 <== mon.1 v1:172.21.15.192:6789/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 2234374065 0 0) 0x7f8d20001ec0 con 0x7f8d2813ccb0 2024-06-03T09:06:38.042 INFO:teuthology.orchestra.run.smithi088.stderr:2024-06-03T09:06:38.040+0000 7f8d0f7fe640 10 monclient(hunting): handle_monmap mon_map magic: 0 v1 2024-06-03T09:06:38.042 INFO:teuthology.orchestra.run.smithi088.stderr:2024-06-03T09:06:38.040+0000 7f8d0f7fe640 10 monclient(hunting): got monmap 1 from mon.noname-b (according to old e1) 2024-06-03T09:06:38.042 INFO:teuthology.orchestra.run.smithi088.stderr:2024-06-03T09:06:38.040+0000 7f8d0f7fe640 10 monclient(hunting): dump: 2024-06-03T09:06:38.042 INFO:teuthology.orchestra.run.smithi088.stderr:epoch 1 2024-06-03T09:06:38.042 INFO:teuthology.orchestra.run.smithi088.stderr:fsid ec3c3fdd-3ac6-4d7d-9412-683130ed8b04 2024-06-03T09:06:38.042 INFO:teuthology.orchestra.run.smithi088.stderr:last_changed 2024-06-03T08:34:31.885125+0000 2024-06-03T09:06:38.042 INFO:teuthology.orchestra.run.smithi088.stderr:created 2024-06-03T08:34:31.885125+0000 2024-06-03T09:06:38.043 INFO:teuthology.orchestra.run.smithi088.stderr:min_mon_release 18 (reef) 2024-06-03T09:06:38.043 INFO:teuthology.orchestra.run.smithi088.stderr:election_strategy: 3 2024-06-03T09:06:38.043 INFO:teuthology.orchestra.run.smithi088.stderr:0: v1:172.21.15.88:6789/0 mon.a 2024-06-03T09:06:38.043 INFO:teuthology.orchestra.run.smithi088.stderr:1: v1:172.21.15.192:6789/0 mon.b 2024-06-03T09:06:38.043 INFO:teuthology.orchestra.run.smithi088.stderr:2: v1:172.21.15.88:6790/0 mon.c 2024-06-03T09:06:38.043 INFO:teuthology.orchestra.run.smithi088.stderr: 2024-06-03T09:06:38.043 INFO:teuthology.orchestra.run.smithi088.stderr:2024-06-03T09:06:38.040+0000 7f8d0f7fe640 10 monclient(hunting): _finish_auth 0 2024-06-03T09:06:38.043 INFO:teuthology.orchestra.run.smithi088.stderr:2024-06-03T09:06:38.041+0000 7f8d0f7fe640 1 -- 172.21.15.88:0/1583682877 <== mon.0 v1:172.21.15.88:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 764+0+0 (unknown 1032236916 0 0) 0x7f8d28137580 con 0x7f8d280a87f0 2024-06-03T09:06:38.043 INFO:teuthology.orchestra.run.smithi088.stderr:2024-06-03T09:06:38.041+0000 7f8d0f7fe640 1 -- 172.21.15.88:0/1583682877 --> v1:172.21.15.88:6789/0 -- auth(proto 2 165 bytes epoch 0) v1 -- 0x7f8d14004eb0 con 0x7f8d280a87f0 2024-06-03T09:06:38.043 INFO:teuthology.orchestra.run.smithi088.stderr:2024-06-03T09:06:38.041+0000 7f8d0f7fe640 1 -- 172.21.15.88:0/1583682877 <== mon.2 v1:172.21.15.88:6790/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 764+0+0 (unknown 1848107605 0 0) 0x7f8d280aa350 con 0x7f8d28136e60 2024-06-03T09:06:38.043 INFO:teuthology.orchestra.run.smithi088.stderr:2024-06-03T09:06:38.041+0000 7f8d0f7fe640 1 -- 172.21.15.88:0/1583682877 --> v1:172.21.15.88:6790/0 -- auth(proto 2 165 bytes epoch 0) v1 -- 0x7f8d28137580 con 0x7f8d28136e60 2024-06-03T09:06:38.043 INFO:teuthology.orchestra.run.smithi088.stderr:2024-06-03T09:06:38.041+0000 7f8d0f7fe640 1 -- 172.21.15.88:0/1583682877 <== mon.2 v1:172.21.15.88:6790/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 2234374065 0 0) 0x7f8d1c001870 con 0x7f8d28136e60 2024-06-03T09:06:38.043 INFO:teuthology.orchestra.run.smithi088.stderr:2024-06-03T09:06:38.041+0000 7f8d0f7fe640 10 monclient(hunting): handle_monmap mon_map magic: 0 v1 2024-06-03T09:06:38.043 INFO:teuthology.orchestra.run.smithi088.stderr:2024-06-03T09:06:38.041+0000 7f8d0f7fe640 10 monclient(hunting): got monmap 1 from mon.c (according to old e1) 2024-06-03T09:06:38.044 INFO:teuthology.orchestra.run.smithi088.stderr:2024-06-03T09:06:38.041+0000 7f8d0f7fe640 10 monclient(hunting): dump: 2024-06-03T09:06:38.044 INFO:teuthology.orchestra.run.smithi088.stderr:epoch 1 2024-06-03T09:06:38.044 INFO:teuthology.orchestra.run.smithi088.stderr:fsid ec3c3fdd-3ac6-4d7d-9412-683130ed8b04 2024-06-03T09:06:38.044 INFO:teuthology.orchestra.run.smithi088.stderr:last_changed 2024-06-03T08:34:31.885125+0000 2024-06-03T09:06:38.044 INFO:teuthology.orchestra.run.smithi088.stderr:created 2024-06-03T08:34:31.885125+0000 2024-06-03T09:06:38.044 INFO:teuthology.orchestra.run.smithi088.stderr:min_mon_release 18 (reef) 2024-06-03T09:06:38.044 INFO:teuthology.orchestra.run.smithi088.stderr:election_strategy: 3 2024-06-03T09:06:38.044 INFO:teuthology.orchestra.run.smithi088.stderr:0: v1:172.21.15.88:6789/0 mon.a 2024-06-03T09:06:38.044 INFO:teuthology.orchestra.run.smithi088.stderr:1: v1:172.21.15.192:6789/0 mon.b 2024-06-03T09:06:38.044 INFO:teuthology.orchestra.run.smithi088.stderr:2: v1:172.21.15.88:6790/0 mon.c 2024-06-03T09:06:38.044 INFO:teuthology.orchestra.run.smithi088.stderr: 2024-06-03T09:06:38.045 INFO:teuthology.orchestra.run.smithi088.stderr:2024-06-03T09:06:38.044+0000 7f8d0f7fe640 1 -- 172.21.15.88:0/1583682877 <== mon.1 v1:172.21.15.192:6789/0 4 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 580+0+0 (unknown 48539087 0 0) 0x7f8d2813d0e0 con 0x7f8d2813ccb0 2024-06-03T09:06:38.045 INFO:teuthology.orchestra.run.smithi088.stderr:2024-06-03T09:06:38.044+0000 7f8d0f7fe640 1 -- 172.21.15.88:0/1583682877 >> v1:172.21.15.88:6790/0 conn(0x7f8d28136e60 legacy=0x7f8d281da020 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-06-03T09:06:38.045 INFO:teuthology.orchestra.run.smithi088.stderr:2024-06-03T09:06:38.044+0000 7f8d0f7fe640 1 -- 172.21.15.88:0/1583682877 >> v1:172.21.15.88:6789/0 conn(0x7f8d280a87f0 legacy=0x7f8d28138c80 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-06-03T09:06:38.046 INFO:teuthology.orchestra.run.smithi088.stderr:2024-06-03T09:06:38.044+0000 7f8d0f7fe640 10 monclient: _finish_hunting 0 2024-06-03T09:06:38.046 INFO:teuthology.orchestra.run.smithi088.stderr:2024-06-03T09:06:38.044+0000 7f8d0f7fe640 1 monclient: found mon.b 2024-06-03T09:06:38.046 INFO:teuthology.orchestra.run.smithi088.stderr:2024-06-03T09:06:38.044+0000 7f8d0f7fe640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-06-03T09:06:38.046 INFO:teuthology.orchestra.run.smithi088.stderr:2024-06-03T09:06:38.044+0000 7f8d0f7fe640 10 monclient: _send_mon_message to mon.b at v1:172.21.15.192:6789/0 2024-06-03T09:06:38.046 INFO:teuthology.orchestra.run.smithi088.stderr:2024-06-03T09:06:38.044+0000 7f8d0f7fe640 1 -- 172.21.15.88:0/1583682877 --> v1:172.21.15.192:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f8d281dae40 con 0x7f8d2813ccb0 2024-06-03T09:06:38.046 INFO:teuthology.orchestra.run.smithi088.stderr:2024-06-03T09:06:38.044+0000 7f8d0f7fe640 10 monclient: _finish_auth 0 2024-06-03T09:06:38.046 INFO:teuthology.orchestra.run.smithi088.stderr:2024-06-03T09:06:38.044+0000 7f8d0f7fe640 10 monclient: _check_auth_tickets 2024-06-03T09:06:38.046 INFO:teuthology.orchestra.run.smithi088.stderr:2024-06-03T09:06:38.044+0000 7f8d0f7fe640 20 monclient: _check_auth_rotating not needed by client.admin 2024-06-03T09:06:38.046 INFO:teuthology.orchestra.run.smithi088.stderr:2024-06-03T09:06:38.044+0000 7f8d2d0b2640 5 monclient: authenticate success, global_id 30244 2024-06-03T09:06:38.046 INFO:teuthology.orchestra.run.smithi088.stderr:2024-06-03T09:06:38.044+0000 7f8d2d0b2640 10 monclient: _renew_subs 2024-06-03T09:06:38.046 INFO:teuthology.orchestra.run.smithi088.stderr:2024-06-03T09:06:38.044+0000 7f8d2d0b2640 10 monclient: _send_mon_message to mon.b at v1:172.21.15.192:6789/0 2024-06-03T09:06:38.046 INFO:teuthology.orchestra.run.smithi088.stderr:2024-06-03T09:06:38.044+0000 7f8d2d0b2640 1 -- 172.21.15.88:0/1583682877 --> v1:172.21.15.192:6789/0 -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f8d2813e300 con 0x7f8d2813ccb0 2024-06-03T09:06:38.047 INFO:teuthology.orchestra.run.smithi088.stderr:2024-06-03T09:06:38.044+0000 7f8d2d0b2640 10 client.30244.objecter _maybe_request_map subscribing (onetime) to next osd map 2024-06-03T09:06:38.047 INFO:teuthology.orchestra.run.smithi088.stderr:2024-06-03T09:06:38.044+0000 7f8d2d0b2640 10 monclient: _renew_subs 2024-06-03T09:06:38.047 INFO:teuthology.orchestra.run.smithi088.stderr:2024-06-03T09:06:38.044+0000 7f8d2d0b2640 10 monclient: _send_mon_message to mon.b at v1:172.21.15.192:6789/0 2024-06-03T09:06:38.047 INFO:teuthology.orchestra.run.smithi088.stderr:2024-06-03T09:06:38.044+0000 7f8d2d0b2640 1 -- 172.21.15.88:0/1583682877 --> v1:172.21.15.192:6789/0 -- mon_subscribe({osdmap=0}) v3 -- 0x7f8d2813eac0 con 0x7f8d2813ccb0 2024-06-03T09:06:38.047 INFO:teuthology.orchestra.run.smithi088.stderr:2024-06-03T09:06:38.044+0000 7f8d2d0b2640 1 librados: init done 2024-06-03T09:06:38.047 INFO:teuthology.orchestra.run.smithi088.stderr:2024-06-03T09:06:38.045+0000 7f8d2d0b2640 10 start_mon_command cmd=[{"prefix": "get_command_descriptions"}] 2024-06-03T09:06:38.047 INFO:teuthology.orchestra.run.smithi088.stderr:2024-06-03T09:06:38.045+0000 7f8d2d0b2640 10 monclient: _send_command 1 [{"prefix": "get_command_descriptions"}] 2024-06-03T09:06:38.047 INFO:teuthology.orchestra.run.smithi088.stderr:2024-06-03T09:06:38.045+0000 7f8d2d0b2640 10 monclient: _send_mon_message to mon.b at v1:172.21.15.192:6789/0 2024-06-03T09:06:38.047 INFO:teuthology.orchestra.run.smithi088.stderr:2024-06-03T09:06:38.045+0000 7f8d2d0b2640 1 -- 172.21.15.88:0/1583682877 --> v1:172.21.15.192:6789/0 -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f8d281389c0 con 0x7f8d2813ccb0 2024-06-03T09:06:38.060 INFO:teuthology.orchestra.run.smithi088.stderr:2024-06-03T09:06:38.060+0000 7f8d0f7fe640 1 -- 172.21.15.88:0/1583682877 <== mon.1 v1:172.21.15.192:6789/0 5 ==== config(0 keys) v1 ==== 4+0+0 (unknown 0 0 0) 0x7f8d200014c0 con 0x7f8d2813ccb0 2024-06-03T09:06:38.060 INFO:teuthology.orchestra.run.smithi088.stderr:2024-06-03T09:06:38.060+0000 7f8d0f7fe640 10 monclient: handle_config config(0 keys) v1 2024-06-03T09:06:38.060 INFO:teuthology.orchestra.run.smithi088.stderr:2024-06-03T09:06:38.060+0000 7f8d0f7fe640 1 -- 172.21.15.88:0/1583682877 <== mon.1 v1:172.21.15.192:6789/0 6 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 2234374065 0 0) 0x7f8d20003c20 con 0x7f8d2813ccb0 2024-06-03T09:06:38.060 INFO:teuthology.orchestra.run.smithi088.stderr:2024-06-03T09:06:38.060+0000 7f8d0f7fe640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-06-03T09:06:38.060 INFO:teuthology.orchestra.run.smithi088.stderr:2024-06-03T09:06:38.060+0000 7f8d0f7fe640 10 monclient: got monmap 1 from mon.b (according to old e1) 2024-06-03T09:06:38.060 INFO:teuthology.orchestra.run.smithi088.stderr:2024-06-03T09:06:38.060+0000 7f8d0f7fe640 10 monclient: dump: 2024-06-03T09:06:38.060 INFO:teuthology.orchestra.run.smithi088.stderr:epoch 1 2024-06-03T09:06:38.060 INFO:teuthology.orchestra.run.smithi088.stderr:fsid ec3c3fdd-3ac6-4d7d-9412-683130ed8b04 2024-06-03T09:06:38.061 INFO:teuthology.orchestra.run.smithi088.stderr:last_changed 2024-06-03T08:34:31.885125+0000 2024-06-03T09:06:38.061 INFO:teuthology.orchestra.run.smithi088.stderr:created 2024-06-03T08:34:31.885125+0000 2024-06-03T09:06:38.061 INFO:teuthology.orchestra.run.smithi088.stderr:min_mon_release 18 (reef) 2024-06-03T09:06:38.061 INFO:teuthology.orchestra.run.smithi088.stderr:election_strategy: 3 2024-06-03T09:06:38.061 INFO:teuthology.orchestra.run.smithi088.stderr:0: v1:172.21.15.88:6789/0 mon.a 2024-06-03T09:06:38.061 INFO:teuthology.orchestra.run.smithi088.stderr:1: v1:172.21.15.192:6789/0 mon.b 2024-06-03T09:06:38.061 INFO:teuthology.orchestra.run.smithi088.stderr:2: v1:172.21.15.88:6790/0 mon.c 2024-06-03T09:06:38.061 INFO:teuthology.orchestra.run.smithi088.stderr: 2024-06-03T09:06:38.061 INFO:teuthology.orchestra.run.smithi088.stderr:2024-06-03T09:06:38.060+0000 7f8d2559a640 4 set_mon_vals no callback set 2024-06-03T09:06:38.062 INFO:teuthology.orchestra.run.smithi088.stderr:2024-06-03T09:06:38.060+0000 7f8d0f7fe640 1 -- 172.21.15.88:0/1583682877 <== mon.1 v1:172.21.15.192:6789/0 7 ==== mgrmap(e 5) v1 ==== 89110+0+0 (unknown 873533973 0 0) 0x7f8d200199a0 con 0x7f8d2813ccb0 2024-06-03T09:06:38.062 INFO:teuthology.orchestra.run.smithi088.stderr:2024-06-03T09:06:38.061+0000 7f8d26d9d640 10 monclient: get_auth_request con 0x7f8d1406d5a0 auth_method 0 2024-06-03T09:06:38.062 INFO:teuthology.orchestra.run.smithi088.stderr:2024-06-03T09:06:38.061+0000 7f8d0f7fe640 10 client.30244.objecter ms_handle_connect 0x7f8d1406d5a0 2024-06-03T09:06:38.062 INFO:teuthology.orchestra.run.smithi088.stderr:2024-06-03T09:06:38.062+0000 7f8d0f7fe640 1 -- 172.21.15.88:0/1583682877 <== mon.1 v1:172.21.15.192:6789/0 8 ==== osd_map(1010..1010 src has 622..1010) v4 ==== 5229+0+0 (unknown 1105883993 0 0) 0x7f8d20053060 con 0x7f8d2813ccb0 2024-06-03T09:06:38.062 INFO:teuthology.orchestra.run.smithi088.stderr:2024-06-03T09:06:38.062+0000 7f8d0f7fe640 10 client.30244.objecter ms_dispatch 0x7f8d28000b90 osd_map(1010..1010 src has 622..1010) v4 2024-06-03T09:06:38.063 INFO:teuthology.orchestra.run.smithi088.stderr:2024-06-03T09:06:38.062+0000 7f8d0f7fe640 3 client.30244.objecter handle_osd_map got epochs [1010,1010] > 0 2024-06-03T09:06:38.063 INFO:teuthology.orchestra.run.smithi088.stderr:2024-06-03T09:06:38.062+0000 7f8d0f7fe640 3 client.30244.objecter handle_osd_map decoding full epoch 1010 2024-06-03T09:06:38.063 INFO:teuthology.orchestra.run.smithi088.stderr:2024-06-03T09:06:38.062+0000 7f8d0f7fe640 20 client.30244.objecter dump_active .. 0 homeless 2024-06-03T09:06:38.303 INFO:teuthology.orchestra.run.smithi088.stderr:2024-06-03T09:06:38.303+0000 7f8d0f7fe640 1 -- 172.21.15.88:0/1583682877 <== mon.1 v1:172.21.15.192:6789/0 9 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+182498 (unknown 1092875540 0 3016760700) 0x7f8d2004ea40 con 0x7f8d2813ccb0 2024-06-03T09:06:38.303 INFO:teuthology.orchestra.run.smithi088.stderr:2024-06-03T09:06:38.303+0000 7f8d0f7fe640 10 monclient: handle_mon_command_ack 1 [{"prefix": "get_command_descriptions"}] 2024-06-03T09:06:38.303 INFO:teuthology.orchestra.run.smithi088.stderr:2024-06-03T09:06:38.303+0000 7f8d0f7fe640 10 monclient: _finish_command 1 = system:0 2024-06-03T09:06:38.429 INFO:tasks.ceph.mon.a.smithi088.stderr:==00:00:31:33.755 34743== Warning: unimplemented fcntl command: 1036 2024-06-03T09:06:38.431 INFO:tasks.ceph.mon.a.smithi088.stderr:==00:00:31:33.757 34743== Warning: unimplemented fcntl command: 1036 2024-06-03T09:06:38.441 INFO:tasks.ceph.mon.a.smithi088.stderr:==00:00:31:33.767 34743== Warning: unimplemented fcntl command: 1036 2024-06-03T09:06:38.476 INFO:tasks.ceph.mon.b.smithi192.stderr:==00:00:31:33.779 34228== Warning: unimplemented fcntl command: 1036 2024-06-03T09:06:38.478 INFO:tasks.ceph.mon.b.smithi192.stderr:==00:00:31:33.781 34228== Warning: unimplemented fcntl command: 1036 2024-06-03T09:06:38.481 INFO:tasks.ceph.mon.c.smithi088.stderr:==00:00:31:33.807 34749== Warning: unimplemented fcntl command: 1036 2024-06-03T09:06:38.482 INFO:tasks.ceph.mon.c.smithi088.stderr:==00:00:31:33.808 34749== Warning: unimplemented fcntl command: 1036 2024-06-03T09:06:38.488 INFO:tasks.ceph.mon.b.smithi192.stderr:==00:00:31:33.791 34228== Warning: unimplemented fcntl command: 1036 2024-06-03T09:06:38.492 INFO:tasks.ceph.mon.c.smithi088.stderr:==00:00:31:33.818 34749== Warning: unimplemented fcntl command: 1036 2024-06-03T09:06:38.557 INFO:teuthology.orchestra.run.smithi088.stderr:2024-06-03T09:06:38.557+0000 7f8d2d0b2640 10 start_mon_command cmd=[{"prefix": "status", "format": "json"}] 2024-06-03T09:06:38.558 INFO:teuthology.orchestra.run.smithi088.stderr:2024-06-03T09:06:38.557+0000 7f8d2d0b2640 10 monclient: _send_command 2 [{"prefix": "status", "format": "json"}] 2024-06-03T09:06:38.558 INFO:teuthology.orchestra.run.smithi088.stderr:2024-06-03T09:06:38.557+0000 7f8d2d0b2640 10 monclient: _send_mon_message to mon.b at v1:172.21.15.192:6789/0 2024-06-03T09:06:38.558 INFO:teuthology.orchestra.run.smithi088.stderr:2024-06-03T09:06:38.557+0000 7f8d2d0b2640 1 -- 172.21.15.88:0/1583682877 --> v1:172.21.15.192:6789/0 -- mon_command({"prefix": "status", "format": "json"} v 0) v1 -- 0x7f8d2813ed10 con 0x7f8d2813ccb0 2024-06-03T09:06:38.571 INFO:teuthology.orchestra.run.smithi088.stderr:2024-06-03T09:06:38.571+0000 7f8d0f7fe640 1 -- 172.21.15.88:0/1583682877 <== mon.1 v1:172.21.15.192:6789/0 10 ==== mon_command_ack([{"prefix": "status", "format": "json"}]=0 v0) v1 ==== 72+0+2108 (unknown 3226119948 0 1597181143) 0x7f8d20050b40 con 0x7f8d2813ccb0 2024-06-03T09:06:38.571 INFO:teuthology.orchestra.run.smithi088.stderr:2024-06-03T09:06:38.571+0000 7f8d0f7fe640 10 monclient: handle_mon_command_ack 2 [{"prefix": "status", "format": "json"}] 2024-06-03T09:06:38.571 INFO:teuthology.orchestra.run.smithi088.stderr:2024-06-03T09:06:38.571+0000 7f8d0f7fe640 10 monclient: _finish_command 2 = system:0 2024-06-03T09:06:38.572 INFO:teuthology.orchestra.run.smithi088.stdout: 2024-06-03T09:06:38.572 INFO:teuthology.orchestra.run.smithi088.stdout:{"fsid":"ec3c3fdd-3ac6-4d7d-9412-683130ed8b04","health":{"status":"HEALTH_WARN","checks":{"OSD_DOWN":{"severity":"HEALTH_WARN","summary":{"message":"1 osds down","count":1},"muted":false},"PG_DEGRADED":{"severity":"HEALTH_WARN","summary":{"message":"Degraded data redundancy: 1/6 objects degraded (16.667%), 1 pg degraded","count":1},"muted":false},"POOL_APP_NOT_ENABLED":{"severity":"HEALTH_WARN","summary":{"message":"1 pool(s) do not have an application enabled","count":1},"muted":false}},"mutes":[]},"election_epoch":10,"quorum":[0,1,2],"quorum_names":["a","b","c"],"quorum_age":1872,"monmap":{"epoch":1,"min_mon_release_name":"reef","num_mons":3},"osdmap":{"epoch":1010,"num_osds":8,"num_up_osds":7,"osd_up_since":1717405562,"num_in_osds":8,"osd_in_since":1717403732,"num_remapped_pgs":0},"pgmap":{"pgs_by_state":[{"state_name":"active+clean","count":27},{"state_name":"active+undersized","count":13},{"state_name":"active+undersized+degraded","count":1}],"num_pgs":41,"num_pools":3,"num_objects":3,"data_bytes":459285,"bytes_used":506417152,"bytes_avail":772587696128,"bytes_total":773094113280,"degraded_objects":1,"degraded_total":6,"degraded_ratio":0.16666666666666663},"fsmap":{"epoch":1,"by_rank":[],"up:standby":0},"mgrmap":{"available":true,"num_standbys":1,"modules":["iostat","nfs","restful"],"services":{}},"servicemap":{"epoch":25,"modified":"2024-06-03T09:06:37.101852+0000","services":{"osd":{"daemons":{"summary":"","0":{"start_epoch":0,"start_stamp":"0.000000","gid":0,"addr":"(unrecognized address family 0)/0","metadata":{},"task_status":{}},"1":{"start_epoch":0,"start_stamp":"0.000000","gid":0,"addr":"(unrecognized address family 0)/0","metadata":{},"task_status":{}},"3":{"start_epoch":0,"start_stamp":"0.000000","gid":0,"addr":"(unrecognized address family 0)/0","metadata":{},"task_status":{}},"4":{"start_epoch":0,"start_stamp":"0.000000","gid":0,"addr":"(unrecognized address family 0)/0","metadata":{},"task_status":{}},"5":{"start_epoch":0,"start_stamp":"0.000000","gid":0,"addr":"(unrecognized address family 0)/0","metadata":{},"task_status":{}}}}}},"progress_events":{}} 2024-06-03T09:06:38.575 INFO:teuthology.orchestra.run.smithi088.stderr:2024-06-03T09:06:38.574+0000 7f8d2d0b2640 10 librados: watch_flush enter 2024-06-03T09:06:38.575 INFO:teuthology.orchestra.run.smithi088.stderr:2024-06-03T09:06:38.574+0000 7f8d2d0b2640 10 librados: watch_flush exit 2024-06-03T09:06:38.575 INFO:teuthology.orchestra.run.smithi088.stderr:2024-06-03T09:06:38.574+0000 7f8d2d0b2640 20 client.30244.objecter shutdown clearing up homeless session... 2024-06-03T09:06:38.575 INFO:teuthology.orchestra.run.smithi088.stderr:2024-06-03T09:06:38.574+0000 7f8d2d0b2640 10 client.30244.objecter successfully canceled tick 2024-06-03T09:06:38.575 INFO:teuthology.orchestra.run.smithi088.stderr:2024-06-03T09:06:38.574+0000 7f8d2d0b2640 1 -- 172.21.15.88:0/1583682877 >> v1:172.21.15.192:6800/34229 conn(0x7f8d1406d5a0 legacy=0x7f8d1406fa60 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-06-03T09:06:38.575 INFO:teuthology.orchestra.run.smithi088.stderr:2024-06-03T09:06:38.574+0000 7f8d2d0b2640 10 monclient: shutdown 2024-06-03T09:06:38.575 INFO:teuthology.orchestra.run.smithi088.stderr:2024-06-03T09:06:38.574+0000 7f8d2d0b2640 20 monclient: shutdown discarding 0 pending message(s) 2024-06-03T09:06:38.575 INFO:teuthology.orchestra.run.smithi088.stderr:2024-06-03T09:06:38.574+0000 7f8d2d0b2640 1 -- 172.21.15.88:0/1583682877 >> v1:172.21.15.192:6789/0 conn(0x7f8d2813ccb0 legacy=0x7f8d281da730 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-06-03T09:06:38.575 INFO:teuthology.orchestra.run.smithi088.stderr:2024-06-03T09:06:38.574+0000 7f8d2d0b2640 1 -- 172.21.15.88:0/1583682877 shutdown_connections 2024-06-03T09:06:38.575 INFO:teuthology.orchestra.run.smithi088.stderr:2024-06-03T09:06:38.574+0000 7f8d2d0b2640 1 -- 172.21.15.88:0/1583682877 >> 172.21.15.88:0/1583682877 conn(0x7f8d280b71e0 msgr2=0x7f8d280b75c0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-06-03T09:06:38.575 INFO:teuthology.orchestra.run.smithi088.stderr:2024-06-03T09:06:38.574+0000 7f8d2d0b2640 1 -- 172.21.15.88:0/1583682877 shutdown_connections 2024-06-03T09:06:38.575 INFO:teuthology.orchestra.run.smithi088.stderr:2024-06-03T09:06:38.574+0000 7f8d2d0b2640 1 -- 172.21.15.88:0/1583682877 wait complete. 2024-06-03T09:06:38.576 INFO:teuthology.orchestra.run.smithi088.stderr:2024-06-03T09:06:38.574+0000 7f8d2d0b2640 1 librados: shutdown 2024-06-03T09:06:38.586 INFO:tasks.ceph.ceph_manager.ceph:no progress seen, keeping timeout for now 2024-06-03T09:06:38.586 DEBUG:teuthology.orchestra.run.smithi088:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph pg dump --format=json 2024-06-03T09:06:38.710 INFO:teuthology.orchestra.run.smithi088.stderr:2024-06-03T09:06:38.710+0000 7ffb38300640 10 monclient: get_monmap_and_config 2024-06-03T09:06:38.710 INFO:teuthology.orchestra.run.smithi088.stderr:2024-06-03T09:06:38.710+0000 7ffb38300640 10 monclient: build_initial_monmap 2024-06-03T09:06:38.710 INFO:teuthology.orchestra.run.smithi088.stderr:2024-06-03T09:06:38.710+0000 7ffb38300640 10 monclient: monmap: 2024-06-03T09:06:38.710 INFO:teuthology.orchestra.run.smithi088.stderr:epoch 0 2024-06-03T09:06:38.710 INFO:teuthology.orchestra.run.smithi088.stderr:fsid ec3c3fdd-3ac6-4d7d-9412-683130ed8b04 2024-06-03T09:06:38.710 INFO:teuthology.orchestra.run.smithi088.stderr:last_changed 2024-06-03T09:06:38.711153+0000 2024-06-03T09:06:38.710 INFO:teuthology.orchestra.run.smithi088.stderr:created 2024-06-03T09:06:38.711153+0000 2024-06-03T09:06:38.710 INFO:teuthology.orchestra.run.smithi088.stderr:min_mon_release 0 (unknown) 2024-06-03T09:06:38.710 INFO:teuthology.orchestra.run.smithi088.stderr:election_strategy: 3 2024-06-03T09:06:38.710 INFO:teuthology.orchestra.run.smithi088.stderr:0: v1:172.21.15.88:6789/0 mon.noname-a 2024-06-03T09:06:38.711 INFO:teuthology.orchestra.run.smithi088.stderr:1: v1:172.21.15.192:6789/0 mon.noname-c 2024-06-03T09:06:38.711 INFO:teuthology.orchestra.run.smithi088.stderr:2: v1:172.21.15.88:6790/0 mon.noname-b 2024-06-03T09:06:38.711 INFO:teuthology.orchestra.run.smithi088.stderr: 2024-06-03T09:06:38.712 INFO:teuthology.orchestra.run.smithi088.stderr:2024-06-03T09:06:38.710+0000 7ffb38300640 1 Processor -- start 2024-06-03T09:06:38.712 INFO:teuthology.orchestra.run.smithi088.stderr:2024-06-03T09:06:38.710+0000 7ffb38300640 1 -- start start 2024-06-03T09:06:38.712 INFO:teuthology.orchestra.run.smithi088.stderr:2024-06-03T09:06:38.710+0000 7ffb38300640 10 monclient: init 2024-06-03T09:06:38.712 INFO:teuthology.orchestra.run.smithi088.stderr:2024-06-03T09:06:38.711+0000 7ffb38300640 10 monclient: _reopen_session rank -1 2024-06-03T09:06:38.712 INFO:teuthology.orchestra.run.smithi088.stderr:2024-06-03T09:06:38.711+0000 7ffb38300640 10 monclient: _add_conns ranks=[1,0,2] 2024-06-03T09:06:38.712 INFO:teuthology.orchestra.run.smithi088.stderr:2024-06-03T09:06:38.711+0000 7ffb38300640 10 monclient(hunting): picked mon.noname-c con 0x7ffb300a8960 addr v1:172.21.15.192:6789/0 2024-06-03T09:06:38.712 INFO:teuthology.orchestra.run.smithi088.stderr:2024-06-03T09:06:38.711+0000 7ffb38300640 10 monclient(hunting): picked mon.noname-a con 0x7ffb301450f0 addr v1:172.21.15.88:6789/0 2024-06-03T09:06:38.712 INFO:teuthology.orchestra.run.smithi088.stderr:2024-06-03T09:06:38.711+0000 7ffb38300640 10 monclient(hunting): picked mon.noname-b con 0x7ffb3013eec0 addr v1:172.21.15.88:6790/0 2024-06-03T09:06:38.712 INFO:teuthology.orchestra.run.smithi088.stderr:2024-06-03T09:06:38.711+0000 7ffb38300640 1 -- --> v1:172.21.15.88:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7ffb30144c60 con 0x7ffb301450f0 2024-06-03T09:06:38.712 INFO:teuthology.orchestra.run.smithi088.stderr:2024-06-03T09:06:38.711+0000 7ffb38300640 1 -- --> v1:172.21.15.192:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7ffb300b3930 con 0x7ffb300a8960 2024-06-03T09:06:38.712 INFO:teuthology.orchestra.run.smithi088.stderr:2024-06-03T09:06:38.711+0000 7ffb38300640 1 -- --> v1:172.21.15.88:6790/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7ffb300b1ad0 con 0x7ffb3013eec0 2024-06-03T09:06:38.713 INFO:teuthology.orchestra.run.smithi088.stderr:2024-06-03T09:06:38.711+0000 7ffb38300640 10 monclient(hunting): _renew_subs 2024-06-03T09:06:38.713 INFO:teuthology.orchestra.run.smithi088.stderr:2024-06-03T09:06:38.711+0000 7ffb38300640 10 monclient(hunting): authenticate will time out at 3054.521503s 2024-06-03T09:06:38.714 INFO:teuthology.orchestra.run.smithi088.stderr:2024-06-03T09:06:38.714+0000 7ffb3589b640 1 --1- >> v1:172.21.15.88:6789/0 conn(0x7ffb301450f0 0x7ffb3013e7b0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.88:6789/0 says I am v1:172.21.15.88:54382/0 (socket says 172.21.15.88:54382) 2024-06-03T09:06:38.714 INFO:teuthology.orchestra.run.smithi088.stderr:2024-06-03T09:06:38.714+0000 7ffb3589b640 1 -- 172.21.15.88:0/3362207617 learned_addr learned my addr 172.21.15.88:0/3362207617 (peer_addr_for_me v1:172.21.15.88:0/0) 2024-06-03T09:06:38.727 INFO:teuthology.orchestra.run.smithi088.stderr:2024-06-03T09:06:38.727+0000 7ffb3509a640 1 -- 172.21.15.88:0/3362207617 <== mon.0 v1:172.21.15.88:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 3127106224 0 0) 0x7ffb30144c60 con 0x7ffb301450f0 2024-06-03T09:06:38.727 INFO:teuthology.orchestra.run.smithi088.stderr:2024-06-03T09:06:38.727+0000 7ffb3509a640 10 monclient(hunting): _negotiate 2024-06-03T09:06:38.727 INFO:teuthology.orchestra.run.smithi088.stderr:2024-06-03T09:06:38.727+0000 7ffb3509a640 10 monclient(hunting): _init_auth method 2 2024-06-03T09:06:38.727 INFO:teuthology.orchestra.run.smithi088.stderr:2024-06-03T09:06:38.727+0000 7ffb3509a640 10 monclient(hunting): _init_auth creating new auth 2024-06-03T09:06:38.728 INFO:teuthology.orchestra.run.smithi088.stderr:2024-06-03T09:06:38.727+0000 7ffb3509a640 10 monclient(hunting): my global_id is 30666 2024-06-03T09:06:38.728 INFO:teuthology.orchestra.run.smithi088.stderr:2024-06-03T09:06:38.727+0000 7ffb3509a640 1 -- 172.21.15.88:0/3362207617 --> v1:172.21.15.88:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7ffb1c003520 con 0x7ffb301450f0 2024-06-03T09:06:38.730 INFO:teuthology.orchestra.run.smithi088.stderr:2024-06-03T09:06:38.730+0000 7ffb3509a640 1 -- 172.21.15.88:0/3362207617 <== mon.2 v1:172.21.15.88:6790/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 2653668045 0 0) 0x7ffb300b1ad0 con 0x7ffb3013eec0 2024-06-03T09:06:38.730 INFO:teuthology.orchestra.run.smithi088.stderr:2024-06-03T09:06:38.730+0000 7ffb3509a640 10 monclient(hunting): _negotiate 2024-06-03T09:06:38.730 INFO:teuthology.orchestra.run.smithi088.stderr:2024-06-03T09:06:38.730+0000 7ffb3509a640 10 monclient(hunting): _init_auth method 2 2024-06-03T09:06:38.730 INFO:teuthology.orchestra.run.smithi088.stderr:2024-06-03T09:06:38.730+0000 7ffb3509a640 10 monclient(hunting): _init_auth creating new auth 2024-06-03T09:06:38.731 INFO:teuthology.orchestra.run.smithi088.stderr:2024-06-03T09:06:38.730+0000 7ffb3509a640 10 monclient(hunting): my global_id is 30170 2024-06-03T09:06:38.731 INFO:teuthology.orchestra.run.smithi088.stderr:2024-06-03T09:06:38.730+0000 7ffb3509a640 1 -- 172.21.15.88:0/3362207617 --> v1:172.21.15.88:6790/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7ffb30144c60 con 0x7ffb3013eec0 2024-06-03T09:06:38.732 INFO:teuthology.orchestra.run.smithi088.stderr:2024-06-03T09:06:38.732+0000 7ffb3509a640 1 -- 172.21.15.88:0/3362207617 <== mon.1 v1:172.21.15.192:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 3280828983 0 0) 0x7ffb300b3930 con 0x7ffb300a8960 2024-06-03T09:06:38.732 INFO:teuthology.orchestra.run.smithi088.stderr:2024-06-03T09:06:38.732+0000 7ffb3509a640 10 monclient(hunting): _negotiate 2024-06-03T09:06:38.732 INFO:teuthology.orchestra.run.smithi088.stderr:2024-06-03T09:06:38.732+0000 7ffb3509a640 10 monclient(hunting): _init_auth method 2 2024-06-03T09:06:38.733 INFO:teuthology.orchestra.run.smithi088.stderr:2024-06-03T09:06:38.732+0000 7ffb3509a640 10 monclient(hunting): _init_auth creating new auth 2024-06-03T09:06:38.733 INFO:teuthology.orchestra.run.smithi088.stderr:2024-06-03T09:06:38.732+0000 7ffb3509a640 10 monclient(hunting): my global_id is 30247 2024-06-03T09:06:38.733 INFO:teuthology.orchestra.run.smithi088.stderr:2024-06-03T09:06:38.732+0000 7ffb3509a640 1 -- 172.21.15.88:0/3362207617 --> v1:172.21.15.192:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7ffb300b1ad0 con 0x7ffb300a8960 2024-06-03T09:06:38.738 INFO:teuthology.orchestra.run.smithi088.stderr:2024-06-03T09:06:38.737+0000 7ffb3509a640 1 -- 172.21.15.88:0/3362207617 <== mon.1 v1:172.21.15.192:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 214+0+0 (unknown 785699705 0 0) 0x7ffb300b1ad0 con 0x7ffb300a8960 2024-06-03T09:06:38.738 INFO:teuthology.orchestra.run.smithi088.stderr:2024-06-03T09:06:38.737+0000 7ffb3509a640 1 -- 172.21.15.88:0/3362207617 >> v1:172.21.15.88:6790/0 conn(0x7ffb3013eec0 legacy=0x7ffb3014ce60 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-06-03T09:06:38.738 INFO:teuthology.orchestra.run.smithi088.stderr:2024-06-03T09:06:38.737+0000 7ffb3509a640 1 -- 172.21.15.88:0/3362207617 >> v1:172.21.15.88:6789/0 conn(0x7ffb301450f0 legacy=0x7ffb3013e7b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-06-03T09:06:38.738 INFO:teuthology.orchestra.run.smithi088.stderr:2024-06-03T09:06:38.737+0000 7ffb3509a640 10 monclient: _finish_hunting 0 2024-06-03T09:06:38.738 INFO:teuthology.orchestra.run.smithi088.stderr:2024-06-03T09:06:38.737+0000 7ffb3509a640 1 monclient: found mon.noname-c 2024-06-03T09:06:38.738 INFO:teuthology.orchestra.run.smithi088.stderr:2024-06-03T09:06:38.737+0000 7ffb3509a640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-06-03T09:06:38.738 INFO:teuthology.orchestra.run.smithi088.stderr:2024-06-03T09:06:38.737+0000 7ffb3509a640 10 monclient: _send_mon_message to mon.noname-c at v1:172.21.15.192:6789/0 2024-06-03T09:06:38.738 INFO:teuthology.orchestra.run.smithi088.stderr:2024-06-03T09:06:38.737+0000 7ffb3509a640 1 -- 172.21.15.88:0/3362207617 --> v1:172.21.15.192:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7ffb300b1c60 con 0x7ffb300a8960 2024-06-03T09:06:38.738 INFO:teuthology.orchestra.run.smithi088.stderr:2024-06-03T09:06:38.737+0000 7ffb3509a640 10 monclient: _finish_auth 0 2024-06-03T09:06:38.738 INFO:teuthology.orchestra.run.smithi088.stderr:2024-06-03T09:06:38.737+0000 7ffb3509a640 10 monclient: _check_auth_tickets 2024-06-03T09:06:38.738 INFO:teuthology.orchestra.run.smithi088.stderr:2024-06-03T09:06:38.737+0000 7ffb3509a640 20 monclient: _check_auth_rotating not needed by client.admin 2024-06-03T09:06:38.739 INFO:teuthology.orchestra.run.smithi088.stderr:2024-06-03T09:06:38.737+0000 7ffb3509a640 1 -- 172.21.15.88:0/3362207617 <== mon.1 v1:172.21.15.192:6789/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 2234374065 0 0) 0x7ffb2c0015b0 con 0x7ffb300a8960 2024-06-03T09:06:38.739 INFO:teuthology.orchestra.run.smithi088.stderr:2024-06-03T09:06:38.737+0000 7ffb3509a640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-06-03T09:06:38.739 INFO:teuthology.orchestra.run.smithi088.stderr:2024-06-03T09:06:38.737+0000 7ffb3509a640 10 monclient: got monmap 1 from mon.noname-c (according to old e1) 2024-06-03T09:06:38.739 INFO:teuthology.orchestra.run.smithi088.stderr:2024-06-03T09:06:38.737+0000 7ffb3509a640 10 monclient: dump: 2024-06-03T09:06:38.739 INFO:teuthology.orchestra.run.smithi088.stderr:epoch 1 2024-06-03T09:06:38.739 INFO:teuthology.orchestra.run.smithi088.stderr:fsid ec3c3fdd-3ac6-4d7d-9412-683130ed8b04 2024-06-03T09:06:38.739 INFO:teuthology.orchestra.run.smithi088.stderr:last_changed 2024-06-03T08:34:31.885125+0000 2024-06-03T09:06:38.739 INFO:teuthology.orchestra.run.smithi088.stderr:created 2024-06-03T08:34:31.885125+0000 2024-06-03T09:06:38.739 INFO:teuthology.orchestra.run.smithi088.stderr:min_mon_release 18 (reef) 2024-06-03T09:06:38.739 INFO:teuthology.orchestra.run.smithi088.stderr:election_strategy: 3 2024-06-03T09:06:38.739 INFO:teuthology.orchestra.run.smithi088.stderr:0: v1:172.21.15.88:6789/0 mon.a 2024-06-03T09:06:38.739 INFO:teuthology.orchestra.run.smithi088.stderr:1: v1:172.21.15.192:6789/0 mon.b 2024-06-03T09:06:38.739 INFO:teuthology.orchestra.run.smithi088.stderr:2: v1:172.21.15.88:6790/0 mon.c 2024-06-03T09:06:38.739 INFO:teuthology.orchestra.run.smithi088.stderr: 2024-06-03T09:06:38.740 INFO:teuthology.orchestra.run.smithi088.stderr:2024-06-03T09:06:38.737+0000 7ffb38300640 5 monclient: authenticate success, global_id 30247 2024-06-03T09:06:38.740 INFO:teuthology.orchestra.run.smithi088.stderr:2024-06-03T09:06:38.737+0000 7ffb38300640 20 monclient: get_monmap_and_config waiting for monmap|config 2024-06-03T09:06:38.763 INFO:teuthology.orchestra.run.smithi088.stderr:2024-06-03T09:06:38.763+0000 7ffb3509a640 1 -- 172.21.15.88:0/3362207617 <== mon.1 v1:172.21.15.192:6789/0 4 ==== config(0 keys) v1 ==== 4+0+0 (unknown 0 0 0) 0x7ffb2c0017d0 con 0x7ffb300a8960 2024-06-03T09:06:38.764 INFO:teuthology.orchestra.run.smithi088.stderr:2024-06-03T09:06:38.764+0000 7ffb3509a640 10 monclient: handle_config config(0 keys) v1 2024-06-03T09:06:38.764 INFO:teuthology.orchestra.run.smithi088.stderr:2024-06-03T09:06:38.764+0000 7ffb3509a640 1 -- 172.21.15.88:0/3362207617 <== mon.1 v1:172.21.15.192:6789/0 5 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 2234374065 0 0) 0x7ffb2c001ae0 con 0x7ffb300a8960 2024-06-03T09:06:38.764 INFO:teuthology.orchestra.run.smithi088.stderr:2024-06-03T09:06:38.764+0000 7ffb38300640 10 monclient: get_monmap_and_config success 2024-06-03T09:06:38.764 INFO:teuthology.orchestra.run.smithi088.stderr:2024-06-03T09:06:38.764+0000 7ffb38300640 4 set_mon_vals no callback set 2024-06-03T09:06:38.764 INFO:teuthology.orchestra.run.smithi088.stderr:2024-06-03T09:06:38.764+0000 7ffb3509a640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-06-03T09:06:38.764 INFO:teuthology.orchestra.run.smithi088.stderr:2024-06-03T09:06:38.764+0000 7ffb3509a640 10 monclient: got monmap 1 from mon.b (according to old e1) 2024-06-03T09:06:38.764 INFO:teuthology.orchestra.run.smithi088.stderr:2024-06-03T09:06:38.764+0000 7ffb3509a640 10 monclient: dump: 2024-06-03T09:06:38.764 INFO:teuthology.orchestra.run.smithi088.stderr:epoch 1 2024-06-03T09:06:38.764 INFO:teuthology.orchestra.run.smithi088.stderr:fsid ec3c3fdd-3ac6-4d7d-9412-683130ed8b04 2024-06-03T09:06:38.764 INFO:teuthology.orchestra.run.smithi088.stderr:last_changed 2024-06-03T08:34:31.885125+0000 2024-06-03T09:06:38.764 INFO:teuthology.orchestra.run.smithi088.stderr:created 2024-06-03T08:34:31.885125+0000 2024-06-03T09:06:38.764 INFO:teuthology.orchestra.run.smithi088.stderr:min_mon_release 18 (reef) 2024-06-03T09:06:38.765 INFO:teuthology.orchestra.run.smithi088.stderr:election_strategy: 3 2024-06-03T09:06:38.765 INFO:teuthology.orchestra.run.smithi088.stderr:0: v1:172.21.15.88:6789/0 mon.a 2024-06-03T09:06:38.765 INFO:teuthology.orchestra.run.smithi088.stderr:1: v1:172.21.15.192:6789/0 mon.b 2024-06-03T09:06:38.765 INFO:teuthology.orchestra.run.smithi088.stderr:2: v1:172.21.15.88:6790/0 mon.c 2024-06-03T09:06:38.765 INFO:teuthology.orchestra.run.smithi088.stderr: 2024-06-03T09:06:38.767 INFO:teuthology.orchestra.run.smithi088.stderr:2024-06-03T09:06:38.764+0000 7ffb38300640 10 monclient: shutdown 2024-06-03T09:06:38.767 INFO:teuthology.orchestra.run.smithi088.stderr:2024-06-03T09:06:38.764+0000 7ffb38300640 20 monclient: shutdown discarding 0 pending message(s) 2024-06-03T09:06:38.767 INFO:teuthology.orchestra.run.smithi088.stderr:2024-06-03T09:06:38.764+0000 7ffb38300640 1 -- 172.21.15.88:0/3362207617 >> v1:172.21.15.192:6789/0 conn(0x7ffb300a8960 legacy=0x7ffb300a8d40 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-06-03T09:06:38.768 INFO:teuthology.orchestra.run.smithi088.stderr:2024-06-03T09:06:38.764+0000 7ffb38300640 1 -- 172.21.15.88:0/3362207617 shutdown_connections 2024-06-03T09:06:38.768 INFO:teuthology.orchestra.run.smithi088.stderr:2024-06-03T09:06:38.764+0000 7ffb38300640 1 -- 172.21.15.88:0/3362207617 >> 172.21.15.88:0/3362207617 conn(0x7ffb300b70d0 msgr2=0x7ffb300b74e0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-06-03T09:06:38.768 INFO:teuthology.orchestra.run.smithi088.stderr:2024-06-03T09:06:38.764+0000 7ffb38300640 1 -- 172.21.15.88:0/3362207617 shutdown_connections 2024-06-03T09:06:38.768 INFO:teuthology.orchestra.run.smithi088.stderr:2024-06-03T09:06:38.764+0000 7ffb38300640 1 -- 172.21.15.88:0/3362207617 wait complete. 2024-06-03T09:06:38.768 INFO:teuthology.orchestra.run.smithi088.stderr:2024-06-03T09:06:38.765+0000 7ffb38300640 10 monclient: build_initial_monmap 2024-06-03T09:06:38.768 INFO:teuthology.orchestra.run.smithi088.stderr:2024-06-03T09:06:38.765+0000 7ffb38300640 10 monclient: monmap: 2024-06-03T09:06:38.768 INFO:teuthology.orchestra.run.smithi088.stderr:epoch 0 2024-06-03T09:06:38.768 INFO:teuthology.orchestra.run.smithi088.stderr:fsid 00000000-0000-0000-0000-000000000000 2024-06-03T09:06:38.768 INFO:teuthology.orchestra.run.smithi088.stderr:last_changed 0.000000 2024-06-03T09:06:38.768 INFO:teuthology.orchestra.run.smithi088.stderr:created 0.000000 2024-06-03T09:06:38.768 INFO:teuthology.orchestra.run.smithi088.stderr:min_mon_release 0 (unknown) 2024-06-03T09:06:38.768 INFO:teuthology.orchestra.run.smithi088.stderr:election_strategy: 1 2024-06-03T09:06:38.768 INFO:teuthology.orchestra.run.smithi088.stderr:0: v1:172.21.15.88:6789/0 mon.noname-a 2024-06-03T09:06:38.768 INFO:teuthology.orchestra.run.smithi088.stderr:1: v1:172.21.15.192:6789/0 mon.noname-b 2024-06-03T09:06:38.769 INFO:teuthology.orchestra.run.smithi088.stderr:2: v1:172.21.15.88:6790/0 mon.noname-c 2024-06-03T09:06:38.769 INFO:teuthology.orchestra.run.smithi088.stderr: 2024-06-03T09:06:38.769 INFO:teuthology.orchestra.run.smithi088.stderr:2024-06-03T09:06:38.766+0000 7ffb38300640 1 librados: starting msgr at 2024-06-03T09:06:38.769 INFO:teuthology.orchestra.run.smithi088.stderr:2024-06-03T09:06:38.766+0000 7ffb38300640 1 librados: starting objecter 2024-06-03T09:06:38.769 INFO:teuthology.orchestra.run.smithi088.stderr:2024-06-03T09:06:38.766+0000 7ffb38300640 1 Processor -- start 2024-06-03T09:06:38.769 INFO:teuthology.orchestra.run.smithi088.stderr:2024-06-03T09:06:38.766+0000 7ffb38300640 1 -- start start 2024-06-03T09:06:38.769 INFO:teuthology.orchestra.run.smithi088.stderr:2024-06-03T09:06:38.766+0000 7ffb38300640 1 librados: setting wanted keys 2024-06-03T09:06:38.769 INFO:teuthology.orchestra.run.smithi088.stderr:2024-06-03T09:06:38.766+0000 7ffb38300640 1 librados: calling monclient init 2024-06-03T09:06:38.769 INFO:teuthology.orchestra.run.smithi088.stderr:2024-06-03T09:06:38.766+0000 7ffb38300640 10 monclient: init 2024-06-03T09:06:38.769 INFO:teuthology.orchestra.run.smithi088.stderr:2024-06-03T09:06:38.767+0000 7ffb38300640 10 monclient: _reopen_session rank -1 2024-06-03T09:06:38.769 INFO:teuthology.orchestra.run.smithi088.stderr:2024-06-03T09:06:38.767+0000 7ffb38300640 10 monclient: _add_conns ranks=[2,0,1] 2024-06-03T09:06:38.769 INFO:teuthology.orchestra.run.smithi088.stderr:2024-06-03T09:06:38.767+0000 7ffb38300640 10 monclient(hunting): picked mon.noname-c con 0x7ffb300a8960 addr v1:172.21.15.88:6790/0 2024-06-03T09:06:38.769 INFO:teuthology.orchestra.run.smithi088.stderr:2024-06-03T09:06:38.767+0000 7ffb38300640 10 monclient(hunting): picked mon.noname-a con 0x7ffb3013eec0 addr v1:172.21.15.88:6789/0 2024-06-03T09:06:38.769 INFO:teuthology.orchestra.run.smithi088.stderr:2024-06-03T09:06:38.767+0000 7ffb38300640 10 monclient(hunting): picked mon.noname-b con 0x7ffb301450f0 addr v1:172.21.15.192:6789/0 2024-06-03T09:06:38.770 INFO:teuthology.orchestra.run.smithi088.stderr:2024-06-03T09:06:38.767+0000 7ffb38300640 1 -- --> v1:172.21.15.88:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7ffb301d6740 con 0x7ffb3013eec0 2024-06-03T09:06:38.770 INFO:teuthology.orchestra.run.smithi088.stderr:2024-06-03T09:06:38.767+0000 7ffb38300640 1 -- --> v1:172.21.15.192:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7ffb30140e90 con 0x7ffb301450f0 2024-06-03T09:06:38.770 INFO:teuthology.orchestra.run.smithi088.stderr:2024-06-03T09:06:38.767+0000 7ffb38300640 1 -- --> v1:172.21.15.88:6790/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7ffb3013f380 con 0x7ffb300a8960 2024-06-03T09:06:38.770 INFO:teuthology.orchestra.run.smithi088.stderr:2024-06-03T09:06:38.767+0000 7ffb38300640 10 monclient(hunting): _renew_subs 2024-06-03T09:06:38.770 INFO:teuthology.orchestra.run.smithi088.stderr:2024-06-03T09:06:38.767+0000 7ffb38300640 10 monclient(hunting): authenticate will time out at 3054.577297s 2024-06-03T09:06:38.770 INFO:teuthology.orchestra.run.smithi088.stderr:2024-06-03T09:06:38.768+0000 7ffb3589b640 1 --1- >> v1:172.21.15.88:6789/0 conn(0x7ffb3013eec0 0x7ffb30141cd0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=1).handle_server_banner_and_identify peer v1:172.21.15.88:6789/0 says I am v1:172.21.15.88:54388/0 (socket says 172.21.15.88:54388) 2024-06-03T09:06:38.770 INFO:teuthology.orchestra.run.smithi088.stderr:2024-06-03T09:06:38.768+0000 7ffb3589b640 1 -- 172.21.15.88:0/373289902 learned_addr learned my addr 172.21.15.88:0/373289902 (peer_addr_for_me v1:172.21.15.88:0/0) 2024-06-03T09:06:38.771 INFO:teuthology.orchestra.run.smithi088.stderr:2024-06-03T09:06:38.769+0000 7ffb26ffd640 10 client.?.objecter ms_handle_connect 0x7ffb3013eec0 2024-06-03T09:06:38.771 INFO:teuthology.orchestra.run.smithi088.stderr:2024-06-03T09:06:38.769+0000 7ffb26ffd640 10 client.?.objecter resend_mon_ops 2024-06-03T09:06:38.776 INFO:teuthology.orchestra.run.smithi088.stderr:2024-06-03T09:06:38.776+0000 7ffb26ffd640 1 -- 172.21.15.88:0/373289902 <== mon.0 v1:172.21.15.88:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 257685716 0 0) 0x7ffb301d6740 con 0x7ffb3013eec0 2024-06-03T09:06:38.776 INFO:teuthology.orchestra.run.smithi088.stderr:2024-06-03T09:06:38.776+0000 7ffb26ffd640 10 monclient(hunting): _negotiate 2024-06-03T09:06:38.776 INFO:teuthology.orchestra.run.smithi088.stderr:2024-06-03T09:06:38.776+0000 7ffb26ffd640 10 monclient(hunting): _init_auth method 2 2024-06-03T09:06:38.777 INFO:teuthology.orchestra.run.smithi088.stderr:2024-06-03T09:06:38.776+0000 7ffb26ffd640 10 monclient(hunting): _init_auth creating new auth 2024-06-03T09:06:38.777 INFO:teuthology.orchestra.run.smithi088.stderr:2024-06-03T09:06:38.776+0000 7ffb26ffd640 10 monclient(hunting): my global_id is 30669 2024-06-03T09:06:38.777 INFO:teuthology.orchestra.run.smithi088.stderr:2024-06-03T09:06:38.776+0000 7ffb26ffd640 1 -- 172.21.15.88:0/373289902 --> v1:172.21.15.88:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7ffb14004eb0 con 0x7ffb3013eec0 2024-06-03T09:06:38.777 INFO:teuthology.orchestra.run.smithi088.stderr:2024-06-03T09:06:38.777+0000 7ffb26ffd640 10 client.?.objecter ms_handle_connect 0x7ffb300a8960 2024-06-03T09:06:38.777 INFO:teuthology.orchestra.run.smithi088.stderr:2024-06-03T09:06:38.777+0000 7ffb26ffd640 10 client.?.objecter resend_mon_ops 2024-06-03T09:06:38.782 INFO:teuthology.orchestra.run.smithi088.stderr:2024-06-03T09:06:38.782+0000 7ffb26ffd640 1 -- 172.21.15.88:0/373289902 <== mon.2 v1:172.21.15.88:6790/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 2947548660 0 0) 0x7ffb3013f380 con 0x7ffb300a8960 2024-06-03T09:06:38.782 INFO:teuthology.orchestra.run.smithi088.stderr:2024-06-03T09:06:38.782+0000 7ffb26ffd640 10 monclient(hunting): _negotiate 2024-06-03T09:06:38.782 INFO:teuthology.orchestra.run.smithi088.stderr:2024-06-03T09:06:38.782+0000 7ffb26ffd640 10 monclient(hunting): _init_auth method 2 2024-06-03T09:06:38.782 INFO:teuthology.orchestra.run.smithi088.stderr:2024-06-03T09:06:38.782+0000 7ffb26ffd640 10 monclient(hunting): _init_auth creating new auth 2024-06-03T09:06:38.782 INFO:teuthology.orchestra.run.smithi088.stderr:2024-06-03T09:06:38.782+0000 7ffb26ffd640 10 monclient(hunting): my global_id is 30173 2024-06-03T09:06:38.782 INFO:teuthology.orchestra.run.smithi088.stderr:2024-06-03T09:06:38.782+0000 7ffb26ffd640 1 -- 172.21.15.88:0/373289902 --> v1:172.21.15.88:6790/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7ffb301d6740 con 0x7ffb300a8960 2024-06-03T09:06:38.787 INFO:teuthology.orchestra.run.smithi088.stderr:2024-06-03T09:06:38.787+0000 7ffb26ffd640 1 -- 172.21.15.88:0/373289902 <== mon.0 v1:172.21.15.88:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 764+0+0 (unknown 1915288945 0 0) 0x7ffb14004eb0 con 0x7ffb3013eec0 2024-06-03T09:06:38.787 INFO:teuthology.orchestra.run.smithi088.stderr:2024-06-03T09:06:38.787+0000 7ffb26ffd640 1 -- 172.21.15.88:0/373289902 --> v1:172.21.15.88:6789/0 -- auth(proto 2 165 bytes epoch 0) v1 -- 0x7ffb3013f380 con 0x7ffb3013eec0 2024-06-03T09:06:38.788 INFO:teuthology.orchestra.run.smithi088.stderr:2024-06-03T09:06:38.788+0000 7ffb26ffd640 1 -- 172.21.15.88:0/373289902 <== mon.0 v1:172.21.15.88:6789/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 2234374065 0 0) 0x7ffb20001860 con 0x7ffb3013eec0 2024-06-03T09:06:38.788 INFO:teuthology.orchestra.run.smithi088.stderr:2024-06-03T09:06:38.788+0000 7ffb26ffd640 10 monclient(hunting): handle_monmap mon_map magic: 0 v1 2024-06-03T09:06:38.788 INFO:teuthology.orchestra.run.smithi088.stderr:2024-06-03T09:06:38.788+0000 7ffb26ffd640 10 monclient(hunting): got monmap 1 from mon.noname-a (according to old e1) 2024-06-03T09:06:38.788 INFO:teuthology.orchestra.run.smithi088.stderr:2024-06-03T09:06:38.788+0000 7ffb26ffd640 10 monclient(hunting): dump: 2024-06-03T09:06:38.788 INFO:teuthology.orchestra.run.smithi088.stderr:epoch 1 2024-06-03T09:06:38.788 INFO:teuthology.orchestra.run.smithi088.stderr:fsid ec3c3fdd-3ac6-4d7d-9412-683130ed8b04 2024-06-03T09:06:38.788 INFO:teuthology.orchestra.run.smithi088.stderr:last_changed 2024-06-03T08:34:31.885125+0000 2024-06-03T09:06:38.789 INFO:teuthology.orchestra.run.smithi088.stderr:created 2024-06-03T08:34:31.885125+0000 2024-06-03T09:06:38.789 INFO:teuthology.orchestra.run.smithi088.stderr:min_mon_release 18 (reef) 2024-06-03T09:06:38.789 INFO:teuthology.orchestra.run.smithi088.stderr:election_strategy: 3 2024-06-03T09:06:38.789 INFO:teuthology.orchestra.run.smithi088.stderr:0: v1:172.21.15.88:6789/0 mon.a 2024-06-03T09:06:38.789 INFO:teuthology.orchestra.run.smithi088.stderr:1: v1:172.21.15.192:6789/0 mon.b 2024-06-03T09:06:38.789 INFO:teuthology.orchestra.run.smithi088.stderr:2: v1:172.21.15.88:6790/0 mon.c 2024-06-03T09:06:38.789 INFO:teuthology.orchestra.run.smithi088.stderr: 2024-06-03T09:06:38.789 INFO:teuthology.orchestra.run.smithi088.stderr:2024-06-03T09:06:38.788+0000 7ffb26ffd640 10 monclient(hunting): _finish_auth 0 2024-06-03T09:06:38.791 INFO:teuthology.orchestra.run.smithi088.stderr:2024-06-03T09:06:38.790+0000 7ffb26ffd640 1 -- 172.21.15.88:0/373289902 <== mon.2 v1:172.21.15.88:6790/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 764+0+0 (unknown 2754393658 0 0) 0x7ffb301d6740 con 0x7ffb300a8960 2024-06-03T09:06:38.791 INFO:teuthology.orchestra.run.smithi088.stderr:2024-06-03T09:06:38.790+0000 7ffb26ffd640 1 -- 172.21.15.88:0/373289902 --> v1:172.21.15.88:6790/0 -- auth(proto 2 165 bytes epoch 0) v1 -- 0x7ffb14004eb0 con 0x7ffb300a8960 2024-06-03T09:06:38.791 INFO:teuthology.orchestra.run.smithi088.stderr:2024-06-03T09:06:38.790+0000 7ffb26ffd640 1 -- 172.21.15.88:0/373289902 <== mon.2 v1:172.21.15.88:6790/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 2234374065 0 0) 0x7ffb2c0014f0 con 0x7ffb300a8960 2024-06-03T09:06:38.791 INFO:teuthology.orchestra.run.smithi088.stderr:2024-06-03T09:06:38.790+0000 7ffb26ffd640 10 monclient(hunting): handle_monmap mon_map magic: 0 v1 2024-06-03T09:06:38.791 INFO:teuthology.orchestra.run.smithi088.stderr:2024-06-03T09:06:38.790+0000 7ffb26ffd640 10 monclient(hunting): got monmap 1 from mon.c (according to old e1) 2024-06-03T09:06:38.791 INFO:teuthology.orchestra.run.smithi088.stderr:2024-06-03T09:06:38.790+0000 7ffb26ffd640 10 monclient(hunting): dump: 2024-06-03T09:06:38.791 INFO:teuthology.orchestra.run.smithi088.stderr:epoch 1 2024-06-03T09:06:38.791 INFO:teuthology.orchestra.run.smithi088.stderr:fsid ec3c3fdd-3ac6-4d7d-9412-683130ed8b04 2024-06-03T09:06:38.791 INFO:teuthology.orchestra.run.smithi088.stderr:last_changed 2024-06-03T08:34:31.885125+0000 2024-06-03T09:06:38.791 INFO:teuthology.orchestra.run.smithi088.stderr:created 2024-06-03T08:34:31.885125+0000 2024-06-03T09:06:38.791 INFO:teuthology.orchestra.run.smithi088.stderr:min_mon_release 18 (reef) 2024-06-03T09:06:38.791 INFO:teuthology.orchestra.run.smithi088.stderr:election_strategy: 3 2024-06-03T09:06:38.791 INFO:teuthology.orchestra.run.smithi088.stderr:0: v1:172.21.15.88:6789/0 mon.a 2024-06-03T09:06:38.792 INFO:teuthology.orchestra.run.smithi088.stderr:1: v1:172.21.15.192:6789/0 mon.b 2024-06-03T09:06:38.792 INFO:teuthology.orchestra.run.smithi088.stderr:2: v1:172.21.15.88:6790/0 mon.c 2024-06-03T09:06:38.792 INFO:teuthology.orchestra.run.smithi088.stderr: 2024-06-03T09:06:38.796 INFO:teuthology.orchestra.run.smithi088.stderr:2024-06-03T09:06:38.795+0000 7ffb26ffd640 1 -- 172.21.15.88:0/373289902 <== mon.0 v1:172.21.15.88:6789/0 4 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 580+0+0 (unknown 2122355327 0 0) 0x7ffb3013f380 con 0x7ffb3013eec0 2024-06-03T09:06:38.796 INFO:teuthology.orchestra.run.smithi088.stderr:2024-06-03T09:06:38.795+0000 7ffb26ffd640 1 -- 172.21.15.88:0/373289902 >> v1:172.21.15.88:6790/0 conn(0x7ffb300a8960 legacy=0x7ffb300aaea0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-06-03T09:06:38.796 INFO:teuthology.orchestra.run.smithi088.stderr:2024-06-03T09:06:38.795+0000 7ffb26ffd640 1 -- 172.21.15.88:0/373289902 >> v1:172.21.15.192:6789/0 conn(0x7ffb301450f0 legacy=0x7ffb301d6bf0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-06-03T09:06:38.796 INFO:teuthology.orchestra.run.smithi088.stderr:2024-06-03T09:06:38.795+0000 7ffb26ffd640 10 monclient: _finish_hunting 0 2024-06-03T09:06:38.796 INFO:teuthology.orchestra.run.smithi088.stderr:2024-06-03T09:06:38.795+0000 7ffb26ffd640 1 monclient: found mon.a 2024-06-03T09:06:38.796 INFO:teuthology.orchestra.run.smithi088.stderr:2024-06-03T09:06:38.795+0000 7ffb26ffd640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-06-03T09:06:38.796 INFO:teuthology.orchestra.run.smithi088.stderr:2024-06-03T09:06:38.795+0000 7ffb26ffd640 10 monclient: _send_mon_message to mon.a at v1:172.21.15.88:6789/0 2024-06-03T09:06:38.796 INFO:teuthology.orchestra.run.smithi088.stderr:2024-06-03T09:06:38.795+0000 7ffb26ffd640 1 -- 172.21.15.88:0/373289902 --> v1:172.21.15.88:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7ffb3013f510 con 0x7ffb3013eec0 2024-06-03T09:06:38.796 INFO:teuthology.orchestra.run.smithi088.stderr:2024-06-03T09:06:38.795+0000 7ffb26ffd640 10 monclient: _finish_auth 0 2024-06-03T09:06:38.796 INFO:teuthology.orchestra.run.smithi088.stderr:2024-06-03T09:06:38.795+0000 7ffb26ffd640 10 monclient: _check_auth_tickets 2024-06-03T09:06:38.797 INFO:teuthology.orchestra.run.smithi088.stderr:2024-06-03T09:06:38.795+0000 7ffb26ffd640 20 monclient: _check_auth_rotating not needed by client.admin 2024-06-03T09:06:38.797 INFO:teuthology.orchestra.run.smithi088.stderr:2024-06-03T09:06:38.795+0000 7ffb38300640 5 monclient: authenticate success, global_id 30669 2024-06-03T09:06:38.797 INFO:teuthology.orchestra.run.smithi088.stderr:2024-06-03T09:06:38.795+0000 7ffb38300640 10 monclient: _renew_subs 2024-06-03T09:06:38.797 INFO:teuthology.orchestra.run.smithi088.stderr:2024-06-03T09:06:38.795+0000 7ffb38300640 10 monclient: _send_mon_message to mon.a at v1:172.21.15.88:6789/0 2024-06-03T09:06:38.797 INFO:teuthology.orchestra.run.smithi088.stderr:2024-06-03T09:06:38.795+0000 7ffb38300640 1 -- 172.21.15.88:0/373289902 --> v1:172.21.15.88:6789/0 -- mon_subscribe({mgrmap=0+}) v3 -- 0x7ffb301d7300 con 0x7ffb3013eec0 2024-06-03T09:06:38.797 INFO:teuthology.orchestra.run.smithi088.stderr:2024-06-03T09:06:38.795+0000 7ffb38300640 10 client.30669.objecter _maybe_request_map subscribing (onetime) to next osd map 2024-06-03T09:06:38.797 INFO:teuthology.orchestra.run.smithi088.stderr:2024-06-03T09:06:38.795+0000 7ffb38300640 10 monclient: _renew_subs 2024-06-03T09:06:38.797 INFO:teuthology.orchestra.run.smithi088.stderr:2024-06-03T09:06:38.795+0000 7ffb38300640 10 monclient: _send_mon_message to mon.a at v1:172.21.15.88:6789/0 2024-06-03T09:06:38.797 INFO:teuthology.orchestra.run.smithi088.stderr:2024-06-03T09:06:38.795+0000 7ffb38300640 1 -- 172.21.15.88:0/373289902 --> v1:172.21.15.88:6789/0 -- mon_subscribe({osdmap=0}) v3 -- 0x7ffb30137260 con 0x7ffb3013eec0 2024-06-03T09:06:38.797 INFO:teuthology.orchestra.run.smithi088.stderr:2024-06-03T09:06:38.795+0000 7ffb38300640 1 librados: init done 2024-06-03T09:06:38.798 INFO:teuthology.orchestra.run.smithi088.stderr:2024-06-03T09:06:38.796+0000 7ffb38300640 10 start_mon_command cmd=[{"prefix": "get_command_descriptions"}] 2024-06-03T09:06:38.798 INFO:teuthology.orchestra.run.smithi088.stderr:2024-06-03T09:06:38.796+0000 7ffb38300640 10 monclient: _send_command 1 [{"prefix": "get_command_descriptions"}] 2024-06-03T09:06:38.798 INFO:teuthology.orchestra.run.smithi088.stderr:2024-06-03T09:06:38.796+0000 7ffb38300640 10 monclient: _send_mon_message to mon.a at v1:172.21.15.88:6789/0 2024-06-03T09:06:38.798 INFO:teuthology.orchestra.run.smithi088.stderr:2024-06-03T09:06:38.796+0000 7ffb38300640 1 -- 172.21.15.88:0/373289902 --> v1:172.21.15.88:6789/0 -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7ffb300a9e60 con 0x7ffb3013eec0 2024-06-03T09:06:39.062 INFO:teuthology.orchestra.run.smithi088.stderr:2024-06-03T09:06:39.062+0000 7ffb26ffd640 1 -- 172.21.15.88:0/373289902 <== mon.0 v1:172.21.15.88:6789/0 5 ==== config(0 keys) v1 ==== 4+0+0 (unknown 0 0 0) 0x7ffb200047e0 con 0x7ffb3013eec0 2024-06-03T09:06:39.062 INFO:teuthology.orchestra.run.smithi088.stderr:2024-06-03T09:06:39.062+0000 7ffb26ffd640 10 monclient: handle_config config(0 keys) v1 2024-06-03T09:06:39.062 INFO:teuthology.orchestra.run.smithi088.stderr:2024-06-03T09:06:39.062+0000 7ffb26ffd640 1 -- 172.21.15.88:0/373289902 <== mon.0 v1:172.21.15.88:6789/0 6 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 2234374065 0 0) 0x7ffb20004950 con 0x7ffb3013eec0 2024-06-03T09:06:39.062 INFO:teuthology.orchestra.run.smithi088.stderr:2024-06-03T09:06:39.062+0000 7ffb26ffd640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-06-03T09:06:39.062 INFO:teuthology.orchestra.run.smithi088.stderr:2024-06-03T09:06:39.062+0000 7ffb26ffd640 10 monclient: got monmap 1 from mon.a (according to old e1) 2024-06-03T09:06:39.062 INFO:teuthology.orchestra.run.smithi088.stderr:2024-06-03T09:06:39.062+0000 7ffb26ffd640 10 monclient: dump: 2024-06-03T09:06:39.062 INFO:teuthology.orchestra.run.smithi088.stderr:epoch 1 2024-06-03T09:06:39.062 INFO:teuthology.orchestra.run.smithi088.stderr:fsid ec3c3fdd-3ac6-4d7d-9412-683130ed8b04 2024-06-03T09:06:39.063 INFO:teuthology.orchestra.run.smithi088.stderr:last_changed 2024-06-03T08:34:31.885125+0000 2024-06-03T09:06:39.063 INFO:teuthology.orchestra.run.smithi088.stderr:created 2024-06-03T08:34:31.885125+0000 2024-06-03T09:06:39.063 INFO:teuthology.orchestra.run.smithi088.stderr:min_mon_release 18 (reef) 2024-06-03T09:06:39.063 INFO:teuthology.orchestra.run.smithi088.stderr:election_strategy: 3 2024-06-03T09:06:39.063 INFO:teuthology.orchestra.run.smithi088.stderr:0: v1:172.21.15.88:6789/0 mon.a 2024-06-03T09:06:39.063 INFO:teuthology.orchestra.run.smithi088.stderr:1: v1:172.21.15.192:6789/0 mon.b 2024-06-03T09:06:39.063 INFO:teuthology.orchestra.run.smithi088.stderr:2: v1:172.21.15.88:6790/0 mon.c 2024-06-03T09:06:39.063 INFO:teuthology.orchestra.run.smithi088.stderr: 2024-06-03T09:06:39.063 INFO:teuthology.orchestra.run.smithi088.stderr:2024-06-03T09:06:39.062+0000 7ffb34899640 4 set_mon_vals no callback set 2024-06-03T09:06:39.064 INFO:teuthology.orchestra.run.smithi088.stderr:2024-06-03T09:06:39.063+0000 7ffb26ffd640 1 -- 172.21.15.88:0/373289902 <== mon.0 v1:172.21.15.88:6789/0 7 ==== mgrmap(e 5) v1 ==== 89110+0+0 (unknown 873533973 0 0) 0x7ffb2001a6d0 con 0x7ffb3013eec0 2024-06-03T09:06:39.064 INFO:teuthology.orchestra.run.smithi088.stderr:2024-06-03T09:06:39.063+0000 7ffb3609c640 10 monclient: get_auth_request con 0x7ffb1406d1b0 auth_method 0 2024-06-03T09:06:39.064 INFO:teuthology.orchestra.run.smithi088.stderr:2024-06-03T09:06:39.064+0000 7ffb26ffd640 10 client.30669.objecter ms_handle_connect 0x7ffb1406d1b0 2024-06-03T09:06:39.070 INFO:teuthology.orchestra.run.smithi088.stderr:2024-06-03T09:06:39.069+0000 7ffb26ffd640 1 -- 172.21.15.88:0/373289902 <== mon.0 v1:172.21.15.88:6789/0 8 ==== osd_map(1010..1010 src has 622..1010) v4 ==== 5229+0+0 (unknown 1105883993 0 0) 0x7ffb20053f80 con 0x7ffb3013eec0 2024-06-03T09:06:39.070 INFO:teuthology.orchestra.run.smithi088.stderr:2024-06-03T09:06:39.069+0000 7ffb26ffd640 10 client.30669.objecter ms_dispatch 0x7ffb30000b90 osd_map(1010..1010 src has 622..1010) v4 2024-06-03T09:06:39.070 INFO:teuthology.orchestra.run.smithi088.stderr:2024-06-03T09:06:39.069+0000 7ffb26ffd640 3 client.30669.objecter handle_osd_map got epochs [1010,1010] > 0 2024-06-03T09:06:39.070 INFO:teuthology.orchestra.run.smithi088.stderr:2024-06-03T09:06:39.069+0000 7ffb26ffd640 3 client.30669.objecter handle_osd_map decoding full epoch 1010 2024-06-03T09:06:39.070 INFO:teuthology.orchestra.run.smithi088.stderr:2024-06-03T09:06:39.069+0000 7ffb26ffd640 20 client.30669.objecter dump_active .. 0 homeless 2024-06-03T09:06:39.070 INFO:teuthology.orchestra.run.smithi088.stderr:2024-06-03T09:06:39.069+0000 7ffb26ffd640 1 -- 172.21.15.88:0/373289902 <== mon.0 v1:172.21.15.88:6789/0 9 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+182498 (unknown 1092875540 0 3016760700) 0x7ffb2004fb30 con 0x7ffb3013eec0 2024-06-03T09:06:39.070 INFO:teuthology.orchestra.run.smithi088.stderr:2024-06-03T09:06:39.069+0000 7ffb26ffd640 10 monclient: handle_mon_command_ack 1 [{"prefix": "get_command_descriptions"}] 2024-06-03T09:06:39.070 INFO:teuthology.orchestra.run.smithi088.stderr:2024-06-03T09:06:39.069+0000 7ffb26ffd640 10 monclient: _finish_command 1 = system:0 2024-06-03T09:06:39.251 INFO:teuthology.orchestra.run.smithi088.stderr:2024-06-03T09:06:39.250+0000 7ffb38300640 1 -- 172.21.15.88:0/373289902 --> v1:172.21.15.192:6800/34229 -- mgr_command(tid 0: {"prefix": "pg dump", "target": ["mon-mgr", ""], "format": "json"}) v1 -- 0x7ffb3004ea60 con 0x7ffb1406d1b0 2024-06-03T09:06:39.255 INFO:teuthology.orchestra.run.smithi088.stderr:2024-06-03T09:06:39.255+0000 7ffb26ffd640 1 -- 172.21.15.88:0/373289902 <== mgr.4110 v1:172.21.15.192:6800/34229 1 ==== mgr_command_reply(tid 0: 0 dumped all) v1 ==== 18+0+151499 (unknown 2965378022 0 3395572766) 0x7ffb3004ea60 con 0x7ffb1406d1b0 2024-06-03T09:06:39.257 INFO:teuthology.orchestra.run.smithi088.stdout: 2024-06-03T09:06:39.258 INFO:teuthology.orchestra.run.smithi088.stderr:dumped all 2024-06-03T09:06:39.259 INFO:teuthology.orchestra.run.smithi088.stderr:2024-06-03T09:06:39.258+0000 7ffb38300640 10 librados: watch_flush enter 2024-06-03T09:06:39.260 INFO:teuthology.orchestra.run.smithi088.stderr:2024-06-03T09:06:39.258+0000 7ffb38300640 10 librados: watch_flush exit 2024-06-03T09:06:39.260 INFO:teuthology.orchestra.run.smithi088.stderr:2024-06-03T09:06:39.258+0000 7ffb38300640 20 client.30669.objecter shutdown clearing up homeless session... 2024-06-03T09:06:39.260 INFO:teuthology.orchestra.run.smithi088.stderr:2024-06-03T09:06:39.258+0000 7ffb38300640 10 client.30669.objecter successfully canceled tick 2024-06-03T09:06:39.260 INFO:teuthology.orchestra.run.smithi088.stderr:2024-06-03T09:06:39.258+0000 7ffb38300640 1 -- 172.21.15.88:0/373289902 >> v1:172.21.15.192:6800/34229 conn(0x7ffb1406d1b0 legacy=0x7ffb1406f670 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-06-03T09:06:39.260 INFO:teuthology.orchestra.run.smithi088.stderr:2024-06-03T09:06:39.258+0000 7ffb38300640 10 monclient: shutdown 2024-06-03T09:06:39.260 INFO:teuthology.orchestra.run.smithi088.stderr:2024-06-03T09:06:39.258+0000 7ffb38300640 20 monclient: shutdown discarding 0 pending message(s) 2024-06-03T09:06:39.260 INFO:teuthology.orchestra.run.smithi088.stderr:2024-06-03T09:06:39.258+0000 7ffb38300640 1 -- 172.21.15.88:0/373289902 >> v1:172.21.15.88:6789/0 conn(0x7ffb3013eec0 legacy=0x7ffb30141cd0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-06-03T09:06:39.260 INFO:teuthology.orchestra.run.smithi088.stderr:2024-06-03T09:06:39.258+0000 7ffb38300640 1 -- 172.21.15.88:0/373289902 shutdown_connections 2024-06-03T09:06:39.260 INFO:teuthology.orchestra.run.smithi088.stderr:2024-06-03T09:06:39.258+0000 7ffb38300640 1 -- 172.21.15.88:0/373289902 >> 172.21.15.88:0/373289902 conn(0x7ffb300b70d0 msgr2=0x7ffb3013e580 unknown :-1 s=STATE_NONE l=0).mark_down 2024-06-03T09:06:39.260 INFO:teuthology.orchestra.run.smithi088.stderr:2024-06-03T09:06:39.258+0000 7ffb38300640 1 -- 172.21.15.88:0/373289902 shutdown_connections 2024-06-03T09:06:39.260 INFO:teuthology.orchestra.run.smithi088.stderr:2024-06-03T09:06:39.258+0000 7ffb38300640 1 -- 172.21.15.88:0/373289902 wait complete. 2024-06-03T09:06:39.261 INFO:teuthology.orchestra.run.smithi088.stderr:2024-06-03T09:06:39.258+0000 7ffb38300640 1 librados: shutdown 2024-06-03T09:06:39.269 INFO:teuthology.orchestra.run.smithi088.stdout:{"pg_ready":true,"pg_map":{"version":1925,"stamp":"2024-06-03T09:06:39.102221+0000","last_osdmap_epoch":0,"last_pg_scan":0,"pg_stats_sum":{"stat_sum":{"num_bytes":459285,"num_objects":3,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":1,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":3,"num_whiteouts":0,"num_read":102,"num_read_kb":88,"num_write":120,"num_write_kb":1379,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2,"num_bytes_recovered":397840,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":83,"ondisk_log_size":83,"up":68,"acting":68,"num_store_stats":0},"osd_stats_sum":{"up_from":0,"seq":0,"num_pgs":100,"num_osds":8,"num_per_pool_osds":8,"num_per_pool_omap_osds":8,"kb":754974720,"kb_used":494548,"kb_used_data":46324,"kb_used_omap":12,"kb_used_meta":448115,"kb_avail":754480172,"statfs":{"total":773094113280,"available":772587696128,"internally_reserved":0,"allocated":47435776,"data_stored":22747954,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":12744,"internal_metadata":458870328},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":109,"apply_latency_ms":109,"commit_latency_ns":109000000,"apply_latency_ns":109000000},"alerts":[],"network_ping_times":[]},"pg_stats_delta":{"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":0,"acting":0,"num_store_stats":0,"stamp_delta":"12.002344"},"pg_stats":[{"pgid":"17.d","version":"0'0","reported_seq":942,"reported_epoch":1009,"state":"active+clean","last_fresh":"2024-06-03T09:06:14.443917+0000","last_change":"2024-06-03T08:38:19.260548+0000","last_active":"2024-06-03T09:06:14.443917+0000","last_peered":"2024-06-03T09:06:14.443917+0000","last_clean":"2024-06-03T09:06:14.443917+0000","last_became_active":"2024-06-03T08:38:19.253559+0000","last_became_peered":"2024-06-03T08:38:19.253559+0000","last_unstale":"2024-06-03T09:06:14.443917+0000","last_undegraded":"2024-06-03T09:06:14.443917+0000","last_fullsized":"2024-06-03T09:06:14.443917+0000","mapping_epoch":28,"log_start":"0'0","ondisk_log_start":"0'0","created":28,"last_epoch_clean":45,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-06-03T08:37:55.614497+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-06-03T08:37:55.614497+0000","last_clean_scrub_stamp":"2024-06-03T08:37:55.614497+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-06-04T19:00:12.470979+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1],"acting":[0,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"17.c","version":"0'0","reported_seq":953,"reported_epoch":1009,"state":"active+undersized","last_fresh":"2024-06-03T09:06:13.668156+0000","last_change":"2024-06-03T09:06:03.440074+0000","last_active":"2024-06-03T09:06:13.668156+0000","last_peered":"2024-06-03T09:06:13.668156+0000","last_clean":"2024-06-03T09:05:02.105638+0000","last_became_active":"2024-06-03T09:06:03.440074+0000","last_became_peered":"2024-06-03T09:06:03.440074+0000","last_unstale":"2024-06-03T09:06:13.668156+0000","last_undegraded":"2024-06-03T09:06:13.668156+0000","last_fullsized":"2024-06-03T09:06:03.377982+0000","mapping_epoch":999,"log_start":"0'0","ondisk_log_start":"0'0","created":28,"last_epoch_clean":255,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-06-03T08:37:55.614497+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-06-03T08:37:55.614497+0000","last_clean_scrub_stamp":"2024-06-03T08:37:55.614497+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-06-04T10:34:20.808557+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3],"acting":[3],"avail_no_missing":["3"],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"17.f","version":"0'0","reported_seq":148,"reported_epoch":1009,"state":"active+undersized","last_fresh":"2024-06-03T09:06:14.450543+0000","last_change":"2024-06-03T09:06:03.420285+0000","last_active":"2024-06-03T09:06:14.450543+0000","last_peered":"2024-06-03T09:06:14.450543+0000","last_clean":"2024-06-03T08:37:55.614497+0000","last_became_active":"2024-06-03T09:06:03.420285+0000","last_became_peered":"2024-06-03T09:06:03.420285+0000","last_unstale":"2024-06-03T09:06:14.450543+0000","last_undegraded":"2024-06-03T09:06:14.450543+0000","last_fullsized":"2024-06-03T09:06:03.380616+0000","mapping_epoch":999,"log_start":"0'0","ondisk_log_start":"0'0","created":28,"last_epoch_clean":255,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-06-03T08:37:55.614497+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-06-03T08:37:55.614497+0000","last_clean_scrub_stamp":"2024-06-03T08:37:55.614497+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-06-04T15:39:41.187731+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4],"acting":[4],"avail_no_missing":["4"],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"17.e","version":"0'0","reported_seq":149,"reported_epoch":1009,"state":"active+undersized","last_fresh":"2024-06-03T09:06:13.726992+0000","last_change":"2024-06-03T09:06:03.412887+0000","last_active":"2024-06-03T09:06:13.726992+0000","last_peered":"2024-06-03T09:06:13.726992+0000","last_clean":"2024-06-03T08:37:55.614497+0000","last_became_active":"2024-06-03T09:06:03.412887+0000","last_became_peered":"2024-06-03T09:06:03.412887+0000","last_unstale":"2024-06-03T09:06:13.726992+0000","last_undegraded":"2024-06-03T09:06:13.726992+0000","last_fullsized":"2024-06-03T09:06:03.332482+0000","mapping_epoch":999,"log_start":"0'0","ondisk_log_start":"0'0","created":28,"last_epoch_clean":255,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-06-03T08:37:55.614497+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-06-03T08:37:55.614497+0000","last_clean_scrub_stamp":"2024-06-03T08:37:55.614497+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-06-04T12:23:22.511601+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5],"acting":[5],"avail_no_missing":["5"],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"17.9","version":"0'0","reported_seq":946,"reported_epoch":1009,"state":"active+clean","last_fresh":"2024-06-03T09:06:14.999299+0000","last_change":"2024-06-03T08:38:04.772260+0000","last_active":"2024-06-03T09:06:14.999299+0000","last_peered":"2024-06-03T09:06:14.999299+0000","last_clean":"2024-06-03T09:06:14.999299+0000","last_became_active":"2024-06-03T08:38:04.763428+0000","last_became_peered":"2024-06-03T08:38:04.763428+0000","last_unstale":"2024-06-03T09:06:14.999299+0000","last_undegraded":"2024-06-03T09:06:14.999299+0000","last_fullsized":"2024-06-03T09:06:14.999299+0000","mapping_epoch":28,"log_start":"0'0","ondisk_log_start":"0'0","created":28,"last_epoch_clean":33,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-06-03T08:37:55.614497+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-06-03T08:37:55.614497+0000","last_clean_scrub_stamp":"2024-06-03T08:37:55.614497+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-06-04T10:55:10.773936+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,4],"acting":[7,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"17.8","version":"0'0","reported_seq":961,"reported_epoch":1009,"state":"active+undersized","last_fresh":"2024-06-03T09:06:14.430375+0000","last_change":"2024-06-03T09:06:03.388786+0000","last_active":"2024-06-03T09:06:14.430375+0000","last_peered":"2024-06-03T09:06:14.430375+0000","last_clean":"2024-06-03T09:04:43.297780+0000","last_became_active":"2024-06-03T09:06:03.388786+0000","last_became_peered":"2024-06-03T09:06:03.388786+0000","last_unstale":"2024-06-03T09:06:14.430375+0000","last_undegraded":"2024-06-03T09:06:14.430375+0000","last_fullsized":"2024-06-03T09:06:03.331754+0000","mapping_epoch":999,"log_start":"0'0","ondisk_log_start":"0'0","created":28,"last_epoch_clean":255,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-06-03T08:37:55.614497+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-06-03T08:37:55.614497+0000","last_clean_scrub_stamp":"2024-06-03T08:37:55.614497+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-06-04T09:18:44.765547+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0],"acting":[0],"avail_no_missing":["0"],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"17.b","version":"0'0","reported_seq":149,"reported_epoch":1009,"state":"active+undersized","last_fresh":"2024-06-03T09:06:14.433735+0000","last_change":"2024-06-03T09:06:03.451203+0000","last_active":"2024-06-03T09:06:14.433735+0000","last_peered":"2024-06-03T09:06:14.433735+0000","last_clean":"2024-06-03T08:37:55.614497+0000","last_became_active":"2024-06-03T09:06:03.451203+0000","last_became_peered":"2024-06-03T09:06:03.451203+0000","last_unstale":"2024-06-03T09:06:14.433735+0000","last_undegraded":"2024-06-03T09:06:14.433735+0000","last_fullsized":"2024-06-03T09:06:03.375523+0000","mapping_epoch":999,"log_start":"0'0","ondisk_log_start":"0'0","created":28,"last_epoch_clean":255,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-06-03T08:37:55.614497+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-06-03T08:37:55.614497+0000","last_clean_scrub_stamp":"2024-06-03T08:37:55.614497+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-06-04T10:16:19.813495+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2],"acting":[2],"avail_no_missing":["2"],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"17.a","version":"0'0","reported_seq":925,"reported_epoch":1010,"state":"active+clean","last_fresh":"2024-06-03T09:06:18.651697+0000","last_change":"2024-06-03T08:38:05.843142+0000","last_active":"2024-06-03T09:06:18.651697+0000","last_peered":"2024-06-03T09:06:18.651697+0000","last_clean":"2024-06-03T09:06:18.651697+0000","last_became_active":"2024-06-03T08:38:05.809257+0000","last_became_peered":"2024-06-03T08:38:05.809257+0000","last_unstale":"2024-06-03T09:06:18.651697+0000","last_undegraded":"2024-06-03T09:06:18.651697+0000","last_fullsized":"2024-06-03T09:06:18.651697+0000","mapping_epoch":28,"log_start":"0'0","ondisk_log_start":"0'0","created":28,"last_epoch_clean":33,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-06-03T08:37:55.614497+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-06-03T08:37:55.614497+0000","last_clean_scrub_stamp":"2024-06-03T08:37:55.614497+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-06-04T12:12:03.686838+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2],"acting":[1,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"17.5","version":"0'0","reported_seq":943,"reported_epoch":1009,"state":"active+clean","last_fresh":"2024-06-03T09:06:14.442357+0000","last_change":"2024-06-03T08:38:23.466051+0000","last_active":"2024-06-03T09:06:14.442357+0000","last_peered":"2024-06-03T09:06:14.442357+0000","last_clean":"2024-06-03T09:06:14.442357+0000","last_became_active":"2024-06-03T08:38:23.458039+0000","last_became_peered":"2024-06-03T08:38:23.458039+0000","last_unstale":"2024-06-03T09:06:14.442357+0000","last_undegraded":"2024-06-03T09:06:14.442357+0000","last_fullsized":"2024-06-03T09:06:14.442357+0000","mapping_epoch":28,"log_start":"0'0","ondisk_log_start":"0'0","created":28,"last_epoch_clean":49,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-06-03T08:37:55.614497+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-06-03T08:37:55.614497+0000","last_clean_scrub_stamp":"2024-06-03T08:37:55.614497+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-06-04T12:27:02.532436+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,4],"acting":[0,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"17.4","version":"0'0","reported_seq":149,"reported_epoch":1009,"state":"active+undersized","last_fresh":"2024-06-03T09:06:13.941522+0000","last_change":"2024-06-03T09:06:03.441889+0000","last_active":"2024-06-03T09:06:13.941522+0000","last_peered":"2024-06-03T09:06:13.941522+0000","last_clean":"2024-06-03T08:37:55.614497+0000","last_became_active":"2024-06-03T09:06:03.441889+0000","last_became_peered":"2024-06-03T09:06:03.441889+0000","last_unstale":"2024-06-03T09:06:13.941522+0000","last_undegraded":"2024-06-03T09:06:13.941522+0000","last_fullsized":"2024-06-03T09:06:03.359582+0000","mapping_epoch":999,"log_start":"0'0","ondisk_log_start":"0'0","created":28,"last_epoch_clean":255,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-06-03T08:37:55.614497+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-06-03T08:37:55.614497+0000","last_clean_scrub_stamp":"2024-06-03T08:37:55.614497+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-06-04T12:49:22.907141+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7],"acting":[7],"avail_no_missing":["7"],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"17.7","version":"0'0","reported_seq":929,"reported_epoch":1009,"state":"active+clean","last_fresh":"2024-06-03T09:06:13.736920+0000","last_change":"2024-06-03T08:38:04.837083+0000","last_active":"2024-06-03T09:06:13.736920+0000","last_peered":"2024-06-03T09:06:13.736920+0000","last_clean":"2024-06-03T09:06:13.736920+0000","last_became_active":"2024-06-03T08:38:04.818246+0000","last_became_peered":"2024-06-03T08:38:04.818246+0000","last_unstale":"2024-06-03T09:06:13.736920+0000","last_undegraded":"2024-06-03T09:06:13.736920+0000","last_fullsized":"2024-06-03T09:06:13.736920+0000","mapping_epoch":28,"log_start":"0'0","ondisk_log_start":"0'0","created":28,"last_epoch_clean":34,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-06-03T08:37:55.614497+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-06-03T08:37:55.614497+0000","last_clean_scrub_stamp":"2024-06-03T08:37:55.614497+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-06-04T17:02:38.204813+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,2],"acting":[5,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"17.6","version":"0'0","reported_seq":960,"reported_epoch":1009,"state":"active+undersized","last_fresh":"2024-06-03T09:06:14.453110+0000","last_change":"2024-06-03T09:06:03.428677+0000","last_active":"2024-06-03T09:06:14.453110+0000","last_peered":"2024-06-03T09:06:14.453110+0000","last_clean":"2024-06-03T09:04:43.312352+0000","last_became_active":"2024-06-03T09:06:03.428677+0000","last_became_peered":"2024-06-03T09:06:03.428677+0000","last_unstale":"2024-06-03T09:06:14.453110+0000","last_undegraded":"2024-06-03T09:06:14.453110+0000","last_fullsized":"2024-06-03T09:06:03.359909+0000","mapping_epoch":999,"log_start":"0'0","ondisk_log_start":"0'0","created":28,"last_epoch_clean":255,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-06-03T08:37:55.614497+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-06-03T08:37:55.614497+0000","last_clean_scrub_stamp":"2024-06-03T08:37:55.614497+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-06-04T15:38:02.180241+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0],"acting":[0],"avail_no_missing":["0"],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"2.7","version":"0'0","reported_seq":940,"reported_epoch":1009,"state":"active+clean","last_fresh":"2024-06-03T09:06:13.684065+0000","last_change":"2024-06-03T08:36:30.385405+0000","last_active":"2024-06-03T09:06:13.684065+0000","last_peered":"2024-06-03T09:06:13.684065+0000","last_clean":"2024-06-03T09:06:13.684065+0000","last_became_active":"2024-06-03T08:36:30.380302+0000","last_became_peered":"2024-06-03T08:36:30.380302+0000","last_unstale":"2024-06-03T09:06:13.684065+0000","last_undegraded":"2024-06-03T09:06:13.684065+0000","last_fullsized":"2024-06-03T09:06:13.684065+0000","mapping_epoch":23,"log_start":"0'0","ondisk_log_start":"0'0","created":23,"last_epoch_clean":24,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-06-03T08:36:28.444764+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-06-03T08:36:28.444764+0000","last_clean_scrub_stamp":"2024-06-03T08:36:28.444764+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-06-04T18:46:33.475181+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0],"acting":[3,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"17.14","version":"0'0","reported_seq":926,"reported_epoch":1009,"state":"active+clean","last_fresh":"2024-06-03T09:06:13.721460+0000","last_change":"2024-06-03T08:38:04.547565+0000","last_active":"2024-06-03T09:06:13.721460+0000","last_peered":"2024-06-03T09:06:13.721460+0000","last_clean":"2024-06-03T09:06:13.721460+0000","last_became_active":"2024-06-03T08:38:04.541781+0000","last_became_peered":"2024-06-03T08:38:04.541781+0000","last_unstale":"2024-06-03T09:06:13.721460+0000","last_undegraded":"2024-06-03T09:06:13.721460+0000","last_fullsized":"2024-06-03T09:06:13.721460+0000","mapping_epoch":28,"log_start":"0'0","ondisk_log_start":"0'0","created":28,"last_epoch_clean":34,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-06-03T08:37:55.614497+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-06-03T08:37:55.614497+0000","last_clean_scrub_stamp":"2024-06-03T08:37:55.614497+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-06-04T16:41:55.170134+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,2],"acting":[5,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"2.6","version":"0'0","reported_seq":949,"reported_epoch":1009,"state":"active+clean","last_fresh":"2024-06-03T09:06:14.448100+0000","last_change":"2024-06-03T08:36:30.430760+0000","last_active":"2024-06-03T09:06:14.448100+0000","last_peered":"2024-06-03T09:06:14.448100+0000","last_clean":"2024-06-03T09:06:14.448100+0000","last_became_active":"2024-06-03T08:36:30.404665+0000","last_became_peered":"2024-06-03T08:36:30.404665+0000","last_unstale":"2024-06-03T09:06:14.448100+0000","last_undegraded":"2024-06-03T09:06:14.448100+0000","last_fullsized":"2024-06-03T09:06:14.448100+0000","mapping_epoch":23,"log_start":"0'0","ondisk_log_start":"0'0","created":23,"last_epoch_clean":24,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-06-03T08:36:28.444764+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-06-03T08:36:28.444764+0000","last_clean_scrub_stamp":"2024-06-03T08:36:28.444764+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-06-04T10:37:37.746074+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,0],"acting":[4,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"17.15","version":"0'0","reported_seq":934,"reported_epoch":1009,"state":"active+clean","last_fresh":"2024-06-03T09:06:13.672940+0000","last_change":"2024-06-03T08:38:20.999903+0000","last_active":"2024-06-03T09:06:13.672940+0000","last_peered":"2024-06-03T09:06:13.672940+0000","last_clean":"2024-06-03T09:06:13.672940+0000","last_became_active":"2024-06-03T08:38:20.979523+0000","last_became_peered":"2024-06-03T08:38:20.979523+0000","last_unstale":"2024-06-03T09:06:13.672940+0000","last_undegraded":"2024-06-03T09:06:13.672940+0000","last_fullsized":"2024-06-03T09:06:13.672940+0000","mapping_epoch":28,"log_start":"0'0","ondisk_log_start":"0'0","created":28,"last_epoch_clean":47,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-06-03T08:37:55.614497+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-06-03T08:37:55.614497+0000","last_clean_scrub_stamp":"2024-06-03T08:37:55.614497+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-06-04T20:29:58.314289+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,1],"acting":[3,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"2.5","version":"0'0","reported_seq":942,"reported_epoch":1010,"state":"active+clean","last_fresh":"2024-06-03T09:06:18.382767+0000","last_change":"2024-06-03T08:36:30.024235+0000","last_active":"2024-06-03T09:06:18.382767+0000","last_peered":"2024-06-03T09:06:18.382767+0000","last_clean":"2024-06-03T09:06:18.382767+0000","last_became_active":"2024-06-03T08:36:30.019040+0000","last_became_peered":"2024-06-03T08:36:30.019040+0000","last_unstale":"2024-06-03T09:06:18.382767+0000","last_undegraded":"2024-06-03T09:06:18.382767+0000","last_fullsized":"2024-06-03T09:06:18.382767+0000","mapping_epoch":23,"log_start":"0'0","ondisk_log_start":"0'0","created":23,"last_epoch_clean":24,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-06-03T08:36:28.444764+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-06-03T08:36:28.444764+0000","last_clean_scrub_stamp":"2024-06-03T08:36:28.444764+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-06-04T15:46:47.109502+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,4],"acting":[3,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"17.16","version":"0'0","reported_seq":972,"reported_epoch":1009,"state":"active+undersized","last_fresh":"2024-06-03T09:06:14.441665+0000","last_change":"2024-06-03T09:06:03.478615+0000","last_active":"2024-06-03T09:06:14.441665+0000","last_peered":"2024-06-03T09:06:14.441665+0000","last_clean":"2024-06-03T09:04:43.328415+0000","last_became_active":"2024-06-03T09:06:03.478615+0000","last_became_peered":"2024-06-03T09:06:03.478615+0000","last_unstale":"2024-06-03T09:06:14.441665+0000","last_undegraded":"2024-06-03T09:06:14.441665+0000","last_fullsized":"2024-06-03T09:06:03.402504+0000","mapping_epoch":999,"log_start":"0'0","ondisk_log_start":"0'0","created":28,"last_epoch_clean":255,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-06-03T08:37:55.614497+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-06-03T08:37:55.614497+0000","last_clean_scrub_stamp":"2024-06-03T08:37:55.614497+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-06-04T08:47:08.079233+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2],"acting":[2],"avail_no_missing":["2"],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"2.4","version":"0'0","reported_seq":952,"reported_epoch":1010,"state":"active+undersized","last_fresh":"2024-06-03T09:06:18.331485+0000","last_change":"2024-06-03T09:06:03.387171+0000","last_active":"2024-06-03T09:06:18.331485+0000","last_peered":"2024-06-03T09:06:18.331485+0000","last_clean":"2024-06-03T09:04:43.202645+0000","last_became_active":"2024-06-03T09:06:03.387171+0000","last_became_peered":"2024-06-03T09:06:03.387171+0000","last_unstale":"2024-06-03T09:06:18.331485+0000","last_undegraded":"2024-06-03T09:06:18.331485+0000","last_fullsized":"2024-06-03T09:06:03.283787+0000","mapping_epoch":999,"log_start":"0'0","ondisk_log_start":"0'0","created":23,"last_epoch_clean":255,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-06-03T08:36:28.444764+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-06-03T08:36:28.444764+0000","last_clean_scrub_stamp":"2024-06-03T08:36:28.444764+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-06-04T20:17:43.485023+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5],"acting":[5],"avail_no_missing":["5"],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"17.17","version":"0'0","reported_seq":954,"reported_epoch":1009,"state":"active+clean","last_fresh":"2024-06-03T09:06:14.447046+0000","last_change":"2024-06-03T08:38:38.483275+0000","last_active":"2024-06-03T09:06:14.447046+0000","last_peered":"2024-06-03T09:06:14.447046+0000","last_clean":"2024-06-03T09:06:14.447046+0000","last_became_active":"2024-06-03T08:38:38.467916+0000","last_became_peered":"2024-06-03T08:38:38.467916+0000","last_unstale":"2024-06-03T09:06:14.447046+0000","last_undegraded":"2024-06-03T09:06:14.447046+0000","last_fullsized":"2024-06-03T09:06:14.447046+0000","mapping_epoch":28,"log_start":"0'0","ondisk_log_start":"0'0","created":28,"last_epoch_clean":61,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-06-03T08:37:55.614497+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-06-03T08:37:55.614497+0000","last_clean_scrub_stamp":"2024-06-03T08:37:55.614497+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-06-04T10:02:23.036420+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,7],"acting":[2,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"2.2","version":"0'0","reported_seq":963,"reported_epoch":1009,"state":"active+undersized","last_fresh":"2024-06-03T09:06:13.657892+0000","last_change":"2024-06-03T09:06:03.410020+0000","last_active":"2024-06-03T09:06:13.657892+0000","last_peered":"2024-06-03T09:06:13.657892+0000","last_clean":"2024-06-03T09:05:37.145889+0000","last_became_active":"2024-06-03T09:06:03.410020+0000","last_became_peered":"2024-06-03T09:06:03.410020+0000","last_unstale":"2024-06-03T09:06:13.657892+0000","last_undegraded":"2024-06-03T09:06:13.657892+0000","last_fullsized":"2024-06-03T09:06:03.346864+0000","mapping_epoch":999,"log_start":"0'0","ondisk_log_start":"0'0","created":23,"last_epoch_clean":255,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-06-03T08:36:28.444764+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-06-03T08:36:28.444764+0000","last_clean_scrub_stamp":"2024-06-03T08:36:28.444764+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-06-04T09:57:12.423360+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3],"acting":[3],"avail_no_missing":["3"],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"17.11","version":"0'0","reported_seq":944,"reported_epoch":1009,"state":"active+clean","last_fresh":"2024-06-03T09:06:14.426670+0000","last_change":"2024-06-03T08:38:06.502302+0000","last_active":"2024-06-03T09:06:14.426670+0000","last_peered":"2024-06-03T09:06:14.426670+0000","last_clean":"2024-06-03T09:06:14.426670+0000","last_became_active":"2024-06-03T08:38:06.490204+0000","last_became_peered":"2024-06-03T08:38:06.490204+0000","last_unstale":"2024-06-03T09:06:14.426670+0000","last_undegraded":"2024-06-03T09:06:14.426670+0000","last_fullsized":"2024-06-03T09:06:14.426670+0000","mapping_epoch":28,"log_start":"0'0","ondisk_log_start":"0'0","created":28,"last_epoch_clean":32,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-06-03T08:37:55.614497+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-06-03T08:37:55.614497+0000","last_clean_scrub_stamp":"2024-06-03T08:37:55.614497+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-06-04T10:47:05.988958+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,2],"acting":[4,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"2.1","version":"0'0","reported_seq":952,"reported_epoch":1009,"state":"active+clean","last_fresh":"2024-06-03T09:06:13.929725+0000","last_change":"2024-06-03T08:36:30.883360+0000","last_active":"2024-06-03T09:06:13.929725+0000","last_peered":"2024-06-03T09:06:13.929725+0000","last_clean":"2024-06-03T09:06:13.929725+0000","last_became_active":"2024-06-03T08:36:30.855073+0000","last_became_peered":"2024-06-03T08:36:30.855073+0000","last_unstale":"2024-06-03T09:06:13.929725+0000","last_undegraded":"2024-06-03T09:06:13.929725+0000","last_fullsized":"2024-06-03T09:06:13.929725+0000","mapping_epoch":23,"log_start":"0'0","ondisk_log_start":"0'0","created":23,"last_epoch_clean":25,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-06-03T08:36:28.444764+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-06-03T08:36:28.444764+0000","last_clean_scrub_stamp":"2024-06-03T08:36:28.444764+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-06-04T18:12:22.363473+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,5],"acting":[7,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"17.12","version":"0'0","reported_seq":945,"reported_epoch":1009,"state":"active+clean","last_fresh":"2024-06-03T09:06:13.935141+0000","last_change":"2024-06-03T08:38:01.751562+0000","last_active":"2024-06-03T09:06:13.935141+0000","last_peered":"2024-06-03T09:06:13.935141+0000","last_clean":"2024-06-03T09:06:13.935141+0000","last_became_active":"2024-06-03T08:38:01.745027+0000","last_became_peered":"2024-06-03T08:38:01.745027+0000","last_unstale":"2024-06-03T09:06:13.935141+0000","last_undegraded":"2024-06-03T09:06:13.935141+0000","last_fullsized":"2024-06-03T09:06:13.935141+0000","mapping_epoch":28,"log_start":"0'0","ondisk_log_start":"0'0","created":28,"last_epoch_clean":32,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-06-03T08:37:55.614497+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-06-03T08:37:55.614497+0000","last_clean_scrub_stamp":"2024-06-03T08:37:55.614497+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-06-04T17:54:43.785326+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,3],"acting":[7,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"2.0","version":"0'0","reported_seq":943,"reported_epoch":1009,"state":"active+clean","last_fresh":"2024-06-03T09:06:13.648112+0000","last_change":"2024-06-03T08:36:29.984096+0000","last_active":"2024-06-03T09:06:13.648112+0000","last_peered":"2024-06-03T09:06:13.648112+0000","last_clean":"2024-06-03T09:06:13.648112+0000","last_became_active":"2024-06-03T08:36:29.957875+0000","last_became_peered":"2024-06-03T08:36:29.957875+0000","last_unstale":"2024-06-03T09:06:13.648112+0000","last_undegraded":"2024-06-03T09:06:13.648112+0000","last_fullsized":"2024-06-03T09:06:13.648112+0000","mapping_epoch":23,"log_start":"0'0","ondisk_log_start":"0'0","created":23,"last_epoch_clean":24,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-06-03T08:36:28.444764+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-06-03T08:36:28.444764+0000","last_clean_scrub_stamp":"2024-06-03T08:36:28.444764+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-06-04T08:41:33.679479+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,5],"acting":[3,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"17.13","version":"0'0","reported_seq":943,"reported_epoch":1009,"state":"active+clean","last_fresh":"2024-06-03T09:06:14.438399+0000","last_change":"2024-06-03T08:38:26.838316+0000","last_active":"2024-06-03T09:06:14.438399+0000","last_peered":"2024-06-03T09:06:14.438399+0000","last_clean":"2024-06-03T09:06:14.438399+0000","last_became_active":"2024-06-03T08:38:26.833214+0000","last_became_peered":"2024-06-03T08:38:26.833214+0000","last_unstale":"2024-06-03T09:06:14.438399+0000","last_undegraded":"2024-06-03T09:06:14.438399+0000","last_fullsized":"2024-06-03T09:06:14.438399+0000","mapping_epoch":28,"log_start":"0'0","ondisk_log_start":"0'0","created":28,"last_epoch_clean":52,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-06-03T08:37:55.614497+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-06-03T08:37:55.614497+0000","last_clean_scrub_stamp":"2024-06-03T08:37:55.614497+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-06-04T12:58:51.617998+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,0],"acting":[4,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"2.3","version":"0'0","reported_seq":943,"reported_epoch":1010,"state":"active+clean","last_fresh":"2024-06-03T09:06:18.362614+0000","last_change":"2024-06-03T08:36:30.351670+0000","last_active":"2024-06-03T09:06:18.362614+0000","last_peered":"2024-06-03T09:06:18.362614+0000","last_clean":"2024-06-03T09:06:18.362614+0000","last_became_active":"2024-06-03T08:36:30.343560+0000","last_became_peered":"2024-06-03T08:36:30.343560+0000","last_unstale":"2024-06-03T09:06:18.362614+0000","last_undegraded":"2024-06-03T09:06:18.362614+0000","last_fullsized":"2024-06-03T09:06:18.362614+0000","mapping_epoch":23,"log_start":"0'0","ondisk_log_start":"0'0","created":23,"last_epoch_clean":24,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-06-03T08:36:28.444764+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-06-03T08:36:28.444764+0000","last_clean_scrub_stamp":"2024-06-03T08:36:28.444764+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-06-04T12:02:24.765135+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,2],"acting":[3,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.0","version":"19'76","reported_seq":1051,"reported_epoch":1009,"state":"active+clean","last_fresh":"2024-06-03T09:06:13.703685+0000","last_change":"2024-06-03T08:36:26.702702+0000","last_active":"2024-06-03T09:06:13.703685+0000","last_peered":"2024-06-03T09:06:13.703685+0000","last_clean":"2024-06-03T09:06:13.703685+0000","last_became_active":"2024-06-03T08:36:26.128378+0000","last_became_peered":"2024-06-03T08:36:26.128378+0000","last_unstale":"2024-06-03T09:06:13.703685+0000","last_undegraded":"2024-06-03T09:06:13.703685+0000","last_fullsized":"2024-06-03T09:06:13.703685+0000","mapping_epoch":20,"log_start":"0'0","ondisk_log_start":"0'0","created":16,"last_epoch_clean":22,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-06-03T08:36:15.542635+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-06-03T08:36:15.542635+0000","last_clean_scrub_stamp":"2024-06-03T08:36:15.542635+0000","objects_scrubbed":0,"log_size":76,"log_dups_size":0,"ondisk_log_size":76,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-06-04T12:41:53.731360+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":459280,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":96,"num_read_kb":82,"num_write":113,"num_write_kb":1372,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2,"num_bytes_recovered":397840,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,5],"acting":[1,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"17.10","version":"0'0","reported_seq":944,"reported_epoch":1009,"state":"active+clean","last_fresh":"2024-06-03T09:06:14.422035+0000","last_change":"2024-06-03T08:38:15.399189+0000","last_active":"2024-06-03T09:06:14.422035+0000","last_peered":"2024-06-03T09:06:14.422035+0000","last_clean":"2024-06-03T09:06:14.422035+0000","last_became_active":"2024-06-03T08:38:15.337723+0000","last_became_peered":"2024-06-03T08:38:15.337723+0000","last_unstale":"2024-06-03T09:06:14.422035+0000","last_undegraded":"2024-06-03T09:06:14.422035+0000","last_fullsized":"2024-06-03T09:06:14.422035+0000","mapping_epoch":28,"log_start":"0'0","ondisk_log_start":"0'0","created":28,"last_epoch_clean":42,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-06-03T08:37:55.614497+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-06-03T08:37:55.614497+0000","last_clean_scrub_stamp":"2024-06-03T08:37:55.614497+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-06-04T09:29:46.427317+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,3],"acting":[4,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"17.1a","version":"0'0","reported_seq":960,"reported_epoch":1009,"state":"active+undersized","last_fresh":"2024-06-03T09:06:14.431224+0000","last_change":"2024-06-03T09:06:03.390734+0000","last_active":"2024-06-03T09:06:14.431224+0000","last_peered":"2024-06-03T09:06:14.431224+0000","last_clean":"2024-06-03T09:05:20.322313+0000","last_became_active":"2024-06-03T09:06:03.390734+0000","last_became_peered":"2024-06-03T09:06:03.390734+0000","last_unstale":"2024-06-03T09:06:14.431224+0000","last_undegraded":"2024-06-03T09:06:14.431224+0000","last_fullsized":"2024-06-03T09:06:03.343197+0000","mapping_epoch":999,"log_start":"0'0","ondisk_log_start":"0'0","created":28,"last_epoch_clean":254,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-06-03T08:37:55.614497+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-06-03T08:37:55.614497+0000","last_clean_scrub_stamp":"2024-06-03T08:37:55.614497+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-06-04T15:56:47.628875+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4],"acting":[4],"avail_no_missing":["4"],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"17.1b","version":"0'0","reported_seq":935,"reported_epoch":1009,"state":"active+clean","last_fresh":"2024-06-03T09:06:13.663279+0000","last_change":"2024-06-03T08:38:03.115748+0000","last_active":"2024-06-03T09:06:13.663279+0000","last_peered":"2024-06-03T09:06:13.663279+0000","last_clean":"2024-06-03T09:06:13.663279+0000","last_became_active":"2024-06-03T08:38:03.110533+0000","last_became_peered":"2024-06-03T08:38:03.110533+0000","last_unstale":"2024-06-03T09:06:13.663279+0000","last_undegraded":"2024-06-03T09:06:13.663279+0000","last_fullsized":"2024-06-03T09:06:13.663279+0000","mapping_epoch":28,"log_start":"0'0","ondisk_log_start":"0'0","created":28,"last_epoch_clean":32,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-06-03T08:37:55.614497+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-06-03T08:37:55.614497+0000","last_clean_scrub_stamp":"2024-06-03T08:37:55.614497+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-06-04T12:36:47.471609+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,7],"acting":[3,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"17.18","version":"0'0","reported_seq":944,"reported_epoch":1009,"state":"active+clean","last_fresh":"2024-06-03T09:06:13.924799+0000","last_change":"2024-06-03T08:38:04.632864+0000","last_active":"2024-06-03T09:06:13.924799+0000","last_peered":"2024-06-03T09:06:13.924799+0000","last_clean":"2024-06-03T09:06:13.924799+0000","last_became_active":"2024-06-03T08:38:04.627856+0000","last_became_peered":"2024-06-03T08:38:04.627856+0000","last_unstale":"2024-06-03T09:06:13.924799+0000","last_undegraded":"2024-06-03T09:06:13.924799+0000","last_fullsized":"2024-06-03T09:06:13.924799+0000","mapping_epoch":28,"log_start":"0'0","ondisk_log_start":"0'0","created":28,"last_epoch_clean":33,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-06-03T08:37:55.614497+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-06-03T08:37:55.614497+0000","last_clean_scrub_stamp":"2024-06-03T08:37:55.614497+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-06-04T19:55:51.181598+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,4],"acting":[7,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"17.19","version":"0'0","reported_seq":934,"reported_epoch":1009,"state":"active+clean","last_fresh":"2024-06-03T09:06:13.652943+0000","last_change":"2024-06-03T08:38:01.839491+0000","last_active":"2024-06-03T09:06:13.652943+0000","last_peered":"2024-06-03T09:06:13.652943+0000","last_clean":"2024-06-03T09:06:13.652943+0000","last_became_active":"2024-06-03T08:38:01.829539+0000","last_became_peered":"2024-06-03T08:38:01.829539+0000","last_unstale":"2024-06-03T09:06:13.652943+0000","last_undegraded":"2024-06-03T09:06:13.652943+0000","last_fullsized":"2024-06-03T09:06:13.652943+0000","mapping_epoch":28,"log_start":"0'0","ondisk_log_start":"0'0","created":28,"last_epoch_clean":30,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-06-03T08:37:55.614497+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-06-03T08:37:55.614497+0000","last_clean_scrub_stamp":"2024-06-03T08:37:55.614497+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-06-04T09:10:43.860229+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,2],"acting":[3,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"17.1e","version":"0'0","reported_seq":148,"reported_epoch":1009,"state":"active+undersized","last_fresh":"2024-06-03T09:06:14.445917+0000","last_change":"2024-06-03T09:06:03.406181+0000","last_active":"2024-06-03T09:06:14.445917+0000","last_peered":"2024-06-03T09:06:14.445917+0000","last_clean":"2024-06-03T08:37:55.614497+0000","last_became_active":"2024-06-03T09:06:03.406181+0000","last_became_peered":"2024-06-03T09:06:03.406181+0000","last_unstale":"2024-06-03T09:06:14.445917+0000","last_undegraded":"2024-06-03T09:06:14.445917+0000","last_fullsized":"2024-06-03T09:06:03.363739+0000","mapping_epoch":999,"log_start":"0'0","ondisk_log_start":"0'0","created":28,"last_epoch_clean":255,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-06-03T08:37:55.614497+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-06-03T08:37:55.614497+0000","last_clean_scrub_stamp":"2024-06-03T08:37:55.614497+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-06-04T11:45:24.486600+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4],"acting":[4],"avail_no_missing":["4"],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"17.1f","version":"0'0","reported_seq":941,"reported_epoch":1009,"state":"active+clean","last_fresh":"2024-06-03T09:06:14.450053+0000","last_change":"2024-06-03T08:38:04.383538+0000","last_active":"2024-06-03T09:06:14.450053+0000","last_peered":"2024-06-03T09:06:14.450053+0000","last_clean":"2024-06-03T09:06:14.450053+0000","last_became_active":"2024-06-03T08:38:04.367278+0000","last_became_peered":"2024-06-03T08:38:04.367278+0000","last_unstale":"2024-06-03T09:06:14.450053+0000","last_undegraded":"2024-06-03T09:06:14.450053+0000","last_fullsized":"2024-06-03T09:06:14.450053+0000","mapping_epoch":28,"log_start":"0'0","ondisk_log_start":"0'0","created":28,"last_epoch_clean":32,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-06-03T08:37:55.614497+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-06-03T08:37:55.614497+0000","last_clean_scrub_stamp":"2024-06-03T08:37:55.614497+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-06-04T10:39:26.552282+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,2],"acting":[0,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"17.1c","version":"0'0","reported_seq":944,"reported_epoch":1010,"state":"active+undersized","last_fresh":"2024-06-03T09:06:18.327492+0000","last_change":"2024-06-03T09:06:03.382210+0000","last_active":"2024-06-03T09:06:18.327492+0000","last_peered":"2024-06-03T09:06:18.327492+0000","last_clean":"2024-06-03T09:04:43.199660+0000","last_became_active":"2024-06-03T09:06:03.382210+0000","last_became_peered":"2024-06-03T09:06:03.382210+0000","last_unstale":"2024-06-03T09:06:18.327492+0000","last_undegraded":"2024-06-03T09:06:18.327492+0000","last_fullsized":"2024-06-03T09:06:03.277874+0000","mapping_epoch":999,"log_start":"0'0","ondisk_log_start":"0'0","created":28,"last_epoch_clean":255,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-06-03T08:37:55.614497+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-06-03T08:37:55.614497+0000","last_clean_scrub_stamp":"2024-06-03T08:37:55.614497+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-06-04T12:18:50.013884+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5],"acting":[5],"avail_no_missing":["5"],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"17.1d","version":"0'0","reported_seq":923,"reported_epoch":1009,"state":"active+clean","last_fresh":"2024-06-03T09:06:13.715906+0000","last_change":"2024-06-03T08:38:06.794630+0000","last_active":"2024-06-03T09:06:13.715906+0000","last_peered":"2024-06-03T09:06:13.715906+0000","last_clean":"2024-06-03T09:06:13.715906+0000","last_became_active":"2024-06-03T08:38:06.592317+0000","last_became_peered":"2024-06-03T08:38:06.592317+0000","last_unstale":"2024-06-03T09:06:13.715906+0000","last_undegraded":"2024-06-03T09:06:13.715906+0000","last_fullsized":"2024-06-03T09:06:13.715906+0000","mapping_epoch":28,"log_start":"0'0","ondisk_log_start":"0'0","created":28,"last_epoch_clean":33,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-06-03T08:37:55.614497+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-06-03T08:37:55.614497+0000","last_clean_scrub_stamp":"2024-06-03T08:37:55.614497+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-06-04T12:33:30.785369+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,7],"acting":[1,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"17.2","version":"0'0","reported_seq":925,"reported_epoch":1009,"state":"active+clean","last_fresh":"2024-06-03T09:06:13.708688+0000","last_change":"2024-06-03T08:38:14.981410+0000","last_active":"2024-06-03T09:06:13.708688+0000","last_peered":"2024-06-03T09:06:13.708688+0000","last_clean":"2024-06-03T09:06:13.708688+0000","last_became_active":"2024-06-03T08:38:14.973825+0000","last_became_peered":"2024-06-03T08:38:14.973825+0000","last_unstale":"2024-06-03T09:06:13.708688+0000","last_undegraded":"2024-06-03T09:06:13.708688+0000","last_fullsized":"2024-06-03T09:06:13.708688+0000","mapping_epoch":28,"log_start":"0'0","ondisk_log_start":"0'0","created":28,"last_epoch_clean":41,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-06-03T08:37:55.614497+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-06-03T08:37:55.614497+0000","last_clean_scrub_stamp":"2024-06-03T08:37:55.614497+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-06-04T17:01:05.518613+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,4],"acting":[1,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"17.3","version":"0'0","reported_seq":943,"reported_epoch":1009,"state":"active+clean","last_fresh":"2024-06-03T09:06:14.437636+0000","last_change":"2024-06-03T08:38:14.619745+0000","last_active":"2024-06-03T09:06:14.437636+0000","last_peered":"2024-06-03T09:06:14.437636+0000","last_clean":"2024-06-03T09:06:14.437636+0000","last_became_active":"2024-06-03T08:38:14.510665+0000","last_became_peered":"2024-06-03T08:38:14.510665+0000","last_unstale":"2024-06-03T09:06:14.437636+0000","last_undegraded":"2024-06-03T09:06:14.437636+0000","last_fullsized":"2024-06-03T09:06:14.437636+0000","mapping_epoch":28,"log_start":"0'0","ondisk_log_start":"0'0","created":28,"last_epoch_clean":42,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-06-03T08:37:55.614497+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-06-03T08:37:55.614497+0000","last_clean_scrub_stamp":"2024-06-03T08:37:55.614497+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-06-04T12:03:47.392064+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,7],"acting":[0,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"17.0","version":"0'0","reported_seq":944,"reported_epoch":1009,"state":"active+clean","last_fresh":"2024-06-03T09:06:14.427123+0000","last_change":"2024-06-03T08:38:04.237118+0000","last_active":"2024-06-03T09:06:14.427123+0000","last_peered":"2024-06-03T09:06:14.427123+0000","last_clean":"2024-06-03T09:06:14.427123+0000","last_became_active":"2024-06-03T08:38:04.231954+0000","last_became_peered":"2024-06-03T08:38:04.231954+0000","last_unstale":"2024-06-03T09:06:14.427123+0000","last_undegraded":"2024-06-03T09:06:14.427123+0000","last_fullsized":"2024-06-03T09:06:14.427123+0000","mapping_epoch":28,"log_start":"0'0","ondisk_log_start":"0'0","created":28,"last_epoch_clean":32,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-06-03T08:37:55.614497+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-06-03T08:37:55.614497+0000","last_clean_scrub_stamp":"2024-06-03T08:37:55.614497+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-06-04T09:56:10.741637+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,2],"acting":[0,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"17.1","version":"65'7","reported_seq":959,"reported_epoch":1009,"state":"active+undersized+degraded","last_fresh":"2024-06-03T09:06:13.709209+0000","last_change":"2024-06-03T09:06:03.370660+0000","last_active":"2024-06-03T09:06:13.709209+0000","last_peered":"2024-06-03T09:06:13.709209+0000","last_clean":"2024-06-03T09:05:03.733864+0000","last_became_active":"2024-06-03T09:06:03.370660+0000","last_became_peered":"2024-06-03T09:06:03.370660+0000","last_unstale":"2024-06-03T09:06:13.709209+0000","last_undegraded":"2024-06-03T09:06:03.266076+0000","last_fullsized":"2024-06-03T09:06:03.264525+0000","mapping_epoch":999,"log_start":"0'0","ondisk_log_start":"0'0","created":28,"last_epoch_clean":255,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-06-03T08:37:55.614497+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-06-03T08:37:55.614497+0000","last_clean_scrub_stamp":"2024-06-03T08:37:55.614497+0000","objects_scrubbed":0,"log_size":7,"log_dups_size":0,"ondisk_log_size":7,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-06-04T11:55:18.194283+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":5,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":1,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":6,"num_read_kb":6,"num_write":7,"num_write_kb":7,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5],"acting":[5],"avail_no_missing":["5"],"object_location_counts":[{"shards":"5","objects":1}],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]}],"pool_stats":[{"poolid":2,"num_pg":8,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":14,"acting":14,"num_store_stats":7},{"poolid":1,"num_pg":1,"stat_sum":{"num_bytes":459280,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":96,"num_read_kb":82,"num_write":113,"num_write_kb":1372,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2,"num_bytes_recovered":397840,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":802816,"data_stored":795680,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":76,"ondisk_log_size":76,"up":2,"acting":2,"num_store_stats":2},{"poolid":17,"num_pg":32,"stat_sum":{"num_bytes":5,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":1,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":6,"num_read_kb":6,"num_write":7,"num_write_kb":7,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":8192,"data_stored":10,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":7,"ondisk_log_size":7,"up":52,"acting":52,"num_store_stats":8}],"osd_stats":[{"osd":6,"up_from":253,"seq":1086626726200,"num_pgs":32,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":54772,"kb_used_data":5608,"kb_used_omap":1,"kb_used_meta":49150,"kb_avail":94317068,"statfs":{"total":96636764160,"available":96580677632,"internally_reserved":0,"allocated":5742592,"data_stored":2767180,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1593,"internal_metadata":50330055},"hb_peers":[0,1,2,3,4,5,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":109,"apply_latency_ms":109,"commit_latency_ns":109000000,"apply_latency_ns":109000000},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Mon Jun 3 09:04:02 2024","interfaces":[{"interface":"back","average":{"1min":1582.4780000000001,"5min":332.72699999999998,"15min":124.113},"min":{"1min":4.4829999999999997,"5min":4.1959999999999997,"15min":4.1959999999999997},"max":{"1min":22766.882000000001,"5min":22766.882000000001,"15min":22766.882000000001},"last":21.917999999999999},{"interface":"front","average":{"1min":1579.5409999999999,"5min":334.94499999999999,"15min":125.84999999999999},"min":{"1min":4.8129999999999997,"5min":4.7539999999999996,"15min":4.4530000000000003},"max":{"1min":22766.289000000001,"5min":22766.289000000001,"15min":22766.289000000001},"last":25.408999999999999}]},{"osd":1,"last update":"Mon Jun 3 09:03:40 2024","interfaces":[{"interface":"back","average":{"1min":36.420999999999999,"5min":77.054000000000002,"15min":132.84800000000001},"min":{"1min":6.0380000000000003,"5min":5.8799999999999999,"15min":5.484},"max":{"1min":589.64099999999996,"5min":4138.1679999999997,"15min":9955.1299999999992},"last":26.84},{"interface":"front","average":{"1min":40.240000000000002,"5min":79.674999999999997,"15min":133.38999999999999},"min":{"1min":8.4589999999999996,"5min":8.4090000000000007,"15min":6.1390000000000002},"max":{"1min":588.60799999999995,"5min":4137.6530000000002,"15min":9827.5509999999995},"last":26.263000000000002}]},{"osd":2,"last update":"Mon Jun 3 09:03:40 2024","interfaces":[{"interface":"back","average":{"1min":12.613,"5min":24.600999999999999,"15min":21.088000000000001},"min":{"1min":6.9740000000000002,"5min":6.9740000000000002,"15min":6.5330000000000004},"max":{"1min":26.076000000000001,"5min":376.24900000000002,"15min":376.24900000000002},"last":24.899000000000001},{"interface":"front","average":{"1min":12.493,"5min":22.207999999999998,"15min":21.219000000000001},"min":{"1min":6.5810000000000004,"5min":6.5110000000000001,"15min":6.0750000000000002},"max":{"1min":29.690999999999999,"5min":271.51100000000002,"15min":434.77300000000002},"last":40.372}]},{"osd":3,"last update":"Mon Jun 3 09:03:40 2024","interfaces":[{"interface":"back","average":{"1min":14.093999999999999,"5min":24.143999999999998,"15min":29.376999999999999},"min":{"1min":10.196999999999999,"5min":6.2919999999999998,"15min":6.1719999999999997},"max":{"1min":24.832000000000001,"5min":197.84899999999999,"15min":1235.5799999999999},"last":24.353999999999999},{"interface":"front","average":{"1min":9.3260000000000005,"5min":18.463999999999999,"15min":27.631},"min":{"1min":6.125,"5min":6.125,"15min":6.125},"max":{"1min":28.166,"5min":200.173,"15min":1157.6780000000001},"last":23.454000000000001}]},{"osd":4,"last update":"Mon Jun 3 09:04:29 2024","interfaces":[{"interface":"back","average":{"1min":10.279,"5min":1253.809,"15min":450.79199999999997},"min":{"1min":7.2300000000000004,"5min":6.093,"15min":6.093},"max":{"1min":15.000999999999999,"5min":28093.096000000001,"15min":28093.096000000001},"last":101.45399999999999},{"interface":"front","average":{"1min":15.693,"5min":1279.759,"15min":463.87099999999998},"min":{"1min":10.551,"5min":7.7480000000000002,"15min":7.7480000000000002},"max":{"1min":27.898,"5min":28323.837,"15min":28323.837},"last":102.262}]},{"osd":5,"last update":"Mon Jun 3 09:04:10 2024","interfaces":[{"interface":"back","average":{"1min":13.728999999999999,"5min":24.670000000000002,"15min":23.815999999999999},"min":{"1min":10.401,"5min":7.516,"15min":7.516},"max":{"1min":25.138000000000002,"5min":320.85000000000002,"15min":445.041},"last":103.52500000000001},{"interface":"front","average":{"1min":13.847,"5min":29.283000000000001,"15min":23.393999999999998},"min":{"1min":7.944,"5min":6.266,"15min":6.266},"max":{"1min":18.817,"5min":530.56700000000001,"15min":530.56700000000001},"last":29.263999999999999}]},{"osd":7,"last update":"Mon Jun 3 09:03:40 2024","interfaces":[{"interface":"back","average":{"1min":20.568999999999999,"5min":30.286999999999999,"15min":27.977},"min":{"1min":13.874000000000001,"5min":10.58,"15min":9.2859999999999996},"max":{"1min":86.959000000000003,"5min":222.07599999999999,"15min":255.08799999999999},"last":37.618000000000002},{"interface":"front","average":{"1min":19.102,"5min":29.550000000000001,"15min":27.763000000000002},"min":{"1min":10.052,"5min":8.6709999999999994,"15min":8.5429999999999993},"max":{"1min":67.968999999999994,"5min":293.62099999999998,"15min":438.14600000000002},"last":111.985}]}]},{"osd":7,"up_from":20,"seq":85899346288,"num_pgs":9,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":58964,"kb_used_data":5704,"kb_used_omap":1,"kb_used_meta":53246,"kb_avail":94312876,"statfs":{"total":96636764160,"available":96576385024,"internally_reserved":0,"allocated":5840896,"data_stored":2740727,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1589,"internal_metadata":54524363},"hb_peers":[0,1,2,3,4,5],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Mon Jun 3 09:06:09 2024","interfaces":[{"interface":"back","average":{"1min":14.073,"5min":894.22400000000005,"15min":308.93400000000003},"min":{"1min":5.5709999999999997,"5min":4.218,"15min":4.0599999999999996},"max":{"1min":90.757999999999996,"5min":21729.001,"15min":21729.001},"last":6.1630000000000003},{"interface":"front","average":{"1min":9.2550000000000008,"5min":906.23500000000001,"15min":314.06},"min":{"1min":5.5789999999999997,"5min":4.8810000000000002,"15min":4.8810000000000002},"max":{"1min":25.937000000000001,"5min":21899.732,"15min":21899.732},"last":7.109}]},{"osd":1,"last update":"Mon Jun 3 09:06:09 2024","interfaces":[{"interface":"back","average":{"1min":15.182,"5min":69.183999999999997,"15min":219.173},"min":{"1min":7.3639999999999999,"5min":6.6200000000000001,"15min":6.1319999999999997},"max":{"1min":76.926000000000002,"5min":2795.1329999999998,"15min":18373.257000000001},"last":9.9030000000000005},{"interface":"front","average":{"1min":21.495999999999999,"5min":74.703999999999994,"15min":225.285},"min":{"1min":8.8399999999999999,"5min":8.8399999999999999,"15min":8.5129999999999999},"max":{"1min":94.977999999999994,"5min":2803.3670000000002,"15min":18374.601999999999},"last":12.782}]},{"osd":2,"last update":"Mon Jun 3 09:06:09 2024","interfaces":[{"interface":"back","average":{"1min":19.164000000000001,"5min":26.902000000000001,"15min":22.920999999999999},"min":{"1min":11.207000000000001,"5min":8.5030000000000001,"15min":8.5030000000000001},"max":{"1min":91.319999999999993,"5min":333.99200000000002,"15min":333.99200000000002},"last":14.696},{"interface":"front","average":{"1min":11.125999999999999,"5min":18.132000000000001,"15min":16.353000000000002},"min":{"1min":6.202,"5min":5.8369999999999997,"15min":5.5069999999999997},"max":{"1min":28.649000000000001,"5min":226.18100000000001,"15min":226.18100000000001},"last":8.0749999999999993}]},{"osd":3,"last update":"Mon Jun 3 09:06:09 2024","interfaces":[{"interface":"back","average":{"1min":22.291,"5min":26.289999999999999,"15min":31.074999999999999},"min":{"1min":11.714,"5min":6.8959999999999999,"15min":6.3760000000000003},"max":{"1min":83.102000000000004,"5min":316.64400000000001,"15min":2039.9059999999999},"last":13.901},{"interface":"front","average":{"1min":13.324999999999999,"5min":17.963000000000001,"15min":25.969999999999999},"min":{"1min":6.1200000000000001,"5min":5.2089999999999996,"15min":5.2089999999999996},"max":{"1min":49.908999999999999,"5min":233.542,"15min":2037.5550000000001},"last":9.3919999999999995}]},{"osd":4,"last update":"Mon Jun 3 09:06:29 2024","interfaces":[{"interface":"back","average":{"1min":17.675999999999998,"5min":1026.325,"15min":472.47199999999998},"min":{"1min":7.306,"5min":5.9089999999999998,"15min":5.9089999999999998},"max":{"1min":140.03299999999999,"5min":28969.369999999999,"15min":30073.917000000001},"last":8.8829999999999991},{"interface":"front","average":{"1min":17.890999999999998,"5min":1044.5139999999999,"15min":480.226},"min":{"1min":5.9279999999999999,"5min":5.9279999999999999,"15min":5.9279999999999999},"max":{"1min":141.179,"5min":29244.138999999999,"15min":30348.021000000001},"last":12.252000000000001}]},{"osd":5,"last update":"Mon Jun 3 09:06:02 2024","interfaces":[{"interface":"back","average":{"1min":24.372,"5min":26.856000000000002,"15min":23.571999999999999},"min":{"1min":12.278,"5min":9.2460000000000004,"15min":6.5650000000000004},"max":{"1min":183.64099999999999,"5min":317.572,"15min":317.572},"last":15.528},{"interface":"front","average":{"1min":21.259,"5min":23.364000000000001,"15min":21.193999999999999},"min":{"1min":7.8499999999999996,"5min":7.0999999999999996,"15min":7.008},"max":{"1min":184.53399999999999,"5min":306.60599999999999,"15min":306.60599999999999},"last":10.446999999999999}]},{"osd":6,"last update":"Mon Jun 3 09:04:31 2024","interfaces":[{"interface":"back","average":{"1min":44.061,"5min":26.821000000000002,"15min":91.176000000000002},"min":{"1min":9.0800000000000001,"5min":7.9390000000000001,"15min":7.9390000000000001},"max":{"1min":268.029,"5min":308.916,"15min":8734.4639999999999},"last":6.6769999999999996},{"interface":"front","average":{"1min":46.643000000000001,"5min":23.879999999999999,"15min":88.765000000000001},"min":{"1min":8.8119999999999994,"5min":6.9240000000000004,"15min":6.742},"max":{"1min":305.64600000000002,"5min":305.64600000000002,"15min":8735.5319999999992},"last":12.327}]}]},{"osd":5,"up_from":20,"seq":85899346263,"num_pgs":9,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":59360,"kb_used_data":6100,"kb_used_omap":1,"kb_used_meta":53246,"kb_avail":94312480,"statfs":{"total":96636764160,"available":96575979520,"internally_reserved":0,"allocated":6246400,"data_stored":3138572,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1594,"internal_metadata":54524358},"hb_peers":[0,1,2,3,4,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Mon Jun 3 09:06:06 2024","interfaces":[{"interface":"back","average":{"1min":7.468,"5min":1703.037,"15min":580.51900000000001},"min":{"1min":4.6509999999999998,"5min":4.6509999999999998,"15min":4.4020000000000001},"max":{"1min":14.711,"5min":26433.764999999999,"15min":26433.764999999999},"last":5.8639999999999999},{"interface":"front","average":{"1min":9.9429999999999996,"5min":1725.27,"15min":587.87800000000004},"min":{"1min":5.5999999999999996,"5min":5.1319999999999997,"15min":4.8170000000000002},"max":{"1min":13.590999999999999,"5min":26704.698,"15min":26704.698},"last":12.747999999999999}]},{"osd":1,"last update":"Mon Jun 3 09:06:00 2024","interfaces":[{"interface":"back","average":{"1min":10.862,"5min":67.731999999999999,"15min":217.364},"min":{"1min":7.1769999999999996,"5min":6.0890000000000004,"15min":5.4930000000000003},"max":{"1min":16.998999999999999,"5min":3923.4090000000001,"15min":18811.696},"last":14.052},{"interface":"front","average":{"1min":10.417,"5min":65.141999999999996,"15min":220.75700000000001},"min":{"1min":5.3209999999999997,"5min":5.3209999999999997,"15min":5.0209999999999999},"max":{"1min":15.821999999999999,"5min":4019.6129999999998,"15min":18968.365000000002},"last":8.7219999999999995}]},{"osd":2,"last update":"Mon Jun 3 09:06:00 2024","interfaces":[{"interface":"back","average":{"1min":11.366,"5min":24.18,"15min":21.137},"min":{"1min":5.6369999999999996,"5min":5.6369999999999996,"15min":5.0510000000000002},"max":{"1min":15.529999999999999,"5min":332.48200000000003,"15min":332.48200000000003},"last":12.167},{"interface":"front","average":{"1min":10.089,"5min":23.117000000000001,"15min":24.050000000000001},"min":{"1min":6.6769999999999996,"5min":6.6769999999999996,"15min":5.8739999999999997},"max":{"1min":15.821,"5min":353.03199999999998,"15min":353.03199999999998},"last":7.2619999999999996}]},{"osd":3,"last update":"Mon Jun 3 09:06:00 2024","interfaces":[{"interface":"back","average":{"1min":13.472,"5min":24.283999999999999,"15min":31.666},"min":{"1min":10.111000000000001,"5min":7.4720000000000004,"15min":7.4210000000000003},"max":{"1min":16.271999999999998,"5min":353.57499999999999,"15min":2855.8519999999999},"last":8.1140000000000008},{"interface":"front","average":{"1min":9.5899999999999999,"5min":15.875,"15min":25.834},"min":{"1min":6.6609999999999996,"5min":5.7729999999999997,"15min":5.1909999999999998},"max":{"1min":15.112,"5min":203.804,"15min":2869.7860000000001},"last":9.3699999999999992}]},{"osd":4,"last update":"Mon Jun 3 09:06:29 2024","interfaces":[{"interface":"back","average":{"1min":11.067,"5min":780.21000000000004,"15min":377.28399999999999},"min":{"1min":5.4779999999999998,"5min":5.4779999999999998,"15min":5.4779999999999998},"max":{"1min":14.654999999999999,"5min":23315.438999999998,"15min":28619.825000000001},"last":6.7199999999999998},{"interface":"front","average":{"1min":14.349,"5min":782.97299999999996,"15min":377.93400000000003},"min":{"1min":7.492,"5min":6.5030000000000001,"15min":6.5030000000000001},"max":{"1min":17.707000000000001,"5min":23317.502,"15min":28621.294000000002},"last":11.621}]},{"osd":6,"last update":"Mon Jun 3 09:03:57 2024","interfaces":[{"interface":"back","average":{"1min":23.536000000000001,"5min":36.457000000000001,"15min":83.799999999999997},"min":{"1min":9.4800000000000004,"5min":9.4800000000000004,"15min":8.0630000000000006},"max":{"1min":89.442999999999998,"5min":508.77999999999997,"15min":9193.5169999999998},"last":157.79300000000001},{"interface":"front","average":{"1min":18.891999999999999,"5min":22.100999999999999,"15min":75.489999999999995},"min":{"1min":5.923,"5min":5.923,"15min":5.923},"max":{"1min":78.424999999999997,"5min":464.62,"15min":9191.4429999999993},"last":148.09}]},{"osd":7,"last update":"Mon Jun 3 09:06:00 2024","interfaces":[{"interface":"back","average":{"1min":13.257,"5min":20.917999999999999,"15min":21.088999999999999},"min":{"1min":10.475,"5min":6.7629999999999999,"15min":6.7629999999999999},"max":{"1min":17.245000000000001,"5min":211.22999999999999,"15min":211.22999999999999},"last":11.061999999999999},{"interface":"front","average":{"1min":12.872999999999999,"5min":20.587,"15min":24.021000000000001},"min":{"1min":8.5760000000000005,"5min":7.7709999999999999,"15min":6.8570000000000002},"max":{"1min":18.359999999999999,"5min":223.191,"15min":286.69499999999999},"last":10.021000000000001}]}]},{"osd":4,"up_from":20,"seq":85899346271,"num_pgs":12,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":63060,"kb_used_data":5704,"kb_used_omap":1,"kb_used_meta":57342,"kb_avail":94308780,"statfs":{"total":96636764160,"available":96572190720,"internally_reserved":0,"allocated":5840896,"data_stored":2740727,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1594,"internal_metadata":58718662},"hb_peers":[0,1,2,3,5,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Mon Jun 3 09:06:35 2024","interfaces":[{"interface":"back","average":{"1min":10.417,"5min":1093.547,"15min":381.64499999999998},"min":{"1min":4.6900000000000004,"5min":4.5279999999999996,"15min":4.3220000000000001},"max":{"1min":52.241,"5min":21726.743999999999,"15min":21726.743999999999},"last":7.0030000000000001},{"interface":"front","average":{"1min":13.175000000000001,"5min":1102.4200000000001,"15min":387.161},"min":{"1min":6.1950000000000003,"5min":5.8380000000000001,"15min":5.4909999999999997},"max":{"1min":53.148000000000003,"5min":21727.993999999999,"15min":21727.993999999999},"last":12.923999999999999}]},{"osd":1,"last update":"Mon Jun 3 09:06:35 2024","interfaces":[{"interface":"back","average":{"1min":9.9909999999999997,"5min":53.250999999999998,"15min":340.02999999999997},"min":{"1min":4.5670000000000002,"5min":4.1970000000000001,"15min":4.1970000000000001},"max":{"1min":47.073,"5min":3346.4389999999999,"15min":22256.098000000002},"last":15.198},{"interface":"front","average":{"1min":12.189,"5min":53.375999999999998,"15min":339.68799999999999},"min":{"1min":6.1539999999999999,"5min":5.0590000000000002,"15min":4.2640000000000002},"max":{"1min":49.558,"5min":3347.181,"15min":22247.317999999999},"last":7.681}]},{"osd":2,"last update":"Mon Jun 3 09:06:35 2024","interfaces":[{"interface":"back","average":{"1min":15.178000000000001,"5min":17.202000000000002,"15min":20.417999999999999},"min":{"1min":6.3010000000000002,"5min":6.3010000000000002,"15min":6.3010000000000002},"max":{"1min":85.340000000000003,"5min":337.60399999999998,"15min":393.58600000000001},"last":12.378},{"interface":"front","average":{"1min":15.423,"5min":14.432,"15min":22.754000000000001},"min":{"1min":6.4889999999999999,"5min":6.0629999999999997,"15min":6.0629999999999997},"max":{"1min":82.120000000000005,"5min":127.913,"15min":394.471},"last":10.843999999999999}]},{"osd":3,"last update":"Mon Jun 3 09:06:35 2024","interfaces":[{"interface":"back","average":{"1min":17.651,"5min":15.5,"15min":18.079000000000001},"min":{"1min":10.359999999999999,"5min":5.8719999999999999,"15min":5.4000000000000004},"max":{"1min":84.555999999999997,"5min":127.43600000000001,"15min":322.63},"last":14.564},{"interface":"front","average":{"1min":16.413,"5min":17.356999999999999,"15min":22.920000000000002},"min":{"1min":9.7349999999999994,"5min":7.484,"15min":7.484},"max":{"1min":71.394000000000005,"5min":204.517,"15min":392.67500000000001},"last":13.446999999999999}]},{"osd":5,"last update":"Mon Jun 3 09:06:35 2024","interfaces":[{"interface":"back","average":{"1min":15.672000000000001,"5min":16.742999999999999,"15min":20.795000000000002},"min":{"1min":8.093,"5min":6.7489999999999997,"15min":6.7489999999999997},"max":{"1min":51.326000000000001,"5min":268.10199999999998,"15min":344.23399999999998},"last":9.2910000000000004},{"interface":"front","average":{"1min":17.297999999999998,"5min":19.356999999999999,"15min":25.713999999999999},"min":{"1min":12.875,"5min":8.5180000000000007,"15min":8.5180000000000007},"max":{"1min":40.829000000000001,"5min":264.75,"15min":345.18700000000001},"last":10.318}]},{"osd":6,"last update":"Mon Jun 3 09:04:32 2024","interfaces":[{"interface":"back","average":{"1min":15.548,"5min":32.957999999999998,"15min":152.738},"min":{"1min":9.141,"5min":9.141,"15min":8.2319999999999993},"max":{"1min":52.57,"5min":395.238,"15min":12496.335999999999},"last":208.90000000000001},{"interface":"front","average":{"1min":12.680999999999999,"5min":25.428000000000001,"15min":149.30600000000001},"min":{"1min":6.4379999999999997,"5min":5.6319999999999997,"15min":5.6319999999999997},"max":{"1min":46.283000000000001,"5min":377.74299999999999,"15min":12616.297},"last":205.88900000000001}]},{"osd":7,"last update":"Mon Jun 3 09:06:35 2024","interfaces":[{"interface":"back","average":{"1min":9.6969999999999992,"5min":14.196,"15min":18.809999999999999},"min":{"1min":6.7190000000000003,"5min":6.2110000000000003,"15min":5.8010000000000002},"max":{"1min":15.960000000000001,"5min":207.25299999999999,"15min":358.94900000000001},"last":8.7810000000000006},{"interface":"front","average":{"1min":17.696000000000002,"5min":19.024999999999999,"15min":23.259},"min":{"1min":11.794,"5min":7.7789999999999999,"15min":7.7789999999999999},"max":{"1min":45.639000000000003,"5min":209.32900000000001,"15min":380.06900000000002},"last":11.362}]}]},{"osd":2,"up_from":20,"seq":85899346290,"num_pgs":11,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":58964,"kb_used_data":5704,"kb_used_omap":1,"kb_used_meta":53246,"kb_avail":94312876,"statfs":{"total":96636764160,"available":96576385024,"internally_reserved":0,"allocated":5840896,"data_stored":2740727,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1594,"internal_metadata":54524358},"hb_peers":[0,1,3,4,5,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Mon Jun 3 09:06:23 2024","interfaces":[{"interface":"back","average":{"1min":8.9629999999999992,"5min":946.25400000000002,"15min":326.36099999999999},"min":{"1min":5.3479999999999999,"5min":4.7050000000000001,"15min":4.1420000000000003},"max":{"1min":26.015000000000001,"5min":23465.18,"15min":23465.18},"last":6.3940000000000001},{"interface":"front","average":{"1min":7.6130000000000004,"5min":944.39800000000002,"15min":326.178},"min":{"1min":4.6790000000000003,"5min":4.6790000000000003,"15min":4.6790000000000003},"max":{"1min":20.802,"5min":23465.966,"15min":23465.966},"last":4.492}]},{"osd":1,"last update":"Mon Jun 3 09:06:23 2024","interfaces":[{"interface":"back","average":{"1min":13.007,"5min":76.507000000000005,"15min":294.72899999999998},"min":{"1min":6.4089999999999998,"5min":5.4199999999999999,"15min":5.4199999999999999},"max":{"1min":85.430000000000007,"5min":4220.5290000000005,"15min":18954.987000000001},"last":7.4249999999999998},{"interface":"front","average":{"1min":15.894,"5min":82.119,"15min":302.92399999999998},"min":{"1min":7.367,"5min":7.367,"15min":7.367},"max":{"1min":98.498000000000005,"5min":4325.9489999999996,"15min":19143.834999999999},"last":9.6929999999999996}]},{"osd":3,"last update":"Mon Jun 3 09:06:23 2024","interfaces":[{"interface":"back","average":{"1min":9.7579999999999991,"5min":12.444000000000001,"15min":28.065000000000001},"min":{"1min":7,"5min":5.9290000000000003,"15min":5.2450000000000001},"max":{"1min":28.164000000000001,"5min":118.913,"15min":2594.8150000000001},"last":8.8740000000000006},{"interface":"front","average":{"1min":12.321999999999999,"5min":15.25,"15min":31.349},"min":{"1min":9.1240000000000006,"5min":8.0760000000000005,"15min":6.0780000000000003},"max":{"1min":32.360999999999997,"5min":156.34700000000001,"15min":2706.5120000000002},"last":11.180999999999999}]},{"osd":4,"last update":"Mon Jun 3 09:05:57 2024","interfaces":[{"interface":"back","average":{"1min":10.573,"5min":1266.894,"15min":438.154},"min":{"1min":8.6379999999999999,"5min":7.8789999999999996,"15min":6.2039999999999997},"max":{"1min":15.105,"5min":26660.861000000001,"15min":26660.861000000001},"last":10.538},{"interface":"front","average":{"1min":12.407999999999999,"5min":1266.7460000000001,"15min":435.75400000000002},"min":{"1min":10.266999999999999,"5min":10.266999999999999,"15min":6.2720000000000002},"max":{"1min":15.228,"5min":26616.243999999999,"15min":26616.243999999999},"last":11.879}]},{"osd":5,"last update":"Mon Jun 3 09:06:23 2024","interfaces":[{"interface":"back","average":{"1min":14.103,"5min":18.091000000000001,"15min":21.119},"min":{"1min":9.6349999999999998,"5min":9.2159999999999993,"15min":6.2050000000000001},"max":{"1min":32.890000000000001,"5min":252.31100000000001,"15min":252.31100000000001},"last":12.696},{"interface":"front","average":{"1min":11.545999999999999,"5min":16.170999999999999,"15min":20.254000000000001},"min":{"1min":7.6070000000000002,"5min":7.4850000000000003,"15min":6.8810000000000002},"max":{"1min":29.228999999999999,"5min":258.31700000000001,"15min":258.31700000000001},"last":8.6699999999999999}]},{"osd":6,"last update":"Mon Jun 3 09:04:28 2024","interfaces":[{"interface":"back","average":{"1min":14.063000000000001,"5min":24.850000000000001,"15min":106.755},"min":{"1min":9.0820000000000007,"5min":8.9649999999999999,"15min":8.9649999999999999},"max":{"1min":24.666,"5min":188.06700000000001,"15min":9521.0010000000002},"last":183.60900000000001},{"interface":"front","average":{"1min":13.933999999999999,"5min":21.081,"15min":106.142},"min":{"1min":5.8719999999999999,"5min":5.8719999999999999,"15min":5.8719999999999999},"max":{"1min":117.551,"5min":193.887,"15min":9585.6620000000003},"last":184.20599999999999}]},{"osd":7,"last update":"Mon Jun 3 09:06:23 2024","interfaces":[{"interface":"back","average":{"1min":17.056000000000001,"5min":20.055,"15min":23.268999999999998},"min":{"1min":10.353999999999999,"5min":6.6130000000000004,"15min":6.6130000000000004},"max":{"1min":88.974999999999994,"5min":242.56800000000001,"15min":242.56800000000001},"last":13.343},{"interface":"front","average":{"1min":18.885000000000002,"5min":21.460000000000001,"15min":25.489999999999998},"min":{"1min":12.497,"5min":9.0660000000000007,"15min":9.0660000000000007},"max":{"1min":102.229,"5min":181.56399999999999,"15min":214.33799999999999},"last":14.042}]}]},{"osd":3,"up_from":14,"seq":60129542504,"num_pgs":11,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":58964,"kb_used_data":5704,"kb_used_omap":1,"kb_used_meta":53246,"kb_avail":94312876,"statfs":{"total":96636764160,"available":96576385024,"internally_reserved":0,"allocated":5840896,"data_stored":2740727,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1592,"internal_metadata":54524360},"hb_peers":[0,1,2,4,5,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Mon Jun 3 09:06:25 2024","interfaces":[{"interface":"back","average":{"1min":7.79,"5min":1060.1959999999999,"15min":367.01799999999997},"min":{"1min":5.6909999999999998,"5min":5.173,"15min":4.6929999999999996},"max":{"1min":10.59,"5min":23275.167000000001,"15min":23275.167000000001},"last":5.1989999999999998},{"interface":"front","average":{"1min":11.215999999999999,"5min":1066.5799999999999,"15min":372.476},"min":{"1min":8.7080000000000002,"5min":7.2510000000000003,"15min":5.9880000000000004},"max":{"1min":13.499000000000001,"5min":23278.536,"15min":23278.536},"last":10.922000000000001}]},{"osd":1,"last update":"Mon Jun 3 09:06:25 2024","interfaces":[{"interface":"back","average":{"1min":7.3760000000000003,"5min":15.614000000000001,"15min":389.83100000000002},"min":{"1min":4.335,"5min":4.335,"15min":4.335},"max":{"1min":11.289,"5min":250.72300000000001,"15min":21658.205999999998},"last":8.5549999999999997},{"interface":"front","average":{"1min":8.9499999999999993,"5min":14.135,"15min":381.10300000000001},"min":{"1min":6.5609999999999999,"5min":5.4100000000000001,"15min":5.2779999999999996},"max":{"1min":14.51,"5min":180.44499999999999,"15min":21307.069},"last":7.04}]},{"osd":2,"last update":"Mon Jun 3 09:06:25 2024","interfaces":[{"interface":"back","average":{"1min":12.343999999999999,"5min":22.059999999999999,"15min":23.715},"min":{"1min":7.9199999999999999,"5min":6.4900000000000002,"15min":6.056},"max":{"1min":15.601000000000001,"5min":377.803,"15min":377.803},"last":12.581},{"interface":"front","average":{"1min":10.319000000000001,"5min":19.364999999999998,"15min":20.242999999999999},"min":{"1min":6.891,"5min":6.1749999999999998,"15min":5.9020000000000001},"max":{"1min":15.432,"5min":371.601,"15min":371.601},"last":10.368}]},{"osd":4,"last update":"Mon Jun 3 09:06:31 2024","interfaces":[{"interface":"back","average":{"1min":9.7460000000000004,"5min":891.23000000000002,"15min":416.48099999999999},"min":{"1min":5.6539999999999999,"5min":5.6539999999999999,"15min":5.6539999999999999},"max":{"1min":15.445,"5min":29040.635999999999,"15min":30144.939999999999},"last":8.0259999999999998},{"interface":"front","average":{"1min":12.398999999999999,"5min":897.58500000000004,"15min":421.27499999999998},"min":{"1min":8.9469999999999992,"5min":7.8109999999999999,"15min":6.2439999999999998},"max":{"1min":16.518000000000001,"5min":29076.955999999998,"15min":30180.689999999999},"last":9.1539999999999999}]},{"osd":5,"last update":"Mon Jun 3 09:06:07 2024","interfaces":[{"interface":"back","average":{"1min":12.802,"5min":19.640999999999998,"15min":22.318000000000001},"min":{"1min":9.0890000000000004,"5min":6.8810000000000002,"15min":6.7690000000000001},"max":{"1min":16.745999999999999,"5min":249.88,"15min":327.34899999999999},"last":14.225},{"interface":"front","average":{"1min":15.002000000000001,"5min":24.003,"15min":26.649999999999999},"min":{"1min":11.984,"5min":9.9149999999999991,"15min":8.9879999999999995},"max":{"1min":18.407,"5min":251.47800000000001,"15min":331.86799999999999},"last":14.91}]},{"osd":6,"last update":"Mon Jun 3 09:04:04 2024","interfaces":[{"interface":"back","average":{"1min":22.815999999999999,"5min":31.513999999999999,"15min":181.74299999999999},"min":{"1min":9.4809999999999999,"5min":7.2240000000000002,"15min":7.2240000000000002},"max":{"1min":160.56999999999999,"5min":380.613,"15min":11893.74},"last":19.359999999999999},{"interface":"front","average":{"1min":25.359999999999999,"5min":30.981000000000002,"15min":179.81399999999999},"min":{"1min":9.8870000000000005,"5min":7.3360000000000003,"15min":6.0899999999999999},"max":{"1min":193.11099999999999,"5min":379.80700000000002,"15min":11893.069},"last":8.202}]},{"osd":7,"last update":"Mon Jun 3 09:06:25 2024","interfaces":[{"interface":"back","average":{"1min":11.321999999999999,"5min":18.443999999999999,"15min":21.568999999999999},"min":{"1min":8.109,"5min":8.109,"15min":7.4279999999999999},"max":{"1min":17.113,"5min":146.267,"15min":178.76300000000001},"last":11.492000000000001},{"interface":"front","average":{"1min":12.512,"5min":20.065999999999999,"15min":24.190000000000001},"min":{"1min":7.4989999999999997,"5min":7.4989999999999997,"15min":7.4989999999999997},"max":{"1min":18.047999999999998,"5min":148.13800000000001,"15min":417.22800000000001},"last":13.419}]}]},{"osd":1,"up_from":14,"seq":60129542502,"num_pgs":6,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":59356,"kb_used_data":6096,"kb_used_omap":1,"kb_used_meta":53246,"kb_avail":94312484,"statfs":{"total":96636764160,"available":96575983616,"internally_reserved":0,"allocated":6242304,"data_stored":3138567,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":54524357},"hb_peers":[0,2,3,4,5,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Mon Jun 3 09:06:07 2024","interfaces":[{"interface":"back","average":{"1min":7.7530000000000001,"5min":185.66999999999999,"15min":77.347999999999999},"min":{"1min":5.0229999999999997,"5min":4.9279999999999999,"15min":4.5449999999999999},"max":{"1min":11.896000000000001,"5min":7470.616,"15min":7470.616},"last":10.537000000000001},{"interface":"front","average":{"1min":9.9220000000000006,"5min":187.14599999999999,"15min":76.325999999999993},"min":{"1min":5.3159999999999998,"5min":5.3159999999999998,"15min":5.3159999999999998},"max":{"1min":18.338000000000001,"5min":7477.0829999999996,"15min":7477.0829999999996},"last":9.5649999999999995}]},{"osd":2,"last update":"Mon Jun 3 09:06:20 2024","interfaces":[{"interface":"back","average":{"1min":10.340999999999999,"5min":17.369,"15min":23.670999999999999},"min":{"1min":6.9740000000000002,"5min":5.8650000000000002,"15min":5.8650000000000002},"max":{"1min":15.452,"5min":228,"15min":657.38},"last":11.122999999999999},{"interface":"front","average":{"1min":13.387,"5min":23.355,"15min":31.524000000000001},"min":{"1min":9.077,"5min":9.077,"15min":7.2160000000000002},"max":{"1min":19.701000000000001,"5min":229.465,"15min":700.06500000000005},"last":15.970000000000001}]},{"osd":3,"last update":"Mon Jun 3 09:06:04 2024","interfaces":[{"interface":"back","average":{"1min":8.6110000000000007,"5min":15.733000000000001,"15min":26.646999999999998},"min":{"1min":5.827,"5min":5.4640000000000004,"15min":5.4640000000000004},"max":{"1min":13.105,"5min":121.619,"15min":1676.009},"last":11.763},{"interface":"front","average":{"1min":9.5850000000000009,"5min":16.161999999999999,"15min":29.065999999999999},"min":{"1min":6.3630000000000004,"5min":5.7699999999999996,"15min":5.7699999999999996},"max":{"1min":15.166,"5min":132.036,"15min":1766.5840000000001},"last":8.7479999999999993}]},{"osd":4,"last update":"Mon Jun 3 09:05:31 2024","interfaces":[{"interface":"back","average":{"1min":10.996,"5min":954.46500000000003,"15min":1281.6859999999999},"min":{"1min":7.702,"5min":6.1760000000000002,"15min":6.0970000000000004},"max":{"1min":21.538,"5min":28044.862000000001,"15min":59438.017999999996},"last":9.3219999999999992},{"interface":"front","average":{"1min":13.646000000000001,"5min":961.24599999999998,"15min":1285.26},"min":{"1min":9.7200000000000006,"5min":9.3940000000000001,"15min":6.4640000000000004},"max":{"1min":23.277999999999999,"5min":28045.624,"15min":59439.283000000003},"last":12.798}]},{"osd":5,"last update":"Mon Jun 3 09:06:04 2024","interfaces":[{"interface":"back","average":{"1min":12.478999999999999,"5min":20.228000000000002,"15min":25.997},"min":{"1min":8.6430000000000007,"5min":7.9720000000000004,"15min":7.234},"max":{"1min":17.442,"5min":139.43199999999999,"15min":701.22000000000003},"last":13.678000000000001},{"interface":"front","average":{"1min":13.507999999999999,"5min":25.760999999999999,"15min":27.553999999999998},"min":{"1min":10.042999999999999,"5min":7.258,"15min":6.242},"max":{"1min":20.611000000000001,"5min":278.63,"15min":598.35000000000002},"last":14.173}]},{"osd":6,"last update":"Mon Jun 3 09:03:54 2024","interfaces":[{"interface":"back","average":{"1min":28.524000000000001,"5min":20.654,"15min":23.878},"min":{"1min":6.0309999999999997,"5min":6.0309999999999997,"15min":6.0309999999999997},"max":{"1min":167.221,"5min":167.221,"15min":704.245},"last":18.721},{"interface":"front","average":{"1min":27.606000000000002,"5min":30.131,"15min":28.628},"min":{"1min":9.8650000000000002,"5min":9.7200000000000006,"15min":8.3580000000000005},"max":{"1min":123.938,"5min":358.23000000000002,"15min":634.05499999999995},"last":173.113}]},{"osd":7,"last update":"Mon Jun 3 09:06:20 2024","interfaces":[{"interface":"back","average":{"1min":14.66,"5min":23.234999999999999,"15min":31.097000000000001},"min":{"1min":11.250999999999999,"5min":8.0559999999999992,"15min":8.0559999999999992},"max":{"1min":21.507999999999999,"5min":270.95100000000002,"15min":703.07799999999997},"last":15.116},{"interface":"front","average":{"1min":15.548,"5min":28.760000000000002,"15min":28.925000000000001},"min":{"1min":12.759,"5min":8.2089999999999996,"15min":7.7599999999999998},"max":{"1min":22.408000000000001,"5min":360.99900000000002,"15min":608.44100000000003},"last":16.640000000000001}]}]},{"osd":0,"up_from":14,"seq":60129542504,"num_pgs":10,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":81108,"kb_used_data":5704,"kb_used_omap":1,"kb_used_meta":75390,"kb_avail":94290732,"statfs":{"total":96636764160,"available":96553709568,"internally_reserved":0,"allocated":5840896,"data_stored":2740727,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1593,"internal_metadata":77199815},"hb_peers":[1,2,3,4,5,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[{"osd":1,"last update":"Mon Jun 3 09:06:04 2024","interfaces":[{"interface":"back","average":{"1min":6.9960000000000004,"5min":56.997,"15min":339.29700000000003},"min":{"1min":4.4530000000000003,"5min":4.4530000000000003,"15min":4.4100000000000001},"max":{"1min":11.720000000000001,"5min":2414.4749999999999,"15min":20691.453000000001},"last":5.7039999999999997},{"interface":"front","average":{"1min":9.548,"5min":57.258000000000003,"15min":337.791},"min":{"1min":5.2300000000000004,"5min":5.2300000000000004,"15min":4.6050000000000004},"max":{"1min":15.337999999999999,"5min":2412.9549999999999,"15min":20640.792000000001},"last":8.2159999999999993}]},{"osd":2,"last update":"Mon Jun 3 09:06:22 2024","interfaces":[{"interface":"back","average":{"1min":11.154,"5min":26.515999999999998,"15min":22.882000000000001},"min":{"1min":5.7869999999999999,"5min":5.766,"15min":5.766},"max":{"1min":72.269999999999996,"5min":627.40700000000004,"15min":627.40700000000004},"last":11.311},{"interface":"front","average":{"1min":14.353999999999999,"5min":33.304000000000002,"15min":29.885000000000002},"min":{"1min":8.2149999999999999,"5min":7.8630000000000004,"15min":7.4089999999999998},"max":{"1min":73.418000000000006,"5min":635.62599999999998,"15min":635.62599999999998},"last":13.371}]},{"osd":3,"last update":"Mon Jun 3 09:06:04 2024","interfaces":[{"interface":"back","average":{"1min":12.105,"5min":26.925000000000001,"15min":26.584},"min":{"1min":7.8410000000000002,"5min":7.1280000000000001,"15min":6.1909999999999998},"max":{"1min":18.425999999999998,"5min":389.72699999999998,"15min":389.72699999999998},"last":9.4990000000000006},{"interface":"front","average":{"1min":13.794,"5min":29.771999999999998,"15min":28.815000000000001},"min":{"1min":10.084,"5min":9.4920000000000009,"15min":6.1849999999999996},"max":{"1min":17.373999999999999,"5min":390.23200000000003,"15min":494.92899999999997},"last":10.595000000000001}]},{"osd":4,"last update":"Mon Jun 3 09:06:27 2024","interfaces":[{"interface":"back","average":{"1min":11.352,"5min":745.30600000000004,"15min":363.48099999999999},"min":{"1min":6.415,"5min":6.415,"15min":6.415},"max":{"1min":16.724,"5min":20248.616000000002,"15min":25555.212},"last":12.867000000000001},{"interface":"front","average":{"1min":13.545,"5min":744.76999999999998,"15min":361.03100000000001},"min":{"1min":9.3889999999999993,"5min":8.8940000000000001,"15min":7.1429999999999998},"max":{"1min":18.411000000000001,"5min":20250.195,"15min":25556.159},"last":14.318}]},{"osd":5,"last update":"Mon Jun 3 09:06:17 2024","interfaces":[{"interface":"back","average":{"1min":9.7040000000000006,"5min":26.533999999999999,"15min":28.683},"min":{"1min":6.8129999999999997,"5min":6.8129999999999997,"15min":6.8129999999999997},"max":{"1min":14.942,"5min":455.78699999999998,"15min":605.05899999999997},"last":6.6539999999999999},{"interface":"front","average":{"1min":14.279,"5min":28.398,"15min":26.192},"min":{"1min":11.696,"5min":6.4420000000000002,"15min":6.2030000000000003},"max":{"1min":17.879000000000001,"5min":391.31200000000001,"15min":541.11000000000001},"last":11.938000000000001}]},{"osd":6,"last update":"Mon Jun 3 09:04:30 2024","interfaces":[{"interface":"back","average":{"1min":52.496000000000002,"5min":33.139000000000003,"15min":160.83199999999999},"min":{"1min":7.5289999999999999,"5min":7.1040000000000001,"15min":6.3739999999999997},"max":{"1min":457.91699999999997,"5min":457.91699999999997,"15min":11115.258},"last":18.959},{"interface":"front","average":{"1min":49.670000000000002,"5min":33.704999999999998,"15min":162.791},"min":{"1min":12.247999999999999,"5min":11.019,"15min":9.9760000000000009},"max":{"1min":407.05099999999999,"5min":407.05099999999999,"15min":11092.194},"last":17.707000000000001}]},{"osd":7,"last update":"Mon Jun 3 09:06:22 2024","interfaces":[{"interface":"back","average":{"1min":11.625999999999999,"5min":26.375,"15min":28.972999999999999},"min":{"1min":6.4480000000000004,"5min":6.4480000000000004,"15min":6.4480000000000004},"max":{"1min":15.352,"5min":461.41699999999997,"15min":598.33699999999999},"last":7.1369999999999996},{"interface":"front","average":{"1min":14.013,"5min":29.696000000000002,"15min":33.630000000000003},"min":{"1min":10.938000000000001,"5min":10.58,"15min":10.263999999999999},"max":{"1min":16.603000000000002,"5min":463.10599999999999,"15min":592.80499999999995},"last":8.7810000000000006}]}]}],"pool_statfs":[{"poolid":1,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":401408,"data_stored":397840,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":5,"total":0,"available":0,"internally_reserved":0,"allocated":401408,"data_stored":397840,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":4,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":5,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":6,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":17,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":17,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":17,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":17,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":17,"osd":4,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":17,"osd":5,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":5,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":17,"osd":6,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":5,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":17,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0}]}} 2024-06-03T09:06:42.272 DEBUG:teuthology.orchestra.run.smithi088:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph pg dump --format=json 2024-06-03T09:06:42.372 INFO:teuthology.orchestra.run.smithi088.stderr:2024-06-03T09:06:42.372+0000 7f9cb9ad5640 10 monclient: get_monmap_and_config 2024-06-03T09:06:42.372 INFO:teuthology.orchestra.run.smithi088.stderr:2024-06-03T09:06:42.372+0000 7f9cb9ad5640 10 monclient: build_initial_monmap 2024-06-03T09:06:42.372 INFO:teuthology.orchestra.run.smithi088.stderr:2024-06-03T09:06:42.372+0000 7f9cb9ad5640 10 monclient: monmap: 2024-06-03T09:06:42.372 INFO:teuthology.orchestra.run.smithi088.stderr:epoch 0 2024-06-03T09:06:42.372 INFO:teuthology.orchestra.run.smithi088.stderr:fsid ec3c3fdd-3ac6-4d7d-9412-683130ed8b04 2024-06-03T09:06:42.372 INFO:teuthology.orchestra.run.smithi088.stderr:last_changed 2024-06-03T09:06:42.373140+0000 2024-06-03T09:06:42.372 INFO:teuthology.orchestra.run.smithi088.stderr:created 2024-06-03T09:06:42.373140+0000 2024-06-03T09:06:42.372 INFO:teuthology.orchestra.run.smithi088.stderr:min_mon_release 0 (unknown) 2024-06-03T09:06:42.372 INFO:teuthology.orchestra.run.smithi088.stderr:election_strategy: 3 2024-06-03T09:06:42.373 INFO:teuthology.orchestra.run.smithi088.stderr:0: v1:172.21.15.88:6789/0 mon.noname-a 2024-06-03T09:06:42.373 INFO:teuthology.orchestra.run.smithi088.stderr:1: v1:172.21.15.192:6789/0 mon.noname-c 2024-06-03T09:06:42.373 INFO:teuthology.orchestra.run.smithi088.stderr:2: v1:172.21.15.88:6790/0 mon.noname-b 2024-06-03T09:06:42.373 INFO:teuthology.orchestra.run.smithi088.stderr: 2024-06-03T09:06:42.374 INFO:teuthology.orchestra.run.smithi088.stderr:2024-06-03T09:06:42.372+0000 7f9cb9ad5640 1 Processor -- start 2024-06-03T09:06:42.374 INFO:teuthology.orchestra.run.smithi088.stderr:2024-06-03T09:06:42.372+0000 7f9cb9ad5640 1 -- start start 2024-06-03T09:06:42.374 INFO:teuthology.orchestra.run.smithi088.stderr:2024-06-03T09:06:42.372+0000 7f9cb9ad5640 10 monclient: init 2024-06-03T09:06:42.374 INFO:teuthology.orchestra.run.smithi088.stderr:2024-06-03T09:06:42.373+0000 7f9cb9ad5640 10 monclient: _reopen_session rank -1 2024-06-03T09:06:42.374 INFO:teuthology.orchestra.run.smithi088.stderr:2024-06-03T09:06:42.373+0000 7f9cb9ad5640 10 monclient: _add_conns ranks=[1,0,2] 2024-06-03T09:06:42.374 INFO:teuthology.orchestra.run.smithi088.stderr:2024-06-03T09:06:42.373+0000 7f9cb9ad5640 10 monclient(hunting): picked mon.noname-c con 0x7f9cb40a8960 addr v1:172.21.15.192:6789/0 2024-06-03T09:06:42.375 INFO:teuthology.orchestra.run.smithi088.stderr:2024-06-03T09:06:42.373+0000 7f9cb9ad5640 10 monclient(hunting): picked mon.noname-a con 0x7f9cb41450f0 addr v1:172.21.15.88:6789/0 2024-06-03T09:06:42.375 INFO:teuthology.orchestra.run.smithi088.stderr:2024-06-03T09:06:42.373+0000 7f9cb9ad5640 10 monclient(hunting): picked mon.noname-b con 0x7f9cb413eec0 addr v1:172.21.15.88:6790/0 2024-06-03T09:06:42.375 INFO:teuthology.orchestra.run.smithi088.stderr:2024-06-03T09:06:42.373+0000 7f9cb9ad5640 1 -- --> v1:172.21.15.88:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f9cb4144c60 con 0x7f9cb41450f0 2024-06-03T09:06:42.375 INFO:teuthology.orchestra.run.smithi088.stderr:2024-06-03T09:06:42.373+0000 7f9cb9ad5640 1 -- --> v1:172.21.15.192:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f9cb40b3930 con 0x7f9cb40a8960 2024-06-03T09:06:42.375 INFO:teuthology.orchestra.run.smithi088.stderr:2024-06-03T09:06:42.373+0000 7f9cb9ad5640 1 -- --> v1:172.21.15.88:6790/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f9cb40b1ad0 con 0x7f9cb413eec0 2024-06-03T09:06:42.375 INFO:teuthology.orchestra.run.smithi088.stderr:2024-06-03T09:06:42.373+0000 7f9cb9ad5640 10 monclient(hunting): _renew_subs 2024-06-03T09:06:42.375 INFO:teuthology.orchestra.run.smithi088.stderr:2024-06-03T09:06:42.373+0000 7f9cb9ad5640 10 monclient(hunting): authenticate will time out at 3058.183458s 2024-06-03T09:06:42.375 INFO:teuthology.orchestra.run.smithi088.stderr:2024-06-03T09:06:42.374+0000 7f9cb3fff640 1 --1- >> v1:172.21.15.88:6790/0 conn(0x7f9cb413eec0 0x7f9cb414ce60 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.88:6790/0 says I am v1:172.21.15.88:41272/0 (socket says 172.21.15.88:41272) 2024-06-03T09:06:42.375 INFO:teuthology.orchestra.run.smithi088.stderr:2024-06-03T09:06:42.374+0000 7f9cb2ffd640 1 --1- >> v1:172.21.15.88:6789/0 conn(0x7f9cb41450f0 0x7f9cb413e7b0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.88:6789/0 says I am v1:172.21.15.88:54394/0 (socket says 172.21.15.88:54394) 2024-06-03T09:06:42.375 INFO:teuthology.orchestra.run.smithi088.stderr:2024-06-03T09:06:42.374+0000 7f9cb3fff640 1 -- 172.21.15.88:0/4183956825 learned_addr learned my addr 172.21.15.88:0/4183956825 (peer_addr_for_me v1:172.21.15.88:0/0) 2024-06-03T09:06:42.376 INFO:teuthology.orchestra.run.smithi088.stderr:2024-06-03T09:06:42.376+0000 7f9cb27fc640 1 -- 172.21.15.88:0/4183956825 <== mon.2 v1:172.21.15.88:6790/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 4056714555 0 0) 0x7f9cb40b1ad0 con 0x7f9cb413eec0 2024-06-03T09:06:42.377 INFO:teuthology.orchestra.run.smithi088.stderr:2024-06-03T09:06:42.376+0000 7f9cb27fc640 10 monclient(hunting): _negotiate 2024-06-03T09:06:42.377 INFO:teuthology.orchestra.run.smithi088.stderr:2024-06-03T09:06:42.376+0000 7f9cb27fc640 10 monclient(hunting): _init_auth method 2 2024-06-03T09:06:42.377 INFO:teuthology.orchestra.run.smithi088.stderr:2024-06-03T09:06:42.376+0000 7f9cb27fc640 10 monclient(hunting): _init_auth creating new auth 2024-06-03T09:06:42.377 INFO:teuthology.orchestra.run.smithi088.stderr:2024-06-03T09:06:42.376+0000 7f9cb27fc640 10 monclient(hunting): my global_id is 30176 2024-06-03T09:06:42.377 INFO:teuthology.orchestra.run.smithi088.stderr:2024-06-03T09:06:42.376+0000 7f9cb27fc640 1 -- 172.21.15.88:0/4183956825 --> v1:172.21.15.88:6790/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f9c9c003520 con 0x7f9cb413eec0 2024-06-03T09:06:42.377 INFO:teuthology.orchestra.run.smithi088.stderr:2024-06-03T09:06:42.376+0000 7f9cb27fc640 1 -- 172.21.15.88:0/4183956825 <== mon.0 v1:172.21.15.88:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 1402940566 0 0) 0x7f9cb4144c60 con 0x7f9cb41450f0 2024-06-03T09:06:42.377 INFO:teuthology.orchestra.run.smithi088.stderr:2024-06-03T09:06:42.376+0000 7f9cb27fc640 10 monclient(hunting): _negotiate 2024-06-03T09:06:42.377 INFO:teuthology.orchestra.run.smithi088.stderr:2024-06-03T09:06:42.376+0000 7f9cb27fc640 10 monclient(hunting): _init_auth method 2 2024-06-03T09:06:42.377 INFO:teuthology.orchestra.run.smithi088.stderr:2024-06-03T09:06:42.376+0000 7f9cb27fc640 10 monclient(hunting): _init_auth creating new auth 2024-06-03T09:06:42.377 INFO:teuthology.orchestra.run.smithi088.stderr:2024-06-03T09:06:42.376+0000 7f9cb27fc640 10 monclient(hunting): my global_id is 30672 2024-06-03T09:06:42.377 INFO:teuthology.orchestra.run.smithi088.stderr:2024-06-03T09:06:42.376+0000 7f9cb27fc640 1 -- 172.21.15.88:0/4183956825 --> v1:172.21.15.88:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f9cb40b1ad0 con 0x7f9cb41450f0 2024-06-03T09:06:42.378 INFO:teuthology.orchestra.run.smithi088.stderr:2024-06-03T09:06:42.376+0000 7f9cb27fc640 1 -- 172.21.15.88:0/4183956825 <== mon.1 v1:172.21.15.192:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 3683595113 0 0) 0x7f9cb40b3930 con 0x7f9cb40a8960 2024-06-03T09:06:42.378 INFO:teuthology.orchestra.run.smithi088.stderr:2024-06-03T09:06:42.376+0000 7f9cb27fc640 10 monclient(hunting): _negotiate 2024-06-03T09:06:42.378 INFO:teuthology.orchestra.run.smithi088.stderr:2024-06-03T09:06:42.376+0000 7f9cb27fc640 10 monclient(hunting): _init_auth method 2 2024-06-03T09:06:42.378 INFO:teuthology.orchestra.run.smithi088.stderr:2024-06-03T09:06:42.376+0000 7f9cb27fc640 10 monclient(hunting): _init_auth creating new auth 2024-06-03T09:06:42.378 INFO:teuthology.orchestra.run.smithi088.stderr:2024-06-03T09:06:42.376+0000 7f9cb27fc640 10 monclient(hunting): my global_id is 30250 2024-06-03T09:06:42.378 INFO:teuthology.orchestra.run.smithi088.stderr:2024-06-03T09:06:42.376+0000 7f9cb27fc640 1 -- 172.21.15.88:0/4183956825 --> v1:172.21.15.192:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f9cb4144c60 con 0x7f9cb40a8960 2024-06-03T09:06:42.380 INFO:teuthology.orchestra.run.smithi088.stderr:2024-06-03T09:06:42.379+0000 7f9cb27fc640 1 -- 172.21.15.88:0/4183956825 <== mon.1 v1:172.21.15.192:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 214+0+0 (unknown 2266702981 0 0) 0x7f9cb4144c60 con 0x7f9cb40a8960 2024-06-03T09:06:42.380 INFO:teuthology.orchestra.run.smithi088.stderr:2024-06-03T09:06:42.379+0000 7f9cb27fc640 1 -- 172.21.15.88:0/4183956825 >> v1:172.21.15.88:6790/0 conn(0x7f9cb413eec0 legacy=0x7f9cb414ce60 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-06-03T09:06:42.380 INFO:teuthology.orchestra.run.smithi088.stderr:2024-06-03T09:06:42.379+0000 7f9cb27fc640 1 -- 172.21.15.88:0/4183956825 >> v1:172.21.15.88:6789/0 conn(0x7f9cb41450f0 legacy=0x7f9cb413e7b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-06-03T09:06:42.380 INFO:teuthology.orchestra.run.smithi088.stderr:2024-06-03T09:06:42.380+0000 7f9cb27fc640 10 monclient: _finish_hunting 0 2024-06-03T09:06:42.380 INFO:teuthology.orchestra.run.smithi088.stderr:2024-06-03T09:06:42.380+0000 7f9cb27fc640 1 monclient: found mon.noname-c 2024-06-03T09:06:42.380 INFO:teuthology.orchestra.run.smithi088.stderr:2024-06-03T09:06:42.380+0000 7f9cb27fc640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-06-03T09:06:42.380 INFO:teuthology.orchestra.run.smithi088.stderr:2024-06-03T09:06:42.380+0000 7f9cb27fc640 10 monclient: _send_mon_message to mon.noname-c at v1:172.21.15.192:6789/0 2024-06-03T09:06:42.380 INFO:teuthology.orchestra.run.smithi088.stderr:2024-06-03T09:06:42.380+0000 7f9cb27fc640 1 -- 172.21.15.88:0/4183956825 --> v1:172.21.15.192:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f9cb40b1c60 con 0x7f9cb40a8960 2024-06-03T09:06:42.381 INFO:teuthology.orchestra.run.smithi088.stderr:2024-06-03T09:06:42.380+0000 7f9cb27fc640 10 monclient: _finish_auth 0 2024-06-03T09:06:42.381 INFO:teuthology.orchestra.run.smithi088.stderr:2024-06-03T09:06:42.380+0000 7f9cb27fc640 10 monclient: _check_auth_tickets 2024-06-03T09:06:42.381 INFO:teuthology.orchestra.run.smithi088.stderr:2024-06-03T09:06:42.380+0000 7f9cb27fc640 20 monclient: _check_auth_rotating not needed by client.admin 2024-06-03T09:06:42.381 INFO:teuthology.orchestra.run.smithi088.stderr:2024-06-03T09:06:42.380+0000 7f9cb27fc640 1 -- 172.21.15.88:0/4183956825 <== mon.1 v1:172.21.15.192:6789/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 2234374065 0 0) 0x7f9ca40015b0 con 0x7f9cb40a8960 2024-06-03T09:06:42.381 INFO:teuthology.orchestra.run.smithi088.stderr:2024-06-03T09:06:42.380+0000 7f9cb27fc640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-06-03T09:06:42.381 INFO:teuthology.orchestra.run.smithi088.stderr:2024-06-03T09:06:42.380+0000 7f9cb27fc640 10 monclient: got monmap 1 from mon.noname-c (according to old e1) 2024-06-03T09:06:42.381 INFO:teuthology.orchestra.run.smithi088.stderr:2024-06-03T09:06:42.380+0000 7f9cb27fc640 10 monclient: dump: 2024-06-03T09:06:42.381 INFO:teuthology.orchestra.run.smithi088.stderr:epoch 1 2024-06-03T09:06:42.381 INFO:teuthology.orchestra.run.smithi088.stderr:fsid ec3c3fdd-3ac6-4d7d-9412-683130ed8b04 2024-06-03T09:06:42.381 INFO:teuthology.orchestra.run.smithi088.stderr:last_changed 2024-06-03T08:34:31.885125+0000 2024-06-03T09:06:42.381 INFO:teuthology.orchestra.run.smithi088.stderr:created 2024-06-03T08:34:31.885125+0000 2024-06-03T09:06:42.381 INFO:teuthology.orchestra.run.smithi088.stderr:min_mon_release 18 (reef) 2024-06-03T09:06:42.381 INFO:teuthology.orchestra.run.smithi088.stderr:election_strategy: 3 2024-06-03T09:06:42.381 INFO:teuthology.orchestra.run.smithi088.stderr:0: v1:172.21.15.88:6789/0 mon.a 2024-06-03T09:06:42.382 INFO:teuthology.orchestra.run.smithi088.stderr:1: v1:172.21.15.192:6789/0 mon.b 2024-06-03T09:06:42.382 INFO:teuthology.orchestra.run.smithi088.stderr:2: v1:172.21.15.88:6790/0 mon.c 2024-06-03T09:06:42.382 INFO:teuthology.orchestra.run.smithi088.stderr: 2024-06-03T09:06:42.382 INFO:teuthology.orchestra.run.smithi088.stderr:2024-06-03T09:06:42.380+0000 7f9cb9ad5640 5 monclient: authenticate success, global_id 30250 2024-06-03T09:06:42.382 INFO:teuthology.orchestra.run.smithi088.stderr:2024-06-03T09:06:42.380+0000 7f9cb9ad5640 20 monclient: get_monmap_and_config waiting for monmap|config 2024-06-03T09:06:42.384 INFO:teuthology.orchestra.run.smithi088.stderr:2024-06-03T09:06:42.381+0000 7f9cb27fc640 1 -- 172.21.15.88:0/4183956825 <== mon.1 v1:172.21.15.192:6789/0 4 ==== config(0 keys) v1 ==== 4+0+0 (unknown 0 0 0) 0x7f9ca40017d0 con 0x7f9cb40a8960 2024-06-03T09:06:42.384 INFO:teuthology.orchestra.run.smithi088.stderr:2024-06-03T09:06:42.381+0000 7f9cb27fc640 10 monclient: handle_config config(0 keys) v1 2024-06-03T09:06:42.384 INFO:teuthology.orchestra.run.smithi088.stderr:2024-06-03T09:06:42.381+0000 7f9cb27fc640 1 -- 172.21.15.88:0/4183956825 <== mon.1 v1:172.21.15.192:6789/0 5 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 2234374065 0 0) 0x7f9ca4001ae0 con 0x7f9cb40a8960 2024-06-03T09:06:42.384 INFO:teuthology.orchestra.run.smithi088.stderr:2024-06-03T09:06:42.381+0000 7f9cb9ad5640 10 monclient: get_monmap_and_config success 2024-06-03T09:06:42.384 INFO:teuthology.orchestra.run.smithi088.stderr:2024-06-03T09:06:42.381+0000 7f9cb9ad5640 4 set_mon_vals no callback set 2024-06-03T09:06:42.385 INFO:teuthology.orchestra.run.smithi088.stderr:2024-06-03T09:06:42.381+0000 7f9cb27fc640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-06-03T09:06:42.385 INFO:teuthology.orchestra.run.smithi088.stderr:2024-06-03T09:06:42.381+0000 7f9cb27fc640 10 monclient: got monmap 1 from mon.b (according to old e1) 2024-06-03T09:06:42.385 INFO:teuthology.orchestra.run.smithi088.stderr:2024-06-03T09:06:42.381+0000 7f9cb27fc640 10 monclient: dump: 2024-06-03T09:06:42.385 INFO:teuthology.orchestra.run.smithi088.stderr:epoch 1 2024-06-03T09:06:42.385 INFO:teuthology.orchestra.run.smithi088.stderr:fsid ec3c3fdd-3ac6-4d7d-9412-683130ed8b04 2024-06-03T09:06:42.385 INFO:teuthology.orchestra.run.smithi088.stderr:last_changed 2024-06-03T08:34:31.885125+0000 2024-06-03T09:06:42.385 INFO:teuthology.orchestra.run.smithi088.stderr:created 2024-06-03T08:34:31.885125+0000 2024-06-03T09:06:42.385 INFO:teuthology.orchestra.run.smithi088.stderr:min_mon_release 18 (reef) 2024-06-03T09:06:42.385 INFO:teuthology.orchestra.run.smithi088.stderr:election_strategy: 3 2024-06-03T09:06:42.385 INFO:teuthology.orchestra.run.smithi088.stderr:0: v1:172.21.15.88:6789/0 mon.a 2024-06-03T09:06:42.385 INFO:teuthology.orchestra.run.smithi088.stderr:1: v1:172.21.15.192:6789/0 mon.b 2024-06-03T09:06:42.385 INFO:teuthology.orchestra.run.smithi088.stderr:2: v1:172.21.15.88:6790/0 mon.c 2024-06-03T09:06:42.385 INFO:teuthology.orchestra.run.smithi088.stderr: 2024-06-03T09:06:42.386 INFO:teuthology.orchestra.run.smithi088.stderr:2024-06-03T09:06:42.382+0000 7f9cb9ad5640 10 monclient: shutdown 2024-06-03T09:06:42.386 INFO:teuthology.orchestra.run.smithi088.stderr:2024-06-03T09:06:42.382+0000 7f9cb9ad5640 20 monclient: shutdown discarding 0 pending message(s) 2024-06-03T09:06:42.386 INFO:teuthology.orchestra.run.smithi088.stderr:2024-06-03T09:06:42.382+0000 7f9cb9ad5640 1 -- 172.21.15.88:0/4183956825 >> v1:172.21.15.192:6789/0 conn(0x7f9cb40a8960 legacy=0x7f9cb40a8d40 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-06-03T09:06:42.386 INFO:teuthology.orchestra.run.smithi088.stderr:2024-06-03T09:06:42.382+0000 7f9cb9ad5640 1 -- 172.21.15.88:0/4183956825 shutdown_connections 2024-06-03T09:06:42.386 INFO:teuthology.orchestra.run.smithi088.stderr:2024-06-03T09:06:42.382+0000 7f9cb9ad5640 1 -- 172.21.15.88:0/4183956825 >> 172.21.15.88:0/4183956825 conn(0x7f9cb40b70d0 msgr2=0x7f9cb40b74e0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-06-03T09:06:42.386 INFO:teuthology.orchestra.run.smithi088.stderr:2024-06-03T09:06:42.382+0000 7f9cb9ad5640 1 -- 172.21.15.88:0/4183956825 shutdown_connections 2024-06-03T09:06:42.386 INFO:teuthology.orchestra.run.smithi088.stderr:2024-06-03T09:06:42.382+0000 7f9cb9ad5640 1 -- 172.21.15.88:0/4183956825 wait complete. 2024-06-03T09:06:42.386 INFO:teuthology.orchestra.run.smithi088.stderr:2024-06-03T09:06:42.383+0000 7f9cb9ad5640 10 monclient: build_initial_monmap 2024-06-03T09:06:42.386 INFO:teuthology.orchestra.run.smithi088.stderr:2024-06-03T09:06:42.383+0000 7f9cb9ad5640 10 monclient: monmap: 2024-06-03T09:06:42.386 INFO:teuthology.orchestra.run.smithi088.stderr:epoch 0 2024-06-03T09:06:42.386 INFO:teuthology.orchestra.run.smithi088.stderr:fsid 00000000-0000-0000-0000-000000000000 2024-06-03T09:06:42.386 INFO:teuthology.orchestra.run.smithi088.stderr:last_changed 0.000000 2024-06-03T09:06:42.386 INFO:teuthology.orchestra.run.smithi088.stderr:created 0.000000 2024-06-03T09:06:42.386 INFO:teuthology.orchestra.run.smithi088.stderr:min_mon_release 0 (unknown) 2024-06-03T09:06:42.387 INFO:teuthology.orchestra.run.smithi088.stderr:election_strategy: 1 2024-06-03T09:06:42.387 INFO:teuthology.orchestra.run.smithi088.stderr:0: v1:172.21.15.88:6789/0 mon.noname-a 2024-06-03T09:06:42.387 INFO:teuthology.orchestra.run.smithi088.stderr:1: v1:172.21.15.192:6789/0 mon.noname-b 2024-06-03T09:06:42.387 INFO:teuthology.orchestra.run.smithi088.stderr:2: v1:172.21.15.88:6790/0 mon.noname-c 2024-06-03T09:06:42.387 INFO:teuthology.orchestra.run.smithi088.stderr: 2024-06-03T09:06:42.387 INFO:teuthology.orchestra.run.smithi088.stderr:2024-06-03T09:06:42.383+0000 7f9cb9ad5640 1 librados: starting msgr at 2024-06-03T09:06:42.387 INFO:teuthology.orchestra.run.smithi088.stderr:2024-06-03T09:06:42.383+0000 7f9cb9ad5640 1 librados: starting objecter 2024-06-03T09:06:42.387 INFO:teuthology.orchestra.run.smithi088.stderr:2024-06-03T09:06:42.383+0000 7f9cb9ad5640 1 Processor -- start 2024-06-03T09:06:42.387 INFO:teuthology.orchestra.run.smithi088.stderr:2024-06-03T09:06:42.383+0000 7f9cb9ad5640 1 -- start start 2024-06-03T09:06:42.387 INFO:teuthology.orchestra.run.smithi088.stderr:2024-06-03T09:06:42.383+0000 7f9cb9ad5640 1 librados: setting wanted keys 2024-06-03T09:06:42.387 INFO:teuthology.orchestra.run.smithi088.stderr:2024-06-03T09:06:42.383+0000 7f9cb9ad5640 1 librados: calling monclient init 2024-06-03T09:06:42.387 INFO:teuthology.orchestra.run.smithi088.stderr:2024-06-03T09:06:42.383+0000 7f9cb9ad5640 10 monclient: init 2024-06-03T09:06:42.387 INFO:teuthology.orchestra.run.smithi088.stderr:2024-06-03T09:06:42.384+0000 7f9cb9ad5640 10 monclient: _reopen_session rank -1 2024-06-03T09:06:42.388 INFO:teuthology.orchestra.run.smithi088.stderr:2024-06-03T09:06:42.384+0000 7f9cb9ad5640 10 monclient: _add_conns ranks=[0,2,1] 2024-06-03T09:06:42.388 INFO:teuthology.orchestra.run.smithi088.stderr:2024-06-03T09:06:42.384+0000 7f9cb9ad5640 10 monclient(hunting): picked mon.noname-a con 0x7f9cb40a8960 addr v1:172.21.15.88:6789/0 2024-06-03T09:06:42.388 INFO:teuthology.orchestra.run.smithi088.stderr:2024-06-03T09:06:42.384+0000 7f9cb9ad5640 10 monclient(hunting): picked mon.noname-c con 0x7f9cb413eec0 addr v1:172.21.15.88:6790/0 2024-06-03T09:06:42.388 INFO:teuthology.orchestra.run.smithi088.stderr:2024-06-03T09:06:42.384+0000 7f9cb9ad5640 10 monclient(hunting): picked mon.noname-b con 0x7f9cb41450f0 addr v1:172.21.15.192:6789/0 2024-06-03T09:06:42.388 INFO:teuthology.orchestra.run.smithi088.stderr:2024-06-03T09:06:42.384+0000 7f9cb9ad5640 1 -- --> v1:172.21.15.88:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f9cb40acfe0 con 0x7f9cb40a8960 2024-06-03T09:06:42.388 INFO:teuthology.orchestra.run.smithi088.stderr:2024-06-03T09:06:42.384+0000 7f9cb9ad5640 1 -- --> v1:172.21.15.192:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f9cb4142ab0 con 0x7f9cb41450f0 2024-06-03T09:06:42.388 INFO:teuthology.orchestra.run.smithi088.stderr:2024-06-03T09:06:42.384+0000 7f9cb9ad5640 1 -- --> v1:172.21.15.88:6790/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f9cb4142c40 con 0x7f9cb413eec0 2024-06-03T09:06:42.388 INFO:teuthology.orchestra.run.smithi088.stderr:2024-06-03T09:06:42.384+0000 7f9cb9ad5640 10 monclient(hunting): _renew_subs 2024-06-03T09:06:42.388 INFO:teuthology.orchestra.run.smithi088.stderr:2024-06-03T09:06:42.384+0000 7f9cb9ad5640 10 monclient(hunting): authenticate will time out at 3058.194304s 2024-06-03T09:06:42.388 INFO:teuthology.orchestra.run.smithi088.stderr:2024-06-03T09:06:42.384+0000 7f9cb37fe640 1 --1- >> v1:172.21.15.88:6789/0 conn(0x7f9cb40a8960 0x7f9cb40ac6f0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=1).handle_server_banner_and_identify peer v1:172.21.15.88:6789/0 says I am v1:172.21.15.88:54410/0 (socket says 172.21.15.88:54410) 2024-06-03T09:06:42.388 INFO:teuthology.orchestra.run.smithi088.stderr:2024-06-03T09:06:42.384+0000 7f9cb37fe640 1 -- 172.21.15.88:0/1057796131 learned_addr learned my addr 172.21.15.88:0/1057796131 (peer_addr_for_me v1:172.21.15.88:0/0) 2024-06-03T09:06:42.389 INFO:teuthology.orchestra.run.smithi088.stderr:2024-06-03T09:06:42.385+0000 7f9c9bfff640 10 client.?.objecter ms_handle_connect 0x7f9cb40a8960 2024-06-03T09:06:42.389 INFO:teuthology.orchestra.run.smithi088.stderr:2024-06-03T09:06:42.385+0000 7f9c9bfff640 10 client.?.objecter resend_mon_ops 2024-06-03T09:06:42.389 INFO:teuthology.orchestra.run.smithi088.stderr:2024-06-03T09:06:42.385+0000 7f9c9bfff640 10 client.?.objecter ms_handle_connect 0x7f9cb413eec0 2024-06-03T09:06:42.390 INFO:teuthology.orchestra.run.smithi088.stderr:2024-06-03T09:06:42.385+0000 7f9c9bfff640 10 client.?.objecter resend_mon_ops 2024-06-03T09:06:42.390 INFO:teuthology.orchestra.run.smithi088.stderr:2024-06-03T09:06:42.385+0000 7f9c9bfff640 10 client.?.objecter ms_handle_connect 0x7f9cb41450f0 2024-06-03T09:06:42.390 INFO:teuthology.orchestra.run.smithi088.stderr:2024-06-03T09:06:42.385+0000 7f9c9bfff640 10 client.?.objecter resend_mon_ops 2024-06-03T09:06:42.390 INFO:teuthology.orchestra.run.smithi088.stderr:2024-06-03T09:06:42.387+0000 7f9c9bfff640 1 -- 172.21.15.88:0/1057796131 <== mon.0 v1:172.21.15.88:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 2523177015 0 0) 0x7f9cb40acfe0 con 0x7f9cb40a8960 2024-06-03T09:06:42.390 INFO:teuthology.orchestra.run.smithi088.stderr:2024-06-03T09:06:42.387+0000 7f9c9bfff640 10 monclient(hunting): _negotiate 2024-06-03T09:06:42.390 INFO:teuthology.orchestra.run.smithi088.stderr:2024-06-03T09:06:42.387+0000 7f9c9bfff640 10 monclient(hunting): _init_auth method 2 2024-06-03T09:06:42.390 INFO:teuthology.orchestra.run.smithi088.stderr:2024-06-03T09:06:42.387+0000 7f9c9bfff640 10 monclient(hunting): _init_auth creating new auth 2024-06-03T09:06:42.390 INFO:teuthology.orchestra.run.smithi088.stderr:2024-06-03T09:06:42.387+0000 7f9c9bfff640 10 monclient(hunting): my global_id is 30675 2024-06-03T09:06:42.390 INFO:teuthology.orchestra.run.smithi088.stderr:2024-06-03T09:06:42.387+0000 7f9c9bfff640 1 -- 172.21.15.88:0/1057796131 --> v1:172.21.15.88:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f9ca0004eb0 con 0x7f9cb40a8960 2024-06-03T09:06:42.390 INFO:teuthology.orchestra.run.smithi088.stderr:2024-06-03T09:06:42.387+0000 7f9c9bfff640 1 -- 172.21.15.88:0/1057796131 <== mon.2 v1:172.21.15.88:6790/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 1302615021 0 0) 0x7f9cb4142c40 con 0x7f9cb413eec0 2024-06-03T09:06:42.390 INFO:teuthology.orchestra.run.smithi088.stderr:2024-06-03T09:06:42.387+0000 7f9c9bfff640 10 monclient(hunting): _negotiate 2024-06-03T09:06:42.390 INFO:teuthology.orchestra.run.smithi088.stderr:2024-06-03T09:06:42.387+0000 7f9c9bfff640 10 monclient(hunting): _init_auth method 2 2024-06-03T09:06:42.391 INFO:teuthology.orchestra.run.smithi088.stderr:2024-06-03T09:06:42.387+0000 7f9c9bfff640 10 monclient(hunting): _init_auth creating new auth 2024-06-03T09:06:42.391 INFO:teuthology.orchestra.run.smithi088.stderr:2024-06-03T09:06:42.387+0000 7f9c9bfff640 10 monclient(hunting): my global_id is 30179 2024-06-03T09:06:42.391 INFO:teuthology.orchestra.run.smithi088.stderr:2024-06-03T09:06:42.387+0000 7f9c9bfff640 1 -- 172.21.15.88:0/1057796131 --> v1:172.21.15.88:6790/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f9cb40acfe0 con 0x7f9cb413eec0 2024-06-03T09:06:42.391 INFO:teuthology.orchestra.run.smithi088.stderr:2024-06-03T09:06:42.387+0000 7f9c9bfff640 1 -- 172.21.15.88:0/1057796131 <== mon.1 v1:172.21.15.192:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 2086863543 0 0) 0x7f9cb4142ab0 con 0x7f9cb41450f0 2024-06-03T09:06:42.391 INFO:teuthology.orchestra.run.smithi088.stderr:2024-06-03T09:06:42.387+0000 7f9c9bfff640 10 monclient(hunting): _negotiate 2024-06-03T09:06:42.391 INFO:teuthology.orchestra.run.smithi088.stderr:2024-06-03T09:06:42.387+0000 7f9c9bfff640 10 monclient(hunting): _init_auth method 2 2024-06-03T09:06:42.391 INFO:teuthology.orchestra.run.smithi088.stderr:2024-06-03T09:06:42.387+0000 7f9c9bfff640 10 monclient(hunting): _init_auth creating new auth 2024-06-03T09:06:42.391 INFO:teuthology.orchestra.run.smithi088.stderr:2024-06-03T09:06:42.387+0000 7f9c9bfff640 10 monclient(hunting): my global_id is 30253 2024-06-03T09:06:42.391 INFO:teuthology.orchestra.run.smithi088.stderr:2024-06-03T09:06:42.387+0000 7f9c9bfff640 1 -- 172.21.15.88:0/1057796131 --> v1:172.21.15.192:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f9cb4142c40 con 0x7f9cb41450f0 2024-06-03T09:06:42.392 INFO:teuthology.orchestra.run.smithi088.stderr:2024-06-03T09:06:42.392+0000 7f9c9bfff640 1 -- 172.21.15.88:0/1057796131 <== mon.1 v1:172.21.15.192:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 764+0+0 (unknown 1311371333 0 0) 0x7f9cb4142c40 con 0x7f9cb41450f0 2024-06-03T09:06:42.392 INFO:teuthology.orchestra.run.smithi088.stderr:2024-06-03T09:06:42.392+0000 7f9c9bfff640 1 -- 172.21.15.88:0/1057796131 --> v1:172.21.15.192:6789/0 -- auth(proto 2 165 bytes epoch 0) v1 -- 0x7f9cb4142ab0 con 0x7f9cb41450f0 2024-06-03T09:06:42.392 INFO:teuthology.orchestra.run.smithi088.stderr:2024-06-03T09:06:42.392+0000 7f9c9bfff640 1 -- 172.21.15.88:0/1057796131 <== mon.1 v1:172.21.15.192:6789/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 2234374065 0 0) 0x7f9cac001c50 con 0x7f9cb41450f0 2024-06-03T09:06:42.393 INFO:teuthology.orchestra.run.smithi088.stderr:2024-06-03T09:06:42.392+0000 7f9c9bfff640 10 monclient(hunting): handle_monmap mon_map magic: 0 v1 2024-06-03T09:06:42.393 INFO:teuthology.orchestra.run.smithi088.stderr:2024-06-03T09:06:42.392+0000 7f9c9bfff640 10 monclient(hunting): got monmap 1 from mon.noname-b (according to old e1) 2024-06-03T09:06:42.393 INFO:teuthology.orchestra.run.smithi088.stderr:2024-06-03T09:06:42.392+0000 7f9c9bfff640 10 monclient(hunting): dump: 2024-06-03T09:06:42.393 INFO:teuthology.orchestra.run.smithi088.stderr:epoch 1 2024-06-03T09:06:42.393 INFO:teuthology.orchestra.run.smithi088.stderr:fsid ec3c3fdd-3ac6-4d7d-9412-683130ed8b04 2024-06-03T09:06:42.393 INFO:teuthology.orchestra.run.smithi088.stderr:last_changed 2024-06-03T08:34:31.885125+0000 2024-06-03T09:06:42.393 INFO:teuthology.orchestra.run.smithi088.stderr:created 2024-06-03T08:34:31.885125+0000 2024-06-03T09:06:42.393 INFO:teuthology.orchestra.run.smithi088.stderr:min_mon_release 18 (reef) 2024-06-03T09:06:42.393 INFO:teuthology.orchestra.run.smithi088.stderr:election_strategy: 3 2024-06-03T09:06:42.393 INFO:teuthology.orchestra.run.smithi088.stderr:0: v1:172.21.15.88:6789/0 mon.a 2024-06-03T09:06:42.393 INFO:teuthology.orchestra.run.smithi088.stderr:1: v1:172.21.15.192:6789/0 mon.b 2024-06-03T09:06:42.393 INFO:teuthology.orchestra.run.smithi088.stderr:2: v1:172.21.15.88:6790/0 mon.c 2024-06-03T09:06:42.393 INFO:teuthology.orchestra.run.smithi088.stderr: 2024-06-03T09:06:42.393 INFO:teuthology.orchestra.run.smithi088.stderr:2024-06-03T09:06:42.392+0000 7f9c9bfff640 10 monclient(hunting): _finish_auth 0 2024-06-03T09:06:42.394 INFO:teuthology.orchestra.run.smithi088.stderr:2024-06-03T09:06:42.392+0000 7f9c9bfff640 1 -- 172.21.15.88:0/1057796131 <== mon.2 v1:172.21.15.88:6790/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 764+0+0 (unknown 2654552567 0 0) 0x7f9cb40acfe0 con 0x7f9cb413eec0 2024-06-03T09:06:42.394 INFO:teuthology.orchestra.run.smithi088.stderr:2024-06-03T09:06:42.392+0000 7f9c9bfff640 1 -- 172.21.15.88:0/1057796131 --> v1:172.21.15.88:6790/0 -- auth(proto 2 165 bytes epoch 0) v1 -- 0x7f9cb4142c40 con 0x7f9cb413eec0 2024-06-03T09:06:42.394 INFO:teuthology.orchestra.run.smithi088.stderr:2024-06-03T09:06:42.392+0000 7f9c9bfff640 1 -- 172.21.15.88:0/1057796131 <== mon.0 v1:172.21.15.88:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 764+0+0 (unknown 3951549656 0 0) 0x7f9ca0004eb0 con 0x7f9cb40a8960 2024-06-03T09:06:42.394 INFO:teuthology.orchestra.run.smithi088.stderr:2024-06-03T09:06:42.392+0000 7f9c9bfff640 1 -- 172.21.15.88:0/1057796131 --> v1:172.21.15.88:6789/0 -- auth(proto 2 165 bytes epoch 0) v1 -- 0x7f9cb40acfe0 con 0x7f9cb40a8960 2024-06-03T09:06:42.394 INFO:teuthology.orchestra.run.smithi088.stderr:2024-06-03T09:06:42.392+0000 7f9c9bfff640 1 -- 172.21.15.88:0/1057796131 <== mon.2 v1:172.21.15.88:6790/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 2234374065 0 0) 0x7f9ca8001750 con 0x7f9cb413eec0 2024-06-03T09:06:42.394 INFO:teuthology.orchestra.run.smithi088.stderr:2024-06-03T09:06:42.392+0000 7f9c9bfff640 10 monclient(hunting): handle_monmap mon_map magic: 0 v1 2024-06-03T09:06:42.394 INFO:teuthology.orchestra.run.smithi088.stderr:2024-06-03T09:06:42.392+0000 7f9c9bfff640 10 monclient(hunting): got monmap 1 from mon.c (according to old e1) 2024-06-03T09:06:42.394 INFO:teuthology.orchestra.run.smithi088.stderr:2024-06-03T09:06:42.392+0000 7f9c9bfff640 10 monclient(hunting): dump: 2024-06-03T09:06:42.394 INFO:teuthology.orchestra.run.smithi088.stderr:epoch 1 2024-06-03T09:06:42.394 INFO:teuthology.orchestra.run.smithi088.stderr:fsid ec3c3fdd-3ac6-4d7d-9412-683130ed8b04 2024-06-03T09:06:42.394 INFO:teuthology.orchestra.run.smithi088.stderr:last_changed 2024-06-03T08:34:31.885125+0000 2024-06-03T09:06:42.394 INFO:teuthology.orchestra.run.smithi088.stderr:created 2024-06-03T08:34:31.885125+0000 2024-06-03T09:06:42.394 INFO:teuthology.orchestra.run.smithi088.stderr:min_mon_release 18 (reef) 2024-06-03T09:06:42.395 INFO:teuthology.orchestra.run.smithi088.stderr:election_strategy: 3 2024-06-03T09:06:42.395 INFO:teuthology.orchestra.run.smithi088.stderr:0: v1:172.21.15.88:6789/0 mon.a 2024-06-03T09:06:42.395 INFO:teuthology.orchestra.run.smithi088.stderr:1: v1:172.21.15.192:6789/0 mon.b 2024-06-03T09:06:42.395 INFO:teuthology.orchestra.run.smithi088.stderr:2: v1:172.21.15.88:6790/0 mon.c 2024-06-03T09:06:42.395 INFO:teuthology.orchestra.run.smithi088.stderr: 2024-06-03T09:06:42.395 INFO:teuthology.orchestra.run.smithi088.stderr:2024-06-03T09:06:42.392+0000 7f9c9bfff640 1 -- 172.21.15.88:0/1057796131 <== mon.0 v1:172.21.15.88:6789/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 2234374065 0 0) 0x7f9ca40014f0 con 0x7f9cb40a8960 2024-06-03T09:06:42.395 INFO:teuthology.orchestra.run.smithi088.stderr:2024-06-03T09:06:42.392+0000 7f9c9bfff640 10 monclient(hunting): handle_monmap mon_map magic: 0 v1 2024-06-03T09:06:42.395 INFO:teuthology.orchestra.run.smithi088.stderr:2024-06-03T09:06:42.392+0000 7f9c9bfff640 10 monclient(hunting): got monmap 1 from mon.a (according to old e1) 2024-06-03T09:06:42.395 INFO:teuthology.orchestra.run.smithi088.stderr:2024-06-03T09:06:42.392+0000 7f9c9bfff640 10 monclient(hunting): dump: 2024-06-03T09:06:42.395 INFO:teuthology.orchestra.run.smithi088.stderr:epoch 1 2024-06-03T09:06:42.395 INFO:teuthology.orchestra.run.smithi088.stderr:fsid ec3c3fdd-3ac6-4d7d-9412-683130ed8b04 2024-06-03T09:06:42.395 INFO:teuthology.orchestra.run.smithi088.stderr:last_changed 2024-06-03T08:34:31.885125+0000 2024-06-03T09:06:42.395 INFO:teuthology.orchestra.run.smithi088.stderr:created 2024-06-03T08:34:31.885125+0000 2024-06-03T09:06:42.395 INFO:teuthology.orchestra.run.smithi088.stderr:min_mon_release 18 (reef) 2024-06-03T09:06:42.396 INFO:teuthology.orchestra.run.smithi088.stderr:election_strategy: 3 2024-06-03T09:06:42.396 INFO:teuthology.orchestra.run.smithi088.stderr:0: v1:172.21.15.88:6789/0 mon.a 2024-06-03T09:06:42.396 INFO:teuthology.orchestra.run.smithi088.stderr:1: v1:172.21.15.192:6789/0 mon.b 2024-06-03T09:06:42.396 INFO:teuthology.orchestra.run.smithi088.stderr:2: v1:172.21.15.88:6790/0 mon.c 2024-06-03T09:06:42.396 INFO:teuthology.orchestra.run.smithi088.stderr: 2024-06-03T09:06:42.397 INFO:teuthology.orchestra.run.smithi088.stderr:2024-06-03T09:06:42.394+0000 7f9c9bfff640 1 -- 172.21.15.88:0/1057796131 <== mon.2 v1:172.21.15.88:6790/0 4 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 580+0+0 (unknown 985272856 0 0) 0x7f9cb4142c40 con 0x7f9cb413eec0 2024-06-03T09:06:42.397 INFO:teuthology.orchestra.run.smithi088.stderr:2024-06-03T09:06:42.394+0000 7f9c9bfff640 1 -- 172.21.15.88:0/1057796131 >> v1:172.21.15.192:6789/0 conn(0x7f9cb41450f0 legacy=0x7f9cb41e0ea0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-06-03T09:06:42.397 INFO:teuthology.orchestra.run.smithi088.stderr:2024-06-03T09:06:42.394+0000 7f9c9bfff640 1 -- 172.21.15.88:0/1057796131 >> v1:172.21.15.88:6789/0 conn(0x7f9cb40a8960 legacy=0x7f9cb40ac6f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-06-03T09:06:42.397 INFO:teuthology.orchestra.run.smithi088.stderr:2024-06-03T09:06:42.394+0000 7f9c9bfff640 10 monclient: _finish_hunting 0 2024-06-03T09:06:42.397 INFO:teuthology.orchestra.run.smithi088.stderr:2024-06-03T09:06:42.394+0000 7f9c9bfff640 1 monclient: found mon.c 2024-06-03T09:06:42.397 INFO:teuthology.orchestra.run.smithi088.stderr:2024-06-03T09:06:42.394+0000 7f9c9bfff640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-06-03T09:06:42.397 INFO:teuthology.orchestra.run.smithi088.stderr:2024-06-03T09:06:42.394+0000 7f9c9bfff640 10 monclient: _send_mon_message to mon.c at v1:172.21.15.88:6790/0 2024-06-03T09:06:42.397 INFO:teuthology.orchestra.run.smithi088.stderr:2024-06-03T09:06:42.394+0000 7f9c9bfff640 1 -- 172.21.15.88:0/1057796131 --> v1:172.21.15.88:6790/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f9cb40ac150 con 0x7f9cb413eec0 2024-06-03T09:06:42.398 INFO:teuthology.orchestra.run.smithi088.stderr:2024-06-03T09:06:42.394+0000 7f9c9bfff640 10 monclient: _finish_auth 0 2024-06-03T09:06:42.398 INFO:teuthology.orchestra.run.smithi088.stderr:2024-06-03T09:06:42.394+0000 7f9c9bfff640 10 monclient: _check_auth_tickets 2024-06-03T09:06:42.398 INFO:teuthology.orchestra.run.smithi088.stderr:2024-06-03T09:06:42.394+0000 7f9c9bfff640 20 monclient: _check_auth_rotating not needed by client.admin 2024-06-03T09:06:42.398 INFO:teuthology.orchestra.run.smithi088.stderr:2024-06-03T09:06:42.394+0000 7f9cb9ad5640 5 monclient: authenticate success, global_id 30179 2024-06-03T09:06:42.398 INFO:teuthology.orchestra.run.smithi088.stderr:2024-06-03T09:06:42.394+0000 7f9cb9ad5640 10 monclient: _renew_subs 2024-06-03T09:06:42.398 INFO:teuthology.orchestra.run.smithi088.stderr:2024-06-03T09:06:42.394+0000 7f9cb9ad5640 10 monclient: _send_mon_message to mon.c at v1:172.21.15.88:6790/0 2024-06-03T09:06:42.398 INFO:teuthology.orchestra.run.smithi088.stderr:2024-06-03T09:06:42.394+0000 7f9cb9ad5640 1 -- 172.21.15.88:0/1057796131 --> v1:172.21.15.88:6790/0 -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f9cb40ac400 con 0x7f9cb413eec0 2024-06-03T09:06:42.398 INFO:teuthology.orchestra.run.smithi088.stderr:2024-06-03T09:06:42.394+0000 7f9cb9ad5640 10 client.30179.objecter _maybe_request_map subscribing (onetime) to next osd map 2024-06-03T09:06:42.398 INFO:teuthology.orchestra.run.smithi088.stderr:2024-06-03T09:06:42.394+0000 7f9cb9ad5640 10 monclient: _renew_subs 2024-06-03T09:06:42.398 INFO:teuthology.orchestra.run.smithi088.stderr:2024-06-03T09:06:42.394+0000 7f9cb9ad5640 10 monclient: _send_mon_message to mon.c at v1:172.21.15.88:6790/0 2024-06-03T09:06:42.398 INFO:teuthology.orchestra.run.smithi088.stderr:2024-06-03T09:06:42.394+0000 7f9cb9ad5640 1 -- 172.21.15.88:0/1057796131 --> v1:172.21.15.88:6790/0 -- mon_subscribe({osdmap=0}) v3 -- 0x7f9cb40ab460 con 0x7f9cb413eec0 2024-06-03T09:06:42.398 INFO:teuthology.orchestra.run.smithi088.stderr:2024-06-03T09:06:42.394+0000 7f9cb9ad5640 1 librados: init done 2024-06-03T09:06:42.398 INFO:teuthology.orchestra.run.smithi088.stderr:2024-06-03T09:06:42.395+0000 7f9cb9ad5640 10 start_mon_command cmd=[{"prefix": "get_command_descriptions"}] 2024-06-03T09:06:42.399 INFO:teuthology.orchestra.run.smithi088.stderr:2024-06-03T09:06:42.395+0000 7f9cb9ad5640 10 monclient: _send_command 1 [{"prefix": "get_command_descriptions"}] 2024-06-03T09:06:42.399 INFO:teuthology.orchestra.run.smithi088.stderr:2024-06-03T09:06:42.395+0000 7f9cb9ad5640 10 monclient: _send_mon_message to mon.c at v1:172.21.15.88:6790/0 2024-06-03T09:06:42.399 INFO:teuthology.orchestra.run.smithi088.stderr:2024-06-03T09:06:42.395+0000 7f9cb9ad5640 1 -- 172.21.15.88:0/1057796131 --> v1:172.21.15.88:6790/0 -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f9cb40a9e60 con 0x7f9cb413eec0 2024-06-03T09:06:42.643 INFO:teuthology.orchestra.run.smithi088.stderr:2024-06-03T09:06:42.643+0000 7f9c9bfff640 1 -- 172.21.15.88:0/1057796131 <== mon.2 v1:172.21.15.88:6790/0 5 ==== config(0 keys) v1 ==== 4+0+0 (unknown 0 0 0) 0x7f9ca8001ce0 con 0x7f9cb413eec0 2024-06-03T09:06:42.643 INFO:teuthology.orchestra.run.smithi088.stderr:2024-06-03T09:06:42.643+0000 7f9c9bfff640 10 monclient: handle_config config(0 keys) v1 2024-06-03T09:06:42.643 INFO:teuthology.orchestra.run.smithi088.stderr:2024-06-03T09:06:42.643+0000 7f9c9bfff640 1 -- 172.21.15.88:0/1057796131 <== mon.2 v1:172.21.15.88:6790/0 6 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 2234374065 0 0) 0x7f9ca8004980 con 0x7f9cb413eec0 2024-06-03T09:06:42.644 INFO:teuthology.orchestra.run.smithi088.stderr:2024-06-03T09:06:42.643+0000 7f9c9bfff640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-06-03T09:06:42.644 INFO:teuthology.orchestra.run.smithi088.stderr:2024-06-03T09:06:42.643+0000 7f9c9bfff640 10 monclient: got monmap 1 from mon.c (according to old e1) 2024-06-03T09:06:42.644 INFO:teuthology.orchestra.run.smithi088.stderr:2024-06-03T09:06:42.643+0000 7f9c9bfff640 10 monclient: dump: 2024-06-03T09:06:42.644 INFO:teuthology.orchestra.run.smithi088.stderr:epoch 1 2024-06-03T09:06:42.644 INFO:teuthology.orchestra.run.smithi088.stderr:fsid ec3c3fdd-3ac6-4d7d-9412-683130ed8b04 2024-06-03T09:06:42.644 INFO:teuthology.orchestra.run.smithi088.stderr:last_changed 2024-06-03T08:34:31.885125+0000 2024-06-03T09:06:42.644 INFO:teuthology.orchestra.run.smithi088.stderr:created 2024-06-03T08:34:31.885125+0000 2024-06-03T09:06:42.644 INFO:teuthology.orchestra.run.smithi088.stderr:min_mon_release 18 (reef) 2024-06-03T09:06:42.644 INFO:teuthology.orchestra.run.smithi088.stderr:election_strategy: 3 2024-06-03T09:06:42.644 INFO:teuthology.orchestra.run.smithi088.stderr:0: v1:172.21.15.88:6789/0 mon.a 2024-06-03T09:06:42.644 INFO:teuthology.orchestra.run.smithi088.stderr:1: v1:172.21.15.192:6789/0 mon.b 2024-06-03T09:06:42.644 INFO:teuthology.orchestra.run.smithi088.stderr:2: v1:172.21.15.88:6790/0 mon.c 2024-06-03T09:06:42.644 INFO:teuthology.orchestra.run.smithi088.stderr: 2024-06-03T09:06:42.645 INFO:teuthology.orchestra.run.smithi088.stderr:2024-06-03T09:06:42.643+0000 7f9cb1ffb640 4 set_mon_vals no callback set 2024-06-03T09:06:42.645 INFO:teuthology.orchestra.run.smithi088.stderr:2024-06-03T09:06:42.643+0000 7f9c9bfff640 1 -- 172.21.15.88:0/1057796131 <== mon.2 v1:172.21.15.88:6790/0 7 ==== mgrmap(e 5) v1 ==== 89110+0+0 (unknown 873533973 0 0) 0x7f9ca801a700 con 0x7f9cb413eec0 2024-06-03T09:06:42.645 INFO:teuthology.orchestra.run.smithi088.stderr:2024-06-03T09:06:42.644+0000 7f9cb37fe640 10 monclient: get_auth_request con 0x7f9ca006d4c0 auth_method 0 2024-06-03T09:06:42.645 INFO:teuthology.orchestra.run.smithi088.stderr:2024-06-03T09:06:42.644+0000 7f9c9bfff640 10 client.30179.objecter ms_handle_connect 0x7f9ca006d4c0 2024-06-03T09:06:42.648 INFO:teuthology.orchestra.run.smithi088.stderr:2024-06-03T09:06:42.648+0000 7f9c9bfff640 1 -- 172.21.15.88:0/1057796131 <== mon.2 v1:172.21.15.88:6790/0 8 ==== osd_map(1010..1010 src has 622..1010) v4 ==== 5229+0+0 (unknown 1105883993 0 0) 0x7f9ca8053e90 con 0x7f9cb413eec0 2024-06-03T09:06:42.648 INFO:teuthology.orchestra.run.smithi088.stderr:2024-06-03T09:06:42.648+0000 7f9c9bfff640 10 client.30179.objecter ms_dispatch 0x7f9cb4000b90 osd_map(1010..1010 src has 622..1010) v4 2024-06-03T09:06:42.648 INFO:teuthology.orchestra.run.smithi088.stderr:2024-06-03T09:06:42.648+0000 7f9c9bfff640 3 client.30179.objecter handle_osd_map got epochs [1010,1010] > 0 2024-06-03T09:06:42.648 INFO:teuthology.orchestra.run.smithi088.stderr:2024-06-03T09:06:42.648+0000 7f9c9bfff640 3 client.30179.objecter handle_osd_map decoding full epoch 1010 2024-06-03T09:06:42.649 INFO:teuthology.orchestra.run.smithi088.stderr:2024-06-03T09:06:42.648+0000 7f9c9bfff640 20 client.30179.objecter dump_active .. 0 homeless 2024-06-03T09:06:42.649 INFO:teuthology.orchestra.run.smithi088.stderr:2024-06-03T09:06:42.648+0000 7f9c9bfff640 1 -- 172.21.15.88:0/1057796131 <== mon.2 v1:172.21.15.88:6790/0 9 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+182498 (unknown 1092875540 0 3016760700) 0x7f9ca804fa40 con 0x7f9cb413eec0 2024-06-03T09:06:42.649 INFO:teuthology.orchestra.run.smithi088.stderr:2024-06-03T09:06:42.648+0000 7f9c9bfff640 10 monclient: handle_mon_command_ack 1 [{"prefix": "get_command_descriptions"}] 2024-06-03T09:06:42.649 INFO:teuthology.orchestra.run.smithi088.stderr:2024-06-03T09:06:42.648+0000 7f9c9bfff640 10 monclient: _finish_command 1 = system:0 2024-06-03T09:06:42.854 INFO:teuthology.orchestra.run.smithi088.stderr:2024-06-03T09:06:42.854+0000 7f9cb9ad5640 1 -- 172.21.15.88:0/1057796131 --> v1:172.21.15.192:6800/34229 -- mgr_command(tid 0: {"prefix": "pg dump", "target": ["mon-mgr", ""], "format": "json"}) v1 -- 0x7f9cb40ab710 con 0x7f9ca006d4c0 2024-06-03T09:06:42.859 INFO:teuthology.orchestra.run.smithi088.stderr:2024-06-03T09:06:42.859+0000 7f9c9bfff640 1 -- 172.21.15.88:0/1057796131 <== mgr.4110 v1:172.21.15.192:6800/34229 1 ==== mgr_command_reply(tid 0: 0 dumped all) v1 ==== 18+0+151537 (unknown 2965378022 0 1086612447) 0x7f9cb40ab710 con 0x7f9ca006d4c0 2024-06-03T09:06:42.860 INFO:teuthology.orchestra.run.smithi088.stdout: 2024-06-03T09:06:42.861 INFO:teuthology.orchestra.run.smithi088.stderr:dumped all 2024-06-03T09:06:42.863 INFO:teuthology.orchestra.run.smithi088.stderr:2024-06-03T09:06:42.861+0000 7f9cb9ad5640 10 librados: watch_flush enter 2024-06-03T09:06:42.863 INFO:teuthology.orchestra.run.smithi088.stderr:2024-06-03T09:06:42.861+0000 7f9cb9ad5640 10 librados: watch_flush exit 2024-06-03T09:06:42.863 INFO:teuthology.orchestra.run.smithi088.stderr:2024-06-03T09:06:42.861+0000 7f9cb9ad5640 20 client.30179.objecter shutdown clearing up homeless session... 2024-06-03T09:06:42.863 INFO:teuthology.orchestra.run.smithi088.stderr:2024-06-03T09:06:42.861+0000 7f9cb9ad5640 10 client.30179.objecter successfully canceled tick 2024-06-03T09:06:42.863 INFO:teuthology.orchestra.run.smithi088.stderr:2024-06-03T09:06:42.861+0000 7f9cb9ad5640 1 -- 172.21.15.88:0/1057796131 >> v1:172.21.15.192:6800/34229 conn(0x7f9ca006d4c0 legacy=0x7f9ca006f980 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-06-03T09:06:42.863 INFO:teuthology.orchestra.run.smithi088.stderr:2024-06-03T09:06:42.861+0000 7f9cb9ad5640 10 monclient: shutdown 2024-06-03T09:06:42.863 INFO:teuthology.orchestra.run.smithi088.stderr:2024-06-03T09:06:42.861+0000 7f9cb9ad5640 20 monclient: shutdown discarding 0 pending message(s) 2024-06-03T09:06:42.863 INFO:teuthology.orchestra.run.smithi088.stderr:2024-06-03T09:06:42.861+0000 7f9cb9ad5640 1 -- 172.21.15.88:0/1057796131 >> v1:172.21.15.88:6790/0 conn(0x7f9cb413eec0 legacy=0x7f9cb40aab70 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-06-03T09:06:42.863 INFO:teuthology.orchestra.run.smithi088.stderr:2024-06-03T09:06:42.861+0000 7f9cb9ad5640 1 -- 172.21.15.88:0/1057796131 shutdown_connections 2024-06-03T09:06:42.863 INFO:teuthology.orchestra.run.smithi088.stderr:2024-06-03T09:06:42.861+0000 7f9cb9ad5640 1 -- 172.21.15.88:0/1057796131 >> 172.21.15.88:0/1057796131 conn(0x7f9cb40b70d0 msgr2=0x7f9cb413e580 unknown :-1 s=STATE_NONE l=0).mark_down 2024-06-03T09:06:42.864 INFO:teuthology.orchestra.run.smithi088.stderr:2024-06-03T09:06:42.862+0000 7f9cb9ad5640 1 -- 172.21.15.88:0/1057796131 shutdown_connections 2024-06-03T09:06:42.864 INFO:teuthology.orchestra.run.smithi088.stderr:2024-06-03T09:06:42.862+0000 7f9cb9ad5640 1 -- 172.21.15.88:0/1057796131 wait complete. 2024-06-03T09:06:42.864 INFO:teuthology.orchestra.run.smithi088.stderr:2024-06-03T09:06:42.862+0000 7f9cb9ad5640 1 librados: shutdown 2024-06-03T09:06:42.873 INFO:teuthology.orchestra.run.smithi088.stdout:{"pg_ready":true,"pg_map":{"version":1927,"stamp":"2024-06-03T09:06:42.834123+0000","last_osdmap_epoch":0,"last_pg_scan":0,"pg_stats_sum":{"stat_sum":{"num_bytes":459285,"num_objects":3,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":1,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":3,"num_whiteouts":0,"num_read":102,"num_read_kb":88,"num_write":120,"num_write_kb":1379,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2,"num_bytes_recovered":397840,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":83,"ondisk_log_size":83,"up":68,"acting":68,"num_store_stats":0},"osd_stats_sum":{"up_from":0,"seq":0,"num_pgs":88,"num_osds":8,"num_per_pool_osds":8,"num_per_pool_omap_osds":8,"kb":754974720,"kb_used":494612,"kb_used_data":46388,"kb_used_omap":12,"kb_used_meta":448115,"kb_avail":754480108,"statfs":{"total":773094113280,"available":772587630592,"internally_reserved":0,"allocated":47501312,"data_stored":22782476,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":12744,"internal_metadata":458870328},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":93,"apply_latency_ms":93,"commit_latency_ns":93000000,"apply_latency_ns":93000000},"alerts":[],"network_ping_times":[]},"pg_stats_delta":{"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":0,"acting":0,"num_store_stats":0,"stamp_delta":"11.733482"},"pg_stats":[{"pgid":"17.d","version":"0'0","reported_seq":942,"reported_epoch":1009,"state":"active+clean","last_fresh":"2024-06-03T09:06:14.443917+0000","last_change":"2024-06-03T08:38:19.260548+0000","last_active":"2024-06-03T09:06:14.443917+0000","last_peered":"2024-06-03T09:06:14.443917+0000","last_clean":"2024-06-03T09:06:14.443917+0000","last_became_active":"2024-06-03T08:38:19.253559+0000","last_became_peered":"2024-06-03T08:38:19.253559+0000","last_unstale":"2024-06-03T09:06:14.443917+0000","last_undegraded":"2024-06-03T09:06:14.443917+0000","last_fullsized":"2024-06-03T09:06:14.443917+0000","mapping_epoch":28,"log_start":"0'0","ondisk_log_start":"0'0","created":28,"last_epoch_clean":45,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-06-03T08:37:55.614497+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-06-03T08:37:55.614497+0000","last_clean_scrub_stamp":"2024-06-03T08:37:55.614497+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-06-04T19:00:12.470979+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1],"acting":[0,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"17.c","version":"0'0","reported_seq":953,"reported_epoch":1009,"state":"active+undersized","last_fresh":"2024-06-03T09:06:13.668156+0000","last_change":"2024-06-03T09:06:03.440074+0000","last_active":"2024-06-03T09:06:13.668156+0000","last_peered":"2024-06-03T09:06:13.668156+0000","last_clean":"2024-06-03T09:05:02.105638+0000","last_became_active":"2024-06-03T09:06:03.440074+0000","last_became_peered":"2024-06-03T09:06:03.440074+0000","last_unstale":"2024-06-03T09:06:13.668156+0000","last_undegraded":"2024-06-03T09:06:13.668156+0000","last_fullsized":"2024-06-03T09:06:03.377982+0000","mapping_epoch":999,"log_start":"0'0","ondisk_log_start":"0'0","created":28,"last_epoch_clean":255,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-06-03T08:37:55.614497+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-06-03T08:37:55.614497+0000","last_clean_scrub_stamp":"2024-06-03T08:37:55.614497+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-06-04T10:34:20.808557+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3],"acting":[3],"avail_no_missing":["3"],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"17.f","version":"0'0","reported_seq":148,"reported_epoch":1009,"state":"active+undersized","last_fresh":"2024-06-03T09:06:14.450543+0000","last_change":"2024-06-03T09:06:03.420285+0000","last_active":"2024-06-03T09:06:14.450543+0000","last_peered":"2024-06-03T09:06:14.450543+0000","last_clean":"2024-06-03T08:37:55.614497+0000","last_became_active":"2024-06-03T09:06:03.420285+0000","last_became_peered":"2024-06-03T09:06:03.420285+0000","last_unstale":"2024-06-03T09:06:14.450543+0000","last_undegraded":"2024-06-03T09:06:14.450543+0000","last_fullsized":"2024-06-03T09:06:03.380616+0000","mapping_epoch":999,"log_start":"0'0","ondisk_log_start":"0'0","created":28,"last_epoch_clean":255,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-06-03T08:37:55.614497+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-06-03T08:37:55.614497+0000","last_clean_scrub_stamp":"2024-06-03T08:37:55.614497+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-06-04T15:39:41.187731+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4],"acting":[4],"avail_no_missing":["4"],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"17.e","version":"0'0","reported_seq":149,"reported_epoch":1009,"state":"active+undersized","last_fresh":"2024-06-03T09:06:13.726992+0000","last_change":"2024-06-03T09:06:03.412887+0000","last_active":"2024-06-03T09:06:13.726992+0000","last_peered":"2024-06-03T09:06:13.726992+0000","last_clean":"2024-06-03T08:37:55.614497+0000","last_became_active":"2024-06-03T09:06:03.412887+0000","last_became_peered":"2024-06-03T09:06:03.412887+0000","last_unstale":"2024-06-03T09:06:13.726992+0000","last_undegraded":"2024-06-03T09:06:13.726992+0000","last_fullsized":"2024-06-03T09:06:03.332482+0000","mapping_epoch":999,"log_start":"0'0","ondisk_log_start":"0'0","created":28,"last_epoch_clean":255,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-06-03T08:37:55.614497+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-06-03T08:37:55.614497+0000","last_clean_scrub_stamp":"2024-06-03T08:37:55.614497+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-06-04T12:23:22.511601+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5],"acting":[5],"avail_no_missing":["5"],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"17.9","version":"0'0","reported_seq":946,"reported_epoch":1009,"state":"active+clean","last_fresh":"2024-06-03T09:06:14.999299+0000","last_change":"2024-06-03T08:38:04.772260+0000","last_active":"2024-06-03T09:06:14.999299+0000","last_peered":"2024-06-03T09:06:14.999299+0000","last_clean":"2024-06-03T09:06:14.999299+0000","last_became_active":"2024-06-03T08:38:04.763428+0000","last_became_peered":"2024-06-03T08:38:04.763428+0000","last_unstale":"2024-06-03T09:06:14.999299+0000","last_undegraded":"2024-06-03T09:06:14.999299+0000","last_fullsized":"2024-06-03T09:06:14.999299+0000","mapping_epoch":28,"log_start":"0'0","ondisk_log_start":"0'0","created":28,"last_epoch_clean":33,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-06-03T08:37:55.614497+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-06-03T08:37:55.614497+0000","last_clean_scrub_stamp":"2024-06-03T08:37:55.614497+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-06-04T10:55:10.773936+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,4],"acting":[7,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"17.8","version":"0'0","reported_seq":961,"reported_epoch":1009,"state":"active+undersized","last_fresh":"2024-06-03T09:06:14.430375+0000","last_change":"2024-06-03T09:06:03.388786+0000","last_active":"2024-06-03T09:06:14.430375+0000","last_peered":"2024-06-03T09:06:14.430375+0000","last_clean":"2024-06-03T09:04:43.297780+0000","last_became_active":"2024-06-03T09:06:03.388786+0000","last_became_peered":"2024-06-03T09:06:03.388786+0000","last_unstale":"2024-06-03T09:06:14.430375+0000","last_undegraded":"2024-06-03T09:06:14.430375+0000","last_fullsized":"2024-06-03T09:06:03.331754+0000","mapping_epoch":999,"log_start":"0'0","ondisk_log_start":"0'0","created":28,"last_epoch_clean":255,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-06-03T08:37:55.614497+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-06-03T08:37:55.614497+0000","last_clean_scrub_stamp":"2024-06-03T08:37:55.614497+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-06-04T09:18:44.765547+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0],"acting":[0],"avail_no_missing":["0"],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"17.b","version":"0'0","reported_seq":149,"reported_epoch":1009,"state":"active+undersized","last_fresh":"2024-06-03T09:06:14.433735+0000","last_change":"2024-06-03T09:06:03.451203+0000","last_active":"2024-06-03T09:06:14.433735+0000","last_peered":"2024-06-03T09:06:14.433735+0000","last_clean":"2024-06-03T08:37:55.614497+0000","last_became_active":"2024-06-03T09:06:03.451203+0000","last_became_peered":"2024-06-03T09:06:03.451203+0000","last_unstale":"2024-06-03T09:06:14.433735+0000","last_undegraded":"2024-06-03T09:06:14.433735+0000","last_fullsized":"2024-06-03T09:06:03.375523+0000","mapping_epoch":999,"log_start":"0'0","ondisk_log_start":"0'0","created":28,"last_epoch_clean":255,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-06-03T08:37:55.614497+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-06-03T08:37:55.614497+0000","last_clean_scrub_stamp":"2024-06-03T08:37:55.614497+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-06-04T10:16:19.813495+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2],"acting":[2],"avail_no_missing":["2"],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"17.a","version":"0'0","reported_seq":925,"reported_epoch":1010,"state":"active+clean","last_fresh":"2024-06-03T09:06:18.651697+0000","last_change":"2024-06-03T08:38:05.843142+0000","last_active":"2024-06-03T09:06:18.651697+0000","last_peered":"2024-06-03T09:06:18.651697+0000","last_clean":"2024-06-03T09:06:18.651697+0000","last_became_active":"2024-06-03T08:38:05.809257+0000","last_became_peered":"2024-06-03T08:38:05.809257+0000","last_unstale":"2024-06-03T09:06:18.651697+0000","last_undegraded":"2024-06-03T09:06:18.651697+0000","last_fullsized":"2024-06-03T09:06:18.651697+0000","mapping_epoch":28,"log_start":"0'0","ondisk_log_start":"0'0","created":28,"last_epoch_clean":33,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-06-03T08:37:55.614497+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-06-03T08:37:55.614497+0000","last_clean_scrub_stamp":"2024-06-03T08:37:55.614497+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-06-04T12:12:03.686838+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2],"acting":[1,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"17.5","version":"0'0","reported_seq":943,"reported_epoch":1009,"state":"active+clean","last_fresh":"2024-06-03T09:06:14.442357+0000","last_change":"2024-06-03T08:38:23.466051+0000","last_active":"2024-06-03T09:06:14.442357+0000","last_peered":"2024-06-03T09:06:14.442357+0000","last_clean":"2024-06-03T09:06:14.442357+0000","last_became_active":"2024-06-03T08:38:23.458039+0000","last_became_peered":"2024-06-03T08:38:23.458039+0000","last_unstale":"2024-06-03T09:06:14.442357+0000","last_undegraded":"2024-06-03T09:06:14.442357+0000","last_fullsized":"2024-06-03T09:06:14.442357+0000","mapping_epoch":28,"log_start":"0'0","ondisk_log_start":"0'0","created":28,"last_epoch_clean":49,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-06-03T08:37:55.614497+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-06-03T08:37:55.614497+0000","last_clean_scrub_stamp":"2024-06-03T08:37:55.614497+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-06-04T12:27:02.532436+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,4],"acting":[0,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"17.4","version":"0'0","reported_seq":149,"reported_epoch":1009,"state":"active+undersized","last_fresh":"2024-06-03T09:06:13.941522+0000","last_change":"2024-06-03T09:06:03.441889+0000","last_active":"2024-06-03T09:06:13.941522+0000","last_peered":"2024-06-03T09:06:13.941522+0000","last_clean":"2024-06-03T08:37:55.614497+0000","last_became_active":"2024-06-03T09:06:03.441889+0000","last_became_peered":"2024-06-03T09:06:03.441889+0000","last_unstale":"2024-06-03T09:06:13.941522+0000","last_undegraded":"2024-06-03T09:06:13.941522+0000","last_fullsized":"2024-06-03T09:06:03.359582+0000","mapping_epoch":999,"log_start":"0'0","ondisk_log_start":"0'0","created":28,"last_epoch_clean":255,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-06-03T08:37:55.614497+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-06-03T08:37:55.614497+0000","last_clean_scrub_stamp":"2024-06-03T08:37:55.614497+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-06-04T12:49:22.907141+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7],"acting":[7],"avail_no_missing":["7"],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"17.7","version":"0'0","reported_seq":929,"reported_epoch":1009,"state":"active+clean","last_fresh":"2024-06-03T09:06:13.736920+0000","last_change":"2024-06-03T08:38:04.837083+0000","last_active":"2024-06-03T09:06:13.736920+0000","last_peered":"2024-06-03T09:06:13.736920+0000","last_clean":"2024-06-03T09:06:13.736920+0000","last_became_active":"2024-06-03T08:38:04.818246+0000","last_became_peered":"2024-06-03T08:38:04.818246+0000","last_unstale":"2024-06-03T09:06:13.736920+0000","last_undegraded":"2024-06-03T09:06:13.736920+0000","last_fullsized":"2024-06-03T09:06:13.736920+0000","mapping_epoch":28,"log_start":"0'0","ondisk_log_start":"0'0","created":28,"last_epoch_clean":34,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-06-03T08:37:55.614497+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-06-03T08:37:55.614497+0000","last_clean_scrub_stamp":"2024-06-03T08:37:55.614497+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-06-04T17:02:38.204813+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,2],"acting":[5,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"17.6","version":"0'0","reported_seq":960,"reported_epoch":1009,"state":"active+undersized","last_fresh":"2024-06-03T09:06:14.453110+0000","last_change":"2024-06-03T09:06:03.428677+0000","last_active":"2024-06-03T09:06:14.453110+0000","last_peered":"2024-06-03T09:06:14.453110+0000","last_clean":"2024-06-03T09:04:43.312352+0000","last_became_active":"2024-06-03T09:06:03.428677+0000","last_became_peered":"2024-06-03T09:06:03.428677+0000","last_unstale":"2024-06-03T09:06:14.453110+0000","last_undegraded":"2024-06-03T09:06:14.453110+0000","last_fullsized":"2024-06-03T09:06:03.359909+0000","mapping_epoch":999,"log_start":"0'0","ondisk_log_start":"0'0","created":28,"last_epoch_clean":255,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-06-03T08:37:55.614497+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-06-03T08:37:55.614497+0000","last_clean_scrub_stamp":"2024-06-03T08:37:55.614497+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-06-04T15:38:02.180241+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0],"acting":[0],"avail_no_missing":["0"],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"2.7","version":"0'0","reported_seq":940,"reported_epoch":1009,"state":"active+clean","last_fresh":"2024-06-03T09:06:13.684065+0000","last_change":"2024-06-03T08:36:30.385405+0000","last_active":"2024-06-03T09:06:13.684065+0000","last_peered":"2024-06-03T09:06:13.684065+0000","last_clean":"2024-06-03T09:06:13.684065+0000","last_became_active":"2024-06-03T08:36:30.380302+0000","last_became_peered":"2024-06-03T08:36:30.380302+0000","last_unstale":"2024-06-03T09:06:13.684065+0000","last_undegraded":"2024-06-03T09:06:13.684065+0000","last_fullsized":"2024-06-03T09:06:13.684065+0000","mapping_epoch":23,"log_start":"0'0","ondisk_log_start":"0'0","created":23,"last_epoch_clean":24,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-06-03T08:36:28.444764+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-06-03T08:36:28.444764+0000","last_clean_scrub_stamp":"2024-06-03T08:36:28.444764+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-06-04T18:46:33.475181+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0],"acting":[3,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"17.14","version":"0'0","reported_seq":926,"reported_epoch":1009,"state":"active+clean","last_fresh":"2024-06-03T09:06:13.721460+0000","last_change":"2024-06-03T08:38:04.547565+0000","last_active":"2024-06-03T09:06:13.721460+0000","last_peered":"2024-06-03T09:06:13.721460+0000","last_clean":"2024-06-03T09:06:13.721460+0000","last_became_active":"2024-06-03T08:38:04.541781+0000","last_became_peered":"2024-06-03T08:38:04.541781+0000","last_unstale":"2024-06-03T09:06:13.721460+0000","last_undegraded":"2024-06-03T09:06:13.721460+0000","last_fullsized":"2024-06-03T09:06:13.721460+0000","mapping_epoch":28,"log_start":"0'0","ondisk_log_start":"0'0","created":28,"last_epoch_clean":34,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-06-03T08:37:55.614497+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-06-03T08:37:55.614497+0000","last_clean_scrub_stamp":"2024-06-03T08:37:55.614497+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-06-04T16:41:55.170134+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,2],"acting":[5,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"2.6","version":"0'0","reported_seq":949,"reported_epoch":1009,"state":"active+clean","last_fresh":"2024-06-03T09:06:14.448100+0000","last_change":"2024-06-03T08:36:30.430760+0000","last_active":"2024-06-03T09:06:14.448100+0000","last_peered":"2024-06-03T09:06:14.448100+0000","last_clean":"2024-06-03T09:06:14.448100+0000","last_became_active":"2024-06-03T08:36:30.404665+0000","last_became_peered":"2024-06-03T08:36:30.404665+0000","last_unstale":"2024-06-03T09:06:14.448100+0000","last_undegraded":"2024-06-03T09:06:14.448100+0000","last_fullsized":"2024-06-03T09:06:14.448100+0000","mapping_epoch":23,"log_start":"0'0","ondisk_log_start":"0'0","created":23,"last_epoch_clean":24,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-06-03T08:36:28.444764+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-06-03T08:36:28.444764+0000","last_clean_scrub_stamp":"2024-06-03T08:36:28.444764+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-06-04T10:37:37.746074+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,0],"acting":[4,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"17.15","version":"0'0","reported_seq":934,"reported_epoch":1009,"state":"active+clean","last_fresh":"2024-06-03T09:06:13.672940+0000","last_change":"2024-06-03T08:38:20.999903+0000","last_active":"2024-06-03T09:06:13.672940+0000","last_peered":"2024-06-03T09:06:13.672940+0000","last_clean":"2024-06-03T09:06:13.672940+0000","last_became_active":"2024-06-03T08:38:20.979523+0000","last_became_peered":"2024-06-03T08:38:20.979523+0000","last_unstale":"2024-06-03T09:06:13.672940+0000","last_undegraded":"2024-06-03T09:06:13.672940+0000","last_fullsized":"2024-06-03T09:06:13.672940+0000","mapping_epoch":28,"log_start":"0'0","ondisk_log_start":"0'0","created":28,"last_epoch_clean":47,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-06-03T08:37:55.614497+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-06-03T08:37:55.614497+0000","last_clean_scrub_stamp":"2024-06-03T08:37:55.614497+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-06-04T20:29:58.314289+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,1],"acting":[3,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"2.5","version":"0'0","reported_seq":942,"reported_epoch":1010,"state":"active+clean","last_fresh":"2024-06-03T09:06:18.382767+0000","last_change":"2024-06-03T08:36:30.024235+0000","last_active":"2024-06-03T09:06:18.382767+0000","last_peered":"2024-06-03T09:06:18.382767+0000","last_clean":"2024-06-03T09:06:18.382767+0000","last_became_active":"2024-06-03T08:36:30.019040+0000","last_became_peered":"2024-06-03T08:36:30.019040+0000","last_unstale":"2024-06-03T09:06:18.382767+0000","last_undegraded":"2024-06-03T09:06:18.382767+0000","last_fullsized":"2024-06-03T09:06:18.382767+0000","mapping_epoch":23,"log_start":"0'0","ondisk_log_start":"0'0","created":23,"last_epoch_clean":24,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-06-03T08:36:28.444764+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-06-03T08:36:28.444764+0000","last_clean_scrub_stamp":"2024-06-03T08:36:28.444764+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-06-04T15:46:47.109502+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,4],"acting":[3,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"17.16","version":"0'0","reported_seq":972,"reported_epoch":1009,"state":"active+undersized","last_fresh":"2024-06-03T09:06:14.441665+0000","last_change":"2024-06-03T09:06:03.478615+0000","last_active":"2024-06-03T09:06:14.441665+0000","last_peered":"2024-06-03T09:06:14.441665+0000","last_clean":"2024-06-03T09:04:43.328415+0000","last_became_active":"2024-06-03T09:06:03.478615+0000","last_became_peered":"2024-06-03T09:06:03.478615+0000","last_unstale":"2024-06-03T09:06:14.441665+0000","last_undegraded":"2024-06-03T09:06:14.441665+0000","last_fullsized":"2024-06-03T09:06:03.402504+0000","mapping_epoch":999,"log_start":"0'0","ondisk_log_start":"0'0","created":28,"last_epoch_clean":255,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-06-03T08:37:55.614497+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-06-03T08:37:55.614497+0000","last_clean_scrub_stamp":"2024-06-03T08:37:55.614497+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-06-04T08:47:08.079233+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2],"acting":[2],"avail_no_missing":["2"],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"2.4","version":"0'0","reported_seq":952,"reported_epoch":1010,"state":"active+undersized","last_fresh":"2024-06-03T09:06:18.331485+0000","last_change":"2024-06-03T09:06:03.387171+0000","last_active":"2024-06-03T09:06:18.331485+0000","last_peered":"2024-06-03T09:06:18.331485+0000","last_clean":"2024-06-03T09:04:43.202645+0000","last_became_active":"2024-06-03T09:06:03.387171+0000","last_became_peered":"2024-06-03T09:06:03.387171+0000","last_unstale":"2024-06-03T09:06:18.331485+0000","last_undegraded":"2024-06-03T09:06:18.331485+0000","last_fullsized":"2024-06-03T09:06:03.283787+0000","mapping_epoch":999,"log_start":"0'0","ondisk_log_start":"0'0","created":23,"last_epoch_clean":255,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-06-03T08:36:28.444764+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-06-03T08:36:28.444764+0000","last_clean_scrub_stamp":"2024-06-03T08:36:28.444764+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-06-04T20:17:43.485023+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5],"acting":[5],"avail_no_missing":["5"],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"17.17","version":"0'0","reported_seq":954,"reported_epoch":1009,"state":"active+clean","last_fresh":"2024-06-03T09:06:14.447046+0000","last_change":"2024-06-03T08:38:38.483275+0000","last_active":"2024-06-03T09:06:14.447046+0000","last_peered":"2024-06-03T09:06:14.447046+0000","last_clean":"2024-06-03T09:06:14.447046+0000","last_became_active":"2024-06-03T08:38:38.467916+0000","last_became_peered":"2024-06-03T08:38:38.467916+0000","last_unstale":"2024-06-03T09:06:14.447046+0000","last_undegraded":"2024-06-03T09:06:14.447046+0000","last_fullsized":"2024-06-03T09:06:14.447046+0000","mapping_epoch":28,"log_start":"0'0","ondisk_log_start":"0'0","created":28,"last_epoch_clean":61,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-06-03T08:37:55.614497+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-06-03T08:37:55.614497+0000","last_clean_scrub_stamp":"2024-06-03T08:37:55.614497+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-06-04T10:02:23.036420+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,7],"acting":[2,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"2.2","version":"0'0","reported_seq":963,"reported_epoch":1009,"state":"active+undersized","last_fresh":"2024-06-03T09:06:13.657892+0000","last_change":"2024-06-03T09:06:03.410020+0000","last_active":"2024-06-03T09:06:13.657892+0000","last_peered":"2024-06-03T09:06:13.657892+0000","last_clean":"2024-06-03T09:05:37.145889+0000","last_became_active":"2024-06-03T09:06:03.410020+0000","last_became_peered":"2024-06-03T09:06:03.410020+0000","last_unstale":"2024-06-03T09:06:13.657892+0000","last_undegraded":"2024-06-03T09:06:13.657892+0000","last_fullsized":"2024-06-03T09:06:03.346864+0000","mapping_epoch":999,"log_start":"0'0","ondisk_log_start":"0'0","created":23,"last_epoch_clean":255,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-06-03T08:36:28.444764+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-06-03T08:36:28.444764+0000","last_clean_scrub_stamp":"2024-06-03T08:36:28.444764+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-06-04T09:57:12.423360+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3],"acting":[3],"avail_no_missing":["3"],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"17.11","version":"0'0","reported_seq":944,"reported_epoch":1009,"state":"active+clean","last_fresh":"2024-06-03T09:06:14.426670+0000","last_change":"2024-06-03T08:38:06.502302+0000","last_active":"2024-06-03T09:06:14.426670+0000","last_peered":"2024-06-03T09:06:14.426670+0000","last_clean":"2024-06-03T09:06:14.426670+0000","last_became_active":"2024-06-03T08:38:06.490204+0000","last_became_peered":"2024-06-03T08:38:06.490204+0000","last_unstale":"2024-06-03T09:06:14.426670+0000","last_undegraded":"2024-06-03T09:06:14.426670+0000","last_fullsized":"2024-06-03T09:06:14.426670+0000","mapping_epoch":28,"log_start":"0'0","ondisk_log_start":"0'0","created":28,"last_epoch_clean":32,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-06-03T08:37:55.614497+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-06-03T08:37:55.614497+0000","last_clean_scrub_stamp":"2024-06-03T08:37:55.614497+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-06-04T10:47:05.988958+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,2],"acting":[4,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"2.1","version":"0'0","reported_seq":952,"reported_epoch":1009,"state":"active+clean","last_fresh":"2024-06-03T09:06:13.929725+0000","last_change":"2024-06-03T08:36:30.883360+0000","last_active":"2024-06-03T09:06:13.929725+0000","last_peered":"2024-06-03T09:06:13.929725+0000","last_clean":"2024-06-03T09:06:13.929725+0000","last_became_active":"2024-06-03T08:36:30.855073+0000","last_became_peered":"2024-06-03T08:36:30.855073+0000","last_unstale":"2024-06-03T09:06:13.929725+0000","last_undegraded":"2024-06-03T09:06:13.929725+0000","last_fullsized":"2024-06-03T09:06:13.929725+0000","mapping_epoch":23,"log_start":"0'0","ondisk_log_start":"0'0","created":23,"last_epoch_clean":25,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-06-03T08:36:28.444764+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-06-03T08:36:28.444764+0000","last_clean_scrub_stamp":"2024-06-03T08:36:28.444764+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-06-04T18:12:22.363473+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,5],"acting":[7,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"17.12","version":"0'0","reported_seq":945,"reported_epoch":1009,"state":"active+clean","last_fresh":"2024-06-03T09:06:13.935141+0000","last_change":"2024-06-03T08:38:01.751562+0000","last_active":"2024-06-03T09:06:13.935141+0000","last_peered":"2024-06-03T09:06:13.935141+0000","last_clean":"2024-06-03T09:06:13.935141+0000","last_became_active":"2024-06-03T08:38:01.745027+0000","last_became_peered":"2024-06-03T08:38:01.745027+0000","last_unstale":"2024-06-03T09:06:13.935141+0000","last_undegraded":"2024-06-03T09:06:13.935141+0000","last_fullsized":"2024-06-03T09:06:13.935141+0000","mapping_epoch":28,"log_start":"0'0","ondisk_log_start":"0'0","created":28,"last_epoch_clean":32,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-06-03T08:37:55.614497+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-06-03T08:37:55.614497+0000","last_clean_scrub_stamp":"2024-06-03T08:37:55.614497+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-06-04T17:54:43.785326+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,3],"acting":[7,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"2.0","version":"0'0","reported_seq":943,"reported_epoch":1009,"state":"active+clean","last_fresh":"2024-06-03T09:06:13.648112+0000","last_change":"2024-06-03T08:36:29.984096+0000","last_active":"2024-06-03T09:06:13.648112+0000","last_peered":"2024-06-03T09:06:13.648112+0000","last_clean":"2024-06-03T09:06:13.648112+0000","last_became_active":"2024-06-03T08:36:29.957875+0000","last_became_peered":"2024-06-03T08:36:29.957875+0000","last_unstale":"2024-06-03T09:06:13.648112+0000","last_undegraded":"2024-06-03T09:06:13.648112+0000","last_fullsized":"2024-06-03T09:06:13.648112+0000","mapping_epoch":23,"log_start":"0'0","ondisk_log_start":"0'0","created":23,"last_epoch_clean":24,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-06-03T08:36:28.444764+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-06-03T08:36:28.444764+0000","last_clean_scrub_stamp":"2024-06-03T08:36:28.444764+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-06-04T08:41:33.679479+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,5],"acting":[3,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"17.13","version":"0'0","reported_seq":943,"reported_epoch":1009,"state":"active+clean","last_fresh":"2024-06-03T09:06:14.438399+0000","last_change":"2024-06-03T08:38:26.838316+0000","last_active":"2024-06-03T09:06:14.438399+0000","last_peered":"2024-06-03T09:06:14.438399+0000","last_clean":"2024-06-03T09:06:14.438399+0000","last_became_active":"2024-06-03T08:38:26.833214+0000","last_became_peered":"2024-06-03T08:38:26.833214+0000","last_unstale":"2024-06-03T09:06:14.438399+0000","last_undegraded":"2024-06-03T09:06:14.438399+0000","last_fullsized":"2024-06-03T09:06:14.438399+0000","mapping_epoch":28,"log_start":"0'0","ondisk_log_start":"0'0","created":28,"last_epoch_clean":52,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-06-03T08:37:55.614497+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-06-03T08:37:55.614497+0000","last_clean_scrub_stamp":"2024-06-03T08:37:55.614497+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-06-04T12:58:51.617998+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,0],"acting":[4,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"2.3","version":"0'0","reported_seq":943,"reported_epoch":1010,"state":"active+clean","last_fresh":"2024-06-03T09:06:18.362614+0000","last_change":"2024-06-03T08:36:30.351670+0000","last_active":"2024-06-03T09:06:18.362614+0000","last_peered":"2024-06-03T09:06:18.362614+0000","last_clean":"2024-06-03T09:06:18.362614+0000","last_became_active":"2024-06-03T08:36:30.343560+0000","last_became_peered":"2024-06-03T08:36:30.343560+0000","last_unstale":"2024-06-03T09:06:18.362614+0000","last_undegraded":"2024-06-03T09:06:18.362614+0000","last_fullsized":"2024-06-03T09:06:18.362614+0000","mapping_epoch":23,"log_start":"0'0","ondisk_log_start":"0'0","created":23,"last_epoch_clean":24,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-06-03T08:36:28.444764+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-06-03T08:36:28.444764+0000","last_clean_scrub_stamp":"2024-06-03T08:36:28.444764+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-06-04T12:02:24.765135+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,2],"acting":[3,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.0","version":"19'76","reported_seq":1051,"reported_epoch":1009,"state":"active+clean","last_fresh":"2024-06-03T09:06:13.703685+0000","last_change":"2024-06-03T08:36:26.702702+0000","last_active":"2024-06-03T09:06:13.703685+0000","last_peered":"2024-06-03T09:06:13.703685+0000","last_clean":"2024-06-03T09:06:13.703685+0000","last_became_active":"2024-06-03T08:36:26.128378+0000","last_became_peered":"2024-06-03T08:36:26.128378+0000","last_unstale":"2024-06-03T09:06:13.703685+0000","last_undegraded":"2024-06-03T09:06:13.703685+0000","last_fullsized":"2024-06-03T09:06:13.703685+0000","mapping_epoch":20,"log_start":"0'0","ondisk_log_start":"0'0","created":16,"last_epoch_clean":22,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-06-03T08:36:15.542635+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-06-03T08:36:15.542635+0000","last_clean_scrub_stamp":"2024-06-03T08:36:15.542635+0000","objects_scrubbed":0,"log_size":76,"log_dups_size":0,"ondisk_log_size":76,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-06-04T12:41:53.731360+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":459280,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":96,"num_read_kb":82,"num_write":113,"num_write_kb":1372,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2,"num_bytes_recovered":397840,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,5],"acting":[1,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"17.10","version":"0'0","reported_seq":944,"reported_epoch":1009,"state":"active+clean","last_fresh":"2024-06-03T09:06:14.422035+0000","last_change":"2024-06-03T08:38:15.399189+0000","last_active":"2024-06-03T09:06:14.422035+0000","last_peered":"2024-06-03T09:06:14.422035+0000","last_clean":"2024-06-03T09:06:14.422035+0000","last_became_active":"2024-06-03T08:38:15.337723+0000","last_became_peered":"2024-06-03T08:38:15.337723+0000","last_unstale":"2024-06-03T09:06:14.422035+0000","last_undegraded":"2024-06-03T09:06:14.422035+0000","last_fullsized":"2024-06-03T09:06:14.422035+0000","mapping_epoch":28,"log_start":"0'0","ondisk_log_start":"0'0","created":28,"last_epoch_clean":42,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-06-03T08:37:55.614497+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-06-03T08:37:55.614497+0000","last_clean_scrub_stamp":"2024-06-03T08:37:55.614497+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-06-04T09:29:46.427317+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,3],"acting":[4,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"17.1a","version":"0'0","reported_seq":960,"reported_epoch":1009,"state":"active+undersized","last_fresh":"2024-06-03T09:06:14.431224+0000","last_change":"2024-06-03T09:06:03.390734+0000","last_active":"2024-06-03T09:06:14.431224+0000","last_peered":"2024-06-03T09:06:14.431224+0000","last_clean":"2024-06-03T09:05:20.322313+0000","last_became_active":"2024-06-03T09:06:03.390734+0000","last_became_peered":"2024-06-03T09:06:03.390734+0000","last_unstale":"2024-06-03T09:06:14.431224+0000","last_undegraded":"2024-06-03T09:06:14.431224+0000","last_fullsized":"2024-06-03T09:06:03.343197+0000","mapping_epoch":999,"log_start":"0'0","ondisk_log_start":"0'0","created":28,"last_epoch_clean":254,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-06-03T08:37:55.614497+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-06-03T08:37:55.614497+0000","last_clean_scrub_stamp":"2024-06-03T08:37:55.614497+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-06-04T15:56:47.628875+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4],"acting":[4],"avail_no_missing":["4"],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"17.1b","version":"0'0","reported_seq":935,"reported_epoch":1009,"state":"active+clean","last_fresh":"2024-06-03T09:06:13.663279+0000","last_change":"2024-06-03T08:38:03.115748+0000","last_active":"2024-06-03T09:06:13.663279+0000","last_peered":"2024-06-03T09:06:13.663279+0000","last_clean":"2024-06-03T09:06:13.663279+0000","last_became_active":"2024-06-03T08:38:03.110533+0000","last_became_peered":"2024-06-03T08:38:03.110533+0000","last_unstale":"2024-06-03T09:06:13.663279+0000","last_undegraded":"2024-06-03T09:06:13.663279+0000","last_fullsized":"2024-06-03T09:06:13.663279+0000","mapping_epoch":28,"log_start":"0'0","ondisk_log_start":"0'0","created":28,"last_epoch_clean":32,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-06-03T08:37:55.614497+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-06-03T08:37:55.614497+0000","last_clean_scrub_stamp":"2024-06-03T08:37:55.614497+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-06-04T12:36:47.471609+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,7],"acting":[3,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"17.18","version":"0'0","reported_seq":944,"reported_epoch":1009,"state":"active+clean","last_fresh":"2024-06-03T09:06:13.924799+0000","last_change":"2024-06-03T08:38:04.632864+0000","last_active":"2024-06-03T09:06:13.924799+0000","last_peered":"2024-06-03T09:06:13.924799+0000","last_clean":"2024-06-03T09:06:13.924799+0000","last_became_active":"2024-06-03T08:38:04.627856+0000","last_became_peered":"2024-06-03T08:38:04.627856+0000","last_unstale":"2024-06-03T09:06:13.924799+0000","last_undegraded":"2024-06-03T09:06:13.924799+0000","last_fullsized":"2024-06-03T09:06:13.924799+0000","mapping_epoch":28,"log_start":"0'0","ondisk_log_start":"0'0","created":28,"last_epoch_clean":33,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-06-03T08:37:55.614497+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-06-03T08:37:55.614497+0000","last_clean_scrub_stamp":"2024-06-03T08:37:55.614497+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-06-04T19:55:51.181598+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,4],"acting":[7,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"17.19","version":"0'0","reported_seq":934,"reported_epoch":1009,"state":"active+clean","last_fresh":"2024-06-03T09:06:13.652943+0000","last_change":"2024-06-03T08:38:01.839491+0000","last_active":"2024-06-03T09:06:13.652943+0000","last_peered":"2024-06-03T09:06:13.652943+0000","last_clean":"2024-06-03T09:06:13.652943+0000","last_became_active":"2024-06-03T08:38:01.829539+0000","last_became_peered":"2024-06-03T08:38:01.829539+0000","last_unstale":"2024-06-03T09:06:13.652943+0000","last_undegraded":"2024-06-03T09:06:13.652943+0000","last_fullsized":"2024-06-03T09:06:13.652943+0000","mapping_epoch":28,"log_start":"0'0","ondisk_log_start":"0'0","created":28,"last_epoch_clean":30,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-06-03T08:37:55.614497+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-06-03T08:37:55.614497+0000","last_clean_scrub_stamp":"2024-06-03T08:37:55.614497+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-06-04T09:10:43.860229+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,2],"acting":[3,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"17.1e","version":"0'0","reported_seq":148,"reported_epoch":1009,"state":"active+undersized","last_fresh":"2024-06-03T09:06:14.445917+0000","last_change":"2024-06-03T09:06:03.406181+0000","last_active":"2024-06-03T09:06:14.445917+0000","last_peered":"2024-06-03T09:06:14.445917+0000","last_clean":"2024-06-03T08:37:55.614497+0000","last_became_active":"2024-06-03T09:06:03.406181+0000","last_became_peered":"2024-06-03T09:06:03.406181+0000","last_unstale":"2024-06-03T09:06:14.445917+0000","last_undegraded":"2024-06-03T09:06:14.445917+0000","last_fullsized":"2024-06-03T09:06:03.363739+0000","mapping_epoch":999,"log_start":"0'0","ondisk_log_start":"0'0","created":28,"last_epoch_clean":255,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-06-03T08:37:55.614497+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-06-03T08:37:55.614497+0000","last_clean_scrub_stamp":"2024-06-03T08:37:55.614497+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-06-04T11:45:24.486600+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4],"acting":[4],"avail_no_missing":["4"],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"17.1f","version":"0'0","reported_seq":941,"reported_epoch":1009,"state":"active+clean","last_fresh":"2024-06-03T09:06:14.450053+0000","last_change":"2024-06-03T08:38:04.383538+0000","last_active":"2024-06-03T09:06:14.450053+0000","last_peered":"2024-06-03T09:06:14.450053+0000","last_clean":"2024-06-03T09:06:14.450053+0000","last_became_active":"2024-06-03T08:38:04.367278+0000","last_became_peered":"2024-06-03T08:38:04.367278+0000","last_unstale":"2024-06-03T09:06:14.450053+0000","last_undegraded":"2024-06-03T09:06:14.450053+0000","last_fullsized":"2024-06-03T09:06:14.450053+0000","mapping_epoch":28,"log_start":"0'0","ondisk_log_start":"0'0","created":28,"last_epoch_clean":32,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-06-03T08:37:55.614497+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-06-03T08:37:55.614497+0000","last_clean_scrub_stamp":"2024-06-03T08:37:55.614497+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-06-04T10:39:26.552282+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,2],"acting":[0,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"17.1c","version":"0'0","reported_seq":944,"reported_epoch":1010,"state":"active+undersized","last_fresh":"2024-06-03T09:06:18.327492+0000","last_change":"2024-06-03T09:06:03.382210+0000","last_active":"2024-06-03T09:06:18.327492+0000","last_peered":"2024-06-03T09:06:18.327492+0000","last_clean":"2024-06-03T09:04:43.199660+0000","last_became_active":"2024-06-03T09:06:03.382210+0000","last_became_peered":"2024-06-03T09:06:03.382210+0000","last_unstale":"2024-06-03T09:06:18.327492+0000","last_undegraded":"2024-06-03T09:06:18.327492+0000","last_fullsized":"2024-06-03T09:06:03.277874+0000","mapping_epoch":999,"log_start":"0'0","ondisk_log_start":"0'0","created":28,"last_epoch_clean":255,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-06-03T08:37:55.614497+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-06-03T08:37:55.614497+0000","last_clean_scrub_stamp":"2024-06-03T08:37:55.614497+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-06-04T12:18:50.013884+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5],"acting":[5],"avail_no_missing":["5"],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"17.1d","version":"0'0","reported_seq":923,"reported_epoch":1009,"state":"active+clean","last_fresh":"2024-06-03T09:06:13.715906+0000","last_change":"2024-06-03T08:38:06.794630+0000","last_active":"2024-06-03T09:06:13.715906+0000","last_peered":"2024-06-03T09:06:13.715906+0000","last_clean":"2024-06-03T09:06:13.715906+0000","last_became_active":"2024-06-03T08:38:06.592317+0000","last_became_peered":"2024-06-03T08:38:06.592317+0000","last_unstale":"2024-06-03T09:06:13.715906+0000","last_undegraded":"2024-06-03T09:06:13.715906+0000","last_fullsized":"2024-06-03T09:06:13.715906+0000","mapping_epoch":28,"log_start":"0'0","ondisk_log_start":"0'0","created":28,"last_epoch_clean":33,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-06-03T08:37:55.614497+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-06-03T08:37:55.614497+0000","last_clean_scrub_stamp":"2024-06-03T08:37:55.614497+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-06-04T12:33:30.785369+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,7],"acting":[1,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"17.2","version":"0'0","reported_seq":925,"reported_epoch":1009,"state":"active+clean","last_fresh":"2024-06-03T09:06:13.708688+0000","last_change":"2024-06-03T08:38:14.981410+0000","last_active":"2024-06-03T09:06:13.708688+0000","last_peered":"2024-06-03T09:06:13.708688+0000","last_clean":"2024-06-03T09:06:13.708688+0000","last_became_active":"2024-06-03T08:38:14.973825+0000","last_became_peered":"2024-06-03T08:38:14.973825+0000","last_unstale":"2024-06-03T09:06:13.708688+0000","last_undegraded":"2024-06-03T09:06:13.708688+0000","last_fullsized":"2024-06-03T09:06:13.708688+0000","mapping_epoch":28,"log_start":"0'0","ondisk_log_start":"0'0","created":28,"last_epoch_clean":41,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-06-03T08:37:55.614497+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-06-03T08:37:55.614497+0000","last_clean_scrub_stamp":"2024-06-03T08:37:55.614497+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-06-04T17:01:05.518613+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,4],"acting":[1,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"17.3","version":"0'0","reported_seq":943,"reported_epoch":1009,"state":"active+clean","last_fresh":"2024-06-03T09:06:14.437636+0000","last_change":"2024-06-03T08:38:14.619745+0000","last_active":"2024-06-03T09:06:14.437636+0000","last_peered":"2024-06-03T09:06:14.437636+0000","last_clean":"2024-06-03T09:06:14.437636+0000","last_became_active":"2024-06-03T08:38:14.510665+0000","last_became_peered":"2024-06-03T08:38:14.510665+0000","last_unstale":"2024-06-03T09:06:14.437636+0000","last_undegraded":"2024-06-03T09:06:14.437636+0000","last_fullsized":"2024-06-03T09:06:14.437636+0000","mapping_epoch":28,"log_start":"0'0","ondisk_log_start":"0'0","created":28,"last_epoch_clean":42,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-06-03T08:37:55.614497+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-06-03T08:37:55.614497+0000","last_clean_scrub_stamp":"2024-06-03T08:37:55.614497+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-06-04T12:03:47.392064+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,7],"acting":[0,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"17.0","version":"0'0","reported_seq":944,"reported_epoch":1009,"state":"active+clean","last_fresh":"2024-06-03T09:06:14.427123+0000","last_change":"2024-06-03T08:38:04.237118+0000","last_active":"2024-06-03T09:06:14.427123+0000","last_peered":"2024-06-03T09:06:14.427123+0000","last_clean":"2024-06-03T09:06:14.427123+0000","last_became_active":"2024-06-03T08:38:04.231954+0000","last_became_peered":"2024-06-03T08:38:04.231954+0000","last_unstale":"2024-06-03T09:06:14.427123+0000","last_undegraded":"2024-06-03T09:06:14.427123+0000","last_fullsized":"2024-06-03T09:06:14.427123+0000","mapping_epoch":28,"log_start":"0'0","ondisk_log_start":"0'0","created":28,"last_epoch_clean":32,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-06-03T08:37:55.614497+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-06-03T08:37:55.614497+0000","last_clean_scrub_stamp":"2024-06-03T08:37:55.614497+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-06-04T09:56:10.741637+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,2],"acting":[0,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"17.1","version":"65'7","reported_seq":959,"reported_epoch":1009,"state":"active+undersized+degraded","last_fresh":"2024-06-03T09:06:13.709209+0000","last_change":"2024-06-03T09:06:03.370660+0000","last_active":"2024-06-03T09:06:13.709209+0000","last_peered":"2024-06-03T09:06:13.709209+0000","last_clean":"2024-06-03T09:05:03.733864+0000","last_became_active":"2024-06-03T09:06:03.370660+0000","last_became_peered":"2024-06-03T09:06:03.370660+0000","last_unstale":"2024-06-03T09:06:13.709209+0000","last_undegraded":"2024-06-03T09:06:03.266076+0000","last_fullsized":"2024-06-03T09:06:03.264525+0000","mapping_epoch":999,"log_start":"0'0","ondisk_log_start":"0'0","created":28,"last_epoch_clean":255,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-06-03T08:37:55.614497+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-06-03T08:37:55.614497+0000","last_clean_scrub_stamp":"2024-06-03T08:37:55.614497+0000","objects_scrubbed":0,"log_size":7,"log_dups_size":0,"ondisk_log_size":7,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-06-04T11:55:18.194283+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":5,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":1,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":6,"num_read_kb":6,"num_write":7,"num_write_kb":7,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5],"acting":[5],"avail_no_missing":["5"],"object_location_counts":[{"shards":"5","objects":1}],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]}],"pool_stats":[{"poolid":2,"num_pg":8,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":14,"acting":14,"num_store_stats":7},{"poolid":1,"num_pg":1,"stat_sum":{"num_bytes":459280,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":96,"num_read_kb":82,"num_write":113,"num_write_kb":1372,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2,"num_bytes_recovered":397840,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":802816,"data_stored":795680,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":76,"ondisk_log_size":76,"up":2,"acting":2,"num_store_stats":2},{"poolid":17,"num_pg":32,"stat_sum":{"num_bytes":5,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":1,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":6,"num_read_kb":6,"num_write":7,"num_write_kb":7,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":8192,"data_stored":10,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":7,"ondisk_log_size":7,"up":52,"acting":52,"num_store_stats":8}],"osd_stats":[{"osd":6,"up_from":253,"seq":1086626726201,"num_pgs":20,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":54836,"kb_used_data":5672,"kb_used_omap":1,"kb_used_meta":49150,"kb_avail":94317004,"statfs":{"total":96636764160,"available":96580612096,"internally_reserved":0,"allocated":5808128,"data_stored":2801702,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1593,"internal_metadata":50330055},"hb_peers":[0,1,2,3,4,5,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":93,"apply_latency_ms":93,"commit_latency_ns":93000000,"apply_latency_ns":93000000},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Mon Jun 3 09:04:02 2024","interfaces":[{"interface":"back","average":{"1min":1582.4780000000001,"5min":332.72699999999998,"15min":124.113},"min":{"1min":4.4829999999999997,"5min":4.1959999999999997,"15min":4.1959999999999997},"max":{"1min":22766.882000000001,"5min":22766.882000000001,"15min":22766.882000000001},"last":5.5880000000000001},{"interface":"front","average":{"1min":1579.5409999999999,"5min":334.94499999999999,"15min":125.84999999999999},"min":{"1min":4.8129999999999997,"5min":4.7539999999999996,"15min":4.4530000000000003},"max":{"1min":22766.289000000001,"5min":22766.289000000001,"15min":22766.289000000001},"last":4.835}]},{"osd":1,"last update":"Mon Jun 3 09:03:40 2024","interfaces":[{"interface":"back","average":{"1min":36.420999999999999,"5min":77.054000000000002,"15min":132.84800000000001},"min":{"1min":6.0380000000000003,"5min":5.8799999999999999,"15min":5.484},"max":{"1min":589.64099999999996,"5min":4138.1679999999997,"15min":9955.1299999999992},"last":8.8469999999999995},{"interface":"front","average":{"1min":40.240000000000002,"5min":79.674999999999997,"15min":133.38999999999999},"min":{"1min":8.4589999999999996,"5min":8.4090000000000007,"15min":6.1390000000000002},"max":{"1min":588.60799999999995,"5min":4137.6530000000002,"15min":9827.5509999999995},"last":11.279999999999999}]},{"osd":2,"last update":"Mon Jun 3 09:03:40 2024","interfaces":[{"interface":"back","average":{"1min":12.613,"5min":24.600999999999999,"15min":21.088000000000001},"min":{"1min":6.9740000000000002,"5min":6.9740000000000002,"15min":6.5330000000000004},"max":{"1min":26.076000000000001,"5min":376.24900000000002,"15min":376.24900000000002},"last":13.292999999999999},{"interface":"front","average":{"1min":12.493,"5min":22.207999999999998,"15min":21.219000000000001},"min":{"1min":6.5810000000000004,"5min":6.5110000000000001,"15min":6.0750000000000002},"max":{"1min":29.690999999999999,"5min":271.51100000000002,"15min":434.77300000000002},"last":8.2739999999999991}]},{"osd":3,"last update":"Mon Jun 3 09:03:40 2024","interfaces":[{"interface":"back","average":{"1min":14.093999999999999,"5min":24.143999999999998,"15min":29.376999999999999},"min":{"1min":10.196999999999999,"5min":6.2919999999999998,"15min":6.1719999999999997},"max":{"1min":24.832000000000001,"5min":197.84899999999999,"15min":1235.5799999999999},"last":14.281000000000001},{"interface":"front","average":{"1min":9.3260000000000005,"5min":18.463999999999999,"15min":27.631},"min":{"1min":6.125,"5min":6.125,"15min":6.125},"max":{"1min":28.166,"5min":200.173,"15min":1157.6780000000001},"last":12.782999999999999}]},{"osd":4,"last update":"Mon Jun 3 09:04:29 2024","interfaces":[{"interface":"back","average":{"1min":10.279,"5min":1253.809,"15min":450.79199999999997},"min":{"1min":7.2300000000000004,"5min":6.093,"15min":6.093},"max":{"1min":15.000999999999999,"5min":28093.096000000001,"15min":28093.096000000001},"last":10.965999999999999},{"interface":"front","average":{"1min":15.693,"5min":1279.759,"15min":463.87099999999998},"min":{"1min":10.551,"5min":7.7480000000000002,"15min":7.7480000000000002},"max":{"1min":27.898,"5min":28323.837,"15min":28323.837},"last":13.581}]},{"osd":5,"last update":"Mon Jun 3 09:04:10 2024","interfaces":[{"interface":"back","average":{"1min":13.728999999999999,"5min":24.670000000000002,"15min":23.815999999999999},"min":{"1min":10.401,"5min":7.516,"15min":7.516},"max":{"1min":25.138000000000002,"5min":320.85000000000002,"15min":445.041},"last":14.928000000000001},{"interface":"front","average":{"1min":13.847,"5min":29.283000000000001,"15min":23.393999999999998},"min":{"1min":7.944,"5min":6.266,"15min":6.266},"max":{"1min":18.817,"5min":530.56700000000001,"15min":530.56700000000001},"last":16.893000000000001}]},{"osd":7,"last update":"Mon Jun 3 09:03:40 2024","interfaces":[{"interface":"back","average":{"1min":20.568999999999999,"5min":30.286999999999999,"15min":27.977},"min":{"1min":13.874000000000001,"5min":10.58,"15min":9.2859999999999996},"max":{"1min":86.959000000000003,"5min":222.07599999999999,"15min":255.08799999999999},"last":17.59},{"interface":"front","average":{"1min":19.102,"5min":29.550000000000001,"15min":27.763000000000002},"min":{"1min":10.052,"5min":8.6709999999999994,"15min":8.5429999999999993},"max":{"1min":67.968999999999994,"5min":293.62099999999998,"15min":438.14600000000002},"last":16.045000000000002}]}]},{"osd":7,"up_from":20,"seq":85899346289,"num_pgs":9,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":58964,"kb_used_data":5704,"kb_used_omap":1,"kb_used_meta":53246,"kb_avail":94312876,"statfs":{"total":96636764160,"available":96576385024,"internally_reserved":0,"allocated":5840896,"data_stored":2740727,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1589,"internal_metadata":54524363},"hb_peers":[0,1,2,3,4,5],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Mon Jun 3 09:06:09 2024","interfaces":[{"interface":"back","average":{"1min":14.073,"5min":894.22400000000005,"15min":308.93400000000003},"min":{"1min":5.5709999999999997,"5min":4.218,"15min":4.0599999999999996},"max":{"1min":90.757999999999996,"5min":21729.001,"15min":21729.001},"last":12.398},{"interface":"front","average":{"1min":9.2550000000000008,"5min":906.23500000000001,"15min":314.06},"min":{"1min":5.5789999999999997,"5min":4.8810000000000002,"15min":4.8810000000000002},"max":{"1min":25.937000000000001,"5min":21899.732,"15min":21899.732},"last":8.6609999999999996}]},{"osd":1,"last update":"Mon Jun 3 09:06:09 2024","interfaces":[{"interface":"back","average":{"1min":15.182,"5min":69.183999999999997,"15min":219.173},"min":{"1min":7.3639999999999999,"5min":6.6200000000000001,"15min":6.1319999999999997},"max":{"1min":76.926000000000002,"5min":2795.1329999999998,"15min":18373.257000000001},"last":9.1489999999999991},{"interface":"front","average":{"1min":21.495999999999999,"5min":74.703999999999994,"15min":225.285},"min":{"1min":8.8399999999999999,"5min":8.8399999999999999,"15min":8.5129999999999999},"max":{"1min":94.977999999999994,"5min":2803.3670000000002,"15min":18374.601999999999},"last":11.051}]},{"osd":2,"last update":"Mon Jun 3 09:06:09 2024","interfaces":[{"interface":"back","average":{"1min":19.164000000000001,"5min":26.902000000000001,"15min":22.920999999999999},"min":{"1min":11.207000000000001,"5min":8.5030000000000001,"15min":8.5030000000000001},"max":{"1min":91.319999999999993,"5min":333.99200000000002,"15min":333.99200000000002},"last":13.628},{"interface":"front","average":{"1min":11.125999999999999,"5min":18.132000000000001,"15min":16.353000000000002},"min":{"1min":6.202,"5min":5.8369999999999997,"15min":5.5069999999999997},"max":{"1min":28.649000000000001,"5min":226.18100000000001,"15min":226.18100000000001},"last":7.4770000000000003}]},{"osd":3,"last update":"Mon Jun 3 09:06:09 2024","interfaces":[{"interface":"back","average":{"1min":22.291,"5min":26.289999999999999,"15min":31.074999999999999},"min":{"1min":11.714,"5min":6.8959999999999999,"15min":6.3760000000000003},"max":{"1min":83.102000000000004,"5min":316.64400000000001,"15min":2039.9059999999999},"last":11.871},{"interface":"front","average":{"1min":13.324999999999999,"5min":17.963000000000001,"15min":25.969999999999999},"min":{"1min":6.1200000000000001,"5min":5.2089999999999996,"15min":5.2089999999999996},"max":{"1min":49.908999999999999,"5min":233.542,"15min":2037.5550000000001},"last":6.75}]},{"osd":4,"last update":"Mon Jun 3 09:06:29 2024","interfaces":[{"interface":"back","average":{"1min":17.675999999999998,"5min":1026.325,"15min":472.47199999999998},"min":{"1min":7.306,"5min":5.9089999999999998,"15min":5.9089999999999998},"max":{"1min":140.03299999999999,"5min":28969.369999999999,"15min":30073.917000000001},"last":5.2709999999999999},{"interface":"front","average":{"1min":17.890999999999998,"5min":1044.5139999999999,"15min":480.226},"min":{"1min":5.9279999999999999,"5min":5.9279999999999999,"15min":5.9279999999999999},"max":{"1min":141.179,"5min":29244.138999999999,"15min":30348.021000000001},"last":9.7810000000000006}]},{"osd":5,"last update":"Mon Jun 3 09:06:02 2024","interfaces":[{"interface":"back","average":{"1min":24.372,"5min":26.856000000000002,"15min":23.571999999999999},"min":{"1min":12.278,"5min":9.2460000000000004,"15min":6.5650000000000004},"max":{"1min":183.64099999999999,"5min":317.572,"15min":317.572},"last":15.048999999999999},{"interface":"front","average":{"1min":21.259,"5min":23.364000000000001,"15min":21.193999999999999},"min":{"1min":7.8499999999999996,"5min":7.0999999999999996,"15min":7.008},"max":{"1min":184.53399999999999,"5min":306.60599999999999,"15min":306.60599999999999},"last":14.843999999999999}]},{"osd":6,"last update":"Mon Jun 3 09:04:31 2024","interfaces":[{"interface":"back","average":{"1min":44.061,"5min":26.821000000000002,"15min":91.176000000000002},"min":{"1min":9.0800000000000001,"5min":7.9390000000000001,"15min":7.9390000000000001},"max":{"1min":268.029,"5min":308.916,"15min":8734.4639999999999},"last":6.6769999999999996},{"interface":"front","average":{"1min":46.643000000000001,"5min":23.879999999999999,"15min":88.765000000000001},"min":{"1min":8.8119999999999994,"5min":6.9240000000000004,"15min":6.742},"max":{"1min":305.64600000000002,"5min":305.64600000000002,"15min":8735.5319999999992},"last":12.327}]}]},{"osd":5,"up_from":20,"seq":85899346263,"num_pgs":9,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":59360,"kb_used_data":6100,"kb_used_omap":1,"kb_used_meta":53246,"kb_avail":94312480,"statfs":{"total":96636764160,"available":96575979520,"internally_reserved":0,"allocated":6246400,"data_stored":3138572,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1594,"internal_metadata":54524358},"hb_peers":[0,1,2,3,4,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Mon Jun 3 09:06:06 2024","interfaces":[{"interface":"back","average":{"1min":7.468,"5min":1703.037,"15min":580.51900000000001},"min":{"1min":4.6509999999999998,"5min":4.6509999999999998,"15min":4.4020000000000001},"max":{"1min":14.711,"5min":26433.764999999999,"15min":26433.764999999999},"last":5.8639999999999999},{"interface":"front","average":{"1min":9.9429999999999996,"5min":1725.27,"15min":587.87800000000004},"min":{"1min":5.5999999999999996,"5min":5.1319999999999997,"15min":4.8170000000000002},"max":{"1min":13.590999999999999,"5min":26704.698,"15min":26704.698},"last":12.747999999999999}]},{"osd":1,"last update":"Mon Jun 3 09:06:00 2024","interfaces":[{"interface":"back","average":{"1min":10.862,"5min":67.731999999999999,"15min":217.364},"min":{"1min":7.1769999999999996,"5min":6.0890000000000004,"15min":5.4930000000000003},"max":{"1min":16.998999999999999,"5min":3923.4090000000001,"15min":18811.696},"last":14.052},{"interface":"front","average":{"1min":10.417,"5min":65.141999999999996,"15min":220.75700000000001},"min":{"1min":5.3209999999999997,"5min":5.3209999999999997,"15min":5.0209999999999999},"max":{"1min":15.821999999999999,"5min":4019.6129999999998,"15min":18968.365000000002},"last":8.7219999999999995}]},{"osd":2,"last update":"Mon Jun 3 09:06:00 2024","interfaces":[{"interface":"back","average":{"1min":11.366,"5min":24.18,"15min":21.137},"min":{"1min":5.6369999999999996,"5min":5.6369999999999996,"15min":5.0510000000000002},"max":{"1min":15.529999999999999,"5min":332.48200000000003,"15min":332.48200000000003},"last":12.167},{"interface":"front","average":{"1min":10.089,"5min":23.117000000000001,"15min":24.050000000000001},"min":{"1min":6.6769999999999996,"5min":6.6769999999999996,"15min":5.8739999999999997},"max":{"1min":15.821,"5min":353.03199999999998,"15min":353.03199999999998},"last":7.2619999999999996}]},{"osd":3,"last update":"Mon Jun 3 09:06:00 2024","interfaces":[{"interface":"back","average":{"1min":13.472,"5min":24.283999999999999,"15min":31.666},"min":{"1min":10.111000000000001,"5min":7.4720000000000004,"15min":7.4210000000000003},"max":{"1min":16.271999999999998,"5min":353.57499999999999,"15min":2855.8519999999999},"last":8.1140000000000008},{"interface":"front","average":{"1min":9.5899999999999999,"5min":15.875,"15min":25.834},"min":{"1min":6.6609999999999996,"5min":5.7729999999999997,"15min":5.1909999999999998},"max":{"1min":15.112,"5min":203.804,"15min":2869.7860000000001},"last":9.3699999999999992}]},{"osd":4,"last update":"Mon Jun 3 09:06:29 2024","interfaces":[{"interface":"back","average":{"1min":11.067,"5min":780.21000000000004,"15min":377.28399999999999},"min":{"1min":5.4779999999999998,"5min":5.4779999999999998,"15min":5.4779999999999998},"max":{"1min":14.654999999999999,"5min":23315.438999999998,"15min":28619.825000000001},"last":6.7199999999999998},{"interface":"front","average":{"1min":14.349,"5min":782.97299999999996,"15min":377.93400000000003},"min":{"1min":7.492,"5min":6.5030000000000001,"15min":6.5030000000000001},"max":{"1min":17.707000000000001,"5min":23317.502,"15min":28621.294000000002},"last":11.621}]},{"osd":6,"last update":"Mon Jun 3 09:03:57 2024","interfaces":[{"interface":"back","average":{"1min":23.536000000000001,"5min":36.457000000000001,"15min":83.799999999999997},"min":{"1min":9.4800000000000004,"5min":9.4800000000000004,"15min":8.0630000000000006},"max":{"1min":89.442999999999998,"5min":508.77999999999997,"15min":9193.5169999999998},"last":157.79300000000001},{"interface":"front","average":{"1min":18.891999999999999,"5min":22.100999999999999,"15min":75.489999999999995},"min":{"1min":5.923,"5min":5.923,"15min":5.923},"max":{"1min":78.424999999999997,"5min":464.62,"15min":9191.4429999999993},"last":148.09}]},{"osd":7,"last update":"Mon Jun 3 09:06:00 2024","interfaces":[{"interface":"back","average":{"1min":13.257,"5min":20.917999999999999,"15min":21.088999999999999},"min":{"1min":10.475,"5min":6.7629999999999999,"15min":6.7629999999999999},"max":{"1min":17.245000000000001,"5min":211.22999999999999,"15min":211.22999999999999},"last":11.061999999999999},{"interface":"front","average":{"1min":12.872999999999999,"5min":20.587,"15min":24.021000000000001},"min":{"1min":8.5760000000000005,"5min":7.7709999999999999,"15min":6.8570000000000002},"max":{"1min":18.359999999999999,"5min":223.191,"15min":286.69499999999999},"last":10.021000000000001}]}]},{"osd":4,"up_from":20,"seq":85899346271,"num_pgs":12,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":63060,"kb_used_data":5704,"kb_used_omap":1,"kb_used_meta":57342,"kb_avail":94308780,"statfs":{"total":96636764160,"available":96572190720,"internally_reserved":0,"allocated":5840896,"data_stored":2740727,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1594,"internal_metadata":58718662},"hb_peers":[0,1,2,3,5,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Mon Jun 3 09:06:35 2024","interfaces":[{"interface":"back","average":{"1min":10.417,"5min":1093.547,"15min":381.64499999999998},"min":{"1min":4.6900000000000004,"5min":4.5279999999999996,"15min":4.3220000000000001},"max":{"1min":52.241,"5min":21726.743999999999,"15min":21726.743999999999},"last":7.0030000000000001},{"interface":"front","average":{"1min":13.175000000000001,"5min":1102.4200000000001,"15min":387.161},"min":{"1min":6.1950000000000003,"5min":5.8380000000000001,"15min":5.4909999999999997},"max":{"1min":53.148000000000003,"5min":21727.993999999999,"15min":21727.993999999999},"last":12.923999999999999}]},{"osd":1,"last update":"Mon Jun 3 09:06:35 2024","interfaces":[{"interface":"back","average":{"1min":9.9909999999999997,"5min":53.250999999999998,"15min":340.02999999999997},"min":{"1min":4.5670000000000002,"5min":4.1970000000000001,"15min":4.1970000000000001},"max":{"1min":47.073,"5min":3346.4389999999999,"15min":22256.098000000002},"last":15.198},{"interface":"front","average":{"1min":12.189,"5min":53.375999999999998,"15min":339.68799999999999},"min":{"1min":6.1539999999999999,"5min":5.0590000000000002,"15min":4.2640000000000002},"max":{"1min":49.558,"5min":3347.181,"15min":22247.317999999999},"last":7.681}]},{"osd":2,"last update":"Mon Jun 3 09:06:35 2024","interfaces":[{"interface":"back","average":{"1min":15.178000000000001,"5min":17.202000000000002,"15min":20.417999999999999},"min":{"1min":6.3010000000000002,"5min":6.3010000000000002,"15min":6.3010000000000002},"max":{"1min":85.340000000000003,"5min":337.60399999999998,"15min":393.58600000000001},"last":12.378},{"interface":"front","average":{"1min":15.423,"5min":14.432,"15min":22.754000000000001},"min":{"1min":6.4889999999999999,"5min":6.0629999999999997,"15min":6.0629999999999997},"max":{"1min":82.120000000000005,"5min":127.913,"15min":394.471},"last":10.843999999999999}]},{"osd":3,"last update":"Mon Jun 3 09:06:35 2024","interfaces":[{"interface":"back","average":{"1min":17.651,"5min":15.5,"15min":18.079000000000001},"min":{"1min":10.359999999999999,"5min":5.8719999999999999,"15min":5.4000000000000004},"max":{"1min":84.555999999999997,"5min":127.43600000000001,"15min":322.63},"last":14.564},{"interface":"front","average":{"1min":16.413,"5min":17.356999999999999,"15min":22.920000000000002},"min":{"1min":9.7349999999999994,"5min":7.484,"15min":7.484},"max":{"1min":71.394000000000005,"5min":204.517,"15min":392.67500000000001},"last":13.446999999999999}]},{"osd":5,"last update":"Mon Jun 3 09:06:35 2024","interfaces":[{"interface":"back","average":{"1min":15.672000000000001,"5min":16.742999999999999,"15min":20.795000000000002},"min":{"1min":8.093,"5min":6.7489999999999997,"15min":6.7489999999999997},"max":{"1min":51.326000000000001,"5min":268.10199999999998,"15min":344.23399999999998},"last":9.2910000000000004},{"interface":"front","average":{"1min":17.297999999999998,"5min":19.356999999999999,"15min":25.713999999999999},"min":{"1min":12.875,"5min":8.5180000000000007,"15min":8.5180000000000007},"max":{"1min":40.829000000000001,"5min":264.75,"15min":345.18700000000001},"last":10.318}]},{"osd":6,"last update":"Mon Jun 3 09:04:32 2024","interfaces":[{"interface":"back","average":{"1min":15.548,"5min":32.957999999999998,"15min":152.738},"min":{"1min":9.141,"5min":9.141,"15min":8.2319999999999993},"max":{"1min":52.57,"5min":395.238,"15min":12496.335999999999},"last":208.90000000000001},{"interface":"front","average":{"1min":12.680999999999999,"5min":25.428000000000001,"15min":149.30600000000001},"min":{"1min":6.4379999999999997,"5min":5.6319999999999997,"15min":5.6319999999999997},"max":{"1min":46.283000000000001,"5min":377.74299999999999,"15min":12616.297},"last":205.88900000000001}]},{"osd":7,"last update":"Mon Jun 3 09:06:35 2024","interfaces":[{"interface":"back","average":{"1min":9.6969999999999992,"5min":14.196,"15min":18.809999999999999},"min":{"1min":6.7190000000000003,"5min":6.2110000000000003,"15min":5.8010000000000002},"max":{"1min":15.960000000000001,"5min":207.25299999999999,"15min":358.94900000000001},"last":8.7810000000000006},{"interface":"front","average":{"1min":17.696000000000002,"5min":19.024999999999999,"15min":23.259},"min":{"1min":11.794,"5min":7.7789999999999999,"15min":7.7789999999999999},"max":{"1min":45.639000000000003,"5min":209.32900000000001,"15min":380.06900000000002},"last":11.362}]}]},{"osd":2,"up_from":20,"seq":85899346291,"num_pgs":11,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":58964,"kb_used_data":5704,"kb_used_omap":1,"kb_used_meta":53246,"kb_avail":94312876,"statfs":{"total":96636764160,"available":96576385024,"internally_reserved":0,"allocated":5840896,"data_stored":2740727,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1594,"internal_metadata":54524358},"hb_peers":[0,1,3,4,5,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Mon Jun 3 09:06:23 2024","interfaces":[{"interface":"back","average":{"1min":8.9629999999999992,"5min":946.25400000000002,"15min":326.36099999999999},"min":{"1min":5.3479999999999999,"5min":4.7050000000000001,"15min":4.1420000000000003},"max":{"1min":26.015000000000001,"5min":23465.18,"15min":23465.18},"last":7.5270000000000001},{"interface":"front","average":{"1min":7.6130000000000004,"5min":944.39800000000002,"15min":326.178},"min":{"1min":4.6790000000000003,"5min":4.6790000000000003,"15min":4.6790000000000003},"max":{"1min":20.802,"5min":23465.966,"15min":23465.966},"last":9.0079999999999991}]},{"osd":1,"last update":"Mon Jun 3 09:06:23 2024","interfaces":[{"interface":"back","average":{"1min":13.007,"5min":76.507000000000005,"15min":294.72899999999998},"min":{"1min":6.4089999999999998,"5min":5.4199999999999999,"15min":5.4199999999999999},"max":{"1min":85.430000000000007,"5min":4220.5290000000005,"15min":18954.987000000001},"last":6.4100000000000001},{"interface":"front","average":{"1min":15.894,"5min":82.119,"15min":302.92399999999998},"min":{"1min":7.367,"5min":7.367,"15min":7.367},"max":{"1min":98.498000000000005,"5min":4325.9489999999996,"15min":19143.834999999999},"last":10.802}]},{"osd":3,"last update":"Mon Jun 3 09:06:23 2024","interfaces":[{"interface":"back","average":{"1min":9.7579999999999991,"5min":12.444000000000001,"15min":28.065000000000001},"min":{"1min":7,"5min":5.9290000000000003,"15min":5.2450000000000001},"max":{"1min":28.164000000000001,"5min":118.913,"15min":2594.8150000000001},"last":10.282},{"interface":"front","average":{"1min":12.321999999999999,"5min":15.25,"15min":31.349},"min":{"1min":9.1240000000000006,"5min":8.0760000000000005,"15min":6.0780000000000003},"max":{"1min":32.360999999999997,"5min":156.34700000000001,"15min":2706.5120000000002},"last":15.382999999999999}]},{"osd":4,"last update":"Mon Jun 3 09:05:57 2024","interfaces":[{"interface":"back","average":{"1min":10.573,"5min":1266.894,"15min":438.154},"min":{"1min":8.6379999999999999,"5min":7.8789999999999996,"15min":6.2039999999999997},"max":{"1min":15.105,"5min":26660.861000000001,"15min":26660.861000000001},"last":11.968999999999999},{"interface":"front","average":{"1min":12.407999999999999,"5min":1266.7460000000001,"15min":435.75400000000002},"min":{"1min":10.266999999999999,"5min":10.266999999999999,"15min":6.2720000000000002},"max":{"1min":15.228,"5min":26616.243999999999,"15min":26616.243999999999},"last":14.516999999999999}]},{"osd":5,"last update":"Mon Jun 3 09:06:23 2024","interfaces":[{"interface":"back","average":{"1min":14.103,"5min":18.091000000000001,"15min":21.119},"min":{"1min":9.6349999999999998,"5min":9.2159999999999993,"15min":6.2050000000000001},"max":{"1min":32.890000000000001,"5min":252.31100000000001,"15min":252.31100000000001},"last":13.661},{"interface":"front","average":{"1min":11.545999999999999,"5min":16.170999999999999,"15min":20.254000000000001},"min":{"1min":7.6070000000000002,"5min":7.4850000000000003,"15min":6.8810000000000002},"max":{"1min":29.228999999999999,"5min":258.31700000000001,"15min":258.31700000000001},"last":8.4199999999999999}]},{"osd":6,"last update":"Mon Jun 3 09:04:28 2024","interfaces":[{"interface":"back","average":{"1min":14.063000000000001,"5min":24.850000000000001,"15min":106.755},"min":{"1min":9.0820000000000007,"5min":8.9649999999999999,"15min":8.9649999999999999},"max":{"1min":24.666,"5min":188.06700000000001,"15min":9521.0010000000002},"last":183.60900000000001},{"interface":"front","average":{"1min":13.933999999999999,"5min":21.081,"15min":106.142},"min":{"1min":5.8719999999999999,"5min":5.8719999999999999,"15min":5.8719999999999999},"max":{"1min":117.551,"5min":193.887,"15min":9585.6620000000003},"last":184.20599999999999}]},{"osd":7,"last update":"Mon Jun 3 09:06:23 2024","interfaces":[{"interface":"back","average":{"1min":17.056000000000001,"5min":20.055,"15min":23.268999999999998},"min":{"1min":10.353999999999999,"5min":6.6130000000000004,"15min":6.6130000000000004},"max":{"1min":88.974999999999994,"5min":242.56800000000001,"15min":242.56800000000001},"last":12.833},{"interface":"front","average":{"1min":18.885000000000002,"5min":21.460000000000001,"15min":25.489999999999998},"min":{"1min":12.497,"5min":9.0660000000000007,"15min":9.0660000000000007},"max":{"1min":102.229,"5min":181.56399999999999,"15min":214.33799999999999},"last":16.236000000000001}]}]},{"osd":3,"up_from":14,"seq":60129542505,"num_pgs":11,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":58964,"kb_used_data":5704,"kb_used_omap":1,"kb_used_meta":53246,"kb_avail":94312876,"statfs":{"total":96636764160,"available":96576385024,"internally_reserved":0,"allocated":5840896,"data_stored":2740727,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1592,"internal_metadata":54524360},"hb_peers":[0,1,2,4,5,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Mon Jun 3 09:06:25 2024","interfaces":[{"interface":"back","average":{"1min":7.79,"5min":1060.1959999999999,"15min":367.01799999999997},"min":{"1min":5.6909999999999998,"5min":5.173,"15min":4.6929999999999996},"max":{"1min":10.59,"5min":23275.167000000001,"15min":23275.167000000001},"last":11.048999999999999},{"interface":"front","average":{"1min":11.215999999999999,"5min":1066.5799999999999,"15min":372.476},"min":{"1min":8.7080000000000002,"5min":7.2510000000000003,"15min":5.9880000000000004},"max":{"1min":13.499000000000001,"5min":23278.536,"15min":23278.536},"last":13.255000000000001}]},{"osd":1,"last update":"Mon Jun 3 09:06:25 2024","interfaces":[{"interface":"back","average":{"1min":7.3760000000000003,"5min":15.614000000000001,"15min":389.83100000000002},"min":{"1min":4.335,"5min":4.335,"15min":4.335},"max":{"1min":11.289,"5min":250.72300000000001,"15min":21658.205999999998},"last":6.0979999999999999},{"interface":"front","average":{"1min":8.9499999999999993,"5min":14.135,"15min":381.10300000000001},"min":{"1min":6.5609999999999999,"5min":5.4100000000000001,"15min":5.2779999999999996},"max":{"1min":14.51,"5min":180.44499999999999,"15min":21307.069},"last":8.6020000000000003}]},{"osd":2,"last update":"Mon Jun 3 09:06:25 2024","interfaces":[{"interface":"back","average":{"1min":12.343999999999999,"5min":22.059999999999999,"15min":23.715},"min":{"1min":7.9199999999999999,"5min":6.4900000000000002,"15min":6.056},"max":{"1min":15.601000000000001,"5min":377.803,"15min":377.803},"last":12.279999999999999},{"interface":"front","average":{"1min":10.319000000000001,"5min":19.364999999999998,"15min":20.242999999999999},"min":{"1min":6.891,"5min":6.1749999999999998,"15min":5.9020000000000001},"max":{"1min":15.432,"5min":371.601,"15min":371.601},"last":6.9720000000000004}]},{"osd":4,"last update":"Mon Jun 3 09:06:31 2024","interfaces":[{"interface":"back","average":{"1min":9.7460000000000004,"5min":891.23000000000002,"15min":416.48099999999999},"min":{"1min":5.6539999999999999,"5min":5.6539999999999999,"15min":5.6539999999999999},"max":{"1min":15.445,"5min":29040.635999999999,"15min":30144.939999999999},"last":7.7599999999999998},{"interface":"front","average":{"1min":12.398999999999999,"5min":897.58500000000004,"15min":421.27499999999998},"min":{"1min":8.9469999999999992,"5min":7.8109999999999999,"15min":6.2439999999999998},"max":{"1min":16.518000000000001,"5min":29076.955999999998,"15min":30180.689999999999},"last":11.598000000000001}]},{"osd":5,"last update":"Mon Jun 3 09:06:07 2024","interfaces":[{"interface":"back","average":{"1min":12.802,"5min":19.640999999999998,"15min":22.318000000000001},"min":{"1min":9.0890000000000004,"5min":6.8810000000000002,"15min":6.7690000000000001},"max":{"1min":16.745999999999999,"5min":249.88,"15min":327.34899999999999},"last":14.084},{"interface":"front","average":{"1min":15.002000000000001,"5min":24.003,"15min":26.649999999999999},"min":{"1min":11.984,"5min":9.9149999999999991,"15min":8.9879999999999995},"max":{"1min":18.407,"5min":251.47800000000001,"15min":331.86799999999999},"last":14.791}]},{"osd":6,"last update":"Mon Jun 3 09:04:04 2024","interfaces":[{"interface":"back","average":{"1min":22.815999999999999,"5min":31.513999999999999,"15min":181.74299999999999},"min":{"1min":9.4809999999999999,"5min":7.2240000000000002,"15min":7.2240000000000002},"max":{"1min":160.56999999999999,"5min":380.613,"15min":11893.74},"last":19.359999999999999},{"interface":"front","average":{"1min":25.359999999999999,"5min":30.981000000000002,"15min":179.81399999999999},"min":{"1min":9.8870000000000005,"5min":7.3360000000000003,"15min":6.0899999999999999},"max":{"1min":193.11099999999999,"5min":379.80700000000002,"15min":11893.069},"last":8.202}]},{"osd":7,"last update":"Mon Jun 3 09:06:25 2024","interfaces":[{"interface":"back","average":{"1min":11.321999999999999,"5min":18.443999999999999,"15min":21.568999999999999},"min":{"1min":8.109,"5min":8.109,"15min":7.4279999999999999},"max":{"1min":17.113,"5min":146.267,"15min":178.76300000000001},"last":9.4009999999999998},{"interface":"front","average":{"1min":12.512,"5min":20.065999999999999,"15min":24.190000000000001},"min":{"1min":7.4989999999999997,"5min":7.4989999999999997,"15min":7.4989999999999997},"max":{"1min":18.047999999999998,"5min":148.13800000000001,"15min":417.22800000000001},"last":10.007}]}]},{"osd":1,"up_from":14,"seq":60129542503,"num_pgs":6,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":59356,"kb_used_data":6096,"kb_used_omap":1,"kb_used_meta":53246,"kb_avail":94312484,"statfs":{"total":96636764160,"available":96575983616,"internally_reserved":0,"allocated":6242304,"data_stored":3138567,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":54524357},"hb_peers":[0,2,3,4,5,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Mon Jun 3 09:06:07 2024","interfaces":[{"interface":"back","average":{"1min":7.7530000000000001,"5min":185.66999999999999,"15min":77.347999999999999},"min":{"1min":5.0229999999999997,"5min":4.9279999999999999,"15min":4.5449999999999999},"max":{"1min":11.896000000000001,"5min":7470.616,"15min":7470.616},"last":4.9960000000000004},{"interface":"front","average":{"1min":9.9220000000000006,"5min":187.14599999999999,"15min":76.325999999999993},"min":{"1min":5.3159999999999998,"5min":5.3159999999999998,"15min":5.3159999999999998},"max":{"1min":18.338000000000001,"5min":7477.0829999999996,"15min":7477.0829999999996},"last":12.051}]},{"osd":2,"last update":"Mon Jun 3 09:06:20 2024","interfaces":[{"interface":"back","average":{"1min":10.340999999999999,"5min":17.369,"15min":23.670999999999999},"min":{"1min":6.9740000000000002,"5min":5.8650000000000002,"15min":5.8650000000000002},"max":{"1min":15.452,"5min":228,"15min":657.38},"last":8.8550000000000004},{"interface":"front","average":{"1min":13.387,"5min":23.355,"15min":31.524000000000001},"min":{"1min":9.077,"5min":9.077,"15min":7.2160000000000002},"max":{"1min":19.701000000000001,"5min":229.465,"15min":700.06500000000005},"last":12.859999999999999}]},{"osd":3,"last update":"Mon Jun 3 09:06:04 2024","interfaces":[{"interface":"back","average":{"1min":8.6110000000000007,"5min":15.733000000000001,"15min":26.646999999999998},"min":{"1min":5.827,"5min":5.4640000000000004,"15min":5.4640000000000004},"max":{"1min":13.105,"5min":121.619,"15min":1676.009},"last":8.6129999999999995},{"interface":"front","average":{"1min":9.5850000000000009,"5min":16.161999999999999,"15min":29.065999999999999},"min":{"1min":6.3630000000000004,"5min":5.7699999999999996,"15min":5.7699999999999996},"max":{"1min":15.166,"5min":132.036,"15min":1766.5840000000001},"last":7.2569999999999997}]},{"osd":4,"last update":"Mon Jun 3 09:06:35 2024","interfaces":[{"interface":"back","average":{"1min":8.6400000000000006,"5min":641.85500000000002,"15min":334.71499999999997},"min":{"1min":6.7640000000000002,"5min":6.1760000000000002,"15min":6.0970000000000004},"max":{"1min":11.811,"5min":26340.968000000001,"15min":28044.862000000001},"last":6.7640000000000002},{"interface":"front","average":{"1min":12.361000000000001,"5min":648.59900000000005,"15min":338.48500000000001},"min":{"1min":9.8949999999999996,"5min":9.3940000000000001,"15min":6.4640000000000004},"max":{"1min":15.507,"5min":26342.311000000002,"15min":28045.624},"last":10.343}]},{"osd":5,"last update":"Mon Jun 3 09:06:04 2024","interfaces":[{"interface":"back","average":{"1min":12.478999999999999,"5min":20.228000000000002,"15min":25.997},"min":{"1min":8.6430000000000007,"5min":7.9720000000000004,"15min":7.234},"max":{"1min":17.442,"5min":139.43199999999999,"15min":701.22000000000003},"last":11.173},{"interface":"front","average":{"1min":13.507999999999999,"5min":25.760999999999999,"15min":27.553999999999998},"min":{"1min":10.042999999999999,"5min":7.258,"15min":6.242},"max":{"1min":20.611000000000001,"5min":278.63,"15min":598.35000000000002},"last":10.097}]},{"osd":6,"last update":"Mon Jun 3 09:03:54 2024","interfaces":[{"interface":"back","average":{"1min":28.524000000000001,"5min":20.654,"15min":23.878},"min":{"1min":6.0309999999999997,"5min":6.0309999999999997,"15min":6.0309999999999997},"max":{"1min":167.221,"5min":167.221,"15min":704.245},"last":18.721},{"interface":"front","average":{"1min":27.606000000000002,"5min":30.131,"15min":28.628},"min":{"1min":9.8650000000000002,"5min":9.7200000000000006,"15min":8.3580000000000005},"max":{"1min":123.938,"5min":358.23000000000002,"15min":634.05499999999995},"last":173.113}]},{"osd":7,"last update":"Mon Jun 3 09:06:20 2024","interfaces":[{"interface":"back","average":{"1min":14.66,"5min":23.234999999999999,"15min":31.097000000000001},"min":{"1min":11.250999999999999,"5min":8.0559999999999992,"15min":8.0559999999999992},"max":{"1min":21.507999999999999,"5min":270.95100000000002,"15min":703.07799999999997},"last":13.523999999999999},{"interface":"front","average":{"1min":15.548,"5min":28.760000000000002,"15min":28.925000000000001},"min":{"1min":12.759,"5min":8.2089999999999996,"15min":7.7599999999999998},"max":{"1min":22.408000000000001,"5min":360.99900000000002,"15min":608.44100000000003},"last":14.153}]}]},{"osd":0,"up_from":14,"seq":60129542505,"num_pgs":10,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":81108,"kb_used_data":5704,"kb_used_omap":1,"kb_used_meta":75390,"kb_avail":94290732,"statfs":{"total":96636764160,"available":96553709568,"internally_reserved":0,"allocated":5840896,"data_stored":2740727,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1593,"internal_metadata":77199815},"hb_peers":[1,2,3,4,5,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[{"osd":1,"last update":"Mon Jun 3 09:06:04 2024","interfaces":[{"interface":"back","average":{"1min":6.9960000000000004,"5min":56.997,"15min":339.29700000000003},"min":{"1min":4.4530000000000003,"5min":4.4530000000000003,"15min":4.4100000000000001},"max":{"1min":11.720000000000001,"5min":2414.4749999999999,"15min":20691.453000000001},"last":5.891},{"interface":"front","average":{"1min":9.548,"5min":57.258000000000003,"15min":337.791},"min":{"1min":5.2300000000000004,"5min":5.2300000000000004,"15min":4.6050000000000004},"max":{"1min":15.337999999999999,"5min":2412.9549999999999,"15min":20640.792000000001},"last":25.372}]},{"osd":2,"last update":"Mon Jun 3 09:06:22 2024","interfaces":[{"interface":"back","average":{"1min":11.154,"5min":26.515999999999998,"15min":22.882000000000001},"min":{"1min":5.7869999999999999,"5min":5.766,"15min":5.766},"max":{"1min":72.269999999999996,"5min":627.40700000000004,"15min":627.40700000000004},"last":22.283999999999999},{"interface":"front","average":{"1min":14.353999999999999,"5min":33.304000000000002,"15min":29.885000000000002},"min":{"1min":8.2149999999999999,"5min":7.8630000000000004,"15min":7.4089999999999998},"max":{"1min":73.418000000000006,"5min":635.62599999999998,"15min":635.62599999999998},"last":29.343}]},{"osd":3,"last update":"Mon Jun 3 09:06:04 2024","interfaces":[{"interface":"back","average":{"1min":12.105,"5min":26.925000000000001,"15min":26.584},"min":{"1min":7.8410000000000002,"5min":7.1280000000000001,"15min":6.1909999999999998},"max":{"1min":18.425999999999998,"5min":389.72699999999998,"15min":389.72699999999998},"last":27.023},{"interface":"front","average":{"1min":13.794,"5min":29.771999999999998,"15min":28.815000000000001},"min":{"1min":10.084,"5min":9.4920000000000009,"15min":6.1849999999999996},"max":{"1min":17.373999999999999,"5min":390.23200000000003,"15min":494.92899999999997},"last":31.582999999999998}]},{"osd":4,"last update":"Mon Jun 3 09:06:27 2024","interfaces":[{"interface":"back","average":{"1min":11.352,"5min":745.30600000000004,"15min":363.48099999999999},"min":{"1min":6.415,"5min":6.415,"15min":6.415},"max":{"1min":16.724,"5min":20248.616000000002,"15min":25555.212},"last":26.516999999999999},{"interface":"front","average":{"1min":13.545,"5min":744.76999999999998,"15min":361.03100000000001},"min":{"1min":9.3889999999999993,"5min":8.8940000000000001,"15min":7.1429999999999998},"max":{"1min":18.411000000000001,"5min":20250.195,"15min":25556.159},"last":27.510000000000002}]},{"osd":5,"last update":"Mon Jun 3 09:06:17 2024","interfaces":[{"interface":"back","average":{"1min":9.7040000000000006,"5min":26.533999999999999,"15min":28.683},"min":{"1min":6.8129999999999997,"5min":6.8129999999999997,"15min":6.8129999999999997},"max":{"1min":14.942,"5min":455.78699999999998,"15min":605.05899999999997},"last":22.044},{"interface":"front","average":{"1min":14.279,"5min":28.398,"15min":26.192},"min":{"1min":11.696,"5min":6.4420000000000002,"15min":6.2030000000000003},"max":{"1min":17.879000000000001,"5min":391.31200000000001,"15min":541.11000000000001},"last":29.972000000000001}]},{"osd":6,"last update":"Mon Jun 3 09:04:30 2024","interfaces":[{"interface":"back","average":{"1min":52.496000000000002,"5min":33.139000000000003,"15min":160.83199999999999},"min":{"1min":7.5289999999999999,"5min":7.1040000000000001,"15min":6.3739999999999997},"max":{"1min":457.91699999999997,"5min":457.91699999999997,"15min":11115.258},"last":18.959},{"interface":"front","average":{"1min":49.670000000000002,"5min":33.704999999999998,"15min":162.791},"min":{"1min":12.247999999999999,"5min":11.019,"15min":9.9760000000000009},"max":{"1min":407.05099999999999,"5min":407.05099999999999,"15min":11092.194},"last":17.707000000000001}]},{"osd":7,"last update":"Mon Jun 3 09:06:22 2024","interfaces":[{"interface":"back","average":{"1min":11.625999999999999,"5min":26.375,"15min":28.972999999999999},"min":{"1min":6.4480000000000004,"5min":6.4480000000000004,"15min":6.4480000000000004},"max":{"1min":15.352,"5min":461.41699999999997,"15min":598.33699999999999},"last":26.029},{"interface":"front","average":{"1min":14.013,"5min":29.696000000000002,"15min":33.630000000000003},"min":{"1min":10.938000000000001,"5min":10.58,"15min":10.263999999999999},"max":{"1min":16.603000000000002,"5min":463.10599999999999,"15min":592.80499999999995},"last":30.530000000000001}]}]}],"pool_statfs":[{"poolid":1,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":401408,"data_stored":397840,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":5,"total":0,"available":0,"internally_reserved":0,"allocated":401408,"data_stored":397840,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":4,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":5,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":6,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":17,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":17,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":17,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":17,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":17,"osd":4,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":17,"osd":5,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":5,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":17,"osd":6,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":5,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":17,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0}]}} 2024-06-03T09:06:42.877 DEBUG:teuthology.orchestra.run.smithi088:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph pg dump --format=json 2024-06-03T09:06:42.978 INFO:teuthology.orchestra.run.smithi088.stderr:2024-06-03T09:06:42.978+0000 7f52234b6640 10 monclient: get_monmap_and_config 2024-06-03T09:06:42.978 INFO:teuthology.orchestra.run.smithi088.stderr:2024-06-03T09:06:42.978+0000 7f52234b6640 10 monclient: build_initial_monmap 2024-06-03T09:06:42.979 INFO:teuthology.orchestra.run.smithi088.stderr:2024-06-03T09:06:42.978+0000 7f52234b6640 10 monclient: monmap: 2024-06-03T09:06:42.979 INFO:teuthology.orchestra.run.smithi088.stderr:epoch 0 2024-06-03T09:06:42.979 INFO:teuthology.orchestra.run.smithi088.stderr:fsid ec3c3fdd-3ac6-4d7d-9412-683130ed8b04 2024-06-03T09:06:42.979 INFO:teuthology.orchestra.run.smithi088.stderr:last_changed 2024-06-03T09:06:42.979791+0000 2024-06-03T09:06:42.979 INFO:teuthology.orchestra.run.smithi088.stderr:created 2024-06-03T09:06:42.979791+0000 2024-06-03T09:06:42.979 INFO:teuthology.orchestra.run.smithi088.stderr:min_mon_release 0 (unknown) 2024-06-03T09:06:42.979 INFO:teuthology.orchestra.run.smithi088.stderr:election_strategy: 3 2024-06-03T09:06:42.979 INFO:teuthology.orchestra.run.smithi088.stderr:0: v1:172.21.15.88:6789/0 mon.noname-a 2024-06-03T09:06:42.979 INFO:teuthology.orchestra.run.smithi088.stderr:1: v1:172.21.15.192:6789/0 mon.noname-c 2024-06-03T09:06:42.979 INFO:teuthology.orchestra.run.smithi088.stderr:2: v1:172.21.15.88:6790/0 mon.noname-b 2024-06-03T09:06:42.979 INFO:teuthology.orchestra.run.smithi088.stderr: 2024-06-03T09:06:42.981 INFO:teuthology.orchestra.run.smithi088.stderr:2024-06-03T09:06:42.979+0000 7f52234b6640 1 Processor -- start 2024-06-03T09:06:42.981 INFO:teuthology.orchestra.run.smithi088.stderr:2024-06-03T09:06:42.979+0000 7f52234b6640 1 -- start start 2024-06-03T09:06:42.981 INFO:teuthology.orchestra.run.smithi088.stderr:2024-06-03T09:06:42.979+0000 7f52234b6640 10 monclient: init 2024-06-03T09:06:42.981 INFO:teuthology.orchestra.run.smithi088.stderr:2024-06-03T09:06:42.980+0000 7f52234b6640 10 monclient: _reopen_session rank -1 2024-06-03T09:06:42.981 INFO:teuthology.orchestra.run.smithi088.stderr:2024-06-03T09:06:42.980+0000 7f52234b6640 10 monclient: _add_conns ranks=[0,2,1] 2024-06-03T09:06:42.981 INFO:teuthology.orchestra.run.smithi088.stderr:2024-06-03T09:06:42.980+0000 7f52234b6640 10 monclient(hunting): picked mon.noname-a con 0x7f521c0a8a60 addr v1:172.21.15.88:6789/0 2024-06-03T09:06:42.981 INFO:teuthology.orchestra.run.smithi088.stderr:2024-06-03T09:06:42.980+0000 7f52234b6640 10 monclient(hunting): picked mon.noname-b con 0x7f521c0b0d00 addr v1:172.21.15.88:6790/0 2024-06-03T09:06:42.981 INFO:teuthology.orchestra.run.smithi088.stderr:2024-06-03T09:06:42.980+0000 7f52234b6640 10 monclient(hunting): picked mon.noname-c con 0x7f521c1451d0 addr v1:172.21.15.192:6789/0 2024-06-03T09:06:42.981 INFO:teuthology.orchestra.run.smithi088.stderr:2024-06-03T09:06:42.980+0000 7f52234b6640 1 -- --> v1:172.21.15.88:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f521c0a9050 con 0x7f521c0a8a60 2024-06-03T09:06:42.981 INFO:teuthology.orchestra.run.smithi088.stderr:2024-06-03T09:06:42.980+0000 7f52234b6640 1 -- --> v1:172.21.15.192:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f521c0a9d90 con 0x7f521c1451d0 2024-06-03T09:06:42.981 INFO:teuthology.orchestra.run.smithi088.stderr:2024-06-03T09:06:42.980+0000 7f52234b6640 1 -- --> v1:172.21.15.88:6790/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f521c0a8e40 con 0x7f521c0b0d00 2024-06-03T09:06:42.982 INFO:teuthology.orchestra.run.smithi088.stderr:2024-06-03T09:06:42.980+0000 7f52234b6640 10 monclient(hunting): _renew_subs 2024-06-03T09:06:42.982 INFO:teuthology.orchestra.run.smithi088.stderr:2024-06-03T09:06:42.980+0000 7f52234b6640 10 monclient(hunting): authenticate will time out at 3058.790211s 2024-06-03T09:06:42.982 INFO:teuthology.orchestra.run.smithi088.stderr:2024-06-03T09:06:42.980+0000 7f5220a51640 1 --1- >> v1:172.21.15.88:6790/0 conn(0x7f521c0b0d00 0x7f521c13e360 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.88:6790/0 says I am v1:172.21.15.88:41292/0 (socket says 172.21.15.88:41292) 2024-06-03T09:06:42.982 INFO:teuthology.orchestra.run.smithi088.stderr:2024-06-03T09:06:42.980+0000 7f5220a51640 1 -- 172.21.15.88:0/380061428 learned_addr learned my addr 172.21.15.88:0/380061428 (peer_addr_for_me v1:172.21.15.88:0/0) 2024-06-03T09:06:42.984 INFO:teuthology.orchestra.run.smithi088.stderr:2024-06-03T09:06:42.983+0000 7f520ffff640 1 -- 172.21.15.88:0/380061428 <== mon.0 v1:172.21.15.88:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 284008639 0 0) 0x7f521c0a9050 con 0x7f521c0a8a60 2024-06-03T09:06:42.984 INFO:teuthology.orchestra.run.smithi088.stderr:2024-06-03T09:06:42.983+0000 7f520ffff640 10 monclient(hunting): _negotiate 2024-06-03T09:06:42.984 INFO:teuthology.orchestra.run.smithi088.stderr:2024-06-03T09:06:42.983+0000 7f520ffff640 10 monclient(hunting): _init_auth method 2 2024-06-03T09:06:42.984 INFO:teuthology.orchestra.run.smithi088.stderr:2024-06-03T09:06:42.983+0000 7f520ffff640 10 monclient(hunting): _init_auth creating new auth 2024-06-03T09:06:42.984 INFO:teuthology.orchestra.run.smithi088.stderr:2024-06-03T09:06:42.983+0000 7f520ffff640 10 monclient(hunting): my global_id is 30678 2024-06-03T09:06:42.984 INFO:teuthology.orchestra.run.smithi088.stderr:2024-06-03T09:06:42.983+0000 7f520ffff640 1 -- 172.21.15.88:0/380061428 --> v1:172.21.15.88:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f5208003520 con 0x7f521c0a8a60 2024-06-03T09:06:42.984 INFO:teuthology.orchestra.run.smithi088.stderr:2024-06-03T09:06:42.983+0000 7f520ffff640 1 -- 172.21.15.88:0/380061428 <== mon.2 v1:172.21.15.88:6790/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 670541417 0 0) 0x7f521c0a8e40 con 0x7f521c0b0d00 2024-06-03T09:06:42.984 INFO:teuthology.orchestra.run.smithi088.stderr:2024-06-03T09:06:42.983+0000 7f520ffff640 10 monclient(hunting): _negotiate 2024-06-03T09:06:42.984 INFO:teuthology.orchestra.run.smithi088.stderr:2024-06-03T09:06:42.983+0000 7f520ffff640 10 monclient(hunting): _init_auth method 2 2024-06-03T09:06:42.984 INFO:teuthology.orchestra.run.smithi088.stderr:2024-06-03T09:06:42.983+0000 7f520ffff640 10 monclient(hunting): _init_auth creating new auth 2024-06-03T09:06:42.985 INFO:teuthology.orchestra.run.smithi088.stderr:2024-06-03T09:06:42.983+0000 7f520ffff640 10 monclient(hunting): my global_id is 30182 2024-06-03T09:06:42.985 INFO:teuthology.orchestra.run.smithi088.stderr:2024-06-03T09:06:42.983+0000 7f520ffff640 1 -- 172.21.15.88:0/380061428 --> v1:172.21.15.88:6790/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f521c0a9050 con 0x7f521c0b0d00 2024-06-03T09:06:42.985 INFO:teuthology.orchestra.run.smithi088.stderr:2024-06-03T09:06:42.983+0000 7f520ffff640 1 -- 172.21.15.88:0/380061428 <== mon.1 v1:172.21.15.192:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 2442754772 0 0) 0x7f521c0a9d90 con 0x7f521c1451d0 2024-06-03T09:06:42.985 INFO:teuthology.orchestra.run.smithi088.stderr:2024-06-03T09:06:42.983+0000 7f520ffff640 10 monclient(hunting): _negotiate 2024-06-03T09:06:42.985 INFO:teuthology.orchestra.run.smithi088.stderr:2024-06-03T09:06:42.983+0000 7f520ffff640 10 monclient(hunting): _init_auth method 2 2024-06-03T09:06:42.985 INFO:teuthology.orchestra.run.smithi088.stderr:2024-06-03T09:06:42.983+0000 7f520ffff640 10 monclient(hunting): _init_auth creating new auth 2024-06-03T09:06:42.985 INFO:teuthology.orchestra.run.smithi088.stderr:2024-06-03T09:06:42.983+0000 7f520ffff640 10 monclient(hunting): my global_id is 30256 2024-06-03T09:06:42.985 INFO:teuthology.orchestra.run.smithi088.stderr:2024-06-03T09:06:42.983+0000 7f520ffff640 1 -- 172.21.15.88:0/380061428 --> v1:172.21.15.192:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f521c0a8e40 con 0x7f521c1451d0 2024-06-03T09:06:42.986 INFO:teuthology.orchestra.run.smithi088.stderr:2024-06-03T09:06:42.986+0000 7f520ffff640 1 -- 172.21.15.88:0/380061428 <== mon.0 v1:172.21.15.88:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 214+0+0 (unknown 3810541648 0 0) 0x7f5208003520 con 0x7f521c0a8a60 2024-06-03T09:06:42.987 INFO:teuthology.orchestra.run.smithi088.stderr:2024-06-03T09:06:42.986+0000 7f520ffff640 1 -- 172.21.15.88:0/380061428 >> v1:172.21.15.88:6790/0 conn(0x7f521c0b0d00 legacy=0x7f521c13e360 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-06-03T09:06:42.987 INFO:teuthology.orchestra.run.smithi088.stderr:2024-06-03T09:06:42.986+0000 7f520ffff640 1 -- 172.21.15.88:0/380061428 >> v1:172.21.15.192:6789/0 conn(0x7f521c1451d0 legacy=0x7f521c13ea70 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-06-03T09:06:42.987 INFO:teuthology.orchestra.run.smithi088.stderr:2024-06-03T09:06:42.986+0000 7f520ffff640 10 monclient: _finish_hunting 0 2024-06-03T09:06:42.987 INFO:teuthology.orchestra.run.smithi088.stderr:2024-06-03T09:06:42.986+0000 7f520ffff640 1 monclient: found mon.noname-a 2024-06-03T09:06:42.987 INFO:teuthology.orchestra.run.smithi088.stderr:2024-06-03T09:06:42.986+0000 7f520ffff640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-06-03T09:06:42.987 INFO:teuthology.orchestra.run.smithi088.stderr:2024-06-03T09:06:42.986+0000 7f520ffff640 10 monclient: _send_mon_message to mon.noname-a at v1:172.21.15.88:6789/0 2024-06-03T09:06:42.987 INFO:teuthology.orchestra.run.smithi088.stderr:2024-06-03T09:06:42.986+0000 7f520ffff640 1 -- 172.21.15.88:0/380061428 --> v1:172.21.15.88:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f521c13f180 con 0x7f521c0a8a60 2024-06-03T09:06:42.987 INFO:teuthology.orchestra.run.smithi088.stderr:2024-06-03T09:06:42.986+0000 7f520ffff640 10 monclient: _finish_auth 0 2024-06-03T09:06:42.987 INFO:teuthology.orchestra.run.smithi088.stderr:2024-06-03T09:06:42.986+0000 7f520ffff640 10 monclient: _check_auth_tickets 2024-06-03T09:06:42.987 INFO:teuthology.orchestra.run.smithi088.stderr:2024-06-03T09:06:42.986+0000 7f520ffff640 20 monclient: _check_auth_rotating not needed by client.admin 2024-06-03T09:06:42.987 INFO:teuthology.orchestra.run.smithi088.stderr:2024-06-03T09:06:42.986+0000 7f520ffff640 1 -- 172.21.15.88:0/380061428 <== mon.0 v1:172.21.15.88:6789/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 2234374065 0 0) 0x7f5214001610 con 0x7f521c0a8a60 2024-06-03T09:06:42.988 INFO:teuthology.orchestra.run.smithi088.stderr:2024-06-03T09:06:42.986+0000 7f520ffff640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-06-03T09:06:42.988 INFO:teuthology.orchestra.run.smithi088.stderr:2024-06-03T09:06:42.986+0000 7f520ffff640 10 monclient: got monmap 1 from mon.noname-a (according to old e1) 2024-06-03T09:06:42.988 INFO:teuthology.orchestra.run.smithi088.stderr:2024-06-03T09:06:42.986+0000 7f520ffff640 10 monclient: dump: 2024-06-03T09:06:42.988 INFO:teuthology.orchestra.run.smithi088.stderr:epoch 1 2024-06-03T09:06:42.988 INFO:teuthology.orchestra.run.smithi088.stderr:fsid ec3c3fdd-3ac6-4d7d-9412-683130ed8b04 2024-06-03T09:06:42.988 INFO:teuthology.orchestra.run.smithi088.stderr:last_changed 2024-06-03T08:34:31.885125+0000 2024-06-03T09:06:42.988 INFO:teuthology.orchestra.run.smithi088.stderr:created 2024-06-03T08:34:31.885125+0000 2024-06-03T09:06:42.988 INFO:teuthology.orchestra.run.smithi088.stderr:min_mon_release 18 (reef) 2024-06-03T09:06:42.988 INFO:teuthology.orchestra.run.smithi088.stderr:election_strategy: 3 2024-06-03T09:06:42.988 INFO:teuthology.orchestra.run.smithi088.stderr:0: v1:172.21.15.88:6789/0 mon.a 2024-06-03T09:06:42.988 INFO:teuthology.orchestra.run.smithi088.stderr:1: v1:172.21.15.192:6789/0 mon.b 2024-06-03T09:06:42.988 INFO:teuthology.orchestra.run.smithi088.stderr:2: v1:172.21.15.88:6790/0 mon.c 2024-06-03T09:06:42.988 INFO:teuthology.orchestra.run.smithi088.stderr: 2024-06-03T09:06:42.989 INFO:teuthology.orchestra.run.smithi088.stderr:2024-06-03T09:06:42.986+0000 7f52234b6640 5 monclient: authenticate success, global_id 30678 2024-06-03T09:06:42.989 INFO:teuthology.orchestra.run.smithi088.stderr:2024-06-03T09:06:42.986+0000 7f52234b6640 20 monclient: get_monmap_and_config waiting for monmap|config 2024-06-03T09:06:42.991 INFO:teuthology.orchestra.run.smithi088.stderr:2024-06-03T09:06:42.988+0000 7f520ffff640 1 -- 172.21.15.88:0/380061428 <== mon.0 v1:172.21.15.88:6789/0 4 ==== config(0 keys) v1 ==== 4+0+0 (unknown 0 0 0) 0x7f5214001830 con 0x7f521c0a8a60 2024-06-03T09:06:42.991 INFO:teuthology.orchestra.run.smithi088.stderr:2024-06-03T09:06:42.988+0000 7f520ffff640 10 monclient: handle_config config(0 keys) v1 2024-06-03T09:06:42.992 INFO:teuthology.orchestra.run.smithi088.stderr:2024-06-03T09:06:42.988+0000 7f520ffff640 1 -- 172.21.15.88:0/380061428 <== mon.0 v1:172.21.15.88:6789/0 5 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 2234374065 0 0) 0x7f5214001b40 con 0x7f521c0a8a60 2024-06-03T09:06:42.992 INFO:teuthology.orchestra.run.smithi088.stderr:2024-06-03T09:06:42.988+0000 7f520ffff640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-06-03T09:06:42.992 INFO:teuthology.orchestra.run.smithi088.stderr:2024-06-03T09:06:42.988+0000 7f520ffff640 10 monclient: got monmap 1 from mon.a (according to old e1) 2024-06-03T09:06:42.992 INFO:teuthology.orchestra.run.smithi088.stderr:2024-06-03T09:06:42.988+0000 7f520ffff640 10 monclient: dump: 2024-06-03T09:06:42.992 INFO:teuthology.orchestra.run.smithi088.stderr:epoch 1 2024-06-03T09:06:42.992 INFO:teuthology.orchestra.run.smithi088.stderr:fsid ec3c3fdd-3ac6-4d7d-9412-683130ed8b04 2024-06-03T09:06:42.992 INFO:teuthology.orchestra.run.smithi088.stderr:last_changed 2024-06-03T08:34:31.885125+0000 2024-06-03T09:06:42.992 INFO:teuthology.orchestra.run.smithi088.stderr:created 2024-06-03T08:34:31.885125+0000 2024-06-03T09:06:42.992 INFO:teuthology.orchestra.run.smithi088.stderr:min_mon_release 18 (reef) 2024-06-03T09:06:42.992 INFO:teuthology.orchestra.run.smithi088.stderr:election_strategy: 3 2024-06-03T09:06:42.992 INFO:teuthology.orchestra.run.smithi088.stderr:0: v1:172.21.15.88:6789/0 mon.a 2024-06-03T09:06:42.992 INFO:teuthology.orchestra.run.smithi088.stderr:1: v1:172.21.15.192:6789/0 mon.b 2024-06-03T09:06:42.993 INFO:teuthology.orchestra.run.smithi088.stderr:2: v1:172.21.15.88:6790/0 mon.c 2024-06-03T09:06:42.993 INFO:teuthology.orchestra.run.smithi088.stderr: 2024-06-03T09:06:42.993 INFO:teuthology.orchestra.run.smithi088.stderr:2024-06-03T09:06:42.988+0000 7f52234b6640 10 monclient: get_monmap_and_config success 2024-06-03T09:06:42.993 INFO:teuthology.orchestra.run.smithi088.stderr:2024-06-03T09:06:42.988+0000 7f52234b6640 4 set_mon_vals no callback set 2024-06-03T09:06:42.993 INFO:teuthology.orchestra.run.smithi088.stderr:2024-06-03T09:06:42.989+0000 7f52234b6640 10 monclient: shutdown 2024-06-03T09:06:42.993 INFO:teuthology.orchestra.run.smithi088.stderr:2024-06-03T09:06:42.989+0000 7f52234b6640 20 monclient: shutdown discarding 0 pending message(s) 2024-06-03T09:06:42.993 INFO:teuthology.orchestra.run.smithi088.stderr:2024-06-03T09:06:42.989+0000 7f52234b6640 1 -- 172.21.15.88:0/380061428 >> v1:172.21.15.88:6789/0 conn(0x7f521c0a8a60 legacy=0x7f521c144ac0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-06-03T09:06:42.993 INFO:teuthology.orchestra.run.smithi088.stderr:2024-06-03T09:06:42.989+0000 7f52234b6640 1 -- 172.21.15.88:0/380061428 shutdown_connections 2024-06-03T09:06:42.993 INFO:teuthology.orchestra.run.smithi088.stderr:2024-06-03T09:06:42.989+0000 7f52234b6640 1 -- 172.21.15.88:0/380061428 >> 172.21.15.88:0/380061428 conn(0x7f521c0b70d0 msgr2=0x7f521c0b74e0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-06-03T09:06:42.993 INFO:teuthology.orchestra.run.smithi088.stderr:2024-06-03T09:06:42.989+0000 7f52234b6640 1 -- 172.21.15.88:0/380061428 shutdown_connections 2024-06-03T09:06:42.993 INFO:teuthology.orchestra.run.smithi088.stderr:2024-06-03T09:06:42.989+0000 7f52234b6640 1 -- 172.21.15.88:0/380061428 wait complete. 2024-06-03T09:06:42.993 INFO:teuthology.orchestra.run.smithi088.stderr:2024-06-03T09:06:42.990+0000 7f52234b6640 10 monclient: build_initial_monmap 2024-06-03T09:06:42.993 INFO:teuthology.orchestra.run.smithi088.stderr:2024-06-03T09:06:42.990+0000 7f52234b6640 10 monclient: monmap: 2024-06-03T09:06:42.994 INFO:teuthology.orchestra.run.smithi088.stderr:epoch 0 2024-06-03T09:06:42.994 INFO:teuthology.orchestra.run.smithi088.stderr:fsid 00000000-0000-0000-0000-000000000000 2024-06-03T09:06:42.994 INFO:teuthology.orchestra.run.smithi088.stderr:last_changed 0.000000 2024-06-03T09:06:42.994 INFO:teuthology.orchestra.run.smithi088.stderr:created 0.000000 2024-06-03T09:06:42.994 INFO:teuthology.orchestra.run.smithi088.stderr:min_mon_release 0 (unknown) 2024-06-03T09:06:42.994 INFO:teuthology.orchestra.run.smithi088.stderr:election_strategy: 1 2024-06-03T09:06:42.994 INFO:teuthology.orchestra.run.smithi088.stderr:0: v1:172.21.15.88:6789/0 mon.noname-a 2024-06-03T09:06:42.994 INFO:teuthology.orchestra.run.smithi088.stderr:1: v1:172.21.15.192:6789/0 mon.noname-b 2024-06-03T09:06:42.994 INFO:teuthology.orchestra.run.smithi088.stderr:2: v1:172.21.15.88:6790/0 mon.noname-c 2024-06-03T09:06:42.994 INFO:teuthology.orchestra.run.smithi088.stderr: 2024-06-03T09:06:42.994 INFO:teuthology.orchestra.run.smithi088.stderr:2024-06-03T09:06:42.990+0000 7f52234b6640 1 librados: starting msgr at 2024-06-03T09:06:42.994 INFO:teuthology.orchestra.run.smithi088.stderr:2024-06-03T09:06:42.990+0000 7f52234b6640 1 librados: starting objecter 2024-06-03T09:06:42.994 INFO:teuthology.orchestra.run.smithi088.stderr:2024-06-03T09:06:42.990+0000 7f52234b6640 1 Processor -- start 2024-06-03T09:06:42.995 INFO:teuthology.orchestra.run.smithi088.stderr:2024-06-03T09:06:42.990+0000 7f52234b6640 1 -- start start 2024-06-03T09:06:42.995 INFO:teuthology.orchestra.run.smithi088.stderr:2024-06-03T09:06:42.990+0000 7f52234b6640 1 librados: setting wanted keys 2024-06-03T09:06:42.995 INFO:teuthology.orchestra.run.smithi088.stderr:2024-06-03T09:06:42.990+0000 7f52234b6640 1 librados: calling monclient init 2024-06-03T09:06:42.995 INFO:teuthology.orchestra.run.smithi088.stderr:2024-06-03T09:06:42.990+0000 7f52234b6640 10 monclient: init 2024-06-03T09:06:42.995 INFO:teuthology.orchestra.run.smithi088.stderr:2024-06-03T09:06:42.991+0000 7f52234b6640 10 monclient: _reopen_session rank -1 2024-06-03T09:06:42.995 INFO:teuthology.orchestra.run.smithi088.stderr:2024-06-03T09:06:42.991+0000 7f52234b6640 10 monclient: _add_conns ranks=[2,1,0] 2024-06-03T09:06:42.995 INFO:teuthology.orchestra.run.smithi088.stderr:2024-06-03T09:06:42.991+0000 7f52234b6640 10 monclient(hunting): picked mon.noname-c con 0x7f521c0a8a60 addr v1:172.21.15.88:6790/0 2024-06-03T09:06:42.995 INFO:teuthology.orchestra.run.smithi088.stderr:2024-06-03T09:06:42.991+0000 7f52234b6640 10 monclient(hunting): picked mon.noname-b con 0x7f521c0b0d00 addr v1:172.21.15.192:6789/0 2024-06-03T09:06:42.995 INFO:teuthology.orchestra.run.smithi088.stderr:2024-06-03T09:06:42.991+0000 7f52234b6640 10 monclient(hunting): picked mon.noname-a con 0x7f521c1451d0 addr v1:172.21.15.88:6789/0 2024-06-03T09:06:42.995 INFO:teuthology.orchestra.run.smithi088.stderr:2024-06-03T09:06:42.991+0000 7f52234b6640 1 -- --> v1:172.21.15.88:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f521c138140 con 0x7f521c1451d0 2024-06-03T09:06:42.995 INFO:teuthology.orchestra.run.smithi088.stderr:2024-06-03T09:06:42.991+0000 7f52234b6640 1 -- --> v1:172.21.15.192:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f521c139a70 con 0x7f521c0b0d00 2024-06-03T09:06:42.995 INFO:teuthology.orchestra.run.smithi088.stderr:2024-06-03T09:06:42.991+0000 7f52234b6640 1 -- --> v1:172.21.15.88:6790/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f521c139d50 con 0x7f521c0a8a60 2024-06-03T09:06:42.996 INFO:teuthology.orchestra.run.smithi088.stderr:2024-06-03T09:06:42.991+0000 7f52234b6640 10 monclient(hunting): _renew_subs 2024-06-03T09:06:42.996 INFO:teuthology.orchestra.run.smithi088.stderr:2024-06-03T09:06:42.991+0000 7f52234b6640 10 monclient(hunting): authenticate will time out at 3058.801068s 2024-06-03T09:06:42.996 INFO:teuthology.orchestra.run.smithi088.stderr:2024-06-03T09:06:42.991+0000 7f5221a53640 1 --1- >> v1:172.21.15.88:6789/0 conn(0x7f521c1451d0 0x7f521c1e18c0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=1).handle_server_banner_and_identify peer v1:172.21.15.88:6789/0 says I am v1:172.21.15.88:54434/0 (socket says 172.21.15.88:54434) 2024-06-03T09:06:42.996 INFO:teuthology.orchestra.run.smithi088.stderr:2024-06-03T09:06:42.991+0000 7f5221252640 1 --1- >> v1:172.21.15.88:6790/0 conn(0x7f521c0a8a60 0x7f521c1385d0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=1).handle_server_banner_and_identify peer v1:172.21.15.88:6790/0 says I am v1:172.21.15.88:41300/0 (socket says 172.21.15.88:41300) 2024-06-03T09:06:42.996 INFO:teuthology.orchestra.run.smithi088.stderr:2024-06-03T09:06:42.991+0000 7f5221a53640 1 -- 172.21.15.88:0/2007844037 learned_addr learned my addr 172.21.15.88:0/2007844037 (peer_addr_for_me v1:172.21.15.88:0/0) 2024-06-03T09:06:42.996 INFO:teuthology.orchestra.run.smithi088.stderr:2024-06-03T09:06:42.991+0000 7f520dffb640 10 client.?.objecter ms_handle_connect 0x7f521c1451d0 2024-06-03T09:06:42.996 INFO:teuthology.orchestra.run.smithi088.stderr:2024-06-03T09:06:42.991+0000 7f520dffb640 10 client.?.objecter resend_mon_ops 2024-06-03T09:06:42.996 INFO:teuthology.orchestra.run.smithi088.stderr:2024-06-03T09:06:42.991+0000 7f520dffb640 10 client.?.objecter ms_handle_connect 0x7f521c0a8a60 2024-06-03T09:06:42.996 INFO:teuthology.orchestra.run.smithi088.stderr:2024-06-03T09:06:42.991+0000 7f520dffb640 10 client.?.objecter resend_mon_ops 2024-06-03T09:06:42.996 INFO:teuthology.orchestra.run.smithi088.stderr:2024-06-03T09:06:42.992+0000 7f520dffb640 10 client.?.objecter ms_handle_connect 0x7f521c0b0d00 2024-06-03T09:06:42.996 INFO:teuthology.orchestra.run.smithi088.stderr:2024-06-03T09:06:42.992+0000 7f520dffb640 10 client.?.objecter resend_mon_ops 2024-06-03T09:06:42.997 INFO:teuthology.orchestra.run.smithi088.stderr:2024-06-03T09:06:42.993+0000 7f520dffb640 1 -- 172.21.15.88:0/2007844037 <== mon.1 v1:172.21.15.192:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 3158249954 0 0) 0x7f521c139a70 con 0x7f521c0b0d00 2024-06-03T09:06:42.997 INFO:teuthology.orchestra.run.smithi088.stderr:2024-06-03T09:06:42.993+0000 7f520dffb640 10 monclient(hunting): _negotiate 2024-06-03T09:06:42.998 INFO:teuthology.orchestra.run.smithi088.stderr:2024-06-03T09:06:42.993+0000 7f520dffb640 10 monclient(hunting): _init_auth method 2 2024-06-03T09:06:42.998 INFO:teuthology.orchestra.run.smithi088.stderr:2024-06-03T09:06:42.993+0000 7f520dffb640 10 monclient(hunting): _init_auth creating new auth 2024-06-03T09:06:42.998 INFO:teuthology.orchestra.run.smithi088.stderr:2024-06-03T09:06:42.993+0000 7f520dffb640 10 monclient(hunting): my global_id is 30259 2024-06-03T09:06:42.998 INFO:teuthology.orchestra.run.smithi088.stderr:2024-06-03T09:06:42.993+0000 7f520dffb640 1 -- 172.21.15.88:0/2007844037 --> v1:172.21.15.192:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f5200004eb0 con 0x7f521c0b0d00 2024-06-03T09:06:42.998 INFO:teuthology.orchestra.run.smithi088.stderr:2024-06-03T09:06:42.994+0000 7f520dffb640 1 -- 172.21.15.88:0/2007844037 <== mon.0 v1:172.21.15.88:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 3183224999 0 0) 0x7f521c138140 con 0x7f521c1451d0 2024-06-03T09:06:42.998 INFO:teuthology.orchestra.run.smithi088.stderr:2024-06-03T09:06:42.994+0000 7f520dffb640 10 monclient(hunting): _negotiate 2024-06-03T09:06:42.998 INFO:teuthology.orchestra.run.smithi088.stderr:2024-06-03T09:06:42.994+0000 7f520dffb640 10 monclient(hunting): _init_auth method 2 2024-06-03T09:06:42.998 INFO:teuthology.orchestra.run.smithi088.stderr:2024-06-03T09:06:42.994+0000 7f520dffb640 10 monclient(hunting): _init_auth creating new auth 2024-06-03T09:06:42.998 INFO:teuthology.orchestra.run.smithi088.stderr:2024-06-03T09:06:42.994+0000 7f520dffb640 10 monclient(hunting): my global_id is 30681 2024-06-03T09:06:42.998 INFO:teuthology.orchestra.run.smithi088.stderr:2024-06-03T09:06:42.994+0000 7f520dffb640 1 -- 172.21.15.88:0/2007844037 --> v1:172.21.15.88:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f521c139a70 con 0x7f521c1451d0 2024-06-03T09:06:42.998 INFO:teuthology.orchestra.run.smithi088.stderr:2024-06-03T09:06:42.994+0000 7f520dffb640 1 -- 172.21.15.88:0/2007844037 <== mon.2 v1:172.21.15.88:6790/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 818780705 0 0) 0x7f521c139d50 con 0x7f521c0a8a60 2024-06-03T09:06:42.998 INFO:teuthology.orchestra.run.smithi088.stderr:2024-06-03T09:06:42.994+0000 7f520dffb640 10 monclient(hunting): _negotiate 2024-06-03T09:06:42.999 INFO:teuthology.orchestra.run.smithi088.stderr:2024-06-03T09:06:42.994+0000 7f520dffb640 10 monclient(hunting): _init_auth method 2 2024-06-03T09:06:42.999 INFO:teuthology.orchestra.run.smithi088.stderr:2024-06-03T09:06:42.994+0000 7f520dffb640 10 monclient(hunting): _init_auth creating new auth 2024-06-03T09:06:42.999 INFO:teuthology.orchestra.run.smithi088.stderr:2024-06-03T09:06:42.994+0000 7f520dffb640 10 monclient(hunting): my global_id is 30185 2024-06-03T09:06:42.999 INFO:teuthology.orchestra.run.smithi088.stderr:2024-06-03T09:06:42.994+0000 7f520dffb640 1 -- 172.21.15.88:0/2007844037 --> v1:172.21.15.88:6790/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f521c138140 con 0x7f521c0a8a60 2024-06-03T09:06:43.000 INFO:teuthology.orchestra.run.smithi088.stderr:2024-06-03T09:06:42.999+0000 7f520dffb640 1 -- 172.21.15.88:0/2007844037 <== mon.2 v1:172.21.15.88:6790/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 764+0+0 (unknown 671617944 0 0) 0x7f521c138140 con 0x7f521c0a8a60 2024-06-03T09:06:43.000 INFO:teuthology.orchestra.run.smithi088.stderr:2024-06-03T09:06:42.999+0000 7f520dffb640 1 -- 172.21.15.88:0/2007844037 --> v1:172.21.15.88:6790/0 -- auth(proto 2 165 bytes epoch 0) v1 -- 0x7f521c139d50 con 0x7f521c0a8a60 2024-06-03T09:06:43.001 INFO:teuthology.orchestra.run.smithi088.stderr:2024-06-03T09:06:42.999+0000 7f520dffb640 1 -- 172.21.15.88:0/2007844037 <== mon.2 v1:172.21.15.88:6790/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 2234374065 0 0) 0x7f5214001510 con 0x7f521c0a8a60 2024-06-03T09:06:43.001 INFO:teuthology.orchestra.run.smithi088.stderr:2024-06-03T09:06:42.999+0000 7f520dffb640 10 monclient(hunting): handle_monmap mon_map magic: 0 v1 2024-06-03T09:06:43.001 INFO:teuthology.orchestra.run.smithi088.stderr:2024-06-03T09:06:42.999+0000 7f520dffb640 10 monclient(hunting): got monmap 1 from mon.noname-c (according to old e1) 2024-06-03T09:06:43.001 INFO:teuthology.orchestra.run.smithi088.stderr:2024-06-03T09:06:42.999+0000 7f520dffb640 10 monclient(hunting): dump: 2024-06-03T09:06:43.001 INFO:teuthology.orchestra.run.smithi088.stderr:epoch 1 2024-06-03T09:06:43.001 INFO:teuthology.orchestra.run.smithi088.stderr:fsid ec3c3fdd-3ac6-4d7d-9412-683130ed8b04 2024-06-03T09:06:43.001 INFO:teuthology.orchestra.run.smithi088.stderr:last_changed 2024-06-03T08:34:31.885125+0000 2024-06-03T09:06:43.001 INFO:teuthology.orchestra.run.smithi088.stderr:created 2024-06-03T08:34:31.885125+0000 2024-06-03T09:06:43.001 INFO:teuthology.orchestra.run.smithi088.stderr:min_mon_release 18 (reef) 2024-06-03T09:06:43.001 INFO:teuthology.orchestra.run.smithi088.stderr:election_strategy: 3 2024-06-03T09:06:43.001 INFO:teuthology.orchestra.run.smithi088.stderr:0: v1:172.21.15.88:6789/0 mon.a 2024-06-03T09:06:43.001 INFO:teuthology.orchestra.run.smithi088.stderr:1: v1:172.21.15.192:6789/0 mon.b 2024-06-03T09:06:43.001 INFO:teuthology.orchestra.run.smithi088.stderr:2: v1:172.21.15.88:6790/0 mon.c 2024-06-03T09:06:43.002 INFO:teuthology.orchestra.run.smithi088.stderr: 2024-06-03T09:06:43.002 INFO:teuthology.orchestra.run.smithi088.stderr:2024-06-03T09:06:42.999+0000 7f520dffb640 10 monclient(hunting): _finish_auth 0 2024-06-03T09:06:43.002 INFO:teuthology.orchestra.run.smithi088.stderr:2024-06-03T09:06:42.999+0000 7f520dffb640 1 -- 172.21.15.88:0/2007844037 <== mon.1 v1:172.21.15.192:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 764+0+0 (unknown 2479447406 0 0) 0x7f5200004eb0 con 0x7f521c0b0d00 2024-06-03T09:06:43.002 INFO:teuthology.orchestra.run.smithi088.stderr:2024-06-03T09:06:42.999+0000 7f520dffb640 1 -- 172.21.15.88:0/2007844037 --> v1:172.21.15.192:6789/0 -- auth(proto 2 165 bytes epoch 0) v1 -- 0x7f521c138140 con 0x7f521c0b0d00 2024-06-03T09:06:43.002 INFO:teuthology.orchestra.run.smithi088.stderr:2024-06-03T09:06:42.999+0000 7f520dffb640 1 -- 172.21.15.88:0/2007844037 <== mon.1 v1:172.21.15.192:6789/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 2234374065 0 0) 0x7f5218001c30 con 0x7f521c0b0d00 2024-06-03T09:06:43.002 INFO:teuthology.orchestra.run.smithi088.stderr:2024-06-03T09:06:42.999+0000 7f520dffb640 10 monclient(hunting): handle_monmap mon_map magic: 0 v1 2024-06-03T09:06:43.002 INFO:teuthology.orchestra.run.smithi088.stderr:2024-06-03T09:06:42.999+0000 7f520dffb640 10 monclient(hunting): got monmap 1 from mon.b (according to old e1) 2024-06-03T09:06:43.002 INFO:teuthology.orchestra.run.smithi088.stderr:2024-06-03T09:06:42.999+0000 7f520dffb640 10 monclient(hunting): dump: 2024-06-03T09:06:43.002 INFO:teuthology.orchestra.run.smithi088.stderr:epoch 1 2024-06-03T09:06:43.002 INFO:teuthology.orchestra.run.smithi088.stderr:fsid ec3c3fdd-3ac6-4d7d-9412-683130ed8b04 2024-06-03T09:06:43.002 INFO:teuthology.orchestra.run.smithi088.stderr:last_changed 2024-06-03T08:34:31.885125+0000 2024-06-03T09:06:43.002 INFO:teuthology.orchestra.run.smithi088.stderr:created 2024-06-03T08:34:31.885125+0000 2024-06-03T09:06:43.003 INFO:teuthology.orchestra.run.smithi088.stderr:min_mon_release 18 (reef) 2024-06-03T09:06:43.003 INFO:teuthology.orchestra.run.smithi088.stderr:election_strategy: 3 2024-06-03T09:06:43.003 INFO:teuthology.orchestra.run.smithi088.stderr:0: v1:172.21.15.88:6789/0 mon.a 2024-06-03T09:06:43.003 INFO:teuthology.orchestra.run.smithi088.stderr:1: v1:172.21.15.192:6789/0 mon.b 2024-06-03T09:06:43.003 INFO:teuthology.orchestra.run.smithi088.stderr:2: v1:172.21.15.88:6790/0 mon.c 2024-06-03T09:06:43.003 INFO:teuthology.orchestra.run.smithi088.stderr: 2024-06-03T09:06:43.003 INFO:teuthology.orchestra.run.smithi088.stderr:2024-06-03T09:06:42.999+0000 7f520dffb640 1 -- 172.21.15.88:0/2007844037 <== mon.0 v1:172.21.15.88:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 764+0+0 (unknown 3534825294 0 0) 0x7f521c139a70 con 0x7f521c1451d0 2024-06-03T09:06:43.003 INFO:teuthology.orchestra.run.smithi088.stderr:2024-06-03T09:06:42.999+0000 7f520dffb640 1 -- 172.21.15.88:0/2007844037 --> v1:172.21.15.88:6789/0 -- auth(proto 2 165 bytes epoch 0) v1 -- 0x7f5200004eb0 con 0x7f521c1451d0 2024-06-03T09:06:43.003 INFO:teuthology.orchestra.run.smithi088.stderr:2024-06-03T09:06:42.999+0000 7f520dffb640 1 -- 172.21.15.88:0/2007844037 <== mon.0 v1:172.21.15.88:6789/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 2234374065 0 0) 0x7f5210001850 con 0x7f521c1451d0 2024-06-03T09:06:43.003 INFO:teuthology.orchestra.run.smithi088.stderr:2024-06-03T09:06:42.999+0000 7f520dffb640 10 monclient(hunting): handle_monmap mon_map magic: 0 v1 2024-06-03T09:06:43.003 INFO:teuthology.orchestra.run.smithi088.stderr:2024-06-03T09:06:42.999+0000 7f520dffb640 10 monclient(hunting): got monmap 1 from mon.a (according to old e1) 2024-06-03T09:06:43.003 INFO:teuthology.orchestra.run.smithi088.stderr:2024-06-03T09:06:42.999+0000 7f520dffb640 10 monclient(hunting): dump: 2024-06-03T09:06:43.003 INFO:teuthology.orchestra.run.smithi088.stderr:epoch 1 2024-06-03T09:06:43.004 INFO:teuthology.orchestra.run.smithi088.stderr:fsid ec3c3fdd-3ac6-4d7d-9412-683130ed8b04 2024-06-03T09:06:43.004 INFO:teuthology.orchestra.run.smithi088.stderr:last_changed 2024-06-03T08:34:31.885125+0000 2024-06-03T09:06:43.004 INFO:teuthology.orchestra.run.smithi088.stderr:created 2024-06-03T08:34:31.885125+0000 2024-06-03T09:06:43.004 INFO:teuthology.orchestra.run.smithi088.stderr:min_mon_release 18 (reef) 2024-06-03T09:06:43.004 INFO:teuthology.orchestra.run.smithi088.stderr:election_strategy: 3 2024-06-03T09:06:43.004 INFO:teuthology.orchestra.run.smithi088.stderr:0: v1:172.21.15.88:6789/0 mon.a 2024-06-03T09:06:43.004 INFO:teuthology.orchestra.run.smithi088.stderr:1: v1:172.21.15.192:6789/0 mon.b 2024-06-03T09:06:43.004 INFO:teuthology.orchestra.run.smithi088.stderr:2: v1:172.21.15.88:6790/0 mon.c 2024-06-03T09:06:43.004 INFO:teuthology.orchestra.run.smithi088.stderr: 2024-06-03T09:06:43.005 INFO:teuthology.orchestra.run.smithi088.stderr:2024-06-03T09:06:43.003+0000 7f520dffb640 1 -- 172.21.15.88:0/2007844037 <== mon.2 v1:172.21.15.88:6790/0 4 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 580+0+0 (unknown 3181311915 0 0) 0x7f521c139d50 con 0x7f521c0a8a60 2024-06-03T09:06:43.005 INFO:teuthology.orchestra.run.smithi088.stderr:2024-06-03T09:06:43.003+0000 7f520dffb640 1 -- 172.21.15.88:0/2007844037 >> v1:172.21.15.192:6789/0 conn(0x7f521c0b0d00 legacy=0x7f521c1de190 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-06-03T09:06:43.005 INFO:teuthology.orchestra.run.smithi088.stderr:2024-06-03T09:06:43.003+0000 7f520dffb640 1 -- 172.21.15.88:0/2007844037 >> v1:172.21.15.88:6789/0 conn(0x7f521c1451d0 legacy=0x7f521c1e18c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-06-03T09:06:43.005 INFO:teuthology.orchestra.run.smithi088.stderr:2024-06-03T09:06:43.003+0000 7f520dffb640 10 monclient: _finish_hunting 0 2024-06-03T09:06:43.005 INFO:teuthology.orchestra.run.smithi088.stderr:2024-06-03T09:06:43.003+0000 7f520dffb640 1 monclient: found mon.c 2024-06-03T09:06:43.006 INFO:teuthology.orchestra.run.smithi088.stderr:2024-06-03T09:06:43.003+0000 7f520dffb640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-06-03T09:06:43.006 INFO:teuthology.orchestra.run.smithi088.stderr:2024-06-03T09:06:43.003+0000 7f520dffb640 10 monclient: _send_mon_message to mon.c at v1:172.21.15.88:6790/0 2024-06-03T09:06:43.006 INFO:teuthology.orchestra.run.smithi088.stderr:2024-06-03T09:06:43.003+0000 7f520dffb640 1 -- 172.21.15.88:0/2007844037 --> v1:172.21.15.88:6790/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f521c141e70 con 0x7f521c0a8a60 2024-06-03T09:06:43.006 INFO:teuthology.orchestra.run.smithi088.stderr:2024-06-03T09:06:43.003+0000 7f520dffb640 10 monclient: _finish_auth 0 2024-06-03T09:06:43.006 INFO:teuthology.orchestra.run.smithi088.stderr:2024-06-03T09:06:43.003+0000 7f520dffb640 10 monclient: _check_auth_tickets 2024-06-03T09:06:43.006 INFO:teuthology.orchestra.run.smithi088.stderr:2024-06-03T09:06:43.003+0000 7f520dffb640 20 monclient: _check_auth_rotating not needed by client.admin 2024-06-03T09:06:43.006 INFO:teuthology.orchestra.run.smithi088.stderr:2024-06-03T09:06:43.003+0000 7f52234b6640 5 monclient: authenticate success, global_id 30185 2024-06-03T09:06:43.006 INFO:teuthology.orchestra.run.smithi088.stderr:2024-06-03T09:06:43.003+0000 7f52234b6640 10 monclient: _renew_subs 2024-06-03T09:06:43.006 INFO:teuthology.orchestra.run.smithi088.stderr:2024-06-03T09:06:43.003+0000 7f52234b6640 10 monclient: _send_mon_message to mon.c at v1:172.21.15.88:6790/0 2024-06-03T09:06:43.006 INFO:teuthology.orchestra.run.smithi088.stderr:2024-06-03T09:06:43.003+0000 7f52234b6640 1 -- 172.21.15.88:0/2007844037 --> v1:172.21.15.88:6790/0 -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f521c137680 con 0x7f521c0a8a60 2024-06-03T09:06:43.006 INFO:teuthology.orchestra.run.smithi088.stderr:2024-06-03T09:06:43.003+0000 7f52234b6640 10 client.30185.objecter _maybe_request_map subscribing (onetime) to next osd map 2024-06-03T09:06:43.006 INFO:teuthology.orchestra.run.smithi088.stderr:2024-06-03T09:06:43.003+0000 7f52234b6640 10 monclient: _renew_subs 2024-06-03T09:06:43.007 INFO:teuthology.orchestra.run.smithi088.stderr:2024-06-03T09:06:43.003+0000 7f52234b6640 10 monclient: _send_mon_message to mon.c at v1:172.21.15.88:6790/0 2024-06-03T09:06:43.007 INFO:teuthology.orchestra.run.smithi088.stderr:2024-06-03T09:06:43.003+0000 7f52234b6640 1 -- 172.21.15.88:0/2007844037 --> v1:172.21.15.88:6790/0 -- mon_subscribe({osdmap=0}) v3 -- 0x7f521c1414f0 con 0x7f521c0a8a60 2024-06-03T09:06:43.007 INFO:teuthology.orchestra.run.smithi088.stderr:2024-06-03T09:06:43.003+0000 7f52234b6640 1 librados: init done 2024-06-03T09:06:43.007 INFO:teuthology.orchestra.run.smithi088.stderr:2024-06-03T09:06:43.004+0000 7f52234b6640 10 start_mon_command cmd=[{"prefix": "get_command_descriptions"}] 2024-06-03T09:06:43.007 INFO:teuthology.orchestra.run.smithi088.stderr:2024-06-03T09:06:43.004+0000 7f52234b6640 10 monclient: _send_command 1 [{"prefix": "get_command_descriptions"}] 2024-06-03T09:06:43.007 INFO:teuthology.orchestra.run.smithi088.stderr:2024-06-03T09:06:43.004+0000 7f52234b6640 10 monclient: _send_mon_message to mon.c at v1:172.21.15.88:6790/0 2024-06-03T09:06:43.007 INFO:teuthology.orchestra.run.smithi088.stderr:2024-06-03T09:06:43.004+0000 7f52234b6640 1 -- 172.21.15.88:0/2007844037 --> v1:172.21.15.88:6790/0 -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f521c04ea60 con 0x7f521c0a8a60 2024-06-03T09:06:43.219 INFO:teuthology.orchestra.run.smithi088.stderr:2024-06-03T09:06:43.219+0000 7f520dffb640 1 -- 172.21.15.88:0/2007844037 <== mon.2 v1:172.21.15.88:6790/0 5 ==== config(0 keys) v1 ==== 4+0+0 (unknown 0 0 0) 0x7f52140011e0 con 0x7f521c0a8a60 2024-06-03T09:06:43.219 INFO:teuthology.orchestra.run.smithi088.stderr:2024-06-03T09:06:43.219+0000 7f520dffb640 10 monclient: handle_config config(0 keys) v1 2024-06-03T09:06:43.219 INFO:teuthology.orchestra.run.smithi088.stderr:2024-06-03T09:06:43.219+0000 7f520f7fe640 4 set_mon_vals no callback set 2024-06-03T09:06:43.254 INFO:teuthology.orchestra.run.smithi088.stderr:2024-06-03T09:06:43.255+0000 7f520dffb640 1 -- 172.21.15.88:0/2007844037 <== mon.2 v1:172.21.15.88:6790/0 6 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 2234374065 0 0) 0x7f5214003a40 con 0x7f521c0a8a60 2024-06-03T09:06:43.255 INFO:teuthology.orchestra.run.smithi088.stderr:2024-06-03T09:06:43.255+0000 7f520dffb640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-06-03T09:06:43.255 INFO:teuthology.orchestra.run.smithi088.stderr:2024-06-03T09:06:43.255+0000 7f520dffb640 10 monclient: got monmap 1 from mon.c (according to old e1) 2024-06-03T09:06:43.255 INFO:teuthology.orchestra.run.smithi088.stderr:2024-06-03T09:06:43.255+0000 7f520dffb640 10 monclient: dump: 2024-06-03T09:06:43.255 INFO:teuthology.orchestra.run.smithi088.stderr:epoch 1 2024-06-03T09:06:43.255 INFO:teuthology.orchestra.run.smithi088.stderr:fsid ec3c3fdd-3ac6-4d7d-9412-683130ed8b04 2024-06-03T09:06:43.255 INFO:teuthology.orchestra.run.smithi088.stderr:last_changed 2024-06-03T08:34:31.885125+0000 2024-06-03T09:06:43.255 INFO:teuthology.orchestra.run.smithi088.stderr:created 2024-06-03T08:34:31.885125+0000 2024-06-03T09:06:43.255 INFO:teuthology.orchestra.run.smithi088.stderr:min_mon_release 18 (reef) 2024-06-03T09:06:43.255 INFO:teuthology.orchestra.run.smithi088.stderr:election_strategy: 3 2024-06-03T09:06:43.255 INFO:teuthology.orchestra.run.smithi088.stderr:0: v1:172.21.15.88:6789/0 mon.a 2024-06-03T09:06:43.255 INFO:teuthology.orchestra.run.smithi088.stderr:1: v1:172.21.15.192:6789/0 mon.b 2024-06-03T09:06:43.255 INFO:teuthology.orchestra.run.smithi088.stderr:2: v1:172.21.15.88:6790/0 mon.c 2024-06-03T09:06:43.256 INFO:teuthology.orchestra.run.smithi088.stderr: 2024-06-03T09:06:43.256 INFO:teuthology.orchestra.run.smithi088.stderr:2024-06-03T09:06:43.255+0000 7f520dffb640 1 -- 172.21.15.88:0/2007844037 <== mon.2 v1:172.21.15.88:6790/0 7 ==== mgrmap(e 5) v1 ==== 89110+0+0 (unknown 873533973 0 0) 0x7f5214003c20 con 0x7f521c0a8a60 2024-06-03T09:06:43.256 INFO:teuthology.orchestra.run.smithi088.stderr:2024-06-03T09:06:43.256+0000 7f5220a51640 10 monclient: get_auth_request con 0x7f520006d400 auth_method 0 2024-06-03T09:06:43.256 INFO:teuthology.orchestra.run.smithi088.stderr:2024-06-03T09:06:43.256+0000 7f520dffb640 10 client.30185.objecter ms_handle_connect 0x7f520006d400 2024-06-03T09:06:43.262 INFO:teuthology.orchestra.run.smithi088.stderr:2024-06-03T09:06:43.262+0000 7f520dffb640 1 -- 172.21.15.88:0/2007844037 <== mon.2 v1:172.21.15.88:6790/0 8 ==== osd_map(1011..1011 src has 622..1011) v4 ==== 5229+0+0 (unknown 1639111431 0 0) 0x7f5214054100 con 0x7f521c0a8a60 2024-06-03T09:06:43.262 INFO:teuthology.orchestra.run.smithi088.stderr:2024-06-03T09:06:43.262+0000 7f520dffb640 10 client.30185.objecter ms_dispatch 0x7f521c000b90 osd_map(1011..1011 src has 622..1011) v4 2024-06-03T09:06:43.262 INFO:teuthology.orchestra.run.smithi088.stderr:2024-06-03T09:06:43.262+0000 7f520dffb640 3 client.30185.objecter handle_osd_map got epochs [1011,1011] > 0 2024-06-03T09:06:43.262 INFO:teuthology.orchestra.run.smithi088.stderr:2024-06-03T09:06:43.262+0000 7f520dffb640 3 client.30185.objecter handle_osd_map decoding full epoch 1011 2024-06-03T09:06:43.262 INFO:teuthology.orchestra.run.smithi088.stderr:2024-06-03T09:06:43.262+0000 7f520dffb640 20 client.30185.objecter dump_active .. 0 homeless 2024-06-03T09:06:43.263 INFO:teuthology.orchestra.run.smithi088.stderr:2024-06-03T09:06:43.262+0000 7f520dffb640 1 -- 172.21.15.88:0/2007844037 <== mon.2 v1:172.21.15.88:6790/0 9 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+182498 (unknown 1092875540 0 3016760700) 0x7f521404fcb0 con 0x7f521c0a8a60 2024-06-03T09:06:43.263 INFO:teuthology.orchestra.run.smithi088.stderr:2024-06-03T09:06:43.262+0000 7f520dffb640 10 monclient: handle_mon_command_ack 1 [{"prefix": "get_command_descriptions"}] 2024-06-03T09:06:43.263 INFO:teuthology.orchestra.run.smithi088.stderr:2024-06-03T09:06:43.262+0000 7f520dffb640 10 monclient: _finish_command 1 = system:0 2024-06-03T09:06:43.448 INFO:teuthology.orchestra.run.smithi088.stderr:2024-06-03T09:06:43.448+0000 7f52234b6640 1 -- 172.21.15.88:0/2007844037 --> v1:172.21.15.192:6800/34229 -- mgr_command(tid 0: {"prefix": "pg dump", "target": ["mon-mgr", ""], "format": "json"}) v1 -- 0x7f521c0a94e0 con 0x7f520006d400 2024-06-03T09:06:43.452 INFO:teuthology.orchestra.run.smithi088.stderr:2024-06-03T09:06:43.452+0000 7f520dffb640 1 -- 172.21.15.88:0/2007844037 <== mgr.4110 v1:172.21.15.192:6800/34229 1 ==== mgr_command_reply(tid 0: 0 dumped all) v1 ==== 18+0+151536 (unknown 2965378022 0 2674254763) 0x7f52180020d0 con 0x7f520006d400 2024-06-03T09:06:43.454 INFO:teuthology.orchestra.run.smithi088.stdout: 2024-06-03T09:06:43.454 INFO:teuthology.orchestra.run.smithi088.stderr:dumped all 2024-06-03T09:06:43.456 INFO:teuthology.orchestra.run.smithi088.stderr:2024-06-03T09:06:43.454+0000 7f52234b6640 10 librados: watch_flush enter 2024-06-03T09:06:43.456 INFO:teuthology.orchestra.run.smithi088.stderr:2024-06-03T09:06:43.455+0000 7f52234b6640 10 librados: watch_flush exit 2024-06-03T09:06:43.456 INFO:teuthology.orchestra.run.smithi088.stderr:2024-06-03T09:06:43.455+0000 7f52234b6640 20 client.30185.objecter shutdown clearing up homeless session... 2024-06-03T09:06:43.456 INFO:teuthology.orchestra.run.smithi088.stderr:2024-06-03T09:06:43.455+0000 7f52234b6640 10 client.30185.objecter successfully canceled tick 2024-06-03T09:06:43.457 INFO:teuthology.orchestra.run.smithi088.stderr:2024-06-03T09:06:43.455+0000 7f52234b6640 1 -- 172.21.15.88:0/2007844037 >> v1:172.21.15.192:6800/34229 conn(0x7f520006d400 legacy=0x7f520006f8c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-06-03T09:06:43.457 INFO:teuthology.orchestra.run.smithi088.stderr:2024-06-03T09:06:43.455+0000 7f52234b6640 10 monclient: shutdown 2024-06-03T09:06:43.457 INFO:teuthology.orchestra.run.smithi088.stderr:2024-06-03T09:06:43.455+0000 7f52234b6640 20 monclient: shutdown discarding 0 pending message(s) 2024-06-03T09:06:43.457 INFO:teuthology.orchestra.run.smithi088.stderr:2024-06-03T09:06:43.455+0000 7f52234b6640 1 -- 172.21.15.88:0/2007844037 >> v1:172.21.15.88:6790/0 conn(0x7f521c0a8a60 legacy=0x7f521c1385d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-06-03T09:06:43.457 INFO:teuthology.orchestra.run.smithi088.stderr:2024-06-03T09:06:43.455+0000 7f52234b6640 1 -- 172.21.15.88:0/2007844037 shutdown_connections 2024-06-03T09:06:43.457 INFO:teuthology.orchestra.run.smithi088.stderr:2024-06-03T09:06:43.455+0000 7f52234b6640 1 -- 172.21.15.88:0/2007844037 >> 172.21.15.88:0/2007844037 conn(0x7f521c0b70d0 msgr2=0x7f521c0b5060 unknown :-1 s=STATE_NONE l=0).mark_down 2024-06-03T09:06:43.457 INFO:teuthology.orchestra.run.smithi088.stderr:2024-06-03T09:06:43.455+0000 7f52234b6640 1 -- 172.21.15.88:0/2007844037 shutdown_connections 2024-06-03T09:06:43.457 INFO:teuthology.orchestra.run.smithi088.stderr:2024-06-03T09:06:43.455+0000 7f52234b6640 1 -- 172.21.15.88:0/2007844037 wait complete. 2024-06-03T09:06:43.457 INFO:teuthology.orchestra.run.smithi088.stderr:2024-06-03T09:06:43.455+0000 7f52234b6640 1 librados: shutdown 2024-06-03T09:06:43.470 INFO:teuthology.orchestra.run.smithi088.stdout:{"pg_ready":true,"pg_map":{"version":1929,"stamp":"2024-06-03T09:06:43.413439+0000","last_osdmap_epoch":0,"last_pg_scan":0,"pg_stats_sum":{"stat_sum":{"num_bytes":459285,"num_objects":3,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":1,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":3,"num_whiteouts":0,"num_read":102,"num_read_kb":88,"num_write":120,"num_write_kb":1379,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2,"num_bytes_recovered":397840,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":83,"ondisk_log_size":83,"up":68,"acting":68,"num_store_stats":0},"osd_stats_sum":{"up_from":0,"seq":0,"num_pgs":88,"num_osds":8,"num_per_pool_osds":8,"num_per_pool_omap_osds":8,"kb":754974720,"kb_used":494612,"kb_used_data":46388,"kb_used_omap":12,"kb_used_meta":448115,"kb_avail":754480108,"statfs":{"total":773094113280,"available":772587630592,"internally_reserved":0,"allocated":47501312,"data_stored":22782476,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":12744,"internal_metadata":458870328},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":93,"apply_latency_ms":93,"commit_latency_ns":93000000,"apply_latency_ns":93000000},"alerts":[],"network_ping_times":[]},"pg_stats_delta":{"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":0,"acting":0,"num_store_stats":0,"stamp_delta":"8.312065"},"pg_stats":[{"pgid":"17.d","version":"0'0","reported_seq":942,"reported_epoch":1009,"state":"active+clean","last_fresh":"2024-06-03T09:06:14.443917+0000","last_change":"2024-06-03T08:38:19.260548+0000","last_active":"2024-06-03T09:06:14.443917+0000","last_peered":"2024-06-03T09:06:14.443917+0000","last_clean":"2024-06-03T09:06:14.443917+0000","last_became_active":"2024-06-03T08:38:19.253559+0000","last_became_peered":"2024-06-03T08:38:19.253559+0000","last_unstale":"2024-06-03T09:06:14.443917+0000","last_undegraded":"2024-06-03T09:06:14.443917+0000","last_fullsized":"2024-06-03T09:06:14.443917+0000","mapping_epoch":28,"log_start":"0'0","ondisk_log_start":"0'0","created":28,"last_epoch_clean":45,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-06-03T08:37:55.614497+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-06-03T08:37:55.614497+0000","last_clean_scrub_stamp":"2024-06-03T08:37:55.614497+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-06-04T19:00:12.470979+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1],"acting":[0,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"17.c","version":"0'0","reported_seq":953,"reported_epoch":1009,"state":"active+undersized","last_fresh":"2024-06-03T09:06:13.668156+0000","last_change":"2024-06-03T09:06:03.440074+0000","last_active":"2024-06-03T09:06:13.668156+0000","last_peered":"2024-06-03T09:06:13.668156+0000","last_clean":"2024-06-03T09:05:02.105638+0000","last_became_active":"2024-06-03T09:06:03.440074+0000","last_became_peered":"2024-06-03T09:06:03.440074+0000","last_unstale":"2024-06-03T09:06:13.668156+0000","last_undegraded":"2024-06-03T09:06:13.668156+0000","last_fullsized":"2024-06-03T09:06:03.377982+0000","mapping_epoch":999,"log_start":"0'0","ondisk_log_start":"0'0","created":28,"last_epoch_clean":255,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-06-03T08:37:55.614497+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-06-03T08:37:55.614497+0000","last_clean_scrub_stamp":"2024-06-03T08:37:55.614497+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-06-04T10:34:20.808557+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3],"acting":[3],"avail_no_missing":["3"],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"17.f","version":"0'0","reported_seq":148,"reported_epoch":1009,"state":"active+undersized","last_fresh":"2024-06-03T09:06:14.450543+0000","last_change":"2024-06-03T09:06:03.420285+0000","last_active":"2024-06-03T09:06:14.450543+0000","last_peered":"2024-06-03T09:06:14.450543+0000","last_clean":"2024-06-03T08:37:55.614497+0000","last_became_active":"2024-06-03T09:06:03.420285+0000","last_became_peered":"2024-06-03T09:06:03.420285+0000","last_unstale":"2024-06-03T09:06:14.450543+0000","last_undegraded":"2024-06-03T09:06:14.450543+0000","last_fullsized":"2024-06-03T09:06:03.380616+0000","mapping_epoch":999,"log_start":"0'0","ondisk_log_start":"0'0","created":28,"last_epoch_clean":255,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-06-03T08:37:55.614497+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-06-03T08:37:55.614497+0000","last_clean_scrub_stamp":"2024-06-03T08:37:55.614497+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-06-04T15:39:41.187731+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4],"acting":[4],"avail_no_missing":["4"],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"17.e","version":"0'0","reported_seq":149,"reported_epoch":1009,"state":"active+undersized","last_fresh":"2024-06-03T09:06:13.726992+0000","last_change":"2024-06-03T09:06:03.412887+0000","last_active":"2024-06-03T09:06:13.726992+0000","last_peered":"2024-06-03T09:06:13.726992+0000","last_clean":"2024-06-03T08:37:55.614497+0000","last_became_active":"2024-06-03T09:06:03.412887+0000","last_became_peered":"2024-06-03T09:06:03.412887+0000","last_unstale":"2024-06-03T09:06:13.726992+0000","last_undegraded":"2024-06-03T09:06:13.726992+0000","last_fullsized":"2024-06-03T09:06:03.332482+0000","mapping_epoch":999,"log_start":"0'0","ondisk_log_start":"0'0","created":28,"last_epoch_clean":255,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-06-03T08:37:55.614497+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-06-03T08:37:55.614497+0000","last_clean_scrub_stamp":"2024-06-03T08:37:55.614497+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-06-04T12:23:22.511601+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5],"acting":[5],"avail_no_missing":["5"],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"17.9","version":"0'0","reported_seq":946,"reported_epoch":1009,"state":"active+clean","last_fresh":"2024-06-03T09:06:14.999299+0000","last_change":"2024-06-03T08:38:04.772260+0000","last_active":"2024-06-03T09:06:14.999299+0000","last_peered":"2024-06-03T09:06:14.999299+0000","last_clean":"2024-06-03T09:06:14.999299+0000","last_became_active":"2024-06-03T08:38:04.763428+0000","last_became_peered":"2024-06-03T08:38:04.763428+0000","last_unstale":"2024-06-03T09:06:14.999299+0000","last_undegraded":"2024-06-03T09:06:14.999299+0000","last_fullsized":"2024-06-03T09:06:14.999299+0000","mapping_epoch":28,"log_start":"0'0","ondisk_log_start":"0'0","created":28,"last_epoch_clean":33,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-06-03T08:37:55.614497+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-06-03T08:37:55.614497+0000","last_clean_scrub_stamp":"2024-06-03T08:37:55.614497+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-06-04T10:55:10.773936+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,4],"acting":[7,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"17.8","version":"0'0","reported_seq":961,"reported_epoch":1009,"state":"active+undersized","last_fresh":"2024-06-03T09:06:14.430375+0000","last_change":"2024-06-03T09:06:03.388786+0000","last_active":"2024-06-03T09:06:14.430375+0000","last_peered":"2024-06-03T09:06:14.430375+0000","last_clean":"2024-06-03T09:04:43.297780+0000","last_became_active":"2024-06-03T09:06:03.388786+0000","last_became_peered":"2024-06-03T09:06:03.388786+0000","last_unstale":"2024-06-03T09:06:14.430375+0000","last_undegraded":"2024-06-03T09:06:14.430375+0000","last_fullsized":"2024-06-03T09:06:03.331754+0000","mapping_epoch":999,"log_start":"0'0","ondisk_log_start":"0'0","created":28,"last_epoch_clean":255,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-06-03T08:37:55.614497+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-06-03T08:37:55.614497+0000","last_clean_scrub_stamp":"2024-06-03T08:37:55.614497+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-06-04T09:18:44.765547+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0],"acting":[0],"avail_no_missing":["0"],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"17.b","version":"0'0","reported_seq":149,"reported_epoch":1009,"state":"active+undersized","last_fresh":"2024-06-03T09:06:14.433735+0000","last_change":"2024-06-03T09:06:03.451203+0000","last_active":"2024-06-03T09:06:14.433735+0000","last_peered":"2024-06-03T09:06:14.433735+0000","last_clean":"2024-06-03T08:37:55.614497+0000","last_became_active":"2024-06-03T09:06:03.451203+0000","last_became_peered":"2024-06-03T09:06:03.451203+0000","last_unstale":"2024-06-03T09:06:14.433735+0000","last_undegraded":"2024-06-03T09:06:14.433735+0000","last_fullsized":"2024-06-03T09:06:03.375523+0000","mapping_epoch":999,"log_start":"0'0","ondisk_log_start":"0'0","created":28,"last_epoch_clean":255,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-06-03T08:37:55.614497+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-06-03T08:37:55.614497+0000","last_clean_scrub_stamp":"2024-06-03T08:37:55.614497+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-06-04T10:16:19.813495+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2],"acting":[2],"avail_no_missing":["2"],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"17.a","version":"0'0","reported_seq":925,"reported_epoch":1010,"state":"active+clean","last_fresh":"2024-06-03T09:06:18.651697+0000","last_change":"2024-06-03T08:38:05.843142+0000","last_active":"2024-06-03T09:06:18.651697+0000","last_peered":"2024-06-03T09:06:18.651697+0000","last_clean":"2024-06-03T09:06:18.651697+0000","last_became_active":"2024-06-03T08:38:05.809257+0000","last_became_peered":"2024-06-03T08:38:05.809257+0000","last_unstale":"2024-06-03T09:06:18.651697+0000","last_undegraded":"2024-06-03T09:06:18.651697+0000","last_fullsized":"2024-06-03T09:06:18.651697+0000","mapping_epoch":28,"log_start":"0'0","ondisk_log_start":"0'0","created":28,"last_epoch_clean":33,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-06-03T08:37:55.614497+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-06-03T08:37:55.614497+0000","last_clean_scrub_stamp":"2024-06-03T08:37:55.614497+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-06-04T12:12:03.686838+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2],"acting":[1,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"17.5","version":"0'0","reported_seq":943,"reported_epoch":1009,"state":"active+clean","last_fresh":"2024-06-03T09:06:14.442357+0000","last_change":"2024-06-03T08:38:23.466051+0000","last_active":"2024-06-03T09:06:14.442357+0000","last_peered":"2024-06-03T09:06:14.442357+0000","last_clean":"2024-06-03T09:06:14.442357+0000","last_became_active":"2024-06-03T08:38:23.458039+0000","last_became_peered":"2024-06-03T08:38:23.458039+0000","last_unstale":"2024-06-03T09:06:14.442357+0000","last_undegraded":"2024-06-03T09:06:14.442357+0000","last_fullsized":"2024-06-03T09:06:14.442357+0000","mapping_epoch":28,"log_start":"0'0","ondisk_log_start":"0'0","created":28,"last_epoch_clean":49,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-06-03T08:37:55.614497+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-06-03T08:37:55.614497+0000","last_clean_scrub_stamp":"2024-06-03T08:37:55.614497+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-06-04T12:27:02.532436+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,4],"acting":[0,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"17.4","version":"0'0","reported_seq":149,"reported_epoch":1009,"state":"active+undersized","last_fresh":"2024-06-03T09:06:13.941522+0000","last_change":"2024-06-03T09:06:03.441889+0000","last_active":"2024-06-03T09:06:13.941522+0000","last_peered":"2024-06-03T09:06:13.941522+0000","last_clean":"2024-06-03T08:37:55.614497+0000","last_became_active":"2024-06-03T09:06:03.441889+0000","last_became_peered":"2024-06-03T09:06:03.441889+0000","last_unstale":"2024-06-03T09:06:13.941522+0000","last_undegraded":"2024-06-03T09:06:13.941522+0000","last_fullsized":"2024-06-03T09:06:03.359582+0000","mapping_epoch":999,"log_start":"0'0","ondisk_log_start":"0'0","created":28,"last_epoch_clean":255,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-06-03T08:37:55.614497+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-06-03T08:37:55.614497+0000","last_clean_scrub_stamp":"2024-06-03T08:37:55.614497+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-06-04T12:49:22.907141+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7],"acting":[7],"avail_no_missing":["7"],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"17.7","version":"0'0","reported_seq":929,"reported_epoch":1009,"state":"active+clean","last_fresh":"2024-06-03T09:06:13.736920+0000","last_change":"2024-06-03T08:38:04.837083+0000","last_active":"2024-06-03T09:06:13.736920+0000","last_peered":"2024-06-03T09:06:13.736920+0000","last_clean":"2024-06-03T09:06:13.736920+0000","last_became_active":"2024-06-03T08:38:04.818246+0000","last_became_peered":"2024-06-03T08:38:04.818246+0000","last_unstale":"2024-06-03T09:06:13.736920+0000","last_undegraded":"2024-06-03T09:06:13.736920+0000","last_fullsized":"2024-06-03T09:06:13.736920+0000","mapping_epoch":28,"log_start":"0'0","ondisk_log_start":"0'0","created":28,"last_epoch_clean":34,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-06-03T08:37:55.614497+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-06-03T08:37:55.614497+0000","last_clean_scrub_stamp":"2024-06-03T08:37:55.614497+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-06-04T17:02:38.204813+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,2],"acting":[5,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"17.6","version":"0'0","reported_seq":960,"reported_epoch":1009,"state":"active+undersized","last_fresh":"2024-06-03T09:06:14.453110+0000","last_change":"2024-06-03T09:06:03.428677+0000","last_active":"2024-06-03T09:06:14.453110+0000","last_peered":"2024-06-03T09:06:14.453110+0000","last_clean":"2024-06-03T09:04:43.312352+0000","last_became_active":"2024-06-03T09:06:03.428677+0000","last_became_peered":"2024-06-03T09:06:03.428677+0000","last_unstale":"2024-06-03T09:06:14.453110+0000","last_undegraded":"2024-06-03T09:06:14.453110+0000","last_fullsized":"2024-06-03T09:06:03.359909+0000","mapping_epoch":999,"log_start":"0'0","ondisk_log_start":"0'0","created":28,"last_epoch_clean":255,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-06-03T08:37:55.614497+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-06-03T08:37:55.614497+0000","last_clean_scrub_stamp":"2024-06-03T08:37:55.614497+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-06-04T15:38:02.180241+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0],"acting":[0],"avail_no_missing":["0"],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"2.7","version":"0'0","reported_seq":940,"reported_epoch":1009,"state":"active+clean","last_fresh":"2024-06-03T09:06:13.684065+0000","last_change":"2024-06-03T08:36:30.385405+0000","last_active":"2024-06-03T09:06:13.684065+0000","last_peered":"2024-06-03T09:06:13.684065+0000","last_clean":"2024-06-03T09:06:13.684065+0000","last_became_active":"2024-06-03T08:36:30.380302+0000","last_became_peered":"2024-06-03T08:36:30.380302+0000","last_unstale":"2024-06-03T09:06:13.684065+0000","last_undegraded":"2024-06-03T09:06:13.684065+0000","last_fullsized":"2024-06-03T09:06:13.684065+0000","mapping_epoch":23,"log_start":"0'0","ondisk_log_start":"0'0","created":23,"last_epoch_clean":24,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-06-03T08:36:28.444764+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-06-03T08:36:28.444764+0000","last_clean_scrub_stamp":"2024-06-03T08:36:28.444764+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-06-04T18:46:33.475181+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0],"acting":[3,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"17.14","version":"0'0","reported_seq":926,"reported_epoch":1009,"state":"active+clean","last_fresh":"2024-06-03T09:06:13.721460+0000","last_change":"2024-06-03T08:38:04.547565+0000","last_active":"2024-06-03T09:06:13.721460+0000","last_peered":"2024-06-03T09:06:13.721460+0000","last_clean":"2024-06-03T09:06:13.721460+0000","last_became_active":"2024-06-03T08:38:04.541781+0000","last_became_peered":"2024-06-03T08:38:04.541781+0000","last_unstale":"2024-06-03T09:06:13.721460+0000","last_undegraded":"2024-06-03T09:06:13.721460+0000","last_fullsized":"2024-06-03T09:06:13.721460+0000","mapping_epoch":28,"log_start":"0'0","ondisk_log_start":"0'0","created":28,"last_epoch_clean":34,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-06-03T08:37:55.614497+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-06-03T08:37:55.614497+0000","last_clean_scrub_stamp":"2024-06-03T08:37:55.614497+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-06-04T16:41:55.170134+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,2],"acting":[5,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"2.6","version":"0'0","reported_seq":949,"reported_epoch":1009,"state":"active+clean","last_fresh":"2024-06-03T09:06:14.448100+0000","last_change":"2024-06-03T08:36:30.430760+0000","last_active":"2024-06-03T09:06:14.448100+0000","last_peered":"2024-06-03T09:06:14.448100+0000","last_clean":"2024-06-03T09:06:14.448100+0000","last_became_active":"2024-06-03T08:36:30.404665+0000","last_became_peered":"2024-06-03T08:36:30.404665+0000","last_unstale":"2024-06-03T09:06:14.448100+0000","last_undegraded":"2024-06-03T09:06:14.448100+0000","last_fullsized":"2024-06-03T09:06:14.448100+0000","mapping_epoch":23,"log_start":"0'0","ondisk_log_start":"0'0","created":23,"last_epoch_clean":24,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-06-03T08:36:28.444764+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-06-03T08:36:28.444764+0000","last_clean_scrub_stamp":"2024-06-03T08:36:28.444764+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-06-04T10:37:37.746074+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,0],"acting":[4,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"17.15","version":"0'0","reported_seq":934,"reported_epoch":1009,"state":"active+clean","last_fresh":"2024-06-03T09:06:13.672940+0000","last_change":"2024-06-03T08:38:20.999903+0000","last_active":"2024-06-03T09:06:13.672940+0000","last_peered":"2024-06-03T09:06:13.672940+0000","last_clean":"2024-06-03T09:06:13.672940+0000","last_became_active":"2024-06-03T08:38:20.979523+0000","last_became_peered":"2024-06-03T08:38:20.979523+0000","last_unstale":"2024-06-03T09:06:13.672940+0000","last_undegraded":"2024-06-03T09:06:13.672940+0000","last_fullsized":"2024-06-03T09:06:13.672940+0000","mapping_epoch":28,"log_start":"0'0","ondisk_log_start":"0'0","created":28,"last_epoch_clean":47,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-06-03T08:37:55.614497+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-06-03T08:37:55.614497+0000","last_clean_scrub_stamp":"2024-06-03T08:37:55.614497+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-06-04T20:29:58.314289+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,1],"acting":[3,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"2.5","version":"0'0","reported_seq":942,"reported_epoch":1010,"state":"active+clean","last_fresh":"2024-06-03T09:06:18.382767+0000","last_change":"2024-06-03T08:36:30.024235+0000","last_active":"2024-06-03T09:06:18.382767+0000","last_peered":"2024-06-03T09:06:18.382767+0000","last_clean":"2024-06-03T09:06:18.382767+0000","last_became_active":"2024-06-03T08:36:30.019040+0000","last_became_peered":"2024-06-03T08:36:30.019040+0000","last_unstale":"2024-06-03T09:06:18.382767+0000","last_undegraded":"2024-06-03T09:06:18.382767+0000","last_fullsized":"2024-06-03T09:06:18.382767+0000","mapping_epoch":23,"log_start":"0'0","ondisk_log_start":"0'0","created":23,"last_epoch_clean":24,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-06-03T08:36:28.444764+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-06-03T08:36:28.444764+0000","last_clean_scrub_stamp":"2024-06-03T08:36:28.444764+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-06-04T15:46:47.109502+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,4],"acting":[3,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"17.16","version":"0'0","reported_seq":972,"reported_epoch":1009,"state":"active+undersized","last_fresh":"2024-06-03T09:06:14.441665+0000","last_change":"2024-06-03T09:06:03.478615+0000","last_active":"2024-06-03T09:06:14.441665+0000","last_peered":"2024-06-03T09:06:14.441665+0000","last_clean":"2024-06-03T09:04:43.328415+0000","last_became_active":"2024-06-03T09:06:03.478615+0000","last_became_peered":"2024-06-03T09:06:03.478615+0000","last_unstale":"2024-06-03T09:06:14.441665+0000","last_undegraded":"2024-06-03T09:06:14.441665+0000","last_fullsized":"2024-06-03T09:06:03.402504+0000","mapping_epoch":999,"log_start":"0'0","ondisk_log_start":"0'0","created":28,"last_epoch_clean":255,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-06-03T08:37:55.614497+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-06-03T08:37:55.614497+0000","last_clean_scrub_stamp":"2024-06-03T08:37:55.614497+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-06-04T08:47:08.079233+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2],"acting":[2],"avail_no_missing":["2"],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"2.4","version":"0'0","reported_seq":952,"reported_epoch":1010,"state":"active+undersized","last_fresh":"2024-06-03T09:06:18.331485+0000","last_change":"2024-06-03T09:06:03.387171+0000","last_active":"2024-06-03T09:06:18.331485+0000","last_peered":"2024-06-03T09:06:18.331485+0000","last_clean":"2024-06-03T09:04:43.202645+0000","last_became_active":"2024-06-03T09:06:03.387171+0000","last_became_peered":"2024-06-03T09:06:03.387171+0000","last_unstale":"2024-06-03T09:06:18.331485+0000","last_undegraded":"2024-06-03T09:06:18.331485+0000","last_fullsized":"2024-06-03T09:06:03.283787+0000","mapping_epoch":999,"log_start":"0'0","ondisk_log_start":"0'0","created":23,"last_epoch_clean":255,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-06-03T08:36:28.444764+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-06-03T08:36:28.444764+0000","last_clean_scrub_stamp":"2024-06-03T08:36:28.444764+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-06-04T20:17:43.485023+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5],"acting":[5],"avail_no_missing":["5"],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"17.17","version":"0'0","reported_seq":954,"reported_epoch":1009,"state":"active+clean","last_fresh":"2024-06-03T09:06:14.447046+0000","last_change":"2024-06-03T08:38:38.483275+0000","last_active":"2024-06-03T09:06:14.447046+0000","last_peered":"2024-06-03T09:06:14.447046+0000","last_clean":"2024-06-03T09:06:14.447046+0000","last_became_active":"2024-06-03T08:38:38.467916+0000","last_became_peered":"2024-06-03T08:38:38.467916+0000","last_unstale":"2024-06-03T09:06:14.447046+0000","last_undegraded":"2024-06-03T09:06:14.447046+0000","last_fullsized":"2024-06-03T09:06:14.447046+0000","mapping_epoch":28,"log_start":"0'0","ondisk_log_start":"0'0","created":28,"last_epoch_clean":61,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-06-03T08:37:55.614497+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-06-03T08:37:55.614497+0000","last_clean_scrub_stamp":"2024-06-03T08:37:55.614497+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-06-04T10:02:23.036420+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,7],"acting":[2,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"2.2","version":"0'0","reported_seq":963,"reported_epoch":1009,"state":"active+undersized","last_fresh":"2024-06-03T09:06:13.657892+0000","last_change":"2024-06-03T09:06:03.410020+0000","last_active":"2024-06-03T09:06:13.657892+0000","last_peered":"2024-06-03T09:06:13.657892+0000","last_clean":"2024-06-03T09:05:37.145889+0000","last_became_active":"2024-06-03T09:06:03.410020+0000","last_became_peered":"2024-06-03T09:06:03.410020+0000","last_unstale":"2024-06-03T09:06:13.657892+0000","last_undegraded":"2024-06-03T09:06:13.657892+0000","last_fullsized":"2024-06-03T09:06:03.346864+0000","mapping_epoch":999,"log_start":"0'0","ondisk_log_start":"0'0","created":23,"last_epoch_clean":255,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-06-03T08:36:28.444764+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-06-03T08:36:28.444764+0000","last_clean_scrub_stamp":"2024-06-03T08:36:28.444764+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-06-04T09:57:12.423360+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3],"acting":[3],"avail_no_missing":["3"],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"17.11","version":"0'0","reported_seq":944,"reported_epoch":1009,"state":"active+clean","last_fresh":"2024-06-03T09:06:14.426670+0000","last_change":"2024-06-03T08:38:06.502302+0000","last_active":"2024-06-03T09:06:14.426670+0000","last_peered":"2024-06-03T09:06:14.426670+0000","last_clean":"2024-06-03T09:06:14.426670+0000","last_became_active":"2024-06-03T08:38:06.490204+0000","last_became_peered":"2024-06-03T08:38:06.490204+0000","last_unstale":"2024-06-03T09:06:14.426670+0000","last_undegraded":"2024-06-03T09:06:14.426670+0000","last_fullsized":"2024-06-03T09:06:14.426670+0000","mapping_epoch":28,"log_start":"0'0","ondisk_log_start":"0'0","created":28,"last_epoch_clean":32,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-06-03T08:37:55.614497+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-06-03T08:37:55.614497+0000","last_clean_scrub_stamp":"2024-06-03T08:37:55.614497+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-06-04T10:47:05.988958+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,2],"acting":[4,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"2.1","version":"0'0","reported_seq":952,"reported_epoch":1009,"state":"active+clean","last_fresh":"2024-06-03T09:06:13.929725+0000","last_change":"2024-06-03T08:36:30.883360+0000","last_active":"2024-06-03T09:06:13.929725+0000","last_peered":"2024-06-03T09:06:13.929725+0000","last_clean":"2024-06-03T09:06:13.929725+0000","last_became_active":"2024-06-03T08:36:30.855073+0000","last_became_peered":"2024-06-03T08:36:30.855073+0000","last_unstale":"2024-06-03T09:06:13.929725+0000","last_undegraded":"2024-06-03T09:06:13.929725+0000","last_fullsized":"2024-06-03T09:06:13.929725+0000","mapping_epoch":23,"log_start":"0'0","ondisk_log_start":"0'0","created":23,"last_epoch_clean":25,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-06-03T08:36:28.444764+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-06-03T08:36:28.444764+0000","last_clean_scrub_stamp":"2024-06-03T08:36:28.444764+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-06-04T18:12:22.363473+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,5],"acting":[7,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"17.12","version":"0'0","reported_seq":945,"reported_epoch":1009,"state":"active+clean","last_fresh":"2024-06-03T09:06:13.935141+0000","last_change":"2024-06-03T08:38:01.751562+0000","last_active":"2024-06-03T09:06:13.935141+0000","last_peered":"2024-06-03T09:06:13.935141+0000","last_clean":"2024-06-03T09:06:13.935141+0000","last_became_active":"2024-06-03T08:38:01.745027+0000","last_became_peered":"2024-06-03T08:38:01.745027+0000","last_unstale":"2024-06-03T09:06:13.935141+0000","last_undegraded":"2024-06-03T09:06:13.935141+0000","last_fullsized":"2024-06-03T09:06:13.935141+0000","mapping_epoch":28,"log_start":"0'0","ondisk_log_start":"0'0","created":28,"last_epoch_clean":32,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-06-03T08:37:55.614497+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-06-03T08:37:55.614497+0000","last_clean_scrub_stamp":"2024-06-03T08:37:55.614497+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-06-04T17:54:43.785326+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,3],"acting":[7,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"2.0","version":"0'0","reported_seq":943,"reported_epoch":1009,"state":"active+clean","last_fresh":"2024-06-03T09:06:13.648112+0000","last_change":"2024-06-03T08:36:29.984096+0000","last_active":"2024-06-03T09:06:13.648112+0000","last_peered":"2024-06-03T09:06:13.648112+0000","last_clean":"2024-06-03T09:06:13.648112+0000","last_became_active":"2024-06-03T08:36:29.957875+0000","last_became_peered":"2024-06-03T08:36:29.957875+0000","last_unstale":"2024-06-03T09:06:13.648112+0000","last_undegraded":"2024-06-03T09:06:13.648112+0000","last_fullsized":"2024-06-03T09:06:13.648112+0000","mapping_epoch":23,"log_start":"0'0","ondisk_log_start":"0'0","created":23,"last_epoch_clean":24,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-06-03T08:36:28.444764+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-06-03T08:36:28.444764+0000","last_clean_scrub_stamp":"2024-06-03T08:36:28.444764+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-06-04T08:41:33.679479+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,5],"acting":[3,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"17.13","version":"0'0","reported_seq":943,"reported_epoch":1009,"state":"active+clean","last_fresh":"2024-06-03T09:06:14.438399+0000","last_change":"2024-06-03T08:38:26.838316+0000","last_active":"2024-06-03T09:06:14.438399+0000","last_peered":"2024-06-03T09:06:14.438399+0000","last_clean":"2024-06-03T09:06:14.438399+0000","last_became_active":"2024-06-03T08:38:26.833214+0000","last_became_peered":"2024-06-03T08:38:26.833214+0000","last_unstale":"2024-06-03T09:06:14.438399+0000","last_undegraded":"2024-06-03T09:06:14.438399+0000","last_fullsized":"2024-06-03T09:06:14.438399+0000","mapping_epoch":28,"log_start":"0'0","ondisk_log_start":"0'0","created":28,"last_epoch_clean":52,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-06-03T08:37:55.614497+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-06-03T08:37:55.614497+0000","last_clean_scrub_stamp":"2024-06-03T08:37:55.614497+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-06-04T12:58:51.617998+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,0],"acting":[4,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"2.3","version":"0'0","reported_seq":943,"reported_epoch":1010,"state":"active+clean","last_fresh":"2024-06-03T09:06:18.362614+0000","last_change":"2024-06-03T08:36:30.351670+0000","last_active":"2024-06-03T09:06:18.362614+0000","last_peered":"2024-06-03T09:06:18.362614+0000","last_clean":"2024-06-03T09:06:18.362614+0000","last_became_active":"2024-06-03T08:36:30.343560+0000","last_became_peered":"2024-06-03T08:36:30.343560+0000","last_unstale":"2024-06-03T09:06:18.362614+0000","last_undegraded":"2024-06-03T09:06:18.362614+0000","last_fullsized":"2024-06-03T09:06:18.362614+0000","mapping_epoch":23,"log_start":"0'0","ondisk_log_start":"0'0","created":23,"last_epoch_clean":24,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-06-03T08:36:28.444764+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-06-03T08:36:28.444764+0000","last_clean_scrub_stamp":"2024-06-03T08:36:28.444764+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-06-04T12:02:24.765135+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,2],"acting":[3,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.0","version":"19'76","reported_seq":1051,"reported_epoch":1009,"state":"active+clean","last_fresh":"2024-06-03T09:06:13.703685+0000","last_change":"2024-06-03T08:36:26.702702+0000","last_active":"2024-06-03T09:06:13.703685+0000","last_peered":"2024-06-03T09:06:13.703685+0000","last_clean":"2024-06-03T09:06:13.703685+0000","last_became_active":"2024-06-03T08:36:26.128378+0000","last_became_peered":"2024-06-03T08:36:26.128378+0000","last_unstale":"2024-06-03T09:06:13.703685+0000","last_undegraded":"2024-06-03T09:06:13.703685+0000","last_fullsized":"2024-06-03T09:06:13.703685+0000","mapping_epoch":20,"log_start":"0'0","ondisk_log_start":"0'0","created":16,"last_epoch_clean":22,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-06-03T08:36:15.542635+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-06-03T08:36:15.542635+0000","last_clean_scrub_stamp":"2024-06-03T08:36:15.542635+0000","objects_scrubbed":0,"log_size":76,"log_dups_size":0,"ondisk_log_size":76,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-06-04T12:41:53.731360+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":459280,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":96,"num_read_kb":82,"num_write":113,"num_write_kb":1372,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2,"num_bytes_recovered":397840,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,5],"acting":[1,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"17.10","version":"0'0","reported_seq":944,"reported_epoch":1009,"state":"active+clean","last_fresh":"2024-06-03T09:06:14.422035+0000","last_change":"2024-06-03T08:38:15.399189+0000","last_active":"2024-06-03T09:06:14.422035+0000","last_peered":"2024-06-03T09:06:14.422035+0000","last_clean":"2024-06-03T09:06:14.422035+0000","last_became_active":"2024-06-03T08:38:15.337723+0000","last_became_peered":"2024-06-03T08:38:15.337723+0000","last_unstale":"2024-06-03T09:06:14.422035+0000","last_undegraded":"2024-06-03T09:06:14.422035+0000","last_fullsized":"2024-06-03T09:06:14.422035+0000","mapping_epoch":28,"log_start":"0'0","ondisk_log_start":"0'0","created":28,"last_epoch_clean":42,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-06-03T08:37:55.614497+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-06-03T08:37:55.614497+0000","last_clean_scrub_stamp":"2024-06-03T08:37:55.614497+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-06-04T09:29:46.427317+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,3],"acting":[4,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"17.1a","version":"0'0","reported_seq":960,"reported_epoch":1009,"state":"active+undersized","last_fresh":"2024-06-03T09:06:14.431224+0000","last_change":"2024-06-03T09:06:03.390734+0000","last_active":"2024-06-03T09:06:14.431224+0000","last_peered":"2024-06-03T09:06:14.431224+0000","last_clean":"2024-06-03T09:05:20.322313+0000","last_became_active":"2024-06-03T09:06:03.390734+0000","last_became_peered":"2024-06-03T09:06:03.390734+0000","last_unstale":"2024-06-03T09:06:14.431224+0000","last_undegraded":"2024-06-03T09:06:14.431224+0000","last_fullsized":"2024-06-03T09:06:03.343197+0000","mapping_epoch":999,"log_start":"0'0","ondisk_log_start":"0'0","created":28,"last_epoch_clean":254,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-06-03T08:37:55.614497+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-06-03T08:37:55.614497+0000","last_clean_scrub_stamp":"2024-06-03T08:37:55.614497+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-06-04T15:56:47.628875+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4],"acting":[4],"avail_no_missing":["4"],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"17.1b","version":"0'0","reported_seq":935,"reported_epoch":1009,"state":"active+clean","last_fresh":"2024-06-03T09:06:13.663279+0000","last_change":"2024-06-03T08:38:03.115748+0000","last_active":"2024-06-03T09:06:13.663279+0000","last_peered":"2024-06-03T09:06:13.663279+0000","last_clean":"2024-06-03T09:06:13.663279+0000","last_became_active":"2024-06-03T08:38:03.110533+0000","last_became_peered":"2024-06-03T08:38:03.110533+0000","last_unstale":"2024-06-03T09:06:13.663279+0000","last_undegraded":"2024-06-03T09:06:13.663279+0000","last_fullsized":"2024-06-03T09:06:13.663279+0000","mapping_epoch":28,"log_start":"0'0","ondisk_log_start":"0'0","created":28,"last_epoch_clean":32,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-06-03T08:37:55.614497+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-06-03T08:37:55.614497+0000","last_clean_scrub_stamp":"2024-06-03T08:37:55.614497+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-06-04T12:36:47.471609+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,7],"acting":[3,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"17.18","version":"0'0","reported_seq":944,"reported_epoch":1009,"state":"active+clean","last_fresh":"2024-06-03T09:06:13.924799+0000","last_change":"2024-06-03T08:38:04.632864+0000","last_active":"2024-06-03T09:06:13.924799+0000","last_peered":"2024-06-03T09:06:13.924799+0000","last_clean":"2024-06-03T09:06:13.924799+0000","last_became_active":"2024-06-03T08:38:04.627856+0000","last_became_peered":"2024-06-03T08:38:04.627856+0000","last_unstale":"2024-06-03T09:06:13.924799+0000","last_undegraded":"2024-06-03T09:06:13.924799+0000","last_fullsized":"2024-06-03T09:06:13.924799+0000","mapping_epoch":28,"log_start":"0'0","ondisk_log_start":"0'0","created":28,"last_epoch_clean":33,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-06-03T08:37:55.614497+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-06-03T08:37:55.614497+0000","last_clean_scrub_stamp":"2024-06-03T08:37:55.614497+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-06-04T19:55:51.181598+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,4],"acting":[7,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"17.19","version":"0'0","reported_seq":934,"reported_epoch":1009,"state":"active+clean","last_fresh":"2024-06-03T09:06:13.652943+0000","last_change":"2024-06-03T08:38:01.839491+0000","last_active":"2024-06-03T09:06:13.652943+0000","last_peered":"2024-06-03T09:06:13.652943+0000","last_clean":"2024-06-03T09:06:13.652943+0000","last_became_active":"2024-06-03T08:38:01.829539+0000","last_became_peered":"2024-06-03T08:38:01.829539+0000","last_unstale":"2024-06-03T09:06:13.652943+0000","last_undegraded":"2024-06-03T09:06:13.652943+0000","last_fullsized":"2024-06-03T09:06:13.652943+0000","mapping_epoch":28,"log_start":"0'0","ondisk_log_start":"0'0","created":28,"last_epoch_clean":30,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-06-03T08:37:55.614497+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-06-03T08:37:55.614497+0000","last_clean_scrub_stamp":"2024-06-03T08:37:55.614497+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-06-04T09:10:43.860229+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,2],"acting":[3,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"17.1e","version":"0'0","reported_seq":148,"reported_epoch":1009,"state":"active+undersized","last_fresh":"2024-06-03T09:06:14.445917+0000","last_change":"2024-06-03T09:06:03.406181+0000","last_active":"2024-06-03T09:06:14.445917+0000","last_peered":"2024-06-03T09:06:14.445917+0000","last_clean":"2024-06-03T08:37:55.614497+0000","last_became_active":"2024-06-03T09:06:03.406181+0000","last_became_peered":"2024-06-03T09:06:03.406181+0000","last_unstale":"2024-06-03T09:06:14.445917+0000","last_undegraded":"2024-06-03T09:06:14.445917+0000","last_fullsized":"2024-06-03T09:06:03.363739+0000","mapping_epoch":999,"log_start":"0'0","ondisk_log_start":"0'0","created":28,"last_epoch_clean":255,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-06-03T08:37:55.614497+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-06-03T08:37:55.614497+0000","last_clean_scrub_stamp":"2024-06-03T08:37:55.614497+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-06-04T11:45:24.486600+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4],"acting":[4],"avail_no_missing":["4"],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"17.1f","version":"0'0","reported_seq":941,"reported_epoch":1009,"state":"active+clean","last_fresh":"2024-06-03T09:06:14.450053+0000","last_change":"2024-06-03T08:38:04.383538+0000","last_active":"2024-06-03T09:06:14.450053+0000","last_peered":"2024-06-03T09:06:14.450053+0000","last_clean":"2024-06-03T09:06:14.450053+0000","last_became_active":"2024-06-03T08:38:04.367278+0000","last_became_peered":"2024-06-03T08:38:04.367278+0000","last_unstale":"2024-06-03T09:06:14.450053+0000","last_undegraded":"2024-06-03T09:06:14.450053+0000","last_fullsized":"2024-06-03T09:06:14.450053+0000","mapping_epoch":28,"log_start":"0'0","ondisk_log_start":"0'0","created":28,"last_epoch_clean":32,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-06-03T08:37:55.614497+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-06-03T08:37:55.614497+0000","last_clean_scrub_stamp":"2024-06-03T08:37:55.614497+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-06-04T10:39:26.552282+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,2],"acting":[0,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"17.1c","version":"0'0","reported_seq":944,"reported_epoch":1010,"state":"active+undersized","last_fresh":"2024-06-03T09:06:18.327492+0000","last_change":"2024-06-03T09:06:03.382210+0000","last_active":"2024-06-03T09:06:18.327492+0000","last_peered":"2024-06-03T09:06:18.327492+0000","last_clean":"2024-06-03T09:04:43.199660+0000","last_became_active":"2024-06-03T09:06:03.382210+0000","last_became_peered":"2024-06-03T09:06:03.382210+0000","last_unstale":"2024-06-03T09:06:18.327492+0000","last_undegraded":"2024-06-03T09:06:18.327492+0000","last_fullsized":"2024-06-03T09:06:03.277874+0000","mapping_epoch":999,"log_start":"0'0","ondisk_log_start":"0'0","created":28,"last_epoch_clean":255,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-06-03T08:37:55.614497+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-06-03T08:37:55.614497+0000","last_clean_scrub_stamp":"2024-06-03T08:37:55.614497+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-06-04T12:18:50.013884+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5],"acting":[5],"avail_no_missing":["5"],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"17.1d","version":"0'0","reported_seq":923,"reported_epoch":1009,"state":"active+clean","last_fresh":"2024-06-03T09:06:13.715906+0000","last_change":"2024-06-03T08:38:06.794630+0000","last_active":"2024-06-03T09:06:13.715906+0000","last_peered":"2024-06-03T09:06:13.715906+0000","last_clean":"2024-06-03T09:06:13.715906+0000","last_became_active":"2024-06-03T08:38:06.592317+0000","last_became_peered":"2024-06-03T08:38:06.592317+0000","last_unstale":"2024-06-03T09:06:13.715906+0000","last_undegraded":"2024-06-03T09:06:13.715906+0000","last_fullsized":"2024-06-03T09:06:13.715906+0000","mapping_epoch":28,"log_start":"0'0","ondisk_log_start":"0'0","created":28,"last_epoch_clean":33,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-06-03T08:37:55.614497+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-06-03T08:37:55.614497+0000","last_clean_scrub_stamp":"2024-06-03T08:37:55.614497+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-06-04T12:33:30.785369+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,7],"acting":[1,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"17.2","version":"0'0","reported_seq":925,"reported_epoch":1009,"state":"active+clean","last_fresh":"2024-06-03T09:06:13.708688+0000","last_change":"2024-06-03T08:38:14.981410+0000","last_active":"2024-06-03T09:06:13.708688+0000","last_peered":"2024-06-03T09:06:13.708688+0000","last_clean":"2024-06-03T09:06:13.708688+0000","last_became_active":"2024-06-03T08:38:14.973825+0000","last_became_peered":"2024-06-03T08:38:14.973825+0000","last_unstale":"2024-06-03T09:06:13.708688+0000","last_undegraded":"2024-06-03T09:06:13.708688+0000","last_fullsized":"2024-06-03T09:06:13.708688+0000","mapping_epoch":28,"log_start":"0'0","ondisk_log_start":"0'0","created":28,"last_epoch_clean":41,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-06-03T08:37:55.614497+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-06-03T08:37:55.614497+0000","last_clean_scrub_stamp":"2024-06-03T08:37:55.614497+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-06-04T17:01:05.518613+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,4],"acting":[1,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"17.3","version":"0'0","reported_seq":943,"reported_epoch":1009,"state":"active+clean","last_fresh":"2024-06-03T09:06:14.437636+0000","last_change":"2024-06-03T08:38:14.619745+0000","last_active":"2024-06-03T09:06:14.437636+0000","last_peered":"2024-06-03T09:06:14.437636+0000","last_clean":"2024-06-03T09:06:14.437636+0000","last_became_active":"2024-06-03T08:38:14.510665+0000","last_became_peered":"2024-06-03T08:38:14.510665+0000","last_unstale":"2024-06-03T09:06:14.437636+0000","last_undegraded":"2024-06-03T09:06:14.437636+0000","last_fullsized":"2024-06-03T09:06:14.437636+0000","mapping_epoch":28,"log_start":"0'0","ondisk_log_start":"0'0","created":28,"last_epoch_clean":42,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-06-03T08:37:55.614497+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-06-03T08:37:55.614497+0000","last_clean_scrub_stamp":"2024-06-03T08:37:55.614497+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-06-04T12:03:47.392064+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,7],"acting":[0,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"17.0","version":"0'0","reported_seq":944,"reported_epoch":1009,"state":"active+clean","last_fresh":"2024-06-03T09:06:14.427123+0000","last_change":"2024-06-03T08:38:04.237118+0000","last_active":"2024-06-03T09:06:14.427123+0000","last_peered":"2024-06-03T09:06:14.427123+0000","last_clean":"2024-06-03T09:06:14.427123+0000","last_became_active":"2024-06-03T08:38:04.231954+0000","last_became_peered":"2024-06-03T08:38:04.231954+0000","last_unstale":"2024-06-03T09:06:14.427123+0000","last_undegraded":"2024-06-03T09:06:14.427123+0000","last_fullsized":"2024-06-03T09:06:14.427123+0000","mapping_epoch":28,"log_start":"0'0","ondisk_log_start":"0'0","created":28,"last_epoch_clean":32,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-06-03T08:37:55.614497+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-06-03T08:37:55.614497+0000","last_clean_scrub_stamp":"2024-06-03T08:37:55.614497+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-06-04T09:56:10.741637+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,2],"acting":[0,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"17.1","version":"65'7","reported_seq":959,"reported_epoch":1009,"state":"active+undersized+degraded","last_fresh":"2024-06-03T09:06:13.709209+0000","last_change":"2024-06-03T09:06:03.370660+0000","last_active":"2024-06-03T09:06:13.709209+0000","last_peered":"2024-06-03T09:06:13.709209+0000","last_clean":"2024-06-03T09:05:03.733864+0000","last_became_active":"2024-06-03T09:06:03.370660+0000","last_became_peered":"2024-06-03T09:06:03.370660+0000","last_unstale":"2024-06-03T09:06:13.709209+0000","last_undegraded":"2024-06-03T09:06:03.266076+0000","last_fullsized":"2024-06-03T09:06:03.264525+0000","mapping_epoch":999,"log_start":"0'0","ondisk_log_start":"0'0","created":28,"last_epoch_clean":255,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-06-03T08:37:55.614497+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-06-03T08:37:55.614497+0000","last_clean_scrub_stamp":"2024-06-03T08:37:55.614497+0000","objects_scrubbed":0,"log_size":7,"log_dups_size":0,"ondisk_log_size":7,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-06-04T11:55:18.194283+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":5,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":1,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":6,"num_read_kb":6,"num_write":7,"num_write_kb":7,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5],"acting":[5],"avail_no_missing":["5"],"object_location_counts":[{"shards":"5","objects":1}],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]}],"pool_stats":[{"poolid":2,"num_pg":8,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":14,"acting":14,"num_store_stats":7},{"poolid":1,"num_pg":1,"stat_sum":{"num_bytes":459280,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":96,"num_read_kb":82,"num_write":113,"num_write_kb":1372,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2,"num_bytes_recovered":397840,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":802816,"data_stored":795680,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":76,"ondisk_log_size":76,"up":2,"acting":2,"num_store_stats":2},{"poolid":17,"num_pg":32,"stat_sum":{"num_bytes":5,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":1,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":6,"num_read_kb":6,"num_write":7,"num_write_kb":7,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":8192,"data_stored":10,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":7,"ondisk_log_size":7,"up":52,"acting":52,"num_store_stats":8}],"osd_stats":[{"osd":6,"up_from":253,"seq":1086626726201,"num_pgs":20,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":54836,"kb_used_data":5672,"kb_used_omap":1,"kb_used_meta":49150,"kb_avail":94317004,"statfs":{"total":96636764160,"available":96580612096,"internally_reserved":0,"allocated":5808128,"data_stored":2801702,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1593,"internal_metadata":50330055},"hb_peers":[0,1,2,3,4,5,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":93,"apply_latency_ms":93,"commit_latency_ns":93000000,"apply_latency_ns":93000000},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Mon Jun 3 09:04:02 2024","interfaces":[{"interface":"back","average":{"1min":1582.4780000000001,"5min":332.72699999999998,"15min":124.113},"min":{"1min":4.4829999999999997,"5min":4.1959999999999997,"15min":4.1959999999999997},"max":{"1min":22766.882000000001,"5min":22766.882000000001,"15min":22766.882000000001},"last":5.5880000000000001},{"interface":"front","average":{"1min":1579.5409999999999,"5min":334.94499999999999,"15min":125.84999999999999},"min":{"1min":4.8129999999999997,"5min":4.7539999999999996,"15min":4.4530000000000003},"max":{"1min":22766.289000000001,"5min":22766.289000000001,"15min":22766.289000000001},"last":4.835}]},{"osd":1,"last update":"Mon Jun 3 09:03:40 2024","interfaces":[{"interface":"back","average":{"1min":36.420999999999999,"5min":77.054000000000002,"15min":132.84800000000001},"min":{"1min":6.0380000000000003,"5min":5.8799999999999999,"15min":5.484},"max":{"1min":589.64099999999996,"5min":4138.1679999999997,"15min":9955.1299999999992},"last":8.8469999999999995},{"interface":"front","average":{"1min":40.240000000000002,"5min":79.674999999999997,"15min":133.38999999999999},"min":{"1min":8.4589999999999996,"5min":8.4090000000000007,"15min":6.1390000000000002},"max":{"1min":588.60799999999995,"5min":4137.6530000000002,"15min":9827.5509999999995},"last":11.279999999999999}]},{"osd":2,"last update":"Mon Jun 3 09:03:40 2024","interfaces":[{"interface":"back","average":{"1min":12.613,"5min":24.600999999999999,"15min":21.088000000000001},"min":{"1min":6.9740000000000002,"5min":6.9740000000000002,"15min":6.5330000000000004},"max":{"1min":26.076000000000001,"5min":376.24900000000002,"15min":376.24900000000002},"last":13.292999999999999},{"interface":"front","average":{"1min":12.493,"5min":22.207999999999998,"15min":21.219000000000001},"min":{"1min":6.5810000000000004,"5min":6.5110000000000001,"15min":6.0750000000000002},"max":{"1min":29.690999999999999,"5min":271.51100000000002,"15min":434.77300000000002},"last":8.2739999999999991}]},{"osd":3,"last update":"Mon Jun 3 09:03:40 2024","interfaces":[{"interface":"back","average":{"1min":14.093999999999999,"5min":24.143999999999998,"15min":29.376999999999999},"min":{"1min":10.196999999999999,"5min":6.2919999999999998,"15min":6.1719999999999997},"max":{"1min":24.832000000000001,"5min":197.84899999999999,"15min":1235.5799999999999},"last":14.281000000000001},{"interface":"front","average":{"1min":9.3260000000000005,"5min":18.463999999999999,"15min":27.631},"min":{"1min":6.125,"5min":6.125,"15min":6.125},"max":{"1min":28.166,"5min":200.173,"15min":1157.6780000000001},"last":12.782999999999999}]},{"osd":4,"last update":"Mon Jun 3 09:04:29 2024","interfaces":[{"interface":"back","average":{"1min":10.279,"5min":1253.809,"15min":450.79199999999997},"min":{"1min":7.2300000000000004,"5min":6.093,"15min":6.093},"max":{"1min":15.000999999999999,"5min":28093.096000000001,"15min":28093.096000000001},"last":10.965999999999999},{"interface":"front","average":{"1min":15.693,"5min":1279.759,"15min":463.87099999999998},"min":{"1min":10.551,"5min":7.7480000000000002,"15min":7.7480000000000002},"max":{"1min":27.898,"5min":28323.837,"15min":28323.837},"last":13.581}]},{"osd":5,"last update":"Mon Jun 3 09:04:10 2024","interfaces":[{"interface":"back","average":{"1min":13.728999999999999,"5min":24.670000000000002,"15min":23.815999999999999},"min":{"1min":10.401,"5min":7.516,"15min":7.516},"max":{"1min":25.138000000000002,"5min":320.85000000000002,"15min":445.041},"last":14.928000000000001},{"interface":"front","average":{"1min":13.847,"5min":29.283000000000001,"15min":23.393999999999998},"min":{"1min":7.944,"5min":6.266,"15min":6.266},"max":{"1min":18.817,"5min":530.56700000000001,"15min":530.56700000000001},"last":16.893000000000001}]},{"osd":7,"last update":"Mon Jun 3 09:03:40 2024","interfaces":[{"interface":"back","average":{"1min":20.568999999999999,"5min":30.286999999999999,"15min":27.977},"min":{"1min":13.874000000000001,"5min":10.58,"15min":9.2859999999999996},"max":{"1min":86.959000000000003,"5min":222.07599999999999,"15min":255.08799999999999},"last":17.59},{"interface":"front","average":{"1min":19.102,"5min":29.550000000000001,"15min":27.763000000000002},"min":{"1min":10.052,"5min":8.6709999999999994,"15min":8.5429999999999993},"max":{"1min":67.968999999999994,"5min":293.62099999999998,"15min":438.14600000000002},"last":16.045000000000002}]}]},{"osd":7,"up_from":20,"seq":85899346289,"num_pgs":9,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":58964,"kb_used_data":5704,"kb_used_omap":1,"kb_used_meta":53246,"kb_avail":94312876,"statfs":{"total":96636764160,"available":96576385024,"internally_reserved":0,"allocated":5840896,"data_stored":2740727,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1589,"internal_metadata":54524363},"hb_peers":[0,1,2,3,4,5],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Mon Jun 3 09:06:09 2024","interfaces":[{"interface":"back","average":{"1min":14.073,"5min":894.22400000000005,"15min":308.93400000000003},"min":{"1min":5.5709999999999997,"5min":4.218,"15min":4.0599999999999996},"max":{"1min":90.757999999999996,"5min":21729.001,"15min":21729.001},"last":12.398},{"interface":"front","average":{"1min":9.2550000000000008,"5min":906.23500000000001,"15min":314.06},"min":{"1min":5.5789999999999997,"5min":4.8810000000000002,"15min":4.8810000000000002},"max":{"1min":25.937000000000001,"5min":21899.732,"15min":21899.732},"last":8.6609999999999996}]},{"osd":1,"last update":"Mon Jun 3 09:06:09 2024","interfaces":[{"interface":"back","average":{"1min":15.182,"5min":69.183999999999997,"15min":219.173},"min":{"1min":7.3639999999999999,"5min":6.6200000000000001,"15min":6.1319999999999997},"max":{"1min":76.926000000000002,"5min":2795.1329999999998,"15min":18373.257000000001},"last":9.1489999999999991},{"interface":"front","average":{"1min":21.495999999999999,"5min":74.703999999999994,"15min":225.285},"min":{"1min":8.8399999999999999,"5min":8.8399999999999999,"15min":8.5129999999999999},"max":{"1min":94.977999999999994,"5min":2803.3670000000002,"15min":18374.601999999999},"last":11.051}]},{"osd":2,"last update":"Mon Jun 3 09:06:09 2024","interfaces":[{"interface":"back","average":{"1min":19.164000000000001,"5min":26.902000000000001,"15min":22.920999999999999},"min":{"1min":11.207000000000001,"5min":8.5030000000000001,"15min":8.5030000000000001},"max":{"1min":91.319999999999993,"5min":333.99200000000002,"15min":333.99200000000002},"last":13.628},{"interface":"front","average":{"1min":11.125999999999999,"5min":18.132000000000001,"15min":16.353000000000002},"min":{"1min":6.202,"5min":5.8369999999999997,"15min":5.5069999999999997},"max":{"1min":28.649000000000001,"5min":226.18100000000001,"15min":226.18100000000001},"last":7.4770000000000003}]},{"osd":3,"last update":"Mon Jun 3 09:06:09 2024","interfaces":[{"interface":"back","average":{"1min":22.291,"5min":26.289999999999999,"15min":31.074999999999999},"min":{"1min":11.714,"5min":6.8959999999999999,"15min":6.3760000000000003},"max":{"1min":83.102000000000004,"5min":316.64400000000001,"15min":2039.9059999999999},"last":11.871},{"interface":"front","average":{"1min":13.324999999999999,"5min":17.963000000000001,"15min":25.969999999999999},"min":{"1min":6.1200000000000001,"5min":5.2089999999999996,"15min":5.2089999999999996},"max":{"1min":49.908999999999999,"5min":233.542,"15min":2037.5550000000001},"last":6.75}]},{"osd":4,"last update":"Mon Jun 3 09:06:29 2024","interfaces":[{"interface":"back","average":{"1min":17.675999999999998,"5min":1026.325,"15min":472.47199999999998},"min":{"1min":7.306,"5min":5.9089999999999998,"15min":5.9089999999999998},"max":{"1min":140.03299999999999,"5min":28969.369999999999,"15min":30073.917000000001},"last":5.2709999999999999},{"interface":"front","average":{"1min":17.890999999999998,"5min":1044.5139999999999,"15min":480.226},"min":{"1min":5.9279999999999999,"5min":5.9279999999999999,"15min":5.9279999999999999},"max":{"1min":141.179,"5min":29244.138999999999,"15min":30348.021000000001},"last":9.7810000000000006}]},{"osd":5,"last update":"Mon Jun 3 09:06:02 2024","interfaces":[{"interface":"back","average":{"1min":24.372,"5min":26.856000000000002,"15min":23.571999999999999},"min":{"1min":12.278,"5min":9.2460000000000004,"15min":6.5650000000000004},"max":{"1min":183.64099999999999,"5min":317.572,"15min":317.572},"last":15.048999999999999},{"interface":"front","average":{"1min":21.259,"5min":23.364000000000001,"15min":21.193999999999999},"min":{"1min":7.8499999999999996,"5min":7.0999999999999996,"15min":7.008},"max":{"1min":184.53399999999999,"5min":306.60599999999999,"15min":306.60599999999999},"last":14.843999999999999}]},{"osd":6,"last update":"Mon Jun 3 09:04:31 2024","interfaces":[{"interface":"back","average":{"1min":44.061,"5min":26.821000000000002,"15min":91.176000000000002},"min":{"1min":9.0800000000000001,"5min":7.9390000000000001,"15min":7.9390000000000001},"max":{"1min":268.029,"5min":308.916,"15min":8734.4639999999999},"last":6.6769999999999996},{"interface":"front","average":{"1min":46.643000000000001,"5min":23.879999999999999,"15min":88.765000000000001},"min":{"1min":8.8119999999999994,"5min":6.9240000000000004,"15min":6.742},"max":{"1min":305.64600000000002,"5min":305.64600000000002,"15min":8735.5319999999992},"last":12.327}]}]},{"osd":5,"up_from":20,"seq":85899346264,"num_pgs":9,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":59360,"kb_used_data":6100,"kb_used_omap":1,"kb_used_meta":53246,"kb_avail":94312480,"statfs":{"total":96636764160,"available":96575979520,"internally_reserved":0,"allocated":6246400,"data_stored":3138572,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1594,"internal_metadata":54524358},"hb_peers":[0,1,2,3,4,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Mon Jun 3 09:06:06 2024","interfaces":[{"interface":"back","average":{"1min":7.468,"5min":1703.037,"15min":580.51900000000001},"min":{"1min":4.6509999999999998,"5min":4.6509999999999998,"15min":4.4020000000000001},"max":{"1min":14.711,"5min":26433.764999999999,"15min":26433.764999999999},"last":5.8639999999999999},{"interface":"front","average":{"1min":9.9429999999999996,"5min":1725.27,"15min":587.87800000000004},"min":{"1min":5.5999999999999996,"5min":5.1319999999999997,"15min":4.8170000000000002},"max":{"1min":13.590999999999999,"5min":26704.698,"15min":26704.698},"last":12.747999999999999}]},{"osd":1,"last update":"Mon Jun 3 09:06:00 2024","interfaces":[{"interface":"back","average":{"1min":10.862,"5min":67.731999999999999,"15min":217.364},"min":{"1min":7.1769999999999996,"5min":6.0890000000000004,"15min":5.4930000000000003},"max":{"1min":16.998999999999999,"5min":3923.4090000000001,"15min":18811.696},"last":14.052},{"interface":"front","average":{"1min":10.417,"5min":65.141999999999996,"15min":220.75700000000001},"min":{"1min":5.3209999999999997,"5min":5.3209999999999997,"15min":5.0209999999999999},"max":{"1min":15.821999999999999,"5min":4019.6129999999998,"15min":18968.365000000002},"last":8.7219999999999995}]},{"osd":2,"last update":"Mon Jun 3 09:06:00 2024","interfaces":[{"interface":"back","average":{"1min":11.366,"5min":24.18,"15min":21.137},"min":{"1min":5.6369999999999996,"5min":5.6369999999999996,"15min":5.0510000000000002},"max":{"1min":15.529999999999999,"5min":332.48200000000003,"15min":332.48200000000003},"last":12.167},{"interface":"front","average":{"1min":10.089,"5min":23.117000000000001,"15min":24.050000000000001},"min":{"1min":6.6769999999999996,"5min":6.6769999999999996,"15min":5.8739999999999997},"max":{"1min":15.821,"5min":353.03199999999998,"15min":353.03199999999998},"last":7.2619999999999996}]},{"osd":3,"last update":"Mon Jun 3 09:06:00 2024","interfaces":[{"interface":"back","average":{"1min":13.472,"5min":24.283999999999999,"15min":31.666},"min":{"1min":10.111000000000001,"5min":7.4720000000000004,"15min":7.4210000000000003},"max":{"1min":16.271999999999998,"5min":353.57499999999999,"15min":2855.8519999999999},"last":8.1140000000000008},{"interface":"front","average":{"1min":9.5899999999999999,"5min":15.875,"15min":25.834},"min":{"1min":6.6609999999999996,"5min":5.7729999999999997,"15min":5.1909999999999998},"max":{"1min":15.112,"5min":203.804,"15min":2869.7860000000001},"last":9.3699999999999992}]},{"osd":4,"last update":"Mon Jun 3 09:06:29 2024","interfaces":[{"interface":"back","average":{"1min":11.067,"5min":780.21000000000004,"15min":377.28399999999999},"min":{"1min":5.4779999999999998,"5min":5.4779999999999998,"15min":5.4779999999999998},"max":{"1min":14.654999999999999,"5min":23315.438999999998,"15min":28619.825000000001},"last":6.7199999999999998},{"interface":"front","average":{"1min":14.349,"5min":782.97299999999996,"15min":377.93400000000003},"min":{"1min":7.492,"5min":6.5030000000000001,"15min":6.5030000000000001},"max":{"1min":17.707000000000001,"5min":23317.502,"15min":28621.294000000002},"last":11.621}]},{"osd":6,"last update":"Mon Jun 3 09:03:57 2024","interfaces":[{"interface":"back","average":{"1min":23.536000000000001,"5min":36.457000000000001,"15min":83.799999999999997},"min":{"1min":9.4800000000000004,"5min":9.4800000000000004,"15min":8.0630000000000006},"max":{"1min":89.442999999999998,"5min":508.77999999999997,"15min":9193.5169999999998},"last":157.79300000000001},{"interface":"front","average":{"1min":18.891999999999999,"5min":22.100999999999999,"15min":75.489999999999995},"min":{"1min":5.923,"5min":5.923,"15min":5.923},"max":{"1min":78.424999999999997,"5min":464.62,"15min":9191.4429999999993},"last":148.09}]},{"osd":7,"last update":"Mon Jun 3 09:06:00 2024","interfaces":[{"interface":"back","average":{"1min":13.257,"5min":20.917999999999999,"15min":21.088999999999999},"min":{"1min":10.475,"5min":6.7629999999999999,"15min":6.7629999999999999},"max":{"1min":17.245000000000001,"5min":211.22999999999999,"15min":211.22999999999999},"last":11.061999999999999},{"interface":"front","average":{"1min":12.872999999999999,"5min":20.587,"15min":24.021000000000001},"min":{"1min":8.5760000000000005,"5min":7.7709999999999999,"15min":6.8570000000000002},"max":{"1min":18.359999999999999,"5min":223.191,"15min":286.69499999999999},"last":10.021000000000001}]}]},{"osd":4,"up_from":20,"seq":85899346271,"num_pgs":12,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":63060,"kb_used_data":5704,"kb_used_omap":1,"kb_used_meta":57342,"kb_avail":94308780,"statfs":{"total":96636764160,"available":96572190720,"internally_reserved":0,"allocated":5840896,"data_stored":2740727,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1594,"internal_metadata":58718662},"hb_peers":[0,1,2,3,5,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Mon Jun 3 09:06:35 2024","interfaces":[{"interface":"back","average":{"1min":10.417,"5min":1093.547,"15min":381.64499999999998},"min":{"1min":4.6900000000000004,"5min":4.5279999999999996,"15min":4.3220000000000001},"max":{"1min":52.241,"5min":21726.743999999999,"15min":21726.743999999999},"last":7.0030000000000001},{"interface":"front","average":{"1min":13.175000000000001,"5min":1102.4200000000001,"15min":387.161},"min":{"1min":6.1950000000000003,"5min":5.8380000000000001,"15min":5.4909999999999997},"max":{"1min":53.148000000000003,"5min":21727.993999999999,"15min":21727.993999999999},"last":12.923999999999999}]},{"osd":1,"last update":"Mon Jun 3 09:06:35 2024","interfaces":[{"interface":"back","average":{"1min":9.9909999999999997,"5min":53.250999999999998,"15min":340.02999999999997},"min":{"1min":4.5670000000000002,"5min":4.1970000000000001,"15min":4.1970000000000001},"max":{"1min":47.073,"5min":3346.4389999999999,"15min":22256.098000000002},"last":15.198},{"interface":"front","average":{"1min":12.189,"5min":53.375999999999998,"15min":339.68799999999999},"min":{"1min":6.1539999999999999,"5min":5.0590000000000002,"15min":4.2640000000000002},"max":{"1min":49.558,"5min":3347.181,"15min":22247.317999999999},"last":7.681}]},{"osd":2,"last update":"Mon Jun 3 09:06:35 2024","interfaces":[{"interface":"back","average":{"1min":15.178000000000001,"5min":17.202000000000002,"15min":20.417999999999999},"min":{"1min":6.3010000000000002,"5min":6.3010000000000002,"15min":6.3010000000000002},"max":{"1min":85.340000000000003,"5min":337.60399999999998,"15min":393.58600000000001},"last":12.378},{"interface":"front","average":{"1min":15.423,"5min":14.432,"15min":22.754000000000001},"min":{"1min":6.4889999999999999,"5min":6.0629999999999997,"15min":6.0629999999999997},"max":{"1min":82.120000000000005,"5min":127.913,"15min":394.471},"last":10.843999999999999}]},{"osd":3,"last update":"Mon Jun 3 09:06:35 2024","interfaces":[{"interface":"back","average":{"1min":17.651,"5min":15.5,"15min":18.079000000000001},"min":{"1min":10.359999999999999,"5min":5.8719999999999999,"15min":5.4000000000000004},"max":{"1min":84.555999999999997,"5min":127.43600000000001,"15min":322.63},"last":14.564},{"interface":"front","average":{"1min":16.413,"5min":17.356999999999999,"15min":22.920000000000002},"min":{"1min":9.7349999999999994,"5min":7.484,"15min":7.484},"max":{"1min":71.394000000000005,"5min":204.517,"15min":392.67500000000001},"last":13.446999999999999}]},{"osd":5,"last update":"Mon Jun 3 09:06:35 2024","interfaces":[{"interface":"back","average":{"1min":15.672000000000001,"5min":16.742999999999999,"15min":20.795000000000002},"min":{"1min":8.093,"5min":6.7489999999999997,"15min":6.7489999999999997},"max":{"1min":51.326000000000001,"5min":268.10199999999998,"15min":344.23399999999998},"last":9.2910000000000004},{"interface":"front","average":{"1min":17.297999999999998,"5min":19.356999999999999,"15min":25.713999999999999},"min":{"1min":12.875,"5min":8.5180000000000007,"15min":8.5180000000000007},"max":{"1min":40.829000000000001,"5min":264.75,"15min":345.18700000000001},"last":10.318}]},{"osd":6,"last update":"Mon Jun 3 09:04:32 2024","interfaces":[{"interface":"back","average":{"1min":15.548,"5min":32.957999999999998,"15min":152.738},"min":{"1min":9.141,"5min":9.141,"15min":8.2319999999999993},"max":{"1min":52.57,"5min":395.238,"15min":12496.335999999999},"last":208.90000000000001},{"interface":"front","average":{"1min":12.680999999999999,"5min":25.428000000000001,"15min":149.30600000000001},"min":{"1min":6.4379999999999997,"5min":5.6319999999999997,"15min":5.6319999999999997},"max":{"1min":46.283000000000001,"5min":377.74299999999999,"15min":12616.297},"last":205.88900000000001}]},{"osd":7,"last update":"Mon Jun 3 09:06:35 2024","interfaces":[{"interface":"back","average":{"1min":9.6969999999999992,"5min":14.196,"15min":18.809999999999999},"min":{"1min":6.7190000000000003,"5min":6.2110000000000003,"15min":5.8010000000000002},"max":{"1min":15.960000000000001,"5min":207.25299999999999,"15min":358.94900000000001},"last":8.7810000000000006},{"interface":"front","average":{"1min":17.696000000000002,"5min":19.024999999999999,"15min":23.259},"min":{"1min":11.794,"5min":7.7789999999999999,"15min":7.7789999999999999},"max":{"1min":45.639000000000003,"5min":209.32900000000001,"15min":380.06900000000002},"last":11.362}]}]},{"osd":2,"up_from":20,"seq":85899346291,"num_pgs":11,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":58964,"kb_used_data":5704,"kb_used_omap":1,"kb_used_meta":53246,"kb_avail":94312876,"statfs":{"total":96636764160,"available":96576385024,"internally_reserved":0,"allocated":5840896,"data_stored":2740727,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1594,"internal_metadata":54524358},"hb_peers":[0,1,3,4,5,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Mon Jun 3 09:06:23 2024","interfaces":[{"interface":"back","average":{"1min":8.9629999999999992,"5min":946.25400000000002,"15min":326.36099999999999},"min":{"1min":5.3479999999999999,"5min":4.7050000000000001,"15min":4.1420000000000003},"max":{"1min":26.015000000000001,"5min":23465.18,"15min":23465.18},"last":7.5270000000000001},{"interface":"front","average":{"1min":7.6130000000000004,"5min":944.39800000000002,"15min":326.178},"min":{"1min":4.6790000000000003,"5min":4.6790000000000003,"15min":4.6790000000000003},"max":{"1min":20.802,"5min":23465.966,"15min":23465.966},"last":9.0079999999999991}]},{"osd":1,"last update":"Mon Jun 3 09:06:23 2024","interfaces":[{"interface":"back","average":{"1min":13.007,"5min":76.507000000000005,"15min":294.72899999999998},"min":{"1min":6.4089999999999998,"5min":5.4199999999999999,"15min":5.4199999999999999},"max":{"1min":85.430000000000007,"5min":4220.5290000000005,"15min":18954.987000000001},"last":6.4100000000000001},{"interface":"front","average":{"1min":15.894,"5min":82.119,"15min":302.92399999999998},"min":{"1min":7.367,"5min":7.367,"15min":7.367},"max":{"1min":98.498000000000005,"5min":4325.9489999999996,"15min":19143.834999999999},"last":10.802}]},{"osd":3,"last update":"Mon Jun 3 09:06:23 2024","interfaces":[{"interface":"back","average":{"1min":9.7579999999999991,"5min":12.444000000000001,"15min":28.065000000000001},"min":{"1min":7,"5min":5.9290000000000003,"15min":5.2450000000000001},"max":{"1min":28.164000000000001,"5min":118.913,"15min":2594.8150000000001},"last":10.282},{"interface":"front","average":{"1min":12.321999999999999,"5min":15.25,"15min":31.349},"min":{"1min":9.1240000000000006,"5min":8.0760000000000005,"15min":6.0780000000000003},"max":{"1min":32.360999999999997,"5min":156.34700000000001,"15min":2706.5120000000002},"last":15.382999999999999}]},{"osd":4,"last update":"Mon Jun 3 09:05:57 2024","interfaces":[{"interface":"back","average":{"1min":10.573,"5min":1266.894,"15min":438.154},"min":{"1min":8.6379999999999999,"5min":7.8789999999999996,"15min":6.2039999999999997},"max":{"1min":15.105,"5min":26660.861000000001,"15min":26660.861000000001},"last":11.968999999999999},{"interface":"front","average":{"1min":12.407999999999999,"5min":1266.7460000000001,"15min":435.75400000000002},"min":{"1min":10.266999999999999,"5min":10.266999999999999,"15min":6.2720000000000002},"max":{"1min":15.228,"5min":26616.243999999999,"15min":26616.243999999999},"last":14.516999999999999}]},{"osd":5,"last update":"Mon Jun 3 09:06:23 2024","interfaces":[{"interface":"back","average":{"1min":14.103,"5min":18.091000000000001,"15min":21.119},"min":{"1min":9.6349999999999998,"5min":9.2159999999999993,"15min":6.2050000000000001},"max":{"1min":32.890000000000001,"5min":252.31100000000001,"15min":252.31100000000001},"last":13.661},{"interface":"front","average":{"1min":11.545999999999999,"5min":16.170999999999999,"15min":20.254000000000001},"min":{"1min":7.6070000000000002,"5min":7.4850000000000003,"15min":6.8810000000000002},"max":{"1min":29.228999999999999,"5min":258.31700000000001,"15min":258.31700000000001},"last":8.4199999999999999}]},{"osd":6,"last update":"Mon Jun 3 09:04:28 2024","interfaces":[{"interface":"back","average":{"1min":14.063000000000001,"5min":24.850000000000001,"15min":106.755},"min":{"1min":9.0820000000000007,"5min":8.9649999999999999,"15min":8.9649999999999999},"max":{"1min":24.666,"5min":188.06700000000001,"15min":9521.0010000000002},"last":183.60900000000001},{"interface":"front","average":{"1min":13.933999999999999,"5min":21.081,"15min":106.142},"min":{"1min":5.8719999999999999,"5min":5.8719999999999999,"15min":5.8719999999999999},"max":{"1min":117.551,"5min":193.887,"15min":9585.6620000000003},"last":184.20599999999999}]},{"osd":7,"last update":"Mon Jun 3 09:06:23 2024","interfaces":[{"interface":"back","average":{"1min":17.056000000000001,"5min":20.055,"15min":23.268999999999998},"min":{"1min":10.353999999999999,"5min":6.6130000000000004,"15min":6.6130000000000004},"max":{"1min":88.974999999999994,"5min":242.56800000000001,"15min":242.56800000000001},"last":12.833},{"interface":"front","average":{"1min":18.885000000000002,"5min":21.460000000000001,"15min":25.489999999999998},"min":{"1min":12.497,"5min":9.0660000000000007,"15min":9.0660000000000007},"max":{"1min":102.229,"5min":181.56399999999999,"15min":214.33799999999999},"last":16.236000000000001}]}]},{"osd":3,"up_from":14,"seq":60129542505,"num_pgs":11,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":58964,"kb_used_data":5704,"kb_used_omap":1,"kb_used_meta":53246,"kb_avail":94312876,"statfs":{"total":96636764160,"available":96576385024,"internally_reserved":0,"allocated":5840896,"data_stored":2740727,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1592,"internal_metadata":54524360},"hb_peers":[0,1,2,4,5,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Mon Jun 3 09:06:25 2024","interfaces":[{"interface":"back","average":{"1min":7.79,"5min":1060.1959999999999,"15min":367.01799999999997},"min":{"1min":5.6909999999999998,"5min":5.173,"15min":4.6929999999999996},"max":{"1min":10.59,"5min":23275.167000000001,"15min":23275.167000000001},"last":11.048999999999999},{"interface":"front","average":{"1min":11.215999999999999,"5min":1066.5799999999999,"15min":372.476},"min":{"1min":8.7080000000000002,"5min":7.2510000000000003,"15min":5.9880000000000004},"max":{"1min":13.499000000000001,"5min":23278.536,"15min":23278.536},"last":13.255000000000001}]},{"osd":1,"last update":"Mon Jun 3 09:06:25 2024","interfaces":[{"interface":"back","average":{"1min":7.3760000000000003,"5min":15.614000000000001,"15min":389.83100000000002},"min":{"1min":4.335,"5min":4.335,"15min":4.335},"max":{"1min":11.289,"5min":250.72300000000001,"15min":21658.205999999998},"last":6.0979999999999999},{"interface":"front","average":{"1min":8.9499999999999993,"5min":14.135,"15min":381.10300000000001},"min":{"1min":6.5609999999999999,"5min":5.4100000000000001,"15min":5.2779999999999996},"max":{"1min":14.51,"5min":180.44499999999999,"15min":21307.069},"last":8.6020000000000003}]},{"osd":2,"last update":"Mon Jun 3 09:06:25 2024","interfaces":[{"interface":"back","average":{"1min":12.343999999999999,"5min":22.059999999999999,"15min":23.715},"min":{"1min":7.9199999999999999,"5min":6.4900000000000002,"15min":6.056},"max":{"1min":15.601000000000001,"5min":377.803,"15min":377.803},"last":12.279999999999999},{"interface":"front","average":{"1min":10.319000000000001,"5min":19.364999999999998,"15min":20.242999999999999},"min":{"1min":6.891,"5min":6.1749999999999998,"15min":5.9020000000000001},"max":{"1min":15.432,"5min":371.601,"15min":371.601},"last":6.9720000000000004}]},{"osd":4,"last update":"Mon Jun 3 09:06:31 2024","interfaces":[{"interface":"back","average":{"1min":9.7460000000000004,"5min":891.23000000000002,"15min":416.48099999999999},"min":{"1min":5.6539999999999999,"5min":5.6539999999999999,"15min":5.6539999999999999},"max":{"1min":15.445,"5min":29040.635999999999,"15min":30144.939999999999},"last":7.7599999999999998},{"interface":"front","average":{"1min":12.398999999999999,"5min":897.58500000000004,"15min":421.27499999999998},"min":{"1min":8.9469999999999992,"5min":7.8109999999999999,"15min":6.2439999999999998},"max":{"1min":16.518000000000001,"5min":29076.955999999998,"15min":30180.689999999999},"last":11.598000000000001}]},{"osd":5,"last update":"Mon Jun 3 09:06:07 2024","interfaces":[{"interface":"back","average":{"1min":12.802,"5min":19.640999999999998,"15min":22.318000000000001},"min":{"1min":9.0890000000000004,"5min":6.8810000000000002,"15min":6.7690000000000001},"max":{"1min":16.745999999999999,"5min":249.88,"15min":327.34899999999999},"last":14.084},{"interface":"front","average":{"1min":15.002000000000001,"5min":24.003,"15min":26.649999999999999},"min":{"1min":11.984,"5min":9.9149999999999991,"15min":8.9879999999999995},"max":{"1min":18.407,"5min":251.47800000000001,"15min":331.86799999999999},"last":14.791}]},{"osd":6,"last update":"Mon Jun 3 09:04:04 2024","interfaces":[{"interface":"back","average":{"1min":22.815999999999999,"5min":31.513999999999999,"15min":181.74299999999999},"min":{"1min":9.4809999999999999,"5min":7.2240000000000002,"15min":7.2240000000000002},"max":{"1min":160.56999999999999,"5min":380.613,"15min":11893.74},"last":19.359999999999999},{"interface":"front","average":{"1min":25.359999999999999,"5min":30.981000000000002,"15min":179.81399999999999},"min":{"1min":9.8870000000000005,"5min":7.3360000000000003,"15min":6.0899999999999999},"max":{"1min":193.11099999999999,"5min":379.80700000000002,"15min":11893.069},"last":8.202}]},{"osd":7,"last update":"Mon Jun 3 09:06:25 2024","interfaces":[{"interface":"back","average":{"1min":11.321999999999999,"5min":18.443999999999999,"15min":21.568999999999999},"min":{"1min":8.109,"5min":8.109,"15min":7.4279999999999999},"max":{"1min":17.113,"5min":146.267,"15min":178.76300000000001},"last":9.4009999999999998},{"interface":"front","average":{"1min":12.512,"5min":20.065999999999999,"15min":24.190000000000001},"min":{"1min":7.4989999999999997,"5min":7.4989999999999997,"15min":7.4989999999999997},"max":{"1min":18.047999999999998,"5min":148.13800000000001,"15min":417.22800000000001},"last":10.007}]}]},{"osd":1,"up_from":14,"seq":60129542503,"num_pgs":6,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":59356,"kb_used_data":6096,"kb_used_omap":1,"kb_used_meta":53246,"kb_avail":94312484,"statfs":{"total":96636764160,"available":96575983616,"internally_reserved":0,"allocated":6242304,"data_stored":3138567,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":54524357},"hb_peers":[0,2,3,4,5,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Mon Jun 3 09:06:07 2024","interfaces":[{"interface":"back","average":{"1min":7.7530000000000001,"5min":185.66999999999999,"15min":77.347999999999999},"min":{"1min":5.0229999999999997,"5min":4.9279999999999999,"15min":4.5449999999999999},"max":{"1min":11.896000000000001,"5min":7470.616,"15min":7470.616},"last":4.9960000000000004},{"interface":"front","average":{"1min":9.9220000000000006,"5min":187.14599999999999,"15min":76.325999999999993},"min":{"1min":5.3159999999999998,"5min":5.3159999999999998,"15min":5.3159999999999998},"max":{"1min":18.338000000000001,"5min":7477.0829999999996,"15min":7477.0829999999996},"last":12.051}]},{"osd":2,"last update":"Mon Jun 3 09:06:20 2024","interfaces":[{"interface":"back","average":{"1min":10.340999999999999,"5min":17.369,"15min":23.670999999999999},"min":{"1min":6.9740000000000002,"5min":5.8650000000000002,"15min":5.8650000000000002},"max":{"1min":15.452,"5min":228,"15min":657.38},"last":8.8550000000000004},{"interface":"front","average":{"1min":13.387,"5min":23.355,"15min":31.524000000000001},"min":{"1min":9.077,"5min":9.077,"15min":7.2160000000000002},"max":{"1min":19.701000000000001,"5min":229.465,"15min":700.06500000000005},"last":12.859999999999999}]},{"osd":3,"last update":"Mon Jun 3 09:06:04 2024","interfaces":[{"interface":"back","average":{"1min":8.6110000000000007,"5min":15.733000000000001,"15min":26.646999999999998},"min":{"1min":5.827,"5min":5.4640000000000004,"15min":5.4640000000000004},"max":{"1min":13.105,"5min":121.619,"15min":1676.009},"last":8.6129999999999995},{"interface":"front","average":{"1min":9.5850000000000009,"5min":16.161999999999999,"15min":29.065999999999999},"min":{"1min":6.3630000000000004,"5min":5.7699999999999996,"15min":5.7699999999999996},"max":{"1min":15.166,"5min":132.036,"15min":1766.5840000000001},"last":7.2569999999999997}]},{"osd":4,"last update":"Mon Jun 3 09:06:35 2024","interfaces":[{"interface":"back","average":{"1min":8.6400000000000006,"5min":641.85500000000002,"15min":334.71499999999997},"min":{"1min":6.7640000000000002,"5min":6.1760000000000002,"15min":6.0970000000000004},"max":{"1min":11.811,"5min":26340.968000000001,"15min":28044.862000000001},"last":6.7640000000000002},{"interface":"front","average":{"1min":12.361000000000001,"5min":648.59900000000005,"15min":338.48500000000001},"min":{"1min":9.8949999999999996,"5min":9.3940000000000001,"15min":6.4640000000000004},"max":{"1min":15.507,"5min":26342.311000000002,"15min":28045.624},"last":10.343}]},{"osd":5,"last update":"Mon Jun 3 09:06:04 2024","interfaces":[{"interface":"back","average":{"1min":12.478999999999999,"5min":20.228000000000002,"15min":25.997},"min":{"1min":8.6430000000000007,"5min":7.9720000000000004,"15min":7.234},"max":{"1min":17.442,"5min":139.43199999999999,"15min":701.22000000000003},"last":11.173},{"interface":"front","average":{"1min":13.507999999999999,"5min":25.760999999999999,"15min":27.553999999999998},"min":{"1min":10.042999999999999,"5min":7.258,"15min":6.242},"max":{"1min":20.611000000000001,"5min":278.63,"15min":598.35000000000002},"last":10.097}]},{"osd":6,"last update":"Mon Jun 3 09:03:54 2024","interfaces":[{"interface":"back","average":{"1min":28.524000000000001,"5min":20.654,"15min":23.878},"min":{"1min":6.0309999999999997,"5min":6.0309999999999997,"15min":6.0309999999999997},"max":{"1min":167.221,"5min":167.221,"15min":704.245},"last":18.721},{"interface":"front","average":{"1min":27.606000000000002,"5min":30.131,"15min":28.628},"min":{"1min":9.8650000000000002,"5min":9.7200000000000006,"15min":8.3580000000000005},"max":{"1min":123.938,"5min":358.23000000000002,"15min":634.05499999999995},"last":173.113}]},{"osd":7,"last update":"Mon Jun 3 09:06:20 2024","interfaces":[{"interface":"back","average":{"1min":14.66,"5min":23.234999999999999,"15min":31.097000000000001},"min":{"1min":11.250999999999999,"5min":8.0559999999999992,"15min":8.0559999999999992},"max":{"1min":21.507999999999999,"5min":270.95100000000002,"15min":703.07799999999997},"last":13.523999999999999},{"interface":"front","average":{"1min":15.548,"5min":28.760000000000002,"15min":28.925000000000001},"min":{"1min":12.759,"5min":8.2089999999999996,"15min":7.7599999999999998},"max":{"1min":22.408000000000001,"5min":360.99900000000002,"15min":608.44100000000003},"last":14.153}]}]},{"osd":0,"up_from":14,"seq":60129542505,"num_pgs":10,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":81108,"kb_used_data":5704,"kb_used_omap":1,"kb_used_meta":75390,"kb_avail":94290732,"statfs":{"total":96636764160,"available":96553709568,"internally_reserved":0,"allocated":5840896,"data_stored":2740727,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1593,"internal_metadata":77199815},"hb_peers":[1,2,3,4,5,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[{"osd":1,"last update":"Mon Jun 3 09:06:04 2024","interfaces":[{"interface":"back","average":{"1min":6.9960000000000004,"5min":56.997,"15min":339.29700000000003},"min":{"1min":4.4530000000000003,"5min":4.4530000000000003,"15min":4.4100000000000001},"max":{"1min":11.720000000000001,"5min":2414.4749999999999,"15min":20691.453000000001},"last":5.891},{"interface":"front","average":{"1min":9.548,"5min":57.258000000000003,"15min":337.791},"min":{"1min":5.2300000000000004,"5min":5.2300000000000004,"15min":4.6050000000000004},"max":{"1min":15.337999999999999,"5min":2412.9549999999999,"15min":20640.792000000001},"last":25.372}]},{"osd":2,"last update":"Mon Jun 3 09:06:22 2024","interfaces":[{"interface":"back","average":{"1min":11.154,"5min":26.515999999999998,"15min":22.882000000000001},"min":{"1min":5.7869999999999999,"5min":5.766,"15min":5.766},"max":{"1min":72.269999999999996,"5min":627.40700000000004,"15min":627.40700000000004},"last":22.283999999999999},{"interface":"front","average":{"1min":14.353999999999999,"5min":33.304000000000002,"15min":29.885000000000002},"min":{"1min":8.2149999999999999,"5min":7.8630000000000004,"15min":7.4089999999999998},"max":{"1min":73.418000000000006,"5min":635.62599999999998,"15min":635.62599999999998},"last":29.343}]},{"osd":3,"last update":"Mon Jun 3 09:06:04 2024","interfaces":[{"interface":"back","average":{"1min":12.105,"5min":26.925000000000001,"15min":26.584},"min":{"1min":7.8410000000000002,"5min":7.1280000000000001,"15min":6.1909999999999998},"max":{"1min":18.425999999999998,"5min":389.72699999999998,"15min":389.72699999999998},"last":27.023},{"interface":"front","average":{"1min":13.794,"5min":29.771999999999998,"15min":28.815000000000001},"min":{"1min":10.084,"5min":9.4920000000000009,"15min":6.1849999999999996},"max":{"1min":17.373999999999999,"5min":390.23200000000003,"15min":494.92899999999997},"last":31.582999999999998}]},{"osd":4,"last update":"Mon Jun 3 09:06:27 2024","interfaces":[{"interface":"back","average":{"1min":11.352,"5min":745.30600000000004,"15min":363.48099999999999},"min":{"1min":6.415,"5min":6.415,"15min":6.415},"max":{"1min":16.724,"5min":20248.616000000002,"15min":25555.212},"last":26.516999999999999},{"interface":"front","average":{"1min":13.545,"5min":744.76999999999998,"15min":361.03100000000001},"min":{"1min":9.3889999999999993,"5min":8.8940000000000001,"15min":7.1429999999999998},"max":{"1min":18.411000000000001,"5min":20250.195,"15min":25556.159},"last":27.510000000000002}]},{"osd":5,"last update":"Mon Jun 3 09:06:17 2024","interfaces":[{"interface":"back","average":{"1min":9.7040000000000006,"5min":26.533999999999999,"15min":28.683},"min":{"1min":6.8129999999999997,"5min":6.8129999999999997,"15min":6.8129999999999997},"max":{"1min":14.942,"5min":455.78699999999998,"15min":605.05899999999997},"last":22.044},{"interface":"front","average":{"1min":14.279,"5min":28.398,"15min":26.192},"min":{"1min":11.696,"5min":6.4420000000000002,"15min":6.2030000000000003},"max":{"1min":17.879000000000001,"5min":391.31200000000001,"15min":541.11000000000001},"last":29.972000000000001}]},{"osd":6,"last update":"Mon Jun 3 09:04:30 2024","interfaces":[{"interface":"back","average":{"1min":52.496000000000002,"5min":33.139000000000003,"15min":160.83199999999999},"min":{"1min":7.5289999999999999,"5min":7.1040000000000001,"15min":6.3739999999999997},"max":{"1min":457.91699999999997,"5min":457.91699999999997,"15min":11115.258},"last":18.959},{"interface":"front","average":{"1min":49.670000000000002,"5min":33.704999999999998,"15min":162.791},"min":{"1min":12.247999999999999,"5min":11.019,"15min":9.9760000000000009},"max":{"1min":407.05099999999999,"5min":407.05099999999999,"15min":11092.194},"last":17.707000000000001}]},{"osd":7,"last update":"Mon Jun 3 09:06:22 2024","interfaces":[{"interface":"back","average":{"1min":11.625999999999999,"5min":26.375,"15min":28.972999999999999},"min":{"1min":6.4480000000000004,"5min":6.4480000000000004,"15min":6.4480000000000004},"max":{"1min":15.352,"5min":461.41699999999997,"15min":598.33699999999999},"last":26.029},{"interface":"front","average":{"1min":14.013,"5min":29.696000000000002,"15min":33.630000000000003},"min":{"1min":10.938000000000001,"5min":10.58,"15min":10.263999999999999},"max":{"1min":16.603000000000002,"5min":463.10599999999999,"15min":592.80499999999995},"last":30.530000000000001}]}]}],"pool_statfs":[{"poolid":1,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":401408,"data_stored":397840,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":5,"total":0,"available":0,"internally_reserved":0,"allocated":401408,"data_stored":397840,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":4,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":5,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":6,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":17,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":17,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":17,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":17,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":17,"osd":4,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":17,"osd":5,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":5,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":17,"osd":6,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":5,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":17,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0}]}} 2024-06-03T09:06:43.473 INFO:tasks.ceph.ceph_manager.ceph:PG 17.c is not active+clean 2024-06-03T09:06:43.473 INFO:tasks.ceph.ceph_manager.ceph:{'pgid': '17.c', 'version': "0'0", 'reported_seq': 953, 'reported_epoch': 1009, 'state': 'active+undersized', 'last_fresh': '2024-06-03T09:06:13.668156+0000', 'last_change': '2024-06-03T09:06:03.440074+0000', 'last_active': '2024-06-03T09:06:13.668156+0000', 'last_peered': '2024-06-03T09:06:13.668156+0000', 'last_clean': '2024-06-03T09:05:02.105638+0000', 'last_became_active': '2024-06-03T09:06:03.440074+0000', 'last_became_peered': '2024-06-03T09:06:03.440074+0000', 'last_unstale': '2024-06-03T09:06:13.668156+0000', 'last_undegraded': '2024-06-03T09:06:13.668156+0000', 'last_fullsized': '2024-06-03T09:06:03.377982+0000', 'mapping_epoch': 999, 'log_start': "0'0", 'ondisk_log_start': "0'0", 'created': 28, 'last_epoch_clean': 255, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2024-06-03T08:37:55.614497+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2024-06-03T08:37:55.614497+0000', 'last_clean_scrub_stamp': '2024-06-03T08:37:55.614497+0000', 'objects_scrubbed': 0, 'log_size': 0, 'log_dups_size': 0, 'ondisk_log_size': 0, 'stats_invalid': False, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': 'periodic scrub scheduled @ 2024-06-04T10:34:20.808557+0000', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 0, 'num_objects': 0, 'num_object_clones': 0, 'num_object_copies': 0, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 0, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 0, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 0, 'num_write_kb': 0, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 0, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [3], 'acting': [3], 'avail_no_missing': ['3'], 'object_location_counts': [], 'blocked_by': [], 'up_primary': 3, 'acting_primary': 3, 'purged_snaps': []} 2024-06-03T09:06:43.473 INFO:tasks.ceph.ceph_manager.ceph:PG 17.f is not active+clean 2024-06-03T09:06:43.473 INFO:tasks.ceph.ceph_manager.ceph:{'pgid': '17.f', 'version': "0'0", 'reported_seq': 148, 'reported_epoch': 1009, 'state': 'active+undersized', 'last_fresh': '2024-06-03T09:06:14.450543+0000', 'last_change': '2024-06-03T09:06:03.420285+0000', 'last_active': '2024-06-03T09:06:14.450543+0000', 'last_peered': '2024-06-03T09:06:14.450543+0000', 'last_clean': '2024-06-03T08:37:55.614497+0000', 'last_became_active': '2024-06-03T09:06:03.420285+0000', 'last_became_peered': '2024-06-03T09:06:03.420285+0000', 'last_unstale': '2024-06-03T09:06:14.450543+0000', 'last_undegraded': '2024-06-03T09:06:14.450543+0000', 'last_fullsized': '2024-06-03T09:06:03.380616+0000', 'mapping_epoch': 999, 'log_start': "0'0", 'ondisk_log_start': "0'0", 'created': 28, 'last_epoch_clean': 255, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2024-06-03T08:37:55.614497+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2024-06-03T08:37:55.614497+0000', 'last_clean_scrub_stamp': '2024-06-03T08:37:55.614497+0000', 'objects_scrubbed': 0, 'log_size': 0, 'log_dups_size': 0, 'ondisk_log_size': 0, 'stats_invalid': False, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': 'periodic scrub scheduled @ 2024-06-04T15:39:41.187731+0000', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 0, 'num_objects': 0, 'num_object_clones': 0, 'num_object_copies': 0, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 0, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 0, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 0, 'num_write_kb': 0, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 0, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [4], 'acting': [4], 'avail_no_missing': ['4'], 'object_location_counts': [], 'blocked_by': [], 'up_primary': 4, 'acting_primary': 4, 'purged_snaps': []} 2024-06-03T09:06:43.474 INFO:tasks.ceph.ceph_manager.ceph:PG 17.e is not active+clean 2024-06-03T09:06:43.474 INFO:tasks.ceph.ceph_manager.ceph:{'pgid': '17.e', 'version': "0'0", 'reported_seq': 149, 'reported_epoch': 1009, 'state': 'active+undersized', 'last_fresh': '2024-06-03T09:06:13.726992+0000', 'last_change': '2024-06-03T09:06:03.412887+0000', 'last_active': '2024-06-03T09:06:13.726992+0000', 'last_peered': '2024-06-03T09:06:13.726992+0000', 'last_clean': '2024-06-03T08:37:55.614497+0000', 'last_became_active': '2024-06-03T09:06:03.412887+0000', 'last_became_peered': '2024-06-03T09:06:03.412887+0000', 'last_unstale': '2024-06-03T09:06:13.726992+0000', 'last_undegraded': '2024-06-03T09:06:13.726992+0000', 'last_fullsized': '2024-06-03T09:06:03.332482+0000', 'mapping_epoch': 999, 'log_start': "0'0", 'ondisk_log_start': "0'0", 'created': 28, 'last_epoch_clean': 255, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2024-06-03T08:37:55.614497+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2024-06-03T08:37:55.614497+0000', 'last_clean_scrub_stamp': '2024-06-03T08:37:55.614497+0000', 'objects_scrubbed': 0, 'log_size': 0, 'log_dups_size': 0, 'ondisk_log_size': 0, 'stats_invalid': False, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': 'periodic scrub scheduled @ 2024-06-04T12:23:22.511601+0000', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 0, 'num_objects': 0, 'num_object_clones': 0, 'num_object_copies': 0, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 0, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 0, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 0, 'num_write_kb': 0, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 0, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [5], 'acting': [5], 'avail_no_missing': ['5'], 'object_location_counts': [], 'blocked_by': [], 'up_primary': 5, 'acting_primary': 5, 'purged_snaps': []} 2024-06-03T09:06:43.474 INFO:tasks.ceph.ceph_manager.ceph:PG 17.8 is not active+clean 2024-06-03T09:06:43.474 INFO:tasks.ceph.ceph_manager.ceph:{'pgid': '17.8', 'version': "0'0", 'reported_seq': 961, 'reported_epoch': 1009, 'state': 'active+undersized', 'last_fresh': '2024-06-03T09:06:14.430375+0000', 'last_change': '2024-06-03T09:06:03.388786+0000', 'last_active': '2024-06-03T09:06:14.430375+0000', 'last_peered': '2024-06-03T09:06:14.430375+0000', 'last_clean': '2024-06-03T09:04:43.297780+0000', 'last_became_active': '2024-06-03T09:06:03.388786+0000', 'last_became_peered': '2024-06-03T09:06:03.388786+0000', 'last_unstale': '2024-06-03T09:06:14.430375+0000', 'last_undegraded': '2024-06-03T09:06:14.430375+0000', 'last_fullsized': '2024-06-03T09:06:03.331754+0000', 'mapping_epoch': 999, 'log_start': "0'0", 'ondisk_log_start': "0'0", 'created': 28, 'last_epoch_clean': 255, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2024-06-03T08:37:55.614497+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2024-06-03T08:37:55.614497+0000', 'last_clean_scrub_stamp': '2024-06-03T08:37:55.614497+0000', 'objects_scrubbed': 0, 'log_size': 0, 'log_dups_size': 0, 'ondisk_log_size': 0, 'stats_invalid': False, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': 'periodic scrub scheduled @ 2024-06-04T09:18:44.765547+0000', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 0, 'num_objects': 0, 'num_object_clones': 0, 'num_object_copies': 0, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 0, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 0, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 0, 'num_write_kb': 0, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 0, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [0], 'acting': [0], 'avail_no_missing': ['0'], 'object_location_counts': [], 'blocked_by': [], 'up_primary': 0, 'acting_primary': 0, 'purged_snaps': []} 2024-06-03T09:06:43.474 INFO:tasks.ceph.ceph_manager.ceph:PG 17.b is not active+clean 2024-06-03T09:06:43.474 INFO:tasks.ceph.ceph_manager.ceph:{'pgid': '17.b', 'version': "0'0", 'reported_seq': 149, 'reported_epoch': 1009, 'state': 'active+undersized', 'last_fresh': '2024-06-03T09:06:14.433735+0000', 'last_change': '2024-06-03T09:06:03.451203+0000', 'last_active': '2024-06-03T09:06:14.433735+0000', 'last_peered': '2024-06-03T09:06:14.433735+0000', 'last_clean': '2024-06-03T08:37:55.614497+0000', 'last_became_active': '2024-06-03T09:06:03.451203+0000', 'last_became_peered': '2024-06-03T09:06:03.451203+0000', 'last_unstale': '2024-06-03T09:06:14.433735+0000', 'last_undegraded': '2024-06-03T09:06:14.433735+0000', 'last_fullsized': '2024-06-03T09:06:03.375523+0000', 'mapping_epoch': 999, 'log_start': "0'0", 'ondisk_log_start': "0'0", 'created': 28, 'last_epoch_clean': 255, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2024-06-03T08:37:55.614497+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2024-06-03T08:37:55.614497+0000', 'last_clean_scrub_stamp': '2024-06-03T08:37:55.614497+0000', 'objects_scrubbed': 0, 'log_size': 0, 'log_dups_size': 0, 'ondisk_log_size': 0, 'stats_invalid': False, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': 'periodic scrub scheduled @ 2024-06-04T10:16:19.813495+0000', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 0, 'num_objects': 0, 'num_object_clones': 0, 'num_object_copies': 0, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 0, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 0, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 0, 'num_write_kb': 0, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 0, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [2], 'acting': [2], 'avail_no_missing': ['2'], 'object_location_counts': [], 'blocked_by': [], 'up_primary': 2, 'acting_primary': 2, 'purged_snaps': []} 2024-06-03T09:06:43.474 INFO:tasks.ceph.ceph_manager.ceph:PG 17.4 is not active+clean 2024-06-03T09:06:43.474 INFO:tasks.ceph.ceph_manager.ceph:{'pgid': '17.4', 'version': "0'0", 'reported_seq': 149, 'reported_epoch': 1009, 'state': 'active+undersized', 'last_fresh': '2024-06-03T09:06:13.941522+0000', 'last_change': '2024-06-03T09:06:03.441889+0000', 'last_active': '2024-06-03T09:06:13.941522+0000', 'last_peered': '2024-06-03T09:06:13.941522+0000', 'last_clean': '2024-06-03T08:37:55.614497+0000', 'last_became_active': '2024-06-03T09:06:03.441889+0000', 'last_became_peered': '2024-06-03T09:06:03.441889+0000', 'last_unstale': '2024-06-03T09:06:13.941522+0000', 'last_undegraded': '2024-06-03T09:06:13.941522+0000', 'last_fullsized': '2024-06-03T09:06:03.359582+0000', 'mapping_epoch': 999, 'log_start': "0'0", 'ondisk_log_start': "0'0", 'created': 28, 'last_epoch_clean': 255, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2024-06-03T08:37:55.614497+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2024-06-03T08:37:55.614497+0000', 'last_clean_scrub_stamp': '2024-06-03T08:37:55.614497+0000', 'objects_scrubbed': 0, 'log_size': 0, 'log_dups_size': 0, 'ondisk_log_size': 0, 'stats_invalid': False, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': 'periodic scrub scheduled @ 2024-06-04T12:49:22.907141+0000', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 0, 'num_objects': 0, 'num_object_clones': 0, 'num_object_copies': 0, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 0, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 0, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 0, 'num_write_kb': 0, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 0, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [7], 'acting': [7], 'avail_no_missing': ['7'], 'object_location_counts': [], 'blocked_by': [], 'up_primary': 7, 'acting_primary': 7, 'purged_snaps': []} 2024-06-03T09:06:43.474 INFO:tasks.ceph.ceph_manager.ceph:PG 17.6 is not active+clean 2024-06-03T09:06:43.475 INFO:tasks.ceph.ceph_manager.ceph:{'pgid': '17.6', 'version': "0'0", 'reported_seq': 960, 'reported_epoch': 1009, 'state': 'active+undersized', 'last_fresh': '2024-06-03T09:06:14.453110+0000', 'last_change': '2024-06-03T09:06:03.428677+0000', 'last_active': '2024-06-03T09:06:14.453110+0000', 'last_peered': '2024-06-03T09:06:14.453110+0000', 'last_clean': '2024-06-03T09:04:43.312352+0000', 'last_became_active': '2024-06-03T09:06:03.428677+0000', 'last_became_peered': '2024-06-03T09:06:03.428677+0000', 'last_unstale': '2024-06-03T09:06:14.453110+0000', 'last_undegraded': '2024-06-03T09:06:14.453110+0000', 'last_fullsized': '2024-06-03T09:06:03.359909+0000', 'mapping_epoch': 999, 'log_start': "0'0", 'ondisk_log_start': "0'0", 'created': 28, 'last_epoch_clean': 255, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2024-06-03T08:37:55.614497+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2024-06-03T08:37:55.614497+0000', 'last_clean_scrub_stamp': '2024-06-03T08:37:55.614497+0000', 'objects_scrubbed': 0, 'log_size': 0, 'log_dups_size': 0, 'ondisk_log_size': 0, 'stats_invalid': False, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': 'periodic scrub scheduled @ 2024-06-04T15:38:02.180241+0000', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 0, 'num_objects': 0, 'num_object_clones': 0, 'num_object_copies': 0, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 0, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 0, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 0, 'num_write_kb': 0, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 0, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [0], 'acting': [0], 'avail_no_missing': ['0'], 'object_location_counts': [], 'blocked_by': [], 'up_primary': 0, 'acting_primary': 0, 'purged_snaps': []} 2024-06-03T09:06:43.475 INFO:tasks.ceph.ceph_manager.ceph:PG 17.16 is not active+clean 2024-06-03T09:06:43.475 INFO:tasks.ceph.ceph_manager.ceph:{'pgid': '17.16', 'version': "0'0", 'reported_seq': 972, 'reported_epoch': 1009, 'state': 'active+undersized', 'last_fresh': '2024-06-03T09:06:14.441665+0000', 'last_change': '2024-06-03T09:06:03.478615+0000', 'last_active': '2024-06-03T09:06:14.441665+0000', 'last_peered': '2024-06-03T09:06:14.441665+0000', 'last_clean': '2024-06-03T09:04:43.328415+0000', 'last_became_active': '2024-06-03T09:06:03.478615+0000', 'last_became_peered': '2024-06-03T09:06:03.478615+0000', 'last_unstale': '2024-06-03T09:06:14.441665+0000', 'last_undegraded': '2024-06-03T09:06:14.441665+0000', 'last_fullsized': '2024-06-03T09:06:03.402504+0000', 'mapping_epoch': 999, 'log_start': "0'0", 'ondisk_log_start': "0'0", 'created': 28, 'last_epoch_clean': 255, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2024-06-03T08:37:55.614497+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2024-06-03T08:37:55.614497+0000', 'last_clean_scrub_stamp': '2024-06-03T08:37:55.614497+0000', 'objects_scrubbed': 0, 'log_size': 0, 'log_dups_size': 0, 'ondisk_log_size': 0, 'stats_invalid': False, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': 'periodic scrub scheduled @ 2024-06-04T08:47:08.079233+0000', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 0, 'num_objects': 0, 'num_object_clones': 0, 'num_object_copies': 0, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 0, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 0, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 0, 'num_write_kb': 0, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 0, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [2], 'acting': [2], 'avail_no_missing': ['2'], 'object_location_counts': [], 'blocked_by': [], 'up_primary': 2, 'acting_primary': 2, 'purged_snaps': []} 2024-06-03T09:06:43.475 INFO:tasks.ceph.ceph_manager.ceph:PG 2.4 is not active+clean 2024-06-03T09:06:43.475 INFO:tasks.ceph.ceph_manager.ceph:{'pgid': '2.4', 'version': "0'0", 'reported_seq': 952, 'reported_epoch': 1010, 'state': 'active+undersized', 'last_fresh': '2024-06-03T09:06:18.331485+0000', 'last_change': '2024-06-03T09:06:03.387171+0000', 'last_active': '2024-06-03T09:06:18.331485+0000', 'last_peered': '2024-06-03T09:06:18.331485+0000', 'last_clean': '2024-06-03T09:04:43.202645+0000', 'last_became_active': '2024-06-03T09:06:03.387171+0000', 'last_became_peered': '2024-06-03T09:06:03.387171+0000', 'last_unstale': '2024-06-03T09:06:18.331485+0000', 'last_undegraded': '2024-06-03T09:06:18.331485+0000', 'last_fullsized': '2024-06-03T09:06:03.283787+0000', 'mapping_epoch': 999, 'log_start': "0'0", 'ondisk_log_start': "0'0", 'created': 23, 'last_epoch_clean': 255, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2024-06-03T08:36:28.444764+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2024-06-03T08:36:28.444764+0000', 'last_clean_scrub_stamp': '2024-06-03T08:36:28.444764+0000', 'objects_scrubbed': 0, 'log_size': 0, 'log_dups_size': 0, 'ondisk_log_size': 0, 'stats_invalid': False, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': 'periodic scrub scheduled @ 2024-06-04T20:17:43.485023+0000', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 0, 'num_objects': 0, 'num_object_clones': 0, 'num_object_copies': 0, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 0, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 0, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 0, 'num_write_kb': 0, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 0, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [5], 'acting': [5], 'avail_no_missing': ['5'], 'object_location_counts': [], 'blocked_by': [], 'up_primary': 5, 'acting_primary': 5, 'purged_snaps': []} 2024-06-03T09:06:43.475 INFO:tasks.ceph.ceph_manager.ceph:PG 2.2 is not active+clean 2024-06-03T09:06:43.475 INFO:tasks.ceph.ceph_manager.ceph:{'pgid': '2.2', 'version': "0'0", 'reported_seq': 963, 'reported_epoch': 1009, 'state': 'active+undersized', 'last_fresh': '2024-06-03T09:06:13.657892+0000', 'last_change': '2024-06-03T09:06:03.410020+0000', 'last_active': '2024-06-03T09:06:13.657892+0000', 'last_peered': '2024-06-03T09:06:13.657892+0000', 'last_clean': '2024-06-03T09:05:37.145889+0000', 'last_became_active': '2024-06-03T09:06:03.410020+0000', 'last_became_peered': '2024-06-03T09:06:03.410020+0000', 'last_unstale': '2024-06-03T09:06:13.657892+0000', 'last_undegraded': '2024-06-03T09:06:13.657892+0000', 'last_fullsized': '2024-06-03T09:06:03.346864+0000', 'mapping_epoch': 999, 'log_start': "0'0", 'ondisk_log_start': "0'0", 'created': 23, 'last_epoch_clean': 255, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2024-06-03T08:36:28.444764+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2024-06-03T08:36:28.444764+0000', 'last_clean_scrub_stamp': '2024-06-03T08:36:28.444764+0000', 'objects_scrubbed': 0, 'log_size': 0, 'log_dups_size': 0, 'ondisk_log_size': 0, 'stats_invalid': False, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': 'periodic scrub scheduled @ 2024-06-04T09:57:12.423360+0000', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 0, 'num_objects': 0, 'num_object_clones': 0, 'num_object_copies': 0, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 0, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 0, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 0, 'num_write_kb': 0, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 0, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [3], 'acting': [3], 'avail_no_missing': ['3'], 'object_location_counts': [], 'blocked_by': [], 'up_primary': 3, 'acting_primary': 3, 'purged_snaps': []} 2024-06-03T09:06:43.475 INFO:tasks.ceph.ceph_manager.ceph:PG 17.1a is not active+clean 2024-06-03T09:06:43.475 INFO:tasks.ceph.ceph_manager.ceph:{'pgid': '17.1a', 'version': "0'0", 'reported_seq': 960, 'reported_epoch': 1009, 'state': 'active+undersized', 'last_fresh': '2024-06-03T09:06:14.431224+0000', 'last_change': '2024-06-03T09:06:03.390734+0000', 'last_active': '2024-06-03T09:06:14.431224+0000', 'last_peered': '2024-06-03T09:06:14.431224+0000', 'last_clean': '2024-06-03T09:05:20.322313+0000', 'last_became_active': '2024-06-03T09:06:03.390734+0000', 'last_became_peered': '2024-06-03T09:06:03.390734+0000', 'last_unstale': '2024-06-03T09:06:14.431224+0000', 'last_undegraded': '2024-06-03T09:06:14.431224+0000', 'last_fullsized': '2024-06-03T09:06:03.343197+0000', 'mapping_epoch': 999, 'log_start': "0'0", 'ondisk_log_start': "0'0", 'created': 28, 'last_epoch_clean': 254, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2024-06-03T08:37:55.614497+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2024-06-03T08:37:55.614497+0000', 'last_clean_scrub_stamp': '2024-06-03T08:37:55.614497+0000', 'objects_scrubbed': 0, 'log_size': 0, 'log_dups_size': 0, 'ondisk_log_size': 0, 'stats_invalid': False, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': 'periodic scrub scheduled @ 2024-06-04T15:56:47.628875+0000', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 0, 'num_objects': 0, 'num_object_clones': 0, 'num_object_copies': 0, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 0, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 0, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 0, 'num_write_kb': 0, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 0, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [4], 'acting': [4], 'avail_no_missing': ['4'], 'object_location_counts': [], 'blocked_by': [], 'up_primary': 4, 'acting_primary': 4, 'purged_snaps': []} 2024-06-03T09:06:43.476 INFO:tasks.ceph.ceph_manager.ceph:PG 17.1e is not active+clean 2024-06-03T09:06:43.476 INFO:tasks.ceph.ceph_manager.ceph:{'pgid': '17.1e', 'version': "0'0", 'reported_seq': 148, 'reported_epoch': 1009, 'state': 'active+undersized', 'last_fresh': '2024-06-03T09:06:14.445917+0000', 'last_change': '2024-06-03T09:06:03.406181+0000', 'last_active': '2024-06-03T09:06:14.445917+0000', 'last_peered': '2024-06-03T09:06:14.445917+0000', 'last_clean': '2024-06-03T08:37:55.614497+0000', 'last_became_active': '2024-06-03T09:06:03.406181+0000', 'last_became_peered': '2024-06-03T09:06:03.406181+0000', 'last_unstale': '2024-06-03T09:06:14.445917+0000', 'last_undegraded': '2024-06-03T09:06:14.445917+0000', 'last_fullsized': '2024-06-03T09:06:03.363739+0000', 'mapping_epoch': 999, 'log_start': "0'0", 'ondisk_log_start': "0'0", 'created': 28, 'last_epoch_clean': 255, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2024-06-03T08:37:55.614497+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2024-06-03T08:37:55.614497+0000', 'last_clean_scrub_stamp': '2024-06-03T08:37:55.614497+0000', 'objects_scrubbed': 0, 'log_size': 0, 'log_dups_size': 0, 'ondisk_log_size': 0, 'stats_invalid': False, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': 'periodic scrub scheduled @ 2024-06-04T11:45:24.486600+0000', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 0, 'num_objects': 0, 'num_object_clones': 0, 'num_object_copies': 0, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 0, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 0, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 0, 'num_write_kb': 0, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 0, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [4], 'acting': [4], 'avail_no_missing': ['4'], 'object_location_counts': [], 'blocked_by': [], 'up_primary': 4, 'acting_primary': 4, 'purged_snaps': []} 2024-06-03T09:06:43.476 INFO:tasks.ceph.ceph_manager.ceph:PG 17.1c is not active+clean 2024-06-03T09:06:43.476 INFO:tasks.ceph.ceph_manager.ceph:{'pgid': '17.1c', 'version': "0'0", 'reported_seq': 944, 'reported_epoch': 1010, 'state': 'active+undersized', 'last_fresh': '2024-06-03T09:06:18.327492+0000', 'last_change': '2024-06-03T09:06:03.382210+0000', 'last_active': '2024-06-03T09:06:18.327492+0000', 'last_peered': '2024-06-03T09:06:18.327492+0000', 'last_clean': '2024-06-03T09:04:43.199660+0000', 'last_became_active': '2024-06-03T09:06:03.382210+0000', 'last_became_peered': '2024-06-03T09:06:03.382210+0000', 'last_unstale': '2024-06-03T09:06:18.327492+0000', 'last_undegraded': '2024-06-03T09:06:18.327492+0000', 'last_fullsized': '2024-06-03T09:06:03.277874+0000', 'mapping_epoch': 999, 'log_start': "0'0", 'ondisk_log_start': "0'0", 'created': 28, 'last_epoch_clean': 255, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2024-06-03T08:37:55.614497+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2024-06-03T08:37:55.614497+0000', 'last_clean_scrub_stamp': '2024-06-03T08:37:55.614497+0000', 'objects_scrubbed': 0, 'log_size': 0, 'log_dups_size': 0, 'ondisk_log_size': 0, 'stats_invalid': False, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': 'periodic scrub scheduled @ 2024-06-04T12:18:50.013884+0000', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 0, 'num_objects': 0, 'num_object_clones': 0, 'num_object_copies': 0, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 0, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 0, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 0, 'num_write_kb': 0, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 0, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [5], 'acting': [5], 'avail_no_missing': ['5'], 'object_location_counts': [], 'blocked_by': [], 'up_primary': 5, 'acting_primary': 5, 'purged_snaps': []} 2024-06-03T09:06:43.476 INFO:tasks.ceph.ceph_manager.ceph:PG 17.1 is not active+clean 2024-06-03T09:06:43.476 INFO:tasks.ceph.ceph_manager.ceph:{'pgid': '17.1', 'version': "65'7", 'reported_seq': 959, 'reported_epoch': 1009, 'state': 'active+undersized+degraded', 'last_fresh': '2024-06-03T09:06:13.709209+0000', 'last_change': '2024-06-03T09:06:03.370660+0000', 'last_active': '2024-06-03T09:06:13.709209+0000', 'last_peered': '2024-06-03T09:06:13.709209+0000', 'last_clean': '2024-06-03T09:05:03.733864+0000', 'last_became_active': '2024-06-03T09:06:03.370660+0000', 'last_became_peered': '2024-06-03T09:06:03.370660+0000', 'last_unstale': '2024-06-03T09:06:13.709209+0000', 'last_undegraded': '2024-06-03T09:06:03.266076+0000', 'last_fullsized': '2024-06-03T09:06:03.264525+0000', 'mapping_epoch': 999, 'log_start': "0'0", 'ondisk_log_start': "0'0", 'created': 28, 'last_epoch_clean': 255, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2024-06-03T08:37:55.614497+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2024-06-03T08:37:55.614497+0000', 'last_clean_scrub_stamp': '2024-06-03T08:37:55.614497+0000', 'objects_scrubbed': 0, 'log_size': 7, 'log_dups_size': 0, 'ondisk_log_size': 7, 'stats_invalid': False, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': 'periodic scrub scheduled @ 2024-06-04T11:55:18.194283+0000', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 5, 'num_objects': 1, 'num_object_clones': 0, 'num_object_copies': 2, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 1, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 1, 'num_whiteouts': 0, 'num_read': 6, 'num_read_kb': 6, 'num_write': 7, 'num_write_kb': 7, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 0, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [5], 'acting': [5], 'avail_no_missing': ['5'], 'object_location_counts': [{'shards': '5', 'objects': 1}], 'blocked_by': [], 'up_primary': 5, 'acting_primary': 5, 'purged_snaps': []} 2024-06-03T09:06:43.476 DEBUG:teuthology.orchestra.run.smithi088:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph status --format=json 2024-06-03T09:06:43.574 INFO:teuthology.orchestra.run.smithi088.stderr:2024-06-03T09:06:43.574+0000 7f5c93888640 10 monclient: get_monmap_and_config 2024-06-03T09:06:43.574 INFO:teuthology.orchestra.run.smithi088.stderr:2024-06-03T09:06:43.574+0000 7f5c93888640 10 monclient: build_initial_monmap 2024-06-03T09:06:43.575 INFO:teuthology.orchestra.run.smithi088.stderr:2024-06-03T09:06:43.574+0000 7f5c93888640 10 monclient: monmap: 2024-06-03T09:06:43.575 INFO:teuthology.orchestra.run.smithi088.stderr:epoch 0 2024-06-03T09:06:43.575 INFO:teuthology.orchestra.run.smithi088.stderr:fsid ec3c3fdd-3ac6-4d7d-9412-683130ed8b04 2024-06-03T09:06:43.575 INFO:teuthology.orchestra.run.smithi088.stderr:last_changed 2024-06-03T09:06:43.575603+0000 2024-06-03T09:06:43.575 INFO:teuthology.orchestra.run.smithi088.stderr:created 2024-06-03T09:06:43.575603+0000 2024-06-03T09:06:43.575 INFO:teuthology.orchestra.run.smithi088.stderr:min_mon_release 0 (unknown) 2024-06-03T09:06:43.575 INFO:teuthology.orchestra.run.smithi088.stderr:election_strategy: 3 2024-06-03T09:06:43.575 INFO:teuthology.orchestra.run.smithi088.stderr:0: v1:172.21.15.88:6789/0 mon.noname-a 2024-06-03T09:06:43.575 INFO:teuthology.orchestra.run.smithi088.stderr:1: v1:172.21.15.192:6789/0 mon.noname-c 2024-06-03T09:06:43.575 INFO:teuthology.orchestra.run.smithi088.stderr:2: v1:172.21.15.88:6790/0 mon.noname-b 2024-06-03T09:06:43.575 INFO:teuthology.orchestra.run.smithi088.stderr: 2024-06-03T09:06:43.576 INFO:teuthology.orchestra.run.smithi088.stderr:2024-06-03T09:06:43.575+0000 7f5c93888640 1 Processor -- start 2024-06-03T09:06:43.577 INFO:teuthology.orchestra.run.smithi088.stderr:2024-06-03T09:06:43.575+0000 7f5c93888640 1 -- start start 2024-06-03T09:06:43.577 INFO:teuthology.orchestra.run.smithi088.stderr:2024-06-03T09:06:43.575+0000 7f5c93888640 10 monclient: init 2024-06-03T09:06:43.577 INFO:teuthology.orchestra.run.smithi088.stderr:2024-06-03T09:06:43.575+0000 7f5c93888640 10 monclient: _reopen_session rank -1 2024-06-03T09:06:43.577 INFO:teuthology.orchestra.run.smithi088.stderr:2024-06-03T09:06:43.575+0000 7f5c93888640 10 monclient: _add_conns ranks=[0,1,2] 2024-06-03T09:06:43.577 INFO:teuthology.orchestra.run.smithi088.stderr:2024-06-03T09:06:43.575+0000 7f5c93888640 10 monclient(hunting): picked mon.noname-a con 0x7f5c8c06f970 addr v1:172.21.15.88:6789/0 2024-06-03T09:06:43.577 INFO:teuthology.orchestra.run.smithi088.stderr:2024-06-03T09:06:43.575+0000 7f5c93888640 10 monclient(hunting): picked mon.noname-c con 0x7f5c8c0ad840 addr v1:172.21.15.192:6789/0 2024-06-03T09:06:43.577 INFO:teuthology.orchestra.run.smithi088.stderr:2024-06-03T09:06:43.575+0000 7f5c93888640 10 monclient(hunting): picked mon.noname-b con 0x7f5c8c06fd50 addr v1:172.21.15.88:6790/0 2024-06-03T09:06:43.577 INFO:teuthology.orchestra.run.smithi088.stderr:2024-06-03T09:06:43.575+0000 7f5c93888640 1 -- --> v1:172.21.15.88:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f5c8c071be0 con 0x7f5c8c06f970 2024-06-03T09:06:43.577 INFO:teuthology.orchestra.run.smithi088.stderr:2024-06-03T09:06:43.575+0000 7f5c93888640 1 -- --> v1:172.21.15.192:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f5c8c071ec0 con 0x7f5c8c0ad840 2024-06-03T09:06:43.577 INFO:teuthology.orchestra.run.smithi088.stderr:2024-06-03T09:06:43.575+0000 7f5c93888640 1 -- --> v1:172.21.15.88:6790/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f5c8c0708e0 con 0x7f5c8c06fd50 2024-06-03T09:06:43.577 INFO:teuthology.orchestra.run.smithi088.stderr:2024-06-03T09:06:43.575+0000 7f5c93888640 10 monclient(hunting): _renew_subs 2024-06-03T09:06:43.577 INFO:teuthology.orchestra.run.smithi088.stderr:2024-06-03T09:06:43.575+0000 7f5c93888640 10 monclient(hunting): authenticate will time out at 3059.385922s 2024-06-03T09:06:43.577 INFO:teuthology.orchestra.run.smithi088.stderr:2024-06-03T09:06:43.576+0000 7f5c91624640 1 --1- >> v1:172.21.15.88:6789/0 conn(0x7f5c8c06f970 0x7f5c8c13fa70 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.88:6789/0 says I am v1:172.21.15.88:54442/0 (socket says 172.21.15.88:54442) 2024-06-03T09:06:43.578 INFO:teuthology.orchestra.run.smithi088.stderr:2024-06-03T09:06:43.576+0000 7f5c91e25640 1 --1- >> v1:172.21.15.88:6790/0 conn(0x7f5c8c06fd50 0x7f5c8c14abf0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.88:6790/0 says I am v1:172.21.15.88:41306/0 (socket says 172.21.15.88:41306) 2024-06-03T09:06:43.578 INFO:teuthology.orchestra.run.smithi088.stderr:2024-06-03T09:06:43.576+0000 7f5c91624640 1 -- 172.21.15.88:0/844628958 learned_addr learned my addr 172.21.15.88:0/844628958 (peer_addr_for_me v1:172.21.15.88:0/0) 2024-06-03T09:06:43.579 INFO:teuthology.orchestra.run.smithi088.stderr:2024-06-03T09:06:43.578+0000 7f5c83fff640 1 -- 172.21.15.88:0/844628958 <== mon.2 v1:172.21.15.88:6790/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 1990438790 0 0) 0x7f5c8c0708e0 con 0x7f5c8c06fd50 2024-06-03T09:06:43.579 INFO:teuthology.orchestra.run.smithi088.stderr:2024-06-03T09:06:43.578+0000 7f5c83fff640 10 monclient(hunting): _negotiate 2024-06-03T09:06:43.579 INFO:teuthology.orchestra.run.smithi088.stderr:2024-06-03T09:06:43.578+0000 7f5c83fff640 10 monclient(hunting): _init_auth method 2 2024-06-03T09:06:43.579 INFO:teuthology.orchestra.run.smithi088.stderr:2024-06-03T09:06:43.578+0000 7f5c83fff640 10 monclient(hunting): _init_auth creating new auth 2024-06-03T09:06:43.579 INFO:teuthology.orchestra.run.smithi088.stderr:2024-06-03T09:06:43.578+0000 7f5c83fff640 10 monclient(hunting): my global_id is 30188 2024-06-03T09:06:43.579 INFO:teuthology.orchestra.run.smithi088.stderr:2024-06-03T09:06:43.578+0000 7f5c83fff640 1 -- 172.21.15.88:0/844628958 --> v1:172.21.15.88:6790/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f5c78003520 con 0x7f5c8c06fd50 2024-06-03T09:06:43.579 INFO:teuthology.orchestra.run.smithi088.stderr:2024-06-03T09:06:43.578+0000 7f5c83fff640 1 -- 172.21.15.88:0/844628958 <== mon.0 v1:172.21.15.88:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 1040273857 0 0) 0x7f5c8c071be0 con 0x7f5c8c06f970 2024-06-03T09:06:43.579 INFO:teuthology.orchestra.run.smithi088.stderr:2024-06-03T09:06:43.578+0000 7f5c83fff640 10 monclient(hunting): _negotiate 2024-06-03T09:06:43.579 INFO:teuthology.orchestra.run.smithi088.stderr:2024-06-03T09:06:43.578+0000 7f5c83fff640 10 monclient(hunting): _init_auth method 2 2024-06-03T09:06:43.579 INFO:teuthology.orchestra.run.smithi088.stderr:2024-06-03T09:06:43.578+0000 7f5c83fff640 10 monclient(hunting): _init_auth creating new auth 2024-06-03T09:06:43.579 INFO:teuthology.orchestra.run.smithi088.stderr:2024-06-03T09:06:43.578+0000 7f5c83fff640 10 monclient(hunting): my global_id is 30684 2024-06-03T09:06:43.579 INFO:teuthology.orchestra.run.smithi088.stderr:2024-06-03T09:06:43.578+0000 7f5c83fff640 1 -- 172.21.15.88:0/844628958 --> v1:172.21.15.88:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f5c8c0708e0 con 0x7f5c8c06f970 2024-06-03T09:06:43.579 INFO:teuthology.orchestra.run.smithi088.stderr:2024-06-03T09:06:43.578+0000 7f5c83fff640 1 -- 172.21.15.88:0/844628958 <== mon.1 v1:172.21.15.192:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 2682984726 0 0) 0x7f5c8c071ec0 con 0x7f5c8c0ad840 2024-06-03T09:06:43.580 INFO:teuthology.orchestra.run.smithi088.stderr:2024-06-03T09:06:43.578+0000 7f5c83fff640 10 monclient(hunting): _negotiate 2024-06-03T09:06:43.580 INFO:teuthology.orchestra.run.smithi088.stderr:2024-06-03T09:06:43.578+0000 7f5c83fff640 10 monclient(hunting): _init_auth method 2 2024-06-03T09:06:43.580 INFO:teuthology.orchestra.run.smithi088.stderr:2024-06-03T09:06:43.578+0000 7f5c83fff640 10 monclient(hunting): _init_auth creating new auth 2024-06-03T09:06:43.580 INFO:teuthology.orchestra.run.smithi088.stderr:2024-06-03T09:06:43.578+0000 7f5c83fff640 10 monclient(hunting): my global_id is 30262 2024-06-03T09:06:43.580 INFO:teuthology.orchestra.run.smithi088.stderr:2024-06-03T09:06:43.578+0000 7f5c83fff640 1 -- 172.21.15.88:0/844628958 --> v1:172.21.15.192:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f5c8c071be0 con 0x7f5c8c0ad840 2024-06-03T09:06:43.582 INFO:teuthology.orchestra.run.smithi088.stderr:2024-06-03T09:06:43.582+0000 7f5c83fff640 1 -- 172.21.15.88:0/844628958 <== mon.0 v1:172.21.15.88:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 214+0+0 (unknown 157799921 0 0) 0x7f5c8c0708e0 con 0x7f5c8c06f970 2024-06-03T09:06:43.582 INFO:teuthology.orchestra.run.smithi088.stderr:2024-06-03T09:06:43.582+0000 7f5c83fff640 1 -- 172.21.15.88:0/844628958 >> v1:172.21.15.88:6790/0 conn(0x7f5c8c06fd50 legacy=0x7f5c8c14abf0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-06-03T09:06:43.582 INFO:teuthology.orchestra.run.smithi088.stderr:2024-06-03T09:06:43.582+0000 7f5c83fff640 1 -- 172.21.15.88:0/844628958 >> v1:172.21.15.192:6789/0 conn(0x7f5c8c0ad840 legacy=0x7f5c8c13c770 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-06-03T09:06:43.582 INFO:teuthology.orchestra.run.smithi088.stderr:2024-06-03T09:06:43.582+0000 7f5c83fff640 10 monclient: _finish_hunting 0 2024-06-03T09:06:43.582 INFO:teuthology.orchestra.run.smithi088.stderr:2024-06-03T09:06:43.582+0000 7f5c83fff640 1 monclient: found mon.noname-a 2024-06-03T09:06:43.582 INFO:teuthology.orchestra.run.smithi088.stderr:2024-06-03T09:06:43.582+0000 7f5c83fff640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-06-03T09:06:43.582 INFO:teuthology.orchestra.run.smithi088.stderr:2024-06-03T09:06:43.582+0000 7f5c83fff640 10 monclient: _send_mon_message to mon.noname-a at v1:172.21.15.88:6789/0 2024-06-03T09:06:43.582 INFO:teuthology.orchestra.run.smithi088.stderr:2024-06-03T09:06:43.582+0000 7f5c83fff640 1 -- 172.21.15.88:0/844628958 --> v1:172.21.15.88:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f5c8c13f550 con 0x7f5c8c06f970 2024-06-03T09:06:43.582 INFO:teuthology.orchestra.run.smithi088.stderr:2024-06-03T09:06:43.582+0000 7f5c83fff640 10 monclient: _finish_auth 0 2024-06-03T09:06:43.583 INFO:teuthology.orchestra.run.smithi088.stderr:2024-06-03T09:06:43.582+0000 7f5c83fff640 10 monclient: _check_auth_tickets 2024-06-03T09:06:43.583 INFO:teuthology.orchestra.run.smithi088.stderr:2024-06-03T09:06:43.582+0000 7f5c83fff640 20 monclient: _check_auth_rotating not needed by client.admin 2024-06-03T09:06:43.583 INFO:teuthology.orchestra.run.smithi088.stderr:2024-06-03T09:06:43.582+0000 7f5c83fff640 1 -- 172.21.15.88:0/844628958 <== mon.0 v1:172.21.15.88:6789/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 2234374065 0 0) 0x7f5c88001570 con 0x7f5c8c06f970 2024-06-03T09:06:43.583 INFO:teuthology.orchestra.run.smithi088.stderr:2024-06-03T09:06:43.582+0000 7f5c93888640 5 monclient: authenticate success, global_id 30684 2024-06-03T09:06:43.583 INFO:teuthology.orchestra.run.smithi088.stderr:2024-06-03T09:06:43.582+0000 7f5c93888640 20 monclient: get_monmap_and_config waiting for monmap|config 2024-06-03T09:06:43.583 INFO:teuthology.orchestra.run.smithi088.stderr:2024-06-03T09:06:43.582+0000 7f5c83fff640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-06-03T09:06:43.583 INFO:teuthology.orchestra.run.smithi088.stderr:2024-06-03T09:06:43.582+0000 7f5c83fff640 10 monclient: got monmap 1 from mon.noname-a (according to old e1) 2024-06-03T09:06:43.583 INFO:teuthology.orchestra.run.smithi088.stderr:2024-06-03T09:06:43.582+0000 7f5c83fff640 10 monclient: dump: 2024-06-03T09:06:43.583 INFO:teuthology.orchestra.run.smithi088.stderr:epoch 1 2024-06-03T09:06:43.583 INFO:teuthology.orchestra.run.smithi088.stderr:fsid ec3c3fdd-3ac6-4d7d-9412-683130ed8b04 2024-06-03T09:06:43.583 INFO:teuthology.orchestra.run.smithi088.stderr:last_changed 2024-06-03T08:34:31.885125+0000 2024-06-03T09:06:43.583 INFO:teuthology.orchestra.run.smithi088.stderr:created 2024-06-03T08:34:31.885125+0000 2024-06-03T09:06:43.583 INFO:teuthology.orchestra.run.smithi088.stderr:min_mon_release 18 (reef) 2024-06-03T09:06:43.583 INFO:teuthology.orchestra.run.smithi088.stderr:election_strategy: 3 2024-06-03T09:06:43.584 INFO:teuthology.orchestra.run.smithi088.stderr:0: v1:172.21.15.88:6789/0 mon.a 2024-06-03T09:06:43.584 INFO:teuthology.orchestra.run.smithi088.stderr:1: v1:172.21.15.192:6789/0 mon.b 2024-06-03T09:06:43.584 INFO:teuthology.orchestra.run.smithi088.stderr:2: v1:172.21.15.88:6790/0 mon.c 2024-06-03T09:06:43.584 INFO:teuthology.orchestra.run.smithi088.stderr: 2024-06-03T09:06:43.584 INFO:teuthology.orchestra.run.smithi088.stderr:2024-06-03T09:06:43.582+0000 7f5c93888640 20 monclient: get_monmap_and_config waiting for monmap|config 2024-06-03T09:06:43.587 INFO:teuthology.orchestra.run.smithi088.stderr:2024-06-03T09:06:43.583+0000 7f5c83fff640 1 -- 172.21.15.88:0/844628958 <== mon.0 v1:172.21.15.88:6789/0 4 ==== config(0 keys) v1 ==== 4+0+0 (unknown 0 0 0) 0x7f5c88001810 con 0x7f5c8c06f970 2024-06-03T09:06:43.587 INFO:teuthology.orchestra.run.smithi088.stderr:2024-06-03T09:06:43.583+0000 7f5c83fff640 10 monclient: handle_config config(0 keys) v1 2024-06-03T09:06:43.587 INFO:teuthology.orchestra.run.smithi088.stderr:2024-06-03T09:06:43.583+0000 7f5c93888640 10 monclient: get_monmap_and_config success 2024-06-03T09:06:43.587 INFO:teuthology.orchestra.run.smithi088.stderr:2024-06-03T09:06:43.583+0000 7f5c93888640 4 set_mon_vals no callback set 2024-06-03T09:06:43.587 INFO:teuthology.orchestra.run.smithi088.stderr:2024-06-03T09:06:43.584+0000 7f5c83fff640 1 -- 172.21.15.88:0/844628958 <== mon.0 v1:172.21.15.88:6789/0 5 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 2234374065 0 0) 0x7f5c88001be0 con 0x7f5c8c06f970 2024-06-03T09:06:43.587 INFO:teuthology.orchestra.run.smithi088.stderr:2024-06-03T09:06:43.584+0000 7f5c83fff640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-06-03T09:06:43.587 INFO:teuthology.orchestra.run.smithi088.stderr:2024-06-03T09:06:43.584+0000 7f5c83fff640 10 monclient: got monmap 1 from mon.a (according to old e1) 2024-06-03T09:06:43.587 INFO:teuthology.orchestra.run.smithi088.stderr:2024-06-03T09:06:43.584+0000 7f5c83fff640 10 monclient: dump: 2024-06-03T09:06:43.588 INFO:teuthology.orchestra.run.smithi088.stderr:epoch 1 2024-06-03T09:06:43.588 INFO:teuthology.orchestra.run.smithi088.stderr:fsid ec3c3fdd-3ac6-4d7d-9412-683130ed8b04 2024-06-03T09:06:43.588 INFO:teuthology.orchestra.run.smithi088.stderr:last_changed 2024-06-03T08:34:31.885125+0000 2024-06-03T09:06:43.588 INFO:teuthology.orchestra.run.smithi088.stderr:created 2024-06-03T08:34:31.885125+0000 2024-06-03T09:06:43.588 INFO:teuthology.orchestra.run.smithi088.stderr:min_mon_release 18 (reef) 2024-06-03T09:06:43.588 INFO:teuthology.orchestra.run.smithi088.stderr:election_strategy: 3 2024-06-03T09:06:43.588 INFO:teuthology.orchestra.run.smithi088.stderr:0: v1:172.21.15.88:6789/0 mon.a 2024-06-03T09:06:43.588 INFO:teuthology.orchestra.run.smithi088.stderr:1: v1:172.21.15.192:6789/0 mon.b 2024-06-03T09:06:43.588 INFO:teuthology.orchestra.run.smithi088.stderr:2: v1:172.21.15.88:6790/0 mon.c 2024-06-03T09:06:43.588 INFO:teuthology.orchestra.run.smithi088.stderr: 2024-06-03T09:06:43.588 INFO:teuthology.orchestra.run.smithi088.stderr:2024-06-03T09:06:43.584+0000 7f5c93888640 10 monclient: shutdown 2024-06-03T09:06:43.588 INFO:teuthology.orchestra.run.smithi088.stderr:2024-06-03T09:06:43.584+0000 7f5c93888640 20 monclient: shutdown discarding 0 pending message(s) 2024-06-03T09:06:43.588 INFO:teuthology.orchestra.run.smithi088.stderr:2024-06-03T09:06:43.584+0000 7f5c93888640 1 -- 172.21.15.88:0/844628958 >> v1:172.21.15.88:6789/0 conn(0x7f5c8c06f970 legacy=0x7f5c8c13fa70 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-06-03T09:06:43.588 INFO:teuthology.orchestra.run.smithi088.stderr:2024-06-03T09:06:43.584+0000 7f5c93888640 1 -- 172.21.15.88:0/844628958 shutdown_connections 2024-06-03T09:06:43.589 INFO:teuthology.orchestra.run.smithi088.stderr:2024-06-03T09:06:43.584+0000 7f5c93888640 1 -- 172.21.15.88:0/844628958 >> 172.21.15.88:0/844628958 conn(0x7f5c8c0b2e40 msgr2=0x7f5c8c0b3250 unknown :-1 s=STATE_NONE l=0).mark_down 2024-06-03T09:06:43.589 INFO:teuthology.orchestra.run.smithi088.stderr:2024-06-03T09:06:43.584+0000 7f5c93888640 1 -- 172.21.15.88:0/844628958 shutdown_connections 2024-06-03T09:06:43.589 INFO:teuthology.orchestra.run.smithi088.stderr:2024-06-03T09:06:43.584+0000 7f5c93888640 1 -- 172.21.15.88:0/844628958 wait complete. 2024-06-03T09:06:43.589 INFO:teuthology.orchestra.run.smithi088.stderr:2024-06-03T09:06:43.585+0000 7f5c93888640 10 monclient: build_initial_monmap 2024-06-03T09:06:43.589 INFO:teuthology.orchestra.run.smithi088.stderr:2024-06-03T09:06:43.585+0000 7f5c93888640 10 monclient: monmap: 2024-06-03T09:06:43.589 INFO:teuthology.orchestra.run.smithi088.stderr:epoch 0 2024-06-03T09:06:43.589 INFO:teuthology.orchestra.run.smithi088.stderr:fsid 00000000-0000-0000-0000-000000000000 2024-06-03T09:06:43.589 INFO:teuthology.orchestra.run.smithi088.stderr:last_changed 0.000000 2024-06-03T09:06:43.589 INFO:teuthology.orchestra.run.smithi088.stderr:created 0.000000 2024-06-03T09:06:43.589 INFO:teuthology.orchestra.run.smithi088.stderr:min_mon_release 0 (unknown) 2024-06-03T09:06:43.589 INFO:teuthology.orchestra.run.smithi088.stderr:election_strategy: 1 2024-06-03T09:06:43.589 INFO:teuthology.orchestra.run.smithi088.stderr:0: v1:172.21.15.88:6789/0 mon.noname-a 2024-06-03T09:06:43.589 INFO:teuthology.orchestra.run.smithi088.stderr:1: v1:172.21.15.192:6789/0 mon.noname-b 2024-06-03T09:06:43.589 INFO:teuthology.orchestra.run.smithi088.stderr:2: v1:172.21.15.88:6790/0 mon.noname-c 2024-06-03T09:06:43.590 INFO:teuthology.orchestra.run.smithi088.stderr: 2024-06-03T09:06:43.590 INFO:teuthology.orchestra.run.smithi088.stderr:2024-06-03T09:06:43.585+0000 7f5c93888640 1 librados: starting msgr at 2024-06-03T09:06:43.590 INFO:teuthology.orchestra.run.smithi088.stderr:2024-06-03T09:06:43.585+0000 7f5c93888640 1 librados: starting objecter 2024-06-03T09:06:43.590 INFO:teuthology.orchestra.run.smithi088.stderr:2024-06-03T09:06:43.585+0000 7f5c93888640 1 Processor -- start 2024-06-03T09:06:43.590 INFO:teuthology.orchestra.run.smithi088.stderr:2024-06-03T09:06:43.585+0000 7f5c93888640 1 -- start start 2024-06-03T09:06:43.590 INFO:teuthology.orchestra.run.smithi088.stderr:2024-06-03T09:06:43.585+0000 7f5c93888640 1 librados: setting wanted keys 2024-06-03T09:06:43.590 INFO:teuthology.orchestra.run.smithi088.stderr:2024-06-03T09:06:43.585+0000 7f5c93888640 1 librados: calling monclient init 2024-06-03T09:06:43.590 INFO:teuthology.orchestra.run.smithi088.stderr:2024-06-03T09:06:43.585+0000 7f5c93888640 10 monclient: init 2024-06-03T09:06:43.590 INFO:teuthology.orchestra.run.smithi088.stderr:2024-06-03T09:06:43.586+0000 7f5c93888640 10 monclient: _reopen_session rank -1 2024-06-03T09:06:43.590 INFO:teuthology.orchestra.run.smithi088.stderr:2024-06-03T09:06:43.586+0000 7f5c93888640 10 monclient: _add_conns ranks=[1,0,2] 2024-06-03T09:06:43.590 INFO:teuthology.orchestra.run.smithi088.stderr:2024-06-03T09:06:43.586+0000 7f5c93888640 10 monclient(hunting): picked mon.noname-b con 0x7f5c8c06f970 addr v1:172.21.15.192:6789/0 2024-06-03T09:06:43.590 INFO:teuthology.orchestra.run.smithi088.stderr:2024-06-03T09:06:43.586+0000 7f5c93888640 10 monclient(hunting): picked mon.noname-a con 0x7f5c8c06fd50 addr v1:172.21.15.88:6789/0 2024-06-03T09:06:43.590 INFO:teuthology.orchestra.run.smithi088.stderr:2024-06-03T09:06:43.586+0000 7f5c93888640 10 monclient(hunting): picked mon.noname-c con 0x7f5c8c0ad840 addr v1:172.21.15.88:6790/0 2024-06-03T09:06:43.590 INFO:teuthology.orchestra.run.smithi088.stderr:2024-06-03T09:06:43.586+0000 7f5c93888640 1 -- --> v1:172.21.15.88:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f5c8c1d6360 con 0x7f5c8c06fd50 2024-06-03T09:06:43.591 INFO:teuthology.orchestra.run.smithi088.stderr:2024-06-03T09:06:43.586+0000 7f5c93888640 1 -- --> v1:172.21.15.192:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f5c8c1d71e0 con 0x7f5c8c06f970 2024-06-03T09:06:43.591 INFO:teuthology.orchestra.run.smithi088.stderr:2024-06-03T09:06:43.586+0000 7f5c93888640 1 -- --> v1:172.21.15.88:6790/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f5c8c1d65b0 con 0x7f5c8c0ad840 2024-06-03T09:06:43.591 INFO:teuthology.orchestra.run.smithi088.stderr:2024-06-03T09:06:43.586+0000 7f5c93888640 10 monclient(hunting): _renew_subs 2024-06-03T09:06:43.591 INFO:teuthology.orchestra.run.smithi088.stderr:2024-06-03T09:06:43.586+0000 7f5c93888640 10 monclient(hunting): authenticate will time out at 3059.396191s 2024-06-03T09:06:43.591 INFO:teuthology.orchestra.run.smithi088.stderr:2024-06-03T09:06:43.586+0000 7f5c90e23640 1 --1- >> v1:172.21.15.88:6789/0 conn(0x7f5c8c06fd50 0x7f5c8c14a3a0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=1).handle_server_banner_and_identify peer v1:172.21.15.88:6789/0 says I am v1:172.21.15.88:54452/0 (socket says 172.21.15.88:54452) 2024-06-03T09:06:43.591 INFO:teuthology.orchestra.run.smithi088.stderr:2024-06-03T09:06:43.586+0000 7f5c91e25640 1 --1- >> v1:172.21.15.88:6790/0 conn(0x7f5c8c0ad840 0x7f5c8c1e1c20 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=1).handle_server_banner_and_identify peer v1:172.21.15.88:6790/0 says I am v1:172.21.15.88:41316/0 (socket says 172.21.15.88:41316) 2024-06-03T09:06:43.591 INFO:teuthology.orchestra.run.smithi088.stderr:2024-06-03T09:06:43.586+0000 7f5c90e23640 1 -- 172.21.15.88:0/1130531237 learned_addr learned my addr 172.21.15.88:0/1130531237 (peer_addr_for_me v1:172.21.15.88:0/0) 2024-06-03T09:06:43.591 INFO:teuthology.orchestra.run.smithi088.stderr:2024-06-03T09:06:43.587+0000 7f5c81ffb640 10 client.?.objecter ms_handle_connect 0x7f5c8c06fd50 2024-06-03T09:06:43.591 INFO:teuthology.orchestra.run.smithi088.stderr:2024-06-03T09:06:43.587+0000 7f5c81ffb640 10 client.?.objecter resend_mon_ops 2024-06-03T09:06:43.591 INFO:teuthology.orchestra.run.smithi088.stderr:2024-06-03T09:06:43.587+0000 7f5c81ffb640 10 client.?.objecter ms_handle_connect 0x7f5c8c06f970 2024-06-03T09:06:43.591 INFO:teuthology.orchestra.run.smithi088.stderr:2024-06-03T09:06:43.587+0000 7f5c81ffb640 10 client.?.objecter resend_mon_ops 2024-06-03T09:06:43.591 INFO:teuthology.orchestra.run.smithi088.stderr:2024-06-03T09:06:43.587+0000 7f5c81ffb640 10 client.?.objecter ms_handle_connect 0x7f5c8c0ad840 2024-06-03T09:06:43.591 INFO:teuthology.orchestra.run.smithi088.stderr:2024-06-03T09:06:43.587+0000 7f5c81ffb640 10 client.?.objecter resend_mon_ops 2024-06-03T09:06:43.592 INFO:teuthology.orchestra.run.smithi088.stderr:2024-06-03T09:06:43.588+0000 7f5c81ffb640 1 -- 172.21.15.88:0/1130531237 <== mon.1 v1:172.21.15.192:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 3179101348 0 0) 0x7f5c8c1d71e0 con 0x7f5c8c06f970 2024-06-03T09:06:43.592 INFO:teuthology.orchestra.run.smithi088.stderr:2024-06-03T09:06:43.588+0000 7f5c81ffb640 10 monclient(hunting): _negotiate 2024-06-03T09:06:43.593 INFO:teuthology.orchestra.run.smithi088.stderr:2024-06-03T09:06:43.588+0000 7f5c81ffb640 10 monclient(hunting): _init_auth method 2 2024-06-03T09:06:43.593 INFO:teuthology.orchestra.run.smithi088.stderr:2024-06-03T09:06:43.588+0000 7f5c81ffb640 10 monclient(hunting): _init_auth creating new auth 2024-06-03T09:06:43.593 INFO:teuthology.orchestra.run.smithi088.stderr:2024-06-03T09:06:43.588+0000 7f5c81ffb640 10 monclient(hunting): my global_id is 30265 2024-06-03T09:06:43.593 INFO:teuthology.orchestra.run.smithi088.stderr:2024-06-03T09:06:43.588+0000 7f5c81ffb640 1 -- 172.21.15.88:0/1130531237 --> v1:172.21.15.192:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f5c70004eb0 con 0x7f5c8c06f970 2024-06-03T09:06:43.593 INFO:teuthology.orchestra.run.smithi088.stderr:2024-06-03T09:06:43.589+0000 7f5c81ffb640 1 -- 172.21.15.88:0/1130531237 <== mon.0 v1:172.21.15.88:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 3531460045 0 0) 0x7f5c8c1d6360 con 0x7f5c8c06fd50 2024-06-03T09:06:43.593 INFO:teuthology.orchestra.run.smithi088.stderr:2024-06-03T09:06:43.589+0000 7f5c81ffb640 10 monclient(hunting): _negotiate 2024-06-03T09:06:43.593 INFO:teuthology.orchestra.run.smithi088.stderr:2024-06-03T09:06:43.589+0000 7f5c81ffb640 10 monclient(hunting): _init_auth method 2 2024-06-03T09:06:43.593 INFO:teuthology.orchestra.run.smithi088.stderr:2024-06-03T09:06:43.589+0000 7f5c81ffb640 10 monclient(hunting): _init_auth creating new auth 2024-06-03T09:06:43.593 INFO:teuthology.orchestra.run.smithi088.stderr:2024-06-03T09:06:43.589+0000 7f5c81ffb640 10 monclient(hunting): my global_id is 30687 2024-06-03T09:06:43.593 INFO:teuthology.orchestra.run.smithi088.stderr:2024-06-03T09:06:43.589+0000 7f5c81ffb640 1 -- 172.21.15.88:0/1130531237 --> v1:172.21.15.88:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f5c8c1d71e0 con 0x7f5c8c06fd50 2024-06-03T09:06:43.593 INFO:teuthology.orchestra.run.smithi088.stderr:2024-06-03T09:06:43.589+0000 7f5c81ffb640 1 -- 172.21.15.88:0/1130531237 <== mon.2 v1:172.21.15.88:6790/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 2923192795 0 0) 0x7f5c8c1d65b0 con 0x7f5c8c0ad840 2024-06-03T09:06:43.593 INFO:teuthology.orchestra.run.smithi088.stderr:2024-06-03T09:06:43.589+0000 7f5c81ffb640 10 monclient(hunting): _negotiate 2024-06-03T09:06:43.593 INFO:teuthology.orchestra.run.smithi088.stderr:2024-06-03T09:06:43.589+0000 7f5c81ffb640 10 monclient(hunting): _init_auth method 2 2024-06-03T09:06:43.593 INFO:teuthology.orchestra.run.smithi088.stderr:2024-06-03T09:06:43.589+0000 7f5c81ffb640 10 monclient(hunting): _init_auth creating new auth 2024-06-03T09:06:43.594 INFO:teuthology.orchestra.run.smithi088.stderr:2024-06-03T09:06:43.589+0000 7f5c81ffb640 10 monclient(hunting): my global_id is 30191 2024-06-03T09:06:43.594 INFO:teuthology.orchestra.run.smithi088.stderr:2024-06-03T09:06:43.589+0000 7f5c81ffb640 1 -- 172.21.15.88:0/1130531237 --> v1:172.21.15.88:6790/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f5c8c1d6360 con 0x7f5c8c0ad840 2024-06-03T09:06:43.595 INFO:teuthology.orchestra.run.smithi088.stderr:2024-06-03T09:06:43.593+0000 7f5c81ffb640 1 -- 172.21.15.88:0/1130531237 <== mon.1 v1:172.21.15.192:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 764+0+0 (unknown 1666572498 0 0) 0x7f5c70004eb0 con 0x7f5c8c06f970 2024-06-03T09:06:43.595 INFO:teuthology.orchestra.run.smithi088.stderr:2024-06-03T09:06:43.593+0000 7f5c81ffb640 1 -- 172.21.15.88:0/1130531237 --> v1:172.21.15.192:6789/0 -- auth(proto 2 165 bytes epoch 0) v1 -- 0x7f5c8c1d65b0 con 0x7f5c8c06f970 2024-06-03T09:06:43.595 INFO:teuthology.orchestra.run.smithi088.stderr:2024-06-03T09:06:43.593+0000 7f5c81ffb640 1 -- 172.21.15.88:0/1130531237 <== mon.1 v1:172.21.15.192:6789/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 2234374065 0 0) 0x7f5c88001570 con 0x7f5c8c06f970 2024-06-03T09:06:43.595 INFO:teuthology.orchestra.run.smithi088.stderr:2024-06-03T09:06:43.593+0000 7f5c81ffb640 10 monclient(hunting): handle_monmap mon_map magic: 0 v1 2024-06-03T09:06:43.595 INFO:teuthology.orchestra.run.smithi088.stderr:2024-06-03T09:06:43.593+0000 7f5c81ffb640 10 monclient(hunting): got monmap 1 from mon.noname-b (according to old e1) 2024-06-03T09:06:43.595 INFO:teuthology.orchestra.run.smithi088.stderr:2024-06-03T09:06:43.593+0000 7f5c81ffb640 10 monclient(hunting): dump: 2024-06-03T09:06:43.595 INFO:teuthology.orchestra.run.smithi088.stderr:epoch 1 2024-06-03T09:06:43.595 INFO:teuthology.orchestra.run.smithi088.stderr:fsid ec3c3fdd-3ac6-4d7d-9412-683130ed8b04 2024-06-03T09:06:43.595 INFO:teuthology.orchestra.run.smithi088.stderr:last_changed 2024-06-03T08:34:31.885125+0000 2024-06-03T09:06:43.596 INFO:teuthology.orchestra.run.smithi088.stderr:created 2024-06-03T08:34:31.885125+0000 2024-06-03T09:06:43.596 INFO:teuthology.orchestra.run.smithi088.stderr:min_mon_release 18 (reef) 2024-06-03T09:06:43.596 INFO:teuthology.orchestra.run.smithi088.stderr:election_strategy: 3 2024-06-03T09:06:43.596 INFO:teuthology.orchestra.run.smithi088.stderr:0: v1:172.21.15.88:6789/0 mon.a 2024-06-03T09:06:43.596 INFO:teuthology.orchestra.run.smithi088.stderr:1: v1:172.21.15.192:6789/0 mon.b 2024-06-03T09:06:43.596 INFO:teuthology.orchestra.run.smithi088.stderr:2: v1:172.21.15.88:6790/0 mon.c 2024-06-03T09:06:43.596 INFO:teuthology.orchestra.run.smithi088.stderr: 2024-06-03T09:06:43.596 INFO:teuthology.orchestra.run.smithi088.stderr:2024-06-03T09:06:43.593+0000 7f5c81ffb640 10 monclient(hunting): _finish_auth 0 2024-06-03T09:06:43.596 INFO:teuthology.orchestra.run.smithi088.stderr:2024-06-03T09:06:43.594+0000 7f5c81ffb640 1 -- 172.21.15.88:0/1130531237 <== mon.2 v1:172.21.15.88:6790/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 764+0+0 (unknown 1340260496 0 0) 0x7f5c8c1d6360 con 0x7f5c8c0ad840 2024-06-03T09:06:43.596 INFO:teuthology.orchestra.run.smithi088.stderr:2024-06-03T09:06:43.594+0000 7f5c81ffb640 1 -- 172.21.15.88:0/1130531237 --> v1:172.21.15.88:6790/0 -- auth(proto 2 165 bytes epoch 0) v1 -- 0x7f5c70004eb0 con 0x7f5c8c0ad840 2024-06-03T09:06:43.596 INFO:teuthology.orchestra.run.smithi088.stderr:2024-06-03T09:06:43.594+0000 7f5c81ffb640 1 -- 172.21.15.88:0/1130531237 <== mon.0 v1:172.21.15.88:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 764+0+0 (unknown 3418376147 0 0) 0x7f5c8c1d71e0 con 0x7f5c8c06fd50 2024-06-03T09:06:43.596 INFO:teuthology.orchestra.run.smithi088.stderr:2024-06-03T09:06:43.594+0000 7f5c81ffb640 1 -- 172.21.15.88:0/1130531237 --> v1:172.21.15.88:6789/0 -- auth(proto 2 165 bytes epoch 0) v1 -- 0x7f5c8c1d6360 con 0x7f5c8c06fd50 2024-06-03T09:06:43.596 INFO:teuthology.orchestra.run.smithi088.stderr:2024-06-03T09:06:43.594+0000 7f5c81ffb640 1 -- 172.21.15.88:0/1130531237 <== mon.0 v1:172.21.15.88:6789/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 2234374065 0 0) 0x7f5c7c001850 con 0x7f5c8c06fd50 2024-06-03T09:06:43.597 INFO:teuthology.orchestra.run.smithi088.stderr:2024-06-03T09:06:43.594+0000 7f5c81ffb640 10 monclient(hunting): handle_monmap mon_map magic: 0 v1 2024-06-03T09:06:43.597 INFO:teuthology.orchestra.run.smithi088.stderr:2024-06-03T09:06:43.594+0000 7f5c81ffb640 10 monclient(hunting): got monmap 1 from mon.a (according to old e1) 2024-06-03T09:06:43.597 INFO:teuthology.orchestra.run.smithi088.stderr:2024-06-03T09:06:43.594+0000 7f5c81ffb640 10 monclient(hunting): dump: 2024-06-03T09:06:43.597 INFO:teuthology.orchestra.run.smithi088.stderr:epoch 1 2024-06-03T09:06:43.597 INFO:teuthology.orchestra.run.smithi088.stderr:fsid ec3c3fdd-3ac6-4d7d-9412-683130ed8b04 2024-06-03T09:06:43.597 INFO:teuthology.orchestra.run.smithi088.stderr:last_changed 2024-06-03T08:34:31.885125+0000 2024-06-03T09:06:43.597 INFO:teuthology.orchestra.run.smithi088.stderr:created 2024-06-03T08:34:31.885125+0000 2024-06-03T09:06:43.597 INFO:teuthology.orchestra.run.smithi088.stderr:min_mon_release 18 (reef) 2024-06-03T09:06:43.597 INFO:teuthology.orchestra.run.smithi088.stderr:election_strategy: 3 2024-06-03T09:06:43.597 INFO:teuthology.orchestra.run.smithi088.stderr:0: v1:172.21.15.88:6789/0 mon.a 2024-06-03T09:06:43.597 INFO:teuthology.orchestra.run.smithi088.stderr:1: v1:172.21.15.192:6789/0 mon.b 2024-06-03T09:06:43.597 INFO:teuthology.orchestra.run.smithi088.stderr:2: v1:172.21.15.88:6790/0 mon.c 2024-06-03T09:06:43.597 INFO:teuthology.orchestra.run.smithi088.stderr: 2024-06-03T09:06:43.597 INFO:teuthology.orchestra.run.smithi088.stderr:2024-06-03T09:06:43.594+0000 7f5c81ffb640 1 -- 172.21.15.88:0/1130531237 <== mon.2 v1:172.21.15.88:6790/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 2234374065 0 0) 0x7f5c840019d0 con 0x7f5c8c0ad840 2024-06-03T09:06:43.597 INFO:teuthology.orchestra.run.smithi088.stderr:2024-06-03T09:06:43.594+0000 7f5c81ffb640 10 monclient(hunting): handle_monmap mon_map magic: 0 v1 2024-06-03T09:06:43.598 INFO:teuthology.orchestra.run.smithi088.stderr:2024-06-03T09:06:43.594+0000 7f5c81ffb640 10 monclient(hunting): got monmap 1 from mon.c (according to old e1) 2024-06-03T09:06:43.598 INFO:teuthology.orchestra.run.smithi088.stderr:2024-06-03T09:06:43.594+0000 7f5c81ffb640 10 monclient(hunting): dump: 2024-06-03T09:06:43.598 INFO:teuthology.orchestra.run.smithi088.stderr:epoch 1 2024-06-03T09:06:43.598 INFO:teuthology.orchestra.run.smithi088.stderr:fsid ec3c3fdd-3ac6-4d7d-9412-683130ed8b04 2024-06-03T09:06:43.598 INFO:teuthology.orchestra.run.smithi088.stderr:last_changed 2024-06-03T08:34:31.885125+0000 2024-06-03T09:06:43.598 INFO:teuthology.orchestra.run.smithi088.stderr:created 2024-06-03T08:34:31.885125+0000 2024-06-03T09:06:43.598 INFO:teuthology.orchestra.run.smithi088.stderr:min_mon_release 18 (reef) 2024-06-03T09:06:43.598 INFO:teuthology.orchestra.run.smithi088.stderr:election_strategy: 3 2024-06-03T09:06:43.598 INFO:teuthology.orchestra.run.smithi088.stderr:0: v1:172.21.15.88:6789/0 mon.a 2024-06-03T09:06:43.598 INFO:teuthology.orchestra.run.smithi088.stderr:1: v1:172.21.15.192:6789/0 mon.b 2024-06-03T09:06:43.598 INFO:teuthology.orchestra.run.smithi088.stderr:2: v1:172.21.15.88:6790/0 mon.c 2024-06-03T09:06:43.598 INFO:teuthology.orchestra.run.smithi088.stderr: 2024-06-03T09:06:43.600 INFO:teuthology.orchestra.run.smithi088.stderr:2024-06-03T09:06:43.596+0000 7f5c81ffb640 1 -- 172.21.15.88:0/1130531237 <== mon.1 v1:172.21.15.192:6789/0 4 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 580+0+0 (unknown 4046477660 0 0) 0x7f5c8c1d65b0 con 0x7f5c8c06f970 2024-06-03T09:06:43.600 INFO:teuthology.orchestra.run.smithi088.stderr:2024-06-03T09:06:43.596+0000 7f5c81ffb640 1 -- 172.21.15.88:0/1130531237 >> v1:172.21.15.88:6790/0 conn(0x7f5c8c0ad840 legacy=0x7f5c8c1e1c20 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-06-03T09:06:43.600 INFO:teuthology.orchestra.run.smithi088.stderr:2024-06-03T09:06:43.596+0000 7f5c81ffb640 1 -- 172.21.15.88:0/1130531237 >> v1:172.21.15.88:6789/0 conn(0x7f5c8c06fd50 legacy=0x7f5c8c14a3a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-06-03T09:06:43.600 INFO:teuthology.orchestra.run.smithi088.stderr:2024-06-03T09:06:43.596+0000 7f5c81ffb640 10 monclient: _finish_hunting 0 2024-06-03T09:06:43.600 INFO:teuthology.orchestra.run.smithi088.stderr:2024-06-03T09:06:43.596+0000 7f5c81ffb640 1 monclient: found mon.b 2024-06-03T09:06:43.600 INFO:teuthology.orchestra.run.smithi088.stderr:2024-06-03T09:06:43.596+0000 7f5c81ffb640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-06-03T09:06:43.600 INFO:teuthology.orchestra.run.smithi088.stderr:2024-06-03T09:06:43.596+0000 7f5c81ffb640 10 monclient: _send_mon_message to mon.b at v1:172.21.15.192:6789/0 2024-06-03T09:06:43.600 INFO:teuthology.orchestra.run.smithi088.stderr:2024-06-03T09:06:43.596+0000 7f5c81ffb640 1 -- 172.21.15.88:0/1130531237 --> v1:172.21.15.192:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f5c8c0ab9e0 con 0x7f5c8c06f970 2024-06-03T09:06:43.600 INFO:teuthology.orchestra.run.smithi088.stderr:2024-06-03T09:06:43.596+0000 7f5c81ffb640 10 monclient: _finish_auth 0 2024-06-03T09:06:43.600 INFO:teuthology.orchestra.run.smithi088.stderr:2024-06-03T09:06:43.596+0000 7f5c81ffb640 10 monclient: _check_auth_tickets 2024-06-03T09:06:43.600 INFO:teuthology.orchestra.run.smithi088.stderr:2024-06-03T09:06:43.596+0000 7f5c81ffb640 20 monclient: _check_auth_rotating not needed by client.admin 2024-06-03T09:06:43.600 INFO:teuthology.orchestra.run.smithi088.stderr:2024-06-03T09:06:43.596+0000 7f5c93888640 5 monclient: authenticate success, global_id 30265 2024-06-03T09:06:43.601 INFO:teuthology.orchestra.run.smithi088.stderr:2024-06-03T09:06:43.596+0000 7f5c93888640 10 monclient: _renew_subs 2024-06-03T09:06:43.601 INFO:teuthology.orchestra.run.smithi088.stderr:2024-06-03T09:06:43.596+0000 7f5c93888640 10 monclient: _send_mon_message to mon.b at v1:172.21.15.192:6789/0 2024-06-03T09:06:43.601 INFO:teuthology.orchestra.run.smithi088.stderr:2024-06-03T09:06:43.596+0000 7f5c93888640 1 -- 172.21.15.88:0/1130531237 --> v1:172.21.15.192:6789/0 -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f5c8c14b9c0 con 0x7f5c8c06f970 2024-06-03T09:06:43.601 INFO:teuthology.orchestra.run.smithi088.stderr:2024-06-03T09:06:43.596+0000 7f5c93888640 10 client.30265.objecter _maybe_request_map subscribing (onetime) to next osd map 2024-06-03T09:06:43.601 INFO:teuthology.orchestra.run.smithi088.stderr:2024-06-03T09:06:43.596+0000 7f5c93888640 10 monclient: _renew_subs 2024-06-03T09:06:43.601 INFO:teuthology.orchestra.run.smithi088.stderr:2024-06-03T09:06:43.596+0000 7f5c93888640 10 monclient: _send_mon_message to mon.b at v1:172.21.15.192:6789/0 2024-06-03T09:06:43.601 INFO:teuthology.orchestra.run.smithi088.stderr:2024-06-03T09:06:43.596+0000 7f5c93888640 1 -- 172.21.15.88:0/1130531237 --> v1:172.21.15.192:6789/0 -- mon_subscribe({osdmap=0}) v3 -- 0x7f5c8c14b6c0 con 0x7f5c8c06f970 2024-06-03T09:06:43.601 INFO:teuthology.orchestra.run.smithi088.stderr:2024-06-03T09:06:43.596+0000 7f5c93888640 1 librados: init done 2024-06-03T09:06:43.601 INFO:teuthology.orchestra.run.smithi088.stderr:2024-06-03T09:06:43.597+0000 7f5c93888640 10 start_mon_command cmd=[{"prefix": "get_command_descriptions"}] 2024-06-03T09:06:43.601 INFO:teuthology.orchestra.run.smithi088.stderr:2024-06-03T09:06:43.597+0000 7f5c93888640 10 monclient: _send_command 1 [{"prefix": "get_command_descriptions"}] 2024-06-03T09:06:43.601 INFO:teuthology.orchestra.run.smithi088.stderr:2024-06-03T09:06:43.597+0000 7f5c93888640 10 monclient: _send_mon_message to mon.b at v1:172.21.15.192:6789/0 2024-06-03T09:06:43.601 INFO:teuthology.orchestra.run.smithi088.stderr:2024-06-03T09:06:43.597+0000 7f5c93888640 1 -- 172.21.15.88:0/1130531237 --> v1:172.21.15.192:6789/0 -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f5c8c1d6740 con 0x7f5c8c06f970 2024-06-03T09:06:43.845 INFO:teuthology.orchestra.run.smithi088.stderr:2024-06-03T09:06:43.845+0000 7f5c81ffb640 1 -- 172.21.15.88:0/1130531237 <== mon.1 v1:172.21.15.192:6789/0 5 ==== config(0 keys) v1 ==== 4+0+0 (unknown 0 0 0) 0x7f5c88004a90 con 0x7f5c8c06f970 2024-06-03T09:06:43.845 INFO:teuthology.orchestra.run.smithi088.stderr:2024-06-03T09:06:43.845+0000 7f5c81ffb640 10 monclient: handle_config config(0 keys) v1 2024-06-03T09:06:43.845 INFO:teuthology.orchestra.run.smithi088.stderr:2024-06-03T09:06:43.845+0000 7f5c81ffb640 1 -- 172.21.15.88:0/1130531237 <== mon.1 v1:172.21.15.192:6789/0 6 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 2234374065 0 0) 0x7f5c88004da0 con 0x7f5c8c06f970 2024-06-03T09:06:43.845 INFO:teuthology.orchestra.run.smithi088.stderr:2024-06-03T09:06:43.845+0000 7f5c81ffb640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-06-03T09:06:43.845 INFO:teuthology.orchestra.run.smithi088.stderr:2024-06-03T09:06:43.845+0000 7f5c81ffb640 10 monclient: got monmap 1 from mon.b (according to old e1) 2024-06-03T09:06:43.845 INFO:teuthology.orchestra.run.smithi088.stderr:2024-06-03T09:06:43.845+0000 7f5c81ffb640 10 monclient: dump: 2024-06-03T09:06:43.845 INFO:teuthology.orchestra.run.smithi088.stderr:epoch 1 2024-06-03T09:06:43.846 INFO:teuthology.orchestra.run.smithi088.stderr:fsid ec3c3fdd-3ac6-4d7d-9412-683130ed8b04 2024-06-03T09:06:43.846 INFO:teuthology.orchestra.run.smithi088.stderr:last_changed 2024-06-03T08:34:31.885125+0000 2024-06-03T09:06:43.846 INFO:teuthology.orchestra.run.smithi088.stderr:created 2024-06-03T08:34:31.885125+0000 2024-06-03T09:06:43.846 INFO:teuthology.orchestra.run.smithi088.stderr:min_mon_release 18 (reef) 2024-06-03T09:06:43.846 INFO:teuthology.orchestra.run.smithi088.stderr:election_strategy: 3 2024-06-03T09:06:43.846 INFO:teuthology.orchestra.run.smithi088.stderr:0: v1:172.21.15.88:6789/0 mon.a 2024-06-03T09:06:43.846 INFO:teuthology.orchestra.run.smithi088.stderr:1: v1:172.21.15.192:6789/0 mon.b 2024-06-03T09:06:43.846 INFO:teuthology.orchestra.run.smithi088.stderr:2: v1:172.21.15.88:6790/0 mon.c 2024-06-03T09:06:43.846 INFO:teuthology.orchestra.run.smithi088.stderr: 2024-06-03T09:06:43.846 INFO:teuthology.orchestra.run.smithi088.stderr:2024-06-03T09:06:43.845+0000 7f5c837fe640 4 set_mon_vals no callback set 2024-06-03T09:06:43.847 INFO:teuthology.orchestra.run.smithi088.stderr:2024-06-03T09:06:43.845+0000 7f5c81ffb640 1 -- 172.21.15.88:0/1130531237 <== mon.1 v1:172.21.15.192:6789/0 7 ==== mgrmap(e 5) v1 ==== 89110+0+0 (unknown 873533973 0 0) 0x7f5c8801ab20 con 0x7f5c8c06f970 2024-06-03T09:06:43.847 INFO:teuthology.orchestra.run.smithi088.stderr:2024-06-03T09:06:43.846+0000 7f5c90e23640 10 monclient: get_auth_request con 0x7f5c7006d400 auth_method 0 2024-06-03T09:06:43.847 INFO:teuthology.orchestra.run.smithi088.stderr:2024-06-03T09:06:43.846+0000 7f5c81ffb640 10 client.30265.objecter ms_handle_connect 0x7f5c7006d400 2024-06-03T09:06:43.848 INFO:teuthology.orchestra.run.smithi088.stderr:2024-06-03T09:06:43.847+0000 7f5c81ffb640 1 -- 172.21.15.88:0/1130531237 <== mon.1 v1:172.21.15.192:6789/0 8 ==== osd_map(1012..1012 src has 622..1012) v4 ==== 5229+0+0 (unknown 2151252637 0 0) 0x7f5c88054150 con 0x7f5c8c06f970 2024-06-03T09:06:43.848 INFO:teuthology.orchestra.run.smithi088.stderr:2024-06-03T09:06:43.847+0000 7f5c81ffb640 10 client.30265.objecter ms_dispatch 0x7f5c8c000b90 osd_map(1012..1012 src has 622..1012) v4 2024-06-03T09:06:43.848 INFO:teuthology.orchestra.run.smithi088.stderr:2024-06-03T09:06:43.847+0000 7f5c81ffb640 3 client.30265.objecter handle_osd_map got epochs [1012,1012] > 0 2024-06-03T09:06:43.848 INFO:teuthology.orchestra.run.smithi088.stderr:2024-06-03T09:06:43.847+0000 7f5c81ffb640 3 client.30265.objecter handle_osd_map decoding full epoch 1012 2024-06-03T09:06:43.848 INFO:teuthology.orchestra.run.smithi088.stderr:2024-06-03T09:06:43.848+0000 7f5c81ffb640 20 client.30265.objecter dump_active .. 0 homeless 2024-06-03T09:06:43.849 INFO:teuthology.orchestra.run.smithi088.stderr:2024-06-03T09:06:43.849+0000 7f5c81ffb640 1 -- 172.21.15.88:0/1130531237 <== mon.1 v1:172.21.15.192:6789/0 9 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+182498 (unknown 1092875540 0 3016760700) 0x7f5c8804fb30 con 0x7f5c8c06f970 2024-06-03T09:06:43.849 INFO:teuthology.orchestra.run.smithi088.stderr:2024-06-03T09:06:43.849+0000 7f5c81ffb640 10 monclient: handle_mon_command_ack 1 [{"prefix": "get_command_descriptions"}] 2024-06-03T09:06:43.849 INFO:teuthology.orchestra.run.smithi088.stderr:2024-06-03T09:06:43.849+0000 7f5c81ffb640 10 monclient: _finish_command 1 = system:0 2024-06-03T09:06:43.949 INFO:tasks.ceph.osd.6.smithi192.stderr:2024-06-03T09:06:43.948+0000 1426f640 -1 osd.6 1011 set_numa_affinity unable to identify public interface '' numa node: (2) No such file or directory 2024-06-03T09:06:44.103 INFO:teuthology.orchestra.run.smithi088.stderr:2024-06-03T09:06:44.103+0000 7f5c93888640 10 start_mon_command cmd=[{"prefix": "status", "format": "json"}] 2024-06-03T09:06:44.104 INFO:teuthology.orchestra.run.smithi088.stderr:2024-06-03T09:06:44.103+0000 7f5c93888640 10 monclient: _send_command 2 [{"prefix": "status", "format": "json"}] 2024-06-03T09:06:44.104 INFO:teuthology.orchestra.run.smithi088.stderr:2024-06-03T09:06:44.103+0000 7f5c93888640 10 monclient: _send_mon_message to mon.b at v1:172.21.15.192:6789/0 2024-06-03T09:06:44.104 INFO:teuthology.orchestra.run.smithi088.stderr:2024-06-03T09:06:44.103+0000 7f5c93888640 1 -- 172.21.15.88:0/1130531237 --> v1:172.21.15.192:6789/0 -- mon_command({"prefix": "status", "format": "json"} v 0) v1 -- 0x7f5c8c04ea60 con 0x7f5c8c06f970 2024-06-03T09:06:44.110 INFO:teuthology.orchestra.run.smithi088.stderr:2024-06-03T09:06:44.110+0000 7f5c81ffb640 1 -- 172.21.15.88:0/1130531237 <== mon.1 v1:172.21.15.192:6789/0 10 ==== mon_command_ack([{"prefix": "status", "format": "json"}]=0 v0) v1 ==== 72+0+2108 (unknown 3226119948 0 3698714486) 0x7f5c88051c30 con 0x7f5c8c06f970 2024-06-03T09:06:44.110 INFO:teuthology.orchestra.run.smithi088.stderr:2024-06-03T09:06:44.110+0000 7f5c81ffb640 10 monclient: handle_mon_command_ack 2 [{"prefix": "status", "format": "json"}] 2024-06-03T09:06:44.110 INFO:teuthology.orchestra.run.smithi088.stderr:2024-06-03T09:06:44.110+0000 7f5c81ffb640 10 monclient: _finish_command 2 = system:0 2024-06-03T09:06:44.110 INFO:teuthology.orchestra.run.smithi088.stdout: 2024-06-03T09:06:44.110 INFO:teuthology.orchestra.run.smithi088.stdout:{"fsid":"ec3c3fdd-3ac6-4d7d-9412-683130ed8b04","health":{"status":"HEALTH_WARN","checks":{"OSD_DOWN":{"severity":"HEALTH_WARN","summary":{"message":"1 osds down","count":1},"muted":false},"PG_DEGRADED":{"severity":"HEALTH_WARN","summary":{"message":"Degraded data redundancy: 1/6 objects degraded (16.667%), 1 pg degraded","count":1},"muted":false},"POOL_APP_NOT_ENABLED":{"severity":"HEALTH_WARN","summary":{"message":"1 pool(s) do not have an application enabled","count":1},"muted":false}},"mutes":[]},"election_epoch":10,"quorum":[0,1,2],"quorum_names":["a","b","c"],"quorum_age":1877,"monmap":{"epoch":1,"min_mon_release_name":"reef","num_mons":3},"osdmap":{"epoch":1012,"num_osds":8,"num_up_osds":7,"osd_up_since":1717405562,"num_in_osds":8,"osd_in_since":1717403732,"num_remapped_pgs":0},"pgmap":{"pgs_by_state":[{"state_name":"active+clean","count":27},{"state_name":"active+undersized","count":13},{"state_name":"active+undersized+degraded","count":1}],"num_pgs":41,"num_pools":3,"num_objects":3,"data_bytes":459285,"bytes_used":506482688,"bytes_avail":772587630592,"bytes_total":773094113280,"degraded_objects":1,"degraded_total":6,"degraded_ratio":0.16666666666666663},"fsmap":{"epoch":1,"by_rank":[],"up:standby":0},"mgrmap":{"available":true,"num_standbys":1,"modules":["iostat","nfs","restful"],"services":{}},"servicemap":{"epoch":25,"modified":"2024-06-03T09:06:37.101852+0000","services":{"osd":{"daemons":{"summary":"","0":{"start_epoch":0,"start_stamp":"0.000000","gid":0,"addr":"(unrecognized address family 0)/0","metadata":{},"task_status":{}},"1":{"start_epoch":0,"start_stamp":"0.000000","gid":0,"addr":"(unrecognized address family 0)/0","metadata":{},"task_status":{}},"3":{"start_epoch":0,"start_stamp":"0.000000","gid":0,"addr":"(unrecognized address family 0)/0","metadata":{},"task_status":{}},"4":{"start_epoch":0,"start_stamp":"0.000000","gid":0,"addr":"(unrecognized address family 0)/0","metadata":{},"task_status":{}},"5":{"start_epoch":0,"start_stamp":"0.000000","gid":0,"addr":"(unrecognized address family 0)/0","metadata":{},"task_status":{}}}}}},"progress_events":{}} 2024-06-03T09:06:44.112 INFO:teuthology.orchestra.run.smithi088.stderr:2024-06-03T09:06:44.112+0000 7f5c93888640 10 librados: watch_flush enter 2024-06-03T09:06:44.112 INFO:teuthology.orchestra.run.smithi088.stderr:2024-06-03T09:06:44.112+0000 7f5c93888640 10 librados: watch_flush exit 2024-06-03T09:06:44.112 INFO:teuthology.orchestra.run.smithi088.stderr:2024-06-03T09:06:44.112+0000 7f5c93888640 20 client.30265.objecter shutdown clearing up homeless session... 2024-06-03T09:06:44.112 INFO:teuthology.orchestra.run.smithi088.stderr:2024-06-03T09:06:44.112+0000 7f5c93888640 10 client.30265.objecter successfully canceled tick 2024-06-03T09:06:44.112 INFO:teuthology.orchestra.run.smithi088.stderr:2024-06-03T09:06:44.112+0000 7f5c93888640 1 -- 172.21.15.88:0/1130531237 >> v1:172.21.15.192:6800/34229 conn(0x7f5c7006d400 legacy=0x7f5c7006f8c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-06-03T09:06:44.113 INFO:teuthology.orchestra.run.smithi088.stderr:2024-06-03T09:06:44.112+0000 7f5c93888640 10 monclient: shutdown 2024-06-03T09:06:44.113 INFO:teuthology.orchestra.run.smithi088.stderr:2024-06-03T09:06:44.112+0000 7f5c93888640 20 monclient: shutdown discarding 0 pending message(s) 2024-06-03T09:06:44.113 INFO:teuthology.orchestra.run.smithi088.stderr:2024-06-03T09:06:44.112+0000 7f5c93888640 1 -- 172.21.15.88:0/1130531237 >> v1:172.21.15.192:6789/0 conn(0x7f5c8c06f970 legacy=0x7f5c8c1d6ad0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-06-03T09:06:44.113 INFO:teuthology.orchestra.run.smithi088.stderr:2024-06-03T09:06:44.112+0000 7f5c93888640 1 -- 172.21.15.88:0/1130531237 shutdown_connections 2024-06-03T09:06:44.113 INFO:teuthology.orchestra.run.smithi088.stderr:2024-06-03T09:06:44.112+0000 7f5c93888640 1 -- 172.21.15.88:0/1130531237 >> 172.21.15.88:0/1130531237 conn(0x7f5c8c0b2e40 msgr2=0x7f5c8c0737f0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-06-03T09:06:44.113 INFO:teuthology.orchestra.run.smithi088.stderr:2024-06-03T09:06:44.112+0000 7f5c93888640 1 -- 172.21.15.88:0/1130531237 shutdown_connections 2024-06-03T09:06:44.113 INFO:teuthology.orchestra.run.smithi088.stderr:2024-06-03T09:06:44.112+0000 7f5c93888640 1 -- 172.21.15.88:0/1130531237 wait complete. 2024-06-03T09:06:44.113 INFO:teuthology.orchestra.run.smithi088.stderr:2024-06-03T09:06:44.112+0000 7f5c93888640 1 librados: shutdown 2024-06-03T09:06:44.123 INFO:tasks.ceph.ceph_manager.ceph:no progress seen, keeping timeout for now 2024-06-03T09:06:44.123 DEBUG:teuthology.orchestra.run.smithi088:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph pg dump --format=json 2024-06-03T09:06:44.221 INFO:teuthology.orchestra.run.smithi088.stderr:2024-06-03T09:06:44.221+0000 7f41610ee640 10 monclient: get_monmap_and_config 2024-06-03T09:06:44.222 INFO:teuthology.orchestra.run.smithi088.stderr:2024-06-03T09:06:44.221+0000 7f41610ee640 10 monclient: build_initial_monmap 2024-06-03T09:06:44.222 INFO:teuthology.orchestra.run.smithi088.stderr:2024-06-03T09:06:44.221+0000 7f41610ee640 10 monclient: monmap: 2024-06-03T09:06:44.222 INFO:teuthology.orchestra.run.smithi088.stderr:epoch 0 2024-06-03T09:06:44.222 INFO:teuthology.orchestra.run.smithi088.stderr:fsid ec3c3fdd-3ac6-4d7d-9412-683130ed8b04 2024-06-03T09:06:44.222 INFO:teuthology.orchestra.run.smithi088.stderr:last_changed 2024-06-03T09:06:44.222799+0000 2024-06-03T09:06:44.222 INFO:teuthology.orchestra.run.smithi088.stderr:created 2024-06-03T09:06:44.222799+0000 2024-06-03T09:06:44.222 INFO:teuthology.orchestra.run.smithi088.stderr:min_mon_release 0 (unknown) 2024-06-03T09:06:44.222 INFO:teuthology.orchestra.run.smithi088.stderr:election_strategy: 3 2024-06-03T09:06:44.222 INFO:teuthology.orchestra.run.smithi088.stderr:0: v1:172.21.15.88:6789/0 mon.noname-a 2024-06-03T09:06:44.222 INFO:teuthology.orchestra.run.smithi088.stderr:1: v1:172.21.15.192:6789/0 mon.noname-c 2024-06-03T09:06:44.222 INFO:teuthology.orchestra.run.smithi088.stderr:2: v1:172.21.15.88:6790/0 mon.noname-b 2024-06-03T09:06:44.222 INFO:teuthology.orchestra.run.smithi088.stderr: 2024-06-03T09:06:44.224 INFO:teuthology.orchestra.run.smithi088.stderr:2024-06-03T09:06:44.222+0000 7f41610ee640 1 Processor -- start 2024-06-03T09:06:44.224 INFO:teuthology.orchestra.run.smithi088.stderr:2024-06-03T09:06:44.222+0000 7f41610ee640 1 -- start start 2024-06-03T09:06:44.224 INFO:teuthology.orchestra.run.smithi088.stderr:2024-06-03T09:06:44.222+0000 7f41610ee640 10 monclient: init 2024-06-03T09:06:44.224 INFO:teuthology.orchestra.run.smithi088.stderr:2024-06-03T09:06:44.222+0000 7f41610ee640 10 monclient: _reopen_session rank -1 2024-06-03T09:06:44.224 INFO:teuthology.orchestra.run.smithi088.stderr:2024-06-03T09:06:44.222+0000 7f41610ee640 10 monclient: _add_conns ranks=[2,0,1] 2024-06-03T09:06:44.224 INFO:teuthology.orchestra.run.smithi088.stderr:2024-06-03T09:06:44.222+0000 7f41610ee640 10 monclient(hunting): picked mon.noname-b con 0x7f415c06fc70 addr v1:172.21.15.88:6790/0 2024-06-03T09:06:44.224 INFO:teuthology.orchestra.run.smithi088.stderr:2024-06-03T09:06:44.222+0000 7f41610ee640 10 monclient(hunting): picked mon.noname-a con 0x7f415c0ad920 addr v1:172.21.15.88:6789/0 2024-06-03T09:06:44.224 INFO:teuthology.orchestra.run.smithi088.stderr:2024-06-03T09:06:44.222+0000 7f41610ee640 10 monclient(hunting): picked mon.noname-c con 0x7f415c1393f0 addr v1:172.21.15.192:6789/0 2024-06-03T09:06:44.224 INFO:teuthology.orchestra.run.smithi088.stderr:2024-06-03T09:06:44.222+0000 7f41610ee640 1 -- --> v1:172.21.15.88:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f415c070420 con 0x7f415c0ad920 2024-06-03T09:06:44.225 INFO:teuthology.orchestra.run.smithi088.stderr:2024-06-03T09:06:44.222+0000 7f41610ee640 1 -- --> v1:172.21.15.192:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f415c071050 con 0x7f415c1393f0 2024-06-03T09:06:44.225 INFO:teuthology.orchestra.run.smithi088.stderr:2024-06-03T09:06:44.223+0000 7f41610ee640 1 -- --> v1:172.21.15.88:6790/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f415c0add60 con 0x7f415c06fc70 2024-06-03T09:06:44.225 INFO:teuthology.orchestra.run.smithi088.stderr:2024-06-03T09:06:44.223+0000 7f41610ee640 10 monclient(hunting): _renew_subs 2024-06-03T09:06:44.225 INFO:teuthology.orchestra.run.smithi088.stderr:2024-06-03T09:06:44.223+0000 7f41610ee640 10 monclient(hunting): authenticate will time out at 3060.032958s 2024-06-03T09:06:44.225 INFO:teuthology.orchestra.run.smithi088.stderr:2024-06-03T09:06:44.223+0000 7f415ad9d640 1 --1- >> v1:172.21.15.88:6790/0 conn(0x7f415c06fc70 0x7f415c13fbb0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.88:6790/0 says I am v1:172.21.15.88:41334/0 (socket says 172.21.15.88:41334) 2024-06-03T09:06:44.225 INFO:teuthology.orchestra.run.smithi088.stderr:2024-06-03T09:06:44.223+0000 7f415ad9d640 1 -- 172.21.15.88:0/2254523300 learned_addr learned my addr 172.21.15.88:0/2254523300 (peer_addr_for_me v1:172.21.15.88:0/0) 2024-06-03T09:06:44.226 INFO:teuthology.orchestra.run.smithi088.stderr:2024-06-03T09:06:44.225+0000 7f4159d9b640 1 -- 172.21.15.88:0/2254523300 <== mon.2 v1:172.21.15.88:6790/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 1401217215 0 0) 0x7f415c0add60 con 0x7f415c06fc70 2024-06-03T09:06:44.226 INFO:teuthology.orchestra.run.smithi088.stderr:2024-06-03T09:06:44.225+0000 7f4159d9b640 10 monclient(hunting): _negotiate 2024-06-03T09:06:44.226 INFO:teuthology.orchestra.run.smithi088.stderr:2024-06-03T09:06:44.225+0000 7f4159d9b640 10 monclient(hunting): _init_auth method 2 2024-06-03T09:06:44.227 INFO:teuthology.orchestra.run.smithi088.stderr:2024-06-03T09:06:44.225+0000 7f4159d9b640 10 monclient(hunting): _init_auth creating new auth 2024-06-03T09:06:44.227 INFO:teuthology.orchestra.run.smithi088.stderr:2024-06-03T09:06:44.225+0000 7f4159d9b640 10 monclient(hunting): my global_id is 30194 2024-06-03T09:06:44.227 INFO:teuthology.orchestra.run.smithi088.stderr:2024-06-03T09:06:44.225+0000 7f4159d9b640 1 -- 172.21.15.88:0/2254523300 --> v1:172.21.15.88:6790/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f4144003520 con 0x7f415c06fc70 2024-06-03T09:06:44.227 INFO:teuthology.orchestra.run.smithi088.stderr:2024-06-03T09:06:44.225+0000 7f4159d9b640 1 -- 172.21.15.88:0/2254523300 <== mon.0 v1:172.21.15.88:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 1228046087 0 0) 0x7f415c070420 con 0x7f415c0ad920 2024-06-03T09:06:44.227 INFO:teuthology.orchestra.run.smithi088.stderr:2024-06-03T09:06:44.225+0000 7f4159d9b640 10 monclient(hunting): _negotiate 2024-06-03T09:06:44.227 INFO:teuthology.orchestra.run.smithi088.stderr:2024-06-03T09:06:44.225+0000 7f4159d9b640 10 monclient(hunting): _init_auth method 2 2024-06-03T09:06:44.227 INFO:teuthology.orchestra.run.smithi088.stderr:2024-06-03T09:06:44.225+0000 7f4159d9b640 10 monclient(hunting): _init_auth creating new auth 2024-06-03T09:06:44.227 INFO:teuthology.orchestra.run.smithi088.stderr:2024-06-03T09:06:44.225+0000 7f4159d9b640 10 monclient(hunting): my global_id is 30690 2024-06-03T09:06:44.227 INFO:teuthology.orchestra.run.smithi088.stderr:2024-06-03T09:06:44.225+0000 7f4159d9b640 1 -- 172.21.15.88:0/2254523300 --> v1:172.21.15.88:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f415c0add60 con 0x7f415c0ad920 2024-06-03T09:06:44.227 INFO:teuthology.orchestra.run.smithi088.stderr:2024-06-03T09:06:44.226+0000 7f4159d9b640 1 -- 172.21.15.88:0/2254523300 <== mon.1 v1:172.21.15.192:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 2566470333 0 0) 0x7f415c071050 con 0x7f415c1393f0 2024-06-03T09:06:44.227 INFO:teuthology.orchestra.run.smithi088.stderr:2024-06-03T09:06:44.226+0000 7f4159d9b640 10 monclient(hunting): _negotiate 2024-06-03T09:06:44.227 INFO:teuthology.orchestra.run.smithi088.stderr:2024-06-03T09:06:44.226+0000 7f4159d9b640 10 monclient(hunting): _init_auth method 2 2024-06-03T09:06:44.227 INFO:teuthology.orchestra.run.smithi088.stderr:2024-06-03T09:06:44.226+0000 7f4159d9b640 10 monclient(hunting): _init_auth creating new auth 2024-06-03T09:06:44.227 INFO:teuthology.orchestra.run.smithi088.stderr:2024-06-03T09:06:44.226+0000 7f4159d9b640 10 monclient(hunting): my global_id is 30268 2024-06-03T09:06:44.228 INFO:teuthology.orchestra.run.smithi088.stderr:2024-06-03T09:06:44.226+0000 7f4159d9b640 1 -- 172.21.15.88:0/2254523300 --> v1:172.21.15.192:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f415c070420 con 0x7f415c1393f0 2024-06-03T09:06:44.229 INFO:teuthology.orchestra.run.smithi088.stderr:2024-06-03T09:06:44.228+0000 7f4159d9b640 1 -- 172.21.15.88:0/2254523300 <== mon.2 v1:172.21.15.88:6790/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 214+0+0 (unknown 2242321922 0 0) 0x7f4144003520 con 0x7f415c06fc70 2024-06-03T09:06:44.229 INFO:teuthology.orchestra.run.smithi088.stderr:2024-06-03T09:06:44.229+0000 7f4159d9b640 1 -- 172.21.15.88:0/2254523300 >> v1:172.21.15.192:6789/0 conn(0x7f415c1393f0 legacy=0x7f415c14ac50 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-06-03T09:06:44.229 INFO:teuthology.orchestra.run.smithi088.stderr:2024-06-03T09:06:44.229+0000 7f4159d9b640 1 -- 172.21.15.88:0/2254523300 >> v1:172.21.15.88:6789/0 conn(0x7f415c0ad920 legacy=0x7f415c138ce0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-06-03T09:06:44.229 INFO:teuthology.orchestra.run.smithi088.stderr:2024-06-03T09:06:44.229+0000 7f4159d9b640 10 monclient: _finish_hunting 0 2024-06-03T09:06:44.229 INFO:teuthology.orchestra.run.smithi088.stderr:2024-06-03T09:06:44.229+0000 7f4159d9b640 1 monclient: found mon.noname-b 2024-06-03T09:06:44.229 INFO:teuthology.orchestra.run.smithi088.stderr:2024-06-03T09:06:44.229+0000 7f4159d9b640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-06-03T09:06:44.229 INFO:teuthology.orchestra.run.smithi088.stderr:2024-06-03T09:06:44.229+0000 7f4159d9b640 10 monclient: _send_mon_message to mon.noname-b at v1:172.21.15.88:6790/0 2024-06-03T09:06:44.230 INFO:teuthology.orchestra.run.smithi088.stderr:2024-06-03T09:06:44.229+0000 7f4159d9b640 1 -- 172.21.15.88:0/2254523300 --> v1:172.21.15.88:6790/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f415c071be0 con 0x7f415c06fc70 2024-06-03T09:06:44.230 INFO:teuthology.orchestra.run.smithi088.stderr:2024-06-03T09:06:44.229+0000 7f4159d9b640 10 monclient: _finish_auth 0 2024-06-03T09:06:44.230 INFO:teuthology.orchestra.run.smithi088.stderr:2024-06-03T09:06:44.229+0000 7f4159d9b640 10 monclient: _check_auth_tickets 2024-06-03T09:06:44.230 INFO:teuthology.orchestra.run.smithi088.stderr:2024-06-03T09:06:44.229+0000 7f4159d9b640 20 monclient: _check_auth_rotating not needed by client.admin 2024-06-03T09:06:44.230 INFO:teuthology.orchestra.run.smithi088.stderr:2024-06-03T09:06:44.229+0000 7f41610ee640 5 monclient: authenticate success, global_id 30194 2024-06-03T09:06:44.230 INFO:teuthology.orchestra.run.smithi088.stderr:2024-06-03T09:06:44.229+0000 7f41610ee640 20 monclient: get_monmap_and_config waiting for monmap|config 2024-06-03T09:06:44.230 INFO:teuthology.orchestra.run.smithi088.stderr:2024-06-03T09:06:44.229+0000 7f4159d9b640 1 -- 172.21.15.88:0/2254523300 <== mon.2 v1:172.21.15.88:6790/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 2234374065 0 0) 0x7f414c001610 con 0x7f415c06fc70 2024-06-03T09:06:44.230 INFO:teuthology.orchestra.run.smithi088.stderr:2024-06-03T09:06:44.229+0000 7f4159d9b640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-06-03T09:06:44.230 INFO:teuthology.orchestra.run.smithi088.stderr:2024-06-03T09:06:44.229+0000 7f4159d9b640 10 monclient: got monmap 1 from mon.noname-b (according to old e1) 2024-06-03T09:06:44.230 INFO:teuthology.orchestra.run.smithi088.stderr:2024-06-03T09:06:44.229+0000 7f4159d9b640 10 monclient: dump: 2024-06-03T09:06:44.230 INFO:teuthology.orchestra.run.smithi088.stderr:epoch 1 2024-06-03T09:06:44.230 INFO:teuthology.orchestra.run.smithi088.stderr:fsid ec3c3fdd-3ac6-4d7d-9412-683130ed8b04 2024-06-03T09:06:44.231 INFO:teuthology.orchestra.run.smithi088.stderr:last_changed 2024-06-03T08:34:31.885125+0000 2024-06-03T09:06:44.231 INFO:teuthology.orchestra.run.smithi088.stderr:created 2024-06-03T08:34:31.885125+0000 2024-06-03T09:06:44.231 INFO:teuthology.orchestra.run.smithi088.stderr:min_mon_release 18 (reef) 2024-06-03T09:06:44.231 INFO:teuthology.orchestra.run.smithi088.stderr:election_strategy: 3 2024-06-03T09:06:44.231 INFO:teuthology.orchestra.run.smithi088.stderr:0: v1:172.21.15.88:6789/0 mon.a 2024-06-03T09:06:44.231 INFO:teuthology.orchestra.run.smithi088.stderr:1: v1:172.21.15.192:6789/0 mon.b 2024-06-03T09:06:44.231 INFO:teuthology.orchestra.run.smithi088.stderr:2: v1:172.21.15.88:6790/0 mon.c 2024-06-03T09:06:44.231 INFO:teuthology.orchestra.run.smithi088.stderr: 2024-06-03T09:06:44.231 INFO:teuthology.orchestra.run.smithi088.stderr:2024-06-03T09:06:44.229+0000 7f41610ee640 20 monclient: get_monmap_and_config waiting for monmap|config 2024-06-03T09:06:44.233 INFO:teuthology.orchestra.run.smithi088.stderr:2024-06-03T09:06:44.231+0000 7f4159d9b640 1 -- 172.21.15.88:0/2254523300 <== mon.2 v1:172.21.15.88:6790/0 4 ==== config(0 keys) v1 ==== 4+0+0 (unknown 0 0 0) 0x7f414c0017f0 con 0x7f415c06fc70 2024-06-03T09:06:44.233 INFO:teuthology.orchestra.run.smithi088.stderr:2024-06-03T09:06:44.231+0000 7f4159d9b640 10 monclient: handle_config config(0 keys) v1 2024-06-03T09:06:44.233 INFO:teuthology.orchestra.run.smithi088.stderr:2024-06-03T09:06:44.231+0000 7f4159d9b640 1 -- 172.21.15.88:0/2254523300 <== mon.2 v1:172.21.15.88:6790/0 5 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 2234374065 0 0) 0x7f414c001b80 con 0x7f415c06fc70 2024-06-03T09:06:44.233 INFO:teuthology.orchestra.run.smithi088.stderr:2024-06-03T09:06:44.231+0000 7f4159d9b640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-06-03T09:06:44.233 INFO:teuthology.orchestra.run.smithi088.stderr:2024-06-03T09:06:44.231+0000 7f4159d9b640 10 monclient: got monmap 1 from mon.c (according to old e1) 2024-06-03T09:06:44.233 INFO:teuthology.orchestra.run.smithi088.stderr:2024-06-03T09:06:44.231+0000 7f4159d9b640 10 monclient: dump: 2024-06-03T09:06:44.233 INFO:teuthology.orchestra.run.smithi088.stderr:epoch 1 2024-06-03T09:06:44.233 INFO:teuthology.orchestra.run.smithi088.stderr:fsid ec3c3fdd-3ac6-4d7d-9412-683130ed8b04 2024-06-03T09:06:44.233 INFO:teuthology.orchestra.run.smithi088.stderr:last_changed 2024-06-03T08:34:31.885125+0000 2024-06-03T09:06:44.234 INFO:teuthology.orchestra.run.smithi088.stderr:created 2024-06-03T08:34:31.885125+0000 2024-06-03T09:06:44.234 INFO:teuthology.orchestra.run.smithi088.stderr:min_mon_release 18 (reef) 2024-06-03T09:06:44.234 INFO:teuthology.orchestra.run.smithi088.stderr:election_strategy: 3 2024-06-03T09:06:44.234 INFO:teuthology.orchestra.run.smithi088.stderr:0: v1:172.21.15.88:6789/0 mon.a 2024-06-03T09:06:44.234 INFO:teuthology.orchestra.run.smithi088.stderr:1: v1:172.21.15.192:6789/0 mon.b 2024-06-03T09:06:44.234 INFO:teuthology.orchestra.run.smithi088.stderr:2: v1:172.21.15.88:6790/0 mon.c 2024-06-03T09:06:44.234 INFO:teuthology.orchestra.run.smithi088.stderr: 2024-06-03T09:06:44.234 INFO:teuthology.orchestra.run.smithi088.stderr:2024-06-03T09:06:44.231+0000 7f41610ee640 10 monclient: get_monmap_and_config success 2024-06-03T09:06:44.234 INFO:teuthology.orchestra.run.smithi088.stderr:2024-06-03T09:06:44.231+0000 7f41610ee640 4 set_mon_vals no callback set 2024-06-03T09:06:44.234 INFO:teuthology.orchestra.run.smithi088.stderr:2024-06-03T09:06:44.231+0000 7f41610ee640 10 monclient: shutdown 2024-06-03T09:06:44.234 INFO:teuthology.orchestra.run.smithi088.stderr:2024-06-03T09:06:44.231+0000 7f41610ee640 20 monclient: shutdown discarding 0 pending message(s) 2024-06-03T09:06:44.234 INFO:teuthology.orchestra.run.smithi088.stderr:2024-06-03T09:06:44.231+0000 7f41610ee640 1 -- 172.21.15.88:0/2254523300 >> v1:172.21.15.88:6790/0 conn(0x7f415c06fc70 legacy=0x7f415c13fbb0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-06-03T09:06:44.234 INFO:teuthology.orchestra.run.smithi088.stderr:2024-06-03T09:06:44.231+0000 7f41610ee640 1 -- 172.21.15.88:0/2254523300 shutdown_connections 2024-06-03T09:06:44.234 INFO:teuthology.orchestra.run.smithi088.stderr:2024-06-03T09:06:44.231+0000 7f41610ee640 1 -- 172.21.15.88:0/2254523300 >> 172.21.15.88:0/2254523300 conn(0x7f415c0b2f20 msgr2=0x7f415c0b3300 unknown :-1 s=STATE_NONE l=0).mark_down 2024-06-03T09:06:44.235 INFO:teuthology.orchestra.run.smithi088.stderr:2024-06-03T09:06:44.231+0000 7f41610ee640 1 -- 172.21.15.88:0/2254523300 shutdown_connections 2024-06-03T09:06:44.235 INFO:teuthology.orchestra.run.smithi088.stderr:2024-06-03T09:06:44.231+0000 7f41610ee640 1 -- 172.21.15.88:0/2254523300 wait complete. 2024-06-03T09:06:44.235 INFO:teuthology.orchestra.run.smithi088.stderr:2024-06-03T09:06:44.232+0000 7f41610ee640 10 monclient: build_initial_monmap 2024-06-03T09:06:44.235 INFO:teuthology.orchestra.run.smithi088.stderr:2024-06-03T09:06:44.232+0000 7f41610ee640 10 monclient: monmap: 2024-06-03T09:06:44.235 INFO:teuthology.orchestra.run.smithi088.stderr:epoch 0 2024-06-03T09:06:44.235 INFO:teuthology.orchestra.run.smithi088.stderr:fsid 00000000-0000-0000-0000-000000000000 2024-06-03T09:06:44.235 INFO:teuthology.orchestra.run.smithi088.stderr:last_changed 0.000000 2024-06-03T09:06:44.235 INFO:teuthology.orchestra.run.smithi088.stderr:created 0.000000 2024-06-03T09:06:44.235 INFO:teuthology.orchestra.run.smithi088.stderr:min_mon_release 0 (unknown) 2024-06-03T09:06:44.235 INFO:teuthology.orchestra.run.smithi088.stderr:election_strategy: 1 2024-06-03T09:06:44.235 INFO:teuthology.orchestra.run.smithi088.stderr:0: v1:172.21.15.88:6789/0 mon.noname-a 2024-06-03T09:06:44.235 INFO:teuthology.orchestra.run.smithi088.stderr:1: v1:172.21.15.192:6789/0 mon.noname-b 2024-06-03T09:06:44.235 INFO:teuthology.orchestra.run.smithi088.stderr:2: v1:172.21.15.88:6790/0 mon.noname-c 2024-06-03T09:06:44.236 INFO:teuthology.orchestra.run.smithi088.stderr: 2024-06-03T09:06:44.236 INFO:teuthology.orchestra.run.smithi088.stderr:2024-06-03T09:06:44.233+0000 7f41610ee640 1 librados: starting msgr at 2024-06-03T09:06:44.236 INFO:teuthology.orchestra.run.smithi088.stderr:2024-06-03T09:06:44.233+0000 7f41610ee640 1 librados: starting objecter 2024-06-03T09:06:44.236 INFO:teuthology.orchestra.run.smithi088.stderr:2024-06-03T09:06:44.233+0000 7f41610ee640 1 Processor -- start 2024-06-03T09:06:44.236 INFO:teuthology.orchestra.run.smithi088.stderr:2024-06-03T09:06:44.233+0000 7f41610ee640 1 -- start start 2024-06-03T09:06:44.236 INFO:teuthology.orchestra.run.smithi088.stderr:2024-06-03T09:06:44.233+0000 7f41610ee640 1 librados: setting wanted keys 2024-06-03T09:06:44.236 INFO:teuthology.orchestra.run.smithi088.stderr:2024-06-03T09:06:44.233+0000 7f41610ee640 1 librados: calling monclient init 2024-06-03T09:06:44.236 INFO:teuthology.orchestra.run.smithi088.stderr:2024-06-03T09:06:44.233+0000 7f41610ee640 10 monclient: init 2024-06-03T09:06:44.238 INFO:teuthology.orchestra.run.smithi088.stderr:2024-06-03T09:06:44.234+0000 7f41610ee640 10 monclient: _reopen_session rank -1 2024-06-03T09:06:44.238 INFO:teuthology.orchestra.run.smithi088.stderr:2024-06-03T09:06:44.234+0000 7f41610ee640 10 monclient: _add_conns ranks=[0,1,2] 2024-06-03T09:06:44.238 INFO:teuthology.orchestra.run.smithi088.stderr:2024-06-03T09:06:44.234+0000 7f41610ee640 10 monclient(hunting): picked mon.noname-a con 0x7f415c06fc70 addr v1:172.21.15.88:6789/0 2024-06-03T09:06:44.238 INFO:teuthology.orchestra.run.smithi088.stderr:2024-06-03T09:06:44.234+0000 7f41610ee640 10 monclient(hunting): picked mon.noname-b con 0x7f415c0ad920 addr v1:172.21.15.192:6789/0 2024-06-03T09:06:44.238 INFO:teuthology.orchestra.run.smithi088.stderr:2024-06-03T09:06:44.234+0000 7f41610ee640 10 monclient(hunting): picked mon.noname-c con 0x7f415c1393f0 addr v1:172.21.15.88:6790/0 2024-06-03T09:06:44.238 INFO:teuthology.orchestra.run.smithi088.stderr:2024-06-03T09:06:44.234+0000 7f41610ee640 1 -- --> v1:172.21.15.88:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f415c143040 con 0x7f415c06fc70 2024-06-03T09:06:44.238 INFO:teuthology.orchestra.run.smithi088.stderr:2024-06-03T09:06:44.234+0000 7f41610ee640 1 -- --> v1:172.21.15.192:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f415c141840 con 0x7f415c0ad920 2024-06-03T09:06:44.238 INFO:teuthology.orchestra.run.smithi088.stderr:2024-06-03T09:06:44.234+0000 7f41610ee640 1 -- --> v1:172.21.15.88:6790/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f415c135e40 con 0x7f415c1393f0 2024-06-03T09:06:44.239 INFO:teuthology.orchestra.run.smithi088.stderr:2024-06-03T09:06:44.234+0000 7f41610ee640 10 monclient(hunting): _renew_subs 2024-06-03T09:06:44.239 INFO:teuthology.orchestra.run.smithi088.stderr:2024-06-03T09:06:44.234+0000 7f41610ee640 10 monclient(hunting): authenticate will time out at 3060.044049s 2024-06-03T09:06:44.239 INFO:teuthology.orchestra.run.smithi088.stderr:2024-06-03T09:06:44.234+0000 7f415b59e640 1 --1- >> v1:172.21.15.88:6790/0 conn(0x7f415c1393f0 0x7f415c1e1180 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=1).handle_server_banner_and_identify peer v1:172.21.15.88:6790/0 says I am v1:172.21.15.88:41338/0 (socket says 172.21.15.88:41338) 2024-06-03T09:06:44.239 INFO:teuthology.orchestra.run.smithi088.stderr:2024-06-03T09:06:44.234+0000 7f415b59e640 1 -- 172.21.15.88:0/2059603519 learned_addr learned my addr 172.21.15.88:0/2059603519 (peer_addr_for_me v1:172.21.15.88:0/0) 2024-06-03T09:06:44.239 INFO:teuthology.orchestra.run.smithi088.stderr:2024-06-03T09:06:44.234+0000 7f41437fe640 10 client.?.objecter ms_handle_connect 0x7f415c1393f0 2024-06-03T09:06:44.239 INFO:teuthology.orchestra.run.smithi088.stderr:2024-06-03T09:06:44.234+0000 7f41437fe640 10 client.?.objecter resend_mon_ops 2024-06-03T09:06:44.239 INFO:teuthology.orchestra.run.smithi088.stderr:2024-06-03T09:06:44.234+0000 7f41437fe640 10 client.?.objecter ms_handle_connect 0x7f415c06fc70 2024-06-03T09:06:44.239 INFO:teuthology.orchestra.run.smithi088.stderr:2024-06-03T09:06:44.234+0000 7f41437fe640 10 client.?.objecter resend_mon_ops 2024-06-03T09:06:44.239 INFO:teuthology.orchestra.run.smithi088.stderr:2024-06-03T09:06:44.235+0000 7f41437fe640 10 client.?.objecter ms_handle_connect 0x7f415c0ad920 2024-06-03T09:06:44.239 INFO:teuthology.orchestra.run.smithi088.stderr:2024-06-03T09:06:44.235+0000 7f41437fe640 10 client.?.objecter resend_mon_ops 2024-06-03T09:06:44.239 INFO:teuthology.orchestra.run.smithi088.stderr:2024-06-03T09:06:44.236+0000 7f41437fe640 1 -- 172.21.15.88:0/2059603519 <== mon.2 v1:172.21.15.88:6790/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 1225470158 0 0) 0x7f415c135e40 con 0x7f415c1393f0 2024-06-03T09:06:44.239 INFO:teuthology.orchestra.run.smithi088.stderr:2024-06-03T09:06:44.236+0000 7f41437fe640 10 monclient(hunting): _negotiate 2024-06-03T09:06:44.239 INFO:teuthology.orchestra.run.smithi088.stderr:2024-06-03T09:06:44.236+0000 7f41437fe640 10 monclient(hunting): _init_auth method 2 2024-06-03T09:06:44.239 INFO:teuthology.orchestra.run.smithi088.stderr:2024-06-03T09:06:44.236+0000 7f41437fe640 10 monclient(hunting): _init_auth creating new auth 2024-06-03T09:06:44.240 INFO:teuthology.orchestra.run.smithi088.stderr:2024-06-03T09:06:44.236+0000 7f41437fe640 10 monclient(hunting): my global_id is 30197 2024-06-03T09:06:44.240 INFO:teuthology.orchestra.run.smithi088.stderr:2024-06-03T09:06:44.236+0000 7f41437fe640 1 -- 172.21.15.88:0/2059603519 --> v1:172.21.15.88:6790/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f4148004eb0 con 0x7f415c1393f0 2024-06-03T09:06:44.240 INFO:teuthology.orchestra.run.smithi088.stderr:2024-06-03T09:06:44.236+0000 7f41437fe640 1 -- 172.21.15.88:0/2059603519 <== mon.1 v1:172.21.15.192:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 3131932764 0 0) 0x7f415c141840 con 0x7f415c0ad920 2024-06-03T09:06:44.240 INFO:teuthology.orchestra.run.smithi088.stderr:2024-06-03T09:06:44.236+0000 7f41437fe640 10 monclient(hunting): _negotiate 2024-06-03T09:06:44.240 INFO:teuthology.orchestra.run.smithi088.stderr:2024-06-03T09:06:44.236+0000 7f41437fe640 10 monclient(hunting): _init_auth method 2 2024-06-03T09:06:44.240 INFO:teuthology.orchestra.run.smithi088.stderr:2024-06-03T09:06:44.236+0000 7f41437fe640 10 monclient(hunting): _init_auth creating new auth 2024-06-03T09:06:44.240 INFO:teuthology.orchestra.run.smithi088.stderr:2024-06-03T09:06:44.236+0000 7f41437fe640 10 monclient(hunting): my global_id is 30271 2024-06-03T09:06:44.240 INFO:teuthology.orchestra.run.smithi088.stderr:2024-06-03T09:06:44.236+0000 7f41437fe640 1 -- 172.21.15.88:0/2059603519 --> v1:172.21.15.192:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f415c135e40 con 0x7f415c0ad920 2024-06-03T09:06:44.240 INFO:teuthology.orchestra.run.smithi088.stderr:2024-06-03T09:06:44.237+0000 7f41437fe640 1 -- 172.21.15.88:0/2059603519 <== mon.0 v1:172.21.15.88:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 2817582198 0 0) 0x7f415c143040 con 0x7f415c06fc70 2024-06-03T09:06:44.240 INFO:teuthology.orchestra.run.smithi088.stderr:2024-06-03T09:06:44.237+0000 7f41437fe640 10 monclient(hunting): _negotiate 2024-06-03T09:06:44.240 INFO:teuthology.orchestra.run.smithi088.stderr:2024-06-03T09:06:44.237+0000 7f41437fe640 10 monclient(hunting): _init_auth method 2 2024-06-03T09:06:44.240 INFO:teuthology.orchestra.run.smithi088.stderr:2024-06-03T09:06:44.237+0000 7f41437fe640 10 monclient(hunting): _init_auth creating new auth 2024-06-03T09:06:44.240 INFO:teuthology.orchestra.run.smithi088.stderr:2024-06-03T09:06:44.237+0000 7f41437fe640 10 monclient(hunting): my global_id is 30693 2024-06-03T09:06:44.240 INFO:teuthology.orchestra.run.smithi088.stderr:2024-06-03T09:06:44.237+0000 7f41437fe640 1 -- 172.21.15.88:0/2059603519 --> v1:172.21.15.88:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f415c141840 con 0x7f415c06fc70 2024-06-03T09:06:44.241 INFO:teuthology.orchestra.run.smithi088.stderr:2024-06-03T09:06:44.241+0000 7f41437fe640 1 -- 172.21.15.88:0/2059603519 <== mon.2 v1:172.21.15.88:6790/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 764+0+0 (unknown 3441862791 0 0) 0x7f4148004eb0 con 0x7f415c1393f0 2024-06-03T09:06:44.241 INFO:teuthology.orchestra.run.smithi088.stderr:2024-06-03T09:06:44.241+0000 7f41437fe640 1 -- 172.21.15.88:0/2059603519 --> v1:172.21.15.88:6790/0 -- auth(proto 2 165 bytes epoch 0) v1 -- 0x7f415c143040 con 0x7f415c1393f0 2024-06-03T09:06:44.242 INFO:teuthology.orchestra.run.smithi088.stderr:2024-06-03T09:06:44.241+0000 7f41437fe640 1 -- 172.21.15.88:0/2059603519 <== mon.1 v1:172.21.15.192:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 764+0+0 (unknown 580055367 0 0) 0x7f415c135e40 con 0x7f415c0ad920 2024-06-03T09:06:44.242 INFO:teuthology.orchestra.run.smithi088.stderr:2024-06-03T09:06:44.241+0000 7f41437fe640 1 -- 172.21.15.88:0/2059603519 --> v1:172.21.15.192:6789/0 -- auth(proto 2 165 bytes epoch 0) v1 -- 0x7f4148004eb0 con 0x7f415c0ad920 2024-06-03T09:06:44.242 INFO:teuthology.orchestra.run.smithi088.stderr:2024-06-03T09:06:44.241+0000 7f41437fe640 1 -- 172.21.15.88:0/2059603519 <== mon.1 v1:172.21.15.192:6789/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 2234374065 0 0) 0x7f41500018b0 con 0x7f415c0ad920 2024-06-03T09:06:44.242 INFO:teuthology.orchestra.run.smithi088.stderr:2024-06-03T09:06:44.241+0000 7f41437fe640 10 monclient(hunting): handle_monmap mon_map magic: 0 v1 2024-06-03T09:06:44.242 INFO:teuthology.orchestra.run.smithi088.stderr:2024-06-03T09:06:44.241+0000 7f41437fe640 10 monclient(hunting): got monmap 1 from mon.noname-b (according to old e1) 2024-06-03T09:06:44.242 INFO:teuthology.orchestra.run.smithi088.stderr:2024-06-03T09:06:44.241+0000 7f41437fe640 10 monclient(hunting): dump: 2024-06-03T09:06:44.242 INFO:teuthology.orchestra.run.smithi088.stderr:epoch 1 2024-06-03T09:06:44.242 INFO:teuthology.orchestra.run.smithi088.stderr:fsid ec3c3fdd-3ac6-4d7d-9412-683130ed8b04 2024-06-03T09:06:44.242 INFO:teuthology.orchestra.run.smithi088.stderr:last_changed 2024-06-03T08:34:31.885125+0000 2024-06-03T09:06:44.242 INFO:teuthology.orchestra.run.smithi088.stderr:created 2024-06-03T08:34:31.885125+0000 2024-06-03T09:06:44.243 INFO:teuthology.orchestra.run.smithi088.stderr:min_mon_release 18 (reef) 2024-06-03T09:06:44.243 INFO:teuthology.orchestra.run.smithi088.stderr:election_strategy: 3 2024-06-03T09:06:44.243 INFO:teuthology.orchestra.run.smithi088.stderr:0: v1:172.21.15.88:6789/0 mon.a 2024-06-03T09:06:44.243 INFO:teuthology.orchestra.run.smithi088.stderr:1: v1:172.21.15.192:6789/0 mon.b 2024-06-03T09:06:44.243 INFO:teuthology.orchestra.run.smithi088.stderr:2: v1:172.21.15.88:6790/0 mon.c 2024-06-03T09:06:44.243 INFO:teuthology.orchestra.run.smithi088.stderr: 2024-06-03T09:06:44.243 INFO:teuthology.orchestra.run.smithi088.stderr:2024-06-03T09:06:44.241+0000 7f41437fe640 10 monclient(hunting): _finish_auth 0 2024-06-03T09:06:44.243 INFO:teuthology.orchestra.run.smithi088.stderr:2024-06-03T09:06:44.241+0000 7f41437fe640 1 -- 172.21.15.88:0/2059603519 <== mon.2 v1:172.21.15.88:6790/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 2234374065 0 0) 0x7f4154001d90 con 0x7f415c1393f0 2024-06-03T09:06:44.243 INFO:teuthology.orchestra.run.smithi088.stderr:2024-06-03T09:06:44.241+0000 7f41437fe640 10 monclient(hunting): handle_monmap mon_map magic: 0 v1 2024-06-03T09:06:44.243 INFO:teuthology.orchestra.run.smithi088.stderr:2024-06-03T09:06:44.241+0000 7f41437fe640 10 monclient(hunting): got monmap 1 from mon.c (according to old e1) 2024-06-03T09:06:44.243 INFO:teuthology.orchestra.run.smithi088.stderr:2024-06-03T09:06:44.241+0000 7f41437fe640 10 monclient(hunting): dump: 2024-06-03T09:06:44.243 INFO:teuthology.orchestra.run.smithi088.stderr:epoch 1 2024-06-03T09:06:44.243 INFO:teuthology.orchestra.run.smithi088.stderr:fsid ec3c3fdd-3ac6-4d7d-9412-683130ed8b04 2024-06-03T09:06:44.244 INFO:teuthology.orchestra.run.smithi088.stderr:last_changed 2024-06-03T08:34:31.885125+0000 2024-06-03T09:06:44.244 INFO:teuthology.orchestra.run.smithi088.stderr:created 2024-06-03T08:34:31.885125+0000 2024-06-03T09:06:44.244 INFO:teuthology.orchestra.run.smithi088.stderr:min_mon_release 18 (reef) 2024-06-03T09:06:44.244 INFO:teuthology.orchestra.run.smithi088.stderr:election_strategy: 3 2024-06-03T09:06:44.244 INFO:teuthology.orchestra.run.smithi088.stderr:0: v1:172.21.15.88:6789/0 mon.a 2024-06-03T09:06:44.244 INFO:teuthology.orchestra.run.smithi088.stderr:1: v1:172.21.15.192:6789/0 mon.b 2024-06-03T09:06:44.244 INFO:teuthology.orchestra.run.smithi088.stderr:2: v1:172.21.15.88:6790/0 mon.c 2024-06-03T09:06:44.244 INFO:teuthology.orchestra.run.smithi088.stderr: 2024-06-03T09:06:44.246 INFO:teuthology.orchestra.run.smithi088.stderr:2024-06-03T09:06:44.242+0000 7f41437fe640 1 -- 172.21.15.88:0/2059603519 <== mon.0 v1:172.21.15.88:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 764+0+0 (unknown 2682444250 0 0) 0x7f415c141840 con 0x7f415c06fc70 2024-06-03T09:06:44.246 INFO:teuthology.orchestra.run.smithi088.stderr:2024-06-03T09:06:44.242+0000 7f41437fe640 1 -- 172.21.15.88:0/2059603519 --> v1:172.21.15.88:6789/0 -- auth(proto 2 165 bytes epoch 0) v1 -- 0x7f415c135e40 con 0x7f415c06fc70 2024-06-03T09:06:44.246 INFO:teuthology.orchestra.run.smithi088.stderr:2024-06-03T09:06:44.242+0000 7f41437fe640 1 -- 172.21.15.88:0/2059603519 <== mon.0 v1:172.21.15.88:6789/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 2234374065 0 0) 0x7f414c001b00 con 0x7f415c06fc70 2024-06-03T09:06:44.246 INFO:teuthology.orchestra.run.smithi088.stderr:2024-06-03T09:06:44.242+0000 7f41437fe640 10 monclient(hunting): handle_monmap mon_map magic: 0 v1 2024-06-03T09:06:44.246 INFO:teuthology.orchestra.run.smithi088.stderr:2024-06-03T09:06:44.242+0000 7f41437fe640 10 monclient(hunting): got monmap 1 from mon.a (according to old e1) 2024-06-03T09:06:44.246 INFO:teuthology.orchestra.run.smithi088.stderr:2024-06-03T09:06:44.242+0000 7f41437fe640 10 monclient(hunting): dump: 2024-06-03T09:06:44.246 INFO:teuthology.orchestra.run.smithi088.stderr:epoch 1 2024-06-03T09:06:44.246 INFO:teuthology.orchestra.run.smithi088.stderr:fsid ec3c3fdd-3ac6-4d7d-9412-683130ed8b04 2024-06-03T09:06:44.247 INFO:teuthology.orchestra.run.smithi088.stderr:last_changed 2024-06-03T08:34:31.885125+0000 2024-06-03T09:06:44.247 INFO:teuthology.orchestra.run.smithi088.stderr:created 2024-06-03T08:34:31.885125+0000 2024-06-03T09:06:44.247 INFO:teuthology.orchestra.run.smithi088.stderr:min_mon_release 18 (reef) 2024-06-03T09:06:44.247 INFO:teuthology.orchestra.run.smithi088.stderr:election_strategy: 3 2024-06-03T09:06:44.247 INFO:teuthology.orchestra.run.smithi088.stderr:0: v1:172.21.15.88:6789/0 mon.a 2024-06-03T09:06:44.247 INFO:teuthology.orchestra.run.smithi088.stderr:1: v1:172.21.15.192:6789/0 mon.b 2024-06-03T09:06:44.247 INFO:teuthology.orchestra.run.smithi088.stderr:2: v1:172.21.15.88:6790/0 mon.c 2024-06-03T09:06:44.247 INFO:teuthology.orchestra.run.smithi088.stderr: 2024-06-03T09:06:44.247 INFO:teuthology.orchestra.run.smithi088.stderr:2024-06-03T09:06:44.244+0000 7f41437fe640 1 -- 172.21.15.88:0/2059603519 <== mon.1 v1:172.21.15.192:6789/0 4 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 580+0+0 (unknown 4106535039 0 0) 0x7f4148004eb0 con 0x7f415c0ad920 2024-06-03T09:06:44.247 INFO:teuthology.orchestra.run.smithi088.stderr:2024-06-03T09:06:44.244+0000 7f41437fe640 1 -- 172.21.15.88:0/2059603519 >> v1:172.21.15.88:6790/0 conn(0x7f415c1393f0 legacy=0x7f415c1e1180 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-06-03T09:06:44.247 INFO:teuthology.orchestra.run.smithi088.stderr:2024-06-03T09:06:44.244+0000 7f41437fe640 1 -- 172.21.15.88:0/2059603519 >> v1:172.21.15.88:6789/0 conn(0x7f415c06fc70 legacy=0x7f415c141130 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-06-03T09:06:44.247 INFO:teuthology.orchestra.run.smithi088.stderr:2024-06-03T09:06:44.244+0000 7f41437fe640 10 monclient: _finish_hunting 0 2024-06-03T09:06:44.247 INFO:teuthology.orchestra.run.smithi088.stderr:2024-06-03T09:06:44.244+0000 7f41437fe640 1 monclient: found mon.b 2024-06-03T09:06:44.247 INFO:teuthology.orchestra.run.smithi088.stderr:2024-06-03T09:06:44.244+0000 7f41437fe640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-06-03T09:06:44.248 INFO:teuthology.orchestra.run.smithi088.stderr:2024-06-03T09:06:44.244+0000 7f41437fe640 10 monclient: _send_mon_message to mon.b at v1:172.21.15.192:6789/0 2024-06-03T09:06:44.248 INFO:teuthology.orchestra.run.smithi088.stderr:2024-06-03T09:06:44.244+0000 7f41437fe640 1 -- 172.21.15.88:0/2059603519 --> v1:172.21.15.192:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f415c1432b0 con 0x7f415c0ad920 2024-06-03T09:06:44.248 INFO:teuthology.orchestra.run.smithi088.stderr:2024-06-03T09:06:44.244+0000 7f41437fe640 10 monclient: _finish_auth 0 2024-06-03T09:06:44.248 INFO:teuthology.orchestra.run.smithi088.stderr:2024-06-03T09:06:44.244+0000 7f41437fe640 10 monclient: _check_auth_tickets 2024-06-03T09:06:44.248 INFO:teuthology.orchestra.run.smithi088.stderr:2024-06-03T09:06:44.244+0000 7f41437fe640 20 monclient: _check_auth_rotating not needed by client.admin 2024-06-03T09:06:44.248 INFO:teuthology.orchestra.run.smithi088.stderr:2024-06-03T09:06:44.244+0000 7f41610ee640 5 monclient: authenticate success, global_id 30271 2024-06-03T09:06:44.248 INFO:teuthology.orchestra.run.smithi088.stderr:2024-06-03T09:06:44.244+0000 7f41610ee640 10 monclient: _renew_subs 2024-06-03T09:06:44.248 INFO:teuthology.orchestra.run.smithi088.stderr:2024-06-03T09:06:44.244+0000 7f41610ee640 10 monclient: _send_mon_message to mon.b at v1:172.21.15.192:6789/0 2024-06-03T09:06:44.248 INFO:teuthology.orchestra.run.smithi088.stderr:2024-06-03T09:06:44.244+0000 7f41610ee640 1 -- 172.21.15.88:0/2059603519 --> v1:172.21.15.192:6789/0 -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f415c143590 con 0x7f415c0ad920 2024-06-03T09:06:44.248 INFO:teuthology.orchestra.run.smithi088.stderr:2024-06-03T09:06:44.244+0000 7f41610ee640 10 client.30271.objecter _maybe_request_map subscribing (onetime) to next osd map 2024-06-03T09:06:44.248 INFO:teuthology.orchestra.run.smithi088.stderr:2024-06-03T09:06:44.244+0000 7f41610ee640 10 monclient: _renew_subs 2024-06-03T09:06:44.249 INFO:teuthology.orchestra.run.smithi088.stderr:2024-06-03T09:06:44.244+0000 7f41610ee640 10 monclient: _send_mon_message to mon.b at v1:172.21.15.192:6789/0 2024-06-03T09:06:44.249 INFO:teuthology.orchestra.run.smithi088.stderr:2024-06-03T09:06:44.244+0000 7f41610ee640 1 -- 172.21.15.88:0/2059603519 --> v1:172.21.15.192:6789/0 -- mon_subscribe({osdmap=0}) v3 -- 0x7f415c141a60 con 0x7f415c0ad920 2024-06-03T09:06:44.249 INFO:teuthology.orchestra.run.smithi088.stderr:2024-06-03T09:06:44.244+0000 7f41610ee640 1 librados: init done 2024-06-03T09:06:44.249 INFO:teuthology.orchestra.run.smithi088.stderr:2024-06-03T09:06:44.245+0000 7f41610ee640 10 start_mon_command cmd=[{"prefix": "get_command_descriptions"}] 2024-06-03T09:06:44.249 INFO:teuthology.orchestra.run.smithi088.stderr:2024-06-03T09:06:44.245+0000 7f41610ee640 10 monclient: _send_command 1 [{"prefix": "get_command_descriptions"}] 2024-06-03T09:06:44.249 INFO:teuthology.orchestra.run.smithi088.stderr:2024-06-03T09:06:44.245+0000 7f41610ee640 10 monclient: _send_mon_message to mon.b at v1:172.21.15.192:6789/0 2024-06-03T09:06:44.249 INFO:teuthology.orchestra.run.smithi088.stderr:2024-06-03T09:06:44.245+0000 7f41610ee640 1 -- 172.21.15.88:0/2059603519 --> v1:172.21.15.192:6789/0 -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f415c141cb0 con 0x7f415c0ad920 2024-06-03T09:06:44.260 INFO:teuthology.orchestra.run.smithi088.stderr:2024-06-03T09:06:44.260+0000 7f41437fe640 1 -- 172.21.15.88:0/2059603519 <== mon.1 v1:172.21.15.192:6789/0 5 ==== config(0 keys) v1 ==== 4+0+0 (unknown 0 0 0) 0x7f41500035f0 con 0x7f415c0ad920 2024-06-03T09:06:44.260 INFO:teuthology.orchestra.run.smithi088.stderr:2024-06-03T09:06:44.260+0000 7f41437fe640 10 monclient: handle_config config(0 keys) v1 2024-06-03T09:06:44.260 INFO:teuthology.orchestra.run.smithi088.stderr:2024-06-03T09:06:44.260+0000 7f41437fe640 1 -- 172.21.15.88:0/2059603519 <== mon.1 v1:172.21.15.192:6789/0 6 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 2234374065 0 0) 0x7f4150001d80 con 0x7f415c0ad920 2024-06-03T09:06:44.260 INFO:teuthology.orchestra.run.smithi088.stderr:2024-06-03T09:06:44.260+0000 7f41437fe640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-06-03T09:06:44.260 INFO:teuthology.orchestra.run.smithi088.stderr:2024-06-03T09:06:44.260+0000 7f41437fe640 10 monclient: got monmap 1 from mon.b (according to old e1) 2024-06-03T09:06:44.260 INFO:teuthology.orchestra.run.smithi088.stderr:2024-06-03T09:06:44.260+0000 7f41437fe640 10 monclient: dump: 2024-06-03T09:06:44.260 INFO:teuthology.orchestra.run.smithi088.stderr:epoch 1 2024-06-03T09:06:44.260 INFO:teuthology.orchestra.run.smithi088.stderr:fsid ec3c3fdd-3ac6-4d7d-9412-683130ed8b04 2024-06-03T09:06:44.260 INFO:teuthology.orchestra.run.smithi088.stderr:last_changed 2024-06-03T08:34:31.885125+0000 2024-06-03T09:06:44.260 INFO:teuthology.orchestra.run.smithi088.stderr:created 2024-06-03T08:34:31.885125+0000 2024-06-03T09:06:44.260 INFO:teuthology.orchestra.run.smithi088.stderr:min_mon_release 18 (reef) 2024-06-03T09:06:44.261 INFO:teuthology.orchestra.run.smithi088.stderr:election_strategy: 3 2024-06-03T09:06:44.261 INFO:teuthology.orchestra.run.smithi088.stderr:0: v1:172.21.15.88:6789/0 mon.a 2024-06-03T09:06:44.261 INFO:teuthology.orchestra.run.smithi088.stderr:1: v1:172.21.15.192:6789/0 mon.b 2024-06-03T09:06:44.261 INFO:teuthology.orchestra.run.smithi088.stderr:2: v1:172.21.15.88:6790/0 mon.c 2024-06-03T09:06:44.261 INFO:teuthology.orchestra.run.smithi088.stderr: 2024-06-03T09:06:44.261 INFO:teuthology.orchestra.run.smithi088.stderr:2024-06-03T09:06:44.260+0000 7f415959a640 4 set_mon_vals no callback set 2024-06-03T09:06:44.261 INFO:teuthology.orchestra.run.smithi088.stderr:2024-06-03T09:06:44.261+0000 7f41437fe640 1 -- 172.21.15.88:0/2059603519 <== mon.1 v1:172.21.15.192:6789/0 7 ==== mgrmap(e 5) v1 ==== 89110+0+0 (unknown 873533973 0 0) 0x7f4150019520 con 0x7f415c0ad920 2024-06-03T09:06:44.262 INFO:teuthology.orchestra.run.smithi088.stderr:2024-06-03T09:06:44.261+0000 7f415ad9d640 10 monclient: get_auth_request con 0x7f414806d420 auth_method 0 2024-06-03T09:06:44.262 INFO:teuthology.orchestra.run.smithi088.stderr:2024-06-03T09:06:44.262+0000 7f41437fe640 10 client.30271.objecter ms_handle_connect 0x7f414806d420 2024-06-03T09:06:44.262 INFO:teuthology.orchestra.run.smithi088.stderr:2024-06-03T09:06:44.262+0000 7f41437fe640 1 -- 172.21.15.88:0/2059603519 <== mon.1 v1:172.21.15.192:6789/0 8 ==== osd_map(1012..1012 src has 622..1012) v4 ==== 5229+0+0 (unknown 2151252637 0 0) 0x7f4150052d70 con 0x7f415c0ad920 2024-06-03T09:06:44.262 INFO:teuthology.orchestra.run.smithi088.stderr:2024-06-03T09:06:44.262+0000 7f41437fe640 10 client.30271.objecter ms_dispatch 0x7f415c000b90 osd_map(1012..1012 src has 622..1012) v4 2024-06-03T09:06:44.262 INFO:teuthology.orchestra.run.smithi088.stderr:2024-06-03T09:06:44.262+0000 7f41437fe640 3 client.30271.objecter handle_osd_map got epochs [1012,1012] > 0 2024-06-03T09:06:44.262 INFO:teuthology.orchestra.run.smithi088.stderr:2024-06-03T09:06:44.262+0000 7f41437fe640 3 client.30271.objecter handle_osd_map decoding full epoch 1012 2024-06-03T09:06:44.263 INFO:teuthology.orchestra.run.smithi088.stderr:2024-06-03T09:06:44.262+0000 7f41437fe640 20 client.30271.objecter dump_active .. 0 homeless 2024-06-03T09:06:44.509 INFO:teuthology.orchestra.run.smithi088.stderr:2024-06-03T09:06:44.509+0000 7f41437fe640 1 -- 172.21.15.88:0/2059603519 <== mon.1 v1:172.21.15.192:6789/0 9 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+182498 (unknown 1092875540 0 3016760700) 0x7f415004e750 con 0x7f415c0ad920 2024-06-03T09:06:44.509 INFO:teuthology.orchestra.run.smithi088.stderr:2024-06-03T09:06:44.509+0000 7f41437fe640 10 monclient: handle_mon_command_ack 1 [{"prefix": "get_command_descriptions"}] 2024-06-03T09:06:44.510 INFO:teuthology.orchestra.run.smithi088.stderr:2024-06-03T09:06:44.509+0000 7f41437fe640 10 monclient: _finish_command 1 = system:0 2024-06-03T09:06:44.701 INFO:teuthology.orchestra.run.smithi088.stderr:2024-06-03T09:06:44.701+0000 7f41610ee640 1 -- 172.21.15.88:0/2059603519 --> v1:172.21.15.192:6800/34229 -- mgr_command(tid 0: {"prefix": "pg dump", "target": ["mon-mgr", ""], "format": "json"}) v1 -- 0x7f415c13ce70 con 0x7f414806d420 2024-06-03T09:06:44.706 INFO:teuthology.orchestra.run.smithi088.stderr:2024-06-03T09:06:44.706+0000 7f41437fe640 1 -- 172.21.15.88:0/2059603519 <== mgr.4110 v1:172.21.15.192:6800/34229 1 ==== mgr_command_reply(tid 0: 0 dumped all) v1 ==== 18+0+151536 (unknown 2965378022 0 2674254763) 0x7f414c007810 con 0x7f414806d420 2024-06-03T09:06:44.707 INFO:teuthology.orchestra.run.smithi088.stdout: 2024-06-03T09:06:44.708 INFO:teuthology.orchestra.run.smithi088.stderr:dumped all 2024-06-03T09:06:44.710 INFO:teuthology.orchestra.run.smithi088.stderr:2024-06-03T09:06:44.708+0000 7f41610ee640 10 librados: watch_flush enter 2024-06-03T09:06:44.710 INFO:teuthology.orchestra.run.smithi088.stderr:2024-06-03T09:06:44.708+0000 7f41610ee640 10 librados: watch_flush exit 2024-06-03T09:06:44.710 INFO:teuthology.orchestra.run.smithi088.stderr:2024-06-03T09:06:44.708+0000 7f41610ee640 20 client.30271.objecter shutdown clearing up homeless session... 2024-06-03T09:06:44.710 INFO:teuthology.orchestra.run.smithi088.stderr:2024-06-03T09:06:44.708+0000 7f41610ee640 10 client.30271.objecter successfully canceled tick 2024-06-03T09:06:44.710 INFO:teuthology.orchestra.run.smithi088.stderr:2024-06-03T09:06:44.708+0000 7f41610ee640 1 -- 172.21.15.88:0/2059603519 >> v1:172.21.15.192:6800/34229 conn(0x7f414806d420 legacy=0x7f414806f8e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-06-03T09:06:44.710 INFO:teuthology.orchestra.run.smithi088.stderr:2024-06-03T09:06:44.708+0000 7f41610ee640 10 monclient: shutdown 2024-06-03T09:06:44.710 INFO:teuthology.orchestra.run.smithi088.stderr:2024-06-03T09:06:44.708+0000 7f41610ee640 20 monclient: shutdown discarding 0 pending message(s) 2024-06-03T09:06:44.710 INFO:teuthology.orchestra.run.smithi088.stderr:2024-06-03T09:06:44.708+0000 7f41610ee640 1 -- 172.21.15.88:0/2059603519 >> v1:172.21.15.192:6789/0 conn(0x7f415c0ad920 legacy=0x7f415c06d370 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-06-03T09:06:44.710 INFO:teuthology.orchestra.run.smithi088.stderr:2024-06-03T09:06:44.708+0000 7f41610ee640 1 -- 172.21.15.88:0/2059603519 shutdown_connections 2024-06-03T09:06:44.710 INFO:teuthology.orchestra.run.smithi088.stderr:2024-06-03T09:06:44.708+0000 7f41610ee640 1 -- 172.21.15.88:0/2059603519 >> 172.21.15.88:0/2059603519 conn(0x7f415c0b2f20 msgr2=0x7f415c0adf80 unknown :-1 s=STATE_NONE l=0).mark_down 2024-06-03T09:06:44.711 INFO:teuthology.orchestra.run.smithi088.stderr:2024-06-03T09:06:44.709+0000 7f41610ee640 1 -- 172.21.15.88:0/2059603519 shutdown_connections 2024-06-03T09:06:44.711 INFO:teuthology.orchestra.run.smithi088.stderr:2024-06-03T09:06:44.709+0000 7f41610ee640 1 -- 172.21.15.88:0/2059603519 wait complete. 2024-06-03T09:06:44.711 INFO:teuthology.orchestra.run.smithi088.stderr:2024-06-03T09:06:44.709+0000 7f41610ee640 1 librados: shutdown 2024-06-03T09:06:44.719 INFO:teuthology.orchestra.run.smithi088.stdout:{"pg_ready":true,"pg_map":{"version":1929,"stamp":"2024-06-03T09:06:43.413439+0000","last_osdmap_epoch":0,"last_pg_scan":0,"pg_stats_sum":{"stat_sum":{"num_bytes":459285,"num_objects":3,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":1,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":3,"num_whiteouts":0,"num_read":102,"num_read_kb":88,"num_write":120,"num_write_kb":1379,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2,"num_bytes_recovered":397840,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":83,"ondisk_log_size":83,"up":68,"acting":68,"num_store_stats":0},"osd_stats_sum":{"up_from":0,"seq":0,"num_pgs":88,"num_osds":8,"num_per_pool_osds":8,"num_per_pool_omap_osds":8,"kb":754974720,"kb_used":494612,"kb_used_data":46388,"kb_used_omap":12,"kb_used_meta":448115,"kb_avail":754480108,"statfs":{"total":773094113280,"available":772587630592,"internally_reserved":0,"allocated":47501312,"data_stored":22782476,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":12744,"internal_metadata":458870328},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":93,"apply_latency_ms":93,"commit_latency_ns":93000000,"apply_latency_ns":93000000},"alerts":[],"network_ping_times":[]},"pg_stats_delta":{"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":0,"acting":0,"num_store_stats":0,"stamp_delta":"8.312065"},"pg_stats":[{"pgid":"17.d","version":"0'0","reported_seq":942,"reported_epoch":1009,"state":"active+clean","last_fresh":"2024-06-03T09:06:14.443917+0000","last_change":"2024-06-03T08:38:19.260548+0000","last_active":"2024-06-03T09:06:14.443917+0000","last_peered":"2024-06-03T09:06:14.443917+0000","last_clean":"2024-06-03T09:06:14.443917+0000","last_became_active":"2024-06-03T08:38:19.253559+0000","last_became_peered":"2024-06-03T08:38:19.253559+0000","last_unstale":"2024-06-03T09:06:14.443917+0000","last_undegraded":"2024-06-03T09:06:14.443917+0000","last_fullsized":"2024-06-03T09:06:14.443917+0000","mapping_epoch":28,"log_start":"0'0","ondisk_log_start":"0'0","created":28,"last_epoch_clean":45,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-06-03T08:37:55.614497+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-06-03T08:37:55.614497+0000","last_clean_scrub_stamp":"2024-06-03T08:37:55.614497+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-06-04T19:00:12.470979+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1],"acting":[0,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"17.c","version":"0'0","reported_seq":953,"reported_epoch":1009,"state":"active+undersized","last_fresh":"2024-06-03T09:06:13.668156+0000","last_change":"2024-06-03T09:06:03.440074+0000","last_active":"2024-06-03T09:06:13.668156+0000","last_peered":"2024-06-03T09:06:13.668156+0000","last_clean":"2024-06-03T09:05:02.105638+0000","last_became_active":"2024-06-03T09:06:03.440074+0000","last_became_peered":"2024-06-03T09:06:03.440074+0000","last_unstale":"2024-06-03T09:06:13.668156+0000","last_undegraded":"2024-06-03T09:06:13.668156+0000","last_fullsized":"2024-06-03T09:06:03.377982+0000","mapping_epoch":999,"log_start":"0'0","ondisk_log_start":"0'0","created":28,"last_epoch_clean":255,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-06-03T08:37:55.614497+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-06-03T08:37:55.614497+0000","last_clean_scrub_stamp":"2024-06-03T08:37:55.614497+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-06-04T10:34:20.808557+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3],"acting":[3],"avail_no_missing":["3"],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"17.f","version":"0'0","reported_seq":148,"reported_epoch":1009,"state":"active+undersized","last_fresh":"2024-06-03T09:06:14.450543+0000","last_change":"2024-06-03T09:06:03.420285+0000","last_active":"2024-06-03T09:06:14.450543+0000","last_peered":"2024-06-03T09:06:14.450543+0000","last_clean":"2024-06-03T08:37:55.614497+0000","last_became_active":"2024-06-03T09:06:03.420285+0000","last_became_peered":"2024-06-03T09:06:03.420285+0000","last_unstale":"2024-06-03T09:06:14.450543+0000","last_undegraded":"2024-06-03T09:06:14.450543+0000","last_fullsized":"2024-06-03T09:06:03.380616+0000","mapping_epoch":999,"log_start":"0'0","ondisk_log_start":"0'0","created":28,"last_epoch_clean":255,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-06-03T08:37:55.614497+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-06-03T08:37:55.614497+0000","last_clean_scrub_stamp":"2024-06-03T08:37:55.614497+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-06-04T15:39:41.187731+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4],"acting":[4],"avail_no_missing":["4"],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"17.e","version":"0'0","reported_seq":149,"reported_epoch":1009,"state":"active+undersized","last_fresh":"2024-06-03T09:06:13.726992+0000","last_change":"2024-06-03T09:06:03.412887+0000","last_active":"2024-06-03T09:06:13.726992+0000","last_peered":"2024-06-03T09:06:13.726992+0000","last_clean":"2024-06-03T08:37:55.614497+0000","last_became_active":"2024-06-03T09:06:03.412887+0000","last_became_peered":"2024-06-03T09:06:03.412887+0000","last_unstale":"2024-06-03T09:06:13.726992+0000","last_undegraded":"2024-06-03T09:06:13.726992+0000","last_fullsized":"2024-06-03T09:06:03.332482+0000","mapping_epoch":999,"log_start":"0'0","ondisk_log_start":"0'0","created":28,"last_epoch_clean":255,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-06-03T08:37:55.614497+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-06-03T08:37:55.614497+0000","last_clean_scrub_stamp":"2024-06-03T08:37:55.614497+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-06-04T12:23:22.511601+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5],"acting":[5],"avail_no_missing":["5"],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"17.9","version":"0'0","reported_seq":946,"reported_epoch":1009,"state":"active+clean","last_fresh":"2024-06-03T09:06:14.999299+0000","last_change":"2024-06-03T08:38:04.772260+0000","last_active":"2024-06-03T09:06:14.999299+0000","last_peered":"2024-06-03T09:06:14.999299+0000","last_clean":"2024-06-03T09:06:14.999299+0000","last_became_active":"2024-06-03T08:38:04.763428+0000","last_became_peered":"2024-06-03T08:38:04.763428+0000","last_unstale":"2024-06-03T09:06:14.999299+0000","last_undegraded":"2024-06-03T09:06:14.999299+0000","last_fullsized":"2024-06-03T09:06:14.999299+0000","mapping_epoch":28,"log_start":"0'0","ondisk_log_start":"0'0","created":28,"last_epoch_clean":33,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-06-03T08:37:55.614497+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-06-03T08:37:55.614497+0000","last_clean_scrub_stamp":"2024-06-03T08:37:55.614497+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-06-04T10:55:10.773936+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,4],"acting":[7,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"17.8","version":"0'0","reported_seq":961,"reported_epoch":1009,"state":"active+undersized","last_fresh":"2024-06-03T09:06:14.430375+0000","last_change":"2024-06-03T09:06:03.388786+0000","last_active":"2024-06-03T09:06:14.430375+0000","last_peered":"2024-06-03T09:06:14.430375+0000","last_clean":"2024-06-03T09:04:43.297780+0000","last_became_active":"2024-06-03T09:06:03.388786+0000","last_became_peered":"2024-06-03T09:06:03.388786+0000","last_unstale":"2024-06-03T09:06:14.430375+0000","last_undegraded":"2024-06-03T09:06:14.430375+0000","last_fullsized":"2024-06-03T09:06:03.331754+0000","mapping_epoch":999,"log_start":"0'0","ondisk_log_start":"0'0","created":28,"last_epoch_clean":255,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-06-03T08:37:55.614497+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-06-03T08:37:55.614497+0000","last_clean_scrub_stamp":"2024-06-03T08:37:55.614497+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-06-04T09:18:44.765547+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0],"acting":[0],"avail_no_missing":["0"],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"17.b","version":"0'0","reported_seq":149,"reported_epoch":1009,"state":"active+undersized","last_fresh":"2024-06-03T09:06:14.433735+0000","last_change":"2024-06-03T09:06:03.451203+0000","last_active":"2024-06-03T09:06:14.433735+0000","last_peered":"2024-06-03T09:06:14.433735+0000","last_clean":"2024-06-03T08:37:55.614497+0000","last_became_active":"2024-06-03T09:06:03.451203+0000","last_became_peered":"2024-06-03T09:06:03.451203+0000","last_unstale":"2024-06-03T09:06:14.433735+0000","last_undegraded":"2024-06-03T09:06:14.433735+0000","last_fullsized":"2024-06-03T09:06:03.375523+0000","mapping_epoch":999,"log_start":"0'0","ondisk_log_start":"0'0","created":28,"last_epoch_clean":255,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-06-03T08:37:55.614497+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-06-03T08:37:55.614497+0000","last_clean_scrub_stamp":"2024-06-03T08:37:55.614497+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-06-04T10:16:19.813495+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2],"acting":[2],"avail_no_missing":["2"],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"17.a","version":"0'0","reported_seq":925,"reported_epoch":1010,"state":"active+clean","last_fresh":"2024-06-03T09:06:18.651697+0000","last_change":"2024-06-03T08:38:05.843142+0000","last_active":"2024-06-03T09:06:18.651697+0000","last_peered":"2024-06-03T09:06:18.651697+0000","last_clean":"2024-06-03T09:06:18.651697+0000","last_became_active":"2024-06-03T08:38:05.809257+0000","last_became_peered":"2024-06-03T08:38:05.809257+0000","last_unstale":"2024-06-03T09:06:18.651697+0000","last_undegraded":"2024-06-03T09:06:18.651697+0000","last_fullsized":"2024-06-03T09:06:18.651697+0000","mapping_epoch":28,"log_start":"0'0","ondisk_log_start":"0'0","created":28,"last_epoch_clean":33,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-06-03T08:37:55.614497+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-06-03T08:37:55.614497+0000","last_clean_scrub_stamp":"2024-06-03T08:37:55.614497+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-06-04T12:12:03.686838+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2],"acting":[1,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"17.5","version":"0'0","reported_seq":943,"reported_epoch":1009,"state":"active+clean","last_fresh":"2024-06-03T09:06:14.442357+0000","last_change":"2024-06-03T08:38:23.466051+0000","last_active":"2024-06-03T09:06:14.442357+0000","last_peered":"2024-06-03T09:06:14.442357+0000","last_clean":"2024-06-03T09:06:14.442357+0000","last_became_active":"2024-06-03T08:38:23.458039+0000","last_became_peered":"2024-06-03T08:38:23.458039+0000","last_unstale":"2024-06-03T09:06:14.442357+0000","last_undegraded":"2024-06-03T09:06:14.442357+0000","last_fullsized":"2024-06-03T09:06:14.442357+0000","mapping_epoch":28,"log_start":"0'0","ondisk_log_start":"0'0","created":28,"last_epoch_clean":49,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-06-03T08:37:55.614497+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-06-03T08:37:55.614497+0000","last_clean_scrub_stamp":"2024-06-03T08:37:55.614497+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-06-04T12:27:02.532436+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,4],"acting":[0,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"17.4","version":"0'0","reported_seq":149,"reported_epoch":1009,"state":"active+undersized","last_fresh":"2024-06-03T09:06:13.941522+0000","last_change":"2024-06-03T09:06:03.441889+0000","last_active":"2024-06-03T09:06:13.941522+0000","last_peered":"2024-06-03T09:06:13.941522+0000","last_clean":"2024-06-03T08:37:55.614497+0000","last_became_active":"2024-06-03T09:06:03.441889+0000","last_became_peered":"2024-06-03T09:06:03.441889+0000","last_unstale":"2024-06-03T09:06:13.941522+0000","last_undegraded":"2024-06-03T09:06:13.941522+0000","last_fullsized":"2024-06-03T09:06:03.359582+0000","mapping_epoch":999,"log_start":"0'0","ondisk_log_start":"0'0","created":28,"last_epoch_clean":255,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-06-03T08:37:55.614497+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-06-03T08:37:55.614497+0000","last_clean_scrub_stamp":"2024-06-03T08:37:55.614497+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-06-04T12:49:22.907141+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7],"acting":[7],"avail_no_missing":["7"],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"17.7","version":"0'0","reported_seq":929,"reported_epoch":1009,"state":"active+clean","last_fresh":"2024-06-03T09:06:13.736920+0000","last_change":"2024-06-03T08:38:04.837083+0000","last_active":"2024-06-03T09:06:13.736920+0000","last_peered":"2024-06-03T09:06:13.736920+0000","last_clean":"2024-06-03T09:06:13.736920+0000","last_became_active":"2024-06-03T08:38:04.818246+0000","last_became_peered":"2024-06-03T08:38:04.818246+0000","last_unstale":"2024-06-03T09:06:13.736920+0000","last_undegraded":"2024-06-03T09:06:13.736920+0000","last_fullsized":"2024-06-03T09:06:13.736920+0000","mapping_epoch":28,"log_start":"0'0","ondisk_log_start":"0'0","created":28,"last_epoch_clean":34,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-06-03T08:37:55.614497+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-06-03T08:37:55.614497+0000","last_clean_scrub_stamp":"2024-06-03T08:37:55.614497+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-06-04T17:02:38.204813+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,2],"acting":[5,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"17.6","version":"0'0","reported_seq":960,"reported_epoch":1009,"state":"active+undersized","last_fresh":"2024-06-03T09:06:14.453110+0000","last_change":"2024-06-03T09:06:03.428677+0000","last_active":"2024-06-03T09:06:14.453110+0000","last_peered":"2024-06-03T09:06:14.453110+0000","last_clean":"2024-06-03T09:04:43.312352+0000","last_became_active":"2024-06-03T09:06:03.428677+0000","last_became_peered":"2024-06-03T09:06:03.428677+0000","last_unstale":"2024-06-03T09:06:14.453110+0000","last_undegraded":"2024-06-03T09:06:14.453110+0000","last_fullsized":"2024-06-03T09:06:03.359909+0000","mapping_epoch":999,"log_start":"0'0","ondisk_log_start":"0'0","created":28,"last_epoch_clean":255,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-06-03T08:37:55.614497+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-06-03T08:37:55.614497+0000","last_clean_scrub_stamp":"2024-06-03T08:37:55.614497+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-06-04T15:38:02.180241+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0],"acting":[0],"avail_no_missing":["0"],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"2.7","version":"0'0","reported_seq":940,"reported_epoch":1009,"state":"active+clean","last_fresh":"2024-06-03T09:06:13.684065+0000","last_change":"2024-06-03T08:36:30.385405+0000","last_active":"2024-06-03T09:06:13.684065+0000","last_peered":"2024-06-03T09:06:13.684065+0000","last_clean":"2024-06-03T09:06:13.684065+0000","last_became_active":"2024-06-03T08:36:30.380302+0000","last_became_peered":"2024-06-03T08:36:30.380302+0000","last_unstale":"2024-06-03T09:06:13.684065+0000","last_undegraded":"2024-06-03T09:06:13.684065+0000","last_fullsized":"2024-06-03T09:06:13.684065+0000","mapping_epoch":23,"log_start":"0'0","ondisk_log_start":"0'0","created":23,"last_epoch_clean":24,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-06-03T08:36:28.444764+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-06-03T08:36:28.444764+0000","last_clean_scrub_stamp":"2024-06-03T08:36:28.444764+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-06-04T18:46:33.475181+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0],"acting":[3,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"17.14","version":"0'0","reported_seq":926,"reported_epoch":1009,"state":"active+clean","last_fresh":"2024-06-03T09:06:13.721460+0000","last_change":"2024-06-03T08:38:04.547565+0000","last_active":"2024-06-03T09:06:13.721460+0000","last_peered":"2024-06-03T09:06:13.721460+0000","last_clean":"2024-06-03T09:06:13.721460+0000","last_became_active":"2024-06-03T08:38:04.541781+0000","last_became_peered":"2024-06-03T08:38:04.541781+0000","last_unstale":"2024-06-03T09:06:13.721460+0000","last_undegraded":"2024-06-03T09:06:13.721460+0000","last_fullsized":"2024-06-03T09:06:13.721460+0000","mapping_epoch":28,"log_start":"0'0","ondisk_log_start":"0'0","created":28,"last_epoch_clean":34,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-06-03T08:37:55.614497+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-06-03T08:37:55.614497+0000","last_clean_scrub_stamp":"2024-06-03T08:37:55.614497+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-06-04T16:41:55.170134+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,2],"acting":[5,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"2.6","version":"0'0","reported_seq":949,"reported_epoch":1009,"state":"active+clean","last_fresh":"2024-06-03T09:06:14.448100+0000","last_change":"2024-06-03T08:36:30.430760+0000","last_active":"2024-06-03T09:06:14.448100+0000","last_peered":"2024-06-03T09:06:14.448100+0000","last_clean":"2024-06-03T09:06:14.448100+0000","last_became_active":"2024-06-03T08:36:30.404665+0000","last_became_peered":"2024-06-03T08:36:30.404665+0000","last_unstale":"2024-06-03T09:06:14.448100+0000","last_undegraded":"2024-06-03T09:06:14.448100+0000","last_fullsized":"2024-06-03T09:06:14.448100+0000","mapping_epoch":23,"log_start":"0'0","ondisk_log_start":"0'0","created":23,"last_epoch_clean":24,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-06-03T08:36:28.444764+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-06-03T08:36:28.444764+0000","last_clean_scrub_stamp":"2024-06-03T08:36:28.444764+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-06-04T10:37:37.746074+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,0],"acting":[4,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"17.15","version":"0'0","reported_seq":934,"reported_epoch":1009,"state":"active+clean","last_fresh":"2024-06-03T09:06:13.672940+0000","last_change":"2024-06-03T08:38:20.999903+0000","last_active":"2024-06-03T09:06:13.672940+0000","last_peered":"2024-06-03T09:06:13.672940+0000","last_clean":"2024-06-03T09:06:13.672940+0000","last_became_active":"2024-06-03T08:38:20.979523+0000","last_became_peered":"2024-06-03T08:38:20.979523+0000","last_unstale":"2024-06-03T09:06:13.672940+0000","last_undegraded":"2024-06-03T09:06:13.672940+0000","last_fullsized":"2024-06-03T09:06:13.672940+0000","mapping_epoch":28,"log_start":"0'0","ondisk_log_start":"0'0","created":28,"last_epoch_clean":47,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-06-03T08:37:55.614497+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-06-03T08:37:55.614497+0000","last_clean_scrub_stamp":"2024-06-03T08:37:55.614497+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-06-04T20:29:58.314289+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,1],"acting":[3,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"2.5","version":"0'0","reported_seq":942,"reported_epoch":1010,"state":"active+clean","last_fresh":"2024-06-03T09:06:18.382767+0000","last_change":"2024-06-03T08:36:30.024235+0000","last_active":"2024-06-03T09:06:18.382767+0000","last_peered":"2024-06-03T09:06:18.382767+0000","last_clean":"2024-06-03T09:06:18.382767+0000","last_became_active":"2024-06-03T08:36:30.019040+0000","last_became_peered":"2024-06-03T08:36:30.019040+0000","last_unstale":"2024-06-03T09:06:18.382767+0000","last_undegraded":"2024-06-03T09:06:18.382767+0000","last_fullsized":"2024-06-03T09:06:18.382767+0000","mapping_epoch":23,"log_start":"0'0","ondisk_log_start":"0'0","created":23,"last_epoch_clean":24,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-06-03T08:36:28.444764+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-06-03T08:36:28.444764+0000","last_clean_scrub_stamp":"2024-06-03T08:36:28.444764+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-06-04T15:46:47.109502+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,4],"acting":[3,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"17.16","version":"0'0","reported_seq":972,"reported_epoch":1009,"state":"active+undersized","last_fresh":"2024-06-03T09:06:14.441665+0000","last_change":"2024-06-03T09:06:03.478615+0000","last_active":"2024-06-03T09:06:14.441665+0000","last_peered":"2024-06-03T09:06:14.441665+0000","last_clean":"2024-06-03T09:04:43.328415+0000","last_became_active":"2024-06-03T09:06:03.478615+0000","last_became_peered":"2024-06-03T09:06:03.478615+0000","last_unstale":"2024-06-03T09:06:14.441665+0000","last_undegraded":"2024-06-03T09:06:14.441665+0000","last_fullsized":"2024-06-03T09:06:03.402504+0000","mapping_epoch":999,"log_start":"0'0","ondisk_log_start":"0'0","created":28,"last_epoch_clean":255,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-06-03T08:37:55.614497+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-06-03T08:37:55.614497+0000","last_clean_scrub_stamp":"2024-06-03T08:37:55.614497+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-06-04T08:47:08.079233+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2],"acting":[2],"avail_no_missing":["2"],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"2.4","version":"0'0","reported_seq":952,"reported_epoch":1010,"state":"active+undersized","last_fresh":"2024-06-03T09:06:18.331485+0000","last_change":"2024-06-03T09:06:03.387171+0000","last_active":"2024-06-03T09:06:18.331485+0000","last_peered":"2024-06-03T09:06:18.331485+0000","last_clean":"2024-06-03T09:04:43.202645+0000","last_became_active":"2024-06-03T09:06:03.387171+0000","last_became_peered":"2024-06-03T09:06:03.387171+0000","last_unstale":"2024-06-03T09:06:18.331485+0000","last_undegraded":"2024-06-03T09:06:18.331485+0000","last_fullsized":"2024-06-03T09:06:03.283787+0000","mapping_epoch":999,"log_start":"0'0","ondisk_log_start":"0'0","created":23,"last_epoch_clean":255,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-06-03T08:36:28.444764+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-06-03T08:36:28.444764+0000","last_clean_scrub_stamp":"2024-06-03T08:36:28.444764+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-06-04T20:17:43.485023+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5],"acting":[5],"avail_no_missing":["5"],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"17.17","version":"0'0","reported_seq":954,"reported_epoch":1009,"state":"active+clean","last_fresh":"2024-06-03T09:06:14.447046+0000","last_change":"2024-06-03T08:38:38.483275+0000","last_active":"2024-06-03T09:06:14.447046+0000","last_peered":"2024-06-03T09:06:14.447046+0000","last_clean":"2024-06-03T09:06:14.447046+0000","last_became_active":"2024-06-03T08:38:38.467916+0000","last_became_peered":"2024-06-03T08:38:38.467916+0000","last_unstale":"2024-06-03T09:06:14.447046+0000","last_undegraded":"2024-06-03T09:06:14.447046+0000","last_fullsized":"2024-06-03T09:06:14.447046+0000","mapping_epoch":28,"log_start":"0'0","ondisk_log_start":"0'0","created":28,"last_epoch_clean":61,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-06-03T08:37:55.614497+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-06-03T08:37:55.614497+0000","last_clean_scrub_stamp":"2024-06-03T08:37:55.614497+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-06-04T10:02:23.036420+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,7],"acting":[2,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"2.2","version":"0'0","reported_seq":963,"reported_epoch":1009,"state":"active+undersized","last_fresh":"2024-06-03T09:06:13.657892+0000","last_change":"2024-06-03T09:06:03.410020+0000","last_active":"2024-06-03T09:06:13.657892+0000","last_peered":"2024-06-03T09:06:13.657892+0000","last_clean":"2024-06-03T09:05:37.145889+0000","last_became_active":"2024-06-03T09:06:03.410020+0000","last_became_peered":"2024-06-03T09:06:03.410020+0000","last_unstale":"2024-06-03T09:06:13.657892+0000","last_undegraded":"2024-06-03T09:06:13.657892+0000","last_fullsized":"2024-06-03T09:06:03.346864+0000","mapping_epoch":999,"log_start":"0'0","ondisk_log_start":"0'0","created":23,"last_epoch_clean":255,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-06-03T08:36:28.444764+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-06-03T08:36:28.444764+0000","last_clean_scrub_stamp":"2024-06-03T08:36:28.444764+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-06-04T09:57:12.423360+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3],"acting":[3],"avail_no_missing":["3"],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"17.11","version":"0'0","reported_seq":944,"reported_epoch":1009,"state":"active+clean","last_fresh":"2024-06-03T09:06:14.426670+0000","last_change":"2024-06-03T08:38:06.502302+0000","last_active":"2024-06-03T09:06:14.426670+0000","last_peered":"2024-06-03T09:06:14.426670+0000","last_clean":"2024-06-03T09:06:14.426670+0000","last_became_active":"2024-06-03T08:38:06.490204+0000","last_became_peered":"2024-06-03T08:38:06.490204+0000","last_unstale":"2024-06-03T09:06:14.426670+0000","last_undegraded":"2024-06-03T09:06:14.426670+0000","last_fullsized":"2024-06-03T09:06:14.426670+0000","mapping_epoch":28,"log_start":"0'0","ondisk_log_start":"0'0","created":28,"last_epoch_clean":32,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-06-03T08:37:55.614497+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-06-03T08:37:55.614497+0000","last_clean_scrub_stamp":"2024-06-03T08:37:55.614497+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-06-04T10:47:05.988958+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,2],"acting":[4,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"2.1","version":"0'0","reported_seq":952,"reported_epoch":1009,"state":"active+clean","last_fresh":"2024-06-03T09:06:13.929725+0000","last_change":"2024-06-03T08:36:30.883360+0000","last_active":"2024-06-03T09:06:13.929725+0000","last_peered":"2024-06-03T09:06:13.929725+0000","last_clean":"2024-06-03T09:06:13.929725+0000","last_became_active":"2024-06-03T08:36:30.855073+0000","last_became_peered":"2024-06-03T08:36:30.855073+0000","last_unstale":"2024-06-03T09:06:13.929725+0000","last_undegraded":"2024-06-03T09:06:13.929725+0000","last_fullsized":"2024-06-03T09:06:13.929725+0000","mapping_epoch":23,"log_start":"0'0","ondisk_log_start":"0'0","created":23,"last_epoch_clean":25,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-06-03T08:36:28.444764+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-06-03T08:36:28.444764+0000","last_clean_scrub_stamp":"2024-06-03T08:36:28.444764+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-06-04T18:12:22.363473+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,5],"acting":[7,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"17.12","version":"0'0","reported_seq":945,"reported_epoch":1009,"state":"active+clean","last_fresh":"2024-06-03T09:06:13.935141+0000","last_change":"2024-06-03T08:38:01.751562+0000","last_active":"2024-06-03T09:06:13.935141+0000","last_peered":"2024-06-03T09:06:13.935141+0000","last_clean":"2024-06-03T09:06:13.935141+0000","last_became_active":"2024-06-03T08:38:01.745027+0000","last_became_peered":"2024-06-03T08:38:01.745027+0000","last_unstale":"2024-06-03T09:06:13.935141+0000","last_undegraded":"2024-06-03T09:06:13.935141+0000","last_fullsized":"2024-06-03T09:06:13.935141+0000","mapping_epoch":28,"log_start":"0'0","ondisk_log_start":"0'0","created":28,"last_epoch_clean":32,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-06-03T08:37:55.614497+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-06-03T08:37:55.614497+0000","last_clean_scrub_stamp":"2024-06-03T08:37:55.614497+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-06-04T17:54:43.785326+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,3],"acting":[7,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"2.0","version":"0'0","reported_seq":943,"reported_epoch":1009,"state":"active+clean","last_fresh":"2024-06-03T09:06:13.648112+0000","last_change":"2024-06-03T08:36:29.984096+0000","last_active":"2024-06-03T09:06:13.648112+0000","last_peered":"2024-06-03T09:06:13.648112+0000","last_clean":"2024-06-03T09:06:13.648112+0000","last_became_active":"2024-06-03T08:36:29.957875+0000","last_became_peered":"2024-06-03T08:36:29.957875+0000","last_unstale":"2024-06-03T09:06:13.648112+0000","last_undegraded":"2024-06-03T09:06:13.648112+0000","last_fullsized":"2024-06-03T09:06:13.648112+0000","mapping_epoch":23,"log_start":"0'0","ondisk_log_start":"0'0","created":23,"last_epoch_clean":24,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-06-03T08:36:28.444764+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-06-03T08:36:28.444764+0000","last_clean_scrub_stamp":"2024-06-03T08:36:28.444764+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-06-04T08:41:33.679479+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,5],"acting":[3,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"17.13","version":"0'0","reported_seq":943,"reported_epoch":1009,"state":"active+clean","last_fresh":"2024-06-03T09:06:14.438399+0000","last_change":"2024-06-03T08:38:26.838316+0000","last_active":"2024-06-03T09:06:14.438399+0000","last_peered":"2024-06-03T09:06:14.438399+0000","last_clean":"2024-06-03T09:06:14.438399+0000","last_became_active":"2024-06-03T08:38:26.833214+0000","last_became_peered":"2024-06-03T08:38:26.833214+0000","last_unstale":"2024-06-03T09:06:14.438399+0000","last_undegraded":"2024-06-03T09:06:14.438399+0000","last_fullsized":"2024-06-03T09:06:14.438399+0000","mapping_epoch":28,"log_start":"0'0","ondisk_log_start":"0'0","created":28,"last_epoch_clean":52,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-06-03T08:37:55.614497+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-06-03T08:37:55.614497+0000","last_clean_scrub_stamp":"2024-06-03T08:37:55.614497+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-06-04T12:58:51.617998+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,0],"acting":[4,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"2.3","version":"0'0","reported_seq":943,"reported_epoch":1010,"state":"active+clean","last_fresh":"2024-06-03T09:06:18.362614+0000","last_change":"2024-06-03T08:36:30.351670+0000","last_active":"2024-06-03T09:06:18.362614+0000","last_peered":"2024-06-03T09:06:18.362614+0000","last_clean":"2024-06-03T09:06:18.362614+0000","last_became_active":"2024-06-03T08:36:30.343560+0000","last_became_peered":"2024-06-03T08:36:30.343560+0000","last_unstale":"2024-06-03T09:06:18.362614+0000","last_undegraded":"2024-06-03T09:06:18.362614+0000","last_fullsized":"2024-06-03T09:06:18.362614+0000","mapping_epoch":23,"log_start":"0'0","ondisk_log_start":"0'0","created":23,"last_epoch_clean":24,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-06-03T08:36:28.444764+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-06-03T08:36:28.444764+0000","last_clean_scrub_stamp":"2024-06-03T08:36:28.444764+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-06-04T12:02:24.765135+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,2],"acting":[3,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.0","version":"19'76","reported_seq":1051,"reported_epoch":1009,"state":"active+clean","last_fresh":"2024-06-03T09:06:13.703685+0000","last_change":"2024-06-03T08:36:26.702702+0000","last_active":"2024-06-03T09:06:13.703685+0000","last_peered":"2024-06-03T09:06:13.703685+0000","last_clean":"2024-06-03T09:06:13.703685+0000","last_became_active":"2024-06-03T08:36:26.128378+0000","last_became_peered":"2024-06-03T08:36:26.128378+0000","last_unstale":"2024-06-03T09:06:13.703685+0000","last_undegraded":"2024-06-03T09:06:13.703685+0000","last_fullsized":"2024-06-03T09:06:13.703685+0000","mapping_epoch":20,"log_start":"0'0","ondisk_log_start":"0'0","created":16,"last_epoch_clean":22,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-06-03T08:36:15.542635+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-06-03T08:36:15.542635+0000","last_clean_scrub_stamp":"2024-06-03T08:36:15.542635+0000","objects_scrubbed":0,"log_size":76,"log_dups_size":0,"ondisk_log_size":76,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-06-04T12:41:53.731360+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":459280,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":96,"num_read_kb":82,"num_write":113,"num_write_kb":1372,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2,"num_bytes_recovered":397840,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,5],"acting":[1,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"17.10","version":"0'0","reported_seq":944,"reported_epoch":1009,"state":"active+clean","last_fresh":"2024-06-03T09:06:14.422035+0000","last_change":"2024-06-03T08:38:15.399189+0000","last_active":"2024-06-03T09:06:14.422035+0000","last_peered":"2024-06-03T09:06:14.422035+0000","last_clean":"2024-06-03T09:06:14.422035+0000","last_became_active":"2024-06-03T08:38:15.337723+0000","last_became_peered":"2024-06-03T08:38:15.337723+0000","last_unstale":"2024-06-03T09:06:14.422035+0000","last_undegraded":"2024-06-03T09:06:14.422035+0000","last_fullsized":"2024-06-03T09:06:14.422035+0000","mapping_epoch":28,"log_start":"0'0","ondisk_log_start":"0'0","created":28,"last_epoch_clean":42,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-06-03T08:37:55.614497+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-06-03T08:37:55.614497+0000","last_clean_scrub_stamp":"2024-06-03T08:37:55.614497+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-06-04T09:29:46.427317+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,3],"acting":[4,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"17.1a","version":"0'0","reported_seq":960,"reported_epoch":1009,"state":"active+undersized","last_fresh":"2024-06-03T09:06:14.431224+0000","last_change":"2024-06-03T09:06:03.390734+0000","last_active":"2024-06-03T09:06:14.431224+0000","last_peered":"2024-06-03T09:06:14.431224+0000","last_clean":"2024-06-03T09:05:20.322313+0000","last_became_active":"2024-06-03T09:06:03.390734+0000","last_became_peered":"2024-06-03T09:06:03.390734+0000","last_unstale":"2024-06-03T09:06:14.431224+0000","last_undegraded":"2024-06-03T09:06:14.431224+0000","last_fullsized":"2024-06-03T09:06:03.343197+0000","mapping_epoch":999,"log_start":"0'0","ondisk_log_start":"0'0","created":28,"last_epoch_clean":254,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-06-03T08:37:55.614497+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-06-03T08:37:55.614497+0000","last_clean_scrub_stamp":"2024-06-03T08:37:55.614497+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-06-04T15:56:47.628875+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4],"acting":[4],"avail_no_missing":["4"],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"17.1b","version":"0'0","reported_seq":935,"reported_epoch":1009,"state":"active+clean","last_fresh":"2024-06-03T09:06:13.663279+0000","last_change":"2024-06-03T08:38:03.115748+0000","last_active":"2024-06-03T09:06:13.663279+0000","last_peered":"2024-06-03T09:06:13.663279+0000","last_clean":"2024-06-03T09:06:13.663279+0000","last_became_active":"2024-06-03T08:38:03.110533+0000","last_became_peered":"2024-06-03T08:38:03.110533+0000","last_unstale":"2024-06-03T09:06:13.663279+0000","last_undegraded":"2024-06-03T09:06:13.663279+0000","last_fullsized":"2024-06-03T09:06:13.663279+0000","mapping_epoch":28,"log_start":"0'0","ondisk_log_start":"0'0","created":28,"last_epoch_clean":32,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-06-03T08:37:55.614497+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-06-03T08:37:55.614497+0000","last_clean_scrub_stamp":"2024-06-03T08:37:55.614497+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-06-04T12:36:47.471609+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,7],"acting":[3,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"17.18","version":"0'0","reported_seq":944,"reported_epoch":1009,"state":"active+clean","last_fresh":"2024-06-03T09:06:13.924799+0000","last_change":"2024-06-03T08:38:04.632864+0000","last_active":"2024-06-03T09:06:13.924799+0000","last_peered":"2024-06-03T09:06:13.924799+0000","last_clean":"2024-06-03T09:06:13.924799+0000","last_became_active":"2024-06-03T08:38:04.627856+0000","last_became_peered":"2024-06-03T08:38:04.627856+0000","last_unstale":"2024-06-03T09:06:13.924799+0000","last_undegraded":"2024-06-03T09:06:13.924799+0000","last_fullsized":"2024-06-03T09:06:13.924799+0000","mapping_epoch":28,"log_start":"0'0","ondisk_log_start":"0'0","created":28,"last_epoch_clean":33,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-06-03T08:37:55.614497+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-06-03T08:37:55.614497+0000","last_clean_scrub_stamp":"2024-06-03T08:37:55.614497+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-06-04T19:55:51.181598+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,4],"acting":[7,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"17.19","version":"0'0","reported_seq":934,"reported_epoch":1009,"state":"active+clean","last_fresh":"2024-06-03T09:06:13.652943+0000","last_change":"2024-06-03T08:38:01.839491+0000","last_active":"2024-06-03T09:06:13.652943+0000","last_peered":"2024-06-03T09:06:13.652943+0000","last_clean":"2024-06-03T09:06:13.652943+0000","last_became_active":"2024-06-03T08:38:01.829539+0000","last_became_peered":"2024-06-03T08:38:01.829539+0000","last_unstale":"2024-06-03T09:06:13.652943+0000","last_undegraded":"2024-06-03T09:06:13.652943+0000","last_fullsized":"2024-06-03T09:06:13.652943+0000","mapping_epoch":28,"log_start":"0'0","ondisk_log_start":"0'0","created":28,"last_epoch_clean":30,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-06-03T08:37:55.614497+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-06-03T08:37:55.614497+0000","last_clean_scrub_stamp":"2024-06-03T08:37:55.614497+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-06-04T09:10:43.860229+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,2],"acting":[3,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"17.1e","version":"0'0","reported_seq":148,"reported_epoch":1009,"state":"active+undersized","last_fresh":"2024-06-03T09:06:14.445917+0000","last_change":"2024-06-03T09:06:03.406181+0000","last_active":"2024-06-03T09:06:14.445917+0000","last_peered":"2024-06-03T09:06:14.445917+0000","last_clean":"2024-06-03T08:37:55.614497+0000","last_became_active":"2024-06-03T09:06:03.406181+0000","last_became_peered":"2024-06-03T09:06:03.406181+0000","last_unstale":"2024-06-03T09:06:14.445917+0000","last_undegraded":"2024-06-03T09:06:14.445917+0000","last_fullsized":"2024-06-03T09:06:03.363739+0000","mapping_epoch":999,"log_start":"0'0","ondisk_log_start":"0'0","created":28,"last_epoch_clean":255,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-06-03T08:37:55.614497+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-06-03T08:37:55.614497+0000","last_clean_scrub_stamp":"2024-06-03T08:37:55.614497+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-06-04T11:45:24.486600+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4],"acting":[4],"avail_no_missing":["4"],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"17.1f","version":"0'0","reported_seq":941,"reported_epoch":1009,"state":"active+clean","last_fresh":"2024-06-03T09:06:14.450053+0000","last_change":"2024-06-03T08:38:04.383538+0000","last_active":"2024-06-03T09:06:14.450053+0000","last_peered":"2024-06-03T09:06:14.450053+0000","last_clean":"2024-06-03T09:06:14.450053+0000","last_became_active":"2024-06-03T08:38:04.367278+0000","last_became_peered":"2024-06-03T08:38:04.367278+0000","last_unstale":"2024-06-03T09:06:14.450053+0000","last_undegraded":"2024-06-03T09:06:14.450053+0000","last_fullsized":"2024-06-03T09:06:14.450053+0000","mapping_epoch":28,"log_start":"0'0","ondisk_log_start":"0'0","created":28,"last_epoch_clean":32,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-06-03T08:37:55.614497+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-06-03T08:37:55.614497+0000","last_clean_scrub_stamp":"2024-06-03T08:37:55.614497+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-06-04T10:39:26.552282+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,2],"acting":[0,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"17.1c","version":"0'0","reported_seq":944,"reported_epoch":1010,"state":"active+undersized","last_fresh":"2024-06-03T09:06:18.327492+0000","last_change":"2024-06-03T09:06:03.382210+0000","last_active":"2024-06-03T09:06:18.327492+0000","last_peered":"2024-06-03T09:06:18.327492+0000","last_clean":"2024-06-03T09:04:43.199660+0000","last_became_active":"2024-06-03T09:06:03.382210+0000","last_became_peered":"2024-06-03T09:06:03.382210+0000","last_unstale":"2024-06-03T09:06:18.327492+0000","last_undegraded":"2024-06-03T09:06:18.327492+0000","last_fullsized":"2024-06-03T09:06:03.277874+0000","mapping_epoch":999,"log_start":"0'0","ondisk_log_start":"0'0","created":28,"last_epoch_clean":255,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-06-03T08:37:55.614497+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-06-03T08:37:55.614497+0000","last_clean_scrub_stamp":"2024-06-03T08:37:55.614497+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-06-04T12:18:50.013884+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5],"acting":[5],"avail_no_missing":["5"],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"17.1d","version":"0'0","reported_seq":923,"reported_epoch":1009,"state":"active+clean","last_fresh":"2024-06-03T09:06:13.715906+0000","last_change":"2024-06-03T08:38:06.794630+0000","last_active":"2024-06-03T09:06:13.715906+0000","last_peered":"2024-06-03T09:06:13.715906+0000","last_clean":"2024-06-03T09:06:13.715906+0000","last_became_active":"2024-06-03T08:38:06.592317+0000","last_became_peered":"2024-06-03T08:38:06.592317+0000","last_unstale":"2024-06-03T09:06:13.715906+0000","last_undegraded":"2024-06-03T09:06:13.715906+0000","last_fullsized":"2024-06-03T09:06:13.715906+0000","mapping_epoch":28,"log_start":"0'0","ondisk_log_start":"0'0","created":28,"last_epoch_clean":33,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-06-03T08:37:55.614497+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-06-03T08:37:55.614497+0000","last_clean_scrub_stamp":"2024-06-03T08:37:55.614497+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-06-04T12:33:30.785369+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,7],"acting":[1,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"17.2","version":"0'0","reported_seq":925,"reported_epoch":1009,"state":"active+clean","last_fresh":"2024-06-03T09:06:13.708688+0000","last_change":"2024-06-03T08:38:14.981410+0000","last_active":"2024-06-03T09:06:13.708688+0000","last_peered":"2024-06-03T09:06:13.708688+0000","last_clean":"2024-06-03T09:06:13.708688+0000","last_became_active":"2024-06-03T08:38:14.973825+0000","last_became_peered":"2024-06-03T08:38:14.973825+0000","last_unstale":"2024-06-03T09:06:13.708688+0000","last_undegraded":"2024-06-03T09:06:13.708688+0000","last_fullsized":"2024-06-03T09:06:13.708688+0000","mapping_epoch":28,"log_start":"0'0","ondisk_log_start":"0'0","created":28,"last_epoch_clean":41,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-06-03T08:37:55.614497+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-06-03T08:37:55.614497+0000","last_clean_scrub_stamp":"2024-06-03T08:37:55.614497+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-06-04T17:01:05.518613+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,4],"acting":[1,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"17.3","version":"0'0","reported_seq":943,"reported_epoch":1009,"state":"active+clean","last_fresh":"2024-06-03T09:06:14.437636+0000","last_change":"2024-06-03T08:38:14.619745+0000","last_active":"2024-06-03T09:06:14.437636+0000","last_peered":"2024-06-03T09:06:14.437636+0000","last_clean":"2024-06-03T09:06:14.437636+0000","last_became_active":"2024-06-03T08:38:14.510665+0000","last_became_peered":"2024-06-03T08:38:14.510665+0000","last_unstale":"2024-06-03T09:06:14.437636+0000","last_undegraded":"2024-06-03T09:06:14.437636+0000","last_fullsized":"2024-06-03T09:06:14.437636+0000","mapping_epoch":28,"log_start":"0'0","ondisk_log_start":"0'0","created":28,"last_epoch_clean":42,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-06-03T08:37:55.614497+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-06-03T08:37:55.614497+0000","last_clean_scrub_stamp":"2024-06-03T08:37:55.614497+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-06-04T12:03:47.392064+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,7],"acting":[0,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"17.0","version":"0'0","reported_seq":944,"reported_epoch":1009,"state":"active+clean","last_fresh":"2024-06-03T09:06:14.427123+0000","last_change":"2024-06-03T08:38:04.237118+0000","last_active":"2024-06-03T09:06:14.427123+0000","last_peered":"2024-06-03T09:06:14.427123+0000","last_clean":"2024-06-03T09:06:14.427123+0000","last_became_active":"2024-06-03T08:38:04.231954+0000","last_became_peered":"2024-06-03T08:38:04.231954+0000","last_unstale":"2024-06-03T09:06:14.427123+0000","last_undegraded":"2024-06-03T09:06:14.427123+0000","last_fullsized":"2024-06-03T09:06:14.427123+0000","mapping_epoch":28,"log_start":"0'0","ondisk_log_start":"0'0","created":28,"last_epoch_clean":32,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-06-03T08:37:55.614497+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-06-03T08:37:55.614497+0000","last_clean_scrub_stamp":"2024-06-03T08:37:55.614497+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-06-04T09:56:10.741637+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,2],"acting":[0,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"17.1","version":"65'7","reported_seq":959,"reported_epoch":1009,"state":"active+undersized+degraded","last_fresh":"2024-06-03T09:06:13.709209+0000","last_change":"2024-06-03T09:06:03.370660+0000","last_active":"2024-06-03T09:06:13.709209+0000","last_peered":"2024-06-03T09:06:13.709209+0000","last_clean":"2024-06-03T09:05:03.733864+0000","last_became_active":"2024-06-03T09:06:03.370660+0000","last_became_peered":"2024-06-03T09:06:03.370660+0000","last_unstale":"2024-06-03T09:06:13.709209+0000","last_undegraded":"2024-06-03T09:06:03.266076+0000","last_fullsized":"2024-06-03T09:06:03.264525+0000","mapping_epoch":999,"log_start":"0'0","ondisk_log_start":"0'0","created":28,"last_epoch_clean":255,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-06-03T08:37:55.614497+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-06-03T08:37:55.614497+0000","last_clean_scrub_stamp":"2024-06-03T08:37:55.614497+0000","objects_scrubbed":0,"log_size":7,"log_dups_size":0,"ondisk_log_size":7,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-06-04T11:55:18.194283+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":5,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":1,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":6,"num_read_kb":6,"num_write":7,"num_write_kb":7,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5],"acting":[5],"avail_no_missing":["5"],"object_location_counts":[{"shards":"5","objects":1}],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]}],"pool_stats":[{"poolid":2,"num_pg":8,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":14,"acting":14,"num_store_stats":7},{"poolid":1,"num_pg":1,"stat_sum":{"num_bytes":459280,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":96,"num_read_kb":82,"num_write":113,"num_write_kb":1372,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2,"num_bytes_recovered":397840,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":802816,"data_stored":795680,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":76,"ondisk_log_size":76,"up":2,"acting":2,"num_store_stats":2},{"poolid":17,"num_pg":32,"stat_sum":{"num_bytes":5,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":1,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":6,"num_read_kb":6,"num_write":7,"num_write_kb":7,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":8192,"data_stored":10,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":7,"ondisk_log_size":7,"up":52,"acting":52,"num_store_stats":8}],"osd_stats":[{"osd":6,"up_from":253,"seq":1086626726201,"num_pgs":20,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":54836,"kb_used_data":5672,"kb_used_omap":1,"kb_used_meta":49150,"kb_avail":94317004,"statfs":{"total":96636764160,"available":96580612096,"internally_reserved":0,"allocated":5808128,"data_stored":2801702,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1593,"internal_metadata":50330055},"hb_peers":[0,1,2,3,4,5,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":93,"apply_latency_ms":93,"commit_latency_ns":93000000,"apply_latency_ns":93000000},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Mon Jun 3 09:04:02 2024","interfaces":[{"interface":"back","average":{"1min":1582.4780000000001,"5min":332.72699999999998,"15min":124.113},"min":{"1min":4.4829999999999997,"5min":4.1959999999999997,"15min":4.1959999999999997},"max":{"1min":22766.882000000001,"5min":22766.882000000001,"15min":22766.882000000001},"last":5.5880000000000001},{"interface":"front","average":{"1min":1579.5409999999999,"5min":334.94499999999999,"15min":125.84999999999999},"min":{"1min":4.8129999999999997,"5min":4.7539999999999996,"15min":4.4530000000000003},"max":{"1min":22766.289000000001,"5min":22766.289000000001,"15min":22766.289000000001},"last":4.835}]},{"osd":1,"last update":"Mon Jun 3 09:03:40 2024","interfaces":[{"interface":"back","average":{"1min":36.420999999999999,"5min":77.054000000000002,"15min":132.84800000000001},"min":{"1min":6.0380000000000003,"5min":5.8799999999999999,"15min":5.484},"max":{"1min":589.64099999999996,"5min":4138.1679999999997,"15min":9955.1299999999992},"last":8.8469999999999995},{"interface":"front","average":{"1min":40.240000000000002,"5min":79.674999999999997,"15min":133.38999999999999},"min":{"1min":8.4589999999999996,"5min":8.4090000000000007,"15min":6.1390000000000002},"max":{"1min":588.60799999999995,"5min":4137.6530000000002,"15min":9827.5509999999995},"last":11.279999999999999}]},{"osd":2,"last update":"Mon Jun 3 09:03:40 2024","interfaces":[{"interface":"back","average":{"1min":12.613,"5min":24.600999999999999,"15min":21.088000000000001},"min":{"1min":6.9740000000000002,"5min":6.9740000000000002,"15min":6.5330000000000004},"max":{"1min":26.076000000000001,"5min":376.24900000000002,"15min":376.24900000000002},"last":13.292999999999999},{"interface":"front","average":{"1min":12.493,"5min":22.207999999999998,"15min":21.219000000000001},"min":{"1min":6.5810000000000004,"5min":6.5110000000000001,"15min":6.0750000000000002},"max":{"1min":29.690999999999999,"5min":271.51100000000002,"15min":434.77300000000002},"last":8.2739999999999991}]},{"osd":3,"last update":"Mon Jun 3 09:03:40 2024","interfaces":[{"interface":"back","average":{"1min":14.093999999999999,"5min":24.143999999999998,"15min":29.376999999999999},"min":{"1min":10.196999999999999,"5min":6.2919999999999998,"15min":6.1719999999999997},"max":{"1min":24.832000000000001,"5min":197.84899999999999,"15min":1235.5799999999999},"last":14.281000000000001},{"interface":"front","average":{"1min":9.3260000000000005,"5min":18.463999999999999,"15min":27.631},"min":{"1min":6.125,"5min":6.125,"15min":6.125},"max":{"1min":28.166,"5min":200.173,"15min":1157.6780000000001},"last":12.782999999999999}]},{"osd":4,"last update":"Mon Jun 3 09:04:29 2024","interfaces":[{"interface":"back","average":{"1min":10.279,"5min":1253.809,"15min":450.79199999999997},"min":{"1min":7.2300000000000004,"5min":6.093,"15min":6.093},"max":{"1min":15.000999999999999,"5min":28093.096000000001,"15min":28093.096000000001},"last":10.965999999999999},{"interface":"front","average":{"1min":15.693,"5min":1279.759,"15min":463.87099999999998},"min":{"1min":10.551,"5min":7.7480000000000002,"15min":7.7480000000000002},"max":{"1min":27.898,"5min":28323.837,"15min":28323.837},"last":13.581}]},{"osd":5,"last update":"Mon Jun 3 09:04:10 2024","interfaces":[{"interface":"back","average":{"1min":13.728999999999999,"5min":24.670000000000002,"15min":23.815999999999999},"min":{"1min":10.401,"5min":7.516,"15min":7.516},"max":{"1min":25.138000000000002,"5min":320.85000000000002,"15min":445.041},"last":14.928000000000001},{"interface":"front","average":{"1min":13.847,"5min":29.283000000000001,"15min":23.393999999999998},"min":{"1min":7.944,"5min":6.266,"15min":6.266},"max":{"1min":18.817,"5min":530.56700000000001,"15min":530.56700000000001},"last":16.893000000000001}]},{"osd":7,"last update":"Mon Jun 3 09:03:40 2024","interfaces":[{"interface":"back","average":{"1min":20.568999999999999,"5min":30.286999999999999,"15min":27.977},"min":{"1min":13.874000000000001,"5min":10.58,"15min":9.2859999999999996},"max":{"1min":86.959000000000003,"5min":222.07599999999999,"15min":255.08799999999999},"last":17.59},{"interface":"front","average":{"1min":19.102,"5min":29.550000000000001,"15min":27.763000000000002},"min":{"1min":10.052,"5min":8.6709999999999994,"15min":8.5429999999999993},"max":{"1min":67.968999999999994,"5min":293.62099999999998,"15min":438.14600000000002},"last":16.045000000000002}]}]},{"osd":7,"up_from":20,"seq":85899346289,"num_pgs":9,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":58964,"kb_used_data":5704,"kb_used_omap":1,"kb_used_meta":53246,"kb_avail":94312876,"statfs":{"total":96636764160,"available":96576385024,"internally_reserved":0,"allocated":5840896,"data_stored":2740727,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1589,"internal_metadata":54524363},"hb_peers":[0,1,2,3,4,5],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Mon Jun 3 09:06:09 2024","interfaces":[{"interface":"back","average":{"1min":14.073,"5min":894.22400000000005,"15min":308.93400000000003},"min":{"1min":5.5709999999999997,"5min":4.218,"15min":4.0599999999999996},"max":{"1min":90.757999999999996,"5min":21729.001,"15min":21729.001},"last":12.398},{"interface":"front","average":{"1min":9.2550000000000008,"5min":906.23500000000001,"15min":314.06},"min":{"1min":5.5789999999999997,"5min":4.8810000000000002,"15min":4.8810000000000002},"max":{"1min":25.937000000000001,"5min":21899.732,"15min":21899.732},"last":8.6609999999999996}]},{"osd":1,"last update":"Mon Jun 3 09:06:09 2024","interfaces":[{"interface":"back","average":{"1min":15.182,"5min":69.183999999999997,"15min":219.173},"min":{"1min":7.3639999999999999,"5min":6.6200000000000001,"15min":6.1319999999999997},"max":{"1min":76.926000000000002,"5min":2795.1329999999998,"15min":18373.257000000001},"last":9.1489999999999991},{"interface":"front","average":{"1min":21.495999999999999,"5min":74.703999999999994,"15min":225.285},"min":{"1min":8.8399999999999999,"5min":8.8399999999999999,"15min":8.5129999999999999},"max":{"1min":94.977999999999994,"5min":2803.3670000000002,"15min":18374.601999999999},"last":11.051}]},{"osd":2,"last update":"Mon Jun 3 09:06:09 2024","interfaces":[{"interface":"back","average":{"1min":19.164000000000001,"5min":26.902000000000001,"15min":22.920999999999999},"min":{"1min":11.207000000000001,"5min":8.5030000000000001,"15min":8.5030000000000001},"max":{"1min":91.319999999999993,"5min":333.99200000000002,"15min":333.99200000000002},"last":13.628},{"interface":"front","average":{"1min":11.125999999999999,"5min":18.132000000000001,"15min":16.353000000000002},"min":{"1min":6.202,"5min":5.8369999999999997,"15min":5.5069999999999997},"max":{"1min":28.649000000000001,"5min":226.18100000000001,"15min":226.18100000000001},"last":7.4770000000000003}]},{"osd":3,"last update":"Mon Jun 3 09:06:09 2024","interfaces":[{"interface":"back","average":{"1min":22.291,"5min":26.289999999999999,"15min":31.074999999999999},"min":{"1min":11.714,"5min":6.8959999999999999,"15min":6.3760000000000003},"max":{"1min":83.102000000000004,"5min":316.64400000000001,"15min":2039.9059999999999},"last":11.871},{"interface":"front","average":{"1min":13.324999999999999,"5min":17.963000000000001,"15min":25.969999999999999},"min":{"1min":6.1200000000000001,"5min":5.2089999999999996,"15min":5.2089999999999996},"max":{"1min":49.908999999999999,"5min":233.542,"15min":2037.5550000000001},"last":6.75}]},{"osd":4,"last update":"Mon Jun 3 09:06:29 2024","interfaces":[{"interface":"back","average":{"1min":17.675999999999998,"5min":1026.325,"15min":472.47199999999998},"min":{"1min":7.306,"5min":5.9089999999999998,"15min":5.9089999999999998},"max":{"1min":140.03299999999999,"5min":28969.369999999999,"15min":30073.917000000001},"last":5.2709999999999999},{"interface":"front","average":{"1min":17.890999999999998,"5min":1044.5139999999999,"15min":480.226},"min":{"1min":5.9279999999999999,"5min":5.9279999999999999,"15min":5.9279999999999999},"max":{"1min":141.179,"5min":29244.138999999999,"15min":30348.021000000001},"last":9.7810000000000006}]},{"osd":5,"last update":"Mon Jun 3 09:06:02 2024","interfaces":[{"interface":"back","average":{"1min":24.372,"5min":26.856000000000002,"15min":23.571999999999999},"min":{"1min":12.278,"5min":9.2460000000000004,"15min":6.5650000000000004},"max":{"1min":183.64099999999999,"5min":317.572,"15min":317.572},"last":15.048999999999999},{"interface":"front","average":{"1min":21.259,"5min":23.364000000000001,"15min":21.193999999999999},"min":{"1min":7.8499999999999996,"5min":7.0999999999999996,"15min":7.008},"max":{"1min":184.53399999999999,"5min":306.60599999999999,"15min":306.60599999999999},"last":14.843999999999999}]},{"osd":6,"last update":"Mon Jun 3 09:04:31 2024","interfaces":[{"interface":"back","average":{"1min":44.061,"5min":26.821000000000002,"15min":91.176000000000002},"min":{"1min":9.0800000000000001,"5min":7.9390000000000001,"15min":7.9390000000000001},"max":{"1min":268.029,"5min":308.916,"15min":8734.4639999999999},"last":6.6769999999999996},{"interface":"front","average":{"1min":46.643000000000001,"5min":23.879999999999999,"15min":88.765000000000001},"min":{"1min":8.8119999999999994,"5min":6.9240000000000004,"15min":6.742},"max":{"1min":305.64600000000002,"5min":305.64600000000002,"15min":8735.5319999999992},"last":12.327}]}]},{"osd":5,"up_from":20,"seq":85899346264,"num_pgs":9,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":59360,"kb_used_data":6100,"kb_used_omap":1,"kb_used_meta":53246,"kb_avail":94312480,"statfs":{"total":96636764160,"available":96575979520,"internally_reserved":0,"allocated":6246400,"data_stored":3138572,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1594,"internal_metadata":54524358},"hb_peers":[0,1,2,3,4,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Mon Jun 3 09:06:06 2024","interfaces":[{"interface":"back","average":{"1min":7.468,"5min":1703.037,"15min":580.51900000000001},"min":{"1min":4.6509999999999998,"5min":4.6509999999999998,"15min":4.4020000000000001},"max":{"1min":14.711,"5min":26433.764999999999,"15min":26433.764999999999},"last":5.8639999999999999},{"interface":"front","average":{"1min":9.9429999999999996,"5min":1725.27,"15min":587.87800000000004},"min":{"1min":5.5999999999999996,"5min":5.1319999999999997,"15min":4.8170000000000002},"max":{"1min":13.590999999999999,"5min":26704.698,"15min":26704.698},"last":12.747999999999999}]},{"osd":1,"last update":"Mon Jun 3 09:06:00 2024","interfaces":[{"interface":"back","average":{"1min":10.862,"5min":67.731999999999999,"15min":217.364},"min":{"1min":7.1769999999999996,"5min":6.0890000000000004,"15min":5.4930000000000003},"max":{"1min":16.998999999999999,"5min":3923.4090000000001,"15min":18811.696},"last":14.052},{"interface":"front","average":{"1min":10.417,"5min":65.141999999999996,"15min":220.75700000000001},"min":{"1min":5.3209999999999997,"5min":5.3209999999999997,"15min":5.0209999999999999},"max":{"1min":15.821999999999999,"5min":4019.6129999999998,"15min":18968.365000000002},"last":8.7219999999999995}]},{"osd":2,"last update":"Mon Jun 3 09:06:00 2024","interfaces":[{"interface":"back","average":{"1min":11.366,"5min":24.18,"15min":21.137},"min":{"1min":5.6369999999999996,"5min":5.6369999999999996,"15min":5.0510000000000002},"max":{"1min":15.529999999999999,"5min":332.48200000000003,"15min":332.48200000000003},"last":12.167},{"interface":"front","average":{"1min":10.089,"5min":23.117000000000001,"15min":24.050000000000001},"min":{"1min":6.6769999999999996,"5min":6.6769999999999996,"15min":5.8739999999999997},"max":{"1min":15.821,"5min":353.03199999999998,"15min":353.03199999999998},"last":7.2619999999999996}]},{"osd":3,"last update":"Mon Jun 3 09:06:00 2024","interfaces":[{"interface":"back","average":{"1min":13.472,"5min":24.283999999999999,"15min":31.666},"min":{"1min":10.111000000000001,"5min":7.4720000000000004,"15min":7.4210000000000003},"max":{"1min":16.271999999999998,"5min":353.57499999999999,"15min":2855.8519999999999},"last":8.1140000000000008},{"interface":"front","average":{"1min":9.5899999999999999,"5min":15.875,"15min":25.834},"min":{"1min":6.6609999999999996,"5min":5.7729999999999997,"15min":5.1909999999999998},"max":{"1min":15.112,"5min":203.804,"15min":2869.7860000000001},"last":9.3699999999999992}]},{"osd":4,"last update":"Mon Jun 3 09:06:29 2024","interfaces":[{"interface":"back","average":{"1min":11.067,"5min":780.21000000000004,"15min":377.28399999999999},"min":{"1min":5.4779999999999998,"5min":5.4779999999999998,"15min":5.4779999999999998},"max":{"1min":14.654999999999999,"5min":23315.438999999998,"15min":28619.825000000001},"last":6.7199999999999998},{"interface":"front","average":{"1min":14.349,"5min":782.97299999999996,"15min":377.93400000000003},"min":{"1min":7.492,"5min":6.5030000000000001,"15min":6.5030000000000001},"max":{"1min":17.707000000000001,"5min":23317.502,"15min":28621.294000000002},"last":11.621}]},{"osd":6,"last update":"Mon Jun 3 09:03:57 2024","interfaces":[{"interface":"back","average":{"1min":23.536000000000001,"5min":36.457000000000001,"15min":83.799999999999997},"min":{"1min":9.4800000000000004,"5min":9.4800000000000004,"15min":8.0630000000000006},"max":{"1min":89.442999999999998,"5min":508.77999999999997,"15min":9193.5169999999998},"last":157.79300000000001},{"interface":"front","average":{"1min":18.891999999999999,"5min":22.100999999999999,"15min":75.489999999999995},"min":{"1min":5.923,"5min":5.923,"15min":5.923},"max":{"1min":78.424999999999997,"5min":464.62,"15min":9191.4429999999993},"last":148.09}]},{"osd":7,"last update":"Mon Jun 3 09:06:00 2024","interfaces":[{"interface":"back","average":{"1min":13.257,"5min":20.917999999999999,"15min":21.088999999999999},"min":{"1min":10.475,"5min":6.7629999999999999,"15min":6.7629999999999999},"max":{"1min":17.245000000000001,"5min":211.22999999999999,"15min":211.22999999999999},"last":11.061999999999999},{"interface":"front","average":{"1min":12.872999999999999,"5min":20.587,"15min":24.021000000000001},"min":{"1min":8.5760000000000005,"5min":7.7709999999999999,"15min":6.8570000000000002},"max":{"1min":18.359999999999999,"5min":223.191,"15min":286.69499999999999},"last":10.021000000000001}]}]},{"osd":4,"up_from":20,"seq":85899346271,"num_pgs":12,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":63060,"kb_used_data":5704,"kb_used_omap":1,"kb_used_meta":57342,"kb_avail":94308780,"statfs":{"total":96636764160,"available":96572190720,"internally_reserved":0,"allocated":5840896,"data_stored":2740727,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1594,"internal_metadata":58718662},"hb_peers":[0,1,2,3,5,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Mon Jun 3 09:06:35 2024","interfaces":[{"interface":"back","average":{"1min":10.417,"5min":1093.547,"15min":381.64499999999998},"min":{"1min":4.6900000000000004,"5min":4.5279999999999996,"15min":4.3220000000000001},"max":{"1min":52.241,"5min":21726.743999999999,"15min":21726.743999999999},"last":7.0030000000000001},{"interface":"front","average":{"1min":13.175000000000001,"5min":1102.4200000000001,"15min":387.161},"min":{"1min":6.1950000000000003,"5min":5.8380000000000001,"15min":5.4909999999999997},"max":{"1min":53.148000000000003,"5min":21727.993999999999,"15min":21727.993999999999},"last":12.923999999999999}]},{"osd":1,"last update":"Mon Jun 3 09:06:35 2024","interfaces":[{"interface":"back","average":{"1min":9.9909999999999997,"5min":53.250999999999998,"15min":340.02999999999997},"min":{"1min":4.5670000000000002,"5min":4.1970000000000001,"15min":4.1970000000000001},"max":{"1min":47.073,"5min":3346.4389999999999,"15min":22256.098000000002},"last":15.198},{"interface":"front","average":{"1min":12.189,"5min":53.375999999999998,"15min":339.68799999999999},"min":{"1min":6.1539999999999999,"5min":5.0590000000000002,"15min":4.2640000000000002},"max":{"1min":49.558,"5min":3347.181,"15min":22247.317999999999},"last":7.681}]},{"osd":2,"last update":"Mon Jun 3 09:06:35 2024","interfaces":[{"interface":"back","average":{"1min":15.178000000000001,"5min":17.202000000000002,"15min":20.417999999999999},"min":{"1min":6.3010000000000002,"5min":6.3010000000000002,"15min":6.3010000000000002},"max":{"1min":85.340000000000003,"5min":337.60399999999998,"15min":393.58600000000001},"last":12.378},{"interface":"front","average":{"1min":15.423,"5min":14.432,"15min":22.754000000000001},"min":{"1min":6.4889999999999999,"5min":6.0629999999999997,"15min":6.0629999999999997},"max":{"1min":82.120000000000005,"5min":127.913,"15min":394.471},"last":10.843999999999999}]},{"osd":3,"last update":"Mon Jun 3 09:06:35 2024","interfaces":[{"interface":"back","average":{"1min":17.651,"5min":15.5,"15min":18.079000000000001},"min":{"1min":10.359999999999999,"5min":5.8719999999999999,"15min":5.4000000000000004},"max":{"1min":84.555999999999997,"5min":127.43600000000001,"15min":322.63},"last":14.564},{"interface":"front","average":{"1min":16.413,"5min":17.356999999999999,"15min":22.920000000000002},"min":{"1min":9.7349999999999994,"5min":7.484,"15min":7.484},"max":{"1min":71.394000000000005,"5min":204.517,"15min":392.67500000000001},"last":13.446999999999999}]},{"osd":5,"last update":"Mon Jun 3 09:06:35 2024","interfaces":[{"interface":"back","average":{"1min":15.672000000000001,"5min":16.742999999999999,"15min":20.795000000000002},"min":{"1min":8.093,"5min":6.7489999999999997,"15min":6.7489999999999997},"max":{"1min":51.326000000000001,"5min":268.10199999999998,"15min":344.23399999999998},"last":9.2910000000000004},{"interface":"front","average":{"1min":17.297999999999998,"5min":19.356999999999999,"15min":25.713999999999999},"min":{"1min":12.875,"5min":8.5180000000000007,"15min":8.5180000000000007},"max":{"1min":40.829000000000001,"5min":264.75,"15min":345.18700000000001},"last":10.318}]},{"osd":6,"last update":"Mon Jun 3 09:04:32 2024","interfaces":[{"interface":"back","average":{"1min":15.548,"5min":32.957999999999998,"15min":152.738},"min":{"1min":9.141,"5min":9.141,"15min":8.2319999999999993},"max":{"1min":52.57,"5min":395.238,"15min":12496.335999999999},"last":208.90000000000001},{"interface":"front","average":{"1min":12.680999999999999,"5min":25.428000000000001,"15min":149.30600000000001},"min":{"1min":6.4379999999999997,"5min":5.6319999999999997,"15min":5.6319999999999997},"max":{"1min":46.283000000000001,"5min":377.74299999999999,"15min":12616.297},"last":205.88900000000001}]},{"osd":7,"last update":"Mon Jun 3 09:06:35 2024","interfaces":[{"interface":"back","average":{"1min":9.6969999999999992,"5min":14.196,"15min":18.809999999999999},"min":{"1min":6.7190000000000003,"5min":6.2110000000000003,"15min":5.8010000000000002},"max":{"1min":15.960000000000001,"5min":207.25299999999999,"15min":358.94900000000001},"last":8.7810000000000006},{"interface":"front","average":{"1min":17.696000000000002,"5min":19.024999999999999,"15min":23.259},"min":{"1min":11.794,"5min":7.7789999999999999,"15min":7.7789999999999999},"max":{"1min":45.639000000000003,"5min":209.32900000000001,"15min":380.06900000000002},"last":11.362}]}]},{"osd":2,"up_from":20,"seq":85899346291,"num_pgs":11,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":58964,"kb_used_data":5704,"kb_used_omap":1,"kb_used_meta":53246,"kb_avail":94312876,"statfs":{"total":96636764160,"available":96576385024,"internally_reserved":0,"allocated":5840896,"data_stored":2740727,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1594,"internal_metadata":54524358},"hb_peers":[0,1,3,4,5,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Mon Jun 3 09:06:23 2024","interfaces":[{"interface":"back","average":{"1min":8.9629999999999992,"5min":946.25400000000002,"15min":326.36099999999999},"min":{"1min":5.3479999999999999,"5min":4.7050000000000001,"15min":4.1420000000000003},"max":{"1min":26.015000000000001,"5min":23465.18,"15min":23465.18},"last":7.5270000000000001},{"interface":"front","average":{"1min":7.6130000000000004,"5min":944.39800000000002,"15min":326.178},"min":{"1min":4.6790000000000003,"5min":4.6790000000000003,"15min":4.6790000000000003},"max":{"1min":20.802,"5min":23465.966,"15min":23465.966},"last":9.0079999999999991}]},{"osd":1,"last update":"Mon Jun 3 09:06:23 2024","interfaces":[{"interface":"back","average":{"1min":13.007,"5min":76.507000000000005,"15min":294.72899999999998},"min":{"1min":6.4089999999999998,"5min":5.4199999999999999,"15min":5.4199999999999999},"max":{"1min":85.430000000000007,"5min":4220.5290000000005,"15min":18954.987000000001},"last":6.4100000000000001},{"interface":"front","average":{"1min":15.894,"5min":82.119,"15min":302.92399999999998},"min":{"1min":7.367,"5min":7.367,"15min":7.367},"max":{"1min":98.498000000000005,"5min":4325.9489999999996,"15min":19143.834999999999},"last":10.802}]},{"osd":3,"last update":"Mon Jun 3 09:06:23 2024","interfaces":[{"interface":"back","average":{"1min":9.7579999999999991,"5min":12.444000000000001,"15min":28.065000000000001},"min":{"1min":7,"5min":5.9290000000000003,"15min":5.2450000000000001},"max":{"1min":28.164000000000001,"5min":118.913,"15min":2594.8150000000001},"last":10.282},{"interface":"front","average":{"1min":12.321999999999999,"5min":15.25,"15min":31.349},"min":{"1min":9.1240000000000006,"5min":8.0760000000000005,"15min":6.0780000000000003},"max":{"1min":32.360999999999997,"5min":156.34700000000001,"15min":2706.5120000000002},"last":15.382999999999999}]},{"osd":4,"last update":"Mon Jun 3 09:05:57 2024","interfaces":[{"interface":"back","average":{"1min":10.573,"5min":1266.894,"15min":438.154},"min":{"1min":8.6379999999999999,"5min":7.8789999999999996,"15min":6.2039999999999997},"max":{"1min":15.105,"5min":26660.861000000001,"15min":26660.861000000001},"last":11.968999999999999},{"interface":"front","average":{"1min":12.407999999999999,"5min":1266.7460000000001,"15min":435.75400000000002},"min":{"1min":10.266999999999999,"5min":10.266999999999999,"15min":6.2720000000000002},"max":{"1min":15.228,"5min":26616.243999999999,"15min":26616.243999999999},"last":14.516999999999999}]},{"osd":5,"last update":"Mon Jun 3 09:06:23 2024","interfaces":[{"interface":"back","average":{"1min":14.103,"5min":18.091000000000001,"15min":21.119},"min":{"1min":9.6349999999999998,"5min":9.2159999999999993,"15min":6.2050000000000001},"max":{"1min":32.890000000000001,"5min":252.31100000000001,"15min":252.31100000000001},"last":13.661},{"interface":"front","average":{"1min":11.545999999999999,"5min":16.170999999999999,"15min":20.254000000000001},"min":{"1min":7.6070000000000002,"5min":7.4850000000000003,"15min":6.8810000000000002},"max":{"1min":29.228999999999999,"5min":258.31700000000001,"15min":258.31700000000001},"last":8.4199999999999999}]},{"osd":6,"last update":"Mon Jun 3 09:04:28 2024","interfaces":[{"interface":"back","average":{"1min":14.063000000000001,"5min":24.850000000000001,"15min":106.755},"min":{"1min":9.0820000000000007,"5min":8.9649999999999999,"15min":8.9649999999999999},"max":{"1min":24.666,"5min":188.06700000000001,"15min":9521.0010000000002},"last":183.60900000000001},{"interface":"front","average":{"1min":13.933999999999999,"5min":21.081,"15min":106.142},"min":{"1min":5.8719999999999999,"5min":5.8719999999999999,"15min":5.8719999999999999},"max":{"1min":117.551,"5min":193.887,"15min":9585.6620000000003},"last":184.20599999999999}]},{"osd":7,"last update":"Mon Jun 3 09:06:23 2024","interfaces":[{"interface":"back","average":{"1min":17.056000000000001,"5min":20.055,"15min":23.268999999999998},"min":{"1min":10.353999999999999,"5min":6.6130000000000004,"15min":6.6130000000000004},"max":{"1min":88.974999999999994,"5min":242.56800000000001,"15min":242.56800000000001},"last":12.833},{"interface":"front","average":{"1min":18.885000000000002,"5min":21.460000000000001,"15min":25.489999999999998},"min":{"1min":12.497,"5min":9.0660000000000007,"15min":9.0660000000000007},"max":{"1min":102.229,"5min":181.56399999999999,"15min":214.33799999999999},"last":16.236000000000001}]}]},{"osd":3,"up_from":14,"seq":60129542505,"num_pgs":11,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":58964,"kb_used_data":5704,"kb_used_omap":1,"kb_used_meta":53246,"kb_avail":94312876,"statfs":{"total":96636764160,"available":96576385024,"internally_reserved":0,"allocated":5840896,"data_stored":2740727,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1592,"internal_metadata":54524360},"hb_peers":[0,1,2,4,5,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Mon Jun 3 09:06:25 2024","interfaces":[{"interface":"back","average":{"1min":7.79,"5min":1060.1959999999999,"15min":367.01799999999997},"min":{"1min":5.6909999999999998,"5min":5.173,"15min":4.6929999999999996},"max":{"1min":10.59,"5min":23275.167000000001,"15min":23275.167000000001},"last":11.048999999999999},{"interface":"front","average":{"1min":11.215999999999999,"5min":1066.5799999999999,"15min":372.476},"min":{"1min":8.7080000000000002,"5min":7.2510000000000003,"15min":5.9880000000000004},"max":{"1min":13.499000000000001,"5min":23278.536,"15min":23278.536},"last":13.255000000000001}]},{"osd":1,"last update":"Mon Jun 3 09:06:25 2024","interfaces":[{"interface":"back","average":{"1min":7.3760000000000003,"5min":15.614000000000001,"15min":389.83100000000002},"min":{"1min":4.335,"5min":4.335,"15min":4.335},"max":{"1min":11.289,"5min":250.72300000000001,"15min":21658.205999999998},"last":6.0979999999999999},{"interface":"front","average":{"1min":8.9499999999999993,"5min":14.135,"15min":381.10300000000001},"min":{"1min":6.5609999999999999,"5min":5.4100000000000001,"15min":5.2779999999999996},"max":{"1min":14.51,"5min":180.44499999999999,"15min":21307.069},"last":8.6020000000000003}]},{"osd":2,"last update":"Mon Jun 3 09:06:25 2024","interfaces":[{"interface":"back","average":{"1min":12.343999999999999,"5min":22.059999999999999,"15min":23.715},"min":{"1min":7.9199999999999999,"5min":6.4900000000000002,"15min":6.056},"max":{"1min":15.601000000000001,"5min":377.803,"15min":377.803},"last":12.279999999999999},{"interface":"front","average":{"1min":10.319000000000001,"5min":19.364999999999998,"15min":20.242999999999999},"min":{"1min":6.891,"5min":6.1749999999999998,"15min":5.9020000000000001},"max":{"1min":15.432,"5min":371.601,"15min":371.601},"last":6.9720000000000004}]},{"osd":4,"last update":"Mon Jun 3 09:06:31 2024","interfaces":[{"interface":"back","average":{"1min":9.7460000000000004,"5min":891.23000000000002,"15min":416.48099999999999},"min":{"1min":5.6539999999999999,"5min":5.6539999999999999,"15min":5.6539999999999999},"max":{"1min":15.445,"5min":29040.635999999999,"15min":30144.939999999999},"last":7.7599999999999998},{"interface":"front","average":{"1min":12.398999999999999,"5min":897.58500000000004,"15min":421.27499999999998},"min":{"1min":8.9469999999999992,"5min":7.8109999999999999,"15min":6.2439999999999998},"max":{"1min":16.518000000000001,"5min":29076.955999999998,"15min":30180.689999999999},"last":11.598000000000001}]},{"osd":5,"last update":"Mon Jun 3 09:06:07 2024","interfaces":[{"interface":"back","average":{"1min":12.802,"5min":19.640999999999998,"15min":22.318000000000001},"min":{"1min":9.0890000000000004,"5min":6.8810000000000002,"15min":6.7690000000000001},"max":{"1min":16.745999999999999,"5min":249.88,"15min":327.34899999999999},"last":14.084},{"interface":"front","average":{"1min":15.002000000000001,"5min":24.003,"15min":26.649999999999999},"min":{"1min":11.984,"5min":9.9149999999999991,"15min":8.9879999999999995},"max":{"1min":18.407,"5min":251.47800000000001,"15min":331.86799999999999},"last":14.791}]},{"osd":6,"last update":"Mon Jun 3 09:04:04 2024","interfaces":[{"interface":"back","average":{"1min":22.815999999999999,"5min":31.513999999999999,"15min":181.74299999999999},"min":{"1min":9.4809999999999999,"5min":7.2240000000000002,"15min":7.2240000000000002},"max":{"1min":160.56999999999999,"5min":380.613,"15min":11893.74},"last":19.359999999999999},{"interface":"front","average":{"1min":25.359999999999999,"5min":30.981000000000002,"15min":179.81399999999999},"min":{"1min":9.8870000000000005,"5min":7.3360000000000003,"15min":6.0899999999999999},"max":{"1min":193.11099999999999,"5min":379.80700000000002,"15min":11893.069},"last":8.202}]},{"osd":7,"last update":"Mon Jun 3 09:06:25 2024","interfaces":[{"interface":"back","average":{"1min":11.321999999999999,"5min":18.443999999999999,"15min":21.568999999999999},"min":{"1min":8.109,"5min":8.109,"15min":7.4279999999999999},"max":{"1min":17.113,"5min":146.267,"15min":178.76300000000001},"last":9.4009999999999998},{"interface":"front","average":{"1min":12.512,"5min":20.065999999999999,"15min":24.190000000000001},"min":{"1min":7.4989999999999997,"5min":7.4989999999999997,"15min":7.4989999999999997},"max":{"1min":18.047999999999998,"5min":148.13800000000001,"15min":417.22800000000001},"last":10.007}]}]},{"osd":1,"up_from":14,"seq":60129542503,"num_pgs":6,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":59356,"kb_used_data":6096,"kb_used_omap":1,"kb_used_meta":53246,"kb_avail":94312484,"statfs":{"total":96636764160,"available":96575983616,"internally_reserved":0,"allocated":6242304,"data_stored":3138567,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":54524357},"hb_peers":[0,2,3,4,5,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Mon Jun 3 09:06:07 2024","interfaces":[{"interface":"back","average":{"1min":7.7530000000000001,"5min":185.66999999999999,"15min":77.347999999999999},"min":{"1min":5.0229999999999997,"5min":4.9279999999999999,"15min":4.5449999999999999},"max":{"1min":11.896000000000001,"5min":7470.616,"15min":7470.616},"last":4.9960000000000004},{"interface":"front","average":{"1min":9.9220000000000006,"5min":187.14599999999999,"15min":76.325999999999993},"min":{"1min":5.3159999999999998,"5min":5.3159999999999998,"15min":5.3159999999999998},"max":{"1min":18.338000000000001,"5min":7477.0829999999996,"15min":7477.0829999999996},"last":12.051}]},{"osd":2,"last update":"Mon Jun 3 09:06:20 2024","interfaces":[{"interface":"back","average":{"1min":10.340999999999999,"5min":17.369,"15min":23.670999999999999},"min":{"1min":6.9740000000000002,"5min":5.8650000000000002,"15min":5.8650000000000002},"max":{"1min":15.452,"5min":228,"15min":657.38},"last":8.8550000000000004},{"interface":"front","average":{"1min":13.387,"5min":23.355,"15min":31.524000000000001},"min":{"1min":9.077,"5min":9.077,"15min":7.2160000000000002},"max":{"1min":19.701000000000001,"5min":229.465,"15min":700.06500000000005},"last":12.859999999999999}]},{"osd":3,"last update":"Mon Jun 3 09:06:04 2024","interfaces":[{"interface":"back","average":{"1min":8.6110000000000007,"5min":15.733000000000001,"15min":26.646999999999998},"min":{"1min":5.827,"5min":5.4640000000000004,"15min":5.4640000000000004},"max":{"1min":13.105,"5min":121.619,"15min":1676.009},"last":8.6129999999999995},{"interface":"front","average":{"1min":9.5850000000000009,"5min":16.161999999999999,"15min":29.065999999999999},"min":{"1min":6.3630000000000004,"5min":5.7699999999999996,"15min":5.7699999999999996},"max":{"1min":15.166,"5min":132.036,"15min":1766.5840000000001},"last":7.2569999999999997}]},{"osd":4,"last update":"Mon Jun 3 09:06:35 2024","interfaces":[{"interface":"back","average":{"1min":8.6400000000000006,"5min":641.85500000000002,"15min":334.71499999999997},"min":{"1min":6.7640000000000002,"5min":6.1760000000000002,"15min":6.0970000000000004},"max":{"1min":11.811,"5min":26340.968000000001,"15min":28044.862000000001},"last":6.7640000000000002},{"interface":"front","average":{"1min":12.361000000000001,"5min":648.59900000000005,"15min":338.48500000000001},"min":{"1min":9.8949999999999996,"5min":9.3940000000000001,"15min":6.4640000000000004},"max":{"1min":15.507,"5min":26342.311000000002,"15min":28045.624},"last":10.343}]},{"osd":5,"last update":"Mon Jun 3 09:06:04 2024","interfaces":[{"interface":"back","average":{"1min":12.478999999999999,"5min":20.228000000000002,"15min":25.997},"min":{"1min":8.6430000000000007,"5min":7.9720000000000004,"15min":7.234},"max":{"1min":17.442,"5min":139.43199999999999,"15min":701.22000000000003},"last":11.173},{"interface":"front","average":{"1min":13.507999999999999,"5min":25.760999999999999,"15min":27.553999999999998},"min":{"1min":10.042999999999999,"5min":7.258,"15min":6.242},"max":{"1min":20.611000000000001,"5min":278.63,"15min":598.35000000000002},"last":10.097}]},{"osd":6,"last update":"Mon Jun 3 09:03:54 2024","interfaces":[{"interface":"back","average":{"1min":28.524000000000001,"5min":20.654,"15min":23.878},"min":{"1min":6.0309999999999997,"5min":6.0309999999999997,"15min":6.0309999999999997},"max":{"1min":167.221,"5min":167.221,"15min":704.245},"last":18.721},{"interface":"front","average":{"1min":27.606000000000002,"5min":30.131,"15min":28.628},"min":{"1min":9.8650000000000002,"5min":9.7200000000000006,"15min":8.3580000000000005},"max":{"1min":123.938,"5min":358.23000000000002,"15min":634.05499999999995},"last":173.113}]},{"osd":7,"last update":"Mon Jun 3 09:06:20 2024","interfaces":[{"interface":"back","average":{"1min":14.66,"5min":23.234999999999999,"15min":31.097000000000001},"min":{"1min":11.250999999999999,"5min":8.0559999999999992,"15min":8.0559999999999992},"max":{"1min":21.507999999999999,"5min":270.95100000000002,"15min":703.07799999999997},"last":13.523999999999999},{"interface":"front","average":{"1min":15.548,"5min":28.760000000000002,"15min":28.925000000000001},"min":{"1min":12.759,"5min":8.2089999999999996,"15min":7.7599999999999998},"max":{"1min":22.408000000000001,"5min":360.99900000000002,"15min":608.44100000000003},"last":14.153}]}]},{"osd":0,"up_from":14,"seq":60129542505,"num_pgs":10,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":81108,"kb_used_data":5704,"kb_used_omap":1,"kb_used_meta":75390,"kb_avail":94290732,"statfs":{"total":96636764160,"available":96553709568,"internally_reserved":0,"allocated":5840896,"data_stored":2740727,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1593,"internal_metadata":77199815},"hb_peers":[1,2,3,4,5,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[{"osd":1,"last update":"Mon Jun 3 09:06:04 2024","interfaces":[{"interface":"back","average":{"1min":6.9960000000000004,"5min":56.997,"15min":339.29700000000003},"min":{"1min":4.4530000000000003,"5min":4.4530000000000003,"15min":4.4100000000000001},"max":{"1min":11.720000000000001,"5min":2414.4749999999999,"15min":20691.453000000001},"last":5.891},{"interface":"front","average":{"1min":9.548,"5min":57.258000000000003,"15min":337.791},"min":{"1min":5.2300000000000004,"5min":5.2300000000000004,"15min":4.6050000000000004},"max":{"1min":15.337999999999999,"5min":2412.9549999999999,"15min":20640.792000000001},"last":25.372}]},{"osd":2,"last update":"Mon Jun 3 09:06:22 2024","interfaces":[{"interface":"back","average":{"1min":11.154,"5min":26.515999999999998,"15min":22.882000000000001},"min":{"1min":5.7869999999999999,"5min":5.766,"15min":5.766},"max":{"1min":72.269999999999996,"5min":627.40700000000004,"15min":627.40700000000004},"last":22.283999999999999},{"interface":"front","average":{"1min":14.353999999999999,"5min":33.304000000000002,"15min":29.885000000000002},"min":{"1min":8.2149999999999999,"5min":7.8630000000000004,"15min":7.4089999999999998},"max":{"1min":73.418000000000006,"5min":635.62599999999998,"15min":635.62599999999998},"last":29.343}]},{"osd":3,"last update":"Mon Jun 3 09:06:04 2024","interfaces":[{"interface":"back","average":{"1min":12.105,"5min":26.925000000000001,"15min":26.584},"min":{"1min":7.8410000000000002,"5min":7.1280000000000001,"15min":6.1909999999999998},"max":{"1min":18.425999999999998,"5min":389.72699999999998,"15min":389.72699999999998},"last":27.023},{"interface":"front","average":{"1min":13.794,"5min":29.771999999999998,"15min":28.815000000000001},"min":{"1min":10.084,"5min":9.4920000000000009,"15min":6.1849999999999996},"max":{"1min":17.373999999999999,"5min":390.23200000000003,"15min":494.92899999999997},"last":31.582999999999998}]},{"osd":4,"last update":"Mon Jun 3 09:06:27 2024","interfaces":[{"interface":"back","average":{"1min":11.352,"5min":745.30600000000004,"15min":363.48099999999999},"min":{"1min":6.415,"5min":6.415,"15min":6.415},"max":{"1min":16.724,"5min":20248.616000000002,"15min":25555.212},"last":26.516999999999999},{"interface":"front","average":{"1min":13.545,"5min":744.76999999999998,"15min":361.03100000000001},"min":{"1min":9.3889999999999993,"5min":8.8940000000000001,"15min":7.1429999999999998},"max":{"1min":18.411000000000001,"5min":20250.195,"15min":25556.159},"last":27.510000000000002}]},{"osd":5,"last update":"Mon Jun 3 09:06:17 2024","interfaces":[{"interface":"back","average":{"1min":9.7040000000000006,"5min":26.533999999999999,"15min":28.683},"min":{"1min":6.8129999999999997,"5min":6.8129999999999997,"15min":6.8129999999999997},"max":{"1min":14.942,"5min":455.78699999999998,"15min":605.05899999999997},"last":22.044},{"interface":"front","average":{"1min":14.279,"5min":28.398,"15min":26.192},"min":{"1min":11.696,"5min":6.4420000000000002,"15min":6.2030000000000003},"max":{"1min":17.879000000000001,"5min":391.31200000000001,"15min":541.11000000000001},"last":29.972000000000001}]},{"osd":6,"last update":"Mon Jun 3 09:04:30 2024","interfaces":[{"interface":"back","average":{"1min":52.496000000000002,"5min":33.139000000000003,"15min":160.83199999999999},"min":{"1min":7.5289999999999999,"5min":7.1040000000000001,"15min":6.3739999999999997},"max":{"1min":457.91699999999997,"5min":457.91699999999997,"15min":11115.258},"last":18.959},{"interface":"front","average":{"1min":49.670000000000002,"5min":33.704999999999998,"15min":162.791},"min":{"1min":12.247999999999999,"5min":11.019,"15min":9.9760000000000009},"max":{"1min":407.05099999999999,"5min":407.05099999999999,"15min":11092.194},"last":17.707000000000001}]},{"osd":7,"last update":"Mon Jun 3 09:06:22 2024","interfaces":[{"interface":"back","average":{"1min":11.625999999999999,"5min":26.375,"15min":28.972999999999999},"min":{"1min":6.4480000000000004,"5min":6.4480000000000004,"15min":6.4480000000000004},"max":{"1min":15.352,"5min":461.41699999999997,"15min":598.33699999999999},"last":26.029},{"interface":"front","average":{"1min":14.013,"5min":29.696000000000002,"15min":33.630000000000003},"min":{"1min":10.938000000000001,"5min":10.58,"15min":10.263999999999999},"max":{"1min":16.603000000000002,"5min":463.10599999999999,"15min":592.80499999999995},"last":30.530000000000001}]}]}],"pool_statfs":[{"poolid":1,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":401408,"data_stored":397840,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":5,"total":0,"available":0,"internally_reserved":0,"allocated":401408,"data_stored":397840,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":4,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":5,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":6,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":17,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":17,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":17,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":17,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":17,"osd":4,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":17,"osd":5,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":5,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":17,"osd":6,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":5,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":17,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0}]}} 2024-06-03T09:06:47.723 DEBUG:teuthology.orchestra.run.smithi088:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph pg dump --format=json 2024-06-03T09:06:47.822 INFO:teuthology.orchestra.run.smithi088.stderr:2024-06-03T09:06:47.822+0000 7fceea937640 10 monclient: get_monmap_and_config 2024-06-03T09:06:47.822 INFO:teuthology.orchestra.run.smithi088.stderr:2024-06-03T09:06:47.822+0000 7fceea937640 10 monclient: build_initial_monmap 2024-06-03T09:06:47.823 INFO:teuthology.orchestra.run.smithi088.stderr:2024-06-03T09:06:47.822+0000 7fceea937640 10 monclient: monmap: 2024-06-03T09:06:47.823 INFO:teuthology.orchestra.run.smithi088.stderr:epoch 0 2024-06-03T09:06:47.823 INFO:teuthology.orchestra.run.smithi088.stderr:fsid ec3c3fdd-3ac6-4d7d-9412-683130ed8b04 2024-06-03T09:06:47.823 INFO:teuthology.orchestra.run.smithi088.stderr:last_changed 2024-06-03T09:06:47.823616+0000 2024-06-03T09:06:47.823 INFO:teuthology.orchestra.run.smithi088.stderr:created 2024-06-03T09:06:47.823616+0000 2024-06-03T09:06:47.823 INFO:teuthology.orchestra.run.smithi088.stderr:min_mon_release 0 (unknown) 2024-06-03T09:06:47.823 INFO:teuthology.orchestra.run.smithi088.stderr:election_strategy: 3 2024-06-03T09:06:47.823 INFO:teuthology.orchestra.run.smithi088.stderr:0: v1:172.21.15.88:6789/0 mon.noname-a 2024-06-03T09:06:47.823 INFO:teuthology.orchestra.run.smithi088.stderr:1: v1:172.21.15.192:6789/0 mon.noname-c 2024-06-03T09:06:47.823 INFO:teuthology.orchestra.run.smithi088.stderr:2: v1:172.21.15.88:6790/0 mon.noname-b 2024-06-03T09:06:47.823 INFO:teuthology.orchestra.run.smithi088.stderr: 2024-06-03T09:06:47.824 INFO:teuthology.orchestra.run.smithi088.stderr:2024-06-03T09:06:47.823+0000 7fceea937640 1 Processor -- start 2024-06-03T09:06:47.824 INFO:teuthology.orchestra.run.smithi088.stderr:2024-06-03T09:06:47.823+0000 7fceea937640 1 -- start start 2024-06-03T09:06:47.825 INFO:teuthology.orchestra.run.smithi088.stderr:2024-06-03T09:06:47.823+0000 7fceea937640 10 monclient: init 2024-06-03T09:06:47.825 INFO:teuthology.orchestra.run.smithi088.stderr:2024-06-03T09:06:47.823+0000 7fceea937640 10 monclient: _reopen_session rank -1 2024-06-03T09:06:47.825 INFO:teuthology.orchestra.run.smithi088.stderr:2024-06-03T09:06:47.823+0000 7fceea937640 10 monclient: _add_conns ranks=[1,0,2] 2024-06-03T09:06:47.825 INFO:teuthology.orchestra.run.smithi088.stderr:2024-06-03T09:06:47.823+0000 7fceea937640 10 monclient(hunting): picked mon.noname-c con 0x7fcee40a8960 addr v1:172.21.15.192:6789/0 2024-06-03T09:06:47.825 INFO:teuthology.orchestra.run.smithi088.stderr:2024-06-03T09:06:47.823+0000 7fceea937640 10 monclient(hunting): picked mon.noname-a con 0x7fcee41450f0 addr v1:172.21.15.88:6789/0 2024-06-03T09:06:47.825 INFO:teuthology.orchestra.run.smithi088.stderr:2024-06-03T09:06:47.824+0000 7fceea937640 10 monclient(hunting): picked mon.noname-b con 0x7fcee413eec0 addr v1:172.21.15.88:6790/0 2024-06-03T09:06:47.825 INFO:teuthology.orchestra.run.smithi088.stderr:2024-06-03T09:06:47.824+0000 7fceea937640 1 -- --> v1:172.21.15.88:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fcee4144c60 con 0x7fcee41450f0 2024-06-03T09:06:47.825 INFO:teuthology.orchestra.run.smithi088.stderr:2024-06-03T09:06:47.824+0000 7fceea937640 1 -- --> v1:172.21.15.192:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fcee40b3930 con 0x7fcee40a8960 2024-06-03T09:06:47.825 INFO:teuthology.orchestra.run.smithi088.stderr:2024-06-03T09:06:47.824+0000 7fceea937640 1 -- --> v1:172.21.15.88:6790/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fcee40b1ad0 con 0x7fcee413eec0 2024-06-03T09:06:47.825 INFO:teuthology.orchestra.run.smithi088.stderr:2024-06-03T09:06:47.824+0000 7fceea937640 10 monclient(hunting): _renew_subs 2024-06-03T09:06:47.825 INFO:teuthology.orchestra.run.smithi088.stderr:2024-06-03T09:06:47.824+0000 7fceea937640 10 monclient(hunting): authenticate will time out at 3063.633986s 2024-06-03T09:06:47.825 INFO:teuthology.orchestra.run.smithi088.stderr:2024-06-03T09:06:47.824+0000 7fcee8ed4640 1 --1- >> v1:172.21.15.88:6790/0 conn(0x7fcee413eec0 0x7fcee414ce60 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.88:6790/0 says I am v1:172.21.15.88:52298/0 (socket says 172.21.15.88:52298) 2024-06-03T09:06:47.825 INFO:teuthology.orchestra.run.smithi088.stderr:2024-06-03T09:06:47.824+0000 7fcee8ed4640 1 -- 172.21.15.88:0/1939170253 learned_addr learned my addr 172.21.15.88:0/1939170253 (peer_addr_for_me v1:172.21.15.88:0/0) 2024-06-03T09:06:47.827 INFO:teuthology.orchestra.run.smithi088.stderr:2024-06-03T09:06:47.826+0000 7fcee2ffd640 1 -- 172.21.15.88:0/1939170253 <== mon.2 v1:172.21.15.88:6790/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 3651996009 0 0) 0x7fcee40b1ad0 con 0x7fcee413eec0 2024-06-03T09:06:47.827 INFO:teuthology.orchestra.run.smithi088.stderr:2024-06-03T09:06:47.826+0000 7fcee2ffd640 10 monclient(hunting): _negotiate 2024-06-03T09:06:47.827 INFO:teuthology.orchestra.run.smithi088.stderr:2024-06-03T09:06:47.826+0000 7fcee2ffd640 10 monclient(hunting): _init_auth method 2 2024-06-03T09:06:47.827 INFO:teuthology.orchestra.run.smithi088.stderr:2024-06-03T09:06:47.826+0000 7fcee2ffd640 10 monclient(hunting): _init_auth creating new auth 2024-06-03T09:06:47.827 INFO:teuthology.orchestra.run.smithi088.stderr:2024-06-03T09:06:47.826+0000 7fcee2ffd640 10 monclient(hunting): my global_id is 30200 2024-06-03T09:06:47.827 INFO:teuthology.orchestra.run.smithi088.stderr:2024-06-03T09:06:47.826+0000 7fcee2ffd640 1 -- 172.21.15.88:0/1939170253 --> v1:172.21.15.88:6790/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7fcecc003520 con 0x7fcee413eec0 2024-06-03T09:06:47.828 INFO:teuthology.orchestra.run.smithi088.stderr:2024-06-03T09:06:47.827+0000 7fcee2ffd640 1 -- 172.21.15.88:0/1939170253 <== mon.0 v1:172.21.15.88:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 3626117997 0 0) 0x7fcee4144c60 con 0x7fcee41450f0 2024-06-03T09:06:47.828 INFO:teuthology.orchestra.run.smithi088.stderr:2024-06-03T09:06:47.827+0000 7fcee2ffd640 10 monclient(hunting): _negotiate 2024-06-03T09:06:47.828 INFO:teuthology.orchestra.run.smithi088.stderr:2024-06-03T09:06:47.827+0000 7fcee2ffd640 10 monclient(hunting): _init_auth method 2 2024-06-03T09:06:47.828 INFO:teuthology.orchestra.run.smithi088.stderr:2024-06-03T09:06:47.827+0000 7fcee2ffd640 10 monclient(hunting): _init_auth creating new auth 2024-06-03T09:06:47.828 INFO:teuthology.orchestra.run.smithi088.stderr:2024-06-03T09:06:47.827+0000 7fcee2ffd640 10 monclient(hunting): my global_id is 30696 2024-06-03T09:06:47.828 INFO:teuthology.orchestra.run.smithi088.stderr:2024-06-03T09:06:47.827+0000 7fcee2ffd640 1 -- 172.21.15.88:0/1939170253 --> v1:172.21.15.88:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7fcee40b1ad0 con 0x7fcee41450f0 2024-06-03T09:06:47.828 INFO:teuthology.orchestra.run.smithi088.stderr:2024-06-03T09:06:47.827+0000 7fcee2ffd640 1 -- 172.21.15.88:0/1939170253 <== mon.1 v1:172.21.15.192:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 3278509676 0 0) 0x7fcee40b3930 con 0x7fcee40a8960 2024-06-03T09:06:47.828 INFO:teuthology.orchestra.run.smithi088.stderr:2024-06-03T09:06:47.827+0000 7fcee2ffd640 10 monclient(hunting): _negotiate 2024-06-03T09:06:47.828 INFO:teuthology.orchestra.run.smithi088.stderr:2024-06-03T09:06:47.827+0000 7fcee2ffd640 10 monclient(hunting): _init_auth method 2 2024-06-03T09:06:47.828 INFO:teuthology.orchestra.run.smithi088.stderr:2024-06-03T09:06:47.827+0000 7fcee2ffd640 10 monclient(hunting): _init_auth creating new auth 2024-06-03T09:06:47.828 INFO:teuthology.orchestra.run.smithi088.stderr:2024-06-03T09:06:47.827+0000 7fcee2ffd640 10 monclient(hunting): my global_id is 30274 2024-06-03T09:06:47.828 INFO:teuthology.orchestra.run.smithi088.stderr:2024-06-03T09:06:47.827+0000 7fcee2ffd640 1 -- 172.21.15.88:0/1939170253 --> v1:172.21.15.192:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7fcee4144c60 con 0x7fcee40a8960 2024-06-03T09:06:47.830 INFO:teuthology.orchestra.run.smithi088.stderr:2024-06-03T09:06:47.829+0000 7fcee2ffd640 1 -- 172.21.15.88:0/1939170253 <== mon.1 v1:172.21.15.192:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 214+0+0 (unknown 1918720830 0 0) 0x7fcee4144c60 con 0x7fcee40a8960 2024-06-03T09:06:47.830 INFO:teuthology.orchestra.run.smithi088.stderr:2024-06-03T09:06:47.829+0000 7fcee2ffd640 1 -- 172.21.15.88:0/1939170253 >> v1:172.21.15.88:6790/0 conn(0x7fcee413eec0 legacy=0x7fcee414ce60 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-06-03T09:06:47.830 INFO:teuthology.orchestra.run.smithi088.stderr:2024-06-03T09:06:47.829+0000 7fcee2ffd640 1 -- 172.21.15.88:0/1939170253 >> v1:172.21.15.88:6789/0 conn(0x7fcee41450f0 legacy=0x7fcee413e7b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-06-03T09:06:47.830 INFO:teuthology.orchestra.run.smithi088.stderr:2024-06-03T09:06:47.829+0000 7fcee2ffd640 10 monclient: _finish_hunting 0 2024-06-03T09:06:47.830 INFO:teuthology.orchestra.run.smithi088.stderr:2024-06-03T09:06:47.829+0000 7fcee2ffd640 1 monclient: found mon.noname-c 2024-06-03T09:06:47.830 INFO:teuthology.orchestra.run.smithi088.stderr:2024-06-03T09:06:47.829+0000 7fcee2ffd640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-06-03T09:06:47.830 INFO:teuthology.orchestra.run.smithi088.stderr:2024-06-03T09:06:47.829+0000 7fcee2ffd640 10 monclient: _send_mon_message to mon.noname-c at v1:172.21.15.192:6789/0 2024-06-03T09:06:47.830 INFO:teuthology.orchestra.run.smithi088.stderr:2024-06-03T09:06:47.829+0000 7fcee2ffd640 1 -- 172.21.15.88:0/1939170253 --> v1:172.21.15.192:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fcee40b1c60 con 0x7fcee40a8960 2024-06-03T09:06:47.830 INFO:teuthology.orchestra.run.smithi088.stderr:2024-06-03T09:06:47.829+0000 7fcee2ffd640 10 monclient: _finish_auth 0 2024-06-03T09:06:47.831 INFO:teuthology.orchestra.run.smithi088.stderr:2024-06-03T09:06:47.829+0000 7fcee2ffd640 10 monclient: _check_auth_tickets 2024-06-03T09:06:47.831 INFO:teuthology.orchestra.run.smithi088.stderr:2024-06-03T09:06:47.829+0000 7fcee2ffd640 20 monclient: _check_auth_rotating not needed by client.admin 2024-06-03T09:06:47.831 INFO:teuthology.orchestra.run.smithi088.stderr:2024-06-03T09:06:47.829+0000 7fceea937640 5 monclient: authenticate success, global_id 30274 2024-06-03T09:06:47.831 INFO:teuthology.orchestra.run.smithi088.stderr:2024-06-03T09:06:47.829+0000 7fcee2ffd640 1 -- 172.21.15.88:0/1939170253 <== mon.1 v1:172.21.15.192:6789/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 2234374065 0 0) 0x7fced40015b0 con 0x7fcee40a8960 2024-06-03T09:06:47.831 INFO:teuthology.orchestra.run.smithi088.stderr:2024-06-03T09:06:47.829+0000 7fcee2ffd640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-06-03T09:06:47.831 INFO:teuthology.orchestra.run.smithi088.stderr:2024-06-03T09:06:47.829+0000 7fcee2ffd640 10 monclient: got monmap 1 from mon.noname-c (according to old e1) 2024-06-03T09:06:47.831 INFO:teuthology.orchestra.run.smithi088.stderr:2024-06-03T09:06:47.830+0000 7fcee2ffd640 10 monclient: dump: 2024-06-03T09:06:47.831 INFO:teuthology.orchestra.run.smithi088.stderr:epoch 1 2024-06-03T09:06:47.831 INFO:teuthology.orchestra.run.smithi088.stderr:fsid ec3c3fdd-3ac6-4d7d-9412-683130ed8b04 2024-06-03T09:06:47.831 INFO:teuthology.orchestra.run.smithi088.stderr:last_changed 2024-06-03T08:34:31.885125+0000 2024-06-03T09:06:47.831 INFO:teuthology.orchestra.run.smithi088.stderr:created 2024-06-03T08:34:31.885125+0000 2024-06-03T09:06:47.832 INFO:teuthology.orchestra.run.smithi088.stderr:min_mon_release 18 (reef) 2024-06-03T09:06:47.832 INFO:teuthology.orchestra.run.smithi088.stderr:election_strategy: 3 2024-06-03T09:06:47.832 INFO:teuthology.orchestra.run.smithi088.stderr:0: v1:172.21.15.88:6789/0 mon.a 2024-06-03T09:06:47.832 INFO:teuthology.orchestra.run.smithi088.stderr:1: v1:172.21.15.192:6789/0 mon.b 2024-06-03T09:06:47.832 INFO:teuthology.orchestra.run.smithi088.stderr:2: v1:172.21.15.88:6790/0 mon.c 2024-06-03T09:06:47.832 INFO:teuthology.orchestra.run.smithi088.stderr: 2024-06-03T09:06:47.832 INFO:teuthology.orchestra.run.smithi088.stderr:2024-06-03T09:06:47.830+0000 7fceea937640 20 monclient: get_monmap_and_config waiting for monmap|config 2024-06-03T09:06:47.836 INFO:teuthology.orchestra.run.smithi088.stderr:2024-06-03T09:06:47.832+0000 7fcee2ffd640 1 -- 172.21.15.88:0/1939170253 <== mon.1 v1:172.21.15.192:6789/0 4 ==== config(0 keys) v1 ==== 4+0+0 (unknown 0 0 0) 0x7fced40017d0 con 0x7fcee40a8960 2024-06-03T09:06:47.836 INFO:teuthology.orchestra.run.smithi088.stderr:2024-06-03T09:06:47.832+0000 7fcee2ffd640 10 monclient: handle_config config(0 keys) v1 2024-06-03T09:06:47.836 INFO:teuthology.orchestra.run.smithi088.stderr:2024-06-03T09:06:47.832+0000 7fcee2ffd640 1 -- 172.21.15.88:0/1939170253 <== mon.1 v1:172.21.15.192:6789/0 5 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 2234374065 0 0) 0x7fced4001ae0 con 0x7fcee40a8960 2024-06-03T09:06:47.836 INFO:teuthology.orchestra.run.smithi088.stderr:2024-06-03T09:06:47.832+0000 7fcee2ffd640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-06-03T09:06:47.836 INFO:teuthology.orchestra.run.smithi088.stderr:2024-06-03T09:06:47.832+0000 7fcee2ffd640 10 monclient: got monmap 1 from mon.b (according to old e1) 2024-06-03T09:06:47.836 INFO:teuthology.orchestra.run.smithi088.stderr:2024-06-03T09:06:47.832+0000 7fcee2ffd640 10 monclient: dump: 2024-06-03T09:06:47.836 INFO:teuthology.orchestra.run.smithi088.stderr:epoch 1 2024-06-03T09:06:47.836 INFO:teuthology.orchestra.run.smithi088.stderr:fsid ec3c3fdd-3ac6-4d7d-9412-683130ed8b04 2024-06-03T09:06:47.836 INFO:teuthology.orchestra.run.smithi088.stderr:last_changed 2024-06-03T08:34:31.885125+0000 2024-06-03T09:06:47.837 INFO:teuthology.orchestra.run.smithi088.stderr:created 2024-06-03T08:34:31.885125+0000 2024-06-03T09:06:47.837 INFO:teuthology.orchestra.run.smithi088.stderr:min_mon_release 18 (reef) 2024-06-03T09:06:47.837 INFO:teuthology.orchestra.run.smithi088.stderr:election_strategy: 3 2024-06-03T09:06:47.837 INFO:teuthology.orchestra.run.smithi088.stderr:0: v1:172.21.15.88:6789/0 mon.a 2024-06-03T09:06:47.837 INFO:teuthology.orchestra.run.smithi088.stderr:1: v1:172.21.15.192:6789/0 mon.b 2024-06-03T09:06:47.837 INFO:teuthology.orchestra.run.smithi088.stderr:2: v1:172.21.15.88:6790/0 mon.c 2024-06-03T09:06:47.837 INFO:teuthology.orchestra.run.smithi088.stderr: 2024-06-03T09:06:47.837 INFO:teuthology.orchestra.run.smithi088.stderr:2024-06-03T09:06:47.832+0000 7fceea937640 10 monclient: get_monmap_and_config success 2024-06-03T09:06:47.837 INFO:teuthology.orchestra.run.smithi088.stderr:2024-06-03T09:06:47.832+0000 7fceea937640 4 set_mon_vals no callback set 2024-06-03T09:06:47.838 INFO:teuthology.orchestra.run.smithi088.stderr:2024-06-03T09:06:47.832+0000 7fceea937640 10 monclient: shutdown 2024-06-03T09:06:47.838 INFO:teuthology.orchestra.run.smithi088.stderr:2024-06-03T09:06:47.832+0000 7fceea937640 20 monclient: shutdown discarding 0 pending message(s) 2024-06-03T09:06:47.838 INFO:teuthology.orchestra.run.smithi088.stderr:2024-06-03T09:06:47.832+0000 7fceea937640 1 -- 172.21.15.88:0/1939170253 >> v1:172.21.15.192:6789/0 conn(0x7fcee40a8960 legacy=0x7fcee40a8d40 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-06-03T09:06:47.838 INFO:teuthology.orchestra.run.smithi088.stderr:2024-06-03T09:06:47.833+0000 7fceea937640 1 -- 172.21.15.88:0/1939170253 shutdown_connections 2024-06-03T09:06:47.838 INFO:teuthology.orchestra.run.smithi088.stderr:2024-06-03T09:06:47.833+0000 7fceea937640 1 -- 172.21.15.88:0/1939170253 >> 172.21.15.88:0/1939170253 conn(0x7fcee40b70d0 msgr2=0x7fcee40b74e0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-06-03T09:06:47.838 INFO:teuthology.orchestra.run.smithi088.stderr:2024-06-03T09:06:47.833+0000 7fceea937640 1 -- 172.21.15.88:0/1939170253 shutdown_connections 2024-06-03T09:06:47.838 INFO:teuthology.orchestra.run.smithi088.stderr:2024-06-03T09:06:47.833+0000 7fceea937640 1 -- 172.21.15.88:0/1939170253 wait complete. 2024-06-03T09:06:47.838 INFO:teuthology.orchestra.run.smithi088.stderr:2024-06-03T09:06:47.833+0000 7fceea937640 10 monclient: build_initial_monmap 2024-06-03T09:06:47.838 INFO:teuthology.orchestra.run.smithi088.stderr:2024-06-03T09:06:47.833+0000 7fceea937640 10 monclient: monmap: 2024-06-03T09:06:47.839 INFO:teuthology.orchestra.run.smithi088.stderr:epoch 0 2024-06-03T09:06:47.839 INFO:teuthology.orchestra.run.smithi088.stderr:fsid 00000000-0000-0000-0000-000000000000 2024-06-03T09:06:47.839 INFO:teuthology.orchestra.run.smithi088.stderr:last_changed 0.000000 2024-06-03T09:06:47.839 INFO:teuthology.orchestra.run.smithi088.stderr:created 0.000000 2024-06-03T09:06:47.839 INFO:teuthology.orchestra.run.smithi088.stderr:min_mon_release 0 (unknown) 2024-06-03T09:06:47.839 INFO:teuthology.orchestra.run.smithi088.stderr:election_strategy: 1 2024-06-03T09:06:47.839 INFO:teuthology.orchestra.run.smithi088.stderr:0: v1:172.21.15.88:6789/0 mon.noname-a 2024-06-03T09:06:47.839 INFO:teuthology.orchestra.run.smithi088.stderr:1: v1:172.21.15.192:6789/0 mon.noname-b 2024-06-03T09:06:47.839 INFO:teuthology.orchestra.run.smithi088.stderr:2: v1:172.21.15.88:6790/0 mon.noname-c 2024-06-03T09:06:47.839 INFO:teuthology.orchestra.run.smithi088.stderr: 2024-06-03T09:06:47.839 INFO:teuthology.orchestra.run.smithi088.stderr:2024-06-03T09:06:47.833+0000 7fceea937640 1 librados: starting msgr at 2024-06-03T09:06:47.839 INFO:teuthology.orchestra.run.smithi088.stderr:2024-06-03T09:06:47.833+0000 7fceea937640 1 librados: starting objecter 2024-06-03T09:06:47.839 INFO:teuthology.orchestra.run.smithi088.stderr:2024-06-03T09:06:47.834+0000 7fceea937640 1 Processor -- start 2024-06-03T09:06:47.840 INFO:teuthology.orchestra.run.smithi088.stderr:2024-06-03T09:06:47.834+0000 7fceea937640 1 -- start start 2024-06-03T09:06:47.840 INFO:teuthology.orchestra.run.smithi088.stderr:2024-06-03T09:06:47.834+0000 7fceea937640 1 librados: setting wanted keys 2024-06-03T09:06:47.840 INFO:teuthology.orchestra.run.smithi088.stderr:2024-06-03T09:06:47.834+0000 7fceea937640 1 librados: calling monclient init 2024-06-03T09:06:47.840 INFO:teuthology.orchestra.run.smithi088.stderr:2024-06-03T09:06:47.834+0000 7fceea937640 10 monclient: init 2024-06-03T09:06:47.840 INFO:teuthology.orchestra.run.smithi088.stderr:2024-06-03T09:06:47.834+0000 7fceea937640 10 monclient: _reopen_session rank -1 2024-06-03T09:06:47.840 INFO:teuthology.orchestra.run.smithi088.stderr:2024-06-03T09:06:47.834+0000 7fceea937640 10 monclient: _add_conns ranks=[2,0,1] 2024-06-03T09:06:47.840 INFO:teuthology.orchestra.run.smithi088.stderr:2024-06-03T09:06:47.834+0000 7fceea937640 10 monclient(hunting): picked mon.noname-c con 0x7fcee40a8960 addr v1:172.21.15.88:6790/0 2024-06-03T09:06:47.840 INFO:teuthology.orchestra.run.smithi088.stderr:2024-06-03T09:06:47.834+0000 7fceea937640 10 monclient(hunting): picked mon.noname-a con 0x7fcee413eec0 addr v1:172.21.15.88:6789/0 2024-06-03T09:06:47.840 INFO:teuthology.orchestra.run.smithi088.stderr:2024-06-03T09:06:47.834+0000 7fceea937640 10 monclient(hunting): picked mon.noname-b con 0x7fcee41450f0 addr v1:172.21.15.192:6789/0 2024-06-03T09:06:47.840 INFO:teuthology.orchestra.run.smithi088.stderr:2024-06-03T09:06:47.834+0000 7fceea937640 1 -- --> v1:172.21.15.88:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fcee41d9c00 con 0x7fcee413eec0 2024-06-03T09:06:47.841 INFO:teuthology.orchestra.run.smithi088.stderr:2024-06-03T09:06:47.834+0000 7fceea937640 1 -- --> v1:172.21.15.192:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fcee4137e50 con 0x7fcee41450f0 2024-06-03T09:06:47.841 INFO:teuthology.orchestra.run.smithi088.stderr:2024-06-03T09:06:47.834+0000 7fceea937640 1 -- --> v1:172.21.15.88:6790/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fcee4141280 con 0x7fcee40a8960 2024-06-03T09:06:47.841 INFO:teuthology.orchestra.run.smithi088.stderr:2024-06-03T09:06:47.834+0000 7fceea937640 10 monclient(hunting): _renew_subs 2024-06-03T09:06:47.841 INFO:teuthology.orchestra.run.smithi088.stderr:2024-06-03T09:06:47.834+0000 7fceea937640 10 monclient(hunting): authenticate will time out at 3063.644751s 2024-06-03T09:06:47.841 INFO:teuthology.orchestra.run.smithi088.stderr:2024-06-03T09:06:47.835+0000 7fcee3fff640 1 --1- >> v1:172.21.15.88:6790/0 conn(0x7fcee40a8960 0x7fcee41da090 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=1).handle_server_banner_and_identify peer v1:172.21.15.88:6790/0 says I am v1:172.21.15.88:52312/0 (socket says 172.21.15.88:52312) 2024-06-03T09:06:47.841 INFO:teuthology.orchestra.run.smithi088.stderr:2024-06-03T09:06:47.835+0000 7fcee3fff640 1 -- 172.21.15.88:0/3312006346 learned_addr learned my addr 172.21.15.88:0/3312006346 (peer_addr_for_me v1:172.21.15.88:0/0) 2024-06-03T09:06:47.841 INFO:teuthology.orchestra.run.smithi088.stderr:2024-06-03T09:06:47.835+0000 7fcee0ff9640 10 client.?.objecter ms_handle_connect 0x7fcee40a8960 2024-06-03T09:06:47.841 INFO:teuthology.orchestra.run.smithi088.stderr:2024-06-03T09:06:47.835+0000 7fcee0ff9640 10 client.?.objecter resend_mon_ops 2024-06-03T09:06:47.841 INFO:teuthology.orchestra.run.smithi088.stderr:2024-06-03T09:06:47.835+0000 7fcee0ff9640 10 client.?.objecter ms_handle_connect 0x7fcee413eec0 2024-06-03T09:06:47.841 INFO:teuthology.orchestra.run.smithi088.stderr:2024-06-03T09:06:47.835+0000 7fcee0ff9640 10 client.?.objecter resend_mon_ops 2024-06-03T09:06:47.841 INFO:teuthology.orchestra.run.smithi088.stderr:2024-06-03T09:06:47.835+0000 7fcee0ff9640 10 client.?.objecter ms_handle_connect 0x7fcee41450f0 2024-06-03T09:06:47.841 INFO:teuthology.orchestra.run.smithi088.stderr:2024-06-03T09:06:47.835+0000 7fcee0ff9640 10 client.?.objecter resend_mon_ops 2024-06-03T09:06:47.843 INFO:teuthology.orchestra.run.smithi088.stderr:2024-06-03T09:06:47.837+0000 7fcee0ff9640 1 -- 172.21.15.88:0/3312006346 <== mon.2 v1:172.21.15.88:6790/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 441005682 0 0) 0x7fcee4141280 con 0x7fcee40a8960 2024-06-03T09:06:47.843 INFO:teuthology.orchestra.run.smithi088.stderr:2024-06-03T09:06:47.837+0000 7fcee0ff9640 10 monclient(hunting): _negotiate 2024-06-03T09:06:47.843 INFO:teuthology.orchestra.run.smithi088.stderr:2024-06-03T09:06:47.837+0000 7fcee0ff9640 10 monclient(hunting): _init_auth method 2 2024-06-03T09:06:47.843 INFO:teuthology.orchestra.run.smithi088.stderr:2024-06-03T09:06:47.837+0000 7fcee0ff9640 10 monclient(hunting): _init_auth creating new auth 2024-06-03T09:06:47.843 INFO:teuthology.orchestra.run.smithi088.stderr:2024-06-03T09:06:47.837+0000 7fcee0ff9640 10 monclient(hunting): my global_id is 30203 2024-06-03T09:06:47.843 INFO:teuthology.orchestra.run.smithi088.stderr:2024-06-03T09:06:47.837+0000 7fcee0ff9640 1 -- 172.21.15.88:0/3312006346 --> v1:172.21.15.88:6790/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7fced0004eb0 con 0x7fcee40a8960 2024-06-03T09:06:47.843 INFO:teuthology.orchestra.run.smithi088.stderr:2024-06-03T09:06:47.837+0000 7fcee0ff9640 1 -- 172.21.15.88:0/3312006346 <== mon.0 v1:172.21.15.88:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 2081126075 0 0) 0x7fcee41d9c00 con 0x7fcee413eec0 2024-06-03T09:06:47.843 INFO:teuthology.orchestra.run.smithi088.stderr:2024-06-03T09:06:47.837+0000 7fcee0ff9640 10 monclient(hunting): _negotiate 2024-06-03T09:06:47.844 INFO:teuthology.orchestra.run.smithi088.stderr:2024-06-03T09:06:47.837+0000 7fcee0ff9640 10 monclient(hunting): _init_auth method 2 2024-06-03T09:06:47.844 INFO:teuthology.orchestra.run.smithi088.stderr:2024-06-03T09:06:47.837+0000 7fcee0ff9640 10 monclient(hunting): _init_auth creating new auth 2024-06-03T09:06:47.844 INFO:teuthology.orchestra.run.smithi088.stderr:2024-06-03T09:06:47.837+0000 7fcee0ff9640 10 monclient(hunting): my global_id is 30699 2024-06-03T09:06:47.844 INFO:teuthology.orchestra.run.smithi088.stderr:2024-06-03T09:06:47.837+0000 7fcee0ff9640 1 -- 172.21.15.88:0/3312006346 --> v1:172.21.15.88:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7fcee4141280 con 0x7fcee413eec0 2024-06-03T09:06:47.844 INFO:teuthology.orchestra.run.smithi088.stderr:2024-06-03T09:06:47.838+0000 7fcee0ff9640 1 -- 172.21.15.88:0/3312006346 <== mon.1 v1:172.21.15.192:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 3005677692 0 0) 0x7fcee4137e50 con 0x7fcee41450f0 2024-06-03T09:06:47.844 INFO:teuthology.orchestra.run.smithi088.stderr:2024-06-03T09:06:47.838+0000 7fcee0ff9640 10 monclient(hunting): _negotiate 2024-06-03T09:06:47.844 INFO:teuthology.orchestra.run.smithi088.stderr:2024-06-03T09:06:47.838+0000 7fcee0ff9640 10 monclient(hunting): _init_auth method 2 2024-06-03T09:06:47.844 INFO:teuthology.orchestra.run.smithi088.stderr:2024-06-03T09:06:47.838+0000 7fcee0ff9640 10 monclient(hunting): _init_auth creating new auth 2024-06-03T09:06:47.844 INFO:teuthology.orchestra.run.smithi088.stderr:2024-06-03T09:06:47.838+0000 7fcee0ff9640 10 monclient(hunting): my global_id is 30277 2024-06-03T09:06:47.844 INFO:teuthology.orchestra.run.smithi088.stderr:2024-06-03T09:06:47.838+0000 7fcee0ff9640 1 -- 172.21.15.88:0/3312006346 --> v1:172.21.15.192:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7fcee41d9c00 con 0x7fcee41450f0 2024-06-03T09:06:47.844 INFO:teuthology.orchestra.run.smithi088.stderr:2024-06-03T09:06:47.842+0000 7fcee0ff9640 1 -- 172.21.15.88:0/3312006346 <== mon.0 v1:172.21.15.88:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 764+0+0 (unknown 752064733 0 0) 0x7fcee4141280 con 0x7fcee413eec0 2024-06-03T09:06:47.844 INFO:teuthology.orchestra.run.smithi088.stderr:2024-06-03T09:06:47.842+0000 7fcee0ff9640 1 -- 172.21.15.88:0/3312006346 --> v1:172.21.15.88:6789/0 -- auth(proto 2 165 bytes epoch 0) v1 -- 0x7fcee4137e50 con 0x7fcee413eec0 2024-06-03T09:06:47.844 INFO:teuthology.orchestra.run.smithi088.stderr:2024-06-03T09:06:47.842+0000 7fcee0ff9640 1 -- 172.21.15.88:0/3312006346 <== mon.0 v1:172.21.15.88:6789/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 2234374065 0 0) 0x7fced80017d0 con 0x7fcee413eec0 2024-06-03T09:06:47.845 INFO:teuthology.orchestra.run.smithi088.stderr:2024-06-03T09:06:47.842+0000 7fcee0ff9640 10 monclient(hunting): handle_monmap mon_map magic: 0 v1 2024-06-03T09:06:47.845 INFO:teuthology.orchestra.run.smithi088.stderr:2024-06-03T09:06:47.842+0000 7fcee0ff9640 10 monclient(hunting): got monmap 1 from mon.noname-a (according to old e1) 2024-06-03T09:06:47.845 INFO:teuthology.orchestra.run.smithi088.stderr:2024-06-03T09:06:47.842+0000 7fcee0ff9640 10 monclient(hunting): dump: 2024-06-03T09:06:47.845 INFO:teuthology.orchestra.run.smithi088.stderr:epoch 1 2024-06-03T09:06:47.845 INFO:teuthology.orchestra.run.smithi088.stderr:fsid ec3c3fdd-3ac6-4d7d-9412-683130ed8b04 2024-06-03T09:06:47.845 INFO:teuthology.orchestra.run.smithi088.stderr:last_changed 2024-06-03T08:34:31.885125+0000 2024-06-03T09:06:47.845 INFO:teuthology.orchestra.run.smithi088.stderr:created 2024-06-03T08:34:31.885125+0000 2024-06-03T09:06:47.845 INFO:teuthology.orchestra.run.smithi088.stderr:min_mon_release 18 (reef) 2024-06-03T09:06:47.845 INFO:teuthology.orchestra.run.smithi088.stderr:election_strategy: 3 2024-06-03T09:06:47.845 INFO:teuthology.orchestra.run.smithi088.stderr:0: v1:172.21.15.88:6789/0 mon.a 2024-06-03T09:06:47.845 INFO:teuthology.orchestra.run.smithi088.stderr:1: v1:172.21.15.192:6789/0 mon.b 2024-06-03T09:06:47.845 INFO:teuthology.orchestra.run.smithi088.stderr:2: v1:172.21.15.88:6790/0 mon.c 2024-06-03T09:06:47.845 INFO:teuthology.orchestra.run.smithi088.stderr: 2024-06-03T09:06:47.845 INFO:teuthology.orchestra.run.smithi088.stderr:2024-06-03T09:06:47.842+0000 7fcee0ff9640 10 monclient(hunting): _finish_auth 0 2024-06-03T09:06:47.847 INFO:teuthology.orchestra.run.smithi088.stderr:2024-06-03T09:06:47.844+0000 7fcee0ff9640 1 -- 172.21.15.88:0/3312006346 <== mon.2 v1:172.21.15.88:6790/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 764+0+0 (unknown 2994781873 0 0) 0x7fced0004eb0 con 0x7fcee40a8960 2024-06-03T09:06:47.847 INFO:teuthology.orchestra.run.smithi088.stderr:2024-06-03T09:06:47.844+0000 7fcee0ff9640 1 -- 172.21.15.88:0/3312006346 --> v1:172.21.15.88:6790/0 -- auth(proto 2 165 bytes epoch 0) v1 -- 0x7fcee4141280 con 0x7fcee40a8960 2024-06-03T09:06:47.847 INFO:teuthology.orchestra.run.smithi088.stderr:2024-06-03T09:06:47.844+0000 7fcee0ff9640 1 -- 172.21.15.88:0/3312006346 <== mon.2 v1:172.21.15.88:6790/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 2234374065 0 0) 0x7fced4001530 con 0x7fcee40a8960 2024-06-03T09:06:47.847 INFO:teuthology.orchestra.run.smithi088.stderr:2024-06-03T09:06:47.844+0000 7fcee0ff9640 10 monclient(hunting): handle_monmap mon_map magic: 0 v1 2024-06-03T09:06:47.847 INFO:teuthology.orchestra.run.smithi088.stderr:2024-06-03T09:06:47.844+0000 7fcee0ff9640 10 monclient(hunting): got monmap 1 from mon.c (according to old e1) 2024-06-03T09:06:47.847 INFO:teuthology.orchestra.run.smithi088.stderr:2024-06-03T09:06:47.844+0000 7fcee0ff9640 10 monclient(hunting): dump: 2024-06-03T09:06:47.848 INFO:teuthology.orchestra.run.smithi088.stderr:epoch 1 2024-06-03T09:06:47.848 INFO:teuthology.orchestra.run.smithi088.stderr:fsid ec3c3fdd-3ac6-4d7d-9412-683130ed8b04 2024-06-03T09:06:47.848 INFO:teuthology.orchestra.run.smithi088.stderr:last_changed 2024-06-03T08:34:31.885125+0000 2024-06-03T09:06:47.848 INFO:teuthology.orchestra.run.smithi088.stderr:created 2024-06-03T08:34:31.885125+0000 2024-06-03T09:06:47.848 INFO:teuthology.orchestra.run.smithi088.stderr:min_mon_release 18 (reef) 2024-06-03T09:06:47.848 INFO:teuthology.orchestra.run.smithi088.stderr:election_strategy: 3 2024-06-03T09:06:47.848 INFO:teuthology.orchestra.run.smithi088.stderr:0: v1:172.21.15.88:6789/0 mon.a 2024-06-03T09:06:47.848 INFO:teuthology.orchestra.run.smithi088.stderr:1: v1:172.21.15.192:6789/0 mon.b 2024-06-03T09:06:47.848 INFO:teuthology.orchestra.run.smithi088.stderr:2: v1:172.21.15.88:6790/0 mon.c 2024-06-03T09:06:47.848 INFO:teuthology.orchestra.run.smithi088.stderr: 2024-06-03T09:06:47.848 INFO:teuthology.orchestra.run.smithi088.stderr:2024-06-03T09:06:47.844+0000 7fcee0ff9640 1 -- 172.21.15.88:0/3312006346 <== mon.1 v1:172.21.15.192:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 764+0+0 (unknown 4056819201 0 0) 0x7fcee41d9c00 con 0x7fcee41450f0 2024-06-03T09:06:47.849 INFO:teuthology.orchestra.run.smithi088.stderr:2024-06-03T09:06:47.844+0000 7fcee0ff9640 1 -- 172.21.15.88:0/3312006346 --> v1:172.21.15.192:6789/0 -- auth(proto 2 165 bytes epoch 0) v1 -- 0x7fced0004eb0 con 0x7fcee41450f0 2024-06-03T09:06:47.849 INFO:teuthology.orchestra.run.smithi088.stderr:2024-06-03T09:06:47.844+0000 7fcee0ff9640 1 -- 172.21.15.88:0/3312006346 <== mon.1 v1:172.21.15.192:6789/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 2234374065 0 0) 0x7fcedc001c50 con 0x7fcee41450f0 2024-06-03T09:06:47.849 INFO:teuthology.orchestra.run.smithi088.stderr:2024-06-03T09:06:47.844+0000 7fcee0ff9640 10 monclient(hunting): handle_monmap mon_map magic: 0 v1 2024-06-03T09:06:47.849 INFO:teuthology.orchestra.run.smithi088.stderr:2024-06-03T09:06:47.844+0000 7fcee0ff9640 10 monclient(hunting): got monmap 1 from mon.b (according to old e1) 2024-06-03T09:06:47.849 INFO:teuthology.orchestra.run.smithi088.stderr:2024-06-03T09:06:47.844+0000 7fcee0ff9640 10 monclient(hunting): dump: 2024-06-03T09:06:47.849 INFO:teuthology.orchestra.run.smithi088.stderr:epoch 1 2024-06-03T09:06:47.849 INFO:teuthology.orchestra.run.smithi088.stderr:fsid ec3c3fdd-3ac6-4d7d-9412-683130ed8b04 2024-06-03T09:06:47.849 INFO:teuthology.orchestra.run.smithi088.stderr:last_changed 2024-06-03T08:34:31.885125+0000 2024-06-03T09:06:47.849 INFO:teuthology.orchestra.run.smithi088.stderr:created 2024-06-03T08:34:31.885125+0000 2024-06-03T09:06:47.849 INFO:teuthology.orchestra.run.smithi088.stderr:min_mon_release 18 (reef) 2024-06-03T09:06:47.849 INFO:teuthology.orchestra.run.smithi088.stderr:election_strategy: 3 2024-06-03T09:06:47.849 INFO:teuthology.orchestra.run.smithi088.stderr:0: v1:172.21.15.88:6789/0 mon.a 2024-06-03T09:06:47.849 INFO:teuthology.orchestra.run.smithi088.stderr:1: v1:172.21.15.192:6789/0 mon.b 2024-06-03T09:06:47.849 INFO:teuthology.orchestra.run.smithi088.stderr:2: v1:172.21.15.88:6790/0 mon.c 2024-06-03T09:06:47.850 INFO:teuthology.orchestra.run.smithi088.stderr: 2024-06-03T09:06:47.850 INFO:teuthology.orchestra.run.smithi088.stderr:2024-06-03T09:06:47.846+0000 7fcee0ff9640 1 -- 172.21.15.88:0/3312006346 <== mon.0 v1:172.21.15.88:6789/0 4 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 580+0+0 (unknown 1723500600 0 0) 0x7fcee4137e50 con 0x7fcee413eec0 2024-06-03T09:06:47.850 INFO:teuthology.orchestra.run.smithi088.stderr:2024-06-03T09:06:47.846+0000 7fcee0ff9640 1 -- 172.21.15.88:0/3312006346 >> v1:172.21.15.88:6790/0 conn(0x7fcee40a8960 legacy=0x7fcee41da090 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-06-03T09:06:47.850 INFO:teuthology.orchestra.run.smithi088.stderr:2024-06-03T09:06:47.846+0000 7fcee0ff9640 1 -- 172.21.15.88:0/3312006346 >> v1:172.21.15.192:6789/0 conn(0x7fcee41450f0 legacy=0x7fcee4137740 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-06-03T09:06:47.850 INFO:teuthology.orchestra.run.smithi088.stderr:2024-06-03T09:06:47.846+0000 7fcee0ff9640 10 monclient: _finish_hunting 0 2024-06-03T09:06:47.850 INFO:teuthology.orchestra.run.smithi088.stderr:2024-06-03T09:06:47.846+0000 7fcee0ff9640 1 monclient: found mon.a 2024-06-03T09:06:47.850 INFO:teuthology.orchestra.run.smithi088.stderr:2024-06-03T09:06:47.846+0000 7fcee0ff9640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-06-03T09:06:47.850 INFO:teuthology.orchestra.run.smithi088.stderr:2024-06-03T09:06:47.846+0000 7fcee0ff9640 10 monclient: _send_mon_message to mon.a at v1:172.21.15.88:6789/0 2024-06-03T09:06:47.850 INFO:teuthology.orchestra.run.smithi088.stderr:2024-06-03T09:06:47.846+0000 7fcee0ff9640 1 -- 172.21.15.88:0/3312006346 --> v1:172.21.15.88:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fcee41426f0 con 0x7fcee413eec0 2024-06-03T09:06:47.850 INFO:teuthology.orchestra.run.smithi088.stderr:2024-06-03T09:06:47.846+0000 7fcee0ff9640 10 monclient: _finish_auth 0 2024-06-03T09:06:47.850 INFO:teuthology.orchestra.run.smithi088.stderr:2024-06-03T09:06:47.846+0000 7fcee0ff9640 10 monclient: _check_auth_tickets 2024-06-03T09:06:47.850 INFO:teuthology.orchestra.run.smithi088.stderr:2024-06-03T09:06:47.846+0000 7fcee0ff9640 20 monclient: _check_auth_rotating not needed by client.admin 2024-06-03T09:06:47.850 INFO:teuthology.orchestra.run.smithi088.stderr:2024-06-03T09:06:47.846+0000 7fceea937640 5 monclient: authenticate success, global_id 30699 2024-06-03T09:06:47.850 INFO:teuthology.orchestra.run.smithi088.stderr:2024-06-03T09:06:47.846+0000 7fceea937640 10 monclient: _renew_subs 2024-06-03T09:06:47.851 INFO:teuthology.orchestra.run.smithi088.stderr:2024-06-03T09:06:47.846+0000 7fceea937640 10 monclient: _send_mon_message to mon.a at v1:172.21.15.88:6789/0 2024-06-03T09:06:47.851 INFO:teuthology.orchestra.run.smithi088.stderr:2024-06-03T09:06:47.846+0000 7fceea937640 1 -- 172.21.15.88:0/3312006346 --> v1:172.21.15.88:6789/0 -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fcee41da7a0 con 0x7fcee413eec0 2024-06-03T09:06:47.851 INFO:teuthology.orchestra.run.smithi088.stderr:2024-06-03T09:06:47.846+0000 7fceea937640 10 client.30699.objecter _maybe_request_map subscribing (onetime) to next osd map 2024-06-03T09:06:47.851 INFO:teuthology.orchestra.run.smithi088.stderr:2024-06-03T09:06:47.846+0000 7fceea937640 10 monclient: _renew_subs 2024-06-03T09:06:47.851 INFO:teuthology.orchestra.run.smithi088.stderr:2024-06-03T09:06:47.846+0000 7fceea937640 10 monclient: _send_mon_message to mon.a at v1:172.21.15.88:6789/0 2024-06-03T09:06:47.851 INFO:teuthology.orchestra.run.smithi088.stderr:2024-06-03T09:06:47.846+0000 7fceea937640 1 -- 172.21.15.88:0/3312006346 --> v1:172.21.15.88:6789/0 -- mon_subscribe({osdmap=0}) v3 -- 0x7fcee413f560 con 0x7fcee413eec0 2024-06-03T09:06:47.851 INFO:teuthology.orchestra.run.smithi088.stderr:2024-06-03T09:06:47.846+0000 7fceea937640 1 librados: init done 2024-06-03T09:06:47.852 INFO:teuthology.orchestra.run.smithi088.stderr:2024-06-03T09:06:47.848+0000 7fceea937640 10 start_mon_command cmd=[{"prefix": "get_command_descriptions"}] 2024-06-03T09:06:47.852 INFO:teuthology.orchestra.run.smithi088.stderr:2024-06-03T09:06:47.848+0000 7fceea937640 10 monclient: _send_command 1 [{"prefix": "get_command_descriptions"}] 2024-06-03T09:06:47.852 INFO:teuthology.orchestra.run.smithi088.stderr:2024-06-03T09:06:47.848+0000 7fceea937640 10 monclient: _send_mon_message to mon.a at v1:172.21.15.88:6789/0 2024-06-03T09:06:47.852 INFO:teuthology.orchestra.run.smithi088.stderr:2024-06-03T09:06:47.848+0000 7fceea937640 1 -- 172.21.15.88:0/3312006346 --> v1:172.21.15.88:6789/0 -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fcee40a9e60 con 0x7fcee413eec0 2024-06-03T09:06:48.094 INFO:teuthology.orchestra.run.smithi088.stderr:2024-06-03T09:06:48.094+0000 7fcee0ff9640 1 -- 172.21.15.88:0/3312006346 <== mon.0 v1:172.21.15.88:6789/0 5 ==== config(0 keys) v1 ==== 4+0+0 (unknown 0 0 0) 0x7fced80035f0 con 0x7fcee413eec0 2024-06-03T09:06:48.094 INFO:teuthology.orchestra.run.smithi088.stderr:2024-06-03T09:06:48.094+0000 7fcee0ff9640 10 monclient: handle_config config(0 keys) v1 2024-06-03T09:06:48.094 INFO:teuthology.orchestra.run.smithi088.stderr:2024-06-03T09:06:48.094+0000 7fcee0ff9640 1 -- 172.21.15.88:0/3312006346 <== mon.0 v1:172.21.15.88:6789/0 6 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 2234374065 0 0) 0x7fced8001d80 con 0x7fcee413eec0 2024-06-03T09:06:48.094 INFO:teuthology.orchestra.run.smithi088.stderr:2024-06-03T09:06:48.094+0000 7fcee0ff9640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-06-03T09:06:48.094 INFO:teuthology.orchestra.run.smithi088.stderr:2024-06-03T09:06:48.094+0000 7fcee0ff9640 10 monclient: got monmap 1 from mon.a (according to old e1) 2024-06-03T09:06:48.094 INFO:teuthology.orchestra.run.smithi088.stderr:2024-06-03T09:06:48.094+0000 7fcee0ff9640 10 monclient: dump: 2024-06-03T09:06:48.094 INFO:teuthology.orchestra.run.smithi088.stderr:epoch 1 2024-06-03T09:06:48.095 INFO:teuthology.orchestra.run.smithi088.stderr:fsid ec3c3fdd-3ac6-4d7d-9412-683130ed8b04 2024-06-03T09:06:48.095 INFO:teuthology.orchestra.run.smithi088.stderr:last_changed 2024-06-03T08:34:31.885125+0000 2024-06-03T09:06:48.095 INFO:teuthology.orchestra.run.smithi088.stderr:created 2024-06-03T08:34:31.885125+0000 2024-06-03T09:06:48.095 INFO:teuthology.orchestra.run.smithi088.stderr:min_mon_release 18 (reef) 2024-06-03T09:06:48.095 INFO:teuthology.orchestra.run.smithi088.stderr:election_strategy: 3 2024-06-03T09:06:48.095 INFO:teuthology.orchestra.run.smithi088.stderr:0: v1:172.21.15.88:6789/0 mon.a 2024-06-03T09:06:48.095 INFO:teuthology.orchestra.run.smithi088.stderr:1: v1:172.21.15.192:6789/0 mon.b 2024-06-03T09:06:48.095 INFO:teuthology.orchestra.run.smithi088.stderr:2: v1:172.21.15.88:6790/0 mon.c 2024-06-03T09:06:48.095 INFO:teuthology.orchestra.run.smithi088.stderr: 2024-06-03T09:06:48.095 INFO:teuthology.orchestra.run.smithi088.stderr:2024-06-03T09:06:48.094+0000 7fcee27fc640 4 set_mon_vals no callback set 2024-06-03T09:06:48.096 INFO:teuthology.orchestra.run.smithi088.stderr:2024-06-03T09:06:48.094+0000 7fcee0ff9640 1 -- 172.21.15.88:0/3312006346 <== mon.0 v1:172.21.15.88:6789/0 7 ==== mgrmap(e 5) v1 ==== 89110+0+0 (unknown 873533973 0 0) 0x7fced8019520 con 0x7fcee413eec0 2024-06-03T09:06:48.096 INFO:teuthology.orchestra.run.smithi088.stderr:2024-06-03T09:06:48.095+0000 7fcee3fff640 10 monclient: get_auth_request con 0x7fced006d450 auth_method 0 2024-06-03T09:06:48.096 INFO:teuthology.orchestra.run.smithi088.stderr:2024-06-03T09:06:48.095+0000 7fcee0ff9640 10 client.30699.objecter ms_handle_connect 0x7fced006d450 2024-06-03T09:06:48.099 INFO:teuthology.orchestra.run.smithi088.stderr:2024-06-03T09:06:48.099+0000 7fcee0ff9640 1 -- 172.21.15.88:0/3312006346 <== mon.0 v1:172.21.15.88:6789/0 8 ==== osd_map(1014..1014 src has 622..1014) v4 ==== 5229+0+0 (unknown 3230192161 0 0) 0x7fced8052dd0 con 0x7fcee413eec0 2024-06-03T09:06:48.099 INFO:teuthology.orchestra.run.smithi088.stderr:2024-06-03T09:06:48.099+0000 7fcee0ff9640 10 client.30699.objecter ms_dispatch 0x7fcee4000b90 osd_map(1014..1014 src has 622..1014) v4 2024-06-03T09:06:48.100 INFO:teuthology.orchestra.run.smithi088.stderr:2024-06-03T09:06:48.099+0000 7fcee0ff9640 3 client.30699.objecter handle_osd_map got epochs [1014,1014] > 0 2024-06-03T09:06:48.100 INFO:teuthology.orchestra.run.smithi088.stderr:2024-06-03T09:06:48.099+0000 7fcee0ff9640 3 client.30699.objecter handle_osd_map decoding full epoch 1014 2024-06-03T09:06:48.100 INFO:teuthology.orchestra.run.smithi088.stderr:2024-06-03T09:06:48.099+0000 7fcee0ff9640 20 client.30699.objecter dump_active .. 0 homeless 2024-06-03T09:06:48.100 INFO:teuthology.orchestra.run.smithi088.stderr:2024-06-03T09:06:48.099+0000 7fcee0ff9640 1 -- 172.21.15.88:0/3312006346 <== mon.0 v1:172.21.15.88:6789/0 9 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+182498 (unknown 1092875540 0 3016760700) 0x7fced804e980 con 0x7fcee413eec0 2024-06-03T09:06:48.100 INFO:teuthology.orchestra.run.smithi088.stderr:2024-06-03T09:06:48.099+0000 7fcee0ff9640 10 monclient: handle_mon_command_ack 1 [{"prefix": "get_command_descriptions"}] 2024-06-03T09:06:48.100 INFO:teuthology.orchestra.run.smithi088.stderr:2024-06-03T09:06:48.099+0000 7fcee0ff9640 10 monclient: _finish_command 1 = system:0 2024-06-03T09:06:48.281 INFO:teuthology.orchestra.run.smithi088.stderr:2024-06-03T09:06:48.280+0000 7fceea937640 1 -- 172.21.15.88:0/3312006346 --> v1:172.21.15.192:6800/34229 -- mgr_command(tid 0: {"prefix": "pg dump", "target": ["mon-mgr", ""], "format": "json"}) v1 -- 0x7fcee41d9db0 con 0x7fced006d450 2024-06-03T09:06:48.285 INFO:teuthology.orchestra.run.smithi088.stderr:2024-06-03T09:06:48.285+0000 7fcee0ff9640 1 -- 172.21.15.88:0/3312006346 <== mgr.4110 v1:172.21.15.192:6800/34229 1 ==== mgr_command_reply(tid 0: 0 dumped all) v1 ==== 18+0+151706 (unknown 2965378022 0 3869215128) 0x7fcee41d9db0 con 0x7fced006d450 2024-06-03T09:06:48.287 INFO:teuthology.orchestra.run.smithi088.stdout: 2024-06-03T09:06:48.288 INFO:teuthology.orchestra.run.smithi088.stderr:dumped all 2024-06-03T09:06:48.291 INFO:teuthology.orchestra.run.smithi088.stderr:2024-06-03T09:06:48.287+0000 7fceea937640 10 librados: watch_flush enter 2024-06-03T09:06:48.291 INFO:teuthology.orchestra.run.smithi088.stderr:2024-06-03T09:06:48.287+0000 7fceea937640 10 librados: watch_flush exit 2024-06-03T09:06:48.292 INFO:teuthology.orchestra.run.smithi088.stderr:2024-06-03T09:06:48.287+0000 7fceea937640 20 client.30699.objecter shutdown clearing up homeless session... 2024-06-03T09:06:48.292 INFO:teuthology.orchestra.run.smithi088.stderr:2024-06-03T09:06:48.287+0000 7fceea937640 10 client.30699.objecter successfully canceled tick 2024-06-03T09:06:48.292 INFO:teuthology.orchestra.run.smithi088.stderr:2024-06-03T09:06:48.287+0000 7fceea937640 1 -- 172.21.15.88:0/3312006346 >> v1:172.21.15.192:6800/34229 conn(0x7fced006d450 legacy=0x7fced006f910 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-06-03T09:06:48.292 INFO:teuthology.orchestra.run.smithi088.stderr:2024-06-03T09:06:48.287+0000 7fceea937640 10 monclient: shutdown 2024-06-03T09:06:48.292 INFO:teuthology.orchestra.run.smithi088.stderr:2024-06-03T09:06:48.287+0000 7fceea937640 20 monclient: shutdown discarding 0 pending message(s) 2024-06-03T09:06:48.293 INFO:teuthology.orchestra.run.smithi088.stderr:2024-06-03T09:06:48.287+0000 7fceea937640 1 -- 172.21.15.88:0/3312006346 >> v1:172.21.15.88:6789/0 conn(0x7fcee413eec0 legacy=0x7fcee4137030 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-06-03T09:06:48.293 INFO:teuthology.orchestra.run.smithi088.stderr:2024-06-03T09:06:48.287+0000 7fceea937640 1 -- 172.21.15.88:0/3312006346 shutdown_connections 2024-06-03T09:06:48.293 INFO:teuthology.orchestra.run.smithi088.stderr:2024-06-03T09:06:48.287+0000 7fceea937640 1 -- 172.21.15.88:0/3312006346 >> 172.21.15.88:0/3312006346 conn(0x7fcee40b70d0 msgr2=0x7fcee413e580 unknown :-1 s=STATE_NONE l=0).mark_down 2024-06-03T09:06:48.293 INFO:teuthology.orchestra.run.smithi088.stderr:2024-06-03T09:06:48.287+0000 7fceea937640 1 -- 172.21.15.88:0/3312006346 shutdown_connections 2024-06-03T09:06:48.293 INFO:teuthology.orchestra.run.smithi088.stderr:2024-06-03T09:06:48.287+0000 7fceea937640 1 -- 172.21.15.88:0/3312006346 wait complete. 2024-06-03T09:06:48.293 INFO:teuthology.orchestra.run.smithi088.stderr:2024-06-03T09:06:48.287+0000 7fceea937640 1 librados: shutdown 2024-06-03T09:06:48.299 INFO:teuthology.orchestra.run.smithi088.stdout:{"pg_ready":true,"pg_map":{"version":1933,"stamp":"2024-06-03T09:06:47.103933+0000","last_osdmap_epoch":0,"last_pg_scan":0,"pg_stats_sum":{"stat_sum":{"num_bytes":459285,"num_objects":3,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":1,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":3,"num_whiteouts":0,"num_read":102,"num_read_kb":88,"num_write":120,"num_write_kb":1379,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2,"num_bytes_recovered":397840,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":83,"ondisk_log_size":83,"up":73,"acting":73,"num_store_stats":0},"osd_stats_sum":{"up_from":0,"seq":0,"num_pgs":82,"num_osds":8,"num_per_pool_osds":8,"num_per_pool_omap_osds":8,"kb":754974720,"kb_used":494764,"kb_used_data":46532,"kb_used_omap":12,"kb_used_meta":448115,"kb_avail":754479956,"statfs":{"total":773094113280,"available":772587474944,"internally_reserved":0,"allocated":47648768,"data_stored":22771099,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":12744,"internal_metadata":458870328},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":422,"apply_latency_ms":422,"commit_latency_ns":422000000,"apply_latency_ns":422000000},"alerts":[],"network_ping_times":[]},"pg_stats_delta":{"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":0,"acting":0,"num_store_stats":0,"stamp_delta":"4.269809"},"pg_stats":[{"pgid":"17.d","version":"0'0","reported_seq":942,"reported_epoch":1009,"state":"active+clean","last_fresh":"2024-06-03T09:06:14.443917+0000","last_change":"2024-06-03T08:38:19.260548+0000","last_active":"2024-06-03T09:06:14.443917+0000","last_peered":"2024-06-03T09:06:14.443917+0000","last_clean":"2024-06-03T09:06:14.443917+0000","last_became_active":"2024-06-03T08:38:19.253559+0000","last_became_peered":"2024-06-03T08:38:19.253559+0000","last_unstale":"2024-06-03T09:06:14.443917+0000","last_undegraded":"2024-06-03T09:06:14.443917+0000","last_fullsized":"2024-06-03T09:06:14.443917+0000","mapping_epoch":28,"log_start":"0'0","ondisk_log_start":"0'0","created":28,"last_epoch_clean":45,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-06-03T08:37:55.614497+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-06-03T08:37:55.614497+0000","last_clean_scrub_stamp":"2024-06-03T08:37:55.614497+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-06-04T19:00:12.470979+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1],"acting":[0,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"17.c","version":"0'0","reported_seq":955,"reported_epoch":1011,"state":"active+undersized","last_fresh":"2024-06-03T09:06:42.798184+0000","last_change":"2024-06-03T09:06:03.440074+0000","last_active":"2024-06-03T09:06:42.798184+0000","last_peered":"2024-06-03T09:06:42.798184+0000","last_clean":"2024-06-03T09:05:02.105638+0000","last_became_active":"2024-06-03T09:06:03.440074+0000","last_became_peered":"2024-06-03T09:06:03.440074+0000","last_unstale":"2024-06-03T09:06:42.798184+0000","last_undegraded":"2024-06-03T09:06:42.798184+0000","last_fullsized":"2024-06-03T09:06:03.377982+0000","mapping_epoch":999,"log_start":"0'0","ondisk_log_start":"0'0","created":28,"last_epoch_clean":255,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-06-03T08:37:55.614497+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-06-03T08:37:55.614497+0000","last_clean_scrub_stamp":"2024-06-03T08:37:55.614497+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-06-04T10:34:20.808557+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3],"acting":[3],"avail_no_missing":["3"],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"17.f","version":"0'0","reported_seq":843,"reported_epoch":1014,"state":"active+clean","last_fresh":"2024-06-03T09:06:46.306998+0000","last_change":"2024-06-03T09:06:46.306998+0000","last_active":"2024-06-03T09:06:46.306998+0000","last_peered":"2024-06-03T09:06:46.306998+0000","last_clean":"2024-06-03T09:06:46.306998+0000","last_became_active":"2024-06-03T09:06:46.301958+0000","last_became_peered":"2024-06-03T09:06:46.301958+0000","last_unstale":"2024-06-03T09:06:46.306998+0000","last_undegraded":"2024-06-03T09:06:46.306998+0000","last_fullsized":"2024-06-03T09:06:46.306998+0000","mapping_epoch":1013,"log_start":"0'0","ondisk_log_start":"0'0","created":28,"last_epoch_clean":1014,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-06-03T08:37:55.614497+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-06-03T08:37:55.614497+0000","last_clean_scrub_stamp":"2024-06-03T08:37:55.614497+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-06-04T14:30:23.089522+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,4],"acting":[6,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"17.e","version":"0'0","reported_seq":843,"reported_epoch":1014,"state":"active+clean","last_fresh":"2024-06-03T09:06:46.278835+0000","last_change":"2024-06-03T09:06:46.278835+0000","last_active":"2024-06-03T09:06:46.278835+0000","last_peered":"2024-06-03T09:06:46.278835+0000","last_clean":"2024-06-03T09:06:46.278835+0000","last_became_active":"2024-06-03T09:06:46.273941+0000","last_became_peered":"2024-06-03T09:06:46.273941+0000","last_unstale":"2024-06-03T09:06:46.278835+0000","last_undegraded":"2024-06-03T09:06:46.278835+0000","last_fullsized":"2024-06-03T09:06:46.278835+0000","mapping_epoch":1013,"log_start":"0'0","ondisk_log_start":"0'0","created":28,"last_epoch_clean":1014,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-06-03T08:37:55.614497+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-06-03T08:37:55.614497+0000","last_clean_scrub_stamp":"2024-06-03T08:37:55.614497+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-06-04T09:31:07.311650+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,5],"acting":[6,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"17.9","version":"0'0","reported_seq":947,"reported_epoch":1010,"state":"active+clean","last_fresh":"2024-06-03T09:06:18.488819+0000","last_change":"2024-06-03T08:38:04.772260+0000","last_active":"2024-06-03T09:06:18.488819+0000","last_peered":"2024-06-03T09:06:18.488819+0000","last_clean":"2024-06-03T09:06:18.488819+0000","last_became_active":"2024-06-03T08:38:04.763428+0000","last_became_peered":"2024-06-03T08:38:04.763428+0000","last_unstale":"2024-06-03T09:06:18.488819+0000","last_undegraded":"2024-06-03T09:06:18.488819+0000","last_fullsized":"2024-06-03T09:06:18.488819+0000","mapping_epoch":28,"log_start":"0'0","ondisk_log_start":"0'0","created":28,"last_epoch_clean":33,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-06-03T08:37:55.614497+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-06-03T08:37:55.614497+0000","last_clean_scrub_stamp":"2024-06-03T08:37:55.614497+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-06-04T10:55:10.773936+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,4],"acting":[7,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"17.8","version":"0'0","reported_seq":961,"reported_epoch":1009,"state":"active+undersized","last_fresh":"2024-06-03T09:06:14.430375+0000","last_change":"2024-06-03T09:06:03.388786+0000","last_active":"2024-06-03T09:06:14.430375+0000","last_peered":"2024-06-03T09:06:14.430375+0000","last_clean":"2024-06-03T09:04:43.297780+0000","last_became_active":"2024-06-03T09:06:03.388786+0000","last_became_peered":"2024-06-03T09:06:03.388786+0000","last_unstale":"2024-06-03T09:06:14.430375+0000","last_undegraded":"2024-06-03T09:06:14.430375+0000","last_fullsized":"2024-06-03T09:06:03.331754+0000","mapping_epoch":999,"log_start":"0'0","ondisk_log_start":"0'0","created":28,"last_epoch_clean":255,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-06-03T08:37:55.614497+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-06-03T08:37:55.614497+0000","last_clean_scrub_stamp":"2024-06-03T08:37:55.614497+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-06-04T09:18:44.765547+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0],"acting":[0],"avail_no_missing":["0"],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"17.b","version":"0'0","reported_seq":843,"reported_epoch":1014,"state":"active+clean","last_fresh":"2024-06-03T09:06:46.350450+0000","last_change":"2024-06-03T09:06:46.350450+0000","last_active":"2024-06-03T09:06:46.350450+0000","last_peered":"2024-06-03T09:06:46.350450+0000","last_clean":"2024-06-03T09:06:46.350450+0000","last_became_active":"2024-06-03T09:06:46.345458+0000","last_became_peered":"2024-06-03T09:06:46.345458+0000","last_unstale":"2024-06-03T09:06:46.350450+0000","last_undegraded":"2024-06-03T09:06:46.350450+0000","last_fullsized":"2024-06-03T09:06:46.350450+0000","mapping_epoch":1013,"log_start":"0'0","ondisk_log_start":"0'0","created":28,"last_epoch_clean":1014,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-06-03T08:37:55.614497+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-06-03T08:37:55.614497+0000","last_clean_scrub_stamp":"2024-06-03T08:37:55.614497+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-06-04T16:51:34.952224+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,2],"acting":[6,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"17.a","version":"0'0","reported_seq":927,"reported_epoch":1011,"state":"active+clean","last_fresh":"2024-06-03T09:06:42.861977+0000","last_change":"2024-06-03T08:38:05.843142+0000","last_active":"2024-06-03T09:06:42.861977+0000","last_peered":"2024-06-03T09:06:42.861977+0000","last_clean":"2024-06-03T09:06:42.861977+0000","last_became_active":"2024-06-03T08:38:05.809257+0000","last_became_peered":"2024-06-03T08:38:05.809257+0000","last_unstale":"2024-06-03T09:06:42.861977+0000","last_undegraded":"2024-06-03T09:06:42.861977+0000","last_fullsized":"2024-06-03T09:06:42.861977+0000","mapping_epoch":28,"log_start":"0'0","ondisk_log_start":"0'0","created":28,"last_epoch_clean":33,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-06-03T08:37:55.614497+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-06-03T08:37:55.614497+0000","last_clean_scrub_stamp":"2024-06-03T08:37:55.614497+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-06-04T12:12:03.686838+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2],"acting":[1,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"17.5","version":"0'0","reported_seq":943,"reported_epoch":1009,"state":"active+clean","last_fresh":"2024-06-03T09:06:14.442357+0000","last_change":"2024-06-03T08:38:23.466051+0000","last_active":"2024-06-03T09:06:14.442357+0000","last_peered":"2024-06-03T09:06:14.442357+0000","last_clean":"2024-06-03T09:06:14.442357+0000","last_became_active":"2024-06-03T08:38:23.458039+0000","last_became_peered":"2024-06-03T08:38:23.458039+0000","last_unstale":"2024-06-03T09:06:14.442357+0000","last_undegraded":"2024-06-03T09:06:14.442357+0000","last_fullsized":"2024-06-03T09:06:14.442357+0000","mapping_epoch":28,"log_start":"0'0","ondisk_log_start":"0'0","created":28,"last_epoch_clean":49,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-06-03T08:37:55.614497+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-06-03T08:37:55.614497+0000","last_clean_scrub_stamp":"2024-06-03T08:37:55.614497+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-06-04T12:27:02.532436+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,4],"acting":[0,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"17.4","version":"0'0","reported_seq":843,"reported_epoch":1014,"state":"active+clean","last_fresh":"2024-06-03T09:06:46.456388+0000","last_change":"2024-06-03T09:06:46.456388+0000","last_active":"2024-06-03T09:06:46.456388+0000","last_peered":"2024-06-03T09:06:46.456388+0000","last_clean":"2024-06-03T09:06:46.456388+0000","last_became_active":"2024-06-03T09:06:46.447534+0000","last_became_peered":"2024-06-03T09:06:46.447534+0000","last_unstale":"2024-06-03T09:06:46.456388+0000","last_undegraded":"2024-06-03T09:06:46.456388+0000","last_fullsized":"2024-06-03T09:06:46.456388+0000","mapping_epoch":1013,"log_start":"0'0","ondisk_log_start":"0'0","created":28,"last_epoch_clean":1014,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-06-03T08:37:55.614497+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-06-03T08:37:55.614497+0000","last_clean_scrub_stamp":"2024-06-03T08:37:55.614497+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-06-04T16:06:17.792570+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,7],"acting":[6,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"17.7","version":"0'0","reported_seq":929,"reported_epoch":1009,"state":"active+clean","last_fresh":"2024-06-03T09:06:13.736920+0000","last_change":"2024-06-03T08:38:04.837083+0000","last_active":"2024-06-03T09:06:13.736920+0000","last_peered":"2024-06-03T09:06:13.736920+0000","last_clean":"2024-06-03T09:06:13.736920+0000","last_became_active":"2024-06-03T08:38:04.818246+0000","last_became_peered":"2024-06-03T08:38:04.818246+0000","last_unstale":"2024-06-03T09:06:13.736920+0000","last_undegraded":"2024-06-03T09:06:13.736920+0000","last_fullsized":"2024-06-03T09:06:13.736920+0000","mapping_epoch":28,"log_start":"0'0","ondisk_log_start":"0'0","created":28,"last_epoch_clean":34,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-06-03T08:37:55.614497+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-06-03T08:37:55.614497+0000","last_clean_scrub_stamp":"2024-06-03T08:37:55.614497+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-06-04T17:02:38.204813+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,2],"acting":[5,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"17.6","version":"0'0","reported_seq":960,"reported_epoch":1009,"state":"active+undersized","last_fresh":"2024-06-03T09:06:14.453110+0000","last_change":"2024-06-03T09:06:03.428677+0000","last_active":"2024-06-03T09:06:14.453110+0000","last_peered":"2024-06-03T09:06:14.453110+0000","last_clean":"2024-06-03T09:04:43.312352+0000","last_became_active":"2024-06-03T09:06:03.428677+0000","last_became_peered":"2024-06-03T09:06:03.428677+0000","last_unstale":"2024-06-03T09:06:14.453110+0000","last_undegraded":"2024-06-03T09:06:14.453110+0000","last_fullsized":"2024-06-03T09:06:03.359909+0000","mapping_epoch":999,"log_start":"0'0","ondisk_log_start":"0'0","created":28,"last_epoch_clean":255,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-06-03T08:37:55.614497+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-06-03T08:37:55.614497+0000","last_clean_scrub_stamp":"2024-06-03T08:37:55.614497+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-06-04T15:38:02.180241+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0],"acting":[0],"avail_no_missing":["0"],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"2.7","version":"0'0","reported_seq":942,"reported_epoch":1011,"state":"active+clean","last_fresh":"2024-06-03T09:06:42.820718+0000","last_change":"2024-06-03T08:36:30.385405+0000","last_active":"2024-06-03T09:06:42.820718+0000","last_peered":"2024-06-03T09:06:42.820718+0000","last_clean":"2024-06-03T09:06:42.820718+0000","last_became_active":"2024-06-03T08:36:30.380302+0000","last_became_peered":"2024-06-03T08:36:30.380302+0000","last_unstale":"2024-06-03T09:06:42.820718+0000","last_undegraded":"2024-06-03T09:06:42.820718+0000","last_fullsized":"2024-06-03T09:06:42.820718+0000","mapping_epoch":23,"log_start":"0'0","ondisk_log_start":"0'0","created":23,"last_epoch_clean":24,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-06-03T08:36:28.444764+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-06-03T08:36:28.444764+0000","last_clean_scrub_stamp":"2024-06-03T08:36:28.444764+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-06-04T18:46:33.475181+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0],"acting":[3,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"17.14","version":"0'0","reported_seq":926,"reported_epoch":1009,"state":"active+clean","last_fresh":"2024-06-03T09:06:13.721460+0000","last_change":"2024-06-03T08:38:04.547565+0000","last_active":"2024-06-03T09:06:13.721460+0000","last_peered":"2024-06-03T09:06:13.721460+0000","last_clean":"2024-06-03T09:06:13.721460+0000","last_became_active":"2024-06-03T08:38:04.541781+0000","last_became_peered":"2024-06-03T08:38:04.541781+0000","last_unstale":"2024-06-03T09:06:13.721460+0000","last_undegraded":"2024-06-03T09:06:13.721460+0000","last_fullsized":"2024-06-03T09:06:13.721460+0000","mapping_epoch":28,"log_start":"0'0","ondisk_log_start":"0'0","created":28,"last_epoch_clean":34,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-06-03T08:37:55.614497+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-06-03T08:37:55.614497+0000","last_clean_scrub_stamp":"2024-06-03T08:37:55.614497+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-06-04T16:41:55.170134+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,2],"acting":[5,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"2.6","version":"0'0","reported_seq":950,"reported_epoch":1010,"state":"active+clean","last_fresh":"2024-06-03T09:06:19.062222+0000","last_change":"2024-06-03T08:36:30.430760+0000","last_active":"2024-06-03T09:06:19.062222+0000","last_peered":"2024-06-03T09:06:19.062222+0000","last_clean":"2024-06-03T09:06:19.062222+0000","last_became_active":"2024-06-03T08:36:30.404665+0000","last_became_peered":"2024-06-03T08:36:30.404665+0000","last_unstale":"2024-06-03T09:06:19.062222+0000","last_undegraded":"2024-06-03T09:06:19.062222+0000","last_fullsized":"2024-06-03T09:06:19.062222+0000","mapping_epoch":23,"log_start":"0'0","ondisk_log_start":"0'0","created":23,"last_epoch_clean":24,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-06-03T08:36:28.444764+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-06-03T08:36:28.444764+0000","last_clean_scrub_stamp":"2024-06-03T08:36:28.444764+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-06-04T10:37:37.746074+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,0],"acting":[4,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"17.15","version":"0'0","reported_seq":936,"reported_epoch":1011,"state":"active+clean","last_fresh":"2024-06-03T09:06:42.808375+0000","last_change":"2024-06-03T08:38:20.999903+0000","last_active":"2024-06-03T09:06:42.808375+0000","last_peered":"2024-06-03T09:06:42.808375+0000","last_clean":"2024-06-03T09:06:42.808375+0000","last_became_active":"2024-06-03T08:38:20.979523+0000","last_became_peered":"2024-06-03T08:38:20.979523+0000","last_unstale":"2024-06-03T09:06:42.808375+0000","last_undegraded":"2024-06-03T09:06:42.808375+0000","last_fullsized":"2024-06-03T09:06:42.808375+0000","mapping_epoch":28,"log_start":"0'0","ondisk_log_start":"0'0","created":28,"last_epoch_clean":47,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-06-03T08:37:55.614497+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-06-03T08:37:55.614497+0000","last_clean_scrub_stamp":"2024-06-03T08:37:55.614497+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-06-04T20:29:58.314289+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,1],"acting":[3,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"2.5","version":"0'0","reported_seq":944,"reported_epoch":1011,"state":"active+clean","last_fresh":"2024-06-03T09:06:42.818103+0000","last_change":"2024-06-03T08:36:30.024235+0000","last_active":"2024-06-03T09:06:42.818103+0000","last_peered":"2024-06-03T09:06:42.818103+0000","last_clean":"2024-06-03T09:06:42.818103+0000","last_became_active":"2024-06-03T08:36:30.019040+0000","last_became_peered":"2024-06-03T08:36:30.019040+0000","last_unstale":"2024-06-03T09:06:42.818103+0000","last_undegraded":"2024-06-03T09:06:42.818103+0000","last_fullsized":"2024-06-03T09:06:42.818103+0000","mapping_epoch":23,"log_start":"0'0","ondisk_log_start":"0'0","created":23,"last_epoch_clean":24,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-06-03T08:36:28.444764+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-06-03T08:36:28.444764+0000","last_clean_scrub_stamp":"2024-06-03T08:36:28.444764+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-06-04T15:46:47.109502+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,4],"acting":[3,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"17.16","version":"0'0","reported_seq":974,"reported_epoch":1011,"state":"active+undersized","last_fresh":"2024-06-03T09:06:42.992868+0000","last_change":"2024-06-03T09:06:03.478615+0000","last_active":"2024-06-03T09:06:42.992868+0000","last_peered":"2024-06-03T09:06:42.992868+0000","last_clean":"2024-06-03T09:04:43.328415+0000","last_became_active":"2024-06-03T09:06:03.478615+0000","last_became_peered":"2024-06-03T09:06:03.478615+0000","last_unstale":"2024-06-03T09:06:42.992868+0000","last_undegraded":"2024-06-03T09:06:42.992868+0000","last_fullsized":"2024-06-03T09:06:03.402504+0000","mapping_epoch":999,"log_start":"0'0","ondisk_log_start":"0'0","created":28,"last_epoch_clean":255,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-06-03T08:37:55.614497+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-06-03T08:37:55.614497+0000","last_clean_scrub_stamp":"2024-06-03T08:37:55.614497+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-06-04T08:47:08.079233+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2],"acting":[2],"avail_no_missing":["2"],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"2.4","version":"0'0","reported_seq":952,"reported_epoch":1010,"state":"active+undersized","last_fresh":"2024-06-03T09:06:18.331485+0000","last_change":"2024-06-03T09:06:03.387171+0000","last_active":"2024-06-03T09:06:18.331485+0000","last_peered":"2024-06-03T09:06:18.331485+0000","last_clean":"2024-06-03T09:04:43.202645+0000","last_became_active":"2024-06-03T09:06:03.387171+0000","last_became_peered":"2024-06-03T09:06:03.387171+0000","last_unstale":"2024-06-03T09:06:18.331485+0000","last_undegraded":"2024-06-03T09:06:18.331485+0000","last_fullsized":"2024-06-03T09:06:03.283787+0000","mapping_epoch":999,"log_start":"0'0","ondisk_log_start":"0'0","created":23,"last_epoch_clean":255,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-06-03T08:36:28.444764+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-06-03T08:36:28.444764+0000","last_clean_scrub_stamp":"2024-06-03T08:36:28.444764+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-06-04T20:17:43.485023+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5],"acting":[5],"avail_no_missing":["5"],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"17.17","version":"0'0","reported_seq":956,"reported_epoch":1011,"state":"active+clean","last_fresh":"2024-06-03T09:06:42.990396+0000","last_change":"2024-06-03T08:38:38.483275+0000","last_active":"2024-06-03T09:06:42.990396+0000","last_peered":"2024-06-03T09:06:42.990396+0000","last_clean":"2024-06-03T09:06:42.990396+0000","last_became_active":"2024-06-03T08:38:38.467916+0000","last_became_peered":"2024-06-03T08:38:38.467916+0000","last_unstale":"2024-06-03T09:06:42.990396+0000","last_undegraded":"2024-06-03T09:06:42.990396+0000","last_fullsized":"2024-06-03T09:06:42.990396+0000","mapping_epoch":28,"log_start":"0'0","ondisk_log_start":"0'0","created":28,"last_epoch_clean":61,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-06-03T08:37:55.614497+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-06-03T08:37:55.614497+0000","last_clean_scrub_stamp":"2024-06-03T08:37:55.614497+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-06-04T10:02:23.036420+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,7],"acting":[2,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"2.2","version":"0'0","reported_seq":965,"reported_epoch":1011,"state":"active+undersized","last_fresh":"2024-06-03T09:06:42.803839+0000","last_change":"2024-06-03T09:06:03.410020+0000","last_active":"2024-06-03T09:06:42.803839+0000","last_peered":"2024-06-03T09:06:42.803839+0000","last_clean":"2024-06-03T09:05:37.145889+0000","last_became_active":"2024-06-03T09:06:03.410020+0000","last_became_peered":"2024-06-03T09:06:03.410020+0000","last_unstale":"2024-06-03T09:06:42.803839+0000","last_undegraded":"2024-06-03T09:06:42.803839+0000","last_fullsized":"2024-06-03T09:06:03.346864+0000","mapping_epoch":999,"log_start":"0'0","ondisk_log_start":"0'0","created":23,"last_epoch_clean":255,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-06-03T08:36:28.444764+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-06-03T08:36:28.444764+0000","last_clean_scrub_stamp":"2024-06-03T08:36:28.444764+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-06-04T09:57:12.423360+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3],"acting":[3],"avail_no_missing":["3"],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"17.11","version":"0'0","reported_seq":944,"reported_epoch":1009,"state":"active+clean","last_fresh":"2024-06-03T09:06:14.426670+0000","last_change":"2024-06-03T08:38:06.502302+0000","last_active":"2024-06-03T09:06:14.426670+0000","last_peered":"2024-06-03T09:06:14.426670+0000","last_clean":"2024-06-03T09:06:14.426670+0000","last_became_active":"2024-06-03T08:38:06.490204+0000","last_became_peered":"2024-06-03T08:38:06.490204+0000","last_unstale":"2024-06-03T09:06:14.426670+0000","last_undegraded":"2024-06-03T09:06:14.426670+0000","last_fullsized":"2024-06-03T09:06:14.426670+0000","mapping_epoch":28,"log_start":"0'0","ondisk_log_start":"0'0","created":28,"last_epoch_clean":32,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-06-03T08:37:55.614497+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-06-03T08:37:55.614497+0000","last_clean_scrub_stamp":"2024-06-03T08:37:55.614497+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-06-04T10:47:05.988958+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,2],"acting":[4,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"2.1","version":"0'0","reported_seq":953,"reported_epoch":1010,"state":"active+clean","last_fresh":"2024-06-03T09:06:18.490362+0000","last_change":"2024-06-03T08:36:30.883360+0000","last_active":"2024-06-03T09:06:18.490362+0000","last_peered":"2024-06-03T09:06:18.490362+0000","last_clean":"2024-06-03T09:06:18.490362+0000","last_became_active":"2024-06-03T08:36:30.855073+0000","last_became_peered":"2024-06-03T08:36:30.855073+0000","last_unstale":"2024-06-03T09:06:18.490362+0000","last_undegraded":"2024-06-03T09:06:18.490362+0000","last_fullsized":"2024-06-03T09:06:18.490362+0000","mapping_epoch":23,"log_start":"0'0","ondisk_log_start":"0'0","created":23,"last_epoch_clean":25,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-06-03T08:36:28.444764+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-06-03T08:36:28.444764+0000","last_clean_scrub_stamp":"2024-06-03T08:36:28.444764+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-06-04T18:12:22.363473+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,5],"acting":[7,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"17.12","version":"0'0","reported_seq":946,"reported_epoch":1010,"state":"active+clean","last_fresh":"2024-06-03T09:06:18.482303+0000","last_change":"2024-06-03T08:38:01.751562+0000","last_active":"2024-06-03T09:06:18.482303+0000","last_peered":"2024-06-03T09:06:18.482303+0000","last_clean":"2024-06-03T09:06:18.482303+0000","last_became_active":"2024-06-03T08:38:01.745027+0000","last_became_peered":"2024-06-03T08:38:01.745027+0000","last_unstale":"2024-06-03T09:06:18.482303+0000","last_undegraded":"2024-06-03T09:06:18.482303+0000","last_fullsized":"2024-06-03T09:06:18.482303+0000","mapping_epoch":28,"log_start":"0'0","ondisk_log_start":"0'0","created":28,"last_epoch_clean":32,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-06-03T08:37:55.614497+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-06-03T08:37:55.614497+0000","last_clean_scrub_stamp":"2024-06-03T08:37:55.614497+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-06-04T17:54:43.785326+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,3],"acting":[7,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"2.0","version":"0'0","reported_seq":946,"reported_epoch":1012,"state":"active+clean","last_fresh":"2024-06-03T09:06:43.473891+0000","last_change":"2024-06-03T08:36:29.984096+0000","last_active":"2024-06-03T09:06:43.473891+0000","last_peered":"2024-06-03T09:06:43.473891+0000","last_clean":"2024-06-03T09:06:43.473891+0000","last_became_active":"2024-06-03T08:36:29.957875+0000","last_became_peered":"2024-06-03T08:36:29.957875+0000","last_unstale":"2024-06-03T09:06:43.473891+0000","last_undegraded":"2024-06-03T09:06:43.473891+0000","last_fullsized":"2024-06-03T09:06:43.473891+0000","mapping_epoch":23,"log_start":"0'0","ondisk_log_start":"0'0","created":23,"last_epoch_clean":24,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-06-03T08:36:28.444764+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-06-03T08:36:28.444764+0000","last_clean_scrub_stamp":"2024-06-03T08:36:28.444764+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-06-04T08:41:33.679479+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,5],"acting":[3,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"17.13","version":"0'0","reported_seq":943,"reported_epoch":1009,"state":"active+clean","last_fresh":"2024-06-03T09:06:14.438399+0000","last_change":"2024-06-03T08:38:26.838316+0000","last_active":"2024-06-03T09:06:14.438399+0000","last_peered":"2024-06-03T09:06:14.438399+0000","last_clean":"2024-06-03T09:06:14.438399+0000","last_became_active":"2024-06-03T08:38:26.833214+0000","last_became_peered":"2024-06-03T08:38:26.833214+0000","last_unstale":"2024-06-03T09:06:14.438399+0000","last_undegraded":"2024-06-03T09:06:14.438399+0000","last_fullsized":"2024-06-03T09:06:14.438399+0000","mapping_epoch":28,"log_start":"0'0","ondisk_log_start":"0'0","created":28,"last_epoch_clean":52,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-06-03T08:37:55.614497+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-06-03T08:37:55.614497+0000","last_clean_scrub_stamp":"2024-06-03T08:37:55.614497+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-06-04T12:58:51.617998+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,0],"acting":[4,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"2.3","version":"0'0","reported_seq":945,"reported_epoch":1011,"state":"active+clean","last_fresh":"2024-06-03T09:06:42.806748+0000","last_change":"2024-06-03T08:36:30.351670+0000","last_active":"2024-06-03T09:06:42.806748+0000","last_peered":"2024-06-03T09:06:42.806748+0000","last_clean":"2024-06-03T09:06:42.806748+0000","last_became_active":"2024-06-03T08:36:30.343560+0000","last_became_peered":"2024-06-03T08:36:30.343560+0000","last_unstale":"2024-06-03T09:06:42.806748+0000","last_undegraded":"2024-06-03T09:06:42.806748+0000","last_fullsized":"2024-06-03T09:06:42.806748+0000","mapping_epoch":23,"log_start":"0'0","ondisk_log_start":"0'0","created":23,"last_epoch_clean":24,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-06-03T08:36:28.444764+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-06-03T08:36:28.444764+0000","last_clean_scrub_stamp":"2024-06-03T08:36:28.444764+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-06-04T12:02:24.765135+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,2],"acting":[3,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.0","version":"19'76","reported_seq":1053,"reported_epoch":1011,"state":"active+clean","last_fresh":"2024-06-03T09:06:42.852938+0000","last_change":"2024-06-03T08:36:26.702702+0000","last_active":"2024-06-03T09:06:42.852938+0000","last_peered":"2024-06-03T09:06:42.852938+0000","last_clean":"2024-06-03T09:06:42.852938+0000","last_became_active":"2024-06-03T08:36:26.128378+0000","last_became_peered":"2024-06-03T08:36:26.128378+0000","last_unstale":"2024-06-03T09:06:42.852938+0000","last_undegraded":"2024-06-03T09:06:42.852938+0000","last_fullsized":"2024-06-03T09:06:42.852938+0000","mapping_epoch":20,"log_start":"0'0","ondisk_log_start":"0'0","created":16,"last_epoch_clean":22,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-06-03T08:36:15.542635+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-06-03T08:36:15.542635+0000","last_clean_scrub_stamp":"2024-06-03T08:36:15.542635+0000","objects_scrubbed":0,"log_size":76,"log_dups_size":0,"ondisk_log_size":76,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-06-04T12:41:53.731360+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":459280,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":96,"num_read_kb":82,"num_write":113,"num_write_kb":1372,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2,"num_bytes_recovered":397840,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,5],"acting":[1,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"17.10","version":"0'0","reported_seq":944,"reported_epoch":1009,"state":"active+clean","last_fresh":"2024-06-03T09:06:14.422035+0000","last_change":"2024-06-03T08:38:15.399189+0000","last_active":"2024-06-03T09:06:14.422035+0000","last_peered":"2024-06-03T09:06:14.422035+0000","last_clean":"2024-06-03T09:06:14.422035+0000","last_became_active":"2024-06-03T08:38:15.337723+0000","last_became_peered":"2024-06-03T08:38:15.337723+0000","last_unstale":"2024-06-03T09:06:14.422035+0000","last_undegraded":"2024-06-03T09:06:14.422035+0000","last_fullsized":"2024-06-03T09:06:14.422035+0000","mapping_epoch":28,"log_start":"0'0","ondisk_log_start":"0'0","created":28,"last_epoch_clean":42,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-06-03T08:37:55.614497+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-06-03T08:37:55.614497+0000","last_clean_scrub_stamp":"2024-06-03T08:37:55.614497+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-06-04T09:29:46.427317+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,3],"acting":[4,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"17.1a","version":"0'0","reported_seq":960,"reported_epoch":1009,"state":"active+undersized","last_fresh":"2024-06-03T09:06:14.431224+0000","last_change":"2024-06-03T09:06:03.390734+0000","last_active":"2024-06-03T09:06:14.431224+0000","last_peered":"2024-06-03T09:06:14.431224+0000","last_clean":"2024-06-03T09:05:20.322313+0000","last_became_active":"2024-06-03T09:06:03.390734+0000","last_became_peered":"2024-06-03T09:06:03.390734+0000","last_unstale":"2024-06-03T09:06:14.431224+0000","last_undegraded":"2024-06-03T09:06:14.431224+0000","last_fullsized":"2024-06-03T09:06:03.343197+0000","mapping_epoch":999,"log_start":"0'0","ondisk_log_start":"0'0","created":28,"last_epoch_clean":254,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-06-03T08:37:55.614497+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-06-03T08:37:55.614497+0000","last_clean_scrub_stamp":"2024-06-03T08:37:55.614497+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-06-04T15:56:47.628875+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4],"acting":[4],"avail_no_missing":["4"],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"17.1b","version":"0'0","reported_seq":937,"reported_epoch":1011,"state":"active+clean","last_fresh":"2024-06-03T09:06:42.815385+0000","last_change":"2024-06-03T08:38:03.115748+0000","last_active":"2024-06-03T09:06:42.815385+0000","last_peered":"2024-06-03T09:06:42.815385+0000","last_clean":"2024-06-03T09:06:42.815385+0000","last_became_active":"2024-06-03T08:38:03.110533+0000","last_became_peered":"2024-06-03T08:38:03.110533+0000","last_unstale":"2024-06-03T09:06:42.815385+0000","last_undegraded":"2024-06-03T09:06:42.815385+0000","last_fullsized":"2024-06-03T09:06:42.815385+0000","mapping_epoch":28,"log_start":"0'0","ondisk_log_start":"0'0","created":28,"last_epoch_clean":32,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-06-03T08:37:55.614497+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-06-03T08:37:55.614497+0000","last_clean_scrub_stamp":"2024-06-03T08:37:55.614497+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-06-04T12:36:47.471609+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,7],"acting":[3,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"17.18","version":"0'0","reported_seq":945,"reported_epoch":1010,"state":"active+clean","last_fresh":"2024-06-03T09:06:18.484214+0000","last_change":"2024-06-03T08:38:04.632864+0000","last_active":"2024-06-03T09:06:18.484214+0000","last_peered":"2024-06-03T09:06:18.484214+0000","last_clean":"2024-06-03T09:06:18.484214+0000","last_became_active":"2024-06-03T08:38:04.627856+0000","last_became_peered":"2024-06-03T08:38:04.627856+0000","last_unstale":"2024-06-03T09:06:18.484214+0000","last_undegraded":"2024-06-03T09:06:18.484214+0000","last_fullsized":"2024-06-03T09:06:18.484214+0000","mapping_epoch":28,"log_start":"0'0","ondisk_log_start":"0'0","created":28,"last_epoch_clean":33,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-06-03T08:37:55.614497+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-06-03T08:37:55.614497+0000","last_clean_scrub_stamp":"2024-06-03T08:37:55.614497+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-06-04T19:55:51.181598+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,4],"acting":[7,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"17.19","version":"0'0","reported_seq":936,"reported_epoch":1011,"state":"active+clean","last_fresh":"2024-06-03T09:06:42.812989+0000","last_change":"2024-06-03T08:38:01.839491+0000","last_active":"2024-06-03T09:06:42.812989+0000","last_peered":"2024-06-03T09:06:42.812989+0000","last_clean":"2024-06-03T09:06:42.812989+0000","last_became_active":"2024-06-03T08:38:01.829539+0000","last_became_peered":"2024-06-03T08:38:01.829539+0000","last_unstale":"2024-06-03T09:06:42.812989+0000","last_undegraded":"2024-06-03T09:06:42.812989+0000","last_fullsized":"2024-06-03T09:06:42.812989+0000","mapping_epoch":28,"log_start":"0'0","ondisk_log_start":"0'0","created":28,"last_epoch_clean":30,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-06-03T08:37:55.614497+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-06-03T08:37:55.614497+0000","last_clean_scrub_stamp":"2024-06-03T08:37:55.614497+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-06-04T09:10:43.860229+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,2],"acting":[3,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"17.1e","version":"0'0","reported_seq":844,"reported_epoch":1014,"state":"active+clean","last_fresh":"2024-06-03T09:06:46.297025+0000","last_change":"2024-06-03T09:06:46.297025+0000","last_active":"2024-06-03T09:06:46.297025+0000","last_peered":"2024-06-03T09:06:46.297025+0000","last_clean":"2024-06-03T09:06:46.297025+0000","last_became_active":"2024-06-03T09:06:46.288593+0000","last_became_peered":"2024-06-03T09:06:46.288593+0000","last_unstale":"2024-06-03T09:06:46.297025+0000","last_undegraded":"2024-06-03T09:06:46.297025+0000","last_fullsized":"2024-06-03T09:06:46.297025+0000","mapping_epoch":1013,"log_start":"0'0","ondisk_log_start":"0'0","created":28,"last_epoch_clean":1014,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-06-03T08:37:55.614497+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-06-03T08:37:55.614497+0000","last_clean_scrub_stamp":"2024-06-03T08:37:55.614497+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-06-04T20:09:30.144366+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,4],"acting":[6,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"17.1f","version":"0'0","reported_seq":941,"reported_epoch":1009,"state":"active+clean","last_fresh":"2024-06-03T09:06:14.450053+0000","last_change":"2024-06-03T08:38:04.383538+0000","last_active":"2024-06-03T09:06:14.450053+0000","last_peered":"2024-06-03T09:06:14.450053+0000","last_clean":"2024-06-03T09:06:14.450053+0000","last_became_active":"2024-06-03T08:38:04.367278+0000","last_became_peered":"2024-06-03T08:38:04.367278+0000","last_unstale":"2024-06-03T09:06:14.450053+0000","last_undegraded":"2024-06-03T09:06:14.450053+0000","last_fullsized":"2024-06-03T09:06:14.450053+0000","mapping_epoch":28,"log_start":"0'0","ondisk_log_start":"0'0","created":28,"last_epoch_clean":32,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-06-03T08:37:55.614497+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-06-03T08:37:55.614497+0000","last_clean_scrub_stamp":"2024-06-03T08:37:55.614497+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-06-04T10:39:26.552282+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,2],"acting":[0,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"17.1c","version":"0'0","reported_seq":944,"reported_epoch":1010,"state":"active+undersized","last_fresh":"2024-06-03T09:06:18.327492+0000","last_change":"2024-06-03T09:06:03.382210+0000","last_active":"2024-06-03T09:06:18.327492+0000","last_peered":"2024-06-03T09:06:18.327492+0000","last_clean":"2024-06-03T09:04:43.199660+0000","last_became_active":"2024-06-03T09:06:03.382210+0000","last_became_peered":"2024-06-03T09:06:03.382210+0000","last_unstale":"2024-06-03T09:06:18.327492+0000","last_undegraded":"2024-06-03T09:06:18.327492+0000","last_fullsized":"2024-06-03T09:06:03.277874+0000","mapping_epoch":999,"log_start":"0'0","ondisk_log_start":"0'0","created":28,"last_epoch_clean":255,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-06-03T08:37:55.614497+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-06-03T08:37:55.614497+0000","last_clean_scrub_stamp":"2024-06-03T08:37:55.614497+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-06-04T12:18:50.013884+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5],"acting":[5],"avail_no_missing":["5"],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"17.1d","version":"0'0","reported_seq":925,"reported_epoch":1011,"state":"active+clean","last_fresh":"2024-06-03T09:06:42.858234+0000","last_change":"2024-06-03T08:38:06.794630+0000","last_active":"2024-06-03T09:06:42.858234+0000","last_peered":"2024-06-03T09:06:42.858234+0000","last_clean":"2024-06-03T09:06:42.858234+0000","last_became_active":"2024-06-03T08:38:06.592317+0000","last_became_peered":"2024-06-03T08:38:06.592317+0000","last_unstale":"2024-06-03T09:06:42.858234+0000","last_undegraded":"2024-06-03T09:06:42.858234+0000","last_fullsized":"2024-06-03T09:06:42.858234+0000","mapping_epoch":28,"log_start":"0'0","ondisk_log_start":"0'0","created":28,"last_epoch_clean":33,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-06-03T08:37:55.614497+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-06-03T08:37:55.614497+0000","last_clean_scrub_stamp":"2024-06-03T08:37:55.614497+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-06-04T12:33:30.785369+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,7],"acting":[1,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"17.2","version":"0'0","reported_seq":927,"reported_epoch":1011,"state":"active+clean","last_fresh":"2024-06-03T09:06:42.859877+0000","last_change":"2024-06-03T08:38:14.981410+0000","last_active":"2024-06-03T09:06:42.859877+0000","last_peered":"2024-06-03T09:06:42.859877+0000","last_clean":"2024-06-03T09:06:42.859877+0000","last_became_active":"2024-06-03T08:38:14.973825+0000","last_became_peered":"2024-06-03T08:38:14.973825+0000","last_unstale":"2024-06-03T09:06:42.859877+0000","last_undegraded":"2024-06-03T09:06:42.859877+0000","last_fullsized":"2024-06-03T09:06:42.859877+0000","mapping_epoch":28,"log_start":"0'0","ondisk_log_start":"0'0","created":28,"last_epoch_clean":41,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-06-03T08:37:55.614497+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-06-03T08:37:55.614497+0000","last_clean_scrub_stamp":"2024-06-03T08:37:55.614497+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-06-04T17:01:05.518613+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,4],"acting":[1,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"17.3","version":"0'0","reported_seq":943,"reported_epoch":1009,"state":"active+clean","last_fresh":"2024-06-03T09:06:14.437636+0000","last_change":"2024-06-03T08:38:14.619745+0000","last_active":"2024-06-03T09:06:14.437636+0000","last_peered":"2024-06-03T09:06:14.437636+0000","last_clean":"2024-06-03T09:06:14.437636+0000","last_became_active":"2024-06-03T08:38:14.510665+0000","last_became_peered":"2024-06-03T08:38:14.510665+0000","last_unstale":"2024-06-03T09:06:14.437636+0000","last_undegraded":"2024-06-03T09:06:14.437636+0000","last_fullsized":"2024-06-03T09:06:14.437636+0000","mapping_epoch":28,"log_start":"0'0","ondisk_log_start":"0'0","created":28,"last_epoch_clean":42,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-06-03T08:37:55.614497+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-06-03T08:37:55.614497+0000","last_clean_scrub_stamp":"2024-06-03T08:37:55.614497+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-06-04T12:03:47.392064+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,7],"acting":[0,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"17.0","version":"0'0","reported_seq":944,"reported_epoch":1009,"state":"active+clean","last_fresh":"2024-06-03T09:06:14.427123+0000","last_change":"2024-06-03T08:38:04.237118+0000","last_active":"2024-06-03T09:06:14.427123+0000","last_peered":"2024-06-03T09:06:14.427123+0000","last_clean":"2024-06-03T09:06:14.427123+0000","last_became_active":"2024-06-03T08:38:04.231954+0000","last_became_peered":"2024-06-03T08:38:04.231954+0000","last_unstale":"2024-06-03T09:06:14.427123+0000","last_undegraded":"2024-06-03T09:06:14.427123+0000","last_fullsized":"2024-06-03T09:06:14.427123+0000","mapping_epoch":28,"log_start":"0'0","ondisk_log_start":"0'0","created":28,"last_epoch_clean":32,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-06-03T08:37:55.614497+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-06-03T08:37:55.614497+0000","last_clean_scrub_stamp":"2024-06-03T08:37:55.614497+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-06-04T09:56:10.741637+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,2],"acting":[0,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"17.1","version":"65'7","reported_seq":959,"reported_epoch":1009,"state":"active+undersized+degraded","last_fresh":"2024-06-03T09:06:13.709209+0000","last_change":"2024-06-03T09:06:03.370660+0000","last_active":"2024-06-03T09:06:13.709209+0000","last_peered":"2024-06-03T09:06:13.709209+0000","last_clean":"2024-06-03T09:05:03.733864+0000","last_became_active":"2024-06-03T09:06:03.370660+0000","last_became_peered":"2024-06-03T09:06:03.370660+0000","last_unstale":"2024-06-03T09:06:13.709209+0000","last_undegraded":"2024-06-03T09:06:03.266076+0000","last_fullsized":"2024-06-03T09:06:03.264525+0000","mapping_epoch":999,"log_start":"0'0","ondisk_log_start":"0'0","created":28,"last_epoch_clean":255,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-06-03T08:37:55.614497+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-06-03T08:37:55.614497+0000","last_clean_scrub_stamp":"2024-06-03T08:37:55.614497+0000","objects_scrubbed":0,"log_size":7,"log_dups_size":0,"ondisk_log_size":7,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-06-04T11:55:18.194283+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":5,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":1,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":6,"num_read_kb":6,"num_write":7,"num_write_kb":7,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5],"acting":[5],"avail_no_missing":["5"],"object_location_counts":[{"shards":"5","objects":1}],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]}],"pool_stats":[{"poolid":2,"num_pg":8,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":14,"acting":14,"num_store_stats":7},{"poolid":1,"num_pg":1,"stat_sum":{"num_bytes":459280,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":96,"num_read_kb":82,"num_write":113,"num_write_kb":1372,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2,"num_bytes_recovered":397840,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":802816,"data_stored":795680,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":76,"ondisk_log_size":76,"up":2,"acting":2,"num_store_stats":2},{"poolid":17,"num_pg":32,"stat_sum":{"num_bytes":5,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":1,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":6,"num_read_kb":6,"num_write":7,"num_write_kb":7,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":8192,"data_stored":10,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":7,"ondisk_log_size":7,"up":57,"acting":57,"num_store_stats":8}],"osd_stats":[{"osd":6,"up_from":1013,"seq":4350801871162,"num_pgs":14,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":54908,"kb_used_data":5744,"kb_used_omap":1,"kb_used_meta":49150,"kb_avail":94316932,"statfs":{"total":96636764160,"available":96580538368,"internally_reserved":0,"allocated":5881856,"data_stored":2757601,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1593,"internal_metadata":50330055},"hb_peers":[0,1,2,3,4,5,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":403,"apply_latency_ms":403,"commit_latency_ns":403000000,"apply_latency_ns":403000000},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Mon Jun 3 09:06:42 2024","interfaces":[{"interface":"back","average":{"1min":5696.3109999999997,"5min":1469.3900000000001,"15min":503.11700000000002},"min":{"1min":5.242,"5min":4.4829999999999997,"15min":4.1959999999999997},"max":{"1min":20463.029999999999,"5min":22766.882000000001,"15min":22766.882000000001},"last":7.3399999999999999},{"interface":"front","average":{"1min":5687.6390000000001,"5min":1468.1669999999999,"15min":504.13200000000001},"min":{"1min":4.835,"5min":4.8129999999999997,"15min":4.4530000000000003},"max":{"1min":20461.830000000002,"5min":22766.289000000001,"15min":22766.289000000001},"last":8.1010000000000009}]},{"osd":1,"last update":"Mon Jun 3 09:06:42 2024","interfaces":[{"interface":"back","average":{"1min":36.496000000000002,"5min":80.162999999999997,"15min":134.59399999999999},"min":{"1min":5.9189999999999996,"5min":5.8799999999999999,"15min":5.7119999999999997},"max":{"1min":544.85799999999995,"5min":4138.1679999999997,"15min":9955.1299999999992},"last":9.4870000000000001},{"interface":"front","average":{"1min":38.866999999999997,"5min":82.631,"15min":135.11500000000001},"min":{"1min":6.5350000000000001,"5min":6.5350000000000001,"15min":6.1390000000000002},"max":{"1min":546.73099999999999,"5min":4137.6530000000002,"15min":9827.5509999999995},"last":15.617000000000001}]},{"osd":2,"last update":"Mon Jun 3 09:06:42 2024","interfaces":[{"interface":"back","average":{"1min":41.093000000000004,"5min":27.917000000000002,"15min":22.329000000000001},"min":{"1min":6.0670000000000002,"5min":6.0670000000000002,"15min":6.0670000000000002},"max":{"1min":548.12900000000002,"5min":548.12900000000002,"15min":548.12900000000002},"last":27.196999999999999},{"interface":"front","average":{"1min":30.798999999999999,"5min":24.276,"15min":22.488},"min":{"1min":6.4370000000000003,"5min":6.4370000000000003,"15min":6.1870000000000003},"max":{"1min":369.14499999999998,"5min":369.14499999999998,"15min":434.77300000000002},"last":28.960999999999999}]},{"osd":3,"last update":"Mon Jun 3 09:06:42 2024","interfaces":[{"interface":"back","average":{"1min":41.411999999999999,"5min":27.434999999999999,"15min":30.657},"min":{"1min":10.061,"5min":8.3179999999999996,"15min":6.1719999999999997},"max":{"1min":549.024,"5min":549.024,"15min":1235.5799999999999},"last":16.423999999999999},{"interface":"front","average":{"1min":37.850999999999999,"5min":22.001000000000001,"15min":28.015999999999998},"min":{"1min":6.0679999999999996,"5min":6.0679999999999996,"15min":6.0679999999999996},"max":{"1min":559.25,"5min":559.25,"15min":1157.6780000000001},"last":14.327}]},{"osd":4,"last update":"Mon Jun 3 09:06:42 2024","interfaces":[{"interface":"back","average":{"1min":165.62899999999999,"5min":1281.1010000000001,"15min":440.59199999999998},"min":{"1min":10.965999999999999,"5min":6.093,"15min":6.093},"max":{"1min":384.46899999999999,"5min":28093.096000000001,"15min":28093.096000000001},"last":11.307},{"interface":"front","average":{"1min":186.97900000000001,"5min":1308.886,"15min":454.97699999999998},"min":{"1min":13.581,"5min":7.7480000000000002,"15min":7.7480000000000002},"max":{"1min":445.096,"5min":28323.837,"15min":28323.837},"last":17.109000000000002}]},{"osd":5,"last update":"Mon Jun 3 09:06:42 2024","interfaces":[{"interface":"back","average":{"1min":81.581000000000003,"5min":34.863,"15min":28.350999999999999},"min":{"1min":10.345000000000001,"5min":8.625,"15min":7.516},"max":{"1min":455.84199999999998,"5min":455.84199999999998,"15min":455.84199999999998},"last":17.791},{"interface":"front","average":{"1min":83.391000000000005,"5min":39.277999999999999,"15min":27.885000000000002},"min":{"1min":8.0280000000000005,"5min":6.266,"15min":6.266},"max":{"1min":550.02300000000002,"5min":550.02300000000002,"15min":550.02300000000002},"last":11.814}]},{"osd":7,"last update":"Mon Jun 3 09:06:42 2024","interfaces":[{"interface":"back","average":{"1min":44.052999999999997,"5min":33.697000000000003,"15min":29.731999999999999},"min":{"1min":9.1500000000000004,"5min":9.1500000000000004,"15min":9.1500000000000004},"max":{"1min":552.09500000000003,"5min":552.09500000000003,"15min":552.09500000000003},"last":18.486999999999998},{"interface":"front","average":{"1min":47.076999999999998,"5min":34.226999999999997,"15min":29.827000000000002},"min":{"1min":9.7780000000000005,"5min":8.6709999999999994,"15min":8.5820000000000007},"max":{"1min":531.08600000000001,"5min":531.08600000000001,"15min":531.08600000000001},"last":13.776}]}]},{"osd":7,"up_from":20,"seq":85899346290,"num_pgs":9,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":58964,"kb_used_data":5704,"kb_used_omap":1,"kb_used_meta":53246,"kb_avail":94312876,"statfs":{"total":96636764160,"available":96576385024,"internally_reserved":0,"allocated":5840896,"data_stored":2740727,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1589,"internal_metadata":54524363},"hb_peers":[0,1,2,3,4,5],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":5,"apply_latency_ms":5,"commit_latency_ns":5000000,"apply_latency_ns":5000000},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Mon Jun 3 09:06:09 2024","interfaces":[{"interface":"back","average":{"1min":14.073,"5min":894.22400000000005,"15min":308.93400000000003},"min":{"1min":5.5709999999999997,"5min":4.218,"15min":4.0599999999999996},"max":{"1min":90.757999999999996,"5min":21729.001,"15min":21729.001},"last":9.4469999999999992},{"interface":"front","average":{"1min":9.2550000000000008,"5min":906.23500000000001,"15min":314.06},"min":{"1min":5.5789999999999997,"5min":4.8810000000000002,"15min":4.8810000000000002},"max":{"1min":25.937000000000001,"5min":21899.732,"15min":21899.732},"last":4.5979999999999999}]},{"osd":1,"last update":"Mon Jun 3 09:06:09 2024","interfaces":[{"interface":"back","average":{"1min":15.182,"5min":69.183999999999997,"15min":219.173},"min":{"1min":7.3639999999999999,"5min":6.6200000000000001,"15min":6.1319999999999997},"max":{"1min":76.926000000000002,"5min":2795.1329999999998,"15min":18373.257000000001},"last":5.8630000000000004},{"interface":"front","average":{"1min":21.495999999999999,"5min":74.703999999999994,"15min":225.285},"min":{"1min":8.8399999999999999,"5min":8.8399999999999999,"15min":8.5129999999999999},"max":{"1min":94.977999999999994,"5min":2803.3670000000002,"15min":18374.601999999999},"last":10.94}]},{"osd":2,"last update":"Mon Jun 3 09:06:09 2024","interfaces":[{"interface":"back","average":{"1min":19.164000000000001,"5min":26.902000000000001,"15min":22.920999999999999},"min":{"1min":11.207000000000001,"5min":8.5030000000000001,"15min":8.5030000000000001},"max":{"1min":91.319999999999993,"5min":333.99200000000002,"15min":333.99200000000002},"last":10.095000000000001},{"interface":"front","average":{"1min":11.125999999999999,"5min":18.132000000000001,"15min":16.353000000000002},"min":{"1min":6.202,"5min":5.8369999999999997,"15min":5.5069999999999997},"max":{"1min":28.649000000000001,"5min":226.18100000000001,"15min":226.18100000000001},"last":12.922000000000001}]},{"osd":3,"last update":"Mon Jun 3 09:06:09 2024","interfaces":[{"interface":"back","average":{"1min":22.291,"5min":26.289999999999999,"15min":31.074999999999999},"min":{"1min":11.714,"5min":6.8959999999999999,"15min":6.3760000000000003},"max":{"1min":83.102000000000004,"5min":316.64400000000001,"15min":2039.9059999999999},"last":7.617},{"interface":"front","average":{"1min":13.324999999999999,"5min":17.963000000000001,"15min":25.969999999999999},"min":{"1min":6.1200000000000001,"5min":5.2089999999999996,"15min":5.2089999999999996},"max":{"1min":49.908999999999999,"5min":233.542,"15min":2037.5550000000001},"last":8.1479999999999997}]},{"osd":4,"last update":"Mon Jun 3 09:06:29 2024","interfaces":[{"interface":"back","average":{"1min":17.675999999999998,"5min":1026.325,"15min":472.47199999999998},"min":{"1min":7.306,"5min":5.9089999999999998,"15min":5.9089999999999998},"max":{"1min":140.03299999999999,"5min":28969.369999999999,"15min":30073.917000000001},"last":12.000999999999999},{"interface":"front","average":{"1min":17.890999999999998,"5min":1044.5139999999999,"15min":480.226},"min":{"1min":5.9279999999999999,"5min":5.9279999999999999,"15min":5.9279999999999999},"max":{"1min":141.179,"5min":29244.138999999999,"15min":30348.021000000001},"last":15.622}]},{"osd":5,"last update":"Mon Jun 3 09:06:02 2024","interfaces":[{"interface":"back","average":{"1min":24.372,"5min":26.856000000000002,"15min":23.571999999999999},"min":{"1min":12.278,"5min":9.2460000000000004,"15min":6.5650000000000004},"max":{"1min":183.64099999999999,"5min":317.572,"15min":317.572},"last":11.449999999999999},{"interface":"front","average":{"1min":21.259,"5min":23.364000000000001,"15min":21.193999999999999},"min":{"1min":7.8499999999999996,"5min":7.0999999999999996,"15min":7.008},"max":{"1min":184.53399999999999,"5min":306.60599999999999,"15min":306.60599999999999},"last":14.676}]},{"osd":6,"last update":"Mon Jun 3 09:04:31 2024","interfaces":[{"interface":"back","average":{"1min":44.061,"5min":26.821000000000002,"15min":91.176000000000002},"min":{"1min":9.0800000000000001,"5min":7.9390000000000001,"15min":7.9390000000000001},"max":{"1min":268.029,"5min":308.916,"15min":8734.4639999999999},"last":6.6769999999999996},{"interface":"front","average":{"1min":46.643000000000001,"5min":23.879999999999999,"15min":88.765000000000001},"min":{"1min":8.8119999999999994,"5min":6.9240000000000004,"15min":6.742},"max":{"1min":305.64600000000002,"5min":305.64600000000002,"15min":8735.5319999999992},"last":12.327}]}]},{"osd":5,"up_from":20,"seq":85899346264,"num_pgs":9,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":59360,"kb_used_data":6100,"kb_used_omap":1,"kb_used_meta":53246,"kb_avail":94312480,"statfs":{"total":96636764160,"available":96575979520,"internally_reserved":0,"allocated":6246400,"data_stored":3138572,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1594,"internal_metadata":54524358},"hb_peers":[0,1,2,3,4,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Mon Jun 3 09:06:06 2024","interfaces":[{"interface":"back","average":{"1min":7.468,"5min":1703.037,"15min":580.51900000000001},"min":{"1min":4.6509999999999998,"5min":4.6509999999999998,"15min":4.4020000000000001},"max":{"1min":14.711,"5min":26433.764999999999,"15min":26433.764999999999},"last":5.8639999999999999},{"interface":"front","average":{"1min":9.9429999999999996,"5min":1725.27,"15min":587.87800000000004},"min":{"1min":5.5999999999999996,"5min":5.1319999999999997,"15min":4.8170000000000002},"max":{"1min":13.590999999999999,"5min":26704.698,"15min":26704.698},"last":12.747999999999999}]},{"osd":1,"last update":"Mon Jun 3 09:06:00 2024","interfaces":[{"interface":"back","average":{"1min":10.862,"5min":67.731999999999999,"15min":217.364},"min":{"1min":7.1769999999999996,"5min":6.0890000000000004,"15min":5.4930000000000003},"max":{"1min":16.998999999999999,"5min":3923.4090000000001,"15min":18811.696},"last":14.052},{"interface":"front","average":{"1min":10.417,"5min":65.141999999999996,"15min":220.75700000000001},"min":{"1min":5.3209999999999997,"5min":5.3209999999999997,"15min":5.0209999999999999},"max":{"1min":15.821999999999999,"5min":4019.6129999999998,"15min":18968.365000000002},"last":8.7219999999999995}]},{"osd":2,"last update":"Mon Jun 3 09:06:00 2024","interfaces":[{"interface":"back","average":{"1min":11.366,"5min":24.18,"15min":21.137},"min":{"1min":5.6369999999999996,"5min":5.6369999999999996,"15min":5.0510000000000002},"max":{"1min":15.529999999999999,"5min":332.48200000000003,"15min":332.48200000000003},"last":12.167},{"interface":"front","average":{"1min":10.089,"5min":23.117000000000001,"15min":24.050000000000001},"min":{"1min":6.6769999999999996,"5min":6.6769999999999996,"15min":5.8739999999999997},"max":{"1min":15.821,"5min":353.03199999999998,"15min":353.03199999999998},"last":7.2619999999999996}]},{"osd":3,"last update":"Mon Jun 3 09:06:00 2024","interfaces":[{"interface":"back","average":{"1min":13.472,"5min":24.283999999999999,"15min":31.666},"min":{"1min":10.111000000000001,"5min":7.4720000000000004,"15min":7.4210000000000003},"max":{"1min":16.271999999999998,"5min":353.57499999999999,"15min":2855.8519999999999},"last":8.1140000000000008},{"interface":"front","average":{"1min":9.5899999999999999,"5min":15.875,"15min":25.834},"min":{"1min":6.6609999999999996,"5min":5.7729999999999997,"15min":5.1909999999999998},"max":{"1min":15.112,"5min":203.804,"15min":2869.7860000000001},"last":9.3699999999999992}]},{"osd":4,"last update":"Mon Jun 3 09:06:29 2024","interfaces":[{"interface":"back","average":{"1min":11.067,"5min":780.21000000000004,"15min":377.28399999999999},"min":{"1min":5.4779999999999998,"5min":5.4779999999999998,"15min":5.4779999999999998},"max":{"1min":14.654999999999999,"5min":23315.438999999998,"15min":28619.825000000001},"last":6.7199999999999998},{"interface":"front","average":{"1min":14.349,"5min":782.97299999999996,"15min":377.93400000000003},"min":{"1min":7.492,"5min":6.5030000000000001,"15min":6.5030000000000001},"max":{"1min":17.707000000000001,"5min":23317.502,"15min":28621.294000000002},"last":11.621}]},{"osd":6,"last update":"Mon Jun 3 09:03:57 2024","interfaces":[{"interface":"back","average":{"1min":23.536000000000001,"5min":36.457000000000001,"15min":83.799999999999997},"min":{"1min":9.4800000000000004,"5min":9.4800000000000004,"15min":8.0630000000000006},"max":{"1min":89.442999999999998,"5min":508.77999999999997,"15min":9193.5169999999998},"last":157.79300000000001},{"interface":"front","average":{"1min":18.891999999999999,"5min":22.100999999999999,"15min":75.489999999999995},"min":{"1min":5.923,"5min":5.923,"15min":5.923},"max":{"1min":78.424999999999997,"5min":464.62,"15min":9191.4429999999993},"last":148.09}]},{"osd":7,"last update":"Mon Jun 3 09:06:00 2024","interfaces":[{"interface":"back","average":{"1min":13.257,"5min":20.917999999999999,"15min":21.088999999999999},"min":{"1min":10.475,"5min":6.7629999999999999,"15min":6.7629999999999999},"max":{"1min":17.245000000000001,"5min":211.22999999999999,"15min":211.22999999999999},"last":11.061999999999999},{"interface":"front","average":{"1min":12.872999999999999,"5min":20.587,"15min":24.021000000000001},"min":{"1min":8.5760000000000005,"5min":7.7709999999999999,"15min":6.8570000000000002},"max":{"1min":18.359999999999999,"5min":223.191,"15min":286.69499999999999},"last":10.021000000000001}]}]},{"osd":4,"up_from":20,"seq":85899346272,"num_pgs":12,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":63060,"kb_used_data":5704,"kb_used_omap":1,"kb_used_meta":57342,"kb_avail":94308780,"statfs":{"total":96636764160,"available":96572190720,"internally_reserved":0,"allocated":5840896,"data_stored":2740727,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1594,"internal_metadata":58718662},"hb_peers":[0,1,2,3,5,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Mon Jun 3 09:06:35 2024","interfaces":[{"interface":"back","average":{"1min":10.417,"5min":1093.547,"15min":381.64499999999998},"min":{"1min":4.6900000000000004,"5min":4.5279999999999996,"15min":4.3220000000000001},"max":{"1min":52.241,"5min":21726.743999999999,"15min":21726.743999999999},"last":9.0980000000000008},{"interface":"front","average":{"1min":13.175000000000001,"5min":1102.4200000000001,"15min":387.161},"min":{"1min":6.1950000000000003,"5min":5.8380000000000001,"15min":5.4909999999999997},"max":{"1min":53.148000000000003,"5min":21727.993999999999,"15min":21727.993999999999},"last":9.6370000000000005}]},{"osd":1,"last update":"Mon Jun 3 09:06:35 2024","interfaces":[{"interface":"back","average":{"1min":9.9909999999999997,"5min":53.250999999999998,"15min":340.02999999999997},"min":{"1min":4.5670000000000002,"5min":4.1970000000000001,"15min":4.1970000000000001},"max":{"1min":47.073,"5min":3346.4389999999999,"15min":22256.098000000002},"last":5.7030000000000003},{"interface":"front","average":{"1min":12.189,"5min":53.375999999999998,"15min":339.68799999999999},"min":{"1min":6.1539999999999999,"5min":5.0590000000000002,"15min":4.2640000000000002},"max":{"1min":49.558,"5min":3347.181,"15min":22247.317999999999},"last":13.794}]},{"osd":2,"last update":"Mon Jun 3 09:06:35 2024","interfaces":[{"interface":"back","average":{"1min":15.178000000000001,"5min":17.202000000000002,"15min":20.417999999999999},"min":{"1min":6.3010000000000002,"5min":6.3010000000000002,"15min":6.3010000000000002},"max":{"1min":85.340000000000003,"5min":337.60399999999998,"15min":393.58600000000001},"last":11.301},{"interface":"front","average":{"1min":15.423,"5min":14.432,"15min":22.754000000000001},"min":{"1min":6.4889999999999999,"5min":6.0629999999999997,"15min":6.0629999999999997},"max":{"1min":82.120000000000005,"5min":127.913,"15min":394.471},"last":14.632}]},{"osd":3,"last update":"Mon Jun 3 09:06:35 2024","interfaces":[{"interface":"back","average":{"1min":17.651,"5min":15.5,"15min":18.079000000000001},"min":{"1min":10.359999999999999,"5min":5.8719999999999999,"15min":5.4000000000000004},"max":{"1min":84.555999999999997,"5min":127.43600000000001,"15min":322.63},"last":15.337999999999999},{"interface":"front","average":{"1min":16.413,"5min":17.356999999999999,"15min":22.920000000000002},"min":{"1min":9.7349999999999994,"5min":7.484,"15min":7.484},"max":{"1min":71.394000000000005,"5min":204.517,"15min":392.67500000000001},"last":12.705}]},{"osd":5,"last update":"Mon Jun 3 09:06:35 2024","interfaces":[{"interface":"back","average":{"1min":15.672000000000001,"5min":16.742999999999999,"15min":20.795000000000002},"min":{"1min":8.093,"5min":6.7489999999999997,"15min":6.7489999999999997},"max":{"1min":51.326000000000001,"5min":268.10199999999998,"15min":344.23399999999998},"last":16.116},{"interface":"front","average":{"1min":17.297999999999998,"5min":19.356999999999999,"15min":25.713999999999999},"min":{"1min":12.875,"5min":8.5180000000000007,"15min":8.5180000000000007},"max":{"1min":40.829000000000001,"5min":264.75,"15min":345.18700000000001},"last":17.427}]},{"osd":6,"last update":"Mon Jun 3 09:04:32 2024","interfaces":[{"interface":"back","average":{"1min":15.548,"5min":32.957999999999998,"15min":152.738},"min":{"1min":9.141,"5min":9.141,"15min":8.2319999999999993},"max":{"1min":52.57,"5min":395.238,"15min":12496.335999999999},"last":208.90000000000001},{"interface":"front","average":{"1min":12.680999999999999,"5min":25.428000000000001,"15min":149.30600000000001},"min":{"1min":6.4379999999999997,"5min":5.6319999999999997,"15min":5.6319999999999997},"max":{"1min":46.283000000000001,"5min":377.74299999999999,"15min":12616.297},"last":205.88900000000001}]},{"osd":7,"last update":"Mon Jun 3 09:06:35 2024","interfaces":[{"interface":"back","average":{"1min":9.6969999999999992,"5min":14.196,"15min":18.809999999999999},"min":{"1min":6.7190000000000003,"5min":6.2110000000000003,"15min":5.8010000000000002},"max":{"1min":15.960000000000001,"5min":207.25299999999999,"15min":358.94900000000001},"last":10.683999999999999},{"interface":"front","average":{"1min":17.696000000000002,"5min":19.024999999999999,"15min":23.259},"min":{"1min":11.794,"5min":7.7789999999999999,"15min":7.7789999999999999},"max":{"1min":45.639000000000003,"5min":209.32900000000001,"15min":380.06900000000002},"last":16.852}]}]},{"osd":2,"up_from":20,"seq":85899346292,"num_pgs":11,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":58988,"kb_used_data":5728,"kb_used_omap":1,"kb_used_meta":53246,"kb_avail":94312852,"statfs":{"total":96636764160,"available":96576360448,"internally_reserved":0,"allocated":5865472,"data_stored":2751635,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1594,"internal_metadata":54524358},"hb_peers":[0,1,3,4,5,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":4,"apply_latency_ms":4,"commit_latency_ns":4000000,"apply_latency_ns":4000000},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Mon Jun 3 09:06:23 2024","interfaces":[{"interface":"back","average":{"1min":8.9629999999999992,"5min":946.25400000000002,"15min":326.36099999999999},"min":{"1min":5.3479999999999999,"5min":4.7050000000000001,"15min":4.1420000000000003},"max":{"1min":26.015000000000001,"5min":23465.18,"15min":23465.18},"last":8.7959999999999994},{"interface":"front","average":{"1min":7.6130000000000004,"5min":944.39800000000002,"15min":326.178},"min":{"1min":4.6790000000000003,"5min":4.6790000000000003,"15min":4.6790000000000003},"max":{"1min":20.802,"5min":23465.966,"15min":23465.966},"last":5.6059999999999999}]},{"osd":1,"last update":"Mon Jun 3 09:06:23 2024","interfaces":[{"interface":"back","average":{"1min":13.007,"5min":76.507000000000005,"15min":294.72899999999998},"min":{"1min":6.4089999999999998,"5min":5.4199999999999999,"15min":5.4199999999999999},"max":{"1min":85.430000000000007,"5min":4220.5290000000005,"15min":18954.987000000001},"last":11.789},{"interface":"front","average":{"1min":15.894,"5min":82.119,"15min":302.92399999999998},"min":{"1min":7.367,"5min":7.367,"15min":7.367},"max":{"1min":98.498000000000005,"5min":4325.9489999999996,"15min":19143.834999999999},"last":15.368}]},{"osd":3,"last update":"Mon Jun 3 09:06:23 2024","interfaces":[{"interface":"back","average":{"1min":9.7579999999999991,"5min":12.444000000000001,"15min":28.065000000000001},"min":{"1min":7,"5min":5.9290000000000003,"15min":5.2450000000000001},"max":{"1min":28.164000000000001,"5min":118.913,"15min":2594.8150000000001},"last":13.339},{"interface":"front","average":{"1min":12.321999999999999,"5min":15.25,"15min":31.349},"min":{"1min":9.1240000000000006,"5min":8.0760000000000005,"15min":6.0780000000000003},"max":{"1min":32.360999999999997,"5min":156.34700000000001,"15min":2706.5120000000002},"last":14.712}]},{"osd":4,"last update":"Mon Jun 3 09:05:57 2024","interfaces":[{"interface":"back","average":{"1min":10.573,"5min":1266.894,"15min":438.154},"min":{"1min":8.6379999999999999,"5min":7.8789999999999996,"15min":6.2039999999999997},"max":{"1min":15.105,"5min":26660.861000000001,"15min":26660.861000000001},"last":7.8780000000000001},{"interface":"front","average":{"1min":12.407999999999999,"5min":1266.7460000000001,"15min":435.75400000000002},"min":{"1min":10.266999999999999,"5min":10.266999999999999,"15min":6.2720000000000002},"max":{"1min":15.228,"5min":26616.243999999999,"15min":26616.243999999999},"last":10.970000000000001}]},{"osd":5,"last update":"Mon Jun 3 09:06:23 2024","interfaces":[{"interface":"back","average":{"1min":14.103,"5min":18.091000000000001,"15min":21.119},"min":{"1min":9.6349999999999998,"5min":9.2159999999999993,"15min":6.2050000000000001},"max":{"1min":32.890000000000001,"5min":252.31100000000001,"15min":252.31100000000001},"last":9.3249999999999993},{"interface":"front","average":{"1min":11.545999999999999,"5min":16.170999999999999,"15min":20.254000000000001},"min":{"1min":7.6070000000000002,"5min":7.4850000000000003,"15min":6.8810000000000002},"max":{"1min":29.228999999999999,"5min":258.31700000000001,"15min":258.31700000000001},"last":7.0940000000000003}]},{"osd":6,"last update":"Mon Jun 3 09:04:28 2024","interfaces":[{"interface":"back","average":{"1min":14.063000000000001,"5min":24.850000000000001,"15min":106.755},"min":{"1min":9.0820000000000007,"5min":8.9649999999999999,"15min":8.9649999999999999},"max":{"1min":24.666,"5min":188.06700000000001,"15min":9521.0010000000002},"last":183.60900000000001},{"interface":"front","average":{"1min":13.933999999999999,"5min":21.081,"15min":106.142},"min":{"1min":5.8719999999999999,"5min":5.8719999999999999,"15min":5.8719999999999999},"max":{"1min":117.551,"5min":193.887,"15min":9585.6620000000003},"last":184.20599999999999}]},{"osd":7,"last update":"Mon Jun 3 09:06:23 2024","interfaces":[{"interface":"back","average":{"1min":17.056000000000001,"5min":20.055,"15min":23.268999999999998},"min":{"1min":10.353999999999999,"5min":6.6130000000000004,"15min":6.6130000000000004},"max":{"1min":88.974999999999994,"5min":242.56800000000001,"15min":242.56800000000001},"last":12.818},{"interface":"front","average":{"1min":18.885000000000002,"5min":21.460000000000001,"15min":25.489999999999998},"min":{"1min":12.497,"5min":9.0660000000000007,"15min":9.0660000000000007},"max":{"1min":102.229,"5min":181.56399999999999,"15min":214.33799999999999},"last":12.301}]}]},{"osd":3,"up_from":14,"seq":60129542506,"num_pgs":11,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":58992,"kb_used_data":5728,"kb_used_omap":1,"kb_used_meta":53246,"kb_avail":94312848,"statfs":{"total":96636764160,"available":96576356352,"internally_reserved":0,"allocated":5865472,"data_stored":2751635,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1592,"internal_metadata":54524360},"hb_peers":[0,1,2,4,5,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":5,"apply_latency_ms":5,"commit_latency_ns":5000000,"apply_latency_ns":5000000},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Mon Jun 3 09:06:25 2024","interfaces":[{"interface":"back","average":{"1min":7.79,"5min":1060.1959999999999,"15min":367.01799999999997},"min":{"1min":5.6909999999999998,"5min":5.173,"15min":4.6929999999999996},"max":{"1min":10.59,"5min":23275.167000000001,"15min":23275.167000000001},"last":8.5389999999999997},{"interface":"front","average":{"1min":11.215999999999999,"5min":1066.5799999999999,"15min":372.476},"min":{"1min":8.7080000000000002,"5min":7.2510000000000003,"15min":5.9880000000000004},"max":{"1min":13.499000000000001,"5min":23278.536,"15min":23278.536},"last":13.473000000000001}]},{"osd":1,"last update":"Mon Jun 3 09:06:25 2024","interfaces":[{"interface":"back","average":{"1min":7.3760000000000003,"5min":15.614000000000001,"15min":389.83100000000002},"min":{"1min":4.335,"5min":4.335,"15min":4.335},"max":{"1min":11.289,"5min":250.72300000000001,"15min":21658.205999999998},"last":5.8440000000000003},{"interface":"front","average":{"1min":8.9499999999999993,"5min":14.135,"15min":381.10300000000001},"min":{"1min":6.5609999999999999,"5min":5.4100000000000001,"15min":5.2779999999999996},"max":{"1min":14.51,"5min":180.44499999999999,"15min":21307.069},"last":7.8959999999999999}]},{"osd":2,"last update":"Mon Jun 3 09:06:25 2024","interfaces":[{"interface":"back","average":{"1min":12.343999999999999,"5min":22.059999999999999,"15min":23.715},"min":{"1min":7.9199999999999999,"5min":6.4900000000000002,"15min":6.056},"max":{"1min":15.601000000000001,"5min":377.803,"15min":377.803},"last":10.523},{"interface":"front","average":{"1min":10.319000000000001,"5min":19.364999999999998,"15min":20.242999999999999},"min":{"1min":6.891,"5min":6.1749999999999998,"15min":5.9020000000000001},"max":{"1min":15.432,"5min":371.601,"15min":371.601},"last":11.694000000000001}]},{"osd":4,"last update":"Mon Jun 3 09:06:31 2024","interfaces":[{"interface":"back","average":{"1min":9.7460000000000004,"5min":891.23000000000002,"15min":416.48099999999999},"min":{"1min":5.6539999999999999,"5min":5.6539999999999999,"15min":5.6539999999999999},"max":{"1min":15.445,"5min":29040.635999999999,"15min":30144.939999999999},"last":6.827},{"interface":"front","average":{"1min":12.398999999999999,"5min":897.58500000000004,"15min":421.27499999999998},"min":{"1min":8.9469999999999992,"5min":7.8109999999999999,"15min":6.2439999999999998},"max":{"1min":16.518000000000001,"5min":29076.955999999998,"15min":30180.689999999999},"last":9.0559999999999992}]},{"osd":5,"last update":"Mon Jun 3 09:06:07 2024","interfaces":[{"interface":"back","average":{"1min":12.802,"5min":19.640999999999998,"15min":22.318000000000001},"min":{"1min":9.0890000000000004,"5min":6.8810000000000002,"15min":6.7690000000000001},"max":{"1min":16.745999999999999,"5min":249.88,"15min":327.34899999999999},"last":11.055},{"interface":"front","average":{"1min":15.002000000000001,"5min":24.003,"15min":26.649999999999999},"min":{"1min":11.984,"5min":9.9149999999999991,"15min":8.9879999999999995},"max":{"1min":18.407,"5min":251.47800000000001,"15min":331.86799999999999},"last":15.125}]},{"osd":6,"last update":"Mon Jun 3 09:04:04 2024","interfaces":[{"interface":"back","average":{"1min":22.815999999999999,"5min":31.513999999999999,"15min":181.74299999999999},"min":{"1min":9.4809999999999999,"5min":7.2240000000000002,"15min":7.2240000000000002},"max":{"1min":160.56999999999999,"5min":380.613,"15min":11893.74},"last":19.359999999999999},{"interface":"front","average":{"1min":25.359999999999999,"5min":30.981000000000002,"15min":179.81399999999999},"min":{"1min":9.8870000000000005,"5min":7.3360000000000003,"15min":6.0899999999999999},"max":{"1min":193.11099999999999,"5min":379.80700000000002,"15min":11893.069},"last":8.202}]},{"osd":7,"last update":"Mon Jun 3 09:06:25 2024","interfaces":[{"interface":"back","average":{"1min":11.321999999999999,"5min":18.443999999999999,"15min":21.568999999999999},"min":{"1min":8.109,"5min":8.109,"15min":7.4279999999999999},"max":{"1min":17.113,"5min":146.267,"15min":178.76300000000001},"last":12.93},{"interface":"front","average":{"1min":12.512,"5min":20.065999999999999,"15min":24.190000000000001},"min":{"1min":7.4989999999999997,"5min":7.4989999999999997,"15min":7.4989999999999997},"max":{"1min":18.047999999999998,"5min":148.13800000000001,"15min":417.22800000000001},"last":14.553000000000001}]}]},{"osd":1,"up_from":14,"seq":60129542504,"num_pgs":6,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":59384,"kb_used_data":6120,"kb_used_omap":1,"kb_used_meta":53246,"kb_avail":94312456,"statfs":{"total":96636764160,"available":96575954944,"internally_reserved":0,"allocated":6266880,"data_stored":3149475,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":54524357},"hb_peers":[0,2,3,4,5,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":5,"apply_latency_ms":5,"commit_latency_ns":5000000,"apply_latency_ns":5000000},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Mon Jun 3 09:06:07 2024","interfaces":[{"interface":"back","average":{"1min":7.7530000000000001,"5min":185.66999999999999,"15min":77.347999999999999},"min":{"1min":5.0229999999999997,"5min":4.9279999999999999,"15min":4.5449999999999999},"max":{"1min":11.896000000000001,"5min":7470.616,"15min":7470.616},"last":7.4450000000000003},{"interface":"front","average":{"1min":9.9220000000000006,"5min":187.14599999999999,"15min":76.325999999999993},"min":{"1min":5.3159999999999998,"5min":5.3159999999999998,"15min":5.3159999999999998},"max":{"1min":18.338000000000001,"5min":7477.0829999999996,"15min":7477.0829999999996},"last":9.8000000000000007}]},{"osd":2,"last update":"Mon Jun 3 09:06:20 2024","interfaces":[{"interface":"back","average":{"1min":10.340999999999999,"5min":17.369,"15min":23.670999999999999},"min":{"1min":6.9740000000000002,"5min":5.8650000000000002,"15min":5.8650000000000002},"max":{"1min":15.452,"5min":228,"15min":657.38},"last":10.438000000000001},{"interface":"front","average":{"1min":13.387,"5min":23.355,"15min":31.524000000000001},"min":{"1min":9.077,"5min":9.077,"15min":7.2160000000000002},"max":{"1min":19.701000000000001,"5min":229.465,"15min":700.06500000000005},"last":10.981999999999999}]},{"osd":3,"last update":"Mon Jun 3 09:06:04 2024","interfaces":[{"interface":"back","average":{"1min":8.6110000000000007,"5min":15.733000000000001,"15min":26.646999999999998},"min":{"1min":5.827,"5min":5.4640000000000004,"15min":5.4640000000000004},"max":{"1min":13.105,"5min":121.619,"15min":1676.009},"last":6.7359999999999998},{"interface":"front","average":{"1min":9.5850000000000009,"5min":16.161999999999999,"15min":29.065999999999999},"min":{"1min":6.3630000000000004,"5min":5.7699999999999996,"15min":5.7699999999999996},"max":{"1min":15.166,"5min":132.036,"15min":1766.5840000000001},"last":9.4640000000000004}]},{"osd":4,"last update":"Mon Jun 3 09:06:35 2024","interfaces":[{"interface":"back","average":{"1min":8.6400000000000006,"5min":641.85500000000002,"15min":334.71499999999997},"min":{"1min":6.7640000000000002,"5min":6.1760000000000002,"15min":6.0970000000000004},"max":{"1min":11.811,"5min":26340.968000000001,"15min":28044.862000000001},"last":9.2050000000000001},{"interface":"front","average":{"1min":12.361000000000001,"5min":648.59900000000005,"15min":338.48500000000001},"min":{"1min":9.8949999999999996,"5min":9.3940000000000001,"15min":6.4640000000000004},"max":{"1min":15.507,"5min":26342.311000000002,"15min":28045.624},"last":14.834}]},{"osd":5,"last update":"Mon Jun 3 09:06:04 2024","interfaces":[{"interface":"back","average":{"1min":12.478999999999999,"5min":20.228000000000002,"15min":25.997},"min":{"1min":8.6430000000000007,"5min":7.9720000000000004,"15min":7.234},"max":{"1min":17.442,"5min":139.43199999999999,"15min":701.22000000000003},"last":15.553000000000001},{"interface":"front","average":{"1min":13.507999999999999,"5min":25.760999999999999,"15min":27.553999999999998},"min":{"1min":10.042999999999999,"5min":7.258,"15min":6.242},"max":{"1min":20.611000000000001,"5min":278.63,"15min":598.35000000000002},"last":12.192}]},{"osd":6,"last update":"Mon Jun 3 09:03:54 2024","interfaces":[{"interface":"back","average":{"1min":28.524000000000001,"5min":20.654,"15min":23.878},"min":{"1min":6.0309999999999997,"5min":6.0309999999999997,"15min":6.0309999999999997},"max":{"1min":167.221,"5min":167.221,"15min":704.245},"last":18.721},{"interface":"front","average":{"1min":27.606000000000002,"5min":30.131,"15min":28.628},"min":{"1min":9.8650000000000002,"5min":9.7200000000000006,"15min":8.3580000000000005},"max":{"1min":123.938,"5min":358.23000000000002,"15min":634.05499999999995},"last":173.113}]},{"osd":7,"last update":"Mon Jun 3 09:06:20 2024","interfaces":[{"interface":"back","average":{"1min":14.66,"5min":23.234999999999999,"15min":31.097000000000001},"min":{"1min":11.250999999999999,"5min":8.0559999999999992,"15min":8.0559999999999992},"max":{"1min":21.507999999999999,"5min":270.95100000000002,"15min":703.07799999999997},"last":11.664999999999999},{"interface":"front","average":{"1min":15.548,"5min":28.760000000000002,"15min":28.925000000000001},"min":{"1min":12.759,"5min":8.2089999999999996,"15min":7.7599999999999998},"max":{"1min":22.408000000000001,"5min":360.99900000000002,"15min":608.44100000000003},"last":13.053000000000001}]}]},{"osd":0,"up_from":14,"seq":60129542506,"num_pgs":10,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":81108,"kb_used_data":5704,"kb_used_omap":1,"kb_used_meta":75390,"kb_avail":94290732,"statfs":{"total":96636764160,"available":96553709568,"internally_reserved":0,"allocated":5840896,"data_stored":2740727,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1593,"internal_metadata":77199815},"hb_peers":[1,2,3,4,5,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[{"osd":1,"last update":"Mon Jun 3 09:06:04 2024","interfaces":[{"interface":"back","average":{"1min":6.9960000000000004,"5min":56.997,"15min":339.29700000000003},"min":{"1min":4.4530000000000003,"5min":4.4530000000000003,"15min":4.4100000000000001},"max":{"1min":11.720000000000001,"5min":2414.4749999999999,"15min":20691.453000000001},"last":5.6639999999999997},{"interface":"front","average":{"1min":9.548,"5min":57.258000000000003,"15min":337.791},"min":{"1min":5.2300000000000004,"5min":5.2300000000000004,"15min":4.6050000000000004},"max":{"1min":15.337999999999999,"5min":2412.9549999999999,"15min":20640.792000000001},"last":8.5690000000000008}]},{"osd":2,"last update":"Mon Jun 3 09:06:22 2024","interfaces":[{"interface":"back","average":{"1min":11.154,"5min":26.515999999999998,"15min":22.882000000000001},"min":{"1min":5.7869999999999999,"5min":5.766,"15min":5.766},"max":{"1min":72.269999999999996,"5min":627.40700000000004,"15min":627.40700000000004},"last":6.6139999999999999},{"interface":"front","average":{"1min":14.353999999999999,"5min":33.304000000000002,"15min":29.885000000000002},"min":{"1min":8.2149999999999999,"5min":7.8630000000000004,"15min":7.4089999999999998},"max":{"1min":73.418000000000006,"5min":635.62599999999998,"15min":635.62599999999998},"last":9.8049999999999997}]},{"osd":3,"last update":"Mon Jun 3 09:06:04 2024","interfaces":[{"interface":"back","average":{"1min":12.105,"5min":26.925000000000001,"15min":26.584},"min":{"1min":7.8410000000000002,"5min":7.1280000000000001,"15min":6.1909999999999998},"max":{"1min":18.425999999999998,"5min":389.72699999999998,"15min":389.72699999999998},"last":9.1590000000000007},{"interface":"front","average":{"1min":13.794,"5min":29.771999999999998,"15min":28.815000000000001},"min":{"1min":10.084,"5min":9.4920000000000009,"15min":6.1849999999999996},"max":{"1min":17.373999999999999,"5min":390.23200000000003,"15min":494.92899999999997},"last":11.637}]},{"osd":4,"last update":"Mon Jun 3 09:06:27 2024","interfaces":[{"interface":"back","average":{"1min":11.352,"5min":745.30600000000004,"15min":363.48099999999999},"min":{"1min":6.415,"5min":6.415,"15min":6.415},"max":{"1min":16.724,"5min":20248.616000000002,"15min":25555.212},"last":7.2590000000000003},{"interface":"front","average":{"1min":13.545,"5min":744.76999999999998,"15min":361.03100000000001},"min":{"1min":9.3889999999999993,"5min":8.8940000000000001,"15min":7.1429999999999998},"max":{"1min":18.411000000000001,"5min":20250.195,"15min":25556.159},"last":10.904}]},{"osd":5,"last update":"Mon Jun 3 09:06:17 2024","interfaces":[{"interface":"back","average":{"1min":9.7040000000000006,"5min":26.533999999999999,"15min":28.683},"min":{"1min":6.8129999999999997,"5min":6.8129999999999997,"15min":6.8129999999999997},"max":{"1min":14.942,"5min":455.78699999999998,"15min":605.05899999999997},"last":14.183999999999999},{"interface":"front","average":{"1min":14.279,"5min":28.398,"15min":26.192},"min":{"1min":11.696,"5min":6.4420000000000002,"15min":6.2030000000000003},"max":{"1min":17.879000000000001,"5min":391.31200000000001,"15min":541.11000000000001},"last":12.648999999999999}]},{"osd":6,"last update":"Mon Jun 3 09:04:30 2024","interfaces":[{"interface":"back","average":{"1min":52.496000000000002,"5min":33.139000000000003,"15min":160.83199999999999},"min":{"1min":7.5289999999999999,"5min":7.1040000000000001,"15min":6.3739999999999997},"max":{"1min":457.91699999999997,"5min":457.91699999999997,"15min":11115.258},"last":18.959},{"interface":"front","average":{"1min":49.670000000000002,"5min":33.704999999999998,"15min":162.791},"min":{"1min":12.247999999999999,"5min":11.019,"15min":9.9760000000000009},"max":{"1min":407.05099999999999,"5min":407.05099999999999,"15min":11092.194},"last":17.707000000000001}]},{"osd":7,"last update":"Mon Jun 3 09:06:22 2024","interfaces":[{"interface":"back","average":{"1min":11.625999999999999,"5min":26.375,"15min":28.972999999999999},"min":{"1min":6.4480000000000004,"5min":6.4480000000000004,"15min":6.4480000000000004},"max":{"1min":15.352,"5min":461.41699999999997,"15min":598.33699999999999},"last":13.51},{"interface":"front","average":{"1min":14.013,"5min":29.696000000000002,"15min":33.630000000000003},"min":{"1min":10.938000000000001,"5min":10.58,"15min":10.263999999999999},"max":{"1min":16.603000000000002,"5min":463.10599999999999,"15min":592.80499999999995},"last":14.896000000000001}]}]}],"pool_statfs":[{"poolid":1,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":401408,"data_stored":397840,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":5,"total":0,"available":0,"internally_reserved":0,"allocated":401408,"data_stored":397840,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":4,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":5,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":6,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":17,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":17,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":17,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":17,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":17,"osd":4,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":17,"osd":5,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":5,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":17,"osd":6,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":5,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":17,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0}]}} 2024-06-03T09:06:48.303 DEBUG:teuthology.orchestra.run.smithi088:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph pg dump --format=json 2024-06-03T09:06:48.401 INFO:teuthology.orchestra.run.smithi088.stderr:2024-06-03T09:06:48.400+0000 7f35567f6640 10 monclient: get_monmap_and_config 2024-06-03T09:06:48.401 INFO:teuthology.orchestra.run.smithi088.stderr:2024-06-03T09:06:48.400+0000 7f35567f6640 10 monclient: build_initial_monmap 2024-06-03T09:06:48.401 INFO:teuthology.orchestra.run.smithi088.stderr:2024-06-03T09:06:48.400+0000 7f35567f6640 10 monclient: monmap: 2024-06-03T09:06:48.401 INFO:teuthology.orchestra.run.smithi088.stderr:epoch 0 2024-06-03T09:06:48.401 INFO:teuthology.orchestra.run.smithi088.stderr:fsid ec3c3fdd-3ac6-4d7d-9412-683130ed8b04 2024-06-03T09:06:48.401 INFO:teuthology.orchestra.run.smithi088.stderr:last_changed 2024-06-03T09:06:48.402003+0000 2024-06-03T09:06:48.401 INFO:teuthology.orchestra.run.smithi088.stderr:created 2024-06-03T09:06:48.402003+0000 2024-06-03T09:06:48.401 INFO:teuthology.orchestra.run.smithi088.stderr:min_mon_release 0 (unknown) 2024-06-03T09:06:48.402 INFO:teuthology.orchestra.run.smithi088.stderr:election_strategy: 3 2024-06-03T09:06:48.402 INFO:teuthology.orchestra.run.smithi088.stderr:0: v1:172.21.15.88:6789/0 mon.noname-a 2024-06-03T09:06:48.402 INFO:teuthology.orchestra.run.smithi088.stderr:1: v1:172.21.15.192:6789/0 mon.noname-c 2024-06-03T09:06:48.402 INFO:teuthology.orchestra.run.smithi088.stderr:2: v1:172.21.15.88:6790/0 mon.noname-b 2024-06-03T09:06:48.402 INFO:teuthology.orchestra.run.smithi088.stderr: 2024-06-03T09:06:48.404 INFO:teuthology.orchestra.run.smithi088.stderr:2024-06-03T09:06:48.401+0000 7f35567f6640 1 Processor -- start 2024-06-03T09:06:48.404 INFO:teuthology.orchestra.run.smithi088.stderr:2024-06-03T09:06:48.401+0000 7f35567f6640 1 -- start start 2024-06-03T09:06:48.404 INFO:teuthology.orchestra.run.smithi088.stderr:2024-06-03T09:06:48.401+0000 7f35567f6640 10 monclient: init 2024-06-03T09:06:48.404 INFO:teuthology.orchestra.run.smithi088.stderr:2024-06-03T09:06:48.402+0000 7f35567f6640 10 monclient: _reopen_session rank -1 2024-06-03T09:06:48.404 INFO:teuthology.orchestra.run.smithi088.stderr:2024-06-03T09:06:48.402+0000 7f35567f6640 10 monclient: _add_conns ranks=[0,2,1] 2024-06-03T09:06:48.404 INFO:teuthology.orchestra.run.smithi088.stderr:2024-06-03T09:06:48.402+0000 7f35567f6640 10 monclient(hunting): picked mon.noname-a con 0x7f35500a8960 addr v1:172.21.15.88:6789/0 2024-06-03T09:06:48.404 INFO:teuthology.orchestra.run.smithi088.stderr:2024-06-03T09:06:48.402+0000 7f35567f6640 10 monclient(hunting): picked mon.noname-b con 0x7f35501450f0 addr v1:172.21.15.88:6790/0 2024-06-03T09:06:48.404 INFO:teuthology.orchestra.run.smithi088.stderr:2024-06-03T09:06:48.402+0000 7f35567f6640 10 monclient(hunting): picked mon.noname-c con 0x7f355013eec0 addr v1:172.21.15.192:6789/0 2024-06-03T09:06:48.404 INFO:teuthology.orchestra.run.smithi088.stderr:2024-06-03T09:06:48.402+0000 7f35567f6640 1 -- --> v1:172.21.15.88:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f3550144c60 con 0x7f35500a8960 2024-06-03T09:06:48.404 INFO:teuthology.orchestra.run.smithi088.stderr:2024-06-03T09:06:48.402+0000 7f35567f6640 1 -- --> v1:172.21.15.192:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f35500b3930 con 0x7f355013eec0 2024-06-03T09:06:48.404 INFO:teuthology.orchestra.run.smithi088.stderr:2024-06-03T09:06:48.402+0000 7f35567f6640 1 -- --> v1:172.21.15.88:6790/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f35500b1ad0 con 0x7f35501450f0 2024-06-03T09:06:48.404 INFO:teuthology.orchestra.run.smithi088.stderr:2024-06-03T09:06:48.402+0000 7f35567f6640 10 monclient(hunting): _renew_subs 2024-06-03T09:06:48.405 INFO:teuthology.orchestra.run.smithi088.stderr:2024-06-03T09:06:48.402+0000 7f35567f6640 10 monclient(hunting): authenticate will time out at 3064.212546s 2024-06-03T09:06:48.405 INFO:teuthology.orchestra.run.smithi088.stderr:2024-06-03T09:06:48.403+0000 7f3547fff640 1 --1- >> v1:172.21.15.88:6790/0 conn(0x7f35501450f0 0x7f355013e7b0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.88:6790/0 says I am v1:172.21.15.88:52314/0 (socket says 172.21.15.88:52314) 2024-06-03T09:06:48.405 INFO:teuthology.orchestra.run.smithi088.stderr:2024-06-03T09:06:48.403+0000 7f3547fff640 1 -- 172.21.15.88:0/1601105214 learned_addr learned my addr 172.21.15.88:0/1601105214 (peer_addr_for_me v1:172.21.15.88:0/0) 2024-06-03T09:06:48.405 INFO:teuthology.orchestra.run.smithi088.stderr:2024-06-03T09:06:48.404+0000 7f3554d93640 1 -- 172.21.15.88:0/1601105214 >> v1:172.21.15.192:6789/0 conn(0x7f355013eec0 legacy=0x7f355014ce60 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).read_bulk peer close file descriptor 15 2024-06-03T09:06:48.405 INFO:teuthology.orchestra.run.smithi088.stderr:2024-06-03T09:06:48.404+0000 7f3554d93640 1 -- 172.21.15.88:0/1601105214 >> v1:172.21.15.192:6789/0 conn(0x7f355013eec0 legacy=0x7f355014ce60 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).read_until read failed 2024-06-03T09:06:48.405 INFO:teuthology.orchestra.run.smithi088.stderr:2024-06-03T09:06:48.404+0000 7f3554d93640 1 --1- 172.21.15.88:0/1601105214 >> v1:172.21.15.192:6789/0 conn(0x7f355013eec0 0x7f355014ce60 :-1 s=OPENED pgs=9270 cs=1 l=1).handle_message read tag failed 2024-06-03T09:06:48.405 INFO:teuthology.orchestra.run.smithi088.stderr:2024-06-03T09:06:48.404+0000 7f3554d93640 1 --1- 172.21.15.88:0/1601105214 >> v1:172.21.15.192:6789/0 conn(0x7f355013eec0 0x7f355014ce60 :-1 s=OPENED pgs=9270 cs=1 l=1).fault on lossy channel, failing 2024-06-03T09:06:48.405 INFO:teuthology.orchestra.run.smithi088.stderr:2024-06-03T09:06:48.404+0000 7f354f7fe640 10 monclient(hunting): ms_handle_reset hunted mon v1:172.21.15.192:6789/0 2024-06-03T09:06:48.406 INFO:teuthology.orchestra.run.smithi088.stderr:2024-06-03T09:06:48.405+0000 7f354f7fe640 1 -- 172.21.15.88:0/1601105214 <== mon.2 v1:172.21.15.88:6790/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 497652384 0 0) 0x7f35500b1ad0 con 0x7f35501450f0 2024-06-03T09:06:48.406 INFO:teuthology.orchestra.run.smithi088.stderr:2024-06-03T09:06:48.405+0000 7f354f7fe640 10 monclient(hunting): _negotiate 2024-06-03T09:06:48.406 INFO:teuthology.orchestra.run.smithi088.stderr:2024-06-03T09:06:48.405+0000 7f354f7fe640 10 monclient(hunting): _init_auth method 2 2024-06-03T09:06:48.406 INFO:teuthology.orchestra.run.smithi088.stderr:2024-06-03T09:06:48.405+0000 7f354f7fe640 10 monclient(hunting): _init_auth creating new auth 2024-06-03T09:06:48.406 INFO:teuthology.orchestra.run.smithi088.stderr:2024-06-03T09:06:48.405+0000 7f354f7fe640 10 monclient(hunting): my global_id is 30206 2024-06-03T09:06:48.406 INFO:teuthology.orchestra.run.smithi088.stderr:2024-06-03T09:06:48.405+0000 7f354f7fe640 1 -- 172.21.15.88:0/1601105214 --> v1:172.21.15.88:6790/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f353c003520 con 0x7f35501450f0 2024-06-03T09:06:48.406 INFO:teuthology.orchestra.run.smithi088.stderr:2024-06-03T09:06:48.405+0000 7f354f7fe640 1 -- 172.21.15.88:0/1601105214 <== mon.0 v1:172.21.15.88:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 4214727660 0 0) 0x7f3550144c60 con 0x7f35500a8960 2024-06-03T09:06:48.406 INFO:teuthology.orchestra.run.smithi088.stderr:2024-06-03T09:06:48.405+0000 7f354f7fe640 10 monclient(hunting): _negotiate 2024-06-03T09:06:48.406 INFO:teuthology.orchestra.run.smithi088.stderr:2024-06-03T09:06:48.405+0000 7f354f7fe640 10 monclient(hunting): _init_auth method 2 2024-06-03T09:06:48.407 INFO:teuthology.orchestra.run.smithi088.stderr:2024-06-03T09:06:48.405+0000 7f354f7fe640 10 monclient(hunting): _init_auth creating new auth 2024-06-03T09:06:48.407 INFO:teuthology.orchestra.run.smithi088.stderr:2024-06-03T09:06:48.405+0000 7f354f7fe640 10 monclient(hunting): my global_id is 30702 2024-06-03T09:06:48.407 INFO:teuthology.orchestra.run.smithi088.stderr:2024-06-03T09:06:48.405+0000 7f354f7fe640 1 -- 172.21.15.88:0/1601105214 --> v1:172.21.15.88:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f35500b1ad0 con 0x7f35500a8960 2024-06-03T09:06:48.408 INFO:teuthology.orchestra.run.smithi088.stderr:2024-06-03T09:06:48.408+0000 7f354f7fe640 1 -- 172.21.15.88:0/1601105214 <== mon.2 v1:172.21.15.88:6790/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 214+0+0 (unknown 1257589220 0 0) 0x7f353c003520 con 0x7f35501450f0 2024-06-03T09:06:48.409 INFO:teuthology.orchestra.run.smithi088.stderr:2024-06-03T09:06:48.408+0000 7f354f7fe640 1 -- 172.21.15.88:0/1601105214 >> v1:172.21.15.192:6789/0 conn(0x7f355013eec0 legacy=0x7f355014ce60 unknown :-1 s=STATE_CLOSED l=1).mark_down 2024-06-03T09:06:48.409 INFO:teuthology.orchestra.run.smithi088.stderr:2024-06-03T09:06:48.408+0000 7f354f7fe640 1 -- 172.21.15.88:0/1601105214 >> v1:172.21.15.88:6789/0 conn(0x7f35500a8960 legacy=0x7f35500a8d40 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-06-03T09:06:48.409 INFO:teuthology.orchestra.run.smithi088.stderr:2024-06-03T09:06:48.408+0000 7f354f7fe640 10 monclient: _finish_hunting 0 2024-06-03T09:06:48.409 INFO:teuthology.orchestra.run.smithi088.stderr:2024-06-03T09:06:48.408+0000 7f354f7fe640 1 monclient: found mon.noname-b 2024-06-03T09:06:48.409 INFO:teuthology.orchestra.run.smithi088.stderr:2024-06-03T09:06:48.408+0000 7f354f7fe640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-06-03T09:06:48.409 INFO:teuthology.orchestra.run.smithi088.stderr:2024-06-03T09:06:48.408+0000 7f354f7fe640 10 monclient: _send_mon_message to mon.noname-b at v1:172.21.15.88:6790/0 2024-06-03T09:06:48.409 INFO:teuthology.orchestra.run.smithi088.stderr:2024-06-03T09:06:48.408+0000 7f354f7fe640 1 -- 172.21.15.88:0/1601105214 --> v1:172.21.15.88:6790/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f35500b1c60 con 0x7f35501450f0 2024-06-03T09:06:48.409 INFO:teuthology.orchestra.run.smithi088.stderr:2024-06-03T09:06:48.408+0000 7f354f7fe640 10 monclient: _finish_auth 0 2024-06-03T09:06:48.409 INFO:teuthology.orchestra.run.smithi088.stderr:2024-06-03T09:06:48.408+0000 7f354f7fe640 10 monclient: _check_auth_tickets 2024-06-03T09:06:48.409 INFO:teuthology.orchestra.run.smithi088.stderr:2024-06-03T09:06:48.408+0000 7f354f7fe640 20 monclient: _check_auth_rotating not needed by client.admin 2024-06-03T09:06:48.409 INFO:teuthology.orchestra.run.smithi088.stderr:2024-06-03T09:06:48.408+0000 7f354f7fe640 1 -- 172.21.15.88:0/1601105214 <== mon.2 v1:172.21.15.88:6790/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 2234374065 0 0) 0x7f3538001610 con 0x7f35501450f0 2024-06-03T09:06:48.410 INFO:teuthology.orchestra.run.smithi088.stderr:2024-06-03T09:06:48.408+0000 7f35567f6640 5 monclient: authenticate success, global_id 30206 2024-06-03T09:06:48.410 INFO:teuthology.orchestra.run.smithi088.stderr:2024-06-03T09:06:48.408+0000 7f35567f6640 20 monclient: get_monmap_and_config waiting for monmap|config 2024-06-03T09:06:48.410 INFO:teuthology.orchestra.run.smithi088.stderr:2024-06-03T09:06:48.408+0000 7f354f7fe640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-06-03T09:06:48.410 INFO:teuthology.orchestra.run.smithi088.stderr:2024-06-03T09:06:48.408+0000 7f354f7fe640 10 monclient: got monmap 1 from mon.noname-b (according to old e1) 2024-06-03T09:06:48.410 INFO:teuthology.orchestra.run.smithi088.stderr:2024-06-03T09:06:48.408+0000 7f354f7fe640 10 monclient: dump: 2024-06-03T09:06:48.410 INFO:teuthology.orchestra.run.smithi088.stderr:epoch 1 2024-06-03T09:06:48.410 INFO:teuthology.orchestra.run.smithi088.stderr:fsid ec3c3fdd-3ac6-4d7d-9412-683130ed8b04 2024-06-03T09:06:48.410 INFO:teuthology.orchestra.run.smithi088.stderr:last_changed 2024-06-03T08:34:31.885125+0000 2024-06-03T09:06:48.410 INFO:teuthology.orchestra.run.smithi088.stderr:created 2024-06-03T08:34:31.885125+0000 2024-06-03T09:06:48.410 INFO:teuthology.orchestra.run.smithi088.stderr:min_mon_release 18 (reef) 2024-06-03T09:06:48.410 INFO:teuthology.orchestra.run.smithi088.stderr:election_strategy: 3 2024-06-03T09:06:48.410 INFO:teuthology.orchestra.run.smithi088.stderr:0: v1:172.21.15.88:6789/0 mon.a 2024-06-03T09:06:48.411 INFO:teuthology.orchestra.run.smithi088.stderr:1: v1:172.21.15.192:6789/0 mon.b 2024-06-03T09:06:48.411 INFO:teuthology.orchestra.run.smithi088.stderr:2: v1:172.21.15.88:6790/0 mon.c 2024-06-03T09:06:48.411 INFO:teuthology.orchestra.run.smithi088.stderr: 2024-06-03T09:06:48.411 INFO:teuthology.orchestra.run.smithi088.stderr:2024-06-03T09:06:48.408+0000 7f35567f6640 20 monclient: get_monmap_and_config waiting for monmap|config 2024-06-03T09:06:48.414 INFO:teuthology.orchestra.run.smithi088.stderr:2024-06-03T09:06:48.410+0000 7f354f7fe640 1 -- 172.21.15.88:0/1601105214 <== mon.2 v1:172.21.15.88:6790/0 4 ==== config(0 keys) v1 ==== 4+0+0 (unknown 0 0 0) 0x7f3538001830 con 0x7f35501450f0 2024-06-03T09:06:48.414 INFO:teuthology.orchestra.run.smithi088.stderr:2024-06-03T09:06:48.410+0000 7f354f7fe640 10 monclient: handle_config config(0 keys) v1 2024-06-03T09:06:48.414 INFO:teuthology.orchestra.run.smithi088.stderr:2024-06-03T09:06:48.410+0000 7f35567f6640 10 monclient: get_monmap_and_config success 2024-06-03T09:06:48.414 INFO:teuthology.orchestra.run.smithi088.stderr:2024-06-03T09:06:48.410+0000 7f35567f6640 4 set_mon_vals no callback set 2024-06-03T09:06:48.414 INFO:teuthology.orchestra.run.smithi088.stderr:2024-06-03T09:06:48.410+0000 7f354f7fe640 1 -- 172.21.15.88:0/1601105214 <== mon.2 v1:172.21.15.88:6790/0 5 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 2234374065 0 0) 0x7f3538001b40 con 0x7f35501450f0 2024-06-03T09:06:48.414 INFO:teuthology.orchestra.run.smithi088.stderr:2024-06-03T09:06:48.410+0000 7f354f7fe640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-06-03T09:06:48.414 INFO:teuthology.orchestra.run.smithi088.stderr:2024-06-03T09:06:48.410+0000 7f354f7fe640 10 monclient: got monmap 1 from mon.c (according to old e1) 2024-06-03T09:06:48.414 INFO:teuthology.orchestra.run.smithi088.stderr:2024-06-03T09:06:48.410+0000 7f354f7fe640 10 monclient: dump: 2024-06-03T09:06:48.414 INFO:teuthology.orchestra.run.smithi088.stderr:epoch 1 2024-06-03T09:06:48.415 INFO:teuthology.orchestra.run.smithi088.stderr:fsid ec3c3fdd-3ac6-4d7d-9412-683130ed8b04 2024-06-03T09:06:48.415 INFO:teuthology.orchestra.run.smithi088.stderr:last_changed 2024-06-03T08:34:31.885125+0000 2024-06-03T09:06:48.415 INFO:teuthology.orchestra.run.smithi088.stderr:created 2024-06-03T08:34:31.885125+0000 2024-06-03T09:06:48.415 INFO:teuthology.orchestra.run.smithi088.stderr:min_mon_release 18 (reef) 2024-06-03T09:06:48.415 INFO:teuthology.orchestra.run.smithi088.stderr:election_strategy: 3 2024-06-03T09:06:48.415 INFO:teuthology.orchestra.run.smithi088.stderr:0: v1:172.21.15.88:6789/0 mon.a 2024-06-03T09:06:48.415 INFO:teuthology.orchestra.run.smithi088.stderr:1: v1:172.21.15.192:6789/0 mon.b 2024-06-03T09:06:48.415 INFO:teuthology.orchestra.run.smithi088.stderr:2: v1:172.21.15.88:6790/0 mon.c 2024-06-03T09:06:48.415 INFO:teuthology.orchestra.run.smithi088.stderr: 2024-06-03T09:06:48.415 INFO:teuthology.orchestra.run.smithi088.stderr:2024-06-03T09:06:48.410+0000 7f35567f6640 10 monclient: shutdown 2024-06-03T09:06:48.415 INFO:teuthology.orchestra.run.smithi088.stderr:2024-06-03T09:06:48.410+0000 7f35567f6640 20 monclient: shutdown discarding 0 pending message(s) 2024-06-03T09:06:48.415 INFO:teuthology.orchestra.run.smithi088.stderr:2024-06-03T09:06:48.410+0000 7f35567f6640 1 -- 172.21.15.88:0/1601105214 >> v1:172.21.15.88:6790/0 conn(0x7f35501450f0 legacy=0x7f355013e7b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-06-03T09:06:48.415 INFO:teuthology.orchestra.run.smithi088.stderr:2024-06-03T09:06:48.410+0000 7f35567f6640 1 -- 172.21.15.88:0/1601105214 shutdown_connections 2024-06-03T09:06:48.416 INFO:teuthology.orchestra.run.smithi088.stderr:2024-06-03T09:06:48.411+0000 7f35567f6640 1 -- 172.21.15.88:0/1601105214 >> 172.21.15.88:0/1601105214 conn(0x7f35500b70d0 msgr2=0x7f35500b74e0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-06-03T09:06:48.416 INFO:teuthology.orchestra.run.smithi088.stderr:2024-06-03T09:06:48.411+0000 7f35567f6640 1 -- 172.21.15.88:0/1601105214 shutdown_connections 2024-06-03T09:06:48.416 INFO:teuthology.orchestra.run.smithi088.stderr:2024-06-03T09:06:48.411+0000 7f35567f6640 1 -- 172.21.15.88:0/1601105214 wait complete. 2024-06-03T09:06:48.416 INFO:teuthology.orchestra.run.smithi088.stderr:2024-06-03T09:06:48.411+0000 7f35567f6640 10 monclient: build_initial_monmap 2024-06-03T09:06:48.416 INFO:teuthology.orchestra.run.smithi088.stderr:2024-06-03T09:06:48.411+0000 7f35567f6640 10 monclient: monmap: 2024-06-03T09:06:48.416 INFO:teuthology.orchestra.run.smithi088.stderr:epoch 0 2024-06-03T09:06:48.416 INFO:teuthology.orchestra.run.smithi088.stderr:fsid 00000000-0000-0000-0000-000000000000 2024-06-03T09:06:48.416 INFO:teuthology.orchestra.run.smithi088.stderr:last_changed 0.000000 2024-06-03T09:06:48.416 INFO:teuthology.orchestra.run.smithi088.stderr:created 0.000000 2024-06-03T09:06:48.416 INFO:teuthology.orchestra.run.smithi088.stderr:min_mon_release 0 (unknown) 2024-06-03T09:06:48.416 INFO:teuthology.orchestra.run.smithi088.stderr:election_strategy: 1 2024-06-03T09:06:48.416 INFO:teuthology.orchestra.run.smithi088.stderr:0: v1:172.21.15.88:6789/0 mon.noname-a 2024-06-03T09:06:48.416 INFO:teuthology.orchestra.run.smithi088.stderr:1: v1:172.21.15.192:6789/0 mon.noname-b 2024-06-03T09:06:48.417 INFO:teuthology.orchestra.run.smithi088.stderr:2: v1:172.21.15.88:6790/0 mon.noname-c 2024-06-03T09:06:48.417 INFO:teuthology.orchestra.run.smithi088.stderr: 2024-06-03T09:06:48.417 INFO:teuthology.orchestra.run.smithi088.stderr:2024-06-03T09:06:48.411+0000 7f35567f6640 1 librados: starting msgr at 2024-06-03T09:06:48.417 INFO:teuthology.orchestra.run.smithi088.stderr:2024-06-03T09:06:48.411+0000 7f35567f6640 1 librados: starting objecter 2024-06-03T09:06:48.417 INFO:teuthology.orchestra.run.smithi088.stderr:2024-06-03T09:06:48.412+0000 7f35567f6640 1 Processor -- start 2024-06-03T09:06:48.417 INFO:teuthology.orchestra.run.smithi088.stderr:2024-06-03T09:06:48.412+0000 7f35567f6640 1 -- start start 2024-06-03T09:06:48.417 INFO:teuthology.orchestra.run.smithi088.stderr:2024-06-03T09:06:48.412+0000 7f35567f6640 1 librados: setting wanted keys 2024-06-03T09:06:48.417 INFO:teuthology.orchestra.run.smithi088.stderr:2024-06-03T09:06:48.412+0000 7f35567f6640 1 librados: calling monclient init 2024-06-03T09:06:48.417 INFO:teuthology.orchestra.run.smithi088.stderr:2024-06-03T09:06:48.412+0000 7f35567f6640 10 monclient: init 2024-06-03T09:06:48.417 INFO:teuthology.orchestra.run.smithi088.stderr:2024-06-03T09:06:48.412+0000 7f35567f6640 10 monclient: _reopen_session rank -1 2024-06-03T09:06:48.417 INFO:teuthology.orchestra.run.smithi088.stderr:2024-06-03T09:06:48.412+0000 7f35567f6640 10 monclient: _add_conns ranks=[1,2,0] 2024-06-03T09:06:48.417 INFO:teuthology.orchestra.run.smithi088.stderr:2024-06-03T09:06:48.412+0000 7f35567f6640 10 monclient(hunting): picked mon.noname-b con 0x7f35500a8960 addr v1:172.21.15.192:6789/0 2024-06-03T09:06:48.417 INFO:teuthology.orchestra.run.smithi088.stderr:2024-06-03T09:06:48.412+0000 7f35567f6640 10 monclient(hunting): picked mon.noname-c con 0x7f355013eec0 addr v1:172.21.15.88:6790/0 2024-06-03T09:06:48.418 INFO:teuthology.orchestra.run.smithi088.stderr:2024-06-03T09:06:48.412+0000 7f35567f6640 10 monclient(hunting): picked mon.noname-a con 0x7f35501450f0 addr v1:172.21.15.88:6789/0 2024-06-03T09:06:48.418 INFO:teuthology.orchestra.run.smithi088.stderr:2024-06-03T09:06:48.412+0000 7f35567f6640 1 -- --> v1:172.21.15.88:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f35501d6450 con 0x7f35501450f0 2024-06-03T09:06:48.418 INFO:teuthology.orchestra.run.smithi088.stderr:2024-06-03T09:06:48.412+0000 7f35567f6640 1 -- --> v1:172.21.15.192:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f35501d7320 con 0x7f35500a8960 2024-06-03T09:06:48.418 INFO:teuthology.orchestra.run.smithi088.stderr:2024-06-03T09:06:48.412+0000 7f35567f6640 1 -- --> v1:172.21.15.88:6790/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f35500ad640 con 0x7f355013eec0 2024-06-03T09:06:48.418 INFO:teuthology.orchestra.run.smithi088.stderr:2024-06-03T09:06:48.412+0000 7f35567f6640 10 monclient(hunting): _renew_subs 2024-06-03T09:06:48.418 INFO:teuthology.orchestra.run.smithi088.stderr:2024-06-03T09:06:48.412+0000 7f35567f6640 10 monclient(hunting): authenticate will time out at 3064.222781s 2024-06-03T09:06:48.418 INFO:teuthology.orchestra.run.smithi088.stderr:2024-06-03T09:06:48.413+0000 7f3547fff640 1 --1- >> v1:172.21.15.88:6790/0 conn(0x7f355013eec0 0x7f35500adc00 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=1).handle_server_banner_and_identify peer v1:172.21.15.88:6790/0 says I am v1:172.21.15.88:52328/0 (socket says 172.21.15.88:52328) 2024-06-03T09:06:48.418 INFO:teuthology.orchestra.run.smithi088.stderr:2024-06-03T09:06:48.413+0000 7f3547fff640 1 -- 172.21.15.88:0/3396045638 learned_addr learned my addr 172.21.15.88:0/3396045638 (peer_addr_for_me v1:172.21.15.88:0/0) 2024-06-03T09:06:48.418 INFO:teuthology.orchestra.run.smithi088.stderr:2024-06-03T09:06:48.413+0000 7f354d7fa640 10 client.?.objecter ms_handle_connect 0x7f355013eec0 2024-06-03T09:06:48.418 INFO:teuthology.orchestra.run.smithi088.stderr:2024-06-03T09:06:48.413+0000 7f354d7fa640 10 client.?.objecter resend_mon_ops 2024-06-03T09:06:48.418 INFO:teuthology.orchestra.run.smithi088.stderr:2024-06-03T09:06:48.413+0000 7f354d7fa640 10 client.?.objecter ms_handle_connect 0x7f35501450f0 2024-06-03T09:06:48.418 INFO:teuthology.orchestra.run.smithi088.stderr:2024-06-03T09:06:48.413+0000 7f354d7fa640 10 client.?.objecter resend_mon_ops 2024-06-03T09:06:48.418 INFO:teuthology.orchestra.run.smithi088.stderr:2024-06-03T09:06:48.413+0000 7f354d7fa640 10 client.?.objecter ms_handle_connect 0x7f35500a8960 2024-06-03T09:06:48.419 INFO:teuthology.orchestra.run.smithi088.stderr:2024-06-03T09:06:48.413+0000 7f354d7fa640 10 client.?.objecter resend_mon_ops 2024-06-03T09:06:48.420 INFO:teuthology.orchestra.run.smithi088.stderr:2024-06-03T09:06:48.415+0000 7f354d7fa640 1 -- 172.21.15.88:0/3396045638 <== mon.2 v1:172.21.15.88:6790/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 2709219890 0 0) 0x7f35500ad640 con 0x7f355013eec0 2024-06-03T09:06:48.420 INFO:teuthology.orchestra.run.smithi088.stderr:2024-06-03T09:06:48.415+0000 7f354d7fa640 10 monclient(hunting): _negotiate 2024-06-03T09:06:48.420 INFO:teuthology.orchestra.run.smithi088.stderr:2024-06-03T09:06:48.415+0000 7f354d7fa640 10 monclient(hunting): _init_auth method 2 2024-06-03T09:06:48.421 INFO:teuthology.orchestra.run.smithi088.stderr:2024-06-03T09:06:48.415+0000 7f354d7fa640 10 monclient(hunting): _init_auth creating new auth 2024-06-03T09:06:48.421 INFO:teuthology.orchestra.run.smithi088.stderr:2024-06-03T09:06:48.415+0000 7f354d7fa640 10 monclient(hunting): my global_id is 30209 2024-06-03T09:06:48.421 INFO:teuthology.orchestra.run.smithi088.stderr:2024-06-03T09:06:48.415+0000 7f354d7fa640 1 -- 172.21.15.88:0/3396045638 --> v1:172.21.15.88:6790/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f3530004eb0 con 0x7f355013eec0 2024-06-03T09:06:48.421 INFO:teuthology.orchestra.run.smithi088.stderr:2024-06-03T09:06:48.415+0000 7f354d7fa640 1 -- 172.21.15.88:0/3396045638 <== mon.0 v1:172.21.15.88:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 4218558234 0 0) 0x7f35501d6450 con 0x7f35501450f0 2024-06-03T09:06:48.421 INFO:teuthology.orchestra.run.smithi088.stderr:2024-06-03T09:06:48.415+0000 7f354d7fa640 10 monclient(hunting): _negotiate 2024-06-03T09:06:48.421 INFO:teuthology.orchestra.run.smithi088.stderr:2024-06-03T09:06:48.415+0000 7f354d7fa640 10 monclient(hunting): _init_auth method 2 2024-06-03T09:06:48.421 INFO:teuthology.orchestra.run.smithi088.stderr:2024-06-03T09:06:48.415+0000 7f354d7fa640 10 monclient(hunting): _init_auth creating new auth 2024-06-03T09:06:48.421 INFO:teuthology.orchestra.run.smithi088.stderr:2024-06-03T09:06:48.415+0000 7f354d7fa640 10 monclient(hunting): my global_id is 30705 2024-06-03T09:06:48.421 INFO:teuthology.orchestra.run.smithi088.stderr:2024-06-03T09:06:48.415+0000 7f354d7fa640 1 -- 172.21.15.88:0/3396045638 --> v1:172.21.15.88:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f35500ad640 con 0x7f35501450f0 2024-06-03T09:06:48.421 INFO:teuthology.orchestra.run.smithi088.stderr:2024-06-03T09:06:48.415+0000 7f354d7fa640 1 -- 172.21.15.88:0/3396045638 <== mon.1 v1:172.21.15.192:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 173754731 0 0) 0x7f35501d7320 con 0x7f35500a8960 2024-06-03T09:06:48.421 INFO:teuthology.orchestra.run.smithi088.stderr:2024-06-03T09:06:48.415+0000 7f354d7fa640 10 monclient(hunting): _negotiate 2024-06-03T09:06:48.421 INFO:teuthology.orchestra.run.smithi088.stderr:2024-06-03T09:06:48.415+0000 7f354d7fa640 10 monclient(hunting): _init_auth method 2 2024-06-03T09:06:48.421 INFO:teuthology.orchestra.run.smithi088.stderr:2024-06-03T09:06:48.415+0000 7f354d7fa640 10 monclient(hunting): _init_auth creating new auth 2024-06-03T09:06:48.422 INFO:teuthology.orchestra.run.smithi088.stderr:2024-06-03T09:06:48.415+0000 7f354d7fa640 10 monclient(hunting): my global_id is 30280 2024-06-03T09:06:48.422 INFO:teuthology.orchestra.run.smithi088.stderr:2024-06-03T09:06:48.415+0000 7f354d7fa640 1 -- 172.21.15.88:0/3396045638 --> v1:172.21.15.192:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f35501d6450 con 0x7f35500a8960 2024-06-03T09:06:48.422 INFO:teuthology.orchestra.run.smithi088.stderr:2024-06-03T09:06:48.420+0000 7f354d7fa640 1 -- 172.21.15.88:0/3396045638 <== mon.2 v1:172.21.15.88:6790/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 764+0+0 (unknown 1874617649 0 0) 0x7f3530004eb0 con 0x7f355013eec0 2024-06-03T09:06:48.422 INFO:teuthology.orchestra.run.smithi088.stderr:2024-06-03T09:06:48.420+0000 7f354d7fa640 1 -- 172.21.15.88:0/3396045638 --> v1:172.21.15.88:6790/0 -- auth(proto 2 165 bytes epoch 0) v1 -- 0x7f35501d7320 con 0x7f355013eec0 2024-06-03T09:06:48.422 INFO:teuthology.orchestra.run.smithi088.stderr:2024-06-03T09:06:48.420+0000 7f354d7fa640 1 -- 172.21.15.88:0/3396045638 <== mon.2 v1:172.21.15.88:6790/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 2234374065 0 0) 0x7f3538001510 con 0x7f355013eec0 2024-06-03T09:06:48.422 INFO:teuthology.orchestra.run.smithi088.stderr:2024-06-03T09:06:48.420+0000 7f354d7fa640 10 monclient(hunting): handle_monmap mon_map magic: 0 v1 2024-06-03T09:06:48.422 INFO:teuthology.orchestra.run.smithi088.stderr:2024-06-03T09:06:48.420+0000 7f354d7fa640 10 monclient(hunting): got monmap 1 from mon.noname-c (according to old e1) 2024-06-03T09:06:48.422 INFO:teuthology.orchestra.run.smithi088.stderr:2024-06-03T09:06:48.420+0000 7f354d7fa640 10 monclient(hunting): dump: 2024-06-03T09:06:48.422 INFO:teuthology.orchestra.run.smithi088.stderr:epoch 1 2024-06-03T09:06:48.422 INFO:teuthology.orchestra.run.smithi088.stderr:fsid ec3c3fdd-3ac6-4d7d-9412-683130ed8b04 2024-06-03T09:06:48.422 INFO:teuthology.orchestra.run.smithi088.stderr:last_changed 2024-06-03T08:34:31.885125+0000 2024-06-03T09:06:48.422 INFO:teuthology.orchestra.run.smithi088.stderr:created 2024-06-03T08:34:31.885125+0000 2024-06-03T09:06:48.423 INFO:teuthology.orchestra.run.smithi088.stderr:min_mon_release 18 (reef) 2024-06-03T09:06:48.423 INFO:teuthology.orchestra.run.smithi088.stderr:election_strategy: 3 2024-06-03T09:06:48.423 INFO:teuthology.orchestra.run.smithi088.stderr:0: v1:172.21.15.88:6789/0 mon.a 2024-06-03T09:06:48.423 INFO:teuthology.orchestra.run.smithi088.stderr:1: v1:172.21.15.192:6789/0 mon.b 2024-06-03T09:06:48.423 INFO:teuthology.orchestra.run.smithi088.stderr:2: v1:172.21.15.88:6790/0 mon.c 2024-06-03T09:06:48.423 INFO:teuthology.orchestra.run.smithi088.stderr: 2024-06-03T09:06:48.423 INFO:teuthology.orchestra.run.smithi088.stderr:2024-06-03T09:06:48.420+0000 7f354d7fa640 10 monclient(hunting): _finish_auth 0 2024-06-03T09:06:48.423 INFO:teuthology.orchestra.run.smithi088.stderr:2024-06-03T09:06:48.420+0000 7f354d7fa640 1 -- 172.21.15.88:0/3396045638 <== mon.1 v1:172.21.15.192:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 764+0+0 (unknown 200700957 0 0) 0x7f35501d6450 con 0x7f35500a8960 2024-06-03T09:06:48.423 INFO:teuthology.orchestra.run.smithi088.stderr:2024-06-03T09:06:48.420+0000 7f354d7fa640 1 -- 172.21.15.88:0/3396045638 --> v1:172.21.15.192:6789/0 -- auth(proto 2 165 bytes epoch 0) v1 -- 0x7f3530004eb0 con 0x7f35500a8960 2024-06-03T09:06:48.423 INFO:teuthology.orchestra.run.smithi088.stderr:2024-06-03T09:06:48.420+0000 7f354d7fa640 1 -- 172.21.15.88:0/3396045638 <== mon.1 v1:172.21.15.192:6789/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 2234374065 0 0) 0x7f35400018b0 con 0x7f35500a8960 2024-06-03T09:06:48.423 INFO:teuthology.orchestra.run.smithi088.stderr:2024-06-03T09:06:48.420+0000 7f354d7fa640 10 monclient(hunting): handle_monmap mon_map magic: 0 v1 2024-06-03T09:06:48.423 INFO:teuthology.orchestra.run.smithi088.stderr:2024-06-03T09:06:48.420+0000 7f354d7fa640 10 monclient(hunting): got monmap 1 from mon.b (according to old e1) 2024-06-03T09:06:48.423 INFO:teuthology.orchestra.run.smithi088.stderr:2024-06-03T09:06:48.420+0000 7f354d7fa640 10 monclient(hunting): dump: 2024-06-03T09:06:48.424 INFO:teuthology.orchestra.run.smithi088.stderr:epoch 1 2024-06-03T09:06:48.424 INFO:teuthology.orchestra.run.smithi088.stderr:fsid ec3c3fdd-3ac6-4d7d-9412-683130ed8b04 2024-06-03T09:06:48.424 INFO:teuthology.orchestra.run.smithi088.stderr:last_changed 2024-06-03T08:34:31.885125+0000 2024-06-03T09:06:48.424 INFO:teuthology.orchestra.run.smithi088.stderr:created 2024-06-03T08:34:31.885125+0000 2024-06-03T09:06:48.424 INFO:teuthology.orchestra.run.smithi088.stderr:min_mon_release 18 (reef) 2024-06-03T09:06:48.424 INFO:teuthology.orchestra.run.smithi088.stderr:election_strategy: 3 2024-06-03T09:06:48.424 INFO:teuthology.orchestra.run.smithi088.stderr:0: v1:172.21.15.88:6789/0 mon.a 2024-06-03T09:06:48.424 INFO:teuthology.orchestra.run.smithi088.stderr:1: v1:172.21.15.192:6789/0 mon.b 2024-06-03T09:06:48.424 INFO:teuthology.orchestra.run.smithi088.stderr:2: v1:172.21.15.88:6790/0 mon.c 2024-06-03T09:06:48.424 INFO:teuthology.orchestra.run.smithi088.stderr: 2024-06-03T09:06:48.424 INFO:teuthology.orchestra.run.smithi088.stderr:2024-06-03T09:06:48.420+0000 7f354d7fa640 1 -- 172.21.15.88:0/3396045638 <== mon.0 v1:172.21.15.88:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 764+0+0 (unknown 539391996 0 0) 0x7f35500ad640 con 0x7f35501450f0 2024-06-03T09:06:48.424 INFO:teuthology.orchestra.run.smithi088.stderr:2024-06-03T09:06:48.420+0000 7f354d7fa640 1 -- 172.21.15.88:0/3396045638 --> v1:172.21.15.88:6789/0 -- auth(proto 2 165 bytes epoch 0) v1 -- 0x7f35501d6450 con 0x7f35501450f0 2024-06-03T09:06:48.424 INFO:teuthology.orchestra.run.smithi088.stderr:2024-06-03T09:06:48.420+0000 7f354d7fa640 1 -- 172.21.15.88:0/3396045638 <== mon.0 v1:172.21.15.88:6789/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 2234374065 0 0) 0x7f3548001c60 con 0x7f35501450f0 2024-06-03T09:06:48.425 INFO:teuthology.orchestra.run.smithi088.stderr:2024-06-03T09:06:48.420+0000 7f354d7fa640 10 monclient(hunting): handle_monmap mon_map magic: 0 v1 2024-06-03T09:06:48.425 INFO:teuthology.orchestra.run.smithi088.stderr:2024-06-03T09:06:48.420+0000 7f354d7fa640 10 monclient(hunting): got monmap 1 from mon.a (according to old e1) 2024-06-03T09:06:48.425 INFO:teuthology.orchestra.run.smithi088.stderr:2024-06-03T09:06:48.420+0000 7f354d7fa640 10 monclient(hunting): dump: 2024-06-03T09:06:48.425 INFO:teuthology.orchestra.run.smithi088.stderr:epoch 1 2024-06-03T09:06:48.425 INFO:teuthology.orchestra.run.smithi088.stderr:fsid ec3c3fdd-3ac6-4d7d-9412-683130ed8b04 2024-06-03T09:06:48.425 INFO:teuthology.orchestra.run.smithi088.stderr:last_changed 2024-06-03T08:34:31.885125+0000 2024-06-03T09:06:48.425 INFO:teuthology.orchestra.run.smithi088.stderr:created 2024-06-03T08:34:31.885125+0000 2024-06-03T09:06:48.425 INFO:teuthology.orchestra.run.smithi088.stderr:min_mon_release 18 (reef) 2024-06-03T09:06:48.425 INFO:teuthology.orchestra.run.smithi088.stderr:election_strategy: 3 2024-06-03T09:06:48.425 INFO:teuthology.orchestra.run.smithi088.stderr:0: v1:172.21.15.88:6789/0 mon.a 2024-06-03T09:06:48.425 INFO:teuthology.orchestra.run.smithi088.stderr:1: v1:172.21.15.192:6789/0 mon.b 2024-06-03T09:06:48.425 INFO:teuthology.orchestra.run.smithi088.stderr:2: v1:172.21.15.88:6790/0 mon.c 2024-06-03T09:06:48.425 INFO:teuthology.orchestra.run.smithi088.stderr: 2024-06-03T09:06:48.427 INFO:teuthology.orchestra.run.smithi088.stderr:2024-06-03T09:06:48.423+0000 7f354d7fa640 1 -- 172.21.15.88:0/3396045638 <== mon.2 v1:172.21.15.88:6790/0 4 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 580+0+0 (unknown 400244475 0 0) 0x7f35501d7320 con 0x7f355013eec0 2024-06-03T09:06:48.427 INFO:teuthology.orchestra.run.smithi088.stderr:2024-06-03T09:06:48.423+0000 7f354d7fa640 1 -- 172.21.15.88:0/3396045638 >> v1:172.21.15.192:6789/0 conn(0x7f35500a8960 legacy=0x7f35501d6c10 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-06-03T09:06:48.427 INFO:teuthology.orchestra.run.smithi088.stderr:2024-06-03T09:06:48.423+0000 7f354d7fa640 1 -- 172.21.15.88:0/3396045638 >> v1:172.21.15.88:6789/0 conn(0x7f35501450f0 legacy=0x7f35501e0f70 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-06-03T09:06:48.427 INFO:teuthology.orchestra.run.smithi088.stderr:2024-06-03T09:06:48.423+0000 7f354d7fa640 10 monclient: _finish_hunting 0 2024-06-03T09:06:48.427 INFO:teuthology.orchestra.run.smithi088.stderr:2024-06-03T09:06:48.423+0000 7f354d7fa640 1 monclient: found mon.c 2024-06-03T09:06:48.427 INFO:teuthology.orchestra.run.smithi088.stderr:2024-06-03T09:06:48.423+0000 7f354d7fa640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-06-03T09:06:48.427 INFO:teuthology.orchestra.run.smithi088.stderr:2024-06-03T09:06:48.423+0000 7f354d7fa640 10 monclient: _send_mon_message to mon.c at v1:172.21.15.88:6790/0 2024-06-03T09:06:48.427 INFO:teuthology.orchestra.run.smithi088.stderr:2024-06-03T09:06:48.423+0000 7f354d7fa640 1 -- 172.21.15.88:0/3396045638 --> v1:172.21.15.88:6790/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f35500ad7d0 con 0x7f355013eec0 2024-06-03T09:06:48.427 INFO:teuthology.orchestra.run.smithi088.stderr:2024-06-03T09:06:48.423+0000 7f354d7fa640 10 monclient: _finish_auth 0 2024-06-03T09:06:48.427 INFO:teuthology.orchestra.run.smithi088.stderr:2024-06-03T09:06:48.423+0000 7f354d7fa640 10 monclient: _check_auth_tickets 2024-06-03T09:06:48.428 INFO:teuthology.orchestra.run.smithi088.stderr:2024-06-03T09:06:48.423+0000 7f354d7fa640 20 monclient: _check_auth_rotating not needed by client.admin 2024-06-03T09:06:48.428 INFO:teuthology.orchestra.run.smithi088.stderr:2024-06-03T09:06:48.423+0000 7f35567f6640 5 monclient: authenticate success, global_id 30209 2024-06-03T09:06:48.428 INFO:teuthology.orchestra.run.smithi088.stderr:2024-06-03T09:06:48.423+0000 7f35567f6640 10 monclient: _renew_subs 2024-06-03T09:06:48.428 INFO:teuthology.orchestra.run.smithi088.stderr:2024-06-03T09:06:48.423+0000 7f35567f6640 10 monclient: _send_mon_message to mon.c at v1:172.21.15.88:6790/0 2024-06-03T09:06:48.428 INFO:teuthology.orchestra.run.smithi088.stderr:2024-06-03T09:06:48.423+0000 7f35567f6640 1 -- 172.21.15.88:0/3396045638 --> v1:172.21.15.88:6790/0 -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f35500ae310 con 0x7f355013eec0 2024-06-03T09:06:48.428 INFO:teuthology.orchestra.run.smithi088.stderr:2024-06-03T09:06:48.424+0000 7f35567f6640 10 client.30209.objecter _maybe_request_map subscribing (onetime) to next osd map 2024-06-03T09:06:48.428 INFO:teuthology.orchestra.run.smithi088.stderr:2024-06-03T09:06:48.424+0000 7f35567f6640 10 monclient: _renew_subs 2024-06-03T09:06:48.428 INFO:teuthology.orchestra.run.smithi088.stderr:2024-06-03T09:06:48.424+0000 7f35567f6640 10 monclient: _send_mon_message to mon.c at v1:172.21.15.88:6790/0 2024-06-03T09:06:48.428 INFO:teuthology.orchestra.run.smithi088.stderr:2024-06-03T09:06:48.424+0000 7f35567f6640 1 -- 172.21.15.88:0/3396045638 --> v1:172.21.15.88:6790/0 -- mon_subscribe({osdmap=0}) v3 -- 0x7f35500afac0 con 0x7f355013eec0 2024-06-03T09:06:48.428 INFO:teuthology.orchestra.run.smithi088.stderr:2024-06-03T09:06:48.424+0000 7f35567f6640 1 librados: init done 2024-06-03T09:06:48.428 INFO:teuthology.orchestra.run.smithi088.stderr:2024-06-03T09:06:48.424+0000 7f35567f6640 10 start_mon_command cmd=[{"prefix": "get_command_descriptions"}] 2024-06-03T09:06:48.428 INFO:teuthology.orchestra.run.smithi088.stderr:2024-06-03T09:06:48.424+0000 7f35567f6640 10 monclient: _send_command 1 [{"prefix": "get_command_descriptions"}] 2024-06-03T09:06:48.428 INFO:teuthology.orchestra.run.smithi088.stderr:2024-06-03T09:06:48.424+0000 7f35567f6640 10 monclient: _send_mon_message to mon.c at v1:172.21.15.88:6790/0 2024-06-03T09:06:48.429 INFO:teuthology.orchestra.run.smithi088.stderr:2024-06-03T09:06:48.424+0000 7f35567f6640 1 -- 172.21.15.88:0/3396045638 --> v1:172.21.15.88:6790/0 -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f35500a9e60 con 0x7f355013eec0 2024-06-03T09:06:48.635 INFO:teuthology.orchestra.run.smithi088.stderr:2024-06-03T09:06:48.635+0000 7f354d7fa640 1 -- 172.21.15.88:0/3396045638 <== mon.2 v1:172.21.15.88:6790/0 5 ==== config(0 keys) v1 ==== 4+0+0 (unknown 0 0 0) 0x7f3538004a90 con 0x7f355013eec0 2024-06-03T09:06:48.635 INFO:teuthology.orchestra.run.smithi088.stderr:2024-06-03T09:06:48.635+0000 7f354d7fa640 10 monclient: handle_config config(0 keys) v1 2024-06-03T09:06:48.635 INFO:teuthology.orchestra.run.smithi088.stderr:2024-06-03T09:06:48.635+0000 7f354effd640 4 set_mon_vals no callback set 2024-06-03T09:06:48.674 INFO:teuthology.orchestra.run.smithi088.stderr:2024-06-03T09:06:48.674+0000 7f354d7fa640 1 -- 172.21.15.88:0/3396045638 <== mon.2 v1:172.21.15.88:6790/0 6 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 2234374065 0 0) 0x7f3538004c00 con 0x7f355013eec0 2024-06-03T09:06:48.675 INFO:teuthology.orchestra.run.smithi088.stderr:2024-06-03T09:06:48.674+0000 7f354d7fa640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-06-03T09:06:48.675 INFO:teuthology.orchestra.run.smithi088.stderr:2024-06-03T09:06:48.674+0000 7f354d7fa640 10 monclient: got monmap 1 from mon.c (according to old e1) 2024-06-03T09:06:48.675 INFO:teuthology.orchestra.run.smithi088.stderr:2024-06-03T09:06:48.674+0000 7f354d7fa640 10 monclient: dump: 2024-06-03T09:06:48.675 INFO:teuthology.orchestra.run.smithi088.stderr:epoch 1 2024-06-03T09:06:48.675 INFO:teuthology.orchestra.run.smithi088.stderr:fsid ec3c3fdd-3ac6-4d7d-9412-683130ed8b04 2024-06-03T09:06:48.675 INFO:teuthology.orchestra.run.smithi088.stderr:last_changed 2024-06-03T08:34:31.885125+0000 2024-06-03T09:06:48.675 INFO:teuthology.orchestra.run.smithi088.stderr:created 2024-06-03T08:34:31.885125+0000 2024-06-03T09:06:48.675 INFO:teuthology.orchestra.run.smithi088.stderr:min_mon_release 18 (reef) 2024-06-03T09:06:48.675 INFO:teuthology.orchestra.run.smithi088.stderr:election_strategy: 3 2024-06-03T09:06:48.675 INFO:teuthology.orchestra.run.smithi088.stderr:0: v1:172.21.15.88:6789/0 mon.a 2024-06-03T09:06:48.675 INFO:teuthology.orchestra.run.smithi088.stderr:1: v1:172.21.15.192:6789/0 mon.b 2024-06-03T09:06:48.675 INFO:teuthology.orchestra.run.smithi088.stderr:2: v1:172.21.15.88:6790/0 mon.c 2024-06-03T09:06:48.675 INFO:teuthology.orchestra.run.smithi088.stderr: 2024-06-03T09:06:48.676 INFO:teuthology.orchestra.run.smithi088.stderr:2024-06-03T09:06:48.675+0000 7f354d7fa640 1 -- 172.21.15.88:0/3396045638 <== mon.2 v1:172.21.15.88:6790/0 7 ==== mgrmap(e 5) v1 ==== 89110+0+0 (unknown 873533973 0 0) 0x7f353801a980 con 0x7f355013eec0 2024-06-03T09:06:48.676 INFO:teuthology.orchestra.run.smithi088.stderr:2024-06-03T09:06:48.676+0000 7f354ffff640 10 monclient: get_auth_request con 0x7f353006d430 auth_method 0 2024-06-03T09:06:48.676 INFO:teuthology.orchestra.run.smithi088.stderr:2024-06-03T09:06:48.676+0000 7f354d7fa640 10 client.30209.objecter ms_handle_connect 0x7f353006d430 2024-06-03T09:06:48.683 INFO:teuthology.orchestra.run.smithi088.stderr:2024-06-03T09:06:48.682+0000 7f354d7fa640 1 -- 172.21.15.88:0/3396045638 <== mon.2 v1:172.21.15.88:6790/0 8 ==== osd_map(1014..1014 src has 622..1014) v4 ==== 5229+0+0 (unknown 3230192161 0 0) 0x7f3538054010 con 0x7f355013eec0 2024-06-03T09:06:48.683 INFO:teuthology.orchestra.run.smithi088.stderr:2024-06-03T09:06:48.682+0000 7f354d7fa640 10 client.30209.objecter ms_dispatch 0x7f3550000b90 osd_map(1014..1014 src has 622..1014) v4 2024-06-03T09:06:48.683 INFO:teuthology.orchestra.run.smithi088.stderr:2024-06-03T09:06:48.682+0000 7f354d7fa640 3 client.30209.objecter handle_osd_map got epochs [1014,1014] > 0 2024-06-03T09:06:48.683 INFO:teuthology.orchestra.run.smithi088.stderr:2024-06-03T09:06:48.682+0000 7f354d7fa640 3 client.30209.objecter handle_osd_map decoding full epoch 1014 2024-06-03T09:06:48.683 INFO:teuthology.orchestra.run.smithi088.stderr:2024-06-03T09:06:48.682+0000 7f354d7fa640 20 client.30209.objecter dump_active .. 0 homeless 2024-06-03T09:06:48.683 INFO:teuthology.orchestra.run.smithi088.stderr:2024-06-03T09:06:48.682+0000 7f354d7fa640 1 -- 172.21.15.88:0/3396045638 <== mon.2 v1:172.21.15.88:6790/0 9 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+182498 (unknown 1092875540 0 3016760700) 0x7f353804f9f0 con 0x7f355013eec0 2024-06-03T09:06:48.683 INFO:teuthology.orchestra.run.smithi088.stderr:2024-06-03T09:06:48.682+0000 7f354d7fa640 10 monclient: handle_mon_command_ack 1 [{"prefix": "get_command_descriptions"}] 2024-06-03T09:06:48.683 INFO:teuthology.orchestra.run.smithi088.stderr:2024-06-03T09:06:48.682+0000 7f354d7fa640 10 monclient: _finish_command 1 = system:0 2024-06-03T09:06:48.854 INFO:teuthology.orchestra.run.smithi088.stderr:2024-06-03T09:06:48.853+0000 7f35567f6640 1 -- 172.21.15.88:0/3396045638 --> v1:172.21.15.192:6800/34229 -- mgr_command(tid 0: {"prefix": "pg dump", "target": ["mon-mgr", ""], "format": "json"}) v1 -- 0x7f35500afd70 con 0x7f353006d430 2024-06-03T09:06:48.858 INFO:teuthology.orchestra.run.smithi088.stderr:2024-06-03T09:06:48.858+0000 7f354d7fa640 1 -- 172.21.15.88:0/3396045638 <== mgr.4110 v1:172.21.15.192:6800/34229 1 ==== mgr_command_reply(tid 0: 0 dumped all) v1 ==== 18+0+151706 (unknown 2965378022 0 3869215128) 0x7f35500afd70 con 0x7f353006d430 2024-06-03T09:06:48.862 INFO:teuthology.orchestra.run.smithi088.stdout: 2024-06-03T09:06:48.863 INFO:teuthology.orchestra.run.smithi088.stderr:dumped all 2024-06-03T09:06:48.867 INFO:teuthology.orchestra.run.smithi088.stderr:2024-06-03T09:06:48.861+0000 7f35567f6640 10 librados: watch_flush enter 2024-06-03T09:06:48.867 INFO:teuthology.orchestra.run.smithi088.stderr:2024-06-03T09:06:48.861+0000 7f35567f6640 10 librados: watch_flush exit 2024-06-03T09:06:48.867 INFO:teuthology.orchestra.run.smithi088.stderr:2024-06-03T09:06:48.861+0000 7f35567f6640 20 client.30209.objecter shutdown clearing up homeless session... 2024-06-03T09:06:48.867 INFO:teuthology.orchestra.run.smithi088.stderr:2024-06-03T09:06:48.861+0000 7f35567f6640 10 client.30209.objecter successfully canceled tick 2024-06-03T09:06:48.867 INFO:teuthology.orchestra.run.smithi088.stderr:2024-06-03T09:06:48.861+0000 7f35567f6640 1 -- 172.21.15.88:0/3396045638 >> v1:172.21.15.192:6800/34229 conn(0x7f353006d430 legacy=0x7f353006f8f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-06-03T09:06:48.868 INFO:teuthology.orchestra.run.smithi088.stderr:2024-06-03T09:06:48.861+0000 7f35567f6640 10 monclient: shutdown 2024-06-03T09:06:48.868 INFO:teuthology.orchestra.run.smithi088.stderr:2024-06-03T09:06:48.861+0000 7f35567f6640 20 monclient: shutdown discarding 0 pending message(s) 2024-06-03T09:06:48.868 INFO:teuthology.orchestra.run.smithi088.stderr:2024-06-03T09:06:48.861+0000 7f35567f6640 1 -- 172.21.15.88:0/3396045638 >> v1:172.21.15.88:6790/0 conn(0x7f355013eec0 legacy=0x7f35500adc00 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-06-03T09:06:48.868 INFO:teuthology.orchestra.run.smithi088.stderr:2024-06-03T09:06:48.861+0000 7f35567f6640 1 -- 172.21.15.88:0/3396045638 shutdown_connections 2024-06-03T09:06:48.868 INFO:teuthology.orchestra.run.smithi088.stderr:2024-06-03T09:06:48.861+0000 7f35567f6640 1 -- 172.21.15.88:0/3396045638 >> 172.21.15.88:0/3396045638 conn(0x7f35500b70d0 msgr2=0x7f355013e580 unknown :-1 s=STATE_NONE l=0).mark_down 2024-06-03T09:06:48.868 INFO:teuthology.orchestra.run.smithi088.stderr:2024-06-03T09:06:48.861+0000 7f35567f6640 1 -- 172.21.15.88:0/3396045638 shutdown_connections 2024-06-03T09:06:48.869 INFO:teuthology.orchestra.run.smithi088.stderr:2024-06-03T09:06:48.861+0000 7f35567f6640 1 -- 172.21.15.88:0/3396045638 wait complete. 2024-06-03T09:06:48.869 INFO:teuthology.orchestra.run.smithi088.stderr:2024-06-03T09:06:48.861+0000 7f35567f6640 1 librados: shutdown 2024-06-03T09:06:48.872 INFO:teuthology.orchestra.run.smithi088.stdout:{"pg_ready":true,"pg_map":{"version":1933,"stamp":"2024-06-03T09:06:47.103933+0000","last_osdmap_epoch":0,"last_pg_scan":0,"pg_stats_sum":{"stat_sum":{"num_bytes":459285,"num_objects":3,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":1,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":3,"num_whiteouts":0,"num_read":102,"num_read_kb":88,"num_write":120,"num_write_kb":1379,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2,"num_bytes_recovered":397840,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":83,"ondisk_log_size":83,"up":73,"acting":73,"num_store_stats":0},"osd_stats_sum":{"up_from":0,"seq":0,"num_pgs":82,"num_osds":8,"num_per_pool_osds":8,"num_per_pool_omap_osds":8,"kb":754974720,"kb_used":494764,"kb_used_data":46532,"kb_used_omap":12,"kb_used_meta":448115,"kb_avail":754479956,"statfs":{"total":773094113280,"available":772587474944,"internally_reserved":0,"allocated":47648768,"data_stored":22771099,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":12744,"internal_metadata":458870328},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":422,"apply_latency_ms":422,"commit_latency_ns":422000000,"apply_latency_ns":422000000},"alerts":[],"network_ping_times":[]},"pg_stats_delta":{"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":0,"acting":0,"num_store_stats":0,"stamp_delta":"4.269809"},"pg_stats":[{"pgid":"17.d","version":"0'0","reported_seq":942,"reported_epoch":1009,"state":"active+clean","last_fresh":"2024-06-03T09:06:14.443917+0000","last_change":"2024-06-03T08:38:19.260548+0000","last_active":"2024-06-03T09:06:14.443917+0000","last_peered":"2024-06-03T09:06:14.443917+0000","last_clean":"2024-06-03T09:06:14.443917+0000","last_became_active":"2024-06-03T08:38:19.253559+0000","last_became_peered":"2024-06-03T08:38:19.253559+0000","last_unstale":"2024-06-03T09:06:14.443917+0000","last_undegraded":"2024-06-03T09:06:14.443917+0000","last_fullsized":"2024-06-03T09:06:14.443917+0000","mapping_epoch":28,"log_start":"0'0","ondisk_log_start":"0'0","created":28,"last_epoch_clean":45,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-06-03T08:37:55.614497+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-06-03T08:37:55.614497+0000","last_clean_scrub_stamp":"2024-06-03T08:37:55.614497+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-06-04T19:00:12.470979+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1],"acting":[0,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"17.c","version":"0'0","reported_seq":955,"reported_epoch":1011,"state":"active+undersized","last_fresh":"2024-06-03T09:06:42.798184+0000","last_change":"2024-06-03T09:06:03.440074+0000","last_active":"2024-06-03T09:06:42.798184+0000","last_peered":"2024-06-03T09:06:42.798184+0000","last_clean":"2024-06-03T09:05:02.105638+0000","last_became_active":"2024-06-03T09:06:03.440074+0000","last_became_peered":"2024-06-03T09:06:03.440074+0000","last_unstale":"2024-06-03T09:06:42.798184+0000","last_undegraded":"2024-06-03T09:06:42.798184+0000","last_fullsized":"2024-06-03T09:06:03.377982+0000","mapping_epoch":999,"log_start":"0'0","ondisk_log_start":"0'0","created":28,"last_epoch_clean":255,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-06-03T08:37:55.614497+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-06-03T08:37:55.614497+0000","last_clean_scrub_stamp":"2024-06-03T08:37:55.614497+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-06-04T10:34:20.808557+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3],"acting":[3],"avail_no_missing":["3"],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"17.f","version":"0'0","reported_seq":843,"reported_epoch":1014,"state":"active+clean","last_fresh":"2024-06-03T09:06:46.306998+0000","last_change":"2024-06-03T09:06:46.306998+0000","last_active":"2024-06-03T09:06:46.306998+0000","last_peered":"2024-06-03T09:06:46.306998+0000","last_clean":"2024-06-03T09:06:46.306998+0000","last_became_active":"2024-06-03T09:06:46.301958+0000","last_became_peered":"2024-06-03T09:06:46.301958+0000","last_unstale":"2024-06-03T09:06:46.306998+0000","last_undegraded":"2024-06-03T09:06:46.306998+0000","last_fullsized":"2024-06-03T09:06:46.306998+0000","mapping_epoch":1013,"log_start":"0'0","ondisk_log_start":"0'0","created":28,"last_epoch_clean":1014,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-06-03T08:37:55.614497+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-06-03T08:37:55.614497+0000","last_clean_scrub_stamp":"2024-06-03T08:37:55.614497+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-06-04T14:30:23.089522+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,4],"acting":[6,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"17.e","version":"0'0","reported_seq":843,"reported_epoch":1014,"state":"active+clean","last_fresh":"2024-06-03T09:06:46.278835+0000","last_change":"2024-06-03T09:06:46.278835+0000","last_active":"2024-06-03T09:06:46.278835+0000","last_peered":"2024-06-03T09:06:46.278835+0000","last_clean":"2024-06-03T09:06:46.278835+0000","last_became_active":"2024-06-03T09:06:46.273941+0000","last_became_peered":"2024-06-03T09:06:46.273941+0000","last_unstale":"2024-06-03T09:06:46.278835+0000","last_undegraded":"2024-06-03T09:06:46.278835+0000","last_fullsized":"2024-06-03T09:06:46.278835+0000","mapping_epoch":1013,"log_start":"0'0","ondisk_log_start":"0'0","created":28,"last_epoch_clean":1014,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-06-03T08:37:55.614497+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-06-03T08:37:55.614497+0000","last_clean_scrub_stamp":"2024-06-03T08:37:55.614497+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-06-04T09:31:07.311650+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,5],"acting":[6,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"17.9","version":"0'0","reported_seq":947,"reported_epoch":1010,"state":"active+clean","last_fresh":"2024-06-03T09:06:18.488819+0000","last_change":"2024-06-03T08:38:04.772260+0000","last_active":"2024-06-03T09:06:18.488819+0000","last_peered":"2024-06-03T09:06:18.488819+0000","last_clean":"2024-06-03T09:06:18.488819+0000","last_became_active":"2024-06-03T08:38:04.763428+0000","last_became_peered":"2024-06-03T08:38:04.763428+0000","last_unstale":"2024-06-03T09:06:18.488819+0000","last_undegraded":"2024-06-03T09:06:18.488819+0000","last_fullsized":"2024-06-03T09:06:18.488819+0000","mapping_epoch":28,"log_start":"0'0","ondisk_log_start":"0'0","created":28,"last_epoch_clean":33,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-06-03T08:37:55.614497+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-06-03T08:37:55.614497+0000","last_clean_scrub_stamp":"2024-06-03T08:37:55.614497+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-06-04T10:55:10.773936+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,4],"acting":[7,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"17.8","version":"0'0","reported_seq":961,"reported_epoch":1009,"state":"active+undersized","last_fresh":"2024-06-03T09:06:14.430375+0000","last_change":"2024-06-03T09:06:03.388786+0000","last_active":"2024-06-03T09:06:14.430375+0000","last_peered":"2024-06-03T09:06:14.430375+0000","last_clean":"2024-06-03T09:04:43.297780+0000","last_became_active":"2024-06-03T09:06:03.388786+0000","last_became_peered":"2024-06-03T09:06:03.388786+0000","last_unstale":"2024-06-03T09:06:14.430375+0000","last_undegraded":"2024-06-03T09:06:14.430375+0000","last_fullsized":"2024-06-03T09:06:03.331754+0000","mapping_epoch":999,"log_start":"0'0","ondisk_log_start":"0'0","created":28,"last_epoch_clean":255,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-06-03T08:37:55.614497+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-06-03T08:37:55.614497+0000","last_clean_scrub_stamp":"2024-06-03T08:37:55.614497+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-06-04T09:18:44.765547+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0],"acting":[0],"avail_no_missing":["0"],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"17.b","version":"0'0","reported_seq":843,"reported_epoch":1014,"state":"active+clean","last_fresh":"2024-06-03T09:06:46.350450+0000","last_change":"2024-06-03T09:06:46.350450+0000","last_active":"2024-06-03T09:06:46.350450+0000","last_peered":"2024-06-03T09:06:46.350450+0000","last_clean":"2024-06-03T09:06:46.350450+0000","last_became_active":"2024-06-03T09:06:46.345458+0000","last_became_peered":"2024-06-03T09:06:46.345458+0000","last_unstale":"2024-06-03T09:06:46.350450+0000","last_undegraded":"2024-06-03T09:06:46.350450+0000","last_fullsized":"2024-06-03T09:06:46.350450+0000","mapping_epoch":1013,"log_start":"0'0","ondisk_log_start":"0'0","created":28,"last_epoch_clean":1014,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-06-03T08:37:55.614497+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-06-03T08:37:55.614497+0000","last_clean_scrub_stamp":"2024-06-03T08:37:55.614497+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-06-04T16:51:34.952224+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,2],"acting":[6,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"17.a","version":"0'0","reported_seq":927,"reported_epoch":1011,"state":"active+clean","last_fresh":"2024-06-03T09:06:42.861977+0000","last_change":"2024-06-03T08:38:05.843142+0000","last_active":"2024-06-03T09:06:42.861977+0000","last_peered":"2024-06-03T09:06:42.861977+0000","last_clean":"2024-06-03T09:06:42.861977+0000","last_became_active":"2024-06-03T08:38:05.809257+0000","last_became_peered":"2024-06-03T08:38:05.809257+0000","last_unstale":"2024-06-03T09:06:42.861977+0000","last_undegraded":"2024-06-03T09:06:42.861977+0000","last_fullsized":"2024-06-03T09:06:42.861977+0000","mapping_epoch":28,"log_start":"0'0","ondisk_log_start":"0'0","created":28,"last_epoch_clean":33,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-06-03T08:37:55.614497+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-06-03T08:37:55.614497+0000","last_clean_scrub_stamp":"2024-06-03T08:37:55.614497+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-06-04T12:12:03.686838+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2],"acting":[1,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"17.5","version":"0'0","reported_seq":943,"reported_epoch":1009,"state":"active+clean","last_fresh":"2024-06-03T09:06:14.442357+0000","last_change":"2024-06-03T08:38:23.466051+0000","last_active":"2024-06-03T09:06:14.442357+0000","last_peered":"2024-06-03T09:06:14.442357+0000","last_clean":"2024-06-03T09:06:14.442357+0000","last_became_active":"2024-06-03T08:38:23.458039+0000","last_became_peered":"2024-06-03T08:38:23.458039+0000","last_unstale":"2024-06-03T09:06:14.442357+0000","last_undegraded":"2024-06-03T09:06:14.442357+0000","last_fullsized":"2024-06-03T09:06:14.442357+0000","mapping_epoch":28,"log_start":"0'0","ondisk_log_start":"0'0","created":28,"last_epoch_clean":49,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-06-03T08:37:55.614497+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-06-03T08:37:55.614497+0000","last_clean_scrub_stamp":"2024-06-03T08:37:55.614497+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-06-04T12:27:02.532436+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,4],"acting":[0,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"17.4","version":"0'0","reported_seq":843,"reported_epoch":1014,"state":"active+clean","last_fresh":"2024-06-03T09:06:46.456388+0000","last_change":"2024-06-03T09:06:46.456388+0000","last_active":"2024-06-03T09:06:46.456388+0000","last_peered":"2024-06-03T09:06:46.456388+0000","last_clean":"2024-06-03T09:06:46.456388+0000","last_became_active":"2024-06-03T09:06:46.447534+0000","last_became_peered":"2024-06-03T09:06:46.447534+0000","last_unstale":"2024-06-03T09:06:46.456388+0000","last_undegraded":"2024-06-03T09:06:46.456388+0000","last_fullsized":"2024-06-03T09:06:46.456388+0000","mapping_epoch":1013,"log_start":"0'0","ondisk_log_start":"0'0","created":28,"last_epoch_clean":1014,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-06-03T08:37:55.614497+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-06-03T08:37:55.614497+0000","last_clean_scrub_stamp":"2024-06-03T08:37:55.614497+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-06-04T16:06:17.792570+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,7],"acting":[6,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"17.7","version":"0'0","reported_seq":929,"reported_epoch":1009,"state":"active+clean","last_fresh":"2024-06-03T09:06:13.736920+0000","last_change":"2024-06-03T08:38:04.837083+0000","last_active":"2024-06-03T09:06:13.736920+0000","last_peered":"2024-06-03T09:06:13.736920+0000","last_clean":"2024-06-03T09:06:13.736920+0000","last_became_active":"2024-06-03T08:38:04.818246+0000","last_became_peered":"2024-06-03T08:38:04.818246+0000","last_unstale":"2024-06-03T09:06:13.736920+0000","last_undegraded":"2024-06-03T09:06:13.736920+0000","last_fullsized":"2024-06-03T09:06:13.736920+0000","mapping_epoch":28,"log_start":"0'0","ondisk_log_start":"0'0","created":28,"last_epoch_clean":34,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-06-03T08:37:55.614497+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-06-03T08:37:55.614497+0000","last_clean_scrub_stamp":"2024-06-03T08:37:55.614497+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-06-04T17:02:38.204813+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,2],"acting":[5,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"17.6","version":"0'0","reported_seq":960,"reported_epoch":1009,"state":"active+undersized","last_fresh":"2024-06-03T09:06:14.453110+0000","last_change":"2024-06-03T09:06:03.428677+0000","last_active":"2024-06-03T09:06:14.453110+0000","last_peered":"2024-06-03T09:06:14.453110+0000","last_clean":"2024-06-03T09:04:43.312352+0000","last_became_active":"2024-06-03T09:06:03.428677+0000","last_became_peered":"2024-06-03T09:06:03.428677+0000","last_unstale":"2024-06-03T09:06:14.453110+0000","last_undegraded":"2024-06-03T09:06:14.453110+0000","last_fullsized":"2024-06-03T09:06:03.359909+0000","mapping_epoch":999,"log_start":"0'0","ondisk_log_start":"0'0","created":28,"last_epoch_clean":255,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-06-03T08:37:55.614497+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-06-03T08:37:55.614497+0000","last_clean_scrub_stamp":"2024-06-03T08:37:55.614497+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-06-04T15:38:02.180241+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0],"acting":[0],"avail_no_missing":["0"],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"2.7","version":"0'0","reported_seq":942,"reported_epoch":1011,"state":"active+clean","last_fresh":"2024-06-03T09:06:42.820718+0000","last_change":"2024-06-03T08:36:30.385405+0000","last_active":"2024-06-03T09:06:42.820718+0000","last_peered":"2024-06-03T09:06:42.820718+0000","last_clean":"2024-06-03T09:06:42.820718+0000","last_became_active":"2024-06-03T08:36:30.380302+0000","last_became_peered":"2024-06-03T08:36:30.380302+0000","last_unstale":"2024-06-03T09:06:42.820718+0000","last_undegraded":"2024-06-03T09:06:42.820718+0000","last_fullsized":"2024-06-03T09:06:42.820718+0000","mapping_epoch":23,"log_start":"0'0","ondisk_log_start":"0'0","created":23,"last_epoch_clean":24,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-06-03T08:36:28.444764+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-06-03T08:36:28.444764+0000","last_clean_scrub_stamp":"2024-06-03T08:36:28.444764+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-06-04T18:46:33.475181+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0],"acting":[3,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"17.14","version":"0'0","reported_seq":926,"reported_epoch":1009,"state":"active+clean","last_fresh":"2024-06-03T09:06:13.721460+0000","last_change":"2024-06-03T08:38:04.547565+0000","last_active":"2024-06-03T09:06:13.721460+0000","last_peered":"2024-06-03T09:06:13.721460+0000","last_clean":"2024-06-03T09:06:13.721460+0000","last_became_active":"2024-06-03T08:38:04.541781+0000","last_became_peered":"2024-06-03T08:38:04.541781+0000","last_unstale":"2024-06-03T09:06:13.721460+0000","last_undegraded":"2024-06-03T09:06:13.721460+0000","last_fullsized":"2024-06-03T09:06:13.721460+0000","mapping_epoch":28,"log_start":"0'0","ondisk_log_start":"0'0","created":28,"last_epoch_clean":34,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-06-03T08:37:55.614497+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-06-03T08:37:55.614497+0000","last_clean_scrub_stamp":"2024-06-03T08:37:55.614497+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-06-04T16:41:55.170134+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,2],"acting":[5,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"2.6","version":"0'0","reported_seq":950,"reported_epoch":1010,"state":"active+clean","last_fresh":"2024-06-03T09:06:19.062222+0000","last_change":"2024-06-03T08:36:30.430760+0000","last_active":"2024-06-03T09:06:19.062222+0000","last_peered":"2024-06-03T09:06:19.062222+0000","last_clean":"2024-06-03T09:06:19.062222+0000","last_became_active":"2024-06-03T08:36:30.404665+0000","last_became_peered":"2024-06-03T08:36:30.404665+0000","last_unstale":"2024-06-03T09:06:19.062222+0000","last_undegraded":"2024-06-03T09:06:19.062222+0000","last_fullsized":"2024-06-03T09:06:19.062222+0000","mapping_epoch":23,"log_start":"0'0","ondisk_log_start":"0'0","created":23,"last_epoch_clean":24,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-06-03T08:36:28.444764+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-06-03T08:36:28.444764+0000","last_clean_scrub_stamp":"2024-06-03T08:36:28.444764+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-06-04T10:37:37.746074+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,0],"acting":[4,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"17.15","version":"0'0","reported_seq":936,"reported_epoch":1011,"state":"active+clean","last_fresh":"2024-06-03T09:06:42.808375+0000","last_change":"2024-06-03T08:38:20.999903+0000","last_active":"2024-06-03T09:06:42.808375+0000","last_peered":"2024-06-03T09:06:42.808375+0000","last_clean":"2024-06-03T09:06:42.808375+0000","last_became_active":"2024-06-03T08:38:20.979523+0000","last_became_peered":"2024-06-03T08:38:20.979523+0000","last_unstale":"2024-06-03T09:06:42.808375+0000","last_undegraded":"2024-06-03T09:06:42.808375+0000","last_fullsized":"2024-06-03T09:06:42.808375+0000","mapping_epoch":28,"log_start":"0'0","ondisk_log_start":"0'0","created":28,"last_epoch_clean":47,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-06-03T08:37:55.614497+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-06-03T08:37:55.614497+0000","last_clean_scrub_stamp":"2024-06-03T08:37:55.614497+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-06-04T20:29:58.314289+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,1],"acting":[3,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"2.5","version":"0'0","reported_seq":944,"reported_epoch":1011,"state":"active+clean","last_fresh":"2024-06-03T09:06:42.818103+0000","last_change":"2024-06-03T08:36:30.024235+0000","last_active":"2024-06-03T09:06:42.818103+0000","last_peered":"2024-06-03T09:06:42.818103+0000","last_clean":"2024-06-03T09:06:42.818103+0000","last_became_active":"2024-06-03T08:36:30.019040+0000","last_became_peered":"2024-06-03T08:36:30.019040+0000","last_unstale":"2024-06-03T09:06:42.818103+0000","last_undegraded":"2024-06-03T09:06:42.818103+0000","last_fullsized":"2024-06-03T09:06:42.818103+0000","mapping_epoch":23,"log_start":"0'0","ondisk_log_start":"0'0","created":23,"last_epoch_clean":24,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-06-03T08:36:28.444764+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-06-03T08:36:28.444764+0000","last_clean_scrub_stamp":"2024-06-03T08:36:28.444764+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-06-04T15:46:47.109502+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,4],"acting":[3,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"17.16","version":"0'0","reported_seq":974,"reported_epoch":1011,"state":"active+undersized","last_fresh":"2024-06-03T09:06:42.992868+0000","last_change":"2024-06-03T09:06:03.478615+0000","last_active":"2024-06-03T09:06:42.992868+0000","last_peered":"2024-06-03T09:06:42.992868+0000","last_clean":"2024-06-03T09:04:43.328415+0000","last_became_active":"2024-06-03T09:06:03.478615+0000","last_became_peered":"2024-06-03T09:06:03.478615+0000","last_unstale":"2024-06-03T09:06:42.992868+0000","last_undegraded":"2024-06-03T09:06:42.992868+0000","last_fullsized":"2024-06-03T09:06:03.402504+0000","mapping_epoch":999,"log_start":"0'0","ondisk_log_start":"0'0","created":28,"last_epoch_clean":255,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-06-03T08:37:55.614497+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-06-03T08:37:55.614497+0000","last_clean_scrub_stamp":"2024-06-03T08:37:55.614497+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-06-04T08:47:08.079233+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2],"acting":[2],"avail_no_missing":["2"],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"2.4","version":"0'0","reported_seq":952,"reported_epoch":1010,"state":"active+undersized","last_fresh":"2024-06-03T09:06:18.331485+0000","last_change":"2024-06-03T09:06:03.387171+0000","last_active":"2024-06-03T09:06:18.331485+0000","last_peered":"2024-06-03T09:06:18.331485+0000","last_clean":"2024-06-03T09:04:43.202645+0000","last_became_active":"2024-06-03T09:06:03.387171+0000","last_became_peered":"2024-06-03T09:06:03.387171+0000","last_unstale":"2024-06-03T09:06:18.331485+0000","last_undegraded":"2024-06-03T09:06:18.331485+0000","last_fullsized":"2024-06-03T09:06:03.283787+0000","mapping_epoch":999,"log_start":"0'0","ondisk_log_start":"0'0","created":23,"last_epoch_clean":255,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-06-03T08:36:28.444764+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-06-03T08:36:28.444764+0000","last_clean_scrub_stamp":"2024-06-03T08:36:28.444764+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-06-04T20:17:43.485023+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5],"acting":[5],"avail_no_missing":["5"],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"17.17","version":"0'0","reported_seq":956,"reported_epoch":1011,"state":"active+clean","last_fresh":"2024-06-03T09:06:42.990396+0000","last_change":"2024-06-03T08:38:38.483275+0000","last_active":"2024-06-03T09:06:42.990396+0000","last_peered":"2024-06-03T09:06:42.990396+0000","last_clean":"2024-06-03T09:06:42.990396+0000","last_became_active":"2024-06-03T08:38:38.467916+0000","last_became_peered":"2024-06-03T08:38:38.467916+0000","last_unstale":"2024-06-03T09:06:42.990396+0000","last_undegraded":"2024-06-03T09:06:42.990396+0000","last_fullsized":"2024-06-03T09:06:42.990396+0000","mapping_epoch":28,"log_start":"0'0","ondisk_log_start":"0'0","created":28,"last_epoch_clean":61,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-06-03T08:37:55.614497+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-06-03T08:37:55.614497+0000","last_clean_scrub_stamp":"2024-06-03T08:37:55.614497+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-06-04T10:02:23.036420+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,7],"acting":[2,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"2.2","version":"0'0","reported_seq":965,"reported_epoch":1011,"state":"active+undersized","last_fresh":"2024-06-03T09:06:42.803839+0000","last_change":"2024-06-03T09:06:03.410020+0000","last_active":"2024-06-03T09:06:42.803839+0000","last_peered":"2024-06-03T09:06:42.803839+0000","last_clean":"2024-06-03T09:05:37.145889+0000","last_became_active":"2024-06-03T09:06:03.410020+0000","last_became_peered":"2024-06-03T09:06:03.410020+0000","last_unstale":"2024-06-03T09:06:42.803839+0000","last_undegraded":"2024-06-03T09:06:42.803839+0000","last_fullsized":"2024-06-03T09:06:03.346864+0000","mapping_epoch":999,"log_start":"0'0","ondisk_log_start":"0'0","created":23,"last_epoch_clean":255,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-06-03T08:36:28.444764+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-06-03T08:36:28.444764+0000","last_clean_scrub_stamp":"2024-06-03T08:36:28.444764+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-06-04T09:57:12.423360+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3],"acting":[3],"avail_no_missing":["3"],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"17.11","version":"0'0","reported_seq":944,"reported_epoch":1009,"state":"active+clean","last_fresh":"2024-06-03T09:06:14.426670+0000","last_change":"2024-06-03T08:38:06.502302+0000","last_active":"2024-06-03T09:06:14.426670+0000","last_peered":"2024-06-03T09:06:14.426670+0000","last_clean":"2024-06-03T09:06:14.426670+0000","last_became_active":"2024-06-03T08:38:06.490204+0000","last_became_peered":"2024-06-03T08:38:06.490204+0000","last_unstale":"2024-06-03T09:06:14.426670+0000","last_undegraded":"2024-06-03T09:06:14.426670+0000","last_fullsized":"2024-06-03T09:06:14.426670+0000","mapping_epoch":28,"log_start":"0'0","ondisk_log_start":"0'0","created":28,"last_epoch_clean":32,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-06-03T08:37:55.614497+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-06-03T08:37:55.614497+0000","last_clean_scrub_stamp":"2024-06-03T08:37:55.614497+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-06-04T10:47:05.988958+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,2],"acting":[4,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"2.1","version":"0'0","reported_seq":953,"reported_epoch":1010,"state":"active+clean","last_fresh":"2024-06-03T09:06:18.490362+0000","last_change":"2024-06-03T08:36:30.883360+0000","last_active":"2024-06-03T09:06:18.490362+0000","last_peered":"2024-06-03T09:06:18.490362+0000","last_clean":"2024-06-03T09:06:18.490362+0000","last_became_active":"2024-06-03T08:36:30.855073+0000","last_became_peered":"2024-06-03T08:36:30.855073+0000","last_unstale":"2024-06-03T09:06:18.490362+0000","last_undegraded":"2024-06-03T09:06:18.490362+0000","last_fullsized":"2024-06-03T09:06:18.490362+0000","mapping_epoch":23,"log_start":"0'0","ondisk_log_start":"0'0","created":23,"last_epoch_clean":25,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-06-03T08:36:28.444764+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-06-03T08:36:28.444764+0000","last_clean_scrub_stamp":"2024-06-03T08:36:28.444764+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-06-04T18:12:22.363473+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,5],"acting":[7,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"17.12","version":"0'0","reported_seq":946,"reported_epoch":1010,"state":"active+clean","last_fresh":"2024-06-03T09:06:18.482303+0000","last_change":"2024-06-03T08:38:01.751562+0000","last_active":"2024-06-03T09:06:18.482303+0000","last_peered":"2024-06-03T09:06:18.482303+0000","last_clean":"2024-06-03T09:06:18.482303+0000","last_became_active":"2024-06-03T08:38:01.745027+0000","last_became_peered":"2024-06-03T08:38:01.745027+0000","last_unstale":"2024-06-03T09:06:18.482303+0000","last_undegraded":"2024-06-03T09:06:18.482303+0000","last_fullsized":"2024-06-03T09:06:18.482303+0000","mapping_epoch":28,"log_start":"0'0","ondisk_log_start":"0'0","created":28,"last_epoch_clean":32,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-06-03T08:37:55.614497+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-06-03T08:37:55.614497+0000","last_clean_scrub_stamp":"2024-06-03T08:37:55.614497+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-06-04T17:54:43.785326+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,3],"acting":[7,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"2.0","version":"0'0","reported_seq":946,"reported_epoch":1012,"state":"active+clean","last_fresh":"2024-06-03T09:06:43.473891+0000","last_change":"2024-06-03T08:36:29.984096+0000","last_active":"2024-06-03T09:06:43.473891+0000","last_peered":"2024-06-03T09:06:43.473891+0000","last_clean":"2024-06-03T09:06:43.473891+0000","last_became_active":"2024-06-03T08:36:29.957875+0000","last_became_peered":"2024-06-03T08:36:29.957875+0000","last_unstale":"2024-06-03T09:06:43.473891+0000","last_undegraded":"2024-06-03T09:06:43.473891+0000","last_fullsized":"2024-06-03T09:06:43.473891+0000","mapping_epoch":23,"log_start":"0'0","ondisk_log_start":"0'0","created":23,"last_epoch_clean":24,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-06-03T08:36:28.444764+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-06-03T08:36:28.444764+0000","last_clean_scrub_stamp":"2024-06-03T08:36:28.444764+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-06-04T08:41:33.679479+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,5],"acting":[3,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"17.13","version":"0'0","reported_seq":943,"reported_epoch":1009,"state":"active+clean","last_fresh":"2024-06-03T09:06:14.438399+0000","last_change":"2024-06-03T08:38:26.838316+0000","last_active":"2024-06-03T09:06:14.438399+0000","last_peered":"2024-06-03T09:06:14.438399+0000","last_clean":"2024-06-03T09:06:14.438399+0000","last_became_active":"2024-06-03T08:38:26.833214+0000","last_became_peered":"2024-06-03T08:38:26.833214+0000","last_unstale":"2024-06-03T09:06:14.438399+0000","last_undegraded":"2024-06-03T09:06:14.438399+0000","last_fullsized":"2024-06-03T09:06:14.438399+0000","mapping_epoch":28,"log_start":"0'0","ondisk_log_start":"0'0","created":28,"last_epoch_clean":52,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-06-03T08:37:55.614497+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-06-03T08:37:55.614497+0000","last_clean_scrub_stamp":"2024-06-03T08:37:55.614497+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-06-04T12:58:51.617998+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,0],"acting":[4,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"2.3","version":"0'0","reported_seq":945,"reported_epoch":1011,"state":"active+clean","last_fresh":"2024-06-03T09:06:42.806748+0000","last_change":"2024-06-03T08:36:30.351670+0000","last_active":"2024-06-03T09:06:42.806748+0000","last_peered":"2024-06-03T09:06:42.806748+0000","last_clean":"2024-06-03T09:06:42.806748+0000","last_became_active":"2024-06-03T08:36:30.343560+0000","last_became_peered":"2024-06-03T08:36:30.343560+0000","last_unstale":"2024-06-03T09:06:42.806748+0000","last_undegraded":"2024-06-03T09:06:42.806748+0000","last_fullsized":"2024-06-03T09:06:42.806748+0000","mapping_epoch":23,"log_start":"0'0","ondisk_log_start":"0'0","created":23,"last_epoch_clean":24,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-06-03T08:36:28.444764+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-06-03T08:36:28.444764+0000","last_clean_scrub_stamp":"2024-06-03T08:36:28.444764+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-06-04T12:02:24.765135+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,2],"acting":[3,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.0","version":"19'76","reported_seq":1053,"reported_epoch":1011,"state":"active+clean","last_fresh":"2024-06-03T09:06:42.852938+0000","last_change":"2024-06-03T08:36:26.702702+0000","last_active":"2024-06-03T09:06:42.852938+0000","last_peered":"2024-06-03T09:06:42.852938+0000","last_clean":"2024-06-03T09:06:42.852938+0000","last_became_active":"2024-06-03T08:36:26.128378+0000","last_became_peered":"2024-06-03T08:36:26.128378+0000","last_unstale":"2024-06-03T09:06:42.852938+0000","last_undegraded":"2024-06-03T09:06:42.852938+0000","last_fullsized":"2024-06-03T09:06:42.852938+0000","mapping_epoch":20,"log_start":"0'0","ondisk_log_start":"0'0","created":16,"last_epoch_clean":22,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-06-03T08:36:15.542635+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-06-03T08:36:15.542635+0000","last_clean_scrub_stamp":"2024-06-03T08:36:15.542635+0000","objects_scrubbed":0,"log_size":76,"log_dups_size":0,"ondisk_log_size":76,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-06-04T12:41:53.731360+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":459280,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":96,"num_read_kb":82,"num_write":113,"num_write_kb":1372,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2,"num_bytes_recovered":397840,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,5],"acting":[1,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"17.10","version":"0'0","reported_seq":944,"reported_epoch":1009,"state":"active+clean","last_fresh":"2024-06-03T09:06:14.422035+0000","last_change":"2024-06-03T08:38:15.399189+0000","last_active":"2024-06-03T09:06:14.422035+0000","last_peered":"2024-06-03T09:06:14.422035+0000","last_clean":"2024-06-03T09:06:14.422035+0000","last_became_active":"2024-06-03T08:38:15.337723+0000","last_became_peered":"2024-06-03T08:38:15.337723+0000","last_unstale":"2024-06-03T09:06:14.422035+0000","last_undegraded":"2024-06-03T09:06:14.422035+0000","last_fullsized":"2024-06-03T09:06:14.422035+0000","mapping_epoch":28,"log_start":"0'0","ondisk_log_start":"0'0","created":28,"last_epoch_clean":42,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-06-03T08:37:55.614497+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-06-03T08:37:55.614497+0000","last_clean_scrub_stamp":"2024-06-03T08:37:55.614497+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-06-04T09:29:46.427317+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,3],"acting":[4,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"17.1a","version":"0'0","reported_seq":960,"reported_epoch":1009,"state":"active+undersized","last_fresh":"2024-06-03T09:06:14.431224+0000","last_change":"2024-06-03T09:06:03.390734+0000","last_active":"2024-06-03T09:06:14.431224+0000","last_peered":"2024-06-03T09:06:14.431224+0000","last_clean":"2024-06-03T09:05:20.322313+0000","last_became_active":"2024-06-03T09:06:03.390734+0000","last_became_peered":"2024-06-03T09:06:03.390734+0000","last_unstale":"2024-06-03T09:06:14.431224+0000","last_undegraded":"2024-06-03T09:06:14.431224+0000","last_fullsized":"2024-06-03T09:06:03.343197+0000","mapping_epoch":999,"log_start":"0'0","ondisk_log_start":"0'0","created":28,"last_epoch_clean":254,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-06-03T08:37:55.614497+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-06-03T08:37:55.614497+0000","last_clean_scrub_stamp":"2024-06-03T08:37:55.614497+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-06-04T15:56:47.628875+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4],"acting":[4],"avail_no_missing":["4"],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"17.1b","version":"0'0","reported_seq":937,"reported_epoch":1011,"state":"active+clean","last_fresh":"2024-06-03T09:06:42.815385+0000","last_change":"2024-06-03T08:38:03.115748+0000","last_active":"2024-06-03T09:06:42.815385+0000","last_peered":"2024-06-03T09:06:42.815385+0000","last_clean":"2024-06-03T09:06:42.815385+0000","last_became_active":"2024-06-03T08:38:03.110533+0000","last_became_peered":"2024-06-03T08:38:03.110533+0000","last_unstale":"2024-06-03T09:06:42.815385+0000","last_undegraded":"2024-06-03T09:06:42.815385+0000","last_fullsized":"2024-06-03T09:06:42.815385+0000","mapping_epoch":28,"log_start":"0'0","ondisk_log_start":"0'0","created":28,"last_epoch_clean":32,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-06-03T08:37:55.614497+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-06-03T08:37:55.614497+0000","last_clean_scrub_stamp":"2024-06-03T08:37:55.614497+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-06-04T12:36:47.471609+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,7],"acting":[3,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"17.18","version":"0'0","reported_seq":945,"reported_epoch":1010,"state":"active+clean","last_fresh":"2024-06-03T09:06:18.484214+0000","last_change":"2024-06-03T08:38:04.632864+0000","last_active":"2024-06-03T09:06:18.484214+0000","last_peered":"2024-06-03T09:06:18.484214+0000","last_clean":"2024-06-03T09:06:18.484214+0000","last_became_active":"2024-06-03T08:38:04.627856+0000","last_became_peered":"2024-06-03T08:38:04.627856+0000","last_unstale":"2024-06-03T09:06:18.484214+0000","last_undegraded":"2024-06-03T09:06:18.484214+0000","last_fullsized":"2024-06-03T09:06:18.484214+0000","mapping_epoch":28,"log_start":"0'0","ondisk_log_start":"0'0","created":28,"last_epoch_clean":33,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-06-03T08:37:55.614497+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-06-03T08:37:55.614497+0000","last_clean_scrub_stamp":"2024-06-03T08:37:55.614497+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-06-04T19:55:51.181598+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,4],"acting":[7,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"17.19","version":"0'0","reported_seq":936,"reported_epoch":1011,"state":"active+clean","last_fresh":"2024-06-03T09:06:42.812989+0000","last_change":"2024-06-03T08:38:01.839491+0000","last_active":"2024-06-03T09:06:42.812989+0000","last_peered":"2024-06-03T09:06:42.812989+0000","last_clean":"2024-06-03T09:06:42.812989+0000","last_became_active":"2024-06-03T08:38:01.829539+0000","last_became_peered":"2024-06-03T08:38:01.829539+0000","last_unstale":"2024-06-03T09:06:42.812989+0000","last_undegraded":"2024-06-03T09:06:42.812989+0000","last_fullsized":"2024-06-03T09:06:42.812989+0000","mapping_epoch":28,"log_start":"0'0","ondisk_log_start":"0'0","created":28,"last_epoch_clean":30,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-06-03T08:37:55.614497+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-06-03T08:37:55.614497+0000","last_clean_scrub_stamp":"2024-06-03T08:37:55.614497+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-06-04T09:10:43.860229+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,2],"acting":[3,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"17.1e","version":"0'0","reported_seq":844,"reported_epoch":1014,"state":"active+clean","last_fresh":"2024-06-03T09:06:46.297025+0000","last_change":"2024-06-03T09:06:46.297025+0000","last_active":"2024-06-03T09:06:46.297025+0000","last_peered":"2024-06-03T09:06:46.297025+0000","last_clean":"2024-06-03T09:06:46.297025+0000","last_became_active":"2024-06-03T09:06:46.288593+0000","last_became_peered":"2024-06-03T09:06:46.288593+0000","last_unstale":"2024-06-03T09:06:46.297025+0000","last_undegraded":"2024-06-03T09:06:46.297025+0000","last_fullsized":"2024-06-03T09:06:46.297025+0000","mapping_epoch":1013,"log_start":"0'0","ondisk_log_start":"0'0","created":28,"last_epoch_clean":1014,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-06-03T08:37:55.614497+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-06-03T08:37:55.614497+0000","last_clean_scrub_stamp":"2024-06-03T08:37:55.614497+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-06-04T20:09:30.144366+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,4],"acting":[6,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"17.1f","version":"0'0","reported_seq":941,"reported_epoch":1009,"state":"active+clean","last_fresh":"2024-06-03T09:06:14.450053+0000","last_change":"2024-06-03T08:38:04.383538+0000","last_active":"2024-06-03T09:06:14.450053+0000","last_peered":"2024-06-03T09:06:14.450053+0000","last_clean":"2024-06-03T09:06:14.450053+0000","last_became_active":"2024-06-03T08:38:04.367278+0000","last_became_peered":"2024-06-03T08:38:04.367278+0000","last_unstale":"2024-06-03T09:06:14.450053+0000","last_undegraded":"2024-06-03T09:06:14.450053+0000","last_fullsized":"2024-06-03T09:06:14.450053+0000","mapping_epoch":28,"log_start":"0'0","ondisk_log_start":"0'0","created":28,"last_epoch_clean":32,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-06-03T08:37:55.614497+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-06-03T08:37:55.614497+0000","last_clean_scrub_stamp":"2024-06-03T08:37:55.614497+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-06-04T10:39:26.552282+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,2],"acting":[0,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"17.1c","version":"0'0","reported_seq":944,"reported_epoch":1010,"state":"active+undersized","last_fresh":"2024-06-03T09:06:18.327492+0000","last_change":"2024-06-03T09:06:03.382210+0000","last_active":"2024-06-03T09:06:18.327492+0000","last_peered":"2024-06-03T09:06:18.327492+0000","last_clean":"2024-06-03T09:04:43.199660+0000","last_became_active":"2024-06-03T09:06:03.382210+0000","last_became_peered":"2024-06-03T09:06:03.382210+0000","last_unstale":"2024-06-03T09:06:18.327492+0000","last_undegraded":"2024-06-03T09:06:18.327492+0000","last_fullsized":"2024-06-03T09:06:03.277874+0000","mapping_epoch":999,"log_start":"0'0","ondisk_log_start":"0'0","created":28,"last_epoch_clean":255,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-06-03T08:37:55.614497+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-06-03T08:37:55.614497+0000","last_clean_scrub_stamp":"2024-06-03T08:37:55.614497+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-06-04T12:18:50.013884+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5],"acting":[5],"avail_no_missing":["5"],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"17.1d","version":"0'0","reported_seq":925,"reported_epoch":1011,"state":"active+clean","last_fresh":"2024-06-03T09:06:42.858234+0000","last_change":"2024-06-03T08:38:06.794630+0000","last_active":"2024-06-03T09:06:42.858234+0000","last_peered":"2024-06-03T09:06:42.858234+0000","last_clean":"2024-06-03T09:06:42.858234+0000","last_became_active":"2024-06-03T08:38:06.592317+0000","last_became_peered":"2024-06-03T08:38:06.592317+0000","last_unstale":"2024-06-03T09:06:42.858234+0000","last_undegraded":"2024-06-03T09:06:42.858234+0000","last_fullsized":"2024-06-03T09:06:42.858234+0000","mapping_epoch":28,"log_start":"0'0","ondisk_log_start":"0'0","created":28,"last_epoch_clean":33,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-06-03T08:37:55.614497+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-06-03T08:37:55.614497+0000","last_clean_scrub_stamp":"2024-06-03T08:37:55.614497+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-06-04T12:33:30.785369+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,7],"acting":[1,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"17.2","version":"0'0","reported_seq":927,"reported_epoch":1011,"state":"active+clean","last_fresh":"2024-06-03T09:06:42.859877+0000","last_change":"2024-06-03T08:38:14.981410+0000","last_active":"2024-06-03T09:06:42.859877+0000","last_peered":"2024-06-03T09:06:42.859877+0000","last_clean":"2024-06-03T09:06:42.859877+0000","last_became_active":"2024-06-03T08:38:14.973825+0000","last_became_peered":"2024-06-03T08:38:14.973825+0000","last_unstale":"2024-06-03T09:06:42.859877+0000","last_undegraded":"2024-06-03T09:06:42.859877+0000","last_fullsized":"2024-06-03T09:06:42.859877+0000","mapping_epoch":28,"log_start":"0'0","ondisk_log_start":"0'0","created":28,"last_epoch_clean":41,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-06-03T08:37:55.614497+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-06-03T08:37:55.614497+0000","last_clean_scrub_stamp":"2024-06-03T08:37:55.614497+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-06-04T17:01:05.518613+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,4],"acting":[1,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"17.3","version":"0'0","reported_seq":943,"reported_epoch":1009,"state":"active+clean","last_fresh":"2024-06-03T09:06:14.437636+0000","last_change":"2024-06-03T08:38:14.619745+0000","last_active":"2024-06-03T09:06:14.437636+0000","last_peered":"2024-06-03T09:06:14.437636+0000","last_clean":"2024-06-03T09:06:14.437636+0000","last_became_active":"2024-06-03T08:38:14.510665+0000","last_became_peered":"2024-06-03T08:38:14.510665+0000","last_unstale":"2024-06-03T09:06:14.437636+0000","last_undegraded":"2024-06-03T09:06:14.437636+0000","last_fullsized":"2024-06-03T09:06:14.437636+0000","mapping_epoch":28,"log_start":"0'0","ondisk_log_start":"0'0","created":28,"last_epoch_clean":42,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-06-03T08:37:55.614497+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-06-03T08:37:55.614497+0000","last_clean_scrub_stamp":"2024-06-03T08:37:55.614497+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-06-04T12:03:47.392064+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,7],"acting":[0,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"17.0","version":"0'0","reported_seq":944,"reported_epoch":1009,"state":"active+clean","last_fresh":"2024-06-03T09:06:14.427123+0000","last_change":"2024-06-03T08:38:04.237118+0000","last_active":"2024-06-03T09:06:14.427123+0000","last_peered":"2024-06-03T09:06:14.427123+0000","last_clean":"2024-06-03T09:06:14.427123+0000","last_became_active":"2024-06-03T08:38:04.231954+0000","last_became_peered":"2024-06-03T08:38:04.231954+0000","last_unstale":"2024-06-03T09:06:14.427123+0000","last_undegraded":"2024-06-03T09:06:14.427123+0000","last_fullsized":"2024-06-03T09:06:14.427123+0000","mapping_epoch":28,"log_start":"0'0","ondisk_log_start":"0'0","created":28,"last_epoch_clean":32,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-06-03T08:37:55.614497+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-06-03T08:37:55.614497+0000","last_clean_scrub_stamp":"2024-06-03T08:37:55.614497+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-06-04T09:56:10.741637+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,2],"acting":[0,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"17.1","version":"65'7","reported_seq":959,"reported_epoch":1009,"state":"active+undersized+degraded","last_fresh":"2024-06-03T09:06:13.709209+0000","last_change":"2024-06-03T09:06:03.370660+0000","last_active":"2024-06-03T09:06:13.709209+0000","last_peered":"2024-06-03T09:06:13.709209+0000","last_clean":"2024-06-03T09:05:03.733864+0000","last_became_active":"2024-06-03T09:06:03.370660+0000","last_became_peered":"2024-06-03T09:06:03.370660+0000","last_unstale":"2024-06-03T09:06:13.709209+0000","last_undegraded":"2024-06-03T09:06:03.266076+0000","last_fullsized":"2024-06-03T09:06:03.264525+0000","mapping_epoch":999,"log_start":"0'0","ondisk_log_start":"0'0","created":28,"last_epoch_clean":255,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-06-03T08:37:55.614497+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-06-03T08:37:55.614497+0000","last_clean_scrub_stamp":"2024-06-03T08:37:55.614497+0000","objects_scrubbed":0,"log_size":7,"log_dups_size":0,"ondisk_log_size":7,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-06-04T11:55:18.194283+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":5,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":1,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":6,"num_read_kb":6,"num_write":7,"num_write_kb":7,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5],"acting":[5],"avail_no_missing":["5"],"object_location_counts":[{"shards":"5","objects":1}],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]}],"pool_stats":[{"poolid":2,"num_pg":8,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":14,"acting":14,"num_store_stats":7},{"poolid":1,"num_pg":1,"stat_sum":{"num_bytes":459280,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":96,"num_read_kb":82,"num_write":113,"num_write_kb":1372,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2,"num_bytes_recovered":397840,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":802816,"data_stored":795680,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":76,"ondisk_log_size":76,"up":2,"acting":2,"num_store_stats":2},{"poolid":17,"num_pg":32,"stat_sum":{"num_bytes":5,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":1,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":6,"num_read_kb":6,"num_write":7,"num_write_kb":7,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":8192,"data_stored":10,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":7,"ondisk_log_size":7,"up":57,"acting":57,"num_store_stats":8}],"osd_stats":[{"osd":6,"up_from":1013,"seq":4350801871162,"num_pgs":14,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":54908,"kb_used_data":5744,"kb_used_omap":1,"kb_used_meta":49150,"kb_avail":94316932,"statfs":{"total":96636764160,"available":96580538368,"internally_reserved":0,"allocated":5881856,"data_stored":2757601,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1593,"internal_metadata":50330055},"hb_peers":[0,1,2,3,4,5,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":403,"apply_latency_ms":403,"commit_latency_ns":403000000,"apply_latency_ns":403000000},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Mon Jun 3 09:06:42 2024","interfaces":[{"interface":"back","average":{"1min":5696.3109999999997,"5min":1469.3900000000001,"15min":503.11700000000002},"min":{"1min":5.242,"5min":4.4829999999999997,"15min":4.1959999999999997},"max":{"1min":20463.029999999999,"5min":22766.882000000001,"15min":22766.882000000001},"last":7.3399999999999999},{"interface":"front","average":{"1min":5687.6390000000001,"5min":1468.1669999999999,"15min":504.13200000000001},"min":{"1min":4.835,"5min":4.8129999999999997,"15min":4.4530000000000003},"max":{"1min":20461.830000000002,"5min":22766.289000000001,"15min":22766.289000000001},"last":8.1010000000000009}]},{"osd":1,"last update":"Mon Jun 3 09:06:42 2024","interfaces":[{"interface":"back","average":{"1min":36.496000000000002,"5min":80.162999999999997,"15min":134.59399999999999},"min":{"1min":5.9189999999999996,"5min":5.8799999999999999,"15min":5.7119999999999997},"max":{"1min":544.85799999999995,"5min":4138.1679999999997,"15min":9955.1299999999992},"last":9.4870000000000001},{"interface":"front","average":{"1min":38.866999999999997,"5min":82.631,"15min":135.11500000000001},"min":{"1min":6.5350000000000001,"5min":6.5350000000000001,"15min":6.1390000000000002},"max":{"1min":546.73099999999999,"5min":4137.6530000000002,"15min":9827.5509999999995},"last":15.617000000000001}]},{"osd":2,"last update":"Mon Jun 3 09:06:42 2024","interfaces":[{"interface":"back","average":{"1min":41.093000000000004,"5min":27.917000000000002,"15min":22.329000000000001},"min":{"1min":6.0670000000000002,"5min":6.0670000000000002,"15min":6.0670000000000002},"max":{"1min":548.12900000000002,"5min":548.12900000000002,"15min":548.12900000000002},"last":27.196999999999999},{"interface":"front","average":{"1min":30.798999999999999,"5min":24.276,"15min":22.488},"min":{"1min":6.4370000000000003,"5min":6.4370000000000003,"15min":6.1870000000000003},"max":{"1min":369.14499999999998,"5min":369.14499999999998,"15min":434.77300000000002},"last":28.960999999999999}]},{"osd":3,"last update":"Mon Jun 3 09:06:42 2024","interfaces":[{"interface":"back","average":{"1min":41.411999999999999,"5min":27.434999999999999,"15min":30.657},"min":{"1min":10.061,"5min":8.3179999999999996,"15min":6.1719999999999997},"max":{"1min":549.024,"5min":549.024,"15min":1235.5799999999999},"last":16.423999999999999},{"interface":"front","average":{"1min":37.850999999999999,"5min":22.001000000000001,"15min":28.015999999999998},"min":{"1min":6.0679999999999996,"5min":6.0679999999999996,"15min":6.0679999999999996},"max":{"1min":559.25,"5min":559.25,"15min":1157.6780000000001},"last":14.327}]},{"osd":4,"last update":"Mon Jun 3 09:06:42 2024","interfaces":[{"interface":"back","average":{"1min":165.62899999999999,"5min":1281.1010000000001,"15min":440.59199999999998},"min":{"1min":10.965999999999999,"5min":6.093,"15min":6.093},"max":{"1min":384.46899999999999,"5min":28093.096000000001,"15min":28093.096000000001},"last":11.307},{"interface":"front","average":{"1min":186.97900000000001,"5min":1308.886,"15min":454.97699999999998},"min":{"1min":13.581,"5min":7.7480000000000002,"15min":7.7480000000000002},"max":{"1min":445.096,"5min":28323.837,"15min":28323.837},"last":17.109000000000002}]},{"osd":5,"last update":"Mon Jun 3 09:06:42 2024","interfaces":[{"interface":"back","average":{"1min":81.581000000000003,"5min":34.863,"15min":28.350999999999999},"min":{"1min":10.345000000000001,"5min":8.625,"15min":7.516},"max":{"1min":455.84199999999998,"5min":455.84199999999998,"15min":455.84199999999998},"last":17.791},{"interface":"front","average":{"1min":83.391000000000005,"5min":39.277999999999999,"15min":27.885000000000002},"min":{"1min":8.0280000000000005,"5min":6.266,"15min":6.266},"max":{"1min":550.02300000000002,"5min":550.02300000000002,"15min":550.02300000000002},"last":11.814}]},{"osd":7,"last update":"Mon Jun 3 09:06:42 2024","interfaces":[{"interface":"back","average":{"1min":44.052999999999997,"5min":33.697000000000003,"15min":29.731999999999999},"min":{"1min":9.1500000000000004,"5min":9.1500000000000004,"15min":9.1500000000000004},"max":{"1min":552.09500000000003,"5min":552.09500000000003,"15min":552.09500000000003},"last":18.486999999999998},{"interface":"front","average":{"1min":47.076999999999998,"5min":34.226999999999997,"15min":29.827000000000002},"min":{"1min":9.7780000000000005,"5min":8.6709999999999994,"15min":8.5820000000000007},"max":{"1min":531.08600000000001,"5min":531.08600000000001,"15min":531.08600000000001},"last":13.776}]}]},{"osd":7,"up_from":20,"seq":85899346290,"num_pgs":9,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":58964,"kb_used_data":5704,"kb_used_omap":1,"kb_used_meta":53246,"kb_avail":94312876,"statfs":{"total":96636764160,"available":96576385024,"internally_reserved":0,"allocated":5840896,"data_stored":2740727,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1589,"internal_metadata":54524363},"hb_peers":[0,1,2,3,4,5],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":5,"apply_latency_ms":5,"commit_latency_ns":5000000,"apply_latency_ns":5000000},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Mon Jun 3 09:06:09 2024","interfaces":[{"interface":"back","average":{"1min":14.073,"5min":894.22400000000005,"15min":308.93400000000003},"min":{"1min":5.5709999999999997,"5min":4.218,"15min":4.0599999999999996},"max":{"1min":90.757999999999996,"5min":21729.001,"15min":21729.001},"last":9.4469999999999992},{"interface":"front","average":{"1min":9.2550000000000008,"5min":906.23500000000001,"15min":314.06},"min":{"1min":5.5789999999999997,"5min":4.8810000000000002,"15min":4.8810000000000002},"max":{"1min":25.937000000000001,"5min":21899.732,"15min":21899.732},"last":4.5979999999999999}]},{"osd":1,"last update":"Mon Jun 3 09:06:09 2024","interfaces":[{"interface":"back","average":{"1min":15.182,"5min":69.183999999999997,"15min":219.173},"min":{"1min":7.3639999999999999,"5min":6.6200000000000001,"15min":6.1319999999999997},"max":{"1min":76.926000000000002,"5min":2795.1329999999998,"15min":18373.257000000001},"last":5.8630000000000004},{"interface":"front","average":{"1min":21.495999999999999,"5min":74.703999999999994,"15min":225.285},"min":{"1min":8.8399999999999999,"5min":8.8399999999999999,"15min":8.5129999999999999},"max":{"1min":94.977999999999994,"5min":2803.3670000000002,"15min":18374.601999999999},"last":10.94}]},{"osd":2,"last update":"Mon Jun 3 09:06:09 2024","interfaces":[{"interface":"back","average":{"1min":19.164000000000001,"5min":26.902000000000001,"15min":22.920999999999999},"min":{"1min":11.207000000000001,"5min":8.5030000000000001,"15min":8.5030000000000001},"max":{"1min":91.319999999999993,"5min":333.99200000000002,"15min":333.99200000000002},"last":10.095000000000001},{"interface":"front","average":{"1min":11.125999999999999,"5min":18.132000000000001,"15min":16.353000000000002},"min":{"1min":6.202,"5min":5.8369999999999997,"15min":5.5069999999999997},"max":{"1min":28.649000000000001,"5min":226.18100000000001,"15min":226.18100000000001},"last":12.922000000000001}]},{"osd":3,"last update":"Mon Jun 3 09:06:09 2024","interfaces":[{"interface":"back","average":{"1min":22.291,"5min":26.289999999999999,"15min":31.074999999999999},"min":{"1min":11.714,"5min":6.8959999999999999,"15min":6.3760000000000003},"max":{"1min":83.102000000000004,"5min":316.64400000000001,"15min":2039.9059999999999},"last":7.617},{"interface":"front","average":{"1min":13.324999999999999,"5min":17.963000000000001,"15min":25.969999999999999},"min":{"1min":6.1200000000000001,"5min":5.2089999999999996,"15min":5.2089999999999996},"max":{"1min":49.908999999999999,"5min":233.542,"15min":2037.5550000000001},"last":8.1479999999999997}]},{"osd":4,"last update":"Mon Jun 3 09:06:29 2024","interfaces":[{"interface":"back","average":{"1min":17.675999999999998,"5min":1026.325,"15min":472.47199999999998},"min":{"1min":7.306,"5min":5.9089999999999998,"15min":5.9089999999999998},"max":{"1min":140.03299999999999,"5min":28969.369999999999,"15min":30073.917000000001},"last":12.000999999999999},{"interface":"front","average":{"1min":17.890999999999998,"5min":1044.5139999999999,"15min":480.226},"min":{"1min":5.9279999999999999,"5min":5.9279999999999999,"15min":5.9279999999999999},"max":{"1min":141.179,"5min":29244.138999999999,"15min":30348.021000000001},"last":15.622}]},{"osd":5,"last update":"Mon Jun 3 09:06:02 2024","interfaces":[{"interface":"back","average":{"1min":24.372,"5min":26.856000000000002,"15min":23.571999999999999},"min":{"1min":12.278,"5min":9.2460000000000004,"15min":6.5650000000000004},"max":{"1min":183.64099999999999,"5min":317.572,"15min":317.572},"last":11.449999999999999},{"interface":"front","average":{"1min":21.259,"5min":23.364000000000001,"15min":21.193999999999999},"min":{"1min":7.8499999999999996,"5min":7.0999999999999996,"15min":7.008},"max":{"1min":184.53399999999999,"5min":306.60599999999999,"15min":306.60599999999999},"last":14.676}]},{"osd":6,"last update":"Mon Jun 3 09:04:31 2024","interfaces":[{"interface":"back","average":{"1min":44.061,"5min":26.821000000000002,"15min":91.176000000000002},"min":{"1min":9.0800000000000001,"5min":7.9390000000000001,"15min":7.9390000000000001},"max":{"1min":268.029,"5min":308.916,"15min":8734.4639999999999},"last":6.6769999999999996},{"interface":"front","average":{"1min":46.643000000000001,"5min":23.879999999999999,"15min":88.765000000000001},"min":{"1min":8.8119999999999994,"5min":6.9240000000000004,"15min":6.742},"max":{"1min":305.64600000000002,"5min":305.64600000000002,"15min":8735.5319999999992},"last":12.327}]}]},{"osd":5,"up_from":20,"seq":85899346264,"num_pgs":9,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":59360,"kb_used_data":6100,"kb_used_omap":1,"kb_used_meta":53246,"kb_avail":94312480,"statfs":{"total":96636764160,"available":96575979520,"internally_reserved":0,"allocated":6246400,"data_stored":3138572,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1594,"internal_metadata":54524358},"hb_peers":[0,1,2,3,4,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Mon Jun 3 09:06:06 2024","interfaces":[{"interface":"back","average":{"1min":7.468,"5min":1703.037,"15min":580.51900000000001},"min":{"1min":4.6509999999999998,"5min":4.6509999999999998,"15min":4.4020000000000001},"max":{"1min":14.711,"5min":26433.764999999999,"15min":26433.764999999999},"last":5.8639999999999999},{"interface":"front","average":{"1min":9.9429999999999996,"5min":1725.27,"15min":587.87800000000004},"min":{"1min":5.5999999999999996,"5min":5.1319999999999997,"15min":4.8170000000000002},"max":{"1min":13.590999999999999,"5min":26704.698,"15min":26704.698},"last":12.747999999999999}]},{"osd":1,"last update":"Mon Jun 3 09:06:00 2024","interfaces":[{"interface":"back","average":{"1min":10.862,"5min":67.731999999999999,"15min":217.364},"min":{"1min":7.1769999999999996,"5min":6.0890000000000004,"15min":5.4930000000000003},"max":{"1min":16.998999999999999,"5min":3923.4090000000001,"15min":18811.696},"last":14.052},{"interface":"front","average":{"1min":10.417,"5min":65.141999999999996,"15min":220.75700000000001},"min":{"1min":5.3209999999999997,"5min":5.3209999999999997,"15min":5.0209999999999999},"max":{"1min":15.821999999999999,"5min":4019.6129999999998,"15min":18968.365000000002},"last":8.7219999999999995}]},{"osd":2,"last update":"Mon Jun 3 09:06:00 2024","interfaces":[{"interface":"back","average":{"1min":11.366,"5min":24.18,"15min":21.137},"min":{"1min":5.6369999999999996,"5min":5.6369999999999996,"15min":5.0510000000000002},"max":{"1min":15.529999999999999,"5min":332.48200000000003,"15min":332.48200000000003},"last":12.167},{"interface":"front","average":{"1min":10.089,"5min":23.117000000000001,"15min":24.050000000000001},"min":{"1min":6.6769999999999996,"5min":6.6769999999999996,"15min":5.8739999999999997},"max":{"1min":15.821,"5min":353.03199999999998,"15min":353.03199999999998},"last":7.2619999999999996}]},{"osd":3,"last update":"Mon Jun 3 09:06:00 2024","interfaces":[{"interface":"back","average":{"1min":13.472,"5min":24.283999999999999,"15min":31.666},"min":{"1min":10.111000000000001,"5min":7.4720000000000004,"15min":7.4210000000000003},"max":{"1min":16.271999999999998,"5min":353.57499999999999,"15min":2855.8519999999999},"last":8.1140000000000008},{"interface":"front","average":{"1min":9.5899999999999999,"5min":15.875,"15min":25.834},"min":{"1min":6.6609999999999996,"5min":5.7729999999999997,"15min":5.1909999999999998},"max":{"1min":15.112,"5min":203.804,"15min":2869.7860000000001},"last":9.3699999999999992}]},{"osd":4,"last update":"Mon Jun 3 09:06:29 2024","interfaces":[{"interface":"back","average":{"1min":11.067,"5min":780.21000000000004,"15min":377.28399999999999},"min":{"1min":5.4779999999999998,"5min":5.4779999999999998,"15min":5.4779999999999998},"max":{"1min":14.654999999999999,"5min":23315.438999999998,"15min":28619.825000000001},"last":6.7199999999999998},{"interface":"front","average":{"1min":14.349,"5min":782.97299999999996,"15min":377.93400000000003},"min":{"1min":7.492,"5min":6.5030000000000001,"15min":6.5030000000000001},"max":{"1min":17.707000000000001,"5min":23317.502,"15min":28621.294000000002},"last":11.621}]},{"osd":6,"last update":"Mon Jun 3 09:03:57 2024","interfaces":[{"interface":"back","average":{"1min":23.536000000000001,"5min":36.457000000000001,"15min":83.799999999999997},"min":{"1min":9.4800000000000004,"5min":9.4800000000000004,"15min":8.0630000000000006},"max":{"1min":89.442999999999998,"5min":508.77999999999997,"15min":9193.5169999999998},"last":157.79300000000001},{"interface":"front","average":{"1min":18.891999999999999,"5min":22.100999999999999,"15min":75.489999999999995},"min":{"1min":5.923,"5min":5.923,"15min":5.923},"max":{"1min":78.424999999999997,"5min":464.62,"15min":9191.4429999999993},"last":148.09}]},{"osd":7,"last update":"Mon Jun 3 09:06:00 2024","interfaces":[{"interface":"back","average":{"1min":13.257,"5min":20.917999999999999,"15min":21.088999999999999},"min":{"1min":10.475,"5min":6.7629999999999999,"15min":6.7629999999999999},"max":{"1min":17.245000000000001,"5min":211.22999999999999,"15min":211.22999999999999},"last":11.061999999999999},{"interface":"front","average":{"1min":12.872999999999999,"5min":20.587,"15min":24.021000000000001},"min":{"1min":8.5760000000000005,"5min":7.7709999999999999,"15min":6.8570000000000002},"max":{"1min":18.359999999999999,"5min":223.191,"15min":286.69499999999999},"last":10.021000000000001}]}]},{"osd":4,"up_from":20,"seq":85899346272,"num_pgs":12,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":63060,"kb_used_data":5704,"kb_used_omap":1,"kb_used_meta":57342,"kb_avail":94308780,"statfs":{"total":96636764160,"available":96572190720,"internally_reserved":0,"allocated":5840896,"data_stored":2740727,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1594,"internal_metadata":58718662},"hb_peers":[0,1,2,3,5,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Mon Jun 3 09:06:35 2024","interfaces":[{"interface":"back","average":{"1min":10.417,"5min":1093.547,"15min":381.64499999999998},"min":{"1min":4.6900000000000004,"5min":4.5279999999999996,"15min":4.3220000000000001},"max":{"1min":52.241,"5min":21726.743999999999,"15min":21726.743999999999},"last":9.0980000000000008},{"interface":"front","average":{"1min":13.175000000000001,"5min":1102.4200000000001,"15min":387.161},"min":{"1min":6.1950000000000003,"5min":5.8380000000000001,"15min":5.4909999999999997},"max":{"1min":53.148000000000003,"5min":21727.993999999999,"15min":21727.993999999999},"last":9.6370000000000005}]},{"osd":1,"last update":"Mon Jun 3 09:06:35 2024","interfaces":[{"interface":"back","average":{"1min":9.9909999999999997,"5min":53.250999999999998,"15min":340.02999999999997},"min":{"1min":4.5670000000000002,"5min":4.1970000000000001,"15min":4.1970000000000001},"max":{"1min":47.073,"5min":3346.4389999999999,"15min":22256.098000000002},"last":5.7030000000000003},{"interface":"front","average":{"1min":12.189,"5min":53.375999999999998,"15min":339.68799999999999},"min":{"1min":6.1539999999999999,"5min":5.0590000000000002,"15min":4.2640000000000002},"max":{"1min":49.558,"5min":3347.181,"15min":22247.317999999999},"last":13.794}]},{"osd":2,"last update":"Mon Jun 3 09:06:35 2024","interfaces":[{"interface":"back","average":{"1min":15.178000000000001,"5min":17.202000000000002,"15min":20.417999999999999},"min":{"1min":6.3010000000000002,"5min":6.3010000000000002,"15min":6.3010000000000002},"max":{"1min":85.340000000000003,"5min":337.60399999999998,"15min":393.58600000000001},"last":11.301},{"interface":"front","average":{"1min":15.423,"5min":14.432,"15min":22.754000000000001},"min":{"1min":6.4889999999999999,"5min":6.0629999999999997,"15min":6.0629999999999997},"max":{"1min":82.120000000000005,"5min":127.913,"15min":394.471},"last":14.632}]},{"osd":3,"last update":"Mon Jun 3 09:06:35 2024","interfaces":[{"interface":"back","average":{"1min":17.651,"5min":15.5,"15min":18.079000000000001},"min":{"1min":10.359999999999999,"5min":5.8719999999999999,"15min":5.4000000000000004},"max":{"1min":84.555999999999997,"5min":127.43600000000001,"15min":322.63},"last":15.337999999999999},{"interface":"front","average":{"1min":16.413,"5min":17.356999999999999,"15min":22.920000000000002},"min":{"1min":9.7349999999999994,"5min":7.484,"15min":7.484},"max":{"1min":71.394000000000005,"5min":204.517,"15min":392.67500000000001},"last":12.705}]},{"osd":5,"last update":"Mon Jun 3 09:06:35 2024","interfaces":[{"interface":"back","average":{"1min":15.672000000000001,"5min":16.742999999999999,"15min":20.795000000000002},"min":{"1min":8.093,"5min":6.7489999999999997,"15min":6.7489999999999997},"max":{"1min":51.326000000000001,"5min":268.10199999999998,"15min":344.23399999999998},"last":16.116},{"interface":"front","average":{"1min":17.297999999999998,"5min":19.356999999999999,"15min":25.713999999999999},"min":{"1min":12.875,"5min":8.5180000000000007,"15min":8.5180000000000007},"max":{"1min":40.829000000000001,"5min":264.75,"15min":345.18700000000001},"last":17.427}]},{"osd":6,"last update":"Mon Jun 3 09:04:32 2024","interfaces":[{"interface":"back","average":{"1min":15.548,"5min":32.957999999999998,"15min":152.738},"min":{"1min":9.141,"5min":9.141,"15min":8.2319999999999993},"max":{"1min":52.57,"5min":395.238,"15min":12496.335999999999},"last":208.90000000000001},{"interface":"front","average":{"1min":12.680999999999999,"5min":25.428000000000001,"15min":149.30600000000001},"min":{"1min":6.4379999999999997,"5min":5.6319999999999997,"15min":5.6319999999999997},"max":{"1min":46.283000000000001,"5min":377.74299999999999,"15min":12616.297},"last":205.88900000000001}]},{"osd":7,"last update":"Mon Jun 3 09:06:35 2024","interfaces":[{"interface":"back","average":{"1min":9.6969999999999992,"5min":14.196,"15min":18.809999999999999},"min":{"1min":6.7190000000000003,"5min":6.2110000000000003,"15min":5.8010000000000002},"max":{"1min":15.960000000000001,"5min":207.25299999999999,"15min":358.94900000000001},"last":10.683999999999999},{"interface":"front","average":{"1min":17.696000000000002,"5min":19.024999999999999,"15min":23.259},"min":{"1min":11.794,"5min":7.7789999999999999,"15min":7.7789999999999999},"max":{"1min":45.639000000000003,"5min":209.32900000000001,"15min":380.06900000000002},"last":16.852}]}]},{"osd":2,"up_from":20,"seq":85899346292,"num_pgs":11,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":58988,"kb_used_data":5728,"kb_used_omap":1,"kb_used_meta":53246,"kb_avail":94312852,"statfs":{"total":96636764160,"available":96576360448,"internally_reserved":0,"allocated":5865472,"data_stored":2751635,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1594,"internal_metadata":54524358},"hb_peers":[0,1,3,4,5,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":4,"apply_latency_ms":4,"commit_latency_ns":4000000,"apply_latency_ns":4000000},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Mon Jun 3 09:06:23 2024","interfaces":[{"interface":"back","average":{"1min":8.9629999999999992,"5min":946.25400000000002,"15min":326.36099999999999},"min":{"1min":5.3479999999999999,"5min":4.7050000000000001,"15min":4.1420000000000003},"max":{"1min":26.015000000000001,"5min":23465.18,"15min":23465.18},"last":8.7959999999999994},{"interface":"front","average":{"1min":7.6130000000000004,"5min":944.39800000000002,"15min":326.178},"min":{"1min":4.6790000000000003,"5min":4.6790000000000003,"15min":4.6790000000000003},"max":{"1min":20.802,"5min":23465.966,"15min":23465.966},"last":5.6059999999999999}]},{"osd":1,"last update":"Mon Jun 3 09:06:23 2024","interfaces":[{"interface":"back","average":{"1min":13.007,"5min":76.507000000000005,"15min":294.72899999999998},"min":{"1min":6.4089999999999998,"5min":5.4199999999999999,"15min":5.4199999999999999},"max":{"1min":85.430000000000007,"5min":4220.5290000000005,"15min":18954.987000000001},"last":11.789},{"interface":"front","average":{"1min":15.894,"5min":82.119,"15min":302.92399999999998},"min":{"1min":7.367,"5min":7.367,"15min":7.367},"max":{"1min":98.498000000000005,"5min":4325.9489999999996,"15min":19143.834999999999},"last":15.368}]},{"osd":3,"last update":"Mon Jun 3 09:06:23 2024","interfaces":[{"interface":"back","average":{"1min":9.7579999999999991,"5min":12.444000000000001,"15min":28.065000000000001},"min":{"1min":7,"5min":5.9290000000000003,"15min":5.2450000000000001},"max":{"1min":28.164000000000001,"5min":118.913,"15min":2594.8150000000001},"last":13.339},{"interface":"front","average":{"1min":12.321999999999999,"5min":15.25,"15min":31.349},"min":{"1min":9.1240000000000006,"5min":8.0760000000000005,"15min":6.0780000000000003},"max":{"1min":32.360999999999997,"5min":156.34700000000001,"15min":2706.5120000000002},"last":14.712}]},{"osd":4,"last update":"Mon Jun 3 09:05:57 2024","interfaces":[{"interface":"back","average":{"1min":10.573,"5min":1266.894,"15min":438.154},"min":{"1min":8.6379999999999999,"5min":7.8789999999999996,"15min":6.2039999999999997},"max":{"1min":15.105,"5min":26660.861000000001,"15min":26660.861000000001},"last":7.8780000000000001},{"interface":"front","average":{"1min":12.407999999999999,"5min":1266.7460000000001,"15min":435.75400000000002},"min":{"1min":10.266999999999999,"5min":10.266999999999999,"15min":6.2720000000000002},"max":{"1min":15.228,"5min":26616.243999999999,"15min":26616.243999999999},"last":10.970000000000001}]},{"osd":5,"last update":"Mon Jun 3 09:06:23 2024","interfaces":[{"interface":"back","average":{"1min":14.103,"5min":18.091000000000001,"15min":21.119},"min":{"1min":9.6349999999999998,"5min":9.2159999999999993,"15min":6.2050000000000001},"max":{"1min":32.890000000000001,"5min":252.31100000000001,"15min":252.31100000000001},"last":9.3249999999999993},{"interface":"front","average":{"1min":11.545999999999999,"5min":16.170999999999999,"15min":20.254000000000001},"min":{"1min":7.6070000000000002,"5min":7.4850000000000003,"15min":6.8810000000000002},"max":{"1min":29.228999999999999,"5min":258.31700000000001,"15min":258.31700000000001},"last":7.0940000000000003}]},{"osd":6,"last update":"Mon Jun 3 09:04:28 2024","interfaces":[{"interface":"back","average":{"1min":14.063000000000001,"5min":24.850000000000001,"15min":106.755},"min":{"1min":9.0820000000000007,"5min":8.9649999999999999,"15min":8.9649999999999999},"max":{"1min":24.666,"5min":188.06700000000001,"15min":9521.0010000000002},"last":183.60900000000001},{"interface":"front","average":{"1min":13.933999999999999,"5min":21.081,"15min":106.142},"min":{"1min":5.8719999999999999,"5min":5.8719999999999999,"15min":5.8719999999999999},"max":{"1min":117.551,"5min":193.887,"15min":9585.6620000000003},"last":184.20599999999999}]},{"osd":7,"last update":"Mon Jun 3 09:06:23 2024","interfaces":[{"interface":"back","average":{"1min":17.056000000000001,"5min":20.055,"15min":23.268999999999998},"min":{"1min":10.353999999999999,"5min":6.6130000000000004,"15min":6.6130000000000004},"max":{"1min":88.974999999999994,"5min":242.56800000000001,"15min":242.56800000000001},"last":12.818},{"interface":"front","average":{"1min":18.885000000000002,"5min":21.460000000000001,"15min":25.489999999999998},"min":{"1min":12.497,"5min":9.0660000000000007,"15min":9.0660000000000007},"max":{"1min":102.229,"5min":181.56399999999999,"15min":214.33799999999999},"last":12.301}]}]},{"osd":3,"up_from":14,"seq":60129542506,"num_pgs":11,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":58992,"kb_used_data":5728,"kb_used_omap":1,"kb_used_meta":53246,"kb_avail":94312848,"statfs":{"total":96636764160,"available":96576356352,"internally_reserved":0,"allocated":5865472,"data_stored":2751635,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1592,"internal_metadata":54524360},"hb_peers":[0,1,2,4,5,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":5,"apply_latency_ms":5,"commit_latency_ns":5000000,"apply_latency_ns":5000000},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Mon Jun 3 09:06:25 2024","interfaces":[{"interface":"back","average":{"1min":7.79,"5min":1060.1959999999999,"15min":367.01799999999997},"min":{"1min":5.6909999999999998,"5min":5.173,"15min":4.6929999999999996},"max":{"1min":10.59,"5min":23275.167000000001,"15min":23275.167000000001},"last":8.5389999999999997},{"interface":"front","average":{"1min":11.215999999999999,"5min":1066.5799999999999,"15min":372.476},"min":{"1min":8.7080000000000002,"5min":7.2510000000000003,"15min":5.9880000000000004},"max":{"1min":13.499000000000001,"5min":23278.536,"15min":23278.536},"last":13.473000000000001}]},{"osd":1,"last update":"Mon Jun 3 09:06:25 2024","interfaces":[{"interface":"back","average":{"1min":7.3760000000000003,"5min":15.614000000000001,"15min":389.83100000000002},"min":{"1min":4.335,"5min":4.335,"15min":4.335},"max":{"1min":11.289,"5min":250.72300000000001,"15min":21658.205999999998},"last":5.8440000000000003},{"interface":"front","average":{"1min":8.9499999999999993,"5min":14.135,"15min":381.10300000000001},"min":{"1min":6.5609999999999999,"5min":5.4100000000000001,"15min":5.2779999999999996},"max":{"1min":14.51,"5min":180.44499999999999,"15min":21307.069},"last":7.8959999999999999}]},{"osd":2,"last update":"Mon Jun 3 09:06:25 2024","interfaces":[{"interface":"back","average":{"1min":12.343999999999999,"5min":22.059999999999999,"15min":23.715},"min":{"1min":7.9199999999999999,"5min":6.4900000000000002,"15min":6.056},"max":{"1min":15.601000000000001,"5min":377.803,"15min":377.803},"last":10.523},{"interface":"front","average":{"1min":10.319000000000001,"5min":19.364999999999998,"15min":20.242999999999999},"min":{"1min":6.891,"5min":6.1749999999999998,"15min":5.9020000000000001},"max":{"1min":15.432,"5min":371.601,"15min":371.601},"last":11.694000000000001}]},{"osd":4,"last update":"Mon Jun 3 09:06:31 2024","interfaces":[{"interface":"back","average":{"1min":9.7460000000000004,"5min":891.23000000000002,"15min":416.48099999999999},"min":{"1min":5.6539999999999999,"5min":5.6539999999999999,"15min":5.6539999999999999},"max":{"1min":15.445,"5min":29040.635999999999,"15min":30144.939999999999},"last":6.827},{"interface":"front","average":{"1min":12.398999999999999,"5min":897.58500000000004,"15min":421.27499999999998},"min":{"1min":8.9469999999999992,"5min":7.8109999999999999,"15min":6.2439999999999998},"max":{"1min":16.518000000000001,"5min":29076.955999999998,"15min":30180.689999999999},"last":9.0559999999999992}]},{"osd":5,"last update":"Mon Jun 3 09:06:07 2024","interfaces":[{"interface":"back","average":{"1min":12.802,"5min":19.640999999999998,"15min":22.318000000000001},"min":{"1min":9.0890000000000004,"5min":6.8810000000000002,"15min":6.7690000000000001},"max":{"1min":16.745999999999999,"5min":249.88,"15min":327.34899999999999},"last":11.055},{"interface":"front","average":{"1min":15.002000000000001,"5min":24.003,"15min":26.649999999999999},"min":{"1min":11.984,"5min":9.9149999999999991,"15min":8.9879999999999995},"max":{"1min":18.407,"5min":251.47800000000001,"15min":331.86799999999999},"last":15.125}]},{"osd":6,"last update":"Mon Jun 3 09:04:04 2024","interfaces":[{"interface":"back","average":{"1min":22.815999999999999,"5min":31.513999999999999,"15min":181.74299999999999},"min":{"1min":9.4809999999999999,"5min":7.2240000000000002,"15min":7.2240000000000002},"max":{"1min":160.56999999999999,"5min":380.613,"15min":11893.74},"last":19.359999999999999},{"interface":"front","average":{"1min":25.359999999999999,"5min":30.981000000000002,"15min":179.81399999999999},"min":{"1min":9.8870000000000005,"5min":7.3360000000000003,"15min":6.0899999999999999},"max":{"1min":193.11099999999999,"5min":379.80700000000002,"15min":11893.069},"last":8.202}]},{"osd":7,"last update":"Mon Jun 3 09:06:25 2024","interfaces":[{"interface":"back","average":{"1min":11.321999999999999,"5min":18.443999999999999,"15min":21.568999999999999},"min":{"1min":8.109,"5min":8.109,"15min":7.4279999999999999},"max":{"1min":17.113,"5min":146.267,"15min":178.76300000000001},"last":12.93},{"interface":"front","average":{"1min":12.512,"5min":20.065999999999999,"15min":24.190000000000001},"min":{"1min":7.4989999999999997,"5min":7.4989999999999997,"15min":7.4989999999999997},"max":{"1min":18.047999999999998,"5min":148.13800000000001,"15min":417.22800000000001},"last":14.553000000000001}]}]},{"osd":1,"up_from":14,"seq":60129542504,"num_pgs":6,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":59384,"kb_used_data":6120,"kb_used_omap":1,"kb_used_meta":53246,"kb_avail":94312456,"statfs":{"total":96636764160,"available":96575954944,"internally_reserved":0,"allocated":6266880,"data_stored":3149475,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":54524357},"hb_peers":[0,2,3,4,5,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":5,"apply_latency_ms":5,"commit_latency_ns":5000000,"apply_latency_ns":5000000},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Mon Jun 3 09:06:07 2024","interfaces":[{"interface":"back","average":{"1min":7.7530000000000001,"5min":185.66999999999999,"15min":77.347999999999999},"min":{"1min":5.0229999999999997,"5min":4.9279999999999999,"15min":4.5449999999999999},"max":{"1min":11.896000000000001,"5min":7470.616,"15min":7470.616},"last":7.4450000000000003},{"interface":"front","average":{"1min":9.9220000000000006,"5min":187.14599999999999,"15min":76.325999999999993},"min":{"1min":5.3159999999999998,"5min":5.3159999999999998,"15min":5.3159999999999998},"max":{"1min":18.338000000000001,"5min":7477.0829999999996,"15min":7477.0829999999996},"last":9.8000000000000007}]},{"osd":2,"last update":"Mon Jun 3 09:06:20 2024","interfaces":[{"interface":"back","average":{"1min":10.340999999999999,"5min":17.369,"15min":23.670999999999999},"min":{"1min":6.9740000000000002,"5min":5.8650000000000002,"15min":5.8650000000000002},"max":{"1min":15.452,"5min":228,"15min":657.38},"last":10.438000000000001},{"interface":"front","average":{"1min":13.387,"5min":23.355,"15min":31.524000000000001},"min":{"1min":9.077,"5min":9.077,"15min":7.2160000000000002},"max":{"1min":19.701000000000001,"5min":229.465,"15min":700.06500000000005},"last":10.981999999999999}]},{"osd":3,"last update":"Mon Jun 3 09:06:04 2024","interfaces":[{"interface":"back","average":{"1min":8.6110000000000007,"5min":15.733000000000001,"15min":26.646999999999998},"min":{"1min":5.827,"5min":5.4640000000000004,"15min":5.4640000000000004},"max":{"1min":13.105,"5min":121.619,"15min":1676.009},"last":6.7359999999999998},{"interface":"front","average":{"1min":9.5850000000000009,"5min":16.161999999999999,"15min":29.065999999999999},"min":{"1min":6.3630000000000004,"5min":5.7699999999999996,"15min":5.7699999999999996},"max":{"1min":15.166,"5min":132.036,"15min":1766.5840000000001},"last":9.4640000000000004}]},{"osd":4,"last update":"Mon Jun 3 09:06:35 2024","interfaces":[{"interface":"back","average":{"1min":8.6400000000000006,"5min":641.85500000000002,"15min":334.71499999999997},"min":{"1min":6.7640000000000002,"5min":6.1760000000000002,"15min":6.0970000000000004},"max":{"1min":11.811,"5min":26340.968000000001,"15min":28044.862000000001},"last":9.2050000000000001},{"interface":"front","average":{"1min":12.361000000000001,"5min":648.59900000000005,"15min":338.48500000000001},"min":{"1min":9.8949999999999996,"5min":9.3940000000000001,"15min":6.4640000000000004},"max":{"1min":15.507,"5min":26342.311000000002,"15min":28045.624},"last":14.834}]},{"osd":5,"last update":"Mon Jun 3 09:06:04 2024","interfaces":[{"interface":"back","average":{"1min":12.478999999999999,"5min":20.228000000000002,"15min":25.997},"min":{"1min":8.6430000000000007,"5min":7.9720000000000004,"15min":7.234},"max":{"1min":17.442,"5min":139.43199999999999,"15min":701.22000000000003},"last":15.553000000000001},{"interface":"front","average":{"1min":13.507999999999999,"5min":25.760999999999999,"15min":27.553999999999998},"min":{"1min":10.042999999999999,"5min":7.258,"15min":6.242},"max":{"1min":20.611000000000001,"5min":278.63,"15min":598.35000000000002},"last":12.192}]},{"osd":6,"last update":"Mon Jun 3 09:03:54 2024","interfaces":[{"interface":"back","average":{"1min":28.524000000000001,"5min":20.654,"15min":23.878},"min":{"1min":6.0309999999999997,"5min":6.0309999999999997,"15min":6.0309999999999997},"max":{"1min":167.221,"5min":167.221,"15min":704.245},"last":18.721},{"interface":"front","average":{"1min":27.606000000000002,"5min":30.131,"15min":28.628},"min":{"1min":9.8650000000000002,"5min":9.7200000000000006,"15min":8.3580000000000005},"max":{"1min":123.938,"5min":358.23000000000002,"15min":634.05499999999995},"last":173.113}]},{"osd":7,"last update":"Mon Jun 3 09:06:20 2024","interfaces":[{"interface":"back","average":{"1min":14.66,"5min":23.234999999999999,"15min":31.097000000000001},"min":{"1min":11.250999999999999,"5min":8.0559999999999992,"15min":8.0559999999999992},"max":{"1min":21.507999999999999,"5min":270.95100000000002,"15min":703.07799999999997},"last":11.664999999999999},{"interface":"front","average":{"1min":15.548,"5min":28.760000000000002,"15min":28.925000000000001},"min":{"1min":12.759,"5min":8.2089999999999996,"15min":7.7599999999999998},"max":{"1min":22.408000000000001,"5min":360.99900000000002,"15min":608.44100000000003},"last":13.053000000000001}]}]},{"osd":0,"up_from":14,"seq":60129542506,"num_pgs":10,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":81108,"kb_used_data":5704,"kb_used_omap":1,"kb_used_meta":75390,"kb_avail":94290732,"statfs":{"total":96636764160,"available":96553709568,"internally_reserved":0,"allocated":5840896,"data_stored":2740727,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1593,"internal_metadata":77199815},"hb_peers":[1,2,3,4,5,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[{"osd":1,"last update":"Mon Jun 3 09:06:04 2024","interfaces":[{"interface":"back","average":{"1min":6.9960000000000004,"5min":56.997,"15min":339.29700000000003},"min":{"1min":4.4530000000000003,"5min":4.4530000000000003,"15min":4.4100000000000001},"max":{"1min":11.720000000000001,"5min":2414.4749999999999,"15min":20691.453000000001},"last":5.6639999999999997},{"interface":"front","average":{"1min":9.548,"5min":57.258000000000003,"15min":337.791},"min":{"1min":5.2300000000000004,"5min":5.2300000000000004,"15min":4.6050000000000004},"max":{"1min":15.337999999999999,"5min":2412.9549999999999,"15min":20640.792000000001},"last":8.5690000000000008}]},{"osd":2,"last update":"Mon Jun 3 09:06:22 2024","interfaces":[{"interface":"back","average":{"1min":11.154,"5min":26.515999999999998,"15min":22.882000000000001},"min":{"1min":5.7869999999999999,"5min":5.766,"15min":5.766},"max":{"1min":72.269999999999996,"5min":627.40700000000004,"15min":627.40700000000004},"last":6.6139999999999999},{"interface":"front","average":{"1min":14.353999999999999,"5min":33.304000000000002,"15min":29.885000000000002},"min":{"1min":8.2149999999999999,"5min":7.8630000000000004,"15min":7.4089999999999998},"max":{"1min":73.418000000000006,"5min":635.62599999999998,"15min":635.62599999999998},"last":9.8049999999999997}]},{"osd":3,"last update":"Mon Jun 3 09:06:04 2024","interfaces":[{"interface":"back","average":{"1min":12.105,"5min":26.925000000000001,"15min":26.584},"min":{"1min":7.8410000000000002,"5min":7.1280000000000001,"15min":6.1909999999999998},"max":{"1min":18.425999999999998,"5min":389.72699999999998,"15min":389.72699999999998},"last":9.1590000000000007},{"interface":"front","average":{"1min":13.794,"5min":29.771999999999998,"15min":28.815000000000001},"min":{"1min":10.084,"5min":9.4920000000000009,"15min":6.1849999999999996},"max":{"1min":17.373999999999999,"5min":390.23200000000003,"15min":494.92899999999997},"last":11.637}]},{"osd":4,"last update":"Mon Jun 3 09:06:27 2024","interfaces":[{"interface":"back","average":{"1min":11.352,"5min":745.30600000000004,"15min":363.48099999999999},"min":{"1min":6.415,"5min":6.415,"15min":6.415},"max":{"1min":16.724,"5min":20248.616000000002,"15min":25555.212},"last":7.2590000000000003},{"interface":"front","average":{"1min":13.545,"5min":744.76999999999998,"15min":361.03100000000001},"min":{"1min":9.3889999999999993,"5min":8.8940000000000001,"15min":7.1429999999999998},"max":{"1min":18.411000000000001,"5min":20250.195,"15min":25556.159},"last":10.904}]},{"osd":5,"last update":"Mon Jun 3 09:06:17 2024","interfaces":[{"interface":"back","average":{"1min":9.7040000000000006,"5min":26.533999999999999,"15min":28.683},"min":{"1min":6.8129999999999997,"5min":6.8129999999999997,"15min":6.8129999999999997},"max":{"1min":14.942,"5min":455.78699999999998,"15min":605.05899999999997},"last":14.183999999999999},{"interface":"front","average":{"1min":14.279,"5min":28.398,"15min":26.192},"min":{"1min":11.696,"5min":6.4420000000000002,"15min":6.2030000000000003},"max":{"1min":17.879000000000001,"5min":391.31200000000001,"15min":541.11000000000001},"last":12.648999999999999}]},{"osd":6,"last update":"Mon Jun 3 09:04:30 2024","interfaces":[{"interface":"back","average":{"1min":52.496000000000002,"5min":33.139000000000003,"15min":160.83199999999999},"min":{"1min":7.5289999999999999,"5min":7.1040000000000001,"15min":6.3739999999999997},"max":{"1min":457.91699999999997,"5min":457.91699999999997,"15min":11115.258},"last":18.959},{"interface":"front","average":{"1min":49.670000000000002,"5min":33.704999999999998,"15min":162.791},"min":{"1min":12.247999999999999,"5min":11.019,"15min":9.9760000000000009},"max":{"1min":407.05099999999999,"5min":407.05099999999999,"15min":11092.194},"last":17.707000000000001}]},{"osd":7,"last update":"Mon Jun 3 09:06:22 2024","interfaces":[{"interface":"back","average":{"1min":11.625999999999999,"5min":26.375,"15min":28.972999999999999},"min":{"1min":6.4480000000000004,"5min":6.4480000000000004,"15min":6.4480000000000004},"max":{"1min":15.352,"5min":461.41699999999997,"15min":598.33699999999999},"last":13.51},{"interface":"front","average":{"1min":14.013,"5min":29.696000000000002,"15min":33.630000000000003},"min":{"1min":10.938000000000001,"5min":10.58,"15min":10.263999999999999},"max":{"1min":16.603000000000002,"5min":463.10599999999999,"15min":592.80499999999995},"last":14.896000000000001}]}]}],"pool_statfs":[{"poolid":1,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":401408,"data_stored":397840,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":5,"total":0,"available":0,"internally_reserved":0,"allocated":401408,"data_stored":397840,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":4,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":5,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":6,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":17,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":17,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":17,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":17,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":17,"osd":4,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":17,"osd":5,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":5,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":17,"osd":6,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":5,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":17,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0}]}} 2024-06-03T09:06:48.876 INFO:tasks.ceph.ceph_manager.ceph:PG 17.c is not active+clean 2024-06-03T09:06:48.876 INFO:tasks.ceph.ceph_manager.ceph:{'pgid': '17.c', 'version': "0'0", 'reported_seq': 955, 'reported_epoch': 1011, 'state': 'active+undersized', 'last_fresh': '2024-06-03T09:06:42.798184+0000', 'last_change': '2024-06-03T09:06:03.440074+0000', 'last_active': '2024-06-03T09:06:42.798184+0000', 'last_peered': '2024-06-03T09:06:42.798184+0000', 'last_clean': '2024-06-03T09:05:02.105638+0000', 'last_became_active': '2024-06-03T09:06:03.440074+0000', 'last_became_peered': '2024-06-03T09:06:03.440074+0000', 'last_unstale': '2024-06-03T09:06:42.798184+0000', 'last_undegraded': '2024-06-03T09:06:42.798184+0000', 'last_fullsized': '2024-06-03T09:06:03.377982+0000', 'mapping_epoch': 999, 'log_start': "0'0", 'ondisk_log_start': "0'0", 'created': 28, 'last_epoch_clean': 255, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2024-06-03T08:37:55.614497+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2024-06-03T08:37:55.614497+0000', 'last_clean_scrub_stamp': '2024-06-03T08:37:55.614497+0000', 'objects_scrubbed': 0, 'log_size': 0, 'log_dups_size': 0, 'ondisk_log_size': 0, 'stats_invalid': False, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': 'periodic scrub scheduled @ 2024-06-04T10:34:20.808557+0000', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 0, 'num_objects': 0, 'num_object_clones': 0, 'num_object_copies': 0, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 0, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 0, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 0, 'num_write_kb': 0, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 0, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [3], 'acting': [3], 'avail_no_missing': ['3'], 'object_location_counts': [], 'blocked_by': [], 'up_primary': 3, 'acting_primary': 3, 'purged_snaps': []} 2024-06-03T09:06:48.876 INFO:tasks.ceph.ceph_manager.ceph:PG 17.8 is not active+clean 2024-06-03T09:06:48.876 INFO:tasks.ceph.ceph_manager.ceph:{'pgid': '17.8', 'version': "0'0", 'reported_seq': 961, 'reported_epoch': 1009, 'state': 'active+undersized', 'last_fresh': '2024-06-03T09:06:14.430375+0000', 'last_change': '2024-06-03T09:06:03.388786+0000', 'last_active': '2024-06-03T09:06:14.430375+0000', 'last_peered': '2024-06-03T09:06:14.430375+0000', 'last_clean': '2024-06-03T09:04:43.297780+0000', 'last_became_active': '2024-06-03T09:06:03.388786+0000', 'last_became_peered': '2024-06-03T09:06:03.388786+0000', 'last_unstale': '2024-06-03T09:06:14.430375+0000', 'last_undegraded': '2024-06-03T09:06:14.430375+0000', 'last_fullsized': '2024-06-03T09:06:03.331754+0000', 'mapping_epoch': 999, 'log_start': "0'0", 'ondisk_log_start': "0'0", 'created': 28, 'last_epoch_clean': 255, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2024-06-03T08:37:55.614497+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2024-06-03T08:37:55.614497+0000', 'last_clean_scrub_stamp': '2024-06-03T08:37:55.614497+0000', 'objects_scrubbed': 0, 'log_size': 0, 'log_dups_size': 0, 'ondisk_log_size': 0, 'stats_invalid': False, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': 'periodic scrub scheduled @ 2024-06-04T09:18:44.765547+0000', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 0, 'num_objects': 0, 'num_object_clones': 0, 'num_object_copies': 0, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 0, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 0, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 0, 'num_write_kb': 0, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 0, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [0], 'acting': [0], 'avail_no_missing': ['0'], 'object_location_counts': [], 'blocked_by': [], 'up_primary': 0, 'acting_primary': 0, 'purged_snaps': []} 2024-06-03T09:06:48.876 INFO:tasks.ceph.ceph_manager.ceph:PG 17.6 is not active+clean 2024-06-03T09:06:48.877 INFO:tasks.ceph.ceph_manager.ceph:{'pgid': '17.6', 'version': "0'0", 'reported_seq': 960, 'reported_epoch': 1009, 'state': 'active+undersized', 'last_fresh': '2024-06-03T09:06:14.453110+0000', 'last_change': '2024-06-03T09:06:03.428677+0000', 'last_active': '2024-06-03T09:06:14.453110+0000', 'last_peered': '2024-06-03T09:06:14.453110+0000', 'last_clean': '2024-06-03T09:04:43.312352+0000', 'last_became_active': '2024-06-03T09:06:03.428677+0000', 'last_became_peered': '2024-06-03T09:06:03.428677+0000', 'last_unstale': '2024-06-03T09:06:14.453110+0000', 'last_undegraded': '2024-06-03T09:06:14.453110+0000', 'last_fullsized': '2024-06-03T09:06:03.359909+0000', 'mapping_epoch': 999, 'log_start': "0'0", 'ondisk_log_start': "0'0", 'created': 28, 'last_epoch_clean': 255, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2024-06-03T08:37:55.614497+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2024-06-03T08:37:55.614497+0000', 'last_clean_scrub_stamp': '2024-06-03T08:37:55.614497+0000', 'objects_scrubbed': 0, 'log_size': 0, 'log_dups_size': 0, 'ondisk_log_size': 0, 'stats_invalid': False, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': 'periodic scrub scheduled @ 2024-06-04T15:38:02.180241+0000', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 0, 'num_objects': 0, 'num_object_clones': 0, 'num_object_copies': 0, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 0, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 0, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 0, 'num_write_kb': 0, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 0, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [0], 'acting': [0], 'avail_no_missing': ['0'], 'object_location_counts': [], 'blocked_by': [], 'up_primary': 0, 'acting_primary': 0, 'purged_snaps': []} 2024-06-03T09:06:48.877 INFO:tasks.ceph.ceph_manager.ceph:PG 17.16 is not active+clean 2024-06-03T09:06:48.877 INFO:tasks.ceph.ceph_manager.ceph:{'pgid': '17.16', 'version': "0'0", 'reported_seq': 974, 'reported_epoch': 1011, 'state': 'active+undersized', 'last_fresh': '2024-06-03T09:06:42.992868+0000', 'last_change': '2024-06-03T09:06:03.478615+0000', 'last_active': '2024-06-03T09:06:42.992868+0000', 'last_peered': '2024-06-03T09:06:42.992868+0000', 'last_clean': '2024-06-03T09:04:43.328415+0000', 'last_became_active': '2024-06-03T09:06:03.478615+0000', 'last_became_peered': '2024-06-03T09:06:03.478615+0000', 'last_unstale': '2024-06-03T09:06:42.992868+0000', 'last_undegraded': '2024-06-03T09:06:42.992868+0000', 'last_fullsized': '2024-06-03T09:06:03.402504+0000', 'mapping_epoch': 999, 'log_start': "0'0", 'ondisk_log_start': "0'0", 'created': 28, 'last_epoch_clean': 255, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2024-06-03T08:37:55.614497+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2024-06-03T08:37:55.614497+0000', 'last_clean_scrub_stamp': '2024-06-03T08:37:55.614497+0000', 'objects_scrubbed': 0, 'log_size': 0, 'log_dups_size': 0, 'ondisk_log_size': 0, 'stats_invalid': False, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': 'periodic scrub scheduled @ 2024-06-04T08:47:08.079233+0000', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 0, 'num_objects': 0, 'num_object_clones': 0, 'num_object_copies': 0, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 0, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 0, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 0, 'num_write_kb': 0, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 0, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [2], 'acting': [2], 'avail_no_missing': ['2'], 'object_location_counts': [], 'blocked_by': [], 'up_primary': 2, 'acting_primary': 2, 'purged_snaps': []} 2024-06-03T09:06:48.877 INFO:tasks.ceph.ceph_manager.ceph:PG 2.4 is not active+clean 2024-06-03T09:06:48.877 INFO:tasks.ceph.ceph_manager.ceph:{'pgid': '2.4', 'version': "0'0", 'reported_seq': 952, 'reported_epoch': 1010, 'state': 'active+undersized', 'last_fresh': '2024-06-03T09:06:18.331485+0000', 'last_change': '2024-06-03T09:06:03.387171+0000', 'last_active': '2024-06-03T09:06:18.331485+0000', 'last_peered': '2024-06-03T09:06:18.331485+0000', 'last_clean': '2024-06-03T09:04:43.202645+0000', 'last_became_active': '2024-06-03T09:06:03.387171+0000', 'last_became_peered': '2024-06-03T09:06:03.387171+0000', 'last_unstale': '2024-06-03T09:06:18.331485+0000', 'last_undegraded': '2024-06-03T09:06:18.331485+0000', 'last_fullsized': '2024-06-03T09:06:03.283787+0000', 'mapping_epoch': 999, 'log_start': "0'0", 'ondisk_log_start': "0'0", 'created': 23, 'last_epoch_clean': 255, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2024-06-03T08:36:28.444764+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2024-06-03T08:36:28.444764+0000', 'last_clean_scrub_stamp': '2024-06-03T08:36:28.444764+0000', 'objects_scrubbed': 0, 'log_size': 0, 'log_dups_size': 0, 'ondisk_log_size': 0, 'stats_invalid': False, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': 'periodic scrub scheduled @ 2024-06-04T20:17:43.485023+0000', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 0, 'num_objects': 0, 'num_object_clones': 0, 'num_object_copies': 0, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 0, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 0, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 0, 'num_write_kb': 0, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 0, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [5], 'acting': [5], 'avail_no_missing': ['5'], 'object_location_counts': [], 'blocked_by': [], 'up_primary': 5, 'acting_primary': 5, 'purged_snaps': []} 2024-06-03T09:06:48.877 INFO:tasks.ceph.ceph_manager.ceph:PG 2.2 is not active+clean 2024-06-03T09:06:48.877 INFO:tasks.ceph.ceph_manager.ceph:{'pgid': '2.2', 'version': "0'0", 'reported_seq': 965, 'reported_epoch': 1011, 'state': 'active+undersized', 'last_fresh': '2024-06-03T09:06:42.803839+0000', 'last_change': '2024-06-03T09:06:03.410020+0000', 'last_active': '2024-06-03T09:06:42.803839+0000', 'last_peered': '2024-06-03T09:06:42.803839+0000', 'last_clean': '2024-06-03T09:05:37.145889+0000', 'last_became_active': '2024-06-03T09:06:03.410020+0000', 'last_became_peered': '2024-06-03T09:06:03.410020+0000', 'last_unstale': '2024-06-03T09:06:42.803839+0000', 'last_undegraded': '2024-06-03T09:06:42.803839+0000', 'last_fullsized': '2024-06-03T09:06:03.346864+0000', 'mapping_epoch': 999, 'log_start': "0'0", 'ondisk_log_start': "0'0", 'created': 23, 'last_epoch_clean': 255, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2024-06-03T08:36:28.444764+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2024-06-03T08:36:28.444764+0000', 'last_clean_scrub_stamp': '2024-06-03T08:36:28.444764+0000', 'objects_scrubbed': 0, 'log_size': 0, 'log_dups_size': 0, 'ondisk_log_size': 0, 'stats_invalid': False, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': 'periodic scrub scheduled @ 2024-06-04T09:57:12.423360+0000', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 0, 'num_objects': 0, 'num_object_clones': 0, 'num_object_copies': 0, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 0, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 0, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 0, 'num_write_kb': 0, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 0, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [3], 'acting': [3], 'avail_no_missing': ['3'], 'object_location_counts': [], 'blocked_by': [], 'up_primary': 3, 'acting_primary': 3, 'purged_snaps': []} 2024-06-03T09:06:48.877 INFO:tasks.ceph.ceph_manager.ceph:PG 17.1a is not active+clean 2024-06-03T09:06:48.877 INFO:tasks.ceph.ceph_manager.ceph:{'pgid': '17.1a', 'version': "0'0", 'reported_seq': 960, 'reported_epoch': 1009, 'state': 'active+undersized', 'last_fresh': '2024-06-03T09:06:14.431224+0000', 'last_change': '2024-06-03T09:06:03.390734+0000', 'last_active': '2024-06-03T09:06:14.431224+0000', 'last_peered': '2024-06-03T09:06:14.431224+0000', 'last_clean': '2024-06-03T09:05:20.322313+0000', 'last_became_active': '2024-06-03T09:06:03.390734+0000', 'last_became_peered': '2024-06-03T09:06:03.390734+0000', 'last_unstale': '2024-06-03T09:06:14.431224+0000', 'last_undegraded': '2024-06-03T09:06:14.431224+0000', 'last_fullsized': '2024-06-03T09:06:03.343197+0000', 'mapping_epoch': 999, 'log_start': "0'0", 'ondisk_log_start': "0'0", 'created': 28, 'last_epoch_clean': 254, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2024-06-03T08:37:55.614497+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2024-06-03T08:37:55.614497+0000', 'last_clean_scrub_stamp': '2024-06-03T08:37:55.614497+0000', 'objects_scrubbed': 0, 'log_size': 0, 'log_dups_size': 0, 'ondisk_log_size': 0, 'stats_invalid': False, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': 'periodic scrub scheduled @ 2024-06-04T15:56:47.628875+0000', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 0, 'num_objects': 0, 'num_object_clones': 0, 'num_object_copies': 0, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 0, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 0, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 0, 'num_write_kb': 0, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 0, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [4], 'acting': [4], 'avail_no_missing': ['4'], 'object_location_counts': [], 'blocked_by': [], 'up_primary': 4, 'acting_primary': 4, 'purged_snaps': []} 2024-06-03T09:06:48.878 INFO:tasks.ceph.ceph_manager.ceph:PG 17.1c is not active+clean 2024-06-03T09:06:48.878 INFO:tasks.ceph.ceph_manager.ceph:{'pgid': '17.1c', 'version': "0'0", 'reported_seq': 944, 'reported_epoch': 1010, 'state': 'active+undersized', 'last_fresh': '2024-06-03T09:06:18.327492+0000', 'last_change': '2024-06-03T09:06:03.382210+0000', 'last_active': '2024-06-03T09:06:18.327492+0000', 'last_peered': '2024-06-03T09:06:18.327492+0000', 'last_clean': '2024-06-03T09:04:43.199660+0000', 'last_became_active': '2024-06-03T09:06:03.382210+0000', 'last_became_peered': '2024-06-03T09:06:03.382210+0000', 'last_unstale': '2024-06-03T09:06:18.327492+0000', 'last_undegraded': '2024-06-03T09:06:18.327492+0000', 'last_fullsized': '2024-06-03T09:06:03.277874+0000', 'mapping_epoch': 999, 'log_start': "0'0", 'ondisk_log_start': "0'0", 'created': 28, 'last_epoch_clean': 255, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2024-06-03T08:37:55.614497+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2024-06-03T08:37:55.614497+0000', 'last_clean_scrub_stamp': '2024-06-03T08:37:55.614497+0000', 'objects_scrubbed': 0, 'log_size': 0, 'log_dups_size': 0, 'ondisk_log_size': 0, 'stats_invalid': False, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': 'periodic scrub scheduled @ 2024-06-04T12:18:50.013884+0000', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 0, 'num_objects': 0, 'num_object_clones': 0, 'num_object_copies': 0, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 0, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 0, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 0, 'num_write_kb': 0, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 0, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [5], 'acting': [5], 'avail_no_missing': ['5'], 'object_location_counts': [], 'blocked_by': [], 'up_primary': 5, 'acting_primary': 5, 'purged_snaps': []} 2024-06-03T09:06:48.878 INFO:tasks.ceph.ceph_manager.ceph:PG 17.1 is not active+clean 2024-06-03T09:06:48.878 INFO:tasks.ceph.ceph_manager.ceph:{'pgid': '17.1', 'version': "65'7", 'reported_seq': 959, 'reported_epoch': 1009, 'state': 'active+undersized+degraded', 'last_fresh': '2024-06-03T09:06:13.709209+0000', 'last_change': '2024-06-03T09:06:03.370660+0000', 'last_active': '2024-06-03T09:06:13.709209+0000', 'last_peered': '2024-06-03T09:06:13.709209+0000', 'last_clean': '2024-06-03T09:05:03.733864+0000', 'last_became_active': '2024-06-03T09:06:03.370660+0000', 'last_became_peered': '2024-06-03T09:06:03.370660+0000', 'last_unstale': '2024-06-03T09:06:13.709209+0000', 'last_undegraded': '2024-06-03T09:06:03.266076+0000', 'last_fullsized': '2024-06-03T09:06:03.264525+0000', 'mapping_epoch': 999, 'log_start': "0'0", 'ondisk_log_start': "0'0", 'created': 28, 'last_epoch_clean': 255, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2024-06-03T08:37:55.614497+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2024-06-03T08:37:55.614497+0000', 'last_clean_scrub_stamp': '2024-06-03T08:37:55.614497+0000', 'objects_scrubbed': 0, 'log_size': 7, 'log_dups_size': 0, 'ondisk_log_size': 7, 'stats_invalid': False, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': 'periodic scrub scheduled @ 2024-06-04T11:55:18.194283+0000', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 5, 'num_objects': 1, 'num_object_clones': 0, 'num_object_copies': 2, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 1, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 1, 'num_whiteouts': 0, 'num_read': 6, 'num_read_kb': 6, 'num_write': 7, 'num_write_kb': 7, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 0, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [5], 'acting': [5], 'avail_no_missing': ['5'], 'object_location_counts': [{'shards': '5', 'objects': 1}], 'blocked_by': [], 'up_primary': 5, 'acting_primary': 5, 'purged_snaps': []} 2024-06-03T09:06:48.878 DEBUG:teuthology.orchestra.run.smithi088:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph status --format=json 2024-06-03T09:06:48.976 INFO:teuthology.orchestra.run.smithi088.stderr:2024-06-03T09:06:48.976+0000 7fa536711640 10 monclient: get_monmap_and_config 2024-06-03T09:06:48.976 INFO:teuthology.orchestra.run.smithi088.stderr:2024-06-03T09:06:48.976+0000 7fa536711640 10 monclient: build_initial_monmap 2024-06-03T09:06:48.976 INFO:teuthology.orchestra.run.smithi088.stderr:2024-06-03T09:06:48.976+0000 7fa536711640 10 monclient: monmap: 2024-06-03T09:06:48.976 INFO:teuthology.orchestra.run.smithi088.stderr:epoch 0 2024-06-03T09:06:48.976 INFO:teuthology.orchestra.run.smithi088.stderr:fsid ec3c3fdd-3ac6-4d7d-9412-683130ed8b04 2024-06-03T09:06:48.977 INFO:teuthology.orchestra.run.smithi088.stderr:last_changed 2024-06-03T09:06:48.977332+0000 2024-06-03T09:06:48.977 INFO:teuthology.orchestra.run.smithi088.stderr:created 2024-06-03T09:06:48.977332+0000 2024-06-03T09:06:48.977 INFO:teuthology.orchestra.run.smithi088.stderr:min_mon_release 0 (unknown) 2024-06-03T09:06:48.977 INFO:teuthology.orchestra.run.smithi088.stderr:election_strategy: 3 2024-06-03T09:06:48.977 INFO:teuthology.orchestra.run.smithi088.stderr:0: v1:172.21.15.88:6789/0 mon.noname-a 2024-06-03T09:06:48.977 INFO:teuthology.orchestra.run.smithi088.stderr:1: v1:172.21.15.192:6789/0 mon.noname-c 2024-06-03T09:06:48.977 INFO:teuthology.orchestra.run.smithi088.stderr:2: v1:172.21.15.88:6790/0 mon.noname-b 2024-06-03T09:06:48.977 INFO:teuthology.orchestra.run.smithi088.stderr: 2024-06-03T09:06:48.978 INFO:teuthology.orchestra.run.smithi088.stderr:2024-06-03T09:06:48.977+0000 7fa536711640 1 Processor -- start 2024-06-03T09:06:48.979 INFO:teuthology.orchestra.run.smithi088.stderr:2024-06-03T09:06:48.977+0000 7fa536711640 1 -- start start 2024-06-03T09:06:48.979 INFO:teuthology.orchestra.run.smithi088.stderr:2024-06-03T09:06:48.977+0000 7fa536711640 10 monclient: init 2024-06-03T09:06:48.979 INFO:teuthology.orchestra.run.smithi088.stderr:2024-06-03T09:06:48.977+0000 7fa536711640 10 monclient: _reopen_session rank -1 2024-06-03T09:06:48.979 INFO:teuthology.orchestra.run.smithi088.stderr:2024-06-03T09:06:48.977+0000 7fa536711640 10 monclient: _add_conns ranks=[2,0,1] 2024-06-03T09:06:48.979 INFO:teuthology.orchestra.run.smithi088.stderr:2024-06-03T09:06:48.977+0000 7fa536711640 10 monclient(hunting): picked mon.noname-b con 0x7fa5300a87f0 addr v1:172.21.15.88:6790/0 2024-06-03T09:06:48.979 INFO:teuthology.orchestra.run.smithi088.stderr:2024-06-03T09:06:48.977+0000 7fa536711640 10 monclient(hunting): picked mon.noname-a con 0x7fa5300a8bd0 addr v1:172.21.15.88:6789/0 2024-06-03T09:06:48.979 INFO:teuthology.orchestra.run.smithi088.stderr:2024-06-03T09:06:48.977+0000 7fa536711640 10 monclient(hunting): picked mon.noname-c con 0x7fa5301414c0 addr v1:172.21.15.192:6789/0 2024-06-03T09:06:48.979 INFO:teuthology.orchestra.run.smithi088.stderr:2024-06-03T09:06:48.977+0000 7fa536711640 1 -- --> v1:172.21.15.88:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fa5300aa350 con 0x7fa5300a8bd0 2024-06-03T09:06:48.979 INFO:teuthology.orchestra.run.smithi088.stderr:2024-06-03T09:06:48.977+0000 7fa536711640 1 -- --> v1:172.21.15.192:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fa5300aa5c0 con 0x7fa5301414c0 2024-06-03T09:06:48.979 INFO:teuthology.orchestra.run.smithi088.stderr:2024-06-03T09:06:48.977+0000 7fa536711640 1 -- --> v1:172.21.15.88:6790/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fa530141950 con 0x7fa5300a87f0 2024-06-03T09:06:48.979 INFO:teuthology.orchestra.run.smithi088.stderr:2024-06-03T09:06:48.977+0000 7fa536711640 10 monclient(hunting): _renew_subs 2024-06-03T09:06:48.979 INFO:teuthology.orchestra.run.smithi088.stderr:2024-06-03T09:06:48.977+0000 7fa536711640 10 monclient(hunting): authenticate will time out at 3064.787785s 2024-06-03T09:06:48.980 INFO:teuthology.orchestra.run.smithi088.stderr:2024-06-03T09:06:48.978+0000 7fa52ffff640 1 --1- >> v1:172.21.15.88:6790/0 conn(0x7fa5300a87f0 0x7fa5300b1be0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.88:6790/0 says I am v1:172.21.15.88:52338/0 (socket says 172.21.15.88:52338) 2024-06-03T09:06:48.980 INFO:teuthology.orchestra.run.smithi088.stderr:2024-06-03T09:06:48.978+0000 7fa52ffff640 1 -- 172.21.15.88:0/1161549713 learned_addr learned my addr 172.21.15.88:0/1161549713 (peer_addr_for_me v1:172.21.15.88:0/0) 2024-06-03T09:06:48.981 INFO:teuthology.orchestra.run.smithi088.stderr:2024-06-03T09:06:48.981+0000 7fa52effd640 1 -- 172.21.15.88:0/1161549713 <== mon.2 v1:172.21.15.88:6790/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 807319651 0 0) 0x7fa530141950 con 0x7fa5300a87f0 2024-06-03T09:06:48.981 INFO:teuthology.orchestra.run.smithi088.stderr:2024-06-03T09:06:48.981+0000 7fa52effd640 10 monclient(hunting): _negotiate 2024-06-03T09:06:48.981 INFO:teuthology.orchestra.run.smithi088.stderr:2024-06-03T09:06:48.981+0000 7fa52effd640 10 monclient(hunting): _init_auth method 2 2024-06-03T09:06:48.981 INFO:teuthology.orchestra.run.smithi088.stderr:2024-06-03T09:06:48.981+0000 7fa52effd640 10 monclient(hunting): _init_auth creating new auth 2024-06-03T09:06:48.981 INFO:teuthology.orchestra.run.smithi088.stderr:2024-06-03T09:06:48.981+0000 7fa52effd640 10 monclient(hunting): my global_id is 30212 2024-06-03T09:06:48.981 INFO:teuthology.orchestra.run.smithi088.stderr:2024-06-03T09:06:48.981+0000 7fa52effd640 1 -- 172.21.15.88:0/1161549713 --> v1:172.21.15.88:6790/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7fa518003520 con 0x7fa5300a87f0 2024-06-03T09:06:48.982 INFO:teuthology.orchestra.run.smithi088.stderr:2024-06-03T09:06:48.981+0000 7fa52effd640 1 -- 172.21.15.88:0/1161549713 <== mon.0 v1:172.21.15.88:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 3867840359 0 0) 0x7fa5300aa350 con 0x7fa5300a8bd0 2024-06-03T09:06:48.982 INFO:teuthology.orchestra.run.smithi088.stderr:2024-06-03T09:06:48.981+0000 7fa52effd640 10 monclient(hunting): _negotiate 2024-06-03T09:06:48.982 INFO:teuthology.orchestra.run.smithi088.stderr:2024-06-03T09:06:48.981+0000 7fa52effd640 10 monclient(hunting): _init_auth method 2 2024-06-03T09:06:48.982 INFO:teuthology.orchestra.run.smithi088.stderr:2024-06-03T09:06:48.981+0000 7fa52effd640 10 monclient(hunting): _init_auth creating new auth 2024-06-03T09:06:48.982 INFO:teuthology.orchestra.run.smithi088.stderr:2024-06-03T09:06:48.981+0000 7fa52effd640 10 monclient(hunting): my global_id is 30708 2024-06-03T09:06:48.982 INFO:teuthology.orchestra.run.smithi088.stderr:2024-06-03T09:06:48.981+0000 7fa52effd640 1 -- 172.21.15.88:0/1161549713 --> v1:172.21.15.88:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7fa530141950 con 0x7fa5300a8bd0 2024-06-03T09:06:48.983 INFO:teuthology.orchestra.run.smithi088.stderr:2024-06-03T09:06:48.982+0000 7fa52effd640 1 -- 172.21.15.88:0/1161549713 <== mon.1 v1:172.21.15.192:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 3229327085 0 0) 0x7fa5300aa5c0 con 0x7fa5301414c0 2024-06-03T09:06:48.983 INFO:teuthology.orchestra.run.smithi088.stderr:2024-06-03T09:06:48.982+0000 7fa52effd640 10 monclient(hunting): _negotiate 2024-06-03T09:06:48.983 INFO:teuthology.orchestra.run.smithi088.stderr:2024-06-03T09:06:48.982+0000 7fa52effd640 10 monclient(hunting): _init_auth method 2 2024-06-03T09:06:48.983 INFO:teuthology.orchestra.run.smithi088.stderr:2024-06-03T09:06:48.982+0000 7fa52effd640 10 monclient(hunting): _init_auth creating new auth 2024-06-03T09:06:48.983 INFO:teuthology.orchestra.run.smithi088.stderr:2024-06-03T09:06:48.982+0000 7fa52effd640 10 monclient(hunting): my global_id is 30283 2024-06-03T09:06:48.983 INFO:teuthology.orchestra.run.smithi088.stderr:2024-06-03T09:06:48.982+0000 7fa52effd640 1 -- 172.21.15.88:0/1161549713 --> v1:172.21.15.192:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7fa5300aa350 con 0x7fa5301414c0 2024-06-03T09:06:48.985 INFO:teuthology.orchestra.run.smithi088.stderr:2024-06-03T09:06:48.985+0000 7fa52effd640 1 -- 172.21.15.88:0/1161549713 <== mon.1 v1:172.21.15.192:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 214+0+0 (unknown 2944571576 0 0) 0x7fa5300aa350 con 0x7fa5301414c0 2024-06-03T09:06:48.985 INFO:teuthology.orchestra.run.smithi088.stderr:2024-06-03T09:06:48.985+0000 7fa52effd640 1 -- 172.21.15.88:0/1161549713 >> v1:172.21.15.88:6790/0 conn(0x7fa5300a87f0 legacy=0x7fa5300b1be0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-06-03T09:06:48.985 INFO:teuthology.orchestra.run.smithi088.stderr:2024-06-03T09:06:48.985+0000 7fa52effd640 1 -- 172.21.15.88:0/1161549713 >> v1:172.21.15.88:6789/0 conn(0x7fa5300a8bd0 legacy=0x7fa530141e70 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-06-03T09:06:48.985 INFO:teuthology.orchestra.run.smithi088.stderr:2024-06-03T09:06:48.985+0000 7fa52effd640 10 monclient: _finish_hunting 0 2024-06-03T09:06:48.986 INFO:teuthology.orchestra.run.smithi088.stderr:2024-06-03T09:06:48.985+0000 7fa52effd640 1 monclient: found mon.noname-c 2024-06-03T09:06:48.986 INFO:teuthology.orchestra.run.smithi088.stderr:2024-06-03T09:06:48.985+0000 7fa52effd640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-06-03T09:06:48.986 INFO:teuthology.orchestra.run.smithi088.stderr:2024-06-03T09:06:48.985+0000 7fa52effd640 10 monclient: _send_mon_message to mon.noname-c at v1:172.21.15.192:6789/0 2024-06-03T09:06:48.986 INFO:teuthology.orchestra.run.smithi088.stderr:2024-06-03T09:06:48.985+0000 7fa52effd640 1 -- 172.21.15.88:0/1161549713 --> v1:172.21.15.192:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fa53013b8c0 con 0x7fa5301414c0 2024-06-03T09:06:48.986 INFO:teuthology.orchestra.run.smithi088.stderr:2024-06-03T09:06:48.985+0000 7fa52effd640 10 monclient: _finish_auth 0 2024-06-03T09:06:48.986 INFO:teuthology.orchestra.run.smithi088.stderr:2024-06-03T09:06:48.985+0000 7fa52effd640 10 monclient: _check_auth_tickets 2024-06-03T09:06:48.986 INFO:teuthology.orchestra.run.smithi088.stderr:2024-06-03T09:06:48.985+0000 7fa52effd640 20 monclient: _check_auth_rotating not needed by client.admin 2024-06-03T09:06:48.986 INFO:teuthology.orchestra.run.smithi088.stderr:2024-06-03T09:06:48.985+0000 7fa52effd640 1 -- 172.21.15.88:0/1161549713 <== mon.1 v1:172.21.15.192:6789/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 2234374065 0 0) 0x7fa528001930 con 0x7fa5301414c0 2024-06-03T09:06:48.986 INFO:teuthology.orchestra.run.smithi088.stderr:2024-06-03T09:06:48.985+0000 7fa536711640 5 monclient: authenticate success, global_id 30283 2024-06-03T09:06:48.986 INFO:teuthology.orchestra.run.smithi088.stderr:2024-06-03T09:06:48.985+0000 7fa52effd640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-06-03T09:06:48.986 INFO:teuthology.orchestra.run.smithi088.stderr:2024-06-03T09:06:48.985+0000 7fa52effd640 10 monclient: got monmap 1 from mon.noname-c (according to old e1) 2024-06-03T09:06:48.986 INFO:teuthology.orchestra.run.smithi088.stderr:2024-06-03T09:06:48.985+0000 7fa52effd640 10 monclient: dump: 2024-06-03T09:06:48.987 INFO:teuthology.orchestra.run.smithi088.stderr:epoch 1 2024-06-03T09:06:48.987 INFO:teuthology.orchestra.run.smithi088.stderr:fsid ec3c3fdd-3ac6-4d7d-9412-683130ed8b04 2024-06-03T09:06:48.987 INFO:teuthology.orchestra.run.smithi088.stderr:last_changed 2024-06-03T08:34:31.885125+0000 2024-06-03T09:06:48.987 INFO:teuthology.orchestra.run.smithi088.stderr:created 2024-06-03T08:34:31.885125+0000 2024-06-03T09:06:48.987 INFO:teuthology.orchestra.run.smithi088.stderr:min_mon_release 18 (reef) 2024-06-03T09:06:48.987 INFO:teuthology.orchestra.run.smithi088.stderr:election_strategy: 3 2024-06-03T09:06:48.987 INFO:teuthology.orchestra.run.smithi088.stderr:0: v1:172.21.15.88:6789/0 mon.a 2024-06-03T09:06:48.987 INFO:teuthology.orchestra.run.smithi088.stderr:1: v1:172.21.15.192:6789/0 mon.b 2024-06-03T09:06:48.987 INFO:teuthology.orchestra.run.smithi088.stderr:2: v1:172.21.15.88:6790/0 mon.c 2024-06-03T09:06:48.987 INFO:teuthology.orchestra.run.smithi088.stderr: 2024-06-03T09:06:48.987 INFO:teuthology.orchestra.run.smithi088.stderr:2024-06-03T09:06:48.985+0000 7fa536711640 20 monclient: get_monmap_and_config waiting for monmap|config 2024-06-03T09:06:48.989 INFO:teuthology.orchestra.run.smithi088.stderr:2024-06-03T09:06:48.987+0000 7fa52effd640 1 -- 172.21.15.88:0/1161549713 <== mon.1 v1:172.21.15.192:6789/0 4 ==== config(0 keys) v1 ==== 4+0+0 (unknown 0 0 0) 0x7fa528001b50 con 0x7fa5301414c0 2024-06-03T09:06:48.989 INFO:teuthology.orchestra.run.smithi088.stderr:2024-06-03T09:06:48.987+0000 7fa52effd640 10 monclient: handle_config config(0 keys) v1 2024-06-03T09:06:48.989 INFO:teuthology.orchestra.run.smithi088.stderr:2024-06-03T09:06:48.987+0000 7fa52effd640 1 -- 172.21.15.88:0/1161549713 <== mon.1 v1:172.21.15.192:6789/0 5 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 2234374065 0 0) 0x7fa528001ee0 con 0x7fa5301414c0 2024-06-03T09:06:48.989 INFO:teuthology.orchestra.run.smithi088.stderr:2024-06-03T09:06:48.987+0000 7fa536711640 10 monclient: get_monmap_and_config success 2024-06-03T09:06:48.989 INFO:teuthology.orchestra.run.smithi088.stderr:2024-06-03T09:06:48.987+0000 7fa536711640 4 set_mon_vals no callback set 2024-06-03T09:06:48.989 INFO:teuthology.orchestra.run.smithi088.stderr:2024-06-03T09:06:48.987+0000 7fa52effd640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-06-03T09:06:48.989 INFO:teuthology.orchestra.run.smithi088.stderr:2024-06-03T09:06:48.987+0000 7fa52effd640 10 monclient: got monmap 1 from mon.b (according to old e1) 2024-06-03T09:06:48.989 INFO:teuthology.orchestra.run.smithi088.stderr:2024-06-03T09:06:48.987+0000 7fa52effd640 10 monclient: dump: 2024-06-03T09:06:48.989 INFO:teuthology.orchestra.run.smithi088.stderr:epoch 1 2024-06-03T09:06:48.989 INFO:teuthology.orchestra.run.smithi088.stderr:fsid ec3c3fdd-3ac6-4d7d-9412-683130ed8b04 2024-06-03T09:06:48.989 INFO:teuthology.orchestra.run.smithi088.stderr:last_changed 2024-06-03T08:34:31.885125+0000 2024-06-03T09:06:48.989 INFO:teuthology.orchestra.run.smithi088.stderr:created 2024-06-03T08:34:31.885125+0000 2024-06-03T09:06:48.989 INFO:teuthology.orchestra.run.smithi088.stderr:min_mon_release 18 (reef) 2024-06-03T09:06:48.990 INFO:teuthology.orchestra.run.smithi088.stderr:election_strategy: 3 2024-06-03T09:06:48.990 INFO:teuthology.orchestra.run.smithi088.stderr:0: v1:172.21.15.88:6789/0 mon.a 2024-06-03T09:06:48.990 INFO:teuthology.orchestra.run.smithi088.stderr:1: v1:172.21.15.192:6789/0 mon.b 2024-06-03T09:06:48.990 INFO:teuthology.orchestra.run.smithi088.stderr:2: v1:172.21.15.88:6790/0 mon.c 2024-06-03T09:06:48.990 INFO:teuthology.orchestra.run.smithi088.stderr: 2024-06-03T09:06:48.990 INFO:teuthology.orchestra.run.smithi088.stderr:2024-06-03T09:06:48.988+0000 7fa536711640 10 monclient: shutdown 2024-06-03T09:06:48.990 INFO:teuthology.orchestra.run.smithi088.stderr:2024-06-03T09:06:48.988+0000 7fa536711640 20 monclient: shutdown discarding 0 pending message(s) 2024-06-03T09:06:48.990 INFO:teuthology.orchestra.run.smithi088.stderr:2024-06-03T09:06:48.988+0000 7fa536711640 1 -- 172.21.15.88:0/1161549713 >> v1:172.21.15.192:6789/0 conn(0x7fa5301414c0 legacy=0x7fa53013b1b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-06-03T09:06:48.991 INFO:teuthology.orchestra.run.smithi088.stderr:2024-06-03T09:06:48.988+0000 7fa536711640 1 -- 172.21.15.88:0/1161549713 shutdown_connections 2024-06-03T09:06:48.991 INFO:teuthology.orchestra.run.smithi088.stderr:2024-06-03T09:06:48.988+0000 7fa536711640 1 -- 172.21.15.88:0/1161549713 >> 172.21.15.88:0/1161549713 conn(0x7fa5300b71e0 msgr2=0x7fa5300b75c0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-06-03T09:06:48.991 INFO:teuthology.orchestra.run.smithi088.stderr:2024-06-03T09:06:48.988+0000 7fa536711640 1 -- 172.21.15.88:0/1161549713 shutdown_connections 2024-06-03T09:06:48.991 INFO:teuthology.orchestra.run.smithi088.stderr:2024-06-03T09:06:48.988+0000 7fa536711640 1 -- 172.21.15.88:0/1161549713 wait complete. 2024-06-03T09:06:48.994 INFO:teuthology.orchestra.run.smithi088.stderr:2024-06-03T09:06:48.989+0000 7fa536711640 10 monclient: build_initial_monmap 2024-06-03T09:06:48.994 INFO:teuthology.orchestra.run.smithi088.stderr:2024-06-03T09:06:48.989+0000 7fa536711640 10 monclient: monmap: 2024-06-03T09:06:48.994 INFO:teuthology.orchestra.run.smithi088.stderr:epoch 0 2024-06-03T09:06:48.994 INFO:teuthology.orchestra.run.smithi088.stderr:fsid 00000000-0000-0000-0000-000000000000 2024-06-03T09:06:48.994 INFO:teuthology.orchestra.run.smithi088.stderr:last_changed 0.000000 2024-06-03T09:06:48.994 INFO:teuthology.orchestra.run.smithi088.stderr:created 0.000000 2024-06-03T09:06:48.994 INFO:teuthology.orchestra.run.smithi088.stderr:min_mon_release 0 (unknown) 2024-06-03T09:06:48.994 INFO:teuthology.orchestra.run.smithi088.stderr:election_strategy: 1 2024-06-03T09:06:48.994 INFO:teuthology.orchestra.run.smithi088.stderr:0: v1:172.21.15.88:6789/0 mon.noname-a 2024-06-03T09:06:48.995 INFO:teuthology.orchestra.run.smithi088.stderr:1: v1:172.21.15.192:6789/0 mon.noname-b 2024-06-03T09:06:48.995 INFO:teuthology.orchestra.run.smithi088.stderr:2: v1:172.21.15.88:6790/0 mon.noname-c 2024-06-03T09:06:48.995 INFO:teuthology.orchestra.run.smithi088.stderr: 2024-06-03T09:06:48.995 INFO:teuthology.orchestra.run.smithi088.stderr:2024-06-03T09:06:48.989+0000 7fa536711640 1 librados: starting msgr at 2024-06-03T09:06:48.995 INFO:teuthology.orchestra.run.smithi088.stderr:2024-06-03T09:06:48.989+0000 7fa536711640 1 librados: starting objecter 2024-06-03T09:06:48.995 INFO:teuthology.orchestra.run.smithi088.stderr:2024-06-03T09:06:48.989+0000 7fa536711640 1 Processor -- start 2024-06-03T09:06:48.995 INFO:teuthology.orchestra.run.smithi088.stderr:2024-06-03T09:06:48.990+0000 7fa536711640 1 -- start start 2024-06-03T09:06:48.995 INFO:teuthology.orchestra.run.smithi088.stderr:2024-06-03T09:06:48.990+0000 7fa536711640 1 librados: setting wanted keys 2024-06-03T09:06:48.995 INFO:teuthology.orchestra.run.smithi088.stderr:2024-06-03T09:06:48.990+0000 7fa536711640 1 librados: calling monclient init 2024-06-03T09:06:48.995 INFO:teuthology.orchestra.run.smithi088.stderr:2024-06-03T09:06:48.990+0000 7fa536711640 10 monclient: init 2024-06-03T09:06:48.995 INFO:teuthology.orchestra.run.smithi088.stderr:2024-06-03T09:06:48.990+0000 7fa536711640 10 monclient: _reopen_session rank -1 2024-06-03T09:06:48.995 INFO:teuthology.orchestra.run.smithi088.stderr:2024-06-03T09:06:48.990+0000 7fa536711640 10 monclient: _add_conns ranks=[2,1,0] 2024-06-03T09:06:48.996 INFO:teuthology.orchestra.run.smithi088.stderr:2024-06-03T09:06:48.990+0000 7fa536711640 10 monclient(hunting): picked mon.noname-c con 0x7fa5300a87f0 addr v1:172.21.15.88:6790/0 2024-06-03T09:06:48.996 INFO:teuthology.orchestra.run.smithi088.stderr:2024-06-03T09:06:48.990+0000 7fa536711640 10 monclient(hunting): picked mon.noname-b con 0x7fa53012ea80 addr v1:172.21.15.192:6789/0 2024-06-03T09:06:48.996 INFO:teuthology.orchestra.run.smithi088.stderr:2024-06-03T09:06:48.990+0000 7fa536711640 10 monclient(hunting): picked mon.noname-a con 0x7fa53013d930 addr v1:172.21.15.88:6789/0 2024-06-03T09:06:48.996 INFO:teuthology.orchestra.run.smithi088.stderr:2024-06-03T09:06:48.990+0000 7fa536711640 1 -- --> v1:172.21.15.88:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fa530130c40 con 0x7fa53013d930 2024-06-03T09:06:48.996 INFO:teuthology.orchestra.run.smithi088.stderr:2024-06-03T09:06:48.990+0000 7fa536711640 1 -- --> v1:172.21.15.192:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fa530130dd0 con 0x7fa53012ea80 2024-06-03T09:06:48.996 INFO:teuthology.orchestra.run.smithi088.stderr:2024-06-03T09:06:48.990+0000 7fa536711640 1 -- --> v1:172.21.15.88:6790/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fa530130f60 con 0x7fa5300a87f0 2024-06-03T09:06:48.996 INFO:teuthology.orchestra.run.smithi088.stderr:2024-06-03T09:06:48.990+0000 7fa536711640 10 monclient(hunting): _renew_subs 2024-06-03T09:06:48.996 INFO:teuthology.orchestra.run.smithi088.stderr:2024-06-03T09:06:48.990+0000 7fa536711640 10 monclient(hunting): authenticate will time out at 3064.800566s 2024-06-03T09:06:48.996 INFO:teuthology.orchestra.run.smithi088.stderr:2024-06-03T09:06:48.991+0000 7fa52ffff640 1 --1- >> v1:172.21.15.88:6790/0 conn(0x7fa5300a87f0 0x7fa53012e370 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=1).handle_server_banner_and_identify peer v1:172.21.15.88:6790/0 says I am v1:172.21.15.88:52346/0 (socket says 172.21.15.88:52346) 2024-06-03T09:06:48.996 INFO:teuthology.orchestra.run.smithi088.stderr:2024-06-03T09:06:48.991+0000 7fa52ffff640 1 -- 172.21.15.88:0/2250717138 learned_addr learned my addr 172.21.15.88:0/2250717138 (peer_addr_for_me v1:172.21.15.88:0/0) 2024-06-03T09:06:48.996 INFO:teuthology.orchestra.run.smithi088.stderr:2024-06-03T09:06:48.991+0000 7fa52cff9640 10 client.?.objecter ms_handle_connect 0x7fa5300a87f0 2024-06-03T09:06:48.996 INFO:teuthology.orchestra.run.smithi088.stderr:2024-06-03T09:06:48.991+0000 7fa52cff9640 10 client.?.objecter resend_mon_ops 2024-06-03T09:06:48.997 INFO:teuthology.orchestra.run.smithi088.stderr:2024-06-03T09:06:48.991+0000 7fa52cff9640 10 client.?.objecter ms_handle_connect 0x7fa53013d930 2024-06-03T09:06:48.997 INFO:teuthology.orchestra.run.smithi088.stderr:2024-06-03T09:06:48.991+0000 7fa52cff9640 10 client.?.objecter resend_mon_ops 2024-06-03T09:06:48.997 INFO:teuthology.orchestra.run.smithi088.stderr:2024-06-03T09:06:48.991+0000 7fa52cff9640 10 client.?.objecter ms_handle_connect 0x7fa53012ea80 2024-06-03T09:06:48.997 INFO:teuthology.orchestra.run.smithi088.stderr:2024-06-03T09:06:48.991+0000 7fa52cff9640 10 client.?.objecter resend_mon_ops 2024-06-03T09:06:48.997 INFO:teuthology.orchestra.run.smithi088.stderr:2024-06-03T09:06:48.993+0000 7fa52cff9640 1 -- 172.21.15.88:0/2250717138 <== mon.2 v1:172.21.15.88:6790/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 3756583553 0 0) 0x7fa530130f60 con 0x7fa5300a87f0 2024-06-03T09:06:48.997 INFO:teuthology.orchestra.run.smithi088.stderr:2024-06-03T09:06:48.993+0000 7fa52cff9640 10 monclient(hunting): _negotiate 2024-06-03T09:06:48.997 INFO:teuthology.orchestra.run.smithi088.stderr:2024-06-03T09:06:48.993+0000 7fa52cff9640 10 monclient(hunting): _init_auth method 2 2024-06-03T09:06:48.997 INFO:teuthology.orchestra.run.smithi088.stderr:2024-06-03T09:06:48.993+0000 7fa52cff9640 10 monclient(hunting): _init_auth creating new auth 2024-06-03T09:06:48.997 INFO:teuthology.orchestra.run.smithi088.stderr:2024-06-03T09:06:48.993+0000 7fa52cff9640 10 monclient(hunting): my global_id is 30215 2024-06-03T09:06:48.997 INFO:teuthology.orchestra.run.smithi088.stderr:2024-06-03T09:06:48.993+0000 7fa52cff9640 1 -- 172.21.15.88:0/2250717138 --> v1:172.21.15.88:6790/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7fa51c004eb0 con 0x7fa5300a87f0 2024-06-03T09:06:48.997 INFO:teuthology.orchestra.run.smithi088.stderr:2024-06-03T09:06:48.993+0000 7fa52cff9640 1 -- 172.21.15.88:0/2250717138 <== mon.0 v1:172.21.15.88:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 39768161 0 0) 0x7fa530130c40 con 0x7fa53013d930 2024-06-03T09:06:48.997 INFO:teuthology.orchestra.run.smithi088.stderr:2024-06-03T09:06:48.993+0000 7fa52cff9640 10 monclient(hunting): _negotiate 2024-06-03T09:06:48.997 INFO:teuthology.orchestra.run.smithi088.stderr:2024-06-03T09:06:48.993+0000 7fa52cff9640 10 monclient(hunting): _init_auth method 2 2024-06-03T09:06:48.998 INFO:teuthology.orchestra.run.smithi088.stderr:2024-06-03T09:06:48.993+0000 7fa52cff9640 10 monclient(hunting): _init_auth creating new auth 2024-06-03T09:06:48.998 INFO:teuthology.orchestra.run.smithi088.stderr:2024-06-03T09:06:48.993+0000 7fa52cff9640 10 monclient(hunting): my global_id is 30711 2024-06-03T09:06:48.998 INFO:teuthology.orchestra.run.smithi088.stderr:2024-06-03T09:06:48.993+0000 7fa52cff9640 1 -- 172.21.15.88:0/2250717138 --> v1:172.21.15.88:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7fa530130f60 con 0x7fa53013d930 2024-06-03T09:06:48.998 INFO:teuthology.orchestra.run.smithi088.stderr:2024-06-03T09:06:48.993+0000 7fa52cff9640 1 -- 172.21.15.88:0/2250717138 <== mon.1 v1:172.21.15.192:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 3970189997 0 0) 0x7fa530130dd0 con 0x7fa53012ea80 2024-06-03T09:06:48.998 INFO:teuthology.orchestra.run.smithi088.stderr:2024-06-03T09:06:48.993+0000 7fa52cff9640 10 monclient(hunting): _negotiate 2024-06-03T09:06:48.998 INFO:teuthology.orchestra.run.smithi088.stderr:2024-06-03T09:06:48.993+0000 7fa52cff9640 10 monclient(hunting): _init_auth method 2 2024-06-03T09:06:48.998 INFO:teuthology.orchestra.run.smithi088.stderr:2024-06-03T09:06:48.993+0000 7fa52cff9640 10 monclient(hunting): _init_auth creating new auth 2024-06-03T09:06:48.998 INFO:teuthology.orchestra.run.smithi088.stderr:2024-06-03T09:06:48.993+0000 7fa52cff9640 10 monclient(hunting): my global_id is 30286 2024-06-03T09:06:48.998 INFO:teuthology.orchestra.run.smithi088.stderr:2024-06-03T09:06:48.993+0000 7fa52cff9640 1 -- 172.21.15.88:0/2250717138 --> v1:172.21.15.192:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7fa530130c40 con 0x7fa53012ea80 2024-06-03T09:06:49.000 INFO:teuthology.orchestra.run.smithi088.stderr:2024-06-03T09:06:48.998+0000 7fa52cff9640 1 -- 172.21.15.88:0/2250717138 <== mon.0 v1:172.21.15.88:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 764+0+0 (unknown 2043540717 0 0) 0x7fa530130f60 con 0x7fa53013d930 2024-06-03T09:06:49.000 INFO:teuthology.orchestra.run.smithi088.stderr:2024-06-03T09:06:48.998+0000 7fa52cff9640 1 -- 172.21.15.88:0/2250717138 --> v1:172.21.15.88:6789/0 -- auth(proto 2 165 bytes epoch 0) v1 -- 0x7fa530130dd0 con 0x7fa53013d930 2024-06-03T09:06:49.000 INFO:teuthology.orchestra.run.smithi088.stderr:2024-06-03T09:06:48.998+0000 7fa52cff9640 1 -- 172.21.15.88:0/2250717138 <== mon.0 v1:172.21.15.88:6789/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 2234374065 0 0) 0x7fa528001e60 con 0x7fa53013d930 2024-06-03T09:06:49.000 INFO:teuthology.orchestra.run.smithi088.stderr:2024-06-03T09:06:48.998+0000 7fa52cff9640 10 monclient(hunting): handle_monmap mon_map magic: 0 v1 2024-06-03T09:06:49.000 INFO:teuthology.orchestra.run.smithi088.stderr:2024-06-03T09:06:48.998+0000 7fa52cff9640 10 monclient(hunting): got monmap 1 from mon.noname-a (according to old e1) 2024-06-03T09:06:49.000 INFO:teuthology.orchestra.run.smithi088.stderr:2024-06-03T09:06:48.998+0000 7fa52cff9640 10 monclient(hunting): dump: 2024-06-03T09:06:49.001 INFO:teuthology.orchestra.run.smithi088.stderr:epoch 1 2024-06-03T09:06:49.001 INFO:teuthology.orchestra.run.smithi088.stderr:fsid ec3c3fdd-3ac6-4d7d-9412-683130ed8b04 2024-06-03T09:06:49.001 INFO:teuthology.orchestra.run.smithi088.stderr:last_changed 2024-06-03T08:34:31.885125+0000 2024-06-03T09:06:49.001 INFO:teuthology.orchestra.run.smithi088.stderr:created 2024-06-03T08:34:31.885125+0000 2024-06-03T09:06:49.001 INFO:teuthology.orchestra.run.smithi088.stderr:min_mon_release 18 (reef) 2024-06-03T09:06:49.001 INFO:teuthology.orchestra.run.smithi088.stderr:election_strategy: 3 2024-06-03T09:06:49.001 INFO:teuthology.orchestra.run.smithi088.stderr:0: v1:172.21.15.88:6789/0 mon.a 2024-06-03T09:06:49.001 INFO:teuthology.orchestra.run.smithi088.stderr:1: v1:172.21.15.192:6789/0 mon.b 2024-06-03T09:06:49.001 INFO:teuthology.orchestra.run.smithi088.stderr:2: v1:172.21.15.88:6790/0 mon.c 2024-06-03T09:06:49.001 INFO:teuthology.orchestra.run.smithi088.stderr: 2024-06-03T09:06:49.001 INFO:teuthology.orchestra.run.smithi088.stderr:2024-06-03T09:06:48.998+0000 7fa52cff9640 10 monclient(hunting): _finish_auth 0 2024-06-03T09:06:49.001 INFO:teuthology.orchestra.run.smithi088.stderr:2024-06-03T09:06:48.998+0000 7fa52cff9640 1 -- 172.21.15.88:0/2250717138 <== mon.2 v1:172.21.15.88:6790/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 764+0+0 (unknown 3258749122 0 0) 0x7fa51c004eb0 con 0x7fa5300a87f0 2024-06-03T09:06:49.001 INFO:teuthology.orchestra.run.smithi088.stderr:2024-06-03T09:06:48.998+0000 7fa52cff9640 1 -- 172.21.15.88:0/2250717138 --> v1:172.21.15.88:6790/0 -- auth(proto 2 165 bytes epoch 0) v1 -- 0x7fa530130f60 con 0x7fa5300a87f0 2024-06-03T09:06:49.002 INFO:teuthology.orchestra.run.smithi088.stderr:2024-06-03T09:06:48.998+0000 7fa52cff9640 1 -- 172.21.15.88:0/2250717138 <== mon.2 v1:172.21.15.88:6790/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 2234374065 0 0) 0x7fa5200017e0 con 0x7fa5300a87f0 2024-06-03T09:06:49.002 INFO:teuthology.orchestra.run.smithi088.stderr:2024-06-03T09:06:48.998+0000 7fa52cff9640 10 monclient(hunting): handle_monmap mon_map magic: 0 v1 2024-06-03T09:06:49.002 INFO:teuthology.orchestra.run.smithi088.stderr:2024-06-03T09:06:48.998+0000 7fa52cff9640 10 monclient(hunting): got monmap 1 from mon.c (according to old e1) 2024-06-03T09:06:49.002 INFO:teuthology.orchestra.run.smithi088.stderr:2024-06-03T09:06:48.998+0000 7fa52cff9640 10 monclient(hunting): dump: 2024-06-03T09:06:49.002 INFO:teuthology.orchestra.run.smithi088.stderr:epoch 1 2024-06-03T09:06:49.002 INFO:teuthology.orchestra.run.smithi088.stderr:fsid ec3c3fdd-3ac6-4d7d-9412-683130ed8b04 2024-06-03T09:06:49.002 INFO:teuthology.orchestra.run.smithi088.stderr:last_changed 2024-06-03T08:34:31.885125+0000 2024-06-03T09:06:49.002 INFO:teuthology.orchestra.run.smithi088.stderr:created 2024-06-03T08:34:31.885125+0000 2024-06-03T09:06:49.002 INFO:teuthology.orchestra.run.smithi088.stderr:min_mon_release 18 (reef) 2024-06-03T09:06:49.002 INFO:teuthology.orchestra.run.smithi088.stderr:election_strategy: 3 2024-06-03T09:06:49.003 INFO:teuthology.orchestra.run.smithi088.stderr:0: v1:172.21.15.88:6789/0 mon.a 2024-06-03T09:06:49.003 INFO:teuthology.orchestra.run.smithi088.stderr:1: v1:172.21.15.192:6789/0 mon.b 2024-06-03T09:06:49.003 INFO:teuthology.orchestra.run.smithi088.stderr:2: v1:172.21.15.88:6790/0 mon.c 2024-06-03T09:06:49.003 INFO:teuthology.orchestra.run.smithi088.stderr: 2024-06-03T09:06:49.003 INFO:teuthology.orchestra.run.smithi088.stderr:2024-06-03T09:06:48.998+0000 7fa52cff9640 1 -- 172.21.15.88:0/2250717138 <== mon.1 v1:172.21.15.192:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 764+0+0 (unknown 2706931132 0 0) 0x7fa530130c40 con 0x7fa53012ea80 2024-06-03T09:06:49.003 INFO:teuthology.orchestra.run.smithi088.stderr:2024-06-03T09:06:48.998+0000 7fa52cff9640 1 -- 172.21.15.88:0/2250717138 --> v1:172.21.15.192:6789/0 -- auth(proto 2 165 bytes epoch 0) v1 -- 0x7fa51c004eb0 con 0x7fa53012ea80 2024-06-03T09:06:49.003 INFO:teuthology.orchestra.run.smithi088.stderr:2024-06-03T09:06:48.998+0000 7fa52cff9640 1 -- 172.21.15.88:0/2250717138 <== mon.1 v1:172.21.15.192:6789/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 2234374065 0 0) 0x7fa5240018b0 con 0x7fa53012ea80 2024-06-03T09:06:49.003 INFO:teuthology.orchestra.run.smithi088.stderr:2024-06-03T09:06:48.998+0000 7fa52cff9640 10 monclient(hunting): handle_monmap mon_map magic: 0 v1 2024-06-03T09:06:49.003 INFO:teuthology.orchestra.run.smithi088.stderr:2024-06-03T09:06:48.998+0000 7fa52cff9640 10 monclient(hunting): got monmap 1 from mon.b (according to old e1) 2024-06-03T09:06:49.003 INFO:teuthology.orchestra.run.smithi088.stderr:2024-06-03T09:06:48.998+0000 7fa52cff9640 10 monclient(hunting): dump: 2024-06-03T09:06:49.003 INFO:teuthology.orchestra.run.smithi088.stderr:epoch 1 2024-06-03T09:06:49.003 INFO:teuthology.orchestra.run.smithi088.stderr:fsid ec3c3fdd-3ac6-4d7d-9412-683130ed8b04 2024-06-03T09:06:49.003 INFO:teuthology.orchestra.run.smithi088.stderr:last_changed 2024-06-03T08:34:31.885125+0000 2024-06-03T09:06:49.004 INFO:teuthology.orchestra.run.smithi088.stderr:created 2024-06-03T08:34:31.885125+0000 2024-06-03T09:06:49.004 INFO:teuthology.orchestra.run.smithi088.stderr:min_mon_release 18 (reef) 2024-06-03T09:06:49.004 INFO:teuthology.orchestra.run.smithi088.stderr:election_strategy: 3 2024-06-03T09:06:49.004 INFO:teuthology.orchestra.run.smithi088.stderr:0: v1:172.21.15.88:6789/0 mon.a 2024-06-03T09:06:49.004 INFO:teuthology.orchestra.run.smithi088.stderr:1: v1:172.21.15.192:6789/0 mon.b 2024-06-03T09:06:49.004 INFO:teuthology.orchestra.run.smithi088.stderr:2: v1:172.21.15.88:6790/0 mon.c 2024-06-03T09:06:49.004 INFO:teuthology.orchestra.run.smithi088.stderr: 2024-06-03T09:06:49.005 INFO:teuthology.orchestra.run.smithi088.stderr:2024-06-03T09:06:49.002+0000 7fa52cff9640 1 -- 172.21.15.88:0/2250717138 <== mon.2 v1:172.21.15.88:6790/0 4 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 580+0+0 (unknown 818021909 0 0) 0x7fa530130f60 con 0x7fa5300a87f0 2024-06-03T09:06:49.005 INFO:teuthology.orchestra.run.smithi088.stderr:2024-06-03T09:06:49.002+0000 7fa52cff9640 1 -- 172.21.15.88:0/2250717138 >> v1:172.21.15.192:6789/0 conn(0x7fa53012ea80 legacy=0x7fa5301d62f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-06-03T09:06:49.006 INFO:teuthology.orchestra.run.smithi088.stderr:2024-06-03T09:06:49.002+0000 7fa52cff9640 1 -- 172.21.15.88:0/2250717138 >> v1:172.21.15.88:6789/0 conn(0x7fa53013d930 legacy=0x7fa5301d6a00 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-06-03T09:06:49.006 INFO:teuthology.orchestra.run.smithi088.stderr:2024-06-03T09:06:49.002+0000 7fa52cff9640 10 monclient: _finish_hunting 0 2024-06-03T09:06:49.006 INFO:teuthology.orchestra.run.smithi088.stderr:2024-06-03T09:06:49.002+0000 7fa52cff9640 1 monclient: found mon.c 2024-06-03T09:06:49.006 INFO:teuthology.orchestra.run.smithi088.stderr:2024-06-03T09:06:49.002+0000 7fa52cff9640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-06-03T09:06:49.006 INFO:teuthology.orchestra.run.smithi088.stderr:2024-06-03T09:06:49.002+0000 7fa52cff9640 10 monclient: _send_mon_message to mon.c at v1:172.21.15.88:6790/0 2024-06-03T09:06:49.006 INFO:teuthology.orchestra.run.smithi088.stderr:2024-06-03T09:06:49.002+0000 7fa52cff9640 1 -- 172.21.15.88:0/2250717138 --> v1:172.21.15.88:6790/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fa53013d240 con 0x7fa5300a87f0 2024-06-03T09:06:49.006 INFO:teuthology.orchestra.run.smithi088.stderr:2024-06-03T09:06:49.002+0000 7fa52cff9640 10 monclient: _finish_auth 0 2024-06-03T09:06:49.006 INFO:teuthology.orchestra.run.smithi088.stderr:2024-06-03T09:06:49.002+0000 7fa52cff9640 10 monclient: _check_auth_tickets 2024-06-03T09:06:49.006 INFO:teuthology.orchestra.run.smithi088.stderr:2024-06-03T09:06:49.002+0000 7fa52cff9640 20 monclient: _check_auth_rotating not needed by client.admin 2024-06-03T09:06:49.006 INFO:teuthology.orchestra.run.smithi088.stderr:2024-06-03T09:06:49.002+0000 7fa536711640 5 monclient: authenticate success, global_id 30215 2024-06-03T09:06:49.006 INFO:teuthology.orchestra.run.smithi088.stderr:2024-06-03T09:06:49.002+0000 7fa536711640 10 monclient: _renew_subs 2024-06-03T09:06:49.006 INFO:teuthology.orchestra.run.smithi088.stderr:2024-06-03T09:06:49.002+0000 7fa536711640 10 monclient: _send_mon_message to mon.c at v1:172.21.15.88:6790/0 2024-06-03T09:06:49.007 INFO:teuthology.orchestra.run.smithi088.stderr:2024-06-03T09:06:49.002+0000 7fa536711640 1 -- 172.21.15.88:0/2250717138 --> v1:172.21.15.88:6790/0 -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fa53013d3d0 con 0x7fa5300a87f0 2024-06-03T09:06:49.007 INFO:teuthology.orchestra.run.smithi088.stderr:2024-06-03T09:06:49.002+0000 7fa536711640 10 client.30215.objecter _maybe_request_map subscribing (onetime) to next osd map 2024-06-03T09:06:49.007 INFO:teuthology.orchestra.run.smithi088.stderr:2024-06-03T09:06:49.002+0000 7fa536711640 10 monclient: _renew_subs 2024-06-03T09:06:49.007 INFO:teuthology.orchestra.run.smithi088.stderr:2024-06-03T09:06:49.002+0000 7fa536711640 10 monclient: _send_mon_message to mon.c at v1:172.21.15.88:6790/0 2024-06-03T09:06:49.007 INFO:teuthology.orchestra.run.smithi088.stderr:2024-06-03T09:06:49.002+0000 7fa536711640 1 -- 172.21.15.88:0/2250717138 --> v1:172.21.15.88:6790/0 -- mon_subscribe({osdmap=0}) v3 -- 0x7fa53013d680 con 0x7fa5300a87f0 2024-06-03T09:06:49.007 INFO:teuthology.orchestra.run.smithi088.stderr:2024-06-03T09:06:49.002+0000 7fa536711640 1 librados: init done 2024-06-03T09:06:49.007 INFO:teuthology.orchestra.run.smithi088.stderr:2024-06-03T09:06:49.003+0000 7fa536711640 10 start_mon_command cmd=[{"prefix": "get_command_descriptions"}] 2024-06-03T09:06:49.007 INFO:teuthology.orchestra.run.smithi088.stderr:2024-06-03T09:06:49.003+0000 7fa536711640 10 monclient: _send_command 1 [{"prefix": "get_command_descriptions"}] 2024-06-03T09:06:49.007 INFO:teuthology.orchestra.run.smithi088.stderr:2024-06-03T09:06:49.003+0000 7fa536711640 10 monclient: _send_mon_message to mon.c at v1:172.21.15.88:6790/0 2024-06-03T09:06:49.007 INFO:teuthology.orchestra.run.smithi088.stderr:2024-06-03T09:06:49.003+0000 7fa536711640 1 -- 172.21.15.88:0/2250717138 --> v1:172.21.15.88:6790/0 -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fa53004ea60 con 0x7fa5300a87f0 2024-06-03T09:06:49.018 INFO:teuthology.orchestra.run.smithi088.stderr:2024-06-03T09:06:49.018+0000 7fa52cff9640 1 -- 172.21.15.88:0/2250717138 <== mon.2 v1:172.21.15.88:6790/0 5 ==== config(0 keys) v1 ==== 4+0+0 (unknown 0 0 0) 0x7fa520001d50 con 0x7fa5300a87f0 2024-06-03T09:06:49.018 INFO:teuthology.orchestra.run.smithi088.stderr:2024-06-03T09:06:49.018+0000 7fa52cff9640 10 monclient: handle_config config(0 keys) v1 2024-06-03T09:06:49.018 INFO:teuthology.orchestra.run.smithi088.stderr:2024-06-03T09:06:49.018+0000 7fa52cff9640 1 -- 172.21.15.88:0/2250717138 <== mon.2 v1:172.21.15.88:6790/0 6 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 2234374065 0 0) 0x7fa520004980 con 0x7fa5300a87f0 2024-06-03T09:06:49.018 INFO:teuthology.orchestra.run.smithi088.stderr:2024-06-03T09:06:49.018+0000 7fa52cff9640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-06-03T09:06:49.018 INFO:teuthology.orchestra.run.smithi088.stderr:2024-06-03T09:06:49.018+0000 7fa52cff9640 10 monclient: got monmap 1 from mon.c (according to old e1) 2024-06-03T09:06:49.018 INFO:teuthology.orchestra.run.smithi088.stderr:2024-06-03T09:06:49.018+0000 7fa52cff9640 10 monclient: dump: 2024-06-03T09:06:49.018 INFO:teuthology.orchestra.run.smithi088.stderr:epoch 1 2024-06-03T09:06:49.018 INFO:teuthology.orchestra.run.smithi088.stderr:fsid ec3c3fdd-3ac6-4d7d-9412-683130ed8b04 2024-06-03T09:06:49.018 INFO:teuthology.orchestra.run.smithi088.stderr:last_changed 2024-06-03T08:34:31.885125+0000 2024-06-03T09:06:49.018 INFO:teuthology.orchestra.run.smithi088.stderr:created 2024-06-03T08:34:31.885125+0000 2024-06-03T09:06:49.018 INFO:teuthology.orchestra.run.smithi088.stderr:min_mon_release 18 (reef) 2024-06-03T09:06:49.019 INFO:teuthology.orchestra.run.smithi088.stderr:election_strategy: 3 2024-06-03T09:06:49.019 INFO:teuthology.orchestra.run.smithi088.stderr:0: v1:172.21.15.88:6789/0 mon.a 2024-06-03T09:06:49.019 INFO:teuthology.orchestra.run.smithi088.stderr:1: v1:172.21.15.192:6789/0 mon.b 2024-06-03T09:06:49.019 INFO:teuthology.orchestra.run.smithi088.stderr:2: v1:172.21.15.88:6790/0 mon.c 2024-06-03T09:06:49.019 INFO:teuthology.orchestra.run.smithi088.stderr: 2024-06-03T09:06:49.019 INFO:teuthology.orchestra.run.smithi088.stderr:2024-06-03T09:06:49.018+0000 7fa52e7fc640 4 set_mon_vals no callback set 2024-06-03T09:06:49.019 INFO:teuthology.orchestra.run.smithi088.stderr:2024-06-03T09:06:49.018+0000 7fa52cff9640 1 -- 172.21.15.88:0/2250717138 <== mon.2 v1:172.21.15.88:6790/0 7 ==== mgrmap(e 5) v1 ==== 89110+0+0 (unknown 873533973 0 0) 0x7fa52001a700 con 0x7fa5300a87f0 2024-06-03T09:06:49.020 INFO:teuthology.orchestra.run.smithi088.stderr:2024-06-03T09:06:49.019+0000 7fa52f7fe640 10 monclient: get_auth_request con 0x7fa51c06d440 auth_method 0 2024-06-03T09:06:49.020 INFO:teuthology.orchestra.run.smithi088.stderr:2024-06-03T09:06:49.019+0000 7fa52cff9640 10 client.30215.objecter ms_handle_connect 0x7fa51c06d440 2024-06-03T09:06:49.020 INFO:teuthology.orchestra.run.smithi088.stderr:2024-06-03T09:06:49.020+0000 7fa52cff9640 1 -- 172.21.15.88:0/2250717138 <== mon.2 v1:172.21.15.88:6790/0 8 ==== osd_map(1014..1014 src has 622..1014) v4 ==== 5229+0+0 (unknown 3230192161 0 0) 0x7fa520054000 con 0x7fa5300a87f0 2024-06-03T09:06:49.021 INFO:teuthology.orchestra.run.smithi088.stderr:2024-06-03T09:06:49.020+0000 7fa52cff9640 10 client.30215.objecter ms_dispatch 0x7fa530000b90 osd_map(1014..1014 src has 622..1014) v4 2024-06-03T09:06:49.021 INFO:teuthology.orchestra.run.smithi088.stderr:2024-06-03T09:06:49.020+0000 7fa52cff9640 3 client.30215.objecter handle_osd_map got epochs [1014,1014] > 0 2024-06-03T09:06:49.021 INFO:teuthology.orchestra.run.smithi088.stderr:2024-06-03T09:06:49.021+0000 7fa52cff9640 3 client.30215.objecter handle_osd_map decoding full epoch 1014 2024-06-03T09:06:49.021 INFO:teuthology.orchestra.run.smithi088.stderr:2024-06-03T09:06:49.021+0000 7fa52cff9640 20 client.30215.objecter dump_active .. 0 homeless 2024-06-03T09:06:49.256 INFO:teuthology.orchestra.run.smithi088.stderr:2024-06-03T09:06:49.256+0000 7fa52cff9640 1 -- 172.21.15.88:0/2250717138 <== mon.2 v1:172.21.15.88:6790/0 9 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+182498 (unknown 1092875540 0 3016760700) 0x7fa52004f9e0 con 0x7fa5300a87f0 2024-06-03T09:06:49.256 INFO:teuthology.orchestra.run.smithi088.stderr:2024-06-03T09:06:49.256+0000 7fa52cff9640 10 monclient: handle_mon_command_ack 1 [{"prefix": "get_command_descriptions"}] 2024-06-03T09:06:49.256 INFO:teuthology.orchestra.run.smithi088.stderr:2024-06-03T09:06:49.256+0000 7fa52cff9640 10 monclient: _finish_command 1 = system:0 2024-06-03T09:06:49.527 INFO:teuthology.orchestra.run.smithi088.stderr:2024-06-03T09:06:49.527+0000 7fa536711640 10 start_mon_command cmd=[{"prefix": "status", "format": "json"}] 2024-06-03T09:06:49.527 INFO:teuthology.orchestra.run.smithi088.stderr:2024-06-03T09:06:49.527+0000 7fa536711640 10 monclient: _send_command 2 [{"prefix": "status", "format": "json"}] 2024-06-03T09:06:49.528 INFO:teuthology.orchestra.run.smithi088.stderr:2024-06-03T09:06:49.527+0000 7fa536711640 10 monclient: _send_mon_message to mon.c at v1:172.21.15.88:6790/0 2024-06-03T09:06:49.528 INFO:teuthology.orchestra.run.smithi088.stderr:2024-06-03T09:06:49.527+0000 7fa536711640 1 -- 172.21.15.88:0/2250717138 --> v1:172.21.15.88:6790/0 -- mon_command({"prefix": "status", "format": "json"} v 0) v1 -- 0x7fa53013e010 con 0x7fa5300a87f0 2024-06-03T09:06:49.536 INFO:teuthology.orchestra.run.smithi088.stderr:2024-06-03T09:06:49.536+0000 7fa52cff9640 1 -- 172.21.15.88:0/2250717138 <== mon.2 v1:172.21.15.88:6790/0 10 ==== mon_command_ack([{"prefix": "status", "format": "json"}]=0 v0) v1 ==== 72+0+2019 (unknown 3226119948 0 1705449476) 0x7fa520051ae0 con 0x7fa5300a87f0 2024-06-03T09:06:49.536 INFO:teuthology.orchestra.run.smithi088.stderr:2024-06-03T09:06:49.536+0000 7fa52cff9640 10 monclient: handle_mon_command_ack 2 [{"prefix": "status", "format": "json"}] 2024-06-03T09:06:49.536 INFO:teuthology.orchestra.run.smithi088.stderr:2024-06-03T09:06:49.536+0000 7fa52cff9640 10 monclient: _finish_command 2 = system:0 2024-06-03T09:06:49.536 INFO:teuthology.orchestra.run.smithi088.stdout: 2024-06-03T09:06:49.537 INFO:teuthology.orchestra.run.smithi088.stdout:{"fsid":"ec3c3fdd-3ac6-4d7d-9412-683130ed8b04","health":{"status":"HEALTH_WARN","checks":{"OSD_SLOW_PING_TIME_BACK":{"severity":"HEALTH_WARN","summary":{"message":"Slow OSD heartbeats on back (longest 5696.311ms)","count":1},"muted":false},"OSD_SLOW_PING_TIME_FRONT":{"severity":"HEALTH_WARN","summary":{"message":"Slow OSD heartbeats on front (longest 5687.639ms)","count":1},"muted":false},"POOL_APP_NOT_ENABLED":{"severity":"HEALTH_WARN","summary":{"message":"1 pool(s) do not have an application enabled","count":1},"muted":false}},"mutes":[]},"election_epoch":10,"quorum":[0,1,2],"quorum_names":["a","b","c"],"quorum_age":1882,"monmap":{"epoch":1,"min_mon_release_name":"reef","num_mons":3},"osdmap":{"epoch":1014,"num_osds":8,"num_up_osds":8,"osd_up_since":1717405604,"num_in_osds":8,"osd_in_since":1717403732,"num_remapped_pgs":0},"pgmap":{"pgs_by_state":[{"state_name":"active+clean","count":36},{"state_name":"active+undersized","count":5}],"num_pgs":41,"num_pools":3,"num_objects":3,"data_bytes":459285,"bytes_used":506748928,"bytes_avail":772587364352,"bytes_total":773094113280},"fsmap":{"epoch":1,"by_rank":[],"up:standby":0},"mgrmap":{"available":true,"num_standbys":1,"modules":["iostat","nfs","restful"],"services":{}},"servicemap":{"epoch":25,"modified":"2024-06-03T09:06:37.101852+0000","services":{"osd":{"daemons":{"summary":"","0":{"start_epoch":0,"start_stamp":"0.000000","gid":0,"addr":"(unrecognized address family 0)/0","metadata":{},"task_status":{}},"1":{"start_epoch":0,"start_stamp":"0.000000","gid":0,"addr":"(unrecognized address family 0)/0","metadata":{},"task_status":{}},"3":{"start_epoch":0,"start_stamp":"0.000000","gid":0,"addr":"(unrecognized address family 0)/0","metadata":{},"task_status":{}},"4":{"start_epoch":0,"start_stamp":"0.000000","gid":0,"addr":"(unrecognized address family 0)/0","metadata":{},"task_status":{}},"5":{"start_epoch":0,"start_stamp":"0.000000","gid":0,"addr":"(unrecognized address family 0)/0","metadata":{},"task_status":{}}}}}},"progress_events":{}} 2024-06-03T09:06:49.539 INFO:teuthology.orchestra.run.smithi088.stderr:2024-06-03T09:06:49.538+0000 7fa536711640 10 librados: watch_flush enter 2024-06-03T09:06:49.539 INFO:teuthology.orchestra.run.smithi088.stderr:2024-06-03T09:06:49.538+0000 7fa536711640 10 librados: watch_flush exit 2024-06-03T09:06:49.539 INFO:teuthology.orchestra.run.smithi088.stderr:2024-06-03T09:06:49.538+0000 7fa536711640 20 client.30215.objecter shutdown clearing up homeless session... 2024-06-03T09:06:49.539 INFO:teuthology.orchestra.run.smithi088.stderr:2024-06-03T09:06:49.538+0000 7fa536711640 10 client.30215.objecter successfully canceled tick 2024-06-03T09:06:49.539 INFO:teuthology.orchestra.run.smithi088.stderr:2024-06-03T09:06:49.539+0000 7fa536711640 1 -- 172.21.15.88:0/2250717138 >> v1:172.21.15.192:6800/34229 conn(0x7fa51c06d440 legacy=0x7fa51c06f900 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-06-03T09:06:49.539 INFO:teuthology.orchestra.run.smithi088.stderr:2024-06-03T09:06:49.539+0000 7fa536711640 10 monclient: shutdown 2024-06-03T09:06:49.539 INFO:teuthology.orchestra.run.smithi088.stderr:2024-06-03T09:06:49.539+0000 7fa536711640 20 monclient: shutdown discarding 0 pending message(s) 2024-06-03T09:06:49.540 INFO:teuthology.orchestra.run.smithi088.stderr:2024-06-03T09:06:49.539+0000 7fa536711640 1 -- 172.21.15.88:0/2250717138 >> v1:172.21.15.88:6790/0 conn(0x7fa5300a87f0 legacy=0x7fa53012e370 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-06-03T09:06:49.540 INFO:teuthology.orchestra.run.smithi088.stderr:2024-06-03T09:06:49.539+0000 7fa536711640 1 -- 172.21.15.88:0/2250717138 shutdown_connections 2024-06-03T09:06:49.540 INFO:teuthology.orchestra.run.smithi088.stderr:2024-06-03T09:06:49.539+0000 7fa536711640 1 -- 172.21.15.88:0/2250717138 >> 172.21.15.88:0/2250717138 conn(0x7fa5300b71e0 msgr2=0x7fa5300b75c0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-06-03T09:06:49.540 INFO:teuthology.orchestra.run.smithi088.stderr:2024-06-03T09:06:49.539+0000 7fa536711640 1 -- 172.21.15.88:0/2250717138 shutdown_connections 2024-06-03T09:06:49.540 INFO:teuthology.orchestra.run.smithi088.stderr:2024-06-03T09:06:49.539+0000 7fa536711640 1 -- 172.21.15.88:0/2250717138 wait complete. 2024-06-03T09:06:49.540 INFO:teuthology.orchestra.run.smithi088.stderr:2024-06-03T09:06:49.539+0000 7fa536711640 1 librados: shutdown 2024-06-03T09:06:49.550 INFO:tasks.ceph.ceph_manager.ceph:no progress seen, keeping timeout for now 2024-06-03T09:06:49.550 DEBUG:teuthology.orchestra.run.smithi088:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph pg dump --format=json 2024-06-03T09:06:49.650 INFO:teuthology.orchestra.run.smithi088.stderr:2024-06-03T09:06:49.650+0000 7f24a2e69640 10 monclient: get_monmap_and_config 2024-06-03T09:06:49.650 INFO:teuthology.orchestra.run.smithi088.stderr:2024-06-03T09:06:49.650+0000 7f24a2e69640 10 monclient: build_initial_monmap 2024-06-03T09:06:49.650 INFO:teuthology.orchestra.run.smithi088.stderr:2024-06-03T09:06:49.650+0000 7f24a2e69640 10 monclient: monmap: 2024-06-03T09:06:49.650 INFO:teuthology.orchestra.run.smithi088.stderr:epoch 0 2024-06-03T09:06:49.650 INFO:teuthology.orchestra.run.smithi088.stderr:fsid ec3c3fdd-3ac6-4d7d-9412-683130ed8b04 2024-06-03T09:06:49.650 INFO:teuthology.orchestra.run.smithi088.stderr:last_changed 2024-06-03T09:06:49.651170+0000 2024-06-03T09:06:49.651 INFO:teuthology.orchestra.run.smithi088.stderr:created 2024-06-03T09:06:49.651170+0000 2024-06-03T09:06:49.651 INFO:teuthology.orchestra.run.smithi088.stderr:min_mon_release 0 (unknown) 2024-06-03T09:06:49.651 INFO:teuthology.orchestra.run.smithi088.stderr:election_strategy: 3 2024-06-03T09:06:49.651 INFO:teuthology.orchestra.run.smithi088.stderr:0: v1:172.21.15.88:6789/0 mon.noname-a 2024-06-03T09:06:49.651 INFO:teuthology.orchestra.run.smithi088.stderr:1: v1:172.21.15.192:6789/0 mon.noname-c 2024-06-03T09:06:49.651 INFO:teuthology.orchestra.run.smithi088.stderr:2: v1:172.21.15.88:6790/0 mon.noname-b 2024-06-03T09:06:49.651 INFO:teuthology.orchestra.run.smithi088.stderr: 2024-06-03T09:06:49.653 INFO:teuthology.orchestra.run.smithi088.stderr:2024-06-03T09:06:49.650+0000 7f24a2e69640 1 Processor -- start 2024-06-03T09:06:49.653 INFO:teuthology.orchestra.run.smithi088.stderr:2024-06-03T09:06:49.650+0000 7f24a2e69640 1 -- start start 2024-06-03T09:06:49.653 INFO:teuthology.orchestra.run.smithi088.stderr:2024-06-03T09:06:49.650+0000 7f24a2e69640 10 monclient: init 2024-06-03T09:06:49.653 INFO:teuthology.orchestra.run.smithi088.stderr:2024-06-03T09:06:49.651+0000 7f24a2e69640 10 monclient: _reopen_session rank -1 2024-06-03T09:06:49.653 INFO:teuthology.orchestra.run.smithi088.stderr:2024-06-03T09:06:49.651+0000 7f24a2e69640 10 monclient: _add_conns ranks=[0,2,1] 2024-06-03T09:06:49.653 INFO:teuthology.orchestra.run.smithi088.stderr:2024-06-03T09:06:49.651+0000 7f24a2e69640 10 monclient(hunting): picked mon.noname-a con 0x7f249c0a8960 addr v1:172.21.15.88:6789/0 2024-06-03T09:06:49.653 INFO:teuthology.orchestra.run.smithi088.stderr:2024-06-03T09:06:49.651+0000 7f24a2e69640 10 monclient(hunting): picked mon.noname-b con 0x7f249c1450f0 addr v1:172.21.15.88:6790/0 2024-06-03T09:06:49.653 INFO:teuthology.orchestra.run.smithi088.stderr:2024-06-03T09:06:49.651+0000 7f24a2e69640 10 monclient(hunting): picked mon.noname-c con 0x7f249c13eec0 addr v1:172.21.15.192:6789/0 2024-06-03T09:06:49.653 INFO:teuthology.orchestra.run.smithi088.stderr:2024-06-03T09:06:49.651+0000 7f24a2e69640 1 -- --> v1:172.21.15.88:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f249c144c60 con 0x7f249c0a8960 2024-06-03T09:06:49.654 INFO:teuthology.orchestra.run.smithi088.stderr:2024-06-03T09:06:49.651+0000 7f24a2e69640 1 -- --> v1:172.21.15.192:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f249c0b3930 con 0x7f249c13eec0 2024-06-03T09:06:49.654 INFO:teuthology.orchestra.run.smithi088.stderr:2024-06-03T09:06:49.651+0000 7f24a2e69640 1 -- --> v1:172.21.15.88:6790/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f249c0b1ad0 con 0x7f249c1450f0 2024-06-03T09:06:49.654 INFO:teuthology.orchestra.run.smithi088.stderr:2024-06-03T09:06:49.651+0000 7f24a2e69640 10 monclient(hunting): _renew_subs 2024-06-03T09:06:49.654 INFO:teuthology.orchestra.run.smithi088.stderr:2024-06-03T09:06:49.651+0000 7f24a2e69640 10 monclient(hunting): authenticate will time out at 3065.461603s 2024-06-03T09:06:49.654 INFO:teuthology.orchestra.run.smithi088.stderr:2024-06-03T09:06:49.652+0000 7f2493fff640 1 --1- >> v1:172.21.15.88:6790/0 conn(0x7f249c1450f0 0x7f249c13e7b0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.88:6790/0 says I am v1:172.21.15.88:52354/0 (socket says 172.21.15.88:52354) 2024-06-03T09:06:49.654 INFO:teuthology.orchestra.run.smithi088.stderr:2024-06-03T09:06:49.652+0000 7f2493fff640 1 -- 172.21.15.88:0/3220038639 learned_addr learned my addr 172.21.15.88:0/3220038639 (peer_addr_for_me v1:172.21.15.88:0/0) 2024-06-03T09:06:49.655 INFO:teuthology.orchestra.run.smithi088.stderr:2024-06-03T09:06:49.654+0000 7f24937fe640 1 -- 172.21.15.88:0/3220038639 <== mon.2 v1:172.21.15.88:6790/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 2904438159 0 0) 0x7f249c0b1ad0 con 0x7f249c1450f0 2024-06-03T09:06:49.655 INFO:teuthology.orchestra.run.smithi088.stderr:2024-06-03T09:06:49.654+0000 7f24937fe640 10 monclient(hunting): _negotiate 2024-06-03T09:06:49.655 INFO:teuthology.orchestra.run.smithi088.stderr:2024-06-03T09:06:49.654+0000 7f24937fe640 10 monclient(hunting): _init_auth method 2 2024-06-03T09:06:49.655 INFO:teuthology.orchestra.run.smithi088.stderr:2024-06-03T09:06:49.654+0000 7f24937fe640 10 monclient(hunting): _init_auth creating new auth 2024-06-03T09:06:49.655 INFO:teuthology.orchestra.run.smithi088.stderr:2024-06-03T09:06:49.654+0000 7f24937fe640 10 monclient(hunting): my global_id is 30218 2024-06-03T09:06:49.655 INFO:teuthology.orchestra.run.smithi088.stderr:2024-06-03T09:06:49.654+0000 7f24937fe640 1 -- 172.21.15.88:0/3220038639 --> v1:172.21.15.88:6790/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f2484003520 con 0x7f249c1450f0 2024-06-03T09:06:49.656 INFO:teuthology.orchestra.run.smithi088.stderr:2024-06-03T09:06:49.654+0000 7f24937fe640 1 -- 172.21.15.88:0/3220038639 <== mon.1 v1:172.21.15.192:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 1269364403 0 0) 0x7f249c0b3930 con 0x7f249c13eec0 2024-06-03T09:06:49.656 INFO:teuthology.orchestra.run.smithi088.stderr:2024-06-03T09:06:49.654+0000 7f24937fe640 10 monclient(hunting): _negotiate 2024-06-03T09:06:49.656 INFO:teuthology.orchestra.run.smithi088.stderr:2024-06-03T09:06:49.654+0000 7f24937fe640 10 monclient(hunting): _init_auth method 2 2024-06-03T09:06:49.656 INFO:teuthology.orchestra.run.smithi088.stderr:2024-06-03T09:06:49.654+0000 7f24937fe640 10 monclient(hunting): _init_auth creating new auth 2024-06-03T09:06:49.656 INFO:teuthology.orchestra.run.smithi088.stderr:2024-06-03T09:06:49.654+0000 7f24937fe640 10 monclient(hunting): my global_id is 30289 2024-06-03T09:06:49.656 INFO:teuthology.orchestra.run.smithi088.stderr:2024-06-03T09:06:49.654+0000 7f24937fe640 1 -- 172.21.15.88:0/3220038639 --> v1:172.21.15.192:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f249c0b1ad0 con 0x7f249c13eec0 2024-06-03T09:06:49.656 INFO:teuthology.orchestra.run.smithi088.stderr:2024-06-03T09:06:49.654+0000 7f24937fe640 1 -- 172.21.15.88:0/3220038639 <== mon.0 v1:172.21.15.88:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 205584596 0 0) 0x7f249c144c60 con 0x7f249c0a8960 2024-06-03T09:06:49.656 INFO:teuthology.orchestra.run.smithi088.stderr:2024-06-03T09:06:49.654+0000 7f24937fe640 10 monclient(hunting): _negotiate 2024-06-03T09:06:49.656 INFO:teuthology.orchestra.run.smithi088.stderr:2024-06-03T09:06:49.654+0000 7f24937fe640 10 monclient(hunting): _init_auth method 2 2024-06-03T09:06:49.656 INFO:teuthology.orchestra.run.smithi088.stderr:2024-06-03T09:06:49.654+0000 7f24937fe640 10 monclient(hunting): _init_auth creating new auth 2024-06-03T09:06:49.656 INFO:teuthology.orchestra.run.smithi088.stderr:2024-06-03T09:06:49.654+0000 7f24937fe640 10 monclient(hunting): my global_id is 30714 2024-06-03T09:06:49.656 INFO:teuthology.orchestra.run.smithi088.stderr:2024-06-03T09:06:49.654+0000 7f24937fe640 1 -- 172.21.15.88:0/3220038639 --> v1:172.21.15.88:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f249c0b3930 con 0x7f249c0a8960 2024-06-03T09:06:49.658 INFO:teuthology.orchestra.run.smithi088.stderr:2024-06-03T09:06:49.657+0000 7f24937fe640 1 -- 172.21.15.88:0/3220038639 <== mon.2 v1:172.21.15.88:6790/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 214+0+0 (unknown 3670052782 0 0) 0x7f2484003520 con 0x7f249c1450f0 2024-06-03T09:06:49.658 INFO:teuthology.orchestra.run.smithi088.stderr:2024-06-03T09:06:49.657+0000 7f24937fe640 1 -- 172.21.15.88:0/3220038639 >> v1:172.21.15.192:6789/0 conn(0x7f249c13eec0 legacy=0x7f249c14ce60 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-06-03T09:06:49.658 INFO:teuthology.orchestra.run.smithi088.stderr:2024-06-03T09:06:49.657+0000 7f24937fe640 1 -- 172.21.15.88:0/3220038639 >> v1:172.21.15.88:6789/0 conn(0x7f249c0a8960 legacy=0x7f249c0a8d40 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-06-03T09:06:49.658 INFO:teuthology.orchestra.run.smithi088.stderr:2024-06-03T09:06:49.657+0000 7f24937fe640 10 monclient: _finish_hunting 0 2024-06-03T09:06:49.658 INFO:teuthology.orchestra.run.smithi088.stderr:2024-06-03T09:06:49.657+0000 7f24937fe640 1 monclient: found mon.noname-b 2024-06-03T09:06:49.658 INFO:teuthology.orchestra.run.smithi088.stderr:2024-06-03T09:06:49.657+0000 7f24937fe640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-06-03T09:06:49.658 INFO:teuthology.orchestra.run.smithi088.stderr:2024-06-03T09:06:49.657+0000 7f24937fe640 10 monclient: _send_mon_message to mon.noname-b at v1:172.21.15.88:6790/0 2024-06-03T09:06:49.658 INFO:teuthology.orchestra.run.smithi088.stderr:2024-06-03T09:06:49.657+0000 7f24937fe640 1 -- 172.21.15.88:0/3220038639 --> v1:172.21.15.88:6790/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f249c0b1c60 con 0x7f249c1450f0 2024-06-03T09:06:49.658 INFO:teuthology.orchestra.run.smithi088.stderr:2024-06-03T09:06:49.657+0000 7f24937fe640 10 monclient: _finish_auth 0 2024-06-03T09:06:49.658 INFO:teuthology.orchestra.run.smithi088.stderr:2024-06-03T09:06:49.657+0000 7f24937fe640 10 monclient: _check_auth_tickets 2024-06-03T09:06:49.659 INFO:teuthology.orchestra.run.smithi088.stderr:2024-06-03T09:06:49.657+0000 7f24937fe640 20 monclient: _check_auth_rotating not needed by client.admin 2024-06-03T09:06:49.659 INFO:teuthology.orchestra.run.smithi088.stderr:2024-06-03T09:06:49.657+0000 7f24937fe640 1 -- 172.21.15.88:0/3220038639 <== mon.2 v1:172.21.15.88:6790/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 2234374065 0 0) 0x7f248c001610 con 0x7f249c1450f0 2024-06-03T09:06:49.659 INFO:teuthology.orchestra.run.smithi088.stderr:2024-06-03T09:06:49.657+0000 7f24937fe640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-06-03T09:06:49.659 INFO:teuthology.orchestra.run.smithi088.stderr:2024-06-03T09:06:49.657+0000 7f24937fe640 10 monclient: got monmap 1 from mon.noname-b (according to old e1) 2024-06-03T09:06:49.659 INFO:teuthology.orchestra.run.smithi088.stderr:2024-06-03T09:06:49.657+0000 7f24937fe640 10 monclient: dump: 2024-06-03T09:06:49.659 INFO:teuthology.orchestra.run.smithi088.stderr:epoch 1 2024-06-03T09:06:49.659 INFO:teuthology.orchestra.run.smithi088.stderr:fsid ec3c3fdd-3ac6-4d7d-9412-683130ed8b04 2024-06-03T09:06:49.659 INFO:teuthology.orchestra.run.smithi088.stderr:last_changed 2024-06-03T08:34:31.885125+0000 2024-06-03T09:06:49.659 INFO:teuthology.orchestra.run.smithi088.stderr:created 2024-06-03T08:34:31.885125+0000 2024-06-03T09:06:49.659 INFO:teuthology.orchestra.run.smithi088.stderr:min_mon_release 18 (reef) 2024-06-03T09:06:49.659 INFO:teuthology.orchestra.run.smithi088.stderr:election_strategy: 3 2024-06-03T09:06:49.660 INFO:teuthology.orchestra.run.smithi088.stderr:0: v1:172.21.15.88:6789/0 mon.a 2024-06-03T09:06:49.660 INFO:teuthology.orchestra.run.smithi088.stderr:1: v1:172.21.15.192:6789/0 mon.b 2024-06-03T09:06:49.660 INFO:teuthology.orchestra.run.smithi088.stderr:2: v1:172.21.15.88:6790/0 mon.c 2024-06-03T09:06:49.660 INFO:teuthology.orchestra.run.smithi088.stderr: 2024-06-03T09:06:49.660 INFO:teuthology.orchestra.run.smithi088.stderr:2024-06-03T09:06:49.657+0000 7f24a2e69640 5 monclient: authenticate success, global_id 30218 2024-06-03T09:06:49.660 INFO:teuthology.orchestra.run.smithi088.stderr:2024-06-03T09:06:49.657+0000 7f24a2e69640 20 monclient: get_monmap_and_config waiting for monmap|config 2024-06-03T09:06:49.663 INFO:teuthology.orchestra.run.smithi088.stderr:2024-06-03T09:06:49.659+0000 7f24937fe640 1 -- 172.21.15.88:0/3220038639 <== mon.2 v1:172.21.15.88:6790/0 4 ==== config(0 keys) v1 ==== 4+0+0 (unknown 0 0 0) 0x7f248c001830 con 0x7f249c1450f0 2024-06-03T09:06:49.663 INFO:teuthology.orchestra.run.smithi088.stderr:2024-06-03T09:06:49.659+0000 7f24937fe640 10 monclient: handle_config config(0 keys) v1 2024-06-03T09:06:49.663 INFO:teuthology.orchestra.run.smithi088.stderr:2024-06-03T09:06:49.659+0000 7f24937fe640 1 -- 172.21.15.88:0/3220038639 <== mon.2 v1:172.21.15.88:6790/0 5 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 2234374065 0 0) 0x7f248c001b80 con 0x7f249c1450f0 2024-06-03T09:06:49.663 INFO:teuthology.orchestra.run.smithi088.stderr:2024-06-03T09:06:49.659+0000 7f24a2e69640 10 monclient: get_monmap_and_config success 2024-06-03T09:06:49.663 INFO:teuthology.orchestra.run.smithi088.stderr:2024-06-03T09:06:49.659+0000 7f24a2e69640 4 set_mon_vals no callback set 2024-06-03T09:06:49.663 INFO:teuthology.orchestra.run.smithi088.stderr:2024-06-03T09:06:49.659+0000 7f24937fe640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-06-03T09:06:49.663 INFO:teuthology.orchestra.run.smithi088.stderr:2024-06-03T09:06:49.659+0000 7f24937fe640 10 monclient: got monmap 1 from mon.c (according to old e1) 2024-06-03T09:06:49.663 INFO:teuthology.orchestra.run.smithi088.stderr:2024-06-03T09:06:49.659+0000 7f24937fe640 10 monclient: dump: 2024-06-03T09:06:49.664 INFO:teuthology.orchestra.run.smithi088.stderr:epoch 1 2024-06-03T09:06:49.664 INFO:teuthology.orchestra.run.smithi088.stderr:fsid ec3c3fdd-3ac6-4d7d-9412-683130ed8b04 2024-06-03T09:06:49.664 INFO:teuthology.orchestra.run.smithi088.stderr:last_changed 2024-06-03T08:34:31.885125+0000 2024-06-03T09:06:49.664 INFO:teuthology.orchestra.run.smithi088.stderr:created 2024-06-03T08:34:31.885125+0000 2024-06-03T09:06:49.664 INFO:teuthology.orchestra.run.smithi088.stderr:min_mon_release 18 (reef) 2024-06-03T09:06:49.664 INFO:teuthology.orchestra.run.smithi088.stderr:election_strategy: 3 2024-06-03T09:06:49.664 INFO:teuthology.orchestra.run.smithi088.stderr:0: v1:172.21.15.88:6789/0 mon.a 2024-06-03T09:06:49.664 INFO:teuthology.orchestra.run.smithi088.stderr:1: v1:172.21.15.192:6789/0 mon.b 2024-06-03T09:06:49.664 INFO:teuthology.orchestra.run.smithi088.stderr:2: v1:172.21.15.88:6790/0 mon.c 2024-06-03T09:06:49.664 INFO:teuthology.orchestra.run.smithi088.stderr: 2024-06-03T09:06:49.664 INFO:teuthology.orchestra.run.smithi088.stderr:2024-06-03T09:06:49.660+0000 7f24a2e69640 10 monclient: shutdown 2024-06-03T09:06:49.664 INFO:teuthology.orchestra.run.smithi088.stderr:2024-06-03T09:06:49.660+0000 7f24a2e69640 20 monclient: shutdown discarding 0 pending message(s) 2024-06-03T09:06:49.665 INFO:teuthology.orchestra.run.smithi088.stderr:2024-06-03T09:06:49.660+0000 7f24a2e69640 1 -- 172.21.15.88:0/3220038639 >> v1:172.21.15.88:6790/0 conn(0x7f249c1450f0 legacy=0x7f249c13e7b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-06-03T09:06:49.665 INFO:teuthology.orchestra.run.smithi088.stderr:2024-06-03T09:06:49.660+0000 7f24a2e69640 1 -- 172.21.15.88:0/3220038639 shutdown_connections 2024-06-03T09:06:49.665 INFO:teuthology.orchestra.run.smithi088.stderr:2024-06-03T09:06:49.660+0000 7f24a2e69640 1 -- 172.21.15.88:0/3220038639 >> 172.21.15.88:0/3220038639 conn(0x7f249c0b70d0 msgr2=0x7f249c0b74e0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-06-03T09:06:49.665 INFO:teuthology.orchestra.run.smithi088.stderr:2024-06-03T09:06:49.660+0000 7f24a2e69640 1 -- 172.21.15.88:0/3220038639 shutdown_connections 2024-06-03T09:06:49.665 INFO:teuthology.orchestra.run.smithi088.stderr:2024-06-03T09:06:49.660+0000 7f24a2e69640 1 -- 172.21.15.88:0/3220038639 wait complete. 2024-06-03T09:06:49.665 INFO:teuthology.orchestra.run.smithi088.stderr:2024-06-03T09:06:49.661+0000 7f24a2e69640 10 monclient: build_initial_monmap 2024-06-03T09:06:49.665 INFO:teuthology.orchestra.run.smithi088.stderr:2024-06-03T09:06:49.661+0000 7f24a2e69640 10 monclient: monmap: 2024-06-03T09:06:49.665 INFO:teuthology.orchestra.run.smithi088.stderr:epoch 0 2024-06-03T09:06:49.665 INFO:teuthology.orchestra.run.smithi088.stderr:fsid 00000000-0000-0000-0000-000000000000 2024-06-03T09:06:49.665 INFO:teuthology.orchestra.run.smithi088.stderr:last_changed 0.000000 2024-06-03T09:06:49.665 INFO:teuthology.orchestra.run.smithi088.stderr:created 0.000000 2024-06-03T09:06:49.665 INFO:teuthology.orchestra.run.smithi088.stderr:min_mon_release 0 (unknown) 2024-06-03T09:06:49.666 INFO:teuthology.orchestra.run.smithi088.stderr:election_strategy: 1 2024-06-03T09:06:49.666 INFO:teuthology.orchestra.run.smithi088.stderr:0: v1:172.21.15.88:6789/0 mon.noname-a 2024-06-03T09:06:49.666 INFO:teuthology.orchestra.run.smithi088.stderr:1: v1:172.21.15.192:6789/0 mon.noname-b 2024-06-03T09:06:49.666 INFO:teuthology.orchestra.run.smithi088.stderr:2: v1:172.21.15.88:6790/0 mon.noname-c 2024-06-03T09:06:49.666 INFO:teuthology.orchestra.run.smithi088.stderr: 2024-06-03T09:06:49.666 INFO:teuthology.orchestra.run.smithi088.stderr:2024-06-03T09:06:49.661+0000 7f24a2e69640 1 librados: starting msgr at 2024-06-03T09:06:49.666 INFO:teuthology.orchestra.run.smithi088.stderr:2024-06-03T09:06:49.661+0000 7f24a2e69640 1 librados: starting objecter 2024-06-03T09:06:49.666 INFO:teuthology.orchestra.run.smithi088.stderr:2024-06-03T09:06:49.661+0000 7f24a2e69640 1 Processor -- start 2024-06-03T09:06:49.666 INFO:teuthology.orchestra.run.smithi088.stderr:2024-06-03T09:06:49.661+0000 7f24a2e69640 1 -- start start 2024-06-03T09:06:49.666 INFO:teuthology.orchestra.run.smithi088.stderr:2024-06-03T09:06:49.661+0000 7f24a2e69640 1 librados: setting wanted keys 2024-06-03T09:06:49.666 INFO:teuthology.orchestra.run.smithi088.stderr:2024-06-03T09:06:49.661+0000 7f24a2e69640 1 librados: calling monclient init 2024-06-03T09:06:49.666 INFO:teuthology.orchestra.run.smithi088.stderr:2024-06-03T09:06:49.661+0000 7f24a2e69640 10 monclient: init 2024-06-03T09:06:49.667 INFO:teuthology.orchestra.run.smithi088.stderr:2024-06-03T09:06:49.662+0000 7f24a2e69640 10 monclient: _reopen_session rank -1 2024-06-03T09:06:49.667 INFO:teuthology.orchestra.run.smithi088.stderr:2024-06-03T09:06:49.662+0000 7f24a2e69640 10 monclient: _add_conns ranks=[2,1,0] 2024-06-03T09:06:49.667 INFO:teuthology.orchestra.run.smithi088.stderr:2024-06-03T09:06:49.662+0000 7f24a2e69640 10 monclient(hunting): picked mon.noname-c con 0x7f249c0a8960 addr v1:172.21.15.88:6790/0 2024-06-03T09:06:49.667 INFO:teuthology.orchestra.run.smithi088.stderr:2024-06-03T09:06:49.662+0000 7f24a2e69640 10 monclient(hunting): picked mon.noname-b con 0x7f249c13eec0 addr v1:172.21.15.192:6789/0 2024-06-03T09:06:49.667 INFO:teuthology.orchestra.run.smithi088.stderr:2024-06-03T09:06:49.662+0000 7f24a2e69640 10 monclient(hunting): picked mon.noname-a con 0x7f249c1450f0 addr v1:172.21.15.88:6789/0 2024-06-03T09:06:49.667 INFO:teuthology.orchestra.run.smithi088.stderr:2024-06-03T09:06:49.662+0000 7f24a2e69640 1 -- --> v1:172.21.15.88:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f249c0a9660 con 0x7f249c1450f0 2024-06-03T09:06:49.667 INFO:teuthology.orchestra.run.smithi088.stderr:2024-06-03T09:06:49.662+0000 7f24a2e69640 1 -- --> v1:172.21.15.192:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f249c140aa0 con 0x7f249c13eec0 2024-06-03T09:06:49.667 INFO:teuthology.orchestra.run.smithi088.stderr:2024-06-03T09:06:49.662+0000 7f24a2e69640 1 -- --> v1:172.21.15.88:6790/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f249c143e10 con 0x7f249c0a8960 2024-06-03T09:06:49.667 INFO:teuthology.orchestra.run.smithi088.stderr:2024-06-03T09:06:49.662+0000 7f24a2e69640 10 monclient(hunting): _renew_subs 2024-06-03T09:06:49.667 INFO:teuthology.orchestra.run.smithi088.stderr:2024-06-03T09:06:49.662+0000 7f24a2e69640 10 monclient(hunting): authenticate will time out at 3065.472180s 2024-06-03T09:06:49.667 INFO:teuthology.orchestra.run.smithi088.stderr:2024-06-03T09:06:49.662+0000 7f24a0c05640 1 --1- >> v1:172.21.15.88:6790/0 conn(0x7f249c0a8960 0x7f249c1da390 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=1).handle_server_banner_and_identify peer v1:172.21.15.88:6790/0 says I am v1:172.21.15.88:52366/0 (socket says 172.21.15.88:52366) 2024-06-03T09:06:49.667 INFO:teuthology.orchestra.run.smithi088.stderr:2024-06-03T09:06:49.662+0000 7f24a0c05640 1 -- 172.21.15.88:0/1698317358 learned_addr learned my addr 172.21.15.88:0/1698317358 (peer_addr_for_me v1:172.21.15.88:0/0) 2024-06-03T09:06:49.668 INFO:teuthology.orchestra.run.smithi088.stderr:2024-06-03T09:06:49.662+0000 7f24917fa640 10 client.?.objecter ms_handle_connect 0x7f249c0a8960 2024-06-03T09:06:49.668 INFO:teuthology.orchestra.run.smithi088.stderr:2024-06-03T09:06:49.662+0000 7f24917fa640 10 client.?.objecter resend_mon_ops 2024-06-03T09:06:49.668 INFO:teuthology.orchestra.run.smithi088.stderr:2024-06-03T09:06:49.663+0000 7f24917fa640 10 client.?.objecter ms_handle_connect 0x7f249c1450f0 2024-06-03T09:06:49.668 INFO:teuthology.orchestra.run.smithi088.stderr:2024-06-03T09:06:49.663+0000 7f24917fa640 10 client.?.objecter resend_mon_ops 2024-06-03T09:06:49.668 INFO:teuthology.orchestra.run.smithi088.stderr:2024-06-03T09:06:49.663+0000 7f24917fa640 10 client.?.objecter ms_handle_connect 0x7f249c13eec0 2024-06-03T09:06:49.668 INFO:teuthology.orchestra.run.smithi088.stderr:2024-06-03T09:06:49.663+0000 7f24917fa640 10 client.?.objecter resend_mon_ops 2024-06-03T09:06:49.669 INFO:teuthology.orchestra.run.smithi088.stderr:2024-06-03T09:06:49.665+0000 7f24917fa640 1 -- 172.21.15.88:0/1698317358 <== mon.2 v1:172.21.15.88:6790/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 2087469209 0 0) 0x7f249c143e10 con 0x7f249c0a8960 2024-06-03T09:06:49.669 INFO:teuthology.orchestra.run.smithi088.stderr:2024-06-03T09:06:49.665+0000 7f24917fa640 10 monclient(hunting): _negotiate 2024-06-03T09:06:49.669 INFO:teuthology.orchestra.run.smithi088.stderr:2024-06-03T09:06:49.665+0000 7f24917fa640 10 monclient(hunting): _init_auth method 2 2024-06-03T09:06:49.669 INFO:teuthology.orchestra.run.smithi088.stderr:2024-06-03T09:06:49.665+0000 7f24917fa640 10 monclient(hunting): _init_auth creating new auth 2024-06-03T09:06:49.669 INFO:teuthology.orchestra.run.smithi088.stderr:2024-06-03T09:06:49.665+0000 7f24917fa640 10 monclient(hunting): my global_id is 30221 2024-06-03T09:06:49.669 INFO:teuthology.orchestra.run.smithi088.stderr:2024-06-03T09:06:49.665+0000 7f24917fa640 1 -- 172.21.15.88:0/1698317358 --> v1:172.21.15.88:6790/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f2488004eb0 con 0x7f249c0a8960 2024-06-03T09:06:49.669 INFO:teuthology.orchestra.run.smithi088.stderr:2024-06-03T09:06:49.665+0000 7f24917fa640 1 -- 172.21.15.88:0/1698317358 <== mon.1 v1:172.21.15.192:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 1929039199 0 0) 0x7f249c140aa0 con 0x7f249c13eec0 2024-06-03T09:06:49.670 INFO:teuthology.orchestra.run.smithi088.stderr:2024-06-03T09:06:49.665+0000 7f24917fa640 10 monclient(hunting): _negotiate 2024-06-03T09:06:49.670 INFO:teuthology.orchestra.run.smithi088.stderr:2024-06-03T09:06:49.665+0000 7f24917fa640 10 monclient(hunting): _init_auth method 2 2024-06-03T09:06:49.670 INFO:teuthology.orchestra.run.smithi088.stderr:2024-06-03T09:06:49.665+0000 7f24917fa640 10 monclient(hunting): _init_auth creating new auth 2024-06-03T09:06:49.670 INFO:teuthology.orchestra.run.smithi088.stderr:2024-06-03T09:06:49.665+0000 7f24917fa640 10 monclient(hunting): my global_id is 30292 2024-06-03T09:06:49.670 INFO:teuthology.orchestra.run.smithi088.stderr:2024-06-03T09:06:49.665+0000 7f24917fa640 1 -- 172.21.15.88:0/1698317358 --> v1:172.21.15.192:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f249c143e10 con 0x7f249c13eec0 2024-06-03T09:06:49.670 INFO:teuthology.orchestra.run.smithi088.stderr:2024-06-03T09:06:49.665+0000 7f24917fa640 1 -- 172.21.15.88:0/1698317358 <== mon.0 v1:172.21.15.88:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 3826829188 0 0) 0x7f249c0a9660 con 0x7f249c1450f0 2024-06-03T09:06:49.670 INFO:teuthology.orchestra.run.smithi088.stderr:2024-06-03T09:06:49.665+0000 7f24917fa640 10 monclient(hunting): _negotiate 2024-06-03T09:06:49.670 INFO:teuthology.orchestra.run.smithi088.stderr:2024-06-03T09:06:49.665+0000 7f24917fa640 10 monclient(hunting): _init_auth method 2 2024-06-03T09:06:49.670 INFO:teuthology.orchestra.run.smithi088.stderr:2024-06-03T09:06:49.665+0000 7f24917fa640 10 monclient(hunting): _init_auth creating new auth 2024-06-03T09:06:49.670 INFO:teuthology.orchestra.run.smithi088.stderr:2024-06-03T09:06:49.665+0000 7f24917fa640 10 monclient(hunting): my global_id is 30717 2024-06-03T09:06:49.671 INFO:teuthology.orchestra.run.smithi088.stderr:2024-06-03T09:06:49.665+0000 7f24917fa640 1 -- 172.21.15.88:0/1698317358 --> v1:172.21.15.88:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f249c140aa0 con 0x7f249c1450f0 2024-06-03T09:06:49.672 INFO:teuthology.orchestra.run.smithi088.stderr:2024-06-03T09:06:49.669+0000 7f24917fa640 1 -- 172.21.15.88:0/1698317358 <== mon.1 v1:172.21.15.192:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 764+0+0 (unknown 1729163604 0 0) 0x7f249c143e10 con 0x7f249c13eec0 2024-06-03T09:06:49.672 INFO:teuthology.orchestra.run.smithi088.stderr:2024-06-03T09:06:49.669+0000 7f24917fa640 1 -- 172.21.15.88:0/1698317358 --> v1:172.21.15.192:6789/0 -- auth(proto 2 165 bytes epoch 0) v1 -- 0x7f249c0a9660 con 0x7f249c13eec0 2024-06-03T09:06:49.672 INFO:teuthology.orchestra.run.smithi088.stderr:2024-06-03T09:06:49.669+0000 7f24917fa640 1 -- 172.21.15.88:0/1698317358 <== mon.1 v1:172.21.15.192:6789/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 2234374065 0 0) 0x7f248c001b40 con 0x7f249c13eec0 2024-06-03T09:06:49.672 INFO:teuthology.orchestra.run.smithi088.stderr:2024-06-03T09:06:49.669+0000 7f24917fa640 10 monclient(hunting): handle_monmap mon_map magic: 0 v1 2024-06-03T09:06:49.672 INFO:teuthology.orchestra.run.smithi088.stderr:2024-06-03T09:06:49.669+0000 7f24917fa640 10 monclient(hunting): got monmap 1 from mon.noname-b (according to old e1) 2024-06-03T09:06:49.672 INFO:teuthology.orchestra.run.smithi088.stderr:2024-06-03T09:06:49.669+0000 7f24917fa640 10 monclient(hunting): dump: 2024-06-03T09:06:49.672 INFO:teuthology.orchestra.run.smithi088.stderr:epoch 1 2024-06-03T09:06:49.672 INFO:teuthology.orchestra.run.smithi088.stderr:fsid ec3c3fdd-3ac6-4d7d-9412-683130ed8b04 2024-06-03T09:06:49.673 INFO:teuthology.orchestra.run.smithi088.stderr:last_changed 2024-06-03T08:34:31.885125+0000 2024-06-03T09:06:49.673 INFO:teuthology.orchestra.run.smithi088.stderr:created 2024-06-03T08:34:31.885125+0000 2024-06-03T09:06:49.673 INFO:teuthology.orchestra.run.smithi088.stderr:min_mon_release 18 (reef) 2024-06-03T09:06:49.673 INFO:teuthology.orchestra.run.smithi088.stderr:election_strategy: 3 2024-06-03T09:06:49.673 INFO:teuthology.orchestra.run.smithi088.stderr:0: v1:172.21.15.88:6789/0 mon.a 2024-06-03T09:06:49.673 INFO:teuthology.orchestra.run.smithi088.stderr:1: v1:172.21.15.192:6789/0 mon.b 2024-06-03T09:06:49.673 INFO:teuthology.orchestra.run.smithi088.stderr:2: v1:172.21.15.88:6790/0 mon.c 2024-06-03T09:06:49.673 INFO:teuthology.orchestra.run.smithi088.stderr: 2024-06-03T09:06:49.673 INFO:teuthology.orchestra.run.smithi088.stderr:2024-06-03T09:06:49.669+0000 7f24917fa640 10 monclient(hunting): _finish_auth 0 2024-06-03T09:06:49.673 INFO:teuthology.orchestra.run.smithi088.stderr:2024-06-03T09:06:49.671+0000 7f24917fa640 1 -- 172.21.15.88:0/1698317358 <== mon.2 v1:172.21.15.88:6790/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 764+0+0 (unknown 1185766618 0 0) 0x7f2488004eb0 con 0x7f249c0a8960 2024-06-03T09:06:49.673 INFO:teuthology.orchestra.run.smithi088.stderr:2024-06-03T09:06:49.671+0000 7f24917fa640 1 -- 172.21.15.88:0/1698317358 --> v1:172.21.15.88:6790/0 -- auth(proto 2 165 bytes epoch 0) v1 -- 0x7f249c143e10 con 0x7f249c0a8960 2024-06-03T09:06:49.673 INFO:teuthology.orchestra.run.smithi088.stderr:2024-06-03T09:06:49.671+0000 7f24917fa640 1 -- 172.21.15.88:0/1698317358 <== mon.2 v1:172.21.15.88:6790/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 2234374065 0 0) 0x7f24980019d0 con 0x7f249c0a8960 2024-06-03T09:06:49.674 INFO:teuthology.orchestra.run.smithi088.stderr:2024-06-03T09:06:49.671+0000 7f24917fa640 10 monclient(hunting): handle_monmap mon_map magic: 0 v1 2024-06-03T09:06:49.674 INFO:teuthology.orchestra.run.smithi088.stderr:2024-06-03T09:06:49.671+0000 7f24917fa640 10 monclient(hunting): got monmap 1 from mon.c (according to old e1) 2024-06-03T09:06:49.674 INFO:teuthology.orchestra.run.smithi088.stderr:2024-06-03T09:06:49.671+0000 7f24917fa640 10 monclient(hunting): dump: 2024-06-03T09:06:49.674 INFO:teuthology.orchestra.run.smithi088.stderr:epoch 1 2024-06-03T09:06:49.674 INFO:teuthology.orchestra.run.smithi088.stderr:fsid ec3c3fdd-3ac6-4d7d-9412-683130ed8b04 2024-06-03T09:06:49.674 INFO:teuthology.orchestra.run.smithi088.stderr:last_changed 2024-06-03T08:34:31.885125+0000 2024-06-03T09:06:49.674 INFO:teuthology.orchestra.run.smithi088.stderr:created 2024-06-03T08:34:31.885125+0000 2024-06-03T09:06:49.674 INFO:teuthology.orchestra.run.smithi088.stderr:min_mon_release 18 (reef) 2024-06-03T09:06:49.674 INFO:teuthology.orchestra.run.smithi088.stderr:election_strategy: 3 2024-06-03T09:06:49.674 INFO:teuthology.orchestra.run.smithi088.stderr:0: v1:172.21.15.88:6789/0 mon.a 2024-06-03T09:06:49.674 INFO:teuthology.orchestra.run.smithi088.stderr:1: v1:172.21.15.192:6789/0 mon.b 2024-06-03T09:06:49.674 INFO:teuthology.orchestra.run.smithi088.stderr:2: v1:172.21.15.88:6790/0 mon.c 2024-06-03T09:06:49.675 INFO:teuthology.orchestra.run.smithi088.stderr: 2024-06-03T09:06:49.675 INFO:teuthology.orchestra.run.smithi088.stderr:2024-06-03T09:06:49.672+0000 7f24917fa640 1 -- 172.21.15.88:0/1698317358 <== mon.0 v1:172.21.15.88:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 764+0+0 (unknown 709257484 0 0) 0x7f249c140aa0 con 0x7f249c1450f0 2024-06-03T09:06:49.675 INFO:teuthology.orchestra.run.smithi088.stderr:2024-06-03T09:06:49.672+0000 7f24917fa640 1 -- 172.21.15.88:0/1698317358 --> v1:172.21.15.88:6789/0 -- auth(proto 2 165 bytes epoch 0) v1 -- 0x7f2488004eb0 con 0x7f249c1450f0 2024-06-03T09:06:49.675 INFO:teuthology.orchestra.run.smithi088.stderr:2024-06-03T09:06:49.672+0000 7f24917fa640 1 -- 172.21.15.88:0/1698317358 <== mon.0 v1:172.21.15.88:6789/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 2234374065 0 0) 0x7f24940019b0 con 0x7f249c1450f0 2024-06-03T09:06:49.675 INFO:teuthology.orchestra.run.smithi088.stderr:2024-06-03T09:06:49.672+0000 7f24917fa640 10 monclient(hunting): handle_monmap mon_map magic: 0 v1 2024-06-03T09:06:49.675 INFO:teuthology.orchestra.run.smithi088.stderr:2024-06-03T09:06:49.672+0000 7f24917fa640 10 monclient(hunting): got monmap 1 from mon.a (according to old e1) 2024-06-03T09:06:49.675 INFO:teuthology.orchestra.run.smithi088.stderr:2024-06-03T09:06:49.672+0000 7f24917fa640 10 monclient(hunting): dump: 2024-06-03T09:06:49.675 INFO:teuthology.orchestra.run.smithi088.stderr:epoch 1 2024-06-03T09:06:49.675 INFO:teuthology.orchestra.run.smithi088.stderr:fsid ec3c3fdd-3ac6-4d7d-9412-683130ed8b04 2024-06-03T09:06:49.675 INFO:teuthology.orchestra.run.smithi088.stderr:last_changed 2024-06-03T08:34:31.885125+0000 2024-06-03T09:06:49.675 INFO:teuthology.orchestra.run.smithi088.stderr:created 2024-06-03T08:34:31.885125+0000 2024-06-03T09:06:49.675 INFO:teuthology.orchestra.run.smithi088.stderr:min_mon_release 18 (reef) 2024-06-03T09:06:49.676 INFO:teuthology.orchestra.run.smithi088.stderr:election_strategy: 3 2024-06-03T09:06:49.676 INFO:teuthology.orchestra.run.smithi088.stderr:0: v1:172.21.15.88:6789/0 mon.a 2024-06-03T09:06:49.676 INFO:teuthology.orchestra.run.smithi088.stderr:1: v1:172.21.15.192:6789/0 mon.b 2024-06-03T09:06:49.676 INFO:teuthology.orchestra.run.smithi088.stderr:2: v1:172.21.15.88:6790/0 mon.c 2024-06-03T09:06:49.676 INFO:teuthology.orchestra.run.smithi088.stderr: 2024-06-03T09:06:49.677 INFO:teuthology.orchestra.run.smithi088.stderr:2024-06-03T09:06:49.673+0000 7f24917fa640 1 -- 172.21.15.88:0/1698317358 <== mon.1 v1:172.21.15.192:6789/0 4 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 580+0+0 (unknown 3270334840 0 0) 0x7f249c0a9660 con 0x7f249c13eec0 2024-06-03T09:06:49.677 INFO:teuthology.orchestra.run.smithi088.stderr:2024-06-03T09:06:49.673+0000 7f24917fa640 1 -- 172.21.15.88:0/1698317358 >> v1:172.21.15.88:6790/0 conn(0x7f249c0a8960 legacy=0x7f249c1da390 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-06-03T09:06:49.677 INFO:teuthology.orchestra.run.smithi088.stderr:2024-06-03T09:06:49.673+0000 7f24917fa640 1 -- 172.21.15.88:0/1698317358 >> v1:172.21.15.88:6789/0 conn(0x7f249c1450f0 legacy=0x7f249c1daaa0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-06-03T09:06:49.677 INFO:teuthology.orchestra.run.smithi088.stderr:2024-06-03T09:06:49.673+0000 7f24917fa640 10 monclient: _finish_hunting 0 2024-06-03T09:06:49.678 INFO:teuthology.orchestra.run.smithi088.stderr:2024-06-03T09:06:49.673+0000 7f24917fa640 1 monclient: found mon.b 2024-06-03T09:06:49.678 INFO:teuthology.orchestra.run.smithi088.stderr:2024-06-03T09:06:49.673+0000 7f24917fa640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-06-03T09:06:49.678 INFO:teuthology.orchestra.run.smithi088.stderr:2024-06-03T09:06:49.673+0000 7f24917fa640 10 monclient: _send_mon_message to mon.b at v1:172.21.15.192:6789/0 2024-06-03T09:06:49.678 INFO:teuthology.orchestra.run.smithi088.stderr:2024-06-03T09:06:49.673+0000 7f24917fa640 1 -- 172.21.15.88:0/1698317358 --> v1:172.21.15.192:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f249c1db1b0 con 0x7f249c13eec0 2024-06-03T09:06:49.678 INFO:teuthology.orchestra.run.smithi088.stderr:2024-06-03T09:06:49.673+0000 7f24917fa640 10 monclient: _finish_auth 0 2024-06-03T09:06:49.678 INFO:teuthology.orchestra.run.smithi088.stderr:2024-06-03T09:06:49.673+0000 7f24917fa640 10 monclient: _check_auth_tickets 2024-06-03T09:06:49.678 INFO:teuthology.orchestra.run.smithi088.stderr:2024-06-03T09:06:49.673+0000 7f24917fa640 20 monclient: _check_auth_rotating not needed by client.admin 2024-06-03T09:06:49.678 INFO:teuthology.orchestra.run.smithi088.stderr:2024-06-03T09:06:49.673+0000 7f24a2e69640 5 monclient: authenticate success, global_id 30292 2024-06-03T09:06:49.678 INFO:teuthology.orchestra.run.smithi088.stderr:2024-06-03T09:06:49.673+0000 7f24a2e69640 10 monclient: _renew_subs 2024-06-03T09:06:49.678 INFO:teuthology.orchestra.run.smithi088.stderr:2024-06-03T09:06:49.673+0000 7f24a2e69640 10 monclient: _send_mon_message to mon.b at v1:172.21.15.192:6789/0 2024-06-03T09:06:49.678 INFO:teuthology.orchestra.run.smithi088.stderr:2024-06-03T09:06:49.673+0000 7f24a2e69640 1 -- 172.21.15.88:0/1698317358 --> v1:172.21.15.192:6789/0 -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f249c140e00 con 0x7f249c13eec0 2024-06-03T09:06:49.678 INFO:teuthology.orchestra.run.smithi088.stderr:2024-06-03T09:06:49.673+0000 7f24a2e69640 10 client.30292.objecter _maybe_request_map subscribing (onetime) to next osd map 2024-06-03T09:06:49.678 INFO:teuthology.orchestra.run.smithi088.stderr:2024-06-03T09:06:49.673+0000 7f24a2e69640 10 monclient: _renew_subs 2024-06-03T09:06:49.679 INFO:teuthology.orchestra.run.smithi088.stderr:2024-06-03T09:06:49.673+0000 7f24a2e69640 10 monclient: _send_mon_message to mon.b at v1:172.21.15.192:6789/0 2024-06-03T09:06:49.679 INFO:teuthology.orchestra.run.smithi088.stderr:2024-06-03T09:06:49.673+0000 7f24a2e69640 1 -- 172.21.15.88:0/1698317358 --> v1:172.21.15.192:6789/0 -- mon_subscribe({osdmap=0}) v3 -- 0x7f249c1444a0 con 0x7f249c13eec0 2024-06-03T09:06:49.679 INFO:teuthology.orchestra.run.smithi088.stderr:2024-06-03T09:06:49.673+0000 7f24a2e69640 1 librados: init done 2024-06-03T09:06:49.679 INFO:teuthology.orchestra.run.smithi088.stderr:2024-06-03T09:06:49.674+0000 7f24a2e69640 10 start_mon_command cmd=[{"prefix": "get_command_descriptions"}] 2024-06-03T09:06:49.679 INFO:teuthology.orchestra.run.smithi088.stderr:2024-06-03T09:06:49.674+0000 7f24a2e69640 10 monclient: _send_command 1 [{"prefix": "get_command_descriptions"}] 2024-06-03T09:06:49.679 INFO:teuthology.orchestra.run.smithi088.stderr:2024-06-03T09:06:49.674+0000 7f24a2e69640 10 monclient: _send_mon_message to mon.b at v1:172.21.15.192:6789/0 2024-06-03T09:06:49.679 INFO:teuthology.orchestra.run.smithi088.stderr:2024-06-03T09:06:49.674+0000 7f24a2e69640 1 -- 172.21.15.88:0/1698317358 --> v1:172.21.15.192:6789/0 -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f249c0a9e60 con 0x7f249c13eec0 2024-06-03T09:06:49.920 INFO:teuthology.orchestra.run.smithi088.stderr:2024-06-03T09:06:49.920+0000 7f24917fa640 1 -- 172.21.15.88:0/1698317358 <== mon.1 v1:172.21.15.192:6789/0 5 ==== config(0 keys) v1 ==== 4+0+0 (unknown 0 0 0) 0x7f248c001170 con 0x7f249c13eec0 2024-06-03T09:06:49.920 INFO:teuthology.orchestra.run.smithi088.stderr:2024-06-03T09:06:49.920+0000 7f24917fa640 10 monclient: handle_config config(0 keys) v1 2024-06-03T09:06:49.920 INFO:teuthology.orchestra.run.smithi088.stderr:2024-06-03T09:06:49.920+0000 7f24917fa640 1 -- 172.21.15.88:0/1698317358 <== mon.1 v1:172.21.15.192:6789/0 6 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 2234374065 0 0) 0x7f248c004a90 con 0x7f249c13eec0 2024-06-03T09:06:49.920 INFO:teuthology.orchestra.run.smithi088.stderr:2024-06-03T09:06:49.920+0000 7f24917fa640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-06-03T09:06:49.920 INFO:teuthology.orchestra.run.smithi088.stderr:2024-06-03T09:06:49.920+0000 7f24917fa640 10 monclient: got monmap 1 from mon.b (according to old e1) 2024-06-03T09:06:49.920 INFO:teuthology.orchestra.run.smithi088.stderr:2024-06-03T09:06:49.920+0000 7f24917fa640 10 monclient: dump: 2024-06-03T09:06:49.920 INFO:teuthology.orchestra.run.smithi088.stderr:epoch 1 2024-06-03T09:06:49.920 INFO:teuthology.orchestra.run.smithi088.stderr:fsid ec3c3fdd-3ac6-4d7d-9412-683130ed8b04 2024-06-03T09:06:49.920 INFO:teuthology.orchestra.run.smithi088.stderr:last_changed 2024-06-03T08:34:31.885125+0000 2024-06-03T09:06:49.921 INFO:teuthology.orchestra.run.smithi088.stderr:created 2024-06-03T08:34:31.885125+0000 2024-06-03T09:06:49.921 INFO:teuthology.orchestra.run.smithi088.stderr:min_mon_release 18 (reef) 2024-06-03T09:06:49.921 INFO:teuthology.orchestra.run.smithi088.stderr:election_strategy: 3 2024-06-03T09:06:49.921 INFO:teuthology.orchestra.run.smithi088.stderr:0: v1:172.21.15.88:6789/0 mon.a 2024-06-03T09:06:49.921 INFO:teuthology.orchestra.run.smithi088.stderr:1: v1:172.21.15.192:6789/0 mon.b 2024-06-03T09:06:49.921 INFO:teuthology.orchestra.run.smithi088.stderr:2: v1:172.21.15.88:6790/0 mon.c 2024-06-03T09:06:49.921 INFO:teuthology.orchestra.run.smithi088.stderr: 2024-06-03T09:06:49.921 INFO:teuthology.orchestra.run.smithi088.stderr:2024-06-03T09:06:49.920+0000 7f2492ffd640 4 set_mon_vals no callback set 2024-06-03T09:06:49.922 INFO:teuthology.orchestra.run.smithi088.stderr:2024-06-03T09:06:49.920+0000 7f24917fa640 1 -- 172.21.15.88:0/1698317358 <== mon.1 v1:172.21.15.192:6789/0 7 ==== mgrmap(e 5) v1 ==== 89110+0+0 (unknown 873533973 0 0) 0x7f248c01a810 con 0x7f249c13eec0 2024-06-03T09:06:49.922 INFO:teuthology.orchestra.run.smithi088.stderr:2024-06-03T09:06:49.921+0000 7f24a0c05640 10 monclient: get_auth_request con 0x7f248806d400 auth_method 0 2024-06-03T09:06:49.922 INFO:teuthology.orchestra.run.smithi088.stderr:2024-06-03T09:06:49.921+0000 7f24917fa640 10 client.30292.objecter ms_handle_connect 0x7f248806d400 2024-06-03T09:06:49.923 INFO:teuthology.orchestra.run.smithi088.stderr:2024-06-03T09:06:49.923+0000 7f24917fa640 1 -- 172.21.15.88:0/1698317358 <== mon.1 v1:172.21.15.192:6789/0 8 ==== osd_map(1014..1014 src has 622..1014) v4 ==== 5229+0+0 (unknown 3230192161 0 0) 0x7f248c053fc0 con 0x7f249c13eec0 2024-06-03T09:06:49.923 INFO:teuthology.orchestra.run.smithi088.stderr:2024-06-03T09:06:49.923+0000 7f24917fa640 10 client.30292.objecter ms_dispatch 0x7f249c000b90 osd_map(1014..1014 src has 622..1014) v4 2024-06-03T09:06:49.923 INFO:teuthology.orchestra.run.smithi088.stderr:2024-06-03T09:06:49.923+0000 7f24917fa640 3 client.30292.objecter handle_osd_map got epochs [1014,1014] > 0 2024-06-03T09:06:49.923 INFO:teuthology.orchestra.run.smithi088.stderr:2024-06-03T09:06:49.923+0000 7f24917fa640 3 client.30292.objecter handle_osd_map decoding full epoch 1014 2024-06-03T09:06:49.923 INFO:teuthology.orchestra.run.smithi088.stderr:2024-06-03T09:06:49.923+0000 7f24917fa640 20 client.30292.objecter dump_active .. 0 homeless 2024-06-03T09:06:49.925 INFO:teuthology.orchestra.run.smithi088.stderr:2024-06-03T09:06:49.925+0000 7f24917fa640 1 -- 172.21.15.88:0/1698317358 <== mon.1 v1:172.21.15.192:6789/0 9 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+182498 (unknown 1092875540 0 3016760700) 0x7f248c04f9a0 con 0x7f249c13eec0 2024-06-03T09:06:49.925 INFO:teuthology.orchestra.run.smithi088.stderr:2024-06-03T09:06:49.925+0000 7f24917fa640 10 monclient: handle_mon_command_ack 1 [{"prefix": "get_command_descriptions"}] 2024-06-03T09:06:49.925 INFO:teuthology.orchestra.run.smithi088.stderr:2024-06-03T09:06:49.925+0000 7f24917fa640 10 monclient: _finish_command 1 = system:0 2024-06-03T09:06:50.096 INFO:teuthology.orchestra.run.smithi088.stderr:2024-06-03T09:06:50.096+0000 7f24a2e69640 1 -- 172.21.15.88:0/1698317358 --> v1:172.21.15.192:6800/34229 -- mgr_command(tid 0: {"prefix": "pg dump", "target": ["mon-mgr", ""], "format": "json"}) v1 -- 0x7f249c13db70 con 0x7f248806d400 2024-06-03T09:06:50.101 INFO:teuthology.orchestra.run.smithi088.stderr:2024-06-03T09:06:50.101+0000 7f24917fa640 1 -- 172.21.15.88:0/1698317358 <== mgr.4110 v1:172.21.15.192:6800/34229 1 ==== mgr_command_reply(tid 0: 0 dumped all) v1 ==== 18+0+151731 (unknown 2965378022 0 2910335067) 0x7f249c13db70 con 0x7f248806d400 2024-06-03T09:06:50.103 INFO:teuthology.orchestra.run.smithi088.stdout: 2024-06-03T09:06:50.103 INFO:teuthology.orchestra.run.smithi088.stderr:dumped all 2024-06-03T09:06:50.105 INFO:teuthology.orchestra.run.smithi088.stderr:2024-06-03T09:06:50.103+0000 7f24a2e69640 10 librados: watch_flush enter 2024-06-03T09:06:50.105 INFO:teuthology.orchestra.run.smithi088.stderr:2024-06-03T09:06:50.103+0000 7f24a2e69640 10 librados: watch_flush exit 2024-06-03T09:06:50.105 INFO:teuthology.orchestra.run.smithi088.stderr:2024-06-03T09:06:50.103+0000 7f24a2e69640 20 client.30292.objecter shutdown clearing up homeless session... 2024-06-03T09:06:50.105 INFO:teuthology.orchestra.run.smithi088.stderr:2024-06-03T09:06:50.103+0000 7f24a2e69640 10 client.30292.objecter successfully canceled tick 2024-06-03T09:06:50.105 INFO:teuthology.orchestra.run.smithi088.stderr:2024-06-03T09:06:50.103+0000 7f24a2e69640 1 -- 172.21.15.88:0/1698317358 >> v1:172.21.15.192:6800/34229 conn(0x7f248806d400 legacy=0x7f248806f8c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-06-03T09:06:50.105 INFO:teuthology.orchestra.run.smithi088.stderr:2024-06-03T09:06:50.103+0000 7f24a2e69640 10 monclient: shutdown 2024-06-03T09:06:50.105 INFO:teuthology.orchestra.run.smithi088.stderr:2024-06-03T09:06:50.103+0000 7f24a2e69640 20 monclient: shutdown discarding 0 pending message(s) 2024-06-03T09:06:50.105 INFO:teuthology.orchestra.run.smithi088.stderr:2024-06-03T09:06:50.103+0000 7f24a2e69640 1 -- 172.21.15.88:0/1698317358 >> v1:172.21.15.192:6789/0 conn(0x7f249c13eec0 legacy=0x7f249c140390 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-06-03T09:06:50.105 INFO:teuthology.orchestra.run.smithi088.stderr:2024-06-03T09:06:50.103+0000 7f24a2e69640 1 -- 172.21.15.88:0/1698317358 shutdown_connections 2024-06-03T09:06:50.106 INFO:teuthology.orchestra.run.smithi088.stderr:2024-06-03T09:06:50.103+0000 7f24a2e69640 1 -- 172.21.15.88:0/1698317358 >> 172.21.15.88:0/1698317358 conn(0x7f249c0b70d0 msgr2=0x7f249c13e580 unknown :-1 s=STATE_NONE l=0).mark_down 2024-06-03T09:06:50.106 INFO:teuthology.orchestra.run.smithi088.stderr:2024-06-03T09:06:50.103+0000 7f24a2e69640 1 -- 172.21.15.88:0/1698317358 shutdown_connections 2024-06-03T09:06:50.106 INFO:teuthology.orchestra.run.smithi088.stderr:2024-06-03T09:06:50.103+0000 7f24a2e69640 1 -- 172.21.15.88:0/1698317358 wait complete. 2024-06-03T09:06:50.106 INFO:teuthology.orchestra.run.smithi088.stderr:2024-06-03T09:06:50.103+0000 7f24a2e69640 1 librados: shutdown 2024-06-03T09:06:50.114 INFO:teuthology.orchestra.run.smithi088.stdout:{"pg_ready":true,"pg_map":{"version":1934,"stamp":"2024-06-03T09:06:49.104509+0000","last_osdmap_epoch":0,"last_pg_scan":0,"pg_stats_sum":{"stat_sum":{"num_bytes":459285,"num_objects":3,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":3,"num_whiteouts":0,"num_read":102,"num_read_kb":88,"num_write":120,"num_write_kb":1379,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2,"num_bytes_recovered":397840,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":83,"ondisk_log_size":83,"up":77,"acting":77,"num_store_stats":0},"osd_stats_sum":{"up_from":0,"seq":0,"num_pgs":82,"num_osds":8,"num_per_pool_osds":8,"num_per_pool_omap_osds":8,"kb":754974720,"kb_used":494872,"kb_used_data":46628,"kb_used_omap":12,"kb_used_meta":448115,"kb_avail":754479848,"statfs":{"total":773094113280,"available":772587364352,"internally_reserved":0,"allocated":47747072,"data_stored":22815791,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":12744,"internal_metadata":458870328},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":465,"apply_latency_ms":465,"commit_latency_ns":465000000,"apply_latency_ns":465000000},"alerts":[],"network_ping_times":[]},"pg_stats_delta":{"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":-1,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":0,"acting":0,"num_store_stats":0,"stamp_delta":"6.001430"},"pg_stats":[{"pgid":"17.d","version":"0'0","reported_seq":942,"reported_epoch":1009,"state":"active+clean","last_fresh":"2024-06-03T09:06:14.443917+0000","last_change":"2024-06-03T08:38:19.260548+0000","last_active":"2024-06-03T09:06:14.443917+0000","last_peered":"2024-06-03T09:06:14.443917+0000","last_clean":"2024-06-03T09:06:14.443917+0000","last_became_active":"2024-06-03T08:38:19.253559+0000","last_became_peered":"2024-06-03T08:38:19.253559+0000","last_unstale":"2024-06-03T09:06:14.443917+0000","last_undegraded":"2024-06-03T09:06:14.443917+0000","last_fullsized":"2024-06-03T09:06:14.443917+0000","mapping_epoch":28,"log_start":"0'0","ondisk_log_start":"0'0","created":28,"last_epoch_clean":45,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-06-03T08:37:55.614497+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-06-03T08:37:55.614497+0000","last_clean_scrub_stamp":"2024-06-03T08:37:55.614497+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-06-04T19:00:12.470979+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1],"acting":[0,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"17.c","version":"0'0","reported_seq":955,"reported_epoch":1011,"state":"active+undersized","last_fresh":"2024-06-03T09:06:42.798184+0000","last_change":"2024-06-03T09:06:03.440074+0000","last_active":"2024-06-03T09:06:42.798184+0000","last_peered":"2024-06-03T09:06:42.798184+0000","last_clean":"2024-06-03T09:05:02.105638+0000","last_became_active":"2024-06-03T09:06:03.440074+0000","last_became_peered":"2024-06-03T09:06:03.440074+0000","last_unstale":"2024-06-03T09:06:42.798184+0000","last_undegraded":"2024-06-03T09:06:42.798184+0000","last_fullsized":"2024-06-03T09:06:03.377982+0000","mapping_epoch":999,"log_start":"0'0","ondisk_log_start":"0'0","created":28,"last_epoch_clean":255,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-06-03T08:37:55.614497+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-06-03T08:37:55.614497+0000","last_clean_scrub_stamp":"2024-06-03T08:37:55.614497+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-06-04T10:34:20.808557+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3],"acting":[3],"avail_no_missing":["3"],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"17.f","version":"0'0","reported_seq":843,"reported_epoch":1014,"state":"active+clean","last_fresh":"2024-06-03T09:06:46.306998+0000","last_change":"2024-06-03T09:06:46.306998+0000","last_active":"2024-06-03T09:06:46.306998+0000","last_peered":"2024-06-03T09:06:46.306998+0000","last_clean":"2024-06-03T09:06:46.306998+0000","last_became_active":"2024-06-03T09:06:46.301958+0000","last_became_peered":"2024-06-03T09:06:46.301958+0000","last_unstale":"2024-06-03T09:06:46.306998+0000","last_undegraded":"2024-06-03T09:06:46.306998+0000","last_fullsized":"2024-06-03T09:06:46.306998+0000","mapping_epoch":1013,"log_start":"0'0","ondisk_log_start":"0'0","created":28,"last_epoch_clean":1014,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-06-03T08:37:55.614497+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-06-03T08:37:55.614497+0000","last_clean_scrub_stamp":"2024-06-03T08:37:55.614497+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-06-04T14:30:23.089522+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,4],"acting":[6,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"17.e","version":"0'0","reported_seq":843,"reported_epoch":1014,"state":"active+clean","last_fresh":"2024-06-03T09:06:46.278835+0000","last_change":"2024-06-03T09:06:46.278835+0000","last_active":"2024-06-03T09:06:46.278835+0000","last_peered":"2024-06-03T09:06:46.278835+0000","last_clean":"2024-06-03T09:06:46.278835+0000","last_became_active":"2024-06-03T09:06:46.273941+0000","last_became_peered":"2024-06-03T09:06:46.273941+0000","last_unstale":"2024-06-03T09:06:46.278835+0000","last_undegraded":"2024-06-03T09:06:46.278835+0000","last_fullsized":"2024-06-03T09:06:46.278835+0000","mapping_epoch":1013,"log_start":"0'0","ondisk_log_start":"0'0","created":28,"last_epoch_clean":1014,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-06-03T08:37:55.614497+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-06-03T08:37:55.614497+0000","last_clean_scrub_stamp":"2024-06-03T08:37:55.614497+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-06-04T09:31:07.311650+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,5],"acting":[6,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"17.9","version":"0'0","reported_seq":947,"reported_epoch":1010,"state":"active+clean","last_fresh":"2024-06-03T09:06:18.488819+0000","last_change":"2024-06-03T08:38:04.772260+0000","last_active":"2024-06-03T09:06:18.488819+0000","last_peered":"2024-06-03T09:06:18.488819+0000","last_clean":"2024-06-03T09:06:18.488819+0000","last_became_active":"2024-06-03T08:38:04.763428+0000","last_became_peered":"2024-06-03T08:38:04.763428+0000","last_unstale":"2024-06-03T09:06:18.488819+0000","last_undegraded":"2024-06-03T09:06:18.488819+0000","last_fullsized":"2024-06-03T09:06:18.488819+0000","mapping_epoch":28,"log_start":"0'0","ondisk_log_start":"0'0","created":28,"last_epoch_clean":33,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-06-03T08:37:55.614497+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-06-03T08:37:55.614497+0000","last_clean_scrub_stamp":"2024-06-03T08:37:55.614497+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-06-04T10:55:10.773936+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,4],"acting":[7,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"17.8","version":"0'0","reported_seq":961,"reported_epoch":1009,"state":"active+undersized","last_fresh":"2024-06-03T09:06:14.430375+0000","last_change":"2024-06-03T09:06:03.388786+0000","last_active":"2024-06-03T09:06:14.430375+0000","last_peered":"2024-06-03T09:06:14.430375+0000","last_clean":"2024-06-03T09:04:43.297780+0000","last_became_active":"2024-06-03T09:06:03.388786+0000","last_became_peered":"2024-06-03T09:06:03.388786+0000","last_unstale":"2024-06-03T09:06:14.430375+0000","last_undegraded":"2024-06-03T09:06:14.430375+0000","last_fullsized":"2024-06-03T09:06:03.331754+0000","mapping_epoch":999,"log_start":"0'0","ondisk_log_start":"0'0","created":28,"last_epoch_clean":255,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-06-03T08:37:55.614497+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-06-03T08:37:55.614497+0000","last_clean_scrub_stamp":"2024-06-03T08:37:55.614497+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-06-04T09:18:44.765547+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0],"acting":[0],"avail_no_missing":["0"],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"17.b","version":"0'0","reported_seq":843,"reported_epoch":1014,"state":"active+clean","last_fresh":"2024-06-03T09:06:46.350450+0000","last_change":"2024-06-03T09:06:46.350450+0000","last_active":"2024-06-03T09:06:46.350450+0000","last_peered":"2024-06-03T09:06:46.350450+0000","last_clean":"2024-06-03T09:06:46.350450+0000","last_became_active":"2024-06-03T09:06:46.345458+0000","last_became_peered":"2024-06-03T09:06:46.345458+0000","last_unstale":"2024-06-03T09:06:46.350450+0000","last_undegraded":"2024-06-03T09:06:46.350450+0000","last_fullsized":"2024-06-03T09:06:46.350450+0000","mapping_epoch":1013,"log_start":"0'0","ondisk_log_start":"0'0","created":28,"last_epoch_clean":1014,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-06-03T08:37:55.614497+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-06-03T08:37:55.614497+0000","last_clean_scrub_stamp":"2024-06-03T08:37:55.614497+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-06-04T16:51:34.952224+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,2],"acting":[6,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"17.a","version":"0'0","reported_seq":927,"reported_epoch":1011,"state":"active+clean","last_fresh":"2024-06-03T09:06:42.861977+0000","last_change":"2024-06-03T08:38:05.843142+0000","last_active":"2024-06-03T09:06:42.861977+0000","last_peered":"2024-06-03T09:06:42.861977+0000","last_clean":"2024-06-03T09:06:42.861977+0000","last_became_active":"2024-06-03T08:38:05.809257+0000","last_became_peered":"2024-06-03T08:38:05.809257+0000","last_unstale":"2024-06-03T09:06:42.861977+0000","last_undegraded":"2024-06-03T09:06:42.861977+0000","last_fullsized":"2024-06-03T09:06:42.861977+0000","mapping_epoch":28,"log_start":"0'0","ondisk_log_start":"0'0","created":28,"last_epoch_clean":33,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-06-03T08:37:55.614497+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-06-03T08:37:55.614497+0000","last_clean_scrub_stamp":"2024-06-03T08:37:55.614497+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-06-04T12:12:03.686838+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2],"acting":[1,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"17.5","version":"0'0","reported_seq":943,"reported_epoch":1009,"state":"active+clean","last_fresh":"2024-06-03T09:06:14.442357+0000","last_change":"2024-06-03T08:38:23.466051+0000","last_active":"2024-06-03T09:06:14.442357+0000","last_peered":"2024-06-03T09:06:14.442357+0000","last_clean":"2024-06-03T09:06:14.442357+0000","last_became_active":"2024-06-03T08:38:23.458039+0000","last_became_peered":"2024-06-03T08:38:23.458039+0000","last_unstale":"2024-06-03T09:06:14.442357+0000","last_undegraded":"2024-06-03T09:06:14.442357+0000","last_fullsized":"2024-06-03T09:06:14.442357+0000","mapping_epoch":28,"log_start":"0'0","ondisk_log_start":"0'0","created":28,"last_epoch_clean":49,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-06-03T08:37:55.614497+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-06-03T08:37:55.614497+0000","last_clean_scrub_stamp":"2024-06-03T08:37:55.614497+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-06-04T12:27:02.532436+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,4],"acting":[0,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"17.4","version":"0'0","reported_seq":843,"reported_epoch":1014,"state":"active+clean","last_fresh":"2024-06-03T09:06:46.456388+0000","last_change":"2024-06-03T09:06:46.456388+0000","last_active":"2024-06-03T09:06:46.456388+0000","last_peered":"2024-06-03T09:06:46.456388+0000","last_clean":"2024-06-03T09:06:46.456388+0000","last_became_active":"2024-06-03T09:06:46.447534+0000","last_became_peered":"2024-06-03T09:06:46.447534+0000","last_unstale":"2024-06-03T09:06:46.456388+0000","last_undegraded":"2024-06-03T09:06:46.456388+0000","last_fullsized":"2024-06-03T09:06:46.456388+0000","mapping_epoch":1013,"log_start":"0'0","ondisk_log_start":"0'0","created":28,"last_epoch_clean":1014,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-06-03T08:37:55.614497+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-06-03T08:37:55.614497+0000","last_clean_scrub_stamp":"2024-06-03T08:37:55.614497+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-06-04T16:06:17.792570+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,7],"acting":[6,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"17.7","version":"0'0","reported_seq":933,"reported_epoch":1013,"state":"active+clean","last_fresh":"2024-06-03T09:06:45.495807+0000","last_change":"2024-06-03T08:38:04.837083+0000","last_active":"2024-06-03T09:06:45.495807+0000","last_peered":"2024-06-03T09:06:45.495807+0000","last_clean":"2024-06-03T09:06:45.495807+0000","last_became_active":"2024-06-03T08:38:04.818246+0000","last_became_peered":"2024-06-03T08:38:04.818246+0000","last_unstale":"2024-06-03T09:06:45.495807+0000","last_undegraded":"2024-06-03T09:06:45.495807+0000","last_fullsized":"2024-06-03T09:06:45.495807+0000","mapping_epoch":28,"log_start":"0'0","ondisk_log_start":"0'0","created":28,"last_epoch_clean":34,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-06-03T08:37:55.614497+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-06-03T08:37:55.614497+0000","last_clean_scrub_stamp":"2024-06-03T08:37:55.614497+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-06-04T17:02:38.204813+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,2],"acting":[5,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"17.6","version":"0'0","reported_seq":960,"reported_epoch":1009,"state":"active+undersized","last_fresh":"2024-06-03T09:06:14.453110+0000","last_change":"2024-06-03T09:06:03.428677+0000","last_active":"2024-06-03T09:06:14.453110+0000","last_peered":"2024-06-03T09:06:14.453110+0000","last_clean":"2024-06-03T09:04:43.312352+0000","last_became_active":"2024-06-03T09:06:03.428677+0000","last_became_peered":"2024-06-03T09:06:03.428677+0000","last_unstale":"2024-06-03T09:06:14.453110+0000","last_undegraded":"2024-06-03T09:06:14.453110+0000","last_fullsized":"2024-06-03T09:06:03.359909+0000","mapping_epoch":999,"log_start":"0'0","ondisk_log_start":"0'0","created":28,"last_epoch_clean":255,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-06-03T08:37:55.614497+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-06-03T08:37:55.614497+0000","last_clean_scrub_stamp":"2024-06-03T08:37:55.614497+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-06-04T15:38:02.180241+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0],"acting":[0],"avail_no_missing":["0"],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"2.7","version":"0'0","reported_seq":942,"reported_epoch":1011,"state":"active+clean","last_fresh":"2024-06-03T09:06:42.820718+0000","last_change":"2024-06-03T08:36:30.385405+0000","last_active":"2024-06-03T09:06:42.820718+0000","last_peered":"2024-06-03T09:06:42.820718+0000","last_clean":"2024-06-03T09:06:42.820718+0000","last_became_active":"2024-06-03T08:36:30.380302+0000","last_became_peered":"2024-06-03T08:36:30.380302+0000","last_unstale":"2024-06-03T09:06:42.820718+0000","last_undegraded":"2024-06-03T09:06:42.820718+0000","last_fullsized":"2024-06-03T09:06:42.820718+0000","mapping_epoch":23,"log_start":"0'0","ondisk_log_start":"0'0","created":23,"last_epoch_clean":24,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-06-03T08:36:28.444764+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-06-03T08:36:28.444764+0000","last_clean_scrub_stamp":"2024-06-03T08:36:28.444764+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-06-04T18:46:33.475181+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0],"acting":[3,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"17.14","version":"0'0","reported_seq":930,"reported_epoch":1013,"state":"active+clean","last_fresh":"2024-06-03T09:06:45.541591+0000","last_change":"2024-06-03T08:38:04.547565+0000","last_active":"2024-06-03T09:06:45.541591+0000","last_peered":"2024-06-03T09:06:45.541591+0000","last_clean":"2024-06-03T09:06:45.541591+0000","last_became_active":"2024-06-03T08:38:04.541781+0000","last_became_peered":"2024-06-03T08:38:04.541781+0000","last_unstale":"2024-06-03T09:06:45.541591+0000","last_undegraded":"2024-06-03T09:06:45.541591+0000","last_fullsized":"2024-06-03T09:06:45.541591+0000","mapping_epoch":28,"log_start":"0'0","ondisk_log_start":"0'0","created":28,"last_epoch_clean":34,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-06-03T08:37:55.614497+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-06-03T08:37:55.614497+0000","last_clean_scrub_stamp":"2024-06-03T08:37:55.614497+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-06-04T16:41:55.170134+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,2],"acting":[5,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"2.6","version":"0'0","reported_seq":953,"reported_epoch":1013,"state":"active+clean","last_fresh":"2024-06-03T09:06:45.018770+0000","last_change":"2024-06-03T08:36:30.430760+0000","last_active":"2024-06-03T09:06:45.018770+0000","last_peered":"2024-06-03T09:06:45.018770+0000","last_clean":"2024-06-03T09:06:45.018770+0000","last_became_active":"2024-06-03T08:36:30.404665+0000","last_became_peered":"2024-06-03T08:36:30.404665+0000","last_unstale":"2024-06-03T09:06:45.018770+0000","last_undegraded":"2024-06-03T09:06:45.018770+0000","last_fullsized":"2024-06-03T09:06:45.018770+0000","mapping_epoch":23,"log_start":"0'0","ondisk_log_start":"0'0","created":23,"last_epoch_clean":24,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-06-03T08:36:28.444764+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-06-03T08:36:28.444764+0000","last_clean_scrub_stamp":"2024-06-03T08:36:28.444764+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-06-04T10:37:37.746074+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,0],"acting":[4,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"17.15","version":"0'0","reported_seq":936,"reported_epoch":1011,"state":"active+clean","last_fresh":"2024-06-03T09:06:42.808375+0000","last_change":"2024-06-03T08:38:20.999903+0000","last_active":"2024-06-03T09:06:42.808375+0000","last_peered":"2024-06-03T09:06:42.808375+0000","last_clean":"2024-06-03T09:06:42.808375+0000","last_became_active":"2024-06-03T08:38:20.979523+0000","last_became_peered":"2024-06-03T08:38:20.979523+0000","last_unstale":"2024-06-03T09:06:42.808375+0000","last_undegraded":"2024-06-03T09:06:42.808375+0000","last_fullsized":"2024-06-03T09:06:42.808375+0000","mapping_epoch":28,"log_start":"0'0","ondisk_log_start":"0'0","created":28,"last_epoch_clean":47,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-06-03T08:37:55.614497+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-06-03T08:37:55.614497+0000","last_clean_scrub_stamp":"2024-06-03T08:37:55.614497+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-06-04T20:29:58.314289+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,1],"acting":[3,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"2.5","version":"0'0","reported_seq":944,"reported_epoch":1011,"state":"active+clean","last_fresh":"2024-06-03T09:06:42.818103+0000","last_change":"2024-06-03T08:36:30.024235+0000","last_active":"2024-06-03T09:06:42.818103+0000","last_peered":"2024-06-03T09:06:42.818103+0000","last_clean":"2024-06-03T09:06:42.818103+0000","last_became_active":"2024-06-03T08:36:30.019040+0000","last_became_peered":"2024-06-03T08:36:30.019040+0000","last_unstale":"2024-06-03T09:06:42.818103+0000","last_undegraded":"2024-06-03T09:06:42.818103+0000","last_fullsized":"2024-06-03T09:06:42.818103+0000","mapping_epoch":23,"log_start":"0'0","ondisk_log_start":"0'0","created":23,"last_epoch_clean":24,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-06-03T08:36:28.444764+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-06-03T08:36:28.444764+0000","last_clean_scrub_stamp":"2024-06-03T08:36:28.444764+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-06-04T15:46:47.109502+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,4],"acting":[3,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"17.16","version":"0'0","reported_seq":974,"reported_epoch":1011,"state":"active+undersized","last_fresh":"2024-06-03T09:06:42.992868+0000","last_change":"2024-06-03T09:06:03.478615+0000","last_active":"2024-06-03T09:06:42.992868+0000","last_peered":"2024-06-03T09:06:42.992868+0000","last_clean":"2024-06-03T09:04:43.328415+0000","last_became_active":"2024-06-03T09:06:03.478615+0000","last_became_peered":"2024-06-03T09:06:03.478615+0000","last_unstale":"2024-06-03T09:06:42.992868+0000","last_undegraded":"2024-06-03T09:06:42.992868+0000","last_fullsized":"2024-06-03T09:06:03.402504+0000","mapping_epoch":999,"log_start":"0'0","ondisk_log_start":"0'0","created":28,"last_epoch_clean":255,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-06-03T08:37:55.614497+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-06-03T08:37:55.614497+0000","last_clean_scrub_stamp":"2024-06-03T08:37:55.614497+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-06-04T08:47:08.079233+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2],"acting":[2],"avail_no_missing":["2"],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"2.4","version":"0'0","reported_seq":964,"reported_epoch":1014,"state":"active+clean","last_fresh":"2024-06-03T09:06:46.377610+0000","last_change":"2024-06-03T09:06:46.377610+0000","last_active":"2024-06-03T09:06:46.377610+0000","last_peered":"2024-06-03T09:06:46.377610+0000","last_clean":"2024-06-03T09:06:46.377610+0000","last_became_active":"2024-06-03T09:06:46.371430+0000","last_became_peered":"2024-06-03T09:06:46.371430+0000","last_unstale":"2024-06-03T09:06:46.377610+0000","last_undegraded":"2024-06-03T09:06:46.377610+0000","last_fullsized":"2024-06-03T09:06:46.377610+0000","mapping_epoch":1013,"log_start":"0'0","ondisk_log_start":"0'0","created":23,"last_epoch_clean":1014,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-06-03T08:36:28.444764+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-06-03T08:36:28.444764+0000","last_clean_scrub_stamp":"2024-06-03T08:36:28.444764+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-06-04T12:20:39.476337+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,6],"acting":[5,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"17.17","version":"0'0","reported_seq":956,"reported_epoch":1011,"state":"active+clean","last_fresh":"2024-06-03T09:06:42.990396+0000","last_change":"2024-06-03T08:38:38.483275+0000","last_active":"2024-06-03T09:06:42.990396+0000","last_peered":"2024-06-03T09:06:42.990396+0000","last_clean":"2024-06-03T09:06:42.990396+0000","last_became_active":"2024-06-03T08:38:38.467916+0000","last_became_peered":"2024-06-03T08:38:38.467916+0000","last_unstale":"2024-06-03T09:06:42.990396+0000","last_undegraded":"2024-06-03T09:06:42.990396+0000","last_fullsized":"2024-06-03T09:06:42.990396+0000","mapping_epoch":28,"log_start":"0'0","ondisk_log_start":"0'0","created":28,"last_epoch_clean":61,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-06-03T08:37:55.614497+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-06-03T08:37:55.614497+0000","last_clean_scrub_stamp":"2024-06-03T08:37:55.614497+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-06-04T10:02:23.036420+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,7],"acting":[2,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"2.2","version":"0'0","reported_seq":965,"reported_epoch":1011,"state":"active+undersized","last_fresh":"2024-06-03T09:06:42.803839+0000","last_change":"2024-06-03T09:06:03.410020+0000","last_active":"2024-06-03T09:06:42.803839+0000","last_peered":"2024-06-03T09:06:42.803839+0000","last_clean":"2024-06-03T09:05:37.145889+0000","last_became_active":"2024-06-03T09:06:03.410020+0000","last_became_peered":"2024-06-03T09:06:03.410020+0000","last_unstale":"2024-06-03T09:06:42.803839+0000","last_undegraded":"2024-06-03T09:06:42.803839+0000","last_fullsized":"2024-06-03T09:06:03.346864+0000","mapping_epoch":999,"log_start":"0'0","ondisk_log_start":"0'0","created":23,"last_epoch_clean":255,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-06-03T08:36:28.444764+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-06-03T08:36:28.444764+0000","last_clean_scrub_stamp":"2024-06-03T08:36:28.444764+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-06-04T09:57:12.423360+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3],"acting":[3],"avail_no_missing":["3"],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"17.11","version":"0'0","reported_seq":947,"reported_epoch":1013,"state":"active+clean","last_fresh":"2024-06-03T09:06:44.976905+0000","last_change":"2024-06-03T08:38:06.502302+0000","last_active":"2024-06-03T09:06:44.976905+0000","last_peered":"2024-06-03T09:06:44.976905+0000","last_clean":"2024-06-03T09:06:44.976905+0000","last_became_active":"2024-06-03T08:38:06.490204+0000","last_became_peered":"2024-06-03T08:38:06.490204+0000","last_unstale":"2024-06-03T09:06:44.976905+0000","last_undegraded":"2024-06-03T09:06:44.976905+0000","last_fullsized":"2024-06-03T09:06:44.976905+0000","mapping_epoch":28,"log_start":"0'0","ondisk_log_start":"0'0","created":28,"last_epoch_clean":32,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-06-03T08:37:55.614497+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-06-03T08:37:55.614497+0000","last_clean_scrub_stamp":"2024-06-03T08:37:55.614497+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-06-04T10:47:05.988958+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,2],"acting":[4,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"2.1","version":"0'0","reported_seq":953,"reported_epoch":1010,"state":"active+clean","last_fresh":"2024-06-03T09:06:18.490362+0000","last_change":"2024-06-03T08:36:30.883360+0000","last_active":"2024-06-03T09:06:18.490362+0000","last_peered":"2024-06-03T09:06:18.490362+0000","last_clean":"2024-06-03T09:06:18.490362+0000","last_became_active":"2024-06-03T08:36:30.855073+0000","last_became_peered":"2024-06-03T08:36:30.855073+0000","last_unstale":"2024-06-03T09:06:18.490362+0000","last_undegraded":"2024-06-03T09:06:18.490362+0000","last_fullsized":"2024-06-03T09:06:18.490362+0000","mapping_epoch":23,"log_start":"0'0","ondisk_log_start":"0'0","created":23,"last_epoch_clean":25,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-06-03T08:36:28.444764+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-06-03T08:36:28.444764+0000","last_clean_scrub_stamp":"2024-06-03T08:36:28.444764+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-06-04T18:12:22.363473+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,5],"acting":[7,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"17.12","version":"0'0","reported_seq":946,"reported_epoch":1010,"state":"active+clean","last_fresh":"2024-06-03T09:06:18.482303+0000","last_change":"2024-06-03T08:38:01.751562+0000","last_active":"2024-06-03T09:06:18.482303+0000","last_peered":"2024-06-03T09:06:18.482303+0000","last_clean":"2024-06-03T09:06:18.482303+0000","last_became_active":"2024-06-03T08:38:01.745027+0000","last_became_peered":"2024-06-03T08:38:01.745027+0000","last_unstale":"2024-06-03T09:06:18.482303+0000","last_undegraded":"2024-06-03T09:06:18.482303+0000","last_fullsized":"2024-06-03T09:06:18.482303+0000","mapping_epoch":28,"log_start":"0'0","ondisk_log_start":"0'0","created":28,"last_epoch_clean":32,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-06-03T08:37:55.614497+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-06-03T08:37:55.614497+0000","last_clean_scrub_stamp":"2024-06-03T08:37:55.614497+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-06-04T17:54:43.785326+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,3],"acting":[7,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"2.0","version":"0'0","reported_seq":946,"reported_epoch":1012,"state":"active+clean","last_fresh":"2024-06-03T09:06:43.473891+0000","last_change":"2024-06-03T08:36:29.984096+0000","last_active":"2024-06-03T09:06:43.473891+0000","last_peered":"2024-06-03T09:06:43.473891+0000","last_clean":"2024-06-03T09:06:43.473891+0000","last_became_active":"2024-06-03T08:36:29.957875+0000","last_became_peered":"2024-06-03T08:36:29.957875+0000","last_unstale":"2024-06-03T09:06:43.473891+0000","last_undegraded":"2024-06-03T09:06:43.473891+0000","last_fullsized":"2024-06-03T09:06:43.473891+0000","mapping_epoch":23,"log_start":"0'0","ondisk_log_start":"0'0","created":23,"last_epoch_clean":24,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-06-03T08:36:28.444764+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-06-03T08:36:28.444764+0000","last_clean_scrub_stamp":"2024-06-03T08:36:28.444764+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-06-04T08:41:33.679479+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,5],"acting":[3,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"17.13","version":"0'0","reported_seq":946,"reported_epoch":1013,"state":"active+clean","last_fresh":"2024-06-03T09:06:44.996126+0000","last_change":"2024-06-03T08:38:26.838316+0000","last_active":"2024-06-03T09:06:44.996126+0000","last_peered":"2024-06-03T09:06:44.996126+0000","last_clean":"2024-06-03T09:06:44.996126+0000","last_became_active":"2024-06-03T08:38:26.833214+0000","last_became_peered":"2024-06-03T08:38:26.833214+0000","last_unstale":"2024-06-03T09:06:44.996126+0000","last_undegraded":"2024-06-03T09:06:44.996126+0000","last_fullsized":"2024-06-03T09:06:44.996126+0000","mapping_epoch":28,"log_start":"0'0","ondisk_log_start":"0'0","created":28,"last_epoch_clean":52,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-06-03T08:37:55.614497+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-06-03T08:37:55.614497+0000","last_clean_scrub_stamp":"2024-06-03T08:37:55.614497+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-06-04T12:58:51.617998+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,0],"acting":[4,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"2.3","version":"0'0","reported_seq":945,"reported_epoch":1011,"state":"active+clean","last_fresh":"2024-06-03T09:06:42.806748+0000","last_change":"2024-06-03T08:36:30.351670+0000","last_active":"2024-06-03T09:06:42.806748+0000","last_peered":"2024-06-03T09:06:42.806748+0000","last_clean":"2024-06-03T09:06:42.806748+0000","last_became_active":"2024-06-03T08:36:30.343560+0000","last_became_peered":"2024-06-03T08:36:30.343560+0000","last_unstale":"2024-06-03T09:06:42.806748+0000","last_undegraded":"2024-06-03T09:06:42.806748+0000","last_fullsized":"2024-06-03T09:06:42.806748+0000","mapping_epoch":23,"log_start":"0'0","ondisk_log_start":"0'0","created":23,"last_epoch_clean":24,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-06-03T08:36:28.444764+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-06-03T08:36:28.444764+0000","last_clean_scrub_stamp":"2024-06-03T08:36:28.444764+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-06-04T12:02:24.765135+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,2],"acting":[3,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.0","version":"19'76","reported_seq":1053,"reported_epoch":1011,"state":"active+clean","last_fresh":"2024-06-03T09:06:42.852938+0000","last_change":"2024-06-03T08:36:26.702702+0000","last_active":"2024-06-03T09:06:42.852938+0000","last_peered":"2024-06-03T09:06:42.852938+0000","last_clean":"2024-06-03T09:06:42.852938+0000","last_became_active":"2024-06-03T08:36:26.128378+0000","last_became_peered":"2024-06-03T08:36:26.128378+0000","last_unstale":"2024-06-03T09:06:42.852938+0000","last_undegraded":"2024-06-03T09:06:42.852938+0000","last_fullsized":"2024-06-03T09:06:42.852938+0000","mapping_epoch":20,"log_start":"0'0","ondisk_log_start":"0'0","created":16,"last_epoch_clean":22,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-06-03T08:36:15.542635+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-06-03T08:36:15.542635+0000","last_clean_scrub_stamp":"2024-06-03T08:36:15.542635+0000","objects_scrubbed":0,"log_size":76,"log_dups_size":0,"ondisk_log_size":76,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-06-04T12:41:53.731360+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":459280,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":96,"num_read_kb":82,"num_write":113,"num_write_kb":1372,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2,"num_bytes_recovered":397840,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,5],"acting":[1,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"17.10","version":"0'0","reported_seq":947,"reported_epoch":1013,"state":"active+clean","last_fresh":"2024-06-03T09:06:44.974251+0000","last_change":"2024-06-03T08:38:15.399189+0000","last_active":"2024-06-03T09:06:44.974251+0000","last_peered":"2024-06-03T09:06:44.974251+0000","last_clean":"2024-06-03T09:06:44.974251+0000","last_became_active":"2024-06-03T08:38:15.337723+0000","last_became_peered":"2024-06-03T08:38:15.337723+0000","last_unstale":"2024-06-03T09:06:44.974251+0000","last_undegraded":"2024-06-03T09:06:44.974251+0000","last_fullsized":"2024-06-03T09:06:44.974251+0000","mapping_epoch":28,"log_start":"0'0","ondisk_log_start":"0'0","created":28,"last_epoch_clean":42,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-06-03T08:37:55.614497+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-06-03T08:37:55.614497+0000","last_clean_scrub_stamp":"2024-06-03T08:37:55.614497+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-06-04T09:29:46.427317+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,3],"acting":[4,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"17.1a","version":"0'0","reported_seq":971,"reported_epoch":1014,"state":"active+clean","last_fresh":"2024-06-03T09:06:46.393342+0000","last_change":"2024-06-03T09:06:46.393342+0000","last_active":"2024-06-03T09:06:46.393342+0000","last_peered":"2024-06-03T09:06:46.393342+0000","last_clean":"2024-06-03T09:06:46.393342+0000","last_became_active":"2024-06-03T09:06:46.386469+0000","last_became_peered":"2024-06-03T09:06:46.386469+0000","last_unstale":"2024-06-03T09:06:46.393342+0000","last_undegraded":"2024-06-03T09:06:46.393342+0000","last_fullsized":"2024-06-03T09:06:46.393342+0000","mapping_epoch":1013,"log_start":"0'0","ondisk_log_start":"0'0","created":28,"last_epoch_clean":1014,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-06-03T08:37:55.614497+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-06-03T08:37:55.614497+0000","last_clean_scrub_stamp":"2024-06-03T08:37:55.614497+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-06-04T13:22:45.876282+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,6],"acting":[4,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"17.1b","version":"0'0","reported_seq":937,"reported_epoch":1011,"state":"active+clean","last_fresh":"2024-06-03T09:06:42.815385+0000","last_change":"2024-06-03T08:38:03.115748+0000","last_active":"2024-06-03T09:06:42.815385+0000","last_peered":"2024-06-03T09:06:42.815385+0000","last_clean":"2024-06-03T09:06:42.815385+0000","last_became_active":"2024-06-03T08:38:03.110533+0000","last_became_peered":"2024-06-03T08:38:03.110533+0000","last_unstale":"2024-06-03T09:06:42.815385+0000","last_undegraded":"2024-06-03T09:06:42.815385+0000","last_fullsized":"2024-06-03T09:06:42.815385+0000","mapping_epoch":28,"log_start":"0'0","ondisk_log_start":"0'0","created":28,"last_epoch_clean":32,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-06-03T08:37:55.614497+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-06-03T08:37:55.614497+0000","last_clean_scrub_stamp":"2024-06-03T08:37:55.614497+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-06-04T12:36:47.471609+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,7],"acting":[3,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"17.18","version":"0'0","reported_seq":945,"reported_epoch":1010,"state":"active+clean","last_fresh":"2024-06-03T09:06:18.484214+0000","last_change":"2024-06-03T08:38:04.632864+0000","last_active":"2024-06-03T09:06:18.484214+0000","last_peered":"2024-06-03T09:06:18.484214+0000","last_clean":"2024-06-03T09:06:18.484214+0000","last_became_active":"2024-06-03T08:38:04.627856+0000","last_became_peered":"2024-06-03T08:38:04.627856+0000","last_unstale":"2024-06-03T09:06:18.484214+0000","last_undegraded":"2024-06-03T09:06:18.484214+0000","last_fullsized":"2024-06-03T09:06:18.484214+0000","mapping_epoch":28,"log_start":"0'0","ondisk_log_start":"0'0","created":28,"last_epoch_clean":33,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-06-03T08:37:55.614497+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-06-03T08:37:55.614497+0000","last_clean_scrub_stamp":"2024-06-03T08:37:55.614497+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-06-04T19:55:51.181598+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,4],"acting":[7,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"17.19","version":"0'0","reported_seq":936,"reported_epoch":1011,"state":"active+clean","last_fresh":"2024-06-03T09:06:42.812989+0000","last_change":"2024-06-03T08:38:01.839491+0000","last_active":"2024-06-03T09:06:42.812989+0000","last_peered":"2024-06-03T09:06:42.812989+0000","last_clean":"2024-06-03T09:06:42.812989+0000","last_became_active":"2024-06-03T08:38:01.829539+0000","last_became_peered":"2024-06-03T08:38:01.829539+0000","last_unstale":"2024-06-03T09:06:42.812989+0000","last_undegraded":"2024-06-03T09:06:42.812989+0000","last_fullsized":"2024-06-03T09:06:42.812989+0000","mapping_epoch":28,"log_start":"0'0","ondisk_log_start":"0'0","created":28,"last_epoch_clean":30,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-06-03T08:37:55.614497+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-06-03T08:37:55.614497+0000","last_clean_scrub_stamp":"2024-06-03T08:37:55.614497+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-06-04T09:10:43.860229+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,2],"acting":[3,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"17.1e","version":"0'0","reported_seq":844,"reported_epoch":1014,"state":"active+clean","last_fresh":"2024-06-03T09:06:46.297025+0000","last_change":"2024-06-03T09:06:46.297025+0000","last_active":"2024-06-03T09:06:46.297025+0000","last_peered":"2024-06-03T09:06:46.297025+0000","last_clean":"2024-06-03T09:06:46.297025+0000","last_became_active":"2024-06-03T09:06:46.288593+0000","last_became_peered":"2024-06-03T09:06:46.288593+0000","last_unstale":"2024-06-03T09:06:46.297025+0000","last_undegraded":"2024-06-03T09:06:46.297025+0000","last_fullsized":"2024-06-03T09:06:46.297025+0000","mapping_epoch":1013,"log_start":"0'0","ondisk_log_start":"0'0","created":28,"last_epoch_clean":1014,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-06-03T08:37:55.614497+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-06-03T08:37:55.614497+0000","last_clean_scrub_stamp":"2024-06-03T08:37:55.614497+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-06-04T20:09:30.144366+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,4],"acting":[6,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"17.1f","version":"0'0","reported_seq":941,"reported_epoch":1009,"state":"active+clean","last_fresh":"2024-06-03T09:06:14.450053+0000","last_change":"2024-06-03T08:38:04.383538+0000","last_active":"2024-06-03T09:06:14.450053+0000","last_peered":"2024-06-03T09:06:14.450053+0000","last_clean":"2024-06-03T09:06:14.450053+0000","last_became_active":"2024-06-03T08:38:04.367278+0000","last_became_peered":"2024-06-03T08:38:04.367278+0000","last_unstale":"2024-06-03T09:06:14.450053+0000","last_undegraded":"2024-06-03T09:06:14.450053+0000","last_fullsized":"2024-06-03T09:06:14.450053+0000","mapping_epoch":28,"log_start":"0'0","ondisk_log_start":"0'0","created":28,"last_epoch_clean":32,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-06-03T08:37:55.614497+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-06-03T08:37:55.614497+0000","last_clean_scrub_stamp":"2024-06-03T08:37:55.614497+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-06-04T10:39:26.552282+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,2],"acting":[0,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"17.1c","version":"0'0","reported_seq":956,"reported_epoch":1014,"state":"active+clean","last_fresh":"2024-06-03T09:06:46.410116+0000","last_change":"2024-06-03T09:06:46.410116+0000","last_active":"2024-06-03T09:06:46.410116+0000","last_peered":"2024-06-03T09:06:46.410116+0000","last_clean":"2024-06-03T09:06:46.410116+0000","last_became_active":"2024-06-03T09:06:46.403724+0000","last_became_peered":"2024-06-03T09:06:46.403724+0000","last_unstale":"2024-06-03T09:06:46.410116+0000","last_undegraded":"2024-06-03T09:06:46.410116+0000","last_fullsized":"2024-06-03T09:06:46.410116+0000","mapping_epoch":1013,"log_start":"0'0","ondisk_log_start":"0'0","created":28,"last_epoch_clean":1014,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-06-03T08:37:55.614497+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-06-03T08:37:55.614497+0000","last_clean_scrub_stamp":"2024-06-03T08:37:55.614497+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-06-04T19:40:47.504202+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,6],"acting":[5,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"17.1d","version":"0'0","reported_seq":925,"reported_epoch":1011,"state":"active+clean","last_fresh":"2024-06-03T09:06:42.858234+0000","last_change":"2024-06-03T08:38:06.794630+0000","last_active":"2024-06-03T09:06:42.858234+0000","last_peered":"2024-06-03T09:06:42.858234+0000","last_clean":"2024-06-03T09:06:42.858234+0000","last_became_active":"2024-06-03T08:38:06.592317+0000","last_became_peered":"2024-06-03T08:38:06.592317+0000","last_unstale":"2024-06-03T09:06:42.858234+0000","last_undegraded":"2024-06-03T09:06:42.858234+0000","last_fullsized":"2024-06-03T09:06:42.858234+0000","mapping_epoch":28,"log_start":"0'0","ondisk_log_start":"0'0","created":28,"last_epoch_clean":33,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-06-03T08:37:55.614497+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-06-03T08:37:55.614497+0000","last_clean_scrub_stamp":"2024-06-03T08:37:55.614497+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-06-04T12:33:30.785369+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,7],"acting":[1,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"17.2","version":"0'0","reported_seq":927,"reported_epoch":1011,"state":"active+clean","last_fresh":"2024-06-03T09:06:42.859877+0000","last_change":"2024-06-03T08:38:14.981410+0000","last_active":"2024-06-03T09:06:42.859877+0000","last_peered":"2024-06-03T09:06:42.859877+0000","last_clean":"2024-06-03T09:06:42.859877+0000","last_became_active":"2024-06-03T08:38:14.973825+0000","last_became_peered":"2024-06-03T08:38:14.973825+0000","last_unstale":"2024-06-03T09:06:42.859877+0000","last_undegraded":"2024-06-03T09:06:42.859877+0000","last_fullsized":"2024-06-03T09:06:42.859877+0000","mapping_epoch":28,"log_start":"0'0","ondisk_log_start":"0'0","created":28,"last_epoch_clean":41,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-06-03T08:37:55.614497+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-06-03T08:37:55.614497+0000","last_clean_scrub_stamp":"2024-06-03T08:37:55.614497+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-06-04T17:01:05.518613+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,4],"acting":[1,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"17.3","version":"0'0","reported_seq":943,"reported_epoch":1009,"state":"active+clean","last_fresh":"2024-06-03T09:06:14.437636+0000","last_change":"2024-06-03T08:38:14.619745+0000","last_active":"2024-06-03T09:06:14.437636+0000","last_peered":"2024-06-03T09:06:14.437636+0000","last_clean":"2024-06-03T09:06:14.437636+0000","last_became_active":"2024-06-03T08:38:14.510665+0000","last_became_peered":"2024-06-03T08:38:14.510665+0000","last_unstale":"2024-06-03T09:06:14.437636+0000","last_undegraded":"2024-06-03T09:06:14.437636+0000","last_fullsized":"2024-06-03T09:06:14.437636+0000","mapping_epoch":28,"log_start":"0'0","ondisk_log_start":"0'0","created":28,"last_epoch_clean":42,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-06-03T08:37:55.614497+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-06-03T08:37:55.614497+0000","last_clean_scrub_stamp":"2024-06-03T08:37:55.614497+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-06-04T12:03:47.392064+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,7],"acting":[0,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"17.0","version":"0'0","reported_seq":944,"reported_epoch":1009,"state":"active+clean","last_fresh":"2024-06-03T09:06:14.427123+0000","last_change":"2024-06-03T08:38:04.237118+0000","last_active":"2024-06-03T09:06:14.427123+0000","last_peered":"2024-06-03T09:06:14.427123+0000","last_clean":"2024-06-03T09:06:14.427123+0000","last_became_active":"2024-06-03T08:38:04.231954+0000","last_became_peered":"2024-06-03T08:38:04.231954+0000","last_unstale":"2024-06-03T09:06:14.427123+0000","last_undegraded":"2024-06-03T09:06:14.427123+0000","last_fullsized":"2024-06-03T09:06:14.427123+0000","mapping_epoch":28,"log_start":"0'0","ondisk_log_start":"0'0","created":28,"last_epoch_clean":32,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-06-03T08:37:55.614497+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-06-03T08:37:55.614497+0000","last_clean_scrub_stamp":"2024-06-03T08:37:55.614497+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-06-04T09:56:10.741637+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,2],"acting":[0,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"17.1","version":"65'7","reported_seq":971,"reported_epoch":1014,"state":"active+clean","last_fresh":"2024-06-03T09:06:46.389760+0000","last_change":"2024-06-03T09:06:46.389760+0000","last_active":"2024-06-03T09:06:46.389760+0000","last_peered":"2024-06-03T09:06:46.389760+0000","last_clean":"2024-06-03T09:06:46.389760+0000","last_became_active":"2024-06-03T09:06:46.383290+0000","last_became_peered":"2024-06-03T09:06:46.383290+0000","last_unstale":"2024-06-03T09:06:46.389760+0000","last_undegraded":"2024-06-03T09:06:46.389760+0000","last_fullsized":"2024-06-03T09:06:46.389760+0000","mapping_epoch":1013,"log_start":"0'0","ondisk_log_start":"0'0","created":28,"last_epoch_clean":1014,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-06-03T08:37:55.614497+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-06-03T08:37:55.614497+0000","last_clean_scrub_stamp":"2024-06-03T08:37:55.614497+0000","objects_scrubbed":0,"log_size":7,"log_dups_size":0,"ondisk_log_size":7,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-06-04T10:38:34.982254+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":5,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":6,"num_read_kb":6,"num_write":7,"num_write_kb":7,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,6],"acting":[5,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]}],"pool_stats":[{"poolid":2,"num_pg":8,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":15,"acting":15,"num_store_stats":7},{"poolid":1,"num_pg":1,"stat_sum":{"num_bytes":459280,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":96,"num_read_kb":82,"num_write":113,"num_write_kb":1372,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2,"num_bytes_recovered":397840,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":802816,"data_stored":795680,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":76,"ondisk_log_size":76,"up":2,"acting":2,"num_store_stats":2},{"poolid":17,"num_pg":32,"stat_sum":{"num_bytes":5,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":6,"num_read_kb":6,"num_write":7,"num_write_kb":7,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":8192,"data_stored":10,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":7,"ondisk_log_size":7,"up":60,"acting":60,"num_store_stats":8}],"osd_stats":[{"osd":6,"up_from":1013,"seq":4350801871162,"num_pgs":14,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":54908,"kb_used_data":5744,"kb_used_omap":1,"kb_used_meta":49150,"kb_avail":94316932,"statfs":{"total":96636764160,"available":96580538368,"internally_reserved":0,"allocated":5881856,"data_stored":2757601,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1593,"internal_metadata":50330055},"hb_peers":[0,1,2,3,4,5,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":403,"apply_latency_ms":403,"commit_latency_ns":403000000,"apply_latency_ns":403000000},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Mon Jun 3 09:06:42 2024","interfaces":[{"interface":"back","average":{"1min":5696.3109999999997,"5min":1469.3900000000001,"15min":503.11700000000002},"min":{"1min":5.242,"5min":4.4829999999999997,"15min":4.1959999999999997},"max":{"1min":20463.029999999999,"5min":22766.882000000001,"15min":22766.882000000001},"last":7.3399999999999999},{"interface":"front","average":{"1min":5687.6390000000001,"5min":1468.1669999999999,"15min":504.13200000000001},"min":{"1min":4.835,"5min":4.8129999999999997,"15min":4.4530000000000003},"max":{"1min":20461.830000000002,"5min":22766.289000000001,"15min":22766.289000000001},"last":8.1010000000000009}]},{"osd":1,"last update":"Mon Jun 3 09:06:42 2024","interfaces":[{"interface":"back","average":{"1min":36.496000000000002,"5min":80.162999999999997,"15min":134.59399999999999},"min":{"1min":5.9189999999999996,"5min":5.8799999999999999,"15min":5.7119999999999997},"max":{"1min":544.85799999999995,"5min":4138.1679999999997,"15min":9955.1299999999992},"last":9.4870000000000001},{"interface":"front","average":{"1min":38.866999999999997,"5min":82.631,"15min":135.11500000000001},"min":{"1min":6.5350000000000001,"5min":6.5350000000000001,"15min":6.1390000000000002},"max":{"1min":546.73099999999999,"5min":4137.6530000000002,"15min":9827.5509999999995},"last":15.617000000000001}]},{"osd":2,"last update":"Mon Jun 3 09:06:42 2024","interfaces":[{"interface":"back","average":{"1min":41.093000000000004,"5min":27.917000000000002,"15min":22.329000000000001},"min":{"1min":6.0670000000000002,"5min":6.0670000000000002,"15min":6.0670000000000002},"max":{"1min":548.12900000000002,"5min":548.12900000000002,"15min":548.12900000000002},"last":27.196999999999999},{"interface":"front","average":{"1min":30.798999999999999,"5min":24.276,"15min":22.488},"min":{"1min":6.4370000000000003,"5min":6.4370000000000003,"15min":6.1870000000000003},"max":{"1min":369.14499999999998,"5min":369.14499999999998,"15min":434.77300000000002},"last":28.960999999999999}]},{"osd":3,"last update":"Mon Jun 3 09:06:42 2024","interfaces":[{"interface":"back","average":{"1min":41.411999999999999,"5min":27.434999999999999,"15min":30.657},"min":{"1min":10.061,"5min":8.3179999999999996,"15min":6.1719999999999997},"max":{"1min":549.024,"5min":549.024,"15min":1235.5799999999999},"last":16.423999999999999},{"interface":"front","average":{"1min":37.850999999999999,"5min":22.001000000000001,"15min":28.015999999999998},"min":{"1min":6.0679999999999996,"5min":6.0679999999999996,"15min":6.0679999999999996},"max":{"1min":559.25,"5min":559.25,"15min":1157.6780000000001},"last":14.327}]},{"osd":4,"last update":"Mon Jun 3 09:06:42 2024","interfaces":[{"interface":"back","average":{"1min":165.62899999999999,"5min":1281.1010000000001,"15min":440.59199999999998},"min":{"1min":10.965999999999999,"5min":6.093,"15min":6.093},"max":{"1min":384.46899999999999,"5min":28093.096000000001,"15min":28093.096000000001},"last":11.307},{"interface":"front","average":{"1min":186.97900000000001,"5min":1308.886,"15min":454.97699999999998},"min":{"1min":13.581,"5min":7.7480000000000002,"15min":7.7480000000000002},"max":{"1min":445.096,"5min":28323.837,"15min":28323.837},"last":17.109000000000002}]},{"osd":5,"last update":"Mon Jun 3 09:06:42 2024","interfaces":[{"interface":"back","average":{"1min":81.581000000000003,"5min":34.863,"15min":28.350999999999999},"min":{"1min":10.345000000000001,"5min":8.625,"15min":7.516},"max":{"1min":455.84199999999998,"5min":455.84199999999998,"15min":455.84199999999998},"last":17.791},{"interface":"front","average":{"1min":83.391000000000005,"5min":39.277999999999999,"15min":27.885000000000002},"min":{"1min":8.0280000000000005,"5min":6.266,"15min":6.266},"max":{"1min":550.02300000000002,"5min":550.02300000000002,"15min":550.02300000000002},"last":11.814}]},{"osd":7,"last update":"Mon Jun 3 09:06:42 2024","interfaces":[{"interface":"back","average":{"1min":44.052999999999997,"5min":33.697000000000003,"15min":29.731999999999999},"min":{"1min":9.1500000000000004,"5min":9.1500000000000004,"15min":9.1500000000000004},"max":{"1min":552.09500000000003,"5min":552.09500000000003,"15min":552.09500000000003},"last":18.486999999999998},{"interface":"front","average":{"1min":47.076999999999998,"5min":34.226999999999997,"15min":29.827000000000002},"min":{"1min":9.7780000000000005,"5min":8.6709999999999994,"15min":8.5820000000000007},"max":{"1min":531.08600000000001,"5min":531.08600000000001,"15min":531.08600000000001},"last":13.776}]}]},{"osd":7,"up_from":20,"seq":85899346290,"num_pgs":9,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":58964,"kb_used_data":5704,"kb_used_omap":1,"kb_used_meta":53246,"kb_avail":94312876,"statfs":{"total":96636764160,"available":96576385024,"internally_reserved":0,"allocated":5840896,"data_stored":2740727,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1589,"internal_metadata":54524363},"hb_peers":[0,1,2,3,4,5],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":5,"apply_latency_ms":5,"commit_latency_ns":5000000,"apply_latency_ns":5000000},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Mon Jun 3 09:06:09 2024","interfaces":[{"interface":"back","average":{"1min":14.073,"5min":894.22400000000005,"15min":308.93400000000003},"min":{"1min":5.5709999999999997,"5min":4.218,"15min":4.0599999999999996},"max":{"1min":90.757999999999996,"5min":21729.001,"15min":21729.001},"last":9.4469999999999992},{"interface":"front","average":{"1min":9.2550000000000008,"5min":906.23500000000001,"15min":314.06},"min":{"1min":5.5789999999999997,"5min":4.8810000000000002,"15min":4.8810000000000002},"max":{"1min":25.937000000000001,"5min":21899.732,"15min":21899.732},"last":4.5979999999999999}]},{"osd":1,"last update":"Mon Jun 3 09:06:09 2024","interfaces":[{"interface":"back","average":{"1min":15.182,"5min":69.183999999999997,"15min":219.173},"min":{"1min":7.3639999999999999,"5min":6.6200000000000001,"15min":6.1319999999999997},"max":{"1min":76.926000000000002,"5min":2795.1329999999998,"15min":18373.257000000001},"last":5.8630000000000004},{"interface":"front","average":{"1min":21.495999999999999,"5min":74.703999999999994,"15min":225.285},"min":{"1min":8.8399999999999999,"5min":8.8399999999999999,"15min":8.5129999999999999},"max":{"1min":94.977999999999994,"5min":2803.3670000000002,"15min":18374.601999999999},"last":10.94}]},{"osd":2,"last update":"Mon Jun 3 09:06:09 2024","interfaces":[{"interface":"back","average":{"1min":19.164000000000001,"5min":26.902000000000001,"15min":22.920999999999999},"min":{"1min":11.207000000000001,"5min":8.5030000000000001,"15min":8.5030000000000001},"max":{"1min":91.319999999999993,"5min":333.99200000000002,"15min":333.99200000000002},"last":10.095000000000001},{"interface":"front","average":{"1min":11.125999999999999,"5min":18.132000000000001,"15min":16.353000000000002},"min":{"1min":6.202,"5min":5.8369999999999997,"15min":5.5069999999999997},"max":{"1min":28.649000000000001,"5min":226.18100000000001,"15min":226.18100000000001},"last":12.922000000000001}]},{"osd":3,"last update":"Mon Jun 3 09:06:09 2024","interfaces":[{"interface":"back","average":{"1min":22.291,"5min":26.289999999999999,"15min":31.074999999999999},"min":{"1min":11.714,"5min":6.8959999999999999,"15min":6.3760000000000003},"max":{"1min":83.102000000000004,"5min":316.64400000000001,"15min":2039.9059999999999},"last":7.617},{"interface":"front","average":{"1min":13.324999999999999,"5min":17.963000000000001,"15min":25.969999999999999},"min":{"1min":6.1200000000000001,"5min":5.2089999999999996,"15min":5.2089999999999996},"max":{"1min":49.908999999999999,"5min":233.542,"15min":2037.5550000000001},"last":8.1479999999999997}]},{"osd":4,"last update":"Mon Jun 3 09:06:29 2024","interfaces":[{"interface":"back","average":{"1min":17.675999999999998,"5min":1026.325,"15min":472.47199999999998},"min":{"1min":7.306,"5min":5.9089999999999998,"15min":5.9089999999999998},"max":{"1min":140.03299999999999,"5min":28969.369999999999,"15min":30073.917000000001},"last":12.000999999999999},{"interface":"front","average":{"1min":17.890999999999998,"5min":1044.5139999999999,"15min":480.226},"min":{"1min":5.9279999999999999,"5min":5.9279999999999999,"15min":5.9279999999999999},"max":{"1min":141.179,"5min":29244.138999999999,"15min":30348.021000000001},"last":15.622}]},{"osd":5,"last update":"Mon Jun 3 09:06:02 2024","interfaces":[{"interface":"back","average":{"1min":24.372,"5min":26.856000000000002,"15min":23.571999999999999},"min":{"1min":12.278,"5min":9.2460000000000004,"15min":6.5650000000000004},"max":{"1min":183.64099999999999,"5min":317.572,"15min":317.572},"last":11.449999999999999},{"interface":"front","average":{"1min":21.259,"5min":23.364000000000001,"15min":21.193999999999999},"min":{"1min":7.8499999999999996,"5min":7.0999999999999996,"15min":7.008},"max":{"1min":184.53399999999999,"5min":306.60599999999999,"15min":306.60599999999999},"last":14.676}]},{"osd":6,"last update":"Mon Jun 3 09:04:31 2024","interfaces":[{"interface":"back","average":{"1min":44.061,"5min":26.821000000000002,"15min":91.176000000000002},"min":{"1min":9.0800000000000001,"5min":7.9390000000000001,"15min":7.9390000000000001},"max":{"1min":268.029,"5min":308.916,"15min":8734.4639999999999},"last":6.6769999999999996},{"interface":"front","average":{"1min":46.643000000000001,"5min":23.879999999999999,"15min":88.765000000000001},"min":{"1min":8.8119999999999994,"5min":6.9240000000000004,"15min":6.742},"max":{"1min":305.64600000000002,"5min":305.64600000000002,"15min":8735.5319999999992},"last":12.327}]}]},{"osd":5,"up_from":20,"seq":85899346265,"num_pgs":9,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":59416,"kb_used_data":6148,"kb_used_omap":1,"kb_used_meta":53246,"kb_avail":94312424,"statfs":{"total":96636764160,"available":96575922176,"internally_reserved":0,"allocated":6295552,"data_stored":3160918,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1594,"internal_metadata":54524358},"hb_peers":[0,1,2,3,4,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":23,"apply_latency_ms":23,"commit_latency_ns":23000000,"apply_latency_ns":23000000},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Mon Jun 3 09:06:06 2024","interfaces":[{"interface":"back","average":{"1min":7.468,"5min":1703.037,"15min":580.51900000000001},"min":{"1min":4.6509999999999998,"5min":4.6509999999999998,"15min":4.4020000000000001},"max":{"1min":14.711,"5min":26433.764999999999,"15min":26433.764999999999},"last":7.9779999999999998},{"interface":"front","average":{"1min":9.9429999999999996,"5min":1725.27,"15min":587.87800000000004},"min":{"1min":5.5999999999999996,"5min":5.1319999999999997,"15min":4.8170000000000002},"max":{"1min":13.590999999999999,"5min":26704.698,"15min":26704.698},"last":9.2810000000000006}]},{"osd":1,"last update":"Mon Jun 3 09:06:00 2024","interfaces":[{"interface":"back","average":{"1min":10.862,"5min":67.731999999999999,"15min":217.364},"min":{"1min":7.1769999999999996,"5min":6.0890000000000004,"15min":5.4930000000000003},"max":{"1min":16.998999999999999,"5min":3923.4090000000001,"15min":18811.696},"last":7.1600000000000001},{"interface":"front","average":{"1min":10.417,"5min":65.141999999999996,"15min":220.75700000000001},"min":{"1min":5.3209999999999997,"5min":5.3209999999999997,"15min":5.0209999999999999},"max":{"1min":15.821999999999999,"5min":4019.6129999999998,"15min":18968.365000000002},"last":6.298}]},{"osd":2,"last update":"Mon Jun 3 09:06:00 2024","interfaces":[{"interface":"back","average":{"1min":11.366,"5min":24.18,"15min":21.137},"min":{"1min":5.6369999999999996,"5min":5.6369999999999996,"15min":5.0510000000000002},"max":{"1min":15.529999999999999,"5min":332.48200000000003,"15min":332.48200000000003},"last":13.682},{"interface":"front","average":{"1min":10.089,"5min":23.117000000000001,"15min":24.050000000000001},"min":{"1min":6.6769999999999996,"5min":6.6769999999999996,"15min":5.8739999999999997},"max":{"1min":15.821,"5min":353.03199999999998,"15min":353.03199999999998},"last":11.372}]},{"osd":3,"last update":"Mon Jun 3 09:06:00 2024","interfaces":[{"interface":"back","average":{"1min":13.472,"5min":24.283999999999999,"15min":31.666},"min":{"1min":10.111000000000001,"5min":7.4720000000000004,"15min":7.4210000000000003},"max":{"1min":16.271999999999998,"5min":353.57499999999999,"15min":2855.8519999999999},"last":12.367000000000001},{"interface":"front","average":{"1min":9.5899999999999999,"5min":15.875,"15min":25.834},"min":{"1min":6.6609999999999996,"5min":5.7729999999999997,"15min":5.1909999999999998},"max":{"1min":15.112,"5min":203.804,"15min":2869.7860000000001},"last":15.904999999999999}]},{"osd":4,"last update":"Mon Jun 3 09:06:29 2024","interfaces":[{"interface":"back","average":{"1min":11.067,"5min":780.21000000000004,"15min":377.28399999999999},"min":{"1min":5.4779999999999998,"5min":5.4779999999999998,"15min":5.4779999999999998},"max":{"1min":14.654999999999999,"5min":23315.438999999998,"15min":28619.825000000001},"last":10.768000000000001},{"interface":"front","average":{"1min":14.349,"5min":782.97299999999996,"15min":377.93400000000003},"min":{"1min":7.492,"5min":6.5030000000000001,"15min":6.5030000000000001},"max":{"1min":17.707000000000001,"5min":23317.502,"15min":28621.294000000002},"last":14.840999999999999}]},{"osd":6,"last update":"Mon Jun 3 09:03:57 2024","interfaces":[{"interface":"back","average":{"1min":23.536000000000001,"5min":36.457000000000001,"15min":83.799999999999997},"min":{"1min":9.4800000000000004,"5min":9.4800000000000004,"15min":8.0630000000000006},"max":{"1min":89.442999999999998,"5min":508.77999999999997,"15min":9193.5169999999998},"last":9.8670000000000009},{"interface":"front","average":{"1min":18.891999999999999,"5min":22.100999999999999,"15min":75.489999999999995},"min":{"1min":5.923,"5min":5.923,"15min":5.923},"max":{"1min":78.424999999999997,"5min":464.62,"15min":9191.4429999999993},"last":16.568000000000001}]},{"osd":7,"last update":"Mon Jun 3 09:06:00 2024","interfaces":[{"interface":"back","average":{"1min":13.257,"5min":20.917999999999999,"15min":21.088999999999999},"min":{"1min":10.475,"5min":6.7629999999999999,"15min":6.7629999999999999},"max":{"1min":17.245000000000001,"5min":211.22999999999999,"15min":211.22999999999999},"last":13.07},{"interface":"front","average":{"1min":12.872999999999999,"5min":20.587,"15min":24.021000000000001},"min":{"1min":8.5760000000000005,"5min":7.7709999999999999,"15min":6.8570000000000002},"max":{"1min":18.359999999999999,"5min":223.191,"15min":286.69499999999999},"last":15.362}]}]},{"osd":4,"up_from":20,"seq":85899346273,"num_pgs":12,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":63112,"kb_used_data":5752,"kb_used_omap":1,"kb_used_meta":57342,"kb_avail":94308728,"statfs":{"total":96636764160,"available":96572137472,"internally_reserved":0,"allocated":5890048,"data_stored":2763073,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1594,"internal_metadata":58718662},"hb_peers":[0,1,2,3,5,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":20,"apply_latency_ms":20,"commit_latency_ns":20000000,"apply_latency_ns":20000000},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Mon Jun 3 09:06:35 2024","interfaces":[{"interface":"back","average":{"1min":10.417,"5min":1093.547,"15min":381.64499999999998},"min":{"1min":4.6900000000000004,"5min":4.5279999999999996,"15min":4.3220000000000001},"max":{"1min":52.241,"5min":21726.743999999999,"15min":21726.743999999999},"last":5.0679999999999996},{"interface":"front","average":{"1min":13.175000000000001,"5min":1102.4200000000001,"15min":387.161},"min":{"1min":6.1950000000000003,"5min":5.8380000000000001,"15min":5.4909999999999997},"max":{"1min":53.148000000000003,"5min":21727.993999999999,"15min":21727.993999999999},"last":21.422000000000001}]},{"osd":1,"last update":"Mon Jun 3 09:06:35 2024","interfaces":[{"interface":"back","average":{"1min":9.9909999999999997,"5min":53.250999999999998,"15min":340.02999999999997},"min":{"1min":4.5670000000000002,"5min":4.1970000000000001,"15min":4.1970000000000001},"max":{"1min":47.073,"5min":3346.4389999999999,"15min":22256.098000000002},"last":6.5800000000000001},{"interface":"front","average":{"1min":12.189,"5min":53.375999999999998,"15min":339.68799999999999},"min":{"1min":6.1539999999999999,"5min":5.0590000000000002,"15min":4.2640000000000002},"max":{"1min":49.558,"5min":3347.181,"15min":22247.317999999999},"last":26.995000000000001}]},{"osd":2,"last update":"Mon Jun 3 09:06:35 2024","interfaces":[{"interface":"back","average":{"1min":15.178000000000001,"5min":17.202000000000002,"15min":20.417999999999999},"min":{"1min":6.3010000000000002,"5min":6.3010000000000002,"15min":6.3010000000000002},"max":{"1min":85.340000000000003,"5min":337.60399999999998,"15min":393.58600000000001},"last":25.855},{"interface":"front","average":{"1min":15.423,"5min":14.432,"15min":22.754000000000001},"min":{"1min":6.4889999999999999,"5min":6.0629999999999997,"15min":6.0629999999999997},"max":{"1min":82.120000000000005,"5min":127.913,"15min":394.471},"last":26.456}]},{"osd":3,"last update":"Mon Jun 3 09:06:35 2024","interfaces":[{"interface":"back","average":{"1min":17.651,"5min":15.5,"15min":18.079000000000001},"min":{"1min":10.359999999999999,"5min":5.8719999999999999,"15min":5.4000000000000004},"max":{"1min":84.555999999999997,"5min":127.43600000000001,"15min":322.63},"last":29.57},{"interface":"front","average":{"1min":16.413,"5min":17.356999999999999,"15min":22.920000000000002},"min":{"1min":9.7349999999999994,"5min":7.484,"15min":7.484},"max":{"1min":71.394000000000005,"5min":204.517,"15min":392.67500000000001},"last":28.472000000000001}]},{"osd":5,"last update":"Mon Jun 3 09:06:35 2024","interfaces":[{"interface":"back","average":{"1min":15.672000000000001,"5min":16.742999999999999,"15min":20.795000000000002},"min":{"1min":8.093,"5min":6.7489999999999997,"15min":6.7489999999999997},"max":{"1min":51.326000000000001,"5min":268.10199999999998,"15min":344.23399999999998},"last":30.233000000000001},{"interface":"front","average":{"1min":17.297999999999998,"5min":19.356999999999999,"15min":25.713999999999999},"min":{"1min":12.875,"5min":8.5180000000000007,"15min":8.5180000000000007},"max":{"1min":40.829000000000001,"5min":264.75,"15min":345.18700000000001},"last":31.635999999999999}]},{"osd":6,"last update":"Mon Jun 3 09:04:32 2024","interfaces":[{"interface":"back","average":{"1min":15.548,"5min":32.957999999999998,"15min":152.738},"min":{"1min":9.141,"5min":9.141,"15min":8.2319999999999993},"max":{"1min":52.57,"5min":395.238,"15min":12496.335999999999},"last":208.90000000000001},{"interface":"front","average":{"1min":12.680999999999999,"5min":25.428000000000001,"15min":149.30600000000001},"min":{"1min":6.4379999999999997,"5min":5.6319999999999997,"15min":5.6319999999999997},"max":{"1min":46.283000000000001,"5min":377.74299999999999,"15min":12616.297},"last":205.88900000000001}]},{"osd":7,"last update":"Mon Jun 3 09:06:35 2024","interfaces":[{"interface":"back","average":{"1min":9.6969999999999992,"5min":14.196,"15min":18.809999999999999},"min":{"1min":6.7190000000000003,"5min":6.2110000000000003,"15min":5.8010000000000002},"max":{"1min":15.960000000000001,"5min":207.25299999999999,"15min":358.94900000000001},"last":5.7249999999999996},{"interface":"front","average":{"1min":17.696000000000002,"5min":19.024999999999999,"15min":23.259},"min":{"1min":11.794,"5min":7.7789999999999999,"15min":7.7789999999999999},"max":{"1min":45.639000000000003,"5min":209.32900000000001,"15min":380.06900000000002},"last":30.835000000000001}]}]},{"osd":2,"up_from":20,"seq":85899346292,"num_pgs":11,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":58988,"kb_used_data":5728,"kb_used_omap":1,"kb_used_meta":53246,"kb_avail":94312852,"statfs":{"total":96636764160,"available":96576360448,"internally_reserved":0,"allocated":5865472,"data_stored":2751635,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1594,"internal_metadata":54524358},"hb_peers":[0,1,3,4,5,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":4,"apply_latency_ms":4,"commit_latency_ns":4000000,"apply_latency_ns":4000000},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Mon Jun 3 09:06:23 2024","interfaces":[{"interface":"back","average":{"1min":8.9629999999999992,"5min":946.25400000000002,"15min":326.36099999999999},"min":{"1min":5.3479999999999999,"5min":4.7050000000000001,"15min":4.1420000000000003},"max":{"1min":26.015000000000001,"5min":23465.18,"15min":23465.18},"last":8.7959999999999994},{"interface":"front","average":{"1min":7.6130000000000004,"5min":944.39800000000002,"15min":326.178},"min":{"1min":4.6790000000000003,"5min":4.6790000000000003,"15min":4.6790000000000003},"max":{"1min":20.802,"5min":23465.966,"15min":23465.966},"last":5.6059999999999999}]},{"osd":1,"last update":"Mon Jun 3 09:06:23 2024","interfaces":[{"interface":"back","average":{"1min":13.007,"5min":76.507000000000005,"15min":294.72899999999998},"min":{"1min":6.4089999999999998,"5min":5.4199999999999999,"15min":5.4199999999999999},"max":{"1min":85.430000000000007,"5min":4220.5290000000005,"15min":18954.987000000001},"last":11.789},{"interface":"front","average":{"1min":15.894,"5min":82.119,"15min":302.92399999999998},"min":{"1min":7.367,"5min":7.367,"15min":7.367},"max":{"1min":98.498000000000005,"5min":4325.9489999999996,"15min":19143.834999999999},"last":15.368}]},{"osd":3,"last update":"Mon Jun 3 09:06:23 2024","interfaces":[{"interface":"back","average":{"1min":9.7579999999999991,"5min":12.444000000000001,"15min":28.065000000000001},"min":{"1min":7,"5min":5.9290000000000003,"15min":5.2450000000000001},"max":{"1min":28.164000000000001,"5min":118.913,"15min":2594.8150000000001},"last":13.339},{"interface":"front","average":{"1min":12.321999999999999,"5min":15.25,"15min":31.349},"min":{"1min":9.1240000000000006,"5min":8.0760000000000005,"15min":6.0780000000000003},"max":{"1min":32.360999999999997,"5min":156.34700000000001,"15min":2706.5120000000002},"last":14.712}]},{"osd":4,"last update":"Mon Jun 3 09:05:57 2024","interfaces":[{"interface":"back","average":{"1min":10.573,"5min":1266.894,"15min":438.154},"min":{"1min":8.6379999999999999,"5min":7.8789999999999996,"15min":6.2039999999999997},"max":{"1min":15.105,"5min":26660.861000000001,"15min":26660.861000000001},"last":7.8780000000000001},{"interface":"front","average":{"1min":12.407999999999999,"5min":1266.7460000000001,"15min":435.75400000000002},"min":{"1min":10.266999999999999,"5min":10.266999999999999,"15min":6.2720000000000002},"max":{"1min":15.228,"5min":26616.243999999999,"15min":26616.243999999999},"last":10.970000000000001}]},{"osd":5,"last update":"Mon Jun 3 09:06:23 2024","interfaces":[{"interface":"back","average":{"1min":14.103,"5min":18.091000000000001,"15min":21.119},"min":{"1min":9.6349999999999998,"5min":9.2159999999999993,"15min":6.2050000000000001},"max":{"1min":32.890000000000001,"5min":252.31100000000001,"15min":252.31100000000001},"last":9.3249999999999993},{"interface":"front","average":{"1min":11.545999999999999,"5min":16.170999999999999,"15min":20.254000000000001},"min":{"1min":7.6070000000000002,"5min":7.4850000000000003,"15min":6.8810000000000002},"max":{"1min":29.228999999999999,"5min":258.31700000000001,"15min":258.31700000000001},"last":7.0940000000000003}]},{"osd":6,"last update":"Mon Jun 3 09:04:28 2024","interfaces":[{"interface":"back","average":{"1min":14.063000000000001,"5min":24.850000000000001,"15min":106.755},"min":{"1min":9.0820000000000007,"5min":8.9649999999999999,"15min":8.9649999999999999},"max":{"1min":24.666,"5min":188.06700000000001,"15min":9521.0010000000002},"last":183.60900000000001},{"interface":"front","average":{"1min":13.933999999999999,"5min":21.081,"15min":106.142},"min":{"1min":5.8719999999999999,"5min":5.8719999999999999,"15min":5.8719999999999999},"max":{"1min":117.551,"5min":193.887,"15min":9585.6620000000003},"last":184.20599999999999}]},{"osd":7,"last update":"Mon Jun 3 09:06:23 2024","interfaces":[{"interface":"back","average":{"1min":17.056000000000001,"5min":20.055,"15min":23.268999999999998},"min":{"1min":10.353999999999999,"5min":6.6130000000000004,"15min":6.6130000000000004},"max":{"1min":88.974999999999994,"5min":242.56800000000001,"15min":242.56800000000001},"last":12.818},{"interface":"front","average":{"1min":18.885000000000002,"5min":21.460000000000001,"15min":25.489999999999998},"min":{"1min":12.497,"5min":9.0660000000000007,"15min":9.0660000000000007},"max":{"1min":102.229,"5min":181.56399999999999,"15min":214.33799999999999},"last":12.301}]}]},{"osd":3,"up_from":14,"seq":60129542506,"num_pgs":11,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":58992,"kb_used_data":5728,"kb_used_omap":1,"kb_used_meta":53246,"kb_avail":94312848,"statfs":{"total":96636764160,"available":96576356352,"internally_reserved":0,"allocated":5865472,"data_stored":2751635,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1592,"internal_metadata":54524360},"hb_peers":[0,1,2,4,5,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":5,"apply_latency_ms":5,"commit_latency_ns":5000000,"apply_latency_ns":5000000},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Mon Jun 3 09:06:25 2024","interfaces":[{"interface":"back","average":{"1min":7.79,"5min":1060.1959999999999,"15min":367.01799999999997},"min":{"1min":5.6909999999999998,"5min":5.173,"15min":4.6929999999999996},"max":{"1min":10.59,"5min":23275.167000000001,"15min":23275.167000000001},"last":8.5389999999999997},{"interface":"front","average":{"1min":11.215999999999999,"5min":1066.5799999999999,"15min":372.476},"min":{"1min":8.7080000000000002,"5min":7.2510000000000003,"15min":5.9880000000000004},"max":{"1min":13.499000000000001,"5min":23278.536,"15min":23278.536},"last":13.473000000000001}]},{"osd":1,"last update":"Mon Jun 3 09:06:25 2024","interfaces":[{"interface":"back","average":{"1min":7.3760000000000003,"5min":15.614000000000001,"15min":389.83100000000002},"min":{"1min":4.335,"5min":4.335,"15min":4.335},"max":{"1min":11.289,"5min":250.72300000000001,"15min":21658.205999999998},"last":5.8440000000000003},{"interface":"front","average":{"1min":8.9499999999999993,"5min":14.135,"15min":381.10300000000001},"min":{"1min":6.5609999999999999,"5min":5.4100000000000001,"15min":5.2779999999999996},"max":{"1min":14.51,"5min":180.44499999999999,"15min":21307.069},"last":7.8959999999999999}]},{"osd":2,"last update":"Mon Jun 3 09:06:25 2024","interfaces":[{"interface":"back","average":{"1min":12.343999999999999,"5min":22.059999999999999,"15min":23.715},"min":{"1min":7.9199999999999999,"5min":6.4900000000000002,"15min":6.056},"max":{"1min":15.601000000000001,"5min":377.803,"15min":377.803},"last":10.523},{"interface":"front","average":{"1min":10.319000000000001,"5min":19.364999999999998,"15min":20.242999999999999},"min":{"1min":6.891,"5min":6.1749999999999998,"15min":5.9020000000000001},"max":{"1min":15.432,"5min":371.601,"15min":371.601},"last":11.694000000000001}]},{"osd":4,"last update":"Mon Jun 3 09:06:31 2024","interfaces":[{"interface":"back","average":{"1min":9.7460000000000004,"5min":891.23000000000002,"15min":416.48099999999999},"min":{"1min":5.6539999999999999,"5min":5.6539999999999999,"15min":5.6539999999999999},"max":{"1min":15.445,"5min":29040.635999999999,"15min":30144.939999999999},"last":6.827},{"interface":"front","average":{"1min":12.398999999999999,"5min":897.58500000000004,"15min":421.27499999999998},"min":{"1min":8.9469999999999992,"5min":7.8109999999999999,"15min":6.2439999999999998},"max":{"1min":16.518000000000001,"5min":29076.955999999998,"15min":30180.689999999999},"last":9.0559999999999992}]},{"osd":5,"last update":"Mon Jun 3 09:06:07 2024","interfaces":[{"interface":"back","average":{"1min":12.802,"5min":19.640999999999998,"15min":22.318000000000001},"min":{"1min":9.0890000000000004,"5min":6.8810000000000002,"15min":6.7690000000000001},"max":{"1min":16.745999999999999,"5min":249.88,"15min":327.34899999999999},"last":11.055},{"interface":"front","average":{"1min":15.002000000000001,"5min":24.003,"15min":26.649999999999999},"min":{"1min":11.984,"5min":9.9149999999999991,"15min":8.9879999999999995},"max":{"1min":18.407,"5min":251.47800000000001,"15min":331.86799999999999},"last":15.125}]},{"osd":6,"last update":"Mon Jun 3 09:04:04 2024","interfaces":[{"interface":"back","average":{"1min":22.815999999999999,"5min":31.513999999999999,"15min":181.74299999999999},"min":{"1min":9.4809999999999999,"5min":7.2240000000000002,"15min":7.2240000000000002},"max":{"1min":160.56999999999999,"5min":380.613,"15min":11893.74},"last":19.359999999999999},{"interface":"front","average":{"1min":25.359999999999999,"5min":30.981000000000002,"15min":179.81399999999999},"min":{"1min":9.8870000000000005,"5min":7.3360000000000003,"15min":6.0899999999999999},"max":{"1min":193.11099999999999,"5min":379.80700000000002,"15min":11893.069},"last":8.202}]},{"osd":7,"last update":"Mon Jun 3 09:06:25 2024","interfaces":[{"interface":"back","average":{"1min":11.321999999999999,"5min":18.443999999999999,"15min":21.568999999999999},"min":{"1min":8.109,"5min":8.109,"15min":7.4279999999999999},"max":{"1min":17.113,"5min":146.267,"15min":178.76300000000001},"last":12.93},{"interface":"front","average":{"1min":12.512,"5min":20.065999999999999,"15min":24.190000000000001},"min":{"1min":7.4989999999999997,"5min":7.4989999999999997,"15min":7.4989999999999997},"max":{"1min":18.047999999999998,"5min":148.13800000000001,"15min":417.22800000000001},"last":14.553000000000001}]}]},{"osd":1,"up_from":14,"seq":60129542504,"num_pgs":6,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":59384,"kb_used_data":6120,"kb_used_omap":1,"kb_used_meta":53246,"kb_avail":94312456,"statfs":{"total":96636764160,"available":96575954944,"internally_reserved":0,"allocated":6266880,"data_stored":3149475,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":54524357},"hb_peers":[0,2,3,4,5,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":5,"apply_latency_ms":5,"commit_latency_ns":5000000,"apply_latency_ns":5000000},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Mon Jun 3 09:06:07 2024","interfaces":[{"interface":"back","average":{"1min":7.7530000000000001,"5min":185.66999999999999,"15min":77.347999999999999},"min":{"1min":5.0229999999999997,"5min":4.9279999999999999,"15min":4.5449999999999999},"max":{"1min":11.896000000000001,"5min":7470.616,"15min":7470.616},"last":7.4450000000000003},{"interface":"front","average":{"1min":9.9220000000000006,"5min":187.14599999999999,"15min":76.325999999999993},"min":{"1min":5.3159999999999998,"5min":5.3159999999999998,"15min":5.3159999999999998},"max":{"1min":18.338000000000001,"5min":7477.0829999999996,"15min":7477.0829999999996},"last":9.8000000000000007}]},{"osd":2,"last update":"Mon Jun 3 09:06:20 2024","interfaces":[{"interface":"back","average":{"1min":10.340999999999999,"5min":17.369,"15min":23.670999999999999},"min":{"1min":6.9740000000000002,"5min":5.8650000000000002,"15min":5.8650000000000002},"max":{"1min":15.452,"5min":228,"15min":657.38},"last":10.438000000000001},{"interface":"front","average":{"1min":13.387,"5min":23.355,"15min":31.524000000000001},"min":{"1min":9.077,"5min":9.077,"15min":7.2160000000000002},"max":{"1min":19.701000000000001,"5min":229.465,"15min":700.06500000000005},"last":10.981999999999999}]},{"osd":3,"last update":"Mon Jun 3 09:06:04 2024","interfaces":[{"interface":"back","average":{"1min":8.6110000000000007,"5min":15.733000000000001,"15min":26.646999999999998},"min":{"1min":5.827,"5min":5.4640000000000004,"15min":5.4640000000000004},"max":{"1min":13.105,"5min":121.619,"15min":1676.009},"last":6.7359999999999998},{"interface":"front","average":{"1min":9.5850000000000009,"5min":16.161999999999999,"15min":29.065999999999999},"min":{"1min":6.3630000000000004,"5min":5.7699999999999996,"15min":5.7699999999999996},"max":{"1min":15.166,"5min":132.036,"15min":1766.5840000000001},"last":9.4640000000000004}]},{"osd":4,"last update":"Mon Jun 3 09:06:35 2024","interfaces":[{"interface":"back","average":{"1min":8.6400000000000006,"5min":641.85500000000002,"15min":334.71499999999997},"min":{"1min":6.7640000000000002,"5min":6.1760000000000002,"15min":6.0970000000000004},"max":{"1min":11.811,"5min":26340.968000000001,"15min":28044.862000000001},"last":9.2050000000000001},{"interface":"front","average":{"1min":12.361000000000001,"5min":648.59900000000005,"15min":338.48500000000001},"min":{"1min":9.8949999999999996,"5min":9.3940000000000001,"15min":6.4640000000000004},"max":{"1min":15.507,"5min":26342.311000000002,"15min":28045.624},"last":14.834}]},{"osd":5,"last update":"Mon Jun 3 09:06:04 2024","interfaces":[{"interface":"back","average":{"1min":12.478999999999999,"5min":20.228000000000002,"15min":25.997},"min":{"1min":8.6430000000000007,"5min":7.9720000000000004,"15min":7.234},"max":{"1min":17.442,"5min":139.43199999999999,"15min":701.22000000000003},"last":15.553000000000001},{"interface":"front","average":{"1min":13.507999999999999,"5min":25.760999999999999,"15min":27.553999999999998},"min":{"1min":10.042999999999999,"5min":7.258,"15min":6.242},"max":{"1min":20.611000000000001,"5min":278.63,"15min":598.35000000000002},"last":12.192}]},{"osd":6,"last update":"Mon Jun 3 09:03:54 2024","interfaces":[{"interface":"back","average":{"1min":28.524000000000001,"5min":20.654,"15min":23.878},"min":{"1min":6.0309999999999997,"5min":6.0309999999999997,"15min":6.0309999999999997},"max":{"1min":167.221,"5min":167.221,"15min":704.245},"last":18.721},{"interface":"front","average":{"1min":27.606000000000002,"5min":30.131,"15min":28.628},"min":{"1min":9.8650000000000002,"5min":9.7200000000000006,"15min":8.3580000000000005},"max":{"1min":123.938,"5min":358.23000000000002,"15min":634.05499999999995},"last":173.113}]},{"osd":7,"last update":"Mon Jun 3 09:06:20 2024","interfaces":[{"interface":"back","average":{"1min":14.66,"5min":23.234999999999999,"15min":31.097000000000001},"min":{"1min":11.250999999999999,"5min":8.0559999999999992,"15min":8.0559999999999992},"max":{"1min":21.507999999999999,"5min":270.95100000000002,"15min":703.07799999999997},"last":11.664999999999999},{"interface":"front","average":{"1min":15.548,"5min":28.760000000000002,"15min":28.925000000000001},"min":{"1min":12.759,"5min":8.2089999999999996,"15min":7.7599999999999998},"max":{"1min":22.408000000000001,"5min":360.99900000000002,"15min":608.44100000000003},"last":13.053000000000001}]}]},{"osd":0,"up_from":14,"seq":60129542506,"num_pgs":10,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":81108,"kb_used_data":5704,"kb_used_omap":1,"kb_used_meta":75390,"kb_avail":94290732,"statfs":{"total":96636764160,"available":96553709568,"internally_reserved":0,"allocated":5840896,"data_stored":2740727,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1593,"internal_metadata":77199815},"hb_peers":[1,2,3,4,5,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[{"osd":1,"last update":"Mon Jun 3 09:06:04 2024","interfaces":[{"interface":"back","average":{"1min":6.9960000000000004,"5min":56.997,"15min":339.29700000000003},"min":{"1min":4.4530000000000003,"5min":4.4530000000000003,"15min":4.4100000000000001},"max":{"1min":11.720000000000001,"5min":2414.4749999999999,"15min":20691.453000000001},"last":5.6639999999999997},{"interface":"front","average":{"1min":9.548,"5min":57.258000000000003,"15min":337.791},"min":{"1min":5.2300000000000004,"5min":5.2300000000000004,"15min":4.6050000000000004},"max":{"1min":15.337999999999999,"5min":2412.9549999999999,"15min":20640.792000000001},"last":8.5690000000000008}]},{"osd":2,"last update":"Mon Jun 3 09:06:22 2024","interfaces":[{"interface":"back","average":{"1min":11.154,"5min":26.515999999999998,"15min":22.882000000000001},"min":{"1min":5.7869999999999999,"5min":5.766,"15min":5.766},"max":{"1min":72.269999999999996,"5min":627.40700000000004,"15min":627.40700000000004},"last":6.6139999999999999},{"interface":"front","average":{"1min":14.353999999999999,"5min":33.304000000000002,"15min":29.885000000000002},"min":{"1min":8.2149999999999999,"5min":7.8630000000000004,"15min":7.4089999999999998},"max":{"1min":73.418000000000006,"5min":635.62599999999998,"15min":635.62599999999998},"last":9.8049999999999997}]},{"osd":3,"last update":"Mon Jun 3 09:06:04 2024","interfaces":[{"interface":"back","average":{"1min":12.105,"5min":26.925000000000001,"15min":26.584},"min":{"1min":7.8410000000000002,"5min":7.1280000000000001,"15min":6.1909999999999998},"max":{"1min":18.425999999999998,"5min":389.72699999999998,"15min":389.72699999999998},"last":9.1590000000000007},{"interface":"front","average":{"1min":13.794,"5min":29.771999999999998,"15min":28.815000000000001},"min":{"1min":10.084,"5min":9.4920000000000009,"15min":6.1849999999999996},"max":{"1min":17.373999999999999,"5min":390.23200000000003,"15min":494.92899999999997},"last":11.637}]},{"osd":4,"last update":"Mon Jun 3 09:06:27 2024","interfaces":[{"interface":"back","average":{"1min":11.352,"5min":745.30600000000004,"15min":363.48099999999999},"min":{"1min":6.415,"5min":6.415,"15min":6.415},"max":{"1min":16.724,"5min":20248.616000000002,"15min":25555.212},"last":7.2590000000000003},{"interface":"front","average":{"1min":13.545,"5min":744.76999999999998,"15min":361.03100000000001},"min":{"1min":9.3889999999999993,"5min":8.8940000000000001,"15min":7.1429999999999998},"max":{"1min":18.411000000000001,"5min":20250.195,"15min":25556.159},"last":10.904}]},{"osd":5,"last update":"Mon Jun 3 09:06:17 2024","interfaces":[{"interface":"back","average":{"1min":9.7040000000000006,"5min":26.533999999999999,"15min":28.683},"min":{"1min":6.8129999999999997,"5min":6.8129999999999997,"15min":6.8129999999999997},"max":{"1min":14.942,"5min":455.78699999999998,"15min":605.05899999999997},"last":14.183999999999999},{"interface":"front","average":{"1min":14.279,"5min":28.398,"15min":26.192},"min":{"1min":11.696,"5min":6.4420000000000002,"15min":6.2030000000000003},"max":{"1min":17.879000000000001,"5min":391.31200000000001,"15min":541.11000000000001},"last":12.648999999999999}]},{"osd":6,"last update":"Mon Jun 3 09:04:30 2024","interfaces":[{"interface":"back","average":{"1min":52.496000000000002,"5min":33.139000000000003,"15min":160.83199999999999},"min":{"1min":7.5289999999999999,"5min":7.1040000000000001,"15min":6.3739999999999997},"max":{"1min":457.91699999999997,"5min":457.91699999999997,"15min":11115.258},"last":18.959},{"interface":"front","average":{"1min":49.670000000000002,"5min":33.704999999999998,"15min":162.791},"min":{"1min":12.247999999999999,"5min":11.019,"15min":9.9760000000000009},"max":{"1min":407.05099999999999,"5min":407.05099999999999,"15min":11092.194},"last":17.707000000000001}]},{"osd":7,"last update":"Mon Jun 3 09:06:22 2024","interfaces":[{"interface":"back","average":{"1min":11.625999999999999,"5min":26.375,"15min":28.972999999999999},"min":{"1min":6.4480000000000004,"5min":6.4480000000000004,"15min":6.4480000000000004},"max":{"1min":15.352,"5min":461.41699999999997,"15min":598.33699999999999},"last":13.51},{"interface":"front","average":{"1min":14.013,"5min":29.696000000000002,"15min":33.630000000000003},"min":{"1min":10.938000000000001,"5min":10.58,"15min":10.263999999999999},"max":{"1min":16.603000000000002,"5min":463.10599999999999,"15min":592.80499999999995},"last":14.896000000000001}]}]}],"pool_statfs":[{"poolid":1,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":401408,"data_stored":397840,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":5,"total":0,"available":0,"internally_reserved":0,"allocated":401408,"data_stored":397840,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":4,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":5,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":6,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":17,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":17,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":17,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":17,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":17,"osd":4,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":17,"osd":5,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":5,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":17,"osd":6,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":5,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":17,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0}]}} 2024-06-03T09:06:53.118 DEBUG:teuthology.orchestra.run.smithi088:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph pg dump --format=json 2024-06-03T09:06:53.235 INFO:teuthology.orchestra.run.smithi088.stderr:2024-06-03T09:06:53.235+0000 7fa2c591b640 10 monclient: get_monmap_and_config 2024-06-03T09:06:53.236 INFO:teuthology.orchestra.run.smithi088.stderr:2024-06-03T09:06:53.235+0000 7fa2c591b640 10 monclient: build_initial_monmap 2024-06-03T09:06:53.236 INFO:teuthology.orchestra.run.smithi088.stderr:2024-06-03T09:06:53.235+0000 7fa2c591b640 10 monclient: monmap: 2024-06-03T09:06:53.236 INFO:teuthology.orchestra.run.smithi088.stderr:epoch 0 2024-06-03T09:06:53.236 INFO:teuthology.orchestra.run.smithi088.stderr:fsid ec3c3fdd-3ac6-4d7d-9412-683130ed8b04 2024-06-03T09:06:53.236 INFO:teuthology.orchestra.run.smithi088.stderr:last_changed 2024-06-03T09:06:53.236802+0000 2024-06-03T09:06:53.236 INFO:teuthology.orchestra.run.smithi088.stderr:created 2024-06-03T09:06:53.236802+0000 2024-06-03T09:06:53.236 INFO:teuthology.orchestra.run.smithi088.stderr:min_mon_release 0 (unknown) 2024-06-03T09:06:53.236 INFO:teuthology.orchestra.run.smithi088.stderr:election_strategy: 3 2024-06-03T09:06:53.236 INFO:teuthology.orchestra.run.smithi088.stderr:0: v1:172.21.15.88:6789/0 mon.noname-a 2024-06-03T09:06:53.236 INFO:teuthology.orchestra.run.smithi088.stderr:1: v1:172.21.15.192:6789/0 mon.noname-c 2024-06-03T09:06:53.237 INFO:teuthology.orchestra.run.smithi088.stderr:2: v1:172.21.15.88:6790/0 mon.noname-b 2024-06-03T09:06:53.237 INFO:teuthology.orchestra.run.smithi088.stderr: 2024-06-03T09:06:53.238 INFO:teuthology.orchestra.run.smithi088.stderr:2024-06-03T09:06:53.236+0000 7fa2c591b640 1 Processor -- start 2024-06-03T09:06:53.238 INFO:teuthology.orchestra.run.smithi088.stderr:2024-06-03T09:06:53.236+0000 7fa2c591b640 1 -- start start 2024-06-03T09:06:53.238 INFO:teuthology.orchestra.run.smithi088.stderr:2024-06-03T09:06:53.236+0000 7fa2c591b640 10 monclient: init 2024-06-03T09:06:53.238 INFO:teuthology.orchestra.run.smithi088.stderr:2024-06-03T09:06:53.237+0000 7fa2c591b640 10 monclient: _reopen_session rank -1 2024-06-03T09:06:53.238 INFO:teuthology.orchestra.run.smithi088.stderr:2024-06-03T09:06:53.237+0000 7fa2c591b640 10 monclient: _add_conns ranks=[1,0,2] 2024-06-03T09:06:53.238 INFO:teuthology.orchestra.run.smithi088.stderr:2024-06-03T09:06:53.237+0000 7fa2c591b640 10 monclient(hunting): picked mon.noname-c con 0x7fa2c00aef20 addr v1:172.21.15.192:6789/0 2024-06-03T09:06:53.238 INFO:teuthology.orchestra.run.smithi088.stderr:2024-06-03T09:06:53.237+0000 7fa2c591b640 10 monclient(hunting): picked mon.noname-a con 0x7fa2c006dc50 addr v1:172.21.15.88:6789/0 2024-06-03T09:06:53.239 INFO:teuthology.orchestra.run.smithi088.stderr:2024-06-03T09:06:53.237+0000 7fa2c591b640 10 monclient(hunting): picked mon.noname-b con 0x7fa2c0072250 addr v1:172.21.15.88:6790/0 2024-06-03T09:06:53.239 INFO:teuthology.orchestra.run.smithi088.stderr:2024-06-03T09:06:53.237+0000 7fa2c591b640 1 -- --> v1:172.21.15.88:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fa2c0144b00 con 0x7fa2c006dc50 2024-06-03T09:06:53.239 INFO:teuthology.orchestra.run.smithi088.stderr:2024-06-03T09:06:53.237+0000 7fa2c591b640 1 -- --> v1:172.21.15.192:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fa2c006f1a0 con 0x7fa2c00aef20 2024-06-03T09:06:53.239 INFO:teuthology.orchestra.run.smithi088.stderr:2024-06-03T09:06:53.237+0000 7fa2c591b640 1 -- --> v1:172.21.15.88:6790/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fa2c00b0c30 con 0x7fa2c0072250 2024-06-03T09:06:53.239 INFO:teuthology.orchestra.run.smithi088.stderr:2024-06-03T09:06:53.237+0000 7fa2c591b640 10 monclient(hunting): _renew_subs 2024-06-03T09:06:53.239 INFO:teuthology.orchestra.run.smithi088.stderr:2024-06-03T09:06:53.237+0000 7fa2c591b640 10 monclient(hunting): authenticate will time out at 3069.047107s 2024-06-03T09:06:53.239 INFO:teuthology.orchestra.run.smithi088.stderr:2024-06-03T09:06:53.237+0000 7fa2bf7fe640 1 --1- >> v1:172.21.15.88:6790/0 conn(0x7fa2c0072250 0x7fa2c014cea0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.88:6790/0 says I am v1:172.21.15.88:52368/0 (socket says 172.21.15.88:52368) 2024-06-03T09:06:53.239 INFO:teuthology.orchestra.run.smithi088.stderr:2024-06-03T09:06:53.237+0000 7fa2bf7fe640 1 -- 172.21.15.88:0/664464655 learned_addr learned my addr 172.21.15.88:0/664464655 (peer_addr_for_me v1:172.21.15.88:0/0) 2024-06-03T09:06:53.240 INFO:teuthology.orchestra.run.smithi088.stderr:2024-06-03T09:06:53.240+0000 7fa2bdffb640 1 -- 172.21.15.88:0/664464655 <== mon.1 v1:172.21.15.192:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 272816066 0 0) 0x7fa2c006f1a0 con 0x7fa2c00aef20 2024-06-03T09:06:53.241 INFO:teuthology.orchestra.run.smithi088.stderr:2024-06-03T09:06:53.240+0000 7fa2bdffb640 10 monclient(hunting): _negotiate 2024-06-03T09:06:53.241 INFO:teuthology.orchestra.run.smithi088.stderr:2024-06-03T09:06:53.240+0000 7fa2bdffb640 10 monclient(hunting): _init_auth method 2 2024-06-03T09:06:53.241 INFO:teuthology.orchestra.run.smithi088.stderr:2024-06-03T09:06:53.240+0000 7fa2bdffb640 10 monclient(hunting): _init_auth creating new auth 2024-06-03T09:06:53.241 INFO:teuthology.orchestra.run.smithi088.stderr:2024-06-03T09:06:53.240+0000 7fa2bdffb640 10 monclient(hunting): my global_id is 30295 2024-06-03T09:06:53.241 INFO:teuthology.orchestra.run.smithi088.stderr:2024-06-03T09:06:53.240+0000 7fa2bdffb640 1 -- 172.21.15.88:0/664464655 --> v1:172.21.15.192:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7fa2a8003520 con 0x7fa2c00aef20 2024-06-03T09:06:53.241 INFO:teuthology.orchestra.run.smithi088.stderr:2024-06-03T09:06:53.240+0000 7fa2bdffb640 1 -- 172.21.15.88:0/664464655 <== mon.0 v1:172.21.15.88:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 715310826 0 0) 0x7fa2c0144b00 con 0x7fa2c006dc50 2024-06-03T09:06:53.241 INFO:teuthology.orchestra.run.smithi088.stderr:2024-06-03T09:06:53.240+0000 7fa2bdffb640 10 monclient(hunting): _negotiate 2024-06-03T09:06:53.241 INFO:teuthology.orchestra.run.smithi088.stderr:2024-06-03T09:06:53.240+0000 7fa2bdffb640 10 monclient(hunting): _init_auth method 2 2024-06-03T09:06:53.241 INFO:teuthology.orchestra.run.smithi088.stderr:2024-06-03T09:06:53.240+0000 7fa2bdffb640 10 monclient(hunting): _init_auth creating new auth 2024-06-03T09:06:53.241 INFO:teuthology.orchestra.run.smithi088.stderr:2024-06-03T09:06:53.240+0000 7fa2bdffb640 10 monclient(hunting): my global_id is 30720 2024-06-03T09:06:53.241 INFO:teuthology.orchestra.run.smithi088.stderr:2024-06-03T09:06:53.240+0000 7fa2bdffb640 1 -- 172.21.15.88:0/664464655 --> v1:172.21.15.88:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7fa2c006f1a0 con 0x7fa2c006dc50 2024-06-03T09:06:53.241 INFO:teuthology.orchestra.run.smithi088.stderr:2024-06-03T09:06:53.240+0000 7fa2bdffb640 1 -- 172.21.15.88:0/664464655 <== mon.2 v1:172.21.15.88:6790/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 281284844 0 0) 0x7fa2c00b0c30 con 0x7fa2c0072250 2024-06-03T09:06:53.241 INFO:teuthology.orchestra.run.smithi088.stderr:2024-06-03T09:06:53.240+0000 7fa2bdffb640 10 monclient(hunting): _negotiate 2024-06-03T09:06:53.241 INFO:teuthology.orchestra.run.smithi088.stderr:2024-06-03T09:06:53.240+0000 7fa2bdffb640 10 monclient(hunting): _init_auth method 2 2024-06-03T09:06:53.242 INFO:teuthology.orchestra.run.smithi088.stderr:2024-06-03T09:06:53.240+0000 7fa2bdffb640 10 monclient(hunting): _init_auth creating new auth 2024-06-03T09:06:53.242 INFO:teuthology.orchestra.run.smithi088.stderr:2024-06-03T09:06:53.240+0000 7fa2bdffb640 10 monclient(hunting): my global_id is 30224 2024-06-03T09:06:53.242 INFO:teuthology.orchestra.run.smithi088.stderr:2024-06-03T09:06:53.240+0000 7fa2bdffb640 1 -- 172.21.15.88:0/664464655 --> v1:172.21.15.88:6790/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7fa2c0144b00 con 0x7fa2c0072250 2024-06-03T09:06:53.244 INFO:teuthology.orchestra.run.smithi088.stderr:2024-06-03T09:06:53.244+0000 7fa2bdffb640 1 -- 172.21.15.88:0/664464655 <== mon.2 v1:172.21.15.88:6790/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 214+0+0 (unknown 325568614 0 0) 0x7fa2c0144b00 con 0x7fa2c0072250 2024-06-03T09:06:53.244 INFO:teuthology.orchestra.run.smithi088.stderr:2024-06-03T09:06:53.244+0000 7fa2bdffb640 1 -- 172.21.15.88:0/664464655 >> v1:172.21.15.192:6789/0 conn(0x7fa2c00aef20 legacy=0x7fa2c0071b40 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-06-03T09:06:53.244 INFO:teuthology.orchestra.run.smithi088.stderr:2024-06-03T09:06:53.244+0000 7fa2bdffb640 1 -- 172.21.15.88:0/664464655 >> v1:172.21.15.88:6789/0 conn(0x7fa2c006dc50 legacy=0x7fa2c0145000 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-06-03T09:06:53.244 INFO:teuthology.orchestra.run.smithi088.stderr:2024-06-03T09:06:53.244+0000 7fa2bdffb640 10 monclient: _finish_hunting 0 2024-06-03T09:06:53.245 INFO:teuthology.orchestra.run.smithi088.stderr:2024-06-03T09:06:53.244+0000 7fa2bdffb640 1 monclient: found mon.noname-b 2024-06-03T09:06:53.245 INFO:teuthology.orchestra.run.smithi088.stderr:2024-06-03T09:06:53.244+0000 7fa2bdffb640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-06-03T09:06:53.245 INFO:teuthology.orchestra.run.smithi088.stderr:2024-06-03T09:06:53.244+0000 7fa2bdffb640 10 monclient: _send_mon_message to mon.noname-b at v1:172.21.15.88:6790/0 2024-06-03T09:06:53.245 INFO:teuthology.orchestra.run.smithi088.stderr:2024-06-03T09:06:53.244+0000 7fa2bdffb640 1 -- 172.21.15.88:0/664464655 --> v1:172.21.15.88:6790/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fa2c0141580 con 0x7fa2c0072250 2024-06-03T09:06:53.245 INFO:teuthology.orchestra.run.smithi088.stderr:2024-06-03T09:06:53.244+0000 7fa2bdffb640 10 monclient: _finish_auth 0 2024-06-03T09:06:53.245 INFO:teuthology.orchestra.run.smithi088.stderr:2024-06-03T09:06:53.244+0000 7fa2bdffb640 10 monclient: _check_auth_tickets 2024-06-03T09:06:53.245 INFO:teuthology.orchestra.run.smithi088.stderr:2024-06-03T09:06:53.244+0000 7fa2bdffb640 20 monclient: _check_auth_rotating not needed by client.admin 2024-06-03T09:06:53.245 INFO:teuthology.orchestra.run.smithi088.stderr:2024-06-03T09:06:53.244+0000 7fa2bdffb640 1 -- 172.21.15.88:0/664464655 <== mon.2 v1:172.21.15.88:6790/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 2234374065 0 0) 0x7fa2b80015b0 con 0x7fa2c0072250 2024-06-03T09:06:53.245 INFO:teuthology.orchestra.run.smithi088.stderr:2024-06-03T09:06:53.244+0000 7fa2c591b640 5 monclient: authenticate success, global_id 30224 2024-06-03T09:06:53.245 INFO:teuthology.orchestra.run.smithi088.stderr:2024-06-03T09:06:53.244+0000 7fa2c591b640 20 monclient: get_monmap_and_config waiting for monmap|config 2024-06-03T09:06:53.245 INFO:teuthology.orchestra.run.smithi088.stderr:2024-06-03T09:06:53.244+0000 7fa2bdffb640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-06-03T09:06:53.245 INFO:teuthology.orchestra.run.smithi088.stderr:2024-06-03T09:06:53.244+0000 7fa2bdffb640 10 monclient: got monmap 1 from mon.noname-b (according to old e1) 2024-06-03T09:06:53.245 INFO:teuthology.orchestra.run.smithi088.stderr:2024-06-03T09:06:53.244+0000 7fa2bdffb640 10 monclient: dump: 2024-06-03T09:06:53.245 INFO:teuthology.orchestra.run.smithi088.stderr:epoch 1 2024-06-03T09:06:53.246 INFO:teuthology.orchestra.run.smithi088.stderr:fsid ec3c3fdd-3ac6-4d7d-9412-683130ed8b04 2024-06-03T09:06:53.246 INFO:teuthology.orchestra.run.smithi088.stderr:last_changed 2024-06-03T08:34:31.885125+0000 2024-06-03T09:06:53.246 INFO:teuthology.orchestra.run.smithi088.stderr:created 2024-06-03T08:34:31.885125+0000 2024-06-03T09:06:53.246 INFO:teuthology.orchestra.run.smithi088.stderr:min_mon_release 18 (reef) 2024-06-03T09:06:53.246 INFO:teuthology.orchestra.run.smithi088.stderr:election_strategy: 3 2024-06-03T09:06:53.246 INFO:teuthology.orchestra.run.smithi088.stderr:0: v1:172.21.15.88:6789/0 mon.a 2024-06-03T09:06:53.246 INFO:teuthology.orchestra.run.smithi088.stderr:1: v1:172.21.15.192:6789/0 mon.b 2024-06-03T09:06:53.246 INFO:teuthology.orchestra.run.smithi088.stderr:2: v1:172.21.15.88:6790/0 mon.c 2024-06-03T09:06:53.246 INFO:teuthology.orchestra.run.smithi088.stderr: 2024-06-03T09:06:53.246 INFO:teuthology.orchestra.run.smithi088.stderr:2024-06-03T09:06:53.244+0000 7fa2c591b640 20 monclient: get_monmap_and_config waiting for monmap|config 2024-06-03T09:06:53.248 INFO:teuthology.orchestra.run.smithi088.stderr:2024-06-03T09:06:53.246+0000 7fa2bdffb640 1 -- 172.21.15.88:0/664464655 <== mon.2 v1:172.21.15.88:6790/0 4 ==== config(0 keys) v1 ==== 4+0+0 (unknown 0 0 0) 0x7fa2b80017d0 con 0x7fa2c0072250 2024-06-03T09:06:53.248 INFO:teuthology.orchestra.run.smithi088.stderr:2024-06-03T09:06:53.246+0000 7fa2bdffb640 10 monclient: handle_config config(0 keys) v1 2024-06-03T09:06:53.248 INFO:teuthology.orchestra.run.smithi088.stderr:2024-06-03T09:06:53.246+0000 7fa2bdffb640 1 -- 172.21.15.88:0/664464655 <== mon.2 v1:172.21.15.88:6790/0 5 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 2234374065 0 0) 0x7fa2b8001b60 con 0x7fa2c0072250 2024-06-03T09:06:53.248 INFO:teuthology.orchestra.run.smithi088.stderr:2024-06-03T09:06:53.246+0000 7fa2bdffb640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-06-03T09:06:53.248 INFO:teuthology.orchestra.run.smithi088.stderr:2024-06-03T09:06:53.246+0000 7fa2bdffb640 10 monclient: got monmap 1 from mon.c (according to old e1) 2024-06-03T09:06:53.248 INFO:teuthology.orchestra.run.smithi088.stderr:2024-06-03T09:06:53.246+0000 7fa2bdffb640 10 monclient: dump: 2024-06-03T09:06:53.248 INFO:teuthology.orchestra.run.smithi088.stderr:epoch 1 2024-06-03T09:06:53.249 INFO:teuthology.orchestra.run.smithi088.stderr:fsid ec3c3fdd-3ac6-4d7d-9412-683130ed8b04 2024-06-03T09:06:53.249 INFO:teuthology.orchestra.run.smithi088.stderr:last_changed 2024-06-03T08:34:31.885125+0000 2024-06-03T09:06:53.249 INFO:teuthology.orchestra.run.smithi088.stderr:created 2024-06-03T08:34:31.885125+0000 2024-06-03T09:06:53.249 INFO:teuthology.orchestra.run.smithi088.stderr:min_mon_release 18 (reef) 2024-06-03T09:06:53.249 INFO:teuthology.orchestra.run.smithi088.stderr:election_strategy: 3 2024-06-03T09:06:53.249 INFO:teuthology.orchestra.run.smithi088.stderr:0: v1:172.21.15.88:6789/0 mon.a 2024-06-03T09:06:53.249 INFO:teuthology.orchestra.run.smithi088.stderr:1: v1:172.21.15.192:6789/0 mon.b 2024-06-03T09:06:53.249 INFO:teuthology.orchestra.run.smithi088.stderr:2: v1:172.21.15.88:6790/0 mon.c 2024-06-03T09:06:53.249 INFO:teuthology.orchestra.run.smithi088.stderr: 2024-06-03T09:06:53.249 INFO:teuthology.orchestra.run.smithi088.stderr:2024-06-03T09:06:53.246+0000 7fa2c591b640 10 monclient: get_monmap_and_config success 2024-06-03T09:06:53.249 INFO:teuthology.orchestra.run.smithi088.stderr:2024-06-03T09:06:53.246+0000 7fa2c591b640 4 set_mon_vals no callback set 2024-06-03T09:06:53.249 INFO:teuthology.orchestra.run.smithi088.stderr:2024-06-03T09:06:53.247+0000 7fa2c591b640 10 monclient: shutdown 2024-06-03T09:06:53.250 INFO:teuthology.orchestra.run.smithi088.stderr:2024-06-03T09:06:53.247+0000 7fa2c591b640 20 monclient: shutdown discarding 0 pending message(s) 2024-06-03T09:06:53.250 INFO:teuthology.orchestra.run.smithi088.stderr:2024-06-03T09:06:53.247+0000 7fa2c591b640 1 -- 172.21.15.88:0/664464655 >> v1:172.21.15.88:6790/0 conn(0x7fa2c0072250 legacy=0x7fa2c014cea0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-06-03T09:06:53.250 INFO:teuthology.orchestra.run.smithi088.stderr:2024-06-03T09:06:53.247+0000 7fa2c591b640 1 -- 172.21.15.88:0/664464655 shutdown_connections 2024-06-03T09:06:53.250 INFO:teuthology.orchestra.run.smithi088.stderr:2024-06-03T09:06:53.247+0000 7fa2c591b640 1 -- 172.21.15.88:0/664464655 >> 172.21.15.88:0/664464655 conn(0x7fa2c00afda0 msgr2=0x7fa2c00b01b0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-06-03T09:06:53.250 INFO:teuthology.orchestra.run.smithi088.stderr:2024-06-03T09:06:53.247+0000 7fa2c591b640 1 -- 172.21.15.88:0/664464655 shutdown_connections 2024-06-03T09:06:53.250 INFO:teuthology.orchestra.run.smithi088.stderr:2024-06-03T09:06:53.247+0000 7fa2c591b640 1 -- 172.21.15.88:0/664464655 wait complete. 2024-06-03T09:06:53.250 INFO:teuthology.orchestra.run.smithi088.stderr:2024-06-03T09:06:53.247+0000 7fa2c591b640 10 monclient: build_initial_monmap 2024-06-03T09:06:53.250 INFO:teuthology.orchestra.run.smithi088.stderr:2024-06-03T09:06:53.247+0000 7fa2c591b640 10 monclient: monmap: 2024-06-03T09:06:53.250 INFO:teuthology.orchestra.run.smithi088.stderr:epoch 0 2024-06-03T09:06:53.250 INFO:teuthology.orchestra.run.smithi088.stderr:fsid 00000000-0000-0000-0000-000000000000 2024-06-03T09:06:53.250 INFO:teuthology.orchestra.run.smithi088.stderr:last_changed 0.000000 2024-06-03T09:06:53.250 INFO:teuthology.orchestra.run.smithi088.stderr:created 0.000000 2024-06-03T09:06:53.250 INFO:teuthology.orchestra.run.smithi088.stderr:min_mon_release 0 (unknown) 2024-06-03T09:06:53.250 INFO:teuthology.orchestra.run.smithi088.stderr:election_strategy: 1 2024-06-03T09:06:53.251 INFO:teuthology.orchestra.run.smithi088.stderr:0: v1:172.21.15.88:6789/0 mon.noname-a 2024-06-03T09:06:53.251 INFO:teuthology.orchestra.run.smithi088.stderr:1: v1:172.21.15.192:6789/0 mon.noname-b 2024-06-03T09:06:53.251 INFO:teuthology.orchestra.run.smithi088.stderr:2: v1:172.21.15.88:6790/0 mon.noname-c 2024-06-03T09:06:53.251 INFO:teuthology.orchestra.run.smithi088.stderr: 2024-06-03T09:06:53.251 INFO:teuthology.orchestra.run.smithi088.stderr:2024-06-03T09:06:53.248+0000 7fa2c591b640 1 librados: starting msgr at 2024-06-03T09:06:53.251 INFO:teuthology.orchestra.run.smithi088.stderr:2024-06-03T09:06:53.248+0000 7fa2c591b640 1 librados: starting objecter 2024-06-03T09:06:53.251 INFO:teuthology.orchestra.run.smithi088.stderr:2024-06-03T09:06:53.248+0000 7fa2c591b640 1 Processor -- start 2024-06-03T09:06:53.251 INFO:teuthology.orchestra.run.smithi088.stderr:2024-06-03T09:06:53.248+0000 7fa2c591b640 1 -- start start 2024-06-03T09:06:53.251 INFO:teuthology.orchestra.run.smithi088.stderr:2024-06-03T09:06:53.248+0000 7fa2c591b640 1 librados: setting wanted keys 2024-06-03T09:06:53.251 INFO:teuthology.orchestra.run.smithi088.stderr:2024-06-03T09:06:53.248+0000 7fa2c591b640 1 librados: calling monclient init 2024-06-03T09:06:53.251 INFO:teuthology.orchestra.run.smithi088.stderr:2024-06-03T09:06:53.248+0000 7fa2c591b640 10 monclient: init 2024-06-03T09:06:53.255 INFO:teuthology.orchestra.run.smithi088.stderr:2024-06-03T09:06:53.249+0000 7fa2c591b640 10 monclient: _reopen_session rank -1 2024-06-03T09:06:53.256 INFO:teuthology.orchestra.run.smithi088.stderr:2024-06-03T09:06:53.249+0000 7fa2c591b640 10 monclient: _add_conns ranks=[0,2,1] 2024-06-03T09:06:53.256 INFO:teuthology.orchestra.run.smithi088.stderr:2024-06-03T09:06:53.249+0000 7fa2c591b640 10 monclient(hunting): picked mon.noname-a con 0x7fa2c006dc50 addr v1:172.21.15.88:6789/0 2024-06-03T09:06:53.256 INFO:teuthology.orchestra.run.smithi088.stderr:2024-06-03T09:06:53.249+0000 7fa2c591b640 10 monclient(hunting): picked mon.noname-c con 0x7fa2c0072250 addr v1:172.21.15.88:6790/0 2024-06-03T09:06:53.256 INFO:teuthology.orchestra.run.smithi088.stderr:2024-06-03T09:06:53.249+0000 7fa2c591b640 10 monclient(hunting): picked mon.noname-b con 0x7fa2c00aef20 addr v1:172.21.15.192:6789/0 2024-06-03T09:06:53.256 INFO:teuthology.orchestra.run.smithi088.stderr:2024-06-03T09:06:53.249+0000 7fa2c591b640 1 -- --> v1:172.21.15.88:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fa2c011dbe0 con 0x7fa2c006dc50 2024-06-03T09:06:53.256 INFO:teuthology.orchestra.run.smithi088.stderr:2024-06-03T09:06:53.249+0000 7fa2c591b640 1 -- --> v1:172.21.15.192:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fa2c011ae80 con 0x7fa2c00aef20 2024-06-03T09:06:53.257 INFO:teuthology.orchestra.run.smithi088.stderr:2024-06-03T09:06:53.249+0000 7fa2c591b640 1 -- --> v1:172.21.15.88:6790/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fa2c011d530 con 0x7fa2c0072250 2024-06-03T09:06:53.257 INFO:teuthology.orchestra.run.smithi088.stderr:2024-06-03T09:06:53.249+0000 7fa2c591b640 10 monclient(hunting): _renew_subs 2024-06-03T09:06:53.257 INFO:teuthology.orchestra.run.smithi088.stderr:2024-06-03T09:06:53.249+0000 7fa2c591b640 10 monclient(hunting): authenticate will time out at 3069.059118s 2024-06-03T09:06:53.257 INFO:teuthology.orchestra.run.smithi088.stderr:2024-06-03T09:06:53.249+0000 7fa2be7fc640 1 --1- >> v1:172.21.15.88:6790/0 conn(0x7fa2c0072250 0x7fa2c011e070 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=1).handle_server_banner_and_identify peer v1:172.21.15.88:6790/0 says I am v1:172.21.15.88:52372/0 (socket says 172.21.15.88:52372) 2024-06-03T09:06:53.257 INFO:teuthology.orchestra.run.smithi088.stderr:2024-06-03T09:06:53.249+0000 7fa2be7fc640 1 -- 172.21.15.88:0/4188580933 learned_addr learned my addr 172.21.15.88:0/4188580933 (peer_addr_for_me v1:172.21.15.88:0/0) 2024-06-03T09:06:53.257 INFO:teuthology.orchestra.run.smithi088.stderr:2024-06-03T09:06:53.249+0000 7fa2a7fff640 10 client.?.objecter ms_handle_connect 0x7fa2c0072250 2024-06-03T09:06:53.258 INFO:teuthology.orchestra.run.smithi088.stderr:2024-06-03T09:06:53.249+0000 7fa2a7fff640 10 client.?.objecter resend_mon_ops 2024-06-03T09:06:53.258 INFO:teuthology.orchestra.run.smithi088.stderr:2024-06-03T09:06:53.250+0000 7fa2a7fff640 10 client.?.objecter ms_handle_connect 0x7fa2c006dc50 2024-06-03T09:06:53.258 INFO:teuthology.orchestra.run.smithi088.stderr:2024-06-03T09:06:53.250+0000 7fa2a7fff640 10 client.?.objecter resend_mon_ops 2024-06-03T09:06:53.258 INFO:teuthology.orchestra.run.smithi088.stderr:2024-06-03T09:06:53.250+0000 7fa2a7fff640 10 client.?.objecter ms_handle_connect 0x7fa2c00aef20 2024-06-03T09:06:53.258 INFO:teuthology.orchestra.run.smithi088.stderr:2024-06-03T09:06:53.250+0000 7fa2a7fff640 10 client.?.objecter resend_mon_ops 2024-06-03T09:06:53.258 INFO:teuthology.orchestra.run.smithi088.stderr:2024-06-03T09:06:53.252+0000 7fa2a7fff640 1 -- 172.21.15.88:0/4188580933 <== mon.2 v1:172.21.15.88:6790/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 1719853319 0 0) 0x7fa2c011d530 con 0x7fa2c0072250 2024-06-03T09:06:53.259 INFO:teuthology.orchestra.run.smithi088.stderr:2024-06-03T09:06:53.252+0000 7fa2a7fff640 10 monclient(hunting): _negotiate 2024-06-03T09:06:53.259 INFO:teuthology.orchestra.run.smithi088.stderr:2024-06-03T09:06:53.252+0000 7fa2a7fff640 10 monclient(hunting): _init_auth method 2 2024-06-03T09:06:53.259 INFO:teuthology.orchestra.run.smithi088.stderr:2024-06-03T09:06:53.252+0000 7fa2a7fff640 10 monclient(hunting): _init_auth creating new auth 2024-06-03T09:06:53.259 INFO:teuthology.orchestra.run.smithi088.stderr:2024-06-03T09:06:53.252+0000 7fa2a7fff640 10 monclient(hunting): my global_id is 30227 2024-06-03T09:06:53.259 INFO:teuthology.orchestra.run.smithi088.stderr:2024-06-03T09:06:53.252+0000 7fa2a7fff640 1 -- 172.21.15.88:0/4188580933 --> v1:172.21.15.88:6790/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7fa2ac004eb0 con 0x7fa2c0072250 2024-06-03T09:06:53.259 INFO:teuthology.orchestra.run.smithi088.stderr:2024-06-03T09:06:53.252+0000 7fa2a7fff640 1 -- 172.21.15.88:0/4188580933 <== mon.0 v1:172.21.15.88:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 1484703905 0 0) 0x7fa2c011dbe0 con 0x7fa2c006dc50 2024-06-03T09:06:53.260 INFO:teuthology.orchestra.run.smithi088.stderr:2024-06-03T09:06:53.252+0000 7fa2a7fff640 10 monclient(hunting): _negotiate 2024-06-03T09:06:53.260 INFO:teuthology.orchestra.run.smithi088.stderr:2024-06-03T09:06:53.252+0000 7fa2a7fff640 10 monclient(hunting): _init_auth method 2 2024-06-03T09:06:53.260 INFO:teuthology.orchestra.run.smithi088.stderr:2024-06-03T09:06:53.252+0000 7fa2a7fff640 10 monclient(hunting): _init_auth creating new auth 2024-06-03T09:06:53.260 INFO:teuthology.orchestra.run.smithi088.stderr:2024-06-03T09:06:53.252+0000 7fa2a7fff640 10 monclient(hunting): my global_id is 30723 2024-06-03T09:06:53.260 INFO:teuthology.orchestra.run.smithi088.stderr:2024-06-03T09:06:53.252+0000 7fa2a7fff640 1 -- 172.21.15.88:0/4188580933 --> v1:172.21.15.88:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7fa2c011d530 con 0x7fa2c006dc50 2024-06-03T09:06:53.260 INFO:teuthology.orchestra.run.smithi088.stderr:2024-06-03T09:06:53.252+0000 7fa2a7fff640 1 -- 172.21.15.88:0/4188580933 <== mon.1 v1:172.21.15.192:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 730627342 0 0) 0x7fa2c011ae80 con 0x7fa2c00aef20 2024-06-03T09:06:53.261 INFO:teuthology.orchestra.run.smithi088.stderr:2024-06-03T09:06:53.252+0000 7fa2a7fff640 10 monclient(hunting): _negotiate 2024-06-03T09:06:53.261 INFO:teuthology.orchestra.run.smithi088.stderr:2024-06-03T09:06:53.252+0000 7fa2a7fff640 10 monclient(hunting): _init_auth method 2 2024-06-03T09:06:53.261 INFO:teuthology.orchestra.run.smithi088.stderr:2024-06-03T09:06:53.252+0000 7fa2a7fff640 10 monclient(hunting): _init_auth creating new auth 2024-06-03T09:06:53.261 INFO:teuthology.orchestra.run.smithi088.stderr:2024-06-03T09:06:53.252+0000 7fa2a7fff640 10 monclient(hunting): my global_id is 30298 2024-06-03T09:06:53.261 INFO:teuthology.orchestra.run.smithi088.stderr:2024-06-03T09:06:53.252+0000 7fa2a7fff640 1 -- 172.21.15.88:0/4188580933 --> v1:172.21.15.192:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7fa2c011dbe0 con 0x7fa2c00aef20 2024-06-03T09:06:53.265 INFO:teuthology.orchestra.run.smithi088.stderr:2024-06-03T09:06:53.257+0000 7fa2a7fff640 1 -- 172.21.15.88:0/4188580933 <== mon.0 v1:172.21.15.88:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 764+0+0 (unknown 864024291 0 0) 0x7fa2c011d530 con 0x7fa2c006dc50 2024-06-03T09:06:53.265 INFO:teuthology.orchestra.run.smithi088.stderr:2024-06-03T09:06:53.257+0000 7fa2a7fff640 1 -- 172.21.15.88:0/4188580933 --> v1:172.21.15.88:6789/0 -- auth(proto 2 165 bytes epoch 0) v1 -- 0x7fa2c011ae80 con 0x7fa2c006dc50 2024-06-03T09:06:53.265 INFO:teuthology.orchestra.run.smithi088.stderr:2024-06-03T09:06:53.257+0000 7fa2a7fff640 1 -- 172.21.15.88:0/4188580933 <== mon.2 v1:172.21.15.88:6790/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 764+0+0 (unknown 4096688033 0 0) 0x7fa2ac004eb0 con 0x7fa2c0072250 2024-06-03T09:06:53.265 INFO:teuthology.orchestra.run.smithi088.stderr:2024-06-03T09:06:53.257+0000 7fa2a7fff640 1 -- 172.21.15.88:0/4188580933 --> v1:172.21.15.88:6790/0 -- auth(proto 2 165 bytes epoch 0) v1 -- 0x7fa2c011d530 con 0x7fa2c0072250 2024-06-03T09:06:53.266 INFO:teuthology.orchestra.run.smithi088.stderr:2024-06-03T09:06:53.257+0000 7fa2a7fff640 1 -- 172.21.15.88:0/4188580933 <== mon.2 v1:172.21.15.88:6790/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 2234374065 0 0) 0x7fa2b40018b0 con 0x7fa2c0072250 2024-06-03T09:06:53.266 INFO:teuthology.orchestra.run.smithi088.stderr:2024-06-03T09:06:53.257+0000 7fa2a7fff640 10 monclient(hunting): handle_monmap mon_map magic: 0 v1 2024-06-03T09:06:53.266 INFO:teuthology.orchestra.run.smithi088.stderr:2024-06-03T09:06:53.257+0000 7fa2a7fff640 10 monclient(hunting): got monmap 1 from mon.noname-c (according to old e1) 2024-06-03T09:06:53.266 INFO:teuthology.orchestra.run.smithi088.stderr:2024-06-03T09:06:53.257+0000 7fa2a7fff640 10 monclient(hunting): dump: 2024-06-03T09:06:53.266 INFO:teuthology.orchestra.run.smithi088.stderr:epoch 1 2024-06-03T09:06:53.266 INFO:teuthology.orchestra.run.smithi088.stderr:fsid ec3c3fdd-3ac6-4d7d-9412-683130ed8b04 2024-06-03T09:06:53.267 INFO:teuthology.orchestra.run.smithi088.stderr:last_changed 2024-06-03T08:34:31.885125+0000 2024-06-03T09:06:53.267 INFO:teuthology.orchestra.run.smithi088.stderr:created 2024-06-03T08:34:31.885125+0000 2024-06-03T09:06:53.267 INFO:teuthology.orchestra.run.smithi088.stderr:min_mon_release 18 (reef) 2024-06-03T09:06:53.267 INFO:teuthology.orchestra.run.smithi088.stderr:election_strategy: 3 2024-06-03T09:06:53.267 INFO:teuthology.orchestra.run.smithi088.stderr:0: v1:172.21.15.88:6789/0 mon.a 2024-06-03T09:06:53.267 INFO:teuthology.orchestra.run.smithi088.stderr:1: v1:172.21.15.192:6789/0 mon.b 2024-06-03T09:06:53.268 INFO:teuthology.orchestra.run.smithi088.stderr:2: v1:172.21.15.88:6790/0 mon.c 2024-06-03T09:06:53.268 INFO:teuthology.orchestra.run.smithi088.stderr: 2024-06-03T09:06:53.268 INFO:teuthology.orchestra.run.smithi088.stderr:2024-06-03T09:06:53.257+0000 7fa2a7fff640 10 monclient(hunting): _finish_auth 0 2024-06-03T09:06:53.268 INFO:teuthology.orchestra.run.smithi088.stderr:2024-06-03T09:06:53.257+0000 7fa2a7fff640 1 -- 172.21.15.88:0/4188580933 <== mon.0 v1:172.21.15.88:6789/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 2234374065 0 0) 0x7fa2b0001cf0 con 0x7fa2c006dc50 2024-06-03T09:06:53.268 INFO:teuthology.orchestra.run.smithi088.stderr:2024-06-03T09:06:53.257+0000 7fa2a7fff640 10 monclient(hunting): handle_monmap mon_map magic: 0 v1 2024-06-03T09:06:53.268 INFO:teuthology.orchestra.run.smithi088.stderr:2024-06-03T09:06:53.257+0000 7fa2a7fff640 10 monclient(hunting): got monmap 1 from mon.a (according to old e1) 2024-06-03T09:06:53.269 INFO:teuthology.orchestra.run.smithi088.stderr:2024-06-03T09:06:53.257+0000 7fa2a7fff640 10 monclient(hunting): dump: 2024-06-03T09:06:53.269 INFO:teuthology.orchestra.run.smithi088.stderr:epoch 1 2024-06-03T09:06:53.269 INFO:teuthology.orchestra.run.smithi088.stderr:fsid ec3c3fdd-3ac6-4d7d-9412-683130ed8b04 2024-06-03T09:06:53.269 INFO:teuthology.orchestra.run.smithi088.stderr:last_changed 2024-06-03T08:34:31.885125+0000 2024-06-03T09:06:53.269 INFO:teuthology.orchestra.run.smithi088.stderr:created 2024-06-03T08:34:31.885125+0000 2024-06-03T09:06:53.269 INFO:teuthology.orchestra.run.smithi088.stderr:min_mon_release 18 (reef) 2024-06-03T09:06:53.270 INFO:teuthology.orchestra.run.smithi088.stderr:election_strategy: 3 2024-06-03T09:06:53.270 INFO:teuthology.orchestra.run.smithi088.stderr:0: v1:172.21.15.88:6789/0 mon.a 2024-06-03T09:06:53.270 INFO:teuthology.orchestra.run.smithi088.stderr:1: v1:172.21.15.192:6789/0 mon.b 2024-06-03T09:06:53.270 INFO:teuthology.orchestra.run.smithi088.stderr:2: v1:172.21.15.88:6790/0 mon.c 2024-06-03T09:06:53.270 INFO:teuthology.orchestra.run.smithi088.stderr: 2024-06-03T09:06:53.271 INFO:teuthology.orchestra.run.smithi088.stderr:2024-06-03T09:06:53.257+0000 7fa2a7fff640 1 -- 172.21.15.88:0/4188580933 <== mon.1 v1:172.21.15.192:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 764+0+0 (unknown 33829048 0 0) 0x7fa2c011dbe0 con 0x7fa2c00aef20 2024-06-03T09:06:53.271 INFO:teuthology.orchestra.run.smithi088.stderr:2024-06-03T09:06:53.257+0000 7fa2a7fff640 1 -- 172.21.15.88:0/4188580933 --> v1:172.21.15.192:6789/0 -- auth(proto 2 165 bytes epoch 0) v1 -- 0x7fa2ac004eb0 con 0x7fa2c00aef20 2024-06-03T09:06:53.271 INFO:teuthology.orchestra.run.smithi088.stderr:2024-06-03T09:06:53.257+0000 7fa2a7fff640 1 -- 172.21.15.88:0/4188580933 <== mon.1 v1:172.21.15.192:6789/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 2234374065 0 0) 0x7fa2b8001ae0 con 0x7fa2c00aef20 2024-06-03T09:06:53.271 INFO:teuthology.orchestra.run.smithi088.stderr:2024-06-03T09:06:53.257+0000 7fa2a7fff640 10 monclient(hunting): handle_monmap mon_map magic: 0 v1 2024-06-03T09:06:53.271 INFO:teuthology.orchestra.run.smithi088.stderr:2024-06-03T09:06:53.257+0000 7fa2a7fff640 10 monclient(hunting): got monmap 1 from mon.b (according to old e1) 2024-06-03T09:06:53.271 INFO:teuthology.orchestra.run.smithi088.stderr:2024-06-03T09:06:53.257+0000 7fa2a7fff640 10 monclient(hunting): dump: 2024-06-03T09:06:53.272 INFO:teuthology.orchestra.run.smithi088.stderr:epoch 1 2024-06-03T09:06:53.272 INFO:teuthology.orchestra.run.smithi088.stderr:fsid ec3c3fdd-3ac6-4d7d-9412-683130ed8b04 2024-06-03T09:06:53.272 INFO:teuthology.orchestra.run.smithi088.stderr:last_changed 2024-06-03T08:34:31.885125+0000 2024-06-03T09:06:53.272 INFO:teuthology.orchestra.run.smithi088.stderr:created 2024-06-03T08:34:31.885125+0000 2024-06-03T09:06:53.272 INFO:teuthology.orchestra.run.smithi088.stderr:min_mon_release 18 (reef) 2024-06-03T09:06:53.272 INFO:teuthology.orchestra.run.smithi088.stderr:election_strategy: 3 2024-06-03T09:06:53.273 INFO:teuthology.orchestra.run.smithi088.stderr:0: v1:172.21.15.88:6789/0 mon.a 2024-06-03T09:06:53.273 INFO:teuthology.orchestra.run.smithi088.stderr:1: v1:172.21.15.192:6789/0 mon.b 2024-06-03T09:06:53.273 INFO:teuthology.orchestra.run.smithi088.stderr:2: v1:172.21.15.88:6790/0 mon.c 2024-06-03T09:06:53.273 INFO:teuthology.orchestra.run.smithi088.stderr: 2024-06-03T09:06:53.273 INFO:teuthology.orchestra.run.smithi088.stderr:2024-06-03T09:06:53.261+0000 7fa2a7fff640 1 -- 172.21.15.88:0/4188580933 <== mon.2 v1:172.21.15.88:6790/0 4 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 580+0+0 (unknown 1977872962 0 0) 0x7fa2c011d530 con 0x7fa2c0072250 2024-06-03T09:06:53.273 INFO:teuthology.orchestra.run.smithi088.stderr:2024-06-03T09:06:53.261+0000 7fa2a7fff640 1 -- 172.21.15.88:0/4188580933 >> v1:172.21.15.192:6789/0 conn(0x7fa2c00aef20 legacy=0x7fa2c01e2050 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-06-03T09:06:53.274 INFO:teuthology.orchestra.run.smithi088.stderr:2024-06-03T09:06:53.261+0000 7fa2a7fff640 1 -- 172.21.15.88:0/4188580933 >> v1:172.21.15.88:6789/0 conn(0x7fa2c006dc50 legacy=0x7fa2c013c510 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-06-03T09:06:53.274 INFO:teuthology.orchestra.run.smithi088.stderr:2024-06-03T09:06:53.261+0000 7fa2a7fff640 10 monclient: _finish_hunting 0 2024-06-03T09:06:53.274 INFO:teuthology.orchestra.run.smithi088.stderr:2024-06-03T09:06:53.261+0000 7fa2a7fff640 1 monclient: found mon.c 2024-06-03T09:06:53.274 INFO:teuthology.orchestra.run.smithi088.stderr:2024-06-03T09:06:53.261+0000 7fa2a7fff640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-06-03T09:06:53.274 INFO:teuthology.orchestra.run.smithi088.stderr:2024-06-03T09:06:53.261+0000 7fa2a7fff640 10 monclient: _send_mon_message to mon.c at v1:172.21.15.88:6790/0 2024-06-03T09:06:53.275 INFO:teuthology.orchestra.run.smithi088.stderr:2024-06-03T09:06:53.261+0000 7fa2a7fff640 1 -- 172.21.15.88:0/4188580933 --> v1:172.21.15.88:6790/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fa2c011c960 con 0x7fa2c0072250 2024-06-03T09:06:53.275 INFO:teuthology.orchestra.run.smithi088.stderr:2024-06-03T09:06:53.261+0000 7fa2a7fff640 10 monclient: _finish_auth 0 2024-06-03T09:06:53.275 INFO:teuthology.orchestra.run.smithi088.stderr:2024-06-03T09:06:53.261+0000 7fa2a7fff640 10 monclient: _check_auth_tickets 2024-06-03T09:06:53.275 INFO:teuthology.orchestra.run.smithi088.stderr:2024-06-03T09:06:53.261+0000 7fa2a7fff640 20 monclient: _check_auth_rotating not needed by client.admin 2024-06-03T09:06:53.275 INFO:teuthology.orchestra.run.smithi088.stderr:2024-06-03T09:06:53.261+0000 7fa2c591b640 5 monclient: authenticate success, global_id 30227 2024-06-03T09:06:53.275 INFO:teuthology.orchestra.run.smithi088.stderr:2024-06-03T09:06:53.261+0000 7fa2c591b640 10 monclient: _renew_subs 2024-06-03T09:06:53.276 INFO:teuthology.orchestra.run.smithi088.stderr:2024-06-03T09:06:53.261+0000 7fa2c591b640 10 monclient: _send_mon_message to mon.c at v1:172.21.15.88:6790/0 2024-06-03T09:06:53.276 INFO:teuthology.orchestra.run.smithi088.stderr:2024-06-03T09:06:53.261+0000 7fa2c591b640 1 -- 172.21.15.88:0/4188580933 --> v1:172.21.15.88:6790/0 -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fa2c011d7e0 con 0x7fa2c0072250 2024-06-03T09:06:53.276 INFO:teuthology.orchestra.run.smithi088.stderr:2024-06-03T09:06:53.261+0000 7fa2c591b640 10 client.30227.objecter _maybe_request_map subscribing (onetime) to next osd map 2024-06-03T09:06:53.276 INFO:teuthology.orchestra.run.smithi088.stderr:2024-06-03T09:06:53.261+0000 7fa2c591b640 10 monclient: _renew_subs 2024-06-03T09:06:53.276 INFO:teuthology.orchestra.run.smithi088.stderr:2024-06-03T09:06:53.261+0000 7fa2c591b640 10 monclient: _send_mon_message to mon.c at v1:172.21.15.88:6790/0 2024-06-03T09:06:53.276 INFO:teuthology.orchestra.run.smithi088.stderr:2024-06-03T09:06:53.261+0000 7fa2c591b640 1 -- 172.21.15.88:0/4188580933 --> v1:172.21.15.88:6790/0 -- mon_subscribe({osdmap=0}) v3 -- 0x7fa2c01dab20 con 0x7fa2c0072250 2024-06-03T09:06:53.277 INFO:teuthology.orchestra.run.smithi088.stderr:2024-06-03T09:06:53.261+0000 7fa2c591b640 1 librados: init done 2024-06-03T09:06:53.277 INFO:teuthology.orchestra.run.smithi088.stderr:2024-06-03T09:06:53.262+0000 7fa2c591b640 10 start_mon_command cmd=[{"prefix": "get_command_descriptions"}] 2024-06-03T09:06:53.277 INFO:teuthology.orchestra.run.smithi088.stderr:2024-06-03T09:06:53.262+0000 7fa2c591b640 10 monclient: _send_command 1 [{"prefix": "get_command_descriptions"}] 2024-06-03T09:06:53.277 INFO:teuthology.orchestra.run.smithi088.stderr:2024-06-03T09:06:53.262+0000 7fa2c591b640 10 monclient: _send_mon_message to mon.c at v1:172.21.15.88:6790/0 2024-06-03T09:06:53.277 INFO:teuthology.orchestra.run.smithi088.stderr:2024-06-03T09:06:53.262+0000 7fa2c591b640 1 -- 172.21.15.88:0/4188580933 --> v1:172.21.15.88:6790/0 -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fa2c004ea60 con 0x7fa2c0072250 2024-06-03T09:06:53.513 INFO:teuthology.orchestra.run.smithi088.stderr:2024-06-03T09:06:53.512+0000 7fa2a7fff640 1 -- 172.21.15.88:0/4188580933 <== mon.2 v1:172.21.15.88:6790/0 5 ==== config(0 keys) v1 ==== 4+0+0 (unknown 0 0 0) 0x7fa2b40035f0 con 0x7fa2c0072250 2024-06-03T09:06:53.513 INFO:teuthology.orchestra.run.smithi088.stderr:2024-06-03T09:06:53.512+0000 7fa2a7fff640 10 monclient: handle_config config(0 keys) v1 2024-06-03T09:06:53.513 INFO:teuthology.orchestra.run.smithi088.stderr:2024-06-03T09:06:53.512+0000 7fa2a7fff640 1 -- 172.21.15.88:0/4188580933 <== mon.2 v1:172.21.15.88:6790/0 6 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 2234374065 0 0) 0x7fa2b4003980 con 0x7fa2c0072250 2024-06-03T09:06:53.513 INFO:teuthology.orchestra.run.smithi088.stderr:2024-06-03T09:06:53.512+0000 7fa2a7fff640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-06-03T09:06:53.513 INFO:teuthology.orchestra.run.smithi088.stderr:2024-06-03T09:06:53.512+0000 7fa2a7fff640 10 monclient: got monmap 1 from mon.c (according to old e1) 2024-06-03T09:06:53.513 INFO:teuthology.orchestra.run.smithi088.stderr:2024-06-03T09:06:53.512+0000 7fa2a7fff640 10 monclient: dump: 2024-06-03T09:06:53.513 INFO:teuthology.orchestra.run.smithi088.stderr:epoch 1 2024-06-03T09:06:53.513 INFO:teuthology.orchestra.run.smithi088.stderr:fsid ec3c3fdd-3ac6-4d7d-9412-683130ed8b04 2024-06-03T09:06:53.513 INFO:teuthology.orchestra.run.smithi088.stderr:last_changed 2024-06-03T08:34:31.885125+0000 2024-06-03T09:06:53.513 INFO:teuthology.orchestra.run.smithi088.stderr:created 2024-06-03T08:34:31.885125+0000 2024-06-03T09:06:53.514 INFO:teuthology.orchestra.run.smithi088.stderr:min_mon_release 18 (reef) 2024-06-03T09:06:53.514 INFO:teuthology.orchestra.run.smithi088.stderr:election_strategy: 3 2024-06-03T09:06:53.514 INFO:teuthology.orchestra.run.smithi088.stderr:0: v1:172.21.15.88:6789/0 mon.a 2024-06-03T09:06:53.514 INFO:teuthology.orchestra.run.smithi088.stderr:1: v1:172.21.15.192:6789/0 mon.b 2024-06-03T09:06:53.514 INFO:teuthology.orchestra.run.smithi088.stderr:2: v1:172.21.15.88:6790/0 mon.c 2024-06-03T09:06:53.514 INFO:teuthology.orchestra.run.smithi088.stderr: 2024-06-03T09:06:53.514 INFO:teuthology.orchestra.run.smithi088.stderr:2024-06-03T09:06:53.512+0000 7fa2bd7fa640 4 set_mon_vals no callback set 2024-06-03T09:06:53.515 INFO:teuthology.orchestra.run.smithi088.stderr:2024-06-03T09:06:53.513+0000 7fa2a7fff640 1 -- 172.21.15.88:0/4188580933 <== mon.2 v1:172.21.15.88:6790/0 7 ==== mgrmap(e 5) v1 ==== 89110+0+0 (unknown 873533973 0 0) 0x7fa2b4003ae0 con 0x7fa2c0072250 2024-06-03T09:06:53.515 INFO:teuthology.orchestra.run.smithi088.stderr:2024-06-03T09:06:53.514+0000 7fa2beffd640 10 monclient: get_auth_request con 0x7fa2ac06d3d0 auth_method 0 2024-06-03T09:06:53.515 INFO:teuthology.orchestra.run.smithi088.stderr:2024-06-03T09:06:53.514+0000 7fa2a7fff640 10 client.30227.objecter ms_handle_connect 0x7fa2ac06d3d0 2024-06-03T09:06:53.522 INFO:teuthology.orchestra.run.smithi088.stderr:2024-06-03T09:06:53.522+0000 7fa2a7fff640 1 -- 172.21.15.88:0/4188580933 <== mon.2 v1:172.21.15.88:6790/0 8 ==== osd_map(1014..1014 src has 622..1014) v4 ==== 5229+0+0 (unknown 3230192161 0 0) 0x7fa2b4054070 con 0x7fa2c0072250 2024-06-03T09:06:53.522 INFO:teuthology.orchestra.run.smithi088.stderr:2024-06-03T09:06:53.522+0000 7fa2a7fff640 10 client.30227.objecter ms_dispatch 0x7fa2c0000b90 osd_map(1014..1014 src has 622..1014) v4 2024-06-03T09:06:53.522 INFO:teuthology.orchestra.run.smithi088.stderr:2024-06-03T09:06:53.522+0000 7fa2a7fff640 3 client.30227.objecter handle_osd_map got epochs [1014,1014] > 0 2024-06-03T09:06:53.523 INFO:teuthology.orchestra.run.smithi088.stderr:2024-06-03T09:06:53.522+0000 7fa2a7fff640 3 client.30227.objecter handle_osd_map decoding full epoch 1014 2024-06-03T09:06:53.523 INFO:teuthology.orchestra.run.smithi088.stderr:2024-06-03T09:06:53.522+0000 7fa2a7fff640 20 client.30227.objecter dump_active .. 0 homeless 2024-06-03T09:06:53.523 INFO:teuthology.orchestra.run.smithi088.stderr:2024-06-03T09:06:53.522+0000 7fa2a7fff640 1 -- 172.21.15.88:0/4188580933 <== mon.2 v1:172.21.15.88:6790/0 9 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+182498 (unknown 1092875540 0 3016760700) 0x7fa2b404fc20 con 0x7fa2c0072250 2024-06-03T09:06:53.523 INFO:teuthology.orchestra.run.smithi088.stderr:2024-06-03T09:06:53.522+0000 7fa2a7fff640 10 monclient: handle_mon_command_ack 1 [{"prefix": "get_command_descriptions"}] 2024-06-03T09:06:53.523 INFO:teuthology.orchestra.run.smithi088.stderr:2024-06-03T09:06:53.522+0000 7fa2a7fff640 10 monclient: _finish_command 1 = system:0 2024-06-03T09:06:53.570 INFO:tasks.ceph.mon.a.smithi088.stderr:==00:00:31:48.896 34743== Warning: unimplemented fcntl command: 1036 2024-06-03T09:06:53.572 INFO:tasks.ceph.mon.a.smithi088.stderr:==00:00:31:48.898 34743== Warning: unimplemented fcntl command: 1036 2024-06-03T09:06:53.593 INFO:tasks.ceph.mon.a.smithi088.stderr:==00:00:31:48.919 34743== Warning: unimplemented fcntl command: 1036 2024-06-03T09:06:53.623 INFO:tasks.ceph.mon.c.smithi088.stderr:==00:00:31:48.949 34749== Warning: unimplemented fcntl command: 1036 2024-06-03T09:06:53.626 INFO:tasks.ceph.mon.c.smithi088.stderr:==00:00:31:48.952 34749== Warning: unimplemented fcntl command: 1036 2024-06-03T09:06:53.640 INFO:tasks.ceph.mon.b.smithi192.stderr:==00:00:31:48.943 34228== Warning: unimplemented fcntl command: 1036 2024-06-03T09:06:53.642 INFO:tasks.ceph.mon.b.smithi192.stderr:==00:00:31:48.945 34228== Warning: unimplemented fcntl command: 1036 2024-06-03T09:06:53.645 INFO:tasks.ceph.mon.c.smithi088.stderr:==00:00:31:48.971 34749== Warning: unimplemented fcntl command: 1036 2024-06-03T09:06:53.662 INFO:tasks.ceph.mon.b.smithi192.stderr:==00:00:31:48.964 34228== Warning: unimplemented fcntl command: 1036 2024-06-03T09:06:53.704 INFO:teuthology.orchestra.run.smithi088.stderr:2024-06-03T09:06:53.704+0000 7fa2c591b640 1 -- 172.21.15.88:0/4188580933 --> v1:172.21.15.192:6800/34229 -- mgr_command(tid 0: {"prefix": "pg dump", "target": ["mon-mgr", ""], "format": "json"}) v1 -- 0x7fa2c00af300 con 0x7fa2ac06d3d0 2024-06-03T09:06:53.713 INFO:teuthology.orchestra.run.smithi088.stderr:2024-06-03T09:06:53.713+0000 7fa2a7fff640 1 -- 172.21.15.88:0/4188580933 <== mgr.4110 v1:172.21.15.192:6800/34229 1 ==== mgr_command_reply(tid 0: 0 dumped all) v1 ==== 18+0+151688 (unknown 2965378022 0 548438894) 0x7fa2b0007b20 con 0x7fa2ac06d3d0 2024-06-03T09:06:53.715 INFO:teuthology.orchestra.run.smithi088.stdout: 2024-06-03T09:06:53.716 INFO:teuthology.orchestra.run.smithi088.stderr:dumped all 2024-06-03T09:06:53.717 INFO:teuthology.orchestra.run.smithi088.stderr:2024-06-03T09:06:53.716+0000 7fa2c591b640 10 librados: watch_flush enter 2024-06-03T09:06:53.717 INFO:teuthology.orchestra.run.smithi088.stderr:2024-06-03T09:06:53.716+0000 7fa2c591b640 10 librados: watch_flush exit 2024-06-03T09:06:53.717 INFO:teuthology.orchestra.run.smithi088.stderr:2024-06-03T09:06:53.716+0000 7fa2c591b640 20 client.30227.objecter shutdown clearing up homeless session... 2024-06-03T09:06:53.718 INFO:teuthology.orchestra.run.smithi088.stderr:2024-06-03T09:06:53.716+0000 7fa2c591b640 10 client.30227.objecter successfully canceled tick 2024-06-03T09:06:53.718 INFO:teuthology.orchestra.run.smithi088.stderr:2024-06-03T09:06:53.716+0000 7fa2c591b640 1 -- 172.21.15.88:0/4188580933 >> v1:172.21.15.192:6800/34229 conn(0x7fa2ac06d3d0 legacy=0x7fa2ac06f890 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-06-03T09:06:53.718 INFO:teuthology.orchestra.run.smithi088.stderr:2024-06-03T09:06:53.716+0000 7fa2c591b640 10 monclient: shutdown 2024-06-03T09:06:53.718 INFO:teuthology.orchestra.run.smithi088.stderr:2024-06-03T09:06:53.716+0000 7fa2c591b640 20 monclient: shutdown discarding 0 pending message(s) 2024-06-03T09:06:53.718 INFO:teuthology.orchestra.run.smithi088.stderr:2024-06-03T09:06:53.716+0000 7fa2c591b640 1 -- 172.21.15.88:0/4188580933 >> v1:172.21.15.88:6790/0 conn(0x7fa2c0072250 legacy=0x7fa2c011e070 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-06-03T09:06:53.718 INFO:teuthology.orchestra.run.smithi088.stderr:2024-06-03T09:06:53.716+0000 7fa2c591b640 1 -- 172.21.15.88:0/4188580933 shutdown_connections 2024-06-03T09:06:53.718 INFO:teuthology.orchestra.run.smithi088.stderr:2024-06-03T09:06:53.716+0000 7fa2c591b640 1 -- 172.21.15.88:0/4188580933 >> 172.21.15.88:0/4188580933 conn(0x7fa2c00afda0 msgr2=0x7fa2c00b2060 unknown :-1 s=STATE_NONE l=0).mark_down 2024-06-03T09:06:53.718 INFO:teuthology.orchestra.run.smithi088.stderr:2024-06-03T09:06:53.716+0000 7fa2c591b640 1 -- 172.21.15.88:0/4188580933 shutdown_connections 2024-06-03T09:06:53.718 INFO:teuthology.orchestra.run.smithi088.stderr:2024-06-03T09:06:53.716+0000 7fa2c591b640 1 -- 172.21.15.88:0/4188580933 wait complete. 2024-06-03T09:06:53.718 INFO:teuthology.orchestra.run.smithi088.stderr:2024-06-03T09:06:53.716+0000 7fa2c591b640 1 librados: shutdown 2024-06-03T09:06:53.731 INFO:teuthology.orchestra.run.smithi088.stdout:{"pg_ready":true,"pg_map":{"version":1936,"stamp":"2024-06-03T09:06:53.105541+0000","last_osdmap_epoch":0,"last_pg_scan":0,"pg_stats_sum":{"stat_sum":{"num_bytes":459285,"num_objects":3,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":3,"num_whiteouts":0,"num_read":102,"num_read_kb":88,"num_write":120,"num_write_kb":1379,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2,"num_bytes_recovered":397840,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":83,"ondisk_log_size":83,"up":82,"acting":82,"num_store_stats":0},"osd_stats_sum":{"up_from":0,"seq":0,"num_pgs":82,"num_osds":8,"num_per_pool_osds":8,"num_per_pool_omap_osds":8,"kb":754974720,"kb_used":495068,"kb_used_data":46808,"kb_used_omap":12,"kb_used_meta":448115,"kb_avail":754479652,"statfs":{"total":773094113280,"available":772587163648,"internally_reserved":0,"allocated":47931392,"data_stored":22900274,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":12744,"internal_metadata":458870328},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":77,"apply_latency_ms":77,"commit_latency_ns":77000000,"apply_latency_ns":77000000},"alerts":[],"network_ping_times":[]},"pg_stats_delta":{"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":-1,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":0,"acting":0,"num_store_stats":0,"stamp_delta":"8.148465"},"pg_stats":[{"pgid":"17.d","version":"0'0","reported_seq":945,"reported_epoch":1013,"state":"active+clean","last_fresh":"2024-06-03T09:06:45.930577+0000","last_change":"2024-06-03T08:38:19.260548+0000","last_active":"2024-06-03T09:06:45.930577+0000","last_peered":"2024-06-03T09:06:45.930577+0000","last_clean":"2024-06-03T09:06:45.930577+0000","last_became_active":"2024-06-03T08:38:19.253559+0000","last_became_peered":"2024-06-03T08:38:19.253559+0000","last_unstale":"2024-06-03T09:06:45.930577+0000","last_undegraded":"2024-06-03T09:06:45.930577+0000","last_fullsized":"2024-06-03T09:06:45.930577+0000","mapping_epoch":28,"log_start":"0'0","ondisk_log_start":"0'0","created":28,"last_epoch_clean":45,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-06-03T08:37:55.614497+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-06-03T08:37:55.614497+0000","last_clean_scrub_stamp":"2024-06-03T08:37:55.614497+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-06-04T19:00:12.470979+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1],"acting":[0,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"17.c","version":"0'0","reported_seq":965,"reported_epoch":1014,"state":"active+clean","last_fresh":"2024-06-03T09:06:46.317689+0000","last_change":"2024-06-03T09:06:46.317689+0000","last_active":"2024-06-03T09:06:46.317689+0000","last_peered":"2024-06-03T09:06:46.317689+0000","last_clean":"2024-06-03T09:06:46.317689+0000","last_became_active":"2024-06-03T09:06:46.312637+0000","last_became_peered":"2024-06-03T09:06:46.312637+0000","last_unstale":"2024-06-03T09:06:46.317689+0000","last_undegraded":"2024-06-03T09:06:46.317689+0000","last_fullsized":"2024-06-03T09:06:46.317689+0000","mapping_epoch":1013,"log_start":"0'0","ondisk_log_start":"0'0","created":28,"last_epoch_clean":1014,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-06-03T08:37:55.614497+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-06-03T08:37:55.614497+0000","last_clean_scrub_stamp":"2024-06-03T08:37:55.614497+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-06-04T17:49:40.110047+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,6],"acting":[3,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"17.f","version":"0'0","reported_seq":843,"reported_epoch":1014,"state":"active+clean","last_fresh":"2024-06-03T09:06:46.306998+0000","last_change":"2024-06-03T09:06:46.306998+0000","last_active":"2024-06-03T09:06:46.306998+0000","last_peered":"2024-06-03T09:06:46.306998+0000","last_clean":"2024-06-03T09:06:46.306998+0000","last_became_active":"2024-06-03T09:06:46.301958+0000","last_became_peered":"2024-06-03T09:06:46.301958+0000","last_unstale":"2024-06-03T09:06:46.306998+0000","last_undegraded":"2024-06-03T09:06:46.306998+0000","last_fullsized":"2024-06-03T09:06:46.306998+0000","mapping_epoch":1013,"log_start":"0'0","ondisk_log_start":"0'0","created":28,"last_epoch_clean":1014,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-06-03T08:37:55.614497+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-06-03T08:37:55.614497+0000","last_clean_scrub_stamp":"2024-06-03T08:37:55.614497+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-06-04T14:30:23.089522+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,4],"acting":[6,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"17.e","version":"0'0","reported_seq":843,"reported_epoch":1014,"state":"active+clean","last_fresh":"2024-06-03T09:06:46.278835+0000","last_change":"2024-06-03T09:06:46.278835+0000","last_active":"2024-06-03T09:06:46.278835+0000","last_peered":"2024-06-03T09:06:46.278835+0000","last_clean":"2024-06-03T09:06:46.278835+0000","last_became_active":"2024-06-03T09:06:46.273941+0000","last_became_peered":"2024-06-03T09:06:46.273941+0000","last_unstale":"2024-06-03T09:06:46.278835+0000","last_undegraded":"2024-06-03T09:06:46.278835+0000","last_fullsized":"2024-06-03T09:06:46.278835+0000","mapping_epoch":1013,"log_start":"0'0","ondisk_log_start":"0'0","created":28,"last_epoch_clean":1014,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-06-03T08:37:55.614497+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-06-03T08:37:55.614497+0000","last_clean_scrub_stamp":"2024-06-03T08:37:55.614497+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-06-04T09:31:07.311650+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,5],"acting":[6,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"17.9","version":"0'0","reported_seq":949,"reported_epoch":1013,"state":"active+clean","last_fresh":"2024-06-03T09:06:45.422478+0000","last_change":"2024-06-03T08:38:04.772260+0000","last_active":"2024-06-03T09:06:45.422478+0000","last_peered":"2024-06-03T09:06:45.422478+0000","last_clean":"2024-06-03T09:06:45.422478+0000","last_became_active":"2024-06-03T08:38:04.763428+0000","last_became_peered":"2024-06-03T08:38:04.763428+0000","last_unstale":"2024-06-03T09:06:45.422478+0000","last_undegraded":"2024-06-03T09:06:45.422478+0000","last_fullsized":"2024-06-03T09:06:45.422478+0000","mapping_epoch":28,"log_start":"0'0","ondisk_log_start":"0'0","created":28,"last_epoch_clean":33,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-06-03T08:37:55.614497+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-06-03T08:37:55.614497+0000","last_clean_scrub_stamp":"2024-06-03T08:37:55.614497+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-06-04T10:55:10.773936+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,4],"acting":[7,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"17.8","version":"0'0","reported_seq":972,"reported_epoch":1014,"state":"active+clean","last_fresh":"2024-06-03T09:06:46.349899+0000","last_change":"2024-06-03T09:06:46.349899+0000","last_active":"2024-06-03T09:06:46.349899+0000","last_peered":"2024-06-03T09:06:46.349899+0000","last_clean":"2024-06-03T09:06:46.349899+0000","last_became_active":"2024-06-03T09:06:46.344756+0000","last_became_peered":"2024-06-03T09:06:46.344756+0000","last_unstale":"2024-06-03T09:06:46.349899+0000","last_undegraded":"2024-06-03T09:06:46.349899+0000","last_fullsized":"2024-06-03T09:06:46.349899+0000","mapping_epoch":1013,"log_start":"0'0","ondisk_log_start":"0'0","created":28,"last_epoch_clean":1014,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-06-03T08:37:55.614497+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-06-03T08:37:55.614497+0000","last_clean_scrub_stamp":"2024-06-03T08:37:55.614497+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-06-04T09:33:29.773440+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,6],"acting":[0,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"17.b","version":"0'0","reported_seq":843,"reported_epoch":1014,"state":"active+clean","last_fresh":"2024-06-03T09:06:46.350450+0000","last_change":"2024-06-03T09:06:46.350450+0000","last_active":"2024-06-03T09:06:46.350450+0000","last_peered":"2024-06-03T09:06:46.350450+0000","last_clean":"2024-06-03T09:06:46.350450+0000","last_became_active":"2024-06-03T09:06:46.345458+0000","last_became_peered":"2024-06-03T09:06:46.345458+0000","last_unstale":"2024-06-03T09:06:46.350450+0000","last_undegraded":"2024-06-03T09:06:46.350450+0000","last_fullsized":"2024-06-03T09:06:46.350450+0000","mapping_epoch":1013,"log_start":"0'0","ondisk_log_start":"0'0","created":28,"last_epoch_clean":1014,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-06-03T08:37:55.614497+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-06-03T08:37:55.614497+0000","last_clean_scrub_stamp":"2024-06-03T08:37:55.614497+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-06-04T16:51:34.952224+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,2],"acting":[6,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"17.a","version":"0'0","reported_seq":929,"reported_epoch":1013,"state":"active+clean","last_fresh":"2024-06-03T09:06:45.882773+0000","last_change":"2024-06-03T08:38:05.843142+0000","last_active":"2024-06-03T09:06:45.882773+0000","last_peered":"2024-06-03T09:06:45.882773+0000","last_clean":"2024-06-03T09:06:45.882773+0000","last_became_active":"2024-06-03T08:38:05.809257+0000","last_became_peered":"2024-06-03T08:38:05.809257+0000","last_unstale":"2024-06-03T09:06:45.882773+0000","last_undegraded":"2024-06-03T09:06:45.882773+0000","last_fullsized":"2024-06-03T09:06:45.882773+0000","mapping_epoch":28,"log_start":"0'0","ondisk_log_start":"0'0","created":28,"last_epoch_clean":33,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-06-03T08:37:55.614497+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-06-03T08:37:55.614497+0000","last_clean_scrub_stamp":"2024-06-03T08:37:55.614497+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-06-04T12:12:03.686838+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2],"acting":[1,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"17.5","version":"0'0","reported_seq":946,"reported_epoch":1013,"state":"active+clean","last_fresh":"2024-06-03T09:06:45.929134+0000","last_change":"2024-06-03T08:38:23.466051+0000","last_active":"2024-06-03T09:06:45.929134+0000","last_peered":"2024-06-03T09:06:45.929134+0000","last_clean":"2024-06-03T09:06:45.929134+0000","last_became_active":"2024-06-03T08:38:23.458039+0000","last_became_peered":"2024-06-03T08:38:23.458039+0000","last_unstale":"2024-06-03T09:06:45.929134+0000","last_undegraded":"2024-06-03T09:06:45.929134+0000","last_fullsized":"2024-06-03T09:06:45.929134+0000","mapping_epoch":28,"log_start":"0'0","ondisk_log_start":"0'0","created":28,"last_epoch_clean":49,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-06-03T08:37:55.614497+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-06-03T08:37:55.614497+0000","last_clean_scrub_stamp":"2024-06-03T08:37:55.614497+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-06-04T12:27:02.532436+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,4],"acting":[0,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"17.4","version":"0'0","reported_seq":843,"reported_epoch":1014,"state":"active+clean","last_fresh":"2024-06-03T09:06:46.456388+0000","last_change":"2024-06-03T09:06:46.456388+0000","last_active":"2024-06-03T09:06:46.456388+0000","last_peered":"2024-06-03T09:06:46.456388+0000","last_clean":"2024-06-03T09:06:46.456388+0000","last_became_active":"2024-06-03T09:06:46.447534+0000","last_became_peered":"2024-06-03T09:06:46.447534+0000","last_unstale":"2024-06-03T09:06:46.456388+0000","last_undegraded":"2024-06-03T09:06:46.456388+0000","last_fullsized":"2024-06-03T09:06:46.456388+0000","mapping_epoch":1013,"log_start":"0'0","ondisk_log_start":"0'0","created":28,"last_epoch_clean":1014,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-06-03T08:37:55.614497+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-06-03T08:37:55.614497+0000","last_clean_scrub_stamp":"2024-06-03T08:37:55.614497+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-06-04T16:06:17.792570+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,7],"acting":[6,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"17.7","version":"0'0","reported_seq":933,"reported_epoch":1013,"state":"active+clean","last_fresh":"2024-06-03T09:06:45.495807+0000","last_change":"2024-06-03T08:38:04.837083+0000","last_active":"2024-06-03T09:06:45.495807+0000","last_peered":"2024-06-03T09:06:45.495807+0000","last_clean":"2024-06-03T09:06:45.495807+0000","last_became_active":"2024-06-03T08:38:04.818246+0000","last_became_peered":"2024-06-03T08:38:04.818246+0000","last_unstale":"2024-06-03T09:06:45.495807+0000","last_undegraded":"2024-06-03T09:06:45.495807+0000","last_fullsized":"2024-06-03T09:06:45.495807+0000","mapping_epoch":28,"log_start":"0'0","ondisk_log_start":"0'0","created":28,"last_epoch_clean":34,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-06-03T08:37:55.614497+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-06-03T08:37:55.614497+0000","last_clean_scrub_stamp":"2024-06-03T08:37:55.614497+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-06-04T17:02:38.204813+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,2],"acting":[5,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"17.6","version":"0'0","reported_seq":971,"reported_epoch":1014,"state":"active+clean","last_fresh":"2024-06-03T09:06:46.362623+0000","last_change":"2024-06-03T09:06:46.362623+0000","last_active":"2024-06-03T09:06:46.362623+0000","last_peered":"2024-06-03T09:06:46.362623+0000","last_clean":"2024-06-03T09:06:46.362623+0000","last_became_active":"2024-06-03T09:06:46.357698+0000","last_became_peered":"2024-06-03T09:06:46.357698+0000","last_unstale":"2024-06-03T09:06:46.362623+0000","last_undegraded":"2024-06-03T09:06:46.362623+0000","last_fullsized":"2024-06-03T09:06:46.362623+0000","mapping_epoch":1013,"log_start":"0'0","ondisk_log_start":"0'0","created":28,"last_epoch_clean":1014,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-06-03T08:37:55.614497+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-06-03T08:37:55.614497+0000","last_clean_scrub_stamp":"2024-06-03T08:37:55.614497+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-06-04T12:51:05.494877+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,6],"acting":[0,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"2.7","version":"0'0","reported_seq":944,"reported_epoch":1013,"state":"active+clean","last_fresh":"2024-06-03T09:06:45.158971+0000","last_change":"2024-06-03T08:36:30.385405+0000","last_active":"2024-06-03T09:06:45.158971+0000","last_peered":"2024-06-03T09:06:45.158971+0000","last_clean":"2024-06-03T09:06:45.158971+0000","last_became_active":"2024-06-03T08:36:30.380302+0000","last_became_peered":"2024-06-03T08:36:30.380302+0000","last_unstale":"2024-06-03T09:06:45.158971+0000","last_undegraded":"2024-06-03T09:06:45.158971+0000","last_fullsized":"2024-06-03T09:06:45.158971+0000","mapping_epoch":23,"log_start":"0'0","ondisk_log_start":"0'0","created":23,"last_epoch_clean":24,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-06-03T08:36:28.444764+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-06-03T08:36:28.444764+0000","last_clean_scrub_stamp":"2024-06-03T08:36:28.444764+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-06-04T18:46:33.475181+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0],"acting":[3,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"17.14","version":"0'0","reported_seq":930,"reported_epoch":1013,"state":"active+clean","last_fresh":"2024-06-03T09:06:45.541591+0000","last_change":"2024-06-03T08:38:04.547565+0000","last_active":"2024-06-03T09:06:45.541591+0000","last_peered":"2024-06-03T09:06:45.541591+0000","last_clean":"2024-06-03T09:06:45.541591+0000","last_became_active":"2024-06-03T08:38:04.541781+0000","last_became_peered":"2024-06-03T08:38:04.541781+0000","last_unstale":"2024-06-03T09:06:45.541591+0000","last_undegraded":"2024-06-03T09:06:45.541591+0000","last_fullsized":"2024-06-03T09:06:45.541591+0000","mapping_epoch":28,"log_start":"0'0","ondisk_log_start":"0'0","created":28,"last_epoch_clean":34,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-06-03T08:37:55.614497+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-06-03T08:37:55.614497+0000","last_clean_scrub_stamp":"2024-06-03T08:37:55.614497+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-06-04T16:41:55.170134+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,2],"acting":[5,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"2.6","version":"0'0","reported_seq":953,"reported_epoch":1013,"state":"active+clean","last_fresh":"2024-06-03T09:06:45.018770+0000","last_change":"2024-06-03T08:36:30.430760+0000","last_active":"2024-06-03T09:06:45.018770+0000","last_peered":"2024-06-03T09:06:45.018770+0000","last_clean":"2024-06-03T09:06:45.018770+0000","last_became_active":"2024-06-03T08:36:30.404665+0000","last_became_peered":"2024-06-03T08:36:30.404665+0000","last_unstale":"2024-06-03T09:06:45.018770+0000","last_undegraded":"2024-06-03T09:06:45.018770+0000","last_fullsized":"2024-06-03T09:06:45.018770+0000","mapping_epoch":23,"log_start":"0'0","ondisk_log_start":"0'0","created":23,"last_epoch_clean":24,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-06-03T08:36:28.444764+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-06-03T08:36:28.444764+0000","last_clean_scrub_stamp":"2024-06-03T08:36:28.444764+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-06-04T10:37:37.746074+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,0],"acting":[4,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"17.15","version":"0'0","reported_seq":938,"reported_epoch":1013,"state":"active+clean","last_fresh":"2024-06-03T09:06:45.160758+0000","last_change":"2024-06-03T08:38:20.999903+0000","last_active":"2024-06-03T09:06:45.160758+0000","last_peered":"2024-06-03T09:06:45.160758+0000","last_clean":"2024-06-03T09:06:45.160758+0000","last_became_active":"2024-06-03T08:38:20.979523+0000","last_became_peered":"2024-06-03T08:38:20.979523+0000","last_unstale":"2024-06-03T09:06:45.160758+0000","last_undegraded":"2024-06-03T09:06:45.160758+0000","last_fullsized":"2024-06-03T09:06:45.160758+0000","mapping_epoch":28,"log_start":"0'0","ondisk_log_start":"0'0","created":28,"last_epoch_clean":47,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-06-03T08:37:55.614497+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-06-03T08:37:55.614497+0000","last_clean_scrub_stamp":"2024-06-03T08:37:55.614497+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-06-04T20:29:58.314289+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,1],"acting":[3,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"2.5","version":"0'0","reported_seq":946,"reported_epoch":1013,"state":"active+clean","last_fresh":"2024-06-03T09:06:45.162304+0000","last_change":"2024-06-03T08:36:30.024235+0000","last_active":"2024-06-03T09:06:45.162304+0000","last_peered":"2024-06-03T09:06:45.162304+0000","last_clean":"2024-06-03T09:06:45.162304+0000","last_became_active":"2024-06-03T08:36:30.019040+0000","last_became_peered":"2024-06-03T08:36:30.019040+0000","last_unstale":"2024-06-03T09:06:45.162304+0000","last_undegraded":"2024-06-03T09:06:45.162304+0000","last_fullsized":"2024-06-03T09:06:45.162304+0000","mapping_epoch":23,"log_start":"0'0","ondisk_log_start":"0'0","created":23,"last_epoch_clean":24,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-06-03T08:36:28.444764+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-06-03T08:36:28.444764+0000","last_clean_scrub_stamp":"2024-06-03T08:36:28.444764+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-06-04T15:46:47.109502+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,4],"acting":[3,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"17.16","version":"0'0","reported_seq":984,"reported_epoch":1014,"state":"active+clean","last_fresh":"2024-06-03T09:06:46.384595+0000","last_change":"2024-06-03T09:06:46.384595+0000","last_active":"2024-06-03T09:06:46.384595+0000","last_peered":"2024-06-03T09:06:46.384595+0000","last_clean":"2024-06-03T09:06:46.384595+0000","last_became_active":"2024-06-03T09:06:46.379518+0000","last_became_peered":"2024-06-03T09:06:46.379518+0000","last_unstale":"2024-06-03T09:06:46.384595+0000","last_undegraded":"2024-06-03T09:06:46.384595+0000","last_fullsized":"2024-06-03T09:06:46.384595+0000","mapping_epoch":1013,"log_start":"0'0","ondisk_log_start":"0'0","created":28,"last_epoch_clean":1014,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-06-03T08:37:55.614497+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-06-03T08:37:55.614497+0000","last_clean_scrub_stamp":"2024-06-03T08:37:55.614497+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-06-04T12:51:20.305056+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,6],"acting":[2,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"2.4","version":"0'0","reported_seq":964,"reported_epoch":1014,"state":"active+clean","last_fresh":"2024-06-03T09:06:46.377610+0000","last_change":"2024-06-03T09:06:46.377610+0000","last_active":"2024-06-03T09:06:46.377610+0000","last_peered":"2024-06-03T09:06:46.377610+0000","last_clean":"2024-06-03T09:06:46.377610+0000","last_became_active":"2024-06-03T09:06:46.371430+0000","last_became_peered":"2024-06-03T09:06:46.371430+0000","last_unstale":"2024-06-03T09:06:46.377610+0000","last_undegraded":"2024-06-03T09:06:46.377610+0000","last_fullsized":"2024-06-03T09:06:46.377610+0000","mapping_epoch":1013,"log_start":"0'0","ondisk_log_start":"0'0","created":23,"last_epoch_clean":1014,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-06-03T08:36:28.444764+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-06-03T08:36:28.444764+0000","last_clean_scrub_stamp":"2024-06-03T08:36:28.444764+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-06-04T12:20:39.476337+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,6],"acting":[5,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"17.17","version":"0'0","reported_seq":958,"reported_epoch":1013,"state":"active+clean","last_fresh":"2024-06-03T09:06:45.796338+0000","last_change":"2024-06-03T08:38:38.483275+0000","last_active":"2024-06-03T09:06:45.796338+0000","last_peered":"2024-06-03T09:06:45.796338+0000","last_clean":"2024-06-03T09:06:45.796338+0000","last_became_active":"2024-06-03T08:38:38.467916+0000","last_became_peered":"2024-06-03T08:38:38.467916+0000","last_unstale":"2024-06-03T09:06:45.796338+0000","last_undegraded":"2024-06-03T09:06:45.796338+0000","last_fullsized":"2024-06-03T09:06:45.796338+0000","mapping_epoch":28,"log_start":"0'0","ondisk_log_start":"0'0","created":28,"last_epoch_clean":61,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-06-03T08:37:55.614497+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-06-03T08:37:55.614497+0000","last_clean_scrub_stamp":"2024-06-03T08:37:55.614497+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-06-04T10:02:23.036420+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,7],"acting":[2,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"2.2","version":"0'0","reported_seq":975,"reported_epoch":1014,"state":"active+clean","last_fresh":"2024-06-03T09:06:46.290105+0000","last_change":"2024-06-03T09:06:46.290105+0000","last_active":"2024-06-03T09:06:46.290105+0000","last_peered":"2024-06-03T09:06:46.290105+0000","last_clean":"2024-06-03T09:06:46.290105+0000","last_became_active":"2024-06-03T09:06:46.284836+0000","last_became_peered":"2024-06-03T09:06:46.284836+0000","last_unstale":"2024-06-03T09:06:46.290105+0000","last_undegraded":"2024-06-03T09:06:46.290105+0000","last_fullsized":"2024-06-03T09:06:46.290105+0000","mapping_epoch":1013,"log_start":"0'0","ondisk_log_start":"0'0","created":23,"last_epoch_clean":1014,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-06-03T08:36:28.444764+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-06-03T08:36:28.444764+0000","last_clean_scrub_stamp":"2024-06-03T08:36:28.444764+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-06-04T14:05:42.945227+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,6],"acting":[3,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"17.11","version":"0'0","reported_seq":947,"reported_epoch":1013,"state":"active+clean","last_fresh":"2024-06-03T09:06:44.976905+0000","last_change":"2024-06-03T08:38:06.502302+0000","last_active":"2024-06-03T09:06:44.976905+0000","last_peered":"2024-06-03T09:06:44.976905+0000","last_clean":"2024-06-03T09:06:44.976905+0000","last_became_active":"2024-06-03T08:38:06.490204+0000","last_became_peered":"2024-06-03T08:38:06.490204+0000","last_unstale":"2024-06-03T09:06:44.976905+0000","last_undegraded":"2024-06-03T09:06:44.976905+0000","last_fullsized":"2024-06-03T09:06:44.976905+0000","mapping_epoch":28,"log_start":"0'0","ondisk_log_start":"0'0","created":28,"last_epoch_clean":32,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-06-03T08:37:55.614497+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-06-03T08:37:55.614497+0000","last_clean_scrub_stamp":"2024-06-03T08:37:55.614497+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-06-04T10:47:05.988958+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,2],"acting":[4,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"2.1","version":"0'0","reported_seq":955,"reported_epoch":1013,"state":"active+clean","last_fresh":"2024-06-03T09:06:45.423921+0000","last_change":"2024-06-03T08:36:30.883360+0000","last_active":"2024-06-03T09:06:45.423921+0000","last_peered":"2024-06-03T09:06:45.423921+0000","last_clean":"2024-06-03T09:06:45.423921+0000","last_became_active":"2024-06-03T08:36:30.855073+0000","last_became_peered":"2024-06-03T08:36:30.855073+0000","last_unstale":"2024-06-03T09:06:45.423921+0000","last_undegraded":"2024-06-03T09:06:45.423921+0000","last_fullsized":"2024-06-03T09:06:45.423921+0000","mapping_epoch":23,"log_start":"0'0","ondisk_log_start":"0'0","created":23,"last_epoch_clean":25,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-06-03T08:36:28.444764+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-06-03T08:36:28.444764+0000","last_clean_scrub_stamp":"2024-06-03T08:36:28.444764+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-06-04T18:12:22.363473+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,5],"acting":[7,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"17.12","version":"0'0","reported_seq":948,"reported_epoch":1013,"state":"active+clean","last_fresh":"2024-06-03T09:06:45.447050+0000","last_change":"2024-06-03T08:38:01.751562+0000","last_active":"2024-06-03T09:06:45.447050+0000","last_peered":"2024-06-03T09:06:45.447050+0000","last_clean":"2024-06-03T09:06:45.447050+0000","last_became_active":"2024-06-03T08:38:01.745027+0000","last_became_peered":"2024-06-03T08:38:01.745027+0000","last_unstale":"2024-06-03T09:06:45.447050+0000","last_undegraded":"2024-06-03T09:06:45.447050+0000","last_fullsized":"2024-06-03T09:06:45.447050+0000","mapping_epoch":28,"log_start":"0'0","ondisk_log_start":"0'0","created":28,"last_epoch_clean":32,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-06-03T08:37:55.614497+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-06-03T08:37:55.614497+0000","last_clean_scrub_stamp":"2024-06-03T08:37:55.614497+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-06-04T17:54:43.785326+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,3],"acting":[7,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"2.0","version":"0'0","reported_seq":948,"reported_epoch":1013,"state":"active+clean","last_fresh":"2024-06-03T09:06:45.108519+0000","last_change":"2024-06-03T08:36:29.984096+0000","last_active":"2024-06-03T09:06:45.108519+0000","last_peered":"2024-06-03T09:06:45.108519+0000","last_clean":"2024-06-03T09:06:45.108519+0000","last_became_active":"2024-06-03T08:36:29.957875+0000","last_became_peered":"2024-06-03T08:36:29.957875+0000","last_unstale":"2024-06-03T09:06:45.108519+0000","last_undegraded":"2024-06-03T09:06:45.108519+0000","last_fullsized":"2024-06-03T09:06:45.108519+0000","mapping_epoch":23,"log_start":"0'0","ondisk_log_start":"0'0","created":23,"last_epoch_clean":24,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-06-03T08:36:28.444764+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-06-03T08:36:28.444764+0000","last_clean_scrub_stamp":"2024-06-03T08:36:28.444764+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-06-04T08:41:33.679479+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,5],"acting":[3,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"17.13","version":"0'0","reported_seq":946,"reported_epoch":1013,"state":"active+clean","last_fresh":"2024-06-03T09:06:44.996126+0000","last_change":"2024-06-03T08:38:26.838316+0000","last_active":"2024-06-03T09:06:44.996126+0000","last_peered":"2024-06-03T09:06:44.996126+0000","last_clean":"2024-06-03T09:06:44.996126+0000","last_became_active":"2024-06-03T08:38:26.833214+0000","last_became_peered":"2024-06-03T08:38:26.833214+0000","last_unstale":"2024-06-03T09:06:44.996126+0000","last_undegraded":"2024-06-03T09:06:44.996126+0000","last_fullsized":"2024-06-03T09:06:44.996126+0000","mapping_epoch":28,"log_start":"0'0","ondisk_log_start":"0'0","created":28,"last_epoch_clean":52,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-06-03T08:37:55.614497+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-06-03T08:37:55.614497+0000","last_clean_scrub_stamp":"2024-06-03T08:37:55.614497+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-06-04T12:58:51.617998+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,0],"acting":[4,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"2.3","version":"0'0","reported_seq":947,"reported_epoch":1013,"state":"active+clean","last_fresh":"2024-06-03T09:06:45.137572+0000","last_change":"2024-06-03T08:36:30.351670+0000","last_active":"2024-06-03T09:06:45.137572+0000","last_peered":"2024-06-03T09:06:45.137572+0000","last_clean":"2024-06-03T09:06:45.137572+0000","last_became_active":"2024-06-03T08:36:30.343560+0000","last_became_peered":"2024-06-03T08:36:30.343560+0000","last_unstale":"2024-06-03T09:06:45.137572+0000","last_undegraded":"2024-06-03T09:06:45.137572+0000","last_fullsized":"2024-06-03T09:06:45.137572+0000","mapping_epoch":23,"log_start":"0'0","ondisk_log_start":"0'0","created":23,"last_epoch_clean":24,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-06-03T08:36:28.444764+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-06-03T08:36:28.444764+0000","last_clean_scrub_stamp":"2024-06-03T08:36:28.444764+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-06-04T12:02:24.765135+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,2],"acting":[3,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.0","version":"19'76","reported_seq":1056,"reported_epoch":1014,"state":"active+clean","last_fresh":"2024-06-03T09:06:46.463038+0000","last_change":"2024-06-03T08:36:26.702702+0000","last_active":"2024-06-03T09:06:46.463038+0000","last_peered":"2024-06-03T09:06:46.463038+0000","last_clean":"2024-06-03T09:06:46.463038+0000","last_became_active":"2024-06-03T08:36:26.128378+0000","last_became_peered":"2024-06-03T08:36:26.128378+0000","last_unstale":"2024-06-03T09:06:46.463038+0000","last_undegraded":"2024-06-03T09:06:46.463038+0000","last_fullsized":"2024-06-03T09:06:46.463038+0000","mapping_epoch":20,"log_start":"0'0","ondisk_log_start":"0'0","created":16,"last_epoch_clean":22,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-06-03T08:36:15.542635+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-06-03T08:36:15.542635+0000","last_clean_scrub_stamp":"2024-06-03T08:36:15.542635+0000","objects_scrubbed":0,"log_size":76,"log_dups_size":0,"ondisk_log_size":76,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-06-04T12:41:53.731360+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":459280,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":96,"num_read_kb":82,"num_write":113,"num_write_kb":1372,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2,"num_bytes_recovered":397840,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,5],"acting":[1,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"17.10","version":"0'0","reported_seq":947,"reported_epoch":1013,"state":"active+clean","last_fresh":"2024-06-03T09:06:44.974251+0000","last_change":"2024-06-03T08:38:15.399189+0000","last_active":"2024-06-03T09:06:44.974251+0000","last_peered":"2024-06-03T09:06:44.974251+0000","last_clean":"2024-06-03T09:06:44.974251+0000","last_became_active":"2024-06-03T08:38:15.337723+0000","last_became_peered":"2024-06-03T08:38:15.337723+0000","last_unstale":"2024-06-03T09:06:44.974251+0000","last_undegraded":"2024-06-03T09:06:44.974251+0000","last_fullsized":"2024-06-03T09:06:44.974251+0000","mapping_epoch":28,"log_start":"0'0","ondisk_log_start":"0'0","created":28,"last_epoch_clean":42,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-06-03T08:37:55.614497+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-06-03T08:37:55.614497+0000","last_clean_scrub_stamp":"2024-06-03T08:37:55.614497+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-06-04T09:29:46.427317+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,3],"acting":[4,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"17.1a","version":"0'0","reported_seq":971,"reported_epoch":1014,"state":"active+clean","last_fresh":"2024-06-03T09:06:46.393342+0000","last_change":"2024-06-03T09:06:46.393342+0000","last_active":"2024-06-03T09:06:46.393342+0000","last_peered":"2024-06-03T09:06:46.393342+0000","last_clean":"2024-06-03T09:06:46.393342+0000","last_became_active":"2024-06-03T09:06:46.386469+0000","last_became_peered":"2024-06-03T09:06:46.386469+0000","last_unstale":"2024-06-03T09:06:46.393342+0000","last_undegraded":"2024-06-03T09:06:46.393342+0000","last_fullsized":"2024-06-03T09:06:46.393342+0000","mapping_epoch":1013,"log_start":"0'0","ondisk_log_start":"0'0","created":28,"last_epoch_clean":1014,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-06-03T08:37:55.614497+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-06-03T08:37:55.614497+0000","last_clean_scrub_stamp":"2024-06-03T08:37:55.614497+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-06-04T13:22:45.876282+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,6],"acting":[4,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"17.1b","version":"0'0","reported_seq":939,"reported_epoch":1013,"state":"active+clean","last_fresh":"2024-06-03T09:06:45.136039+0000","last_change":"2024-06-03T08:38:03.115748+0000","last_active":"2024-06-03T09:06:45.136039+0000","last_peered":"2024-06-03T09:06:45.136039+0000","last_clean":"2024-06-03T09:06:45.136039+0000","last_became_active":"2024-06-03T08:38:03.110533+0000","last_became_peered":"2024-06-03T08:38:03.110533+0000","last_unstale":"2024-06-03T09:06:45.136039+0000","last_undegraded":"2024-06-03T09:06:45.136039+0000","last_fullsized":"2024-06-03T09:06:45.136039+0000","mapping_epoch":28,"log_start":"0'0","ondisk_log_start":"0'0","created":28,"last_epoch_clean":32,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-06-03T08:37:55.614497+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-06-03T08:37:55.614497+0000","last_clean_scrub_stamp":"2024-06-03T08:37:55.614497+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-06-04T12:36:47.471609+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,7],"acting":[3,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"17.18","version":"0'0","reported_seq":947,"reported_epoch":1013,"state":"active+clean","last_fresh":"2024-06-03T09:06:45.420817+0000","last_change":"2024-06-03T08:38:04.632864+0000","last_active":"2024-06-03T09:06:45.420817+0000","last_peered":"2024-06-03T09:06:45.420817+0000","last_clean":"2024-06-03T09:06:45.420817+0000","last_became_active":"2024-06-03T08:38:04.627856+0000","last_became_peered":"2024-06-03T08:38:04.627856+0000","last_unstale":"2024-06-03T09:06:45.420817+0000","last_undegraded":"2024-06-03T09:06:45.420817+0000","last_fullsized":"2024-06-03T09:06:45.420817+0000","mapping_epoch":28,"log_start":"0'0","ondisk_log_start":"0'0","created":28,"last_epoch_clean":33,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-06-03T08:37:55.614497+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-06-03T08:37:55.614497+0000","last_clean_scrub_stamp":"2024-06-03T08:37:55.614497+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-06-04T19:55:51.181598+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,4],"acting":[7,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"17.19","version":"0'0","reported_seq":938,"reported_epoch":1013,"state":"active+clean","last_fresh":"2024-06-03T09:06:45.129069+0000","last_change":"2024-06-03T08:38:01.839491+0000","last_active":"2024-06-03T09:06:45.129069+0000","last_peered":"2024-06-03T09:06:45.129069+0000","last_clean":"2024-06-03T09:06:45.129069+0000","last_became_active":"2024-06-03T08:38:01.829539+0000","last_became_peered":"2024-06-03T08:38:01.829539+0000","last_unstale":"2024-06-03T09:06:45.129069+0000","last_undegraded":"2024-06-03T09:06:45.129069+0000","last_fullsized":"2024-06-03T09:06:45.129069+0000","mapping_epoch":28,"log_start":"0'0","ondisk_log_start":"0'0","created":28,"last_epoch_clean":30,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-06-03T08:37:55.614497+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-06-03T08:37:55.614497+0000","last_clean_scrub_stamp":"2024-06-03T08:37:55.614497+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-06-04T09:10:43.860229+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,2],"acting":[3,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"17.1e","version":"0'0","reported_seq":844,"reported_epoch":1014,"state":"active+clean","last_fresh":"2024-06-03T09:06:46.297025+0000","last_change":"2024-06-03T09:06:46.297025+0000","last_active":"2024-06-03T09:06:46.297025+0000","last_peered":"2024-06-03T09:06:46.297025+0000","last_clean":"2024-06-03T09:06:46.297025+0000","last_became_active":"2024-06-03T09:06:46.288593+0000","last_became_peered":"2024-06-03T09:06:46.288593+0000","last_unstale":"2024-06-03T09:06:46.297025+0000","last_undegraded":"2024-06-03T09:06:46.297025+0000","last_fullsized":"2024-06-03T09:06:46.297025+0000","mapping_epoch":1013,"log_start":"0'0","ondisk_log_start":"0'0","created":28,"last_epoch_clean":1014,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-06-03T08:37:55.614497+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-06-03T08:37:55.614497+0000","last_clean_scrub_stamp":"2024-06-03T08:37:55.614497+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-06-04T20:09:30.144366+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,4],"acting":[6,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"17.1f","version":"0'0","reported_seq":944,"reported_epoch":1013,"state":"active+clean","last_fresh":"2024-06-03T09:06:45.933220+0000","last_change":"2024-06-03T08:38:04.383538+0000","last_active":"2024-06-03T09:06:45.933220+0000","last_peered":"2024-06-03T09:06:45.933220+0000","last_clean":"2024-06-03T09:06:45.933220+0000","last_became_active":"2024-06-03T08:38:04.367278+0000","last_became_peered":"2024-06-03T08:38:04.367278+0000","last_unstale":"2024-06-03T09:06:45.933220+0000","last_undegraded":"2024-06-03T09:06:45.933220+0000","last_fullsized":"2024-06-03T09:06:45.933220+0000","mapping_epoch":28,"log_start":"0'0","ondisk_log_start":"0'0","created":28,"last_epoch_clean":32,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-06-03T08:37:55.614497+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-06-03T08:37:55.614497+0000","last_clean_scrub_stamp":"2024-06-03T08:37:55.614497+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-06-04T10:39:26.552282+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,2],"acting":[0,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"17.1c","version":"0'0","reported_seq":956,"reported_epoch":1014,"state":"active+clean","last_fresh":"2024-06-03T09:06:46.410116+0000","last_change":"2024-06-03T09:06:46.410116+0000","last_active":"2024-06-03T09:06:46.410116+0000","last_peered":"2024-06-03T09:06:46.410116+0000","last_clean":"2024-06-03T09:06:46.410116+0000","last_became_active":"2024-06-03T09:06:46.403724+0000","last_became_peered":"2024-06-03T09:06:46.403724+0000","last_unstale":"2024-06-03T09:06:46.410116+0000","last_undegraded":"2024-06-03T09:06:46.410116+0000","last_fullsized":"2024-06-03T09:06:46.410116+0000","mapping_epoch":1013,"log_start":"0'0","ondisk_log_start":"0'0","created":28,"last_epoch_clean":1014,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-06-03T08:37:55.614497+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-06-03T08:37:55.614497+0000","last_clean_scrub_stamp":"2024-06-03T08:37:55.614497+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-06-04T19:40:47.504202+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,6],"acting":[5,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"17.1d","version":"0'0","reported_seq":927,"reported_epoch":1013,"state":"active+clean","last_fresh":"2024-06-03T09:06:45.879478+0000","last_change":"2024-06-03T08:38:06.794630+0000","last_active":"2024-06-03T09:06:45.879478+0000","last_peered":"2024-06-03T09:06:45.879478+0000","last_clean":"2024-06-03T09:06:45.879478+0000","last_became_active":"2024-06-03T08:38:06.592317+0000","last_became_peered":"2024-06-03T08:38:06.592317+0000","last_unstale":"2024-06-03T09:06:45.879478+0000","last_undegraded":"2024-06-03T09:06:45.879478+0000","last_fullsized":"2024-06-03T09:06:45.879478+0000","mapping_epoch":28,"log_start":"0'0","ondisk_log_start":"0'0","created":28,"last_epoch_clean":33,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-06-03T08:37:55.614497+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-06-03T08:37:55.614497+0000","last_clean_scrub_stamp":"2024-06-03T08:37:55.614497+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-06-04T12:33:30.785369+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,7],"acting":[1,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"17.2","version":"0'0","reported_seq":929,"reported_epoch":1013,"state":"active+clean","last_fresh":"2024-06-03T09:06:45.881178+0000","last_change":"2024-06-03T08:38:14.981410+0000","last_active":"2024-06-03T09:06:45.881178+0000","last_peered":"2024-06-03T09:06:45.881178+0000","last_clean":"2024-06-03T09:06:45.881178+0000","last_became_active":"2024-06-03T08:38:14.973825+0000","last_became_peered":"2024-06-03T08:38:14.973825+0000","last_unstale":"2024-06-03T09:06:45.881178+0000","last_undegraded":"2024-06-03T09:06:45.881178+0000","last_fullsized":"2024-06-03T09:06:45.881178+0000","mapping_epoch":28,"log_start":"0'0","ondisk_log_start":"0'0","created":28,"last_epoch_clean":41,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-06-03T08:37:55.614497+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-06-03T08:37:55.614497+0000","last_clean_scrub_stamp":"2024-06-03T08:37:55.614497+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-06-04T17:01:05.518613+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,4],"acting":[1,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"17.3","version":"0'0","reported_seq":946,"reported_epoch":1013,"state":"active+clean","last_fresh":"2024-06-03T09:06:45.934804+0000","last_change":"2024-06-03T08:38:14.619745+0000","last_active":"2024-06-03T09:06:45.934804+0000","last_peered":"2024-06-03T09:06:45.934804+0000","last_clean":"2024-06-03T09:06:45.934804+0000","last_became_active":"2024-06-03T08:38:14.510665+0000","last_became_peered":"2024-06-03T08:38:14.510665+0000","last_unstale":"2024-06-03T09:06:45.934804+0000","last_undegraded":"2024-06-03T09:06:45.934804+0000","last_fullsized":"2024-06-03T09:06:45.934804+0000","mapping_epoch":28,"log_start":"0'0","ondisk_log_start":"0'0","created":28,"last_epoch_clean":42,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-06-03T08:37:55.614497+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-06-03T08:37:55.614497+0000","last_clean_scrub_stamp":"2024-06-03T08:37:55.614497+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-06-04T12:03:47.392064+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,7],"acting":[0,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"17.0","version":"0'0","reported_seq":947,"reported_epoch":1013,"state":"active+clean","last_fresh":"2024-06-03T09:06:45.920467+0000","last_change":"2024-06-03T08:38:04.237118+0000","last_active":"2024-06-03T09:06:45.920467+0000","last_peered":"2024-06-03T09:06:45.920467+0000","last_clean":"2024-06-03T09:06:45.920467+0000","last_became_active":"2024-06-03T08:38:04.231954+0000","last_became_peered":"2024-06-03T08:38:04.231954+0000","last_unstale":"2024-06-03T09:06:45.920467+0000","last_undegraded":"2024-06-03T09:06:45.920467+0000","last_fullsized":"2024-06-03T09:06:45.920467+0000","mapping_epoch":28,"log_start":"0'0","ondisk_log_start":"0'0","created":28,"last_epoch_clean":32,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-06-03T08:37:55.614497+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-06-03T08:37:55.614497+0000","last_clean_scrub_stamp":"2024-06-03T08:37:55.614497+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-06-04T09:56:10.741637+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,2],"acting":[0,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"17.1","version":"65'7","reported_seq":971,"reported_epoch":1014,"state":"active+clean","last_fresh":"2024-06-03T09:06:46.389760+0000","last_change":"2024-06-03T09:06:46.389760+0000","last_active":"2024-06-03T09:06:46.389760+0000","last_peered":"2024-06-03T09:06:46.389760+0000","last_clean":"2024-06-03T09:06:46.389760+0000","last_became_active":"2024-06-03T09:06:46.383290+0000","last_became_peered":"2024-06-03T09:06:46.383290+0000","last_unstale":"2024-06-03T09:06:46.389760+0000","last_undegraded":"2024-06-03T09:06:46.389760+0000","last_fullsized":"2024-06-03T09:06:46.389760+0000","mapping_epoch":1013,"log_start":"0'0","ondisk_log_start":"0'0","created":28,"last_epoch_clean":1014,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-06-03T08:37:55.614497+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-06-03T08:37:55.614497+0000","last_clean_scrub_stamp":"2024-06-03T08:37:55.614497+0000","objects_scrubbed":0,"log_size":7,"log_dups_size":0,"ondisk_log_size":7,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-06-04T10:38:34.982254+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":5,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":6,"num_read_kb":6,"num_write":7,"num_write_kb":7,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,6],"acting":[5,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]}],"pool_stats":[{"poolid":2,"num_pg":8,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":16,"acting":16,"num_store_stats":7},{"poolid":1,"num_pg":1,"stat_sum":{"num_bytes":459280,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":96,"num_read_kb":82,"num_write":113,"num_write_kb":1372,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2,"num_bytes_recovered":397840,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":802816,"data_stored":795680,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":76,"ondisk_log_size":76,"up":2,"acting":2,"num_store_stats":2},{"poolid":17,"num_pg":32,"stat_sum":{"num_bytes":5,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":6,"num_read_kb":6,"num_write":7,"num_write_kb":7,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":8192,"data_stored":10,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":7,"ondisk_log_size":7,"up":64,"acting":64,"num_store_stats":8}],"osd_stats":[{"osd":6,"up_from":1013,"seq":4350801871163,"num_pgs":14,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":54924,"kb_used_data":5756,"kb_used_omap":1,"kb_used_meta":49150,"kb_avail":94316916,"statfs":{"total":96636764160,"available":96580521984,"internally_reserved":0,"allocated":5894144,"data_stored":2763078,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1593,"internal_metadata":50330055},"hb_peers":[0,1,2,3,4,5,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Mon Jun 3 09:06:42 2024","interfaces":[{"interface":"back","average":{"1min":5696.3109999999997,"5min":1469.3900000000001,"15min":503.11700000000002},"min":{"1min":5.242,"5min":4.4829999999999997,"15min":4.1959999999999997},"max":{"1min":20463.029999999999,"5min":22766.882000000001,"15min":22766.882000000001},"last":5.1619999999999999},{"interface":"front","average":{"1min":5687.6390000000001,"5min":1468.1669999999999,"15min":504.13200000000001},"min":{"1min":4.835,"5min":4.8129999999999997,"15min":4.4530000000000003},"max":{"1min":20461.830000000002,"5min":22766.289000000001,"15min":22766.289000000001},"last":9.1899999999999995}]},{"osd":1,"last update":"Mon Jun 3 09:06:42 2024","interfaces":[{"interface":"back","average":{"1min":36.496000000000002,"5min":80.162999999999997,"15min":134.59399999999999},"min":{"1min":5.9189999999999996,"5min":5.8799999999999999,"15min":5.7119999999999997},"max":{"1min":544.85799999999995,"5min":4138.1679999999997,"15min":9955.1299999999992},"last":12.109999999999999},{"interface":"front","average":{"1min":38.866999999999997,"5min":82.631,"15min":135.11500000000001},"min":{"1min":6.5350000000000001,"5min":6.5350000000000001,"15min":6.1390000000000002},"max":{"1min":546.73099999999999,"5min":4137.6530000000002,"15min":9827.5509999999995},"last":10.295999999999999}]},{"osd":2,"last update":"Mon Jun 3 09:06:42 2024","interfaces":[{"interface":"back","average":{"1min":41.093000000000004,"5min":27.917000000000002,"15min":22.329000000000001},"min":{"1min":6.0670000000000002,"5min":6.0670000000000002,"15min":6.0670000000000002},"max":{"1min":548.12900000000002,"5min":548.12900000000002,"15min":548.12900000000002},"last":6.452},{"interface":"front","average":{"1min":30.798999999999999,"5min":24.276,"15min":22.488},"min":{"1min":6.4370000000000003,"5min":6.4370000000000003,"15min":6.1870000000000003},"max":{"1min":369.14499999999998,"5min":369.14499999999998,"15min":434.77300000000002},"last":11.317}]},{"osd":3,"last update":"Mon Jun 3 09:06:42 2024","interfaces":[{"interface":"back","average":{"1min":41.411999999999999,"5min":27.434999999999999,"15min":30.657},"min":{"1min":10.061,"5min":8.3179999999999996,"15min":6.1719999999999997},"max":{"1min":549.024,"5min":549.024,"15min":1235.5799999999999},"last":9.7409999999999997},{"interface":"front","average":{"1min":37.850999999999999,"5min":22.001000000000001,"15min":28.015999999999998},"min":{"1min":6.0679999999999996,"5min":6.0679999999999996,"15min":6.0679999999999996},"max":{"1min":559.25,"5min":559.25,"15min":1157.6780000000001},"last":7.0650000000000004}]},{"osd":4,"last update":"Mon Jun 3 09:06:42 2024","interfaces":[{"interface":"back","average":{"1min":165.62899999999999,"5min":1281.1010000000001,"15min":440.59199999999998},"min":{"1min":10.965999999999999,"5min":6.093,"15min":6.093},"max":{"1min":384.46899999999999,"5min":28093.096000000001,"15min":28093.096000000001},"last":12.743},{"interface":"front","average":{"1min":186.97900000000001,"5min":1308.886,"15min":454.97699999999998},"min":{"1min":13.581,"5min":7.7480000000000002,"15min":7.7480000000000002},"max":{"1min":445.096,"5min":28323.837,"15min":28323.837},"last":14.276}]},{"osd":5,"last update":"Mon Jun 3 09:06:42 2024","interfaces":[{"interface":"back","average":{"1min":81.581000000000003,"5min":34.863,"15min":28.350999999999999},"min":{"1min":10.345000000000001,"5min":8.625,"15min":7.516},"max":{"1min":455.84199999999998,"5min":455.84199999999998,"15min":455.84199999999998},"last":13.510999999999999},{"interface":"front","average":{"1min":83.391000000000005,"5min":39.277999999999999,"15min":27.885000000000002},"min":{"1min":8.0280000000000005,"5min":6.266,"15min":6.266},"max":{"1min":550.02300000000002,"5min":550.02300000000002,"15min":550.02300000000002},"last":15.76}]},{"osd":7,"last update":"Mon Jun 3 09:06:42 2024","interfaces":[{"interface":"back","average":{"1min":44.052999999999997,"5min":33.697000000000003,"15min":29.731999999999999},"min":{"1min":9.1500000000000004,"5min":9.1500000000000004,"15min":9.1500000000000004},"max":{"1min":552.09500000000003,"5min":552.09500000000003,"15min":552.09500000000003},"last":14.996},{"interface":"front","average":{"1min":47.076999999999998,"5min":34.226999999999997,"15min":29.827000000000002},"min":{"1min":9.7780000000000005,"5min":8.6709999999999994,"15min":8.5820000000000007},"max":{"1min":531.08600000000001,"5min":531.08600000000001,"15min":531.08600000000001},"last":7.5579999999999998}]}]},{"osd":7,"up_from":20,"seq":85899346291,"num_pgs":9,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":59020,"kb_used_data":5752,"kb_used_omap":1,"kb_used_meta":53246,"kb_avail":94312820,"statfs":{"total":96636764160,"available":96576327680,"internally_reserved":0,"allocated":5890048,"data_stored":2763073,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1589,"internal_metadata":54524363},"hb_peers":[0,1,2,3,4,5,6],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":10,"apply_latency_ms":10,"commit_latency_ns":10000000,"apply_latency_ns":10000000},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Mon Jun 3 09:06:09 2024","interfaces":[{"interface":"back","average":{"1min":14.073,"5min":894.22400000000005,"15min":308.93400000000003},"min":{"1min":5.5709999999999997,"5min":4.218,"15min":4.0599999999999996},"max":{"1min":90.757999999999996,"5min":21729.001,"15min":21729.001},"last":11.026999999999999},{"interface":"front","average":{"1min":9.2550000000000008,"5min":906.23500000000001,"15min":314.06},"min":{"1min":5.5789999999999997,"5min":4.8810000000000002,"15min":4.8810000000000002},"max":{"1min":25.937000000000001,"5min":21899.732,"15min":21899.732},"last":6.4160000000000004}]},{"osd":1,"last update":"Mon Jun 3 09:06:09 2024","interfaces":[{"interface":"back","average":{"1min":15.182,"5min":69.183999999999997,"15min":219.173},"min":{"1min":7.3639999999999999,"5min":6.6200000000000001,"15min":6.1319999999999997},"max":{"1min":76.926000000000002,"5min":2795.1329999999998,"15min":18373.257000000001},"last":7.4720000000000004},{"interface":"front","average":{"1min":21.495999999999999,"5min":74.703999999999994,"15min":225.285},"min":{"1min":8.8399999999999999,"5min":8.8399999999999999,"15min":8.5129999999999999},"max":{"1min":94.977999999999994,"5min":2803.3670000000002,"15min":18374.601999999999},"last":15.045}]},{"osd":2,"last update":"Mon Jun 3 09:06:09 2024","interfaces":[{"interface":"back","average":{"1min":19.164000000000001,"5min":26.902000000000001,"15min":22.920999999999999},"min":{"1min":11.207000000000001,"5min":8.5030000000000001,"15min":8.5030000000000001},"max":{"1min":91.319999999999993,"5min":333.99200000000002,"15min":333.99200000000002},"last":10.000999999999999},{"interface":"front","average":{"1min":11.125999999999999,"5min":18.132000000000001,"15min":16.353000000000002},"min":{"1min":6.202,"5min":5.8369999999999997,"15min":5.5069999999999997},"max":{"1min":28.649000000000001,"5min":226.18100000000001,"15min":226.18100000000001},"last":8.452}]},{"osd":3,"last update":"Mon Jun 3 09:06:09 2024","interfaces":[{"interface":"back","average":{"1min":22.291,"5min":26.289999999999999,"15min":31.074999999999999},"min":{"1min":11.714,"5min":6.8959999999999999,"15min":6.3760000000000003},"max":{"1min":83.102000000000004,"5min":316.64400000000001,"15min":2039.9059999999999},"last":12.130000000000001},{"interface":"front","average":{"1min":13.324999999999999,"5min":17.963000000000001,"15min":25.969999999999999},"min":{"1min":6.1200000000000001,"5min":5.2089999999999996,"15min":5.2089999999999996},"max":{"1min":49.908999999999999,"5min":233.542,"15min":2037.5550000000001},"last":9.4380000000000006}]},{"osd":4,"last update":"Mon Jun 3 09:06:29 2024","interfaces":[{"interface":"back","average":{"1min":17.675999999999998,"5min":1026.325,"15min":472.47199999999998},"min":{"1min":7.306,"5min":5.9089999999999998,"15min":5.9089999999999998},"max":{"1min":140.03299999999999,"5min":28969.369999999999,"15min":30073.917000000001},"last":13.851000000000001},{"interface":"front","average":{"1min":17.890999999999998,"5min":1044.5139999999999,"15min":480.226},"min":{"1min":5.9279999999999999,"5min":5.9279999999999999,"15min":5.9279999999999999},"max":{"1min":141.179,"5min":29244.138999999999,"15min":30348.021000000001},"last":15.601000000000001}]},{"osd":5,"last update":"Mon Jun 3 09:06:02 2024","interfaces":[{"interface":"back","average":{"1min":24.372,"5min":26.856000000000002,"15min":23.571999999999999},"min":{"1min":12.278,"5min":9.2460000000000004,"15min":6.5650000000000004},"max":{"1min":183.64099999999999,"5min":317.572,"15min":317.572},"last":16.84},{"interface":"front","average":{"1min":21.259,"5min":23.364000000000001,"15min":21.193999999999999},"min":{"1min":7.8499999999999996,"5min":7.0999999999999996,"15min":7.008},"max":{"1min":184.53399999999999,"5min":306.60599999999999,"15min":306.60599999999999},"last":14.491}]},{"osd":6,"last update":"Mon Jun 3 09:04:31 2024","interfaces":[{"interface":"back","average":{"1min":44.061,"5min":26.821000000000002,"15min":91.176000000000002},"min":{"1min":9.0800000000000001,"5min":7.9390000000000001,"15min":7.9390000000000001},"max":{"1min":268.029,"5min":308.916,"15min":8734.4639999999999},"last":10.551},{"interface":"front","average":{"1min":46.643000000000001,"5min":23.879999999999999,"15min":88.765000000000001},"min":{"1min":8.8119999999999994,"5min":6.9240000000000004,"15min":6.742},"max":{"1min":305.64600000000002,"5min":305.64600000000002,"15min":8735.5319999999992},"last":12.933}]}]},{"osd":5,"up_from":20,"seq":85899346266,"num_pgs":9,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":59416,"kb_used_data":6148,"kb_used_omap":1,"kb_used_meta":53246,"kb_avail":94312424,"statfs":{"total":96636764160,"available":96575922176,"internally_reserved":0,"allocated":6295552,"data_stored":3160918,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1594,"internal_metadata":54524358},"hb_peers":[0,1,2,3,4,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Mon Jun 3 09:06:06 2024","interfaces":[{"interface":"back","average":{"1min":7.468,"5min":1703.037,"15min":580.51900000000001},"min":{"1min":4.6509999999999998,"5min":4.6509999999999998,"15min":4.4020000000000001},"max":{"1min":14.711,"5min":26433.764999999999,"15min":26433.764999999999},"last":16.231999999999999},{"interface":"front","average":{"1min":9.9429999999999996,"5min":1725.27,"15min":587.87800000000004},"min":{"1min":5.5999999999999996,"5min":5.1319999999999997,"15min":4.8170000000000002},"max":{"1min":13.590999999999999,"5min":26704.698,"15min":26704.698},"last":10.154}]},{"osd":1,"last update":"Mon Jun 3 09:06:00 2024","interfaces":[{"interface":"back","average":{"1min":10.862,"5min":67.731999999999999,"15min":217.364},"min":{"1min":7.1769999999999996,"5min":6.0890000000000004,"15min":5.4930000000000003},"max":{"1min":16.998999999999999,"5min":3923.4090000000001,"15min":18811.696},"last":10.711},{"interface":"front","average":{"1min":10.417,"5min":65.141999999999996,"15min":220.75700000000001},"min":{"1min":5.3209999999999997,"5min":5.3209999999999997,"15min":5.0209999999999999},"max":{"1min":15.821999999999999,"5min":4019.6129999999998,"15min":18968.365000000002},"last":14.157999999999999}]},{"osd":2,"last update":"Mon Jun 3 09:06:00 2024","interfaces":[{"interface":"back","average":{"1min":11.366,"5min":24.18,"15min":21.137},"min":{"1min":5.6369999999999996,"5min":5.6369999999999996,"15min":5.0510000000000002},"max":{"1min":15.529999999999999,"5min":332.48200000000003,"15min":332.48200000000003},"last":11.706},{"interface":"front","average":{"1min":10.089,"5min":23.117000000000001,"15min":24.050000000000001},"min":{"1min":6.6769999999999996,"5min":6.6769999999999996,"15min":5.8739999999999997},"max":{"1min":15.821,"5min":353.03199999999998,"15min":353.03199999999998},"last":6.9550000000000001}]},{"osd":3,"last update":"Mon Jun 3 09:06:00 2024","interfaces":[{"interface":"back","average":{"1min":13.472,"5min":24.283999999999999,"15min":31.666},"min":{"1min":10.111000000000001,"5min":7.4720000000000004,"15min":7.4210000000000003},"max":{"1min":16.271999999999998,"5min":353.57499999999999,"15min":2855.8519999999999},"last":12.326000000000001},{"interface":"front","average":{"1min":9.5899999999999999,"5min":15.875,"15min":25.834},"min":{"1min":6.6609999999999996,"5min":5.7729999999999997,"15min":5.1909999999999998},"max":{"1min":15.112,"5min":203.804,"15min":2869.7860000000001},"last":8.3740000000000006}]},{"osd":4,"last update":"Mon Jun 3 09:06:29 2024","interfaces":[{"interface":"back","average":{"1min":11.067,"5min":780.21000000000004,"15min":377.28399999999999},"min":{"1min":5.4779999999999998,"5min":5.4779999999999998,"15min":5.4779999999999998},"max":{"1min":14.654999999999999,"5min":23315.438999999998,"15min":28619.825000000001},"last":9.1669999999999998},{"interface":"front","average":{"1min":14.349,"5min":782.97299999999996,"15min":377.93400000000003},"min":{"1min":7.492,"5min":6.5030000000000001,"15min":6.5030000000000001},"max":{"1min":17.707000000000001,"5min":23317.502,"15min":28621.294000000002},"last":12.898999999999999}]},{"osd":6,"last update":"Mon Jun 3 09:03:57 2024","interfaces":[{"interface":"back","average":{"1min":23.536000000000001,"5min":36.457000000000001,"15min":83.799999999999997},"min":{"1min":9.4800000000000004,"5min":9.4800000000000004,"15min":8.0630000000000006},"max":{"1min":89.442999999999998,"5min":508.77999999999997,"15min":9193.5169999999998},"last":15.467000000000001},{"interface":"front","average":{"1min":18.891999999999999,"5min":22.100999999999999,"15min":75.489999999999995},"min":{"1min":5.923,"5min":5.923,"15min":5.923},"max":{"1min":78.424999999999997,"5min":464.62,"15min":9191.4429999999993},"last":13.491}]},{"osd":7,"last update":"Mon Jun 3 09:06:00 2024","interfaces":[{"interface":"back","average":{"1min":13.257,"5min":20.917999999999999,"15min":21.088999999999999},"min":{"1min":10.475,"5min":6.7629999999999999,"15min":6.7629999999999999},"max":{"1min":17.245000000000001,"5min":211.22999999999999,"15min":211.22999999999999},"last":16.960999999999999},{"interface":"front","average":{"1min":12.872999999999999,"5min":20.587,"15min":24.021000000000001},"min":{"1min":8.5760000000000005,"5min":7.7709999999999999,"15min":6.8570000000000002},"max":{"1min":18.359999999999999,"5min":223.191,"15min":286.69499999999999},"last":7.8499999999999996}]}]},{"osd":4,"up_from":20,"seq":85899346273,"num_pgs":12,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":63112,"kb_used_data":5752,"kb_used_omap":1,"kb_used_meta":57342,"kb_avail":94308728,"statfs":{"total":96636764160,"available":96572137472,"internally_reserved":0,"allocated":5890048,"data_stored":2763073,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1594,"internal_metadata":58718662},"hb_peers":[0,1,2,3,5,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":20,"apply_latency_ms":20,"commit_latency_ns":20000000,"apply_latency_ns":20000000},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Mon Jun 3 09:06:35 2024","interfaces":[{"interface":"back","average":{"1min":10.417,"5min":1093.547,"15min":381.64499999999998},"min":{"1min":4.6900000000000004,"5min":4.5279999999999996,"15min":4.3220000000000001},"max":{"1min":52.241,"5min":21726.743999999999,"15min":21726.743999999999},"last":5.0679999999999996},{"interface":"front","average":{"1min":13.175000000000001,"5min":1102.4200000000001,"15min":387.161},"min":{"1min":6.1950000000000003,"5min":5.8380000000000001,"15min":5.4909999999999997},"max":{"1min":53.148000000000003,"5min":21727.993999999999,"15min":21727.993999999999},"last":21.422000000000001}]},{"osd":1,"last update":"Mon Jun 3 09:06:35 2024","interfaces":[{"interface":"back","average":{"1min":9.9909999999999997,"5min":53.250999999999998,"15min":340.02999999999997},"min":{"1min":4.5670000000000002,"5min":4.1970000000000001,"15min":4.1970000000000001},"max":{"1min":47.073,"5min":3346.4389999999999,"15min":22256.098000000002},"last":6.5800000000000001},{"interface":"front","average":{"1min":12.189,"5min":53.375999999999998,"15min":339.68799999999999},"min":{"1min":6.1539999999999999,"5min":5.0590000000000002,"15min":4.2640000000000002},"max":{"1min":49.558,"5min":3347.181,"15min":22247.317999999999},"last":26.995000000000001}]},{"osd":2,"last update":"Mon Jun 3 09:06:35 2024","interfaces":[{"interface":"back","average":{"1min":15.178000000000001,"5min":17.202000000000002,"15min":20.417999999999999},"min":{"1min":6.3010000000000002,"5min":6.3010000000000002,"15min":6.3010000000000002},"max":{"1min":85.340000000000003,"5min":337.60399999999998,"15min":393.58600000000001},"last":25.855},{"interface":"front","average":{"1min":15.423,"5min":14.432,"15min":22.754000000000001},"min":{"1min":6.4889999999999999,"5min":6.0629999999999997,"15min":6.0629999999999997},"max":{"1min":82.120000000000005,"5min":127.913,"15min":394.471},"last":26.456}]},{"osd":3,"last update":"Mon Jun 3 09:06:35 2024","interfaces":[{"interface":"back","average":{"1min":17.651,"5min":15.5,"15min":18.079000000000001},"min":{"1min":10.359999999999999,"5min":5.8719999999999999,"15min":5.4000000000000004},"max":{"1min":84.555999999999997,"5min":127.43600000000001,"15min":322.63},"last":29.57},{"interface":"front","average":{"1min":16.413,"5min":17.356999999999999,"15min":22.920000000000002},"min":{"1min":9.7349999999999994,"5min":7.484,"15min":7.484},"max":{"1min":71.394000000000005,"5min":204.517,"15min":392.67500000000001},"last":28.472000000000001}]},{"osd":5,"last update":"Mon Jun 3 09:06:35 2024","interfaces":[{"interface":"back","average":{"1min":15.672000000000001,"5min":16.742999999999999,"15min":20.795000000000002},"min":{"1min":8.093,"5min":6.7489999999999997,"15min":6.7489999999999997},"max":{"1min":51.326000000000001,"5min":268.10199999999998,"15min":344.23399999999998},"last":30.233000000000001},{"interface":"front","average":{"1min":17.297999999999998,"5min":19.356999999999999,"15min":25.713999999999999},"min":{"1min":12.875,"5min":8.5180000000000007,"15min":8.5180000000000007},"max":{"1min":40.829000000000001,"5min":264.75,"15min":345.18700000000001},"last":31.635999999999999}]},{"osd":6,"last update":"Mon Jun 3 09:04:32 2024","interfaces":[{"interface":"back","average":{"1min":15.548,"5min":32.957999999999998,"15min":152.738},"min":{"1min":9.141,"5min":9.141,"15min":8.2319999999999993},"max":{"1min":52.57,"5min":395.238,"15min":12496.335999999999},"last":208.90000000000001},{"interface":"front","average":{"1min":12.680999999999999,"5min":25.428000000000001,"15min":149.30600000000001},"min":{"1min":6.4379999999999997,"5min":5.6319999999999997,"15min":5.6319999999999997},"max":{"1min":46.283000000000001,"5min":377.74299999999999,"15min":12616.297},"last":205.88900000000001}]},{"osd":7,"last update":"Mon Jun 3 09:06:35 2024","interfaces":[{"interface":"back","average":{"1min":9.6969999999999992,"5min":14.196,"15min":18.809999999999999},"min":{"1min":6.7190000000000003,"5min":6.2110000000000003,"15min":5.8010000000000002},"max":{"1min":15.960000000000001,"5min":207.25299999999999,"15min":358.94900000000001},"last":5.7249999999999996},{"interface":"front","average":{"1min":17.696000000000002,"5min":19.024999999999999,"15min":23.259},"min":{"1min":11.794,"5min":7.7789999999999999,"15min":7.7789999999999999},"max":{"1min":45.639000000000003,"5min":209.32900000000001,"15min":380.06900000000002},"last":30.835000000000001}]}]},{"osd":2,"up_from":20,"seq":85899346293,"num_pgs":11,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":59020,"kb_used_data":5752,"kb_used_omap":1,"kb_used_meta":53246,"kb_avail":94312820,"statfs":{"total":96636764160,"available":96576327680,"internally_reserved":0,"allocated":5890048,"data_stored":2763073,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1594,"internal_metadata":54524358},"hb_peers":[0,1,3,4,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":14,"apply_latency_ms":14,"commit_latency_ns":14000000,"apply_latency_ns":14000000},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Mon Jun 3 09:06:23 2024","interfaces":[{"interface":"back","average":{"1min":8.9629999999999992,"5min":946.25400000000002,"15min":326.36099999999999},"min":{"1min":5.3479999999999999,"5min":4.7050000000000001,"15min":4.1420000000000003},"max":{"1min":26.015000000000001,"5min":23465.18,"15min":23465.18},"last":8.6600000000000001},{"interface":"front","average":{"1min":7.6130000000000004,"5min":944.39800000000002,"15min":326.178},"min":{"1min":4.6790000000000003,"5min":4.6790000000000003,"15min":4.6790000000000003},"max":{"1min":20.802,"5min":23465.966,"15min":23465.966},"last":5.6900000000000004}]},{"osd":1,"last update":"Mon Jun 3 09:06:23 2024","interfaces":[{"interface":"back","average":{"1min":13.007,"5min":76.507000000000005,"15min":294.72899999999998},"min":{"1min":6.4089999999999998,"5min":5.4199999999999999,"15min":5.4199999999999999},"max":{"1min":85.430000000000007,"5min":4220.5290000000005,"15min":18954.987000000001},"last":7.6230000000000002},{"interface":"front","average":{"1min":15.894,"5min":82.119,"15min":302.92399999999998},"min":{"1min":7.367,"5min":7.367,"15min":7.367},"max":{"1min":98.498000000000005,"5min":4325.9489999999996,"15min":19143.834999999999},"last":12.243}]},{"osd":3,"last update":"Mon Jun 3 09:06:23 2024","interfaces":[{"interface":"back","average":{"1min":9.7579999999999991,"5min":12.444000000000001,"15min":28.065000000000001},"min":{"1min":7,"5min":5.9290000000000003,"15min":5.2450000000000001},"max":{"1min":28.164000000000001,"5min":118.913,"15min":2594.8150000000001},"last":9.3569999999999993},{"interface":"front","average":{"1min":12.321999999999999,"5min":15.25,"15min":31.349},"min":{"1min":9.1240000000000006,"5min":8.0760000000000005,"15min":6.0780000000000003},"max":{"1min":32.360999999999997,"5min":156.34700000000001,"15min":2706.5120000000002},"last":11.468999999999999}]},{"osd":4,"last update":"Mon Jun 3 09:05:57 2024","interfaces":[{"interface":"back","average":{"1min":10.573,"5min":1266.894,"15min":438.154},"min":{"1min":8.6379999999999999,"5min":7.8789999999999996,"15min":6.2039999999999997},"max":{"1min":15.105,"5min":26660.861000000001,"15min":26660.861000000001},"last":10.225},{"interface":"front","average":{"1min":12.407999999999999,"5min":1266.7460000000001,"15min":435.75400000000002},"min":{"1min":10.266999999999999,"5min":10.266999999999999,"15min":6.2720000000000002},"max":{"1min":15.228,"5min":26616.243999999999,"15min":26616.243999999999},"last":15.326000000000001}]},{"osd":5,"last update":"Mon Jun 3 09:06:23 2024","interfaces":[{"interface":"back","average":{"1min":14.103,"5min":18.091000000000001,"15min":21.119},"min":{"1min":9.6349999999999998,"5min":9.2159999999999993,"15min":6.2050000000000001},"max":{"1min":32.890000000000001,"5min":252.31100000000001,"15min":252.31100000000001},"last":23.541},{"interface":"front","average":{"1min":11.545999999999999,"5min":16.170999999999999,"15min":20.254000000000001},"min":{"1min":7.6070000000000002,"5min":7.4850000000000003,"15min":6.8810000000000002},"max":{"1min":29.228999999999999,"5min":258.31700000000001,"15min":258.31700000000001},"last":12.821999999999999}]},{"osd":6,"last update":"Mon Jun 3 09:04:28 2024","interfaces":[{"interface":"back","average":{"1min":14.063000000000001,"5min":24.850000000000001,"15min":106.755},"min":{"1min":9.0820000000000007,"5min":8.9649999999999999,"15min":8.9649999999999999},"max":{"1min":24.666,"5min":188.06700000000001,"15min":9521.0010000000002},"last":39.628999999999998},{"interface":"front","average":{"1min":13.933999999999999,"5min":21.081,"15min":106.142},"min":{"1min":5.8719999999999999,"5min":5.8719999999999999,"15min":5.8719999999999999},"max":{"1min":117.551,"5min":193.887,"15min":9585.6620000000003},"last":38.316000000000003}]},{"osd":7,"last update":"Mon Jun 3 09:06:23 2024","interfaces":[{"interface":"back","average":{"1min":17.056000000000001,"5min":20.055,"15min":23.268999999999998},"min":{"1min":10.353999999999999,"5min":6.6130000000000004,"15min":6.6130000000000004},"max":{"1min":88.974999999999994,"5min":242.56800000000001,"15min":242.56800000000001},"last":16.387},{"interface":"front","average":{"1min":18.885000000000002,"5min":21.460000000000001,"15min":25.489999999999998},"min":{"1min":12.497,"5min":9.0660000000000007,"15min":9.0660000000000007},"max":{"1min":102.229,"5min":181.56399999999999,"15min":214.33799999999999},"last":22.489000000000001}]}]},{"osd":3,"up_from":14,"seq":60129542507,"num_pgs":11,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":59016,"kb_used_data":5752,"kb_used_omap":1,"kb_used_meta":53246,"kb_avail":94312824,"statfs":{"total":96636764160,"available":96576331776,"internally_reserved":0,"allocated":5890048,"data_stored":2763073,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1592,"internal_metadata":54524360},"hb_peers":[0,1,2,4,5,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":13,"apply_latency_ms":13,"commit_latency_ns":13000000,"apply_latency_ns":13000000},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Mon Jun 3 09:06:25 2024","interfaces":[{"interface":"back","average":{"1min":7.79,"5min":1060.1959999999999,"15min":367.01799999999997},"min":{"1min":5.6909999999999998,"5min":5.173,"15min":4.6929999999999996},"max":{"1min":10.59,"5min":23275.167000000001,"15min":23275.167000000001},"last":5.4189999999999996},{"interface":"front","average":{"1min":11.215999999999999,"5min":1066.5799999999999,"15min":372.476},"min":{"1min":8.7080000000000002,"5min":7.2510000000000003,"15min":5.9880000000000004},"max":{"1min":13.499000000000001,"5min":23278.536,"15min":23278.536},"last":11.593999999999999}]},{"osd":1,"last update":"Mon Jun 3 09:06:25 2024","interfaces":[{"interface":"back","average":{"1min":7.3760000000000003,"5min":15.614000000000001,"15min":389.83100000000002},"min":{"1min":4.335,"5min":4.335,"15min":4.335},"max":{"1min":11.289,"5min":250.72300000000001,"15min":21658.205999999998},"last":6.6079999999999997},{"interface":"front","average":{"1min":8.9499999999999993,"5min":14.135,"15min":381.10300000000001},"min":{"1min":6.5609999999999999,"5min":5.4100000000000001,"15min":5.2779999999999996},"max":{"1min":14.51,"5min":180.44499999999999,"15min":21307.069},"last":12.449}]},{"osd":2,"last update":"Mon Jun 3 09:06:25 2024","interfaces":[{"interface":"back","average":{"1min":12.343999999999999,"5min":22.059999999999999,"15min":23.715},"min":{"1min":7.9199999999999999,"5min":6.4900000000000002,"15min":6.056},"max":{"1min":15.601000000000001,"5min":377.803,"15min":377.803},"last":14.433999999999999},{"interface":"front","average":{"1min":10.319000000000001,"5min":19.364999999999998,"15min":20.242999999999999},"min":{"1min":6.891,"5min":6.1749999999999998,"15min":5.9020000000000001},"max":{"1min":15.432,"5min":371.601,"15min":371.601},"last":7.6109999999999998}]},{"osd":4,"last update":"Mon Jun 3 09:06:31 2024","interfaces":[{"interface":"back","average":{"1min":9.7460000000000004,"5min":891.23000000000002,"15min":416.48099999999999},"min":{"1min":5.6539999999999999,"5min":5.6539999999999999,"15min":5.6539999999999999},"max":{"1min":15.445,"5min":29040.635999999999,"15min":30144.939999999999},"last":8.266},{"interface":"front","average":{"1min":12.398999999999999,"5min":897.58500000000004,"15min":421.27499999999998},"min":{"1min":8.9469999999999992,"5min":7.8109999999999999,"15min":6.2439999999999998},"max":{"1min":16.518000000000001,"5min":29076.955999999998,"15min":30180.689999999999},"last":13.709}]},{"osd":5,"last update":"Mon Jun 3 09:06:07 2024","interfaces":[{"interface":"back","average":{"1min":12.802,"5min":19.640999999999998,"15min":22.318000000000001},"min":{"1min":9.0890000000000004,"5min":6.8810000000000002,"15min":6.7690000000000001},"max":{"1min":16.745999999999999,"5min":249.88,"15min":327.34899999999999},"last":15.243},{"interface":"front","average":{"1min":15.002000000000001,"5min":24.003,"15min":26.649999999999999},"min":{"1min":11.984,"5min":9.9149999999999991,"15min":8.9879999999999995},"max":{"1min":18.407,"5min":251.47800000000001,"15min":331.86799999999999},"last":25.289999999999999}]},{"osd":6,"last update":"Mon Jun 3 09:04:04 2024","interfaces":[{"interface":"back","average":{"1min":22.815999999999999,"5min":31.513999999999999,"15min":181.74299999999999},"min":{"1min":9.4809999999999999,"5min":7.2240000000000002,"15min":7.2240000000000002},"max":{"1min":160.56999999999999,"5min":380.613,"15min":11893.74},"last":19.359999999999999},{"interface":"front","average":{"1min":25.359999999999999,"5min":30.981000000000002,"15min":179.81399999999999},"min":{"1min":9.8870000000000005,"5min":7.3360000000000003,"15min":6.0899999999999999},"max":{"1min":193.11099999999999,"5min":379.80700000000002,"15min":11893.069},"last":8.202}]},{"osd":7,"last update":"Mon Jun 3 09:06:25 2024","interfaces":[{"interface":"back","average":{"1min":11.321999999999999,"5min":18.443999999999999,"15min":21.568999999999999},"min":{"1min":8.109,"5min":8.109,"15min":7.4279999999999999},"max":{"1min":17.113,"5min":146.267,"15min":178.76300000000001},"last":9.8670000000000009},{"interface":"front","average":{"1min":12.512,"5min":20.065999999999999,"15min":24.190000000000001},"min":{"1min":7.4989999999999997,"5min":7.4989999999999997,"15min":7.4989999999999997},"max":{"1min":18.047999999999998,"5min":148.13800000000001,"15min":417.22800000000001},"last":10.81}]}]},{"osd":1,"up_from":14,"seq":60129542505,"num_pgs":6,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":59408,"kb_used_data":6144,"kb_used_omap":1,"kb_used_meta":53246,"kb_avail":94312432,"statfs":{"total":96636764160,"available":96575930368,"internally_reserved":0,"allocated":6291456,"data_stored":3160913,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":54524357},"hb_peers":[0,2,3,4,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":5,"apply_latency_ms":5,"commit_latency_ns":5000000,"apply_latency_ns":5000000},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Mon Jun 3 09:06:07 2024","interfaces":[{"interface":"back","average":{"1min":7.7530000000000001,"5min":185.66999999999999,"15min":77.347999999999999},"min":{"1min":5.0229999999999997,"5min":4.9279999999999999,"15min":4.5449999999999999},"max":{"1min":11.896000000000001,"5min":7470.616,"15min":7470.616},"last":16.219000000000001},{"interface":"front","average":{"1min":9.9220000000000006,"5min":187.14599999999999,"15min":76.325999999999993},"min":{"1min":5.3159999999999998,"5min":5.3159999999999998,"15min":5.3159999999999998},"max":{"1min":18.338000000000001,"5min":7477.0829999999996,"15min":7477.0829999999996},"last":16.843}]},{"osd":2,"last update":"Mon Jun 3 09:06:20 2024","interfaces":[{"interface":"back","average":{"1min":10.340999999999999,"5min":17.369,"15min":23.670999999999999},"min":{"1min":6.9740000000000002,"5min":5.8650000000000002,"15min":5.8650000000000002},"max":{"1min":15.452,"5min":228,"15min":657.38},"last":8.1440000000000001},{"interface":"front","average":{"1min":13.387,"5min":23.355,"15min":31.524000000000001},"min":{"1min":9.077,"5min":9.077,"15min":7.2160000000000002},"max":{"1min":19.701000000000001,"5min":229.465,"15min":700.06500000000005},"last":14.17}]},{"osd":3,"last update":"Mon Jun 3 09:06:04 2024","interfaces":[{"interface":"back","average":{"1min":8.6110000000000007,"5min":15.733000000000001,"15min":26.646999999999998},"min":{"1min":5.827,"5min":5.4640000000000004,"15min":5.4640000000000004},"max":{"1min":13.105,"5min":121.619,"15min":1676.009},"last":10.58},{"interface":"front","average":{"1min":9.5850000000000009,"5min":16.161999999999999,"15min":29.065999999999999},"min":{"1min":6.3630000000000004,"5min":5.7699999999999996,"15min":5.7699999999999996},"max":{"1min":15.166,"5min":132.036,"15min":1766.5840000000001},"last":7.21}]},{"osd":4,"last update":"Mon Jun 3 09:06:35 2024","interfaces":[{"interface":"back","average":{"1min":8.6400000000000006,"5min":641.85500000000002,"15min":334.71499999999997},"min":{"1min":6.7640000000000002,"5min":6.1760000000000002,"15min":6.0970000000000004},"max":{"1min":11.811,"5min":26340.968000000001,"15min":28044.862000000001},"last":8.8770000000000007},{"interface":"front","average":{"1min":12.361000000000001,"5min":648.59900000000005,"15min":338.48500000000001},"min":{"1min":9.8949999999999996,"5min":9.3940000000000001,"15min":6.4640000000000004},"max":{"1min":15.507,"5min":26342.311000000002,"15min":28045.624},"last":13.067}]},{"osd":5,"last update":"Mon Jun 3 09:06:04 2024","interfaces":[{"interface":"back","average":{"1min":12.478999999999999,"5min":20.228000000000002,"15min":25.997},"min":{"1min":8.6430000000000007,"5min":7.9720000000000004,"15min":7.234},"max":{"1min":17.442,"5min":139.43199999999999,"15min":701.22000000000003},"last":11.135},{"interface":"front","average":{"1min":13.507999999999999,"5min":25.760999999999999,"15min":27.553999999999998},"min":{"1min":10.042999999999999,"5min":7.258,"15min":6.242},"max":{"1min":20.611000000000001,"5min":278.63,"15min":598.35000000000002},"last":9.6969999999999992}]},{"osd":6,"last update":"Mon Jun 3 09:03:54 2024","interfaces":[{"interface":"back","average":{"1min":28.524000000000001,"5min":20.654,"15min":23.878},"min":{"1min":6.0309999999999997,"5min":6.0309999999999997,"15min":6.0309999999999997},"max":{"1min":167.221,"5min":167.221,"15min":704.245},"last":12.303000000000001},{"interface":"front","average":{"1min":27.606000000000002,"5min":30.131,"15min":28.628},"min":{"1min":9.8650000000000002,"5min":9.7200000000000006,"15min":8.3580000000000005},"max":{"1min":123.938,"5min":358.23000000000002,"15min":634.05499999999995},"last":12.534000000000001}]},{"osd":7,"last update":"Mon Jun 3 09:06:20 2024","interfaces":[{"interface":"back","average":{"1min":14.66,"5min":23.234999999999999,"15min":31.097000000000001},"min":{"1min":11.250999999999999,"5min":8.0559999999999992,"15min":8.0559999999999992},"max":{"1min":21.507999999999999,"5min":270.95100000000002,"15min":703.07799999999997},"last":15.563000000000001},{"interface":"front","average":{"1min":15.548,"5min":28.760000000000002,"15min":28.925000000000001},"min":{"1min":12.759,"5min":8.2089999999999996,"15min":7.7599999999999998},"max":{"1min":22.408000000000001,"5min":360.99900000000002,"15min":608.44100000000003},"last":14.904999999999999}]}]},{"osd":0,"up_from":14,"seq":60129542507,"num_pgs":10,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":81152,"kb_used_data":5752,"kb_used_omap":1,"kb_used_meta":75390,"kb_avail":94290688,"statfs":{"total":96636764160,"available":96553664512,"internally_reserved":0,"allocated":5890048,"data_stored":2763073,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1593,"internal_metadata":77199815},"hb_peers":[1,2,3,4,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":15,"apply_latency_ms":15,"commit_latency_ns":15000000,"apply_latency_ns":15000000},"alerts":[],"network_ping_times":[{"osd":1,"last update":"Mon Jun 3 09:06:04 2024","interfaces":[{"interface":"back","average":{"1min":6.9960000000000004,"5min":56.997,"15min":339.29700000000003},"min":{"1min":4.4530000000000003,"5min":4.4530000000000003,"15min":4.4100000000000001},"max":{"1min":11.720000000000001,"5min":2414.4749999999999,"15min":20691.453000000001},"last":7.8399999999999999},{"interface":"front","average":{"1min":9.548,"5min":57.258000000000003,"15min":337.791},"min":{"1min":5.2300000000000004,"5min":5.2300000000000004,"15min":4.6050000000000004},"max":{"1min":15.337999999999999,"5min":2412.9549999999999,"15min":20640.792000000001},"last":10.067}]},{"osd":2,"last update":"Mon Jun 3 09:06:22 2024","interfaces":[{"interface":"back","average":{"1min":11.154,"5min":26.515999999999998,"15min":22.882000000000001},"min":{"1min":5.7869999999999999,"5min":5.766,"15min":5.766},"max":{"1min":72.269999999999996,"5min":627.40700000000004,"15min":627.40700000000004},"last":8.4190000000000005},{"interface":"front","average":{"1min":14.353999999999999,"5min":33.304000000000002,"15min":29.885000000000002},"min":{"1min":8.2149999999999999,"5min":7.8630000000000004,"15min":7.4089999999999998},"max":{"1min":73.418000000000006,"5min":635.62599999999998,"15min":635.62599999999998},"last":14.103}]},{"osd":3,"last update":"Mon Jun 3 09:06:04 2024","interfaces":[{"interface":"back","average":{"1min":12.105,"5min":26.925000000000001,"15min":26.584},"min":{"1min":7.8410000000000002,"5min":7.1280000000000001,"15min":6.1909999999999998},"max":{"1min":18.425999999999998,"5min":389.72699999999998,"15min":389.72699999999998},"last":6.8739999999999997},{"interface":"front","average":{"1min":13.794,"5min":29.771999999999998,"15min":28.815000000000001},"min":{"1min":10.084,"5min":9.4920000000000009,"15min":6.1849999999999996},"max":{"1min":17.373999999999999,"5min":390.23200000000003,"15min":494.92899999999997},"last":15.661}]},{"osd":4,"last update":"Mon Jun 3 09:06:27 2024","interfaces":[{"interface":"back","average":{"1min":11.352,"5min":745.30600000000004,"15min":363.48099999999999},"min":{"1min":6.415,"5min":6.415,"15min":6.415},"max":{"1min":16.724,"5min":20248.616000000002,"15min":25555.212},"last":12.611000000000001},{"interface":"front","average":{"1min":13.545,"5min":744.76999999999998,"15min":361.03100000000001},"min":{"1min":9.3889999999999993,"5min":8.8940000000000001,"15min":7.1429999999999998},"max":{"1min":18.411000000000001,"5min":20250.195,"15min":25556.159},"last":17.414999999999999}]},{"osd":5,"last update":"Mon Jun 3 09:06:17 2024","interfaces":[{"interface":"back","average":{"1min":9.7040000000000006,"5min":26.533999999999999,"15min":28.683},"min":{"1min":6.8129999999999997,"5min":6.8129999999999997,"15min":6.8129999999999997},"max":{"1min":14.942,"5min":455.78699999999998,"15min":605.05899999999997},"last":10.74},{"interface":"front","average":{"1min":14.279,"5min":28.398,"15min":26.192},"min":{"1min":11.696,"5min":6.4420000000000002,"15min":6.2030000000000003},"max":{"1min":17.879000000000001,"5min":391.31200000000001,"15min":541.11000000000001},"last":11.941000000000001}]},{"osd":6,"last update":"Mon Jun 3 09:04:30 2024","interfaces":[{"interface":"back","average":{"1min":52.496000000000002,"5min":33.139000000000003,"15min":160.83199999999999},"min":{"1min":7.5289999999999999,"5min":7.1040000000000001,"15min":6.3739999999999997},"max":{"1min":457.91699999999997,"5min":457.91699999999997,"15min":11115.258},"last":16.187999999999999},{"interface":"front","average":{"1min":49.670000000000002,"5min":33.704999999999998,"15min":162.791},"min":{"1min":12.247999999999999,"5min":11.019,"15min":9.9760000000000009},"max":{"1min":407.05099999999999,"5min":407.05099999999999,"15min":11092.194},"last":14.961}]},{"osd":7,"last update":"Mon Jun 3 09:06:22 2024","interfaces":[{"interface":"back","average":{"1min":11.625999999999999,"5min":26.375,"15min":28.972999999999999},"min":{"1min":6.4480000000000004,"5min":6.4480000000000004,"15min":6.4480000000000004},"max":{"1min":15.352,"5min":461.41699999999997,"15min":598.33699999999999},"last":16.821999999999999},{"interface":"front","average":{"1min":14.013,"5min":29.696000000000002,"15min":33.630000000000003},"min":{"1min":10.938000000000001,"5min":10.58,"15min":10.263999999999999},"max":{"1min":16.603000000000002,"5min":463.10599999999999,"15min":592.80499999999995},"last":18.053999999999998}]}]}],"pool_statfs":[{"poolid":1,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":401408,"data_stored":397840,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":5,"total":0,"available":0,"internally_reserved":0,"allocated":401408,"data_stored":397840,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":4,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":5,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":6,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":17,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":17,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":17,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":17,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":17,"osd":4,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":17,"osd":5,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":5,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":17,"osd":6,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":5,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":17,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0}]}} 2024-06-03T09:06:53.734 INFO:tasks.ceph.ceph_manager.ceph:clean! 2024-06-03T09:06:53.734 DEBUG:teuthology.orchestra.run.smithi088:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph pg dump --format=json 2024-06-03T09:06:53.834 INFO:teuthology.orchestra.run.smithi088.stderr:2024-06-03T09:06:53.834+0000 7fc80e1e7640 10 monclient: get_monmap_and_config 2024-06-03T09:06:53.835 INFO:teuthology.orchestra.run.smithi088.stderr:2024-06-03T09:06:53.834+0000 7fc80e1e7640 10 monclient: build_initial_monmap 2024-06-03T09:06:53.835 INFO:teuthology.orchestra.run.smithi088.stderr:2024-06-03T09:06:53.834+0000 7fc80e1e7640 10 monclient: monmap: 2024-06-03T09:06:53.835 INFO:teuthology.orchestra.run.smithi088.stderr:epoch 0 2024-06-03T09:06:53.835 INFO:teuthology.orchestra.run.smithi088.stderr:fsid ec3c3fdd-3ac6-4d7d-9412-683130ed8b04 2024-06-03T09:06:53.835 INFO:teuthology.orchestra.run.smithi088.stderr:last_changed 2024-06-03T09:06:53.835929+0000 2024-06-03T09:06:53.835 INFO:teuthology.orchestra.run.smithi088.stderr:created 2024-06-03T09:06:53.835929+0000 2024-06-03T09:06:53.835 INFO:teuthology.orchestra.run.smithi088.stderr:min_mon_release 0 (unknown) 2024-06-03T09:06:53.835 INFO:teuthology.orchestra.run.smithi088.stderr:election_strategy: 3 2024-06-03T09:06:53.835 INFO:teuthology.orchestra.run.smithi088.stderr:0: v1:172.21.15.88:6789/0 mon.noname-a 2024-06-03T09:06:53.835 INFO:teuthology.orchestra.run.smithi088.stderr:1: v1:172.21.15.192:6789/0 mon.noname-c 2024-06-03T09:06:53.835 INFO:teuthology.orchestra.run.smithi088.stderr:2: v1:172.21.15.88:6790/0 mon.noname-b 2024-06-03T09:06:53.836 INFO:teuthology.orchestra.run.smithi088.stderr: 2024-06-03T09:06:53.837 INFO:teuthology.orchestra.run.smithi088.stderr:2024-06-03T09:06:53.835+0000 7fc80e1e7640 1 Processor -- start 2024-06-03T09:06:53.837 INFO:teuthology.orchestra.run.smithi088.stderr:2024-06-03T09:06:53.835+0000 7fc80e1e7640 1 -- start start 2024-06-03T09:06:53.837 INFO:teuthology.orchestra.run.smithi088.stderr:2024-06-03T09:06:53.835+0000 7fc80e1e7640 10 monclient: init 2024-06-03T09:06:53.837 INFO:teuthology.orchestra.run.smithi088.stderr:2024-06-03T09:06:53.836+0000 7fc80e1e7640 10 monclient: _reopen_session rank -1 2024-06-03T09:06:53.837 INFO:teuthology.orchestra.run.smithi088.stderr:2024-06-03T09:06:53.836+0000 7fc80e1e7640 10 monclient: _add_conns ranks=[0,1,2] 2024-06-03T09:06:53.837 INFO:teuthology.orchestra.run.smithi088.stderr:2024-06-03T09:06:53.836+0000 7fc80e1e7640 10 monclient(hunting): picked mon.noname-a con 0x7fc80806ba10 addr v1:172.21.15.88:6789/0 2024-06-03T09:06:53.837 INFO:teuthology.orchestra.run.smithi088.stderr:2024-06-03T09:06:53.836+0000 7fc80e1e7640 10 monclient(hunting): picked mon.noname-c con 0x7fc80806bdf0 addr v1:172.21.15.192:6789/0 2024-06-03T09:06:53.837 INFO:teuthology.orchestra.run.smithi088.stderr:2024-06-03T09:06:53.836+0000 7fc80e1e7640 10 monclient(hunting): picked mon.noname-b con 0x7fc80813eee0 addr v1:172.21.15.88:6790/0 2024-06-03T09:06:53.837 INFO:teuthology.orchestra.run.smithi088.stderr:2024-06-03T09:06:53.836+0000 7fc80e1e7640 1 -- --> v1:172.21.15.88:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fc8080ad400 con 0x7fc80806ba10 2024-06-03T09:06:53.837 INFO:teuthology.orchestra.run.smithi088.stderr:2024-06-03T09:06:53.836+0000 7fc80e1e7640 1 -- --> v1:172.21.15.192:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fc808142530 con 0x7fc80806bdf0 2024-06-03T09:06:53.838 INFO:teuthology.orchestra.run.smithi088.stderr:2024-06-03T09:06:53.836+0000 7fc80e1e7640 1 -- --> v1:172.21.15.88:6790/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fc8080ad630 con 0x7fc80813eee0 2024-06-03T09:06:53.838 INFO:teuthology.orchestra.run.smithi088.stderr:2024-06-03T09:06:53.836+0000 7fc80e1e7640 10 monclient(hunting): _renew_subs 2024-06-03T09:06:53.838 INFO:teuthology.orchestra.run.smithi088.stderr:2024-06-03T09:06:53.836+0000 7fc80e1e7640 10 monclient(hunting): authenticate will time out at 3069.646301s 2024-06-03T09:06:53.838 INFO:teuthology.orchestra.run.smithi088.stderr:2024-06-03T09:06:53.838+0000 7fc807fff640 1 --1- >> v1:172.21.15.88:6790/0 conn(0x7fc80813eee0 0x7fc80814ce60 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.88:6790/0 says I am v1:172.21.15.88:52386/0 (socket says 172.21.15.88:52386) 2024-06-03T09:06:53.838 INFO:teuthology.orchestra.run.smithi088.stderr:2024-06-03T09:06:53.838+0000 7fc807fff640 1 -- 172.21.15.88:0/533618875 learned_addr learned my addr 172.21.15.88:0/533618875 (peer_addr_for_me v1:172.21.15.88:0/0) 2024-06-03T09:06:53.855 INFO:teuthology.orchestra.run.smithi088.stderr:2024-06-03T09:06:53.855+0000 7fc8067fc640 1 -- 172.21.15.88:0/533618875 <== mon.2 v1:172.21.15.88:6790/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 2637229227 0 0) 0x7fc8080ad630 con 0x7fc80813eee0 2024-06-03T09:06:53.855 INFO:teuthology.orchestra.run.smithi088.stderr:2024-06-03T09:06:53.855+0000 7fc8067fc640 10 monclient(hunting): _negotiate 2024-06-03T09:06:53.855 INFO:teuthology.orchestra.run.smithi088.stderr:2024-06-03T09:06:53.855+0000 7fc8067fc640 10 monclient(hunting): _init_auth method 2 2024-06-03T09:06:53.855 INFO:teuthology.orchestra.run.smithi088.stderr:2024-06-03T09:06:53.855+0000 7fc8067fc640 10 monclient(hunting): _init_auth creating new auth 2024-06-03T09:06:53.855 INFO:teuthology.orchestra.run.smithi088.stderr:2024-06-03T09:06:53.855+0000 7fc8067fc640 10 monclient(hunting): my global_id is 30230 2024-06-03T09:06:53.855 INFO:teuthology.orchestra.run.smithi088.stderr:2024-06-03T09:06:53.855+0000 7fc8067fc640 1 -- 172.21.15.88:0/533618875 --> v1:172.21.15.88:6790/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7fc7f0003520 con 0x7fc80813eee0 2024-06-03T09:06:53.869 INFO:teuthology.orchestra.run.smithi088.stderr:2024-06-03T09:06:53.869+0000 7fc8067fc640 1 -- 172.21.15.88:0/533618875 <== mon.2 v1:172.21.15.88:6790/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 214+0+0 (unknown 3960078359 0 0) 0x7fc7f0003520 con 0x7fc80813eee0 2024-06-03T09:06:53.869 INFO:teuthology.orchestra.run.smithi088.stderr:2024-06-03T09:06:53.869+0000 7fc8067fc640 1 -- 172.21.15.88:0/533618875 >> v1:172.21.15.192:6789/0 conn(0x7fc80806bdf0 legacy=0x7fc80813e7d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-06-03T09:06:53.869 INFO:teuthology.orchestra.run.smithi088.stderr:2024-06-03T09:06:53.869+0000 7fc8067fc640 1 -- 172.21.15.88:0/533618875 >> v1:172.21.15.88:6789/0 conn(0x7fc80806ba10 legacy=0x7fc808141e20 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-06-03T09:06:53.869 INFO:teuthology.orchestra.run.smithi088.stderr:2024-06-03T09:06:53.869+0000 7fc8067fc640 10 monclient: _finish_hunting 0 2024-06-03T09:06:53.869 INFO:teuthology.orchestra.run.smithi088.stderr:2024-06-03T09:06:53.869+0000 7fc8067fc640 1 monclient: found mon.noname-b 2024-06-03T09:06:53.869 INFO:teuthology.orchestra.run.smithi088.stderr:2024-06-03T09:06:53.869+0000 7fc8067fc640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-06-03T09:06:53.869 INFO:teuthology.orchestra.run.smithi088.stderr:2024-06-03T09:06:53.869+0000 7fc8067fc640 10 monclient: _send_mon_message to mon.noname-b at v1:172.21.15.88:6790/0 2024-06-03T09:06:53.870 INFO:teuthology.orchestra.run.smithi088.stderr:2024-06-03T09:06:53.869+0000 7fc8067fc640 1 -- 172.21.15.88:0/533618875 --> v1:172.21.15.88:6790/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fc80806c840 con 0x7fc80813eee0 2024-06-03T09:06:53.870 INFO:teuthology.orchestra.run.smithi088.stderr:2024-06-03T09:06:53.869+0000 7fc8067fc640 10 monclient: _finish_auth 0 2024-06-03T09:06:53.870 INFO:teuthology.orchestra.run.smithi088.stderr:2024-06-03T09:06:53.869+0000 7fc8067fc640 10 monclient: _check_auth_tickets 2024-06-03T09:06:53.870 INFO:teuthology.orchestra.run.smithi088.stderr:2024-06-03T09:06:53.869+0000 7fc8067fc640 20 monclient: _check_auth_rotating not needed by client.admin 2024-06-03T09:06:53.870 INFO:teuthology.orchestra.run.smithi088.stderr:2024-06-03T09:06:53.869+0000 7fc8067fc640 1 -- 172.21.15.88:0/533618875 <== mon.2 v1:172.21.15.88:6790/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 2234374065 0 0) 0x7fc800001990 con 0x7fc80813eee0 2024-06-03T09:06:53.870 INFO:teuthology.orchestra.run.smithi088.stderr:2024-06-03T09:06:53.869+0000 7fc8067fc640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-06-03T09:06:53.870 INFO:teuthology.orchestra.run.smithi088.stderr:2024-06-03T09:06:53.869+0000 7fc8067fc640 10 monclient: got monmap 1 from mon.noname-b (according to old e1) 2024-06-03T09:06:53.870 INFO:teuthology.orchestra.run.smithi088.stderr:2024-06-03T09:06:53.869+0000 7fc8067fc640 10 monclient: dump: 2024-06-03T09:06:53.870 INFO:teuthology.orchestra.run.smithi088.stderr:epoch 1 2024-06-03T09:06:53.870 INFO:teuthology.orchestra.run.smithi088.stderr:fsid ec3c3fdd-3ac6-4d7d-9412-683130ed8b04 2024-06-03T09:06:53.870 INFO:teuthology.orchestra.run.smithi088.stderr:last_changed 2024-06-03T08:34:31.885125+0000 2024-06-03T09:06:53.871 INFO:teuthology.orchestra.run.smithi088.stderr:created 2024-06-03T08:34:31.885125+0000 2024-06-03T09:06:53.871 INFO:teuthology.orchestra.run.smithi088.stderr:min_mon_release 18 (reef) 2024-06-03T09:06:53.871 INFO:teuthology.orchestra.run.smithi088.stderr:election_strategy: 3 2024-06-03T09:06:53.871 INFO:teuthology.orchestra.run.smithi088.stderr:0: v1:172.21.15.88:6789/0 mon.a 2024-06-03T09:06:53.871 INFO:teuthology.orchestra.run.smithi088.stderr:1: v1:172.21.15.192:6789/0 mon.b 2024-06-03T09:06:53.871 INFO:teuthology.orchestra.run.smithi088.stderr:2: v1:172.21.15.88:6790/0 mon.c 2024-06-03T09:06:53.871 INFO:teuthology.orchestra.run.smithi088.stderr: 2024-06-03T09:06:53.871 INFO:teuthology.orchestra.run.smithi088.stderr:2024-06-03T09:06:53.869+0000 7fc80e1e7640 5 monclient: authenticate success, global_id 30230 2024-06-03T09:06:53.871 INFO:teuthology.orchestra.run.smithi088.stderr:2024-06-03T09:06:53.869+0000 7fc80e1e7640 20 monclient: get_monmap_and_config waiting for monmap|config 2024-06-03T09:06:53.885 INFO:teuthology.orchestra.run.smithi088.stderr:2024-06-03T09:06:53.885+0000 7fc8067fc640 1 -- 172.21.15.88:0/533618875 <== mon.2 v1:172.21.15.88:6790/0 4 ==== config(0 keys) v1 ==== 4+0+0 (unknown 0 0 0) 0x7fc800001bb0 con 0x7fc80813eee0 2024-06-03T09:06:53.885 INFO:teuthology.orchestra.run.smithi088.stderr:2024-06-03T09:06:53.885+0000 7fc8067fc640 10 monclient: handle_config config(0 keys) v1 2024-06-03T09:06:53.885 INFO:teuthology.orchestra.run.smithi088.stderr:2024-06-03T09:06:53.885+0000 7fc8067fc640 1 -- 172.21.15.88:0/533618875 <== mon.2 v1:172.21.15.88:6790/0 5 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 2234374065 0 0) 0x7fc800001f40 con 0x7fc80813eee0 2024-06-03T09:06:53.886 INFO:teuthology.orchestra.run.smithi088.stderr:2024-06-03T09:06:53.885+0000 7fc80e1e7640 10 monclient: get_monmap_and_config success 2024-06-03T09:06:53.886 INFO:teuthology.orchestra.run.smithi088.stderr:2024-06-03T09:06:53.885+0000 7fc80e1e7640 4 set_mon_vals no callback set 2024-06-03T09:06:53.886 INFO:teuthology.orchestra.run.smithi088.stderr:2024-06-03T09:06:53.885+0000 7fc8067fc640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-06-03T09:06:53.886 INFO:teuthology.orchestra.run.smithi088.stderr:2024-06-03T09:06:53.885+0000 7fc8067fc640 10 monclient: got monmap 1 from mon.c (according to old e1) 2024-06-03T09:06:53.886 INFO:teuthology.orchestra.run.smithi088.stderr:2024-06-03T09:06:53.885+0000 7fc8067fc640 10 monclient: dump: 2024-06-03T09:06:53.886 INFO:teuthology.orchestra.run.smithi088.stderr:epoch 1 2024-06-03T09:06:53.886 INFO:teuthology.orchestra.run.smithi088.stderr:fsid ec3c3fdd-3ac6-4d7d-9412-683130ed8b04 2024-06-03T09:06:53.886 INFO:teuthology.orchestra.run.smithi088.stderr:last_changed 2024-06-03T08:34:31.885125+0000 2024-06-03T09:06:53.886 INFO:teuthology.orchestra.run.smithi088.stderr:created 2024-06-03T08:34:31.885125+0000 2024-06-03T09:06:53.886 INFO:teuthology.orchestra.run.smithi088.stderr:min_mon_release 18 (reef) 2024-06-03T09:06:53.886 INFO:teuthology.orchestra.run.smithi088.stderr:election_strategy: 3 2024-06-03T09:06:53.886 INFO:teuthology.orchestra.run.smithi088.stderr:0: v1:172.21.15.88:6789/0 mon.a 2024-06-03T09:06:53.887 INFO:teuthology.orchestra.run.smithi088.stderr:1: v1:172.21.15.192:6789/0 mon.b 2024-06-03T09:06:53.887 INFO:teuthology.orchestra.run.smithi088.stderr:2: v1:172.21.15.88:6790/0 mon.c 2024-06-03T09:06:53.887 INFO:teuthology.orchestra.run.smithi088.stderr: 2024-06-03T09:06:53.889 INFO:teuthology.orchestra.run.smithi088.stderr:2024-06-03T09:06:53.886+0000 7fc80e1e7640 10 monclient: shutdown 2024-06-03T09:06:53.889 INFO:teuthology.orchestra.run.smithi088.stderr:2024-06-03T09:06:53.886+0000 7fc80e1e7640 20 monclient: shutdown discarding 0 pending message(s) 2024-06-03T09:06:53.889 INFO:teuthology.orchestra.run.smithi088.stderr:2024-06-03T09:06:53.886+0000 7fc80e1e7640 1 -- 172.21.15.88:0/533618875 >> v1:172.21.15.88:6790/0 conn(0x7fc80813eee0 legacy=0x7fc80814ce60 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-06-03T09:06:53.889 INFO:teuthology.orchestra.run.smithi088.stderr:2024-06-03T09:06:53.886+0000 7fc80e1e7640 1 -- 172.21.15.88:0/533618875 shutdown_connections 2024-06-03T09:06:53.889 INFO:teuthology.orchestra.run.smithi088.stderr:2024-06-03T09:06:53.886+0000 7fc80e1e7640 1 -- 172.21.15.88:0/533618875 >> 172.21.15.88:0/533618875 conn(0x7fc8080b4fb0 msgr2=0x7fc8080b53c0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-06-03T09:06:53.889 INFO:teuthology.orchestra.run.smithi088.stderr:2024-06-03T09:06:53.886+0000 7fc80e1e7640 1 -- 172.21.15.88:0/533618875 shutdown_connections 2024-06-03T09:06:53.889 INFO:teuthology.orchestra.run.smithi088.stderr:2024-06-03T09:06:53.886+0000 7fc80e1e7640 1 -- 172.21.15.88:0/533618875 wait complete. 2024-06-03T09:06:53.889 INFO:teuthology.orchestra.run.smithi088.stderr:2024-06-03T09:06:53.886+0000 7fc80e1e7640 10 monclient: build_initial_monmap 2024-06-03T09:06:53.889 INFO:teuthology.orchestra.run.smithi088.stderr:2024-06-03T09:06:53.886+0000 7fc80e1e7640 10 monclient: monmap: 2024-06-03T09:06:53.890 INFO:teuthology.orchestra.run.smithi088.stderr:epoch 0 2024-06-03T09:06:53.890 INFO:teuthology.orchestra.run.smithi088.stderr:fsid 00000000-0000-0000-0000-000000000000 2024-06-03T09:06:53.890 INFO:teuthology.orchestra.run.smithi088.stderr:last_changed 0.000000 2024-06-03T09:06:53.890 INFO:teuthology.orchestra.run.smithi088.stderr:created 0.000000 2024-06-03T09:06:53.890 INFO:teuthology.orchestra.run.smithi088.stderr:min_mon_release 0 (unknown) 2024-06-03T09:06:53.890 INFO:teuthology.orchestra.run.smithi088.stderr:election_strategy: 1 2024-06-03T09:06:53.890 INFO:teuthology.orchestra.run.smithi088.stderr:0: v1:172.21.15.88:6789/0 mon.noname-a 2024-06-03T09:06:53.890 INFO:teuthology.orchestra.run.smithi088.stderr:1: v1:172.21.15.192:6789/0 mon.noname-b 2024-06-03T09:06:53.890 INFO:teuthology.orchestra.run.smithi088.stderr:2: v1:172.21.15.88:6790/0 mon.noname-c 2024-06-03T09:06:53.890 INFO:teuthology.orchestra.run.smithi088.stderr: 2024-06-03T09:06:53.890 INFO:teuthology.orchestra.run.smithi088.stderr:2024-06-03T09:06:53.887+0000 7fc80e1e7640 1 librados: starting msgr at 2024-06-03T09:06:53.890 INFO:teuthology.orchestra.run.smithi088.stderr:2024-06-03T09:06:53.887+0000 7fc80e1e7640 1 librados: starting objecter 2024-06-03T09:06:53.890 INFO:teuthology.orchestra.run.smithi088.stderr:2024-06-03T09:06:53.887+0000 7fc80e1e7640 1 Processor -- start 2024-06-03T09:06:53.891 INFO:teuthology.orchestra.run.smithi088.stderr:2024-06-03T09:06:53.887+0000 7fc80e1e7640 1 -- start start 2024-06-03T09:06:53.891 INFO:teuthology.orchestra.run.smithi088.stderr:2024-06-03T09:06:53.887+0000 7fc80e1e7640 1 librados: setting wanted keys 2024-06-03T09:06:53.891 INFO:teuthology.orchestra.run.smithi088.stderr:2024-06-03T09:06:53.887+0000 7fc80e1e7640 1 librados: calling monclient init 2024-06-03T09:06:53.891 INFO:teuthology.orchestra.run.smithi088.stderr:2024-06-03T09:06:53.887+0000 7fc80e1e7640 10 monclient: init 2024-06-03T09:06:53.891 INFO:teuthology.orchestra.run.smithi088.stderr:2024-06-03T09:06:53.888+0000 7fc80e1e7640 10 monclient: _reopen_session rank -1 2024-06-03T09:06:53.891 INFO:teuthology.orchestra.run.smithi088.stderr:2024-06-03T09:06:53.888+0000 7fc80e1e7640 10 monclient: _add_conns ranks=[0,2,1] 2024-06-03T09:06:53.891 INFO:teuthology.orchestra.run.smithi088.stderr:2024-06-03T09:06:53.888+0000 7fc80e1e7640 10 monclient(hunting): picked mon.noname-a con 0x7fc80806ba10 addr v1:172.21.15.88:6789/0 2024-06-03T09:06:53.891 INFO:teuthology.orchestra.run.smithi088.stderr:2024-06-03T09:06:53.888+0000 7fc80e1e7640 10 monclient(hunting): picked mon.noname-c con 0x7fc80806bdf0 addr v1:172.21.15.88:6790/0 2024-06-03T09:06:53.891 INFO:teuthology.orchestra.run.smithi088.stderr:2024-06-03T09:06:53.888+0000 7fc80e1e7640 10 monclient(hunting): picked mon.noname-b con 0x7fc80813eee0 addr v1:172.21.15.192:6789/0 2024-06-03T09:06:53.891 INFO:teuthology.orchestra.run.smithi088.stderr:2024-06-03T09:06:53.888+0000 7fc80e1e7640 1 -- --> v1:172.21.15.88:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fc8081da310 con 0x7fc80806ba10 2024-06-03T09:06:53.891 INFO:teuthology.orchestra.run.smithi088.stderr:2024-06-03T09:06:53.888+0000 7fc80e1e7640 1 -- --> v1:172.21.15.192:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fc8080b49d0 con 0x7fc80813eee0 2024-06-03T09:06:53.892 INFO:teuthology.orchestra.run.smithi088.stderr:2024-06-03T09:06:53.888+0000 7fc80e1e7640 1 -- --> v1:172.21.15.88:6790/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fc8081d9ec0 con 0x7fc80806bdf0 2024-06-03T09:06:53.892 INFO:teuthology.orchestra.run.smithi088.stderr:2024-06-03T09:06:53.888+0000 7fc80e1e7640 10 monclient(hunting): _renew_subs 2024-06-03T09:06:53.892 INFO:teuthology.orchestra.run.smithi088.stderr:2024-06-03T09:06:53.888+0000 7fc80e1e7640 10 monclient(hunting): authenticate will time out at 3069.698095s 2024-06-03T09:06:53.892 INFO:teuthology.orchestra.run.smithi088.stderr:2024-06-03T09:06:53.892+0000 7fc8077fe640 1 --1- >> v1:172.21.15.88:6789/0 conn(0x7fc80806ba10 0x7fc808070a90 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=1).handle_server_banner_and_identify peer v1:172.21.15.88:6789/0 says I am v1:172.21.15.88:43338/0 (socket says 172.21.15.88:43338) 2024-06-03T09:06:53.892 INFO:teuthology.orchestra.run.smithi088.stderr:2024-06-03T09:06:53.892+0000 7fc8077fe640 1 -- 172.21.15.88:0/420261594 learned_addr learned my addr 172.21.15.88:0/420261594 (peer_addr_for_me v1:172.21.15.88:0/0) 2024-06-03T09:06:53.894 INFO:teuthology.orchestra.run.smithi088.stderr:2024-06-03T09:06:53.894+0000 7fc7effff640 10 client.?.objecter ms_handle_connect 0x7fc80806ba10 2024-06-03T09:06:53.894 INFO:teuthology.orchestra.run.smithi088.stderr:2024-06-03T09:06:53.894+0000 7fc7effff640 10 client.?.objecter resend_mon_ops 2024-06-03T09:06:53.899 INFO:teuthology.orchestra.run.smithi088.stderr:2024-06-03T09:06:53.900+0000 7fc7effff640 10 client.?.objecter ms_handle_connect 0x7fc80806bdf0 2024-06-03T09:06:53.900 INFO:teuthology.orchestra.run.smithi088.stderr:2024-06-03T09:06:53.900+0000 7fc7effff640 10 client.?.objecter resend_mon_ops 2024-06-03T09:06:53.918 INFO:teuthology.orchestra.run.smithi088.stderr:2024-06-03T09:06:53.918+0000 7fc7effff640 1 -- 172.21.15.88:0/420261594 <== mon.0 v1:172.21.15.88:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 3373734499 0 0) 0x7fc8081da310 con 0x7fc80806ba10 2024-06-03T09:06:53.918 INFO:teuthology.orchestra.run.smithi088.stderr:2024-06-03T09:06:53.918+0000 7fc7effff640 10 monclient(hunting): _negotiate 2024-06-03T09:06:53.918 INFO:teuthology.orchestra.run.smithi088.stderr:2024-06-03T09:06:53.918+0000 7fc7effff640 10 monclient(hunting): _init_auth method 2 2024-06-03T09:06:53.918 INFO:teuthology.orchestra.run.smithi088.stderr:2024-06-03T09:06:53.918+0000 7fc7effff640 10 monclient(hunting): _init_auth creating new auth 2024-06-03T09:06:53.918 INFO:teuthology.orchestra.run.smithi088.stderr:2024-06-03T09:06:53.918+0000 7fc7effff640 10 monclient(hunting): my global_id is 30729 2024-06-03T09:06:53.918 INFO:teuthology.orchestra.run.smithi088.stderr:2024-06-03T09:06:53.918+0000 7fc7effff640 1 -- 172.21.15.88:0/420261594 --> v1:172.21.15.88:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7fc7f4004eb0 con 0x7fc80806ba10 2024-06-03T09:06:53.932 INFO:teuthology.orchestra.run.smithi088.stderr:2024-06-03T09:06:53.932+0000 7fc7effff640 1 -- 172.21.15.88:0/420261594 <== mon.2 v1:172.21.15.88:6790/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 2955361439 0 0) 0x7fc8081d9ec0 con 0x7fc80806bdf0 2024-06-03T09:06:53.932 INFO:teuthology.orchestra.run.smithi088.stderr:2024-06-03T09:06:53.932+0000 7fc7effff640 10 monclient(hunting): _negotiate 2024-06-03T09:06:53.932 INFO:teuthology.orchestra.run.smithi088.stderr:2024-06-03T09:06:53.932+0000 7fc7effff640 10 monclient(hunting): _init_auth method 2 2024-06-03T09:06:53.932 INFO:teuthology.orchestra.run.smithi088.stderr:2024-06-03T09:06:53.932+0000 7fc7effff640 10 monclient(hunting): _init_auth creating new auth 2024-06-03T09:06:53.932 INFO:teuthology.orchestra.run.smithi088.stderr:2024-06-03T09:06:53.932+0000 7fc7effff640 10 monclient(hunting): my global_id is 30233 2024-06-03T09:06:53.932 INFO:teuthology.orchestra.run.smithi088.stderr:2024-06-03T09:06:53.932+0000 7fc7effff640 1 -- 172.21.15.88:0/420261594 --> v1:172.21.15.88:6790/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7fc8081da310 con 0x7fc80806bdf0 2024-06-03T09:06:53.940 INFO:teuthology.orchestra.run.smithi088.stderr:2024-06-03T09:06:53.940+0000 7fc7effff640 1 -- 172.21.15.88:0/420261594 <== mon.0 v1:172.21.15.88:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 764+0+0 (unknown 3060385517 0 0) 0x7fc7f4004eb0 con 0x7fc80806ba10 2024-06-03T09:06:53.940 INFO:teuthology.orchestra.run.smithi088.stderr:2024-06-03T09:06:53.940+0000 7fc7effff640 1 -- 172.21.15.88:0/420261594 --> v1:172.21.15.88:6789/0 -- auth(proto 2 165 bytes epoch 0) v1 -- 0x7fc8081d9ec0 con 0x7fc80806ba10 2024-06-03T09:06:53.940 INFO:teuthology.orchestra.run.smithi088.stderr:2024-06-03T09:06:53.940+0000 7fc7effff640 1 -- 172.21.15.88:0/420261594 <== mon.0 v1:172.21.15.88:6789/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 2234374065 0 0) 0x7fc7f8001970 con 0x7fc80806ba10 2024-06-03T09:06:53.940 INFO:teuthology.orchestra.run.smithi088.stderr:2024-06-03T09:06:53.940+0000 7fc7effff640 10 monclient(hunting): handle_monmap mon_map magic: 0 v1 2024-06-03T09:06:53.941 INFO:teuthology.orchestra.run.smithi088.stderr:2024-06-03T09:06:53.940+0000 7fc7effff640 10 monclient(hunting): got monmap 1 from mon.noname-a (according to old e1) 2024-06-03T09:06:53.941 INFO:teuthology.orchestra.run.smithi088.stderr:2024-06-03T09:06:53.940+0000 7fc7effff640 10 monclient(hunting): dump: 2024-06-03T09:06:53.941 INFO:teuthology.orchestra.run.smithi088.stderr:epoch 1 2024-06-03T09:06:53.941 INFO:teuthology.orchestra.run.smithi088.stderr:fsid ec3c3fdd-3ac6-4d7d-9412-683130ed8b04 2024-06-03T09:06:53.941 INFO:teuthology.orchestra.run.smithi088.stderr:last_changed 2024-06-03T08:34:31.885125+0000 2024-06-03T09:06:53.941 INFO:teuthology.orchestra.run.smithi088.stderr:created 2024-06-03T08:34:31.885125+0000 2024-06-03T09:06:53.941 INFO:teuthology.orchestra.run.smithi088.stderr:min_mon_release 18 (reef) 2024-06-03T09:06:53.941 INFO:teuthology.orchestra.run.smithi088.stderr:election_strategy: 3 2024-06-03T09:06:53.941 INFO:teuthology.orchestra.run.smithi088.stderr:0: v1:172.21.15.88:6789/0 mon.a 2024-06-03T09:06:53.941 INFO:teuthology.orchestra.run.smithi088.stderr:1: v1:172.21.15.192:6789/0 mon.b 2024-06-03T09:06:53.941 INFO:teuthology.orchestra.run.smithi088.stderr:2: v1:172.21.15.88:6790/0 mon.c 2024-06-03T09:06:53.941 INFO:teuthology.orchestra.run.smithi088.stderr: 2024-06-03T09:06:53.941 INFO:teuthology.orchestra.run.smithi088.stderr:2024-06-03T09:06:53.940+0000 7fc7effff640 10 monclient(hunting): _finish_auth 0 2024-06-03T09:06:53.978 INFO:teuthology.orchestra.run.smithi088.stderr:2024-06-03T09:06:53.977+0000 7fc7effff640 1 -- 172.21.15.88:0/420261594 <== mon.0 v1:172.21.15.88:6789/0 4 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 580+0+0 (unknown 3726466457 0 0) 0x7fc8081d9ec0 con 0x7fc80806ba10 2024-06-03T09:06:53.978 INFO:teuthology.orchestra.run.smithi088.stderr:2024-06-03T09:06:53.977+0000 7fc7effff640 1 -- 172.21.15.88:0/420261594 >> v1:172.21.15.88:6790/0 conn(0x7fc80806bdf0 legacy=0x7fc8081da830 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-06-03T09:06:53.978 INFO:teuthology.orchestra.run.smithi088.stderr:2024-06-03T09:06:53.977+0000 7fc7effff640 1 -- 172.21.15.88:0/420261594 >> v1:172.21.15.192:6789/0 conn(0x7fc80813eee0 legacy=0x7fc8081360e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-06-03T09:06:53.978 INFO:teuthology.orchestra.run.smithi088.stderr:2024-06-03T09:06:53.977+0000 7fc7effff640 10 monclient: _finish_hunting 0 2024-06-03T09:06:53.978 INFO:teuthology.orchestra.run.smithi088.stderr:2024-06-03T09:06:53.977+0000 7fc7effff640 1 monclient: found mon.a 2024-06-03T09:06:53.979 INFO:teuthology.orchestra.run.smithi088.stderr:2024-06-03T09:06:53.977+0000 7fc7effff640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-06-03T09:06:53.979 INFO:teuthology.orchestra.run.smithi088.stderr:2024-06-03T09:06:53.977+0000 7fc7effff640 10 monclient: _send_mon_message to mon.a at v1:172.21.15.88:6789/0 2024-06-03T09:06:53.979 INFO:teuthology.orchestra.run.smithi088.stderr:2024-06-03T09:06:53.977+0000 7fc7effff640 1 -- 172.21.15.88:0/420261594 --> v1:172.21.15.88:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fc8081da050 con 0x7fc80806ba10 2024-06-03T09:06:53.979 INFO:teuthology.orchestra.run.smithi088.stderr:2024-06-03T09:06:53.977+0000 7fc7effff640 10 monclient: _finish_auth 0 2024-06-03T09:06:53.979 INFO:teuthology.orchestra.run.smithi088.stderr:2024-06-03T09:06:53.977+0000 7fc7effff640 10 monclient: _check_auth_tickets 2024-06-03T09:06:53.979 INFO:teuthology.orchestra.run.smithi088.stderr:2024-06-03T09:06:53.977+0000 7fc7effff640 20 monclient: _check_auth_rotating not needed by client.admin 2024-06-03T09:06:53.979 INFO:teuthology.orchestra.run.smithi088.stderr:2024-06-03T09:06:53.977+0000 7fc80e1e7640 5 monclient: authenticate success, global_id 30729 2024-06-03T09:06:53.979 INFO:teuthology.orchestra.run.smithi088.stderr:2024-06-03T09:06:53.977+0000 7fc80e1e7640 10 monclient: _renew_subs 2024-06-03T09:06:53.979 INFO:teuthology.orchestra.run.smithi088.stderr:2024-06-03T09:06:53.977+0000 7fc80e1e7640 10 monclient: _send_mon_message to mon.a at v1:172.21.15.88:6789/0 2024-06-03T09:06:53.979 INFO:teuthology.orchestra.run.smithi088.stderr:2024-06-03T09:06:53.977+0000 7fc80e1e7640 1 -- 172.21.15.88:0/420261594 --> v1:172.21.15.88:6789/0 -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fc808136dc0 con 0x7fc80806ba10 2024-06-03T09:06:53.979 INFO:teuthology.orchestra.run.smithi088.stderr:2024-06-03T09:06:53.977+0000 7fc80e1e7640 10 client.30729.objecter _maybe_request_map subscribing (onetime) to next osd map 2024-06-03T09:06:53.980 INFO:teuthology.orchestra.run.smithi088.stderr:2024-06-03T09:06:53.977+0000 7fc80e1e7640 10 monclient: _renew_subs 2024-06-03T09:06:53.980 INFO:teuthology.orchestra.run.smithi088.stderr:2024-06-03T09:06:53.977+0000 7fc80e1e7640 10 monclient: _send_mon_message to mon.a at v1:172.21.15.88:6789/0 2024-06-03T09:06:53.980 INFO:teuthology.orchestra.run.smithi088.stderr:2024-06-03T09:06:53.977+0000 7fc80e1e7640 1 -- 172.21.15.88:0/420261594 --> v1:172.21.15.88:6789/0 -- mon_subscribe({osdmap=0}) v3 -- 0x7fc8080b3e90 con 0x7fc80806ba10 2024-06-03T09:06:53.980 INFO:teuthology.orchestra.run.smithi088.stderr:2024-06-03T09:06:53.977+0000 7fc80e1e7640 1 librados: init done 2024-06-03T09:06:53.980 INFO:teuthology.orchestra.run.smithi088.stderr:2024-06-03T09:06:53.978+0000 7fc80e1e7640 10 start_mon_command cmd=[{"prefix": "get_command_descriptions"}] 2024-06-03T09:06:53.980 INFO:teuthology.orchestra.run.smithi088.stderr:2024-06-03T09:06:53.978+0000 7fc80e1e7640 10 monclient: _send_command 1 [{"prefix": "get_command_descriptions"}] 2024-06-03T09:06:53.980 INFO:teuthology.orchestra.run.smithi088.stderr:2024-06-03T09:06:53.978+0000 7fc80e1e7640 10 monclient: _send_mon_message to mon.a at v1:172.21.15.88:6789/0 2024-06-03T09:06:53.980 INFO:teuthology.orchestra.run.smithi088.stderr:2024-06-03T09:06:53.978+0000 7fc80e1e7640 1 -- 172.21.15.88:0/420261594 --> v1:172.21.15.88:6789/0 -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fc8080aea20 con 0x7fc80806ba10 2024-06-03T09:06:54.242 INFO:teuthology.orchestra.run.smithi088.stderr:2024-06-03T09:06:54.241+0000 7fc7effff640 1 -- 172.21.15.88:0/420261594 <== mon.0 v1:172.21.15.88:6789/0 5 ==== config(0 keys) v1 ==== 4+0+0 (unknown 0 0 0) 0x7fc7f8003610 con 0x7fc80806ba10 2024-06-03T09:06:54.242 INFO:teuthology.orchestra.run.smithi088.stderr:2024-06-03T09:06:54.241+0000 7fc7effff640 10 monclient: handle_config config(0 keys) v1 2024-06-03T09:06:54.242 INFO:teuthology.orchestra.run.smithi088.stderr:2024-06-03T09:06:54.241+0000 7fc7effff640 1 -- 172.21.15.88:0/420261594 <== mon.0 v1:172.21.15.88:6789/0 6 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 2234374065 0 0) 0x7fc7f8003920 con 0x7fc80806ba10 2024-06-03T09:06:54.242 INFO:teuthology.orchestra.run.smithi088.stderr:2024-06-03T09:06:54.241+0000 7fc7effff640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-06-03T09:06:54.242 INFO:teuthology.orchestra.run.smithi088.stderr:2024-06-03T09:06:54.241+0000 7fc7effff640 10 monclient: got monmap 1 from mon.a (according to old e1) 2024-06-03T09:06:54.242 INFO:teuthology.orchestra.run.smithi088.stderr:2024-06-03T09:06:54.241+0000 7fc7effff640 10 monclient: dump: 2024-06-03T09:06:54.242 INFO:teuthology.orchestra.run.smithi088.stderr:epoch 1 2024-06-03T09:06:54.242 INFO:teuthology.orchestra.run.smithi088.stderr:fsid ec3c3fdd-3ac6-4d7d-9412-683130ed8b04 2024-06-03T09:06:54.242 INFO:teuthology.orchestra.run.smithi088.stderr:last_changed 2024-06-03T08:34:31.885125+0000 2024-06-03T09:06:54.242 INFO:teuthology.orchestra.run.smithi088.stderr:created 2024-06-03T08:34:31.885125+0000 2024-06-03T09:06:54.243 INFO:teuthology.orchestra.run.smithi088.stderr:min_mon_release 18 (reef) 2024-06-03T09:06:54.243 INFO:teuthology.orchestra.run.smithi088.stderr:election_strategy: 3 2024-06-03T09:06:54.243 INFO:teuthology.orchestra.run.smithi088.stderr:0: v1:172.21.15.88:6789/0 mon.a 2024-06-03T09:06:54.243 INFO:teuthology.orchestra.run.smithi088.stderr:1: v1:172.21.15.192:6789/0 mon.b 2024-06-03T09:06:54.243 INFO:teuthology.orchestra.run.smithi088.stderr:2: v1:172.21.15.88:6790/0 mon.c 2024-06-03T09:06:54.243 INFO:teuthology.orchestra.run.smithi088.stderr: 2024-06-03T09:06:54.243 INFO:teuthology.orchestra.run.smithi088.stderr:2024-06-03T09:06:54.241+0000 7fc805ffb640 4 set_mon_vals no callback set 2024-06-03T09:06:54.244 INFO:teuthology.orchestra.run.smithi088.stderr:2024-06-03T09:06:54.242+0000 7fc7effff640 1 -- 172.21.15.88:0/420261594 <== mon.0 v1:172.21.15.88:6789/0 7 ==== mgrmap(e 5) v1 ==== 89110+0+0 (unknown 873533973 0 0) 0x7fc7f8003a80 con 0x7fc80806ba10 2024-06-03T09:06:54.244 INFO:teuthology.orchestra.run.smithi088.stderr:2024-06-03T09:06:54.243+0000 7fc806ffd640 10 monclient: get_auth_request con 0x7fc7f406d190 auth_method 0 2024-06-03T09:06:54.244 INFO:teuthology.orchestra.run.smithi088.stderr:2024-06-03T09:06:54.243+0000 7fc7effff640 10 client.30729.objecter ms_handle_connect 0x7fc7f406d190 2024-06-03T09:06:54.249 INFO:teuthology.orchestra.run.smithi088.stderr:2024-06-03T09:06:54.249+0000 7fc7effff640 1 -- 172.21.15.88:0/420261594 <== mon.0 v1:172.21.15.88:6789/0 8 ==== osd_map(1014..1014 src has 622..1014) v4 ==== 5229+0+0 (unknown 3230192161 0 0) 0x7fc7f8054140 con 0x7fc80806ba10 2024-06-03T09:06:54.249 INFO:teuthology.orchestra.run.smithi088.stderr:2024-06-03T09:06:54.249+0000 7fc7effff640 10 client.30729.objecter ms_dispatch 0x7fc808000b90 osd_map(1014..1014 src has 622..1014) v4 2024-06-03T09:06:54.249 INFO:teuthology.orchestra.run.smithi088.stderr:2024-06-03T09:06:54.249+0000 7fc7effff640 3 client.30729.objecter handle_osd_map got epochs [1014,1014] > 0 2024-06-03T09:06:54.249 INFO:teuthology.orchestra.run.smithi088.stderr:2024-06-03T09:06:54.249+0000 7fc7effff640 3 client.30729.objecter handle_osd_map decoding full epoch 1014 2024-06-03T09:06:54.249 INFO:teuthology.orchestra.run.smithi088.stderr:2024-06-03T09:06:54.249+0000 7fc7effff640 20 client.30729.objecter dump_active .. 0 homeless 2024-06-03T09:06:54.249 INFO:teuthology.orchestra.run.smithi088.stderr:2024-06-03T09:06:54.249+0000 7fc7effff640 1 -- 172.21.15.88:0/420261594 <== mon.0 v1:172.21.15.88:6789/0 9 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+182498 (unknown 1092875540 0 3016760700) 0x7fc7f804fcf0 con 0x7fc80806ba10 2024-06-03T09:06:54.250 INFO:teuthology.orchestra.run.smithi088.stderr:2024-06-03T09:06:54.249+0000 7fc7effff640 10 monclient: handle_mon_command_ack 1 [{"prefix": "get_command_descriptions"}] 2024-06-03T09:06:54.250 INFO:teuthology.orchestra.run.smithi088.stderr:2024-06-03T09:06:54.249+0000 7fc7effff640 10 monclient: _finish_command 1 = system:0 2024-06-03T09:06:54.299 INFO:tasks.ceph.mon.b.smithi192.stderr:==00:00:31:49.602 34228== Warning: unimplemented fcntl command: 1036 2024-06-03T09:06:54.300 INFO:tasks.ceph.mon.b.smithi192.stderr:==00:00:31:49.603 34228== Warning: unimplemented fcntl command: 1036 2024-06-03T09:06:54.308 INFO:tasks.ceph.mon.b.smithi192.stderr:==00:00:31:49.611 34228== Warning: unimplemented fcntl command: 1036 2024-06-03T09:06:54.327 INFO:tasks.ceph.mon.c.smithi088.stderr:==00:00:31:49.653 34749== Warning: unimplemented fcntl command: 1036 2024-06-03T09:06:54.329 INFO:tasks.ceph.mon.c.smithi088.stderr:==00:00:31:49.655 34749== Warning: unimplemented fcntl command: 1036 2024-06-03T09:06:54.338 INFO:tasks.ceph.mon.c.smithi088.stderr:==00:00:31:49.663 34749== Warning: unimplemented fcntl command: 1036 2024-06-03T09:06:54.436 INFO:teuthology.orchestra.run.smithi088.stderr:2024-06-03T09:06:54.435+0000 7fc80e1e7640 1 -- 172.21.15.88:0/420261594 --> v1:172.21.15.192:6800/34229 -- mgr_command(tid 0: {"prefix": "pg dump", "target": ["mon-mgr", ""], "format": "json"}) v1 -- 0x7fc8081416b0 con 0x7fc7f406d190 2024-06-03T09:06:54.440 INFO:teuthology.orchestra.run.smithi088.stderr:2024-06-03T09:06:54.440+0000 7fc7effff640 1 -- 172.21.15.88:0/420261594 <== mgr.4110 v1:172.21.15.192:6800/34229 1 ==== mgr_command_reply(tid 0: 0 dumped all) v1 ==== 18+0+151688 (unknown 2965378022 0 548438894) 0x7fc7fc0017a0 con 0x7fc7f406d190 2024-06-03T09:06:54.442 INFO:teuthology.orchestra.run.smithi088.stdout: 2024-06-03T09:06:54.442 INFO:teuthology.orchestra.run.smithi088.stderr:dumped all 2024-06-03T09:06:54.444 INFO:teuthology.orchestra.run.smithi088.stderr:2024-06-03T09:06:54.443+0000 7fc80e1e7640 10 librados: watch_flush enter 2024-06-03T09:06:54.444 INFO:teuthology.orchestra.run.smithi088.stderr:2024-06-03T09:06:54.443+0000 7fc80e1e7640 10 librados: watch_flush exit 2024-06-03T09:06:54.444 INFO:teuthology.orchestra.run.smithi088.stderr:2024-06-03T09:06:54.443+0000 7fc80e1e7640 20 client.30729.objecter shutdown clearing up homeless session... 2024-06-03T09:06:54.444 INFO:teuthology.orchestra.run.smithi088.stderr:2024-06-03T09:06:54.443+0000 7fc80e1e7640 10 client.30729.objecter successfully canceled tick 2024-06-03T09:06:54.444 INFO:teuthology.orchestra.run.smithi088.stderr:2024-06-03T09:06:54.443+0000 7fc80e1e7640 1 -- 172.21.15.88:0/420261594 >> v1:172.21.15.192:6800/34229 conn(0x7fc7f406d190 legacy=0x7fc7f406f650 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-06-03T09:06:54.444 INFO:teuthology.orchestra.run.smithi088.stderr:2024-06-03T09:06:54.443+0000 7fc80e1e7640 10 monclient: shutdown 2024-06-03T09:06:54.444 INFO:teuthology.orchestra.run.smithi088.stderr:2024-06-03T09:06:54.443+0000 7fc80e1e7640 20 monclient: shutdown discarding 0 pending message(s) 2024-06-03T09:06:54.444 INFO:teuthology.orchestra.run.smithi088.stderr:2024-06-03T09:06:54.443+0000 7fc80e1e7640 1 -- 172.21.15.88:0/420261594 >> v1:172.21.15.88:6789/0 conn(0x7fc80806ba10 legacy=0x7fc808070a90 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-06-03T09:06:54.444 INFO:teuthology.orchestra.run.smithi088.stderr:2024-06-03T09:06:54.443+0000 7fc80e1e7640 1 -- 172.21.15.88:0/420261594 shutdown_connections 2024-06-03T09:06:54.445 INFO:teuthology.orchestra.run.smithi088.stderr:2024-06-03T09:06:54.443+0000 7fc80e1e7640 1 -- 172.21.15.88:0/420261594 >> 172.21.15.88:0/420261594 conn(0x7fc8080b4fb0 msgr2=0x7fc8080b22d0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-06-03T09:06:54.445 INFO:teuthology.orchestra.run.smithi088.stderr:2024-06-03T09:06:54.443+0000 7fc80e1e7640 1 -- 172.21.15.88:0/420261594 shutdown_connections 2024-06-03T09:06:54.445 INFO:teuthology.orchestra.run.smithi088.stderr:2024-06-03T09:06:54.443+0000 7fc80e1e7640 1 -- 172.21.15.88:0/420261594 wait complete. 2024-06-03T09:06:54.445 INFO:teuthology.orchestra.run.smithi088.stderr:2024-06-03T09:06:54.443+0000 7fc80e1e7640 1 librados: shutdown 2024-06-03T09:06:54.455 INFO:teuthology.orchestra.run.smithi088.stdout:{"pg_ready":true,"pg_map":{"version":1936,"stamp":"2024-06-03T09:06:53.105541+0000","last_osdmap_epoch":0,"last_pg_scan":0,"pg_stats_sum":{"stat_sum":{"num_bytes":459285,"num_objects":3,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":3,"num_whiteouts":0,"num_read":102,"num_read_kb":88,"num_write":120,"num_write_kb":1379,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2,"num_bytes_recovered":397840,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":83,"ondisk_log_size":83,"up":82,"acting":82,"num_store_stats":0},"osd_stats_sum":{"up_from":0,"seq":0,"num_pgs":82,"num_osds":8,"num_per_pool_osds":8,"num_per_pool_omap_osds":8,"kb":754974720,"kb_used":495068,"kb_used_data":46808,"kb_used_omap":12,"kb_used_meta":448115,"kb_avail":754479652,"statfs":{"total":773094113280,"available":772587163648,"internally_reserved":0,"allocated":47931392,"data_stored":22900274,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":12744,"internal_metadata":458870328},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":77,"apply_latency_ms":77,"commit_latency_ns":77000000,"apply_latency_ns":77000000},"alerts":[],"network_ping_times":[]},"pg_stats_delta":{"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":-1,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":0,"acting":0,"num_store_stats":0,"stamp_delta":"8.148465"},"pg_stats":[{"pgid":"17.d","version":"0'0","reported_seq":945,"reported_epoch":1013,"state":"active+clean","last_fresh":"2024-06-03T09:06:45.930577+0000","last_change":"2024-06-03T08:38:19.260548+0000","last_active":"2024-06-03T09:06:45.930577+0000","last_peered":"2024-06-03T09:06:45.930577+0000","last_clean":"2024-06-03T09:06:45.930577+0000","last_became_active":"2024-06-03T08:38:19.253559+0000","last_became_peered":"2024-06-03T08:38:19.253559+0000","last_unstale":"2024-06-03T09:06:45.930577+0000","last_undegraded":"2024-06-03T09:06:45.930577+0000","last_fullsized":"2024-06-03T09:06:45.930577+0000","mapping_epoch":28,"log_start":"0'0","ondisk_log_start":"0'0","created":28,"last_epoch_clean":45,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-06-03T08:37:55.614497+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-06-03T08:37:55.614497+0000","last_clean_scrub_stamp":"2024-06-03T08:37:55.614497+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-06-04T19:00:12.470979+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1],"acting":[0,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"17.c","version":"0'0","reported_seq":965,"reported_epoch":1014,"state":"active+clean","last_fresh":"2024-06-03T09:06:46.317689+0000","last_change":"2024-06-03T09:06:46.317689+0000","last_active":"2024-06-03T09:06:46.317689+0000","last_peered":"2024-06-03T09:06:46.317689+0000","last_clean":"2024-06-03T09:06:46.317689+0000","last_became_active":"2024-06-03T09:06:46.312637+0000","last_became_peered":"2024-06-03T09:06:46.312637+0000","last_unstale":"2024-06-03T09:06:46.317689+0000","last_undegraded":"2024-06-03T09:06:46.317689+0000","last_fullsized":"2024-06-03T09:06:46.317689+0000","mapping_epoch":1013,"log_start":"0'0","ondisk_log_start":"0'0","created":28,"last_epoch_clean":1014,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-06-03T08:37:55.614497+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-06-03T08:37:55.614497+0000","last_clean_scrub_stamp":"2024-06-03T08:37:55.614497+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-06-04T17:49:40.110047+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,6],"acting":[3,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"17.f","version":"0'0","reported_seq":843,"reported_epoch":1014,"state":"active+clean","last_fresh":"2024-06-03T09:06:46.306998+0000","last_change":"2024-06-03T09:06:46.306998+0000","last_active":"2024-06-03T09:06:46.306998+0000","last_peered":"2024-06-03T09:06:46.306998+0000","last_clean":"2024-06-03T09:06:46.306998+0000","last_became_active":"2024-06-03T09:06:46.301958+0000","last_became_peered":"2024-06-03T09:06:46.301958+0000","last_unstale":"2024-06-03T09:06:46.306998+0000","last_undegraded":"2024-06-03T09:06:46.306998+0000","last_fullsized":"2024-06-03T09:06:46.306998+0000","mapping_epoch":1013,"log_start":"0'0","ondisk_log_start":"0'0","created":28,"last_epoch_clean":1014,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-06-03T08:37:55.614497+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-06-03T08:37:55.614497+0000","last_clean_scrub_stamp":"2024-06-03T08:37:55.614497+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-06-04T14:30:23.089522+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,4],"acting":[6,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"17.e","version":"0'0","reported_seq":843,"reported_epoch":1014,"state":"active+clean","last_fresh":"2024-06-03T09:06:46.278835+0000","last_change":"2024-06-03T09:06:46.278835+0000","last_active":"2024-06-03T09:06:46.278835+0000","last_peered":"2024-06-03T09:06:46.278835+0000","last_clean":"2024-06-03T09:06:46.278835+0000","last_became_active":"2024-06-03T09:06:46.273941+0000","last_became_peered":"2024-06-03T09:06:46.273941+0000","last_unstale":"2024-06-03T09:06:46.278835+0000","last_undegraded":"2024-06-03T09:06:46.278835+0000","last_fullsized":"2024-06-03T09:06:46.278835+0000","mapping_epoch":1013,"log_start":"0'0","ondisk_log_start":"0'0","created":28,"last_epoch_clean":1014,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-06-03T08:37:55.614497+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-06-03T08:37:55.614497+0000","last_clean_scrub_stamp":"2024-06-03T08:37:55.614497+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-06-04T09:31:07.311650+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,5],"acting":[6,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"17.9","version":"0'0","reported_seq":949,"reported_epoch":1013,"state":"active+clean","last_fresh":"2024-06-03T09:06:45.422478+0000","last_change":"2024-06-03T08:38:04.772260+0000","last_active":"2024-06-03T09:06:45.422478+0000","last_peered":"2024-06-03T09:06:45.422478+0000","last_clean":"2024-06-03T09:06:45.422478+0000","last_became_active":"2024-06-03T08:38:04.763428+0000","last_became_peered":"2024-06-03T08:38:04.763428+0000","last_unstale":"2024-06-03T09:06:45.422478+0000","last_undegraded":"2024-06-03T09:06:45.422478+0000","last_fullsized":"2024-06-03T09:06:45.422478+0000","mapping_epoch":28,"log_start":"0'0","ondisk_log_start":"0'0","created":28,"last_epoch_clean":33,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-06-03T08:37:55.614497+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-06-03T08:37:55.614497+0000","last_clean_scrub_stamp":"2024-06-03T08:37:55.614497+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-06-04T10:55:10.773936+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,4],"acting":[7,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"17.8","version":"0'0","reported_seq":972,"reported_epoch":1014,"state":"active+clean","last_fresh":"2024-06-03T09:06:46.349899+0000","last_change":"2024-06-03T09:06:46.349899+0000","last_active":"2024-06-03T09:06:46.349899+0000","last_peered":"2024-06-03T09:06:46.349899+0000","last_clean":"2024-06-03T09:06:46.349899+0000","last_became_active":"2024-06-03T09:06:46.344756+0000","last_became_peered":"2024-06-03T09:06:46.344756+0000","last_unstale":"2024-06-03T09:06:46.349899+0000","last_undegraded":"2024-06-03T09:06:46.349899+0000","last_fullsized":"2024-06-03T09:06:46.349899+0000","mapping_epoch":1013,"log_start":"0'0","ondisk_log_start":"0'0","created":28,"last_epoch_clean":1014,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-06-03T08:37:55.614497+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-06-03T08:37:55.614497+0000","last_clean_scrub_stamp":"2024-06-03T08:37:55.614497+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-06-04T09:33:29.773440+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,6],"acting":[0,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"17.b","version":"0'0","reported_seq":843,"reported_epoch":1014,"state":"active+clean","last_fresh":"2024-06-03T09:06:46.350450+0000","last_change":"2024-06-03T09:06:46.350450+0000","last_active":"2024-06-03T09:06:46.350450+0000","last_peered":"2024-06-03T09:06:46.350450+0000","last_clean":"2024-06-03T09:06:46.350450+0000","last_became_active":"2024-06-03T09:06:46.345458+0000","last_became_peered":"2024-06-03T09:06:46.345458+0000","last_unstale":"2024-06-03T09:06:46.350450+0000","last_undegraded":"2024-06-03T09:06:46.350450+0000","last_fullsized":"2024-06-03T09:06:46.350450+0000","mapping_epoch":1013,"log_start":"0'0","ondisk_log_start":"0'0","created":28,"last_epoch_clean":1014,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-06-03T08:37:55.614497+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-06-03T08:37:55.614497+0000","last_clean_scrub_stamp":"2024-06-03T08:37:55.614497+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-06-04T16:51:34.952224+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,2],"acting":[6,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"17.a","version":"0'0","reported_seq":929,"reported_epoch":1013,"state":"active+clean","last_fresh":"2024-06-03T09:06:45.882773+0000","last_change":"2024-06-03T08:38:05.843142+0000","last_active":"2024-06-03T09:06:45.882773+0000","last_peered":"2024-06-03T09:06:45.882773+0000","last_clean":"2024-06-03T09:06:45.882773+0000","last_became_active":"2024-06-03T08:38:05.809257+0000","last_became_peered":"2024-06-03T08:38:05.809257+0000","last_unstale":"2024-06-03T09:06:45.882773+0000","last_undegraded":"2024-06-03T09:06:45.882773+0000","last_fullsized":"2024-06-03T09:06:45.882773+0000","mapping_epoch":28,"log_start":"0'0","ondisk_log_start":"0'0","created":28,"last_epoch_clean":33,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-06-03T08:37:55.614497+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-06-03T08:37:55.614497+0000","last_clean_scrub_stamp":"2024-06-03T08:37:55.614497+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-06-04T12:12:03.686838+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2],"acting":[1,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"17.5","version":"0'0","reported_seq":946,"reported_epoch":1013,"state":"active+clean","last_fresh":"2024-06-03T09:06:45.929134+0000","last_change":"2024-06-03T08:38:23.466051+0000","last_active":"2024-06-03T09:06:45.929134+0000","last_peered":"2024-06-03T09:06:45.929134+0000","last_clean":"2024-06-03T09:06:45.929134+0000","last_became_active":"2024-06-03T08:38:23.458039+0000","last_became_peered":"2024-06-03T08:38:23.458039+0000","last_unstale":"2024-06-03T09:06:45.929134+0000","last_undegraded":"2024-06-03T09:06:45.929134+0000","last_fullsized":"2024-06-03T09:06:45.929134+0000","mapping_epoch":28,"log_start":"0'0","ondisk_log_start":"0'0","created":28,"last_epoch_clean":49,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-06-03T08:37:55.614497+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-06-03T08:37:55.614497+0000","last_clean_scrub_stamp":"2024-06-03T08:37:55.614497+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-06-04T12:27:02.532436+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,4],"acting":[0,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"17.4","version":"0'0","reported_seq":843,"reported_epoch":1014,"state":"active+clean","last_fresh":"2024-06-03T09:06:46.456388+0000","last_change":"2024-06-03T09:06:46.456388+0000","last_active":"2024-06-03T09:06:46.456388+0000","last_peered":"2024-06-03T09:06:46.456388+0000","last_clean":"2024-06-03T09:06:46.456388+0000","last_became_active":"2024-06-03T09:06:46.447534+0000","last_became_peered":"2024-06-03T09:06:46.447534+0000","last_unstale":"2024-06-03T09:06:46.456388+0000","last_undegraded":"2024-06-03T09:06:46.456388+0000","last_fullsized":"2024-06-03T09:06:46.456388+0000","mapping_epoch":1013,"log_start":"0'0","ondisk_log_start":"0'0","created":28,"last_epoch_clean":1014,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-06-03T08:37:55.614497+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-06-03T08:37:55.614497+0000","last_clean_scrub_stamp":"2024-06-03T08:37:55.614497+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-06-04T16:06:17.792570+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,7],"acting":[6,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"17.7","version":"0'0","reported_seq":933,"reported_epoch":1013,"state":"active+clean","last_fresh":"2024-06-03T09:06:45.495807+0000","last_change":"2024-06-03T08:38:04.837083+0000","last_active":"2024-06-03T09:06:45.495807+0000","last_peered":"2024-06-03T09:06:45.495807+0000","last_clean":"2024-06-03T09:06:45.495807+0000","last_became_active":"2024-06-03T08:38:04.818246+0000","last_became_peered":"2024-06-03T08:38:04.818246+0000","last_unstale":"2024-06-03T09:06:45.495807+0000","last_undegraded":"2024-06-03T09:06:45.495807+0000","last_fullsized":"2024-06-03T09:06:45.495807+0000","mapping_epoch":28,"log_start":"0'0","ondisk_log_start":"0'0","created":28,"last_epoch_clean":34,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-06-03T08:37:55.614497+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-06-03T08:37:55.614497+0000","last_clean_scrub_stamp":"2024-06-03T08:37:55.614497+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-06-04T17:02:38.204813+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,2],"acting":[5,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"17.6","version":"0'0","reported_seq":971,"reported_epoch":1014,"state":"active+clean","last_fresh":"2024-06-03T09:06:46.362623+0000","last_change":"2024-06-03T09:06:46.362623+0000","last_active":"2024-06-03T09:06:46.362623+0000","last_peered":"2024-06-03T09:06:46.362623+0000","last_clean":"2024-06-03T09:06:46.362623+0000","last_became_active":"2024-06-03T09:06:46.357698+0000","last_became_peered":"2024-06-03T09:06:46.357698+0000","last_unstale":"2024-06-03T09:06:46.362623+0000","last_undegraded":"2024-06-03T09:06:46.362623+0000","last_fullsized":"2024-06-03T09:06:46.362623+0000","mapping_epoch":1013,"log_start":"0'0","ondisk_log_start":"0'0","created":28,"last_epoch_clean":1014,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-06-03T08:37:55.614497+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-06-03T08:37:55.614497+0000","last_clean_scrub_stamp":"2024-06-03T08:37:55.614497+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-06-04T12:51:05.494877+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,6],"acting":[0,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"2.7","version":"0'0","reported_seq":944,"reported_epoch":1013,"state":"active+clean","last_fresh":"2024-06-03T09:06:45.158971+0000","last_change":"2024-06-03T08:36:30.385405+0000","last_active":"2024-06-03T09:06:45.158971+0000","last_peered":"2024-06-03T09:06:45.158971+0000","last_clean":"2024-06-03T09:06:45.158971+0000","last_became_active":"2024-06-03T08:36:30.380302+0000","last_became_peered":"2024-06-03T08:36:30.380302+0000","last_unstale":"2024-06-03T09:06:45.158971+0000","last_undegraded":"2024-06-03T09:06:45.158971+0000","last_fullsized":"2024-06-03T09:06:45.158971+0000","mapping_epoch":23,"log_start":"0'0","ondisk_log_start":"0'0","created":23,"last_epoch_clean":24,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-06-03T08:36:28.444764+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-06-03T08:36:28.444764+0000","last_clean_scrub_stamp":"2024-06-03T08:36:28.444764+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-06-04T18:46:33.475181+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0],"acting":[3,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"17.14","version":"0'0","reported_seq":930,"reported_epoch":1013,"state":"active+clean","last_fresh":"2024-06-03T09:06:45.541591+0000","last_change":"2024-06-03T08:38:04.547565+0000","last_active":"2024-06-03T09:06:45.541591+0000","last_peered":"2024-06-03T09:06:45.541591+0000","last_clean":"2024-06-03T09:06:45.541591+0000","last_became_active":"2024-06-03T08:38:04.541781+0000","last_became_peered":"2024-06-03T08:38:04.541781+0000","last_unstale":"2024-06-03T09:06:45.541591+0000","last_undegraded":"2024-06-03T09:06:45.541591+0000","last_fullsized":"2024-06-03T09:06:45.541591+0000","mapping_epoch":28,"log_start":"0'0","ondisk_log_start":"0'0","created":28,"last_epoch_clean":34,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-06-03T08:37:55.614497+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-06-03T08:37:55.614497+0000","last_clean_scrub_stamp":"2024-06-03T08:37:55.614497+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-06-04T16:41:55.170134+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,2],"acting":[5,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"2.6","version":"0'0","reported_seq":953,"reported_epoch":1013,"state":"active+clean","last_fresh":"2024-06-03T09:06:45.018770+0000","last_change":"2024-06-03T08:36:30.430760+0000","last_active":"2024-06-03T09:06:45.018770+0000","last_peered":"2024-06-03T09:06:45.018770+0000","last_clean":"2024-06-03T09:06:45.018770+0000","last_became_active":"2024-06-03T08:36:30.404665+0000","last_became_peered":"2024-06-03T08:36:30.404665+0000","last_unstale":"2024-06-03T09:06:45.018770+0000","last_undegraded":"2024-06-03T09:06:45.018770+0000","last_fullsized":"2024-06-03T09:06:45.018770+0000","mapping_epoch":23,"log_start":"0'0","ondisk_log_start":"0'0","created":23,"last_epoch_clean":24,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-06-03T08:36:28.444764+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-06-03T08:36:28.444764+0000","last_clean_scrub_stamp":"2024-06-03T08:36:28.444764+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-06-04T10:37:37.746074+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,0],"acting":[4,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"17.15","version":"0'0","reported_seq":938,"reported_epoch":1013,"state":"active+clean","last_fresh":"2024-06-03T09:06:45.160758+0000","last_change":"2024-06-03T08:38:20.999903+0000","last_active":"2024-06-03T09:06:45.160758+0000","last_peered":"2024-06-03T09:06:45.160758+0000","last_clean":"2024-06-03T09:06:45.160758+0000","last_became_active":"2024-06-03T08:38:20.979523+0000","last_became_peered":"2024-06-03T08:38:20.979523+0000","last_unstale":"2024-06-03T09:06:45.160758+0000","last_undegraded":"2024-06-03T09:06:45.160758+0000","last_fullsized":"2024-06-03T09:06:45.160758+0000","mapping_epoch":28,"log_start":"0'0","ondisk_log_start":"0'0","created":28,"last_epoch_clean":47,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-06-03T08:37:55.614497+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-06-03T08:37:55.614497+0000","last_clean_scrub_stamp":"2024-06-03T08:37:55.614497+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-06-04T20:29:58.314289+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,1],"acting":[3,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"2.5","version":"0'0","reported_seq":946,"reported_epoch":1013,"state":"active+clean","last_fresh":"2024-06-03T09:06:45.162304+0000","last_change":"2024-06-03T08:36:30.024235+0000","last_active":"2024-06-03T09:06:45.162304+0000","last_peered":"2024-06-03T09:06:45.162304+0000","last_clean":"2024-06-03T09:06:45.162304+0000","last_became_active":"2024-06-03T08:36:30.019040+0000","last_became_peered":"2024-06-03T08:36:30.019040+0000","last_unstale":"2024-06-03T09:06:45.162304+0000","last_undegraded":"2024-06-03T09:06:45.162304+0000","last_fullsized":"2024-06-03T09:06:45.162304+0000","mapping_epoch":23,"log_start":"0'0","ondisk_log_start":"0'0","created":23,"last_epoch_clean":24,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-06-03T08:36:28.444764+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-06-03T08:36:28.444764+0000","last_clean_scrub_stamp":"2024-06-03T08:36:28.444764+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-06-04T15:46:47.109502+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,4],"acting":[3,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"17.16","version":"0'0","reported_seq":984,"reported_epoch":1014,"state":"active+clean","last_fresh":"2024-06-03T09:06:46.384595+0000","last_change":"2024-06-03T09:06:46.384595+0000","last_active":"2024-06-03T09:06:46.384595+0000","last_peered":"2024-06-03T09:06:46.384595+0000","last_clean":"2024-06-03T09:06:46.384595+0000","last_became_active":"2024-06-03T09:06:46.379518+0000","last_became_peered":"2024-06-03T09:06:46.379518+0000","last_unstale":"2024-06-03T09:06:46.384595+0000","last_undegraded":"2024-06-03T09:06:46.384595+0000","last_fullsized":"2024-06-03T09:06:46.384595+0000","mapping_epoch":1013,"log_start":"0'0","ondisk_log_start":"0'0","created":28,"last_epoch_clean":1014,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-06-03T08:37:55.614497+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-06-03T08:37:55.614497+0000","last_clean_scrub_stamp":"2024-06-03T08:37:55.614497+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-06-04T12:51:20.305056+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,6],"acting":[2,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"2.4","version":"0'0","reported_seq":964,"reported_epoch":1014,"state":"active+clean","last_fresh":"2024-06-03T09:06:46.377610+0000","last_change":"2024-06-03T09:06:46.377610+0000","last_active":"2024-06-03T09:06:46.377610+0000","last_peered":"2024-06-03T09:06:46.377610+0000","last_clean":"2024-06-03T09:06:46.377610+0000","last_became_active":"2024-06-03T09:06:46.371430+0000","last_became_peered":"2024-06-03T09:06:46.371430+0000","last_unstale":"2024-06-03T09:06:46.377610+0000","last_undegraded":"2024-06-03T09:06:46.377610+0000","last_fullsized":"2024-06-03T09:06:46.377610+0000","mapping_epoch":1013,"log_start":"0'0","ondisk_log_start":"0'0","created":23,"last_epoch_clean":1014,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-06-03T08:36:28.444764+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-06-03T08:36:28.444764+0000","last_clean_scrub_stamp":"2024-06-03T08:36:28.444764+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-06-04T12:20:39.476337+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,6],"acting":[5,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"17.17","version":"0'0","reported_seq":958,"reported_epoch":1013,"state":"active+clean","last_fresh":"2024-06-03T09:06:45.796338+0000","last_change":"2024-06-03T08:38:38.483275+0000","last_active":"2024-06-03T09:06:45.796338+0000","last_peered":"2024-06-03T09:06:45.796338+0000","last_clean":"2024-06-03T09:06:45.796338+0000","last_became_active":"2024-06-03T08:38:38.467916+0000","last_became_peered":"2024-06-03T08:38:38.467916+0000","last_unstale":"2024-06-03T09:06:45.796338+0000","last_undegraded":"2024-06-03T09:06:45.796338+0000","last_fullsized":"2024-06-03T09:06:45.796338+0000","mapping_epoch":28,"log_start":"0'0","ondisk_log_start":"0'0","created":28,"last_epoch_clean":61,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-06-03T08:37:55.614497+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-06-03T08:37:55.614497+0000","last_clean_scrub_stamp":"2024-06-03T08:37:55.614497+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-06-04T10:02:23.036420+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,7],"acting":[2,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"2.2","version":"0'0","reported_seq":975,"reported_epoch":1014,"state":"active+clean","last_fresh":"2024-06-03T09:06:46.290105+0000","last_change":"2024-06-03T09:06:46.290105+0000","last_active":"2024-06-03T09:06:46.290105+0000","last_peered":"2024-06-03T09:06:46.290105+0000","last_clean":"2024-06-03T09:06:46.290105+0000","last_became_active":"2024-06-03T09:06:46.284836+0000","last_became_peered":"2024-06-03T09:06:46.284836+0000","last_unstale":"2024-06-03T09:06:46.290105+0000","last_undegraded":"2024-06-03T09:06:46.290105+0000","last_fullsized":"2024-06-03T09:06:46.290105+0000","mapping_epoch":1013,"log_start":"0'0","ondisk_log_start":"0'0","created":23,"last_epoch_clean":1014,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-06-03T08:36:28.444764+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-06-03T08:36:28.444764+0000","last_clean_scrub_stamp":"2024-06-03T08:36:28.444764+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-06-04T14:05:42.945227+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,6],"acting":[3,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"17.11","version":"0'0","reported_seq":947,"reported_epoch":1013,"state":"active+clean","last_fresh":"2024-06-03T09:06:44.976905+0000","last_change":"2024-06-03T08:38:06.502302+0000","last_active":"2024-06-03T09:06:44.976905+0000","last_peered":"2024-06-03T09:06:44.976905+0000","last_clean":"2024-06-03T09:06:44.976905+0000","last_became_active":"2024-06-03T08:38:06.490204+0000","last_became_peered":"2024-06-03T08:38:06.490204+0000","last_unstale":"2024-06-03T09:06:44.976905+0000","last_undegraded":"2024-06-03T09:06:44.976905+0000","last_fullsized":"2024-06-03T09:06:44.976905+0000","mapping_epoch":28,"log_start":"0'0","ondisk_log_start":"0'0","created":28,"last_epoch_clean":32,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-06-03T08:37:55.614497+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-06-03T08:37:55.614497+0000","last_clean_scrub_stamp":"2024-06-03T08:37:55.614497+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-06-04T10:47:05.988958+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,2],"acting":[4,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"2.1","version":"0'0","reported_seq":955,"reported_epoch":1013,"state":"active+clean","last_fresh":"2024-06-03T09:06:45.423921+0000","last_change":"2024-06-03T08:36:30.883360+0000","last_active":"2024-06-03T09:06:45.423921+0000","last_peered":"2024-06-03T09:06:45.423921+0000","last_clean":"2024-06-03T09:06:45.423921+0000","last_became_active":"2024-06-03T08:36:30.855073+0000","last_became_peered":"2024-06-03T08:36:30.855073+0000","last_unstale":"2024-06-03T09:06:45.423921+0000","last_undegraded":"2024-06-03T09:06:45.423921+0000","last_fullsized":"2024-06-03T09:06:45.423921+0000","mapping_epoch":23,"log_start":"0'0","ondisk_log_start":"0'0","created":23,"last_epoch_clean":25,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-06-03T08:36:28.444764+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-06-03T08:36:28.444764+0000","last_clean_scrub_stamp":"2024-06-03T08:36:28.444764+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-06-04T18:12:22.363473+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,5],"acting":[7,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"17.12","version":"0'0","reported_seq":948,"reported_epoch":1013,"state":"active+clean","last_fresh":"2024-06-03T09:06:45.447050+0000","last_change":"2024-06-03T08:38:01.751562+0000","last_active":"2024-06-03T09:06:45.447050+0000","last_peered":"2024-06-03T09:06:45.447050+0000","last_clean":"2024-06-03T09:06:45.447050+0000","last_became_active":"2024-06-03T08:38:01.745027+0000","last_became_peered":"2024-06-03T08:38:01.745027+0000","last_unstale":"2024-06-03T09:06:45.447050+0000","last_undegraded":"2024-06-03T09:06:45.447050+0000","last_fullsized":"2024-06-03T09:06:45.447050+0000","mapping_epoch":28,"log_start":"0'0","ondisk_log_start":"0'0","created":28,"last_epoch_clean":32,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-06-03T08:37:55.614497+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-06-03T08:37:55.614497+0000","last_clean_scrub_stamp":"2024-06-03T08:37:55.614497+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-06-04T17:54:43.785326+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,3],"acting":[7,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"2.0","version":"0'0","reported_seq":948,"reported_epoch":1013,"state":"active+clean","last_fresh":"2024-06-03T09:06:45.108519+0000","last_change":"2024-06-03T08:36:29.984096+0000","last_active":"2024-06-03T09:06:45.108519+0000","last_peered":"2024-06-03T09:06:45.108519+0000","last_clean":"2024-06-03T09:06:45.108519+0000","last_became_active":"2024-06-03T08:36:29.957875+0000","last_became_peered":"2024-06-03T08:36:29.957875+0000","last_unstale":"2024-06-03T09:06:45.108519+0000","last_undegraded":"2024-06-03T09:06:45.108519+0000","last_fullsized":"2024-06-03T09:06:45.108519+0000","mapping_epoch":23,"log_start":"0'0","ondisk_log_start":"0'0","created":23,"last_epoch_clean":24,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-06-03T08:36:28.444764+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-06-03T08:36:28.444764+0000","last_clean_scrub_stamp":"2024-06-03T08:36:28.444764+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-06-04T08:41:33.679479+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,5],"acting":[3,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"17.13","version":"0'0","reported_seq":946,"reported_epoch":1013,"state":"active+clean","last_fresh":"2024-06-03T09:06:44.996126+0000","last_change":"2024-06-03T08:38:26.838316+0000","last_active":"2024-06-03T09:06:44.996126+0000","last_peered":"2024-06-03T09:06:44.996126+0000","last_clean":"2024-06-03T09:06:44.996126+0000","last_became_active":"2024-06-03T08:38:26.833214+0000","last_became_peered":"2024-06-03T08:38:26.833214+0000","last_unstale":"2024-06-03T09:06:44.996126+0000","last_undegraded":"2024-06-03T09:06:44.996126+0000","last_fullsized":"2024-06-03T09:06:44.996126+0000","mapping_epoch":28,"log_start":"0'0","ondisk_log_start":"0'0","created":28,"last_epoch_clean":52,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-06-03T08:37:55.614497+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-06-03T08:37:55.614497+0000","last_clean_scrub_stamp":"2024-06-03T08:37:55.614497+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-06-04T12:58:51.617998+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,0],"acting":[4,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"2.3","version":"0'0","reported_seq":947,"reported_epoch":1013,"state":"active+clean","last_fresh":"2024-06-03T09:06:45.137572+0000","last_change":"2024-06-03T08:36:30.351670+0000","last_active":"2024-06-03T09:06:45.137572+0000","last_peered":"2024-06-03T09:06:45.137572+0000","last_clean":"2024-06-03T09:06:45.137572+0000","last_became_active":"2024-06-03T08:36:30.343560+0000","last_became_peered":"2024-06-03T08:36:30.343560+0000","last_unstale":"2024-06-03T09:06:45.137572+0000","last_undegraded":"2024-06-03T09:06:45.137572+0000","last_fullsized":"2024-06-03T09:06:45.137572+0000","mapping_epoch":23,"log_start":"0'0","ondisk_log_start":"0'0","created":23,"last_epoch_clean":24,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-06-03T08:36:28.444764+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-06-03T08:36:28.444764+0000","last_clean_scrub_stamp":"2024-06-03T08:36:28.444764+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-06-04T12:02:24.765135+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,2],"acting":[3,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.0","version":"19'76","reported_seq":1056,"reported_epoch":1014,"state":"active+clean","last_fresh":"2024-06-03T09:06:46.463038+0000","last_change":"2024-06-03T08:36:26.702702+0000","last_active":"2024-06-03T09:06:46.463038+0000","last_peered":"2024-06-03T09:06:46.463038+0000","last_clean":"2024-06-03T09:06:46.463038+0000","last_became_active":"2024-06-03T08:36:26.128378+0000","last_became_peered":"2024-06-03T08:36:26.128378+0000","last_unstale":"2024-06-03T09:06:46.463038+0000","last_undegraded":"2024-06-03T09:06:46.463038+0000","last_fullsized":"2024-06-03T09:06:46.463038+0000","mapping_epoch":20,"log_start":"0'0","ondisk_log_start":"0'0","created":16,"last_epoch_clean":22,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-06-03T08:36:15.542635+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-06-03T08:36:15.542635+0000","last_clean_scrub_stamp":"2024-06-03T08:36:15.542635+0000","objects_scrubbed":0,"log_size":76,"log_dups_size":0,"ondisk_log_size":76,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-06-04T12:41:53.731360+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":459280,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":96,"num_read_kb":82,"num_write":113,"num_write_kb":1372,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2,"num_bytes_recovered":397840,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,5],"acting":[1,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"17.10","version":"0'0","reported_seq":947,"reported_epoch":1013,"state":"active+clean","last_fresh":"2024-06-03T09:06:44.974251+0000","last_change":"2024-06-03T08:38:15.399189+0000","last_active":"2024-06-03T09:06:44.974251+0000","last_peered":"2024-06-03T09:06:44.974251+0000","last_clean":"2024-06-03T09:06:44.974251+0000","last_became_active":"2024-06-03T08:38:15.337723+0000","last_became_peered":"2024-06-03T08:38:15.337723+0000","last_unstale":"2024-06-03T09:06:44.974251+0000","last_undegraded":"2024-06-03T09:06:44.974251+0000","last_fullsized":"2024-06-03T09:06:44.974251+0000","mapping_epoch":28,"log_start":"0'0","ondisk_log_start":"0'0","created":28,"last_epoch_clean":42,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-06-03T08:37:55.614497+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-06-03T08:37:55.614497+0000","last_clean_scrub_stamp":"2024-06-03T08:37:55.614497+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-06-04T09:29:46.427317+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,3],"acting":[4,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"17.1a","version":"0'0","reported_seq":971,"reported_epoch":1014,"state":"active+clean","last_fresh":"2024-06-03T09:06:46.393342+0000","last_change":"2024-06-03T09:06:46.393342+0000","last_active":"2024-06-03T09:06:46.393342+0000","last_peered":"2024-06-03T09:06:46.393342+0000","last_clean":"2024-06-03T09:06:46.393342+0000","last_became_active":"2024-06-03T09:06:46.386469+0000","last_became_peered":"2024-06-03T09:06:46.386469+0000","last_unstale":"2024-06-03T09:06:46.393342+0000","last_undegraded":"2024-06-03T09:06:46.393342+0000","last_fullsized":"2024-06-03T09:06:46.393342+0000","mapping_epoch":1013,"log_start":"0'0","ondisk_log_start":"0'0","created":28,"last_epoch_clean":1014,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-06-03T08:37:55.614497+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-06-03T08:37:55.614497+0000","last_clean_scrub_stamp":"2024-06-03T08:37:55.614497+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-06-04T13:22:45.876282+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,6],"acting":[4,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"17.1b","version":"0'0","reported_seq":939,"reported_epoch":1013,"state":"active+clean","last_fresh":"2024-06-03T09:06:45.136039+0000","last_change":"2024-06-03T08:38:03.115748+0000","last_active":"2024-06-03T09:06:45.136039+0000","last_peered":"2024-06-03T09:06:45.136039+0000","last_clean":"2024-06-03T09:06:45.136039+0000","last_became_active":"2024-06-03T08:38:03.110533+0000","last_became_peered":"2024-06-03T08:38:03.110533+0000","last_unstale":"2024-06-03T09:06:45.136039+0000","last_undegraded":"2024-06-03T09:06:45.136039+0000","last_fullsized":"2024-06-03T09:06:45.136039+0000","mapping_epoch":28,"log_start":"0'0","ondisk_log_start":"0'0","created":28,"last_epoch_clean":32,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-06-03T08:37:55.614497+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-06-03T08:37:55.614497+0000","last_clean_scrub_stamp":"2024-06-03T08:37:55.614497+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-06-04T12:36:47.471609+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,7],"acting":[3,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"17.18","version":"0'0","reported_seq":947,"reported_epoch":1013,"state":"active+clean","last_fresh":"2024-06-03T09:06:45.420817+0000","last_change":"2024-06-03T08:38:04.632864+0000","last_active":"2024-06-03T09:06:45.420817+0000","last_peered":"2024-06-03T09:06:45.420817+0000","last_clean":"2024-06-03T09:06:45.420817+0000","last_became_active":"2024-06-03T08:38:04.627856+0000","last_became_peered":"2024-06-03T08:38:04.627856+0000","last_unstale":"2024-06-03T09:06:45.420817+0000","last_undegraded":"2024-06-03T09:06:45.420817+0000","last_fullsized":"2024-06-03T09:06:45.420817+0000","mapping_epoch":28,"log_start":"0'0","ondisk_log_start":"0'0","created":28,"last_epoch_clean":33,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-06-03T08:37:55.614497+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-06-03T08:37:55.614497+0000","last_clean_scrub_stamp":"2024-06-03T08:37:55.614497+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-06-04T19:55:51.181598+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,4],"acting":[7,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"17.19","version":"0'0","reported_seq":938,"reported_epoch":1013,"state":"active+clean","last_fresh":"2024-06-03T09:06:45.129069+0000","last_change":"2024-06-03T08:38:01.839491+0000","last_active":"2024-06-03T09:06:45.129069+0000","last_peered":"2024-06-03T09:06:45.129069+0000","last_clean":"2024-06-03T09:06:45.129069+0000","last_became_active":"2024-06-03T08:38:01.829539+0000","last_became_peered":"2024-06-03T08:38:01.829539+0000","last_unstale":"2024-06-03T09:06:45.129069+0000","last_undegraded":"2024-06-03T09:06:45.129069+0000","last_fullsized":"2024-06-03T09:06:45.129069+0000","mapping_epoch":28,"log_start":"0'0","ondisk_log_start":"0'0","created":28,"last_epoch_clean":30,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-06-03T08:37:55.614497+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-06-03T08:37:55.614497+0000","last_clean_scrub_stamp":"2024-06-03T08:37:55.614497+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-06-04T09:10:43.860229+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,2],"acting":[3,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"17.1e","version":"0'0","reported_seq":844,"reported_epoch":1014,"state":"active+clean","last_fresh":"2024-06-03T09:06:46.297025+0000","last_change":"2024-06-03T09:06:46.297025+0000","last_active":"2024-06-03T09:06:46.297025+0000","last_peered":"2024-06-03T09:06:46.297025+0000","last_clean":"2024-06-03T09:06:46.297025+0000","last_became_active":"2024-06-03T09:06:46.288593+0000","last_became_peered":"2024-06-03T09:06:46.288593+0000","last_unstale":"2024-06-03T09:06:46.297025+0000","last_undegraded":"2024-06-03T09:06:46.297025+0000","last_fullsized":"2024-06-03T09:06:46.297025+0000","mapping_epoch":1013,"log_start":"0'0","ondisk_log_start":"0'0","created":28,"last_epoch_clean":1014,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-06-03T08:37:55.614497+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-06-03T08:37:55.614497+0000","last_clean_scrub_stamp":"2024-06-03T08:37:55.614497+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-06-04T20:09:30.144366+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,4],"acting":[6,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"17.1f","version":"0'0","reported_seq":944,"reported_epoch":1013,"state":"active+clean","last_fresh":"2024-06-03T09:06:45.933220+0000","last_change":"2024-06-03T08:38:04.383538+0000","last_active":"2024-06-03T09:06:45.933220+0000","last_peered":"2024-06-03T09:06:45.933220+0000","last_clean":"2024-06-03T09:06:45.933220+0000","last_became_active":"2024-06-03T08:38:04.367278+0000","last_became_peered":"2024-06-03T08:38:04.367278+0000","last_unstale":"2024-06-03T09:06:45.933220+0000","last_undegraded":"2024-06-03T09:06:45.933220+0000","last_fullsized":"2024-06-03T09:06:45.933220+0000","mapping_epoch":28,"log_start":"0'0","ondisk_log_start":"0'0","created":28,"last_epoch_clean":32,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-06-03T08:37:55.614497+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-06-03T08:37:55.614497+0000","last_clean_scrub_stamp":"2024-06-03T08:37:55.614497+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-06-04T10:39:26.552282+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,2],"acting":[0,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"17.1c","version":"0'0","reported_seq":956,"reported_epoch":1014,"state":"active+clean","last_fresh":"2024-06-03T09:06:46.410116+0000","last_change":"2024-06-03T09:06:46.410116+0000","last_active":"2024-06-03T09:06:46.410116+0000","last_peered":"2024-06-03T09:06:46.410116+0000","last_clean":"2024-06-03T09:06:46.410116+0000","last_became_active":"2024-06-03T09:06:46.403724+0000","last_became_peered":"2024-06-03T09:06:46.403724+0000","last_unstale":"2024-06-03T09:06:46.410116+0000","last_undegraded":"2024-06-03T09:06:46.410116+0000","last_fullsized":"2024-06-03T09:06:46.410116+0000","mapping_epoch":1013,"log_start":"0'0","ondisk_log_start":"0'0","created":28,"last_epoch_clean":1014,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-06-03T08:37:55.614497+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-06-03T08:37:55.614497+0000","last_clean_scrub_stamp":"2024-06-03T08:37:55.614497+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-06-04T19:40:47.504202+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,6],"acting":[5,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"17.1d","version":"0'0","reported_seq":927,"reported_epoch":1013,"state":"active+clean","last_fresh":"2024-06-03T09:06:45.879478+0000","last_change":"2024-06-03T08:38:06.794630+0000","last_active":"2024-06-03T09:06:45.879478+0000","last_peered":"2024-06-03T09:06:45.879478+0000","last_clean":"2024-06-03T09:06:45.879478+0000","last_became_active":"2024-06-03T08:38:06.592317+0000","last_became_peered":"2024-06-03T08:38:06.592317+0000","last_unstale":"2024-06-03T09:06:45.879478+0000","last_undegraded":"2024-06-03T09:06:45.879478+0000","last_fullsized":"2024-06-03T09:06:45.879478+0000","mapping_epoch":28,"log_start":"0'0","ondisk_log_start":"0'0","created":28,"last_epoch_clean":33,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-06-03T08:37:55.614497+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-06-03T08:37:55.614497+0000","last_clean_scrub_stamp":"2024-06-03T08:37:55.614497+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-06-04T12:33:30.785369+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,7],"acting":[1,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"17.2","version":"0'0","reported_seq":929,"reported_epoch":1013,"state":"active+clean","last_fresh":"2024-06-03T09:06:45.881178+0000","last_change":"2024-06-03T08:38:14.981410+0000","last_active":"2024-06-03T09:06:45.881178+0000","last_peered":"2024-06-03T09:06:45.881178+0000","last_clean":"2024-06-03T09:06:45.881178+0000","last_became_active":"2024-06-03T08:38:14.973825+0000","last_became_peered":"2024-06-03T08:38:14.973825+0000","last_unstale":"2024-06-03T09:06:45.881178+0000","last_undegraded":"2024-06-03T09:06:45.881178+0000","last_fullsized":"2024-06-03T09:06:45.881178+0000","mapping_epoch":28,"log_start":"0'0","ondisk_log_start":"0'0","created":28,"last_epoch_clean":41,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-06-03T08:37:55.614497+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-06-03T08:37:55.614497+0000","last_clean_scrub_stamp":"2024-06-03T08:37:55.614497+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-06-04T17:01:05.518613+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,4],"acting":[1,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"17.3","version":"0'0","reported_seq":946,"reported_epoch":1013,"state":"active+clean","last_fresh":"2024-06-03T09:06:45.934804+0000","last_change":"2024-06-03T08:38:14.619745+0000","last_active":"2024-06-03T09:06:45.934804+0000","last_peered":"2024-06-03T09:06:45.934804+0000","last_clean":"2024-06-03T09:06:45.934804+0000","last_became_active":"2024-06-03T08:38:14.510665+0000","last_became_peered":"2024-06-03T08:38:14.510665+0000","last_unstale":"2024-06-03T09:06:45.934804+0000","last_undegraded":"2024-06-03T09:06:45.934804+0000","last_fullsized":"2024-06-03T09:06:45.934804+0000","mapping_epoch":28,"log_start":"0'0","ondisk_log_start":"0'0","created":28,"last_epoch_clean":42,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-06-03T08:37:55.614497+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-06-03T08:37:55.614497+0000","last_clean_scrub_stamp":"2024-06-03T08:37:55.614497+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-06-04T12:03:47.392064+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,7],"acting":[0,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"17.0","version":"0'0","reported_seq":947,"reported_epoch":1013,"state":"active+clean","last_fresh":"2024-06-03T09:06:45.920467+0000","last_change":"2024-06-03T08:38:04.237118+0000","last_active":"2024-06-03T09:06:45.920467+0000","last_peered":"2024-06-03T09:06:45.920467+0000","last_clean":"2024-06-03T09:06:45.920467+0000","last_became_active":"2024-06-03T08:38:04.231954+0000","last_became_peered":"2024-06-03T08:38:04.231954+0000","last_unstale":"2024-06-03T09:06:45.920467+0000","last_undegraded":"2024-06-03T09:06:45.920467+0000","last_fullsized":"2024-06-03T09:06:45.920467+0000","mapping_epoch":28,"log_start":"0'0","ondisk_log_start":"0'0","created":28,"last_epoch_clean":32,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-06-03T08:37:55.614497+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-06-03T08:37:55.614497+0000","last_clean_scrub_stamp":"2024-06-03T08:37:55.614497+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-06-04T09:56:10.741637+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,2],"acting":[0,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"17.1","version":"65'7","reported_seq":971,"reported_epoch":1014,"state":"active+clean","last_fresh":"2024-06-03T09:06:46.389760+0000","last_change":"2024-06-03T09:06:46.389760+0000","last_active":"2024-06-03T09:06:46.389760+0000","last_peered":"2024-06-03T09:06:46.389760+0000","last_clean":"2024-06-03T09:06:46.389760+0000","last_became_active":"2024-06-03T09:06:46.383290+0000","last_became_peered":"2024-06-03T09:06:46.383290+0000","last_unstale":"2024-06-03T09:06:46.389760+0000","last_undegraded":"2024-06-03T09:06:46.389760+0000","last_fullsized":"2024-06-03T09:06:46.389760+0000","mapping_epoch":1013,"log_start":"0'0","ondisk_log_start":"0'0","created":28,"last_epoch_clean":1014,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-06-03T08:37:55.614497+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-06-03T08:37:55.614497+0000","last_clean_scrub_stamp":"2024-06-03T08:37:55.614497+0000","objects_scrubbed":0,"log_size":7,"log_dups_size":0,"ondisk_log_size":7,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-06-04T10:38:34.982254+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":5,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":6,"num_read_kb":6,"num_write":7,"num_write_kb":7,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,6],"acting":[5,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]}],"pool_stats":[{"poolid":2,"num_pg":8,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":16,"acting":16,"num_store_stats":7},{"poolid":1,"num_pg":1,"stat_sum":{"num_bytes":459280,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":96,"num_read_kb":82,"num_write":113,"num_write_kb":1372,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2,"num_bytes_recovered":397840,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":802816,"data_stored":795680,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":76,"ondisk_log_size":76,"up":2,"acting":2,"num_store_stats":2},{"poolid":17,"num_pg":32,"stat_sum":{"num_bytes":5,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":6,"num_read_kb":6,"num_write":7,"num_write_kb":7,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":8192,"data_stored":10,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":7,"ondisk_log_size":7,"up":64,"acting":64,"num_store_stats":8}],"osd_stats":[{"osd":6,"up_from":1013,"seq":4350801871163,"num_pgs":14,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":54924,"kb_used_data":5756,"kb_used_omap":1,"kb_used_meta":49150,"kb_avail":94316916,"statfs":{"total":96636764160,"available":96580521984,"internally_reserved":0,"allocated":5894144,"data_stored":2763078,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1593,"internal_metadata":50330055},"hb_peers":[0,1,2,3,4,5,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Mon Jun 3 09:06:42 2024","interfaces":[{"interface":"back","average":{"1min":5696.3109999999997,"5min":1469.3900000000001,"15min":503.11700000000002},"min":{"1min":5.242,"5min":4.4829999999999997,"15min":4.1959999999999997},"max":{"1min":20463.029999999999,"5min":22766.882000000001,"15min":22766.882000000001},"last":5.1619999999999999},{"interface":"front","average":{"1min":5687.6390000000001,"5min":1468.1669999999999,"15min":504.13200000000001},"min":{"1min":4.835,"5min":4.8129999999999997,"15min":4.4530000000000003},"max":{"1min":20461.830000000002,"5min":22766.289000000001,"15min":22766.289000000001},"last":9.1899999999999995}]},{"osd":1,"last update":"Mon Jun 3 09:06:42 2024","interfaces":[{"interface":"back","average":{"1min":36.496000000000002,"5min":80.162999999999997,"15min":134.59399999999999},"min":{"1min":5.9189999999999996,"5min":5.8799999999999999,"15min":5.7119999999999997},"max":{"1min":544.85799999999995,"5min":4138.1679999999997,"15min":9955.1299999999992},"last":12.109999999999999},{"interface":"front","average":{"1min":38.866999999999997,"5min":82.631,"15min":135.11500000000001},"min":{"1min":6.5350000000000001,"5min":6.5350000000000001,"15min":6.1390000000000002},"max":{"1min":546.73099999999999,"5min":4137.6530000000002,"15min":9827.5509999999995},"last":10.295999999999999}]},{"osd":2,"last update":"Mon Jun 3 09:06:42 2024","interfaces":[{"interface":"back","average":{"1min":41.093000000000004,"5min":27.917000000000002,"15min":22.329000000000001},"min":{"1min":6.0670000000000002,"5min":6.0670000000000002,"15min":6.0670000000000002},"max":{"1min":548.12900000000002,"5min":548.12900000000002,"15min":548.12900000000002},"last":6.452},{"interface":"front","average":{"1min":30.798999999999999,"5min":24.276,"15min":22.488},"min":{"1min":6.4370000000000003,"5min":6.4370000000000003,"15min":6.1870000000000003},"max":{"1min":369.14499999999998,"5min":369.14499999999998,"15min":434.77300000000002},"last":11.317}]},{"osd":3,"last update":"Mon Jun 3 09:06:42 2024","interfaces":[{"interface":"back","average":{"1min":41.411999999999999,"5min":27.434999999999999,"15min":30.657},"min":{"1min":10.061,"5min":8.3179999999999996,"15min":6.1719999999999997},"max":{"1min":549.024,"5min":549.024,"15min":1235.5799999999999},"last":9.7409999999999997},{"interface":"front","average":{"1min":37.850999999999999,"5min":22.001000000000001,"15min":28.015999999999998},"min":{"1min":6.0679999999999996,"5min":6.0679999999999996,"15min":6.0679999999999996},"max":{"1min":559.25,"5min":559.25,"15min":1157.6780000000001},"last":7.0650000000000004}]},{"osd":4,"last update":"Mon Jun 3 09:06:42 2024","interfaces":[{"interface":"back","average":{"1min":165.62899999999999,"5min":1281.1010000000001,"15min":440.59199999999998},"min":{"1min":10.965999999999999,"5min":6.093,"15min":6.093},"max":{"1min":384.46899999999999,"5min":28093.096000000001,"15min":28093.096000000001},"last":12.743},{"interface":"front","average":{"1min":186.97900000000001,"5min":1308.886,"15min":454.97699999999998},"min":{"1min":13.581,"5min":7.7480000000000002,"15min":7.7480000000000002},"max":{"1min":445.096,"5min":28323.837,"15min":28323.837},"last":14.276}]},{"osd":5,"last update":"Mon Jun 3 09:06:42 2024","interfaces":[{"interface":"back","average":{"1min":81.581000000000003,"5min":34.863,"15min":28.350999999999999},"min":{"1min":10.345000000000001,"5min":8.625,"15min":7.516},"max":{"1min":455.84199999999998,"5min":455.84199999999998,"15min":455.84199999999998},"last":13.510999999999999},{"interface":"front","average":{"1min":83.391000000000005,"5min":39.277999999999999,"15min":27.885000000000002},"min":{"1min":8.0280000000000005,"5min":6.266,"15min":6.266},"max":{"1min":550.02300000000002,"5min":550.02300000000002,"15min":550.02300000000002},"last":15.76}]},{"osd":7,"last update":"Mon Jun 3 09:06:42 2024","interfaces":[{"interface":"back","average":{"1min":44.052999999999997,"5min":33.697000000000003,"15min":29.731999999999999},"min":{"1min":9.1500000000000004,"5min":9.1500000000000004,"15min":9.1500000000000004},"max":{"1min":552.09500000000003,"5min":552.09500000000003,"15min":552.09500000000003},"last":14.996},{"interface":"front","average":{"1min":47.076999999999998,"5min":34.226999999999997,"15min":29.827000000000002},"min":{"1min":9.7780000000000005,"5min":8.6709999999999994,"15min":8.5820000000000007},"max":{"1min":531.08600000000001,"5min":531.08600000000001,"15min":531.08600000000001},"last":7.5579999999999998}]}]},{"osd":7,"up_from":20,"seq":85899346291,"num_pgs":9,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":59020,"kb_used_data":5752,"kb_used_omap":1,"kb_used_meta":53246,"kb_avail":94312820,"statfs":{"total":96636764160,"available":96576327680,"internally_reserved":0,"allocated":5890048,"data_stored":2763073,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1589,"internal_metadata":54524363},"hb_peers":[0,1,2,3,4,5,6],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":10,"apply_latency_ms":10,"commit_latency_ns":10000000,"apply_latency_ns":10000000},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Mon Jun 3 09:06:09 2024","interfaces":[{"interface":"back","average":{"1min":14.073,"5min":894.22400000000005,"15min":308.93400000000003},"min":{"1min":5.5709999999999997,"5min":4.218,"15min":4.0599999999999996},"max":{"1min":90.757999999999996,"5min":21729.001,"15min":21729.001},"last":11.026999999999999},{"interface":"front","average":{"1min":9.2550000000000008,"5min":906.23500000000001,"15min":314.06},"min":{"1min":5.5789999999999997,"5min":4.8810000000000002,"15min":4.8810000000000002},"max":{"1min":25.937000000000001,"5min":21899.732,"15min":21899.732},"last":6.4160000000000004}]},{"osd":1,"last update":"Mon Jun 3 09:06:09 2024","interfaces":[{"interface":"back","average":{"1min":15.182,"5min":69.183999999999997,"15min":219.173},"min":{"1min":7.3639999999999999,"5min":6.6200000000000001,"15min":6.1319999999999997},"max":{"1min":76.926000000000002,"5min":2795.1329999999998,"15min":18373.257000000001},"last":7.4720000000000004},{"interface":"front","average":{"1min":21.495999999999999,"5min":74.703999999999994,"15min":225.285},"min":{"1min":8.8399999999999999,"5min":8.8399999999999999,"15min":8.5129999999999999},"max":{"1min":94.977999999999994,"5min":2803.3670000000002,"15min":18374.601999999999},"last":15.045}]},{"osd":2,"last update":"Mon Jun 3 09:06:09 2024","interfaces":[{"interface":"back","average":{"1min":19.164000000000001,"5min":26.902000000000001,"15min":22.920999999999999},"min":{"1min":11.207000000000001,"5min":8.5030000000000001,"15min":8.5030000000000001},"max":{"1min":91.319999999999993,"5min":333.99200000000002,"15min":333.99200000000002},"last":10.000999999999999},{"interface":"front","average":{"1min":11.125999999999999,"5min":18.132000000000001,"15min":16.353000000000002},"min":{"1min":6.202,"5min":5.8369999999999997,"15min":5.5069999999999997},"max":{"1min":28.649000000000001,"5min":226.18100000000001,"15min":226.18100000000001},"last":8.452}]},{"osd":3,"last update":"Mon Jun 3 09:06:09 2024","interfaces":[{"interface":"back","average":{"1min":22.291,"5min":26.289999999999999,"15min":31.074999999999999},"min":{"1min":11.714,"5min":6.8959999999999999,"15min":6.3760000000000003},"max":{"1min":83.102000000000004,"5min":316.64400000000001,"15min":2039.9059999999999},"last":12.130000000000001},{"interface":"front","average":{"1min":13.324999999999999,"5min":17.963000000000001,"15min":25.969999999999999},"min":{"1min":6.1200000000000001,"5min":5.2089999999999996,"15min":5.2089999999999996},"max":{"1min":49.908999999999999,"5min":233.542,"15min":2037.5550000000001},"last":9.4380000000000006}]},{"osd":4,"last update":"Mon Jun 3 09:06:29 2024","interfaces":[{"interface":"back","average":{"1min":17.675999999999998,"5min":1026.325,"15min":472.47199999999998},"min":{"1min":7.306,"5min":5.9089999999999998,"15min":5.9089999999999998},"max":{"1min":140.03299999999999,"5min":28969.369999999999,"15min":30073.917000000001},"last":13.851000000000001},{"interface":"front","average":{"1min":17.890999999999998,"5min":1044.5139999999999,"15min":480.226},"min":{"1min":5.9279999999999999,"5min":5.9279999999999999,"15min":5.9279999999999999},"max":{"1min":141.179,"5min":29244.138999999999,"15min":30348.021000000001},"last":15.601000000000001}]},{"osd":5,"last update":"Mon Jun 3 09:06:02 2024","interfaces":[{"interface":"back","average":{"1min":24.372,"5min":26.856000000000002,"15min":23.571999999999999},"min":{"1min":12.278,"5min":9.2460000000000004,"15min":6.5650000000000004},"max":{"1min":183.64099999999999,"5min":317.572,"15min":317.572},"last":16.84},{"interface":"front","average":{"1min":21.259,"5min":23.364000000000001,"15min":21.193999999999999},"min":{"1min":7.8499999999999996,"5min":7.0999999999999996,"15min":7.008},"max":{"1min":184.53399999999999,"5min":306.60599999999999,"15min":306.60599999999999},"last":14.491}]},{"osd":6,"last update":"Mon Jun 3 09:04:31 2024","interfaces":[{"interface":"back","average":{"1min":44.061,"5min":26.821000000000002,"15min":91.176000000000002},"min":{"1min":9.0800000000000001,"5min":7.9390000000000001,"15min":7.9390000000000001},"max":{"1min":268.029,"5min":308.916,"15min":8734.4639999999999},"last":10.551},{"interface":"front","average":{"1min":46.643000000000001,"5min":23.879999999999999,"15min":88.765000000000001},"min":{"1min":8.8119999999999994,"5min":6.9240000000000004,"15min":6.742},"max":{"1min":305.64600000000002,"5min":305.64600000000002,"15min":8735.5319999999992},"last":12.933}]}]},{"osd":5,"up_from":20,"seq":85899346266,"num_pgs":9,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":59416,"kb_used_data":6148,"kb_used_omap":1,"kb_used_meta":53246,"kb_avail":94312424,"statfs":{"total":96636764160,"available":96575922176,"internally_reserved":0,"allocated":6295552,"data_stored":3160918,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1594,"internal_metadata":54524358},"hb_peers":[0,1,2,3,4,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Mon Jun 3 09:06:06 2024","interfaces":[{"interface":"back","average":{"1min":7.468,"5min":1703.037,"15min":580.51900000000001},"min":{"1min":4.6509999999999998,"5min":4.6509999999999998,"15min":4.4020000000000001},"max":{"1min":14.711,"5min":26433.764999999999,"15min":26433.764999999999},"last":16.231999999999999},{"interface":"front","average":{"1min":9.9429999999999996,"5min":1725.27,"15min":587.87800000000004},"min":{"1min":5.5999999999999996,"5min":5.1319999999999997,"15min":4.8170000000000002},"max":{"1min":13.590999999999999,"5min":26704.698,"15min":26704.698},"last":10.154}]},{"osd":1,"last update":"Mon Jun 3 09:06:00 2024","interfaces":[{"interface":"back","average":{"1min":10.862,"5min":67.731999999999999,"15min":217.364},"min":{"1min":7.1769999999999996,"5min":6.0890000000000004,"15min":5.4930000000000003},"max":{"1min":16.998999999999999,"5min":3923.4090000000001,"15min":18811.696},"last":10.711},{"interface":"front","average":{"1min":10.417,"5min":65.141999999999996,"15min":220.75700000000001},"min":{"1min":5.3209999999999997,"5min":5.3209999999999997,"15min":5.0209999999999999},"max":{"1min":15.821999999999999,"5min":4019.6129999999998,"15min":18968.365000000002},"last":14.157999999999999}]},{"osd":2,"last update":"Mon Jun 3 09:06:00 2024","interfaces":[{"interface":"back","average":{"1min":11.366,"5min":24.18,"15min":21.137},"min":{"1min":5.6369999999999996,"5min":5.6369999999999996,"15min":5.0510000000000002},"max":{"1min":15.529999999999999,"5min":332.48200000000003,"15min":332.48200000000003},"last":11.706},{"interface":"front","average":{"1min":10.089,"5min":23.117000000000001,"15min":24.050000000000001},"min":{"1min":6.6769999999999996,"5min":6.6769999999999996,"15min":5.8739999999999997},"max":{"1min":15.821,"5min":353.03199999999998,"15min":353.03199999999998},"last":6.9550000000000001}]},{"osd":3,"last update":"Mon Jun 3 09:06:00 2024","interfaces":[{"interface":"back","average":{"1min":13.472,"5min":24.283999999999999,"15min":31.666},"min":{"1min":10.111000000000001,"5min":7.4720000000000004,"15min":7.4210000000000003},"max":{"1min":16.271999999999998,"5min":353.57499999999999,"15min":2855.8519999999999},"last":12.326000000000001},{"interface":"front","average":{"1min":9.5899999999999999,"5min":15.875,"15min":25.834},"min":{"1min":6.6609999999999996,"5min":5.7729999999999997,"15min":5.1909999999999998},"max":{"1min":15.112,"5min":203.804,"15min":2869.7860000000001},"last":8.3740000000000006}]},{"osd":4,"last update":"Mon Jun 3 09:06:29 2024","interfaces":[{"interface":"back","average":{"1min":11.067,"5min":780.21000000000004,"15min":377.28399999999999},"min":{"1min":5.4779999999999998,"5min":5.4779999999999998,"15min":5.4779999999999998},"max":{"1min":14.654999999999999,"5min":23315.438999999998,"15min":28619.825000000001},"last":9.1669999999999998},{"interface":"front","average":{"1min":14.349,"5min":782.97299999999996,"15min":377.93400000000003},"min":{"1min":7.492,"5min":6.5030000000000001,"15min":6.5030000000000001},"max":{"1min":17.707000000000001,"5min":23317.502,"15min":28621.294000000002},"last":12.898999999999999}]},{"osd":6,"last update":"Mon Jun 3 09:03:57 2024","interfaces":[{"interface":"back","average":{"1min":23.536000000000001,"5min":36.457000000000001,"15min":83.799999999999997},"min":{"1min":9.4800000000000004,"5min":9.4800000000000004,"15min":8.0630000000000006},"max":{"1min":89.442999999999998,"5min":508.77999999999997,"15min":9193.5169999999998},"last":15.467000000000001},{"interface":"front","average":{"1min":18.891999999999999,"5min":22.100999999999999,"15min":75.489999999999995},"min":{"1min":5.923,"5min":5.923,"15min":5.923},"max":{"1min":78.424999999999997,"5min":464.62,"15min":9191.4429999999993},"last":13.491}]},{"osd":7,"last update":"Mon Jun 3 09:06:00 2024","interfaces":[{"interface":"back","average":{"1min":13.257,"5min":20.917999999999999,"15min":21.088999999999999},"min":{"1min":10.475,"5min":6.7629999999999999,"15min":6.7629999999999999},"max":{"1min":17.245000000000001,"5min":211.22999999999999,"15min":211.22999999999999},"last":16.960999999999999},{"interface":"front","average":{"1min":12.872999999999999,"5min":20.587,"15min":24.021000000000001},"min":{"1min":8.5760000000000005,"5min":7.7709999999999999,"15min":6.8570000000000002},"max":{"1min":18.359999999999999,"5min":223.191,"15min":286.69499999999999},"last":7.8499999999999996}]}]},{"osd":4,"up_from":20,"seq":85899346273,"num_pgs":12,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":63112,"kb_used_data":5752,"kb_used_omap":1,"kb_used_meta":57342,"kb_avail":94308728,"statfs":{"total":96636764160,"available":96572137472,"internally_reserved":0,"allocated":5890048,"data_stored":2763073,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1594,"internal_metadata":58718662},"hb_peers":[0,1,2,3,5,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":20,"apply_latency_ms":20,"commit_latency_ns":20000000,"apply_latency_ns":20000000},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Mon Jun 3 09:06:35 2024","interfaces":[{"interface":"back","average":{"1min":10.417,"5min":1093.547,"15min":381.64499999999998},"min":{"1min":4.6900000000000004,"5min":4.5279999999999996,"15min":4.3220000000000001},"max":{"1min":52.241,"5min":21726.743999999999,"15min":21726.743999999999},"last":5.0679999999999996},{"interface":"front","average":{"1min":13.175000000000001,"5min":1102.4200000000001,"15min":387.161},"min":{"1min":6.1950000000000003,"5min":5.8380000000000001,"15min":5.4909999999999997},"max":{"1min":53.148000000000003,"5min":21727.993999999999,"15min":21727.993999999999},"last":21.422000000000001}]},{"osd":1,"last update":"Mon Jun 3 09:06:35 2024","interfaces":[{"interface":"back","average":{"1min":9.9909999999999997,"5min":53.250999999999998,"15min":340.02999999999997},"min":{"1min":4.5670000000000002,"5min":4.1970000000000001,"15min":4.1970000000000001},"max":{"1min":47.073,"5min":3346.4389999999999,"15min":22256.098000000002},"last":6.5800000000000001},{"interface":"front","average":{"1min":12.189,"5min":53.375999999999998,"15min":339.68799999999999},"min":{"1min":6.1539999999999999,"5min":5.0590000000000002,"15min":4.2640000000000002},"max":{"1min":49.558,"5min":3347.181,"15min":22247.317999999999},"last":26.995000000000001}]},{"osd":2,"last update":"Mon Jun 3 09:06:35 2024","interfaces":[{"interface":"back","average":{"1min":15.178000000000001,"5min":17.202000000000002,"15min":20.417999999999999},"min":{"1min":6.3010000000000002,"5min":6.3010000000000002,"15min":6.3010000000000002},"max":{"1min":85.340000000000003,"5min":337.60399999999998,"15min":393.58600000000001},"last":25.855},{"interface":"front","average":{"1min":15.423,"5min":14.432,"15min":22.754000000000001},"min":{"1min":6.4889999999999999,"5min":6.0629999999999997,"15min":6.0629999999999997},"max":{"1min":82.120000000000005,"5min":127.913,"15min":394.471},"last":26.456}]},{"osd":3,"last update":"Mon Jun 3 09:06:35 2024","interfaces":[{"interface":"back","average":{"1min":17.651,"5min":15.5,"15min":18.079000000000001},"min":{"1min":10.359999999999999,"5min":5.8719999999999999,"15min":5.4000000000000004},"max":{"1min":84.555999999999997,"5min":127.43600000000001,"15min":322.63},"last":29.57},{"interface":"front","average":{"1min":16.413,"5min":17.356999999999999,"15min":22.920000000000002},"min":{"1min":9.7349999999999994,"5min":7.484,"15min":7.484},"max":{"1min":71.394000000000005,"5min":204.517,"15min":392.67500000000001},"last":28.472000000000001}]},{"osd":5,"last update":"Mon Jun 3 09:06:35 2024","interfaces":[{"interface":"back","average":{"1min":15.672000000000001,"5min":16.742999999999999,"15min":20.795000000000002},"min":{"1min":8.093,"5min":6.7489999999999997,"15min":6.7489999999999997},"max":{"1min":51.326000000000001,"5min":268.10199999999998,"15min":344.23399999999998},"last":30.233000000000001},{"interface":"front","average":{"1min":17.297999999999998,"5min":19.356999999999999,"15min":25.713999999999999},"min":{"1min":12.875,"5min":8.5180000000000007,"15min":8.5180000000000007},"max":{"1min":40.829000000000001,"5min":264.75,"15min":345.18700000000001},"last":31.635999999999999}]},{"osd":6,"last update":"Mon Jun 3 09:04:32 2024","interfaces":[{"interface":"back","average":{"1min":15.548,"5min":32.957999999999998,"15min":152.738},"min":{"1min":9.141,"5min":9.141,"15min":8.2319999999999993},"max":{"1min":52.57,"5min":395.238,"15min":12496.335999999999},"last":208.90000000000001},{"interface":"front","average":{"1min":12.680999999999999,"5min":25.428000000000001,"15min":149.30600000000001},"min":{"1min":6.4379999999999997,"5min":5.6319999999999997,"15min":5.6319999999999997},"max":{"1min":46.283000000000001,"5min":377.74299999999999,"15min":12616.297},"last":205.88900000000001}]},{"osd":7,"last update":"Mon Jun 3 09:06:35 2024","interfaces":[{"interface":"back","average":{"1min":9.6969999999999992,"5min":14.196,"15min":18.809999999999999},"min":{"1min":6.7190000000000003,"5min":6.2110000000000003,"15min":5.8010000000000002},"max":{"1min":15.960000000000001,"5min":207.25299999999999,"15min":358.94900000000001},"last":5.7249999999999996},{"interface":"front","average":{"1min":17.696000000000002,"5min":19.024999999999999,"15min":23.259},"min":{"1min":11.794,"5min":7.7789999999999999,"15min":7.7789999999999999},"max":{"1min":45.639000000000003,"5min":209.32900000000001,"15min":380.06900000000002},"last":30.835000000000001}]}]},{"osd":2,"up_from":20,"seq":85899346293,"num_pgs":11,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":59020,"kb_used_data":5752,"kb_used_omap":1,"kb_used_meta":53246,"kb_avail":94312820,"statfs":{"total":96636764160,"available":96576327680,"internally_reserved":0,"allocated":5890048,"data_stored":2763073,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1594,"internal_metadata":54524358},"hb_peers":[0,1,3,4,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":14,"apply_latency_ms":14,"commit_latency_ns":14000000,"apply_latency_ns":14000000},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Mon Jun 3 09:06:23 2024","interfaces":[{"interface":"back","average":{"1min":8.9629999999999992,"5min":946.25400000000002,"15min":326.36099999999999},"min":{"1min":5.3479999999999999,"5min":4.7050000000000001,"15min":4.1420000000000003},"max":{"1min":26.015000000000001,"5min":23465.18,"15min":23465.18},"last":8.6600000000000001},{"interface":"front","average":{"1min":7.6130000000000004,"5min":944.39800000000002,"15min":326.178},"min":{"1min":4.6790000000000003,"5min":4.6790000000000003,"15min":4.6790000000000003},"max":{"1min":20.802,"5min":23465.966,"15min":23465.966},"last":5.6900000000000004}]},{"osd":1,"last update":"Mon Jun 3 09:06:23 2024","interfaces":[{"interface":"back","average":{"1min":13.007,"5min":76.507000000000005,"15min":294.72899999999998},"min":{"1min":6.4089999999999998,"5min":5.4199999999999999,"15min":5.4199999999999999},"max":{"1min":85.430000000000007,"5min":4220.5290000000005,"15min":18954.987000000001},"last":7.6230000000000002},{"interface":"front","average":{"1min":15.894,"5min":82.119,"15min":302.92399999999998},"min":{"1min":7.367,"5min":7.367,"15min":7.367},"max":{"1min":98.498000000000005,"5min":4325.9489999999996,"15min":19143.834999999999},"last":12.243}]},{"osd":3,"last update":"Mon Jun 3 09:06:23 2024","interfaces":[{"interface":"back","average":{"1min":9.7579999999999991,"5min":12.444000000000001,"15min":28.065000000000001},"min":{"1min":7,"5min":5.9290000000000003,"15min":5.2450000000000001},"max":{"1min":28.164000000000001,"5min":118.913,"15min":2594.8150000000001},"last":9.3569999999999993},{"interface":"front","average":{"1min":12.321999999999999,"5min":15.25,"15min":31.349},"min":{"1min":9.1240000000000006,"5min":8.0760000000000005,"15min":6.0780000000000003},"max":{"1min":32.360999999999997,"5min":156.34700000000001,"15min":2706.5120000000002},"last":11.468999999999999}]},{"osd":4,"last update":"Mon Jun 3 09:05:57 2024","interfaces":[{"interface":"back","average":{"1min":10.573,"5min":1266.894,"15min":438.154},"min":{"1min":8.6379999999999999,"5min":7.8789999999999996,"15min":6.2039999999999997},"max":{"1min":15.105,"5min":26660.861000000001,"15min":26660.861000000001},"last":10.225},{"interface":"front","average":{"1min":12.407999999999999,"5min":1266.7460000000001,"15min":435.75400000000002},"min":{"1min":10.266999999999999,"5min":10.266999999999999,"15min":6.2720000000000002},"max":{"1min":15.228,"5min":26616.243999999999,"15min":26616.243999999999},"last":15.326000000000001}]},{"osd":5,"last update":"Mon Jun 3 09:06:23 2024","interfaces":[{"interface":"back","average":{"1min":14.103,"5min":18.091000000000001,"15min":21.119},"min":{"1min":9.6349999999999998,"5min":9.2159999999999993,"15min":6.2050000000000001},"max":{"1min":32.890000000000001,"5min":252.31100000000001,"15min":252.31100000000001},"last":23.541},{"interface":"front","average":{"1min":11.545999999999999,"5min":16.170999999999999,"15min":20.254000000000001},"min":{"1min":7.6070000000000002,"5min":7.4850000000000003,"15min":6.8810000000000002},"max":{"1min":29.228999999999999,"5min":258.31700000000001,"15min":258.31700000000001},"last":12.821999999999999}]},{"osd":6,"last update":"Mon Jun 3 09:04:28 2024","interfaces":[{"interface":"back","average":{"1min":14.063000000000001,"5min":24.850000000000001,"15min":106.755},"min":{"1min":9.0820000000000007,"5min":8.9649999999999999,"15min":8.9649999999999999},"max":{"1min":24.666,"5min":188.06700000000001,"15min":9521.0010000000002},"last":39.628999999999998},{"interface":"front","average":{"1min":13.933999999999999,"5min":21.081,"15min":106.142},"min":{"1min":5.8719999999999999,"5min":5.8719999999999999,"15min":5.8719999999999999},"max":{"1min":117.551,"5min":193.887,"15min":9585.6620000000003},"last":38.316000000000003}]},{"osd":7,"last update":"Mon Jun 3 09:06:23 2024","interfaces":[{"interface":"back","average":{"1min":17.056000000000001,"5min":20.055,"15min":23.268999999999998},"min":{"1min":10.353999999999999,"5min":6.6130000000000004,"15min":6.6130000000000004},"max":{"1min":88.974999999999994,"5min":242.56800000000001,"15min":242.56800000000001},"last":16.387},{"interface":"front","average":{"1min":18.885000000000002,"5min":21.460000000000001,"15min":25.489999999999998},"min":{"1min":12.497,"5min":9.0660000000000007,"15min":9.0660000000000007},"max":{"1min":102.229,"5min":181.56399999999999,"15min":214.33799999999999},"last":22.489000000000001}]}]},{"osd":3,"up_from":14,"seq":60129542507,"num_pgs":11,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":59016,"kb_used_data":5752,"kb_used_omap":1,"kb_used_meta":53246,"kb_avail":94312824,"statfs":{"total":96636764160,"available":96576331776,"internally_reserved":0,"allocated":5890048,"data_stored":2763073,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1592,"internal_metadata":54524360},"hb_peers":[0,1,2,4,5,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":13,"apply_latency_ms":13,"commit_latency_ns":13000000,"apply_latency_ns":13000000},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Mon Jun 3 09:06:25 2024","interfaces":[{"interface":"back","average":{"1min":7.79,"5min":1060.1959999999999,"15min":367.01799999999997},"min":{"1min":5.6909999999999998,"5min":5.173,"15min":4.6929999999999996},"max":{"1min":10.59,"5min":23275.167000000001,"15min":23275.167000000001},"last":5.4189999999999996},{"interface":"front","average":{"1min":11.215999999999999,"5min":1066.5799999999999,"15min":372.476},"min":{"1min":8.7080000000000002,"5min":7.2510000000000003,"15min":5.9880000000000004},"max":{"1min":13.499000000000001,"5min":23278.536,"15min":23278.536},"last":11.593999999999999}]},{"osd":1,"last update":"Mon Jun 3 09:06:25 2024","interfaces":[{"interface":"back","average":{"1min":7.3760000000000003,"5min":15.614000000000001,"15min":389.83100000000002},"min":{"1min":4.335,"5min":4.335,"15min":4.335},"max":{"1min":11.289,"5min":250.72300000000001,"15min":21658.205999999998},"last":6.6079999999999997},{"interface":"front","average":{"1min":8.9499999999999993,"5min":14.135,"15min":381.10300000000001},"min":{"1min":6.5609999999999999,"5min":5.4100000000000001,"15min":5.2779999999999996},"max":{"1min":14.51,"5min":180.44499999999999,"15min":21307.069},"last":12.449}]},{"osd":2,"last update":"Mon Jun 3 09:06:25 2024","interfaces":[{"interface":"back","average":{"1min":12.343999999999999,"5min":22.059999999999999,"15min":23.715},"min":{"1min":7.9199999999999999,"5min":6.4900000000000002,"15min":6.056},"max":{"1min":15.601000000000001,"5min":377.803,"15min":377.803},"last":14.433999999999999},{"interface":"front","average":{"1min":10.319000000000001,"5min":19.364999999999998,"15min":20.242999999999999},"min":{"1min":6.891,"5min":6.1749999999999998,"15min":5.9020000000000001},"max":{"1min":15.432,"5min":371.601,"15min":371.601},"last":7.6109999999999998}]},{"osd":4,"last update":"Mon Jun 3 09:06:31 2024","interfaces":[{"interface":"back","average":{"1min":9.7460000000000004,"5min":891.23000000000002,"15min":416.48099999999999},"min":{"1min":5.6539999999999999,"5min":5.6539999999999999,"15min":5.6539999999999999},"max":{"1min":15.445,"5min":29040.635999999999,"15min":30144.939999999999},"last":8.266},{"interface":"front","average":{"1min":12.398999999999999,"5min":897.58500000000004,"15min":421.27499999999998},"min":{"1min":8.9469999999999992,"5min":7.8109999999999999,"15min":6.2439999999999998},"max":{"1min":16.518000000000001,"5min":29076.955999999998,"15min":30180.689999999999},"last":13.709}]},{"osd":5,"last update":"Mon Jun 3 09:06:07 2024","interfaces":[{"interface":"back","average":{"1min":12.802,"5min":19.640999999999998,"15min":22.318000000000001},"min":{"1min":9.0890000000000004,"5min":6.8810000000000002,"15min":6.7690000000000001},"max":{"1min":16.745999999999999,"5min":249.88,"15min":327.34899999999999},"last":15.243},{"interface":"front","average":{"1min":15.002000000000001,"5min":24.003,"15min":26.649999999999999},"min":{"1min":11.984,"5min":9.9149999999999991,"15min":8.9879999999999995},"max":{"1min":18.407,"5min":251.47800000000001,"15min":331.86799999999999},"last":25.289999999999999}]},{"osd":6,"last update":"Mon Jun 3 09:04:04 2024","interfaces":[{"interface":"back","average":{"1min":22.815999999999999,"5min":31.513999999999999,"15min":181.74299999999999},"min":{"1min":9.4809999999999999,"5min":7.2240000000000002,"15min":7.2240000000000002},"max":{"1min":160.56999999999999,"5min":380.613,"15min":11893.74},"last":19.359999999999999},{"interface":"front","average":{"1min":25.359999999999999,"5min":30.981000000000002,"15min":179.81399999999999},"min":{"1min":9.8870000000000005,"5min":7.3360000000000003,"15min":6.0899999999999999},"max":{"1min":193.11099999999999,"5min":379.80700000000002,"15min":11893.069},"last":8.202}]},{"osd":7,"last update":"Mon Jun 3 09:06:25 2024","interfaces":[{"interface":"back","average":{"1min":11.321999999999999,"5min":18.443999999999999,"15min":21.568999999999999},"min":{"1min":8.109,"5min":8.109,"15min":7.4279999999999999},"max":{"1min":17.113,"5min":146.267,"15min":178.76300000000001},"last":9.8670000000000009},{"interface":"front","average":{"1min":12.512,"5min":20.065999999999999,"15min":24.190000000000001},"min":{"1min":7.4989999999999997,"5min":7.4989999999999997,"15min":7.4989999999999997},"max":{"1min":18.047999999999998,"5min":148.13800000000001,"15min":417.22800000000001},"last":10.81}]}]},{"osd":1,"up_from":14,"seq":60129542505,"num_pgs":6,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":59408,"kb_used_data":6144,"kb_used_omap":1,"kb_used_meta":53246,"kb_avail":94312432,"statfs":{"total":96636764160,"available":96575930368,"internally_reserved":0,"allocated":6291456,"data_stored":3160913,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":54524357},"hb_peers":[0,2,3,4,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":5,"apply_latency_ms":5,"commit_latency_ns":5000000,"apply_latency_ns":5000000},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Mon Jun 3 09:06:07 2024","interfaces":[{"interface":"back","average":{"1min":7.7530000000000001,"5min":185.66999999999999,"15min":77.347999999999999},"min":{"1min":5.0229999999999997,"5min":4.9279999999999999,"15min":4.5449999999999999},"max":{"1min":11.896000000000001,"5min":7470.616,"15min":7470.616},"last":16.219000000000001},{"interface":"front","average":{"1min":9.9220000000000006,"5min":187.14599999999999,"15min":76.325999999999993},"min":{"1min":5.3159999999999998,"5min":5.3159999999999998,"15min":5.3159999999999998},"max":{"1min":18.338000000000001,"5min":7477.0829999999996,"15min":7477.0829999999996},"last":16.843}]},{"osd":2,"last update":"Mon Jun 3 09:06:20 2024","interfaces":[{"interface":"back","average":{"1min":10.340999999999999,"5min":17.369,"15min":23.670999999999999},"min":{"1min":6.9740000000000002,"5min":5.8650000000000002,"15min":5.8650000000000002},"max":{"1min":15.452,"5min":228,"15min":657.38},"last":8.1440000000000001},{"interface":"front","average":{"1min":13.387,"5min":23.355,"15min":31.524000000000001},"min":{"1min":9.077,"5min":9.077,"15min":7.2160000000000002},"max":{"1min":19.701000000000001,"5min":229.465,"15min":700.06500000000005},"last":14.17}]},{"osd":3,"last update":"Mon Jun 3 09:06:04 2024","interfaces":[{"interface":"back","average":{"1min":8.6110000000000007,"5min":15.733000000000001,"15min":26.646999999999998},"min":{"1min":5.827,"5min":5.4640000000000004,"15min":5.4640000000000004},"max":{"1min":13.105,"5min":121.619,"15min":1676.009},"last":10.58},{"interface":"front","average":{"1min":9.5850000000000009,"5min":16.161999999999999,"15min":29.065999999999999},"min":{"1min":6.3630000000000004,"5min":5.7699999999999996,"15min":5.7699999999999996},"max":{"1min":15.166,"5min":132.036,"15min":1766.5840000000001},"last":7.21}]},{"osd":4,"last update":"Mon Jun 3 09:06:35 2024","interfaces":[{"interface":"back","average":{"1min":8.6400000000000006,"5min":641.85500000000002,"15min":334.71499999999997},"min":{"1min":6.7640000000000002,"5min":6.1760000000000002,"15min":6.0970000000000004},"max":{"1min":11.811,"5min":26340.968000000001,"15min":28044.862000000001},"last":8.8770000000000007},{"interface":"front","average":{"1min":12.361000000000001,"5min":648.59900000000005,"15min":338.48500000000001},"min":{"1min":9.8949999999999996,"5min":9.3940000000000001,"15min":6.4640000000000004},"max":{"1min":15.507,"5min":26342.311000000002,"15min":28045.624},"last":13.067}]},{"osd":5,"last update":"Mon Jun 3 09:06:04 2024","interfaces":[{"interface":"back","average":{"1min":12.478999999999999,"5min":20.228000000000002,"15min":25.997},"min":{"1min":8.6430000000000007,"5min":7.9720000000000004,"15min":7.234},"max":{"1min":17.442,"5min":139.43199999999999,"15min":701.22000000000003},"last":11.135},{"interface":"front","average":{"1min":13.507999999999999,"5min":25.760999999999999,"15min":27.553999999999998},"min":{"1min":10.042999999999999,"5min":7.258,"15min":6.242},"max":{"1min":20.611000000000001,"5min":278.63,"15min":598.35000000000002},"last":9.6969999999999992}]},{"osd":6,"last update":"Mon Jun 3 09:03:54 2024","interfaces":[{"interface":"back","average":{"1min":28.524000000000001,"5min":20.654,"15min":23.878},"min":{"1min":6.0309999999999997,"5min":6.0309999999999997,"15min":6.0309999999999997},"max":{"1min":167.221,"5min":167.221,"15min":704.245},"last":12.303000000000001},{"interface":"front","average":{"1min":27.606000000000002,"5min":30.131,"15min":28.628},"min":{"1min":9.8650000000000002,"5min":9.7200000000000006,"15min":8.3580000000000005},"max":{"1min":123.938,"5min":358.23000000000002,"15min":634.05499999999995},"last":12.534000000000001}]},{"osd":7,"last update":"Mon Jun 3 09:06:20 2024","interfaces":[{"interface":"back","average":{"1min":14.66,"5min":23.234999999999999,"15min":31.097000000000001},"min":{"1min":11.250999999999999,"5min":8.0559999999999992,"15min":8.0559999999999992},"max":{"1min":21.507999999999999,"5min":270.95100000000002,"15min":703.07799999999997},"last":15.563000000000001},{"interface":"front","average":{"1min":15.548,"5min":28.760000000000002,"15min":28.925000000000001},"min":{"1min":12.759,"5min":8.2089999999999996,"15min":7.7599999999999998},"max":{"1min":22.408000000000001,"5min":360.99900000000002,"15min":608.44100000000003},"last":14.904999999999999}]}]},{"osd":0,"up_from":14,"seq":60129542507,"num_pgs":10,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":81152,"kb_used_data":5752,"kb_used_omap":1,"kb_used_meta":75390,"kb_avail":94290688,"statfs":{"total":96636764160,"available":96553664512,"internally_reserved":0,"allocated":5890048,"data_stored":2763073,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1593,"internal_metadata":77199815},"hb_peers":[1,2,3,4,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":15,"apply_latency_ms":15,"commit_latency_ns":15000000,"apply_latency_ns":15000000},"alerts":[],"network_ping_times":[{"osd":1,"last update":"Mon Jun 3 09:06:04 2024","interfaces":[{"interface":"back","average":{"1min":6.9960000000000004,"5min":56.997,"15min":339.29700000000003},"min":{"1min":4.4530000000000003,"5min":4.4530000000000003,"15min":4.4100000000000001},"max":{"1min":11.720000000000001,"5min":2414.4749999999999,"15min":20691.453000000001},"last":7.8399999999999999},{"interface":"front","average":{"1min":9.548,"5min":57.258000000000003,"15min":337.791},"min":{"1min":5.2300000000000004,"5min":5.2300000000000004,"15min":4.6050000000000004},"max":{"1min":15.337999999999999,"5min":2412.9549999999999,"15min":20640.792000000001},"last":10.067}]},{"osd":2,"last update":"Mon Jun 3 09:06:22 2024","interfaces":[{"interface":"back","average":{"1min":11.154,"5min":26.515999999999998,"15min":22.882000000000001},"min":{"1min":5.7869999999999999,"5min":5.766,"15min":5.766},"max":{"1min":72.269999999999996,"5min":627.40700000000004,"15min":627.40700000000004},"last":8.4190000000000005},{"interface":"front","average":{"1min":14.353999999999999,"5min":33.304000000000002,"15min":29.885000000000002},"min":{"1min":8.2149999999999999,"5min":7.8630000000000004,"15min":7.4089999999999998},"max":{"1min":73.418000000000006,"5min":635.62599999999998,"15min":635.62599999999998},"last":14.103}]},{"osd":3,"last update":"Mon Jun 3 09:06:04 2024","interfaces":[{"interface":"back","average":{"1min":12.105,"5min":26.925000000000001,"15min":26.584},"min":{"1min":7.8410000000000002,"5min":7.1280000000000001,"15min":6.1909999999999998},"max":{"1min":18.425999999999998,"5min":389.72699999999998,"15min":389.72699999999998},"last":6.8739999999999997},{"interface":"front","average":{"1min":13.794,"5min":29.771999999999998,"15min":28.815000000000001},"min":{"1min":10.084,"5min":9.4920000000000009,"15min":6.1849999999999996},"max":{"1min":17.373999999999999,"5min":390.23200000000003,"15min":494.92899999999997},"last":15.661}]},{"osd":4,"last update":"Mon Jun 3 09:06:27 2024","interfaces":[{"interface":"back","average":{"1min":11.352,"5min":745.30600000000004,"15min":363.48099999999999},"min":{"1min":6.415,"5min":6.415,"15min":6.415},"max":{"1min":16.724,"5min":20248.616000000002,"15min":25555.212},"last":12.611000000000001},{"interface":"front","average":{"1min":13.545,"5min":744.76999999999998,"15min":361.03100000000001},"min":{"1min":9.3889999999999993,"5min":8.8940000000000001,"15min":7.1429999999999998},"max":{"1min":18.411000000000001,"5min":20250.195,"15min":25556.159},"last":17.414999999999999}]},{"osd":5,"last update":"Mon Jun 3 09:06:17 2024","interfaces":[{"interface":"back","average":{"1min":9.7040000000000006,"5min":26.533999999999999,"15min":28.683},"min":{"1min":6.8129999999999997,"5min":6.8129999999999997,"15min":6.8129999999999997},"max":{"1min":14.942,"5min":455.78699999999998,"15min":605.05899999999997},"last":10.74},{"interface":"front","average":{"1min":14.279,"5min":28.398,"15min":26.192},"min":{"1min":11.696,"5min":6.4420000000000002,"15min":6.2030000000000003},"max":{"1min":17.879000000000001,"5min":391.31200000000001,"15min":541.11000000000001},"last":11.941000000000001}]},{"osd":6,"last update":"Mon Jun 3 09:04:30 2024","interfaces":[{"interface":"back","average":{"1min":52.496000000000002,"5min":33.139000000000003,"15min":160.83199999999999},"min":{"1min":7.5289999999999999,"5min":7.1040000000000001,"15min":6.3739999999999997},"max":{"1min":457.91699999999997,"5min":457.91699999999997,"15min":11115.258},"last":16.187999999999999},{"interface":"front","average":{"1min":49.670000000000002,"5min":33.704999999999998,"15min":162.791},"min":{"1min":12.247999999999999,"5min":11.019,"15min":9.9760000000000009},"max":{"1min":407.05099999999999,"5min":407.05099999999999,"15min":11092.194},"last":14.961}]},{"osd":7,"last update":"Mon Jun 3 09:06:22 2024","interfaces":[{"interface":"back","average":{"1min":11.625999999999999,"5min":26.375,"15min":28.972999999999999},"min":{"1min":6.4480000000000004,"5min":6.4480000000000004,"15min":6.4480000000000004},"max":{"1min":15.352,"5min":461.41699999999997,"15min":598.33699999999999},"last":16.821999999999999},{"interface":"front","average":{"1min":14.013,"5min":29.696000000000002,"15min":33.630000000000003},"min":{"1min":10.938000000000001,"5min":10.58,"15min":10.263999999999999},"max":{"1min":16.603000000000002,"5min":463.10599999999999,"15min":592.80499999999995},"last":18.053999999999998}]}]}],"pool_statfs":[{"poolid":1,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":401408,"data_stored":397840,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":5,"total":0,"available":0,"internally_reserved":0,"allocated":401408,"data_stored":397840,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":4,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":5,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":6,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":17,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":17,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":17,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":17,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":17,"osd":4,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":17,"osd":5,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":5,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":17,"osd":6,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":5,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":17,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0}]}} 2024-06-03T09:06:54.458 DEBUG:teuthology.orchestra.run.smithi088:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd dump --format=json 2024-06-03T09:06:54.560 INFO:tasks.ceph.mon.a.smithi088.stderr:==00:00:31:49.886 34743== Warning: unimplemented fcntl command: 1036 2024-06-03T09:06:54.561 INFO:tasks.ceph.mon.a.smithi088.stderr:==00:00:31:49.887 34743== Warning: unimplemented fcntl command: 1036 2024-06-03T09:06:54.570 INFO:teuthology.orchestra.run.smithi088.stderr:2024-06-03T09:06:54.570+0000 7fc8128d7640 10 monclient: get_monmap_and_config 2024-06-03T09:06:54.570 INFO:teuthology.orchestra.run.smithi088.stderr:2024-06-03T09:06:54.570+0000 7fc8128d7640 10 monclient: build_initial_monmap 2024-06-03T09:06:54.570 INFO:teuthology.orchestra.run.smithi088.stderr:2024-06-03T09:06:54.570+0000 7fc8128d7640 10 monclient: monmap: 2024-06-03T09:06:54.570 INFO:teuthology.orchestra.run.smithi088.stderr:epoch 0 2024-06-03T09:06:54.570 INFO:teuthology.orchestra.run.smithi088.stderr:fsid ec3c3fdd-3ac6-4d7d-9412-683130ed8b04 2024-06-03T09:06:54.570 INFO:teuthology.orchestra.run.smithi088.stderr:last_changed 2024-06-03T09:06:54.571051+0000 2024-06-03T09:06:54.570 INFO:teuthology.orchestra.run.smithi088.stderr:created 2024-06-03T09:06:54.571051+0000 2024-06-03T09:06:54.570 INFO:teuthology.orchestra.run.smithi088.stderr:min_mon_release 0 (unknown) 2024-06-03T09:06:54.571 INFO:teuthology.orchestra.run.smithi088.stderr:election_strategy: 3 2024-06-03T09:06:54.571 INFO:teuthology.orchestra.run.smithi088.stderr:0: v1:172.21.15.88:6789/0 mon.noname-a 2024-06-03T09:06:54.571 INFO:teuthology.orchestra.run.smithi088.stderr:1: v1:172.21.15.192:6789/0 mon.noname-c 2024-06-03T09:06:54.571 INFO:teuthology.orchestra.run.smithi088.stderr:2: v1:172.21.15.88:6790/0 mon.noname-b 2024-06-03T09:06:54.571 INFO:teuthology.orchestra.run.smithi088.stderr: 2024-06-03T09:06:54.571 INFO:tasks.ceph.mon.a.smithi088.stderr:==00:00:31:49.895 34743== Warning: unimplemented fcntl command: 1036 2024-06-03T09:06:54.572 INFO:teuthology.orchestra.run.smithi088.stderr:2024-06-03T09:06:54.570+0000 7fc8128d7640 1 Processor -- start 2024-06-03T09:06:54.572 INFO:teuthology.orchestra.run.smithi088.stderr:2024-06-03T09:06:54.570+0000 7fc8128d7640 1 -- start start 2024-06-03T09:06:54.572 INFO:teuthology.orchestra.run.smithi088.stderr:2024-06-03T09:06:54.570+0000 7fc8128d7640 10 monclient: init 2024-06-03T09:06:54.572 INFO:teuthology.orchestra.run.smithi088.stderr:2024-06-03T09:06:54.571+0000 7fc8128d7640 10 monclient: _reopen_session rank -1 2024-06-03T09:06:54.572 INFO:teuthology.orchestra.run.smithi088.stderr:2024-06-03T09:06:54.571+0000 7fc8128d7640 10 monclient: _add_conns ranks=[2,1,0] 2024-06-03T09:06:54.572 INFO:teuthology.orchestra.run.smithi088.stderr:2024-06-03T09:06:54.571+0000 7fc8128d7640 10 monclient(hunting): picked mon.noname-b con 0x7fc80c0a8960 addr v1:172.21.15.88:6790/0 2024-06-03T09:06:54.572 INFO:teuthology.orchestra.run.smithi088.stderr:2024-06-03T09:06:54.571+0000 7fc8128d7640 10 monclient(hunting): picked mon.noname-c con 0x7fc80c1450f0 addr v1:172.21.15.192:6789/0 2024-06-03T09:06:54.572 INFO:teuthology.orchestra.run.smithi088.stderr:2024-06-03T09:06:54.571+0000 7fc8128d7640 10 monclient(hunting): picked mon.noname-a con 0x7fc80c13eec0 addr v1:172.21.15.88:6789/0 2024-06-03T09:06:54.573 INFO:teuthology.orchestra.run.smithi088.stderr:2024-06-03T09:06:54.571+0000 7fc8128d7640 1 -- --> v1:172.21.15.88:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fc80c144c60 con 0x7fc80c13eec0 2024-06-03T09:06:54.573 INFO:teuthology.orchestra.run.smithi088.stderr:2024-06-03T09:06:54.571+0000 7fc8128d7640 1 -- --> v1:172.21.15.192:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fc80c0b3930 con 0x7fc80c1450f0 2024-06-03T09:06:54.573 INFO:teuthology.orchestra.run.smithi088.stderr:2024-06-03T09:06:54.571+0000 7fc8128d7640 1 -- --> v1:172.21.15.88:6790/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fc80c0b1ad0 con 0x7fc80c0a8960 2024-06-03T09:06:54.573 INFO:teuthology.orchestra.run.smithi088.stderr:2024-06-03T09:06:54.571+0000 7fc8128d7640 10 monclient(hunting): _renew_subs 2024-06-03T09:06:54.573 INFO:teuthology.orchestra.run.smithi088.stderr:2024-06-03T09:06:54.571+0000 7fc8128d7640 10 monclient(hunting): authenticate will time out at 3070.381562s 2024-06-03T09:06:54.574 INFO:teuthology.orchestra.run.smithi088.stderr:2024-06-03T09:06:54.575+0000 7fc810e74640 1 --1- >> v1:172.21.15.88:6789/0 conn(0x7fc80c13eec0 0x7fc80c14ce60 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.88:6789/0 says I am v1:172.21.15.88:43354/0 (socket says 172.21.15.88:43354) 2024-06-03T09:06:54.574 INFO:teuthology.orchestra.run.smithi088.stderr:2024-06-03T09:06:54.575+0000 7fc810e74640 1 -- 172.21.15.88:0/2396873647 learned_addr learned my addr 172.21.15.88:0/2396873647 (peer_addr_for_me v1:172.21.15.88:0/0) 2024-06-03T09:06:54.584 INFO:teuthology.orchestra.run.smithi088.stderr:2024-06-03T09:06:54.584+0000 7fc80affd640 1 -- 172.21.15.88:0/2396873647 <== mon.0 v1:172.21.15.88:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 33377265 0 0) 0x7fc80c144c60 con 0x7fc80c13eec0 2024-06-03T09:06:54.584 INFO:teuthology.orchestra.run.smithi088.stderr:2024-06-03T09:06:54.584+0000 7fc80affd640 10 monclient(hunting): _negotiate 2024-06-03T09:06:54.584 INFO:teuthology.orchestra.run.smithi088.stderr:2024-06-03T09:06:54.584+0000 7fc80affd640 10 monclient(hunting): _init_auth method 2 2024-06-03T09:06:54.584 INFO:teuthology.orchestra.run.smithi088.stderr:2024-06-03T09:06:54.584+0000 7fc80affd640 10 monclient(hunting): _init_auth creating new auth 2024-06-03T09:06:54.584 INFO:teuthology.orchestra.run.smithi088.stderr:2024-06-03T09:06:54.584+0000 7fc80affd640 10 monclient(hunting): my global_id is 30732 2024-06-03T09:06:54.584 INFO:teuthology.orchestra.run.smithi088.stderr:2024-06-03T09:06:54.584+0000 7fc80affd640 1 -- 172.21.15.88:0/2396873647 --> v1:172.21.15.88:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7fc7f4003520 con 0x7fc80c13eec0 2024-06-03T09:06:54.584 INFO:teuthology.orchestra.run.smithi088.stderr:2024-06-03T09:06:54.584+0000 7fc80affd640 1 -- 172.21.15.88:0/2396873647 <== mon.1 v1:172.21.15.192:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 1402514556 0 0) 0x7fc80c0b3930 con 0x7fc80c1450f0 2024-06-03T09:06:54.585 INFO:teuthology.orchestra.run.smithi088.stderr:2024-06-03T09:06:54.584+0000 7fc80affd640 10 monclient(hunting): _negotiate 2024-06-03T09:06:54.585 INFO:teuthology.orchestra.run.smithi088.stderr:2024-06-03T09:06:54.584+0000 7fc80affd640 10 monclient(hunting): _init_auth method 2 2024-06-03T09:06:54.585 INFO:teuthology.orchestra.run.smithi088.stderr:2024-06-03T09:06:54.584+0000 7fc80affd640 10 monclient(hunting): _init_auth creating new auth 2024-06-03T09:06:54.585 INFO:teuthology.orchestra.run.smithi088.stderr:2024-06-03T09:06:54.584+0000 7fc80affd640 10 monclient(hunting): my global_id is 30301 2024-06-03T09:06:54.585 INFO:teuthology.orchestra.run.smithi088.stderr:2024-06-03T09:06:54.584+0000 7fc80affd640 1 -- 172.21.15.88:0/2396873647 --> v1:172.21.15.192:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7fc80c144c60 con 0x7fc80c1450f0 2024-06-03T09:06:54.588 INFO:teuthology.orchestra.run.smithi088.stderr:2024-06-03T09:06:54.588+0000 7fc80affd640 1 -- 172.21.15.88:0/2396873647 <== mon.0 v1:172.21.15.88:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 214+0+0 (unknown 3735388166 0 0) 0x7fc7f4003520 con 0x7fc80c13eec0 2024-06-03T09:06:54.588 INFO:teuthology.orchestra.run.smithi088.stderr:2024-06-03T09:06:54.588+0000 7fc80affd640 1 -- 172.21.15.88:0/2396873647 >> v1:172.21.15.88:6790/0 conn(0x7fc80c0a8960 legacy=0x7fc80c0a8d40 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-06-03T09:06:54.588 INFO:teuthology.orchestra.run.smithi088.stderr:2024-06-03T09:06:54.588+0000 7fc80affd640 1 -- 172.21.15.88:0/2396873647 >> v1:172.21.15.192:6789/0 conn(0x7fc80c1450f0 legacy=0x7fc80c13e7b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-06-03T09:06:54.588 INFO:teuthology.orchestra.run.smithi088.stderr:2024-06-03T09:06:54.588+0000 7fc80affd640 10 monclient: _finish_hunting 0 2024-06-03T09:06:54.588 INFO:teuthology.orchestra.run.smithi088.stderr:2024-06-03T09:06:54.588+0000 7fc80affd640 1 monclient: found mon.noname-a 2024-06-03T09:06:54.588 INFO:teuthology.orchestra.run.smithi088.stderr:2024-06-03T09:06:54.588+0000 7fc80affd640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-06-03T09:06:54.589 INFO:teuthology.orchestra.run.smithi088.stderr:2024-06-03T09:06:54.588+0000 7fc80affd640 10 monclient: _send_mon_message to mon.noname-a at v1:172.21.15.88:6789/0 2024-06-03T09:06:54.589 INFO:teuthology.orchestra.run.smithi088.stderr:2024-06-03T09:06:54.588+0000 7fc80affd640 1 -- 172.21.15.88:0/2396873647 --> v1:172.21.15.88:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fc80c0b1c60 con 0x7fc80c13eec0 2024-06-03T09:06:54.589 INFO:teuthology.orchestra.run.smithi088.stderr:2024-06-03T09:06:54.588+0000 7fc80affd640 10 monclient: _finish_auth 0 2024-06-03T09:06:54.589 INFO:teuthology.orchestra.run.smithi088.stderr:2024-06-03T09:06:54.588+0000 7fc80affd640 10 monclient: _check_auth_tickets 2024-06-03T09:06:54.589 INFO:teuthology.orchestra.run.smithi088.stderr:2024-06-03T09:06:54.588+0000 7fc80affd640 20 monclient: _check_auth_rotating not needed by client.admin 2024-06-03T09:06:54.589 INFO:teuthology.orchestra.run.smithi088.stderr:2024-06-03T09:06:54.588+0000 7fc80affd640 1 -- 172.21.15.88:0/2396873647 <== mon.0 v1:172.21.15.88:6789/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 2234374065 0 0) 0x7fc804001990 con 0x7fc80c13eec0 2024-06-03T09:06:54.589 INFO:teuthology.orchestra.run.smithi088.stderr:2024-06-03T09:06:54.588+0000 7fc8128d7640 5 monclient: authenticate success, global_id 30732 2024-06-03T09:06:54.589 INFO:teuthology.orchestra.run.smithi088.stderr:2024-06-03T09:06:54.588+0000 7fc80affd640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-06-03T09:06:54.589 INFO:teuthology.orchestra.run.smithi088.stderr:2024-06-03T09:06:54.588+0000 7fc80affd640 10 monclient: got monmap 1 from mon.noname-a (according to old e1) 2024-06-03T09:06:54.589 INFO:teuthology.orchestra.run.smithi088.stderr:2024-06-03T09:06:54.588+0000 7fc80affd640 10 monclient: dump: 2024-06-03T09:06:54.589 INFO:teuthology.orchestra.run.smithi088.stderr:epoch 1 2024-06-03T09:06:54.589 INFO:teuthology.orchestra.run.smithi088.stderr:fsid ec3c3fdd-3ac6-4d7d-9412-683130ed8b04 2024-06-03T09:06:54.589 INFO:teuthology.orchestra.run.smithi088.stderr:last_changed 2024-06-03T08:34:31.885125+0000 2024-06-03T09:06:54.590 INFO:teuthology.orchestra.run.smithi088.stderr:created 2024-06-03T08:34:31.885125+0000 2024-06-03T09:06:54.590 INFO:teuthology.orchestra.run.smithi088.stderr:min_mon_release 18 (reef) 2024-06-03T09:06:54.590 INFO:teuthology.orchestra.run.smithi088.stderr:election_strategy: 3 2024-06-03T09:06:54.590 INFO:teuthology.orchestra.run.smithi088.stderr:0: v1:172.21.15.88:6789/0 mon.a 2024-06-03T09:06:54.590 INFO:teuthology.orchestra.run.smithi088.stderr:1: v1:172.21.15.192:6789/0 mon.b 2024-06-03T09:06:54.590 INFO:teuthology.orchestra.run.smithi088.stderr:2: v1:172.21.15.88:6790/0 mon.c 2024-06-03T09:06:54.590 INFO:teuthology.orchestra.run.smithi088.stderr: 2024-06-03T09:06:54.590 INFO:teuthology.orchestra.run.smithi088.stderr:2024-06-03T09:06:54.588+0000 7fc8128d7640 20 monclient: get_monmap_and_config waiting for monmap|config 2024-06-03T09:06:54.603 INFO:teuthology.orchestra.run.smithi088.stderr:2024-06-03T09:06:54.603+0000 7fc80affd640 1 -- 172.21.15.88:0/2396873647 <== mon.0 v1:172.21.15.88:6789/0 4 ==== config(0 keys) v1 ==== 4+0+0 (unknown 0 0 0) 0x7fc804001bb0 con 0x7fc80c13eec0 2024-06-03T09:06:54.603 INFO:teuthology.orchestra.run.smithi088.stderr:2024-06-03T09:06:54.603+0000 7fc80affd640 10 monclient: handle_config config(0 keys) v1 2024-06-03T09:06:54.603 INFO:teuthology.orchestra.run.smithi088.stderr:2024-06-03T09:06:54.603+0000 7fc8128d7640 10 monclient: get_monmap_and_config success 2024-06-03T09:06:54.603 INFO:teuthology.orchestra.run.smithi088.stderr:2024-06-03T09:06:54.603+0000 7fc8128d7640 4 set_mon_vals no callback set 2024-06-03T09:06:54.605 INFO:teuthology.orchestra.run.smithi088.stderr:2024-06-03T09:06:54.603+0000 7fc8128d7640 10 monclient: shutdown 2024-06-03T09:06:54.605 INFO:teuthology.orchestra.run.smithi088.stderr:2024-06-03T09:06:54.603+0000 7fc8128d7640 20 monclient: shutdown discarding 0 pending message(s) 2024-06-03T09:06:54.605 INFO:teuthology.orchestra.run.smithi088.stderr:2024-06-03T09:06:54.603+0000 7fc8128d7640 1 -- 172.21.15.88:0/2396873647 >> v1:172.21.15.88:6789/0 conn(0x7fc80c13eec0 legacy=0x7fc80c14ce60 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-06-03T09:06:54.605 INFO:teuthology.orchestra.run.smithi088.stderr:2024-06-03T09:06:54.603+0000 7fc8128d7640 1 -- 172.21.15.88:0/2396873647 shutdown_connections 2024-06-03T09:06:54.605 INFO:teuthology.orchestra.run.smithi088.stderr:2024-06-03T09:06:54.603+0000 7fc8128d7640 1 -- 172.21.15.88:0/2396873647 >> 172.21.15.88:0/2396873647 conn(0x7fc80c0b70d0 msgr2=0x7fc80c0b74e0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-06-03T09:06:54.605 INFO:teuthology.orchestra.run.smithi088.stderr:2024-06-03T09:06:54.604+0000 7fc8128d7640 1 -- 172.21.15.88:0/2396873647 shutdown_connections 2024-06-03T09:06:54.605 INFO:teuthology.orchestra.run.smithi088.stderr:2024-06-03T09:06:54.604+0000 7fc8128d7640 1 -- 172.21.15.88:0/2396873647 wait complete. 2024-06-03T09:06:54.605 INFO:teuthology.orchestra.run.smithi088.stderr:2024-06-03T09:06:54.604+0000 7fc8128d7640 10 monclient: build_initial_monmap 2024-06-03T09:06:54.605 INFO:teuthology.orchestra.run.smithi088.stderr:2024-06-03T09:06:54.604+0000 7fc8128d7640 10 monclient: monmap: 2024-06-03T09:06:54.606 INFO:teuthology.orchestra.run.smithi088.stderr:epoch 0 2024-06-03T09:06:54.606 INFO:teuthology.orchestra.run.smithi088.stderr:fsid 00000000-0000-0000-0000-000000000000 2024-06-03T09:06:54.606 INFO:teuthology.orchestra.run.smithi088.stderr:last_changed 0.000000 2024-06-03T09:06:54.606 INFO:teuthology.orchestra.run.smithi088.stderr:created 0.000000 2024-06-03T09:06:54.606 INFO:teuthology.orchestra.run.smithi088.stderr:min_mon_release 0 (unknown) 2024-06-03T09:06:54.606 INFO:teuthology.orchestra.run.smithi088.stderr:election_strategy: 1 2024-06-03T09:06:54.606 INFO:teuthology.orchestra.run.smithi088.stderr:0: v1:172.21.15.88:6789/0 mon.noname-a 2024-06-03T09:06:54.606 INFO:teuthology.orchestra.run.smithi088.stderr:1: v1:172.21.15.192:6789/0 mon.noname-b 2024-06-03T09:06:54.606 INFO:teuthology.orchestra.run.smithi088.stderr:2: v1:172.21.15.88:6790/0 mon.noname-c 2024-06-03T09:06:54.606 INFO:teuthology.orchestra.run.smithi088.stderr: 2024-06-03T09:06:54.606 INFO:teuthology.orchestra.run.smithi088.stderr:2024-06-03T09:06:54.604+0000 7fc8128d7640 1 librados: starting msgr at 2024-06-03T09:06:54.606 INFO:teuthology.orchestra.run.smithi088.stderr:2024-06-03T09:06:54.604+0000 7fc8128d7640 1 librados: starting objecter 2024-06-03T09:06:54.606 INFO:teuthology.orchestra.run.smithi088.stderr:2024-06-03T09:06:54.605+0000 7fc8128d7640 1 Processor -- start 2024-06-03T09:06:54.607 INFO:teuthology.orchestra.run.smithi088.stderr:2024-06-03T09:06:54.605+0000 7fc8128d7640 1 -- start start 2024-06-03T09:06:54.607 INFO:teuthology.orchestra.run.smithi088.stderr:2024-06-03T09:06:54.605+0000 7fc8128d7640 1 librados: setting wanted keys 2024-06-03T09:06:54.607 INFO:teuthology.orchestra.run.smithi088.stderr:2024-06-03T09:06:54.605+0000 7fc8128d7640 1 librados: calling monclient init 2024-06-03T09:06:54.607 INFO:teuthology.orchestra.run.smithi088.stderr:2024-06-03T09:06:54.605+0000 7fc8128d7640 10 monclient: init 2024-06-03T09:06:54.608 INFO:teuthology.orchestra.run.smithi088.stderr:2024-06-03T09:06:54.605+0000 7fc8128d7640 10 monclient: _reopen_session rank -1 2024-06-03T09:06:54.608 INFO:teuthology.orchestra.run.smithi088.stderr:2024-06-03T09:06:54.605+0000 7fc8128d7640 10 monclient: _add_conns ranks=[0,1,2] 2024-06-03T09:06:54.608 INFO:teuthology.orchestra.run.smithi088.stderr:2024-06-03T09:06:54.605+0000 7fc8128d7640 10 monclient(hunting): picked mon.noname-a con 0x7fc80c0a8960 addr v1:172.21.15.88:6789/0 2024-06-03T09:06:54.608 INFO:teuthology.orchestra.run.smithi088.stderr:2024-06-03T09:06:54.605+0000 7fc8128d7640 10 monclient(hunting): picked mon.noname-b con 0x7fc80c13eec0 addr v1:172.21.15.192:6789/0 2024-06-03T09:06:54.608 INFO:teuthology.orchestra.run.smithi088.stderr:2024-06-03T09:06:54.605+0000 7fc8128d7640 10 monclient(hunting): picked mon.noname-c con 0x7fc80c1450f0 addr v1:172.21.15.88:6790/0 2024-06-03T09:06:54.608 INFO:teuthology.orchestra.run.smithi088.stderr:2024-06-03T09:06:54.605+0000 7fc8128d7640 1 -- --> v1:172.21.15.88:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fc80c13f380 con 0x7fc80c0a8960 2024-06-03T09:06:54.608 INFO:teuthology.orchestra.run.smithi088.stderr:2024-06-03T09:06:54.605+0000 7fc8128d7640 1 -- --> v1:172.21.15.192:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fc80c0ac6c0 con 0x7fc80c13eec0 2024-06-03T09:06:54.608 INFO:teuthology.orchestra.run.smithi088.stderr:2024-06-03T09:06:54.605+0000 7fc8128d7640 1 -- --> v1:172.21.15.88:6790/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fc80c137c50 con 0x7fc80c1450f0 2024-06-03T09:06:54.608 INFO:teuthology.orchestra.run.smithi088.stderr:2024-06-03T09:06:54.605+0000 7fc8128d7640 10 monclient(hunting): _renew_subs 2024-06-03T09:06:54.608 INFO:teuthology.orchestra.run.smithi088.stderr:2024-06-03T09:06:54.605+0000 7fc8128d7640 10 monclient(hunting): authenticate will time out at 3070.415859s 2024-06-03T09:06:54.609 INFO:teuthology.orchestra.run.smithi088.stderr:2024-06-03T09:06:54.609+0000 7fc810e74640 1 --1- >> v1:172.21.15.88:6790/0 conn(0x7fc80c1450f0 0x7fc80c137540 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=1).handle_server_banner_and_identify peer v1:172.21.15.88:6790/0 says I am v1:172.21.15.88:52416/0 (socket says 172.21.15.88:52416) 2024-06-03T09:06:54.609 INFO:teuthology.orchestra.run.smithi088.stderr:2024-06-03T09:06:54.609+0000 7fc810e74640 1 -- 172.21.15.88:0/3798012535 learned_addr learned my addr 172.21.15.88:0/3798012535 (peer_addr_for_me v1:172.21.15.88:0/0) 2024-06-03T09:06:54.612 INFO:teuthology.orchestra.run.smithi088.stderr:2024-06-03T09:06:54.612+0000 7fc808ff9640 10 client.?.objecter ms_handle_connect 0x7fc80c0a8960 2024-06-03T09:06:54.612 INFO:teuthology.orchestra.run.smithi088.stderr:2024-06-03T09:06:54.612+0000 7fc808ff9640 10 client.?.objecter resend_mon_ops 2024-06-03T09:06:54.613 INFO:teuthology.orchestra.run.smithi088.stderr:2024-06-03T09:06:54.613+0000 7fc808ff9640 10 client.?.objecter ms_handle_connect 0x7fc80c1450f0 2024-06-03T09:06:54.613 INFO:teuthology.orchestra.run.smithi088.stderr:2024-06-03T09:06:54.613+0000 7fc808ff9640 10 client.?.objecter resend_mon_ops 2024-06-03T09:06:54.621 INFO:teuthology.orchestra.run.smithi088.stderr:2024-06-03T09:06:54.621+0000 7fc808ff9640 1 -- 172.21.15.88:0/3798012535 <== mon.2 v1:172.21.15.88:6790/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 2961493768 0 0) 0x7fc80c137c50 con 0x7fc80c1450f0 2024-06-03T09:06:54.621 INFO:teuthology.orchestra.run.smithi088.stderr:2024-06-03T09:06:54.621+0000 7fc808ff9640 10 monclient(hunting): _negotiate 2024-06-03T09:06:54.622 INFO:teuthology.orchestra.run.smithi088.stderr:2024-06-03T09:06:54.621+0000 7fc808ff9640 10 monclient(hunting): _init_auth method 2 2024-06-03T09:06:54.622 INFO:teuthology.orchestra.run.smithi088.stderr:2024-06-03T09:06:54.621+0000 7fc808ff9640 10 monclient(hunting): _init_auth creating new auth 2024-06-03T09:06:54.622 INFO:teuthology.orchestra.run.smithi088.stderr:2024-06-03T09:06:54.621+0000 7fc808ff9640 10 monclient(hunting): my global_id is 30239 2024-06-03T09:06:54.622 INFO:teuthology.orchestra.run.smithi088.stderr:2024-06-03T09:06:54.621+0000 7fc808ff9640 1 -- 172.21.15.88:0/3798012535 --> v1:172.21.15.88:6790/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7fc7f8004eb0 con 0x7fc80c1450f0 2024-06-03T09:06:54.626 INFO:teuthology.orchestra.run.smithi088.stderr:2024-06-03T09:06:54.624+0000 7fc808ff9640 1 -- 172.21.15.88:0/3798012535 <== mon.0 v1:172.21.15.88:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 3746241810 0 0) 0x7fc80c13f380 con 0x7fc80c0a8960 2024-06-03T09:06:54.626 INFO:teuthology.orchestra.run.smithi088.stderr:2024-06-03T09:06:54.624+0000 7fc808ff9640 10 monclient(hunting): _negotiate 2024-06-03T09:06:54.626 INFO:teuthology.orchestra.run.smithi088.stderr:2024-06-03T09:06:54.624+0000 7fc808ff9640 10 monclient(hunting): _init_auth method 2 2024-06-03T09:06:54.626 INFO:teuthology.orchestra.run.smithi088.stderr:2024-06-03T09:06:54.624+0000 7fc808ff9640 10 monclient(hunting): _init_auth creating new auth 2024-06-03T09:06:54.626 INFO:teuthology.orchestra.run.smithi088.stderr:2024-06-03T09:06:54.624+0000 7fc808ff9640 10 monclient(hunting): my global_id is 30735 2024-06-03T09:06:54.626 INFO:teuthology.orchestra.run.smithi088.stderr:2024-06-03T09:06:54.624+0000 7fc808ff9640 1 -- 172.21.15.88:0/3798012535 --> v1:172.21.15.88:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7fc80c137c50 con 0x7fc80c0a8960 2024-06-03T09:06:54.636 INFO:teuthology.orchestra.run.smithi088.stderr:2024-06-03T09:06:54.636+0000 7fc808ff9640 10 client.?.objecter ms_handle_connect 0x7fc80c13eec0 2024-06-03T09:06:54.636 INFO:teuthology.orchestra.run.smithi088.stderr:2024-06-03T09:06:54.636+0000 7fc808ff9640 10 client.?.objecter resend_mon_ops 2024-06-03T09:06:54.640 INFO:teuthology.orchestra.run.smithi088.stderr:2024-06-03T09:06:54.639+0000 7fc808ff9640 1 -- 172.21.15.88:0/3798012535 <== mon.2 v1:172.21.15.88:6790/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 764+0+0 (unknown 3959412981 0 0) 0x7fc7f8004eb0 con 0x7fc80c1450f0 2024-06-03T09:06:54.640 INFO:teuthology.orchestra.run.smithi088.stderr:2024-06-03T09:06:54.640+0000 7fc808ff9640 1 -- 172.21.15.88:0/3798012535 --> v1:172.21.15.88:6790/0 -- auth(proto 2 165 bytes epoch 0) v1 -- 0x7fc80c13f380 con 0x7fc80c1450f0 2024-06-03T09:06:54.640 INFO:teuthology.orchestra.run.smithi088.stderr:2024-06-03T09:06:54.640+0000 7fc808ff9640 1 -- 172.21.15.88:0/3798012535 <== mon.2 v1:172.21.15.88:6790/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 2234374065 0 0) 0x7fc8040018d0 con 0x7fc80c1450f0 2024-06-03T09:06:54.640 INFO:teuthology.orchestra.run.smithi088.stderr:2024-06-03T09:06:54.640+0000 7fc808ff9640 10 monclient(hunting): handle_monmap mon_map magic: 0 v1 2024-06-03T09:06:54.640 INFO:teuthology.orchestra.run.smithi088.stderr:2024-06-03T09:06:54.640+0000 7fc808ff9640 10 monclient(hunting): got monmap 1 from mon.noname-c (according to old e1) 2024-06-03T09:06:54.640 INFO:teuthology.orchestra.run.smithi088.stderr:2024-06-03T09:06:54.640+0000 7fc808ff9640 10 monclient(hunting): dump: 2024-06-03T09:06:54.640 INFO:teuthology.orchestra.run.smithi088.stderr:epoch 1 2024-06-03T09:06:54.640 INFO:teuthology.orchestra.run.smithi088.stderr:fsid ec3c3fdd-3ac6-4d7d-9412-683130ed8b04 2024-06-03T09:06:54.640 INFO:teuthology.orchestra.run.smithi088.stderr:last_changed 2024-06-03T08:34:31.885125+0000 2024-06-03T09:06:54.640 INFO:teuthology.orchestra.run.smithi088.stderr:created 2024-06-03T08:34:31.885125+0000 2024-06-03T09:06:54.640 INFO:teuthology.orchestra.run.smithi088.stderr:min_mon_release 18 (reef) 2024-06-03T09:06:54.641 INFO:teuthology.orchestra.run.smithi088.stderr:election_strategy: 3 2024-06-03T09:06:54.641 INFO:teuthology.orchestra.run.smithi088.stderr:0: v1:172.21.15.88:6789/0 mon.a 2024-06-03T09:06:54.641 INFO:teuthology.orchestra.run.smithi088.stderr:1: v1:172.21.15.192:6789/0 mon.b 2024-06-03T09:06:54.641 INFO:teuthology.orchestra.run.smithi088.stderr:2: v1:172.21.15.88:6790/0 mon.c 2024-06-03T09:06:54.641 INFO:teuthology.orchestra.run.smithi088.stderr: 2024-06-03T09:06:54.641 INFO:teuthology.orchestra.run.smithi088.stderr:2024-06-03T09:06:54.640+0000 7fc808ff9640 10 monclient(hunting): _finish_auth 0 2024-06-03T09:06:54.645 INFO:teuthology.orchestra.run.smithi088.stderr:2024-06-03T09:06:54.645+0000 7fc808ff9640 1 -- 172.21.15.88:0/3798012535 <== mon.1 v1:172.21.15.192:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 3072300194 0 0) 0x7fc80c0ac6c0 con 0x7fc80c13eec0 2024-06-03T09:06:54.645 INFO:teuthology.orchestra.run.smithi088.stderr:2024-06-03T09:06:54.645+0000 7fc808ff9640 10 monclient(hunting): _negotiate 2024-06-03T09:06:54.645 INFO:teuthology.orchestra.run.smithi088.stderr:2024-06-03T09:06:54.645+0000 7fc808ff9640 10 monclient(hunting): _init_auth method 2 2024-06-03T09:06:54.645 INFO:teuthology.orchestra.run.smithi088.stderr:2024-06-03T09:06:54.645+0000 7fc808ff9640 10 monclient(hunting): _init_auth creating new auth 2024-06-03T09:06:54.646 INFO:teuthology.orchestra.run.smithi088.stderr:2024-06-03T09:06:54.645+0000 7fc808ff9640 10 monclient(hunting): my global_id is 30304 2024-06-03T09:06:54.646 INFO:teuthology.orchestra.run.smithi088.stderr:2024-06-03T09:06:54.645+0000 7fc808ff9640 1 -- 172.21.15.88:0/3798012535 --> v1:172.21.15.192:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7fc7f8004eb0 con 0x7fc80c13eec0 2024-06-03T09:06:54.652 INFO:teuthology.orchestra.run.smithi088.stderr:2024-06-03T09:06:54.652+0000 7fc808ff9640 1 -- 172.21.15.88:0/3798012535 <== mon.0 v1:172.21.15.88:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 764+0+0 (unknown 2940623861 0 0) 0x7fc80c137c50 con 0x7fc80c0a8960 2024-06-03T09:06:54.652 INFO:teuthology.orchestra.run.smithi088.stderr:2024-06-03T09:06:54.652+0000 7fc808ff9640 1 -- 172.21.15.88:0/3798012535 --> v1:172.21.15.88:6789/0 -- auth(proto 2 165 bytes epoch 0) v1 -- 0x7fc80c0ac6c0 con 0x7fc80c0a8960 2024-06-03T09:06:54.652 INFO:teuthology.orchestra.run.smithi088.stderr:2024-06-03T09:06:54.652+0000 7fc808ff9640 1 -- 172.21.15.88:0/3798012535 <== mon.0 v1:172.21.15.88:6789/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 2234374065 0 0) 0x7fc7fc0017e0 con 0x7fc80c0a8960 2024-06-03T09:06:54.652 INFO:teuthology.orchestra.run.smithi088.stderr:2024-06-03T09:06:54.652+0000 7fc808ff9640 10 monclient(hunting): handle_monmap mon_map magic: 0 v1 2024-06-03T09:06:54.652 INFO:teuthology.orchestra.run.smithi088.stderr:2024-06-03T09:06:54.652+0000 7fc808ff9640 10 monclient(hunting): got monmap 1 from mon.a (according to old e1) 2024-06-03T09:06:54.652 INFO:teuthology.orchestra.run.smithi088.stderr:2024-06-03T09:06:54.652+0000 7fc808ff9640 10 monclient(hunting): dump: 2024-06-03T09:06:54.652 INFO:teuthology.orchestra.run.smithi088.stderr:epoch 1 2024-06-03T09:06:54.652 INFO:teuthology.orchestra.run.smithi088.stderr:fsid ec3c3fdd-3ac6-4d7d-9412-683130ed8b04 2024-06-03T09:06:54.653 INFO:teuthology.orchestra.run.smithi088.stderr:last_changed 2024-06-03T08:34:31.885125+0000 2024-06-03T09:06:54.653 INFO:teuthology.orchestra.run.smithi088.stderr:created 2024-06-03T08:34:31.885125+0000 2024-06-03T09:06:54.653 INFO:teuthology.orchestra.run.smithi088.stderr:min_mon_release 18 (reef) 2024-06-03T09:06:54.653 INFO:teuthology.orchestra.run.smithi088.stderr:election_strategy: 3 2024-06-03T09:06:54.653 INFO:teuthology.orchestra.run.smithi088.stderr:0: v1:172.21.15.88:6789/0 mon.a 2024-06-03T09:06:54.653 INFO:teuthology.orchestra.run.smithi088.stderr:1: v1:172.21.15.192:6789/0 mon.b 2024-06-03T09:06:54.653 INFO:teuthology.orchestra.run.smithi088.stderr:2: v1:172.21.15.88:6790/0 mon.c 2024-06-03T09:06:54.653 INFO:teuthology.orchestra.run.smithi088.stderr: 2024-06-03T09:06:54.654 INFO:teuthology.orchestra.run.smithi088.stderr:2024-06-03T09:06:54.653+0000 7fc808ff9640 1 -- 172.21.15.88:0/3798012535 <== mon.1 v1:172.21.15.192:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 764+0+0 (unknown 58979205 0 0) 0x7fc7f8004eb0 con 0x7fc80c13eec0 2024-06-03T09:06:54.654 INFO:teuthology.orchestra.run.smithi088.stderr:2024-06-03T09:06:54.653+0000 7fc808ff9640 1 -- 172.21.15.88:0/3798012535 --> v1:172.21.15.192:6789/0 -- auth(proto 2 165 bytes epoch 0) v1 -- 0x7fc80c137c50 con 0x7fc80c13eec0 2024-06-03T09:06:54.654 INFO:teuthology.orchestra.run.smithi088.stderr:2024-06-03T09:06:54.653+0000 7fc808ff9640 1 -- 172.21.15.88:0/3798012535 <== mon.1 v1:172.21.15.192:6789/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 2234374065 0 0) 0x7fc8000017d0 con 0x7fc80c13eec0 2024-06-03T09:06:54.654 INFO:teuthology.orchestra.run.smithi088.stderr:2024-06-03T09:06:54.653+0000 7fc808ff9640 10 monclient(hunting): handle_monmap mon_map magic: 0 v1 2024-06-03T09:06:54.654 INFO:teuthology.orchestra.run.smithi088.stderr:2024-06-03T09:06:54.653+0000 7fc808ff9640 10 monclient(hunting): got monmap 1 from mon.b (according to old e1) 2024-06-03T09:06:54.654 INFO:teuthology.orchestra.run.smithi088.stderr:2024-06-03T09:06:54.653+0000 7fc808ff9640 10 monclient(hunting): dump: 2024-06-03T09:06:54.654 INFO:teuthology.orchestra.run.smithi088.stderr:epoch 1 2024-06-03T09:06:54.654 INFO:teuthology.orchestra.run.smithi088.stderr:fsid ec3c3fdd-3ac6-4d7d-9412-683130ed8b04 2024-06-03T09:06:54.654 INFO:teuthology.orchestra.run.smithi088.stderr:last_changed 2024-06-03T08:34:31.885125+0000 2024-06-03T09:06:54.654 INFO:teuthology.orchestra.run.smithi088.stderr:created 2024-06-03T08:34:31.885125+0000 2024-06-03T09:06:54.654 INFO:teuthology.orchestra.run.smithi088.stderr:min_mon_release 18 (reef) 2024-06-03T09:06:54.654 INFO:teuthology.orchestra.run.smithi088.stderr:election_strategy: 3 2024-06-03T09:06:54.655 INFO:teuthology.orchestra.run.smithi088.stderr:0: v1:172.21.15.88:6789/0 mon.a 2024-06-03T09:06:54.655 INFO:teuthology.orchestra.run.smithi088.stderr:1: v1:172.21.15.192:6789/0 mon.b 2024-06-03T09:06:54.655 INFO:teuthology.orchestra.run.smithi088.stderr:2: v1:172.21.15.88:6790/0 mon.c 2024-06-03T09:06:54.655 INFO:teuthology.orchestra.run.smithi088.stderr: 2024-06-03T09:06:54.659 INFO:teuthology.orchestra.run.smithi088.stderr:2024-06-03T09:06:54.658+0000 7fc808ff9640 1 -- 172.21.15.88:0/3798012535 <== mon.2 v1:172.21.15.88:6790/0 4 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 580+0+0 (unknown 981801547 0 0) 0x7fc80c13f380 con 0x7fc80c1450f0 2024-06-03T09:06:54.659 INFO:teuthology.orchestra.run.smithi088.stderr:2024-06-03T09:06:54.658+0000 7fc808ff9640 1 -- 172.21.15.88:0/3798012535 >> v1:172.21.15.192:6789/0 conn(0x7fc80c13eec0 legacy=0x7fc80c136e30 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-06-03T09:06:54.659 INFO:teuthology.orchestra.run.smithi088.stderr:2024-06-03T09:06:54.658+0000 7fc808ff9640 1 -- 172.21.15.88:0/3798012535 >> v1:172.21.15.88:6789/0 conn(0x7fc80c0a8960 legacy=0x7fc80c1d6870 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-06-03T09:06:54.660 INFO:teuthology.orchestra.run.smithi088.stderr:2024-06-03T09:06:54.658+0000 7fc808ff9640 10 monclient: _finish_hunting 0 2024-06-03T09:06:54.660 INFO:teuthology.orchestra.run.smithi088.stderr:2024-06-03T09:06:54.658+0000 7fc808ff9640 1 monclient: found mon.c 2024-06-03T09:06:54.660 INFO:teuthology.orchestra.run.smithi088.stderr:2024-06-03T09:06:54.658+0000 7fc808ff9640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-06-03T09:06:54.660 INFO:teuthology.orchestra.run.smithi088.stderr:2024-06-03T09:06:54.658+0000 7fc808ff9640 10 monclient: _send_mon_message to mon.c at v1:172.21.15.88:6790/0 2024-06-03T09:06:54.660 INFO:teuthology.orchestra.run.smithi088.stderr:2024-06-03T09:06:54.658+0000 7fc808ff9640 1 -- 172.21.15.88:0/3798012535 --> v1:172.21.15.88:6790/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fc80c141170 con 0x7fc80c1450f0 2024-06-03T09:06:54.660 INFO:teuthology.orchestra.run.smithi088.stderr:2024-06-03T09:06:54.658+0000 7fc808ff9640 10 monclient: _finish_auth 0 2024-06-03T09:06:54.660 INFO:teuthology.orchestra.run.smithi088.stderr:2024-06-03T09:06:54.658+0000 7fc808ff9640 10 monclient: _check_auth_tickets 2024-06-03T09:06:54.660 INFO:teuthology.orchestra.run.smithi088.stderr:2024-06-03T09:06:54.659+0000 7fc808ff9640 20 monclient: _check_auth_rotating not needed by client.admin 2024-06-03T09:06:54.660 INFO:teuthology.orchestra.run.smithi088.stderr:2024-06-03T09:06:54.659+0000 7fc8128d7640 5 monclient: authenticate success, global_id 30239 2024-06-03T09:06:54.660 INFO:teuthology.orchestra.run.smithi088.stderr:2024-06-03T09:06:54.659+0000 7fc8128d7640 10 monclient: _renew_subs 2024-06-03T09:06:54.660 INFO:teuthology.orchestra.run.smithi088.stderr:2024-06-03T09:06:54.659+0000 7fc8128d7640 10 monclient: _send_mon_message to mon.c at v1:172.21.15.88:6790/0 2024-06-03T09:06:54.660 INFO:teuthology.orchestra.run.smithi088.stderr:2024-06-03T09:06:54.659+0000 7fc8128d7640 1 -- 172.21.15.88:0/3798012535 --> v1:172.21.15.88:6790/0 -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fc80c142250 con 0x7fc80c1450f0 2024-06-03T09:06:54.660 INFO:teuthology.orchestra.run.smithi088.stderr:2024-06-03T09:06:54.659+0000 7fc8128d7640 10 client.30239.objecter _maybe_request_map subscribing (onetime) to next osd map 2024-06-03T09:06:54.661 INFO:teuthology.orchestra.run.smithi088.stderr:2024-06-03T09:06:54.659+0000 7fc8128d7640 10 monclient: _renew_subs 2024-06-03T09:06:54.661 INFO:teuthology.orchestra.run.smithi088.stderr:2024-06-03T09:06:54.659+0000 7fc8128d7640 10 monclient: _send_mon_message to mon.c at v1:172.21.15.88:6790/0 2024-06-03T09:06:54.661 INFO:teuthology.orchestra.run.smithi088.stderr:2024-06-03T09:06:54.659+0000 7fc8128d7640 1 -- 172.21.15.88:0/3798012535 --> v1:172.21.15.88:6790/0 -- mon_subscribe({osdmap=0}) v3 -- 0x7fc80c0ab4f0 con 0x7fc80c1450f0 2024-06-03T09:06:54.661 INFO:teuthology.orchestra.run.smithi088.stderr:2024-06-03T09:06:54.659+0000 7fc8128d7640 1 librados: init done 2024-06-03T09:06:54.661 INFO:teuthology.orchestra.run.smithi088.stderr:2024-06-03T09:06:54.659+0000 7fc8128d7640 10 start_mon_command cmd=[{"prefix": "get_command_descriptions"}] 2024-06-03T09:06:54.661 INFO:teuthology.orchestra.run.smithi088.stderr:2024-06-03T09:06:54.659+0000 7fc8128d7640 10 monclient: _send_command 1 [{"prefix": "get_command_descriptions"}] 2024-06-03T09:06:54.661 INFO:teuthology.orchestra.run.smithi088.stderr:2024-06-03T09:06:54.659+0000 7fc8128d7640 10 monclient: _send_mon_message to mon.c at v1:172.21.15.88:6790/0 2024-06-03T09:06:54.661 INFO:teuthology.orchestra.run.smithi088.stderr:2024-06-03T09:06:54.659+0000 7fc8128d7640 1 -- 172.21.15.88:0/3798012535 --> v1:172.21.15.88:6790/0 -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fc80c0a9e60 con 0x7fc80c1450f0 2024-06-03T09:06:54.941 INFO:teuthology.orchestra.run.smithi088.stderr:2024-06-03T09:06:54.941+0000 7fc808ff9640 1 -- 172.21.15.88:0/3798012535 <== mon.2 v1:172.21.15.88:6790/0 5 ==== config(0 keys) v1 ==== 4+0+0 (unknown 0 0 0) 0x7fc804004e10 con 0x7fc80c1450f0 2024-06-03T09:06:54.941 INFO:teuthology.orchestra.run.smithi088.stderr:2024-06-03T09:06:54.941+0000 7fc808ff9640 10 monclient: handle_config config(0 keys) v1 2024-06-03T09:06:54.941 INFO:teuthology.orchestra.run.smithi088.stderr:2024-06-03T09:06:54.941+0000 7fc808ff9640 1 -- 172.21.15.88:0/3798012535 <== mon.2 v1:172.21.15.88:6790/0 6 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 2234374065 0 0) 0x7fc804005120 con 0x7fc80c1450f0 2024-06-03T09:06:54.941 INFO:teuthology.orchestra.run.smithi088.stderr:2024-06-03T09:06:54.941+0000 7fc808ff9640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-06-03T09:06:54.941 INFO:teuthology.orchestra.run.smithi088.stderr:2024-06-03T09:06:54.941+0000 7fc808ff9640 10 monclient: got monmap 1 from mon.c (according to old e1) 2024-06-03T09:06:54.942 INFO:teuthology.orchestra.run.smithi088.stderr:2024-06-03T09:06:54.941+0000 7fc808ff9640 10 monclient: dump: 2024-06-03T09:06:54.942 INFO:teuthology.orchestra.run.smithi088.stderr:epoch 1 2024-06-03T09:06:54.942 INFO:teuthology.orchestra.run.smithi088.stderr:fsid ec3c3fdd-3ac6-4d7d-9412-683130ed8b04 2024-06-03T09:06:54.942 INFO:teuthology.orchestra.run.smithi088.stderr:last_changed 2024-06-03T08:34:31.885125+0000 2024-06-03T09:06:54.942 INFO:teuthology.orchestra.run.smithi088.stderr:created 2024-06-03T08:34:31.885125+0000 2024-06-03T09:06:54.942 INFO:teuthology.orchestra.run.smithi088.stderr:min_mon_release 18 (reef) 2024-06-03T09:06:54.942 INFO:teuthology.orchestra.run.smithi088.stderr:election_strategy: 3 2024-06-03T09:06:54.942 INFO:teuthology.orchestra.run.smithi088.stderr:0: v1:172.21.15.88:6789/0 mon.a 2024-06-03T09:06:54.942 INFO:teuthology.orchestra.run.smithi088.stderr:1: v1:172.21.15.192:6789/0 mon.b 2024-06-03T09:06:54.942 INFO:teuthology.orchestra.run.smithi088.stderr:2: v1:172.21.15.88:6790/0 mon.c 2024-06-03T09:06:54.942 INFO:teuthology.orchestra.run.smithi088.stderr: 2024-06-03T09:06:54.942 INFO:teuthology.orchestra.run.smithi088.stderr:2024-06-03T09:06:54.941+0000 7fc80a7fc640 4 set_mon_vals no callback set 2024-06-03T09:06:54.943 INFO:teuthology.orchestra.run.smithi088.stderr:2024-06-03T09:06:54.941+0000 7fc808ff9640 1 -- 172.21.15.88:0/3798012535 <== mon.2 v1:172.21.15.88:6790/0 7 ==== mgrmap(e 5) v1 ==== 89110+0+0 (unknown 873533973 0 0) 0x7fc80401aea0 con 0x7fc80c1450f0 2024-06-03T09:06:54.943 INFO:teuthology.orchestra.run.smithi088.stderr:2024-06-03T09:06:54.942+0000 7fc80bfff640 10 monclient: get_auth_request con 0x7fc7f806d470 auth_method 0 2024-06-03T09:06:54.943 INFO:teuthology.orchestra.run.smithi088.stderr:2024-06-03T09:06:54.943+0000 7fc808ff9640 10 client.30239.objecter ms_handle_connect 0x7fc7f806d470 2024-06-03T09:06:54.947 INFO:teuthology.orchestra.run.smithi088.stderr:2024-06-03T09:06:54.947+0000 7fc808ff9640 1 -- 172.21.15.88:0/3798012535 <== mon.2 v1:172.21.15.88:6790/0 8 ==== osd_map(1014..1014 src has 622..1014) v4 ==== 5229+0+0 (unknown 3230192161 0 0) 0x7fc8040545c0 con 0x7fc80c1450f0 2024-06-03T09:06:54.947 INFO:teuthology.orchestra.run.smithi088.stderr:2024-06-03T09:06:54.947+0000 7fc808ff9640 10 client.30239.objecter ms_dispatch 0x7fc80c000b90 osd_map(1014..1014 src has 622..1014) v4 2024-06-03T09:06:54.947 INFO:teuthology.orchestra.run.smithi088.stderr:2024-06-03T09:06:54.947+0000 7fc808ff9640 3 client.30239.objecter handle_osd_map got epochs [1014,1014] > 0 2024-06-03T09:06:54.947 INFO:teuthology.orchestra.run.smithi088.stderr:2024-06-03T09:06:54.947+0000 7fc808ff9640 3 client.30239.objecter handle_osd_map decoding full epoch 1014 2024-06-03T09:06:54.948 INFO:teuthology.orchestra.run.smithi088.stderr:2024-06-03T09:06:54.947+0000 7fc808ff9640 20 client.30239.objecter dump_active .. 0 homeless 2024-06-03T09:06:54.948 INFO:teuthology.orchestra.run.smithi088.stderr:2024-06-03T09:06:54.947+0000 7fc808ff9640 1 -- 172.21.15.88:0/3798012535 <== mon.2 v1:172.21.15.88:6790/0 9 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+182498 (unknown 1092875540 0 3016760700) 0x7fc804050170 con 0x7fc80c1450f0 2024-06-03T09:06:54.948 INFO:teuthology.orchestra.run.smithi088.stderr:2024-06-03T09:06:54.947+0000 7fc808ff9640 10 monclient: handle_mon_command_ack 1 [{"prefix": "get_command_descriptions"}] 2024-06-03T09:06:54.948 INFO:teuthology.orchestra.run.smithi088.stderr:2024-06-03T09:06:54.947+0000 7fc808ff9640 10 monclient: _finish_command 1 = system:0 2024-06-03T09:06:55.145 INFO:teuthology.orchestra.run.smithi088.stderr:2024-06-03T09:06:55.144+0000 7fc8128d7640 10 start_mon_command cmd=[{"prefix": "osd dump", "format": "json"}] 2024-06-03T09:06:55.145 INFO:teuthology.orchestra.run.smithi088.stderr:2024-06-03T09:06:55.144+0000 7fc8128d7640 10 monclient: _send_command 2 [{"prefix": "osd dump", "format": "json"}] 2024-06-03T09:06:55.145 INFO:teuthology.orchestra.run.smithi088.stderr:2024-06-03T09:06:55.144+0000 7fc8128d7640 10 monclient: _send_mon_message to mon.c at v1:172.21.15.88:6790/0 2024-06-03T09:06:55.145 INFO:teuthology.orchestra.run.smithi088.stderr:2024-06-03T09:06:55.144+0000 7fc8128d7640 1 -- 172.21.15.88:0/3798012535 --> v1:172.21.15.88:6790/0 -- mon_command({"prefix": "osd dump", "format": "json"} v 0) v1 -- 0x7fc80c04ea60 con 0x7fc80c1450f0 2024-06-03T09:06:55.193 INFO:teuthology.orchestra.run.smithi088.stderr:2024-06-03T09:06:55.193+0000 7fc808ff9640 1 -- 172.21.15.88:0/3798012535 <== mon.2 v1:172.21.15.88:6790/0 10 ==== mon_command_ack([{"prefix": "osd dump", "format": "json"}]=0 v1014) v1 ==== 74+0+15448 (unknown 3937929044 0 3324103678) 0x7fc8040520a0 con 0x7fc80c1450f0 2024-06-03T09:06:55.194 INFO:teuthology.orchestra.run.smithi088.stderr:2024-06-03T09:06:55.193+0000 7fc808ff9640 10 monclient: handle_mon_command_ack 2 [{"prefix": "osd dump", "format": "json"}] 2024-06-03T09:06:55.194 INFO:teuthology.orchestra.run.smithi088.stderr:2024-06-03T09:06:55.193+0000 7fc808ff9640 10 monclient: _finish_command 2 = system:0 2024-06-03T09:06:55.194 INFO:teuthology.orchestra.run.smithi088.stdout: 2024-06-03T09:06:55.194 INFO:teuthology.orchestra.run.smithi088.stdout:{"epoch":1014,"fsid":"ec3c3fdd-3ac6-4d7d-9412-683130ed8b04","created":"2024-06-03T08:35:26.799638+0000","modified":"2024-06-03T09:06:46.034698+0000","last_up_change":"2024-06-03T09:06:44.859078+0000","last_in_change":"2024-06-03T08:35:32.183068+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":100,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":260,"max_osd":8,"require_min_compat_client":"luminous","min_compat_client":"jewel","require_osd_release":"reef","allow_crimson":false,"pools":[{"pool":1,"pool_name":".mgr","create_time":"2024-06-03T08:36:15.192388+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":1,"pg_placement_num":1,"pg_placement_num_target":1,"pg_num_target":1,"pg_num_pending":1,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"19","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_max":32,"pg_num_min":1},"application_metadata":{"mgr":{}},"read_balance":{"score_acting":8,"score_stable":8,"optimal_score":0.25,"raw_score_acting":2,"raw_score_stable":2,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":2,"pool_name":"rbd","create_time":"2024-06-03T08:36:28.156336+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":8,"pg_placement_num":8,"pg_placement_num_target":8,"pg_num_target":8,"pg_num_pending":8,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"26","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{},"application_metadata":{"rbd":{}},"read_balance":{"score_acting":4.9800000190734863,"score_stable":4.9800000190734863,"optimal_score":0.87999999523162842,"raw_score_acting":4.380000114440918,"raw_score_stable":4.380000114440918,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":17,"pool_name":"ceph_test_rados_api_asio","create_time":"2024-06-03T08:37:54.698183+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":32,"pg_placement_num":32,"pg_placement_num_target":32,"pg_num_target":32,"pg_num_pending":32,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"66","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{},"application_metadata":{},"read_balance":{"score_acting":1.75,"score_stable":1.75,"optimal_score":1,"raw_score_acting":1.75,"raw_score_stable":1.75,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}}],"osds":[{"osd":0,"uuid":"b25ab3bb-3587-45d0-991a-6d41bb529b37","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":14,"up_thru":1013,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.88:6808","nonce":35066}]},"cluster_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.88:6809","nonce":35066}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.88:6811","nonce":35066}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.88:6810","nonce":35066}]},"public_addr":"172.21.15.88:6808/35066","cluster_addr":"172.21.15.88:6809/35066","heartbeat_back_addr":"172.21.15.88:6811/35066","heartbeat_front_addr":"172.21.15.88:6810/35066","state":["exists","up"]},{"osd":1,"uuid":"00d160e7-9773-41d8-8d17-21e70620a0fd","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":14,"up_thru":1003,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.88:6800","nonce":35067}]},"cluster_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.88:6801","nonce":35067}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.88:6804","nonce":35067}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.88:6803","nonce":35067}]},"public_addr":"172.21.15.88:6800/35067","cluster_addr":"172.21.15.88:6801/35067","heartbeat_back_addr":"172.21.15.88:6804/35067","heartbeat_front_addr":"172.21.15.88:6803/35067","state":["exists","up"]},{"osd":2,"uuid":"addfb2a7-2bde-4cc3-98cf-17fd2bdfa3a9","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":20,"up_thru":1013,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.88:6802","nonce":35065}]},"cluster_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.88:6805","nonce":35065}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.88:6807","nonce":35065}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.88:6806","nonce":35065}]},"public_addr":"172.21.15.88:6802/35065","cluster_addr":"172.21.15.88:6805/35065","heartbeat_back_addr":"172.21.15.88:6807/35065","heartbeat_front_addr":"172.21.15.88:6806/35065","state":["exists","up"]},{"osd":3,"uuid":"6c5770b0-c93c-4101-8c53-bd761b8abcde","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":14,"up_thru":1013,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.88:6812","nonce":35068}]},"cluster_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.88:6813","nonce":35068}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.88:6815","nonce":35068}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.88:6814","nonce":35068}]},"public_addr":"172.21.15.88:6812/35068","cluster_addr":"172.21.15.88:6813/35068","heartbeat_back_addr":"172.21.15.88:6815/35068","heartbeat_front_addr":"172.21.15.88:6814/35068","state":["exists","up"]},{"osd":4,"uuid":"f001b3cb-2081-428b-bac3-b3df3c19b978","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":20,"up_thru":1013,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.192:6809","nonce":34785}]},"cluster_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.192:6810","nonce":34785}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.192:6812","nonce":34785}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.192:6811","nonce":34785}]},"public_addr":"172.21.15.192:6809/34785","cluster_addr":"172.21.15.192:6810/34785","heartbeat_back_addr":"172.21.15.192:6812/34785","heartbeat_front_addr":"172.21.15.192:6811/34785","state":["exists","up"]},{"osd":5,"uuid":"3cc263e9-e9bf-45e0-a3b7-83a76469eb70","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":20,"up_thru":1013,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.192:6805","nonce":34786}]},"cluster_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.192:6806","nonce":34786}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.192:6808","nonce":34786}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.192:6807","nonce":34786}]},"public_addr":"172.21.15.192:6805/34786","cluster_addr":"172.21.15.192:6806/34786","heartbeat_back_addr":"172.21.15.192:6808/34786","heartbeat_front_addr":"172.21.15.192:6807/34786","state":["exists","up"]},{"osd":6,"uuid":"82c83316-809f-4901-a5ed-34b6a075b393","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":21,"last_clean_end":1011,"up_from":1013,"up_thru":1013,"down_at":999,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.192:6813","nonce":34788}]},"cluster_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.192:6814","nonce":2034788}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.192:6816","nonce":34788}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.192:6815","nonce":34788}]},"public_addr":"172.21.15.192:6813/34788","cluster_addr":"172.21.15.192:6814/2034788","heartbeat_back_addr":"172.21.15.192:6816/34788","heartbeat_front_addr":"172.21.15.192:6815/34788","state":["exists","up"]},{"osd":7,"uuid":"aafe0716-1102-4eae-aac6-0f81bb154e7d","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":20,"up_thru":1003,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.192:6801","nonce":34787}]},"cluster_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.192:6802","nonce":34787}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.192:6804","nonce":34787}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.192:6803","nonce":34787}]},"public_addr":"172.21.15.192:6801/34787","cluster_addr":"172.21.15.192:6802/34787","heartbeat_back_addr":"172.21.15.192:6804/34787","heartbeat_front_addr":"172.21.15.192:6803/34787","state":["exists","up"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2024-06-03T08:36:10.027533+0000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2024-06-03T08:36:09.853335+0000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2024-06-03T08:36:10.789609+0000","dead_epoch":0},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2024-06-03T08:36:09.824468+0000","dead_epoch":0},{"osd":4,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2024-06-03T08:36:10.117232+0000","dead_epoch":0},{"osd":5,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2024-06-03T08:36:10.469993+0000","dead_epoch":0},{"osd":6,"down_stamp":"2024-06-03T09:06:02.454918+0000","laggy_probability":0.50999999046325684,"laggy_interval":22,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2024-06-03T08:36:10.789874+0000","dead_epoch":1010},{"osd":7,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2024-06-03T08:36:09.541793+0000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[],"pg_upmap_primaries":[],"pg_temp":[],"primary_temp":[],"blocklist":{},"range_blocklist":{},"erasure_code_profiles":{"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","technique":"reed_sol_van"}},"removed_snaps_queue":[{"pool":9,"snaps":[{"begin":1,"length":1},{"begin":3,"length":1},{"begin":5,"length":1},{"begin":7,"length":1}]},{"pool":18,"snaps":[{"begin":5,"length":1},{"begin":7,"length":1},{"begin":9,"length":2}]},{"pool":50,"snaps":[{"begin":11,"length":2}]},{"pool":53,"snaps":[{"begin":23,"length":1}]},{"pool":79,"snaps":[{"begin":3,"length":1},{"begin":5,"length":1},{"begin":7,"length":1}]},{"pool":87,"snaps":[{"begin":2,"length":2},{"begin":6,"length":2}]},{"pool":93,"snaps":[{"begin":2,"length":1}]},{"pool":97,"snaps":[{"begin":3,"length":1},{"begin":5,"length":1},{"begin":7,"length":1}]},{"pool":99,"snaps":[{"begin":2,"length":1}]},{"pool":127,"snaps":[{"begin":2,"length":2},{"begin":6,"length":2},{"begin":10,"length":2}]},{"pool":148,"snaps":[{"begin":4,"length":4}]},{"pool":179,"snaps":[{"begin":12,"length":1}]},{"pool":189,"snaps":[{"begin":14,"length":1}]},{"pool":194,"snaps":[{"begin":17,"length":1}]},{"pool":195,"snaps":[{"begin":19,"length":2}]},{"pool":198,"snaps":[{"begin":24,"length":1}]},{"pool":241,"snaps":[{"begin":2,"length":1}]},{"pool":242,"snaps":[{"begin":4,"length":1}]},{"pool":245,"snaps":[{"begin":6,"length":1}]},{"pool":249,"snaps":[{"begin":9,"length":1}]}],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2024-06-03T09:06:55.196 INFO:teuthology.orchestra.run.smithi088.stderr:2024-06-03T09:06:55.196+0000 7fc8128d7640 10 librados: watch_flush enter 2024-06-03T09:06:55.196 INFO:teuthology.orchestra.run.smithi088.stderr:2024-06-03T09:06:55.196+0000 7fc8128d7640 10 librados: watch_flush exit 2024-06-03T09:06:55.196 INFO:teuthology.orchestra.run.smithi088.stderr:2024-06-03T09:06:55.196+0000 7fc8128d7640 20 client.30239.objecter shutdown clearing up homeless session... 2024-06-03T09:06:55.196 INFO:teuthology.orchestra.run.smithi088.stderr:2024-06-03T09:06:55.196+0000 7fc8128d7640 10 client.30239.objecter successfully canceled tick 2024-06-03T09:06:55.197 INFO:teuthology.orchestra.run.smithi088.stderr:2024-06-03T09:06:55.196+0000 7fc8128d7640 1 -- 172.21.15.88:0/3798012535 >> v1:172.21.15.192:6800/34229 conn(0x7fc7f806d470 legacy=0x7fc7f806f930 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-06-03T09:06:55.197 INFO:teuthology.orchestra.run.smithi088.stderr:2024-06-03T09:06:55.196+0000 7fc8128d7640 10 monclient: shutdown 2024-06-03T09:06:55.197 INFO:teuthology.orchestra.run.smithi088.stderr:2024-06-03T09:06:55.196+0000 7fc8128d7640 20 monclient: shutdown discarding 0 pending message(s) 2024-06-03T09:06:55.197 INFO:teuthology.orchestra.run.smithi088.stderr:2024-06-03T09:06:55.196+0000 7fc8128d7640 1 -- 172.21.15.88:0/3798012535 >> v1:172.21.15.88:6790/0 conn(0x7fc80c1450f0 legacy=0x7fc80c137540 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-06-03T09:06:55.197 INFO:teuthology.orchestra.run.smithi088.stderr:2024-06-03T09:06:55.196+0000 7fc8128d7640 1 -- 172.21.15.88:0/3798012535 shutdown_connections 2024-06-03T09:06:55.197 INFO:teuthology.orchestra.run.smithi088.stderr:2024-06-03T09:06:55.196+0000 7fc8128d7640 1 -- 172.21.15.88:0/3798012535 >> 172.21.15.88:0/3798012535 conn(0x7fc80c0b70d0 msgr2=0x7fc80c0a8d40 unknown :-1 s=STATE_NONE l=0).mark_down 2024-06-03T09:06:55.197 INFO:teuthology.orchestra.run.smithi088.stderr:2024-06-03T09:06:55.196+0000 7fc8128d7640 1 -- 172.21.15.88:0/3798012535 shutdown_connections 2024-06-03T09:06:55.197 INFO:teuthology.orchestra.run.smithi088.stderr:2024-06-03T09:06:55.196+0000 7fc8128d7640 1 -- 172.21.15.88:0/3798012535 wait complete. 2024-06-03T09:06:55.197 INFO:teuthology.orchestra.run.smithi088.stderr:2024-06-03T09:06:55.196+0000 7fc8128d7640 1 librados: shutdown 2024-06-03T09:06:56.211 INFO:tasks.ceph:Scrubbing osd.0 2024-06-03T09:06:56.212 DEBUG:teuthology.orchestra.run.smithi088:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph tell osd.0 config set osd_debug_deep_scrub_sleep 0 2024-06-03T09:06:56.314 INFO:teuthology.orchestra.run.smithi088.stderr:2024-06-03T09:06:56.314+0000 7f1b45081640 10 monclient: get_monmap_and_config 2024-06-03T09:06:56.314 INFO:teuthology.orchestra.run.smithi088.stderr:2024-06-03T09:06:56.314+0000 7f1b45081640 10 monclient: build_initial_monmap 2024-06-03T09:06:56.314 INFO:teuthology.orchestra.run.smithi088.stderr:2024-06-03T09:06:56.314+0000 7f1b45081640 10 monclient: monmap: 2024-06-03T09:06:56.314 INFO:teuthology.orchestra.run.smithi088.stderr:epoch 0 2024-06-03T09:06:56.314 INFO:teuthology.orchestra.run.smithi088.stderr:fsid ec3c3fdd-3ac6-4d7d-9412-683130ed8b04 2024-06-03T09:06:56.314 INFO:teuthology.orchestra.run.smithi088.stderr:last_changed 2024-06-03T09:06:56.315110+0000 2024-06-03T09:06:56.314 INFO:teuthology.orchestra.run.smithi088.stderr:created 2024-06-03T09:06:56.315110+0000 2024-06-03T09:06:56.314 INFO:teuthology.orchestra.run.smithi088.stderr:min_mon_release 0 (unknown) 2024-06-03T09:06:56.315 INFO:teuthology.orchestra.run.smithi088.stderr:election_strategy: 3 2024-06-03T09:06:56.315 INFO:teuthology.orchestra.run.smithi088.stderr:0: v1:172.21.15.88:6789/0 mon.noname-a 2024-06-03T09:06:56.315 INFO:teuthology.orchestra.run.smithi088.stderr:1: v1:172.21.15.192:6789/0 mon.noname-c 2024-06-03T09:06:56.315 INFO:teuthology.orchestra.run.smithi088.stderr:2: v1:172.21.15.88:6790/0 mon.noname-b 2024-06-03T09:06:56.315 INFO:teuthology.orchestra.run.smithi088.stderr: 2024-06-03T09:06:56.316 INFO:teuthology.orchestra.run.smithi088.stderr:2024-06-03T09:06:56.314+0000 7f1b45081640 1 Processor -- start 2024-06-03T09:06:56.316 INFO:teuthology.orchestra.run.smithi088.stderr:2024-06-03T09:06:56.314+0000 7f1b45081640 1 -- start start 2024-06-03T09:06:56.316 INFO:teuthology.orchestra.run.smithi088.stderr:2024-06-03T09:06:56.314+0000 7f1b45081640 10 monclient: init 2024-06-03T09:06:56.317 INFO:teuthology.orchestra.run.smithi088.stderr:2024-06-03T09:06:56.315+0000 7f1b45081640 10 monclient: _reopen_session rank -1 2024-06-03T09:06:56.317 INFO:teuthology.orchestra.run.smithi088.stderr:2024-06-03T09:06:56.315+0000 7f1b45081640 10 monclient: _add_conns ranks=[2,1,0] 2024-06-03T09:06:56.317 INFO:teuthology.orchestra.run.smithi088.stderr:2024-06-03T09:06:56.315+0000 7f1b45081640 10 monclient(hunting): picked mon.noname-b con 0x7f1b400a8a60 addr v1:172.21.15.88:6790/0 2024-06-03T09:06:56.317 INFO:teuthology.orchestra.run.smithi088.stderr:2024-06-03T09:06:56.315+0000 7f1b45081640 10 monclient(hunting): picked mon.noname-c con 0x7f1b400b1b70 addr v1:172.21.15.192:6789/0 2024-06-03T09:06:56.317 INFO:teuthology.orchestra.run.smithi088.stderr:2024-06-03T09:06:56.315+0000 7f1b45081640 10 monclient(hunting): picked mon.noname-a con 0x7f1b4014ae00 addr v1:172.21.15.88:6789/0 2024-06-03T09:06:56.317 INFO:teuthology.orchestra.run.smithi088.stderr:2024-06-03T09:06:56.315+0000 7f1b45081640 1 -- --> v1:172.21.15.88:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f1b400b18c0 con 0x7f1b4014ae00 2024-06-03T09:06:56.317 INFO:teuthology.orchestra.run.smithi088.stderr:2024-06-03T09:06:56.315+0000 7f1b45081640 1 -- --> v1:172.21.15.192:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f1b400a92a0 con 0x7f1b400b1b70 2024-06-03T09:06:56.317 INFO:teuthology.orchestra.run.smithi088.stderr:2024-06-03T09:06:56.315+0000 7f1b45081640 1 -- --> v1:172.21.15.88:6790/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f1b400b1fb0 con 0x7f1b400a8a60 2024-06-03T09:06:56.317 INFO:teuthology.orchestra.run.smithi088.stderr:2024-06-03T09:06:56.315+0000 7f1b45081640 10 monclient(hunting): _renew_subs 2024-06-03T09:06:56.317 INFO:teuthology.orchestra.run.smithi088.stderr:2024-06-03T09:06:56.315+0000 7f1b45081640 10 monclient(hunting): authenticate will time out at 3072.125453s 2024-06-03T09:06:56.317 INFO:teuthology.orchestra.run.smithi088.stderr:2024-06-03T09:06:56.315+0000 7f1b3ed9d640 1 --1- >> v1:172.21.15.88:6790/0 conn(0x7f1b400a8a60 0x7f1b40141d70 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.88:6790/0 says I am v1:172.21.15.88:52422/0 (socket says 172.21.15.88:52422) 2024-06-03T09:06:56.317 INFO:teuthology.orchestra.run.smithi088.stderr:2024-06-03T09:06:56.315+0000 7f1b3ed9d640 1 -- 172.21.15.88:0/4184876467 learned_addr learned my addr 172.21.15.88:0/4184876467 (peer_addr_for_me v1:172.21.15.88:0/0) 2024-06-03T09:06:56.318 INFO:teuthology.orchestra.run.smithi088.stderr:2024-06-03T09:06:56.318+0000 7f1b3dd9b640 1 -- 172.21.15.88:0/4184876467 <== mon.1 v1:172.21.15.192:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 2129173367 0 0) 0x7f1b400a92a0 con 0x7f1b400b1b70 2024-06-03T09:06:56.319 INFO:teuthology.orchestra.run.smithi088.stderr:2024-06-03T09:06:56.318+0000 7f1b3dd9b640 10 monclient(hunting): _negotiate 2024-06-03T09:06:56.319 INFO:teuthology.orchestra.run.smithi088.stderr:2024-06-03T09:06:56.318+0000 7f1b3dd9b640 10 monclient(hunting): _init_auth method 2 2024-06-03T09:06:56.319 INFO:teuthology.orchestra.run.smithi088.stderr:2024-06-03T09:06:56.318+0000 7f1b3dd9b640 10 monclient(hunting): _init_auth creating new auth 2024-06-03T09:06:56.319 INFO:teuthology.orchestra.run.smithi088.stderr:2024-06-03T09:06:56.318+0000 7f1b3dd9b640 10 monclient(hunting): my global_id is 30307 2024-06-03T09:06:56.319 INFO:teuthology.orchestra.run.smithi088.stderr:2024-06-03T09:06:56.318+0000 7f1b3dd9b640 1 -- 172.21.15.88:0/4184876467 --> v1:172.21.15.192:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f1b28003520 con 0x7f1b400b1b70 2024-06-03T09:06:56.319 INFO:teuthology.orchestra.run.smithi088.stderr:2024-06-03T09:06:56.318+0000 7f1b3dd9b640 1 -- 172.21.15.88:0/4184876467 <== mon.2 v1:172.21.15.88:6790/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 2155870776 0 0) 0x7f1b400b1fb0 con 0x7f1b400a8a60 2024-06-03T09:06:56.319 INFO:teuthology.orchestra.run.smithi088.stderr:2024-06-03T09:06:56.318+0000 7f1b3dd9b640 10 monclient(hunting): _negotiate 2024-06-03T09:06:56.319 INFO:teuthology.orchestra.run.smithi088.stderr:2024-06-03T09:06:56.318+0000 7f1b3dd9b640 10 monclient(hunting): _init_auth method 2 2024-06-03T09:06:56.319 INFO:teuthology.orchestra.run.smithi088.stderr:2024-06-03T09:06:56.318+0000 7f1b3dd9b640 10 monclient(hunting): _init_auth creating new auth 2024-06-03T09:06:56.319 INFO:teuthology.orchestra.run.smithi088.stderr:2024-06-03T09:06:56.318+0000 7f1b3dd9b640 10 monclient(hunting): my global_id is 30242 2024-06-03T09:06:56.319 INFO:teuthology.orchestra.run.smithi088.stderr:2024-06-03T09:06:56.318+0000 7f1b3dd9b640 1 -- 172.21.15.88:0/4184876467 --> v1:172.21.15.88:6790/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f1b400a92a0 con 0x7f1b400a8a60 2024-06-03T09:06:56.319 INFO:teuthology.orchestra.run.smithi088.stderr:2024-06-03T09:06:56.318+0000 7f1b3dd9b640 1 -- 172.21.15.88:0/4184876467 <== mon.0 v1:172.21.15.88:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 447342560 0 0) 0x7f1b400b18c0 con 0x7f1b4014ae00 2024-06-03T09:06:56.319 INFO:teuthology.orchestra.run.smithi088.stderr:2024-06-03T09:06:56.318+0000 7f1b3dd9b640 10 monclient(hunting): _negotiate 2024-06-03T09:06:56.319 INFO:teuthology.orchestra.run.smithi088.stderr:2024-06-03T09:06:56.318+0000 7f1b3dd9b640 10 monclient(hunting): _init_auth method 2 2024-06-03T09:06:56.320 INFO:teuthology.orchestra.run.smithi088.stderr:2024-06-03T09:06:56.318+0000 7f1b3dd9b640 10 monclient(hunting): _init_auth creating new auth 2024-06-03T09:06:56.320 INFO:teuthology.orchestra.run.smithi088.stderr:2024-06-03T09:06:56.318+0000 7f1b3dd9b640 10 monclient(hunting): my global_id is 30738 2024-06-03T09:06:56.320 INFO:teuthology.orchestra.run.smithi088.stderr:2024-06-03T09:06:56.318+0000 7f1b3dd9b640 1 -- 172.21.15.88:0/4184876467 --> v1:172.21.15.88:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f1b400b1fb0 con 0x7f1b4014ae00 2024-06-03T09:06:56.321 INFO:teuthology.orchestra.run.smithi088.stderr:2024-06-03T09:06:56.321+0000 7f1b3dd9b640 1 -- 172.21.15.88:0/4184876467 <== mon.1 v1:172.21.15.192:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 214+0+0 (unknown 1538759584 0 0) 0x7f1b28003520 con 0x7f1b400b1b70 2024-06-03T09:06:56.321 INFO:teuthology.orchestra.run.smithi088.stderr:2024-06-03T09:06:56.321+0000 7f1b3dd9b640 1 -- 172.21.15.88:0/4184876467 >> v1:172.21.15.88:6790/0 conn(0x7f1b400a8a60 legacy=0x7f1b40141d70 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-06-03T09:06:56.321 INFO:teuthology.orchestra.run.smithi088.stderr:2024-06-03T09:06:56.321+0000 7f1b3dd9b640 1 -- 172.21.15.88:0/4184876467 >> v1:172.21.15.88:6789/0 conn(0x7f1b4014ae00 legacy=0x7f1b4014d1f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-06-03T09:06:56.322 INFO:teuthology.orchestra.run.smithi088.stderr:2024-06-03T09:06:56.321+0000 7f1b3dd9b640 10 monclient: _finish_hunting 0 2024-06-03T09:06:56.322 INFO:teuthology.orchestra.run.smithi088.stderr:2024-06-03T09:06:56.321+0000 7f1b3dd9b640 1 monclient: found mon.noname-c 2024-06-03T09:06:56.322 INFO:teuthology.orchestra.run.smithi088.stderr:2024-06-03T09:06:56.321+0000 7f1b3dd9b640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-06-03T09:06:56.322 INFO:teuthology.orchestra.run.smithi088.stderr:2024-06-03T09:06:56.321+0000 7f1b3dd9b640 10 monclient: _send_mon_message to mon.noname-c at v1:172.21.15.192:6789/0 2024-06-03T09:06:56.322 INFO:teuthology.orchestra.run.smithi088.stderr:2024-06-03T09:06:56.321+0000 7f1b3dd9b640 1 -- 172.21.15.88:0/4184876467 --> v1:172.21.15.192:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f1b400aa070 con 0x7f1b400b1b70 2024-06-03T09:06:56.322 INFO:teuthology.orchestra.run.smithi088.stderr:2024-06-03T09:06:56.321+0000 7f1b3dd9b640 10 monclient: _finish_auth 0 2024-06-03T09:06:56.322 INFO:teuthology.orchestra.run.smithi088.stderr:2024-06-03T09:06:56.321+0000 7f1b3dd9b640 10 monclient: _check_auth_tickets 2024-06-03T09:06:56.322 INFO:teuthology.orchestra.run.smithi088.stderr:2024-06-03T09:06:56.321+0000 7f1b3dd9b640 20 monclient: _check_auth_rotating not needed by client.admin 2024-06-03T09:06:56.322 INFO:teuthology.orchestra.run.smithi088.stderr:2024-06-03T09:06:56.321+0000 7f1b3dd9b640 1 -- 172.21.15.88:0/4184876467 <== mon.1 v1:172.21.15.192:6789/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 2234374065 0 0) 0x7f1b34001610 con 0x7f1b400b1b70 2024-06-03T09:06:56.322 INFO:teuthology.orchestra.run.smithi088.stderr:2024-06-03T09:06:56.321+0000 7f1b3dd9b640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-06-03T09:06:56.322 INFO:teuthology.orchestra.run.smithi088.stderr:2024-06-03T09:06:56.321+0000 7f1b3dd9b640 10 monclient: got monmap 1 from mon.noname-c (according to old e1) 2024-06-03T09:06:56.322 INFO:teuthology.orchestra.run.smithi088.stderr:2024-06-03T09:06:56.321+0000 7f1b3dd9b640 10 monclient: dump: 2024-06-03T09:06:56.323 INFO:teuthology.orchestra.run.smithi088.stderr:epoch 1 2024-06-03T09:06:56.323 INFO:teuthology.orchestra.run.smithi088.stderr:fsid ec3c3fdd-3ac6-4d7d-9412-683130ed8b04 2024-06-03T09:06:56.323 INFO:teuthology.orchestra.run.smithi088.stderr:last_changed 2024-06-03T08:34:31.885125+0000 2024-06-03T09:06:56.323 INFO:teuthology.orchestra.run.smithi088.stderr:created 2024-06-03T08:34:31.885125+0000 2024-06-03T09:06:56.323 INFO:teuthology.orchestra.run.smithi088.stderr:min_mon_release 18 (reef) 2024-06-03T09:06:56.323 INFO:teuthology.orchestra.run.smithi088.stderr:election_strategy: 3 2024-06-03T09:06:56.323 INFO:teuthology.orchestra.run.smithi088.stderr:0: v1:172.21.15.88:6789/0 mon.a 2024-06-03T09:06:56.323 INFO:teuthology.orchestra.run.smithi088.stderr:1: v1:172.21.15.192:6789/0 mon.b 2024-06-03T09:06:56.323 INFO:teuthology.orchestra.run.smithi088.stderr:2: v1:172.21.15.88:6790/0 mon.c 2024-06-03T09:06:56.323 INFO:teuthology.orchestra.run.smithi088.stderr: 2024-06-03T09:06:56.323 INFO:teuthology.orchestra.run.smithi088.stderr:2024-06-03T09:06:56.321+0000 7f1b45081640 5 monclient: authenticate success, global_id 30307 2024-06-03T09:06:56.323 INFO:teuthology.orchestra.run.smithi088.stderr:2024-06-03T09:06:56.321+0000 7f1b45081640 20 monclient: get_monmap_and_config waiting for monmap|config 2024-06-03T09:06:56.325 INFO:teuthology.orchestra.run.smithi088.stderr:2024-06-03T09:06:56.323+0000 7f1b3dd9b640 1 -- 172.21.15.88:0/4184876467 <== mon.1 v1:172.21.15.192:6789/0 4 ==== config(0 keys) v1 ==== 4+0+0 (unknown 0 0 0) 0x7f1b34001830 con 0x7f1b400b1b70 2024-06-03T09:06:56.325 INFO:teuthology.orchestra.run.smithi088.stderr:2024-06-03T09:06:56.323+0000 7f1b3dd9b640 10 monclient: handle_config config(0 keys) v1 2024-06-03T09:06:56.325 INFO:teuthology.orchestra.run.smithi088.stderr:2024-06-03T09:06:56.323+0000 7f1b45081640 10 monclient: get_monmap_and_config success 2024-06-03T09:06:56.325 INFO:teuthology.orchestra.run.smithi088.stderr:2024-06-03T09:06:56.323+0000 7f1b3dd9b640 1 -- 172.21.15.88:0/4184876467 <== mon.1 v1:172.21.15.192:6789/0 5 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 2234374065 0 0) 0x7f1b34001b40 con 0x7f1b400b1b70 2024-06-03T09:06:56.325 INFO:teuthology.orchestra.run.smithi088.stderr:2024-06-03T09:06:56.323+0000 7f1b3dd9b640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-06-03T09:06:56.325 INFO:teuthology.orchestra.run.smithi088.stderr:2024-06-03T09:06:56.323+0000 7f1b45081640 4 set_mon_vals no callback set 2024-06-03T09:06:56.325 INFO:teuthology.orchestra.run.smithi088.stderr:2024-06-03T09:06:56.323+0000 7f1b3dd9b640 10 monclient: got monmap 1 from mon.b (according to old e1) 2024-06-03T09:06:56.325 INFO:teuthology.orchestra.run.smithi088.stderr:2024-06-03T09:06:56.323+0000 7f1b3dd9b640 10 monclient: dump: 2024-06-03T09:06:56.326 INFO:teuthology.orchestra.run.smithi088.stderr:epoch 1 2024-06-03T09:06:56.326 INFO:teuthology.orchestra.run.smithi088.stderr:fsid ec3c3fdd-3ac6-4d7d-9412-683130ed8b04 2024-06-03T09:06:56.326 INFO:teuthology.orchestra.run.smithi088.stderr:last_changed 2024-06-03T08:34:31.885125+0000 2024-06-03T09:06:56.326 INFO:teuthology.orchestra.run.smithi088.stderr:created 2024-06-03T08:34:31.885125+0000 2024-06-03T09:06:56.326 INFO:teuthology.orchestra.run.smithi088.stderr:min_mon_release 18 (reef) 2024-06-03T09:06:56.326 INFO:teuthology.orchestra.run.smithi088.stderr:election_strategy: 3 2024-06-03T09:06:56.326 INFO:teuthology.orchestra.run.smithi088.stderr:0: v1:172.21.15.88:6789/0 mon.a 2024-06-03T09:06:56.326 INFO:teuthology.orchestra.run.smithi088.stderr:1: v1:172.21.15.192:6789/0 mon.b 2024-06-03T09:06:56.326 INFO:teuthology.orchestra.run.smithi088.stderr:2: v1:172.21.15.88:6790/0 mon.c 2024-06-03T09:06:56.326 INFO:teuthology.orchestra.run.smithi088.stderr: 2024-06-03T09:06:56.326 INFO:teuthology.orchestra.run.smithi088.stderr:2024-06-03T09:06:56.324+0000 7f1b45081640 10 monclient: shutdown 2024-06-03T09:06:56.326 INFO:teuthology.orchestra.run.smithi088.stderr:2024-06-03T09:06:56.324+0000 7f1b45081640 20 monclient: shutdown discarding 0 pending message(s) 2024-06-03T09:06:56.326 INFO:teuthology.orchestra.run.smithi088.stderr:2024-06-03T09:06:56.324+0000 7f1b45081640 1 -- 172.21.15.88:0/4184876467 >> v1:172.21.15.192:6789/0 conn(0x7f1b400b1b70 legacy=0x7f1b4013b140 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-06-03T09:06:56.327 INFO:teuthology.orchestra.run.smithi088.stderr:2024-06-03T09:06:56.324+0000 7f1b45081640 1 -- 172.21.15.88:0/4184876467 shutdown_connections 2024-06-03T09:06:56.327 INFO:teuthology.orchestra.run.smithi088.stderr:2024-06-03T09:06:56.324+0000 7f1b45081640 1 -- 172.21.15.88:0/4184876467 >> 172.21.15.88:0/4184876467 conn(0x7f1b400b7170 msgr2=0x7f1b400b7580 unknown :-1 s=STATE_NONE l=0).mark_down 2024-06-03T09:06:56.327 INFO:teuthology.orchestra.run.smithi088.stderr:2024-06-03T09:06:56.324+0000 7f1b45081640 1 -- 172.21.15.88:0/4184876467 shutdown_connections 2024-06-03T09:06:56.327 INFO:teuthology.orchestra.run.smithi088.stderr:2024-06-03T09:06:56.324+0000 7f1b45081640 1 -- 172.21.15.88:0/4184876467 wait complete. 2024-06-03T09:06:56.327 INFO:teuthology.orchestra.run.smithi088.stderr:2024-06-03T09:06:56.324+0000 7f1b45081640 10 monclient: build_initial_monmap 2024-06-03T09:06:56.327 INFO:teuthology.orchestra.run.smithi088.stderr:2024-06-03T09:06:56.324+0000 7f1b45081640 10 monclient: monmap: 2024-06-03T09:06:56.327 INFO:teuthology.orchestra.run.smithi088.stderr:epoch 0 2024-06-03T09:06:56.327 INFO:teuthology.orchestra.run.smithi088.stderr:fsid 00000000-0000-0000-0000-000000000000 2024-06-03T09:06:56.327 INFO:teuthology.orchestra.run.smithi088.stderr:last_changed 0.000000 2024-06-03T09:06:56.327 INFO:teuthology.orchestra.run.smithi088.stderr:created 0.000000 2024-06-03T09:06:56.327 INFO:teuthology.orchestra.run.smithi088.stderr:min_mon_release 0 (unknown) 2024-06-03T09:06:56.327 INFO:teuthology.orchestra.run.smithi088.stderr:election_strategy: 1 2024-06-03T09:06:56.327 INFO:teuthology.orchestra.run.smithi088.stderr:0: v1:172.21.15.88:6789/0 mon.noname-a 2024-06-03T09:06:56.327 INFO:teuthology.orchestra.run.smithi088.stderr:1: v1:172.21.15.192:6789/0 mon.noname-b 2024-06-03T09:06:56.328 INFO:teuthology.orchestra.run.smithi088.stderr:2: v1:172.21.15.88:6790/0 mon.noname-c 2024-06-03T09:06:56.328 INFO:teuthology.orchestra.run.smithi088.stderr: 2024-06-03T09:06:56.328 INFO:teuthology.orchestra.run.smithi088.stderr:2024-06-03T09:06:56.325+0000 7f1b45081640 1 librados: starting msgr at 2024-06-03T09:06:56.328 INFO:teuthology.orchestra.run.smithi088.stderr:2024-06-03T09:06:56.325+0000 7f1b45081640 1 librados: starting objecter 2024-06-03T09:06:56.328 INFO:teuthology.orchestra.run.smithi088.stderr:2024-06-03T09:06:56.325+0000 7f1b45081640 1 Processor -- start 2024-06-03T09:06:56.328 INFO:teuthology.orchestra.run.smithi088.stderr:2024-06-03T09:06:56.325+0000 7f1b45081640 1 -- start start 2024-06-03T09:06:56.328 INFO:teuthology.orchestra.run.smithi088.stderr:2024-06-03T09:06:56.325+0000 7f1b45081640 1 librados: setting wanted keys 2024-06-03T09:06:56.328 INFO:teuthology.orchestra.run.smithi088.stderr:2024-06-03T09:06:56.325+0000 7f1b45081640 1 librados: calling monclient init 2024-06-03T09:06:56.328 INFO:teuthology.orchestra.run.smithi088.stderr:2024-06-03T09:06:56.325+0000 7f1b45081640 10 monclient: init 2024-06-03T09:06:56.330 INFO:teuthology.orchestra.run.smithi088.stderr:2024-06-03T09:06:56.326+0000 7f1b45081640 10 monclient: _reopen_session rank -1 2024-06-03T09:06:56.330 INFO:teuthology.orchestra.run.smithi088.stderr:2024-06-03T09:06:56.326+0000 7f1b45081640 10 monclient: _add_conns ranks=[1,2,0] 2024-06-03T09:06:56.330 INFO:teuthology.orchestra.run.smithi088.stderr:2024-06-03T09:06:56.326+0000 7f1b45081640 10 monclient(hunting): picked mon.noname-b con 0x7f1b400a8a60 addr v1:172.21.15.192:6789/0 2024-06-03T09:06:56.330 INFO:teuthology.orchestra.run.smithi088.stderr:2024-06-03T09:06:56.326+0000 7f1b45081640 10 monclient(hunting): picked mon.noname-c con 0x7f1b400b1b70 addr v1:172.21.15.88:6790/0 2024-06-03T09:06:56.330 INFO:teuthology.orchestra.run.smithi088.stderr:2024-06-03T09:06:56.326+0000 7f1b45081640 10 monclient(hunting): picked mon.noname-a con 0x7f1b4014ae00 addr v1:172.21.15.88:6789/0 2024-06-03T09:06:56.331 INFO:teuthology.orchestra.run.smithi088.stderr:2024-06-03T09:06:56.326+0000 7f1b45081640 1 -- --> v1:172.21.15.88:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f1b401dab20 con 0x7f1b4014ae00 2024-06-03T09:06:56.331 INFO:teuthology.orchestra.run.smithi088.stderr:2024-06-03T09:06:56.326+0000 7f1b45081640 1 -- --> v1:172.21.15.192:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f1b401d9c10 con 0x7f1b400a8a60 2024-06-03T09:06:56.331 INFO:teuthology.orchestra.run.smithi088.stderr:2024-06-03T09:06:56.326+0000 7f1b45081640 1 -- --> v1:172.21.15.88:6790/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f1b401d9ef0 con 0x7f1b400b1b70 2024-06-03T09:06:56.331 INFO:teuthology.orchestra.run.smithi088.stderr:2024-06-03T09:06:56.326+0000 7f1b45081640 10 monclient(hunting): _renew_subs 2024-06-03T09:06:56.331 INFO:teuthology.orchestra.run.smithi088.stderr:2024-06-03T09:06:56.326+0000 7f1b45081640 10 monclient(hunting): authenticate will time out at 3072.136045s 2024-06-03T09:06:56.331 INFO:teuthology.orchestra.run.smithi088.stderr:2024-06-03T09:06:56.326+0000 7f1b3e59c640 1 --1- >> v1:172.21.15.88:6790/0 conn(0x7f1b400b1b70 0x7f1b40135980 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=1).handle_server_banner_and_identify peer v1:172.21.15.88:6790/0 says I am v1:172.21.15.88:52424/0 (socket says 172.21.15.88:52424) 2024-06-03T09:06:56.331 INFO:teuthology.orchestra.run.smithi088.stderr:2024-06-03T09:06:56.326+0000 7f1b3e59c640 1 -- 172.21.15.88:0/1846551412 learned_addr learned my addr 172.21.15.88:0/1846551412 (peer_addr_for_me v1:172.21.15.88:0/0) 2024-06-03T09:06:56.331 INFO:teuthology.orchestra.run.smithi088.stderr:2024-06-03T09:06:56.326+0000 7f1b277fe640 10 client.?.objecter ms_handle_connect 0x7f1b400b1b70 2024-06-03T09:06:56.331 INFO:teuthology.orchestra.run.smithi088.stderr:2024-06-03T09:06:56.326+0000 7f1b277fe640 10 client.?.objecter resend_mon_ops 2024-06-03T09:06:56.331 INFO:teuthology.orchestra.run.smithi088.stderr:2024-06-03T09:06:56.327+0000 7f1b277fe640 10 client.?.objecter ms_handle_connect 0x7f1b4014ae00 2024-06-03T09:06:56.331 INFO:teuthology.orchestra.run.smithi088.stderr:2024-06-03T09:06:56.327+0000 7f1b277fe640 10 client.?.objecter resend_mon_ops 2024-06-03T09:06:56.331 INFO:teuthology.orchestra.run.smithi088.stderr:2024-06-03T09:06:56.327+0000 7f1b277fe640 10 client.?.objecter ms_handle_connect 0x7f1b400a8a60 2024-06-03T09:06:56.331 INFO:teuthology.orchestra.run.smithi088.stderr:2024-06-03T09:06:56.327+0000 7f1b277fe640 10 client.?.objecter resend_mon_ops 2024-06-03T09:06:56.332 INFO:teuthology.orchestra.run.smithi088.stderr:2024-06-03T09:06:56.328+0000 7f1b277fe640 1 -- 172.21.15.88:0/1846551412 <== mon.0 v1:172.21.15.88:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 2271051580 0 0) 0x7f1b401dab20 con 0x7f1b4014ae00 2024-06-03T09:06:56.332 INFO:teuthology.orchestra.run.smithi088.stderr:2024-06-03T09:06:56.328+0000 7f1b277fe640 10 monclient(hunting): _negotiate 2024-06-03T09:06:56.332 INFO:teuthology.orchestra.run.smithi088.stderr:2024-06-03T09:06:56.328+0000 7f1b277fe640 10 monclient(hunting): _init_auth method 2 2024-06-03T09:06:56.332 INFO:teuthology.orchestra.run.smithi088.stderr:2024-06-03T09:06:56.328+0000 7f1b277fe640 10 monclient(hunting): _init_auth creating new auth 2024-06-03T09:06:56.332 INFO:teuthology.orchestra.run.smithi088.stderr:2024-06-03T09:06:56.328+0000 7f1b277fe640 10 monclient(hunting): my global_id is 30741 2024-06-03T09:06:56.332 INFO:teuthology.orchestra.run.smithi088.stderr:2024-06-03T09:06:56.328+0000 7f1b277fe640 1 -- 172.21.15.88:0/1846551412 --> v1:172.21.15.88:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f1b2c004eb0 con 0x7f1b4014ae00 2024-06-03T09:06:56.332 INFO:teuthology.orchestra.run.smithi088.stderr:2024-06-03T09:06:56.329+0000 7f1b277fe640 1 -- 172.21.15.88:0/1846551412 <== mon.1 v1:172.21.15.192:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 200692609 0 0) 0x7f1b401d9c10 con 0x7f1b400a8a60 2024-06-03T09:06:56.332 INFO:teuthology.orchestra.run.smithi088.stderr:2024-06-03T09:06:56.329+0000 7f1b277fe640 10 monclient(hunting): _negotiate 2024-06-03T09:06:56.332 INFO:teuthology.orchestra.run.smithi088.stderr:2024-06-03T09:06:56.329+0000 7f1b277fe640 10 monclient(hunting): _init_auth method 2 2024-06-03T09:06:56.332 INFO:teuthology.orchestra.run.smithi088.stderr:2024-06-03T09:06:56.329+0000 7f1b277fe640 10 monclient(hunting): _init_auth creating new auth 2024-06-03T09:06:56.332 INFO:teuthology.orchestra.run.smithi088.stderr:2024-06-03T09:06:56.329+0000 7f1b277fe640 10 monclient(hunting): my global_id is 30310 2024-06-03T09:06:56.332 INFO:teuthology.orchestra.run.smithi088.stderr:2024-06-03T09:06:56.329+0000 7f1b277fe640 1 -- 172.21.15.88:0/1846551412 --> v1:172.21.15.192:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f1b401dab20 con 0x7f1b400a8a60 2024-06-03T09:06:56.332 INFO:teuthology.orchestra.run.smithi088.stderr:2024-06-03T09:06:56.329+0000 7f1b277fe640 1 -- 172.21.15.88:0/1846551412 <== mon.2 v1:172.21.15.88:6790/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 205083979 0 0) 0x7f1b401d9ef0 con 0x7f1b400b1b70 2024-06-03T09:06:56.332 INFO:teuthology.orchestra.run.smithi088.stderr:2024-06-03T09:06:56.329+0000 7f1b277fe640 10 monclient(hunting): _negotiate 2024-06-03T09:06:56.333 INFO:teuthology.orchestra.run.smithi088.stderr:2024-06-03T09:06:56.329+0000 7f1b277fe640 10 monclient(hunting): _init_auth method 2 2024-06-03T09:06:56.333 INFO:teuthology.orchestra.run.smithi088.stderr:2024-06-03T09:06:56.329+0000 7f1b277fe640 10 monclient(hunting): _init_auth creating new auth 2024-06-03T09:06:56.333 INFO:teuthology.orchestra.run.smithi088.stderr:2024-06-03T09:06:56.329+0000 7f1b277fe640 10 monclient(hunting): my global_id is 30245 2024-06-03T09:06:56.333 INFO:teuthology.orchestra.run.smithi088.stderr:2024-06-03T09:06:56.329+0000 7f1b277fe640 1 -- 172.21.15.88:0/1846551412 --> v1:172.21.15.88:6790/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f1b401d9c10 con 0x7f1b400b1b70 2024-06-03T09:06:56.333 INFO:teuthology.orchestra.run.smithi088.stderr:2024-06-03T09:06:56.333+0000 7f1b277fe640 1 -- 172.21.15.88:0/1846551412 <== mon.1 v1:172.21.15.192:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 764+0+0 (unknown 2657557695 0 0) 0x7f1b401dab20 con 0x7f1b400a8a60 2024-06-03T09:06:56.333 INFO:teuthology.orchestra.run.smithi088.stderr:2024-06-03T09:06:56.333+0000 7f1b277fe640 1 -- 172.21.15.88:0/1846551412 --> v1:172.21.15.192:6789/0 -- auth(proto 2 165 bytes epoch 0) v1 -- 0x7f1b401d9ef0 con 0x7f1b400a8a60 2024-06-03T09:06:56.334 INFO:teuthology.orchestra.run.smithi088.stderr:2024-06-03T09:06:56.333+0000 7f1b277fe640 1 -- 172.21.15.88:0/1846551412 <== mon.1 v1:172.21.15.192:6789/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 2234374065 0 0) 0x7f1b30001c30 con 0x7f1b400a8a60 2024-06-03T09:06:56.334 INFO:teuthology.orchestra.run.smithi088.stderr:2024-06-03T09:06:56.333+0000 7f1b277fe640 10 monclient(hunting): handle_monmap mon_map magic: 0 v1 2024-06-03T09:06:56.334 INFO:teuthology.orchestra.run.smithi088.stderr:2024-06-03T09:06:56.333+0000 7f1b277fe640 10 monclient(hunting): got monmap 1 from mon.noname-b (according to old e1) 2024-06-03T09:06:56.334 INFO:teuthology.orchestra.run.smithi088.stderr:2024-06-03T09:06:56.333+0000 7f1b277fe640 10 monclient(hunting): dump: 2024-06-03T09:06:56.334 INFO:teuthology.orchestra.run.smithi088.stderr:epoch 1 2024-06-03T09:06:56.334 INFO:teuthology.orchestra.run.smithi088.stderr:fsid ec3c3fdd-3ac6-4d7d-9412-683130ed8b04 2024-06-03T09:06:56.334 INFO:teuthology.orchestra.run.smithi088.stderr:last_changed 2024-06-03T08:34:31.885125+0000 2024-06-03T09:06:56.334 INFO:teuthology.orchestra.run.smithi088.stderr:created 2024-06-03T08:34:31.885125+0000 2024-06-03T09:06:56.334 INFO:teuthology.orchestra.run.smithi088.stderr:min_mon_release 18 (reef) 2024-06-03T09:06:56.334 INFO:teuthology.orchestra.run.smithi088.stderr:election_strategy: 3 2024-06-03T09:06:56.334 INFO:teuthology.orchestra.run.smithi088.stderr:0: v1:172.21.15.88:6789/0 mon.a 2024-06-03T09:06:56.334 INFO:teuthology.orchestra.run.smithi088.stderr:1: v1:172.21.15.192:6789/0 mon.b 2024-06-03T09:06:56.334 INFO:teuthology.orchestra.run.smithi088.stderr:2: v1:172.21.15.88:6790/0 mon.c 2024-06-03T09:06:56.334 INFO:teuthology.orchestra.run.smithi088.stderr: 2024-06-03T09:06:56.335 INFO:teuthology.orchestra.run.smithi088.stderr:2024-06-03T09:06:56.333+0000 7f1b277fe640 10 monclient(hunting): _finish_auth 0 2024-06-03T09:06:56.335 INFO:teuthology.orchestra.run.smithi088.stderr:2024-06-03T09:06:56.335+0000 7f1b277fe640 1 -- 172.21.15.88:0/1846551412 <== mon.0 v1:172.21.15.88:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 764+0+0 (unknown 325664604 0 0) 0x7f1b2c004eb0 con 0x7f1b4014ae00 2024-06-03T09:06:56.335 INFO:teuthology.orchestra.run.smithi088.stderr:2024-06-03T09:06:56.335+0000 7f1b277fe640 1 -- 172.21.15.88:0/1846551412 --> v1:172.21.15.88:6789/0 -- auth(proto 2 165 bytes epoch 0) v1 -- 0x7f1b401dab20 con 0x7f1b4014ae00 2024-06-03T09:06:56.336 INFO:teuthology.orchestra.run.smithi088.stderr:2024-06-03T09:06:56.335+0000 7f1b277fe640 1 -- 172.21.15.88:0/1846551412 <== mon.2 v1:172.21.15.88:6790/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 764+0+0 (unknown 951893408 0 0) 0x7f1b401d9c10 con 0x7f1b400b1b70 2024-06-03T09:06:56.336 INFO:teuthology.orchestra.run.smithi088.stderr:2024-06-03T09:06:56.335+0000 7f1b277fe640 1 -- 172.21.15.88:0/1846551412 --> v1:172.21.15.88:6790/0 -- auth(proto 2 165 bytes epoch 0) v1 -- 0x7f1b2c004eb0 con 0x7f1b400b1b70 2024-06-03T09:06:56.336 INFO:teuthology.orchestra.run.smithi088.stderr:2024-06-03T09:06:56.335+0000 7f1b277fe640 1 -- 172.21.15.88:0/1846551412 <== mon.0 v1:172.21.15.88:6789/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 2234374065 0 0) 0x7f1b38001810 con 0x7f1b4014ae00 2024-06-03T09:06:56.336 INFO:teuthology.orchestra.run.smithi088.stderr:2024-06-03T09:06:56.335+0000 7f1b277fe640 10 monclient(hunting): handle_monmap mon_map magic: 0 v1 2024-06-03T09:06:56.336 INFO:teuthology.orchestra.run.smithi088.stderr:2024-06-03T09:06:56.335+0000 7f1b277fe640 10 monclient(hunting): got monmap 1 from mon.a (according to old e1) 2024-06-03T09:06:56.336 INFO:teuthology.orchestra.run.smithi088.stderr:2024-06-03T09:06:56.335+0000 7f1b277fe640 10 monclient(hunting): dump: 2024-06-03T09:06:56.336 INFO:teuthology.orchestra.run.smithi088.stderr:epoch 1 2024-06-03T09:06:56.336 INFO:teuthology.orchestra.run.smithi088.stderr:fsid ec3c3fdd-3ac6-4d7d-9412-683130ed8b04 2024-06-03T09:06:56.336 INFO:teuthology.orchestra.run.smithi088.stderr:last_changed 2024-06-03T08:34:31.885125+0000 2024-06-03T09:06:56.336 INFO:teuthology.orchestra.run.smithi088.stderr:created 2024-06-03T08:34:31.885125+0000 2024-06-03T09:06:56.336 INFO:teuthology.orchestra.run.smithi088.stderr:min_mon_release 18 (reef) 2024-06-03T09:06:56.336 INFO:teuthology.orchestra.run.smithi088.stderr:election_strategy: 3 2024-06-03T09:06:56.336 INFO:teuthology.orchestra.run.smithi088.stderr:0: v1:172.21.15.88:6789/0 mon.a 2024-06-03T09:06:56.337 INFO:teuthology.orchestra.run.smithi088.stderr:1: v1:172.21.15.192:6789/0 mon.b 2024-06-03T09:06:56.337 INFO:teuthology.orchestra.run.smithi088.stderr:2: v1:172.21.15.88:6790/0 mon.c 2024-06-03T09:06:56.337 INFO:teuthology.orchestra.run.smithi088.stderr: 2024-06-03T09:06:56.337 INFO:teuthology.orchestra.run.smithi088.stderr:2024-06-03T09:06:56.335+0000 7f1b277fe640 1 -- 172.21.15.88:0/1846551412 <== mon.2 v1:172.21.15.88:6790/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 2234374065 0 0) 0x7f1b34001510 con 0x7f1b400b1b70 2024-06-03T09:06:56.337 INFO:teuthology.orchestra.run.smithi088.stderr:2024-06-03T09:06:56.335+0000 7f1b277fe640 10 monclient(hunting): handle_monmap mon_map magic: 0 v1 2024-06-03T09:06:56.337 INFO:teuthology.orchestra.run.smithi088.stderr:2024-06-03T09:06:56.335+0000 7f1b277fe640 10 monclient(hunting): got monmap 1 from mon.c (according to old e1) 2024-06-03T09:06:56.337 INFO:teuthology.orchestra.run.smithi088.stderr:2024-06-03T09:06:56.335+0000 7f1b277fe640 10 monclient(hunting): dump: 2024-06-03T09:06:56.337 INFO:teuthology.orchestra.run.smithi088.stderr:epoch 1 2024-06-03T09:06:56.337 INFO:teuthology.orchestra.run.smithi088.stderr:fsid ec3c3fdd-3ac6-4d7d-9412-683130ed8b04 2024-06-03T09:06:56.337 INFO:teuthology.orchestra.run.smithi088.stderr:last_changed 2024-06-03T08:34:31.885125+0000 2024-06-03T09:06:56.337 INFO:teuthology.orchestra.run.smithi088.stderr:created 2024-06-03T08:34:31.885125+0000 2024-06-03T09:06:56.337 INFO:teuthology.orchestra.run.smithi088.stderr:min_mon_release 18 (reef) 2024-06-03T09:06:56.337 INFO:teuthology.orchestra.run.smithi088.stderr:election_strategy: 3 2024-06-03T09:06:56.338 INFO:teuthology.orchestra.run.smithi088.stderr:0: v1:172.21.15.88:6789/0 mon.a 2024-06-03T09:06:56.338 INFO:teuthology.orchestra.run.smithi088.stderr:1: v1:172.21.15.192:6789/0 mon.b 2024-06-03T09:06:56.338 INFO:teuthology.orchestra.run.smithi088.stderr:2: v1:172.21.15.88:6790/0 mon.c 2024-06-03T09:06:56.338 INFO:teuthology.orchestra.run.smithi088.stderr: 2024-06-03T09:06:56.339 INFO:teuthology.orchestra.run.smithi088.stderr:2024-06-03T09:06:56.337+0000 7f1b277fe640 1 -- 172.21.15.88:0/1846551412 <== mon.1 v1:172.21.15.192:6789/0 4 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 580+0+0 (unknown 2057179030 0 0) 0x7f1b401d9ef0 con 0x7f1b400a8a60 2024-06-03T09:06:56.339 INFO:teuthology.orchestra.run.smithi088.stderr:2024-06-03T09:06:56.337+0000 7f1b277fe640 1 -- 172.21.15.88:0/1846551412 >> v1:172.21.15.88:6790/0 conn(0x7f1b400b1b70 legacy=0x7f1b40135980 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-06-03T09:06:56.339 INFO:teuthology.orchestra.run.smithi088.stderr:2024-06-03T09:06:56.337+0000 7f1b277fe640 1 -- 172.21.15.88:0/1846551412 >> v1:172.21.15.88:6789/0 conn(0x7f1b4014ae00 legacy=0x7f1b401e0f60 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-06-03T09:06:56.339 INFO:teuthology.orchestra.run.smithi088.stderr:2024-06-03T09:06:56.337+0000 7f1b277fe640 10 monclient: _finish_hunting 0 2024-06-03T09:06:56.339 INFO:teuthology.orchestra.run.smithi088.stderr:2024-06-03T09:06:56.337+0000 7f1b277fe640 1 monclient: found mon.b 2024-06-03T09:06:56.339 INFO:teuthology.orchestra.run.smithi088.stderr:2024-06-03T09:06:56.337+0000 7f1b277fe640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-06-03T09:06:56.339 INFO:teuthology.orchestra.run.smithi088.stderr:2024-06-03T09:06:56.337+0000 7f1b277fe640 10 monclient: _send_mon_message to mon.b at v1:172.21.15.192:6789/0 2024-06-03T09:06:56.339 INFO:teuthology.orchestra.run.smithi088.stderr:2024-06-03T09:06:56.337+0000 7f1b277fe640 1 -- 172.21.15.88:0/1846551412 --> v1:172.21.15.192:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f1b40136090 con 0x7f1b400a8a60 2024-06-03T09:06:56.339 INFO:teuthology.orchestra.run.smithi088.stderr:2024-06-03T09:06:56.337+0000 7f1b277fe640 10 monclient: _finish_auth 0 2024-06-03T09:06:56.339 INFO:teuthology.orchestra.run.smithi088.stderr:2024-06-03T09:06:56.337+0000 7f1b277fe640 10 monclient: _check_auth_tickets 2024-06-03T09:06:56.340 INFO:teuthology.orchestra.run.smithi088.stderr:2024-06-03T09:06:56.337+0000 7f1b277fe640 20 monclient: _check_auth_rotating not needed by client.admin 2024-06-03T09:06:56.340 INFO:teuthology.orchestra.run.smithi088.stderr:2024-06-03T09:06:56.337+0000 7f1b45081640 5 monclient: authenticate success, global_id 30310 2024-06-03T09:06:56.340 INFO:teuthology.orchestra.run.smithi088.stderr:2024-06-03T09:06:56.337+0000 7f1b45081640 10 monclient: _renew_subs 2024-06-03T09:06:56.340 INFO:teuthology.orchestra.run.smithi088.stderr:2024-06-03T09:06:56.337+0000 7f1b45081640 10 monclient: _send_mon_message to mon.b at v1:172.21.15.192:6789/0 2024-06-03T09:06:56.340 INFO:teuthology.orchestra.run.smithi088.stderr:2024-06-03T09:06:56.337+0000 7f1b45081640 1 -- 172.21.15.88:0/1846551412 --> v1:172.21.15.192:6789/0 -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f1b4013e4f0 con 0x7f1b400a8a60 2024-06-03T09:06:56.340 INFO:teuthology.orchestra.run.smithi088.stderr:2024-06-03T09:06:56.337+0000 7f1b45081640 10 client.30310.objecter _maybe_request_map subscribing (onetime) to next osd map 2024-06-03T09:06:56.340 INFO:teuthology.orchestra.run.smithi088.stderr:2024-06-03T09:06:56.337+0000 7f1b45081640 10 monclient: _renew_subs 2024-06-03T09:06:56.340 INFO:teuthology.orchestra.run.smithi088.stderr:2024-06-03T09:06:56.337+0000 7f1b45081640 10 monclient: _send_mon_message to mon.b at v1:172.21.15.192:6789/0 2024-06-03T09:06:56.340 INFO:teuthology.orchestra.run.smithi088.stderr:2024-06-03T09:06:56.337+0000 7f1b45081640 1 -- 172.21.15.88:0/1846551412 --> v1:172.21.15.192:6789/0 -- mon_subscribe({osdmap=0}) v3 -- 0x7f1b401e2680 con 0x7f1b400a8a60 2024-06-03T09:06:56.340 INFO:teuthology.orchestra.run.smithi088.stderr:2024-06-03T09:06:56.337+0000 7f1b45081640 1 librados: init done 2024-06-03T09:06:56.340 INFO:teuthology.orchestra.run.smithi088.stderr:2024-06-03T09:06:56.338+0000 7f1b45081640 10 client.30310.objecter _submit_command 1 [{"prefix": "get_command_descriptions"}] 2024-06-03T09:06:56.340 INFO:teuthology.orchestra.run.smithi088.stderr:2024-06-03T09:06:56.338+0000 7f1b45081640 15 client.30310.objecter _session_command_op_assign -1 1 2024-06-03T09:06:56.340 INFO:teuthology.orchestra.run.smithi088.stderr:2024-06-03T09:06:56.338+0000 7f1b45081640 20 client.30310.objecter _get_session osd=-1 returning homeless 2024-06-03T09:06:56.340 INFO:teuthology.orchestra.run.smithi088.stderr:2024-06-03T09:06:56.338+0000 7f1b45081640 10 client.30310.objecter _maybe_request_map subscribing (onetime) to next osd map 2024-06-03T09:06:56.341 INFO:teuthology.orchestra.run.smithi088.stderr:2024-06-03T09:06:56.338+0000 7f1b45081640 10 monclient: _send_mon_message to mon.b at v1:172.21.15.192:6789/0 2024-06-03T09:06:56.341 INFO:teuthology.orchestra.run.smithi088.stderr:2024-06-03T09:06:56.338+0000 7f1b45081640 1 -- 172.21.15.88:0/1846551412 --> v1:172.21.15.192:6789/0 -- mon_get_version(what=osdmap handle=1) v1 -- 0x7f1b401362e0 con 0x7f1b400a8a60 2024-06-03T09:06:56.353 INFO:teuthology.orchestra.run.smithi088.stderr:2024-06-03T09:06:56.353+0000 7f1b277fe640 1 -- 172.21.15.88:0/1846551412 <== mon.1 v1:172.21.15.192:6789/0 5 ==== config(0 keys) v1 ==== 4+0+0 (unknown 0 0 0) 0x7f1b30004b60 con 0x7f1b400a8a60 2024-06-03T09:06:56.353 INFO:teuthology.orchestra.run.smithi088.stderr:2024-06-03T09:06:56.353+0000 7f1b277fe640 10 monclient: handle_config config(0 keys) v1 2024-06-03T09:06:56.353 INFO:teuthology.orchestra.run.smithi088.stderr:2024-06-03T09:06:56.353+0000 7f1b277fe640 1 -- 172.21.15.88:0/1846551412 <== mon.1 v1:172.21.15.192:6789/0 6 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 2234374065 0 0) 0x7f1b30004e70 con 0x7f1b400a8a60 2024-06-03T09:06:56.353 INFO:teuthology.orchestra.run.smithi088.stderr:2024-06-03T09:06:56.353+0000 7f1b277fe640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-06-03T09:06:56.353 INFO:teuthology.orchestra.run.smithi088.stderr:2024-06-03T09:06:56.353+0000 7f1b277fe640 10 monclient: got monmap 1 from mon.b (according to old e1) 2024-06-03T09:06:56.353 INFO:teuthology.orchestra.run.smithi088.stderr:2024-06-03T09:06:56.353+0000 7f1b277fe640 10 monclient: dump: 2024-06-03T09:06:56.353 INFO:teuthology.orchestra.run.smithi088.stderr:epoch 1 2024-06-03T09:06:56.353 INFO:teuthology.orchestra.run.smithi088.stderr:fsid ec3c3fdd-3ac6-4d7d-9412-683130ed8b04 2024-06-03T09:06:56.353 INFO:teuthology.orchestra.run.smithi088.stderr:last_changed 2024-06-03T08:34:31.885125+0000 2024-06-03T09:06:56.353 INFO:teuthology.orchestra.run.smithi088.stderr:created 2024-06-03T08:34:31.885125+0000 2024-06-03T09:06:56.354 INFO:teuthology.orchestra.run.smithi088.stderr:min_mon_release 18 (reef) 2024-06-03T09:06:56.354 INFO:teuthology.orchestra.run.smithi088.stderr:election_strategy: 3 2024-06-03T09:06:56.354 INFO:teuthology.orchestra.run.smithi088.stderr:0: v1:172.21.15.88:6789/0 mon.a 2024-06-03T09:06:56.354 INFO:teuthology.orchestra.run.smithi088.stderr:1: v1:172.21.15.192:6789/0 mon.b 2024-06-03T09:06:56.354 INFO:teuthology.orchestra.run.smithi088.stderr:2: v1:172.21.15.88:6790/0 mon.c 2024-06-03T09:06:56.354 INFO:teuthology.orchestra.run.smithi088.stderr: 2024-06-03T09:06:56.354 INFO:teuthology.orchestra.run.smithi088.stderr:2024-06-03T09:06:56.353+0000 7f1b3d59a640 4 set_mon_vals no callback set 2024-06-03T09:06:56.355 INFO:teuthology.orchestra.run.smithi088.stderr:2024-06-03T09:06:56.353+0000 7f1b277fe640 1 -- 172.21.15.88:0/1846551412 <== mon.1 v1:172.21.15.192:6789/0 7 ==== mgrmap(e 5) v1 ==== 89110+0+0 (unknown 873533973 0 0) 0x7f1b3001abf0 con 0x7f1b400a8a60 2024-06-03T09:06:56.355 INFO:teuthology.orchestra.run.smithi088.stderr:2024-06-03T09:06:56.354+0000 7f1b3e59c640 10 monclient: get_auth_request con 0x7f1b2c06d390 auth_method 0 2024-06-03T09:06:56.355 INFO:teuthology.orchestra.run.smithi088.stderr:2024-06-03T09:06:56.354+0000 7f1b277fe640 10 client.30310.objecter ms_handle_connect 0x7f1b2c06d390 2024-06-03T09:06:56.356 INFO:teuthology.orchestra.run.smithi088.stderr:2024-06-03T09:06:56.355+0000 7f1b277fe640 1 -- 172.21.15.88:0/1846551412 <== mon.1 v1:172.21.15.192:6789/0 8 ==== osd_map(1014..1014 src has 622..1014) v4 ==== 5229+0+0 (unknown 3230192161 0 0) 0x7f1b30054470 con 0x7f1b400a8a60 2024-06-03T09:06:56.356 INFO:teuthology.orchestra.run.smithi088.stderr:2024-06-03T09:06:56.355+0000 7f1b277fe640 10 client.30310.objecter ms_dispatch 0x7f1b40000b90 osd_map(1014..1014 src has 622..1014) v4 2024-06-03T09:06:56.356 INFO:teuthology.orchestra.run.smithi088.stderr:2024-06-03T09:06:56.355+0000 7f1b277fe640 3 client.30310.objecter handle_osd_map got epochs [1014,1014] > 0 2024-06-03T09:06:56.356 INFO:teuthology.orchestra.run.smithi088.stderr:2024-06-03T09:06:56.355+0000 7f1b277fe640 3 client.30310.objecter handle_osd_map decoding full epoch 1014 2024-06-03T09:06:56.356 INFO:teuthology.orchestra.run.smithi088.stderr:2024-06-03T09:06:56.355+0000 7f1b277fe640 10 client.30310.objecter checking command 1 2024-06-03T09:06:56.356 INFO:teuthology.orchestra.run.smithi088.stderr:2024-06-03T09:06:56.355+0000 7f1b277fe640 20 client.30310.objecter _get_session s=0x7f1b2c0752d0 osd=0 3 2024-06-03T09:06:56.356 INFO:teuthology.orchestra.run.smithi088.stderr:2024-06-03T09:06:56.355+0000 7f1b277fe640 20 client.30310.objecter put_session s=0x7f1b2c0752d0 osd=0 3 2024-06-03T09:06:56.356 INFO:teuthology.orchestra.run.smithi088.stderr:2024-06-03T09:06:56.355+0000 7f1b277fe640 15 client.30310.objecter _session_command_op_remove -1 1 2024-06-03T09:06:56.356 INFO:teuthology.orchestra.run.smithi088.stderr:2024-06-03T09:06:56.355+0000 7f1b277fe640 20 client.30310.objecter _get_session s=0x7f1b2c0752d0 osd=0 3 2024-06-03T09:06:56.356 INFO:teuthology.orchestra.run.smithi088.stderr:2024-06-03T09:06:56.355+0000 7f1b277fe640 20 client.30310.objecter get_session s=0x7f1b2c0752d0 osd=0 3 2024-06-03T09:06:56.356 INFO:teuthology.orchestra.run.smithi088.stderr:2024-06-03T09:06:56.355+0000 7f1b277fe640 15 client.30310.objecter _session_command_op_assign 0 1 2024-06-03T09:06:56.357 INFO:teuthology.orchestra.run.smithi088.stderr:2024-06-03T09:06:56.355+0000 7f1b277fe640 20 client.30310.objecter put_session s=0x7f1b2c0752d0 osd=0 4 2024-06-03T09:06:56.357 INFO:teuthology.orchestra.run.smithi088.stderr:2024-06-03T09:06:56.355+0000 7f1b277fe640 10 client.30310.objecter _send_command 1 2024-06-03T09:06:56.357 INFO:teuthology.orchestra.run.smithi088.stderr:2024-06-03T09:06:56.355+0000 7f1b277fe640 1 -- 172.21.15.88:0/1846551412 --> v1:172.21.15.88:6808/35066 -- command(tid 1: {"prefix": "get_command_descriptions"}) v1 -- 0x7f1b2c079690 con 0x7f1b2c075980 2024-06-03T09:06:56.357 INFO:teuthology.orchestra.run.smithi088.stderr:2024-06-03T09:06:56.355+0000 7f1b277fe640 20 client.30310.objecter dump_active .. 0 homeless 2024-06-03T09:06:56.357 INFO:teuthology.orchestra.run.smithi088.stderr:2024-06-03T09:06:56.356+0000 7f1b3f59e640 10 monclient: get_auth_request con 0x7f1b2c075980 auth_method 0 2024-06-03T09:06:56.357 INFO:teuthology.orchestra.run.smithi088.stderr:2024-06-03T09:06:56.356+0000 7f1b277fe640 1 -- 172.21.15.88:0/1846551412 <== mon.1 v1:172.21.15.192:6789/0 9 ==== mon_get_version_reply(handle=1 version=1014) v2 ==== 24+0+0 (unknown 3501658476 0 0) 0x7f1b30050020 con 0x7f1b400a8a60 2024-06-03T09:06:56.357 INFO:teuthology.orchestra.run.smithi088.stderr:2024-06-03T09:06:56.356+0000 7f1b277fe640 10 monclient: handle_get_version_reply finishing 1 version 1014 2024-06-03T09:06:56.357 INFO:teuthology.orchestra.run.smithi088.stderr:2024-06-03T09:06:56.357+0000 7f1b277fe640 10 client.30310.objecter ms_handle_connect 0x7f1b2c075980 2024-06-03T09:06:56.380 INFO:teuthology.orchestra.run.smithi088.stderr:2024-06-03T09:06:56.380+0000 7f1b277fe640 1 -- 172.21.15.88:0/1846551412 <== osd.0 v1:172.21.15.88:6808/35066 1 ==== command_reply(tid 1: 0 ) v1 ==== 8+0+26118 (unknown 0 0 1686565167) 0x7f1b2c079690 con 0x7f1b2c075980 2024-06-03T09:06:56.380 INFO:teuthology.orchestra.run.smithi088.stderr:2024-06-03T09:06:56.380+0000 7f1b277fe640 10 client.30310.objecter ms_dispatch 0x7f1b40000b90 command_reply(tid 1: 0 ) v1 2024-06-03T09:06:56.381 INFO:teuthology.orchestra.run.smithi088.stderr:2024-06-03T09:06:56.380+0000 7f1b277fe640 10 client.30310.objecter _finish_command 1 = system:0 2024-06-03T09:06:56.381 INFO:teuthology.orchestra.run.smithi088.stderr:2024-06-03T09:06:56.380+0000 7f1b277fe640 20 client.30310.objecter put_session s=0x7f1b2c0752d0 osd=0 4 2024-06-03T09:06:56.381 INFO:teuthology.orchestra.run.smithi088.stderr:2024-06-03T09:06:56.380+0000 7f1b277fe640 15 client.30310.objecter _session_command_op_remove 0 1 2024-06-03T09:06:56.393 INFO:teuthology.orchestra.run.smithi088.stderr:2024-06-03T09:06:56.393+0000 7f1b45081640 10 client.30310.objecter _submit_command 2 [{"prefix": "config set", "var": "osd_debug_deep_scrub_sleep", "val": ["0"]}] 2024-06-03T09:06:56.393 INFO:teuthology.orchestra.run.smithi088.stderr:2024-06-03T09:06:56.393+0000 7f1b45081640 15 client.30310.objecter _session_command_op_assign -1 2 2024-06-03T09:06:56.393 INFO:teuthology.orchestra.run.smithi088.stderr:2024-06-03T09:06:56.393+0000 7f1b45081640 20 client.30310.objecter _get_session s=0x7f1b2c0752d0 osd=0 3 2024-06-03T09:06:56.393 INFO:teuthology.orchestra.run.smithi088.stderr:2024-06-03T09:06:56.393+0000 7f1b45081640 20 client.30310.objecter put_session s=0x7f1b2c0752d0 osd=0 3 2024-06-03T09:06:56.393 INFO:teuthology.orchestra.run.smithi088.stderr:2024-06-03T09:06:56.393+0000 7f1b45081640 20 client.30310.objecter _get_session s=0x7f1b2c0752d0 osd=0 3 2024-06-03T09:06:56.393 INFO:teuthology.orchestra.run.smithi088.stderr:2024-06-03T09:06:56.393+0000 7f1b45081640 15 client.30310.objecter _session_command_op_remove -1 2 2024-06-03T09:06:56.393 INFO:teuthology.orchestra.run.smithi088.stderr:2024-06-03T09:06:56.393+0000 7f1b45081640 20 client.30310.objecter get_session s=0x7f1b2c0752d0 osd=0 3 2024-06-03T09:06:56.393 INFO:teuthology.orchestra.run.smithi088.stderr:2024-06-03T09:06:56.393+0000 7f1b45081640 15 client.30310.objecter _session_command_op_assign 0 2 2024-06-03T09:06:56.393 INFO:teuthology.orchestra.run.smithi088.stderr:2024-06-03T09:06:56.393+0000 7f1b45081640 20 client.30310.objecter put_session s=0x7f1b2c0752d0 osd=0 4 2024-06-03T09:06:56.393 INFO:teuthology.orchestra.run.smithi088.stderr:2024-06-03T09:06:56.393+0000 7f1b45081640 10 client.30310.objecter _send_command 2 2024-06-03T09:06:56.393 INFO:teuthology.orchestra.run.smithi088.stderr:2024-06-03T09:06:56.393+0000 7f1b45081640 1 -- 172.21.15.88:0/1846551412 --> v1:172.21.15.88:6808/35066 -- command(tid 2: {"prefix": "config set", "var": "osd_debug_deep_scrub_sleep", "val": ["0"]}) v1 -- 0x7f1b4013d180 con 0x7f1b2c075980 2024-06-03T09:06:56.406 INFO:teuthology.orchestra.run.smithi088.stderr:2024-06-03T09:06:56.406+0000 7f1b277fe640 1 -- 172.21.15.88:0/1846551412 <== osd.0 v1:172.21.15.88:6808/35066 2 ==== command_reply(tid 2: 0 ) v1 ==== 8+0+97 (unknown 0 0 119112422) 0x7f1b4013d180 con 0x7f1b2c075980 2024-06-03T09:06:56.406 INFO:teuthology.orchestra.run.smithi088.stderr:2024-06-03T09:06:56.406+0000 7f1b277fe640 10 client.30310.objecter ms_dispatch 0x7f1b40000b90 command_reply(tid 2: 0 ) v1 2024-06-03T09:06:56.406 INFO:teuthology.orchestra.run.smithi088.stderr:2024-06-03T09:06:56.406+0000 7f1b277fe640 10 client.30310.objecter _finish_command 2 = system:0 2024-06-03T09:06:56.406 INFO:teuthology.orchestra.run.smithi088.stderr:2024-06-03T09:06:56.406+0000 7f1b277fe640 20 client.30310.objecter put_session s=0x7f1b2c0752d0 osd=0 4 2024-06-03T09:06:56.406 INFO:teuthology.orchestra.run.smithi088.stderr:2024-06-03T09:06:56.406+0000 7f1b277fe640 15 client.30310.objecter _session_command_op_remove 0 2 2024-06-03T09:06:56.406 INFO:teuthology.orchestra.run.smithi088.stdout:{ 2024-06-03T09:06:56.407 INFO:teuthology.orchestra.run.smithi088.stdout: "success": "osd_debug_deep_scrub_sleep = '' (not observed, change may require restart) " 2024-06-03T09:06:56.407 INFO:teuthology.orchestra.run.smithi088.stdout:} 2024-06-03T09:06:56.408 INFO:teuthology.orchestra.run.smithi088.stderr:2024-06-03T09:06:56.406+0000 7f1b45081640 10 librados: watch_flush enter 2024-06-03T09:06:56.408 INFO:teuthology.orchestra.run.smithi088.stderr:2024-06-03T09:06:56.406+0000 7f1b45081640 10 librados: watch_flush exit 2024-06-03T09:06:56.408 INFO:teuthology.orchestra.run.smithi088.stderr:2024-06-03T09:06:56.406+0000 7f1b45081640 10 client.30310.objecter close_session for osd.0 2024-06-03T09:06:56.408 INFO:teuthology.orchestra.run.smithi088.stderr:2024-06-03T09:06:56.406+0000 7f1b45081640 1 -- 172.21.15.88:0/1846551412 >> v1:172.21.15.88:6808/35066 conn(0x7f1b2c075980 legacy=0x7f1b2c077e20 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-06-03T09:06:56.408 INFO:teuthology.orchestra.run.smithi088.stderr:2024-06-03T09:06:56.406+0000 7f1b45081640 20 client.30310.objecter put_session s=0x7f1b2c0752d0 osd=0 1 2024-06-03T09:06:56.408 INFO:teuthology.orchestra.run.smithi088.stderr:2024-06-03T09:06:56.406+0000 7f1b45081640 20 client.30310.objecter shutdown clearing up homeless session... 2024-06-03T09:06:56.408 INFO:teuthology.orchestra.run.smithi088.stderr:2024-06-03T09:06:56.406+0000 7f1b45081640 10 client.30310.objecter successfully canceled tick 2024-06-03T09:06:56.408 INFO:teuthology.orchestra.run.smithi088.stderr:2024-06-03T09:06:56.406+0000 7f1b45081640 1 -- 172.21.15.88:0/1846551412 >> v1:172.21.15.192:6800/34229 conn(0x7f1b2c06d390 legacy=0x7f1b2c06f850 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-06-03T09:06:56.408 INFO:teuthology.orchestra.run.smithi088.stderr:2024-06-03T09:06:56.406+0000 7f1b45081640 10 monclient: shutdown 2024-06-03T09:06:56.408 INFO:teuthology.orchestra.run.smithi088.stderr:2024-06-03T09:06:56.406+0000 7f1b45081640 20 monclient: shutdown discarding 0 pending message(s) 2024-06-03T09:06:56.409 INFO:teuthology.orchestra.run.smithi088.stderr:2024-06-03T09:06:56.406+0000 7f1b45081640 1 -- 172.21.15.88:0/1846551412 >> v1:172.21.15.192:6789/0 conn(0x7f1b400a8a60 legacy=0x7f1b401da410 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-06-03T09:06:56.409 INFO:teuthology.orchestra.run.smithi088.stderr:2024-06-03T09:06:56.406+0000 7f1b3f59e640 1 -- 172.21.15.88:0/1846551412 reap_dead start 2024-06-03T09:06:56.409 INFO:teuthology.orchestra.run.smithi088.stderr:2024-06-03T09:06:56.406+0000 7f1b45081640 1 -- 172.21.15.88:0/1846551412 shutdown_connections 2024-06-03T09:06:56.409 INFO:teuthology.orchestra.run.smithi088.stderr:2024-06-03T09:06:56.406+0000 7f1b45081640 1 -- 172.21.15.88:0/1846551412 >> 172.21.15.88:0/1846551412 conn(0x7f1b400b7170 msgr2=0x7f1b40134b90 unknown :-1 s=STATE_NONE l=0).mark_down 2024-06-03T09:06:56.409 INFO:teuthology.orchestra.run.smithi088.stderr:2024-06-03T09:06:56.406+0000 7f1b45081640 1 -- 172.21.15.88:0/1846551412 shutdown_connections 2024-06-03T09:06:56.409 INFO:teuthology.orchestra.run.smithi088.stderr:2024-06-03T09:06:56.406+0000 7f1b45081640 1 -- 172.21.15.88:0/1846551412 wait complete. 2024-06-03T09:06:56.409 INFO:teuthology.orchestra.run.smithi088.stderr:2024-06-03T09:06:56.406+0000 7f1b45081640 1 librados: shutdown 2024-06-03T09:06:56.418 DEBUG:teuthology.orchestra.run.smithi088:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd deep-scrub 0 2024-06-03T09:06:56.520 INFO:teuthology.orchestra.run.smithi088.stderr:2024-06-03T09:06:56.519+0000 7fc02dee2640 10 monclient: get_monmap_and_config 2024-06-03T09:06:56.520 INFO:teuthology.orchestra.run.smithi088.stderr:2024-06-03T09:06:56.519+0000 7fc02dee2640 10 monclient: build_initial_monmap 2024-06-03T09:06:56.520 INFO:teuthology.orchestra.run.smithi088.stderr:2024-06-03T09:06:56.519+0000 7fc02dee2640 10 monclient: monmap: 2024-06-03T09:06:56.520 INFO:teuthology.orchestra.run.smithi088.stderr:epoch 0 2024-06-03T09:06:56.520 INFO:teuthology.orchestra.run.smithi088.stderr:fsid ec3c3fdd-3ac6-4d7d-9412-683130ed8b04 2024-06-03T09:06:56.520 INFO:teuthology.orchestra.run.smithi088.stderr:last_changed 2024-06-03T09:06:56.520904+0000 2024-06-03T09:06:56.520 INFO:teuthology.orchestra.run.smithi088.stderr:created 2024-06-03T09:06:56.520904+0000 2024-06-03T09:06:56.520 INFO:teuthology.orchestra.run.smithi088.stderr:min_mon_release 0 (unknown) 2024-06-03T09:06:56.520 INFO:teuthology.orchestra.run.smithi088.stderr:election_strategy: 3 2024-06-03T09:06:56.520 INFO:teuthology.orchestra.run.smithi088.stderr:0: v1:172.21.15.88:6789/0 mon.noname-a 2024-06-03T09:06:56.520 INFO:teuthology.orchestra.run.smithi088.stderr:1: v1:172.21.15.192:6789/0 mon.noname-c 2024-06-03T09:06:56.521 INFO:teuthology.orchestra.run.smithi088.stderr:2: v1:172.21.15.88:6790/0 mon.noname-b 2024-06-03T09:06:56.521 INFO:teuthology.orchestra.run.smithi088.stderr: 2024-06-03T09:06:56.522 INFO:teuthology.orchestra.run.smithi088.stderr:2024-06-03T09:06:56.520+0000 7fc02dee2640 1 Processor -- start 2024-06-03T09:06:56.522 INFO:teuthology.orchestra.run.smithi088.stderr:2024-06-03T09:06:56.520+0000 7fc02dee2640 1 -- start start 2024-06-03T09:06:56.522 INFO:teuthology.orchestra.run.smithi088.stderr:2024-06-03T09:06:56.520+0000 7fc02dee2640 10 monclient: init 2024-06-03T09:06:56.522 INFO:teuthology.orchestra.run.smithi088.stderr:2024-06-03T09:06:56.521+0000 7fc02dee2640 10 monclient: _reopen_session rank -1 2024-06-03T09:06:56.522 INFO:teuthology.orchestra.run.smithi088.stderr:2024-06-03T09:06:56.521+0000 7fc02dee2640 10 monclient: _add_conns ranks=[2,1,0] 2024-06-03T09:06:56.522 INFO:teuthology.orchestra.run.smithi088.stderr:2024-06-03T09:06:56.521+0000 7fc02dee2640 10 monclient(hunting): picked mon.noname-b con 0x7fc0280a8960 addr v1:172.21.15.88:6790/0 2024-06-03T09:06:56.522 INFO:teuthology.orchestra.run.smithi088.stderr:2024-06-03T09:06:56.521+0000 7fc02dee2640 10 monclient(hunting): picked mon.noname-c con 0x7fc0281450f0 addr v1:172.21.15.192:6789/0 2024-06-03T09:06:56.523 INFO:teuthology.orchestra.run.smithi088.stderr:2024-06-03T09:06:56.521+0000 7fc02dee2640 10 monclient(hunting): picked mon.noname-a con 0x7fc02813eec0 addr v1:172.21.15.88:6789/0 2024-06-03T09:06:56.523 INFO:teuthology.orchestra.run.smithi088.stderr:2024-06-03T09:06:56.521+0000 7fc02dee2640 1 -- --> v1:172.21.15.88:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fc028144c60 con 0x7fc02813eec0 2024-06-03T09:06:56.523 INFO:teuthology.orchestra.run.smithi088.stderr:2024-06-03T09:06:56.521+0000 7fc02dee2640 1 -- --> v1:172.21.15.192:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fc0280b3930 con 0x7fc0281450f0 2024-06-03T09:06:56.523 INFO:teuthology.orchestra.run.smithi088.stderr:2024-06-03T09:06:56.521+0000 7fc02dee2640 1 -- --> v1:172.21.15.88:6790/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fc0280b1ad0 con 0x7fc0280a8960 2024-06-03T09:06:56.523 INFO:teuthology.orchestra.run.smithi088.stderr:2024-06-03T09:06:56.521+0000 7fc02dee2640 10 monclient(hunting): _renew_subs 2024-06-03T09:06:56.523 INFO:teuthology.orchestra.run.smithi088.stderr:2024-06-03T09:06:56.521+0000 7fc02dee2640 10 monclient(hunting): authenticate will time out at 3072.331590s 2024-06-03T09:06:56.523 INFO:teuthology.orchestra.run.smithi088.stderr:2024-06-03T09:06:56.522+0000 7fc0277fe640 1 --1- >> v1:172.21.15.88:6790/0 conn(0x7fc0280a8960 0x7fc0280a8d40 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.88:6790/0 says I am v1:172.21.15.88:52428/0 (socket says 172.21.15.88:52428) 2024-06-03T09:06:56.523 INFO:teuthology.orchestra.run.smithi088.stderr:2024-06-03T09:06:56.522+0000 7fc0277fe640 1 -- 172.21.15.88:0/3717201828 learned_addr learned my addr 172.21.15.88:0/3717201828 (peer_addr_for_me v1:172.21.15.88:0/0) 2024-06-03T09:06:56.523 INFO:teuthology.orchestra.run.smithi088.stderr:2024-06-03T09:06:56.524+0000 7fc026ffd640 1 -- 172.21.15.88:0/3717201828 <== mon.1 v1:172.21.15.192:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 1915111280 0 0) 0x7fc0280b3930 con 0x7fc0281450f0 2024-06-03T09:06:56.524 INFO:teuthology.orchestra.run.smithi088.stderr:2024-06-03T09:06:56.524+0000 7fc026ffd640 10 monclient(hunting): _negotiate 2024-06-03T09:06:56.524 INFO:teuthology.orchestra.run.smithi088.stderr:2024-06-03T09:06:56.524+0000 7fc026ffd640 10 monclient(hunting): _init_auth method 2 2024-06-03T09:06:56.524 INFO:teuthology.orchestra.run.smithi088.stderr:2024-06-03T09:06:56.524+0000 7fc026ffd640 10 monclient(hunting): _init_auth creating new auth 2024-06-03T09:06:56.524 INFO:teuthology.orchestra.run.smithi088.stderr:2024-06-03T09:06:56.524+0000 7fc026ffd640 10 monclient(hunting): my global_id is 30313 2024-06-03T09:06:56.524 INFO:teuthology.orchestra.run.smithi088.stderr:2024-06-03T09:06:56.524+0000 7fc026ffd640 1 -- 172.21.15.88:0/3717201828 --> v1:172.21.15.192:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7fc014003520 con 0x7fc0281450f0 2024-06-03T09:06:56.525 INFO:teuthology.orchestra.run.smithi088.stderr:2024-06-03T09:06:56.525+0000 7fc026ffd640 1 -- 172.21.15.88:0/3717201828 <== mon.2 v1:172.21.15.88:6790/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 1852288509 0 0) 0x7fc0280b1ad0 con 0x7fc0280a8960 2024-06-03T09:06:56.525 INFO:teuthology.orchestra.run.smithi088.stderr:2024-06-03T09:06:56.525+0000 7fc026ffd640 10 monclient(hunting): _negotiate 2024-06-03T09:06:56.525 INFO:teuthology.orchestra.run.smithi088.stderr:2024-06-03T09:06:56.525+0000 7fc026ffd640 10 monclient(hunting): _init_auth method 2 2024-06-03T09:06:56.525 INFO:teuthology.orchestra.run.smithi088.stderr:2024-06-03T09:06:56.525+0000 7fc026ffd640 10 monclient(hunting): _init_auth creating new auth 2024-06-03T09:06:56.525 INFO:teuthology.orchestra.run.smithi088.stderr:2024-06-03T09:06:56.525+0000 7fc026ffd640 10 monclient(hunting): my global_id is 30248 2024-06-03T09:06:56.526 INFO:teuthology.orchestra.run.smithi088.stderr:2024-06-03T09:06:56.525+0000 7fc026ffd640 1 -- 172.21.15.88:0/3717201828 --> v1:172.21.15.88:6790/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7fc0280b3930 con 0x7fc0280a8960 2024-06-03T09:06:56.526 INFO:teuthology.orchestra.run.smithi088.stderr:2024-06-03T09:06:56.525+0000 7fc026ffd640 1 -- 172.21.15.88:0/3717201828 <== mon.0 v1:172.21.15.88:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 114139513 0 0) 0x7fc028144c60 con 0x7fc02813eec0 2024-06-03T09:06:56.526 INFO:teuthology.orchestra.run.smithi088.stderr:2024-06-03T09:06:56.525+0000 7fc026ffd640 10 monclient(hunting): _negotiate 2024-06-03T09:06:56.526 INFO:teuthology.orchestra.run.smithi088.stderr:2024-06-03T09:06:56.525+0000 7fc026ffd640 10 monclient(hunting): _init_auth method 2 2024-06-03T09:06:56.526 INFO:teuthology.orchestra.run.smithi088.stderr:2024-06-03T09:06:56.525+0000 7fc026ffd640 10 monclient(hunting): _init_auth creating new auth 2024-06-03T09:06:56.526 INFO:teuthology.orchestra.run.smithi088.stderr:2024-06-03T09:06:56.525+0000 7fc026ffd640 10 monclient(hunting): my global_id is 30744 2024-06-03T09:06:56.526 INFO:teuthology.orchestra.run.smithi088.stderr:2024-06-03T09:06:56.525+0000 7fc026ffd640 1 -- 172.21.15.88:0/3717201828 --> v1:172.21.15.88:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7fc0280b1ad0 con 0x7fc02813eec0 2024-06-03T09:06:56.528 INFO:teuthology.orchestra.run.smithi088.stderr:2024-06-03T09:06:56.527+0000 7fc026ffd640 1 -- 172.21.15.88:0/3717201828 <== mon.1 v1:172.21.15.192:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 214+0+0 (unknown 2418338584 0 0) 0x7fc014003520 con 0x7fc0281450f0 2024-06-03T09:06:56.528 INFO:teuthology.orchestra.run.smithi088.stderr:2024-06-03T09:06:56.527+0000 7fc026ffd640 1 -- 172.21.15.88:0/3717201828 >> v1:172.21.15.88:6790/0 conn(0x7fc0280a8960 legacy=0x7fc0280a8d40 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-06-03T09:06:56.528 INFO:teuthology.orchestra.run.smithi088.stderr:2024-06-03T09:06:56.527+0000 7fc026ffd640 1 -- 172.21.15.88:0/3717201828 >> v1:172.21.15.88:6789/0 conn(0x7fc02813eec0 legacy=0x7fc02814ce60 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-06-03T09:06:56.528 INFO:teuthology.orchestra.run.smithi088.stderr:2024-06-03T09:06:56.527+0000 7fc026ffd640 10 monclient: _finish_hunting 0 2024-06-03T09:06:56.528 INFO:teuthology.orchestra.run.smithi088.stderr:2024-06-03T09:06:56.527+0000 7fc026ffd640 1 monclient: found mon.noname-c 2024-06-03T09:06:56.528 INFO:teuthology.orchestra.run.smithi088.stderr:2024-06-03T09:06:56.527+0000 7fc026ffd640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-06-03T09:06:56.528 INFO:teuthology.orchestra.run.smithi088.stderr:2024-06-03T09:06:56.527+0000 7fc026ffd640 10 monclient: _send_mon_message to mon.noname-c at v1:172.21.15.192:6789/0 2024-06-03T09:06:56.528 INFO:teuthology.orchestra.run.smithi088.stderr:2024-06-03T09:06:56.527+0000 7fc026ffd640 1 -- 172.21.15.88:0/3717201828 --> v1:172.21.15.192:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fc0280b1c60 con 0x7fc0281450f0 2024-06-03T09:06:56.528 INFO:teuthology.orchestra.run.smithi088.stderr:2024-06-03T09:06:56.527+0000 7fc026ffd640 10 monclient: _finish_auth 0 2024-06-03T09:06:56.528 INFO:teuthology.orchestra.run.smithi088.stderr:2024-06-03T09:06:56.527+0000 7fc026ffd640 10 monclient: _check_auth_tickets 2024-06-03T09:06:56.528 INFO:teuthology.orchestra.run.smithi088.stderr:2024-06-03T09:06:56.527+0000 7fc026ffd640 20 monclient: _check_auth_rotating not needed by client.admin 2024-06-03T09:06:56.528 INFO:teuthology.orchestra.run.smithi088.stderr:2024-06-03T09:06:56.527+0000 7fc026ffd640 1 -- 172.21.15.88:0/3717201828 <== mon.1 v1:172.21.15.192:6789/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 2234374065 0 0) 0x7fc010001610 con 0x7fc0281450f0 2024-06-03T09:06:56.529 INFO:teuthology.orchestra.run.smithi088.stderr:2024-06-03T09:06:56.527+0000 7fc026ffd640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-06-03T09:06:56.529 INFO:teuthology.orchestra.run.smithi088.stderr:2024-06-03T09:06:56.527+0000 7fc026ffd640 10 monclient: got monmap 1 from mon.noname-c (according to old e1) 2024-06-03T09:06:56.529 INFO:teuthology.orchestra.run.smithi088.stderr:2024-06-03T09:06:56.527+0000 7fc026ffd640 10 monclient: dump: 2024-06-03T09:06:56.529 INFO:teuthology.orchestra.run.smithi088.stderr:epoch 1 2024-06-03T09:06:56.529 INFO:teuthology.orchestra.run.smithi088.stderr:fsid ec3c3fdd-3ac6-4d7d-9412-683130ed8b04 2024-06-03T09:06:56.529 INFO:teuthology.orchestra.run.smithi088.stderr:last_changed 2024-06-03T08:34:31.885125+0000 2024-06-03T09:06:56.529 INFO:teuthology.orchestra.run.smithi088.stderr:created 2024-06-03T08:34:31.885125+0000 2024-06-03T09:06:56.529 INFO:teuthology.orchestra.run.smithi088.stderr:min_mon_release 18 (reef) 2024-06-03T09:06:56.529 INFO:teuthology.orchestra.run.smithi088.stderr:election_strategy: 3 2024-06-03T09:06:56.529 INFO:teuthology.orchestra.run.smithi088.stderr:0: v1:172.21.15.88:6789/0 mon.a 2024-06-03T09:06:56.529 INFO:teuthology.orchestra.run.smithi088.stderr:1: v1:172.21.15.192:6789/0 mon.b 2024-06-03T09:06:56.529 INFO:teuthology.orchestra.run.smithi088.stderr:2: v1:172.21.15.88:6790/0 mon.c 2024-06-03T09:06:56.529 INFO:teuthology.orchestra.run.smithi088.stderr: 2024-06-03T09:06:56.529 INFO:teuthology.orchestra.run.smithi088.stderr:2024-06-03T09:06:56.527+0000 7fc02dee2640 5 monclient: authenticate success, global_id 30313 2024-06-03T09:06:56.530 INFO:teuthology.orchestra.run.smithi088.stderr:2024-06-03T09:06:56.527+0000 7fc02dee2640 20 monclient: get_monmap_and_config waiting for monmap|config 2024-06-03T09:06:56.532 INFO:teuthology.orchestra.run.smithi088.stderr:2024-06-03T09:06:56.529+0000 7fc026ffd640 1 -- 172.21.15.88:0/3717201828 <== mon.1 v1:172.21.15.192:6789/0 4 ==== config(0 keys) v1 ==== 4+0+0 (unknown 0 0 0) 0x7fc010001830 con 0x7fc0281450f0 2024-06-03T09:06:56.532 INFO:teuthology.orchestra.run.smithi088.stderr:2024-06-03T09:06:56.529+0000 7fc026ffd640 10 monclient: handle_config config(0 keys) v1 2024-06-03T09:06:56.533 INFO:teuthology.orchestra.run.smithi088.stderr:2024-06-03T09:06:56.529+0000 7fc026ffd640 1 -- 172.21.15.88:0/3717201828 <== mon.1 v1:172.21.15.192:6789/0 5 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 2234374065 0 0) 0x7fc010001b40 con 0x7fc0281450f0 2024-06-03T09:06:56.533 INFO:teuthology.orchestra.run.smithi088.stderr:2024-06-03T09:06:56.529+0000 7fc026ffd640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-06-03T09:06:56.533 INFO:teuthology.orchestra.run.smithi088.stderr:2024-06-03T09:06:56.529+0000 7fc026ffd640 10 monclient: got monmap 1 from mon.b (according to old e1) 2024-06-03T09:06:56.533 INFO:teuthology.orchestra.run.smithi088.stderr:2024-06-03T09:06:56.529+0000 7fc026ffd640 10 monclient: dump: 2024-06-03T09:06:56.533 INFO:teuthology.orchestra.run.smithi088.stderr:epoch 1 2024-06-03T09:06:56.533 INFO:teuthology.orchestra.run.smithi088.stderr:fsid ec3c3fdd-3ac6-4d7d-9412-683130ed8b04 2024-06-03T09:06:56.533 INFO:teuthology.orchestra.run.smithi088.stderr:last_changed 2024-06-03T08:34:31.885125+0000 2024-06-03T09:06:56.533 INFO:teuthology.orchestra.run.smithi088.stderr:created 2024-06-03T08:34:31.885125+0000 2024-06-03T09:06:56.533 INFO:teuthology.orchestra.run.smithi088.stderr:min_mon_release 18 (reef) 2024-06-03T09:06:56.533 INFO:teuthology.orchestra.run.smithi088.stderr:election_strategy: 3 2024-06-03T09:06:56.533 INFO:teuthology.orchestra.run.smithi088.stderr:0: v1:172.21.15.88:6789/0 mon.a 2024-06-03T09:06:56.533 INFO:teuthology.orchestra.run.smithi088.stderr:1: v1:172.21.15.192:6789/0 mon.b 2024-06-03T09:06:56.533 INFO:teuthology.orchestra.run.smithi088.stderr:2: v1:172.21.15.88:6790/0 mon.c 2024-06-03T09:06:56.534 INFO:teuthology.orchestra.run.smithi088.stderr: 2024-06-03T09:06:56.534 INFO:teuthology.orchestra.run.smithi088.stderr:2024-06-03T09:06:56.529+0000 7fc02dee2640 10 monclient: get_monmap_and_config success 2024-06-03T09:06:56.534 INFO:teuthology.orchestra.run.smithi088.stderr:2024-06-03T09:06:56.529+0000 7fc02dee2640 4 set_mon_vals no callback set 2024-06-03T09:06:56.534 INFO:teuthology.orchestra.run.smithi088.stderr:2024-06-03T09:06:56.530+0000 7fc02dee2640 10 monclient: shutdown 2024-06-03T09:06:56.534 INFO:teuthology.orchestra.run.smithi088.stderr:2024-06-03T09:06:56.530+0000 7fc02dee2640 20 monclient: shutdown discarding 0 pending message(s) 2024-06-03T09:06:56.534 INFO:teuthology.orchestra.run.smithi088.stderr:2024-06-03T09:06:56.530+0000 7fc02dee2640 1 -- 172.21.15.88:0/3717201828 >> v1:172.21.15.192:6789/0 conn(0x7fc0281450f0 legacy=0x7fc02813e7b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-06-03T09:06:56.534 INFO:teuthology.orchestra.run.smithi088.stderr:2024-06-03T09:06:56.530+0000 7fc02dee2640 1 -- 172.21.15.88:0/3717201828 shutdown_connections 2024-06-03T09:06:56.534 INFO:teuthology.orchestra.run.smithi088.stderr:2024-06-03T09:06:56.530+0000 7fc02dee2640 1 -- 172.21.15.88:0/3717201828 >> 172.21.15.88:0/3717201828 conn(0x7fc0280b70d0 msgr2=0x7fc0280b74e0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-06-03T09:06:56.534 INFO:teuthology.orchestra.run.smithi088.stderr:2024-06-03T09:06:56.530+0000 7fc02dee2640 1 -- 172.21.15.88:0/3717201828 shutdown_connections 2024-06-03T09:06:56.534 INFO:teuthology.orchestra.run.smithi088.stderr:2024-06-03T09:06:56.530+0000 7fc02dee2640 1 -- 172.21.15.88:0/3717201828 wait complete. 2024-06-03T09:06:56.534 INFO:teuthology.orchestra.run.smithi088.stderr:2024-06-03T09:06:56.530+0000 7fc02dee2640 10 monclient: build_initial_monmap 2024-06-03T09:06:56.534 INFO:teuthology.orchestra.run.smithi088.stderr:2024-06-03T09:06:56.530+0000 7fc02dee2640 10 monclient: monmap: 2024-06-03T09:06:56.534 INFO:teuthology.orchestra.run.smithi088.stderr:epoch 0 2024-06-03T09:06:56.534 INFO:teuthology.orchestra.run.smithi088.stderr:fsid 00000000-0000-0000-0000-000000000000 2024-06-03T09:06:56.535 INFO:teuthology.orchestra.run.smithi088.stderr:last_changed 0.000000 2024-06-03T09:06:56.535 INFO:teuthology.orchestra.run.smithi088.stderr:created 0.000000 2024-06-03T09:06:56.535 INFO:teuthology.orchestra.run.smithi088.stderr:min_mon_release 0 (unknown) 2024-06-03T09:06:56.535 INFO:teuthology.orchestra.run.smithi088.stderr:election_strategy: 1 2024-06-03T09:06:56.535 INFO:teuthology.orchestra.run.smithi088.stderr:0: v1:172.21.15.88:6789/0 mon.noname-a 2024-06-03T09:06:56.535 INFO:teuthology.orchestra.run.smithi088.stderr:1: v1:172.21.15.192:6789/0 mon.noname-b 2024-06-03T09:06:56.535 INFO:teuthology.orchestra.run.smithi088.stderr:2: v1:172.21.15.88:6790/0 mon.noname-c 2024-06-03T09:06:56.535 INFO:teuthology.orchestra.run.smithi088.stderr: 2024-06-03T09:06:56.535 INFO:teuthology.orchestra.run.smithi088.stderr:2024-06-03T09:06:56.531+0000 7fc02dee2640 1 librados: starting msgr at 2024-06-03T09:06:56.535 INFO:teuthology.orchestra.run.smithi088.stderr:2024-06-03T09:06:56.531+0000 7fc02dee2640 1 librados: starting objecter 2024-06-03T09:06:56.535 INFO:teuthology.orchestra.run.smithi088.stderr:2024-06-03T09:06:56.531+0000 7fc02dee2640 1 Processor -- start 2024-06-03T09:06:56.535 INFO:teuthology.orchestra.run.smithi088.stderr:2024-06-03T09:06:56.531+0000 7fc02dee2640 1 -- start start 2024-06-03T09:06:56.535 INFO:teuthology.orchestra.run.smithi088.stderr:2024-06-03T09:06:56.531+0000 7fc02dee2640 1 librados: setting wanted keys 2024-06-03T09:06:56.536 INFO:teuthology.orchestra.run.smithi088.stderr:2024-06-03T09:06:56.531+0000 7fc02dee2640 1 librados: calling monclient init 2024-06-03T09:06:56.536 INFO:teuthology.orchestra.run.smithi088.stderr:2024-06-03T09:06:56.531+0000 7fc02dee2640 10 monclient: init 2024-06-03T09:06:56.536 INFO:teuthology.orchestra.run.smithi088.stderr:2024-06-03T09:06:56.531+0000 7fc02dee2640 10 monclient: _reopen_session rank -1 2024-06-03T09:06:56.536 INFO:teuthology.orchestra.run.smithi088.stderr:2024-06-03T09:06:56.531+0000 7fc02dee2640 10 monclient: _add_conns ranks=[0,1,2] 2024-06-03T09:06:56.536 INFO:teuthology.orchestra.run.smithi088.stderr:2024-06-03T09:06:56.531+0000 7fc02dee2640 10 monclient(hunting): picked mon.noname-a con 0x7fc0280a8960 addr v1:172.21.15.88:6789/0 2024-06-03T09:06:56.536 INFO:teuthology.orchestra.run.smithi088.stderr:2024-06-03T09:06:56.531+0000 7fc02dee2640 10 monclient(hunting): picked mon.noname-b con 0x7fc02813eec0 addr v1:172.21.15.192:6789/0 2024-06-03T09:06:56.536 INFO:teuthology.orchestra.run.smithi088.stderr:2024-06-03T09:06:56.531+0000 7fc02dee2640 10 monclient(hunting): picked mon.noname-c con 0x7fc0281450f0 addr v1:172.21.15.88:6790/0 2024-06-03T09:06:56.536 INFO:teuthology.orchestra.run.smithi088.stderr:2024-06-03T09:06:56.531+0000 7fc02dee2640 1 -- --> v1:172.21.15.88:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fc0280ab540 con 0x7fc0280a8960 2024-06-03T09:06:56.536 INFO:teuthology.orchestra.run.smithi088.stderr:2024-06-03T09:06:56.531+0000 7fc02dee2640 1 -- --> v1:172.21.15.192:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fc0281d6460 con 0x7fc02813eec0 2024-06-03T09:06:56.536 INFO:teuthology.orchestra.run.smithi088.stderr:2024-06-03T09:06:56.531+0000 7fc02dee2640 1 -- --> v1:172.21.15.88:6790/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fc028140fe0 con 0x7fc0281450f0 2024-06-03T09:06:56.536 INFO:teuthology.orchestra.run.smithi088.stderr:2024-06-03T09:06:56.531+0000 7fc02dee2640 10 monclient(hunting): _renew_subs 2024-06-03T09:06:56.536 INFO:teuthology.orchestra.run.smithi088.stderr:2024-06-03T09:06:56.531+0000 7fc02dee2640 10 monclient(hunting): authenticate will time out at 3072.341781s 2024-06-03T09:06:56.536 INFO:teuthology.orchestra.run.smithi088.stderr:2024-06-03T09:06:56.532+0000 7fc027fff640 1 --1- >> v1:172.21.15.88:6790/0 conn(0x7fc0281450f0 0x7fc0281e1d80 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=1).handle_server_banner_and_identify peer v1:172.21.15.88:6790/0 says I am v1:172.21.15.88:52440/0 (socket says 172.21.15.88:52440) 2024-06-03T09:06:56.536 INFO:teuthology.orchestra.run.smithi088.stderr:2024-06-03T09:06:56.532+0000 7fc027fff640 1 -- 172.21.15.88:0/3384061971 learned_addr learned my addr 172.21.15.88:0/3384061971 (peer_addr_for_me v1:172.21.15.88:0/0) 2024-06-03T09:06:56.537 INFO:teuthology.orchestra.run.smithi088.stderr:2024-06-03T09:06:56.532+0000 7fc024ff9640 10 client.?.objecter ms_handle_connect 0x7fc0281450f0 2024-06-03T09:06:56.537 INFO:teuthology.orchestra.run.smithi088.stderr:2024-06-03T09:06:56.532+0000 7fc024ff9640 10 client.?.objecter resend_mon_ops 2024-06-03T09:06:56.537 INFO:teuthology.orchestra.run.smithi088.stderr:2024-06-03T09:06:56.532+0000 7fc024ff9640 10 client.?.objecter ms_handle_connect 0x7fc0280a8960 2024-06-03T09:06:56.537 INFO:teuthology.orchestra.run.smithi088.stderr:2024-06-03T09:06:56.532+0000 7fc024ff9640 10 client.?.objecter resend_mon_ops 2024-06-03T09:06:56.537 INFO:teuthology.orchestra.run.smithi088.stderr:2024-06-03T09:06:56.532+0000 7fc024ff9640 10 client.?.objecter ms_handle_connect 0x7fc02813eec0 2024-06-03T09:06:56.537 INFO:teuthology.orchestra.run.smithi088.stderr:2024-06-03T09:06:56.532+0000 7fc024ff9640 10 client.?.objecter resend_mon_ops 2024-06-03T09:06:56.538 INFO:teuthology.orchestra.run.smithi088.stderr:2024-06-03T09:06:56.534+0000 7fc024ff9640 1 -- 172.21.15.88:0/3384061971 <== mon.1 v1:172.21.15.192:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 924611662 0 0) 0x7fc0281d6460 con 0x7fc02813eec0 2024-06-03T09:06:56.538 INFO:teuthology.orchestra.run.smithi088.stderr:2024-06-03T09:06:56.534+0000 7fc024ff9640 10 monclient(hunting): _negotiate 2024-06-03T09:06:56.538 INFO:teuthology.orchestra.run.smithi088.stderr:2024-06-03T09:06:56.534+0000 7fc024ff9640 10 monclient(hunting): _init_auth method 2 2024-06-03T09:06:56.538 INFO:teuthology.orchestra.run.smithi088.stderr:2024-06-03T09:06:56.534+0000 7fc024ff9640 10 monclient(hunting): _init_auth creating new auth 2024-06-03T09:06:56.538 INFO:teuthology.orchestra.run.smithi088.stderr:2024-06-03T09:06:56.534+0000 7fc024ff9640 10 monclient(hunting): my global_id is 30316 2024-06-03T09:06:56.538 INFO:teuthology.orchestra.run.smithi088.stderr:2024-06-03T09:06:56.534+0000 7fc024ff9640 1 -- 172.21.15.88:0/3384061971 --> v1:172.21.15.192:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7fc008004eb0 con 0x7fc02813eec0 2024-06-03T09:06:56.538 INFO:teuthology.orchestra.run.smithi088.stderr:2024-06-03T09:06:56.534+0000 7fc024ff9640 1 -- 172.21.15.88:0/3384061971 <== mon.2 v1:172.21.15.88:6790/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 2214925646 0 0) 0x7fc028140fe0 con 0x7fc0281450f0 2024-06-03T09:06:56.538 INFO:teuthology.orchestra.run.smithi088.stderr:2024-06-03T09:06:56.534+0000 7fc024ff9640 10 monclient(hunting): _negotiate 2024-06-03T09:06:56.538 INFO:teuthology.orchestra.run.smithi088.stderr:2024-06-03T09:06:56.534+0000 7fc024ff9640 10 monclient(hunting): _init_auth method 2 2024-06-03T09:06:56.538 INFO:teuthology.orchestra.run.smithi088.stderr:2024-06-03T09:06:56.534+0000 7fc024ff9640 10 monclient(hunting): _init_auth creating new auth 2024-06-03T09:06:56.538 INFO:teuthology.orchestra.run.smithi088.stderr:2024-06-03T09:06:56.534+0000 7fc024ff9640 10 monclient(hunting): my global_id is 30251 2024-06-03T09:06:56.538 INFO:teuthology.orchestra.run.smithi088.stderr:2024-06-03T09:06:56.534+0000 7fc024ff9640 1 -- 172.21.15.88:0/3384061971 --> v1:172.21.15.88:6790/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7fc0281d6460 con 0x7fc0281450f0 2024-06-03T09:06:56.538 INFO:teuthology.orchestra.run.smithi088.stderr:2024-06-03T09:06:56.534+0000 7fc024ff9640 1 -- 172.21.15.88:0/3384061971 <== mon.0 v1:172.21.15.88:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 1193948567 0 0) 0x7fc0280ab540 con 0x7fc0280a8960 2024-06-03T09:06:56.539 INFO:teuthology.orchestra.run.smithi088.stderr:2024-06-03T09:06:56.534+0000 7fc024ff9640 10 monclient(hunting): _negotiate 2024-06-03T09:06:56.539 INFO:teuthology.orchestra.run.smithi088.stderr:2024-06-03T09:06:56.534+0000 7fc024ff9640 10 monclient(hunting): _init_auth method 2 2024-06-03T09:06:56.539 INFO:teuthology.orchestra.run.smithi088.stderr:2024-06-03T09:06:56.534+0000 7fc024ff9640 10 monclient(hunting): _init_auth creating new auth 2024-06-03T09:06:56.539 INFO:teuthology.orchestra.run.smithi088.stderr:2024-06-03T09:06:56.534+0000 7fc024ff9640 10 monclient(hunting): my global_id is 30747 2024-06-03T09:06:56.539 INFO:teuthology.orchestra.run.smithi088.stderr:2024-06-03T09:06:56.534+0000 7fc024ff9640 1 -- 172.21.15.88:0/3384061971 --> v1:172.21.15.88:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7fc028140fe0 con 0x7fc0280a8960 2024-06-03T09:06:56.540 INFO:teuthology.orchestra.run.smithi088.stderr:2024-06-03T09:06:56.538+0000 7fc024ff9640 1 -- 172.21.15.88:0/3384061971 <== mon.1 v1:172.21.15.192:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 764+0+0 (unknown 579313552 0 0) 0x7fc008004eb0 con 0x7fc02813eec0 2024-06-03T09:06:56.540 INFO:teuthology.orchestra.run.smithi088.stderr:2024-06-03T09:06:56.539+0000 7fc024ff9640 1 -- 172.21.15.88:0/3384061971 --> v1:172.21.15.192:6789/0 -- auth(proto 2 165 bytes epoch 0) v1 -- 0x7fc0280ab540 con 0x7fc02813eec0 2024-06-03T09:06:56.540 INFO:teuthology.orchestra.run.smithi088.stderr:2024-06-03T09:06:56.539+0000 7fc024ff9640 1 -- 172.21.15.88:0/3384061971 <== mon.1 v1:172.21.15.192:6789/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 2234374065 0 0) 0x7fc010001b40 con 0x7fc02813eec0 2024-06-03T09:06:56.540 INFO:teuthology.orchestra.run.smithi088.stderr:2024-06-03T09:06:56.539+0000 7fc024ff9640 10 monclient(hunting): handle_monmap mon_map magic: 0 v1 2024-06-03T09:06:56.540 INFO:teuthology.orchestra.run.smithi088.stderr:2024-06-03T09:06:56.539+0000 7fc024ff9640 10 monclient(hunting): got monmap 1 from mon.noname-b (according to old e1) 2024-06-03T09:06:56.540 INFO:teuthology.orchestra.run.smithi088.stderr:2024-06-03T09:06:56.539+0000 7fc024ff9640 10 monclient(hunting): dump: 2024-06-03T09:06:56.540 INFO:teuthology.orchestra.run.smithi088.stderr:epoch 1 2024-06-03T09:06:56.540 INFO:teuthology.orchestra.run.smithi088.stderr:fsid ec3c3fdd-3ac6-4d7d-9412-683130ed8b04 2024-06-03T09:06:56.540 INFO:teuthology.orchestra.run.smithi088.stderr:last_changed 2024-06-03T08:34:31.885125+0000 2024-06-03T09:06:56.540 INFO:teuthology.orchestra.run.smithi088.stderr:created 2024-06-03T08:34:31.885125+0000 2024-06-03T09:06:56.541 INFO:teuthology.orchestra.run.smithi088.stderr:min_mon_release 18 (reef) 2024-06-03T09:06:56.541 INFO:teuthology.orchestra.run.smithi088.stderr:election_strategy: 3 2024-06-03T09:06:56.541 INFO:teuthology.orchestra.run.smithi088.stderr:0: v1:172.21.15.88:6789/0 mon.a 2024-06-03T09:06:56.541 INFO:teuthology.orchestra.run.smithi088.stderr:1: v1:172.21.15.192:6789/0 mon.b 2024-06-03T09:06:56.541 INFO:teuthology.orchestra.run.smithi088.stderr:2: v1:172.21.15.88:6790/0 mon.c 2024-06-03T09:06:56.541 INFO:teuthology.orchestra.run.smithi088.stderr: 2024-06-03T09:06:56.541 INFO:teuthology.orchestra.run.smithi088.stderr:2024-06-03T09:06:56.539+0000 7fc024ff9640 10 monclient(hunting): _finish_auth 0 2024-06-03T09:06:56.541 INFO:teuthology.orchestra.run.smithi088.stderr:2024-06-03T09:06:56.539+0000 7fc024ff9640 1 -- 172.21.15.88:0/3384061971 <== mon.0 v1:172.21.15.88:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 764+0+0 (unknown 2727226113 0 0) 0x7fc028140fe0 con 0x7fc0280a8960 2024-06-03T09:06:56.541 INFO:teuthology.orchestra.run.smithi088.stderr:2024-06-03T09:06:56.539+0000 7fc024ff9640 1 -- 172.21.15.88:0/3384061971 --> v1:172.21.15.88:6789/0 -- auth(proto 2 165 bytes epoch 0) v1 -- 0x7fc008004eb0 con 0x7fc0280a8960 2024-06-03T09:06:56.541 INFO:teuthology.orchestra.run.smithi088.stderr:2024-06-03T09:06:56.539+0000 7fc024ff9640 1 -- 172.21.15.88:0/3384061971 <== mon.2 v1:172.21.15.88:6790/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 764+0+0 (unknown 2813798943 0 0) 0x7fc0281d6460 con 0x7fc0281450f0 2024-06-03T09:06:56.541 INFO:teuthology.orchestra.run.smithi088.stderr:2024-06-03T09:06:56.539+0000 7fc024ff9640 1 -- 172.21.15.88:0/3384061971 --> v1:172.21.15.88:6790/0 -- auth(proto 2 165 bytes epoch 0) v1 -- 0x7fc028140fe0 con 0x7fc0281450f0 2024-06-03T09:06:56.541 INFO:teuthology.orchestra.run.smithi088.stderr:2024-06-03T09:06:56.539+0000 7fc024ff9640 1 -- 172.21.15.88:0/3384061971 <== mon.2 v1:172.21.15.88:6790/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 2234374065 0 0) 0x7fc020001c30 con 0x7fc0281450f0 2024-06-03T09:06:56.541 INFO:teuthology.orchestra.run.smithi088.stderr:2024-06-03T09:06:56.539+0000 7fc024ff9640 10 monclient(hunting): handle_monmap mon_map magic: 0 v1 2024-06-03T09:06:56.541 INFO:teuthology.orchestra.run.smithi088.stderr:2024-06-03T09:06:56.539+0000 7fc024ff9640 10 monclient(hunting): got monmap 1 from mon.c (according to old e1) 2024-06-03T09:06:56.542 INFO:teuthology.orchestra.run.smithi088.stderr:2024-06-03T09:06:56.539+0000 7fc024ff9640 10 monclient(hunting): dump: 2024-06-03T09:06:56.542 INFO:teuthology.orchestra.run.smithi088.stderr:epoch 1 2024-06-03T09:06:56.542 INFO:teuthology.orchestra.run.smithi088.stderr:fsid ec3c3fdd-3ac6-4d7d-9412-683130ed8b04 2024-06-03T09:06:56.542 INFO:teuthology.orchestra.run.smithi088.stderr:last_changed 2024-06-03T08:34:31.885125+0000 2024-06-03T09:06:56.542 INFO:teuthology.orchestra.run.smithi088.stderr:created 2024-06-03T08:34:31.885125+0000 2024-06-03T09:06:56.542 INFO:teuthology.orchestra.run.smithi088.stderr:min_mon_release 18 (reef) 2024-06-03T09:06:56.542 INFO:teuthology.orchestra.run.smithi088.stderr:election_strategy: 3 2024-06-03T09:06:56.542 INFO:teuthology.orchestra.run.smithi088.stderr:0: v1:172.21.15.88:6789/0 mon.a 2024-06-03T09:06:56.542 INFO:teuthology.orchestra.run.smithi088.stderr:1: v1:172.21.15.192:6789/0 mon.b 2024-06-03T09:06:56.542 INFO:teuthology.orchestra.run.smithi088.stderr:2: v1:172.21.15.88:6790/0 mon.c 2024-06-03T09:06:56.542 INFO:teuthology.orchestra.run.smithi088.stderr: 2024-06-03T09:06:56.543 INFO:teuthology.orchestra.run.smithi088.stderr:2024-06-03T09:06:56.541+0000 7fc024ff9640 1 -- 172.21.15.88:0/3384061971 <== mon.1 v1:172.21.15.192:6789/0 4 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 580+0+0 (unknown 3968874463 0 0) 0x7fc0280ab540 con 0x7fc02813eec0 2024-06-03T09:06:56.544 INFO:teuthology.orchestra.run.smithi088.stderr:2024-06-03T09:06:56.541+0000 7fc024ff9640 1 -- 172.21.15.88:0/3384061971 >> v1:172.21.15.88:6790/0 conn(0x7fc0281450f0 legacy=0x7fc0281e1d80 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-06-03T09:06:56.544 INFO:teuthology.orchestra.run.smithi088.stderr:2024-06-03T09:06:56.541+0000 7fc024ff9640 1 -- 172.21.15.88:0/3384061971 >> v1:172.21.15.88:6789/0 conn(0x7fc0280a8960 legacy=0x7fc0281d68f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-06-03T09:06:56.544 INFO:teuthology.orchestra.run.smithi088.stderr:2024-06-03T09:06:56.541+0000 7fc024ff9640 10 monclient: _finish_hunting 0 2024-06-03T09:06:56.544 INFO:teuthology.orchestra.run.smithi088.stderr:2024-06-03T09:06:56.541+0000 7fc024ff9640 1 monclient: found mon.b 2024-06-03T09:06:56.544 INFO:teuthology.orchestra.run.smithi088.stderr:2024-06-03T09:06:56.541+0000 7fc024ff9640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-06-03T09:06:56.544 INFO:teuthology.orchestra.run.smithi088.stderr:2024-06-03T09:06:56.541+0000 7fc024ff9640 10 monclient: _send_mon_message to mon.b at v1:172.21.15.192:6789/0 2024-06-03T09:06:56.544 INFO:teuthology.orchestra.run.smithi088.stderr:2024-06-03T09:06:56.541+0000 7fc024ff9640 1 -- 172.21.15.88:0/3384061971 --> v1:172.21.15.192:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fc02813f380 con 0x7fc02813eec0 2024-06-03T09:06:56.544 INFO:teuthology.orchestra.run.smithi088.stderr:2024-06-03T09:06:56.541+0000 7fc024ff9640 10 monclient: _finish_auth 0 2024-06-03T09:06:56.544 INFO:teuthology.orchestra.run.smithi088.stderr:2024-06-03T09:06:56.541+0000 7fc024ff9640 10 monclient: _check_auth_tickets 2024-06-03T09:06:56.544 INFO:teuthology.orchestra.run.smithi088.stderr:2024-06-03T09:06:56.541+0000 7fc024ff9640 20 monclient: _check_auth_rotating not needed by client.admin 2024-06-03T09:06:56.544 INFO:teuthology.orchestra.run.smithi088.stderr:2024-06-03T09:06:56.541+0000 7fc02dee2640 5 monclient: authenticate success, global_id 30316 2024-06-03T09:06:56.544 INFO:teuthology.orchestra.run.smithi088.stderr:2024-06-03T09:06:56.541+0000 7fc02dee2640 10 monclient: _renew_subs 2024-06-03T09:06:56.544 INFO:teuthology.orchestra.run.smithi088.stderr:2024-06-03T09:06:56.541+0000 7fc02dee2640 10 monclient: _send_mon_message to mon.b at v1:172.21.15.192:6789/0 2024-06-03T09:06:56.545 INFO:teuthology.orchestra.run.smithi088.stderr:2024-06-03T09:06:56.541+0000 7fc02dee2640 1 -- 172.21.15.88:0/3384061971 --> v1:172.21.15.192:6789/0 -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fc028142400 con 0x7fc02813eec0 2024-06-03T09:06:56.545 INFO:teuthology.orchestra.run.smithi088.stderr:2024-06-03T09:06:56.541+0000 7fc02dee2640 10 client.30316.objecter _maybe_request_map subscribing (onetime) to next osd map 2024-06-03T09:06:56.545 INFO:teuthology.orchestra.run.smithi088.stderr:2024-06-03T09:06:56.541+0000 7fc02dee2640 10 monclient: _renew_subs 2024-06-03T09:06:56.545 INFO:teuthology.orchestra.run.smithi088.stderr:2024-06-03T09:06:56.541+0000 7fc02dee2640 10 monclient: _send_mon_message to mon.b at v1:172.21.15.192:6789/0 2024-06-03T09:06:56.545 INFO:teuthology.orchestra.run.smithi088.stderr:2024-06-03T09:06:56.541+0000 7fc02dee2640 1 -- 172.21.15.88:0/3384061971 --> v1:172.21.15.192:6789/0 -- mon_subscribe({osdmap=0}) v3 -- 0x7fc0280ab1b0 con 0x7fc02813eec0 2024-06-03T09:06:56.545 INFO:teuthology.orchestra.run.smithi088.stderr:2024-06-03T09:06:56.541+0000 7fc02dee2640 1 librados: init done 2024-06-03T09:06:56.545 INFO:teuthology.orchestra.run.smithi088.stderr:2024-06-03T09:06:56.542+0000 7fc02dee2640 10 start_mon_command cmd=[{"prefix": "get_command_descriptions"}] 2024-06-03T09:06:56.545 INFO:teuthology.orchestra.run.smithi088.stderr:2024-06-03T09:06:56.542+0000 7fc02dee2640 10 monclient: _send_command 1 [{"prefix": "get_command_descriptions"}] 2024-06-03T09:06:56.545 INFO:teuthology.orchestra.run.smithi088.stderr:2024-06-03T09:06:56.542+0000 7fc02dee2640 10 monclient: _send_mon_message to mon.b at v1:172.21.15.192:6789/0 2024-06-03T09:06:56.545 INFO:teuthology.orchestra.run.smithi088.stderr:2024-06-03T09:06:56.542+0000 7fc02dee2640 1 -- 172.21.15.88:0/3384061971 --> v1:172.21.15.192:6789/0 -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fc0280a9e60 con 0x7fc02813eec0 2024-06-03T09:06:56.794 INFO:teuthology.orchestra.run.smithi088.stderr:2024-06-03T09:06:56.794+0000 7fc024ff9640 1 -- 172.21.15.88:0/3384061971 <== mon.1 v1:172.21.15.192:6789/0 5 ==== config(0 keys) v1 ==== 4+0+0 (unknown 0 0 0) 0x7fc010001d60 con 0x7fc02813eec0 2024-06-03T09:06:56.794 INFO:teuthology.orchestra.run.smithi088.stderr:2024-06-03T09:06:56.794+0000 7fc024ff9640 10 monclient: handle_config config(0 keys) v1 2024-06-03T09:06:56.794 INFO:teuthology.orchestra.run.smithi088.stderr:2024-06-03T09:06:56.794+0000 7fc024ff9640 1 -- 172.21.15.88:0/3384061971 <== mon.1 v1:172.21.15.192:6789/0 6 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 2234374065 0 0) 0x7fc0100038a0 con 0x7fc02813eec0 2024-06-03T09:06:56.794 INFO:teuthology.orchestra.run.smithi088.stderr:2024-06-03T09:06:56.794+0000 7fc024ff9640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-06-03T09:06:56.794 INFO:teuthology.orchestra.run.smithi088.stderr:2024-06-03T09:06:56.794+0000 7fc024ff9640 10 monclient: got monmap 1 from mon.b (according to old e1) 2024-06-03T09:06:56.794 INFO:teuthology.orchestra.run.smithi088.stderr:2024-06-03T09:06:56.794+0000 7fc024ff9640 10 monclient: dump: 2024-06-03T09:06:56.795 INFO:teuthology.orchestra.run.smithi088.stderr:epoch 1 2024-06-03T09:06:56.795 INFO:teuthology.orchestra.run.smithi088.stderr:fsid ec3c3fdd-3ac6-4d7d-9412-683130ed8b04 2024-06-03T09:06:56.795 INFO:teuthology.orchestra.run.smithi088.stderr:last_changed 2024-06-03T08:34:31.885125+0000 2024-06-03T09:06:56.795 INFO:teuthology.orchestra.run.smithi088.stderr:created 2024-06-03T08:34:31.885125+0000 2024-06-03T09:06:56.795 INFO:teuthology.orchestra.run.smithi088.stderr:min_mon_release 18 (reef) 2024-06-03T09:06:56.795 INFO:teuthology.orchestra.run.smithi088.stderr:election_strategy: 3 2024-06-03T09:06:56.795 INFO:teuthology.orchestra.run.smithi088.stderr:0: v1:172.21.15.88:6789/0 mon.a 2024-06-03T09:06:56.795 INFO:teuthology.orchestra.run.smithi088.stderr:1: v1:172.21.15.192:6789/0 mon.b 2024-06-03T09:06:56.795 INFO:teuthology.orchestra.run.smithi088.stderr:2: v1:172.21.15.88:6790/0 mon.c 2024-06-03T09:06:56.795 INFO:teuthology.orchestra.run.smithi088.stderr: 2024-06-03T09:06:56.795 INFO:teuthology.orchestra.run.smithi088.stderr:2024-06-03T09:06:56.794+0000 7fc0267fc640 4 set_mon_vals no callback set 2024-06-03T09:06:56.796 INFO:teuthology.orchestra.run.smithi088.stderr:2024-06-03T09:06:56.794+0000 7fc024ff9640 1 -- 172.21.15.88:0/3384061971 <== mon.1 v1:172.21.15.192:6789/0 7 ==== mgrmap(e 5) v1 ==== 89110+0+0 (unknown 873533973 0 0) 0x7fc010019620 con 0x7fc02813eec0 2024-06-03T09:06:56.796 INFO:teuthology.orchestra.run.smithi088.stderr:2024-06-03T09:06:56.795+0000 7fc0277fe640 10 monclient: get_auth_request con 0x7fc00806d560 auth_method 0 2024-06-03T09:06:56.796 INFO:teuthology.orchestra.run.smithi088.stderr:2024-06-03T09:06:56.795+0000 7fc024ff9640 10 client.30316.objecter ms_handle_connect 0x7fc00806d560 2024-06-03T09:06:56.797 INFO:teuthology.orchestra.run.smithi088.stderr:2024-06-03T09:06:56.796+0000 7fc024ff9640 1 -- 172.21.15.88:0/3384061971 <== mon.1 v1:172.21.15.192:6789/0 8 ==== osd_map(1014..1014 src has 622..1014) v4 ==== 5229+0+0 (unknown 3230192161 0 0) 0x7fc010052e20 con 0x7fc02813eec0 2024-06-03T09:06:56.797 INFO:teuthology.orchestra.run.smithi088.stderr:2024-06-03T09:06:56.796+0000 7fc024ff9640 10 client.30316.objecter ms_dispatch 0x7fc028000b90 osd_map(1014..1014 src has 622..1014) v4 2024-06-03T09:06:56.797 INFO:teuthology.orchestra.run.smithi088.stderr:2024-06-03T09:06:56.796+0000 7fc024ff9640 3 client.30316.objecter handle_osd_map got epochs [1014,1014] > 0 2024-06-03T09:06:56.797 INFO:teuthology.orchestra.run.smithi088.stderr:2024-06-03T09:06:56.796+0000 7fc024ff9640 3 client.30316.objecter handle_osd_map decoding full epoch 1014 2024-06-03T09:06:56.797 INFO:teuthology.orchestra.run.smithi088.stderr:2024-06-03T09:06:56.796+0000 7fc024ff9640 20 client.30316.objecter dump_active .. 0 homeless 2024-06-03T09:06:56.798 INFO:teuthology.orchestra.run.smithi088.stderr:2024-06-03T09:06:56.799+0000 7fc024ff9640 1 -- 172.21.15.88:0/3384061971 <== mon.1 v1:172.21.15.192:6789/0 9 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+182498 (unknown 1092875540 0 3016760700) 0x7fc01004e800 con 0x7fc02813eec0 2024-06-03T09:06:56.799 INFO:teuthology.orchestra.run.smithi088.stderr:2024-06-03T09:06:56.799+0000 7fc024ff9640 10 monclient: handle_mon_command_ack 1 [{"prefix": "get_command_descriptions"}] 2024-06-03T09:06:56.799 INFO:teuthology.orchestra.run.smithi088.stderr:2024-06-03T09:06:56.799+0000 7fc024ff9640 10 monclient: _finish_command 1 = system:0 2024-06-03T09:06:56.987 INFO:teuthology.orchestra.run.smithi088.stderr:2024-06-03T09:06:56.987+0000 7fc02dee2640 1 -- 172.21.15.88:0/3384061971 --> v1:172.21.15.192:6800/34229 -- mgr_command(tid 0: {"prefix": "osd deep-scrub", "who": "0", "target": ["mon-mgr", ""]}) v1 -- 0x7fc0281d6610 con 0x7fc00806d560 2024-06-03T09:06:56.987 INFO:teuthology.orchestra.run.smithi088.stderr:2024-06-03T09:06:56.987+0000 7fc024ff9640 1 -- 172.21.15.88:0/3384061971 <== mgr.4110 v1:172.21.15.192:6800/34229 1 ==== mgr_command_reply(tid 0: 0 instructed osd(s) 0 to deep-scrub) v1 ==== 41+0+0 (unknown 2057319476 0 0) 0x7fc0281d6610 con 0x7fc00806d560 2024-06-03T09:06:56.987 INFO:teuthology.orchestra.run.smithi088.stderr:instructed osd(s) 0 to deep-scrub 2024-06-03T09:06:56.989 INFO:teuthology.orchestra.run.smithi088.stderr:2024-06-03T09:06:56.989+0000 7fc02dee2640 10 librados: watch_flush enter 2024-06-03T09:06:56.990 INFO:teuthology.orchestra.run.smithi088.stderr:2024-06-03T09:06:56.989+0000 7fc02dee2640 10 librados: watch_flush exit 2024-06-03T09:06:56.990 INFO:teuthology.orchestra.run.smithi088.stderr:2024-06-03T09:06:56.989+0000 7fc02dee2640 20 client.30316.objecter shutdown clearing up homeless session... 2024-06-03T09:06:56.990 INFO:teuthology.orchestra.run.smithi088.stderr:2024-06-03T09:06:56.989+0000 7fc02dee2640 10 client.30316.objecter successfully canceled tick 2024-06-03T09:06:56.990 INFO:teuthology.orchestra.run.smithi088.stderr:2024-06-03T09:06:56.989+0000 7fc02dee2640 1 -- 172.21.15.88:0/3384061971 >> v1:172.21.15.192:6800/34229 conn(0x7fc00806d560 legacy=0x7fc00806fa20 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-06-03T09:06:56.990 INFO:teuthology.orchestra.run.smithi088.stderr:2024-06-03T09:06:56.989+0000 7fc02dee2640 10 monclient: shutdown 2024-06-03T09:06:56.990 INFO:teuthology.orchestra.run.smithi088.stderr:2024-06-03T09:06:56.989+0000 7fc02dee2640 20 monclient: shutdown discarding 0 pending message(s) 2024-06-03T09:06:56.990 INFO:teuthology.orchestra.run.smithi088.stderr:2024-06-03T09:06:56.989+0000 7fc02dee2640 1 -- 172.21.15.88:0/3384061971 >> v1:172.21.15.192:6789/0 conn(0x7fc02813eec0 legacy=0x7fc028137180 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-06-03T09:06:56.990 INFO:teuthology.orchestra.run.smithi088.stderr:2024-06-03T09:06:56.989+0000 7fc02dee2640 1 -- 172.21.15.88:0/3384061971 shutdown_connections 2024-06-03T09:06:56.990 INFO:teuthology.orchestra.run.smithi088.stderr:2024-06-03T09:06:56.989+0000 7fc02dee2640 1 -- 172.21.15.88:0/3384061971 >> 172.21.15.88:0/3384061971 conn(0x7fc0280b70d0 msgr2=0x7fc02813db70 unknown :-1 s=STATE_NONE l=0).mark_down 2024-06-03T09:06:56.990 INFO:teuthology.orchestra.run.smithi088.stderr:2024-06-03T09:06:56.989+0000 7fc02dee2640 1 -- 172.21.15.88:0/3384061971 shutdown_connections 2024-06-03T09:06:56.990 INFO:teuthology.orchestra.run.smithi088.stderr:2024-06-03T09:06:56.989+0000 7fc02dee2640 1 -- 172.21.15.88:0/3384061971 wait complete. 2024-06-03T09:06:56.990 INFO:teuthology.orchestra.run.smithi088.stderr:2024-06-03T09:06:56.989+0000 7fc02dee2640 1 librados: shutdown 2024-06-03T09:06:57.001 INFO:tasks.ceph:Scrubbing osd.1 2024-06-03T09:06:57.001 DEBUG:teuthology.orchestra.run.smithi088:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph tell osd.1 config set osd_debug_deep_scrub_sleep 0 2024-06-03T09:06:57.102 INFO:teuthology.orchestra.run.smithi088.stderr:2024-06-03T09:06:57.102+0000 7f3f1a329640 10 monclient: get_monmap_and_config 2024-06-03T09:06:57.102 INFO:teuthology.orchestra.run.smithi088.stderr:2024-06-03T09:06:57.102+0000 7f3f1a329640 10 monclient: build_initial_monmap 2024-06-03T09:06:57.102 INFO:teuthology.orchestra.run.smithi088.stderr:2024-06-03T09:06:57.102+0000 7f3f1a329640 10 monclient: monmap: 2024-06-03T09:06:57.102 INFO:teuthology.orchestra.run.smithi088.stderr:epoch 0 2024-06-03T09:06:57.102 INFO:teuthology.orchestra.run.smithi088.stderr:fsid ec3c3fdd-3ac6-4d7d-9412-683130ed8b04 2024-06-03T09:06:57.102 INFO:teuthology.orchestra.run.smithi088.stderr:last_changed 2024-06-03T09:06:57.103295+0000 2024-06-03T09:06:57.103 INFO:teuthology.orchestra.run.smithi088.stderr:created 2024-06-03T09:06:57.103295+0000 2024-06-03T09:06:57.103 INFO:teuthology.orchestra.run.smithi088.stderr:min_mon_release 0 (unknown) 2024-06-03T09:06:57.103 INFO:teuthology.orchestra.run.smithi088.stderr:election_strategy: 3 2024-06-03T09:06:57.103 INFO:teuthology.orchestra.run.smithi088.stderr:0: v1:172.21.15.88:6789/0 mon.noname-a 2024-06-03T09:06:57.103 INFO:teuthology.orchestra.run.smithi088.stderr:1: v1:172.21.15.192:6789/0 mon.noname-c 2024-06-03T09:06:57.103 INFO:teuthology.orchestra.run.smithi088.stderr:2: v1:172.21.15.88:6790/0 mon.noname-b 2024-06-03T09:06:57.103 INFO:teuthology.orchestra.run.smithi088.stderr: 2024-06-03T09:06:57.104 INFO:teuthology.orchestra.run.smithi088.stderr:2024-06-03T09:06:57.103+0000 7f3f1a329640 1 Processor -- start 2024-06-03T09:06:57.104 INFO:teuthology.orchestra.run.smithi088.stderr:2024-06-03T09:06:57.103+0000 7f3f1a329640 1 -- start start 2024-06-03T09:06:57.104 INFO:teuthology.orchestra.run.smithi088.stderr:2024-06-03T09:06:57.103+0000 7f3f1a329640 10 monclient: init 2024-06-03T09:06:57.104 INFO:teuthology.orchestra.run.smithi088.stderr:2024-06-03T09:06:57.103+0000 7f3f1a329640 10 monclient: _reopen_session rank -1 2024-06-03T09:06:57.105 INFO:teuthology.orchestra.run.smithi088.stderr:2024-06-03T09:06:57.103+0000 7f3f1a329640 10 monclient: _add_conns ranks=[2,1,0] 2024-06-03T09:06:57.105 INFO:teuthology.orchestra.run.smithi088.stderr:2024-06-03T09:06:57.103+0000 7f3f1a329640 10 monclient(hunting): picked mon.noname-b con 0x7f3f140a8a60 addr v1:172.21.15.88:6790/0 2024-06-03T09:06:57.105 INFO:teuthology.orchestra.run.smithi088.stderr:2024-06-03T09:06:57.103+0000 7f3f1a329640 10 monclient(hunting): picked mon.noname-c con 0x7f3f140b1b70 addr v1:172.21.15.192:6789/0 2024-06-03T09:06:57.105 INFO:teuthology.orchestra.run.smithi088.stderr:2024-06-03T09:06:57.103+0000 7f3f1a329640 10 monclient(hunting): picked mon.noname-a con 0x7f3f1414ae00 addr v1:172.21.15.88:6789/0 2024-06-03T09:06:57.105 INFO:teuthology.orchestra.run.smithi088.stderr:2024-06-03T09:06:57.103+0000 7f3f1a329640 1 -- --> v1:172.21.15.88:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f3f140b18c0 con 0x7f3f1414ae00 2024-06-03T09:06:57.105 INFO:teuthology.orchestra.run.smithi088.stderr:2024-06-03T09:06:57.103+0000 7f3f1a329640 1 -- --> v1:172.21.15.192:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f3f140a92a0 con 0x7f3f140b1b70 2024-06-03T09:06:57.105 INFO:teuthology.orchestra.run.smithi088.stderr:2024-06-03T09:06:57.103+0000 7f3f1a329640 1 -- --> v1:172.21.15.88:6790/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f3f140b1fb0 con 0x7f3f140a8a60 2024-06-03T09:06:57.105 INFO:teuthology.orchestra.run.smithi088.stderr:2024-06-03T09:06:57.103+0000 7f3f1a329640 10 monclient(hunting): _renew_subs 2024-06-03T09:06:57.105 INFO:teuthology.orchestra.run.smithi088.stderr:2024-06-03T09:06:57.103+0000 7f3f1a329640 10 monclient(hunting): authenticate will time out at 3072.913742s 2024-06-03T09:06:57.105 INFO:teuthology.orchestra.run.smithi088.stderr:2024-06-03T09:06:57.104+0000 7f3f13fff640 1 --1- >> v1:172.21.15.88:6790/0 conn(0x7f3f140a8a60 0x7f3f14141d70 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.88:6790/0 says I am v1:172.21.15.88:50976/0 (socket says 172.21.15.88:50976) 2024-06-03T09:06:57.105 INFO:teuthology.orchestra.run.smithi088.stderr:2024-06-03T09:06:57.104+0000 7f3f13fff640 1 -- 172.21.15.88:0/3333167593 learned_addr learned my addr 172.21.15.88:0/3333167593 (peer_addr_for_me v1:172.21.15.88:0/0) 2024-06-03T09:06:57.106 INFO:teuthology.orchestra.run.smithi088.stderr:2024-06-03T09:06:57.106+0000 7f3f12ffd640 1 -- 172.21.15.88:0/3333167593 <== mon.2 v1:172.21.15.88:6790/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 6795795 0 0) 0x7f3f140b1fb0 con 0x7f3f140a8a60 2024-06-03T09:06:57.106 INFO:teuthology.orchestra.run.smithi088.stderr:2024-06-03T09:06:57.106+0000 7f3f12ffd640 10 monclient(hunting): _negotiate 2024-06-03T09:06:57.106 INFO:teuthology.orchestra.run.smithi088.stderr:2024-06-03T09:06:57.106+0000 7f3f12ffd640 10 monclient(hunting): _init_auth method 2 2024-06-03T09:06:57.106 INFO:teuthology.orchestra.run.smithi088.stderr:2024-06-03T09:06:57.106+0000 7f3f12ffd640 10 monclient(hunting): _init_auth creating new auth 2024-06-03T09:06:57.106 INFO:teuthology.orchestra.run.smithi088.stderr:2024-06-03T09:06:57.106+0000 7f3f12ffd640 10 monclient(hunting): my global_id is 30254 2024-06-03T09:06:57.107 INFO:teuthology.orchestra.run.smithi088.stderr:2024-06-03T09:06:57.106+0000 7f3f12ffd640 1 -- 172.21.15.88:0/3333167593 --> v1:172.21.15.88:6790/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f3f00003520 con 0x7f3f140a8a60 2024-06-03T09:06:57.107 INFO:teuthology.orchestra.run.smithi088.stderr:2024-06-03T09:06:57.106+0000 7f3f12ffd640 1 -- 172.21.15.88:0/3333167593 <== mon.0 v1:172.21.15.88:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 3256490695 0 0) 0x7f3f140b18c0 con 0x7f3f1414ae00 2024-06-03T09:06:57.107 INFO:teuthology.orchestra.run.smithi088.stderr:2024-06-03T09:06:57.106+0000 7f3f12ffd640 10 monclient(hunting): _negotiate 2024-06-03T09:06:57.107 INFO:teuthology.orchestra.run.smithi088.stderr:2024-06-03T09:06:57.106+0000 7f3f12ffd640 10 monclient(hunting): _init_auth method 2 2024-06-03T09:06:57.107 INFO:teuthology.orchestra.run.smithi088.stderr:2024-06-03T09:06:57.106+0000 7f3f12ffd640 10 monclient(hunting): _init_auth creating new auth 2024-06-03T09:06:57.107 INFO:teuthology.orchestra.run.smithi088.stderr:2024-06-03T09:06:57.106+0000 7f3f12ffd640 10 monclient(hunting): my global_id is 30750 2024-06-03T09:06:57.107 INFO:teuthology.orchestra.run.smithi088.stderr:2024-06-03T09:06:57.106+0000 7f3f12ffd640 1 -- 172.21.15.88:0/3333167593 --> v1:172.21.15.88:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f3f140b1fb0 con 0x7f3f1414ae00 2024-06-03T09:06:57.107 INFO:teuthology.orchestra.run.smithi088.stderr:2024-06-03T09:06:57.107+0000 7f3f12ffd640 1 -- 172.21.15.88:0/3333167593 <== mon.1 v1:172.21.15.192:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 2273317864 0 0) 0x7f3f140a92a0 con 0x7f3f140b1b70 2024-06-03T09:06:57.107 INFO:teuthology.orchestra.run.smithi088.stderr:2024-06-03T09:06:57.107+0000 7f3f12ffd640 10 monclient(hunting): _negotiate 2024-06-03T09:06:57.108 INFO:teuthology.orchestra.run.smithi088.stderr:2024-06-03T09:06:57.107+0000 7f3f12ffd640 10 monclient(hunting): _init_auth method 2 2024-06-03T09:06:57.108 INFO:teuthology.orchestra.run.smithi088.stderr:2024-06-03T09:06:57.107+0000 7f3f12ffd640 10 monclient(hunting): _init_auth creating new auth 2024-06-03T09:06:57.108 INFO:teuthology.orchestra.run.smithi088.stderr:2024-06-03T09:06:57.107+0000 7f3f12ffd640 10 monclient(hunting): my global_id is 30319 2024-06-03T09:06:57.108 INFO:teuthology.orchestra.run.smithi088.stderr:2024-06-03T09:06:57.107+0000 7f3f12ffd640 1 -- 172.21.15.88:0/3333167593 --> v1:172.21.15.192:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f3f140b18c0 con 0x7f3f140b1b70 2024-06-03T09:06:57.109 INFO:teuthology.orchestra.run.smithi088.stderr:2024-06-03T09:06:57.109+0000 7f3f12ffd640 1 -- 172.21.15.88:0/3333167593 <== mon.2 v1:172.21.15.88:6790/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 214+0+0 (unknown 2401980214 0 0) 0x7f3f00003520 con 0x7f3f140a8a60 2024-06-03T09:06:57.110 INFO:teuthology.orchestra.run.smithi088.stderr:2024-06-03T09:06:57.109+0000 7f3f12ffd640 1 -- 172.21.15.88:0/3333167593 >> v1:172.21.15.192:6789/0 conn(0x7f3f140b1b70 legacy=0x7f3f1413b140 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-06-03T09:06:57.110 INFO:teuthology.orchestra.run.smithi088.stderr:2024-06-03T09:06:57.109+0000 7f3f12ffd640 1 -- 172.21.15.88:0/3333167593 >> v1:172.21.15.88:6789/0 conn(0x7f3f1414ae00 legacy=0x7f3f1414d1f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-06-03T09:06:57.110 INFO:teuthology.orchestra.run.smithi088.stderr:2024-06-03T09:06:57.109+0000 7f3f12ffd640 10 monclient: _finish_hunting 0 2024-06-03T09:06:57.110 INFO:teuthology.orchestra.run.smithi088.stderr:2024-06-03T09:06:57.109+0000 7f3f12ffd640 1 monclient: found mon.noname-b 2024-06-03T09:06:57.110 INFO:teuthology.orchestra.run.smithi088.stderr:2024-06-03T09:06:57.109+0000 7f3f12ffd640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-06-03T09:06:57.110 INFO:teuthology.orchestra.run.smithi088.stderr:2024-06-03T09:06:57.109+0000 7f3f12ffd640 10 monclient: _send_mon_message to mon.noname-b at v1:172.21.15.88:6790/0 2024-06-03T09:06:57.111 INFO:teuthology.orchestra.run.smithi088.stderr:2024-06-03T09:06:57.109+0000 7f3f12ffd640 1 -- 172.21.15.88:0/3333167593 --> v1:172.21.15.88:6790/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f3f140aa070 con 0x7f3f140a8a60 2024-06-03T09:06:57.111 INFO:teuthology.orchestra.run.smithi088.stderr:2024-06-03T09:06:57.109+0000 7f3f12ffd640 10 monclient: _finish_auth 0 2024-06-03T09:06:57.111 INFO:teuthology.orchestra.run.smithi088.stderr:2024-06-03T09:06:57.109+0000 7f3f12ffd640 10 monclient: _check_auth_tickets 2024-06-03T09:06:57.111 INFO:teuthology.orchestra.run.smithi088.stderr:2024-06-03T09:06:57.109+0000 7f3f12ffd640 20 monclient: _check_auth_rotating not needed by client.admin 2024-06-03T09:06:57.111 INFO:teuthology.orchestra.run.smithi088.stderr:2024-06-03T09:06:57.109+0000 7f3f12ffd640 1 -- 172.21.15.88:0/3333167593 <== mon.2 v1:172.21.15.88:6790/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 2234374065 0 0) 0x7f3f04001610 con 0x7f3f140a8a60 2024-06-03T09:06:57.111 INFO:teuthology.orchestra.run.smithi088.stderr:2024-06-03T09:06:57.109+0000 7f3f12ffd640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-06-03T09:06:57.111 INFO:teuthology.orchestra.run.smithi088.stderr:2024-06-03T09:06:57.109+0000 7f3f12ffd640 10 monclient: got monmap 1 from mon.noname-b (according to old e1) 2024-06-03T09:06:57.111 INFO:teuthology.orchestra.run.smithi088.stderr:2024-06-03T09:06:57.109+0000 7f3f12ffd640 10 monclient: dump: 2024-06-03T09:06:57.111 INFO:teuthology.orchestra.run.smithi088.stderr:epoch 1 2024-06-03T09:06:57.111 INFO:teuthology.orchestra.run.smithi088.stderr:fsid ec3c3fdd-3ac6-4d7d-9412-683130ed8b04 2024-06-03T09:06:57.111 INFO:teuthology.orchestra.run.smithi088.stderr:last_changed 2024-06-03T08:34:31.885125+0000 2024-06-03T09:06:57.111 INFO:teuthology.orchestra.run.smithi088.stderr:created 2024-06-03T08:34:31.885125+0000 2024-06-03T09:06:57.111 INFO:teuthology.orchestra.run.smithi088.stderr:min_mon_release 18 (reef) 2024-06-03T09:06:57.112 INFO:teuthology.orchestra.run.smithi088.stderr:election_strategy: 3 2024-06-03T09:06:57.112 INFO:teuthology.orchestra.run.smithi088.stderr:0: v1:172.21.15.88:6789/0 mon.a 2024-06-03T09:06:57.112 INFO:teuthology.orchestra.run.smithi088.stderr:1: v1:172.21.15.192:6789/0 mon.b 2024-06-03T09:06:57.112 INFO:teuthology.orchestra.run.smithi088.stderr:2: v1:172.21.15.88:6790/0 mon.c 2024-06-03T09:06:57.112 INFO:teuthology.orchestra.run.smithi088.stderr: 2024-06-03T09:06:57.112 INFO:teuthology.orchestra.run.smithi088.stderr:2024-06-03T09:06:57.109+0000 7f3f1a329640 5 monclient: authenticate success, global_id 30254 2024-06-03T09:06:57.112 INFO:teuthology.orchestra.run.smithi088.stderr:2024-06-03T09:06:57.109+0000 7f3f1a329640 20 monclient: get_monmap_and_config waiting for monmap|config 2024-06-03T09:06:57.114 INFO:teuthology.orchestra.run.smithi088.stderr:2024-06-03T09:06:57.111+0000 7f3f12ffd640 1 -- 172.21.15.88:0/3333167593 <== mon.2 v1:172.21.15.88:6790/0 4 ==== config(0 keys) v1 ==== 4+0+0 (unknown 0 0 0) 0x7f3f04001830 con 0x7f3f140a8a60 2024-06-03T09:06:57.114 INFO:teuthology.orchestra.run.smithi088.stderr:2024-06-03T09:06:57.111+0000 7f3f12ffd640 10 monclient: handle_config config(0 keys) v1 2024-06-03T09:06:57.115 INFO:teuthology.orchestra.run.smithi088.stderr:2024-06-03T09:06:57.111+0000 7f3f12ffd640 1 -- 172.21.15.88:0/3333167593 <== mon.2 v1:172.21.15.88:6790/0 5 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 2234374065 0 0) 0x7f3f04001b40 con 0x7f3f140a8a60 2024-06-03T09:06:57.115 INFO:teuthology.orchestra.run.smithi088.stderr:2024-06-03T09:06:57.111+0000 7f3f1a329640 10 monclient: get_monmap_and_config success 2024-06-03T09:06:57.115 INFO:teuthology.orchestra.run.smithi088.stderr:2024-06-03T09:06:57.111+0000 7f3f1a329640 4 set_mon_vals no callback set 2024-06-03T09:06:57.115 INFO:teuthology.orchestra.run.smithi088.stderr:2024-06-03T09:06:57.111+0000 7f3f12ffd640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-06-03T09:06:57.115 INFO:teuthology.orchestra.run.smithi088.stderr:2024-06-03T09:06:57.111+0000 7f3f12ffd640 10 monclient: got monmap 1 from mon.c (according to old e1) 2024-06-03T09:06:57.115 INFO:teuthology.orchestra.run.smithi088.stderr:2024-06-03T09:06:57.111+0000 7f3f12ffd640 10 monclient: dump: 2024-06-03T09:06:57.115 INFO:teuthology.orchestra.run.smithi088.stderr:epoch 1 2024-06-03T09:06:57.115 INFO:teuthology.orchestra.run.smithi088.stderr:fsid ec3c3fdd-3ac6-4d7d-9412-683130ed8b04 2024-06-03T09:06:57.115 INFO:teuthology.orchestra.run.smithi088.stderr:last_changed 2024-06-03T08:34:31.885125+0000 2024-06-03T09:06:57.115 INFO:teuthology.orchestra.run.smithi088.stderr:created 2024-06-03T08:34:31.885125+0000 2024-06-03T09:06:57.115 INFO:teuthology.orchestra.run.smithi088.stderr:min_mon_release 18 (reef) 2024-06-03T09:06:57.115 INFO:teuthology.orchestra.run.smithi088.stderr:election_strategy: 3 2024-06-03T09:06:57.115 INFO:teuthology.orchestra.run.smithi088.stderr:0: v1:172.21.15.88:6789/0 mon.a 2024-06-03T09:06:57.116 INFO:teuthology.orchestra.run.smithi088.stderr:1: v1:172.21.15.192:6789/0 mon.b 2024-06-03T09:06:57.116 INFO:teuthology.orchestra.run.smithi088.stderr:2: v1:172.21.15.88:6790/0 mon.c 2024-06-03T09:06:57.116 INFO:teuthology.orchestra.run.smithi088.stderr: 2024-06-03T09:06:57.116 INFO:teuthology.orchestra.run.smithi088.stderr:2024-06-03T09:06:57.112+0000 7f3f1a329640 10 monclient: shutdown 2024-06-03T09:06:57.116 INFO:teuthology.orchestra.run.smithi088.stderr:2024-06-03T09:06:57.112+0000 7f3f1a329640 20 monclient: shutdown discarding 0 pending message(s) 2024-06-03T09:06:57.116 INFO:teuthology.orchestra.run.smithi088.stderr:2024-06-03T09:06:57.112+0000 7f3f1a329640 1 -- 172.21.15.88:0/3333167593 >> v1:172.21.15.88:6790/0 conn(0x7f3f140a8a60 legacy=0x7f3f14141d70 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-06-03T09:06:57.116 INFO:teuthology.orchestra.run.smithi088.stderr:2024-06-03T09:06:57.112+0000 7f3f1a329640 1 -- 172.21.15.88:0/3333167593 shutdown_connections 2024-06-03T09:06:57.116 INFO:teuthology.orchestra.run.smithi088.stderr:2024-06-03T09:06:57.112+0000 7f3f1a329640 1 -- 172.21.15.88:0/3333167593 >> 172.21.15.88:0/3333167593 conn(0x7f3f140b7170 msgr2=0x7f3f140b7580 unknown :-1 s=STATE_NONE l=0).mark_down 2024-06-03T09:06:57.116 INFO:teuthology.orchestra.run.smithi088.stderr:2024-06-03T09:06:57.112+0000 7f3f1a329640 1 -- 172.21.15.88:0/3333167593 shutdown_connections 2024-06-03T09:06:57.116 INFO:teuthology.orchestra.run.smithi088.stderr:2024-06-03T09:06:57.112+0000 7f3f1a329640 1 -- 172.21.15.88:0/3333167593 wait complete. 2024-06-03T09:06:57.116 INFO:teuthology.orchestra.run.smithi088.stderr:2024-06-03T09:06:57.112+0000 7f3f1a329640 10 monclient: build_initial_monmap 2024-06-03T09:06:57.116 INFO:teuthology.orchestra.run.smithi088.stderr:2024-06-03T09:06:57.112+0000 7f3f1a329640 10 monclient: monmap: 2024-06-03T09:06:57.116 INFO:teuthology.orchestra.run.smithi088.stderr:epoch 0 2024-06-03T09:06:57.116 INFO:teuthology.orchestra.run.smithi088.stderr:fsid 00000000-0000-0000-0000-000000000000 2024-06-03T09:06:57.116 INFO:teuthology.orchestra.run.smithi088.stderr:last_changed 0.000000 2024-06-03T09:06:57.117 INFO:teuthology.orchestra.run.smithi088.stderr:created 0.000000 2024-06-03T09:06:57.117 INFO:teuthology.orchestra.run.smithi088.stderr:min_mon_release 0 (unknown) 2024-06-03T09:06:57.117 INFO:teuthology.orchestra.run.smithi088.stderr:election_strategy: 1 2024-06-03T09:06:57.117 INFO:teuthology.orchestra.run.smithi088.stderr:0: v1:172.21.15.88:6789/0 mon.noname-a 2024-06-03T09:06:57.117 INFO:teuthology.orchestra.run.smithi088.stderr:1: v1:172.21.15.192:6789/0 mon.noname-b 2024-06-03T09:06:57.117 INFO:teuthology.orchestra.run.smithi088.stderr:2: v1:172.21.15.88:6790/0 mon.noname-c 2024-06-03T09:06:57.117 INFO:teuthology.orchestra.run.smithi088.stderr: 2024-06-03T09:06:57.117 INFO:teuthology.orchestra.run.smithi088.stderr:2024-06-03T09:06:57.113+0000 7f3f1a329640 1 librados: starting msgr at 2024-06-03T09:06:57.117 INFO:teuthology.orchestra.run.smithi088.stderr:2024-06-03T09:06:57.113+0000 7f3f1a329640 1 librados: starting objecter 2024-06-03T09:06:57.117 INFO:teuthology.orchestra.run.smithi088.stderr:2024-06-03T09:06:57.113+0000 7f3f1a329640 1 Processor -- start 2024-06-03T09:06:57.117 INFO:teuthology.orchestra.run.smithi088.stderr:2024-06-03T09:06:57.113+0000 7f3f1a329640 1 -- start start 2024-06-03T09:06:57.117 INFO:teuthology.orchestra.run.smithi088.stderr:2024-06-03T09:06:57.113+0000 7f3f1a329640 1 librados: setting wanted keys 2024-06-03T09:06:57.117 INFO:teuthology.orchestra.run.smithi088.stderr:2024-06-03T09:06:57.113+0000 7f3f1a329640 1 librados: calling monclient init 2024-06-03T09:06:57.117 INFO:teuthology.orchestra.run.smithi088.stderr:2024-06-03T09:06:57.113+0000 7f3f1a329640 10 monclient: init 2024-06-03T09:06:57.118 INFO:teuthology.orchestra.run.smithi088.stderr:2024-06-03T09:06:57.114+0000 7f3f1a329640 10 monclient: _reopen_session rank -1 2024-06-03T09:06:57.118 INFO:teuthology.orchestra.run.smithi088.stderr:2024-06-03T09:06:57.114+0000 7f3f1a329640 10 monclient: _add_conns ranks=[1,0,2] 2024-06-03T09:06:57.118 INFO:teuthology.orchestra.run.smithi088.stderr:2024-06-03T09:06:57.114+0000 7f3f1a329640 10 monclient(hunting): picked mon.noname-b con 0x7f3f140a8a60 addr v1:172.21.15.192:6789/0 2024-06-03T09:06:57.118 INFO:teuthology.orchestra.run.smithi088.stderr:2024-06-03T09:06:57.114+0000 7f3f1a329640 10 monclient(hunting): picked mon.noname-a con 0x7f3f140b1b70 addr v1:172.21.15.88:6789/0 2024-06-03T09:06:57.118 INFO:teuthology.orchestra.run.smithi088.stderr:2024-06-03T09:06:57.114+0000 7f3f1a329640 10 monclient(hunting): picked mon.noname-c con 0x7f3f1414ae00 addr v1:172.21.15.88:6790/0 2024-06-03T09:06:57.118 INFO:teuthology.orchestra.run.smithi088.stderr:2024-06-03T09:06:57.114+0000 7f3f1a329640 1 -- --> v1:172.21.15.88:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f3f141370e0 con 0x7f3f140b1b70 2024-06-03T09:06:57.118 INFO:teuthology.orchestra.run.smithi088.stderr:2024-06-03T09:06:57.114+0000 7f3f1a329640 1 -- --> v1:172.21.15.192:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f3f141431f0 con 0x7f3f140a8a60 2024-06-03T09:06:57.118 INFO:teuthology.orchestra.run.smithi088.stderr:2024-06-03T09:06:57.114+0000 7f3f1a329640 1 -- --> v1:172.21.15.88:6790/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f3f14143380 con 0x7f3f1414ae00 2024-06-03T09:06:57.118 INFO:teuthology.orchestra.run.smithi088.stderr:2024-06-03T09:06:57.114+0000 7f3f1a329640 10 monclient(hunting): _renew_subs 2024-06-03T09:06:57.118 INFO:teuthology.orchestra.run.smithi088.stderr:2024-06-03T09:06:57.114+0000 7f3f1a329640 10 monclient(hunting): authenticate will time out at 3072.924119s 2024-06-03T09:06:57.118 INFO:teuthology.orchestra.run.smithi088.stderr:2024-06-03T09:06:57.114+0000 7f3f188c6640 1 --1- >> v1:172.21.15.88:6790/0 conn(0x7f3f1414ae00 0x7f3f14144e60 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=1).handle_server_banner_and_identify peer v1:172.21.15.88:6790/0 says I am v1:172.21.15.88:50978/0 (socket says 172.21.15.88:50978) 2024-06-03T09:06:57.118 INFO:teuthology.orchestra.run.smithi088.stderr:2024-06-03T09:06:57.114+0000 7f3f188c6640 1 -- 172.21.15.88:0/4233859610 learned_addr learned my addr 172.21.15.88:0/4233859610 (peer_addr_for_me v1:172.21.15.88:0/0) 2024-06-03T09:06:57.118 INFO:teuthology.orchestra.run.smithi088.stderr:2024-06-03T09:06:57.114+0000 7f3f10ff9640 10 client.?.objecter ms_handle_connect 0x7f3f1414ae00 2024-06-03T09:06:57.119 INFO:teuthology.orchestra.run.smithi088.stderr:2024-06-03T09:06:57.114+0000 7f3f10ff9640 10 client.?.objecter resend_mon_ops 2024-06-03T09:06:57.119 INFO:teuthology.orchestra.run.smithi088.stderr:2024-06-03T09:06:57.115+0000 7f3f10ff9640 10 client.?.objecter ms_handle_connect 0x7f3f140a8a60 2024-06-03T09:06:57.119 INFO:teuthology.orchestra.run.smithi088.stderr:2024-06-03T09:06:57.115+0000 7f3f10ff9640 10 client.?.objecter resend_mon_ops 2024-06-03T09:06:57.119 INFO:teuthology.orchestra.run.smithi088.stderr:2024-06-03T09:06:57.116+0000 7f3f10ff9640 1 -- 172.21.15.88:0/4233859610 <== mon.2 v1:172.21.15.88:6790/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 227641039 0 0) 0x7f3f14143380 con 0x7f3f1414ae00 2024-06-03T09:06:57.119 INFO:teuthology.orchestra.run.smithi088.stderr:2024-06-03T09:06:57.116+0000 7f3f10ff9640 10 monclient(hunting): _negotiate 2024-06-03T09:06:57.120 INFO:teuthology.orchestra.run.smithi088.stderr:2024-06-03T09:06:57.116+0000 7f3f10ff9640 10 monclient(hunting): _init_auth method 2 2024-06-03T09:06:57.120 INFO:teuthology.orchestra.run.smithi088.stderr:2024-06-03T09:06:57.116+0000 7f3f10ff9640 10 monclient(hunting): _init_auth creating new auth 2024-06-03T09:06:57.120 INFO:teuthology.orchestra.run.smithi088.stderr:2024-06-03T09:06:57.116+0000 7f3f10ff9640 10 monclient(hunting): my global_id is 30257 2024-06-03T09:06:57.120 INFO:teuthology.orchestra.run.smithi088.stderr:2024-06-03T09:06:57.116+0000 7f3f10ff9640 1 -- 172.21.15.88:0/4233859610 --> v1:172.21.15.88:6790/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f3ef4004eb0 con 0x7f3f1414ae00 2024-06-03T09:06:57.120 INFO:teuthology.orchestra.run.smithi088.stderr:2024-06-03T09:06:57.117+0000 7f3f10ff9640 1 -- 172.21.15.88:0/4233859610 <== mon.1 v1:172.21.15.192:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 3154738428 0 0) 0x7f3f141431f0 con 0x7f3f140a8a60 2024-06-03T09:06:57.120 INFO:teuthology.orchestra.run.smithi088.stderr:2024-06-03T09:06:57.117+0000 7f3f10ff9640 10 monclient(hunting): _negotiate 2024-06-03T09:06:57.120 INFO:teuthology.orchestra.run.smithi088.stderr:2024-06-03T09:06:57.117+0000 7f3f10ff9640 10 monclient(hunting): _init_auth method 2 2024-06-03T09:06:57.120 INFO:teuthology.orchestra.run.smithi088.stderr:2024-06-03T09:06:57.117+0000 7f3f10ff9640 10 monclient(hunting): _init_auth creating new auth 2024-06-03T09:06:57.120 INFO:teuthology.orchestra.run.smithi088.stderr:2024-06-03T09:06:57.117+0000 7f3f10ff9640 10 monclient(hunting): my global_id is 30322 2024-06-03T09:06:57.120 INFO:teuthology.orchestra.run.smithi088.stderr:2024-06-03T09:06:57.117+0000 7f3f10ff9640 1 -- 172.21.15.88:0/4233859610 --> v1:172.21.15.192:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f3f14143380 con 0x7f3f140a8a60 2024-06-03T09:06:57.121 INFO:teuthology.orchestra.run.smithi088.stderr:2024-06-03T09:06:57.121+0000 7f3f10ff9640 1 -- 172.21.15.88:0/4233859610 <== mon.2 v1:172.21.15.88:6790/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 764+0+0 (unknown 2655222684 0 0) 0x7f3ef4004eb0 con 0x7f3f1414ae00 2024-06-03T09:06:57.122 INFO:teuthology.orchestra.run.smithi088.stderr:2024-06-03T09:06:57.121+0000 7f3f10ff9640 1 -- 172.21.15.88:0/4233859610 --> v1:172.21.15.88:6790/0 -- auth(proto 2 165 bytes epoch 0) v1 -- 0x7f3f141431f0 con 0x7f3f1414ae00 2024-06-03T09:06:57.122 INFO:teuthology.orchestra.run.smithi088.stderr:2024-06-03T09:06:57.121+0000 7f3f10ff9640 1 -- 172.21.15.88:0/4233859610 <== mon.2 v1:172.21.15.88:6790/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 2234374065 0 0) 0x7f3f0c001b90 con 0x7f3f1414ae00 2024-06-03T09:06:57.122 INFO:teuthology.orchestra.run.smithi088.stderr:2024-06-03T09:06:57.121+0000 7f3f10ff9640 10 monclient(hunting): handle_monmap mon_map magic: 0 v1 2024-06-03T09:06:57.122 INFO:teuthology.orchestra.run.smithi088.stderr:2024-06-03T09:06:57.121+0000 7f3f10ff9640 10 monclient(hunting): got monmap 1 from mon.noname-c (according to old e1) 2024-06-03T09:06:57.122 INFO:teuthology.orchestra.run.smithi088.stderr:2024-06-03T09:06:57.121+0000 7f3f10ff9640 10 monclient(hunting): dump: 2024-06-03T09:06:57.122 INFO:teuthology.orchestra.run.smithi088.stderr:epoch 1 2024-06-03T09:06:57.122 INFO:teuthology.orchestra.run.smithi088.stderr:fsid ec3c3fdd-3ac6-4d7d-9412-683130ed8b04 2024-06-03T09:06:57.122 INFO:teuthology.orchestra.run.smithi088.stderr:last_changed 2024-06-03T08:34:31.885125+0000 2024-06-03T09:06:57.122 INFO:teuthology.orchestra.run.smithi088.stderr:created 2024-06-03T08:34:31.885125+0000 2024-06-03T09:06:57.122 INFO:teuthology.orchestra.run.smithi088.stderr:min_mon_release 18 (reef) 2024-06-03T09:06:57.122 INFO:teuthology.orchestra.run.smithi088.stderr:election_strategy: 3 2024-06-03T09:06:57.122 INFO:teuthology.orchestra.run.smithi088.stderr:0: v1:172.21.15.88:6789/0 mon.a 2024-06-03T09:06:57.123 INFO:teuthology.orchestra.run.smithi088.stderr:1: v1:172.21.15.192:6789/0 mon.b 2024-06-03T09:06:57.123 INFO:teuthology.orchestra.run.smithi088.stderr:2: v1:172.21.15.88:6790/0 mon.c 2024-06-03T09:06:57.123 INFO:teuthology.orchestra.run.smithi088.stderr: 2024-06-03T09:06:57.123 INFO:teuthology.orchestra.run.smithi088.stderr:2024-06-03T09:06:57.121+0000 7f3f10ff9640 10 monclient(hunting): _finish_auth 0 2024-06-03T09:06:57.123 INFO:teuthology.orchestra.run.smithi088.stderr:2024-06-03T09:06:57.121+0000 7f3f10ff9640 1 -- 172.21.15.88:0/4233859610 <== mon.1 v1:172.21.15.192:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 764+0+0 (unknown 1066809523 0 0) 0x7f3f14143380 con 0x7f3f140a8a60 2024-06-03T09:06:57.123 INFO:teuthology.orchestra.run.smithi088.stderr:2024-06-03T09:06:57.121+0000 7f3f10ff9640 1 -- 172.21.15.88:0/4233859610 --> v1:172.21.15.192:6789/0 -- auth(proto 2 165 bytes epoch 0) v1 -- 0x7f3ef4004eb0 con 0x7f3f140a8a60 2024-06-03T09:06:57.123 INFO:teuthology.orchestra.run.smithi088.stderr:2024-06-03T09:06:57.121+0000 7f3f10ff9640 1 -- 172.21.15.88:0/4233859610 <== mon.1 v1:172.21.15.192:6789/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 2234374065 0 0) 0x7f3f040010d0 con 0x7f3f140a8a60 2024-06-03T09:06:57.123 INFO:teuthology.orchestra.run.smithi088.stderr:2024-06-03T09:06:57.121+0000 7f3f10ff9640 10 monclient(hunting): handle_monmap mon_map magic: 0 v1 2024-06-03T09:06:57.123 INFO:teuthology.orchestra.run.smithi088.stderr:2024-06-03T09:06:57.121+0000 7f3f10ff9640 10 monclient(hunting): got monmap 1 from mon.b (according to old e1) 2024-06-03T09:06:57.123 INFO:teuthology.orchestra.run.smithi088.stderr:2024-06-03T09:06:57.121+0000 7f3f10ff9640 10 monclient(hunting): dump: 2024-06-03T09:06:57.123 INFO:teuthology.orchestra.run.smithi088.stderr:epoch 1 2024-06-03T09:06:57.123 INFO:teuthology.orchestra.run.smithi088.stderr:fsid ec3c3fdd-3ac6-4d7d-9412-683130ed8b04 2024-06-03T09:06:57.123 INFO:teuthology.orchestra.run.smithi088.stderr:last_changed 2024-06-03T08:34:31.885125+0000 2024-06-03T09:06:57.123 INFO:teuthology.orchestra.run.smithi088.stderr:created 2024-06-03T08:34:31.885125+0000 2024-06-03T09:06:57.124 INFO:teuthology.orchestra.run.smithi088.stderr:min_mon_release 18 (reef) 2024-06-03T09:06:57.124 INFO:teuthology.orchestra.run.smithi088.stderr:election_strategy: 3 2024-06-03T09:06:57.124 INFO:teuthology.orchestra.run.smithi088.stderr:0: v1:172.21.15.88:6789/0 mon.a 2024-06-03T09:06:57.124 INFO:teuthology.orchestra.run.smithi088.stderr:1: v1:172.21.15.192:6789/0 mon.b 2024-06-03T09:06:57.124 INFO:teuthology.orchestra.run.smithi088.stderr:2: v1:172.21.15.88:6790/0 mon.c 2024-06-03T09:06:57.124 INFO:teuthology.orchestra.run.smithi088.stderr: 2024-06-03T09:06:57.125 INFO:teuthology.orchestra.run.smithi088.stderr:2024-06-03T09:06:57.124+0000 7f3f10ff9640 1 -- 172.21.15.88:0/4233859610 <== mon.1 v1:172.21.15.192:6789/0 4 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 580+0+0 (unknown 2174396778 0 0) 0x7f3ef4004eb0 con 0x7f3f140a8a60 2024-06-03T09:06:57.125 INFO:teuthology.orchestra.run.smithi088.stderr:2024-06-03T09:06:57.124+0000 7f3f10ff9640 1 -- 172.21.15.88:0/4233859610 >> v1:172.21.15.88:6790/0 conn(0x7f3f1414ae00 legacy=0x7f3f14144e60 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-06-03T09:06:57.125 INFO:teuthology.orchestra.run.smithi088.stderr:2024-06-03T09:06:57.124+0000 7f3f10ff9640 1 -- 172.21.15.88:0/4233859610 >> v1:172.21.15.88:6789/0 conn(0x7f3f140b1b70 legacy=0x7f3f141d6cb0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-06-03T09:06:57.125 INFO:teuthology.orchestra.run.smithi088.stderr:2024-06-03T09:06:57.124+0000 7f3f10ff9640 10 monclient: _finish_hunting 0 2024-06-03T09:06:57.125 INFO:teuthology.orchestra.run.smithi088.stderr:2024-06-03T09:06:57.124+0000 7f3f10ff9640 1 monclient: found mon.b 2024-06-03T09:06:57.126 INFO:teuthology.orchestra.run.smithi088.stderr:2024-06-03T09:06:57.124+0000 7f3f10ff9640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-06-03T09:06:57.126 INFO:teuthology.orchestra.run.smithi088.stderr:2024-06-03T09:06:57.124+0000 7f3f10ff9640 10 monclient: _send_mon_message to mon.b at v1:172.21.15.192:6789/0 2024-06-03T09:06:57.126 INFO:teuthology.orchestra.run.smithi088.stderr:2024-06-03T09:06:57.124+0000 7f3f10ff9640 1 -- 172.21.15.88:0/4233859610 --> v1:172.21.15.192:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f3f141d73c0 con 0x7f3f140a8a60 2024-06-03T09:06:57.126 INFO:teuthology.orchestra.run.smithi088.stderr:2024-06-03T09:06:57.124+0000 7f3f10ff9640 10 monclient: _finish_auth 0 2024-06-03T09:06:57.126 INFO:teuthology.orchestra.run.smithi088.stderr:2024-06-03T09:06:57.124+0000 7f3f10ff9640 10 monclient: _check_auth_tickets 2024-06-03T09:06:57.126 INFO:teuthology.orchestra.run.smithi088.stderr:2024-06-03T09:06:57.124+0000 7f3f10ff9640 20 monclient: _check_auth_rotating not needed by client.admin 2024-06-03T09:06:57.126 INFO:teuthology.orchestra.run.smithi088.stderr:2024-06-03T09:06:57.124+0000 7f3f1a329640 5 monclient: authenticate success, global_id 30322 2024-06-03T09:06:57.126 INFO:teuthology.orchestra.run.smithi088.stderr:2024-06-03T09:06:57.124+0000 7f3f1a329640 10 monclient: _renew_subs 2024-06-03T09:06:57.126 INFO:teuthology.orchestra.run.smithi088.stderr:2024-06-03T09:06:57.124+0000 7f3f1a329640 10 monclient: _send_mon_message to mon.b at v1:172.21.15.192:6789/0 2024-06-03T09:06:57.126 INFO:teuthology.orchestra.run.smithi088.stderr:2024-06-03T09:06:57.124+0000 7f3f1a329640 1 -- 172.21.15.88:0/4233859610 --> v1:172.21.15.192:6789/0 -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f3f141447b0 con 0x7f3f140a8a60 2024-06-03T09:06:57.126 INFO:teuthology.orchestra.run.smithi088.stderr:2024-06-03T09:06:57.125+0000 7f3f1a329640 10 client.30322.objecter _maybe_request_map subscribing (onetime) to next osd map 2024-06-03T09:06:57.126 INFO:teuthology.orchestra.run.smithi088.stderr:2024-06-03T09:06:57.125+0000 7f3f1a329640 10 monclient: _renew_subs 2024-06-03T09:06:57.126 INFO:teuthology.orchestra.run.smithi088.stderr:2024-06-03T09:06:57.125+0000 7f3f1a329640 10 monclient: _send_mon_message to mon.b at v1:172.21.15.192:6789/0 2024-06-03T09:06:57.127 INFO:teuthology.orchestra.run.smithi088.stderr:2024-06-03T09:06:57.125+0000 7f3f1a329640 1 -- 172.21.15.88:0/4233859610 --> v1:172.21.15.192:6789/0 -- mon_subscribe({osdmap=0}) v3 -- 0x7f3f14145750 con 0x7f3f140a8a60 2024-06-03T09:06:57.127 INFO:teuthology.orchestra.run.smithi088.stderr:2024-06-03T09:06:57.125+0000 7f3f1a329640 1 librados: init done 2024-06-03T09:06:57.127 INFO:teuthology.orchestra.run.smithi088.stderr:2024-06-03T09:06:57.125+0000 7f3f1a329640 10 client.30322.objecter _submit_command 1 [{"prefix": "get_command_descriptions"}] 2024-06-03T09:06:57.127 INFO:teuthology.orchestra.run.smithi088.stderr:2024-06-03T09:06:57.125+0000 7f3f1a329640 15 client.30322.objecter _session_command_op_assign -1 1 2024-06-03T09:06:57.127 INFO:teuthology.orchestra.run.smithi088.stderr:2024-06-03T09:06:57.125+0000 7f3f1a329640 20 client.30322.objecter _get_session osd=-1 returning homeless 2024-06-03T09:06:57.127 INFO:teuthology.orchestra.run.smithi088.stderr:2024-06-03T09:06:57.125+0000 7f3f1a329640 10 client.30322.objecter _maybe_request_map subscribing (onetime) to next osd map 2024-06-03T09:06:57.127 INFO:teuthology.orchestra.run.smithi088.stderr:2024-06-03T09:06:57.125+0000 7f3f1a329640 10 monclient: _send_mon_message to mon.b at v1:172.21.15.192:6789/0 2024-06-03T09:06:57.127 INFO:teuthology.orchestra.run.smithi088.stderr:2024-06-03T09:06:57.125+0000 7f3f1a329640 1 -- 172.21.15.88:0/4233859610 --> v1:172.21.15.192:6789/0 -- mon_get_version(what=osdmap handle=1) v1 -- 0x7f3f1413f030 con 0x7f3f140a8a60 2024-06-03T09:06:57.141 INFO:teuthology.orchestra.run.smithi088.stderr:2024-06-03T09:06:57.141+0000 7f3f10ff9640 1 -- 172.21.15.88:0/4233859610 <== mon.1 v1:172.21.15.192:6789/0 5 ==== config(0 keys) v1 ==== 4+0+0 (unknown 0 0 0) 0x7f3f04001650 con 0x7f3f140a8a60 2024-06-03T09:06:57.141 INFO:teuthology.orchestra.run.smithi088.stderr:2024-06-03T09:06:57.141+0000 7f3f10ff9640 10 monclient: handle_config config(0 keys) v1 2024-06-03T09:06:57.141 INFO:teuthology.orchestra.run.smithi088.stderr:2024-06-03T09:06:57.141+0000 7f3f10ff9640 1 -- 172.21.15.88:0/4233859610 <== mon.1 v1:172.21.15.192:6789/0 6 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 2234374065 0 0) 0x7f3f04004c30 con 0x7f3f140a8a60 2024-06-03T09:06:57.141 INFO:teuthology.orchestra.run.smithi088.stderr:2024-06-03T09:06:57.141+0000 7f3f10ff9640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-06-03T09:06:57.141 INFO:teuthology.orchestra.run.smithi088.stderr:2024-06-03T09:06:57.141+0000 7f3f10ff9640 10 monclient: got monmap 1 from mon.b (according to old e1) 2024-06-03T09:06:57.141 INFO:teuthology.orchestra.run.smithi088.stderr:2024-06-03T09:06:57.141+0000 7f3f10ff9640 10 monclient: dump: 2024-06-03T09:06:57.141 INFO:teuthology.orchestra.run.smithi088.stderr:epoch 1 2024-06-03T09:06:57.141 INFO:teuthology.orchestra.run.smithi088.stderr:fsid ec3c3fdd-3ac6-4d7d-9412-683130ed8b04 2024-06-03T09:06:57.141 INFO:teuthology.orchestra.run.smithi088.stderr:last_changed 2024-06-03T08:34:31.885125+0000 2024-06-03T09:06:57.141 INFO:teuthology.orchestra.run.smithi088.stderr:created 2024-06-03T08:34:31.885125+0000 2024-06-03T09:06:57.141 INFO:teuthology.orchestra.run.smithi088.stderr:min_mon_release 18 (reef) 2024-06-03T09:06:57.142 INFO:teuthology.orchestra.run.smithi088.stderr:election_strategy: 3 2024-06-03T09:06:57.142 INFO:teuthology.orchestra.run.smithi088.stderr:0: v1:172.21.15.88:6789/0 mon.a 2024-06-03T09:06:57.142 INFO:teuthology.orchestra.run.smithi088.stderr:1: v1:172.21.15.192:6789/0 mon.b 2024-06-03T09:06:57.142 INFO:teuthology.orchestra.run.smithi088.stderr:2: v1:172.21.15.88:6790/0 mon.c 2024-06-03T09:06:57.142 INFO:teuthology.orchestra.run.smithi088.stderr: 2024-06-03T09:06:57.142 INFO:teuthology.orchestra.run.smithi088.stderr:2024-06-03T09:06:57.141+0000 7f3f127fc640 4 set_mon_vals no callback set 2024-06-03T09:06:57.143 INFO:teuthology.orchestra.run.smithi088.stderr:2024-06-03T09:06:57.141+0000 7f3f10ff9640 1 -- 172.21.15.88:0/4233859610 <== mon.1 v1:172.21.15.192:6789/0 7 ==== mgrmap(e 5) v1 ==== 89110+0+0 (unknown 873533973 0 0) 0x7f3f0401a9b0 con 0x7f3f140a8a60 2024-06-03T09:06:57.143 INFO:teuthology.orchestra.run.smithi088.stderr:2024-06-03T09:06:57.142+0000 7f3f137fe640 10 monclient: get_auth_request con 0x7f3ef406d200 auth_method 0 2024-06-03T09:06:57.143 INFO:teuthology.orchestra.run.smithi088.stderr:2024-06-03T09:06:57.142+0000 7f3f10ff9640 10 client.30322.objecter ms_handle_connect 0x7f3ef406d200 2024-06-03T09:06:57.146 INFO:teuthology.orchestra.run.smithi088.stderr:2024-06-03T09:06:57.145+0000 7f3f10ff9640 1 -- 172.21.15.88:0/4233859610 <== mon.1 v1:172.21.15.192:6789/0 8 ==== osd_map(1014..1014 src has 622..1014) v4 ==== 5229+0+0 (unknown 3230192161 0 0) 0x7f3f040540f0 con 0x7f3f140a8a60 2024-06-03T09:06:57.146 INFO:teuthology.orchestra.run.smithi088.stderr:2024-06-03T09:06:57.145+0000 7f3f10ff9640 10 client.30322.objecter ms_dispatch 0x7f3f14000b90 osd_map(1014..1014 src has 622..1014) v4 2024-06-03T09:06:57.146 INFO:teuthology.orchestra.run.smithi088.stderr:2024-06-03T09:06:57.145+0000 7f3f10ff9640 3 client.30322.objecter handle_osd_map got epochs [1014,1014] > 0 2024-06-03T09:06:57.146 INFO:teuthology.orchestra.run.smithi088.stderr:2024-06-03T09:06:57.145+0000 7f3f10ff9640 3 client.30322.objecter handle_osd_map decoding full epoch 1014 2024-06-03T09:06:57.146 INFO:teuthology.orchestra.run.smithi088.stderr:2024-06-03T09:06:57.145+0000 7f3f10ff9640 10 client.30322.objecter checking command 1 2024-06-03T09:06:57.146 INFO:teuthology.orchestra.run.smithi088.stderr:2024-06-03T09:06:57.145+0000 7f3f10ff9640 20 client.30322.objecter _get_session s=0x7f3ef4075140 osd=1 3 2024-06-03T09:06:57.146 INFO:teuthology.orchestra.run.smithi088.stderr:2024-06-03T09:06:57.145+0000 7f3f10ff9640 20 client.30322.objecter put_session s=0x7f3ef4075140 osd=1 3 2024-06-03T09:06:57.146 INFO:teuthology.orchestra.run.smithi088.stderr:2024-06-03T09:06:57.145+0000 7f3f10ff9640 15 client.30322.objecter _session_command_op_remove -1 1 2024-06-03T09:06:57.146 INFO:teuthology.orchestra.run.smithi088.stderr:2024-06-03T09:06:57.145+0000 7f3f10ff9640 20 client.30322.objecter _get_session s=0x7f3ef4075140 osd=1 3 2024-06-03T09:06:57.146 INFO:teuthology.orchestra.run.smithi088.stderr:2024-06-03T09:06:57.145+0000 7f3f10ff9640 20 client.30322.objecter get_session s=0x7f3ef4075140 osd=1 3 2024-06-03T09:06:57.146 INFO:teuthology.orchestra.run.smithi088.stderr:2024-06-03T09:06:57.145+0000 7f3f10ff9640 15 client.30322.objecter _session_command_op_assign 1 1 2024-06-03T09:06:57.146 INFO:teuthology.orchestra.run.smithi088.stderr:2024-06-03T09:06:57.145+0000 7f3f10ff9640 20 client.30322.objecter put_session s=0x7f3ef4075140 osd=1 4 2024-06-03T09:06:57.147 INFO:teuthology.orchestra.run.smithi088.stderr:2024-06-03T09:06:57.145+0000 7f3f10ff9640 10 client.30322.objecter _send_command 1 2024-06-03T09:06:57.147 INFO:teuthology.orchestra.run.smithi088.stderr:2024-06-03T09:06:57.145+0000 7f3f10ff9640 1 -- 172.21.15.88:0/4233859610 --> v1:172.21.15.88:6800/35067 -- command(tid 1: {"prefix": "get_command_descriptions"}) v1 -- 0x7f3ef4079500 con 0x7f3ef40757f0 2024-06-03T09:06:57.147 INFO:teuthology.orchestra.run.smithi088.stderr:2024-06-03T09:06:57.145+0000 7f3f10ff9640 20 client.30322.objecter dump_active .. 0 homeless 2024-06-03T09:06:57.147 INFO:teuthology.orchestra.run.smithi088.stderr:2024-06-03T09:06:57.145+0000 7f3f10ff9640 1 -- 172.21.15.88:0/4233859610 <== mon.1 v1:172.21.15.192:6789/0 9 ==== mon_get_version_reply(handle=1 version=1014) v2 ==== 24+0+0 (unknown 3501658476 0 0) 0x7f3f040725a0 con 0x7f3f140a8a60 2024-06-03T09:06:57.147 INFO:teuthology.orchestra.run.smithi088.stderr:2024-06-03T09:06:57.145+0000 7f3f10ff9640 10 monclient: handle_get_version_reply finishing 1 version 1014 2024-06-03T09:06:57.147 INFO:teuthology.orchestra.run.smithi088.stderr:2024-06-03T09:06:57.146+0000 7f3f188c6640 10 monclient: get_auth_request con 0x7f3ef40757f0 auth_method 0 2024-06-03T09:06:57.147 INFO:teuthology.orchestra.run.smithi088.stderr:2024-06-03T09:06:57.147+0000 7f3f10ff9640 10 client.30322.objecter ms_handle_connect 0x7f3ef40757f0 2024-06-03T09:06:57.170 INFO:teuthology.orchestra.run.smithi088.stderr:2024-06-03T09:06:57.170+0000 7f3f10ff9640 1 -- 172.21.15.88:0/4233859610 <== osd.1 v1:172.21.15.88:6800/35067 1 ==== command_reply(tid 1: 0 ) v1 ==== 8+0+26118 (unknown 0 0 1686565167) 0x7f3ef4079500 con 0x7f3ef40757f0 2024-06-03T09:06:57.170 INFO:teuthology.orchestra.run.smithi088.stderr:2024-06-03T09:06:57.170+0000 7f3f10ff9640 10 client.30322.objecter ms_dispatch 0x7f3f14000b90 command_reply(tid 1: 0 ) v1 2024-06-03T09:06:57.170 INFO:teuthology.orchestra.run.smithi088.stderr:2024-06-03T09:06:57.170+0000 7f3f10ff9640 10 client.30322.objecter _finish_command 1 = system:0 2024-06-03T09:06:57.170 INFO:teuthology.orchestra.run.smithi088.stderr:2024-06-03T09:06:57.170+0000 7f3f10ff9640 20 client.30322.objecter put_session s=0x7f3ef4075140 osd=1 4 2024-06-03T09:06:57.170 INFO:teuthology.orchestra.run.smithi088.stderr:2024-06-03T09:06:57.170+0000 7f3f10ff9640 15 client.30322.objecter _session_command_op_remove 1 1 2024-06-03T09:06:57.184 INFO:teuthology.orchestra.run.smithi088.stderr:2024-06-03T09:06:57.184+0000 7f3f1a329640 10 client.30322.objecter _submit_command 2 [{"prefix": "config set", "var": "osd_debug_deep_scrub_sleep", "val": ["0"]}] 2024-06-03T09:06:57.184 INFO:teuthology.orchestra.run.smithi088.stderr:2024-06-03T09:06:57.184+0000 7f3f1a329640 15 client.30322.objecter _session_command_op_assign -1 2 2024-06-03T09:06:57.184 INFO:teuthology.orchestra.run.smithi088.stderr:2024-06-03T09:06:57.184+0000 7f3f1a329640 20 client.30322.objecter _get_session s=0x7f3ef4075140 osd=1 3 2024-06-03T09:06:57.184 INFO:teuthology.orchestra.run.smithi088.stderr:2024-06-03T09:06:57.184+0000 7f3f1a329640 20 client.30322.objecter put_session s=0x7f3ef4075140 osd=1 3 2024-06-03T09:06:57.184 INFO:teuthology.orchestra.run.smithi088.stderr:2024-06-03T09:06:57.184+0000 7f3f1a329640 20 client.30322.objecter _get_session s=0x7f3ef4075140 osd=1 3 2024-06-03T09:06:57.184 INFO:teuthology.orchestra.run.smithi088.stderr:2024-06-03T09:06:57.184+0000 7f3f1a329640 15 client.30322.objecter _session_command_op_remove -1 2 2024-06-03T09:06:57.184 INFO:teuthology.orchestra.run.smithi088.stderr:2024-06-03T09:06:57.184+0000 7f3f1a329640 20 client.30322.objecter get_session s=0x7f3ef4075140 osd=1 3 2024-06-03T09:06:57.184 INFO:teuthology.orchestra.run.smithi088.stderr:2024-06-03T09:06:57.184+0000 7f3f1a329640 15 client.30322.objecter _session_command_op_assign 1 2 2024-06-03T09:06:57.184 INFO:teuthology.orchestra.run.smithi088.stderr:2024-06-03T09:06:57.184+0000 7f3f1a329640 20 client.30322.objecter put_session s=0x7f3ef4075140 osd=1 4 2024-06-03T09:06:57.184 INFO:teuthology.orchestra.run.smithi088.stderr:2024-06-03T09:06:57.184+0000 7f3f1a329640 10 client.30322.objecter _send_command 2 2024-06-03T09:06:57.184 INFO:teuthology.orchestra.run.smithi088.stderr:2024-06-03T09:06:57.184+0000 7f3f1a329640 1 -- 172.21.15.88:0/4233859610 --> v1:172.21.15.88:6800/35067 -- command(tid 2: {"prefix": "config set", "var": "osd_debug_deep_scrub_sleep", "val": ["0"]}) v1 -- 0x7f3f140aa630 con 0x7f3ef40757f0 2024-06-03T09:06:57.197 INFO:teuthology.orchestra.run.smithi088.stderr:2024-06-03T09:06:57.197+0000 7f3f10ff9640 1 -- 172.21.15.88:0/4233859610 <== osd.1 v1:172.21.15.88:6800/35067 2 ==== command_reply(tid 2: 0 ) v1 ==== 8+0+97 (unknown 0 0 119112422) 0x7f3f0c007b60 con 0x7f3ef40757f0 2024-06-03T09:06:57.197 INFO:teuthology.orchestra.run.smithi088.stderr:2024-06-03T09:06:57.197+0000 7f3f10ff9640 10 client.30322.objecter ms_dispatch 0x7f3f14000b90 command_reply(tid 2: 0 ) v1 2024-06-03T09:06:57.197 INFO:teuthology.orchestra.run.smithi088.stderr:2024-06-03T09:06:57.197+0000 7f3f10ff9640 10 client.30322.objecter _finish_command 2 = system:0 2024-06-03T09:06:57.197 INFO:teuthology.orchestra.run.smithi088.stderr:2024-06-03T09:06:57.197+0000 7f3f10ff9640 20 client.30322.objecter put_session s=0x7f3ef4075140 osd=1 4 2024-06-03T09:06:57.197 INFO:teuthology.orchestra.run.smithi088.stderr:2024-06-03T09:06:57.197+0000 7f3f10ff9640 15 client.30322.objecter _session_command_op_remove 1 2 2024-06-03T09:06:57.198 INFO:teuthology.orchestra.run.smithi088.stdout:{ 2024-06-03T09:06:57.198 INFO:teuthology.orchestra.run.smithi088.stdout: "success": "osd_debug_deep_scrub_sleep = '' (not observed, change may require restart) " 2024-06-03T09:06:57.198 INFO:teuthology.orchestra.run.smithi088.stdout:} 2024-06-03T09:06:57.199 INFO:teuthology.orchestra.run.smithi088.stderr:2024-06-03T09:06:57.197+0000 7f3f1a329640 10 librados: watch_flush enter 2024-06-03T09:06:57.199 INFO:teuthology.orchestra.run.smithi088.stderr:2024-06-03T09:06:57.197+0000 7f3f1a329640 10 librados: watch_flush exit 2024-06-03T09:06:57.199 INFO:teuthology.orchestra.run.smithi088.stderr:2024-06-03T09:06:57.197+0000 7f3f1a329640 10 client.30322.objecter close_session for osd.1 2024-06-03T09:06:57.199 INFO:teuthology.orchestra.run.smithi088.stderr:2024-06-03T09:06:57.197+0000 7f3f1a329640 1 -- 172.21.15.88:0/4233859610 >> v1:172.21.15.88:6800/35067 conn(0x7f3ef40757f0 legacy=0x7f3ef4077c90 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-06-03T09:06:57.199 INFO:teuthology.orchestra.run.smithi088.stderr:2024-06-03T09:06:57.197+0000 7f3f1a329640 20 client.30322.objecter put_session s=0x7f3ef4075140 osd=1 1 2024-06-03T09:06:57.199 INFO:teuthology.orchestra.run.smithi088.stderr:2024-06-03T09:06:57.197+0000 7f3f1a329640 20 client.30322.objecter shutdown clearing up homeless session... 2024-06-03T09:06:57.199 INFO:teuthology.orchestra.run.smithi088.stderr:2024-06-03T09:06:57.197+0000 7f3f1a329640 10 client.30322.objecter successfully canceled tick 2024-06-03T09:06:57.200 INFO:teuthology.orchestra.run.smithi088.stderr:2024-06-03T09:06:57.197+0000 7f3f1a329640 1 -- 172.21.15.88:0/4233859610 >> v1:172.21.15.192:6800/34229 conn(0x7f3ef406d200 legacy=0x7f3ef406f6c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-06-03T09:06:57.200 INFO:teuthology.orchestra.run.smithi088.stderr:2024-06-03T09:06:57.197+0000 7f3f1a329640 10 monclient: shutdown 2024-06-03T09:06:57.200 INFO:teuthology.orchestra.run.smithi088.stderr:2024-06-03T09:06:57.197+0000 7f3f1a329640 20 monclient: shutdown discarding 0 pending message(s) 2024-06-03T09:06:57.200 INFO:teuthology.orchestra.run.smithi088.stderr:2024-06-03T09:06:57.197+0000 7f3f1a329640 1 -- 172.21.15.88:0/4233859610 >> v1:172.21.15.192:6789/0 conn(0x7f3f140a8a60 legacy=0x7f3f14142ae0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-06-03T09:06:57.200 INFO:teuthology.orchestra.run.smithi088.stderr:2024-06-03T09:06:57.197+0000 7f3f188c6640 1 -- 172.21.15.88:0/4233859610 reap_dead start 2024-06-03T09:06:57.200 INFO:teuthology.orchestra.run.smithi088.stderr:2024-06-03T09:06:57.197+0000 7f3f1a329640 1 -- 172.21.15.88:0/4233859610 shutdown_connections 2024-06-03T09:06:57.200 INFO:teuthology.orchestra.run.smithi088.stderr:2024-06-03T09:06:57.197+0000 7f3f1a329640 1 -- 172.21.15.88:0/4233859610 >> 172.21.15.88:0/4233859610 conn(0x7f3f140b7170 msgr2=0x7f3f140b5420 unknown :-1 s=STATE_NONE l=0).mark_down 2024-06-03T09:06:57.200 INFO:teuthology.orchestra.run.smithi088.stderr:2024-06-03T09:06:57.198+0000 7f3f1a329640 1 -- 172.21.15.88:0/4233859610 shutdown_connections 2024-06-03T09:06:57.200 INFO:teuthology.orchestra.run.smithi088.stderr:2024-06-03T09:06:57.198+0000 7f3f1a329640 1 -- 172.21.15.88:0/4233859610 wait complete. 2024-06-03T09:06:57.200 INFO:teuthology.orchestra.run.smithi088.stderr:2024-06-03T09:06:57.198+0000 7f3f1a329640 1 librados: shutdown 2024-06-03T09:06:57.209 DEBUG:teuthology.orchestra.run.smithi088:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd deep-scrub 1 2024-06-03T09:06:57.309 INFO:teuthology.orchestra.run.smithi088.stderr:2024-06-03T09:06:57.309+0000 7fbbdf88f640 10 monclient: get_monmap_and_config 2024-06-03T09:06:57.309 INFO:teuthology.orchestra.run.smithi088.stderr:2024-06-03T09:06:57.309+0000 7fbbdf88f640 10 monclient: build_initial_monmap 2024-06-03T09:06:57.310 INFO:teuthology.orchestra.run.smithi088.stderr:2024-06-03T09:06:57.309+0000 7fbbdf88f640 10 monclient: monmap: 2024-06-03T09:06:57.310 INFO:teuthology.orchestra.run.smithi088.stderr:epoch 0 2024-06-03T09:06:57.310 INFO:teuthology.orchestra.run.smithi088.stderr:fsid ec3c3fdd-3ac6-4d7d-9412-683130ed8b04 2024-06-03T09:06:57.310 INFO:teuthology.orchestra.run.smithi088.stderr:last_changed 2024-06-03T09:06:57.310696+0000 2024-06-03T09:06:57.310 INFO:teuthology.orchestra.run.smithi088.stderr:created 2024-06-03T09:06:57.310696+0000 2024-06-03T09:06:57.310 INFO:teuthology.orchestra.run.smithi088.stderr:min_mon_release 0 (unknown) 2024-06-03T09:06:57.310 INFO:teuthology.orchestra.run.smithi088.stderr:election_strategy: 3 2024-06-03T09:06:57.310 INFO:teuthology.orchestra.run.smithi088.stderr:0: v1:172.21.15.88:6789/0 mon.noname-a 2024-06-03T09:06:57.310 INFO:teuthology.orchestra.run.smithi088.stderr:1: v1:172.21.15.192:6789/0 mon.noname-c 2024-06-03T09:06:57.310 INFO:teuthology.orchestra.run.smithi088.stderr:2: v1:172.21.15.88:6790/0 mon.noname-b 2024-06-03T09:06:57.310 INFO:teuthology.orchestra.run.smithi088.stderr: 2024-06-03T09:06:57.311 INFO:teuthology.orchestra.run.smithi088.stderr:2024-06-03T09:06:57.310+0000 7fbbdf88f640 1 Processor -- start 2024-06-03T09:06:57.312 INFO:teuthology.orchestra.run.smithi088.stderr:2024-06-03T09:06:57.310+0000 7fbbdf88f640 1 -- start start 2024-06-03T09:06:57.312 INFO:teuthology.orchestra.run.smithi088.stderr:2024-06-03T09:06:57.310+0000 7fbbdf88f640 10 monclient: init 2024-06-03T09:06:57.312 INFO:teuthology.orchestra.run.smithi088.stderr:2024-06-03T09:06:57.310+0000 7fbbdf88f640 10 monclient: _reopen_session rank -1 2024-06-03T09:06:57.312 INFO:teuthology.orchestra.run.smithi088.stderr:2024-06-03T09:06:57.311+0000 7fbbdf88f640 10 monclient: _add_conns ranks=[1,0,2] 2024-06-03T09:06:57.312 INFO:teuthology.orchestra.run.smithi088.stderr:2024-06-03T09:06:57.311+0000 7fbbdf88f640 10 monclient(hunting): picked mon.noname-c con 0x7fbbd80adb90 addr v1:172.21.15.192:6789/0 2024-06-03T09:06:57.312 INFO:teuthology.orchestra.run.smithi088.stderr:2024-06-03T09:06:57.311+0000 7fbbdf88f640 10 monclient(hunting): picked mon.noname-a con 0x7fbbd806ee00 addr v1:172.21.15.88:6789/0 2024-06-03T09:06:57.312 INFO:teuthology.orchestra.run.smithi088.stderr:2024-06-03T09:06:57.311+0000 7fbbdf88f640 10 monclient(hunting): picked mon.noname-b con 0x7fbbd81383e0 addr v1:172.21.15.88:6790/0 2024-06-03T09:06:57.312 INFO:teuthology.orchestra.run.smithi088.stderr:2024-06-03T09:06:57.311+0000 7fbbdf88f640 1 -- --> v1:172.21.15.88:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fbbd806f2c0 con 0x7fbbd806ee00 2024-06-03T09:06:57.312 INFO:teuthology.orchestra.run.smithi088.stderr:2024-06-03T09:06:57.311+0000 7fbbdf88f640 1 -- --> v1:172.21.15.192:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fbbd80710c0 con 0x7fbbd80adb90 2024-06-03T09:06:57.312 INFO:teuthology.orchestra.run.smithi088.stderr:2024-06-03T09:06:57.311+0000 7fbbdf88f640 1 -- --> v1:172.21.15.88:6790/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fbbd8071fd0 con 0x7fbbd81383e0 2024-06-03T09:06:57.312 INFO:teuthology.orchestra.run.smithi088.stderr:2024-06-03T09:06:57.311+0000 7fbbdf88f640 10 monclient(hunting): _renew_subs 2024-06-03T09:06:57.312 INFO:teuthology.orchestra.run.smithi088.stderr:2024-06-03T09:06:57.311+0000 7fbbdf88f640 10 monclient(hunting): authenticate will time out at 3073.121013s 2024-06-03T09:06:57.312 INFO:teuthology.orchestra.run.smithi088.stderr:2024-06-03T09:06:57.311+0000 7fbbdde2c640 1 --1- >> v1:172.21.15.88:6790/0 conn(0x7fbbd81383e0 0x7fbbd8149af0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.88:6790/0 says I am v1:172.21.15.88:50990/0 (socket says 172.21.15.88:50990) 2024-06-03T09:06:57.313 INFO:teuthology.orchestra.run.smithi088.stderr:2024-06-03T09:06:57.311+0000 7fbbdde2c640 1 -- 172.21.15.88:0/3150754724 learned_addr learned my addr 172.21.15.88:0/3150754724 (peer_addr_for_me v1:172.21.15.88:0/0) 2024-06-03T09:06:57.314 INFO:teuthology.orchestra.run.smithi088.stderr:2024-06-03T09:06:57.314+0000 7fbbcffff640 1 -- 172.21.15.88:0/3150754724 <== mon.1 v1:172.21.15.192:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 3601031322 0 0) 0x7fbbd80710c0 con 0x7fbbd80adb90 2024-06-03T09:06:57.314 INFO:teuthology.orchestra.run.smithi088.stderr:2024-06-03T09:06:57.314+0000 7fbbcffff640 10 monclient(hunting): _negotiate 2024-06-03T09:06:57.314 INFO:teuthology.orchestra.run.smithi088.stderr:2024-06-03T09:06:57.314+0000 7fbbcffff640 10 monclient(hunting): _init_auth method 2 2024-06-03T09:06:57.314 INFO:teuthology.orchestra.run.smithi088.stderr:2024-06-03T09:06:57.314+0000 7fbbcffff640 10 monclient(hunting): _init_auth creating new auth 2024-06-03T09:06:57.314 INFO:teuthology.orchestra.run.smithi088.stderr:2024-06-03T09:06:57.314+0000 7fbbcffff640 10 monclient(hunting): my global_id is 30325 2024-06-03T09:06:57.314 INFO:teuthology.orchestra.run.smithi088.stderr:2024-06-03T09:06:57.314+0000 7fbbcffff640 1 -- 172.21.15.88:0/3150754724 --> v1:172.21.15.192:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7fbbc4003520 con 0x7fbbd80adb90 2024-06-03T09:06:57.314 INFO:teuthology.orchestra.run.smithi088.stderr:2024-06-03T09:06:57.314+0000 7fbbcffff640 1 -- 172.21.15.88:0/3150754724 <== mon.0 v1:172.21.15.88:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 584669458 0 0) 0x7fbbd806f2c0 con 0x7fbbd806ee00 2024-06-03T09:06:57.314 INFO:teuthology.orchestra.run.smithi088.stderr:2024-06-03T09:06:57.314+0000 7fbbcffff640 10 monclient(hunting): _negotiate 2024-06-03T09:06:57.315 INFO:teuthology.orchestra.run.smithi088.stderr:2024-06-03T09:06:57.314+0000 7fbbcffff640 10 monclient(hunting): _init_auth method 2 2024-06-03T09:06:57.315 INFO:teuthology.orchestra.run.smithi088.stderr:2024-06-03T09:06:57.314+0000 7fbbcffff640 10 monclient(hunting): _init_auth creating new auth 2024-06-03T09:06:57.315 INFO:teuthology.orchestra.run.smithi088.stderr:2024-06-03T09:06:57.314+0000 7fbbcffff640 10 monclient(hunting): my global_id is 30753 2024-06-03T09:06:57.315 INFO:teuthology.orchestra.run.smithi088.stderr:2024-06-03T09:06:57.314+0000 7fbbcffff640 1 -- 172.21.15.88:0/3150754724 --> v1:172.21.15.88:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7fbbd80710c0 con 0x7fbbd806ee00 2024-06-03T09:06:57.315 INFO:teuthology.orchestra.run.smithi088.stderr:2024-06-03T09:06:57.314+0000 7fbbcffff640 1 -- 172.21.15.88:0/3150754724 <== mon.2 v1:172.21.15.88:6790/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 2810050075 0 0) 0x7fbbd8071fd0 con 0x7fbbd81383e0 2024-06-03T09:06:57.315 INFO:teuthology.orchestra.run.smithi088.stderr:2024-06-03T09:06:57.314+0000 7fbbcffff640 10 monclient(hunting): _negotiate 2024-06-03T09:06:57.315 INFO:teuthology.orchestra.run.smithi088.stderr:2024-06-03T09:06:57.314+0000 7fbbcffff640 10 monclient(hunting): _init_auth method 2 2024-06-03T09:06:57.315 INFO:teuthology.orchestra.run.smithi088.stderr:2024-06-03T09:06:57.314+0000 7fbbcffff640 10 monclient(hunting): _init_auth creating new auth 2024-06-03T09:06:57.315 INFO:teuthology.orchestra.run.smithi088.stderr:2024-06-03T09:06:57.314+0000 7fbbcffff640 10 monclient(hunting): my global_id is 30260 2024-06-03T09:06:57.315 INFO:teuthology.orchestra.run.smithi088.stderr:2024-06-03T09:06:57.314+0000 7fbbcffff640 1 -- 172.21.15.88:0/3150754724 --> v1:172.21.15.88:6790/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7fbbd806f2c0 con 0x7fbbd81383e0 2024-06-03T09:06:57.317 INFO:teuthology.orchestra.run.smithi088.stderr:2024-06-03T09:06:57.317+0000 7fbbcffff640 1 -- 172.21.15.88:0/3150754724 <== mon.1 v1:172.21.15.192:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 214+0+0 (unknown 2593781624 0 0) 0x7fbbc4003520 con 0x7fbbd80adb90 2024-06-03T09:06:57.317 INFO:teuthology.orchestra.run.smithi088.stderr:2024-06-03T09:06:57.317+0000 7fbbcffff640 1 -- 172.21.15.88:0/3150754724 >> v1:172.21.15.88:6790/0 conn(0x7fbbd81383e0 legacy=0x7fbbd8149af0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-06-03T09:06:57.317 INFO:teuthology.orchestra.run.smithi088.stderr:2024-06-03T09:06:57.317+0000 7fbbcffff640 1 -- 172.21.15.88:0/3150754724 >> v1:172.21.15.88:6789/0 conn(0x7fbbd806ee00 legacy=0x7fbbd8137cd0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-06-03T09:06:57.317 INFO:teuthology.orchestra.run.smithi088.stderr:2024-06-03T09:06:57.317+0000 7fbbcffff640 10 monclient: _finish_hunting 0 2024-06-03T09:06:57.317 INFO:teuthology.orchestra.run.smithi088.stderr:2024-06-03T09:06:57.317+0000 7fbbcffff640 1 monclient: found mon.noname-c 2024-06-03T09:06:57.317 INFO:teuthology.orchestra.run.smithi088.stderr:2024-06-03T09:06:57.317+0000 7fbbcffff640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-06-03T09:06:57.317 INFO:teuthology.orchestra.run.smithi088.stderr:2024-06-03T09:06:57.317+0000 7fbbcffff640 10 monclient: _send_mon_message to mon.noname-c at v1:172.21.15.192:6789/0 2024-06-03T09:06:57.317 INFO:teuthology.orchestra.run.smithi088.stderr:2024-06-03T09:06:57.317+0000 7fbbcffff640 1 -- 172.21.15.88:0/3150754724 --> v1:172.21.15.192:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fbbd8141750 con 0x7fbbd80adb90 2024-06-03T09:06:57.318 INFO:teuthology.orchestra.run.smithi088.stderr:2024-06-03T09:06:57.317+0000 7fbbcffff640 10 monclient: _finish_auth 0 2024-06-03T09:06:57.318 INFO:teuthology.orchestra.run.smithi088.stderr:2024-06-03T09:06:57.317+0000 7fbbcffff640 10 monclient: _check_auth_tickets 2024-06-03T09:06:57.318 INFO:teuthology.orchestra.run.smithi088.stderr:2024-06-03T09:06:57.317+0000 7fbbcffff640 20 monclient: _check_auth_rotating not needed by client.admin 2024-06-03T09:06:57.318 INFO:teuthology.orchestra.run.smithi088.stderr:2024-06-03T09:06:57.317+0000 7fbbcffff640 1 -- 172.21.15.88:0/3150754724 <== mon.1 v1:172.21.15.192:6789/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 2234374065 0 0) 0x7fbbd4001990 con 0x7fbbd80adb90 2024-06-03T09:06:57.318 INFO:teuthology.orchestra.run.smithi088.stderr:2024-06-03T09:06:57.317+0000 7fbbcffff640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-06-03T09:06:57.318 INFO:teuthology.orchestra.run.smithi088.stderr:2024-06-03T09:06:57.317+0000 7fbbcffff640 10 monclient: got monmap 1 from mon.noname-c (according to old e1) 2024-06-03T09:06:57.318 INFO:teuthology.orchestra.run.smithi088.stderr:2024-06-03T09:06:57.317+0000 7fbbcffff640 10 monclient: dump: 2024-06-03T09:06:57.318 INFO:teuthology.orchestra.run.smithi088.stderr:epoch 1 2024-06-03T09:06:57.318 INFO:teuthology.orchestra.run.smithi088.stderr:fsid ec3c3fdd-3ac6-4d7d-9412-683130ed8b04 2024-06-03T09:06:57.318 INFO:teuthology.orchestra.run.smithi088.stderr:last_changed 2024-06-03T08:34:31.885125+0000 2024-06-03T09:06:57.318 INFO:teuthology.orchestra.run.smithi088.stderr:created 2024-06-03T08:34:31.885125+0000 2024-06-03T09:06:57.318 INFO:teuthology.orchestra.run.smithi088.stderr:min_mon_release 18 (reef) 2024-06-03T09:06:57.318 INFO:teuthology.orchestra.run.smithi088.stderr:election_strategy: 3 2024-06-03T09:06:57.319 INFO:teuthology.orchestra.run.smithi088.stderr:0: v1:172.21.15.88:6789/0 mon.a 2024-06-03T09:06:57.319 INFO:teuthology.orchestra.run.smithi088.stderr:1: v1:172.21.15.192:6789/0 mon.b 2024-06-03T09:06:57.319 INFO:teuthology.orchestra.run.smithi088.stderr:2: v1:172.21.15.88:6790/0 mon.c 2024-06-03T09:06:57.319 INFO:teuthology.orchestra.run.smithi088.stderr: 2024-06-03T09:06:57.319 INFO:teuthology.orchestra.run.smithi088.stderr:2024-06-03T09:06:57.317+0000 7fbbdf88f640 5 monclient: authenticate success, global_id 30325 2024-06-03T09:06:57.319 INFO:teuthology.orchestra.run.smithi088.stderr:2024-06-03T09:06:57.317+0000 7fbbdf88f640 20 monclient: get_monmap_and_config waiting for monmap|config 2024-06-03T09:06:57.320 INFO:teuthology.orchestra.run.smithi088.stderr:2024-06-03T09:06:57.319+0000 7fbbcffff640 1 -- 172.21.15.88:0/3150754724 <== mon.1 v1:172.21.15.192:6789/0 4 ==== config(0 keys) v1 ==== 4+0+0 (unknown 0 0 0) 0x7fbbd4001bb0 con 0x7fbbd80adb90 2024-06-03T09:06:57.321 INFO:teuthology.orchestra.run.smithi088.stderr:2024-06-03T09:06:57.319+0000 7fbbcffff640 10 monclient: handle_config config(0 keys) v1 2024-06-03T09:06:57.321 INFO:teuthology.orchestra.run.smithi088.stderr:2024-06-03T09:06:57.319+0000 7fbbcffff640 1 -- 172.21.15.88:0/3150754724 <== mon.1 v1:172.21.15.192:6789/0 5 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 2234374065 0 0) 0x7fbbd4001f40 con 0x7fbbd80adb90 2024-06-03T09:06:57.321 INFO:teuthology.orchestra.run.smithi088.stderr:2024-06-03T09:06:57.319+0000 7fbbdf88f640 10 monclient: get_monmap_and_config success 2024-06-03T09:06:57.321 INFO:teuthology.orchestra.run.smithi088.stderr:2024-06-03T09:06:57.319+0000 7fbbdf88f640 4 set_mon_vals no callback set 2024-06-03T09:06:57.321 INFO:teuthology.orchestra.run.smithi088.stderr:2024-06-03T09:06:57.319+0000 7fbbcffff640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-06-03T09:06:57.321 INFO:teuthology.orchestra.run.smithi088.stderr:2024-06-03T09:06:57.319+0000 7fbbcffff640 10 monclient: got monmap 1 from mon.b (according to old e1) 2024-06-03T09:06:57.321 INFO:teuthology.orchestra.run.smithi088.stderr:2024-06-03T09:06:57.319+0000 7fbbcffff640 10 monclient: dump: 2024-06-03T09:06:57.321 INFO:teuthology.orchestra.run.smithi088.stderr:epoch 1 2024-06-03T09:06:57.321 INFO:teuthology.orchestra.run.smithi088.stderr:fsid ec3c3fdd-3ac6-4d7d-9412-683130ed8b04 2024-06-03T09:06:57.321 INFO:teuthology.orchestra.run.smithi088.stderr:last_changed 2024-06-03T08:34:31.885125+0000 2024-06-03T09:06:57.321 INFO:teuthology.orchestra.run.smithi088.stderr:created 2024-06-03T08:34:31.885125+0000 2024-06-03T09:06:57.321 INFO:teuthology.orchestra.run.smithi088.stderr:min_mon_release 18 (reef) 2024-06-03T09:06:57.321 INFO:teuthology.orchestra.run.smithi088.stderr:election_strategy: 3 2024-06-03T09:06:57.321 INFO:teuthology.orchestra.run.smithi088.stderr:0: v1:172.21.15.88:6789/0 mon.a 2024-06-03T09:06:57.322 INFO:teuthology.orchestra.run.smithi088.stderr:1: v1:172.21.15.192:6789/0 mon.b 2024-06-03T09:06:57.322 INFO:teuthology.orchestra.run.smithi088.stderr:2: v1:172.21.15.88:6790/0 mon.c 2024-06-03T09:06:57.322 INFO:teuthology.orchestra.run.smithi088.stderr: 2024-06-03T09:06:57.322 INFO:teuthology.orchestra.run.smithi088.stderr:2024-06-03T09:06:57.319+0000 7fbbdf88f640 10 monclient: shutdown 2024-06-03T09:06:57.322 INFO:teuthology.orchestra.run.smithi088.stderr:2024-06-03T09:06:57.319+0000 7fbbdf88f640 20 monclient: shutdown discarding 0 pending message(s) 2024-06-03T09:06:57.322 INFO:teuthology.orchestra.run.smithi088.stderr:2024-06-03T09:06:57.319+0000 7fbbdf88f640 1 -- 172.21.15.88:0/3150754724 >> v1:172.21.15.192:6789/0 conn(0x7fbbd80adb90 legacy=0x7fbbd80718c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-06-03T09:06:57.322 INFO:teuthology.orchestra.run.smithi088.stderr:2024-06-03T09:06:57.319+0000 7fbbdf88f640 1 -- 172.21.15.88:0/3150754724 shutdown_connections 2024-06-03T09:06:57.322 INFO:teuthology.orchestra.run.smithi088.stderr:2024-06-03T09:06:57.319+0000 7fbbdf88f640 1 -- 172.21.15.88:0/3150754724 >> 172.21.15.88:0/3150754724 conn(0x7fbbd80b2f20 msgr2=0x7fbbd80b3300 unknown :-1 s=STATE_NONE l=0).mark_down 2024-06-03T09:06:57.322 INFO:teuthology.orchestra.run.smithi088.stderr:2024-06-03T09:06:57.319+0000 7fbbdf88f640 1 -- 172.21.15.88:0/3150754724 shutdown_connections 2024-06-03T09:06:57.322 INFO:teuthology.orchestra.run.smithi088.stderr:2024-06-03T09:06:57.319+0000 7fbbdf88f640 1 -- 172.21.15.88:0/3150754724 wait complete. 2024-06-03T09:06:57.322 INFO:teuthology.orchestra.run.smithi088.stderr:2024-06-03T09:06:57.320+0000 7fbbdf88f640 10 monclient: build_initial_monmap 2024-06-03T09:06:57.322 INFO:teuthology.orchestra.run.smithi088.stderr:2024-06-03T09:06:57.320+0000 7fbbdf88f640 10 monclient: monmap: 2024-06-03T09:06:57.322 INFO:teuthology.orchestra.run.smithi088.stderr:epoch 0 2024-06-03T09:06:57.323 INFO:teuthology.orchestra.run.smithi088.stderr:fsid 00000000-0000-0000-0000-000000000000 2024-06-03T09:06:57.323 INFO:teuthology.orchestra.run.smithi088.stderr:last_changed 0.000000 2024-06-03T09:06:57.323 INFO:teuthology.orchestra.run.smithi088.stderr:created 0.000000 2024-06-03T09:06:57.323 INFO:teuthology.orchestra.run.smithi088.stderr:min_mon_release 0 (unknown) 2024-06-03T09:06:57.323 INFO:teuthology.orchestra.run.smithi088.stderr:election_strategy: 1 2024-06-03T09:06:57.323 INFO:teuthology.orchestra.run.smithi088.stderr:0: v1:172.21.15.88:6789/0 mon.noname-a 2024-06-03T09:06:57.323 INFO:teuthology.orchestra.run.smithi088.stderr:1: v1:172.21.15.192:6789/0 mon.noname-b 2024-06-03T09:06:57.323 INFO:teuthology.orchestra.run.smithi088.stderr:2: v1:172.21.15.88:6790/0 mon.noname-c 2024-06-03T09:06:57.323 INFO:teuthology.orchestra.run.smithi088.stderr: 2024-06-03T09:06:57.323 INFO:teuthology.orchestra.run.smithi088.stderr:2024-06-03T09:06:57.320+0000 7fbbdf88f640 1 librados: starting msgr at 2024-06-03T09:06:57.323 INFO:teuthology.orchestra.run.smithi088.stderr:2024-06-03T09:06:57.320+0000 7fbbdf88f640 1 librados: starting objecter 2024-06-03T09:06:57.323 INFO:teuthology.orchestra.run.smithi088.stderr:2024-06-03T09:06:57.320+0000 7fbbdf88f640 1 Processor -- start 2024-06-03T09:06:57.323 INFO:teuthology.orchestra.run.smithi088.stderr:2024-06-03T09:06:57.320+0000 7fbbdf88f640 1 -- start start 2024-06-03T09:06:57.323 INFO:teuthology.orchestra.run.smithi088.stderr:2024-06-03T09:06:57.320+0000 7fbbdf88f640 1 librados: setting wanted keys 2024-06-03T09:06:57.324 INFO:teuthology.orchestra.run.smithi088.stderr:2024-06-03T09:06:57.320+0000 7fbbdf88f640 1 librados: calling monclient init 2024-06-03T09:06:57.324 INFO:teuthology.orchestra.run.smithi088.stderr:2024-06-03T09:06:57.320+0000 7fbbdf88f640 10 monclient: init 2024-06-03T09:06:57.325 INFO:teuthology.orchestra.run.smithi088.stderr:2024-06-03T09:06:57.321+0000 7fbbdf88f640 10 monclient: _reopen_session rank -1 2024-06-03T09:06:57.325 INFO:teuthology.orchestra.run.smithi088.stderr:2024-06-03T09:06:57.321+0000 7fbbdf88f640 10 monclient: _add_conns ranks=[0,1,2] 2024-06-03T09:06:57.326 INFO:teuthology.orchestra.run.smithi088.stderr:2024-06-03T09:06:57.321+0000 7fbbdf88f640 10 monclient(hunting): picked mon.noname-a con 0x7fbbd806ee00 addr v1:172.21.15.88:6789/0 2024-06-03T09:06:57.326 INFO:teuthology.orchestra.run.smithi088.stderr:2024-06-03T09:06:57.321+0000 7fbbdf88f640 10 monclient(hunting): picked mon.noname-b con 0x7fbbd80adb90 addr v1:172.21.15.192:6789/0 2024-06-03T09:06:57.326 INFO:teuthology.orchestra.run.smithi088.stderr:2024-06-03T09:06:57.321+0000 7fbbdf88f640 10 monclient(hunting): picked mon.noname-c con 0x7fbbd81383e0 addr v1:172.21.15.88:6790/0 2024-06-03T09:06:57.326 INFO:teuthology.orchestra.run.smithi088.stderr:2024-06-03T09:06:57.321+0000 7fbbdf88f640 1 -- --> v1:172.21.15.88:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fbbd80aad50 con 0x7fbbd806ee00 2024-06-03T09:06:57.326 INFO:teuthology.orchestra.run.smithi088.stderr:2024-06-03T09:06:57.321+0000 7fbbdf88f640 1 -- --> v1:172.21.15.192:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fbbd80ab1e0 con 0x7fbbd80adb90 2024-06-03T09:06:57.326 INFO:teuthology.orchestra.run.smithi088.stderr:2024-06-03T09:06:57.321+0000 7fbbdf88f640 1 -- --> v1:172.21.15.88:6790/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fbbd80aa7a0 con 0x7fbbd81383e0 2024-06-03T09:06:57.326 INFO:teuthology.orchestra.run.smithi088.stderr:2024-06-03T09:06:57.321+0000 7fbbdf88f640 10 monclient(hunting): _renew_subs 2024-06-03T09:06:57.326 INFO:teuthology.orchestra.run.smithi088.stderr:2024-06-03T09:06:57.321+0000 7fbbdf88f640 10 monclient(hunting): authenticate will time out at 3073.131537s 2024-06-03T09:06:57.326 INFO:teuthology.orchestra.run.smithi088.stderr:2024-06-03T09:06:57.321+0000 7fbbdde2c640 1 --1- >> v1:172.21.15.88:6790/0 conn(0x7fbbd81383e0 0x7fbbd81ddbb0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=1).handle_server_banner_and_identify peer v1:172.21.15.88:6790/0 says I am v1:172.21.15.88:50992/0 (socket says 172.21.15.88:50992) 2024-06-03T09:06:57.326 INFO:teuthology.orchestra.run.smithi088.stderr:2024-06-03T09:06:57.321+0000 7fbbdde2c640 1 -- 172.21.15.88:0/2354298640 learned_addr learned my addr 172.21.15.88:0/2354298640 (peer_addr_for_me v1:172.21.15.88:0/0) 2024-06-03T09:06:57.326 INFO:teuthology.orchestra.run.smithi088.stderr:2024-06-03T09:06:57.322+0000 7fbbcdffb640 10 client.?.objecter ms_handle_connect 0x7fbbd81383e0 2024-06-03T09:06:57.326 INFO:teuthology.orchestra.run.smithi088.stderr:2024-06-03T09:06:57.322+0000 7fbbcdffb640 10 client.?.objecter resend_mon_ops 2024-06-03T09:06:57.327 INFO:teuthology.orchestra.run.smithi088.stderr:2024-06-03T09:06:57.322+0000 7fbbcdffb640 10 client.?.objecter ms_handle_connect 0x7fbbd80adb90 2024-06-03T09:06:57.327 INFO:teuthology.orchestra.run.smithi088.stderr:2024-06-03T09:06:57.322+0000 7fbbcdffb640 10 client.?.objecter resend_mon_ops 2024-06-03T09:06:57.327 INFO:teuthology.orchestra.run.smithi088.stderr:2024-06-03T09:06:57.322+0000 7fbbcdffb640 10 client.?.objecter ms_handle_connect 0x7fbbd806ee00 2024-06-03T09:06:57.327 INFO:teuthology.orchestra.run.smithi088.stderr:2024-06-03T09:06:57.322+0000 7fbbcdffb640 10 client.?.objecter resend_mon_ops 2024-06-03T09:06:57.327 INFO:teuthology.orchestra.run.smithi088.stderr:2024-06-03T09:06:57.324+0000 7fbbcdffb640 1 -- 172.21.15.88:0/2354298640 <== mon.2 v1:172.21.15.88:6790/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 167517967 0 0) 0x7fbbd80aa7a0 con 0x7fbbd81383e0 2024-06-03T09:06:57.327 INFO:teuthology.orchestra.run.smithi088.stderr:2024-06-03T09:06:57.324+0000 7fbbcdffb640 10 monclient(hunting): _negotiate 2024-06-03T09:06:57.327 INFO:teuthology.orchestra.run.smithi088.stderr:2024-06-03T09:06:57.324+0000 7fbbcdffb640 10 monclient(hunting): _init_auth method 2 2024-06-03T09:06:57.327 INFO:teuthology.orchestra.run.smithi088.stderr:2024-06-03T09:06:57.324+0000 7fbbcdffb640 10 monclient(hunting): _init_auth creating new auth 2024-06-03T09:06:57.327 INFO:teuthology.orchestra.run.smithi088.stderr:2024-06-03T09:06:57.324+0000 7fbbcdffb640 10 monclient(hunting): my global_id is 30263 2024-06-03T09:06:57.327 INFO:teuthology.orchestra.run.smithi088.stderr:2024-06-03T09:06:57.324+0000 7fbbcdffb640 1 -- 172.21.15.88:0/2354298640 --> v1:172.21.15.88:6790/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7fbbbc004eb0 con 0x7fbbd81383e0 2024-06-03T09:06:57.327 INFO:teuthology.orchestra.run.smithi088.stderr:2024-06-03T09:06:57.324+0000 7fbbcdffb640 1 -- 172.21.15.88:0/2354298640 <== mon.1 v1:172.21.15.192:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 2200607540 0 0) 0x7fbbd80ab1e0 con 0x7fbbd80adb90 2024-06-03T09:06:57.327 INFO:teuthology.orchestra.run.smithi088.stderr:2024-06-03T09:06:57.324+0000 7fbbcdffb640 10 monclient(hunting): _negotiate 2024-06-03T09:06:57.327 INFO:teuthology.orchestra.run.smithi088.stderr:2024-06-03T09:06:57.324+0000 7fbbcdffb640 10 monclient(hunting): _init_auth method 2 2024-06-03T09:06:57.327 INFO:teuthology.orchestra.run.smithi088.stderr:2024-06-03T09:06:57.324+0000 7fbbcdffb640 10 monclient(hunting): _init_auth creating new auth 2024-06-03T09:06:57.328 INFO:teuthology.orchestra.run.smithi088.stderr:2024-06-03T09:06:57.324+0000 7fbbcdffb640 10 monclient(hunting): my global_id is 30328 2024-06-03T09:06:57.328 INFO:teuthology.orchestra.run.smithi088.stderr:2024-06-03T09:06:57.324+0000 7fbbcdffb640 1 -- 172.21.15.88:0/2354298640 --> v1:172.21.15.192:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7fbbd80aa7a0 con 0x7fbbd80adb90 2024-06-03T09:06:57.328 INFO:teuthology.orchestra.run.smithi088.stderr:2024-06-03T09:06:57.324+0000 7fbbcdffb640 1 -- 172.21.15.88:0/2354298640 <== mon.0 v1:172.21.15.88:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 3699918369 0 0) 0x7fbbd80aad50 con 0x7fbbd806ee00 2024-06-03T09:06:57.328 INFO:teuthology.orchestra.run.smithi088.stderr:2024-06-03T09:06:57.324+0000 7fbbcdffb640 10 monclient(hunting): _negotiate 2024-06-03T09:06:57.328 INFO:teuthology.orchestra.run.smithi088.stderr:2024-06-03T09:06:57.324+0000 7fbbcdffb640 10 monclient(hunting): _init_auth method 2 2024-06-03T09:06:57.328 INFO:teuthology.orchestra.run.smithi088.stderr:2024-06-03T09:06:57.324+0000 7fbbcdffb640 10 monclient(hunting): _init_auth creating new auth 2024-06-03T09:06:57.328 INFO:teuthology.orchestra.run.smithi088.stderr:2024-06-03T09:06:57.324+0000 7fbbcdffb640 10 monclient(hunting): my global_id is 30756 2024-06-03T09:06:57.328 INFO:teuthology.orchestra.run.smithi088.stderr:2024-06-03T09:06:57.324+0000 7fbbcdffb640 1 -- 172.21.15.88:0/2354298640 --> v1:172.21.15.88:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7fbbd80ab1e0 con 0x7fbbd806ee00 2024-06-03T09:06:57.330 INFO:teuthology.orchestra.run.smithi088.stderr:2024-06-03T09:06:57.329+0000 7fbbcdffb640 1 -- 172.21.15.88:0/2354298640 <== mon.1 v1:172.21.15.192:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 764+0+0 (unknown 3897740960 0 0) 0x7fbbd80aa7a0 con 0x7fbbd80adb90 2024-06-03T09:06:57.330 INFO:teuthology.orchestra.run.smithi088.stderr:2024-06-03T09:06:57.329+0000 7fbbcdffb640 1 -- 172.21.15.88:0/2354298640 --> v1:172.21.15.192:6789/0 -- auth(proto 2 165 bytes epoch 0) v1 -- 0x7fbbd80aad50 con 0x7fbbd80adb90 2024-06-03T09:06:57.330 INFO:teuthology.orchestra.run.smithi088.stderr:2024-06-03T09:06:57.329+0000 7fbbcdffb640 1 -- 172.21.15.88:0/2354298640 <== mon.1 v1:172.21.15.192:6789/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 2234374065 0 0) 0x7fbbc80018b0 con 0x7fbbd80adb90 2024-06-03T09:06:57.330 INFO:teuthology.orchestra.run.smithi088.stderr:2024-06-03T09:06:57.329+0000 7fbbcdffb640 10 monclient(hunting): handle_monmap mon_map magic: 0 v1 2024-06-03T09:06:57.330 INFO:teuthology.orchestra.run.smithi088.stderr:2024-06-03T09:06:57.329+0000 7fbbcdffb640 10 monclient(hunting): got monmap 1 from mon.noname-b (according to old e1) 2024-06-03T09:06:57.330 INFO:teuthology.orchestra.run.smithi088.stderr:2024-06-03T09:06:57.329+0000 7fbbcdffb640 10 monclient(hunting): dump: 2024-06-03T09:06:57.330 INFO:teuthology.orchestra.run.smithi088.stderr:epoch 1 2024-06-03T09:06:57.330 INFO:teuthology.orchestra.run.smithi088.stderr:fsid ec3c3fdd-3ac6-4d7d-9412-683130ed8b04 2024-06-03T09:06:57.330 INFO:teuthology.orchestra.run.smithi088.stderr:last_changed 2024-06-03T08:34:31.885125+0000 2024-06-03T09:06:57.330 INFO:teuthology.orchestra.run.smithi088.stderr:created 2024-06-03T08:34:31.885125+0000 2024-06-03T09:06:57.330 INFO:teuthology.orchestra.run.smithi088.stderr:min_mon_release 18 (reef) 2024-06-03T09:06:57.330 INFO:teuthology.orchestra.run.smithi088.stderr:election_strategy: 3 2024-06-03T09:06:57.330 INFO:teuthology.orchestra.run.smithi088.stderr:0: v1:172.21.15.88:6789/0 mon.a 2024-06-03T09:06:57.331 INFO:teuthology.orchestra.run.smithi088.stderr:1: v1:172.21.15.192:6789/0 mon.b 2024-06-03T09:06:57.331 INFO:teuthology.orchestra.run.smithi088.stderr:2: v1:172.21.15.88:6790/0 mon.c 2024-06-03T09:06:57.331 INFO:teuthology.orchestra.run.smithi088.stderr: 2024-06-03T09:06:57.331 INFO:teuthology.orchestra.run.smithi088.stderr:2024-06-03T09:06:57.329+0000 7fbbcdffb640 10 monclient(hunting): _finish_auth 0 2024-06-03T09:06:57.331 INFO:teuthology.orchestra.run.smithi088.stderr:2024-06-03T09:06:57.329+0000 7fbbcdffb640 1 -- 172.21.15.88:0/2354298640 <== mon.2 v1:172.21.15.88:6790/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 764+0+0 (unknown 4004219464 0 0) 0x7fbbbc004eb0 con 0x7fbbd81383e0 2024-06-03T09:06:57.331 INFO:teuthology.orchestra.run.smithi088.stderr:2024-06-03T09:06:57.329+0000 7fbbcdffb640 1 -- 172.21.15.88:0/2354298640 --> v1:172.21.15.88:6790/0 -- auth(proto 2 165 bytes epoch 0) v1 -- 0x7fbbd80aa7a0 con 0x7fbbd81383e0 2024-06-03T09:06:57.331 INFO:teuthology.orchestra.run.smithi088.stderr:2024-06-03T09:06:57.329+0000 7fbbcdffb640 1 -- 172.21.15.88:0/2354298640 <== mon.2 v1:172.21.15.88:6790/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 2234374065 0 0) 0x7fbbd0001810 con 0x7fbbd81383e0 2024-06-03T09:06:57.331 INFO:teuthology.orchestra.run.smithi088.stderr:2024-06-03T09:06:57.329+0000 7fbbcdffb640 10 monclient(hunting): handle_monmap mon_map magic: 0 v1 2024-06-03T09:06:57.331 INFO:teuthology.orchestra.run.smithi088.stderr:2024-06-03T09:06:57.329+0000 7fbbcdffb640 10 monclient(hunting): got monmap 1 from mon.c (according to old e1) 2024-06-03T09:06:57.331 INFO:teuthology.orchestra.run.smithi088.stderr:2024-06-03T09:06:57.329+0000 7fbbcdffb640 10 monclient(hunting): dump: 2024-06-03T09:06:57.331 INFO:teuthology.orchestra.run.smithi088.stderr:epoch 1 2024-06-03T09:06:57.331 INFO:teuthology.orchestra.run.smithi088.stderr:fsid ec3c3fdd-3ac6-4d7d-9412-683130ed8b04 2024-06-03T09:06:57.331 INFO:teuthology.orchestra.run.smithi088.stderr:last_changed 2024-06-03T08:34:31.885125+0000 2024-06-03T09:06:57.331 INFO:teuthology.orchestra.run.smithi088.stderr:created 2024-06-03T08:34:31.885125+0000 2024-06-03T09:06:57.332 INFO:teuthology.orchestra.run.smithi088.stderr:min_mon_release 18 (reef) 2024-06-03T09:06:57.332 INFO:teuthology.orchestra.run.smithi088.stderr:election_strategy: 3 2024-06-03T09:06:57.332 INFO:teuthology.orchestra.run.smithi088.stderr:0: v1:172.21.15.88:6789/0 mon.a 2024-06-03T09:06:57.332 INFO:teuthology.orchestra.run.smithi088.stderr:1: v1:172.21.15.192:6789/0 mon.b 2024-06-03T09:06:57.332 INFO:teuthology.orchestra.run.smithi088.stderr:2: v1:172.21.15.88:6790/0 mon.c 2024-06-03T09:06:57.332 INFO:teuthology.orchestra.run.smithi088.stderr: 2024-06-03T09:06:57.332 INFO:teuthology.orchestra.run.smithi088.stderr:2024-06-03T09:06:57.329+0000 7fbbcdffb640 1 -- 172.21.15.88:0/2354298640 <== mon.0 v1:172.21.15.88:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 764+0+0 (unknown 3646190357 0 0) 0x7fbbd80ab1e0 con 0x7fbbd806ee00 2024-06-03T09:06:57.332 INFO:teuthology.orchestra.run.smithi088.stderr:2024-06-03T09:06:57.329+0000 7fbbcdffb640 1 -- 172.21.15.88:0/2354298640 --> v1:172.21.15.88:6789/0 -- auth(proto 2 165 bytes epoch 0) v1 -- 0x7fbbbc004eb0 con 0x7fbbd806ee00 2024-06-03T09:06:57.332 INFO:teuthology.orchestra.run.smithi088.stderr:2024-06-03T09:06:57.329+0000 7fbbcdffb640 1 -- 172.21.15.88:0/2354298640 <== mon.0 v1:172.21.15.88:6789/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 2234374065 0 0) 0x7fbbd4001ec0 con 0x7fbbd806ee00 2024-06-03T09:06:57.332 INFO:teuthology.orchestra.run.smithi088.stderr:2024-06-03T09:06:57.329+0000 7fbbcdffb640 10 monclient(hunting): handle_monmap mon_map magic: 0 v1 2024-06-03T09:06:57.332 INFO:teuthology.orchestra.run.smithi088.stderr:2024-06-03T09:06:57.329+0000 7fbbcdffb640 10 monclient(hunting): got monmap 1 from mon.a (according to old e1) 2024-06-03T09:06:57.332 INFO:teuthology.orchestra.run.smithi088.stderr:2024-06-03T09:06:57.329+0000 7fbbcdffb640 10 monclient(hunting): dump: 2024-06-03T09:06:57.332 INFO:teuthology.orchestra.run.smithi088.stderr:epoch 1 2024-06-03T09:06:57.332 INFO:teuthology.orchestra.run.smithi088.stderr:fsid ec3c3fdd-3ac6-4d7d-9412-683130ed8b04 2024-06-03T09:06:57.332 INFO:teuthology.orchestra.run.smithi088.stderr:last_changed 2024-06-03T08:34:31.885125+0000 2024-06-03T09:06:57.333 INFO:teuthology.orchestra.run.smithi088.stderr:created 2024-06-03T08:34:31.885125+0000 2024-06-03T09:06:57.333 INFO:teuthology.orchestra.run.smithi088.stderr:min_mon_release 18 (reef) 2024-06-03T09:06:57.333 INFO:teuthology.orchestra.run.smithi088.stderr:election_strategy: 3 2024-06-03T09:06:57.333 INFO:teuthology.orchestra.run.smithi088.stderr:0: v1:172.21.15.88:6789/0 mon.a 2024-06-03T09:06:57.333 INFO:teuthology.orchestra.run.smithi088.stderr:1: v1:172.21.15.192:6789/0 mon.b 2024-06-03T09:06:57.333 INFO:teuthology.orchestra.run.smithi088.stderr:2: v1:172.21.15.88:6790/0 mon.c 2024-06-03T09:06:57.333 INFO:teuthology.orchestra.run.smithi088.stderr: 2024-06-03T09:06:57.334 INFO:teuthology.orchestra.run.smithi088.stderr:2024-06-03T09:06:57.331+0000 7fbbcdffb640 1 -- 172.21.15.88:0/2354298640 <== mon.1 v1:172.21.15.192:6789/0 4 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 580+0+0 (unknown 2373695795 0 0) 0x7fbbd80aad50 con 0x7fbbd80adb90 2024-06-03T09:06:57.334 INFO:teuthology.orchestra.run.smithi088.stderr:2024-06-03T09:06:57.331+0000 7fbbcdffb640 1 -- 172.21.15.88:0/2354298640 >> v1:172.21.15.88:6790/0 conn(0x7fbbd81383e0 legacy=0x7fbbd81ddbb0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-06-03T09:06:57.334 INFO:teuthology.orchestra.run.smithi088.stderr:2024-06-03T09:06:57.331+0000 7fbbcdffb640 1 -- 172.21.15.88:0/2354298640 >> v1:172.21.15.88:6789/0 conn(0x7fbbd806ee00 legacy=0x7fbbd81370a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-06-03T09:06:57.335 INFO:teuthology.orchestra.run.smithi088.stderr:2024-06-03T09:06:57.331+0000 7fbbcdffb640 10 monclient: _finish_hunting 0 2024-06-03T09:06:57.335 INFO:teuthology.orchestra.run.smithi088.stderr:2024-06-03T09:06:57.331+0000 7fbbcdffb640 1 monclient: found mon.b 2024-06-03T09:06:57.335 INFO:teuthology.orchestra.run.smithi088.stderr:2024-06-03T09:06:57.331+0000 7fbbcdffb640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-06-03T09:06:57.335 INFO:teuthology.orchestra.run.smithi088.stderr:2024-06-03T09:06:57.331+0000 7fbbcdffb640 10 monclient: _send_mon_message to mon.b at v1:172.21.15.192:6789/0 2024-06-03T09:06:57.335 INFO:teuthology.orchestra.run.smithi088.stderr:2024-06-03T09:06:57.331+0000 7fbbcdffb640 1 -- 172.21.15.88:0/2354298640 --> v1:172.21.15.192:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fbbd80aa930 con 0x7fbbd80adb90 2024-06-03T09:06:57.335 INFO:teuthology.orchestra.run.smithi088.stderr:2024-06-03T09:06:57.331+0000 7fbbcdffb640 10 monclient: _finish_auth 0 2024-06-03T09:06:57.335 INFO:teuthology.orchestra.run.smithi088.stderr:2024-06-03T09:06:57.331+0000 7fbbcdffb640 10 monclient: _check_auth_tickets 2024-06-03T09:06:57.335 INFO:teuthology.orchestra.run.smithi088.stderr:2024-06-03T09:06:57.331+0000 7fbbcdffb640 20 monclient: _check_auth_rotating not needed by client.admin 2024-06-03T09:06:57.335 INFO:teuthology.orchestra.run.smithi088.stderr:2024-06-03T09:06:57.331+0000 7fbbdf88f640 5 monclient: authenticate success, global_id 30328 2024-06-03T09:06:57.335 INFO:teuthology.orchestra.run.smithi088.stderr:2024-06-03T09:06:57.331+0000 7fbbdf88f640 10 monclient: _renew_subs 2024-06-03T09:06:57.335 INFO:teuthology.orchestra.run.smithi088.stderr:2024-06-03T09:06:57.331+0000 7fbbdf88f640 10 monclient: _send_mon_message to mon.b at v1:172.21.15.192:6789/0 2024-06-03T09:06:57.335 INFO:teuthology.orchestra.run.smithi088.stderr:2024-06-03T09:06:57.331+0000 7fbbdf88f640 1 -- 172.21.15.88:0/2354298640 --> v1:172.21.15.192:6789/0 -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fbbd813ae00 con 0x7fbbd80adb90 2024-06-03T09:06:57.335 INFO:teuthology.orchestra.run.smithi088.stderr:2024-06-03T09:06:57.332+0000 7fbbdf88f640 10 client.30328.objecter _maybe_request_map subscribing (onetime) to next osd map 2024-06-03T09:06:57.335 INFO:teuthology.orchestra.run.smithi088.stderr:2024-06-03T09:06:57.332+0000 7fbbdf88f640 10 monclient: _renew_subs 2024-06-03T09:06:57.336 INFO:teuthology.orchestra.run.smithi088.stderr:2024-06-03T09:06:57.332+0000 7fbbdf88f640 10 monclient: _send_mon_message to mon.b at v1:172.21.15.192:6789/0 2024-06-03T09:06:57.336 INFO:teuthology.orchestra.run.smithi088.stderr:2024-06-03T09:06:57.332+0000 7fbbdf88f640 1 -- 172.21.15.88:0/2354298640 --> v1:172.21.15.192:6789/0 -- mon_subscribe({osdmap=0}) v3 -- 0x7fbbd81377b0 con 0x7fbbd80adb90 2024-06-03T09:06:57.336 INFO:teuthology.orchestra.run.smithi088.stderr:2024-06-03T09:06:57.332+0000 7fbbdf88f640 1 librados: init done 2024-06-03T09:06:57.336 INFO:teuthology.orchestra.run.smithi088.stderr:2024-06-03T09:06:57.332+0000 7fbbdf88f640 10 start_mon_command cmd=[{"prefix": "get_command_descriptions"}] 2024-06-03T09:06:57.336 INFO:teuthology.orchestra.run.smithi088.stderr:2024-06-03T09:06:57.333+0000 7fbbdf88f640 10 monclient: _send_command 1 [{"prefix": "get_command_descriptions"}] 2024-06-03T09:06:57.336 INFO:teuthology.orchestra.run.smithi088.stderr:2024-06-03T09:06:57.333+0000 7fbbdf88f640 10 monclient: _send_mon_message to mon.b at v1:172.21.15.192:6789/0 2024-06-03T09:06:57.336 INFO:teuthology.orchestra.run.smithi088.stderr:2024-06-03T09:06:57.333+0000 7fbbdf88f640 1 -- 172.21.15.88:0/2354298640 --> v1:172.21.15.192:6789/0 -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fbbd80aab20 con 0x7fbbd80adb90 2024-06-03T09:06:57.579 INFO:teuthology.orchestra.run.smithi088.stderr:2024-06-03T09:06:57.578+0000 7fbbcdffb640 1 -- 172.21.15.88:0/2354298640 <== mon.1 v1:172.21.15.192:6789/0 5 ==== config(0 keys) v1 ==== 4+0+0 (unknown 0 0 0) 0x7fbbc80035f0 con 0x7fbbd80adb90 2024-06-03T09:06:57.579 INFO:teuthology.orchestra.run.smithi088.stderr:2024-06-03T09:06:57.578+0000 7fbbcdffb640 10 monclient: handle_config config(0 keys) v1 2024-06-03T09:06:57.579 INFO:teuthology.orchestra.run.smithi088.stderr:2024-06-03T09:06:57.578+0000 7fbbcdffb640 1 -- 172.21.15.88:0/2354298640 <== mon.1 v1:172.21.15.192:6789/0 6 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 2234374065 0 0) 0x7fbbc8003760 con 0x7fbbd80adb90 2024-06-03T09:06:57.579 INFO:teuthology.orchestra.run.smithi088.stderr:2024-06-03T09:06:57.578+0000 7fbbcdffb640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-06-03T09:06:57.579 INFO:teuthology.orchestra.run.smithi088.stderr:2024-06-03T09:06:57.578+0000 7fbbcdffb640 10 monclient: got monmap 1 from mon.b (according to old e1) 2024-06-03T09:06:57.579 INFO:teuthology.orchestra.run.smithi088.stderr:2024-06-03T09:06:57.578+0000 7fbbcdffb640 10 monclient: dump: 2024-06-03T09:06:57.579 INFO:teuthology.orchestra.run.smithi088.stderr:epoch 1 2024-06-03T09:06:57.579 INFO:teuthology.orchestra.run.smithi088.stderr:fsid ec3c3fdd-3ac6-4d7d-9412-683130ed8b04 2024-06-03T09:06:57.579 INFO:teuthology.orchestra.run.smithi088.stderr:last_changed 2024-06-03T08:34:31.885125+0000 2024-06-03T09:06:57.579 INFO:teuthology.orchestra.run.smithi088.stderr:created 2024-06-03T08:34:31.885125+0000 2024-06-03T09:06:57.579 INFO:teuthology.orchestra.run.smithi088.stderr:min_mon_release 18 (reef) 2024-06-03T09:06:57.580 INFO:teuthology.orchestra.run.smithi088.stderr:election_strategy: 3 2024-06-03T09:06:57.580 INFO:teuthology.orchestra.run.smithi088.stderr:0: v1:172.21.15.88:6789/0 mon.a 2024-06-03T09:06:57.580 INFO:teuthology.orchestra.run.smithi088.stderr:1: v1:172.21.15.192:6789/0 mon.b 2024-06-03T09:06:57.580 INFO:teuthology.orchestra.run.smithi088.stderr:2: v1:172.21.15.88:6790/0 mon.c 2024-06-03T09:06:57.580 INFO:teuthology.orchestra.run.smithi088.stderr: 2024-06-03T09:06:57.580 INFO:teuthology.orchestra.run.smithi088.stderr:2024-06-03T09:06:57.578+0000 7fbbcf7fe640 4 set_mon_vals no callback set 2024-06-03T09:06:57.580 INFO:teuthology.orchestra.run.smithi088.stderr:2024-06-03T09:06:57.579+0000 7fbbcdffb640 1 -- 172.21.15.88:0/2354298640 <== mon.1 v1:172.21.15.192:6789/0 7 ==== mgrmap(e 5) v1 ==== 89110+0+0 (unknown 873533973 0 0) 0x7fbbc80194e0 con 0x7fbbd80adb90 2024-06-03T09:06:57.581 INFO:teuthology.orchestra.run.smithi088.stderr:2024-06-03T09:06:57.580+0000 7fbbdd62b640 10 monclient: get_auth_request con 0x7fbbbc06d450 auth_method 0 2024-06-03T09:06:57.581 INFO:teuthology.orchestra.run.smithi088.stderr:2024-06-03T09:06:57.580+0000 7fbbcdffb640 10 client.30328.objecter ms_handle_connect 0x7fbbbc06d450 2024-06-03T09:06:57.582 INFO:teuthology.orchestra.run.smithi088.stderr:2024-06-03T09:06:57.582+0000 7fbbcdffb640 1 -- 172.21.15.88:0/2354298640 <== mon.1 v1:172.21.15.192:6789/0 8 ==== osd_map(1014..1014 src has 622..1014) v4 ==== 5229+0+0 (unknown 3230192161 0 0) 0x7fbbc8052d00 con 0x7fbbd80adb90 2024-06-03T09:06:57.582 INFO:teuthology.orchestra.run.smithi088.stderr:2024-06-03T09:06:57.582+0000 7fbbcdffb640 10 client.30328.objecter ms_dispatch 0x7fbbd8000b90 osd_map(1014..1014 src has 622..1014) v4 2024-06-03T09:06:57.582 INFO:teuthology.orchestra.run.smithi088.stderr:2024-06-03T09:06:57.582+0000 7fbbcdffb640 3 client.30328.objecter handle_osd_map got epochs [1014,1014] > 0 2024-06-03T09:06:57.582 INFO:teuthology.orchestra.run.smithi088.stderr:2024-06-03T09:06:57.582+0000 7fbbcdffb640 3 client.30328.objecter handle_osd_map decoding full epoch 1014 2024-06-03T09:06:57.582 INFO:teuthology.orchestra.run.smithi088.stderr:2024-06-03T09:06:57.582+0000 7fbbcdffb640 20 client.30328.objecter dump_active .. 0 homeless 2024-06-03T09:06:57.585 INFO:teuthology.orchestra.run.smithi088.stderr:2024-06-03T09:06:57.585+0000 7fbbcdffb640 1 -- 172.21.15.88:0/2354298640 <== mon.1 v1:172.21.15.192:6789/0 9 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+182498 (unknown 1092875540 0 3016760700) 0x7fbbc804e8b0 con 0x7fbbd80adb90 2024-06-03T09:06:57.585 INFO:teuthology.orchestra.run.smithi088.stderr:2024-06-03T09:06:57.585+0000 7fbbcdffb640 10 monclient: handle_mon_command_ack 1 [{"prefix": "get_command_descriptions"}] 2024-06-03T09:06:57.585 INFO:teuthology.orchestra.run.smithi088.stderr:2024-06-03T09:06:57.585+0000 7fbbcdffb640 10 monclient: _finish_command 1 = system:0 2024-06-03T09:06:57.760 INFO:teuthology.orchestra.run.smithi088.stderr:2024-06-03T09:06:57.760+0000 7fbbdf88f640 1 -- 172.21.15.88:0/2354298640 --> v1:172.21.15.192:6800/34229 -- mgr_command(tid 0: {"prefix": "osd deep-scrub", "who": "1", "target": ["mon-mgr", ""]}) v1 -- 0x7fbbd80adf70 con 0x7fbbbc06d450 2024-06-03T09:06:57.760 INFO:teuthology.orchestra.run.smithi088.stderr:2024-06-03T09:06:57.760+0000 7fbbcdffb640 1 -- 172.21.15.88:0/2354298640 <== mgr.4110 v1:172.21.15.192:6800/34229 1 ==== mgr_command_reply(tid 0: 0 instructed osd(s) 1 to deep-scrub) v1 ==== 41+0+0 (unknown 489605429 0 0) 0x7fbbd4005b90 con 0x7fbbbc06d450 2024-06-03T09:06:57.760 INFO:teuthology.orchestra.run.smithi088.stderr:instructed osd(s) 1 to deep-scrub 2024-06-03T09:06:57.763 INFO:teuthology.orchestra.run.smithi088.stderr:2024-06-03T09:06:57.763+0000 7fbbdf88f640 10 librados: watch_flush enter 2024-06-03T09:06:57.763 INFO:teuthology.orchestra.run.smithi088.stderr:2024-06-03T09:06:57.763+0000 7fbbdf88f640 10 librados: watch_flush exit 2024-06-03T09:06:57.763 INFO:teuthology.orchestra.run.smithi088.stderr:2024-06-03T09:06:57.763+0000 7fbbdf88f640 20 client.30328.objecter shutdown clearing up homeless session... 2024-06-03T09:06:57.763 INFO:teuthology.orchestra.run.smithi088.stderr:2024-06-03T09:06:57.763+0000 7fbbdf88f640 10 client.30328.objecter successfully canceled tick 2024-06-03T09:06:57.763 INFO:teuthology.orchestra.run.smithi088.stderr:2024-06-03T09:06:57.763+0000 7fbbdf88f640 1 -- 172.21.15.88:0/2354298640 >> v1:172.21.15.192:6800/34229 conn(0x7fbbbc06d450 legacy=0x7fbbbc06f910 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-06-03T09:06:57.763 INFO:teuthology.orchestra.run.smithi088.stderr:2024-06-03T09:06:57.763+0000 7fbbdf88f640 10 monclient: shutdown 2024-06-03T09:06:57.764 INFO:teuthology.orchestra.run.smithi088.stderr:2024-06-03T09:06:57.763+0000 7fbbdf88f640 20 monclient: shutdown discarding 0 pending message(s) 2024-06-03T09:06:57.764 INFO:teuthology.orchestra.run.smithi088.stderr:2024-06-03T09:06:57.763+0000 7fbbdf88f640 1 -- 172.21.15.88:0/2354298640 >> v1:172.21.15.192:6789/0 conn(0x7fbbd80adb90 legacy=0x7fbbd80ac6f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-06-03T09:06:57.764 INFO:teuthology.orchestra.run.smithi088.stderr:2024-06-03T09:06:57.763+0000 7fbbdf88f640 1 -- 172.21.15.88:0/2354298640 shutdown_connections 2024-06-03T09:06:57.764 INFO:teuthology.orchestra.run.smithi088.stderr:2024-06-03T09:06:57.763+0000 7fbbdf88f640 1 -- 172.21.15.88:0/2354298640 >> 172.21.15.88:0/2354298640 conn(0x7fbbd80b2f20 msgr2=0x7fbbd80ae150 unknown :-1 s=STATE_NONE l=0).mark_down 2024-06-03T09:06:57.764 INFO:teuthology.orchestra.run.smithi088.stderr:2024-06-03T09:06:57.763+0000 7fbbdf88f640 1 -- 172.21.15.88:0/2354298640 shutdown_connections 2024-06-03T09:06:57.764 INFO:teuthology.orchestra.run.smithi088.stderr:2024-06-03T09:06:57.763+0000 7fbbdf88f640 1 -- 172.21.15.88:0/2354298640 wait complete. 2024-06-03T09:06:57.764 INFO:teuthology.orchestra.run.smithi088.stderr:2024-06-03T09:06:57.763+0000 7fbbdf88f640 1 librados: shutdown 2024-06-03T09:06:57.774 INFO:tasks.ceph:Scrubbing osd.2 2024-06-03T09:06:57.774 DEBUG:teuthology.orchestra.run.smithi088:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph tell osd.2 config set osd_debug_deep_scrub_sleep 0 2024-06-03T09:06:57.882 INFO:teuthology.orchestra.run.smithi088.stderr:2024-06-03T09:06:57.881+0000 7f64bf173640 10 monclient: get_monmap_and_config 2024-06-03T09:06:57.882 INFO:teuthology.orchestra.run.smithi088.stderr:2024-06-03T09:06:57.881+0000 7f64bf173640 10 monclient: build_initial_monmap 2024-06-03T09:06:57.882 INFO:teuthology.orchestra.run.smithi088.stderr:2024-06-03T09:06:57.881+0000 7f64bf173640 10 monclient: monmap: 2024-06-03T09:06:57.882 INFO:teuthology.orchestra.run.smithi088.stderr:epoch 0 2024-06-03T09:06:57.882 INFO:teuthology.orchestra.run.smithi088.stderr:fsid ec3c3fdd-3ac6-4d7d-9412-683130ed8b04 2024-06-03T09:06:57.882 INFO:teuthology.orchestra.run.smithi088.stderr:last_changed 2024-06-03T09:06:57.882986+0000 2024-06-03T09:06:57.882 INFO:teuthology.orchestra.run.smithi088.stderr:created 2024-06-03T09:06:57.882986+0000 2024-06-03T09:06:57.882 INFO:teuthology.orchestra.run.smithi088.stderr:min_mon_release 0 (unknown) 2024-06-03T09:06:57.882 INFO:teuthology.orchestra.run.smithi088.stderr:election_strategy: 3 2024-06-03T09:06:57.882 INFO:teuthology.orchestra.run.smithi088.stderr:0: v1:172.21.15.88:6789/0 mon.noname-a 2024-06-03T09:06:57.883 INFO:teuthology.orchestra.run.smithi088.stderr:1: v1:172.21.15.192:6789/0 mon.noname-c 2024-06-03T09:06:57.883 INFO:teuthology.orchestra.run.smithi088.stderr:2: v1:172.21.15.88:6790/0 mon.noname-b 2024-06-03T09:06:57.883 INFO:teuthology.orchestra.run.smithi088.stderr: 2024-06-03T09:06:57.884 INFO:teuthology.orchestra.run.smithi088.stderr:2024-06-03T09:06:57.882+0000 7f64bf173640 1 Processor -- start 2024-06-03T09:06:57.884 INFO:teuthology.orchestra.run.smithi088.stderr:2024-06-03T09:06:57.882+0000 7f64bf173640 1 -- start start 2024-06-03T09:06:57.884 INFO:teuthology.orchestra.run.smithi088.stderr:2024-06-03T09:06:57.882+0000 7f64bf173640 10 monclient: init 2024-06-03T09:06:57.884 INFO:teuthology.orchestra.run.smithi088.stderr:2024-06-03T09:06:57.883+0000 7f64bf173640 10 monclient: _reopen_session rank -1 2024-06-03T09:06:57.884 INFO:teuthology.orchestra.run.smithi088.stderr:2024-06-03T09:06:57.883+0000 7f64bf173640 10 monclient: _add_conns ranks=[1,2,0] 2024-06-03T09:06:57.884 INFO:teuthology.orchestra.run.smithi088.stderr:2024-06-03T09:06:57.883+0000 7f64bf173640 10 monclient(hunting): picked mon.noname-c con 0x7f64b80adb90 addr v1:172.21.15.192:6789/0 2024-06-03T09:06:57.884 INFO:teuthology.orchestra.run.smithi088.stderr:2024-06-03T09:06:57.883+0000 7f64bf173640 10 monclient(hunting): picked mon.noname-b con 0x7f64b806ee00 addr v1:172.21.15.88:6790/0 2024-06-03T09:06:57.884 INFO:teuthology.orchestra.run.smithi088.stderr:2024-06-03T09:06:57.883+0000 7f64bf173640 10 monclient(hunting): picked mon.noname-a con 0x7f64b81383e0 addr v1:172.21.15.88:6789/0 2024-06-03T09:06:57.885 INFO:teuthology.orchestra.run.smithi088.stderr:2024-06-03T09:06:57.883+0000 7f64bf173640 1 -- --> v1:172.21.15.88:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f64b806f2c0 con 0x7f64b81383e0 2024-06-03T09:06:57.885 INFO:teuthology.orchestra.run.smithi088.stderr:2024-06-03T09:06:57.883+0000 7f64bf173640 1 -- --> v1:172.21.15.192:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f64b80710c0 con 0x7f64b80adb90 2024-06-03T09:06:57.885 INFO:teuthology.orchestra.run.smithi088.stderr:2024-06-03T09:06:57.883+0000 7f64bf173640 1 -- --> v1:172.21.15.88:6790/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f64b8071fd0 con 0x7f64b806ee00 2024-06-03T09:06:57.885 INFO:teuthology.orchestra.run.smithi088.stderr:2024-06-03T09:06:57.883+0000 7f64bf173640 10 monclient(hunting): _renew_subs 2024-06-03T09:06:57.885 INFO:teuthology.orchestra.run.smithi088.stderr:2024-06-03T09:06:57.883+0000 7f64bf173640 10 monclient(hunting): authenticate will time out at 3073.693387s 2024-06-03T09:06:57.885 INFO:teuthology.orchestra.run.smithi088.stderr:2024-06-03T09:06:57.883+0000 7f64b7fff640 1 --1- >> v1:172.21.15.88:6790/0 conn(0x7f64b806ee00 0x7f64b8137cd0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.88:6790/0 says I am v1:172.21.15.88:50998/0 (socket says 172.21.15.88:50998) 2024-06-03T09:06:57.885 INFO:teuthology.orchestra.run.smithi088.stderr:2024-06-03T09:06:57.883+0000 7f64b7fff640 1 -- 172.21.15.88:0/1109563611 learned_addr learned my addr 172.21.15.88:0/1109563611 (peer_addr_for_me v1:172.21.15.88:0/0) 2024-06-03T09:06:57.886 INFO:teuthology.orchestra.run.smithi088.stderr:2024-06-03T09:06:57.886+0000 7f64b77fe640 1 -- 172.21.15.88:0/1109563611 <== mon.0 v1:172.21.15.88:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 594539188 0 0) 0x7f64b806f2c0 con 0x7f64b81383e0 2024-06-03T09:06:57.887 INFO:teuthology.orchestra.run.smithi088.stderr:2024-06-03T09:06:57.886+0000 7f64b77fe640 10 monclient(hunting): _negotiate 2024-06-03T09:06:57.887 INFO:teuthology.orchestra.run.smithi088.stderr:2024-06-03T09:06:57.886+0000 7f64b77fe640 10 monclient(hunting): _init_auth method 2 2024-06-03T09:06:57.887 INFO:teuthology.orchestra.run.smithi088.stderr:2024-06-03T09:06:57.886+0000 7f64b77fe640 10 monclient(hunting): _init_auth creating new auth 2024-06-03T09:06:57.887 INFO:teuthology.orchestra.run.smithi088.stderr:2024-06-03T09:06:57.886+0000 7f64b77fe640 10 monclient(hunting): my global_id is 30759 2024-06-03T09:06:57.887 INFO:teuthology.orchestra.run.smithi088.stderr:2024-06-03T09:06:57.886+0000 7f64b77fe640 1 -- 172.21.15.88:0/1109563611 --> v1:172.21.15.88:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f64a4005730 con 0x7f64b81383e0 2024-06-03T09:06:57.887 INFO:teuthology.orchestra.run.smithi088.stderr:2024-06-03T09:06:57.886+0000 7f64b77fe640 1 -- 172.21.15.88:0/1109563611 <== mon.1 v1:172.21.15.192:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 2635996119 0 0) 0x7f64b80710c0 con 0x7f64b80adb90 2024-06-03T09:06:57.887 INFO:teuthology.orchestra.run.smithi088.stderr:2024-06-03T09:06:57.886+0000 7f64b77fe640 10 monclient(hunting): _negotiate 2024-06-03T09:06:57.887 INFO:teuthology.orchestra.run.smithi088.stderr:2024-06-03T09:06:57.886+0000 7f64b77fe640 10 monclient(hunting): _init_auth method 2 2024-06-03T09:06:57.887 INFO:teuthology.orchestra.run.smithi088.stderr:2024-06-03T09:06:57.886+0000 7f64b77fe640 10 monclient(hunting): _init_auth creating new auth 2024-06-03T09:06:57.887 INFO:teuthology.orchestra.run.smithi088.stderr:2024-06-03T09:06:57.886+0000 7f64b77fe640 10 monclient(hunting): my global_id is 30331 2024-06-03T09:06:57.887 INFO:teuthology.orchestra.run.smithi088.stderr:2024-06-03T09:06:57.886+0000 7f64b77fe640 1 -- 172.21.15.88:0/1109563611 --> v1:172.21.15.192:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f64b806f2c0 con 0x7f64b80adb90 2024-06-03T09:06:57.887 INFO:teuthology.orchestra.run.smithi088.stderr:2024-06-03T09:06:57.887+0000 7f64b77fe640 1 -- 172.21.15.88:0/1109563611 <== mon.2 v1:172.21.15.88:6790/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 3406834360 0 0) 0x7f64b8071fd0 con 0x7f64b806ee00 2024-06-03T09:06:57.887 INFO:teuthology.orchestra.run.smithi088.stderr:2024-06-03T09:06:57.887+0000 7f64b77fe640 10 monclient(hunting): _negotiate 2024-06-03T09:06:57.888 INFO:teuthology.orchestra.run.smithi088.stderr:2024-06-03T09:06:57.887+0000 7f64b77fe640 10 monclient(hunting): _init_auth method 2 2024-06-03T09:06:57.888 INFO:teuthology.orchestra.run.smithi088.stderr:2024-06-03T09:06:57.887+0000 7f64b77fe640 10 monclient(hunting): _init_auth creating new auth 2024-06-03T09:06:57.888 INFO:teuthology.orchestra.run.smithi088.stderr:2024-06-03T09:06:57.887+0000 7f64b77fe640 10 monclient(hunting): my global_id is 30266 2024-06-03T09:06:57.888 INFO:teuthology.orchestra.run.smithi088.stderr:2024-06-03T09:06:57.887+0000 7f64b77fe640 1 -- 172.21.15.88:0/1109563611 --> v1:172.21.15.88:6790/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f64b80710c0 con 0x7f64b806ee00 2024-06-03T09:06:57.890 INFO:teuthology.orchestra.run.smithi088.stderr:2024-06-03T09:06:57.889+0000 7f64b77fe640 1 -- 172.21.15.88:0/1109563611 <== mon.1 v1:172.21.15.192:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 214+0+0 (unknown 311938474 0 0) 0x7f64b806f2c0 con 0x7f64b80adb90 2024-06-03T09:06:57.890 INFO:teuthology.orchestra.run.smithi088.stderr:2024-06-03T09:06:57.889+0000 7f64b77fe640 1 -- 172.21.15.88:0/1109563611 >> v1:172.21.15.88:6790/0 conn(0x7f64b806ee00 legacy=0x7f64b8137cd0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-06-03T09:06:57.890 INFO:teuthology.orchestra.run.smithi088.stderr:2024-06-03T09:06:57.889+0000 7f64b77fe640 1 -- 172.21.15.88:0/1109563611 >> v1:172.21.15.88:6789/0 conn(0x7f64b81383e0 legacy=0x7f64b8149af0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-06-03T09:06:57.890 INFO:teuthology.orchestra.run.smithi088.stderr:2024-06-03T09:06:57.889+0000 7f64b77fe640 10 monclient: _finish_hunting 0 2024-06-03T09:06:57.890 INFO:teuthology.orchestra.run.smithi088.stderr:2024-06-03T09:06:57.889+0000 7f64b77fe640 1 monclient: found mon.noname-c 2024-06-03T09:06:57.890 INFO:teuthology.orchestra.run.smithi088.stderr:2024-06-03T09:06:57.889+0000 7f64b77fe640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-06-03T09:06:57.890 INFO:teuthology.orchestra.run.smithi088.stderr:2024-06-03T09:06:57.889+0000 7f64b77fe640 10 monclient: _send_mon_message to mon.noname-c at v1:172.21.15.192:6789/0 2024-06-03T09:06:57.890 INFO:teuthology.orchestra.run.smithi088.stderr:2024-06-03T09:06:57.889+0000 7f64b77fe640 1 -- 172.21.15.88:0/1109563611 --> v1:172.21.15.192:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f64b8141750 con 0x7f64b80adb90 2024-06-03T09:06:57.890 INFO:teuthology.orchestra.run.smithi088.stderr:2024-06-03T09:06:57.889+0000 7f64b77fe640 10 monclient: _finish_auth 0 2024-06-03T09:06:57.890 INFO:teuthology.orchestra.run.smithi088.stderr:2024-06-03T09:06:57.889+0000 7f64b77fe640 10 monclient: _check_auth_tickets 2024-06-03T09:06:57.890 INFO:teuthology.orchestra.run.smithi088.stderr:2024-06-03T09:06:57.889+0000 7f64b77fe640 20 monclient: _check_auth_rotating not needed by client.admin 2024-06-03T09:06:57.890 INFO:teuthology.orchestra.run.smithi088.stderr:2024-06-03T09:06:57.889+0000 7f64b77fe640 1 -- 172.21.15.88:0/1109563611 <== mon.1 v1:172.21.15.192:6789/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 2234374065 0 0) 0x7f64b00015f0 con 0x7f64b80adb90 2024-06-03T09:06:57.891 INFO:teuthology.orchestra.run.smithi088.stderr:2024-06-03T09:06:57.889+0000 7f64b77fe640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-06-03T09:06:57.891 INFO:teuthology.orchestra.run.smithi088.stderr:2024-06-03T09:06:57.889+0000 7f64b77fe640 10 monclient: got monmap 1 from mon.noname-c (according to old e1) 2024-06-03T09:06:57.894 INFO:teuthology.orchestra.run.smithi088.stderr:2024-06-03T09:06:57.889+0000 7f64b77fe640 10 monclient: dump: 2024-06-03T09:06:57.894 INFO:teuthology.orchestra.run.smithi088.stderr:epoch 1 2024-06-03T09:06:57.894 INFO:teuthology.orchestra.run.smithi088.stderr:fsid ec3c3fdd-3ac6-4d7d-9412-683130ed8b04 2024-06-03T09:06:57.894 INFO:teuthology.orchestra.run.smithi088.stderr:last_changed 2024-06-03T08:34:31.885125+0000 2024-06-03T09:06:57.894 INFO:teuthology.orchestra.run.smithi088.stderr:created 2024-06-03T08:34:31.885125+0000 2024-06-03T09:06:57.894 INFO:teuthology.orchestra.run.smithi088.stderr:min_mon_release 18 (reef) 2024-06-03T09:06:57.895 INFO:teuthology.orchestra.run.smithi088.stderr:election_strategy: 3 2024-06-03T09:06:57.895 INFO:teuthology.orchestra.run.smithi088.stderr:0: v1:172.21.15.88:6789/0 mon.a 2024-06-03T09:06:57.895 INFO:teuthology.orchestra.run.smithi088.stderr:1: v1:172.21.15.192:6789/0 mon.b 2024-06-03T09:06:57.895 INFO:teuthology.orchestra.run.smithi088.stderr:2: v1:172.21.15.88:6790/0 mon.c 2024-06-03T09:06:57.895 INFO:teuthology.orchestra.run.smithi088.stderr: 2024-06-03T09:06:57.895 INFO:teuthology.orchestra.run.smithi088.stderr:2024-06-03T09:06:57.889+0000 7f64bf173640 5 monclient: authenticate success, global_id 30331 2024-06-03T09:06:57.895 INFO:teuthology.orchestra.run.smithi088.stderr:2024-06-03T09:06:57.889+0000 7f64bf173640 20 monclient: get_monmap_and_config waiting for monmap|config 2024-06-03T09:06:57.898 INFO:teuthology.orchestra.run.smithi088.stderr:2024-06-03T09:06:57.891+0000 7f64b77fe640 1 -- 172.21.15.88:0/1109563611 <== mon.1 v1:172.21.15.192:6789/0 4 ==== config(0 keys) v1 ==== 4+0+0 (unknown 0 0 0) 0x7f64b0001810 con 0x7f64b80adb90 2024-06-03T09:06:57.898 INFO:teuthology.orchestra.run.smithi088.stderr:2024-06-03T09:06:57.891+0000 7f64b77fe640 10 monclient: handle_config config(0 keys) v1 2024-06-03T09:06:57.899 INFO:teuthology.orchestra.run.smithi088.stderr:2024-06-03T09:06:57.891+0000 7f64b77fe640 1 -- 172.21.15.88:0/1109563611 <== mon.1 v1:172.21.15.192:6789/0 5 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 2234374065 0 0) 0x7f64b0001be0 con 0x7f64b80adb90 2024-06-03T09:06:57.899 INFO:teuthology.orchestra.run.smithi088.stderr:2024-06-03T09:06:57.891+0000 7f64bf173640 10 monclient: get_monmap_and_config success 2024-06-03T09:06:57.899 INFO:teuthology.orchestra.run.smithi088.stderr:2024-06-03T09:06:57.891+0000 7f64bf173640 4 set_mon_vals no callback set 2024-06-03T09:06:57.899 INFO:teuthology.orchestra.run.smithi088.stderr:2024-06-03T09:06:57.891+0000 7f64b77fe640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-06-03T09:06:57.899 INFO:teuthology.orchestra.run.smithi088.stderr:2024-06-03T09:06:57.891+0000 7f64b77fe640 10 monclient: got monmap 1 from mon.b (according to old e1) 2024-06-03T09:06:57.899 INFO:teuthology.orchestra.run.smithi088.stderr:2024-06-03T09:06:57.891+0000 7f64b77fe640 10 monclient: dump: 2024-06-03T09:06:57.899 INFO:teuthology.orchestra.run.smithi088.stderr:epoch 1 2024-06-03T09:06:57.899 INFO:teuthology.orchestra.run.smithi088.stderr:fsid ec3c3fdd-3ac6-4d7d-9412-683130ed8b04 2024-06-03T09:06:57.899 INFO:teuthology.orchestra.run.smithi088.stderr:last_changed 2024-06-03T08:34:31.885125+0000 2024-06-03T09:06:57.899 INFO:teuthology.orchestra.run.smithi088.stderr:created 2024-06-03T08:34:31.885125+0000 2024-06-03T09:06:57.899 INFO:teuthology.orchestra.run.smithi088.stderr:min_mon_release 18 (reef) 2024-06-03T09:06:57.899 INFO:teuthology.orchestra.run.smithi088.stderr:election_strategy: 3 2024-06-03T09:06:57.899 INFO:teuthology.orchestra.run.smithi088.stderr:0: v1:172.21.15.88:6789/0 mon.a 2024-06-03T09:06:57.899 INFO:teuthology.orchestra.run.smithi088.stderr:1: v1:172.21.15.192:6789/0 mon.b 2024-06-03T09:06:57.899 INFO:teuthology.orchestra.run.smithi088.stderr:2: v1:172.21.15.88:6790/0 mon.c 2024-06-03T09:06:57.900 INFO:teuthology.orchestra.run.smithi088.stderr: 2024-06-03T09:06:57.900 INFO:teuthology.orchestra.run.smithi088.stderr:2024-06-03T09:06:57.892+0000 7f64bf173640 10 monclient: shutdown 2024-06-03T09:06:57.900 INFO:teuthology.orchestra.run.smithi088.stderr:2024-06-03T09:06:57.892+0000 7f64bf173640 20 monclient: shutdown discarding 0 pending message(s) 2024-06-03T09:06:57.900 INFO:teuthology.orchestra.run.smithi088.stderr:2024-06-03T09:06:57.892+0000 7f64bf173640 1 -- 172.21.15.88:0/1109563611 >> v1:172.21.15.192:6789/0 conn(0x7f64b80adb90 legacy=0x7f64b80718c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-06-03T09:06:57.900 INFO:teuthology.orchestra.run.smithi088.stderr:2024-06-03T09:06:57.892+0000 7f64bf173640 1 -- 172.21.15.88:0/1109563611 shutdown_connections 2024-06-03T09:06:57.900 INFO:teuthology.orchestra.run.smithi088.stderr:2024-06-03T09:06:57.892+0000 7f64bf173640 1 -- 172.21.15.88:0/1109563611 >> 172.21.15.88:0/1109563611 conn(0x7f64b80b2f20 msgr2=0x7f64b80b3300 unknown :-1 s=STATE_NONE l=0).mark_down 2024-06-03T09:06:57.900 INFO:teuthology.orchestra.run.smithi088.stderr:2024-06-03T09:06:57.892+0000 7f64bf173640 1 -- 172.21.15.88:0/1109563611 shutdown_connections 2024-06-03T09:06:57.900 INFO:teuthology.orchestra.run.smithi088.stderr:2024-06-03T09:06:57.892+0000 7f64bf173640 1 -- 172.21.15.88:0/1109563611 wait complete. 2024-06-03T09:06:57.900 INFO:teuthology.orchestra.run.smithi088.stderr:2024-06-03T09:06:57.892+0000 7f64bf173640 10 monclient: build_initial_monmap 2024-06-03T09:06:57.900 INFO:teuthology.orchestra.run.smithi088.stderr:2024-06-03T09:06:57.892+0000 7f64bf173640 10 monclient: monmap: 2024-06-03T09:06:57.900 INFO:teuthology.orchestra.run.smithi088.stderr:epoch 0 2024-06-03T09:06:57.900 INFO:teuthology.orchestra.run.smithi088.stderr:fsid 00000000-0000-0000-0000-000000000000 2024-06-03T09:06:57.900 INFO:teuthology.orchestra.run.smithi088.stderr:last_changed 0.000000 2024-06-03T09:06:57.900 INFO:teuthology.orchestra.run.smithi088.stderr:created 0.000000 2024-06-03T09:06:57.901 INFO:teuthology.orchestra.run.smithi088.stderr:min_mon_release 0 (unknown) 2024-06-03T09:06:57.901 INFO:teuthology.orchestra.run.smithi088.stderr:election_strategy: 1 2024-06-03T09:06:57.901 INFO:teuthology.orchestra.run.smithi088.stderr:0: v1:172.21.15.88:6789/0 mon.noname-a 2024-06-03T09:06:57.901 INFO:teuthology.orchestra.run.smithi088.stderr:1: v1:172.21.15.192:6789/0 mon.noname-b 2024-06-03T09:06:57.901 INFO:teuthology.orchestra.run.smithi088.stderr:2: v1:172.21.15.88:6790/0 mon.noname-c 2024-06-03T09:06:57.901 INFO:teuthology.orchestra.run.smithi088.stderr: 2024-06-03T09:06:57.901 INFO:teuthology.orchestra.run.smithi088.stderr:2024-06-03T09:06:57.893+0000 7f64bf173640 1 librados: starting msgr at 2024-06-03T09:06:57.901 INFO:teuthology.orchestra.run.smithi088.stderr:2024-06-03T09:06:57.893+0000 7f64bf173640 1 librados: starting objecter 2024-06-03T09:06:57.901 INFO:teuthology.orchestra.run.smithi088.stderr:2024-06-03T09:06:57.893+0000 7f64bf173640 1 Processor -- start 2024-06-03T09:06:57.901 INFO:teuthology.orchestra.run.smithi088.stderr:2024-06-03T09:06:57.893+0000 7f64bf173640 1 -- start start 2024-06-03T09:06:57.901 INFO:teuthology.orchestra.run.smithi088.stderr:2024-06-03T09:06:57.893+0000 7f64bf173640 1 librados: setting wanted keys 2024-06-03T09:06:57.901 INFO:teuthology.orchestra.run.smithi088.stderr:2024-06-03T09:06:57.893+0000 7f64bf173640 1 librados: calling monclient init 2024-06-03T09:06:57.901 INFO:teuthology.orchestra.run.smithi088.stderr:2024-06-03T09:06:57.893+0000 7f64bf173640 10 monclient: init 2024-06-03T09:06:57.902 INFO:teuthology.orchestra.run.smithi088.stderr:2024-06-03T09:06:57.894+0000 7f64bf173640 10 monclient: _reopen_session rank -1 2024-06-03T09:06:57.902 INFO:teuthology.orchestra.run.smithi088.stderr:2024-06-03T09:06:57.894+0000 7f64bf173640 10 monclient: _add_conns ranks=[2,0,1] 2024-06-03T09:06:57.902 INFO:teuthology.orchestra.run.smithi088.stderr:2024-06-03T09:06:57.894+0000 7f64bf173640 10 monclient(hunting): picked mon.noname-c con 0x7f64b806ee00 addr v1:172.21.15.88:6790/0 2024-06-03T09:06:57.902 INFO:teuthology.orchestra.run.smithi088.stderr:2024-06-03T09:06:57.894+0000 7f64bf173640 10 monclient(hunting): picked mon.noname-a con 0x7f64b80adb90 addr v1:172.21.15.88:6789/0 2024-06-03T09:06:57.902 INFO:teuthology.orchestra.run.smithi088.stderr:2024-06-03T09:06:57.894+0000 7f64bf173640 10 monclient(hunting): picked mon.noname-b con 0x7f64b81383e0 addr v1:172.21.15.192:6789/0 2024-06-03T09:06:57.902 INFO:teuthology.orchestra.run.smithi088.stderr:2024-06-03T09:06:57.894+0000 7f64bf173640 1 -- --> v1:172.21.15.88:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f64b813ef40 con 0x7f64b80adb90 2024-06-03T09:06:57.902 INFO:teuthology.orchestra.run.smithi088.stderr:2024-06-03T09:06:57.894+0000 7f64bf173640 1 -- --> v1:172.21.15.192:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f64b811ca30 con 0x7f64b81383e0 2024-06-03T09:06:57.902 INFO:teuthology.orchestra.run.smithi088.stderr:2024-06-03T09:06:57.894+0000 7f64bf173640 1 -- --> v1:172.21.15.88:6790/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f64b81da550 con 0x7f64b806ee00 2024-06-03T09:06:57.903 INFO:teuthology.orchestra.run.smithi088.stderr:2024-06-03T09:06:57.894+0000 7f64bf173640 10 monclient(hunting): _renew_subs 2024-06-03T09:06:57.903 INFO:teuthology.orchestra.run.smithi088.stderr:2024-06-03T09:06:57.894+0000 7f64bf173640 10 monclient(hunting): authenticate will time out at 3073.704085s 2024-06-03T09:06:57.903 INFO:teuthology.orchestra.run.smithi088.stderr:2024-06-03T09:06:57.894+0000 7f64bcf0f640 1 --1- >> v1:172.21.15.88:6790/0 conn(0x7f64b806ee00 0x7f64b81d9e40 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=1).handle_server_banner_and_identify peer v1:172.21.15.88:6790/0 says I am v1:172.21.15.88:51002/0 (socket says 172.21.15.88:51002) 2024-06-03T09:06:57.903 INFO:teuthology.orchestra.run.smithi088.stderr:2024-06-03T09:06:57.894+0000 7f64bcf0f640 1 -- 172.21.15.88:0/1662221529 learned_addr learned my addr 172.21.15.88:0/1662221529 (peer_addr_for_me v1:172.21.15.88:0/0) 2024-06-03T09:06:57.903 INFO:teuthology.orchestra.run.smithi088.stderr:2024-06-03T09:06:57.894+0000 7f64b57fa640 10 client.?.objecter ms_handle_connect 0x7f64b806ee00 2024-06-03T09:06:57.903 INFO:teuthology.orchestra.run.smithi088.stderr:2024-06-03T09:06:57.894+0000 7f64b57fa640 10 client.?.objecter resend_mon_ops 2024-06-03T09:06:57.903 INFO:teuthology.orchestra.run.smithi088.stderr:2024-06-03T09:06:57.895+0000 7f64b57fa640 10 client.?.objecter ms_handle_connect 0x7f64b81383e0 2024-06-03T09:06:57.903 INFO:teuthology.orchestra.run.smithi088.stderr:2024-06-03T09:06:57.895+0000 7f64b57fa640 10 client.?.objecter resend_mon_ops 2024-06-03T09:06:57.903 INFO:teuthology.orchestra.run.smithi088.stderr:2024-06-03T09:06:57.895+0000 7f64b57fa640 10 client.?.objecter ms_handle_connect 0x7f64b80adb90 2024-06-03T09:06:57.903 INFO:teuthology.orchestra.run.smithi088.stderr:2024-06-03T09:06:57.895+0000 7f64b57fa640 10 client.?.objecter resend_mon_ops 2024-06-03T09:06:57.903 INFO:teuthology.orchestra.run.smithi088.stderr:2024-06-03T09:06:57.896+0000 7f64b57fa640 1 -- 172.21.15.88:0/1662221529 <== mon.1 v1:172.21.15.192:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 3222291670 0 0) 0x7f64b811ca30 con 0x7f64b81383e0 2024-06-03T09:06:57.903 INFO:teuthology.orchestra.run.smithi088.stderr:2024-06-03T09:06:57.896+0000 7f64b57fa640 10 monclient(hunting): _negotiate 2024-06-03T09:06:57.903 INFO:teuthology.orchestra.run.smithi088.stderr:2024-06-03T09:06:57.896+0000 7f64b57fa640 10 monclient(hunting): _init_auth method 2 2024-06-03T09:06:57.903 INFO:teuthology.orchestra.run.smithi088.stderr:2024-06-03T09:06:57.896+0000 7f64b57fa640 10 monclient(hunting): _init_auth creating new auth 2024-06-03T09:06:57.904 INFO:teuthology.orchestra.run.smithi088.stderr:2024-06-03T09:06:57.896+0000 7f64b57fa640 10 monclient(hunting): my global_id is 30334 2024-06-03T09:06:57.904 INFO:teuthology.orchestra.run.smithi088.stderr:2024-06-03T09:06:57.896+0000 7f64b57fa640 1 -- 172.21.15.88:0/1662221529 --> v1:172.21.15.192:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f6498004eb0 con 0x7f64b81383e0 2024-06-03T09:06:57.904 INFO:teuthology.orchestra.run.smithi088.stderr:2024-06-03T09:06:57.897+0000 7f64b57fa640 1 -- 172.21.15.88:0/1662221529 <== mon.2 v1:172.21.15.88:6790/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 3976620634 0 0) 0x7f64b81da550 con 0x7f64b806ee00 2024-06-03T09:06:57.904 INFO:teuthology.orchestra.run.smithi088.stderr:2024-06-03T09:06:57.897+0000 7f64b57fa640 10 monclient(hunting): _negotiate 2024-06-03T09:06:57.904 INFO:teuthology.orchestra.run.smithi088.stderr:2024-06-03T09:06:57.897+0000 7f64b57fa640 10 monclient(hunting): _init_auth method 2 2024-06-03T09:06:57.904 INFO:teuthology.orchestra.run.smithi088.stderr:2024-06-03T09:06:57.897+0000 7f64b57fa640 10 monclient(hunting): _init_auth creating new auth 2024-06-03T09:06:57.904 INFO:teuthology.orchestra.run.smithi088.stderr:2024-06-03T09:06:57.897+0000 7f64b57fa640 10 monclient(hunting): my global_id is 30269 2024-06-03T09:06:57.904 INFO:teuthology.orchestra.run.smithi088.stderr:2024-06-03T09:06:57.897+0000 7f64b57fa640 1 -- 172.21.15.88:0/1662221529 --> v1:172.21.15.88:6790/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f64b811ca30 con 0x7f64b806ee00 2024-06-03T09:06:57.904 INFO:teuthology.orchestra.run.smithi088.stderr:2024-06-03T09:06:57.897+0000 7f64b57fa640 1 -- 172.21.15.88:0/1662221529 <== mon.0 v1:172.21.15.88:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 3598398884 0 0) 0x7f64b813ef40 con 0x7f64b80adb90 2024-06-03T09:06:57.904 INFO:teuthology.orchestra.run.smithi088.stderr:2024-06-03T09:06:57.897+0000 7f64b57fa640 10 monclient(hunting): _negotiate 2024-06-03T09:06:57.904 INFO:teuthology.orchestra.run.smithi088.stderr:2024-06-03T09:06:57.897+0000 7f64b57fa640 10 monclient(hunting): _init_auth method 2 2024-06-03T09:06:57.904 INFO:teuthology.orchestra.run.smithi088.stderr:2024-06-03T09:06:57.897+0000 7f64b57fa640 10 monclient(hunting): _init_auth creating new auth 2024-06-03T09:06:57.904 INFO:teuthology.orchestra.run.smithi088.stderr:2024-06-03T09:06:57.897+0000 7f64b57fa640 10 monclient(hunting): my global_id is 30762 2024-06-03T09:06:57.904 INFO:teuthology.orchestra.run.smithi088.stderr:2024-06-03T09:06:57.897+0000 7f64b57fa640 1 -- 172.21.15.88:0/1662221529 --> v1:172.21.15.88:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f64b81da550 con 0x7f64b80adb90 2024-06-03T09:06:57.907 INFO:teuthology.orchestra.run.smithi088.stderr:2024-06-03T09:06:57.901+0000 7f64b57fa640 1 -- 172.21.15.88:0/1662221529 <== mon.2 v1:172.21.15.88:6790/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 764+0+0 (unknown 1400257402 0 0) 0x7f64b811ca30 con 0x7f64b806ee00 2024-06-03T09:06:57.907 INFO:teuthology.orchestra.run.smithi088.stderr:2024-06-03T09:06:57.901+0000 7f64b57fa640 1 -- 172.21.15.88:0/1662221529 --> v1:172.21.15.88:6790/0 -- auth(proto 2 165 bytes epoch 0) v1 -- 0x7f64b813ef40 con 0x7f64b806ee00 2024-06-03T09:06:57.907 INFO:teuthology.orchestra.run.smithi088.stderr:2024-06-03T09:06:57.901+0000 7f64b57fa640 1 -- 172.21.15.88:0/1662221529 <== mon.2 v1:172.21.15.88:6790/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 2234374065 0 0) 0x7f64b00015b0 con 0x7f64b806ee00 2024-06-03T09:06:57.907 INFO:teuthology.orchestra.run.smithi088.stderr:2024-06-03T09:06:57.901+0000 7f64b57fa640 10 monclient(hunting): handle_monmap mon_map magic: 0 v1 2024-06-03T09:06:57.907 INFO:teuthology.orchestra.run.smithi088.stderr:2024-06-03T09:06:57.901+0000 7f64b57fa640 10 monclient(hunting): got monmap 1 from mon.noname-c (according to old e1) 2024-06-03T09:06:57.907 INFO:teuthology.orchestra.run.smithi088.stderr:2024-06-03T09:06:57.901+0000 7f64b57fa640 10 monclient(hunting): dump: 2024-06-03T09:06:57.907 INFO:teuthology.orchestra.run.smithi088.stderr:epoch 1 2024-06-03T09:06:57.908 INFO:teuthology.orchestra.run.smithi088.stderr:fsid ec3c3fdd-3ac6-4d7d-9412-683130ed8b04 2024-06-03T09:06:57.908 INFO:teuthology.orchestra.run.smithi088.stderr:last_changed 2024-06-03T08:34:31.885125+0000 2024-06-03T09:06:57.908 INFO:teuthology.orchestra.run.smithi088.stderr:created 2024-06-03T08:34:31.885125+0000 2024-06-03T09:06:57.908 INFO:teuthology.orchestra.run.smithi088.stderr:min_mon_release 18 (reef) 2024-06-03T09:06:57.908 INFO:teuthology.orchestra.run.smithi088.stderr:election_strategy: 3 2024-06-03T09:06:57.908 INFO:teuthology.orchestra.run.smithi088.stderr:0: v1:172.21.15.88:6789/0 mon.a 2024-06-03T09:06:57.908 INFO:teuthology.orchestra.run.smithi088.stderr:1: v1:172.21.15.192:6789/0 mon.b 2024-06-03T09:06:57.908 INFO:teuthology.orchestra.run.smithi088.stderr:2: v1:172.21.15.88:6790/0 mon.c 2024-06-03T09:06:57.908 INFO:teuthology.orchestra.run.smithi088.stderr: 2024-06-03T09:06:57.908 INFO:teuthology.orchestra.run.smithi088.stderr:2024-06-03T09:06:57.901+0000 7f64b57fa640 10 monclient(hunting): _finish_auth 0 2024-06-03T09:06:57.909 INFO:teuthology.orchestra.run.smithi088.stderr:2024-06-03T09:06:57.902+0000 7f64b57fa640 1 -- 172.21.15.88:0/1662221529 <== mon.0 v1:172.21.15.88:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 764+0+0 (unknown 521834298 0 0) 0x7f64b81da550 con 0x7f64b80adb90 2024-06-03T09:06:57.909 INFO:teuthology.orchestra.run.smithi088.stderr:2024-06-03T09:06:57.902+0000 7f64b57fa640 1 -- 172.21.15.88:0/1662221529 --> v1:172.21.15.88:6789/0 -- auth(proto 2 165 bytes epoch 0) v1 -- 0x7f64b811ca30 con 0x7f64b80adb90 2024-06-03T09:06:57.909 INFO:teuthology.orchestra.run.smithi088.stderr:2024-06-03T09:06:57.902+0000 7f64b57fa640 1 -- 172.21.15.88:0/1662221529 <== mon.0 v1:172.21.15.88:6789/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 2234374065 0 0) 0x7f64a8001bf0 con 0x7f64b80adb90 2024-06-03T09:06:57.909 INFO:teuthology.orchestra.run.smithi088.stderr:2024-06-03T09:06:57.902+0000 7f64b57fa640 10 monclient(hunting): handle_monmap mon_map magic: 0 v1 2024-06-03T09:06:57.909 INFO:teuthology.orchestra.run.smithi088.stderr:2024-06-03T09:06:57.902+0000 7f64b57fa640 10 monclient(hunting): got monmap 1 from mon.a (according to old e1) 2024-06-03T09:06:57.909 INFO:teuthology.orchestra.run.smithi088.stderr:2024-06-03T09:06:57.902+0000 7f64b57fa640 10 monclient(hunting): dump: 2024-06-03T09:06:57.909 INFO:teuthology.orchestra.run.smithi088.stderr:epoch 1 2024-06-03T09:06:57.909 INFO:teuthology.orchestra.run.smithi088.stderr:fsid ec3c3fdd-3ac6-4d7d-9412-683130ed8b04 2024-06-03T09:06:57.909 INFO:teuthology.orchestra.run.smithi088.stderr:last_changed 2024-06-03T08:34:31.885125+0000 2024-06-03T09:06:57.909 INFO:teuthology.orchestra.run.smithi088.stderr:created 2024-06-03T08:34:31.885125+0000 2024-06-03T09:06:57.910 INFO:teuthology.orchestra.run.smithi088.stderr:min_mon_release 18 (reef) 2024-06-03T09:06:57.910 INFO:teuthology.orchestra.run.smithi088.stderr:election_strategy: 3 2024-06-03T09:06:57.910 INFO:teuthology.orchestra.run.smithi088.stderr:0: v1:172.21.15.88:6789/0 mon.a 2024-06-03T09:06:57.910 INFO:teuthology.orchestra.run.smithi088.stderr:1: v1:172.21.15.192:6789/0 mon.b 2024-06-03T09:06:57.910 INFO:teuthology.orchestra.run.smithi088.stderr:2: v1:172.21.15.88:6790/0 mon.c 2024-06-03T09:06:57.910 INFO:teuthology.orchestra.run.smithi088.stderr: 2024-06-03T09:06:57.910 INFO:teuthology.orchestra.run.smithi088.stderr:2024-06-03T09:06:57.902+0000 7f64b57fa640 1 -- 172.21.15.88:0/1662221529 <== mon.1 v1:172.21.15.192:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 764+0+0 (unknown 2693182249 0 0) 0x7f6498004eb0 con 0x7f64b81383e0 2024-06-03T09:06:57.910 INFO:teuthology.orchestra.run.smithi088.stderr:2024-06-03T09:06:57.902+0000 7f64b57fa640 1 -- 172.21.15.88:0/1662221529 --> v1:172.21.15.192:6789/0 -- auth(proto 2 165 bytes epoch 0) v1 -- 0x7f64b81da550 con 0x7f64b81383e0 2024-06-03T09:06:57.910 INFO:teuthology.orchestra.run.smithi088.stderr:2024-06-03T09:06:57.902+0000 7f64b57fa640 1 -- 172.21.15.88:0/1662221529 <== mon.1 v1:172.21.15.192:6789/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 2234374065 0 0) 0x7f64a00017d0 con 0x7f64b81383e0 2024-06-03T09:06:57.910 INFO:teuthology.orchestra.run.smithi088.stderr:2024-06-03T09:06:57.902+0000 7f64b57fa640 10 monclient(hunting): handle_monmap mon_map magic: 0 v1 2024-06-03T09:06:57.910 INFO:teuthology.orchestra.run.smithi088.stderr:2024-06-03T09:06:57.902+0000 7f64b57fa640 10 monclient(hunting): got monmap 1 from mon.b (according to old e1) 2024-06-03T09:06:57.911 INFO:teuthology.orchestra.run.smithi088.stderr:2024-06-03T09:06:57.902+0000 7f64b57fa640 10 monclient(hunting): dump: 2024-06-03T09:06:57.911 INFO:teuthology.orchestra.run.smithi088.stderr:epoch 1 2024-06-03T09:06:57.911 INFO:teuthology.orchestra.run.smithi088.stderr:fsid ec3c3fdd-3ac6-4d7d-9412-683130ed8b04 2024-06-03T09:06:57.911 INFO:teuthology.orchestra.run.smithi088.stderr:last_changed 2024-06-03T08:34:31.885125+0000 2024-06-03T09:06:57.911 INFO:teuthology.orchestra.run.smithi088.stderr:created 2024-06-03T08:34:31.885125+0000 2024-06-03T09:06:57.911 INFO:teuthology.orchestra.run.smithi088.stderr:min_mon_release 18 (reef) 2024-06-03T09:06:57.911 INFO:teuthology.orchestra.run.smithi088.stderr:election_strategy: 3 2024-06-03T09:06:57.911 INFO:teuthology.orchestra.run.smithi088.stderr:0: v1:172.21.15.88:6789/0 mon.a 2024-06-03T09:06:57.911 INFO:teuthology.orchestra.run.smithi088.stderr:1: v1:172.21.15.192:6789/0 mon.b 2024-06-03T09:06:57.911 INFO:teuthology.orchestra.run.smithi088.stderr:2: v1:172.21.15.88:6790/0 mon.c 2024-06-03T09:06:57.912 INFO:teuthology.orchestra.run.smithi088.stderr: 2024-06-03T09:06:57.912 INFO:teuthology.orchestra.run.smithi088.stderr:2024-06-03T09:06:57.904+0000 7f64b57fa640 1 -- 172.21.15.88:0/1662221529 <== mon.0 v1:172.21.15.88:6789/0 4 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 580+0+0 (unknown 3928286611 0 0) 0x7f64b811ca30 con 0x7f64b80adb90 2024-06-03T09:06:57.912 INFO:teuthology.orchestra.run.smithi088.stderr:2024-06-03T09:06:57.904+0000 7f64b57fa640 1 -- 172.21.15.88:0/1662221529 >> v1:172.21.15.88:6790/0 conn(0x7f64b806ee00 legacy=0x7f64b81d9e40 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-06-03T09:06:57.912 INFO:teuthology.orchestra.run.smithi088.stderr:2024-06-03T09:06:57.904+0000 7f64b57fa640 1 -- 172.21.15.88:0/1662221529 >> v1:172.21.15.192:6789/0 conn(0x7f64b81383e0 legacy=0x7f64b813f3d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-06-03T09:06:57.912 INFO:teuthology.orchestra.run.smithi088.stderr:2024-06-03T09:06:57.904+0000 7f64b57fa640 10 monclient: _finish_hunting 0 2024-06-03T09:06:57.912 INFO:teuthology.orchestra.run.smithi088.stderr:2024-06-03T09:06:57.904+0000 7f64b57fa640 1 monclient: found mon.a 2024-06-03T09:06:57.912 INFO:teuthology.orchestra.run.smithi088.stderr:2024-06-03T09:06:57.904+0000 7f64b57fa640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-06-03T09:06:57.912 INFO:teuthology.orchestra.run.smithi088.stderr:2024-06-03T09:06:57.904+0000 7f64b57fa640 10 monclient: _send_mon_message to mon.a at v1:172.21.15.88:6789/0 2024-06-03T09:06:57.912 INFO:teuthology.orchestra.run.smithi088.stderr:2024-06-03T09:06:57.904+0000 7f64b57fa640 1 -- 172.21.15.88:0/1662221529 --> v1:172.21.15.88:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f64b81da6e0 con 0x7f64b80adb90 2024-06-03T09:06:57.912 INFO:teuthology.orchestra.run.smithi088.stderr:2024-06-03T09:06:57.904+0000 7f64b57fa640 10 monclient: _finish_auth 0 2024-06-03T09:06:57.912 INFO:teuthology.orchestra.run.smithi088.stderr:2024-06-03T09:06:57.904+0000 7f64b57fa640 10 monclient: _check_auth_tickets 2024-06-03T09:06:57.913 INFO:teuthology.orchestra.run.smithi088.stderr:2024-06-03T09:06:57.904+0000 7f64b57fa640 20 monclient: _check_auth_rotating not needed by client.admin 2024-06-03T09:06:57.913 INFO:teuthology.orchestra.run.smithi088.stderr:2024-06-03T09:06:57.905+0000 7f64bf173640 5 monclient: authenticate success, global_id 30762 2024-06-03T09:06:57.913 INFO:teuthology.orchestra.run.smithi088.stderr:2024-06-03T09:06:57.905+0000 7f64bf173640 10 monclient: _renew_subs 2024-06-03T09:06:57.913 INFO:teuthology.orchestra.run.smithi088.stderr:2024-06-03T09:06:57.905+0000 7f64bf173640 10 monclient: _send_mon_message to mon.a at v1:172.21.15.88:6789/0 2024-06-03T09:06:57.913 INFO:teuthology.orchestra.run.smithi088.stderr:2024-06-03T09:06:57.905+0000 7f64bf173640 1 -- 172.21.15.88:0/1662221529 --> v1:172.21.15.88:6789/0 -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f64b80acad0 con 0x7f64b80adb90 2024-06-03T09:06:57.913 INFO:teuthology.orchestra.run.smithi088.stderr:2024-06-03T09:06:57.905+0000 7f64bf173640 10 client.30762.objecter _maybe_request_map subscribing (onetime) to next osd map 2024-06-03T09:06:57.913 INFO:teuthology.orchestra.run.smithi088.stderr:2024-06-03T09:06:57.905+0000 7f64bf173640 10 monclient: _renew_subs 2024-06-03T09:06:57.913 INFO:teuthology.orchestra.run.smithi088.stderr:2024-06-03T09:06:57.905+0000 7f64bf173640 10 monclient: _send_mon_message to mon.a at v1:172.21.15.88:6789/0 2024-06-03T09:06:57.913 INFO:teuthology.orchestra.run.smithi088.stderr:2024-06-03T09:06:57.905+0000 7f64bf173640 1 -- 172.21.15.88:0/1662221529 --> v1:172.21.15.88:6789/0 -- mon_subscribe({osdmap=0}) v3 -- 0x7f64b8137770 con 0x7f64b80adb90 2024-06-03T09:06:57.913 INFO:teuthology.orchestra.run.smithi088.stderr:2024-06-03T09:06:57.905+0000 7f64bf173640 1 librados: init done 2024-06-03T09:06:57.913 INFO:teuthology.orchestra.run.smithi088.stderr:2024-06-03T09:06:57.905+0000 7f64bf173640 10 client.30762.objecter _submit_command 1 [{"prefix": "get_command_descriptions"}] 2024-06-03T09:06:57.913 INFO:teuthology.orchestra.run.smithi088.stderr:2024-06-03T09:06:57.905+0000 7f64bf173640 15 client.30762.objecter _session_command_op_assign -1 1 2024-06-03T09:06:57.913 INFO:teuthology.orchestra.run.smithi088.stderr:2024-06-03T09:06:57.905+0000 7f64bf173640 20 client.30762.objecter _get_session osd=-1 returning homeless 2024-06-03T09:06:57.913 INFO:teuthology.orchestra.run.smithi088.stderr:2024-06-03T09:06:57.905+0000 7f64bf173640 10 client.30762.objecter _maybe_request_map subscribing (onetime) to next osd map 2024-06-03T09:06:57.914 INFO:teuthology.orchestra.run.smithi088.stderr:2024-06-03T09:06:57.905+0000 7f64bf173640 10 monclient: _send_mon_message to mon.a at v1:172.21.15.88:6789/0 2024-06-03T09:06:57.914 INFO:teuthology.orchestra.run.smithi088.stderr:2024-06-03T09:06:57.905+0000 7f64bf173640 1 -- 172.21.15.88:0/1662221529 --> v1:172.21.15.88:6789/0 -- mon_get_version(what=osdmap handle=1) v1 -- 0x7f64b80b38a0 con 0x7f64b80adb90 2024-06-03T09:06:57.922 INFO:teuthology.orchestra.run.smithi088.stderr:2024-06-03T09:06:57.922+0000 7f64b57fa640 1 -- 172.21.15.88:0/1662221529 <== mon.0 v1:172.21.15.88:6789/0 5 ==== config(0 keys) v1 ==== 4+0+0 (unknown 0 0 0) 0x7f64b80b38a0 con 0x7f64b80adb90 2024-06-03T09:06:57.923 INFO:teuthology.orchestra.run.smithi088.stderr:2024-06-03T09:06:57.922+0000 7f64b57fa640 10 monclient: handle_config config(0 keys) v1 2024-06-03T09:06:57.923 INFO:teuthology.orchestra.run.smithi088.stderr:2024-06-03T09:06:57.922+0000 7f64b57fa640 1 -- 172.21.15.88:0/1662221529 <== mon.0 v1:172.21.15.88:6789/0 6 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 2234374065 0 0) 0x7f64a8002100 con 0x7f64b80adb90 2024-06-03T09:06:57.923 INFO:teuthology.orchestra.run.smithi088.stderr:2024-06-03T09:06:57.922+0000 7f64b57fa640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-06-03T09:06:57.923 INFO:teuthology.orchestra.run.smithi088.stderr:2024-06-03T09:06:57.922+0000 7f64b57fa640 10 monclient: got monmap 1 from mon.a (according to old e1) 2024-06-03T09:06:57.923 INFO:teuthology.orchestra.run.smithi088.stderr:2024-06-03T09:06:57.922+0000 7f64b57fa640 10 monclient: dump: 2024-06-03T09:06:57.923 INFO:teuthology.orchestra.run.smithi088.stderr:epoch 1 2024-06-03T09:06:57.924 INFO:teuthology.orchestra.run.smithi088.stderr:fsid ec3c3fdd-3ac6-4d7d-9412-683130ed8b04 2024-06-03T09:06:57.924 INFO:teuthology.orchestra.run.smithi088.stderr:last_changed 2024-06-03T08:34:31.885125+0000 2024-06-03T09:06:57.924 INFO:teuthology.orchestra.run.smithi088.stderr:created 2024-06-03T08:34:31.885125+0000 2024-06-03T09:06:57.924 INFO:teuthology.orchestra.run.smithi088.stderr:min_mon_release 18 (reef) 2024-06-03T09:06:57.924 INFO:teuthology.orchestra.run.smithi088.stderr:election_strategy: 3 2024-06-03T09:06:57.924 INFO:teuthology.orchestra.run.smithi088.stderr:0: v1:172.21.15.88:6789/0 mon.a 2024-06-03T09:06:57.924 INFO:teuthology.orchestra.run.smithi088.stderr:1: v1:172.21.15.192:6789/0 mon.b 2024-06-03T09:06:57.924 INFO:teuthology.orchestra.run.smithi088.stderr:2: v1:172.21.15.88:6790/0 mon.c 2024-06-03T09:06:57.925 INFO:teuthology.orchestra.run.smithi088.stderr: 2024-06-03T09:06:57.925 INFO:teuthology.orchestra.run.smithi088.stderr:2024-06-03T09:06:57.922+0000 7f64b6ffd640 4 set_mon_vals no callback set 2024-06-03T09:06:57.927 INFO:teuthology.orchestra.run.smithi088.stderr:2024-06-03T09:06:57.923+0000 7f64b57fa640 1 -- 172.21.15.88:0/1662221529 <== mon.0 v1:172.21.15.88:6789/0 7 ==== mgrmap(e 5) v1 ==== 89110+0+0 (unknown 873533973 0 0) 0x7f64a801a920 con 0x7f64b80adb90 2024-06-03T09:06:57.927 INFO:teuthology.orchestra.run.smithi088.stderr:2024-06-03T09:06:57.924+0000 7f64bcf0f640 10 monclient: get_auth_request con 0x7f649806d470 auth_method 0 2024-06-03T09:06:57.927 INFO:teuthology.orchestra.run.smithi088.stderr:2024-06-03T09:06:57.924+0000 7f64b57fa640 10 client.30762.objecter ms_handle_connect 0x7f649806d470 2024-06-03T09:06:57.927 INFO:teuthology.orchestra.run.smithi088.stderr:2024-06-03T09:06:57.925+0000 7f64b57fa640 1 -- 172.21.15.88:0/1662221529 <== mon.0 v1:172.21.15.88:6789/0 8 ==== osd_map(1014..1014 src has 622..1014) v4 ==== 5229+0+0 (unknown 3230192161 0 0) 0x7f64a8054270 con 0x7f64b80adb90 2024-06-03T09:06:57.927 INFO:teuthology.orchestra.run.smithi088.stderr:2024-06-03T09:06:57.925+0000 7f64b57fa640 10 client.30762.objecter ms_dispatch 0x7f64b8000b90 osd_map(1014..1014 src has 622..1014) v4 2024-06-03T09:06:57.927 INFO:teuthology.orchestra.run.smithi088.stderr:2024-06-03T09:06:57.925+0000 7f64b57fa640 3 client.30762.objecter handle_osd_map got epochs [1014,1014] > 0 2024-06-03T09:06:57.928 INFO:teuthology.orchestra.run.smithi088.stderr:2024-06-03T09:06:57.925+0000 7f64b57fa640 3 client.30762.objecter handle_osd_map decoding full epoch 1014 2024-06-03T09:06:57.928 INFO:teuthology.orchestra.run.smithi088.stderr:2024-06-03T09:06:57.925+0000 7f64b57fa640 10 client.30762.objecter checking command 1 2024-06-03T09:06:57.928 INFO:teuthology.orchestra.run.smithi088.stderr:2024-06-03T09:06:57.925+0000 7f64b57fa640 20 client.30762.objecter _get_session s=0x7f64980753b0 osd=2 3 2024-06-03T09:06:57.928 INFO:teuthology.orchestra.run.smithi088.stderr:2024-06-03T09:06:57.925+0000 7f64b57fa640 20 client.30762.objecter put_session s=0x7f64980753b0 osd=2 3 2024-06-03T09:06:57.928 INFO:teuthology.orchestra.run.smithi088.stderr:2024-06-03T09:06:57.925+0000 7f64b57fa640 15 client.30762.objecter _session_command_op_remove -1 1 2024-06-03T09:06:57.928 INFO:teuthology.orchestra.run.smithi088.stderr:2024-06-03T09:06:57.925+0000 7f64b57fa640 20 client.30762.objecter _get_session s=0x7f64980753b0 osd=2 3 2024-06-03T09:06:57.929 INFO:teuthology.orchestra.run.smithi088.stderr:2024-06-03T09:06:57.925+0000 7f64b57fa640 20 client.30762.objecter get_session s=0x7f64980753b0 osd=2 3 2024-06-03T09:06:57.929 INFO:teuthology.orchestra.run.smithi088.stderr:2024-06-03T09:06:57.925+0000 7f64b57fa640 15 client.30762.objecter _session_command_op_assign 2 1 2024-06-03T09:06:57.929 INFO:teuthology.orchestra.run.smithi088.stderr:2024-06-03T09:06:57.925+0000 7f64b57fa640 20 client.30762.objecter put_session s=0x7f64980753b0 osd=2 4 2024-06-03T09:06:57.929 INFO:teuthology.orchestra.run.smithi088.stderr:2024-06-03T09:06:57.925+0000 7f64b57fa640 10 client.30762.objecter _send_command 1 2024-06-03T09:06:57.929 INFO:teuthology.orchestra.run.smithi088.stderr:2024-06-03T09:06:57.925+0000 7f64b57fa640 1 -- 172.21.15.88:0/1662221529 --> v1:172.21.15.88:6802/35065 -- command(tid 1: {"prefix": "get_command_descriptions"}) v1 -- 0x7f64980796d0 con 0x7f6498075a60 2024-06-03T09:06:57.929 INFO:teuthology.orchestra.run.smithi088.stderr:2024-06-03T09:06:57.925+0000 7f64b57fa640 20 client.30762.objecter dump_active .. 0 homeless 2024-06-03T09:06:57.930 INFO:teuthology.orchestra.run.smithi088.stderr:2024-06-03T09:06:57.925+0000 7f64b57fa640 1 -- 172.21.15.88:0/1662221529 <== mon.0 v1:172.21.15.88:6789/0 9 ==== mon_get_version_reply(handle=1 version=1014) v2 ==== 24+0+0 (unknown 3501658476 0 0) 0x7f64a804fe20 con 0x7f64b80adb90 2024-06-03T09:06:57.930 INFO:teuthology.orchestra.run.smithi088.stderr:2024-06-03T09:06:57.925+0000 7f64b57fa640 10 monclient: handle_get_version_reply finishing 1 version 1014 2024-06-03T09:06:57.930 INFO:teuthology.orchestra.run.smithi088.stderr:2024-06-03T09:06:57.926+0000 7f64bd710640 10 monclient: get_auth_request con 0x7f6498075a60 auth_method 0 2024-06-03T09:06:57.930 INFO:teuthology.orchestra.run.smithi088.stderr:2024-06-03T09:06:57.927+0000 7f64b57fa640 10 client.30762.objecter ms_handle_connect 0x7f6498075a60 2024-06-03T09:06:57.949 INFO:teuthology.orchestra.run.smithi088.stderr:2024-06-03T09:06:57.949+0000 7f64b57fa640 1 -- 172.21.15.88:0/1662221529 <== osd.2 v1:172.21.15.88:6802/35065 1 ==== command_reply(tid 1: 0 ) v1 ==== 8+0+26118 (unknown 0 0 1686565167) 0x7f64980796d0 con 0x7f6498075a60 2024-06-03T09:06:57.949 INFO:teuthology.orchestra.run.smithi088.stderr:2024-06-03T09:06:57.949+0000 7f64b57fa640 10 client.30762.objecter ms_dispatch 0x7f64b8000b90 command_reply(tid 1: 0 ) v1 2024-06-03T09:06:57.949 INFO:teuthology.orchestra.run.smithi088.stderr:2024-06-03T09:06:57.949+0000 7f64b57fa640 10 client.30762.objecter _finish_command 1 = system:0 2024-06-03T09:06:57.949 INFO:teuthology.orchestra.run.smithi088.stderr:2024-06-03T09:06:57.949+0000 7f64b57fa640 20 client.30762.objecter put_session s=0x7f64980753b0 osd=2 4 2024-06-03T09:06:57.949 INFO:teuthology.orchestra.run.smithi088.stderr:2024-06-03T09:06:57.949+0000 7f64b57fa640 15 client.30762.objecter _session_command_op_remove 2 1 2024-06-03T09:06:57.964 INFO:teuthology.orchestra.run.smithi088.stderr:2024-06-03T09:06:57.964+0000 7f64bf173640 10 client.30762.objecter _submit_command 2 [{"prefix": "config set", "var": "osd_debug_deep_scrub_sleep", "val": ["0"]}] 2024-06-03T09:06:57.964 INFO:teuthology.orchestra.run.smithi088.stderr:2024-06-03T09:06:57.964+0000 7f64bf173640 15 client.30762.objecter _session_command_op_assign -1 2 2024-06-03T09:06:57.964 INFO:teuthology.orchestra.run.smithi088.stderr:2024-06-03T09:06:57.964+0000 7f64bf173640 20 client.30762.objecter _get_session s=0x7f64980753b0 osd=2 3 2024-06-03T09:06:57.964 INFO:teuthology.orchestra.run.smithi088.stderr:2024-06-03T09:06:57.964+0000 7f64bf173640 20 client.30762.objecter put_session s=0x7f64980753b0 osd=2 3 2024-06-03T09:06:57.964 INFO:teuthology.orchestra.run.smithi088.stderr:2024-06-03T09:06:57.964+0000 7f64bf173640 20 client.30762.objecter _get_session s=0x7f64980753b0 osd=2 3 2024-06-03T09:06:57.964 INFO:teuthology.orchestra.run.smithi088.stderr:2024-06-03T09:06:57.964+0000 7f64bf173640 15 client.30762.objecter _session_command_op_remove -1 2 2024-06-03T09:06:57.964 INFO:teuthology.orchestra.run.smithi088.stderr:2024-06-03T09:06:57.964+0000 7f64bf173640 20 client.30762.objecter get_session s=0x7f64980753b0 osd=2 3 2024-06-03T09:06:57.964 INFO:teuthology.orchestra.run.smithi088.stderr:2024-06-03T09:06:57.964+0000 7f64bf173640 15 client.30762.objecter _session_command_op_assign 2 2 2024-06-03T09:06:57.964 INFO:teuthology.orchestra.run.smithi088.stderr:2024-06-03T09:06:57.964+0000 7f64bf173640 20 client.30762.objecter put_session s=0x7f64980753b0 osd=2 4 2024-06-03T09:06:57.964 INFO:teuthology.orchestra.run.smithi088.stderr:2024-06-03T09:06:57.964+0000 7f64bf173640 10 client.30762.objecter _send_command 2 2024-06-03T09:06:57.964 INFO:teuthology.orchestra.run.smithi088.stderr:2024-06-03T09:06:57.964+0000 7f64bf173640 1 -- 172.21.15.88:0/1662221529 --> v1:172.21.15.88:6802/35065 -- command(tid 2: {"prefix": "config set", "var": "osd_debug_deep_scrub_sleep", "val": ["0"]}) v1 -- 0x7f64b80adf70 con 0x7f6498075a60 2024-06-03T09:06:57.979 INFO:teuthology.orchestra.run.smithi088.stderr:2024-06-03T09:06:57.979+0000 7f64b57fa640 1 -- 172.21.15.88:0/1662221529 <== osd.2 v1:172.21.15.88:6802/35065 2 ==== command_reply(tid 2: 0 ) v1 ==== 8+0+97 (unknown 0 0 119112422) 0x7f64a00077c0 con 0x7f6498075a60 2024-06-03T09:06:57.979 INFO:teuthology.orchestra.run.smithi088.stderr:2024-06-03T09:06:57.979+0000 7f64b57fa640 10 client.30762.objecter ms_dispatch 0x7f64b8000b90 command_reply(tid 2: 0 ) v1 2024-06-03T09:06:57.980 INFO:teuthology.orchestra.run.smithi088.stderr:2024-06-03T09:06:57.979+0000 7f64b57fa640 10 client.30762.objecter _finish_command 2 = system:0 2024-06-03T09:06:57.980 INFO:teuthology.orchestra.run.smithi088.stderr:2024-06-03T09:06:57.979+0000 7f64b57fa640 20 client.30762.objecter put_session s=0x7f64980753b0 osd=2 4 2024-06-03T09:06:57.980 INFO:teuthology.orchestra.run.smithi088.stderr:2024-06-03T09:06:57.979+0000 7f64b57fa640 15 client.30762.objecter _session_command_op_remove 2 2 2024-06-03T09:06:57.980 INFO:teuthology.orchestra.run.smithi088.stdout:{ 2024-06-03T09:06:57.980 INFO:teuthology.orchestra.run.smithi088.stdout: "success": "osd_debug_deep_scrub_sleep = '' (not observed, change may require restart) " 2024-06-03T09:06:57.980 INFO:teuthology.orchestra.run.smithi088.stdout:} 2024-06-03T09:06:57.981 INFO:teuthology.orchestra.run.smithi088.stderr:2024-06-03T09:06:57.979+0000 7f64bf173640 10 librados: watch_flush enter 2024-06-03T09:06:57.981 INFO:teuthology.orchestra.run.smithi088.stderr:2024-06-03T09:06:57.979+0000 7f64bf173640 10 librados: watch_flush exit 2024-06-03T09:06:57.981 INFO:teuthology.orchestra.run.smithi088.stderr:2024-06-03T09:06:57.979+0000 7f64bf173640 10 client.30762.objecter close_session for osd.2 2024-06-03T09:06:57.982 INFO:teuthology.orchestra.run.smithi088.stderr:2024-06-03T09:06:57.979+0000 7f64bf173640 1 -- 172.21.15.88:0/1662221529 >> v1:172.21.15.88:6802/35065 conn(0x7f6498075a60 legacy=0x7f6498077f00 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-06-03T09:06:57.982 INFO:teuthology.orchestra.run.smithi088.stderr:2024-06-03T09:06:57.979+0000 7f64bf173640 20 client.30762.objecter put_session s=0x7f64980753b0 osd=2 1 2024-06-03T09:06:57.982 INFO:teuthology.orchestra.run.smithi088.stderr:2024-06-03T09:06:57.979+0000 7f64bf173640 20 client.30762.objecter shutdown clearing up homeless session... 2024-06-03T09:06:57.982 INFO:teuthology.orchestra.run.smithi088.stderr:2024-06-03T09:06:57.979+0000 7f64bf173640 10 client.30762.objecter successfully canceled tick 2024-06-03T09:06:57.982 INFO:teuthology.orchestra.run.smithi088.stderr:2024-06-03T09:06:57.979+0000 7f64bf173640 1 -- 172.21.15.88:0/1662221529 >> v1:172.21.15.192:6800/34229 conn(0x7f649806d470 legacy=0x7f649806f930 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-06-03T09:06:57.982 INFO:teuthology.orchestra.run.smithi088.stderr:2024-06-03T09:06:57.979+0000 7f64bf173640 10 monclient: shutdown 2024-06-03T09:06:57.982 INFO:teuthology.orchestra.run.smithi088.stderr:2024-06-03T09:06:57.979+0000 7f64bf173640 20 monclient: shutdown discarding 0 pending message(s) 2024-06-03T09:06:57.982 INFO:teuthology.orchestra.run.smithi088.stderr:2024-06-03T09:06:57.979+0000 7f64bf173640 1 -- 172.21.15.88:0/1662221529 >> v1:172.21.15.88:6789/0 conn(0x7f64b80adb90 legacy=0x7f64b8137060 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-06-03T09:06:57.982 INFO:teuthology.orchestra.run.smithi088.stderr:2024-06-03T09:06:57.979+0000 7f64bd710640 1 -- 172.21.15.88:0/1662221529 reap_dead start 2024-06-03T09:06:57.982 INFO:teuthology.orchestra.run.smithi088.stderr:2024-06-03T09:06:57.980+0000 7f64bf173640 1 -- 172.21.15.88:0/1662221529 shutdown_connections 2024-06-03T09:06:57.982 INFO:teuthology.orchestra.run.smithi088.stderr:2024-06-03T09:06:57.980+0000 7f64bf173640 1 -- 172.21.15.88:0/1662221529 >> 172.21.15.88:0/1662221529 conn(0x7f64b80b2f20 msgr2=0x7f64b80af9e0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-06-03T09:06:57.982 INFO:teuthology.orchestra.run.smithi088.stderr:2024-06-03T09:06:57.980+0000 7f64bf173640 1 -- 172.21.15.88:0/1662221529 shutdown_connections 2024-06-03T09:06:57.983 INFO:teuthology.orchestra.run.smithi088.stderr:2024-06-03T09:06:57.980+0000 7f64bf173640 1 -- 172.21.15.88:0/1662221529 wait complete. 2024-06-03T09:06:57.983 INFO:teuthology.orchestra.run.smithi088.stderr:2024-06-03T09:06:57.980+0000 7f64bf173640 1 librados: shutdown 2024-06-03T09:06:57.990 DEBUG:teuthology.orchestra.run.smithi088:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd deep-scrub 2 2024-06-03T09:06:58.088 INFO:teuthology.orchestra.run.smithi088.stderr:2024-06-03T09:06:58.088+0000 7fadb0781640 10 monclient: get_monmap_and_config 2024-06-03T09:06:58.088 INFO:teuthology.orchestra.run.smithi088.stderr:2024-06-03T09:06:58.088+0000 7fadb0781640 10 monclient: build_initial_monmap 2024-06-03T09:06:58.089 INFO:teuthology.orchestra.run.smithi088.stderr:2024-06-03T09:06:58.088+0000 7fadb0781640 10 monclient: monmap: 2024-06-03T09:06:58.089 INFO:teuthology.orchestra.run.smithi088.stderr:epoch 0 2024-06-03T09:06:58.089 INFO:teuthology.orchestra.run.smithi088.stderr:fsid ec3c3fdd-3ac6-4d7d-9412-683130ed8b04 2024-06-03T09:06:58.089 INFO:teuthology.orchestra.run.smithi088.stderr:last_changed 2024-06-03T09:06:58.089660+0000 2024-06-03T09:06:58.089 INFO:teuthology.orchestra.run.smithi088.stderr:created 2024-06-03T09:06:58.089660+0000 2024-06-03T09:06:58.089 INFO:teuthology.orchestra.run.smithi088.stderr:min_mon_release 0 (unknown) 2024-06-03T09:06:58.089 INFO:teuthology.orchestra.run.smithi088.stderr:election_strategy: 3 2024-06-03T09:06:58.089 INFO:teuthology.orchestra.run.smithi088.stderr:0: v1:172.21.15.88:6789/0 mon.noname-a 2024-06-03T09:06:58.089 INFO:teuthology.orchestra.run.smithi088.stderr:1: v1:172.21.15.192:6789/0 mon.noname-c 2024-06-03T09:06:58.089 INFO:teuthology.orchestra.run.smithi088.stderr:2: v1:172.21.15.88:6790/0 mon.noname-b 2024-06-03T09:06:58.089 INFO:teuthology.orchestra.run.smithi088.stderr: 2024-06-03T09:06:58.091 INFO:teuthology.orchestra.run.smithi088.stderr:2024-06-03T09:06:58.089+0000 7fadb0781640 1 Processor -- start 2024-06-03T09:06:58.091 INFO:teuthology.orchestra.run.smithi088.stderr:2024-06-03T09:06:58.089+0000 7fadb0781640 1 -- start start 2024-06-03T09:06:58.091 INFO:teuthology.orchestra.run.smithi088.stderr:2024-06-03T09:06:58.089+0000 7fadb0781640 10 monclient: init 2024-06-03T09:06:58.091 INFO:teuthology.orchestra.run.smithi088.stderr:2024-06-03T09:06:58.089+0000 7fadb0781640 10 monclient: _reopen_session rank -1 2024-06-03T09:06:58.091 INFO:teuthology.orchestra.run.smithi088.stderr:2024-06-03T09:06:58.089+0000 7fadb0781640 10 monclient: _add_conns ranks=[0,2,1] 2024-06-03T09:06:58.091 INFO:teuthology.orchestra.run.smithi088.stderr:2024-06-03T09:06:58.089+0000 7fadb0781640 10 monclient(hunting): picked mon.noname-a con 0x7fada80adb90 addr v1:172.21.15.88:6789/0 2024-06-03T09:06:58.092 INFO:teuthology.orchestra.run.smithi088.stderr:2024-06-03T09:06:58.089+0000 7fadb0781640 10 monclient(hunting): picked mon.noname-b con 0x7fada806ee00 addr v1:172.21.15.88:6790/0 2024-06-03T09:06:58.092 INFO:teuthology.orchestra.run.smithi088.stderr:2024-06-03T09:06:58.089+0000 7fadb0781640 10 monclient(hunting): picked mon.noname-c con 0x7fada81383e0 addr v1:172.21.15.192:6789/0 2024-06-03T09:06:58.092 INFO:teuthology.orchestra.run.smithi088.stderr:2024-06-03T09:06:58.089+0000 7fadb0781640 1 -- --> v1:172.21.15.88:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fada806f2c0 con 0x7fada80adb90 2024-06-03T09:06:58.092 INFO:teuthology.orchestra.run.smithi088.stderr:2024-06-03T09:06:58.090+0000 7fadb0781640 1 -- --> v1:172.21.15.192:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fada80710c0 con 0x7fada81383e0 2024-06-03T09:06:58.092 INFO:teuthology.orchestra.run.smithi088.stderr:2024-06-03T09:06:58.090+0000 7fadb0781640 1 -- --> v1:172.21.15.88:6790/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fada8071fd0 con 0x7fada806ee00 2024-06-03T09:06:58.092 INFO:teuthology.orchestra.run.smithi088.stderr:2024-06-03T09:06:58.090+0000 7fadb0781640 10 monclient(hunting): _renew_subs 2024-06-03T09:06:58.092 INFO:teuthology.orchestra.run.smithi088.stderr:2024-06-03T09:06:58.090+0000 7fadb0781640 10 monclient(hunting): authenticate will time out at 3073.899961s 2024-06-03T09:06:58.092 INFO:teuthology.orchestra.run.smithi088.stderr:2024-06-03T09:06:58.090+0000 7fadadd1c640 1 --1- >> v1:172.21.15.88:6790/0 conn(0x7fada806ee00 0x7fada8137cd0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.88:6790/0 says I am v1:172.21.15.88:51018/0 (socket says 172.21.15.88:51018) 2024-06-03T09:06:58.092 INFO:teuthology.orchestra.run.smithi088.stderr:2024-06-03T09:06:58.090+0000 7fadadd1c640 1 -- 172.21.15.88:0/3458365234 learned_addr learned my addr 172.21.15.88:0/3458365234 (peer_addr_for_me v1:172.21.15.88:0/0) 2024-06-03T09:06:58.093 INFO:teuthology.orchestra.run.smithi088.stderr:2024-06-03T09:06:58.092+0000 7fadad51b640 1 -- 172.21.15.88:0/3458365234 <== mon.1 v1:172.21.15.192:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 3802718456 0 0) 0x7fada80710c0 con 0x7fada81383e0 2024-06-03T09:06:58.093 INFO:teuthology.orchestra.run.smithi088.stderr:2024-06-03T09:06:58.093+0000 7fadad51b640 10 monclient(hunting): _negotiate 2024-06-03T09:06:58.093 INFO:teuthology.orchestra.run.smithi088.stderr:2024-06-03T09:06:58.093+0000 7fadad51b640 10 monclient(hunting): _init_auth method 2 2024-06-03T09:06:58.093 INFO:teuthology.orchestra.run.smithi088.stderr:2024-06-03T09:06:58.093+0000 7fadad51b640 10 monclient(hunting): _init_auth creating new auth 2024-06-03T09:06:58.093 INFO:teuthology.orchestra.run.smithi088.stderr:2024-06-03T09:06:58.093+0000 7fadad51b640 10 monclient(hunting): my global_id is 30337 2024-06-03T09:06:58.093 INFO:teuthology.orchestra.run.smithi088.stderr:2024-06-03T09:06:58.093+0000 7fadad51b640 1 -- 172.21.15.88:0/3458365234 --> v1:172.21.15.192:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7fad94003520 con 0x7fada81383e0 2024-06-03T09:06:58.093 INFO:teuthology.orchestra.run.smithi088.stderr:2024-06-03T09:06:58.093+0000 7fadad51b640 1 -- 172.21.15.88:0/3458365234 <== mon.2 v1:172.21.15.88:6790/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 3890350089 0 0) 0x7fada8071fd0 con 0x7fada806ee00 2024-06-03T09:06:58.094 INFO:teuthology.orchestra.run.smithi088.stderr:2024-06-03T09:06:58.093+0000 7fadad51b640 10 monclient(hunting): _negotiate 2024-06-03T09:06:58.094 INFO:teuthology.orchestra.run.smithi088.stderr:2024-06-03T09:06:58.093+0000 7fadad51b640 10 monclient(hunting): _init_auth method 2 2024-06-03T09:06:58.094 INFO:teuthology.orchestra.run.smithi088.stderr:2024-06-03T09:06:58.093+0000 7fadad51b640 10 monclient(hunting): _init_auth creating new auth 2024-06-03T09:06:58.094 INFO:teuthology.orchestra.run.smithi088.stderr:2024-06-03T09:06:58.093+0000 7fadad51b640 10 monclient(hunting): my global_id is 30272 2024-06-03T09:06:58.094 INFO:teuthology.orchestra.run.smithi088.stderr:2024-06-03T09:06:58.093+0000 7fadad51b640 1 -- 172.21.15.88:0/3458365234 --> v1:172.21.15.88:6790/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7fada80710c0 con 0x7fada806ee00 2024-06-03T09:06:58.094 INFO:teuthology.orchestra.run.smithi088.stderr:2024-06-03T09:06:58.093+0000 7fadad51b640 1 -- 172.21.15.88:0/3458365234 <== mon.0 v1:172.21.15.88:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 1273326951 0 0) 0x7fada806f2c0 con 0x7fada80adb90 2024-06-03T09:06:58.094 INFO:teuthology.orchestra.run.smithi088.stderr:2024-06-03T09:06:58.093+0000 7fadad51b640 10 monclient(hunting): _negotiate 2024-06-03T09:06:58.094 INFO:teuthology.orchestra.run.smithi088.stderr:2024-06-03T09:06:58.093+0000 7fadad51b640 10 monclient(hunting): _init_auth method 2 2024-06-03T09:06:58.094 INFO:teuthology.orchestra.run.smithi088.stderr:2024-06-03T09:06:58.093+0000 7fadad51b640 10 monclient(hunting): _init_auth creating new auth 2024-06-03T09:06:58.094 INFO:teuthology.orchestra.run.smithi088.stderr:2024-06-03T09:06:58.093+0000 7fadad51b640 10 monclient(hunting): my global_id is 30765 2024-06-03T09:06:58.094 INFO:teuthology.orchestra.run.smithi088.stderr:2024-06-03T09:06:58.093+0000 7fadad51b640 1 -- 172.21.15.88:0/3458365234 --> v1:172.21.15.88:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7fada8071fd0 con 0x7fada80adb90 2024-06-03T09:06:58.096 INFO:teuthology.orchestra.run.smithi088.stderr:2024-06-03T09:06:58.096+0000 7fadad51b640 1 -- 172.21.15.88:0/3458365234 <== mon.1 v1:172.21.15.192:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 214+0+0 (unknown 3813763835 0 0) 0x7fad94003520 con 0x7fada81383e0 2024-06-03T09:06:58.096 INFO:teuthology.orchestra.run.smithi088.stderr:2024-06-03T09:06:58.096+0000 7fadad51b640 1 -- 172.21.15.88:0/3458365234 >> v1:172.21.15.88:6790/0 conn(0x7fada806ee00 legacy=0x7fada8137cd0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-06-03T09:06:58.096 INFO:teuthology.orchestra.run.smithi088.stderr:2024-06-03T09:06:58.096+0000 7fadad51b640 1 -- 172.21.15.88:0/3458365234 >> v1:172.21.15.88:6789/0 conn(0x7fada80adb90 legacy=0x7fada80718c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-06-03T09:06:58.096 INFO:teuthology.orchestra.run.smithi088.stderr:2024-06-03T09:06:58.096+0000 7fadad51b640 10 monclient: _finish_hunting 0 2024-06-03T09:06:58.096 INFO:teuthology.orchestra.run.smithi088.stderr:2024-06-03T09:06:58.096+0000 7fadad51b640 1 monclient: found mon.noname-c 2024-06-03T09:06:58.097 INFO:teuthology.orchestra.run.smithi088.stderr:2024-06-03T09:06:58.096+0000 7fadad51b640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-06-03T09:06:58.097 INFO:teuthology.orchestra.run.smithi088.stderr:2024-06-03T09:06:58.096+0000 7fadad51b640 10 monclient: _send_mon_message to mon.noname-c at v1:172.21.15.192:6789/0 2024-06-03T09:06:58.097 INFO:teuthology.orchestra.run.smithi088.stderr:2024-06-03T09:06:58.096+0000 7fadad51b640 1 -- 172.21.15.88:0/3458365234 --> v1:172.21.15.192:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fada8141750 con 0x7fada81383e0 2024-06-03T09:06:58.097 INFO:teuthology.orchestra.run.smithi088.stderr:2024-06-03T09:06:58.096+0000 7fadad51b640 10 monclient: _finish_auth 0 2024-06-03T09:06:58.097 INFO:teuthology.orchestra.run.smithi088.stderr:2024-06-03T09:06:58.096+0000 7fadad51b640 10 monclient: _check_auth_tickets 2024-06-03T09:06:58.097 INFO:teuthology.orchestra.run.smithi088.stderr:2024-06-03T09:06:58.096+0000 7fadad51b640 20 monclient: _check_auth_rotating not needed by client.admin 2024-06-03T09:06:58.097 INFO:teuthology.orchestra.run.smithi088.stderr:2024-06-03T09:06:58.096+0000 7fadad51b640 1 -- 172.21.15.88:0/3458365234 <== mon.1 v1:172.21.15.192:6789/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 2234374065 0 0) 0x7fada0001990 con 0x7fada81383e0 2024-06-03T09:06:58.097 INFO:teuthology.orchestra.run.smithi088.stderr:2024-06-03T09:06:58.096+0000 7fadad51b640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-06-03T09:06:58.097 INFO:teuthology.orchestra.run.smithi088.stderr:2024-06-03T09:06:58.096+0000 7fadad51b640 10 monclient: got monmap 1 from mon.noname-c (according to old e1) 2024-06-03T09:06:58.097 INFO:teuthology.orchestra.run.smithi088.stderr:2024-06-03T09:06:58.096+0000 7fadad51b640 10 monclient: dump: 2024-06-03T09:06:58.097 INFO:teuthology.orchestra.run.smithi088.stderr:epoch 1 2024-06-03T09:06:58.097 INFO:teuthology.orchestra.run.smithi088.stderr:fsid ec3c3fdd-3ac6-4d7d-9412-683130ed8b04 2024-06-03T09:06:58.097 INFO:teuthology.orchestra.run.smithi088.stderr:last_changed 2024-06-03T08:34:31.885125+0000 2024-06-03T09:06:58.098 INFO:teuthology.orchestra.run.smithi088.stderr:created 2024-06-03T08:34:31.885125+0000 2024-06-03T09:06:58.098 INFO:teuthology.orchestra.run.smithi088.stderr:min_mon_release 18 (reef) 2024-06-03T09:06:58.098 INFO:teuthology.orchestra.run.smithi088.stderr:election_strategy: 3 2024-06-03T09:06:58.098 INFO:teuthology.orchestra.run.smithi088.stderr:0: v1:172.21.15.88:6789/0 mon.a 2024-06-03T09:06:58.098 INFO:teuthology.orchestra.run.smithi088.stderr:1: v1:172.21.15.192:6789/0 mon.b 2024-06-03T09:06:58.098 INFO:teuthology.orchestra.run.smithi088.stderr:2: v1:172.21.15.88:6790/0 mon.c 2024-06-03T09:06:58.098 INFO:teuthology.orchestra.run.smithi088.stderr: 2024-06-03T09:06:58.098 INFO:teuthology.orchestra.run.smithi088.stderr:2024-06-03T09:06:58.096+0000 7fadb0781640 5 monclient: authenticate success, global_id 30337 2024-06-03T09:06:58.098 INFO:teuthology.orchestra.run.smithi088.stderr:2024-06-03T09:06:58.096+0000 7fadb0781640 20 monclient: get_monmap_and_config waiting for monmap|config 2024-06-03T09:06:58.101 INFO:teuthology.orchestra.run.smithi088.stderr:2024-06-03T09:06:58.098+0000 7fadad51b640 1 -- 172.21.15.88:0/3458365234 <== mon.1 v1:172.21.15.192:6789/0 4 ==== config(0 keys) v1 ==== 4+0+0 (unknown 0 0 0) 0x7fada0001bb0 con 0x7fada81383e0 2024-06-03T09:06:58.101 INFO:teuthology.orchestra.run.smithi088.stderr:2024-06-03T09:06:58.098+0000 7fadad51b640 10 monclient: handle_config config(0 keys) v1 2024-06-03T09:06:58.101 INFO:teuthology.orchestra.run.smithi088.stderr:2024-06-03T09:06:58.098+0000 7fadad51b640 1 -- 172.21.15.88:0/3458365234 <== mon.1 v1:172.21.15.192:6789/0 5 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 2234374065 0 0) 0x7fada0001f40 con 0x7fada81383e0 2024-06-03T09:06:58.101 INFO:teuthology.orchestra.run.smithi088.stderr:2024-06-03T09:06:58.098+0000 7fadad51b640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-06-03T09:06:58.101 INFO:teuthology.orchestra.run.smithi088.stderr:2024-06-03T09:06:58.098+0000 7fadad51b640 10 monclient: got monmap 1 from mon.b (according to old e1) 2024-06-03T09:06:58.101 INFO:teuthology.orchestra.run.smithi088.stderr:2024-06-03T09:06:58.098+0000 7fadad51b640 10 monclient: dump: 2024-06-03T09:06:58.101 INFO:teuthology.orchestra.run.smithi088.stderr:epoch 1 2024-06-03T09:06:58.101 INFO:teuthology.orchestra.run.smithi088.stderr:fsid ec3c3fdd-3ac6-4d7d-9412-683130ed8b04 2024-06-03T09:06:58.101 INFO:teuthology.orchestra.run.smithi088.stderr:last_changed 2024-06-03T08:34:31.885125+0000 2024-06-03T09:06:58.101 INFO:teuthology.orchestra.run.smithi088.stderr:created 2024-06-03T08:34:31.885125+0000 2024-06-03T09:06:58.102 INFO:teuthology.orchestra.run.smithi088.stderr:min_mon_release 18 (reef) 2024-06-03T09:06:58.102 INFO:teuthology.orchestra.run.smithi088.stderr:election_strategy: 3 2024-06-03T09:06:58.102 INFO:teuthology.orchestra.run.smithi088.stderr:0: v1:172.21.15.88:6789/0 mon.a 2024-06-03T09:06:58.102 INFO:teuthology.orchestra.run.smithi088.stderr:1: v1:172.21.15.192:6789/0 mon.b 2024-06-03T09:06:58.102 INFO:teuthology.orchestra.run.smithi088.stderr:2: v1:172.21.15.88:6790/0 mon.c 2024-06-03T09:06:58.102 INFO:teuthology.orchestra.run.smithi088.stderr: 2024-06-03T09:06:58.102 INFO:teuthology.orchestra.run.smithi088.stderr:2024-06-03T09:06:58.098+0000 7fadb0781640 10 monclient: get_monmap_and_config success 2024-06-03T09:06:58.102 INFO:teuthology.orchestra.run.smithi088.stderr:2024-06-03T09:06:58.098+0000 7fadb0781640 4 set_mon_vals no callback set 2024-06-03T09:06:58.102 INFO:teuthology.orchestra.run.smithi088.stderr:2024-06-03T09:06:58.098+0000 7fadb0781640 10 monclient: shutdown 2024-06-03T09:06:58.102 INFO:teuthology.orchestra.run.smithi088.stderr:2024-06-03T09:06:58.098+0000 7fadb0781640 20 monclient: shutdown discarding 0 pending message(s) 2024-06-03T09:06:58.102 INFO:teuthology.orchestra.run.smithi088.stderr:2024-06-03T09:06:58.098+0000 7fadb0781640 1 -- 172.21.15.88:0/3458365234 >> v1:172.21.15.192:6789/0 conn(0x7fada81383e0 legacy=0x7fada8149af0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-06-03T09:06:58.102 INFO:teuthology.orchestra.run.smithi088.stderr:2024-06-03T09:06:58.098+0000 7fadb0781640 1 -- 172.21.15.88:0/3458365234 shutdown_connections 2024-06-03T09:06:58.102 INFO:teuthology.orchestra.run.smithi088.stderr:2024-06-03T09:06:58.098+0000 7fadb0781640 1 -- 172.21.15.88:0/3458365234 >> 172.21.15.88:0/3458365234 conn(0x7fada80b2f20 msgr2=0x7fada80b3300 unknown :-1 s=STATE_NONE l=0).mark_down 2024-06-03T09:06:58.103 INFO:teuthology.orchestra.run.smithi088.stderr:2024-06-03T09:06:58.098+0000 7fadb0781640 1 -- 172.21.15.88:0/3458365234 shutdown_connections 2024-06-03T09:06:58.103 INFO:teuthology.orchestra.run.smithi088.stderr:2024-06-03T09:06:58.098+0000 7fadb0781640 1 -- 172.21.15.88:0/3458365234 wait complete. 2024-06-03T09:06:58.103 INFO:teuthology.orchestra.run.smithi088.stderr:2024-06-03T09:06:58.099+0000 7fadb0781640 10 monclient: build_initial_monmap 2024-06-03T09:06:58.103 INFO:teuthology.orchestra.run.smithi088.stderr:2024-06-03T09:06:58.099+0000 7fadb0781640 10 monclient: monmap: 2024-06-03T09:06:58.103 INFO:teuthology.orchestra.run.smithi088.stderr:epoch 0 2024-06-03T09:06:58.103 INFO:teuthology.orchestra.run.smithi088.stderr:fsid 00000000-0000-0000-0000-000000000000 2024-06-03T09:06:58.103 INFO:teuthology.orchestra.run.smithi088.stderr:last_changed 0.000000 2024-06-03T09:06:58.103 INFO:teuthology.orchestra.run.smithi088.stderr:created 0.000000 2024-06-03T09:06:58.103 INFO:teuthology.orchestra.run.smithi088.stderr:min_mon_release 0 (unknown) 2024-06-03T09:06:58.103 INFO:teuthology.orchestra.run.smithi088.stderr:election_strategy: 1 2024-06-03T09:06:58.103 INFO:teuthology.orchestra.run.smithi088.stderr:0: v1:172.21.15.88:6789/0 mon.noname-a 2024-06-03T09:06:58.103 INFO:teuthology.orchestra.run.smithi088.stderr:1: v1:172.21.15.192:6789/0 mon.noname-b 2024-06-03T09:06:58.103 INFO:teuthology.orchestra.run.smithi088.stderr:2: v1:172.21.15.88:6790/0 mon.noname-c 2024-06-03T09:06:58.103 INFO:teuthology.orchestra.run.smithi088.stderr: 2024-06-03T09:06:58.104 INFO:teuthology.orchestra.run.smithi088.stderr:2024-06-03T09:06:58.099+0000 7fadb0781640 1 librados: starting msgr at 2024-06-03T09:06:58.104 INFO:teuthology.orchestra.run.smithi088.stderr:2024-06-03T09:06:58.099+0000 7fadb0781640 1 librados: starting objecter 2024-06-03T09:06:58.104 INFO:teuthology.orchestra.run.smithi088.stderr:2024-06-03T09:06:58.099+0000 7fadb0781640 1 Processor -- start 2024-06-03T09:06:58.104 INFO:teuthology.orchestra.run.smithi088.stderr:2024-06-03T09:06:58.099+0000 7fadb0781640 1 -- start start 2024-06-03T09:06:58.104 INFO:teuthology.orchestra.run.smithi088.stderr:2024-06-03T09:06:58.099+0000 7fadb0781640 1 librados: setting wanted keys 2024-06-03T09:06:58.104 INFO:teuthology.orchestra.run.smithi088.stderr:2024-06-03T09:06:58.099+0000 7fadb0781640 1 librados: calling monclient init 2024-06-03T09:06:58.104 INFO:teuthology.orchestra.run.smithi088.stderr:2024-06-03T09:06:58.099+0000 7fadb0781640 10 monclient: init 2024-06-03T09:06:58.104 INFO:teuthology.orchestra.run.smithi088.stderr:2024-06-03T09:06:58.100+0000 7fadb0781640 10 monclient: _reopen_session rank -1 2024-06-03T09:06:58.104 INFO:teuthology.orchestra.run.smithi088.stderr:2024-06-03T09:06:58.100+0000 7fadb0781640 10 monclient: _add_conns ranks=[0,1,2] 2024-06-03T09:06:58.104 INFO:teuthology.orchestra.run.smithi088.stderr:2024-06-03T09:06:58.100+0000 7fadb0781640 10 monclient(hunting): picked mon.noname-a con 0x7fada806ee00 addr v1:172.21.15.88:6789/0 2024-06-03T09:06:58.104 INFO:teuthology.orchestra.run.smithi088.stderr:2024-06-03T09:06:58.100+0000 7fadb0781640 10 monclient(hunting): picked mon.noname-b con 0x7fada80adb90 addr v1:172.21.15.192:6789/0 2024-06-03T09:06:58.104 INFO:teuthology.orchestra.run.smithi088.stderr:2024-06-03T09:06:58.100+0000 7fadb0781640 10 monclient(hunting): picked mon.noname-c con 0x7fada81383e0 addr v1:172.21.15.88:6790/0 2024-06-03T09:06:58.104 INFO:teuthology.orchestra.run.smithi088.stderr:2024-06-03T09:06:58.100+0000 7fadb0781640 1 -- --> v1:172.21.15.88:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fada8135b50 con 0x7fada806ee00 2024-06-03T09:06:58.104 INFO:teuthology.orchestra.run.smithi088.stderr:2024-06-03T09:06:58.100+0000 7fadb0781640 1 -- --> v1:172.21.15.192:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fada80a9440 con 0x7fada80adb90 2024-06-03T09:06:58.105 INFO:teuthology.orchestra.run.smithi088.stderr:2024-06-03T09:06:58.100+0000 7fadb0781640 1 -- --> v1:172.21.15.88:6790/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fada80ad520 con 0x7fada81383e0 2024-06-03T09:06:58.105 INFO:teuthology.orchestra.run.smithi088.stderr:2024-06-03T09:06:58.100+0000 7fadb0781640 10 monclient(hunting): _renew_subs 2024-06-03T09:06:58.105 INFO:teuthology.orchestra.run.smithi088.stderr:2024-06-03T09:06:58.100+0000 7fadb0781640 10 monclient(hunting): authenticate will time out at 3073.910458s 2024-06-03T09:06:58.105 INFO:teuthology.orchestra.run.smithi088.stderr:2024-06-03T09:06:58.100+0000 7fadaed1e640 1 --1- >> v1:172.21.15.88:6790/0 conn(0x7fada81383e0 0x7fada8135440 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=1).handle_server_banner_and_identify peer v1:172.21.15.88:6790/0 says I am v1:172.21.15.88:51034/0 (socket says 172.21.15.88:51034) 2024-06-03T09:06:58.105 INFO:teuthology.orchestra.run.smithi088.stderr:2024-06-03T09:06:58.100+0000 7fadaed1e640 1 -- 172.21.15.88:0/176455414 learned_addr learned my addr 172.21.15.88:0/176455414 (peer_addr_for_me v1:172.21.15.88:0/0) 2024-06-03T09:06:58.105 INFO:teuthology.orchestra.run.smithi088.stderr:2024-06-03T09:06:58.101+0000 7fad9effd640 10 client.?.objecter ms_handle_connect 0x7fada81383e0 2024-06-03T09:06:58.105 INFO:teuthology.orchestra.run.smithi088.stderr:2024-06-03T09:06:58.101+0000 7fad9effd640 10 client.?.objecter resend_mon_ops 2024-06-03T09:06:58.105 INFO:teuthology.orchestra.run.smithi088.stderr:2024-06-03T09:06:58.101+0000 7fad9effd640 10 client.?.objecter ms_handle_connect 0x7fada806ee00 2024-06-03T09:06:58.105 INFO:teuthology.orchestra.run.smithi088.stderr:2024-06-03T09:06:58.101+0000 7fad9effd640 10 client.?.objecter resend_mon_ops 2024-06-03T09:06:58.105 INFO:teuthology.orchestra.run.smithi088.stderr:2024-06-03T09:06:58.101+0000 7fad9effd640 10 client.?.objecter ms_handle_connect 0x7fada80adb90 2024-06-03T09:06:58.105 INFO:teuthology.orchestra.run.smithi088.stderr:2024-06-03T09:06:58.101+0000 7fad9effd640 10 client.?.objecter resend_mon_ops 2024-06-03T09:06:58.106 INFO:teuthology.orchestra.run.smithi088.stderr:2024-06-03T09:06:58.103+0000 7fad9effd640 1 -- 172.21.15.88:0/176455414 <== mon.2 v1:172.21.15.88:6790/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 1364165866 0 0) 0x7fada80ad520 con 0x7fada81383e0 2024-06-03T09:06:58.106 INFO:teuthology.orchestra.run.smithi088.stderr:2024-06-03T09:06:58.103+0000 7fad9effd640 10 monclient(hunting): _negotiate 2024-06-03T09:06:58.107 INFO:teuthology.orchestra.run.smithi088.stderr:2024-06-03T09:06:58.103+0000 7fad9effd640 10 monclient(hunting): _init_auth method 2 2024-06-03T09:06:58.107 INFO:teuthology.orchestra.run.smithi088.stderr:2024-06-03T09:06:58.103+0000 7fad9effd640 10 monclient(hunting): _init_auth creating new auth 2024-06-03T09:06:58.107 INFO:teuthology.orchestra.run.smithi088.stderr:2024-06-03T09:06:58.103+0000 7fad9effd640 10 monclient(hunting): my global_id is 30275 2024-06-03T09:06:58.107 INFO:teuthology.orchestra.run.smithi088.stderr:2024-06-03T09:06:58.103+0000 7fad9effd640 1 -- 172.21.15.88:0/176455414 --> v1:172.21.15.88:6790/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7fad8c004eb0 con 0x7fada81383e0 2024-06-03T09:06:58.107 INFO:teuthology.orchestra.run.smithi088.stderr:2024-06-03T09:06:58.103+0000 7fad9effd640 1 -- 172.21.15.88:0/176455414 <== mon.0 v1:172.21.15.88:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 3658581930 0 0) 0x7fada8135b50 con 0x7fada806ee00 2024-06-03T09:06:58.107 INFO:teuthology.orchestra.run.smithi088.stderr:2024-06-03T09:06:58.103+0000 7fad9effd640 10 monclient(hunting): _negotiate 2024-06-03T09:06:58.107 INFO:teuthology.orchestra.run.smithi088.stderr:2024-06-03T09:06:58.103+0000 7fad9effd640 10 monclient(hunting): _init_auth method 2 2024-06-03T09:06:58.107 INFO:teuthology.orchestra.run.smithi088.stderr:2024-06-03T09:06:58.103+0000 7fad9effd640 10 monclient(hunting): _init_auth creating new auth 2024-06-03T09:06:58.107 INFO:teuthology.orchestra.run.smithi088.stderr:2024-06-03T09:06:58.103+0000 7fad9effd640 10 monclient(hunting): my global_id is 30768 2024-06-03T09:06:58.107 INFO:teuthology.orchestra.run.smithi088.stderr:2024-06-03T09:06:58.103+0000 7fad9effd640 1 -- 172.21.15.88:0/176455414 --> v1:172.21.15.88:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7fada80ad520 con 0x7fada806ee00 2024-06-03T09:06:58.107 INFO:teuthology.orchestra.run.smithi088.stderr:2024-06-03T09:06:58.103+0000 7fad9effd640 1 -- 172.21.15.88:0/176455414 <== mon.1 v1:172.21.15.192:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 3708262655 0 0) 0x7fada80a9440 con 0x7fada80adb90 2024-06-03T09:06:58.107 INFO:teuthology.orchestra.run.smithi088.stderr:2024-06-03T09:06:58.103+0000 7fad9effd640 10 monclient(hunting): _negotiate 2024-06-03T09:06:58.107 INFO:teuthology.orchestra.run.smithi088.stderr:2024-06-03T09:06:58.103+0000 7fad9effd640 10 monclient(hunting): _init_auth method 2 2024-06-03T09:06:58.107 INFO:teuthology.orchestra.run.smithi088.stderr:2024-06-03T09:06:58.103+0000 7fad9effd640 10 monclient(hunting): _init_auth creating new auth 2024-06-03T09:06:58.108 INFO:teuthology.orchestra.run.smithi088.stderr:2024-06-03T09:06:58.103+0000 7fad9effd640 10 monclient(hunting): my global_id is 30340 2024-06-03T09:06:58.108 INFO:teuthology.orchestra.run.smithi088.stderr:2024-06-03T09:06:58.103+0000 7fad9effd640 1 -- 172.21.15.88:0/176455414 --> v1:172.21.15.192:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7fada8135b50 con 0x7fada80adb90 2024-06-03T09:06:58.109 INFO:teuthology.orchestra.run.smithi088.stderr:2024-06-03T09:06:58.107+0000 7fad9effd640 1 -- 172.21.15.88:0/176455414 <== mon.2 v1:172.21.15.88:6790/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 764+0+0 (unknown 1570845558 0 0) 0x7fad8c004eb0 con 0x7fada81383e0 2024-06-03T09:06:58.109 INFO:teuthology.orchestra.run.smithi088.stderr:2024-06-03T09:06:58.107+0000 7fad9effd640 1 -- 172.21.15.88:0/176455414 --> v1:172.21.15.88:6790/0 -- auth(proto 2 165 bytes epoch 0) v1 -- 0x7fada80a9440 con 0x7fada81383e0 2024-06-03T09:06:58.109 INFO:teuthology.orchestra.run.smithi088.stderr:2024-06-03T09:06:58.107+0000 7fad9effd640 1 -- 172.21.15.88:0/176455414 <== mon.2 v1:172.21.15.88:6790/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 2234374065 0 0) 0x7fada0001ec0 con 0x7fada81383e0 2024-06-03T09:06:58.109 INFO:teuthology.orchestra.run.smithi088.stderr:2024-06-03T09:06:58.107+0000 7fad9effd640 10 monclient(hunting): handle_monmap mon_map magic: 0 v1 2024-06-03T09:06:58.109 INFO:teuthology.orchestra.run.smithi088.stderr:2024-06-03T09:06:58.107+0000 7fad9effd640 10 monclient(hunting): got monmap 1 from mon.noname-c (according to old e1) 2024-06-03T09:06:58.110 INFO:teuthology.orchestra.run.smithi088.stderr:2024-06-03T09:06:58.107+0000 7fad9effd640 10 monclient(hunting): dump: 2024-06-03T09:06:58.110 INFO:teuthology.orchestra.run.smithi088.stderr:epoch 1 2024-06-03T09:06:58.110 INFO:teuthology.orchestra.run.smithi088.stderr:fsid ec3c3fdd-3ac6-4d7d-9412-683130ed8b04 2024-06-03T09:06:58.110 INFO:teuthology.orchestra.run.smithi088.stderr:last_changed 2024-06-03T08:34:31.885125+0000 2024-06-03T09:06:58.110 INFO:teuthology.orchestra.run.smithi088.stderr:created 2024-06-03T08:34:31.885125+0000 2024-06-03T09:06:58.110 INFO:teuthology.orchestra.run.smithi088.stderr:min_mon_release 18 (reef) 2024-06-03T09:06:58.110 INFO:teuthology.orchestra.run.smithi088.stderr:election_strategy: 3 2024-06-03T09:06:58.110 INFO:teuthology.orchestra.run.smithi088.stderr:0: v1:172.21.15.88:6789/0 mon.a 2024-06-03T09:06:58.110 INFO:teuthology.orchestra.run.smithi088.stderr:1: v1:172.21.15.192:6789/0 mon.b 2024-06-03T09:06:58.110 INFO:teuthology.orchestra.run.smithi088.stderr:2: v1:172.21.15.88:6790/0 mon.c 2024-06-03T09:06:58.110 INFO:teuthology.orchestra.run.smithi088.stderr: 2024-06-03T09:06:58.111 INFO:teuthology.orchestra.run.smithi088.stderr:2024-06-03T09:06:58.107+0000 7fad9effd640 10 monclient(hunting): _finish_auth 0 2024-06-03T09:06:58.111 INFO:teuthology.orchestra.run.smithi088.stderr:2024-06-03T09:06:58.108+0000 7fad9effd640 1 -- 172.21.15.88:0/176455414 <== mon.1 v1:172.21.15.192:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 764+0+0 (unknown 1706565168 0 0) 0x7fada8135b50 con 0x7fada80adb90 2024-06-03T09:06:58.111 INFO:teuthology.orchestra.run.smithi088.stderr:2024-06-03T09:06:58.108+0000 7fad9effd640 1 -- 172.21.15.88:0/176455414 --> v1:172.21.15.192:6789/0 -- auth(proto 2 165 bytes epoch 0) v1 -- 0x7fad8c004eb0 con 0x7fada80adb90 2024-06-03T09:06:58.111 INFO:teuthology.orchestra.run.smithi088.stderr:2024-06-03T09:06:58.108+0000 7fad9effd640 1 -- 172.21.15.88:0/176455414 <== mon.1 v1:172.21.15.192:6789/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 2234374065 0 0) 0x7fad98001870 con 0x7fada80adb90 2024-06-03T09:06:58.111 INFO:teuthology.orchestra.run.smithi088.stderr:2024-06-03T09:06:58.108+0000 7fad9effd640 10 monclient(hunting): handle_monmap mon_map magic: 0 v1 2024-06-03T09:06:58.111 INFO:teuthology.orchestra.run.smithi088.stderr:2024-06-03T09:06:58.108+0000 7fad9effd640 10 monclient(hunting): got monmap 1 from mon.b (according to old e1) 2024-06-03T09:06:58.111 INFO:teuthology.orchestra.run.smithi088.stderr:2024-06-03T09:06:58.108+0000 7fad9effd640 10 monclient(hunting): dump: 2024-06-03T09:06:58.111 INFO:teuthology.orchestra.run.smithi088.stderr:epoch 1 2024-06-03T09:06:58.111 INFO:teuthology.orchestra.run.smithi088.stderr:fsid ec3c3fdd-3ac6-4d7d-9412-683130ed8b04 2024-06-03T09:06:58.111 INFO:teuthology.orchestra.run.smithi088.stderr:last_changed 2024-06-03T08:34:31.885125+0000 2024-06-03T09:06:58.111 INFO:teuthology.orchestra.run.smithi088.stderr:created 2024-06-03T08:34:31.885125+0000 2024-06-03T09:06:58.112 INFO:teuthology.orchestra.run.smithi088.stderr:min_mon_release 18 (reef) 2024-06-03T09:06:58.112 INFO:teuthology.orchestra.run.smithi088.stderr:election_strategy: 3 2024-06-03T09:06:58.112 INFO:teuthology.orchestra.run.smithi088.stderr:0: v1:172.21.15.88:6789/0 mon.a 2024-06-03T09:06:58.112 INFO:teuthology.orchestra.run.smithi088.stderr:1: v1:172.21.15.192:6789/0 mon.b 2024-06-03T09:06:58.112 INFO:teuthology.orchestra.run.smithi088.stderr:2: v1:172.21.15.88:6790/0 mon.c 2024-06-03T09:06:58.112 INFO:teuthology.orchestra.run.smithi088.stderr: 2024-06-03T09:06:58.112 INFO:teuthology.orchestra.run.smithi088.stderr:2024-06-03T09:06:58.108+0000 7fad9effd640 1 -- 172.21.15.88:0/176455414 <== mon.0 v1:172.21.15.88:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 764+0+0 (unknown 695380182 0 0) 0x7fada80ad520 con 0x7fada806ee00 2024-06-03T09:06:58.112 INFO:teuthology.orchestra.run.smithi088.stderr:2024-06-03T09:06:58.108+0000 7fad9effd640 1 -- 172.21.15.88:0/176455414 --> v1:172.21.15.88:6789/0 -- auth(proto 2 165 bytes epoch 0) v1 -- 0x7fada8135b50 con 0x7fada806ee00 2024-06-03T09:06:58.112 INFO:teuthology.orchestra.run.smithi088.stderr:2024-06-03T09:06:58.108+0000 7fad9effd640 1 -- 172.21.15.88:0/176455414 <== mon.0 v1:172.21.15.88:6789/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 2234374065 0 0) 0x7fada40018b0 con 0x7fada806ee00 2024-06-03T09:06:58.112 INFO:teuthology.orchestra.run.smithi088.stderr:2024-06-03T09:06:58.108+0000 7fad9effd640 10 monclient(hunting): handle_monmap mon_map magic: 0 v1 2024-06-03T09:06:58.112 INFO:teuthology.orchestra.run.smithi088.stderr:2024-06-03T09:06:58.108+0000 7fad9effd640 10 monclient(hunting): got monmap 1 from mon.a (according to old e1) 2024-06-03T09:06:58.112 INFO:teuthology.orchestra.run.smithi088.stderr:2024-06-03T09:06:58.108+0000 7fad9effd640 10 monclient(hunting): dump: 2024-06-03T09:06:58.113 INFO:teuthology.orchestra.run.smithi088.stderr:epoch 1 2024-06-03T09:06:58.113 INFO:teuthology.orchestra.run.smithi088.stderr:fsid ec3c3fdd-3ac6-4d7d-9412-683130ed8b04 2024-06-03T09:06:58.113 INFO:teuthology.orchestra.run.smithi088.stderr:last_changed 2024-06-03T08:34:31.885125+0000 2024-06-03T09:06:58.113 INFO:teuthology.orchestra.run.smithi088.stderr:created 2024-06-03T08:34:31.885125+0000 2024-06-03T09:06:58.113 INFO:teuthology.orchestra.run.smithi088.stderr:min_mon_release 18 (reef) 2024-06-03T09:06:58.113 INFO:teuthology.orchestra.run.smithi088.stderr:election_strategy: 3 2024-06-03T09:06:58.113 INFO:teuthology.orchestra.run.smithi088.stderr:0: v1:172.21.15.88:6789/0 mon.a 2024-06-03T09:06:58.113 INFO:teuthology.orchestra.run.smithi088.stderr:1: v1:172.21.15.192:6789/0 mon.b 2024-06-03T09:06:58.113 INFO:teuthology.orchestra.run.smithi088.stderr:2: v1:172.21.15.88:6790/0 mon.c 2024-06-03T09:06:58.113 INFO:teuthology.orchestra.run.smithi088.stderr: 2024-06-03T09:06:58.114 INFO:teuthology.orchestra.run.smithi088.stderr:2024-06-03T09:06:58.111+0000 7fad9effd640 1 -- 172.21.15.88:0/176455414 <== mon.2 v1:172.21.15.88:6790/0 4 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 580+0+0 (unknown 991082751 0 0) 0x7fada80a9440 con 0x7fada81383e0 2024-06-03T09:06:58.115 INFO:teuthology.orchestra.run.smithi088.stderr:2024-06-03T09:06:58.111+0000 7fad9effd640 1 -- 172.21.15.88:0/176455414 >> v1:172.21.15.192:6789/0 conn(0x7fada80adb90 legacy=0x7fada8134d30 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-06-03T09:06:58.115 INFO:teuthology.orchestra.run.smithi088.stderr:2024-06-03T09:06:58.111+0000 7fad9effd640 1 -- 172.21.15.88:0/176455414 >> v1:172.21.15.88:6789/0 conn(0x7fada806ee00 legacy=0x7fada81d6e20 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-06-03T09:06:58.115 INFO:teuthology.orchestra.run.smithi088.stderr:2024-06-03T09:06:58.111+0000 7fad9effd640 10 monclient: _finish_hunting 0 2024-06-03T09:06:58.115 INFO:teuthology.orchestra.run.smithi088.stderr:2024-06-03T09:06:58.111+0000 7fad9effd640 1 monclient: found mon.c 2024-06-03T09:06:58.115 INFO:teuthology.orchestra.run.smithi088.stderr:2024-06-03T09:06:58.111+0000 7fad9effd640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-06-03T09:06:58.115 INFO:teuthology.orchestra.run.smithi088.stderr:2024-06-03T09:06:58.111+0000 7fad9effd640 10 monclient: _send_mon_message to mon.c at v1:172.21.15.88:6790/0 2024-06-03T09:06:58.115 INFO:teuthology.orchestra.run.smithi088.stderr:2024-06-03T09:06:58.111+0000 7fad9effd640 1 -- 172.21.15.88:0/176455414 --> v1:172.21.15.88:6790/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fada81d6900 con 0x7fada81383e0 2024-06-03T09:06:58.115 INFO:teuthology.orchestra.run.smithi088.stderr:2024-06-03T09:06:58.111+0000 7fad9effd640 10 monclient: _finish_auth 0 2024-06-03T09:06:58.115 INFO:teuthology.orchestra.run.smithi088.stderr:2024-06-03T09:06:58.111+0000 7fad9effd640 10 monclient: _check_auth_tickets 2024-06-03T09:06:58.115 INFO:teuthology.orchestra.run.smithi088.stderr:2024-06-03T09:06:58.111+0000 7fad9effd640 20 monclient: _check_auth_rotating not needed by client.admin 2024-06-03T09:06:58.115 INFO:teuthology.orchestra.run.smithi088.stderr:2024-06-03T09:06:58.111+0000 7fadb0781640 5 monclient: authenticate success, global_id 30275 2024-06-03T09:06:58.116 INFO:teuthology.orchestra.run.smithi088.stderr:2024-06-03T09:06:58.111+0000 7fadb0781640 10 monclient: _renew_subs 2024-06-03T09:06:58.116 INFO:teuthology.orchestra.run.smithi088.stderr:2024-06-03T09:06:58.111+0000 7fadb0781640 10 monclient: _send_mon_message to mon.c at v1:172.21.15.88:6790/0 2024-06-03T09:06:58.116 INFO:teuthology.orchestra.run.smithi088.stderr:2024-06-03T09:06:58.111+0000 7fadb0781640 1 -- 172.21.15.88:0/176455414 --> v1:172.21.15.88:6790/0 -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fada813ad50 con 0x7fada81383e0 2024-06-03T09:06:58.116 INFO:teuthology.orchestra.run.smithi088.stderr:2024-06-03T09:06:58.111+0000 7fadb0781640 10 client.30275.objecter _maybe_request_map subscribing (onetime) to next osd map 2024-06-03T09:06:58.116 INFO:teuthology.orchestra.run.smithi088.stderr:2024-06-03T09:06:58.111+0000 7fadb0781640 10 monclient: _renew_subs 2024-06-03T09:06:58.116 INFO:teuthology.orchestra.run.smithi088.stderr:2024-06-03T09:06:58.111+0000 7fadb0781640 10 monclient: _send_mon_message to mon.c at v1:172.21.15.88:6790/0 2024-06-03T09:06:58.116 INFO:teuthology.orchestra.run.smithi088.stderr:2024-06-03T09:06:58.111+0000 7fadb0781640 1 -- 172.21.15.88:0/176455414 --> v1:172.21.15.88:6790/0 -- mon_subscribe({osdmap=0}) v3 -- 0x7fada80a8450 con 0x7fada81383e0 2024-06-03T09:06:58.116 INFO:teuthology.orchestra.run.smithi088.stderr:2024-06-03T09:06:58.111+0000 7fadb0781640 1 librados: init done 2024-06-03T09:06:58.116 INFO:teuthology.orchestra.run.smithi088.stderr:2024-06-03T09:06:58.112+0000 7fadb0781640 10 start_mon_command cmd=[{"prefix": "get_command_descriptions"}] 2024-06-03T09:06:58.116 INFO:teuthology.orchestra.run.smithi088.stderr:2024-06-03T09:06:58.112+0000 7fadb0781640 10 monclient: _send_command 1 [{"prefix": "get_command_descriptions"}] 2024-06-03T09:06:58.116 INFO:teuthology.orchestra.run.smithi088.stderr:2024-06-03T09:06:58.112+0000 7fadb0781640 10 monclient: _send_mon_message to mon.c at v1:172.21.15.88:6790/0 2024-06-03T09:06:58.116 INFO:teuthology.orchestra.run.smithi088.stderr:2024-06-03T09:06:58.112+0000 7fadb0781640 1 -- 172.21.15.88:0/176455414 --> v1:172.21.15.88:6790/0 -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fada81d6760 con 0x7fada81383e0 2024-06-03T09:06:58.355 INFO:teuthology.orchestra.run.smithi088.stderr:2024-06-03T09:06:58.355+0000 7fad9effd640 1 -- 172.21.15.88:0/176455414 <== mon.2 v1:172.21.15.88:6790/0 5 ==== config(0 keys) v1 ==== 4+0+0 (unknown 0 0 0) 0x7fada0001530 con 0x7fada81383e0 2024-06-03T09:06:58.355 INFO:teuthology.orchestra.run.smithi088.stderr:2024-06-03T09:06:58.355+0000 7fad9effd640 10 monclient: handle_config config(0 keys) v1 2024-06-03T09:06:58.356 INFO:teuthology.orchestra.run.smithi088.stderr:2024-06-03T09:06:58.355+0000 7fad9effd640 1 -- 172.21.15.88:0/176455414 <== mon.2 v1:172.21.15.88:6790/0 6 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 2234374065 0 0) 0x7fada0003c20 con 0x7fada81383e0 2024-06-03T09:06:58.356 INFO:teuthology.orchestra.run.smithi088.stderr:2024-06-03T09:06:58.355+0000 7fad9effd640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-06-03T09:06:58.356 INFO:teuthology.orchestra.run.smithi088.stderr:2024-06-03T09:06:58.355+0000 7fad9effd640 10 monclient: got monmap 1 from mon.c (according to old e1) 2024-06-03T09:06:58.356 INFO:teuthology.orchestra.run.smithi088.stderr:2024-06-03T09:06:58.355+0000 7fad9effd640 10 monclient: dump: 2024-06-03T09:06:58.356 INFO:teuthology.orchestra.run.smithi088.stderr:epoch 1 2024-06-03T09:06:58.356 INFO:teuthology.orchestra.run.smithi088.stderr:fsid ec3c3fdd-3ac6-4d7d-9412-683130ed8b04 2024-06-03T09:06:58.356 INFO:teuthology.orchestra.run.smithi088.stderr:last_changed 2024-06-03T08:34:31.885125+0000 2024-06-03T09:06:58.356 INFO:teuthology.orchestra.run.smithi088.stderr:created 2024-06-03T08:34:31.885125+0000 2024-06-03T09:06:58.356 INFO:teuthology.orchestra.run.smithi088.stderr:min_mon_release 18 (reef) 2024-06-03T09:06:58.356 INFO:teuthology.orchestra.run.smithi088.stderr:election_strategy: 3 2024-06-03T09:06:58.356 INFO:teuthology.orchestra.run.smithi088.stderr:0: v1:172.21.15.88:6789/0 mon.a 2024-06-03T09:06:58.356 INFO:teuthology.orchestra.run.smithi088.stderr:1: v1:172.21.15.192:6789/0 mon.b 2024-06-03T09:06:58.356 INFO:teuthology.orchestra.run.smithi088.stderr:2: v1:172.21.15.88:6790/0 mon.c 2024-06-03T09:06:58.357 INFO:teuthology.orchestra.run.smithi088.stderr: 2024-06-03T09:06:58.357 INFO:teuthology.orchestra.run.smithi088.stderr:2024-06-03T09:06:58.355+0000 7fadacd1a640 4 set_mon_vals no callback set 2024-06-03T09:06:58.357 INFO:teuthology.orchestra.run.smithi088.stderr:2024-06-03T09:06:58.355+0000 7fad9effd640 1 -- 172.21.15.88:0/176455414 <== mon.2 v1:172.21.15.88:6790/0 7 ==== mgrmap(e 5) v1 ==== 89110+0+0 (unknown 873533973 0 0) 0x7fada0003d80 con 0x7fada81383e0 2024-06-03T09:06:58.357 INFO:teuthology.orchestra.run.smithi088.stderr:2024-06-03T09:06:58.356+0000 7fadae51d640 10 monclient: get_auth_request con 0x7fad8c06d3b0 auth_method 0 2024-06-03T09:06:58.357 INFO:teuthology.orchestra.run.smithi088.stderr:2024-06-03T09:06:58.356+0000 7fad9effd640 10 client.30275.objecter ms_handle_connect 0x7fad8c06d3b0 2024-06-03T09:06:58.361 INFO:teuthology.orchestra.run.smithi088.stderr:2024-06-03T09:06:58.360+0000 7fad9effd640 1 -- 172.21.15.88:0/176455414 <== mon.2 v1:172.21.15.88:6790/0 8 ==== osd_map(1014..1014 src has 622..1014) v4 ==== 5229+0+0 (unknown 3230192161 0 0) 0x7fada0054220 con 0x7fada81383e0 2024-06-03T09:06:58.361 INFO:teuthology.orchestra.run.smithi088.stderr:2024-06-03T09:06:58.360+0000 7fad9effd640 10 client.30275.objecter ms_dispatch 0x7fada8000b90 osd_map(1014..1014 src has 622..1014) v4 2024-06-03T09:06:58.361 INFO:teuthology.orchestra.run.smithi088.stderr:2024-06-03T09:06:58.360+0000 7fad9effd640 3 client.30275.objecter handle_osd_map got epochs [1014,1014] > 0 2024-06-03T09:06:58.361 INFO:teuthology.orchestra.run.smithi088.stderr:2024-06-03T09:06:58.360+0000 7fad9effd640 3 client.30275.objecter handle_osd_map decoding full epoch 1014 2024-06-03T09:06:58.361 INFO:teuthology.orchestra.run.smithi088.stderr:2024-06-03T09:06:58.360+0000 7fad9effd640 20 client.30275.objecter dump_active .. 0 homeless 2024-06-03T09:06:58.361 INFO:teuthology.orchestra.run.smithi088.stderr:2024-06-03T09:06:58.360+0000 7fad9effd640 1 -- 172.21.15.88:0/176455414 <== mon.2 v1:172.21.15.88:6790/0 9 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+182498 (unknown 1092875540 0 3016760700) 0x7fada004fdd0 con 0x7fada81383e0 2024-06-03T09:06:58.361 INFO:teuthology.orchestra.run.smithi088.stderr:2024-06-03T09:06:58.360+0000 7fad9effd640 10 monclient: handle_mon_command_ack 1 [{"prefix": "get_command_descriptions"}] 2024-06-03T09:06:58.362 INFO:teuthology.orchestra.run.smithi088.stderr:2024-06-03T09:06:58.360+0000 7fad9effd640 10 monclient: _finish_command 1 = system:0 2024-06-03T09:06:58.540 INFO:teuthology.orchestra.run.smithi088.stderr:2024-06-03T09:06:58.540+0000 7fadb0781640 1 -- 172.21.15.88:0/176455414 --> v1:172.21.15.192:6800/34229 -- mgr_command(tid 0: {"prefix": "osd deep-scrub", "who": "2", "target": ["mon-mgr", ""]}) v1 -- 0x7fada806ff80 con 0x7fad8c06d3b0 2024-06-03T09:06:58.541 INFO:teuthology.orchestra.run.smithi088.stderr:2024-06-03T09:06:58.541+0000 7fad9effd640 1 -- 172.21.15.88:0/176455414 <== mgr.4110 v1:172.21.15.192:6800/34229 1 ==== mgr_command_reply(tid 0: 0 instructed osd(s) 2 to deep-scrub) v1 ==== 41+0+0 (unknown 3049115190 0 0) 0x7fada40057a0 con 0x7fad8c06d3b0 2024-06-03T09:06:58.541 INFO:teuthology.orchestra.run.smithi088.stderr:instructed osd(s) 2 to deep-scrub 2024-06-03T09:06:58.543 INFO:teuthology.orchestra.run.smithi088.stderr:2024-06-03T09:06:58.543+0000 7fadb0781640 10 librados: watch_flush enter 2024-06-03T09:06:58.543 INFO:teuthology.orchestra.run.smithi088.stderr:2024-06-03T09:06:58.543+0000 7fadb0781640 10 librados: watch_flush exit 2024-06-03T09:06:58.543 INFO:teuthology.orchestra.run.smithi088.stderr:2024-06-03T09:06:58.543+0000 7fadb0781640 20 client.30275.objecter shutdown clearing up homeless session... 2024-06-03T09:06:58.543 INFO:teuthology.orchestra.run.smithi088.stderr:2024-06-03T09:06:58.543+0000 7fadb0781640 10 client.30275.objecter successfully canceled tick 2024-06-03T09:06:58.544 INFO:teuthology.orchestra.run.smithi088.stderr:2024-06-03T09:06:58.543+0000 7fadb0781640 1 -- 172.21.15.88:0/176455414 >> v1:172.21.15.192:6800/34229 conn(0x7fad8c06d3b0 legacy=0x7fad8c06f870 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-06-03T09:06:58.544 INFO:teuthology.orchestra.run.smithi088.stderr:2024-06-03T09:06:58.543+0000 7fadb0781640 10 monclient: shutdown 2024-06-03T09:06:58.544 INFO:teuthology.orchestra.run.smithi088.stderr:2024-06-03T09:06:58.543+0000 7fadb0781640 20 monclient: shutdown discarding 0 pending message(s) 2024-06-03T09:06:58.544 INFO:teuthology.orchestra.run.smithi088.stderr:2024-06-03T09:06:58.543+0000 7fadb0781640 1 -- 172.21.15.88:0/176455414 >> v1:172.21.15.88:6790/0 conn(0x7fada81383e0 legacy=0x7fada8135440 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-06-03T09:06:58.544 INFO:teuthology.orchestra.run.smithi088.stderr:2024-06-03T09:06:58.543+0000 7fadb0781640 1 -- 172.21.15.88:0/176455414 shutdown_connections 2024-06-03T09:06:58.544 INFO:teuthology.orchestra.run.smithi088.stderr:2024-06-03T09:06:58.543+0000 7fadb0781640 1 -- 172.21.15.88:0/176455414 >> 172.21.15.88:0/176455414 conn(0x7fada80b2f20 msgr2=0x7fada813eb60 unknown :-1 s=STATE_NONE l=0).mark_down 2024-06-03T09:06:58.544 INFO:teuthology.orchestra.run.smithi088.stderr:2024-06-03T09:06:58.543+0000 7fadb0781640 1 -- 172.21.15.88:0/176455414 shutdown_connections 2024-06-03T09:06:58.544 INFO:teuthology.orchestra.run.smithi088.stderr:2024-06-03T09:06:58.543+0000 7fadb0781640 1 -- 172.21.15.88:0/176455414 wait complete. 2024-06-03T09:06:58.544 INFO:teuthology.orchestra.run.smithi088.stderr:2024-06-03T09:06:58.543+0000 7fadb0781640 1 librados: shutdown 2024-06-03T09:06:58.554 INFO:tasks.ceph:Scrubbing osd.3 2024-06-03T09:06:58.555 DEBUG:teuthology.orchestra.run.smithi088:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph tell osd.3 config set osd_debug_deep_scrub_sleep 0 2024-06-03T09:06:58.656 INFO:teuthology.orchestra.run.smithi088.stderr:2024-06-03T09:06:58.656+0000 7fd2b85eb640 10 monclient: get_monmap_and_config 2024-06-03T09:06:58.656 INFO:teuthology.orchestra.run.smithi088.stderr:2024-06-03T09:06:58.656+0000 7fd2b85eb640 10 monclient: build_initial_monmap 2024-06-03T09:06:58.656 INFO:teuthology.orchestra.run.smithi088.stderr:2024-06-03T09:06:58.656+0000 7fd2b85eb640 10 monclient: monmap: 2024-06-03T09:06:58.656 INFO:teuthology.orchestra.run.smithi088.stderr:epoch 0 2024-06-03T09:06:58.657 INFO:teuthology.orchestra.run.smithi088.stderr:fsid ec3c3fdd-3ac6-4d7d-9412-683130ed8b04 2024-06-03T09:06:58.657 INFO:teuthology.orchestra.run.smithi088.stderr:last_changed 2024-06-03T09:06:58.657477+0000 2024-06-03T09:06:58.657 INFO:teuthology.orchestra.run.smithi088.stderr:created 2024-06-03T09:06:58.657477+0000 2024-06-03T09:06:58.657 INFO:teuthology.orchestra.run.smithi088.stderr:min_mon_release 0 (unknown) 2024-06-03T09:06:58.657 INFO:teuthology.orchestra.run.smithi088.stderr:election_strategy: 3 2024-06-03T09:06:58.657 INFO:teuthology.orchestra.run.smithi088.stderr:0: v1:172.21.15.88:6789/0 mon.noname-a 2024-06-03T09:06:58.657 INFO:teuthology.orchestra.run.smithi088.stderr:1: v1:172.21.15.192:6789/0 mon.noname-c 2024-06-03T09:06:58.657 INFO:teuthology.orchestra.run.smithi088.stderr:2: v1:172.21.15.88:6790/0 mon.noname-b 2024-06-03T09:06:58.657 INFO:teuthology.orchestra.run.smithi088.stderr: 2024-06-03T09:06:58.658 INFO:teuthology.orchestra.run.smithi088.stderr:2024-06-03T09:06:58.657+0000 7fd2b85eb640 1 Processor -- start 2024-06-03T09:06:58.659 INFO:teuthology.orchestra.run.smithi088.stderr:2024-06-03T09:06:58.657+0000 7fd2b85eb640 1 -- start start 2024-06-03T09:06:58.659 INFO:teuthology.orchestra.run.smithi088.stderr:2024-06-03T09:06:58.657+0000 7fd2b85eb640 10 monclient: init 2024-06-03T09:06:58.659 INFO:teuthology.orchestra.run.smithi088.stderr:2024-06-03T09:06:58.657+0000 7fd2b85eb640 10 monclient: _reopen_session rank -1 2024-06-03T09:06:58.659 INFO:teuthology.orchestra.run.smithi088.stderr:2024-06-03T09:06:58.657+0000 7fd2b85eb640 10 monclient: _add_conns ranks=[1,0,2] 2024-06-03T09:06:58.659 INFO:teuthology.orchestra.run.smithi088.stderr:2024-06-03T09:06:58.657+0000 7fd2b85eb640 10 monclient(hunting): picked mon.noname-c con 0x7fd2b006e050 addr v1:172.21.15.192:6789/0 2024-06-03T09:06:58.659 INFO:teuthology.orchestra.run.smithi088.stderr:2024-06-03T09:06:58.657+0000 7fd2b85eb640 10 monclient(hunting): picked mon.noname-a con 0x7fd2b006e430 addr v1:172.21.15.88:6789/0 2024-06-03T09:06:58.659 INFO:teuthology.orchestra.run.smithi088.stderr:2024-06-03T09:06:58.657+0000 7fd2b85eb640 10 monclient(hunting): picked mon.noname-b con 0x7fd2b00717c0 addr v1:172.21.15.88:6790/0 2024-06-03T09:06:58.659 INFO:teuthology.orchestra.run.smithi088.stderr:2024-06-03T09:06:58.657+0000 7fd2b85eb640 1 -- --> v1:172.21.15.88:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fd2b0070c10 con 0x7fd2b006e430 2024-06-03T09:06:58.659 INFO:teuthology.orchestra.run.smithi088.stderr:2024-06-03T09:06:58.657+0000 7fd2b85eb640 1 -- --> v1:172.21.15.192:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fd2b0070230 con 0x7fd2b006e050 2024-06-03T09:06:58.659 INFO:teuthology.orchestra.run.smithi088.stderr:2024-06-03T09:06:58.657+0000 7fd2b85eb640 1 -- --> v1:172.21.15.88:6790/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fd2b0070510 con 0x7fd2b00717c0 2024-06-03T09:06:58.659 INFO:teuthology.orchestra.run.smithi088.stderr:2024-06-03T09:06:58.657+0000 7fd2b85eb640 10 monclient(hunting): _renew_subs 2024-06-03T09:06:58.659 INFO:teuthology.orchestra.run.smithi088.stderr:2024-06-03T09:06:58.657+0000 7fd2b85eb640 10 monclient(hunting): authenticate will time out at 3074.467883s 2024-06-03T09:06:58.659 INFO:teuthology.orchestra.run.smithi088.stderr:2024-06-03T09:06:58.658+0000 7fd2b6b88640 1 --1- >> v1:172.21.15.88:6790/0 conn(0x7fd2b00717c0 0x7fd2b0149240 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.88:6790/0 says I am v1:172.21.15.88:51050/0 (socket says 172.21.15.88:51050) 2024-06-03T09:06:58.660 INFO:teuthology.orchestra.run.smithi088.stderr:2024-06-03T09:06:58.658+0000 7fd2b6b88640 1 -- 172.21.15.88:0/3183569582 learned_addr learned my addr 172.21.15.88:0/3183569582 (peer_addr_for_me v1:172.21.15.88:0/0) 2024-06-03T09:06:58.661 INFO:teuthology.orchestra.run.smithi088.stderr:2024-06-03T09:06:58.660+0000 7fd2b5385640 1 -- 172.21.15.88:0/3183569582 <== mon.1 v1:172.21.15.192:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 264736503 0 0) 0x7fd2b0070230 con 0x7fd2b006e050 2024-06-03T09:06:58.661 INFO:teuthology.orchestra.run.smithi088.stderr:2024-06-03T09:06:58.660+0000 7fd2b5385640 10 monclient(hunting): _negotiate 2024-06-03T09:06:58.661 INFO:teuthology.orchestra.run.smithi088.stderr:2024-06-03T09:06:58.660+0000 7fd2b5385640 10 monclient(hunting): _init_auth method 2 2024-06-03T09:06:58.661 INFO:teuthology.orchestra.run.smithi088.stderr:2024-06-03T09:06:58.660+0000 7fd2b5385640 10 monclient(hunting): _init_auth creating new auth 2024-06-03T09:06:58.661 INFO:teuthology.orchestra.run.smithi088.stderr:2024-06-03T09:06:58.660+0000 7fd2b5385640 10 monclient(hunting): my global_id is 30343 2024-06-03T09:06:58.661 INFO:teuthology.orchestra.run.smithi088.stderr:2024-06-03T09:06:58.660+0000 7fd2b5385640 1 -- 172.21.15.88:0/3183569582 --> v1:172.21.15.192:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7fd29c003520 con 0x7fd2b006e050 2024-06-03T09:06:58.661 INFO:teuthology.orchestra.run.smithi088.stderr:2024-06-03T09:06:58.660+0000 7fd2b5385640 1 -- 172.21.15.88:0/3183569582 <== mon.0 v1:172.21.15.88:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 1361743160 0 0) 0x7fd2b0070c10 con 0x7fd2b006e430 2024-06-03T09:06:58.661 INFO:teuthology.orchestra.run.smithi088.stderr:2024-06-03T09:06:58.660+0000 7fd2b5385640 10 monclient(hunting): _negotiate 2024-06-03T09:06:58.661 INFO:teuthology.orchestra.run.smithi088.stderr:2024-06-03T09:06:58.660+0000 7fd2b5385640 10 monclient(hunting): _init_auth method 2 2024-06-03T09:06:58.661 INFO:teuthology.orchestra.run.smithi088.stderr:2024-06-03T09:06:58.660+0000 7fd2b5385640 10 monclient(hunting): _init_auth creating new auth 2024-06-03T09:06:58.661 INFO:teuthology.orchestra.run.smithi088.stderr:2024-06-03T09:06:58.660+0000 7fd2b5385640 10 monclient(hunting): my global_id is 30771 2024-06-03T09:06:58.662 INFO:teuthology.orchestra.run.smithi088.stderr:2024-06-03T09:06:58.660+0000 7fd2b5385640 1 -- 172.21.15.88:0/3183569582 --> v1:172.21.15.88:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7fd2b0070230 con 0x7fd2b006e430 2024-06-03T09:06:58.662 INFO:teuthology.orchestra.run.smithi088.stderr:2024-06-03T09:06:58.660+0000 7fd2b5385640 1 -- 172.21.15.88:0/3183569582 <== mon.2 v1:172.21.15.88:6790/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 2281221951 0 0) 0x7fd2b0070510 con 0x7fd2b00717c0 2024-06-03T09:06:58.662 INFO:teuthology.orchestra.run.smithi088.stderr:2024-06-03T09:06:58.660+0000 7fd2b5385640 10 monclient(hunting): _negotiate 2024-06-03T09:06:58.662 INFO:teuthology.orchestra.run.smithi088.stderr:2024-06-03T09:06:58.660+0000 7fd2b5385640 10 monclient(hunting): _init_auth method 2 2024-06-03T09:06:58.662 INFO:teuthology.orchestra.run.smithi088.stderr:2024-06-03T09:06:58.660+0000 7fd2b5385640 10 monclient(hunting): _init_auth creating new auth 2024-06-03T09:06:58.662 INFO:teuthology.orchestra.run.smithi088.stderr:2024-06-03T09:06:58.660+0000 7fd2b5385640 10 monclient(hunting): my global_id is 30278 2024-06-03T09:06:58.662 INFO:teuthology.orchestra.run.smithi088.stderr:2024-06-03T09:06:58.660+0000 7fd2b5385640 1 -- 172.21.15.88:0/3183569582 --> v1:172.21.15.88:6790/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7fd2b0070c10 con 0x7fd2b00717c0 2024-06-03T09:06:58.664 INFO:teuthology.orchestra.run.smithi088.stderr:2024-06-03T09:06:58.664+0000 7fd2b5385640 1 -- 172.21.15.88:0/3183569582 <== mon.0 v1:172.21.15.88:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 214+0+0 (unknown 2951192727 0 0) 0x7fd2b0070230 con 0x7fd2b006e430 2024-06-03T09:06:58.664 INFO:teuthology.orchestra.run.smithi088.stderr:2024-06-03T09:06:58.664+0000 7fd2b5385640 1 -- 172.21.15.88:0/3183569582 >> v1:172.21.15.88:6790/0 conn(0x7fd2b00717c0 legacy=0x7fd2b0149240 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-06-03T09:06:58.664 INFO:teuthology.orchestra.run.smithi088.stderr:2024-06-03T09:06:58.664+0000 7fd2b5385640 1 -- 172.21.15.88:0/3183569582 >> v1:172.21.15.192:6789/0 conn(0x7fd2b006e050 legacy=0x7fd2b013ad00 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-06-03T09:06:58.664 INFO:teuthology.orchestra.run.smithi088.stderr:2024-06-03T09:06:58.664+0000 7fd2b5385640 10 monclient: _finish_hunting 0 2024-06-03T09:06:58.664 INFO:teuthology.orchestra.run.smithi088.stderr:2024-06-03T09:06:58.664+0000 7fd2b5385640 1 monclient: found mon.noname-a 2024-06-03T09:06:58.664 INFO:teuthology.orchestra.run.smithi088.stderr:2024-06-03T09:06:58.664+0000 7fd2b5385640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-06-03T09:06:58.664 INFO:teuthology.orchestra.run.smithi088.stderr:2024-06-03T09:06:58.664+0000 7fd2b5385640 10 monclient: _send_mon_message to mon.noname-a at v1:172.21.15.88:6789/0 2024-06-03T09:06:58.664 INFO:teuthology.orchestra.run.smithi088.stderr:2024-06-03T09:06:58.664+0000 7fd2b5385640 1 -- 172.21.15.88:0/3183569582 --> v1:172.21.15.88:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fd2b00707f0 con 0x7fd2b006e430 2024-06-03T09:06:58.665 INFO:teuthology.orchestra.run.smithi088.stderr:2024-06-03T09:06:58.664+0000 7fd2b5385640 10 monclient: _finish_auth 0 2024-06-03T09:06:58.665 INFO:teuthology.orchestra.run.smithi088.stderr:2024-06-03T09:06:58.664+0000 7fd2b5385640 10 monclient: _check_auth_tickets 2024-06-03T09:06:58.665 INFO:teuthology.orchestra.run.smithi088.stderr:2024-06-03T09:06:58.664+0000 7fd2b5385640 20 monclient: _check_auth_rotating not needed by client.admin 2024-06-03T09:06:58.665 INFO:teuthology.orchestra.run.smithi088.stderr:2024-06-03T09:06:58.664+0000 7fd2b5385640 1 -- 172.21.15.88:0/3183569582 <== mon.0 v1:172.21.15.88:6789/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 2234374065 0 0) 0x7fd2a00015f0 con 0x7fd2b006e430 2024-06-03T09:06:58.665 INFO:teuthology.orchestra.run.smithi088.stderr:2024-06-03T09:06:58.664+0000 7fd2b85eb640 5 monclient: authenticate success, global_id 30771 2024-06-03T09:06:58.665 INFO:teuthology.orchestra.run.smithi088.stderr:2024-06-03T09:06:58.664+0000 7fd2b85eb640 20 monclient: get_monmap_and_config waiting for monmap|config 2024-06-03T09:06:58.665 INFO:teuthology.orchestra.run.smithi088.stderr:2024-06-03T09:06:58.664+0000 7fd2b5385640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-06-03T09:06:58.665 INFO:teuthology.orchestra.run.smithi088.stderr:2024-06-03T09:06:58.664+0000 7fd2b5385640 10 monclient: got monmap 1 from mon.noname-a (according to old e1) 2024-06-03T09:06:58.665 INFO:teuthology.orchestra.run.smithi088.stderr:2024-06-03T09:06:58.664+0000 7fd2b5385640 10 monclient: dump: 2024-06-03T09:06:58.665 INFO:teuthology.orchestra.run.smithi088.stderr:epoch 1 2024-06-03T09:06:58.665 INFO:teuthology.orchestra.run.smithi088.stderr:fsid ec3c3fdd-3ac6-4d7d-9412-683130ed8b04 2024-06-03T09:06:58.665 INFO:teuthology.orchestra.run.smithi088.stderr:last_changed 2024-06-03T08:34:31.885125+0000 2024-06-03T09:06:58.665 INFO:teuthology.orchestra.run.smithi088.stderr:created 2024-06-03T08:34:31.885125+0000 2024-06-03T09:06:58.665 INFO:teuthology.orchestra.run.smithi088.stderr:min_mon_release 18 (reef) 2024-06-03T09:06:58.666 INFO:teuthology.orchestra.run.smithi088.stderr:election_strategy: 3 2024-06-03T09:06:58.666 INFO:teuthology.orchestra.run.smithi088.stderr:0: v1:172.21.15.88:6789/0 mon.a 2024-06-03T09:06:58.666 INFO:teuthology.orchestra.run.smithi088.stderr:1: v1:172.21.15.192:6789/0 mon.b 2024-06-03T09:06:58.666 INFO:teuthology.orchestra.run.smithi088.stderr:2: v1:172.21.15.88:6790/0 mon.c 2024-06-03T09:06:58.666 INFO:teuthology.orchestra.run.smithi088.stderr: 2024-06-03T09:06:58.666 INFO:teuthology.orchestra.run.smithi088.stderr:2024-06-03T09:06:58.664+0000 7fd2b85eb640 20 monclient: get_monmap_and_config waiting for monmap|config 2024-06-03T09:06:58.669 INFO:teuthology.orchestra.run.smithi088.stderr:2024-06-03T09:06:58.665+0000 7fd2b5385640 1 -- 172.21.15.88:0/3183569582 <== mon.0 v1:172.21.15.88:6789/0 4 ==== config(0 keys) v1 ==== 4+0+0 (unknown 0 0 0) 0x7fd2a0001890 con 0x7fd2b006e430 2024-06-03T09:06:58.669 INFO:teuthology.orchestra.run.smithi088.stderr:2024-06-03T09:06:58.665+0000 7fd2b5385640 10 monclient: handle_config config(0 keys) v1 2024-06-03T09:06:58.669 INFO:teuthology.orchestra.run.smithi088.stderr:2024-06-03T09:06:58.665+0000 7fd2b5385640 1 -- 172.21.15.88:0/3183569582 <== mon.0 v1:172.21.15.88:6789/0 5 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 2234374065 0 0) 0x7fd2a0001c60 con 0x7fd2b006e430 2024-06-03T09:06:58.669 INFO:teuthology.orchestra.run.smithi088.stderr:2024-06-03T09:06:58.665+0000 7fd2b5385640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-06-03T09:06:58.669 INFO:teuthology.orchestra.run.smithi088.stderr:2024-06-03T09:06:58.665+0000 7fd2b5385640 10 monclient: got monmap 1 from mon.a (according to old e1) 2024-06-03T09:06:58.669 INFO:teuthology.orchestra.run.smithi088.stderr:2024-06-03T09:06:58.665+0000 7fd2b5385640 10 monclient: dump: 2024-06-03T09:06:58.669 INFO:teuthology.orchestra.run.smithi088.stderr:epoch 1 2024-06-03T09:06:58.669 INFO:teuthology.orchestra.run.smithi088.stderr:fsid ec3c3fdd-3ac6-4d7d-9412-683130ed8b04 2024-06-03T09:06:58.669 INFO:teuthology.orchestra.run.smithi088.stderr:last_changed 2024-06-03T08:34:31.885125+0000 2024-06-03T09:06:58.670 INFO:teuthology.orchestra.run.smithi088.stderr:created 2024-06-03T08:34:31.885125+0000 2024-06-03T09:06:58.670 INFO:teuthology.orchestra.run.smithi088.stderr:min_mon_release 18 (reef) 2024-06-03T09:06:58.670 INFO:teuthology.orchestra.run.smithi088.stderr:election_strategy: 3 2024-06-03T09:06:58.670 INFO:teuthology.orchestra.run.smithi088.stderr:0: v1:172.21.15.88:6789/0 mon.a 2024-06-03T09:06:58.670 INFO:teuthology.orchestra.run.smithi088.stderr:1: v1:172.21.15.192:6789/0 mon.b 2024-06-03T09:06:58.670 INFO:teuthology.orchestra.run.smithi088.stderr:2: v1:172.21.15.88:6790/0 mon.c 2024-06-03T09:06:58.670 INFO:teuthology.orchestra.run.smithi088.stderr: 2024-06-03T09:06:58.670 INFO:teuthology.orchestra.run.smithi088.stderr:2024-06-03T09:06:58.665+0000 7fd2b85eb640 10 monclient: get_monmap_and_config success 2024-06-03T09:06:58.670 INFO:teuthology.orchestra.run.smithi088.stderr:2024-06-03T09:06:58.665+0000 7fd2b85eb640 4 set_mon_vals no callback set 2024-06-03T09:06:58.670 INFO:teuthology.orchestra.run.smithi088.stderr:2024-06-03T09:06:58.666+0000 7fd2b85eb640 10 monclient: shutdown 2024-06-03T09:06:58.670 INFO:teuthology.orchestra.run.smithi088.stderr:2024-06-03T09:06:58.666+0000 7fd2b85eb640 20 monclient: shutdown discarding 0 pending message(s) 2024-06-03T09:06:58.670 INFO:teuthology.orchestra.run.smithi088.stderr:2024-06-03T09:06:58.666+0000 7fd2b85eb640 1 -- 172.21.15.88:0/3183569582 >> v1:172.21.15.88:6789/0 conn(0x7fd2b006e430 legacy=0x7fd2b00710b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-06-03T09:06:58.670 INFO:teuthology.orchestra.run.smithi088.stderr:2024-06-03T09:06:58.666+0000 7fd2b85eb640 1 -- 172.21.15.88:0/3183569582 shutdown_connections 2024-06-03T09:06:58.671 INFO:teuthology.orchestra.run.smithi088.stderr:2024-06-03T09:06:58.666+0000 7fd2b85eb640 1 -- 172.21.15.88:0/3183569582 >> 172.21.15.88:0/3183569582 conn(0x7fd2b00b2f20 msgr2=0x7fd2b00b3300 unknown :-1 s=STATE_NONE l=0).mark_down 2024-06-03T09:06:58.671 INFO:teuthology.orchestra.run.smithi088.stderr:2024-06-03T09:06:58.666+0000 7fd2b85eb640 1 -- 172.21.15.88:0/3183569582 shutdown_connections 2024-06-03T09:06:58.671 INFO:teuthology.orchestra.run.smithi088.stderr:2024-06-03T09:06:58.666+0000 7fd2b85eb640 1 -- 172.21.15.88:0/3183569582 wait complete. 2024-06-03T09:06:58.671 INFO:teuthology.orchestra.run.smithi088.stderr:2024-06-03T09:06:58.667+0000 7fd2b85eb640 10 monclient: build_initial_monmap 2024-06-03T09:06:58.671 INFO:teuthology.orchestra.run.smithi088.stderr:2024-06-03T09:06:58.667+0000 7fd2b85eb640 10 monclient: monmap: 2024-06-03T09:06:58.671 INFO:teuthology.orchestra.run.smithi088.stderr:epoch 0 2024-06-03T09:06:58.671 INFO:teuthology.orchestra.run.smithi088.stderr:fsid 00000000-0000-0000-0000-000000000000 2024-06-03T09:06:58.671 INFO:teuthology.orchestra.run.smithi088.stderr:last_changed 0.000000 2024-06-03T09:06:58.671 INFO:teuthology.orchestra.run.smithi088.stderr:created 0.000000 2024-06-03T09:06:58.671 INFO:teuthology.orchestra.run.smithi088.stderr:min_mon_release 0 (unknown) 2024-06-03T09:06:58.671 INFO:teuthology.orchestra.run.smithi088.stderr:election_strategy: 1 2024-06-03T09:06:58.671 INFO:teuthology.orchestra.run.smithi088.stderr:0: v1:172.21.15.88:6789/0 mon.noname-a 2024-06-03T09:06:58.671 INFO:teuthology.orchestra.run.smithi088.stderr:1: v1:172.21.15.192:6789/0 mon.noname-b 2024-06-03T09:06:58.671 INFO:teuthology.orchestra.run.smithi088.stderr:2: v1:172.21.15.88:6790/0 mon.noname-c 2024-06-03T09:06:58.672 INFO:teuthology.orchestra.run.smithi088.stderr: 2024-06-03T09:06:58.672 INFO:teuthology.orchestra.run.smithi088.stderr:2024-06-03T09:06:58.667+0000 7fd2b85eb640 1 librados: starting msgr at 2024-06-03T09:06:58.672 INFO:teuthology.orchestra.run.smithi088.stderr:2024-06-03T09:06:58.667+0000 7fd2b85eb640 1 librados: starting objecter 2024-06-03T09:06:58.672 INFO:teuthology.orchestra.run.smithi088.stderr:2024-06-03T09:06:58.667+0000 7fd2b85eb640 1 Processor -- start 2024-06-03T09:06:58.672 INFO:teuthology.orchestra.run.smithi088.stderr:2024-06-03T09:06:58.667+0000 7fd2b85eb640 1 -- start start 2024-06-03T09:06:58.672 INFO:teuthology.orchestra.run.smithi088.stderr:2024-06-03T09:06:58.667+0000 7fd2b85eb640 1 librados: setting wanted keys 2024-06-03T09:06:58.672 INFO:teuthology.orchestra.run.smithi088.stderr:2024-06-03T09:06:58.667+0000 7fd2b85eb640 1 librados: calling monclient init 2024-06-03T09:06:58.672 INFO:teuthology.orchestra.run.smithi088.stderr:2024-06-03T09:06:58.667+0000 7fd2b85eb640 10 monclient: init 2024-06-03T09:06:58.672 INFO:teuthology.orchestra.run.smithi088.stderr:2024-06-03T09:06:58.668+0000 7fd2b85eb640 10 monclient: _reopen_session rank -1 2024-06-03T09:06:58.672 INFO:teuthology.orchestra.run.smithi088.stderr:2024-06-03T09:06:58.668+0000 7fd2b85eb640 10 monclient: _add_conns ranks=[0,2,1] 2024-06-03T09:06:58.672 INFO:teuthology.orchestra.run.smithi088.stderr:2024-06-03T09:06:58.668+0000 7fd2b85eb640 10 monclient(hunting): picked mon.noname-a con 0x7fd2b006e050 addr v1:172.21.15.88:6789/0 2024-06-03T09:06:58.672 INFO:teuthology.orchestra.run.smithi088.stderr:2024-06-03T09:06:58.668+0000 7fd2b85eb640 10 monclient(hunting): picked mon.noname-c con 0x7fd2b00717c0 addr v1:172.21.15.88:6790/0 2024-06-03T09:06:58.672 INFO:teuthology.orchestra.run.smithi088.stderr:2024-06-03T09:06:58.668+0000 7fd2b85eb640 10 monclient(hunting): picked mon.noname-b con 0x7fd2b0143640 addr v1:172.21.15.192:6789/0 2024-06-03T09:06:58.672 INFO:teuthology.orchestra.run.smithi088.stderr:2024-06-03T09:06:58.668+0000 7fd2b85eb640 1 -- --> v1:172.21.15.88:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fd2b013e250 con 0x7fd2b006e050 2024-06-03T09:06:58.673 INFO:teuthology.orchestra.run.smithi088.stderr:2024-06-03T09:06:58.668+0000 7fd2b85eb640 1 -- --> v1:172.21.15.192:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fd2b01457c0 con 0x7fd2b0143640 2024-06-03T09:06:58.673 INFO:teuthology.orchestra.run.smithi088.stderr:2024-06-03T09:06:58.668+0000 7fd2b85eb640 1 -- --> v1:172.21.15.88:6790/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fd2b0145a30 con 0x7fd2b00717c0 2024-06-03T09:06:58.673 INFO:teuthology.orchestra.run.smithi088.stderr:2024-06-03T09:06:58.668+0000 7fd2b85eb640 10 monclient(hunting): _renew_subs 2024-06-03T09:06:58.673 INFO:teuthology.orchestra.run.smithi088.stderr:2024-06-03T09:06:58.668+0000 7fd2b85eb640 10 monclient(hunting): authenticate will time out at 3074.478302s 2024-06-03T09:06:58.673 INFO:teuthology.orchestra.run.smithi088.stderr:2024-06-03T09:06:58.668+0000 7fd2b5b86640 1 --1- >> v1:172.21.15.88:6790/0 conn(0x7fd2b00717c0 0x7fd2b0144070 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=1).handle_server_banner_and_identify peer v1:172.21.15.88:6790/0 says I am v1:172.21.15.88:51052/0 (socket says 172.21.15.88:51052) 2024-06-03T09:06:58.673 INFO:teuthology.orchestra.run.smithi088.stderr:2024-06-03T09:06:58.668+0000 7fd2b5b86640 1 -- 172.21.15.88:0/1531698337 learned_addr learned my addr 172.21.15.88:0/1531698337 (peer_addr_for_me v1:172.21.15.88:0/0) 2024-06-03T09:06:58.673 INFO:teuthology.orchestra.run.smithi088.stderr:2024-06-03T09:06:58.668+0000 7fd2a6ffd640 10 client.?.objecter ms_handle_connect 0x7fd2b00717c0 2024-06-03T09:06:58.673 INFO:teuthology.orchestra.run.smithi088.stderr:2024-06-03T09:06:58.668+0000 7fd2a6ffd640 10 client.?.objecter resend_mon_ops 2024-06-03T09:06:58.673 INFO:teuthology.orchestra.run.smithi088.stderr:2024-06-03T09:06:58.669+0000 7fd2a6ffd640 10 client.?.objecter ms_handle_connect 0x7fd2b006e050 2024-06-03T09:06:58.673 INFO:teuthology.orchestra.run.smithi088.stderr:2024-06-03T09:06:58.669+0000 7fd2a6ffd640 10 client.?.objecter resend_mon_ops 2024-06-03T09:06:58.673 INFO:teuthology.orchestra.run.smithi088.stderr:2024-06-03T09:06:58.669+0000 7fd2a6ffd640 10 client.?.objecter ms_handle_connect 0x7fd2b0143640 2024-06-03T09:06:58.673 INFO:teuthology.orchestra.run.smithi088.stderr:2024-06-03T09:06:58.669+0000 7fd2a6ffd640 10 client.?.objecter resend_mon_ops 2024-06-03T09:06:58.674 INFO:teuthology.orchestra.run.smithi088.stderr:2024-06-03T09:06:58.670+0000 7fd2a6ffd640 1 -- 172.21.15.88:0/1531698337 <== mon.2 v1:172.21.15.88:6790/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 1994003611 0 0) 0x7fd2b0145a30 con 0x7fd2b00717c0 2024-06-03T09:06:58.674 INFO:teuthology.orchestra.run.smithi088.stderr:2024-06-03T09:06:58.670+0000 7fd2a6ffd640 10 monclient(hunting): _negotiate 2024-06-03T09:06:58.674 INFO:teuthology.orchestra.run.smithi088.stderr:2024-06-03T09:06:58.670+0000 7fd2a6ffd640 10 monclient(hunting): _init_auth method 2 2024-06-03T09:06:58.675 INFO:teuthology.orchestra.run.smithi088.stderr:2024-06-03T09:06:58.670+0000 7fd2a6ffd640 10 monclient(hunting): _init_auth creating new auth 2024-06-03T09:06:58.675 INFO:teuthology.orchestra.run.smithi088.stderr:2024-06-03T09:06:58.670+0000 7fd2a6ffd640 10 monclient(hunting): my global_id is 30281 2024-06-03T09:06:58.675 INFO:teuthology.orchestra.run.smithi088.stderr:2024-06-03T09:06:58.670+0000 7fd2a6ffd640 1 -- 172.21.15.88:0/1531698337 --> v1:172.21.15.88:6790/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7fd294004eb0 con 0x7fd2b00717c0 2024-06-03T09:06:58.675 INFO:teuthology.orchestra.run.smithi088.stderr:2024-06-03T09:06:58.671+0000 7fd2a6ffd640 1 -- 172.21.15.88:0/1531698337 <== mon.0 v1:172.21.15.88:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 1038190891 0 0) 0x7fd2b013e250 con 0x7fd2b006e050 2024-06-03T09:06:58.675 INFO:teuthology.orchestra.run.smithi088.stderr:2024-06-03T09:06:58.671+0000 7fd2a6ffd640 10 monclient(hunting): _negotiate 2024-06-03T09:06:58.675 INFO:teuthology.orchestra.run.smithi088.stderr:2024-06-03T09:06:58.671+0000 7fd2a6ffd640 10 monclient(hunting): _init_auth method 2 2024-06-03T09:06:58.675 INFO:teuthology.orchestra.run.smithi088.stderr:2024-06-03T09:06:58.671+0000 7fd2a6ffd640 10 monclient(hunting): _init_auth creating new auth 2024-06-03T09:06:58.675 INFO:teuthology.orchestra.run.smithi088.stderr:2024-06-03T09:06:58.671+0000 7fd2a6ffd640 10 monclient(hunting): my global_id is 30774 2024-06-03T09:06:58.675 INFO:teuthology.orchestra.run.smithi088.stderr:2024-06-03T09:06:58.671+0000 7fd2a6ffd640 1 -- 172.21.15.88:0/1531698337 --> v1:172.21.15.88:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7fd2b0145a30 con 0x7fd2b006e050 2024-06-03T09:06:58.675 INFO:teuthology.orchestra.run.smithi088.stderr:2024-06-03T09:06:58.671+0000 7fd2a6ffd640 1 -- 172.21.15.88:0/1531698337 <== mon.1 v1:172.21.15.192:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 34106622 0 0) 0x7fd2b01457c0 con 0x7fd2b0143640 2024-06-03T09:06:58.675 INFO:teuthology.orchestra.run.smithi088.stderr:2024-06-03T09:06:58.671+0000 7fd2a6ffd640 10 monclient(hunting): _negotiate 2024-06-03T09:06:58.675 INFO:teuthology.orchestra.run.smithi088.stderr:2024-06-03T09:06:58.671+0000 7fd2a6ffd640 10 monclient(hunting): _init_auth method 2 2024-06-03T09:06:58.675 INFO:teuthology.orchestra.run.smithi088.stderr:2024-06-03T09:06:58.671+0000 7fd2a6ffd640 10 monclient(hunting): _init_auth creating new auth 2024-06-03T09:06:58.676 INFO:teuthology.orchestra.run.smithi088.stderr:2024-06-03T09:06:58.671+0000 7fd2a6ffd640 10 monclient(hunting): my global_id is 30346 2024-06-03T09:06:58.676 INFO:teuthology.orchestra.run.smithi088.stderr:2024-06-03T09:06:58.671+0000 7fd2a6ffd640 1 -- 172.21.15.88:0/1531698337 --> v1:172.21.15.192:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7fd2b013e250 con 0x7fd2b0143640 2024-06-03T09:06:58.677 INFO:teuthology.orchestra.run.smithi088.stderr:2024-06-03T09:06:58.675+0000 7fd2a6ffd640 1 -- 172.21.15.88:0/1531698337 <== mon.1 v1:172.21.15.192:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 764+0+0 (unknown 4058440865 0 0) 0x7fd2b013e250 con 0x7fd2b0143640 2024-06-03T09:06:58.677 INFO:teuthology.orchestra.run.smithi088.stderr:2024-06-03T09:06:58.676+0000 7fd2a6ffd640 1 -- 172.21.15.88:0/1531698337 --> v1:172.21.15.192:6789/0 -- auth(proto 2 165 bytes epoch 0) v1 -- 0x7fd2b01457c0 con 0x7fd2b0143640 2024-06-03T09:06:58.677 INFO:teuthology.orchestra.run.smithi088.stderr:2024-06-03T09:06:58.676+0000 7fd2a6ffd640 1 -- 172.21.15.88:0/1531698337 <== mon.1 v1:172.21.15.192:6789/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 2234374065 0 0) 0x7fd2a8001b50 con 0x7fd2b0143640 2024-06-03T09:06:58.677 INFO:teuthology.orchestra.run.smithi088.stderr:2024-06-03T09:06:58.676+0000 7fd2a6ffd640 10 monclient(hunting): handle_monmap mon_map magic: 0 v1 2024-06-03T09:06:58.677 INFO:teuthology.orchestra.run.smithi088.stderr:2024-06-03T09:06:58.676+0000 7fd2a6ffd640 10 monclient(hunting): got monmap 1 from mon.noname-b (according to old e1) 2024-06-03T09:06:58.677 INFO:teuthology.orchestra.run.smithi088.stderr:2024-06-03T09:06:58.676+0000 7fd2a6ffd640 10 monclient(hunting): dump: 2024-06-03T09:06:58.677 INFO:teuthology.orchestra.run.smithi088.stderr:epoch 1 2024-06-03T09:06:58.677 INFO:teuthology.orchestra.run.smithi088.stderr:fsid ec3c3fdd-3ac6-4d7d-9412-683130ed8b04 2024-06-03T09:06:58.677 INFO:teuthology.orchestra.run.smithi088.stderr:last_changed 2024-06-03T08:34:31.885125+0000 2024-06-03T09:06:58.678 INFO:teuthology.orchestra.run.smithi088.stderr:created 2024-06-03T08:34:31.885125+0000 2024-06-03T09:06:58.678 INFO:teuthology.orchestra.run.smithi088.stderr:min_mon_release 18 (reef) 2024-06-03T09:06:58.678 INFO:teuthology.orchestra.run.smithi088.stderr:election_strategy: 3 2024-06-03T09:06:58.678 INFO:teuthology.orchestra.run.smithi088.stderr:0: v1:172.21.15.88:6789/0 mon.a 2024-06-03T09:06:58.678 INFO:teuthology.orchestra.run.smithi088.stderr:1: v1:172.21.15.192:6789/0 mon.b 2024-06-03T09:06:58.678 INFO:teuthology.orchestra.run.smithi088.stderr:2: v1:172.21.15.88:6790/0 mon.c 2024-06-03T09:06:58.678 INFO:teuthology.orchestra.run.smithi088.stderr: 2024-06-03T09:06:58.678 INFO:teuthology.orchestra.run.smithi088.stderr:2024-06-03T09:06:58.676+0000 7fd2a6ffd640 10 monclient(hunting): _finish_auth 0 2024-06-03T09:06:58.678 INFO:teuthology.orchestra.run.smithi088.stderr:2024-06-03T09:06:58.676+0000 7fd2a6ffd640 1 -- 172.21.15.88:0/1531698337 <== mon.2 v1:172.21.15.88:6790/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 764+0+0 (unknown 227437408 0 0) 0x7fd294004eb0 con 0x7fd2b00717c0 2024-06-03T09:06:58.678 INFO:teuthology.orchestra.run.smithi088.stderr:2024-06-03T09:06:58.676+0000 7fd2a6ffd640 1 -- 172.21.15.88:0/1531698337 --> v1:172.21.15.88:6790/0 -- auth(proto 2 165 bytes epoch 0) v1 -- 0x7fd2b013e250 con 0x7fd2b00717c0 2024-06-03T09:06:58.678 INFO:teuthology.orchestra.run.smithi088.stderr:2024-06-03T09:06:58.676+0000 7fd2a6ffd640 1 -- 172.21.15.88:0/1531698337 <== mon.2 v1:172.21.15.88:6790/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 2234374065 0 0) 0x7fd2a0001c60 con 0x7fd2b00717c0 2024-06-03T09:06:58.678 INFO:teuthology.orchestra.run.smithi088.stderr:2024-06-03T09:06:58.676+0000 7fd2a6ffd640 10 monclient(hunting): handle_monmap mon_map magic: 0 v1 2024-06-03T09:06:58.678 INFO:teuthology.orchestra.run.smithi088.stderr:2024-06-03T09:06:58.676+0000 7fd2a6ffd640 10 monclient(hunting): got monmap 1 from mon.c (according to old e1) 2024-06-03T09:06:58.679 INFO:teuthology.orchestra.run.smithi088.stderr:2024-06-03T09:06:58.676+0000 7fd2a6ffd640 10 monclient(hunting): dump: 2024-06-03T09:06:58.679 INFO:teuthology.orchestra.run.smithi088.stderr:epoch 1 2024-06-03T09:06:58.679 INFO:teuthology.orchestra.run.smithi088.stderr:fsid ec3c3fdd-3ac6-4d7d-9412-683130ed8b04 2024-06-03T09:06:58.679 INFO:teuthology.orchestra.run.smithi088.stderr:last_changed 2024-06-03T08:34:31.885125+0000 2024-06-03T09:06:58.679 INFO:teuthology.orchestra.run.smithi088.stderr:created 2024-06-03T08:34:31.885125+0000 2024-06-03T09:06:58.679 INFO:teuthology.orchestra.run.smithi088.stderr:min_mon_release 18 (reef) 2024-06-03T09:06:58.679 INFO:teuthology.orchestra.run.smithi088.stderr:election_strategy: 3 2024-06-03T09:06:58.679 INFO:teuthology.orchestra.run.smithi088.stderr:0: v1:172.21.15.88:6789/0 mon.a 2024-06-03T09:06:58.679 INFO:teuthology.orchestra.run.smithi088.stderr:1: v1:172.21.15.192:6789/0 mon.b 2024-06-03T09:06:58.679 INFO:teuthology.orchestra.run.smithi088.stderr:2: v1:172.21.15.88:6790/0 mon.c 2024-06-03T09:06:58.679 INFO:teuthology.orchestra.run.smithi088.stderr: 2024-06-03T09:06:58.679 INFO:teuthology.orchestra.run.smithi088.stderr:2024-06-03T09:06:58.676+0000 7fd2a6ffd640 1 -- 172.21.15.88:0/1531698337 <== mon.0 v1:172.21.15.88:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 764+0+0 (unknown 1431173249 0 0) 0x7fd2b0145a30 con 0x7fd2b006e050 2024-06-03T09:06:58.679 INFO:teuthology.orchestra.run.smithi088.stderr:2024-06-03T09:06:58.676+0000 7fd2a6ffd640 1 -- 172.21.15.88:0/1531698337 --> v1:172.21.15.88:6789/0 -- auth(proto 2 165 bytes epoch 0) v1 -- 0x7fd294004eb0 con 0x7fd2b006e050 2024-06-03T09:06:58.679 INFO:teuthology.orchestra.run.smithi088.stderr:2024-06-03T09:06:58.676+0000 7fd2a6ffd640 1 -- 172.21.15.88:0/1531698337 <== mon.0 v1:172.21.15.88:6789/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 2234374065 0 0) 0x7fd2ac001650 con 0x7fd2b006e050 2024-06-03T09:06:58.680 INFO:teuthology.orchestra.run.smithi088.stderr:2024-06-03T09:06:58.676+0000 7fd2a6ffd640 10 monclient(hunting): handle_monmap mon_map magic: 0 v1 2024-06-03T09:06:58.680 INFO:teuthology.orchestra.run.smithi088.stderr:2024-06-03T09:06:58.676+0000 7fd2a6ffd640 10 monclient(hunting): got monmap 1 from mon.a (according to old e1) 2024-06-03T09:06:58.680 INFO:teuthology.orchestra.run.smithi088.stderr:2024-06-03T09:06:58.676+0000 7fd2a6ffd640 10 monclient(hunting): dump: 2024-06-03T09:06:58.680 INFO:teuthology.orchestra.run.smithi088.stderr:epoch 1 2024-06-03T09:06:58.680 INFO:teuthology.orchestra.run.smithi088.stderr:fsid ec3c3fdd-3ac6-4d7d-9412-683130ed8b04 2024-06-03T09:06:58.680 INFO:teuthology.orchestra.run.smithi088.stderr:last_changed 2024-06-03T08:34:31.885125+0000 2024-06-03T09:06:58.680 INFO:teuthology.orchestra.run.smithi088.stderr:created 2024-06-03T08:34:31.885125+0000 2024-06-03T09:06:58.680 INFO:teuthology.orchestra.run.smithi088.stderr:min_mon_release 18 (reef) 2024-06-03T09:06:58.680 INFO:teuthology.orchestra.run.smithi088.stderr:election_strategy: 3 2024-06-03T09:06:58.680 INFO:teuthology.orchestra.run.smithi088.stderr:0: v1:172.21.15.88:6789/0 mon.a 2024-06-03T09:06:58.680 INFO:teuthology.orchestra.run.smithi088.stderr:1: v1:172.21.15.192:6789/0 mon.b 2024-06-03T09:06:58.680 INFO:teuthology.orchestra.run.smithi088.stderr:2: v1:172.21.15.88:6790/0 mon.c 2024-06-03T09:06:58.680 INFO:teuthology.orchestra.run.smithi088.stderr: 2024-06-03T09:06:58.681 INFO:teuthology.orchestra.run.smithi088.stderr:2024-06-03T09:06:58.679+0000 7fd2a6ffd640 1 -- 172.21.15.88:0/1531698337 <== mon.1 v1:172.21.15.192:6789/0 4 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 580+0+0 (unknown 3229608592 0 0) 0x7fd2b01457c0 con 0x7fd2b0143640 2024-06-03T09:06:58.682 INFO:teuthology.orchestra.run.smithi088.stderr:2024-06-03T09:06:58.679+0000 7fd2a6ffd640 1 -- 172.21.15.88:0/1531698337 >> v1:172.21.15.88:6790/0 conn(0x7fd2b00717c0 legacy=0x7fd2b0144070 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-06-03T09:06:58.682 INFO:teuthology.orchestra.run.smithi088.stderr:2024-06-03T09:06:58.679+0000 7fd2a6ffd640 1 -- 172.21.15.88:0/1531698337 >> v1:172.21.15.88:6789/0 conn(0x7fd2b006e050 legacy=0x7fd2b013da10 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-06-03T09:06:58.682 INFO:teuthology.orchestra.run.smithi088.stderr:2024-06-03T09:06:58.679+0000 7fd2a6ffd640 10 monclient: _finish_hunting 0 2024-06-03T09:06:58.682 INFO:teuthology.orchestra.run.smithi088.stderr:2024-06-03T09:06:58.679+0000 7fd2a6ffd640 1 monclient: found mon.b 2024-06-03T09:06:58.682 INFO:teuthology.orchestra.run.smithi088.stderr:2024-06-03T09:06:58.679+0000 7fd2a6ffd640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-06-03T09:06:58.682 INFO:teuthology.orchestra.run.smithi088.stderr:2024-06-03T09:06:58.679+0000 7fd2a6ffd640 10 monclient: _send_mon_message to mon.b at v1:172.21.15.192:6789/0 2024-06-03T09:06:58.682 INFO:teuthology.orchestra.run.smithi088.stderr:2024-06-03T09:06:58.679+0000 7fd2a6ffd640 1 -- 172.21.15.88:0/1531698337 --> v1:172.21.15.192:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fd2b0144780 con 0x7fd2b0143640 2024-06-03T09:06:58.682 INFO:teuthology.orchestra.run.smithi088.stderr:2024-06-03T09:06:58.679+0000 7fd2a6ffd640 10 monclient: _finish_auth 0 2024-06-03T09:06:58.682 INFO:teuthology.orchestra.run.smithi088.stderr:2024-06-03T09:06:58.679+0000 7fd2a6ffd640 10 monclient: _check_auth_tickets 2024-06-03T09:06:58.682 INFO:teuthology.orchestra.run.smithi088.stderr:2024-06-03T09:06:58.679+0000 7fd2a6ffd640 20 monclient: _check_auth_rotating not needed by client.admin 2024-06-03T09:06:58.682 INFO:teuthology.orchestra.run.smithi088.stderr:2024-06-03T09:06:58.679+0000 7fd2b85eb640 5 monclient: authenticate success, global_id 30346 2024-06-03T09:06:58.682 INFO:teuthology.orchestra.run.smithi088.stderr:2024-06-03T09:06:58.679+0000 7fd2b85eb640 10 monclient: _renew_subs 2024-06-03T09:06:58.682 INFO:teuthology.orchestra.run.smithi088.stderr:2024-06-03T09:06:58.679+0000 7fd2b85eb640 10 monclient: _send_mon_message to mon.b at v1:172.21.15.192:6789/0 2024-06-03T09:06:58.683 INFO:teuthology.orchestra.run.smithi088.stderr:2024-06-03T09:06:58.679+0000 7fd2b85eb640 1 -- 172.21.15.88:0/1531698337 --> v1:172.21.15.192:6789/0 -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fd2b0141ff0 con 0x7fd2b0143640 2024-06-03T09:06:58.683 INFO:teuthology.orchestra.run.smithi088.stderr:2024-06-03T09:06:58.679+0000 7fd2b85eb640 10 client.30346.objecter _maybe_request_map subscribing (onetime) to next osd map 2024-06-03T09:06:58.683 INFO:teuthology.orchestra.run.smithi088.stderr:2024-06-03T09:06:58.679+0000 7fd2b85eb640 10 monclient: _renew_subs 2024-06-03T09:06:58.683 INFO:teuthology.orchestra.run.smithi088.stderr:2024-06-03T09:06:58.679+0000 7fd2b85eb640 10 monclient: _send_mon_message to mon.b at v1:172.21.15.192:6789/0 2024-06-03T09:06:58.683 INFO:teuthology.orchestra.run.smithi088.stderr:2024-06-03T09:06:58.679+0000 7fd2b85eb640 1 -- 172.21.15.88:0/1531698337 --> v1:172.21.15.192:6789/0 -- mon_subscribe({osdmap=0}) v3 -- 0x7fd2b01e2550 con 0x7fd2b0143640 2024-06-03T09:06:58.683 INFO:teuthology.orchestra.run.smithi088.stderr:2024-06-03T09:06:58.679+0000 7fd2b85eb640 1 librados: init done 2024-06-03T09:06:58.683 INFO:teuthology.orchestra.run.smithi088.stderr:2024-06-03T09:06:58.680+0000 7fd2b85eb640 10 client.30346.objecter _submit_command 1 [{"prefix": "get_command_descriptions"}] 2024-06-03T09:06:58.683 INFO:teuthology.orchestra.run.smithi088.stderr:2024-06-03T09:06:58.680+0000 7fd2b85eb640 15 client.30346.objecter _session_command_op_assign -1 1 2024-06-03T09:06:58.683 INFO:teuthology.orchestra.run.smithi088.stderr:2024-06-03T09:06:58.680+0000 7fd2b85eb640 20 client.30346.objecter _get_session osd=-1 returning homeless 2024-06-03T09:06:58.683 INFO:teuthology.orchestra.run.smithi088.stderr:2024-06-03T09:06:58.680+0000 7fd2b85eb640 10 client.30346.objecter _maybe_request_map subscribing (onetime) to next osd map 2024-06-03T09:06:58.683 INFO:teuthology.orchestra.run.smithi088.stderr:2024-06-03T09:06:58.680+0000 7fd2b85eb640 10 monclient: _send_mon_message to mon.b at v1:172.21.15.192:6789/0 2024-06-03T09:06:58.683 INFO:teuthology.orchestra.run.smithi088.stderr:2024-06-03T09:06:58.680+0000 7fd2b85eb640 1 -- 172.21.15.88:0/1531698337 --> v1:172.21.15.192:6789/0 -- mon_get_version(what=osdmap handle=1) v1 -- 0x7fd2b006ee80 con 0x7fd2b0143640 2024-06-03T09:06:58.695 INFO:teuthology.orchestra.run.smithi088.stderr:2024-06-03T09:06:58.695+0000 7fd2a6ffd640 1 -- 172.21.15.88:0/1531698337 <== mon.1 v1:172.21.15.192:6789/0 5 ==== config(0 keys) v1 ==== 4+0+0 (unknown 0 0 0) 0x7fd2b006ee80 con 0x7fd2b0143640 2024-06-03T09:06:58.695 INFO:teuthology.orchestra.run.smithi088.stderr:2024-06-03T09:06:58.695+0000 7fd2a6ffd640 10 monclient: handle_config config(0 keys) v1 2024-06-03T09:06:58.695 INFO:teuthology.orchestra.run.smithi088.stderr:2024-06-03T09:06:58.695+0000 7fd2a6ffd640 1 -- 172.21.15.88:0/1531698337 <== mon.1 v1:172.21.15.192:6789/0 6 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 2234374065 0 0) 0x7fd2a8004b60 con 0x7fd2b0143640 2024-06-03T09:06:58.695 INFO:teuthology.orchestra.run.smithi088.stderr:2024-06-03T09:06:58.695+0000 7fd2a6ffd640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-06-03T09:06:58.695 INFO:teuthology.orchestra.run.smithi088.stderr:2024-06-03T09:06:58.695+0000 7fd2a6ffd640 10 monclient: got monmap 1 from mon.b (according to old e1) 2024-06-03T09:06:58.695 INFO:teuthology.orchestra.run.smithi088.stderr:2024-06-03T09:06:58.695+0000 7fd2a6ffd640 10 monclient: dump: 2024-06-03T09:06:58.695 INFO:teuthology.orchestra.run.smithi088.stderr:epoch 1 2024-06-03T09:06:58.695 INFO:teuthology.orchestra.run.smithi088.stderr:fsid ec3c3fdd-3ac6-4d7d-9412-683130ed8b04 2024-06-03T09:06:58.695 INFO:teuthology.orchestra.run.smithi088.stderr:last_changed 2024-06-03T08:34:31.885125+0000 2024-06-03T09:06:58.696 INFO:teuthology.orchestra.run.smithi088.stderr:created 2024-06-03T08:34:31.885125+0000 2024-06-03T09:06:58.696 INFO:teuthology.orchestra.run.smithi088.stderr:min_mon_release 18 (reef) 2024-06-03T09:06:58.696 INFO:teuthology.orchestra.run.smithi088.stderr:election_strategy: 3 2024-06-03T09:06:58.696 INFO:teuthology.orchestra.run.smithi088.stderr:0: v1:172.21.15.88:6789/0 mon.a 2024-06-03T09:06:58.696 INFO:teuthology.orchestra.run.smithi088.stderr:1: v1:172.21.15.192:6789/0 mon.b 2024-06-03T09:06:58.696 INFO:teuthology.orchestra.run.smithi088.stderr:2: v1:172.21.15.88:6790/0 mon.c 2024-06-03T09:06:58.696 INFO:teuthology.orchestra.run.smithi088.stderr: 2024-06-03T09:06:58.696 INFO:teuthology.orchestra.run.smithi088.stderr:2024-06-03T09:06:58.695+0000 7fd2b4b84640 4 set_mon_vals no callback set 2024-06-03T09:06:58.697 INFO:teuthology.orchestra.run.smithi088.stderr:2024-06-03T09:06:58.696+0000 7fd2a6ffd640 1 -- 172.21.15.88:0/1531698337 <== mon.1 v1:172.21.15.192:6789/0 7 ==== mgrmap(e 5) v1 ==== 89110+0+0 (unknown 873533973 0 0) 0x7fd2a801a8e0 con 0x7fd2b0143640 2024-06-03T09:06:58.697 INFO:teuthology.orchestra.run.smithi088.stderr:2024-06-03T09:06:58.696+0000 7fd2b6387640 10 monclient: get_auth_request con 0x7fd29406d450 auth_method 0 2024-06-03T09:06:58.697 INFO:teuthology.orchestra.run.smithi088.stderr:2024-06-03T09:06:58.696+0000 7fd2a6ffd640 10 client.30346.objecter ms_handle_connect 0x7fd29406d450 2024-06-03T09:06:58.698 INFO:teuthology.orchestra.run.smithi088.stderr:2024-06-03T09:06:58.697+0000 7fd2a6ffd640 1 -- 172.21.15.88:0/1531698337 <== mon.1 v1:172.21.15.192:6789/0 8 ==== osd_map(1014..1014 src has 622..1014) v4 ==== 5229+0+0 (unknown 3230192161 0 0) 0x7fd2a8054170 con 0x7fd2b0143640 2024-06-03T09:06:58.698 INFO:teuthology.orchestra.run.smithi088.stderr:2024-06-03T09:06:58.697+0000 7fd2a6ffd640 10 client.30346.objecter ms_dispatch 0x7fd2b0000b90 osd_map(1014..1014 src has 622..1014) v4 2024-06-03T09:06:58.698 INFO:teuthology.orchestra.run.smithi088.stderr:2024-06-03T09:06:58.697+0000 7fd2a6ffd640 3 client.30346.objecter handle_osd_map got epochs [1014,1014] > 0 2024-06-03T09:06:58.698 INFO:teuthology.orchestra.run.smithi088.stderr:2024-06-03T09:06:58.697+0000 7fd2a6ffd640 3 client.30346.objecter handle_osd_map decoding full epoch 1014 2024-06-03T09:06:58.698 INFO:teuthology.orchestra.run.smithi088.stderr:2024-06-03T09:06:58.697+0000 7fd2a6ffd640 10 client.30346.objecter checking command 1 2024-06-03T09:06:58.698 INFO:teuthology.orchestra.run.smithi088.stderr:2024-06-03T09:06:58.697+0000 7fd2a6ffd640 20 client.30346.objecter _get_session s=0x7fd294075390 osd=3 3 2024-06-03T09:06:58.698 INFO:teuthology.orchestra.run.smithi088.stderr:2024-06-03T09:06:58.697+0000 7fd2a6ffd640 20 client.30346.objecter put_session s=0x7fd294075390 osd=3 3 2024-06-03T09:06:58.698 INFO:teuthology.orchestra.run.smithi088.stderr:2024-06-03T09:06:58.697+0000 7fd2a6ffd640 15 client.30346.objecter _session_command_op_remove -1 1 2024-06-03T09:06:58.698 INFO:teuthology.orchestra.run.smithi088.stderr:2024-06-03T09:06:58.697+0000 7fd2a6ffd640 20 client.30346.objecter _get_session s=0x7fd294075390 osd=3 3 2024-06-03T09:06:58.698 INFO:teuthology.orchestra.run.smithi088.stderr:2024-06-03T09:06:58.697+0000 7fd2a6ffd640 20 client.30346.objecter get_session s=0x7fd294075390 osd=3 3 2024-06-03T09:06:58.699 INFO:teuthology.orchestra.run.smithi088.stderr:2024-06-03T09:06:58.697+0000 7fd2a6ffd640 15 client.30346.objecter _session_command_op_assign 3 1 2024-06-03T09:06:58.699 INFO:teuthology.orchestra.run.smithi088.stderr:2024-06-03T09:06:58.697+0000 7fd2a6ffd640 20 client.30346.objecter put_session s=0x7fd294075390 osd=3 4 2024-06-03T09:06:58.699 INFO:teuthology.orchestra.run.smithi088.stderr:2024-06-03T09:06:58.697+0000 7fd2a6ffd640 10 client.30346.objecter _send_command 1 2024-06-03T09:06:58.699 INFO:teuthology.orchestra.run.smithi088.stderr:2024-06-03T09:06:58.697+0000 7fd2a6ffd640 1 -- 172.21.15.88:0/1531698337 --> v1:172.21.15.88:6812/35068 -- command(tid 1: {"prefix": "get_command_descriptions"}) v1 -- 0x7fd2940796b0 con 0x7fd294075a40 2024-06-03T09:06:58.699 INFO:teuthology.orchestra.run.smithi088.stderr:2024-06-03T09:06:58.697+0000 7fd2a6ffd640 20 client.30346.objecter dump_active .. 0 homeless 2024-06-03T09:06:58.699 INFO:teuthology.orchestra.run.smithi088.stderr:2024-06-03T09:06:58.698+0000 7fd2b5b86640 10 monclient: get_auth_request con 0x7fd294075a40 auth_method 0 2024-06-03T09:06:58.699 INFO:teuthology.orchestra.run.smithi088.stderr:2024-06-03T09:06:58.699+0000 7fd2a6ffd640 1 -- 172.21.15.88:0/1531698337 <== mon.1 v1:172.21.15.192:6789/0 9 ==== mon_get_version_reply(handle=1 version=1014) v2 ==== 24+0+0 (unknown 3501658476 0 0) 0x7fd2a804fd20 con 0x7fd2b0143640 2024-06-03T09:06:58.699 INFO:teuthology.orchestra.run.smithi088.stderr:2024-06-03T09:06:58.699+0000 7fd2a6ffd640 10 monclient: handle_get_version_reply finishing 1 version 1014 2024-06-03T09:06:58.700 INFO:teuthology.orchestra.run.smithi088.stderr:2024-06-03T09:06:58.699+0000 7fd2a6ffd640 10 client.30346.objecter ms_handle_connect 0x7fd294075a40 2024-06-03T09:06:58.722 INFO:teuthology.orchestra.run.smithi088.stderr:2024-06-03T09:06:58.722+0000 7fd2a6ffd640 1 -- 172.21.15.88:0/1531698337 <== osd.3 v1:172.21.15.88:6812/35068 1 ==== command_reply(tid 1: 0 ) v1 ==== 8+0+26118 (unknown 0 0 1686565167) 0x7fd2940796b0 con 0x7fd294075a40 2024-06-03T09:06:58.723 INFO:teuthology.orchestra.run.smithi088.stderr:2024-06-03T09:06:58.722+0000 7fd2a6ffd640 10 client.30346.objecter ms_dispatch 0x7fd2b0000b90 command_reply(tid 1: 0 ) v1 2024-06-03T09:06:58.723 INFO:teuthology.orchestra.run.smithi088.stderr:2024-06-03T09:06:58.722+0000 7fd2a6ffd640 10 client.30346.objecter _finish_command 1 = system:0 2024-06-03T09:06:58.723 INFO:teuthology.orchestra.run.smithi088.stderr:2024-06-03T09:06:58.722+0000 7fd2a6ffd640 20 client.30346.objecter put_session s=0x7fd294075390 osd=3 4 2024-06-03T09:06:58.723 INFO:teuthology.orchestra.run.smithi088.stderr:2024-06-03T09:06:58.722+0000 7fd2a6ffd640 15 client.30346.objecter _session_command_op_remove 3 1 2024-06-03T09:06:58.736 INFO:teuthology.orchestra.run.smithi088.stderr:2024-06-03T09:06:58.736+0000 7fd2b85eb640 10 client.30346.objecter _submit_command 2 [{"prefix": "config set", "var": "osd_debug_deep_scrub_sleep", "val": ["0"]}] 2024-06-03T09:06:58.736 INFO:teuthology.orchestra.run.smithi088.stderr:2024-06-03T09:06:58.736+0000 7fd2b85eb640 15 client.30346.objecter _session_command_op_assign -1 2 2024-06-03T09:06:58.736 INFO:teuthology.orchestra.run.smithi088.stderr:2024-06-03T09:06:58.736+0000 7fd2b85eb640 20 client.30346.objecter _get_session s=0x7fd294075390 osd=3 3 2024-06-03T09:06:58.736 INFO:teuthology.orchestra.run.smithi088.stderr:2024-06-03T09:06:58.736+0000 7fd2b85eb640 20 client.30346.objecter put_session s=0x7fd294075390 osd=3 3 2024-06-03T09:06:58.736 INFO:teuthology.orchestra.run.smithi088.stderr:2024-06-03T09:06:58.736+0000 7fd2b85eb640 20 client.30346.objecter _get_session s=0x7fd294075390 osd=3 3 2024-06-03T09:06:58.736 INFO:teuthology.orchestra.run.smithi088.stderr:2024-06-03T09:06:58.736+0000 7fd2b85eb640 15 client.30346.objecter _session_command_op_remove -1 2 2024-06-03T09:06:58.736 INFO:teuthology.orchestra.run.smithi088.stderr:2024-06-03T09:06:58.736+0000 7fd2b85eb640 20 client.30346.objecter get_session s=0x7fd294075390 osd=3 3 2024-06-03T09:06:58.736 INFO:teuthology.orchestra.run.smithi088.stderr:2024-06-03T09:06:58.736+0000 7fd2b85eb640 15 client.30346.objecter _session_command_op_assign 3 2 2024-06-03T09:06:58.736 INFO:teuthology.orchestra.run.smithi088.stderr:2024-06-03T09:06:58.736+0000 7fd2b85eb640 20 client.30346.objecter put_session s=0x7fd294075390 osd=3 4 2024-06-03T09:06:58.737 INFO:teuthology.orchestra.run.smithi088.stderr:2024-06-03T09:06:58.736+0000 7fd2b85eb640 10 client.30346.objecter _send_command 2 2024-06-03T09:06:58.737 INFO:teuthology.orchestra.run.smithi088.stderr:2024-06-03T09:06:58.736+0000 7fd2b85eb640 1 -- 172.21.15.88:0/1531698337 --> v1:172.21.15.88:6812/35068 -- command(tid 2: {"prefix": "config set", "var": "osd_debug_deep_scrub_sleep", "val": ["0"]}) v1 -- 0x7fd2b004ea60 con 0x7fd294075a40 2024-06-03T09:06:58.749 INFO:teuthology.orchestra.run.smithi088.stderr:2024-06-03T09:06:58.749+0000 7fd2a6ffd640 1 -- 172.21.15.88:0/1531698337 <== osd.3 v1:172.21.15.88:6812/35068 2 ==== command_reply(tid 2: 0 ) v1 ==== 8+0+97 (unknown 0 0 119112422) 0x7fd2b004ea60 con 0x7fd294075a40 2024-06-03T09:06:58.749 INFO:teuthology.orchestra.run.smithi088.stderr:2024-06-03T09:06:58.749+0000 7fd2a6ffd640 10 client.30346.objecter ms_dispatch 0x7fd2b0000b90 command_reply(tid 2: 0 ) v1 2024-06-03T09:06:58.749 INFO:teuthology.orchestra.run.smithi088.stderr:2024-06-03T09:06:58.749+0000 7fd2a6ffd640 10 client.30346.objecter _finish_command 2 = system:0 2024-06-03T09:06:58.749 INFO:teuthology.orchestra.run.smithi088.stderr:2024-06-03T09:06:58.749+0000 7fd2a6ffd640 20 client.30346.objecter put_session s=0x7fd294075390 osd=3 4 2024-06-03T09:06:58.749 INFO:teuthology.orchestra.run.smithi088.stderr:2024-06-03T09:06:58.749+0000 7fd2a6ffd640 15 client.30346.objecter _session_command_op_remove 3 2 2024-06-03T09:06:58.749 INFO:teuthology.orchestra.run.smithi088.stdout:{ 2024-06-03T09:06:58.750 INFO:teuthology.orchestra.run.smithi088.stdout: "success": "osd_debug_deep_scrub_sleep = '' (not observed, change may require restart) " 2024-06-03T09:06:58.750 INFO:teuthology.orchestra.run.smithi088.stdout:} 2024-06-03T09:06:58.751 INFO:teuthology.orchestra.run.smithi088.stderr:2024-06-03T09:06:58.749+0000 7fd2b85eb640 10 librados: watch_flush enter 2024-06-03T09:06:58.751 INFO:teuthology.orchestra.run.smithi088.stderr:2024-06-03T09:06:58.749+0000 7fd2b85eb640 10 librados: watch_flush exit 2024-06-03T09:06:58.751 INFO:teuthology.orchestra.run.smithi088.stderr:2024-06-03T09:06:58.749+0000 7fd2b85eb640 10 client.30346.objecter close_session for osd.3 2024-06-03T09:06:58.751 INFO:teuthology.orchestra.run.smithi088.stderr:2024-06-03T09:06:58.749+0000 7fd2b85eb640 1 -- 172.21.15.88:0/1531698337 >> v1:172.21.15.88:6812/35068 conn(0x7fd294075a40 legacy=0x7fd294077ee0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-06-03T09:06:58.751 INFO:teuthology.orchestra.run.smithi088.stderr:2024-06-03T09:06:58.749+0000 7fd2b85eb640 20 client.30346.objecter put_session s=0x7fd294075390 osd=3 1 2024-06-03T09:06:58.751 INFO:teuthology.orchestra.run.smithi088.stderr:2024-06-03T09:06:58.749+0000 7fd2b85eb640 20 client.30346.objecter shutdown clearing up homeless session... 2024-06-03T09:06:58.752 INFO:teuthology.orchestra.run.smithi088.stderr:2024-06-03T09:06:58.749+0000 7fd2b85eb640 10 client.30346.objecter successfully canceled tick 2024-06-03T09:06:58.752 INFO:teuthology.orchestra.run.smithi088.stderr:2024-06-03T09:06:58.749+0000 7fd2b85eb640 1 -- 172.21.15.88:0/1531698337 >> v1:172.21.15.192:6800/34229 conn(0x7fd29406d450 legacy=0x7fd29406f910 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-06-03T09:06:58.752 INFO:teuthology.orchestra.run.smithi088.stderr:2024-06-03T09:06:58.749+0000 7fd2b85eb640 10 monclient: shutdown 2024-06-03T09:06:58.752 INFO:teuthology.orchestra.run.smithi088.stderr:2024-06-03T09:06:58.749+0000 7fd2b85eb640 20 monclient: shutdown discarding 0 pending message(s) 2024-06-03T09:06:58.752 INFO:teuthology.orchestra.run.smithi088.stderr:2024-06-03T09:06:58.749+0000 7fd2b85eb640 1 -- 172.21.15.88:0/1531698337 >> v1:172.21.15.192:6789/0 conn(0x7fd2b0143640 legacy=0x7fd2b01e0e30 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-06-03T09:06:58.752 INFO:teuthology.orchestra.run.smithi088.stderr:2024-06-03T09:06:58.749+0000 7fd2b6b88640 1 -- 172.21.15.88:0/1531698337 reap_dead start 2024-06-03T09:06:58.752 INFO:teuthology.orchestra.run.smithi088.stderr:2024-06-03T09:06:58.749+0000 7fd2b85eb640 1 -- 172.21.15.88:0/1531698337 shutdown_connections 2024-06-03T09:06:58.752 INFO:teuthology.orchestra.run.smithi088.stderr:2024-06-03T09:06:58.749+0000 7fd2b85eb640 1 -- 172.21.15.88:0/1531698337 >> 172.21.15.88:0/1531698337 conn(0x7fd2b00b2f20 msgr2=0x7fd2b00b0170 unknown :-1 s=STATE_NONE l=0).mark_down 2024-06-03T09:06:58.752 INFO:teuthology.orchestra.run.smithi088.stderr:2024-06-03T09:06:58.749+0000 7fd2b85eb640 1 -- 172.21.15.88:0/1531698337 shutdown_connections 2024-06-03T09:06:58.752 INFO:teuthology.orchestra.run.smithi088.stderr:2024-06-03T09:06:58.749+0000 7fd2b85eb640 1 -- 172.21.15.88:0/1531698337 wait complete. 2024-06-03T09:06:58.752 INFO:teuthology.orchestra.run.smithi088.stderr:2024-06-03T09:06:58.749+0000 7fd2b85eb640 1 librados: shutdown 2024-06-03T09:06:58.760 DEBUG:teuthology.orchestra.run.smithi088:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd deep-scrub 3 2024-06-03T09:06:58.858 INFO:teuthology.orchestra.run.smithi088.stderr:2024-06-03T09:06:58.858+0000 7f2422741640 10 monclient: get_monmap_and_config 2024-06-03T09:06:58.858 INFO:teuthology.orchestra.run.smithi088.stderr:2024-06-03T09:06:58.858+0000 7f2422741640 10 monclient: build_initial_monmap 2024-06-03T09:06:58.858 INFO:teuthology.orchestra.run.smithi088.stderr:2024-06-03T09:06:58.858+0000 7f2422741640 10 monclient: monmap: 2024-06-03T09:06:58.858 INFO:teuthology.orchestra.run.smithi088.stderr:epoch 0 2024-06-03T09:06:58.858 INFO:teuthology.orchestra.run.smithi088.stderr:fsid ec3c3fdd-3ac6-4d7d-9412-683130ed8b04 2024-06-03T09:06:58.858 INFO:teuthology.orchestra.run.smithi088.stderr:last_changed 2024-06-03T09:06:58.859322+0000 2024-06-03T09:06:58.859 INFO:teuthology.orchestra.run.smithi088.stderr:created 2024-06-03T09:06:58.859322+0000 2024-06-03T09:06:58.859 INFO:teuthology.orchestra.run.smithi088.stderr:min_mon_release 0 (unknown) 2024-06-03T09:06:58.859 INFO:teuthology.orchestra.run.smithi088.stderr:election_strategy: 3 2024-06-03T09:06:58.859 INFO:teuthology.orchestra.run.smithi088.stderr:0: v1:172.21.15.88:6789/0 mon.noname-a 2024-06-03T09:06:58.859 INFO:teuthology.orchestra.run.smithi088.stderr:1: v1:172.21.15.192:6789/0 mon.noname-c 2024-06-03T09:06:58.859 INFO:teuthology.orchestra.run.smithi088.stderr:2: v1:172.21.15.88:6790/0 mon.noname-b 2024-06-03T09:06:58.859 INFO:teuthology.orchestra.run.smithi088.stderr: 2024-06-03T09:06:58.860 INFO:teuthology.orchestra.run.smithi088.stderr:2024-06-03T09:06:58.859+0000 7f2422741640 1 Processor -- start 2024-06-03T09:06:58.860 INFO:teuthology.orchestra.run.smithi088.stderr:2024-06-03T09:06:58.859+0000 7f2422741640 1 -- start start 2024-06-03T09:06:58.861 INFO:teuthology.orchestra.run.smithi088.stderr:2024-06-03T09:06:58.859+0000 7f2422741640 10 monclient: init 2024-06-03T09:06:58.861 INFO:teuthology.orchestra.run.smithi088.stderr:2024-06-03T09:06:58.859+0000 7f2422741640 10 monclient: _reopen_session rank -1 2024-06-03T09:06:58.861 INFO:teuthology.orchestra.run.smithi088.stderr:2024-06-03T09:06:58.859+0000 7f2422741640 10 monclient: _add_conns ranks=[2,1,0] 2024-06-03T09:06:58.861 INFO:teuthology.orchestra.run.smithi088.stderr:2024-06-03T09:06:58.859+0000 7f2422741640 10 monclient(hunting): picked mon.noname-b con 0x7f241c0a8a60 addr v1:172.21.15.88:6790/0 2024-06-03T09:06:58.861 INFO:teuthology.orchestra.run.smithi088.stderr:2024-06-03T09:06:58.859+0000 7f2422741640 10 monclient(hunting): picked mon.noname-c con 0x7f241c0b0d00 addr v1:172.21.15.192:6789/0 2024-06-03T09:06:58.861 INFO:teuthology.orchestra.run.smithi088.stderr:2024-06-03T09:06:58.859+0000 7f2422741640 10 monclient(hunting): picked mon.noname-a con 0x7f241c1451d0 addr v1:172.21.15.88:6789/0 2024-06-03T09:06:58.861 INFO:teuthology.orchestra.run.smithi088.stderr:2024-06-03T09:06:58.859+0000 7f2422741640 1 -- --> v1:172.21.15.88:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f241c0a9050 con 0x7f241c1451d0 2024-06-03T09:06:58.861 INFO:teuthology.orchestra.run.smithi088.stderr:2024-06-03T09:06:58.859+0000 7f2422741640 1 -- --> v1:172.21.15.192:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f241c0a9d90 con 0x7f241c0b0d00 2024-06-03T09:06:58.861 INFO:teuthology.orchestra.run.smithi088.stderr:2024-06-03T09:06:58.859+0000 7f2422741640 1 -- --> v1:172.21.15.88:6790/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f241c0a8e40 con 0x7f241c0a8a60 2024-06-03T09:06:58.861 INFO:teuthology.orchestra.run.smithi088.stderr:2024-06-03T09:06:58.859+0000 7f2422741640 10 monclient(hunting): _renew_subs 2024-06-03T09:06:58.862 INFO:teuthology.orchestra.run.smithi088.stderr:2024-06-03T09:06:58.859+0000 7f2422741640 10 monclient(hunting): authenticate will time out at 3074.669783s 2024-06-03T09:06:58.862 INFO:teuthology.orchestra.run.smithi088.stderr:2024-06-03T09:06:58.860+0000 7f241bfff640 1 --1- >> v1:172.21.15.88:6790/0 conn(0x7f241c0a8a60 0x7f241c144ac0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.88:6790/0 says I am v1:172.21.15.88:51058/0 (socket says 172.21.15.88:51058) 2024-06-03T09:06:58.862 INFO:teuthology.orchestra.run.smithi088.stderr:2024-06-03T09:06:58.860+0000 7f241bfff640 1 -- 172.21.15.88:0/4011089670 learned_addr learned my addr 172.21.15.88:0/4011089670 (peer_addr_for_me v1:172.21.15.88:0/0) 2024-06-03T09:06:58.863 INFO:teuthology.orchestra.run.smithi088.stderr:2024-06-03T09:06:58.862+0000 7f241affd640 1 -- 172.21.15.88:0/4011089670 <== mon.1 v1:172.21.15.192:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 2851596261 0 0) 0x7f241c0a9d90 con 0x7f241c0b0d00 2024-06-03T09:06:58.863 INFO:teuthology.orchestra.run.smithi088.stderr:2024-06-03T09:06:58.862+0000 7f241affd640 10 monclient(hunting): _negotiate 2024-06-03T09:06:58.863 INFO:teuthology.orchestra.run.smithi088.stderr:2024-06-03T09:06:58.862+0000 7f241affd640 10 monclient(hunting): _init_auth method 2 2024-06-03T09:06:58.863 INFO:teuthology.orchestra.run.smithi088.stderr:2024-06-03T09:06:58.862+0000 7f241affd640 10 monclient(hunting): _init_auth creating new auth 2024-06-03T09:06:58.863 INFO:teuthology.orchestra.run.smithi088.stderr:2024-06-03T09:06:58.862+0000 7f241affd640 10 monclient(hunting): my global_id is 30349 2024-06-03T09:06:58.863 INFO:teuthology.orchestra.run.smithi088.stderr:2024-06-03T09:06:58.862+0000 7f241affd640 1 -- 172.21.15.88:0/4011089670 --> v1:172.21.15.192:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f2404003520 con 0x7f241c0b0d00 2024-06-03T09:06:58.863 INFO:teuthology.orchestra.run.smithi088.stderr:2024-06-03T09:06:58.863+0000 7f241affd640 1 -- 172.21.15.88:0/4011089670 <== mon.2 v1:172.21.15.88:6790/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 2470013327 0 0) 0x7f241c0a8e40 con 0x7f241c0a8a60 2024-06-03T09:06:58.863 INFO:teuthology.orchestra.run.smithi088.stderr:2024-06-03T09:06:58.863+0000 7f241affd640 10 monclient(hunting): _negotiate 2024-06-03T09:06:58.863 INFO:teuthology.orchestra.run.smithi088.stderr:2024-06-03T09:06:58.863+0000 7f241affd640 10 monclient(hunting): _init_auth method 2 2024-06-03T09:06:58.863 INFO:teuthology.orchestra.run.smithi088.stderr:2024-06-03T09:06:58.863+0000 7f241affd640 10 monclient(hunting): _init_auth creating new auth 2024-06-03T09:06:58.863 INFO:teuthology.orchestra.run.smithi088.stderr:2024-06-03T09:06:58.863+0000 7f241affd640 10 monclient(hunting): my global_id is 30284 2024-06-03T09:06:58.863 INFO:teuthology.orchestra.run.smithi088.stderr:2024-06-03T09:06:58.863+0000 7f241affd640 1 -- 172.21.15.88:0/4011089670 --> v1:172.21.15.88:6790/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f241c0a9d90 con 0x7f241c0a8a60 2024-06-03T09:06:58.864 INFO:teuthology.orchestra.run.smithi088.stderr:2024-06-03T09:06:58.863+0000 7f241affd640 1 -- 172.21.15.88:0/4011089670 <== mon.0 v1:172.21.15.88:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 2355753605 0 0) 0x7f241c0a9050 con 0x7f241c1451d0 2024-06-03T09:06:58.864 INFO:teuthology.orchestra.run.smithi088.stderr:2024-06-03T09:06:58.863+0000 7f241affd640 10 monclient(hunting): _negotiate 2024-06-03T09:06:58.864 INFO:teuthology.orchestra.run.smithi088.stderr:2024-06-03T09:06:58.863+0000 7f241affd640 10 monclient(hunting): _init_auth method 2 2024-06-03T09:06:58.864 INFO:teuthology.orchestra.run.smithi088.stderr:2024-06-03T09:06:58.863+0000 7f241affd640 10 monclient(hunting): _init_auth creating new auth 2024-06-03T09:06:58.864 INFO:teuthology.orchestra.run.smithi088.stderr:2024-06-03T09:06:58.863+0000 7f241affd640 10 monclient(hunting): my global_id is 30777 2024-06-03T09:06:58.864 INFO:teuthology.orchestra.run.smithi088.stderr:2024-06-03T09:06:58.863+0000 7f241affd640 1 -- 172.21.15.88:0/4011089670 --> v1:172.21.15.88:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f241c0a8e40 con 0x7f241c1451d0 2024-06-03T09:06:58.866 INFO:teuthology.orchestra.run.smithi088.stderr:2024-06-03T09:06:58.865+0000 7f241affd640 1 -- 172.21.15.88:0/4011089670 <== mon.1 v1:172.21.15.192:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 214+0+0 (unknown 1938463829 0 0) 0x7f2404003520 con 0x7f241c0b0d00 2024-06-03T09:06:58.866 INFO:teuthology.orchestra.run.smithi088.stderr:2024-06-03T09:06:58.866+0000 7f241affd640 1 -- 172.21.15.88:0/4011089670 >> v1:172.21.15.88:6790/0 conn(0x7f241c0a8a60 legacy=0x7f241c144ac0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-06-03T09:06:58.866 INFO:teuthology.orchestra.run.smithi088.stderr:2024-06-03T09:06:58.866+0000 7f241affd640 1 -- 172.21.15.88:0/4011089670 >> v1:172.21.15.88:6789/0 conn(0x7f241c1451d0 legacy=0x7f241c13ea70 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-06-03T09:06:58.866 INFO:teuthology.orchestra.run.smithi088.stderr:2024-06-03T09:06:58.866+0000 7f241affd640 10 monclient: _finish_hunting 0 2024-06-03T09:06:58.866 INFO:teuthology.orchestra.run.smithi088.stderr:2024-06-03T09:06:58.866+0000 7f241affd640 1 monclient: found mon.noname-c 2024-06-03T09:06:58.866 INFO:teuthology.orchestra.run.smithi088.stderr:2024-06-03T09:06:58.866+0000 7f241affd640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-06-03T09:06:58.866 INFO:teuthology.orchestra.run.smithi088.stderr:2024-06-03T09:06:58.866+0000 7f241affd640 10 monclient: _send_mon_message to mon.noname-c at v1:172.21.15.192:6789/0 2024-06-03T09:06:58.866 INFO:teuthology.orchestra.run.smithi088.stderr:2024-06-03T09:06:58.866+0000 7f241affd640 1 -- 172.21.15.88:0/4011089670 --> v1:172.21.15.192:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f241c13f180 con 0x7f241c0b0d00 2024-06-03T09:06:58.866 INFO:teuthology.orchestra.run.smithi088.stderr:2024-06-03T09:06:58.866+0000 7f241affd640 10 monclient: _finish_auth 0 2024-06-03T09:06:58.867 INFO:teuthology.orchestra.run.smithi088.stderr:2024-06-03T09:06:58.866+0000 7f241affd640 10 monclient: _check_auth_tickets 2024-06-03T09:06:58.867 INFO:teuthology.orchestra.run.smithi088.stderr:2024-06-03T09:06:58.866+0000 7f241affd640 20 monclient: _check_auth_rotating not needed by client.admin 2024-06-03T09:06:58.867 INFO:teuthology.orchestra.run.smithi088.stderr:2024-06-03T09:06:58.866+0000 7f241affd640 1 -- 172.21.15.88:0/4011089670 <== mon.1 v1:172.21.15.192:6789/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 2234374065 0 0) 0x7f2410001610 con 0x7f241c0b0d00 2024-06-03T09:06:58.867 INFO:teuthology.orchestra.run.smithi088.stderr:2024-06-03T09:06:58.866+0000 7f241affd640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-06-03T09:06:58.867 INFO:teuthology.orchestra.run.smithi088.stderr:2024-06-03T09:06:58.866+0000 7f241affd640 10 monclient: got monmap 1 from mon.noname-c (according to old e1) 2024-06-03T09:06:58.867 INFO:teuthology.orchestra.run.smithi088.stderr:2024-06-03T09:06:58.866+0000 7f241affd640 10 monclient: dump: 2024-06-03T09:06:58.867 INFO:teuthology.orchestra.run.smithi088.stderr:epoch 1 2024-06-03T09:06:58.867 INFO:teuthology.orchestra.run.smithi088.stderr:fsid ec3c3fdd-3ac6-4d7d-9412-683130ed8b04 2024-06-03T09:06:58.867 INFO:teuthology.orchestra.run.smithi088.stderr:last_changed 2024-06-03T08:34:31.885125+0000 2024-06-03T09:06:58.867 INFO:teuthology.orchestra.run.smithi088.stderr:created 2024-06-03T08:34:31.885125+0000 2024-06-03T09:06:58.867 INFO:teuthology.orchestra.run.smithi088.stderr:min_mon_release 18 (reef) 2024-06-03T09:06:58.867 INFO:teuthology.orchestra.run.smithi088.stderr:election_strategy: 3 2024-06-03T09:06:58.867 INFO:teuthology.orchestra.run.smithi088.stderr:0: v1:172.21.15.88:6789/0 mon.a 2024-06-03T09:06:58.867 INFO:teuthology.orchestra.run.smithi088.stderr:1: v1:172.21.15.192:6789/0 mon.b 2024-06-03T09:06:58.867 INFO:teuthology.orchestra.run.smithi088.stderr:2: v1:172.21.15.88:6790/0 mon.c 2024-06-03T09:06:58.868 INFO:teuthology.orchestra.run.smithi088.stderr: 2024-06-03T09:06:58.868 INFO:teuthology.orchestra.run.smithi088.stderr:2024-06-03T09:06:58.866+0000 7f2422741640 5 monclient: authenticate success, global_id 30349 2024-06-03T09:06:58.868 INFO:teuthology.orchestra.run.smithi088.stderr:2024-06-03T09:06:58.866+0000 7f2422741640 20 monclient: get_monmap_and_config waiting for monmap|config 2024-06-03T09:06:58.869 INFO:teuthology.orchestra.run.smithi088.stderr:2024-06-03T09:06:58.867+0000 7f241affd640 1 -- 172.21.15.88:0/4011089670 <== mon.1 v1:172.21.15.192:6789/0 4 ==== config(0 keys) v1 ==== 4+0+0 (unknown 0 0 0) 0x7f2410001830 con 0x7f241c0b0d00 2024-06-03T09:06:58.869 INFO:teuthology.orchestra.run.smithi088.stderr:2024-06-03T09:06:58.867+0000 7f241affd640 10 monclient: handle_config config(0 keys) v1 2024-06-03T09:06:58.869 INFO:teuthology.orchestra.run.smithi088.stderr:2024-06-03T09:06:58.867+0000 7f241affd640 1 -- 172.21.15.88:0/4011089670 <== mon.1 v1:172.21.15.192:6789/0 5 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 2234374065 0 0) 0x7f2410001b40 con 0x7f241c0b0d00 2024-06-03T09:06:58.869 INFO:teuthology.orchestra.run.smithi088.stderr:2024-06-03T09:06:58.867+0000 7f241affd640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-06-03T09:06:58.869 INFO:teuthology.orchestra.run.smithi088.stderr:2024-06-03T09:06:58.867+0000 7f241affd640 10 monclient: got monmap 1 from mon.b (according to old e1) 2024-06-03T09:06:58.870 INFO:teuthology.orchestra.run.smithi088.stderr:2024-06-03T09:06:58.867+0000 7f241affd640 10 monclient: dump: 2024-06-03T09:06:58.870 INFO:teuthology.orchestra.run.smithi088.stderr:epoch 1 2024-06-03T09:06:58.870 INFO:teuthology.orchestra.run.smithi088.stderr:fsid ec3c3fdd-3ac6-4d7d-9412-683130ed8b04 2024-06-03T09:06:58.870 INFO:teuthology.orchestra.run.smithi088.stderr:last_changed 2024-06-03T08:34:31.885125+0000 2024-06-03T09:06:58.870 INFO:teuthology.orchestra.run.smithi088.stderr:created 2024-06-03T08:34:31.885125+0000 2024-06-03T09:06:58.870 INFO:teuthology.orchestra.run.smithi088.stderr:min_mon_release 18 (reef) 2024-06-03T09:06:58.870 INFO:teuthology.orchestra.run.smithi088.stderr:election_strategy: 3 2024-06-03T09:06:58.870 INFO:teuthology.orchestra.run.smithi088.stderr:0: v1:172.21.15.88:6789/0 mon.a 2024-06-03T09:06:58.870 INFO:teuthology.orchestra.run.smithi088.stderr:1: v1:172.21.15.192:6789/0 mon.b 2024-06-03T09:06:58.870 INFO:teuthology.orchestra.run.smithi088.stderr:2: v1:172.21.15.88:6790/0 mon.c 2024-06-03T09:06:58.870 INFO:teuthology.orchestra.run.smithi088.stderr: 2024-06-03T09:06:58.870 INFO:teuthology.orchestra.run.smithi088.stderr:2024-06-03T09:06:58.867+0000 7f2422741640 10 monclient: get_monmap_and_config success 2024-06-03T09:06:58.870 INFO:teuthology.orchestra.run.smithi088.stderr:2024-06-03T09:06:58.867+0000 7f2422741640 4 set_mon_vals no callback set 2024-06-03T09:06:58.870 INFO:teuthology.orchestra.run.smithi088.stderr:2024-06-03T09:06:58.868+0000 7f2422741640 10 monclient: shutdown 2024-06-03T09:06:58.871 INFO:teuthology.orchestra.run.smithi088.stderr:2024-06-03T09:06:58.868+0000 7f2422741640 20 monclient: shutdown discarding 0 pending message(s) 2024-06-03T09:06:58.871 INFO:teuthology.orchestra.run.smithi088.stderr:2024-06-03T09:06:58.868+0000 7f2422741640 1 -- 172.21.15.88:0/4011089670 >> v1:172.21.15.192:6789/0 conn(0x7f241c0b0d00 legacy=0x7f241c13e360 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-06-03T09:06:58.871 INFO:teuthology.orchestra.run.smithi088.stderr:2024-06-03T09:06:58.868+0000 7f2422741640 1 -- 172.21.15.88:0/4011089670 shutdown_connections 2024-06-03T09:06:58.871 INFO:teuthology.orchestra.run.smithi088.stderr:2024-06-03T09:06:58.868+0000 7f2422741640 1 -- 172.21.15.88:0/4011089670 >> 172.21.15.88:0/4011089670 conn(0x7f241c0b70d0 msgr2=0x7f241c0b74e0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-06-03T09:06:58.871 INFO:teuthology.orchestra.run.smithi088.stderr:2024-06-03T09:06:58.868+0000 7f2422741640 1 -- 172.21.15.88:0/4011089670 shutdown_connections 2024-06-03T09:06:58.871 INFO:teuthology.orchestra.run.smithi088.stderr:2024-06-03T09:06:58.868+0000 7f2422741640 1 -- 172.21.15.88:0/4011089670 wait complete. 2024-06-03T09:06:58.871 INFO:teuthology.orchestra.run.smithi088.stderr:2024-06-03T09:06:58.869+0000 7f2422741640 10 monclient: build_initial_monmap 2024-06-03T09:06:58.871 INFO:teuthology.orchestra.run.smithi088.stderr:2024-06-03T09:06:58.869+0000 7f2422741640 10 monclient: monmap: 2024-06-03T09:06:58.871 INFO:teuthology.orchestra.run.smithi088.stderr:epoch 0 2024-06-03T09:06:58.871 INFO:teuthology.orchestra.run.smithi088.stderr:fsid 00000000-0000-0000-0000-000000000000 2024-06-03T09:06:58.871 INFO:teuthology.orchestra.run.smithi088.stderr:last_changed 0.000000 2024-06-03T09:06:58.871 INFO:teuthology.orchestra.run.smithi088.stderr:created 0.000000 2024-06-03T09:06:58.871 INFO:teuthology.orchestra.run.smithi088.stderr:min_mon_release 0 (unknown) 2024-06-03T09:06:58.872 INFO:teuthology.orchestra.run.smithi088.stderr:election_strategy: 1 2024-06-03T09:06:58.872 INFO:teuthology.orchestra.run.smithi088.stderr:0: v1:172.21.15.88:6789/0 mon.noname-a 2024-06-03T09:06:58.872 INFO:teuthology.orchestra.run.smithi088.stderr:1: v1:172.21.15.192:6789/0 mon.noname-b 2024-06-03T09:06:58.872 INFO:teuthology.orchestra.run.smithi088.stderr:2: v1:172.21.15.88:6790/0 mon.noname-c 2024-06-03T09:06:58.872 INFO:teuthology.orchestra.run.smithi088.stderr: 2024-06-03T09:06:58.872 INFO:teuthology.orchestra.run.smithi088.stderr:2024-06-03T09:06:58.869+0000 7f2422741640 1 librados: starting msgr at 2024-06-03T09:06:58.872 INFO:teuthology.orchestra.run.smithi088.stderr:2024-06-03T09:06:58.869+0000 7f2422741640 1 librados: starting objecter 2024-06-03T09:06:58.872 INFO:teuthology.orchestra.run.smithi088.stderr:2024-06-03T09:06:58.869+0000 7f2422741640 1 Processor -- start 2024-06-03T09:06:58.872 INFO:teuthology.orchestra.run.smithi088.stderr:2024-06-03T09:06:58.869+0000 7f2422741640 1 -- start start 2024-06-03T09:06:58.872 INFO:teuthology.orchestra.run.smithi088.stderr:2024-06-03T09:06:58.869+0000 7f2422741640 1 librados: setting wanted keys 2024-06-03T09:06:58.872 INFO:teuthology.orchestra.run.smithi088.stderr:2024-06-03T09:06:58.869+0000 7f2422741640 1 librados: calling monclient init 2024-06-03T09:06:58.872 INFO:teuthology.orchestra.run.smithi088.stderr:2024-06-03T09:06:58.869+0000 7f2422741640 10 monclient: init 2024-06-03T09:06:58.874 INFO:teuthology.orchestra.run.smithi088.stderr:2024-06-03T09:06:58.870+0000 7f2422741640 10 monclient: _reopen_session rank -1 2024-06-03T09:06:58.874 INFO:teuthology.orchestra.run.smithi088.stderr:2024-06-03T09:06:58.870+0000 7f2422741640 10 monclient: _add_conns ranks=[0,2,1] 2024-06-03T09:06:58.874 INFO:teuthology.orchestra.run.smithi088.stderr:2024-06-03T09:06:58.870+0000 7f2422741640 10 monclient(hunting): picked mon.noname-a con 0x7f241c0a8a60 addr v1:172.21.15.88:6789/0 2024-06-03T09:06:58.875 INFO:teuthology.orchestra.run.smithi088.stderr:2024-06-03T09:06:58.870+0000 7f2422741640 10 monclient(hunting): picked mon.noname-c con 0x7f241c0b0d00 addr v1:172.21.15.88:6790/0 2024-06-03T09:06:58.875 INFO:teuthology.orchestra.run.smithi088.stderr:2024-06-03T09:06:58.870+0000 7f2422741640 10 monclient(hunting): picked mon.noname-b con 0x7f241c1451d0 addr v1:172.21.15.192:6789/0 2024-06-03T09:06:58.875 INFO:teuthology.orchestra.run.smithi088.stderr:2024-06-03T09:06:58.870+0000 7f2422741640 1 -- --> v1:172.21.15.88:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f241c138d40 con 0x7f241c0a8a60 2024-06-03T09:06:58.875 INFO:teuthology.orchestra.run.smithi088.stderr:2024-06-03T09:06:58.870+0000 7f2422741640 1 -- --> v1:172.21.15.192:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f241c1380f0 con 0x7f241c1451d0 2024-06-03T09:06:58.875 INFO:teuthology.orchestra.run.smithi088.stderr:2024-06-03T09:06:58.870+0000 7f2422741640 1 -- --> v1:172.21.15.88:6790/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f241c135950 con 0x7f241c0b0d00 2024-06-03T09:06:58.875 INFO:teuthology.orchestra.run.smithi088.stderr:2024-06-03T09:06:58.870+0000 7f2422741640 10 monclient(hunting): _renew_subs 2024-06-03T09:06:58.875 INFO:teuthology.orchestra.run.smithi088.stderr:2024-06-03T09:06:58.870+0000 7f2422741640 10 monclient(hunting): authenticate will time out at 3074.680368s 2024-06-03T09:06:58.875 INFO:teuthology.orchestra.run.smithi088.stderr:2024-06-03T09:06:58.870+0000 7f241b7fe640 1 --1- >> v1:172.21.15.88:6790/0 conn(0x7f241c0b0d00 0x7f241c138410 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=1).handle_server_banner_and_identify peer v1:172.21.15.88:6790/0 says I am v1:172.21.15.88:51072/0 (socket says 172.21.15.88:51072) 2024-06-03T09:06:58.875 INFO:teuthology.orchestra.run.smithi088.stderr:2024-06-03T09:06:58.870+0000 7f241b7fe640 1 -- 172.21.15.88:0/3073260023 learned_addr learned my addr 172.21.15.88:0/3073260023 (peer_addr_for_me v1:172.21.15.88:0/0) 2024-06-03T09:06:58.875 INFO:teuthology.orchestra.run.smithi088.stderr:2024-06-03T09:06:58.871+0000 7f2418ff9640 10 client.?.objecter ms_handle_connect 0x7f241c0b0d00 2024-06-03T09:06:58.875 INFO:teuthology.orchestra.run.smithi088.stderr:2024-06-03T09:06:58.871+0000 7f2418ff9640 10 client.?.objecter resend_mon_ops 2024-06-03T09:06:58.875 INFO:teuthology.orchestra.run.smithi088.stderr:2024-06-03T09:06:58.871+0000 7f2418ff9640 10 client.?.objecter ms_handle_connect 0x7f241c0a8a60 2024-06-03T09:06:58.875 INFO:teuthology.orchestra.run.smithi088.stderr:2024-06-03T09:06:58.871+0000 7f2418ff9640 10 client.?.objecter resend_mon_ops 2024-06-03T09:06:58.876 INFO:teuthology.orchestra.run.smithi088.stderr:2024-06-03T09:06:58.871+0000 7f2418ff9640 10 client.?.objecter ms_handle_connect 0x7f241c1451d0 2024-06-03T09:06:58.876 INFO:teuthology.orchestra.run.smithi088.stderr:2024-06-03T09:06:58.871+0000 7f2418ff9640 10 client.?.objecter resend_mon_ops 2024-06-03T09:06:58.876 INFO:teuthology.orchestra.run.smithi088.stderr:2024-06-03T09:06:58.873+0000 7f2418ff9640 1 -- 172.21.15.88:0/3073260023 <== mon.0 v1:172.21.15.88:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 2081578029 0 0) 0x7f241c138d40 con 0x7f241c0a8a60 2024-06-03T09:06:58.876 INFO:teuthology.orchestra.run.smithi088.stderr:2024-06-03T09:06:58.873+0000 7f2418ff9640 10 monclient(hunting): _negotiate 2024-06-03T09:06:58.876 INFO:teuthology.orchestra.run.smithi088.stderr:2024-06-03T09:06:58.873+0000 7f2418ff9640 10 monclient(hunting): _init_auth method 2 2024-06-03T09:06:58.876 INFO:teuthology.orchestra.run.smithi088.stderr:2024-06-03T09:06:58.873+0000 7f2418ff9640 10 monclient(hunting): _init_auth creating new auth 2024-06-03T09:06:58.876 INFO:teuthology.orchestra.run.smithi088.stderr:2024-06-03T09:06:58.873+0000 7f2418ff9640 10 monclient(hunting): my global_id is 30780 2024-06-03T09:06:58.876 INFO:teuthology.orchestra.run.smithi088.stderr:2024-06-03T09:06:58.873+0000 7f2418ff9640 1 -- 172.21.15.88:0/3073260023 --> v1:172.21.15.88:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f2408004eb0 con 0x7f241c0a8a60 2024-06-03T09:06:58.876 INFO:teuthology.orchestra.run.smithi088.stderr:2024-06-03T09:06:58.873+0000 7f2418ff9640 1 -- 172.21.15.88:0/3073260023 <== mon.2 v1:172.21.15.88:6790/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 3009475801 0 0) 0x7f241c135950 con 0x7f241c0b0d00 2024-06-03T09:06:58.876 INFO:teuthology.orchestra.run.smithi088.stderr:2024-06-03T09:06:58.873+0000 7f2418ff9640 10 monclient(hunting): _negotiate 2024-06-03T09:06:58.876 INFO:teuthology.orchestra.run.smithi088.stderr:2024-06-03T09:06:58.873+0000 7f2418ff9640 10 monclient(hunting): _init_auth method 2 2024-06-03T09:06:58.876 INFO:teuthology.orchestra.run.smithi088.stderr:2024-06-03T09:06:58.873+0000 7f2418ff9640 10 monclient(hunting): _init_auth creating new auth 2024-06-03T09:06:58.876 INFO:teuthology.orchestra.run.smithi088.stderr:2024-06-03T09:06:58.873+0000 7f2418ff9640 10 monclient(hunting): my global_id is 30287 2024-06-03T09:06:58.876 INFO:teuthology.orchestra.run.smithi088.stderr:2024-06-03T09:06:58.873+0000 7f2418ff9640 1 -- 172.21.15.88:0/3073260023 --> v1:172.21.15.88:6790/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f241c138d40 con 0x7f241c0b0d00 2024-06-03T09:06:58.877 INFO:teuthology.orchestra.run.smithi088.stderr:2024-06-03T09:06:58.873+0000 7f2418ff9640 1 -- 172.21.15.88:0/3073260023 <== mon.1 v1:172.21.15.192:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 2261448920 0 0) 0x7f241c1380f0 con 0x7f241c1451d0 2024-06-03T09:06:58.877 INFO:teuthology.orchestra.run.smithi088.stderr:2024-06-03T09:06:58.873+0000 7f2418ff9640 10 monclient(hunting): _negotiate 2024-06-03T09:06:58.877 INFO:teuthology.orchestra.run.smithi088.stderr:2024-06-03T09:06:58.873+0000 7f2418ff9640 10 monclient(hunting): _init_auth method 2 2024-06-03T09:06:58.877 INFO:teuthology.orchestra.run.smithi088.stderr:2024-06-03T09:06:58.873+0000 7f2418ff9640 10 monclient(hunting): _init_auth creating new auth 2024-06-03T09:06:58.877 INFO:teuthology.orchestra.run.smithi088.stderr:2024-06-03T09:06:58.873+0000 7f2418ff9640 10 monclient(hunting): my global_id is 30352 2024-06-03T09:06:58.877 INFO:teuthology.orchestra.run.smithi088.stderr:2024-06-03T09:06:58.873+0000 7f2418ff9640 1 -- 172.21.15.88:0/3073260023 --> v1:172.21.15.192:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f241c135950 con 0x7f241c1451d0 2024-06-03T09:06:58.880 INFO:teuthology.orchestra.run.smithi088.stderr:2024-06-03T09:06:58.880+0000 7f2418ff9640 1 -- 172.21.15.88:0/3073260023 <== mon.2 v1:172.21.15.88:6790/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 764+0+0 (unknown 1846180248 0 0) 0x7f241c138d40 con 0x7f241c0b0d00 2024-06-03T09:06:58.880 INFO:teuthology.orchestra.run.smithi088.stderr:2024-06-03T09:06:58.880+0000 7f2418ff9640 1 -- 172.21.15.88:0/3073260023 --> v1:172.21.15.88:6790/0 -- auth(proto 2 165 bytes epoch 0) v1 -- 0x7f241c1380f0 con 0x7f241c0b0d00 2024-06-03T09:06:58.880 INFO:teuthology.orchestra.run.smithi088.stderr:2024-06-03T09:06:58.880+0000 7f2418ff9640 1 -- 172.21.15.88:0/3073260023 <== mon.2 v1:172.21.15.88:6790/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 2234374065 0 0) 0x7f2410001510 con 0x7f241c0b0d00 2024-06-03T09:06:58.880 INFO:teuthology.orchestra.run.smithi088.stderr:2024-06-03T09:06:58.880+0000 7f2418ff9640 10 monclient(hunting): handle_monmap mon_map magic: 0 v1 2024-06-03T09:06:58.880 INFO:teuthology.orchestra.run.smithi088.stderr:2024-06-03T09:06:58.880+0000 7f2418ff9640 10 monclient(hunting): got monmap 1 from mon.noname-c (according to old e1) 2024-06-03T09:06:58.880 INFO:teuthology.orchestra.run.smithi088.stderr:2024-06-03T09:06:58.880+0000 7f2418ff9640 10 monclient(hunting): dump: 2024-06-03T09:06:58.880 INFO:teuthology.orchestra.run.smithi088.stderr:epoch 1 2024-06-03T09:06:58.880 INFO:teuthology.orchestra.run.smithi088.stderr:fsid ec3c3fdd-3ac6-4d7d-9412-683130ed8b04 2024-06-03T09:06:58.881 INFO:teuthology.orchestra.run.smithi088.stderr:last_changed 2024-06-03T08:34:31.885125+0000 2024-06-03T09:06:58.881 INFO:teuthology.orchestra.run.smithi088.stderr:created 2024-06-03T08:34:31.885125+0000 2024-06-03T09:06:58.881 INFO:teuthology.orchestra.run.smithi088.stderr:min_mon_release 18 (reef) 2024-06-03T09:06:58.881 INFO:teuthology.orchestra.run.smithi088.stderr:election_strategy: 3 2024-06-03T09:06:58.881 INFO:teuthology.orchestra.run.smithi088.stderr:0: v1:172.21.15.88:6789/0 mon.a 2024-06-03T09:06:58.881 INFO:teuthology.orchestra.run.smithi088.stderr:1: v1:172.21.15.192:6789/0 mon.b 2024-06-03T09:06:58.881 INFO:teuthology.orchestra.run.smithi088.stderr:2: v1:172.21.15.88:6790/0 mon.c 2024-06-03T09:06:58.881 INFO:teuthology.orchestra.run.smithi088.stderr: 2024-06-03T09:06:58.881 INFO:teuthology.orchestra.run.smithi088.stderr:2024-06-03T09:06:58.880+0000 7f2418ff9640 10 monclient(hunting): _finish_auth 0 2024-06-03T09:06:58.881 INFO:teuthology.orchestra.run.smithi088.stderr:2024-06-03T09:06:58.880+0000 7f2418ff9640 1 -- 172.21.15.88:0/3073260023 <== mon.0 v1:172.21.15.88:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 764+0+0 (unknown 447516604 0 0) 0x7f2408004eb0 con 0x7f241c0a8a60 2024-06-03T09:06:58.881 INFO:teuthology.orchestra.run.smithi088.stderr:2024-06-03T09:06:58.880+0000 7f2418ff9640 1 -- 172.21.15.88:0/3073260023 --> v1:172.21.15.88:6789/0 -- auth(proto 2 165 bytes epoch 0) v1 -- 0x7f241c138d40 con 0x7f241c0a8a60 2024-06-03T09:06:58.881 INFO:teuthology.orchestra.run.smithi088.stderr:2024-06-03T09:06:58.880+0000 7f2418ff9640 1 -- 172.21.15.88:0/3073260023 <== mon.0 v1:172.21.15.88:6789/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 2234374065 0 0) 0x7f240c0018b0 con 0x7f241c0a8a60 2024-06-03T09:06:58.881 INFO:teuthology.orchestra.run.smithi088.stderr:2024-06-03T09:06:58.880+0000 7f2418ff9640 10 monclient(hunting): handle_monmap mon_map magic: 0 v1 2024-06-03T09:06:58.882 INFO:teuthology.orchestra.run.smithi088.stderr:2024-06-03T09:06:58.880+0000 7f2418ff9640 10 monclient(hunting): got monmap 1 from mon.a (according to old e1) 2024-06-03T09:06:58.882 INFO:teuthology.orchestra.run.smithi088.stderr:2024-06-03T09:06:58.880+0000 7f2418ff9640 10 monclient(hunting): dump: 2024-06-03T09:06:58.882 INFO:teuthology.orchestra.run.smithi088.stderr:epoch 1 2024-06-03T09:06:58.882 INFO:teuthology.orchestra.run.smithi088.stderr:fsid ec3c3fdd-3ac6-4d7d-9412-683130ed8b04 2024-06-03T09:06:58.882 INFO:teuthology.orchestra.run.smithi088.stderr:last_changed 2024-06-03T08:34:31.885125+0000 2024-06-03T09:06:58.882 INFO:teuthology.orchestra.run.smithi088.stderr:created 2024-06-03T08:34:31.885125+0000 2024-06-03T09:06:58.882 INFO:teuthology.orchestra.run.smithi088.stderr:min_mon_release 18 (reef) 2024-06-03T09:06:58.882 INFO:teuthology.orchestra.run.smithi088.stderr:election_strategy: 3 2024-06-03T09:06:58.882 INFO:teuthology.orchestra.run.smithi088.stderr:0: v1:172.21.15.88:6789/0 mon.a 2024-06-03T09:06:58.882 INFO:teuthology.orchestra.run.smithi088.stderr:1: v1:172.21.15.192:6789/0 mon.b 2024-06-03T09:06:58.882 INFO:teuthology.orchestra.run.smithi088.stderr:2: v1:172.21.15.88:6790/0 mon.c 2024-06-03T09:06:58.882 INFO:teuthology.orchestra.run.smithi088.stderr: 2024-06-03T09:06:58.885 INFO:teuthology.orchestra.run.smithi088.stderr:2024-06-03T09:06:58.884+0000 7f2418ff9640 1 -- 172.21.15.88:0/3073260023 <== mon.2 v1:172.21.15.88:6790/0 4 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 580+0+0 (unknown 746055325 0 0) 0x7f241c1380f0 con 0x7f241c0b0d00 2024-06-03T09:06:58.885 INFO:teuthology.orchestra.run.smithi088.stderr:2024-06-03T09:06:58.884+0000 7f2418ff9640 1 -- 172.21.15.88:0/3073260023 >> v1:172.21.15.192:6789/0 conn(0x7f241c1451d0 legacy=0x7f241c135240 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-06-03T09:06:58.885 INFO:teuthology.orchestra.run.smithi088.stderr:2024-06-03T09:06:58.884+0000 7f2418ff9640 1 -- 172.21.15.88:0/3073260023 >> v1:172.21.15.88:6789/0 conn(0x7f241c0a8a60 legacy=0x7f241c1d9b70 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-06-03T09:06:58.885 INFO:teuthology.orchestra.run.smithi088.stderr:2024-06-03T09:06:58.884+0000 7f2418ff9640 10 monclient: _finish_hunting 0 2024-06-03T09:06:58.885 INFO:teuthology.orchestra.run.smithi088.stderr:2024-06-03T09:06:58.884+0000 7f2418ff9640 1 monclient: found mon.c 2024-06-03T09:06:58.885 INFO:teuthology.orchestra.run.smithi088.stderr:2024-06-03T09:06:58.884+0000 7f2418ff9640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-06-03T09:06:58.885 INFO:teuthology.orchestra.run.smithi088.stderr:2024-06-03T09:06:58.884+0000 7f2418ff9640 10 monclient: _send_mon_message to mon.c at v1:172.21.15.88:6790/0 2024-06-03T09:06:58.885 INFO:teuthology.orchestra.run.smithi088.stderr:2024-06-03T09:06:58.884+0000 7f2418ff9640 1 -- 172.21.15.88:0/3073260023 --> v1:172.21.15.88:6790/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f241c135ae0 con 0x7f241c0b0d00 2024-06-03T09:06:58.885 INFO:teuthology.orchestra.run.smithi088.stderr:2024-06-03T09:06:58.884+0000 7f2418ff9640 10 monclient: _finish_auth 0 2024-06-03T09:06:58.885 INFO:teuthology.orchestra.run.smithi088.stderr:2024-06-03T09:06:58.884+0000 7f2418ff9640 10 monclient: _check_auth_tickets 2024-06-03T09:06:58.885 INFO:teuthology.orchestra.run.smithi088.stderr:2024-06-03T09:06:58.884+0000 7f2418ff9640 20 monclient: _check_auth_rotating not needed by client.admin 2024-06-03T09:06:58.886 INFO:teuthology.orchestra.run.smithi088.stderr:2024-06-03T09:06:58.884+0000 7f2422741640 5 monclient: authenticate success, global_id 30287 2024-06-03T09:06:58.886 INFO:teuthology.orchestra.run.smithi088.stderr:2024-06-03T09:06:58.884+0000 7f2422741640 10 monclient: _renew_subs 2024-06-03T09:06:58.886 INFO:teuthology.orchestra.run.smithi088.stderr:2024-06-03T09:06:58.884+0000 7f2422741640 10 monclient: _send_mon_message to mon.c at v1:172.21.15.88:6790/0 2024-06-03T09:06:58.886 INFO:teuthology.orchestra.run.smithi088.stderr:2024-06-03T09:06:58.884+0000 7f2422741640 1 -- 172.21.15.88:0/3073260023 --> v1:172.21.15.88:6790/0 -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f241c141d80 con 0x7f241c0b0d00 2024-06-03T09:06:58.886 INFO:teuthology.orchestra.run.smithi088.stderr:2024-06-03T09:06:58.884+0000 7f2422741640 10 client.30287.objecter _maybe_request_map subscribing (onetime) to next osd map 2024-06-03T09:06:58.886 INFO:teuthology.orchestra.run.smithi088.stderr:2024-06-03T09:06:58.884+0000 7f2422741640 10 monclient: _renew_subs 2024-06-03T09:06:58.886 INFO:teuthology.orchestra.run.smithi088.stderr:2024-06-03T09:06:58.884+0000 7f2422741640 10 monclient: _send_mon_message to mon.c at v1:172.21.15.88:6790/0 2024-06-03T09:06:58.886 INFO:teuthology.orchestra.run.smithi088.stderr:2024-06-03T09:06:58.884+0000 7f2422741640 1 -- 172.21.15.88:0/3073260023 --> v1:172.21.15.88:6790/0 -- mon_subscribe({osdmap=0}) v3 -- 0x7f241c139260 con 0x7f241c0b0d00 2024-06-03T09:06:58.886 INFO:teuthology.orchestra.run.smithi088.stderr:2024-06-03T09:06:58.884+0000 7f2422741640 1 librados: init done 2024-06-03T09:06:58.886 INFO:teuthology.orchestra.run.smithi088.stderr:2024-06-03T09:06:58.885+0000 7f2422741640 10 start_mon_command cmd=[{"prefix": "get_command_descriptions"}] 2024-06-03T09:06:58.886 INFO:teuthology.orchestra.run.smithi088.stderr:2024-06-03T09:06:58.885+0000 7f2422741640 10 monclient: _send_command 1 [{"prefix": "get_command_descriptions"}] 2024-06-03T09:06:58.886 INFO:teuthology.orchestra.run.smithi088.stderr:2024-06-03T09:06:58.885+0000 7f2422741640 10 monclient: _send_mon_message to mon.c at v1:172.21.15.88:6790/0 2024-06-03T09:06:58.886 INFO:teuthology.orchestra.run.smithi088.stderr:2024-06-03T09:06:58.885+0000 7f2422741640 1 -- 172.21.15.88:0/3073260023 --> v1:172.21.15.88:6790/0 -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f241c135d30 con 0x7f241c0b0d00 2024-06-03T09:06:58.887 INFO:teuthology.orchestra.run.smithi088.stderr:2024-06-03T09:06:58.887+0000 7f2418ff9640 1 -- 172.21.15.88:0/3073260023 <== mon.2 v1:172.21.15.88:6790/0 5 ==== config(0 keys) v1 ==== 4+0+0 (unknown 0 0 0) 0x7f24100038a0 con 0x7f241c0b0d00 2024-06-03T09:06:58.887 INFO:teuthology.orchestra.run.smithi088.stderr:2024-06-03T09:06:58.887+0000 7f2418ff9640 10 monclient: handle_config config(0 keys) v1 2024-06-03T09:06:58.887 INFO:teuthology.orchestra.run.smithi088.stderr:2024-06-03T09:06:58.887+0000 7f241a7fc640 4 set_mon_vals no callback set 2024-06-03T09:06:59.137 INFO:teuthology.orchestra.run.smithi088.stderr:2024-06-03T09:06:59.137+0000 7f2418ff9640 1 -- 172.21.15.88:0/3073260023 <== mon.2 v1:172.21.15.88:6790/0 6 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 2234374065 0 0) 0x7f2410003c30 con 0x7f241c0b0d00 2024-06-03T09:06:59.138 INFO:teuthology.orchestra.run.smithi088.stderr:2024-06-03T09:06:59.137+0000 7f2418ff9640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-06-03T09:06:59.138 INFO:teuthology.orchestra.run.smithi088.stderr:2024-06-03T09:06:59.137+0000 7f2418ff9640 10 monclient: got monmap 1 from mon.c (according to old e1) 2024-06-03T09:06:59.138 INFO:teuthology.orchestra.run.smithi088.stderr:2024-06-03T09:06:59.137+0000 7f2418ff9640 10 monclient: dump: 2024-06-03T09:06:59.138 INFO:teuthology.orchestra.run.smithi088.stderr:epoch 1 2024-06-03T09:06:59.138 INFO:teuthology.orchestra.run.smithi088.stderr:fsid ec3c3fdd-3ac6-4d7d-9412-683130ed8b04 2024-06-03T09:06:59.138 INFO:teuthology.orchestra.run.smithi088.stderr:last_changed 2024-06-03T08:34:31.885125+0000 2024-06-03T09:06:59.138 INFO:teuthology.orchestra.run.smithi088.stderr:created 2024-06-03T08:34:31.885125+0000 2024-06-03T09:06:59.138 INFO:teuthology.orchestra.run.smithi088.stderr:min_mon_release 18 (reef) 2024-06-03T09:06:59.138 INFO:teuthology.orchestra.run.smithi088.stderr:election_strategy: 3 2024-06-03T09:06:59.138 INFO:teuthology.orchestra.run.smithi088.stderr:0: v1:172.21.15.88:6789/0 mon.a 2024-06-03T09:06:59.138 INFO:teuthology.orchestra.run.smithi088.stderr:1: v1:172.21.15.192:6789/0 mon.b 2024-06-03T09:06:59.138 INFO:teuthology.orchestra.run.smithi088.stderr:2: v1:172.21.15.88:6790/0 mon.c 2024-06-03T09:06:59.138 INFO:teuthology.orchestra.run.smithi088.stderr: 2024-06-03T09:06:59.139 INFO:teuthology.orchestra.run.smithi088.stderr:2024-06-03T09:06:59.139+0000 7f2418ff9640 1 -- 172.21.15.88:0/3073260023 <== mon.2 v1:172.21.15.88:6790/0 7 ==== mgrmap(e 5) v1 ==== 89110+0+0 (unknown 873533973 0 0) 0x7f2410003e10 con 0x7f241c0b0d00 2024-06-03T09:06:59.139 INFO:teuthology.orchestra.run.smithi088.stderr:2024-06-03T09:06:59.139+0000 7f241bfff640 10 monclient: get_auth_request con 0x7f240806d2b0 auth_method 0 2024-06-03T09:06:59.140 INFO:teuthology.orchestra.run.smithi088.stderr:2024-06-03T09:06:59.140+0000 7f2418ff9640 10 client.30287.objecter ms_handle_connect 0x7f240806d2b0 2024-06-03T09:06:59.143 INFO:teuthology.orchestra.run.smithi088.stderr:2024-06-03T09:06:59.143+0000 7f2418ff9640 1 -- 172.21.15.88:0/3073260023 <== mon.2 v1:172.21.15.88:6790/0 8 ==== osd_map(1014..1014 src has 622..1014) v4 ==== 5229+0+0 (unknown 3230192161 0 0) 0x7f24100011b0 con 0x7f241c0b0d00 2024-06-03T09:06:59.143 INFO:teuthology.orchestra.run.smithi088.stderr:2024-06-03T09:06:59.143+0000 7f2418ff9640 10 client.30287.objecter ms_dispatch 0x7f241c000b90 osd_map(1014..1014 src has 622..1014) v4 2024-06-03T09:06:59.143 INFO:teuthology.orchestra.run.smithi088.stderr:2024-06-03T09:06:59.143+0000 7f2418ff9640 3 client.30287.objecter handle_osd_map got epochs [1014,1014] > 0 2024-06-03T09:06:59.143 INFO:teuthology.orchestra.run.smithi088.stderr:2024-06-03T09:06:59.143+0000 7f2418ff9640 3 client.30287.objecter handle_osd_map decoding full epoch 1014 2024-06-03T09:06:59.143 INFO:teuthology.orchestra.run.smithi088.stderr:2024-06-03T09:06:59.143+0000 7f2418ff9640 20 client.30287.objecter dump_active .. 0 homeless 2024-06-03T09:06:59.143 INFO:teuthology.orchestra.run.smithi088.stderr:2024-06-03T09:06:59.143+0000 7f2418ff9640 1 -- 172.21.15.88:0/3073260023 <== mon.2 v1:172.21.15.88:6790/0 9 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+182498 (unknown 1092875540 0 3016760700) 0x7f241004fdb0 con 0x7f241c0b0d00 2024-06-03T09:06:59.143 INFO:teuthology.orchestra.run.smithi088.stderr:2024-06-03T09:06:59.143+0000 7f2418ff9640 10 monclient: handle_mon_command_ack 1 [{"prefix": "get_command_descriptions"}] 2024-06-03T09:06:59.143 INFO:teuthology.orchestra.run.smithi088.stderr:2024-06-03T09:06:59.143+0000 7f2418ff9640 10 monclient: _finish_command 1 = system:0 2024-06-03T09:06:59.320 INFO:teuthology.orchestra.run.smithi088.stderr:2024-06-03T09:06:59.320+0000 7f2422741640 1 -- 172.21.15.88:0/3073260023 --> v1:172.21.15.192:6800/34229 -- mgr_command(tid 0: {"prefix": "osd deep-scrub", "who": "3", "target": ["mon-mgr", ""]}) v1 -- 0x7f241c0a94e0 con 0x7f240806d2b0 2024-06-03T09:06:59.321 INFO:teuthology.orchestra.run.smithi088.stderr:2024-06-03T09:06:59.321+0000 7f2418ff9640 1 -- 172.21.15.88:0/3073260023 <== mgr.4110 v1:172.21.15.192:6800/34229 1 ==== mgr_command_reply(tid 0: 0 instructed osd(s) 3 to deep-scrub) v1 ==== 41+0+0 (unknown 3526570807 0 0) 0x7f240c001ce0 con 0x7f240806d2b0 2024-06-03T09:06:59.321 INFO:teuthology.orchestra.run.smithi088.stderr:instructed osd(s) 3 to deep-scrub 2024-06-03T09:06:59.323 INFO:teuthology.orchestra.run.smithi088.stderr:2024-06-03T09:06:59.322+0000 7f2422741640 10 librados: watch_flush enter 2024-06-03T09:06:59.323 INFO:teuthology.orchestra.run.smithi088.stderr:2024-06-03T09:06:59.322+0000 7f2422741640 10 librados: watch_flush exit 2024-06-03T09:06:59.323 INFO:teuthology.orchestra.run.smithi088.stderr:2024-06-03T09:06:59.322+0000 7f2422741640 20 client.30287.objecter shutdown clearing up homeless session... 2024-06-03T09:06:59.323 INFO:teuthology.orchestra.run.smithi088.stderr:2024-06-03T09:06:59.322+0000 7f2422741640 10 client.30287.objecter successfully canceled tick 2024-06-03T09:06:59.323 INFO:teuthology.orchestra.run.smithi088.stderr:2024-06-03T09:06:59.323+0000 7f2422741640 1 -- 172.21.15.88:0/3073260023 >> v1:172.21.15.192:6800/34229 conn(0x7f240806d2b0 legacy=0x7f240806f770 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-06-03T09:06:59.323 INFO:teuthology.orchestra.run.smithi088.stderr:2024-06-03T09:06:59.323+0000 7f2422741640 10 monclient: shutdown 2024-06-03T09:06:59.323 INFO:teuthology.orchestra.run.smithi088.stderr:2024-06-03T09:06:59.323+0000 7f2422741640 20 monclient: shutdown discarding 0 pending message(s) 2024-06-03T09:06:59.323 INFO:teuthology.orchestra.run.smithi088.stderr:2024-06-03T09:06:59.323+0000 7f2422741640 1 -- 172.21.15.88:0/3073260023 >> v1:172.21.15.88:6790/0 conn(0x7f241c0b0d00 legacy=0x7f241c138410 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-06-03T09:06:59.324 INFO:teuthology.orchestra.run.smithi088.stderr:2024-06-03T09:06:59.323+0000 7f2422741640 1 -- 172.21.15.88:0/3073260023 shutdown_connections 2024-06-03T09:06:59.324 INFO:teuthology.orchestra.run.smithi088.stderr:2024-06-03T09:06:59.323+0000 7f2422741640 1 -- 172.21.15.88:0/3073260023 >> 172.21.15.88:0/3073260023 conn(0x7f241c0b70d0 msgr2=0x7f241c0b5b20 unknown :-1 s=STATE_NONE l=0).mark_down 2024-06-03T09:06:59.324 INFO:teuthology.orchestra.run.smithi088.stderr:2024-06-03T09:06:59.323+0000 7f2422741640 1 -- 172.21.15.88:0/3073260023 shutdown_connections 2024-06-03T09:06:59.324 INFO:teuthology.orchestra.run.smithi088.stderr:2024-06-03T09:06:59.323+0000 7f2422741640 1 -- 172.21.15.88:0/3073260023 wait complete. 2024-06-03T09:06:59.324 INFO:teuthology.orchestra.run.smithi088.stderr:2024-06-03T09:06:59.323+0000 7f2422741640 1 librados: shutdown 2024-06-03T09:06:59.334 INFO:tasks.ceph:Scrubbing osd.4 2024-06-03T09:06:59.334 DEBUG:teuthology.orchestra.run.smithi088:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph tell osd.4 config set osd_debug_deep_scrub_sleep 0 2024-06-03T09:06:59.438 INFO:teuthology.orchestra.run.smithi088.stderr:2024-06-03T09:06:59.438+0000 7f1861ba7640 10 monclient: get_monmap_and_config 2024-06-03T09:06:59.438 INFO:teuthology.orchestra.run.smithi088.stderr:2024-06-03T09:06:59.438+0000 7f1861ba7640 10 monclient: build_initial_monmap 2024-06-03T09:06:59.438 INFO:teuthology.orchestra.run.smithi088.stderr:2024-06-03T09:06:59.438+0000 7f1861ba7640 10 monclient: monmap: 2024-06-03T09:06:59.438 INFO:teuthology.orchestra.run.smithi088.stderr:epoch 0 2024-06-03T09:06:59.438 INFO:teuthology.orchestra.run.smithi088.stderr:fsid ec3c3fdd-3ac6-4d7d-9412-683130ed8b04 2024-06-03T09:06:59.438 INFO:teuthology.orchestra.run.smithi088.stderr:last_changed 2024-06-03T09:06:59.439397+0000 2024-06-03T09:06:59.439 INFO:teuthology.orchestra.run.smithi088.stderr:created 2024-06-03T09:06:59.439397+0000 2024-06-03T09:06:59.439 INFO:teuthology.orchestra.run.smithi088.stderr:min_mon_release 0 (unknown) 2024-06-03T09:06:59.439 INFO:teuthology.orchestra.run.smithi088.stderr:election_strategy: 3 2024-06-03T09:06:59.439 INFO:teuthology.orchestra.run.smithi088.stderr:0: v1:172.21.15.88:6789/0 mon.noname-a 2024-06-03T09:06:59.439 INFO:teuthology.orchestra.run.smithi088.stderr:1: v1:172.21.15.192:6789/0 mon.noname-c 2024-06-03T09:06:59.439 INFO:teuthology.orchestra.run.smithi088.stderr:2: v1:172.21.15.88:6790/0 mon.noname-b 2024-06-03T09:06:59.439 INFO:teuthology.orchestra.run.smithi088.stderr: 2024-06-03T09:06:59.440 INFO:teuthology.orchestra.run.smithi088.stderr:2024-06-03T09:06:59.439+0000 7f1861ba7640 1 Processor -- start 2024-06-03T09:06:59.440 INFO:teuthology.orchestra.run.smithi088.stderr:2024-06-03T09:06:59.439+0000 7f1861ba7640 1 -- start start 2024-06-03T09:06:59.440 INFO:teuthology.orchestra.run.smithi088.stderr:2024-06-03T09:06:59.439+0000 7f1861ba7640 10 monclient: init 2024-06-03T09:06:59.440 INFO:teuthology.orchestra.run.smithi088.stderr:2024-06-03T09:06:59.439+0000 7f1861ba7640 10 monclient: _reopen_session rank -1 2024-06-03T09:06:59.441 INFO:teuthology.orchestra.run.smithi088.stderr:2024-06-03T09:06:59.439+0000 7f1861ba7640 10 monclient: _add_conns ranks=[0,2,1] 2024-06-03T09:06:59.441 INFO:teuthology.orchestra.run.smithi088.stderr:2024-06-03T09:06:59.439+0000 7f1861ba7640 10 monclient(hunting): picked mon.noname-a con 0x7f185c0a8c10 addr v1:172.21.15.88:6789/0 2024-06-03T09:06:59.441 INFO:teuthology.orchestra.run.smithi088.stderr:2024-06-03T09:06:59.439+0000 7f1861ba7640 10 monclient(hunting): picked mon.noname-b con 0x7f185c0b1ad0 addr v1:172.21.15.88:6790/0 2024-06-03T09:06:59.441 INFO:teuthology.orchestra.run.smithi088.stderr:2024-06-03T09:06:59.439+0000 7f1861ba7640 10 monclient(hunting): picked mon.noname-c con 0x7f185c14ae30 addr v1:172.21.15.192:6789/0 2024-06-03T09:06:59.441 INFO:teuthology.orchestra.run.smithi088.stderr:2024-06-03T09:06:59.439+0000 7f1861ba7640 1 -- --> v1:172.21.15.88:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f185c144aa0 con 0x7f185c0a8c10 2024-06-03T09:06:59.441 INFO:teuthology.orchestra.run.smithi088.stderr:2024-06-03T09:06:59.439+0000 7f1861ba7640 1 -- --> v1:172.21.15.192:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f185c144d80 con 0x7f185c14ae30 2024-06-03T09:06:59.441 INFO:teuthology.orchestra.run.smithi088.stderr:2024-06-03T09:06:59.439+0000 7f1861ba7640 1 -- --> v1:172.21.15.88:6790/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f185c145920 con 0x7f185c0b1ad0 2024-06-03T09:06:59.441 INFO:teuthology.orchestra.run.smithi088.stderr:2024-06-03T09:06:59.439+0000 7f1861ba7640 10 monclient(hunting): _renew_subs 2024-06-03T09:06:59.441 INFO:teuthology.orchestra.run.smithi088.stderr:2024-06-03T09:06:59.439+0000 7f1861ba7640 10 monclient(hunting): authenticate will time out at 3075.249855s 2024-06-03T09:06:59.441 INFO:teuthology.orchestra.run.smithi088.stderr:2024-06-03T09:06:59.440+0000 7f185bfff640 1 --1- >> v1:172.21.15.192:6789/0 conn(0x7f185c14ae30 0x7f185c14d220 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.192:6789/0 says I am v1:172.21.15.88:50574/0 (socket says 172.21.15.88:50574) 2024-06-03T09:06:59.441 INFO:teuthology.orchestra.run.smithi088.stderr:2024-06-03T09:06:59.440+0000 7f185bfff640 1 -- 172.21.15.88:0/3411910040 learned_addr learned my addr 172.21.15.88:0/3411910040 (peer_addr_for_me v1:172.21.15.88:0/0) 2024-06-03T09:06:59.458 INFO:teuthology.orchestra.run.smithi088.stderr:2024-06-03T09:06:59.458+0000 7f185a7fc640 1 -- 172.21.15.88:0/3411910040 <== mon.0 v1:172.21.15.88:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 2740699571 0 0) 0x7f185c144aa0 con 0x7f185c0a8c10 2024-06-03T09:06:59.458 INFO:teuthology.orchestra.run.smithi088.stderr:2024-06-03T09:06:59.458+0000 7f185a7fc640 10 monclient(hunting): _negotiate 2024-06-03T09:06:59.459 INFO:teuthology.orchestra.run.smithi088.stderr:2024-06-03T09:06:59.458+0000 7f185a7fc640 10 monclient(hunting): _init_auth method 2 2024-06-03T09:06:59.459 INFO:teuthology.orchestra.run.smithi088.stderr:2024-06-03T09:06:59.458+0000 7f185a7fc640 10 monclient(hunting): _init_auth creating new auth 2024-06-03T09:06:59.459 INFO:teuthology.orchestra.run.smithi088.stderr:2024-06-03T09:06:59.458+0000 7f185a7fc640 10 monclient(hunting): my global_id is 30783 2024-06-03T09:06:59.459 INFO:teuthology.orchestra.run.smithi088.stderr:2024-06-03T09:06:59.458+0000 7f185a7fc640 1 -- 172.21.15.88:0/3411910040 --> v1:172.21.15.88:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f1844003520 con 0x7f185c0a8c10 2024-06-03T09:06:59.461 INFO:teuthology.orchestra.run.smithi088.stderr:2024-06-03T09:06:59.461+0000 7f185a7fc640 1 -- 172.21.15.88:0/3411910040 <== mon.1 v1:172.21.15.192:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 309437106 0 0) 0x7f185c144d80 con 0x7f185c14ae30 2024-06-03T09:06:59.461 INFO:teuthology.orchestra.run.smithi088.stderr:2024-06-03T09:06:59.461+0000 7f185a7fc640 10 monclient(hunting): _negotiate 2024-06-03T09:06:59.461 INFO:teuthology.orchestra.run.smithi088.stderr:2024-06-03T09:06:59.461+0000 7f185a7fc640 10 monclient(hunting): _init_auth method 2 2024-06-03T09:06:59.461 INFO:teuthology.orchestra.run.smithi088.stderr:2024-06-03T09:06:59.461+0000 7f185a7fc640 10 monclient(hunting): _init_auth creating new auth 2024-06-03T09:06:59.461 INFO:teuthology.orchestra.run.smithi088.stderr:2024-06-03T09:06:59.461+0000 7f185a7fc640 10 monclient(hunting): my global_id is 30355 2024-06-03T09:06:59.462 INFO:teuthology.orchestra.run.smithi088.stderr:2024-06-03T09:06:59.461+0000 7f185a7fc640 1 -- 172.21.15.88:0/3411910040 --> v1:172.21.15.192:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f185c144aa0 con 0x7f185c14ae30 2024-06-03T09:06:59.464 INFO:teuthology.orchestra.run.smithi088.stderr:2024-06-03T09:06:59.463+0000 7f185a7fc640 1 -- 172.21.15.88:0/3411910040 <== mon.0 v1:172.21.15.88:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 214+0+0 (unknown 2638028507 0 0) 0x7f1844003520 con 0x7f185c0a8c10 2024-06-03T09:06:59.464 INFO:teuthology.orchestra.run.smithi088.stderr:2024-06-03T09:06:59.463+0000 7f185a7fc640 1 -- 172.21.15.88:0/3411910040 >> v1:172.21.15.88:6790/0 conn(0x7f185c0b1ad0 legacy=0x7f185c13e8d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-06-03T09:06:59.464 INFO:teuthology.orchestra.run.smithi088.stderr:2024-06-03T09:06:59.463+0000 7f185a7fc640 1 -- 172.21.15.88:0/3411910040 >> v1:172.21.15.192:6789/0 conn(0x7f185c14ae30 legacy=0x7f185c14d220 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-06-03T09:06:59.464 INFO:teuthology.orchestra.run.smithi088.stderr:2024-06-03T09:06:59.463+0000 7f185a7fc640 10 monclient: _finish_hunting 0 2024-06-03T09:06:59.464 INFO:teuthology.orchestra.run.smithi088.stderr:2024-06-03T09:06:59.463+0000 7f185a7fc640 1 monclient: found mon.noname-a 2024-06-03T09:06:59.464 INFO:teuthology.orchestra.run.smithi088.stderr:2024-06-03T09:06:59.463+0000 7f185a7fc640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-06-03T09:06:59.464 INFO:teuthology.orchestra.run.smithi088.stderr:2024-06-03T09:06:59.463+0000 7f185a7fc640 10 monclient: _send_mon_message to mon.noname-a at v1:172.21.15.88:6789/0 2024-06-03T09:06:59.464 INFO:teuthology.orchestra.run.smithi088.stderr:2024-06-03T09:06:59.463+0000 7f185a7fc640 1 -- 172.21.15.88:0/3411910040 --> v1:172.21.15.88:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f185c0a9d20 con 0x7f185c0a8c10 2024-06-03T09:06:59.465 INFO:teuthology.orchestra.run.smithi088.stderr:2024-06-03T09:06:59.463+0000 7f185a7fc640 10 monclient: _finish_auth 0 2024-06-03T09:06:59.465 INFO:teuthology.orchestra.run.smithi088.stderr:2024-06-03T09:06:59.463+0000 7f185a7fc640 10 monclient: _check_auth_tickets 2024-06-03T09:06:59.465 INFO:teuthology.orchestra.run.smithi088.stderr:2024-06-03T09:06:59.463+0000 7f185a7fc640 20 monclient: _check_auth_rotating not needed by client.admin 2024-06-03T09:06:59.465 INFO:teuthology.orchestra.run.smithi088.stderr:2024-06-03T09:06:59.463+0000 7f185a7fc640 1 -- 172.21.15.88:0/3411910040 <== mon.0 v1:172.21.15.88:6789/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 2234374065 0 0) 0x7f184c001610 con 0x7f185c0a8c10 2024-06-03T09:06:59.465 INFO:teuthology.orchestra.run.smithi088.stderr:2024-06-03T09:06:59.463+0000 7f185a7fc640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-06-03T09:06:59.465 INFO:teuthology.orchestra.run.smithi088.stderr:2024-06-03T09:06:59.463+0000 7f185a7fc640 10 monclient: got monmap 1 from mon.noname-a (according to old e1) 2024-06-03T09:06:59.465 INFO:teuthology.orchestra.run.smithi088.stderr:2024-06-03T09:06:59.463+0000 7f185a7fc640 10 monclient: dump: 2024-06-03T09:06:59.465 INFO:teuthology.orchestra.run.smithi088.stderr:epoch 1 2024-06-03T09:06:59.465 INFO:teuthology.orchestra.run.smithi088.stderr:fsid ec3c3fdd-3ac6-4d7d-9412-683130ed8b04 2024-06-03T09:06:59.465 INFO:teuthology.orchestra.run.smithi088.stderr:last_changed 2024-06-03T08:34:31.885125+0000 2024-06-03T09:06:59.465 INFO:teuthology.orchestra.run.smithi088.stderr:created 2024-06-03T08:34:31.885125+0000 2024-06-03T09:06:59.465 INFO:teuthology.orchestra.run.smithi088.stderr:min_mon_release 18 (reef) 2024-06-03T09:06:59.465 INFO:teuthology.orchestra.run.smithi088.stderr:election_strategy: 3 2024-06-03T09:06:59.465 INFO:teuthology.orchestra.run.smithi088.stderr:0: v1:172.21.15.88:6789/0 mon.a 2024-06-03T09:06:59.466 INFO:teuthology.orchestra.run.smithi088.stderr:1: v1:172.21.15.192:6789/0 mon.b 2024-06-03T09:06:59.466 INFO:teuthology.orchestra.run.smithi088.stderr:2: v1:172.21.15.88:6790/0 mon.c 2024-06-03T09:06:59.466 INFO:teuthology.orchestra.run.smithi088.stderr: 2024-06-03T09:06:59.466 INFO:teuthology.orchestra.run.smithi088.stderr:2024-06-03T09:06:59.463+0000 7f1861ba7640 5 monclient: authenticate success, global_id 30783 2024-06-03T09:06:59.466 INFO:teuthology.orchestra.run.smithi088.stderr:2024-06-03T09:06:59.463+0000 7f1861ba7640 20 monclient: get_monmap_and_config waiting for monmap|config 2024-06-03T09:06:59.466 INFO:teuthology.orchestra.run.smithi088.stderr:2024-06-03T09:06:59.466+0000 7f185a7fc640 1 -- 172.21.15.88:0/3411910040 <== mon.0 v1:172.21.15.88:6789/0 4 ==== config(0 keys) v1 ==== 4+0+0 (unknown 0 0 0) 0x7f184c001830 con 0x7f185c0a8c10 2024-06-03T09:06:59.466 INFO:teuthology.orchestra.run.smithi088.stderr:2024-06-03T09:06:59.466+0000 7f185a7fc640 10 monclient: handle_config config(0 keys) v1 2024-06-03T09:06:59.467 INFO:teuthology.orchestra.run.smithi088.stderr:2024-06-03T09:06:59.466+0000 7f1861ba7640 10 monclient: get_monmap_and_config success 2024-06-03T09:06:59.467 INFO:teuthology.orchestra.run.smithi088.stderr:2024-06-03T09:06:59.466+0000 7f1861ba7640 4 set_mon_vals no callback set 2024-06-03T09:06:59.468 INFO:teuthology.orchestra.run.smithi088.stderr:2024-06-03T09:06:59.467+0000 7f1861ba7640 10 monclient: shutdown 2024-06-03T09:06:59.468 INFO:teuthology.orchestra.run.smithi088.stderr:2024-06-03T09:06:59.467+0000 7f1861ba7640 20 monclient: shutdown discarding 0 pending message(s) 2024-06-03T09:06:59.468 INFO:teuthology.orchestra.run.smithi088.stderr:2024-06-03T09:06:59.467+0000 7f1861ba7640 1 -- 172.21.15.88:0/3411910040 >> v1:172.21.15.88:6789/0 conn(0x7f185c0a8c10 legacy=0x7f185c145210 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-06-03T09:06:59.468 INFO:teuthology.orchestra.run.smithi088.stderr:2024-06-03T09:06:59.467+0000 7f1861ba7640 1 -- 172.21.15.88:0/3411910040 shutdown_connections 2024-06-03T09:06:59.468 INFO:teuthology.orchestra.run.smithi088.stderr:2024-06-03T09:06:59.467+0000 7f1861ba7640 1 -- 172.21.15.88:0/3411910040 >> 172.21.15.88:0/3411910040 conn(0x7f185c0b70d0 msgr2=0x7f185c0b74e0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-06-03T09:06:59.468 INFO:teuthology.orchestra.run.smithi088.stderr:2024-06-03T09:06:59.467+0000 7f1861ba7640 1 -- 172.21.15.88:0/3411910040 shutdown_connections 2024-06-03T09:06:59.468 INFO:teuthology.orchestra.run.smithi088.stderr:2024-06-03T09:06:59.467+0000 7f1861ba7640 1 -- 172.21.15.88:0/3411910040 wait complete. 2024-06-03T09:06:59.468 INFO:teuthology.orchestra.run.smithi088.stderr:2024-06-03T09:06:59.468+0000 7f1861ba7640 10 monclient: build_initial_monmap 2024-06-03T09:06:59.468 INFO:teuthology.orchestra.run.smithi088.stderr:2024-06-03T09:06:59.468+0000 7f1861ba7640 10 monclient: monmap: 2024-06-03T09:06:59.469 INFO:teuthology.orchestra.run.smithi088.stderr:epoch 0 2024-06-03T09:06:59.469 INFO:teuthology.orchestra.run.smithi088.stderr:fsid 00000000-0000-0000-0000-000000000000 2024-06-03T09:06:59.469 INFO:teuthology.orchestra.run.smithi088.stderr:last_changed 0.000000 2024-06-03T09:06:59.469 INFO:teuthology.orchestra.run.smithi088.stderr:created 0.000000 2024-06-03T09:06:59.469 INFO:teuthology.orchestra.run.smithi088.stderr:min_mon_release 0 (unknown) 2024-06-03T09:06:59.469 INFO:teuthology.orchestra.run.smithi088.stderr:election_strategy: 1 2024-06-03T09:06:59.469 INFO:teuthology.orchestra.run.smithi088.stderr:0: v1:172.21.15.88:6789/0 mon.noname-a 2024-06-03T09:06:59.469 INFO:teuthology.orchestra.run.smithi088.stderr:1: v1:172.21.15.192:6789/0 mon.noname-b 2024-06-03T09:06:59.469 INFO:teuthology.orchestra.run.smithi088.stderr:2: v1:172.21.15.88:6790/0 mon.noname-c 2024-06-03T09:06:59.469 INFO:teuthology.orchestra.run.smithi088.stderr: 2024-06-03T09:06:59.469 INFO:teuthology.orchestra.run.smithi088.stderr:2024-06-03T09:06:59.468+0000 7f1861ba7640 1 librados: starting msgr at 2024-06-03T09:06:59.469 INFO:teuthology.orchestra.run.smithi088.stderr:2024-06-03T09:06:59.468+0000 7f1861ba7640 1 librados: starting objecter 2024-06-03T09:06:59.469 INFO:teuthology.orchestra.run.smithi088.stderr:2024-06-03T09:06:59.468+0000 7f1861ba7640 1 Processor -- start 2024-06-03T09:06:59.471 INFO:teuthology.orchestra.run.smithi088.stderr:2024-06-03T09:06:59.468+0000 7f1861ba7640 1 -- start start 2024-06-03T09:06:59.471 INFO:teuthology.orchestra.run.smithi088.stderr:2024-06-03T09:06:59.468+0000 7f1861ba7640 1 librados: setting wanted keys 2024-06-03T09:06:59.471 INFO:teuthology.orchestra.run.smithi088.stderr:2024-06-03T09:06:59.468+0000 7f1861ba7640 1 librados: calling monclient init 2024-06-03T09:06:59.471 INFO:teuthology.orchestra.run.smithi088.stderr:2024-06-03T09:06:59.468+0000 7f1861ba7640 10 monclient: init 2024-06-03T09:06:59.471 INFO:teuthology.orchestra.run.smithi088.stderr:2024-06-03T09:06:59.469+0000 7f1861ba7640 10 monclient: _reopen_session rank -1 2024-06-03T09:06:59.472 INFO:teuthology.orchestra.run.smithi088.stderr:2024-06-03T09:06:59.469+0000 7f1861ba7640 10 monclient: _add_conns ranks=[2,1,0] 2024-06-03T09:06:59.472 INFO:teuthology.orchestra.run.smithi088.stderr:2024-06-03T09:06:59.469+0000 7f1861ba7640 10 monclient(hunting): picked mon.noname-c con 0x7f185c0a8c10 addr v1:172.21.15.88:6790/0 2024-06-03T09:06:59.472 INFO:teuthology.orchestra.run.smithi088.stderr:2024-06-03T09:06:59.469+0000 7f1861ba7640 10 monclient(hunting): picked mon.noname-b con 0x7f185c0b1ad0 addr v1:172.21.15.192:6789/0 2024-06-03T09:06:59.472 INFO:teuthology.orchestra.run.smithi088.stderr:2024-06-03T09:06:59.469+0000 7f1861ba7640 10 monclient(hunting): picked mon.noname-a con 0x7f185c14ae30 addr v1:172.21.15.88:6789/0 2024-06-03T09:06:59.472 INFO:teuthology.orchestra.run.smithi088.stderr:2024-06-03T09:06:59.469+0000 7f1861ba7640 1 -- --> v1:172.21.15.88:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f185c147e90 con 0x7f185c14ae30 2024-06-03T09:06:59.472 INFO:teuthology.orchestra.run.smithi088.stderr:2024-06-03T09:06:59.469+0000 7f1861ba7640 1 -- --> v1:172.21.15.192:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f185c149450 con 0x7f185c0b1ad0 2024-06-03T09:06:59.472 INFO:teuthology.orchestra.run.smithi088.stderr:2024-06-03T09:06:59.469+0000 7f1861ba7640 1 -- --> v1:172.21.15.88:6790/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f185c145b40 con 0x7f185c0a8c10 2024-06-03T09:06:59.472 INFO:teuthology.orchestra.run.smithi088.stderr:2024-06-03T09:06:59.469+0000 7f1861ba7640 10 monclient(hunting): _renew_subs 2024-06-03T09:06:59.472 INFO:teuthology.orchestra.run.smithi088.stderr:2024-06-03T09:06:59.469+0000 7f1861ba7640 10 monclient(hunting): authenticate will time out at 3075.279339s 2024-06-03T09:06:59.472 INFO:teuthology.orchestra.run.smithi088.stderr:2024-06-03T09:06:59.469+0000 7f185b7fe640 1 --1- >> v1:172.21.15.88:6790/0 conn(0x7f185c0a8c10 0x7f185c148d40 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=1).handle_server_banner_and_identify peer v1:172.21.15.88:6790/0 says I am v1:172.21.15.88:51094/0 (socket says 172.21.15.88:51094) 2024-06-03T09:06:59.473 INFO:teuthology.orchestra.run.smithi088.stderr:2024-06-03T09:06:59.469+0000 7f185b7fe640 1 -- 172.21.15.88:0/2163385430 learned_addr learned my addr 172.21.15.88:0/2163385430 (peer_addr_for_me v1:172.21.15.88:0/0) 2024-06-03T09:06:59.473 INFO:teuthology.orchestra.run.smithi088.stderr:2024-06-03T09:06:59.470+0000 7f1843fff640 10 client.?.objecter ms_handle_connect 0x7f185c0a8c10 2024-06-03T09:06:59.473 INFO:teuthology.orchestra.run.smithi088.stderr:2024-06-03T09:06:59.470+0000 7f1843fff640 10 client.?.objecter resend_mon_ops 2024-06-03T09:06:59.473 INFO:teuthology.orchestra.run.smithi088.stderr:2024-06-03T09:06:59.470+0000 7f1843fff640 10 client.?.objecter ms_handle_connect 0x7f185c0b1ad0 2024-06-03T09:06:59.473 INFO:teuthology.orchestra.run.smithi088.stderr:2024-06-03T09:06:59.470+0000 7f1843fff640 10 client.?.objecter resend_mon_ops 2024-06-03T09:06:59.522 INFO:teuthology.orchestra.run.smithi088.stderr:2024-06-03T09:06:59.521+0000 7f1843fff640 10 client.?.objecter ms_handle_connect 0x7f185c14ae30 2024-06-03T09:06:59.523 INFO:teuthology.orchestra.run.smithi088.stderr:2024-06-03T09:06:59.521+0000 7f1843fff640 10 client.?.objecter resend_mon_ops 2024-06-03T09:06:59.524 INFO:teuthology.orchestra.run.smithi088.stderr:2024-06-03T09:06:59.524+0000 7f1843fff640 1 -- 172.21.15.88:0/2163385430 <== mon.0 v1:172.21.15.88:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 305979125 0 0) 0x7f185c147e90 con 0x7f185c14ae30 2024-06-03T09:06:59.525 INFO:teuthology.orchestra.run.smithi088.stderr:2024-06-03T09:06:59.524+0000 7f1843fff640 10 monclient(hunting): _negotiate 2024-06-03T09:06:59.525 INFO:teuthology.orchestra.run.smithi088.stderr:2024-06-03T09:06:59.524+0000 7f1843fff640 10 monclient(hunting): _init_auth method 2 2024-06-03T09:06:59.525 INFO:teuthology.orchestra.run.smithi088.stderr:2024-06-03T09:06:59.524+0000 7f1843fff640 10 monclient(hunting): _init_auth creating new auth 2024-06-03T09:06:59.525 INFO:teuthology.orchestra.run.smithi088.stderr:2024-06-03T09:06:59.524+0000 7f1843fff640 10 monclient(hunting): my global_id is 30786 2024-06-03T09:06:59.525 INFO:teuthology.orchestra.run.smithi088.stderr:2024-06-03T09:06:59.524+0000 7f1843fff640 1 -- 172.21.15.88:0/2163385430 --> v1:172.21.15.88:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f1848004eb0 con 0x7f185c14ae30 2024-06-03T09:06:59.530 INFO:teuthology.orchestra.run.smithi088.stderr:2024-06-03T09:06:59.530+0000 7f1843fff640 1 -- 172.21.15.88:0/2163385430 <== mon.0 v1:172.21.15.88:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 764+0+0 (unknown 881529289 0 0) 0x7f1848004eb0 con 0x7f185c14ae30 2024-06-03T09:06:59.530 INFO:teuthology.orchestra.run.smithi088.stderr:2024-06-03T09:06:59.530+0000 7f1843fff640 1 -- 172.21.15.88:0/2163385430 --> v1:172.21.15.88:6789/0 -- auth(proto 2 165 bytes epoch 0) v1 -- 0x7f185c147e90 con 0x7f185c14ae30 2024-06-03T09:06:59.530 INFO:teuthology.orchestra.run.smithi088.stderr:2024-06-03T09:06:59.530+0000 7f1843fff640 1 -- 172.21.15.88:0/2163385430 <== mon.0 v1:172.21.15.88:6789/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 2234374065 0 0) 0x7f1854001ad0 con 0x7f185c14ae30 2024-06-03T09:06:59.530 INFO:teuthology.orchestra.run.smithi088.stderr:2024-06-03T09:06:59.530+0000 7f1843fff640 10 monclient(hunting): handle_monmap mon_map magic: 0 v1 2024-06-03T09:06:59.530 INFO:teuthology.orchestra.run.smithi088.stderr:2024-06-03T09:06:59.530+0000 7f1843fff640 10 monclient(hunting): got monmap 1 from mon.noname-a (according to old e1) 2024-06-03T09:06:59.530 INFO:teuthology.orchestra.run.smithi088.stderr:2024-06-03T09:06:59.530+0000 7f1843fff640 10 monclient(hunting): dump: 2024-06-03T09:06:59.530 INFO:teuthology.orchestra.run.smithi088.stderr:epoch 1 2024-06-03T09:06:59.530 INFO:teuthology.orchestra.run.smithi088.stderr:fsid ec3c3fdd-3ac6-4d7d-9412-683130ed8b04 2024-06-03T09:06:59.531 INFO:teuthology.orchestra.run.smithi088.stderr:last_changed 2024-06-03T08:34:31.885125+0000 2024-06-03T09:06:59.531 INFO:teuthology.orchestra.run.smithi088.stderr:created 2024-06-03T08:34:31.885125+0000 2024-06-03T09:06:59.531 INFO:teuthology.orchestra.run.smithi088.stderr:min_mon_release 18 (reef) 2024-06-03T09:06:59.531 INFO:teuthology.orchestra.run.smithi088.stderr:election_strategy: 3 2024-06-03T09:06:59.531 INFO:teuthology.orchestra.run.smithi088.stderr:0: v1:172.21.15.88:6789/0 mon.a 2024-06-03T09:06:59.531 INFO:teuthology.orchestra.run.smithi088.stderr:1: v1:172.21.15.192:6789/0 mon.b 2024-06-03T09:06:59.531 INFO:teuthology.orchestra.run.smithi088.stderr:2: v1:172.21.15.88:6790/0 mon.c 2024-06-03T09:06:59.531 INFO:teuthology.orchestra.run.smithi088.stderr: 2024-06-03T09:06:59.531 INFO:teuthology.orchestra.run.smithi088.stderr:2024-06-03T09:06:59.530+0000 7f1843fff640 10 monclient(hunting): _finish_auth 0 2024-06-03T09:06:59.534 INFO:teuthology.orchestra.run.smithi088.stderr:2024-06-03T09:06:59.533+0000 7f1843fff640 1 -- 172.21.15.88:0/2163385430 <== mon.0 v1:172.21.15.88:6789/0 4 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 580+0+0 (unknown 4118029838 0 0) 0x7f185c147e90 con 0x7f185c14ae30 2024-06-03T09:06:59.534 INFO:teuthology.orchestra.run.smithi088.stderr:2024-06-03T09:06:59.533+0000 7f1843fff640 1 -- 172.21.15.88:0/2163385430 >> v1:172.21.15.88:6790/0 conn(0x7f185c0a8c10 legacy=0x7f185c148d40 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-06-03T09:06:59.534 INFO:teuthology.orchestra.run.smithi088.stderr:2024-06-03T09:06:59.533+0000 7f1843fff640 1 -- 172.21.15.88:0/2163385430 >> v1:172.21.15.192:6789/0 conn(0x7f185c0b1ad0 legacy=0x7f185c135400 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-06-03T09:06:59.534 INFO:teuthology.orchestra.run.smithi088.stderr:2024-06-03T09:06:59.533+0000 7f1843fff640 10 monclient: _finish_hunting 0 2024-06-03T09:06:59.534 INFO:teuthology.orchestra.run.smithi088.stderr:2024-06-03T09:06:59.533+0000 7f1843fff640 1 monclient: found mon.a 2024-06-03T09:06:59.534 INFO:teuthology.orchestra.run.smithi088.stderr:2024-06-03T09:06:59.533+0000 7f1843fff640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-06-03T09:06:59.535 INFO:teuthology.orchestra.run.smithi088.stderr:2024-06-03T09:06:59.533+0000 7f1843fff640 10 monclient: _send_mon_message to mon.a at v1:172.21.15.88:6789/0 2024-06-03T09:06:59.535 INFO:teuthology.orchestra.run.smithi088.stderr:2024-06-03T09:06:59.533+0000 7f1843fff640 1 -- 172.21.15.88:0/2163385430 --> v1:172.21.15.88:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f185c145cd0 con 0x7f185c14ae30 2024-06-03T09:06:59.535 INFO:teuthology.orchestra.run.smithi088.stderr:2024-06-03T09:06:59.533+0000 7f1843fff640 10 monclient: _finish_auth 0 2024-06-03T09:06:59.535 INFO:teuthology.orchestra.run.smithi088.stderr:2024-06-03T09:06:59.533+0000 7f1843fff640 10 monclient: _check_auth_tickets 2024-06-03T09:06:59.535 INFO:teuthology.orchestra.run.smithi088.stderr:2024-06-03T09:06:59.533+0000 7f1843fff640 20 monclient: _check_auth_rotating not needed by client.admin 2024-06-03T09:06:59.535 INFO:teuthology.orchestra.run.smithi088.stderr:2024-06-03T09:06:59.533+0000 7f1861ba7640 5 monclient: authenticate success, global_id 30786 2024-06-03T09:06:59.535 INFO:teuthology.orchestra.run.smithi088.stderr:2024-06-03T09:06:59.533+0000 7f1861ba7640 10 monclient: _renew_subs 2024-06-03T09:06:59.535 INFO:teuthology.orchestra.run.smithi088.stderr:2024-06-03T09:06:59.533+0000 7f1861ba7640 10 monclient: _send_mon_message to mon.a at v1:172.21.15.88:6789/0 2024-06-03T09:06:59.535 INFO:teuthology.orchestra.run.smithi088.stderr:2024-06-03T09:06:59.533+0000 7f1861ba7640 1 -- 172.21.15.88:0/2163385430 --> v1:172.21.15.88:6789/0 -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f185c135b10 con 0x7f185c14ae30 2024-06-03T09:06:59.535 INFO:teuthology.orchestra.run.smithi088.stderr:2024-06-03T09:06:59.533+0000 7f1861ba7640 10 client.30786.objecter _maybe_request_map subscribing (onetime) to next osd map 2024-06-03T09:06:59.535 INFO:teuthology.orchestra.run.smithi088.stderr:2024-06-03T09:06:59.533+0000 7f1861ba7640 10 monclient: _renew_subs 2024-06-03T09:06:59.535 INFO:teuthology.orchestra.run.smithi088.stderr:2024-06-03T09:06:59.533+0000 7f1861ba7640 10 monclient: _send_mon_message to mon.a at v1:172.21.15.88:6789/0 2024-06-03T09:06:59.535 INFO:teuthology.orchestra.run.smithi088.stderr:2024-06-03T09:06:59.533+0000 7f1861ba7640 1 -- 172.21.15.88:0/2163385430 --> v1:172.21.15.88:6789/0 -- mon_subscribe({osdmap=0}) v3 -- 0x7f185c148790 con 0x7f185c14ae30 2024-06-03T09:06:59.535 INFO:teuthology.orchestra.run.smithi088.stderr:2024-06-03T09:06:59.533+0000 7f1861ba7640 1 librados: init done 2024-06-03T09:06:59.536 INFO:teuthology.orchestra.run.smithi088.stderr:2024-06-03T09:06:59.534+0000 7f1861ba7640 10 client.30786.objecter _submit_command 1 [{"prefix": "get_command_descriptions"}] 2024-06-03T09:06:59.536 INFO:teuthology.orchestra.run.smithi088.stderr:2024-06-03T09:06:59.534+0000 7f1861ba7640 15 client.30786.objecter _session_command_op_assign -1 1 2024-06-03T09:06:59.536 INFO:teuthology.orchestra.run.smithi088.stderr:2024-06-03T09:06:59.534+0000 7f1861ba7640 20 client.30786.objecter _get_session osd=-1 returning homeless 2024-06-03T09:06:59.536 INFO:teuthology.orchestra.run.smithi088.stderr:2024-06-03T09:06:59.534+0000 7f1861ba7640 10 client.30786.objecter _maybe_request_map subscribing (onetime) to next osd map 2024-06-03T09:06:59.536 INFO:teuthology.orchestra.run.smithi088.stderr:2024-06-03T09:06:59.534+0000 7f1861ba7640 10 monclient: _send_mon_message to mon.a at v1:172.21.15.88:6789/0 2024-06-03T09:06:59.536 INFO:teuthology.orchestra.run.smithi088.stderr:2024-06-03T09:06:59.534+0000 7f1861ba7640 1 -- 172.21.15.88:0/2163385430 --> v1:172.21.15.88:6789/0 -- mon_get_version(what=osdmap handle=1) v1 -- 0x7f185c0aa2e0 con 0x7f185c14ae30 2024-06-03T09:06:59.542 INFO:teuthology.orchestra.run.smithi088.stderr:2024-06-03T09:06:59.542+0000 7f1843fff640 1 -- 172.21.15.88:0/2163385430 <== mon.0 v1:172.21.15.88:6789/0 5 ==== config(0 keys) v1 ==== 4+0+0 (unknown 0 0 0) 0x7f1854002000 con 0x7f185c14ae30 2024-06-03T09:06:59.542 INFO:teuthology.orchestra.run.smithi088.stderr:2024-06-03T09:06:59.542+0000 7f1843fff640 10 monclient: handle_config config(0 keys) v1 2024-06-03T09:06:59.542 INFO:teuthology.orchestra.run.smithi088.stderr:2024-06-03T09:06:59.542+0000 7f1859ffb640 4 set_mon_vals no callback set 2024-06-03T09:06:59.557 INFO:teuthology.orchestra.run.smithi088.stderr:2024-06-03T09:06:59.557+0000 7f1843fff640 1 -- 172.21.15.88:0/2163385430 <== mon.0 v1:172.21.15.88:6789/0 6 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 2234374065 0 0) 0x7f1854004b60 con 0x7f185c14ae30 2024-06-03T09:06:59.557 INFO:teuthology.orchestra.run.smithi088.stderr:2024-06-03T09:06:59.557+0000 7f1843fff640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-06-03T09:06:59.557 INFO:teuthology.orchestra.run.smithi088.stderr:2024-06-03T09:06:59.557+0000 7f1843fff640 10 monclient: got monmap 1 from mon.a (according to old e1) 2024-06-03T09:06:59.558 INFO:teuthology.orchestra.run.smithi088.stderr:2024-06-03T09:06:59.557+0000 7f1843fff640 10 monclient: dump: 2024-06-03T09:06:59.558 INFO:teuthology.orchestra.run.smithi088.stderr:epoch 1 2024-06-03T09:06:59.558 INFO:teuthology.orchestra.run.smithi088.stderr:fsid ec3c3fdd-3ac6-4d7d-9412-683130ed8b04 2024-06-03T09:06:59.558 INFO:teuthology.orchestra.run.smithi088.stderr:last_changed 2024-06-03T08:34:31.885125+0000 2024-06-03T09:06:59.558 INFO:teuthology.orchestra.run.smithi088.stderr:created 2024-06-03T08:34:31.885125+0000 2024-06-03T09:06:59.558 INFO:teuthology.orchestra.run.smithi088.stderr:min_mon_release 18 (reef) 2024-06-03T09:06:59.558 INFO:teuthology.orchestra.run.smithi088.stderr:election_strategy: 3 2024-06-03T09:06:59.558 INFO:teuthology.orchestra.run.smithi088.stderr:0: v1:172.21.15.88:6789/0 mon.a 2024-06-03T09:06:59.558 INFO:teuthology.orchestra.run.smithi088.stderr:1: v1:172.21.15.192:6789/0 mon.b 2024-06-03T09:06:59.558 INFO:teuthology.orchestra.run.smithi088.stderr:2: v1:172.21.15.88:6790/0 mon.c 2024-06-03T09:06:59.558 INFO:teuthology.orchestra.run.smithi088.stderr: 2024-06-03T09:06:59.559 INFO:teuthology.orchestra.run.smithi088.stderr:2024-06-03T09:06:59.558+0000 7f1843fff640 1 -- 172.21.15.88:0/2163385430 <== mon.0 v1:172.21.15.88:6789/0 7 ==== mgrmap(e 5) v1 ==== 89110+0+0 (unknown 873533973 0 0) 0x7f1854004d40 con 0x7f185c14ae30 2024-06-03T09:06:59.559 INFO:teuthology.orchestra.run.smithi088.stderr:2024-06-03T09:06:59.559+0000 7f185b7fe640 10 monclient: get_auth_request con 0x7f184806cff0 auth_method 0 2024-06-03T09:06:59.559 INFO:teuthology.orchestra.run.smithi088.stderr:2024-06-03T09:06:59.559+0000 7f1843fff640 10 client.30786.objecter ms_handle_connect 0x7f184806cff0 2024-06-03T09:06:59.562 INFO:teuthology.orchestra.run.smithi088.stderr:2024-06-03T09:06:59.562+0000 7f1843fff640 1 -- 172.21.15.88:0/2163385430 <== mon.0 v1:172.21.15.88:6789/0 8 ==== osd_map(1014..1014 src has 622..1014) v4 ==== 5229+0+0 (unknown 3230192161 0 0) 0x7f1854054240 con 0x7f185c14ae30 2024-06-03T09:06:59.563 INFO:teuthology.orchestra.run.smithi088.stderr:2024-06-03T09:06:59.562+0000 7f1843fff640 10 client.30786.objecter ms_dispatch 0x7f185c000b90 osd_map(1014..1014 src has 622..1014) v4 2024-06-03T09:06:59.563 INFO:teuthology.orchestra.run.smithi088.stderr:2024-06-03T09:06:59.562+0000 7f1843fff640 3 client.30786.objecter handle_osd_map got epochs [1014,1014] > 0 2024-06-03T09:06:59.563 INFO:teuthology.orchestra.run.smithi088.stderr:2024-06-03T09:06:59.562+0000 7f1843fff640 3 client.30786.objecter handle_osd_map decoding full epoch 1014 2024-06-03T09:06:59.563 INFO:teuthology.orchestra.run.smithi088.stderr:2024-06-03T09:06:59.562+0000 7f1843fff640 10 client.30786.objecter checking command 1 2024-06-03T09:06:59.563 INFO:teuthology.orchestra.run.smithi088.stderr:2024-06-03T09:06:59.562+0000 7f1843fff640 20 client.30786.objecter _get_session s=0x7f1848074fc0 osd=4 3 2024-06-03T09:06:59.563 INFO:teuthology.orchestra.run.smithi088.stderr:2024-06-03T09:06:59.562+0000 7f1843fff640 20 client.30786.objecter put_session s=0x7f1848074fc0 osd=4 3 2024-06-03T09:06:59.563 INFO:teuthology.orchestra.run.smithi088.stderr:2024-06-03T09:06:59.562+0000 7f1843fff640 15 client.30786.objecter _session_command_op_remove -1 1 2024-06-03T09:06:59.563 INFO:teuthology.orchestra.run.smithi088.stderr:2024-06-03T09:06:59.562+0000 7f1843fff640 20 client.30786.objecter _get_session s=0x7f1848074fc0 osd=4 3 2024-06-03T09:06:59.564 INFO:teuthology.orchestra.run.smithi088.stderr:2024-06-03T09:06:59.562+0000 7f1843fff640 20 client.30786.objecter get_session s=0x7f1848074fc0 osd=4 3 2024-06-03T09:06:59.564 INFO:teuthology.orchestra.run.smithi088.stderr:2024-06-03T09:06:59.562+0000 7f1843fff640 15 client.30786.objecter _session_command_op_assign 4 1 2024-06-03T09:06:59.564 INFO:teuthology.orchestra.run.smithi088.stderr:2024-06-03T09:06:59.562+0000 7f1843fff640 20 client.30786.objecter put_session s=0x7f1848074fc0 osd=4 4 2024-06-03T09:06:59.564 INFO:teuthology.orchestra.run.smithi088.stderr:2024-06-03T09:06:59.562+0000 7f1843fff640 10 client.30786.objecter _send_command 1 2024-06-03T09:06:59.564 INFO:teuthology.orchestra.run.smithi088.stderr:2024-06-03T09:06:59.562+0000 7f1843fff640 1 -- 172.21.15.88:0/2163385430 --> v1:172.21.15.192:6809/34785 -- command(tid 1: {"prefix": "get_command_descriptions"}) v1 -- 0x7f1848079380 con 0x7f1848075670 2024-06-03T09:06:59.564 INFO:teuthology.orchestra.run.smithi088.stderr:2024-06-03T09:06:59.562+0000 7f1843fff640 20 client.30786.objecter dump_active .. 0 homeless 2024-06-03T09:06:59.564 INFO:teuthology.orchestra.run.smithi088.stderr:2024-06-03T09:06:59.563+0000 7f185affd640 10 monclient: get_auth_request con 0x7f1848075670 auth_method 0 2024-06-03T09:06:59.565 INFO:teuthology.orchestra.run.smithi088.stderr:2024-06-03T09:06:59.564+0000 7f1843fff640 1 -- 172.21.15.88:0/2163385430 <== mon.0 v1:172.21.15.88:6789/0 9 ==== mon_get_version_reply(handle=1 version=1014) v2 ==== 24+0+0 (unknown 3501658476 0 0) 0x7f185404fdf0 con 0x7f185c14ae30 2024-06-03T09:06:59.565 INFO:teuthology.orchestra.run.smithi088.stderr:2024-06-03T09:06:59.564+0000 7f1843fff640 10 monclient: handle_get_version_reply finishing 1 version 1014 2024-06-03T09:06:59.565 INFO:teuthology.orchestra.run.smithi088.stderr:2024-06-03T09:06:59.564+0000 7f1843fff640 10 client.30786.objecter ms_handle_connect 0x7f1848075670 2024-06-03T09:06:59.587 INFO:teuthology.orchestra.run.smithi088.stderr:2024-06-03T09:06:59.587+0000 7f1843fff640 1 -- 172.21.15.88:0/2163385430 <== osd.4 v1:172.21.15.192:6809/34785 1 ==== command_reply(tid 1: 0 ) v1 ==== 8+0+26118 (unknown 0 0 1686565167) 0x7f1848079380 con 0x7f1848075670 2024-06-03T09:06:59.587 INFO:teuthology.orchestra.run.smithi088.stderr:2024-06-03T09:06:59.587+0000 7f1843fff640 10 client.30786.objecter ms_dispatch 0x7f185c000b90 command_reply(tid 1: 0 ) v1 2024-06-03T09:06:59.587 INFO:teuthology.orchestra.run.smithi088.stderr:2024-06-03T09:06:59.587+0000 7f1843fff640 10 client.30786.objecter _finish_command 1 = system:0 2024-06-03T09:06:59.587 INFO:teuthology.orchestra.run.smithi088.stderr:2024-06-03T09:06:59.587+0000 7f1843fff640 20 client.30786.objecter put_session s=0x7f1848074fc0 osd=4 4 2024-06-03T09:06:59.587 INFO:teuthology.orchestra.run.smithi088.stderr:2024-06-03T09:06:59.587+0000 7f1843fff640 15 client.30786.objecter _session_command_op_remove 4 1 2024-06-03T09:06:59.600 INFO:teuthology.orchestra.run.smithi088.stderr:2024-06-03T09:06:59.599+0000 7f1861ba7640 10 client.30786.objecter _submit_command 2 [{"prefix": "config set", "var": "osd_debug_deep_scrub_sleep", "val": ["0"]}] 2024-06-03T09:06:59.600 INFO:teuthology.orchestra.run.smithi088.stderr:2024-06-03T09:06:59.599+0000 7f1861ba7640 15 client.30786.objecter _session_command_op_assign -1 2 2024-06-03T09:06:59.600 INFO:teuthology.orchestra.run.smithi088.stderr:2024-06-03T09:06:59.599+0000 7f1861ba7640 20 client.30786.objecter _get_session s=0x7f1848074fc0 osd=4 3 2024-06-03T09:06:59.600 INFO:teuthology.orchestra.run.smithi088.stderr:2024-06-03T09:06:59.599+0000 7f1861ba7640 20 client.30786.objecter put_session s=0x7f1848074fc0 osd=4 3 2024-06-03T09:06:59.600 INFO:teuthology.orchestra.run.smithi088.stderr:2024-06-03T09:06:59.599+0000 7f1861ba7640 20 client.30786.objecter _get_session s=0x7f1848074fc0 osd=4 3 2024-06-03T09:06:59.600 INFO:teuthology.orchestra.run.smithi088.stderr:2024-06-03T09:06:59.599+0000 7f1861ba7640 15 client.30786.objecter _session_command_op_remove -1 2 2024-06-03T09:06:59.600 INFO:teuthology.orchestra.run.smithi088.stderr:2024-06-03T09:06:59.599+0000 7f1861ba7640 20 client.30786.objecter get_session s=0x7f1848074fc0 osd=4 3 2024-06-03T09:06:59.600 INFO:teuthology.orchestra.run.smithi088.stderr:2024-06-03T09:06:59.599+0000 7f1861ba7640 15 client.30786.objecter _session_command_op_assign 4 2 2024-06-03T09:06:59.600 INFO:teuthology.orchestra.run.smithi088.stderr:2024-06-03T09:06:59.599+0000 7f1861ba7640 20 client.30786.objecter put_session s=0x7f1848074fc0 osd=4 4 2024-06-03T09:06:59.600 INFO:teuthology.orchestra.run.smithi088.stderr:2024-06-03T09:06:59.599+0000 7f1861ba7640 10 client.30786.objecter _send_command 2 2024-06-03T09:06:59.601 INFO:teuthology.orchestra.run.smithi088.stderr:2024-06-03T09:06:59.599+0000 7f1861ba7640 1 -- 172.21.15.88:0/2163385430 --> v1:172.21.15.192:6809/34785 -- command(tid 2: {"prefix": "config set", "var": "osd_debug_deep_scrub_sleep", "val": ["0"]}) v1 -- 0x7f185c0a8ff0 con 0x7f1848075670 2024-06-03T09:06:59.614 INFO:teuthology.orchestra.run.smithi088.stderr:2024-06-03T09:06:59.615+0000 7f1843fff640 1 -- 172.21.15.88:0/2163385430 <== osd.4 v1:172.21.15.192:6809/34785 2 ==== command_reply(tid 2: 0 ) v1 ==== 8+0+97 (unknown 0 0 119112422) 0x7f1850001720 con 0x7f1848075670 2024-06-03T09:06:59.615 INFO:teuthology.orchestra.run.smithi088.stderr:2024-06-03T09:06:59.615+0000 7f1843fff640 10 client.30786.objecter ms_dispatch 0x7f185c000b90 command_reply(tid 2: 0 ) v1 2024-06-03T09:06:59.615 INFO:teuthology.orchestra.run.smithi088.stderr:2024-06-03T09:06:59.615+0000 7f1843fff640 10 client.30786.objecter _finish_command 2 = system:0 2024-06-03T09:06:59.615 INFO:teuthology.orchestra.run.smithi088.stderr:2024-06-03T09:06:59.615+0000 7f1843fff640 20 client.30786.objecter put_session s=0x7f1848074fc0 osd=4 4 2024-06-03T09:06:59.615 INFO:teuthology.orchestra.run.smithi088.stderr:2024-06-03T09:06:59.615+0000 7f1843fff640 15 client.30786.objecter _session_command_op_remove 4 2 2024-06-03T09:06:59.615 INFO:teuthology.orchestra.run.smithi088.stdout:{ 2024-06-03T09:06:59.615 INFO:teuthology.orchestra.run.smithi088.stdout: "success": "osd_debug_deep_scrub_sleep = '' (not observed, change may require restart) " 2024-06-03T09:06:59.615 INFO:teuthology.orchestra.run.smithi088.stdout:} 2024-06-03T09:06:59.617 INFO:teuthology.orchestra.run.smithi088.stderr:2024-06-03T09:06:59.615+0000 7f1861ba7640 10 librados: watch_flush enter 2024-06-03T09:06:59.617 INFO:teuthology.orchestra.run.smithi088.stderr:2024-06-03T09:06:59.615+0000 7f1861ba7640 10 librados: watch_flush exit 2024-06-03T09:06:59.617 INFO:teuthology.orchestra.run.smithi088.stderr:2024-06-03T09:06:59.615+0000 7f1861ba7640 10 client.30786.objecter close_session for osd.4 2024-06-03T09:06:59.617 INFO:teuthology.orchestra.run.smithi088.stderr:2024-06-03T09:06:59.615+0000 7f1861ba7640 1 -- 172.21.15.88:0/2163385430 >> v1:172.21.15.192:6809/34785 conn(0x7f1848075670 legacy=0x7f1848077b10 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-06-03T09:06:59.617 INFO:teuthology.orchestra.run.smithi088.stderr:2024-06-03T09:06:59.615+0000 7f1861ba7640 20 client.30786.objecter put_session s=0x7f1848074fc0 osd=4 1 2024-06-03T09:06:59.617 INFO:teuthology.orchestra.run.smithi088.stderr:2024-06-03T09:06:59.615+0000 7f1861ba7640 20 client.30786.objecter shutdown clearing up homeless session... 2024-06-03T09:06:59.617 INFO:teuthology.orchestra.run.smithi088.stderr:2024-06-03T09:06:59.615+0000 7f1861ba7640 10 client.30786.objecter successfully canceled tick 2024-06-03T09:06:59.617 INFO:teuthology.orchestra.run.smithi088.stderr:2024-06-03T09:06:59.615+0000 7f1861ba7640 1 -- 172.21.15.88:0/2163385430 >> v1:172.21.15.192:6800/34229 conn(0x7f184806cff0 legacy=0x7f184806f4b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-06-03T09:06:59.617 INFO:teuthology.orchestra.run.smithi088.stderr:2024-06-03T09:06:59.615+0000 7f1861ba7640 10 monclient: shutdown 2024-06-03T09:06:59.617 INFO:teuthology.orchestra.run.smithi088.stderr:2024-06-03T09:06:59.615+0000 7f1861ba7640 20 monclient: shutdown discarding 0 pending message(s) 2024-06-03T09:06:59.617 INFO:teuthology.orchestra.run.smithi088.stderr:2024-06-03T09:06:59.615+0000 7f1861ba7640 1 -- 172.21.15.88:0/2163385430 >> v1:172.21.15.88:6789/0 conn(0x7f185c14ae30 legacy=0x7f185c1d6980 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-06-03T09:06:59.617 INFO:teuthology.orchestra.run.smithi088.stderr:2024-06-03T09:06:59.615+0000 7f185bfff640 1 -- 172.21.15.88:0/2163385430 reap_dead start 2024-06-03T09:06:59.618 INFO:teuthology.orchestra.run.smithi088.stderr:2024-06-03T09:06:59.615+0000 7f1861ba7640 1 -- 172.21.15.88:0/2163385430 shutdown_connections 2024-06-03T09:06:59.618 INFO:teuthology.orchestra.run.smithi088.stderr:2024-06-03T09:06:59.615+0000 7f1861ba7640 1 -- 172.21.15.88:0/2163385430 >> 172.21.15.88:0/2163385430 conn(0x7f185c0b70d0 msgr2=0x7f185c0b74b0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-06-03T09:06:59.618 INFO:teuthology.orchestra.run.smithi088.stderr:2024-06-03T09:06:59.615+0000 7f1861ba7640 1 -- 172.21.15.88:0/2163385430 shutdown_connections 2024-06-03T09:06:59.618 INFO:teuthology.orchestra.run.smithi088.stderr:2024-06-03T09:06:59.615+0000 7f1861ba7640 1 -- 172.21.15.88:0/2163385430 wait complete. 2024-06-03T09:06:59.618 INFO:teuthology.orchestra.run.smithi088.stderr:2024-06-03T09:06:59.615+0000 7f1861ba7640 1 librados: shutdown 2024-06-03T09:06:59.625 DEBUG:teuthology.orchestra.run.smithi088:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd deep-scrub 4 2024-06-03T09:06:59.724 INFO:teuthology.orchestra.run.smithi088.stderr:2024-06-03T09:06:59.724+0000 7ff921d63640 10 monclient: get_monmap_and_config 2024-06-03T09:06:59.724 INFO:teuthology.orchestra.run.smithi088.stderr:2024-06-03T09:06:59.724+0000 7ff921d63640 10 monclient: build_initial_monmap 2024-06-03T09:06:59.725 INFO:teuthology.orchestra.run.smithi088.stderr:2024-06-03T09:06:59.724+0000 7ff921d63640 10 monclient: monmap: 2024-06-03T09:06:59.725 INFO:teuthology.orchestra.run.smithi088.stderr:epoch 0 2024-06-03T09:06:59.725 INFO:teuthology.orchestra.run.smithi088.stderr:fsid ec3c3fdd-3ac6-4d7d-9412-683130ed8b04 2024-06-03T09:06:59.725 INFO:teuthology.orchestra.run.smithi088.stderr:last_changed 2024-06-03T09:06:59.725674+0000 2024-06-03T09:06:59.725 INFO:teuthology.orchestra.run.smithi088.stderr:created 2024-06-03T09:06:59.725674+0000 2024-06-03T09:06:59.725 INFO:teuthology.orchestra.run.smithi088.stderr:min_mon_release 0 (unknown) 2024-06-03T09:06:59.725 INFO:teuthology.orchestra.run.smithi088.stderr:election_strategy: 3 2024-06-03T09:06:59.725 INFO:teuthology.orchestra.run.smithi088.stderr:0: v1:172.21.15.88:6789/0 mon.noname-a 2024-06-03T09:06:59.725 INFO:teuthology.orchestra.run.smithi088.stderr:1: v1:172.21.15.192:6789/0 mon.noname-c 2024-06-03T09:06:59.725 INFO:teuthology.orchestra.run.smithi088.stderr:2: v1:172.21.15.88:6790/0 mon.noname-b 2024-06-03T09:06:59.725 INFO:teuthology.orchestra.run.smithi088.stderr: 2024-06-03T09:06:59.727 INFO:teuthology.orchestra.run.smithi088.stderr:2024-06-03T09:06:59.725+0000 7ff921d63640 1 Processor -- start 2024-06-03T09:06:59.727 INFO:teuthology.orchestra.run.smithi088.stderr:2024-06-03T09:06:59.725+0000 7ff921d63640 1 -- start start 2024-06-03T09:06:59.727 INFO:teuthology.orchestra.run.smithi088.stderr:2024-06-03T09:06:59.725+0000 7ff921d63640 10 monclient: init 2024-06-03T09:06:59.727 INFO:teuthology.orchestra.run.smithi088.stderr:2024-06-03T09:06:59.726+0000 7ff921d63640 10 monclient: _reopen_session rank -1 2024-06-03T09:06:59.727 INFO:teuthology.orchestra.run.smithi088.stderr:2024-06-03T09:06:59.726+0000 7ff921d63640 10 monclient: _add_conns ranks=[1,0,2] 2024-06-03T09:06:59.727 INFO:teuthology.orchestra.run.smithi088.stderr:2024-06-03T09:06:59.726+0000 7ff921d63640 10 monclient(hunting): picked mon.noname-c con 0x7ff91c0adb90 addr v1:172.21.15.192:6789/0 2024-06-03T09:06:59.727 INFO:teuthology.orchestra.run.smithi088.stderr:2024-06-03T09:06:59.726+0000 7ff921d63640 10 monclient(hunting): picked mon.noname-a con 0x7ff91c06ee00 addr v1:172.21.15.88:6789/0 2024-06-03T09:06:59.727 INFO:teuthology.orchestra.run.smithi088.stderr:2024-06-03T09:06:59.726+0000 7ff921d63640 10 monclient(hunting): picked mon.noname-b con 0x7ff91c1383e0 addr v1:172.21.15.88:6790/0 2024-06-03T09:06:59.727 INFO:teuthology.orchestra.run.smithi088.stderr:2024-06-03T09:06:59.726+0000 7ff921d63640 1 -- --> v1:172.21.15.88:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7ff91c06f2c0 con 0x7ff91c06ee00 2024-06-03T09:06:59.727 INFO:teuthology.orchestra.run.smithi088.stderr:2024-06-03T09:06:59.726+0000 7ff921d63640 1 -- --> v1:172.21.15.192:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7ff91c0710c0 con 0x7ff91c0adb90 2024-06-03T09:06:59.727 INFO:teuthology.orchestra.run.smithi088.stderr:2024-06-03T09:06:59.726+0000 7ff921d63640 1 -- --> v1:172.21.15.88:6790/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7ff91c071fd0 con 0x7ff91c1383e0 2024-06-03T09:06:59.728 INFO:teuthology.orchestra.run.smithi088.stderr:2024-06-03T09:06:59.726+0000 7ff921d63640 10 monclient(hunting): _renew_subs 2024-06-03T09:06:59.728 INFO:teuthology.orchestra.run.smithi088.stderr:2024-06-03T09:06:59.726+0000 7ff921d63640 10 monclient(hunting): authenticate will time out at 3075.536216s 2024-06-03T09:06:59.728 INFO:teuthology.orchestra.run.smithi088.stderr:2024-06-03T09:06:59.726+0000 7ff91bfff640 1 --1- >> v1:172.21.15.88:6790/0 conn(0x7ff91c1383e0 0x7ff91c149af0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.88:6790/0 says I am v1:172.21.15.88:51108/0 (socket says 172.21.15.88:51108) 2024-06-03T09:06:59.728 INFO:teuthology.orchestra.run.smithi088.stderr:2024-06-03T09:06:59.726+0000 7ff91bfff640 1 -- 172.21.15.88:0/791202267 learned_addr learned my addr 172.21.15.88:0/791202267 (peer_addr_for_me v1:172.21.15.88:0/0) 2024-06-03T09:06:59.729 INFO:teuthology.orchestra.run.smithi088.stderr:2024-06-03T09:06:59.729+0000 7ff91affd640 1 -- 172.21.15.88:0/791202267 <== mon.2 v1:172.21.15.88:6790/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 4019990126 0 0) 0x7ff91c071fd0 con 0x7ff91c1383e0 2024-06-03T09:06:59.729 INFO:teuthology.orchestra.run.smithi088.stderr:2024-06-03T09:06:59.729+0000 7ff91affd640 10 monclient(hunting): _negotiate 2024-06-03T09:06:59.729 INFO:teuthology.orchestra.run.smithi088.stderr:2024-06-03T09:06:59.729+0000 7ff91affd640 10 monclient(hunting): _init_auth method 2 2024-06-03T09:06:59.729 INFO:teuthology.orchestra.run.smithi088.stderr:2024-06-03T09:06:59.729+0000 7ff91affd640 10 monclient(hunting): _init_auth creating new auth 2024-06-03T09:06:59.729 INFO:teuthology.orchestra.run.smithi088.stderr:2024-06-03T09:06:59.729+0000 7ff91affd640 10 monclient(hunting): my global_id is 30293 2024-06-03T09:06:59.730 INFO:teuthology.orchestra.run.smithi088.stderr:2024-06-03T09:06:59.729+0000 7ff91affd640 1 -- 172.21.15.88:0/791202267 --> v1:172.21.15.88:6790/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7ff908005730 con 0x7ff91c1383e0 2024-06-03T09:06:59.730 INFO:teuthology.orchestra.run.smithi088.stderr:2024-06-03T09:06:59.729+0000 7ff91affd640 1 -- 172.21.15.88:0/791202267 <== mon.0 v1:172.21.15.88:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 2326476640 0 0) 0x7ff91c06f2c0 con 0x7ff91c06ee00 2024-06-03T09:06:59.730 INFO:teuthology.orchestra.run.smithi088.stderr:2024-06-03T09:06:59.729+0000 7ff91affd640 10 monclient(hunting): _negotiate 2024-06-03T09:06:59.730 INFO:teuthology.orchestra.run.smithi088.stderr:2024-06-03T09:06:59.729+0000 7ff91affd640 10 monclient(hunting): _init_auth method 2 2024-06-03T09:06:59.730 INFO:teuthology.orchestra.run.smithi088.stderr:2024-06-03T09:06:59.729+0000 7ff91affd640 10 monclient(hunting): _init_auth creating new auth 2024-06-03T09:06:59.730 INFO:teuthology.orchestra.run.smithi088.stderr:2024-06-03T09:06:59.729+0000 7ff91affd640 10 monclient(hunting): my global_id is 30789 2024-06-03T09:06:59.730 INFO:teuthology.orchestra.run.smithi088.stderr:2024-06-03T09:06:59.729+0000 7ff91affd640 1 -- 172.21.15.88:0/791202267 --> v1:172.21.15.88:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7ff91c071fd0 con 0x7ff91c06ee00 2024-06-03T09:06:59.730 INFO:teuthology.orchestra.run.smithi088.stderr:2024-06-03T09:06:59.729+0000 7ff91affd640 1 -- 172.21.15.88:0/791202267 <== mon.1 v1:172.21.15.192:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 2531793325 0 0) 0x7ff91c0710c0 con 0x7ff91c0adb90 2024-06-03T09:06:59.730 INFO:teuthology.orchestra.run.smithi088.stderr:2024-06-03T09:06:59.729+0000 7ff91affd640 10 monclient(hunting): _negotiate 2024-06-03T09:06:59.730 INFO:teuthology.orchestra.run.smithi088.stderr:2024-06-03T09:06:59.729+0000 7ff91affd640 10 monclient(hunting): _init_auth method 2 2024-06-03T09:06:59.730 INFO:teuthology.orchestra.run.smithi088.stderr:2024-06-03T09:06:59.729+0000 7ff91affd640 10 monclient(hunting): _init_auth creating new auth 2024-06-03T09:06:59.730 INFO:teuthology.orchestra.run.smithi088.stderr:2024-06-03T09:06:59.729+0000 7ff91affd640 10 monclient(hunting): my global_id is 30361 2024-06-03T09:06:59.730 INFO:teuthology.orchestra.run.smithi088.stderr:2024-06-03T09:06:59.729+0000 7ff91affd640 1 -- 172.21.15.88:0/791202267 --> v1:172.21.15.192:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7ff91c06f2c0 con 0x7ff91c0adb90 2024-06-03T09:06:59.732 INFO:teuthology.orchestra.run.smithi088.stderr:2024-06-03T09:06:59.732+0000 7ff91affd640 1 -- 172.21.15.88:0/791202267 <== mon.2 v1:172.21.15.88:6790/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 214+0+0 (unknown 2667537687 0 0) 0x7ff908005730 con 0x7ff91c1383e0 2024-06-03T09:06:59.732 INFO:teuthology.orchestra.run.smithi088.stderr:2024-06-03T09:06:59.732+0000 7ff91affd640 1 -- 172.21.15.88:0/791202267 >> v1:172.21.15.192:6789/0 conn(0x7ff91c0adb90 legacy=0x7ff91c0718c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-06-03T09:06:59.732 INFO:teuthology.orchestra.run.smithi088.stderr:2024-06-03T09:06:59.732+0000 7ff91affd640 1 -- 172.21.15.88:0/791202267 >> v1:172.21.15.88:6789/0 conn(0x7ff91c06ee00 legacy=0x7ff91c137cd0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-06-03T09:06:59.732 INFO:teuthology.orchestra.run.smithi088.stderr:2024-06-03T09:06:59.732+0000 7ff91affd640 10 monclient: _finish_hunting 0 2024-06-03T09:06:59.732 INFO:teuthology.orchestra.run.smithi088.stderr:2024-06-03T09:06:59.732+0000 7ff91affd640 1 monclient: found mon.noname-b 2024-06-03T09:06:59.733 INFO:teuthology.orchestra.run.smithi088.stderr:2024-06-03T09:06:59.732+0000 7ff91affd640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-06-03T09:06:59.733 INFO:teuthology.orchestra.run.smithi088.stderr:2024-06-03T09:06:59.732+0000 7ff91affd640 10 monclient: _send_mon_message to mon.noname-b at v1:172.21.15.88:6790/0 2024-06-03T09:06:59.733 INFO:teuthology.orchestra.run.smithi088.stderr:2024-06-03T09:06:59.732+0000 7ff91affd640 1 -- 172.21.15.88:0/791202267 --> v1:172.21.15.88:6790/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7ff91c141750 con 0x7ff91c1383e0 2024-06-03T09:06:59.733 INFO:teuthology.orchestra.run.smithi088.stderr:2024-06-03T09:06:59.732+0000 7ff91affd640 10 monclient: _finish_auth 0 2024-06-03T09:06:59.733 INFO:teuthology.orchestra.run.smithi088.stderr:2024-06-03T09:06:59.732+0000 7ff91affd640 10 monclient: _check_auth_tickets 2024-06-03T09:06:59.733 INFO:teuthology.orchestra.run.smithi088.stderr:2024-06-03T09:06:59.732+0000 7ff91affd640 20 monclient: _check_auth_rotating not needed by client.admin 2024-06-03T09:06:59.733 INFO:teuthology.orchestra.run.smithi088.stderr:2024-06-03T09:06:59.732+0000 7ff91affd640 1 -- 172.21.15.88:0/791202267 <== mon.2 v1:172.21.15.88:6790/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 2234374065 0 0) 0x7ff914001990 con 0x7ff91c1383e0 2024-06-03T09:06:59.733 INFO:teuthology.orchestra.run.smithi088.stderr:2024-06-03T09:06:59.732+0000 7ff921d63640 5 monclient: authenticate success, global_id 30293 2024-06-03T09:06:59.733 INFO:teuthology.orchestra.run.smithi088.stderr:2024-06-03T09:06:59.732+0000 7ff921d63640 20 monclient: get_monmap_and_config waiting for monmap|config 2024-06-03T09:06:59.733 INFO:teuthology.orchestra.run.smithi088.stderr:2024-06-03T09:06:59.732+0000 7ff91affd640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-06-03T09:06:59.733 INFO:teuthology.orchestra.run.smithi088.stderr:2024-06-03T09:06:59.732+0000 7ff91affd640 10 monclient: got monmap 1 from mon.noname-b (according to old e1) 2024-06-03T09:06:59.733 INFO:teuthology.orchestra.run.smithi088.stderr:2024-06-03T09:06:59.732+0000 7ff91affd640 10 monclient: dump: 2024-06-03T09:06:59.733 INFO:teuthology.orchestra.run.smithi088.stderr:epoch 1 2024-06-03T09:06:59.734 INFO:teuthology.orchestra.run.smithi088.stderr:fsid ec3c3fdd-3ac6-4d7d-9412-683130ed8b04 2024-06-03T09:06:59.734 INFO:teuthology.orchestra.run.smithi088.stderr:last_changed 2024-06-03T08:34:31.885125+0000 2024-06-03T09:06:59.734 INFO:teuthology.orchestra.run.smithi088.stderr:created 2024-06-03T08:34:31.885125+0000 2024-06-03T09:06:59.734 INFO:teuthology.orchestra.run.smithi088.stderr:min_mon_release 18 (reef) 2024-06-03T09:06:59.734 INFO:teuthology.orchestra.run.smithi088.stderr:election_strategy: 3 2024-06-03T09:06:59.734 INFO:teuthology.orchestra.run.smithi088.stderr:0: v1:172.21.15.88:6789/0 mon.a 2024-06-03T09:06:59.734 INFO:teuthology.orchestra.run.smithi088.stderr:1: v1:172.21.15.192:6789/0 mon.b 2024-06-03T09:06:59.734 INFO:teuthology.orchestra.run.smithi088.stderr:2: v1:172.21.15.88:6790/0 mon.c 2024-06-03T09:06:59.734 INFO:teuthology.orchestra.run.smithi088.stderr: 2024-06-03T09:06:59.734 INFO:teuthology.orchestra.run.smithi088.stderr:2024-06-03T09:06:59.732+0000 7ff921d63640 20 monclient: get_monmap_and_config waiting for monmap|config 2024-06-03T09:06:59.740 INFO:teuthology.orchestra.run.smithi088.stderr:2024-06-03T09:06:59.734+0000 7ff91affd640 1 -- 172.21.15.88:0/791202267 <== mon.2 v1:172.21.15.88:6790/0 4 ==== config(0 keys) v1 ==== 4+0+0 (unknown 0 0 0) 0x7ff914001bb0 con 0x7ff91c1383e0 2024-06-03T09:06:59.741 INFO:teuthology.orchestra.run.smithi088.stderr:2024-06-03T09:06:59.734+0000 7ff91affd640 10 monclient: handle_config config(0 keys) v1 2024-06-03T09:06:59.741 INFO:teuthology.orchestra.run.smithi088.stderr:2024-06-03T09:06:59.734+0000 7ff91affd640 1 -- 172.21.15.88:0/791202267 <== mon.2 v1:172.21.15.88:6790/0 5 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 2234374065 0 0) 0x7ff914001f40 con 0x7ff91c1383e0 2024-06-03T09:06:59.741 INFO:teuthology.orchestra.run.smithi088.stderr:2024-06-03T09:06:59.734+0000 7ff91affd640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-06-03T09:06:59.741 INFO:teuthology.orchestra.run.smithi088.stderr:2024-06-03T09:06:59.734+0000 7ff91affd640 10 monclient: got monmap 1 from mon.c (according to old e1) 2024-06-03T09:06:59.741 INFO:teuthology.orchestra.run.smithi088.stderr:2024-06-03T09:06:59.734+0000 7ff91affd640 10 monclient: dump: 2024-06-03T09:06:59.741 INFO:teuthology.orchestra.run.smithi088.stderr:epoch 1 2024-06-03T09:06:59.741 INFO:teuthology.orchestra.run.smithi088.stderr:fsid ec3c3fdd-3ac6-4d7d-9412-683130ed8b04 2024-06-03T09:06:59.741 INFO:teuthology.orchestra.run.smithi088.stderr:last_changed 2024-06-03T08:34:31.885125+0000 2024-06-03T09:06:59.741 INFO:teuthology.orchestra.run.smithi088.stderr:created 2024-06-03T08:34:31.885125+0000 2024-06-03T09:06:59.741 INFO:teuthology.orchestra.run.smithi088.stderr:min_mon_release 18 (reef) 2024-06-03T09:06:59.741 INFO:teuthology.orchestra.run.smithi088.stderr:election_strategy: 3 2024-06-03T09:06:59.741 INFO:teuthology.orchestra.run.smithi088.stderr:0: v1:172.21.15.88:6789/0 mon.a 2024-06-03T09:06:59.742 INFO:teuthology.orchestra.run.smithi088.stderr:1: v1:172.21.15.192:6789/0 mon.b 2024-06-03T09:06:59.742 INFO:teuthology.orchestra.run.smithi088.stderr:2: v1:172.21.15.88:6790/0 mon.c 2024-06-03T09:06:59.742 INFO:teuthology.orchestra.run.smithi088.stderr: 2024-06-03T09:06:59.742 INFO:teuthology.orchestra.run.smithi088.stderr:2024-06-03T09:06:59.734+0000 7ff921d63640 10 monclient: get_monmap_and_config success 2024-06-03T09:06:59.742 INFO:teuthology.orchestra.run.smithi088.stderr:2024-06-03T09:06:59.734+0000 7ff921d63640 4 set_mon_vals no callback set 2024-06-03T09:06:59.742 INFO:teuthology.orchestra.run.smithi088.stderr:2024-06-03T09:06:59.734+0000 7ff921d63640 10 monclient: shutdown 2024-06-03T09:06:59.742 INFO:teuthology.orchestra.run.smithi088.stderr:2024-06-03T09:06:59.734+0000 7ff921d63640 20 monclient: shutdown discarding 0 pending message(s) 2024-06-03T09:06:59.742 INFO:teuthology.orchestra.run.smithi088.stderr:2024-06-03T09:06:59.734+0000 7ff921d63640 1 -- 172.21.15.88:0/791202267 >> v1:172.21.15.88:6790/0 conn(0x7ff91c1383e0 legacy=0x7ff91c149af0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-06-03T09:06:59.742 INFO:teuthology.orchestra.run.smithi088.stderr:2024-06-03T09:06:59.734+0000 7ff921d63640 1 -- 172.21.15.88:0/791202267 shutdown_connections 2024-06-03T09:06:59.742 INFO:teuthology.orchestra.run.smithi088.stderr:2024-06-03T09:06:59.734+0000 7ff921d63640 1 -- 172.21.15.88:0/791202267 >> 172.21.15.88:0/791202267 conn(0x7ff91c0b2f20 msgr2=0x7ff91c0b3300 unknown :-1 s=STATE_NONE l=0).mark_down 2024-06-03T09:06:59.742 INFO:teuthology.orchestra.run.smithi088.stderr:2024-06-03T09:06:59.734+0000 7ff921d63640 1 -- 172.21.15.88:0/791202267 shutdown_connections 2024-06-03T09:06:59.742 INFO:teuthology.orchestra.run.smithi088.stderr:2024-06-03T09:06:59.734+0000 7ff921d63640 1 -- 172.21.15.88:0/791202267 wait complete. 2024-06-03T09:06:59.742 INFO:teuthology.orchestra.run.smithi088.stderr:2024-06-03T09:06:59.735+0000 7ff921d63640 10 monclient: build_initial_monmap 2024-06-03T09:06:59.743 INFO:teuthology.orchestra.run.smithi088.stderr:2024-06-03T09:06:59.735+0000 7ff921d63640 10 monclient: monmap: 2024-06-03T09:06:59.743 INFO:teuthology.orchestra.run.smithi088.stderr:epoch 0 2024-06-03T09:06:59.743 INFO:teuthology.orchestra.run.smithi088.stderr:fsid 00000000-0000-0000-0000-000000000000 2024-06-03T09:06:59.743 INFO:teuthology.orchestra.run.smithi088.stderr:last_changed 0.000000 2024-06-03T09:06:59.743 INFO:teuthology.orchestra.run.smithi088.stderr:created 0.000000 2024-06-03T09:06:59.743 INFO:teuthology.orchestra.run.smithi088.stderr:min_mon_release 0 (unknown) 2024-06-03T09:06:59.743 INFO:teuthology.orchestra.run.smithi088.stderr:election_strategy: 1 2024-06-03T09:06:59.743 INFO:teuthology.orchestra.run.smithi088.stderr:0: v1:172.21.15.88:6789/0 mon.noname-a 2024-06-03T09:06:59.743 INFO:teuthology.orchestra.run.smithi088.stderr:1: v1:172.21.15.192:6789/0 mon.noname-b 2024-06-03T09:06:59.743 INFO:teuthology.orchestra.run.smithi088.stderr:2: v1:172.21.15.88:6790/0 mon.noname-c 2024-06-03T09:06:59.743 INFO:teuthology.orchestra.run.smithi088.stderr: 2024-06-03T09:06:59.743 INFO:teuthology.orchestra.run.smithi088.stderr:2024-06-03T09:06:59.735+0000 7ff921d63640 1 librados: starting msgr at 2024-06-03T09:06:59.743 INFO:teuthology.orchestra.run.smithi088.stderr:2024-06-03T09:06:59.735+0000 7ff921d63640 1 librados: starting objecter 2024-06-03T09:06:59.744 INFO:teuthology.orchestra.run.smithi088.stderr:2024-06-03T09:06:59.735+0000 7ff921d63640 1 Processor -- start 2024-06-03T09:06:59.744 INFO:teuthology.orchestra.run.smithi088.stderr:2024-06-03T09:06:59.736+0000 7ff921d63640 1 -- start start 2024-06-03T09:06:59.744 INFO:teuthology.orchestra.run.smithi088.stderr:2024-06-03T09:06:59.736+0000 7ff921d63640 1 librados: setting wanted keys 2024-06-03T09:06:59.744 INFO:teuthology.orchestra.run.smithi088.stderr:2024-06-03T09:06:59.736+0000 7ff921d63640 1 librados: calling monclient init 2024-06-03T09:06:59.744 INFO:teuthology.orchestra.run.smithi088.stderr:2024-06-03T09:06:59.736+0000 7ff921d63640 10 monclient: init 2024-06-03T09:06:59.744 INFO:teuthology.orchestra.run.smithi088.stderr:2024-06-03T09:06:59.736+0000 7ff921d63640 10 monclient: _reopen_session rank -1 2024-06-03T09:06:59.744 INFO:teuthology.orchestra.run.smithi088.stderr:2024-06-03T09:06:59.736+0000 7ff921d63640 10 monclient: _add_conns ranks=[2,0,1] 2024-06-03T09:06:59.744 INFO:teuthology.orchestra.run.smithi088.stderr:2024-06-03T09:06:59.736+0000 7ff921d63640 10 monclient(hunting): picked mon.noname-c con 0x7ff91c06ee00 addr v1:172.21.15.88:6790/0 2024-06-03T09:06:59.744 INFO:teuthology.orchestra.run.smithi088.stderr:2024-06-03T09:06:59.736+0000 7ff921d63640 10 monclient(hunting): picked mon.noname-a con 0x7ff91c0adb90 addr v1:172.21.15.88:6789/0 2024-06-03T09:06:59.744 INFO:teuthology.orchestra.run.smithi088.stderr:2024-06-03T09:06:59.736+0000 7ff921d63640 10 monclient(hunting): picked mon.noname-b con 0x7ff91c1383e0 addr v1:172.21.15.192:6789/0 2024-06-03T09:06:59.744 INFO:teuthology.orchestra.run.smithi088.stderr:2024-06-03T09:06:59.736+0000 7ff921d63640 1 -- --> v1:172.21.15.88:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7ff91c140890 con 0x7ff91c0adb90 2024-06-03T09:06:59.744 INFO:teuthology.orchestra.run.smithi088.stderr:2024-06-03T09:06:59.736+0000 7ff921d63640 1 -- --> v1:172.21.15.192:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7ff91c13d940 con 0x7ff91c1383e0 2024-06-03T09:06:59.744 INFO:teuthology.orchestra.run.smithi088.stderr:2024-06-03T09:06:59.736+0000 7ff921d63640 1 -- --> v1:172.21.15.88:6790/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7ff91c13e510 con 0x7ff91c06ee00 2024-06-03T09:06:59.744 INFO:teuthology.orchestra.run.smithi088.stderr:2024-06-03T09:06:59.736+0000 7ff921d63640 10 monclient(hunting): _renew_subs 2024-06-03T09:06:59.745 INFO:teuthology.orchestra.run.smithi088.stderr:2024-06-03T09:06:59.736+0000 7ff921d63640 10 monclient(hunting): authenticate will time out at 3075.546607s 2024-06-03T09:06:59.745 INFO:teuthology.orchestra.run.smithi088.stderr:2024-06-03T09:06:59.736+0000 7ff91b7fe640 1 --1- >> v1:172.21.15.88:6790/0 conn(0x7ff91c06ee00 0x7ff91c0ac870 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=1).handle_server_banner_and_identify peer v1:172.21.15.88:6790/0 says I am v1:172.21.15.88:51110/0 (socket says 172.21.15.88:51110) 2024-06-03T09:06:59.745 INFO:teuthology.orchestra.run.smithi088.stderr:2024-06-03T09:06:59.736+0000 7ff91b7fe640 1 -- 172.21.15.88:0/3202044714 learned_addr learned my addr 172.21.15.88:0/3202044714 (peer_addr_for_me v1:172.21.15.88:0/0) 2024-06-03T09:06:59.745 INFO:teuthology.orchestra.run.smithi088.stderr:2024-06-03T09:06:59.737+0000 7ff918ff9640 10 client.?.objecter ms_handle_connect 0x7ff91c06ee00 2024-06-03T09:06:59.745 INFO:teuthology.orchestra.run.smithi088.stderr:2024-06-03T09:06:59.737+0000 7ff918ff9640 10 client.?.objecter resend_mon_ops 2024-06-03T09:06:59.745 INFO:teuthology.orchestra.run.smithi088.stderr:2024-06-03T09:06:59.737+0000 7ff918ff9640 10 client.?.objecter ms_handle_connect 0x7ff91c0adb90 2024-06-03T09:06:59.745 INFO:teuthology.orchestra.run.smithi088.stderr:2024-06-03T09:06:59.737+0000 7ff918ff9640 10 client.?.objecter resend_mon_ops 2024-06-03T09:06:59.745 INFO:teuthology.orchestra.run.smithi088.stderr:2024-06-03T09:06:59.737+0000 7ff918ff9640 10 client.?.objecter ms_handle_connect 0x7ff91c1383e0 2024-06-03T09:06:59.745 INFO:teuthology.orchestra.run.smithi088.stderr:2024-06-03T09:06:59.737+0000 7ff918ff9640 10 client.?.objecter resend_mon_ops 2024-06-03T09:06:59.745 INFO:teuthology.orchestra.run.smithi088.stderr:2024-06-03T09:06:59.739+0000 7ff918ff9640 1 -- 172.21.15.88:0/3202044714 <== mon.2 v1:172.21.15.88:6790/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 2145353769 0 0) 0x7ff91c13e510 con 0x7ff91c06ee00 2024-06-03T09:06:59.745 INFO:teuthology.orchestra.run.smithi088.stderr:2024-06-03T09:06:59.739+0000 7ff918ff9640 10 monclient(hunting): _negotiate 2024-06-03T09:06:59.745 INFO:teuthology.orchestra.run.smithi088.stderr:2024-06-03T09:06:59.739+0000 7ff918ff9640 10 monclient(hunting): _init_auth method 2 2024-06-03T09:06:59.745 INFO:teuthology.orchestra.run.smithi088.stderr:2024-06-03T09:06:59.739+0000 7ff918ff9640 10 monclient(hunting): _init_auth creating new auth 2024-06-03T09:06:59.745 INFO:teuthology.orchestra.run.smithi088.stderr:2024-06-03T09:06:59.739+0000 7ff918ff9640 10 monclient(hunting): my global_id is 30296 2024-06-03T09:06:59.746 INFO:teuthology.orchestra.run.smithi088.stderr:2024-06-03T09:06:59.739+0000 7ff918ff9640 1 -- 172.21.15.88:0/3202044714 --> v1:172.21.15.88:6790/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7ff8fc004eb0 con 0x7ff91c06ee00 2024-06-03T09:06:59.746 INFO:teuthology.orchestra.run.smithi088.stderr:2024-06-03T09:06:59.739+0000 7ff918ff9640 1 -- 172.21.15.88:0/3202044714 <== mon.1 v1:172.21.15.192:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 292221573 0 0) 0x7ff91c13d940 con 0x7ff91c1383e0 2024-06-03T09:06:59.746 INFO:teuthology.orchestra.run.smithi088.stderr:2024-06-03T09:06:59.739+0000 7ff918ff9640 10 monclient(hunting): _negotiate 2024-06-03T09:06:59.746 INFO:teuthology.orchestra.run.smithi088.stderr:2024-06-03T09:06:59.739+0000 7ff918ff9640 10 monclient(hunting): _init_auth method 2 2024-06-03T09:06:59.746 INFO:teuthology.orchestra.run.smithi088.stderr:2024-06-03T09:06:59.739+0000 7ff918ff9640 10 monclient(hunting): _init_auth creating new auth 2024-06-03T09:06:59.746 INFO:teuthology.orchestra.run.smithi088.stderr:2024-06-03T09:06:59.739+0000 7ff918ff9640 10 monclient(hunting): my global_id is 30364 2024-06-03T09:06:59.746 INFO:teuthology.orchestra.run.smithi088.stderr:2024-06-03T09:06:59.739+0000 7ff918ff9640 1 -- 172.21.15.88:0/3202044714 --> v1:172.21.15.192:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7ff91c13e510 con 0x7ff91c1383e0 2024-06-03T09:06:59.746 INFO:teuthology.orchestra.run.smithi088.stderr:2024-06-03T09:06:59.739+0000 7ff918ff9640 1 -- 172.21.15.88:0/3202044714 <== mon.0 v1:172.21.15.88:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 3824532835 0 0) 0x7ff91c140890 con 0x7ff91c0adb90 2024-06-03T09:06:59.746 INFO:teuthology.orchestra.run.smithi088.stderr:2024-06-03T09:06:59.739+0000 7ff918ff9640 10 monclient(hunting): _negotiate 2024-06-03T09:06:59.746 INFO:teuthology.orchestra.run.smithi088.stderr:2024-06-03T09:06:59.739+0000 7ff918ff9640 10 monclient(hunting): _init_auth method 2 2024-06-03T09:06:59.746 INFO:teuthology.orchestra.run.smithi088.stderr:2024-06-03T09:06:59.739+0000 7ff918ff9640 10 monclient(hunting): _init_auth creating new auth 2024-06-03T09:06:59.746 INFO:teuthology.orchestra.run.smithi088.stderr:2024-06-03T09:06:59.739+0000 7ff918ff9640 10 monclient(hunting): my global_id is 30792 2024-06-03T09:06:59.746 INFO:teuthology.orchestra.run.smithi088.stderr:2024-06-03T09:06:59.739+0000 7ff918ff9640 1 -- 172.21.15.88:0/3202044714 --> v1:172.21.15.88:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7ff91c13d940 con 0x7ff91c0adb90 2024-06-03T09:06:59.748 INFO:teuthology.orchestra.run.smithi088.stderr:2024-06-03T09:06:59.744+0000 7ff918ff9640 1 -- 172.21.15.88:0/3202044714 <== mon.1 v1:172.21.15.192:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 764+0+0 (unknown 1636161447 0 0) 0x7ff91c13e510 con 0x7ff91c1383e0 2024-06-03T09:06:59.748 INFO:teuthology.orchestra.run.smithi088.stderr:2024-06-03T09:06:59.744+0000 7ff918ff9640 1 -- 172.21.15.88:0/3202044714 --> v1:172.21.15.192:6789/0 -- auth(proto 2 165 bytes epoch 0) v1 -- 0x7ff91c140890 con 0x7ff91c1383e0 2024-06-03T09:06:59.749 INFO:teuthology.orchestra.run.smithi088.stderr:2024-06-03T09:06:59.744+0000 7ff918ff9640 1 -- 172.21.15.88:0/3202044714 <== mon.1 v1:172.21.15.192:6789/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 2234374065 0 0) 0x7ff914001ec0 con 0x7ff91c1383e0 2024-06-03T09:06:59.749 INFO:teuthology.orchestra.run.smithi088.stderr:2024-06-03T09:06:59.744+0000 7ff918ff9640 10 monclient(hunting): handle_monmap mon_map magic: 0 v1 2024-06-03T09:06:59.749 INFO:teuthology.orchestra.run.smithi088.stderr:2024-06-03T09:06:59.744+0000 7ff918ff9640 10 monclient(hunting): got monmap 1 from mon.noname-b (according to old e1) 2024-06-03T09:06:59.749 INFO:teuthology.orchestra.run.smithi088.stderr:2024-06-03T09:06:59.744+0000 7ff918ff9640 10 monclient(hunting): dump: 2024-06-03T09:06:59.749 INFO:teuthology.orchestra.run.smithi088.stderr:epoch 1 2024-06-03T09:06:59.749 INFO:teuthology.orchestra.run.smithi088.stderr:fsid ec3c3fdd-3ac6-4d7d-9412-683130ed8b04 2024-06-03T09:06:59.749 INFO:teuthology.orchestra.run.smithi088.stderr:last_changed 2024-06-03T08:34:31.885125+0000 2024-06-03T09:06:59.749 INFO:teuthology.orchestra.run.smithi088.stderr:created 2024-06-03T08:34:31.885125+0000 2024-06-03T09:06:59.749 INFO:teuthology.orchestra.run.smithi088.stderr:min_mon_release 18 (reef) 2024-06-03T09:06:59.749 INFO:teuthology.orchestra.run.smithi088.stderr:election_strategy: 3 2024-06-03T09:06:59.749 INFO:teuthology.orchestra.run.smithi088.stderr:0: v1:172.21.15.88:6789/0 mon.a 2024-06-03T09:06:59.749 INFO:teuthology.orchestra.run.smithi088.stderr:1: v1:172.21.15.192:6789/0 mon.b 2024-06-03T09:06:59.750 INFO:teuthology.orchestra.run.smithi088.stderr:2: v1:172.21.15.88:6790/0 mon.c 2024-06-03T09:06:59.750 INFO:teuthology.orchestra.run.smithi088.stderr: 2024-06-03T09:06:59.750 INFO:teuthology.orchestra.run.smithi088.stderr:2024-06-03T09:06:59.744+0000 7ff918ff9640 10 monclient(hunting): _finish_auth 0 2024-06-03T09:06:59.750 INFO:teuthology.orchestra.run.smithi088.stderr:2024-06-03T09:06:59.745+0000 7ff918ff9640 1 -- 172.21.15.88:0/3202044714 <== mon.2 v1:172.21.15.88:6790/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 764+0+0 (unknown 1040943777 0 0) 0x7ff8fc004eb0 con 0x7ff91c06ee00 2024-06-03T09:06:59.750 INFO:teuthology.orchestra.run.smithi088.stderr:2024-06-03T09:06:59.745+0000 7ff918ff9640 1 -- 172.21.15.88:0/3202044714 --> v1:172.21.15.88:6790/0 -- auth(proto 2 165 bytes epoch 0) v1 -- 0x7ff91c13e510 con 0x7ff91c06ee00 2024-06-03T09:06:59.750 INFO:teuthology.orchestra.run.smithi088.stderr:2024-06-03T09:06:59.745+0000 7ff918ff9640 1 -- 172.21.15.88:0/3202044714 <== mon.2 v1:172.21.15.88:6790/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 2234374065 0 0) 0x7ff90c001990 con 0x7ff91c06ee00 2024-06-03T09:06:59.750 INFO:teuthology.orchestra.run.smithi088.stderr:2024-06-03T09:06:59.745+0000 7ff918ff9640 10 monclient(hunting): handle_monmap mon_map magic: 0 v1 2024-06-03T09:06:59.750 INFO:teuthology.orchestra.run.smithi088.stderr:2024-06-03T09:06:59.745+0000 7ff918ff9640 10 monclient(hunting): got monmap 1 from mon.c (according to old e1) 2024-06-03T09:06:59.750 INFO:teuthology.orchestra.run.smithi088.stderr:2024-06-03T09:06:59.745+0000 7ff918ff9640 10 monclient(hunting): dump: 2024-06-03T09:06:59.750 INFO:teuthology.orchestra.run.smithi088.stderr:epoch 1 2024-06-03T09:06:59.750 INFO:teuthology.orchestra.run.smithi088.stderr:fsid ec3c3fdd-3ac6-4d7d-9412-683130ed8b04 2024-06-03T09:06:59.750 INFO:teuthology.orchestra.run.smithi088.stderr:last_changed 2024-06-03T08:34:31.885125+0000 2024-06-03T09:06:59.750 INFO:teuthology.orchestra.run.smithi088.stderr:created 2024-06-03T08:34:31.885125+0000 2024-06-03T09:06:59.750 INFO:teuthology.orchestra.run.smithi088.stderr:min_mon_release 18 (reef) 2024-06-03T09:06:59.751 INFO:teuthology.orchestra.run.smithi088.stderr:election_strategy: 3 2024-06-03T09:06:59.751 INFO:teuthology.orchestra.run.smithi088.stderr:0: v1:172.21.15.88:6789/0 mon.a 2024-06-03T09:06:59.751 INFO:teuthology.orchestra.run.smithi088.stderr:1: v1:172.21.15.192:6789/0 mon.b 2024-06-03T09:06:59.751 INFO:teuthology.orchestra.run.smithi088.stderr:2: v1:172.21.15.88:6790/0 mon.c 2024-06-03T09:06:59.751 INFO:teuthology.orchestra.run.smithi088.stderr: 2024-06-03T09:06:59.751 INFO:teuthology.orchestra.run.smithi088.stderr:2024-06-03T09:06:59.746+0000 7ff918ff9640 1 -- 172.21.15.88:0/3202044714 <== mon.0 v1:172.21.15.88:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 764+0+0 (unknown 1053588460 0 0) 0x7ff91c13d940 con 0x7ff91c0adb90 2024-06-03T09:06:59.751 INFO:teuthology.orchestra.run.smithi088.stderr:2024-06-03T09:06:59.746+0000 7ff918ff9640 1 -- 172.21.15.88:0/3202044714 --> v1:172.21.15.88:6789/0 -- auth(proto 2 165 bytes epoch 0) v1 -- 0x7ff8fc004eb0 con 0x7ff91c0adb90 2024-06-03T09:06:59.751 INFO:teuthology.orchestra.run.smithi088.stderr:2024-06-03T09:06:59.746+0000 7ff918ff9640 1 -- 172.21.15.88:0/3202044714 <== mon.0 v1:172.21.15.88:6789/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 2234374065 0 0) 0x7ff9040018b0 con 0x7ff91c0adb90 2024-06-03T09:06:59.751 INFO:teuthology.orchestra.run.smithi088.stderr:2024-06-03T09:06:59.746+0000 7ff918ff9640 10 monclient(hunting): handle_monmap mon_map magic: 0 v1 2024-06-03T09:06:59.751 INFO:teuthology.orchestra.run.smithi088.stderr:2024-06-03T09:06:59.746+0000 7ff918ff9640 10 monclient(hunting): got monmap 1 from mon.a (according to old e1) 2024-06-03T09:06:59.751 INFO:teuthology.orchestra.run.smithi088.stderr:2024-06-03T09:06:59.746+0000 7ff918ff9640 10 monclient(hunting): dump: 2024-06-03T09:06:59.751 INFO:teuthology.orchestra.run.smithi088.stderr:epoch 1 2024-06-03T09:06:59.751 INFO:teuthology.orchestra.run.smithi088.stderr:fsid ec3c3fdd-3ac6-4d7d-9412-683130ed8b04 2024-06-03T09:06:59.751 INFO:teuthology.orchestra.run.smithi088.stderr:last_changed 2024-06-03T08:34:31.885125+0000 2024-06-03T09:06:59.752 INFO:teuthology.orchestra.run.smithi088.stderr:created 2024-06-03T08:34:31.885125+0000 2024-06-03T09:06:59.752 INFO:teuthology.orchestra.run.smithi088.stderr:min_mon_release 18 (reef) 2024-06-03T09:06:59.752 INFO:teuthology.orchestra.run.smithi088.stderr:election_strategy: 3 2024-06-03T09:06:59.752 INFO:teuthology.orchestra.run.smithi088.stderr:0: v1:172.21.15.88:6789/0 mon.a 2024-06-03T09:06:59.752 INFO:teuthology.orchestra.run.smithi088.stderr:1: v1:172.21.15.192:6789/0 mon.b 2024-06-03T09:06:59.752 INFO:teuthology.orchestra.run.smithi088.stderr:2: v1:172.21.15.88:6790/0 mon.c 2024-06-03T09:06:59.752 INFO:teuthology.orchestra.run.smithi088.stderr: 2024-06-03T09:06:59.752 INFO:teuthology.orchestra.run.smithi088.stderr:2024-06-03T09:06:59.748+0000 7ff918ff9640 1 -- 172.21.15.88:0/3202044714 <== mon.1 v1:172.21.15.192:6789/0 4 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 580+0+0 (unknown 414459558 0 0) 0x7ff91c140890 con 0x7ff91c1383e0 2024-06-03T09:06:59.752 INFO:teuthology.orchestra.run.smithi088.stderr:2024-06-03T09:06:59.748+0000 7ff918ff9640 1 -- 172.21.15.88:0/3202044714 >> v1:172.21.15.88:6790/0 conn(0x7ff91c06ee00 legacy=0x7ff91c0ac870 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-06-03T09:06:59.752 INFO:teuthology.orchestra.run.smithi088.stderr:2024-06-03T09:06:59.748+0000 7ff918ff9640 1 -- 172.21.15.88:0/3202044714 >> v1:172.21.15.88:6789/0 conn(0x7ff91c0adb90 legacy=0x7ff91c13ee10 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-06-03T09:06:59.752 INFO:teuthology.orchestra.run.smithi088.stderr:2024-06-03T09:06:59.748+0000 7ff918ff9640 10 monclient: _finish_hunting 0 2024-06-03T09:06:59.752 INFO:teuthology.orchestra.run.smithi088.stderr:2024-06-03T09:06:59.748+0000 7ff918ff9640 1 monclient: found mon.b 2024-06-03T09:06:59.752 INFO:teuthology.orchestra.run.smithi088.stderr:2024-06-03T09:06:59.748+0000 7ff918ff9640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-06-03T09:06:59.753 INFO:teuthology.orchestra.run.smithi088.stderr:2024-06-03T09:06:59.748+0000 7ff918ff9640 10 monclient: _send_mon_message to mon.b at v1:172.21.15.192:6789/0 2024-06-03T09:06:59.753 INFO:teuthology.orchestra.run.smithi088.stderr:2024-06-03T09:06:59.748+0000 7ff918ff9640 1 -- 172.21.15.88:0/3202044714 --> v1:172.21.15.192:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7ff91c13e6a0 con 0x7ff91c1383e0 2024-06-03T09:06:59.753 INFO:teuthology.orchestra.run.smithi088.stderr:2024-06-03T09:06:59.748+0000 7ff918ff9640 10 monclient: _finish_auth 0 2024-06-03T09:06:59.753 INFO:teuthology.orchestra.run.smithi088.stderr:2024-06-03T09:06:59.748+0000 7ff918ff9640 10 monclient: _check_auth_tickets 2024-06-03T09:06:59.753 INFO:teuthology.orchestra.run.smithi088.stderr:2024-06-03T09:06:59.748+0000 7ff918ff9640 20 monclient: _check_auth_rotating not needed by client.admin 2024-06-03T09:06:59.753 INFO:teuthology.orchestra.run.smithi088.stderr:2024-06-03T09:06:59.748+0000 7ff921d63640 5 monclient: authenticate success, global_id 30364 2024-06-03T09:06:59.753 INFO:teuthology.orchestra.run.smithi088.stderr:2024-06-03T09:06:59.748+0000 7ff921d63640 10 monclient: _renew_subs 2024-06-03T09:06:59.753 INFO:teuthology.orchestra.run.smithi088.stderr:2024-06-03T09:06:59.748+0000 7ff921d63640 10 monclient: _send_mon_message to mon.b at v1:172.21.15.192:6789/0 2024-06-03T09:06:59.753 INFO:teuthology.orchestra.run.smithi088.stderr:2024-06-03T09:06:59.748+0000 7ff921d63640 1 -- 172.21.15.88:0/3202044714 --> v1:172.21.15.192:6789/0 -- mon_subscribe({mgrmap=0+}) v3 -- 0x7ff91c1d62f0 con 0x7ff91c1383e0 2024-06-03T09:06:59.753 INFO:teuthology.orchestra.run.smithi088.stderr:2024-06-03T09:06:59.748+0000 7ff921d63640 10 client.30364.objecter _maybe_request_map subscribing (onetime) to next osd map 2024-06-03T09:06:59.753 INFO:teuthology.orchestra.run.smithi088.stderr:2024-06-03T09:06:59.748+0000 7ff921d63640 10 monclient: _renew_subs 2024-06-03T09:06:59.753 INFO:teuthology.orchestra.run.smithi088.stderr:2024-06-03T09:06:59.748+0000 7ff921d63640 10 monclient: _send_mon_message to mon.b at v1:172.21.15.192:6789/0 2024-06-03T09:06:59.753 INFO:teuthology.orchestra.run.smithi088.stderr:2024-06-03T09:06:59.748+0000 7ff921d63640 1 -- 172.21.15.88:0/3202044714 --> v1:172.21.15.192:6789/0 -- mon_subscribe({osdmap=0}) v3 -- 0x7ff91c1d58d0 con 0x7ff91c1383e0 2024-06-03T09:06:59.753 INFO:teuthology.orchestra.run.smithi088.stderr:2024-06-03T09:06:59.748+0000 7ff921d63640 1 librados: init done 2024-06-03T09:06:59.754 INFO:teuthology.orchestra.run.smithi088.stderr:2024-06-03T09:06:59.749+0000 7ff921d63640 10 start_mon_command cmd=[{"prefix": "get_command_descriptions"}] 2024-06-03T09:06:59.754 INFO:teuthology.orchestra.run.smithi088.stderr:2024-06-03T09:06:59.749+0000 7ff921d63640 10 monclient: _send_command 1 [{"prefix": "get_command_descriptions"}] 2024-06-03T09:06:59.754 INFO:teuthology.orchestra.run.smithi088.stderr:2024-06-03T09:06:59.749+0000 7ff921d63640 10 monclient: _send_mon_message to mon.b at v1:172.21.15.192:6789/0 2024-06-03T09:06:59.754 INFO:teuthology.orchestra.run.smithi088.stderr:2024-06-03T09:06:59.749+0000 7ff921d63640 1 -- 172.21.15.88:0/3202044714 --> v1:172.21.15.192:6789/0 -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7ff91c13ec60 con 0x7ff91c1383e0 2024-06-03T09:07:00.005 INFO:teuthology.orchestra.run.smithi088.stderr:2024-06-03T09:07:00.004+0000 7ff918ff9640 1 -- 172.21.15.88:0/3202044714 <== mon.1 v1:172.21.15.192:6789/0 5 ==== config(0 keys) v1 ==== 4+0+0 (unknown 0 0 0) 0x7ff9140020e0 con 0x7ff91c1383e0 2024-06-03T09:07:00.005 INFO:teuthology.orchestra.run.smithi088.stderr:2024-06-03T09:07:00.004+0000 7ff918ff9640 10 monclient: handle_config config(0 keys) v1 2024-06-03T09:07:00.005 INFO:teuthology.orchestra.run.smithi088.stderr:2024-06-03T09:07:00.004+0000 7ff918ff9640 1 -- 172.21.15.88:0/3202044714 <== mon.1 v1:172.21.15.192:6789/0 6 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 2234374065 0 0) 0x7ff914004e10 con 0x7ff91c1383e0 2024-06-03T09:07:00.005 INFO:teuthology.orchestra.run.smithi088.stderr:2024-06-03T09:07:00.004+0000 7ff918ff9640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-06-03T09:07:00.005 INFO:teuthology.orchestra.run.smithi088.stderr:2024-06-03T09:07:00.004+0000 7ff918ff9640 10 monclient: got monmap 1 from mon.b (according to old e1) 2024-06-03T09:07:00.005 INFO:teuthology.orchestra.run.smithi088.stderr:2024-06-03T09:07:00.004+0000 7ff918ff9640 10 monclient: dump: 2024-06-03T09:07:00.005 INFO:teuthology.orchestra.run.smithi088.stderr:epoch 1 2024-06-03T09:07:00.005 INFO:teuthology.orchestra.run.smithi088.stderr:fsid ec3c3fdd-3ac6-4d7d-9412-683130ed8b04 2024-06-03T09:07:00.005 INFO:teuthology.orchestra.run.smithi088.stderr:last_changed 2024-06-03T08:34:31.885125+0000 2024-06-03T09:07:00.005 INFO:teuthology.orchestra.run.smithi088.stderr:created 2024-06-03T08:34:31.885125+0000 2024-06-03T09:07:00.005 INFO:teuthology.orchestra.run.smithi088.stderr:min_mon_release 18 (reef) 2024-06-03T09:07:00.006 INFO:teuthology.orchestra.run.smithi088.stderr:election_strategy: 3 2024-06-03T09:07:00.006 INFO:teuthology.orchestra.run.smithi088.stderr:0: v1:172.21.15.88:6789/0 mon.a 2024-06-03T09:07:00.006 INFO:teuthology.orchestra.run.smithi088.stderr:1: v1:172.21.15.192:6789/0 mon.b 2024-06-03T09:07:00.006 INFO:teuthology.orchestra.run.smithi088.stderr:2: v1:172.21.15.88:6790/0 mon.c 2024-06-03T09:07:00.006 INFO:teuthology.orchestra.run.smithi088.stderr: 2024-06-03T09:07:00.006 INFO:teuthology.orchestra.run.smithi088.stderr:2024-06-03T09:07:00.004+0000 7ff91a7fc640 4 set_mon_vals no callback set 2024-06-03T09:07:00.007 INFO:teuthology.orchestra.run.smithi088.stderr:2024-06-03T09:07:00.005+0000 7ff918ff9640 1 -- 172.21.15.88:0/3202044714 <== mon.1 v1:172.21.15.192:6789/0 7 ==== mgrmap(e 5) v1 ==== 89110+0+0 (unknown 873533973 0 0) 0x7ff91401ab90 con 0x7ff91c1383e0 2024-06-03T09:07:00.007 INFO:teuthology.orchestra.run.smithi088.stderr:2024-06-03T09:07:00.006+0000 7ff91b7fe640 10 monclient: get_auth_request con 0x7ff8fc06d450 auth_method 0 2024-06-03T09:07:00.007 INFO:teuthology.orchestra.run.smithi088.stderr:2024-06-03T09:07:00.006+0000 7ff918ff9640 10 client.30364.objecter ms_handle_connect 0x7ff8fc06d450 2024-06-03T09:07:00.008 INFO:teuthology.orchestra.run.smithi088.stderr:2024-06-03T09:07:00.008+0000 7ff918ff9640 1 -- 172.21.15.88:0/3202044714 <== mon.1 v1:172.21.15.192:6789/0 8 ==== osd_map(1014..1014 src has 622..1014) v4 ==== 5229+0+0 (unknown 3230192161 0 0) 0x7ff914054320 con 0x7ff91c1383e0 2024-06-03T09:07:00.008 INFO:teuthology.orchestra.run.smithi088.stderr:2024-06-03T09:07:00.008+0000 7ff918ff9640 10 client.30364.objecter ms_dispatch 0x7ff91c000b90 osd_map(1014..1014 src has 622..1014) v4 2024-06-03T09:07:00.008 INFO:teuthology.orchestra.run.smithi088.stderr:2024-06-03T09:07:00.008+0000 7ff918ff9640 3 client.30364.objecter handle_osd_map got epochs [1014,1014] > 0 2024-06-03T09:07:00.009 INFO:teuthology.orchestra.run.smithi088.stderr:2024-06-03T09:07:00.008+0000 7ff918ff9640 3 client.30364.objecter handle_osd_map decoding full epoch 1014 2024-06-03T09:07:00.009 INFO:teuthology.orchestra.run.smithi088.stderr:2024-06-03T09:07:00.008+0000 7ff918ff9640 20 client.30364.objecter dump_active .. 0 homeless 2024-06-03T09:07:00.010 INFO:teuthology.orchestra.run.smithi088.stderr:2024-06-03T09:07:00.010+0000 7ff918ff9640 1 -- 172.21.15.88:0/3202044714 <== mon.1 v1:172.21.15.192:6789/0 9 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+182498 (unknown 1092875540 0 3016760700) 0x7ff91404fd00 con 0x7ff91c1383e0 2024-06-03T09:07:00.010 INFO:teuthology.orchestra.run.smithi088.stderr:2024-06-03T09:07:00.010+0000 7ff918ff9640 10 monclient: handle_mon_command_ack 1 [{"prefix": "get_command_descriptions"}] 2024-06-03T09:07:00.010 INFO:teuthology.orchestra.run.smithi088.stderr:2024-06-03T09:07:00.010+0000 7ff918ff9640 10 monclient: _finish_command 1 = system:0 2024-06-03T09:07:00.183 INFO:teuthology.orchestra.run.smithi088.stderr:2024-06-03T09:07:00.183+0000 7ff921d63640 1 -- 172.21.15.88:0/3202044714 --> v1:172.21.15.192:6800/34229 -- mgr_command(tid 0: {"prefix": "osd deep-scrub", "who": "4", "target": ["mon-mgr", ""]}) v1 -- 0x7ff91c0adf70 con 0x7ff8fc06d450 2024-06-03T09:07:00.183 INFO:teuthology.orchestra.run.smithi088.stderr:2024-06-03T09:07:00.183+0000 7ff918ff9640 1 -- 172.21.15.88:0/3202044714 <== mgr.4110 v1:172.21.15.192:6800/34229 1 ==== mgr_command_reply(tid 0: 0 instructed osd(s) 4 to deep-scrub) v1 ==== 41+0+0 (unknown 3782719169 0 0) 0x7ff90c005850 con 0x7ff8fc06d450 2024-06-03T09:07:00.183 INFO:teuthology.orchestra.run.smithi088.stderr:instructed osd(s) 4 to deep-scrub 2024-06-03T09:07:00.186 INFO:teuthology.orchestra.run.smithi088.stderr:2024-06-03T09:07:00.185+0000 7ff921d63640 10 librados: watch_flush enter 2024-06-03T09:07:00.186 INFO:teuthology.orchestra.run.smithi088.stderr:2024-06-03T09:07:00.185+0000 7ff921d63640 10 librados: watch_flush exit 2024-06-03T09:07:00.186 INFO:teuthology.orchestra.run.smithi088.stderr:2024-06-03T09:07:00.185+0000 7ff921d63640 20 client.30364.objecter shutdown clearing up homeless session... 2024-06-03T09:07:00.186 INFO:teuthology.orchestra.run.smithi088.stderr:2024-06-03T09:07:00.185+0000 7ff921d63640 10 client.30364.objecter successfully canceled tick 2024-06-03T09:07:00.186 INFO:teuthology.orchestra.run.smithi088.stderr:2024-06-03T09:07:00.185+0000 7ff921d63640 1 -- 172.21.15.88:0/3202044714 >> v1:172.21.15.192:6800/34229 conn(0x7ff8fc06d450 legacy=0x7ff8fc06f910 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-06-03T09:07:00.186 INFO:teuthology.orchestra.run.smithi088.stderr:2024-06-03T09:07:00.185+0000 7ff921d63640 10 monclient: shutdown 2024-06-03T09:07:00.186 INFO:teuthology.orchestra.run.smithi088.stderr:2024-06-03T09:07:00.185+0000 7ff921d63640 20 monclient: shutdown discarding 0 pending message(s) 2024-06-03T09:07:00.186 INFO:teuthology.orchestra.run.smithi088.stderr:2024-06-03T09:07:00.185+0000 7ff921d63640 1 -- 172.21.15.88:0/3202044714 >> v1:172.21.15.192:6789/0 conn(0x7ff91c1383e0 legacy=0x7ff91c1dda00 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-06-03T09:07:00.186 INFO:teuthology.orchestra.run.smithi088.stderr:2024-06-03T09:07:00.185+0000 7ff921d63640 1 -- 172.21.15.88:0/3202044714 shutdown_connections 2024-06-03T09:07:00.186 INFO:teuthology.orchestra.run.smithi088.stderr:2024-06-03T09:07:00.185+0000 7ff921d63640 1 -- 172.21.15.88:0/3202044714 >> 172.21.15.88:0/3202044714 conn(0x7ff91c0b2f20 msgr2=0x7ff91c0aea20 unknown :-1 s=STATE_NONE l=0).mark_down 2024-06-03T09:07:00.187 INFO:teuthology.orchestra.run.smithi088.stderr:2024-06-03T09:07:00.185+0000 7ff921d63640 1 -- 172.21.15.88:0/3202044714 shutdown_connections 2024-06-03T09:07:00.187 INFO:teuthology.orchestra.run.smithi088.stderr:2024-06-03T09:07:00.185+0000 7ff921d63640 1 -- 172.21.15.88:0/3202044714 wait complete. 2024-06-03T09:07:00.187 INFO:teuthology.orchestra.run.smithi088.stderr:2024-06-03T09:07:00.185+0000 7ff921d63640 1 librados: shutdown 2024-06-03T09:07:00.196 INFO:tasks.ceph:Scrubbing osd.5 2024-06-03T09:07:00.196 DEBUG:teuthology.orchestra.run.smithi088:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph tell osd.5 config set osd_debug_deep_scrub_sleep 0 2024-06-03T09:07:00.301 INFO:teuthology.orchestra.run.smithi088.stderr:2024-06-03T09:07:00.300+0000 7f4f7da3b640 10 monclient: get_monmap_and_config 2024-06-03T09:07:00.301 INFO:teuthology.orchestra.run.smithi088.stderr:2024-06-03T09:07:00.300+0000 7f4f7da3b640 10 monclient: build_initial_monmap 2024-06-03T09:07:00.301 INFO:teuthology.orchestra.run.smithi088.stderr:2024-06-03T09:07:00.300+0000 7f4f7da3b640 10 monclient: monmap: 2024-06-03T09:07:00.301 INFO:teuthology.orchestra.run.smithi088.stderr:epoch 0 2024-06-03T09:07:00.301 INFO:teuthology.orchestra.run.smithi088.stderr:fsid ec3c3fdd-3ac6-4d7d-9412-683130ed8b04 2024-06-03T09:07:00.301 INFO:teuthology.orchestra.run.smithi088.stderr:last_changed 2024-06-03T09:07:00.301992+0000 2024-06-03T09:07:00.301 INFO:teuthology.orchestra.run.smithi088.stderr:created 2024-06-03T09:07:00.301992+0000 2024-06-03T09:07:00.301 INFO:teuthology.orchestra.run.smithi088.stderr:min_mon_release 0 (unknown) 2024-06-03T09:07:00.301 INFO:teuthology.orchestra.run.smithi088.stderr:election_strategy: 3 2024-06-03T09:07:00.301 INFO:teuthology.orchestra.run.smithi088.stderr:0: v1:172.21.15.88:6789/0 mon.noname-a 2024-06-03T09:07:00.302 INFO:teuthology.orchestra.run.smithi088.stderr:1: v1:172.21.15.192:6789/0 mon.noname-c 2024-06-03T09:07:00.302 INFO:teuthology.orchestra.run.smithi088.stderr:2: v1:172.21.15.88:6790/0 mon.noname-b 2024-06-03T09:07:00.302 INFO:teuthology.orchestra.run.smithi088.stderr: 2024-06-03T09:07:00.303 INFO:teuthology.orchestra.run.smithi088.stderr:2024-06-03T09:07:00.301+0000 7f4f7da3b640 1 Processor -- start 2024-06-03T09:07:00.304 INFO:teuthology.orchestra.run.smithi088.stderr:2024-06-03T09:07:00.301+0000 7f4f7da3b640 1 -- start start 2024-06-03T09:07:00.304 INFO:teuthology.orchestra.run.smithi088.stderr:2024-06-03T09:07:00.301+0000 7f4f7da3b640 10 monclient: init 2024-06-03T09:07:00.304 INFO:teuthology.orchestra.run.smithi088.stderr:2024-06-03T09:07:00.302+0000 7f4f7da3b640 10 monclient: _reopen_session rank -1 2024-06-03T09:07:00.304 INFO:teuthology.orchestra.run.smithi088.stderr:2024-06-03T09:07:00.302+0000 7f4f7da3b640 10 monclient: _add_conns ranks=[0,2,1] 2024-06-03T09:07:00.304 INFO:teuthology.orchestra.run.smithi088.stderr:2024-06-03T09:07:00.302+0000 7f4f7da3b640 10 monclient(hunting): picked mon.noname-a con 0x7f4f7806e050 addr v1:172.21.15.88:6789/0 2024-06-03T09:07:00.304 INFO:teuthology.orchestra.run.smithi088.stderr:2024-06-03T09:07:00.302+0000 7f4f7da3b640 10 monclient(hunting): picked mon.noname-b con 0x7f4f7806e430 addr v1:172.21.15.88:6790/0 2024-06-03T09:07:00.304 INFO:teuthology.orchestra.run.smithi088.stderr:2024-06-03T09:07:00.302+0000 7f4f7da3b640 10 monclient(hunting): picked mon.noname-c con 0x7f4f780717c0 addr v1:172.21.15.192:6789/0 2024-06-03T09:07:00.304 INFO:teuthology.orchestra.run.smithi088.stderr:2024-06-03T09:07:00.302+0000 7f4f7da3b640 1 -- --> v1:172.21.15.88:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f4f78070c10 con 0x7f4f7806e050 2024-06-03T09:07:00.304 INFO:teuthology.orchestra.run.smithi088.stderr:2024-06-03T09:07:00.302+0000 7f4f7da3b640 1 -- --> v1:172.21.15.192:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f4f78070230 con 0x7f4f780717c0 2024-06-03T09:07:00.304 INFO:teuthology.orchestra.run.smithi088.stderr:2024-06-03T09:07:00.302+0000 7f4f7da3b640 1 -- --> v1:172.21.15.88:6790/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f4f78070510 con 0x7f4f7806e430 2024-06-03T09:07:00.304 INFO:teuthology.orchestra.run.smithi088.stderr:2024-06-03T09:07:00.302+0000 7f4f7da3b640 10 monclient(hunting): _renew_subs 2024-06-03T09:07:00.305 INFO:teuthology.orchestra.run.smithi088.stderr:2024-06-03T09:07:00.302+0000 7f4f7da3b640 10 monclient(hunting): authenticate will time out at 3076.112254s 2024-06-03T09:07:00.305 INFO:teuthology.orchestra.run.smithi088.stderr:2024-06-03T09:07:00.302+0000 7f4f767fc640 1 --1- >> v1:172.21.15.88:6790/0 conn(0x7f4f7806e430 0x7f4f780710b0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.88:6790/0 says I am v1:172.21.15.88:51114/0 (socket says 172.21.15.88:51114) 2024-06-03T09:07:00.305 INFO:teuthology.orchestra.run.smithi088.stderr:2024-06-03T09:07:00.302+0000 7f4f767fc640 1 -- 172.21.15.88:0/3771859321 learned_addr learned my addr 172.21.15.88:0/3771859321 (peer_addr_for_me v1:172.21.15.88:0/0) 2024-06-03T09:07:00.306 INFO:teuthology.orchestra.run.smithi088.stderr:2024-06-03T09:07:00.305+0000 7f4f75ffb640 1 -- 172.21.15.88:0/3771859321 <== mon.2 v1:172.21.15.88:6790/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 3825689353 0 0) 0x7f4f78070510 con 0x7f4f7806e430 2024-06-03T09:07:00.306 INFO:teuthology.orchestra.run.smithi088.stderr:2024-06-03T09:07:00.305+0000 7f4f75ffb640 10 monclient(hunting): _negotiate 2024-06-03T09:07:00.306 INFO:teuthology.orchestra.run.smithi088.stderr:2024-06-03T09:07:00.305+0000 7f4f75ffb640 10 monclient(hunting): _init_auth method 2 2024-06-03T09:07:00.306 INFO:teuthology.orchestra.run.smithi088.stderr:2024-06-03T09:07:00.305+0000 7f4f75ffb640 10 monclient(hunting): _init_auth creating new auth 2024-06-03T09:07:00.306 INFO:teuthology.orchestra.run.smithi088.stderr:2024-06-03T09:07:00.305+0000 7f4f75ffb640 10 monclient(hunting): my global_id is 30299 2024-06-03T09:07:00.306 INFO:teuthology.orchestra.run.smithi088.stderr:2024-06-03T09:07:00.305+0000 7f4f75ffb640 1 -- 172.21.15.88:0/3771859321 --> v1:172.21.15.88:6790/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f4f60004630 con 0x7f4f7806e430 2024-06-03T09:07:00.306 INFO:teuthology.orchestra.run.smithi088.stderr:2024-06-03T09:07:00.305+0000 7f4f75ffb640 1 -- 172.21.15.88:0/3771859321 <== mon.0 v1:172.21.15.88:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 2187031873 0 0) 0x7f4f78070c10 con 0x7f4f7806e050 2024-06-03T09:07:00.307 INFO:teuthology.orchestra.run.smithi088.stderr:2024-06-03T09:07:00.305+0000 7f4f75ffb640 10 monclient(hunting): _negotiate 2024-06-03T09:07:00.307 INFO:teuthology.orchestra.run.smithi088.stderr:2024-06-03T09:07:00.305+0000 7f4f75ffb640 10 monclient(hunting): _init_auth method 2 2024-06-03T09:07:00.307 INFO:teuthology.orchestra.run.smithi088.stderr:2024-06-03T09:07:00.305+0000 7f4f75ffb640 10 monclient(hunting): _init_auth creating new auth 2024-06-03T09:07:00.307 INFO:teuthology.orchestra.run.smithi088.stderr:2024-06-03T09:07:00.305+0000 7f4f75ffb640 10 monclient(hunting): my global_id is 30795 2024-06-03T09:07:00.307 INFO:teuthology.orchestra.run.smithi088.stderr:2024-06-03T09:07:00.305+0000 7f4f75ffb640 1 -- 172.21.15.88:0/3771859321 --> v1:172.21.15.88:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f4f78070510 con 0x7f4f7806e050 2024-06-03T09:07:00.307 INFO:teuthology.orchestra.run.smithi088.stderr:2024-06-03T09:07:00.305+0000 7f4f75ffb640 1 -- 172.21.15.88:0/3771859321 <== mon.1 v1:172.21.15.192:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 1173493887 0 0) 0x7f4f78070230 con 0x7f4f780717c0 2024-06-03T09:07:00.307 INFO:teuthology.orchestra.run.smithi088.stderr:2024-06-03T09:07:00.305+0000 7f4f75ffb640 10 monclient(hunting): _negotiate 2024-06-03T09:07:00.307 INFO:teuthology.orchestra.run.smithi088.stderr:2024-06-03T09:07:00.305+0000 7f4f75ffb640 10 monclient(hunting): _init_auth method 2 2024-06-03T09:07:00.307 INFO:teuthology.orchestra.run.smithi088.stderr:2024-06-03T09:07:00.305+0000 7f4f75ffb640 10 monclient(hunting): _init_auth creating new auth 2024-06-03T09:07:00.307 INFO:teuthology.orchestra.run.smithi088.stderr:2024-06-03T09:07:00.305+0000 7f4f75ffb640 10 monclient(hunting): my global_id is 30367 2024-06-03T09:07:00.307 INFO:teuthology.orchestra.run.smithi088.stderr:2024-06-03T09:07:00.305+0000 7f4f75ffb640 1 -- 172.21.15.88:0/3771859321 --> v1:172.21.15.192:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f4f78070c10 con 0x7f4f780717c0 2024-06-03T09:07:00.308 INFO:teuthology.orchestra.run.smithi088.stderr:2024-06-03T09:07:00.308+0000 7f4f75ffb640 1 -- 172.21.15.88:0/3771859321 <== mon.2 v1:172.21.15.88:6790/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 214+0+0 (unknown 1021517462 0 0) 0x7f4f60004630 con 0x7f4f7806e430 2024-06-03T09:07:00.308 INFO:teuthology.orchestra.run.smithi088.stderr:2024-06-03T09:07:00.308+0000 7f4f75ffb640 1 -- 172.21.15.88:0/3771859321 >> v1:172.21.15.192:6789/0 conn(0x7f4f780717c0 legacy=0x7f4f78149240 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-06-03T09:07:00.308 INFO:teuthology.orchestra.run.smithi088.stderr:2024-06-03T09:07:00.308+0000 7f4f75ffb640 1 -- 172.21.15.88:0/3771859321 >> v1:172.21.15.88:6789/0 conn(0x7f4f7806e050 legacy=0x7f4f7813ad00 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-06-03T09:07:00.309 INFO:teuthology.orchestra.run.smithi088.stderr:2024-06-03T09:07:00.308+0000 7f4f75ffb640 10 monclient: _finish_hunting 0 2024-06-03T09:07:00.309 INFO:teuthology.orchestra.run.smithi088.stderr:2024-06-03T09:07:00.308+0000 7f4f75ffb640 1 monclient: found mon.noname-b 2024-06-03T09:07:00.309 INFO:teuthology.orchestra.run.smithi088.stderr:2024-06-03T09:07:00.308+0000 7f4f75ffb640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-06-03T09:07:00.309 INFO:teuthology.orchestra.run.smithi088.stderr:2024-06-03T09:07:00.308+0000 7f4f75ffb640 10 monclient: _send_mon_message to mon.noname-b at v1:172.21.15.88:6790/0 2024-06-03T09:07:00.309 INFO:teuthology.orchestra.run.smithi088.stderr:2024-06-03T09:07:00.308+0000 7f4f75ffb640 1 -- 172.21.15.88:0/3771859321 --> v1:172.21.15.88:6790/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f4f780707f0 con 0x7f4f7806e430 2024-06-03T09:07:00.309 INFO:teuthology.orchestra.run.smithi088.stderr:2024-06-03T09:07:00.308+0000 7f4f75ffb640 10 monclient: _finish_auth 0 2024-06-03T09:07:00.310 INFO:teuthology.orchestra.run.smithi088.stderr:2024-06-03T09:07:00.308+0000 7f4f75ffb640 10 monclient: _check_auth_tickets 2024-06-03T09:07:00.310 INFO:teuthology.orchestra.run.smithi088.stderr:2024-06-03T09:07:00.308+0000 7f4f75ffb640 20 monclient: _check_auth_rotating not needed by client.admin 2024-06-03T09:07:00.310 INFO:teuthology.orchestra.run.smithi088.stderr:2024-06-03T09:07:00.308+0000 7f4f75ffb640 1 -- 172.21.15.88:0/3771859321 <== mon.2 v1:172.21.15.88:6790/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 2234374065 0 0) 0x7f4f6c001610 con 0x7f4f7806e430 2024-06-03T09:07:00.310 INFO:teuthology.orchestra.run.smithi088.stderr:2024-06-03T09:07:00.308+0000 7f4f7da3b640 5 monclient: authenticate success, global_id 30299 2024-06-03T09:07:00.310 INFO:teuthology.orchestra.run.smithi088.stderr:2024-06-03T09:07:00.308+0000 7f4f7da3b640 20 monclient: get_monmap_and_config waiting for monmap|config 2024-06-03T09:07:00.310 INFO:teuthology.orchestra.run.smithi088.stderr:2024-06-03T09:07:00.308+0000 7f4f75ffb640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-06-03T09:07:00.311 INFO:teuthology.orchestra.run.smithi088.stderr:2024-06-03T09:07:00.308+0000 7f4f75ffb640 10 monclient: got monmap 1 from mon.noname-b (according to old e1) 2024-06-03T09:07:00.311 INFO:teuthology.orchestra.run.smithi088.stderr:2024-06-03T09:07:00.308+0000 7f4f75ffb640 10 monclient: dump: 2024-06-03T09:07:00.311 INFO:teuthology.orchestra.run.smithi088.stderr:epoch 1 2024-06-03T09:07:00.312 INFO:teuthology.orchestra.run.smithi088.stderr:fsid ec3c3fdd-3ac6-4d7d-9412-683130ed8b04 2024-06-03T09:07:00.312 INFO:teuthology.orchestra.run.smithi088.stderr:last_changed 2024-06-03T08:34:31.885125+0000 2024-06-03T09:07:00.312 INFO:teuthology.orchestra.run.smithi088.stderr:created 2024-06-03T08:34:31.885125+0000 2024-06-03T09:07:00.312 INFO:teuthology.orchestra.run.smithi088.stderr:min_mon_release 18 (reef) 2024-06-03T09:07:00.312 INFO:teuthology.orchestra.run.smithi088.stderr:election_strategy: 3 2024-06-03T09:07:00.312 INFO:teuthology.orchestra.run.smithi088.stderr:0: v1:172.21.15.88:6789/0 mon.a 2024-06-03T09:07:00.313 INFO:teuthology.orchestra.run.smithi088.stderr:1: v1:172.21.15.192:6789/0 mon.b 2024-06-03T09:07:00.313 INFO:teuthology.orchestra.run.smithi088.stderr:2: v1:172.21.15.88:6790/0 mon.c 2024-06-03T09:07:00.313 INFO:teuthology.orchestra.run.smithi088.stderr: 2024-06-03T09:07:00.313 INFO:teuthology.orchestra.run.smithi088.stderr:2024-06-03T09:07:00.308+0000 7f4f7da3b640 20 monclient: get_monmap_and_config waiting for monmap|config 2024-06-03T09:07:00.325 INFO:teuthology.orchestra.run.smithi088.stderr:2024-06-03T09:07:00.310+0000 7f4f75ffb640 1 -- 172.21.15.88:0/3771859321 <== mon.2 v1:172.21.15.88:6790/0 4 ==== config(0 keys) v1 ==== 4+0+0 (unknown 0 0 0) 0x7f4f6c001830 con 0x7f4f7806e430 2024-06-03T09:07:00.325 INFO:teuthology.orchestra.run.smithi088.stderr:2024-06-03T09:07:00.310+0000 7f4f75ffb640 10 monclient: handle_config config(0 keys) v1 2024-06-03T09:07:00.325 INFO:teuthology.orchestra.run.smithi088.stderr:2024-06-03T09:07:00.310+0000 7f4f7da3b640 10 monclient: get_monmap_and_config success 2024-06-03T09:07:00.326 INFO:teuthology.orchestra.run.smithi088.stderr:2024-06-03T09:07:00.310+0000 7f4f7da3b640 4 set_mon_vals no callback set 2024-06-03T09:07:00.326 INFO:teuthology.orchestra.run.smithi088.stderr:2024-06-03T09:07:00.310+0000 7f4f75ffb640 1 -- 172.21.15.88:0/3771859321 <== mon.2 v1:172.21.15.88:6790/0 5 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 2234374065 0 0) 0x7f4f6c001bc0 con 0x7f4f7806e430 2024-06-03T09:07:00.326 INFO:teuthology.orchestra.run.smithi088.stderr:2024-06-03T09:07:00.310+0000 7f4f75ffb640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-06-03T09:07:00.326 INFO:teuthology.orchestra.run.smithi088.stderr:2024-06-03T09:07:00.310+0000 7f4f75ffb640 10 monclient: got monmap 1 from mon.c (according to old e1) 2024-06-03T09:07:00.326 INFO:teuthology.orchestra.run.smithi088.stderr:2024-06-03T09:07:00.310+0000 7f4f75ffb640 10 monclient: dump: 2024-06-03T09:07:00.327 INFO:teuthology.orchestra.run.smithi088.stderr:epoch 1 2024-06-03T09:07:00.327 INFO:teuthology.orchestra.run.smithi088.stderr:fsid ec3c3fdd-3ac6-4d7d-9412-683130ed8b04 2024-06-03T09:07:00.327 INFO:teuthology.orchestra.run.smithi088.stderr:last_changed 2024-06-03T08:34:31.885125+0000 2024-06-03T09:07:00.327 INFO:teuthology.orchestra.run.smithi088.stderr:created 2024-06-03T08:34:31.885125+0000 2024-06-03T09:07:00.327 INFO:teuthology.orchestra.run.smithi088.stderr:min_mon_release 18 (reef) 2024-06-03T09:07:00.328 INFO:teuthology.orchestra.run.smithi088.stderr:election_strategy: 3 2024-06-03T09:07:00.328 INFO:teuthology.orchestra.run.smithi088.stderr:0: v1:172.21.15.88:6789/0 mon.a 2024-06-03T09:07:00.328 INFO:teuthology.orchestra.run.smithi088.stderr:1: v1:172.21.15.192:6789/0 mon.b 2024-06-03T09:07:00.328 INFO:teuthology.orchestra.run.smithi088.stderr:2: v1:172.21.15.88:6790/0 mon.c 2024-06-03T09:07:00.328 INFO:teuthology.orchestra.run.smithi088.stderr: 2024-06-03T09:07:00.329 INFO:teuthology.orchestra.run.smithi088.stderr:2024-06-03T09:07:00.310+0000 7f4f7da3b640 10 monclient: shutdown 2024-06-03T09:07:00.329 INFO:teuthology.orchestra.run.smithi088.stderr:2024-06-03T09:07:00.310+0000 7f4f7da3b640 20 monclient: shutdown discarding 0 pending message(s) 2024-06-03T09:07:00.329 INFO:teuthology.orchestra.run.smithi088.stderr:2024-06-03T09:07:00.310+0000 7f4f7da3b640 1 -- 172.21.15.88:0/3771859321 >> v1:172.21.15.88:6790/0 conn(0x7f4f7806e430 legacy=0x7f4f780710b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-06-03T09:07:00.329 INFO:teuthology.orchestra.run.smithi088.stderr:2024-06-03T09:07:00.310+0000 7f4f7da3b640 1 -- 172.21.15.88:0/3771859321 shutdown_connections 2024-06-03T09:07:00.329 INFO:teuthology.orchestra.run.smithi088.stderr:2024-06-03T09:07:00.310+0000 7f4f7da3b640 1 -- 172.21.15.88:0/3771859321 >> 172.21.15.88:0/3771859321 conn(0x7f4f780b2f20 msgr2=0x7f4f780b3300 unknown :-1 s=STATE_NONE l=0).mark_down 2024-06-03T09:07:00.329 INFO:teuthology.orchestra.run.smithi088.stderr:2024-06-03T09:07:00.310+0000 7f4f7da3b640 1 -- 172.21.15.88:0/3771859321 shutdown_connections 2024-06-03T09:07:00.330 INFO:teuthology.orchestra.run.smithi088.stderr:2024-06-03T09:07:00.310+0000 7f4f7da3b640 1 -- 172.21.15.88:0/3771859321 wait complete. 2024-06-03T09:07:00.330 INFO:teuthology.orchestra.run.smithi088.stderr:2024-06-03T09:07:00.311+0000 7f4f7da3b640 10 monclient: build_initial_monmap 2024-06-03T09:07:00.330 INFO:teuthology.orchestra.run.smithi088.stderr:2024-06-03T09:07:00.311+0000 7f4f7da3b640 10 monclient: monmap: 2024-06-03T09:07:00.330 INFO:teuthology.orchestra.run.smithi088.stderr:epoch 0 2024-06-03T09:07:00.330 INFO:teuthology.orchestra.run.smithi088.stderr:fsid 00000000-0000-0000-0000-000000000000 2024-06-03T09:07:00.331 INFO:teuthology.orchestra.run.smithi088.stderr:last_changed 0.000000 2024-06-03T09:07:00.331 INFO:teuthology.orchestra.run.smithi088.stderr:created 0.000000 2024-06-03T09:07:00.331 INFO:teuthology.orchestra.run.smithi088.stderr:min_mon_release 0 (unknown) 2024-06-03T09:07:00.331 INFO:teuthology.orchestra.run.smithi088.stderr:election_strategy: 1 2024-06-03T09:07:00.331 INFO:teuthology.orchestra.run.smithi088.stderr:0: v1:172.21.15.88:6789/0 mon.noname-a 2024-06-03T09:07:00.332 INFO:teuthology.orchestra.run.smithi088.stderr:1: v1:172.21.15.192:6789/0 mon.noname-b 2024-06-03T09:07:00.332 INFO:teuthology.orchestra.run.smithi088.stderr:2: v1:172.21.15.88:6790/0 mon.noname-c 2024-06-03T09:07:00.332 INFO:teuthology.orchestra.run.smithi088.stderr: 2024-06-03T09:07:00.332 INFO:teuthology.orchestra.run.smithi088.stderr:2024-06-03T09:07:00.311+0000 7f4f7da3b640 1 librados: starting msgr at 2024-06-03T09:07:00.332 INFO:teuthology.orchestra.run.smithi088.stderr:2024-06-03T09:07:00.311+0000 7f4f7da3b640 1 librados: starting objecter 2024-06-03T09:07:00.332 INFO:teuthology.orchestra.run.smithi088.stderr:2024-06-03T09:07:00.311+0000 7f4f7da3b640 1 Processor -- start 2024-06-03T09:07:00.333 INFO:teuthology.orchestra.run.smithi088.stderr:2024-06-03T09:07:00.312+0000 7f4f7da3b640 1 -- start start 2024-06-03T09:07:00.333 INFO:teuthology.orchestra.run.smithi088.stderr:2024-06-03T09:07:00.312+0000 7f4f7da3b640 1 librados: setting wanted keys 2024-06-03T09:07:00.333 INFO:teuthology.orchestra.run.smithi088.stderr:2024-06-03T09:07:00.312+0000 7f4f7da3b640 1 librados: calling monclient init 2024-06-03T09:07:00.333 INFO:teuthology.orchestra.run.smithi088.stderr:2024-06-03T09:07:00.312+0000 7f4f7da3b640 10 monclient: init 2024-06-03T09:07:00.333 INFO:teuthology.orchestra.run.smithi088.stderr:2024-06-03T09:07:00.312+0000 7f4f7da3b640 10 monclient: _reopen_session rank -1 2024-06-03T09:07:00.334 INFO:teuthology.orchestra.run.smithi088.stderr:2024-06-03T09:07:00.312+0000 7f4f7da3b640 10 monclient: _add_conns ranks=[2,1,0] 2024-06-03T09:07:00.334 INFO:teuthology.orchestra.run.smithi088.stderr:2024-06-03T09:07:00.312+0000 7f4f7da3b640 10 monclient(hunting): picked mon.noname-c con 0x7f4f7806e050 addr v1:172.21.15.88:6790/0 2024-06-03T09:07:00.334 INFO:teuthology.orchestra.run.smithi088.stderr:2024-06-03T09:07:00.312+0000 7f4f7da3b640 10 monclient(hunting): picked mon.noname-b con 0x7f4f780717c0 addr v1:172.21.15.192:6789/0 2024-06-03T09:07:00.334 INFO:teuthology.orchestra.run.smithi088.stderr:2024-06-03T09:07:00.312+0000 7f4f7da3b640 10 monclient(hunting): picked mon.noname-a con 0x7f4f780ac700 addr v1:172.21.15.88:6789/0 2024-06-03T09:07:00.334 INFO:teuthology.orchestra.run.smithi088.stderr:2024-06-03T09:07:00.312+0000 7f4f7da3b640 1 -- --> v1:172.21.15.88:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f4f780ac1c0 con 0x7f4f780ac700 2024-06-03T09:07:00.335 INFO:teuthology.orchestra.run.smithi088.stderr:2024-06-03T09:07:00.312+0000 7f4f7da3b640 1 -- --> v1:172.21.15.192:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f4f780ad1a0 con 0x7f4f780717c0 2024-06-03T09:07:00.335 INFO:teuthology.orchestra.run.smithi088.stderr:2024-06-03T09:07:00.312+0000 7f4f7da3b640 1 -- --> v1:172.21.15.88:6790/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f4f781dfd60 con 0x7f4f7806e050 2024-06-03T09:07:00.335 INFO:teuthology.orchestra.run.smithi088.stderr:2024-06-03T09:07:00.312+0000 7f4f7da3b640 10 monclient(hunting): _renew_subs 2024-06-03T09:07:00.335 INFO:teuthology.orchestra.run.smithi088.stderr:2024-06-03T09:07:00.312+0000 7f4f7da3b640 10 monclient(hunting): authenticate will time out at 3076.122731s 2024-06-03T09:07:00.335 INFO:teuthology.orchestra.run.smithi088.stderr:2024-06-03T09:07:00.313+0000 7f4f76ffd640 1 --1- >> v1:172.21.15.88:6790/0 conn(0x7f4f7806e050 0x7f4f780ab8a0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=1).handle_server_banner_and_identify peer v1:172.21.15.88:6790/0 says I am v1:172.21.15.88:51126/0 (socket says 172.21.15.88:51126) 2024-06-03T09:07:00.335 INFO:teuthology.orchestra.run.smithi088.stderr:2024-06-03T09:07:00.313+0000 7f4f76ffd640 1 -- 172.21.15.88:0/396134345 learned_addr learned my addr 172.21.15.88:0/396134345 (peer_addr_for_me v1:172.21.15.88:0/0) 2024-06-03T09:07:00.335 INFO:teuthology.orchestra.run.smithi088.stderr:2024-06-03T09:07:00.313+0000 7f4f5ffff640 10 client.?.objecter ms_handle_connect 0x7f4f7806e050 2024-06-03T09:07:00.335 INFO:teuthology.orchestra.run.smithi088.stderr:2024-06-03T09:07:00.313+0000 7f4f5ffff640 10 client.?.objecter resend_mon_ops 2024-06-03T09:07:00.335 INFO:teuthology.orchestra.run.smithi088.stderr:2024-06-03T09:07:00.313+0000 7f4f5ffff640 10 client.?.objecter ms_handle_connect 0x7f4f780ac700 2024-06-03T09:07:00.335 INFO:teuthology.orchestra.run.smithi088.stderr:2024-06-03T09:07:00.313+0000 7f4f5ffff640 10 client.?.objecter resend_mon_ops 2024-06-03T09:07:00.335 INFO:teuthology.orchestra.run.smithi088.stderr:2024-06-03T09:07:00.313+0000 7f4f5ffff640 10 client.?.objecter ms_handle_connect 0x7f4f780717c0 2024-06-03T09:07:00.336 INFO:teuthology.orchestra.run.smithi088.stderr:2024-06-03T09:07:00.313+0000 7f4f5ffff640 10 client.?.objecter resend_mon_ops 2024-06-03T09:07:00.336 INFO:teuthology.orchestra.run.smithi088.stderr:2024-06-03T09:07:00.315+0000 7f4f5ffff640 1 -- 172.21.15.88:0/396134345 <== mon.0 v1:172.21.15.88:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 1051657188 0 0) 0x7f4f780ac1c0 con 0x7f4f780ac700 2024-06-03T09:07:00.336 INFO:teuthology.orchestra.run.smithi088.stderr:2024-06-03T09:07:00.315+0000 7f4f5ffff640 10 monclient(hunting): _negotiate 2024-06-03T09:07:00.336 INFO:teuthology.orchestra.run.smithi088.stderr:2024-06-03T09:07:00.315+0000 7f4f5ffff640 10 monclient(hunting): _init_auth method 2 2024-06-03T09:07:00.336 INFO:teuthology.orchestra.run.smithi088.stderr:2024-06-03T09:07:00.315+0000 7f4f5ffff640 10 monclient(hunting): _init_auth creating new auth 2024-06-03T09:07:00.336 INFO:teuthology.orchestra.run.smithi088.stderr:2024-06-03T09:07:00.315+0000 7f4f5ffff640 10 monclient(hunting): my global_id is 30798 2024-06-03T09:07:00.336 INFO:teuthology.orchestra.run.smithi088.stderr:2024-06-03T09:07:00.315+0000 7f4f5ffff640 1 -- 172.21.15.88:0/396134345 --> v1:172.21.15.88:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f4f64004eb0 con 0x7f4f780ac700 2024-06-03T09:07:00.336 INFO:teuthology.orchestra.run.smithi088.stderr:2024-06-03T09:07:00.315+0000 7f4f5ffff640 1 -- 172.21.15.88:0/396134345 <== mon.2 v1:172.21.15.88:6790/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 1922715483 0 0) 0x7f4f781dfd60 con 0x7f4f7806e050 2024-06-03T09:07:00.336 INFO:teuthology.orchestra.run.smithi088.stderr:2024-06-03T09:07:00.315+0000 7f4f5ffff640 10 monclient(hunting): _negotiate 2024-06-03T09:07:00.336 INFO:teuthology.orchestra.run.smithi088.stderr:2024-06-03T09:07:00.315+0000 7f4f5ffff640 10 monclient(hunting): _init_auth method 2 2024-06-03T09:07:00.336 INFO:teuthology.orchestra.run.smithi088.stderr:2024-06-03T09:07:00.315+0000 7f4f5ffff640 10 monclient(hunting): _init_auth creating new auth 2024-06-03T09:07:00.336 INFO:teuthology.orchestra.run.smithi088.stderr:2024-06-03T09:07:00.315+0000 7f4f5ffff640 10 monclient(hunting): my global_id is 30302 2024-06-03T09:07:00.336 INFO:teuthology.orchestra.run.smithi088.stderr:2024-06-03T09:07:00.315+0000 7f4f5ffff640 1 -- 172.21.15.88:0/396134345 --> v1:172.21.15.88:6790/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f4f780ac1c0 con 0x7f4f7806e050 2024-06-03T09:07:00.336 INFO:teuthology.orchestra.run.smithi088.stderr:2024-06-03T09:07:00.315+0000 7f4f5ffff640 1 -- 172.21.15.88:0/396134345 <== mon.1 v1:172.21.15.192:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 758595673 0 0) 0x7f4f780ad1a0 con 0x7f4f780717c0 2024-06-03T09:07:00.337 INFO:teuthology.orchestra.run.smithi088.stderr:2024-06-03T09:07:00.315+0000 7f4f5ffff640 10 monclient(hunting): _negotiate 2024-06-03T09:07:00.337 INFO:teuthology.orchestra.run.smithi088.stderr:2024-06-03T09:07:00.315+0000 7f4f5ffff640 10 monclient(hunting): _init_auth method 2 2024-06-03T09:07:00.337 INFO:teuthology.orchestra.run.smithi088.stderr:2024-06-03T09:07:00.315+0000 7f4f5ffff640 10 monclient(hunting): _init_auth creating new auth 2024-06-03T09:07:00.337 INFO:teuthology.orchestra.run.smithi088.stderr:2024-06-03T09:07:00.315+0000 7f4f5ffff640 10 monclient(hunting): my global_id is 30370 2024-06-03T09:07:00.337 INFO:teuthology.orchestra.run.smithi088.stderr:2024-06-03T09:07:00.315+0000 7f4f5ffff640 1 -- 172.21.15.88:0/396134345 --> v1:172.21.15.192:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f4f781dfd60 con 0x7f4f780717c0 2024-06-03T09:07:00.337 INFO:teuthology.orchestra.run.smithi088.stderr:2024-06-03T09:07:00.320+0000 7f4f5ffff640 1 -- 172.21.15.88:0/396134345 <== mon.2 v1:172.21.15.88:6790/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 764+0+0 (unknown 1298535161 0 0) 0x7f4f780ac1c0 con 0x7f4f7806e050 2024-06-03T09:07:00.337 INFO:teuthology.orchestra.run.smithi088.stderr:2024-06-03T09:07:00.320+0000 7f4f5ffff640 1 -- 172.21.15.88:0/396134345 --> v1:172.21.15.88:6790/0 -- auth(proto 2 165 bytes epoch 0) v1 -- 0x7f4f780ad1a0 con 0x7f4f7806e050 2024-06-03T09:07:00.337 INFO:teuthology.orchestra.run.smithi088.stderr:2024-06-03T09:07:00.320+0000 7f4f5ffff640 1 -- 172.21.15.88:0/396134345 <== mon.2 v1:172.21.15.88:6790/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 2234374065 0 0) 0x7f4f68001970 con 0x7f4f7806e050 2024-06-03T09:07:00.337 INFO:teuthology.orchestra.run.smithi088.stderr:2024-06-03T09:07:00.320+0000 7f4f5ffff640 10 monclient(hunting): handle_monmap mon_map magic: 0 v1 2024-06-03T09:07:00.337 INFO:teuthology.orchestra.run.smithi088.stderr:2024-06-03T09:07:00.320+0000 7f4f5ffff640 10 monclient(hunting): got monmap 1 from mon.noname-c (according to old e1) 2024-06-03T09:07:00.337 INFO:teuthology.orchestra.run.smithi088.stderr:2024-06-03T09:07:00.320+0000 7f4f5ffff640 10 monclient(hunting): dump: 2024-06-03T09:07:00.337 INFO:teuthology.orchestra.run.smithi088.stderr:epoch 1 2024-06-03T09:07:00.337 INFO:teuthology.orchestra.run.smithi088.stderr:fsid ec3c3fdd-3ac6-4d7d-9412-683130ed8b04 2024-06-03T09:07:00.338 INFO:teuthology.orchestra.run.smithi088.stderr:last_changed 2024-06-03T08:34:31.885125+0000 2024-06-03T09:07:00.338 INFO:teuthology.orchestra.run.smithi088.stderr:created 2024-06-03T08:34:31.885125+0000 2024-06-03T09:07:00.338 INFO:teuthology.orchestra.run.smithi088.stderr:min_mon_release 18 (reef) 2024-06-03T09:07:00.338 INFO:teuthology.orchestra.run.smithi088.stderr:election_strategy: 3 2024-06-03T09:07:00.338 INFO:teuthology.orchestra.run.smithi088.stderr:0: v1:172.21.15.88:6789/0 mon.a 2024-06-03T09:07:00.338 INFO:teuthology.orchestra.run.smithi088.stderr:1: v1:172.21.15.192:6789/0 mon.b 2024-06-03T09:07:00.338 INFO:teuthology.orchestra.run.smithi088.stderr:2: v1:172.21.15.88:6790/0 mon.c 2024-06-03T09:07:00.338 INFO:teuthology.orchestra.run.smithi088.stderr: 2024-06-03T09:07:00.338 INFO:teuthology.orchestra.run.smithi088.stderr:2024-06-03T09:07:00.320+0000 7f4f5ffff640 10 monclient(hunting): _finish_auth 0 2024-06-03T09:07:00.338 INFO:teuthology.orchestra.run.smithi088.stderr:2024-06-03T09:07:00.320+0000 7f4f5ffff640 1 -- 172.21.15.88:0/396134345 <== mon.0 v1:172.21.15.88:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 764+0+0 (unknown 3926464318 0 0) 0x7f4f64004eb0 con 0x7f4f780ac700 2024-06-03T09:07:00.338 INFO:teuthology.orchestra.run.smithi088.stderr:2024-06-03T09:07:00.320+0000 7f4f5ffff640 1 -- 172.21.15.88:0/396134345 --> v1:172.21.15.88:6789/0 -- auth(proto 2 165 bytes epoch 0) v1 -- 0x7f4f780ac1c0 con 0x7f4f780ac700 2024-06-03T09:07:00.338 INFO:teuthology.orchestra.run.smithi088.stderr:2024-06-03T09:07:00.320+0000 7f4f5ffff640 1 -- 172.21.15.88:0/396134345 <== mon.0 v1:172.21.15.88:6789/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 2234374065 0 0) 0x7f4f70001b90 con 0x7f4f780ac700 2024-06-03T09:07:00.338 INFO:teuthology.orchestra.run.smithi088.stderr:2024-06-03T09:07:00.320+0000 7f4f5ffff640 10 monclient(hunting): handle_monmap mon_map magic: 0 v1 2024-06-03T09:07:00.339 INFO:teuthology.orchestra.run.smithi088.stderr:2024-06-03T09:07:00.320+0000 7f4f5ffff640 10 monclient(hunting): got monmap 1 from mon.a (according to old e1) 2024-06-03T09:07:00.339 INFO:teuthology.orchestra.run.smithi088.stderr:2024-06-03T09:07:00.320+0000 7f4f5ffff640 10 monclient(hunting): dump: 2024-06-03T09:07:00.339 INFO:teuthology.orchestra.run.smithi088.stderr:epoch 1 2024-06-03T09:07:00.339 INFO:teuthology.orchestra.run.smithi088.stderr:fsid ec3c3fdd-3ac6-4d7d-9412-683130ed8b04 2024-06-03T09:07:00.339 INFO:teuthology.orchestra.run.smithi088.stderr:last_changed 2024-06-03T08:34:31.885125+0000 2024-06-03T09:07:00.339 INFO:teuthology.orchestra.run.smithi088.stderr:created 2024-06-03T08:34:31.885125+0000 2024-06-03T09:07:00.339 INFO:teuthology.orchestra.run.smithi088.stderr:min_mon_release 18 (reef) 2024-06-03T09:07:00.339 INFO:teuthology.orchestra.run.smithi088.stderr:election_strategy: 3 2024-06-03T09:07:00.339 INFO:teuthology.orchestra.run.smithi088.stderr:0: v1:172.21.15.88:6789/0 mon.a 2024-06-03T09:07:00.339 INFO:teuthology.orchestra.run.smithi088.stderr:1: v1:172.21.15.192:6789/0 mon.b 2024-06-03T09:07:00.339 INFO:teuthology.orchestra.run.smithi088.stderr:2: v1:172.21.15.88:6790/0 mon.c 2024-06-03T09:07:00.339 INFO:teuthology.orchestra.run.smithi088.stderr: 2024-06-03T09:07:00.339 INFO:teuthology.orchestra.run.smithi088.stderr:2024-06-03T09:07:00.320+0000 7f4f5ffff640 1 -- 172.21.15.88:0/396134345 <== mon.1 v1:172.21.15.192:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 764+0+0 (unknown 3946221614 0 0) 0x7f4f781dfd60 con 0x7f4f780717c0 2024-06-03T09:07:00.339 INFO:teuthology.orchestra.run.smithi088.stderr:2024-06-03T09:07:00.320+0000 7f4f5ffff640 1 -- 172.21.15.88:0/396134345 --> v1:172.21.15.192:6789/0 -- auth(proto 2 165 bytes epoch 0) v1 -- 0x7f4f64004eb0 con 0x7f4f780717c0 2024-06-03T09:07:00.340 INFO:teuthology.orchestra.run.smithi088.stderr:2024-06-03T09:07:00.320+0000 7f4f5ffff640 1 -- 172.21.15.88:0/396134345 <== mon.1 v1:172.21.15.192:6789/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 2234374065 0 0) 0x7f4f6c001510 con 0x7f4f780717c0 2024-06-03T09:07:00.340 INFO:teuthology.orchestra.run.smithi088.stderr:2024-06-03T09:07:00.320+0000 7f4f5ffff640 10 monclient(hunting): handle_monmap mon_map magic: 0 v1 2024-06-03T09:07:00.340 INFO:teuthology.orchestra.run.smithi088.stderr:2024-06-03T09:07:00.320+0000 7f4f5ffff640 10 monclient(hunting): got monmap 1 from mon.b (according to old e1) 2024-06-03T09:07:00.340 INFO:teuthology.orchestra.run.smithi088.stderr:2024-06-03T09:07:00.320+0000 7f4f5ffff640 10 monclient(hunting): dump: 2024-06-03T09:07:00.340 INFO:teuthology.orchestra.run.smithi088.stderr:epoch 1 2024-06-03T09:07:00.340 INFO:teuthology.orchestra.run.smithi088.stderr:fsid ec3c3fdd-3ac6-4d7d-9412-683130ed8b04 2024-06-03T09:07:00.340 INFO:teuthology.orchestra.run.smithi088.stderr:last_changed 2024-06-03T08:34:31.885125+0000 2024-06-03T09:07:00.340 INFO:teuthology.orchestra.run.smithi088.stderr:created 2024-06-03T08:34:31.885125+0000 2024-06-03T09:07:00.340 INFO:teuthology.orchestra.run.smithi088.stderr:min_mon_release 18 (reef) 2024-06-03T09:07:00.340 INFO:teuthology.orchestra.run.smithi088.stderr:election_strategy: 3 2024-06-03T09:07:00.340 INFO:teuthology.orchestra.run.smithi088.stderr:0: v1:172.21.15.88:6789/0 mon.a 2024-06-03T09:07:00.340 INFO:teuthology.orchestra.run.smithi088.stderr:1: v1:172.21.15.192:6789/0 mon.b 2024-06-03T09:07:00.340 INFO:teuthology.orchestra.run.smithi088.stderr:2: v1:172.21.15.88:6790/0 mon.c 2024-06-03T09:07:00.340 INFO:teuthology.orchestra.run.smithi088.stderr: 2024-06-03T09:07:00.341 INFO:teuthology.orchestra.run.smithi088.stderr:2024-06-03T09:07:00.320+0000 7f4f777fe640 1 -- 172.21.15.88:0/396134345 >> v1:172.21.15.88:6789/0 conn(0x7f4f780ac700 legacy=0x7f4f781d2a00 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).read_bulk peer close file descriptor 18 2024-06-03T09:07:00.341 INFO:teuthology.orchestra.run.smithi088.stderr:2024-06-03T09:07:00.320+0000 7f4f777fe640 1 -- 172.21.15.88:0/396134345 >> v1:172.21.15.88:6789/0 conn(0x7f4f780ac700 legacy=0x7f4f781d2a00 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).read_until read failed 2024-06-03T09:07:00.341 INFO:teuthology.orchestra.run.smithi088.stderr:2024-06-03T09:07:00.320+0000 7f4f777fe640 1 --1- 172.21.15.88:0/396134345 >> v1:172.21.15.88:6789/0 conn(0x7f4f780ac700 0x7f4f781d2a00 :-1 s=OPENED pgs=9338 cs=1 l=1).handle_message read tag failed 2024-06-03T09:07:00.341 INFO:teuthology.orchestra.run.smithi088.stderr:2024-06-03T09:07:00.320+0000 7f4f777fe640 1 --1- 172.21.15.88:0/396134345 >> v1:172.21.15.88:6789/0 conn(0x7f4f780ac700 0x7f4f781d2a00 :-1 s=OPENED pgs=9338 cs=1 l=1).fault on lossy channel, failing 2024-06-03T09:07:00.341 INFO:teuthology.orchestra.run.smithi088.stderr:2024-06-03T09:07:00.320+0000 7f4f5ffff640 10 monclient(hunting): ms_handle_reset hunted mon v1:172.21.15.88:6789/0 2024-06-03T09:07:00.341 INFO:teuthology.orchestra.run.smithi088.stderr:2024-06-03T09:07:00.323+0000 7f4f5ffff640 1 -- 172.21.15.88:0/396134345 <== mon.2 v1:172.21.15.88:6790/0 4 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 580+0+0 (unknown 1001617212 0 0) 0x7f4f780ad1a0 con 0x7f4f7806e050 2024-06-03T09:07:00.341 INFO:teuthology.orchestra.run.smithi088.stderr:2024-06-03T09:07:00.323+0000 7f4f5ffff640 1 -- 172.21.15.88:0/396134345 >> v1:172.21.15.192:6789/0 conn(0x7f4f780717c0 legacy=0x7f4f781d22f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-06-03T09:07:00.341 INFO:teuthology.orchestra.run.smithi088.stderr:2024-06-03T09:07:00.324+0000 7f4f5ffff640 1 -- 172.21.15.88:0/396134345 >> v1:172.21.15.88:6789/0 conn(0x7f4f780ac700 legacy=0x7f4f781d2a00 unknown :-1 s=STATE_CLOSED l=1).mark_down 2024-06-03T09:07:00.341 INFO:teuthology.orchestra.run.smithi088.stderr:2024-06-03T09:07:00.324+0000 7f4f5ffff640 10 monclient: _finish_hunting 0 2024-06-03T09:07:00.341 INFO:teuthology.orchestra.run.smithi088.stderr:2024-06-03T09:07:00.324+0000 7f4f5ffff640 1 monclient: found mon.c 2024-06-03T09:07:00.341 INFO:teuthology.orchestra.run.smithi088.stderr:2024-06-03T09:07:00.324+0000 7f4f5ffff640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-06-03T09:07:00.341 INFO:teuthology.orchestra.run.smithi088.stderr:2024-06-03T09:07:00.324+0000 7f4f5ffff640 10 monclient: _send_mon_message to mon.c at v1:172.21.15.88:6790/0 2024-06-03T09:07:00.341 INFO:teuthology.orchestra.run.smithi088.stderr:2024-06-03T09:07:00.324+0000 7f4f5ffff640 1 -- 172.21.15.88:0/396134345 --> v1:172.21.15.88:6790/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f4f781e0f00 con 0x7f4f7806e050 2024-06-03T09:07:00.341 INFO:teuthology.orchestra.run.smithi088.stderr:2024-06-03T09:07:00.324+0000 7f4f5ffff640 10 monclient: _finish_auth 0 2024-06-03T09:07:00.342 INFO:teuthology.orchestra.run.smithi088.stderr:2024-06-03T09:07:00.324+0000 7f4f5ffff640 10 monclient: _check_auth_tickets 2024-06-03T09:07:00.342 INFO:teuthology.orchestra.run.smithi088.stderr:2024-06-03T09:07:00.324+0000 7f4f5ffff640 20 monclient: _check_auth_rotating not needed by client.admin 2024-06-03T09:07:00.342 INFO:teuthology.orchestra.run.smithi088.stderr:2024-06-03T09:07:00.324+0000 7f4f7da3b640 5 monclient: authenticate success, global_id 30302 2024-06-03T09:07:00.342 INFO:teuthology.orchestra.run.smithi088.stderr:2024-06-03T09:07:00.324+0000 7f4f7da3b640 10 monclient: _renew_subs 2024-06-03T09:07:00.342 INFO:teuthology.orchestra.run.smithi088.stderr:2024-06-03T09:07:00.324+0000 7f4f7da3b640 10 monclient: _send_mon_message to mon.c at v1:172.21.15.88:6790/0 2024-06-03T09:07:00.342 INFO:teuthology.orchestra.run.smithi088.stderr:2024-06-03T09:07:00.324+0000 7f4f7da3b640 1 -- 172.21.15.88:0/396134345 --> v1:172.21.15.88:6790/0 -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f4f781dfef0 con 0x7f4f7806e050 2024-06-03T09:07:00.342 INFO:teuthology.orchestra.run.smithi088.stderr:2024-06-03T09:07:00.324+0000 7f4f7da3b640 10 client.30302.objecter _maybe_request_map subscribing (onetime) to next osd map 2024-06-03T09:07:00.342 INFO:teuthology.orchestra.run.smithi088.stderr:2024-06-03T09:07:00.324+0000 7f4f7da3b640 10 monclient: _renew_subs 2024-06-03T09:07:00.342 INFO:teuthology.orchestra.run.smithi088.stderr:2024-06-03T09:07:00.324+0000 7f4f7da3b640 10 monclient: _send_mon_message to mon.c at v1:172.21.15.88:6790/0 2024-06-03T09:07:00.342 INFO:teuthology.orchestra.run.smithi088.stderr:2024-06-03T09:07:00.324+0000 7f4f7da3b640 1 -- 172.21.15.88:0/396134345 --> v1:172.21.15.88:6790/0 -- mon_subscribe({osdmap=0}) v3 -- 0x7f4f781e0380 con 0x7f4f7806e050 2024-06-03T09:07:00.342 INFO:teuthology.orchestra.run.smithi088.stderr:2024-06-03T09:07:00.324+0000 7f4f7da3b640 1 librados: init done 2024-06-03T09:07:00.342 INFO:teuthology.orchestra.run.smithi088.stderr:2024-06-03T09:07:00.324+0000 7f4f7da3b640 10 client.30302.objecter _submit_command 1 [{"prefix": "get_command_descriptions"}] 2024-06-03T09:07:00.342 INFO:teuthology.orchestra.run.smithi088.stderr:2024-06-03T09:07:00.324+0000 7f4f7da3b640 15 client.30302.objecter _session_command_op_assign -1 1 2024-06-03T09:07:00.343 INFO:teuthology.orchestra.run.smithi088.stderr:2024-06-03T09:07:00.324+0000 7f4f7da3b640 20 client.30302.objecter _get_session osd=-1 returning homeless 2024-06-03T09:07:00.343 INFO:teuthology.orchestra.run.smithi088.stderr:2024-06-03T09:07:00.324+0000 7f4f7da3b640 10 client.30302.objecter _maybe_request_map subscribing (onetime) to next osd map 2024-06-03T09:07:00.343 INFO:teuthology.orchestra.run.smithi088.stderr:2024-06-03T09:07:00.324+0000 7f4f7da3b640 10 monclient: _send_mon_message to mon.c at v1:172.21.15.88:6790/0 2024-06-03T09:07:00.343 INFO:teuthology.orchestra.run.smithi088.stderr:2024-06-03T09:07:00.324+0000 7f4f7da3b640 1 -- 172.21.15.88:0/396134345 --> v1:172.21.15.88:6790/0 -- mon_get_version(what=osdmap handle=1) v1 -- 0x7f4f7813e7b0 con 0x7f4f7806e050 2024-06-03T09:07:00.345 INFO:teuthology.orchestra.run.smithi088.stderr:2024-06-03T09:07:00.339+0000 7f4f5ffff640 1 -- 172.21.15.88:0/396134345 <== mon.2 v1:172.21.15.88:6790/0 5 ==== config(0 keys) v1 ==== 4+0+0 (unknown 0 0 0) 0x7f4f7813e7b0 con 0x7f4f7806e050 2024-06-03T09:07:00.345 INFO:teuthology.orchestra.run.smithi088.stderr:2024-06-03T09:07:00.339+0000 7f4f5ffff640 10 monclient: handle_config config(0 keys) v1 2024-06-03T09:07:00.345 INFO:teuthology.orchestra.run.smithi088.stderr:2024-06-03T09:07:00.339+0000 7f4f5ffff640 1 -- 172.21.15.88:0/396134345 <== mon.2 v1:172.21.15.88:6790/0 6 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 2234374065 0 0) 0x7f4f680037b0 con 0x7f4f7806e050 2024-06-03T09:07:00.345 INFO:teuthology.orchestra.run.smithi088.stderr:2024-06-03T09:07:00.339+0000 7f4f5ffff640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-06-03T09:07:00.345 INFO:teuthology.orchestra.run.smithi088.stderr:2024-06-03T09:07:00.339+0000 7f4f5ffff640 10 monclient: got monmap 1 from mon.c (according to old e1) 2024-06-03T09:07:00.345 INFO:teuthology.orchestra.run.smithi088.stderr:2024-06-03T09:07:00.339+0000 7f4f5ffff640 10 monclient: dump: 2024-06-03T09:07:00.345 INFO:teuthology.orchestra.run.smithi088.stderr:epoch 1 2024-06-03T09:07:00.345 INFO:teuthology.orchestra.run.smithi088.stderr:fsid ec3c3fdd-3ac6-4d7d-9412-683130ed8b04 2024-06-03T09:07:00.345 INFO:teuthology.orchestra.run.smithi088.stderr:last_changed 2024-06-03T08:34:31.885125+0000 2024-06-03T09:07:00.346 INFO:teuthology.orchestra.run.smithi088.stderr:created 2024-06-03T08:34:31.885125+0000 2024-06-03T09:07:00.346 INFO:teuthology.orchestra.run.smithi088.stderr:min_mon_release 18 (reef) 2024-06-03T09:07:00.346 INFO:teuthology.orchestra.run.smithi088.stderr:election_strategy: 3 2024-06-03T09:07:00.346 INFO:teuthology.orchestra.run.smithi088.stderr:0: v1:172.21.15.88:6789/0 mon.a 2024-06-03T09:07:00.346 INFO:teuthology.orchestra.run.smithi088.stderr:1: v1:172.21.15.192:6789/0 mon.b 2024-06-03T09:07:00.346 INFO:teuthology.orchestra.run.smithi088.stderr:2: v1:172.21.15.88:6790/0 mon.c 2024-06-03T09:07:00.346 INFO:teuthology.orchestra.run.smithi088.stderr: 2024-06-03T09:07:00.346 INFO:teuthology.orchestra.run.smithi088.stderr:2024-06-03T09:07:00.339+0000 7f4f757fa640 4 set_mon_vals no callback set 2024-06-03T09:07:00.346 INFO:teuthology.orchestra.run.smithi088.stderr:2024-06-03T09:07:00.340+0000 7f4f5ffff640 1 -- 172.21.15.88:0/396134345 <== mon.2 v1:172.21.15.88:6790/0 7 ==== mgrmap(e 5) v1 ==== 89110+0+0 (unknown 873533973 0 0) 0x7f4f68003910 con 0x7f4f7806e050 2024-06-03T09:07:00.346 INFO:teuthology.orchestra.run.smithi088.stderr:2024-06-03T09:07:00.341+0000 7f4f767fc640 10 monclient: get_auth_request con 0x7f4f6406d3c0 auth_method 0 2024-06-03T09:07:00.346 INFO:teuthology.orchestra.run.smithi088.stderr:2024-06-03T09:07:00.341+0000 7f4f5ffff640 10 client.30302.objecter ms_handle_connect 0x7f4f6406d3c0 2024-06-03T09:07:00.346 INFO:teuthology.orchestra.run.smithi088.stderr:2024-06-03T09:07:00.343+0000 7f4f5ffff640 1 -- 172.21.15.88:0/396134345 <== mon.2 v1:172.21.15.88:6790/0 8 ==== osd_map(1014..1014 src has 622..1014) v4 ==== 5229+0+0 (unknown 3230192161 0 0) 0x7f4f68053fb0 con 0x7f4f7806e050 2024-06-03T09:07:00.346 INFO:teuthology.orchestra.run.smithi088.stderr:2024-06-03T09:07:00.343+0000 7f4f5ffff640 10 client.30302.objecter ms_dispatch 0x7f4f78000b90 osd_map(1014..1014 src has 622..1014) v4 2024-06-03T09:07:00.347 INFO:teuthology.orchestra.run.smithi088.stderr:2024-06-03T09:07:00.343+0000 7f4f5ffff640 3 client.30302.objecter handle_osd_map got epochs [1014,1014] > 0 2024-06-03T09:07:00.347 INFO:teuthology.orchestra.run.smithi088.stderr:2024-06-03T09:07:00.343+0000 7f4f5ffff640 3 client.30302.objecter handle_osd_map decoding full epoch 1014 2024-06-03T09:07:00.347 INFO:teuthology.orchestra.run.smithi088.stderr:2024-06-03T09:07:00.343+0000 7f4f5ffff640 10 client.30302.objecter checking command 1 2024-06-03T09:07:00.347 INFO:teuthology.orchestra.run.smithi088.stderr:2024-06-03T09:07:00.343+0000 7f4f5ffff640 20 client.30302.objecter _get_session s=0x7f4f64075390 osd=5 3 2024-06-03T09:07:00.347 INFO:teuthology.orchestra.run.smithi088.stderr:2024-06-03T09:07:00.343+0000 7f4f5ffff640 20 client.30302.objecter put_session s=0x7f4f64075390 osd=5 3 2024-06-03T09:07:00.347 INFO:teuthology.orchestra.run.smithi088.stderr:2024-06-03T09:07:00.343+0000 7f4f5ffff640 15 client.30302.objecter _session_command_op_remove -1 1 2024-06-03T09:07:00.347 INFO:teuthology.orchestra.run.smithi088.stderr:2024-06-03T09:07:00.343+0000 7f4f5ffff640 20 client.30302.objecter _get_session s=0x7f4f64075390 osd=5 3 2024-06-03T09:07:00.347 INFO:teuthology.orchestra.run.smithi088.stderr:2024-06-03T09:07:00.343+0000 7f4f5ffff640 20 client.30302.objecter get_session s=0x7f4f64075390 osd=5 3 2024-06-03T09:07:00.347 INFO:teuthology.orchestra.run.smithi088.stderr:2024-06-03T09:07:00.343+0000 7f4f5ffff640 15 client.30302.objecter _session_command_op_assign 5 1 2024-06-03T09:07:00.347 INFO:teuthology.orchestra.run.smithi088.stderr:2024-06-03T09:07:00.343+0000 7f4f5ffff640 20 client.30302.objecter put_session s=0x7f4f64075390 osd=5 4 2024-06-03T09:07:00.347 INFO:teuthology.orchestra.run.smithi088.stderr:2024-06-03T09:07:00.343+0000 7f4f5ffff640 10 client.30302.objecter _send_command 1 2024-06-03T09:07:00.347 INFO:teuthology.orchestra.run.smithi088.stderr:2024-06-03T09:07:00.343+0000 7f4f5ffff640 1 -- 172.21.15.88:0/396134345 --> v1:172.21.15.192:6805/34786 -- command(tid 1: {"prefix": "get_command_descriptions"}) v1 -- 0x7f4f640796b0 con 0x7f4f64075a40 2024-06-03T09:07:00.347 INFO:teuthology.orchestra.run.smithi088.stderr:2024-06-03T09:07:00.343+0000 7f4f5ffff640 20 client.30302.objecter dump_active .. 0 homeless 2024-06-03T09:07:00.348 INFO:teuthology.orchestra.run.smithi088.stderr:2024-06-03T09:07:00.343+0000 7f4f777fe640 10 monclient: get_auth_request con 0x7f4f64075a40 auth_method 0 2024-06-03T09:07:00.348 INFO:teuthology.orchestra.run.smithi088.stderr:2024-06-03T09:07:00.344+0000 7f4f5ffff640 1 -- 172.21.15.88:0/396134345 <== mon.2 v1:172.21.15.88:6790/0 9 ==== mon_get_version_reply(handle=1 version=1014) v2 ==== 24+0+0 (unknown 3501658476 0 0) 0x7f4f6804fb60 con 0x7f4f7806e050 2024-06-03T09:07:00.348 INFO:teuthology.orchestra.run.smithi088.stderr:2024-06-03T09:07:00.344+0000 7f4f5ffff640 10 monclient: handle_get_version_reply finishing 1 version 1014 2024-06-03T09:07:00.348 INFO:teuthology.orchestra.run.smithi088.stderr:2024-06-03T09:07:00.345+0000 7f4f5ffff640 10 client.30302.objecter ms_handle_connect 0x7f4f64075a40 2024-06-03T09:07:00.369 INFO:teuthology.orchestra.run.smithi088.stderr:2024-06-03T09:07:00.368+0000 7f4f5ffff640 1 -- 172.21.15.88:0/396134345 <== osd.5 v1:172.21.15.192:6805/34786 1 ==== command_reply(tid 1: 0 ) v1 ==== 8+0+26118 (unknown 0 0 1686565167) 0x7f4f640796b0 con 0x7f4f64075a40 2024-06-03T09:07:00.369 INFO:teuthology.orchestra.run.smithi088.stderr:2024-06-03T09:07:00.368+0000 7f4f5ffff640 10 client.30302.objecter ms_dispatch 0x7f4f78000b90 command_reply(tid 1: 0 ) v1 2024-06-03T09:07:00.369 INFO:teuthology.orchestra.run.smithi088.stderr:2024-06-03T09:07:00.368+0000 7f4f5ffff640 10 client.30302.objecter _finish_command 1 = system:0 2024-06-03T09:07:00.369 INFO:teuthology.orchestra.run.smithi088.stderr:2024-06-03T09:07:00.368+0000 7f4f5ffff640 20 client.30302.objecter put_session s=0x7f4f64075390 osd=5 4 2024-06-03T09:07:00.369 INFO:teuthology.orchestra.run.smithi088.stderr:2024-06-03T09:07:00.368+0000 7f4f5ffff640 15 client.30302.objecter _session_command_op_remove 5 1 2024-06-03T09:07:00.381 INFO:teuthology.orchestra.run.smithi088.stderr:2024-06-03T09:07:00.381+0000 7f4f7da3b640 10 client.30302.objecter _submit_command 2 [{"prefix": "config set", "var": "osd_debug_deep_scrub_sleep", "val": ["0"]}] 2024-06-03T09:07:00.382 INFO:teuthology.orchestra.run.smithi088.stderr:2024-06-03T09:07:00.381+0000 7f4f7da3b640 15 client.30302.objecter _session_command_op_assign -1 2 2024-06-03T09:07:00.382 INFO:teuthology.orchestra.run.smithi088.stderr:2024-06-03T09:07:00.381+0000 7f4f7da3b640 20 client.30302.objecter _get_session s=0x7f4f64075390 osd=5 3 2024-06-03T09:07:00.382 INFO:teuthology.orchestra.run.smithi088.stderr:2024-06-03T09:07:00.381+0000 7f4f7da3b640 20 client.30302.objecter put_session s=0x7f4f64075390 osd=5 3 2024-06-03T09:07:00.382 INFO:teuthology.orchestra.run.smithi088.stderr:2024-06-03T09:07:00.381+0000 7f4f7da3b640 20 client.30302.objecter _get_session s=0x7f4f64075390 osd=5 3 2024-06-03T09:07:00.382 INFO:teuthology.orchestra.run.smithi088.stderr:2024-06-03T09:07:00.381+0000 7f4f7da3b640 15 client.30302.objecter _session_command_op_remove -1 2 2024-06-03T09:07:00.382 INFO:teuthology.orchestra.run.smithi088.stderr:2024-06-03T09:07:00.381+0000 7f4f7da3b640 20 client.30302.objecter get_session s=0x7f4f64075390 osd=5 3 2024-06-03T09:07:00.382 INFO:teuthology.orchestra.run.smithi088.stderr:2024-06-03T09:07:00.381+0000 7f4f7da3b640 15 client.30302.objecter _session_command_op_assign 5 2 2024-06-03T09:07:00.382 INFO:teuthology.orchestra.run.smithi088.stderr:2024-06-03T09:07:00.381+0000 7f4f7da3b640 20 client.30302.objecter put_session s=0x7f4f64075390 osd=5 4 2024-06-03T09:07:00.382 INFO:teuthology.orchestra.run.smithi088.stderr:2024-06-03T09:07:00.381+0000 7f4f7da3b640 10 client.30302.objecter _send_command 2 2024-06-03T09:07:00.382 INFO:teuthology.orchestra.run.smithi088.stderr:2024-06-03T09:07:00.381+0000 7f4f7da3b640 1 -- 172.21.15.88:0/396134345 --> v1:172.21.15.192:6805/34786 -- command(tid 2: {"prefix": "config set", "var": "osd_debug_deep_scrub_sleep", "val": ["0"]}) v1 -- 0x7f4f7806eeb0 con 0x7f4f64075a40 2024-06-03T09:07:00.398 INFO:teuthology.orchestra.run.smithi088.stderr:2024-06-03T09:07:00.398+0000 7f4f5ffff640 1 -- 172.21.15.88:0/396134345 <== osd.5 v1:172.21.15.192:6805/34786 2 ==== command_reply(tid 2: 0 ) v1 ==== 8+0+97 (unknown 0 0 119112422) 0x7f4f70007b40 con 0x7f4f64075a40 2024-06-03T09:07:00.398 INFO:teuthology.orchestra.run.smithi088.stderr:2024-06-03T09:07:00.398+0000 7f4f5ffff640 10 client.30302.objecter ms_dispatch 0x7f4f78000b90 command_reply(tid 2: 0 ) v1 2024-06-03T09:07:00.398 INFO:teuthology.orchestra.run.smithi088.stderr:2024-06-03T09:07:00.398+0000 7f4f5ffff640 10 client.30302.objecter _finish_command 2 = system:0 2024-06-03T09:07:00.399 INFO:teuthology.orchestra.run.smithi088.stderr:2024-06-03T09:07:00.398+0000 7f4f5ffff640 20 client.30302.objecter put_session s=0x7f4f64075390 osd=5 4 2024-06-03T09:07:00.399 INFO:teuthology.orchestra.run.smithi088.stderr:2024-06-03T09:07:00.398+0000 7f4f5ffff640 15 client.30302.objecter _session_command_op_remove 5 2 2024-06-03T09:07:00.399 INFO:teuthology.orchestra.run.smithi088.stdout:{ 2024-06-03T09:07:00.399 INFO:teuthology.orchestra.run.smithi088.stdout: "success": "osd_debug_deep_scrub_sleep = '' (not observed, change may require restart) " 2024-06-03T09:07:00.399 INFO:teuthology.orchestra.run.smithi088.stdout:} 2024-06-03T09:07:00.400 INFO:teuthology.orchestra.run.smithi088.stderr:2024-06-03T09:07:00.398+0000 7f4f7da3b640 10 librados: watch_flush enter 2024-06-03T09:07:00.400 INFO:teuthology.orchestra.run.smithi088.stderr:2024-06-03T09:07:00.398+0000 7f4f7da3b640 10 librados: watch_flush exit 2024-06-03T09:07:00.401 INFO:teuthology.orchestra.run.smithi088.stderr:2024-06-03T09:07:00.398+0000 7f4f7da3b640 10 client.30302.objecter close_session for osd.5 2024-06-03T09:07:00.401 INFO:teuthology.orchestra.run.smithi088.stderr:2024-06-03T09:07:00.398+0000 7f4f7da3b640 1 -- 172.21.15.88:0/396134345 >> v1:172.21.15.192:6805/34786 conn(0x7f4f64075a40 legacy=0x7f4f64077ee0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-06-03T09:07:00.401 INFO:teuthology.orchestra.run.smithi088.stderr:2024-06-03T09:07:00.398+0000 7f4f7da3b640 20 client.30302.objecter put_session s=0x7f4f64075390 osd=5 1 2024-06-03T09:07:00.401 INFO:teuthology.orchestra.run.smithi088.stderr:2024-06-03T09:07:00.398+0000 7f4f7da3b640 20 client.30302.objecter shutdown clearing up homeless session... 2024-06-03T09:07:00.401 INFO:teuthology.orchestra.run.smithi088.stderr:2024-06-03T09:07:00.398+0000 7f4f7da3b640 10 client.30302.objecter successfully canceled tick 2024-06-03T09:07:00.401 INFO:teuthology.orchestra.run.smithi088.stderr:2024-06-03T09:07:00.398+0000 7f4f7da3b640 1 -- 172.21.15.88:0/396134345 >> v1:172.21.15.192:6800/34229 conn(0x7f4f6406d3c0 legacy=0x7f4f6406f880 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-06-03T09:07:00.401 INFO:teuthology.orchestra.run.smithi088.stderr:2024-06-03T09:07:00.399+0000 7f4f7da3b640 10 monclient: shutdown 2024-06-03T09:07:00.401 INFO:teuthology.orchestra.run.smithi088.stderr:2024-06-03T09:07:00.399+0000 7f4f7da3b640 20 monclient: shutdown discarding 0 pending message(s) 2024-06-03T09:07:00.401 INFO:teuthology.orchestra.run.smithi088.stderr:2024-06-03T09:07:00.399+0000 7f4f7da3b640 1 -- 172.21.15.88:0/396134345 >> v1:172.21.15.88:6790/0 conn(0x7f4f7806e050 legacy=0x7f4f780ab8a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-06-03T09:07:00.401 INFO:teuthology.orchestra.run.smithi088.stderr:2024-06-03T09:07:00.399+0000 7f4f777fe640 1 -- 172.21.15.88:0/396134345 reap_dead start 2024-06-03T09:07:00.401 INFO:teuthology.orchestra.run.smithi088.stderr:2024-06-03T09:07:00.399+0000 7f4f7da3b640 1 -- 172.21.15.88:0/396134345 shutdown_connections 2024-06-03T09:07:00.401 INFO:teuthology.orchestra.run.smithi088.stderr:2024-06-03T09:07:00.399+0000 7f4f7da3b640 1 -- 172.21.15.88:0/396134345 >> 172.21.15.88:0/396134345 conn(0x7f4f780b2f20 msgr2=0x7f4f780b3300 unknown :-1 s=STATE_NONE l=0).mark_down 2024-06-03T09:07:00.401 INFO:teuthology.orchestra.run.smithi088.stderr:2024-06-03T09:07:00.399+0000 7f4f7da3b640 1 -- 172.21.15.88:0/396134345 shutdown_connections 2024-06-03T09:07:00.402 INFO:teuthology.orchestra.run.smithi088.stderr:2024-06-03T09:07:00.399+0000 7f4f7da3b640 1 -- 172.21.15.88:0/396134345 wait complete. 2024-06-03T09:07:00.402 INFO:teuthology.orchestra.run.smithi088.stderr:2024-06-03T09:07:00.399+0000 7f4f7da3b640 1 librados: shutdown 2024-06-03T09:07:00.409 DEBUG:teuthology.orchestra.run.smithi088:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd deep-scrub 5 2024-06-03T09:07:00.512 INFO:teuthology.orchestra.run.smithi088.stderr:2024-06-03T09:07:00.512+0000 7f95d55c5640 10 monclient: get_monmap_and_config 2024-06-03T09:07:00.512 INFO:teuthology.orchestra.run.smithi088.stderr:2024-06-03T09:07:00.512+0000 7f95d55c5640 10 monclient: build_initial_monmap 2024-06-03T09:07:00.512 INFO:teuthology.orchestra.run.smithi088.stderr:2024-06-03T09:07:00.512+0000 7f95d55c5640 10 monclient: monmap: 2024-06-03T09:07:00.512 INFO:teuthology.orchestra.run.smithi088.stderr:epoch 0 2024-06-03T09:07:00.513 INFO:teuthology.orchestra.run.smithi088.stderr:fsid ec3c3fdd-3ac6-4d7d-9412-683130ed8b04 2024-06-03T09:07:00.513 INFO:teuthology.orchestra.run.smithi088.stderr:last_changed 2024-06-03T09:07:00.513236+0000 2024-06-03T09:07:00.513 INFO:teuthology.orchestra.run.smithi088.stderr:created 2024-06-03T09:07:00.513236+0000 2024-06-03T09:07:00.513 INFO:teuthology.orchestra.run.smithi088.stderr:min_mon_release 0 (unknown) 2024-06-03T09:07:00.513 INFO:teuthology.orchestra.run.smithi088.stderr:election_strategy: 3 2024-06-03T09:07:00.513 INFO:teuthology.orchestra.run.smithi088.stderr:0: v1:172.21.15.88:6789/0 mon.noname-a 2024-06-03T09:07:00.513 INFO:teuthology.orchestra.run.smithi088.stderr:1: v1:172.21.15.192:6789/0 mon.noname-c 2024-06-03T09:07:00.513 INFO:teuthology.orchestra.run.smithi088.stderr:2: v1:172.21.15.88:6790/0 mon.noname-b 2024-06-03T09:07:00.513 INFO:teuthology.orchestra.run.smithi088.stderr: 2024-06-03T09:07:00.515 INFO:teuthology.orchestra.run.smithi088.stderr:2024-06-03T09:07:00.512+0000 7f95d55c5640 1 Processor -- start 2024-06-03T09:07:00.515 INFO:teuthology.orchestra.run.smithi088.stderr:2024-06-03T09:07:00.512+0000 7f95d55c5640 1 -- start start 2024-06-03T09:07:00.515 INFO:teuthology.orchestra.run.smithi088.stderr:2024-06-03T09:07:00.512+0000 7f95d55c5640 10 monclient: init 2024-06-03T09:07:00.515 INFO:teuthology.orchestra.run.smithi088.stderr:2024-06-03T09:07:00.513+0000 7f95d55c5640 10 monclient: _reopen_session rank -1 2024-06-03T09:07:00.515 INFO:teuthology.orchestra.run.smithi088.stderr:2024-06-03T09:07:00.513+0000 7f95d55c5640 10 monclient: _add_conns ranks=[0,2,1] 2024-06-03T09:07:00.515 INFO:teuthology.orchestra.run.smithi088.stderr:2024-06-03T09:07:00.513+0000 7f95d55c5640 10 monclient(hunting): picked mon.noname-a con 0x7f95d00a8a60 addr v1:172.21.15.88:6789/0 2024-06-03T09:07:00.515 INFO:teuthology.orchestra.run.smithi088.stderr:2024-06-03T09:07:00.513+0000 7f95d55c5640 10 monclient(hunting): picked mon.noname-b con 0x7f95d00b0d00 addr v1:172.21.15.88:6790/0 2024-06-03T09:07:00.516 INFO:teuthology.orchestra.run.smithi088.stderr:2024-06-03T09:07:00.513+0000 7f95d55c5640 10 monclient(hunting): picked mon.noname-c con 0x7f95d01451d0 addr v1:172.21.15.192:6789/0 2024-06-03T09:07:00.516 INFO:teuthology.orchestra.run.smithi088.stderr:2024-06-03T09:07:00.513+0000 7f95d55c5640 1 -- --> v1:172.21.15.88:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f95d00a9050 con 0x7f95d00a8a60 2024-06-03T09:07:00.516 INFO:teuthology.orchestra.run.smithi088.stderr:2024-06-03T09:07:00.513+0000 7f95d55c5640 1 -- --> v1:172.21.15.192:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f95d00a9d90 con 0x7f95d01451d0 2024-06-03T09:07:00.516 INFO:teuthology.orchestra.run.smithi088.stderr:2024-06-03T09:07:00.513+0000 7f95d55c5640 1 -- --> v1:172.21.15.88:6790/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f95d00a8e40 con 0x7f95d00b0d00 2024-06-03T09:07:00.516 INFO:teuthology.orchestra.run.smithi088.stderr:2024-06-03T09:07:00.513+0000 7f95d55c5640 10 monclient(hunting): _renew_subs 2024-06-03T09:07:00.516 INFO:teuthology.orchestra.run.smithi088.stderr:2024-06-03T09:07:00.513+0000 7f95d55c5640 10 monclient(hunting): authenticate will time out at 3076.323532s 2024-06-03T09:07:00.535 INFO:teuthology.orchestra.run.smithi088.stderr:2024-06-03T09:07:00.535+0000 7f95ceffd640 1 --1- >> v1:172.21.15.88:6789/0 conn(0x7f95d00a8a60 0x7f95d0144ac0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.88:6789/0 says I am v1:172.21.15.88:36310/0 (socket says 172.21.15.88:36310) 2024-06-03T09:07:00.535 INFO:teuthology.orchestra.run.smithi088.stderr:2024-06-03T09:07:00.535+0000 7f95ceffd640 1 -- 172.21.15.88:0/2433877060 learned_addr learned my addr 172.21.15.88:0/2433877060 (peer_addr_for_me v1:172.21.15.88:0/0) 2024-06-03T09:07:00.537 INFO:teuthology.orchestra.run.smithi088.stderr:2024-06-03T09:07:00.537+0000 7f95cdffb640 1 -- 172.21.15.88:0/2433877060 <== mon.0 v1:172.21.15.88:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 2518624675 0 0) 0x7f95d00a9050 con 0x7f95d00a8a60 2024-06-03T09:07:00.537 INFO:teuthology.orchestra.run.smithi088.stderr:2024-06-03T09:07:00.537+0000 7f95cdffb640 10 monclient(hunting): _negotiate 2024-06-03T09:07:00.537 INFO:teuthology.orchestra.run.smithi088.stderr:2024-06-03T09:07:00.537+0000 7f95cdffb640 10 monclient(hunting): _init_auth method 2 2024-06-03T09:07:00.537 INFO:teuthology.orchestra.run.smithi088.stderr:2024-06-03T09:07:00.537+0000 7f95cdffb640 10 monclient(hunting): _init_auth creating new auth 2024-06-03T09:07:00.537 INFO:teuthology.orchestra.run.smithi088.stderr:2024-06-03T09:07:00.537+0000 7f95cdffb640 10 monclient(hunting): my global_id is 30801 2024-06-03T09:07:00.537 INFO:teuthology.orchestra.run.smithi088.stderr:2024-06-03T09:07:00.537+0000 7f95cdffb640 1 -- 172.21.15.88:0/2433877060 --> v1:172.21.15.88:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f95b8003520 con 0x7f95d00a8a60 2024-06-03T09:07:00.541 INFO:teuthology.orchestra.run.smithi088.stderr:2024-06-03T09:07:00.540+0000 7f95cdffb640 1 -- 172.21.15.88:0/2433877060 <== mon.0 v1:172.21.15.88:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 214+0+0 (unknown 2031874751 0 0) 0x7f95b8003520 con 0x7f95d00a8a60 2024-06-03T09:07:00.541 INFO:teuthology.orchestra.run.smithi088.stderr:2024-06-03T09:07:00.540+0000 7f95cdffb640 1 -- 172.21.15.88:0/2433877060 >> v1:172.21.15.88:6790/0 conn(0x7f95d00b0d00 legacy=0x7f95d013e360 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-06-03T09:07:00.541 INFO:teuthology.orchestra.run.smithi088.stderr:2024-06-03T09:07:00.540+0000 7f95cdffb640 1 -- 172.21.15.88:0/2433877060 >> v1:172.21.15.192:6789/0 conn(0x7f95d01451d0 legacy=0x7f95d013ea70 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-06-03T09:07:00.541 INFO:teuthology.orchestra.run.smithi088.stderr:2024-06-03T09:07:00.540+0000 7f95cdffb640 10 monclient: _finish_hunting 0 2024-06-03T09:07:00.541 INFO:teuthology.orchestra.run.smithi088.stderr:2024-06-03T09:07:00.540+0000 7f95cdffb640 1 monclient: found mon.noname-a 2024-06-03T09:07:00.541 INFO:teuthology.orchestra.run.smithi088.stderr:2024-06-03T09:07:00.540+0000 7f95cdffb640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-06-03T09:07:00.541 INFO:teuthology.orchestra.run.smithi088.stderr:2024-06-03T09:07:00.540+0000 7f95cdffb640 10 monclient: _send_mon_message to mon.noname-a at v1:172.21.15.88:6789/0 2024-06-03T09:07:00.541 INFO:teuthology.orchestra.run.smithi088.stderr:2024-06-03T09:07:00.540+0000 7f95cdffb640 1 -- 172.21.15.88:0/2433877060 --> v1:172.21.15.88:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f95d013f180 con 0x7f95d00a8a60 2024-06-03T09:07:00.541 INFO:teuthology.orchestra.run.smithi088.stderr:2024-06-03T09:07:00.540+0000 7f95cdffb640 10 monclient: _finish_auth 0 2024-06-03T09:07:00.541 INFO:teuthology.orchestra.run.smithi088.stderr:2024-06-03T09:07:00.540+0000 7f95cdffb640 10 monclient: _check_auth_tickets 2024-06-03T09:07:00.541 INFO:teuthology.orchestra.run.smithi088.stderr:2024-06-03T09:07:00.540+0000 7f95cdffb640 20 monclient: _check_auth_rotating not needed by client.admin 2024-06-03T09:07:00.542 INFO:teuthology.orchestra.run.smithi088.stderr:2024-06-03T09:07:00.540+0000 7f95cdffb640 1 -- 172.21.15.88:0/2433877060 <== mon.0 v1:172.21.15.88:6789/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 2234374065 0 0) 0x7f95c0001610 con 0x7f95d00a8a60 2024-06-03T09:07:00.542 INFO:teuthology.orchestra.run.smithi088.stderr:2024-06-03T09:07:00.540+0000 7f95d55c5640 5 monclient: authenticate success, global_id 30801 2024-06-03T09:07:00.542 INFO:teuthology.orchestra.run.smithi088.stderr:2024-06-03T09:07:00.540+0000 7f95d55c5640 20 monclient: get_monmap_and_config waiting for monmap|config 2024-06-03T09:07:00.542 INFO:teuthology.orchestra.run.smithi088.stderr:2024-06-03T09:07:00.540+0000 7f95cdffb640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-06-03T09:07:00.542 INFO:teuthology.orchestra.run.smithi088.stderr:2024-06-03T09:07:00.540+0000 7f95cdffb640 10 monclient: got monmap 1 from mon.noname-a (according to old e1) 2024-06-03T09:07:00.542 INFO:teuthology.orchestra.run.smithi088.stderr:2024-06-03T09:07:00.540+0000 7f95cdffb640 10 monclient: dump: 2024-06-03T09:07:00.542 INFO:teuthology.orchestra.run.smithi088.stderr:epoch 1 2024-06-03T09:07:00.542 INFO:teuthology.orchestra.run.smithi088.stderr:fsid ec3c3fdd-3ac6-4d7d-9412-683130ed8b04 2024-06-03T09:07:00.542 INFO:teuthology.orchestra.run.smithi088.stderr:last_changed 2024-06-03T08:34:31.885125+0000 2024-06-03T09:07:00.542 INFO:teuthology.orchestra.run.smithi088.stderr:created 2024-06-03T08:34:31.885125+0000 2024-06-03T09:07:00.542 INFO:teuthology.orchestra.run.smithi088.stderr:min_mon_release 18 (reef) 2024-06-03T09:07:00.542 INFO:teuthology.orchestra.run.smithi088.stderr:election_strategy: 3 2024-06-03T09:07:00.543 INFO:teuthology.orchestra.run.smithi088.stderr:0: v1:172.21.15.88:6789/0 mon.a 2024-06-03T09:07:00.543 INFO:teuthology.orchestra.run.smithi088.stderr:1: v1:172.21.15.192:6789/0 mon.b 2024-06-03T09:07:00.543 INFO:teuthology.orchestra.run.smithi088.stderr:2: v1:172.21.15.88:6790/0 mon.c 2024-06-03T09:07:00.543 INFO:teuthology.orchestra.run.smithi088.stderr: 2024-06-03T09:07:00.543 INFO:teuthology.orchestra.run.smithi088.stderr:2024-06-03T09:07:00.540+0000 7f95d55c5640 20 monclient: get_monmap_and_config waiting for monmap|config 2024-06-03T09:07:00.545 INFO:teuthology.orchestra.run.smithi088.stderr:2024-06-03T09:07:00.545+0000 7f95cdffb640 1 -- 172.21.15.88:0/2433877060 <== mon.0 v1:172.21.15.88:6789/0 4 ==== config(0 keys) v1 ==== 4+0+0 (unknown 0 0 0) 0x7f95c00018c0 con 0x7f95d00a8a60 2024-06-03T09:07:00.545 INFO:teuthology.orchestra.run.smithi088.stderr:2024-06-03T09:07:00.545+0000 7f95cdffb640 10 monclient: handle_config config(0 keys) v1 2024-06-03T09:07:00.545 INFO:teuthology.orchestra.run.smithi088.stderr:2024-06-03T09:07:00.545+0000 7f95cdffb640 1 -- 172.21.15.88:0/2433877060 <== mon.0 v1:172.21.15.88:6789/0 5 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 2234374065 0 0) 0x7f95c0001c10 con 0x7f95d00a8a60 2024-06-03T09:07:00.545 INFO:teuthology.orchestra.run.smithi088.stderr:2024-06-03T09:07:00.545+0000 7f95cdffb640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-06-03T09:07:00.545 INFO:teuthology.orchestra.run.smithi088.stderr:2024-06-03T09:07:00.545+0000 7f95cdffb640 10 monclient: got monmap 1 from mon.a (according to old e1) 2024-06-03T09:07:00.545 INFO:teuthology.orchestra.run.smithi088.stderr:2024-06-03T09:07:00.545+0000 7f95cdffb640 10 monclient: dump: 2024-06-03T09:07:00.545 INFO:teuthology.orchestra.run.smithi088.stderr:epoch 1 2024-06-03T09:07:00.546 INFO:teuthology.orchestra.run.smithi088.stderr:fsid ec3c3fdd-3ac6-4d7d-9412-683130ed8b04 2024-06-03T09:07:00.546 INFO:teuthology.orchestra.run.smithi088.stderr:last_changed 2024-06-03T08:34:31.885125+0000 2024-06-03T09:07:00.546 INFO:teuthology.orchestra.run.smithi088.stderr:created 2024-06-03T08:34:31.885125+0000 2024-06-03T09:07:00.546 INFO:teuthology.orchestra.run.smithi088.stderr:min_mon_release 18 (reef) 2024-06-03T09:07:00.546 INFO:teuthology.orchestra.run.smithi088.stderr:election_strategy: 3 2024-06-03T09:07:00.546 INFO:teuthology.orchestra.run.smithi088.stderr:0: v1:172.21.15.88:6789/0 mon.a 2024-06-03T09:07:00.546 INFO:teuthology.orchestra.run.smithi088.stderr:1: v1:172.21.15.192:6789/0 mon.b 2024-06-03T09:07:00.546 INFO:teuthology.orchestra.run.smithi088.stderr:2: v1:172.21.15.88:6790/0 mon.c 2024-06-03T09:07:00.546 INFO:teuthology.orchestra.run.smithi088.stderr: 2024-06-03T09:07:00.546 INFO:teuthology.orchestra.run.smithi088.stderr:2024-06-03T09:07:00.545+0000 7f95d55c5640 10 monclient: get_monmap_and_config success 2024-06-03T09:07:00.546 INFO:teuthology.orchestra.run.smithi088.stderr:2024-06-03T09:07:00.545+0000 7f95d55c5640 4 set_mon_vals no callback set 2024-06-03T09:07:00.549 INFO:teuthology.orchestra.run.smithi088.stderr:2024-06-03T09:07:00.546+0000 7f95d55c5640 10 monclient: shutdown 2024-06-03T09:07:00.549 INFO:teuthology.orchestra.run.smithi088.stderr:2024-06-03T09:07:00.546+0000 7f95d55c5640 20 monclient: shutdown discarding 0 pending message(s) 2024-06-03T09:07:00.549 INFO:teuthology.orchestra.run.smithi088.stderr:2024-06-03T09:07:00.546+0000 7f95d55c5640 1 -- 172.21.15.88:0/2433877060 >> v1:172.21.15.88:6789/0 conn(0x7f95d00a8a60 legacy=0x7f95d0144ac0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-06-03T09:07:00.549 INFO:teuthology.orchestra.run.smithi088.stderr:2024-06-03T09:07:00.546+0000 7f95d55c5640 1 -- 172.21.15.88:0/2433877060 shutdown_connections 2024-06-03T09:07:00.549 INFO:teuthology.orchestra.run.smithi088.stderr:2024-06-03T09:07:00.546+0000 7f95d55c5640 1 -- 172.21.15.88:0/2433877060 >> 172.21.15.88:0/2433877060 conn(0x7f95d00b70d0 msgr2=0x7f95d00b74e0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-06-03T09:07:00.549 INFO:teuthology.orchestra.run.smithi088.stderr:2024-06-03T09:07:00.546+0000 7f95d55c5640 1 -- 172.21.15.88:0/2433877060 shutdown_connections 2024-06-03T09:07:00.549 INFO:teuthology.orchestra.run.smithi088.stderr:2024-06-03T09:07:00.546+0000 7f95d55c5640 1 -- 172.21.15.88:0/2433877060 wait complete. 2024-06-03T09:07:00.549 INFO:teuthology.orchestra.run.smithi088.stderr:2024-06-03T09:07:00.547+0000 7f95d55c5640 10 monclient: build_initial_monmap 2024-06-03T09:07:00.549 INFO:teuthology.orchestra.run.smithi088.stderr:2024-06-03T09:07:00.547+0000 7f95d55c5640 10 monclient: monmap: 2024-06-03T09:07:00.549 INFO:teuthology.orchestra.run.smithi088.stderr:epoch 0 2024-06-03T09:07:00.549 INFO:teuthology.orchestra.run.smithi088.stderr:fsid 00000000-0000-0000-0000-000000000000 2024-06-03T09:07:00.550 INFO:teuthology.orchestra.run.smithi088.stderr:last_changed 0.000000 2024-06-03T09:07:00.550 INFO:teuthology.orchestra.run.smithi088.stderr:created 0.000000 2024-06-03T09:07:00.550 INFO:teuthology.orchestra.run.smithi088.stderr:min_mon_release 0 (unknown) 2024-06-03T09:07:00.550 INFO:teuthology.orchestra.run.smithi088.stderr:election_strategy: 1 2024-06-03T09:07:00.550 INFO:teuthology.orchestra.run.smithi088.stderr:0: v1:172.21.15.88:6789/0 mon.noname-a 2024-06-03T09:07:00.550 INFO:teuthology.orchestra.run.smithi088.stderr:1: v1:172.21.15.192:6789/0 mon.noname-b 2024-06-03T09:07:00.550 INFO:teuthology.orchestra.run.smithi088.stderr:2: v1:172.21.15.88:6790/0 mon.noname-c 2024-06-03T09:07:00.550 INFO:teuthology.orchestra.run.smithi088.stderr: 2024-06-03T09:07:00.550 INFO:teuthology.orchestra.run.smithi088.stderr:2024-06-03T09:07:00.547+0000 7f95d55c5640 1 librados: starting msgr at 2024-06-03T09:07:00.550 INFO:teuthology.orchestra.run.smithi088.stderr:2024-06-03T09:07:00.547+0000 7f95d55c5640 1 librados: starting objecter 2024-06-03T09:07:00.550 INFO:teuthology.orchestra.run.smithi088.stderr:2024-06-03T09:07:00.547+0000 7f95d55c5640 1 Processor -- start 2024-06-03T09:07:00.553 INFO:teuthology.orchestra.run.smithi088.stderr:2024-06-03T09:07:00.547+0000 7f95d55c5640 1 -- start start 2024-06-03T09:07:00.553 INFO:teuthology.orchestra.run.smithi088.stderr:2024-06-03T09:07:00.547+0000 7f95d55c5640 1 librados: setting wanted keys 2024-06-03T09:07:00.553 INFO:teuthology.orchestra.run.smithi088.stderr:2024-06-03T09:07:00.547+0000 7f95d55c5640 1 librados: calling monclient init 2024-06-03T09:07:00.554 INFO:teuthology.orchestra.run.smithi088.stderr:2024-06-03T09:07:00.547+0000 7f95d55c5640 10 monclient: init 2024-06-03T09:07:00.554 INFO:teuthology.orchestra.run.smithi088.stderr:2024-06-03T09:07:00.548+0000 7f95d55c5640 10 monclient: _reopen_session rank -1 2024-06-03T09:07:00.554 INFO:teuthology.orchestra.run.smithi088.stderr:2024-06-03T09:07:00.548+0000 7f95d55c5640 10 monclient: _add_conns ranks=[0,2,1] 2024-06-03T09:07:00.554 INFO:teuthology.orchestra.run.smithi088.stderr:2024-06-03T09:07:00.548+0000 7f95d55c5640 10 monclient(hunting): picked mon.noname-a con 0x7f95d00a8a60 addr v1:172.21.15.88:6789/0 2024-06-03T09:07:00.554 INFO:teuthology.orchestra.run.smithi088.stderr:2024-06-03T09:07:00.548+0000 7f95d55c5640 10 monclient(hunting): picked mon.noname-c con 0x7f95d00b0d00 addr v1:172.21.15.88:6790/0 2024-06-03T09:07:00.554 INFO:teuthology.orchestra.run.smithi088.stderr:2024-06-03T09:07:00.548+0000 7f95d55c5640 10 monclient(hunting): picked mon.noname-b con 0x7f95d01451d0 addr v1:172.21.15.192:6789/0 2024-06-03T09:07:00.554 INFO:teuthology.orchestra.run.smithi088.stderr:2024-06-03T09:07:00.548+0000 7f95d55c5640 1 -- --> v1:172.21.15.88:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f95d01daf90 con 0x7f95d00a8a60 2024-06-03T09:07:00.554 INFO:teuthology.orchestra.run.smithi088.stderr:2024-06-03T09:07:00.548+0000 7f95d55c5640 1 -- --> v1:172.21.15.192:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f95d013ac40 con 0x7f95d01451d0 2024-06-03T09:07:00.554 INFO:teuthology.orchestra.run.smithi088.stderr:2024-06-03T09:07:00.548+0000 7f95d55c5640 1 -- --> v1:172.21.15.88:6790/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f95d013aee0 con 0x7f95d00b0d00 2024-06-03T09:07:00.554 INFO:teuthology.orchestra.run.smithi088.stderr:2024-06-03T09:07:00.548+0000 7f95d55c5640 10 monclient(hunting): _renew_subs 2024-06-03T09:07:00.554 INFO:teuthology.orchestra.run.smithi088.stderr:2024-06-03T09:07:00.548+0000 7f95d55c5640 10 monclient(hunting): authenticate will time out at 3076.358384s 2024-06-03T09:07:00.554 INFO:teuthology.orchestra.run.smithi088.stderr:2024-06-03T09:07:00.548+0000 7f95ce7fc640 1 --1- >> v1:172.21.15.88:6790/0 conn(0x7f95d00b0d00 0x7f95d01da170 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=1).handle_server_banner_and_identify peer v1:172.21.15.88:6790/0 says I am v1:172.21.15.88:51146/0 (socket says 172.21.15.88:51146) 2024-06-03T09:07:00.554 INFO:teuthology.orchestra.run.smithi088.stderr:2024-06-03T09:07:00.548+0000 7f95ce7fc640 1 -- 172.21.15.88:0/1654250444 learned_addr learned my addr 172.21.15.88:0/1654250444 (peer_addr_for_me v1:172.21.15.88:0/0) 2024-06-03T09:07:00.555 INFO:teuthology.orchestra.run.smithi088.stderr:2024-06-03T09:07:00.549+0000 7f95b77fe640 10 client.?.objecter ms_handle_connect 0x7f95d00b0d00 2024-06-03T09:07:00.555 INFO:teuthology.orchestra.run.smithi088.stderr:2024-06-03T09:07:00.549+0000 7f95b77fe640 10 client.?.objecter resend_mon_ops 2024-06-03T09:07:00.555 INFO:teuthology.orchestra.run.smithi088.stderr:2024-06-03T09:07:00.549+0000 7f95b77fe640 10 client.?.objecter ms_handle_connect 0x7f95d01451d0 2024-06-03T09:07:00.555 INFO:teuthology.orchestra.run.smithi088.stderr:2024-06-03T09:07:00.549+0000 7f95b77fe640 10 client.?.objecter resend_mon_ops 2024-06-03T09:07:00.555 INFO:teuthology.orchestra.run.smithi088.stderr:2024-06-03T09:07:00.549+0000 7f95b77fe640 10 client.?.objecter ms_handle_connect 0x7f95d00a8a60 2024-06-03T09:07:00.555 INFO:teuthology.orchestra.run.smithi088.stderr:2024-06-03T09:07:00.549+0000 7f95b77fe640 10 client.?.objecter resend_mon_ops 2024-06-03T09:07:00.556 INFO:teuthology.orchestra.run.smithi088.stderr:2024-06-03T09:07:00.551+0000 7f95b77fe640 1 -- 172.21.15.88:0/1654250444 <== mon.0 v1:172.21.15.88:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 1971363289 0 0) 0x7f95d01daf90 con 0x7f95d00a8a60 2024-06-03T09:07:00.556 INFO:teuthology.orchestra.run.smithi088.stderr:2024-06-03T09:07:00.551+0000 7f95b77fe640 10 monclient(hunting): _negotiate 2024-06-03T09:07:00.556 INFO:teuthology.orchestra.run.smithi088.stderr:2024-06-03T09:07:00.551+0000 7f95b77fe640 10 monclient(hunting): _init_auth method 2 2024-06-03T09:07:00.556 INFO:teuthology.orchestra.run.smithi088.stderr:2024-06-03T09:07:00.551+0000 7f95b77fe640 10 monclient(hunting): _init_auth creating new auth 2024-06-03T09:07:00.557 INFO:teuthology.orchestra.run.smithi088.stderr:2024-06-03T09:07:00.551+0000 7f95b77fe640 10 monclient(hunting): my global_id is 30804 2024-06-03T09:07:00.557 INFO:teuthology.orchestra.run.smithi088.stderr:2024-06-03T09:07:00.551+0000 7f95b77fe640 1 -- 172.21.15.88:0/1654250444 --> v1:172.21.15.88:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f95bc004eb0 con 0x7f95d00a8a60 2024-06-03T09:07:00.557 INFO:teuthology.orchestra.run.smithi088.stderr:2024-06-03T09:07:00.551+0000 7f95b77fe640 1 -- 172.21.15.88:0/1654250444 <== mon.2 v1:172.21.15.88:6790/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 3065413924 0 0) 0x7f95d013aee0 con 0x7f95d00b0d00 2024-06-03T09:07:00.557 INFO:teuthology.orchestra.run.smithi088.stderr:2024-06-03T09:07:00.551+0000 7f95b77fe640 10 monclient(hunting): _negotiate 2024-06-03T09:07:00.557 INFO:teuthology.orchestra.run.smithi088.stderr:2024-06-03T09:07:00.551+0000 7f95b77fe640 10 monclient(hunting): _init_auth method 2 2024-06-03T09:07:00.557 INFO:teuthology.orchestra.run.smithi088.stderr:2024-06-03T09:07:00.551+0000 7f95b77fe640 10 monclient(hunting): _init_auth creating new auth 2024-06-03T09:07:00.557 INFO:teuthology.orchestra.run.smithi088.stderr:2024-06-03T09:07:00.551+0000 7f95b77fe640 10 monclient(hunting): my global_id is 30308 2024-06-03T09:07:00.557 INFO:teuthology.orchestra.run.smithi088.stderr:2024-06-03T09:07:00.551+0000 7f95b77fe640 1 -- 172.21.15.88:0/1654250444 --> v1:172.21.15.88:6790/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f95d01daf90 con 0x7f95d00b0d00 2024-06-03T09:07:00.557 INFO:teuthology.orchestra.run.smithi088.stderr:2024-06-03T09:07:00.551+0000 7f95b77fe640 1 -- 172.21.15.88:0/1654250444 <== mon.1 v1:172.21.15.192:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 1800896140 0 0) 0x7f95d013ac40 con 0x7f95d01451d0 2024-06-03T09:07:00.557 INFO:teuthology.orchestra.run.smithi088.stderr:2024-06-03T09:07:00.551+0000 7f95b77fe640 10 monclient(hunting): _negotiate 2024-06-03T09:07:00.557 INFO:teuthology.orchestra.run.smithi088.stderr:2024-06-03T09:07:00.551+0000 7f95b77fe640 10 monclient(hunting): _init_auth method 2 2024-06-03T09:07:00.557 INFO:teuthology.orchestra.run.smithi088.stderr:2024-06-03T09:07:00.551+0000 7f95b77fe640 10 monclient(hunting): _init_auth creating new auth 2024-06-03T09:07:00.557 INFO:teuthology.orchestra.run.smithi088.stderr:2024-06-03T09:07:00.551+0000 7f95b77fe640 10 monclient(hunting): my global_id is 30373 2024-06-03T09:07:00.558 INFO:teuthology.orchestra.run.smithi088.stderr:2024-06-03T09:07:00.551+0000 7f95b77fe640 1 -- 172.21.15.88:0/1654250444 --> v1:172.21.15.192:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f95d013aee0 con 0x7f95d01451d0 2024-06-03T09:07:00.558 INFO:teuthology.orchestra.run.smithi088.stderr:2024-06-03T09:07:00.556+0000 7f95b77fe640 1 -- 172.21.15.88:0/1654250444 <== mon.0 v1:172.21.15.88:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 764+0+0 (unknown 1103842658 0 0) 0x7f95bc004eb0 con 0x7f95d00a8a60 2024-06-03T09:07:00.558 INFO:teuthology.orchestra.run.smithi088.stderr:2024-06-03T09:07:00.556+0000 7f95b77fe640 1 -- 172.21.15.88:0/1654250444 --> v1:172.21.15.88:6789/0 -- auth(proto 2 165 bytes epoch 0) v1 -- 0x7f95d013ac40 con 0x7f95d00a8a60 2024-06-03T09:07:00.558 INFO:teuthology.orchestra.run.smithi088.stderr:2024-06-03T09:07:00.556+0000 7f95b77fe640 1 -- 172.21.15.88:0/1654250444 <== mon.0 v1:172.21.15.88:6789/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 2234374065 0 0) 0x7f95c00015d0 con 0x7f95d00a8a60 2024-06-03T09:07:00.558 INFO:teuthology.orchestra.run.smithi088.stderr:2024-06-03T09:07:00.556+0000 7f95b77fe640 10 monclient(hunting): handle_monmap mon_map magic: 0 v1 2024-06-03T09:07:00.558 INFO:teuthology.orchestra.run.smithi088.stderr:2024-06-03T09:07:00.556+0000 7f95b77fe640 10 monclient(hunting): got monmap 1 from mon.noname-a (according to old e1) 2024-06-03T09:07:00.558 INFO:teuthology.orchestra.run.smithi088.stderr:2024-06-03T09:07:00.556+0000 7f95b77fe640 10 monclient(hunting): dump: 2024-06-03T09:07:00.558 INFO:teuthology.orchestra.run.smithi088.stderr:epoch 1 2024-06-03T09:07:00.558 INFO:teuthology.orchestra.run.smithi088.stderr:fsid ec3c3fdd-3ac6-4d7d-9412-683130ed8b04 2024-06-03T09:07:00.558 INFO:teuthology.orchestra.run.smithi088.stderr:last_changed 2024-06-03T08:34:31.885125+0000 2024-06-03T09:07:00.558 INFO:teuthology.orchestra.run.smithi088.stderr:created 2024-06-03T08:34:31.885125+0000 2024-06-03T09:07:00.558 INFO:teuthology.orchestra.run.smithi088.stderr:min_mon_release 18 (reef) 2024-06-03T09:07:00.558 INFO:teuthology.orchestra.run.smithi088.stderr:election_strategy: 3 2024-06-03T09:07:00.558 INFO:teuthology.orchestra.run.smithi088.stderr:0: v1:172.21.15.88:6789/0 mon.a 2024-06-03T09:07:00.559 INFO:teuthology.orchestra.run.smithi088.stderr:1: v1:172.21.15.192:6789/0 mon.b 2024-06-03T09:07:00.559 INFO:teuthology.orchestra.run.smithi088.stderr:2: v1:172.21.15.88:6790/0 mon.c 2024-06-03T09:07:00.559 INFO:teuthology.orchestra.run.smithi088.stderr: 2024-06-03T09:07:00.559 INFO:teuthology.orchestra.run.smithi088.stderr:2024-06-03T09:07:00.556+0000 7f95b77fe640 10 monclient(hunting): _finish_auth 0 2024-06-03T09:07:00.559 INFO:teuthology.orchestra.run.smithi088.stderr:2024-06-03T09:07:00.557+0000 7f95b77fe640 1 -- 172.21.15.88:0/1654250444 <== mon.2 v1:172.21.15.88:6790/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 764+0+0 (unknown 1591011965 0 0) 0x7f95d01daf90 con 0x7f95d00b0d00 2024-06-03T09:07:00.560 INFO:teuthology.orchestra.run.smithi088.stderr:2024-06-03T09:07:00.557+0000 7f95b77fe640 1 -- 172.21.15.88:0/1654250444 --> v1:172.21.15.88:6790/0 -- auth(proto 2 165 bytes epoch 0) v1 -- 0x7f95bc004eb0 con 0x7f95d00b0d00 2024-06-03T09:07:00.560 INFO:teuthology.orchestra.run.smithi088.stderr:2024-06-03T09:07:00.557+0000 7f95b77fe640 1 -- 172.21.15.88:0/1654250444 <== mon.2 v1:172.21.15.88:6790/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 2234374065 0 0) 0x7f95c40018b0 con 0x7f95d00b0d00 2024-06-03T09:07:00.560 INFO:teuthology.orchestra.run.smithi088.stderr:2024-06-03T09:07:00.557+0000 7f95b77fe640 10 monclient(hunting): handle_monmap mon_map magic: 0 v1 2024-06-03T09:07:00.560 INFO:teuthology.orchestra.run.smithi088.stderr:2024-06-03T09:07:00.557+0000 7f95b77fe640 10 monclient(hunting): got monmap 1 from mon.c (according to old e1) 2024-06-03T09:07:00.560 INFO:teuthology.orchestra.run.smithi088.stderr:2024-06-03T09:07:00.557+0000 7f95b77fe640 10 monclient(hunting): dump: 2024-06-03T09:07:00.560 INFO:teuthology.orchestra.run.smithi088.stderr:epoch 1 2024-06-03T09:07:00.560 INFO:teuthology.orchestra.run.smithi088.stderr:fsid ec3c3fdd-3ac6-4d7d-9412-683130ed8b04 2024-06-03T09:07:00.560 INFO:teuthology.orchestra.run.smithi088.stderr:last_changed 2024-06-03T08:34:31.885125+0000 2024-06-03T09:07:00.560 INFO:teuthology.orchestra.run.smithi088.stderr:created 2024-06-03T08:34:31.885125+0000 2024-06-03T09:07:00.560 INFO:teuthology.orchestra.run.smithi088.stderr:min_mon_release 18 (reef) 2024-06-03T09:07:00.560 INFO:teuthology.orchestra.run.smithi088.stderr:election_strategy: 3 2024-06-03T09:07:00.560 INFO:teuthology.orchestra.run.smithi088.stderr:0: v1:172.21.15.88:6789/0 mon.a 2024-06-03T09:07:00.560 INFO:teuthology.orchestra.run.smithi088.stderr:1: v1:172.21.15.192:6789/0 mon.b 2024-06-03T09:07:00.561 INFO:teuthology.orchestra.run.smithi088.stderr:2: v1:172.21.15.88:6790/0 mon.c 2024-06-03T09:07:00.561 INFO:teuthology.orchestra.run.smithi088.stderr: 2024-06-03T09:07:00.561 INFO:teuthology.orchestra.run.smithi088.stderr:2024-06-03T09:07:00.557+0000 7f95b77fe640 1 -- 172.21.15.88:0/1654250444 <== mon.1 v1:172.21.15.192:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 764+0+0 (unknown 1599811680 0 0) 0x7f95d013aee0 con 0x7f95d01451d0 2024-06-03T09:07:00.561 INFO:teuthology.orchestra.run.smithi088.stderr:2024-06-03T09:07:00.557+0000 7f95b77fe640 1 -- 172.21.15.88:0/1654250444 --> v1:172.21.15.192:6789/0 -- auth(proto 2 165 bytes epoch 0) v1 -- 0x7f95d01daf90 con 0x7f95d01451d0 2024-06-03T09:07:00.561 INFO:teuthology.orchestra.run.smithi088.stderr:2024-06-03T09:07:00.557+0000 7f95b77fe640 1 -- 172.21.15.88:0/1654250444 <== mon.1 v1:172.21.15.192:6789/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 2234374065 0 0) 0x7f95c8001a50 con 0x7f95d01451d0 2024-06-03T09:07:00.561 INFO:teuthology.orchestra.run.smithi088.stderr:2024-06-03T09:07:00.557+0000 7f95b77fe640 10 monclient(hunting): handle_monmap mon_map magic: 0 v1 2024-06-03T09:07:00.561 INFO:teuthology.orchestra.run.smithi088.stderr:2024-06-03T09:07:00.557+0000 7f95b77fe640 10 monclient(hunting): got monmap 1 from mon.b (according to old e1) 2024-06-03T09:07:00.561 INFO:teuthology.orchestra.run.smithi088.stderr:2024-06-03T09:07:00.557+0000 7f95b77fe640 10 monclient(hunting): dump: 2024-06-03T09:07:00.561 INFO:teuthology.orchestra.run.smithi088.stderr:epoch 1 2024-06-03T09:07:00.561 INFO:teuthology.orchestra.run.smithi088.stderr:fsid ec3c3fdd-3ac6-4d7d-9412-683130ed8b04 2024-06-03T09:07:00.561 INFO:teuthology.orchestra.run.smithi088.stderr:last_changed 2024-06-03T08:34:31.885125+0000 2024-06-03T09:07:00.561 INFO:teuthology.orchestra.run.smithi088.stderr:created 2024-06-03T08:34:31.885125+0000 2024-06-03T09:07:00.561 INFO:teuthology.orchestra.run.smithi088.stderr:min_mon_release 18 (reef) 2024-06-03T09:07:00.561 INFO:teuthology.orchestra.run.smithi088.stderr:election_strategy: 3 2024-06-03T09:07:00.562 INFO:teuthology.orchestra.run.smithi088.stderr:0: v1:172.21.15.88:6789/0 mon.a 2024-06-03T09:07:00.562 INFO:teuthology.orchestra.run.smithi088.stderr:1: v1:172.21.15.192:6789/0 mon.b 2024-06-03T09:07:00.562 INFO:teuthology.orchestra.run.smithi088.stderr:2: v1:172.21.15.88:6790/0 mon.c 2024-06-03T09:07:00.562 INFO:teuthology.orchestra.run.smithi088.stderr: 2024-06-03T09:07:00.563 INFO:teuthology.orchestra.run.smithi088.stderr:2024-06-03T09:07:00.560+0000 7f95b77fe640 1 -- 172.21.15.88:0/1654250444 <== mon.0 v1:172.21.15.88:6789/0 4 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 580+0+0 (unknown 3379862142 0 0) 0x7f95d013ac40 con 0x7f95d00a8a60 2024-06-03T09:07:00.563 INFO:teuthology.orchestra.run.smithi088.stderr:2024-06-03T09:07:00.560+0000 7f95b77fe640 1 -- 172.21.15.88:0/1654250444 >> v1:172.21.15.88:6790/0 conn(0x7f95d00b0d00 legacy=0x7f95d01da170 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-06-03T09:07:00.563 INFO:teuthology.orchestra.run.smithi088.stderr:2024-06-03T09:07:00.560+0000 7f95b77fe640 1 -- 172.21.15.88:0/1654250444 >> v1:172.21.15.192:6789/0 conn(0x7f95d01451d0 legacy=0x7f95d01da880 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-06-03T09:07:00.563 INFO:teuthology.orchestra.run.smithi088.stderr:2024-06-03T09:07:00.560+0000 7f95b77fe640 10 monclient: _finish_hunting 0 2024-06-03T09:07:00.563 INFO:teuthology.orchestra.run.smithi088.stderr:2024-06-03T09:07:00.560+0000 7f95b77fe640 1 monclient: found mon.a 2024-06-03T09:07:00.563 INFO:teuthology.orchestra.run.smithi088.stderr:2024-06-03T09:07:00.560+0000 7f95b77fe640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-06-03T09:07:00.563 INFO:teuthology.orchestra.run.smithi088.stderr:2024-06-03T09:07:00.560+0000 7f95b77fe640 10 monclient: _send_mon_message to mon.a at v1:172.21.15.88:6789/0 2024-06-03T09:07:00.563 INFO:teuthology.orchestra.run.smithi088.stderr:2024-06-03T09:07:00.560+0000 7f95b77fe640 1 -- 172.21.15.88:0/1654250444 --> v1:172.21.15.88:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f95d00aa9a0 con 0x7f95d00a8a60 2024-06-03T09:07:00.564 INFO:teuthology.orchestra.run.smithi088.stderr:2024-06-03T09:07:00.560+0000 7f95b77fe640 10 monclient: _finish_auth 0 2024-06-03T09:07:00.564 INFO:teuthology.orchestra.run.smithi088.stderr:2024-06-03T09:07:00.560+0000 7f95b77fe640 10 monclient: _check_auth_tickets 2024-06-03T09:07:00.564 INFO:teuthology.orchestra.run.smithi088.stderr:2024-06-03T09:07:00.560+0000 7f95b77fe640 20 monclient: _check_auth_rotating not needed by client.admin 2024-06-03T09:07:00.564 INFO:teuthology.orchestra.run.smithi088.stderr:2024-06-03T09:07:00.560+0000 7f95d55c5640 5 monclient: authenticate success, global_id 30804 2024-06-03T09:07:00.564 INFO:teuthology.orchestra.run.smithi088.stderr:2024-06-03T09:07:00.560+0000 7f95d55c5640 10 monclient: _renew_subs 2024-06-03T09:07:00.564 INFO:teuthology.orchestra.run.smithi088.stderr:2024-06-03T09:07:00.560+0000 7f95d55c5640 10 monclient: _send_mon_message to mon.a at v1:172.21.15.88:6789/0 2024-06-03T09:07:00.564 INFO:teuthology.orchestra.run.smithi088.stderr:2024-06-03T09:07:00.560+0000 7f95d55c5640 1 -- 172.21.15.88:0/1654250444 --> v1:172.21.15.88:6789/0 -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f95d013a820 con 0x7f95d00a8a60 2024-06-03T09:07:00.564 INFO:teuthology.orchestra.run.smithi088.stderr:2024-06-03T09:07:00.560+0000 7f95d55c5640 10 client.30804.objecter _maybe_request_map subscribing (onetime) to next osd map 2024-06-03T09:07:00.564 INFO:teuthology.orchestra.run.smithi088.stderr:2024-06-03T09:07:00.560+0000 7f95d55c5640 10 monclient: _renew_subs 2024-06-03T09:07:00.564 INFO:teuthology.orchestra.run.smithi088.stderr:2024-06-03T09:07:00.560+0000 7f95d55c5640 10 monclient: _send_mon_message to mon.a at v1:172.21.15.88:6789/0 2024-06-03T09:07:00.564 INFO:teuthology.orchestra.run.smithi088.stderr:2024-06-03T09:07:00.560+0000 7f95d55c5640 1 -- 172.21.15.88:0/1654250444 --> v1:172.21.15.88:6789/0 -- mon_subscribe({osdmap=0}) v3 -- 0x7f95d0138150 con 0x7f95d00a8a60 2024-06-03T09:07:00.564 INFO:teuthology.orchestra.run.smithi088.stderr:2024-06-03T09:07:00.560+0000 7f95d55c5640 1 librados: init done 2024-06-03T09:07:00.564 INFO:teuthology.orchestra.run.smithi088.stderr:2024-06-03T09:07:00.561+0000 7f95d55c5640 10 start_mon_command cmd=[{"prefix": "get_command_descriptions"}] 2024-06-03T09:07:00.564 INFO:teuthology.orchestra.run.smithi088.stderr:2024-06-03T09:07:00.561+0000 7f95d55c5640 10 monclient: _send_command 1 [{"prefix": "get_command_descriptions"}] 2024-06-03T09:07:00.565 INFO:teuthology.orchestra.run.smithi088.stderr:2024-06-03T09:07:00.561+0000 7f95d55c5640 10 monclient: _send_mon_message to mon.a at v1:172.21.15.88:6789/0 2024-06-03T09:07:00.565 INFO:teuthology.orchestra.run.smithi088.stderr:2024-06-03T09:07:00.561+0000 7f95d55c5640 1 -- 172.21.15.88:0/1654250444 --> v1:172.21.15.88:6789/0 -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f95d013a640 con 0x7f95d00a8a60 2024-06-03T09:07:00.576 INFO:teuthology.orchestra.run.smithi088.stderr:2024-06-03T09:07:00.576+0000 7f95b77fe640 1 -- 172.21.15.88:0/1654250444 <== mon.0 v1:172.21.15.88:6789/0 5 ==== config(0 keys) v1 ==== 4+0+0 (unknown 0 0 0) 0x7f95c0004a90 con 0x7f95d00a8a60 2024-06-03T09:07:00.576 INFO:teuthology.orchestra.run.smithi088.stderr:2024-06-03T09:07:00.576+0000 7f95b77fe640 10 monclient: handle_config config(0 keys) v1 2024-06-03T09:07:00.576 INFO:teuthology.orchestra.run.smithi088.stderr:2024-06-03T09:07:00.576+0000 7f95b77fe640 1 -- 172.21.15.88:0/1654250444 <== mon.0 v1:172.21.15.88:6789/0 6 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 2234374065 0 0) 0x7f95c0004da0 con 0x7f95d00a8a60 2024-06-03T09:07:00.576 INFO:teuthology.orchestra.run.smithi088.stderr:2024-06-03T09:07:00.576+0000 7f95b77fe640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-06-03T09:07:00.576 INFO:teuthology.orchestra.run.smithi088.stderr:2024-06-03T09:07:00.576+0000 7f95b77fe640 10 monclient: got monmap 1 from mon.a (according to old e1) 2024-06-03T09:07:00.576 INFO:teuthology.orchestra.run.smithi088.stderr:2024-06-03T09:07:00.576+0000 7f95b77fe640 10 monclient: dump: 2024-06-03T09:07:00.577 INFO:teuthology.orchestra.run.smithi088.stderr:epoch 1 2024-06-03T09:07:00.577 INFO:teuthology.orchestra.run.smithi088.stderr:fsid ec3c3fdd-3ac6-4d7d-9412-683130ed8b04 2024-06-03T09:07:00.577 INFO:teuthology.orchestra.run.smithi088.stderr:last_changed 2024-06-03T08:34:31.885125+0000 2024-06-03T09:07:00.577 INFO:teuthology.orchestra.run.smithi088.stderr:created 2024-06-03T08:34:31.885125+0000 2024-06-03T09:07:00.577 INFO:teuthology.orchestra.run.smithi088.stderr:min_mon_release 18 (reef) 2024-06-03T09:07:00.577 INFO:teuthology.orchestra.run.smithi088.stderr:election_strategy: 3 2024-06-03T09:07:00.577 INFO:teuthology.orchestra.run.smithi088.stderr:0: v1:172.21.15.88:6789/0 mon.a 2024-06-03T09:07:00.577 INFO:teuthology.orchestra.run.smithi088.stderr:1: v1:172.21.15.192:6789/0 mon.b 2024-06-03T09:07:00.577 INFO:teuthology.orchestra.run.smithi088.stderr:2: v1:172.21.15.88:6790/0 mon.c 2024-06-03T09:07:00.577 INFO:teuthology.orchestra.run.smithi088.stderr: 2024-06-03T09:07:00.577 INFO:teuthology.orchestra.run.smithi088.stderr:2024-06-03T09:07:00.576+0000 7f95cd7fa640 4 set_mon_vals no callback set 2024-06-03T09:07:00.578 INFO:teuthology.orchestra.run.smithi088.stderr:2024-06-03T09:07:00.577+0000 7f95b77fe640 1 -- 172.21.15.88:0/1654250444 <== mon.0 v1:172.21.15.88:6789/0 7 ==== mgrmap(e 5) v1 ==== 89110+0+0 (unknown 873533973 0 0) 0x7f95c001ab20 con 0x7f95d00a8a60 2024-06-03T09:07:00.578 INFO:teuthology.orchestra.run.smithi088.stderr:2024-06-03T09:07:00.578+0000 7f95ce7fc640 10 monclient: get_auth_request con 0x7f95bc06d400 auth_method 0 2024-06-03T09:07:00.578 INFO:teuthology.orchestra.run.smithi088.stderr:2024-06-03T09:07:00.578+0000 7f95b77fe640 10 client.30804.objecter ms_handle_connect 0x7f95bc06d400 2024-06-03T09:07:00.579 INFO:teuthology.orchestra.run.smithi088.stderr:2024-06-03T09:07:00.579+0000 7f95b77fe640 1 -- 172.21.15.88:0/1654250444 <== mon.0 v1:172.21.15.88:6789/0 8 ==== osd_map(1014..1014 src has 622..1014) v4 ==== 5229+0+0 (unknown 3230192161 0 0) 0x7f95c00541e0 con 0x7f95d00a8a60 2024-06-03T09:07:00.579 INFO:teuthology.orchestra.run.smithi088.stderr:2024-06-03T09:07:00.579+0000 7f95b77fe640 10 client.30804.objecter ms_dispatch 0x7f95d0000b90 osd_map(1014..1014 src has 622..1014) v4 2024-06-03T09:07:00.579 INFO:teuthology.orchestra.run.smithi088.stderr:2024-06-03T09:07:00.579+0000 7f95b77fe640 3 client.30804.objecter handle_osd_map got epochs [1014,1014] > 0 2024-06-03T09:07:00.579 INFO:teuthology.orchestra.run.smithi088.stderr:2024-06-03T09:07:00.579+0000 7f95b77fe640 3 client.30804.objecter handle_osd_map decoding full epoch 1014 2024-06-03T09:07:00.579 INFO:teuthology.orchestra.run.smithi088.stderr:2024-06-03T09:07:00.579+0000 7f95b77fe640 20 client.30804.objecter dump_active .. 0 homeless 2024-06-03T09:07:00.809 INFO:teuthology.orchestra.run.smithi088.stderr:2024-06-03T09:07:00.808+0000 7f95b77fe640 1 -- 172.21.15.88:0/1654250444 <== mon.0 v1:172.21.15.88:6789/0 9 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+182498 (unknown 1092875540 0 3016760700) 0x7f95c004fd90 con 0x7f95d00a8a60 2024-06-03T09:07:00.809 INFO:teuthology.orchestra.run.smithi088.stderr:2024-06-03T09:07:00.809+0000 7f95b77fe640 10 monclient: handle_mon_command_ack 1 [{"prefix": "get_command_descriptions"}] 2024-06-03T09:07:00.809 INFO:teuthology.orchestra.run.smithi088.stderr:2024-06-03T09:07:00.809+0000 7f95b77fe640 10 monclient: _finish_command 1 = system:0 2024-06-03T09:07:00.992 INFO:teuthology.orchestra.run.smithi088.stderr:2024-06-03T09:07:00.991+0000 7f95d55c5640 1 -- 172.21.15.88:0/1654250444 --> v1:172.21.15.192:6800/34229 -- mgr_command(tid 0: {"prefix": "osd deep-scrub", "who": "5", "target": ["mon-mgr", ""]}) v1 -- 0x7f95d00a94e0 con 0x7f95bc06d400 2024-06-03T09:07:00.992 INFO:teuthology.orchestra.run.smithi088.stderr:2024-06-03T09:07:00.992+0000 7f95b77fe640 1 -- 172.21.15.88:0/1654250444 <== mgr.4110 v1:172.21.15.192:6800/34229 1 ==== mgr_command_reply(tid 0: 0 instructed osd(s) 5 to deep-scrub) v1 ==== 41+0+0 (unknown 2264484800 0 0) 0x7f95c40057c0 con 0x7f95bc06d400 2024-06-03T09:07:00.992 INFO:teuthology.orchestra.run.smithi088.stderr:instructed osd(s) 5 to deep-scrub 2024-06-03T09:07:00.995 INFO:teuthology.orchestra.run.smithi088.stderr:2024-06-03T09:07:00.994+0000 7f95d55c5640 10 librados: watch_flush enter 2024-06-03T09:07:00.995 INFO:teuthology.orchestra.run.smithi088.stderr:2024-06-03T09:07:00.994+0000 7f95d55c5640 10 librados: watch_flush exit 2024-06-03T09:07:00.995 INFO:teuthology.orchestra.run.smithi088.stderr:2024-06-03T09:07:00.994+0000 7f95d55c5640 20 client.30804.objecter shutdown clearing up homeless session... 2024-06-03T09:07:00.995 INFO:teuthology.orchestra.run.smithi088.stderr:2024-06-03T09:07:00.994+0000 7f95d55c5640 10 client.30804.objecter successfully canceled tick 2024-06-03T09:07:00.995 INFO:teuthology.orchestra.run.smithi088.stderr:2024-06-03T09:07:00.994+0000 7f95d55c5640 1 -- 172.21.15.88:0/1654250444 >> v1:172.21.15.192:6800/34229 conn(0x7f95bc06d400 legacy=0x7f95bc06f8c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-06-03T09:07:00.995 INFO:teuthology.orchestra.run.smithi088.stderr:2024-06-03T09:07:00.994+0000 7f95d55c5640 10 monclient: shutdown 2024-06-03T09:07:00.995 INFO:teuthology.orchestra.run.smithi088.stderr:2024-06-03T09:07:00.994+0000 7f95d55c5640 20 monclient: shutdown discarding 0 pending message(s) 2024-06-03T09:07:00.995 INFO:teuthology.orchestra.run.smithi088.stderr:2024-06-03T09:07:00.994+0000 7f95d55c5640 1 -- 172.21.15.88:0/1654250444 >> v1:172.21.15.88:6789/0 conn(0x7f95d00a8a60 legacy=0x7f95d0138c80 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-06-03T09:07:00.995 INFO:teuthology.orchestra.run.smithi088.stderr:2024-06-03T09:07:00.994+0000 7f95d55c5640 1 -- 172.21.15.88:0/1654250444 shutdown_connections 2024-06-03T09:07:00.996 INFO:teuthology.orchestra.run.smithi088.stderr:2024-06-03T09:07:00.994+0000 7f95d55c5640 1 -- 172.21.15.88:0/1654250444 >> 172.21.15.88:0/1654250444 conn(0x7f95d00b70d0 msgr2=0x7f95d00b5e30 unknown :-1 s=STATE_NONE l=0).mark_down 2024-06-03T09:07:00.996 INFO:teuthology.orchestra.run.smithi088.stderr:2024-06-03T09:07:00.994+0000 7f95d55c5640 1 -- 172.21.15.88:0/1654250444 shutdown_connections 2024-06-03T09:07:00.996 INFO:teuthology.orchestra.run.smithi088.stderr:2024-06-03T09:07:00.994+0000 7f95d55c5640 1 -- 172.21.15.88:0/1654250444 wait complete. 2024-06-03T09:07:00.996 INFO:teuthology.orchestra.run.smithi088.stderr:2024-06-03T09:07:00.994+0000 7f95d55c5640 1 librados: shutdown 2024-06-03T09:07:01.005 INFO:tasks.ceph:Scrubbing osd.6 2024-06-03T09:07:01.006 DEBUG:teuthology.orchestra.run.smithi088:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph tell osd.6 config set osd_debug_deep_scrub_sleep 0 2024-06-03T09:07:01.112 INFO:teuthology.orchestra.run.smithi088.stderr:2024-06-03T09:07:01.112+0000 7f05b54a8640 10 monclient: get_monmap_and_config 2024-06-03T09:07:01.112 INFO:teuthology.orchestra.run.smithi088.stderr:2024-06-03T09:07:01.112+0000 7f05b54a8640 10 monclient: build_initial_monmap 2024-06-03T09:07:01.112 INFO:teuthology.orchestra.run.smithi088.stderr:2024-06-03T09:07:01.112+0000 7f05b54a8640 10 monclient: monmap: 2024-06-03T09:07:01.112 INFO:teuthology.orchestra.run.smithi088.stderr:epoch 0 2024-06-03T09:07:01.113 INFO:teuthology.orchestra.run.smithi088.stderr:fsid ec3c3fdd-3ac6-4d7d-9412-683130ed8b04 2024-06-03T09:07:01.113 INFO:teuthology.orchestra.run.smithi088.stderr:last_changed 2024-06-03T09:07:01.113522+0000 2024-06-03T09:07:01.113 INFO:teuthology.orchestra.run.smithi088.stderr:created 2024-06-03T09:07:01.113522+0000 2024-06-03T09:07:01.113 INFO:teuthology.orchestra.run.smithi088.stderr:min_mon_release 0 (unknown) 2024-06-03T09:07:01.113 INFO:teuthology.orchestra.run.smithi088.stderr:election_strategy: 3 2024-06-03T09:07:01.113 INFO:teuthology.orchestra.run.smithi088.stderr:0: v1:172.21.15.88:6789/0 mon.noname-a 2024-06-03T09:07:01.113 INFO:teuthology.orchestra.run.smithi088.stderr:1: v1:172.21.15.192:6789/0 mon.noname-c 2024-06-03T09:07:01.113 INFO:teuthology.orchestra.run.smithi088.stderr:2: v1:172.21.15.88:6790/0 mon.noname-b 2024-06-03T09:07:01.113 INFO:teuthology.orchestra.run.smithi088.stderr: 2024-06-03T09:07:01.114 INFO:teuthology.orchestra.run.smithi088.stderr:2024-06-03T09:07:01.113+0000 7f05b54a8640 1 Processor -- start 2024-06-03T09:07:01.115 INFO:teuthology.orchestra.run.smithi088.stderr:2024-06-03T09:07:01.113+0000 7f05b54a8640 1 -- start start 2024-06-03T09:07:01.115 INFO:teuthology.orchestra.run.smithi088.stderr:2024-06-03T09:07:01.113+0000 7f05b54a8640 10 monclient: init 2024-06-03T09:07:01.115 INFO:teuthology.orchestra.run.smithi088.stderr:2024-06-03T09:07:01.113+0000 7f05b54a8640 10 monclient: _reopen_session rank -1 2024-06-03T09:07:01.115 INFO:teuthology.orchestra.run.smithi088.stderr:2024-06-03T09:07:01.113+0000 7f05b54a8640 10 monclient: _add_conns ranks=[1,2,0] 2024-06-03T09:07:01.115 INFO:teuthology.orchestra.run.smithi088.stderr:2024-06-03T09:07:01.113+0000 7f05b54a8640 10 monclient(hunting): picked mon.noname-c con 0x7f05b006ba10 addr v1:172.21.15.192:6789/0 2024-06-03T09:07:01.115 INFO:teuthology.orchestra.run.smithi088.stderr:2024-06-03T09:07:01.113+0000 7f05b54a8640 10 monclient(hunting): picked mon.noname-b con 0x7f05b006bdf0 addr v1:172.21.15.88:6790/0 2024-06-03T09:07:01.115 INFO:teuthology.orchestra.run.smithi088.stderr:2024-06-03T09:07:01.113+0000 7f05b54a8640 10 monclient(hunting): picked mon.noname-a con 0x7f05b013eee0 addr v1:172.21.15.88:6789/0 2024-06-03T09:07:01.115 INFO:teuthology.orchestra.run.smithi088.stderr:2024-06-03T09:07:01.113+0000 7f05b54a8640 1 -- --> v1:172.21.15.88:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f05b00ad400 con 0x7f05b013eee0 2024-06-03T09:07:01.115 INFO:teuthology.orchestra.run.smithi088.stderr:2024-06-03T09:07:01.113+0000 7f05b54a8640 1 -- --> v1:172.21.15.192:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f05b0142530 con 0x7f05b006ba10 2024-06-03T09:07:01.115 INFO:teuthology.orchestra.run.smithi088.stderr:2024-06-03T09:07:01.113+0000 7f05b54a8640 1 -- --> v1:172.21.15.88:6790/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f05b00ad630 con 0x7f05b006bdf0 2024-06-03T09:07:01.115 INFO:teuthology.orchestra.run.smithi088.stderr:2024-06-03T09:07:01.113+0000 7f05b54a8640 10 monclient(hunting): _renew_subs 2024-06-03T09:07:01.115 INFO:teuthology.orchestra.run.smithi088.stderr:2024-06-03T09:07:01.113+0000 7f05b54a8640 10 monclient(hunting): authenticate will time out at 3076.923855s 2024-06-03T09:07:01.116 INFO:teuthology.orchestra.run.smithi088.stderr:2024-06-03T09:07:01.114+0000 7f05a67fc640 1 --1- >> v1:172.21.15.88:6790/0 conn(0x7f05b006bdf0 0x7f05b013e7d0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.88:6790/0 says I am v1:172.21.15.88:51148/0 (socket says 172.21.15.88:51148) 2024-06-03T09:07:01.116 INFO:teuthology.orchestra.run.smithi088.stderr:2024-06-03T09:07:01.114+0000 7f05a67fc640 1 -- 172.21.15.88:0/2807935574 learned_addr learned my addr 172.21.15.88:0/2807935574 (peer_addr_for_me v1:172.21.15.88:0/0) 2024-06-03T09:07:01.116 INFO:teuthology.orchestra.run.smithi088.stderr:2024-06-03T09:07:01.116+0000 7f05ae7fc640 1 -- 172.21.15.88:0/2807935574 <== mon.2 v1:172.21.15.88:6790/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 4198184169 0 0) 0x7f05b00ad630 con 0x7f05b006bdf0 2024-06-03T09:07:01.116 INFO:teuthology.orchestra.run.smithi088.stderr:2024-06-03T09:07:01.116+0000 7f05ae7fc640 10 monclient(hunting): _negotiate 2024-06-03T09:07:01.117 INFO:teuthology.orchestra.run.smithi088.stderr:2024-06-03T09:07:01.116+0000 7f05ae7fc640 10 monclient(hunting): _init_auth method 2 2024-06-03T09:07:01.117 INFO:teuthology.orchestra.run.smithi088.stderr:2024-06-03T09:07:01.116+0000 7f05ae7fc640 10 monclient(hunting): _init_auth creating new auth 2024-06-03T09:07:01.117 INFO:teuthology.orchestra.run.smithi088.stderr:2024-06-03T09:07:01.116+0000 7f05ae7fc640 10 monclient(hunting): my global_id is 30311 2024-06-03T09:07:01.117 INFO:teuthology.orchestra.run.smithi088.stderr:2024-06-03T09:07:01.116+0000 7f05ae7fc640 1 -- 172.21.15.88:0/2807935574 --> v1:172.21.15.88:6790/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f059c003520 con 0x7f05b006bdf0 2024-06-03T09:07:01.117 INFO:teuthology.orchestra.run.smithi088.stderr:2024-06-03T09:07:01.116+0000 7f05ae7fc640 1 -- 172.21.15.88:0/2807935574 <== mon.1 v1:172.21.15.192:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 3237782465 0 0) 0x7f05b0142530 con 0x7f05b006ba10 2024-06-03T09:07:01.117 INFO:teuthology.orchestra.run.smithi088.stderr:2024-06-03T09:07:01.116+0000 7f05ae7fc640 10 monclient(hunting): _negotiate 2024-06-03T09:07:01.117 INFO:teuthology.orchestra.run.smithi088.stderr:2024-06-03T09:07:01.116+0000 7f05ae7fc640 10 monclient(hunting): _init_auth method 2 2024-06-03T09:07:01.117 INFO:teuthology.orchestra.run.smithi088.stderr:2024-06-03T09:07:01.116+0000 7f05ae7fc640 10 monclient(hunting): _init_auth creating new auth 2024-06-03T09:07:01.117 INFO:teuthology.orchestra.run.smithi088.stderr:2024-06-03T09:07:01.116+0000 7f05ae7fc640 10 monclient(hunting): my global_id is 30376 2024-06-03T09:07:01.117 INFO:teuthology.orchestra.run.smithi088.stderr:2024-06-03T09:07:01.116+0000 7f05ae7fc640 1 -- 172.21.15.88:0/2807935574 --> v1:172.21.15.192:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f05b00ad630 con 0x7f05b006ba10 2024-06-03T09:07:01.120 INFO:teuthology.orchestra.run.smithi088.stderr:2024-06-03T09:07:01.119+0000 7f05ae7fc640 1 -- 172.21.15.88:0/2807935574 <== mon.2 v1:172.21.15.88:6790/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 214+0+0 (unknown 2506742685 0 0) 0x7f059c003520 con 0x7f05b006bdf0 2024-06-03T09:07:01.120 INFO:teuthology.orchestra.run.smithi088.stderr:2024-06-03T09:07:01.119+0000 7f05ae7fc640 1 -- 172.21.15.88:0/2807935574 >> v1:172.21.15.192:6789/0 conn(0x7f05b006ba10 legacy=0x7f05b0141e20 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-06-03T09:07:01.120 INFO:teuthology.orchestra.run.smithi088.stderr:2024-06-03T09:07:01.119+0000 7f05ae7fc640 1 -- 172.21.15.88:0/2807935574 >> v1:172.21.15.88:6789/0 conn(0x7f05b013eee0 legacy=0x7f05b014ce60 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-06-03T09:07:01.120 INFO:teuthology.orchestra.run.smithi088.stderr:2024-06-03T09:07:01.119+0000 7f05ae7fc640 10 monclient: _finish_hunting 0 2024-06-03T09:07:01.120 INFO:teuthology.orchestra.run.smithi088.stderr:2024-06-03T09:07:01.119+0000 7f05ae7fc640 1 monclient: found mon.noname-b 2024-06-03T09:07:01.120 INFO:teuthology.orchestra.run.smithi088.stderr:2024-06-03T09:07:01.119+0000 7f05ae7fc640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-06-03T09:07:01.120 INFO:teuthology.orchestra.run.smithi088.stderr:2024-06-03T09:07:01.119+0000 7f05ae7fc640 10 monclient: _send_mon_message to mon.noname-b at v1:172.21.15.88:6790/0 2024-06-03T09:07:01.120 INFO:teuthology.orchestra.run.smithi088.stderr:2024-06-03T09:07:01.119+0000 7f05ae7fc640 1 -- 172.21.15.88:0/2807935574 --> v1:172.21.15.88:6790/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f05b006c840 con 0x7f05b006bdf0 2024-06-03T09:07:01.120 INFO:teuthology.orchestra.run.smithi088.stderr:2024-06-03T09:07:01.119+0000 7f05ae7fc640 10 monclient: _finish_auth 0 2024-06-03T09:07:01.120 INFO:teuthology.orchestra.run.smithi088.stderr:2024-06-03T09:07:01.119+0000 7f05ae7fc640 10 monclient: _check_auth_tickets 2024-06-03T09:07:01.120 INFO:teuthology.orchestra.run.smithi088.stderr:2024-06-03T09:07:01.119+0000 7f05ae7fc640 20 monclient: _check_auth_rotating not needed by client.admin 2024-06-03T09:07:01.120 INFO:teuthology.orchestra.run.smithi088.stderr:2024-06-03T09:07:01.119+0000 7f05ae7fc640 1 -- 172.21.15.88:0/2807935574 <== mon.2 v1:172.21.15.88:6790/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 2234374065 0 0) 0x7f0598001610 con 0x7f05b006bdf0 2024-06-03T09:07:01.120 INFO:teuthology.orchestra.run.smithi088.stderr:2024-06-03T09:07:01.119+0000 7f05b54a8640 5 monclient: authenticate success, global_id 30311 2024-06-03T09:07:01.121 INFO:teuthology.orchestra.run.smithi088.stderr:2024-06-03T09:07:01.119+0000 7f05b54a8640 20 monclient: get_monmap_and_config waiting for monmap|config 2024-06-03T09:07:01.121 INFO:teuthology.orchestra.run.smithi088.stderr:2024-06-03T09:07:01.119+0000 7f05ae7fc640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-06-03T09:07:01.121 INFO:teuthology.orchestra.run.smithi088.stderr:2024-06-03T09:07:01.119+0000 7f05ae7fc640 10 monclient: got monmap 1 from mon.noname-b (according to old e1) 2024-06-03T09:07:01.121 INFO:teuthology.orchestra.run.smithi088.stderr:2024-06-03T09:07:01.119+0000 7f05ae7fc640 10 monclient: dump: 2024-06-03T09:07:01.121 INFO:teuthology.orchestra.run.smithi088.stderr:epoch 1 2024-06-03T09:07:01.121 INFO:teuthology.orchestra.run.smithi088.stderr:fsid ec3c3fdd-3ac6-4d7d-9412-683130ed8b04 2024-06-03T09:07:01.121 INFO:teuthology.orchestra.run.smithi088.stderr:last_changed 2024-06-03T08:34:31.885125+0000 2024-06-03T09:07:01.121 INFO:teuthology.orchestra.run.smithi088.stderr:created 2024-06-03T08:34:31.885125+0000 2024-06-03T09:07:01.121 INFO:teuthology.orchestra.run.smithi088.stderr:min_mon_release 18 (reef) 2024-06-03T09:07:01.121 INFO:teuthology.orchestra.run.smithi088.stderr:election_strategy: 3 2024-06-03T09:07:01.121 INFO:teuthology.orchestra.run.smithi088.stderr:0: v1:172.21.15.88:6789/0 mon.a 2024-06-03T09:07:01.121 INFO:teuthology.orchestra.run.smithi088.stderr:1: v1:172.21.15.192:6789/0 mon.b 2024-06-03T09:07:01.121 INFO:teuthology.orchestra.run.smithi088.stderr:2: v1:172.21.15.88:6790/0 mon.c 2024-06-03T09:07:01.121 INFO:teuthology.orchestra.run.smithi088.stderr: 2024-06-03T09:07:01.122 INFO:teuthology.orchestra.run.smithi088.stderr:2024-06-03T09:07:01.119+0000 7f05b54a8640 20 monclient: get_monmap_and_config waiting for monmap|config 2024-06-03T09:07:01.124 INFO:teuthology.orchestra.run.smithi088.stderr:2024-06-03T09:07:01.121+0000 7f05ae7fc640 1 -- 172.21.15.88:0/2807935574 <== mon.2 v1:172.21.15.88:6790/0 4 ==== config(0 keys) v1 ==== 4+0+0 (unknown 0 0 0) 0x7f0598001830 con 0x7f05b006bdf0 2024-06-03T09:07:01.124 INFO:teuthology.orchestra.run.smithi088.stderr:2024-06-03T09:07:01.121+0000 7f05ae7fc640 10 monclient: handle_config config(0 keys) v1 2024-06-03T09:07:01.124 INFO:teuthology.orchestra.run.smithi088.stderr:2024-06-03T09:07:01.121+0000 7f05ae7fc640 1 -- 172.21.15.88:0/2807935574 <== mon.2 v1:172.21.15.88:6790/0 5 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 2234374065 0 0) 0x7f0598001b80 con 0x7f05b006bdf0 2024-06-03T09:07:01.124 INFO:teuthology.orchestra.run.smithi088.stderr:2024-06-03T09:07:01.121+0000 7f05ae7fc640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-06-03T09:07:01.124 INFO:teuthology.orchestra.run.smithi088.stderr:2024-06-03T09:07:01.121+0000 7f05ae7fc640 10 monclient: got monmap 1 from mon.c (according to old e1) 2024-06-03T09:07:01.124 INFO:teuthology.orchestra.run.smithi088.stderr:2024-06-03T09:07:01.121+0000 7f05ae7fc640 10 monclient: dump: 2024-06-03T09:07:01.125 INFO:teuthology.orchestra.run.smithi088.stderr:epoch 1 2024-06-03T09:07:01.125 INFO:teuthology.orchestra.run.smithi088.stderr:fsid ec3c3fdd-3ac6-4d7d-9412-683130ed8b04 2024-06-03T09:07:01.125 INFO:teuthology.orchestra.run.smithi088.stderr:last_changed 2024-06-03T08:34:31.885125+0000 2024-06-03T09:07:01.125 INFO:teuthology.orchestra.run.smithi088.stderr:created 2024-06-03T08:34:31.885125+0000 2024-06-03T09:07:01.125 INFO:teuthology.orchestra.run.smithi088.stderr:min_mon_release 18 (reef) 2024-06-03T09:07:01.125 INFO:teuthology.orchestra.run.smithi088.stderr:election_strategy: 3 2024-06-03T09:07:01.125 INFO:teuthology.orchestra.run.smithi088.stderr:0: v1:172.21.15.88:6789/0 mon.a 2024-06-03T09:07:01.125 INFO:teuthology.orchestra.run.smithi088.stderr:1: v1:172.21.15.192:6789/0 mon.b 2024-06-03T09:07:01.125 INFO:teuthology.orchestra.run.smithi088.stderr:2: v1:172.21.15.88:6790/0 mon.c 2024-06-03T09:07:01.125 INFO:teuthology.orchestra.run.smithi088.stderr: 2024-06-03T09:07:01.125 INFO:teuthology.orchestra.run.smithi088.stderr:2024-06-03T09:07:01.121+0000 7f05b54a8640 10 monclient: get_monmap_and_config success 2024-06-03T09:07:01.125 INFO:teuthology.orchestra.run.smithi088.stderr:2024-06-03T09:07:01.121+0000 7f05b54a8640 4 set_mon_vals no callback set 2024-06-03T09:07:01.125 INFO:teuthology.orchestra.run.smithi088.stderr:2024-06-03T09:07:01.122+0000 7f05b54a8640 10 monclient: shutdown 2024-06-03T09:07:01.126 INFO:teuthology.orchestra.run.smithi088.stderr:2024-06-03T09:07:01.122+0000 7f05b54a8640 20 monclient: shutdown discarding 0 pending message(s) 2024-06-03T09:07:01.126 INFO:teuthology.orchestra.run.smithi088.stderr:2024-06-03T09:07:01.122+0000 7f05b54a8640 1 -- 172.21.15.88:0/2807935574 >> v1:172.21.15.88:6790/0 conn(0x7f05b006bdf0 legacy=0x7f05b013e7d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-06-03T09:07:01.126 INFO:teuthology.orchestra.run.smithi088.stderr:2024-06-03T09:07:01.122+0000 7f05b54a8640 1 -- 172.21.15.88:0/2807935574 shutdown_connections 2024-06-03T09:07:01.126 INFO:teuthology.orchestra.run.smithi088.stderr:2024-06-03T09:07:01.122+0000 7f05b54a8640 1 -- 172.21.15.88:0/2807935574 >> 172.21.15.88:0/2807935574 conn(0x7f05b00b4fb0 msgr2=0x7f05b00b53c0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-06-03T09:07:01.126 INFO:teuthology.orchestra.run.smithi088.stderr:2024-06-03T09:07:01.122+0000 7f05b54a8640 1 -- 172.21.15.88:0/2807935574 shutdown_connections 2024-06-03T09:07:01.126 INFO:teuthology.orchestra.run.smithi088.stderr:2024-06-03T09:07:01.122+0000 7f05b54a8640 1 -- 172.21.15.88:0/2807935574 wait complete. 2024-06-03T09:07:01.126 INFO:teuthology.orchestra.run.smithi088.stderr:2024-06-03T09:07:01.122+0000 7f05b54a8640 10 monclient: build_initial_monmap 2024-06-03T09:07:01.126 INFO:teuthology.orchestra.run.smithi088.stderr:2024-06-03T09:07:01.122+0000 7f05b54a8640 10 monclient: monmap: 2024-06-03T09:07:01.126 INFO:teuthology.orchestra.run.smithi088.stderr:epoch 0 2024-06-03T09:07:01.126 INFO:teuthology.orchestra.run.smithi088.stderr:fsid 00000000-0000-0000-0000-000000000000 2024-06-03T09:07:01.126 INFO:teuthology.orchestra.run.smithi088.stderr:last_changed 0.000000 2024-06-03T09:07:01.126 INFO:teuthology.orchestra.run.smithi088.stderr:created 0.000000 2024-06-03T09:07:01.126 INFO:teuthology.orchestra.run.smithi088.stderr:min_mon_release 0 (unknown) 2024-06-03T09:07:01.127 INFO:teuthology.orchestra.run.smithi088.stderr:election_strategy: 1 2024-06-03T09:07:01.127 INFO:teuthology.orchestra.run.smithi088.stderr:0: v1:172.21.15.88:6789/0 mon.noname-a 2024-06-03T09:07:01.127 INFO:teuthology.orchestra.run.smithi088.stderr:1: v1:172.21.15.192:6789/0 mon.noname-b 2024-06-03T09:07:01.127 INFO:teuthology.orchestra.run.smithi088.stderr:2: v1:172.21.15.88:6790/0 mon.noname-c 2024-06-03T09:07:01.127 INFO:teuthology.orchestra.run.smithi088.stderr: 2024-06-03T09:07:01.127 INFO:teuthology.orchestra.run.smithi088.stderr:2024-06-03T09:07:01.123+0000 7f05b54a8640 1 librados: starting msgr at 2024-06-03T09:07:01.127 INFO:teuthology.orchestra.run.smithi088.stderr:2024-06-03T09:07:01.123+0000 7f05b54a8640 1 librados: starting objecter 2024-06-03T09:07:01.127 INFO:teuthology.orchestra.run.smithi088.stderr:2024-06-03T09:07:01.123+0000 7f05b54a8640 1 Processor -- start 2024-06-03T09:07:01.127 INFO:teuthology.orchestra.run.smithi088.stderr:2024-06-03T09:07:01.123+0000 7f05b54a8640 1 -- start start 2024-06-03T09:07:01.127 INFO:teuthology.orchestra.run.smithi088.stderr:2024-06-03T09:07:01.123+0000 7f05b54a8640 1 librados: setting wanted keys 2024-06-03T09:07:01.127 INFO:teuthology.orchestra.run.smithi088.stderr:2024-06-03T09:07:01.123+0000 7f05b54a8640 1 librados: calling monclient init 2024-06-03T09:07:01.127 INFO:teuthology.orchestra.run.smithi088.stderr:2024-06-03T09:07:01.123+0000 7f05b54a8640 10 monclient: init 2024-06-03T09:07:01.127 INFO:teuthology.orchestra.run.smithi088.stderr:2024-06-03T09:07:01.123+0000 7f05b54a8640 10 monclient: _reopen_session rank -1 2024-06-03T09:07:01.127 INFO:teuthology.orchestra.run.smithi088.stderr:2024-06-03T09:07:01.123+0000 7f05b54a8640 10 monclient: _add_conns ranks=[0,2,1] 2024-06-03T09:07:01.128 INFO:teuthology.orchestra.run.smithi088.stderr:2024-06-03T09:07:01.123+0000 7f05b54a8640 10 monclient(hunting): picked mon.noname-a con 0x7f05b006ba10 addr v1:172.21.15.88:6789/0 2024-06-03T09:07:01.128 INFO:teuthology.orchestra.run.smithi088.stderr:2024-06-03T09:07:01.124+0000 7f05b54a8640 10 monclient(hunting): picked mon.noname-c con 0x7f05b006bdf0 addr v1:172.21.15.88:6790/0 2024-06-03T09:07:01.128 INFO:teuthology.orchestra.run.smithi088.stderr:2024-06-03T09:07:01.124+0000 7f05b54a8640 10 monclient(hunting): picked mon.noname-b con 0x7f05b013eee0 addr v1:172.21.15.192:6789/0 2024-06-03T09:07:01.128 INFO:teuthology.orchestra.run.smithi088.stderr:2024-06-03T09:07:01.124+0000 7f05b54a8640 1 -- --> v1:172.21.15.88:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f05b01cab80 con 0x7f05b006ba10 2024-06-03T09:07:01.128 INFO:teuthology.orchestra.run.smithi088.stderr:2024-06-03T09:07:01.124+0000 7f05b54a8640 1 -- --> v1:172.21.15.192:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f05b00b2de0 con 0x7f05b013eee0 2024-06-03T09:07:01.128 INFO:teuthology.orchestra.run.smithi088.stderr:2024-06-03T09:07:01.124+0000 7f05b54a8640 1 -- --> v1:172.21.15.88:6790/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f05b00b4a50 con 0x7f05b006bdf0 2024-06-03T09:07:01.128 INFO:teuthology.orchestra.run.smithi088.stderr:2024-06-03T09:07:01.124+0000 7f05b54a8640 10 monclient(hunting): _renew_subs 2024-06-03T09:07:01.128 INFO:teuthology.orchestra.run.smithi088.stderr:2024-06-03T09:07:01.124+0000 7f05b54a8640 10 monclient(hunting): authenticate will time out at 3076.933981s 2024-06-03T09:07:01.128 INFO:teuthology.orchestra.run.smithi088.stderr:2024-06-03T09:07:01.124+0000 7f05a67fc640 1 --1- >> v1:172.21.15.88:6790/0 conn(0x7f05b006bdf0 0x7f05b01d62e0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=1).handle_server_banner_and_identify peer v1:172.21.15.88:6790/0 says I am v1:172.21.15.88:51154/0 (socket says 172.21.15.88:51154) 2024-06-03T09:07:01.128 INFO:teuthology.orchestra.run.smithi088.stderr:2024-06-03T09:07:01.124+0000 7f05a67fc640 1 -- 172.21.15.88:0/3717066664 learned_addr learned my addr 172.21.15.88:0/3717066664 (peer_addr_for_me v1:172.21.15.88:0/0) 2024-06-03T09:07:01.128 INFO:teuthology.orchestra.run.smithi088.stderr:2024-06-03T09:07:01.124+0000 7f05a7fff640 10 client.?.objecter ms_handle_connect 0x7f05b006bdf0 2024-06-03T09:07:01.128 INFO:teuthology.orchestra.run.smithi088.stderr:2024-06-03T09:07:01.124+0000 7f05a7fff640 10 client.?.objecter resend_mon_ops 2024-06-03T09:07:01.129 INFO:teuthology.orchestra.run.smithi088.stderr:2024-06-03T09:07:01.125+0000 7f05a7fff640 10 client.?.objecter ms_handle_connect 0x7f05b013eee0 2024-06-03T09:07:01.129 INFO:teuthology.orchestra.run.smithi088.stderr:2024-06-03T09:07:01.125+0000 7f05a7fff640 10 client.?.objecter resend_mon_ops 2024-06-03T09:07:01.129 INFO:teuthology.orchestra.run.smithi088.stderr:2024-06-03T09:07:01.126+0000 7f05a7fff640 1 -- 172.21.15.88:0/3717066664 <== mon.2 v1:172.21.15.88:6790/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 1165519109 0 0) 0x7f05b00b4a50 con 0x7f05b006bdf0 2024-06-03T09:07:01.129 INFO:teuthology.orchestra.run.smithi088.stderr:2024-06-03T09:07:01.126+0000 7f05a7fff640 10 monclient(hunting): _negotiate 2024-06-03T09:07:01.129 INFO:teuthology.orchestra.run.smithi088.stderr:2024-06-03T09:07:01.126+0000 7f05a7fff640 10 monclient(hunting): _init_auth method 2 2024-06-03T09:07:01.130 INFO:teuthology.orchestra.run.smithi088.stderr:2024-06-03T09:07:01.126+0000 7f05a7fff640 10 monclient(hunting): _init_auth creating new auth 2024-06-03T09:07:01.130 INFO:teuthology.orchestra.run.smithi088.stderr:2024-06-03T09:07:01.126+0000 7f05a7fff640 10 monclient(hunting): my global_id is 30314 2024-06-03T09:07:01.130 INFO:teuthology.orchestra.run.smithi088.stderr:2024-06-03T09:07:01.126+0000 7f05a7fff640 1 -- 172.21.15.88:0/3717066664 --> v1:172.21.15.88:6790/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f0590004eb0 con 0x7f05b006bdf0 2024-06-03T09:07:01.130 INFO:teuthology.orchestra.run.smithi088.stderr:2024-06-03T09:07:01.127+0000 7f05a7fff640 1 -- 172.21.15.88:0/3717066664 <== mon.1 v1:172.21.15.192:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 3784382585 0 0) 0x7f05b00b2de0 con 0x7f05b013eee0 2024-06-03T09:07:01.130 INFO:teuthology.orchestra.run.smithi088.stderr:2024-06-03T09:07:01.127+0000 7f05a7fff640 10 monclient(hunting): _negotiate 2024-06-03T09:07:01.130 INFO:teuthology.orchestra.run.smithi088.stderr:2024-06-03T09:07:01.127+0000 7f05a7fff640 10 monclient(hunting): _init_auth method 2 2024-06-03T09:07:01.130 INFO:teuthology.orchestra.run.smithi088.stderr:2024-06-03T09:07:01.127+0000 7f05a7fff640 10 monclient(hunting): _init_auth creating new auth 2024-06-03T09:07:01.130 INFO:teuthology.orchestra.run.smithi088.stderr:2024-06-03T09:07:01.127+0000 7f05a7fff640 10 monclient(hunting): my global_id is 30379 2024-06-03T09:07:01.130 INFO:teuthology.orchestra.run.smithi088.stderr:2024-06-03T09:07:01.127+0000 7f05a7fff640 1 -- 172.21.15.88:0/3717066664 --> v1:172.21.15.192:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f05b00b4a50 con 0x7f05b013eee0 2024-06-03T09:07:01.131 INFO:teuthology.orchestra.run.smithi088.stderr:2024-06-03T09:07:01.131+0000 7f05a7fff640 1 -- 172.21.15.88:0/3717066664 <== mon.2 v1:172.21.15.88:6790/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 764+0+0 (unknown 1189901293 0 0) 0x7f0590004eb0 con 0x7f05b006bdf0 2024-06-03T09:07:01.131 INFO:teuthology.orchestra.run.smithi088.stderr:2024-06-03T09:07:01.131+0000 7f05a7fff640 1 -- 172.21.15.88:0/3717066664 --> v1:172.21.15.88:6790/0 -- auth(proto 2 165 bytes epoch 0) v1 -- 0x7f05b00b2de0 con 0x7f05b006bdf0 2024-06-03T09:07:01.131 INFO:teuthology.orchestra.run.smithi088.stderr:2024-06-03T09:07:01.131+0000 7f05a7fff640 1 -- 172.21.15.88:0/3717066664 <== mon.2 v1:172.21.15.88:6790/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 2234374065 0 0) 0x7f05980015d0 con 0x7f05b006bdf0 2024-06-03T09:07:01.131 INFO:teuthology.orchestra.run.smithi088.stderr:2024-06-03T09:07:01.131+0000 7f05a7fff640 10 monclient(hunting): handle_monmap mon_map magic: 0 v1 2024-06-03T09:07:01.131 INFO:teuthology.orchestra.run.smithi088.stderr:2024-06-03T09:07:01.131+0000 7f05a7fff640 10 monclient(hunting): got monmap 1 from mon.noname-c (according to old e1) 2024-06-03T09:07:01.131 INFO:teuthology.orchestra.run.smithi088.stderr:2024-06-03T09:07:01.131+0000 7f05a7fff640 10 monclient(hunting): dump: 2024-06-03T09:07:01.131 INFO:teuthology.orchestra.run.smithi088.stderr:epoch 1 2024-06-03T09:07:01.131 INFO:teuthology.orchestra.run.smithi088.stderr:fsid ec3c3fdd-3ac6-4d7d-9412-683130ed8b04 2024-06-03T09:07:01.131 INFO:teuthology.orchestra.run.smithi088.stderr:last_changed 2024-06-03T08:34:31.885125+0000 2024-06-03T09:07:01.131 INFO:teuthology.orchestra.run.smithi088.stderr:created 2024-06-03T08:34:31.885125+0000 2024-06-03T09:07:01.132 INFO:teuthology.orchestra.run.smithi088.stderr:min_mon_release 18 (reef) 2024-06-03T09:07:01.132 INFO:teuthology.orchestra.run.smithi088.stderr:election_strategy: 3 2024-06-03T09:07:01.132 INFO:teuthology.orchestra.run.smithi088.stderr:0: v1:172.21.15.88:6789/0 mon.a 2024-06-03T09:07:01.132 INFO:teuthology.orchestra.run.smithi088.stderr:1: v1:172.21.15.192:6789/0 mon.b 2024-06-03T09:07:01.132 INFO:teuthology.orchestra.run.smithi088.stderr:2: v1:172.21.15.88:6790/0 mon.c 2024-06-03T09:07:01.132 INFO:teuthology.orchestra.run.smithi088.stderr: 2024-06-03T09:07:01.132 INFO:teuthology.orchestra.run.smithi088.stderr:2024-06-03T09:07:01.131+0000 7f05a7fff640 10 monclient(hunting): _finish_auth 0 2024-06-03T09:07:01.133 INFO:teuthology.orchestra.run.smithi088.stderr:2024-06-03T09:07:01.133+0000 7f05a7fff640 1 -- 172.21.15.88:0/3717066664 <== mon.1 v1:172.21.15.192:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 764+0+0 (unknown 2302193912 0 0) 0x7f05b00b4a50 con 0x7f05b013eee0 2024-06-03T09:07:01.133 INFO:teuthology.orchestra.run.smithi088.stderr:2024-06-03T09:07:01.133+0000 7f05a7fff640 1 -- 172.21.15.88:0/3717066664 --> v1:172.21.15.192:6789/0 -- auth(proto 2 165 bytes epoch 0) v1 -- 0x7f0590004eb0 con 0x7f05b013eee0 2024-06-03T09:07:01.133 INFO:teuthology.orchestra.run.smithi088.stderr:2024-06-03T09:07:01.133+0000 7f05a7fff640 1 -- 172.21.15.88:0/3717066664 <== mon.1 v1:172.21.15.192:6789/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 2234374065 0 0) 0x7f05a8001a50 con 0x7f05b013eee0 2024-06-03T09:07:01.133 INFO:teuthology.orchestra.run.smithi088.stderr:2024-06-03T09:07:01.133+0000 7f05a7fff640 10 monclient(hunting): handle_monmap mon_map magic: 0 v1 2024-06-03T09:07:01.133 INFO:teuthology.orchestra.run.smithi088.stderr:2024-06-03T09:07:01.133+0000 7f05a7fff640 10 monclient(hunting): got monmap 1 from mon.b (according to old e1) 2024-06-03T09:07:01.133 INFO:teuthology.orchestra.run.smithi088.stderr:2024-06-03T09:07:01.133+0000 7f05a7fff640 10 monclient(hunting): dump: 2024-06-03T09:07:01.133 INFO:teuthology.orchestra.run.smithi088.stderr:epoch 1 2024-06-03T09:07:01.133 INFO:teuthology.orchestra.run.smithi088.stderr:fsid ec3c3fdd-3ac6-4d7d-9412-683130ed8b04 2024-06-03T09:07:01.133 INFO:teuthology.orchestra.run.smithi088.stderr:last_changed 2024-06-03T08:34:31.885125+0000 2024-06-03T09:07:01.133 INFO:teuthology.orchestra.run.smithi088.stderr:created 2024-06-03T08:34:31.885125+0000 2024-06-03T09:07:01.133 INFO:teuthology.orchestra.run.smithi088.stderr:min_mon_release 18 (reef) 2024-06-03T09:07:01.133 INFO:teuthology.orchestra.run.smithi088.stderr:election_strategy: 3 2024-06-03T09:07:01.134 INFO:teuthology.orchestra.run.smithi088.stderr:0: v1:172.21.15.88:6789/0 mon.a 2024-06-03T09:07:01.134 INFO:teuthology.orchestra.run.smithi088.stderr:1: v1:172.21.15.192:6789/0 mon.b 2024-06-03T09:07:01.134 INFO:teuthology.orchestra.run.smithi088.stderr:2: v1:172.21.15.88:6790/0 mon.c 2024-06-03T09:07:01.134 INFO:teuthology.orchestra.run.smithi088.stderr: 2024-06-03T09:07:01.135 INFO:teuthology.orchestra.run.smithi088.stderr:2024-06-03T09:07:01.134+0000 7f05a7fff640 1 -- 172.21.15.88:0/3717066664 <== mon.2 v1:172.21.15.88:6790/0 4 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 580+0+0 (unknown 2324481934 0 0) 0x7f05b00b2de0 con 0x7f05b006bdf0 2024-06-03T09:07:01.135 INFO:teuthology.orchestra.run.smithi088.stderr:2024-06-03T09:07:01.134+0000 7f05a7fff640 1 -- 172.21.15.88:0/3717066664 >> v1:172.21.15.192:6789/0 conn(0x7f05b013eee0 legacy=0x7f05b01d69f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-06-03T09:07:01.135 INFO:teuthology.orchestra.run.smithi088.stderr:2024-06-03T09:07:01.134+0000 7f05a7fff640 1 -- 172.21.15.88:0/3717066664 >> v1:172.21.15.88:6789/0 conn(0x7f05b006ba10 legacy=0x7f05b00b3c20 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-06-03T09:07:01.135 INFO:teuthology.orchestra.run.smithi088.stderr:2024-06-03T09:07:01.134+0000 7f05a7fff640 10 monclient: _finish_hunting 0 2024-06-03T09:07:01.135 INFO:teuthology.orchestra.run.smithi088.stderr:2024-06-03T09:07:01.134+0000 7f05a7fff640 1 monclient: found mon.c 2024-06-03T09:07:01.135 INFO:teuthology.orchestra.run.smithi088.stderr:2024-06-03T09:07:01.134+0000 7f05a7fff640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-06-03T09:07:01.135 INFO:teuthology.orchestra.run.smithi088.stderr:2024-06-03T09:07:01.134+0000 7f05a7fff640 10 monclient: _send_mon_message to mon.c at v1:172.21.15.88:6790/0 2024-06-03T09:07:01.135 INFO:teuthology.orchestra.run.smithi088.stderr:2024-06-03T09:07:01.134+0000 7f05a7fff640 1 -- 172.21.15.88:0/3717066664 --> v1:172.21.15.88:6790/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f05b00b4430 con 0x7f05b006bdf0 2024-06-03T09:07:01.135 INFO:teuthology.orchestra.run.smithi088.stderr:2024-06-03T09:07:01.134+0000 7f05a7fff640 10 monclient: _finish_auth 0 2024-06-03T09:07:01.135 INFO:teuthology.orchestra.run.smithi088.stderr:2024-06-03T09:07:01.134+0000 7f05a7fff640 10 monclient: _check_auth_tickets 2024-06-03T09:07:01.136 INFO:teuthology.orchestra.run.smithi088.stderr:2024-06-03T09:07:01.134+0000 7f05a7fff640 20 monclient: _check_auth_rotating not needed by client.admin 2024-06-03T09:07:01.136 INFO:teuthology.orchestra.run.smithi088.stderr:2024-06-03T09:07:01.134+0000 7f05b54a8640 5 monclient: authenticate success, global_id 30314 2024-06-03T09:07:01.136 INFO:teuthology.orchestra.run.smithi088.stderr:2024-06-03T09:07:01.134+0000 7f05b54a8640 10 monclient: _renew_subs 2024-06-03T09:07:01.136 INFO:teuthology.orchestra.run.smithi088.stderr:2024-06-03T09:07:01.134+0000 7f05b54a8640 10 monclient: _send_mon_message to mon.c at v1:172.21.15.88:6790/0 2024-06-03T09:07:01.136 INFO:teuthology.orchestra.run.smithi088.stderr:2024-06-03T09:07:01.134+0000 7f05b54a8640 1 -- 172.21.15.88:0/3717066664 --> v1:172.21.15.88:6790/0 -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f05b00b4680 con 0x7f05b006bdf0 2024-06-03T09:07:01.136 INFO:teuthology.orchestra.run.smithi088.stderr:2024-06-03T09:07:01.134+0000 7f05b54a8640 10 client.30314.objecter _maybe_request_map subscribing (onetime) to next osd map 2024-06-03T09:07:01.136 INFO:teuthology.orchestra.run.smithi088.stderr:2024-06-03T09:07:01.134+0000 7f05b54a8640 10 monclient: _renew_subs 2024-06-03T09:07:01.136 INFO:teuthology.orchestra.run.smithi088.stderr:2024-06-03T09:07:01.134+0000 7f05b54a8640 10 monclient: _send_mon_message to mon.c at v1:172.21.15.88:6790/0 2024-06-03T09:07:01.136 INFO:teuthology.orchestra.run.smithi088.stderr:2024-06-03T09:07:01.134+0000 7f05b54a8640 1 -- 172.21.15.88:0/3717066664 --> v1:172.21.15.88:6790/0 -- mon_subscribe({osdmap=0}) v3 -- 0x7f05b01e1f80 con 0x7f05b006bdf0 2024-06-03T09:07:01.136 INFO:teuthology.orchestra.run.smithi088.stderr:2024-06-03T09:07:01.134+0000 7f05b54a8640 1 librados: init done 2024-06-03T09:07:01.137 INFO:teuthology.orchestra.run.smithi088.stderr:2024-06-03T09:07:01.135+0000 7f05b54a8640 10 client.30314.objecter _submit_command 1 [{"prefix": "get_command_descriptions"}] 2024-06-03T09:07:01.137 INFO:teuthology.orchestra.run.smithi088.stderr:2024-06-03T09:07:01.135+0000 7f05b54a8640 15 client.30314.objecter _session_command_op_assign -1 1 2024-06-03T09:07:01.137 INFO:teuthology.orchestra.run.smithi088.stderr:2024-06-03T09:07:01.135+0000 7f05b54a8640 20 client.30314.objecter _get_session osd=-1 returning homeless 2024-06-03T09:07:01.137 INFO:teuthology.orchestra.run.smithi088.stderr:2024-06-03T09:07:01.135+0000 7f05b54a8640 10 client.30314.objecter _maybe_request_map subscribing (onetime) to next osd map 2024-06-03T09:07:01.137 INFO:teuthology.orchestra.run.smithi088.stderr:2024-06-03T09:07:01.135+0000 7f05b54a8640 10 monclient: _send_mon_message to mon.c at v1:172.21.15.88:6790/0 2024-06-03T09:07:01.137 INFO:teuthology.orchestra.run.smithi088.stderr:2024-06-03T09:07:01.135+0000 7f05b54a8640 1 -- 172.21.15.88:0/3717066664 --> v1:172.21.15.88:6790/0 -- mon_get_version(what=osdmap handle=1) v1 -- 0x7f05b01416b0 con 0x7f05b006bdf0 2024-06-03T09:07:01.151 INFO:teuthology.orchestra.run.smithi088.stderr:2024-06-03T09:07:01.151+0000 7f05a7fff640 1 -- 172.21.15.88:0/3717066664 <== mon.2 v1:172.21.15.88:6790/0 5 ==== config(0 keys) v1 ==== 4+0+0 (unknown 0 0 0) 0x7f0598004a90 con 0x7f05b006bdf0 2024-06-03T09:07:01.151 INFO:teuthology.orchestra.run.smithi088.stderr:2024-06-03T09:07:01.151+0000 7f05a7fff640 10 monclient: handle_config config(0 keys) v1 2024-06-03T09:07:01.152 INFO:teuthology.orchestra.run.smithi088.stderr:2024-06-03T09:07:01.151+0000 7f05a7fff640 1 -- 172.21.15.88:0/3717066664 <== mon.2 v1:172.21.15.88:6790/0 6 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 2234374065 0 0) 0x7f0598004e20 con 0x7f05b006bdf0 2024-06-03T09:07:01.152 INFO:teuthology.orchestra.run.smithi088.stderr:2024-06-03T09:07:01.151+0000 7f05a7fff640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-06-03T09:07:01.152 INFO:teuthology.orchestra.run.smithi088.stderr:2024-06-03T09:07:01.151+0000 7f05a7fff640 10 monclient: got monmap 1 from mon.c (according to old e1) 2024-06-03T09:07:01.152 INFO:teuthology.orchestra.run.smithi088.stderr:2024-06-03T09:07:01.151+0000 7f05a7fff640 10 monclient: dump: 2024-06-03T09:07:01.152 INFO:teuthology.orchestra.run.smithi088.stderr:epoch 1 2024-06-03T09:07:01.152 INFO:teuthology.orchestra.run.smithi088.stderr:fsid ec3c3fdd-3ac6-4d7d-9412-683130ed8b04 2024-06-03T09:07:01.152 INFO:teuthology.orchestra.run.smithi088.stderr:last_changed 2024-06-03T08:34:31.885125+0000 2024-06-03T09:07:01.152 INFO:teuthology.orchestra.run.smithi088.stderr:created 2024-06-03T08:34:31.885125+0000 2024-06-03T09:07:01.152 INFO:teuthology.orchestra.run.smithi088.stderr:min_mon_release 18 (reef) 2024-06-03T09:07:01.152 INFO:teuthology.orchestra.run.smithi088.stderr:election_strategy: 3 2024-06-03T09:07:01.152 INFO:teuthology.orchestra.run.smithi088.stderr:0: v1:172.21.15.88:6789/0 mon.a 2024-06-03T09:07:01.152 INFO:teuthology.orchestra.run.smithi088.stderr:1: v1:172.21.15.192:6789/0 mon.b 2024-06-03T09:07:01.153 INFO:teuthology.orchestra.run.smithi088.stderr:2: v1:172.21.15.88:6790/0 mon.c 2024-06-03T09:07:01.153 INFO:teuthology.orchestra.run.smithi088.stderr: 2024-06-03T09:07:01.153 INFO:teuthology.orchestra.run.smithi088.stderr:2024-06-03T09:07:01.151+0000 7f05adffb640 4 set_mon_vals no callback set 2024-06-03T09:07:01.153 INFO:teuthology.orchestra.run.smithi088.stderr:2024-06-03T09:07:01.152+0000 7f05a7fff640 1 -- 172.21.15.88:0/3717066664 <== mon.2 v1:172.21.15.88:6790/0 7 ==== mgrmap(e 5) v1 ==== 89110+0+0 (unknown 873533973 0 0) 0x7f0598004f80 con 0x7f05b006bdf0 2024-06-03T09:07:01.153 INFO:teuthology.orchestra.run.smithi088.stderr:2024-06-03T09:07:01.153+0000 7f05aeffd640 10 monclient: get_auth_request con 0x7f059006d200 auth_method 0 2024-06-03T09:07:01.154 INFO:teuthology.orchestra.run.smithi088.stderr:2024-06-03T09:07:01.153+0000 7f05a7fff640 10 client.30314.objecter ms_handle_connect 0x7f059006d200 2024-06-03T09:07:01.155 INFO:teuthology.orchestra.run.smithi088.stderr:2024-06-03T09:07:01.155+0000 7f05a7fff640 1 -- 172.21.15.88:0/3717066664 <== mon.2 v1:172.21.15.88:6790/0 8 ==== osd_map(1014..1014 src has 622..1014) v4 ==== 5229+0+0 (unknown 3230192161 0 0) 0x7f0598054260 con 0x7f05b006bdf0 2024-06-03T09:07:01.155 INFO:teuthology.orchestra.run.smithi088.stderr:2024-06-03T09:07:01.155+0000 7f05a7fff640 10 client.30314.objecter ms_dispatch 0x7f05b0000b90 osd_map(1014..1014 src has 622..1014) v4 2024-06-03T09:07:01.155 INFO:teuthology.orchestra.run.smithi088.stderr:2024-06-03T09:07:01.155+0000 7f05a7fff640 3 client.30314.objecter handle_osd_map got epochs [1014,1014] > 0 2024-06-03T09:07:01.155 INFO:teuthology.orchestra.run.smithi088.stderr:2024-06-03T09:07:01.155+0000 7f05a7fff640 3 client.30314.objecter handle_osd_map decoding full epoch 1014 2024-06-03T09:07:01.155 INFO:teuthology.orchestra.run.smithi088.stderr:2024-06-03T09:07:01.155+0000 7f05a7fff640 10 client.30314.objecter checking command 1 2024-06-03T09:07:01.155 INFO:teuthology.orchestra.run.smithi088.stderr:2024-06-03T09:07:01.155+0000 7f05a7fff640 20 client.30314.objecter _get_session s=0x7f0590075140 osd=6 3 2024-06-03T09:07:01.155 INFO:teuthology.orchestra.run.smithi088.stderr:2024-06-03T09:07:01.155+0000 7f05a7fff640 20 client.30314.objecter put_session s=0x7f0590075140 osd=6 3 2024-06-03T09:07:01.155 INFO:teuthology.orchestra.run.smithi088.stderr:2024-06-03T09:07:01.155+0000 7f05a7fff640 15 client.30314.objecter _session_command_op_remove -1 1 2024-06-03T09:07:01.155 INFO:teuthology.orchestra.run.smithi088.stderr:2024-06-03T09:07:01.155+0000 7f05a7fff640 20 client.30314.objecter _get_session s=0x7f0590075140 osd=6 3 2024-06-03T09:07:01.156 INFO:teuthology.orchestra.run.smithi088.stderr:2024-06-03T09:07:01.155+0000 7f05a7fff640 20 client.30314.objecter get_session s=0x7f0590075140 osd=6 3 2024-06-03T09:07:01.156 INFO:teuthology.orchestra.run.smithi088.stderr:2024-06-03T09:07:01.155+0000 7f05a7fff640 15 client.30314.objecter _session_command_op_assign 6 1 2024-06-03T09:07:01.156 INFO:teuthology.orchestra.run.smithi088.stderr:2024-06-03T09:07:01.155+0000 7f05a7fff640 20 client.30314.objecter put_session s=0x7f0590075140 osd=6 4 2024-06-03T09:07:01.156 INFO:teuthology.orchestra.run.smithi088.stderr:2024-06-03T09:07:01.155+0000 7f05a7fff640 10 client.30314.objecter _send_command 1 2024-06-03T09:07:01.156 INFO:teuthology.orchestra.run.smithi088.stderr:2024-06-03T09:07:01.155+0000 7f05a7fff640 1 -- 172.21.15.88:0/3717066664 --> v1:172.21.15.192:6813/34788 -- command(tid 1: {"prefix": "get_command_descriptions"}) v1 -- 0x7f0590079500 con 0x7f05900757f0 2024-06-03T09:07:01.156 INFO:teuthology.orchestra.run.smithi088.stderr:2024-06-03T09:07:01.155+0000 7f05a7fff640 20 client.30314.objecter dump_active .. 0 homeless 2024-06-03T09:07:01.156 INFO:teuthology.orchestra.run.smithi088.stderr:2024-06-03T09:07:01.155+0000 7f05a7fff640 1 -- 172.21.15.88:0/3717066664 <== mon.2 v1:172.21.15.88:6790/0 9 ==== mon_get_version_reply(handle=1 version=1014) v2 ==== 24+0+0 (unknown 3501658476 0 0) 0x7f0598072710 con 0x7f05b006bdf0 2024-06-03T09:07:01.156 INFO:teuthology.orchestra.run.smithi088.stderr:2024-06-03T09:07:01.155+0000 7f05a7fff640 10 monclient: handle_get_version_reply finishing 1 version 1014 2024-06-03T09:07:01.156 INFO:teuthology.orchestra.run.smithi088.stderr:2024-06-03T09:07:01.155+0000 7f05af7fe640 10 monclient: get_auth_request con 0x7f05900757f0 auth_method 0 2024-06-03T09:07:01.157 INFO:teuthology.orchestra.run.smithi088.stderr:2024-06-03T09:07:01.157+0000 7f05a7fff640 10 client.30314.objecter ms_handle_connect 0x7f05900757f0 2024-06-03T09:07:01.180 INFO:teuthology.orchestra.run.smithi088.stderr:2024-06-03T09:07:01.180+0000 7f05a7fff640 1 -- 172.21.15.88:0/3717066664 <== osd.6 v1:172.21.15.192:6813/34788 1 ==== command_reply(tid 1: 0 ) v1 ==== 8+0+26118 (unknown 0 0 1686565167) 0x7f0590079500 con 0x7f05900757f0 2024-06-03T09:07:01.180 INFO:teuthology.orchestra.run.smithi088.stderr:2024-06-03T09:07:01.180+0000 7f05a7fff640 10 client.30314.objecter ms_dispatch 0x7f05b0000b90 command_reply(tid 1: 0 ) v1 2024-06-03T09:07:01.180 INFO:teuthology.orchestra.run.smithi088.stderr:2024-06-03T09:07:01.180+0000 7f05a7fff640 10 client.30314.objecter _finish_command 1 = system:0 2024-06-03T09:07:01.180 INFO:teuthology.orchestra.run.smithi088.stderr:2024-06-03T09:07:01.180+0000 7f05a7fff640 20 client.30314.objecter put_session s=0x7f0590075140 osd=6 4 2024-06-03T09:07:01.180 INFO:teuthology.orchestra.run.smithi088.stderr:2024-06-03T09:07:01.180+0000 7f05a7fff640 15 client.30314.objecter _session_command_op_remove 6 1 2024-06-03T09:07:01.192 INFO:teuthology.orchestra.run.smithi088.stderr:2024-06-03T09:07:01.192+0000 7f05b54a8640 10 client.30314.objecter _submit_command 2 [{"prefix": "config set", "var": "osd_debug_deep_scrub_sleep", "val": ["0"]}] 2024-06-03T09:07:01.192 INFO:teuthology.orchestra.run.smithi088.stderr:2024-06-03T09:07:01.192+0000 7f05b54a8640 15 client.30314.objecter _session_command_op_assign -1 2 2024-06-03T09:07:01.192 INFO:teuthology.orchestra.run.smithi088.stderr:2024-06-03T09:07:01.192+0000 7f05b54a8640 20 client.30314.objecter _get_session s=0x7f0590075140 osd=6 3 2024-06-03T09:07:01.192 INFO:teuthology.orchestra.run.smithi088.stderr:2024-06-03T09:07:01.192+0000 7f05b54a8640 20 client.30314.objecter put_session s=0x7f0590075140 osd=6 3 2024-06-03T09:07:01.192 INFO:teuthology.orchestra.run.smithi088.stderr:2024-06-03T09:07:01.192+0000 7f05b54a8640 20 client.30314.objecter _get_session s=0x7f0590075140 osd=6 3 2024-06-03T09:07:01.192 INFO:teuthology.orchestra.run.smithi088.stderr:2024-06-03T09:07:01.192+0000 7f05b54a8640 15 client.30314.objecter _session_command_op_remove -1 2 2024-06-03T09:07:01.192 INFO:teuthology.orchestra.run.smithi088.stderr:2024-06-03T09:07:01.192+0000 7f05b54a8640 20 client.30314.objecter get_session s=0x7f0590075140 osd=6 3 2024-06-03T09:07:01.192 INFO:teuthology.orchestra.run.smithi088.stderr:2024-06-03T09:07:01.192+0000 7f05b54a8640 15 client.30314.objecter _session_command_op_assign 6 2 2024-06-03T09:07:01.192 INFO:teuthology.orchestra.run.smithi088.stderr:2024-06-03T09:07:01.192+0000 7f05b54a8640 20 client.30314.objecter put_session s=0x7f0590075140 osd=6 4 2024-06-03T09:07:01.192 INFO:teuthology.orchestra.run.smithi088.stderr:2024-06-03T09:07:01.192+0000 7f05b54a8640 10 client.30314.objecter _send_command 2 2024-06-03T09:07:01.193 INFO:teuthology.orchestra.run.smithi088.stderr:2024-06-03T09:07:01.192+0000 7f05b54a8640 1 -- 172.21.15.88:0/3717066664 --> v1:172.21.15.192:6813/34788 -- command(tid 2: {"prefix": "config set", "var": "osd_debug_deep_scrub_sleep", "val": ["0"]}) v1 -- 0x7f05b00b4c40 con 0x7f05900757f0 2024-06-03T09:07:01.207 INFO:teuthology.orchestra.run.smithi088.stderr:2024-06-03T09:07:01.207+0000 7f05a7fff640 1 -- 172.21.15.88:0/3717066664 <== osd.6 v1:172.21.15.192:6813/34788 2 ==== command_reply(tid 2: 0 ) v1 ==== 8+0+97 (unknown 0 0 119112422) 0x7f05b00b4c40 con 0x7f05900757f0 2024-06-03T09:07:01.207 INFO:teuthology.orchestra.run.smithi088.stderr:2024-06-03T09:07:01.207+0000 7f05a7fff640 10 client.30314.objecter ms_dispatch 0x7f05b0000b90 command_reply(tid 2: 0 ) v1 2024-06-03T09:07:01.207 INFO:teuthology.orchestra.run.smithi088.stderr:2024-06-03T09:07:01.207+0000 7f05a7fff640 10 client.30314.objecter _finish_command 2 = system:0 2024-06-03T09:07:01.207 INFO:teuthology.orchestra.run.smithi088.stderr:2024-06-03T09:07:01.207+0000 7f05a7fff640 20 client.30314.objecter put_session s=0x7f0590075140 osd=6 4 2024-06-03T09:07:01.207 INFO:teuthology.orchestra.run.smithi088.stderr:2024-06-03T09:07:01.207+0000 7f05a7fff640 15 client.30314.objecter _session_command_op_remove 6 2 2024-06-03T09:07:01.208 INFO:teuthology.orchestra.run.smithi088.stdout:{ 2024-06-03T09:07:01.208 INFO:teuthology.orchestra.run.smithi088.stdout: "success": "osd_debug_deep_scrub_sleep = '' (not observed, change may require restart) " 2024-06-03T09:07:01.208 INFO:teuthology.orchestra.run.smithi088.stdout:} 2024-06-03T09:07:01.209 INFO:teuthology.orchestra.run.smithi088.stderr:2024-06-03T09:07:01.207+0000 7f05b54a8640 10 librados: watch_flush enter 2024-06-03T09:07:01.210 INFO:teuthology.orchestra.run.smithi088.stderr:2024-06-03T09:07:01.207+0000 7f05b54a8640 10 librados: watch_flush exit 2024-06-03T09:07:01.210 INFO:teuthology.orchestra.run.smithi088.stderr:2024-06-03T09:07:01.207+0000 7f05b54a8640 10 client.30314.objecter close_session for osd.6 2024-06-03T09:07:01.210 INFO:teuthology.orchestra.run.smithi088.stderr:2024-06-03T09:07:01.207+0000 7f05b54a8640 1 -- 172.21.15.88:0/3717066664 >> v1:172.21.15.192:6813/34788 conn(0x7f05900757f0 legacy=0x7f0590077c90 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-06-03T09:07:01.210 INFO:teuthology.orchestra.run.smithi088.stderr:2024-06-03T09:07:01.207+0000 7f05b54a8640 20 client.30314.objecter put_session s=0x7f0590075140 osd=6 1 2024-06-03T09:07:01.210 INFO:teuthology.orchestra.run.smithi088.stderr:2024-06-03T09:07:01.207+0000 7f05b54a8640 20 client.30314.objecter shutdown clearing up homeless session... 2024-06-03T09:07:01.210 INFO:teuthology.orchestra.run.smithi088.stderr:2024-06-03T09:07:01.207+0000 7f05b54a8640 10 client.30314.objecter successfully canceled tick 2024-06-03T09:07:01.210 INFO:teuthology.orchestra.run.smithi088.stderr:2024-06-03T09:07:01.207+0000 7f05b54a8640 1 -- 172.21.15.88:0/3717066664 >> v1:172.21.15.192:6800/34229 conn(0x7f059006d200 legacy=0x7f059006f6c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-06-03T09:07:01.210 INFO:teuthology.orchestra.run.smithi088.stderr:2024-06-03T09:07:01.207+0000 7f05b54a8640 10 monclient: shutdown 2024-06-03T09:07:01.210 INFO:teuthology.orchestra.run.smithi088.stderr:2024-06-03T09:07:01.207+0000 7f05b54a8640 20 monclient: shutdown discarding 0 pending message(s) 2024-06-03T09:07:01.210 INFO:teuthology.orchestra.run.smithi088.stderr:2024-06-03T09:07:01.207+0000 7f05b54a8640 1 -- 172.21.15.88:0/3717066664 >> v1:172.21.15.88:6790/0 conn(0x7f05b006bdf0 legacy=0x7f05b01d62e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-06-03T09:07:01.210 INFO:teuthology.orchestra.run.smithi088.stderr:2024-06-03T09:07:01.207+0000 7f05af7fe640 1 -- 172.21.15.88:0/3717066664 reap_dead start 2024-06-03T09:07:01.210 INFO:teuthology.orchestra.run.smithi088.stderr:2024-06-03T09:07:01.207+0000 7f05b54a8640 1 -- 172.21.15.88:0/3717066664 shutdown_connections 2024-06-03T09:07:01.211 INFO:teuthology.orchestra.run.smithi088.stderr:2024-06-03T09:07:01.207+0000 7f05b54a8640 1 -- 172.21.15.88:0/3717066664 >> 172.21.15.88:0/3717066664 conn(0x7f05b00b4fb0 msgr2=0x7f05b00b0770 unknown :-1 s=STATE_NONE l=0).mark_down 2024-06-03T09:07:01.211 INFO:teuthology.orchestra.run.smithi088.stderr:2024-06-03T09:07:01.208+0000 7f05b54a8640 1 -- 172.21.15.88:0/3717066664 shutdown_connections 2024-06-03T09:07:01.211 INFO:teuthology.orchestra.run.smithi088.stderr:2024-06-03T09:07:01.208+0000 7f05b54a8640 1 -- 172.21.15.88:0/3717066664 wait complete. 2024-06-03T09:07:01.211 INFO:teuthology.orchestra.run.smithi088.stderr:2024-06-03T09:07:01.208+0000 7f05b54a8640 1 librados: shutdown 2024-06-03T09:07:01.218 DEBUG:teuthology.orchestra.run.smithi088:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd deep-scrub 6 2024-06-03T09:07:01.318 INFO:teuthology.orchestra.run.smithi088.stderr:2024-06-03T09:07:01.317+0000 7efc84e77640 10 monclient: get_monmap_and_config 2024-06-03T09:07:01.318 INFO:teuthology.orchestra.run.smithi088.stderr:2024-06-03T09:07:01.317+0000 7efc84e77640 10 monclient: build_initial_monmap 2024-06-03T09:07:01.318 INFO:teuthology.orchestra.run.smithi088.stderr:2024-06-03T09:07:01.317+0000 7efc84e77640 10 monclient: monmap: 2024-06-03T09:07:01.318 INFO:teuthology.orchestra.run.smithi088.stderr:epoch 0 2024-06-03T09:07:01.318 INFO:teuthology.orchestra.run.smithi088.stderr:fsid ec3c3fdd-3ac6-4d7d-9412-683130ed8b04 2024-06-03T09:07:01.318 INFO:teuthology.orchestra.run.smithi088.stderr:last_changed 2024-06-03T09:07:01.318980+0000 2024-06-03T09:07:01.318 INFO:teuthology.orchestra.run.smithi088.stderr:created 2024-06-03T09:07:01.318980+0000 2024-06-03T09:07:01.318 INFO:teuthology.orchestra.run.smithi088.stderr:min_mon_release 0 (unknown) 2024-06-03T09:07:01.319 INFO:teuthology.orchestra.run.smithi088.stderr:election_strategy: 3 2024-06-03T09:07:01.319 INFO:teuthology.orchestra.run.smithi088.stderr:0: v1:172.21.15.88:6789/0 mon.noname-a 2024-06-03T09:07:01.319 INFO:teuthology.orchestra.run.smithi088.stderr:1: v1:172.21.15.192:6789/0 mon.noname-c 2024-06-03T09:07:01.319 INFO:teuthology.orchestra.run.smithi088.stderr:2: v1:172.21.15.88:6790/0 mon.noname-b 2024-06-03T09:07:01.319 INFO:teuthology.orchestra.run.smithi088.stderr: 2024-06-03T09:07:01.320 INFO:teuthology.orchestra.run.smithi088.stderr:2024-06-03T09:07:01.318+0000 7efc84e77640 1 Processor -- start 2024-06-03T09:07:01.321 INFO:teuthology.orchestra.run.smithi088.stderr:2024-06-03T09:07:01.318+0000 7efc84e77640 1 -- start start 2024-06-03T09:07:01.321 INFO:teuthology.orchestra.run.smithi088.stderr:2024-06-03T09:07:01.318+0000 7efc84e77640 10 monclient: init 2024-06-03T09:07:01.321 INFO:teuthology.orchestra.run.smithi088.stderr:2024-06-03T09:07:01.319+0000 7efc84e77640 10 monclient: _reopen_session rank -1 2024-06-03T09:07:01.321 INFO:teuthology.orchestra.run.smithi088.stderr:2024-06-03T09:07:01.319+0000 7efc84e77640 10 monclient: _add_conns ranks=[2,1,0] 2024-06-03T09:07:01.321 INFO:teuthology.orchestra.run.smithi088.stderr:2024-06-03T09:07:01.319+0000 7efc84e77640 10 monclient(hunting): picked mon.noname-b con 0x7efc8006e050 addr v1:172.21.15.88:6790/0 2024-06-03T09:07:01.321 INFO:teuthology.orchestra.run.smithi088.stderr:2024-06-03T09:07:01.319+0000 7efc84e77640 10 monclient(hunting): picked mon.noname-c con 0x7efc8006e430 addr v1:172.21.15.192:6789/0 2024-06-03T09:07:01.321 INFO:teuthology.orchestra.run.smithi088.stderr:2024-06-03T09:07:01.319+0000 7efc84e77640 10 monclient(hunting): picked mon.noname-a con 0x7efc800717c0 addr v1:172.21.15.88:6789/0 2024-06-03T09:07:01.321 INFO:teuthology.orchestra.run.smithi088.stderr:2024-06-03T09:07:01.319+0000 7efc84e77640 1 -- --> v1:172.21.15.88:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7efc80070c10 con 0x7efc800717c0 2024-06-03T09:07:01.321 INFO:teuthology.orchestra.run.smithi088.stderr:2024-06-03T09:07:01.319+0000 7efc84e77640 1 -- --> v1:172.21.15.192:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7efc80070230 con 0x7efc8006e430 2024-06-03T09:07:01.321 INFO:teuthology.orchestra.run.smithi088.stderr:2024-06-03T09:07:01.319+0000 7efc84e77640 1 -- --> v1:172.21.15.88:6790/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7efc80070510 con 0x7efc8006e050 2024-06-03T09:07:01.321 INFO:teuthology.orchestra.run.smithi088.stderr:2024-06-03T09:07:01.319+0000 7efc84e77640 10 monclient(hunting): _renew_subs 2024-06-03T09:07:01.321 INFO:teuthology.orchestra.run.smithi088.stderr:2024-06-03T09:07:01.319+0000 7efc84e77640 10 monclient(hunting): authenticate will time out at 3077.129430s 2024-06-03T09:07:01.321 INFO:teuthology.orchestra.run.smithi088.stderr:2024-06-03T09:07:01.319+0000 7efc7e59c640 1 --1- >> v1:172.21.15.88:6790/0 conn(0x7efc8006e050 0x7efc8013ad00 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.88:6790/0 says I am v1:172.21.15.88:51170/0 (socket says 172.21.15.88:51170) 2024-06-03T09:07:01.322 INFO:teuthology.orchestra.run.smithi088.stderr:2024-06-03T09:07:01.319+0000 7efc7e59c640 1 -- 172.21.15.88:0/1217553920 learned_addr learned my addr 172.21.15.88:0/1217553920 (peer_addr_for_me v1:172.21.15.88:0/0) 2024-06-03T09:07:01.322 INFO:teuthology.orchestra.run.smithi088.stderr:2024-06-03T09:07:01.322+0000 7efc7d59a640 1 -- 172.21.15.88:0/1217553920 <== mon.1 v1:172.21.15.192:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 3757243824 0 0) 0x7efc80070230 con 0x7efc8006e430 2024-06-03T09:07:01.323 INFO:teuthology.orchestra.run.smithi088.stderr:2024-06-03T09:07:01.322+0000 7efc7d59a640 10 monclient(hunting): _negotiate 2024-06-03T09:07:01.323 INFO:teuthology.orchestra.run.smithi088.stderr:2024-06-03T09:07:01.322+0000 7efc7d59a640 10 monclient(hunting): _init_auth method 2 2024-06-03T09:07:01.323 INFO:teuthology.orchestra.run.smithi088.stderr:2024-06-03T09:07:01.322+0000 7efc7d59a640 10 monclient(hunting): _init_auth creating new auth 2024-06-03T09:07:01.323 INFO:teuthology.orchestra.run.smithi088.stderr:2024-06-03T09:07:01.322+0000 7efc7d59a640 10 monclient(hunting): my global_id is 30382 2024-06-03T09:07:01.323 INFO:teuthology.orchestra.run.smithi088.stderr:2024-06-03T09:07:01.322+0000 7efc7d59a640 1 -- 172.21.15.88:0/1217553920 --> v1:172.21.15.192:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7efc68003520 con 0x7efc8006e430 2024-06-03T09:07:01.323 INFO:teuthology.orchestra.run.smithi088.stderr:2024-06-03T09:07:01.322+0000 7efc7d59a640 1 -- 172.21.15.88:0/1217553920 <== mon.2 v1:172.21.15.88:6790/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 721941754 0 0) 0x7efc80070510 con 0x7efc8006e050 2024-06-03T09:07:01.323 INFO:teuthology.orchestra.run.smithi088.stderr:2024-06-03T09:07:01.322+0000 7efc7d59a640 10 monclient(hunting): _negotiate 2024-06-03T09:07:01.323 INFO:teuthology.orchestra.run.smithi088.stderr:2024-06-03T09:07:01.322+0000 7efc7d59a640 10 monclient(hunting): _init_auth method 2 2024-06-03T09:07:01.323 INFO:teuthology.orchestra.run.smithi088.stderr:2024-06-03T09:07:01.322+0000 7efc7d59a640 10 monclient(hunting): _init_auth creating new auth 2024-06-03T09:07:01.323 INFO:teuthology.orchestra.run.smithi088.stderr:2024-06-03T09:07:01.322+0000 7efc7d59a640 10 monclient(hunting): my global_id is 30317 2024-06-03T09:07:01.323 INFO:teuthology.orchestra.run.smithi088.stderr:2024-06-03T09:07:01.322+0000 7efc7d59a640 1 -- 172.21.15.88:0/1217553920 --> v1:172.21.15.88:6790/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7efc80070230 con 0x7efc8006e050 2024-06-03T09:07:01.324 INFO:teuthology.orchestra.run.smithi088.stderr:2024-06-03T09:07:01.323+0000 7efc7d59a640 1 -- 172.21.15.88:0/1217553920 <== mon.0 v1:172.21.15.88:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 800504597 0 0) 0x7efc80070c10 con 0x7efc800717c0 2024-06-03T09:07:01.324 INFO:teuthology.orchestra.run.smithi088.stderr:2024-06-03T09:07:01.323+0000 7efc7d59a640 10 monclient(hunting): _negotiate 2024-06-03T09:07:01.324 INFO:teuthology.orchestra.run.smithi088.stderr:2024-06-03T09:07:01.323+0000 7efc7d59a640 10 monclient(hunting): _init_auth method 2 2024-06-03T09:07:01.324 INFO:teuthology.orchestra.run.smithi088.stderr:2024-06-03T09:07:01.323+0000 7efc7d59a640 10 monclient(hunting): _init_auth creating new auth 2024-06-03T09:07:01.324 INFO:teuthology.orchestra.run.smithi088.stderr:2024-06-03T09:07:01.323+0000 7efc7d59a640 10 monclient(hunting): my global_id is 30807 2024-06-03T09:07:01.324 INFO:teuthology.orchestra.run.smithi088.stderr:2024-06-03T09:07:01.323+0000 7efc7d59a640 1 -- 172.21.15.88:0/1217553920 --> v1:172.21.15.88:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7efc80070510 con 0x7efc800717c0 2024-06-03T09:07:01.326 INFO:teuthology.orchestra.run.smithi088.stderr:2024-06-03T09:07:01.325+0000 7efc7d59a640 1 -- 172.21.15.88:0/1217553920 <== mon.1 v1:172.21.15.192:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 214+0+0 (unknown 1420806814 0 0) 0x7efc68003520 con 0x7efc8006e430 2024-06-03T09:07:01.326 INFO:teuthology.orchestra.run.smithi088.stderr:2024-06-03T09:07:01.325+0000 7efc7d59a640 1 -- 172.21.15.88:0/1217553920 >> v1:172.21.15.88:6790/0 conn(0x7efc8006e050 legacy=0x7efc8013ad00 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-06-03T09:07:01.326 INFO:teuthology.orchestra.run.smithi088.stderr:2024-06-03T09:07:01.325+0000 7efc7d59a640 1 -- 172.21.15.88:0/1217553920 >> v1:172.21.15.88:6789/0 conn(0x7efc800717c0 legacy=0x7efc80149240 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-06-03T09:07:01.326 INFO:teuthology.orchestra.run.smithi088.stderr:2024-06-03T09:07:01.325+0000 7efc7d59a640 10 monclient: _finish_hunting 0 2024-06-03T09:07:01.326 INFO:teuthology.orchestra.run.smithi088.stderr:2024-06-03T09:07:01.325+0000 7efc7d59a640 1 monclient: found mon.noname-c 2024-06-03T09:07:01.326 INFO:teuthology.orchestra.run.smithi088.stderr:2024-06-03T09:07:01.325+0000 7efc7d59a640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-06-03T09:07:01.326 INFO:teuthology.orchestra.run.smithi088.stderr:2024-06-03T09:07:01.325+0000 7efc7d59a640 10 monclient: _send_mon_message to mon.noname-c at v1:172.21.15.192:6789/0 2024-06-03T09:07:01.326 INFO:teuthology.orchestra.run.smithi088.stderr:2024-06-03T09:07:01.325+0000 7efc7d59a640 1 -- 172.21.15.88:0/1217553920 --> v1:172.21.15.192:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7efc800707f0 con 0x7efc8006e430 2024-06-03T09:07:01.327 INFO:teuthology.orchestra.run.smithi088.stderr:2024-06-03T09:07:01.325+0000 7efc7d59a640 10 monclient: _finish_auth 0 2024-06-03T09:07:01.327 INFO:teuthology.orchestra.run.smithi088.stderr:2024-06-03T09:07:01.325+0000 7efc7d59a640 10 monclient: _check_auth_tickets 2024-06-03T09:07:01.327 INFO:teuthology.orchestra.run.smithi088.stderr:2024-06-03T09:07:01.325+0000 7efc7d59a640 20 monclient: _check_auth_rotating not needed by client.admin 2024-06-03T09:07:01.327 INFO:teuthology.orchestra.run.smithi088.stderr:2024-06-03T09:07:01.325+0000 7efc7d59a640 1 -- 172.21.15.88:0/1217553920 <== mon.1 v1:172.21.15.192:6789/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 2234374065 0 0) 0x7efc74001610 con 0x7efc8006e430 2024-06-03T09:07:01.327 INFO:teuthology.orchestra.run.smithi088.stderr:2024-06-03T09:07:01.325+0000 7efc84e77640 5 monclient: authenticate success, global_id 30382 2024-06-03T09:07:01.327 INFO:teuthology.orchestra.run.smithi088.stderr:2024-06-03T09:07:01.325+0000 7efc84e77640 20 monclient: get_monmap_and_config waiting for monmap|config 2024-06-03T09:07:01.327 INFO:teuthology.orchestra.run.smithi088.stderr:2024-06-03T09:07:01.325+0000 7efc7d59a640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-06-03T09:07:01.327 INFO:teuthology.orchestra.run.smithi088.stderr:2024-06-03T09:07:01.325+0000 7efc7d59a640 10 monclient: got monmap 1 from mon.noname-c (according to old e1) 2024-06-03T09:07:01.327 INFO:teuthology.orchestra.run.smithi088.stderr:2024-06-03T09:07:01.325+0000 7efc7d59a640 10 monclient: dump: 2024-06-03T09:07:01.327 INFO:teuthology.orchestra.run.smithi088.stderr:epoch 1 2024-06-03T09:07:01.327 INFO:teuthology.orchestra.run.smithi088.stderr:fsid ec3c3fdd-3ac6-4d7d-9412-683130ed8b04 2024-06-03T09:07:01.327 INFO:teuthology.orchestra.run.smithi088.stderr:last_changed 2024-06-03T08:34:31.885125+0000 2024-06-03T09:07:01.328 INFO:teuthology.orchestra.run.smithi088.stderr:created 2024-06-03T08:34:31.885125+0000 2024-06-03T09:07:01.328 INFO:teuthology.orchestra.run.smithi088.stderr:min_mon_release 18 (reef) 2024-06-03T09:07:01.328 INFO:teuthology.orchestra.run.smithi088.stderr:election_strategy: 3 2024-06-03T09:07:01.328 INFO:teuthology.orchestra.run.smithi088.stderr:0: v1:172.21.15.88:6789/0 mon.a 2024-06-03T09:07:01.328 INFO:teuthology.orchestra.run.smithi088.stderr:1: v1:172.21.15.192:6789/0 mon.b 2024-06-03T09:07:01.328 INFO:teuthology.orchestra.run.smithi088.stderr:2: v1:172.21.15.88:6790/0 mon.c 2024-06-03T09:07:01.328 INFO:teuthology.orchestra.run.smithi088.stderr: 2024-06-03T09:07:01.328 INFO:teuthology.orchestra.run.smithi088.stderr:2024-06-03T09:07:01.325+0000 7efc84e77640 20 monclient: get_monmap_and_config waiting for monmap|config 2024-06-03T09:07:01.331 INFO:teuthology.orchestra.run.smithi088.stderr:2024-06-03T09:07:01.327+0000 7efc7d59a640 1 -- 172.21.15.88:0/1217553920 <== mon.1 v1:172.21.15.192:6789/0 4 ==== config(0 keys) v1 ==== 4+0+0 (unknown 0 0 0) 0x7efc74001830 con 0x7efc8006e430 2024-06-03T09:07:01.331 INFO:teuthology.orchestra.run.smithi088.stderr:2024-06-03T09:07:01.327+0000 7efc7d59a640 10 monclient: handle_config config(0 keys) v1 2024-06-03T09:07:01.332 INFO:teuthology.orchestra.run.smithi088.stderr:2024-06-03T09:07:01.327+0000 7efc7d59a640 1 -- 172.21.15.88:0/1217553920 <== mon.1 v1:172.21.15.192:6789/0 5 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 2234374065 0 0) 0x7efc74001bc0 con 0x7efc8006e430 2024-06-03T09:07:01.332 INFO:teuthology.orchestra.run.smithi088.stderr:2024-06-03T09:07:01.327+0000 7efc84e77640 10 monclient: get_monmap_and_config success 2024-06-03T09:07:01.332 INFO:teuthology.orchestra.run.smithi088.stderr:2024-06-03T09:07:01.327+0000 7efc84e77640 4 set_mon_vals no callback set 2024-06-03T09:07:01.332 INFO:teuthology.orchestra.run.smithi088.stderr:2024-06-03T09:07:01.327+0000 7efc7d59a640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-06-03T09:07:01.332 INFO:teuthology.orchestra.run.smithi088.stderr:2024-06-03T09:07:01.327+0000 7efc7d59a640 10 monclient: got monmap 1 from mon.b (according to old e1) 2024-06-03T09:07:01.332 INFO:teuthology.orchestra.run.smithi088.stderr:2024-06-03T09:07:01.327+0000 7efc7d59a640 10 monclient: dump: 2024-06-03T09:07:01.332 INFO:teuthology.orchestra.run.smithi088.stderr:epoch 1 2024-06-03T09:07:01.332 INFO:teuthology.orchestra.run.smithi088.stderr:fsid ec3c3fdd-3ac6-4d7d-9412-683130ed8b04 2024-06-03T09:07:01.332 INFO:teuthology.orchestra.run.smithi088.stderr:last_changed 2024-06-03T08:34:31.885125+0000 2024-06-03T09:07:01.332 INFO:teuthology.orchestra.run.smithi088.stderr:created 2024-06-03T08:34:31.885125+0000 2024-06-03T09:07:01.332 INFO:teuthology.orchestra.run.smithi088.stderr:min_mon_release 18 (reef) 2024-06-03T09:07:01.332 INFO:teuthology.orchestra.run.smithi088.stderr:election_strategy: 3 2024-06-03T09:07:01.332 INFO:teuthology.orchestra.run.smithi088.stderr:0: v1:172.21.15.88:6789/0 mon.a 2024-06-03T09:07:01.333 INFO:teuthology.orchestra.run.smithi088.stderr:1: v1:172.21.15.192:6789/0 mon.b 2024-06-03T09:07:01.333 INFO:teuthology.orchestra.run.smithi088.stderr:2: v1:172.21.15.88:6790/0 mon.c 2024-06-03T09:07:01.333 INFO:teuthology.orchestra.run.smithi088.stderr: 2024-06-03T09:07:01.333 INFO:teuthology.orchestra.run.smithi088.stderr:2024-06-03T09:07:01.328+0000 7efc84e77640 10 monclient: shutdown 2024-06-03T09:07:01.333 INFO:teuthology.orchestra.run.smithi088.stderr:2024-06-03T09:07:01.328+0000 7efc84e77640 20 monclient: shutdown discarding 0 pending message(s) 2024-06-03T09:07:01.333 INFO:teuthology.orchestra.run.smithi088.stderr:2024-06-03T09:07:01.328+0000 7efc84e77640 1 -- 172.21.15.88:0/1217553920 >> v1:172.21.15.192:6789/0 conn(0x7efc8006e430 legacy=0x7efc800710b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-06-03T09:07:01.333 INFO:teuthology.orchestra.run.smithi088.stderr:2024-06-03T09:07:01.328+0000 7efc84e77640 1 -- 172.21.15.88:0/1217553920 shutdown_connections 2024-06-03T09:07:01.333 INFO:teuthology.orchestra.run.smithi088.stderr:2024-06-03T09:07:01.328+0000 7efc84e77640 1 -- 172.21.15.88:0/1217553920 >> 172.21.15.88:0/1217553920 conn(0x7efc800b2f20 msgr2=0x7efc800b3300 unknown :-1 s=STATE_NONE l=0).mark_down 2024-06-03T09:07:01.333 INFO:teuthology.orchestra.run.smithi088.stderr:2024-06-03T09:07:01.328+0000 7efc84e77640 1 -- 172.21.15.88:0/1217553920 shutdown_connections 2024-06-03T09:07:01.333 INFO:teuthology.orchestra.run.smithi088.stderr:2024-06-03T09:07:01.328+0000 7efc84e77640 1 -- 172.21.15.88:0/1217553920 wait complete. 2024-06-03T09:07:01.333 INFO:teuthology.orchestra.run.smithi088.stderr:2024-06-03T09:07:01.329+0000 7efc84e77640 10 monclient: build_initial_monmap 2024-06-03T09:07:01.333 INFO:teuthology.orchestra.run.smithi088.stderr:2024-06-03T09:07:01.329+0000 7efc84e77640 10 monclient: monmap: 2024-06-03T09:07:01.333 INFO:teuthology.orchestra.run.smithi088.stderr:epoch 0 2024-06-03T09:07:01.334 INFO:teuthology.orchestra.run.smithi088.stderr:fsid 00000000-0000-0000-0000-000000000000 2024-06-03T09:07:01.334 INFO:teuthology.orchestra.run.smithi088.stderr:last_changed 0.000000 2024-06-03T09:07:01.334 INFO:teuthology.orchestra.run.smithi088.stderr:created 0.000000 2024-06-03T09:07:01.334 INFO:teuthology.orchestra.run.smithi088.stderr:min_mon_release 0 (unknown) 2024-06-03T09:07:01.334 INFO:teuthology.orchestra.run.smithi088.stderr:election_strategy: 1 2024-06-03T09:07:01.334 INFO:teuthology.orchestra.run.smithi088.stderr:0: v1:172.21.15.88:6789/0 mon.noname-a 2024-06-03T09:07:01.334 INFO:teuthology.orchestra.run.smithi088.stderr:1: v1:172.21.15.192:6789/0 mon.noname-b 2024-06-03T09:07:01.334 INFO:teuthology.orchestra.run.smithi088.stderr:2: v1:172.21.15.88:6790/0 mon.noname-c 2024-06-03T09:07:01.334 INFO:teuthology.orchestra.run.smithi088.stderr: 2024-06-03T09:07:01.334 INFO:teuthology.orchestra.run.smithi088.stderr:2024-06-03T09:07:01.329+0000 7efc84e77640 1 librados: starting msgr at 2024-06-03T09:07:01.334 INFO:teuthology.orchestra.run.smithi088.stderr:2024-06-03T09:07:01.329+0000 7efc84e77640 1 librados: starting objecter 2024-06-03T09:07:01.334 INFO:teuthology.orchestra.run.smithi088.stderr:2024-06-03T09:07:01.329+0000 7efc84e77640 1 Processor -- start 2024-06-03T09:07:01.334 INFO:teuthology.orchestra.run.smithi088.stderr:2024-06-03T09:07:01.329+0000 7efc84e77640 1 -- start start 2024-06-03T09:07:01.334 INFO:teuthology.orchestra.run.smithi088.stderr:2024-06-03T09:07:01.329+0000 7efc84e77640 1 librados: setting wanted keys 2024-06-03T09:07:01.335 INFO:teuthology.orchestra.run.smithi088.stderr:2024-06-03T09:07:01.329+0000 7efc84e77640 1 librados: calling monclient init 2024-06-03T09:07:01.335 INFO:teuthology.orchestra.run.smithi088.stderr:2024-06-03T09:07:01.329+0000 7efc84e77640 10 monclient: init 2024-06-03T09:07:01.335 INFO:teuthology.orchestra.run.smithi088.stderr:2024-06-03T09:07:01.330+0000 7efc84e77640 10 monclient: _reopen_session rank -1 2024-06-03T09:07:01.335 INFO:teuthology.orchestra.run.smithi088.stderr:2024-06-03T09:07:01.330+0000 7efc84e77640 10 monclient: _add_conns ranks=[0,1,2] 2024-06-03T09:07:01.335 INFO:teuthology.orchestra.run.smithi088.stderr:2024-06-03T09:07:01.330+0000 7efc84e77640 10 monclient(hunting): picked mon.noname-a con 0x7efc8006e050 addr v1:172.21.15.88:6789/0 2024-06-03T09:07:01.335 INFO:teuthology.orchestra.run.smithi088.stderr:2024-06-03T09:07:01.330+0000 7efc84e77640 10 monclient(hunting): picked mon.noname-b con 0x7efc800717c0 addr v1:172.21.15.192:6789/0 2024-06-03T09:07:01.335 INFO:teuthology.orchestra.run.smithi088.stderr:2024-06-03T09:07:01.330+0000 7efc84e77640 10 monclient(hunting): picked mon.noname-c con 0x7efc8012fa40 addr v1:172.21.15.88:6790/0 2024-06-03T09:07:01.335 INFO:teuthology.orchestra.run.smithi088.stderr:2024-06-03T09:07:01.330+0000 7efc84e77640 1 -- --> v1:172.21.15.88:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7efc80135230 con 0x7efc8006e050 2024-06-03T09:07:01.335 INFO:teuthology.orchestra.run.smithi088.stderr:2024-06-03T09:07:01.330+0000 7efc84e77640 1 -- --> v1:172.21.15.192:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7efc80140350 con 0x7efc800717c0 2024-06-03T09:07:01.335 INFO:teuthology.orchestra.run.smithi088.stderr:2024-06-03T09:07:01.330+0000 7efc84e77640 1 -- --> v1:172.21.15.88:6790/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7efc80140030 con 0x7efc8012fa40 2024-06-03T09:07:01.335 INFO:teuthology.orchestra.run.smithi088.stderr:2024-06-03T09:07:01.330+0000 7efc84e77640 10 monclient(hunting): _renew_subs 2024-06-03T09:07:01.335 INFO:teuthology.orchestra.run.smithi088.stderr:2024-06-03T09:07:01.330+0000 7efc84e77640 10 monclient(hunting): authenticate will time out at 3077.140188s 2024-06-03T09:07:01.335 INFO:teuthology.orchestra.run.smithi088.stderr:2024-06-03T09:07:01.330+0000 7efc7ed9d640 1 --1- >> v1:172.21.15.88:6790/0 conn(0x7efc8012fa40 0x7efc801d9a40 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=1).handle_server_banner_and_identify peer v1:172.21.15.88:6790/0 says I am v1:172.21.15.88:51172/0 (socket says 172.21.15.88:51172) 2024-06-03T09:07:01.335 INFO:teuthology.orchestra.run.smithi088.stderr:2024-06-03T09:07:01.330+0000 7efc7ed9d640 1 -- 172.21.15.88:0/1074203684 learned_addr learned my addr 172.21.15.88:0/1074203684 (peer_addr_for_me v1:172.21.15.88:0/0) 2024-06-03T09:07:01.336 INFO:teuthology.orchestra.run.smithi088.stderr:2024-06-03T09:07:01.330+0000 7efc6effd640 10 client.?.objecter ms_handle_connect 0x7efc8012fa40 2024-06-03T09:07:01.336 INFO:teuthology.orchestra.run.smithi088.stderr:2024-06-03T09:07:01.330+0000 7efc6effd640 10 client.?.objecter resend_mon_ops 2024-06-03T09:07:01.336 INFO:teuthology.orchestra.run.smithi088.stderr:2024-06-03T09:07:01.331+0000 7efc6effd640 10 client.?.objecter ms_handle_connect 0x7efc8006e050 2024-06-03T09:07:01.336 INFO:teuthology.orchestra.run.smithi088.stderr:2024-06-03T09:07:01.331+0000 7efc6effd640 10 client.?.objecter resend_mon_ops 2024-06-03T09:07:01.336 INFO:teuthology.orchestra.run.smithi088.stderr:2024-06-03T09:07:01.331+0000 7efc6effd640 10 client.?.objecter ms_handle_connect 0x7efc800717c0 2024-06-03T09:07:01.336 INFO:teuthology.orchestra.run.smithi088.stderr:2024-06-03T09:07:01.331+0000 7efc6effd640 10 client.?.objecter resend_mon_ops 2024-06-03T09:07:01.337 INFO:teuthology.orchestra.run.smithi088.stderr:2024-06-03T09:07:01.332+0000 7efc6effd640 1 -- 172.21.15.88:0/1074203684 <== mon.0 v1:172.21.15.88:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 1067891685 0 0) 0x7efc80135230 con 0x7efc8006e050 2024-06-03T09:07:01.337 INFO:teuthology.orchestra.run.smithi088.stderr:2024-06-03T09:07:01.333+0000 7efc6effd640 10 monclient(hunting): _negotiate 2024-06-03T09:07:01.338 INFO:teuthology.orchestra.run.smithi088.stderr:2024-06-03T09:07:01.333+0000 7efc6effd640 10 monclient(hunting): _init_auth method 2 2024-06-03T09:07:01.338 INFO:teuthology.orchestra.run.smithi088.stderr:2024-06-03T09:07:01.333+0000 7efc6effd640 10 monclient(hunting): _init_auth creating new auth 2024-06-03T09:07:01.338 INFO:teuthology.orchestra.run.smithi088.stderr:2024-06-03T09:07:01.333+0000 7efc6effd640 10 monclient(hunting): my global_id is 30810 2024-06-03T09:07:01.338 INFO:teuthology.orchestra.run.smithi088.stderr:2024-06-03T09:07:01.333+0000 7efc6effd640 1 -- 172.21.15.88:0/1074203684 --> v1:172.21.15.88:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7efc60004eb0 con 0x7efc8006e050 2024-06-03T09:07:01.338 INFO:teuthology.orchestra.run.smithi088.stderr:2024-06-03T09:07:01.333+0000 7efc6effd640 1 -- 172.21.15.88:0/1074203684 <== mon.1 v1:172.21.15.192:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 821866290 0 0) 0x7efc80140350 con 0x7efc800717c0 2024-06-03T09:07:01.338 INFO:teuthology.orchestra.run.smithi088.stderr:2024-06-03T09:07:01.333+0000 7efc6effd640 10 monclient(hunting): _negotiate 2024-06-03T09:07:01.338 INFO:teuthology.orchestra.run.smithi088.stderr:2024-06-03T09:07:01.333+0000 7efc6effd640 10 monclient(hunting): _init_auth method 2 2024-06-03T09:07:01.338 INFO:teuthology.orchestra.run.smithi088.stderr:2024-06-03T09:07:01.333+0000 7efc6effd640 10 monclient(hunting): _init_auth creating new auth 2024-06-03T09:07:01.338 INFO:teuthology.orchestra.run.smithi088.stderr:2024-06-03T09:07:01.333+0000 7efc6effd640 10 monclient(hunting): my global_id is 30385 2024-06-03T09:07:01.338 INFO:teuthology.orchestra.run.smithi088.stderr:2024-06-03T09:07:01.333+0000 7efc6effd640 1 -- 172.21.15.88:0/1074203684 --> v1:172.21.15.192:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7efc80135230 con 0x7efc800717c0 2024-06-03T09:07:01.338 INFO:teuthology.orchestra.run.smithi088.stderr:2024-06-03T09:07:01.333+0000 7efc6effd640 1 -- 172.21.15.88:0/1074203684 <== mon.2 v1:172.21.15.88:6790/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 1895637774 0 0) 0x7efc80140030 con 0x7efc8012fa40 2024-06-03T09:07:01.339 INFO:teuthology.orchestra.run.smithi088.stderr:2024-06-03T09:07:01.333+0000 7efc6effd640 10 monclient(hunting): _negotiate 2024-06-03T09:07:01.339 INFO:teuthology.orchestra.run.smithi088.stderr:2024-06-03T09:07:01.333+0000 7efc6effd640 10 monclient(hunting): _init_auth method 2 2024-06-03T09:07:01.339 INFO:teuthology.orchestra.run.smithi088.stderr:2024-06-03T09:07:01.333+0000 7efc6effd640 10 monclient(hunting): _init_auth creating new auth 2024-06-03T09:07:01.339 INFO:teuthology.orchestra.run.smithi088.stderr:2024-06-03T09:07:01.333+0000 7efc6effd640 10 monclient(hunting): my global_id is 30320 2024-06-03T09:07:01.339 INFO:teuthology.orchestra.run.smithi088.stderr:2024-06-03T09:07:01.333+0000 7efc6effd640 1 -- 172.21.15.88:0/1074203684 --> v1:172.21.15.88:6790/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7efc80140350 con 0x7efc8012fa40 2024-06-03T09:07:01.339 INFO:teuthology.orchestra.run.smithi088.stderr:2024-06-03T09:07:01.337+0000 7efc6effd640 1 -- 172.21.15.88:0/1074203684 <== mon.1 v1:172.21.15.192:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 764+0+0 (unknown 4075611886 0 0) 0x7efc80135230 con 0x7efc800717c0 2024-06-03T09:07:01.339 INFO:teuthology.orchestra.run.smithi088.stderr:2024-06-03T09:07:01.337+0000 7efc6effd640 1 -- 172.21.15.88:0/1074203684 --> v1:172.21.15.192:6789/0 -- auth(proto 2 165 bytes epoch 0) v1 -- 0x7efc80140030 con 0x7efc800717c0 2024-06-03T09:07:01.339 INFO:teuthology.orchestra.run.smithi088.stderr:2024-06-03T09:07:01.337+0000 7efc6effd640 1 -- 172.21.15.88:0/1074203684 <== mon.1 v1:172.21.15.192:6789/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 2234374065 0 0) 0x7efc74001b40 con 0x7efc800717c0 2024-06-03T09:07:01.339 INFO:teuthology.orchestra.run.smithi088.stderr:2024-06-03T09:07:01.337+0000 7efc6effd640 10 monclient(hunting): handle_monmap mon_map magic: 0 v1 2024-06-03T09:07:01.339 INFO:teuthology.orchestra.run.smithi088.stderr:2024-06-03T09:07:01.337+0000 7efc6effd640 10 monclient(hunting): got monmap 1 from mon.noname-b (according to old e1) 2024-06-03T09:07:01.340 INFO:teuthology.orchestra.run.smithi088.stderr:2024-06-03T09:07:01.337+0000 7efc6effd640 10 monclient(hunting): dump: 2024-06-03T09:07:01.340 INFO:teuthology.orchestra.run.smithi088.stderr:epoch 1 2024-06-03T09:07:01.340 INFO:teuthology.orchestra.run.smithi088.stderr:fsid ec3c3fdd-3ac6-4d7d-9412-683130ed8b04 2024-06-03T09:07:01.340 INFO:teuthology.orchestra.run.smithi088.stderr:last_changed 2024-06-03T08:34:31.885125+0000 2024-06-03T09:07:01.340 INFO:teuthology.orchestra.run.smithi088.stderr:created 2024-06-03T08:34:31.885125+0000 2024-06-03T09:07:01.340 INFO:teuthology.orchestra.run.smithi088.stderr:min_mon_release 18 (reef) 2024-06-03T09:07:01.340 INFO:teuthology.orchestra.run.smithi088.stderr:election_strategy: 3 2024-06-03T09:07:01.340 INFO:teuthology.orchestra.run.smithi088.stderr:0: v1:172.21.15.88:6789/0 mon.a 2024-06-03T09:07:01.340 INFO:teuthology.orchestra.run.smithi088.stderr:1: v1:172.21.15.192:6789/0 mon.b 2024-06-03T09:07:01.340 INFO:teuthology.orchestra.run.smithi088.stderr:2: v1:172.21.15.88:6790/0 mon.c 2024-06-03T09:07:01.340 INFO:teuthology.orchestra.run.smithi088.stderr: 2024-06-03T09:07:01.340 INFO:teuthology.orchestra.run.smithi088.stderr:2024-06-03T09:07:01.337+0000 7efc6effd640 10 monclient(hunting): _finish_auth 0 2024-06-03T09:07:01.340 INFO:teuthology.orchestra.run.smithi088.stderr:2024-06-03T09:07:01.337+0000 7efc6effd640 1 -- 172.21.15.88:0/1074203684 <== mon.2 v1:172.21.15.88:6790/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 764+0+0 (unknown 1866921723 0 0) 0x7efc80140350 con 0x7efc8012fa40 2024-06-03T09:07:01.340 INFO:teuthology.orchestra.run.smithi088.stderr:2024-06-03T09:07:01.337+0000 7efc6effd640 1 -- 172.21.15.88:0/1074203684 --> v1:172.21.15.88:6790/0 -- auth(proto 2 165 bytes epoch 0) v1 -- 0x7efc80135230 con 0x7efc8012fa40 2024-06-03T09:07:01.342 INFO:teuthology.orchestra.run.smithi088.stderr:2024-06-03T09:07:01.338+0000 7efc6effd640 1 -- 172.21.15.88:0/1074203684 <== mon.2 v1:172.21.15.88:6790/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 2234374065 0 0) 0x7efc78001c50 con 0x7efc8012fa40 2024-06-03T09:07:01.342 INFO:teuthology.orchestra.run.smithi088.stderr:2024-06-03T09:07:01.338+0000 7efc6effd640 10 monclient(hunting): handle_monmap mon_map magic: 0 v1 2024-06-03T09:07:01.342 INFO:teuthology.orchestra.run.smithi088.stderr:2024-06-03T09:07:01.338+0000 7efc6effd640 10 monclient(hunting): got monmap 1 from mon.c (according to old e1) 2024-06-03T09:07:01.343 INFO:teuthology.orchestra.run.smithi088.stderr:2024-06-03T09:07:01.338+0000 7efc6effd640 10 monclient(hunting): dump: 2024-06-03T09:07:01.343 INFO:teuthology.orchestra.run.smithi088.stderr:epoch 1 2024-06-03T09:07:01.343 INFO:teuthology.orchestra.run.smithi088.stderr:fsid ec3c3fdd-3ac6-4d7d-9412-683130ed8b04 2024-06-03T09:07:01.343 INFO:teuthology.orchestra.run.smithi088.stderr:last_changed 2024-06-03T08:34:31.885125+0000 2024-06-03T09:07:01.343 INFO:teuthology.orchestra.run.smithi088.stderr:created 2024-06-03T08:34:31.885125+0000 2024-06-03T09:07:01.343 INFO:teuthology.orchestra.run.smithi088.stderr:min_mon_release 18 (reef) 2024-06-03T09:07:01.343 INFO:teuthology.orchestra.run.smithi088.stderr:election_strategy: 3 2024-06-03T09:07:01.343 INFO:teuthology.orchestra.run.smithi088.stderr:0: v1:172.21.15.88:6789/0 mon.a 2024-06-03T09:07:01.343 INFO:teuthology.orchestra.run.smithi088.stderr:1: v1:172.21.15.192:6789/0 mon.b 2024-06-03T09:07:01.343 INFO:teuthology.orchestra.run.smithi088.stderr:2: v1:172.21.15.88:6790/0 mon.c 2024-06-03T09:07:01.343 INFO:teuthology.orchestra.run.smithi088.stderr: 2024-06-03T09:07:01.343 INFO:teuthology.orchestra.run.smithi088.stderr:2024-06-03T09:07:01.338+0000 7efc6effd640 1 -- 172.21.15.88:0/1074203684 <== mon.0 v1:172.21.15.88:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 764+0+0 (unknown 518884044 0 0) 0x7efc60004eb0 con 0x7efc8006e050 2024-06-03T09:07:01.343 INFO:teuthology.orchestra.run.smithi088.stderr:2024-06-03T09:07:01.338+0000 7efc6effd640 1 -- 172.21.15.88:0/1074203684 --> v1:172.21.15.88:6789/0 -- auth(proto 2 165 bytes epoch 0) v1 -- 0x7efc80140350 con 0x7efc8006e050 2024-06-03T09:07:01.344 INFO:teuthology.orchestra.run.smithi088.stderr:2024-06-03T09:07:01.338+0000 7efc6effd640 1 -- 172.21.15.88:0/1074203684 <== mon.0 v1:172.21.15.88:6789/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 2234374065 0 0) 0x7efc700018b0 con 0x7efc8006e050 2024-06-03T09:07:01.344 INFO:teuthology.orchestra.run.smithi088.stderr:2024-06-03T09:07:01.338+0000 7efc6effd640 10 monclient(hunting): handle_monmap mon_map magic: 0 v1 2024-06-03T09:07:01.344 INFO:teuthology.orchestra.run.smithi088.stderr:2024-06-03T09:07:01.338+0000 7efc6effd640 10 monclient(hunting): got monmap 1 from mon.a (according to old e1) 2024-06-03T09:07:01.344 INFO:teuthology.orchestra.run.smithi088.stderr:2024-06-03T09:07:01.338+0000 7efc6effd640 10 monclient(hunting): dump: 2024-06-03T09:07:01.344 INFO:teuthology.orchestra.run.smithi088.stderr:epoch 1 2024-06-03T09:07:01.344 INFO:teuthology.orchestra.run.smithi088.stderr:fsid ec3c3fdd-3ac6-4d7d-9412-683130ed8b04 2024-06-03T09:07:01.344 INFO:teuthology.orchestra.run.smithi088.stderr:last_changed 2024-06-03T08:34:31.885125+0000 2024-06-03T09:07:01.344 INFO:teuthology.orchestra.run.smithi088.stderr:created 2024-06-03T08:34:31.885125+0000 2024-06-03T09:07:01.344 INFO:teuthology.orchestra.run.smithi088.stderr:min_mon_release 18 (reef) 2024-06-03T09:07:01.344 INFO:teuthology.orchestra.run.smithi088.stderr:election_strategy: 3 2024-06-03T09:07:01.344 INFO:teuthology.orchestra.run.smithi088.stderr:0: v1:172.21.15.88:6789/0 mon.a 2024-06-03T09:07:01.344 INFO:teuthology.orchestra.run.smithi088.stderr:1: v1:172.21.15.192:6789/0 mon.b 2024-06-03T09:07:01.344 INFO:teuthology.orchestra.run.smithi088.stderr:2: v1:172.21.15.88:6790/0 mon.c 2024-06-03T09:07:01.344 INFO:teuthology.orchestra.run.smithi088.stderr: 2024-06-03T09:07:01.345 INFO:teuthology.orchestra.run.smithi088.stderr:2024-06-03T09:07:01.340+0000 7efc6effd640 1 -- 172.21.15.88:0/1074203684 <== mon.1 v1:172.21.15.192:6789/0 4 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 580+0+0 (unknown 1980072261 0 0) 0x7efc80140030 con 0x7efc800717c0 2024-06-03T09:07:01.345 INFO:teuthology.orchestra.run.smithi088.stderr:2024-06-03T09:07:01.341+0000 7efc6effd640 1 -- 172.21.15.88:0/1074203684 >> v1:172.21.15.88:6790/0 conn(0x7efc8012fa40 legacy=0x7efc801d9a40 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-06-03T09:07:01.345 INFO:teuthology.orchestra.run.smithi088.stderr:2024-06-03T09:07:01.341+0000 7efc6effd640 1 -- 172.21.15.88:0/1074203684 >> v1:172.21.15.88:6789/0 conn(0x7efc8006e050 legacy=0x7efc8013f920 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-06-03T09:07:01.345 INFO:teuthology.orchestra.run.smithi088.stderr:2024-06-03T09:07:01.341+0000 7efc6effd640 10 monclient: _finish_hunting 0 2024-06-03T09:07:01.345 INFO:teuthology.orchestra.run.smithi088.stderr:2024-06-03T09:07:01.341+0000 7efc6effd640 1 monclient: found mon.b 2024-06-03T09:07:01.345 INFO:teuthology.orchestra.run.smithi088.stderr:2024-06-03T09:07:01.341+0000 7efc6effd640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-06-03T09:07:01.345 INFO:teuthology.orchestra.run.smithi088.stderr:2024-06-03T09:07:01.341+0000 7efc6effd640 10 monclient: _send_mon_message to mon.b at v1:172.21.15.192:6789/0 2024-06-03T09:07:01.345 INFO:teuthology.orchestra.run.smithi088.stderr:2024-06-03T09:07:01.341+0000 7efc6effd640 1 -- 172.21.15.88:0/1074203684 --> v1:172.21.15.192:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7efc800acad0 con 0x7efc800717c0 2024-06-03T09:07:01.345 INFO:teuthology.orchestra.run.smithi088.stderr:2024-06-03T09:07:01.341+0000 7efc6effd640 10 monclient: _finish_auth 0 2024-06-03T09:07:01.345 INFO:teuthology.orchestra.run.smithi088.stderr:2024-06-03T09:07:01.341+0000 7efc6effd640 10 monclient: _check_auth_tickets 2024-06-03T09:07:01.345 INFO:teuthology.orchestra.run.smithi088.stderr:2024-06-03T09:07:01.341+0000 7efc6effd640 20 monclient: _check_auth_rotating not needed by client.admin 2024-06-03T09:07:01.345 INFO:teuthology.orchestra.run.smithi088.stderr:2024-06-03T09:07:01.341+0000 7efc84e77640 5 monclient: authenticate success, global_id 30385 2024-06-03T09:07:01.345 INFO:teuthology.orchestra.run.smithi088.stderr:2024-06-03T09:07:01.341+0000 7efc84e77640 10 monclient: _renew_subs 2024-06-03T09:07:01.345 INFO:teuthology.orchestra.run.smithi088.stderr:2024-06-03T09:07:01.341+0000 7efc84e77640 10 monclient: _send_mon_message to mon.b at v1:172.21.15.192:6789/0 2024-06-03T09:07:01.346 INFO:teuthology.orchestra.run.smithi088.stderr:2024-06-03T09:07:01.341+0000 7efc84e77640 1 -- 172.21.15.88:0/1074203684 --> v1:172.21.15.192:6789/0 -- mon_subscribe({mgrmap=0+}) v3 -- 0x7efc800acd20 con 0x7efc800717c0 2024-06-03T09:07:01.346 INFO:teuthology.orchestra.run.smithi088.stderr:2024-06-03T09:07:01.341+0000 7efc84e77640 10 client.30385.objecter _maybe_request_map subscribing (onetime) to next osd map 2024-06-03T09:07:01.346 INFO:teuthology.orchestra.run.smithi088.stderr:2024-06-03T09:07:01.341+0000 7efc84e77640 10 monclient: _renew_subs 2024-06-03T09:07:01.346 INFO:teuthology.orchestra.run.smithi088.stderr:2024-06-03T09:07:01.341+0000 7efc84e77640 10 monclient: _send_mon_message to mon.b at v1:172.21.15.192:6789/0 2024-06-03T09:07:01.346 INFO:teuthology.orchestra.run.smithi088.stderr:2024-06-03T09:07:01.341+0000 7efc84e77640 1 -- 172.21.15.88:0/1074203684 --> v1:172.21.15.192:6789/0 -- mon_subscribe({osdmap=0}) v3 -- 0x7efc801da330 con 0x7efc800717c0 2024-06-03T09:07:01.346 INFO:teuthology.orchestra.run.smithi088.stderr:2024-06-03T09:07:01.341+0000 7efc84e77640 1 librados: init done 2024-06-03T09:07:01.346 INFO:teuthology.orchestra.run.smithi088.stderr:2024-06-03T09:07:01.342+0000 7efc84e77640 10 start_mon_command cmd=[{"prefix": "get_command_descriptions"}] 2024-06-03T09:07:01.346 INFO:teuthology.orchestra.run.smithi088.stderr:2024-06-03T09:07:01.342+0000 7efc84e77640 10 monclient: _send_command 1 [{"prefix": "get_command_descriptions"}] 2024-06-03T09:07:01.346 INFO:teuthology.orchestra.run.smithi088.stderr:2024-06-03T09:07:01.342+0000 7efc84e77640 10 monclient: _send_mon_message to mon.b at v1:172.21.15.192:6789/0 2024-06-03T09:07:01.346 INFO:teuthology.orchestra.run.smithi088.stderr:2024-06-03T09:07:01.342+0000 7efc84e77640 1 -- 172.21.15.88:0/1074203684 --> v1:172.21.15.192:6789/0 -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7efc801401c0 con 0x7efc800717c0 2024-06-03T09:07:01.555 INFO:teuthology.orchestra.run.smithi088.stderr:2024-06-03T09:07:01.555+0000 7efc6effd640 1 -- 172.21.15.88:0/1074203684 <== mon.1 v1:172.21.15.192:6789/0 5 ==== config(0 keys) v1 ==== 4+0+0 (unknown 0 0 0) 0x7efc740011c0 con 0x7efc800717c0 2024-06-03T09:07:01.555 INFO:teuthology.orchestra.run.smithi088.stderr:2024-06-03T09:07:01.555+0000 7efc6effd640 10 monclient: handle_config config(0 keys) v1 2024-06-03T09:07:01.555 INFO:teuthology.orchestra.run.smithi088.stderr:2024-06-03T09:07:01.555+0000 7efc7cd99640 4 set_mon_vals no callback set 2024-06-03T09:07:01.584 INFO:teuthology.orchestra.run.smithi088.stderr:2024-06-03T09:07:01.584+0000 7efc6effd640 1 -- 172.21.15.88:0/1074203684 <== mon.1 v1:172.21.15.192:6789/0 6 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 2234374065 0 0) 0x7efc74003a40 con 0x7efc800717c0 2024-06-03T09:07:01.584 INFO:teuthology.orchestra.run.smithi088.stderr:2024-06-03T09:07:01.584+0000 7efc6effd640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-06-03T09:07:01.584 INFO:teuthology.orchestra.run.smithi088.stderr:2024-06-03T09:07:01.584+0000 7efc6effd640 10 monclient: got monmap 1 from mon.b (according to old e1) 2024-06-03T09:07:01.584 INFO:teuthology.orchestra.run.smithi088.stderr:2024-06-03T09:07:01.584+0000 7efc6effd640 10 monclient: dump: 2024-06-03T09:07:01.584 INFO:teuthology.orchestra.run.smithi088.stderr:epoch 1 2024-06-03T09:07:01.585 INFO:teuthology.orchestra.run.smithi088.stderr:fsid ec3c3fdd-3ac6-4d7d-9412-683130ed8b04 2024-06-03T09:07:01.585 INFO:teuthology.orchestra.run.smithi088.stderr:last_changed 2024-06-03T08:34:31.885125+0000 2024-06-03T09:07:01.585 INFO:teuthology.orchestra.run.smithi088.stderr:created 2024-06-03T08:34:31.885125+0000 2024-06-03T09:07:01.585 INFO:teuthology.orchestra.run.smithi088.stderr:min_mon_release 18 (reef) 2024-06-03T09:07:01.585 INFO:teuthology.orchestra.run.smithi088.stderr:election_strategy: 3 2024-06-03T09:07:01.585 INFO:teuthology.orchestra.run.smithi088.stderr:0: v1:172.21.15.88:6789/0 mon.a 2024-06-03T09:07:01.585 INFO:teuthology.orchestra.run.smithi088.stderr:1: v1:172.21.15.192:6789/0 mon.b 2024-06-03T09:07:01.585 INFO:teuthology.orchestra.run.smithi088.stderr:2: v1:172.21.15.88:6790/0 mon.c 2024-06-03T09:07:01.585 INFO:teuthology.orchestra.run.smithi088.stderr: 2024-06-03T09:07:01.586 INFO:teuthology.orchestra.run.smithi088.stderr:2024-06-03T09:07:01.585+0000 7efc6effd640 1 -- 172.21.15.88:0/1074203684 <== mon.1 v1:172.21.15.192:6789/0 7 ==== mgrmap(e 5) v1 ==== 89110+0+0 (unknown 873533973 0 0) 0x7efc740197c0 con 0x7efc800717c0 2024-06-03T09:07:01.586 INFO:teuthology.orchestra.run.smithi088.stderr:2024-06-03T09:07:01.586+0000 7efc7e59c640 10 monclient: get_auth_request con 0x7efc6006d390 auth_method 0 2024-06-03T09:07:01.586 INFO:teuthology.orchestra.run.smithi088.stderr:2024-06-03T09:07:01.586+0000 7efc6effd640 10 client.30385.objecter ms_handle_connect 0x7efc6006d390 2024-06-03T09:07:01.588 INFO:teuthology.orchestra.run.smithi088.stderr:2024-06-03T09:07:01.588+0000 7efc6effd640 1 -- 172.21.15.88:0/1074203684 <== mon.1 v1:172.21.15.192:6789/0 8 ==== osd_map(1014..1014 src has 622..1014) v4 ==== 5229+0+0 (unknown 3230192161 0 0) 0x7efc74052f80 con 0x7efc800717c0 2024-06-03T09:07:01.588 INFO:teuthology.orchestra.run.smithi088.stderr:2024-06-03T09:07:01.588+0000 7efc6effd640 10 client.30385.objecter ms_dispatch 0x7efc80000b90 osd_map(1014..1014 src has 622..1014) v4 2024-06-03T09:07:01.588 INFO:teuthology.orchestra.run.smithi088.stderr:2024-06-03T09:07:01.588+0000 7efc6effd640 3 client.30385.objecter handle_osd_map got epochs [1014,1014] > 0 2024-06-03T09:07:01.588 INFO:teuthology.orchestra.run.smithi088.stderr:2024-06-03T09:07:01.588+0000 7efc6effd640 3 client.30385.objecter handle_osd_map decoding full epoch 1014 2024-06-03T09:07:01.588 INFO:teuthology.orchestra.run.smithi088.stderr:2024-06-03T09:07:01.588+0000 7efc6effd640 20 client.30385.objecter dump_active .. 0 homeless 2024-06-03T09:07:01.590 INFO:teuthology.orchestra.run.smithi088.stderr:2024-06-03T09:07:01.591+0000 7efc6effd640 1 -- 172.21.15.88:0/1074203684 <== mon.1 v1:172.21.15.192:6789/0 9 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+182498 (unknown 1092875540 0 3016760700) 0x7efc7404eb30 con 0x7efc800717c0 2024-06-03T09:07:01.591 INFO:teuthology.orchestra.run.smithi088.stderr:2024-06-03T09:07:01.591+0000 7efc6effd640 10 monclient: handle_mon_command_ack 1 [{"prefix": "get_command_descriptions"}] 2024-06-03T09:07:01.591 INFO:teuthology.orchestra.run.smithi088.stderr:2024-06-03T09:07:01.591+0000 7efc6effd640 10 monclient: _finish_command 1 = system:0 2024-06-03T09:07:01.774 INFO:teuthology.orchestra.run.smithi088.stderr:2024-06-03T09:07:01.774+0000 7efc84e77640 1 -- 172.21.15.88:0/1074203684 --> v1:172.21.15.192:6800/34229 -- mgr_command(tid 0: {"prefix": "osd deep-scrub", "who": "6", "target": ["mon-mgr", ""]}) v1 -- 0x7efc8006ee80 con 0x7efc6006d390 2024-06-03T09:07:01.775 INFO:teuthology.orchestra.run.smithi088.stderr:2024-06-03T09:07:01.775+0000 7efc6effd640 1 -- 172.21.15.88:0/1074203684 <== mgr.4110 v1:172.21.15.192:6800/34229 1 ==== mgr_command_reply(tid 0: 0 instructed osd(s) 6 to deep-scrub) v1 ==== 41+0+0 (unknown 778718403 0 0) 0x7efc8006ee80 con 0x7efc6006d390 2024-06-03T09:07:01.775 INFO:teuthology.orchestra.run.smithi088.stderr:instructed osd(s) 6 to deep-scrub 2024-06-03T09:07:01.778 INFO:teuthology.orchestra.run.smithi088.stderr:2024-06-03T09:07:01.777+0000 7efc84e77640 10 librados: watch_flush enter 2024-06-03T09:07:01.778 INFO:teuthology.orchestra.run.smithi088.stderr:2024-06-03T09:07:01.777+0000 7efc84e77640 10 librados: watch_flush exit 2024-06-03T09:07:01.778 INFO:teuthology.orchestra.run.smithi088.stderr:2024-06-03T09:07:01.777+0000 7efc84e77640 20 client.30385.objecter shutdown clearing up homeless session... 2024-06-03T09:07:01.778 INFO:teuthology.orchestra.run.smithi088.stderr:2024-06-03T09:07:01.777+0000 7efc84e77640 10 client.30385.objecter successfully canceled tick 2024-06-03T09:07:01.778 INFO:teuthology.orchestra.run.smithi088.stderr:2024-06-03T09:07:01.777+0000 7efc84e77640 1 -- 172.21.15.88:0/1074203684 >> v1:172.21.15.192:6800/34229 conn(0x7efc6006d390 legacy=0x7efc6006f850 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-06-03T09:07:01.778 INFO:teuthology.orchestra.run.smithi088.stderr:2024-06-03T09:07:01.777+0000 7efc84e77640 10 monclient: shutdown 2024-06-03T09:07:01.778 INFO:teuthology.orchestra.run.smithi088.stderr:2024-06-03T09:07:01.777+0000 7efc84e77640 20 monclient: shutdown discarding 0 pending message(s) 2024-06-03T09:07:01.778 INFO:teuthology.orchestra.run.smithi088.stderr:2024-06-03T09:07:01.777+0000 7efc84e77640 1 -- 172.21.15.88:0/1074203684 >> v1:172.21.15.192:6789/0 conn(0x7efc800717c0 legacy=0x7efc8012f330 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-06-03T09:07:01.778 INFO:teuthology.orchestra.run.smithi088.stderr:2024-06-03T09:07:01.777+0000 7efc84e77640 1 -- 172.21.15.88:0/1074203684 shutdown_connections 2024-06-03T09:07:01.779 INFO:teuthology.orchestra.run.smithi088.stderr:2024-06-03T09:07:01.777+0000 7efc84e77640 1 -- 172.21.15.88:0/1074203684 >> 172.21.15.88:0/1074203684 conn(0x7efc800b2f20 msgr2=0x7efc800b3300 unknown :-1 s=STATE_NONE l=0).mark_down 2024-06-03T09:07:01.779 INFO:teuthology.orchestra.run.smithi088.stderr:2024-06-03T09:07:01.777+0000 7efc84e77640 1 -- 172.21.15.88:0/1074203684 shutdown_connections 2024-06-03T09:07:01.779 INFO:teuthology.orchestra.run.smithi088.stderr:2024-06-03T09:07:01.777+0000 7efc84e77640 1 -- 172.21.15.88:0/1074203684 wait complete. 2024-06-03T09:07:01.779 INFO:teuthology.orchestra.run.smithi088.stderr:2024-06-03T09:07:01.777+0000 7efc84e77640 1 librados: shutdown 2024-06-03T09:07:01.788 INFO:tasks.ceph:Scrubbing osd.7 2024-06-03T09:07:01.788 DEBUG:teuthology.orchestra.run.smithi088:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph tell osd.7 config set osd_debug_deep_scrub_sleep 0 2024-06-03T09:07:01.889 INFO:teuthology.orchestra.run.smithi088.stderr:2024-06-03T09:07:01.889+0000 7fbddc8e7640 10 monclient: get_monmap_and_config 2024-06-03T09:07:01.890 INFO:teuthology.orchestra.run.smithi088.stderr:2024-06-03T09:07:01.889+0000 7fbddc8e7640 10 monclient: build_initial_monmap 2024-06-03T09:07:01.890 INFO:teuthology.orchestra.run.smithi088.stderr:2024-06-03T09:07:01.889+0000 7fbddc8e7640 10 monclient: monmap: 2024-06-03T09:07:01.890 INFO:teuthology.orchestra.run.smithi088.stderr:epoch 0 2024-06-03T09:07:01.890 INFO:teuthology.orchestra.run.smithi088.stderr:fsid ec3c3fdd-3ac6-4d7d-9412-683130ed8b04 2024-06-03T09:07:01.890 INFO:teuthology.orchestra.run.smithi088.stderr:last_changed 2024-06-03T09:07:01.890752+0000 2024-06-03T09:07:01.890 INFO:teuthology.orchestra.run.smithi088.stderr:created 2024-06-03T09:07:01.890752+0000 2024-06-03T09:07:01.890 INFO:teuthology.orchestra.run.smithi088.stderr:min_mon_release 0 (unknown) 2024-06-03T09:07:01.890 INFO:teuthology.orchestra.run.smithi088.stderr:election_strategy: 3 2024-06-03T09:07:01.890 INFO:teuthology.orchestra.run.smithi088.stderr:0: v1:172.21.15.88:6789/0 mon.noname-a 2024-06-03T09:07:01.891 INFO:teuthology.orchestra.run.smithi088.stderr:1: v1:172.21.15.192:6789/0 mon.noname-c 2024-06-03T09:07:01.891 INFO:teuthology.orchestra.run.smithi088.stderr:2: v1:172.21.15.88:6790/0 mon.noname-b 2024-06-03T09:07:01.891 INFO:teuthology.orchestra.run.smithi088.stderr: 2024-06-03T09:07:01.892 INFO:teuthology.orchestra.run.smithi088.stderr:2024-06-03T09:07:01.890+0000 7fbddc8e7640 1 Processor -- start 2024-06-03T09:07:01.892 INFO:teuthology.orchestra.run.smithi088.stderr:2024-06-03T09:07:01.890+0000 7fbddc8e7640 1 -- start start 2024-06-03T09:07:01.892 INFO:teuthology.orchestra.run.smithi088.stderr:2024-06-03T09:07:01.890+0000 7fbddc8e7640 10 monclient: init 2024-06-03T09:07:01.893 INFO:teuthology.orchestra.run.smithi088.stderr:2024-06-03T09:07:01.891+0000 7fbddc8e7640 10 monclient: _reopen_session rank -1 2024-06-03T09:07:01.893 INFO:teuthology.orchestra.run.smithi088.stderr:2024-06-03T09:07:01.891+0000 7fbddc8e7640 10 monclient: _add_conns ranks=[1,0,2] 2024-06-03T09:07:01.893 INFO:teuthology.orchestra.run.smithi088.stderr:2024-06-03T09:07:01.891+0000 7fbddc8e7640 10 monclient(hunting): picked mon.noname-c con 0x7fbdd806fc70 addr v1:172.21.15.192:6789/0 2024-06-03T09:07:01.893 INFO:teuthology.orchestra.run.smithi088.stderr:2024-06-03T09:07:01.891+0000 7fbddc8e7640 10 monclient(hunting): picked mon.noname-a con 0x7fbdd80ad920 addr v1:172.21.15.88:6789/0 2024-06-03T09:07:01.893 INFO:teuthology.orchestra.run.smithi088.stderr:2024-06-03T09:07:01.891+0000 7fbddc8e7640 10 monclient(hunting): picked mon.noname-b con 0x7fbdd81393f0 addr v1:172.21.15.88:6790/0 2024-06-03T09:07:01.893 INFO:teuthology.orchestra.run.smithi088.stderr:2024-06-03T09:07:01.891+0000 7fbddc8e7640 1 -- --> v1:172.21.15.88:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fbdd8070420 con 0x7fbdd80ad920 2024-06-03T09:07:01.893 INFO:teuthology.orchestra.run.smithi088.stderr:2024-06-03T09:07:01.891+0000 7fbddc8e7640 1 -- --> v1:172.21.15.192:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fbdd8071050 con 0x7fbdd806fc70 2024-06-03T09:07:01.893 INFO:teuthology.orchestra.run.smithi088.stderr:2024-06-03T09:07:01.891+0000 7fbddc8e7640 1 -- --> v1:172.21.15.88:6790/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fbdd80add60 con 0x7fbdd81393f0 2024-06-03T09:07:01.893 INFO:teuthology.orchestra.run.smithi088.stderr:2024-06-03T09:07:01.891+0000 7fbddc8e7640 10 monclient(hunting): _renew_subs 2024-06-03T09:07:01.893 INFO:teuthology.orchestra.run.smithi088.stderr:2024-06-03T09:07:01.891+0000 7fbddc8e7640 10 monclient(hunting): authenticate will time out at 3077.701293s 2024-06-03T09:07:01.893 INFO:teuthology.orchestra.run.smithi088.stderr:2024-06-03T09:07:01.891+0000 7fbdd6d9d640 1 --1- >> v1:172.21.15.88:6790/0 conn(0x7fbdd81393f0 0x7fbdd814ac50 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.88:6790/0 says I am v1:172.21.15.88:51176/0 (socket says 172.21.15.88:51176) 2024-06-03T09:07:01.893 INFO:teuthology.orchestra.run.smithi088.stderr:2024-06-03T09:07:01.891+0000 7fbdd6d9d640 1 -- 172.21.15.88:0/1013672306 learned_addr learned my addr 172.21.15.88:0/1013672306 (peer_addr_for_me v1:172.21.15.88:0/0) 2024-06-03T09:07:01.894 INFO:teuthology.orchestra.run.smithi088.stderr:2024-06-03T09:07:01.894+0000 7fbdd559a640 1 -- 172.21.15.88:0/1013672306 <== mon.0 v1:172.21.15.88:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 3067602329 0 0) 0x7fbdd8070420 con 0x7fbdd80ad920 2024-06-03T09:07:01.894 INFO:teuthology.orchestra.run.smithi088.stderr:2024-06-03T09:07:01.894+0000 7fbdd559a640 10 monclient(hunting): _negotiate 2024-06-03T09:07:01.894 INFO:teuthology.orchestra.run.smithi088.stderr:2024-06-03T09:07:01.894+0000 7fbdd559a640 10 monclient(hunting): _init_auth method 2 2024-06-03T09:07:01.895 INFO:teuthology.orchestra.run.smithi088.stderr:2024-06-03T09:07:01.894+0000 7fbdd559a640 10 monclient(hunting): _init_auth creating new auth 2024-06-03T09:07:01.895 INFO:teuthology.orchestra.run.smithi088.stderr:2024-06-03T09:07:01.894+0000 7fbdd559a640 10 monclient(hunting): my global_id is 30813 2024-06-03T09:07:01.895 INFO:teuthology.orchestra.run.smithi088.stderr:2024-06-03T09:07:01.894+0000 7fbdd559a640 1 -- 172.21.15.88:0/1013672306 --> v1:172.21.15.88:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7fbdc0003520 con 0x7fbdd80ad920 2024-06-03T09:07:01.895 INFO:teuthology.orchestra.run.smithi088.stderr:2024-06-03T09:07:01.894+0000 7fbdd559a640 1 -- 172.21.15.88:0/1013672306 <== mon.2 v1:172.21.15.88:6790/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 3680198858 0 0) 0x7fbdd80add60 con 0x7fbdd81393f0 2024-06-03T09:07:01.895 INFO:teuthology.orchestra.run.smithi088.stderr:2024-06-03T09:07:01.894+0000 7fbdd559a640 10 monclient(hunting): _negotiate 2024-06-03T09:07:01.895 INFO:teuthology.orchestra.run.smithi088.stderr:2024-06-03T09:07:01.894+0000 7fbdd559a640 10 monclient(hunting): _init_auth method 2 2024-06-03T09:07:01.895 INFO:teuthology.orchestra.run.smithi088.stderr:2024-06-03T09:07:01.894+0000 7fbdd559a640 10 monclient(hunting): _init_auth creating new auth 2024-06-03T09:07:01.895 INFO:teuthology.orchestra.run.smithi088.stderr:2024-06-03T09:07:01.894+0000 7fbdd559a640 10 monclient(hunting): my global_id is 30323 2024-06-03T09:07:01.895 INFO:teuthology.orchestra.run.smithi088.stderr:2024-06-03T09:07:01.894+0000 7fbdd559a640 1 -- 172.21.15.88:0/1013672306 --> v1:172.21.15.88:6790/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7fbdd8070420 con 0x7fbdd81393f0 2024-06-03T09:07:01.895 INFO:teuthology.orchestra.run.smithi088.stderr:2024-06-03T09:07:01.894+0000 7fbdd559a640 1 -- 172.21.15.88:0/1013672306 <== mon.1 v1:172.21.15.192:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 2503868253 0 0) 0x7fbdd8071050 con 0x7fbdd806fc70 2024-06-03T09:07:01.895 INFO:teuthology.orchestra.run.smithi088.stderr:2024-06-03T09:07:01.894+0000 7fbdd559a640 10 monclient(hunting): _negotiate 2024-06-03T09:07:01.895 INFO:teuthology.orchestra.run.smithi088.stderr:2024-06-03T09:07:01.894+0000 7fbdd559a640 10 monclient(hunting): _init_auth method 2 2024-06-03T09:07:01.895 INFO:teuthology.orchestra.run.smithi088.stderr:2024-06-03T09:07:01.894+0000 7fbdd559a640 10 monclient(hunting): _init_auth creating new auth 2024-06-03T09:07:01.896 INFO:teuthology.orchestra.run.smithi088.stderr:2024-06-03T09:07:01.894+0000 7fbdd559a640 10 monclient(hunting): my global_id is 30388 2024-06-03T09:07:01.896 INFO:teuthology.orchestra.run.smithi088.stderr:2024-06-03T09:07:01.894+0000 7fbdd559a640 1 -- 172.21.15.88:0/1013672306 --> v1:172.21.15.192:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7fbdd80add60 con 0x7fbdd806fc70 2024-06-03T09:07:01.897 INFO:teuthology.orchestra.run.smithi088.stderr:2024-06-03T09:07:01.897+0000 7fbdd559a640 1 -- 172.21.15.88:0/1013672306 <== mon.0 v1:172.21.15.88:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 214+0+0 (unknown 1517630555 0 0) 0x7fbdc0003520 con 0x7fbdd80ad920 2024-06-03T09:07:01.898 INFO:teuthology.orchestra.run.smithi088.stderr:2024-06-03T09:07:01.897+0000 7fbdd559a640 1 -- 172.21.15.88:0/1013672306 >> v1:172.21.15.88:6790/0 conn(0x7fbdd81393f0 legacy=0x7fbdd814ac50 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-06-03T09:07:01.898 INFO:teuthology.orchestra.run.smithi088.stderr:2024-06-03T09:07:01.897+0000 7fbdd559a640 1 -- 172.21.15.88:0/1013672306 >> v1:172.21.15.192:6789/0 conn(0x7fbdd806fc70 legacy=0x7fbdd813fbb0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-06-03T09:07:01.898 INFO:teuthology.orchestra.run.smithi088.stderr:2024-06-03T09:07:01.897+0000 7fbdd559a640 10 monclient: _finish_hunting 0 2024-06-03T09:07:01.898 INFO:teuthology.orchestra.run.smithi088.stderr:2024-06-03T09:07:01.897+0000 7fbdd559a640 1 monclient: found mon.noname-a 2024-06-03T09:07:01.898 INFO:teuthology.orchestra.run.smithi088.stderr:2024-06-03T09:07:01.897+0000 7fbdd559a640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-06-03T09:07:01.898 INFO:teuthology.orchestra.run.smithi088.stderr:2024-06-03T09:07:01.897+0000 7fbdd559a640 10 monclient: _send_mon_message to mon.noname-a at v1:172.21.15.88:6789/0 2024-06-03T09:07:01.898 INFO:teuthology.orchestra.run.smithi088.stderr:2024-06-03T09:07:01.897+0000 7fbdd559a640 1 -- 172.21.15.88:0/1013672306 --> v1:172.21.15.88:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fbdd8071be0 con 0x7fbdd80ad920 2024-06-03T09:07:01.898 INFO:teuthology.orchestra.run.smithi088.stderr:2024-06-03T09:07:01.897+0000 7fbdd559a640 10 monclient: _finish_auth 0 2024-06-03T09:07:01.898 INFO:teuthology.orchestra.run.smithi088.stderr:2024-06-03T09:07:01.897+0000 7fbdd559a640 10 monclient: _check_auth_tickets 2024-06-03T09:07:01.898 INFO:teuthology.orchestra.run.smithi088.stderr:2024-06-03T09:07:01.897+0000 7fbdd559a640 20 monclient: _check_auth_rotating not needed by client.admin 2024-06-03T09:07:01.898 INFO:teuthology.orchestra.run.smithi088.stderr:2024-06-03T09:07:01.897+0000 7fbddc8e7640 5 monclient: authenticate success, global_id 30813 2024-06-03T09:07:01.898 INFO:teuthology.orchestra.run.smithi088.stderr:2024-06-03T09:07:01.897+0000 7fbddc8e7640 20 monclient: get_monmap_and_config waiting for monmap|config 2024-06-03T09:07:01.898 INFO:teuthology.orchestra.run.smithi088.stderr:2024-06-03T09:07:01.897+0000 7fbdd559a640 1 -- 172.21.15.88:0/1013672306 <== mon.0 v1:172.21.15.88:6789/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 2234374065 0 0) 0x7fbdcc001610 con 0x7fbdd80ad920 2024-06-03T09:07:01.899 INFO:teuthology.orchestra.run.smithi088.stderr:2024-06-03T09:07:01.897+0000 7fbdd559a640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-06-03T09:07:01.899 INFO:teuthology.orchestra.run.smithi088.stderr:2024-06-03T09:07:01.897+0000 7fbdd559a640 10 monclient: got monmap 1 from mon.noname-a (according to old e1) 2024-06-03T09:07:01.899 INFO:teuthology.orchestra.run.smithi088.stderr:2024-06-03T09:07:01.897+0000 7fbdd559a640 10 monclient: dump: 2024-06-03T09:07:01.899 INFO:teuthology.orchestra.run.smithi088.stderr:epoch 1 2024-06-03T09:07:01.899 INFO:teuthology.orchestra.run.smithi088.stderr:fsid ec3c3fdd-3ac6-4d7d-9412-683130ed8b04 2024-06-03T09:07:01.899 INFO:teuthology.orchestra.run.smithi088.stderr:last_changed 2024-06-03T08:34:31.885125+0000 2024-06-03T09:07:01.899 INFO:teuthology.orchestra.run.smithi088.stderr:created 2024-06-03T08:34:31.885125+0000 2024-06-03T09:07:01.899 INFO:teuthology.orchestra.run.smithi088.stderr:min_mon_release 18 (reef) 2024-06-03T09:07:01.899 INFO:teuthology.orchestra.run.smithi088.stderr:election_strategy: 3 2024-06-03T09:07:01.899 INFO:teuthology.orchestra.run.smithi088.stderr:0: v1:172.21.15.88:6789/0 mon.a 2024-06-03T09:07:01.899 INFO:teuthology.orchestra.run.smithi088.stderr:1: v1:172.21.15.192:6789/0 mon.b 2024-06-03T09:07:01.899 INFO:teuthology.orchestra.run.smithi088.stderr:2: v1:172.21.15.88:6790/0 mon.c 2024-06-03T09:07:01.899 INFO:teuthology.orchestra.run.smithi088.stderr: 2024-06-03T09:07:01.899 INFO:teuthology.orchestra.run.smithi088.stderr:2024-06-03T09:07:01.897+0000 7fbddc8e7640 20 monclient: get_monmap_and_config waiting for monmap|config 2024-06-03T09:07:01.903 INFO:teuthology.orchestra.run.smithi088.stderr:2024-06-03T09:07:01.899+0000 7fbdd559a640 1 -- 172.21.15.88:0/1013672306 <== mon.0 v1:172.21.15.88:6789/0 4 ==== config(0 keys) v1 ==== 4+0+0 (unknown 0 0 0) 0x7fbdcc001830 con 0x7fbdd80ad920 2024-06-03T09:07:01.903 INFO:teuthology.orchestra.run.smithi088.stderr:2024-06-03T09:07:01.899+0000 7fbdd559a640 10 monclient: handle_config config(0 keys) v1 2024-06-03T09:07:01.903 INFO:teuthology.orchestra.run.smithi088.stderr:2024-06-03T09:07:01.899+0000 7fbdd559a640 1 -- 172.21.15.88:0/1013672306 <== mon.0 v1:172.21.15.88:6789/0 5 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 2234374065 0 0) 0x7fbdcc001b40 con 0x7fbdd80ad920 2024-06-03T09:07:01.903 INFO:teuthology.orchestra.run.smithi088.stderr:2024-06-03T09:07:01.899+0000 7fbddc8e7640 10 monclient: get_monmap_and_config success 2024-06-03T09:07:01.903 INFO:teuthology.orchestra.run.smithi088.stderr:2024-06-03T09:07:01.899+0000 7fbddc8e7640 4 set_mon_vals no callback set 2024-06-03T09:07:01.903 INFO:teuthology.orchestra.run.smithi088.stderr:2024-06-03T09:07:01.899+0000 7fbdd559a640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-06-03T09:07:01.903 INFO:teuthology.orchestra.run.smithi088.stderr:2024-06-03T09:07:01.899+0000 7fbdd559a640 10 monclient: got monmap 1 from mon.a (according to old e1) 2024-06-03T09:07:01.903 INFO:teuthology.orchestra.run.smithi088.stderr:2024-06-03T09:07:01.899+0000 7fbdd559a640 10 monclient: dump: 2024-06-03T09:07:01.903 INFO:teuthology.orchestra.run.smithi088.stderr:epoch 1 2024-06-03T09:07:01.903 INFO:teuthology.orchestra.run.smithi088.stderr:fsid ec3c3fdd-3ac6-4d7d-9412-683130ed8b04 2024-06-03T09:07:01.903 INFO:teuthology.orchestra.run.smithi088.stderr:last_changed 2024-06-03T08:34:31.885125+0000 2024-06-03T09:07:01.903 INFO:teuthology.orchestra.run.smithi088.stderr:created 2024-06-03T08:34:31.885125+0000 2024-06-03T09:07:01.903 INFO:teuthology.orchestra.run.smithi088.stderr:min_mon_release 18 (reef) 2024-06-03T09:07:01.904 INFO:teuthology.orchestra.run.smithi088.stderr:election_strategy: 3 2024-06-03T09:07:01.904 INFO:teuthology.orchestra.run.smithi088.stderr:0: v1:172.21.15.88:6789/0 mon.a 2024-06-03T09:07:01.904 INFO:teuthology.orchestra.run.smithi088.stderr:1: v1:172.21.15.192:6789/0 mon.b 2024-06-03T09:07:01.904 INFO:teuthology.orchestra.run.smithi088.stderr:2: v1:172.21.15.88:6790/0 mon.c 2024-06-03T09:07:01.904 INFO:teuthology.orchestra.run.smithi088.stderr: 2024-06-03T09:07:01.904 INFO:teuthology.orchestra.run.smithi088.stderr:2024-06-03T09:07:01.899+0000 7fbddc8e7640 10 monclient: shutdown 2024-06-03T09:07:01.904 INFO:teuthology.orchestra.run.smithi088.stderr:2024-06-03T09:07:01.899+0000 7fbddc8e7640 20 monclient: shutdown discarding 0 pending message(s) 2024-06-03T09:07:01.904 INFO:teuthology.orchestra.run.smithi088.stderr:2024-06-03T09:07:01.899+0000 7fbddc8e7640 1 -- 172.21.15.88:0/1013672306 >> v1:172.21.15.88:6789/0 conn(0x7fbdd80ad920 legacy=0x7fbdd8138ce0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-06-03T09:07:01.904 INFO:teuthology.orchestra.run.smithi088.stderr:2024-06-03T09:07:01.900+0000 7fbddc8e7640 1 -- 172.21.15.88:0/1013672306 shutdown_connections 2024-06-03T09:07:01.904 INFO:teuthology.orchestra.run.smithi088.stderr:2024-06-03T09:07:01.900+0000 7fbddc8e7640 1 -- 172.21.15.88:0/1013672306 >> 172.21.15.88:0/1013672306 conn(0x7fbdd80b2f20 msgr2=0x7fbdd80b3300 unknown :-1 s=STATE_NONE l=0).mark_down 2024-06-03T09:07:01.904 INFO:teuthology.orchestra.run.smithi088.stderr:2024-06-03T09:07:01.900+0000 7fbddc8e7640 1 -- 172.21.15.88:0/1013672306 shutdown_connections 2024-06-03T09:07:01.904 INFO:teuthology.orchestra.run.smithi088.stderr:2024-06-03T09:07:01.900+0000 7fbddc8e7640 1 -- 172.21.15.88:0/1013672306 wait complete. 2024-06-03T09:07:01.904 INFO:teuthology.orchestra.run.smithi088.stderr:2024-06-03T09:07:01.901+0000 7fbddc8e7640 10 monclient: build_initial_monmap 2024-06-03T09:07:01.904 INFO:teuthology.orchestra.run.smithi088.stderr:2024-06-03T09:07:01.901+0000 7fbddc8e7640 10 monclient: monmap: 2024-06-03T09:07:01.905 INFO:teuthology.orchestra.run.smithi088.stderr:epoch 0 2024-06-03T09:07:01.905 INFO:teuthology.orchestra.run.smithi088.stderr:fsid 00000000-0000-0000-0000-000000000000 2024-06-03T09:07:01.905 INFO:teuthology.orchestra.run.smithi088.stderr:last_changed 0.000000 2024-06-03T09:07:01.905 INFO:teuthology.orchestra.run.smithi088.stderr:created 0.000000 2024-06-03T09:07:01.905 INFO:teuthology.orchestra.run.smithi088.stderr:min_mon_release 0 (unknown) 2024-06-03T09:07:01.905 INFO:teuthology.orchestra.run.smithi088.stderr:election_strategy: 1 2024-06-03T09:07:01.905 INFO:teuthology.orchestra.run.smithi088.stderr:0: v1:172.21.15.88:6789/0 mon.noname-a 2024-06-03T09:07:01.905 INFO:teuthology.orchestra.run.smithi088.stderr:1: v1:172.21.15.192:6789/0 mon.noname-b 2024-06-03T09:07:01.905 INFO:teuthology.orchestra.run.smithi088.stderr:2: v1:172.21.15.88:6790/0 mon.noname-c 2024-06-03T09:07:01.905 INFO:teuthology.orchestra.run.smithi088.stderr: 2024-06-03T09:07:01.905 INFO:teuthology.orchestra.run.smithi088.stderr:2024-06-03T09:07:01.901+0000 7fbddc8e7640 1 librados: starting msgr at 2024-06-03T09:07:01.905 INFO:teuthology.orchestra.run.smithi088.stderr:2024-06-03T09:07:01.901+0000 7fbddc8e7640 1 librados: starting objecter 2024-06-03T09:07:01.905 INFO:teuthology.orchestra.run.smithi088.stderr:2024-06-03T09:07:01.901+0000 7fbddc8e7640 1 Processor -- start 2024-06-03T09:07:01.906 INFO:teuthology.orchestra.run.smithi088.stderr:2024-06-03T09:07:01.901+0000 7fbddc8e7640 1 -- start start 2024-06-03T09:07:01.906 INFO:teuthology.orchestra.run.smithi088.stderr:2024-06-03T09:07:01.901+0000 7fbddc8e7640 1 librados: setting wanted keys 2024-06-03T09:07:01.906 INFO:teuthology.orchestra.run.smithi088.stderr:2024-06-03T09:07:01.901+0000 7fbddc8e7640 1 librados: calling monclient init 2024-06-03T09:07:01.906 INFO:teuthology.orchestra.run.smithi088.stderr:2024-06-03T09:07:01.901+0000 7fbddc8e7640 10 monclient: init 2024-06-03T09:07:01.906 INFO:teuthology.orchestra.run.smithi088.stderr:2024-06-03T09:07:01.901+0000 7fbddc8e7640 10 monclient: _reopen_session rank -1 2024-06-03T09:07:01.906 INFO:teuthology.orchestra.run.smithi088.stderr:2024-06-03T09:07:01.901+0000 7fbddc8e7640 10 monclient: _add_conns ranks=[0,1,2] 2024-06-03T09:07:01.906 INFO:teuthology.orchestra.run.smithi088.stderr:2024-06-03T09:07:01.901+0000 7fbddc8e7640 10 monclient(hunting): picked mon.noname-a con 0x7fbdd806fc70 addr v1:172.21.15.88:6789/0 2024-06-03T09:07:01.906 INFO:teuthology.orchestra.run.smithi088.stderr:2024-06-03T09:07:01.901+0000 7fbddc8e7640 10 monclient(hunting): picked mon.noname-b con 0x7fbdd80ad920 addr v1:172.21.15.192:6789/0 2024-06-03T09:07:01.906 INFO:teuthology.orchestra.run.smithi088.stderr:2024-06-03T09:07:01.901+0000 7fbddc8e7640 10 monclient(hunting): picked mon.noname-c con 0x7fbdd81393f0 addr v1:172.21.15.88:6790/0 2024-06-03T09:07:01.906 INFO:teuthology.orchestra.run.smithi088.stderr:2024-06-03T09:07:01.901+0000 7fbddc8e7640 1 -- --> v1:172.21.15.88:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fbdd8142be0 con 0x7fbdd806fc70 2024-06-03T09:07:01.906 INFO:teuthology.orchestra.run.smithi088.stderr:2024-06-03T09:07:01.902+0000 7fbddc8e7640 1 -- --> v1:172.21.15.192:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fbdd8142e30 con 0x7fbdd80ad920 2024-06-03T09:07:01.906 INFO:teuthology.orchestra.run.smithi088.stderr:2024-06-03T09:07:01.902+0000 7fbddc8e7640 1 -- --> v1:172.21.15.88:6790/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fbdd8143110 con 0x7fbdd81393f0 2024-06-03T09:07:01.906 INFO:teuthology.orchestra.run.smithi088.stderr:2024-06-03T09:07:01.902+0000 7fbddc8e7640 10 monclient(hunting): _renew_subs 2024-06-03T09:07:01.907 INFO:teuthology.orchestra.run.smithi088.stderr:2024-06-03T09:07:01.902+0000 7fbddc8e7640 10 monclient(hunting): authenticate will time out at 3077.711954s 2024-06-03T09:07:01.907 INFO:teuthology.orchestra.run.smithi088.stderr:2024-06-03T09:07:01.902+0000 7fbdd6d9d640 1 --1- >> v1:172.21.15.88:6790/0 conn(0x7fbdd81393f0 0x7fbdd8134770 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=1).handle_server_banner_and_identify peer v1:172.21.15.88:6790/0 says I am v1:172.21.15.88:51188/0 (socket says 172.21.15.88:51188) 2024-06-03T09:07:01.907 INFO:teuthology.orchestra.run.smithi088.stderr:2024-06-03T09:07:01.902+0000 7fbdd6d9d640 1 -- 172.21.15.88:0/1234511928 learned_addr learned my addr 172.21.15.88:0/1234511928 (peer_addr_for_me v1:172.21.15.88:0/0) 2024-06-03T09:07:01.907 INFO:teuthology.orchestra.run.smithi088.stderr:2024-06-03T09:07:01.902+0000 7fbdc6ffd640 10 client.?.objecter ms_handle_connect 0x7fbdd806fc70 2024-06-03T09:07:01.907 INFO:teuthology.orchestra.run.smithi088.stderr:2024-06-03T09:07:01.902+0000 7fbdc6ffd640 10 client.?.objecter resend_mon_ops 2024-06-03T09:07:01.907 INFO:teuthology.orchestra.run.smithi088.stderr:2024-06-03T09:07:01.902+0000 7fbdc6ffd640 10 client.?.objecter ms_handle_connect 0x7fbdd81393f0 2024-06-03T09:07:01.907 INFO:teuthology.orchestra.run.smithi088.stderr:2024-06-03T09:07:01.902+0000 7fbdc6ffd640 10 client.?.objecter resend_mon_ops 2024-06-03T09:07:01.907 INFO:teuthology.orchestra.run.smithi088.stderr:2024-06-03T09:07:01.902+0000 7fbdc6ffd640 10 client.?.objecter ms_handle_connect 0x7fbdd80ad920 2024-06-03T09:07:01.907 INFO:teuthology.orchestra.run.smithi088.stderr:2024-06-03T09:07:01.902+0000 7fbdc6ffd640 10 client.?.objecter resend_mon_ops 2024-06-03T09:07:01.908 INFO:teuthology.orchestra.run.smithi088.stderr:2024-06-03T09:07:01.904+0000 7fbdc6ffd640 1 -- 172.21.15.88:0/1234511928 <== mon.0 v1:172.21.15.88:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 129302744 0 0) 0x7fbdd8142be0 con 0x7fbdd806fc70 2024-06-03T09:07:01.908 INFO:teuthology.orchestra.run.smithi088.stderr:2024-06-03T09:07:01.904+0000 7fbdc6ffd640 10 monclient(hunting): _negotiate 2024-06-03T09:07:01.908 INFO:teuthology.orchestra.run.smithi088.stderr:2024-06-03T09:07:01.904+0000 7fbdc6ffd640 10 monclient(hunting): _init_auth method 2 2024-06-03T09:07:01.908 INFO:teuthology.orchestra.run.smithi088.stderr:2024-06-03T09:07:01.904+0000 7fbdc6ffd640 10 monclient(hunting): _init_auth creating new auth 2024-06-03T09:07:01.909 INFO:teuthology.orchestra.run.smithi088.stderr:2024-06-03T09:07:01.904+0000 7fbdc6ffd640 10 monclient(hunting): my global_id is 30816 2024-06-03T09:07:01.909 INFO:teuthology.orchestra.run.smithi088.stderr:2024-06-03T09:07:01.904+0000 7fbdc6ffd640 1 -- 172.21.15.88:0/1234511928 --> v1:172.21.15.88:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7fbdb8004eb0 con 0x7fbdd806fc70 2024-06-03T09:07:01.909 INFO:teuthology.orchestra.run.smithi088.stderr:2024-06-03T09:07:01.904+0000 7fbdc6ffd640 1 -- 172.21.15.88:0/1234511928 <== mon.1 v1:172.21.15.192:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 2116378239 0 0) 0x7fbdd8142e30 con 0x7fbdd80ad920 2024-06-03T09:07:01.909 INFO:teuthology.orchestra.run.smithi088.stderr:2024-06-03T09:07:01.904+0000 7fbdc6ffd640 10 monclient(hunting): _negotiate 2024-06-03T09:07:01.909 INFO:teuthology.orchestra.run.smithi088.stderr:2024-06-03T09:07:01.904+0000 7fbdc6ffd640 10 monclient(hunting): _init_auth method 2 2024-06-03T09:07:01.909 INFO:teuthology.orchestra.run.smithi088.stderr:2024-06-03T09:07:01.904+0000 7fbdc6ffd640 10 monclient(hunting): _init_auth creating new auth 2024-06-03T09:07:01.909 INFO:teuthology.orchestra.run.smithi088.stderr:2024-06-03T09:07:01.904+0000 7fbdc6ffd640 10 monclient(hunting): my global_id is 30391 2024-06-03T09:07:01.909 INFO:teuthology.orchestra.run.smithi088.stderr:2024-06-03T09:07:01.904+0000 7fbdc6ffd640 1 -- 172.21.15.88:0/1234511928 --> v1:172.21.15.192:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7fbdd8142be0 con 0x7fbdd80ad920 2024-06-03T09:07:01.909 INFO:teuthology.orchestra.run.smithi088.stderr:2024-06-03T09:07:01.904+0000 7fbdc6ffd640 1 -- 172.21.15.88:0/1234511928 <== mon.2 v1:172.21.15.88:6790/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 1316936864 0 0) 0x7fbdd8143110 con 0x7fbdd81393f0 2024-06-03T09:07:01.909 INFO:teuthology.orchestra.run.smithi088.stderr:2024-06-03T09:07:01.904+0000 7fbdc6ffd640 10 monclient(hunting): _negotiate 2024-06-03T09:07:01.909 INFO:teuthology.orchestra.run.smithi088.stderr:2024-06-03T09:07:01.904+0000 7fbdc6ffd640 10 monclient(hunting): _init_auth method 2 2024-06-03T09:07:01.909 INFO:teuthology.orchestra.run.smithi088.stderr:2024-06-03T09:07:01.904+0000 7fbdc6ffd640 10 monclient(hunting): _init_auth creating new auth 2024-06-03T09:07:01.910 INFO:teuthology.orchestra.run.smithi088.stderr:2024-06-03T09:07:01.904+0000 7fbdc6ffd640 10 monclient(hunting): my global_id is 30326 2024-06-03T09:07:01.910 INFO:teuthology.orchestra.run.smithi088.stderr:2024-06-03T09:07:01.904+0000 7fbdc6ffd640 1 -- 172.21.15.88:0/1234511928 --> v1:172.21.15.88:6790/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7fbdd8142e30 con 0x7fbdd81393f0 2024-06-03T09:07:01.911 INFO:teuthology.orchestra.run.smithi088.stderr:2024-06-03T09:07:01.909+0000 7fbdc6ffd640 1 -- 172.21.15.88:0/1234511928 <== mon.1 v1:172.21.15.192:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 764+0+0 (unknown 3290847721 0 0) 0x7fbdd8142be0 con 0x7fbdd80ad920 2024-06-03T09:07:01.911 INFO:teuthology.orchestra.run.smithi088.stderr:2024-06-03T09:07:01.909+0000 7fbdc6ffd640 1 -- 172.21.15.88:0/1234511928 --> v1:172.21.15.192:6789/0 -- auth(proto 2 165 bytes epoch 0) v1 -- 0x7fbdd8143110 con 0x7fbdd80ad920 2024-06-03T09:07:01.911 INFO:teuthology.orchestra.run.smithi088.stderr:2024-06-03T09:07:01.909+0000 7fbdc6ffd640 1 -- 172.21.15.88:0/1234511928 <== mon.1 v1:172.21.15.192:6789/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 2234374065 0 0) 0x7fbdcc001510 con 0x7fbdd80ad920 2024-06-03T09:07:01.911 INFO:teuthology.orchestra.run.smithi088.stderr:2024-06-03T09:07:01.909+0000 7fbdc6ffd640 10 monclient(hunting): handle_monmap mon_map magic: 0 v1 2024-06-03T09:07:01.912 INFO:teuthology.orchestra.run.smithi088.stderr:2024-06-03T09:07:01.909+0000 7fbdc6ffd640 10 monclient(hunting): got monmap 1 from mon.noname-b (according to old e1) 2024-06-03T09:07:01.912 INFO:teuthology.orchestra.run.smithi088.stderr:2024-06-03T09:07:01.909+0000 7fbdc6ffd640 10 monclient(hunting): dump: 2024-06-03T09:07:01.912 INFO:teuthology.orchestra.run.smithi088.stderr:epoch 1 2024-06-03T09:07:01.912 INFO:teuthology.orchestra.run.smithi088.stderr:fsid ec3c3fdd-3ac6-4d7d-9412-683130ed8b04 2024-06-03T09:07:01.912 INFO:teuthology.orchestra.run.smithi088.stderr:last_changed 2024-06-03T08:34:31.885125+0000 2024-06-03T09:07:01.912 INFO:teuthology.orchestra.run.smithi088.stderr:created 2024-06-03T08:34:31.885125+0000 2024-06-03T09:07:01.912 INFO:teuthology.orchestra.run.smithi088.stderr:min_mon_release 18 (reef) 2024-06-03T09:07:01.912 INFO:teuthology.orchestra.run.smithi088.stderr:election_strategy: 3 2024-06-03T09:07:01.912 INFO:teuthology.orchestra.run.smithi088.stderr:0: v1:172.21.15.88:6789/0 mon.a 2024-06-03T09:07:01.913 INFO:teuthology.orchestra.run.smithi088.stderr:1: v1:172.21.15.192:6789/0 mon.b 2024-06-03T09:07:01.913 INFO:teuthology.orchestra.run.smithi088.stderr:2: v1:172.21.15.88:6790/0 mon.c 2024-06-03T09:07:01.913 INFO:teuthology.orchestra.run.smithi088.stderr: 2024-06-03T09:07:01.913 INFO:teuthology.orchestra.run.smithi088.stderr:2024-06-03T09:07:01.909+0000 7fbdc6ffd640 10 monclient(hunting): _finish_auth 0 2024-06-03T09:07:01.913 INFO:teuthology.orchestra.run.smithi088.stderr:2024-06-03T09:07:01.909+0000 7fbdc6ffd640 1 -- 172.21.15.88:0/1234511928 <== mon.2 v1:172.21.15.88:6790/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 764+0+0 (unknown 437521675 0 0) 0x7fbdd8142e30 con 0x7fbdd81393f0 2024-06-03T09:07:01.913 INFO:teuthology.orchestra.run.smithi088.stderr:2024-06-03T09:07:01.909+0000 7fbdc6ffd640 1 -- 172.21.15.88:0/1234511928 --> v1:172.21.15.88:6790/0 -- auth(proto 2 165 bytes epoch 0) v1 -- 0x7fbdd8142be0 con 0x7fbdd81393f0 2024-06-03T09:07:01.913 INFO:teuthology.orchestra.run.smithi088.stderr:2024-06-03T09:07:01.909+0000 7fbdc6ffd640 1 -- 172.21.15.88:0/1234511928 <== mon.2 v1:172.21.15.88:6790/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 2234374065 0 0) 0x7fbdd0001c00 con 0x7fbdd81393f0 2024-06-03T09:07:01.913 INFO:teuthology.orchestra.run.smithi088.stderr:2024-06-03T09:07:01.909+0000 7fbdc6ffd640 10 monclient(hunting): handle_monmap mon_map magic: 0 v1 2024-06-03T09:07:01.913 INFO:teuthology.orchestra.run.smithi088.stderr:2024-06-03T09:07:01.909+0000 7fbdc6ffd640 10 monclient(hunting): got monmap 1 from mon.c (according to old e1) 2024-06-03T09:07:01.913 INFO:teuthology.orchestra.run.smithi088.stderr:2024-06-03T09:07:01.909+0000 7fbdc6ffd640 10 monclient(hunting): dump: 2024-06-03T09:07:01.913 INFO:teuthology.orchestra.run.smithi088.stderr:epoch 1 2024-06-03T09:07:01.913 INFO:teuthology.orchestra.run.smithi088.stderr:fsid ec3c3fdd-3ac6-4d7d-9412-683130ed8b04 2024-06-03T09:07:01.913 INFO:teuthology.orchestra.run.smithi088.stderr:last_changed 2024-06-03T08:34:31.885125+0000 2024-06-03T09:07:01.914 INFO:teuthology.orchestra.run.smithi088.stderr:created 2024-06-03T08:34:31.885125+0000 2024-06-03T09:07:01.914 INFO:teuthology.orchestra.run.smithi088.stderr:min_mon_release 18 (reef) 2024-06-03T09:07:01.914 INFO:teuthology.orchestra.run.smithi088.stderr:election_strategy: 3 2024-06-03T09:07:01.914 INFO:teuthology.orchestra.run.smithi088.stderr:0: v1:172.21.15.88:6789/0 mon.a 2024-06-03T09:07:01.914 INFO:teuthology.orchestra.run.smithi088.stderr:1: v1:172.21.15.192:6789/0 mon.b 2024-06-03T09:07:01.914 INFO:teuthology.orchestra.run.smithi088.stderr:2: v1:172.21.15.88:6790/0 mon.c 2024-06-03T09:07:01.914 INFO:teuthology.orchestra.run.smithi088.stderr: 2024-06-03T09:07:01.914 INFO:teuthology.orchestra.run.smithi088.stderr:2024-06-03T09:07:01.910+0000 7fbdc6ffd640 1 -- 172.21.15.88:0/1234511928 <== mon.0 v1:172.21.15.88:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 764+0+0 (unknown 3636425291 0 0) 0x7fbdb8004eb0 con 0x7fbdd806fc70 2024-06-03T09:07:01.914 INFO:teuthology.orchestra.run.smithi088.stderr:2024-06-03T09:07:01.910+0000 7fbdc6ffd640 1 -- 172.21.15.88:0/1234511928 --> v1:172.21.15.88:6789/0 -- auth(proto 2 165 bytes epoch 0) v1 -- 0x7fbdd8142e30 con 0x7fbdd806fc70 2024-06-03T09:07:01.914 INFO:teuthology.orchestra.run.smithi088.stderr:2024-06-03T09:07:01.910+0000 7fbdc6ffd640 1 -- 172.21.15.88:0/1234511928 <== mon.0 v1:172.21.15.88:6789/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 2234374065 0 0) 0x7fbdc80018b0 con 0x7fbdd806fc70 2024-06-03T09:07:01.914 INFO:teuthology.orchestra.run.smithi088.stderr:2024-06-03T09:07:01.910+0000 7fbdc6ffd640 10 monclient(hunting): handle_monmap mon_map magic: 0 v1 2024-06-03T09:07:01.914 INFO:teuthology.orchestra.run.smithi088.stderr:2024-06-03T09:07:01.910+0000 7fbdc6ffd640 10 monclient(hunting): got monmap 1 from mon.a (according to old e1) 2024-06-03T09:07:01.915 INFO:teuthology.orchestra.run.smithi088.stderr:2024-06-03T09:07:01.910+0000 7fbdc6ffd640 10 monclient(hunting): dump: 2024-06-03T09:07:01.915 INFO:teuthology.orchestra.run.smithi088.stderr:epoch 1 2024-06-03T09:07:01.915 INFO:teuthology.orchestra.run.smithi088.stderr:fsid ec3c3fdd-3ac6-4d7d-9412-683130ed8b04 2024-06-03T09:07:01.915 INFO:teuthology.orchestra.run.smithi088.stderr:last_changed 2024-06-03T08:34:31.885125+0000 2024-06-03T09:07:01.915 INFO:teuthology.orchestra.run.smithi088.stderr:created 2024-06-03T08:34:31.885125+0000 2024-06-03T09:07:01.915 INFO:teuthology.orchestra.run.smithi088.stderr:min_mon_release 18 (reef) 2024-06-03T09:07:01.915 INFO:teuthology.orchestra.run.smithi088.stderr:election_strategy: 3 2024-06-03T09:07:01.915 INFO:teuthology.orchestra.run.smithi088.stderr:0: v1:172.21.15.88:6789/0 mon.a 2024-06-03T09:07:01.915 INFO:teuthology.orchestra.run.smithi088.stderr:1: v1:172.21.15.192:6789/0 mon.b 2024-06-03T09:07:01.915 INFO:teuthology.orchestra.run.smithi088.stderr:2: v1:172.21.15.88:6790/0 mon.c 2024-06-03T09:07:01.916 INFO:teuthology.orchestra.run.smithi088.stderr: 2024-06-03T09:07:01.917 INFO:teuthology.orchestra.run.smithi088.stderr:2024-06-03T09:07:01.912+0000 7fbdc6ffd640 1 -- 172.21.15.88:0/1234511928 <== mon.1 v1:172.21.15.192:6789/0 4 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 580+0+0 (unknown 2077720875 0 0) 0x7fbdd8143110 con 0x7fbdd80ad920 2024-06-03T09:07:01.917 INFO:teuthology.orchestra.run.smithi088.stderr:2024-06-03T09:07:01.912+0000 7fbdc6ffd640 1 -- 172.21.15.88:0/1234511928 >> v1:172.21.15.88:6790/0 conn(0x7fbdd81393f0 legacy=0x7fbdd8134770 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-06-03T09:07:01.917 INFO:teuthology.orchestra.run.smithi088.stderr:2024-06-03T09:07:01.912+0000 7fbdc6ffd640 1 -- 172.21.15.88:0/1234511928 >> v1:172.21.15.88:6789/0 conn(0x7fbdd806fc70 legacy=0x7fbdd806d370 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-06-03T09:07:01.917 INFO:teuthology.orchestra.run.smithi088.stderr:2024-06-03T09:07:01.912+0000 7fbdc6ffd640 10 monclient: _finish_hunting 0 2024-06-03T09:07:01.917 INFO:teuthology.orchestra.run.smithi088.stderr:2024-06-03T09:07:01.912+0000 7fbdc6ffd640 1 monclient: found mon.b 2024-06-03T09:07:01.917 INFO:teuthology.orchestra.run.smithi088.stderr:2024-06-03T09:07:01.912+0000 7fbdc6ffd640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-06-03T09:07:01.917 INFO:teuthology.orchestra.run.smithi088.stderr:2024-06-03T09:07:01.912+0000 7fbdc6ffd640 10 monclient: _send_mon_message to mon.b at v1:172.21.15.192:6789/0 2024-06-03T09:07:01.918 INFO:teuthology.orchestra.run.smithi088.stderr:2024-06-03T09:07:01.912+0000 7fbdc6ffd640 1 -- 172.21.15.88:0/1234511928 --> v1:172.21.15.192:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fbdd8140e90 con 0x7fbdd80ad920 2024-06-03T09:07:01.918 INFO:teuthology.orchestra.run.smithi088.stderr:2024-06-03T09:07:01.912+0000 7fbdc6ffd640 10 monclient: _finish_auth 0 2024-06-03T09:07:01.918 INFO:teuthology.orchestra.run.smithi088.stderr:2024-06-03T09:07:01.912+0000 7fbdc6ffd640 10 monclient: _check_auth_tickets 2024-06-03T09:07:01.918 INFO:teuthology.orchestra.run.smithi088.stderr:2024-06-03T09:07:01.912+0000 7fbdc6ffd640 20 monclient: _check_auth_rotating not needed by client.admin 2024-06-03T09:07:01.918 INFO:teuthology.orchestra.run.smithi088.stderr:2024-06-03T09:07:01.912+0000 7fbddc8e7640 5 monclient: authenticate success, global_id 30391 2024-06-03T09:07:01.918 INFO:teuthology.orchestra.run.smithi088.stderr:2024-06-03T09:07:01.912+0000 7fbddc8e7640 10 monclient: _renew_subs 2024-06-03T09:07:01.918 INFO:teuthology.orchestra.run.smithi088.stderr:2024-06-03T09:07:01.912+0000 7fbddc8e7640 10 monclient: _send_mon_message to mon.b at v1:172.21.15.192:6789/0 2024-06-03T09:07:01.918 INFO:teuthology.orchestra.run.smithi088.stderr:2024-06-03T09:07:01.912+0000 7fbddc8e7640 1 -- 172.21.15.88:0/1234511928 --> v1:172.21.15.192:6789/0 -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fbdd8134e80 con 0x7fbdd80ad920 2024-06-03T09:07:01.918 INFO:teuthology.orchestra.run.smithi088.stderr:2024-06-03T09:07:01.912+0000 7fbddc8e7640 10 client.30391.objecter _maybe_request_map subscribing (onetime) to next osd map 2024-06-03T09:07:01.918 INFO:teuthology.orchestra.run.smithi088.stderr:2024-06-03T09:07:01.912+0000 7fbddc8e7640 10 monclient: _renew_subs 2024-06-03T09:07:01.918 INFO:teuthology.orchestra.run.smithi088.stderr:2024-06-03T09:07:01.912+0000 7fbddc8e7640 10 monclient: _send_mon_message to mon.b at v1:172.21.15.192:6789/0 2024-06-03T09:07:01.918 INFO:teuthology.orchestra.run.smithi088.stderr:2024-06-03T09:07:01.912+0000 7fbddc8e7640 1 -- 172.21.15.88:0/1234511928 --> v1:172.21.15.192:6789/0 -- mon_subscribe({osdmap=0}) v3 -- 0x7fbdd8143530 con 0x7fbdd80ad920 2024-06-03T09:07:01.918 INFO:teuthology.orchestra.run.smithi088.stderr:2024-06-03T09:07:01.912+0000 7fbddc8e7640 1 librados: init done 2024-06-03T09:07:01.919 INFO:teuthology.orchestra.run.smithi088.stderr:2024-06-03T09:07:01.913+0000 7fbddc8e7640 10 client.30391.objecter _submit_command 1 [{"prefix": "get_command_descriptions"}] 2024-06-03T09:07:01.919 INFO:teuthology.orchestra.run.smithi088.stderr:2024-06-03T09:07:01.913+0000 7fbddc8e7640 15 client.30391.objecter _session_command_op_assign -1 1 2024-06-03T09:07:01.919 INFO:teuthology.orchestra.run.smithi088.stderr:2024-06-03T09:07:01.913+0000 7fbddc8e7640 20 client.30391.objecter _get_session osd=-1 returning homeless 2024-06-03T09:07:01.919 INFO:teuthology.orchestra.run.smithi088.stderr:2024-06-03T09:07:01.913+0000 7fbddc8e7640 10 client.30391.objecter _maybe_request_map subscribing (onetime) to next osd map 2024-06-03T09:07:01.919 INFO:teuthology.orchestra.run.smithi088.stderr:2024-06-03T09:07:01.913+0000 7fbddc8e7640 10 monclient: _send_mon_message to mon.b at v1:172.21.15.192:6789/0 2024-06-03T09:07:01.919 INFO:teuthology.orchestra.run.smithi088.stderr:2024-06-03T09:07:01.913+0000 7fbddc8e7640 1 -- 172.21.15.88:0/1234511928 --> v1:172.21.15.192:6789/0 -- mon_get_version(what=osdmap handle=1) v1 -- 0x7fbdd813ce70 con 0x7fbdd80ad920 2024-06-03T09:07:01.928 INFO:teuthology.orchestra.run.smithi088.stderr:2024-06-03T09:07:01.927+0000 7fbdc6ffd640 1 -- 172.21.15.88:0/1234511928 <== mon.1 v1:172.21.15.192:6789/0 5 ==== config(0 keys) v1 ==== 4+0+0 (unknown 0 0 0) 0x7fbdcc0038a0 con 0x7fbdd80ad920 2024-06-03T09:07:01.928 INFO:teuthology.orchestra.run.smithi088.stderr:2024-06-03T09:07:01.927+0000 7fbdc6ffd640 10 monclient: handle_config config(0 keys) v1 2024-06-03T09:07:01.928 INFO:teuthology.orchestra.run.smithi088.stderr:2024-06-03T09:07:01.927+0000 7fbdc6ffd640 1 -- 172.21.15.88:0/1234511928 <== mon.1 v1:172.21.15.192:6789/0 6 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 2234374065 0 0) 0x7fbdcc001210 con 0x7fbdd80ad920 2024-06-03T09:07:01.928 INFO:teuthology.orchestra.run.smithi088.stderr:2024-06-03T09:07:01.927+0000 7fbdc6ffd640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-06-03T09:07:01.928 INFO:teuthology.orchestra.run.smithi088.stderr:2024-06-03T09:07:01.927+0000 7fbdc6ffd640 10 monclient: got monmap 1 from mon.b (according to old e1) 2024-06-03T09:07:01.928 INFO:teuthology.orchestra.run.smithi088.stderr:2024-06-03T09:07:01.927+0000 7fbdc6ffd640 10 monclient: dump: 2024-06-03T09:07:01.928 INFO:teuthology.orchestra.run.smithi088.stderr:epoch 1 2024-06-03T09:07:01.929 INFO:teuthology.orchestra.run.smithi088.stderr:fsid ec3c3fdd-3ac6-4d7d-9412-683130ed8b04 2024-06-03T09:07:01.929 INFO:teuthology.orchestra.run.smithi088.stderr:last_changed 2024-06-03T08:34:31.885125+0000 2024-06-03T09:07:01.929 INFO:teuthology.orchestra.run.smithi088.stderr:created 2024-06-03T08:34:31.885125+0000 2024-06-03T09:07:01.929 INFO:teuthology.orchestra.run.smithi088.stderr:min_mon_release 18 (reef) 2024-06-03T09:07:01.929 INFO:teuthology.orchestra.run.smithi088.stderr:election_strategy: 3 2024-06-03T09:07:01.929 INFO:teuthology.orchestra.run.smithi088.stderr:0: v1:172.21.15.88:6789/0 mon.a 2024-06-03T09:07:01.929 INFO:teuthology.orchestra.run.smithi088.stderr:1: v1:172.21.15.192:6789/0 mon.b 2024-06-03T09:07:01.930 INFO:teuthology.orchestra.run.smithi088.stderr:2: v1:172.21.15.88:6790/0 mon.c 2024-06-03T09:07:01.930 INFO:teuthology.orchestra.run.smithi088.stderr: 2024-06-03T09:07:01.930 INFO:teuthology.orchestra.run.smithi088.stderr:2024-06-03T09:07:01.927+0000 7fbdd4d99640 4 set_mon_vals no callback set 2024-06-03T09:07:01.931 INFO:teuthology.orchestra.run.smithi088.stderr:2024-06-03T09:07:01.928+0000 7fbdc6ffd640 1 -- 172.21.15.88:0/1234511928 <== mon.1 v1:172.21.15.192:6789/0 7 ==== mgrmap(e 5) v1 ==== 89110+0+0 (unknown 873533973 0 0) 0x7fbdcc0197d0 con 0x7fbdd80ad920 2024-06-03T09:07:01.931 INFO:teuthology.orchestra.run.smithi088.stderr:2024-06-03T09:07:01.929+0000 7fbdd659c640 10 monclient: get_auth_request con 0x7fbdb806d380 auth_method 0 2024-06-03T09:07:01.931 INFO:teuthology.orchestra.run.smithi088.stderr:2024-06-03T09:07:01.929+0000 7fbdc6ffd640 10 client.30391.objecter ms_handle_connect 0x7fbdb806d380 2024-06-03T09:07:01.932 INFO:teuthology.orchestra.run.smithi088.stderr:2024-06-03T09:07:01.930+0000 7fbdc6ffd640 1 -- 172.21.15.88:0/1234511928 <== mon.1 v1:172.21.15.192:6789/0 8 ==== osd_map(1014..1014 src has 622..1014) v4 ==== 5229+0+0 (unknown 3230192161 0 0) 0x7fbdcc052fc0 con 0x7fbdd80ad920 2024-06-03T09:07:01.932 INFO:teuthology.orchestra.run.smithi088.stderr:2024-06-03T09:07:01.930+0000 7fbdc6ffd640 10 client.30391.objecter ms_dispatch 0x7fbdd8000b90 osd_map(1014..1014 src has 622..1014) v4 2024-06-03T09:07:01.932 INFO:teuthology.orchestra.run.smithi088.stderr:2024-06-03T09:07:01.930+0000 7fbdc6ffd640 3 client.30391.objecter handle_osd_map got epochs [1014,1014] > 0 2024-06-03T09:07:01.932 INFO:teuthology.orchestra.run.smithi088.stderr:2024-06-03T09:07:01.930+0000 7fbdc6ffd640 3 client.30391.objecter handle_osd_map decoding full epoch 1014 2024-06-03T09:07:01.932 INFO:teuthology.orchestra.run.smithi088.stderr:2024-06-03T09:07:01.930+0000 7fbdc6ffd640 10 client.30391.objecter checking command 1 2024-06-03T09:07:01.932 INFO:teuthology.orchestra.run.smithi088.stderr:2024-06-03T09:07:01.930+0000 7fbdc6ffd640 20 client.30391.objecter _get_session s=0x7fbdb8075350 osd=7 3 2024-06-03T09:07:01.932 INFO:teuthology.orchestra.run.smithi088.stderr:2024-06-03T09:07:01.930+0000 7fbdc6ffd640 20 client.30391.objecter put_session s=0x7fbdb8075350 osd=7 3 2024-06-03T09:07:01.932 INFO:teuthology.orchestra.run.smithi088.stderr:2024-06-03T09:07:01.930+0000 7fbdc6ffd640 15 client.30391.objecter _session_command_op_remove -1 1 2024-06-03T09:07:01.932 INFO:teuthology.orchestra.run.smithi088.stderr:2024-06-03T09:07:01.930+0000 7fbdc6ffd640 20 client.30391.objecter _get_session s=0x7fbdb8075350 osd=7 3 2024-06-03T09:07:01.932 INFO:teuthology.orchestra.run.smithi088.stderr:2024-06-03T09:07:01.930+0000 7fbdc6ffd640 20 client.30391.objecter get_session s=0x7fbdb8075350 osd=7 3 2024-06-03T09:07:01.933 INFO:teuthology.orchestra.run.smithi088.stderr:2024-06-03T09:07:01.930+0000 7fbdc6ffd640 15 client.30391.objecter _session_command_op_assign 7 1 2024-06-03T09:07:01.933 INFO:teuthology.orchestra.run.smithi088.stderr:2024-06-03T09:07:01.930+0000 7fbdc6ffd640 20 client.30391.objecter put_session s=0x7fbdb8075350 osd=7 4 2024-06-03T09:07:01.933 INFO:teuthology.orchestra.run.smithi088.stderr:2024-06-03T09:07:01.930+0000 7fbdc6ffd640 10 client.30391.objecter _send_command 1 2024-06-03T09:07:01.933 INFO:teuthology.orchestra.run.smithi088.stderr:2024-06-03T09:07:01.930+0000 7fbdc6ffd640 1 -- 172.21.15.88:0/1234511928 --> v1:172.21.15.192:6801/34787 -- command(tid 1: {"prefix": "get_command_descriptions"}) v1 -- 0x7fbdb8079670 con 0x7fbdb8075a00 2024-06-03T09:07:01.933 INFO:teuthology.orchestra.run.smithi088.stderr:2024-06-03T09:07:01.930+0000 7fbdc6ffd640 20 client.30391.objecter dump_active .. 0 homeless 2024-06-03T09:07:01.933 INFO:teuthology.orchestra.run.smithi088.stderr:2024-06-03T09:07:01.931+0000 7fbdd6d9d640 10 monclient: get_auth_request con 0x7fbdb8075a00 auth_method 0 2024-06-03T09:07:01.933 INFO:teuthology.orchestra.run.smithi088.stderr:2024-06-03T09:07:01.931+0000 7fbdc6ffd640 1 -- 172.21.15.88:0/1234511928 <== mon.1 v1:172.21.15.192:6789/0 9 ==== mon_get_version_reply(handle=1 version=1014) v2 ==== 24+0+0 (unknown 3501658476 0 0) 0x7fbdcc04eb70 con 0x7fbdd80ad920 2024-06-03T09:07:01.933 INFO:teuthology.orchestra.run.smithi088.stderr:2024-06-03T09:07:01.931+0000 7fbdc6ffd640 10 monclient: handle_get_version_reply finishing 1 version 1014 2024-06-03T09:07:01.934 INFO:teuthology.orchestra.run.smithi088.stderr:2024-06-03T09:07:01.932+0000 7fbdc6ffd640 10 client.30391.objecter ms_handle_connect 0x7fbdb8075a00 2024-06-03T09:07:01.957 INFO:teuthology.orchestra.run.smithi088.stderr:2024-06-03T09:07:01.957+0000 7fbdc6ffd640 1 -- 172.21.15.88:0/1234511928 <== osd.7 v1:172.21.15.192:6801/34787 1 ==== command_reply(tid 1: 0 ) v1 ==== 8+0+26118 (unknown 0 0 1686565167) 0x7fbdb8079670 con 0x7fbdb8075a00 2024-06-03T09:07:01.957 INFO:teuthology.orchestra.run.smithi088.stderr:2024-06-03T09:07:01.957+0000 7fbdc6ffd640 10 client.30391.objecter ms_dispatch 0x7fbdd8000b90 command_reply(tid 1: 0 ) v1 2024-06-03T09:07:01.957 INFO:teuthology.orchestra.run.smithi088.stderr:2024-06-03T09:07:01.957+0000 7fbdc6ffd640 10 client.30391.objecter _finish_command 1 = system:0 2024-06-03T09:07:01.957 INFO:teuthology.orchestra.run.smithi088.stderr:2024-06-03T09:07:01.957+0000 7fbdc6ffd640 20 client.30391.objecter put_session s=0x7fbdb8075350 osd=7 4 2024-06-03T09:07:01.957 INFO:teuthology.orchestra.run.smithi088.stderr:2024-06-03T09:07:01.957+0000 7fbdc6ffd640 15 client.30391.objecter _session_command_op_remove 7 1 2024-06-03T09:07:01.970 INFO:teuthology.orchestra.run.smithi088.stderr:2024-06-03T09:07:01.970+0000 7fbddc8e7640 10 client.30391.objecter _submit_command 2 [{"prefix": "config set", "var": "osd_debug_deep_scrub_sleep", "val": ["0"]}] 2024-06-03T09:07:01.970 INFO:teuthology.orchestra.run.smithi088.stderr:2024-06-03T09:07:01.970+0000 7fbddc8e7640 15 client.30391.objecter _session_command_op_assign -1 2 2024-06-03T09:07:01.970 INFO:teuthology.orchestra.run.smithi088.stderr:2024-06-03T09:07:01.970+0000 7fbddc8e7640 20 client.30391.objecter _get_session s=0x7fbdb8075350 osd=7 3 2024-06-03T09:07:01.970 INFO:teuthology.orchestra.run.smithi088.stderr:2024-06-03T09:07:01.970+0000 7fbddc8e7640 20 client.30391.objecter put_session s=0x7fbdb8075350 osd=7 3 2024-06-03T09:07:01.970 INFO:teuthology.orchestra.run.smithi088.stderr:2024-06-03T09:07:01.970+0000 7fbddc8e7640 20 client.30391.objecter _get_session s=0x7fbdb8075350 osd=7 3 2024-06-03T09:07:01.970 INFO:teuthology.orchestra.run.smithi088.stderr:2024-06-03T09:07:01.970+0000 7fbddc8e7640 15 client.30391.objecter _session_command_op_remove -1 2 2024-06-03T09:07:01.970 INFO:teuthology.orchestra.run.smithi088.stderr:2024-06-03T09:07:01.970+0000 7fbddc8e7640 20 client.30391.objecter get_session s=0x7fbdb8075350 osd=7 3 2024-06-03T09:07:01.970 INFO:teuthology.orchestra.run.smithi088.stderr:2024-06-03T09:07:01.970+0000 7fbddc8e7640 15 client.30391.objecter _session_command_op_assign 7 2 2024-06-03T09:07:01.971 INFO:teuthology.orchestra.run.smithi088.stderr:2024-06-03T09:07:01.970+0000 7fbddc8e7640 20 client.30391.objecter put_session s=0x7fbdb8075350 osd=7 4 2024-06-03T09:07:01.971 INFO:teuthology.orchestra.run.smithi088.stderr:2024-06-03T09:07:01.970+0000 7fbddc8e7640 10 client.30391.objecter _send_command 2 2024-06-03T09:07:01.971 INFO:teuthology.orchestra.run.smithi088.stderr:2024-06-03T09:07:01.970+0000 7fbddc8e7640 1 -- 172.21.15.88:0/1234511928 --> v1:172.21.15.192:6801/34787 -- command(tid 2: {"prefix": "config set", "var": "osd_debug_deep_scrub_sleep", "val": ["0"]}) v1 -- 0x7fbdd8140c90 con 0x7fbdb8075a00 2024-06-03T09:07:01.985 INFO:teuthology.orchestra.run.smithi088.stderr:2024-06-03T09:07:01.985+0000 7fbdc6ffd640 1 -- 172.21.15.88:0/1234511928 <== osd.7 v1:172.21.15.192:6801/34787 2 ==== command_reply(tid 2: 0 ) v1 ==== 8+0+97 (unknown 0 0 119112422) 0x7fbdd8140c90 con 0x7fbdb8075a00 2024-06-03T09:07:01.985 INFO:teuthology.orchestra.run.smithi088.stderr:2024-06-03T09:07:01.985+0000 7fbdc6ffd640 10 client.30391.objecter ms_dispatch 0x7fbdd8000b90 command_reply(tid 2: 0 ) v1 2024-06-03T09:07:01.985 INFO:teuthology.orchestra.run.smithi088.stderr:2024-06-03T09:07:01.985+0000 7fbdc6ffd640 10 client.30391.objecter _finish_command 2 = system:0 2024-06-03T09:07:01.986 INFO:teuthology.orchestra.run.smithi088.stderr:2024-06-03T09:07:01.985+0000 7fbdc6ffd640 20 client.30391.objecter put_session s=0x7fbdb8075350 osd=7 4 2024-06-03T09:07:01.986 INFO:teuthology.orchestra.run.smithi088.stderr:2024-06-03T09:07:01.985+0000 7fbdc6ffd640 15 client.30391.objecter _session_command_op_remove 7 2 2024-06-03T09:07:01.986 INFO:teuthology.orchestra.run.smithi088.stdout:{ 2024-06-03T09:07:01.986 INFO:teuthology.orchestra.run.smithi088.stdout: "success": "osd_debug_deep_scrub_sleep = '' (not observed, change may require restart) " 2024-06-03T09:07:01.986 INFO:teuthology.orchestra.run.smithi088.stdout:} 2024-06-03T09:07:01.987 INFO:teuthology.orchestra.run.smithi088.stderr:2024-06-03T09:07:01.985+0000 7fbddc8e7640 10 librados: watch_flush enter 2024-06-03T09:07:01.987 INFO:teuthology.orchestra.run.smithi088.stderr:2024-06-03T09:07:01.985+0000 7fbddc8e7640 10 librados: watch_flush exit 2024-06-03T09:07:01.987 INFO:teuthology.orchestra.run.smithi088.stderr:2024-06-03T09:07:01.985+0000 7fbddc8e7640 10 client.30391.objecter close_session for osd.7 2024-06-03T09:07:01.988 INFO:teuthology.orchestra.run.smithi088.stderr:2024-06-03T09:07:01.985+0000 7fbddc8e7640 1 -- 172.21.15.88:0/1234511928 >> v1:172.21.15.192:6801/34787 conn(0x7fbdb8075a00 legacy=0x7fbdb8077ea0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-06-03T09:07:01.988 INFO:teuthology.orchestra.run.smithi088.stderr:2024-06-03T09:07:01.985+0000 7fbddc8e7640 20 client.30391.objecter put_session s=0x7fbdb8075350 osd=7 1 2024-06-03T09:07:01.988 INFO:teuthology.orchestra.run.smithi088.stderr:2024-06-03T09:07:01.985+0000 7fbddc8e7640 20 client.30391.objecter shutdown clearing up homeless session... 2024-06-03T09:07:01.988 INFO:teuthology.orchestra.run.smithi088.stderr:2024-06-03T09:07:01.985+0000 7fbddc8e7640 10 client.30391.objecter successfully canceled tick 2024-06-03T09:07:01.988 INFO:teuthology.orchestra.run.smithi088.stderr:2024-06-03T09:07:01.985+0000 7fbddc8e7640 1 -- 172.21.15.88:0/1234511928 >> v1:172.21.15.192:6800/34229 conn(0x7fbdb806d380 legacy=0x7fbdb806f840 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-06-03T09:07:01.988 INFO:teuthology.orchestra.run.smithi088.stderr:2024-06-03T09:07:01.985+0000 7fbddc8e7640 10 monclient: shutdown 2024-06-03T09:07:01.988 INFO:teuthology.orchestra.run.smithi088.stderr:2024-06-03T09:07:01.985+0000 7fbddc8e7640 20 monclient: shutdown discarding 0 pending message(s) 2024-06-03T09:07:01.988 INFO:teuthology.orchestra.run.smithi088.stderr:2024-06-03T09:07:01.985+0000 7fbddc8e7640 1 -- 172.21.15.88:0/1234511928 >> v1:172.21.15.192:6789/0 conn(0x7fbdd80ad920 legacy=0x7fbdd8134060 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-06-03T09:07:01.988 INFO:teuthology.orchestra.run.smithi088.stderr:2024-06-03T09:07:01.985+0000 7fbdd6d9d640 1 -- 172.21.15.88:0/1234511928 reap_dead start 2024-06-03T09:07:01.988 INFO:teuthology.orchestra.run.smithi088.stderr:2024-06-03T09:07:01.985+0000 7fbddc8e7640 1 -- 172.21.15.88:0/1234511928 shutdown_connections 2024-06-03T09:07:01.989 INFO:teuthology.orchestra.run.smithi088.stderr:2024-06-03T09:07:01.985+0000 7fbddc8e7640 1 -- 172.21.15.88:0/1234511928 >> 172.21.15.88:0/1234511928 conn(0x7fbdd80b2f20 msgr2=0x7fbdd813d9f0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-06-03T09:07:01.989 INFO:teuthology.orchestra.run.smithi088.stderr:2024-06-03T09:07:01.985+0000 7fbddc8e7640 1 -- 172.21.15.88:0/1234511928 shutdown_connections 2024-06-03T09:07:01.989 INFO:teuthology.orchestra.run.smithi088.stderr:2024-06-03T09:07:01.985+0000 7fbddc8e7640 1 -- 172.21.15.88:0/1234511928 wait complete. 2024-06-03T09:07:01.989 INFO:teuthology.orchestra.run.smithi088.stderr:2024-06-03T09:07:01.985+0000 7fbddc8e7640 1 librados: shutdown 2024-06-03T09:07:01.996 DEBUG:teuthology.orchestra.run.smithi088:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd deep-scrub 7 2024-06-03T09:07:02.095 INFO:teuthology.orchestra.run.smithi088.stderr:2024-06-03T09:07:02.095+0000 7f8a01d34640 10 monclient: get_monmap_and_config 2024-06-03T09:07:02.104 INFO:teuthology.orchestra.run.smithi088.stderr:2024-06-03T09:07:02.095+0000 7f8a01d34640 10 monclient: build_initial_monmap 2024-06-03T09:07:02.104 INFO:teuthology.orchestra.run.smithi088.stderr:2024-06-03T09:07:02.095+0000 7f8a01d34640 10 monclient: monmap: 2024-06-03T09:07:02.104 INFO:teuthology.orchestra.run.smithi088.stderr:epoch 0 2024-06-03T09:07:02.104 INFO:teuthology.orchestra.run.smithi088.stderr:fsid ec3c3fdd-3ac6-4d7d-9412-683130ed8b04 2024-06-03T09:07:02.104 INFO:teuthology.orchestra.run.smithi088.stderr:last_changed 2024-06-03T09:07:02.096364+0000 2024-06-03T09:07:02.104 INFO:teuthology.orchestra.run.smithi088.stderr:created 2024-06-03T09:07:02.096364+0000 2024-06-03T09:07:02.104 INFO:teuthology.orchestra.run.smithi088.stderr:min_mon_release 0 (unknown) 2024-06-03T09:07:02.104 INFO:teuthology.orchestra.run.smithi088.stderr:election_strategy: 3 2024-06-03T09:07:02.104 INFO:teuthology.orchestra.run.smithi088.stderr:0: v1:172.21.15.88:6789/0 mon.noname-a 2024-06-03T09:07:02.104 INFO:teuthology.orchestra.run.smithi088.stderr:1: v1:172.21.15.192:6789/0 mon.noname-c 2024-06-03T09:07:02.104 INFO:teuthology.orchestra.run.smithi088.stderr:2: v1:172.21.15.88:6790/0 mon.noname-b 2024-06-03T09:07:02.104 INFO:teuthology.orchestra.run.smithi088.stderr: 2024-06-03T09:07:02.110 INFO:teuthology.orchestra.run.smithi088.stderr:2024-06-03T09:07:02.096+0000 7f8a01d34640 1 Processor -- start 2024-06-03T09:07:02.110 INFO:teuthology.orchestra.run.smithi088.stderr:2024-06-03T09:07:02.096+0000 7f8a01d34640 1 -- start start 2024-06-03T09:07:02.110 INFO:teuthology.orchestra.run.smithi088.stderr:2024-06-03T09:07:02.096+0000 7f8a01d34640 10 monclient: init 2024-06-03T09:07:02.110 INFO:teuthology.orchestra.run.smithi088.stderr:2024-06-03T09:07:02.096+0000 7f8a01d34640 10 monclient: _reopen_session rank -1 2024-06-03T09:07:02.110 INFO:teuthology.orchestra.run.smithi088.stderr:2024-06-03T09:07:02.096+0000 7f8a01d34640 10 monclient: _add_conns ranks=[1,0,2] 2024-06-03T09:07:02.110 INFO:teuthology.orchestra.run.smithi088.stderr:2024-06-03T09:07:02.096+0000 7f8a01d34640 10 monclient(hunting): picked mon.noname-c con 0x7f89fc0adb90 addr v1:172.21.15.192:6789/0 2024-06-03T09:07:02.110 INFO:teuthology.orchestra.run.smithi088.stderr:2024-06-03T09:07:02.096+0000 7f8a01d34640 10 monclient(hunting): picked mon.noname-a con 0x7f89fc06f020 addr v1:172.21.15.88:6789/0 2024-06-03T09:07:02.110 INFO:teuthology.orchestra.run.smithi088.stderr:2024-06-03T09:07:02.096+0000 7f8a01d34640 10 monclient(hunting): picked mon.noname-b con 0x7f89fc06fc80 addr v1:172.21.15.88:6790/0 2024-06-03T09:07:02.110 INFO:teuthology.orchestra.run.smithi088.stderr:2024-06-03T09:07:02.096+0000 7f8a01d34640 1 -- --> v1:172.21.15.88:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f89fc0716a0 con 0x7f89fc06f020 2024-06-03T09:07:02.110 INFO:teuthology.orchestra.run.smithi088.stderr:2024-06-03T09:07:02.096+0000 7f8a01d34640 1 -- --> v1:172.21.15.192:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f89fc0af7d0 con 0x7f89fc0adb90 2024-06-03T09:07:02.111 INFO:teuthology.orchestra.run.smithi088.stderr:2024-06-03T09:07:02.096+0000 7f8a01d34640 1 -- --> v1:172.21.15.88:6790/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f89fc13ba00 con 0x7f89fc06fc80 2024-06-03T09:07:02.111 INFO:teuthology.orchestra.run.smithi088.stderr:2024-06-03T09:07:02.096+0000 7f8a01d34640 10 monclient(hunting): _renew_subs 2024-06-03T09:07:02.111 INFO:teuthology.orchestra.run.smithi088.stderr:2024-06-03T09:07:02.096+0000 7f8a01d34640 10 monclient(hunting): authenticate will time out at 3077.906700s 2024-06-03T09:07:02.111 INFO:teuthology.orchestra.run.smithi088.stderr:2024-06-03T09:07:02.097+0000 7f89fbfff640 1 --1- >> v1:172.21.15.88:6790/0 conn(0x7f89fc06fc80 0x7f89fc13b2f0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.88:6790/0 says I am v1:172.21.15.88:51196/0 (socket says 172.21.15.88:51196) 2024-06-03T09:07:02.111 INFO:teuthology.orchestra.run.smithi088.stderr:2024-06-03T09:07:02.097+0000 7f89fbfff640 1 -- 172.21.15.88:0/974650501 learned_addr learned my addr 172.21.15.88:0/974650501 (peer_addr_for_me v1:172.21.15.88:0/0) 2024-06-03T09:07:02.111 INFO:teuthology.orchestra.run.smithi088.stderr:2024-06-03T09:07:02.099+0000 7f89fa7fc640 1 -- 172.21.15.88:0/974650501 <== mon.1 v1:172.21.15.192:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 301843214 0 0) 0x7f89fc0af7d0 con 0x7f89fc0adb90 2024-06-03T09:07:02.111 INFO:teuthology.orchestra.run.smithi088.stderr:2024-06-03T09:07:02.099+0000 7f89fa7fc640 10 monclient(hunting): _negotiate 2024-06-03T09:07:02.111 INFO:teuthology.orchestra.run.smithi088.stderr:2024-06-03T09:07:02.099+0000 7f89fa7fc640 10 monclient(hunting): _init_auth method 2 2024-06-03T09:07:02.111 INFO:teuthology.orchestra.run.smithi088.stderr:2024-06-03T09:07:02.099+0000 7f89fa7fc640 10 monclient(hunting): _init_auth creating new auth 2024-06-03T09:07:02.111 INFO:teuthology.orchestra.run.smithi088.stderr:2024-06-03T09:07:02.099+0000 7f89fa7fc640 10 monclient(hunting): my global_id is 30394 2024-06-03T09:07:02.111 INFO:teuthology.orchestra.run.smithi088.stderr:2024-06-03T09:07:02.099+0000 7f89fa7fc640 1 -- 172.21.15.88:0/974650501 --> v1:172.21.15.192:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f89e4003520 con 0x7f89fc0adb90 2024-06-03T09:07:02.111 INFO:teuthology.orchestra.run.smithi088.stderr:2024-06-03T09:07:02.099+0000 7f89fa7fc640 1 -- 172.21.15.88:0/974650501 <== mon.2 v1:172.21.15.88:6790/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 2468954174 0 0) 0x7f89fc13ba00 con 0x7f89fc06fc80 2024-06-03T09:07:02.111 INFO:teuthology.orchestra.run.smithi088.stderr:2024-06-03T09:07:02.099+0000 7f89fa7fc640 10 monclient(hunting): _negotiate 2024-06-03T09:07:02.112 INFO:teuthology.orchestra.run.smithi088.stderr:2024-06-03T09:07:02.099+0000 7f89fa7fc640 10 monclient(hunting): _init_auth method 2 2024-06-03T09:07:02.112 INFO:teuthology.orchestra.run.smithi088.stderr:2024-06-03T09:07:02.099+0000 7f89fa7fc640 10 monclient(hunting): _init_auth creating new auth 2024-06-03T09:07:02.112 INFO:teuthology.orchestra.run.smithi088.stderr:2024-06-03T09:07:02.099+0000 7f89fa7fc640 10 monclient(hunting): my global_id is 30329 2024-06-03T09:07:02.112 INFO:teuthology.orchestra.run.smithi088.stderr:2024-06-03T09:07:02.099+0000 7f89fa7fc640 1 -- 172.21.15.88:0/974650501 --> v1:172.21.15.88:6790/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f89fc0af7d0 con 0x7f89fc06fc80 2024-06-03T09:07:02.112 INFO:teuthology.orchestra.run.smithi088.stderr:2024-06-03T09:07:02.099+0000 7f89fa7fc640 1 -- 172.21.15.88:0/974650501 <== mon.0 v1:172.21.15.88:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 1797561533 0 0) 0x7f89fc0716a0 con 0x7f89fc06f020 2024-06-03T09:07:02.112 INFO:teuthology.orchestra.run.smithi088.stderr:2024-06-03T09:07:02.099+0000 7f89fa7fc640 10 monclient(hunting): _negotiate 2024-06-03T09:07:02.112 INFO:teuthology.orchestra.run.smithi088.stderr:2024-06-03T09:07:02.099+0000 7f89fa7fc640 10 monclient(hunting): _init_auth method 2 2024-06-03T09:07:02.112 INFO:teuthology.orchestra.run.smithi088.stderr:2024-06-03T09:07:02.099+0000 7f89fa7fc640 10 monclient(hunting): _init_auth creating new auth 2024-06-03T09:07:02.112 INFO:teuthology.orchestra.run.smithi088.stderr:2024-06-03T09:07:02.099+0000 7f89fa7fc640 10 monclient(hunting): my global_id is 30819 2024-06-03T09:07:02.112 INFO:teuthology.orchestra.run.smithi088.stderr:2024-06-03T09:07:02.099+0000 7f89fa7fc640 1 -- 172.21.15.88:0/974650501 --> v1:172.21.15.88:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f89fc13ba00 con 0x7f89fc06f020 2024-06-03T09:07:02.112 INFO:teuthology.orchestra.run.smithi088.stderr:2024-06-03T09:07:02.102+0000 7f89fa7fc640 1 -- 172.21.15.88:0/974650501 <== mon.1 v1:172.21.15.192:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 214+0+0 (unknown 3704248712 0 0) 0x7f89e4003520 con 0x7f89fc0adb90 2024-06-03T09:07:02.112 INFO:teuthology.orchestra.run.smithi088.stderr:2024-06-03T09:07:02.102+0000 7f89fa7fc640 1 -- 172.21.15.88:0/974650501 >> v1:172.21.15.88:6790/0 conn(0x7f89fc06fc80 legacy=0x7f89fc13b2f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-06-03T09:07:02.112 INFO:teuthology.orchestra.run.smithi088.stderr:2024-06-03T09:07:02.102+0000 7f89fa7fc640 1 -- 172.21.15.88:0/974650501 >> v1:172.21.15.88:6789/0 conn(0x7f89fc06f020 legacy=0x7f89fc141730 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-06-03T09:07:02.112 INFO:teuthology.orchestra.run.smithi088.stderr:2024-06-03T09:07:02.102+0000 7f89fa7fc640 10 monclient: _finish_hunting 0 2024-06-03T09:07:02.113 INFO:teuthology.orchestra.run.smithi088.stderr:2024-06-03T09:07:02.102+0000 7f89fa7fc640 1 monclient: found mon.noname-c 2024-06-03T09:07:02.113 INFO:teuthology.orchestra.run.smithi088.stderr:2024-06-03T09:07:02.102+0000 7f89fa7fc640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-06-03T09:07:02.113 INFO:teuthology.orchestra.run.smithi088.stderr:2024-06-03T09:07:02.102+0000 7f89fa7fc640 10 monclient: _send_mon_message to mon.noname-c at v1:172.21.15.192:6789/0 2024-06-03T09:07:02.113 INFO:teuthology.orchestra.run.smithi088.stderr:2024-06-03T09:07:02.102+0000 7f89fa7fc640 1 -- 172.21.15.88:0/974650501 --> v1:172.21.15.192:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f89fc13bb90 con 0x7f89fc0adb90 2024-06-03T09:07:02.113 INFO:teuthology.orchestra.run.smithi088.stderr:2024-06-03T09:07:02.102+0000 7f89fa7fc640 10 monclient: _finish_auth 0 2024-06-03T09:07:02.113 INFO:teuthology.orchestra.run.smithi088.stderr:2024-06-03T09:07:02.102+0000 7f89fa7fc640 10 monclient: _check_auth_tickets 2024-06-03T09:07:02.113 INFO:teuthology.orchestra.run.smithi088.stderr:2024-06-03T09:07:02.102+0000 7f89fa7fc640 20 monclient: _check_auth_rotating not needed by client.admin 2024-06-03T09:07:02.113 INFO:teuthology.orchestra.run.smithi088.stderr:2024-06-03T09:07:02.102+0000 7f8a01d34640 5 monclient: authenticate success, global_id 30394 2024-06-03T09:07:02.113 INFO:teuthology.orchestra.run.smithi088.stderr:2024-06-03T09:07:02.102+0000 7f8a01d34640 20 monclient: get_monmap_and_config waiting for monmap|config 2024-06-03T09:07:02.113 INFO:teuthology.orchestra.run.smithi088.stderr:2024-06-03T09:07:02.103+0000 7f89fa7fc640 1 -- 172.21.15.88:0/974650501 <== mon.1 v1:172.21.15.192:6789/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 2234374065 0 0) 0x7f89ec001610 con 0x7f89fc0adb90 2024-06-03T09:07:02.113 INFO:teuthology.orchestra.run.smithi088.stderr:2024-06-03T09:07:02.103+0000 7f89fa7fc640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-06-03T09:07:02.113 INFO:teuthology.orchestra.run.smithi088.stderr:2024-06-03T09:07:02.103+0000 7f89fa7fc640 10 monclient: got monmap 1 from mon.noname-c (according to old e1) 2024-06-03T09:07:02.113 INFO:teuthology.orchestra.run.smithi088.stderr:2024-06-03T09:07:02.103+0000 7f89fa7fc640 10 monclient: dump: 2024-06-03T09:07:02.114 INFO:teuthology.orchestra.run.smithi088.stderr:epoch 1 2024-06-03T09:07:02.114 INFO:teuthology.orchestra.run.smithi088.stderr:fsid ec3c3fdd-3ac6-4d7d-9412-683130ed8b04 2024-06-03T09:07:02.114 INFO:teuthology.orchestra.run.smithi088.stderr:last_changed 2024-06-03T08:34:31.885125+0000 2024-06-03T09:07:02.114 INFO:teuthology.orchestra.run.smithi088.stderr:created 2024-06-03T08:34:31.885125+0000 2024-06-03T09:07:02.114 INFO:teuthology.orchestra.run.smithi088.stderr:min_mon_release 18 (reef) 2024-06-03T09:07:02.114 INFO:teuthology.orchestra.run.smithi088.stderr:election_strategy: 3 2024-06-03T09:07:02.114 INFO:teuthology.orchestra.run.smithi088.stderr:0: v1:172.21.15.88:6789/0 mon.a 2024-06-03T09:07:02.114 INFO:teuthology.orchestra.run.smithi088.stderr:1: v1:172.21.15.192:6789/0 mon.b 2024-06-03T09:07:02.114 INFO:teuthology.orchestra.run.smithi088.stderr:2: v1:172.21.15.88:6790/0 mon.c 2024-06-03T09:07:02.114 INFO:teuthology.orchestra.run.smithi088.stderr: 2024-06-03T09:07:02.114 INFO:teuthology.orchestra.run.smithi088.stderr:2024-06-03T09:07:02.103+0000 7f8a01d34640 20 monclient: get_monmap_and_config waiting for monmap|config 2024-06-03T09:07:02.114 INFO:teuthology.orchestra.run.smithi088.stderr:2024-06-03T09:07:02.104+0000 7f89fa7fc640 1 -- 172.21.15.88:0/974650501 <== mon.1 v1:172.21.15.192:6789/0 4 ==== config(0 keys) v1 ==== 4+0+0 (unknown 0 0 0) 0x7f89ec001860 con 0x7f89fc0adb90 2024-06-03T09:07:02.114 INFO:teuthology.orchestra.run.smithi088.stderr:2024-06-03T09:07:02.104+0000 7f89fa7fc640 10 monclient: handle_config config(0 keys) v1 2024-06-03T09:07:02.115 INFO:teuthology.orchestra.run.smithi088.stderr:2024-06-03T09:07:02.104+0000 7f89fa7fc640 1 -- 172.21.15.88:0/974650501 <== mon.1 v1:172.21.15.192:6789/0 5 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 2234374065 0 0) 0x7f89ec001bf0 con 0x7f89fc0adb90 2024-06-03T09:07:02.115 INFO:teuthology.orchestra.run.smithi088.stderr:2024-06-03T09:07:02.104+0000 7f8a01d34640 10 monclient: get_monmap_and_config success 2024-06-03T09:07:02.115 INFO:teuthology.orchestra.run.smithi088.stderr:2024-06-03T09:07:02.104+0000 7f8a01d34640 4 set_mon_vals no callback set 2024-06-03T09:07:02.115 INFO:teuthology.orchestra.run.smithi088.stderr:2024-06-03T09:07:02.104+0000 7f89fa7fc640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-06-03T09:07:02.115 INFO:teuthology.orchestra.run.smithi088.stderr:2024-06-03T09:07:02.105+0000 7f89fa7fc640 10 monclient: got monmap 1 from mon.b (according to old e1) 2024-06-03T09:07:02.115 INFO:teuthology.orchestra.run.smithi088.stderr:2024-06-03T09:07:02.105+0000 7f89fa7fc640 10 monclient: dump: 2024-06-03T09:07:02.115 INFO:teuthology.orchestra.run.smithi088.stderr:epoch 1 2024-06-03T09:07:02.115 INFO:teuthology.orchestra.run.smithi088.stderr:fsid ec3c3fdd-3ac6-4d7d-9412-683130ed8b04 2024-06-03T09:07:02.115 INFO:teuthology.orchestra.run.smithi088.stderr:last_changed 2024-06-03T08:34:31.885125+0000 2024-06-03T09:07:02.115 INFO:teuthology.orchestra.run.smithi088.stderr:created 2024-06-03T08:34:31.885125+0000 2024-06-03T09:07:02.115 INFO:teuthology.orchestra.run.smithi088.stderr:min_mon_release 18 (reef) 2024-06-03T09:07:02.115 INFO:teuthology.orchestra.run.smithi088.stderr:election_strategy: 3 2024-06-03T09:07:02.115 INFO:teuthology.orchestra.run.smithi088.stderr:0: v1:172.21.15.88:6789/0 mon.a 2024-06-03T09:07:02.115 INFO:teuthology.orchestra.run.smithi088.stderr:1: v1:172.21.15.192:6789/0 mon.b 2024-06-03T09:07:02.116 INFO:teuthology.orchestra.run.smithi088.stderr:2: v1:172.21.15.88:6790/0 mon.c 2024-06-03T09:07:02.116 INFO:teuthology.orchestra.run.smithi088.stderr: 2024-06-03T09:07:02.116 INFO:teuthology.orchestra.run.smithi088.stderr:2024-06-03T09:07:02.105+0000 7f8a01d34640 10 monclient: shutdown 2024-06-03T09:07:02.116 INFO:teuthology.orchestra.run.smithi088.stderr:2024-06-03T09:07:02.105+0000 7f8a01d34640 20 monclient: shutdown discarding 0 pending message(s) 2024-06-03T09:07:02.116 INFO:teuthology.orchestra.run.smithi088.stderr:2024-06-03T09:07:02.105+0000 7f8a01d34640 1 -- 172.21.15.88:0/974650501 >> v1:172.21.15.192:6789/0 conn(0x7f89fc0adb90 legacy=0x7f89fc071b30 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-06-03T09:07:02.116 INFO:teuthology.orchestra.run.smithi088.stderr:2024-06-03T09:07:02.105+0000 7f8a01d34640 1 -- 172.21.15.88:0/974650501 shutdown_connections 2024-06-03T09:07:02.116 INFO:teuthology.orchestra.run.smithi088.stderr:2024-06-03T09:07:02.105+0000 7f8a01d34640 1 -- 172.21.15.88:0/974650501 >> 172.21.15.88:0/974650501 conn(0x7f89fc0667e0 msgr2=0x7f89fc0b2f50 unknown :-1 s=STATE_NONE l=0).mark_down 2024-06-03T09:07:02.116 INFO:teuthology.orchestra.run.smithi088.stderr:2024-06-03T09:07:02.105+0000 7f8a01d34640 1 -- 172.21.15.88:0/974650501 shutdown_connections 2024-06-03T09:07:02.116 INFO:teuthology.orchestra.run.smithi088.stderr:2024-06-03T09:07:02.105+0000 7f8a01d34640 1 -- 172.21.15.88:0/974650501 wait complete. 2024-06-03T09:07:02.116 INFO:teuthology.orchestra.run.smithi088.stderr:2024-06-03T09:07:02.106+0000 7f8a01d34640 10 monclient: build_initial_monmap 2024-06-03T09:07:02.116 INFO:teuthology.orchestra.run.smithi088.stderr:2024-06-03T09:07:02.106+0000 7f8a01d34640 10 monclient: monmap: 2024-06-03T09:07:02.116 INFO:teuthology.orchestra.run.smithi088.stderr:epoch 0 2024-06-03T09:07:02.117 INFO:teuthology.orchestra.run.smithi088.stderr:fsid 00000000-0000-0000-0000-000000000000 2024-06-03T09:07:02.117 INFO:teuthology.orchestra.run.smithi088.stderr:last_changed 0.000000 2024-06-03T09:07:02.117 INFO:teuthology.orchestra.run.smithi088.stderr:created 0.000000 2024-06-03T09:07:02.117 INFO:teuthology.orchestra.run.smithi088.stderr:min_mon_release 0 (unknown) 2024-06-03T09:07:02.117 INFO:teuthology.orchestra.run.smithi088.stderr:election_strategy: 1 2024-06-03T09:07:02.117 INFO:teuthology.orchestra.run.smithi088.stderr:0: v1:172.21.15.88:6789/0 mon.noname-a 2024-06-03T09:07:02.117 INFO:teuthology.orchestra.run.smithi088.stderr:1: v1:172.21.15.192:6789/0 mon.noname-b 2024-06-03T09:07:02.117 INFO:teuthology.orchestra.run.smithi088.stderr:2: v1:172.21.15.88:6790/0 mon.noname-c 2024-06-03T09:07:02.117 INFO:teuthology.orchestra.run.smithi088.stderr: 2024-06-03T09:07:02.117 INFO:teuthology.orchestra.run.smithi088.stderr:2024-06-03T09:07:02.106+0000 7f8a01d34640 1 librados: starting msgr at 2024-06-03T09:07:02.117 INFO:teuthology.orchestra.run.smithi088.stderr:2024-06-03T09:07:02.106+0000 7f8a01d34640 1 librados: starting objecter 2024-06-03T09:07:02.117 INFO:teuthology.orchestra.run.smithi088.stderr:2024-06-03T09:07:02.106+0000 7f8a01d34640 1 Processor -- start 2024-06-03T09:07:02.117 INFO:teuthology.orchestra.run.smithi088.stderr:2024-06-03T09:07:02.107+0000 7f8a01d34640 1 -- start start 2024-06-03T09:07:02.117 INFO:teuthology.orchestra.run.smithi088.stderr:2024-06-03T09:07:02.107+0000 7f8a01d34640 1 librados: setting wanted keys 2024-06-03T09:07:02.118 INFO:teuthology.orchestra.run.smithi088.stderr:2024-06-03T09:07:02.107+0000 7f8a01d34640 1 librados: calling monclient init 2024-06-03T09:07:02.118 INFO:teuthology.orchestra.run.smithi088.stderr:2024-06-03T09:07:02.107+0000 7f8a01d34640 10 monclient: init 2024-06-03T09:07:02.118 INFO:teuthology.orchestra.run.smithi088.stderr:2024-06-03T09:07:02.107+0000 7f8a01d34640 10 monclient: _reopen_session rank -1 2024-06-03T09:07:02.118 INFO:teuthology.orchestra.run.smithi088.stderr:2024-06-03T09:07:02.107+0000 7f8a01d34640 10 monclient: _add_conns ranks=[0,1,2] 2024-06-03T09:07:02.118 INFO:teuthology.orchestra.run.smithi088.stderr:2024-06-03T09:07:02.107+0000 7f8a01d34640 10 monclient(hunting): picked mon.noname-a con 0x7f89fc06f020 addr v1:172.21.15.88:6789/0 2024-06-03T09:07:02.118 INFO:teuthology.orchestra.run.smithi088.stderr:2024-06-03T09:07:02.107+0000 7f8a01d34640 10 monclient(hunting): picked mon.noname-b con 0x7f89fc06fc80 addr v1:172.21.15.192:6789/0 2024-06-03T09:07:02.118 INFO:teuthology.orchestra.run.smithi088.stderr:2024-06-03T09:07:02.107+0000 7f8a01d34640 10 monclient(hunting): picked mon.noname-c con 0x7f89fc0adb90 addr v1:172.21.15.88:6790/0 2024-06-03T09:07:02.118 INFO:teuthology.orchestra.run.smithi088.stderr:2024-06-03T09:07:02.107+0000 7f8a01d34640 1 -- --> v1:172.21.15.88:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f89fc0aa9b0 con 0x7f89fc06f020 2024-06-03T09:07:02.118 INFO:teuthology.orchestra.run.smithi088.stderr:2024-06-03T09:07:02.107+0000 7f8a01d34640 1 -- --> v1:172.21.15.192:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f89fc0aa400 con 0x7f89fc06fc80 2024-06-03T09:07:02.118 INFO:teuthology.orchestra.run.smithi088.stderr:2024-06-03T09:07:02.107+0000 7f8a01d34640 1 -- --> v1:172.21.15.88:6790/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f89fc0aa590 con 0x7f89fc0adb90 2024-06-03T09:07:02.118 INFO:teuthology.orchestra.run.smithi088.stderr:2024-06-03T09:07:02.107+0000 7f8a01d34640 10 monclient(hunting): _renew_subs 2024-06-03T09:07:02.118 INFO:teuthology.orchestra.run.smithi088.stderr:2024-06-03T09:07:02.107+0000 7f8a01d34640 10 monclient(hunting): authenticate will time out at 3077.917528s 2024-06-03T09:07:02.118 INFO:teuthology.orchestra.run.smithi088.stderr:2024-06-03T09:07:02.107+0000 7f89fbfff640 1 --1- >> v1:172.21.15.88:6790/0 conn(0x7f89fc0adb90 0x7f89fc0aca60 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=1).handle_server_banner_and_identify peer v1:172.21.15.88:6790/0 says I am v1:172.21.15.88:51208/0 (socket says 172.21.15.88:51208) 2024-06-03T09:07:02.118 INFO:teuthology.orchestra.run.smithi088.stderr:2024-06-03T09:07:02.107+0000 7f89fbfff640 1 -- 172.21.15.88:0/27458526 learned_addr learned my addr 172.21.15.88:0/27458526 (peer_addr_for_me v1:172.21.15.88:0/0) 2024-06-03T09:07:02.119 INFO:teuthology.orchestra.run.smithi088.stderr:2024-06-03T09:07:02.108+0000 7f89e3fff640 10 client.?.objecter ms_handle_connect 0x7f89fc0adb90 2024-06-03T09:07:02.119 INFO:teuthology.orchestra.run.smithi088.stderr:2024-06-03T09:07:02.108+0000 7f89e3fff640 10 client.?.objecter resend_mon_ops 2024-06-03T09:07:02.119 INFO:teuthology.orchestra.run.smithi088.stderr:2024-06-03T09:07:02.108+0000 7f89e3fff640 10 client.?.objecter ms_handle_connect 0x7f89fc06f020 2024-06-03T09:07:02.119 INFO:teuthology.orchestra.run.smithi088.stderr:2024-06-03T09:07:02.108+0000 7f89e3fff640 10 client.?.objecter resend_mon_ops 2024-06-03T09:07:02.119 INFO:teuthology.orchestra.run.smithi088.stderr:2024-06-03T09:07:02.108+0000 7f89e3fff640 10 client.?.objecter ms_handle_connect 0x7f89fc06fc80 2024-06-03T09:07:02.119 INFO:teuthology.orchestra.run.smithi088.stderr:2024-06-03T09:07:02.108+0000 7f89e3fff640 10 client.?.objecter resend_mon_ops 2024-06-03T09:07:02.121 INFO:teuthology.orchestra.run.smithi088.stderr:2024-06-03T09:07:02.110+0000 7f89e3fff640 1 -- 172.21.15.88:0/27458526 <== mon.2 v1:172.21.15.88:6790/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 1615164395 0 0) 0x7f89fc0aa590 con 0x7f89fc0adb90 2024-06-03T09:07:02.122 INFO:teuthology.orchestra.run.smithi088.stderr:2024-06-03T09:07:02.110+0000 7f89e3fff640 10 monclient(hunting): _negotiate 2024-06-03T09:07:02.122 INFO:teuthology.orchestra.run.smithi088.stderr:2024-06-03T09:07:02.110+0000 7f89e3fff640 10 monclient(hunting): _init_auth method 2 2024-06-03T09:07:02.122 INFO:teuthology.orchestra.run.smithi088.stderr:2024-06-03T09:07:02.110+0000 7f89e3fff640 10 monclient(hunting): _init_auth creating new auth 2024-06-03T09:07:02.122 INFO:teuthology.orchestra.run.smithi088.stderr:2024-06-03T09:07:02.110+0000 7f89e3fff640 10 monclient(hunting): my global_id is 30332 2024-06-03T09:07:02.122 INFO:teuthology.orchestra.run.smithi088.stderr:2024-06-03T09:07:02.110+0000 7f89e3fff640 1 -- 172.21.15.88:0/27458526 --> v1:172.21.15.88:6790/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f89e8004eb0 con 0x7f89fc0adb90 2024-06-03T09:07:02.122 INFO:teuthology.orchestra.run.smithi088.stderr:2024-06-03T09:07:02.110+0000 7f89e3fff640 1 -- 172.21.15.88:0/27458526 <== mon.1 v1:172.21.15.192:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 1599398606 0 0) 0x7f89fc0aa400 con 0x7f89fc06fc80 2024-06-03T09:07:02.122 INFO:teuthology.orchestra.run.smithi088.stderr:2024-06-03T09:07:02.110+0000 7f89e3fff640 10 monclient(hunting): _negotiate 2024-06-03T09:07:02.122 INFO:teuthology.orchestra.run.smithi088.stderr:2024-06-03T09:07:02.110+0000 7f89e3fff640 10 monclient(hunting): _init_auth method 2 2024-06-03T09:07:02.122 INFO:teuthology.orchestra.run.smithi088.stderr:2024-06-03T09:07:02.110+0000 7f89e3fff640 10 monclient(hunting): _init_auth creating new auth 2024-06-03T09:07:02.122 INFO:teuthology.orchestra.run.smithi088.stderr:2024-06-03T09:07:02.110+0000 7f89e3fff640 10 monclient(hunting): my global_id is 30397 2024-06-03T09:07:02.122 INFO:teuthology.orchestra.run.smithi088.stderr:2024-06-03T09:07:02.110+0000 7f89e3fff640 1 -- 172.21.15.88:0/27458526 --> v1:172.21.15.192:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f89fc0aa590 con 0x7f89fc06fc80 2024-06-03T09:07:02.122 INFO:teuthology.orchestra.run.smithi088.stderr:2024-06-03T09:07:02.110+0000 7f89e3fff640 1 -- 172.21.15.88:0/27458526 <== mon.0 v1:172.21.15.88:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 69630869 0 0) 0x7f89fc0aa9b0 con 0x7f89fc06f020 2024-06-03T09:07:02.123 INFO:teuthology.orchestra.run.smithi088.stderr:2024-06-03T09:07:02.110+0000 7f89e3fff640 10 monclient(hunting): _negotiate 2024-06-03T09:07:02.123 INFO:teuthology.orchestra.run.smithi088.stderr:2024-06-03T09:07:02.110+0000 7f89e3fff640 10 monclient(hunting): _init_auth method 2 2024-06-03T09:07:02.123 INFO:teuthology.orchestra.run.smithi088.stderr:2024-06-03T09:07:02.110+0000 7f89e3fff640 10 monclient(hunting): _init_auth creating new auth 2024-06-03T09:07:02.123 INFO:teuthology.orchestra.run.smithi088.stderr:2024-06-03T09:07:02.110+0000 7f89e3fff640 10 monclient(hunting): my global_id is 30822 2024-06-03T09:07:02.123 INFO:teuthology.orchestra.run.smithi088.stderr:2024-06-03T09:07:02.110+0000 7f89e3fff640 1 -- 172.21.15.88:0/27458526 --> v1:172.21.15.88:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f89fc0aa400 con 0x7f89fc06f020 2024-06-03T09:07:02.123 INFO:teuthology.orchestra.run.smithi088.stderr:2024-06-03T09:07:02.115+0000 7f89e3fff640 1 -- 172.21.15.88:0/27458526 <== mon.1 v1:172.21.15.192:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 764+0+0 (unknown 2816277996 0 0) 0x7f89fc0aa590 con 0x7f89fc06fc80 2024-06-03T09:07:02.123 INFO:teuthology.orchestra.run.smithi088.stderr:2024-06-03T09:07:02.115+0000 7f89e3fff640 1 -- 172.21.15.88:0/27458526 --> v1:172.21.15.192:6789/0 -- auth(proto 2 165 bytes epoch 0) v1 -- 0x7f89fc0aa9b0 con 0x7f89fc06fc80 2024-06-03T09:07:02.123 INFO:teuthology.orchestra.run.smithi088.stderr:2024-06-03T09:07:02.115+0000 7f89e3fff640 1 -- 172.21.15.88:0/27458526 <== mon.1 v1:172.21.15.192:6789/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 2234374065 0 0) 0x7f89f00018b0 con 0x7f89fc06fc80 2024-06-03T09:07:02.123 INFO:teuthology.orchestra.run.smithi088.stderr:2024-06-03T09:07:02.115+0000 7f89e3fff640 10 monclient(hunting): handle_monmap mon_map magic: 0 v1 2024-06-03T09:07:02.123 INFO:teuthology.orchestra.run.smithi088.stderr:2024-06-03T09:07:02.115+0000 7f89e3fff640 10 monclient(hunting): got monmap 1 from mon.noname-b (according to old e1) 2024-06-03T09:07:02.123 INFO:teuthology.orchestra.run.smithi088.stderr:2024-06-03T09:07:02.115+0000 7f89e3fff640 10 monclient(hunting): dump: 2024-06-03T09:07:02.123 INFO:teuthology.orchestra.run.smithi088.stderr:epoch 1 2024-06-03T09:07:02.123 INFO:teuthology.orchestra.run.smithi088.stderr:fsid ec3c3fdd-3ac6-4d7d-9412-683130ed8b04 2024-06-03T09:07:02.124 INFO:teuthology.orchestra.run.smithi088.stderr:last_changed 2024-06-03T08:34:31.885125+0000 2024-06-03T09:07:02.124 INFO:teuthology.orchestra.run.smithi088.stderr:created 2024-06-03T08:34:31.885125+0000 2024-06-03T09:07:02.124 INFO:teuthology.orchestra.run.smithi088.stderr:min_mon_release 18 (reef) 2024-06-03T09:07:02.124 INFO:teuthology.orchestra.run.smithi088.stderr:election_strategy: 3 2024-06-03T09:07:02.124 INFO:teuthology.orchestra.run.smithi088.stderr:0: v1:172.21.15.88:6789/0 mon.a 2024-06-03T09:07:02.124 INFO:teuthology.orchestra.run.smithi088.stderr:1: v1:172.21.15.192:6789/0 mon.b 2024-06-03T09:07:02.124 INFO:teuthology.orchestra.run.smithi088.stderr:2: v1:172.21.15.88:6790/0 mon.c 2024-06-03T09:07:02.124 INFO:teuthology.orchestra.run.smithi088.stderr: 2024-06-03T09:07:02.124 INFO:teuthology.orchestra.run.smithi088.stderr:2024-06-03T09:07:02.115+0000 7f89e3fff640 10 monclient(hunting): _finish_auth 0 2024-06-03T09:07:02.124 INFO:teuthology.orchestra.run.smithi088.stderr:2024-06-03T09:07:02.115+0000 7f89e3fff640 1 -- 172.21.15.88:0/27458526 <== mon.2 v1:172.21.15.88:6790/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 764+0+0 (unknown 1761082262 0 0) 0x7f89e8004eb0 con 0x7f89fc0adb90 2024-06-03T09:07:02.124 INFO:teuthology.orchestra.run.smithi088.stderr:2024-06-03T09:07:02.115+0000 7f89e3fff640 1 -- 172.21.15.88:0/27458526 --> v1:172.21.15.88:6790/0 -- auth(proto 2 165 bytes epoch 0) v1 -- 0x7f89fc0aa590 con 0x7f89fc0adb90 2024-06-03T09:07:02.124 INFO:teuthology.orchestra.run.smithi088.stderr:2024-06-03T09:07:02.115+0000 7f89e3fff640 1 -- 172.21.15.88:0/27458526 <== mon.2 v1:172.21.15.88:6790/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 2234374065 0 0) 0x7f89f4001bb0 con 0x7f89fc0adb90 2024-06-03T09:07:02.124 INFO:teuthology.orchestra.run.smithi088.stderr:2024-06-03T09:07:02.115+0000 7f89e3fff640 10 monclient(hunting): handle_monmap mon_map magic: 0 v1 2024-06-03T09:07:02.124 INFO:teuthology.orchestra.run.smithi088.stderr:2024-06-03T09:07:02.115+0000 7f89e3fff640 10 monclient(hunting): got monmap 1 from mon.c (according to old e1) 2024-06-03T09:07:02.125 INFO:teuthology.orchestra.run.smithi088.stderr:2024-06-03T09:07:02.115+0000 7f89e3fff640 10 monclient(hunting): dump: 2024-06-03T09:07:02.125 INFO:teuthology.orchestra.run.smithi088.stderr:epoch 1 2024-06-03T09:07:02.125 INFO:teuthology.orchestra.run.smithi088.stderr:fsid ec3c3fdd-3ac6-4d7d-9412-683130ed8b04 2024-06-03T09:07:02.125 INFO:teuthology.orchestra.run.smithi088.stderr:last_changed 2024-06-03T08:34:31.885125+0000 2024-06-03T09:07:02.125 INFO:teuthology.orchestra.run.smithi088.stderr:created 2024-06-03T08:34:31.885125+0000 2024-06-03T09:07:02.125 INFO:teuthology.orchestra.run.smithi088.stderr:min_mon_release 18 (reef) 2024-06-03T09:07:02.125 INFO:teuthology.orchestra.run.smithi088.stderr:election_strategy: 3 2024-06-03T09:07:02.125 INFO:teuthology.orchestra.run.smithi088.stderr:0: v1:172.21.15.88:6789/0 mon.a 2024-06-03T09:07:02.125 INFO:teuthology.orchestra.run.smithi088.stderr:1: v1:172.21.15.192:6789/0 mon.b 2024-06-03T09:07:02.125 INFO:teuthology.orchestra.run.smithi088.stderr:2: v1:172.21.15.88:6790/0 mon.c 2024-06-03T09:07:02.125 INFO:teuthology.orchestra.run.smithi088.stderr: 2024-06-03T09:07:02.125 INFO:teuthology.orchestra.run.smithi088.stderr:2024-06-03T09:07:02.115+0000 7f89e3fff640 1 -- 172.21.15.88:0/27458526 <== mon.0 v1:172.21.15.88:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 764+0+0 (unknown 3201098567 0 0) 0x7f89fc0aa400 con 0x7f89fc06f020 2024-06-03T09:07:02.125 INFO:teuthology.orchestra.run.smithi088.stderr:2024-06-03T09:07:02.115+0000 7f89e3fff640 1 -- 172.21.15.88:0/27458526 --> v1:172.21.15.88:6789/0 -- auth(proto 2 165 bytes epoch 0) v1 -- 0x7f89e8004eb0 con 0x7f89fc06f020 2024-06-03T09:07:02.125 INFO:teuthology.orchestra.run.smithi088.stderr:2024-06-03T09:07:02.116+0000 7f89e3fff640 1 -- 172.21.15.88:0/27458526 <== mon.0 v1:172.21.15.88:6789/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 2234374065 0 0) 0x7f89ec001b70 con 0x7f89fc06f020 2024-06-03T09:07:02.125 INFO:teuthology.orchestra.run.smithi088.stderr:2024-06-03T09:07:02.116+0000 7f89e3fff640 10 monclient(hunting): handle_monmap mon_map magic: 0 v1 2024-06-03T09:07:02.126 INFO:teuthology.orchestra.run.smithi088.stderr:2024-06-03T09:07:02.116+0000 7f89e3fff640 10 monclient(hunting): got monmap 1 from mon.a (according to old e1) 2024-06-03T09:07:02.126 INFO:teuthology.orchestra.run.smithi088.stderr:2024-06-03T09:07:02.116+0000 7f89e3fff640 10 monclient(hunting): dump: 2024-06-03T09:07:02.126 INFO:teuthology.orchestra.run.smithi088.stderr:epoch 1 2024-06-03T09:07:02.126 INFO:teuthology.orchestra.run.smithi088.stderr:fsid ec3c3fdd-3ac6-4d7d-9412-683130ed8b04 2024-06-03T09:07:02.126 INFO:teuthology.orchestra.run.smithi088.stderr:last_changed 2024-06-03T08:34:31.885125+0000 2024-06-03T09:07:02.126 INFO:teuthology.orchestra.run.smithi088.stderr:created 2024-06-03T08:34:31.885125+0000 2024-06-03T09:07:02.126 INFO:teuthology.orchestra.run.smithi088.stderr:min_mon_release 18 (reef) 2024-06-03T09:07:02.126 INFO:teuthology.orchestra.run.smithi088.stderr:election_strategy: 3 2024-06-03T09:07:02.126 INFO:teuthology.orchestra.run.smithi088.stderr:0: v1:172.21.15.88:6789/0 mon.a 2024-06-03T09:07:02.126 INFO:teuthology.orchestra.run.smithi088.stderr:1: v1:172.21.15.192:6789/0 mon.b 2024-06-03T09:07:02.126 INFO:teuthology.orchestra.run.smithi088.stderr:2: v1:172.21.15.88:6790/0 mon.c 2024-06-03T09:07:02.126 INFO:teuthology.orchestra.run.smithi088.stderr: 2024-06-03T09:07:02.126 INFO:teuthology.orchestra.run.smithi088.stderr:2024-06-03T09:07:02.118+0000 7f89e3fff640 1 -- 172.21.15.88:0/27458526 <== mon.2 v1:172.21.15.88:6790/0 4 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 580+0+0 (unknown 3957477800 0 0) 0x7f89fc0aa590 con 0x7f89fc0adb90 2024-06-03T09:07:02.127 INFO:teuthology.orchestra.run.smithi088.stderr:2024-06-03T09:07:02.118+0000 7f89e3fff640 1 -- 172.21.15.88:0/27458526 >> v1:172.21.15.192:6789/0 conn(0x7f89fc06fc80 legacy=0x7f89fc0ac350 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-06-03T09:07:02.127 INFO:teuthology.orchestra.run.smithi088.stderr:2024-06-03T09:07:02.118+0000 7f89e3fff640 1 -- 172.21.15.88:0/27458526 >> v1:172.21.15.88:6789/0 conn(0x7f89fc06f020 legacy=0x7f89fc1370a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-06-03T09:07:02.127 INFO:teuthology.orchestra.run.smithi088.stderr:2024-06-03T09:07:02.118+0000 7f89e3fff640 10 monclient: _finish_hunting 0 2024-06-03T09:07:02.127 INFO:teuthology.orchestra.run.smithi088.stderr:2024-06-03T09:07:02.118+0000 7f89e3fff640 1 monclient: found mon.c 2024-06-03T09:07:02.127 INFO:teuthology.orchestra.run.smithi088.stderr:2024-06-03T09:07:02.118+0000 7f89e3fff640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-06-03T09:07:02.127 INFO:teuthology.orchestra.run.smithi088.stderr:2024-06-03T09:07:02.118+0000 7f89e3fff640 10 monclient: _send_mon_message to mon.c at v1:172.21.15.88:6790/0 2024-06-03T09:07:02.127 INFO:teuthology.orchestra.run.smithi088.stderr:2024-06-03T09:07:02.118+0000 7f89e3fff640 1 -- 172.21.15.88:0/27458526 --> v1:172.21.15.88:6790/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f89fc0aa720 con 0x7f89fc0adb90 2024-06-03T09:07:02.127 INFO:teuthology.orchestra.run.smithi088.stderr:2024-06-03T09:07:02.118+0000 7f89e3fff640 10 monclient: _finish_auth 0 2024-06-03T09:07:02.127 INFO:teuthology.orchestra.run.smithi088.stderr:2024-06-03T09:07:02.118+0000 7f89e3fff640 10 monclient: _check_auth_tickets 2024-06-03T09:07:02.127 INFO:teuthology.orchestra.run.smithi088.stderr:2024-06-03T09:07:02.118+0000 7f89e3fff640 20 monclient: _check_auth_rotating not needed by client.admin 2024-06-03T09:07:02.127 INFO:teuthology.orchestra.run.smithi088.stderr:2024-06-03T09:07:02.118+0000 7f8a01d34640 5 monclient: authenticate success, global_id 30332 2024-06-03T09:07:02.127 INFO:teuthology.orchestra.run.smithi088.stderr:2024-06-03T09:07:02.118+0000 7f8a01d34640 10 monclient: _renew_subs 2024-06-03T09:07:02.127 INFO:teuthology.orchestra.run.smithi088.stderr:2024-06-03T09:07:02.118+0000 7f8a01d34640 10 monclient: _send_mon_message to mon.c at v1:172.21.15.88:6790/0 2024-06-03T09:07:02.127 INFO:teuthology.orchestra.run.smithi088.stderr:2024-06-03T09:07:02.118+0000 7f8a01d34640 1 -- 172.21.15.88:0/27458526 --> v1:172.21.15.88:6790/0 -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f89fc1377b0 con 0x7f89fc0adb90 2024-06-03T09:07:02.128 INFO:teuthology.orchestra.run.smithi088.stderr:2024-06-03T09:07:02.118+0000 7f8a01d34640 10 client.30332.objecter _maybe_request_map subscribing (onetime) to next osd map 2024-06-03T09:07:02.128 INFO:teuthology.orchestra.run.smithi088.stderr:2024-06-03T09:07:02.118+0000 7f8a01d34640 10 monclient: _renew_subs 2024-06-03T09:07:02.128 INFO:teuthology.orchestra.run.smithi088.stderr:2024-06-03T09:07:02.118+0000 7f8a01d34640 10 monclient: _send_mon_message to mon.c at v1:172.21.15.88:6790/0 2024-06-03T09:07:02.128 INFO:teuthology.orchestra.run.smithi088.stderr:2024-06-03T09:07:02.118+0000 7f8a01d34640 1 -- 172.21.15.88:0/27458526 --> v1:172.21.15.88:6790/0 -- mon_subscribe({osdmap=0}) v3 -- 0x7f89fc13df60 con 0x7f89fc0adb90 2024-06-03T09:07:02.128 INFO:teuthology.orchestra.run.smithi088.stderr:2024-06-03T09:07:02.118+0000 7f8a01d34640 1 librados: init done 2024-06-03T09:07:02.128 INFO:teuthology.orchestra.run.smithi088.stderr:2024-06-03T09:07:02.119+0000 7f8a01d34640 10 start_mon_command cmd=[{"prefix": "get_command_descriptions"}] 2024-06-03T09:07:02.128 INFO:teuthology.orchestra.run.smithi088.stderr:2024-06-03T09:07:02.119+0000 7f8a01d34640 10 monclient: _send_command 1 [{"prefix": "get_command_descriptions"}] 2024-06-03T09:07:02.128 INFO:teuthology.orchestra.run.smithi088.stderr:2024-06-03T09:07:02.119+0000 7f8a01d34640 10 monclient: _send_mon_message to mon.c at v1:172.21.15.88:6790/0 2024-06-03T09:07:02.128 INFO:teuthology.orchestra.run.smithi088.stderr:2024-06-03T09:07:02.119+0000 7f8a01d34640 1 -- 172.21.15.88:0/27458526 --> v1:172.21.15.88:6790/0 -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f89fc0aab40 con 0x7f89fc0adb90 2024-06-03T09:07:02.364 INFO:teuthology.orchestra.run.smithi088.stderr:2024-06-03T09:07:02.364+0000 7f89e3fff640 1 -- 172.21.15.88:0/27458526 <== mon.2 v1:172.21.15.88:6790/0 5 ==== config(0 keys) v1 ==== 4+0+0 (unknown 0 0 0) 0x7f89f4002080 con 0x7f89fc0adb90 2024-06-03T09:07:02.364 INFO:teuthology.orchestra.run.smithi088.stderr:2024-06-03T09:07:02.364+0000 7f89e3fff640 10 monclient: handle_config config(0 keys) v1 2024-06-03T09:07:02.364 INFO:teuthology.orchestra.run.smithi088.stderr:2024-06-03T09:07:02.364+0000 7f89e3fff640 1 -- 172.21.15.88:0/27458526 <== mon.2 v1:172.21.15.88:6790/0 6 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 2234374065 0 0) 0x7f89f4004d00 con 0x7f89fc0adb90 2024-06-03T09:07:02.365 INFO:teuthology.orchestra.run.smithi088.stderr:2024-06-03T09:07:02.364+0000 7f89e3fff640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-06-03T09:07:02.365 INFO:teuthology.orchestra.run.smithi088.stderr:2024-06-03T09:07:02.364+0000 7f89e3fff640 10 monclient: got monmap 1 from mon.c (according to old e1) 2024-06-03T09:07:02.365 INFO:teuthology.orchestra.run.smithi088.stderr:2024-06-03T09:07:02.364+0000 7f89e3fff640 10 monclient: dump: 2024-06-03T09:07:02.365 INFO:teuthology.orchestra.run.smithi088.stderr:epoch 1 2024-06-03T09:07:02.365 INFO:teuthology.orchestra.run.smithi088.stderr:fsid ec3c3fdd-3ac6-4d7d-9412-683130ed8b04 2024-06-03T09:07:02.365 INFO:teuthology.orchestra.run.smithi088.stderr:last_changed 2024-06-03T08:34:31.885125+0000 2024-06-03T09:07:02.365 INFO:teuthology.orchestra.run.smithi088.stderr:created 2024-06-03T08:34:31.885125+0000 2024-06-03T09:07:02.365 INFO:teuthology.orchestra.run.smithi088.stderr:min_mon_release 18 (reef) 2024-06-03T09:07:02.365 INFO:teuthology.orchestra.run.smithi088.stderr:election_strategy: 3 2024-06-03T09:07:02.365 INFO:teuthology.orchestra.run.smithi088.stderr:0: v1:172.21.15.88:6789/0 mon.a 2024-06-03T09:07:02.365 INFO:teuthology.orchestra.run.smithi088.stderr:1: v1:172.21.15.192:6789/0 mon.b 2024-06-03T09:07:02.365 INFO:teuthology.orchestra.run.smithi088.stderr:2: v1:172.21.15.88:6790/0 mon.c 2024-06-03T09:07:02.365 INFO:teuthology.orchestra.run.smithi088.stderr: 2024-06-03T09:07:02.365 INFO:teuthology.orchestra.run.smithi088.stderr:2024-06-03T09:07:02.364+0000 7f89f9ffb640 4 set_mon_vals no callback set 2024-06-03T09:07:02.366 INFO:teuthology.orchestra.run.smithi088.stderr:2024-06-03T09:07:02.364+0000 7f89e3fff640 1 -- 172.21.15.88:0/27458526 <== mon.2 v1:172.21.15.88:6790/0 7 ==== mgrmap(e 5) v1 ==== 89110+0+0 (unknown 873533973 0 0) 0x7f89f4004ee0 con 0x7f89fc0adb90 2024-06-03T09:07:02.366 INFO:teuthology.orchestra.run.smithi088.stderr:2024-06-03T09:07:02.365+0000 7f89fb7fe640 10 monclient: get_auth_request con 0x7f89e806d420 auth_method 0 2024-06-03T09:07:02.366 INFO:teuthology.orchestra.run.smithi088.stderr:2024-06-03T09:07:02.365+0000 7f89e3fff640 10 client.30332.objecter ms_handle_connect 0x7f89e806d420 2024-06-03T09:07:02.370 INFO:teuthology.orchestra.run.smithi088.stderr:2024-06-03T09:07:02.370+0000 7f89e3fff640 1 -- 172.21.15.88:0/27458526 <== mon.2 v1:172.21.15.88:6790/0 8 ==== osd_map(1014..1014 src has 622..1014) v4 ==== 5229+0+0 (unknown 3230192161 0 0) 0x7f89f4054350 con 0x7f89fc0adb90 2024-06-03T09:07:02.370 INFO:teuthology.orchestra.run.smithi088.stderr:2024-06-03T09:07:02.370+0000 7f89e3fff640 10 client.30332.objecter ms_dispatch 0x7f89fc000b90 osd_map(1014..1014 src has 622..1014) v4 2024-06-03T09:07:02.371 INFO:teuthology.orchestra.run.smithi088.stderr:2024-06-03T09:07:02.370+0000 7f89e3fff640 3 client.30332.objecter handle_osd_map got epochs [1014,1014] > 0 2024-06-03T09:07:02.371 INFO:teuthology.orchestra.run.smithi088.stderr:2024-06-03T09:07:02.370+0000 7f89e3fff640 3 client.30332.objecter handle_osd_map decoding full epoch 1014 2024-06-03T09:07:02.371 INFO:teuthology.orchestra.run.smithi088.stderr:2024-06-03T09:07:02.370+0000 7f89e3fff640 20 client.30332.objecter dump_active .. 0 homeless 2024-06-03T09:07:02.371 INFO:teuthology.orchestra.run.smithi088.stderr:2024-06-03T09:07:02.370+0000 7f89e3fff640 1 -- 172.21.15.88:0/27458526 <== mon.2 v1:172.21.15.88:6790/0 9 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+182498 (unknown 1092875540 0 3016760700) 0x7f89f404ff00 con 0x7f89fc0adb90 2024-06-03T09:07:02.371 INFO:teuthology.orchestra.run.smithi088.stderr:2024-06-03T09:07:02.370+0000 7f89e3fff640 10 monclient: handle_mon_command_ack 1 [{"prefix": "get_command_descriptions"}] 2024-06-03T09:07:02.371 INFO:teuthology.orchestra.run.smithi088.stderr:2024-06-03T09:07:02.370+0000 7f89e3fff640 10 monclient: _finish_command 1 = system:0 2024-06-03T09:07:02.545 INFO:teuthology.orchestra.run.smithi088.stderr:2024-06-03T09:07:02.545+0000 7f8a01d34640 1 -- 172.21.15.88:0/27458526 --> v1:172.21.15.192:6800/34229 -- mgr_command(tid 0: {"prefix": "osd deep-scrub", "who": "7", "target": ["mon-mgr", ""]}) v1 -- 0x7f89fc0adf70 con 0x7f89e806d420 2024-06-03T09:07:02.546 INFO:teuthology.orchestra.run.smithi088.stderr:2024-06-03T09:07:02.546+0000 7f89e3fff640 1 -- 172.21.15.88:0/27458526 <== mgr.4110 v1:172.21.15.192:6800/34229 1 ==== mgr_command_reply(tid 0: 0 instructed osd(s) 7 to deep-scrub) v1 ==== 41+0+0 (unknown 1239724482 0 0) 0x7f89ec005810 con 0x7f89e806d420 2024-06-03T09:07:02.546 INFO:teuthology.orchestra.run.smithi088.stderr:instructed osd(s) 7 to deep-scrub 2024-06-03T09:07:02.548 INFO:teuthology.orchestra.run.smithi088.stderr:2024-06-03T09:07:02.548+0000 7f8a01d34640 10 librados: watch_flush enter 2024-06-03T09:07:02.548 INFO:teuthology.orchestra.run.smithi088.stderr:2024-06-03T09:07:02.548+0000 7f8a01d34640 10 librados: watch_flush exit 2024-06-03T09:07:02.549 INFO:teuthology.orchestra.run.smithi088.stderr:2024-06-03T09:07:02.548+0000 7f8a01d34640 20 client.30332.objecter shutdown clearing up homeless session... 2024-06-03T09:07:02.549 INFO:teuthology.orchestra.run.smithi088.stderr:2024-06-03T09:07:02.548+0000 7f8a01d34640 10 client.30332.objecter successfully canceled tick 2024-06-03T09:07:02.549 INFO:teuthology.orchestra.run.smithi088.stderr:2024-06-03T09:07:02.548+0000 7f8a01d34640 1 -- 172.21.15.88:0/27458526 >> v1:172.21.15.192:6800/34229 conn(0x7f89e806d420 legacy=0x7f89e806f8e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-06-03T09:07:02.549 INFO:teuthology.orchestra.run.smithi088.stderr:2024-06-03T09:07:02.548+0000 7f8a01d34640 10 monclient: shutdown 2024-06-03T09:07:02.549 INFO:teuthology.orchestra.run.smithi088.stderr:2024-06-03T09:07:02.548+0000 7f8a01d34640 20 monclient: shutdown discarding 0 pending message(s) 2024-06-03T09:07:02.549 INFO:teuthology.orchestra.run.smithi088.stderr:2024-06-03T09:07:02.548+0000 7f8a01d34640 1 -- 172.21.15.88:0/27458526 >> v1:172.21.15.88:6790/0 conn(0x7f89fc0adb90 legacy=0x7f89fc0aca60 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-06-03T09:07:02.549 INFO:teuthology.orchestra.run.smithi088.stderr:2024-06-03T09:07:02.548+0000 7f8a01d34640 1 -- 172.21.15.88:0/27458526 shutdown_connections 2024-06-03T09:07:02.549 INFO:teuthology.orchestra.run.smithi088.stderr:2024-06-03T09:07:02.548+0000 7f8a01d34640 1 -- 172.21.15.88:0/27458526 >> 172.21.15.88:0/27458526 conn(0x7f89fc0667e0 msgr2=0x7f89fc13b0c0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-06-03T09:07:02.549 INFO:teuthology.orchestra.run.smithi088.stderr:2024-06-03T09:07:02.548+0000 7f8a01d34640 1 -- 172.21.15.88:0/27458526 shutdown_connections 2024-06-03T09:07:02.549 INFO:teuthology.orchestra.run.smithi088.stderr:2024-06-03T09:07:02.548+0000 7f8a01d34640 1 -- 172.21.15.88:0/27458526 wait complete. 2024-06-03T09:07:02.549 INFO:teuthology.orchestra.run.smithi088.stderr:2024-06-03T09:07:02.548+0000 7f8a01d34640 1 librados: shutdown 2024-06-03T09:07:02.559 DEBUG:teuthology.orchestra.run.smithi088:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph pg dump --format=json 2024-06-03T09:07:02.677 INFO:teuthology.orchestra.run.smithi088.stderr:2024-06-03T09:07:02.677+0000 7f5a73d34640 10 monclient: get_monmap_and_config 2024-06-03T09:07:02.677 INFO:teuthology.orchestra.run.smithi088.stderr:2024-06-03T09:07:02.677+0000 7f5a73d34640 10 monclient: build_initial_monmap 2024-06-03T09:07:02.678 INFO:teuthology.orchestra.run.smithi088.stderr:2024-06-03T09:07:02.677+0000 7f5a73d34640 10 monclient: monmap: 2024-06-03T09:07:02.678 INFO:teuthology.orchestra.run.smithi088.stderr:epoch 0 2024-06-03T09:07:02.678 INFO:teuthology.orchestra.run.smithi088.stderr:fsid ec3c3fdd-3ac6-4d7d-9412-683130ed8b04 2024-06-03T09:07:02.678 INFO:teuthology.orchestra.run.smithi088.stderr:last_changed 2024-06-03T09:07:02.678621+0000 2024-06-03T09:07:02.678 INFO:teuthology.orchestra.run.smithi088.stderr:created 2024-06-03T09:07:02.678621+0000 2024-06-03T09:07:02.678 INFO:teuthology.orchestra.run.smithi088.stderr:min_mon_release 0 (unknown) 2024-06-03T09:07:02.678 INFO:teuthology.orchestra.run.smithi088.stderr:election_strategy: 3 2024-06-03T09:07:02.678 INFO:teuthology.orchestra.run.smithi088.stderr:0: v1:172.21.15.88:6789/0 mon.noname-a 2024-06-03T09:07:02.678 INFO:teuthology.orchestra.run.smithi088.stderr:1: v1:172.21.15.192:6789/0 mon.noname-c 2024-06-03T09:07:02.678 INFO:teuthology.orchestra.run.smithi088.stderr:2: v1:172.21.15.88:6790/0 mon.noname-b 2024-06-03T09:07:02.678 INFO:teuthology.orchestra.run.smithi088.stderr: 2024-06-03T09:07:02.680 INFO:teuthology.orchestra.run.smithi088.stderr:2024-06-03T09:07:02.678+0000 7f5a73d34640 1 Processor -- start 2024-06-03T09:07:02.680 INFO:teuthology.orchestra.run.smithi088.stderr:2024-06-03T09:07:02.678+0000 7f5a73d34640 1 -- start start 2024-06-03T09:07:02.680 INFO:teuthology.orchestra.run.smithi088.stderr:2024-06-03T09:07:02.678+0000 7f5a73d34640 10 monclient: init 2024-06-03T09:07:02.680 INFO:teuthology.orchestra.run.smithi088.stderr:2024-06-03T09:07:02.678+0000 7f5a73d34640 10 monclient: _reopen_session rank -1 2024-06-03T09:07:02.680 INFO:teuthology.orchestra.run.smithi088.stderr:2024-06-03T09:07:02.679+0000 7f5a73d34640 10 monclient: _add_conns ranks=[2,0,1] 2024-06-03T09:07:02.680 INFO:teuthology.orchestra.run.smithi088.stderr:2024-06-03T09:07:02.679+0000 7f5a73d34640 10 monclient(hunting): picked mon.noname-b con 0x7f5a6c0adb90 addr v1:172.21.15.88:6790/0 2024-06-03T09:07:02.680 INFO:teuthology.orchestra.run.smithi088.stderr:2024-06-03T09:07:02.679+0000 7f5a73d34640 10 monclient(hunting): picked mon.noname-a con 0x7f5a6c06ee00 addr v1:172.21.15.88:6789/0 2024-06-03T09:07:02.681 INFO:teuthology.orchestra.run.smithi088.stderr:2024-06-03T09:07:02.679+0000 7f5a73d34640 10 monclient(hunting): picked mon.noname-c con 0x7f5a6c1383e0 addr v1:172.21.15.192:6789/0 2024-06-03T09:07:02.681 INFO:teuthology.orchestra.run.smithi088.stderr:2024-06-03T09:07:02.679+0000 7f5a73d34640 1 -- --> v1:172.21.15.88:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f5a6c06f2c0 con 0x7f5a6c06ee00 2024-06-03T09:07:02.681 INFO:teuthology.orchestra.run.smithi088.stderr:2024-06-03T09:07:02.679+0000 7f5a73d34640 1 -- --> v1:172.21.15.192:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f5a6c0710c0 con 0x7f5a6c1383e0 2024-06-03T09:07:02.681 INFO:teuthology.orchestra.run.smithi088.stderr:2024-06-03T09:07:02.679+0000 7f5a73d34640 1 -- --> v1:172.21.15.88:6790/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f5a6c071fd0 con 0x7f5a6c0adb90 2024-06-03T09:07:02.681 INFO:teuthology.orchestra.run.smithi088.stderr:2024-06-03T09:07:02.679+0000 7f5a73d34640 10 monclient(hunting): _renew_subs 2024-06-03T09:07:02.681 INFO:teuthology.orchestra.run.smithi088.stderr:2024-06-03T09:07:02.679+0000 7f5a73d34640 10 monclient(hunting): authenticate will time out at 3078.489075s 2024-06-03T09:07:02.681 INFO:teuthology.orchestra.run.smithi088.stderr:2024-06-03T09:07:02.679+0000 7f5a71ad0640 1 --1- >> v1:172.21.15.88:6790/0 conn(0x7f5a6c0adb90 0x7f5a6c0718c0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.88:6790/0 says I am v1:172.21.15.88:51218/0 (socket says 172.21.15.88:51218) 2024-06-03T09:07:02.681 INFO:teuthology.orchestra.run.smithi088.stderr:2024-06-03T09:07:02.679+0000 7f5a71ad0640 1 -- 172.21.15.88:0/2511842687 learned_addr learned my addr 172.21.15.88:0/2511842687 (peer_addr_for_me v1:172.21.15.88:0/0) 2024-06-03T09:07:02.690 INFO:teuthology.orchestra.run.smithi088.stderr:2024-06-03T09:07:02.690+0000 7f5a70ace640 1 -- 172.21.15.88:0/2511842687 <== mon.0 v1:172.21.15.88:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 1466628655 0 0) 0x7f5a6c06f2c0 con 0x7f5a6c06ee00 2024-06-03T09:07:02.690 INFO:teuthology.orchestra.run.smithi088.stderr:2024-06-03T09:07:02.690+0000 7f5a70ace640 10 monclient(hunting): _negotiate 2024-06-03T09:07:02.691 INFO:teuthology.orchestra.run.smithi088.stderr:2024-06-03T09:07:02.690+0000 7f5a70ace640 10 monclient(hunting): _init_auth method 2 2024-06-03T09:07:02.691 INFO:teuthology.orchestra.run.smithi088.stderr:2024-06-03T09:07:02.690+0000 7f5a70ace640 10 monclient(hunting): _init_auth creating new auth 2024-06-03T09:07:02.691 INFO:teuthology.orchestra.run.smithi088.stderr:2024-06-03T09:07:02.690+0000 7f5a70ace640 10 monclient(hunting): my global_id is 30825 2024-06-03T09:07:02.691 INFO:teuthology.orchestra.run.smithi088.stderr:2024-06-03T09:07:02.690+0000 7f5a70ace640 1 -- 172.21.15.88:0/2511842687 --> v1:172.21.15.88:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f5a58003520 con 0x7f5a6c06ee00 2024-06-03T09:07:02.691 INFO:teuthology.orchestra.run.smithi088.stderr:2024-06-03T09:07:02.690+0000 7f5a70ace640 1 -- 172.21.15.88:0/2511842687 <== mon.2 v1:172.21.15.88:6790/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 3868313119 0 0) 0x7f5a6c071fd0 con 0x7f5a6c0adb90 2024-06-03T09:07:02.691 INFO:teuthology.orchestra.run.smithi088.stderr:2024-06-03T09:07:02.690+0000 7f5a70ace640 10 monclient(hunting): _negotiate 2024-06-03T09:07:02.691 INFO:teuthology.orchestra.run.smithi088.stderr:2024-06-03T09:07:02.690+0000 7f5a70ace640 10 monclient(hunting): _init_auth method 2 2024-06-03T09:07:02.691 INFO:teuthology.orchestra.run.smithi088.stderr:2024-06-03T09:07:02.690+0000 7f5a70ace640 10 monclient(hunting): _init_auth creating new auth 2024-06-03T09:07:02.691 INFO:teuthology.orchestra.run.smithi088.stderr:2024-06-03T09:07:02.690+0000 7f5a70ace640 10 monclient(hunting): my global_id is 30335 2024-06-03T09:07:02.691 INFO:teuthology.orchestra.run.smithi088.stderr:2024-06-03T09:07:02.690+0000 7f5a70ace640 1 -- 172.21.15.88:0/2511842687 --> v1:172.21.15.88:6790/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f5a6c06f2c0 con 0x7f5a6c0adb90 2024-06-03T09:07:02.694 INFO:teuthology.orchestra.run.smithi088.stderr:2024-06-03T09:07:02.694+0000 7f5a70ace640 1 -- 172.21.15.88:0/2511842687 <== mon.2 v1:172.21.15.88:6790/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 214+0+0 (unknown 4171637338 0 0) 0x7f5a6c06f2c0 con 0x7f5a6c0adb90 2024-06-03T09:07:02.694 INFO:teuthology.orchestra.run.smithi088.stderr:2024-06-03T09:07:02.694+0000 7f5a70ace640 1 -- 172.21.15.88:0/2511842687 >> v1:172.21.15.192:6789/0 conn(0x7f5a6c1383e0 legacy=0x7f5a6c149af0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-06-03T09:07:02.694 INFO:teuthology.orchestra.run.smithi088.stderr:2024-06-03T09:07:02.694+0000 7f5a70ace640 1 -- 172.21.15.88:0/2511842687 >> v1:172.21.15.88:6789/0 conn(0x7f5a6c06ee00 legacy=0x7f5a6c137cd0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-06-03T09:07:02.695 INFO:teuthology.orchestra.run.smithi088.stderr:2024-06-03T09:07:02.694+0000 7f5a70ace640 10 monclient: _finish_hunting 0 2024-06-03T09:07:02.695 INFO:teuthology.orchestra.run.smithi088.stderr:2024-06-03T09:07:02.694+0000 7f5a70ace640 1 monclient: found mon.noname-b 2024-06-03T09:07:02.695 INFO:teuthology.orchestra.run.smithi088.stderr:2024-06-03T09:07:02.694+0000 7f5a70ace640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-06-03T09:07:02.695 INFO:teuthology.orchestra.run.smithi088.stderr:2024-06-03T09:07:02.694+0000 7f5a70ace640 10 monclient: _send_mon_message to mon.noname-b at v1:172.21.15.88:6790/0 2024-06-03T09:07:02.695 INFO:teuthology.orchestra.run.smithi088.stderr:2024-06-03T09:07:02.694+0000 7f5a70ace640 1 -- 172.21.15.88:0/2511842687 --> v1:172.21.15.88:6790/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f5a6c141750 con 0x7f5a6c0adb90 2024-06-03T09:07:02.695 INFO:teuthology.orchestra.run.smithi088.stderr:2024-06-03T09:07:02.694+0000 7f5a70ace640 10 monclient: _finish_auth 0 2024-06-03T09:07:02.695 INFO:teuthology.orchestra.run.smithi088.stderr:2024-06-03T09:07:02.694+0000 7f5a70ace640 10 monclient: _check_auth_tickets 2024-06-03T09:07:02.695 INFO:teuthology.orchestra.run.smithi088.stderr:2024-06-03T09:07:02.694+0000 7f5a70ace640 20 monclient: _check_auth_rotating not needed by client.admin 2024-06-03T09:07:02.695 INFO:teuthology.orchestra.run.smithi088.stderr:2024-06-03T09:07:02.694+0000 7f5a70ace640 1 -- 172.21.15.88:0/2511842687 <== mon.2 v1:172.21.15.88:6790/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 2234374065 0 0) 0x7f5a680015f0 con 0x7f5a6c0adb90 2024-06-03T09:07:02.695 INFO:teuthology.orchestra.run.smithi088.stderr:2024-06-03T09:07:02.694+0000 7f5a70ace640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-06-03T09:07:02.695 INFO:teuthology.orchestra.run.smithi088.stderr:2024-06-03T09:07:02.694+0000 7f5a70ace640 10 monclient: got monmap 1 from mon.noname-b (according to old e1) 2024-06-03T09:07:02.695 INFO:teuthology.orchestra.run.smithi088.stderr:2024-06-03T09:07:02.694+0000 7f5a70ace640 10 monclient: dump: 2024-06-03T09:07:02.695 INFO:teuthology.orchestra.run.smithi088.stderr:epoch 1 2024-06-03T09:07:02.696 INFO:teuthology.orchestra.run.smithi088.stderr:fsid ec3c3fdd-3ac6-4d7d-9412-683130ed8b04 2024-06-03T09:07:02.696 INFO:teuthology.orchestra.run.smithi088.stderr:last_changed 2024-06-03T08:34:31.885125+0000 2024-06-03T09:07:02.696 INFO:teuthology.orchestra.run.smithi088.stderr:created 2024-06-03T08:34:31.885125+0000 2024-06-03T09:07:02.696 INFO:teuthology.orchestra.run.smithi088.stderr:min_mon_release 18 (reef) 2024-06-03T09:07:02.696 INFO:teuthology.orchestra.run.smithi088.stderr:election_strategy: 3 2024-06-03T09:07:02.696 INFO:teuthology.orchestra.run.smithi088.stderr:0: v1:172.21.15.88:6789/0 mon.a 2024-06-03T09:07:02.696 INFO:teuthology.orchestra.run.smithi088.stderr:1: v1:172.21.15.192:6789/0 mon.b 2024-06-03T09:07:02.696 INFO:teuthology.orchestra.run.smithi088.stderr:2: v1:172.21.15.88:6790/0 mon.c 2024-06-03T09:07:02.696 INFO:teuthology.orchestra.run.smithi088.stderr: 2024-06-03T09:07:02.696 INFO:teuthology.orchestra.run.smithi088.stderr:2024-06-03T09:07:02.694+0000 7f5a73d34640 5 monclient: authenticate success, global_id 30335 2024-06-03T09:07:02.696 INFO:teuthology.orchestra.run.smithi088.stderr:2024-06-03T09:07:02.694+0000 7f5a73d34640 20 monclient: get_monmap_and_config waiting for monmap|config 2024-06-03T09:07:02.697 INFO:teuthology.orchestra.run.smithi088.stderr:2024-06-03T09:07:02.697+0000 7f5a70ace640 1 -- 172.21.15.88:0/2511842687 <== mon.2 v1:172.21.15.88:6790/0 4 ==== config(0 keys) v1 ==== 4+0+0 (unknown 0 0 0) 0x7f5a68001810 con 0x7f5a6c0adb90 2024-06-03T09:07:02.697 INFO:teuthology.orchestra.run.smithi088.stderr:2024-06-03T09:07:02.697+0000 7f5a70ace640 10 monclient: handle_config config(0 keys) v1 2024-06-03T09:07:02.697 INFO:teuthology.orchestra.run.smithi088.stderr:2024-06-03T09:07:02.697+0000 7f5a70ace640 1 -- 172.21.15.88:0/2511842687 <== mon.2 v1:172.21.15.88:6790/0 5 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 2234374065 0 0) 0x7f5a68001ba0 con 0x7f5a6c0adb90 2024-06-03T09:07:02.697 INFO:teuthology.orchestra.run.smithi088.stderr:2024-06-03T09:07:02.697+0000 7f5a70ace640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-06-03T09:07:02.697 INFO:teuthology.orchestra.run.smithi088.stderr:2024-06-03T09:07:02.697+0000 7f5a70ace640 10 monclient: got monmap 1 from mon.c (according to old e1) 2024-06-03T09:07:02.697 INFO:teuthology.orchestra.run.smithi088.stderr:2024-06-03T09:07:02.697+0000 7f5a70ace640 10 monclient: dump: 2024-06-03T09:07:02.697 INFO:teuthology.orchestra.run.smithi088.stderr:epoch 1 2024-06-03T09:07:02.697 INFO:teuthology.orchestra.run.smithi088.stderr:fsid ec3c3fdd-3ac6-4d7d-9412-683130ed8b04 2024-06-03T09:07:02.697 INFO:teuthology.orchestra.run.smithi088.stderr:last_changed 2024-06-03T08:34:31.885125+0000 2024-06-03T09:07:02.697 INFO:teuthology.orchestra.run.smithi088.stderr:created 2024-06-03T08:34:31.885125+0000 2024-06-03T09:07:02.697 INFO:teuthology.orchestra.run.smithi088.stderr:min_mon_release 18 (reef) 2024-06-03T09:07:02.698 INFO:teuthology.orchestra.run.smithi088.stderr:election_strategy: 3 2024-06-03T09:07:02.698 INFO:teuthology.orchestra.run.smithi088.stderr:0: v1:172.21.15.88:6789/0 mon.a 2024-06-03T09:07:02.698 INFO:teuthology.orchestra.run.smithi088.stderr:1: v1:172.21.15.192:6789/0 mon.b 2024-06-03T09:07:02.698 INFO:teuthology.orchestra.run.smithi088.stderr:2: v1:172.21.15.88:6790/0 mon.c 2024-06-03T09:07:02.698 INFO:teuthology.orchestra.run.smithi088.stderr: 2024-06-03T09:07:02.698 INFO:teuthology.orchestra.run.smithi088.stderr:2024-06-03T09:07:02.697+0000 7f5a73d34640 10 monclient: get_monmap_and_config success 2024-06-03T09:07:02.698 INFO:teuthology.orchestra.run.smithi088.stderr:2024-06-03T09:07:02.697+0000 7f5a73d34640 4 set_mon_vals no callback set 2024-06-03T09:07:02.700 INFO:teuthology.orchestra.run.smithi088.stderr:2024-06-03T09:07:02.697+0000 7f5a73d34640 10 monclient: shutdown 2024-06-03T09:07:02.701 INFO:teuthology.orchestra.run.smithi088.stderr:2024-06-03T09:07:02.697+0000 7f5a73d34640 20 monclient: shutdown discarding 0 pending message(s) 2024-06-03T09:07:02.701 INFO:teuthology.orchestra.run.smithi088.stderr:2024-06-03T09:07:02.697+0000 7f5a73d34640 1 -- 172.21.15.88:0/2511842687 >> v1:172.21.15.88:6790/0 conn(0x7f5a6c0adb90 legacy=0x7f5a6c0718c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-06-03T09:07:02.701 INFO:teuthology.orchestra.run.smithi088.stderr:2024-06-03T09:07:02.698+0000 7f5a73d34640 1 -- 172.21.15.88:0/2511842687 shutdown_connections 2024-06-03T09:07:02.701 INFO:teuthology.orchestra.run.smithi088.stderr:2024-06-03T09:07:02.698+0000 7f5a73d34640 1 -- 172.21.15.88:0/2511842687 >> 172.21.15.88:0/2511842687 conn(0x7f5a6c0b2f20 msgr2=0x7f5a6c0b3300 unknown :-1 s=STATE_NONE l=0).mark_down 2024-06-03T09:07:02.701 INFO:teuthology.orchestra.run.smithi088.stderr:2024-06-03T09:07:02.698+0000 7f5a73d34640 1 -- 172.21.15.88:0/2511842687 shutdown_connections 2024-06-03T09:07:02.701 INFO:teuthology.orchestra.run.smithi088.stderr:2024-06-03T09:07:02.698+0000 7f5a73d34640 1 -- 172.21.15.88:0/2511842687 wait complete. 2024-06-03T09:07:02.701 INFO:teuthology.orchestra.run.smithi088.stderr:2024-06-03T09:07:02.699+0000 7f5a73d34640 10 monclient: build_initial_monmap 2024-06-03T09:07:02.701 INFO:teuthology.orchestra.run.smithi088.stderr:2024-06-03T09:07:02.699+0000 7f5a73d34640 10 monclient: monmap: 2024-06-03T09:07:02.701 INFO:teuthology.orchestra.run.smithi088.stderr:epoch 0 2024-06-03T09:07:02.701 INFO:teuthology.orchestra.run.smithi088.stderr:fsid 00000000-0000-0000-0000-000000000000 2024-06-03T09:07:02.701 INFO:teuthology.orchestra.run.smithi088.stderr:last_changed 0.000000 2024-06-03T09:07:02.701 INFO:teuthology.orchestra.run.smithi088.stderr:created 0.000000 2024-06-03T09:07:02.701 INFO:teuthology.orchestra.run.smithi088.stderr:min_mon_release 0 (unknown) 2024-06-03T09:07:02.701 INFO:teuthology.orchestra.run.smithi088.stderr:election_strategy: 1 2024-06-03T09:07:02.702 INFO:teuthology.orchestra.run.smithi088.stderr:0: v1:172.21.15.88:6789/0 mon.noname-a 2024-06-03T09:07:02.702 INFO:teuthology.orchestra.run.smithi088.stderr:1: v1:172.21.15.192:6789/0 mon.noname-b 2024-06-03T09:07:02.702 INFO:teuthology.orchestra.run.smithi088.stderr:2: v1:172.21.15.88:6790/0 mon.noname-c 2024-06-03T09:07:02.702 INFO:teuthology.orchestra.run.smithi088.stderr: 2024-06-03T09:07:02.702 INFO:teuthology.orchestra.run.smithi088.stderr:2024-06-03T09:07:02.699+0000 7f5a73d34640 1 librados: starting msgr at 2024-06-03T09:07:02.702 INFO:teuthology.orchestra.run.smithi088.stderr:2024-06-03T09:07:02.699+0000 7f5a73d34640 1 librados: starting objecter 2024-06-03T09:07:02.702 INFO:teuthology.orchestra.run.smithi088.stderr:2024-06-03T09:07:02.699+0000 7f5a73d34640 1 Processor -- start 2024-06-03T09:07:02.702 INFO:teuthology.orchestra.run.smithi088.stderr:2024-06-03T09:07:02.699+0000 7f5a73d34640 1 -- start start 2024-06-03T09:07:02.702 INFO:teuthology.orchestra.run.smithi088.stderr:2024-06-03T09:07:02.699+0000 7f5a73d34640 1 librados: setting wanted keys 2024-06-03T09:07:02.702 INFO:teuthology.orchestra.run.smithi088.stderr:2024-06-03T09:07:02.699+0000 7f5a73d34640 1 librados: calling monclient init 2024-06-03T09:07:02.702 INFO:teuthology.orchestra.run.smithi088.stderr:2024-06-03T09:07:02.699+0000 7f5a73d34640 10 monclient: init 2024-06-03T09:07:02.702 INFO:teuthology.orchestra.run.smithi088.stderr:2024-06-03T09:07:02.699+0000 7f5a73d34640 10 monclient: _reopen_session rank -1 2024-06-03T09:07:02.702 INFO:teuthology.orchestra.run.smithi088.stderr:2024-06-03T09:07:02.699+0000 7f5a73d34640 10 monclient: _add_conns ranks=[0,1,2] 2024-06-03T09:07:02.703 INFO:teuthology.orchestra.run.smithi088.stderr:2024-06-03T09:07:02.699+0000 7f5a73d34640 10 monclient(hunting): picked mon.noname-a con 0x7f5a6c06ee00 addr v1:172.21.15.88:6789/0 2024-06-03T09:07:02.703 INFO:teuthology.orchestra.run.smithi088.stderr:2024-06-03T09:07:02.700+0000 7f5a73d34640 10 monclient(hunting): picked mon.noname-b con 0x7f5a6c0adb90 addr v1:172.21.15.192:6789/0 2024-06-03T09:07:02.703 INFO:teuthology.orchestra.run.smithi088.stderr:2024-06-03T09:07:02.700+0000 7f5a73d34640 10 monclient(hunting): picked mon.noname-c con 0x7f5a6c1383e0 addr v1:172.21.15.88:6790/0 2024-06-03T09:07:02.703 INFO:teuthology.orchestra.run.smithi088.stderr:2024-06-03T09:07:02.700+0000 7f5a73d34640 1 -- --> v1:172.21.15.88:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f5a6c13fe50 con 0x7f5a6c06ee00 2024-06-03T09:07:02.703 INFO:teuthology.orchestra.run.smithi088.stderr:2024-06-03T09:07:02.700+0000 7f5a73d34640 1 -- --> v1:172.21.15.192:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f5a6c0aa330 con 0x7f5a6c0adb90 2024-06-03T09:07:02.703 INFO:teuthology.orchestra.run.smithi088.stderr:2024-06-03T09:07:02.700+0000 7f5a73d34640 1 -- --> v1:172.21.15.88:6790/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f5a6c0ad360 con 0x7f5a6c1383e0 2024-06-03T09:07:02.703 INFO:teuthology.orchestra.run.smithi088.stderr:2024-06-03T09:07:02.700+0000 7f5a73d34640 10 monclient(hunting): _renew_subs 2024-06-03T09:07:02.703 INFO:teuthology.orchestra.run.smithi088.stderr:2024-06-03T09:07:02.700+0000 7f5a73d34640 10 monclient(hunting): authenticate will time out at 3078.510006s 2024-06-03T09:07:02.703 INFO:teuthology.orchestra.run.smithi088.stderr:2024-06-03T09:07:02.700+0000 7f5a722d1640 1 --1- >> v1:172.21.15.88:6790/0 conn(0x7f5a6c1383e0 0x7f5a6c1dd940 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=1).handle_server_banner_and_identify peer v1:172.21.15.88:6790/0 says I am v1:172.21.15.88:51228/0 (socket says 172.21.15.88:51228) 2024-06-03T09:07:02.703 INFO:teuthology.orchestra.run.smithi088.stderr:2024-06-03T09:07:02.700+0000 7f5a722d1640 1 -- 172.21.15.88:0/1520866693 learned_addr learned my addr 172.21.15.88:0/1520866693 (peer_addr_for_me v1:172.21.15.88:0/0) 2024-06-03T09:07:02.703 INFO:teuthology.orchestra.run.smithi088.stderr:2024-06-03T09:07:02.700+0000 7f5a627fc640 10 client.?.objecter ms_handle_connect 0x7f5a6c1383e0 2024-06-03T09:07:02.703 INFO:teuthology.orchestra.run.smithi088.stderr:2024-06-03T09:07:02.700+0000 7f5a627fc640 10 client.?.objecter resend_mon_ops 2024-06-03T09:07:02.703 INFO:teuthology.orchestra.run.smithi088.stderr:2024-06-03T09:07:02.701+0000 7f5a627fc640 10 client.?.objecter ms_handle_connect 0x7f5a6c0adb90 2024-06-03T09:07:02.703 INFO:teuthology.orchestra.run.smithi088.stderr:2024-06-03T09:07:02.701+0000 7f5a627fc640 10 client.?.objecter resend_mon_ops 2024-06-03T09:07:02.704 INFO:teuthology.orchestra.run.smithi088.stderr:2024-06-03T09:07:02.702+0000 7f5a627fc640 1 -- 172.21.15.88:0/1520866693 <== mon.2 v1:172.21.15.88:6790/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 3628827948 0 0) 0x7f5a6c0ad360 con 0x7f5a6c1383e0 2024-06-03T09:07:02.704 INFO:teuthology.orchestra.run.smithi088.stderr:2024-06-03T09:07:02.702+0000 7f5a627fc640 10 monclient(hunting): _negotiate 2024-06-03T09:07:02.704 INFO:teuthology.orchestra.run.smithi088.stderr:2024-06-03T09:07:02.702+0000 7f5a627fc640 10 monclient(hunting): _init_auth method 2 2024-06-03T09:07:02.704 INFO:teuthology.orchestra.run.smithi088.stderr:2024-06-03T09:07:02.702+0000 7f5a627fc640 10 monclient(hunting): _init_auth creating new auth 2024-06-03T09:07:02.705 INFO:teuthology.orchestra.run.smithi088.stderr:2024-06-03T09:07:02.702+0000 7f5a627fc640 10 monclient(hunting): my global_id is 30338 2024-06-03T09:07:02.705 INFO:teuthology.orchestra.run.smithi088.stderr:2024-06-03T09:07:02.702+0000 7f5a627fc640 1 -- 172.21.15.88:0/1520866693 --> v1:172.21.15.88:6790/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f5a50004eb0 con 0x7f5a6c1383e0 2024-06-03T09:07:02.705 INFO:teuthology.orchestra.run.smithi088.stderr:2024-06-03T09:07:02.703+0000 7f5a627fc640 1 -- 172.21.15.88:0/1520866693 <== mon.1 v1:172.21.15.192:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 4035597243 0 0) 0x7f5a6c0aa330 con 0x7f5a6c0adb90 2024-06-03T09:07:02.705 INFO:teuthology.orchestra.run.smithi088.stderr:2024-06-03T09:07:02.703+0000 7f5a627fc640 10 monclient(hunting): _negotiate 2024-06-03T09:07:02.705 INFO:teuthology.orchestra.run.smithi088.stderr:2024-06-03T09:07:02.703+0000 7f5a627fc640 10 monclient(hunting): _init_auth method 2 2024-06-03T09:07:02.705 INFO:teuthology.orchestra.run.smithi088.stderr:2024-06-03T09:07:02.703+0000 7f5a627fc640 10 monclient(hunting): _init_auth creating new auth 2024-06-03T09:07:02.705 INFO:teuthology.orchestra.run.smithi088.stderr:2024-06-03T09:07:02.703+0000 7f5a627fc640 10 monclient(hunting): my global_id is 30403 2024-06-03T09:07:02.705 INFO:teuthology.orchestra.run.smithi088.stderr:2024-06-03T09:07:02.703+0000 7f5a627fc640 1 -- 172.21.15.88:0/1520866693 --> v1:172.21.15.192:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f5a6c0ad360 con 0x7f5a6c0adb90 2024-06-03T09:07:02.705 INFO:teuthology.orchestra.run.smithi088.stderr:2024-06-03T09:07:02.704+0000 7f5a627fc640 10 client.?.objecter ms_handle_connect 0x7f5a6c06ee00 2024-06-03T09:07:02.705 INFO:teuthology.orchestra.run.smithi088.stderr:2024-06-03T09:07:02.704+0000 7f5a627fc640 10 client.?.objecter resend_mon_ops 2024-06-03T09:07:02.706 INFO:teuthology.orchestra.run.smithi088.stderr:2024-06-03T09:07:02.706+0000 7f5a627fc640 1 -- 172.21.15.88:0/1520866693 <== mon.0 v1:172.21.15.88:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 2278711492 0 0) 0x7f5a6c13fe50 con 0x7f5a6c06ee00 2024-06-03T09:07:02.706 INFO:teuthology.orchestra.run.smithi088.stderr:2024-06-03T09:07:02.706+0000 7f5a627fc640 10 monclient(hunting): _negotiate 2024-06-03T09:07:02.706 INFO:teuthology.orchestra.run.smithi088.stderr:2024-06-03T09:07:02.706+0000 7f5a627fc640 10 monclient(hunting): _init_auth method 2 2024-06-03T09:07:02.706 INFO:teuthology.orchestra.run.smithi088.stderr:2024-06-03T09:07:02.706+0000 7f5a627fc640 10 monclient(hunting): _init_auth creating new auth 2024-06-03T09:07:02.706 INFO:teuthology.orchestra.run.smithi088.stderr:2024-06-03T09:07:02.706+0000 7f5a627fc640 10 monclient(hunting): my global_id is 30828 2024-06-03T09:07:02.706 INFO:teuthology.orchestra.run.smithi088.stderr:2024-06-03T09:07:02.706+0000 7f5a627fc640 1 -- 172.21.15.88:0/1520866693 --> v1:172.21.15.88:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f5a6c0aa330 con 0x7f5a6c06ee00 2024-06-03T09:07:02.708 INFO:teuthology.orchestra.run.smithi088.stderr:2024-06-03T09:07:02.707+0000 7f5a627fc640 1 -- 172.21.15.88:0/1520866693 <== mon.2 v1:172.21.15.88:6790/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 764+0+0 (unknown 3686005386 0 0) 0x7f5a50004eb0 con 0x7f5a6c1383e0 2024-06-03T09:07:02.708 INFO:teuthology.orchestra.run.smithi088.stderr:2024-06-03T09:07:02.707+0000 7f5a627fc640 1 -- 172.21.15.88:0/1520866693 --> v1:172.21.15.88:6790/0 -- auth(proto 2 165 bytes epoch 0) v1 -- 0x7f5a6c13fe50 con 0x7f5a6c1383e0 2024-06-03T09:07:02.708 INFO:teuthology.orchestra.run.smithi088.stderr:2024-06-03T09:07:02.707+0000 7f5a627fc640 1 -- 172.21.15.88:0/1520866693 <== mon.2 v1:172.21.15.88:6790/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 2234374065 0 0) 0x7f5a64001b10 con 0x7f5a6c1383e0 2024-06-03T09:07:02.708 INFO:teuthology.orchestra.run.smithi088.stderr:2024-06-03T09:07:02.707+0000 7f5a627fc640 10 monclient(hunting): handle_monmap mon_map magic: 0 v1 2024-06-03T09:07:02.708 INFO:teuthology.orchestra.run.smithi088.stderr:2024-06-03T09:07:02.707+0000 7f5a627fc640 10 monclient(hunting): got monmap 1 from mon.noname-c (according to old e1) 2024-06-03T09:07:02.708 INFO:teuthology.orchestra.run.smithi088.stderr:2024-06-03T09:07:02.707+0000 7f5a627fc640 10 monclient(hunting): dump: 2024-06-03T09:07:02.708 INFO:teuthology.orchestra.run.smithi088.stderr:epoch 1 2024-06-03T09:07:02.708 INFO:teuthology.orchestra.run.smithi088.stderr:fsid ec3c3fdd-3ac6-4d7d-9412-683130ed8b04 2024-06-03T09:07:02.708 INFO:teuthology.orchestra.run.smithi088.stderr:last_changed 2024-06-03T08:34:31.885125+0000 2024-06-03T09:07:02.708 INFO:teuthology.orchestra.run.smithi088.stderr:created 2024-06-03T08:34:31.885125+0000 2024-06-03T09:07:02.708 INFO:teuthology.orchestra.run.smithi088.stderr:min_mon_release 18 (reef) 2024-06-03T09:07:02.709 INFO:teuthology.orchestra.run.smithi088.stderr:election_strategy: 3 2024-06-03T09:07:02.709 INFO:teuthology.orchestra.run.smithi088.stderr:0: v1:172.21.15.88:6789/0 mon.a 2024-06-03T09:07:02.709 INFO:teuthology.orchestra.run.smithi088.stderr:1: v1:172.21.15.192:6789/0 mon.b 2024-06-03T09:07:02.709 INFO:teuthology.orchestra.run.smithi088.stderr:2: v1:172.21.15.88:6790/0 mon.c 2024-06-03T09:07:02.709 INFO:teuthology.orchestra.run.smithi088.stderr: 2024-06-03T09:07:02.709 INFO:teuthology.orchestra.run.smithi088.stderr:2024-06-03T09:07:02.707+0000 7f5a627fc640 10 monclient(hunting): _finish_auth 0 2024-06-03T09:07:02.709 INFO:teuthology.orchestra.run.smithi088.stderr:2024-06-03T09:07:02.707+0000 7f5a627fc640 1 -- 172.21.15.88:0/1520866693 <== mon.1 v1:172.21.15.192:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 764+0+0 (unknown 3715096573 0 0) 0x7f5a6c0ad360 con 0x7f5a6c0adb90 2024-06-03T09:07:02.709 INFO:teuthology.orchestra.run.smithi088.stderr:2024-06-03T09:07:02.708+0000 7f5a627fc640 1 -- 172.21.15.88:0/1520866693 --> v1:172.21.15.192:6789/0 -- auth(proto 2 165 bytes epoch 0) v1 -- 0x7f5a50004eb0 con 0x7f5a6c0adb90 2024-06-03T09:07:02.709 INFO:teuthology.orchestra.run.smithi088.stderr:2024-06-03T09:07:02.708+0000 7f5a627fc640 1 -- 172.21.15.88:0/1520866693 <== mon.1 v1:172.21.15.192:6789/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 2234374065 0 0) 0x7f5a5c0018d0 con 0x7f5a6c0adb90 2024-06-03T09:07:02.709 INFO:teuthology.orchestra.run.smithi088.stderr:2024-06-03T09:07:02.708+0000 7f5a627fc640 10 monclient(hunting): handle_monmap mon_map magic: 0 v1 2024-06-03T09:07:02.709 INFO:teuthology.orchestra.run.smithi088.stderr:2024-06-03T09:07:02.708+0000 7f5a627fc640 10 monclient(hunting): got monmap 1 from mon.b (according to old e1) 2024-06-03T09:07:02.709 INFO:teuthology.orchestra.run.smithi088.stderr:2024-06-03T09:07:02.708+0000 7f5a627fc640 10 monclient(hunting): dump: 2024-06-03T09:07:02.709 INFO:teuthology.orchestra.run.smithi088.stderr:epoch 1 2024-06-03T09:07:02.710 INFO:teuthology.orchestra.run.smithi088.stderr:fsid ec3c3fdd-3ac6-4d7d-9412-683130ed8b04 2024-06-03T09:07:02.710 INFO:teuthology.orchestra.run.smithi088.stderr:last_changed 2024-06-03T08:34:31.885125+0000 2024-06-03T09:07:02.710 INFO:teuthology.orchestra.run.smithi088.stderr:created 2024-06-03T08:34:31.885125+0000 2024-06-03T09:07:02.710 INFO:teuthology.orchestra.run.smithi088.stderr:min_mon_release 18 (reef) 2024-06-03T09:07:02.710 INFO:teuthology.orchestra.run.smithi088.stderr:election_strategy: 3 2024-06-03T09:07:02.710 INFO:teuthology.orchestra.run.smithi088.stderr:0: v1:172.21.15.88:6789/0 mon.a 2024-06-03T09:07:02.710 INFO:teuthology.orchestra.run.smithi088.stderr:1: v1:172.21.15.192:6789/0 mon.b 2024-06-03T09:07:02.710 INFO:teuthology.orchestra.run.smithi088.stderr:2: v1:172.21.15.88:6790/0 mon.c 2024-06-03T09:07:02.710 INFO:teuthology.orchestra.run.smithi088.stderr: 2024-06-03T09:07:02.711 INFO:teuthology.orchestra.run.smithi088.stderr:2024-06-03T09:07:02.710+0000 7f5a627fc640 1 -- 172.21.15.88:0/1520866693 <== mon.0 v1:172.21.15.88:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 764+0+0 (unknown 2053758566 0 0) 0x7f5a6c0aa330 con 0x7f5a6c06ee00 2024-06-03T09:07:02.711 INFO:teuthology.orchestra.run.smithi088.stderr:2024-06-03T09:07:02.710+0000 7f5a627fc640 1 -- 172.21.15.88:0/1520866693 --> v1:172.21.15.88:6789/0 -- auth(proto 2 165 bytes epoch 0) v1 -- 0x7f5a6c0ad360 con 0x7f5a6c06ee00 2024-06-03T09:07:02.712 INFO:teuthology.orchestra.run.smithi088.stderr:2024-06-03T09:07:02.710+0000 7f5a627fc640 1 -- 172.21.15.88:0/1520866693 <== mon.0 v1:172.21.15.88:6789/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 2234374065 0 0) 0x7f5a680014f0 con 0x7f5a6c06ee00 2024-06-03T09:07:02.712 INFO:teuthology.orchestra.run.smithi088.stderr:2024-06-03T09:07:02.710+0000 7f5a627fc640 10 monclient(hunting): handle_monmap mon_map magic: 0 v1 2024-06-03T09:07:02.712 INFO:teuthology.orchestra.run.smithi088.stderr:2024-06-03T09:07:02.710+0000 7f5a627fc640 10 monclient(hunting): got monmap 1 from mon.a (according to old e1) 2024-06-03T09:07:02.712 INFO:teuthology.orchestra.run.smithi088.stderr:2024-06-03T09:07:02.710+0000 7f5a627fc640 10 monclient(hunting): dump: 2024-06-03T09:07:02.712 INFO:teuthology.orchestra.run.smithi088.stderr:epoch 1 2024-06-03T09:07:02.712 INFO:teuthology.orchestra.run.smithi088.stderr:fsid ec3c3fdd-3ac6-4d7d-9412-683130ed8b04 2024-06-03T09:07:02.712 INFO:teuthology.orchestra.run.smithi088.stderr:last_changed 2024-06-03T08:34:31.885125+0000 2024-06-03T09:07:02.712 INFO:teuthology.orchestra.run.smithi088.stderr:created 2024-06-03T08:34:31.885125+0000 2024-06-03T09:07:02.712 INFO:teuthology.orchestra.run.smithi088.stderr:min_mon_release 18 (reef) 2024-06-03T09:07:02.712 INFO:teuthology.orchestra.run.smithi088.stderr:election_strategy: 3 2024-06-03T09:07:02.712 INFO:teuthology.orchestra.run.smithi088.stderr:0: v1:172.21.15.88:6789/0 mon.a 2024-06-03T09:07:02.712 INFO:teuthology.orchestra.run.smithi088.stderr:1: v1:172.21.15.192:6789/0 mon.b 2024-06-03T09:07:02.712 INFO:teuthology.orchestra.run.smithi088.stderr:2: v1:172.21.15.88:6790/0 mon.c 2024-06-03T09:07:02.713 INFO:teuthology.orchestra.run.smithi088.stderr: 2024-06-03T09:07:02.713 INFO:teuthology.orchestra.run.smithi088.stderr:2024-06-03T09:07:02.711+0000 7f5a627fc640 1 -- 172.21.15.88:0/1520866693 <== mon.2 v1:172.21.15.88:6790/0 4 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 580+0+0 (unknown 1364546699 0 0) 0x7f5a6c13fe50 con 0x7f5a6c1383e0 2024-06-03T09:07:02.713 INFO:teuthology.orchestra.run.smithi088.stderr:2024-06-03T09:07:02.711+0000 7f5a627fc640 1 -- 172.21.15.88:0/1520866693 >> v1:172.21.15.192:6789/0 conn(0x7f5a6c0adb90 legacy=0x7f5a6c137110 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-06-03T09:07:02.713 INFO:teuthology.orchestra.run.smithi088.stderr:2024-06-03T09:07:02.711+0000 7f5a627fc640 1 -- 172.21.15.88:0/1520866693 >> v1:172.21.15.88:6789/0 conn(0x7f5a6c06ee00 legacy=0x7f5a6c0ac410 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-06-03T09:07:02.713 INFO:teuthology.orchestra.run.smithi088.stderr:2024-06-03T09:07:02.711+0000 7f5a627fc640 10 monclient: _finish_hunting 0 2024-06-03T09:07:02.713 INFO:teuthology.orchestra.run.smithi088.stderr:2024-06-03T09:07:02.711+0000 7f5a627fc640 1 monclient: found mon.c 2024-06-03T09:07:02.713 INFO:teuthology.orchestra.run.smithi088.stderr:2024-06-03T09:07:02.711+0000 7f5a627fc640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-06-03T09:07:02.713 INFO:teuthology.orchestra.run.smithi088.stderr:2024-06-03T09:07:02.711+0000 7f5a627fc640 10 monclient: _send_mon_message to mon.c at v1:172.21.15.88:6790/0 2024-06-03T09:07:02.713 INFO:teuthology.orchestra.run.smithi088.stderr:2024-06-03T09:07:02.711+0000 7f5a627fc640 1 -- 172.21.15.88:0/1520866693 --> v1:172.21.15.88:6790/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f5a6c0ad4f0 con 0x7f5a6c1383e0 2024-06-03T09:07:02.713 INFO:teuthology.orchestra.run.smithi088.stderr:2024-06-03T09:07:02.711+0000 7f5a627fc640 10 monclient: _finish_auth 0 2024-06-03T09:07:02.713 INFO:teuthology.orchestra.run.smithi088.stderr:2024-06-03T09:07:02.711+0000 7f5a627fc640 10 monclient: _check_auth_tickets 2024-06-03T09:07:02.713 INFO:teuthology.orchestra.run.smithi088.stderr:2024-06-03T09:07:02.711+0000 7f5a627fc640 20 monclient: _check_auth_rotating not needed by client.admin 2024-06-03T09:07:02.713 INFO:teuthology.orchestra.run.smithi088.stderr:2024-06-03T09:07:02.711+0000 7f5a73d34640 5 monclient: authenticate success, global_id 30338 2024-06-03T09:07:02.713 INFO:teuthology.orchestra.run.smithi088.stderr:2024-06-03T09:07:02.711+0000 7f5a73d34640 10 monclient: _renew_subs 2024-06-03T09:07:02.714 INFO:teuthology.orchestra.run.smithi088.stderr:2024-06-03T09:07:02.711+0000 7f5a73d34640 10 monclient: _send_mon_message to mon.c at v1:172.21.15.88:6790/0 2024-06-03T09:07:02.714 INFO:teuthology.orchestra.run.smithi088.stderr:2024-06-03T09:07:02.711+0000 7f5a73d34640 1 -- 172.21.15.88:0/1520866693 --> v1:172.21.15.88:6790/0 -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f5a6c0acb20 con 0x7f5a6c1383e0 2024-06-03T09:07:02.714 INFO:teuthology.orchestra.run.smithi088.stderr:2024-06-03T09:07:02.711+0000 7f5a73d34640 10 client.30338.objecter _maybe_request_map subscribing (onetime) to next osd map 2024-06-03T09:07:02.714 INFO:teuthology.orchestra.run.smithi088.stderr:2024-06-03T09:07:02.711+0000 7f5a73d34640 10 monclient: _renew_subs 2024-06-03T09:07:02.714 INFO:teuthology.orchestra.run.smithi088.stderr:2024-06-03T09:07:02.711+0000 7f5a73d34640 10 monclient: _send_mon_message to mon.c at v1:172.21.15.88:6790/0 2024-06-03T09:07:02.714 INFO:teuthology.orchestra.run.smithi088.stderr:2024-06-03T09:07:02.711+0000 7f5a73d34640 1 -- 172.21.15.88:0/1520866693 --> v1:172.21.15.88:6790/0 -- mon_subscribe({osdmap=0}) v3 -- 0x7f5a6c0aacd0 con 0x7f5a6c1383e0 2024-06-03T09:07:02.714 INFO:teuthology.orchestra.run.smithi088.stderr:2024-06-03T09:07:02.711+0000 7f5a73d34640 1 librados: init done 2024-06-03T09:07:02.714 INFO:teuthology.orchestra.run.smithi088.stderr:2024-06-03T09:07:02.712+0000 7f5a73d34640 10 start_mon_command cmd=[{"prefix": "get_command_descriptions"}] 2024-06-03T09:07:02.715 INFO:teuthology.orchestra.run.smithi088.stderr:2024-06-03T09:07:02.712+0000 7f5a73d34640 10 monclient: _send_command 1 [{"prefix": "get_command_descriptions"}] 2024-06-03T09:07:02.715 INFO:teuthology.orchestra.run.smithi088.stderr:2024-06-03T09:07:02.712+0000 7f5a73d34640 10 monclient: _send_mon_message to mon.c at v1:172.21.15.88:6790/0 2024-06-03T09:07:02.715 INFO:teuthology.orchestra.run.smithi088.stderr:2024-06-03T09:07:02.712+0000 7f5a73d34640 1 -- 172.21.15.88:0/1520866693 --> v1:172.21.15.88:6790/0 -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f5a6c070b00 con 0x7f5a6c1383e0 2024-06-03T09:07:02.727 INFO:teuthology.orchestra.run.smithi088.stderr:2024-06-03T09:07:02.727+0000 7f5a627fc640 1 -- 172.21.15.88:0/1520866693 <== mon.2 v1:172.21.15.88:6790/0 5 ==== config(0 keys) v1 ==== 4+0+0 (unknown 0 0 0) 0x7f5a64002060 con 0x7f5a6c1383e0 2024-06-03T09:07:02.727 INFO:teuthology.orchestra.run.smithi088.stderr:2024-06-03T09:07:02.727+0000 7f5a627fc640 10 monclient: handle_config config(0 keys) v1 2024-06-03T09:07:02.727 INFO:teuthology.orchestra.run.smithi088.stderr:2024-06-03T09:07:02.727+0000 7f5a627fc640 1 -- 172.21.15.88:0/1520866693 <== mon.2 v1:172.21.15.88:6790/0 6 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 2234374065 0 0) 0x7f5a64003b30 con 0x7f5a6c1383e0 2024-06-03T09:07:02.727 INFO:teuthology.orchestra.run.smithi088.stderr:2024-06-03T09:07:02.727+0000 7f5a627fc640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-06-03T09:07:02.727 INFO:teuthology.orchestra.run.smithi088.stderr:2024-06-03T09:07:02.727+0000 7f5a627fc640 10 monclient: got monmap 1 from mon.c (according to old e1) 2024-06-03T09:07:02.727 INFO:teuthology.orchestra.run.smithi088.stderr:2024-06-03T09:07:02.727+0000 7f5a627fc640 10 monclient: dump: 2024-06-03T09:07:02.728 INFO:teuthology.orchestra.run.smithi088.stderr:epoch 1 2024-06-03T09:07:02.728 INFO:teuthology.orchestra.run.smithi088.stderr:fsid ec3c3fdd-3ac6-4d7d-9412-683130ed8b04 2024-06-03T09:07:02.728 INFO:teuthology.orchestra.run.smithi088.stderr:last_changed 2024-06-03T08:34:31.885125+0000 2024-06-03T09:07:02.728 INFO:teuthology.orchestra.run.smithi088.stderr:created 2024-06-03T08:34:31.885125+0000 2024-06-03T09:07:02.728 INFO:teuthology.orchestra.run.smithi088.stderr:min_mon_release 18 (reef) 2024-06-03T09:07:02.728 INFO:teuthology.orchestra.run.smithi088.stderr:election_strategy: 3 2024-06-03T09:07:02.728 INFO:teuthology.orchestra.run.smithi088.stderr:0: v1:172.21.15.88:6789/0 mon.a 2024-06-03T09:07:02.728 INFO:teuthology.orchestra.run.smithi088.stderr:1: v1:172.21.15.192:6789/0 mon.b 2024-06-03T09:07:02.728 INFO:teuthology.orchestra.run.smithi088.stderr:2: v1:172.21.15.88:6790/0 mon.c 2024-06-03T09:07:02.728 INFO:teuthology.orchestra.run.smithi088.stderr: 2024-06-03T09:07:02.728 INFO:teuthology.orchestra.run.smithi088.stderr:2024-06-03T09:07:02.727+0000 7f5a63fff640 4 set_mon_vals no callback set 2024-06-03T09:07:02.729 INFO:teuthology.orchestra.run.smithi088.stderr:2024-06-03T09:07:02.728+0000 7f5a627fc640 1 -- 172.21.15.88:0/1520866693 <== mon.2 v1:172.21.15.88:6790/0 7 ==== mgrmap(e 5) v1 ==== 89110+0+0 (unknown 873533973 0 0) 0x7f5a64003c90 con 0x7f5a6c1383e0 2024-06-03T09:07:02.729 INFO:teuthology.orchestra.run.smithi088.stderr:2024-06-03T09:07:02.728+0000 7f5a71ad0640 10 monclient: get_auth_request con 0x7f5a5006d450 auth_method 0 2024-06-03T09:07:02.729 INFO:teuthology.orchestra.run.smithi088.stderr:2024-06-03T09:07:02.729+0000 7f5a627fc640 10 client.30338.objecter ms_handle_connect 0x7f5a5006d450 2024-06-03T09:07:02.730 INFO:teuthology.orchestra.run.smithi088.stderr:2024-06-03T09:07:02.730+0000 7f5a627fc640 1 -- 172.21.15.88:0/1520866693 <== mon.2 v1:172.21.15.88:6790/0 8 ==== osd_map(1014..1014 src has 622..1014) v4 ==== 5229+0+0 (unknown 3230192161 0 0) 0x7f5a640542d0 con 0x7f5a6c1383e0 2024-06-03T09:07:02.731 INFO:teuthology.orchestra.run.smithi088.stderr:2024-06-03T09:07:02.731+0000 7f5a627fc640 10 client.30338.objecter ms_dispatch 0x7f5a6c000b90 osd_map(1014..1014 src has 622..1014) v4 2024-06-03T09:07:02.731 INFO:teuthology.orchestra.run.smithi088.stderr:2024-06-03T09:07:02.731+0000 7f5a627fc640 3 client.30338.objecter handle_osd_map got epochs [1014,1014] > 0 2024-06-03T09:07:02.731 INFO:teuthology.orchestra.run.smithi088.stderr:2024-06-03T09:07:02.731+0000 7f5a627fc640 3 client.30338.objecter handle_osd_map decoding full epoch 1014 2024-06-03T09:07:02.731 INFO:teuthology.orchestra.run.smithi088.stderr:2024-06-03T09:07:02.731+0000 7f5a627fc640 20 client.30338.objecter dump_active .. 0 homeless 2024-06-03T09:07:02.965 INFO:teuthology.orchestra.run.smithi088.stderr:2024-06-03T09:07:02.965+0000 7f5a627fc640 1 -- 172.21.15.88:0/1520866693 <== mon.2 v1:172.21.15.88:6790/0 9 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+182498 (unknown 1092875540 0 3016760700) 0x7f5a6404fcb0 con 0x7f5a6c1383e0 2024-06-03T09:07:02.966 INFO:teuthology.orchestra.run.smithi088.stderr:2024-06-03T09:07:02.965+0000 7f5a627fc640 10 monclient: handle_mon_command_ack 1 [{"prefix": "get_command_descriptions"}] 2024-06-03T09:07:02.966 INFO:teuthology.orchestra.run.smithi088.stderr:2024-06-03T09:07:02.965+0000 7f5a627fc640 10 monclient: _finish_command 1 = system:0 2024-06-03T09:07:03.142 INFO:teuthology.orchestra.run.smithi088.stderr:2024-06-03T09:07:03.142+0000 7f5a73d34640 1 -- 172.21.15.88:0/1520866693 --> v1:172.21.15.192:6800/34229 -- mgr_command(tid 0: {"prefix": "pg dump", "target": ["mon-mgr", ""], "format": "json"}) v1 -- 0x7f5a6c0adf70 con 0x7f5a5006d450 2024-06-03T09:07:03.147 INFO:teuthology.orchestra.run.smithi088.stderr:2024-06-03T09:07:03.147+0000 7f5a627fc640 1 -- 172.21.15.88:0/1520866693 <== mgr.4110 v1:172.21.15.192:6800/34229 1 ==== mgr_command_reply(tid 0: 0 dumped all) v1 ==== 18+0+150977 (unknown 2965378022 0 3569599376) 0x7f5a68007810 con 0x7f5a5006d450 2024-06-03T09:07:03.150 INFO:teuthology.orchestra.run.smithi088.stdout: 2024-06-03T09:07:03.151 INFO:teuthology.orchestra.run.smithi088.stderr:dumped all 2024-06-03T09:07:03.154 INFO:teuthology.orchestra.run.smithi088.stderr:2024-06-03T09:07:03.149+0000 7f5a73d34640 10 librados: watch_flush enter 2024-06-03T09:07:03.154 INFO:teuthology.orchestra.run.smithi088.stderr:2024-06-03T09:07:03.149+0000 7f5a73d34640 10 librados: watch_flush exit 2024-06-03T09:07:03.154 INFO:teuthology.orchestra.run.smithi088.stderr:2024-06-03T09:07:03.149+0000 7f5a73d34640 20 client.30338.objecter shutdown clearing up homeless session... 2024-06-03T09:07:03.155 INFO:teuthology.orchestra.run.smithi088.stderr:2024-06-03T09:07:03.149+0000 7f5a73d34640 10 client.30338.objecter successfully canceled tick 2024-06-03T09:07:03.155 INFO:teuthology.orchestra.run.smithi088.stderr:2024-06-03T09:07:03.149+0000 7f5a73d34640 1 -- 172.21.15.88:0/1520866693 >> v1:172.21.15.192:6800/34229 conn(0x7f5a5006d450 legacy=0x7f5a5006f910 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-06-03T09:07:03.155 INFO:teuthology.orchestra.run.smithi088.stderr:2024-06-03T09:07:03.149+0000 7f5a73d34640 10 monclient: shutdown 2024-06-03T09:07:03.155 INFO:teuthology.orchestra.run.smithi088.stderr:2024-06-03T09:07:03.149+0000 7f5a73d34640 20 monclient: shutdown discarding 0 pending message(s) 2024-06-03T09:07:03.155 INFO:teuthology.orchestra.run.smithi088.stderr:2024-06-03T09:07:03.149+0000 7f5a73d34640 1 -- 172.21.15.88:0/1520866693 >> v1:172.21.15.88:6790/0 conn(0x7f5a6c1383e0 legacy=0x7f5a6c1dd940 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-06-03T09:07:03.155 INFO:teuthology.orchestra.run.smithi088.stderr:2024-06-03T09:07:03.149+0000 7f5a73d34640 1 -- 172.21.15.88:0/1520866693 shutdown_connections 2024-06-03T09:07:03.156 INFO:teuthology.orchestra.run.smithi088.stderr:2024-06-03T09:07:03.149+0000 7f5a73d34640 1 -- 172.21.15.88:0/1520866693 >> 172.21.15.88:0/1520866693 conn(0x7f5a6c0b2f20 msgr2=0x7f5a6c0b13e0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-06-03T09:07:03.156 INFO:teuthology.orchestra.run.smithi088.stderr:2024-06-03T09:07:03.149+0000 7f5a73d34640 1 -- 172.21.15.88:0/1520866693 shutdown_connections 2024-06-03T09:07:03.156 INFO:teuthology.orchestra.run.smithi088.stderr:2024-06-03T09:07:03.149+0000 7f5a73d34640 1 -- 172.21.15.88:0/1520866693 wait complete. 2024-06-03T09:07:03.156 INFO:teuthology.orchestra.run.smithi088.stderr:2024-06-03T09:07:03.149+0000 7f5a73d34640 1 librados: shutdown 2024-06-03T09:07:03.161 INFO:teuthology.orchestra.run.smithi088.stdout:{"pg_ready":true,"pg_map":{"version":1941,"stamp":"2024-06-03T09:07:03.107957+0000","last_osdmap_epoch":0,"last_pg_scan":0,"pg_stats_sum":{"stat_sum":{"num_bytes":459285,"num_objects":3,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":3,"num_whiteouts":0,"num_read":102,"num_read_kb":88,"num_write":120,"num_write_kb":1379,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2,"num_bytes_recovered":397840,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":85,"ondisk_log_size":85,"up":82,"acting":82,"num_store_stats":0},"osd_stats_sum":{"up_from":0,"seq":0,"num_pgs":82,"num_osds":8,"num_per_pool_osds":8,"num_per_pool_omap_osds":8,"kb":754974720,"kb_used":495040,"kb_used_data":46808,"kb_used_omap":12,"kb_used_meta":448115,"kb_avail":754479680,"statfs":{"total":773094113280,"available":772587192320,"internally_reserved":0,"allocated":47931392,"data_stored":22900274,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":12744,"internal_metadata":458870328},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":39,"apply_latency_ms":39,"commit_latency_ns":39000000,"apply_latency_ns":39000000},"alerts":[],"network_ping_times":[]},"pg_stats_delta":{"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":0,"acting":0,"num_store_stats":0,"stamp_delta":"12.002877"},"pg_stats":[{"pgid":"17.d","version":"0'0","reported_seq":946,"reported_epoch":1014,"state":"active+clean","last_fresh":"2024-06-03T09:06:46.148129+0000","last_change":"2024-06-03T08:38:19.260548+0000","last_active":"2024-06-03T09:06:46.148129+0000","last_peered":"2024-06-03T09:06:46.148129+0000","last_clean":"2024-06-03T09:06:46.148129+0000","last_became_active":"2024-06-03T08:38:19.253559+0000","last_became_peered":"2024-06-03T08:38:19.253559+0000","last_unstale":"2024-06-03T09:06:46.148129+0000","last_undegraded":"2024-06-03T09:06:46.148129+0000","last_fullsized":"2024-06-03T09:06:46.148129+0000","mapping_epoch":28,"log_start":"0'0","ondisk_log_start":"0'0","created":28,"last_epoch_clean":45,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-06-03T08:37:55.614497+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-06-03T08:37:55.614497+0000","last_clean_scrub_stamp":"2024-06-03T08:37:55.614497+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for deep scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1],"acting":[0,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"17.c","version":"0'0","reported_seq":966,"reported_epoch":1014,"state":"active+clean","last_fresh":"2024-06-03T09:06:46.325813+0000","last_change":"2024-06-03T09:06:46.317689+0000","last_active":"2024-06-03T09:06:46.325813+0000","last_peered":"2024-06-03T09:06:46.325813+0000","last_clean":"2024-06-03T09:06:46.325813+0000","last_became_active":"2024-06-03T09:06:46.312637+0000","last_became_peered":"2024-06-03T09:06:46.312637+0000","last_unstale":"2024-06-03T09:06:46.325813+0000","last_undegraded":"2024-06-03T09:06:46.325813+0000","last_fullsized":"2024-06-03T09:06:46.325813+0000","mapping_epoch":1013,"log_start":"0'0","ondisk_log_start":"0'0","created":28,"last_epoch_clean":1014,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-06-03T08:37:55.614497+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-06-03T08:37:55.614497+0000","last_clean_scrub_stamp":"2024-06-03T08:37:55.614497+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for deep scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,6],"acting":[3,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"17.f","version":"0'0","reported_seq":843,"reported_epoch":1014,"state":"active+clean","last_fresh":"2024-06-03T09:06:46.306998+0000","last_change":"2024-06-03T09:06:46.306998+0000","last_active":"2024-06-03T09:06:46.306998+0000","last_peered":"2024-06-03T09:06:46.306998+0000","last_clean":"2024-06-03T09:06:46.306998+0000","last_became_active":"2024-06-03T09:06:46.301958+0000","last_became_peered":"2024-06-03T09:06:46.301958+0000","last_unstale":"2024-06-03T09:06:46.306998+0000","last_undegraded":"2024-06-03T09:06:46.306998+0000","last_fullsized":"2024-06-03T09:06:46.306998+0000","mapping_epoch":1013,"log_start":"0'0","ondisk_log_start":"0'0","created":28,"last_epoch_clean":1014,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-06-03T08:37:55.614497+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-06-03T08:37:55.614497+0000","last_clean_scrub_stamp":"2024-06-03T08:37:55.614497+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-06-04T14:30:23.089522+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,4],"acting":[6,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"17.e","version":"0'0","reported_seq":843,"reported_epoch":1014,"state":"active+clean","last_fresh":"2024-06-03T09:06:46.278835+0000","last_change":"2024-06-03T09:06:46.278835+0000","last_active":"2024-06-03T09:06:46.278835+0000","last_peered":"2024-06-03T09:06:46.278835+0000","last_clean":"2024-06-03T09:06:46.278835+0000","last_became_active":"2024-06-03T09:06:46.273941+0000","last_became_peered":"2024-06-03T09:06:46.273941+0000","last_unstale":"2024-06-03T09:06:46.278835+0000","last_undegraded":"2024-06-03T09:06:46.278835+0000","last_fullsized":"2024-06-03T09:06:46.278835+0000","mapping_epoch":1013,"log_start":"0'0","ondisk_log_start":"0'0","created":28,"last_epoch_clean":1014,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-06-03T08:37:55.614497+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-06-03T08:37:55.614497+0000","last_clean_scrub_stamp":"2024-06-03T08:37:55.614497+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-06-04T09:31:07.311650+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,5],"acting":[6,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"17.9","version":"0'0","reported_seq":949,"reported_epoch":1013,"state":"active+clean","last_fresh":"2024-06-03T09:06:45.422478+0000","last_change":"2024-06-03T08:38:04.772260+0000","last_active":"2024-06-03T09:06:45.422478+0000","last_peered":"2024-06-03T09:06:45.422478+0000","last_clean":"2024-06-03T09:06:45.422478+0000","last_became_active":"2024-06-03T08:38:04.763428+0000","last_became_peered":"2024-06-03T08:38:04.763428+0000","last_unstale":"2024-06-03T09:06:45.422478+0000","last_undegraded":"2024-06-03T09:06:45.422478+0000","last_fullsized":"2024-06-03T09:06:45.422478+0000","mapping_epoch":28,"log_start":"0'0","ondisk_log_start":"0'0","created":28,"last_epoch_clean":33,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-06-03T08:37:55.614497+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-06-03T08:37:55.614497+0000","last_clean_scrub_stamp":"2024-06-03T08:37:55.614497+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-06-04T10:55:10.773936+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,4],"acting":[7,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"17.8","version":"0'0","reported_seq":984,"reported_epoch":1014,"state":"active+clean","last_fresh":"2024-06-03T09:06:59.146085+0000","last_change":"2024-06-03T09:06:59.150900+0000","last_active":"2024-06-03T09:06:59.146085+0000","last_peered":"2024-06-03T09:06:59.146085+0000","last_clean":"2024-06-03T09:06:59.146085+0000","last_became_active":"2024-06-03T09:06:46.344756+0000","last_became_peered":"2024-06-03T09:06:46.344756+0000","last_unstale":"2024-06-03T09:06:59.146085+0000","last_undegraded":"2024-06-03T09:06:59.146085+0000","last_fullsized":"2024-06-03T09:06:59.146085+0000","mapping_epoch":1013,"log_start":"0'0","ondisk_log_start":"0'0","created":28,"last_epoch_clean":1014,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-06-03T09:06:59.144935+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-06-03T09:06:59.144935+0000","last_clean_scrub_stamp":"2024-06-03T09:06:59.144935+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-06-04T15:52:37.265440+0000","scrub_duration":0.031008068,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,6],"acting":[0,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"17.b","version":"0'0","reported_seq":843,"reported_epoch":1014,"state":"active+clean","last_fresh":"2024-06-03T09:06:46.350450+0000","last_change":"2024-06-03T09:06:46.350450+0000","last_active":"2024-06-03T09:06:46.350450+0000","last_peered":"2024-06-03T09:06:46.350450+0000","last_clean":"2024-06-03T09:06:46.350450+0000","last_became_active":"2024-06-03T09:06:46.345458+0000","last_became_peered":"2024-06-03T09:06:46.345458+0000","last_unstale":"2024-06-03T09:06:46.350450+0000","last_undegraded":"2024-06-03T09:06:46.350450+0000","last_fullsized":"2024-06-03T09:06:46.350450+0000","mapping_epoch":1013,"log_start":"0'0","ondisk_log_start":"0'0","created":28,"last_epoch_clean":1014,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-06-03T08:37:55.614497+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-06-03T08:37:55.614497+0000","last_clean_scrub_stamp":"2024-06-03T08:37:55.614497+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-06-04T16:51:34.952224+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,2],"acting":[6,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"17.a","version":"0'0","reported_seq":930,"reported_epoch":1014,"state":"active+clean","last_fresh":"2024-06-03T09:06:46.473398+0000","last_change":"2024-06-03T08:38:05.843142+0000","last_active":"2024-06-03T09:06:46.473398+0000","last_peered":"2024-06-03T09:06:46.473398+0000","last_clean":"2024-06-03T09:06:46.473398+0000","last_became_active":"2024-06-03T08:38:05.809257+0000","last_became_peered":"2024-06-03T08:38:05.809257+0000","last_unstale":"2024-06-03T09:06:46.473398+0000","last_undegraded":"2024-06-03T09:06:46.473398+0000","last_fullsized":"2024-06-03T09:06:46.473398+0000","mapping_epoch":28,"log_start":"0'0","ondisk_log_start":"0'0","created":28,"last_epoch_clean":33,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-06-03T08:37:55.614497+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-06-03T08:37:55.614497+0000","last_clean_scrub_stamp":"2024-06-03T08:37:55.614497+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for deep scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2],"acting":[1,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"17.5","version":"0'0","reported_seq":947,"reported_epoch":1014,"state":"active+clean","last_fresh":"2024-06-03T09:06:46.146564+0000","last_change":"2024-06-03T08:38:23.466051+0000","last_active":"2024-06-03T09:06:46.146564+0000","last_peered":"2024-06-03T09:06:46.146564+0000","last_clean":"2024-06-03T09:06:46.146564+0000","last_became_active":"2024-06-03T08:38:23.458039+0000","last_became_peered":"2024-06-03T08:38:23.458039+0000","last_unstale":"2024-06-03T09:06:46.146564+0000","last_undegraded":"2024-06-03T09:06:46.146564+0000","last_fullsized":"2024-06-03T09:06:46.146564+0000","mapping_epoch":28,"log_start":"0'0","ondisk_log_start":"0'0","created":28,"last_epoch_clean":49,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-06-03T08:37:55.614497+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-06-03T08:37:55.614497+0000","last_clean_scrub_stamp":"2024-06-03T08:37:55.614497+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for deep scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,4],"acting":[0,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"17.4","version":"0'0","reported_seq":843,"reported_epoch":1014,"state":"active+clean","last_fresh":"2024-06-03T09:06:46.456388+0000","last_change":"2024-06-03T09:06:46.456388+0000","last_active":"2024-06-03T09:06:46.456388+0000","last_peered":"2024-06-03T09:06:46.456388+0000","last_clean":"2024-06-03T09:06:46.456388+0000","last_became_active":"2024-06-03T09:06:46.447534+0000","last_became_peered":"2024-06-03T09:06:46.447534+0000","last_unstale":"2024-06-03T09:06:46.456388+0000","last_undegraded":"2024-06-03T09:06:46.456388+0000","last_fullsized":"2024-06-03T09:06:46.456388+0000","mapping_epoch":1013,"log_start":"0'0","ondisk_log_start":"0'0","created":28,"last_epoch_clean":1014,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-06-03T08:37:55.614497+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-06-03T08:37:55.614497+0000","last_clean_scrub_stamp":"2024-06-03T08:37:55.614497+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-06-04T16:06:17.792570+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,7],"acting":[6,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"17.7","version":"0'0","reported_seq":935,"reported_epoch":1014,"state":"active+clean","last_fresh":"2024-06-03T09:07:00.999537+0000","last_change":"2024-06-03T08:38:04.837083+0000","last_active":"2024-06-03T09:07:00.999537+0000","last_peered":"2024-06-03T09:07:00.999537+0000","last_clean":"2024-06-03T09:07:00.999537+0000","last_became_active":"2024-06-03T08:38:04.818246+0000","last_became_peered":"2024-06-03T08:38:04.818246+0000","last_unstale":"2024-06-03T09:07:00.999537+0000","last_undegraded":"2024-06-03T09:07:00.999537+0000","last_fullsized":"2024-06-03T09:07:00.999537+0000","mapping_epoch":28,"log_start":"0'0","ondisk_log_start":"0'0","created":28,"last_epoch_clean":34,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-06-03T08:37:55.614497+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-06-03T08:37:55.614497+0000","last_clean_scrub_stamp":"2024-06-03T08:37:55.614497+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for deep scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,2],"acting":[5,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"17.6","version":"0'0","reported_seq":972,"reported_epoch":1014,"state":"active+clean","last_fresh":"2024-06-03T09:06:46.371177+0000","last_change":"2024-06-03T09:06:46.362623+0000","last_active":"2024-06-03T09:06:46.371177+0000","last_peered":"2024-06-03T09:06:46.371177+0000","last_clean":"2024-06-03T09:06:46.371177+0000","last_became_active":"2024-06-03T09:06:46.357698+0000","last_became_peered":"2024-06-03T09:06:46.357698+0000","last_unstale":"2024-06-03T09:06:46.371177+0000","last_undegraded":"2024-06-03T09:06:46.371177+0000","last_fullsized":"2024-06-03T09:06:46.371177+0000","mapping_epoch":1013,"log_start":"0'0","ondisk_log_start":"0'0","created":28,"last_epoch_clean":1014,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-06-03T08:37:55.614497+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-06-03T08:37:55.614497+0000","last_clean_scrub_stamp":"2024-06-03T08:37:55.614497+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for deep scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,6],"acting":[0,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"2.7","version":"0'0","reported_seq":945,"reported_epoch":1014,"state":"active+clean","last_fresh":"2024-06-03T09:06:46.145709+0000","last_change":"2024-06-03T08:36:30.385405+0000","last_active":"2024-06-03T09:06:46.145709+0000","last_peered":"2024-06-03T09:06:46.145709+0000","last_clean":"2024-06-03T09:06:46.145709+0000","last_became_active":"2024-06-03T08:36:30.380302+0000","last_became_peered":"2024-06-03T08:36:30.380302+0000","last_unstale":"2024-06-03T09:06:46.145709+0000","last_undegraded":"2024-06-03T09:06:46.145709+0000","last_fullsized":"2024-06-03T09:06:46.145709+0000","mapping_epoch":23,"log_start":"0'0","ondisk_log_start":"0'0","created":23,"last_epoch_clean":24,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-06-03T08:36:28.444764+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-06-03T08:36:28.444764+0000","last_clean_scrub_stamp":"2024-06-03T08:36:28.444764+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for deep scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0],"acting":[3,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"17.14","version":"0'0","reported_seq":931,"reported_epoch":1014,"state":"active+clean","last_fresh":"2024-06-03T09:06:46.203346+0000","last_change":"2024-06-03T08:38:04.547565+0000","last_active":"2024-06-03T09:06:46.203346+0000","last_peered":"2024-06-03T09:06:46.203346+0000","last_clean":"2024-06-03T09:06:46.203346+0000","last_became_active":"2024-06-03T08:38:04.541781+0000","last_became_peered":"2024-06-03T08:38:04.541781+0000","last_unstale":"2024-06-03T09:06:46.203346+0000","last_undegraded":"2024-06-03T09:06:46.203346+0000","last_fullsized":"2024-06-03T09:06:46.203346+0000","mapping_epoch":28,"log_start":"0'0","ondisk_log_start":"0'0","created":28,"last_epoch_clean":34,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-06-03T08:37:55.614497+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-06-03T08:37:55.614497+0000","last_clean_scrub_stamp":"2024-06-03T08:37:55.614497+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for deep scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,2],"acting":[5,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"2.6","version":"0'0","reported_seq":953,"reported_epoch":1013,"state":"active+clean","last_fresh":"2024-06-03T09:06:45.018770+0000","last_change":"2024-06-03T08:36:30.430760+0000","last_active":"2024-06-03T09:06:45.018770+0000","last_peered":"2024-06-03T09:06:45.018770+0000","last_clean":"2024-06-03T09:06:45.018770+0000","last_became_active":"2024-06-03T08:36:30.404665+0000","last_became_peered":"2024-06-03T08:36:30.404665+0000","last_unstale":"2024-06-03T09:06:45.018770+0000","last_undegraded":"2024-06-03T09:06:45.018770+0000","last_fullsized":"2024-06-03T09:06:45.018770+0000","mapping_epoch":23,"log_start":"0'0","ondisk_log_start":"0'0","created":23,"last_epoch_clean":24,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-06-03T08:36:28.444764+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-06-03T08:36:28.444764+0000","last_clean_scrub_stamp":"2024-06-03T08:36:28.444764+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-06-04T10:37:37.746074+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,0],"acting":[4,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"17.15","version":"0'0","reported_seq":939,"reported_epoch":1014,"state":"active+clean","last_fresh":"2024-06-03T09:06:46.147503+0000","last_change":"2024-06-03T08:38:20.999903+0000","last_active":"2024-06-03T09:06:46.147503+0000","last_peered":"2024-06-03T09:06:46.147503+0000","last_clean":"2024-06-03T09:06:46.147503+0000","last_became_active":"2024-06-03T08:38:20.979523+0000","last_became_peered":"2024-06-03T08:38:20.979523+0000","last_unstale":"2024-06-03T09:06:46.147503+0000","last_undegraded":"2024-06-03T09:06:46.147503+0000","last_fullsized":"2024-06-03T09:06:46.147503+0000","mapping_epoch":28,"log_start":"0'0","ondisk_log_start":"0'0","created":28,"last_epoch_clean":47,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-06-03T08:37:55.614497+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-06-03T08:37:55.614497+0000","last_clean_scrub_stamp":"2024-06-03T08:37:55.614497+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for deep scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,1],"acting":[3,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"2.5","version":"0'0","reported_seq":947,"reported_epoch":1014,"state":"active+clean","last_fresh":"2024-06-03T09:06:46.149070+0000","last_change":"2024-06-03T08:36:30.024235+0000","last_active":"2024-06-03T09:06:46.149070+0000","last_peered":"2024-06-03T09:06:46.149070+0000","last_clean":"2024-06-03T09:06:46.149070+0000","last_became_active":"2024-06-03T08:36:30.019040+0000","last_became_peered":"2024-06-03T08:36:30.019040+0000","last_unstale":"2024-06-03T09:06:46.149070+0000","last_undegraded":"2024-06-03T09:06:46.149070+0000","last_fullsized":"2024-06-03T09:06:46.149070+0000","mapping_epoch":23,"log_start":"0'0","ondisk_log_start":"0'0","created":23,"last_epoch_clean":24,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-06-03T08:36:28.444764+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-06-03T08:36:28.444764+0000","last_clean_scrub_stamp":"2024-06-03T08:36:28.444764+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for deep scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,4],"acting":[3,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"17.16","version":"0'0","reported_seq":996,"reported_epoch":1014,"state":"active+clean","last_fresh":"2024-06-03T09:06:59.968118+0000","last_change":"2024-06-03T09:06:59.969669+0000","last_active":"2024-06-03T09:06:59.968118+0000","last_peered":"2024-06-03T09:06:59.968118+0000","last_clean":"2024-06-03T09:06:59.968118+0000","last_became_active":"2024-06-03T09:06:46.379518+0000","last_became_peered":"2024-06-03T09:06:46.379518+0000","last_unstale":"2024-06-03T09:06:59.968118+0000","last_undegraded":"2024-06-03T09:06:59.968118+0000","last_fullsized":"2024-06-03T09:06:59.968118+0000","mapping_epoch":1013,"log_start":"0'0","ondisk_log_start":"0'0","created":28,"last_epoch_clean":1014,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-06-03T09:06:59.966233+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-06-03T09:06:59.966233+0000","last_clean_scrub_stamp":"2024-06-03T09:06:59.966233+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-06-04T18:45:08.505938+0000","scrub_duration":0.037443454000000001,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,6],"acting":[2,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"2.4","version":"0'0","reported_seq":965,"reported_epoch":1014,"state":"active+clean","last_fresh":"2024-06-03T09:06:46.417775+0000","last_change":"2024-06-03T09:06:46.377610+0000","last_active":"2024-06-03T09:06:46.417775+0000","last_peered":"2024-06-03T09:06:46.417775+0000","last_clean":"2024-06-03T09:06:46.417775+0000","last_became_active":"2024-06-03T09:06:46.371430+0000","last_became_peered":"2024-06-03T09:06:46.371430+0000","last_unstale":"2024-06-03T09:06:46.417775+0000","last_undegraded":"2024-06-03T09:06:46.417775+0000","last_fullsized":"2024-06-03T09:06:46.417775+0000","mapping_epoch":1013,"log_start":"0'0","ondisk_log_start":"0'0","created":23,"last_epoch_clean":1014,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-06-03T08:36:28.444764+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-06-03T08:36:28.444764+0000","last_clean_scrub_stamp":"2024-06-03T08:36:28.444764+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for deep scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,6],"acting":[5,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"17.17","version":"0'0","reported_seq":959,"reported_epoch":1014,"state":"active+clean","last_fresh":"2024-06-03T09:06:46.218924+0000","last_change":"2024-06-03T08:38:38.483275+0000","last_active":"2024-06-03T09:06:46.218924+0000","last_peered":"2024-06-03T09:06:46.218924+0000","last_clean":"2024-06-03T09:06:46.218924+0000","last_became_active":"2024-06-03T08:38:38.467916+0000","last_became_peered":"2024-06-03T08:38:38.467916+0000","last_unstale":"2024-06-03T09:06:46.218924+0000","last_undegraded":"2024-06-03T09:06:46.218924+0000","last_fullsized":"2024-06-03T09:06:46.218924+0000","mapping_epoch":28,"log_start":"0'0","ondisk_log_start":"0'0","created":28,"last_epoch_clean":61,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-06-03T08:37:55.614497+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-06-03T08:37:55.614497+0000","last_clean_scrub_stamp":"2024-06-03T08:37:55.614497+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for deep scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,7],"acting":[2,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"2.2","version":"0'0","reported_seq":976,"reported_epoch":1014,"state":"active+clean","last_fresh":"2024-06-03T09:06:46.295944+0000","last_change":"2024-06-03T09:06:46.290105+0000","last_active":"2024-06-03T09:06:46.295944+0000","last_peered":"2024-06-03T09:06:46.295944+0000","last_clean":"2024-06-03T09:06:46.295944+0000","last_became_active":"2024-06-03T09:06:46.284836+0000","last_became_peered":"2024-06-03T09:06:46.284836+0000","last_unstale":"2024-06-03T09:06:46.295944+0000","last_undegraded":"2024-06-03T09:06:46.295944+0000","last_fullsized":"2024-06-03T09:06:46.295944+0000","mapping_epoch":1013,"log_start":"0'0","ondisk_log_start":"0'0","created":23,"last_epoch_clean":1014,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-06-03T08:36:28.444764+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-06-03T08:36:28.444764+0000","last_clean_scrub_stamp":"2024-06-03T08:36:28.444764+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for deep scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,6],"acting":[3,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"17.11","version":"0'0","reported_seq":947,"reported_epoch":1013,"state":"active+clean","last_fresh":"2024-06-03T09:06:44.976905+0000","last_change":"2024-06-03T08:38:06.502302+0000","last_active":"2024-06-03T09:06:44.976905+0000","last_peered":"2024-06-03T09:06:44.976905+0000","last_clean":"2024-06-03T09:06:44.976905+0000","last_became_active":"2024-06-03T08:38:06.490204+0000","last_became_peered":"2024-06-03T08:38:06.490204+0000","last_unstale":"2024-06-03T09:06:44.976905+0000","last_undegraded":"2024-06-03T09:06:44.976905+0000","last_fullsized":"2024-06-03T09:06:44.976905+0000","mapping_epoch":28,"log_start":"0'0","ondisk_log_start":"0'0","created":28,"last_epoch_clean":32,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-06-03T08:37:55.614497+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-06-03T08:37:55.614497+0000","last_clean_scrub_stamp":"2024-06-03T08:37:55.614497+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-06-04T10:47:05.988958+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,2],"acting":[4,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"2.1","version":"0'0","reported_seq":955,"reported_epoch":1013,"state":"active+clean","last_fresh":"2024-06-03T09:06:45.423921+0000","last_change":"2024-06-03T08:36:30.883360+0000","last_active":"2024-06-03T09:06:45.423921+0000","last_peered":"2024-06-03T09:06:45.423921+0000","last_clean":"2024-06-03T09:06:45.423921+0000","last_became_active":"2024-06-03T08:36:30.855073+0000","last_became_peered":"2024-06-03T08:36:30.855073+0000","last_unstale":"2024-06-03T09:06:45.423921+0000","last_undegraded":"2024-06-03T09:06:45.423921+0000","last_fullsized":"2024-06-03T09:06:45.423921+0000","mapping_epoch":23,"log_start":"0'0","ondisk_log_start":"0'0","created":23,"last_epoch_clean":25,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-06-03T08:36:28.444764+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-06-03T08:36:28.444764+0000","last_clean_scrub_stamp":"2024-06-03T08:36:28.444764+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-06-04T18:12:22.363473+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,5],"acting":[7,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"17.12","version":"0'0","reported_seq":948,"reported_epoch":1013,"state":"active+clean","last_fresh":"2024-06-03T09:06:45.447050+0000","last_change":"2024-06-03T08:38:01.751562+0000","last_active":"2024-06-03T09:06:45.447050+0000","last_peered":"2024-06-03T09:06:45.447050+0000","last_clean":"2024-06-03T09:06:45.447050+0000","last_became_active":"2024-06-03T08:38:01.745027+0000","last_became_peered":"2024-06-03T08:38:01.745027+0000","last_unstale":"2024-06-03T09:06:45.447050+0000","last_undegraded":"2024-06-03T09:06:45.447050+0000","last_fullsized":"2024-06-03T09:06:45.447050+0000","mapping_epoch":28,"log_start":"0'0","ondisk_log_start":"0'0","created":28,"last_epoch_clean":32,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-06-03T08:37:55.614497+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-06-03T08:37:55.614497+0000","last_clean_scrub_stamp":"2024-06-03T08:37:55.614497+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-06-04T17:54:43.785326+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,3],"acting":[7,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"2.0","version":"0'0","reported_seq":949,"reported_epoch":1014,"state":"active+clean","last_fresh":"2024-06-03T09:06:46.142803+0000","last_change":"2024-06-03T08:36:29.984096+0000","last_active":"2024-06-03T09:06:46.142803+0000","last_peered":"2024-06-03T09:06:46.142803+0000","last_clean":"2024-06-03T09:06:46.142803+0000","last_became_active":"2024-06-03T08:36:29.957875+0000","last_became_peered":"2024-06-03T08:36:29.957875+0000","last_unstale":"2024-06-03T09:06:46.142803+0000","last_undegraded":"2024-06-03T09:06:46.142803+0000","last_fullsized":"2024-06-03T09:06:46.142803+0000","mapping_epoch":23,"log_start":"0'0","ondisk_log_start":"0'0","created":23,"last_epoch_clean":24,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-06-03T08:36:28.444764+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-06-03T08:36:28.444764+0000","last_clean_scrub_stamp":"2024-06-03T08:36:28.444764+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for deep scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,5],"acting":[3,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"17.13","version":"0'0","reported_seq":946,"reported_epoch":1013,"state":"active+clean","last_fresh":"2024-06-03T09:06:44.996126+0000","last_change":"2024-06-03T08:38:26.838316+0000","last_active":"2024-06-03T09:06:44.996126+0000","last_peered":"2024-06-03T09:06:44.996126+0000","last_clean":"2024-06-03T09:06:44.996126+0000","last_became_active":"2024-06-03T08:38:26.833214+0000","last_became_peered":"2024-06-03T08:38:26.833214+0000","last_unstale":"2024-06-03T09:06:44.996126+0000","last_undegraded":"2024-06-03T09:06:44.996126+0000","last_fullsized":"2024-06-03T09:06:44.996126+0000","mapping_epoch":28,"log_start":"0'0","ondisk_log_start":"0'0","created":28,"last_epoch_clean":52,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-06-03T08:37:55.614497+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-06-03T08:37:55.614497+0000","last_clean_scrub_stamp":"2024-06-03T08:37:55.614497+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-06-04T12:58:51.617998+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,0],"acting":[4,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"2.3","version":"0'0","reported_seq":948,"reported_epoch":1014,"state":"active+clean","last_fresh":"2024-06-03T09:06:46.130092+0000","last_change":"2024-06-03T08:36:30.351670+0000","last_active":"2024-06-03T09:06:46.130092+0000","last_peered":"2024-06-03T09:06:46.130092+0000","last_clean":"2024-06-03T09:06:46.130092+0000","last_became_active":"2024-06-03T08:36:30.343560+0000","last_became_peered":"2024-06-03T08:36:30.343560+0000","last_unstale":"2024-06-03T09:06:46.130092+0000","last_undegraded":"2024-06-03T09:06:46.130092+0000","last_fullsized":"2024-06-03T09:06:46.130092+0000","mapping_epoch":23,"log_start":"0'0","ondisk_log_start":"0'0","created":23,"last_epoch_clean":24,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-06-03T08:36:28.444764+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-06-03T08:36:28.444764+0000","last_clean_scrub_stamp":"2024-06-03T08:36:28.444764+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for deep scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,2],"acting":[3,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.0","version":"1014'78","reported_seq":1070,"reported_epoch":1014,"state":"active+clean","last_fresh":"2024-06-03T09:06:57.887708+0000","last_change":"2024-06-03T09:06:57.889997+0000","last_active":"2024-06-03T09:06:57.887708+0000","last_peered":"2024-06-03T09:06:57.887708+0000","last_clean":"2024-06-03T09:06:57.887708+0000","last_became_active":"2024-06-03T08:36:26.128378+0000","last_became_peered":"2024-06-03T08:36:26.128378+0000","last_unstale":"2024-06-03T09:06:57.887708+0000","last_undegraded":"2024-06-03T09:06:57.887708+0000","last_fullsized":"2024-06-03T09:06:57.887708+0000","mapping_epoch":20,"log_start":"0'0","ondisk_log_start":"0'0","created":16,"last_epoch_clean":22,"parent":"0.0","parent_split_bits":0,"last_scrub":"1014'78","last_scrub_stamp":"2024-06-03T09:06:57.886322+0000","last_deep_scrub":"1014'78","last_deep_scrub_stamp":"2024-06-03T09:06:57.886322+0000","last_clean_scrub_stamp":"2024-06-03T09:06:57.886322+0000","objects_scrubbed":2,"log_size":78,"log_dups_size":0,"ondisk_log_size":78,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-06-04T19:15:53.804903+0000","scrub_duration":0.098050433000000006,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":459280,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":96,"num_read_kb":82,"num_write":113,"num_write_kb":1372,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2,"num_bytes_recovered":397840,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,5],"acting":[1,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"17.10","version":"0'0","reported_seq":947,"reported_epoch":1013,"state":"active+clean","last_fresh":"2024-06-03T09:06:44.974251+0000","last_change":"2024-06-03T08:38:15.399189+0000","last_active":"2024-06-03T09:06:44.974251+0000","last_peered":"2024-06-03T09:06:44.974251+0000","last_clean":"2024-06-03T09:06:44.974251+0000","last_became_active":"2024-06-03T08:38:15.337723+0000","last_became_peered":"2024-06-03T08:38:15.337723+0000","last_unstale":"2024-06-03T09:06:44.974251+0000","last_undegraded":"2024-06-03T09:06:44.974251+0000","last_fullsized":"2024-06-03T09:06:44.974251+0000","mapping_epoch":28,"log_start":"0'0","ondisk_log_start":"0'0","created":28,"last_epoch_clean":42,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-06-03T08:37:55.614497+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-06-03T08:37:55.614497+0000","last_clean_scrub_stamp":"2024-06-03T08:37:55.614497+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-06-04T09:29:46.427317+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,3],"acting":[4,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"17.1a","version":"0'0","reported_seq":971,"reported_epoch":1014,"state":"active+clean","last_fresh":"2024-06-03T09:06:46.393342+0000","last_change":"2024-06-03T09:06:46.393342+0000","last_active":"2024-06-03T09:06:46.393342+0000","last_peered":"2024-06-03T09:06:46.393342+0000","last_clean":"2024-06-03T09:06:46.393342+0000","last_became_active":"2024-06-03T09:06:46.386469+0000","last_became_peered":"2024-06-03T09:06:46.386469+0000","last_unstale":"2024-06-03T09:06:46.393342+0000","last_undegraded":"2024-06-03T09:06:46.393342+0000","last_fullsized":"2024-06-03T09:06:46.393342+0000","mapping_epoch":1013,"log_start":"0'0","ondisk_log_start":"0'0","created":28,"last_epoch_clean":1014,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-06-03T08:37:55.614497+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-06-03T08:37:55.614497+0000","last_clean_scrub_stamp":"2024-06-03T08:37:55.614497+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-06-04T13:22:45.876282+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,6],"acting":[4,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"17.1b","version":"0'0","reported_seq":940,"reported_epoch":1014,"state":"active+clean","last_fresh":"2024-06-03T09:06:46.128510+0000","last_change":"2024-06-03T08:38:03.115748+0000","last_active":"2024-06-03T09:06:46.128510+0000","last_peered":"2024-06-03T09:06:46.128510+0000","last_clean":"2024-06-03T09:06:46.128510+0000","last_became_active":"2024-06-03T08:38:03.110533+0000","last_became_peered":"2024-06-03T08:38:03.110533+0000","last_unstale":"2024-06-03T09:06:46.128510+0000","last_undegraded":"2024-06-03T09:06:46.128510+0000","last_fullsized":"2024-06-03T09:06:46.128510+0000","mapping_epoch":28,"log_start":"0'0","ondisk_log_start":"0'0","created":28,"last_epoch_clean":32,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-06-03T08:37:55.614497+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-06-03T08:37:55.614497+0000","last_clean_scrub_stamp":"2024-06-03T08:37:55.614497+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for deep scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,7],"acting":[3,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"17.18","version":"0'0","reported_seq":947,"reported_epoch":1013,"state":"active+clean","last_fresh":"2024-06-03T09:06:45.420817+0000","last_change":"2024-06-03T08:38:04.632864+0000","last_active":"2024-06-03T09:06:45.420817+0000","last_peered":"2024-06-03T09:06:45.420817+0000","last_clean":"2024-06-03T09:06:45.420817+0000","last_became_active":"2024-06-03T08:38:04.627856+0000","last_became_peered":"2024-06-03T08:38:04.627856+0000","last_unstale":"2024-06-03T09:06:45.420817+0000","last_undegraded":"2024-06-03T09:06:45.420817+0000","last_fullsized":"2024-06-03T09:06:45.420817+0000","mapping_epoch":28,"log_start":"0'0","ondisk_log_start":"0'0","created":28,"last_epoch_clean":33,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-06-03T08:37:55.614497+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-06-03T08:37:55.614497+0000","last_clean_scrub_stamp":"2024-06-03T08:37:55.614497+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-06-04T19:55:51.181598+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,4],"acting":[7,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"17.19","version":"0'0","reported_seq":939,"reported_epoch":1014,"state":"active+clean","last_fresh":"2024-06-03T09:06:46.121629+0000","last_change":"2024-06-03T08:38:01.839491+0000","last_active":"2024-06-03T09:06:46.121629+0000","last_peered":"2024-06-03T09:06:46.121629+0000","last_clean":"2024-06-03T09:06:46.121629+0000","last_became_active":"2024-06-03T08:38:01.829539+0000","last_became_peered":"2024-06-03T08:38:01.829539+0000","last_unstale":"2024-06-03T09:06:46.121629+0000","last_undegraded":"2024-06-03T09:06:46.121629+0000","last_fullsized":"2024-06-03T09:06:46.121629+0000","mapping_epoch":28,"log_start":"0'0","ondisk_log_start":"0'0","created":28,"last_epoch_clean":30,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-06-03T08:37:55.614497+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-06-03T08:37:55.614497+0000","last_clean_scrub_stamp":"2024-06-03T08:37:55.614497+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for deep scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,2],"acting":[3,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"17.1e","version":"0'0","reported_seq":844,"reported_epoch":1014,"state":"active+clean","last_fresh":"2024-06-03T09:06:46.297025+0000","last_change":"2024-06-03T09:06:46.297025+0000","last_active":"2024-06-03T09:06:46.297025+0000","last_peered":"2024-06-03T09:06:46.297025+0000","last_clean":"2024-06-03T09:06:46.297025+0000","last_became_active":"2024-06-03T09:06:46.288593+0000","last_became_peered":"2024-06-03T09:06:46.288593+0000","last_unstale":"2024-06-03T09:06:46.297025+0000","last_undegraded":"2024-06-03T09:06:46.297025+0000","last_fullsized":"2024-06-03T09:06:46.297025+0000","mapping_epoch":1013,"log_start":"0'0","ondisk_log_start":"0'0","created":28,"last_epoch_clean":1014,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-06-03T08:37:55.614497+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-06-03T08:37:55.614497+0000","last_clean_scrub_stamp":"2024-06-03T08:37:55.614497+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-06-04T20:09:30.144366+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,4],"acting":[6,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"17.1f","version":"0'0","reported_seq":946,"reported_epoch":1014,"state":"active+clean","last_fresh":"2024-06-03T09:06:57.003482+0000","last_change":"2024-06-03T08:38:04.383538+0000","last_active":"2024-06-03T09:06:57.003482+0000","last_peered":"2024-06-03T09:06:57.003482+0000","last_clean":"2024-06-03T09:06:57.003482+0000","last_became_active":"2024-06-03T08:38:04.367278+0000","last_became_peered":"2024-06-03T08:38:04.367278+0000","last_unstale":"2024-06-03T09:06:57.003482+0000","last_undegraded":"2024-06-03T09:06:57.003482+0000","last_fullsized":"2024-06-03T09:06:57.003482+0000","mapping_epoch":28,"log_start":"0'0","ondisk_log_start":"0'0","created":28,"last_epoch_clean":32,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-06-03T08:37:55.614497+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-06-03T08:37:55.614497+0000","last_clean_scrub_stamp":"2024-06-03T08:37:55.614497+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for deep scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,2],"acting":[0,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"17.1c","version":"0'0","reported_seq":957,"reported_epoch":1014,"state":"active+clean","last_fresh":"2024-06-03T09:06:46.419841+0000","last_change":"2024-06-03T09:06:46.410116+0000","last_active":"2024-06-03T09:06:46.419841+0000","last_peered":"2024-06-03T09:06:46.419841+0000","last_clean":"2024-06-03T09:06:46.419841+0000","last_became_active":"2024-06-03T09:06:46.403724+0000","last_became_peered":"2024-06-03T09:06:46.403724+0000","last_unstale":"2024-06-03T09:06:46.419841+0000","last_undegraded":"2024-06-03T09:06:46.419841+0000","last_fullsized":"2024-06-03T09:06:46.419841+0000","mapping_epoch":1013,"log_start":"0'0","ondisk_log_start":"0'0","created":28,"last_epoch_clean":1014,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-06-03T08:37:55.614497+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-06-03T08:37:55.614497+0000","last_clean_scrub_stamp":"2024-06-03T08:37:55.614497+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for deep scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,6],"acting":[5,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"17.1d","version":"0'0","reported_seq":928,"reported_epoch":1014,"state":"active+clean","last_fresh":"2024-06-03T09:06:46.468532+0000","last_change":"2024-06-03T08:38:06.794630+0000","last_active":"2024-06-03T09:06:46.468532+0000","last_peered":"2024-06-03T09:06:46.468532+0000","last_clean":"2024-06-03T09:06:46.468532+0000","last_became_active":"2024-06-03T08:38:06.592317+0000","last_became_peered":"2024-06-03T08:38:06.592317+0000","last_unstale":"2024-06-03T09:06:46.468532+0000","last_undegraded":"2024-06-03T09:06:46.468532+0000","last_fullsized":"2024-06-03T09:06:46.468532+0000","mapping_epoch":28,"log_start":"0'0","ondisk_log_start":"0'0","created":28,"last_epoch_clean":33,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-06-03T08:37:55.614497+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-06-03T08:37:55.614497+0000","last_clean_scrub_stamp":"2024-06-03T08:37:55.614497+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for deep scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,7],"acting":[1,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"17.2","version":"0'0","reported_seq":930,"reported_epoch":1014,"state":"active+clean","last_fresh":"2024-06-03T09:06:46.471827+0000","last_change":"2024-06-03T08:38:14.981410+0000","last_active":"2024-06-03T09:06:46.471827+0000","last_peered":"2024-06-03T09:06:46.471827+0000","last_clean":"2024-06-03T09:06:46.471827+0000","last_became_active":"2024-06-03T08:38:14.973825+0000","last_became_peered":"2024-06-03T08:38:14.973825+0000","last_unstale":"2024-06-03T09:06:46.471827+0000","last_undegraded":"2024-06-03T09:06:46.471827+0000","last_fullsized":"2024-06-03T09:06:46.471827+0000","mapping_epoch":28,"log_start":"0'0","ondisk_log_start":"0'0","created":28,"last_epoch_clean":41,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-06-03T08:37:55.614497+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-06-03T08:37:55.614497+0000","last_clean_scrub_stamp":"2024-06-03T08:37:55.614497+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for deep scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,4],"acting":[1,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"17.3","version":"0'0","reported_seq":947,"reported_epoch":1014,"state":"active+clean","last_fresh":"2024-06-03T09:06:46.134488+0000","last_change":"2024-06-03T08:38:14.619745+0000","last_active":"2024-06-03T09:06:46.134488+0000","last_peered":"2024-06-03T09:06:46.134488+0000","last_clean":"2024-06-03T09:06:46.134488+0000","last_became_active":"2024-06-03T08:38:14.510665+0000","last_became_peered":"2024-06-03T08:38:14.510665+0000","last_unstale":"2024-06-03T09:06:46.134488+0000","last_undegraded":"2024-06-03T09:06:46.134488+0000","last_fullsized":"2024-06-03T09:06:46.134488+0000","mapping_epoch":28,"log_start":"0'0","ondisk_log_start":"0'0","created":28,"last_epoch_clean":42,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-06-03T08:37:55.614497+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-06-03T08:37:55.614497+0000","last_clean_scrub_stamp":"2024-06-03T08:37:55.614497+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for deep scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,7],"acting":[0,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"17.0","version":"0'0","reported_seq":948,"reported_epoch":1014,"state":"active+clean","last_fresh":"2024-06-03T09:06:46.120310+0000","last_change":"2024-06-03T08:38:04.237118+0000","last_active":"2024-06-03T09:06:46.120310+0000","last_peered":"2024-06-03T09:06:46.120310+0000","last_clean":"2024-06-03T09:06:46.120310+0000","last_became_active":"2024-06-03T08:38:04.231954+0000","last_became_peered":"2024-06-03T08:38:04.231954+0000","last_unstale":"2024-06-03T09:06:46.120310+0000","last_undegraded":"2024-06-03T09:06:46.120310+0000","last_fullsized":"2024-06-03T09:06:46.120310+0000","mapping_epoch":28,"log_start":"0'0","ondisk_log_start":"0'0","created":28,"last_epoch_clean":32,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-06-03T08:37:55.614497+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-06-03T08:37:55.614497+0000","last_clean_scrub_stamp":"2024-06-03T08:37:55.614497+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for deep scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,2],"acting":[0,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"17.1","version":"65'7","reported_seq":972,"reported_epoch":1014,"state":"active+clean","last_fresh":"2024-06-03T09:06:46.418975+0000","last_change":"2024-06-03T09:06:46.389760+0000","last_active":"2024-06-03T09:06:46.418975+0000","last_peered":"2024-06-03T09:06:46.418975+0000","last_clean":"2024-06-03T09:06:46.418975+0000","last_became_active":"2024-06-03T09:06:46.383290+0000","last_became_peered":"2024-06-03T09:06:46.383290+0000","last_unstale":"2024-06-03T09:06:46.418975+0000","last_undegraded":"2024-06-03T09:06:46.418975+0000","last_fullsized":"2024-06-03T09:06:46.418975+0000","mapping_epoch":1013,"log_start":"0'0","ondisk_log_start":"0'0","created":28,"last_epoch_clean":1014,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-06-03T08:37:55.614497+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-06-03T08:37:55.614497+0000","last_clean_scrub_stamp":"2024-06-03T08:37:55.614497+0000","objects_scrubbed":0,"log_size":7,"log_dups_size":0,"ondisk_log_size":7,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for deep scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":5,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":6,"num_read_kb":6,"num_write":7,"num_write_kb":7,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,6],"acting":[5,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]}],"pool_stats":[{"poolid":2,"num_pg":8,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":16,"acting":16,"num_store_stats":7},{"poolid":1,"num_pg":1,"stat_sum":{"num_bytes":459280,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":96,"num_read_kb":82,"num_write":113,"num_write_kb":1372,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2,"num_bytes_recovered":397840,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":802816,"data_stored":795680,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":78,"ondisk_log_size":78,"up":2,"acting":2,"num_store_stats":2},{"poolid":17,"num_pg":32,"stat_sum":{"num_bytes":5,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":6,"num_read_kb":6,"num_write":7,"num_write_kb":7,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":8192,"data_stored":10,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":7,"ondisk_log_size":7,"up":64,"acting":64,"num_store_stats":8}],"osd_stats":[{"osd":6,"up_from":1013,"seq":4350801871165,"num_pgs":14,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":54916,"kb_used_data":5756,"kb_used_omap":1,"kb_used_meta":49150,"kb_avail":94316924,"statfs":{"total":96636764160,"available":96580530176,"internally_reserved":0,"allocated":5894144,"data_stored":2763078,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1593,"internal_metadata":50330055},"hb_peers":[0,1,2,3,4,5,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":3,"apply_latency_ms":3,"commit_latency_ns":3000000,"apply_latency_ns":3000000},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Mon Jun 3 09:06:42 2024","interfaces":[{"interface":"back","average":{"1min":5696.3109999999997,"5min":1469.3900000000001,"15min":503.11700000000002},"min":{"1min":5.242,"5min":4.4829999999999997,"15min":4.1959999999999997},"max":{"1min":20463.029999999999,"5min":22766.882000000001,"15min":22766.882000000001},"last":9.6579999999999995},{"interface":"front","average":{"1min":5687.6390000000001,"5min":1468.1669999999999,"15min":504.13200000000001},"min":{"1min":4.835,"5min":4.8129999999999997,"15min":4.4530000000000003},"max":{"1min":20461.830000000002,"5min":22766.289000000001,"15min":22766.289000000001},"last":5.1559999999999997}]},{"osd":1,"last update":"Mon Jun 3 09:06:42 2024","interfaces":[{"interface":"back","average":{"1min":36.496000000000002,"5min":80.162999999999997,"15min":134.59399999999999},"min":{"1min":5.9189999999999996,"5min":5.8799999999999999,"15min":5.7119999999999997},"max":{"1min":544.85799999999995,"5min":4138.1679999999997,"15min":9955.1299999999992},"last":6.0119999999999996},{"interface":"front","average":{"1min":38.866999999999997,"5min":82.631,"15min":135.11500000000001},"min":{"1min":6.5350000000000001,"5min":6.5350000000000001,"15min":6.1390000000000002},"max":{"1min":546.73099999999999,"5min":4137.6530000000002,"15min":9827.5509999999995},"last":11.869}]},{"osd":2,"last update":"Mon Jun 3 09:06:42 2024","interfaces":[{"interface":"back","average":{"1min":41.093000000000004,"5min":27.917000000000002,"15min":22.329000000000001},"min":{"1min":6.0670000000000002,"5min":6.0670000000000002,"15min":6.0670000000000002},"max":{"1min":548.12900000000002,"5min":548.12900000000002,"15min":548.12900000000002},"last":8.3450000000000006},{"interface":"front","average":{"1min":30.798999999999999,"5min":24.276,"15min":22.488},"min":{"1min":6.4370000000000003,"5min":6.4370000000000003,"15min":6.1870000000000003},"max":{"1min":369.14499999999998,"5min":369.14499999999998,"15min":434.77300000000002},"last":14.374000000000001}]},{"osd":3,"last update":"Mon Jun 3 09:06:42 2024","interfaces":[{"interface":"back","average":{"1min":41.411999999999999,"5min":27.434999999999999,"15min":30.657},"min":{"1min":10.061,"5min":8.3179999999999996,"15min":6.1719999999999997},"max":{"1min":549.024,"5min":549.024,"15min":1235.5799999999999},"last":12.497999999999999},{"interface":"front","average":{"1min":37.850999999999999,"5min":22.001000000000001,"15min":28.015999999999998},"min":{"1min":6.0679999999999996,"5min":6.0679999999999996,"15min":6.0679999999999996},"max":{"1min":559.25,"5min":559.25,"15min":1157.6780000000001},"last":13.395}]},{"osd":4,"last update":"Mon Jun 3 09:06:42 2024","interfaces":[{"interface":"back","average":{"1min":165.62899999999999,"5min":1281.1010000000001,"15min":440.59199999999998},"min":{"1min":10.965999999999999,"5min":6.093,"15min":6.093},"max":{"1min":384.46899999999999,"5min":28093.096000000001,"15min":28093.096000000001},"last":16.952999999999999},{"interface":"front","average":{"1min":186.97900000000001,"5min":1308.886,"15min":454.97699999999998},"min":{"1min":13.581,"5min":7.7480000000000002,"15min":7.7480000000000002},"max":{"1min":445.096,"5min":28323.837,"15min":28323.837},"last":16.318999999999999}]},{"osd":5,"last update":"Mon Jun 3 09:06:42 2024","interfaces":[{"interface":"back","average":{"1min":81.581000000000003,"5min":34.863,"15min":28.350999999999999},"min":{"1min":10.345000000000001,"5min":8.625,"15min":7.516},"max":{"1min":455.84199999999998,"5min":455.84199999999998,"15min":455.84199999999998},"last":14.914999999999999},{"interface":"front","average":{"1min":83.391000000000005,"5min":39.277999999999999,"15min":27.885000000000002},"min":{"1min":8.0280000000000005,"5min":6.266,"15min":6.266},"max":{"1min":550.02300000000002,"5min":550.02300000000002,"15min":550.02300000000002},"last":17.957000000000001}]},{"osd":7,"last update":"Mon Jun 3 09:06:42 2024","interfaces":[{"interface":"back","average":{"1min":44.052999999999997,"5min":33.697000000000003,"15min":29.731999999999999},"min":{"1min":9.1500000000000004,"5min":9.1500000000000004,"15min":9.1500000000000004},"max":{"1min":552.09500000000003,"5min":552.09500000000003,"15min":552.09500000000003},"last":15.795},{"interface":"front","average":{"1min":47.076999999999998,"5min":34.226999999999997,"15min":29.827000000000002},"min":{"1min":9.7780000000000005,"5min":8.6709999999999994,"15min":8.5820000000000007},"max":{"1min":531.08600000000001,"5min":531.08600000000001,"15min":531.08600000000001},"last":10.949999999999999}]}]},{"osd":7,"up_from":20,"seq":85899346293,"num_pgs":9,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":59020,"kb_used_data":5752,"kb_used_omap":1,"kb_used_meta":53246,"kb_avail":94312820,"statfs":{"total":96636764160,"available":96576327680,"internally_reserved":0,"allocated":5890048,"data_stored":2763073,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1589,"internal_metadata":54524363},"hb_peers":[0,1,2,3,4,5,6],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Mon Jun 3 09:06:09 2024","interfaces":[{"interface":"back","average":{"1min":14.073,"5min":894.22400000000005,"15min":308.93400000000003},"min":{"1min":5.5709999999999997,"5min":4.218,"15min":4.0599999999999996},"max":{"1min":90.757999999999996,"5min":21729.001,"15min":21729.001},"last":6.9100000000000001},{"interface":"front","average":{"1min":9.2550000000000008,"5min":906.23500000000001,"15min":314.06},"min":{"1min":5.5789999999999997,"5min":4.8810000000000002,"15min":4.8810000000000002},"max":{"1min":25.937000000000001,"5min":21899.732,"15min":21899.732},"last":5.7759999999999998}]},{"osd":1,"last update":"Mon Jun 3 09:06:09 2024","interfaces":[{"interface":"back","average":{"1min":15.182,"5min":69.183999999999997,"15min":219.173},"min":{"1min":7.3639999999999999,"5min":6.6200000000000001,"15min":6.1319999999999997},"max":{"1min":76.926000000000002,"5min":2795.1329999999998,"15min":18373.257000000001},"last":8.7750000000000004},{"interface":"front","average":{"1min":21.495999999999999,"5min":74.703999999999994,"15min":225.285},"min":{"1min":8.8399999999999999,"5min":8.8399999999999999,"15min":8.5129999999999999},"max":{"1min":94.977999999999994,"5min":2803.3670000000002,"15min":18374.601999999999},"last":12.195}]},{"osd":2,"last update":"Mon Jun 3 09:06:09 2024","interfaces":[{"interface":"back","average":{"1min":19.164000000000001,"5min":26.902000000000001,"15min":22.920999999999999},"min":{"1min":11.207000000000001,"5min":8.5030000000000001,"15min":8.5030000000000001},"max":{"1min":91.319999999999993,"5min":333.99200000000002,"15min":333.99200000000002},"last":10.461},{"interface":"front","average":{"1min":11.125999999999999,"5min":18.132000000000001,"15min":16.353000000000002},"min":{"1min":6.202,"5min":5.8369999999999997,"15min":5.5069999999999997},"max":{"1min":28.649000000000001,"5min":226.18100000000001,"15min":226.18100000000001},"last":7.7720000000000002}]},{"osd":3,"last update":"Mon Jun 3 09:06:09 2024","interfaces":[{"interface":"back","average":{"1min":22.291,"5min":26.289999999999999,"15min":31.074999999999999},"min":{"1min":11.714,"5min":6.8959999999999999,"15min":6.3760000000000003},"max":{"1min":83.102000000000004,"5min":316.64400000000001,"15min":2039.9059999999999},"last":12.754},{"interface":"front","average":{"1min":13.324999999999999,"5min":17.963000000000001,"15min":25.969999999999999},"min":{"1min":6.1200000000000001,"5min":5.2089999999999996,"15min":5.2089999999999996},"max":{"1min":49.908999999999999,"5min":233.542,"15min":2037.5550000000001},"last":13.276999999999999}]},{"osd":4,"last update":"Mon Jun 3 09:06:29 2024","interfaces":[{"interface":"back","average":{"1min":17.675999999999998,"5min":1026.325,"15min":472.47199999999998},"min":{"1min":7.306,"5min":5.9089999999999998,"15min":5.9089999999999998},"max":{"1min":140.03299999999999,"5min":28969.369999999999,"15min":30073.917000000001},"last":13.792},{"interface":"front","average":{"1min":17.890999999999998,"5min":1044.5139999999999,"15min":480.226},"min":{"1min":5.9279999999999999,"5min":5.9279999999999999,"15min":5.9279999999999999},"max":{"1min":141.179,"5min":29244.138999999999,"15min":30348.021000000001},"last":11.130000000000001}]},{"osd":5,"last update":"Mon Jun 3 09:06:02 2024","interfaces":[{"interface":"back","average":{"1min":24.372,"5min":26.856000000000002,"15min":23.571999999999999},"min":{"1min":12.278,"5min":9.2460000000000004,"15min":6.5650000000000004},"max":{"1min":183.64099999999999,"5min":317.572,"15min":317.572},"last":16.481000000000002},{"interface":"front","average":{"1min":21.259,"5min":23.364000000000001,"15min":21.193999999999999},"min":{"1min":7.8499999999999996,"5min":7.0999999999999996,"15min":7.008},"max":{"1min":184.53399999999999,"5min":306.60599999999999,"15min":306.60599999999999},"last":9.8949999999999996}]},{"osd":6,"last update":"Mon Jun 3 09:04:31 2024","interfaces":[{"interface":"back","average":{"1min":44.061,"5min":26.821000000000002,"15min":91.176000000000002},"min":{"1min":9.0800000000000001,"5min":7.9390000000000001,"15min":7.9390000000000001},"max":{"1min":268.029,"5min":308.916,"15min":8734.4639999999999},"last":14.321999999999999},{"interface":"front","average":{"1min":46.643000000000001,"5min":23.879999999999999,"15min":88.765000000000001},"min":{"1min":8.8119999999999994,"5min":6.9240000000000004,"15min":6.742},"max":{"1min":305.64600000000002,"5min":305.64600000000002,"15min":8735.5319999999992},"last":15.736000000000001}]}]},{"osd":5,"up_from":20,"seq":85899346268,"num_pgs":9,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":59404,"kb_used_data":6148,"kb_used_omap":1,"kb_used_meta":53246,"kb_avail":94312436,"statfs":{"total":96636764160,"available":96575934464,"internally_reserved":0,"allocated":6295552,"data_stored":3160918,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1594,"internal_metadata":54524358},"hb_peers":[0,1,2,3,4,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":3,"apply_latency_ms":3,"commit_latency_ns":3000000,"apply_latency_ns":3000000},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Mon Jun 3 09:06:06 2024","interfaces":[{"interface":"back","average":{"1min":7.468,"5min":1703.037,"15min":580.51900000000001},"min":{"1min":4.6509999999999998,"5min":4.6509999999999998,"15min":4.4020000000000001},"max":{"1min":14.711,"5min":26433.764999999999,"15min":26433.764999999999},"last":5.2629999999999999},{"interface":"front","average":{"1min":9.9429999999999996,"5min":1725.27,"15min":587.87800000000004},"min":{"1min":5.5999999999999996,"5min":5.1319999999999997,"15min":4.8170000000000002},"max":{"1min":13.590999999999999,"5min":26704.698,"15min":26704.698},"last":11.627000000000001}]},{"osd":1,"last update":"Mon Jun 3 09:07:01 2024","interfaces":[{"interface":"back","average":{"1min":11.929,"5min":66.978999999999999,"15min":216.91999999999999},"min":{"1min":5.9790000000000001,"5min":5.9790000000000001,"15min":5.4930000000000003},"max":{"1min":22.202999999999999,"5min":3923.4090000000001,"15min":18811.696},"last":12.859999999999999},{"interface":"front","average":{"1min":11.496,"5min":64.528000000000006,"15min":220.06100000000001},"min":{"1min":6.298,"5min":5.3209999999999997,"15min":5.0209999999999999},"max":{"1min":27.747,"5min":4019.6129999999998,"15min":18968.365000000002},"last":8.0459999999999994}]},{"osd":2,"last update":"Mon Jun 3 09:07:01 2024","interfaces":[{"interface":"back","average":{"1min":13.903,"5min":24.381,"15min":20.638999999999999},"min":{"1min":9.3559999999999999,"5min":5.6369999999999996,"15min":5.0510000000000002},"max":{"1min":27.013999999999999,"5min":332.48200000000003,"15min":332.48200000000003},"last":13.73},{"interface":"front","average":{"1min":9.2989999999999995,"5min":22.521999999999998,"15min":23.033999999999999},"min":{"1min":6.2439999999999998,"5min":6.2439999999999998,"15min":6.2439999999999998},"max":{"1min":18.395,"5min":353.03199999999998,"15min":353.03199999999998},"last":9.7780000000000005}]},{"osd":3,"last update":"Mon Jun 3 09:07:01 2024","interfaces":[{"interface":"back","average":{"1min":14.686999999999999,"5min":24.370999999999999,"15min":31.327000000000002},"min":{"1min":8.1140000000000008,"5min":7.4720000000000004,"15min":7.4210000000000003},"max":{"1min":30.178000000000001,"5min":353.57499999999999,"15min":2855.8519999999999},"last":14.49},{"interface":"front","average":{"1min":11.590999999999999,"5min":16.498999999999999,"15min":25.437999999999999},"min":{"1min":6.3769999999999998,"5min":5.9660000000000002,"15min":5.1909999999999998},"max":{"1min":22.972000000000001,"5min":203.804,"15min":2869.7860000000001},"last":10.446}]},{"osd":4,"last update":"Mon Jun 3 09:06:29 2024","interfaces":[{"interface":"back","average":{"1min":11.067,"5min":780.21000000000004,"15min":377.28399999999999},"min":{"1min":5.4779999999999998,"5min":5.4779999999999998,"15min":5.4779999999999998},"max":{"1min":14.654999999999999,"5min":23315.438999999998,"15min":28619.825000000001},"last":15.369},{"interface":"front","average":{"1min":14.349,"5min":782.97299999999996,"15min":377.93400000000003},"min":{"1min":7.492,"5min":6.5030000000000001,"15min":6.5030000000000001},"max":{"1min":17.707000000000001,"5min":23317.502,"15min":28621.294000000002},"last":23.309000000000001}]},{"osd":6,"last update":"Mon Jun 3 09:03:57 2024","interfaces":[{"interface":"back","average":{"1min":23.536000000000001,"5min":36.457000000000001,"15min":83.799999999999997},"min":{"1min":9.4800000000000004,"5min":9.4800000000000004,"15min":8.0630000000000006},"max":{"1min":89.442999999999998,"5min":508.77999999999997,"15min":9193.5169999999998},"last":10.976000000000001},{"interface":"front","average":{"1min":18.891999999999999,"5min":22.100999999999999,"15min":75.489999999999995},"min":{"1min":5.923,"5min":5.923,"15min":5.923},"max":{"1min":78.424999999999997,"5min":464.62,"15min":9191.4429999999993},"last":16.289000000000001}]},{"osd":7,"last update":"Mon Jun 3 09:07:01 2024","interfaces":[{"interface":"back","average":{"1min":15.648999999999999,"5min":21.43,"15min":20.760999999999999},"min":{"1min":8.6319999999999997,"5min":7.4020000000000001,"15min":6.7629999999999999},"max":{"1min":28.895,"5min":211.22999999999999,"15min":211.22999999999999},"last":8.6319999999999997},{"interface":"front","average":{"1min":13.545999999999999,"5min":20.359000000000002,"15min":23.413},"min":{"1min":6.8239999999999998,"5min":6.8239999999999998,"15min":6.8239999999999998},"max":{"1min":36.515000000000001,"5min":223.191,"15min":286.69499999999999},"last":7.0800000000000001}]}]},{"osd":4,"up_from":20,"seq":85899346275,"num_pgs":12,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":63112,"kb_used_data":5752,"kb_used_omap":1,"kb_used_meta":57342,"kb_avail":94308728,"statfs":{"total":96636764160,"available":96572137472,"internally_reserved":0,"allocated":5890048,"data_stored":2763073,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1594,"internal_metadata":58718662},"hb_peers":[0,1,2,3,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Mon Jun 3 09:06:35 2024","interfaces":[{"interface":"back","average":{"1min":10.417,"5min":1093.547,"15min":381.64499999999998},"min":{"1min":4.6900000000000004,"5min":4.5279999999999996,"15min":4.3220000000000001},"max":{"1min":52.241,"5min":21726.743999999999,"15min":21726.743999999999},"last":8.3840000000000003},{"interface":"front","average":{"1min":13.175000000000001,"5min":1102.4200000000001,"15min":387.161},"min":{"1min":6.1950000000000003,"5min":5.8380000000000001,"15min":5.4909999999999997},"max":{"1min":53.148000000000003,"5min":21727.993999999999,"15min":21727.993999999999},"last":11.837999999999999}]},{"osd":1,"last update":"Mon Jun 3 09:06:35 2024","interfaces":[{"interface":"back","average":{"1min":9.9909999999999997,"5min":53.250999999999998,"15min":340.02999999999997},"min":{"1min":4.5670000000000002,"5min":4.1970000000000001,"15min":4.1970000000000001},"max":{"1min":47.073,"5min":3346.4389999999999,"15min":22256.098000000002},"last":4.9359999999999999},{"interface":"front","average":{"1min":12.189,"5min":53.375999999999998,"15min":339.68799999999999},"min":{"1min":6.1539999999999999,"5min":5.0590000000000002,"15min":4.2640000000000002},"max":{"1min":49.558,"5min":3347.181,"15min":22247.317999999999},"last":9.9570000000000007}]},{"osd":2,"last update":"Mon Jun 3 09:06:35 2024","interfaces":[{"interface":"back","average":{"1min":15.178000000000001,"5min":17.202000000000002,"15min":20.417999999999999},"min":{"1min":6.3010000000000002,"5min":6.3010000000000002,"15min":6.3010000000000002},"max":{"1min":85.340000000000003,"5min":337.60399999999998,"15min":393.58600000000001},"last":9.3230000000000004},{"interface":"front","average":{"1min":15.423,"5min":14.432,"15min":22.754000000000001},"min":{"1min":6.4889999999999999,"5min":6.0629999999999997,"15min":6.0629999999999997},"max":{"1min":82.120000000000005,"5min":127.913,"15min":394.471},"last":12.503}]},{"osd":3,"last update":"Mon Jun 3 09:06:35 2024","interfaces":[{"interface":"back","average":{"1min":17.651,"5min":15.5,"15min":18.079000000000001},"min":{"1min":10.359999999999999,"5min":5.8719999999999999,"15min":5.4000000000000004},"max":{"1min":84.555999999999997,"5min":127.43600000000001,"15min":322.63},"last":13.558},{"interface":"front","average":{"1min":16.413,"5min":17.356999999999999,"15min":22.920000000000002},"min":{"1min":9.7349999999999994,"5min":7.484,"15min":7.484},"max":{"1min":71.394000000000005,"5min":204.517,"15min":392.67500000000001},"last":11.318}]},{"osd":5,"last update":"Mon Jun 3 09:06:35 2024","interfaces":[{"interface":"back","average":{"1min":15.672000000000001,"5min":16.742999999999999,"15min":20.795000000000002},"min":{"1min":8.093,"5min":6.7489999999999997,"15min":6.7489999999999997},"max":{"1min":51.326000000000001,"5min":268.10199999999998,"15min":344.23399999999998},"last":14.734999999999999},{"interface":"front","average":{"1min":17.297999999999998,"5min":19.356999999999999,"15min":25.713999999999999},"min":{"1min":12.875,"5min":8.5180000000000007,"15min":8.5180000000000007},"max":{"1min":40.829000000000001,"5min":264.75,"15min":345.18700000000001},"last":17.323}]},{"osd":6,"last update":"Mon Jun 3 09:04:32 2024","interfaces":[{"interface":"back","average":{"1min":15.548,"5min":32.957999999999998,"15min":152.738},"min":{"1min":9.141,"5min":9.141,"15min":8.2319999999999993},"max":{"1min":52.57,"5min":395.238,"15min":12496.335999999999},"last":10.609999999999999},{"interface":"front","average":{"1min":12.680999999999999,"5min":25.428000000000001,"15min":149.30600000000001},"min":{"1min":6.4379999999999997,"5min":5.6319999999999997,"15min":5.6319999999999997},"max":{"1min":46.283000000000001,"5min":377.74299999999999,"15min":12616.297},"last":15.973000000000001}]},{"osd":7,"last update":"Mon Jun 3 09:06:35 2024","interfaces":[{"interface":"back","average":{"1min":9.6969999999999992,"5min":14.196,"15min":18.809999999999999},"min":{"1min":6.7190000000000003,"5min":6.2110000000000003,"15min":5.8010000000000002},"max":{"1min":15.960000000000001,"5min":207.25299999999999,"15min":358.94900000000001},"last":14.101000000000001},{"interface":"front","average":{"1min":17.696000000000002,"5min":19.024999999999999,"15min":23.259},"min":{"1min":11.794,"5min":7.7789999999999999,"15min":7.7789999999999999},"max":{"1min":45.639000000000003,"5min":209.32900000000001,"15min":380.06900000000002},"last":16.663}]}]},{"osd":2,"up_from":20,"seq":85899346295,"num_pgs":11,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":59020,"kb_used_data":5752,"kb_used_omap":1,"kb_used_meta":53246,"kb_avail":94312820,"statfs":{"total":96636764160,"available":96576327680,"internally_reserved":0,"allocated":5890048,"data_stored":2763073,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1594,"internal_metadata":54524358},"hb_peers":[0,1,3,4,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":12,"apply_latency_ms":12,"commit_latency_ns":12000000,"apply_latency_ns":12000000},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Mon Jun 3 09:06:23 2024","interfaces":[{"interface":"back","average":{"1min":8.9629999999999992,"5min":946.25400000000002,"15min":326.36099999999999},"min":{"1min":5.3479999999999999,"5min":4.7050000000000001,"15min":4.1420000000000003},"max":{"1min":26.015000000000001,"5min":23465.18,"15min":23465.18},"last":6.6219999999999999},{"interface":"front","average":{"1min":7.6130000000000004,"5min":944.39800000000002,"15min":326.178},"min":{"1min":4.6790000000000003,"5min":4.6790000000000003,"15min":4.6790000000000003},"max":{"1min":20.802,"5min":23465.966,"15min":23465.966},"last":12.722}]},{"osd":1,"last update":"Mon Jun 3 09:06:23 2024","interfaces":[{"interface":"back","average":{"1min":13.007,"5min":76.507000000000005,"15min":294.72899999999998},"min":{"1min":6.4089999999999998,"5min":5.4199999999999999,"15min":5.4199999999999999},"max":{"1min":85.430000000000007,"5min":4220.5290000000005,"15min":18954.987000000001},"last":8.8819999999999997},{"interface":"front","average":{"1min":15.894,"5min":82.119,"15min":302.92399999999998},"min":{"1min":7.367,"5min":7.367,"15min":7.367},"max":{"1min":98.498000000000005,"5min":4325.9489999999996,"15min":19143.834999999999},"last":13.397}]},{"osd":3,"last update":"Mon Jun 3 09:06:23 2024","interfaces":[{"interface":"back","average":{"1min":9.7579999999999991,"5min":12.444000000000001,"15min":28.065000000000001},"min":{"1min":7,"5min":5.9290000000000003,"15min":5.2450000000000001},"max":{"1min":28.164000000000001,"5min":118.913,"15min":2594.8150000000001},"last":7.4560000000000004},{"interface":"front","average":{"1min":12.321999999999999,"5min":15.25,"15min":31.349},"min":{"1min":9.1240000000000006,"5min":8.0760000000000005,"15min":6.0780000000000003},"max":{"1min":32.360999999999997,"5min":156.34700000000001,"15min":2706.5120000000002},"last":9.6129999999999995}]},{"osd":4,"last update":"Mon Jun 3 09:06:59 2024","interfaces":[{"interface":"back","average":{"1min":11.795999999999999,"5min":16.489000000000001,"15min":437.19299999999998},"min":{"1min":7.8780000000000001,"5min":7.8780000000000001,"15min":6.8049999999999997},"max":{"1min":30.163,"5min":182.07400000000001,"15min":26660.861000000001},"last":7.9589999999999996},{"interface":"front","average":{"1min":14.726000000000001,"5min":20.835000000000001,"15min":435.017},"min":{"1min":10.970000000000001,"5min":10.266999999999999,"15min":6.2720000000000002},"max":{"1min":33.402999999999999,"5min":216.476,"15min":26616.243999999999},"last":11.629}]},{"osd":5,"last update":"Mon Jun 3 09:06:23 2024","interfaces":[{"interface":"back","average":{"1min":14.103,"5min":18.091000000000001,"15min":21.119},"min":{"1min":9.6349999999999998,"5min":9.2159999999999993,"15min":6.2050000000000001},"max":{"1min":32.890000000000001,"5min":252.31100000000001,"15min":252.31100000000001},"last":12.173999999999999},{"interface":"front","average":{"1min":11.545999999999999,"5min":16.170999999999999,"15min":20.254000000000001},"min":{"1min":7.6070000000000002,"5min":7.4850000000000003,"15min":6.8810000000000002},"max":{"1min":29.228999999999999,"5min":258.31700000000001,"15min":258.31700000000001},"last":11.129}]},{"osd":6,"last update":"Mon Jun 3 09:04:28 2024","interfaces":[{"interface":"back","average":{"1min":14.063000000000001,"5min":24.850000000000001,"15min":106.755},"min":{"1min":9.0820000000000007,"5min":8.9649999999999999,"15min":8.9649999999999999},"max":{"1min":24.666,"5min":188.06700000000001,"15min":9521.0010000000002},"last":15.291},{"interface":"front","average":{"1min":13.933999999999999,"5min":21.081,"15min":106.142},"min":{"1min":5.8719999999999999,"5min":5.8719999999999999,"15min":5.8719999999999999},"max":{"1min":117.551,"5min":193.887,"15min":9585.6620000000003},"last":10.324999999999999}]},{"osd":7,"last update":"Mon Jun 3 09:06:23 2024","interfaces":[{"interface":"back","average":{"1min":17.056000000000001,"5min":20.055,"15min":23.268999999999998},"min":{"1min":10.353999999999999,"5min":6.6130000000000004,"15min":6.6130000000000004},"max":{"1min":88.974999999999994,"5min":242.56800000000001,"15min":242.56800000000001},"last":14.670999999999999},{"interface":"front","average":{"1min":18.885000000000002,"5min":21.460000000000001,"15min":25.489999999999998},"min":{"1min":12.497,"5min":9.0660000000000007,"15min":9.0660000000000007},"max":{"1min":102.229,"5min":181.56399999999999,"15min":214.33799999999999},"last":16.253}]}]},{"osd":3,"up_from":14,"seq":60129542509,"num_pgs":11,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":59016,"kb_used_data":5752,"kb_used_omap":1,"kb_used_meta":53246,"kb_avail":94312824,"statfs":{"total":96636764160,"available":96576331776,"internally_reserved":0,"allocated":5890048,"data_stored":2763073,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1592,"internal_metadata":54524360},"hb_peers":[0,1,2,4,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Mon Jun 3 09:06:25 2024","interfaces":[{"interface":"back","average":{"1min":7.79,"5min":1060.1959999999999,"15min":367.01799999999997},"min":{"1min":5.6909999999999998,"5min":5.173,"15min":4.6929999999999996},"max":{"1min":10.59,"5min":23275.167000000001,"15min":23275.167000000001},"last":9.4420000000000002},{"interface":"front","average":{"1min":11.215999999999999,"5min":1066.5799999999999,"15min":372.476},"min":{"1min":8.7080000000000002,"5min":7.2510000000000003,"15min":5.9880000000000004},"max":{"1min":13.499000000000001,"5min":23278.536,"15min":23278.536},"last":16.870000000000001}]},{"osd":1,"last update":"Mon Jun 3 09:06:25 2024","interfaces":[{"interface":"back","average":{"1min":7.3760000000000003,"5min":15.614000000000001,"15min":389.83100000000002},"min":{"1min":4.335,"5min":4.335,"15min":4.335},"max":{"1min":11.289,"5min":250.72300000000001,"15min":21658.205999999998},"last":7.9359999999999999},{"interface":"front","average":{"1min":8.9499999999999993,"5min":14.135,"15min":381.10300000000001},"min":{"1min":6.5609999999999999,"5min":5.4100000000000001,"15min":5.2779999999999996},"max":{"1min":14.51,"5min":180.44499999999999,"15min":21307.069},"last":10.613}]},{"osd":2,"last update":"Mon Jun 3 09:06:25 2024","interfaces":[{"interface":"back","average":{"1min":12.343999999999999,"5min":22.059999999999999,"15min":23.715},"min":{"1min":7.9199999999999999,"5min":6.4900000000000002,"15min":6.056},"max":{"1min":15.601000000000001,"5min":377.803,"15min":377.803},"last":18.875},{"interface":"front","average":{"1min":10.319000000000001,"5min":19.364999999999998,"15min":20.242999999999999},"min":{"1min":6.891,"5min":6.1749999999999998,"15min":5.9020000000000001},"max":{"1min":15.432,"5min":371.601,"15min":371.601},"last":17.957999999999998}]},{"osd":4,"last update":"Mon Jun 3 09:06:31 2024","interfaces":[{"interface":"back","average":{"1min":9.7460000000000004,"5min":891.23000000000002,"15min":416.48099999999999},"min":{"1min":5.6539999999999999,"5min":5.6539999999999999,"15min":5.6539999999999999},"max":{"1min":15.445,"5min":29040.635999999999,"15min":30144.939999999999},"last":12.01},{"interface":"front","average":{"1min":12.398999999999999,"5min":897.58500000000004,"15min":421.27499999999998},"min":{"1min":8.9469999999999992,"5min":7.8109999999999999,"15min":6.2439999999999998},"max":{"1min":16.518000000000001,"5min":29076.955999999998,"15min":30180.689999999999},"last":12.824999999999999}]},{"osd":5,"last update":"Mon Jun 3 09:06:07 2024","interfaces":[{"interface":"back","average":{"1min":12.802,"5min":19.640999999999998,"15min":22.318000000000001},"min":{"1min":9.0890000000000004,"5min":6.8810000000000002,"15min":6.7690000000000001},"max":{"1min":16.745999999999999,"5min":249.88,"15min":327.34899999999999},"last":20.768999999999998},{"interface":"front","average":{"1min":15.002000000000001,"5min":24.003,"15min":26.649999999999999},"min":{"1min":11.984,"5min":9.9149999999999991,"15min":8.9879999999999995},"max":{"1min":18.407,"5min":251.47800000000001,"15min":331.86799999999999},"last":22.526}]},{"osd":6,"last update":"Mon Jun 3 09:04:04 2024","interfaces":[{"interface":"back","average":{"1min":22.815999999999999,"5min":31.513999999999999,"15min":181.74299999999999},"min":{"1min":9.4809999999999999,"5min":7.2240000000000002,"15min":7.2240000000000002},"max":{"1min":160.56999999999999,"5min":380.613,"15min":11893.74},"last":16.056999999999999},{"interface":"front","average":{"1min":25.359999999999999,"5min":30.981000000000002,"15min":179.81399999999999},"min":{"1min":9.8870000000000005,"5min":7.3360000000000003,"15min":6.0899999999999999},"max":{"1min":193.11099999999999,"5min":379.80700000000002,"15min":11893.069},"last":14.619}]},{"osd":7,"last update":"Mon Jun 3 09:06:25 2024","interfaces":[{"interface":"back","average":{"1min":11.321999999999999,"5min":18.443999999999999,"15min":21.568999999999999},"min":{"1min":8.109,"5min":8.109,"15min":7.4279999999999999},"max":{"1min":17.113,"5min":146.267,"15min":178.76300000000001},"last":19.881},{"interface":"front","average":{"1min":12.512,"5min":20.065999999999999,"15min":24.190000000000001},"min":{"1min":7.4989999999999997,"5min":7.4989999999999997,"15min":7.4989999999999997},"max":{"1min":18.047999999999998,"5min":148.13800000000001,"15min":417.22800000000001},"last":23.483000000000001}]}]},{"osd":1,"up_from":14,"seq":60129542507,"num_pgs":6,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":59400,"kb_used_data":6144,"kb_used_omap":1,"kb_used_meta":53246,"kb_avail":94312440,"statfs":{"total":96636764160,"available":96575938560,"internally_reserved":0,"allocated":6291456,"data_stored":3160913,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":54524357},"hb_peers":[0,2,3,4,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":12,"apply_latency_ms":12,"commit_latency_ns":12000000,"apply_latency_ns":12000000},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Mon Jun 3 09:06:07 2024","interfaces":[{"interface":"back","average":{"1min":7.7530000000000001,"5min":185.66999999999999,"15min":77.347999999999999},"min":{"1min":5.0229999999999997,"5min":4.9279999999999999,"15min":4.5449999999999999},"max":{"1min":11.896000000000001,"5min":7470.616,"15min":7470.616},"last":5.4109999999999996},{"interface":"front","average":{"1min":9.9220000000000006,"5min":187.14599999999999,"15min":76.325999999999993},"min":{"1min":5.3159999999999998,"5min":5.3159999999999998,"15min":5.3159999999999998},"max":{"1min":18.338000000000001,"5min":7477.0829999999996,"15min":7477.0829999999996},"last":7.8250000000000002}]},{"osd":2,"last update":"Mon Jun 3 09:06:20 2024","interfaces":[{"interface":"back","average":{"1min":10.340999999999999,"5min":17.369,"15min":23.670999999999999},"min":{"1min":6.9740000000000002,"5min":5.8650000000000002,"15min":5.8650000000000002},"max":{"1min":15.452,"5min":228,"15min":657.38},"last":8.5239999999999991},{"interface":"front","average":{"1min":13.387,"5min":23.355,"15min":31.524000000000001},"min":{"1min":9.077,"5min":9.077,"15min":7.2160000000000002},"max":{"1min":19.701000000000001,"5min":229.465,"15min":700.06500000000005},"last":15.786}]},{"osd":3,"last update":"Mon Jun 3 09:06:04 2024","interfaces":[{"interface":"back","average":{"1min":8.6110000000000007,"5min":15.733000000000001,"15min":26.646999999999998},"min":{"1min":5.827,"5min":5.4640000000000004,"15min":5.4640000000000004},"max":{"1min":13.105,"5min":121.619,"15min":1676.009},"last":10.117000000000001},{"interface":"front","average":{"1min":9.5850000000000009,"5min":16.161999999999999,"15min":29.065999999999999},"min":{"1min":6.3630000000000004,"5min":5.7699999999999996,"15min":5.7699999999999996},"max":{"1min":15.166,"5min":132.036,"15min":1766.5840000000001},"last":9.0570000000000004}]},{"osd":4,"last update":"Mon Jun 3 09:06:35 2024","interfaces":[{"interface":"back","average":{"1min":8.6400000000000006,"5min":641.85500000000002,"15min":334.71499999999997},"min":{"1min":6.7640000000000002,"5min":6.1760000000000002,"15min":6.0970000000000004},"max":{"1min":11.811,"5min":26340.968000000001,"15min":28044.862000000001},"last":11.622},{"interface":"front","average":{"1min":12.361000000000001,"5min":648.59900000000005,"15min":338.48500000000001},"min":{"1min":9.8949999999999996,"5min":9.3940000000000001,"15min":6.4640000000000004},"max":{"1min":15.507,"5min":26342.311000000002,"15min":28045.624},"last":14.451000000000001}]},{"osd":5,"last update":"Mon Jun 3 09:06:04 2024","interfaces":[{"interface":"back","average":{"1min":12.478999999999999,"5min":20.228000000000002,"15min":25.997},"min":{"1min":8.6430000000000007,"5min":7.9720000000000004,"15min":7.234},"max":{"1min":17.442,"5min":139.43199999999999,"15min":701.22000000000003},"last":12.804},{"interface":"front","average":{"1min":13.507999999999999,"5min":25.760999999999999,"15min":27.553999999999998},"min":{"1min":10.042999999999999,"5min":7.258,"15min":6.242},"max":{"1min":20.611000000000001,"5min":278.63,"15min":598.35000000000002},"last":10.843}]},{"osd":6,"last update":"Mon Jun 3 09:03:54 2024","interfaces":[{"interface":"back","average":{"1min":28.524000000000001,"5min":20.654,"15min":23.878},"min":{"1min":6.0309999999999997,"5min":6.0309999999999997,"15min":6.0309999999999997},"max":{"1min":167.221,"5min":167.221,"15min":704.245},"last":12.298999999999999},{"interface":"front","average":{"1min":27.606000000000002,"5min":30.131,"15min":28.628},"min":{"1min":9.8650000000000002,"5min":9.7200000000000006,"15min":8.3580000000000005},"max":{"1min":123.938,"5min":358.23000000000002,"15min":634.05499999999995},"last":13.926}]},{"osd":7,"last update":"Mon Jun 3 09:06:20 2024","interfaces":[{"interface":"back","average":{"1min":14.66,"5min":23.234999999999999,"15min":31.097000000000001},"min":{"1min":11.250999999999999,"5min":8.0559999999999992,"15min":8.0559999999999992},"max":{"1min":21.507999999999999,"5min":270.95100000000002,"15min":703.07799999999997},"last":13.355},{"interface":"front","average":{"1min":15.548,"5min":28.760000000000002,"15min":28.925000000000001},"min":{"1min":12.759,"5min":8.2089999999999996,"15min":7.7599999999999998},"max":{"1min":22.408000000000001,"5min":360.99900000000002,"15min":608.44100000000003},"last":16.329999999999998}]}]},{"osd":0,"up_from":14,"seq":60129542509,"num_pgs":10,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":81152,"kb_used_data":5752,"kb_used_omap":1,"kb_used_meta":75390,"kb_avail":94290688,"statfs":{"total":96636764160,"available":96553664512,"internally_reserved":0,"allocated":5890048,"data_stored":2763073,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1593,"internal_metadata":77199815},"hb_peers":[1,2,3,4,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":9,"apply_latency_ms":9,"commit_latency_ns":9000000,"apply_latency_ns":9000000},"alerts":[],"network_ping_times":[{"osd":1,"last update":"Mon Jun 3 09:06:04 2024","interfaces":[{"interface":"back","average":{"1min":6.9960000000000004,"5min":56.997,"15min":339.29700000000003},"min":{"1min":4.4530000000000003,"5min":4.4530000000000003,"15min":4.4100000000000001},"max":{"1min":11.720000000000001,"5min":2414.4749999999999,"15min":20691.453000000001},"last":9.2639999999999993},{"interface":"front","average":{"1min":9.548,"5min":57.258000000000003,"15min":337.791},"min":{"1min":5.2300000000000004,"5min":5.2300000000000004,"15min":4.6050000000000004},"max":{"1min":15.337999999999999,"5min":2412.9549999999999,"15min":20640.792000000001},"last":19.876000000000001}]},{"osd":2,"last update":"Mon Jun 3 09:06:22 2024","interfaces":[{"interface":"back","average":{"1min":11.154,"5min":26.515999999999998,"15min":22.882000000000001},"min":{"1min":5.7869999999999999,"5min":5.766,"15min":5.766},"max":{"1min":72.269999999999996,"5min":627.40700000000004,"15min":627.40700000000004},"last":10.609999999999999},{"interface":"front","average":{"1min":14.353999999999999,"5min":33.304000000000002,"15min":29.885000000000002},"min":{"1min":8.2149999999999999,"5min":7.8630000000000004,"15min":7.4089999999999998},"max":{"1min":73.418000000000006,"5min":635.62599999999998,"15min":635.62599999999998},"last":14.324999999999999}]},{"osd":3,"last update":"Mon Jun 3 09:06:04 2024","interfaces":[{"interface":"back","average":{"1min":12.105,"5min":26.925000000000001,"15min":26.584},"min":{"1min":7.8410000000000002,"5min":7.1280000000000001,"15min":6.1909999999999998},"max":{"1min":18.425999999999998,"5min":389.72699999999998,"15min":389.72699999999998},"last":16.462},{"interface":"front","average":{"1min":13.794,"5min":29.771999999999998,"15min":28.815000000000001},"min":{"1min":10.084,"5min":9.4920000000000009,"15min":6.1849999999999996},"max":{"1min":17.373999999999999,"5min":390.23200000000003,"15min":494.92899999999997},"last":17.754000000000001}]},{"osd":4,"last update":"Mon Jun 3 09:06:27 2024","interfaces":[{"interface":"back","average":{"1min":11.352,"5min":745.30600000000004,"15min":363.48099999999999},"min":{"1min":6.415,"5min":6.415,"15min":6.415},"max":{"1min":16.724,"5min":20248.616000000002,"15min":25555.212},"last":13.329000000000001},{"interface":"front","average":{"1min":13.545,"5min":744.76999999999998,"15min":361.03100000000001},"min":{"1min":9.3889999999999993,"5min":8.8940000000000001,"15min":7.1429999999999998},"max":{"1min":18.411000000000001,"5min":20250.195,"15min":25556.159},"last":18.879000000000001}]},{"osd":5,"last update":"Mon Jun 3 09:06:17 2024","interfaces":[{"interface":"back","average":{"1min":9.7040000000000006,"5min":26.533999999999999,"15min":28.683},"min":{"1min":6.8129999999999997,"5min":6.8129999999999997,"15min":6.8129999999999997},"max":{"1min":14.942,"5min":455.78699999999998,"15min":605.05899999999997},"last":8.4659999999999993},{"interface":"front","average":{"1min":14.279,"5min":28.398,"15min":26.192},"min":{"1min":11.696,"5min":6.4420000000000002,"15min":6.2030000000000003},"max":{"1min":17.879000000000001,"5min":391.31200000000001,"15min":541.11000000000001},"last":20.969999999999999}]},{"osd":6,"last update":"Mon Jun 3 09:04:30 2024","interfaces":[{"interface":"back","average":{"1min":52.496000000000002,"5min":33.139000000000003,"15min":160.83199999999999},"min":{"1min":7.5289999999999999,"5min":7.1040000000000001,"15min":6.3739999999999997},"max":{"1min":457.91699999999997,"5min":457.91699999999997,"15min":11115.258},"last":15.154},{"interface":"front","average":{"1min":49.670000000000002,"5min":33.704999999999998,"15min":162.791},"min":{"1min":12.247999999999999,"5min":11.019,"15min":9.9760000000000009},"max":{"1min":407.05099999999999,"5min":407.05099999999999,"15min":11092.194},"last":9.9009999999999998}]},{"osd":7,"last update":"Mon Jun 3 09:06:22 2024","interfaces":[{"interface":"back","average":{"1min":11.625999999999999,"5min":26.375,"15min":28.972999999999999},"min":{"1min":6.4480000000000004,"5min":6.4480000000000004,"15min":6.4480000000000004},"max":{"1min":15.352,"5min":461.41699999999997,"15min":598.33699999999999},"last":11.227},{"interface":"front","average":{"1min":14.013,"5min":29.696000000000002,"15min":33.630000000000003},"min":{"1min":10.938000000000001,"5min":10.58,"15min":10.263999999999999},"max":{"1min":16.603000000000002,"5min":463.10599999999999,"15min":592.80499999999995},"last":11.832000000000001}]}]}],"pool_statfs":[{"poolid":1,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":401408,"data_stored":397840,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":5,"total":0,"available":0,"internally_reserved":0,"allocated":401408,"data_stored":397840,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":4,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":5,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":6,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":17,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":17,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":17,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":17,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":17,"osd":4,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":17,"osd":5,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":5,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":17,"osd":6,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":5,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":17,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0}]}} 2024-06-03T09:07:03.168 INFO:tasks.ceph:pgid 17.d last_scrub_stamp 2024-06-03T08:37:55.614497+0000 time.struct_time(tm_year=2024, tm_mon=6, tm_mday=3, tm_hour=8, tm_min=37, tm_sec=55, tm_wday=0, tm_yday=155, tm_isdst=-1) <= time.struct_time(tm_year=2024, tm_mon=6, tm_mday=3, tm_hour=9, tm_min=6, tm_sec=55, tm_wday=0, tm_yday=155, tm_isdst=0) 2024-06-03T09:07:03.169 INFO:tasks.ceph:pgid 17.c last_scrub_stamp 2024-06-03T08:37:55.614497+0000 time.struct_time(tm_year=2024, tm_mon=6, tm_mday=3, tm_hour=8, tm_min=37, tm_sec=55, tm_wday=0, tm_yday=155, tm_isdst=-1) <= time.struct_time(tm_year=2024, tm_mon=6, tm_mday=3, tm_hour=9, tm_min=6, tm_sec=55, tm_wday=0, tm_yday=155, tm_isdst=0) 2024-06-03T09:07:03.169 INFO:tasks.ceph:pgid 17.f last_scrub_stamp 2024-06-03T08:37:55.614497+0000 time.struct_time(tm_year=2024, tm_mon=6, tm_mday=3, tm_hour=8, tm_min=37, tm_sec=55, tm_wday=0, tm_yday=155, tm_isdst=-1) <= time.struct_time(tm_year=2024, tm_mon=6, tm_mday=3, tm_hour=9, tm_min=6, tm_sec=55, tm_wday=0, tm_yday=155, tm_isdst=0) 2024-06-03T09:07:03.169 INFO:tasks.ceph:pgid 17.e last_scrub_stamp 2024-06-03T08:37:55.614497+0000 time.struct_time(tm_year=2024, tm_mon=6, tm_mday=3, tm_hour=8, tm_min=37, tm_sec=55, tm_wday=0, tm_yday=155, tm_isdst=-1) <= time.struct_time(tm_year=2024, tm_mon=6, tm_mday=3, tm_hour=9, tm_min=6, tm_sec=55, tm_wday=0, tm_yday=155, tm_isdst=0) 2024-06-03T09:07:03.169 INFO:tasks.ceph:pgid 17.9 last_scrub_stamp 2024-06-03T08:37:55.614497+0000 time.struct_time(tm_year=2024, tm_mon=6, tm_mday=3, tm_hour=8, tm_min=37, tm_sec=55, tm_wday=0, tm_yday=155, tm_isdst=-1) <= time.struct_time(tm_year=2024, tm_mon=6, tm_mday=3, tm_hour=9, tm_min=6, tm_sec=55, tm_wday=0, tm_yday=155, tm_isdst=0) 2024-06-03T09:07:03.169 INFO:tasks.ceph:pgid 17.b last_scrub_stamp 2024-06-03T08:37:55.614497+0000 time.struct_time(tm_year=2024, tm_mon=6, tm_mday=3, tm_hour=8, tm_min=37, tm_sec=55, tm_wday=0, tm_yday=155, tm_isdst=-1) <= time.struct_time(tm_year=2024, tm_mon=6, tm_mday=3, tm_hour=9, tm_min=6, tm_sec=55, tm_wday=0, tm_yday=155, tm_isdst=0) 2024-06-03T09:07:03.169 INFO:tasks.ceph:pgid 17.a last_scrub_stamp 2024-06-03T08:37:55.614497+0000 time.struct_time(tm_year=2024, tm_mon=6, tm_mday=3, tm_hour=8, tm_min=37, tm_sec=55, tm_wday=0, tm_yday=155, tm_isdst=-1) <= time.struct_time(tm_year=2024, tm_mon=6, tm_mday=3, tm_hour=9, tm_min=6, tm_sec=55, tm_wday=0, tm_yday=155, tm_isdst=0) 2024-06-03T09:07:03.169 INFO:tasks.ceph:pgid 17.5 last_scrub_stamp 2024-06-03T08:37:55.614497+0000 time.struct_time(tm_year=2024, tm_mon=6, tm_mday=3, tm_hour=8, tm_min=37, tm_sec=55, tm_wday=0, tm_yday=155, tm_isdst=-1) <= time.struct_time(tm_year=2024, tm_mon=6, tm_mday=3, tm_hour=9, tm_min=6, tm_sec=55, tm_wday=0, tm_yday=155, tm_isdst=0) 2024-06-03T09:07:03.169 INFO:tasks.ceph:pgid 17.4 last_scrub_stamp 2024-06-03T08:37:55.614497+0000 time.struct_time(tm_year=2024, tm_mon=6, tm_mday=3, tm_hour=8, tm_min=37, tm_sec=55, tm_wday=0, tm_yday=155, tm_isdst=-1) <= time.struct_time(tm_year=2024, tm_mon=6, tm_mday=3, tm_hour=9, tm_min=6, tm_sec=55, tm_wday=0, tm_yday=155, tm_isdst=0) 2024-06-03T09:07:03.170 INFO:tasks.ceph:pgid 17.7 last_scrub_stamp 2024-06-03T08:37:55.614497+0000 time.struct_time(tm_year=2024, tm_mon=6, tm_mday=3, tm_hour=8, tm_min=37, tm_sec=55, tm_wday=0, tm_yday=155, tm_isdst=-1) <= time.struct_time(tm_year=2024, tm_mon=6, tm_mday=3, tm_hour=9, tm_min=6, tm_sec=55, tm_wday=0, tm_yday=155, tm_isdst=0) 2024-06-03T09:07:03.170 INFO:tasks.ceph:pgid 17.6 last_scrub_stamp 2024-06-03T08:37:55.614497+0000 time.struct_time(tm_year=2024, tm_mon=6, tm_mday=3, tm_hour=8, tm_min=37, tm_sec=55, tm_wday=0, tm_yday=155, tm_isdst=-1) <= time.struct_time(tm_year=2024, tm_mon=6, tm_mday=3, tm_hour=9, tm_min=6, tm_sec=55, tm_wday=0, tm_yday=155, tm_isdst=0) 2024-06-03T09:07:03.170 INFO:tasks.ceph:pgid 2.7 last_scrub_stamp 2024-06-03T08:36:28.444764+0000 time.struct_time(tm_year=2024, tm_mon=6, tm_mday=3, tm_hour=8, tm_min=36, tm_sec=28, tm_wday=0, tm_yday=155, tm_isdst=-1) <= time.struct_time(tm_year=2024, tm_mon=6, tm_mday=3, tm_hour=9, tm_min=6, tm_sec=55, tm_wday=0, tm_yday=155, tm_isdst=0) 2024-06-03T09:07:03.170 INFO:tasks.ceph:pgid 17.14 last_scrub_stamp 2024-06-03T08:37:55.614497+0000 time.struct_time(tm_year=2024, tm_mon=6, tm_mday=3, tm_hour=8, tm_min=37, tm_sec=55, tm_wday=0, tm_yday=155, tm_isdst=-1) <= time.struct_time(tm_year=2024, tm_mon=6, tm_mday=3, tm_hour=9, tm_min=6, tm_sec=55, tm_wday=0, tm_yday=155, tm_isdst=0) 2024-06-03T09:07:03.170 INFO:tasks.ceph:pgid 2.6 last_scrub_stamp 2024-06-03T08:36:28.444764+0000 time.struct_time(tm_year=2024, tm_mon=6, tm_mday=3, tm_hour=8, tm_min=36, tm_sec=28, tm_wday=0, tm_yday=155, tm_isdst=-1) <= time.struct_time(tm_year=2024, tm_mon=6, tm_mday=3, tm_hour=9, tm_min=6, tm_sec=55, tm_wday=0, tm_yday=155, tm_isdst=0) 2024-06-03T09:07:03.170 INFO:tasks.ceph:pgid 17.15 last_scrub_stamp 2024-06-03T08:37:55.614497+0000 time.struct_time(tm_year=2024, tm_mon=6, tm_mday=3, tm_hour=8, tm_min=37, tm_sec=55, tm_wday=0, tm_yday=155, tm_isdst=-1) <= time.struct_time(tm_year=2024, tm_mon=6, tm_mday=3, tm_hour=9, tm_min=6, tm_sec=55, tm_wday=0, tm_yday=155, tm_isdst=0) 2024-06-03T09:07:03.170 INFO:tasks.ceph:pgid 2.5 last_scrub_stamp 2024-06-03T08:36:28.444764+0000 time.struct_time(tm_year=2024, tm_mon=6, tm_mday=3, tm_hour=8, tm_min=36, tm_sec=28, tm_wday=0, tm_yday=155, tm_isdst=-1) <= time.struct_time(tm_year=2024, tm_mon=6, tm_mday=3, tm_hour=9, tm_min=6, tm_sec=55, tm_wday=0, tm_yday=155, tm_isdst=0) 2024-06-03T09:07:03.170 INFO:tasks.ceph:pgid 2.4 last_scrub_stamp 2024-06-03T08:36:28.444764+0000 time.struct_time(tm_year=2024, tm_mon=6, tm_mday=3, tm_hour=8, tm_min=36, tm_sec=28, tm_wday=0, tm_yday=155, tm_isdst=-1) <= time.struct_time(tm_year=2024, tm_mon=6, tm_mday=3, tm_hour=9, tm_min=6, tm_sec=55, tm_wday=0, tm_yday=155, tm_isdst=0) 2024-06-03T09:07:03.170 INFO:tasks.ceph:pgid 17.17 last_scrub_stamp 2024-06-03T08:37:55.614497+0000 time.struct_time(tm_year=2024, tm_mon=6, tm_mday=3, tm_hour=8, tm_min=37, tm_sec=55, tm_wday=0, tm_yday=155, tm_isdst=-1) <= time.struct_time(tm_year=2024, tm_mon=6, tm_mday=3, tm_hour=9, tm_min=6, tm_sec=55, tm_wday=0, tm_yday=155, tm_isdst=0) 2024-06-03T09:07:03.171 INFO:tasks.ceph:pgid 2.2 last_scrub_stamp 2024-06-03T08:36:28.444764+0000 time.struct_time(tm_year=2024, tm_mon=6, tm_mday=3, tm_hour=8, tm_min=36, tm_sec=28, tm_wday=0, tm_yday=155, tm_isdst=-1) <= time.struct_time(tm_year=2024, tm_mon=6, tm_mday=3, tm_hour=9, tm_min=6, tm_sec=55, tm_wday=0, tm_yday=155, tm_isdst=0) 2024-06-03T09:07:03.171 INFO:tasks.ceph:pgid 17.11 last_scrub_stamp 2024-06-03T08:37:55.614497+0000 time.struct_time(tm_year=2024, tm_mon=6, tm_mday=3, tm_hour=8, tm_min=37, tm_sec=55, tm_wday=0, tm_yday=155, tm_isdst=-1) <= time.struct_time(tm_year=2024, tm_mon=6, tm_mday=3, tm_hour=9, tm_min=6, tm_sec=55, tm_wday=0, tm_yday=155, tm_isdst=0) 2024-06-03T09:07:03.171 INFO:tasks.ceph:pgid 2.1 last_scrub_stamp 2024-06-03T08:36:28.444764+0000 time.struct_time(tm_year=2024, tm_mon=6, tm_mday=3, tm_hour=8, tm_min=36, tm_sec=28, tm_wday=0, tm_yday=155, tm_isdst=-1) <= time.struct_time(tm_year=2024, tm_mon=6, tm_mday=3, tm_hour=9, tm_min=6, tm_sec=55, tm_wday=0, tm_yday=155, tm_isdst=0) 2024-06-03T09:07:03.171 INFO:tasks.ceph:pgid 17.12 last_scrub_stamp 2024-06-03T08:37:55.614497+0000 time.struct_time(tm_year=2024, tm_mon=6, tm_mday=3, tm_hour=8, tm_min=37, tm_sec=55, tm_wday=0, tm_yday=155, tm_isdst=-1) <= time.struct_time(tm_year=2024, tm_mon=6, tm_mday=3, tm_hour=9, tm_min=6, tm_sec=55, tm_wday=0, tm_yday=155, tm_isdst=0) 2024-06-03T09:07:03.171 INFO:tasks.ceph:pgid 2.0 last_scrub_stamp 2024-06-03T08:36:28.444764+0000 time.struct_time(tm_year=2024, tm_mon=6, tm_mday=3, tm_hour=8, tm_min=36, tm_sec=28, tm_wday=0, tm_yday=155, tm_isdst=-1) <= time.struct_time(tm_year=2024, tm_mon=6, tm_mday=3, tm_hour=9, tm_min=6, tm_sec=55, tm_wday=0, tm_yday=155, tm_isdst=0) 2024-06-03T09:07:03.171 INFO:tasks.ceph:pgid 17.13 last_scrub_stamp 2024-06-03T08:37:55.614497+0000 time.struct_time(tm_year=2024, tm_mon=6, tm_mday=3, tm_hour=8, tm_min=37, tm_sec=55, tm_wday=0, tm_yday=155, tm_isdst=-1) <= time.struct_time(tm_year=2024, tm_mon=6, tm_mday=3, tm_hour=9, tm_min=6, tm_sec=55, tm_wday=0, tm_yday=155, tm_isdst=0) 2024-06-03T09:07:03.171 INFO:tasks.ceph:pgid 2.3 last_scrub_stamp 2024-06-03T08:36:28.444764+0000 time.struct_time(tm_year=2024, tm_mon=6, tm_mday=3, tm_hour=8, tm_min=36, tm_sec=28, tm_wday=0, tm_yday=155, tm_isdst=-1) <= time.struct_time(tm_year=2024, tm_mon=6, tm_mday=3, tm_hour=9, tm_min=6, tm_sec=55, tm_wday=0, tm_yday=155, tm_isdst=0) 2024-06-03T09:07:03.171 INFO:tasks.ceph:pgid 17.10 last_scrub_stamp 2024-06-03T08:37:55.614497+0000 time.struct_time(tm_year=2024, tm_mon=6, tm_mday=3, tm_hour=8, tm_min=37, tm_sec=55, tm_wday=0, tm_yday=155, tm_isdst=-1) <= time.struct_time(tm_year=2024, tm_mon=6, tm_mday=3, tm_hour=9, tm_min=6, tm_sec=55, tm_wday=0, tm_yday=155, tm_isdst=0) 2024-06-03T09:07:03.172 INFO:tasks.ceph:pgid 17.1a last_scrub_stamp 2024-06-03T08:37:55.614497+0000 time.struct_time(tm_year=2024, tm_mon=6, tm_mday=3, tm_hour=8, tm_min=37, tm_sec=55, tm_wday=0, tm_yday=155, tm_isdst=-1) <= time.struct_time(tm_year=2024, tm_mon=6, tm_mday=3, tm_hour=9, tm_min=6, tm_sec=55, tm_wday=0, tm_yday=155, tm_isdst=0) 2024-06-03T09:07:03.172 INFO:tasks.ceph:pgid 17.1b last_scrub_stamp 2024-06-03T08:37:55.614497+0000 time.struct_time(tm_year=2024, tm_mon=6, tm_mday=3, tm_hour=8, tm_min=37, tm_sec=55, tm_wday=0, tm_yday=155, tm_isdst=-1) <= time.struct_time(tm_year=2024, tm_mon=6, tm_mday=3, tm_hour=9, tm_min=6, tm_sec=55, tm_wday=0, tm_yday=155, tm_isdst=0) 2024-06-03T09:07:03.172 INFO:tasks.ceph:pgid 17.18 last_scrub_stamp 2024-06-03T08:37:55.614497+0000 time.struct_time(tm_year=2024, tm_mon=6, tm_mday=3, tm_hour=8, tm_min=37, tm_sec=55, tm_wday=0, tm_yday=155, tm_isdst=-1) <= time.struct_time(tm_year=2024, tm_mon=6, tm_mday=3, tm_hour=9, tm_min=6, tm_sec=55, tm_wday=0, tm_yday=155, tm_isdst=0) 2024-06-03T09:07:03.172 INFO:tasks.ceph:pgid 17.19 last_scrub_stamp 2024-06-03T08:37:55.614497+0000 time.struct_time(tm_year=2024, tm_mon=6, tm_mday=3, tm_hour=8, tm_min=37, tm_sec=55, tm_wday=0, tm_yday=155, tm_isdst=-1) <= time.struct_time(tm_year=2024, tm_mon=6, tm_mday=3, tm_hour=9, tm_min=6, tm_sec=55, tm_wday=0, tm_yday=155, tm_isdst=0) 2024-06-03T09:07:03.172 INFO:tasks.ceph:pgid 17.1e last_scrub_stamp 2024-06-03T08:37:55.614497+0000 time.struct_time(tm_year=2024, tm_mon=6, tm_mday=3, tm_hour=8, tm_min=37, tm_sec=55, tm_wday=0, tm_yday=155, tm_isdst=-1) <= time.struct_time(tm_year=2024, tm_mon=6, tm_mday=3, tm_hour=9, tm_min=6, tm_sec=55, tm_wday=0, tm_yday=155, tm_isdst=0) 2024-06-03T09:07:03.172 INFO:tasks.ceph:pgid 17.1f last_scrub_stamp 2024-06-03T08:37:55.614497+0000 time.struct_time(tm_year=2024, tm_mon=6, tm_mday=3, tm_hour=8, tm_min=37, tm_sec=55, tm_wday=0, tm_yday=155, tm_isdst=-1) <= time.struct_time(tm_year=2024, tm_mon=6, tm_mday=3, tm_hour=9, tm_min=6, tm_sec=55, tm_wday=0, tm_yday=155, tm_isdst=0) 2024-06-03T09:07:03.172 INFO:tasks.ceph:pgid 17.1c last_scrub_stamp 2024-06-03T08:37:55.614497+0000 time.struct_time(tm_year=2024, tm_mon=6, tm_mday=3, tm_hour=8, tm_min=37, tm_sec=55, tm_wday=0, tm_yday=155, tm_isdst=-1) <= time.struct_time(tm_year=2024, tm_mon=6, tm_mday=3, tm_hour=9, tm_min=6, tm_sec=55, tm_wday=0, tm_yday=155, tm_isdst=0) 2024-06-03T09:07:03.172 INFO:tasks.ceph:pgid 17.1d last_scrub_stamp 2024-06-03T08:37:55.614497+0000 time.struct_time(tm_year=2024, tm_mon=6, tm_mday=3, tm_hour=8, tm_min=37, tm_sec=55, tm_wday=0, tm_yday=155, tm_isdst=-1) <= time.struct_time(tm_year=2024, tm_mon=6, tm_mday=3, tm_hour=9, tm_min=6, tm_sec=55, tm_wday=0, tm_yday=155, tm_isdst=0) 2024-06-03T09:07:03.172 INFO:tasks.ceph:pgid 17.2 last_scrub_stamp 2024-06-03T08:37:55.614497+0000 time.struct_time(tm_year=2024, tm_mon=6, tm_mday=3, tm_hour=8, tm_min=37, tm_sec=55, tm_wday=0, tm_yday=155, tm_isdst=-1) <= time.struct_time(tm_year=2024, tm_mon=6, tm_mday=3, tm_hour=9, tm_min=6, tm_sec=55, tm_wday=0, tm_yday=155, tm_isdst=0) 2024-06-03T09:07:03.172 INFO:tasks.ceph:pgid 17.3 last_scrub_stamp 2024-06-03T08:37:55.614497+0000 time.struct_time(tm_year=2024, tm_mon=6, tm_mday=3, tm_hour=8, tm_min=37, tm_sec=55, tm_wday=0, tm_yday=155, tm_isdst=-1) <= time.struct_time(tm_year=2024, tm_mon=6, tm_mday=3, tm_hour=9, tm_min=6, tm_sec=55, tm_wday=0, tm_yday=155, tm_isdst=0) 2024-06-03T09:07:03.173 INFO:tasks.ceph:pgid 17.0 last_scrub_stamp 2024-06-03T08:37:55.614497+0000 time.struct_time(tm_year=2024, tm_mon=6, tm_mday=3, tm_hour=8, tm_min=37, tm_sec=55, tm_wday=0, tm_yday=155, tm_isdst=-1) <= time.struct_time(tm_year=2024, tm_mon=6, tm_mday=3, tm_hour=9, tm_min=6, tm_sec=55, tm_wday=0, tm_yday=155, tm_isdst=0) 2024-06-03T09:07:03.173 INFO:tasks.ceph:pgid 17.1 last_scrub_stamp 2024-06-03T08:37:55.614497+0000 time.struct_time(tm_year=2024, tm_mon=6, tm_mday=3, tm_hour=8, tm_min=37, tm_sec=55, tm_wday=0, tm_yday=155, tm_isdst=-1) <= time.struct_time(tm_year=2024, tm_mon=6, tm_mday=3, tm_hour=9, tm_min=6, tm_sec=55, tm_wday=0, tm_yday=155, tm_isdst=0) 2024-06-03T09:07:03.173 INFO:tasks.ceph:Still waiting for all pgs to be scrubbed. 2024-06-03T09:07:08.462 INFO:tasks.ceph.mon.a.smithi088.stderr:==00:00:32:03.788 34743== Warning: unimplemented fcntl command: 1036 2024-06-03T09:07:08.509 INFO:tasks.ceph.mon.a.smithi088.stderr:==00:00:32:03.835 34743== Warning: unimplemented fcntl command: 1036 2024-06-03T09:07:08.531 INFO:tasks.ceph.mon.a.smithi088.stderr:==00:00:32:03.857 34743== Warning: unimplemented fcntl command: 1036 2024-06-03T09:07:08.569 INFO:tasks.ceph.mon.b.smithi192.stderr:==00:00:32:03.871 34228== Warning: unimplemented fcntl command: 1036 2024-06-03T09:07:08.571 INFO:tasks.ceph.mon.b.smithi192.stderr:==00:00:32:03.873 34228== Warning: unimplemented fcntl command: 1036 2024-06-03T09:07:08.575 INFO:tasks.ceph.mon.c.smithi088.stderr:==00:00:32:03.901 34749== Warning: unimplemented fcntl command: 1036 2024-06-03T09:07:08.578 INFO:tasks.ceph.mon.c.smithi088.stderr:==00:00:32:03.904 34749== Warning: unimplemented fcntl command: 1036 2024-06-03T09:07:08.589 INFO:tasks.ceph.mon.b.smithi192.stderr:==00:00:32:03.892 34228== Warning: unimplemented fcntl command: 1036 2024-06-03T09:07:08.598 INFO:tasks.ceph.mon.c.smithi088.stderr:==00:00:32:03.924 34749== Warning: unimplemented fcntl command: 1036 2024-06-03T09:07:13.571 INFO:tasks.ceph.mon.a.smithi088.stderr:==00:00:32:08.897 34743== Warning: unimplemented fcntl command: 1036 2024-06-03T09:07:13.575 INFO:tasks.ceph.mon.a.smithi088.stderr:==00:00:32:08.901 34743== Warning: unimplemented fcntl command: 1036 2024-06-03T09:07:13.591 INFO:tasks.ceph.mon.a.smithi088.stderr:==00:00:32:08.916 34743== Warning: unimplemented fcntl command: 1036 2024-06-03T09:07:13.627 INFO:tasks.ceph.mon.b.smithi192.stderr:==00:00:32:08.930 34228== Warning: unimplemented fcntl command: 1036 2024-06-03T09:07:13.630 INFO:tasks.ceph.mon.b.smithi192.stderr:==00:00:32:08.932 34228== Warning: unimplemented fcntl command: 1036 2024-06-03T09:07:13.644 INFO:tasks.ceph.mon.b.smithi192.stderr:==00:00:32:08.947 34228== Warning: unimplemented fcntl command: 1036 2024-06-03T09:07:13.659 INFO:tasks.ceph.mon.c.smithi088.stderr:==00:00:32:08.984 34749== Warning: unimplemented fcntl command: 1036 2024-06-03T09:07:13.661 INFO:tasks.ceph.mon.c.smithi088.stderr:==00:00:32:08.987 34749== Warning: unimplemented fcntl command: 1036 2024-06-03T09:07:13.680 INFO:tasks.ceph.mon.c.smithi088.stderr:==00:00:32:09.005 34749== Warning: unimplemented fcntl command: 1036 2024-06-03T09:07:14.364 INFO:tasks.ceph.mon.b.smithi192.stderr:==00:00:32:09.666 34228== Warning: unimplemented fcntl command: 1036 2024-06-03T09:07:14.365 INFO:tasks.ceph.mon.b.smithi192.stderr:==00:00:32:09.667 34228== Warning: unimplemented fcntl command: 1036 2024-06-03T09:07:14.379 INFO:tasks.ceph.mon.b.smithi192.stderr:==00:00:32:09.682 34228== Warning: unimplemented fcntl command: 1036 2024-06-03T09:07:14.444 INFO:tasks.ceph.mon.a.smithi088.stderr:==00:00:32:09.769 34743== Warning: unimplemented fcntl command: 1036 2024-06-03T09:07:14.445 INFO:tasks.ceph.mon.a.smithi088.stderr:==00:00:32:09.771 34743== Warning: unimplemented fcntl command: 1036 2024-06-03T09:07:14.454 INFO:tasks.ceph.mon.a.smithi088.stderr:==00:00:32:09.780 34743== Warning: unimplemented fcntl command: 1036 2024-06-03T09:07:14.481 INFO:tasks.ceph.mon.c.smithi088.stderr:==00:00:32:09.806 34749== Warning: unimplemented fcntl command: 1036 2024-06-03T09:07:14.481 INFO:tasks.ceph.mon.c.smithi088.stderr:==00:00:32:09.807 34749== Warning: unimplemented fcntl command: 1036 2024-06-03T09:07:14.490 INFO:tasks.ceph.mon.c.smithi088.stderr:==00:00:32:09.816 34749== Warning: unimplemented fcntl command: 1036 2024-06-03T09:07:18.585 INFO:tasks.ceph.mon.a.smithi088.stderr:==00:00:32:13.911 34743== Warning: unimplemented fcntl command: 1036 2024-06-03T09:07:18.589 INFO:tasks.ceph.mon.a.smithi088.stderr:==00:00:32:13.915 34743== Warning: unimplemented fcntl command: 1036 2024-06-03T09:07:18.602 INFO:tasks.ceph.mon.a.smithi088.stderr:==00:00:32:13.928 34743== Warning: unimplemented fcntl command: 1036 2024-06-03T09:07:18.633 INFO:tasks.ceph.mon.b.smithi192.stderr:==00:00:32:13.935 34228== Warning: unimplemented fcntl command: 1036 2024-06-03T09:07:18.634 INFO:tasks.ceph.mon.b.smithi192.stderr:==00:00:32:13.937 34228== Warning: unimplemented fcntl command: 1036 2024-06-03T09:07:18.640 INFO:tasks.ceph.mon.c.smithi088.stderr:==00:00:32:13.966 34749== Warning: unimplemented fcntl command: 1036 2024-06-03T09:07:18.642 INFO:tasks.ceph.mon.c.smithi088.stderr:==00:00:32:13.968 34749== Warning: unimplemented fcntl command: 1036 2024-06-03T09:07:18.643 INFO:tasks.ceph.mon.b.smithi192.stderr:==00:00:32:13.946 34228== Warning: unimplemented fcntl command: 1036 2024-06-03T09:07:18.651 INFO:tasks.ceph.mon.c.smithi088.stderr:==00:00:32:13.977 34749== Warning: unimplemented fcntl command: 1036 2024-06-03T09:07:21.717 INFO:tasks.ceph.mon.a.smithi088.stderr:==00:00:32:17.043 34743== Warning: unimplemented fcntl command: 1036 2024-06-03T09:07:21.728 INFO:tasks.ceph.mon.c.smithi088.stderr:==00:00:32:17.054 34749== Warning: unimplemented fcntl command: 1036 2024-06-03T09:07:21.729 INFO:tasks.ceph.mon.b.smithi192.stderr:==00:00:32:17.032 34228== Warning: unimplemented fcntl command: 1036 2024-06-03T09:07:21.730 INFO:tasks.ceph.mon.c.smithi088.stderr:==00:00:32:17.055 34749== Warning: unimplemented fcntl command: 1036 2024-06-03T09:07:21.766 INFO:tasks.ceph.mon.a.smithi088.stderr:==00:00:32:17.092 34743== Warning: unimplemented fcntl command: 1036 2024-06-03T09:07:21.775 INFO:tasks.ceph.mon.b.smithi192.stderr:==00:00:32:17.077 34228== Warning: unimplemented fcntl command: 1036 2024-06-03T09:07:21.780 INFO:tasks.ceph.mon.a.smithi088.stderr:==00:00:32:17.106 34743== Warning: unimplemented fcntl command: 1036 2024-06-03T09:07:21.790 INFO:tasks.ceph.mon.c.smithi088.stderr:==00:00:32:17.116 34749== Warning: unimplemented fcntl command: 1036 2024-06-03T09:07:21.790 INFO:tasks.ceph.mon.b.smithi192.stderr:==00:00:32:17.093 34228== Warning: unimplemented fcntl command: 1036 2024-06-03T09:07:23.174 DEBUG:teuthology.orchestra.run.smithi088:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph pg dump --format=json 2024-06-03T09:07:23.276 INFO:teuthology.orchestra.run.smithi088.stderr:2024-06-03T09:07:23.276+0000 7f017875c640 10 monclient: get_monmap_and_config 2024-06-03T09:07:23.276 INFO:teuthology.orchestra.run.smithi088.stderr:2024-06-03T09:07:23.276+0000 7f017875c640 10 monclient: build_initial_monmap 2024-06-03T09:07:23.276 INFO:teuthology.orchestra.run.smithi088.stderr:2024-06-03T09:07:23.276+0000 7f017875c640 10 monclient: monmap: 2024-06-03T09:07:23.276 INFO:teuthology.orchestra.run.smithi088.stderr:epoch 0 2024-06-03T09:07:23.276 INFO:teuthology.orchestra.run.smithi088.stderr:fsid ec3c3fdd-3ac6-4d7d-9412-683130ed8b04 2024-06-03T09:07:23.277 INFO:teuthology.orchestra.run.smithi088.stderr:last_changed 2024-06-03T09:07:23.277128+0000 2024-06-03T09:07:23.277 INFO:teuthology.orchestra.run.smithi088.stderr:created 2024-06-03T09:07:23.277128+0000 2024-06-03T09:07:23.277 INFO:teuthology.orchestra.run.smithi088.stderr:min_mon_release 0 (unknown) 2024-06-03T09:07:23.277 INFO:teuthology.orchestra.run.smithi088.stderr:election_strategy: 3 2024-06-03T09:07:23.277 INFO:teuthology.orchestra.run.smithi088.stderr:0: v1:172.21.15.88:6789/0 mon.noname-a 2024-06-03T09:07:23.277 INFO:teuthology.orchestra.run.smithi088.stderr:1: v1:172.21.15.192:6789/0 mon.noname-c 2024-06-03T09:07:23.277 INFO:teuthology.orchestra.run.smithi088.stderr:2: v1:172.21.15.88:6790/0 mon.noname-b 2024-06-03T09:07:23.277 INFO:teuthology.orchestra.run.smithi088.stderr: 2024-06-03T09:07:23.278 INFO:teuthology.orchestra.run.smithi088.stderr:2024-06-03T09:07:23.276+0000 7f017875c640 1 Processor -- start 2024-06-03T09:07:23.279 INFO:teuthology.orchestra.run.smithi088.stderr:2024-06-03T09:07:23.276+0000 7f017875c640 1 -- start start 2024-06-03T09:07:23.279 INFO:teuthology.orchestra.run.smithi088.stderr:2024-06-03T09:07:23.276+0000 7f017875c640 10 monclient: init 2024-06-03T09:07:23.279 INFO:teuthology.orchestra.run.smithi088.stderr:2024-06-03T09:07:23.277+0000 7f017875c640 10 monclient: _reopen_session rank -1 2024-06-03T09:07:23.279 INFO:teuthology.orchestra.run.smithi088.stderr:2024-06-03T09:07:23.277+0000 7f017875c640 10 monclient: _add_conns ranks=[2,1,0] 2024-06-03T09:07:23.279 INFO:teuthology.orchestra.run.smithi088.stderr:2024-06-03T09:07:23.277+0000 7f017875c640 10 monclient(hunting): picked mon.noname-b con 0x7f01700a8c10 addr v1:172.21.15.88:6790/0 2024-06-03T09:07:23.279 INFO:teuthology.orchestra.run.smithi088.stderr:2024-06-03T09:07:23.277+0000 7f017875c640 10 monclient(hunting): picked mon.noname-c con 0x7f01700b1ad0 addr v1:172.21.15.192:6789/0 2024-06-03T09:07:23.279 INFO:teuthology.orchestra.run.smithi088.stderr:2024-06-03T09:07:23.277+0000 7f017875c640 10 monclient(hunting): picked mon.noname-a con 0x7f017014ae30 addr v1:172.21.15.88:6789/0 2024-06-03T09:07:23.279 INFO:teuthology.orchestra.run.smithi088.stderr:2024-06-03T09:07:23.277+0000 7f017875c640 1 -- --> v1:172.21.15.88:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f0170144aa0 con 0x7f017014ae30 2024-06-03T09:07:23.279 INFO:teuthology.orchestra.run.smithi088.stderr:2024-06-03T09:07:23.277+0000 7f017875c640 1 -- --> v1:172.21.15.192:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f0170144d80 con 0x7f01700b1ad0 2024-06-03T09:07:23.279 INFO:teuthology.orchestra.run.smithi088.stderr:2024-06-03T09:07:23.277+0000 7f017875c640 1 -- --> v1:172.21.15.88:6790/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f0170145920 con 0x7f01700a8c10 2024-06-03T09:07:23.279 INFO:teuthology.orchestra.run.smithi088.stderr:2024-06-03T09:07:23.277+0000 7f017875c640 10 monclient(hunting): _renew_subs 2024-06-03T09:07:23.280 INFO:teuthology.orchestra.run.smithi088.stderr:2024-06-03T09:07:23.277+0000 7f017875c640 10 monclient(hunting): authenticate will time out at 3099.087607s 2024-06-03T09:07:23.280 INFO:teuthology.orchestra.run.smithi088.stderr:2024-06-03T09:07:23.278+0000 7f0176cf9640 1 --1- >> v1:172.21.15.88:6789/0 conn(0x7f017014ae30 0x7f017014d220 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.88:6789/0 says I am v1:172.21.15.88:38354/0 (socket says 172.21.15.88:38354) 2024-06-03T09:07:23.280 INFO:teuthology.orchestra.run.smithi088.stderr:2024-06-03T09:07:23.278+0000 7f0176cf9640 1 -- 172.21.15.88:0/4009955801 learned_addr learned my addr 172.21.15.88:0/4009955801 (peer_addr_for_me v1:172.21.15.88:0/0) 2024-06-03T09:07:23.281 INFO:teuthology.orchestra.run.smithi088.stderr:2024-06-03T09:07:23.280+0000 7f01754f6640 1 -- 172.21.15.88:0/4009955801 <== mon.2 v1:172.21.15.88:6790/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 2331974956 0 0) 0x7f0170145920 con 0x7f01700a8c10 2024-06-03T09:07:23.281 INFO:teuthology.orchestra.run.smithi088.stderr:2024-06-03T09:07:23.280+0000 7f01754f6640 10 monclient(hunting): _negotiate 2024-06-03T09:07:23.281 INFO:teuthology.orchestra.run.smithi088.stderr:2024-06-03T09:07:23.280+0000 7f01754f6640 10 monclient(hunting): _init_auth method 2 2024-06-03T09:07:23.281 INFO:teuthology.orchestra.run.smithi088.stderr:2024-06-03T09:07:23.280+0000 7f01754f6640 10 monclient(hunting): _init_auth creating new auth 2024-06-03T09:07:23.281 INFO:teuthology.orchestra.run.smithi088.stderr:2024-06-03T09:07:23.280+0000 7f01754f6640 10 monclient(hunting): my global_id is 30341 2024-06-03T09:07:23.281 INFO:teuthology.orchestra.run.smithi088.stderr:2024-06-03T09:07:23.280+0000 7f01754f6640 1 -- 172.21.15.88:0/4009955801 --> v1:172.21.15.88:6790/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f015c003520 con 0x7f01700a8c10 2024-06-03T09:07:23.281 INFO:teuthology.orchestra.run.smithi088.stderr:2024-06-03T09:07:23.280+0000 7f01754f6640 1 -- 172.21.15.88:0/4009955801 <== mon.0 v1:172.21.15.88:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 865679816 0 0) 0x7f0170144aa0 con 0x7f017014ae30 2024-06-03T09:07:23.281 INFO:teuthology.orchestra.run.smithi088.stderr:2024-06-03T09:07:23.280+0000 7f01754f6640 10 monclient(hunting): _negotiate 2024-06-03T09:07:23.281 INFO:teuthology.orchestra.run.smithi088.stderr:2024-06-03T09:07:23.280+0000 7f01754f6640 10 monclient(hunting): _init_auth method 2 2024-06-03T09:07:23.281 INFO:teuthology.orchestra.run.smithi088.stderr:2024-06-03T09:07:23.280+0000 7f01754f6640 10 monclient(hunting): _init_auth creating new auth 2024-06-03T09:07:23.281 INFO:teuthology.orchestra.run.smithi088.stderr:2024-06-03T09:07:23.280+0000 7f01754f6640 10 monclient(hunting): my global_id is 30831 2024-06-03T09:07:23.281 INFO:teuthology.orchestra.run.smithi088.stderr:2024-06-03T09:07:23.280+0000 7f01754f6640 1 -- 172.21.15.88:0/4009955801 --> v1:172.21.15.88:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f0170145920 con 0x7f017014ae30 2024-06-03T09:07:23.282 INFO:teuthology.orchestra.run.smithi088.stderr:2024-06-03T09:07:23.281+0000 7f01754f6640 1 -- 172.21.15.88:0/4009955801 <== mon.1 v1:172.21.15.192:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 288792628 0 0) 0x7f0170144d80 con 0x7f01700b1ad0 2024-06-03T09:07:23.282 INFO:teuthology.orchestra.run.smithi088.stderr:2024-06-03T09:07:23.281+0000 7f01754f6640 10 monclient(hunting): _negotiate 2024-06-03T09:07:23.282 INFO:teuthology.orchestra.run.smithi088.stderr:2024-06-03T09:07:23.281+0000 7f01754f6640 10 monclient(hunting): _init_auth method 2 2024-06-03T09:07:23.282 INFO:teuthology.orchestra.run.smithi088.stderr:2024-06-03T09:07:23.281+0000 7f01754f6640 10 monclient(hunting): _init_auth creating new auth 2024-06-03T09:07:23.282 INFO:teuthology.orchestra.run.smithi088.stderr:2024-06-03T09:07:23.281+0000 7f01754f6640 10 monclient(hunting): my global_id is 30406 2024-06-03T09:07:23.282 INFO:teuthology.orchestra.run.smithi088.stderr:2024-06-03T09:07:23.281+0000 7f01754f6640 1 -- 172.21.15.88:0/4009955801 --> v1:172.21.15.192:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f0170144aa0 con 0x7f01700b1ad0 2024-06-03T09:07:23.284 INFO:teuthology.orchestra.run.smithi088.stderr:2024-06-03T09:07:23.284+0000 7f01754f6640 1 -- 172.21.15.88:0/4009955801 <== mon.1 v1:172.21.15.192:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 214+0+0 (unknown 2315538629 0 0) 0x7f0170144aa0 con 0x7f01700b1ad0 2024-06-03T09:07:23.284 INFO:teuthology.orchestra.run.smithi088.stderr:2024-06-03T09:07:23.284+0000 7f01754f6640 1 -- 172.21.15.88:0/4009955801 >> v1:172.21.15.88:6790/0 conn(0x7f01700a8c10 legacy=0x7f0170145210 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-06-03T09:07:23.285 INFO:teuthology.orchestra.run.smithi088.stderr:2024-06-03T09:07:23.284+0000 7f01754f6640 1 -- 172.21.15.88:0/4009955801 >> v1:172.21.15.88:6789/0 conn(0x7f017014ae30 legacy=0x7f017014d220 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-06-03T09:07:23.285 INFO:teuthology.orchestra.run.smithi088.stderr:2024-06-03T09:07:23.284+0000 7f01754f6640 10 monclient: _finish_hunting 0 2024-06-03T09:07:23.285 INFO:teuthology.orchestra.run.smithi088.stderr:2024-06-03T09:07:23.284+0000 7f01754f6640 1 monclient: found mon.noname-c 2024-06-03T09:07:23.285 INFO:teuthology.orchestra.run.smithi088.stderr:2024-06-03T09:07:23.284+0000 7f01754f6640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-06-03T09:07:23.285 INFO:teuthology.orchestra.run.smithi088.stderr:2024-06-03T09:07:23.284+0000 7f01754f6640 10 monclient: _send_mon_message to mon.noname-c at v1:172.21.15.192:6789/0 2024-06-03T09:07:23.285 INFO:teuthology.orchestra.run.smithi088.stderr:2024-06-03T09:07:23.284+0000 7f01754f6640 1 -- 172.21.15.88:0/4009955801 --> v1:172.21.15.192:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f01700a9d20 con 0x7f01700b1ad0 2024-06-03T09:07:23.285 INFO:teuthology.orchestra.run.smithi088.stderr:2024-06-03T09:07:23.284+0000 7f01754f6640 10 monclient: _finish_auth 0 2024-06-03T09:07:23.285 INFO:teuthology.orchestra.run.smithi088.stderr:2024-06-03T09:07:23.284+0000 7f01754f6640 10 monclient: _check_auth_tickets 2024-06-03T09:07:23.285 INFO:teuthology.orchestra.run.smithi088.stderr:2024-06-03T09:07:23.284+0000 7f01754f6640 20 monclient: _check_auth_rotating not needed by client.admin 2024-06-03T09:07:23.285 INFO:teuthology.orchestra.run.smithi088.stderr:2024-06-03T09:07:23.284+0000 7f01754f6640 1 -- 172.21.15.88:0/4009955801 <== mon.1 v1:172.21.15.192:6789/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 2234374065 0 0) 0x7f01600015b0 con 0x7f01700b1ad0 2024-06-03T09:07:23.285 INFO:teuthology.orchestra.run.smithi088.stderr:2024-06-03T09:07:23.284+0000 7f01754f6640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-06-03T09:07:23.285 INFO:teuthology.orchestra.run.smithi088.stderr:2024-06-03T09:07:23.284+0000 7f01754f6640 10 monclient: got monmap 1 from mon.noname-c (according to old e1) 2024-06-03T09:07:23.286 INFO:teuthology.orchestra.run.smithi088.stderr:2024-06-03T09:07:23.284+0000 7f01754f6640 10 monclient: dump: 2024-06-03T09:07:23.286 INFO:teuthology.orchestra.run.smithi088.stderr:epoch 1 2024-06-03T09:07:23.286 INFO:teuthology.orchestra.run.smithi088.stderr:fsid ec3c3fdd-3ac6-4d7d-9412-683130ed8b04 2024-06-03T09:07:23.286 INFO:teuthology.orchestra.run.smithi088.stderr:last_changed 2024-06-03T08:34:31.885125+0000 2024-06-03T09:07:23.286 INFO:teuthology.orchestra.run.smithi088.stderr:created 2024-06-03T08:34:31.885125+0000 2024-06-03T09:07:23.286 INFO:teuthology.orchestra.run.smithi088.stderr:min_mon_release 18 (reef) 2024-06-03T09:07:23.286 INFO:teuthology.orchestra.run.smithi088.stderr:election_strategy: 3 2024-06-03T09:07:23.286 INFO:teuthology.orchestra.run.smithi088.stderr:0: v1:172.21.15.88:6789/0 mon.a 2024-06-03T09:07:23.286 INFO:teuthology.orchestra.run.smithi088.stderr:1: v1:172.21.15.192:6789/0 mon.b 2024-06-03T09:07:23.286 INFO:teuthology.orchestra.run.smithi088.stderr:2: v1:172.21.15.88:6790/0 mon.c 2024-06-03T09:07:23.286 INFO:teuthology.orchestra.run.smithi088.stderr: 2024-06-03T09:07:23.286 INFO:teuthology.orchestra.run.smithi088.stderr:2024-06-03T09:07:23.284+0000 7f017875c640 5 monclient: authenticate success, global_id 30406 2024-06-03T09:07:23.287 INFO:teuthology.orchestra.run.smithi088.stderr:2024-06-03T09:07:23.284+0000 7f017875c640 20 monclient: get_monmap_and_config waiting for monmap|config 2024-06-03T09:07:23.289 INFO:teuthology.orchestra.run.smithi088.stderr:2024-06-03T09:07:23.286+0000 7f01754f6640 1 -- 172.21.15.88:0/4009955801 <== mon.1 v1:172.21.15.192:6789/0 4 ==== config(0 keys) v1 ==== 4+0+0 (unknown 0 0 0) 0x7f01600017d0 con 0x7f01700b1ad0 2024-06-03T09:07:23.289 INFO:teuthology.orchestra.run.smithi088.stderr:2024-06-03T09:07:23.286+0000 7f01754f6640 10 monclient: handle_config config(0 keys) v1 2024-06-03T09:07:23.290 INFO:teuthology.orchestra.run.smithi088.stderr:2024-06-03T09:07:23.286+0000 7f01754f6640 1 -- 172.21.15.88:0/4009955801 <== mon.1 v1:172.21.15.192:6789/0 5 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 2234374065 0 0) 0x7f0160001b60 con 0x7f01700b1ad0 2024-06-03T09:07:23.290 INFO:teuthology.orchestra.run.smithi088.stderr:2024-06-03T09:07:23.286+0000 7f01754f6640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-06-03T09:07:23.290 INFO:teuthology.orchestra.run.smithi088.stderr:2024-06-03T09:07:23.286+0000 7f01754f6640 10 monclient: got monmap 1 from mon.b (according to old e1) 2024-06-03T09:07:23.290 INFO:teuthology.orchestra.run.smithi088.stderr:2024-06-03T09:07:23.286+0000 7f01754f6640 10 monclient: dump: 2024-06-03T09:07:23.290 INFO:teuthology.orchestra.run.smithi088.stderr:epoch 1 2024-06-03T09:07:23.290 INFO:teuthology.orchestra.run.smithi088.stderr:fsid ec3c3fdd-3ac6-4d7d-9412-683130ed8b04 2024-06-03T09:07:23.290 INFO:teuthology.orchestra.run.smithi088.stderr:last_changed 2024-06-03T08:34:31.885125+0000 2024-06-03T09:07:23.290 INFO:teuthology.orchestra.run.smithi088.stderr:created 2024-06-03T08:34:31.885125+0000 2024-06-03T09:07:23.290 INFO:teuthology.orchestra.run.smithi088.stderr:min_mon_release 18 (reef) 2024-06-03T09:07:23.290 INFO:teuthology.orchestra.run.smithi088.stderr:election_strategy: 3 2024-06-03T09:07:23.290 INFO:teuthology.orchestra.run.smithi088.stderr:0: v1:172.21.15.88:6789/0 mon.a 2024-06-03T09:07:23.290 INFO:teuthology.orchestra.run.smithi088.stderr:1: v1:172.21.15.192:6789/0 mon.b 2024-06-03T09:07:23.290 INFO:teuthology.orchestra.run.smithi088.stderr:2: v1:172.21.15.88:6790/0 mon.c 2024-06-03T09:07:23.291 INFO:teuthology.orchestra.run.smithi088.stderr: 2024-06-03T09:07:23.291 INFO:teuthology.orchestra.run.smithi088.stderr:2024-06-03T09:07:23.286+0000 7f017875c640 10 monclient: get_monmap_and_config success 2024-06-03T09:07:23.291 INFO:teuthology.orchestra.run.smithi088.stderr:2024-06-03T09:07:23.286+0000 7f017875c640 4 set_mon_vals no callback set 2024-06-03T09:07:23.291 INFO:teuthology.orchestra.run.smithi088.stderr:2024-06-03T09:07:23.287+0000 7f017875c640 10 monclient: shutdown 2024-06-03T09:07:23.291 INFO:teuthology.orchestra.run.smithi088.stderr:2024-06-03T09:07:23.287+0000 7f017875c640 20 monclient: shutdown discarding 0 pending message(s) 2024-06-03T09:07:23.291 INFO:teuthology.orchestra.run.smithi088.stderr:2024-06-03T09:07:23.287+0000 7f017875c640 1 -- 172.21.15.88:0/4009955801 >> v1:172.21.15.192:6789/0 conn(0x7f01700b1ad0 legacy=0x7f017013e8d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-06-03T09:07:23.291 INFO:teuthology.orchestra.run.smithi088.stderr:2024-06-03T09:07:23.287+0000 7f017875c640 1 -- 172.21.15.88:0/4009955801 shutdown_connections 2024-06-03T09:07:23.291 INFO:teuthology.orchestra.run.smithi088.stderr:2024-06-03T09:07:23.287+0000 7f017875c640 1 -- 172.21.15.88:0/4009955801 >> 172.21.15.88:0/4009955801 conn(0x7f01700b70d0 msgr2=0x7f01700b74e0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-06-03T09:07:23.291 INFO:teuthology.orchestra.run.smithi088.stderr:2024-06-03T09:07:23.287+0000 7f017875c640 1 -- 172.21.15.88:0/4009955801 shutdown_connections 2024-06-03T09:07:23.291 INFO:teuthology.orchestra.run.smithi088.stderr:2024-06-03T09:07:23.287+0000 7f017875c640 1 -- 172.21.15.88:0/4009955801 wait complete. 2024-06-03T09:07:23.291 INFO:teuthology.orchestra.run.smithi088.stderr:2024-06-03T09:07:23.287+0000 7f017875c640 10 monclient: build_initial_monmap 2024-06-03T09:07:23.291 INFO:teuthology.orchestra.run.smithi088.stderr:2024-06-03T09:07:23.287+0000 7f017875c640 10 monclient: monmap: 2024-06-03T09:07:23.291 INFO:teuthology.orchestra.run.smithi088.stderr:epoch 0 2024-06-03T09:07:23.292 INFO:teuthology.orchestra.run.smithi088.stderr:fsid 00000000-0000-0000-0000-000000000000 2024-06-03T09:07:23.292 INFO:teuthology.orchestra.run.smithi088.stderr:last_changed 0.000000 2024-06-03T09:07:23.292 INFO:teuthology.orchestra.run.smithi088.stderr:created 0.000000 2024-06-03T09:07:23.292 INFO:teuthology.orchestra.run.smithi088.stderr:min_mon_release 0 (unknown) 2024-06-03T09:07:23.292 INFO:teuthology.orchestra.run.smithi088.stderr:election_strategy: 1 2024-06-03T09:07:23.292 INFO:teuthology.orchestra.run.smithi088.stderr:0: v1:172.21.15.88:6789/0 mon.noname-a 2024-06-03T09:07:23.292 INFO:teuthology.orchestra.run.smithi088.stderr:1: v1:172.21.15.192:6789/0 mon.noname-b 2024-06-03T09:07:23.292 INFO:teuthology.orchestra.run.smithi088.stderr:2: v1:172.21.15.88:6790/0 mon.noname-c 2024-06-03T09:07:23.292 INFO:teuthology.orchestra.run.smithi088.stderr: 2024-06-03T09:07:23.292 INFO:teuthology.orchestra.run.smithi088.stderr:2024-06-03T09:07:23.288+0000 7f017875c640 1 librados: starting msgr at 2024-06-03T09:07:23.292 INFO:teuthology.orchestra.run.smithi088.stderr:2024-06-03T09:07:23.288+0000 7f017875c640 1 librados: starting objecter 2024-06-03T09:07:23.292 INFO:teuthology.orchestra.run.smithi088.stderr:2024-06-03T09:07:23.288+0000 7f017875c640 1 Processor -- start 2024-06-03T09:07:23.292 INFO:teuthology.orchestra.run.smithi088.stderr:2024-06-03T09:07:23.288+0000 7f017875c640 1 -- start start 2024-06-03T09:07:23.293 INFO:teuthology.orchestra.run.smithi088.stderr:2024-06-03T09:07:23.288+0000 7f017875c640 1 librados: setting wanted keys 2024-06-03T09:07:23.293 INFO:teuthology.orchestra.run.smithi088.stderr:2024-06-03T09:07:23.288+0000 7f017875c640 1 librados: calling monclient init 2024-06-03T09:07:23.293 INFO:teuthology.orchestra.run.smithi088.stderr:2024-06-03T09:07:23.288+0000 7f017875c640 10 monclient: init 2024-06-03T09:07:23.293 INFO:teuthology.orchestra.run.smithi088.stderr:2024-06-03T09:07:23.288+0000 7f017875c640 10 monclient: _reopen_session rank -1 2024-06-03T09:07:23.293 INFO:teuthology.orchestra.run.smithi088.stderr:2024-06-03T09:07:23.288+0000 7f017875c640 10 monclient: _add_conns ranks=[1,0,2] 2024-06-03T09:07:23.293 INFO:teuthology.orchestra.run.smithi088.stderr:2024-06-03T09:07:23.288+0000 7f017875c640 10 monclient(hunting): picked mon.noname-b con 0x7f01700a8c10 addr v1:172.21.15.192:6789/0 2024-06-03T09:07:23.293 INFO:teuthology.orchestra.run.smithi088.stderr:2024-06-03T09:07:23.288+0000 7f017875c640 10 monclient(hunting): picked mon.noname-a con 0x7f01700b1ad0 addr v1:172.21.15.88:6789/0 2024-06-03T09:07:23.293 INFO:teuthology.orchestra.run.smithi088.stderr:2024-06-03T09:07:23.288+0000 7f017875c640 10 monclient(hunting): picked mon.noname-c con 0x7f017014ae30 addr v1:172.21.15.88:6790/0 2024-06-03T09:07:23.293 INFO:teuthology.orchestra.run.smithi088.stderr:2024-06-03T09:07:23.288+0000 7f017875c640 1 -- --> v1:172.21.15.88:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f01701d6470 con 0x7f01700b1ad0 2024-06-03T09:07:23.293 INFO:teuthology.orchestra.run.smithi088.stderr:2024-06-03T09:07:23.288+0000 7f017875c640 1 -- --> v1:172.21.15.192:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f0170143c40 con 0x7f01700a8c10 2024-06-03T09:07:23.293 INFO:teuthology.orchestra.run.smithi088.stderr:2024-06-03T09:07:23.288+0000 7f017875c640 1 -- --> v1:172.21.15.88:6790/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f0170138f50 con 0x7f017014ae30 2024-06-03T09:07:23.293 INFO:teuthology.orchestra.run.smithi088.stderr:2024-06-03T09:07:23.288+0000 7f017875c640 10 monclient(hunting): _renew_subs 2024-06-03T09:07:23.293 INFO:teuthology.orchestra.run.smithi088.stderr:2024-06-03T09:07:23.288+0000 7f017875c640 10 monclient(hunting): authenticate will time out at 3099.098893s 2024-06-03T09:07:23.293 INFO:teuthology.orchestra.run.smithi088.stderr:2024-06-03T09:07:23.289+0000 7f0175cf7640 1 --1- >> v1:172.21.15.88:6789/0 conn(0x7f01700b1ad0 0x7f01701d6900 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=1).handle_server_banner_and_identify peer v1:172.21.15.88:6789/0 says I am v1:172.21.15.88:38364/0 (socket says 172.21.15.88:38364) 2024-06-03T09:07:23.294 INFO:teuthology.orchestra.run.smithi088.stderr:2024-06-03T09:07:23.289+0000 7f0175cf7640 1 -- 172.21.15.88:0/241813640 learned_addr learned my addr 172.21.15.88:0/241813640 (peer_addr_for_me v1:172.21.15.88:0/0) 2024-06-03T09:07:23.294 INFO:teuthology.orchestra.run.smithi088.stderr:2024-06-03T09:07:23.289+0000 7f0166ffd640 10 client.?.objecter ms_handle_connect 0x7f01700b1ad0 2024-06-03T09:07:23.294 INFO:teuthology.orchestra.run.smithi088.stderr:2024-06-03T09:07:23.289+0000 7f0166ffd640 10 client.?.objecter resend_mon_ops 2024-06-03T09:07:23.294 INFO:teuthology.orchestra.run.smithi088.stderr:2024-06-03T09:07:23.289+0000 7f0166ffd640 10 client.?.objecter ms_handle_connect 0x7f017014ae30 2024-06-03T09:07:23.294 INFO:teuthology.orchestra.run.smithi088.stderr:2024-06-03T09:07:23.289+0000 7f0166ffd640 10 client.?.objecter resend_mon_ops 2024-06-03T09:07:23.294 INFO:teuthology.orchestra.run.smithi088.stderr:2024-06-03T09:07:23.289+0000 7f0166ffd640 10 client.?.objecter ms_handle_connect 0x7f01700a8c10 2024-06-03T09:07:23.294 INFO:teuthology.orchestra.run.smithi088.stderr:2024-06-03T09:07:23.289+0000 7f0166ffd640 10 client.?.objecter resend_mon_ops 2024-06-03T09:07:23.295 INFO:teuthology.orchestra.run.smithi088.stderr:2024-06-03T09:07:23.291+0000 7f0166ffd640 1 -- 172.21.15.88:0/241813640 <== mon.0 v1:172.21.15.88:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 791537239 0 0) 0x7f01701d6470 con 0x7f01700b1ad0 2024-06-03T09:07:23.295 INFO:teuthology.orchestra.run.smithi088.stderr:2024-06-03T09:07:23.291+0000 7f0166ffd640 10 monclient(hunting): _negotiate 2024-06-03T09:07:23.295 INFO:teuthology.orchestra.run.smithi088.stderr:2024-06-03T09:07:23.291+0000 7f0166ffd640 10 monclient(hunting): _init_auth method 2 2024-06-03T09:07:23.295 INFO:teuthology.orchestra.run.smithi088.stderr:2024-06-03T09:07:23.291+0000 7f0166ffd640 10 monclient(hunting): _init_auth creating new auth 2024-06-03T09:07:23.295 INFO:teuthology.orchestra.run.smithi088.stderr:2024-06-03T09:07:23.291+0000 7f0166ffd640 10 monclient(hunting): my global_id is 30834 2024-06-03T09:07:23.295 INFO:teuthology.orchestra.run.smithi088.stderr:2024-06-03T09:07:23.291+0000 7f0166ffd640 1 -- 172.21.15.88:0/241813640 --> v1:172.21.15.88:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f0154004eb0 con 0x7f01700b1ad0 2024-06-03T09:07:23.295 INFO:teuthology.orchestra.run.smithi088.stderr:2024-06-03T09:07:23.291+0000 7f0166ffd640 1 -- 172.21.15.88:0/241813640 <== mon.2 v1:172.21.15.88:6790/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 817563057 0 0) 0x7f0170138f50 con 0x7f017014ae30 2024-06-03T09:07:23.295 INFO:teuthology.orchestra.run.smithi088.stderr:2024-06-03T09:07:23.291+0000 7f0166ffd640 10 monclient(hunting): _negotiate 2024-06-03T09:07:23.295 INFO:teuthology.orchestra.run.smithi088.stderr:2024-06-03T09:07:23.291+0000 7f0166ffd640 10 monclient(hunting): _init_auth method 2 2024-06-03T09:07:23.296 INFO:teuthology.orchestra.run.smithi088.stderr:2024-06-03T09:07:23.291+0000 7f0166ffd640 10 monclient(hunting): _init_auth creating new auth 2024-06-03T09:07:23.296 INFO:teuthology.orchestra.run.smithi088.stderr:2024-06-03T09:07:23.291+0000 7f0166ffd640 10 monclient(hunting): my global_id is 30344 2024-06-03T09:07:23.296 INFO:teuthology.orchestra.run.smithi088.stderr:2024-06-03T09:07:23.291+0000 7f0166ffd640 1 -- 172.21.15.88:0/241813640 --> v1:172.21.15.88:6790/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f01701d6470 con 0x7f017014ae30 2024-06-03T09:07:23.296 INFO:teuthology.orchestra.run.smithi088.stderr:2024-06-03T09:07:23.291+0000 7f0166ffd640 1 -- 172.21.15.88:0/241813640 <== mon.1 v1:172.21.15.192:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 2380108698 0 0) 0x7f0170143c40 con 0x7f01700a8c10 2024-06-03T09:07:23.296 INFO:teuthology.orchestra.run.smithi088.stderr:2024-06-03T09:07:23.291+0000 7f0166ffd640 10 monclient(hunting): _negotiate 2024-06-03T09:07:23.296 INFO:teuthology.orchestra.run.smithi088.stderr:2024-06-03T09:07:23.291+0000 7f0166ffd640 10 monclient(hunting): _init_auth method 2 2024-06-03T09:07:23.296 INFO:teuthology.orchestra.run.smithi088.stderr:2024-06-03T09:07:23.291+0000 7f0166ffd640 10 monclient(hunting): _init_auth creating new auth 2024-06-03T09:07:23.296 INFO:teuthology.orchestra.run.smithi088.stderr:2024-06-03T09:07:23.291+0000 7f0166ffd640 10 monclient(hunting): my global_id is 30409 2024-06-03T09:07:23.296 INFO:teuthology.orchestra.run.smithi088.stderr:2024-06-03T09:07:23.291+0000 7f0166ffd640 1 -- 172.21.15.88:0/241813640 --> v1:172.21.15.192:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f0170138f50 con 0x7f01700a8c10 2024-06-03T09:07:23.297 INFO:teuthology.orchestra.run.smithi088.stderr:2024-06-03T09:07:23.296+0000 7f0166ffd640 1 -- 172.21.15.88:0/241813640 <== mon.1 v1:172.21.15.192:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 764+0+0 (unknown 3238349615 0 0) 0x7f0170138f50 con 0x7f01700a8c10 2024-06-03T09:07:23.297 INFO:teuthology.orchestra.run.smithi088.stderr:2024-06-03T09:07:23.296+0000 7f0166ffd640 1 -- 172.21.15.88:0/241813640 --> v1:172.21.15.192:6789/0 -- auth(proto 2 165 bytes epoch 0) v1 -- 0x7f0170143c40 con 0x7f01700a8c10 2024-06-03T09:07:23.297 INFO:teuthology.orchestra.run.smithi088.stderr:2024-06-03T09:07:23.296+0000 7f0166ffd640 1 -- 172.21.15.88:0/241813640 <== mon.1 v1:172.21.15.192:6789/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 2234374065 0 0) 0x7f016c001c50 con 0x7f01700a8c10 2024-06-03T09:07:23.297 INFO:teuthology.orchestra.run.smithi088.stderr:2024-06-03T09:07:23.296+0000 7f0166ffd640 10 monclient(hunting): handle_monmap mon_map magic: 0 v1 2024-06-03T09:07:23.298 INFO:teuthology.orchestra.run.smithi088.stderr:2024-06-03T09:07:23.296+0000 7f0166ffd640 10 monclient(hunting): got monmap 1 from mon.noname-b (according to old e1) 2024-06-03T09:07:23.298 INFO:teuthology.orchestra.run.smithi088.stderr:2024-06-03T09:07:23.296+0000 7f0166ffd640 10 monclient(hunting): dump: 2024-06-03T09:07:23.298 INFO:teuthology.orchestra.run.smithi088.stderr:epoch 1 2024-06-03T09:07:23.298 INFO:teuthology.orchestra.run.smithi088.stderr:fsid ec3c3fdd-3ac6-4d7d-9412-683130ed8b04 2024-06-03T09:07:23.298 INFO:teuthology.orchestra.run.smithi088.stderr:last_changed 2024-06-03T08:34:31.885125+0000 2024-06-03T09:07:23.298 INFO:teuthology.orchestra.run.smithi088.stderr:created 2024-06-03T08:34:31.885125+0000 2024-06-03T09:07:23.298 INFO:teuthology.orchestra.run.smithi088.stderr:min_mon_release 18 (reef) 2024-06-03T09:07:23.298 INFO:teuthology.orchestra.run.smithi088.stderr:election_strategy: 3 2024-06-03T09:07:23.298 INFO:teuthology.orchestra.run.smithi088.stderr:0: v1:172.21.15.88:6789/0 mon.a 2024-06-03T09:07:23.298 INFO:teuthology.orchestra.run.smithi088.stderr:1: v1:172.21.15.192:6789/0 mon.b 2024-06-03T09:07:23.298 INFO:teuthology.orchestra.run.smithi088.stderr:2: v1:172.21.15.88:6790/0 mon.c 2024-06-03T09:07:23.298 INFO:teuthology.orchestra.run.smithi088.stderr: 2024-06-03T09:07:23.298 INFO:teuthology.orchestra.run.smithi088.stderr:2024-06-03T09:07:23.296+0000 7f0166ffd640 10 monclient(hunting): _finish_auth 0 2024-06-03T09:07:23.299 INFO:teuthology.orchestra.run.smithi088.stderr:2024-06-03T09:07:23.296+0000 7f0166ffd640 1 -- 172.21.15.88:0/241813640 <== mon.0 v1:172.21.15.88:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 764+0+0 (unknown 1384180015 0 0) 0x7f0154004eb0 con 0x7f01700b1ad0 2024-06-03T09:07:23.299 INFO:teuthology.orchestra.run.smithi088.stderr:2024-06-03T09:07:23.296+0000 7f0166ffd640 1 -- 172.21.15.88:0/241813640 --> v1:172.21.15.88:6789/0 -- auth(proto 2 165 bytes epoch 0) v1 -- 0x7f0170138f50 con 0x7f01700b1ad0 2024-06-03T09:07:23.299 INFO:teuthology.orchestra.run.smithi088.stderr:2024-06-03T09:07:23.296+0000 7f0166ffd640 1 -- 172.21.15.88:0/241813640 <== mon.0 v1:172.21.15.88:6789/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 2234374065 0 0) 0x7f0160001ba0 con 0x7f01700b1ad0 2024-06-03T09:07:23.299 INFO:teuthology.orchestra.run.smithi088.stderr:2024-06-03T09:07:23.296+0000 7f0166ffd640 10 monclient(hunting): handle_monmap mon_map magic: 0 v1 2024-06-03T09:07:23.299 INFO:teuthology.orchestra.run.smithi088.stderr:2024-06-03T09:07:23.296+0000 7f0166ffd640 10 monclient(hunting): got monmap 1 from mon.a (according to old e1) 2024-06-03T09:07:23.299 INFO:teuthology.orchestra.run.smithi088.stderr:2024-06-03T09:07:23.296+0000 7f0166ffd640 10 monclient(hunting): dump: 2024-06-03T09:07:23.299 INFO:teuthology.orchestra.run.smithi088.stderr:epoch 1 2024-06-03T09:07:23.299 INFO:teuthology.orchestra.run.smithi088.stderr:fsid ec3c3fdd-3ac6-4d7d-9412-683130ed8b04 2024-06-03T09:07:23.299 INFO:teuthology.orchestra.run.smithi088.stderr:last_changed 2024-06-03T08:34:31.885125+0000 2024-06-03T09:07:23.299 INFO:teuthology.orchestra.run.smithi088.stderr:created 2024-06-03T08:34:31.885125+0000 2024-06-03T09:07:23.299 INFO:teuthology.orchestra.run.smithi088.stderr:min_mon_release 18 (reef) 2024-06-03T09:07:23.299 INFO:teuthology.orchestra.run.smithi088.stderr:election_strategy: 3 2024-06-03T09:07:23.299 INFO:teuthology.orchestra.run.smithi088.stderr:0: v1:172.21.15.88:6789/0 mon.a 2024-06-03T09:07:23.300 INFO:teuthology.orchestra.run.smithi088.stderr:1: v1:172.21.15.192:6789/0 mon.b 2024-06-03T09:07:23.300 INFO:teuthology.orchestra.run.smithi088.stderr:2: v1:172.21.15.88:6790/0 mon.c 2024-06-03T09:07:23.300 INFO:teuthology.orchestra.run.smithi088.stderr: 2024-06-03T09:07:23.300 INFO:teuthology.orchestra.run.smithi088.stderr:2024-06-03T09:07:23.296+0000 7f0166ffd640 1 -- 172.21.15.88:0/241813640 <== mon.2 v1:172.21.15.88:6790/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 764+0+0 (unknown 537607345 0 0) 0x7f01701d6470 con 0x7f017014ae30 2024-06-03T09:07:23.300 INFO:teuthology.orchestra.run.smithi088.stderr:2024-06-03T09:07:23.296+0000 7f0166ffd640 1 -- 172.21.15.88:0/241813640 --> v1:172.21.15.88:6790/0 -- auth(proto 2 165 bytes epoch 0) v1 -- 0x7f0154004eb0 con 0x7f017014ae30 2024-06-03T09:07:23.300 INFO:teuthology.orchestra.run.smithi088.stderr:2024-06-03T09:07:23.296+0000 7f0166ffd640 1 -- 172.21.15.88:0/241813640 <== mon.2 v1:172.21.15.88:6790/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 2234374065 0 0) 0x7f0168001750 con 0x7f017014ae30 2024-06-03T09:07:23.300 INFO:teuthology.orchestra.run.smithi088.stderr:2024-06-03T09:07:23.296+0000 7f0166ffd640 10 monclient(hunting): handle_monmap mon_map magic: 0 v1 2024-06-03T09:07:23.300 INFO:teuthology.orchestra.run.smithi088.stderr:2024-06-03T09:07:23.296+0000 7f0166ffd640 10 monclient(hunting): got monmap 1 from mon.c (according to old e1) 2024-06-03T09:07:23.300 INFO:teuthology.orchestra.run.smithi088.stderr:2024-06-03T09:07:23.296+0000 7f0166ffd640 10 monclient(hunting): dump: 2024-06-03T09:07:23.300 INFO:teuthology.orchestra.run.smithi088.stderr:epoch 1 2024-06-03T09:07:23.300 INFO:teuthology.orchestra.run.smithi088.stderr:fsid ec3c3fdd-3ac6-4d7d-9412-683130ed8b04 2024-06-03T09:07:23.300 INFO:teuthology.orchestra.run.smithi088.stderr:last_changed 2024-06-03T08:34:31.885125+0000 2024-06-03T09:07:23.300 INFO:teuthology.orchestra.run.smithi088.stderr:created 2024-06-03T08:34:31.885125+0000 2024-06-03T09:07:23.300 INFO:teuthology.orchestra.run.smithi088.stderr:min_mon_release 18 (reef) 2024-06-03T09:07:23.301 INFO:teuthology.orchestra.run.smithi088.stderr:election_strategy: 3 2024-06-03T09:07:23.301 INFO:teuthology.orchestra.run.smithi088.stderr:0: v1:172.21.15.88:6789/0 mon.a 2024-06-03T09:07:23.301 INFO:teuthology.orchestra.run.smithi088.stderr:1: v1:172.21.15.192:6789/0 mon.b 2024-06-03T09:07:23.301 INFO:teuthology.orchestra.run.smithi088.stderr:2: v1:172.21.15.88:6790/0 mon.c 2024-06-03T09:07:23.301 INFO:teuthology.orchestra.run.smithi088.stderr: 2024-06-03T09:07:23.302 INFO:teuthology.orchestra.run.smithi088.stderr:2024-06-03T09:07:23.299+0000 7f0166ffd640 1 -- 172.21.15.88:0/241813640 <== mon.1 v1:172.21.15.192:6789/0 4 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 580+0+0 (unknown 3615970177 0 0) 0x7f0170143c40 con 0x7f01700a8c10 2024-06-03T09:07:23.302 INFO:teuthology.orchestra.run.smithi088.stderr:2024-06-03T09:07:23.299+0000 7f0166ffd640 1 -- 172.21.15.88:0/241813640 >> v1:172.21.15.88:6790/0 conn(0x7f017014ae30 legacy=0x7f0170133240 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-06-03T09:07:23.302 INFO:teuthology.orchestra.run.smithi088.stderr:2024-06-03T09:07:23.299+0000 7f0166ffd640 1 -- 172.21.15.88:0/241813640 >> v1:172.21.15.88:6789/0 conn(0x7f01700b1ad0 legacy=0x7f01701d6900 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-06-03T09:07:23.302 INFO:teuthology.orchestra.run.smithi088.stderr:2024-06-03T09:07:23.299+0000 7f0166ffd640 10 monclient: _finish_hunting 0 2024-06-03T09:07:23.302 INFO:teuthology.orchestra.run.smithi088.stderr:2024-06-03T09:07:23.299+0000 7f0166ffd640 1 monclient: found mon.b 2024-06-03T09:07:23.302 INFO:teuthology.orchestra.run.smithi088.stderr:2024-06-03T09:07:23.299+0000 7f0166ffd640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-06-03T09:07:23.302 INFO:teuthology.orchestra.run.smithi088.stderr:2024-06-03T09:07:23.299+0000 7f0166ffd640 10 monclient: _send_mon_message to mon.b at v1:172.21.15.192:6789/0 2024-06-03T09:07:23.303 INFO:teuthology.orchestra.run.smithi088.stderr:2024-06-03T09:07:23.299+0000 7f0166ffd640 1 -- 172.21.15.88:0/241813640 --> v1:172.21.15.192:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f01701390e0 con 0x7f01700a8c10 2024-06-03T09:07:23.303 INFO:teuthology.orchestra.run.smithi088.stderr:2024-06-03T09:07:23.299+0000 7f0166ffd640 10 monclient: _finish_auth 0 2024-06-03T09:07:23.303 INFO:teuthology.orchestra.run.smithi088.stderr:2024-06-03T09:07:23.299+0000 7f0166ffd640 10 monclient: _check_auth_tickets 2024-06-03T09:07:23.303 INFO:teuthology.orchestra.run.smithi088.stderr:2024-06-03T09:07:23.299+0000 7f0166ffd640 20 monclient: _check_auth_rotating not needed by client.admin 2024-06-03T09:07:23.303 INFO:teuthology.orchestra.run.smithi088.stderr:2024-06-03T09:07:23.299+0000 7f017875c640 5 monclient: authenticate success, global_id 30409 2024-06-03T09:07:23.303 INFO:teuthology.orchestra.run.smithi088.stderr:2024-06-03T09:07:23.299+0000 7f017875c640 10 monclient: _renew_subs 2024-06-03T09:07:23.303 INFO:teuthology.orchestra.run.smithi088.stderr:2024-06-03T09:07:23.299+0000 7f017875c640 10 monclient: _send_mon_message to mon.b at v1:172.21.15.192:6789/0 2024-06-03T09:07:23.303 INFO:teuthology.orchestra.run.smithi088.stderr:2024-06-03T09:07:23.299+0000 7f017875c640 1 -- 172.21.15.88:0/241813640 --> v1:172.21.15.192:6789/0 -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f017013f380 con 0x7f01700a8c10 2024-06-03T09:07:23.303 INFO:teuthology.orchestra.run.smithi088.stderr:2024-06-03T09:07:23.299+0000 7f017875c640 10 client.30409.objecter _maybe_request_map subscribing (onetime) to next osd map 2024-06-03T09:07:23.303 INFO:teuthology.orchestra.run.smithi088.stderr:2024-06-03T09:07:23.299+0000 7f017875c640 10 monclient: _renew_subs 2024-06-03T09:07:23.303 INFO:teuthology.orchestra.run.smithi088.stderr:2024-06-03T09:07:23.299+0000 7f017875c640 10 monclient: _send_mon_message to mon.b at v1:172.21.15.192:6789/0 2024-06-03T09:07:23.303 INFO:teuthology.orchestra.run.smithi088.stderr:2024-06-03T09:07:23.299+0000 7f017875c640 1 -- 172.21.15.88:0/241813640 --> v1:172.21.15.192:6789/0 -- mon_subscribe({osdmap=0}) v3 -- 0x7f0170144100 con 0x7f01700a8c10 2024-06-03T09:07:23.303 INFO:teuthology.orchestra.run.smithi088.stderr:2024-06-03T09:07:23.299+0000 7f017875c640 1 librados: init done 2024-06-03T09:07:23.303 INFO:teuthology.orchestra.run.smithi088.stderr:2024-06-03T09:07:23.300+0000 7f017875c640 10 start_mon_command cmd=[{"prefix": "get_command_descriptions"}] 2024-06-03T09:07:23.304 INFO:teuthology.orchestra.run.smithi088.stderr:2024-06-03T09:07:23.300+0000 7f017875c640 10 monclient: _send_command 1 [{"prefix": "get_command_descriptions"}] 2024-06-03T09:07:23.304 INFO:teuthology.orchestra.run.smithi088.stderr:2024-06-03T09:07:23.300+0000 7f017875c640 10 monclient: _send_mon_message to mon.b at v1:172.21.15.192:6789/0 2024-06-03T09:07:23.304 INFO:teuthology.orchestra.run.smithi088.stderr:2024-06-03T09:07:23.300+0000 7f017875c640 1 -- 172.21.15.88:0/241813640 --> v1:172.21.15.192:6789/0 -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f017004ea60 con 0x7f01700a8c10 2024-06-03T09:07:23.315 INFO:teuthology.orchestra.run.smithi088.stderr:2024-06-03T09:07:23.315+0000 7f0166ffd640 1 -- 172.21.15.88:0/241813640 <== mon.1 v1:172.21.15.192:6789/0 5 ==== config(0 keys) v1 ==== 4+0+0 (unknown 0 0 0) 0x7f016c003990 con 0x7f01700a8c10 2024-06-03T09:07:23.315 INFO:teuthology.orchestra.run.smithi088.stderr:2024-06-03T09:07:23.315+0000 7f0166ffd640 10 monclient: handle_config config(0 keys) v1 2024-06-03T09:07:23.315 INFO:teuthology.orchestra.run.smithi088.stderr:2024-06-03T09:07:23.315+0000 7f0166ffd640 1 -- 172.21.15.88:0/241813640 <== mon.1 v1:172.21.15.192:6789/0 6 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 2234374065 0 0) 0x7f016c003b00 con 0x7f01700a8c10 2024-06-03T09:07:23.315 INFO:teuthology.orchestra.run.smithi088.stderr:2024-06-03T09:07:23.315+0000 7f0166ffd640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-06-03T09:07:23.316 INFO:teuthology.orchestra.run.smithi088.stderr:2024-06-03T09:07:23.315+0000 7f0166ffd640 10 monclient: got monmap 1 from mon.b (according to old e1) 2024-06-03T09:07:23.316 INFO:teuthology.orchestra.run.smithi088.stderr:2024-06-03T09:07:23.315+0000 7f0166ffd640 10 monclient: dump: 2024-06-03T09:07:23.316 INFO:teuthology.orchestra.run.smithi088.stderr:epoch 1 2024-06-03T09:07:23.316 INFO:teuthology.orchestra.run.smithi088.stderr:fsid ec3c3fdd-3ac6-4d7d-9412-683130ed8b04 2024-06-03T09:07:23.316 INFO:teuthology.orchestra.run.smithi088.stderr:last_changed 2024-06-03T08:34:31.885125+0000 2024-06-03T09:07:23.316 INFO:teuthology.orchestra.run.smithi088.stderr:created 2024-06-03T08:34:31.885125+0000 2024-06-03T09:07:23.316 INFO:teuthology.orchestra.run.smithi088.stderr:min_mon_release 18 (reef) 2024-06-03T09:07:23.316 INFO:teuthology.orchestra.run.smithi088.stderr:election_strategy: 3 2024-06-03T09:07:23.316 INFO:teuthology.orchestra.run.smithi088.stderr:0: v1:172.21.15.88:6789/0 mon.a 2024-06-03T09:07:23.316 INFO:teuthology.orchestra.run.smithi088.stderr:1: v1:172.21.15.192:6789/0 mon.b 2024-06-03T09:07:23.316 INFO:teuthology.orchestra.run.smithi088.stderr:2: v1:172.21.15.88:6790/0 mon.c 2024-06-03T09:07:23.316 INFO:teuthology.orchestra.run.smithi088.stderr: 2024-06-03T09:07:23.316 INFO:teuthology.orchestra.run.smithi088.stderr:2024-06-03T09:07:23.315+0000 7f0174cf5640 4 set_mon_vals no callback set 2024-06-03T09:07:23.317 INFO:teuthology.orchestra.run.smithi088.stderr:2024-06-03T09:07:23.316+0000 7f0166ffd640 1 -- 172.21.15.88:0/241813640 <== mon.1 v1:172.21.15.192:6789/0 7 ==== mgrmap(e 5) v1 ==== 89110+0+0 (unknown 873533973 0 0) 0x7f016c019880 con 0x7f01700a8c10 2024-06-03T09:07:23.317 INFO:teuthology.orchestra.run.smithi088.stderr:2024-06-03T09:07:23.316+0000 7f0175cf7640 10 monclient: get_auth_request con 0x7f015406d420 auth_method 0 2024-06-03T09:07:23.318 INFO:teuthology.orchestra.run.smithi088.stderr:2024-06-03T09:07:23.316+0000 7f0166ffd640 10 client.30409.objecter ms_handle_connect 0x7f015406d420 2024-06-03T09:07:23.318 INFO:teuthology.orchestra.run.smithi088.stderr:2024-06-03T09:07:23.317+0000 7f0166ffd640 1 -- 172.21.15.88:0/241813640 <== mon.1 v1:172.21.15.192:6789/0 8 ==== osd_map(1014..1014 src has 735..1014) v4 ==== 5229+0+0 (unknown 332246716 0 0) 0x7f016c052f90 con 0x7f01700a8c10 2024-06-03T09:07:23.318 INFO:teuthology.orchestra.run.smithi088.stderr:2024-06-03T09:07:23.317+0000 7f0166ffd640 10 client.30409.objecter ms_dispatch 0x7f0170000b90 osd_map(1014..1014 src has 735..1014) v4 2024-06-03T09:07:23.318 INFO:teuthology.orchestra.run.smithi088.stderr:2024-06-03T09:07:23.317+0000 7f0166ffd640 3 client.30409.objecter handle_osd_map got epochs [1014,1014] > 0 2024-06-03T09:07:23.318 INFO:teuthology.orchestra.run.smithi088.stderr:2024-06-03T09:07:23.317+0000 7f0166ffd640 3 client.30409.objecter handle_osd_map decoding full epoch 1014 2024-06-03T09:07:23.318 INFO:teuthology.orchestra.run.smithi088.stderr:2024-06-03T09:07:23.317+0000 7f0166ffd640 20 client.30409.objecter dump_active .. 0 homeless 2024-06-03T09:07:23.546 INFO:teuthology.orchestra.run.smithi088.stderr:2024-06-03T09:07:23.546+0000 7f0166ffd640 1 -- 172.21.15.88:0/241813640 <== mon.1 v1:172.21.15.192:6789/0 9 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+182498 (unknown 1092875540 0 3016760700) 0x7f016c04eb40 con 0x7f01700a8c10 2024-06-03T09:07:23.547 INFO:teuthology.orchestra.run.smithi088.stderr:2024-06-03T09:07:23.546+0000 7f0166ffd640 10 monclient: handle_mon_command_ack 1 [{"prefix": "get_command_descriptions"}] 2024-06-03T09:07:23.547 INFO:teuthology.orchestra.run.smithi088.stderr:2024-06-03T09:07:23.546+0000 7f0166ffd640 10 monclient: _finish_command 1 = system:0 2024-06-03T09:07:23.717 INFO:teuthology.orchestra.run.smithi088.stderr:2024-06-03T09:07:23.716+0000 7f017875c640 1 -- 172.21.15.88:0/241813640 --> v1:172.21.15.192:6800/34229 -- mgr_command(tid 0: {"prefix": "pg dump", "target": ["mon-mgr", ""], "format": "json"}) v1 -- 0x7f01700aa2e0 con 0x7f015406d420 2024-06-03T09:07:23.721 INFO:teuthology.orchestra.run.smithi088.stderr:2024-06-03T09:07:23.722+0000 7f0166ffd640 1 -- 172.21.15.88:0/241813640 <== mgr.4110 v1:172.21.15.192:6800/34229 1 ==== mgr_command_reply(tid 0: 0 dumped all) v1 ==== 18+0+152136 (unknown 2965378022 0 1952366288) 0x7f0160007a50 con 0x7f015406d420 2024-06-03T09:07:23.723 INFO:teuthology.orchestra.run.smithi088.stdout: 2024-06-03T09:07:23.724 INFO:teuthology.orchestra.run.smithi088.stderr:dumped all 2024-06-03T09:07:23.726 INFO:teuthology.orchestra.run.smithi088.stderr:2024-06-03T09:07:23.724+0000 7f017875c640 10 librados: watch_flush enter 2024-06-03T09:07:23.726 INFO:teuthology.orchestra.run.smithi088.stderr:2024-06-03T09:07:23.724+0000 7f017875c640 10 librados: watch_flush exit 2024-06-03T09:07:23.726 INFO:teuthology.orchestra.run.smithi088.stderr:2024-06-03T09:07:23.724+0000 7f017875c640 20 client.30409.objecter shutdown clearing up homeless session... 2024-06-03T09:07:23.726 INFO:teuthology.orchestra.run.smithi088.stderr:2024-06-03T09:07:23.724+0000 7f017875c640 10 client.30409.objecter successfully canceled tick 2024-06-03T09:07:23.726 INFO:teuthology.orchestra.run.smithi088.stderr:2024-06-03T09:07:23.724+0000 7f017875c640 1 -- 172.21.15.88:0/241813640 >> v1:172.21.15.192:6800/34229 conn(0x7f015406d420 legacy=0x7f015406f8e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-06-03T09:07:23.726 INFO:teuthology.orchestra.run.smithi088.stderr:2024-06-03T09:07:23.724+0000 7f017875c640 10 monclient: shutdown 2024-06-03T09:07:23.726 INFO:teuthology.orchestra.run.smithi088.stderr:2024-06-03T09:07:23.724+0000 7f017875c640 20 monclient: shutdown discarding 0 pending message(s) 2024-06-03T09:07:23.726 INFO:teuthology.orchestra.run.smithi088.stderr:2024-06-03T09:07:23.724+0000 7f017875c640 1 -- 172.21.15.88:0/241813640 >> v1:172.21.15.192:6789/0 conn(0x7f01700a8c10 legacy=0x7f0170143530 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-06-03T09:07:23.726 INFO:teuthology.orchestra.run.smithi088.stderr:2024-06-03T09:07:23.724+0000 7f017875c640 1 -- 172.21.15.88:0/241813640 shutdown_connections 2024-06-03T09:07:23.726 INFO:teuthology.orchestra.run.smithi088.stderr:2024-06-03T09:07:23.724+0000 7f017875c640 1 -- 172.21.15.88:0/241813640 >> 172.21.15.88:0/241813640 conn(0x7f01700b70d0 msgr2=0x7f01700b74b0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-06-03T09:07:23.726 INFO:teuthology.orchestra.run.smithi088.stderr:2024-06-03T09:07:23.724+0000 7f017875c640 1 -- 172.21.15.88:0/241813640 shutdown_connections 2024-06-03T09:07:23.727 INFO:teuthology.orchestra.run.smithi088.stderr:2024-06-03T09:07:23.724+0000 7f017875c640 1 -- 172.21.15.88:0/241813640 wait complete. 2024-06-03T09:07:23.727 INFO:teuthology.orchestra.run.smithi088.stderr:2024-06-03T09:07:23.724+0000 7f017875c640 1 librados: shutdown 2024-06-03T09:07:23.736 INFO:teuthology.orchestra.run.smithi088.stdout:{"pg_ready":true,"pg_map":{"version":1951,"stamp":"2024-06-03T09:07:23.112654+0000","last_osdmap_epoch":0,"last_pg_scan":0,"pg_stats_sum":{"stat_sum":{"num_bytes":459285,"num_objects":3,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":3,"num_whiteouts":0,"num_read":102,"num_read_kb":88,"num_write":120,"num_write_kb":1379,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2,"num_bytes_recovered":397840,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":86,"ondisk_log_size":86,"up":82,"acting":82,"num_store_stats":0},"osd_stats_sum":{"up_from":0,"seq":0,"num_pgs":82,"num_osds":8,"num_per_pool_osds":8,"num_per_pool_omap_osds":8,"kb":754974720,"kb_used":495000,"kb_used_data":46808,"kb_used_omap":12,"kb_used_meta":448115,"kb_avail":754479720,"statfs":{"total":773094113280,"available":772587233280,"internally_reserved":0,"allocated":47931392,"data_stored":22900274,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":12744,"internal_metadata":458870328},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":40,"apply_latency_ms":40,"commit_latency_ns":40000000,"apply_latency_ns":40000000},"alerts":[],"network_ping_times":[]},"pg_stats_delta":{"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":0,"acting":0,"num_store_stats":0,"stamp_delta":"12.002859"},"pg_stats":[{"pgid":"17.d","version":"0'0","reported_seq":958,"reported_epoch":1014,"state":"active+clean","last_fresh":"2024-06-03T09:07:15.243547+0000","last_change":"2024-06-03T09:07:15.243547+0000","last_active":"2024-06-03T09:07:15.243547+0000","last_peered":"2024-06-03T09:07:15.243547+0000","last_clean":"2024-06-03T09:07:15.243547+0000","last_became_active":"2024-06-03T08:38:19.253559+0000","last_became_peered":"2024-06-03T08:38:19.253559+0000","last_unstale":"2024-06-03T09:07:15.243547+0000","last_undegraded":"2024-06-03T09:07:15.243547+0000","last_fullsized":"2024-06-03T09:07:15.243547+0000","mapping_epoch":28,"log_start":"0'0","ondisk_log_start":"0'0","created":28,"last_epoch_clean":45,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-06-03T09:07:15.239490+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-06-03T09:07:15.239490+0000","last_clean_scrub_stamp":"2024-06-03T09:07:15.239490+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-06-04T09:27:58.788791+0000","scrub_duration":0.060706005,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1],"acting":[0,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"17.c","version":"0'0","reported_seq":967,"reported_epoch":1014,"state":"active+clean","last_fresh":"2024-06-03T09:06:59.334574+0000","last_change":"2024-06-03T09:06:46.317689+0000","last_active":"2024-06-03T09:06:59.334574+0000","last_peered":"2024-06-03T09:06:59.334574+0000","last_clean":"2024-06-03T09:06:59.334574+0000","last_became_active":"2024-06-03T09:06:46.312637+0000","last_became_peered":"2024-06-03T09:06:46.312637+0000","last_unstale":"2024-06-03T09:06:59.334574+0000","last_undegraded":"2024-06-03T09:06:59.334574+0000","last_fullsized":"2024-06-03T09:06:59.334574+0000","mapping_epoch":1013,"log_start":"0'0","ondisk_log_start":"0'0","created":28,"last_epoch_clean":1014,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-06-03T08:37:55.614497+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-06-03T08:37:55.614497+0000","last_clean_scrub_stamp":"2024-06-03T08:37:55.614497+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for deep scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,6],"acting":[3,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"17.f","version":"0'0","reported_seq":855,"reported_epoch":1014,"state":"active+clean","last_fresh":"2024-06-03T09:07:09.949918+0000","last_change":"2024-06-03T09:07:09.955521+0000","last_active":"2024-06-03T09:07:09.949918+0000","last_peered":"2024-06-03T09:07:09.949918+0000","last_clean":"2024-06-03T09:07:09.949918+0000","last_became_active":"2024-06-03T09:06:46.301958+0000","last_became_peered":"2024-06-03T09:06:46.301958+0000","last_unstale":"2024-06-03T09:07:09.949918+0000","last_undegraded":"2024-06-03T09:07:09.949918+0000","last_fullsized":"2024-06-03T09:07:09.949918+0000","mapping_epoch":1013,"log_start":"0'0","ondisk_log_start":"0'0","created":28,"last_epoch_clean":1014,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-06-03T09:07:09.948711+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-06-03T09:07:09.948711+0000","last_clean_scrub_stamp":"2024-06-03T09:07:09.948711+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-06-04T19:32:16.465187+0000","scrub_duration":0.073847419999999997,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,4],"acting":[6,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"17.e","version":"0'0","reported_seq":855,"reported_epoch":1014,"state":"active+clean","last_fresh":"2024-06-03T09:07:07.901760+0000","last_change":"2024-06-03T09:07:07.903376+0000","last_active":"2024-06-03T09:07:07.901760+0000","last_peered":"2024-06-03T09:07:07.901760+0000","last_clean":"2024-06-03T09:07:07.901760+0000","last_became_active":"2024-06-03T09:06:46.273941+0000","last_became_peered":"2024-06-03T09:06:46.273941+0000","last_unstale":"2024-06-03T09:07:07.901760+0000","last_undegraded":"2024-06-03T09:07:07.901760+0000","last_fullsized":"2024-06-03T09:07:07.901760+0000","mapping_epoch":1013,"log_start":"0'0","ondisk_log_start":"0'0","created":28,"last_epoch_clean":1014,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-06-03T09:07:07.901032+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-06-03T09:07:07.901032+0000","last_clean_scrub_stamp":"2024-06-03T09:07:07.901032+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-06-04T16:06:06.004585+0000","scrub_duration":0.047268636000000003,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,5],"acting":[6,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"17.9","version":"0'0","reported_seq":961,"reported_epoch":1014,"state":"active+clean","last_fresh":"2024-06-03T09:07:08.477260+0000","last_change":"2024-06-03T09:07:08.478860+0000","last_active":"2024-06-03T09:07:08.477260+0000","last_peered":"2024-06-03T09:07:08.477260+0000","last_clean":"2024-06-03T09:07:08.477260+0000","last_became_active":"2024-06-03T08:38:04.763428+0000","last_became_peered":"2024-06-03T08:38:04.763428+0000","last_unstale":"2024-06-03T09:07:08.477260+0000","last_undegraded":"2024-06-03T09:07:08.477260+0000","last_fullsized":"2024-06-03T09:07:08.477260+0000","mapping_epoch":28,"log_start":"0'0","ondisk_log_start":"0'0","created":28,"last_epoch_clean":33,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-06-03T09:07:08.476536+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-06-03T09:07:08.476536+0000","last_clean_scrub_stamp":"2024-06-03T09:07:08.476536+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-06-04T16:39:49.232748+0000","scrub_duration":0.044858958999999997,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,4],"acting":[7,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"17.8","version":"0'0","reported_seq":984,"reported_epoch":1014,"state":"active+clean","last_fresh":"2024-06-03T09:06:59.146085+0000","last_change":"2024-06-03T09:06:59.150900+0000","last_active":"2024-06-03T09:06:59.146085+0000","last_peered":"2024-06-03T09:06:59.146085+0000","last_clean":"2024-06-03T09:06:59.146085+0000","last_became_active":"2024-06-03T09:06:46.344756+0000","last_became_peered":"2024-06-03T09:06:46.344756+0000","last_unstale":"2024-06-03T09:06:59.146085+0000","last_undegraded":"2024-06-03T09:06:59.146085+0000","last_fullsized":"2024-06-03T09:06:59.146085+0000","mapping_epoch":1013,"log_start":"0'0","ondisk_log_start":"0'0","created":28,"last_epoch_clean":1014,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-06-03T09:06:59.144935+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-06-03T09:06:59.144935+0000","last_clean_scrub_stamp":"2024-06-03T09:06:59.144935+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-06-04T15:52:37.265440+0000","scrub_duration":0.031008068,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,6],"acting":[0,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"17.b","version":"0'0","reported_seq":855,"reported_epoch":1014,"state":"active+clean","last_fresh":"2024-06-03T09:07:06.907843+0000","last_change":"2024-06-03T09:07:06.909464+0000","last_active":"2024-06-03T09:07:06.907843+0000","last_peered":"2024-06-03T09:07:06.907843+0000","last_clean":"2024-06-03T09:07:06.907843+0000","last_became_active":"2024-06-03T09:06:46.345458+0000","last_became_peered":"2024-06-03T09:06:46.345458+0000","last_unstale":"2024-06-03T09:07:06.907843+0000","last_undegraded":"2024-06-03T09:07:06.907843+0000","last_fullsized":"2024-06-03T09:07:06.907843+0000","mapping_epoch":1013,"log_start":"0'0","ondisk_log_start":"0'0","created":28,"last_epoch_clean":1014,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-06-03T09:07:06.907110+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-06-03T09:07:06.907110+0000","last_clean_scrub_stamp":"2024-06-03T09:07:06.907110+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-06-04T18:08:34.356218+0000","scrub_duration":0.080445640999999998,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,2],"acting":[6,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"17.a","version":"0'0","reported_seq":941,"reported_epoch":1014,"state":"active+clean","last_fresh":"2024-06-03T09:07:02.857089+0000","last_change":"2024-06-03T09:07:02.858770+0000","last_active":"2024-06-03T09:07:02.857089+0000","last_peered":"2024-06-03T09:07:02.857089+0000","last_clean":"2024-06-03T09:07:02.857089+0000","last_became_active":"2024-06-03T08:38:05.809257+0000","last_became_peered":"2024-06-03T08:38:05.809257+0000","last_unstale":"2024-06-03T09:07:02.857089+0000","last_undegraded":"2024-06-03T09:07:02.857089+0000","last_fullsized":"2024-06-03T09:07:02.857089+0000","mapping_epoch":28,"log_start":"0'0","ondisk_log_start":"0'0","created":28,"last_epoch_clean":33,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-06-03T09:07:02.856327+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-06-03T09:07:02.856327+0000","last_clean_scrub_stamp":"2024-06-03T09:07:02.856327+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-06-04T11:53:39.468676+0000","scrub_duration":0.048148466000000001,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2],"acting":[1,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"17.5","version":"0'0","reported_seq":958,"reported_epoch":1014,"state":"active+clean","last_fresh":"2024-06-03T09:07:14.251865+0000","last_change":"2024-06-03T09:07:14.253701+0000","last_active":"2024-06-03T09:07:14.251865+0000","last_peered":"2024-06-03T09:07:14.251865+0000","last_clean":"2024-06-03T09:07:14.251865+0000","last_became_active":"2024-06-03T08:38:23.458039+0000","last_became_peered":"2024-06-03T08:38:23.458039+0000","last_unstale":"2024-06-03T09:07:14.251865+0000","last_undegraded":"2024-06-03T09:07:14.251865+0000","last_fullsized":"2024-06-03T09:07:14.251865+0000","mapping_epoch":28,"log_start":"0'0","ondisk_log_start":"0'0","created":28,"last_epoch_clean":49,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-06-03T09:07:14.251098+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-06-03T09:07:14.251098+0000","last_clean_scrub_stamp":"2024-06-03T09:07:14.251098+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-06-04T16:53:12.820223+0000","scrub_duration":0.060259851000000003,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,4],"acting":[0,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"17.4","version":"0'0","reported_seq":855,"reported_epoch":1014,"state":"active+clean","last_fresh":"2024-06-03T09:07:03.820792+0000","last_change":"2024-06-03T09:07:03.822380+0000","last_active":"2024-06-03T09:07:03.820792+0000","last_peered":"2024-06-03T09:07:03.820792+0000","last_clean":"2024-06-03T09:07:03.820792+0000","last_became_active":"2024-06-03T09:06:46.447534+0000","last_became_peered":"2024-06-03T09:06:46.447534+0000","last_unstale":"2024-06-03T09:07:03.820792+0000","last_undegraded":"2024-06-03T09:07:03.820792+0000","last_fullsized":"2024-06-03T09:07:03.820792+0000","mapping_epoch":1013,"log_start":"0'0","ondisk_log_start":"0'0","created":28,"last_epoch_clean":1014,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-06-03T09:07:03.820058+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-06-03T09:07:03.820058+0000","last_clean_scrub_stamp":"2024-06-03T09:07:03.820058+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-06-04T13:11:32.675403+0000","scrub_duration":0.027398738999999998,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,7],"acting":[6,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"17.7","version":"0'0","reported_seq":946,"reported_epoch":1014,"state":"active+clean","last_fresh":"2024-06-03T09:07:11.552030+0000","last_change":"2024-06-03T09:07:11.553608+0000","last_active":"2024-06-03T09:07:11.552030+0000","last_peered":"2024-06-03T09:07:11.552030+0000","last_clean":"2024-06-03T09:07:11.552030+0000","last_became_active":"2024-06-03T08:38:04.818246+0000","last_became_peered":"2024-06-03T08:38:04.818246+0000","last_unstale":"2024-06-03T09:07:11.552030+0000","last_undegraded":"2024-06-03T09:07:11.552030+0000","last_fullsized":"2024-06-03T09:07:11.552030+0000","mapping_epoch":28,"log_start":"0'0","ondisk_log_start":"0'0","created":28,"last_epoch_clean":34,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-06-03T09:07:11.551324+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-06-03T09:07:11.551324+0000","last_clean_scrub_stamp":"2024-06-03T09:07:11.551324+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-06-04T12:33:33.185479+0000","scrub_duration":0.064916682000000003,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,2],"acting":[5,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"17.6","version":"0'0","reported_seq":972,"reported_epoch":1014,"state":"active+clean","last_fresh":"2024-06-03T09:06:46.371177+0000","last_change":"2024-06-03T09:06:46.362623+0000","last_active":"2024-06-03T09:06:46.371177+0000","last_peered":"2024-06-03T09:06:46.371177+0000","last_clean":"2024-06-03T09:06:46.371177+0000","last_became_active":"2024-06-03T09:06:46.357698+0000","last_became_peered":"2024-06-03T09:06:46.357698+0000","last_unstale":"2024-06-03T09:06:46.371177+0000","last_undegraded":"2024-06-03T09:06:46.371177+0000","last_fullsized":"2024-06-03T09:06:46.371177+0000","mapping_epoch":1013,"log_start":"0'0","ondisk_log_start":"0'0","created":28,"last_epoch_clean":1014,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-06-03T08:37:55.614497+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-06-03T08:37:55.614497+0000","last_clean_scrub_stamp":"2024-06-03T08:37:55.614497+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for deep scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,6],"acting":[0,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"2.7","version":"0'0","reported_seq":957,"reported_epoch":1014,"state":"active+clean","last_fresh":"2024-06-03T09:07:12.435349+0000","last_change":"2024-06-03T09:07:12.439483+0000","last_active":"2024-06-03T09:07:12.435349+0000","last_peered":"2024-06-03T09:07:12.435349+0000","last_clean":"2024-06-03T09:07:12.435349+0000","last_became_active":"2024-06-03T08:36:30.380302+0000","last_became_peered":"2024-06-03T08:36:30.380302+0000","last_unstale":"2024-06-03T09:07:12.435349+0000","last_undegraded":"2024-06-03T09:07:12.435349+0000","last_fullsized":"2024-06-03T09:07:12.435349+0000","mapping_epoch":23,"log_start":"0'0","ondisk_log_start":"0'0","created":23,"last_epoch_clean":24,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-06-03T09:07:12.434624+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-06-03T09:07:12.434624+0000","last_clean_scrub_stamp":"2024-06-03T09:07:12.434624+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-06-04T09:11:59.759962+0000","scrub_duration":0.029406762999999999,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0],"acting":[3,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"17.14","version":"0'0","reported_seq":942,"reported_epoch":1014,"state":"active+clean","last_fresh":"2024-06-03T09:07:12.551344+0000","last_change":"2024-06-03T09:07:12.552995+0000","last_active":"2024-06-03T09:07:12.551344+0000","last_peered":"2024-06-03T09:07:12.551344+0000","last_clean":"2024-06-03T09:07:12.551344+0000","last_became_active":"2024-06-03T08:38:04.541781+0000","last_became_peered":"2024-06-03T08:38:04.541781+0000","last_unstale":"2024-06-03T09:07:12.551344+0000","last_undegraded":"2024-06-03T09:07:12.551344+0000","last_fullsized":"2024-06-03T09:07:12.551344+0000","mapping_epoch":28,"log_start":"0'0","ondisk_log_start":"0'0","created":28,"last_epoch_clean":34,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-06-03T09:07:12.550638+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-06-03T09:07:12.550638+0000","last_clean_scrub_stamp":"2024-06-03T09:07:12.550638+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-06-04T15:35:25.145190+0000","scrub_duration":0.041798374999999999,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,2],"acting":[5,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"2.6","version":"0'0","reported_seq":965,"reported_epoch":1014,"state":"active+clean","last_fresh":"2024-06-03T09:07:01.091185+0000","last_change":"2024-06-03T09:07:01.092831+0000","last_active":"2024-06-03T09:07:01.091185+0000","last_peered":"2024-06-03T09:07:01.091185+0000","last_clean":"2024-06-03T09:07:01.091185+0000","last_became_active":"2024-06-03T08:36:30.404665+0000","last_became_peered":"2024-06-03T08:36:30.404665+0000","last_unstale":"2024-06-03T09:07:01.091185+0000","last_undegraded":"2024-06-03T09:07:01.091185+0000","last_fullsized":"2024-06-03T09:07:01.091185+0000","mapping_epoch":23,"log_start":"0'0","ondisk_log_start":"0'0","created":23,"last_epoch_clean":24,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-06-03T09:07:01.090441+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-06-03T09:07:01.090441+0000","last_clean_scrub_stamp":"2024-06-03T09:07:01.090441+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-06-04T12:37:48.579420+0000","scrub_duration":0.038448474000000003,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,0],"acting":[4,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"17.15","version":"0'0","reported_seq":940,"reported_epoch":1014,"state":"active+clean","last_fresh":"2024-06-03T09:06:59.336193+0000","last_change":"2024-06-03T08:38:20.999903+0000","last_active":"2024-06-03T09:06:59.336193+0000","last_peered":"2024-06-03T09:06:59.336193+0000","last_clean":"2024-06-03T09:06:59.336193+0000","last_became_active":"2024-06-03T08:38:20.979523+0000","last_became_peered":"2024-06-03T08:38:20.979523+0000","last_unstale":"2024-06-03T09:06:59.336193+0000","last_undegraded":"2024-06-03T09:06:59.336193+0000","last_fullsized":"2024-06-03T09:06:59.336193+0000","mapping_epoch":28,"log_start":"0'0","ondisk_log_start":"0'0","created":28,"last_epoch_clean":47,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-06-03T08:37:55.614497+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-06-03T08:37:55.614497+0000","last_clean_scrub_stamp":"2024-06-03T08:37:55.614497+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for deep scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,1],"acting":[3,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"2.5","version":"0'0","reported_seq":958,"reported_epoch":1014,"state":"active+clean","last_fresh":"2024-06-03T09:07:17.567853+0000","last_change":"2024-06-03T09:07:17.569584+0000","last_active":"2024-06-03T09:07:17.567853+0000","last_peered":"2024-06-03T09:07:17.567853+0000","last_clean":"2024-06-03T09:07:17.567853+0000","last_became_active":"2024-06-03T08:36:30.019040+0000","last_became_peered":"2024-06-03T08:36:30.019040+0000","last_unstale":"2024-06-03T09:07:17.567853+0000","last_undegraded":"2024-06-03T09:07:17.567853+0000","last_fullsized":"2024-06-03T09:07:17.567853+0000","mapping_epoch":23,"log_start":"0'0","ondisk_log_start":"0'0","created":23,"last_epoch_clean":24,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-06-03T09:07:17.567142+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-06-03T09:07:17.567142+0000","last_clean_scrub_stamp":"2024-06-03T09:07:17.567142+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-06-04T16:31:04.548317+0000","scrub_duration":0.034251373000000002,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,4],"acting":[3,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"17.16","version":"0'0","reported_seq":997,"reported_epoch":1014,"state":"active+clean","last_fresh":"2024-06-03T09:06:59.969669+0000","last_change":"2024-06-03T09:06:59.969669+0000","last_active":"2024-06-03T09:06:59.969669+0000","last_peered":"2024-06-03T09:06:59.969669+0000","last_clean":"2024-06-03T09:06:59.969669+0000","last_became_active":"2024-06-03T09:06:46.379518+0000","last_became_peered":"2024-06-03T09:06:46.379518+0000","last_unstale":"2024-06-03T09:06:59.969669+0000","last_undegraded":"2024-06-03T09:06:59.969669+0000","last_fullsized":"2024-06-03T09:06:59.969669+0000","mapping_epoch":1013,"log_start":"0'0","ondisk_log_start":"0'0","created":28,"last_epoch_clean":1014,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-06-03T09:06:59.966233+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-06-03T09:06:59.966233+0000","last_clean_scrub_stamp":"2024-06-03T09:06:59.966233+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-06-04T18:45:08.505938+0000","scrub_duration":0.037443454000000001,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,6],"acting":[2,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"2.4","version":"0'0","reported_seq":976,"reported_epoch":1014,"state":"active+clean","last_fresh":"2024-06-03T09:07:03.492481+0000","last_change":"2024-06-03T09:07:03.494168+0000","last_active":"2024-06-03T09:07:03.492481+0000","last_peered":"2024-06-03T09:07:03.492481+0000","last_clean":"2024-06-03T09:07:03.492481+0000","last_became_active":"2024-06-03T09:06:46.371430+0000","last_became_peered":"2024-06-03T09:06:46.371430+0000","last_unstale":"2024-06-03T09:07:03.492481+0000","last_undegraded":"2024-06-03T09:07:03.492481+0000","last_fullsized":"2024-06-03T09:07:03.492481+0000","mapping_epoch":1013,"log_start":"0'0","ondisk_log_start":"0'0","created":23,"last_epoch_clean":1014,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-06-03T09:07:03.491693+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-06-03T09:07:03.491693+0000","last_clean_scrub_stamp":"2024-06-03T09:07:03.491693+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-06-04T13:54:17.148134+0000","scrub_duration":0.038561466000000003,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,6],"acting":[5,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"17.17","version":"0'0","reported_seq":971,"reported_epoch":1014,"state":"active+clean","last_fresh":"2024-06-03T09:07:02.032651+0000","last_change":"2024-06-03T09:07:02.032651+0000","last_active":"2024-06-03T09:07:02.032651+0000","last_peered":"2024-06-03T09:07:02.032651+0000","last_clean":"2024-06-03T09:07:02.032651+0000","last_became_active":"2024-06-03T08:38:38.467916+0000","last_became_peered":"2024-06-03T08:38:38.467916+0000","last_unstale":"2024-06-03T09:07:02.032651+0000","last_undegraded":"2024-06-03T09:07:02.032651+0000","last_fullsized":"2024-06-03T09:07:02.032651+0000","mapping_epoch":28,"log_start":"0'0","ondisk_log_start":"0'0","created":28,"last_epoch_clean":61,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-06-03T09:07:02.030144+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-06-03T09:07:02.030144+0000","last_clean_scrub_stamp":"2024-06-03T09:07:02.030144+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-06-04T11:23:00.182920+0000","scrub_duration":0.081032390999999995,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,7],"acting":[2,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"2.2","version":"0'0","reported_seq":987,"reported_epoch":1014,"state":"active+clean","last_fresh":"2024-06-03T09:07:04.375923+0000","last_change":"2024-06-03T09:07:04.377466+0000","last_active":"2024-06-03T09:07:04.375923+0000","last_peered":"2024-06-03T09:07:04.375923+0000","last_clean":"2024-06-03T09:07:04.375923+0000","last_became_active":"2024-06-03T09:06:46.284836+0000","last_became_peered":"2024-06-03T09:06:46.284836+0000","last_unstale":"2024-06-03T09:07:04.375923+0000","last_undegraded":"2024-06-03T09:07:04.375923+0000","last_fullsized":"2024-06-03T09:07:04.375923+0000","mapping_epoch":1013,"log_start":"0'0","ondisk_log_start":"0'0","created":23,"last_epoch_clean":1014,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-06-03T09:07:04.375206+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-06-03T09:07:04.375206+0000","last_clean_scrub_stamp":"2024-06-03T09:07:04.375206+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-06-04T12:17:58.541830+0000","scrub_duration":0.032522373,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,6],"acting":[3,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"17.11","version":"0'0","reported_seq":959,"reported_epoch":1014,"state":"active+clean","last_fresh":"2024-06-03T09:07:03.087030+0000","last_change":"2024-06-03T09:07:03.088717+0000","last_active":"2024-06-03T09:07:03.087030+0000","last_peered":"2024-06-03T09:07:03.087030+0000","last_clean":"2024-06-03T09:07:03.087030+0000","last_became_active":"2024-06-03T08:38:06.490204+0000","last_became_peered":"2024-06-03T08:38:06.490204+0000","last_unstale":"2024-06-03T09:07:03.087030+0000","last_undegraded":"2024-06-03T09:07:03.087030+0000","last_fullsized":"2024-06-03T09:07:03.087030+0000","mapping_epoch":28,"log_start":"0'0","ondisk_log_start":"0'0","created":28,"last_epoch_clean":32,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-06-03T09:07:03.086295+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-06-03T09:07:03.086295+0000","last_clean_scrub_stamp":"2024-06-03T09:07:03.086295+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-06-04T09:36:44.758386+0000","scrub_duration":0.038154848999999998,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,2],"acting":[4,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"2.1","version":"0'0","reported_seq":967,"reported_epoch":1014,"state":"active+clean","last_fresh":"2024-06-03T09:07:03.630924+0000","last_change":"2024-06-03T09:07:03.642560+0000","last_active":"2024-06-03T09:07:03.630924+0000","last_peered":"2024-06-03T09:07:03.630924+0000","last_clean":"2024-06-03T09:07:03.630924+0000","last_became_active":"2024-06-03T08:36:30.855073+0000","last_became_peered":"2024-06-03T08:36:30.855073+0000","last_unstale":"2024-06-03T09:07:03.630924+0000","last_undegraded":"2024-06-03T09:07:03.630924+0000","last_fullsized":"2024-06-03T09:07:03.630924+0000","mapping_epoch":23,"log_start":"0'0","ondisk_log_start":"0'0","created":23,"last_epoch_clean":25,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-06-03T09:07:03.628701+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-06-03T09:07:03.628701+0000","last_clean_scrub_stamp":"2024-06-03T09:07:03.628701+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-06-04T11:26:15.349709+0000","scrub_duration":0.19363576900000001,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,5],"acting":[7,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"17.12","version":"0'0","reported_seq":960,"reported_epoch":1014,"state":"active+clean","last_fresh":"2024-06-03T09:07:07.469298+0000","last_change":"2024-06-03T09:07:07.470940+0000","last_active":"2024-06-03T09:07:07.469298+0000","last_peered":"2024-06-03T09:07:07.469298+0000","last_clean":"2024-06-03T09:07:07.469298+0000","last_became_active":"2024-06-03T08:38:01.745027+0000","last_became_peered":"2024-06-03T08:38:01.745027+0000","last_unstale":"2024-06-03T09:07:07.469298+0000","last_undegraded":"2024-06-03T09:07:07.469298+0000","last_fullsized":"2024-06-03T09:07:07.469298+0000","mapping_epoch":28,"log_start":"0'0","ondisk_log_start":"0'0","created":28,"last_epoch_clean":32,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-06-03T09:07:07.468586+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-06-03T09:07:07.468586+0000","last_clean_scrub_stamp":"2024-06-03T09:07:07.468586+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-06-04T19:15:15.696921+0000","scrub_duration":0.03822209,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,3],"acting":[7,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"2.0","version":"0'0","reported_seq":960,"reported_epoch":1014,"state":"active+clean","last_fresh":"2024-06-03T09:07:00.260891+0000","last_change":"2024-06-03T09:07:00.262894+0000","last_active":"2024-06-03T09:07:00.260891+0000","last_peered":"2024-06-03T09:07:00.260891+0000","last_clean":"2024-06-03T09:07:00.260891+0000","last_became_active":"2024-06-03T08:36:29.957875+0000","last_became_peered":"2024-06-03T08:36:29.957875+0000","last_unstale":"2024-06-03T09:07:00.260891+0000","last_undegraded":"2024-06-03T09:07:00.260891+0000","last_fullsized":"2024-06-03T09:07:00.260891+0000","mapping_epoch":23,"log_start":"0'0","ondisk_log_start":"0'0","created":23,"last_epoch_clean":24,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-06-03T09:07:00.260150+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-06-03T09:07:00.260150+0000","last_clean_scrub_stamp":"2024-06-03T09:07:00.260150+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-06-04T12:21:00.168796+0000","scrub_duration":0.031538745999999999,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,5],"acting":[3,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"17.13","version":"0'0","reported_seq":958,"reported_epoch":1014,"state":"active+clean","last_fresh":"2024-06-03T09:07:05.211420+0000","last_change":"2024-06-03T09:07:05.213047+0000","last_active":"2024-06-03T09:07:05.211420+0000","last_peered":"2024-06-03T09:07:05.211420+0000","last_clean":"2024-06-03T09:07:05.211420+0000","last_became_active":"2024-06-03T08:38:26.833214+0000","last_became_peered":"2024-06-03T08:38:26.833214+0000","last_unstale":"2024-06-03T09:07:05.211420+0000","last_undegraded":"2024-06-03T09:07:05.211420+0000","last_fullsized":"2024-06-03T09:07:05.211420+0000","mapping_epoch":28,"log_start":"0'0","ondisk_log_start":"0'0","created":28,"last_epoch_clean":52,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-06-03T09:07:05.210654+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-06-03T09:07:05.210654+0000","last_clean_scrub_stamp":"2024-06-03T09:07:05.210654+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-06-04T12:08:44.116700+0000","scrub_duration":0.087386555000000005,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,0],"acting":[4,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"2.3","version":"0'0","reported_seq":959,"reported_epoch":1014,"state":"active+clean","last_fresh":"2024-06-03T09:07:11.388404+0000","last_change":"2024-06-03T09:07:11.394811+0000","last_active":"2024-06-03T09:07:11.388404+0000","last_peered":"2024-06-03T09:07:11.388404+0000","last_clean":"2024-06-03T09:07:11.388404+0000","last_became_active":"2024-06-03T08:36:30.343560+0000","last_became_peered":"2024-06-03T08:36:30.343560+0000","last_unstale":"2024-06-03T09:07:11.388404+0000","last_undegraded":"2024-06-03T09:07:11.388404+0000","last_fullsized":"2024-06-03T09:07:11.388404+0000","mapping_epoch":23,"log_start":"0'0","ondisk_log_start":"0'0","created":23,"last_epoch_clean":24,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-06-03T09:07:11.387606+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-06-03T09:07:11.387606+0000","last_clean_scrub_stamp":"2024-06-03T09:07:11.387606+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-06-04T14:12:02.854001+0000","scrub_duration":0.033186076000000002,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,2],"acting":[3,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.0","version":"1014'78","reported_seq":1070,"reported_epoch":1014,"state":"active+clean","last_fresh":"2024-06-03T09:06:57.887708+0000","last_change":"2024-06-03T09:06:57.889997+0000","last_active":"2024-06-03T09:06:57.887708+0000","last_peered":"2024-06-03T09:06:57.887708+0000","last_clean":"2024-06-03T09:06:57.887708+0000","last_became_active":"2024-06-03T08:36:26.128378+0000","last_became_peered":"2024-06-03T08:36:26.128378+0000","last_unstale":"2024-06-03T09:06:57.887708+0000","last_undegraded":"2024-06-03T09:06:57.887708+0000","last_fullsized":"2024-06-03T09:06:57.887708+0000","mapping_epoch":20,"log_start":"0'0","ondisk_log_start":"0'0","created":16,"last_epoch_clean":22,"parent":"0.0","parent_split_bits":0,"last_scrub":"1014'78","last_scrub_stamp":"2024-06-03T09:06:57.886322+0000","last_deep_scrub":"1014'78","last_deep_scrub_stamp":"2024-06-03T09:06:57.886322+0000","last_clean_scrub_stamp":"2024-06-03T09:06:57.886322+0000","objects_scrubbed":2,"log_size":78,"log_dups_size":0,"ondisk_log_size":78,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-06-04T19:15:53.804903+0000","scrub_duration":0.098050433000000006,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":459280,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":96,"num_read_kb":82,"num_write":113,"num_write_kb":1372,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2,"num_bytes_recovered":397840,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,5],"acting":[1,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"17.10","version":"0'0","reported_seq":959,"reported_epoch":1014,"state":"active+clean","last_fresh":"2024-06-03T09:07:02.104119+0000","last_change":"2024-06-03T09:07:02.105812+0000","last_active":"2024-06-03T09:07:02.104119+0000","last_peered":"2024-06-03T09:07:02.104119+0000","last_clean":"2024-06-03T09:07:02.104119+0000","last_became_active":"2024-06-03T08:38:15.337723+0000","last_became_peered":"2024-06-03T08:38:15.337723+0000","last_unstale":"2024-06-03T09:07:02.104119+0000","last_undegraded":"2024-06-03T09:07:02.104119+0000","last_fullsized":"2024-06-03T09:07:02.104119+0000","mapping_epoch":28,"log_start":"0'0","ondisk_log_start":"0'0","created":28,"last_epoch_clean":42,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-06-03T09:07:02.103345+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-06-03T09:07:02.103345+0000","last_clean_scrub_stamp":"2024-06-03T09:07:02.103345+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-06-04T20:06:45.041047+0000","scrub_duration":0.036150559999999998,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,3],"acting":[4,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"17.1a","version":"0'0","reported_seq":983,"reported_epoch":1014,"state":"active+clean","last_fresh":"2024-06-03T09:07:07.163797+0000","last_change":"2024-06-03T09:07:07.165427+0000","last_active":"2024-06-03T09:07:07.163797+0000","last_peered":"2024-06-03T09:07:07.163797+0000","last_clean":"2024-06-03T09:07:07.163797+0000","last_became_active":"2024-06-03T09:06:46.386469+0000","last_became_peered":"2024-06-03T09:06:46.386469+0000","last_unstale":"2024-06-03T09:07:07.163797+0000","last_undegraded":"2024-06-03T09:07:07.163797+0000","last_fullsized":"2024-06-03T09:07:07.163797+0000","mapping_epoch":1013,"log_start":"0'0","ondisk_log_start":"0'0","created":28,"last_epoch_clean":1014,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-06-03T09:07:07.163037+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-06-03T09:07:07.163037+0000","last_clean_scrub_stamp":"2024-06-03T09:07:07.163037+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-06-04T18:10:14.782734+0000","scrub_duration":0.031532482000000001,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,6],"acting":[4,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"17.1b","version":"0'0","reported_seq":940,"reported_epoch":1014,"state":"active+clean","last_fresh":"2024-06-03T09:06:46.128510+0000","last_change":"2024-06-03T08:38:03.115748+0000","last_active":"2024-06-03T09:06:46.128510+0000","last_peered":"2024-06-03T09:06:46.128510+0000","last_clean":"2024-06-03T09:06:46.128510+0000","last_became_active":"2024-06-03T08:38:03.110533+0000","last_became_peered":"2024-06-03T08:38:03.110533+0000","last_unstale":"2024-06-03T09:06:46.128510+0000","last_undegraded":"2024-06-03T09:06:46.128510+0000","last_fullsized":"2024-06-03T09:06:46.128510+0000","mapping_epoch":28,"log_start":"0'0","ondisk_log_start":"0'0","created":28,"last_epoch_clean":32,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-06-03T08:37:55.614497+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-06-03T08:37:55.614497+0000","last_clean_scrub_stamp":"2024-06-03T08:37:55.614497+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for deep scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,7],"acting":[3,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"17.18","version":"0'0","reported_seq":960,"reported_epoch":1014,"state":"active+clean","last_fresh":"2024-06-03T09:07:04.460345+0000","last_change":"2024-06-03T09:07:04.460345+0000","last_active":"2024-06-03T09:07:04.460345+0000","last_peered":"2024-06-03T09:07:04.460345+0000","last_clean":"2024-06-03T09:07:04.460345+0000","last_became_active":"2024-06-03T08:38:04.627856+0000","last_became_peered":"2024-06-03T08:38:04.627856+0000","last_unstale":"2024-06-03T09:07:04.460345+0000","last_undegraded":"2024-06-03T09:07:04.460345+0000","last_fullsized":"2024-06-03T09:07:04.460345+0000","mapping_epoch":28,"log_start":"0'0","ondisk_log_start":"0'0","created":28,"last_epoch_clean":33,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-06-03T09:07:04.458032+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-06-03T09:07:04.458032+0000","last_clean_scrub_stamp":"2024-06-03T09:07:04.458032+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-06-04T17:11:27.305960+0000","scrub_duration":0.03213684,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,4],"acting":[7,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"17.19","version":"0'0","reported_seq":940,"reported_epoch":1014,"state":"active+clean","last_fresh":"2024-06-03T09:06:59.338742+0000","last_change":"2024-06-03T08:38:01.839491+0000","last_active":"2024-06-03T09:06:59.338742+0000","last_peered":"2024-06-03T09:06:59.338742+0000","last_clean":"2024-06-03T09:06:59.338742+0000","last_became_active":"2024-06-03T08:38:01.829539+0000","last_became_peered":"2024-06-03T08:38:01.829539+0000","last_unstale":"2024-06-03T09:06:59.338742+0000","last_undegraded":"2024-06-03T09:06:59.338742+0000","last_fullsized":"2024-06-03T09:06:59.338742+0000","mapping_epoch":28,"log_start":"0'0","ondisk_log_start":"0'0","created":28,"last_epoch_clean":30,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-06-03T08:37:55.614497+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-06-03T08:37:55.614497+0000","last_clean_scrub_stamp":"2024-06-03T08:37:55.614497+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for deep scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,2],"acting":[3,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"17.1e","version":"0'0","reported_seq":856,"reported_epoch":1014,"state":"active+clean","last_fresh":"2024-06-03T09:07:10.882123+0000","last_change":"2024-06-03T09:07:10.883772+0000","last_active":"2024-06-03T09:07:10.882123+0000","last_peered":"2024-06-03T09:07:10.882123+0000","last_clean":"2024-06-03T09:07:10.882123+0000","last_became_active":"2024-06-03T09:06:46.288593+0000","last_became_peered":"2024-06-03T09:06:46.288593+0000","last_unstale":"2024-06-03T09:07:10.882123+0000","last_undegraded":"2024-06-03T09:07:10.882123+0000","last_fullsized":"2024-06-03T09:07:10.882123+0000","mapping_epoch":1013,"log_start":"0'0","ondisk_log_start":"0'0","created":28,"last_epoch_clean":1014,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-06-03T09:07:10.881377+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-06-03T09:07:10.881377+0000","last_clean_scrub_stamp":"2024-06-03T09:07:10.881377+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-06-04T15:00:04.646181+0000","scrub_duration":0.035250741000000002,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,4],"acting":[6,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"17.1f","version":"0'0","reported_seq":957,"reported_epoch":1014,"state":"active+clean","last_fresh":"2024-06-03T09:07:16.249909+0000","last_change":"2024-06-03T09:07:16.251476+0000","last_active":"2024-06-03T09:07:16.249909+0000","last_peered":"2024-06-03T09:07:16.249909+0000","last_clean":"2024-06-03T09:07:16.249909+0000","last_became_active":"2024-06-03T08:38:04.367278+0000","last_became_peered":"2024-06-03T08:38:04.367278+0000","last_unstale":"2024-06-03T09:07:16.249909+0000","last_undegraded":"2024-06-03T09:07:16.249909+0000","last_fullsized":"2024-06-03T09:07:16.249909+0000","mapping_epoch":28,"log_start":"0'0","ondisk_log_start":"0'0","created":28,"last_epoch_clean":32,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-06-03T09:07:16.249190+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-06-03T09:07:16.249190+0000","last_clean_scrub_stamp":"2024-06-03T09:07:16.249190+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-06-04T19:04:33.697530+0000","scrub_duration":0.069893753000000003,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,2],"acting":[0,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"17.1c","version":"0'0","reported_seq":957,"reported_epoch":1014,"state":"active+clean","last_fresh":"2024-06-03T09:06:46.419841+0000","last_change":"2024-06-03T09:06:46.410116+0000","last_active":"2024-06-03T09:06:46.419841+0000","last_peered":"2024-06-03T09:06:46.419841+0000","last_clean":"2024-06-03T09:06:46.419841+0000","last_became_active":"2024-06-03T09:06:46.403724+0000","last_became_peered":"2024-06-03T09:06:46.403724+0000","last_unstale":"2024-06-03T09:06:46.419841+0000","last_undegraded":"2024-06-03T09:06:46.419841+0000","last_fullsized":"2024-06-03T09:06:46.419841+0000","mapping_epoch":1013,"log_start":"0'0","ondisk_log_start":"0'0","created":28,"last_epoch_clean":1014,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-06-03T08:37:55.614497+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-06-03T08:37:55.614497+0000","last_clean_scrub_stamp":"2024-06-03T08:37:55.614497+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for deep scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,6],"acting":[5,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"17.1d","version":"0'0","reported_seq":939,"reported_epoch":1014,"state":"active+clean","last_fresh":"2024-06-03T09:07:10.965559+0000","last_change":"2024-06-03T09:07:10.967165+0000","last_active":"2024-06-03T09:07:10.965559+0000","last_peered":"2024-06-03T09:07:10.965559+0000","last_clean":"2024-06-03T09:07:10.965559+0000","last_became_active":"2024-06-03T08:38:06.592317+0000","last_became_peered":"2024-06-03T08:38:06.592317+0000","last_unstale":"2024-06-03T09:07:10.965559+0000","last_undegraded":"2024-06-03T09:07:10.965559+0000","last_fullsized":"2024-06-03T09:07:10.965559+0000","mapping_epoch":28,"log_start":"0'0","ondisk_log_start":"0'0","created":28,"last_epoch_clean":33,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-06-03T09:07:10.964826+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-06-03T09:07:10.964826+0000","last_clean_scrub_stamp":"2024-06-03T09:07:10.964826+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-06-04T15:19:56.203494+0000","scrub_duration":0.062732020999999999,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,7],"acting":[1,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"17.2","version":"0'0","reported_seq":942,"reported_epoch":1014,"state":"active+clean","last_fresh":"2024-06-03T09:07:04.809640+0000","last_change":"2024-06-03T09:07:04.809640+0000","last_active":"2024-06-03T09:07:04.809640+0000","last_peered":"2024-06-03T09:07:04.809640+0000","last_clean":"2024-06-03T09:07:04.809640+0000","last_became_active":"2024-06-03T08:38:14.973825+0000","last_became_peered":"2024-06-03T08:38:14.973825+0000","last_unstale":"2024-06-03T09:07:04.809640+0000","last_undegraded":"2024-06-03T09:07:04.809640+0000","last_fullsized":"2024-06-03T09:07:04.809640+0000","mapping_epoch":28,"log_start":"0'0","ondisk_log_start":"0'0","created":28,"last_epoch_clean":41,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-06-03T09:07:04.807216+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-06-03T09:07:04.807216+0000","last_clean_scrub_stamp":"2024-06-03T09:07:04.807216+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-06-04T14:52:55.434564+0000","scrub_duration":0.030095302000000001,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,4],"acting":[1,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"17.3","version":"0'0","reported_seq":959,"reported_epoch":1014,"state":"active+clean","last_fresh":"2024-06-03T09:07:09.259431+0000","last_change":"2024-06-03T09:07:09.259431+0000","last_active":"2024-06-03T09:07:09.259431+0000","last_peered":"2024-06-03T09:07:09.259431+0000","last_clean":"2024-06-03T09:07:09.259431+0000","last_became_active":"2024-06-03T08:38:14.510665+0000","last_became_peered":"2024-06-03T08:38:14.510665+0000","last_unstale":"2024-06-03T09:07:09.259431+0000","last_undegraded":"2024-06-03T09:07:09.259431+0000","last_fullsized":"2024-06-03T09:07:09.259431+0000","mapping_epoch":28,"log_start":"0'0","ondisk_log_start":"0'0","created":28,"last_epoch_clean":42,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-06-03T09:07:09.253053+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-06-03T09:07:09.253053+0000","last_clean_scrub_stamp":"2024-06-03T09:07:09.253053+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-06-04T14:30:59.802025+0000","scrub_duration":0.068038479999999998,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,7],"acting":[0,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"17.0","version":"0'0","reported_seq":959,"reported_epoch":1014,"state":"active+clean","last_fresh":"2024-06-03T09:07:06.233712+0000","last_change":"2024-06-03T09:07:06.235311+0000","last_active":"2024-06-03T09:07:06.233712+0000","last_peered":"2024-06-03T09:07:06.233712+0000","last_clean":"2024-06-03T09:07:06.233712+0000","last_became_active":"2024-06-03T08:38:04.231954+0000","last_became_peered":"2024-06-03T08:38:04.231954+0000","last_unstale":"2024-06-03T09:07:06.233712+0000","last_undegraded":"2024-06-03T09:07:06.233712+0000","last_fullsized":"2024-06-03T09:07:06.233712+0000","mapping_epoch":28,"log_start":"0'0","ondisk_log_start":"0'0","created":28,"last_epoch_clean":32,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-06-03T09:07:06.232980+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-06-03T09:07:06.232980+0000","last_clean_scrub_stamp":"2024-06-03T09:07:06.232980+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-06-04T14:04:39.143315+0000","scrub_duration":0.033029297999999999,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,2],"acting":[0,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"17.1","version":"1014'8","reported_seq":985,"reported_epoch":1014,"state":"active+clean","last_fresh":"2024-06-03T09:07:05.543762+0000","last_change":"2024-06-03T09:07:05.543762+0000","last_active":"2024-06-03T09:07:05.543762+0000","last_peered":"2024-06-03T09:07:05.543762+0000","last_clean":"2024-06-03T09:07:05.543762+0000","last_became_active":"2024-06-03T09:06:46.383290+0000","last_became_peered":"2024-06-03T09:06:46.383290+0000","last_unstale":"2024-06-03T09:07:05.543762+0000","last_undegraded":"2024-06-03T09:07:05.543762+0000","last_fullsized":"2024-06-03T09:07:05.543762+0000","mapping_epoch":1013,"log_start":"0'0","ondisk_log_start":"0'0","created":28,"last_epoch_clean":1014,"parent":"0.0","parent_split_bits":0,"last_scrub":"1014'8","last_scrub_stamp":"2024-06-03T09:07:05.540994+0000","last_deep_scrub":"1014'8","last_deep_scrub_stamp":"2024-06-03T09:07:05.540994+0000","last_clean_scrub_stamp":"2024-06-03T09:07:05.540994+0000","objects_scrubbed":1,"log_size":8,"log_dups_size":0,"ondisk_log_size":8,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-06-04T18:34:55.019060+0000","scrub_duration":0.091133806999999997,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":5,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":6,"num_read_kb":6,"num_write":7,"num_write_kb":7,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,6],"acting":[5,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]}],"pool_stats":[{"poolid":2,"num_pg":8,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":16,"acting":16,"num_store_stats":7},{"poolid":1,"num_pg":1,"stat_sum":{"num_bytes":459280,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":96,"num_read_kb":82,"num_write":113,"num_write_kb":1372,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2,"num_bytes_recovered":397840,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":802816,"data_stored":795680,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":78,"ondisk_log_size":78,"up":2,"acting":2,"num_store_stats":2},{"poolid":17,"num_pg":32,"stat_sum":{"num_bytes":5,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":6,"num_read_kb":6,"num_write":7,"num_write_kb":7,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":8192,"data_stored":10,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":8,"ondisk_log_size":8,"up":64,"acting":64,"num_store_stats":8}],"osd_stats":[{"osd":6,"up_from":1013,"seq":4350801871169,"num_pgs":14,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":54916,"kb_used_data":5756,"kb_used_omap":1,"kb_used_meta":49150,"kb_avail":94316924,"statfs":{"total":96636764160,"available":96580530176,"internally_reserved":0,"allocated":5894144,"data_stored":2763078,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1593,"internal_metadata":50330055},"hb_peers":[0,1,2,3,4,5,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":3,"apply_latency_ms":3,"commit_latency_ns":3000000,"apply_latency_ns":3000000},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Mon Jun 3 09:06:42 2024","interfaces":[{"interface":"back","average":{"1min":5696.3109999999997,"5min":1469.3900000000001,"15min":503.11700000000002},"min":{"1min":5.242,"5min":4.4829999999999997,"15min":4.1959999999999997},"max":{"1min":20463.029999999999,"5min":22766.882000000001,"15min":22766.882000000001},"last":11.582000000000001},{"interface":"front","average":{"1min":5687.6390000000001,"5min":1468.1669999999999,"15min":504.13200000000001},"min":{"1min":4.835,"5min":4.8129999999999997,"15min":4.4530000000000003},"max":{"1min":20461.830000000002,"5min":22766.289000000001,"15min":22766.289000000001},"last":8.0109999999999992}]},{"osd":1,"last update":"Mon Jun 3 09:06:42 2024","interfaces":[{"interface":"back","average":{"1min":36.496000000000002,"5min":80.162999999999997,"15min":134.59399999999999},"min":{"1min":5.9189999999999996,"5min":5.8799999999999999,"15min":5.7119999999999997},"max":{"1min":544.85799999999995,"5min":4138.1679999999997,"15min":9955.1299999999992},"last":9.1869999999999994},{"interface":"front","average":{"1min":38.866999999999997,"5min":82.631,"15min":135.11500000000001},"min":{"1min":6.5350000000000001,"5min":6.5350000000000001,"15min":6.1390000000000002},"max":{"1min":546.73099999999999,"5min":4137.6530000000002,"15min":9827.5509999999995},"last":15.436999999999999}]},{"osd":2,"last update":"Mon Jun 3 09:06:42 2024","interfaces":[{"interface":"back","average":{"1min":41.093000000000004,"5min":27.917000000000002,"15min":22.329000000000001},"min":{"1min":6.0670000000000002,"5min":6.0670000000000002,"15min":6.0670000000000002},"max":{"1min":548.12900000000002,"5min":548.12900000000002,"15min":548.12900000000002},"last":7.6219999999999999},{"interface":"front","average":{"1min":30.798999999999999,"5min":24.276,"15min":22.488},"min":{"1min":6.4370000000000003,"5min":6.4370000000000003,"15min":6.1870000000000003},"max":{"1min":369.14499999999998,"5min":369.14499999999998,"15min":434.77300000000002},"last":7.141}]},{"osd":3,"last update":"Mon Jun 3 09:06:42 2024","interfaces":[{"interface":"back","average":{"1min":41.411999999999999,"5min":27.434999999999999,"15min":30.657},"min":{"1min":10.061,"5min":8.3179999999999996,"15min":6.1719999999999997},"max":{"1min":549.024,"5min":549.024,"15min":1235.5799999999999},"last":13.637},{"interface":"front","average":{"1min":37.850999999999999,"5min":22.001000000000001,"15min":28.015999999999998},"min":{"1min":6.0679999999999996,"5min":6.0679999999999996,"15min":6.0679999999999996},"max":{"1min":559.25,"5min":559.25,"15min":1157.6780000000001},"last":10.356}]},{"osd":4,"last update":"Mon Jun 3 09:06:42 2024","interfaces":[{"interface":"back","average":{"1min":165.62899999999999,"5min":1281.1010000000001,"15min":440.59199999999998},"min":{"1min":10.965999999999999,"5min":6.093,"15min":6.093},"max":{"1min":384.46899999999999,"5min":28093.096000000001,"15min":28093.096000000001},"last":14.646000000000001},{"interface":"front","average":{"1min":186.97900000000001,"5min":1308.886,"15min":454.97699999999998},"min":{"1min":13.581,"5min":7.7480000000000002,"15min":7.7480000000000002},"max":{"1min":445.096,"5min":28323.837,"15min":28323.837},"last":17.154}]},{"osd":5,"last update":"Mon Jun 3 09:06:42 2024","interfaces":[{"interface":"back","average":{"1min":81.581000000000003,"5min":34.863,"15min":28.350999999999999},"min":{"1min":10.345000000000001,"5min":8.625,"15min":7.516},"max":{"1min":455.84199999999998,"5min":455.84199999999998,"15min":455.84199999999998},"last":18.800000000000001},{"interface":"front","average":{"1min":83.391000000000005,"5min":39.277999999999999,"15min":27.885000000000002},"min":{"1min":8.0280000000000005,"5min":6.266,"15min":6.266},"max":{"1min":550.02300000000002,"5min":550.02300000000002,"15min":550.02300000000002},"last":16.338000000000001}]},{"osd":7,"last update":"Mon Jun 3 09:06:42 2024","interfaces":[{"interface":"back","average":{"1min":44.052999999999997,"5min":33.697000000000003,"15min":29.731999999999999},"min":{"1min":9.1500000000000004,"5min":9.1500000000000004,"15min":9.1500000000000004},"max":{"1min":552.09500000000003,"5min":552.09500000000003,"15min":552.09500000000003},"last":18.065999999999999},{"interface":"front","average":{"1min":47.076999999999998,"5min":34.226999999999997,"15min":29.827000000000002},"min":{"1min":9.7780000000000005,"5min":8.6709999999999994,"15min":8.5820000000000007},"max":{"1min":531.08600000000001,"5min":531.08600000000001,"15min":531.08600000000001},"last":12.265000000000001}]}]},{"osd":7,"up_from":20,"seq":85899346297,"num_pgs":9,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":59008,"kb_used_data":5752,"kb_used_omap":1,"kb_used_meta":53246,"kb_avail":94312832,"statfs":{"total":96636764160,"available":96576339968,"internally_reserved":0,"allocated":5890048,"data_stored":2763073,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1589,"internal_metadata":54524363},"hb_peers":[0,1,2,3,4,5,6],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Mon Jun 3 09:07:10 2024","interfaces":[{"interface":"back","average":{"1min":8.9380000000000006,"5min":893.83100000000002,"15min":307.63400000000001},"min":{"1min":4.032,"5min":4.032,"15min":4.032},"max":{"1min":17.491,"5min":21729.001,"15min":21729.001},"last":8.1859999999999999},{"interface":"front","average":{"1min":7.8970000000000002,"5min":905.91999999999996,"15min":312.15100000000001},"min":{"1min":4.5979999999999999,"5min":4.5979999999999999,"15min":4.5979999999999999},"max":{"1min":16.053999999999998,"5min":21899.732,"15min":21899.732},"last":7.1609999999999996}]},{"osd":1,"last update":"Mon Jun 3 09:07:10 2024","interfaces":[{"interface":"back","average":{"1min":8.5670000000000002,"5min":68.463999999999999,"15min":218.52199999999999},"min":{"1min":5.3419999999999996,"5min":5.3419999999999996,"15min":5.3419999999999996},"max":{"1min":11.128,"5min":2795.1329999999998,"15min":18373.257000000001},"last":10},{"interface":"front","average":{"1min":12.664999999999999,"5min":73.626000000000005,"15min":224.589},"min":{"1min":9.6460000000000008,"5min":8.8399999999999999,"15min":8.5129999999999999},"max":{"1min":15.045,"5min":2803.3670000000002,"15min":18374.601999999999},"last":14.865}]},{"osd":2,"last update":"Mon Jun 3 09:07:10 2024","interfaces":[{"interface":"back","average":{"1min":13.367000000000001,"5min":25.641999999999999,"15min":22.251999999999999},"min":{"1min":10.000999999999999,"5min":8.5030000000000001,"15min":8.5030000000000001},"max":{"1min":18.968,"5min":333.99200000000002,"15min":333.99200000000002},"last":14.234},{"interface":"front","average":{"1min":9.5410000000000004,"5min":17.655000000000001,"15min":15.645},"min":{"1min":6.8840000000000003,"5min":5.8369999999999997,"15min":5.5069999999999997},"max":{"1min":13.875,"5min":226.18100000000001,"15min":226.18100000000001},"last":10.923}]},{"osd":3,"last update":"Mon Jun 3 09:07:10 2024","interfaces":[{"interface":"back","average":{"1min":12.894,"5min":25.039000000000001,"15min":30.681999999999999},"min":{"1min":7.617,"5min":6.8959999999999999,"15min":6.3760000000000003},"max":{"1min":16.376000000000001,"5min":316.64400000000001,"15min":2039.9059999999999},"last":17.507000000000001},{"interface":"front","average":{"1min":9.3800000000000008,"5min":17.294,"15min":24.911999999999999},"min":{"1min":6.1929999999999996,"5min":5.2089999999999996,"15min":5.2089999999999996},"max":{"1min":13.622999999999999,"5min":233.542,"15min":2037.5550000000001},"last":10.452}]},{"osd":4,"last update":"Mon Jun 3 09:06:29 2024","interfaces":[{"interface":"back","average":{"1min":17.675999999999998,"5min":1026.325,"15min":472.47199999999998},"min":{"1min":7.306,"5min":5.9089999999999998,"15min":5.9089999999999998},"max":{"1min":140.03299999999999,"5min":28969.369999999999,"15min":30073.917000000001},"last":11.754},{"interface":"front","average":{"1min":17.890999999999998,"5min":1044.5139999999999,"15min":480.226},"min":{"1min":5.9279999999999999,"5min":5.9279999999999999,"15min":5.9279999999999999},"max":{"1min":141.179,"5min":29244.138999999999,"15min":30348.021000000001},"last":12.298}]},{"osd":5,"last update":"Mon Jun 3 09:07:04 2024","interfaces":[{"interface":"back","average":{"1min":18.949999999999999,"5min":26.824000000000002,"15min":23.492000000000001},"min":{"1min":10.933999999999999,"5min":9.2460000000000004,"15min":7.0010000000000003},"max":{"1min":95.691999999999993,"5min":317.572,"15min":317.572},"last":16.297999999999998},{"interface":"front","average":{"1min":17.582999999999998,"5min":24.216999999999999,"15min":20.686},"min":{"1min":8.5760000000000005,"5min":7.0999999999999996,"15min":7.008},"max":{"1min":92.829999999999998,"5min":306.60599999999999,"15min":306.60599999999999},"last":12.920999999999999}]},{"osd":6,"last update":"Mon Jun 3 09:04:31 2024","interfaces":[{"interface":"back","average":{"1min":44.061,"5min":26.821000000000002,"15min":91.176000000000002},"min":{"1min":9.0800000000000001,"5min":7.9390000000000001,"15min":7.9390000000000001},"max":{"1min":268.029,"5min":308.916,"15min":8734.4639999999999},"last":15.590999999999999},{"interface":"front","average":{"1min":46.643000000000001,"5min":23.879999999999999,"15min":88.765000000000001},"min":{"1min":8.8119999999999994,"5min":6.9240000000000004,"15min":6.742},"max":{"1min":305.64600000000002,"5min":305.64600000000002,"15min":8735.5319999999992},"last":18.106000000000002}]}]},{"osd":5,"up_from":20,"seq":85899346272,"num_pgs":9,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":59404,"kb_used_data":6148,"kb_used_omap":1,"kb_used_meta":53246,"kb_avail":94312436,"statfs":{"total":96636764160,"available":96575934464,"internally_reserved":0,"allocated":6295552,"data_stored":3160918,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1594,"internal_metadata":54524358},"hb_peers":[0,1,2,3,4,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Mon Jun 3 09:07:06 2024","interfaces":[{"interface":"back","average":{"1min":7.5899999999999999,"5min":1702.617,"15min":579.77099999999996},"min":{"1min":4.5380000000000003,"5min":4.5380000000000003,"15min":4.4020000000000001},"max":{"1min":16.231999999999999,"5min":26433.764999999999,"15min":26433.764999999999},"last":12.119},{"interface":"front","average":{"1min":10.436,"5min":1725.6030000000001,"15min":585.86500000000001},"min":{"1min":6.9699999999999998,"5min":5.1319999999999997,"15min":4.8170000000000002},"max":{"1min":17.734000000000002,"5min":26704.698,"15min":26704.698},"last":13.348000000000001}]},{"osd":1,"last update":"Mon Jun 3 09:07:01 2024","interfaces":[{"interface":"back","average":{"1min":11.929,"5min":66.978999999999999,"15min":216.91999999999999},"min":{"1min":5.9790000000000001,"5min":5.9790000000000001,"15min":5.4930000000000003},"max":{"1min":22.202999999999999,"5min":3923.4090000000001,"15min":18811.696},"last":14.686999999999999},{"interface":"front","average":{"1min":11.496,"5min":64.528000000000006,"15min":220.06100000000001},"min":{"1min":6.298,"5min":5.3209999999999997,"15min":5.0209999999999999},"max":{"1min":27.747,"5min":4019.6129999999998,"15min":18968.365000000002},"last":12.608000000000001}]},{"osd":2,"last update":"Mon Jun 3 09:07:01 2024","interfaces":[{"interface":"back","average":{"1min":13.903,"5min":24.381,"15min":20.638999999999999},"min":{"1min":9.3559999999999999,"5min":5.6369999999999996,"15min":5.0510000000000002},"max":{"1min":27.013999999999999,"5min":332.48200000000003,"15min":332.48200000000003},"last":16.734000000000002},{"interface":"front","average":{"1min":9.2989999999999995,"5min":22.521999999999998,"15min":23.033999999999999},"min":{"1min":6.2439999999999998,"5min":6.2439999999999998,"15min":6.2439999999999998},"max":{"1min":18.395,"5min":353.03199999999998,"15min":353.03199999999998},"last":7.0049999999999999}]},{"osd":3,"last update":"Mon Jun 3 09:07:01 2024","interfaces":[{"interface":"back","average":{"1min":14.686999999999999,"5min":24.370999999999999,"15min":31.327000000000002},"min":{"1min":8.1140000000000008,"5min":7.4720000000000004,"15min":7.4210000000000003},"max":{"1min":30.178000000000001,"5min":353.57499999999999,"15min":2855.8519999999999},"last":15.798},{"interface":"front","average":{"1min":11.590999999999999,"5min":16.498999999999999,"15min":25.437999999999999},"min":{"1min":6.3769999999999998,"5min":5.9660000000000002,"15min":5.1909999999999998},"max":{"1min":22.972000000000001,"5min":203.804,"15min":2869.7860000000001},"last":8.6950000000000003}]},{"osd":4,"last update":"Mon Jun 3 09:06:29 2024","interfaces":[{"interface":"back","average":{"1min":11.067,"5min":780.21000000000004,"15min":377.28399999999999},"min":{"1min":5.4779999999999998,"5min":5.4779999999999998,"15min":5.4779999999999998},"max":{"1min":14.654999999999999,"5min":23315.438999999998,"15min":28619.825000000001},"last":7.835},{"interface":"front","average":{"1min":14.349,"5min":782.97299999999996,"15min":377.93400000000003},"min":{"1min":7.492,"5min":6.5030000000000001,"15min":6.5030000000000001},"max":{"1min":17.707000000000001,"5min":23317.502,"15min":28621.294000000002},"last":18.669}]},{"osd":6,"last update":"Mon Jun 3 09:03:57 2024","interfaces":[{"interface":"back","average":{"1min":23.536000000000001,"5min":36.457000000000001,"15min":83.799999999999997},"min":{"1min":9.4800000000000004,"5min":9.4800000000000004,"15min":8.0630000000000006},"max":{"1min":89.442999999999998,"5min":508.77999999999997,"15min":9193.5169999999998},"last":9.8610000000000007},{"interface":"front","average":{"1min":18.891999999999999,"5min":22.100999999999999,"15min":75.489999999999995},"min":{"1min":5.923,"5min":5.923,"15min":5.923},"max":{"1min":78.424999999999997,"5min":464.62,"15min":9191.4429999999993},"last":17.712}]},{"osd":7,"last update":"Mon Jun 3 09:07:01 2024","interfaces":[{"interface":"back","average":{"1min":15.648999999999999,"5min":21.43,"15min":20.760999999999999},"min":{"1min":8.6319999999999997,"5min":7.4020000000000001,"15min":6.7629999999999999},"max":{"1min":28.895,"5min":211.22999999999999,"15min":211.22999999999999},"last":11.529},{"interface":"front","average":{"1min":13.545999999999999,"5min":20.359000000000002,"15min":23.413},"min":{"1min":6.8239999999999998,"5min":6.8239999999999998,"15min":6.8239999999999998},"max":{"1min":36.515000000000001,"5min":223.191,"15min":286.69499999999999},"last":10.734999999999999}]}]},{"osd":4,"up_from":20,"seq":85899346279,"num_pgs":12,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":63104,"kb_used_data":5752,"kb_used_omap":1,"kb_used_meta":57342,"kb_avail":94308736,"statfs":{"total":96636764160,"available":96572145664,"internally_reserved":0,"allocated":5890048,"data_stored":2763073,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1594,"internal_metadata":58718662},"hb_peers":[0,1,2,3,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":3,"apply_latency_ms":3,"commit_latency_ns":3000000,"apply_latency_ns":3000000},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Mon Jun 3 09:06:35 2024","interfaces":[{"interface":"back","average":{"1min":10.417,"5min":1093.547,"15min":381.64499999999998},"min":{"1min":4.6900000000000004,"5min":4.5279999999999996,"15min":4.3220000000000001},"max":{"1min":52.241,"5min":21726.743999999999,"15min":21726.743999999999},"last":6.4180000000000001},{"interface":"front","average":{"1min":13.175000000000001,"5min":1102.4200000000001,"15min":387.161},"min":{"1min":6.1950000000000003,"5min":5.8380000000000001,"15min":5.4909999999999997},"max":{"1min":53.148000000000003,"5min":21727.993999999999,"15min":21727.993999999999},"last":8.4659999999999993}]},{"osd":1,"last update":"Mon Jun 3 09:06:35 2024","interfaces":[{"interface":"back","average":{"1min":9.9909999999999997,"5min":53.250999999999998,"15min":340.02999999999997},"min":{"1min":4.5670000000000002,"5min":4.1970000000000001,"15min":4.1970000000000001},"max":{"1min":47.073,"5min":3346.4389999999999,"15min":22256.098000000002},"last":6.8520000000000003},{"interface":"front","average":{"1min":12.189,"5min":53.375999999999998,"15min":339.68799999999999},"min":{"1min":6.1539999999999999,"5min":5.0590000000000002,"15min":4.2640000000000002},"max":{"1min":49.558,"5min":3347.181,"15min":22247.317999999999},"last":6.1669999999999998}]},{"osd":2,"last update":"Mon Jun 3 09:06:35 2024","interfaces":[{"interface":"back","average":{"1min":15.178000000000001,"5min":17.202000000000002,"15min":20.417999999999999},"min":{"1min":6.3010000000000002,"5min":6.3010000000000002,"15min":6.3010000000000002},"max":{"1min":85.340000000000003,"5min":337.60399999999998,"15min":393.58600000000001},"last":11.648999999999999},{"interface":"front","average":{"1min":15.423,"5min":14.432,"15min":22.754000000000001},"min":{"1min":6.4889999999999999,"5min":6.0629999999999997,"15min":6.0629999999999997},"max":{"1min":82.120000000000005,"5min":127.913,"15min":394.471},"last":13.965}]},{"osd":3,"last update":"Mon Jun 3 09:06:35 2024","interfaces":[{"interface":"back","average":{"1min":17.651,"5min":15.5,"15min":18.079000000000001},"min":{"1min":10.359999999999999,"5min":5.8719999999999999,"15min":5.4000000000000004},"max":{"1min":84.555999999999997,"5min":127.43600000000001,"15min":322.63},"last":13.286},{"interface":"front","average":{"1min":16.413,"5min":17.356999999999999,"15min":22.920000000000002},"min":{"1min":9.7349999999999994,"5min":7.484,"15min":7.484},"max":{"1min":71.394000000000005,"5min":204.517,"15min":392.67500000000001},"last":12.401}]},{"osd":5,"last update":"Mon Jun 3 09:06:35 2024","interfaces":[{"interface":"back","average":{"1min":15.672000000000001,"5min":16.742999999999999,"15min":20.795000000000002},"min":{"1min":8.093,"5min":6.7489999999999997,"15min":6.7489999999999997},"max":{"1min":51.326000000000001,"5min":268.10199999999998,"15min":344.23399999999998},"last":14.686999999999999},{"interface":"front","average":{"1min":17.297999999999998,"5min":19.356999999999999,"15min":25.713999999999999},"min":{"1min":12.875,"5min":8.5180000000000007,"15min":8.5180000000000007},"max":{"1min":40.829000000000001,"5min":264.75,"15min":345.18700000000001},"last":15.436999999999999}]},{"osd":6,"last update":"Mon Jun 3 09:04:32 2024","interfaces":[{"interface":"back","average":{"1min":15.548,"5min":32.957999999999998,"15min":152.738},"min":{"1min":9.141,"5min":9.141,"15min":8.2319999999999993},"max":{"1min":52.57,"5min":395.238,"15min":12496.335999999999},"last":8.7910000000000004},{"interface":"front","average":{"1min":12.680999999999999,"5min":25.428000000000001,"15min":149.30600000000001},"min":{"1min":6.4379999999999997,"5min":5.6319999999999997,"15min":5.6319999999999997},"max":{"1min":46.283000000000001,"5min":377.74299999999999,"15min":12616.297},"last":16.247}]},{"osd":7,"last update":"Mon Jun 3 09:06:35 2024","interfaces":[{"interface":"back","average":{"1min":9.6969999999999992,"5min":14.196,"15min":18.809999999999999},"min":{"1min":6.7190000000000003,"5min":6.2110000000000003,"15min":5.8010000000000002},"max":{"1min":15.960000000000001,"5min":207.25299999999999,"15min":358.94900000000001},"last":9.3079999999999998},{"interface":"front","average":{"1min":17.696000000000002,"5min":19.024999999999999,"15min":23.259},"min":{"1min":11.794,"5min":7.7789999999999999,"15min":7.7789999999999999},"max":{"1min":45.639000000000003,"5min":209.32900000000001,"15min":380.06900000000002},"last":17.077000000000002}]}]},{"osd":2,"up_from":20,"seq":85899346299,"num_pgs":11,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":59008,"kb_used_data":5752,"kb_used_omap":1,"kb_used_meta":53246,"kb_avail":94312832,"statfs":{"total":96636764160,"available":96576339968,"internally_reserved":0,"allocated":5890048,"data_stored":2763073,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1594,"internal_metadata":54524358},"hb_peers":[0,1,3,4,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":3,"apply_latency_ms":3,"commit_latency_ns":3000000,"apply_latency_ns":3000000},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Mon Jun 3 09:06:23 2024","interfaces":[{"interface":"back","average":{"1min":8.9629999999999992,"5min":946.25400000000002,"15min":326.36099999999999},"min":{"1min":5.3479999999999999,"5min":4.7050000000000001,"15min":4.1420000000000003},"max":{"1min":26.015000000000001,"5min":23465.18,"15min":23465.18},"last":7.109},{"interface":"front","average":{"1min":7.6130000000000004,"5min":944.39800000000002,"15min":326.178},"min":{"1min":4.6790000000000003,"5min":4.6790000000000003,"15min":4.6790000000000003},"max":{"1min":20.802,"5min":23465.966,"15min":23465.966},"last":11.827}]},{"osd":1,"last update":"Mon Jun 3 09:06:23 2024","interfaces":[{"interface":"back","average":{"1min":13.007,"5min":76.507000000000005,"15min":294.72899999999998},"min":{"1min":6.4089999999999998,"5min":5.4199999999999999,"15min":5.4199999999999999},"max":{"1min":85.430000000000007,"5min":4220.5290000000005,"15min":18954.987000000001},"last":8.5690000000000008},{"interface":"front","average":{"1min":15.894,"5min":82.119,"15min":302.92399999999998},"min":{"1min":7.367,"5min":7.367,"15min":7.367},"max":{"1min":98.498000000000005,"5min":4325.9489999999996,"15min":19143.834999999999},"last":12.678000000000001}]},{"osd":3,"last update":"Mon Jun 3 09:06:23 2024","interfaces":[{"interface":"back","average":{"1min":9.7579999999999991,"5min":12.444000000000001,"15min":28.065000000000001},"min":{"1min":7,"5min":5.9290000000000003,"15min":5.2450000000000001},"max":{"1min":28.164000000000001,"5min":118.913,"15min":2594.8150000000001},"last":10.355},{"interface":"front","average":{"1min":12.321999999999999,"5min":15.25,"15min":31.349},"min":{"1min":9.1240000000000006,"5min":8.0760000000000005,"15min":6.0780000000000003},"max":{"1min":32.360999999999997,"5min":156.34700000000001,"15min":2706.5120000000002},"last":13.307}]},{"osd":4,"last update":"Mon Jun 3 09:06:59 2024","interfaces":[{"interface":"back","average":{"1min":11.795999999999999,"5min":16.489000000000001,"15min":437.19299999999998},"min":{"1min":7.8780000000000001,"5min":7.8780000000000001,"15min":6.8049999999999997},"max":{"1min":30.163,"5min":182.07400000000001,"15min":26660.861000000001},"last":11.215999999999999},{"interface":"front","average":{"1min":14.726000000000001,"5min":20.835000000000001,"15min":435.017},"min":{"1min":10.970000000000001,"5min":10.266999999999999,"15min":6.2720000000000002},"max":{"1min":33.402999999999999,"5min":216.476,"15min":26616.243999999999},"last":15.301}]},{"osd":5,"last update":"Mon Jun 3 09:06:23 2024","interfaces":[{"interface":"back","average":{"1min":14.103,"5min":18.091000000000001,"15min":21.119},"min":{"1min":9.6349999999999998,"5min":9.2159999999999993,"15min":6.2050000000000001},"max":{"1min":32.890000000000001,"5min":252.31100000000001,"15min":252.31100000000001},"last":16.004999999999999},{"interface":"front","average":{"1min":11.545999999999999,"5min":16.170999999999999,"15min":20.254000000000001},"min":{"1min":7.6070000000000002,"5min":7.4850000000000003,"15min":6.8810000000000002},"max":{"1min":29.228999999999999,"5min":258.31700000000001,"15min":258.31700000000001},"last":9.7330000000000005}]},{"osd":6,"last update":"Mon Jun 3 09:04:28 2024","interfaces":[{"interface":"back","average":{"1min":14.063000000000001,"5min":24.850000000000001,"15min":106.755},"min":{"1min":9.0820000000000007,"5min":8.9649999999999999,"15min":8.9649999999999999},"max":{"1min":24.666,"5min":188.06700000000001,"15min":9521.0010000000002},"last":14.417},{"interface":"front","average":{"1min":13.933999999999999,"5min":21.081,"15min":106.142},"min":{"1min":5.8719999999999999,"5min":5.8719999999999999,"15min":5.8719999999999999},"max":{"1min":117.551,"5min":193.887,"15min":9585.6620000000003},"last":6.4509999999999996}]},{"osd":7,"last update":"Mon Jun 3 09:06:23 2024","interfaces":[{"interface":"back","average":{"1min":17.056000000000001,"5min":20.055,"15min":23.268999999999998},"min":{"1min":10.353999999999999,"5min":6.6130000000000004,"15min":6.6130000000000004},"max":{"1min":88.974999999999994,"5min":242.56800000000001,"15min":242.56800000000001},"last":17.433},{"interface":"front","average":{"1min":18.885000000000002,"5min":21.460000000000001,"15min":25.489999999999998},"min":{"1min":12.497,"5min":9.0660000000000007,"15min":9.0660000000000007},"max":{"1min":102.229,"5min":181.56399999999999,"15min":214.33799999999999},"last":16.780999999999999}]}]},{"osd":3,"up_from":14,"seq":60129542513,"num_pgs":11,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":59008,"kb_used_data":5752,"kb_used_omap":1,"kb_used_meta":53246,"kb_avail":94312832,"statfs":{"total":96636764160,"available":96576339968,"internally_reserved":0,"allocated":5890048,"data_stored":2763073,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1592,"internal_metadata":54524360},"hb_peers":[0,1,2,4,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":9,"apply_latency_ms":9,"commit_latency_ns":9000000,"apply_latency_ns":9000000},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Mon Jun 3 09:06:25 2024","interfaces":[{"interface":"back","average":{"1min":7.79,"5min":1060.1959999999999,"15min":367.01799999999997},"min":{"1min":5.6909999999999998,"5min":5.173,"15min":4.6929999999999996},"max":{"1min":10.59,"5min":23275.167000000001,"15min":23275.167000000001},"last":5.7850000000000001},{"interface":"front","average":{"1min":11.215999999999999,"5min":1066.5799999999999,"15min":372.476},"min":{"1min":8.7080000000000002,"5min":7.2510000000000003,"15min":5.9880000000000004},"max":{"1min":13.499000000000001,"5min":23278.536,"15min":23278.536},"last":12.856}]},{"osd":1,"last update":"Mon Jun 3 09:06:25 2024","interfaces":[{"interface":"back","average":{"1min":7.3760000000000003,"5min":15.614000000000001,"15min":389.83100000000002},"min":{"1min":4.335,"5min":4.335,"15min":4.335},"max":{"1min":11.289,"5min":250.72300000000001,"15min":21658.205999999998},"last":8.6579999999999995},{"interface":"front","average":{"1min":8.9499999999999993,"5min":14.135,"15min":381.10300000000001},"min":{"1min":6.5609999999999999,"5min":5.4100000000000001,"15min":5.2779999999999996},"max":{"1min":14.51,"5min":180.44499999999999,"15min":21307.069},"last":13.512}]},{"osd":2,"last update":"Mon Jun 3 09:06:25 2024","interfaces":[{"interface":"back","average":{"1min":12.343999999999999,"5min":22.059999999999999,"15min":23.715},"min":{"1min":7.9199999999999999,"5min":6.4900000000000002,"15min":6.056},"max":{"1min":15.601000000000001,"5min":377.803,"15min":377.803},"last":14.071},{"interface":"front","average":{"1min":10.319000000000001,"5min":19.364999999999998,"15min":20.242999999999999},"min":{"1min":6.891,"5min":6.1749999999999998,"15min":5.9020000000000001},"max":{"1min":15.432,"5min":371.601,"15min":371.601},"last":7.3879999999999999}]},{"osd":4,"last update":"Mon Jun 3 09:06:31 2024","interfaces":[{"interface":"back","average":{"1min":9.7460000000000004,"5min":891.23000000000002,"15min":416.48099999999999},"min":{"1min":5.6539999999999999,"5min":5.6539999999999999,"15min":5.6539999999999999},"max":{"1min":15.445,"5min":29040.635999999999,"15min":30144.939999999999},"last":10.395},{"interface":"front","average":{"1min":12.398999999999999,"5min":897.58500000000004,"15min":421.27499999999998},"min":{"1min":8.9469999999999992,"5min":7.8109999999999999,"15min":6.2439999999999998},"max":{"1min":16.518000000000001,"5min":29076.955999999998,"15min":30180.689999999999},"last":17.635999999999999}]},{"osd":5,"last update":"Mon Jun 3 09:07:10 2024","interfaces":[{"interface":"back","average":{"1min":13.355,"5min":17.076000000000001,"15min":21.516999999999999},"min":{"1min":8.8510000000000009,"5min":6.8810000000000002,"15min":6.7690000000000001},"max":{"1min":20.768999999999998,"5min":137.64699999999999,"15min":327.34899999999999},"last":15.835000000000001},{"interface":"front","average":{"1min":15.882999999999999,"5min":21.332999999999998,"15min":26.346},"min":{"1min":12.529,"5min":9.9149999999999991,"15min":9.8849999999999998},"max":{"1min":25.289999999999999,"5min":182.32599999999999,"15min":331.86799999999999},"last":16.457000000000001}]},{"osd":6,"last update":"Mon Jun 3 09:04:04 2024","interfaces":[{"interface":"back","average":{"1min":22.815999999999999,"5min":31.513999999999999,"15min":181.74299999999999},"min":{"1min":9.4809999999999999,"5min":7.2240000000000002,"15min":7.2240000000000002},"max":{"1min":160.56999999999999,"5min":380.613,"15min":11893.74},"last":10.625},{"interface":"front","average":{"1min":25.359999999999999,"5min":30.981000000000002,"15min":179.81399999999999},"min":{"1min":9.8870000000000005,"5min":7.3360000000000003,"15min":6.0899999999999999},"max":{"1min":193.11099999999999,"5min":379.80700000000002,"15min":11893.069},"last":15.231}]},{"osd":7,"last update":"Mon Jun 3 09:06:25 2024","interfaces":[{"interface":"back","average":{"1min":11.321999999999999,"5min":18.443999999999999,"15min":21.568999999999999},"min":{"1min":8.109,"5min":8.109,"15min":7.4279999999999999},"max":{"1min":17.113,"5min":146.267,"15min":178.76300000000001},"last":11.247999999999999},{"interface":"front","average":{"1min":12.512,"5min":20.065999999999999,"15min":24.190000000000001},"min":{"1min":7.4989999999999997,"5min":7.4989999999999997,"15min":7.4989999999999997},"max":{"1min":18.047999999999998,"5min":148.13800000000001,"15min":417.22800000000001},"last":11.853}]}]},{"osd":1,"up_from":14,"seq":60129542511,"num_pgs":6,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":59400,"kb_used_data":6144,"kb_used_omap":1,"kb_used_meta":53246,"kb_avail":94312440,"statfs":{"total":96636764160,"available":96575938560,"internally_reserved":0,"allocated":6291456,"data_stored":3160913,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":54524357},"hb_peers":[0,2,3,4,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":3,"apply_latency_ms":3,"commit_latency_ns":3000000,"apply_latency_ns":3000000},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Mon Jun 3 09:07:08 2024","interfaces":[{"interface":"back","average":{"1min":7.8280000000000003,"5min":185.30600000000001,"15min":75.007999999999996},"min":{"1min":4.5090000000000003,"5min":4.5090000000000003,"15min":4.5090000000000003},"max":{"1min":16.443999999999999,"5min":7470.616,"15min":7470.616},"last":8.2400000000000002},{"interface":"front","average":{"1min":10.396000000000001,"5min":187.196,"15min":73.921000000000006},"min":{"1min":5.5709999999999997,"5min":5.3159999999999998,"15min":5.3159999999999998},"max":{"1min":20.145,"5min":7477.0829999999996,"15min":7477.0829999999996},"last":6.0289999999999999}]},{"osd":2,"last update":"Mon Jun 3 09:06:20 2024","interfaces":[{"interface":"back","average":{"1min":10.340999999999999,"5min":17.369,"15min":23.670999999999999},"min":{"1min":6.9740000000000002,"5min":5.8650000000000002,"15min":5.8650000000000002},"max":{"1min":15.452,"5min":228,"15min":657.38},"last":8.8230000000000004},{"interface":"front","average":{"1min":13.387,"5min":23.355,"15min":31.524000000000001},"min":{"1min":9.077,"5min":9.077,"15min":7.2160000000000002},"max":{"1min":19.701000000000001,"5min":229.465,"15min":700.06500000000005},"last":15.097}]},{"osd":3,"last update":"Mon Jun 3 09:07:04 2024","interfaces":[{"interface":"back","average":{"1min":8.5999999999999996,"5min":14.451000000000001,"15min":25.416},"min":{"1min":6.2830000000000004,"5min":5.4640000000000004,"15min":5.4640000000000004},"max":{"1min":11.763,"5min":121.619,"15min":1676.009},"last":6.649},{"interface":"front","average":{"1min":9.5679999999999996,"5min":15.148,"15min":27.706},"min":{"1min":6.0709999999999997,"5min":5.7699999999999996,"15min":5.7699999999999996},"max":{"1min":17.129999999999999,"5min":132.036,"15min":1766.5840000000001},"last":10.007999999999999}]},{"osd":4,"last update":"Mon Jun 3 09:06:35 2024","interfaces":[{"interface":"back","average":{"1min":8.6400000000000006,"5min":641.85500000000002,"15min":334.71499999999997},"min":{"1min":6.7640000000000002,"5min":6.1760000000000002,"15min":6.0970000000000004},"max":{"1min":11.811,"5min":26340.968000000001,"15min":28044.862000000001},"last":9.2119999999999997},{"interface":"front","average":{"1min":12.361000000000001,"5min":648.59900000000005,"15min":338.48500000000001},"min":{"1min":9.8949999999999996,"5min":9.3940000000000001,"15min":6.4640000000000004},"max":{"1min":15.507,"5min":26342.311000000002,"15min":28045.624},"last":12.273999999999999}]},{"osd":5,"last update":"Mon Jun 3 09:07:04 2024","interfaces":[{"interface":"back","average":{"1min":12.039,"5min":19.311,"15min":24.228999999999999},"min":{"1min":7.9050000000000002,"5min":7.9050000000000002,"15min":7.9050000000000002},"max":{"1min":17.462,"5min":139.43199999999999,"15min":701.22000000000003},"last":11.175000000000001},{"interface":"front","average":{"1min":12.936,"5min":24.681999999999999,"15min":26.170000000000002},"min":{"1min":9.6969999999999992,"5min":7.258,"15min":7.2480000000000002},"max":{"1min":16.297999999999998,"5min":278.63,"15min":598.35000000000002},"last":11.385}]},{"osd":6,"last update":"Mon Jun 3 09:03:54 2024","interfaces":[{"interface":"back","average":{"1min":28.524000000000001,"5min":20.654,"15min":23.878},"min":{"1min":6.0309999999999997,"5min":6.0309999999999997,"15min":6.0309999999999997},"max":{"1min":167.221,"5min":167.221,"15min":704.245},"last":13.756},{"interface":"front","average":{"1min":27.606000000000002,"5min":30.131,"15min":28.628},"min":{"1min":9.8650000000000002,"5min":9.7200000000000006,"15min":8.3580000000000005},"max":{"1min":123.938,"5min":358.23000000000002,"15min":634.05499999999995},"last":12.880000000000001}]},{"osd":7,"last update":"Mon Jun 3 09:06:20 2024","interfaces":[{"interface":"back","average":{"1min":14.66,"5min":23.234999999999999,"15min":31.097000000000001},"min":{"1min":11.250999999999999,"5min":8.0559999999999992,"15min":8.0559999999999992},"max":{"1min":21.507999999999999,"5min":270.95100000000002,"15min":703.07799999999997},"last":14.48},{"interface":"front","average":{"1min":15.548,"5min":28.760000000000002,"15min":28.925000000000001},"min":{"1min":12.759,"5min":8.2089999999999996,"15min":7.7599999999999998},"max":{"1min":22.408000000000001,"5min":360.99900000000002,"15min":608.44100000000003},"last":15.781000000000001}]}]},{"osd":0,"up_from":14,"seq":60129542513,"num_pgs":10,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":81152,"kb_used_data":5752,"kb_used_omap":1,"kb_used_meta":75390,"kb_avail":94290688,"statfs":{"total":96636764160,"available":96553664512,"internally_reserved":0,"allocated":5890048,"data_stored":2763073,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1593,"internal_metadata":77199815},"hb_peers":[1,2,3,4,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":19,"apply_latency_ms":19,"commit_latency_ns":19000000,"apply_latency_ns":19000000},"alerts":[],"network_ping_times":[{"osd":1,"last update":"Mon Jun 3 09:07:05 2024","interfaces":[{"interface":"back","average":{"1min":7.46,"5min":56.137,"15min":335.75799999999998},"min":{"1min":4.6040000000000001,"5min":4.4530000000000003,"15min":4.4100000000000001},"max":{"1min":13.083,"5min":2414.4749999999999,"15min":20691.453000000001},"last":11.874000000000001},{"interface":"front","average":{"1min":10.339,"5min":56.201999999999998,"15min":336.238},"min":{"1min":4.7199999999999998,"5min":4.7199999999999998,"15min":4.6050000000000004},"max":{"1min":25.372,"5min":2412.9549999999999,"15min":20640.792000000001},"last":6.04}]},{"osd":2,"last update":"Mon Jun 3 09:06:22 2024","interfaces":[{"interface":"back","average":{"1min":11.154,"5min":26.515999999999998,"15min":22.882000000000001},"min":{"1min":5.7869999999999999,"5min":5.766,"15min":5.766},"max":{"1min":72.269999999999996,"5min":627.40700000000004,"15min":627.40700000000004},"last":7.9050000000000002},{"interface":"front","average":{"1min":14.353999999999999,"5min":33.304000000000002,"15min":29.885000000000002},"min":{"1min":8.2149999999999999,"5min":7.8630000000000004,"15min":7.4089999999999998},"max":{"1min":73.418000000000006,"5min":635.62599999999998,"15min":635.62599999999998},"last":9.6460000000000008}]},{"osd":3,"last update":"Mon Jun 3 09:07:05 2024","interfaces":[{"interface":"back","average":{"1min":12.941000000000001,"5min":26.172999999999998,"15min":24.896999999999998},"min":{"1min":6.8739999999999997,"5min":6.8739999999999997,"15min":6.1909999999999998},"max":{"1min":33.140000000000001,"5min":389.72699999999998,"15min":389.72699999999998},"last":11.661},{"interface":"front","average":{"1min":15.823,"5min":29.198,"15min":26.542999999999999},"min":{"1min":10.595000000000001,"5min":9.8070000000000004,"15min":6.1849999999999996},"max":{"1min":31.582999999999998,"5min":390.23200000000003,"15min":390.23200000000003},"last":12.451000000000001}]},{"osd":4,"last update":"Mon Jun 3 09:06:27 2024","interfaces":[{"interface":"back","average":{"1min":11.352,"5min":745.30600000000004,"15min":363.48099999999999},"min":{"1min":6.415,"5min":6.415,"15min":6.415},"max":{"1min":16.724,"5min":20248.616000000002,"15min":25555.212},"last":10.298},{"interface":"front","average":{"1min":13.545,"5min":744.76999999999998,"15min":361.03100000000001},"min":{"1min":9.3889999999999993,"5min":8.8940000000000001,"15min":7.1429999999999998},"max":{"1min":18.411000000000001,"5min":20250.195,"15min":25556.159},"last":7.1669999999999998}]},{"osd":5,"last update":"Mon Jun 3 09:07:17 2024","interfaces":[{"interface":"back","average":{"1min":11.417999999999999,"5min":24.597000000000001,"15min":25.398},"min":{"1min":6.6539999999999999,"5min":6.6539999999999999,"15min":6.6539999999999999},"max":{"1min":25.068000000000001,"5min":455.78699999999998,"15min":455.78699999999998},"last":13.779},{"interface":"front","average":{"1min":16.913,"5min":28.289999999999999,"15min":23.193999999999999},"min":{"1min":10.563000000000001,"5min":9.2330000000000005,"15min":6.2030000000000003},"max":{"1min":29.972000000000001,"5min":391.31200000000001,"15min":391.31200000000001},"last":13.109999999999999}]},{"osd":6,"last update":"Mon Jun 3 09:04:30 2024","interfaces":[{"interface":"back","average":{"1min":52.496000000000002,"5min":33.139000000000003,"15min":160.83199999999999},"min":{"1min":7.5289999999999999,"5min":7.1040000000000001,"15min":6.3739999999999997},"max":{"1min":457.91699999999997,"5min":457.91699999999997,"15min":11115.258},"last":8.5210000000000008},{"interface":"front","average":{"1min":49.670000000000002,"5min":33.704999999999998,"15min":162.791},"min":{"1min":12.247999999999999,"5min":11.019,"15min":9.9760000000000009},"max":{"1min":407.05099999999999,"5min":407.05099999999999,"15min":11092.194},"last":15.032999999999999}]},{"osd":7,"last update":"Mon Jun 3 09:06:22 2024","interfaces":[{"interface":"back","average":{"1min":11.625999999999999,"5min":26.375,"15min":28.972999999999999},"min":{"1min":6.4480000000000004,"5min":6.4480000000000004,"15min":6.4480000000000004},"max":{"1min":15.352,"5min":461.41699999999997,"15min":598.33699999999999},"last":15.738},{"interface":"front","average":{"1min":14.013,"5min":29.696000000000002,"15min":33.630000000000003},"min":{"1min":10.938000000000001,"5min":10.58,"15min":10.263999999999999},"max":{"1min":16.603000000000002,"5min":463.10599999999999,"15min":592.80499999999995},"last":16.398}]}]}],"pool_statfs":[{"poolid":1,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":401408,"data_stored":397840,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":5,"total":0,"available":0,"internally_reserved":0,"allocated":401408,"data_stored":397840,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":4,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":5,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":6,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":17,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":17,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":17,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":17,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":17,"osd":4,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":17,"osd":5,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":5,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":17,"osd":6,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":5,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":17,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0}]}} 2024-06-03T09:07:23.739 INFO:tasks.ceph:pgid 17.c last_scrub_stamp 2024-06-03T08:37:55.614497+0000 time.struct_time(tm_year=2024, tm_mon=6, tm_mday=3, tm_hour=8, tm_min=37, tm_sec=55, tm_wday=0, tm_yday=155, tm_isdst=-1) <= time.struct_time(tm_year=2024, tm_mon=6, tm_mday=3, tm_hour=9, tm_min=6, tm_sec=55, tm_wday=0, tm_yday=155, tm_isdst=0) 2024-06-03T09:07:23.740 INFO:tasks.ceph:pgid 17.6 last_scrub_stamp 2024-06-03T08:37:55.614497+0000 time.struct_time(tm_year=2024, tm_mon=6, tm_mday=3, tm_hour=8, tm_min=37, tm_sec=55, tm_wday=0, tm_yday=155, tm_isdst=-1) <= time.struct_time(tm_year=2024, tm_mon=6, tm_mday=3, tm_hour=9, tm_min=6, tm_sec=55, tm_wday=0, tm_yday=155, tm_isdst=0) 2024-06-03T09:07:23.740 INFO:tasks.ceph:pgid 17.15 last_scrub_stamp 2024-06-03T08:37:55.614497+0000 time.struct_time(tm_year=2024, tm_mon=6, tm_mday=3, tm_hour=8, tm_min=37, tm_sec=55, tm_wday=0, tm_yday=155, tm_isdst=-1) <= time.struct_time(tm_year=2024, tm_mon=6, tm_mday=3, tm_hour=9, tm_min=6, tm_sec=55, tm_wday=0, tm_yday=155, tm_isdst=0) 2024-06-03T09:07:23.740 INFO:tasks.ceph:pgid 17.1b last_scrub_stamp 2024-06-03T08:37:55.614497+0000 time.struct_time(tm_year=2024, tm_mon=6, tm_mday=3, tm_hour=8, tm_min=37, tm_sec=55, tm_wday=0, tm_yday=155, tm_isdst=-1) <= time.struct_time(tm_year=2024, tm_mon=6, tm_mday=3, tm_hour=9, tm_min=6, tm_sec=55, tm_wday=0, tm_yday=155, tm_isdst=0) 2024-06-03T09:07:23.740 INFO:tasks.ceph:pgid 17.19 last_scrub_stamp 2024-06-03T08:37:55.614497+0000 time.struct_time(tm_year=2024, tm_mon=6, tm_mday=3, tm_hour=8, tm_min=37, tm_sec=55, tm_wday=0, tm_yday=155, tm_isdst=-1) <= time.struct_time(tm_year=2024, tm_mon=6, tm_mday=3, tm_hour=9, tm_min=6, tm_sec=55, tm_wday=0, tm_yday=155, tm_isdst=0) 2024-06-03T09:07:23.740 INFO:tasks.ceph:pgid 17.1c last_scrub_stamp 2024-06-03T08:37:55.614497+0000 time.struct_time(tm_year=2024, tm_mon=6, tm_mday=3, tm_hour=8, tm_min=37, tm_sec=55, tm_wday=0, tm_yday=155, tm_isdst=-1) <= time.struct_time(tm_year=2024, tm_mon=6, tm_mday=3, tm_hour=9, tm_min=6, tm_sec=55, tm_wday=0, tm_yday=155, tm_isdst=0) 2024-06-03T09:07:23.741 INFO:tasks.ceph:Still waiting for all pgs to be scrubbed. 2024-06-03T09:07:33.636 INFO:tasks.ceph.mon.a.smithi088.stderr:==00:00:32:28.962 34743== Warning: unimplemented fcntl command: 1036 2024-06-03T09:07:33.638 INFO:tasks.ceph.mon.a.smithi088.stderr:==00:00:32:28.964 34743== Warning: unimplemented fcntl command: 1036 2024-06-03T09:07:33.658 INFO:tasks.ceph.mon.a.smithi088.stderr:==00:00:32:28.984 34743== Warning: unimplemented fcntl command: 1036 2024-06-03T09:07:33.689 INFO:tasks.ceph.mon.b.smithi192.stderr:==00:00:32:28.992 34228== Warning: unimplemented fcntl command: 1036 2024-06-03T09:07:33.691 INFO:tasks.ceph.mon.c.smithi088.stderr:==00:00:32:29.017 34749== Warning: unimplemented fcntl command: 1036 2024-06-03T09:07:33.693 INFO:tasks.ceph.mon.b.smithi192.stderr:==00:00:32:28.996 34228== Warning: unimplemented fcntl command: 1036 2024-06-03T09:07:33.696 INFO:tasks.ceph.mon.c.smithi088.stderr:==00:00:32:29.021 34749== Warning: unimplemented fcntl command: 1036 2024-06-03T09:07:33.707 INFO:tasks.ceph.mon.b.smithi192.stderr:==00:00:32:29.010 34228== Warning: unimplemented fcntl command: 1036 2024-06-03T09:07:33.710 INFO:tasks.ceph.mon.c.smithi088.stderr:==00:00:32:29.036 34749== Warning: unimplemented fcntl command: 1036 2024-06-03T09:07:34.251 INFO:tasks.ceph.mon.b.smithi192.stderr:==00:00:32:29.553 34228== Warning: unimplemented fcntl command: 1036 2024-06-03T09:07:34.252 INFO:tasks.ceph.mon.b.smithi192.stderr:==00:00:32:29.555 34228== Warning: unimplemented fcntl command: 1036 2024-06-03T09:07:34.260 INFO:tasks.ceph.mon.b.smithi192.stderr:==00:00:32:29.562 34228== Warning: unimplemented fcntl command: 1036 2024-06-03T09:07:34.263 INFO:tasks.ceph.mon.a.smithi088.stderr:==00:00:32:29.589 34743== Warning: unimplemented fcntl command: 1036 2024-06-03T09:07:34.264 INFO:tasks.ceph.mon.a.smithi088.stderr:==00:00:32:29.590 34743== Warning: unimplemented fcntl command: 1036 2024-06-03T09:07:34.274 INFO:tasks.ceph.mon.a.smithi088.stderr:==00:00:32:29.600 34743== Warning: unimplemented fcntl command: 1036 2024-06-03T09:07:34.303 INFO:tasks.ceph.mon.c.smithi088.stderr:==00:00:32:29.629 34749== Warning: unimplemented fcntl command: 1036 2024-06-03T09:07:34.304 INFO:tasks.ceph.mon.c.smithi088.stderr:==00:00:32:29.630 34749== Warning: unimplemented fcntl command: 1036 2024-06-03T09:07:34.312 INFO:tasks.ceph.mon.c.smithi088.stderr:==00:00:32:29.638 34749== Warning: unimplemented fcntl command: 1036 2024-06-03T09:07:43.742 DEBUG:teuthology.orchestra.run.smithi088:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph pg dump --format=json 2024-06-03T09:07:43.841 INFO:teuthology.orchestra.run.smithi088.stderr:2024-06-03T09:07:43.841+0000 7fddd3a70640 10 monclient: get_monmap_and_config 2024-06-03T09:07:43.842 INFO:teuthology.orchestra.run.smithi088.stderr:2024-06-03T09:07:43.841+0000 7fddd3a70640 10 monclient: build_initial_monmap 2024-06-03T09:07:43.842 INFO:teuthology.orchestra.run.smithi088.stderr:2024-06-03T09:07:43.841+0000 7fddd3a70640 10 monclient: monmap: 2024-06-03T09:07:43.842 INFO:teuthology.orchestra.run.smithi088.stderr:epoch 0 2024-06-03T09:07:43.842 INFO:teuthology.orchestra.run.smithi088.stderr:fsid ec3c3fdd-3ac6-4d7d-9412-683130ed8b04 2024-06-03T09:07:43.842 INFO:teuthology.orchestra.run.smithi088.stderr:last_changed 2024-06-03T09:07:43.842802+0000 2024-06-03T09:07:43.842 INFO:teuthology.orchestra.run.smithi088.stderr:created 2024-06-03T09:07:43.842802+0000 2024-06-03T09:07:43.842 INFO:teuthology.orchestra.run.smithi088.stderr:min_mon_release 0 (unknown) 2024-06-03T09:07:43.842 INFO:teuthology.orchestra.run.smithi088.stderr:election_strategy: 3 2024-06-03T09:07:43.842 INFO:teuthology.orchestra.run.smithi088.stderr:0: v1:172.21.15.88:6789/0 mon.noname-a 2024-06-03T09:07:43.842 INFO:teuthology.orchestra.run.smithi088.stderr:1: v1:172.21.15.192:6789/0 mon.noname-c 2024-06-03T09:07:43.842 INFO:teuthology.orchestra.run.smithi088.stderr:2: v1:172.21.15.88:6790/0 mon.noname-b 2024-06-03T09:07:43.843 INFO:teuthology.orchestra.run.smithi088.stderr: 2024-06-03T09:07:43.844 INFO:teuthology.orchestra.run.smithi088.stderr:2024-06-03T09:07:43.842+0000 7fddd3a70640 1 Processor -- start 2024-06-03T09:07:43.844 INFO:teuthology.orchestra.run.smithi088.stderr:2024-06-03T09:07:43.842+0000 7fddd3a70640 1 -- start start 2024-06-03T09:07:43.844 INFO:teuthology.orchestra.run.smithi088.stderr:2024-06-03T09:07:43.842+0000 7fddd3a70640 10 monclient: init 2024-06-03T09:07:43.844 INFO:teuthology.orchestra.run.smithi088.stderr:2024-06-03T09:07:43.843+0000 7fddd3a70640 10 monclient: _reopen_session rank -1 2024-06-03T09:07:43.844 INFO:teuthology.orchestra.run.smithi088.stderr:2024-06-03T09:07:43.843+0000 7fddd3a70640 10 monclient: _add_conns ranks=[1,2,0] 2024-06-03T09:07:43.844 INFO:teuthology.orchestra.run.smithi088.stderr:2024-06-03T09:07:43.843+0000 7fddd3a70640 10 monclient(hunting): picked mon.noname-c con 0x7fddcc06e050 addr v1:172.21.15.192:6789/0 2024-06-03T09:07:43.844 INFO:teuthology.orchestra.run.smithi088.stderr:2024-06-03T09:07:43.843+0000 7fddd3a70640 10 monclient(hunting): picked mon.noname-b con 0x7fddcc06e430 addr v1:172.21.15.88:6790/0 2024-06-03T09:07:43.844 INFO:teuthology.orchestra.run.smithi088.stderr:2024-06-03T09:07:43.843+0000 7fddd3a70640 10 monclient(hunting): picked mon.noname-a con 0x7fddcc0717c0 addr v1:172.21.15.88:6789/0 2024-06-03T09:07:43.844 INFO:teuthology.orchestra.run.smithi088.stderr:2024-06-03T09:07:43.843+0000 7fddd3a70640 1 -- --> v1:172.21.15.88:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fddcc070c10 con 0x7fddcc0717c0 2024-06-03T09:07:43.844 INFO:teuthology.orchestra.run.smithi088.stderr:2024-06-03T09:07:43.843+0000 7fddd3a70640 1 -- --> v1:172.21.15.192:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fddcc070230 con 0x7fddcc06e050 2024-06-03T09:07:43.844 INFO:teuthology.orchestra.run.smithi088.stderr:2024-06-03T09:07:43.843+0000 7fddd3a70640 1 -- --> v1:172.21.15.88:6790/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fddcc070510 con 0x7fddcc06e430 2024-06-03T09:07:43.845 INFO:teuthology.orchestra.run.smithi088.stderr:2024-06-03T09:07:43.843+0000 7fddd3a70640 10 monclient(hunting): _renew_subs 2024-06-03T09:07:43.845 INFO:teuthology.orchestra.run.smithi088.stderr:2024-06-03T09:07:43.843+0000 7fddd3a70640 10 monclient(hunting): authenticate will time out at 3119.653168s 2024-06-03T09:07:43.845 INFO:teuthology.orchestra.run.smithi088.stderr:2024-06-03T09:07:43.843+0000 7fddd200d640 1 --1- >> v1:172.21.15.88:6789/0 conn(0x7fddcc0717c0 0x7fddcc149240 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.88:6789/0 says I am v1:172.21.15.88:43268/0 (socket says 172.21.15.88:43268) 2024-06-03T09:07:43.845 INFO:teuthology.orchestra.run.smithi088.stderr:2024-06-03T09:07:43.843+0000 7fddd200d640 1 -- 172.21.15.88:0/2919879374 learned_addr learned my addr 172.21.15.88:0/2919879374 (peer_addr_for_me v1:172.21.15.88:0/0) 2024-06-03T09:07:43.845 INFO:teuthology.orchestra.run.smithi088.stderr:2024-06-03T09:07:43.845+0000 7fddd080a640 1 -- 172.21.15.88:0/2919879374 <== mon.0 v1:172.21.15.88:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 374512455 0 0) 0x7fddcc070c10 con 0x7fddcc0717c0 2024-06-03T09:07:43.846 INFO:teuthology.orchestra.run.smithi088.stderr:2024-06-03T09:07:43.846+0000 7fddd080a640 10 monclient(hunting): _negotiate 2024-06-03T09:07:43.846 INFO:teuthology.orchestra.run.smithi088.stderr:2024-06-03T09:07:43.846+0000 7fddd080a640 10 monclient(hunting): _init_auth method 2 2024-06-03T09:07:43.846 INFO:teuthology.orchestra.run.smithi088.stderr:2024-06-03T09:07:43.846+0000 7fddd080a640 10 monclient(hunting): _init_auth creating new auth 2024-06-03T09:07:43.846 INFO:teuthology.orchestra.run.smithi088.stderr:2024-06-03T09:07:43.846+0000 7fddd080a640 10 monclient(hunting): my global_id is 30837 2024-06-03T09:07:43.846 INFO:teuthology.orchestra.run.smithi088.stderr:2024-06-03T09:07:43.846+0000 7fddd080a640 1 -- 172.21.15.88:0/2919879374 --> v1:172.21.15.88:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7fddb8004630 con 0x7fddcc0717c0 2024-06-03T09:07:43.846 INFO:teuthology.orchestra.run.smithi088.stderr:2024-06-03T09:07:43.846+0000 7fddd080a640 1 -- 172.21.15.88:0/2919879374 <== mon.2 v1:172.21.15.88:6790/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 2433726623 0 0) 0x7fddcc070510 con 0x7fddcc06e430 2024-06-03T09:07:43.846 INFO:teuthology.orchestra.run.smithi088.stderr:2024-06-03T09:07:43.846+0000 7fddd080a640 10 monclient(hunting): _negotiate 2024-06-03T09:07:43.846 INFO:teuthology.orchestra.run.smithi088.stderr:2024-06-03T09:07:43.846+0000 7fddd080a640 10 monclient(hunting): _init_auth method 2 2024-06-03T09:07:43.846 INFO:teuthology.orchestra.run.smithi088.stderr:2024-06-03T09:07:43.846+0000 7fddd080a640 10 monclient(hunting): _init_auth creating new auth 2024-06-03T09:07:43.846 INFO:teuthology.orchestra.run.smithi088.stderr:2024-06-03T09:07:43.846+0000 7fddd080a640 10 monclient(hunting): my global_id is 30347 2024-06-03T09:07:43.851 INFO:teuthology.orchestra.run.smithi088.stderr:2024-06-03T09:07:43.846+0000 7fddd080a640 1 -- 172.21.15.88:0/2919879374 --> v1:172.21.15.88:6790/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7fddcc070c10 con 0x7fddcc06e430 2024-06-03T09:07:43.853 INFO:teuthology.orchestra.run.smithi088.stderr:2024-06-03T09:07:43.846+0000 7fddd080a640 1 -- 172.21.15.88:0/2919879374 <== mon.1 v1:172.21.15.192:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 1486522991 0 0) 0x7fddcc070230 con 0x7fddcc06e050 2024-06-03T09:07:43.853 INFO:teuthology.orchestra.run.smithi088.stderr:2024-06-03T09:07:43.846+0000 7fddd080a640 10 monclient(hunting): _negotiate 2024-06-03T09:07:43.853 INFO:teuthology.orchestra.run.smithi088.stderr:2024-06-03T09:07:43.846+0000 7fddd080a640 10 monclient(hunting): _init_auth method 2 2024-06-03T09:07:43.853 INFO:teuthology.orchestra.run.smithi088.stderr:2024-06-03T09:07:43.846+0000 7fddd080a640 10 monclient(hunting): _init_auth creating new auth 2024-06-03T09:07:43.853 INFO:teuthology.orchestra.run.smithi088.stderr:2024-06-03T09:07:43.846+0000 7fddd080a640 10 monclient(hunting): my global_id is 30412 2024-06-03T09:07:43.853 INFO:teuthology.orchestra.run.smithi088.stderr:2024-06-03T09:07:43.846+0000 7fddd080a640 1 -- 172.21.15.88:0/2919879374 --> v1:172.21.15.192:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7fddcc070510 con 0x7fddcc06e050 2024-06-03T09:07:43.853 INFO:teuthology.orchestra.run.smithi088.stderr:2024-06-03T09:07:43.850+0000 7fddd080a640 1 -- 172.21.15.88:0/2919879374 <== mon.2 v1:172.21.15.88:6790/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 214+0+0 (unknown 2311491735 0 0) 0x7fddcc070c10 con 0x7fddcc06e430 2024-06-03T09:07:43.853 INFO:teuthology.orchestra.run.smithi088.stderr:2024-06-03T09:07:43.850+0000 7fddd080a640 1 -- 172.21.15.88:0/2919879374 >> v1:172.21.15.192:6789/0 conn(0x7fddcc06e050 legacy=0x7fddcc13ad00 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-06-03T09:07:43.853 INFO:teuthology.orchestra.run.smithi088.stderr:2024-06-03T09:07:43.850+0000 7fddd080a640 1 -- 172.21.15.88:0/2919879374 >> v1:172.21.15.88:6789/0 conn(0x7fddcc0717c0 legacy=0x7fddcc149240 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-06-03T09:07:43.853 INFO:teuthology.orchestra.run.smithi088.stderr:2024-06-03T09:07:43.850+0000 7fddd080a640 10 monclient: _finish_hunting 0 2024-06-03T09:07:43.853 INFO:teuthology.orchestra.run.smithi088.stderr:2024-06-03T09:07:43.850+0000 7fddd080a640 1 monclient: found mon.noname-b 2024-06-03T09:07:43.853 INFO:teuthology.orchestra.run.smithi088.stderr:2024-06-03T09:07:43.850+0000 7fddd080a640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-06-03T09:07:43.854 INFO:teuthology.orchestra.run.smithi088.stderr:2024-06-03T09:07:43.850+0000 7fddd080a640 10 monclient: _send_mon_message to mon.noname-b at v1:172.21.15.88:6790/0 2024-06-03T09:07:43.854 INFO:teuthology.orchestra.run.smithi088.stderr:2024-06-03T09:07:43.850+0000 7fddd080a640 1 -- 172.21.15.88:0/2919879374 --> v1:172.21.15.88:6790/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fddcc0707f0 con 0x7fddcc06e430 2024-06-03T09:07:43.854 INFO:teuthology.orchestra.run.smithi088.stderr:2024-06-03T09:07:43.850+0000 7fddd080a640 10 monclient: _finish_auth 0 2024-06-03T09:07:43.854 INFO:teuthology.orchestra.run.smithi088.stderr:2024-06-03T09:07:43.850+0000 7fddd080a640 10 monclient: _check_auth_tickets 2024-06-03T09:07:43.854 INFO:teuthology.orchestra.run.smithi088.stderr:2024-06-03T09:07:43.850+0000 7fddd080a640 20 monclient: _check_auth_rotating not needed by client.admin 2024-06-03T09:07:43.854 INFO:teuthology.orchestra.run.smithi088.stderr:2024-06-03T09:07:43.850+0000 7fddd3a70640 5 monclient: authenticate success, global_id 30347 2024-06-03T09:07:43.854 INFO:teuthology.orchestra.run.smithi088.stderr:2024-06-03T09:07:43.850+0000 7fddd3a70640 20 monclient: get_monmap_and_config waiting for monmap|config 2024-06-03T09:07:43.854 INFO:teuthology.orchestra.run.smithi088.stderr:2024-06-03T09:07:43.850+0000 7fddd080a640 1 -- 172.21.15.88:0/2919879374 <== mon.2 v1:172.21.15.88:6790/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 2234374065 0 0) 0x7fddbc0015f0 con 0x7fddcc06e430 2024-06-03T09:07:43.854 INFO:teuthology.orchestra.run.smithi088.stderr:2024-06-03T09:07:43.850+0000 7fddd080a640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-06-03T09:07:43.854 INFO:teuthology.orchestra.run.smithi088.stderr:2024-06-03T09:07:43.850+0000 7fddd080a640 10 monclient: got monmap 1 from mon.noname-b (according to old e1) 2024-06-03T09:07:43.854 INFO:teuthology.orchestra.run.smithi088.stderr:2024-06-03T09:07:43.850+0000 7fddd080a640 10 monclient: dump: 2024-06-03T09:07:43.855 INFO:teuthology.orchestra.run.smithi088.stderr:epoch 1 2024-06-03T09:07:43.855 INFO:teuthology.orchestra.run.smithi088.stderr:fsid ec3c3fdd-3ac6-4d7d-9412-683130ed8b04 2024-06-03T09:07:43.855 INFO:teuthology.orchestra.run.smithi088.stderr:last_changed 2024-06-03T08:34:31.885125+0000 2024-06-03T09:07:43.855 INFO:teuthology.orchestra.run.smithi088.stderr:created 2024-06-03T08:34:31.885125+0000 2024-06-03T09:07:43.855 INFO:teuthology.orchestra.run.smithi088.stderr:min_mon_release 18 (reef) 2024-06-03T09:07:43.855 INFO:teuthology.orchestra.run.smithi088.stderr:election_strategy: 3 2024-06-03T09:07:43.855 INFO:teuthology.orchestra.run.smithi088.stderr:0: v1:172.21.15.88:6789/0 mon.a 2024-06-03T09:07:43.855 INFO:teuthology.orchestra.run.smithi088.stderr:1: v1:172.21.15.192:6789/0 mon.b 2024-06-03T09:07:43.855 INFO:teuthology.orchestra.run.smithi088.stderr:2: v1:172.21.15.88:6790/0 mon.c 2024-06-03T09:07:43.855 INFO:teuthology.orchestra.run.smithi088.stderr: 2024-06-03T09:07:43.855 INFO:teuthology.orchestra.run.smithi088.stderr:2024-06-03T09:07:43.850+0000 7fddd3a70640 20 monclient: get_monmap_and_config waiting for monmap|config 2024-06-03T09:07:43.855 INFO:teuthology.orchestra.run.smithi088.stderr:2024-06-03T09:07:43.852+0000 7fddd080a640 1 -- 172.21.15.88:0/2919879374 <== mon.2 v1:172.21.15.88:6790/0 4 ==== config(0 keys) v1 ==== 4+0+0 (unknown 0 0 0) 0x7fddbc001840 con 0x7fddcc06e430 2024-06-03T09:07:43.855 INFO:teuthology.orchestra.run.smithi088.stderr:2024-06-03T09:07:43.852+0000 7fddd080a640 10 monclient: handle_config config(0 keys) v1 2024-06-03T09:07:43.855 INFO:teuthology.orchestra.run.smithi088.stderr:2024-06-03T09:07:43.852+0000 7fddd080a640 1 -- 172.21.15.88:0/2919879374 <== mon.2 v1:172.21.15.88:6790/0 5 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 2234374065 0 0) 0x7fddbc001c10 con 0x7fddcc06e430 2024-06-03T09:07:43.856 INFO:teuthology.orchestra.run.smithi088.stderr:2024-06-03T09:07:43.852+0000 7fddd080a640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-06-03T09:07:43.856 INFO:teuthology.orchestra.run.smithi088.stderr:2024-06-03T09:07:43.852+0000 7fddd080a640 10 monclient: got monmap 1 from mon.c (according to old e1) 2024-06-03T09:07:43.856 INFO:teuthology.orchestra.run.smithi088.stderr:2024-06-03T09:07:43.852+0000 7fddd080a640 10 monclient: dump: 2024-06-03T09:07:43.856 INFO:teuthology.orchestra.run.smithi088.stderr:epoch 1 2024-06-03T09:07:43.856 INFO:teuthology.orchestra.run.smithi088.stderr:fsid ec3c3fdd-3ac6-4d7d-9412-683130ed8b04 2024-06-03T09:07:43.856 INFO:teuthology.orchestra.run.smithi088.stderr:last_changed 2024-06-03T08:34:31.885125+0000 2024-06-03T09:07:43.856 INFO:teuthology.orchestra.run.smithi088.stderr:created 2024-06-03T08:34:31.885125+0000 2024-06-03T09:07:43.856 INFO:teuthology.orchestra.run.smithi088.stderr:min_mon_release 18 (reef) 2024-06-03T09:07:43.856 INFO:teuthology.orchestra.run.smithi088.stderr:election_strategy: 3 2024-06-03T09:07:43.856 INFO:teuthology.orchestra.run.smithi088.stderr:0: v1:172.21.15.88:6789/0 mon.a 2024-06-03T09:07:43.856 INFO:teuthology.orchestra.run.smithi088.stderr:1: v1:172.21.15.192:6789/0 mon.b 2024-06-03T09:07:43.856 INFO:teuthology.orchestra.run.smithi088.stderr:2: v1:172.21.15.88:6790/0 mon.c 2024-06-03T09:07:43.856 INFO:teuthology.orchestra.run.smithi088.stderr: 2024-06-03T09:07:43.857 INFO:teuthology.orchestra.run.smithi088.stderr:2024-06-03T09:07:43.852+0000 7fddd3a70640 10 monclient: get_monmap_and_config success 2024-06-03T09:07:43.857 INFO:teuthology.orchestra.run.smithi088.stderr:2024-06-03T09:07:43.852+0000 7fddd3a70640 4 set_mon_vals no callback set 2024-06-03T09:07:43.857 INFO:teuthology.orchestra.run.smithi088.stderr:2024-06-03T09:07:43.852+0000 7fddd3a70640 10 monclient: shutdown 2024-06-03T09:07:43.857 INFO:teuthology.orchestra.run.smithi088.stderr:2024-06-03T09:07:43.852+0000 7fddd3a70640 20 monclient: shutdown discarding 0 pending message(s) 2024-06-03T09:07:43.857 INFO:teuthology.orchestra.run.smithi088.stderr:2024-06-03T09:07:43.852+0000 7fddd3a70640 1 -- 172.21.15.88:0/2919879374 >> v1:172.21.15.88:6790/0 conn(0x7fddcc06e430 legacy=0x7fddcc0710b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-06-03T09:07:43.857 INFO:teuthology.orchestra.run.smithi088.stderr:2024-06-03T09:07:43.852+0000 7fddd3a70640 1 -- 172.21.15.88:0/2919879374 shutdown_connections 2024-06-03T09:07:43.857 INFO:teuthology.orchestra.run.smithi088.stderr:2024-06-03T09:07:43.852+0000 7fddd3a70640 1 -- 172.21.15.88:0/2919879374 >> 172.21.15.88:0/2919879374 conn(0x7fddcc0b2f20 msgr2=0x7fddcc0b3300 unknown :-1 s=STATE_NONE l=0).mark_down 2024-06-03T09:07:43.857 INFO:teuthology.orchestra.run.smithi088.stderr:2024-06-03T09:07:43.852+0000 7fddd3a70640 1 -- 172.21.15.88:0/2919879374 shutdown_connections 2024-06-03T09:07:43.857 INFO:teuthology.orchestra.run.smithi088.stderr:2024-06-03T09:07:43.852+0000 7fddd3a70640 1 -- 172.21.15.88:0/2919879374 wait complete. 2024-06-03T09:07:43.860 INFO:teuthology.orchestra.run.smithi088.stderr:2024-06-03T09:07:43.853+0000 7fddd3a70640 10 monclient: build_initial_monmap 2024-06-03T09:07:43.860 INFO:teuthology.orchestra.run.smithi088.stderr:2024-06-03T09:07:43.853+0000 7fddd3a70640 10 monclient: monmap: 2024-06-03T09:07:43.860 INFO:teuthology.orchestra.run.smithi088.stderr:epoch 0 2024-06-03T09:07:43.860 INFO:teuthology.orchestra.run.smithi088.stderr:fsid 00000000-0000-0000-0000-000000000000 2024-06-03T09:07:43.860 INFO:teuthology.orchestra.run.smithi088.stderr:last_changed 0.000000 2024-06-03T09:07:43.860 INFO:teuthology.orchestra.run.smithi088.stderr:created 0.000000 2024-06-03T09:07:43.860 INFO:teuthology.orchestra.run.smithi088.stderr:min_mon_release 0 (unknown) 2024-06-03T09:07:43.860 INFO:teuthology.orchestra.run.smithi088.stderr:election_strategy: 1 2024-06-03T09:07:43.860 INFO:teuthology.orchestra.run.smithi088.stderr:0: v1:172.21.15.88:6789/0 mon.noname-a 2024-06-03T09:07:43.860 INFO:teuthology.orchestra.run.smithi088.stderr:1: v1:172.21.15.192:6789/0 mon.noname-b 2024-06-03T09:07:43.860 INFO:teuthology.orchestra.run.smithi088.stderr:2: v1:172.21.15.88:6790/0 mon.noname-c 2024-06-03T09:07:43.861 INFO:teuthology.orchestra.run.smithi088.stderr: 2024-06-03T09:07:43.861 INFO:teuthology.orchestra.run.smithi088.stderr:2024-06-03T09:07:43.853+0000 7fddd3a70640 1 librados: starting msgr at 2024-06-03T09:07:43.861 INFO:teuthology.orchestra.run.smithi088.stderr:2024-06-03T09:07:43.853+0000 7fddd3a70640 1 librados: starting objecter 2024-06-03T09:07:43.861 INFO:teuthology.orchestra.run.smithi088.stderr:2024-06-03T09:07:43.853+0000 7fddd3a70640 1 Processor -- start 2024-06-03T09:07:43.861 INFO:teuthology.orchestra.run.smithi088.stderr:2024-06-03T09:07:43.853+0000 7fddd3a70640 1 -- start start 2024-06-03T09:07:43.861 INFO:teuthology.orchestra.run.smithi088.stderr:2024-06-03T09:07:43.853+0000 7fddd3a70640 1 librados: setting wanted keys 2024-06-03T09:07:43.862 INFO:teuthology.orchestra.run.smithi088.stderr:2024-06-03T09:07:43.854+0000 7fddd3a70640 1 librados: calling monclient init 2024-06-03T09:07:43.862 INFO:teuthology.orchestra.run.smithi088.stderr:2024-06-03T09:07:43.854+0000 7fddd3a70640 10 monclient: init 2024-06-03T09:07:43.862 INFO:teuthology.orchestra.run.smithi088.stderr:2024-06-03T09:07:43.854+0000 7fddd3a70640 10 monclient: _reopen_session rank -1 2024-06-03T09:07:43.862 INFO:teuthology.orchestra.run.smithi088.stderr:2024-06-03T09:07:43.854+0000 7fddd3a70640 10 monclient: _add_conns ranks=[0,2,1] 2024-06-03T09:07:43.862 INFO:teuthology.orchestra.run.smithi088.stderr:2024-06-03T09:07:43.854+0000 7fddd3a70640 10 monclient(hunting): picked mon.noname-a con 0x7fddcc06e050 addr v1:172.21.15.88:6789/0 2024-06-03T09:07:43.862 INFO:teuthology.orchestra.run.smithi088.stderr:2024-06-03T09:07:43.854+0000 7fddd3a70640 10 monclient(hunting): picked mon.noname-c con 0x7fddcc0717c0 addr v1:172.21.15.88:6790/0 2024-06-03T09:07:43.863 INFO:teuthology.orchestra.run.smithi088.stderr:2024-06-03T09:07:43.854+0000 7fddd3a70640 10 monclient(hunting): picked mon.noname-b con 0x7fddcc0a93c0 addr v1:172.21.15.192:6789/0 2024-06-03T09:07:43.863 INFO:teuthology.orchestra.run.smithi088.stderr:2024-06-03T09:07:43.854+0000 7fddd3a70640 1 -- --> v1:172.21.15.88:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fddcc0a98e0 con 0x7fddcc06e050 2024-06-03T09:07:43.863 INFO:teuthology.orchestra.run.smithi088.stderr:2024-06-03T09:07:43.854+0000 7fddd3a70640 1 -- --> v1:172.21.15.192:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fddcc0aac80 con 0x7fddcc0a93c0 2024-06-03T09:07:43.863 INFO:teuthology.orchestra.run.smithi088.stderr:2024-06-03T09:07:43.854+0000 7fddd3a70640 1 -- --> v1:172.21.15.88:6790/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fddcc0a91a0 con 0x7fddcc0717c0 2024-06-03T09:07:43.863 INFO:teuthology.orchestra.run.smithi088.stderr:2024-06-03T09:07:43.854+0000 7fddd3a70640 10 monclient(hunting): _renew_subs 2024-06-03T09:07:43.863 INFO:teuthology.orchestra.run.smithi088.stderr:2024-06-03T09:07:43.854+0000 7fddd3a70640 10 monclient(hunting): authenticate will time out at 3119.664542s 2024-06-03T09:07:43.863 INFO:teuthology.orchestra.run.smithi088.stderr:2024-06-03T09:07:43.854+0000 7fddd180c640 1 --1- >> v1:172.21.15.88:6789/0 conn(0x7fddcc06e050 0x7fddcc0a8a90 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=1).handle_server_banner_and_identify peer v1:172.21.15.88:6789/0 says I am v1:172.21.15.88:43282/0 (socket says 172.21.15.88:43282) 2024-06-03T09:07:43.863 INFO:teuthology.orchestra.run.smithi088.stderr:2024-06-03T09:07:43.854+0000 7fddd180c640 1 -- 172.21.15.88:0/1349199537 learned_addr learned my addr 172.21.15.88:0/1349199537 (peer_addr_for_me v1:172.21.15.88:0/0) 2024-06-03T09:07:43.863 INFO:teuthology.orchestra.run.smithi088.stderr:2024-06-03T09:07:43.855+0000 7fddc27fc640 10 client.?.objecter ms_handle_connect 0x7fddcc06e050 2024-06-03T09:07:43.863 INFO:teuthology.orchestra.run.smithi088.stderr:2024-06-03T09:07:43.855+0000 7fddc27fc640 10 client.?.objecter resend_mon_ops 2024-06-03T09:07:43.863 INFO:teuthology.orchestra.run.smithi088.stderr:2024-06-03T09:07:43.855+0000 7fddc27fc640 10 client.?.objecter ms_handle_connect 0x7fddcc0717c0 2024-06-03T09:07:43.863 INFO:teuthology.orchestra.run.smithi088.stderr:2024-06-03T09:07:43.855+0000 7fddc27fc640 10 client.?.objecter resend_mon_ops 2024-06-03T09:07:43.864 INFO:teuthology.orchestra.run.smithi088.stderr:2024-06-03T09:07:43.855+0000 7fddc27fc640 10 client.?.objecter ms_handle_connect 0x7fddcc0a93c0 2024-06-03T09:07:43.864 INFO:teuthology.orchestra.run.smithi088.stderr:2024-06-03T09:07:43.855+0000 7fddc27fc640 10 client.?.objecter resend_mon_ops 2024-06-03T09:07:43.864 INFO:teuthology.orchestra.run.smithi088.stderr:2024-06-03T09:07:43.857+0000 7fddc27fc640 1 -- 172.21.15.88:0/1349199537 <== mon.0 v1:172.21.15.88:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 3413959645 0 0) 0x7fddcc0a98e0 con 0x7fddcc06e050 2024-06-03T09:07:43.864 INFO:teuthology.orchestra.run.smithi088.stderr:2024-06-03T09:07:43.857+0000 7fddc27fc640 10 monclient(hunting): _negotiate 2024-06-03T09:07:43.864 INFO:teuthology.orchestra.run.smithi088.stderr:2024-06-03T09:07:43.857+0000 7fddc27fc640 10 monclient(hunting): _init_auth method 2 2024-06-03T09:07:43.864 INFO:teuthology.orchestra.run.smithi088.stderr:2024-06-03T09:07:43.857+0000 7fddc27fc640 10 monclient(hunting): _init_auth creating new auth 2024-06-03T09:07:43.864 INFO:teuthology.orchestra.run.smithi088.stderr:2024-06-03T09:07:43.857+0000 7fddc27fc640 10 monclient(hunting): my global_id is 30840 2024-06-03T09:07:43.864 INFO:teuthology.orchestra.run.smithi088.stderr:2024-06-03T09:07:43.857+0000 7fddc27fc640 1 -- 172.21.15.88:0/1349199537 --> v1:172.21.15.88:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7fddb0004eb0 con 0x7fddcc06e050 2024-06-03T09:07:43.864 INFO:teuthology.orchestra.run.smithi088.stderr:2024-06-03T09:07:43.857+0000 7fddc27fc640 1 -- 172.21.15.88:0/1349199537 <== mon.2 v1:172.21.15.88:6790/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 2597538197 0 0) 0x7fddcc0a91a0 con 0x7fddcc0717c0 2024-06-03T09:07:43.864 INFO:teuthology.orchestra.run.smithi088.stderr:2024-06-03T09:07:43.857+0000 7fddc27fc640 10 monclient(hunting): _negotiate 2024-06-03T09:07:43.864 INFO:teuthology.orchestra.run.smithi088.stderr:2024-06-03T09:07:43.857+0000 7fddc27fc640 10 monclient(hunting): _init_auth method 2 2024-06-03T09:07:43.864 INFO:teuthology.orchestra.run.smithi088.stderr:2024-06-03T09:07:43.857+0000 7fddc27fc640 10 monclient(hunting): _init_auth creating new auth 2024-06-03T09:07:43.864 INFO:teuthology.orchestra.run.smithi088.stderr:2024-06-03T09:07:43.857+0000 7fddc27fc640 10 monclient(hunting): my global_id is 30350 2024-06-03T09:07:43.864 INFO:teuthology.orchestra.run.smithi088.stderr:2024-06-03T09:07:43.857+0000 7fddc27fc640 1 -- 172.21.15.88:0/1349199537 --> v1:172.21.15.88:6790/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7fddcc0a98e0 con 0x7fddcc0717c0 2024-06-03T09:07:43.865 INFO:teuthology.orchestra.run.smithi088.stderr:2024-06-03T09:07:43.857+0000 7fddc27fc640 1 -- 172.21.15.88:0/1349199537 <== mon.1 v1:172.21.15.192:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 1688846679 0 0) 0x7fddcc0aac80 con 0x7fddcc0a93c0 2024-06-03T09:07:43.865 INFO:teuthology.orchestra.run.smithi088.stderr:2024-06-03T09:07:43.857+0000 7fddc27fc640 10 monclient(hunting): _negotiate 2024-06-03T09:07:43.865 INFO:teuthology.orchestra.run.smithi088.stderr:2024-06-03T09:07:43.857+0000 7fddc27fc640 10 monclient(hunting): _init_auth method 2 2024-06-03T09:07:43.865 INFO:teuthology.orchestra.run.smithi088.stderr:2024-06-03T09:07:43.857+0000 7fddc27fc640 10 monclient(hunting): _init_auth creating new auth 2024-06-03T09:07:43.865 INFO:teuthology.orchestra.run.smithi088.stderr:2024-06-03T09:07:43.857+0000 7fddc27fc640 10 monclient(hunting): my global_id is 30415 2024-06-03T09:07:43.865 INFO:teuthology.orchestra.run.smithi088.stderr:2024-06-03T09:07:43.857+0000 7fddc27fc640 1 -- 172.21.15.88:0/1349199537 --> v1:172.21.15.192:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7fddcc0a91a0 con 0x7fddcc0a93c0 2024-06-03T09:07:43.867 INFO:teuthology.orchestra.run.smithi088.stderr:2024-06-03T09:07:43.861+0000 7fddc27fc640 1 -- 172.21.15.88:0/1349199537 <== mon.2 v1:172.21.15.88:6790/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 764+0+0 (unknown 2188466870 0 0) 0x7fddcc0a98e0 con 0x7fddcc0717c0 2024-06-03T09:07:43.867 INFO:teuthology.orchestra.run.smithi088.stderr:2024-06-03T09:07:43.861+0000 7fddc27fc640 1 -- 172.21.15.88:0/1349199537 --> v1:172.21.15.88:6790/0 -- auth(proto 2 165 bytes epoch 0) v1 -- 0x7fddcc0aac80 con 0x7fddcc0717c0 2024-06-03T09:07:43.867 INFO:teuthology.orchestra.run.smithi088.stderr:2024-06-03T09:07:43.861+0000 7fddc27fc640 1 -- 172.21.15.88:0/1349199537 <== mon.2 v1:172.21.15.88:6790/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 2234374065 0 0) 0x7fddbc001b50 con 0x7fddcc0717c0 2024-06-03T09:07:43.867 INFO:teuthology.orchestra.run.smithi088.stderr:2024-06-03T09:07:43.861+0000 7fddc27fc640 10 monclient(hunting): handle_monmap mon_map magic: 0 v1 2024-06-03T09:07:43.867 INFO:teuthology.orchestra.run.smithi088.stderr:2024-06-03T09:07:43.861+0000 7fddc27fc640 10 monclient(hunting): got monmap 1 from mon.noname-c (according to old e1) 2024-06-03T09:07:43.867 INFO:teuthology.orchestra.run.smithi088.stderr:2024-06-03T09:07:43.861+0000 7fddc27fc640 10 monclient(hunting): dump: 2024-06-03T09:07:43.867 INFO:teuthology.orchestra.run.smithi088.stderr:epoch 1 2024-06-03T09:07:43.867 INFO:teuthology.orchestra.run.smithi088.stderr:fsid ec3c3fdd-3ac6-4d7d-9412-683130ed8b04 2024-06-03T09:07:43.868 INFO:teuthology.orchestra.run.smithi088.stderr:last_changed 2024-06-03T08:34:31.885125+0000 2024-06-03T09:07:43.868 INFO:teuthology.orchestra.run.smithi088.stderr:created 2024-06-03T08:34:31.885125+0000 2024-06-03T09:07:43.868 INFO:teuthology.orchestra.run.smithi088.stderr:min_mon_release 18 (reef) 2024-06-03T09:07:43.868 INFO:teuthology.orchestra.run.smithi088.stderr:election_strategy: 3 2024-06-03T09:07:43.868 INFO:teuthology.orchestra.run.smithi088.stderr:0: v1:172.21.15.88:6789/0 mon.a 2024-06-03T09:07:43.868 INFO:teuthology.orchestra.run.smithi088.stderr:1: v1:172.21.15.192:6789/0 mon.b 2024-06-03T09:07:43.868 INFO:teuthology.orchestra.run.smithi088.stderr:2: v1:172.21.15.88:6790/0 mon.c 2024-06-03T09:07:43.868 INFO:teuthology.orchestra.run.smithi088.stderr: 2024-06-03T09:07:43.868 INFO:teuthology.orchestra.run.smithi088.stderr:2024-06-03T09:07:43.861+0000 7fddc27fc640 10 monclient(hunting): _finish_auth 0 2024-06-03T09:07:43.868 INFO:teuthology.orchestra.run.smithi088.stderr:2024-06-03T09:07:43.861+0000 7fddc27fc640 1 -- 172.21.15.88:0/1349199537 <== mon.0 v1:172.21.15.88:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 764+0+0 (unknown 1499899278 0 0) 0x7fddb0004eb0 con 0x7fddcc06e050 2024-06-03T09:07:43.868 INFO:teuthology.orchestra.run.smithi088.stderr:2024-06-03T09:07:43.861+0000 7fddc27fc640 1 -- 172.21.15.88:0/1349199537 --> v1:172.21.15.88:6789/0 -- auth(proto 2 165 bytes epoch 0) v1 -- 0x7fddcc0a98e0 con 0x7fddcc06e050 2024-06-03T09:07:43.868 INFO:teuthology.orchestra.run.smithi088.stderr:2024-06-03T09:07:43.861+0000 7fddc27fc640 1 -- 172.21.15.88:0/1349199537 <== mon.0 v1:172.21.15.88:6789/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 2234374065 0 0) 0x7fddc8001910 con 0x7fddcc06e050 2024-06-03T09:07:43.868 INFO:teuthology.orchestra.run.smithi088.stderr:2024-06-03T09:07:43.862+0000 7fddc27fc640 10 monclient(hunting): handle_monmap mon_map magic: 0 v1 2024-06-03T09:07:43.868 INFO:teuthology.orchestra.run.smithi088.stderr:2024-06-03T09:07:43.862+0000 7fddc27fc640 10 monclient(hunting): got monmap 1 from mon.a (according to old e1) 2024-06-03T09:07:43.869 INFO:teuthology.orchestra.run.smithi088.stderr:2024-06-03T09:07:43.862+0000 7fddc27fc640 10 monclient(hunting): dump: 2024-06-03T09:07:43.869 INFO:teuthology.orchestra.run.smithi088.stderr:epoch 1 2024-06-03T09:07:43.869 INFO:teuthology.orchestra.run.smithi088.stderr:fsid ec3c3fdd-3ac6-4d7d-9412-683130ed8b04 2024-06-03T09:07:43.869 INFO:teuthology.orchestra.run.smithi088.stderr:last_changed 2024-06-03T08:34:31.885125+0000 2024-06-03T09:07:43.869 INFO:teuthology.orchestra.run.smithi088.stderr:created 2024-06-03T08:34:31.885125+0000 2024-06-03T09:07:43.869 INFO:teuthology.orchestra.run.smithi088.stderr:min_mon_release 18 (reef) 2024-06-03T09:07:43.869 INFO:teuthology.orchestra.run.smithi088.stderr:election_strategy: 3 2024-06-03T09:07:43.869 INFO:teuthology.orchestra.run.smithi088.stderr:0: v1:172.21.15.88:6789/0 mon.a 2024-06-03T09:07:43.869 INFO:teuthology.orchestra.run.smithi088.stderr:1: v1:172.21.15.192:6789/0 mon.b 2024-06-03T09:07:43.869 INFO:teuthology.orchestra.run.smithi088.stderr:2: v1:172.21.15.88:6790/0 mon.c 2024-06-03T09:07:43.869 INFO:teuthology.orchestra.run.smithi088.stderr: 2024-06-03T09:07:43.869 INFO:teuthology.orchestra.run.smithi088.stderr:2024-06-03T09:07:43.862+0000 7fddc27fc640 1 -- 172.21.15.88:0/1349199537 <== mon.1 v1:172.21.15.192:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 764+0+0 (unknown 1210034586 0 0) 0x7fddcc0a91a0 con 0x7fddcc0a93c0 2024-06-03T09:07:43.869 INFO:teuthology.orchestra.run.smithi088.stderr:2024-06-03T09:07:43.862+0000 7fddc27fc640 1 -- 172.21.15.88:0/1349199537 --> v1:172.21.15.192:6789/0 -- auth(proto 2 165 bytes epoch 0) v1 -- 0x7fddb0004eb0 con 0x7fddcc0a93c0 2024-06-03T09:07:43.869 INFO:teuthology.orchestra.run.smithi088.stderr:2024-06-03T09:07:43.862+0000 7fddc27fc640 1 -- 172.21.15.88:0/1349199537 <== mon.1 v1:172.21.15.192:6789/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 2234374065 0 0) 0x7fddc4001b50 con 0x7fddcc0a93c0 2024-06-03T09:07:43.870 INFO:teuthology.orchestra.run.smithi088.stderr:2024-06-03T09:07:43.862+0000 7fddc27fc640 10 monclient(hunting): handle_monmap mon_map magic: 0 v1 2024-06-03T09:07:43.870 INFO:teuthology.orchestra.run.smithi088.stderr:2024-06-03T09:07:43.862+0000 7fddc27fc640 10 monclient(hunting): got monmap 1 from mon.b (according to old e1) 2024-06-03T09:07:43.870 INFO:teuthology.orchestra.run.smithi088.stderr:2024-06-03T09:07:43.862+0000 7fddc27fc640 10 monclient(hunting): dump: 2024-06-03T09:07:43.870 INFO:teuthology.orchestra.run.smithi088.stderr:epoch 1 2024-06-03T09:07:43.870 INFO:teuthology.orchestra.run.smithi088.stderr:fsid ec3c3fdd-3ac6-4d7d-9412-683130ed8b04 2024-06-03T09:07:43.870 INFO:teuthology.orchestra.run.smithi088.stderr:last_changed 2024-06-03T08:34:31.885125+0000 2024-06-03T09:07:43.870 INFO:teuthology.orchestra.run.smithi088.stderr:created 2024-06-03T08:34:31.885125+0000 2024-06-03T09:07:43.870 INFO:teuthology.orchestra.run.smithi088.stderr:min_mon_release 18 (reef) 2024-06-03T09:07:43.870 INFO:teuthology.orchestra.run.smithi088.stderr:election_strategy: 3 2024-06-03T09:07:43.871 INFO:teuthology.orchestra.run.smithi088.stderr:0: v1:172.21.15.88:6789/0 mon.a 2024-06-03T09:07:43.871 INFO:teuthology.orchestra.run.smithi088.stderr:1: v1:172.21.15.192:6789/0 mon.b 2024-06-03T09:07:43.871 INFO:teuthology.orchestra.run.smithi088.stderr:2: v1:172.21.15.88:6790/0 mon.c 2024-06-03T09:07:43.871 INFO:teuthology.orchestra.run.smithi088.stderr: 2024-06-03T09:07:43.871 INFO:teuthology.orchestra.run.smithi088.stderr:2024-06-03T09:07:43.864+0000 7fddc27fc640 1 -- 172.21.15.88:0/1349199537 <== mon.0 v1:172.21.15.88:6789/0 4 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 580+0+0 (unknown 3378662017 0 0) 0x7fddcc0a98e0 con 0x7fddcc06e050 2024-06-03T09:07:43.871 INFO:teuthology.orchestra.run.smithi088.stderr:2024-06-03T09:07:43.864+0000 7fddc27fc640 1 -- 172.21.15.88:0/1349199537 >> v1:172.21.15.88:6790/0 conn(0x7fddcc0717c0 legacy=0x7fddcc13c570 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-06-03T09:07:43.871 INFO:teuthology.orchestra.run.smithi088.stderr:2024-06-03T09:07:43.864+0000 7fddc27fc640 1 -- 172.21.15.88:0/1349199537 >> v1:172.21.15.192:6789/0 conn(0x7fddcc0a93c0 legacy=0x7fddcc1384e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-06-03T09:07:43.871 INFO:teuthology.orchestra.run.smithi088.stderr:2024-06-03T09:07:43.864+0000 7fddc27fc640 10 monclient: _finish_hunting 0 2024-06-03T09:07:43.871 INFO:teuthology.orchestra.run.smithi088.stderr:2024-06-03T09:07:43.864+0000 7fddc27fc640 1 monclient: found mon.a 2024-06-03T09:07:43.871 INFO:teuthology.orchestra.run.smithi088.stderr:2024-06-03T09:07:43.864+0000 7fddc27fc640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-06-03T09:07:43.871 INFO:teuthology.orchestra.run.smithi088.stderr:2024-06-03T09:07:43.864+0000 7fddc27fc640 10 monclient: _send_mon_message to mon.a at v1:172.21.15.88:6789/0 2024-06-03T09:07:43.871 INFO:teuthology.orchestra.run.smithi088.stderr:2024-06-03T09:07:43.864+0000 7fddc27fc640 1 -- 172.21.15.88:0/1349199537 --> v1:172.21.15.88:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fddcc0a9d70 con 0x7fddcc06e050 2024-06-03T09:07:43.871 INFO:teuthology.orchestra.run.smithi088.stderr:2024-06-03T09:07:43.864+0000 7fddc27fc640 10 monclient: _finish_auth 0 2024-06-03T09:07:43.871 INFO:teuthology.orchestra.run.smithi088.stderr:2024-06-03T09:07:43.864+0000 7fddc27fc640 10 monclient: _check_auth_tickets 2024-06-03T09:07:43.872 INFO:teuthology.orchestra.run.smithi088.stderr:2024-06-03T09:07:43.864+0000 7fddc27fc640 20 monclient: _check_auth_rotating not needed by client.admin 2024-06-03T09:07:43.872 INFO:teuthology.orchestra.run.smithi088.stderr:2024-06-03T09:07:43.864+0000 7fddd3a70640 5 monclient: authenticate success, global_id 30840 2024-06-03T09:07:43.872 INFO:teuthology.orchestra.run.smithi088.stderr:2024-06-03T09:07:43.864+0000 7fddd3a70640 10 monclient: _renew_subs 2024-06-03T09:07:43.872 INFO:teuthology.orchestra.run.smithi088.stderr:2024-06-03T09:07:43.864+0000 7fddd3a70640 10 monclient: _send_mon_message to mon.a at v1:172.21.15.88:6789/0 2024-06-03T09:07:43.872 INFO:teuthology.orchestra.run.smithi088.stderr:2024-06-03T09:07:43.864+0000 7fddd3a70640 1 -- 172.21.15.88:0/1349199537 --> v1:172.21.15.88:6789/0 -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fddcc13e2b0 con 0x7fddcc06e050 2024-06-03T09:07:43.872 INFO:teuthology.orchestra.run.smithi088.stderr:2024-06-03T09:07:43.864+0000 7fddd3a70640 10 client.30840.objecter _maybe_request_map subscribing (onetime) to next osd map 2024-06-03T09:07:43.872 INFO:teuthology.orchestra.run.smithi088.stderr:2024-06-03T09:07:43.864+0000 7fddd3a70640 10 monclient: _renew_subs 2024-06-03T09:07:43.872 INFO:teuthology.orchestra.run.smithi088.stderr:2024-06-03T09:07:43.864+0000 7fddd3a70640 10 monclient: _send_mon_message to mon.a at v1:172.21.15.88:6789/0 2024-06-03T09:07:43.872 INFO:teuthology.orchestra.run.smithi088.stderr:2024-06-03T09:07:43.864+0000 7fddd3a70640 1 -- 172.21.15.88:0/1349199537 --> v1:172.21.15.88:6789/0 -- mon_subscribe({osdmap=0}) v3 -- 0x7fddcc13ce60 con 0x7fddcc06e050 2024-06-03T09:07:43.872 INFO:teuthology.orchestra.run.smithi088.stderr:2024-06-03T09:07:43.864+0000 7fddd3a70640 1 librados: init done 2024-06-03T09:07:43.872 INFO:teuthology.orchestra.run.smithi088.stderr:2024-06-03T09:07:43.865+0000 7fddd3a70640 10 start_mon_command cmd=[{"prefix": "get_command_descriptions"}] 2024-06-03T09:07:43.872 INFO:teuthology.orchestra.run.smithi088.stderr:2024-06-03T09:07:43.865+0000 7fddd3a70640 10 monclient: _send_command 1 [{"prefix": "get_command_descriptions"}] 2024-06-03T09:07:43.872 INFO:teuthology.orchestra.run.smithi088.stderr:2024-06-03T09:07:43.865+0000 7fddd3a70640 10 monclient: _send_mon_message to mon.a at v1:172.21.15.88:6789/0 2024-06-03T09:07:43.872 INFO:teuthology.orchestra.run.smithi088.stderr:2024-06-03T09:07:43.865+0000 7fddd3a70640 1 -- 172.21.15.88:0/1349199537 --> v1:172.21.15.88:6789/0 -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fddcc04ea60 con 0x7fddcc06e050 2024-06-03T09:07:44.111 INFO:teuthology.orchestra.run.smithi088.stderr:2024-06-03T09:07:44.110+0000 7fddc27fc640 1 -- 172.21.15.88:0/1349199537 <== mon.0 v1:172.21.15.88:6789/0 5 ==== config(0 keys) v1 ==== 4+0+0 (unknown 0 0 0) 0x7fddc8003610 con 0x7fddcc06e050 2024-06-03T09:07:44.111 INFO:teuthology.orchestra.run.smithi088.stderr:2024-06-03T09:07:44.111+0000 7fddc27fc640 10 monclient: handle_config config(0 keys) v1 2024-06-03T09:07:44.111 INFO:teuthology.orchestra.run.smithi088.stderr:2024-06-03T09:07:44.111+0000 7fddc27fc640 1 -- 172.21.15.88:0/1349199537 <== mon.0 v1:172.21.15.88:6789/0 6 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 2234374065 0 0) 0x7fddc8003920 con 0x7fddcc06e050 2024-06-03T09:07:44.111 INFO:teuthology.orchestra.run.smithi088.stderr:2024-06-03T09:07:44.111+0000 7fddc27fc640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-06-03T09:07:44.111 INFO:teuthology.orchestra.run.smithi088.stderr:2024-06-03T09:07:44.111+0000 7fddc27fc640 10 monclient: got monmap 1 from mon.a (according to old e1) 2024-06-03T09:07:44.111 INFO:teuthology.orchestra.run.smithi088.stderr:2024-06-03T09:07:44.111+0000 7fddc27fc640 10 monclient: dump: 2024-06-03T09:07:44.112 INFO:teuthology.orchestra.run.smithi088.stderr:epoch 1 2024-06-03T09:07:44.112 INFO:teuthology.orchestra.run.smithi088.stderr:fsid ec3c3fdd-3ac6-4d7d-9412-683130ed8b04 2024-06-03T09:07:44.112 INFO:teuthology.orchestra.run.smithi088.stderr:last_changed 2024-06-03T08:34:31.885125+0000 2024-06-03T09:07:44.112 INFO:teuthology.orchestra.run.smithi088.stderr:created 2024-06-03T08:34:31.885125+0000 2024-06-03T09:07:44.112 INFO:teuthology.orchestra.run.smithi088.stderr:min_mon_release 18 (reef) 2024-06-03T09:07:44.112 INFO:teuthology.orchestra.run.smithi088.stderr:election_strategy: 3 2024-06-03T09:07:44.113 INFO:teuthology.orchestra.run.smithi088.stderr:0: v1:172.21.15.88:6789/0 mon.a 2024-06-03T09:07:44.113 INFO:teuthology.orchestra.run.smithi088.stderr:1: v1:172.21.15.192:6789/0 mon.b 2024-06-03T09:07:44.113 INFO:teuthology.orchestra.run.smithi088.stderr:2: v1:172.21.15.88:6790/0 mon.c 2024-06-03T09:07:44.113 INFO:teuthology.orchestra.run.smithi088.stderr: 2024-06-03T09:07:44.113 INFO:teuthology.orchestra.run.smithi088.stderr:2024-06-03T09:07:44.111+0000 7fddc3fff640 4 set_mon_vals no callback set 2024-06-03T09:07:44.113 INFO:teuthology.orchestra.run.smithi088.stderr:2024-06-03T09:07:44.111+0000 7fddc27fc640 1 -- 172.21.15.88:0/1349199537 <== mon.0 v1:172.21.15.88:6789/0 7 ==== mgrmap(e 5) v1 ==== 89110+0+0 (unknown 873533973 0 0) 0x7fddc8003ac0 con 0x7fddcc06e050 2024-06-03T09:07:44.113 INFO:teuthology.orchestra.run.smithi088.stderr:2024-06-03T09:07:44.112+0000 7fddd100b640 10 monclient: get_auth_request con 0x7fddb006d420 auth_method 0 2024-06-03T09:07:44.114 INFO:teuthology.orchestra.run.smithi088.stderr:2024-06-03T09:07:44.112+0000 7fddc27fc640 10 client.30840.objecter ms_handle_connect 0x7fddb006d420 2024-06-03T09:07:44.116 INFO:teuthology.orchestra.run.smithi088.stderr:2024-06-03T09:07:44.116+0000 7fddc27fc640 1 -- 172.21.15.88:0/1349199537 <== mon.0 v1:172.21.15.88:6789/0 8 ==== osd_map(1014..1014 src has 735..1014) v4 ==== 5229+0+0 (unknown 332246716 0 0) 0x7fddc80540c0 con 0x7fddcc06e050 2024-06-03T09:07:44.116 INFO:teuthology.orchestra.run.smithi088.stderr:2024-06-03T09:07:44.116+0000 7fddc27fc640 10 client.30840.objecter ms_dispatch 0x7fddcc000b90 osd_map(1014..1014 src has 735..1014) v4 2024-06-03T09:07:44.116 INFO:teuthology.orchestra.run.smithi088.stderr:2024-06-03T09:07:44.116+0000 7fddc27fc640 3 client.30840.objecter handle_osd_map got epochs [1014,1014] > 0 2024-06-03T09:07:44.116 INFO:teuthology.orchestra.run.smithi088.stderr:2024-06-03T09:07:44.116+0000 7fddc27fc640 3 client.30840.objecter handle_osd_map decoding full epoch 1014 2024-06-03T09:07:44.116 INFO:teuthology.orchestra.run.smithi088.stderr:2024-06-03T09:07:44.116+0000 7fddc27fc640 20 client.30840.objecter dump_active .. 0 homeless 2024-06-03T09:07:44.116 INFO:teuthology.orchestra.run.smithi088.stderr:2024-06-03T09:07:44.116+0000 7fddc27fc640 1 -- 172.21.15.88:0/1349199537 <== mon.0 v1:172.21.15.88:6789/0 9 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+182498 (unknown 1092875540 0 3016760700) 0x7fddc804fc70 con 0x7fddcc06e050 2024-06-03T09:07:44.116 INFO:teuthology.orchestra.run.smithi088.stderr:2024-06-03T09:07:44.116+0000 7fddc27fc640 10 monclient: handle_mon_command_ack 1 [{"prefix": "get_command_descriptions"}] 2024-06-03T09:07:44.116 INFO:teuthology.orchestra.run.smithi088.stderr:2024-06-03T09:07:44.116+0000 7fddc27fc640 10 monclient: _finish_command 1 = system:0 2024-06-03T09:07:44.287 INFO:teuthology.orchestra.run.smithi088.stderr:2024-06-03T09:07:44.287+0000 7fddd3a70640 1 -- 172.21.15.88:0/1349199537 --> v1:172.21.15.192:6800/34229 -- mgr_command(tid 0: {"prefix": "pg dump", "target": ["mon-mgr", ""], "format": "json"}) v1 -- 0x7fddcc06eeb0 con 0x7fddb006d420 2024-06-03T09:07:44.292 INFO:teuthology.orchestra.run.smithi088.stderr:2024-06-03T09:07:44.292+0000 7fddc27fc640 1 -- 172.21.15.88:0/1349199537 <== mgr.4110 v1:172.21.15.192:6800/34229 1 ==== mgr_command_reply(tid 0: 0 dumped all) v1 ==== 18+0+152317 (unknown 2965378022 0 4002106950) 0x7fddbc007860 con 0x7fddb006d420 2024-06-03T09:07:44.294 INFO:teuthology.orchestra.run.smithi088.stdout: 2024-06-03T09:07:44.294 INFO:teuthology.orchestra.run.smithi088.stderr:dumped all 2024-06-03T09:07:44.296 INFO:teuthology.orchestra.run.smithi088.stderr:2024-06-03T09:07:44.294+0000 7fddd3a70640 10 librados: watch_flush enter 2024-06-03T09:07:44.296 INFO:teuthology.orchestra.run.smithi088.stderr:2024-06-03T09:07:44.294+0000 7fddd3a70640 10 librados: watch_flush exit 2024-06-03T09:07:44.296 INFO:teuthology.orchestra.run.smithi088.stderr:2024-06-03T09:07:44.294+0000 7fddd3a70640 20 client.30840.objecter shutdown clearing up homeless session... 2024-06-03T09:07:44.296 INFO:teuthology.orchestra.run.smithi088.stderr:2024-06-03T09:07:44.294+0000 7fddd3a70640 10 client.30840.objecter successfully canceled tick 2024-06-03T09:07:44.296 INFO:teuthology.orchestra.run.smithi088.stderr:2024-06-03T09:07:44.294+0000 7fddd3a70640 1 -- 172.21.15.88:0/1349199537 >> v1:172.21.15.192:6800/34229 conn(0x7fddb006d420 legacy=0x7fddb006f8e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-06-03T09:07:44.296 INFO:teuthology.orchestra.run.smithi088.stderr:2024-06-03T09:07:44.294+0000 7fddd3a70640 10 monclient: shutdown 2024-06-03T09:07:44.296 INFO:teuthology.orchestra.run.smithi088.stderr:2024-06-03T09:07:44.294+0000 7fddd3a70640 20 monclient: shutdown discarding 0 pending message(s) 2024-06-03T09:07:44.297 INFO:teuthology.orchestra.run.smithi088.stderr:2024-06-03T09:07:44.294+0000 7fddd3a70640 1 -- 172.21.15.88:0/1349199537 >> v1:172.21.15.88:6789/0 conn(0x7fddcc06e050 legacy=0x7fddcc0a8a90 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-06-03T09:07:44.297 INFO:teuthology.orchestra.run.smithi088.stderr:2024-06-03T09:07:44.294+0000 7fddd3a70640 1 -- 172.21.15.88:0/1349199537 shutdown_connections 2024-06-03T09:07:44.297 INFO:teuthology.orchestra.run.smithi088.stderr:2024-06-03T09:07:44.294+0000 7fddd3a70640 1 -- 172.21.15.88:0/1349199537 >> 172.21.15.88:0/1349199537 conn(0x7fddcc0b2f20 msgr2=0x7fddcc0b0ff0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-06-03T09:07:44.297 INFO:teuthology.orchestra.run.smithi088.stderr:2024-06-03T09:07:44.294+0000 7fddd3a70640 1 -- 172.21.15.88:0/1349199537 shutdown_connections 2024-06-03T09:07:44.297 INFO:teuthology.orchestra.run.smithi088.stderr:2024-06-03T09:07:44.294+0000 7fddd3a70640 1 -- 172.21.15.88:0/1349199537 wait complete. 2024-06-03T09:07:44.297 INFO:teuthology.orchestra.run.smithi088.stderr:2024-06-03T09:07:44.294+0000 7fddd3a70640 1 librados: shutdown 2024-06-03T09:07:44.305 INFO:teuthology.orchestra.run.smithi088.stdout:{"pg_ready":true,"pg_map":{"version":1961,"stamp":"2024-06-03T09:07:43.118141+0000","last_osdmap_epoch":0,"last_pg_scan":0,"pg_stats_sum":{"stat_sum":{"num_bytes":459285,"num_objects":3,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":3,"num_whiteouts":0,"num_read":102,"num_read_kb":88,"num_write":120,"num_write_kb":1379,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2,"num_bytes_recovered":397840,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":86,"ondisk_log_size":86,"up":82,"acting":82,"num_store_stats":0},"osd_stats_sum":{"up_from":0,"seq":0,"num_pgs":82,"num_osds":8,"num_per_pool_osds":8,"num_per_pool_omap_osds":8,"kb":754974720,"kb_used":495000,"kb_used_data":46808,"kb_used_omap":12,"kb_used_meta":448115,"kb_avail":754479720,"statfs":{"total":773094113280,"available":772587233280,"internally_reserved":0,"allocated":47931392,"data_stored":22900274,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":12744,"internal_metadata":458870328},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[]},"pg_stats_delta":{"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":0,"acting":0,"num_store_stats":0,"stamp_delta":"12.003471"},"pg_stats":[{"pgid":"17.d","version":"0'0","reported_seq":958,"reported_epoch":1014,"state":"active+clean","last_fresh":"2024-06-03T09:07:15.243547+0000","last_change":"2024-06-03T09:07:15.243547+0000","last_active":"2024-06-03T09:07:15.243547+0000","last_peered":"2024-06-03T09:07:15.243547+0000","last_clean":"2024-06-03T09:07:15.243547+0000","last_became_active":"2024-06-03T08:38:19.253559+0000","last_became_peered":"2024-06-03T08:38:19.253559+0000","last_unstale":"2024-06-03T09:07:15.243547+0000","last_undegraded":"2024-06-03T09:07:15.243547+0000","last_fullsized":"2024-06-03T09:07:15.243547+0000","mapping_epoch":28,"log_start":"0'0","ondisk_log_start":"0'0","created":28,"last_epoch_clean":45,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-06-03T09:07:15.239490+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-06-03T09:07:15.239490+0000","last_clean_scrub_stamp":"2024-06-03T09:07:15.239490+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-06-04T09:27:58.788791+0000","scrub_duration":0.060706005,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1],"acting":[0,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"17.c","version":"0'0","reported_seq":978,"reported_epoch":1014,"state":"active+clean","last_fresh":"2024-06-03T09:07:27.609487+0000","last_change":"2024-06-03T09:07:27.611934+0000","last_active":"2024-06-03T09:07:27.609487+0000","last_peered":"2024-06-03T09:07:27.609487+0000","last_clean":"2024-06-03T09:07:27.609487+0000","last_became_active":"2024-06-03T09:06:46.312637+0000","last_became_peered":"2024-06-03T09:06:46.312637+0000","last_unstale":"2024-06-03T09:07:27.609487+0000","last_undegraded":"2024-06-03T09:07:27.609487+0000","last_fullsized":"2024-06-03T09:07:27.609487+0000","mapping_epoch":1013,"log_start":"0'0","ondisk_log_start":"0'0","created":28,"last_epoch_clean":1014,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-06-03T09:07:27.608376+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-06-03T09:07:27.608376+0000","last_clean_scrub_stamp":"2024-06-03T09:07:27.608376+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-06-04T11:26:54.022023+0000","scrub_duration":0.044138264000000003,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,6],"acting":[3,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"17.f","version":"0'0","reported_seq":855,"reported_epoch":1014,"state":"active+clean","last_fresh":"2024-06-03T09:07:09.949918+0000","last_change":"2024-06-03T09:07:09.955521+0000","last_active":"2024-06-03T09:07:09.949918+0000","last_peered":"2024-06-03T09:07:09.949918+0000","last_clean":"2024-06-03T09:07:09.949918+0000","last_became_active":"2024-06-03T09:06:46.301958+0000","last_became_peered":"2024-06-03T09:06:46.301958+0000","last_unstale":"2024-06-03T09:07:09.949918+0000","last_undegraded":"2024-06-03T09:07:09.949918+0000","last_fullsized":"2024-06-03T09:07:09.949918+0000","mapping_epoch":1013,"log_start":"0'0","ondisk_log_start":"0'0","created":28,"last_epoch_clean":1014,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-06-03T09:07:09.948711+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-06-03T09:07:09.948711+0000","last_clean_scrub_stamp":"2024-06-03T09:07:09.948711+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-06-04T19:32:16.465187+0000","scrub_duration":0.073847419999999997,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,4],"acting":[6,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"17.e","version":"0'0","reported_seq":855,"reported_epoch":1014,"state":"active+clean","last_fresh":"2024-06-03T09:07:07.901760+0000","last_change":"2024-06-03T09:07:07.903376+0000","last_active":"2024-06-03T09:07:07.901760+0000","last_peered":"2024-06-03T09:07:07.901760+0000","last_clean":"2024-06-03T09:07:07.901760+0000","last_became_active":"2024-06-03T09:06:46.273941+0000","last_became_peered":"2024-06-03T09:06:46.273941+0000","last_unstale":"2024-06-03T09:07:07.901760+0000","last_undegraded":"2024-06-03T09:07:07.901760+0000","last_fullsized":"2024-06-03T09:07:07.901760+0000","mapping_epoch":1013,"log_start":"0'0","ondisk_log_start":"0'0","created":28,"last_epoch_clean":1014,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-06-03T09:07:07.901032+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-06-03T09:07:07.901032+0000","last_clean_scrub_stamp":"2024-06-03T09:07:07.901032+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-06-04T16:06:06.004585+0000","scrub_duration":0.047268636000000003,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,5],"acting":[6,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"17.9","version":"0'0","reported_seq":961,"reported_epoch":1014,"state":"active+clean","last_fresh":"2024-06-03T09:07:08.477260+0000","last_change":"2024-06-03T09:07:08.478860+0000","last_active":"2024-06-03T09:07:08.477260+0000","last_peered":"2024-06-03T09:07:08.477260+0000","last_clean":"2024-06-03T09:07:08.477260+0000","last_became_active":"2024-06-03T08:38:04.763428+0000","last_became_peered":"2024-06-03T08:38:04.763428+0000","last_unstale":"2024-06-03T09:07:08.477260+0000","last_undegraded":"2024-06-03T09:07:08.477260+0000","last_fullsized":"2024-06-03T09:07:08.477260+0000","mapping_epoch":28,"log_start":"0'0","ondisk_log_start":"0'0","created":28,"last_epoch_clean":33,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-06-03T09:07:08.476536+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-06-03T09:07:08.476536+0000","last_clean_scrub_stamp":"2024-06-03T09:07:08.476536+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-06-04T16:39:49.232748+0000","scrub_duration":0.044858958999999997,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,4],"acting":[7,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"17.8","version":"0'0","reported_seq":984,"reported_epoch":1014,"state":"active+clean","last_fresh":"2024-06-03T09:06:59.146085+0000","last_change":"2024-06-03T09:06:59.150900+0000","last_active":"2024-06-03T09:06:59.146085+0000","last_peered":"2024-06-03T09:06:59.146085+0000","last_clean":"2024-06-03T09:06:59.146085+0000","last_became_active":"2024-06-03T09:06:46.344756+0000","last_became_peered":"2024-06-03T09:06:46.344756+0000","last_unstale":"2024-06-03T09:06:59.146085+0000","last_undegraded":"2024-06-03T09:06:59.146085+0000","last_fullsized":"2024-06-03T09:06:59.146085+0000","mapping_epoch":1013,"log_start":"0'0","ondisk_log_start":"0'0","created":28,"last_epoch_clean":1014,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-06-03T09:06:59.144935+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-06-03T09:06:59.144935+0000","last_clean_scrub_stamp":"2024-06-03T09:06:59.144935+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-06-04T15:52:37.265440+0000","scrub_duration":0.031008068,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,6],"acting":[0,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"17.b","version":"0'0","reported_seq":855,"reported_epoch":1014,"state":"active+clean","last_fresh":"2024-06-03T09:07:06.907843+0000","last_change":"2024-06-03T09:07:06.909464+0000","last_active":"2024-06-03T09:07:06.907843+0000","last_peered":"2024-06-03T09:07:06.907843+0000","last_clean":"2024-06-03T09:07:06.907843+0000","last_became_active":"2024-06-03T09:06:46.345458+0000","last_became_peered":"2024-06-03T09:06:46.345458+0000","last_unstale":"2024-06-03T09:07:06.907843+0000","last_undegraded":"2024-06-03T09:07:06.907843+0000","last_fullsized":"2024-06-03T09:07:06.907843+0000","mapping_epoch":1013,"log_start":"0'0","ondisk_log_start":"0'0","created":28,"last_epoch_clean":1014,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-06-03T09:07:06.907110+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-06-03T09:07:06.907110+0000","last_clean_scrub_stamp":"2024-06-03T09:07:06.907110+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-06-04T18:08:34.356218+0000","scrub_duration":0.080445640999999998,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,2],"acting":[6,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"17.a","version":"0'0","reported_seq":941,"reported_epoch":1014,"state":"active+clean","last_fresh":"2024-06-03T09:07:02.857089+0000","last_change":"2024-06-03T09:07:02.858770+0000","last_active":"2024-06-03T09:07:02.857089+0000","last_peered":"2024-06-03T09:07:02.857089+0000","last_clean":"2024-06-03T09:07:02.857089+0000","last_became_active":"2024-06-03T08:38:05.809257+0000","last_became_peered":"2024-06-03T08:38:05.809257+0000","last_unstale":"2024-06-03T09:07:02.857089+0000","last_undegraded":"2024-06-03T09:07:02.857089+0000","last_fullsized":"2024-06-03T09:07:02.857089+0000","mapping_epoch":28,"log_start":"0'0","ondisk_log_start":"0'0","created":28,"last_epoch_clean":33,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-06-03T09:07:02.856327+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-06-03T09:07:02.856327+0000","last_clean_scrub_stamp":"2024-06-03T09:07:02.856327+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-06-04T11:53:39.468676+0000","scrub_duration":0.048148466000000001,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2],"acting":[1,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"17.5","version":"0'0","reported_seq":959,"reported_epoch":1014,"state":"active+clean","last_fresh":"2024-06-03T09:07:14.253701+0000","last_change":"2024-06-03T09:07:14.253701+0000","last_active":"2024-06-03T09:07:14.253701+0000","last_peered":"2024-06-03T09:07:14.253701+0000","last_clean":"2024-06-03T09:07:14.253701+0000","last_became_active":"2024-06-03T08:38:23.458039+0000","last_became_peered":"2024-06-03T08:38:23.458039+0000","last_unstale":"2024-06-03T09:07:14.253701+0000","last_undegraded":"2024-06-03T09:07:14.253701+0000","last_fullsized":"2024-06-03T09:07:14.253701+0000","mapping_epoch":28,"log_start":"0'0","ondisk_log_start":"0'0","created":28,"last_epoch_clean":49,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-06-03T09:07:14.251098+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-06-03T09:07:14.251098+0000","last_clean_scrub_stamp":"2024-06-03T09:07:14.251098+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-06-04T16:53:12.820223+0000","scrub_duration":0.060259851000000003,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,4],"acting":[0,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"17.4","version":"0'0","reported_seq":855,"reported_epoch":1014,"state":"active+clean","last_fresh":"2024-06-03T09:07:03.820792+0000","last_change":"2024-06-03T09:07:03.822380+0000","last_active":"2024-06-03T09:07:03.820792+0000","last_peered":"2024-06-03T09:07:03.820792+0000","last_clean":"2024-06-03T09:07:03.820792+0000","last_became_active":"2024-06-03T09:06:46.447534+0000","last_became_peered":"2024-06-03T09:06:46.447534+0000","last_unstale":"2024-06-03T09:07:03.820792+0000","last_undegraded":"2024-06-03T09:07:03.820792+0000","last_fullsized":"2024-06-03T09:07:03.820792+0000","mapping_epoch":1013,"log_start":"0'0","ondisk_log_start":"0'0","created":28,"last_epoch_clean":1014,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-06-03T09:07:03.820058+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-06-03T09:07:03.820058+0000","last_clean_scrub_stamp":"2024-06-03T09:07:03.820058+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-06-04T13:11:32.675403+0000","scrub_duration":0.027398738999999998,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,7],"acting":[6,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"17.7","version":"0'0","reported_seq":946,"reported_epoch":1014,"state":"active+clean","last_fresh":"2024-06-03T09:07:11.552030+0000","last_change":"2024-06-03T09:07:11.553608+0000","last_active":"2024-06-03T09:07:11.552030+0000","last_peered":"2024-06-03T09:07:11.552030+0000","last_clean":"2024-06-03T09:07:11.552030+0000","last_became_active":"2024-06-03T08:38:04.818246+0000","last_became_peered":"2024-06-03T08:38:04.818246+0000","last_unstale":"2024-06-03T09:07:11.552030+0000","last_undegraded":"2024-06-03T09:07:11.552030+0000","last_fullsized":"2024-06-03T09:07:11.552030+0000","mapping_epoch":28,"log_start":"0'0","ondisk_log_start":"0'0","created":28,"last_epoch_clean":34,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-06-03T09:07:11.551324+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-06-03T09:07:11.551324+0000","last_clean_scrub_stamp":"2024-06-03T09:07:11.551324+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-06-04T12:33:33.185479+0000","scrub_duration":0.064916682000000003,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,2],"acting":[5,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"17.6","version":"0'0","reported_seq":983,"reported_epoch":1014,"state":"active+clean","last_fresh":"2024-06-03T09:07:20.214682+0000","last_change":"2024-06-03T09:07:20.216289+0000","last_active":"2024-06-03T09:07:20.214682+0000","last_peered":"2024-06-03T09:07:20.214682+0000","last_clean":"2024-06-03T09:07:20.214682+0000","last_became_active":"2024-06-03T09:06:46.357698+0000","last_became_peered":"2024-06-03T09:06:46.357698+0000","last_unstale":"2024-06-03T09:07:20.214682+0000","last_undegraded":"2024-06-03T09:07:20.214682+0000","last_fullsized":"2024-06-03T09:07:20.214682+0000","mapping_epoch":1013,"log_start":"0'0","ondisk_log_start":"0'0","created":28,"last_epoch_clean":1014,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-06-03T09:07:20.213929+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-06-03T09:07:20.213929+0000","last_clean_scrub_stamp":"2024-06-03T09:07:20.213929+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-06-04T09:34:35.913266+0000","scrub_duration":0.038979437999999998,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,6],"acting":[0,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"2.7","version":"0'0","reported_seq":957,"reported_epoch":1014,"state":"active+clean","last_fresh":"2024-06-03T09:07:12.435349+0000","last_change":"2024-06-03T09:07:12.439483+0000","last_active":"2024-06-03T09:07:12.435349+0000","last_peered":"2024-06-03T09:07:12.435349+0000","last_clean":"2024-06-03T09:07:12.435349+0000","last_became_active":"2024-06-03T08:36:30.380302+0000","last_became_peered":"2024-06-03T08:36:30.380302+0000","last_unstale":"2024-06-03T09:07:12.435349+0000","last_undegraded":"2024-06-03T09:07:12.435349+0000","last_fullsized":"2024-06-03T09:07:12.435349+0000","mapping_epoch":23,"log_start":"0'0","ondisk_log_start":"0'0","created":23,"last_epoch_clean":24,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-06-03T09:07:12.434624+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-06-03T09:07:12.434624+0000","last_clean_scrub_stamp":"2024-06-03T09:07:12.434624+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-06-04T09:11:59.759962+0000","scrub_duration":0.029406762999999999,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0],"acting":[3,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"17.14","version":"0'0","reported_seq":943,"reported_epoch":1014,"state":"active+clean","last_fresh":"2024-06-03T09:07:12.552995+0000","last_change":"2024-06-03T09:07:12.552995+0000","last_active":"2024-06-03T09:07:12.552995+0000","last_peered":"2024-06-03T09:07:12.552995+0000","last_clean":"2024-06-03T09:07:12.552995+0000","last_became_active":"2024-06-03T08:38:04.541781+0000","last_became_peered":"2024-06-03T08:38:04.541781+0000","last_unstale":"2024-06-03T09:07:12.552995+0000","last_undegraded":"2024-06-03T09:07:12.552995+0000","last_fullsized":"2024-06-03T09:07:12.552995+0000","mapping_epoch":28,"log_start":"0'0","ondisk_log_start":"0'0","created":28,"last_epoch_clean":34,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-06-03T09:07:12.550638+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-06-03T09:07:12.550638+0000","last_clean_scrub_stamp":"2024-06-03T09:07:12.550638+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-06-04T15:35:25.145190+0000","scrub_duration":0.041798374999999999,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,2],"acting":[5,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"2.6","version":"0'0","reported_seq":965,"reported_epoch":1014,"state":"active+clean","last_fresh":"2024-06-03T09:07:01.091185+0000","last_change":"2024-06-03T09:07:01.092831+0000","last_active":"2024-06-03T09:07:01.091185+0000","last_peered":"2024-06-03T09:07:01.091185+0000","last_clean":"2024-06-03T09:07:01.091185+0000","last_became_active":"2024-06-03T08:36:30.404665+0000","last_became_peered":"2024-06-03T08:36:30.404665+0000","last_unstale":"2024-06-03T09:07:01.091185+0000","last_undegraded":"2024-06-03T09:07:01.091185+0000","last_fullsized":"2024-06-03T09:07:01.091185+0000","mapping_epoch":23,"log_start":"0'0","ondisk_log_start":"0'0","created":23,"last_epoch_clean":24,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-06-03T09:07:01.090441+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-06-03T09:07:01.090441+0000","last_clean_scrub_stamp":"2024-06-03T09:07:01.090441+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-06-04T12:37:48.579420+0000","scrub_duration":0.038448474000000003,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,0],"acting":[4,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"17.15","version":"0'0","reported_seq":951,"reported_epoch":1014,"state":"active+clean","last_fresh":"2024-06-03T09:07:28.641419+0000","last_change":"2024-06-03T09:07:28.643003+0000","last_active":"2024-06-03T09:07:28.641419+0000","last_peered":"2024-06-03T09:07:28.641419+0000","last_clean":"2024-06-03T09:07:28.641419+0000","last_became_active":"2024-06-03T08:38:20.979523+0000","last_became_peered":"2024-06-03T08:38:20.979523+0000","last_unstale":"2024-06-03T09:07:28.641419+0000","last_undegraded":"2024-06-03T09:07:28.641419+0000","last_fullsized":"2024-06-03T09:07:28.641419+0000","mapping_epoch":28,"log_start":"0'0","ondisk_log_start":"0'0","created":28,"last_epoch_clean":47,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-06-03T09:07:28.640607+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-06-03T09:07:28.640607+0000","last_clean_scrub_stamp":"2024-06-03T09:07:28.640607+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-06-04T12:08:46.674890+0000","scrub_duration":0.070846264000000006,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,1],"acting":[3,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"2.5","version":"0'0","reported_seq":958,"reported_epoch":1014,"state":"active+clean","last_fresh":"2024-06-03T09:07:17.567853+0000","last_change":"2024-06-03T09:07:17.569584+0000","last_active":"2024-06-03T09:07:17.567853+0000","last_peered":"2024-06-03T09:07:17.567853+0000","last_clean":"2024-06-03T09:07:17.567853+0000","last_became_active":"2024-06-03T08:36:30.019040+0000","last_became_peered":"2024-06-03T08:36:30.019040+0000","last_unstale":"2024-06-03T09:07:17.567853+0000","last_undegraded":"2024-06-03T09:07:17.567853+0000","last_fullsized":"2024-06-03T09:07:17.567853+0000","mapping_epoch":23,"log_start":"0'0","ondisk_log_start":"0'0","created":23,"last_epoch_clean":24,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-06-03T09:07:17.567142+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-06-03T09:07:17.567142+0000","last_clean_scrub_stamp":"2024-06-03T09:07:17.567142+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-06-04T16:31:04.548317+0000","scrub_duration":0.034251373000000002,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,4],"acting":[3,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"17.16","version":"0'0","reported_seq":997,"reported_epoch":1014,"state":"active+clean","last_fresh":"2024-06-03T09:06:59.969669+0000","last_change":"2024-06-03T09:06:59.969669+0000","last_active":"2024-06-03T09:06:59.969669+0000","last_peered":"2024-06-03T09:06:59.969669+0000","last_clean":"2024-06-03T09:06:59.969669+0000","last_became_active":"2024-06-03T09:06:46.379518+0000","last_became_peered":"2024-06-03T09:06:46.379518+0000","last_unstale":"2024-06-03T09:06:59.969669+0000","last_undegraded":"2024-06-03T09:06:59.969669+0000","last_fullsized":"2024-06-03T09:06:59.969669+0000","mapping_epoch":1013,"log_start":"0'0","ondisk_log_start":"0'0","created":28,"last_epoch_clean":1014,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-06-03T09:06:59.966233+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-06-03T09:06:59.966233+0000","last_clean_scrub_stamp":"2024-06-03T09:06:59.966233+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-06-04T18:45:08.505938+0000","scrub_duration":0.037443454000000001,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,6],"acting":[2,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"2.4","version":"0'0","reported_seq":976,"reported_epoch":1014,"state":"active+clean","last_fresh":"2024-06-03T09:07:03.492481+0000","last_change":"2024-06-03T09:07:03.494168+0000","last_active":"2024-06-03T09:07:03.492481+0000","last_peered":"2024-06-03T09:07:03.492481+0000","last_clean":"2024-06-03T09:07:03.492481+0000","last_became_active":"2024-06-03T09:06:46.371430+0000","last_became_peered":"2024-06-03T09:06:46.371430+0000","last_unstale":"2024-06-03T09:07:03.492481+0000","last_undegraded":"2024-06-03T09:07:03.492481+0000","last_fullsized":"2024-06-03T09:07:03.492481+0000","mapping_epoch":1013,"log_start":"0'0","ondisk_log_start":"0'0","created":23,"last_epoch_clean":1014,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-06-03T09:07:03.491693+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-06-03T09:07:03.491693+0000","last_clean_scrub_stamp":"2024-06-03T09:07:03.491693+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-06-04T13:54:17.148134+0000","scrub_duration":0.038561466000000003,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,6],"acting":[5,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"17.17","version":"0'0","reported_seq":971,"reported_epoch":1014,"state":"active+clean","last_fresh":"2024-06-03T09:07:02.032651+0000","last_change":"2024-06-03T09:07:02.032651+0000","last_active":"2024-06-03T09:07:02.032651+0000","last_peered":"2024-06-03T09:07:02.032651+0000","last_clean":"2024-06-03T09:07:02.032651+0000","last_became_active":"2024-06-03T08:38:38.467916+0000","last_became_peered":"2024-06-03T08:38:38.467916+0000","last_unstale":"2024-06-03T09:07:02.032651+0000","last_undegraded":"2024-06-03T09:07:02.032651+0000","last_fullsized":"2024-06-03T09:07:02.032651+0000","mapping_epoch":28,"log_start":"0'0","ondisk_log_start":"0'0","created":28,"last_epoch_clean":61,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-06-03T09:07:02.030144+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-06-03T09:07:02.030144+0000","last_clean_scrub_stamp":"2024-06-03T09:07:02.030144+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-06-04T11:23:00.182920+0000","scrub_duration":0.081032390999999995,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,7],"acting":[2,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"2.2","version":"0'0","reported_seq":988,"reported_epoch":1014,"state":"active+clean","last_fresh":"2024-06-03T09:07:04.377466+0000","last_change":"2024-06-03T09:07:04.377466+0000","last_active":"2024-06-03T09:07:04.377466+0000","last_peered":"2024-06-03T09:07:04.377466+0000","last_clean":"2024-06-03T09:07:04.377466+0000","last_became_active":"2024-06-03T09:06:46.284836+0000","last_became_peered":"2024-06-03T09:06:46.284836+0000","last_unstale":"2024-06-03T09:07:04.377466+0000","last_undegraded":"2024-06-03T09:07:04.377466+0000","last_fullsized":"2024-06-03T09:07:04.377466+0000","mapping_epoch":1013,"log_start":"0'0","ondisk_log_start":"0'0","created":23,"last_epoch_clean":1014,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-06-03T09:07:04.375206+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-06-03T09:07:04.375206+0000","last_clean_scrub_stamp":"2024-06-03T09:07:04.375206+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-06-04T12:17:58.541830+0000","scrub_duration":0.032522373,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,6],"acting":[3,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"17.11","version":"0'0","reported_seq":959,"reported_epoch":1014,"state":"active+clean","last_fresh":"2024-06-03T09:07:03.087030+0000","last_change":"2024-06-03T09:07:03.088717+0000","last_active":"2024-06-03T09:07:03.087030+0000","last_peered":"2024-06-03T09:07:03.087030+0000","last_clean":"2024-06-03T09:07:03.087030+0000","last_became_active":"2024-06-03T08:38:06.490204+0000","last_became_peered":"2024-06-03T08:38:06.490204+0000","last_unstale":"2024-06-03T09:07:03.087030+0000","last_undegraded":"2024-06-03T09:07:03.087030+0000","last_fullsized":"2024-06-03T09:07:03.087030+0000","mapping_epoch":28,"log_start":"0'0","ondisk_log_start":"0'0","created":28,"last_epoch_clean":32,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-06-03T09:07:03.086295+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-06-03T09:07:03.086295+0000","last_clean_scrub_stamp":"2024-06-03T09:07:03.086295+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-06-04T09:36:44.758386+0000","scrub_duration":0.038154848999999998,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,2],"acting":[4,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"2.1","version":"0'0","reported_seq":967,"reported_epoch":1014,"state":"active+clean","last_fresh":"2024-06-03T09:07:03.630924+0000","last_change":"2024-06-03T09:07:03.642560+0000","last_active":"2024-06-03T09:07:03.630924+0000","last_peered":"2024-06-03T09:07:03.630924+0000","last_clean":"2024-06-03T09:07:03.630924+0000","last_became_active":"2024-06-03T08:36:30.855073+0000","last_became_peered":"2024-06-03T08:36:30.855073+0000","last_unstale":"2024-06-03T09:07:03.630924+0000","last_undegraded":"2024-06-03T09:07:03.630924+0000","last_fullsized":"2024-06-03T09:07:03.630924+0000","mapping_epoch":23,"log_start":"0'0","ondisk_log_start":"0'0","created":23,"last_epoch_clean":25,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-06-03T09:07:03.628701+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-06-03T09:07:03.628701+0000","last_clean_scrub_stamp":"2024-06-03T09:07:03.628701+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-06-04T11:26:15.349709+0000","scrub_duration":0.19363576900000001,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,5],"acting":[7,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"17.12","version":"0'0","reported_seq":960,"reported_epoch":1014,"state":"active+clean","last_fresh":"2024-06-03T09:07:07.469298+0000","last_change":"2024-06-03T09:07:07.470940+0000","last_active":"2024-06-03T09:07:07.469298+0000","last_peered":"2024-06-03T09:07:07.469298+0000","last_clean":"2024-06-03T09:07:07.469298+0000","last_became_active":"2024-06-03T08:38:01.745027+0000","last_became_peered":"2024-06-03T08:38:01.745027+0000","last_unstale":"2024-06-03T09:07:07.469298+0000","last_undegraded":"2024-06-03T09:07:07.469298+0000","last_fullsized":"2024-06-03T09:07:07.469298+0000","mapping_epoch":28,"log_start":"0'0","ondisk_log_start":"0'0","created":28,"last_epoch_clean":32,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-06-03T09:07:07.468586+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-06-03T09:07:07.468586+0000","last_clean_scrub_stamp":"2024-06-03T09:07:07.468586+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-06-04T19:15:15.696921+0000","scrub_duration":0.03822209,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,3],"acting":[7,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"2.0","version":"0'0","reported_seq":960,"reported_epoch":1014,"state":"active+clean","last_fresh":"2024-06-03T09:07:00.260891+0000","last_change":"2024-06-03T09:07:00.262894+0000","last_active":"2024-06-03T09:07:00.260891+0000","last_peered":"2024-06-03T09:07:00.260891+0000","last_clean":"2024-06-03T09:07:00.260891+0000","last_became_active":"2024-06-03T08:36:29.957875+0000","last_became_peered":"2024-06-03T08:36:29.957875+0000","last_unstale":"2024-06-03T09:07:00.260891+0000","last_undegraded":"2024-06-03T09:07:00.260891+0000","last_fullsized":"2024-06-03T09:07:00.260891+0000","mapping_epoch":23,"log_start":"0'0","ondisk_log_start":"0'0","created":23,"last_epoch_clean":24,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-06-03T09:07:00.260150+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-06-03T09:07:00.260150+0000","last_clean_scrub_stamp":"2024-06-03T09:07:00.260150+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-06-04T12:21:00.168796+0000","scrub_duration":0.031538745999999999,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,5],"acting":[3,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"17.13","version":"0'0","reported_seq":958,"reported_epoch":1014,"state":"active+clean","last_fresh":"2024-06-03T09:07:05.211420+0000","last_change":"2024-06-03T09:07:05.213047+0000","last_active":"2024-06-03T09:07:05.211420+0000","last_peered":"2024-06-03T09:07:05.211420+0000","last_clean":"2024-06-03T09:07:05.211420+0000","last_became_active":"2024-06-03T08:38:26.833214+0000","last_became_peered":"2024-06-03T08:38:26.833214+0000","last_unstale":"2024-06-03T09:07:05.211420+0000","last_undegraded":"2024-06-03T09:07:05.211420+0000","last_fullsized":"2024-06-03T09:07:05.211420+0000","mapping_epoch":28,"log_start":"0'0","ondisk_log_start":"0'0","created":28,"last_epoch_clean":52,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-06-03T09:07:05.210654+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-06-03T09:07:05.210654+0000","last_clean_scrub_stamp":"2024-06-03T09:07:05.210654+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-06-04T12:08:44.116700+0000","scrub_duration":0.087386555000000005,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,0],"acting":[4,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"2.3","version":"0'0","reported_seq":959,"reported_epoch":1014,"state":"active+clean","last_fresh":"2024-06-03T09:07:11.388404+0000","last_change":"2024-06-03T09:07:11.394811+0000","last_active":"2024-06-03T09:07:11.388404+0000","last_peered":"2024-06-03T09:07:11.388404+0000","last_clean":"2024-06-03T09:07:11.388404+0000","last_became_active":"2024-06-03T08:36:30.343560+0000","last_became_peered":"2024-06-03T08:36:30.343560+0000","last_unstale":"2024-06-03T09:07:11.388404+0000","last_undegraded":"2024-06-03T09:07:11.388404+0000","last_fullsized":"2024-06-03T09:07:11.388404+0000","mapping_epoch":23,"log_start":"0'0","ondisk_log_start":"0'0","created":23,"last_epoch_clean":24,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-06-03T09:07:11.387606+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-06-03T09:07:11.387606+0000","last_clean_scrub_stamp":"2024-06-03T09:07:11.387606+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-06-04T14:12:02.854001+0000","scrub_duration":0.033186076000000002,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,2],"acting":[3,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.0","version":"1014'78","reported_seq":1070,"reported_epoch":1014,"state":"active+clean","last_fresh":"2024-06-03T09:06:57.887708+0000","last_change":"2024-06-03T09:06:57.889997+0000","last_active":"2024-06-03T09:06:57.887708+0000","last_peered":"2024-06-03T09:06:57.887708+0000","last_clean":"2024-06-03T09:06:57.887708+0000","last_became_active":"2024-06-03T08:36:26.128378+0000","last_became_peered":"2024-06-03T08:36:26.128378+0000","last_unstale":"2024-06-03T09:06:57.887708+0000","last_undegraded":"2024-06-03T09:06:57.887708+0000","last_fullsized":"2024-06-03T09:06:57.887708+0000","mapping_epoch":20,"log_start":"0'0","ondisk_log_start":"0'0","created":16,"last_epoch_clean":22,"parent":"0.0","parent_split_bits":0,"last_scrub":"1014'78","last_scrub_stamp":"2024-06-03T09:06:57.886322+0000","last_deep_scrub":"1014'78","last_deep_scrub_stamp":"2024-06-03T09:06:57.886322+0000","last_clean_scrub_stamp":"2024-06-03T09:06:57.886322+0000","objects_scrubbed":2,"log_size":78,"log_dups_size":0,"ondisk_log_size":78,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-06-04T19:15:53.804903+0000","scrub_duration":0.098050433000000006,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":459280,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":96,"num_read_kb":82,"num_write":113,"num_write_kb":1372,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2,"num_bytes_recovered":397840,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,5],"acting":[1,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"17.10","version":"0'0","reported_seq":959,"reported_epoch":1014,"state":"active+clean","last_fresh":"2024-06-03T09:07:02.104119+0000","last_change":"2024-06-03T09:07:02.105812+0000","last_active":"2024-06-03T09:07:02.104119+0000","last_peered":"2024-06-03T09:07:02.104119+0000","last_clean":"2024-06-03T09:07:02.104119+0000","last_became_active":"2024-06-03T08:38:15.337723+0000","last_became_peered":"2024-06-03T08:38:15.337723+0000","last_unstale":"2024-06-03T09:07:02.104119+0000","last_undegraded":"2024-06-03T09:07:02.104119+0000","last_fullsized":"2024-06-03T09:07:02.104119+0000","mapping_epoch":28,"log_start":"0'0","ondisk_log_start":"0'0","created":28,"last_epoch_clean":42,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-06-03T09:07:02.103345+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-06-03T09:07:02.103345+0000","last_clean_scrub_stamp":"2024-06-03T09:07:02.103345+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-06-04T20:06:45.041047+0000","scrub_duration":0.036150559999999998,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,3],"acting":[4,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"17.1a","version":"0'0","reported_seq":983,"reported_epoch":1014,"state":"active+clean","last_fresh":"2024-06-03T09:07:07.163797+0000","last_change":"2024-06-03T09:07:07.165427+0000","last_active":"2024-06-03T09:07:07.163797+0000","last_peered":"2024-06-03T09:07:07.163797+0000","last_clean":"2024-06-03T09:07:07.163797+0000","last_became_active":"2024-06-03T09:06:46.386469+0000","last_became_peered":"2024-06-03T09:06:46.386469+0000","last_unstale":"2024-06-03T09:07:07.163797+0000","last_undegraded":"2024-06-03T09:07:07.163797+0000","last_fullsized":"2024-06-03T09:07:07.163797+0000","mapping_epoch":1013,"log_start":"0'0","ondisk_log_start":"0'0","created":28,"last_epoch_clean":1014,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-06-03T09:07:07.163037+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-06-03T09:07:07.163037+0000","last_clean_scrub_stamp":"2024-06-03T09:07:07.163037+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-06-04T18:10:14.782734+0000","scrub_duration":0.031532482000000001,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,6],"acting":[4,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"17.1b","version":"0'0","reported_seq":951,"reported_epoch":1014,"state":"active+clean","last_fresh":"2024-06-03T09:07:30.699539+0000","last_change":"2024-06-03T09:07:30.701204+0000","last_active":"2024-06-03T09:07:30.699539+0000","last_peered":"2024-06-03T09:07:30.699539+0000","last_clean":"2024-06-03T09:07:30.699539+0000","last_became_active":"2024-06-03T08:38:03.110533+0000","last_became_peered":"2024-06-03T08:38:03.110533+0000","last_unstale":"2024-06-03T09:07:30.699539+0000","last_undegraded":"2024-06-03T09:07:30.699539+0000","last_fullsized":"2024-06-03T09:07:30.699539+0000","mapping_epoch":28,"log_start":"0'0","ondisk_log_start":"0'0","created":28,"last_epoch_clean":32,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-06-03T09:07:30.698820+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-06-03T09:07:30.698820+0000","last_clean_scrub_stamp":"2024-06-03T09:07:30.698820+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-06-04T20:02:45.742502+0000","scrub_duration":0.077492022999999993,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,7],"acting":[3,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"17.18","version":"0'0","reported_seq":960,"reported_epoch":1014,"state":"active+clean","last_fresh":"2024-06-03T09:07:04.460345+0000","last_change":"2024-06-03T09:07:04.460345+0000","last_active":"2024-06-03T09:07:04.460345+0000","last_peered":"2024-06-03T09:07:04.460345+0000","last_clean":"2024-06-03T09:07:04.460345+0000","last_became_active":"2024-06-03T08:38:04.627856+0000","last_became_peered":"2024-06-03T08:38:04.627856+0000","last_unstale":"2024-06-03T09:07:04.460345+0000","last_undegraded":"2024-06-03T09:07:04.460345+0000","last_fullsized":"2024-06-03T09:07:04.460345+0000","mapping_epoch":28,"log_start":"0'0","ondisk_log_start":"0'0","created":28,"last_epoch_clean":33,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-06-03T09:07:04.458032+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-06-03T09:07:04.458032+0000","last_clean_scrub_stamp":"2024-06-03T09:07:04.458032+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-06-04T17:11:27.305960+0000","scrub_duration":0.03213684,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,4],"acting":[7,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"17.19","version":"0'0","reported_seq":951,"reported_epoch":1014,"state":"active+clean","last_fresh":"2024-06-03T09:07:24.698335+0000","last_change":"2024-06-03T09:07:24.703492+0000","last_active":"2024-06-03T09:07:24.698335+0000","last_peered":"2024-06-03T09:07:24.698335+0000","last_clean":"2024-06-03T09:07:24.698335+0000","last_became_active":"2024-06-03T08:38:01.829539+0000","last_became_peered":"2024-06-03T08:38:01.829539+0000","last_unstale":"2024-06-03T09:07:24.698335+0000","last_undegraded":"2024-06-03T09:07:24.698335+0000","last_fullsized":"2024-06-03T09:07:24.698335+0000","mapping_epoch":28,"log_start":"0'0","ondisk_log_start":"0'0","created":28,"last_epoch_clean":30,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-06-03T09:07:24.697555+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-06-03T09:07:24.697555+0000","last_clean_scrub_stamp":"2024-06-03T09:07:24.697555+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-06-04T14:44:15.637527+0000","scrub_duration":0.051436971999999997,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,2],"acting":[3,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"17.1e","version":"0'0","reported_seq":856,"reported_epoch":1014,"state":"active+clean","last_fresh":"2024-06-03T09:07:10.882123+0000","last_change":"2024-06-03T09:07:10.883772+0000","last_active":"2024-06-03T09:07:10.882123+0000","last_peered":"2024-06-03T09:07:10.882123+0000","last_clean":"2024-06-03T09:07:10.882123+0000","last_became_active":"2024-06-03T09:06:46.288593+0000","last_became_peered":"2024-06-03T09:06:46.288593+0000","last_unstale":"2024-06-03T09:07:10.882123+0000","last_undegraded":"2024-06-03T09:07:10.882123+0000","last_fullsized":"2024-06-03T09:07:10.882123+0000","mapping_epoch":1013,"log_start":"0'0","ondisk_log_start":"0'0","created":28,"last_epoch_clean":1014,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-06-03T09:07:10.881377+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-06-03T09:07:10.881377+0000","last_clean_scrub_stamp":"2024-06-03T09:07:10.881377+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-06-04T15:00:04.646181+0000","scrub_duration":0.035250741000000002,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,4],"acting":[6,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"17.1f","version":"0'0","reported_seq":957,"reported_epoch":1014,"state":"active+clean","last_fresh":"2024-06-03T09:07:16.249909+0000","last_change":"2024-06-03T09:07:16.251476+0000","last_active":"2024-06-03T09:07:16.249909+0000","last_peered":"2024-06-03T09:07:16.249909+0000","last_clean":"2024-06-03T09:07:16.249909+0000","last_became_active":"2024-06-03T08:38:04.367278+0000","last_became_peered":"2024-06-03T08:38:04.367278+0000","last_unstale":"2024-06-03T09:07:16.249909+0000","last_undegraded":"2024-06-03T09:07:16.249909+0000","last_fullsized":"2024-06-03T09:07:16.249909+0000","mapping_epoch":28,"log_start":"0'0","ondisk_log_start":"0'0","created":28,"last_epoch_clean":32,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-06-03T09:07:16.249190+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-06-03T09:07:16.249190+0000","last_clean_scrub_stamp":"2024-06-03T09:07:16.249190+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-06-04T19:04:33.697530+0000","scrub_duration":0.069893753000000003,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,2],"acting":[0,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"17.1c","version":"0'0","reported_seq":968,"reported_epoch":1014,"state":"active+clean","last_fresh":"2024-06-03T09:07:28.411410+0000","last_change":"2024-06-03T09:07:28.413058+0000","last_active":"2024-06-03T09:07:28.411410+0000","last_peered":"2024-06-03T09:07:28.411410+0000","last_clean":"2024-06-03T09:07:28.411410+0000","last_became_active":"2024-06-03T09:06:46.403724+0000","last_became_peered":"2024-06-03T09:06:46.403724+0000","last_unstale":"2024-06-03T09:07:28.411410+0000","last_undegraded":"2024-06-03T09:07:28.411410+0000","last_fullsized":"2024-06-03T09:07:28.411410+0000","mapping_epoch":1013,"log_start":"0'0","ondisk_log_start":"0'0","created":28,"last_epoch_clean":1014,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-06-03T09:07:28.410663+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-06-03T09:07:28.410663+0000","last_clean_scrub_stamp":"2024-06-03T09:07:28.410663+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-06-04T13:25:43.654123+0000","scrub_duration":0.040100175000000002,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,6],"acting":[5,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"17.1d","version":"0'0","reported_seq":939,"reported_epoch":1014,"state":"active+clean","last_fresh":"2024-06-03T09:07:10.965559+0000","last_change":"2024-06-03T09:07:10.967165+0000","last_active":"2024-06-03T09:07:10.965559+0000","last_peered":"2024-06-03T09:07:10.965559+0000","last_clean":"2024-06-03T09:07:10.965559+0000","last_became_active":"2024-06-03T08:38:06.592317+0000","last_became_peered":"2024-06-03T08:38:06.592317+0000","last_unstale":"2024-06-03T09:07:10.965559+0000","last_undegraded":"2024-06-03T09:07:10.965559+0000","last_fullsized":"2024-06-03T09:07:10.965559+0000","mapping_epoch":28,"log_start":"0'0","ondisk_log_start":"0'0","created":28,"last_epoch_clean":33,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-06-03T09:07:10.964826+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-06-03T09:07:10.964826+0000","last_clean_scrub_stamp":"2024-06-03T09:07:10.964826+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-06-04T15:19:56.203494+0000","scrub_duration":0.062732020999999999,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,7],"acting":[1,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"17.2","version":"0'0","reported_seq":942,"reported_epoch":1014,"state":"active+clean","last_fresh":"2024-06-03T09:07:04.809640+0000","last_change":"2024-06-03T09:07:04.809640+0000","last_active":"2024-06-03T09:07:04.809640+0000","last_peered":"2024-06-03T09:07:04.809640+0000","last_clean":"2024-06-03T09:07:04.809640+0000","last_became_active":"2024-06-03T08:38:14.973825+0000","last_became_peered":"2024-06-03T08:38:14.973825+0000","last_unstale":"2024-06-03T09:07:04.809640+0000","last_undegraded":"2024-06-03T09:07:04.809640+0000","last_fullsized":"2024-06-03T09:07:04.809640+0000","mapping_epoch":28,"log_start":"0'0","ondisk_log_start":"0'0","created":28,"last_epoch_clean":41,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-06-03T09:07:04.807216+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-06-03T09:07:04.807216+0000","last_clean_scrub_stamp":"2024-06-03T09:07:04.807216+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-06-04T14:52:55.434564+0000","scrub_duration":0.030095302000000001,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,4],"acting":[1,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"17.3","version":"0'0","reported_seq":959,"reported_epoch":1014,"state":"active+clean","last_fresh":"2024-06-03T09:07:09.259431+0000","last_change":"2024-06-03T09:07:09.259431+0000","last_active":"2024-06-03T09:07:09.259431+0000","last_peered":"2024-06-03T09:07:09.259431+0000","last_clean":"2024-06-03T09:07:09.259431+0000","last_became_active":"2024-06-03T08:38:14.510665+0000","last_became_peered":"2024-06-03T08:38:14.510665+0000","last_unstale":"2024-06-03T09:07:09.259431+0000","last_undegraded":"2024-06-03T09:07:09.259431+0000","last_fullsized":"2024-06-03T09:07:09.259431+0000","mapping_epoch":28,"log_start":"0'0","ondisk_log_start":"0'0","created":28,"last_epoch_clean":42,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-06-03T09:07:09.253053+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-06-03T09:07:09.253053+0000","last_clean_scrub_stamp":"2024-06-03T09:07:09.253053+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-06-04T14:30:59.802025+0000","scrub_duration":0.068038479999999998,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,7],"acting":[0,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"17.0","version":"0'0","reported_seq":959,"reported_epoch":1014,"state":"active+clean","last_fresh":"2024-06-03T09:07:06.233712+0000","last_change":"2024-06-03T09:07:06.235311+0000","last_active":"2024-06-03T09:07:06.233712+0000","last_peered":"2024-06-03T09:07:06.233712+0000","last_clean":"2024-06-03T09:07:06.233712+0000","last_became_active":"2024-06-03T08:38:04.231954+0000","last_became_peered":"2024-06-03T08:38:04.231954+0000","last_unstale":"2024-06-03T09:07:06.233712+0000","last_undegraded":"2024-06-03T09:07:06.233712+0000","last_fullsized":"2024-06-03T09:07:06.233712+0000","mapping_epoch":28,"log_start":"0'0","ondisk_log_start":"0'0","created":28,"last_epoch_clean":32,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-06-03T09:07:06.232980+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-06-03T09:07:06.232980+0000","last_clean_scrub_stamp":"2024-06-03T09:07:06.232980+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-06-04T14:04:39.143315+0000","scrub_duration":0.033029297999999999,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,2],"acting":[0,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"17.1","version":"1014'8","reported_seq":985,"reported_epoch":1014,"state":"active+clean","last_fresh":"2024-06-03T09:07:05.543762+0000","last_change":"2024-06-03T09:07:05.543762+0000","last_active":"2024-06-03T09:07:05.543762+0000","last_peered":"2024-06-03T09:07:05.543762+0000","last_clean":"2024-06-03T09:07:05.543762+0000","last_became_active":"2024-06-03T09:06:46.383290+0000","last_became_peered":"2024-06-03T09:06:46.383290+0000","last_unstale":"2024-06-03T09:07:05.543762+0000","last_undegraded":"2024-06-03T09:07:05.543762+0000","last_fullsized":"2024-06-03T09:07:05.543762+0000","mapping_epoch":1013,"log_start":"0'0","ondisk_log_start":"0'0","created":28,"last_epoch_clean":1014,"parent":"0.0","parent_split_bits":0,"last_scrub":"1014'8","last_scrub_stamp":"2024-06-03T09:07:05.540994+0000","last_deep_scrub":"1014'8","last_deep_scrub_stamp":"2024-06-03T09:07:05.540994+0000","last_clean_scrub_stamp":"2024-06-03T09:07:05.540994+0000","objects_scrubbed":1,"log_size":8,"log_dups_size":0,"ondisk_log_size":8,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-06-04T18:34:55.019060+0000","scrub_duration":0.091133806999999997,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":5,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":6,"num_read_kb":6,"num_write":7,"num_write_kb":7,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,6],"acting":[5,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]}],"pool_stats":[{"poolid":2,"num_pg":8,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":16,"acting":16,"num_store_stats":7},{"poolid":1,"num_pg":1,"stat_sum":{"num_bytes":459280,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":96,"num_read_kb":82,"num_write":113,"num_write_kb":1372,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2,"num_bytes_recovered":397840,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":802816,"data_stored":795680,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":78,"ondisk_log_size":78,"up":2,"acting":2,"num_store_stats":2},{"poolid":17,"num_pg":32,"stat_sum":{"num_bytes":5,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":6,"num_read_kb":6,"num_write":7,"num_write_kb":7,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":8192,"data_stored":10,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":8,"ondisk_log_size":8,"up":64,"acting":64,"num_store_stats":8}],"osd_stats":[{"osd":6,"up_from":1013,"seq":4350801871173,"num_pgs":14,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":54916,"kb_used_data":5756,"kb_used_omap":1,"kb_used_meta":49150,"kb_avail":94316924,"statfs":{"total":96636764160,"available":96580530176,"internally_reserved":0,"allocated":5894144,"data_stored":2763078,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1593,"internal_metadata":50330055},"hb_peers":[0,1,2,3,4,5,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Mon Jun 3 09:06:42 2024","interfaces":[{"interface":"back","average":{"1min":5696.3109999999997,"5min":1469.3900000000001,"15min":503.11700000000002},"min":{"1min":5.242,"5min":4.4829999999999997,"15min":4.1959999999999997},"max":{"1min":20463.029999999999,"5min":22766.882000000001,"15min":22766.882000000001},"last":6.7460000000000004},{"interface":"front","average":{"1min":5687.6390000000001,"5min":1468.1669999999999,"15min":504.13200000000001},"min":{"1min":4.835,"5min":4.8129999999999997,"15min":4.4530000000000003},"max":{"1min":20461.830000000002,"5min":22766.289000000001,"15min":22766.289000000001},"last":13.135999999999999}]},{"osd":1,"last update":"Mon Jun 3 09:06:42 2024","interfaces":[{"interface":"back","average":{"1min":36.496000000000002,"5min":80.162999999999997,"15min":134.59399999999999},"min":{"1min":5.9189999999999996,"5min":5.8799999999999999,"15min":5.7119999999999997},"max":{"1min":544.85799999999995,"5min":4138.1679999999997,"15min":9955.1299999999992},"last":12.404},{"interface":"front","average":{"1min":38.866999999999997,"5min":82.631,"15min":135.11500000000001},"min":{"1min":6.5350000000000001,"5min":6.5350000000000001,"15min":6.1390000000000002},"max":{"1min":546.73099999999999,"5min":4137.6530000000002,"15min":9827.5509999999995},"last":14.433999999999999}]},{"osd":2,"last update":"Mon Jun 3 09:06:42 2024","interfaces":[{"interface":"back","average":{"1min":41.093000000000004,"5min":27.917000000000002,"15min":22.329000000000001},"min":{"1min":6.0670000000000002,"5min":6.0670000000000002,"15min":6.0670000000000002},"max":{"1min":548.12900000000002,"5min":548.12900000000002,"15min":548.12900000000002},"last":9.8439999999999994},{"interface":"front","average":{"1min":30.798999999999999,"5min":24.276,"15min":22.488},"min":{"1min":6.4370000000000003,"5min":6.4370000000000003,"15min":6.1870000000000003},"max":{"1min":369.14499999999998,"5min":369.14499999999998,"15min":434.77300000000002},"last":7.8399999999999999}]},{"osd":3,"last update":"Mon Jun 3 09:06:42 2024","interfaces":[{"interface":"back","average":{"1min":41.411999999999999,"5min":27.434999999999999,"15min":30.657},"min":{"1min":10.061,"5min":8.3179999999999996,"15min":6.1719999999999997},"max":{"1min":549.024,"5min":549.024,"15min":1235.5799999999999},"last":16.960999999999999},{"interface":"front","average":{"1min":37.850999999999999,"5min":22.001000000000001,"15min":28.015999999999998},"min":{"1min":6.0679999999999996,"5min":6.0679999999999996,"15min":6.0679999999999996},"max":{"1min":559.25,"5min":559.25,"15min":1157.6780000000001},"last":13.744999999999999}]},{"osd":4,"last update":"Mon Jun 3 09:06:42 2024","interfaces":[{"interface":"back","average":{"1min":165.62899999999999,"5min":1281.1010000000001,"15min":440.59199999999998},"min":{"1min":10.965999999999999,"5min":6.093,"15min":6.093},"max":{"1min":384.46899999999999,"5min":28093.096000000001,"15min":28093.096000000001},"last":11.144},{"interface":"front","average":{"1min":186.97900000000001,"5min":1308.886,"15min":454.97699999999998},"min":{"1min":13.581,"5min":7.7480000000000002,"15min":7.7480000000000002},"max":{"1min":445.096,"5min":28323.837,"15min":28323.837},"last":17.690999999999999}]},{"osd":5,"last update":"Mon Jun 3 09:06:42 2024","interfaces":[{"interface":"back","average":{"1min":81.581000000000003,"5min":34.863,"15min":28.350999999999999},"min":{"1min":10.345000000000001,"5min":8.625,"15min":7.516},"max":{"1min":455.84199999999998,"5min":455.84199999999998,"15min":455.84199999999998},"last":16.248000000000001},{"interface":"front","average":{"1min":83.391000000000005,"5min":39.277999999999999,"15min":27.885000000000002},"min":{"1min":8.0280000000000005,"5min":6.266,"15min":6.266},"max":{"1min":550.02300000000002,"5min":550.02300000000002,"15min":550.02300000000002},"last":10.523}]},{"osd":7,"last update":"Mon Jun 3 09:06:42 2024","interfaces":[{"interface":"back","average":{"1min":44.052999999999997,"5min":33.697000000000003,"15min":29.731999999999999},"min":{"1min":9.1500000000000004,"5min":9.1500000000000004,"15min":9.1500000000000004},"max":{"1min":552.09500000000003,"5min":552.09500000000003,"15min":552.09500000000003},"last":18.515999999999998},{"interface":"front","average":{"1min":47.076999999999998,"5min":34.226999999999997,"15min":29.827000000000002},"min":{"1min":9.7780000000000005,"5min":8.6709999999999994,"15min":8.5820000000000007},"max":{"1min":531.08600000000001,"5min":531.08600000000001,"15min":531.08600000000001},"last":15.013999999999999}]}]},{"osd":7,"up_from":20,"seq":85899346301,"num_pgs":9,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":59008,"kb_used_data":5752,"kb_used_omap":1,"kb_used_meta":53246,"kb_avail":94312832,"statfs":{"total":96636764160,"available":96576339968,"internally_reserved":0,"allocated":5890048,"data_stored":2763073,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1589,"internal_metadata":54524363},"hb_peers":[0,1,2,3,4,5,6],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Mon Jun 3 09:07:10 2024","interfaces":[{"interface":"back","average":{"1min":8.9380000000000006,"5min":893.83100000000002,"15min":307.63400000000001},"min":{"1min":4.032,"5min":4.032,"15min":4.032},"max":{"1min":17.491,"5min":21729.001,"15min":21729.001},"last":8.8209999999999997},{"interface":"front","average":{"1min":7.8970000000000002,"5min":905.91999999999996,"15min":312.15100000000001},"min":{"1min":4.5979999999999999,"5min":4.5979999999999999,"15min":4.5979999999999999},"max":{"1min":16.053999999999998,"5min":21899.732,"15min":21899.732},"last":7.0369999999999999}]},{"osd":1,"last update":"Mon Jun 3 09:07:10 2024","interfaces":[{"interface":"back","average":{"1min":8.5670000000000002,"5min":68.463999999999999,"15min":218.52199999999999},"min":{"1min":5.3419999999999996,"5min":5.3419999999999996,"15min":5.3419999999999996},"max":{"1min":11.128,"5min":2795.1329999999998,"15min":18373.257000000001},"last":10.263},{"interface":"front","average":{"1min":12.664999999999999,"5min":73.626000000000005,"15min":224.589},"min":{"1min":9.6460000000000008,"5min":8.8399999999999999,"15min":8.5129999999999999},"max":{"1min":15.045,"5min":2803.3670000000002,"15min":18374.601999999999},"last":13.753}]},{"osd":2,"last update":"Mon Jun 3 09:07:10 2024","interfaces":[{"interface":"back","average":{"1min":13.367000000000001,"5min":25.641999999999999,"15min":22.251999999999999},"min":{"1min":10.000999999999999,"5min":8.5030000000000001,"15min":8.5030000000000001},"max":{"1min":18.968,"5min":333.99200000000002,"15min":333.99200000000002},"last":11.254},{"interface":"front","average":{"1min":9.5410000000000004,"5min":17.655000000000001,"15min":15.645},"min":{"1min":6.8840000000000003,"5min":5.8369999999999997,"15min":5.5069999999999997},"max":{"1min":13.875,"5min":226.18100000000001,"15min":226.18100000000001},"last":12.648999999999999}]},{"osd":3,"last update":"Mon Jun 3 09:07:10 2024","interfaces":[{"interface":"back","average":{"1min":12.894,"5min":25.039000000000001,"15min":30.681999999999999},"min":{"1min":7.617,"5min":6.8959999999999999,"15min":6.3760000000000003},"max":{"1min":16.376000000000001,"5min":316.64400000000001,"15min":2039.9059999999999},"last":16.859999999999999},{"interface":"front","average":{"1min":9.3800000000000008,"5min":17.294,"15min":24.911999999999999},"min":{"1min":6.1929999999999996,"5min":5.2089999999999996,"15min":5.2089999999999996},"max":{"1min":13.622999999999999,"5min":233.542,"15min":2037.5550000000001},"last":7.9349999999999996}]},{"osd":4,"last update":"Mon Jun 3 09:07:30 2024","interfaces":[{"interface":"back","average":{"1min":12.256,"5min":18.018999999999998,"15min":471.79300000000001},"min":{"1min":5.2709999999999999,"5min":5.2709999999999999,"15min":5.2709999999999999},"max":{"1min":16.548999999999999,"5min":224.54300000000001,"15min":30073.917000000001},"last":9.7390000000000008},{"interface":"front","average":{"1min":12.271000000000001,"5min":19.523,"15min":479.38600000000002},"min":{"1min":9.2680000000000007,"5min":5.9279999999999999,"15min":5.9279999999999999},"max":{"1min":17.760000000000002,"5min":305.05099999999999,"15min":30348.021000000001},"last":14.346}]},{"osd":5,"last update":"Mon Jun 3 09:07:04 2024","interfaces":[{"interface":"back","average":{"1min":18.949999999999999,"5min":26.824000000000002,"15min":23.492000000000001},"min":{"1min":10.933999999999999,"5min":9.2460000000000004,"15min":7.0010000000000003},"max":{"1min":95.691999999999993,"5min":317.572,"15min":317.572},"last":15.880000000000001},{"interface":"front","average":{"1min":17.582999999999998,"5min":24.216999999999999,"15min":20.686},"min":{"1min":8.5760000000000005,"5min":7.0999999999999996,"15min":7.008},"max":{"1min":92.829999999999998,"5min":306.60599999999999,"15min":306.60599999999999},"last":15.352}]},{"osd":6,"last update":"Mon Jun 3 09:04:31 2024","interfaces":[{"interface":"back","average":{"1min":44.061,"5min":26.821000000000002,"15min":91.176000000000002},"min":{"1min":9.0800000000000001,"5min":7.9390000000000001,"15min":7.9390000000000001},"max":{"1min":268.029,"5min":308.916,"15min":8734.4639999999999},"last":11.974},{"interface":"front","average":{"1min":46.643000000000001,"5min":23.879999999999999,"15min":88.765000000000001},"min":{"1min":8.8119999999999994,"5min":6.9240000000000004,"15min":6.742},"max":{"1min":305.64600000000002,"5min":305.64600000000002,"15min":8735.5319999999992},"last":17.687000000000001}]}]},{"osd":5,"up_from":20,"seq":85899346276,"num_pgs":9,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":59404,"kb_used_data":6148,"kb_used_omap":1,"kb_used_meta":53246,"kb_avail":94312436,"statfs":{"total":96636764160,"available":96575934464,"internally_reserved":0,"allocated":6295552,"data_stored":3160918,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1594,"internal_metadata":54524358},"hb_peers":[0,1,2,3,4,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Mon Jun 3 09:07:06 2024","interfaces":[{"interface":"back","average":{"1min":7.5899999999999999,"5min":1702.617,"15min":579.77099999999996},"min":{"1min":4.5380000000000003,"5min":4.5380000000000003,"15min":4.4020000000000001},"max":{"1min":16.231999999999999,"5min":26433.764999999999,"15min":26433.764999999999},"last":12.41},{"interface":"front","average":{"1min":10.436,"5min":1725.6030000000001,"15min":585.86500000000001},"min":{"1min":6.9699999999999998,"5min":5.1319999999999997,"15min":4.8170000000000002},"max":{"1min":17.734000000000002,"5min":26704.698,"15min":26704.698},"last":15.801}]},{"osd":1,"last update":"Mon Jun 3 09:07:01 2024","interfaces":[{"interface":"back","average":{"1min":11.929,"5min":66.978999999999999,"15min":216.91999999999999},"min":{"1min":5.9790000000000001,"5min":5.9790000000000001,"15min":5.4930000000000003},"max":{"1min":22.202999999999999,"5min":3923.4090000000001,"15min":18811.696},"last":16.931999999999999},{"interface":"front","average":{"1min":11.496,"5min":64.528000000000006,"15min":220.06100000000001},"min":{"1min":6.298,"5min":5.3209999999999997,"15min":5.0209999999999999},"max":{"1min":27.747,"5min":4019.6129999999998,"15min":18968.365000000002},"last":9.5030000000000001}]},{"osd":2,"last update":"Mon Jun 3 09:07:01 2024","interfaces":[{"interface":"back","average":{"1min":13.903,"5min":24.381,"15min":20.638999999999999},"min":{"1min":9.3559999999999999,"5min":5.6369999999999996,"15min":5.0510000000000002},"max":{"1min":27.013999999999999,"5min":332.48200000000003,"15min":332.48200000000003},"last":17.695},{"interface":"front","average":{"1min":9.2989999999999995,"5min":22.521999999999998,"15min":23.033999999999999},"min":{"1min":6.2439999999999998,"5min":6.2439999999999998,"15min":6.2439999999999998},"max":{"1min":18.395,"5min":353.03199999999998,"15min":353.03199999999998},"last":12.932}]},{"osd":3,"last update":"Mon Jun 3 09:07:01 2024","interfaces":[{"interface":"back","average":{"1min":14.686999999999999,"5min":24.370999999999999,"15min":31.327000000000002},"min":{"1min":8.1140000000000008,"5min":7.4720000000000004,"15min":7.4210000000000003},"max":{"1min":30.178000000000001,"5min":353.57499999999999,"15min":2855.8519999999999},"last":18.943999999999999},{"interface":"front","average":{"1min":11.590999999999999,"5min":16.498999999999999,"15min":25.437999999999999},"min":{"1min":6.3769999999999998,"5min":5.9660000000000002,"15min":5.1909999999999998},"max":{"1min":22.972000000000001,"5min":203.804,"15min":2869.7860000000001},"last":10.396000000000001}]},{"osd":4,"last update":"Mon Jun 3 09:07:30 2024","interfaces":[{"interface":"back","average":{"1min":12.182,"5min":15.939,"15min":376.45100000000002},"min":{"1min":6.7199999999999998,"5min":5.4779999999999998,"15min":5.4779999999999998},"max":{"1min":28.286999999999999,"5min":156.55199999999999,"15min":28619.825000000001},"last":14.048},{"interface":"front","average":{"1min":16.363,"5min":21.972000000000001,"15min":377.21499999999997},"min":{"1min":11.121,"5min":6.5030000000000001,"15min":6.5030000000000001},"max":{"1min":31.370000000000001,"5min":289.51400000000001,"15min":28621.294000000002},"last":18.434000000000001}]},{"osd":6,"last update":"Mon Jun 3 09:03:57 2024","interfaces":[{"interface":"back","average":{"1min":23.536000000000001,"5min":36.457000000000001,"15min":83.799999999999997},"min":{"1min":9.4800000000000004,"5min":9.4800000000000004,"15min":8.0630000000000006},"max":{"1min":89.442999999999998,"5min":508.77999999999997,"15min":9193.5169999999998},"last":14.609},{"interface":"front","average":{"1min":18.891999999999999,"5min":22.100999999999999,"15min":75.489999999999995},"min":{"1min":5.923,"5min":5.923,"15min":5.923},"max":{"1min":78.424999999999997,"5min":464.62,"15min":9191.4429999999993},"last":19.965}]},{"osd":7,"last update":"Mon Jun 3 09:07:01 2024","interfaces":[{"interface":"back","average":{"1min":15.648999999999999,"5min":21.43,"15min":20.760999999999999},"min":{"1min":8.6319999999999997,"5min":7.4020000000000001,"15min":6.7629999999999999},"max":{"1min":28.895,"5min":211.22999999999999,"15min":211.22999999999999},"last":15.111000000000001},{"interface":"front","average":{"1min":13.545999999999999,"5min":20.359000000000002,"15min":23.413},"min":{"1min":6.8239999999999998,"5min":6.8239999999999998,"15min":6.8239999999999998},"max":{"1min":36.515000000000001,"5min":223.191,"15min":286.69499999999999},"last":11.803000000000001}]}]},{"osd":4,"up_from":20,"seq":85899346283,"num_pgs":12,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":63104,"kb_used_data":5752,"kb_used_omap":1,"kb_used_meta":57342,"kb_avail":94308736,"statfs":{"total":96636764160,"available":96572145664,"internally_reserved":0,"allocated":5890048,"data_stored":2763073,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1594,"internal_metadata":58718662},"hb_peers":[0,1,2,3,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Mon Jun 3 09:07:38 2024","interfaces":[{"interface":"back","average":{"1min":7.7130000000000001,"5min":1092.357,"15min":379.98500000000001},"min":{"1min":5.0679999999999996,"5min":4.5279999999999996,"15min":4.3220000000000001},"max":{"1min":13.69,"5min":21726.743999999999,"15min":21726.743999999999},"last":6.7599999999999998},{"interface":"front","average":{"1min":12.016,"5min":1100.883,"15min":386.40199999999999},"min":{"1min":7.9640000000000004,"5min":5.8380000000000001,"15min":5.8380000000000001},"max":{"1min":21.422000000000001,"5min":21727.993999999999,"15min":21727.993999999999},"last":13.712}]},{"osd":1,"last update":"Mon Jun 3 09:07:38 2024","interfaces":[{"interface":"back","average":{"1min":10.042999999999999,"5min":50.198,"15min":339.28199999999998},"min":{"1min":4.3799999999999999,"5min":4.1970000000000001,"15min":4.1970000000000001},"max":{"1min":26.73,"5min":3346.4389999999999,"15min":22256.098000000002},"last":8.2270000000000003},{"interface":"front","average":{"1min":12.218999999999999,"5min":51.030000000000001,"15min":338.81700000000001},"min":{"1min":6.1669999999999998,"5min":5.0590000000000002,"15min":4.2640000000000002},"max":{"1min":26.995000000000001,"5min":3347.181,"15min":22247.317999999999},"last":12.491}]},{"osd":2,"last update":"Mon Jun 3 09:07:38 2024","interfaces":[{"interface":"back","average":{"1min":11.913,"5min":16.399000000000001,"15min":19.949999999999999},"min":{"1min":6.5129999999999999,"5min":6.3010000000000002,"15min":6.3010000000000002},"max":{"1min":25.855,"5min":337.60399999999998,"15min":393.58600000000001},"last":10.952},{"interface":"front","average":{"1min":13.304,"5min":13.912000000000001,"15min":21.908000000000001},"min":{"1min":7.3200000000000003,"5min":6.0629999999999997,"15min":6.0629999999999997},"max":{"1min":26.456,"5min":127.913,"15min":394.471},"last":14.436}]},{"osd":3,"last update":"Mon Jun 3 09:07:38 2024","interfaces":[{"interface":"back","average":{"1min":14.153,"5min":15.855,"15min":17.602},"min":{"1min":8.4939999999999998,"5min":5.8719999999999999,"15min":5.4000000000000004},"max":{"1min":29.57,"5min":127.43600000000001,"15min":322.63},"last":13.113},{"interface":"front","average":{"1min":13.311999999999999,"5min":16.239999999999998,"15min":22.145},"min":{"1min":6.7789999999999999,"5min":6.7789999999999999,"15min":6.7789999999999999},"max":{"1min":28.472000000000001,"5min":204.517,"15min":392.67500000000001},"last":9.2129999999999992}]},{"osd":5,"last update":"Mon Jun 3 09:07:38 2024","interfaces":[{"interface":"back","average":{"1min":14.016999999999999,"5min":16.088000000000001,"15min":20.417999999999999},"min":{"1min":8.1690000000000005,"5min":6.7489999999999997,"15min":6.7489999999999997},"max":{"1min":30.233000000000001,"5min":268.10199999999998,"15min":344.23399999999998},"last":11.494999999999999},{"interface":"front","average":{"1min":16.742000000000001,"5min":18.704000000000001,"15min":24.957999999999998},"min":{"1min":10.318,"5min":10.318,"15min":8.5180000000000007},"max":{"1min":31.635999999999999,"5min":264.75,"15min":345.18700000000001},"last":15.683}]},{"osd":6,"last update":"Mon Jun 3 09:04:32 2024","interfaces":[{"interface":"back","average":{"1min":15.548,"5min":32.957999999999998,"15min":152.738},"min":{"1min":9.141,"5min":9.141,"15min":8.2319999999999993},"max":{"1min":52.57,"5min":395.238,"15min":12496.335999999999},"last":5.9189999999999996},{"interface":"front","average":{"1min":12.680999999999999,"5min":25.428000000000001,"15min":149.30600000000001},"min":{"1min":6.4379999999999997,"5min":5.6319999999999997,"15min":5.6319999999999997},"max":{"1min":46.283000000000001,"5min":377.74299999999999,"15min":12616.297},"last":17.193000000000001}]},{"osd":7,"last update":"Mon Jun 3 09:07:38 2024","interfaces":[{"interface":"back","average":{"1min":11.619,"5min":13.195,"15min":18.27},"min":{"1min":5.7249999999999996,"5min":5.7249999999999996,"15min":5.7249999999999996},"max":{"1min":29.298999999999999,"5min":207.25299999999999,"15min":358.94900000000001},"last":9.8339999999999996},{"interface":"front","average":{"1min":17.565000000000001,"5min":18.817,"15min":22.152999999999999},"min":{"1min":11.362,"5min":10.313000000000001,"15min":7.7789999999999999},"max":{"1min":30.835000000000001,"5min":209.32900000000001,"15min":380.06900000000002},"last":16.434000000000001}]}]},{"osd":2,"up_from":20,"seq":85899346303,"num_pgs":11,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":59008,"kb_used_data":5752,"kb_used_omap":1,"kb_used_meta":53246,"kb_avail":94312832,"statfs":{"total":96636764160,"available":96576339968,"internally_reserved":0,"allocated":5890048,"data_stored":2763073,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1594,"internal_metadata":54524358},"hb_peers":[0,1,3,4,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Mon Jun 3 09:07:26 2024","interfaces":[{"interface":"back","average":{"1min":7.9020000000000001,"5min":945.92200000000003,"15min":325.45800000000003},"min":{"1min":4.7519999999999998,"5min":4.7290000000000001,"15min":4.1420000000000003},"max":{"1min":11.744999999999999,"5min":23465.18,"15min":23465.18},"last":10.664999999999999},{"interface":"front","average":{"1min":7.8810000000000002,"5min":944.41099999999994,"15min":325.19400000000002},"min":{"1min":4.492,"5min":4.492,"15min":4.492},"max":{"1min":13.558,"5min":23465.966,"15min":23465.966},"last":5.4989999999999997}]},{"osd":1,"last update":"Mon Jun 3 09:07:26 2024","interfaces":[{"interface":"back","average":{"1min":8.9459999999999997,"5min":75.915999999999997,"15min":293.77300000000002},"min":{"1min":6.4100000000000001,"5min":5.4199999999999999,"15min":5.4199999999999999},"max":{"1min":14.981999999999999,"5min":4220.5290000000005,"15min":18954.987000000001},"last":11.500999999999999},{"interface":"front","average":{"1min":11.898999999999999,"5min":81.325000000000003,"15min":301.93400000000003},"min":{"1min":9.1620000000000008,"5min":7.367,"15min":7.367},"max":{"1min":15.513,"5min":4325.9489999999996,"15min":19143.834999999999},"last":14.589}]},{"osd":3,"last update":"Mon Jun 3 09:07:26 2024","interfaces":[{"interface":"back","average":{"1min":10.456,"5min":12.208,"15min":27.460999999999999},"min":{"1min":7.0700000000000003,"5min":5.9290000000000003,"15min":5.2450000000000001},"max":{"1min":20.806999999999999,"5min":118.913,"15min":2594.8150000000001},"last":7.9710000000000001},{"interface":"front","average":{"1min":12.379,"5min":14.701000000000001,"15min":30.411000000000001},"min":{"1min":8.6649999999999991,"5min":8.0760000000000005,"15min":6.0780000000000003},"max":{"1min":16.465,"5min":156.34700000000001,"15min":2706.5120000000002},"last":10.115}]},{"osd":4,"last update":"Mon Jun 3 09:06:59 2024","interfaces":[{"interface":"back","average":{"1min":11.795999999999999,"5min":16.489000000000001,"15min":437.19299999999998},"min":{"1min":7.8780000000000001,"5min":7.8780000000000001,"15min":6.8049999999999997},"max":{"1min":30.163,"5min":182.07400000000001,"15min":26660.861000000001},"last":9.0150000000000006},{"interface":"front","average":{"1min":14.726000000000001,"5min":20.835000000000001,"15min":435.017},"min":{"1min":10.970000000000001,"5min":10.266999999999999,"15min":6.2720000000000002},"max":{"1min":33.402999999999999,"5min":216.476,"15min":26616.243999999999},"last":12.795999999999999}]},{"osd":5,"last update":"Mon Jun 3 09:07:26 2024","interfaces":[{"interface":"back","average":{"1min":14.26,"5min":17.93,"15min":20.533999999999999},"min":{"1min":9.3249999999999993,"5min":9.2159999999999993,"15min":8.4900000000000002},"max":{"1min":23.541,"5min":252.31100000000001,"15min":252.31100000000001},"last":11.994},{"interface":"front","average":{"1min":10.9,"5min":15.696,"15min":18.986999999999998},"min":{"1min":7.0940000000000003,"5min":7.0940000000000003,"15min":6.8810000000000002},"max":{"1min":18.452999999999999,"5min":258.31700000000001,"15min":258.31700000000001},"last":15.82}]},{"osd":6,"last update":"Mon Jun 3 09:04:28 2024","interfaces":[{"interface":"back","average":{"1min":14.063000000000001,"5min":24.850000000000001,"15min":106.755},"min":{"1min":9.0820000000000007,"5min":8.9649999999999999,"15min":8.9649999999999999},"max":{"1min":24.666,"5min":188.06700000000001,"15min":9521.0010000000002},"last":16.550999999999998},{"interface":"front","average":{"1min":13.933999999999999,"5min":21.081,"15min":106.142},"min":{"1min":5.8719999999999999,"5min":5.8719999999999999,"15min":5.8719999999999999},"max":{"1min":117.551,"5min":193.887,"15min":9585.6620000000003},"last":8.2170000000000005}]},{"osd":7,"last update":"Mon Jun 3 09:07:26 2024","interfaces":[{"interface":"back","average":{"1min":14.223000000000001,"5min":19.32,"15min":22.773},"min":{"1min":10.624000000000001,"5min":6.6130000000000004,"15min":6.6130000000000004},"max":{"1min":18.039000000000001,"5min":242.56800000000001,"15min":242.56800000000001},"last":13.337999999999999},{"interface":"front","average":{"1min":16.09,"5min":20.786999999999999,"15min":24.945},"min":{"1min":12.301,"5min":9.0660000000000007,"15min":9.0660000000000007},"max":{"1min":22.489000000000001,"5min":181.56399999999999,"15min":214.33799999999999},"last":13.964}]}]},{"osd":3,"up_from":14,"seq":60129542517,"num_pgs":11,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":59008,"kb_used_data":5752,"kb_used_omap":1,"kb_used_meta":53246,"kb_avail":94312832,"statfs":{"total":96636764160,"available":96576339968,"internally_reserved":0,"allocated":5890048,"data_stored":2763073,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1592,"internal_metadata":54524360},"hb_peers":[0,1,2,4,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Mon Jun 3 09:07:26 2024","interfaces":[{"interface":"back","average":{"1min":7.3159999999999998,"5min":1054.8779999999999,"15min":365.84899999999999},"min":{"1min":5.1989999999999998,"5min":5.173,"15min":5.173},"max":{"1min":11.048999999999999,"5min":23275.167000000001,"15min":23275.167000000001},"last":7.0800000000000001},{"interface":"front","average":{"1min":12.42,"5min":1061.508,"15min":371.42000000000002},"min":{"1min":7.8159999999999998,"5min":7.2510000000000003,"15min":5.9880000000000004},"max":{"1min":16.870000000000001,"5min":23278.536,"15min":23278.536},"last":12.805}]},{"osd":1,"last update":"Mon Jun 3 09:07:26 2024","interfaces":[{"interface":"back","average":{"1min":8.5920000000000005,"5min":10.272,"15min":389.517},"min":{"1min":5.3689999999999998,"5min":4.335,"15min":4.335},"max":{"1min":14.029,"5min":68.688000000000002,"15min":21658.205999999998},"last":7.6600000000000001},{"interface":"front","average":{"1min":9.1739999999999995,"5min":10.202,"15min":380.74000000000001},"min":{"1min":6.109,"5min":5.4100000000000001,"15min":5.2779999999999996},"max":{"1min":14.237,"5min":67.974999999999994,"15min":21307.069},"last":9.1560000000000006}]},{"osd":2,"last update":"Mon Jun 3 09:07:26 2024","interfaces":[{"interface":"back","average":{"1min":12.747,"5min":17.745999999999999,"15min":23.242999999999999},"min":{"1min":8.1519999999999992,"5min":6.4900000000000002,"15min":6.056},"max":{"1min":18.875,"5min":377.803,"15min":377.803},"last":8.4350000000000005},{"interface":"front","average":{"1min":10.028,"5min":16.437999999999999,"15min":19.620000000000001},"min":{"1min":6.9720000000000004,"5min":6.3769999999999998,"15min":5.9020000000000001},"max":{"1min":17.957999999999998,"5min":371.601,"15min":371.601},"last":10.895}]},{"osd":4,"last update":"Mon Jun 3 09:07:36 2024","interfaces":[{"interface":"back","average":{"1min":10.268000000000001,"5min":12.093,"15min":416.07100000000003},"min":{"1min":6.827,"5min":5.6539999999999999,"15min":5.6539999999999999},"max":{"1min":14.907,"5min":83.629999999999995,"15min":30144.939999999999},"last":9.8049999999999997},{"interface":"front","average":{"1min":12.494999999999999,"5min":15.882999999999999,"15min":420.495},"min":{"1min":8.5530000000000008,"5min":7.8109999999999999,"15min":6.2439999999999998},"max":{"1min":17.635999999999999,"5min":232.20699999999999,"15min":30180.689999999999},"last":10.323}]},{"osd":5,"last update":"Mon Jun 3 09:07:10 2024","interfaces":[{"interface":"back","average":{"1min":13.355,"5min":17.076000000000001,"15min":21.516999999999999},"min":{"1min":8.8510000000000009,"5min":6.8810000000000002,"15min":6.7690000000000001},"max":{"1min":20.768999999999998,"5min":137.64699999999999,"15min":327.34899999999999},"last":15.679},{"interface":"front","average":{"1min":15.882999999999999,"5min":21.332999999999998,"15min":26.346},"min":{"1min":12.529,"5min":9.9149999999999991,"15min":9.8849999999999998},"max":{"1min":25.289999999999999,"5min":182.32599999999999,"15min":331.86799999999999},"last":16.475999999999999}]},{"osd":6,"last update":"Mon Jun 3 09:04:04 2024","interfaces":[{"interface":"back","average":{"1min":22.815999999999999,"5min":31.513999999999999,"15min":181.74299999999999},"min":{"1min":9.4809999999999999,"5min":7.2240000000000002,"15min":7.2240000000000002},"max":{"1min":160.56999999999999,"5min":380.613,"15min":11893.74},"last":13.369999999999999},{"interface":"front","average":{"1min":25.359999999999999,"5min":30.981000000000002,"15min":179.81399999999999},"min":{"1min":9.8870000000000005,"5min":7.3360000000000003,"15min":6.0899999999999999},"max":{"1min":193.11099999999999,"5min":379.80700000000002,"15min":11893.069},"last":11.413}]},{"osd":7,"last update":"Mon Jun 3 09:07:26 2024","interfaces":[{"interface":"back","average":{"1min":12.329000000000001,"5min":14.183,"15min":20.984999999999999},"min":{"1min":7.915,"5min":7.915,"15min":7.4279999999999999},"max":{"1min":19.881,"5min":93.637,"15min":178.76300000000001},"last":14.502000000000001},{"interface":"front","average":{"1min":14.262,"5min":16.318000000000001,"15min":23.893999999999998},"min":{"1min":9.8960000000000008,"5min":7.4989999999999997,"15min":7.4989999999999997},"max":{"1min":23.483000000000001,"5min":103.571,"15min":417.22800000000001},"last":14.025}]}]},{"osd":1,"up_from":14,"seq":60129542515,"num_pgs":6,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":59400,"kb_used_data":6144,"kb_used_omap":1,"kb_used_meta":53246,"kb_avail":94312440,"statfs":{"total":96636764160,"available":96575938560,"internally_reserved":0,"allocated":6291456,"data_stored":3160913,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":54524357},"hb_peers":[0,2,3,4,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Mon Jun 3 09:07:08 2024","interfaces":[{"interface":"back","average":{"1min":7.8280000000000003,"5min":185.30600000000001,"15min":75.007999999999996},"min":{"1min":4.5090000000000003,"5min":4.5090000000000003,"15min":4.5090000000000003},"max":{"1min":16.443999999999999,"5min":7470.616,"15min":7470.616},"last":7.077},{"interface":"front","average":{"1min":10.396000000000001,"5min":187.196,"15min":73.921000000000006},"min":{"1min":5.5709999999999997,"5min":5.3159999999999998,"15min":5.3159999999999998},"max":{"1min":20.145,"5min":7477.0829999999996,"15min":7477.0829999999996},"last":6.827}]},{"osd":2,"last update":"Mon Jun 3 09:07:21 2024","interfaces":[{"interface":"back","average":{"1min":10.005000000000001,"5min":15.744999999999999,"15min":23.135999999999999},"min":{"1min":7.5259999999999998,"5min":5.8650000000000002,"15min":5.8650000000000002},"max":{"1min":14.269,"5min":228,"15min":657.38},"last":8.4640000000000004},{"interface":"front","average":{"1min":13.448,"5min":19.934000000000001,"15min":29.882999999999999},"min":{"1min":8.9789999999999992,"5min":8.9789999999999992,"15min":7.2160000000000002},"max":{"1min":17.422000000000001,"5min":229.465,"15min":700.06500000000005},"last":11.632999999999999}]},{"osd":3,"last update":"Mon Jun 3 09:07:04 2024","interfaces":[{"interface":"back","average":{"1min":8.5999999999999996,"5min":14.451000000000001,"15min":25.416},"min":{"1min":6.2830000000000004,"5min":5.4640000000000004,"15min":5.4640000000000004},"max":{"1min":11.763,"5min":121.619,"15min":1676.009},"last":11.848000000000001},{"interface":"front","average":{"1min":9.5679999999999996,"5min":15.148,"15min":27.706},"min":{"1min":6.0709999999999997,"5min":5.7699999999999996,"15min":5.7699999999999996},"max":{"1min":17.129999999999999,"5min":132.036,"15min":1766.5840000000001},"last":9.5609999999999999}]},{"osd":4,"last update":"Mon Jun 3 09:07:37 2024","interfaces":[{"interface":"back","average":{"1min":10.769,"5min":13.884,"15min":333.529},"min":{"1min":7.8120000000000003,"5min":6.1760000000000002,"15min":6.0970000000000004},"max":{"1min":24.535,"5min":122.178,"15min":28044.862000000001},"last":13.231},{"interface":"front","average":{"1min":14.084,"5min":20.059999999999999,"15min":337.06599999999997},"min":{"1min":10.898,"5min":9.3940000000000001,"15min":6.4640000000000004},"max":{"1min":30.678000000000001,"5min":170.34999999999999,"15min":28045.624},"last":14.238}]},{"osd":5,"last update":"Mon Jun 3 09:07:04 2024","interfaces":[{"interface":"back","average":{"1min":12.039,"5min":19.311,"15min":24.228999999999999},"min":{"1min":7.9050000000000002,"5min":7.9050000000000002,"15min":7.9050000000000002},"max":{"1min":17.462,"5min":139.43199999999999,"15min":701.22000000000003},"last":15.891999999999999},{"interface":"front","average":{"1min":12.936,"5min":24.681999999999999,"15min":26.170000000000002},"min":{"1min":9.6969999999999992,"5min":7.258,"15min":7.2480000000000002},"max":{"1min":16.297999999999998,"5min":278.63,"15min":598.35000000000002},"last":12.765000000000001}]},{"osd":6,"last update":"Mon Jun 3 09:03:54 2024","interfaces":[{"interface":"back","average":{"1min":28.524000000000001,"5min":20.654,"15min":23.878},"min":{"1min":6.0309999999999997,"5min":6.0309999999999997,"15min":6.0309999999999997},"max":{"1min":167.221,"5min":167.221,"15min":704.245},"last":15.487},{"interface":"front","average":{"1min":27.606000000000002,"5min":30.131,"15min":28.628},"min":{"1min":9.8650000000000002,"5min":9.7200000000000006,"15min":8.3580000000000005},"max":{"1min":123.938,"5min":358.23000000000002,"15min":634.05499999999995},"last":18.117999999999999}]},{"osd":7,"last update":"Mon Jun 3 09:07:21 2024","interfaces":[{"interface":"back","average":{"1min":14.694000000000001,"5min":20.913,"15min":30.079000000000001},"min":{"1min":11.664999999999999,"5min":8.0559999999999992,"15min":8.0559999999999992},"max":{"1min":18.327000000000002,"5min":270.95100000000002,"15min":703.07799999999997},"last":16.611999999999998},{"interface":"front","average":{"1min":15.957000000000001,"5min":25.134,"15min":28.449000000000002},"min":{"1min":13.053000000000001,"5min":8.2089999999999996,"15min":7.7599999999999998},"max":{"1min":19.308,"5min":360.99900000000002,"15min":608.44100000000003},"last":17.280000000000001}]}]},{"osd":0,"up_from":14,"seq":60129542517,"num_pgs":10,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":81152,"kb_used_data":5752,"kb_used_omap":1,"kb_used_meta":75390,"kb_avail":94290688,"statfs":{"total":96636764160,"available":96553664512,"internally_reserved":0,"allocated":5890048,"data_stored":2763073,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1593,"internal_metadata":77199815},"hb_peers":[1,2,3,4,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[{"osd":1,"last update":"Mon Jun 3 09:07:05 2024","interfaces":[{"interface":"back","average":{"1min":7.46,"5min":56.137,"15min":335.75799999999998},"min":{"1min":4.6040000000000001,"5min":4.4530000000000003,"15min":4.4100000000000001},"max":{"1min":13.083,"5min":2414.4749999999999,"15min":20691.453000000001},"last":6.7329999999999997},{"interface":"front","average":{"1min":10.339,"5min":56.201999999999998,"15min":336.238},"min":{"1min":4.7199999999999998,"5min":4.7199999999999998,"15min":4.6050000000000004},"max":{"1min":25.372,"5min":2412.9549999999999,"15min":20640.792000000001},"last":5.8639999999999999}]},{"osd":2,"last update":"Mon Jun 3 09:07:22 2024","interfaces":[{"interface":"back","average":{"1min":9.5009999999999994,"5min":24.390999999999998,"15min":20.867000000000001},"min":{"1min":6.6139999999999999,"5min":5.766,"15min":5.766},"max":{"1min":22.283999999999999,"5min":627.40700000000004,"15min":627.40700000000004},"last":8.2230000000000008},{"interface":"front","average":{"1min":12.978999999999999,"5min":29.117999999999999,"15min":27.751999999999999},"min":{"1min":9.0449999999999999,"5min":7.8630000000000004,"15min":7.4089999999999998},"max":{"1min":29.343,"5min":635.62599999999998,"15min":635.62599999999998},"last":10.143000000000001}]},{"osd":3,"last update":"Mon Jun 3 09:07:05 2024","interfaces":[{"interface":"back","average":{"1min":12.941000000000001,"5min":26.172999999999998,"15min":24.896999999999998},"min":{"1min":6.8739999999999997,"5min":6.8739999999999997,"15min":6.1909999999999998},"max":{"1min":33.140000000000001,"5min":389.72699999999998,"15min":389.72699999999998},"last":10.787000000000001},{"interface":"front","average":{"1min":15.823,"5min":29.198,"15min":26.542999999999999},"min":{"1min":10.595000000000001,"5min":9.8070000000000004,"15min":6.1849999999999996},"max":{"1min":31.582999999999998,"5min":390.23200000000003,"15min":390.23200000000003},"last":12.941000000000001}]},{"osd":4,"last update":"Mon Jun 3 09:07:31 2024","interfaces":[{"interface":"back","average":{"1min":13.292999999999999,"5min":24.899000000000001,"15min":360.56400000000002},"min":{"1min":7.2590000000000003,"5min":6.415,"15min":6.415},"max":{"1min":26.516999999999999,"5min":558.01300000000003,"15min":25555.212},"last":14.917999999999999},{"interface":"front","average":{"1min":15.044,"5min":24.082000000000001,"15min":358.38499999999999},"min":{"1min":7.1669999999999998,"5min":7.1669999999999998,"15min":7.1429999999999998},"max":{"1min":27.510000000000002,"5min":390.791,"15min":25556.159},"last":13.51}]},{"osd":5,"last update":"Mon Jun 3 09:07:17 2024","interfaces":[{"interface":"back","average":{"1min":11.417999999999999,"5min":24.597000000000001,"15min":25.398},"min":{"1min":6.6539999999999999,"5min":6.6539999999999999,"15min":6.6539999999999999},"max":{"1min":25.068000000000001,"5min":455.78699999999998,"15min":455.78699999999998},"last":14.119},{"interface":"front","average":{"1min":16.913,"5min":28.289999999999999,"15min":23.193999999999999},"min":{"1min":10.563000000000001,"5min":9.2330000000000005,"15min":6.2030000000000003},"max":{"1min":29.972000000000001,"5min":391.31200000000001,"15min":391.31200000000001},"last":16.192}]},{"osd":6,"last update":"Mon Jun 3 09:04:30 2024","interfaces":[{"interface":"back","average":{"1min":52.496000000000002,"5min":33.139000000000003,"15min":160.83199999999999},"min":{"1min":7.5289999999999999,"5min":7.1040000000000001,"15min":6.3739999999999997},"max":{"1min":457.91699999999997,"5min":457.91699999999997,"15min":11115.258},"last":15.436},{"interface":"front","average":{"1min":49.670000000000002,"5min":33.704999999999998,"15min":162.791},"min":{"1min":12.247999999999999,"5min":11.019,"15min":9.9760000000000009},"max":{"1min":407.05099999999999,"5min":407.05099999999999,"15min":11092.194},"last":18.585999999999999}]},{"osd":7,"last update":"Mon Jun 3 09:07:23 2024","interfaces":[{"interface":"back","average":{"1min":12.662000000000001,"5min":23.452999999999999,"15min":25.288},"min":{"1min":7.1369999999999996,"5min":6.4480000000000004,"15min":6.4480000000000004},"max":{"1min":26.029,"5min":461.41699999999997,"15min":461.41699999999997},"last":21.033000000000001},{"interface":"front","average":{"1min":14.715,"5min":26.327000000000002,"15min":30.222000000000001},"min":{"1min":8.7810000000000006,"5min":8.7810000000000006,"15min":8.7810000000000006},"max":{"1min":30.530000000000001,"5min":463.10599999999999,"15min":463.10599999999999},"last":21.856000000000002}]}]}],"pool_statfs":[{"poolid":1,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":401408,"data_stored":397840,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":5,"total":0,"available":0,"internally_reserved":0,"allocated":401408,"data_stored":397840,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":4,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":5,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":6,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":17,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":17,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":17,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":17,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":17,"osd":4,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":17,"osd":5,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":5,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":17,"osd":6,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":5,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":17,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0}]}} 2024-06-03T09:07:44.309 DEBUG:teuthology.orchestra.run.smithi088:> sudo ceph --cluster ceph config set global mon_health_to_clog false 2024-06-03T09:07:44.406 INFO:teuthology.orchestra.run.smithi088.stderr:2024-06-03T09:07:44.406+0000 7ff21ee3b640 10 monclient: get_monmap_and_config 2024-06-03T09:07:44.407 INFO:teuthology.orchestra.run.smithi088.stderr:2024-06-03T09:07:44.406+0000 7ff21ee3b640 10 monclient: build_initial_monmap 2024-06-03T09:07:44.407 INFO:teuthology.orchestra.run.smithi088.stderr:2024-06-03T09:07:44.406+0000 7ff21ee3b640 10 monclient: monmap: 2024-06-03T09:07:44.407 INFO:teuthology.orchestra.run.smithi088.stderr:epoch 0 2024-06-03T09:07:44.407 INFO:teuthology.orchestra.run.smithi088.stderr:fsid ec3c3fdd-3ac6-4d7d-9412-683130ed8b04 2024-06-03T09:07:44.407 INFO:teuthology.orchestra.run.smithi088.stderr:last_changed 2024-06-03T09:07:44.407775+0000 2024-06-03T09:07:44.407 INFO:teuthology.orchestra.run.smithi088.stderr:created 2024-06-03T09:07:44.407775+0000 2024-06-03T09:07:44.407 INFO:teuthology.orchestra.run.smithi088.stderr:min_mon_release 0 (unknown) 2024-06-03T09:07:44.407 INFO:teuthology.orchestra.run.smithi088.stderr:election_strategy: 3 2024-06-03T09:07:44.407 INFO:teuthology.orchestra.run.smithi088.stderr:0: v1:172.21.15.88:6789/0 mon.noname-a 2024-06-03T09:07:44.407 INFO:teuthology.orchestra.run.smithi088.stderr:1: v1:172.21.15.192:6789/0 mon.noname-c 2024-06-03T09:07:44.407 INFO:teuthology.orchestra.run.smithi088.stderr:2: v1:172.21.15.88:6790/0 mon.noname-b 2024-06-03T09:07:44.407 INFO:teuthology.orchestra.run.smithi088.stderr: 2024-06-03T09:07:44.409 INFO:teuthology.orchestra.run.smithi088.stderr:2024-06-03T09:07:44.407+0000 7ff21ee3b640 1 Processor -- start 2024-06-03T09:07:44.409 INFO:teuthology.orchestra.run.smithi088.stderr:2024-06-03T09:07:44.407+0000 7ff21ee3b640 1 -- start start 2024-06-03T09:07:44.409 INFO:teuthology.orchestra.run.smithi088.stderr:2024-06-03T09:07:44.407+0000 7ff21ee3b640 10 monclient: init 2024-06-03T09:07:44.409 INFO:teuthology.orchestra.run.smithi088.stderr:2024-06-03T09:07:44.408+0000 7ff21ee3b640 10 monclient: _reopen_session rank -1 2024-06-03T09:07:44.409 INFO:teuthology.orchestra.run.smithi088.stderr:2024-06-03T09:07:44.408+0000 7ff21ee3b640 10 monclient: _add_conns ranks=[1,0,2] 2024-06-03T09:07:44.409 INFO:teuthology.orchestra.run.smithi088.stderr:2024-06-03T09:07:44.408+0000 7ff21ee3b640 10 monclient(hunting): picked mon.noname-c con 0x7ff2180adb90 addr v1:172.21.15.192:6789/0 2024-06-03T09:07:44.409 INFO:teuthology.orchestra.run.smithi088.stderr:2024-06-03T09:07:44.408+0000 7ff21ee3b640 10 monclient(hunting): picked mon.noname-a con 0x7ff21806ee00 addr v1:172.21.15.88:6789/0 2024-06-03T09:07:44.409 INFO:teuthology.orchestra.run.smithi088.stderr:2024-06-03T09:07:44.408+0000 7ff21ee3b640 10 monclient(hunting): picked mon.noname-b con 0x7ff2181383e0 addr v1:172.21.15.88:6790/0 2024-06-03T09:07:44.409 INFO:teuthology.orchestra.run.smithi088.stderr:2024-06-03T09:07:44.408+0000 7ff21ee3b640 1 -- --> v1:172.21.15.88:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7ff21806f2c0 con 0x7ff21806ee00 2024-06-03T09:07:44.409 INFO:teuthology.orchestra.run.smithi088.stderr:2024-06-03T09:07:44.408+0000 7ff21ee3b640 1 -- --> v1:172.21.15.192:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7ff2180710c0 con 0x7ff2180adb90 2024-06-03T09:07:44.409 INFO:teuthology.orchestra.run.smithi088.stderr:2024-06-03T09:07:44.408+0000 7ff21ee3b640 1 -- --> v1:172.21.15.88:6790/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7ff218071fd0 con 0x7ff2181383e0 2024-06-03T09:07:44.409 INFO:teuthology.orchestra.run.smithi088.stderr:2024-06-03T09:07:44.408+0000 7ff21ee3b640 10 monclient(hunting): _renew_subs 2024-06-03T09:07:44.410 INFO:teuthology.orchestra.run.smithi088.stderr:2024-06-03T09:07:44.408+0000 7ff21ee3b640 10 monclient(hunting): authenticate will time out at 3120.218075s 2024-06-03T09:07:44.410 INFO:teuthology.orchestra.run.smithi088.stderr:2024-06-03T09:07:44.408+0000 7ff217fff640 1 --1- >> v1:172.21.15.88:6789/0 conn(0x7ff21806ee00 0x7ff218137cd0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.88:6789/0 says I am v1:172.21.15.88:43292/0 (socket says 172.21.15.88:43292) 2024-06-03T09:07:44.410 INFO:teuthology.orchestra.run.smithi088.stderr:2024-06-03T09:07:44.408+0000 7ff217fff640 1 -- 172.21.15.88:0/3058611720 learned_addr learned my addr 172.21.15.88:0/3058611720 (peer_addr_for_me v1:172.21.15.88:0/0) 2024-06-03T09:07:44.411 INFO:teuthology.orchestra.run.smithi088.stderr:2024-06-03T09:07:44.411+0000 7ff2177fe640 1 -- 172.21.15.88:0/3058611720 <== mon.1 v1:172.21.15.192:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 1397823281 0 0) 0x7ff2180710c0 con 0x7ff2180adb90 2024-06-03T09:07:44.411 INFO:teuthology.orchestra.run.smithi088.stderr:2024-06-03T09:07:44.411+0000 7ff2177fe640 10 monclient(hunting): _negotiate 2024-06-03T09:07:44.412 INFO:teuthology.orchestra.run.smithi088.stderr:2024-06-03T09:07:44.411+0000 7ff2177fe640 10 monclient(hunting): _init_auth method 2 2024-06-03T09:07:44.412 INFO:teuthology.orchestra.run.smithi088.stderr:2024-06-03T09:07:44.411+0000 7ff2177fe640 10 monclient(hunting): _init_auth creating new auth 2024-06-03T09:07:44.412 INFO:teuthology.orchestra.run.smithi088.stderr:2024-06-03T09:07:44.411+0000 7ff2177fe640 10 monclient(hunting): my global_id is 30418 2024-06-03T09:07:44.412 INFO:teuthology.orchestra.run.smithi088.stderr:2024-06-03T09:07:44.411+0000 7ff2177fe640 1 -- 172.21.15.88:0/3058611720 --> v1:172.21.15.192:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7ff208003520 con 0x7ff2180adb90 2024-06-03T09:07:44.412 INFO:teuthology.orchestra.run.smithi088.stderr:2024-06-03T09:07:44.411+0000 7ff2177fe640 1 -- 172.21.15.88:0/3058611720 <== mon.2 v1:172.21.15.88:6790/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 3771711262 0 0) 0x7ff218071fd0 con 0x7ff2181383e0 2024-06-03T09:07:44.412 INFO:teuthology.orchestra.run.smithi088.stderr:2024-06-03T09:07:44.411+0000 7ff2177fe640 10 monclient(hunting): _negotiate 2024-06-03T09:07:44.412 INFO:teuthology.orchestra.run.smithi088.stderr:2024-06-03T09:07:44.411+0000 7ff2177fe640 10 monclient(hunting): _init_auth method 2 2024-06-03T09:07:44.412 INFO:teuthology.orchestra.run.smithi088.stderr:2024-06-03T09:07:44.411+0000 7ff2177fe640 10 monclient(hunting): _init_auth creating new auth 2024-06-03T09:07:44.412 INFO:teuthology.orchestra.run.smithi088.stderr:2024-06-03T09:07:44.411+0000 7ff2177fe640 10 monclient(hunting): my global_id is 30353 2024-06-03T09:07:44.412 INFO:teuthology.orchestra.run.smithi088.stderr:2024-06-03T09:07:44.411+0000 7ff2177fe640 1 -- 172.21.15.88:0/3058611720 --> v1:172.21.15.88:6790/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7ff2180710c0 con 0x7ff2181383e0 2024-06-03T09:07:44.412 INFO:teuthology.orchestra.run.smithi088.stderr:2024-06-03T09:07:44.411+0000 7ff2177fe640 1 -- 172.21.15.88:0/3058611720 <== mon.0 v1:172.21.15.88:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 740103368 0 0) 0x7ff21806f2c0 con 0x7ff21806ee00 2024-06-03T09:07:44.412 INFO:teuthology.orchestra.run.smithi088.stderr:2024-06-03T09:07:44.411+0000 7ff2177fe640 10 monclient(hunting): _negotiate 2024-06-03T09:07:44.412 INFO:teuthology.orchestra.run.smithi088.stderr:2024-06-03T09:07:44.411+0000 7ff2177fe640 10 monclient(hunting): _init_auth method 2 2024-06-03T09:07:44.412 INFO:teuthology.orchestra.run.smithi088.stderr:2024-06-03T09:07:44.411+0000 7ff2177fe640 10 monclient(hunting): _init_auth creating new auth 2024-06-03T09:07:44.413 INFO:teuthology.orchestra.run.smithi088.stderr:2024-06-03T09:07:44.411+0000 7ff2177fe640 10 monclient(hunting): my global_id is 30843 2024-06-03T09:07:44.413 INFO:teuthology.orchestra.run.smithi088.stderr:2024-06-03T09:07:44.411+0000 7ff2177fe640 1 -- 172.21.15.88:0/3058611720 --> v1:172.21.15.88:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7ff218071fd0 con 0x7ff21806ee00 2024-06-03T09:07:44.414 INFO:teuthology.orchestra.run.smithi088.stderr:2024-06-03T09:07:44.414+0000 7ff2177fe640 1 -- 172.21.15.88:0/3058611720 <== mon.1 v1:172.21.15.192:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 214+0+0 (unknown 2923051338 0 0) 0x7ff208003520 con 0x7ff2180adb90 2024-06-03T09:07:44.414 INFO:teuthology.orchestra.run.smithi088.stderr:2024-06-03T09:07:44.414+0000 7ff2177fe640 1 -- 172.21.15.88:0/3058611720 >> v1:172.21.15.88:6790/0 conn(0x7ff2181383e0 legacy=0x7ff218149af0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-06-03T09:07:44.415 INFO:teuthology.orchestra.run.smithi088.stderr:2024-06-03T09:07:44.414+0000 7ff2177fe640 1 -- 172.21.15.88:0/3058611720 >> v1:172.21.15.88:6789/0 conn(0x7ff21806ee00 legacy=0x7ff218137cd0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-06-03T09:07:44.415 INFO:teuthology.orchestra.run.smithi088.stderr:2024-06-03T09:07:44.414+0000 7ff2177fe640 10 monclient: _finish_hunting 0 2024-06-03T09:07:44.415 INFO:teuthology.orchestra.run.smithi088.stderr:2024-06-03T09:07:44.414+0000 7ff2177fe640 1 monclient: found mon.noname-c 2024-06-03T09:07:44.415 INFO:teuthology.orchestra.run.smithi088.stderr:2024-06-03T09:07:44.414+0000 7ff2177fe640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-06-03T09:07:44.415 INFO:teuthology.orchestra.run.smithi088.stderr:2024-06-03T09:07:44.414+0000 7ff2177fe640 10 monclient: _send_mon_message to mon.noname-c at v1:172.21.15.192:6789/0 2024-06-03T09:07:44.415 INFO:teuthology.orchestra.run.smithi088.stderr:2024-06-03T09:07:44.414+0000 7ff2177fe640 1 -- 172.21.15.88:0/3058611720 --> v1:172.21.15.192:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7ff218141750 con 0x7ff2180adb90 2024-06-03T09:07:44.415 INFO:teuthology.orchestra.run.smithi088.stderr:2024-06-03T09:07:44.414+0000 7ff2177fe640 10 monclient: _finish_auth 0 2024-06-03T09:07:44.415 INFO:teuthology.orchestra.run.smithi088.stderr:2024-06-03T09:07:44.414+0000 7ff2177fe640 10 monclient: _check_auth_tickets 2024-06-03T09:07:44.415 INFO:teuthology.orchestra.run.smithi088.stderr:2024-06-03T09:07:44.414+0000 7ff2177fe640 20 monclient: _check_auth_rotating not needed by client.admin 2024-06-03T09:07:44.415 INFO:teuthology.orchestra.run.smithi088.stderr:2024-06-03T09:07:44.414+0000 7ff2177fe640 1 -- 172.21.15.88:0/3058611720 <== mon.1 v1:172.21.15.192:6789/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 2234374065 0 0) 0x7ff20c001610 con 0x7ff2180adb90 2024-06-03T09:07:44.415 INFO:teuthology.orchestra.run.smithi088.stderr:2024-06-03T09:07:44.414+0000 7ff2177fe640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-06-03T09:07:44.415 INFO:teuthology.orchestra.run.smithi088.stderr:2024-06-03T09:07:44.414+0000 7ff2177fe640 10 monclient: got monmap 1 from mon.noname-c (according to old e1) 2024-06-03T09:07:44.415 INFO:teuthology.orchestra.run.smithi088.stderr:2024-06-03T09:07:44.414+0000 7ff2177fe640 10 monclient: dump: 2024-06-03T09:07:44.416 INFO:teuthology.orchestra.run.smithi088.stderr:epoch 1 2024-06-03T09:07:44.416 INFO:teuthology.orchestra.run.smithi088.stderr:fsid ec3c3fdd-3ac6-4d7d-9412-683130ed8b04 2024-06-03T09:07:44.416 INFO:teuthology.orchestra.run.smithi088.stderr:last_changed 2024-06-03T08:34:31.885125+0000 2024-06-03T09:07:44.416 INFO:teuthology.orchestra.run.smithi088.stderr:created 2024-06-03T08:34:31.885125+0000 2024-06-03T09:07:44.416 INFO:teuthology.orchestra.run.smithi088.stderr:min_mon_release 18 (reef) 2024-06-03T09:07:44.416 INFO:teuthology.orchestra.run.smithi088.stderr:election_strategy: 3 2024-06-03T09:07:44.416 INFO:teuthology.orchestra.run.smithi088.stderr:0: v1:172.21.15.88:6789/0 mon.a 2024-06-03T09:07:44.416 INFO:teuthology.orchestra.run.smithi088.stderr:1: v1:172.21.15.192:6789/0 mon.b 2024-06-03T09:07:44.416 INFO:teuthology.orchestra.run.smithi088.stderr:2: v1:172.21.15.88:6790/0 mon.c 2024-06-03T09:07:44.416 INFO:teuthology.orchestra.run.smithi088.stderr: 2024-06-03T09:07:44.416 INFO:teuthology.orchestra.run.smithi088.stderr:2024-06-03T09:07:44.414+0000 7ff21ee3b640 5 monclient: authenticate success, global_id 30418 2024-06-03T09:07:44.416 INFO:teuthology.orchestra.run.smithi088.stderr:2024-06-03T09:07:44.414+0000 7ff21ee3b640 20 monclient: get_monmap_and_config waiting for monmap|config 2024-06-03T09:07:44.417 INFO:teuthology.orchestra.run.smithi088.stderr:2024-06-03T09:07:44.416+0000 7ff2177fe640 1 -- 172.21.15.88:0/3058611720 <== mon.1 v1:172.21.15.192:6789/0 4 ==== config(0 keys) v1 ==== 4+0+0 (unknown 0 0 0) 0x7ff20c001830 con 0x7ff2180adb90 2024-06-03T09:07:44.417 INFO:teuthology.orchestra.run.smithi088.stderr:2024-06-03T09:07:44.416+0000 7ff2177fe640 10 monclient: handle_config config(0 keys) v1 2024-06-03T09:07:44.418 INFO:teuthology.orchestra.run.smithi088.stderr:2024-06-03T09:07:44.416+0000 7ff2177fe640 1 -- 172.21.15.88:0/3058611720 <== mon.1 v1:172.21.15.192:6789/0 5 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 2234374065 0 0) 0x7ff20c001bc0 con 0x7ff2180adb90 2024-06-03T09:07:44.418 INFO:teuthology.orchestra.run.smithi088.stderr:2024-06-03T09:07:44.416+0000 7ff2177fe640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-06-03T09:07:44.418 INFO:teuthology.orchestra.run.smithi088.stderr:2024-06-03T09:07:44.416+0000 7ff2177fe640 10 monclient: got monmap 1 from mon.b (according to old e1) 2024-06-03T09:07:44.418 INFO:teuthology.orchestra.run.smithi088.stderr:2024-06-03T09:07:44.416+0000 7ff2177fe640 10 monclient: dump: 2024-06-03T09:07:44.418 INFO:teuthology.orchestra.run.smithi088.stderr:epoch 1 2024-06-03T09:07:44.418 INFO:teuthology.orchestra.run.smithi088.stderr:fsid ec3c3fdd-3ac6-4d7d-9412-683130ed8b04 2024-06-03T09:07:44.418 INFO:teuthology.orchestra.run.smithi088.stderr:last_changed 2024-06-03T08:34:31.885125+0000 2024-06-03T09:07:44.418 INFO:teuthology.orchestra.run.smithi088.stderr:created 2024-06-03T08:34:31.885125+0000 2024-06-03T09:07:44.418 INFO:teuthology.orchestra.run.smithi088.stderr:min_mon_release 18 (reef) 2024-06-03T09:07:44.418 INFO:teuthology.orchestra.run.smithi088.stderr:election_strategy: 3 2024-06-03T09:07:44.418 INFO:teuthology.orchestra.run.smithi088.stderr:0: v1:172.21.15.88:6789/0 mon.a 2024-06-03T09:07:44.418 INFO:teuthology.orchestra.run.smithi088.stderr:1: v1:172.21.15.192:6789/0 mon.b 2024-06-03T09:07:44.418 INFO:teuthology.orchestra.run.smithi088.stderr:2: v1:172.21.15.88:6790/0 mon.c 2024-06-03T09:07:44.418 INFO:teuthology.orchestra.run.smithi088.stderr: 2024-06-03T09:07:44.419 INFO:teuthology.orchestra.run.smithi088.stderr:2024-06-03T09:07:44.416+0000 7ff21ee3b640 10 monclient: get_monmap_and_config success 2024-06-03T09:07:44.419 INFO:teuthology.orchestra.run.smithi088.stderr:2024-06-03T09:07:44.416+0000 7ff21ee3b640 4 set_mon_vals no callback set 2024-06-03T09:07:44.419 INFO:teuthology.orchestra.run.smithi088.stderr:2024-06-03T09:07:44.417+0000 7ff21ee3b640 10 monclient: shutdown 2024-06-03T09:07:44.419 INFO:teuthology.orchestra.run.smithi088.stderr:2024-06-03T09:07:44.417+0000 7ff21ee3b640 20 monclient: shutdown discarding 0 pending message(s) 2024-06-03T09:07:44.419 INFO:teuthology.orchestra.run.smithi088.stderr:2024-06-03T09:07:44.417+0000 7ff21ee3b640 1 -- 172.21.15.88:0/3058611720 >> v1:172.21.15.192:6789/0 conn(0x7ff2180adb90 legacy=0x7ff2180718c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-06-03T09:07:44.419 INFO:teuthology.orchestra.run.smithi088.stderr:2024-06-03T09:07:44.417+0000 7ff21ee3b640 1 -- 172.21.15.88:0/3058611720 shutdown_connections 2024-06-03T09:07:44.419 INFO:teuthology.orchestra.run.smithi088.stderr:2024-06-03T09:07:44.417+0000 7ff21ee3b640 1 -- 172.21.15.88:0/3058611720 >> 172.21.15.88:0/3058611720 conn(0x7ff2180b2f20 msgr2=0x7ff2180b3300 unknown :-1 s=STATE_NONE l=0).mark_down 2024-06-03T09:07:44.419 INFO:teuthology.orchestra.run.smithi088.stderr:2024-06-03T09:07:44.417+0000 7ff21ee3b640 1 -- 172.21.15.88:0/3058611720 shutdown_connections 2024-06-03T09:07:44.419 INFO:teuthology.orchestra.run.smithi088.stderr:2024-06-03T09:07:44.417+0000 7ff21ee3b640 1 -- 172.21.15.88:0/3058611720 wait complete. 2024-06-03T09:07:44.421 INFO:teuthology.orchestra.run.smithi088.stderr:2024-06-03T09:07:44.418+0000 7ff21ee3b640 10 monclient: build_initial_monmap 2024-06-03T09:07:44.421 INFO:teuthology.orchestra.run.smithi088.stderr:2024-06-03T09:07:44.418+0000 7ff21ee3b640 10 monclient: monmap: 2024-06-03T09:07:44.421 INFO:teuthology.orchestra.run.smithi088.stderr:epoch 0 2024-06-03T09:07:44.421 INFO:teuthology.orchestra.run.smithi088.stderr:fsid 00000000-0000-0000-0000-000000000000 2024-06-03T09:07:44.421 INFO:teuthology.orchestra.run.smithi088.stderr:last_changed 0.000000 2024-06-03T09:07:44.421 INFO:teuthology.orchestra.run.smithi088.stderr:created 0.000000 2024-06-03T09:07:44.421 INFO:teuthology.orchestra.run.smithi088.stderr:min_mon_release 0 (unknown) 2024-06-03T09:07:44.422 INFO:teuthology.orchestra.run.smithi088.stderr:election_strategy: 1 2024-06-03T09:07:44.422 INFO:teuthology.orchestra.run.smithi088.stderr:0: v1:172.21.15.88:6789/0 mon.noname-a 2024-06-03T09:07:44.422 INFO:teuthology.orchestra.run.smithi088.stderr:1: v1:172.21.15.192:6789/0 mon.noname-b 2024-06-03T09:07:44.422 INFO:teuthology.orchestra.run.smithi088.stderr:2: v1:172.21.15.88:6790/0 mon.noname-c 2024-06-03T09:07:44.422 INFO:teuthology.orchestra.run.smithi088.stderr: 2024-06-03T09:07:44.422 INFO:teuthology.orchestra.run.smithi088.stderr:2024-06-03T09:07:44.418+0000 7ff21ee3b640 1 librados: starting msgr at 2024-06-03T09:07:44.422 INFO:teuthology.orchestra.run.smithi088.stderr:2024-06-03T09:07:44.418+0000 7ff21ee3b640 1 librados: starting objecter 2024-06-03T09:07:44.422 INFO:teuthology.orchestra.run.smithi088.stderr:2024-06-03T09:07:44.418+0000 7ff21ee3b640 1 Processor -- start 2024-06-03T09:07:44.422 INFO:teuthology.orchestra.run.smithi088.stderr:2024-06-03T09:07:44.418+0000 7ff21ee3b640 1 -- start start 2024-06-03T09:07:44.422 INFO:teuthology.orchestra.run.smithi088.stderr:2024-06-03T09:07:44.418+0000 7ff21ee3b640 1 librados: setting wanted keys 2024-06-03T09:07:44.422 INFO:teuthology.orchestra.run.smithi088.stderr:2024-06-03T09:07:44.418+0000 7ff21ee3b640 1 librados: calling monclient init 2024-06-03T09:07:44.422 INFO:teuthology.orchestra.run.smithi088.stderr:2024-06-03T09:07:44.418+0000 7ff21ee3b640 10 monclient: init 2024-06-03T09:07:44.423 INFO:teuthology.orchestra.run.smithi088.stderr:2024-06-03T09:07:44.419+0000 7ff21ee3b640 10 monclient: _reopen_session rank -1 2024-06-03T09:07:44.423 INFO:teuthology.orchestra.run.smithi088.stderr:2024-06-03T09:07:44.419+0000 7ff21ee3b640 10 monclient: _add_conns ranks=[2,1,0] 2024-06-03T09:07:44.423 INFO:teuthology.orchestra.run.smithi088.stderr:2024-06-03T09:07:44.419+0000 7ff21ee3b640 10 monclient(hunting): picked mon.noname-c con 0x7ff21806ee00 addr v1:172.21.15.88:6790/0 2024-06-03T09:07:44.423 INFO:teuthology.orchestra.run.smithi088.stderr:2024-06-03T09:07:44.419+0000 7ff21ee3b640 10 monclient(hunting): picked mon.noname-b con 0x7ff2180adb90 addr v1:172.21.15.192:6789/0 2024-06-03T09:07:44.423 INFO:teuthology.orchestra.run.smithi088.stderr:2024-06-03T09:07:44.419+0000 7ff21ee3b640 10 monclient(hunting): picked mon.noname-a con 0x7ff2181383e0 addr v1:172.21.15.88:6789/0 2024-06-03T09:07:44.423 INFO:teuthology.orchestra.run.smithi088.stderr:2024-06-03T09:07:44.419+0000 7ff21ee3b640 1 -- --> v1:172.21.15.88:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7ff218133ce0 con 0x7ff2181383e0 2024-06-03T09:07:44.423 INFO:teuthology.orchestra.run.smithi088.stderr:2024-06-03T09:07:44.419+0000 7ff21ee3b640 1 -- --> v1:172.21.15.192:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7ff218132f50 con 0x7ff2180adb90 2024-06-03T09:07:44.423 INFO:teuthology.orchestra.run.smithi088.stderr:2024-06-03T09:07:44.419+0000 7ff21ee3b640 1 -- --> v1:172.21.15.88:6790/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7ff2180aab00 con 0x7ff21806ee00 2024-06-03T09:07:44.424 INFO:teuthology.orchestra.run.smithi088.stderr:2024-06-03T09:07:44.419+0000 7ff21ee3b640 10 monclient(hunting): _renew_subs 2024-06-03T09:07:44.424 INFO:teuthology.orchestra.run.smithi088.stderr:2024-06-03T09:07:44.419+0000 7ff21ee3b640 10 monclient(hunting): authenticate will time out at 3120.229424s 2024-06-03T09:07:44.424 INFO:teuthology.orchestra.run.smithi088.stderr:2024-06-03T09:07:44.419+0000 7ff21d3d8640 1 --1- >> v1:172.21.15.88:6789/0 conn(0x7ff2181383e0 0x7ff218132840 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=1).handle_server_banner_and_identify peer v1:172.21.15.88:6789/0 says I am v1:172.21.15.88:43308/0 (socket says 172.21.15.88:43308) 2024-06-03T09:07:44.424 INFO:teuthology.orchestra.run.smithi088.stderr:2024-06-03T09:07:44.419+0000 7ff21d3d8640 1 -- 172.21.15.88:0/402045423 learned_addr learned my addr 172.21.15.88:0/402045423 (peer_addr_for_me v1:172.21.15.88:0/0) 2024-06-03T09:07:44.424 INFO:teuthology.orchestra.run.smithi088.stderr:2024-06-03T09:07:44.420+0000 7ff2157fa640 10 client.?.objecter ms_handle_connect 0x7ff2181383e0 2024-06-03T09:07:44.424 INFO:teuthology.orchestra.run.smithi088.stderr:2024-06-03T09:07:44.420+0000 7ff2157fa640 10 client.?.objecter resend_mon_ops 2024-06-03T09:07:44.424 INFO:teuthology.orchestra.run.smithi088.stderr:2024-06-03T09:07:44.420+0000 7ff2157fa640 10 client.?.objecter ms_handle_connect 0x7ff21806ee00 2024-06-03T09:07:44.424 INFO:teuthology.orchestra.run.smithi088.stderr:2024-06-03T09:07:44.420+0000 7ff2157fa640 10 client.?.objecter resend_mon_ops 2024-06-03T09:07:44.424 INFO:teuthology.orchestra.run.smithi088.stderr:2024-06-03T09:07:44.420+0000 7ff2157fa640 10 client.?.objecter ms_handle_connect 0x7ff2180adb90 2024-06-03T09:07:44.424 INFO:teuthology.orchestra.run.smithi088.stderr:2024-06-03T09:07:44.420+0000 7ff2157fa640 10 client.?.objecter resend_mon_ops 2024-06-03T09:07:44.425 INFO:teuthology.orchestra.run.smithi088.stderr:2024-06-03T09:07:44.422+0000 7ff2157fa640 1 -- 172.21.15.88:0/402045423 <== mon.2 v1:172.21.15.88:6790/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 2573175571 0 0) 0x7ff2180aab00 con 0x7ff21806ee00 2024-06-03T09:07:44.425 INFO:teuthology.orchestra.run.smithi088.stderr:2024-06-03T09:07:44.422+0000 7ff2157fa640 10 monclient(hunting): _negotiate 2024-06-03T09:07:44.426 INFO:teuthology.orchestra.run.smithi088.stderr:2024-06-03T09:07:44.422+0000 7ff2157fa640 10 monclient(hunting): _init_auth method 2 2024-06-03T09:07:44.426 INFO:teuthology.orchestra.run.smithi088.stderr:2024-06-03T09:07:44.422+0000 7ff2157fa640 10 monclient(hunting): _init_auth creating new auth 2024-06-03T09:07:44.426 INFO:teuthology.orchestra.run.smithi088.stderr:2024-06-03T09:07:44.422+0000 7ff2157fa640 10 monclient(hunting): my global_id is 30356 2024-06-03T09:07:44.426 INFO:teuthology.orchestra.run.smithi088.stderr:2024-06-03T09:07:44.422+0000 7ff2157fa640 1 -- 172.21.15.88:0/402045423 --> v1:172.21.15.88:6790/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7ff1fc004eb0 con 0x7ff21806ee00 2024-06-03T09:07:44.426 INFO:teuthology.orchestra.run.smithi088.stderr:2024-06-03T09:07:44.422+0000 7ff2157fa640 1 -- 172.21.15.88:0/402045423 <== mon.1 v1:172.21.15.192:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 1267371913 0 0) 0x7ff218132f50 con 0x7ff2180adb90 2024-06-03T09:07:44.426 INFO:teuthology.orchestra.run.smithi088.stderr:2024-06-03T09:07:44.422+0000 7ff2157fa640 10 monclient(hunting): _negotiate 2024-06-03T09:07:44.426 INFO:teuthology.orchestra.run.smithi088.stderr:2024-06-03T09:07:44.422+0000 7ff2157fa640 10 monclient(hunting): _init_auth method 2 2024-06-03T09:07:44.426 INFO:teuthology.orchestra.run.smithi088.stderr:2024-06-03T09:07:44.422+0000 7ff2157fa640 10 monclient(hunting): _init_auth creating new auth 2024-06-03T09:07:44.426 INFO:teuthology.orchestra.run.smithi088.stderr:2024-06-03T09:07:44.422+0000 7ff2157fa640 10 monclient(hunting): my global_id is 30421 2024-06-03T09:07:44.426 INFO:teuthology.orchestra.run.smithi088.stderr:2024-06-03T09:07:44.422+0000 7ff2157fa640 1 -- 172.21.15.88:0/402045423 --> v1:172.21.15.192:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7ff2180aab00 con 0x7ff2180adb90 2024-06-03T09:07:44.426 INFO:teuthology.orchestra.run.smithi088.stderr:2024-06-03T09:07:44.422+0000 7ff2157fa640 1 -- 172.21.15.88:0/402045423 <== mon.0 v1:172.21.15.88:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 3920115220 0 0) 0x7ff218133ce0 con 0x7ff2181383e0 2024-06-03T09:07:44.426 INFO:teuthology.orchestra.run.smithi088.stderr:2024-06-03T09:07:44.422+0000 7ff2157fa640 10 monclient(hunting): _negotiate 2024-06-03T09:07:44.427 INFO:teuthology.orchestra.run.smithi088.stderr:2024-06-03T09:07:44.422+0000 7ff2157fa640 10 monclient(hunting): _init_auth method 2 2024-06-03T09:07:44.427 INFO:teuthology.orchestra.run.smithi088.stderr:2024-06-03T09:07:44.422+0000 7ff2157fa640 10 monclient(hunting): _init_auth creating new auth 2024-06-03T09:07:44.427 INFO:teuthology.orchestra.run.smithi088.stderr:2024-06-03T09:07:44.422+0000 7ff2157fa640 10 monclient(hunting): my global_id is 30846 2024-06-03T09:07:44.427 INFO:teuthology.orchestra.run.smithi088.stderr:2024-06-03T09:07:44.422+0000 7ff2157fa640 1 -- 172.21.15.88:0/402045423 --> v1:172.21.15.88:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7ff218132f50 con 0x7ff2181383e0 2024-06-03T09:07:44.428 INFO:teuthology.orchestra.run.smithi088.stderr:2024-06-03T09:07:44.427+0000 7ff2157fa640 1 -- 172.21.15.88:0/402045423 <== mon.2 v1:172.21.15.88:6790/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 764+0+0 (unknown 2972135207 0 0) 0x7ff1fc004eb0 con 0x7ff21806ee00 2024-06-03T09:07:44.428 INFO:teuthology.orchestra.run.smithi088.stderr:2024-06-03T09:07:44.427+0000 7ff2157fa640 1 -- 172.21.15.88:0/402045423 --> v1:172.21.15.88:6790/0 -- auth(proto 2 165 bytes epoch 0) v1 -- 0x7ff218133ce0 con 0x7ff21806ee00 2024-06-03T09:07:44.428 INFO:teuthology.orchestra.run.smithi088.stderr:2024-06-03T09:07:44.427+0000 7ff2157fa640 1 -- 172.21.15.88:0/402045423 <== mon.1 v1:172.21.15.192:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 764+0+0 (unknown 1735940544 0 0) 0x7ff2180aab00 con 0x7ff2180adb90 2024-06-03T09:07:44.428 INFO:teuthology.orchestra.run.smithi088.stderr:2024-06-03T09:07:44.427+0000 7ff2157fa640 1 -- 172.21.15.88:0/402045423 --> v1:172.21.15.192:6789/0 -- auth(proto 2 165 bytes epoch 0) v1 -- 0x7ff1fc004eb0 con 0x7ff2180adb90 2024-06-03T09:07:44.428 INFO:teuthology.orchestra.run.smithi088.stderr:2024-06-03T09:07:44.427+0000 7ff2157fa640 1 -- 172.21.15.88:0/402045423 <== mon.2 v1:172.21.15.88:6790/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 2234374065 0 0) 0x7ff20c001b40 con 0x7ff21806ee00 2024-06-03T09:07:44.429 INFO:teuthology.orchestra.run.smithi088.stderr:2024-06-03T09:07:44.427+0000 7ff2157fa640 10 monclient(hunting): handle_monmap mon_map magic: 0 v1 2024-06-03T09:07:44.429 INFO:teuthology.orchestra.run.smithi088.stderr:2024-06-03T09:07:44.427+0000 7ff2157fa640 10 monclient(hunting): got monmap 1 from mon.noname-c (according to old e1) 2024-06-03T09:07:44.429 INFO:teuthology.orchestra.run.smithi088.stderr:2024-06-03T09:07:44.427+0000 7ff2157fa640 10 monclient(hunting): dump: 2024-06-03T09:07:44.429 INFO:teuthology.orchestra.run.smithi088.stderr:epoch 1 2024-06-03T09:07:44.430 INFO:teuthology.orchestra.run.smithi088.stderr:fsid ec3c3fdd-3ac6-4d7d-9412-683130ed8b04 2024-06-03T09:07:44.430 INFO:teuthology.orchestra.run.smithi088.stderr:last_changed 2024-06-03T08:34:31.885125+0000 2024-06-03T09:07:44.430 INFO:teuthology.orchestra.run.smithi088.stderr:created 2024-06-03T08:34:31.885125+0000 2024-06-03T09:07:44.430 INFO:teuthology.orchestra.run.smithi088.stderr:min_mon_release 18 (reef) 2024-06-03T09:07:44.430 INFO:teuthology.orchestra.run.smithi088.stderr:election_strategy: 3 2024-06-03T09:07:44.430 INFO:teuthology.orchestra.run.smithi088.stderr:0: v1:172.21.15.88:6789/0 mon.a 2024-06-03T09:07:44.431 INFO:teuthology.orchestra.run.smithi088.stderr:1: v1:172.21.15.192:6789/0 mon.b 2024-06-03T09:07:44.431 INFO:teuthology.orchestra.run.smithi088.stderr:2: v1:172.21.15.88:6790/0 mon.c 2024-06-03T09:07:44.431 INFO:teuthology.orchestra.run.smithi088.stderr: 2024-06-03T09:07:44.431 INFO:teuthology.orchestra.run.smithi088.stderr:2024-06-03T09:07:44.427+0000 7ff2157fa640 10 monclient(hunting): _finish_auth 0 2024-06-03T09:07:44.431 INFO:teuthology.orchestra.run.smithi088.stderr:2024-06-03T09:07:44.427+0000 7ff2157fa640 1 -- 172.21.15.88:0/402045423 <== mon.1 v1:172.21.15.192:6789/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 2234374065 0 0) 0x7ff2040017f0 con 0x7ff2180adb90 2024-06-03T09:07:44.431 INFO:teuthology.orchestra.run.smithi088.stderr:2024-06-03T09:07:44.427+0000 7ff2157fa640 10 monclient(hunting): handle_monmap mon_map magic: 0 v1 2024-06-03T09:07:44.431 INFO:teuthology.orchestra.run.smithi088.stderr:2024-06-03T09:07:44.427+0000 7ff2157fa640 10 monclient(hunting): got monmap 1 from mon.b (according to old e1) 2024-06-03T09:07:44.431 INFO:teuthology.orchestra.run.smithi088.stderr:2024-06-03T09:07:44.427+0000 7ff2157fa640 10 monclient(hunting): dump: 2024-06-03T09:07:44.431 INFO:teuthology.orchestra.run.smithi088.stderr:epoch 1 2024-06-03T09:07:44.431 INFO:teuthology.orchestra.run.smithi088.stderr:fsid ec3c3fdd-3ac6-4d7d-9412-683130ed8b04 2024-06-03T09:07:44.431 INFO:teuthology.orchestra.run.smithi088.stderr:last_changed 2024-06-03T08:34:31.885125+0000 2024-06-03T09:07:44.432 INFO:teuthology.orchestra.run.smithi088.stderr:created 2024-06-03T08:34:31.885125+0000 2024-06-03T09:07:44.432 INFO:teuthology.orchestra.run.smithi088.stderr:min_mon_release 18 (reef) 2024-06-03T09:07:44.432 INFO:teuthology.orchestra.run.smithi088.stderr:election_strategy: 3 2024-06-03T09:07:44.432 INFO:teuthology.orchestra.run.smithi088.stderr:0: v1:172.21.15.88:6789/0 mon.a 2024-06-03T09:07:44.432 INFO:teuthology.orchestra.run.smithi088.stderr:1: v1:172.21.15.192:6789/0 mon.b 2024-06-03T09:07:44.432 INFO:teuthology.orchestra.run.smithi088.stderr:2: v1:172.21.15.88:6790/0 mon.c 2024-06-03T09:07:44.432 INFO:teuthology.orchestra.run.smithi088.stderr: 2024-06-03T09:07:44.432 INFO:teuthology.orchestra.run.smithi088.stderr:2024-06-03T09:07:44.427+0000 7ff2157fa640 1 -- 172.21.15.88:0/402045423 <== mon.0 v1:172.21.15.88:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 764+0+0 (unknown 4105970725 0 0) 0x7ff218132f50 con 0x7ff2181383e0 2024-06-03T09:07:44.432 INFO:teuthology.orchestra.run.smithi088.stderr:2024-06-03T09:07:44.427+0000 7ff2157fa640 1 -- 172.21.15.88:0/402045423 --> v1:172.21.15.88:6789/0 -- auth(proto 2 165 bytes epoch 0) v1 -- 0x7ff2180aab00 con 0x7ff2181383e0 2024-06-03T09:07:44.432 INFO:teuthology.orchestra.run.smithi088.stderr:2024-06-03T09:07:44.427+0000 7ff2157fa640 1 -- 172.21.15.88:0/402045423 <== mon.0 v1:172.21.15.88:6789/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 2234374065 0 0) 0x7ff210001b10 con 0x7ff2181383e0 2024-06-03T09:07:44.432 INFO:teuthology.orchestra.run.smithi088.stderr:2024-06-03T09:07:44.427+0000 7ff2157fa640 10 monclient(hunting): handle_monmap mon_map magic: 0 v1 2024-06-03T09:07:44.432 INFO:teuthology.orchestra.run.smithi088.stderr:2024-06-03T09:07:44.427+0000 7ff2157fa640 10 monclient(hunting): got monmap 1 from mon.a (according to old e1) 2024-06-03T09:07:44.432 INFO:teuthology.orchestra.run.smithi088.stderr:2024-06-03T09:07:44.427+0000 7ff2157fa640 10 monclient(hunting): dump: 2024-06-03T09:07:44.432 INFO:teuthology.orchestra.run.smithi088.stderr:epoch 1 2024-06-03T09:07:44.433 INFO:teuthology.orchestra.run.smithi088.stderr:fsid ec3c3fdd-3ac6-4d7d-9412-683130ed8b04 2024-06-03T09:07:44.433 INFO:teuthology.orchestra.run.smithi088.stderr:last_changed 2024-06-03T08:34:31.885125+0000 2024-06-03T09:07:44.433 INFO:teuthology.orchestra.run.smithi088.stderr:created 2024-06-03T08:34:31.885125+0000 2024-06-03T09:07:44.433 INFO:teuthology.orchestra.run.smithi088.stderr:min_mon_release 18 (reef) 2024-06-03T09:07:44.433 INFO:teuthology.orchestra.run.smithi088.stderr:election_strategy: 3 2024-06-03T09:07:44.433 INFO:teuthology.orchestra.run.smithi088.stderr:0: v1:172.21.15.88:6789/0 mon.a 2024-06-03T09:07:44.433 INFO:teuthology.orchestra.run.smithi088.stderr:1: v1:172.21.15.192:6789/0 mon.b 2024-06-03T09:07:44.433 INFO:teuthology.orchestra.run.smithi088.stderr:2: v1:172.21.15.88:6790/0 mon.c 2024-06-03T09:07:44.433 INFO:teuthology.orchestra.run.smithi088.stderr: 2024-06-03T09:07:44.434 INFO:teuthology.orchestra.run.smithi088.stderr:2024-06-03T09:07:44.430+0000 7ff2157fa640 1 -- 172.21.15.88:0/402045423 <== mon.2 v1:172.21.15.88:6790/0 4 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 580+0+0 (unknown 2508487464 0 0) 0x7ff218133ce0 con 0x7ff21806ee00 2024-06-03T09:07:44.435 INFO:teuthology.orchestra.run.smithi088.stderr:2024-06-03T09:07:44.430+0000 7ff2157fa640 1 -- 172.21.15.88:0/402045423 >> v1:172.21.15.192:6789/0 conn(0x7ff2180adb90 legacy=0x7ff218132130 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-06-03T09:07:44.435 INFO:teuthology.orchestra.run.smithi088.stderr:2024-06-03T09:07:44.430+0000 7ff2157fa640 1 -- 172.21.15.88:0/402045423 >> v1:172.21.15.88:6789/0 conn(0x7ff2181383e0 legacy=0x7ff218132840 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-06-03T09:07:44.435 INFO:teuthology.orchestra.run.smithi088.stderr:2024-06-03T09:07:44.430+0000 7ff2157fa640 10 monclient: _finish_hunting 0 2024-06-03T09:07:44.435 INFO:teuthology.orchestra.run.smithi088.stderr:2024-06-03T09:07:44.430+0000 7ff2157fa640 1 monclient: found mon.c 2024-06-03T09:07:44.435 INFO:teuthology.orchestra.run.smithi088.stderr:2024-06-03T09:07:44.430+0000 7ff2157fa640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-06-03T09:07:44.435 INFO:teuthology.orchestra.run.smithi088.stderr:2024-06-03T09:07:44.430+0000 7ff2157fa640 10 monclient: _send_mon_message to mon.c at v1:172.21.15.88:6790/0 2024-06-03T09:07:44.435 INFO:teuthology.orchestra.run.smithi088.stderr:2024-06-03T09:07:44.430+0000 7ff2157fa640 1 -- 172.21.15.88:0/402045423 --> v1:172.21.15.88:6790/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7ff2180a9fd0 con 0x7ff21806ee00 2024-06-03T09:07:44.435 INFO:teuthology.orchestra.run.smithi088.stderr:2024-06-03T09:07:44.430+0000 7ff2157fa640 10 monclient: _finish_auth 0 2024-06-03T09:07:44.435 INFO:teuthology.orchestra.run.smithi088.stderr:2024-06-03T09:07:44.430+0000 7ff2157fa640 10 monclient: _check_auth_tickets 2024-06-03T09:07:44.436 INFO:teuthology.orchestra.run.smithi088.stderr:2024-06-03T09:07:44.430+0000 7ff2157fa640 20 monclient: _check_auth_rotating not needed by client.admin 2024-06-03T09:07:44.436 INFO:teuthology.orchestra.run.smithi088.stderr:2024-06-03T09:07:44.430+0000 7ff21ee3b640 5 monclient: authenticate success, global_id 30356 2024-06-03T09:07:44.436 INFO:teuthology.orchestra.run.smithi088.stderr:2024-06-03T09:07:44.430+0000 7ff21ee3b640 10 monclient: _renew_subs 2024-06-03T09:07:44.436 INFO:teuthology.orchestra.run.smithi088.stderr:2024-06-03T09:07:44.430+0000 7ff21ee3b640 10 monclient: _send_mon_message to mon.c at v1:172.21.15.88:6790/0 2024-06-03T09:07:44.436 INFO:teuthology.orchestra.run.smithi088.stderr:2024-06-03T09:07:44.430+0000 7ff21ee3b640 1 -- 172.21.15.88:0/402045423 --> v1:172.21.15.88:6790/0 -- mon_subscribe({mgrmap=0+}) v3 -- 0x7ff2180aa1c0 con 0x7ff21806ee00 2024-06-03T09:07:44.436 INFO:teuthology.orchestra.run.smithi088.stderr:2024-06-03T09:07:44.430+0000 7ff21ee3b640 10 client.30356.objecter _maybe_request_map subscribing (onetime) to next osd map 2024-06-03T09:07:44.436 INFO:teuthology.orchestra.run.smithi088.stderr:2024-06-03T09:07:44.430+0000 7ff21ee3b640 10 monclient: _renew_subs 2024-06-03T09:07:44.436 INFO:teuthology.orchestra.run.smithi088.stderr:2024-06-03T09:07:44.430+0000 7ff21ee3b640 10 monclient: _send_mon_message to mon.c at v1:172.21.15.88:6790/0 2024-06-03T09:07:44.436 INFO:teuthology.orchestra.run.smithi088.stderr:2024-06-03T09:07:44.430+0000 7ff21ee3b640 1 -- 172.21.15.88:0/402045423 --> v1:172.21.15.88:6790/0 -- mon_subscribe({osdmap=0}) v3 -- 0x7ff2181333b0 con 0x7ff21806ee00 2024-06-03T09:07:44.436 INFO:teuthology.orchestra.run.smithi088.stderr:2024-06-03T09:07:44.430+0000 7ff21ee3b640 1 librados: init done 2024-06-03T09:07:44.436 INFO:teuthology.orchestra.run.smithi088.stderr:2024-06-03T09:07:44.431+0000 7ff21ee3b640 10 start_mon_command cmd=[{"prefix": "get_command_descriptions"}] 2024-06-03T09:07:44.436 INFO:teuthology.orchestra.run.smithi088.stderr:2024-06-03T09:07:44.431+0000 7ff21ee3b640 10 monclient: _send_command 1 [{"prefix": "get_command_descriptions"}] 2024-06-03T09:07:44.437 INFO:teuthology.orchestra.run.smithi088.stderr:2024-06-03T09:07:44.431+0000 7ff21ee3b640 10 monclient: _send_mon_message to mon.c at v1:172.21.15.88:6790/0 2024-06-03T09:07:44.437 INFO:teuthology.orchestra.run.smithi088.stderr:2024-06-03T09:07:44.431+0000 7ff21ee3b640 1 -- 172.21.15.88:0/402045423 --> v1:172.21.15.88:6790/0 -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7ff2180aa3b0 con 0x7ff21806ee00 2024-06-03T09:07:44.643 INFO:teuthology.orchestra.run.smithi088.stderr:2024-06-03T09:07:44.643+0000 7ff2157fa640 1 -- 172.21.15.88:0/402045423 <== mon.2 v1:172.21.15.88:6790/0 5 ==== config(0 keys) v1 ==== 4+0+0 (unknown 0 0 0) 0x7ff20c003910 con 0x7ff21806ee00 2024-06-03T09:07:44.643 INFO:teuthology.orchestra.run.smithi088.stderr:2024-06-03T09:07:44.643+0000 7ff2157fa640 10 monclient: handle_config config(0 keys) v1 2024-06-03T09:07:44.643 INFO:teuthology.orchestra.run.smithi088.stderr:2024-06-03T09:07:44.643+0000 7ff216ffd640 4 set_mon_vals no callback set 2024-06-03T09:07:44.676 INFO:teuthology.orchestra.run.smithi088.stderr:2024-06-03T09:07:44.676+0000 7ff2157fa640 1 -- 172.21.15.88:0/402045423 <== mon.2 v1:172.21.15.88:6790/0 6 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 2234374065 0 0) 0x7ff20c003c20 con 0x7ff21806ee00 2024-06-03T09:07:44.676 INFO:teuthology.orchestra.run.smithi088.stderr:2024-06-03T09:07:44.676+0000 7ff2157fa640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-06-03T09:07:44.676 INFO:teuthology.orchestra.run.smithi088.stderr:2024-06-03T09:07:44.676+0000 7ff2157fa640 10 monclient: got monmap 1 from mon.c (according to old e1) 2024-06-03T09:07:44.676 INFO:teuthology.orchestra.run.smithi088.stderr:2024-06-03T09:07:44.676+0000 7ff2157fa640 10 monclient: dump: 2024-06-03T09:07:44.676 INFO:teuthology.orchestra.run.smithi088.stderr:epoch 1 2024-06-03T09:07:44.676 INFO:teuthology.orchestra.run.smithi088.stderr:fsid ec3c3fdd-3ac6-4d7d-9412-683130ed8b04 2024-06-03T09:07:44.676 INFO:teuthology.orchestra.run.smithi088.stderr:last_changed 2024-06-03T08:34:31.885125+0000 2024-06-03T09:07:44.676 INFO:teuthology.orchestra.run.smithi088.stderr:created 2024-06-03T08:34:31.885125+0000 2024-06-03T09:07:44.677 INFO:teuthology.orchestra.run.smithi088.stderr:min_mon_release 18 (reef) 2024-06-03T09:07:44.677 INFO:teuthology.orchestra.run.smithi088.stderr:election_strategy: 3 2024-06-03T09:07:44.677 INFO:teuthology.orchestra.run.smithi088.stderr:0: v1:172.21.15.88:6789/0 mon.a 2024-06-03T09:07:44.677 INFO:teuthology.orchestra.run.smithi088.stderr:1: v1:172.21.15.192:6789/0 mon.b 2024-06-03T09:07:44.677 INFO:teuthology.orchestra.run.smithi088.stderr:2: v1:172.21.15.88:6790/0 mon.c 2024-06-03T09:07:44.677 INFO:teuthology.orchestra.run.smithi088.stderr: 2024-06-03T09:07:44.677 INFO:teuthology.orchestra.run.smithi088.stderr:2024-06-03T09:07:44.676+0000 7ff2157fa640 1 -- 172.21.15.88:0/402045423 <== mon.2 v1:172.21.15.88:6790/0 7 ==== mgrmap(e 5) v1 ==== 89110+0+0 (unknown 873533973 0 0) 0x7ff20c003dc0 con 0x7ff21806ee00 2024-06-03T09:07:44.678 INFO:teuthology.orchestra.run.smithi088.stderr:2024-06-03T09:07:44.677+0000 7ff217fff640 10 monclient: get_auth_request con 0x7ff1fc06d410 auth_method 0 2024-06-03T09:07:44.678 INFO:teuthology.orchestra.run.smithi088.stderr:2024-06-03T09:07:44.678+0000 7ff2157fa640 10 client.30356.objecter ms_handle_connect 0x7ff1fc06d410 2024-06-03T09:07:44.681 INFO:teuthology.orchestra.run.smithi088.stderr:2024-06-03T09:07:44.680+0000 7ff2157fa640 1 -- 172.21.15.88:0/402045423 <== mon.2 v1:172.21.15.88:6790/0 8 ==== osd_map(1014..1014 src has 735..1014) v4 ==== 5229+0+0 (unknown 332246716 0 0) 0x7ff20c052fe0 con 0x7ff21806ee00 2024-06-03T09:07:44.681 INFO:teuthology.orchestra.run.smithi088.stderr:2024-06-03T09:07:44.680+0000 7ff2157fa640 10 client.30356.objecter ms_dispatch 0x7ff218000b90 osd_map(1014..1014 src has 735..1014) v4 2024-06-03T09:07:44.681 INFO:teuthology.orchestra.run.smithi088.stderr:2024-06-03T09:07:44.680+0000 7ff2157fa640 3 client.30356.objecter handle_osd_map got epochs [1014,1014] > 0 2024-06-03T09:07:44.681 INFO:teuthology.orchestra.run.smithi088.stderr:2024-06-03T09:07:44.680+0000 7ff2157fa640 3 client.30356.objecter handle_osd_map decoding full epoch 1014 2024-06-03T09:07:44.681 INFO:teuthology.orchestra.run.smithi088.stderr:2024-06-03T09:07:44.681+0000 7ff2157fa640 20 client.30356.objecter dump_active .. 0 homeless 2024-06-03T09:07:44.681 INFO:teuthology.orchestra.run.smithi088.stderr:2024-06-03T09:07:44.681+0000 7ff2157fa640 1 -- 172.21.15.88:0/402045423 <== mon.2 v1:172.21.15.88:6790/0 9 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+182498 (unknown 1092875540 0 3016760700) 0x7ff20c04eb90 con 0x7ff21806ee00 2024-06-03T09:07:44.681 INFO:teuthology.orchestra.run.smithi088.stderr:2024-06-03T09:07:44.681+0000 7ff2157fa640 10 monclient: handle_mon_command_ack 1 [{"prefix": "get_command_descriptions"}] 2024-06-03T09:07:44.681 INFO:teuthology.orchestra.run.smithi088.stderr:2024-06-03T09:07:44.681+0000 7ff2157fa640 10 monclient: _finish_command 1 = system:0 2024-06-03T09:07:44.862 INFO:teuthology.orchestra.run.smithi088.stderr:2024-06-03T09:07:44.862+0000 7ff21ee3b640 10 start_mon_command cmd=[{"prefix": "config set", "who": "global", "name": "mon_health_to_clog", "value": "false"}] 2024-06-03T09:07:44.862 INFO:teuthology.orchestra.run.smithi088.stderr:2024-06-03T09:07:44.862+0000 7ff21ee3b640 10 monclient: _send_command 2 [{"prefix": "config set", "who": "global", "name": "mon_health_to_clog", "value": "false"}] 2024-06-03T09:07:44.862 INFO:teuthology.orchestra.run.smithi088.stderr:2024-06-03T09:07:44.862+0000 7ff21ee3b640 10 monclient: _send_mon_message to mon.c at v1:172.21.15.88:6790/0 2024-06-03T09:07:44.862 INFO:teuthology.orchestra.run.smithi088.stderr:2024-06-03T09:07:44.862+0000 7ff21ee3b640 1 -- 172.21.15.88:0/402045423 --> v1:172.21.15.88:6790/0 -- mon_command([{prefix=config set, name=mon_health_to_clog}] v 0) v1 -- 0x7ff21804ea60 con 0x7ff21806ee00 2024-06-03T09:07:45.034 INFO:teuthology.orchestra.run.smithi088.stderr:2024-06-03T09:07:45.034+0000 7ff2157fa640 1 -- 172.21.15.88:0/402045423 <== mon.2 v1:172.21.15.88:6790/0 10 ==== config(1 keys) v1 ==== 35+0+0 (unknown 2555425225 0 0) 0x7ff20c04eb90 con 0x7ff21806ee00 2024-06-03T09:07:45.034 INFO:teuthology.orchestra.run.smithi088.stderr:2024-06-03T09:07:45.034+0000 7ff2157fa640 10 monclient: handle_config config(1 keys) v1 2024-06-03T09:07:45.034 INFO:teuthology.orchestra.run.smithi088.stderr:2024-06-03T09:07:45.034+0000 7ff216ffd640 4 set_mon_vals no callback set 2024-06-03T09:07:45.034 INFO:teuthology.orchestra.run.smithi088.stderr:2024-06-03T09:07:45.034+0000 7ff216ffd640 10 set_mon_vals mon_health_to_clog = false 2024-06-03T09:07:45.034 INFO:teuthology.orchestra.run.smithi088.stderr:2024-06-03T09:07:45.034+0000 7ff2157fa640 1 -- 172.21.15.88:0/402045423 <== mon.2 v1:172.21.15.88:6790/0 11 ==== mon_command_ack([{prefix=config set, name=mon_health_to_clog}]=0 v7)=0 v7) v1 ==== 123+0+0 (unknown 3414686427 0 0) 0x7ff20c050ac0 con 0x7ff21806ee00 2024-06-03T09:07:45.034 INFO:teuthology.orchestra.run.smithi088.stderr:2024-06-03T09:07:45.034+0000 7ff2157fa640 10 monclient: handle_mon_command_ack 2 [{"prefix": "config set", "who": "global", "name": "mon_health_to_clog", "value": "false"}] 2024-06-03T09:07:45.034 INFO:teuthology.orchestra.run.smithi088.stderr:2024-06-03T09:07:45.034+0000 7ff2157fa640 10 monclient: _finish_command 2 = system:0 2024-06-03T09:07:45.036 INFO:teuthology.orchestra.run.smithi088.stderr:2024-06-03T09:07:45.035+0000 7ff21ee3b640 10 librados: watch_flush enter 2024-06-03T09:07:45.036 INFO:teuthology.orchestra.run.smithi088.stderr:2024-06-03T09:07:45.036+0000 7ff21ee3b640 10 librados: watch_flush exit 2024-06-03T09:07:45.036 INFO:teuthology.orchestra.run.smithi088.stderr:2024-06-03T09:07:45.036+0000 7ff21ee3b640 20 client.30356.objecter shutdown clearing up homeless session... 2024-06-03T09:07:45.036 INFO:teuthology.orchestra.run.smithi088.stderr:2024-06-03T09:07:45.036+0000 7ff21ee3b640 10 client.30356.objecter successfully canceled tick 2024-06-03T09:07:45.036 INFO:teuthology.orchestra.run.smithi088.stderr:2024-06-03T09:07:45.036+0000 7ff21ee3b640 1 -- 172.21.15.88:0/402045423 >> v1:172.21.15.192:6800/34229 conn(0x7ff1fc06d410 legacy=0x7ff1fc06f8d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-06-03T09:07:45.037 INFO:teuthology.orchestra.run.smithi088.stderr:2024-06-03T09:07:45.036+0000 7ff21ee3b640 10 monclient: shutdown 2024-06-03T09:07:45.037 INFO:teuthology.orchestra.run.smithi088.stderr:2024-06-03T09:07:45.036+0000 7ff21ee3b640 20 monclient: shutdown discarding 0 pending message(s) 2024-06-03T09:07:45.037 INFO:teuthology.orchestra.run.smithi088.stderr:2024-06-03T09:07:45.036+0000 7ff21ee3b640 1 -- 172.21.15.88:0/402045423 >> v1:172.21.15.88:6790/0 conn(0x7ff21806ee00 legacy=0x7ff2180aca50 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-06-03T09:07:45.037 INFO:teuthology.orchestra.run.smithi088.stderr:2024-06-03T09:07:45.036+0000 7ff21ee3b640 1 -- 172.21.15.88:0/402045423 shutdown_connections 2024-06-03T09:07:45.037 INFO:teuthology.orchestra.run.smithi088.stderr:2024-06-03T09:07:45.036+0000 7ff21ee3b640 1 -- 172.21.15.88:0/402045423 >> 172.21.15.88:0/402045423 conn(0x7ff2180b2f20 msgr2=0x7ff2180af9e0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-06-03T09:07:45.037 INFO:teuthology.orchestra.run.smithi088.stderr:2024-06-03T09:07:45.036+0000 7ff21ee3b640 1 -- 172.21.15.88:0/402045423 shutdown_connections 2024-06-03T09:07:45.037 INFO:teuthology.orchestra.run.smithi088.stderr:2024-06-03T09:07:45.036+0000 7ff21ee3b640 1 -- 172.21.15.88:0/402045423 wait complete. 2024-06-03T09:07:45.037 INFO:teuthology.orchestra.run.smithi088.stderr:2024-06-03T09:07:45.036+0000 7ff21ee3b640 1 librados: shutdown 2024-06-03T09:07:45.049 INFO:teuthology.misc:Shutting down mds daemons... 2024-06-03T09:07:45.049 INFO:teuthology.misc:Shutting down osd daemons... 2024-06-03T09:07:45.049 DEBUG:tasks.ceph.osd.0:waiting for process to exit 2024-06-03T09:07:45.050 INFO:teuthology.orchestra.run:waiting for 300 2024-06-03T09:07:45.063 INFO:tasks.ceph.osd.0.smithi088.stderr:2024-06-03T09:07:45.054+0000 b48a640 -1 received signal: Terminated from /usr/bin/python3 /bin/daemon-helper term env OPENSSL_ia32cap=~0x1000000000000000 valgrind --trace-children=no --child-silent-after-fork=yes --soname-synonyms=somalloc=*tcmalloc* --num-callers=50 --suppressions=/home/ubuntu/cephtest/valgrind.supp --xml=yes --xml-file=/var/log/ceph/valgrind/osd.0.log --time-stamp=yes --vgdb=yes --exit-on-first-error=yes --error-exitcode=42 --tool=memcheck ceph-osd -f --cluster ceph -i 0 (PID: 35036) UID: 0 2024-06-03T09:07:45.063 INFO:tasks.ceph.osd.0.smithi088.stderr:2024-06-03T09:07:45.056+0000 b48a640 -1 osd.0 1014 *** Got signal Terminated *** 2024-06-03T09:07:51.151 INFO:tasks.ceph.osd.0:Stopped 2024-06-03T09:07:51.152 DEBUG:tasks.ceph.osd.1:waiting for process to exit 2024-06-03T09:07:51.152 INFO:teuthology.orchestra.run:waiting for 300 2024-06-03T09:07:51.167 INFO:tasks.ceph.osd.1.smithi088.stderr:2024-06-03T09:07:51.156+0000 b48a640 -1 received signal: Terminated from /usr/bin/python3 /bin/daemon-helper term env OPENSSL_ia32cap=~0x1000000000000000 valgrind --trace-children=no --child-silent-after-fork=yes --soname-synonyms=somalloc=*tcmalloc* --num-callers=50 --suppressions=/home/ubuntu/cephtest/valgrind.supp --xml=yes --xml-file=/var/log/ceph/valgrind/osd.1.log --time-stamp=yes --vgdb=yes --exit-on-first-error=yes --error-exitcode=42 --tool=memcheck ceph-osd -f --cluster ceph -i 1 (PID: 35037) UID: 0 2024-06-03T09:07:51.167 INFO:tasks.ceph.osd.1.smithi088.stderr:2024-06-03T09:07:51.158+0000 b48a640 -1 osd.1 1018 *** Got signal Terminated *** 2024-06-03T09:07:51.349 INFO:tasks.daemonwatchdog.daemon_watchdog:daemon ceph.osd.0 is failed for ~0s 2024-06-03T09:07:53.796 INFO:tasks.ceph.mon.a.smithi088.stderr:==00:00:32:49.122 34743== Warning: unimplemented fcntl command: 1036 2024-06-03T09:07:53.802 INFO:tasks.ceph.mon.a.smithi088.stderr:==00:00:32:49.128 34743== Warning: unimplemented fcntl command: 1036 2024-06-03T09:07:53.832 INFO:tasks.ceph.mon.a.smithi088.stderr:==00:00:32:49.157 34743== Warning: unimplemented fcntl command: 1036 2024-06-03T09:07:53.860 INFO:tasks.ceph.mon.b.smithi192.stderr:==00:00:32:49.163 34228== Warning: unimplemented fcntl command: 1036 2024-06-03T09:07:53.863 INFO:tasks.ceph.mon.b.smithi192.stderr:==00:00:32:49.166 34228== Warning: unimplemented fcntl command: 1036 2024-06-03T09:07:53.864 INFO:tasks.ceph.mon.c.smithi088.stderr:==00:00:32:49.189 34749== Warning: unimplemented fcntl command: 1036 2024-06-03T09:07:53.866 INFO:tasks.ceph.mon.c.smithi088.stderr:==00:00:32:49.192 34749== Warning: unimplemented fcntl command: 1036 2024-06-03T09:07:53.889 INFO:tasks.ceph.mon.b.smithi192.stderr:==00:00:32:49.192 34228== Warning: unimplemented fcntl command: 1036 2024-06-03T09:07:53.893 INFO:tasks.ceph.mon.c.smithi088.stderr:==00:00:32:49.219 34749== Warning: unimplemented fcntl command: 1036 2024-06-03T09:07:54.387 INFO:tasks.ceph.mon.a.smithi088.stderr:==00:00:32:49.712 34743== Warning: unimplemented fcntl command: 1036 2024-06-03T09:07:54.388 INFO:tasks.ceph.mon.a.smithi088.stderr:==00:00:32:49.714 34743== Warning: unimplemented fcntl command: 1036 2024-06-03T09:07:54.397 INFO:tasks.ceph.mon.a.smithi088.stderr:==00:00:32:49.722 34743== Warning: unimplemented fcntl command: 1036 2024-06-03T09:07:54.424 INFO:tasks.ceph.mon.b.smithi192.stderr:==00:00:32:49.727 34228== Warning: unimplemented fcntl command: 1036 2024-06-03T09:07:54.426 INFO:tasks.ceph.mon.b.smithi192.stderr:==00:00:32:49.728 34228== Warning: unimplemented fcntl command: 1036 2024-06-03T09:07:54.434 INFO:tasks.ceph.mon.b.smithi192.stderr:==00:00:32:49.737 34228== Warning: unimplemented fcntl command: 1036 2024-06-03T09:07:54.458 INFO:tasks.ceph.mon.c.smithi088.stderr:==00:00:32:49.784 34749== Warning: unimplemented fcntl command: 1036 2024-06-03T09:07:54.459 INFO:tasks.ceph.mon.c.smithi088.stderr:==00:00:32:49.785 34749== Warning: unimplemented fcntl command: 1036 2024-06-03T09:07:54.468 INFO:tasks.ceph.mon.c.smithi088.stderr:==00:00:32:49.794 34749== Warning: unimplemented fcntl command: 1036 2024-06-03T09:07:57.559 INFO:tasks.daemonwatchdog.daemon_watchdog:daemon ceph.osd.0 has been restored 2024-06-03T09:08:13.834 INFO:tasks.ceph.mon.a.smithi088.stderr:==00:00:33:09.159 34743== Warning: unimplemented fcntl command: 1036 2024-06-03T09:08:13.837 INFO:tasks.ceph.mon.a.smithi088.stderr:==00:00:33:09.162 34743== Warning: unimplemented fcntl command: 1036 2024-06-03T09:08:13.856 INFO:tasks.ceph.mon.a.smithi088.stderr:==00:00:33:09.181 34743== Warning: unimplemented fcntl command: 1036 2024-06-03T09:08:13.886 INFO:tasks.ceph.mon.b.smithi192.stderr:==00:00:33:09.189 34228== Warning: unimplemented fcntl command: 1036 2024-06-03T09:08:13.888 INFO:tasks.ceph.mon.b.smithi192.stderr:==00:00:33:09.190 34228== Warning: unimplemented fcntl command: 1036 2024-06-03T09:08:13.901 INFO:tasks.ceph.mon.c.smithi088.stderr:==00:00:33:09.227 34749== Warning: unimplemented fcntl command: 1036 2024-06-03T09:08:13.903 INFO:tasks.ceph.mon.b.smithi192.stderr:==00:00:33:09.205 34228== Warning: unimplemented fcntl command: 1036 2024-06-03T09:08:13.905 INFO:tasks.ceph.mon.c.smithi088.stderr:==00:00:33:09.230 34749== Warning: unimplemented fcntl command: 1036 2024-06-03T09:08:13.922 INFO:tasks.ceph.mon.c.smithi088.stderr:==00:00:33:09.248 34749== Warning: unimplemented fcntl command: 1036 2024-06-03T09:08:14.473 INFO:tasks.ceph.mon.b.smithi192.stderr:==00:00:33:09.776 34228== Warning: unimplemented fcntl command: 1036 2024-06-03T09:08:14.474 INFO:tasks.ceph.mon.b.smithi192.stderr:==00:00:33:09.777 34228== Warning: unimplemented fcntl command: 1036 2024-06-03T09:08:14.482 INFO:tasks.ceph.mon.b.smithi192.stderr:==00:00:33:09.785 34228== Warning: unimplemented fcntl command: 1036 2024-06-03T09:08:14.515 INFO:tasks.ceph.mon.a.smithi088.stderr:==00:00:33:09.840 34743== Warning: unimplemented fcntl command: 1036 2024-06-03T09:08:14.516 INFO:tasks.ceph.mon.a.smithi088.stderr:==00:00:33:09.842 34743== Warning: unimplemented fcntl command: 1036 2024-06-03T09:08:14.526 INFO:tasks.ceph.mon.a.smithi088.stderr:==00:00:33:09.851 34743== Warning: unimplemented fcntl command: 1036 2024-06-03T09:08:14.553 INFO:tasks.ceph.mon.c.smithi088.stderr:==00:00:33:09.879 34749== Warning: unimplemented fcntl command: 1036 2024-06-03T09:08:14.554 INFO:tasks.ceph.mon.c.smithi088.stderr:==00:00:33:09.880 34749== Warning: unimplemented fcntl command: 1036 2024-06-03T09:08:14.562 INFO:tasks.ceph.mon.c.smithi088.stderr:==00:00:33:09.888 34749== Warning: unimplemented fcntl command: 1036 2024-06-03T09:08:33.899 INFO:tasks.ceph.mon.a.smithi088.stderr:==00:00:33:29.224 34743== Warning: unimplemented fcntl command: 1036 2024-06-03T09:08:33.900 INFO:tasks.ceph.mon.a.smithi088.stderr:==00:00:33:29.226 34743== Warning: unimplemented fcntl command: 1036 2024-06-03T09:08:33.916 INFO:tasks.ceph.mon.a.smithi088.stderr:==00:00:33:29.242 34743== Warning: unimplemented fcntl command: 1036 2024-06-03T09:08:33.950 INFO:tasks.ceph.mon.b.smithi192.stderr:==00:00:33:29.253 34228== Warning: unimplemented fcntl command: 1036 2024-06-03T09:08:33.954 INFO:tasks.ceph.mon.b.smithi192.stderr:==00:00:33:29.256 34228== Warning: unimplemented fcntl command: 1036 2024-06-03T09:08:33.969 INFO:tasks.ceph.mon.c.smithi088.stderr:==00:00:33:29.295 34749== Warning: unimplemented fcntl command: 1036 2024-06-03T09:08:33.970 INFO:tasks.ceph.mon.b.smithi192.stderr:==00:00:33:29.272 34228== Warning: unimplemented fcntl command: 1036 2024-06-03T09:08:33.975 INFO:tasks.ceph.mon.c.smithi088.stderr:==00:00:33:29.301 34749== Warning: unimplemented fcntl command: 1036 2024-06-03T09:08:33.991 INFO:tasks.ceph.mon.c.smithi088.stderr:==00:00:33:29.317 34749== Warning: unimplemented fcntl command: 1036 2024-06-03T09:08:34.528 INFO:tasks.ceph.mon.a.smithi088.stderr:==00:00:33:29.854 34743== Warning: unimplemented fcntl command: 1036 2024-06-03T09:08:34.529 INFO:tasks.ceph.mon.a.smithi088.stderr:==00:00:33:29.855 34743== Warning: unimplemented fcntl command: 1036 2024-06-03T09:08:34.542 INFO:tasks.ceph.mon.a.smithi088.stderr:==00:00:33:29.868 34743== Warning: unimplemented fcntl command: 1036 2024-06-03T09:08:34.551 INFO:tasks.ceph.mon.b.smithi192.stderr:==00:00:33:29.854 34228== Warning: unimplemented fcntl command: 1036 2024-06-03T09:08:34.552 INFO:tasks.ceph.mon.b.smithi192.stderr:==00:00:33:29.855 34228== Warning: unimplemented fcntl command: 1036 2024-06-03T09:08:34.560 INFO:tasks.ceph.mon.b.smithi192.stderr:==00:00:33:29.863 34228== Warning: unimplemented fcntl command: 1036 2024-06-03T09:08:34.590 INFO:tasks.ceph.mon.c.smithi088.stderr:==00:00:33:29.916 34749== Warning: unimplemented fcntl command: 1036 2024-06-03T09:08:34.591 INFO:tasks.ceph.mon.c.smithi088.stderr:==00:00:33:29.917 34749== Warning: unimplemented fcntl command: 1036 2024-06-03T09:08:34.600 INFO:tasks.ceph.mon.c.smithi088.stderr:==00:00:33:29.926 34749== Warning: unimplemented fcntl command: 1036 2024-06-03T09:08:46.064 INFO:tasks.ceph.osd.1:Stopped 2024-06-03T09:08:46.065 DEBUG:tasks.ceph.osd.2:waiting for process to exit 2024-06-03T09:08:46.065 INFO:teuthology.orchestra.run:waiting for 300 2024-06-03T09:08:46.080 INFO:tasks.ceph.osd.2.smithi088.stderr:2024-06-03T09:08:46.069+0000 b48a640 -1 received signal: Terminated from /usr/bin/python3 /bin/daemon-helper term env OPENSSL_ia32cap=~0x1000000000000000 valgrind --trace-children=no --child-silent-after-fork=yes --soname-synonyms=somalloc=*tcmalloc* --num-callers=50 --suppressions=/home/ubuntu/cephtest/valgrind.supp --xml=yes --xml-file=/var/log/ceph/valgrind/osd.2.log --time-stamp=yes --vgdb=yes --exit-on-first-error=yes --error-exitcode=42 --tool=memcheck ceph-osd -f --cluster ceph -i 2 (PID: 35054) UID: 0 2024-06-03T09:08:46.080 INFO:tasks.ceph.osd.2.smithi088.stderr:2024-06-03T09:08:46.070+0000 b48a640 -1 osd.2 1020 *** Got signal Terminated *** 2024-06-03T09:08:47.137 INFO:tasks.daemonwatchdog.daemon_watchdog:daemon ceph.osd.1 is failed for ~0s 2024-06-03T09:08:53.246 INFO:tasks.daemonwatchdog.daemon_watchdog:daemon ceph.osd.1 has been restored 2024-06-03T09:08:53.906 INFO:tasks.ceph.mon.a.smithi088.stderr:==00:00:33:49.231 34743== Warning: unimplemented fcntl command: 1036 2024-06-03T09:08:53.955 INFO:tasks.ceph.mon.a.smithi088.stderr:==00:00:33:49.281 34743== Warning: unimplemented fcntl command: 1036 2024-06-03T09:08:53.974 INFO:tasks.ceph.mon.a.smithi088.stderr:==00:00:33:49.300 34743== Warning: unimplemented fcntl command: 1036 2024-06-03T09:08:54.006 INFO:tasks.ceph.mon.c.smithi088.stderr:==00:00:33:49.332 34749== Warning: unimplemented fcntl command: 1036 2024-06-03T09:08:54.010 INFO:tasks.ceph.mon.c.smithi088.stderr:==00:00:33:49.335 34749== Warning: unimplemented fcntl command: 1036 2024-06-03T09:08:54.027 INFO:tasks.ceph.mon.c.smithi088.stderr:==00:00:33:49.353 34749== Warning: unimplemented fcntl command: 1036 2024-06-03T09:08:54.042 INFO:tasks.ceph.mon.b.smithi192.stderr:==00:00:33:49.344 34228== Warning: unimplemented fcntl command: 1036 2024-06-03T09:08:54.043 INFO:tasks.ceph.mon.b.smithi192.stderr:==00:00:33:49.346 34228== Warning: unimplemented fcntl command: 1036 2024-06-03T09:08:54.061 INFO:tasks.ceph.mon.b.smithi192.stderr:==00:00:33:49.364 34228== Warning: unimplemented fcntl command: 1036 2024-06-03T09:08:54.643 INFO:tasks.ceph.mon.b.smithi192.stderr:==00:00:33:49.945 34228== Warning: unimplemented fcntl command: 1036 2024-06-03T09:08:54.644 INFO:tasks.ceph.mon.b.smithi192.stderr:==00:00:33:49.946 34228== Warning: unimplemented fcntl command: 1036 2024-06-03T09:08:54.647 INFO:tasks.ceph.mon.c.smithi088.stderr:==00:00:33:49.972 34749== Warning: unimplemented fcntl command: 1036 2024-06-03T09:08:54.648 INFO:tasks.ceph.mon.c.smithi088.stderr:==00:00:33:49.974 34749== Warning: unimplemented fcntl command: 1036 2024-06-03T09:08:54.653 INFO:tasks.ceph.mon.b.smithi192.stderr:==00:00:33:49.956 34228== Warning: unimplemented fcntl command: 1036 2024-06-03T09:08:54.656 INFO:tasks.ceph.mon.c.smithi088.stderr:==00:00:33:49.982 34749== Warning: unimplemented fcntl command: 1036 2024-06-03T09:08:54.678 INFO:tasks.ceph.mon.a.smithi088.stderr:==00:00:33:50.004 34743== Warning: unimplemented fcntl command: 1036 2024-06-03T09:08:54.679 INFO:tasks.ceph.mon.a.smithi088.stderr:==00:00:33:50.005 34743== Warning: unimplemented fcntl command: 1036 2024-06-03T09:08:54.691 INFO:tasks.ceph.mon.a.smithi088.stderr:==00:00:33:50.017 34743== Warning: unimplemented fcntl command: 1036 2024-06-03T09:09:13.997 INFO:tasks.ceph.mon.a.smithi088.stderr:==00:00:34:09.322 34743== Warning: unimplemented fcntl command: 1036 2024-06-03T09:09:13.998 INFO:tasks.ceph.mon.a.smithi088.stderr:==00:00:34:09.324 34743== Warning: unimplemented fcntl command: 1036 2024-06-03T09:09:14.014 INFO:tasks.ceph.mon.a.smithi088.stderr:==00:00:34:09.340 34743== Warning: unimplemented fcntl command: 1036 2024-06-03T09:09:14.044 INFO:tasks.ceph.mon.b.smithi192.stderr:==00:00:34:09.346 34228== Warning: unimplemented fcntl command: 1036 2024-06-03T09:09:14.046 INFO:tasks.ceph.mon.b.smithi192.stderr:==00:00:34:09.348 34228== Warning: unimplemented fcntl command: 1036 2024-06-03T09:09:14.063 INFO:tasks.ceph.mon.b.smithi192.stderr:==00:00:34:09.366 34228== Warning: unimplemented fcntl command: 1036 2024-06-03T09:09:14.073 INFO:tasks.ceph.mon.c.smithi088.stderr:==00:00:34:09.398 34749== Warning: unimplemented fcntl command: 1036 2024-06-03T09:09:14.076 INFO:tasks.ceph.mon.c.smithi088.stderr:==00:00:34:09.402 34749== Warning: unimplemented fcntl command: 1036 2024-06-03T09:09:14.096 INFO:tasks.ceph.mon.c.smithi088.stderr:==00:00:34:09.421 34749== Warning: unimplemented fcntl command: 1036 2024-06-03T09:09:14.577 INFO:tasks.ceph.mon.a.smithi088.stderr:==00:00:34:09.903 34743== Warning: unimplemented fcntl command: 1036 2024-06-03T09:09:14.580 INFO:tasks.ceph.mon.b.smithi192.stderr:==00:00:34:09.882 34228== Warning: unimplemented fcntl command: 1036 2024-06-03T09:09:14.580 INFO:tasks.ceph.mon.a.smithi088.stderr:==00:00:34:09.906 34743== Warning: unimplemented fcntl command: 1036 2024-06-03T09:09:14.581 INFO:tasks.ceph.mon.b.smithi192.stderr:==00:00:34:09.883 34228== Warning: unimplemented fcntl command: 1036 2024-06-03T09:09:14.589 INFO:tasks.ceph.mon.b.smithi192.stderr:==00:00:34:09.891 34228== Warning: unimplemented fcntl command: 1036 2024-06-03T09:09:14.601 INFO:tasks.ceph.mon.a.smithi088.stderr:==00:00:34:09.927 34743== Warning: unimplemented fcntl command: 1036 2024-06-03T09:09:14.687 INFO:tasks.ceph.mon.c.smithi088.stderr:==00:00:34:10.012 34749== Warning: unimplemented fcntl command: 1036 2024-06-03T09:09:14.688 INFO:tasks.ceph.mon.c.smithi088.stderr:==00:00:34:10.013 34749== Warning: unimplemented fcntl command: 1036 2024-06-03T09:09:14.697 INFO:tasks.ceph.mon.c.smithi088.stderr:==00:00:34:10.022 34749== Warning: unimplemented fcntl command: 1036 2024-06-03T09:09:34.044 INFO:tasks.ceph.mon.a.smithi088.stderr:==00:00:34:29.369 34743== Warning: unimplemented fcntl command: 1036 2024-06-03T09:09:34.046 INFO:tasks.ceph.mon.a.smithi088.stderr:==00:00:34:29.372 34743== Warning: unimplemented fcntl command: 1036 2024-06-03T09:09:34.064 INFO:tasks.ceph.mon.a.smithi088.stderr:==00:00:34:29.390 34743== Warning: unimplemented fcntl command: 1036 2024-06-03T09:09:34.098 INFO:tasks.ceph.mon.b.smithi192.stderr:==00:00:34:29.400 34228== Warning: unimplemented fcntl command: 1036 2024-06-03T09:09:34.099 INFO:tasks.ceph.mon.b.smithi192.stderr:==00:00:34:29.402 34228== Warning: unimplemented fcntl command: 1036 2024-06-03T09:09:34.114 INFO:tasks.ceph.mon.c.smithi088.stderr:==00:00:34:29.440 34749== Warning: unimplemented fcntl command: 1036 2024-06-03T09:09:34.116 INFO:tasks.ceph.mon.b.smithi192.stderr:==00:00:34:29.419 34228== Warning: unimplemented fcntl command: 1036 2024-06-03T09:09:34.118 INFO:tasks.ceph.mon.c.smithi088.stderr:==00:00:34:29.444 34749== Warning: unimplemented fcntl command: 1036 2024-06-03T09:09:34.136 INFO:tasks.ceph.mon.c.smithi088.stderr:==00:00:34:29.461 34749== Warning: unimplemented fcntl command: 1036 2024-06-03T09:09:34.705 INFO:tasks.ceph.mon.b.smithi192.stderr:==00:00:34:30.007 34228== Warning: unimplemented fcntl command: 1036 2024-06-03T09:09:34.706 INFO:tasks.ceph.mon.b.smithi192.stderr:==00:00:34:30.008 34228== Warning: unimplemented fcntl command: 1036 2024-06-03T09:09:34.714 INFO:tasks.ceph.mon.b.smithi192.stderr:==00:00:34:30.017 34228== Warning: unimplemented fcntl command: 1036 2024-06-03T09:09:34.731 INFO:tasks.ceph.mon.a.smithi088.stderr:==00:00:34:30.057 34743== Warning: unimplemented fcntl command: 1036 2024-06-03T09:09:34.733 INFO:tasks.ceph.mon.a.smithi088.stderr:==00:00:34:30.059 34743== Warning: unimplemented fcntl command: 1036 2024-06-03T09:09:34.742 INFO:tasks.ceph.mon.c.smithi088.stderr:==00:00:34:30.068 34749== Warning: unimplemented fcntl command: 1036 2024-06-03T09:09:34.743 INFO:tasks.ceph.mon.c.smithi088.stderr:==00:00:34:30.069 34749== Warning: unimplemented fcntl command: 1036 2024-06-03T09:09:34.746 INFO:tasks.ceph.mon.a.smithi088.stderr:==00:00:34:30.071 34743== Warning: unimplemented fcntl command: 1036 2024-06-03T09:09:34.752 INFO:tasks.ceph.mon.c.smithi088.stderr:==00:00:34:30.078 34749== Warning: unimplemented fcntl command: 1036 2024-06-03T09:09:47.081 INFO:tasks.ceph.osd.2:Stopped 2024-06-03T09:09:47.082 DEBUG:tasks.ceph.osd.3:waiting for process to exit 2024-06-03T09:09:47.082 INFO:teuthology.orchestra.run:waiting for 300 2024-06-03T09:09:47.096 INFO:tasks.ceph.osd.3.smithi088.stderr:2024-06-03T09:09:47.086+0000 b48a640 -1 received signal: Terminated from /usr/bin/python3 /bin/daemon-helper term env OPENSSL_ia32cap=~0x1000000000000000 valgrind --trace-children=no --child-silent-after-fork=yes --soname-synonyms=somalloc=*tcmalloc* --num-callers=50 --suppressions=/home/ubuntu/cephtest/valgrind.supp --xml=yes --xml-file=/var/log/ceph/valgrind/osd.3.log --time-stamp=yes --vgdb=yes --exit-on-first-error=yes --error-exitcode=42 --tool=memcheck ceph-osd -f --cluster ceph -i 3 (PID: 35055) UID: 0 2024-06-03T09:09:47.096 INFO:tasks.ceph.osd.3.smithi088.stderr:2024-06-03T09:09:47.087+0000 b48a640 -1 osd.3 1022 *** Got signal Terminated *** 2024-06-03T09:09:54.206 INFO:tasks.ceph.mon.a.smithi088.stderr:==00:00:34:49.532 34743== Warning: unimplemented fcntl command: 1036 2024-06-03T09:09:54.208 INFO:tasks.ceph.mon.a.smithi088.stderr:==00:00:34:49.534 34743== Warning: unimplemented fcntl command: 1036 2024-06-03T09:09:54.224 INFO:tasks.ceph.mon.a.smithi088.stderr:==00:00:34:49.550 34743== Warning: unimplemented fcntl command: 1036 2024-06-03T09:09:54.259 INFO:tasks.ceph.mon.c.smithi088.stderr:==00:00:34:49.584 34749== Warning: unimplemented fcntl command: 1036 2024-06-03T09:09:54.260 INFO:tasks.ceph.mon.b.smithi192.stderr:==00:00:34:49.562 34228== Warning: unimplemented fcntl command: 1036 2024-06-03T09:09:54.260 INFO:tasks.ceph.mon.c.smithi088.stderr:==00:00:34:49.586 34749== Warning: unimplemented fcntl command: 1036 2024-06-03T09:09:54.263 INFO:tasks.ceph.mon.b.smithi192.stderr:==00:00:34:49.565 34228== Warning: unimplemented fcntl command: 1036 2024-06-03T09:09:54.276 INFO:tasks.ceph.mon.c.smithi088.stderr:==00:00:34:49.602 34749== Warning: unimplemented fcntl command: 1036 2024-06-03T09:09:54.280 INFO:tasks.ceph.mon.b.smithi192.stderr:==00:00:34:49.582 34228== Warning: unimplemented fcntl command: 1036 2024-06-03T09:10:14.146 INFO:tasks.ceph.mon.a.smithi088.stderr:==00:00:35:09.471 34743== Warning: unimplemented fcntl command: 1036 2024-06-03T09:10:14.147 INFO:tasks.ceph.mon.a.smithi088.stderr:==00:00:35:09.473 34743== Warning: unimplemented fcntl command: 1036 2024-06-03T09:10:14.163 INFO:tasks.ceph.mon.a.smithi088.stderr:==00:00:35:09.489 34743== Warning: unimplemented fcntl command: 1036 2024-06-03T09:10:14.197 INFO:tasks.ceph.mon.c.smithi088.stderr:==00:00:35:09.522 34749== Warning: unimplemented fcntl command: 1036 2024-06-03T09:10:14.199 INFO:tasks.ceph.mon.b.smithi192.stderr:==00:00:35:09.502 34228== Warning: unimplemented fcntl command: 1036 2024-06-03T09:10:14.201 INFO:tasks.ceph.mon.b.smithi192.stderr:==00:00:35:09.503 34228== Warning: unimplemented fcntl command: 1036 2024-06-03T09:10:14.203 INFO:tasks.ceph.mon.c.smithi088.stderr:==00:00:35:09.529 34749== Warning: unimplemented fcntl command: 1036 2024-06-03T09:10:14.217 INFO:tasks.ceph.mon.b.smithi192.stderr:==00:00:35:09.520 34228== Warning: unimplemented fcntl command: 1036 2024-06-03T09:10:14.219 INFO:tasks.ceph.mon.c.smithi088.stderr:==00:00:35:09.545 34749== Warning: unimplemented fcntl command: 1036 2024-06-03T09:10:34.190 INFO:tasks.ceph.mon.a.smithi088.stderr:==00:00:35:29.515 34743== Warning: unimplemented fcntl command: 1036 2024-06-03T09:10:34.193 INFO:tasks.ceph.mon.a.smithi088.stderr:==00:00:35:29.519 34743== Warning: unimplemented fcntl command: 1036 2024-06-03T09:10:34.211 INFO:tasks.ceph.mon.a.smithi088.stderr:==00:00:35:29.537 34743== Warning: unimplemented fcntl command: 1036 2024-06-03T09:10:34.246 INFO:tasks.ceph.mon.b.smithi192.stderr:==00:00:35:29.548 34228== Warning: unimplemented fcntl command: 1036 2024-06-03T09:10:34.247 INFO:tasks.ceph.mon.b.smithi192.stderr:==00:00:35:29.549 34228== Warning: unimplemented fcntl command: 1036 2024-06-03T09:10:34.249 INFO:tasks.ceph.mon.c.smithi088.stderr:==00:00:35:29.574 34749== Warning: unimplemented fcntl command: 1036 2024-06-03T09:10:34.251 INFO:tasks.ceph.mon.c.smithi088.stderr:==00:00:35:29.577 34749== Warning: unimplemented fcntl command: 1036 2024-06-03T09:10:34.264 INFO:tasks.ceph.mon.b.smithi192.stderr:==00:00:35:29.566 34228== Warning: unimplemented fcntl command: 1036 2024-06-03T09:10:34.267 INFO:tasks.ceph.mon.c.smithi088.stderr:==00:00:35:29.593 34749== Warning: unimplemented fcntl command: 1036 2024-06-03T09:10:34.872 INFO:tasks.ceph.mon.b.smithi192.stderr:==00:00:35:30.175 34228== Warning: unimplemented fcntl command: 1036 2024-06-03T09:10:34.875 INFO:tasks.ceph.mon.b.smithi192.stderr:==00:00:35:30.178 34228== Warning: unimplemented fcntl command: 1036 2024-06-03T09:10:34.885 INFO:tasks.ceph.mon.b.smithi192.stderr:==00:00:35:30.187 34228== Warning: unimplemented fcntl command: 1036 2024-06-03T09:10:34.893 INFO:tasks.ceph.mon.a.smithi088.stderr:==00:00:35:30.219 34743== Warning: unimplemented fcntl command: 1036 2024-06-03T09:10:34.894 INFO:tasks.ceph.mon.a.smithi088.stderr:==00:00:35:30.220 34743== Warning: unimplemented fcntl command: 1036 2024-06-03T09:10:34.909 INFO:tasks.ceph.mon.a.smithi088.stderr:==00:00:35:30.234 34743== Warning: unimplemented fcntl command: 1036 2024-06-03T09:10:34.935 INFO:tasks.ceph.mon.c.smithi088.stderr:==00:00:35:30.261 34749== Warning: unimplemented fcntl command: 1036 2024-06-03T09:10:34.937 INFO:tasks.ceph.mon.c.smithi088.stderr:==00:00:35:30.262 34749== Warning: unimplemented fcntl command: 1036 2024-06-03T09:10:34.959 INFO:tasks.ceph.mon.c.smithi088.stderr:==00:00:35:30.284 34749== Warning: unimplemented fcntl command: 1036 2024-06-03T09:10:54.243 INFO:tasks.ceph.mon.a.smithi088.stderr:==00:00:35:49.564 34743== Warning: unimplemented fcntl command: 1036 2024-06-03T09:10:54.243 INFO:tasks.ceph.mon.a.smithi088.stderr:==00:00:35:49.567 34743== Warning: unimplemented fcntl command: 1036 2024-06-03T09:10:54.258 INFO:tasks.ceph.mon.a.smithi088.stderr:==00:00:35:49.583 34743== Warning: unimplemented fcntl command: 1036 2024-06-03T09:10:54.294 INFO:tasks.ceph.mon.b.smithi192.stderr:==00:00:35:49.597 34228== Warning: unimplemented fcntl command: 1036 2024-06-03T09:10:54.296 INFO:tasks.ceph.mon.b.smithi192.stderr:==00:00:35:49.598 34228== Warning: unimplemented fcntl command: 1036 2024-06-03T09:10:54.312 INFO:tasks.ceph.mon.b.smithi192.stderr:==00:00:35:49.615 34228== Warning: unimplemented fcntl command: 1036 2024-06-03T09:10:54.314 INFO:tasks.ceph.mon.c.smithi088.stderr:==00:00:35:49.639 34749== Warning: unimplemented fcntl command: 1036 2024-06-03T09:10:54.318 INFO:tasks.ceph.mon.c.smithi088.stderr:==00:00:35:49.644 34749== Warning: unimplemented fcntl command: 1036 2024-06-03T09:10:54.334 INFO:tasks.ceph.mon.c.smithi088.stderr:==00:00:35:49.660 34749== Warning: unimplemented fcntl command: 1036 2024-06-03T09:10:54.923 INFO:tasks.ceph.mon.b.smithi192.stderr:==00:00:35:50.225 34228== Warning: unimplemented fcntl command: 1036 2024-06-03T09:10:54.923 INFO:tasks.ceph.mon.b.smithi192.stderr:==00:00:35:50.226 34228== Warning: unimplemented fcntl command: 1036 2024-06-03T09:10:54.936 INFO:tasks.ceph.mon.b.smithi192.stderr:==00:00:35:50.239 34228== Warning: unimplemented fcntl command: 1036 2024-06-03T09:10:55.000 INFO:tasks.ceph.mon.a.smithi088.stderr:==00:00:35:50.325 34743== Warning: unimplemented fcntl command: 1036 2024-06-03T09:10:55.001 INFO:tasks.ceph.mon.a.smithi088.stderr:==00:00:35:50.327 34743== Warning: unimplemented fcntl command: 1036 2024-06-03T09:10:55.014 INFO:tasks.ceph.mon.a.smithi088.stderr:==00:00:35:50.340 34743== Warning: unimplemented fcntl command: 1036 2024-06-03T09:10:55.024 INFO:tasks.ceph.mon.c.smithi088.stderr:==00:00:35:50.350 34749== Warning: unimplemented fcntl command: 1036 2024-06-03T09:10:55.025 INFO:tasks.ceph.mon.c.smithi088.stderr:==00:00:35:50.351 34749== Warning: unimplemented fcntl command: 1036 2024-06-03T09:10:55.038 INFO:tasks.ceph.mon.c.smithi088.stderr:==00:00:35:50.364 34749== Warning: unimplemented fcntl command: 1036 2024-06-03T09:11:06.130 INFO:tasks.daemonwatchdog.daemon_watchdog:daemon ceph.osd.3 is failed for ~0s 2024-06-03T09:11:06.403 INFO:tasks.ceph.osd.3:Stopped 2024-06-03T09:11:06.403 DEBUG:tasks.ceph.osd.4:waiting for process to exit 2024-06-03T09:11:06.403 INFO:teuthology.orchestra.run:waiting for 300 2024-06-03T09:11:06.416 INFO:tasks.ceph.osd.4.smithi192.stderr:2024-06-03T09:11:06.407+0000 b48a640 -1 received signal: Terminated from /usr/bin/python3 /bin/daemon-helper term env OPENSSL_ia32cap=~0x1000000000000000 valgrind --trace-children=no --child-silent-after-fork=yes --soname-synonyms=somalloc=*tcmalloc* --num-callers=50 --suppressions=/home/ubuntu/cephtest/valgrind.supp --xml=yes --xml-file=/var/log/ceph/valgrind/osd.4.log --time-stamp=yes --vgdb=yes --exit-on-first-error=yes --error-exitcode=42 --tool=memcheck ceph-osd -f --cluster ceph -i 4 (PID: 34774) UID: 0 2024-06-03T09:11:06.419 INFO:tasks.ceph.osd.4.smithi192.stderr:2024-06-03T09:11:06.408+0000 b48a640 -1 osd.4 1024 *** Got signal Terminated *** 2024-06-03T09:11:09.277 INFO:tasks.ceph.mon.a.smithi088.stderr:==00:00:36:04.603 34743== Warning: unimplemented fcntl command: 1036 2024-06-03T09:11:09.281 INFO:tasks.ceph.mon.a.smithi088.stderr:==00:00:36:04.607 34743== Warning: unimplemented fcntl command: 1036 2024-06-03T09:11:09.300 INFO:tasks.ceph.mon.a.smithi088.stderr:==00:00:36:04.625 34743== Warning: unimplemented fcntl command: 1036 2024-06-03T09:11:09.327 INFO:tasks.ceph.mon.b.smithi192.stderr:==00:00:36:04.629 34228== Warning: unimplemented fcntl command: 1036 2024-06-03T09:11:09.328 INFO:tasks.ceph.mon.b.smithi192.stderr:==00:00:36:04.631 34228== Warning: unimplemented fcntl command: 1036 2024-06-03T09:11:09.333 INFO:tasks.ceph.mon.c.smithi088.stderr:==00:00:36:04.659 34749== Warning: unimplemented fcntl command: 1036 2024-06-03T09:11:09.335 INFO:tasks.ceph.mon.c.smithi088.stderr:==00:00:36:04.661 34749== Warning: unimplemented fcntl command: 1036 2024-06-03T09:11:09.345 INFO:tasks.ceph.mon.b.smithi192.stderr:==00:00:36:04.648 34228== Warning: unimplemented fcntl command: 1036 2024-06-03T09:11:09.352 INFO:tasks.ceph.mon.c.smithi088.stderr:==00:00:36:04.678 34749== Warning: unimplemented fcntl command: 1036 2024-06-03T09:11:11.936 INFO:tasks.daemonwatchdog.daemon_watchdog:daemon ceph.osd.4 is failed for ~0s 2024-06-03T09:11:11.936 INFO:tasks.daemonwatchdog.daemon_watchdog:daemon ceph.osd.3 has been restored 2024-06-03T09:11:12.505 INFO:tasks.ceph.osd.4:Stopped 2024-06-03T09:11:12.505 DEBUG:tasks.ceph.osd.5:waiting for process to exit 2024-06-03T09:11:12.505 INFO:teuthology.orchestra.run:waiting for 300 2024-06-03T09:11:12.518 INFO:tasks.ceph.osd.5.smithi192.stderr:2024-06-03T09:11:12.510+0000 b48a640 -1 received signal: Terminated from /usr/bin/python3 /bin/daemon-helper term env OPENSSL_ia32cap=~0x1000000000000000 valgrind --trace-children=no --child-silent-after-fork=yes --soname-synonyms=somalloc=*tcmalloc* --num-callers=50 --suppressions=/home/ubuntu/cephtest/valgrind.supp --xml=yes --xml-file=/var/log/ceph/valgrind/osd.5.log --time-stamp=yes --vgdb=yes --exit-on-first-error=yes --error-exitcode=42 --tool=memcheck ceph-osd -f --cluster ceph -i 5 (PID: 34778) UID: 0 2024-06-03T09:11:12.519 INFO:tasks.ceph.osd.5.smithi192.stderr:2024-06-03T09:11:12.511+0000 b48a640 -1 osd.5 1026 *** Got signal Terminated *** 2024-06-03T09:11:14.374 INFO:tasks.ceph.mon.a.smithi088.stderr:==00:00:36:09.699 34743== Warning: unimplemented fcntl command: 1036 2024-06-03T09:11:14.376 INFO:tasks.ceph.mon.a.smithi088.stderr:==00:00:36:09.702 34743== Warning: unimplemented fcntl command: 1036 2024-06-03T09:11:14.391 INFO:tasks.ceph.mon.a.smithi088.stderr:==00:00:36:09.717 34743== Warning: unimplemented fcntl command: 1036 2024-06-03T09:11:14.428 INFO:tasks.ceph.mon.c.smithi088.stderr:==00:00:36:09.753 34749== Warning: unimplemented fcntl command: 1036 2024-06-03T09:11:14.429 INFO:tasks.ceph.mon.c.smithi088.stderr:==00:00:36:09.755 34749== Warning: unimplemented fcntl command: 1036 2024-06-03T09:11:14.430 INFO:tasks.ceph.mon.b.smithi192.stderr:==00:00:36:09.733 34228== Warning: unimplemented fcntl command: 1036 2024-06-03T09:11:14.434 INFO:tasks.ceph.mon.b.smithi192.stderr:==00:00:36:09.736 34228== Warning: unimplemented fcntl command: 1036 2024-06-03T09:11:14.442 INFO:tasks.ceph.mon.c.smithi088.stderr:==00:00:36:09.767 34749== Warning: unimplemented fcntl command: 1036 2024-06-03T09:11:14.447 INFO:tasks.ceph.mon.b.smithi192.stderr:==00:00:36:09.750 34228== Warning: unimplemented fcntl command: 1036 2024-06-03T09:11:17.640 INFO:tasks.daemonwatchdog.daemon_watchdog:daemon ceph.osd.5 is failed for ~0s 2024-06-03T09:11:17.641 INFO:tasks.daemonwatchdog.daemon_watchdog:daemon ceph.osd.4 has been restored 2024-06-03T09:11:18.607 INFO:tasks.ceph.osd.5:Stopped 2024-06-03T09:11:18.607 DEBUG:tasks.ceph.osd.6:waiting for process to exit 2024-06-03T09:11:18.608 INFO:teuthology.orchestra.run:waiting for 300 2024-06-03T09:11:18.620 INFO:tasks.ceph.osd.6.smithi192.stderr:2024-06-03T09:11:18.612+0000 b48a640 -1 received signal: Terminated from /usr/bin/python3 /bin/daemon-helper term env OPENSSL_ia32cap=~0x1000000000000000 valgrind --trace-children=no --child-silent-after-fork=yes --soname-synonyms=somalloc=*tcmalloc* --num-callers=50 --suppressions=/home/ubuntu/cephtest/valgrind.supp --xml=yes --xml-file=/var/log/ceph/valgrind/osd.6.log --time-stamp=yes --vgdb=yes --exit-on-first-error=yes --error-exitcode=42 --tool=memcheck ceph-osd -f --cluster ceph -i 6 (PID: 34779) UID: 0 2024-06-03T09:11:18.620 INFO:tasks.ceph.osd.6.smithi192.stderr:2024-06-03T09:11:18.613+0000 b48a640 -1 osd.6 1028 *** Got signal Terminated *** 2024-06-03T09:11:23.246 INFO:tasks.daemonwatchdog.daemon_watchdog:daemon ceph.osd.6 is failed for ~0s 2024-06-03T09:11:23.246 INFO:tasks.daemonwatchdog.daemon_watchdog:daemon ceph.osd.5 has been restored 2024-06-03T09:11:24.709 INFO:tasks.ceph.osd.6:Stopped 2024-06-03T09:11:24.709 DEBUG:tasks.ceph.osd.7:waiting for process to exit 2024-06-03T09:11:24.709 INFO:teuthology.orchestra.run:waiting for 300 2024-06-03T09:11:24.722 INFO:tasks.ceph.osd.7.smithi192.stderr:2024-06-03T09:11:24.714+0000 b48a640 -1 received signal: Terminated from /usr/bin/python3 /bin/daemon-helper term env OPENSSL_ia32cap=~0x1000000000000000 valgrind --trace-children=no --child-silent-after-fork=yes --soname-synonyms=somalloc=*tcmalloc* --num-callers=50 --suppressions=/home/ubuntu/cephtest/valgrind.supp --xml=yes --xml-file=/var/log/ceph/valgrind/osd.7.log --time-stamp=yes --vgdb=yes --exit-on-first-error=yes --error-exitcode=42 --tool=memcheck ceph-osd -f --cluster ceph -i 7 (PID: 34783) UID: 0 2024-06-03T09:11:24.722 INFO:tasks.ceph.osd.7.smithi192.stderr:2024-06-03T09:11:24.715+0000 b48a640 -1 osd.7 1028 *** Got signal Terminated *** 2024-06-03T09:11:28.751 INFO:tasks.daemonwatchdog.daemon_watchdog:daemon ceph.osd.7 is failed for ~0s 2024-06-03T09:11:28.751 INFO:tasks.daemonwatchdog.daemon_watchdog:daemon ceph.osd.6 has been restored 2024-06-03T09:11:29.342 INFO:tasks.ceph.mon.a.smithi088.stderr:==00:00:36:24.667 34743== Warning: unimplemented fcntl command: 1036 2024-06-03T09:11:29.343 INFO:tasks.ceph.mon.a.smithi088.stderr:==00:00:36:24.669 34743== Warning: unimplemented fcntl command: 1036 2024-06-03T09:11:29.357 INFO:tasks.ceph.mon.a.smithi088.stderr:==00:00:36:24.682 34743== Warning: unimplemented fcntl command: 1036 2024-06-03T09:11:29.390 INFO:tasks.ceph.mon.b.smithi192.stderr:==00:00:36:24.692 34228== Warning: unimplemented fcntl command: 1036 2024-06-03T09:11:29.391 INFO:tasks.ceph.mon.b.smithi192.stderr:==00:00:36:24.693 34228== Warning: unimplemented fcntl command: 1036 2024-06-03T09:11:29.394 INFO:tasks.ceph.mon.c.smithi088.stderr:==00:00:36:24.720 34749== Warning: unimplemented fcntl command: 1036 2024-06-03T09:11:29.396 INFO:tasks.ceph.mon.c.smithi088.stderr:==00:00:36:24.722 34749== Warning: unimplemented fcntl command: 1036 2024-06-03T09:11:29.405 INFO:tasks.ceph.mon.b.smithi192.stderr:==00:00:36:24.707 34228== Warning: unimplemented fcntl command: 1036 2024-06-03T09:11:29.410 INFO:tasks.ceph.mon.c.smithi088.stderr:==00:00:36:24.736 34749== Warning: unimplemented fcntl command: 1036 2024-06-03T09:11:30.811 INFO:tasks.ceph.osd.7:Stopped 2024-06-03T09:11:30.811 INFO:teuthology.misc:Shutting down mgr daemons... 2024-06-03T09:11:30.811 DEBUG:tasks.ceph.mgr.y:waiting for process to exit 2024-06-03T09:11:30.812 INFO:teuthology.orchestra.run:waiting for 300 2024-06-03T09:11:30.835 INFO:tasks.ceph.mgr.y:Stopped 2024-06-03T09:11:30.836 DEBUG:tasks.ceph.mgr.x:waiting for process to exit 2024-06-03T09:11:30.836 INFO:teuthology.orchestra.run:waiting for 300 2024-06-03T09:11:30.837 INFO:tasks.ceph.mgr.x.smithi192.stderr:2024-06-03T09:11:30.837+0000 7f0e67702640 -1 received signal: Terminated from /usr/bin/python3 /bin/daemon-helper term ceph-mgr -f --cluster ceph -i x (PID: 34226) UID: 0 2024-06-03T09:11:30.837 INFO:tasks.ceph.mgr.x.smithi192.stderr:2024-06-03T09:11:30.837+0000 7f0e67702640 -1 mgr handle_mgr_signal *** Got signal Terminated *** 2024-06-03T09:11:30.863 INFO:tasks.ceph.mgr.x:Stopped 2024-06-03T09:11:30.864 INFO:teuthology.misc:Shutting down mon daemons... 2024-06-03T09:11:30.864 DEBUG:tasks.ceph.mon.a:waiting for process to exit 2024-06-03T09:11:30.864 INFO:teuthology.orchestra.run:waiting for 300 2024-06-03T09:11:30.891 INFO:tasks.ceph.mon.a.smithi088.stderr:2024-06-03T09:11:30.880+0000 9f5e640 -1 received signal: Terminated from /usr/bin/python3 /bin/daemon-helper term env OPENSSL_ia32cap=~0x1000000000000000 valgrind --trace-children=no --child-silent-after-fork=yes --soname-synonyms=somalloc=*tcmalloc* --num-callers=50 --suppressions=/home/ubuntu/cephtest/valgrind.supp --xml=yes --xml-file=/var/log/ceph/valgrind/mon.a.log --time-stamp=yes --vgdb=yes --exit-on-first-error=yes --error-exitcode=42 --tool=memcheck --leak-check=full --show-reachable=yes ceph-mon -f --cluster ceph -i a (PID: 34719) UID: 0 2024-06-03T09:11:30.892 INFO:tasks.ceph.mon.a.smithi088.stderr:2024-06-03T09:11:30.882+0000 9f5e640 -1 mon.a@0(leader) e1 *** Got Signal Terminated *** 2024-06-03T09:11:34.054 INFO:tasks.daemonwatchdog.daemon_watchdog:daemon ceph.osd.7 has been restored 2024-06-03T09:11:40.687 INFO:tasks.ceph.mon.a.smithi088.stderr:==00:00:36:36.012 34743== 2024-06-03T09:11:40.687 INFO:tasks.ceph.mon.a.smithi088.stderr:==00:00:36:36.012 34743== Exit program on first error (--exit-on-first-error=yes) 2024-06-03T09:11:40.707 INFO:tasks.ceph.mon.a.smithi088.stderr:daemon-helper: command failed with exit status 42 2024-06-03T09:11:43.067 DEBUG:teuthology.orchestra.run:got remote process result: 42 2024-06-03T09:11:43.067 ERROR:teuthology.orchestra.daemon.state:Error while waiting for process to exit Traceback (most recent call last): File "/home/teuthworker/src/git.ceph.com_teuthology_861a8dcf7aa816a26e13f039336f7ed0a9aec0fa/teuthology/orchestra/daemon/state.py", line 139, in stop run.wait([self.proc], timeout=timeout) File "/home/teuthworker/src/git.ceph.com_teuthology_861a8dcf7aa816a26e13f039336f7ed0a9aec0fa/teuthology/orchestra/run.py", line 479, in wait proc.wait() File "/home/teuthworker/src/git.ceph.com_teuthology_861a8dcf7aa816a26e13f039336f7ed0a9aec0fa/teuthology/orchestra/run.py", line 161, in wait self._raise_for_status() File "/home/teuthworker/src/git.ceph.com_teuthology_861a8dcf7aa816a26e13f039336f7ed0a9aec0fa/teuthology/orchestra/run.py", line 181, in _raise_for_status raise CommandFailedError( teuthology.exceptions.CommandFailedError: Command failed on smithi088 with status 42: "cd /home/ubuntu/cephtest && sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage daemon-helper term env 'OPENSSL_ia32cap=~0x1000000000000000' valgrind --trace-children=no --child-silent-after-fork=yes '--soname-synonyms=somalloc=*tcmalloc*' --num-callers=50 --suppressions=/home/ubuntu/cephtest/valgrind.supp --xml=yes --xml-file=/var/log/ceph/valgrind/mon.a.log --time-stamp=yes --vgdb=yes --exit-on-first-error=yes --error-exitcode=42 --tool=memcheck --leak-check=full --show-reachable=yes ceph-mon -f --cluster ceph -i a" 2024-06-03T09:11:43.068 INFO:tasks.ceph.mon.a:Stopped 2024-06-03T09:11:43.069 DEBUG:tasks.ceph.mon.c:waiting for process to exit 2024-06-03T09:11:43.069 INFO:teuthology.orchestra.run:waiting for 300 2024-06-03T09:11:43.086 INFO:tasks.ceph.mon.c.smithi088.stderr:2024-06-03T09:11:43.073+0000 9f5e640 -1 received signal: Terminated from /usr/bin/python3 /bin/daemon-helper term env OPENSSL_ia32cap=~0x1000000000000000 valgrind --trace-children=no --child-silent-after-fork=yes --soname-synonyms=somalloc=*tcmalloc* --num-callers=50 --suppressions=/home/ubuntu/cephtest/valgrind.supp --xml=yes --xml-file=/var/log/ceph/valgrind/mon.c.log --time-stamp=yes --vgdb=yes --exit-on-first-error=yes --error-exitcode=42 --tool=memcheck --leak-check=full --show-reachable=yes ceph-mon -f --cluster ceph -i c (PID: 34721) UID: 0 2024-06-03T09:11:43.086 INFO:tasks.ceph.mon.c.smithi088.stderr:2024-06-03T09:11:43.074+0000 9f5e640 -1 mon.c@2(electing) e1 *** Got Signal Terminated *** 2024-06-03T09:11:43.589 INFO:tasks.ceph.mon.c.smithi088.stderr:==00:00:36:38.915 34749== 2024-06-03T09:11:43.590 INFO:tasks.ceph.mon.c.smithi088.stderr:==00:00:36:38.915 34749== Exit program on first error (--exit-on-first-error=yes) 2024-06-03T09:11:43.606 INFO:tasks.ceph.mon.c.smithi088.stderr:daemon-helper: command failed with exit status 42 2024-06-03T09:11:44.460 INFO:tasks.daemonwatchdog.daemon_watchdog:daemon ceph.mon.c is failed for ~0s 2024-06-03T09:11:49.170 DEBUG:teuthology.orchestra.run:got remote process result: 42 2024-06-03T09:11:49.171 ERROR:teuthology.orchestra.daemon.state:Error while waiting for process to exit Traceback (most recent call last): File "/home/teuthworker/src/git.ceph.com_teuthology_861a8dcf7aa816a26e13f039336f7ed0a9aec0fa/teuthology/orchestra/daemon/state.py", line 139, in stop run.wait([self.proc], timeout=timeout) File "/home/teuthworker/src/git.ceph.com_teuthology_861a8dcf7aa816a26e13f039336f7ed0a9aec0fa/teuthology/orchestra/run.py", line 479, in wait proc.wait() File "/home/teuthworker/src/git.ceph.com_teuthology_861a8dcf7aa816a26e13f039336f7ed0a9aec0fa/teuthology/orchestra/run.py", line 161, in wait self._raise_for_status() File "/home/teuthworker/src/git.ceph.com_teuthology_861a8dcf7aa816a26e13f039336f7ed0a9aec0fa/teuthology/orchestra/run.py", line 181, in _raise_for_status raise CommandFailedError( teuthology.exceptions.CommandFailedError: Command failed on smithi088 with status 42: "cd /home/ubuntu/cephtest && sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage daemon-helper term env 'OPENSSL_ia32cap=~0x1000000000000000' valgrind --trace-children=no --child-silent-after-fork=yes '--soname-synonyms=somalloc=*tcmalloc*' --num-callers=50 --suppressions=/home/ubuntu/cephtest/valgrind.supp --xml=yes --xml-file=/var/log/ceph/valgrind/mon.c.log --time-stamp=yes --vgdb=yes --exit-on-first-error=yes --error-exitcode=42 --tool=memcheck --leak-check=full --show-reachable=yes ceph-mon -f --cluster ceph -i c" 2024-06-03T09:11:49.171 INFO:tasks.ceph.mon.c:Stopped 2024-06-03T09:11:49.171 DEBUG:tasks.ceph.mon.b:waiting for process to exit 2024-06-03T09:11:49.172 INFO:teuthology.orchestra.run:waiting for 300 2024-06-03T09:11:49.187 INFO:tasks.ceph.mon.b.smithi192.stderr:2024-06-03T09:11:49.176+0000 9f5e640 -1 received signal: Terminated from /usr/bin/python3 /bin/daemon-helper term env OPENSSL_ia32cap=~0x1000000000000000 valgrind --trace-children=no --child-silent-after-fork=yes --soname-synonyms=somalloc=*tcmalloc* --num-callers=50 --suppressions=/home/ubuntu/cephtest/valgrind.supp --xml=yes --xml-file=/var/log/ceph/valgrind/mon.b.log --time-stamp=yes --vgdb=yes --exit-on-first-error=yes --error-exitcode=42 --tool=memcheck --leak-check=full --show-reachable=yes ceph-mon -f --cluster ceph -i b (PID: 34223) UID: 0 2024-06-03T09:11:49.188 INFO:tasks.ceph.mon.b.smithi192.stderr:2024-06-03T09:11:49.177+0000 9f5e640 -1 mon.b@1(electing) e1 *** Got Signal Terminated *** 2024-06-03T09:11:49.562 INFO:tasks.daemonwatchdog.daemon_watchdog:daemon ceph.mon.c has been restored 2024-06-03T09:13:11.684 INFO:tasks.ceph.mon.b.smithi192.stderr:==00:00:38:06.985 34228== 2024-06-03T09:13:11.684 INFO:tasks.ceph.mon.b.smithi192.stderr:==00:00:38:06.985 34228== Exit program on first error (--exit-on-first-error=yes) 2024-06-03T09:13:11.701 INFO:tasks.ceph.mon.b.smithi192.stderr:daemon-helper: command failed with exit status 42 2024-06-03T09:13:14.591 DEBUG:teuthology.orchestra.run:got remote process result: 42 2024-06-03T09:13:14.591 ERROR:teuthology.orchestra.daemon.state:Error while waiting for process to exit Traceback (most recent call last): File "/home/teuthworker/src/git.ceph.com_teuthology_861a8dcf7aa816a26e13f039336f7ed0a9aec0fa/teuthology/orchestra/daemon/state.py", line 139, in stop run.wait([self.proc], timeout=timeout) File "/home/teuthworker/src/git.ceph.com_teuthology_861a8dcf7aa816a26e13f039336f7ed0a9aec0fa/teuthology/orchestra/run.py", line 479, in wait proc.wait() File "/home/teuthworker/src/git.ceph.com_teuthology_861a8dcf7aa816a26e13f039336f7ed0a9aec0fa/teuthology/orchestra/run.py", line 161, in wait self._raise_for_status() File "/home/teuthworker/src/git.ceph.com_teuthology_861a8dcf7aa816a26e13f039336f7ed0a9aec0fa/teuthology/orchestra/run.py", line 181, in _raise_for_status raise CommandFailedError( teuthology.exceptions.CommandFailedError: Command failed on smithi192 with status 42: "cd /home/ubuntu/cephtest && sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage daemon-helper term env 'OPENSSL_ia32cap=~0x1000000000000000' valgrind --trace-children=no --child-silent-after-fork=yes '--soname-synonyms=somalloc=*tcmalloc*' --num-callers=50 --suppressions=/home/ubuntu/cephtest/valgrind.supp --xml=yes --xml-file=/var/log/ceph/valgrind/mon.b.log --time-stamp=yes --vgdb=yes --exit-on-first-error=yes --error-exitcode=42 --tool=memcheck --leak-check=full --show-reachable=yes ceph-mon -f --cluster ceph -i b" 2024-06-03T09:13:14.592 INFO:tasks.ceph.mon.b:Stopped 2024-06-03T09:13:14.592 INFO:tasks.ceph:Checking cluster log for badness... 2024-06-03T09:13:14.593 DEBUG:teuthology.orchestra.run.smithi088:> sudo egrep '\[ERR\]|\[WRN\]|\[SEC\]' /var/log/ceph/ceph.log | egrep -v '\(MDS_ALL_DOWN\)' | egrep -v '\(MDS_UP_LESS_THAN_MAX\)' | egrep -v '\(OSD_SLOW_PING_TIME' | egrep -v 'reached quota' | egrep -v 'overall HEALTH_' | egrep -v '\(CACHE_POOL_NO_HIT_SET\)' | egrep -v '\(POOL_FULL\)' | egrep -v '\(SMALLER_PGP_NUM\)' | egrep -v '\(SLOW_OPS\)' | egrep -v '\(CACHE_POOL_NEAR_FULL\)' | egrep -v '\(POOL_APP_NOT_ENABLED\)' | egrep -v '\(PG_AVAILABILITY\)' | egrep -v '\(OBJECT_MISPLACED\)' | egrep -v 'slow request' | egrep -v 'overall HEALTH_' | egrep -v '\(PG_' | egrep -v '\(MON_DOWN\)' | egrep -v '\(SLOW_OPS\)' | egrep -v 'slow request' | head -n 1 2024-06-03T09:13:14.632 INFO:teuthology.orchestra.run.smithi088.stdout:2024-06-03T08:40:55.510454+0000 osd.1 (osd.1) 6 : cluster [WRN] osd.1 ep: 122 scrubber::ReplicaReservations pg[11.7]: timeout on replica reservations (since 2024-06-03 08:40:50) 2024-06-03T09:13:14.632 WARNING:tasks.ceph:Found errors (ERR|WRN|SEC) in cluster log 2024-06-03T09:13:14.632 DEBUG:teuthology.orchestra.run.smithi088:> sudo egrep '\[SEC\]' /var/log/ceph/ceph.log | egrep -v '\(MDS_ALL_DOWN\)' | egrep -v '\(MDS_UP_LESS_THAN_MAX\)' | egrep -v '\(OSD_SLOW_PING_TIME' | egrep -v 'reached quota' | egrep -v 'overall HEALTH_' | egrep -v '\(CACHE_POOL_NO_HIT_SET\)' | egrep -v '\(POOL_FULL\)' | egrep -v '\(SMALLER_PGP_NUM\)' | egrep -v '\(SLOW_OPS\)' | egrep -v '\(CACHE_POOL_NEAR_FULL\)' | egrep -v '\(POOL_APP_NOT_ENABLED\)' | egrep -v '\(PG_AVAILABILITY\)' | egrep -v '\(OBJECT_MISPLACED\)' | egrep -v 'slow request' | egrep -v 'overall HEALTH_' | egrep -v '\(PG_' | egrep -v '\(MON_DOWN\)' | egrep -v '\(SLOW_OPS\)' | egrep -v 'slow request' | head -n 1 2024-06-03T09:13:14.705 DEBUG:teuthology.orchestra.run.smithi088:> sudo egrep '\[ERR\]' /var/log/ceph/ceph.log | egrep -v '\(MDS_ALL_DOWN\)' | egrep -v '\(MDS_UP_LESS_THAN_MAX\)' | egrep -v '\(OSD_SLOW_PING_TIME' | egrep -v 'reached quota' | egrep -v 'overall HEALTH_' | egrep -v '\(CACHE_POOL_NO_HIT_SET\)' | egrep -v '\(POOL_FULL\)' | egrep -v '\(SMALLER_PGP_NUM\)' | egrep -v '\(SLOW_OPS\)' | egrep -v '\(CACHE_POOL_NEAR_FULL\)' | egrep -v '\(POOL_APP_NOT_ENABLED\)' | egrep -v '\(PG_AVAILABILITY\)' | egrep -v '\(OBJECT_MISPLACED\)' | egrep -v 'slow request' | egrep -v 'overall HEALTH_' | egrep -v '\(PG_' | egrep -v '\(MON_DOWN\)' | egrep -v '\(SLOW_OPS\)' | egrep -v 'slow request' | head -n 1 2024-06-03T09:13:14.777 DEBUG:teuthology.orchestra.run.smithi088:> sudo egrep '\[WRN\]' /var/log/ceph/ceph.log | egrep -v '\(MDS_ALL_DOWN\)' | egrep -v '\(MDS_UP_LESS_THAN_MAX\)' | egrep -v '\(OSD_SLOW_PING_TIME' | egrep -v 'reached quota' | egrep -v 'overall HEALTH_' | egrep -v '\(CACHE_POOL_NO_HIT_SET\)' | egrep -v '\(POOL_FULL\)' | egrep -v '\(SMALLER_PGP_NUM\)' | egrep -v '\(SLOW_OPS\)' | egrep -v '\(CACHE_POOL_NEAR_FULL\)' | egrep -v '\(POOL_APP_NOT_ENABLED\)' | egrep -v '\(PG_AVAILABILITY\)' | egrep -v '\(OBJECT_MISPLACED\)' | egrep -v 'slow request' | egrep -v 'overall HEALTH_' | egrep -v '\(PG_' | egrep -v '\(MON_DOWN\)' | egrep -v '\(SLOW_OPS\)' | egrep -v 'slow request' | head -n 1 2024-06-03T09:13:14.848 INFO:teuthology.orchestra.run.smithi088.stdout:2024-06-03T08:40:55.510454+0000 osd.1 (osd.1) 6 : cluster [WRN] osd.1 ep: 122 scrubber::ReplicaReservations pg[11.7]: timeout on replica reservations (since 2024-06-03 08:40:50) 2024-06-03T09:13:14.849 INFO:tasks.ceph:Unmounting /var/lib/ceph/osd/ceph-0 on ubuntu@smithi088.front.sepia.ceph.com 2024-06-03T09:13:14.849 DEBUG:teuthology.orchestra.run.smithi088:> sync && sudo umount -f /var/lib/ceph/osd/ceph-0 2024-06-03T09:13:14.958 INFO:tasks.ceph:Unmounting /var/lib/ceph/osd/ceph-1 on ubuntu@smithi088.front.sepia.ceph.com 2024-06-03T09:13:14.958 DEBUG:teuthology.orchestra.run.smithi088:> sync && sudo umount -f /var/lib/ceph/osd/ceph-1 2024-06-03T09:13:15.026 INFO:tasks.ceph:Unmounting /var/lib/ceph/osd/ceph-2 on ubuntu@smithi088.front.sepia.ceph.com 2024-06-03T09:13:15.026 DEBUG:teuthology.orchestra.run.smithi088:> sync && sudo umount -f /var/lib/ceph/osd/ceph-2 2024-06-03T09:13:15.099 INFO:tasks.ceph:Unmounting /var/lib/ceph/osd/ceph-3 on ubuntu@smithi088.front.sepia.ceph.com 2024-06-03T09:13:15.099 DEBUG:teuthology.orchestra.run.smithi088:> sync && sudo umount -f /var/lib/ceph/osd/ceph-3 2024-06-03T09:13:15.157 INFO:tasks.ceph:Unmounting /var/lib/ceph/osd/ceph-4 on ubuntu@smithi192.front.sepia.ceph.com 2024-06-03T09:13:15.157 DEBUG:teuthology.orchestra.run.smithi192:> sync && sudo umount -f /var/lib/ceph/osd/ceph-4 2024-06-03T09:13:15.303 INFO:tasks.ceph:Unmounting /var/lib/ceph/osd/ceph-5 on ubuntu@smithi192.front.sepia.ceph.com 2024-06-03T09:13:15.303 DEBUG:teuthology.orchestra.run.smithi192:> sync && sudo umount -f /var/lib/ceph/osd/ceph-5 2024-06-03T09:13:15.366 INFO:tasks.ceph:Unmounting /var/lib/ceph/osd/ceph-6 on ubuntu@smithi192.front.sepia.ceph.com 2024-06-03T09:13:15.367 DEBUG:teuthology.orchestra.run.smithi192:> sync && sudo umount -f /var/lib/ceph/osd/ceph-6 2024-06-03T09:13:15.424 INFO:tasks.ceph:Unmounting /var/lib/ceph/osd/ceph-7 on ubuntu@smithi192.front.sepia.ceph.com 2024-06-03T09:13:15.424 DEBUG:teuthology.orchestra.run.smithi192:> sync && sudo umount -f /var/lib/ceph/osd/ceph-7 2024-06-03T09:13:15.482 INFO:tasks.ceph:Archiving mon data... 2024-06-03T09:13:15.484 DEBUG:teuthology.misc:Transferring archived files from smithi088:/var/lib/ceph/mon/ceph-a to /home/teuthworker/archive/teuthology-2024-06-02_22:00:03-rados-reef-distro-default-smithi/7739049/data/mon.a.tgz 2024-06-03T09:13:15.484 DEBUG:teuthology.orchestra.run.smithi088:> mktemp 2024-06-03T09:13:15.500 INFO:teuthology.orchestra.run.smithi088.stdout:/tmp/tmp.2JoZ8LvMMm 2024-06-03T09:13:15.500 DEBUG:teuthology.orchestra.run.smithi088:> sudo tar cz -f - -C /var/lib/ceph/mon/ceph-a -- . > /tmp/tmp.2JoZ8LvMMm 2024-06-03T09:13:15.647 DEBUG:teuthology.orchestra.run.smithi088:> sudo chmod 0666 /tmp/tmp.2JoZ8LvMMm 2024-06-03T09:13:15.730 DEBUG:teuthology.orchestra.remote:smithi088:/tmp/tmp.2JoZ8LvMMm is 452KB 2024-06-03T09:13:15.829 DEBUG:teuthology.orchestra.run.smithi088:> rm -fr /tmp/tmp.2JoZ8LvMMm 2024-06-03T09:13:15.843 DEBUG:teuthology.misc:Transferring archived files from smithi088:/var/lib/ceph/mon/ceph-c to /home/teuthworker/archive/teuthology-2024-06-02_22:00:03-rados-reef-distro-default-smithi/7739049/data/mon.c.tgz 2024-06-03T09:13:15.844 DEBUG:teuthology.orchestra.run.smithi088:> mktemp 2024-06-03T09:13:15.898 INFO:teuthology.orchestra.run.smithi088.stdout:/tmp/tmp.S7djLUwjg7 2024-06-03T09:13:15.898 DEBUG:teuthology.orchestra.run.smithi088:> sudo tar cz -f - -C /var/lib/ceph/mon/ceph-c -- . > /tmp/tmp.S7djLUwjg7 2024-06-03T09:13:16.034 DEBUG:teuthology.orchestra.run.smithi088:> sudo chmod 0666 /tmp/tmp.S7djLUwjg7 2024-06-03T09:13:16.116 DEBUG:teuthology.orchestra.remote:smithi088:/tmp/tmp.S7djLUwjg7 is 453KB 2024-06-03T09:13:16.213 DEBUG:teuthology.orchestra.run.smithi088:> rm -fr /tmp/tmp.S7djLUwjg7 2024-06-03T09:13:16.227 DEBUG:teuthology.misc:Transferring archived files from smithi192:/var/lib/ceph/mon/ceph-b to /home/teuthworker/archive/teuthology-2024-06-02_22:00:03-rados-reef-distro-default-smithi/7739049/data/mon.b.tgz 2024-06-03T09:13:16.227 DEBUG:teuthology.orchestra.run.smithi192:> mktemp 2024-06-03T09:13:16.246 INFO:teuthology.orchestra.run.smithi192.stdout:/tmp/tmp.qu5uhIBgum 2024-06-03T09:13:16.246 DEBUG:teuthology.orchestra.run.smithi192:> sudo tar cz -f - -C /var/lib/ceph/mon/ceph-b -- . > /tmp/tmp.qu5uhIBgum 2024-06-03T09:13:16.407 DEBUG:teuthology.orchestra.run.smithi192:> sudo chmod 0666 /tmp/tmp.qu5uhIBgum 2024-06-03T09:13:16.491 DEBUG:teuthology.orchestra.remote:smithi192:/tmp/tmp.qu5uhIBgum is 453KB 2024-06-03T09:13:16.587 DEBUG:teuthology.orchestra.run.smithi192:> rm -fr /tmp/tmp.qu5uhIBgum 2024-06-03T09:13:16.602 INFO:tasks.ceph:Cleaning ceph cluster... 2024-06-03T09:13:16.602 DEBUG:teuthology.orchestra.run.smithi088:> sudo rm -rf -- /etc/ceph/ceph.conf /etc/ceph/ceph.keyring /home/ubuntu/cephtest/ceph.data /home/ubuntu/cephtest/ceph.monmap /home/ubuntu/cephtest/../*.pid 2024-06-03T09:13:16.604 DEBUG:teuthology.orchestra.run.smithi192:> sudo rm -rf -- /etc/ceph/ceph.conf /etc/ceph/ceph.keyring /home/ubuntu/cephtest/ceph.data /home/ubuntu/cephtest/ceph.monmap /home/ubuntu/cephtest/../*.pid 2024-06-03T09:13:16.666 INFO:tasks.ceph:Checking for errors in any valgrind logs... 2024-06-03T09:13:16.666 DEBUG:teuthology.orchestra.run.smithi088:> sudo zgrep '' /var/log/ceph/valgrind/* /dev/null | sort | uniq 2024-06-03T09:13:16.669 DEBUG:teuthology.orchestra.run.smithi192:> sudo zgrep '' /var/log/ceph/valgrind/* /dev/null | sort | uniq 2024-06-03T09:13:16.707 INFO:teuthology.orchestra.run.smithi088.stdout:/var/log/ceph/valgrind/mon.a.log: Leak_StillReachable 2024-06-03T09:13:16.707 INFO:teuthology.orchestra.run.smithi088.stdout:/var/log/ceph/valgrind/mon.c.log: Leak_StillReachable 2024-06-03T09:13:16.709 DEBUG:tasks.ceph:file /var/log/ceph/valgrind/mon.a.log kind Leak_StillReachable 2024-06-03T09:13:16.709 ERROR:tasks.ceph:saw valgrind issue Leak_StillReachable in /var/log/ceph/valgrind/mon.a.log 2024-06-03T09:13:16.709 DEBUG:tasks.ceph:file /var/log/ceph/valgrind/mon.c.log kind Leak_StillReachable 2024-06-03T09:13:16.709 ERROR:tasks.ceph:saw valgrind issue Leak_StillReachable in /var/log/ceph/valgrind/mon.c.log 2024-06-03T09:13:16.745 INFO:teuthology.orchestra.run.smithi192.stdout:/var/log/ceph/valgrind/mon.b.log: Leak_StillReachable 2024-06-03T09:13:16.745 DEBUG:tasks.ceph:file /var/log/ceph/valgrind/mon.b.log kind Leak_StillReachable 2024-06-03T09:13:16.745 ERROR:tasks.ceph:saw valgrind issue Leak_StillReachable in /var/log/ceph/valgrind/mon.b.log 2024-06-03T09:13:16.745 INFO:tasks.ceph:Archiving crash dumps... 2024-06-03T09:13:16.747 DEBUG:teuthology.misc:Transferring archived files from smithi088:/var/lib/ceph/crash to /home/teuthworker/archive/teuthology-2024-06-02_22:00:03-rados-reef-distro-default-smithi/7739049/remote/smithi088/crash 2024-06-03T09:13:16.748 DEBUG:teuthology.orchestra.run.smithi088:> sudo tar c -f - -C /var/lib/ceph/crash -- . 2024-06-03T09:13:16.773 DEBUG:teuthology.misc:Transferring archived files from smithi192:/var/lib/ceph/crash to /home/teuthworker/archive/teuthology-2024-06-02_22:00:03-rados-reef-distro-default-smithi/7739049/remote/smithi192/crash 2024-06-03T09:13:16.774 DEBUG:teuthology.orchestra.run.smithi192:> sudo tar c -f - -C /var/lib/ceph/crash -- . 2024-06-03T09:13:16.808 INFO:tasks.ceph:Compressing logs... 2024-06-03T09:13:16.809 DEBUG:teuthology.orchestra.run.smithi088:> sudo find /var/log/ceph -name '*.log' -print0 | sudo xargs -0 --no-run-if-empty -- gzip -- 2024-06-03T09:13:16.817 DEBUG:teuthology.orchestra.run.smithi192:> sudo find /var/log/ceph -name '*.log' -print0 | sudo xargs -0 --no-run-if-empty -- gzip -- 2024-06-03T09:13:41.069 INFO:tasks.ceph:Archiving logs... 2024-06-03T09:13:41.070 DEBUG:teuthology.misc:Transferring archived files from smithi088:/var/log/ceph to /home/teuthworker/archive/teuthology-2024-06-02_22:00:03-rados-reef-distro-default-smithi/7739049/remote/smithi088/log 2024-06-03T09:13:41.071 DEBUG:teuthology.orchestra.run.smithi088:> sudo tar c -f - -C /var/log/ceph -- . 2024-06-03T09:13:44.415 DEBUG:teuthology.misc:Transferring archived files from smithi192:/var/log/ceph to /home/teuthworker/archive/teuthology-2024-06-02_22:00:03-rados-reef-distro-default-smithi/7739049/remote/smithi192/log 2024-06-03T09:13:44.416 DEBUG:teuthology.orchestra.run.smithi192:> sudo tar c -f - -C /var/log/ceph -- . 2024-06-03T09:13:46.880 ERROR:teuthology.run_tasks:Manager failed: ceph Traceback (most recent call last): File "/home/teuthworker/src/git.ceph.com_teuthology_861a8dcf7aa816a26e13f039336f7ed0a9aec0fa/teuthology/run_tasks.py", line 154, in run_tasks suppress = manager.__exit__(*exc_info) File "/usr/lib/python3.10/contextlib.py", line 142, in __exit__ next(self.gen) File "/home/teuthworker/src/git.ceph.com_ceph_f1abf795f4fc15bf2f30ac90c811a53ab21d5d65/qa/tasks/ceph.py", line 1929, in task with contextutil.nested(*subtasks): File "/usr/lib/python3.10/contextlib.py", line 142, in __exit__ next(self.gen) File "/home/teuthworker/src/git.ceph.com_teuthology_861a8dcf7aa816a26e13f039336f7ed0a9aec0fa/teuthology/contextutil.py", line 54, in nested raise exc[1] File "/usr/lib/python3.10/contextlib.py", line 153, in __exit__ self.gen.throw(typ, value, traceback) File "/home/teuthworker/src/git.ceph.com_ceph_f1abf795f4fc15bf2f30ac90c811a53ab21d5d65/qa/tasks/ceph.py", line 251, in ceph_log yield File "/home/teuthworker/src/git.ceph.com_teuthology_861a8dcf7aa816a26e13f039336f7ed0a9aec0fa/teuthology/contextutil.py", line 46, in nested if exit(*exc): File "/usr/lib/python3.10/contextlib.py", line 142, in __exit__ next(self.gen) File "/home/teuthworker/src/git.ceph.com_ceph_f1abf795f4fc15bf2f30ac90c811a53ab21d5d65/qa/tasks/ceph.py", line 362, in valgrind_post raise valgrind_exception Exception: saw valgrind issues 2024-06-03T09:13:46.882 DEBUG:teuthology.run_tasks:Unwinding manager install 2024-06-03T09:13:46.891 ERROR:teuthology.contextutil:Saw exception from nested tasks Traceback (most recent call last): File "/home/teuthworker/src/git.ceph.com_teuthology_861a8dcf7aa816a26e13f039336f7ed0a9aec0fa/teuthology/contextutil.py", line 32, in nested yield vars File "/home/teuthworker/src/git.ceph.com_teuthology_861a8dcf7aa816a26e13f039336f7ed0a9aec0fa/teuthology/task/install/__init__.py", line 641, in task yield File "/home/teuthworker/src/git.ceph.com_teuthology_861a8dcf7aa816a26e13f039336f7ed0a9aec0fa/teuthology/run_tasks.py", line 154, in run_tasks suppress = manager.__exit__(*exc_info) File "/usr/lib/python3.10/contextlib.py", line 142, in __exit__ next(self.gen) File "/home/teuthworker/src/git.ceph.com_ceph_f1abf795f4fc15bf2f30ac90c811a53ab21d5d65/qa/tasks/ceph.py", line 1929, in task with contextutil.nested(*subtasks): File "/usr/lib/python3.10/contextlib.py", line 142, in __exit__ next(self.gen) File "/home/teuthworker/src/git.ceph.com_teuthology_861a8dcf7aa816a26e13f039336f7ed0a9aec0fa/teuthology/contextutil.py", line 54, in nested raise exc[1] File "/usr/lib/python3.10/contextlib.py", line 153, in __exit__ self.gen.throw(typ, value, traceback) File "/home/teuthworker/src/git.ceph.com_ceph_f1abf795f4fc15bf2f30ac90c811a53ab21d5d65/qa/tasks/ceph.py", line 251, in ceph_log yield File "/home/teuthworker/src/git.ceph.com_teuthology_861a8dcf7aa816a26e13f039336f7ed0a9aec0fa/teuthology/contextutil.py", line 46, in nested if exit(*exc): File "/usr/lib/python3.10/contextlib.py", line 142, in __exit__ next(self.gen) File "/home/teuthworker/src/git.ceph.com_ceph_f1abf795f4fc15bf2f30ac90c811a53ab21d5d65/qa/tasks/ceph.py", line 362, in valgrind_post raise valgrind_exception Exception: saw valgrind issues 2024-06-03T09:13:46.891 INFO:teuthology.task.install.util:Removing shipped files: /home/ubuntu/cephtest/valgrind.supp /usr/bin/daemon-helper /usr/bin/adjust-ulimits /usr/bin/stdin-killer... 2024-06-03T09:13:46.891 DEBUG:teuthology.orchestra.run.smithi088:> sudo rm -f -- /home/ubuntu/cephtest/valgrind.supp /usr/bin/daemon-helper /usr/bin/adjust-ulimits /usr/bin/stdin-killer 2024-06-03T09:13:46.894 DEBUG:teuthology.orchestra.run.smithi192:> sudo rm -f -- /home/ubuntu/cephtest/valgrind.supp /usr/bin/daemon-helper /usr/bin/adjust-ulimits /usr/bin/stdin-killer 2024-06-03T09:13:46.984 DEBUG:teuthology.run_tasks:Unwinding manager clock 2024-06-03T09:13:46.993 INFO:teuthology.task.clock:Checking final clock skew... 2024-06-03T09:13:46.993 DEBUG:teuthology.orchestra.run.smithi088:> PATH=/usr/bin:/usr/sbin ntpq -p || PATH=/usr/bin:/usr/sbin chronyc sources || true 2024-06-03T09:13:46.995 DEBUG:teuthology.orchestra.run.smithi192:> PATH=/usr/bin:/usr/sbin ntpq -p || PATH=/usr/bin:/usr/sbin chronyc sources || true 2024-06-03T09:13:47.010 INFO:teuthology.orchestra.run.smithi088.stderr:bash: line 1: ntpq: command not found 2024-06-03T09:13:47.013 INFO:teuthology.orchestra.run.smithi192.stderr:bash: line 1: ntpq: command not found 2024-06-03T09:13:47.015 INFO:teuthology.orchestra.run.smithi088.stdout:MS Name/IP address Stratum Poll Reach LastRx Last sample 2024-06-03T09:13:47.015 INFO:teuthology.orchestra.run.smithi088.stdout:=============================================================================== 2024-06-03T09:13:47.015 INFO:teuthology.orchestra.run.smithi088.stdout:^* hv01.front.sepia.ceph.com 3 8 377 163 -313us[ -378us] +/- 50ms 2024-06-03T09:13:47.015 INFO:teuthology.orchestra.run.smithi088.stdout:^+ hv02.front.sepia.ceph.com 3 7 377 101 -680us[ -680us] +/- 51ms 2024-06-03T09:13:47.015 INFO:teuthology.orchestra.run.smithi088.stdout:^+ hv03.front.sepia.ceph.com 3 7 377 98 +1227us[+1227us] +/- 70ms 2024-06-03T09:13:47.015 INFO:teuthology.orchestra.run.smithi088.stdout:^? hv04.front.sepia.ceph.com 0 9 0 - +0ns[ +0ns] +/- 0ns 2024-06-03T09:13:47.018 INFO:teuthology.orchestra.run.smithi192.stdout:MS Name/IP address Stratum Poll Reach LastRx Last sample 2024-06-03T09:13:47.018 INFO:teuthology.orchestra.run.smithi192.stdout:=============================================================================== 2024-06-03T09:13:47.018 INFO:teuthology.orchestra.run.smithi192.stdout:^* hv01.front.sepia.ceph.com 3 6 377 39 -319us[ -317us] +/- 52ms 2024-06-03T09:13:47.018 INFO:teuthology.orchestra.run.smithi192.stdout:^+ hv02.front.sepia.ceph.com 3 6 377 41 -675us[ -673us] +/- 51ms 2024-06-03T09:13:47.018 INFO:teuthology.orchestra.run.smithi192.stdout:^+ hv03.front.sepia.ceph.com 3 6 377 36 +1247us[+1247us] +/- 70ms 2024-06-03T09:13:47.018 INFO:teuthology.orchestra.run.smithi192.stdout:^? hv04.front.sepia.ceph.com 0 9 0 - +0ns[ +0ns] +/- 0ns 2024-06-03T09:13:47.019 DEBUG:teuthology.run_tasks:Unwinding manager ansible.cephlab 2024-06-03T09:13:47.028 INFO:teuthology.task.ansible:Skipping ansible cleanup... 2024-06-03T09:13:47.028 DEBUG:teuthology.run_tasks:Unwinding manager selinux 2024-06-03T09:13:47.037 DEBUG:teuthology.orchestra.run.smithi088:> mkdir /home/ubuntu/cephtest/archive/audit && sudo cp /var/log/audit/audit.log /home/ubuntu/cephtest/archive/audit && sudo chown $USER /home/ubuntu/cephtest/archive/audit/audit.log && gzip /home/ubuntu/cephtest/archive/audit/audit.log 2024-06-03T09:13:47.123 DEBUG:teuthology.orchestra.run.smithi192:> mkdir /home/ubuntu/cephtest/archive/audit && sudo cp /var/log/audit/audit.log /home/ubuntu/cephtest/archive/audit && sudo chown $USER /home/ubuntu/cephtest/archive/audit/audit.log && gzip /home/ubuntu/cephtest/archive/audit/audit.log 2024-06-03T09:13:47.190 DEBUG:teuthology.orchestra.run.smithi088:> sudo grep -a 'avc: .*denied' /var/log/audit/audit.log | grep -av -e 'comm="dmidecode"' -e chronyd.service -e 'name="cephtest"' -e scontext=system_u:system_r:nrpe_t:s0 -e scontext=system_u:system_r:pcp_pmlogger_t -e scontext=system_u:system_r:pcp_pmcd_t:s0 -e 'comm="rhsmd"' -e scontext=system_u:system_r:syslogd_t:s0 -e tcontext=system_u:system_r:nrpe_t:s0 -e 'comm="updatedb"' -e 'comm="smartd"' -e 'comm="rhsmcertd-worke"' -e 'comm="setroubleshootd"' -e 'comm="rpm"' -e tcontext=system_u:object_r:container_runtime_exec_t:s0 -e 'comm="ksmtuned"' -e 'comm="sssd"' -e 'comm="sss_cache"' -e context=system_u:system_r:NetworkManager_dispatcher_t:s0 2024-06-03T09:13:47.219 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-03T09:13:47.219 DEBUG:teuthology.orchestra.run.smithi192:> sudo grep -a 'avc: .*denied' /var/log/audit/audit.log | grep -av -e 'comm="dmidecode"' -e chronyd.service -e 'name="cephtest"' -e scontext=system_u:system_r:nrpe_t:s0 -e scontext=system_u:system_r:pcp_pmlogger_t -e scontext=system_u:system_r:pcp_pmcd_t:s0 -e 'comm="rhsmd"' -e scontext=system_u:system_r:syslogd_t:s0 -e tcontext=system_u:system_r:nrpe_t:s0 -e 'comm="updatedb"' -e 'comm="smartd"' -e 'comm="rhsmcertd-worke"' -e 'comm="setroubleshootd"' -e 'comm="rpm"' -e tcontext=system_u:object_r:container_runtime_exec_t:s0 -e 'comm="ksmtuned"' -e 'comm="sssd"' -e 'comm="sss_cache"' -e context=system_u:system_r:NetworkManager_dispatcher_t:s0 2024-06-03T09:13:47.248 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-03T09:13:47.249 DEBUG:teuthology.run_tasks:Unwinding manager pcp 2024-06-03T09:13:47.259 DEBUG:teuthology.run_tasks:Unwinding manager internal.timer 2024-06-03T09:13:47.269 INFO:teuthology.task.internal:Duration was 3150.162831 seconds 2024-06-03T09:13:47.269 DEBUG:teuthology.run_tasks:Unwinding manager internal.syslog 2024-06-03T09:13:47.278 INFO:teuthology.task.internal.syslog:Shutting down syslog monitoring... 2024-06-03T09:13:47.278 DEBUG:teuthology.orchestra.run.smithi088:> sudo rm -f -- /etc/rsyslog.d/80-cephtest.conf && sudo service rsyslog restart 2024-06-03T09:13:47.281 DEBUG:teuthology.orchestra.run.smithi192:> sudo rm -f -- /etc/rsyslog.d/80-cephtest.conf && sudo service rsyslog restart 2024-06-03T09:13:47.321 INFO:teuthology.orchestra.run.smithi088.stderr:Redirecting to /bin/systemctl restart rsyslog.service 2024-06-03T09:13:47.329 INFO:teuthology.orchestra.run.smithi192.stderr:Redirecting to /bin/systemctl restart rsyslog.service 2024-06-03T09:13:47.558 INFO:teuthology.task.internal.syslog:Checking logs for errors... 2024-06-03T09:13:47.558 DEBUG:teuthology.task.internal.syslog:Checking ubuntu@smithi088.front.sepia.ceph.com 2024-06-03T09:13:47.559 DEBUG:teuthology.orchestra.run.smithi088:> egrep --binary-files=text '\bBUG\b|\bINFO\b|\bDEADLOCK\b' /home/ubuntu/cephtest/archive/syslog/kern.log | grep -v 'task .* blocked for more than .* seconds' | grep -v 'lockdep is turned off' | grep -v 'trying to register non-static key' | grep -v 'DEBUG: fsize' | grep -v CRON | grep -v 'BUG: bad unlock balance detected' | grep -v 'inconsistent lock state' | grep -v '*** DEADLOCK ***' | grep -v 'INFO: possible irq lock inversion dependency detected' | grep -v 'INFO: NMI handler (perf_event_nmi_handler) took too long to run' | grep -v 'INFO: recovery required on readonly' | grep -v 'ceph-create-keys: INFO' | grep -v INFO:ceph-create-keys | grep -v 'Loaded datasource DataSourceOpenStack' | grep -v 'container-storage-setup: INFO: Volume group backing root filesystem could not be determined' | egrep -v '\bsalt-master\b|\bsalt-minion\b|\bsalt-api\b' | grep -v ceph-crash | egrep -v '\btcmu-runner\b.*\bINFO\b' | head -n 1 2024-06-03T09:13:47.595 DEBUG:teuthology.task.internal.syslog:Checking ubuntu@smithi192.front.sepia.ceph.com 2024-06-03T09:13:47.595 DEBUG:teuthology.orchestra.run.smithi192:> egrep --binary-files=text '\bBUG\b|\bINFO\b|\bDEADLOCK\b' /home/ubuntu/cephtest/archive/syslog/kern.log | grep -v 'task .* blocked for more than .* seconds' | grep -v 'lockdep is turned off' | grep -v 'trying to register non-static key' | grep -v 'DEBUG: fsize' | grep -v CRON | grep -v 'BUG: bad unlock balance detected' | grep -v 'inconsistent lock state' | grep -v '*** DEADLOCK ***' | grep -v 'INFO: possible irq lock inversion dependency detected' | grep -v 'INFO: NMI handler (perf_event_nmi_handler) took too long to run' | grep -v 'INFO: recovery required on readonly' | grep -v 'ceph-create-keys: INFO' | grep -v INFO:ceph-create-keys | grep -v 'Loaded datasource DataSourceOpenStack' | grep -v 'container-storage-setup: INFO: Volume group backing root filesystem could not be determined' | egrep -v '\bsalt-master\b|\bsalt-minion\b|\bsalt-api\b' | grep -v ceph-crash | egrep -v '\btcmu-runner\b.*\bINFO\b' | head -n 1 2024-06-03T09:13:47.642 INFO:teuthology.task.internal.syslog:Compressing syslogs... 2024-06-03T09:13:47.643 DEBUG:teuthology.orchestra.run.smithi088:> find /home/ubuntu/cephtest/archive/syslog -name '*.log' -print0 | sudo xargs -0 --no-run-if-empty -- gzip -- 2024-06-03T09:13:47.645 DEBUG:teuthology.orchestra.run.smithi192:> find /home/ubuntu/cephtest/archive/syslog -name '*.log' -print0 | sudo xargs -0 --no-run-if-empty -- gzip -- 2024-06-03T09:13:47.710 INFO:teuthology.task.internal.syslog:Gathering journactl -b0... 2024-06-03T09:13:47.710 DEBUG:teuthology.orchestra.run.smithi088:> sudo journalctl -b0 | gzip -9 > /home/ubuntu/cephtest/archive/syslog/journalctl-b0.gz 2024-06-03T09:13:47.712 DEBUG:teuthology.orchestra.run.smithi192:> sudo journalctl -b0 | gzip -9 > /home/ubuntu/cephtest/archive/syslog/journalctl-b0.gz 2024-06-03T09:13:47.849 DEBUG:teuthology.run_tasks:Unwinding manager internal.sudo 2024-06-03T09:13:47.860 INFO:teuthology.task.internal:Restoring /etc/sudoers... 2024-06-03T09:13:47.860 DEBUG:teuthology.orchestra.run.smithi088:> sudo mv -f /etc/sudoers.orig.teuthology /etc/sudoers 2024-06-03T09:13:47.885 DEBUG:teuthology.orchestra.run.smithi192:> sudo mv -f /etc/sudoers.orig.teuthology /etc/sudoers 2024-06-03T09:13:47.910 DEBUG:teuthology.run_tasks:Unwinding manager internal.coredump 2024-06-03T09:13:47.919 DEBUG:teuthology.orchestra.run.smithi088:> sudo sysctl -w kernel.core_pattern=core && sudo bash -c 'for f in `find /home/ubuntu/cephtest/archive/coredump -type f`; do file $f | grep -q systemd-sysusers && rm $f || true ; done' && rmdir --ignore-fail-on-non-empty -- /home/ubuntu/cephtest/archive/coredump 2024-06-03T09:13:47.927 DEBUG:teuthology.orchestra.run.smithi192:> sudo sysctl -w kernel.core_pattern=core && sudo bash -c 'for f in `find /home/ubuntu/cephtest/archive/coredump -type f`; do file $f | grep -q systemd-sysusers && rm $f || true ; done' && rmdir --ignore-fail-on-non-empty -- /home/ubuntu/cephtest/archive/coredump 2024-06-03T09:13:47.948 INFO:teuthology.orchestra.run.smithi088.stdout:kernel.core_pattern = core 2024-06-03T09:13:47.972 INFO:teuthology.orchestra.run.smithi192.stdout:kernel.core_pattern = core 2024-06-03T09:13:47.997 DEBUG:teuthology.orchestra.run.smithi088:> test -e /home/ubuntu/cephtest/archive/coredump 2024-06-03T09:13:48.015 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-03T09:13:48.016 DEBUG:teuthology.orchestra.run.smithi192:> test -e /home/ubuntu/cephtest/archive/coredump 2024-06-03T09:13:48.051 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-03T09:13:48.052 DEBUG:teuthology.run_tasks:Unwinding manager internal.archive 2024-06-03T09:13:48.062 INFO:teuthology.task.internal:Transferring archived files... 2024-06-03T09:13:48.062 DEBUG:teuthology.misc:Transferring archived files from smithi088:/home/ubuntu/cephtest/archive to /home/teuthworker/archive/teuthology-2024-06-02_22:00:03-rados-reef-distro-default-smithi/7739049/remote/smithi088 2024-06-03T09:13:48.063 DEBUG:teuthology.orchestra.run.smithi088:> sudo tar c -f - -C /home/ubuntu/cephtest/archive -- . 2024-06-03T09:13:48.097 DEBUG:teuthology.misc:Transferring archived files from smithi192:/home/ubuntu/cephtest/archive to /home/teuthworker/archive/teuthology-2024-06-02_22:00:03-rados-reef-distro-default-smithi/7739049/remote/smithi192 2024-06-03T09:13:48.098 DEBUG:teuthology.orchestra.run.smithi192:> sudo tar c -f - -C /home/ubuntu/cephtest/archive -- . 2024-06-03T09:13:48.131 INFO:teuthology.task.internal:Removing archive directory... 2024-06-03T09:13:48.131 DEBUG:teuthology.orchestra.run.smithi088:> rm -rf -- /home/ubuntu/cephtest/archive 2024-06-03T09:13:48.134 DEBUG:teuthology.orchestra.run.smithi192:> rm -rf -- /home/ubuntu/cephtest/archive 2024-06-03T09:13:48.180 DEBUG:teuthology.run_tasks:Unwinding manager internal.archive_upload 2024-06-03T09:13:48.191 INFO:teuthology.task.internal:Not uploading archives. 2024-06-03T09:13:48.192 DEBUG:teuthology.run_tasks:Unwinding manager internal.base 2024-06-03T09:13:48.202 INFO:teuthology.task.internal:Tidying up after the test... 2024-06-03T09:13:48.202 DEBUG:teuthology.orchestra.run.smithi088:> find /home/ubuntu/cephtest -ls ; rmdir -- /home/ubuntu/cephtest 2024-06-03T09:13:48.205 DEBUG:teuthology.orchestra.run.smithi192:> find /home/ubuntu/cephtest -ls ; rmdir -- /home/ubuntu/cephtest 2024-06-03T09:13:48.217 INFO:teuthology.orchestra.run.smithi088.stdout: 265304 4 drwxr-xr-x 2 ubuntu ubuntu 4096 Jun 3 09:13 /home/ubuntu/cephtest 2024-06-03T09:13:48.234 INFO:teuthology.orchestra.run.smithi192.stdout: 265304 4 drwxr-xr-x 2 ubuntu ubuntu 4096 Jun 3 09:13 /home/ubuntu/cephtest 2024-06-03T09:13:48.236 DEBUG:teuthology.run_tasks:Unwinding manager kernel 2024-06-03T09:13:48.247 DEBUG:teuthology.run_tasks:Unwinding manager console_log 2024-06-03T09:13:48.257 DEBUG:teuthology.run_tasks:Exception was not quenched, exiting: Exception: saw valgrind issues 2024-06-03T09:13:48.260 INFO:teuthology.run:Summary data: description: rados/verify/{centos_latest ceph clusters/{fixed-2 openstack} d-thrash/none mon_election/connectivity msgr-failures/few msgr/async-v1only objectstore/bluestore-stupid rados tasks/rados_api_tests validater/valgrind} duration: 3150.162831068039 failure_reason: '"2024-06-03T08:40:55.510454+0000 osd.1 (osd.1) 6 : cluster [WRN] osd.1 ep: 122 scrubber::ReplicaReservations pg[11.7]: timeout on replica reservations (since 2024-06-03 08:40:50)" in cluster log' flavor: default owner: scheduled_teuthology@teuthology status: fail success: false 2024-06-03T09:13:48.260 DEBUG:teuthology.report:Pushing job info to https://paddles.front.sepia.ceph.com/ 2024-06-03T09:13:48.334 INFO:teuthology.run:FAIL